Job Array Example
This page shows an example of submitting and running a Slurm job array. This is a set of similar jobs submitted with a single sbatch command and job script.
Job arrays are useful when submitting large numbers of similar jobs, like one might in high throughput cases or with embarassingly parallel calculations. Although you could in theory submit each job with a separate sbatch command (or write a simple script to do that), that can put a serious load on the scheduler when the number of jobs runs into the hundreds or more. Instead, by using a job array, the whole set of jobs becomes a single entry in the scheduler. The individual element jobs of the array can be referenced by a job number of the form BASEJOBNUM_INDEX where BASEJOBNUM is a base job number given to the entire array, and INDEX is the index to the job within the array.
The scheduler then periodically sees if there are enough resources to start the next job in the array. If there are, that element gets it own job number (although it can still be referenced as BASEJOBNUM_INDEX) and a separate entry in the Slurm job table. If a job starts, the scheduler can check if the next job in the array can start, and so forth; if the next job in the array cannot start, the scheduler knows that none of the jobs in the array can start, and can look at the next job in the queue. This can greatly reduce the load on the scheduler since determining whether there are resources to start a job can be quite resource intensive.
The job script for all of the jobs are the same, so all of the jobs in the array must request identical amounts of HPC resources. You can use the following environmental variables in your script for information about the job array:
- SLURM_ARRAY_JOB_ID: The base job id shared by all elements of the array
- SLURM_ARRAY_TASK_ID: The index into the job array for the current job. This can be used to make the job script perform different calculations for each job index
- SLURM_ARRAY_TASK_COUNT: The total number of elements in the array
- SLURM_ARRAY_TASK_MIN: The smallest index in the array
- SLURM_ARRAY_TASK_MAX: The largest index in the array
Overview
The example shown is a rather trivial case of a job array. We invoke a python script which prints out the prime factorization of the integer given on the command line.
We run it as a job array, with each element in the array being a job which factors a given integer. The integers to factor are contained in a list defined in the job script, and each job in the array will use the SLURM_ARRAY_TASK_ID variable to find the number it should factor.
The submission script
The submission script can be downloaded as plain text, suitable for you to submit to the scheduler with the sbatch command. We present a copy with line numbers for discussion below.
Source of submission script
The line numbers in the listing below are to facilitate a line-by-line discussion of the script; they should not present in the actual script. Downloaded a plain text version of the script.
Example
1 #!/bin/bash
2 # The line above this is the "shebang" line. It must be first line in script
3 #-----------------------------------------------------
4 # Example Submission Script: job array example
5 # Runs a simple python script to show n-th element of the
6 # Fibonacci sequence
7 #-----------------------------------------------------
8 #
9 # Job array, with 10 elements (0 through 9)
10 #SBATCH --array=0-9
11 # Sequential code, 1 core per job array element
12 #SBATCH -c 1
13 # Request 5 minutes of walltime per job array element
14 #SBATCH -t 10
15 # Allow other jobs to run on same node
16 #SBATCH --oversubscribe
17 # Run on debug partition for rapid turnaround. You will need
18 # to change this (remove the line) if walltime > 15 minutes
19 #SBATCH --partition=debug
20 # Process our dot files (and in particular define the "module" command)
21 source /etc/profile
22 # Module load section
23 # First clear our module list
24 module purge
25 # and reload the standard modules
26 module load hpcc/zaratan
27 # Load the desired compiler, MPI, and python modules
28 # NOTE: You need to use the same compiler and MPI module used
29 # when compiling the python (and its mpi4py package). Module load-ing
30 # python first, followed by openmpi will ensure you get a compatible set
31 module load python
32 # Numbers to factor, one for each job in the array
33 numbers_to_factor=(
34 54
35 1133
36 400219
37 400221
38 400227
39 400239
40 6700489
41 9990123
42 12240321
43 76340789
44 )
45 # Get the number to factor for this job in the array
46 N="${numbers_to_factor[$SLURM_ARRAY_TASK_ID]}"
47 # Get the full path to our python executable. It is best
48 # to provide full path of our executable to , etc.
49 MYPYTHON=`which python`
50 MYPYSCRIPT="${SLURM_SUBMIT_DIR}/factorize.py"
51 echo "Slurm job ${SLURM_JOBID} (${SLURM_ARRAY_JOB_ID} task ${SLURM_ARRAY_TASK_ID})"
52 echo "Slurm job {$SLURM_JOB_NAME} running on:"
53 hostname
54 echo "Ntasks=${SLURM_NTASKS}; CPU cores per task=${SLURM_CPUS_PER_TASK}"
55 echo "On ${SLURM_JOB_NUM_NODES} nodes: ${SLURM_JOB_NODELIST}"
56 date
57 pwd
58 echo "Loaded modules are:"
59 module list
60 echo "Using python $MYPYTHON"
61 echo "Using python script $MYPYSCRIPT"
62 echo "N=${N}"
63 # Run the python script, saving the results to slurm-JOBNUM_TASKID.out
64 $MYPYTHON $MYPYSCRIPT $N
65 # Save the exit code from the previous command
66 ECODE=$?
67 echo "Job finished with exit code $ECODE."
68 date
69 # Exit with the cached exit code
70 exit $ECODE
Discussion of submission script
The following is a detailed, more or less line-by-line discussion of the job submission script:
- Line 1: This shebang line
specifies that the script is to be processed using the
bash shell.
For more information on the shebang line. - Lines 2-8: These are comment lines,
which are ignored by the bash shell.
They are only present to make the script more readable to humans. For more information on comment lines. - Lines 9-19: These lines specify various flags to be passed to the sbatch command. For more information about #SBATCH directives. With the exception of the "--array" directive in lines 9-10, the directive act on each individual job in the job array.
- Lines 9-10: This line defines the array indices we should use for this job array. This is what makes it this job script a job array script instead of a normal job script; it causes the Slurm scheduler to treat this as a job array submission. In this case, an array with ten elements numbers 0 through 9.
- Lines 11-12: As this is a simple job, we request a single CPU cores for each job in the array.
- Lines 13-14: This is a trivial job, so we only request 10 minutes for it. It should finish in well under 10 minutes. For more information about maximum walltime settings
- Lines 15-16: This
#SBATCHdirective says that we are allowing other jobs to share the same node(s) as our job. This is the default (so this directive can usually be omitted) and is recommended, especially for jobs using only a fraction of the cores available on a node (128 cores per node on the Zaratan cluster), otherwise a large number of the cores on the node will not be used by your job nor will the scheduler allow other jobs to use them. For more information about exclusive/shared mode. - Lines 17-19: This tells the scheduler to put our job in the debug partition. Since this is a short tutorial job, the debug partition is fine and generally will allow the job to spend less time in the queue. For more information about selecting partitions.
- Lines 20-31: These lines set up the
environment within which to run your job. These are explained in
more detail in the main job examples
page, but in this case the lines
basically:
- Ensure the module command is defined
- Clear out any previously loaded modules (inherited from .bashrc or similar)
- Load the modules for python
- Lines 32-46: These lines define the list of numbers to factor, one for each element of the job array. Line 44 then defines the variable N to be the number for this specific job in the array.
- Lines 47-50: These lines get the paths for our python interpretter and python script, storing in the environmental variables MYPYTHON and MYPYSCRIPT.
- Lines 51-62: These lines print out various pieces of information to identify the job. The output will go into the file "slurm-JOB_NUMBER_TASKID.out" in the directory from which you issued the sbatch command (where JOB_NUMBER is the Slurm job number and TASKID is the index in the job array). There are options you can pass to sbatch to change the name of the file.
- Line 64: This line actually does the work. It invokes the python interpretter to invoke our python script to do the actual factorization for each number in our list. Because the output is short, we just allow the output of the factorize.py script go to the Slurm standard output script. (This should only be done when you know the output is small, like in this case).
- Line 66: This line saves the exit code
from the python command so that be used when the script ends.
For more information about exit codes. - Lines 67-68: These lines print
some additional information just before the job terminates.
These lines will go in the same file as those from the start of the script, usually "slurm-JOBNUMBER_TASKID.out".
This just prints out:- The exit code from the
hello-umdcommand (as stashed in theECODEenvironment variable) - The time when the job finished
- The exit code from the
- Line 70: This line actually ends the job
script, and terminates the job. The script and job will actually
end at the end of the file even without an explicit
exitcommand, but in that case the exit code for the job script will be that of the last command which ran, which typically will be part of the cleanup commands, and not the meat of the script.
Here we explicitly set the exit code to the exit code from thehello-umdcommand (which we previously saved in the variableECODE).
The python script
The pythonscript can be downloaded as plain text. We present a copy with line numbers for discussion below.
Source of python script
The line numbers in the listing below are to facilitate a line-by-line discussion pf the script; they should not present in the actual script. Downloaded as plain text version of the script.
Example
1 #!/bin/env python
2 #
3 # Simple python code for brute-force factorization of number
4 import sys
5 import math
6 def prime_factorize(n):
7 """Returns the prime factors of n.
8 Return value is a dictionary, key is a prime number which
9 divides n and value is the power of that prime number in
10 the prime factorization of n.
11 E.g. prime_factorize(24) would return { 2: 3, 3:1 }
12 as 24 = 2^3 * 3 ^1
13 """
14 def how_many_times_is_a_factor(n,m):
15 """See if m divides n, and how many time
16 Returns power p and newn where
17 n = newn * m^p
18 If m does not divide n, returns p=0 and newn=n
19 """
20 newn = n
21 p = 0
22 while newn % m == 0:
23 p = p + 1
24 newn = int(newn / m)
25 return p, newn
26 results = {}
27 # Handle special cases
28 if n < 1:
29 raise ValueError("n must be a positive integer")
30 if n == 1:
31 return { 1:1 }
32 # If n has a factor, it must be less than max
33 max = int(math.floor( math.sqrt(n) ) + 1)
34 # Start with 2
35 i = 2
36 while i < max and n > 1:
37 p, newn = how_many_times_is_a_factor(n, i)
38 if p > 0:
39 # Found a factor
40 results[i] = p
42 n = newn
43 max = int(math.floor( math.sqrt(n) ) + 1)
44 i = i + 1
45 # Add n as factor if > 1
46 if n > 1:
47 results[n] = 1
48 return results
49 def print_prime_factorization(n):
50 """Print the prime factorization of n
51 Uses prime_factorize function
52 """
53 results = prime_factorize(n)
54 sys.stdout.write("Prime factorization of {}\n".format(n))
55 sys.stdout.write("{}=\n".format(n))
56 keys =sorted( results.keys() )
57 for prime in keys:
58 power = results[prime]
59 sys.stdout.write("\t{}^{}\n".format(prime,power))
60 return
61 if ( __name__ == '__main__' ):
62 # Run the function if called as a script
63 import argparse
64 parser = argparse.ArgumentParser(
65 prog='factorize',
66 description='Prints the prime factorization of a number'
67 )
68 parser.add_argument('n')
69 args = parser.parse_args()
70 n = int(args.n)
71 print_prime_factorization(n)
Discussion of python script
The following is a detailed, more or less line-by-line discussion of the python script:
- Line 1: This shebang line
specifies that the script is to be processed as a python script.
The use of
/bin/env pythonmeans that it will run under thepythoncommand found in the PATH of the calling process.
As we will be invoking the script as the argument of the anpythoncommand, it is not really necessary (the shebang line really only is used if one tries to execute the script directly). But it is good practice to include it to identify this as a python script. - Lines 2-3: These lines are comments, describing the python script.
- Lines 4-5: These lines import some useful Python packages, in particular the "sys" package (for writing to standard output and error) and "math" (for the floor and sqrt functions).
- Lines 6-48: These lines define the python function "prime_factorize",which does the main work of this script. It basically takes a positive integer n, and returns a dictionary containing the factorization of the integer n. The keys of the returned dictionary are prime factors of n, and the value is the power of that factor. E.g., for n=12, the results would be a dictionary with keys 2 with value of 2, and 3 with value of 1, as 12 = 2^2 * 3^1.
- Lines 7-13: These lines form a python docstring, which are used to document the function "prime_factorize". It is essentially a special form of comment.
- Lines 15-25: These lines define a helper function "how_many_times_is_a_factor" within the "prime_factorize" function. This function returns the number of times one number goes into another; returning 0 if it is not a divisor, 1 if it goes into it only once, 2 if the square is a divisor, 3 if the cube is a divisor, and so forth. It also returns the result of dividing the original number by the power of the divisor.
- Lines 16-19: Another docstring, this time for the helper function.
- Lines 20-21: These lines initialize some variables; p which is the number of times the divisor divides n, and newn the result of the division.
- Lines 22-24: These is a while loop which keeps dividing the newn by the divisor until it can no longer do so. It increments p and resets newn with each iteration.
- Line 25: This line terminates the helper function, returning the power and the result of dividing the original number by that power of the divisor.
- Line 26: We are now back in the body of the "prime_factorize" function, and initializing the return dictionary to an empty dictionary.
- Lines 27-31: This raises an exception if the given value is not positive, and handling the special case if n equals 1.
- Line 33: This computes the square root of n, to find the maximum value of a factor of n.
- Line 35: This initializes the first factor to try; the smallest prime number 2.
- Lines 36-44: This loops over the potential factors. If a factor is found (using the helper function), we determine its power and add it to the results dictionary, and divide the initial input by that power of the divisor.
- Lines 45-47: If we have not fully factorized our input, what is left over in n is then the last prime factor, so we add it to the results dictionary.
- Line 48: This line returns our dictionary.
- Lines 49-60: This defines a function "print_prime_factorization" which invokes "prime_factorization" and then prints the results.
- Lines 61-73: These lines are only invoked if the script is executed as a script (as oppsed to being imported). It imports the "argparse" package and uses it to parse the value of n from the command line. It then converts it to an intenger, and then invokes "print_prime_factorization" to factorize n and print the results.
Submitting the example
To submit the script, you can just:
- Download the submission script and
Python script as plain text and copy
them to one of the cluster login nodes.
Move them to an appropriate directory. - From the directory with the job script, run the command
sbatch submit-python-job-array.sh(replacing the sbatch argument with the name you called the script you downloaded). The scheduler should return something likeSubmitted batch job 11964119--- the number will vary and is the base job number for this job array). The job number can be used to reference the job in various Slurm commands, etc. (Please always give the job number(s) when requesting help about a job you submitted).
The job will be queued for partition specified (or defaulted to); in this example we specified the the debug partition so it will be in that partition. One by one the jobs in the array should start up once resources are available (if the debug partition is not very busy, it is likely they will all start up almost at the same time).
If the debug partition is busier (or you submit a real job array), you will likely see something like:
Example
login-2:~$ sbatch submit-python-job-array.sh
Submitted batch job 11964119
login-2:~$
login-2:~$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
11964119_[0-9] debug submit-p clarkkent PD 0:00 1 (None)
login-2:~$
login-2:~$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
11964119_[2-9] debug submit-p clarkkent PD 0:00 1 (Resources)
11964119_0 debug submit-p clarkkent R 0:00 1 compute-b8-57
11964119_1 debug submit-p clarkkent R 0:00 1 compute-b8-57
login-2:~$
login-2:~$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
11964119_[4-9] debug submit-p clarkkent PD 0:00 1 (Resources)
11964119_2 debug submit-p clarkkent R 0:00 1 compute-b8-58
11964119_3 debug submit-p clarkkent R 0:00 1 compute-b8-58
login-2:~$
login-2:~$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
11964119_3 debug submit-p clarkkent CG 0:01 1 compute-b8-58
11964119_[5-9] debug submit-p clarkkent PD 0:00 1 (Resources)
11964119_4 debug submit-p clarkkent R 0:01 1 compute-b8-57
login-2:~$
login-2:~$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
11964119_5 debug submit-p clarkkent CG 0:00 1 compute-b8-57
11964119_[7-9] debug submit-p clarkkent PD 0:00 1 (Resources)
11964119_6 debug submit-p clarkkent R 0:01 1 compute-b8-59
login-2:~$
login-2:~$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
11964119_7 debug submit-p clarkkent CG 0:00 1 compute-b8-58
11964119_[8-9] debug submit-p clarkkent PD 0:00 1 (Resources)
11964119_6 debug submit-p clarkkent R 0:02 1 compute-b8-59
login-2:~$
login-2:~$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
11964119_8 debug submit-p clarkkent CG 0:00 1 compute-b8-57
11964119_9 debug submit-p clarkkent R 0:01 1 compute-b8-57
login-2:~$
login-2:~$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
As you can can see in the above script, immediately after submitting the job array, we see a single entry with the squeue command. This uses a slightly different notation then you might be used to, with "11964119_[0-9]" in the field labels "JOBID". This can be read as tasks 0 through 9 of the job array with id 11964119. As previously mentioned, it appears to the scheduler as a single entry, even though it represents 10 jobs. After a little while, the scheduler finds resources to start the first 2 tasks, which show up as separate entries in the next squeue output, and only tasks 2-9 are still pending.
As we repeatedly call squeue, additional tasks are promoted to full-fledged jobs and start to run as resources become available (in this case when the other taks finish, but in general many tasks can run at once).
When finished, you should have a collection of slurm output files with names like "slurm-JOBNUMBER_TASKNUMBER". These files should look like:
Example
Loading python/gcc/11.3.0/nocuda/linux-rhel8-zen2/3.10.10
Loading requirement: gcc/11.3.0
Slurm job 11964128 (11964119 task 8)
Slurm job {submit-python-factorize.sh} running on:
compute-b8-57.zaratan.umd.edu
Ntasks=; CPU cores per task=1
On 1 nodes: compute-b8-57
Tue May 20 17:30:52 EDT 2025
/home/payerle/example-jobs/job-array-factorize
Loaded modules are:
Currently Loaded Modulefiles:
1) hpcc/zaratan(default)
2) gcc/11.3.0
3) python/gcc/11.3.0/nocuda/linux-rhel8-zen2/3.10.10
Using python /cvmfs/hpcsw.umd.edu/spack-software/2023.11.20/views/2023/linux-rhel8-zen2/gcc@11.3.0/python-3.10.10/mpi-nocuda/linux-rhel8-zen2/gcc/11.3.0/bin/python
Using python script /home/payerle/example-jobs/job-array-factorize/factorize.py
N=12240321
Prime factorization of 12240321
12240321=
3^1
61^1
211^1
317^1
Job finished with exit code 0.
Tue May 20 17:30:52 EDT 2025
