FF chevron

Multithreaded Job Submission Example

This page shows an example of submitting a job that uses multithreading for parallelization. This example is for purely multithreaded jobs without any MPI parallelization --- for an example of a job in which both MPI and multithreaded are used, see the hybrid job example.

Many of the codes used on HPC clusters support parallelizatin via MPI, and so this example can generally be adapted to support such.

Overview

This example basically consists of a single file, the job script submit.sh (see for a listing and explanation of the script) which gets submitted to the cluster via the sbatch command.

The script is designed to show many good practices; including: - setting standard sbatch options within the script - loading the needed modules within the script - printing some useful diagnostic information at start of the script - creating a job specific temporary work directory - running the code and saving the exit code - copying back any files that should be retained - exiting with the exit code from the main application

Many of the practices above are rather overkill for such a simple job --- indeed, the vast majority of lines are for these "good practices" rather than the running of the intended code, but are included for educational purposes.

The example shown is a rather trivial Hello World type program in which each thread prints out a "Hello" message identifying the thread number and task number (always task 0 of 1 for this case), along with the hostname and process ID. Although trivial, it shows the basics of submitting such a job, along with various other good practices, including:

setting standard sbatch options within the script
loading the needed modules within the script
printing some useful diagnostic information at start of the script
creating a job specific temporary work directory
running the code and saving the exit code
copying important output files back to the submission directory
exiting with the exit code from the main application

Many of the practices above are rather overkill for such a simple job --- indeed, the vast majority of lines are for these "good practices" rather than the running of the intended code, but are included for educational purposes.

The submission script

The submission script can be downloaded as plain text, suitable for you to submit to the scheduler with the sbatch command. We present a copy with line numbers for discussion below.

Source of submit.sh

The line numbers in the listing below are to facilitate a line-by-line discussion pf the script; they should not present in the actual script. Downloaded as plain text version of the script.

Example

1  #!/bin/bash
2  # The line above this is the "shebang" line.  It must be first line in script
3  #-----------------------------------------------------
4  #    Example Submission Script: Hello-UMD, multithreaded
5  #    Runs a multithreaded hello-umd command
6  #-----------------------------------------------------
7  #
8  # Slurm sbatch parameters section:
9  #    Request a single task using 10 CPU cores
10 #SBATCH --ntasks=1
11 #SBATCH --cpus-per-task=10
12 #    Request 5 minute of walltime
13 #SBATCH -t 5
14 #    Request 10 GiB of total memory for the job
15 #SBATCH --mem=10240
16 #    Allow other jobs to run on same node
17 #SBATCH --oversubscribe
18 #    Run on debug partition for rapid turnaround.  You will need
19 #    to change this (remove the line) if walltime > 15 minutes
20 #SBATCH --partition=debug

21 # This job will run our hello-umd demo binary in multithreaded mode
22 # Output will go to local /tmp scratch space on the node we are running
23 # on, and then will be copied back to our work directory.

24 # Process our dot files (and in particular define the "module" command)
25 source /etc/profile

26 # Module load section
27 # First clear our module list 
28 module purge
29 # and reload the standard modules
30 module load hpcc/zaratan
31 # Set up environment for hello-umd
32 module load hello-umd

33 # Section to make a scratch directory for this job
34 # Since this is a single node job, the local /tmp filesystem is a good choice
35 # We include the SLURM jobid in the #directory name to avoid interference if 
36 # multiple jobs running at same time.
37 TMPWORKDIR="/tmp/job.${SLURM_JOB_ID}"
38 mkdir $TMPWORKDIR
39 cd $TMPWORKDIR

40 # Section to output information identifying the job, etc.
41 echo "Slurm job submit-hello-umd-multithreaded.sh ${SLURM_JOB_ID} running on:"
42 hostname
43 echo "Ntasks=${SLURM_NTASKS}; CPU cores per task=${SLURM_CPUS_PER_TASK}"
44 echo "On ${SLURM_JOB_NUM_NODES} nodes: ${SLURM_JOB_NODELIST}"
45 date
46 pwd
47 echo "Loaded modules are:"
48 module list

49 # Run our code, giving -t 0 to use all available CPUs (10 in this case)
50 hello-umd -t 0 > hello.out 2>&1
51 # Save the exit code from the previous command
52 ECODE=$?

53 # Copy results back to submit dir
54 cp hello.out ${SLURM_SUBMIT_DIR}/hello-${SLURM_JOB_ID}.out

55 echo "Job finished with exit code $ECODE"
56 date

57 # Exit with the cached exit code
58 exit $ECODE

Discussion of submit.sh

The following is a detailed, more or less line-by-line discussion of the job submission script:

Line 1: This shebang line specifies that the script is to be processed using the bash shell.
For more information on the shebang line.
Lines 2-7: These are comment lines, which are ignored by the bash shell.
They are only present to make the script more readable to humans. For more information on comment lines.
Lines 8-20: These lines specify various flags to be passed to the sbatch command. For more information about #SBATCH directives.
Lines 10-11: As this is a multithreaded job, we request a single single task. with 10 CPU cores per task, for a total of 10 cores. As all resources for a task are allocated to the same node by the scheduler, these 10 cores will all be allocated on the same node.
Line 13: This is a trivial job, so we only request 5 minutes for it. It should finish in well under 5 minutes. For more information about maximum walltime settings
Line 15: This is a trivial job which does not need much RAM, so we request a total of 10 GiB for it, and even that is well over what the job probably needs. We used the --mem flag to request the total memory for the job; we could have used --mem-per-cpu=1024 to request 1 GiB per CPU core (which also would allocation 10 GiB for the job), but for multithreaded jobs the memory requirement is typically independent of the number of threads, so specifying the total amount of memory is usually more convenient. For more information about memory settings
Line 17: This #SBATCH directive says that we are allowing other jobs to share the same node(s) as our job. This is the default (so this directive can usually be omitted) and is recommended, especially for jobs using only a fraction of the cores on a node.
Otherwise, a significant number of cores would not be used by your job and would sit idly since other jobs would be prevented from using them --- this is an inefficient use of the cluster and your allocation account. For more information about exclusive/shared mode.
Line 20: This tells the scheduler to put our job in the debug partition. Since this is a short tutorial job, the debug partition is fine and generally will allow the job to spend less time in the queue. For more information about selecting partitions.
Lines 24-32: These lines set up the environment within which to run your job. These are explained in more detail in the main job examples page, but in this case the lines basically:
- Ensure the module command is defined
- Clear out any previously loaded modules (inherited from .bashrc or similar)
- Load the module for hello-umd
Lines 33-39: These lines set up a work directory for our job, in this case since it is a multithreaded job and therefore necessarily a single node job, we use a work directory in the node local /tmp filesystem. For more information about setting up work directories.
Lines 40-48: These lines print out various pieces of information to identify the job. The output will go into the file "slurm-JOB_NUMBER.out" in the directory from which you issued the sbatch command (where JOB_NUMBER is the Slurm job number). There are options you can pass to sbatch to change the name of the file.
Line 50: This line actually runs the hello-umd command, saving all output to the file hello.out in the temporary working directory created in lines 33-39. I.e., this one line is the meat of the job script; everything else basically just exists to get to this point or cleanup afterwards.

When writing your own job script to do something useful, this is where you will be putting most of your effort, possibly expanding to multiple lines.

In this simple case, we basically just run the hello-umd command with the flag -t 0. The -t flag instructs hello-umd to run with the specified number of threads; by default it runs a single thread.
The value 0 is a special value meaning that it should start a thread on every CPU core it finds; the scheduler is set up so that the job can only see the cores allocated to it, so this will run one thread for each of the 10 cores allocated to the job. We could also have used -t $SLURM_CPUS_PER_TASK which would have doen the same thing (using the SLURM_CPUS_PER_TASK variable set by Slurm to the value of the --cpus-per-task or -c option from line 11). You could have also simple used -t 10, but we recommend using one of the other methods for consistency; at some point you might try running with a different number of threads, and if you used -t 0 or -t $SLURM_CPUS_PER_TASK in this line you would only need to change the value in line 11. Otherwise you would have to change both lines, and if they are not in sync it would lead to in efficient usage of the cluster.

The "> hello.out" part of this command means that the standard output from the command will be saved in the file hello.out in the working directory. The "2>&1" part means that any output to the standard error output stream will be redirected to the standard output. (NOTE: if you wish to save both standard output and standard error to the same file, you need to give these output redirection arguments in that order; redirect standard output first, then redirect standard error to standard output.)
Line 52: This line saves the exit code from the hello-umd command so thatt be used when the script ends.
For more information about exit codes.
Lines 53-56: These lines just do some cleanup and other minor tasks at the end of the job.
Line 54: This line copies the hello.out file from our working directory in the node local /tmp filesystem back to the directory from which we submitted the job (i.e. where you ran the sbatch command). We add the Slurm job number to the file name during the copy; this prevents the clobbering of the output file if we submit the job multiple times.

This step is needed because we were using the node local /tmp filesystem, which will be deleted after the job terminates. If you modified the script to use the scratch filesystem instead, you can skip this set, as the scratch filesystem is more permanent.

NOTE: When using the node local /tmp filesystem, you generally need to do this or a similar transfer of any output files, as Slurm will delete any data in the /tmp filesystem after your job terminates.
Lines 55-56: These lines print some additional information just before the job terminates.
These lines will go in the same file as those from the start of the script, usually "slurm-JOBNUMBER.out".
This just prints out:
- The exit code from the hello-umd command (as stashed in the ECODE environment variable)
- The time when the job finished
Line 58: This line actually ends the job script, and terminates the job. The script and job will actually end at the end of the file even without an explicit exit command, but in that case the exit code for the job script will be that of the last command which ran, which typically will be part of the cleanup commands, and not the meat of the script.

Here we explicitly set the exit code to the exit code from the hello-umd command (which we previously saved in the variable ECODE).

Submitting the example

To submit the script, you can just: - Download the script as plain text and copy to one of the cluster login nodes. Move it to an appropriate directory. - From the directory with the job script, run the command sbatch submit.sh (replacing submit.sh with the name of the script you downloaded). The scheduler should return something like Submitted batch job 11274515 --- the number will vary and is the job number for this job). The job number can be used to reference the job in various Slurm commands, etc. (Please always give the job number(s) when requesting help about a job you submitted).

The job will be queued for partition specified (or defaulted to); in this example we specified the the debug partition so it will be in that partition. It should typically start running within 15 minutes or so, and finish shortly after it starts (as it is a rather trivial job).

When it finishes running, you should be able to find the files "slurm-JOBNUMBER.out" and "hello-JOBNUMBER.out" in the directory from which you ran the sbatch command; the former should contain the output of the diagnostic commands (the job number, when it started, what hosts running on, when it ended and exit code, etc), something like:

Example

Loading hello-umd/gcc/11.3.0/nompi/zen2/1.5
  Loading requirement: gcc/11.3.0
Slurm job submit-hello-umd-multithreaded.sh 11381255 running on:
compute-b8-57.zaratan.umd.edu
Ntasks=1; CPU cores per task=10
On 1 nodes: compute-b8-57
Fri Apr 25 09:31:11 EDT 2025
/tmp/job.11381255
Loaded modules are:
Currently Loaded Modulefiles:
 1) hpcc/zaratan(default)   3) hello-umd/gcc/11.3.0/nompi/zen2/1.5
 2) gcc/11.3.0
Job finished with exit code 0
Fri Apr 25 09:31:11 EDT 2025

The "hello-JOBNUMNER.out" file should contain the output of the actual hello-umd command, something like:

Example

hello-umd: Version 1.5
Built for compiler: gcc/11.3.0
Hello UMD from thread 0 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 2 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 8 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 4 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 5 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 7 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 3 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 1 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 9 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 6 of 0, task 0 of 1 (pid=934333 on host compute-b8-57.zaratan.umd.edu

In both cases, some of the specifics (e.g. job numbers, timestamps, pids, and node names) will differ from run to run. Also note that the ordering of the lines from the different threads will vary from run to run.