Skip to content

FF chevron

Hybrid Multithreaded + MPI Job Submission Example

This page shows an example of submitting a job that uses a hybrid parallel job which uses both distributed memory parallelization (via MPI) and shared memory parallelization (via multithreading). Basically, the job spawns N MPI tasks on one or more nodes, with each task running M threads.

Overview

The example shown is a rather trivial Hello World type program in which each thread of each tasks prints out a "Hello" message identifying the thread and task number (along with hostname and process ID). Although trivial, it shows the basics of submitting such a job, along with various other good practices, including:

  • setting standard sbatch options within the script
  • loading the needed modules within the script
  • printing some useful diagnostic information at start of the script
  • creating a job specific work directory
  • running the code and saving the exit code
  • exiting with the exit code from the main application

Many of the practices above are rather overkill for such a simple job --- indeed, the vast majority of lines are for these "good practices" rather than the running of the intended code, but are included for educational purposes.

The submission script

The submission script can be downloaded as plain text, suitable for you to submit to the scheduler with the sbatch command. We present a copy with line numbers for discussion below.

Source of submit.sh

The line numbers in the listing below are to facilitate a line-by-line discussion pf the script; they should not present in the actual script. Downloaded as plain text version of the script.

Example

1  #!/bin/bash
2  # The line above this is the "shebang" line.  It must be first line in script
3  #-----------------------------------------------------
4  #    Example Submission Script: Hello-UMD, hybrid (MPI + multithreaded)
5  #    Runs a hello-umd command with mpi and multiple threads per task
6  #-----------------------------------------------------
7  #
8  # Slurm sbatch parameters section:
9  #    These SBATCH lines should immediately follow the "shebang" line
10 #    They must occur before any non-empty/non-comment lines in the script
11 #
12 #    Request 3 MPI tasks with 15 CPU cores each
13 #SBATCH -n 3
14 #SBATCH -c 15
15 #    Request 5 minutes of walltime
16 #SBATCH -t 5
17 #    Request 1 GB of memory per CPU core
18 #SBATCH --mem-per-cpu=1024
19 #    Allow other jobs to run on same node
20 #SBATCH --oversubscribe
21 #    Run on debug partition for rapid turnaround.  You will need
22 #    to change this (remove the line) if walltime > 15 minutes
23 #SBATCH --partition=debug

24 # This job will run the hello-umd with multiple MPI tasks each using 
25 # multiple threads.  We create a directory on parallel filesystem from 
26 # where we actually will run the job.

27 # Process our dot files (and in particular define the "module" command)
28 source /etc/profile

29 # Module load section
30 # First clear our module list 
31 module purge
32 # and reload the standard modules
33 module load hpcc/zaratan
34 # Load the desired compiler,  MPI, and package modules
35 # NOTE: You need to use the same compiler and MPI module used
36 # when compiling the MPI-enabled code you wish to run (in this
37 # case hello-umd). 
38 # NOTE: You need to module load the MPI library (openmpi) before hello-umd;
39 # otherwise you will get a non-MPI version of hello-umd
40 module load gcc
41 module load openmpi
42 module load hello-umd

43 # Section to make a scratch directory for this job
44 # Because different MPI tasks, which might be on different nodes, and will need
45 # access to it, we put it in a parallel file system.  
46 # We include the SLURM jobid in the directory name to avoid interference 
47 # if multiple jobs running at same time.
48 TMPWORKDIR="${HOME}/scratch/job.${SLURM_JOB_ID}"
49 mkdir $TMPWORKDIR
50 cd $TMPWORKDIR

51 # Section to output information identifying the job, etc.
52 echo "Slurm job submit-hello-umd-hybrid.sh ${SLURM_JOB_ID} running on:"
53 hostname
54 echo "Ntasks=${SLURM_NTASKS}; CPU cores per task=${SLURM_CPUS_PER_TASK}"
55 echo "On ${SLURM_JOB_NUM_NODES} nodes: ${SLURM_JOB_NODELIST}"
56 date
57 pwd
58 echo "Loaded modules are:"
59 module list
60 echo "Job will be run in $TMPWORKDIR"

61 # Setting this variable will suppress the warnings
62 # about lack of CUDA support on non-GPU enabled nodes.  We
63 # are not using CUDA, so warning is harmless.
64 export OMPI_MCA_mpi_cuda_support=0

65 # Get the full path to our hello-umd executable.  It is best
66 # to provide the full path of our executable to mpirun, etc.
67 MYEXE=`which hello-umd`
68 echo "Using executable $MYEXE"

69 # Run our code using mpirun to launch the different tasks
70 # We do not specify the number of tasks here, and instead rely on
71 # it defaulting to the number of tasks requested of Slurm
72 # We add -t $SLURM_CPUS_PER_TASK to have hello-umd run threads on all cores 
73 # allocated to the job
74 mpirun  ${MYEXE} -t $SLURM_CPUS_PER_TASK  > hello.out 2>&1
75 # Save the exit code from the previous command
76 ECODE=$?

77 echo "Job finished with exit code $ECODE.  Work dir is $TMPWORKDIR"
78 date

79 # Exit with the cached exit code
80 exit $ECODE

Discussion of submit.sh

The following is a detailed, more or less line-by-line discussion of the job submission script:

  • Line 1: This shebang line specifies that the script is to be processed using the bash shell.
    For more information on the shebang line.
  • Lines 2-10: These are comment lines, which are ignored by the bash shell.
    They are only present to make the script more readable to humans. For more information on comment lines.
  • Lines 11-23: These lines specify various flags to be passed to the sbatch command. For more information about #SBATCH directives.
  • Lines 13-14: As this is a hybrid job, we request a three MPI takss with each task having 15 threads, for a total of 45 CPU cores. All 15 cores for the same task will be allocated on the same node; the cores for different tasks might or might not be allocated to the same node. For more information about core, task, and node settings
  • Line 16: This is a trivial job, so we only request 5 minutes for it. It should finish in well under 5 minutes. For more information about maximum walltime settings
  • Line 18: This is a trivial job which does not need much RAM, so we only request 1 GiB per CPU core (so 45 GiB in total). Normally jobs needing no more than 4 GiB per CPU core can just omit this and get the default value of 4 GiB per core. For more information about memory settings
  • Line 20: This #SBATCH directive says that we are allowing other jobs to share the same node(s) as our job. This is the default (so this directive can usually be omitted) and is recommended, especially for jobs using only a fraction of the cores available on a node (128 cores per node on the Zaratan cluster), otherwise a large number of the cores on the node will not be used by your job nor will the scheduler allow other jobs to use them. For more information about exclusive/shared mode.
  • Line 23: This tells the scheduler to put our job in the debug partition. Since this is a short tutorial job, the debug partition is fine and generally will allow the job to spend less time in the queue. For more information about selecting partitions.
  • Lines 28-42: These lines set up the environment within which to run your job. These are explained in more detail in the main job examples page, but in this case the lines basically:
    • Ensure the module command is defined
    • Clear out any previously loaded modules (inherited from .bashrc or similar)
    • Load the modules for compiler, MPI library, and hello-umd. We must module load the MPI library before hello-umd to ensure that we get the MPI build of hello-umd.
  • Lines 43-50: These lines set up a work directory for our job, in this case since it is an MPI based job and will potentially run on multiple nodes, a shared filesystem is appropriate, so we use a work directory in the scratch. For more information about setting up work directories.
  • Lines 51-60: These lines print out various pieces of information to identify the job. The output will go into the file "slurm-JOB_NUMBER.out" in the directory from which you issued the sbatch command (where JOB_NUMBER is the Slurm job number). There are options you can pass to sbatch to change the name of the file.

  • Line 64: This line suppresses some uninmportant warnings.

  • Line 67: This line uses the Unix which command to determine the full path to the hello-umd command. Generally, when using MPI, it is best to provide the full path to the command to run to the mpirun command.

  • Line 74: This line actually runs the hello-umd command, saving all output to the file hello.out in the temporary working directory created in lines 43-50. I.e., this one line is the meat of the job script; everything else basically just exists to get to this point or cleanup afterwards.

    When writing your own job script to do something useful, this is where you will be putting most of your effort, possibly expanding to multiple lines.

    In this simple case, we basically just run the hello-umd command with the -t $SLURM_CPUS_PER_TASK argument. The environmental variable SLURM_CPUS_PER_TASK contains the value of the --cpus-per-task argument from line 14 to ensure the correct number of threads are run in each MPI task. The "> hello.out" part means that the standard output from the command will be saved in the file hello.out in the working directory. The "2>&1" part means that any output to the standard error output stream will be redirected to the standard output. (NOTE: if you wish to save both standard output and standard error to the same file, you need to give these output redirection arguments in that order; redirect standard output first, then redirect standard error to standard output.)
  • Line 76: This line saves the exit code from the hello-umd command so thatt be used when the script ends.
    For more information about exit codes.
  • Lines 77-78: These lines print some additional information just before the job terminates.
    These lines will go in the same file as those from the start of the script, usually "slurm-JOBNUMBER.out".
    This just prints out:
    • The exit code from the hello-umd command (as stashed in the ECODE environment variable)
    • The time when the job finished
  • Line 80: This line actually ends the job script, and terminates the job. The script and job will actually end at the end of the file even without an explicit exit command, but in that case the exit code for the job script will be that of the last command which ran, which typically will be part of the cleanup commands, and not the meat of the script.

    Here we explicitly set the exit code to the exit code from the hello-umd command (which we previously saved in the variable ECODE).

Submitting the example

To submit the script, you can just: - Download the script as plain text and copy to one of the cluster login nodes. Move it to an appropriate directory. - From the directory with the job script, run the command sbatch submit.sh (replacing submit.sh with the name of the script you downloaded). The scheduler should return something like Submitted batch job 11274515 --- the number will vary and is the job number for this job). The job number can be used to reference the job in various Slurm commands, etc. (Please always give the job number(s) when requesting help about a job you submitted).

The job will be queued for partition specified (or defaulted to); in this example we specified the the debug partition so it will be in that partition. It should typically start running within 15 minutes or so, and finish shortly after it starts (as it is a rather trivial job).

When it finishes running, you should be able to find the file "slurm-JOBNUMBER.out" in the directory from which you ran the sbatch command which should contain the output of the diagnostic commands (the job number, when it started, what hosts running on, when it ended and exit code, etc), something like:

Example

Slurm job submit-hello-umd-hybrid.sh 11381243 running on:
compute-b8-57.zaratan.umd.edu
Ntasks=3; CPU cores per task=15
On 2 nodes: compute-b8-[57-58]
Fri Apr 25 09:12:30 EDT 2025
/home/payerle/scratch/job.11381243
Loaded modules are:
Currently Loaded Modulefiles:
 1) hpcc/zaratan(default)   3) openmpi/gcc/11.3.0/zen2/4.1.5
 2) gcc/11.3.0              4) hello-umd/gcc/11.3.0/openmpi/4.1.5/zen2/1.5
Job will be run in /home/payerle/scratch/job.11381243
Using executable /cvmfs/hpcsw.umd.edu/spack-software/2023.11.20/linux-rhel8-zen2/gcc-11.3.0/hello-umd-1.5-eqwagrojqwhd6pvhlt5jawh2szvtcyg4/bin/hello-umd
Job finished with exit code 0.  Work dir is /home/payerle/scratch/job.11381243
Fri Apr 25 09:12:32 EDT 2025

Similar, in the working directory of the job (normally "~/scratch/job.JOBNUMBER") the file "hello.out" should contain the output of the actual "hello-umd" command, something like:

Example

hello-umd: Version 1.5
Built for compiler: gcc/11.3.0
with MPI support( usgin MPI library openmpi/4.1.5)
Hello UMD from thread 1 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 0 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 1 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 0 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 2 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 3 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 4 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 2 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 3 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 5 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 4 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 5 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 6 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 7 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 6 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 8 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 7 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 8 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 9 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 10 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 11 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 12 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 9 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 10 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 11 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 13 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 14 of 15, task 1 of 3 (pid=930442 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 12 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 13 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 14 of 15, task 0 of 3 (pid=930441 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 14 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 1 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 2 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 3 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 4 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 5 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 6 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 7 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 8 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 9 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 10 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 11 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 12 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 13 of 15, task 2 of 3 (pid=826907 on host compute-b8-58.zaratan.umd.edu

In both cases, some of the specifics (e.g. job numbers, timestamps, pids, and node names) will differ from run to run. Also note that the ordering of the lines from various tasks and threads will vary from run to run.