Skip to content

FF chevron

MPI Job Submission Example

This page shows an example of submitting a job that uses a MPI for parallelization. This example is for a "pure" MPI job, in which each MPI task is single threaded --- for an example of a job in which the MPI tasks are multithreaded, see the hybrid job example.

Many of the codes used on HPC clusters support parallelization via MPI, and so this example can generally be adapted to support such.

Overview

The example shown is a rather trivial Hello World type program in which the single thread of each tasks prints out a "Hello" message identifying the thread (always thread 0 of 1 for this case) and task number (along with hostname and process ID). Although trivial, it shows the basics of submitting such a job, along with various other good practices, including:

  • setting standard sbatch options within the script
  • loading the needed modules within the script
  • printing some useful diagnostic information at start of the script
  • creating a job specific work directory
  • running the code and saving the exit code
  • exiting with the exit code from the main application

Many of the practices above are rather overkill for such a simple job --- indeed, the vast majority of lines are for these "good practices" rather than the running of the intended code, but are included for educational purposes.

The submission script

The submission script can be downloaded as plain text, suitable for you to submit to the scheduler with the sbatch command. We present a copy with line numbers for discussion below.

Source of submit.sh

The line numbers in the listing below are to facilitate a line-by-line discussion pf the script; they should not present in the actual script. Downloaded as plain text version of the script.

Example

1  #!/bin/bash
2  # The line above this is the "shebang" line.  It must be first line in script
3  #-----------------------------------------------------
4  #    Example Submission Script for Hello-UMD, MPI version
5  #    Runs a simple MPI enabled hello-world code
6  #-----------------------------------------------------
7  #
8  # Slurm sbatch parameters section:
9  #    Request 60 MPI tasks with 1 CPU core each
10 #SBATCH -n 60
11 #SBATCH -c 1
12 #    Request 5 minutes of walltime
13 #SBATCH -t 5
14 #    Request 1 GB of memory per CPU core
15 #SBATCH --mem-per-cpu=1024
16 #    Allow other jobs to run on same node
17 #SBATCH --oversubscribe
18 #    Run on debug partition for rapid turnaround.  You will need
19 #    to change this (remove the line) if walltime > 15 minutes
20 #SBATCH --partition=debug

21 # This job will run the MPI enabled version of hello-umd
22 # We create a directory on parallel filesystem from where we actually 
23 # will run the job.

24 # Process our dot files (and in particular define the "module" command)
25 source /etc/profile

26 # Module load section
27 # First clear our module list 
28 module purge
29 # and reload the standard modules
30 module load hpcc/zaratan
31 # Load the desired compiler,  MPI, and package modules
32 # NOTE: You need to use the same compiler and MPI module used
33 # when compiling the MPI-enabled code you wish to run (in this
34 # case hello-umd).  
35 # NOTE: You need to module load the MPI library (openmpi) before hello-umd;
36 # otherwise you will get a non-MPI version of hello-umd
37 module load gcc
38 module load openmpi
39 module load hello-umd

40 # Section to make a scratch directory for this job
41 # Because different MPI tasks, which might be on different nodes, and will need
42 # access to it, we put it in a parallel file system.  
43 # We include the SLURM jobid in the directory name to avoid interference 
44 # if multiple jobs running at same time.
45 TMPWORKDIR="${HOME}/scratch/job.${SLURM_JOB_ID}"
46 mkdir $TMPWORKDIR
47 cd $TMPWORKDIR

48 # Section to output information identifying the job, etc.
49 echo "Slurm job submit-hello-umd-mpi.sh ${SLURM_JOB_ID} running on:"
50 hostname
51 echo "Ntasks=${SLURM_NTASKS}; CPU cores per task=${SLURM_CPUS_PER_TASK}"
52 echo "On ${SLURM_JOB_NUM_NODES} nodes: ${SLURM_JOB_NODELIST}"
53 date
54 pwd
55 echo "Loaded modules are:"
56 module list
57 echo "Job will run in $TMPWORKDIR"

58 # Setting this variable will suppress the warnings
59 # about lack of CUDA support on non-GPU enabled nodes.  We
60 # are not using CUDA, so warning is harmless.
61 export OMPI_MCA_mpi_cuda_support=0

62 # Get the full path to our hello-umd executable.  It is best
63 # to provide the full path of our executable to mpirun, etc.
64 MYEXE=`which hello-umd`
65 echo "Using executable $MYEXE"

66 # Run our code using mpirun to launch the different tasks
67 # We do not specify the number of tasks here, and instead rely on
68 # it defaulting to the number of tasks requested of Slurm
69 mpirun  ${MYEXE}  > hello.out 2>&1
70 # Save the exit code from the previous command
71 ECODE=$?

72 echo "Job finished with exit code $ECODE.  Work dir is $TMPWORKDIR"
73 date

74 # Exit with the cached exit code
75 exit $ECODE

Discussion of submit.sh

The following is a detailed, more or less line-by-line discussion of the job submission script:

  • Line 1: This shebang line specifies that the script is to be processed using the bash shell.
    For more information on the shebang line.
  • Lines 2-9: These are comment lines, which are ignored by the bash shell.
    They are only present to make the script more readable to humans. For more information on comment lines.
  • Lines 8-20: These lines specify various flags to be passed to the sbatch command. For more information about #SBATCH directives.
  • Lines 10-11: As this is a pure MPI job, we request a 60 MPI takss with each task having a single thread, for a total of 60 CPU cores. The scheduler is free to allocate these cores on any number (from 1 to 60) nodes as it desires. For more information about core, task, and node settings
  • Line 13: This is a trivial job, so we only request 5 minutes for it. It should finish in well under 5 minutes. For more information about maximum walltime settings
  • Line 15: This is a trivial job which does not need much RAM, so we only request 1 GiB per CPU core (so 60 GiB in total). Normally jobs needing no more than 4 GiB per CPU core can just omit this and get the default value of 4 GiB per core. For more information about memory settings
  • Line 17: This #SBATCH directive says that we are allowing other jobs to share the same node(s) as our job. This is the default (so this directive can usually be omitted) and is recommended, especially for jobs using only a fraction of the cores available on a node (128 cores per node on the Zaratan cluster), otherwise a large number of the cores on the node will not be used by your job nor will the scheduler allow other jobs to use them. For more information about exclusive/shared mode.
  • Line 20: This tells the scheduler to put our job in the debug partition. Since this is a short tutorial job, the debug partition is fine and generally will allow the job to spend less time in the queue. For more information about selecting partitions.
  • Lines 25-39: These lines set up the environment within which to run your job. These are explained in more detail in the main job examples page, but in this case the lines basically:
    • Ensure the module command is defined
    • Clear out any previously loaded modules (inherited from .bashrc or similar)
    • Load the modules for compiler, MPI library, and hello-umd. We must module load the MPI library before hello-umd to ensure that we get the MPI build of hello-umd.
  • Lines 40-47: These lines set up a work directory for our job, in this case since it is an MPI based job and will potentially run on multiple nodes, a shared filesystem is appropriate, so we use a work directory in the scratch. For more information about setting up work directories.
  • Lines 48-57: These lines print out various pieces of information to identify the job. The output will go into the file "slurm-JOB_NUMBER.out" in the directory from which you issued the sbatch command (where JOB_NUMBER is the Slurm job number). There are options you can pass to sbatch to change the name of the file.

  • Line 61: This line suppresses some uninmportant warnings.

  • Line 64: This line uses the Unix which command to determine the full path to the hello-umd command. Generally, when using MPI, it is best to provide the full path to the command to run to the mpirun command.

  • Line 69: This line actually runs the hello-umd command, saving all output to the file hello.out in the temporary working directory created in lines 40-47. I.e., this one line is the meat of the job script; everything else basically just exists to get to this point or cleanup afterwards.

    When writing your own job script to do something useful, this is where you will be putting most of your effort, possibly expanding to multiple lines.

    In this simple case, we basically just run the hello-umd command with the -t $SLURM_CPUS_PER_TASK argument. The environmental variable SLURM_CPUS_PER_TASK contains the value of the --cpus-per-task argument from line 14 to ensure the correct number of threads are run in each MPI task. The "> hello.out" part means that the standard output from the command will be saved in the file hello.out in the working directory. The "2>&1" part means that any output to the standard error output stream will be redirected to the standard output. (NOTE: if you wish to save both standard output and standard error to the same file, you need to give these output redirection arguments in that order; redirect standard output first, then redirect standard error to standard output.)
  • Line 71: This line saves the exit code from the hello-umd command so thatt be used when the script ends.
    For more information about exit codes.
  • Lines 72-73: These lines print some additional information just before the job terminates.
    These lines will go in the same file as those from the start of the script, usually "slurm-JOBNUMBER.out".
    This just prints out:
    • The exit code from the hello-umd command (as stashed in the ECODE environment variable)
    • The time when the job finished
  • Line 75: This line actually ends the job script, and terminates the job. The script and job will actually end at the end of the file even without an explicit exit command, but in that case the exit code for the job script will be that of the last command which ran, which typically will be part of the cleanup commands, and not the meat of the script.

    Here we explicitly set the exit code to the exit code from the hello-umd command (which we previously saved in the variable ECODE).

Submitting the example

To submit the script, you can just: - Download the script as plain text and copy to one of the cluster login nodes. Move it to an appropriate directory. - From the directory with the job script, run the command sbatch submit.sh (replacing submit.sh with the name of the script you downloaded). The scheduler should return something like Submitted batch job 11274515 --- the number will vary and is the job number for this job). The job number can be used to reference the job in various Slurm commands, etc. (Please always give the job number(s) when requesting help about a job you submitted).

The job will be queued for partition specified (or defaulted to); in this example we specified the the debug partition so it will be in that partition. It should typically start running within 15 minutes or so, and finish shortly after it starts (as it is a rather trivial job).

When it finishes running, you should be able to find the file "slurm-JOBNUMBER.out" in the directory from which you ran the sbatch command which should contain the output of the diagnostic commands (the job number, when it started, what hosts running on, when it ended and exit code, etc), something like:

Example

Slurm job submit-hello-umd-mpi.sh 11381245 running on:
compute-b8-57.zaratan.umd.edu
Ntasks=60; CPU cores per task=1
On 4 nodes: compute-b8-[57-60]
Fri Apr 25 09:13:01 EDT 2025
/home/payerle/scratch/job.11381245
Loaded modules are:
Currently Loaded Modulefiles:
 1) hpcc/zaratan(default)   3) openmpi/gcc/11.3.0/zen2/4.1.5
 2) gcc/11.3.0              4) hello-umd/gcc/11.3.0/openmpi/4.1.5/zen2/1.5
Job will run in /home/payerle/scratch/job.11381245
Using executable /cvmfs/hpcsw.umd.edu/spack-software/2023.11.20/linux-rhel8-zen2/gcc-11.3.0/hello-umd-1.5-eqwagrojqwhd6pvhlt5jawh2szvtcyg4/bin/hello-umd
Job finished with exit code 0.  Work dir is /home/payerle/scratch/job.11381245
Fri Apr 25 09:13:02 EDT 2025

Similar, in the working directory of the job (normally "~/scratch/job.JOBNUMBER") the file "hello.out" should contain the output of the actual "hello-umd" command, something like:

Example

Hello UMD from thread 0 of 1, task 0 of 60 (pid=931078 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 1 of 60 (pid=931079 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 2 of 60 (pid=931080 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 3 of 60 (pid=931081 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 4 of 60 (pid=931082 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 5 of 60 (pid=931083 on host compute-b8-57.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 6 of 60 (pid=931084 on host compute-b8-57.zaratan.umd.edu
hello-umd: Version 1.5
Built for compiler: gcc/11.3.0
with MPI support( usgin MPI library openmpi/4.1.5)
Hello UMD from thread 0 of 1, task 54 of 60 (pid=3994191 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 49 of 60 (pid=3994186 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 58 of 60 (pid=3104797 on host compute-b8-60.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 59 of 60 (pid=3104799 on host compute-b8-60.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 57 of 60 (pid=3104796 on host compute-b8-60.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 47 of 60 (pid=3994184 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 51 of 60 (pid=3994188 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 52 of 60 (pid=3994189 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 55 of 60 (pid=3994195 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 12 of 60 (pid=827134 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 56 of 60 (pid=3994197 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 43 of 60 (pid=3994180 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 44 of 60 (pid=3994181 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 45 of 60 (pid=3994182 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 46 of 60 (pid=3994183 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 48 of 60 (pid=3994185 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 50 of 60 (pid=3994187 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 53 of 60 (pid=3994190 on host compute-b8-59.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 14 of 60 (pid=827211 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 18 of 60 (pid=827214 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 30 of 60 (pid=827220 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 32 of 60 (pid=827140 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 33 of 60 (pid=827195 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 34 of 60 (pid=827221 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 35 of 60 (pid=827161 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 38 of 60 (pid=827223 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 8 of 60 (pid=827133 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 9 of 60 (pid=827174 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 10 of 60 (pid=827132 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 11 of 60 (pid=827141 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 17 of 60 (pid=827182 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 20 of 60 (pid=827136 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 22 of 60 (pid=827216 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 24 of 60 (pid=827137 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 25 of 60 (pid=827188 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 26 of 60 (pid=827218 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 27 of 60 (pid=827155 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 28 of 60 (pid=827138 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 29 of 60 (pid=827192 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 42 of 60 (pid=827227 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 19 of 60 (pid=827151 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 23 of 60 (pid=827153 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 37 of 60 (pid=827199 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 40 of 60 (pid=827147 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 41 of 60 (pid=827201 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 13 of 60 (pid=827178 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 15 of 60 (pid=827146 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 16 of 60 (pid=827135 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 21 of 60 (pid=827185 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 36 of 60 (pid=827144 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 31 of 60 (pid=827157 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 39 of 60 (pid=827163 on host compute-b8-58.zaratan.umd.edu
Hello UMD from thread 0 of 1, task 7 of 60 (pid=827139 on host compute-b8-58.zaratan.umd.edu

In both cases, some of the specifics (e.g. job numbers, timestamps, pids, and node names) will differ from run to run. Also note that the ordering of the lines from the various tasks will vary.