Skip to content

FF chevron

Job Examples

Here we present and discuss various example jobs of various types.

Job Script Examples

Below are example job scripts for a number of different types of jobs. The examples generally do not do much --- typically they just run a simple Hello World! type of program.

As examples, the scripts are generally designed to display many good scripting practices, including:

  • setting standard sbatch options within the script
  • loading the needed modules within the script
  • printing some useful diagnostic information at start of the script
  • creating a job specific work directory
  • running the code and saving the exit code
  • copying back any files if necessary
  • exiting with the exit code from the main application

Many of the practices above are rather overkill for such simple jobs --- indeed, the vast majority of lines in these example scripts pertain to these "good practices" rather than the actual running of the intended code, but are included for educational purposes.

The example scripts include many comments to explain what it is doing, and the web pages include an even more detailed, line-by-line discussion of the scripts.

Our examples are:

Common parts of job scripts

Shebang line

Job scripts must start with a shebang line which specifies which shell or interpretter should be used to process the script. Although you are free to use any suitable shell available on the system you wish to use, We generally recommend using #!/bin/bash for this, as:

  • Bash is the default login shell for users on the current UMD HPC clusters, so you are likely to be at least as familiar with it as any other shell.
  • Bash has many features which some other shells lack and which are often useful when scripting

Comment lines

Bash and most other shells and interpretters treat any lines starting with a an octothorpe/pound/number sign (i.e. '#') as comments. This includes some special lines like the shebang line which define the program used to interpret the script, and the various SBATCH lines which specify sbatch flags --- the bash shell treats these as comments even though they have some meaning to the system and/or scheduler.

But other than those cases, these comment lines are ignored and only exist to convey meaning to any humans reading the script. Feel free to use and add comment lines to your scripts remind yourself (and maybe others reading your script) of what the script is doing.

Flags to the sbatch command

The sbatch command is a little bit unusual because it recognizes options and/or flags not only when they are included in the command line, but also when they are included in the job script in special lines starting with #SBATCH.

In order to be effective, the various #SBATCH lines must precede any "executable" lines (i.e. anything other than a comment or blank line) in your script --- any #SBATCH lines after an executable line will simply and silently be ignored.

Tip

It is recommended that you place any #SBATCH lines in your script immediately after the shebang line and any initial comment lines. They must occur before any non-comment or non-blank line in the script in order to be effective.

It is often convenient to put flags providing specifics about the job inside the job script rather than on the command line, as:

  • This often results in less typing or typing errors (as jobs often require a fair number of options to be set)
  • HPC users often are submitting many variations of the same job, and many of the job parameters (like number of tasks, cores, amount of memory, number and type of GPUs, etc) do not vary much among these jobs
  • Including the parameters in the job script records/documents the job settings, and improves the reproducibility of jobs.

You can provide all of the flags on the sbatch command line if desired, or even provide some in the job script and others on the command line.
When providing a flag in an #SBATCH directive, just give the flag (same as you would on the command line) after the #SBATCH directive, with some whitespace between the directive and the flag. To provide multiple flags, you can either put multiple flags in a single #SBATCH directive, or give each flag in a separate #SBATCH directive.

E.g. the command

Example

login-1:~$ sbatch -n 1 -c 1 --mem=4096 -t 60 foo.sh

is the functionally the same as

Example

login-1:~$ sbatch foo.sh

if foo.sh contains the lines

Example

#SBATCH -n 1 -c 1
#SBATCH --mem=4096
#SBATCH -t 60

immediately after the shebang line.

If the same flag is given on the command line and in an #SBATCH directive, the command line version has precedence and will take effect.

The following list links to more information about some of the more commonly used sbatch flags:

Setup of the environment

In order to ensure that the job runs as expected, the job script usually has some lines to setup the proper environment. These are usually the first, or some of the first, non-comment lines in the script file.

Your job is not always required to do such setup; it can instead rely on environmental settings it inherits via your ~/.bashrc or similar initialization files, but explicitly setting up the environment for your job within the job script is strongly recommended. Doing such will serve to document the dependencies of the job, and improve reproducibility. From time to time HPC staff get tickets from users who had jobs relying on their ~/.bashrc or similar to set up the environment, only to discover that when they attempt to re-run a job which ran successfully months (or longer) ago it no longer runs properly. This usually turns out because in the intervening months they made small changes to their dot files breaking the environmental dependencies the previously successful job required. This can be difficult to resolve, as the changes to the .bashrc file were made over a period of weeks or months or longer and are probably forgotten, and there was likely no record made of what they were.

Relying on the environment set up in your ~/.bashrc or other dot files is also problematic if you have jobs with different requirements; e.g. jobs A and B both require a specific compiler, etc., but the compiler requirement for job A differs from that of job B. In such a case, a job will not run properly unless the correct version of the .bashrc is in place when the job starts to run.

Tip

We strongly recommend that you completely define the environment for your jobs inside the job script.

They typically will look something like:

Example

source /etc/profile
module purge
module load hpcc/zaratan
module load gcc/11.3.0
module load openmpi
module load hello-umd

The first line, source /etc/profile makes sure that the appropriate initialization files (aka "dot files") are processed. The /etc/profile is the standard initialization file for the bash shell --- if you have opted for a different shell in your shebang line, this will need to be modified. This initialization file processes all of the system-wide initialization scripts for bash, and then invokes your ~/.bash_local script if present (NOTE: it does not source your ~/.bashrc or ~/.bash_profile scripts by default; although the ~/.bashrc dot file will have automatically been invoked when the bash shell for the job script started). The important part is that it will process the code which defines the module command.

The following lines then invoke a set of module commands to define your environment; this is typically what would be done if your job is making use of software installed as part of the Research Software Library (RSWL), as packages in the RSWL typically need module load commands to set up their environment.

In the above example, we start with a module purge command which unloads any and all previously loaded modulefiles --- this ensures that the job script starts with a "clean" slate. We then load the standard module for the cluster (in this case for the Zaratan cluster; change if you are using a different cluster).

We then proceed to load the modulefiles for the software which we will be using. Note that the module command expands incomplete tags (i.e. software packages without version numbers or other details) depending on what modules have already been "loaded", so the order of the module load commands may matter. Generally we advise that you load modules more or less in the following order:

  • First load the compiler or toolchain you plan to use. We recommend including a version number here, as the default compiler/toolchain version for a given family is subject to change when we update the library.
  • If using an MPI library, load the MPI library next. NOTE: some packages in the RSWL have MPI-enabled as well as non-MPI builds available, and unless an MPI library is loaded, the "module load" command will generally load the non-MPI build in such cases.
  • If using CUDA, load the CUDA library next. NOTE: some packages in the RSWL have both CUDA and non-CUDA enabled builds available, and unless the CUDA library is loaded the "module load" command will generally load the non-CUDA build in such cases.
  • Then load any packages your job script will require. Generally it is best to load library-type modulefiles first, then application style modulefiles.

Tip

Once the version of the compiler and/or toolchain is selected, most other packages only have a single version available for that toolchain, and so versions do not need to be explicitly given.

However we recommend that you specify the compiler version, otherwise you might get a different version of the application than expected, especially if you run the code months later.

If your job will be using code you wrote and/or installed on the cluster yourself, you should set up the environment for that as well. If the code to be used is based on a conda-type environment, then the module load commands are probably not necessary (as conda generally will not play well with packages in the RSWL) and you should instead give some commands to activate the conda environment, e.g.

Example

# Define a variable pointing to the root of the conda install
CONDA_ROOT="~/scratch/conda-envs/conda"
# Initialize your conda environment (if as recommended you
# moved the initialization code out of your ~/.bashrc script)
source "${CONDA_ROOT}/bashrc.conda"
# Activate your conda environment
conda active my_conda_env

If using conda-pack, you will probably want something more like

Example

# Define the directory where the conda-pack tarball is
CONDAPACK_ROOT="${HOME}/scratch/conda-envs"
# and the actual conda env tarball to use
CONDAENV="${CONDAPACK_ROOT}/foo.tar.gz"

${CONDAPACK_ROOT}/conda-pack-unpacker.sh -f $CONDAENV
# or if using MPI
# srun ${CONDAPACK_ROOT}/conda-pack-unpacker.sh -f $CONDAENV
if [ $? -ne 0 ]; then
    echo >&2 "[ERROR] Error unpackage $CONDAENV"
    exit 1
fi

Setup working directory

It is good practice to setup a job specific working directory for your job. This is useful in case you submit the job multiple times so that the output of one run does not clobber the output of a previous (or even worse, concurrent) job. Again, although not strictly required, it is a good and recommended practice.

We generally recommend using the environmental variable SLURM_JOB_ID in the directory name; this variable is set up by the Slurm scheduler at the start of the job to contain the value of the Slurm job id; thus it will be unique for every job.

The next question is on which storage tier to place this directory. SHELL storage is not accessible to the compute nodes so that is not an option; and your home directory is not optimized for performance and is quite small, so it is generally not an option either. That leaves the scratch filesystem or the node local /tmp filesystem --- each has its own unique strengths and weaknesses as discussed below.

For most jobs, the scratch filesystem will be a good fit. It is a network accessible shared file system which is optimized to handle heavy I/O. However, it can have bottlenecks if there a lot of metadata operations (e.g. file opens and/or closes); thus it is not good for jobs which involve a lot of small files. In such cases, the node local /tmp filesystem might be better.

The node local /tmp filesysem is, as the name suggests, only accessible from a single compute node; although every compute node has a node local /tmp filesystem, it is a different filesystem for every compute node --- files saved under /tmp on one node will not be accessible on any other node. In particular, you will need to copy any data worth saving when the job completes from /tmp to the scratch filesystem.

Tip

Checkpoint files should generally be saved to scratch. Saving them to /tmp will be ineffective unless you also implement a process of automatically moving them to scratch; otherwise if the job dies for any reason, the files under /tmp (including your checkpoint files) will be deleted, and therefore you will not be able to restart the job from the checkpoint as it was deleted.

It is also possible to use a combination of the two; i.e. use the node local /tmp filesystem to save some temporary output files, but save checkpoint files and valuable files to scratch.

When using scratch, you can include near the start of your job script a code snippet like:

Example

TMPWORKDIR="${HOME}/scratch/job.${SLURM_JOB_ID}"
mkdir $TMPWORKDIR
cd $TMPWORKDIR

This will define the variable TMPWORKDIR with the value of the path to working directory. In this case we set it to a subdirectory of your ~/scratch directory named job. with the Slurm job number appended. The script then creates the directory and changes to that directory.

Using the node local /tmp filesystem is slightly more complicated. The code snippet might look like:

Example

TMPWORKDIR="/tmp/job.${SLURM_JOB_ID}"
mkdir $TMPWORKDIR
cd $TMPWORKDIR

This is basically identical to the snippet from the scratch filesystem case, the only difference is with the path to the working directory; in this case being placed under /tmp instead of ~/scratch.

In either case, if you need to have any input files in the directory before the job any input files in the directory, you should copy them right after this. To avoid issues with the small file bottleneck on scratch, you might wish to store the files

For the node local /tmp filesystem case, just before the end of your job script you will need copy any data that should be retained from /tmp to a location in scratch; any data left in /tmp will be deleted after your job finishes.

Job identification output

Once things are set up, it is common practice to print out some text identifying the job, etc. This is not strictly necessary, but it is useful in many cases, especially if there is a need to diagnose issues. First, this diagnostic output generally confirms that the job actually started, and can verify that the resources you expected to be allocated to the job were actually allocated to it. As they are generally rather simple commands that do not take much time, there is very little overhead associated with them, and their value if and when there are issues to debug makes up for that.

Typically you will see something along the lines of:

Example

# Identify our job
echo "Starting job JOBNAME.sh ${SLURM_JOB_ID} running on:"
hostname
echo "Ntasks=${SLURM_NTASKS}; CPU cores per task=${SLURM_CPUS_PER_TASK}"
echo "On ${SLURM_JOB_NUM_NODES} nodes: ${SLURM_JOB_NODELIST}"
date
pwd
echo "Loaded modules are:"
module list

The exact format of these lines and what information is displayed is highly variable; as previously mentioned none of this is strictly needed, but can often prove helpful when diagnosing issues. The above example prints out the following:

  • The name of the job script (you will need to change JOBNAME.sh for this, but this should in theory be fixed for a given job script)
  • The job number (via the SLURM_JOB_ID environment variable)
  • The hostname of the node the job script is running on (via the hostname command)
  • The number of tasks (via the SLURM_NTASKS variable)
  • The number of cores per task (via the SLURM_CPUS_PER_TASK variable)
  • The number of compute nodes the job was assigned (via the SLURM_JOB_NUM_NODES variable)
  • The names of the compute nodes the job was assigned (via the SLURM_JOB_NODELIST variable)
  • The time the job started (via the date command)
  • The working directory for the job (via the pwd command)
  • The modules loaded (via the module list command)

You can add other data points that you think might be useful (or omit things you feel unhelpful); e.g. a job based on a conda environment would probably want to omit the module list command and provide more information on the conda environment. It can also be useful to include information on the paths to various commands (e.g. including something like the output of the which mpirun command), or even dumping the entire environment (e.g. the printenv command; although this last is quite verbose so was not included in our example).

The output from the example will go into the file collecting the standard output from the job; by default this is named "slurm-JOBNUMBER.out" (where JOBNUMBER is the Slurm job number) in the directory from which you issued the sbatch command.
There are options you can pass to sbatch to change the name of that file.

Suppressing the MPI no-CUDA warning

The standard builds of OpenMPI in the Research Software Library include CUDA support. Despite being built with CUDA support, the MPI libraries will work without issue on non-GPU nodes, although since we do not have CUDA drivers on the non-GPU nodes the OpenMPI system will normally print out an warning message when run on a non-GPU node, something like:

Example

--------------------------------------------------------------------------
The library attempted to open the following supporting CUDA libraries,
but each of them failed.  CUDA-aware support is disabled.
libcuda.so.1: cannot open shared object file: No such file or directory
/usr/lib64/libcuda.so.1: cannot open shared object file: No such file or directory
If you are not interested in CUDA-aware support, then run with
--mca mpi_cuda_support 0 to suppress this message.  If you are interested
in CUDA-aware support, then try setting LD_LIBRARY_PATH to the location
of libcuda.so.1 to get passed this issue.
--------------------------------------------------------------------------

If you get this or a similar warning when running an MPI code and you are not intending to use GPUs, then this warning is harmless and can be ignored. Although harmless, it can be annoying, but it can be suppressed by either:

  • adding the flag --mca mpi_cuda_support 0 to the mpirun command, or
  • setting the environmental variable OMPI_MCA_mpi_cuda_support to 0 before the mpirun command.

If you were intending to use GPUs and you get this message, then something is wrong and you should contact HPC staff.

Exit codes

On Unix and Linux systems, the exit code of a process, program, or script is used to pass information about whether the process completed successfully or encountered errors. In particular, the Slurm scheduler will consider any job which did not have a Slurm error and whose job script has an exit code of 0 as having completed successfully, and will consider a non-zero exit code as having failed.

Therefore it is quite useful to ensure that your job script returns an appropriate exit code so that Slurm will report the status of the job properly.

Normally the exit code of a script (job script or otherwise) is the exit code of the last command that was run. The problem is that often the last command to be executed in the job script is not the command that you are really interested in, but rather some less interesting cleanup commands. To get around that, you can use a code snippet something like:

Example

#Run the main command you are interested in, e.g.
mpirun foo
# Capture the exit code of the mpirun command into variable ECODE
ECODE=$?

# Run assorted cleanup commands
...

# Exit script with the saved exit code
exit $ECODE

In this snippet, the special variable $? holds the exit code of the last command ran. We cannot simply use this at the end of the script as it will be overwritten by the exit codes from any commands run in the "cleanup section". So we instead save the value into the environmental variable ECODE. We can then do whatever we need to do in the "cleanup" section, and then we finish the script with the command exit $ECODE which terminates the script and sets the exit code to the value we saved in ECODE.