Sequential Job Submission Example

Although the HPC cluster is primarily designed for the submission of parallel jobs, there are occasions wherein one wishes to submit sequential jobs (i.e. which use only a single CPU core. This can be for embarassingly parallel jobs or for high-throughput computing tasks.

This page provides an example of submitting such a sequential job.

This job makes use of a simple Hello World! class of program called hello-umd available in the UMD Research Software Library (RSWL) and which supports sequential, multithreaded, and MPI modes of operation. The code simply prints an identifying message from each thread of each task --- for this simple sequential case o we will only be using a single task of one thread.

Overview

This example basically consists of a single file, the job script submit.sh (see for a listing and explanation of the script) which gets submitted to the cluster via the sbatch command.

The script is designed to show many good practices; including: - setting standard sbatch options within the script - loading the needed modules within the script - printing some useful diagnostic information at start of the script - creating a job specific temporary work directory - running the code and saving the exit code - copying back any files that should be retained - exiting with the exit code from the main application

Many of the practices above are rather overkill for such a simple job --- indeed, the vast majority of lines are for these "good practices" rather than the running of the intended code, but are included for educational purposes.

This code runs hello-umd in sequential mode, saving the output to a file in the temporary work directory, and then copying back to the submission directory. We could have forgone all that and simply have the output of hello-umd go to standard output, which would be available in the "slurm-JOBNUMBER.out" file (or whatever file you instructed Slurm to use instead). Doing such is acceptable as long as the code is not producing an excessive amount (many MBs) of output --- if the code produces a lot of output having it all sent to Slurm output file can cause problems, and it is better to redirect to a file.

The submission script

The submission script can be downloaded as plain text, suitable for you to submit to the scheduler with the sbatch command. We present a copy with line numbers for discussion below.

Source of submit.sh

The line numbers in the listing below are to facilitate a line-by-line discussion pf the script; they should not present in the actual script. Downloaded as plain text version of the script.

Example

1  #!/bin/bash
2  # The line above this is the "shebang" line.  It must be first line in script
3  #-----------------------------------------------------
4  #    Example Submission Script: Hello-UMD sequential
5  #       Runs a sequential hello-umd command
6  #-----------------------------------------------------
7  #
8  # Slurm sbatch parameters section: 
9  #    These SBATCH lines should immediately follow the "shebang" line
10 #    They must occur before any non-empty/non-comment lines in the script
11 #
12 #    Request a single task using a single CPU core
13 #SBATCH --ntasks=1
14 #SBATCH --cpus-per-task=1
15 #    Request 5 minute of walltime
16 #SBATCH -t 5
17 #    Request 1 GB of memory per CPU core
18 #SBATCH --mem-per-cpu=1024
19 #    Allow other jobs to run on same node
20 #SBATCH --oversubscribe
21 #    Run on debug partition for rapid turnaround.  You will need
22 #    to change this (remove the line) if walltime > 15 minutes
23 #SBATCH --partition=debug

24 # This job will run our hello-umd demo binary in sequential mode
25 # Output will go to local /tmp scratch space on the node we are running
26 # on, and then will be copied back to our work directory.

27 # Process our dot files (and in particular define the "module" command)
28 source /etc/profile

29 # Module load section
30 # First clear our module list 
31 module purge
32 # and reload the standard modules
33 module load hpcc/zaratan
34 # Set up environment for hello-umd
35 module load hello-umd

36 # Section to make a temporary directory for this job
37 # For sequential jobs, local /tmp filesystem is a good choice
38 # We include the SLURM jobid in the #directory name to avoid interference if 
39 # multiple jobs running at same time.
40 TMPWORKDIR="/tmp/job.${SLURM_JOB_ID}"
41 mkdir $TMPWORKDIR
42 cd $TMPWORKDIR

43 # Section to output information identifying the job, etc.
44 echo "Slurm job submit-hello-umd-sequential.sh ${SLURM_JOB_ID} running on:"
45 hostname
46 echo "Ntasks=${SLURM_NTASKS}; CPU cores per task=${SLURM_CPUS_PER_TASK}"
47 echo "On ${SLURM_JOB_NUM_NODES} nodes: ${SLURM_JOB_NODELIST}"
48 date
49 pwd
50 echo "Loaded modules are:"
51 module list

52 # Run our code
53 hello-umd > hello.out 2>&1
54 # Save the exit code from the previous command
55 ECODE=$?

56 # Copy results back to submit dir
57 cp hello.out ${SLURM_SUBMIT_DIR}/hello-${SLURM_JOB_ID}.out

58 echo "Job finished with exit code $ECODE"
59 date

60 # Exit with the cached exit code
61 exit $ECODE

Discussion of submit.sh

The following is a detailed, more or less line-by-line discussion of the job submission script:

• Line 1: This shebang line specifies that the script is to be processed using the bash shell.
  For more information on the shebang line.
• Lines 2-12: These are comment lines, which are ignored by the bash shell.
  They are only present to make the script more readable to humans. For more information on comment lines.
• Lines 13-23: These lines specify various flags to be passed to the sbatch command. For more information about #SBATCH directives.
• Lines 13-14: As this is a sequential job, we request a single CPU core in a single task.
  We could have omitted these two lines, as both flags default to 1, but it is useful to explicitly state such. For more information about core, task, and node settings
• Line 16: This is a trivial job, so we only request 5 minutes for it. It should finish in well under 5 minutes. For more information about maximum walltime settings
• Line 18: This is a trivial job which does not need much RAM, so we only request 1 GiB for it. Normally jobs needing no more than 4 GiB per CPU core can just omit this and get the default value of 4 GiB per core. For more information about memory settings
• Line 20: This #SBATCH directive says that we are allowing other jobs to share the same node(s) as our job. This is the default (so this directive can usually be omitted) and is recommended, especially for sequential to make most efficient use of the cluster; otherwise this sequential job would prevent any other jobs from using any of the 127 CPU cores on the node even though this job is not using them. For more information about exclusive/shared mode.
• Line 22: This tells the scheduler to put our job in the debug partition. Since this is a short tutorial job, the debug partition is fine and generally will allow the job to spend less time in the queue. For more information about selecting partitions.
• Lines 28-35: These lines set up the environment within which to run your job. These are explained in more detail in the main job examples page, but in this case the lines basically:
  • Ensure the module command is defined
  • Clear out any previously loaded modules (inherited from .bashrc or similar)
  • Load the module for hello-umd
• Lines 36-41: These lines set up a work directory for our job, in this case since it is a sequential job and therefore necessarily a single node job, we use a work directory in the node local /tmp filesystem. For more information about setting up work directories.
• Lines 43-51: These lines print out various pieces of information to identify the job. The output will go into the file "slurm-JOB_NUMBER.out" in the directory from which you issued the sbatch command (where JOB_NUMBER is the Slurm job number). There are options you can pass to sbatch to change the name of the file.
• Line 53: This line actually runs the hello-umd command, saving all output to the file hello.out in the temporary working directory created in lines 36-42. I.e., this one line is the meat of the job script; everything else basically just exists to get to this point or cleanup afterwards.
  
  When writing your own job script to do something useful, this is where you will be putting most of your effort, possibly expanding to multiple lines.
  
  In this simple case, we basically just run the hello-umd command without any arguments. The "> hello.out" part means that the standard output from the command will be saved in the file hello.out in the working directory. The "2>&1" part means that any output to the standard error output stream will be redirected to the standard output. (NOTE: if you wish to save both standard output and standard error to the same file, you need to give these output redirection arguments in that order; redirect standard output first, then redirect standard error to standard output.)
• Line 56: This line saves the exit code from the hello-umd command so thatt be used when the script ends.
  For more information about exit codes.
• Lines 56-59: These lines just do some cleanup and other minor tasks at the end of the job.
• Line 57: This line copies the hello.out file from our working directory in the node local /tmp filesystem back to the directory from which we submitted the job (i.e. where you ran the sbatch command). We add the Slurm job number to the file name during the copy; this prevents the clobbering of the output file if we submit the job multiple times.
  
  This step is needed because we were using the node local /tmp filesystem, which will be deleted after the job terminates. If you modified the script to use the scratch filesystem instead, you can skip this set, as the scratch filesystem is more permanent.
  
  NOTE: When using the node local /tmp filesystem, you generally need to do this or a similar transfer of any output files, as Slurm will delete any data in the /tmp filesystem after your job terminates.
• Lines 58-59: These lines print some additional information just before the job terminates.
  These lines will go in the same file as those from the start of the script, usually "slurm-JOBNUMBER.out".
  This just prints out:
  • The exit code from the hello-umd command (as stashed in the ECODE environment variable)
  • The time when the job finished
• Line 60: This line actually ends the job script, and terminates the job. The script and job will actually end at the end of the file even without an explicit exit command, but in that case the exit code for the job script will be that of the last command which ran, which typically will be part of the cleanup commands, and not the meat of the script.
  
  Here we explicitly set the exit code to the exit code from the hello-umd command (which we previously saved in the variable ECODE).

Submitting the example

To submit the script, you can just: - Download the script as plain text and copy to one of the cluster login nodes. Move it to an appropriate directory. - From the directory with the job script, run the command sbatch submit.sh (replacing submit.sh with the name of the script you downloaded). The scheduler should return something like Submitted batch job 11274515 --- the number will vary and is the job number for this job). The job number can be used to reference the job in various Slurm commands, etc. (Please always give the job number(s) when requesting help about a job you submitted).

The job will be queued for partition specified (or defaulted to); in this example we specified the the debug partition so it will be in that partition. It should typically start running within 15 minutes or so, and finish shortly after it starts (as it is a rather trivial job).

When it finishes running, you should be able to find the files "slurm-JOBNUMBER.out" and "hello-JOBNUMBER.out" in the directory from which you ran the sbatch command; the former should contain the output of the diagnostic commands (the job number, when it started, what hosts running on, when it ended and exit code, etc), something like:

Example

Loading hello-umd/gcc/11.3.0/nompi/zen2/1.5
  Loading requirement: gcc/11.3.0
Slurm job submit-hello-umd-sequential.sh 11381246 running on:
compute-b8-60.zaratan.umd.edu
Ntasks=1; CPU cores per task=1
On 1 nodes: compute-b8-60
Fri Apr 25 09:13:01 EDT 2025
/tmp/job.11381246
Loaded modules are:
Currently Loaded Modulefiles:
 1) hpcc/zaratan(default)   3) hello-umd/gcc/11.3.0/nompi/zen2/1.5
 2) gcc/11.3.0
Job finished with exit code 0
Fri Apr 25 09:13:01 EDT 2025

The "hello-JOBNUMNER.out" file should contain the output of the actual hello-umd command, something like:

Example

hello-umd: Version 1.5
Built for compiler: gcc/11.3.0
Hello UMD from thread 0 of 1, task 0 of 1 (pid=3104794 on host compute-b8-60.zaratan.umd.edu

In both cases, some of the specifics (e.g. job numbers, timestamps, pids, and node names) will differ from run to run.