Skip to content

FF chevron

Submitting Jobs

This page describes how to use the sbatch command to submit jobs on the University of Maryland HPC clusters.

Basic Job Submission

The Zaratan HPC cluster uses a batch scheduling system called Slurm to handle the queuing, scheduling, and execution of jobs. This scheduler is used in many recent HPC clusters throughout the world. This page will attempt to discuss the Slurm commands for submitting jobs, and how to specify the job requirements.

Users generally submit jobs by writing a job script file and submitting the job to Slurm with the sbatch command.
The sbatch command takes a number of options (some of which can be omitted or defaulted). These options define various requirements of the job, which are used by the scheduler to figure out what is needed to run your job, and to schedule it to run as soon as possible, subject to the constraints on the system, usage policies, and considering the other users of the cluster. It is also possible to submit an interactive job, but that is usually most useful for debugging purposes.

The options to sbatch can be given on the command line, but in most cases it usually more convenient to provide them within the job script, as this is generally easier to do, keeps the submission parameters with the script, and increases reproducibility. When given inside the job script, the option is placed alone on a line starting with #SBATCH (you must include a space after the SBATCH). These #SBATCH lines SHOULD come before any non-comment/non-blank line in the script --- any #SBATCH lines AFTER a non-comment/non-blank line in the script might get ignored by the scheduler. See the examples page for examples. The # at the start of these lines means they are comments to the shell and so will be ignored by the shell; i.e. only the sbatch command will read them.

Your Job Script

The most basic parameter given to the sbatch command is the script to run. This obviously must be given on command line, not inside the script file. This job script must start with a shebang line which specifies the shell under which the script is to run. I.e., the very first line of your script should generally be either #!/bin/tcsh or #!/bin/bash

for the tcsh or bash shell, respectively. This must be the first line, and no spaces before the name of the shell. This line is typically followed by a bunch of #SBATCH lines specifying the job requirements (these will be discussed below), and then the actual commands that you wish to have executed when the job is started on the compute nodes.

There are many options you can give to sbatch either on the command line or using #SBATCH lines within your script file. Other parts of this page discuss the more common ones. It is strongly recommended that you include at least the following directives to specify:

Note

NOTE: The #SBATCH lines must come BEFORE any non-blank/non-comment lines in your script file. Any #SBATCH lines which come after non-blank/non-comment lines might get ignored by the scheduler.

Generally, we recommend that the first non-comment line in your script be an invocation to your initial dot files. For bash shells it would be source /etc/profile For csh or tcsh shells it would source /etc/csh.login

This will properly set up your environment on the cluster, including defining the module command. It is also recommended that after that line, you include module load commands to set up your environment for any packages in the Research Software library you wish to use.

The remainder of the file should be the commands to run to do the calculations you want. After you submit the job to the scheduler, the job will wait for a while in the queue until resources are available. Once resources are available, the scheduler will run this script on the first node assigned to your job. If your job involves multiple nodes (or even multiple cores on the same node), it is this script's responsibility to launch all the tasks or processes for the job. When the script exits, the job is considered to have finished. More information can be found in the section on Running parallel codes and in the examples section.

Warning

Do NOT run jobs from your home directory; the home directories are not optimized for intensive I/O, and are quite limited in size. Your allocation will have scratch storage allotted to it at /scratch/zt1/projects/PROJECT (where PROJECT is the name of the project to which the allocation belongs); you should use a directory underneath that instead.

Warning

Note: If your script does not end with a proper Unix end-of-line (EOF) character, the last line of your script will usually be ignored by the shell when it is run. This can often happen if you transfer files from Windows (which uses different character to designate the end-of-line) to Unix, and can sometimes be quite confusing, as you submit you job, it runs, and finishes almost immediately without errors, and there is seemingly no output because the last line of your job script, which got ignored, is the command to actually do the work for the calculation. Although you can use a command like dos2unix to fix the script file, it is usually easiest to just remember to add a couple blank lines to the end of your file.

Tip

We recommend leaving some blank lines at the end of your script. This will effectively solve the issue with missing EOL characters. While it does not technically fix the issue, the last line is now blank, and so if it is ignored due to a missing/improper EOL character, it does not effect the functionality of the script since it is blank.

Sometimes your job script might need specific information about job parameters specified to sbatch. To facilitate this, the Slurm scheduler will make a number of these values available in environmental variables. These are detailed in the man page for sbatch, and also in this page listing the more commonly used variables. These can be useful in many cases, e.g. if you are running an OpenMP program on a single node which needs to be passed an argument with the number of threads to run, you probably want to give it $SLURM_CPUS_PER_TASK for this value to ensure that the number you pass it always equals the number of cores requested from Slurm. This can help avoid issues if you change the number of cores requested in a later run, as you only change things in one place now.

The following sections discuss a number of commonly used arguments to the sbatch command; the sbatch command has a very large number of options/flags/arguments; we only cover a rather small number of the more commonly used options here; for a more complete explanatino of the command usage and all flags, please see the sbatch manual page (or run the command man sbatch on the cluster).

Sbatch options for requesting resources

Specifying Walltime

When submitting a job, it is very important to specify the amount of time you expect your job to take. If you specify a time that is too short, your job will be terminated by the scheduler before it completes. So you should always add a buffer to account for variability in run times; you do not want your job to be killed when it is 99.9% complete. However, if you specify a time that is too long, you may run the risk of having your job sit in the queue for longer than it should, as the scheduler attempts to find available resources on which to run your job. See the section on scheduling for more information on the scheduling process and advice regarding the setting of walltime limits.

In general, on the Zaratan cluster, all users can run submit jobs to the standard partition which currently allows for jobs of up to one week.

To specify your estimated runtime, use the --time=TIME or -t TIME parameter to sbatch. This value TIME can in any of the following formats:

  • M (M minutes)
  • M:S (M minutes, S seconds)
  • H:M:S (H hours, M minutes, S seconds)
  • D-H (D days, H hours)
  • D-H:M (D days, H hours, M minutes)
  • D-H:M:S (D days, H hours, M minutes, S seconds)

Warning

NOTE: If you do not specify a walltime, the default walltime on the Zaratan HPC cluster is 15 minutes.
I.e., your job will be killed after 15 minutes.


Since that is not likely to be sufficient for it to complete, specify a reasonable walltime. This greatly aids the scheduler in making the best utilization of resources.

The following example specifies a walltime of 60 seconds, which should be more than enough for the job to complete.

Example

#SBATCH -n 1
#SBATCH -t 0:60

hostname

Node and Core Requirements

Slurm provides many options for specifying your node and core requirements, and we only cover the basics here. More details can be found at the official Slurm site. Also see the man pages for sbatch (i.e. man sbatch).

From your job's perspective, of most concern is the number of CPU cores and how they are distributed over the nodes. Jobs generally use a combination of MPI tasks and/or multithreading for their for their parallelism.
We let N represent the number of MPI tasks, and M represent the number of threads needed by the job. Most jobs then fall into one of these categories:

  • sequential jobs: these jobs will run on a single CPU core (and therefore a single node). In this case N = M = 1
  • shared memory parallelism These jobs typcially use some sort of multithreading (e.g. OpenMP), and so for best performance require M CPU cores on a single node. Here M depends on the code and/or the problem being solved, and N >=1.
  • Pure MPI jobs jobs require a certain number of CPU cores (one for each MPI task but they can be spread out over multiple nodes (and the job generally does not care how they are spread over the nodes). In this case, M=1 and N > 1 depends on the code/problem being solved.
  • Hybrid MPI jobs: these jobs use MPI, but each MPI task uses multithreading as well. For these jobs both N > 1 and M > 1. and jobs, you will need N x M CPU cores, with each set of M cores must residing on the same node. The optimal values of both N and M depend on the code and the problem, and will be greater than 1.

The sbatch and related commands provide several options for controlling this behavior:

  • -n <var>N<var> or --ntasks=<var>N</var>: This sets the number of MPI tasks for the job, defaulting to 1. You should keep this at 1 (either explicitly or using the default) for sequential or purely multithreaded jobs. For MPI you should set this to the number of MPI tasks desired. As MPI jobs can use resources on multiple nodes, the scheduler is free to assign resources for different MPI tasks to different nodes (all resources for the same MPI task will be assigned to the same node).
  • -c <var>M</var> or --cpus-per-task=<var>M</var>: This sets the number of CPU cores to use for each (MPI) task. For non-MPI job, the entire job is treated as a single task. All the CPU cores or other resources for a given task will be allocated on the same node (resources for other tasks might or might not be assigned to the same node) This defaults to 1. For sequential jobs and pure MPI jobs jobs, this should be set to 1 (either explicitly or by default); for pure or hybrid or multithreaded jobs, this should be set to the number of threads desired (per MPI task for hybrid jobs).
  • -N <var>MINNODES[-MAXNODES]</var> or --nodes=MINNODES[-MAXNODES]: This specifies the number of nodes to use. This flag is generally not needed, and we recommend that you do not use this option and instead use the --ntasks and --cpus-per-task options, and let the schedule work out how many nodes are needed. Using it properly requires a good knowledge of the hardware available on the cluster, and using it improperly can result in your job wasting resources, being overcharged, and/or spending excessive time waiting in the queue. If you use it, you can give a range on the number of nodes to use. If MAXNODES is omitted, it will default to the value of MINNODES. But in general, it is best to omit this entirely and just give --ntasks and --cpus-per-task, and the scheduler will allocate just enough nodes to satisfy those specifications.

There are other flags you can use as well, see the sbatch man page (e.g. man sbatch) for more information, but the flags shown above are the most commonly usea, are simple to understand,d and cover almost all cases and so are what we generally recommend.

So for a sequential job you would usually use arguments like

Example

#SBATCH -n 1
#SBATCH -c 1

at the top of your job script. Both of those settings are the default, but it is usually recommended that you give at least one of them. See the example sequential job script for a full example submission script for a sequential job, along with a line by line commentary.

Similarly, for [multithreaded job[(/kb/glossary.md#thread], e.g. where you wish to run 10 threads and therefore want 10 CPU cores on the same node for maximum parallelization (which is usually the case when using an HPC cluster, you can use the arguments:

Example

#SBATCH -n 1
#SBATCH -c 10

See the example multithreaded job script for a full example.

For a pure MPI job, where each task is running only a single thread, you will want a single CPU core per task, and you generally do not really care whether all of the cores allocated to the job are on a single node or multiple nodes. In such a case, you could use something like:

Example

#SBATCH -n 60
#SBATCH -c 1

for a job with 60 tasks. See the example MPI job script for a full example.

For a hybrid parallel job, basically an MPI job in which each task has multiple threads, for full parallelism we will need to request enough CPU cores so that each thread in each task gets their own core. Furthermore, although the resources for each task can be on separate nodes, all of the resources for a single task must be on the same node. For such a case, we could use something like:

Example

#SBATCH -n 3
#SBATCH -c 15

for a job consisting of three tasks with each task having 15 cores or threads. For a script, see the example hybrid parallel job script.

If you are concerned about how your cores are allocated, you can also give the --nodes=*NUMNODESDESC* or -N *NUMNODESDESC*.
*NUMNODESDESC* can be of the form *MINNODES* or *MINNODES-MAXNODES*. In the former case, MAXNODES is set to the same as MINNODES. The scheduler will attempt to allocate between MINNODES and MAXNODES (inclusive) nodes to your job. So for the above example (--ntasks=12), we might have

  • all cores assigned on the same node if -N 1 is given.
  • the cores split among two nodes if -N 2 is given. You might get an even split, 6 cores each node, or an assymetric split, e.g. 4 on one node and 8 on the other. But you will get two distinct nodes.
  • either of the two above cases if -N 1-2 is given.

For most distributed memory (e.g. MPI) jobs, the job does not really care how many nodes the tasks are distributed over, and we recommend that most users just specify the --ntasks or -n parameter and let Slurm figure out how to best divide the cores among the nodes; this is likely to allow the job to run as early as possible. Of course, for shared memory (e.g. OpenMP or multithreaded) jobs, all of the resources need to be on a single node. You can ensure this by either using "--cpus-per-task" to request the number of cores, either with "--ntasks=1" or leaving it unset (so that it defaults to one), or you could explicitly give "--nodes=1".

Slurm's sbatch command has a large number of other options allowing you to specify node and CPU requirements for a wide variety of cases; the above is just the basics. More detail can be found reading the man page, e.g. man sbatch

Memory Requirements

You can explicitly request the amount of memory for your job with the "--mem" flag to the sbatch command. This flag takes a value specifying the amount of memory to reserve for the job, accepting values in the following formats:

  • number: the amount of RAM in megabytes
  • numberunit: the amount of RAM in the specified unit. Unit can be any of:
  • The special value 0 requests all of the memory on the node. This should be used sparingly, as you will be charged for the full node.

The "--mem" flag specifies the maximum amount of RAM for the job, per node. For sequential or purely multithreaded or jobs which are otherwise restricted to a single node, this is the total memory for the job.

For MPI jobs or other jobs which are (potentially) distributed across multiple nodes, it is usually more useful to request memory as a function of the number of CPU cores. Slurm provides a special flag to handle this case, "--mem-per-cpu". This also takes a value, in the same format as the "--mem" flag, but in this case the total memory allocated to the job will be the number of CPU cores requested times this value.

Actually, for both flags, the memory is actually allocated on a node basis. With the "--mem" flag, the specified value is allocated on each node allocated to the job. With the "--mem-per-core" flag, the amount of memory allocated to each node allocated to the job is equal to this value times the number of CPU cores on the node allocated to the job.

Generally, the "--mem-per-cpu" flag is recommended for multinode jobs. Single node jobs can use either "--mem" or the "--mem-per-cpu" flag, but usually the "--mem" form is more convenient (the two are equivalent for single core, i.e. sequential jobs, and for multithreaded jobs the memory required for a calculation is usually fairly indepent of the number of threads).

Example

#SBATCH --mem=5g

The above example requests 5 GB of RAM per node.

Example

#SBATCH --mem-per-cpu=1024

This example would request 1 GiB of RAM per CPU core allocated to the job.

By default, if neither the "--mem" or "--mem-per-cpu" or similar flag is given, your job will get a default of 4 GiB per CPU core requested.

Tip

Note that node selection does not count memory used by the operating system, etc. So a node which nominally has 512 GiB of RAM might only show 510 or so GiB available, due to some of the memory being in use by the operating system. So if you are intending to request all of the memory on a node, you should use the value of 0

Note

The hourly SU cost of the job will be more than the nominal 1 SU per CPU core requested if the job is requesting more than the average amount of memory per core for the nodes in the respective partition. For the standard compute nodes on the Zaratan cluster, that is 4 GiB per CPU core.

Your job will be charged based on the amount of memory requested, not what is actually used.

If requesting more than the average amount of memory per core for the partition, you should be monitoring successful jobs to ensure the memory being requested is actually being used. The seff command is useful for this purpose. Generally you should be aiming to be using at least 80% of the memory requested, and ideally more. If you are typically using less than that, please adjust the amount of memory requested to make more efficient use of the cluster. The Allocations and Advisory Committee will expect you to be monitoring your jobs to ensure efficient use before they will award additional resources.

Using GPUs

Certain types of calculations can greatly benefit from the use of GPUs. These GPU resources are expensive, both in terms of the purchase price of the hardware, and correspondingly the SU cost for running jobs on them; but for some calculations the performance gains from using GPUs are worth the cost. You should check the specific cluster being used to see the types of GPUs available on that cluster (along with their hourly SU cost):

Tip

Codes must be written specially in order to make use of the GPU hardware, typically using CUDA or similar. Some applications in the Research Software Library (RSWL) support GPUs, and some do not. Some packages which support GPUs have multiple builds; to ensure the GPU-enabled version is used, you should generally "module load cuda" first.

If you run a non-GPU enabled package in the GPU partition, it will only run on the CPUs, which means that your job will be wating valuable GPU resources without getting the performance benefit.

To request GPUs for your job on the Zaratan cluster, you must pass the proper parameters to the sbatch command to tell the scheduler that you want to use GPUs, and how many and what types of GPUs to use.

The sbatch command has a fair number of arguments specifically for specifying the GPU related resources a job needs, most of which are beyond the scope of this document; you can learn more about them with the man pages for sbatch (i.e. man sbatch). Here we discuss a more limited set of flags which are suitable for most cases.

The main parameter for specifying the GPUs required by the job is the --gpus flag; this should be provided the type and number of GPUs being requested, in a format like "name:number". The term number specifies the number of the GPUs of the specified type being requested for the job, and the name specifies the type of GPU. This is the recommended flag to use for requesting GPUs in a single node job.

Tip

While it is possible to use the more generic "--gres" flag to specify GPU requirements (in addition to any other GRESes), we recommend using the "--gpu" flag as it is more flexible.

Note

If you use the "--gres" flag, GPU types should be prefixed with "gres/gpu:", e.g. "gres/gpu:a100"

If you use the "--gpu" flag, do not include the "gres/gpu:" prefix, e.g. "a100"

For example:

Example

#SBATCH --gpus=a100:2

will request 2 GPUs of type A100.

Note

You are required to specify a GPU type.

If your job will potentially span multiple nodes, then it is recommended that you use the "--gpus-per-task" flag again. This flag takes an argument in the same name:number format as the aformentioned "--gpus" flag, except that in this case number refers to the number of GPUs allocated per MPI task. All resources (CPU, memory, GPUs) for the same task will be allocated on the same node, which is why this version of the flag is recommended for multinode jobs. Again, you are required to specify a GPU type.

On Zaratan, we have split the GPU nodes into a separate gpu partition to avoid GPU nodes being consumed by non-GPU jobs. So when submitting jobs requiring GPUs, you also need to specify the GPU partition.
You can do this with the --partition flag option to sbatch, e.g. "--partition=gpu".

For example:

Example

#!/bin/bash
#SBATCH -t 1:00:00
#SBATCH -n 1
#SBATCH -c 4
#SBATCH --gpus=a100:4
#SBATCH --partition=gpu

source /etc/profile
module load python
module load tensorflow
python my_tf_code.py

The example above requests a single task with 4 CPU cores and 4 A100 GPUs for 1 hour.

Tip

If no partition is specified for the job, the partition will default to "standard", unless GPUs were requested, in which case it will default to "gpu". While this should default the partition correctly in most cases, it might get it wrong in some cases, and we encourage you to explicitly set the partition correctly.

GPUs are expensive, and accordingly their use will consume more of the SUs in your allocation, with more expensive GPUs having a higher hourly SU rate. You should be consider the choice of GPU for your job carefully. The different GPU types have different amounts of memory, and you need to consider what the minimal GPU memory requirements of your job is.

Note

The GPU memory is an integral part of the GPU, and is distinct from the CPU memory usually discussed. When you request a GPU, you get all of the memory on that GPU --- you cannot request more or less GPU memory except by changing the number or type of GPUs being requested.

The Slurm/sbatch flags for specifying the job's memory requirements are for CPU memory only; they have no effect with respect to GPU memory.

Once you determine which of the available GPUs your job is able to run on due to GPU memory restrictions, you should conduct tests of your code on different GPU types to determine which GPU configuration is optimal for you. E.g., an H100 GPU on Zaratan coststhree times as many SU per hour than an A100 on the same cluster; so unless you get about 3 times or more better performance on an H100 vs an A100 (and assuming the job will fit on either), you probably should be using the A100 GPUs. For smaller jobs with modest GPU memory requirements, you should consider using the fractional a100_1g.5gb GPUs.

Currently, the GPU nodes on Zaratan have 4 GPUs each. It is possible for four single GPU jobs to run on the same node in "shared" mode. Slurm will set the environmental variable CUDA_VISIBLE_DEVICES to the GPU(s) which it allocated to your job, e.g. to 0 if it assigned you only the first GPU, 1 if it assigned only the second, or 0,1 if it assigned both. By default, CUDA will use this variable and will only use the specified GPU(s). So two single GPU CUDA jobs should be able to coexist on the same node without interfering with each other. (However, problems might occur if one of the jobs is not CUDA based, or if the job doing things it should not be doing.)

Amount of Temporary Space Needed

All of the compute nodes have a node local filesystem mounted at /tmp which you can use to read and write to while your job is running. This is temporary space, and your files will be deleted once your job completes. If your job requires more than a small amount (10GB) of space on this node local filesystem, it is recommended that you specify how much you need when you submit the job so that the scheduler can assign appropriate nodes to you.

All of the compute nodes on the Zaratan cluster currently have at least 1 TB of temporary space, based on solid state disk technology.

The following example specifies a scratch space requirement of 5GB. Note however that if you do this, the scheduler will set a filesize limit of 5GB. If you then try to create a file larger than that, your job will automatically be killed, so be sure to specify a size large enough for your needs.

Example

#!/bin/sh
#SBATCH --ntasks=8
#SBATCH --tmp=5120

myjob

For the disk space size, you can give an integer (which will be interpretted as that many MB, or an integer followed by "G" or "T" suffix for that many GB or TB.

Node features or resources

Sometimes your job requires nodes with specific features or resources. I.e., some jobs will make use of GPUs for processing, or some jobs may require specific processor models. Such requirements need to be told to the scheduler to ensure you get assigned appropriate nodes.

In slurm, we break this situations into two cases:

You can see which nodes support which features and resources with sinfo command or the scontrol show node command. The features and resources available by cluster are discussed in the cluster specific pages:

To request a specific feature, use the "--constraint" option to sbatch. In its simplest form, you just give "--constraint=TAG", where TAG is the name of the feature you are requesting. E.g., to request a node running Red Hat Enterprise Linux 8, (the rhel8 feature) you would use something like:

Example

#!/bin/bash
#SBATCH -t 15:00
#SBATCH --ntasks=8
#SBATCH --constraint="rhel8"

The --constraint option can get rather more complicated, as Slurm allows multiple constraints to be given, with the constraints either ANDed or ORed. You can request that only a subset (e.g. 2 out of 4) nodes need to have the constraint, or that either of two features are acceptable, but all nodes assigned must have the same feature. If you need that level of complexity, please see the sbatch man page (man sbatch)

Resources are requested with the --gres option to sbatch. The usage is "--gres=RESOURCE_LIST", where RESOURCE_LIST is a comma delimitted list of resource names, optionally followed by a colon and a count. The resources specified are required on each node assigned to the job. There are also special options for requesting GPUs which you are recommended to use when using GPUs; these are discussed in the section on using gpus.

Requesting specific CPU types

As our clusters grow, they generally become less homogeneous.
While that might not matter for some workloads, for others you might need a specific CPU architecture.

We have provided features on all of our nodes to help facilitate the specification of the desired CPU architecture. This can be done at various levels, ranging from the most general (whether to use AMD or Intel based cpus with the features "amd" or "intel") to very specific (selecting a specific CPU model).

You can instruct the sbatch command to use a particular subset of nodes matching the desired CPU architectures by adding a --constraint flag, as discussed in the section on 'features'. A listing of the various CPUs you can select from on a given cluster is included in the feature list for that cluster:

Whether other jobs can share the node

The standard compute nodes on the Zaratan are rather hefty, having 128 CPU cores and 512 GiB of memory. Although some jobs on the cluster need an entire node, or even multiple nodes for optimal performance, many jobs require far less resources. For such jobs the allocation of an entire node would be excessive, so for better utilization of the cluster the scheduler will generally schedule multiple jobs on the same node as long as the requested resources will fit on the node. E.g., if there are two jobs, jobA requesting 64 CPU cores and 300 GiB of RAM, and jobB requesting 40 cores and 160 GiB of RAM, both jobs can fit comfortably on the same node.

The default behavior of the scheduler is to allow multiple jobs to coexist on the same the node. The sbatch command has the additional flags "--exclusive" and "--oversubscribe" which control this behavior. The default behavior is equivalent to the "--oversubscribe" flag. If the "--exclusive" flag is given, the scheduler will enforce the opposite behavior, no other jobs will be allowed on any of the nodes on which the "exclusive" mode job it running. However, because no other jobs can run on the node while your job is running, your job will be charged for the usage of the entire node or nodes which the job is running on. E.g., if you submit a job requesting 32 CPU cores and 128 GiB of RAM in exclusive mode on a standard Zaratan compute node, you will be charged for all 128 cores on the node even though your job is not using it.

Miscellaneous sbatch options

Choosing a Partition

On the Zaratan cluster, you have a choice of several partitions in which to run your job. Partitions are used to separate resources or job types. For example, a partition for jobs that need GPUs, or one for debug jobs. By default, if you do not specify a partition, your job will be placed in the standard partition.

For quick test jobs you may want to use the debug partition. This partition is intended for quick turn around for small debugging jobs, but has a 15 minute maximum walltime so is not suitable for production.

For a complete list of partitions and their description.

To specify a partition, you just give the sbatch command the --partition=PARTNAME argument, or equivalently the -p PARTNAME argument, replacing PARTNAME with the name of the partition. E.g., to submit a job to the debug partition, you could add to your sbatch command the arguments -p debug. Similarly, to submit a job to the scavenger partition, you could add -p scavenger to the sbatch. In either case, you can either append the partition flag to the end of the command line, or add it near the top of your job script with a #SBATCH prefix, e.g. for the debug partition, e.g. #SBATCH -p debug

Jobs charged to normal accounts take precedence over low-priority scavenger partition jobs. With the exception of scavenger partiton jobs, no job will preempted by another job; i.e. once a job in the standard, gpu, or debug partitions starts, it will continue to run until it finishes (either successfully or crashes) or it walltime expires. However, jobs in the scavenger partition are subject to preemption, so after a scavenger job starts its execution, another if another, non-scavenger job wants the resources the scavenger job is using, the scavenger job will be stopped and (normally) placed back in the queue.

Tip

Do not use the scavenger partition unless you know what you are doing. Generally, all scavenger jobs need to make use of checkpointing so that they can make progress with small slices of runtime here and there.

The account to be charged

All users of the HPC cluster belong to at least one project associated with the cluster, and each project has at least one allocation account its users can charge against. You can see which allocations you belong to with the sbalance command, e.g

Example

login-3:~$ sbalance
Account: test-aac (DEFAULT)
Limit:     50.00 kSU
Unused:    50.00 kSU 
Used:      0.00 kSU (0.0 % of limit)

Account: test-paid (DEFAULT)
Limit:     50.00 kSU
Unused:    5.99 kSU 
Used:      44.01 kSU (88.0 % of limit)

The above shows that the user which ran the command belongs to two allocations, named "test-aac" and "test-paid". Many users only belong to a single allocation while others may belong to two or even many allocations.

Every user also has a default allocation, and if no allocation is explicitly specified when you submit a job, the job will charge against the submitter's default allocation. If you only belong to a single allocation, that will be your default allocation and you can skip this section.

If you belong to multiple allocations, you need to select an appropriate allocation for the the job you are about to submit to charge against. Guidance on doing such can be found on the allocations page. But basically, if you belong to multiple projects, figure out which project the job benefits most. If there are multiple allocations within that project, you should find one with sufficient funds to run the job. If multiple allocations have sufficient funds, we suggest you use allocations which replenish quarterly before using "one-time" grants (like AAC allocations).

To submit jobs to an account other than your default account, use the "-A" or "--account" flag to the sbatch command:

Example

login-1:~: sbatch -A test test.sh
Submitted batch job 4194

or

Example

#!/bin/bash
#SBATCH --account=test
# ... rest of script

If no account is explicitly specified, your job will be charged against your default account. You can view and/or change your default account with the sacctmgr command.

Email Options

The scheduler can email you when certain events related to your job occur, e.g. on start of execution, or when it completes. By default, any such mail is sent to your "@umd.edu" email address, but you can specify otherwise with the "--mail-user=EMAILADDR" flag to sbatch.

You can control when mail is sent with the "--mail-type=TYPE" option. Valid options are:

  • BEGIN: when the job starts to execute
  • END: when the job completes
  • FAIL: if and when the job fails
  • REQUEUE: if and when the job is requeued.
  • ALL: for all of the above cases.

To have mail sent on multiple conditions, give a comma delimitted list of conditions as the argument of the --mail-type flag. E.g., the following job script will send mail to "spider@avengers.com" when the job starts and when the job finishes:

Example

#!/bin/bash
#SBATCH --mail-user=spider@avengers.com
#SBATCH --mail-type=BEGIN,END

Tip

NOTE: It is recommended that you use care with these options, especially if you are submitting a large number of jobs. Not only will you get a large amount of email, but it can cause issues with some email systems (e.g. GMail imposes limits on the number of emails you can receive in a given time period.

Warning

Do not repeat the '--mail-type' flag for different conditions. If multiple '--mail-type' flags are given, the results are not cumulative; instead the last flag given completely replaces any previously given --mail-type flags. E.g. in the submission script

#!/bin/bash
#SBATCH --mail-user=spider@avengers.com
#SBATCH --mail-type=BEGIN
#SBATCH --mail-type=END
#SBATCH --mail-type=FAIL

the job will only send email when it ends, not on start, end, and failure as was likely intended. To have email sent on all three conditions, you would need to use something like:

#!/bin/bash
#SBATCH --mail-user=spider@avengers.com
#SBATCH --mail-type=BEGIN,END,FAIL

Output options

By default, Slurm will direct both the stdout and stderr streams for your job to a file named "slurm-JOBNUMBER.out" in the directory where you submitted the sbatch command. For job arrays, the file will be "slurm-JOBNUMBER_ARRAYINDEX.out". In both cases, JOBNUMBER is the number for the job, and in the latter case ARRAYINDEX is the index of the job within the job array.

You can override this with the option "--output=FILESPEC" (or "-o FILESPEC" for short). FILESPEC is the name of the file to write to, but the following replacement symbols are supported:

  • "%A": The master job allocation number for job arrays master allocation number for the job array.
  • "%a": The job array index number, only meaningful for job arrays.
  • "%j": The job allocation number.
  • "%N": The name of the first node in the job.
  • "%u": Your username

Multiple replacements symbols are allowed in the same FILESPEC. The default values are "slurm-%j.out" for non-array jobs and "slurm-%A_%a.out" for array jobs.

You can also use --error=*FILESPEC* (or -e *FILESPEC*) to have the stderr sent to a different file from stdout. The same replacement symbols are allowed here.

If only the "--output" or "-o" flag is set, both stdout and stderr will be directed to the file specified for that flag. If only "--error" or "-e" is specified, then stdout will go to the default file (e.g. "slurm-%j.out") and stderr will go to the specified file. If both are given, both streams are redirected.

Tip

It is generally a good idea to keep the job number (or job number and array index) in the filenames. This will prevent future jobs from overwriting the outputs of previous jobs, which is extremely helpful when diagnosing issues.

Tip

It is generally a good idea to keep both stdout and stderr in the same file. Doing so allows one to get a rough idea of where in the execution of the job the diagnostic messages in stderr occur (i.e., one can see where the diagnostic message occurred relative to the standard output messages).

If stdout and stderr are sent to different files, one cannot do such correlation of the outputted text, especially if the diagnostic message is not immediately fatal.

Keeping both streams in the same file is sometimes helpful with diagnosing problems.

Submitting Job Arrays

Job arrays are a means by which you can submit a large number of similar jobs at a single time, without overloading the scheduler. Job arrays are submitted with a single sbatch command, taking a single job script file. Within the job script, you can use the environmental variable "SLURM_ARRAY_TASK_ID" to make the script perform a different calculation for each element of the array. The example job array job shows an example of doing this.

Job arrays are created by passing the additional argument "--array" or "-a" to the sbatch command. The argument to this flag specifies the indices to use for the job array. It should be a comma delimited list of index numbers, or ranges of index numbers using a form like "INDEX_LO-INDEX_HI" to mean the numbers between INDEX_LO and INDEX_HI, inclusive. You can append a ":STEP" suffix to the range, so e.g. "--array=0-15:4" would be equivalent to "--array=0,4,8,12".

You can add the suffix "%MAX" to control how many jobs the scheduler will allow to run at once. E.g. "--array=0-100,120-140%4" will run at most 4 jobs from the array at any given time (once one of the jobs terminates, another job can start, and so forth until all jobs have run).

All of the other #SBATCH directives or flags apply to each individual job in the array; e.g. if you have a job array with

Example

#SBATCH -t 4:00:00
#SBATCH -n 4
#SBATCH -c 8
#SBATCH --mem-per-cpu 2048
#SBATCH --array=0-99

this will define a job array consisting of 100 jobs, with each job requesting 4 [MPI tasks](/kb/glossary.md#task] of 8 CPU cores per task, and 2 GiB per CPU core (for a total of 64 GiB per job.

The shell to run in

By default, your job will be executed in whatever shell is specified by the by the shebang line in the script.

Tip

NOTE: If you wish to use a bourne style shell, we strongly recommend #!/bin/bash instead of #!/bin/sh. Under linux, both run the same bash executable, but in the latter case certain non-backwards compatible features are disabled which can causes problems.

Tip

We recommend most users use bash for both interactive use and for their job submission scripts.

Directory the job runs in

By default, the working directory in which your job runs will be the directory from which you ran the sbatch command, unless you specify otherwise.

You can use the "-D WDIR" or "--chdir=WDIR" flags to the sbatch command. This will change the working directory for the job. This will also affect the directory into which the Slurm output files.

Alternatively, and usually more easily, you can just include an appropriate "cd" command in your script. This alternative will not change the directory in which the Slurm output files are written.

If you are using MPI, you may also need to add either the "-wd" or "-wdir" option to the mpirun to specify the working directory.

Specifying a reservation

On rare occasions, a reservation might be set up for a certain allocation account. This means that certain nodes and/or resources have been reserved for a specific period of time for that allocation. This is not done often, usually only for limited use for the occasional lab sessions for classes or workshops (so that students can launch jobs and get the results back while the class/workshop is still in session. Again, this is only done rarely.

You should only be specifying a reservation for your job when you have been instructed to do so, either by HPC staff or the instructor of the class. Most users do not have access to any reservations and can ignore this section. If you specify a reservation which does not exist, or to which you do not have access, or which is not currently active, your job will either not be accepted by the scheduler, or it will end up sitting forever in the queue and never run.

If you do have access to a reservation which is active, you can submit jobs which use the reserved resources by adding the following flag to your sbatch command: "--reservation=RESERVATION" where RESERVATION is the name of the reservation (which should have been provided to you by the person who instructed you to use the reservation). If you were not told of a reservation name, you most likely do not have access to a reservation and this section does not apply to you. this section does not apply to you. The "--reservation=RESERVATION" flag can either be given as an explicit argument on the "sbatch" command line, or as a #SBATCH directive in your job script.

Note

NOTE: to effectively use a reservation, the following conditions must hold:

  • You must be charging the job to an allocation account that has access to the reservation. For class reservations, this typically means that you must be charging the job against the class allocation account.
  • You must specify that the job should use the reservation, i.e. use the "--reservation" flag described above.
  • The reservation must be "active". Class and workshop reservations are typically only active during the hours the class/workshop meets for days for which the instructor requested a reservation. If you submit a job specifying a reservation when the reservation is not active, instead of expediting things it will likely delay the job until the reservation becomes active (or forever if the reservation already passed).