Skip to content

FF chevron

Commands and Utilities

This page lists various commands or utilities available on the UMD DIT-maintained HPC clusters.

fs

The fs command is the main interface to the Auristor cache manager. This is actually a whole suite of commands underneath the main "fs" command token. Most of the subcommands are more for administrative purposes or for rather arcane situations, so we only discuss a very small number of subcommands, but those compromise most of the use cases you are likely to encounter.

fs listacl

The "fs listacl" command will display the Auristor ACL for a directory. The typical usage is just to give the name of the directory as an argument, like:

Example

login-3:~$ fs listacl ~/SHELL/spells
Access list for /home/hpotter/SHELL/spells/ is
Normal rights:
  hpotter rlidwka
  proj:gryffindor rl
  proj:gryffindor_mgr rlidwka
  system:administrators rlidwka

In this case, the spells directory gives full access to the owner (hpotter), as well as full access to the managers of the gryffindor project and the system administrators. The regular members of the gryffindor project have read access.

fs setacl

The "fs setacl" command cab modify the Auristor ACL for a directory. You generally need to provide the path to the directory as argument to the "-path" flag, and the ACL as argument to the "-acl" flag, like:

Example

login-3:~$ fs listacl ~/SHELL/spells
Access list for /home/hpotter/SHELL/spells/ is
Normal rights:
  hpotter rlidwka
  proj:gryffindor rl
  proj:gryffindor_mgr rlidwka
  system:administrators rlidwka
login-3:~$ fs setacl -path ~/SHELL/spells -acl proj:gryffindor none
login-3:~$ fs setacl -path ~/SHELL/spells -acl hermione read
login-3:~$ fs listacl ~/SHELL/spells
Access list for /home/hpotter/SHELL/spells/ is
Normal rights:
  hpotter rlidwka
  hermione rl
  proj:gryffindor_mgr rlidwka
  system:administrators rlidwka

In this case, we remove the ACL for the gryffindor project members (by setting the permissions to "none") and then add read permission to hermione exclusively. We list the acl before and after to see the change.

getfacl

The getfacl command will display the POSIX ACLs associated with a file or directory. Although there are a number flags that can be given to the command (which you can read on the getfacl man page or run man getfacl), the basic usage is simply to just provide the name of a directory or filename.

Example

login-2:~$ ls -l /scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt
-rwx------.  1 hermione zt-gryffindor 4652867 Oct 15  2024 polyjuice-recipe.txt
login-2:~$ getfacl /scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt
getfacl: Removing leading '/' from absolute path names
# file: scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt
# owner: hermione
# group: zt-gryffindor
 user::rwx
 user:hpotter:r--
 group::---
 mask::r--
 other::---

The above example shows the ACLs for the file "/scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt". It displays the name of the file, it ownea (hermione)r and group owner (zt-gryffindor) in the lines starting with the octothorpe/number sign/hash tag ("#"). The lines "user::rwx", "group::---", and "other::---" show the traditional Unix file permissions, indicating that the standard permissions are "rwx------", agreeing with the ls output.

The line "user:hpotter:r--" indicates that an ACL was added giving the user "hpotter" read access to the file.

For directories, you may see additional ACLs starting with "default:", these represent the default ACLs for the directory.

The getfacl command will work for files and directories in the scratch filesystem. You can use the setfacl command to set ACLs.

See also:

The home_quota command will show your your usage and quota on the home directory. It will also show information on the "grace" period; we normally allow users to exceed their 10 GB homespace quota by up to another 10 GB for about 1 week; if you see something other than "[n/a]" in the "Grace Ends" field, you have exceeded your homespace quota and you will be unable to save any data to your home directory once the grace period ends (or you go over the 20 GB hard limit).

home_quota

The home_quota command will show your your usage and quota on the home directory. It will also show information on the "grace" period; we normally allow users to exceed their 10 GB homespace quota by up to another 10 GB for about 1 week; if you see something other than "[n/a]" in the "Grace Ends" field, you have exceeded your homespace quota and you will be unable to save any data to your home directory once the grace period ends (or you go over the 20 GB hard limit).

Example

login-2:~# home_quota
#Username:   Used space/Limit      (%)    [Grace Ends]; Used files/Limit    (%)   ;
voldemort:   3.578 GB/10.240 GB  34.9%    [n/a];        1.808 k/650.000 k    0.3% ;

In the above example, user voldemort is using about 3.6 EB of the 10 GB quota, or just under 35%. If the user was over their home directory quota/limit, there will be a date in the Grace Ends field giving the date at which the user's "grace period" expires --- at that point the directory will be officially overquota and new new content will be able to be added. (The directory can also go over quota if usage exceeds 20 GB; whichever occurs first).

Tip

If you see a date in the Grace Ends field when you run home_quota, you should quickly clean up your home space; either deleting unneeded files or moving files elsewhere (e.g. scratch or SHELL storage.

module

The Research Software Library (RSWL) contains many software packages, and often multiple versions of the same package. The environmental settings needed to run a given package can sometimes conflict with other packages, and will usually conflict with other versions of the same package. Due to these conflicts, we cannot have all packages available to you as soon as you log in.

Instead, we use the "module" command to control the environment as it relates to software packages. The "module" command has several commonly used modes which control its behavior, namely:

  • "module list": lists which modules you have loaded
  • "module avail STRING": this will list all known modules starting the the string STRING.
  • "module help": this will display help for the module command itself
  • "module help MODTAG": this will display the help for the module named MODTAG
  • "module load MODTAG": this will load the environment settings for the package corresponding to the tag MADTAG
  • "module unload MODTAG": this will undo the environment changes from the corresponding "module load" command
  • "module purge": this is equivalent to doing "module unload" for every loaded package

The MODTAG text in the above is the name of the module. This typically includes the base name of the package, the version of the package, and possibly additional information like the compiler and or MPI library with which it was built, and the CPU for which it was optimized.

Generally, the tag for a package can be composed in one of two ways: - 1. The basename of the package 2. The version of the package 3. The compiler (family / version) with which the package was built 4. The MPI library (family / version) with which the package was built, or "nompi" for a non-MPI version of an MPI aware package. 5. The CUDA or (cuda/version) or ROCm (rocm/version), or "nocuda" for non-GPU enabled version of GPU-aware packages. 6. The CPU architecture the package was optimized for. - 1. The basename of the package 2. The compiler (family / version) with which the package was built 3. The MPI library (family / version) with which the package was built, or "nompi" for a non-MPI version of an MPI aware package. 4. The CUDA or (cuda/version) or ROCm (rocm/version), or "nocuda" for non-GPU enabled version of GPU-aware packages. 5. The CPU architecture the package was optimized for. 6. The version of the package

For example, the tag:

hdf5/1.14.0/gcc/11.3.0/openmpi/4.1.5/zen2

represents the package "hdf5", version 1.14.0, built with gcc version 11.3.0 and OpenMPI 4.1.5 and optimized for the AMD zen2 chipset. The same exact build of the package is represented by the tag

hdf5/gcc/11.3.0/openmpi/4.1.5/zen2/1.14.0

But generally you can get away with just specifying the basename of the package or the basename and the version, e.g. "hdf5" or "hdf5/1.14.0". The module command will attempt to default the rest; if you have a compiler or MPI library "module loaded" already, the "module command" will use that by default. The package version will default to the most recent version available, and the CPU architecture will default to the most advanced CPU the package is built for which is compatible with the CPU the process is running on.

Normally we recommend you start by module loading the compiler and, if appropriate, the MPI and CUDA modules. This will ensure that you load the correct version of the package.

Tip

For packages with both MPI and non-MPI variants, you must load an MPI module before loading the package to ensure you get the MPI-enabled build.

Similarly, for packages with both GPU and non-GPU variants, you must load a CUDA or ROCm module before loading the package to ensure you get the GPU-enabled build.

So you to get the hdf5 package from the example above, you can do something like:

Example

login-2:~> module load gcc/11.3.0
login-2:~> module load openmpi
login-2:~> module load hdf5

mpiexec

See the mpirun command, the two commands are equivalent for most modern MPI implementations.

mpirun

The mpirun command is typically used to launch MPI-enabled codes. Generally you can just module load the correct compiler, MPI library, and other needed software packages and then use a command like: mpirun myMPIcode to launch your MPI-enabled program "myMPIcode".

Tip

For most modern MPI implementations, The "mpiexec" command is just another name for the "mpirun" command and either command can be used interchangably.

Tip

Each implementation of MPI has its own version of the "mpirun" and "mpiexec" commands (in most implementations, the two are actually a single command).

It is strongly recommended that you use the exact same build of the MPI library (and its corresponding mpirun command) when running your code as was used to compile the code.

The MPI libraries in the Research Software Library (RSWL) are generally aware of the Slurm scheduler used on the cluster and are therefore able to obtain information about the number of MPI tasks and the nodes allocated to the job from the environmental variables set by Slurm at the start of the job, and therefore you generally do not need to explicitly provide such as arguements to the command line.

The mpirun command can take many options; you can generally see this using the "man mpirun" command after module loading the appropriate MPI library; many of the more advanced options are specific to the specific implementation.

renew

Some of the resources on the UMD HPC clusters require either valid Kerberos tickets or valid AFS tokens.

Although you should automatically get both kerberos tickets and AFS tokens when you log into a login node (unless you were using SSH public key authentication, these tickets and tokens have an expiration, typically only being valid for one day.

If your ssh or other session is long-lived, your tickets and tokens might have expired. If so, you can use the "renew" command to refresh them.

When you run the renew command, if your kerberos tickets are still valid (e.g. have not expired), it will silently refresh your tickets and tokens (extending the expiration). If your kerberos tickets already expired (or you did not have any to begin with), you will be prompted for your password and a new set of tickets and tokens will be issued.

sacct

The sacct command can be used to view the accounting records of jobs and job steps, both past and currently running (although for running and especially queued jobs, many fields are not populated). It can take some time to run, and can display a fair amount of information, and so it has quite of few flags available to control both aspects. We discuss a small number of more commonly used flags below; for more information on these flags or on the other flags available, please see the sacct man page (or run "man sacct").

As there are millions of jobs per year run on the cluster, you will almost certainly want to filter the jobs which are displayed. The following flags are useful for that purpose:

  • "--accounts=ACCTS" or "-A ACCTS": Only display jobs charging against one of the specified Slurm allocation accounts. ACCTS should be a comma separated list of allocation account names.
  • "--allocations" or "-X": only show data relevant to the job allocation itself, instead of show records for every job step.
  • "--allusers" or "-a": By default, the sacct command only shows your jobs; with this flag it will display jobs for all users which match the filter.
  • "--endtime=DATE" or "-E DATE": only display jobs which finished before DATE. DATE should be of the format "YYYY-MM-DD)" or "_YYYY-MM-DD_T_hh:mm:ss".
  • "--jobs=JOBLIST" or "-j JOBLIST": Only display information for the specified jobs. JOBLIST should be a comma delimited list of one of more job numbers.
  • "--nodelist=NODES" or "-N NODES": Only display jobs that ran on one of the nodes listed in NODES. NODES can be a comma delimited list, and can include ranges (e.g. compute-b8-[1-5]).
  • "--partition=PARTITION" or "-r PARTITION": Only display running in the specified partition.
  • "--starttime=DATE" or "-S DATE": only display jobs which started after DATE. DATE should be of the format "YYYY-MM-DD)" or "_YYYY-MM-DD_T_hh:mm:ss".
  • "-uid=USERNAME" or "-u USERNAME": only display jobs submitted by the specified user. (Note that this is not useful unless you also include "--allusers".)

The following flags control what information is displayed and how:

  • "--format=FIELDS" or "-o FIELDS": This controls which fields are displayed. FIELDS should be a comma separated list of field names. Use the "--helpformat" flag to see a list of all available field names.
  • "--helpformat": This displays the list of fields available for the "--format" flag.
  • "--parsable" or "-p": This changes the output format to print fields separated by the pipe (|) character. This is useful as often the content of the fields exceeds the default field widths, truncating the data often to the point of being unusable.
  • "--parsable2" or "-P": Like "--parsable", but no trailing pipe (|) character is printed at the end of each line.

For example, to see all of the jobs charging against the allocation account foo-aac for the month of November 2024, one could use a command like:

Example

login-1> sacct -P -X --start 2024-10-01 --end 2024-10-31 -A foo-aac 
JobID|JobName|Partition|Account|AllocCPUS|State|ExitCode
8057991|test1|gpu|foo-aac|FAILED|2:0
8148243|test_job2|gpu|foo-aac|COMPLETED|0:0
8148253|job3|gpu|foo-aac|CANCELLED by 123456|0:0
8148261|job4|gpu|foo-aac|FAILED|127:0
8148381|job5|gpu|foo-aac|FAILED|1:0
8148416|test_job3|gpu|foo-aac|COMPLETED|0:0

As seen above, the default fields displayed are not the most useful, it is often more useful to provide a specific list of fields you are interested in, like:

Example

login-1> sacct --format=JobID,State,AllocTRES,Elapsed \\
-P -X --start 2024-10-01 --end 2024-10-31 -A foo-aac 
JobID|State|ReqTRES|AllocTRES|Elapsed
8057991|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:25
8148243|COMPLETED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|01:26:04
8148253|CANCELLED by 123456|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:06
8148261|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:03:05
8148381|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:02:07
8148416|COMPLETED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|02:13:37

Some useful output fields are:

  • AllocTRES: shows the amount of resources that were actually allocated to the job. This normally should be the same as what was requested, but can vary in some cases (e.g. if the jobs were run in exclusive mode.
  • AveCPU: The average CPU time (system + user) for all tasks in the job
  • AveRSS: The average resident set size (RSS) of all tasks in the job. This is essentially the average memory usage per task for the job.
  • Elapsed: shows the amount of time the job ran, in hours:minutes:seconds
  • End: The end time of the job.
  • ExitCode: The exit code returned by the job script. If the job exited with a signal, it will be the exit code followed by a colon and the number of the signal.
  • JobID: this is the Slurm job ID number
  • JobName: the name of the job or job step.
  • MaxRSS: The maximum resident set size (RSS) of all tasks in the job. This is essentially the maximum memory usage per task for the job.
  • MaxRSSNode: The name of the node which had the MaxRSS.
  • MaxRSSTask: The number of the task which had the MaxRSS.
  • MinCPU: The minimum CPU time (system + user) for all tasks in the job
  • MinCPUNode: The name of the node which has the MinCPU.
  • MinCPUNode: The number of the task which has the MinCPU.
  • NCPUS: The total number of CPU cores allocated to the job.
  • NNodes: The total number of nodes allocated to the job.
  • NodeList: The names of all the nodes allocated to the job.
  • NTasks: The number of tasks in the job.
  • Partition: The partition the job ran in
  • ReqTRES: shows the resources that were requested by the job. This includes the number of CPU cores(cpu), the total amount of CPU memory(mem), the number of nodes(node) and number and type of GPUs. The special flag billing gives the hourly cost of the job in SU.
  • Start: The start time of the job.
  • State: this state of the job. In this case we see a few jobs which FAILED, a couple which COMPLETED successfully, and one that was CANCELLED.
  • Timelimit: The timelimit for the job.
  • TotalCPU: The total CPU time (system + user) for the job.
  • TresUsageInAve or TresUsageOutAve: The average TRES usage for all tasks in the job.
  • TresUsageInMax or TresUsageOutMax: The maximum usage for each TRES among all tasks in the job.
  • TresUsageInMaxNode or TresUsageOutMaxNode: The name of the node for which TresUsageInMax or TresUsageOutMax, respectively occurred, by TRES.
  • TresUsageInMaxTask or TresUsageOutMaxTask: The number of the task for which TresUsageInMax or TresUsageOutMax, respectively occurred, by TRES.
  • TresUsageInMin, TresUsageOutMin, TresUsageInMinNode, TresUsageOutMinNode, TresUsageInMinTask, and TresUsageOutMinTask: Similar to the corresponding Max fields, but for the minimum usage of the TRES.
  • TresUsageInTot or TresUsageOutTot: The total usage for each TRES among all tasks in the job.
  • User: The username of the user whose job it is

The In/Out distinction for the various TresUsage* fields is only relevant for resources which can be either ingress or egress, i.e. disk reads (In) or disk writes (Out), etc. For other fields it does not matter.

For a more complete listing of the available fields, see the sacct man page (or run "man sacct") or use the "sacct --helpformat" command.

In particular, you can use the information above to calculate the SU cost per job: the value of the billing trackable resource (TRES) in the AllocTRES field represents the hourly SU cost for the job. Multiplying that by the elapsed time in hours yields the SU cost of the job. E.g., job 8148416 in the above example has an hourly billing rate of 96 SU/hour (due to the use of two A100 GPU), and it ran for 2.23 hours, so the SU cost for the job is 214.1 SU.

sacctmgr

The sacctmgr command is primarily used for administrative operations which you as an normal user will not have access to. However, there is one task for which it is useful to end users, the viewing and setting of your default allocation account. This is the allocation account against which jobs will charge if no allocation account is explicitly passed to the sbatch command.

The following example shows how user "pbparker" can view his default allocation and then change it from "dailybugle-paid" to "avengers-paid".

Example

login-2:~$ sacctmgr list user pbparker -p
User|Def Acct|Admin|
pbparker|dailybugle-paid|None|
login-2:~$ 
login-2:~$ sacctmgr modify user pbparker set DefaultAllocation=avengers-paid
 Modified users...
  pbparker
Would you like to commit changes? (You have 30 seconds to decide)
(N/y): y
login-1:~/slurm-tests:
login-1:~/slurm-tests: sacctmgr list user pbparker -p
User|Def Acct|Admin|
pbparker|avengers-paid|None|

The "-p" option to the sacctmgr command causes the command to produce "pipe" ("|") delimitted output; without it the allocation names will usually be abbreviated and omit the allocation type suffix, which is usually important.

Users have permission to change the default allocation of their login account. Note that you must already be a member of the allocation you are choosing as your default allocation.

salloc

The salloc command will request that Slurm allocate a set of resources (CPU cores, memory, GPUs, etc) for a specified amount of time. You can optionally have the salloc command run a command on those resources, or you can use the srun command to launch a command on those resources.

Generally we recommend using the sbatch command instead.

sbalance

The sbalance command allows you to see the amount of compute time (in SU your allocation(s) have been allotted, have used, and have unused. Without any arguements, it will display summary information for all of the allocations to which you belong, e.g.

Example

        login-1:~> sbalance
        Account: test-aac
        Limit:     50.0 kSU
        Available: 37.47 kSU 
        Used:      12.53 kSU (25.1 % of limit)

        Account: test-paid
        Limit:     150.00 kSU
        Available: 102.33 kSU 
        Used:      47.67 kSU (31.8 % of limit)

The above example shows that the user who ran the command belongs to two allocations (test-aac and test-paid). The former allocation is for 50 kSU, 12.5 kSU of which have been consumed. The latter is for 150 kSU, of which 47.7 kSU had been used thus far this quarter. All usages are for the current quarter.

The --account or -A flag accepts the name of a Slurm allocation account, and will show information about that account. The --user or -u flag accepts an username, and will list information about all allocations to which the specified user belongs.

If you add the -all flag to the command, it will break information about which users have consumed resources and how much, e.g

Example

        login-1: sbalance -account smith-prj-aac -all
        Account: smith-prj-aac (DEFAULT)
        Limit:     163.52 kSU
        Available: 102.07 kSU 
        Used:      61.45 kSU (37.6 % of limit)
                User jones used 17.6044 kSU (28.6 % of total usage)
                User smith used 13.3456 kSU (21.7 % of total usage)
                User wong used 30.5000 kSU (49.6 % of total usage)

This lists the same information as before, with the addition of showing every user who has used the account for the current quarter, showing not only the number of kSU they consumed, but what percentage of the total usage for the account. E.g., in the example above, you can see that user wong is using almost as much as users smith and jones combined. You can add the flag --nosuppress0 if you want to also see lines for everyone with access to the allocation but who did not consume any time since the start of the quarter.

The --help option to sbalance will display usage options, most of which were discussed above.

sbatch

The sbatch command is one of the main ways to submit jobs to the scheduler to be run on the compute nodes.

This command has a large number of options, and some of the more commonly used options are discussed in detail on the page about submitting jobs. For a more complete description of the sbatch command and all of its options please see the sbatch manual page (or run the command man sbatch on the cluster).

scancel

The scancel command allows you to cancel jobs that are running or queued. Generally the only argument needed is one or more job numbers, or a range of job numbers.For more information on the acceptable options, see the scancel man page (or run "man scancel"). Job numbers can be provided in the following formats:

  • Single job number: e.g. "8148253"
  • Single job array element id: e.g. "8148250_4"
  • A list of job numbers and/or job array elements: e.g. "8148253 8148250_4"
  • A range of job array elements: e.g. "8148250_[4-7]"
  • A range of job ids: e.g. "8148250-8148257"

For queued jobs, scancel will simply remove the job from the queue. For running jobs, the processes in the job will be terminated, and then the job will be removed from the queue.

scontrol

The scontrol command is a general purpose tool for viewing and/or modifying the configuration and/or state of the Slurm scheduler. As a regular user, you will not be able to modify anything using this command. However, there are some useful things you can do with it:

Job info

You can get detailed information about running or pending jobs using the command

Example

login-2:~> scontrol show job 20412556
JobId=20412556 JobName=submit.scr
UserId=lincoln(452711) GroupId=zt-presidents(11564) MCS_label=N/A
Priority=0 Nice=0 Account=presidents-aac QOS=normal
JobState=PENDING Reason=JobHeldUser Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=00:15:00 TimeMin=N/A
SubmitTime=2025-02-12T09:46:28 EligibleTime=Unknown
AccrueTime=Unknown
StartTime=Unknown EndTime=Unknown Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-03-06T12:50:21 Scheduler=Main
Partition=standard AllocNode:Sid=login-2-internal.zaratan.umd.edu:378885
ReqNodeList=(null) ExcNodeList=(null)
NodeList=
NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
ReqTRES=cpu=1,mem=4000M,node=1,billing=1
AllocTRES=(null)
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0
Features=rhel8&amd DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/scratch/zt1/project/presidents/user/lincoln/namd/scd/scd_sl/submit.scr
WorkDir=/scratch/zt1/project/presidents/user/lincoln/namd/scd/scd_sl
StdErr=/scratch/zt1/project/presidents/user/lincoln/namd/scd/scd_sl/slurm-20412556.out
StdIn=/dev/null
StdOut=/scratch/zt1/project/presidents/user/lincoln/namd/scd/scd_sl/slurm-20412556.out

Updating jobs

You might also be able to use the "scontrol update job" command to update some of the cofiguration specifics of a job. Generally, this will only work on jobs which are still queued (i.e. have not started to run) ---
Once a job has started to run you will not be able to modify the job configuration.

Generally, we recommend that you simply cancel the job and resubmit it; there are a number of scripts which run after a sbatch command which ensure certain groups of settings are consistent, and these scripts do not run when you use the scontrol command to adjust things. Since your job has not started yet, it is generally safer to just cancel the erroneous job and resubmit a new job as this will ensure all dependent settings are set correctly.

If you insist on updating the settings, you can use a command like the following

Example

login-1:~> scontrol update job=20412556 TimeLimit=03:00:00

to increase the time limit on job 20412556 from 15 minutes (from the example in the previous section) to 3 hours.

Holding and/or releasing jobs

Sometimes it might be useful to delay the execution of a job; e.g. you have an important job that needs some limited resources (like GPUs but a less important job that you submitted earlier job would use the same resources, and you want to make sure the more important job runs first.

When there is a hold on the job, the job remains in the queue but it will started until the hold is removed. I.e., the job will just be skipped by the scheduler. To put a hold on the job, do something like

Example

login-2:~> scontrol hold *JOBNUMBER*

Although you can put a hold on a running job, it generally does not do anything --- the job will not be terminated, and the hold will only have effect if the job gets queued again.

When you are ready to remove the hold, you can do something like

Example

login-2:~> scontrol release *JOBNUMBER*

There are various advanced options you can pass to sbatch to control the timing of jobs relative to each other, e.g. making sure one job finishes before the current job starts. Look at the sbatch page (or run "man sbatch").

Node info

The scontrol command can also display a lot of details about a node or nodes. The format is better suited for display information about one node at a time. For example, to show details about node gpu-b11-5 one could use the command:

Example

login-1:~> scontrol show node gpu-b11-5
NodeName=gpu-b11-5 Arch=x86_64 CoresPerSocket=64
   CPUAlloc=4 CPUEfctv=128 CPUTot=128 CPULoad=1.43
   AvailableFeatures=rhel8,amd,epyc_7763,ib,nvme,beeond
   ActiveFeatures=rhel8,amd,epyc_7763,ib,nvme,beeond
   Gres=gpu:a100:4(S:0-1)
   NodeAddr=gpu-b11-5 NodeHostName=gpu-b11-5 Version=24.05.4
   OS=Linux 4.18.0-553.27.1.el8_10.x86_64 #1 SMP Fri Oct 18 06:18:15 EDT 2024
   RealMemory=512000 AllocMem=512000 FreeMem=430728 Sockets=2 Boards=1
   State=MIXED ThreadsPerCore=1 TmpDisk=12000000 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=gpu
   BootTime=2024-12-30T21:34:20 SlurmdStartTime=2024-12-30T21:35:46
   LastBusyTime=2025-05-07T15:53:07 ResumeAfterTime=None
   CfgTRES=cpu=128,mem=500G,billing=192,gres/gpu=4,gres/gpu:a100=4
   AllocTRES=cpu=4,mem=500G,gres/gpu=1,gres/gpu:a100=1
   CurrentWatts=650 AveWatts=525

Node name expansion

When Slurm commands present node names, they usually do so using a compact notation, in which square brackets (i.e. [ and ]) are used to delimit lists and/or ranges of numeric values. This compact form saves space in the environment and in displays, but is often not the most useful in scripts, where a fully expanded list might be more convenient.

To convert between the two formats, there are two useful subcommands of the scontrol command, show hostnames and show hostlist. The former will expand from the compact notation to a list of hostnames, e.g.

Example

login-2:~: scontrol show hostnames 'compute-b24-[1-3,5-9],compute-b25-[1,4,8]'
compute-b24-1
compute-b24-2
compute-b24-3
compute-b24-5
compute-b24-6
compute-b24-7
compute-b24-8
compute-b24-9
compute-b25-1
compute-b25-4
compute-b25-8
login-2:~:

The show hostlist form will do the reverse, e.g:

Example

login-2:~: scontrol show hostlist 'compute-b24-1,compute-b24-2,compute-b24-3,compute-b24-5,compute-b24-6,compute-b24-7,compute-b24-8,compute-b24-9,compute-b25-1,compute-b25-4,compute-b25-8'
compute-b24-[1-3,5-9],compute-b25-[1,4,8]
login-2:~:

scratch_quota

The scratch_quota command will show the quota and usage for the high-performance or scratch tier of storage on the cluster. Without any arguments, it will display the usage and quotas on the scratch filesystem for all groups to which you belong. This should include a group named "zt-PROJECTNAME" for each project to which you belong. Currently, the list might also include other groups which do not correspond to projects; these will be shown with no usage or quota. After that, it will show the total usage (across all projects/groups to which you belong) for you. Most people only belong to a single project, in which case the output would be similar to:

Example

login-1:~> scratch_quota            
# Group quotas
          Group name     Space used    Space quota   % quota used
            zt-hogwarts  532.462 GB       1.100 TB         48.41%
# User quotas
           User name     Space used    Space quota   % quota used  % of GrpTotal 
             hermione    316.717 GB      unlimited              0         59.48%

In the example above, the user belongs to a single project, with a corresponding Unix group zt-hogwarts, which has a scratch quota of 1.1 TB. The members of the project have a combined usage of 532 GB, which is 48% of the quota. In the User quotas section, we see the username of the user who ran the command (hermione), and that she is using 317 GB, which ic 59% of the total usage of the project (the 532 GB above). The unlimited under the Space quota column for that user means that that there is not a user-specific quota being applied; this user is still subject to the group quota (1.1 TB in this case). The % quota used column displays the usage as a percentage of the quota; for the user line, since there is no user level quota, this value is zero. If an user level quota was imposed, it would show how much of that quota has been used.

You can use the --group flag to specify a group to display usage and quotas for. If you specify one or more groups, it will display the usage and quotas for each of those groups.

You can also use the --user flag. If no groups are explicitly provided, the script will list usages and quotas for all groups the named users belong to (if neither users nor groups are explicitly given, the code will act as if your username was explicitly given).

If any users were given (explicitly, or if your username was added by default), after the usages and quotas for all groups are listed, the script will then list the total scratch usage by each of the specified users. If the flag --all-users is provided, this will be done for all members of the groups being displayed.

Note

When looking at usages for users, the numbers shown by this command are always the total usage for that user across all projects to which they belong (or once belonged). It is not their usage for the listed project.


Unfortunately the underlying filesystem quota system only calculates usage by group or separately by user, but not by the ordered pair group and user. So there is no ready way to determine usage by an user inside a single project except to traverse the directory tree and sum file sizes, which can potentially take a significant amount of time.

Tip

When using the du command to view usage of the scratch file system, you should always use the --apparent-size --si flags. This most closely matches the algorithm used by the quota system and the scratch_quota command for determining the usage.
See this FAQ entry for more information.

There is also an option --show_files; thus far we have only discussed the quota in terms of the amount of disk space, there also is a quota in terms of the number of files allowed. The scratch filesystem is optimized to handle large files; very large numbers of small files can be problematic to performance. To catch such problems before they stall the system, we limit the number of files per TB of disk storage. We are still working out what the thresholds at which things become problematic, and we will automatically increase the file count limits for groups as they approach our more conservative limits, but we may contact you if there seems to be a problematic trend.
Anyway, this flag will display the file count limits and usage.

Other options are documented in the help option of the script; use --help to see them.

seff

The seff command provides some information about the "efficiency" of a job, i.e. how well the job made use of the CPU cores and memory requested by the job.

Tip

If you are submitting jobs using more than the standard 4 GiB per CPU core, we recommend that you check the memory utilization of your jobs. Although you want to request slightly more memory than your job will be using (to avoid the job crashing due to running out of memory), you should not be requesting significantly more memory than the job will be using, as that is inconsiderate to other users and wasteful of your allocation.

The AAC will also wish to see such information when you apply for additional resources.

As an example

Example

login-1:~$ seff 10603094
Job ID: 10603094
Cluster: zaratan
User/Group: payerle/glue-staff
State: CANCELLED (exit code 0)
Cores: 1
CPU Utilized: 00:51:13
CPU Efficiency: 99.35% of 00:51:33 core-walltime
Job Wall-clock time: 00:51:33
Memory Utilized: 432.00 KB
Memory Efficiency: 0.01% of 3.91 GB
login-1:~$ 
login-1:~$ seff 11105784
Job ID: 11105784
Cluster: zaratan
User/Group: payerle/glue-staff
State: CANCELLED (exit code 0)
Cores: 1
CPU Utilized: 00:00:03
CPU Efficiency: 0.47% of 00:10:45 core-walltime
Job Wall-clock time: 00:10:45
Memory Utilized: 3.88 GB
Memory Efficiency: 99.26% of 3.91 GB

oThe two jobs above are simple test jobs. As seen from the above, job 10603094 was rather efficient from a CPU viewpoint (over 99% efficiency) but extremely poor (less than 1%) from a memory standpoint. Job 11105784 is the opposite; quite efficient from a memory viewpoint (again over 99%) but extremently inefficient form a CPU perspective (again less than 1%).

For production jobs you will want to reasonable values for both metrics.

setfacl

The setfacl command will modify the POSIX ACLs associated with a file or directory. There are several ways it can be used.

The most basic usage, to add an access control for a new user or group, or modify the permissions in an existing access control, you should use the "-m" or "--modify" flag, followed by an ACL and the path to a file. For example,

Example

login-2:~$ ls -l /scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt
-rwx------.  1 hermione zt-gryffindor 4652867 Oct 15  2024 polyjuice-recipe.txt
login-2:~$ getfacl /scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt
getfacl: Removing leading '/' from absolute path names
# file: scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt
# owner: hermione
# group: zt-gryffindor
 user::rwx
 group::---
 mask::---
 other::---
login-2:~$ setfacl -m u:hpotter:r /scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt
login-2:~$ setfacl -m u:hpotter:rx /scratch/zt1/project/gryffindor/user/hermione/potions
login-2:~$ setfacl -m u:hpotter:rx /scratch/zt1/project/gryffindor/user/hermione
login-2:~$
login-2:~$ getfacl /scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt
getfacl: Removing leading '/' from absolute path names
# file: scratch/zt1/project/gryffindor/user/hermione/potions/polyjuice-recipe.txt
# owner: hermione
# group: zt-gryffindor
 user::rwx
 user:hpotter:r--
 group::---
 mask::r--
 other::---

In the above example, we start by doing an "ls" and getfacl on the file to see its initial state. As indicated by both, the file starts as owned by user "hermione" and group "zt-gryffindor", with permissions "rwx------".

We then grant user "hpotter" read permission to the file, and rx permission to the containing directory and its parent. In order to access the file, an user needs "rx" permission on all of the directories in the path.
Assuming user "hpotter" is already a member of the zt-gryffindor group, he should have access to the "/scratch/zt1/project/gryffindor/user" directory, so he just needs access to the "hermione" and "hermione/potions" directories underneath that.

We then repeat the getfacl command, and this time note that there is an entry granting user "hpotter" read access to the file.

You can repeat the setfacl command for hpotter on the file to change the permissions that user hpotter has, just replace the access control for him. To delete an access control from the list, use the "-x" or "--remove" flag followed by the ACL to remove.

You can add the "-d" or "--default" flag to cause the "-m" or "-x" flag to work on default rather than access ACLs.

See also:

shell_quota

The shell_quota command will show the quota and usage for the medium term SHELL tier of storage on the cluster.
Without any arguments, it will display the usage and quotas on the SHELL filesystem for all projects to which you belong.

You can use the --project flag to specify a project to display usage and quotas for. Projects should be specified by the name of their root directory under /afs/shell.umd.edu/project.

Remember that SHELL storage is volume based and individual volumes are subject to per volume caps. You can use the flag --show_volumes to see a list of all volumes belonging to the project and their respective usages and volume caps.

Note

The information returned by this command is based on data from a cron job that runs every six hours or so. Therefore, the numbers are from a snapshot in time that might be as much as 6 hours old, it might take six hours or so for any changes (file additions or deletions) to be reported.

Tip

When using the du command to view usage of the SHELL file system, you should always use the --apparent-size --si flags. This most closely matches the algorithm used by the quota system and the shell_quota command for determining the usage.
See this FAQ entry for more information.

Other options are documented in the help option of the script; use --help to see them.

sinfo

The sinfo command displays information about how Slurm sees nodes and partitions.

When used without any arguments it will display information for all partitions, grouped by the state of the nodes. You can use the "-p" option to specify a specific partition, e.g.

Example

login-1:~$ sinfo -p gpu
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpu          up 7-00:00:00     28    mix gpu-a6-[2-9],gpu-b9-[1-7],gpu-b10-[1-7],gpu-b11-[1-6]
gpu          up 7-00:00:00      1   idle gpu-a5-1

The sinfo command also has a "-N" option which will display information in a format which is more node oriented. To see a list of nodes in the standard partition along with the features and GRESes on the node, you can use something like:

Example

login-1> sinfo -N -p standard --format="%N %5T %.4c %.8z %.8m %.8d %50f %10G"
NODELIST STATE CPUS    S:C:T   MEMORY TMP_DISK AVAIL_FEATURES                                     GRES
compute-a5-3 mixed  128   2:64:1   512000   300000 rhel8,amd,epyc_7702,ib                             (null)
compute-a5-4 alloc  128   2:64:1   512000   300000 rhel8,amd,epyc_7702,ib                             (null)
compute-a5-5 alloc  128   2:64:1   512000   300000 rhel8,amd,epyc_7702,ib                             (null)
compute-a5-6 mixed  128   2:64:1   512000   300000 rhel8,amd,epyc_7702,ib                             (null)
compute-a5-7 mixed  128   2:64:1   512000   300000 rhel8,amd,epyc_7702,ib                             (null)
compute-a5-8 mixed  128   2:64:1   512000   300000 rhel8,amd,epyc_7702,ib                             (null)
compute-a5-9 mixed  128   2:64:1   512000   300000 rhel8,amd,epyc_7702,ib                             (null)
compute-a5-10 mixed  128   2:64:1   512000   300000 rhel8,amd,epyc_7702,ib                             (null)
compute-a5-11 mixed  128   2:64:1   512000   300000 rhel8,amd,epyc_7702,ib                             (null)
compute-a7-1 mixed  128   2:64:1   512000  1600000 rhel8,amd,epyc_7763,ib,beeond                      (null)
compute-a7-2 maint  128   2:64:1   512000  1600000 rhel8,amd,epyc_7763,ib,beeond                      (null)
...

sinteractive

The sinteractive command is a command line utility for starting an interactive job. It basically is a wrapper around the salloc and the srun commands which creates a job to launch a job with the requested resources which will open a shell prompt in the current window when the scheduler launches the job.

Tip

You might wish to instead use the Interactive Desktop Application of the OnDemand Portal. This is generally easier to use, especially for beginners.

The following options are recognized:

  • -a ACCT: The Slurm allocation account against which to charge the job.. If omitted, your default allocation will be used.
  • -c NUM: The number of CPU cores to request. Defaults to 1
  • -d: Submit the job to the debug partition. In this case, the -t option if given is ignored and the wall time limit is set to 15 minutes.
  • -D: Dry-run mode; just print out the command that would have been run, but do not actually run it.
  • -f FLIST: only use nodes with features given in FLIST.
  • -g GRESLIST: request the specified GRESes -G: submit the job to the gpu partition.
  • -h: Print this help text and exit
  • -J NAME: The name for the job (defaults in interactive)
  • -m MEM: specify the memory per CPU (in MiB. Cannot be used with -M
  • -M MEM: specify the memory per node (in MiB. Cannot be used with -m
  • -N NUM: specify the numbers of nodes (defaults to 1). You will only get a prompt on the first node assigned, you are responsible for launching processes on any additional nodes. We recommend just leaving the default of 1.
  • -s SHELL: the SHELL to used. Defaults to your current shell.
  • -S: submit the job to the scavenger partition.
  • -t NUM: set the wall time limit in minutes (defaults to 60 minutes, unless submitting to the debug partition.
  • -V: print the version number of sinteractive and exit
  • -x: reserve the node(s) in exclusive mode. Default is to run the job in non-exclusive/shared mode.

Example

login-2:$ sinteractive -d -c 1
salloc: Pending job allocation 11139395
salloc: job 11139395 queued and waiting for resources
salloc: job 11139395 has been allocated resources
salloc: Granted job allocation 11139395
salloc: Waiting for resource configuration
salloc: Nodes compute-b8-57 are ready for job
compute-b8-57:bin$ hostname
compute-b8-57.zaratan.umd.edu
compute-b8-57:bin$ echo $SLURM_JOB_ID
11139395
compute-b8-57:bin$ exit
logout
salloc: Relinquishing job allocation 11139395

slurm_hostnames_by_tasks

Although Slurm provides a good deal of information to jobs via environmental variables, the format of the provided information does not always in a format that is immediately useful. In particular, the variables SLURM_JOB_NODELIST and SLURM_TASKS_PER_NODE provide information on the nodes assigned to the various tasks in the job (see the page on Slurm environmental variables for more information), but in a compact form which is often not the most useful.

The slurm_hostnames_by_tasks is a script developed at UMD to translate some of the data about the hostnames of the nodes on which the job is running into various formats.

You can provide a -h or -M flag to see basic usage or a man page for the script, respectively.

Otherwise, you must use the -m flag to provide a mode, which controls the basic operation of the script. Recognized modes are:

  • hostnames: this just prints a list of the hostnames of the nodes allocated to this job. Basically the same as scontrol show hostnames. Each hostname will occur exactly once.
  • hosts_by_tasks: This also prints a list of the hostnames of the nodes allocated to this job, but the hostnames will be repeated for each task allocated to the corresponding node.
  • host_count: This will print the hostnames of the nodes allocated to the job, without repetition, with each hostname followed by the number of tasks assigned to the node.
  • counts: This basically just expands the SLURM_TASKS_BY_NODE variable, listing the numbers of tasks for each node.

You can further customize the delimiter to use between records (e.g. hostname lines) with the -r flag; by default each record is on a separate line. The -f flag allows you to customize the field separator (e.g. the character between the hostname and the task count in the host_count mode), by default it is a colon (:).

For example, if your job has seveb tasks, divided among three nodes compute-b7-50, compute-b8-1, and compute-b8-2, with three tasks allocated to the first node, two to the other two nodes. Then we would have:

Example

compute-b7-50:~$ echo $SLURM_JOB_NODELIST
compute-b7-50,compute-b8-[1-2]
compute-b7-50:~$ echo $SLURM_TASKS_PER_NODE
3,2(x2)
compute-b7-50:~$ slurm_hostnames_by_tasks -m hostnames -r ", "
compute-b7-50, compute-b8-1, compute-b8-2
compute-b7-50:~$ slurm_hostnames_by_tasks -m hosts_by_tasks
compute-b7-50
compute-b7-50
compute-b7-50
compute-b8-1
compute-b8-1
compute-b8-2
compute-b8-2
compute-b7-50:~$ slurm_hostnames_by_tasks -m host_count
compute-b7-50:3
compute-b8-1:2
compute-b8-2:2
compute-b7-50:~$ slurm_hostnames_by_tasks -m counts
3
2
2

The script is available on the Zaratan cluster. A copy of the script is also available for download.

squeue

The squeue command can be used to report what jobs are in the queue and/or running. It takes a large number of options to filter what jobs are displayed and control which pieces of information is reported. We only cover some of the more basic ones here; for a more information on the command see the squeue man page or run "man squeue".

Useful squeue options:

  • "-A" or "--account": Only display jobs charging against the specified Slurm allocation account.
  • "-j" or "--jobs": Only display jobs matching the given job IDs.
  • "--me": Only display jobs belonging to you (the user running the command).
  • "-p" or "--partition": Only display jobs running in the specified partition.
  • "-t" or "--states": Only display jobs in the current state. See the subsection on job states for more information on job states.
  • "-u" or "--user": Only display jobs belonging to the specified user
  • "-w" or "--nodelist": Only display jobs running on the specified node.
  • "-O" or "--Format": This controls what fields are displayed for each job. See below for more information.

Queued and running jobs should always be visible with this command. Once a job finishes, it only remains visible to this command for a short period of time (around 30 minutes).

Job States

The following states are commonly seen. Please see the squeue man page or run "man squeue" for a complete list of job states:

  • R for running. The job has resources assigned and is running on those resources.
  • PD for pending. The job is in the queue and is awaiting resources. Normally it will run once resources are available, unless it is held or blocked from running for some reason. See the Reason output field.
  • CG for completing. The job is completing. Execution of the job has finished, and Slurm is in the process of cleaning up the job and releasing the resources.
  • CD for completed jobs.

Format options

You can use the "--Format" flag to specify what fields to list for each job display. It takes for an argument a comma delimited list of field specifiers, of the form "FIELDNAME" or "FILEDNAME:.SIZESUFFIX".

  • The dot after the colon (:) indicates the field should be right justified if it is given, in which case a SIZE is required.
    If omitted, the field will be left justified
  • SIZE is the minimum size for the field; it should be an integer. It defaults to 20. This field is required if the right justification dot (.) is given.
  • SUFFIX is an rbitrary string to append to the field

Useful field names are:

  • Account: the allocation account the job is charging against
  • ArrayTaskID: The task ID/index of a job in a
    job array.
  • EndTime: the (actual or anticipated) end time of the job
  • Feature: features/constraints specified by the job.
  • JobArrayID: the base Job ID for a job array. For non-array jobs, this will be the same as JobID.
  • JobID: the Job ID or number of the job.
  • NodeList: the list of nodes allocated to the job
  • NumCPUs: the number of CPU cores allocated to the job
  • NumNodes: the number of nodes allocated to the job
  • NumTasks: the number of tasks allocated to the job
  • Partition: thepartition for the job
  • Reason: <a name="squeue-reason"The reason the job is pending, etc.
  • ReasonList: Will display the Reason for a pending job, or the NodeList for a running job.
  • State: The job state of the job, in extended form, e.g. "Running" or "Pending"
  • StateCompact: The job state of the job, in compact form, e.g. "R" for running or "PD" for pending.
  • StartTime: The (actual or anticipated) start time of the job
  • TimeLeft: The amount of time left for the job (based on the time limit of the job less the amount of time it already ran)
  • TimeLimit: The time limit for the job.
  • TimeUsed: The amount of time the job has ued so far
  • TimeUsed: The amount of time the job has ued so far
  • tres-alloc: The TRESes allocated to the job (or requested if job is not running yet).
  • Username: The username of user owning the job

An useful Format specification is --Format="JobId:.8 ,Account:18 ,Partition:6 ,UserName:8 ,StateCompact:2 ,NumNodes:2 ,NumCPUs:3 ,TimeLimit:12 ,Feature: ,tres-alloc: ,ReasonList:"

The field sizes listed are good for most cases. For example,

Example

login-2:~$ squeue --Format="JobId:.8 ,Account:18 ,Partition:6 ,UserName:8 ,StateCompact:2 ,NumNodes:2 ,NumCPUs:3 ,TimeLimit:12 ,Feature: ,tres-alloc: ,ReasonList:" --me
   JOBID ACCOUNT            PARTIT USER     ST NO CPU TIME_LIMIT   FEATURES TRES_ALLOC NODELIST(REASON)
11310197 starfleet-aac      gpu    jtkirk   PD 1  128 6-00:00:00   rhel8&amd cpu=128,mem=500G,node=1,gres/gpu=4,gres/gpu:h100=4,billing=576 (AssocGrpBillingMinutes)
11310199 starfleet-paid     standa jtkirk   PD 4  512 8:00:00      rhel8&amd cpu=512,mem=2000G,node=4,billing=512 (Priority)
11310127 starfleet-paid     standa jtkirk   R  4  512 8:00:00      rhel8&amd cpu=512,mem=2000G,node=4,billing=512 compute-a8-[53-55,60]
11310131 starfleet-paid     standa jtkirk   R  4  512 8:00:00      rhel8&amd cpu=512,mem=2000G,node=4,billing=512 compute-b7-[32,40],compute-b8-[21,42]

The last field, "ReasonList" or "NODELIST(REASON)" displays the nodes the job is running on when the job is running, or the reason the job is pending if it is pending. Commonly seen reasons are:

  • Resources: This means the scheduler is unable to find sufficient idle resources to run the job at this time. This usually means that the job is one of the next in line to run, but cannot be started until resources become available.
  • Priority: This is similar to Resources, but generally means that there are other jobs ahead of it in the queue.
  • AssocGrpBillingMinutes: This means the job is unable to run due to insufficient SUs in the allocation account. The scheduler will not start a job unless there are sufficient SUs to complete this job and all currently running jobs charging against the allocation account. The SU costs for the job are based on the resources allocated or requested, and the walltime limits set for the jobs. If some running jobs terminate before their walltime limits, that might free up enough SUs for the pending job to start. Otherwise, the job will not start until additional SUs are added to the allocation account; e.g. if the account is replenished at the start of the quarter, or the AAC awarded additional resources.
  • QoSResourceLimit: This means the job is blocked because the user is currently using too many of certain resources, typically GPUs. To enourage fairer provisioning of some more valuable resources like GPUs, we restrict how many such resources a given user can be using at a given time. Once some of your running jobs finish and release the restricted resources your pending job will be eligible to run.

In the above example from user "jtkirk" showing two pending and two running jobs. The two running jobs are each using 512 cores over 4 nodes, as listed in the last field. This ReasonList field will display the nodes the job is running on if the job is running, or the reason it is in the pending state if it is pending. One pending job is requesting four H100 GPUs, and is pending due to lack of SUs in the allocation account it is charging against, and the other due to "Priority".

sreport

The sreport command is a command-line utility that allows one to view reports about usage, etc. The general usage is

sreport [FLAGS] REPORTTYPE REPORTSUBTYPE [OPTIONS]

where common values for REPORTYPE and REPORTSUBTYPE are:

  • "cluster AccountUtilizationByUser": This reports the usage for by user, groups by allocation account.
  • "cluster UserUtilizatonByAccount": This reports the usage by user and allocation account, list in order of heaviest to lightest usage.
  • "jobs SizesByAccount": This reports the usage broken down by ranges of the number CPU cores allocated to the job
  • "user TopUsage": This lists the top usage by user and allocation account.

Commonly used FLAGS are:

  • --tres TRESNAME: This will report on the usage of the specified TRESNAME. It defaults to cpu, you can use gres/gpu for GPU usage (or gres/gpu/a100 for specifically A100 usage), or billing for SU usage.
  • -y TPERIOD: This will set the time period to use when reporting usages. The default is Minute; generally Hour would be more usable.

Tip

By default, usages are reported in terms of minutes; e.g. CPU usage is listed in CPU-minutes (with 60 CPU-minutes essentially being 1 SU).

Use the FLAG -t Hour to convert to resource-hours.
Use the FLAG --tres billing to get SUs (the default TRES is cpu, which will only report CPU usage, not what is actually billed as SUs).

The following OPTIONS can be used to control what jobs are reported:

  • Accounts=ACCOUNTS: Only consider the named allocation accounts
  • End=ENDTIME: Only consider jobs which ended before ENDTIME
  • Start=STARTTIME: Only consider jobs which started after STARTTIME
  • Users=USERS: Only consider jobs belinging to the named users.

More detail can be found in the sreport man page or running "man sreport".

For example, to see the usage of A100 GPUs by users of the hogwarts-aac allocation in March of 2024, one could give the command

Example

    login-1:  sreport --tres=gres/gpu:a100 -t Hour cluster UserUtilizationByAccount Start=2025-03-01 End=2025-03-31 Account=hogwarts-aac
    --------------------------------------------------------------------------------
    Cluster/User/Account Utilization 2025-03-01T00:00:00 - 2025-03-30T23:59:59 (2588400 secs)
    Usage reported in TRES Hours
    --------------------------------------------------------------------------------
      Cluster     Login     Proper Name         Account      TRES Name     Used 
    --------- --------- --------------- --------------- -------------- -------- 
      zaratan  hpotter  Harry Potter        hogwarts-aac  gres/gpu:a100     4328 
      zaratan  hermione Hermione Granger    hogwarts-aac  gres/gpu:a100        0 
      zaratan  voldemort Tom Riddle         hogwarts-aac  gres/gpu:a100      269 
      zaratan  weasleyr Ron Weasley         hogwarts-aac  gres/gpu:a100        0

All usages in the above example are GPU-hours

srun

The srun command can be used to run a command on resources which the Slurm scheduler allocated to you. It is used "under the hood" by many implementations of mpirun, and you can also use it within your job script to launch processes on the nodes allocated to the job.

sview

The sview command is an X11 based GUI application which shows the graphical overview of the nodes in the cluster and their state, as well as the job queue.

Warning

Please set the Refresh Interval to something like 300 seconds (5 minutes). Go to the Options dropdown in the top menu, and then the Set Refresh Interval to adjust the interval.

The default value is far too frequent and causes performance issues for node it is running on and the scheduler in general.