FF chevron

Monitoring and Managing Your Jobs, etc.

This page discusses various ways by which you you can monitor and manage your jobs.

The Slurm scheduler generally maintains records about jobs in two distinct places, one primarily dealing with jobs which are queued or running, and one primarily useful for jobs which have already completed/terminated, and this page is broken down along those lines.

Managing running and/or queued jobs
Viewing information about completed/terminated jobs

Managing running and/or queued jobs

The Slurm scheduler maintains a list of jobs which are running or queued so that it can properly schedule jobs, etc. Once a job finishes, after about 30 minutes, the job is removed from this list (as it is no longer relevant to Slurm's scheduling process), but you can issue commands to query information about the job while it still is in this list.

Listing Running/Queued jobs
When will my queued job start?
Details about running/queued jobs?
Viewing output of jobs in progress
Logging into nodes your jobs are running on
Holding/releasing jobs
Cancelling jobs

What jobs are running/queued

The Slurm command to list what jobs are running and/or waiting in the queue is the squeue command. This command will generally only show running or pending jobs (completed or failed jobs only show up in the squeue output for about 30 minutes or so after they finish).

There are a lot of options to control output and filter which jobs are displayed. We recommend using the flag --Format="JobId:.8 ,Account:18 ,Partition:6 ,UserName:8 ,StateCompact:2 ,NumNodes:2 ,NumCPUs:3 ,TimeLimit:12 ,Feature: ,tres-alloc: ,ReasonList:"

as this displays an useful set of information. E.g.

Example

login-2:~$ squeue --Format="JobId:.8 ,Account:18 ,Partition:6 ,UserName:8 ,StateCompact:2 ,NumNodes:2 ,NumCPUs:3 ,TimeLimit:12 ,Feature: ,tres-alloc: ,ReasonList:" --me
   JOBID ACCOUNT            PARTIT USER     ST NO CPU TIME_LIMIT   FEATURES TRES_ALLOC NODELIST(REASON)
11310197 starfleet-aac      gpu    jtkirk   PD 1  128 6-00:00:00   rhel8&amd cpu=128,mem=500G,node=1,gres/gpu=4,gres/gpu:h100=4,billing=576 (AssocGrpBillingMinutes)
11310199 starfleet-paid     standa jtkirk   PD 4  512 8:00:00      rhel8&amd cpu=512,mem=2000G,node=4,billing=512 (Priority)
11310127 starfleet-paid     standa jtkirk   R  4  512 8:00:00      rhel8&amd cpu=512,mem=2000G,node=4,billing=512 compute-a8-[53-55,60]
11310131 starfleet-paid     standa jtkirk   R  4  512 8:00:00      rhel8&amd cpu=512,mem=2000G,node=4,billing=512 compute-b7-[32,40],compute-b8-[21,42]

When will my job start?

The scheduler tries to schedule all jobs as quickly as possible, subject to cluster policies, available hardware, etc. Although most jobs start within a hour or two, when the cluster is more heavily utilized there can be significantly longer delays, especially if your job is requesting many resources or resources that are in high demand (like GPUs).

The squeue command, with the appropriate arguments, can show you the scheduler's estimate of when a pending/idle job will start running. It is, of course, just the scheduler's best estimate, given current conditions, and the actual time a job starts might be earlier or later than that depending on factors such as the behavior of currently running jobs, the submission of new jobs, and hardware issues, etc.

To see the estimated start time, you should tell squeue to include the "StartTime" field in the output, e.g.

Example

login-2:~$ squeue --Format="JobId:.8 ,Account:18 ,Partition:6 ,UserName:8 ,StateCompact:2 ,TimeLimit:12 ,Feature: ,tres-alloc: ,StartTime:" --me
   JOBID ACCOUNT            PARTIT USER     ST TIME_LIMIT   FEATURES TRES_ALLOC NODELIST(REASON)
11310197 starfleet-aac      gpu    jtkirk   PD 6-00:00:00   rhel8&amd cpu=128,mem=500G,node=1,gres/gpu=4,gres/gpu:h100=4,billing=576 N/A
11310199 starfleet-paid     standa jtkirk   PD 8:00:00      rhel8&amd cpu=512,mem=2000G,node=4,billing=512 2025-05-29T18:40:21
11310127 starfleet-paid     standa jtkirk   R  8:00:00      rhel8&amd cpu=512,mem=2000G,node=4,billing=512 2025-05-23T20:03:01
11310131 starfleet-paid     standa jtkirk   R  8:00:00      rhel8&amd cpu=512,mem=2000G,node=4,billing=512  2025-05-23T18:21:11

The scheduler currently does not have an estimated start time for job 11310197. Job 11310199 is estimated to start on 2025-05-29T18:40:21. The StartTime for the two running jobs are the actual time the jobs started.

Detailed information

To get more detailed information about a job that is currently running or in the queue, you can use the the scontrol show job command. This command takes a job number as on argument, and provides a lot of information about provides much detail about your job, e.g.

Example

login-2:~$ scontrol show job 4861267
JobId=4861267 JobName=test1.sh
   UserId=tonystark(31676) GroupId=avengers(5675) MSC_label=N/A
   Priority=500026 Nice=0 Account=avengers-paid QOS=normal
   JobState=PENDING Reason=Priority Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A
   SubmitTime=2025-05-28T15:12:10 EligibleTime=2025-05-28T15:12:10
   AccrueTime=2025-05-28T15:12:10
   StartTime=Unknown EndTime=Unknown Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-05-28T15:41:39 Scheduler=Main
   Partition=standard AllocNode:Sid=login-3-internal.zaratan.umd.edu:1886930
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=
   NumNodes=1-1 NumCPUs=16 NumTasks=16 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=16,mem=62.50G,node=1,billing=16
   AllocTRES=(null)
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0
   Features=rhel8&amd DelayBoot=00:00:00
   OverSubscribe=YES Contiguous=0 Licenses=(null) Network=(null)
   Command=/scratch/zt1/project/avengers/tonystark/suit8/sim114.sh
   WorkDir=/scratch/zt1/project/avengers/tonystark/suit8
   StdErr=/scratch/zt1/project/avengers/tonystark/suit8/slurm-4861267.out
   StdIn=/dev/null
   StdOut=/scratch/zt1/project/avengers/tonystark/suit8/slurm-4861267.out
   TresPerTask=cpu=1

Output of running jobs

Slurm outputs the stdout and stderr streams for your job to the files you specified on the shared filesystem in more or less real time (there might be a few minutes delay for updates to become visible across nodes).

Logging into nodes

Although normally you are not allowed to directly ssh into compute nodes, you are allowed to login to any compute nodes allocated to one of your jobs while that job is running.

This can be useful for diagnosing problems with one of your jobs and/or debugging issues. Simple but common cases include just verifying that the correct number of processes are starting on all of the nodes allocated to the job, or that all GPUs allocated to the job are in use. More advanced diagnostics can be done, but such would be specific to the codes being used.

Modifying jobs

You cannot modify a job once it starts running. While the job is queued but not running, you can in theory use the scontrol update job command, but in general we recommend that you simply cancel the erroneous job and resubmit it with the correct parameters. Doing things that way ensures that all dependent setting are set correctly.

Holding/releasing jobs

Slurm outputs the stdout and stderr streams for your job to the files you specified on the shared filesystem in more or less real time (there might be a few minutes delay for updates to become visible across nodes).

Cancelling jobs

At times you will need to remove a job from the queue, or kill a running job. To do this you can use the scancel command; give it the job number of a list of job numbers to cancel.

Running jobs will be terminated and removed from the queue, and queued jobs will simply be removed from the queue.

Cancelling Your Jobs

Sometimes one needs to kill a job. To kill/cancel a job that is waiting in the queue, or is already running, use the scancel command:

Example

login-1> scancel -i 122488
Cancel job_id=122488 name=test1.sh partition=standard [y/n]? y
login-1>

Viewing information about completed/terminated jobs

Information on completed jobs

Slurm retains a database of information about completed jobs and job steps.
Queued and running jobs will also have entries in this database, but most of the information on jobs is only added once the job completes, so this database is generally not very useful for jobs that have not completed.

Listing jobs

To see jobs which have been run, you can use the sacct command. This command takes a large number of potential arguments, both to filter which jobs are displayed, and what information is shown for the displayed jobs.

Generally you will want to provide a time range for the jobs, and probably your username and or allocation account name to limit the jobs. E.g.

Example

login-1> sacct --format=JobID,State,ReqTRES,AllocTRES,Elapsed \\
-P -X -v -v -v --start 2024-10-01 --end 2024-10-31 -A foo-aac
JobID|State|ReqTRES|AllocTRES|Elapsed
8057991|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:25
8148243|COMPLETED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:04
8148253|CANCELLED by 123456|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:06
8148261|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:05
8148381|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:02:07 
8148416|COMPLETED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|02:13:37

Memory/compute efficiency

A common tasks for users is to ensure that their jobs are making efficient use of the HPC resources allocated to the job, particularly CPU and memory resources. The seff command provides an useful way to do so.

It is typically invoked by just providing a job number (or job array element id, and it will return information about the amount of actual compute time and memory used by the job, as well as comparing such to the requested (and allocated) resources.

Jobs which are requesting more than the standard 4 GiB per CPU core should be monitoring the memory usage in particular; you should be seeing at least about 80% utilization (and ideally at least 90% or even 95%) --- memory is a valuable resource and any memory allocated to your job is unavailable to other jobs running on the node while your job is running. While you want to make sure the requested memory is at least as much as your job will need (otherwise your job will die due to OOM (out of memory) errors), you do not wish to request excessive amounts of memory as that is wasting resources. See the corresponding section in the Running Jobs page for more information.

The seff command also shows computational efficiency. This currently only works well for CPU-only jobs; for GPU jobs, the CPUs will usually be fairly idle while GPU computation is going on, and so the CPU efficiency will show a low value. I/O and other issues can also lower CPU efficiency; although it would be ideal to adjust I/O patterns, etc. in your code to improve CPU efficiency, we realize that this is not always feasible. See the section on checking CPU efficiency for other ways to monitor that your jobs are utilizing the CPU effectively.

Example

login-2:~$ seff 11975422
Job ID:  11975422
Cluster: zaratan
User/Group: taft/zt-presidents
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 4
CPU Utilized: 05:52:00
CPU Efficiency: 12.61% of 1-22:30:40 core-walltime
Job Wall-clock time: 02:54:25
Memory Utilized: 15.07 GB (estimated maximum)
Memory Efficiency: 94.17% of 16.00 GB (16.00 GB/node)

The example job above made heavy use of GPUs, and so the reported CPU efficiency was low (as expected and explained above). However, the memory efficiency was quite good.

Note

The seff command uses the [Slurm](/kb/glossary.md#slurm) accounting
database, and the information about CPU and memory usage is not
updated until the job terminates.  So using the seff command on
a running job will not generally produce anything useful.

Detailed information about finished jobs

The Slurm accounting database retains a good amount of information about completed jobs and job steps. (Queued and running jobs will also have entries in this database, but most of the information is only added once the job completes.) The sacct command will allow you to view information from this database.

We discuss the various arguments and flags you can pass to the sacct command to filter jobs and control the fields displayed, here we discuss some of the things you might wish to do with this data.

Determine the SU costs of a job

You can use the sacct command To determine the SU costs of individual jobs. Run the sacct command with the appropriate flags to display the fields AllocTRES and Elapsed, e.g.

Example

login-1> sacct --format=JobID,State,AllocTRES,Elapsed \\
-P -X --start 2024-10-01 --end 2024-10-31 -A foo-aac
JobID|State|ReqTRES|AllocTRES|Elapsed
8057991|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:25
8148243|COMPLETED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|01:26:04
8148253|CANCELLED by 123456|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:06
8148261|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:03:05
8148381|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:02:07
8148416|COMPLETED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|02:13:37

The important values are the TRES labelled billing, and the Elapsed field. If you convert the time in the Elapsed field to hours and multiply by the value of the billing TRES, and this will yield the SU cost for this job. For the jobs in this example we have

Job Number	Billing	Elapsed	Elapsed Hours	SU for Job
8057991	96	00:00:25	0.007	0.67
8148243	96	01:26:04	1.434	42.70
8148253	96	00:00:06	0.002	0.19
8148261	96	00:03:05	0.051	4.90
8148381	96	00:02:07	0.035	3.38
8148416	96	02:13:37	2.227	23.78
Total	96	03:45:24	3.756	75.6

Job performance data

The sacct command can also provide information about the performance of a job. In particular, there are a slew of fields with names starting with TresUsage which are helpful in this regard. These basically report the usage for each TRES in various fashions:

There are two sets of values, TresUsageIn* and TresUsageOut*. For TRES values which are directional, the TresUsageIn* are usage going into the process (e.g. disk reads, incoming network traffic), and the TresUsageOut* values are for usage leaving the process (e.g. disk writes, outgoing network traffic). For non-directional TRESes (e.g. CPU usage, etc) there should be no difference.
There are sets of values labelled Ave, Max, and Min. The Ave values represent average over all of the tasks, while the Min and Max values represent the minimum and maximum values over all of the tasks.
The Min and Max sets have additional variables with Node and Task suffices which give the name of the node and task number, respectively, where the minimum or maximum value occurred.

Note

Many of the TRES related fields are specific to each job step, so you will need to view each step of a job and look at their individual TRES values.

For example, to find the memory usage of a job, you could look at "mem" TRES for the various steps in the job. E.g. looking at the TresUsageInMax or TresUsageOutMax values for the TRES "mem" and finding the maximum (or looking at the maximum MaxRSS value for each step) will let you know which task/node used the most memory for the job, and how much. Although for that specific use case, the seff command is generally easier to use. But comparing the minimum, average, and maximum usage values might reveal useful information; e.g. if there are large differences in usages for certain TRESes among the tasks, it might indicate that things are not being balanced well among the tasks.