Skip to content

FF chevron

FAQ: Slurm Errors

This page provides answers to some frequently asked questions related to Slurm errors, etc. Please review this page before asking HPC staff for help.

What does sbatch: error: This does not look like a batch script mean?

The Slurm sbatch command requires that your job scripts start with a shebang line, that is a line beginning with #! followed by the path to the shell to be used to process the script. For example, if you have a script written in "bash" shell your script should start like:

Bash example

#!/bin/bash
#SBATCH -n 1
#SBATCH -t 30:00

. ~/.profile
WORKDIR=$SLURM_SUBMIT_DIR
...

A similar script in the tcsh shell would be

Tcsh example

#!/bin/tcsh
#SBATCH -n 1
#SBATCH -t 30:00

setenv WORKDIR $SLURM_SUBMIT_DIR

Slurm requires the shebang line, because it searches for that and uses that to determine which shell it should use to run your script. This error is telling you that Slurm could not find a valid shebang line.

Generally, you just need to add an appropriate shebang line for the shell you are using, typically something like #!/bin/bash.

Sbatch errors with Batch job submission failed: Job violates accounting/QOS policy (job sumbit limit, user\'s size and/or time limits). What does that mean?

Although this cryptic error message can mean a number of things, most often on the Zaratan cluster it means that the allocation account you are trying to charge the job against is out of funds.

Normally, if there are insufficient funds for the completion of your job and all currently running jobs charging against the same allocation account, sbatch will accept the job and place it in the queue, and it will simply refuse to run until sufficient SUs are available (either due to replenishment of the allocation, or due to currently running jobs using less SUs than anticipated by the scheduler). These jobs will remain queued with the reason "AssocGrpCPUMinsLimit", "AssociationJobLimit" or "AssocGrpBillingMinutes"

However, if the anticipated cost (based on the walltime limit and CPU cores requested) of the job you are trying to submit exceeds the limit on the allocation account, the job will refuse to even be queued with the error Batch job submission failed: Job violates accounting/QOS policy (job sumbit limit, user\'s size and/or time limits)

If the allocation account you are charging the job against is not nearly empty (see the sbalance command), then please contact system staff. Please include the exact sbatch command you gave, and the path to the job submission script.

What do (AssocGrpCPUMinsLimit), (AssociationJobLimit) or (AssocGrpBillingMinutes) mean?

If you see one of

  • (AssocGrpCPUMinsLimit)
  • (AssocGrpBillingMinutes) or
  • (AssociationJobLimit)

in the NODELIST(REASON) field of the output of the squeue command for your job, that means that the job is pending because there are insufficient SUs in the allocation account that the job is being charged against.

(Actually, there are in general a number of different factors that could be causing this message in Slurm, but on the Zaratan cluster the only one which is usually relevant is the amount of SUs granted to the account you are charging against.)

Note that for a job to start running, we require that the account has sufficient funds to complete it and all currently running jobs (the amount of funds required for completion is computed based on the walltime requested for the job(s). E.g., even if your account has 3 kSU remaining and your job would only consume 0.5 kSU, if you (or others in your group) have other jobs currently running that are anticipated to need more than 2.5 kSU to finish, your new job will not run and be left pending with this status. If the running jobs finish before expected, so that there is now enough funds for your job, it will be started when the scheduler nexts examines it, assuming there are sufficient idle resources on the cluster and no other jobs are in front of it.

My job failed with an error like slurmstepd: error: *** JOBNUMBERONNodeNameCANCELLED ATTimeDUE TO TIME LIMIT ***. What does that mean?

Every job has a time limit. You are strongly encouraged to explicitly specify a time limit in your job when you submit it as the default time limit is rather small (about 15 minutes). If your job runs past the amount of time that was given to it, it will be killed with an error message like the above.

The time limit for the job is needed for the scheduler to efficiently schedule jobs. Your job will spend less time in the queue if you give a good value for this --- you need to specify a time in which you are sure the job will complete within (because otherwise it will be killed), and you might want to add some modest padding to that time just to be safe. But you do not want to be excessive, either. If you expect your job will only run for an hour, specifying a walltime of 2 hours is not unreasonable (giving it some padding), but 10 hours is excessive and may delay the start of your job because it makes backfilling harder.

When the next job in the queue requires many resources, the scheduler will often "reserve" free resources for that job until enough resources are available to start the job. These "reserved" resources are not available to other jobs (e.g. your job) unless your job will complete before the scheduler expects the other resources needed to start the next job in the queue will be available --- all of these timing considerations are based on the requested walltimes of jobs. Providing reasonable walltimes to the scheduler improves this process, and generally will all your jobs to be scheduled more quickly.

What does it mean when sacct reports OUT_OF_MEMORY for my job?

Slurm monitors the memory usage of your jobs, and will cancel you job if it uses more memory than it was allocated (which basically will be what was requested). This is necessary because memory is a shared resource and if your job tries to use more memory than was allocated to it, this could negatively impact other jobs on the same node.

This error indicates that at some point, your job used more memory than was allocated to it and there was killed.

If you are getting this error, you can try increasing the amount of memory that you request. The standard Zaratan compute nodes all have 4 GiB of RAM per core, for 512 GiB per node, as these nodes have 128 cores. There are also a small number of nodes with 2048 GiB per node on the Zaratan cluster in the bigmem partition. See the section on specifying memory requirements of your job for more information.

Note

Requesting more the 4 GiB per core could result in additional SU charges.

You should try to do an estimate of how much memory your code will need to ensure that you are not hitting a memory leak which will consume however much memory you throw at the job.

Once you find a memory setting with which your job completes successfully, you should use the seff command or similar to determine how much memory the job actually consumed, and then seek to tighten the memory requirements for future jobs.

My job failed with an error slurmstepd error: DetectedNoom_kill even in StepID=STEPID. Some of the step tasks have been OOM killed.

This error message may appear in the Slurm error output if the job exceeded its allotted memory. See the OUT_OF_MEMORY FAQ entry for more information.

My job has the warning slurmstepd error: Exceeded step memory limit at some point. What does that mean?

This warning means that at some point, the processes for your job on one of the nodes it was running on exceeded the memory allotted to it.

Unlike the OUT_OF_MEMORY errors, however, this warning means that the system did not kill one of your processes. Linux will often store files that are being read from or written to in memory to improve performance; the use of memory for that counts against the total memory allotment for your job. However, if the job exceeds the memory allotted to it, the operating system will first try to recover memory from this and similar uses of memory; this generally does not impact the running of the job.

If you see a wanring like this, I would recommend checking that the results of your calculations look reasonable. If so, in all likelihood your job completed without issue. If things look suspicious, you might want to re-run the job requesting more memory.

What does (QOSResourceLimit) mean?

If you see (QOSResourceLimit) in the NODELIST(REASON) field of the squeue output for your job, that means that the job has hit against a limit imposed by HPC staff. This could be something like:

  • a limit on the maximum number of GPUs of a given type that a user can use at the same time.
  • a limit on the maximum number of running jobs allowed for a single user on a given partition

Some users on the cluster legitimately submit hundreds or thousands of jobs at the same time which can run for several days. If unchecked,
this could result in a few users "hogging" all of the resources on the cluster; this can be particulartly problematic for resources like GPUs which are often in high demand. To try to balance this, we have imposed limits on the number of jobs and the number of GPUs (by type) that can be simultaneously running from a single user. You can submit jobs over this limit, but they will remain in pending states (with a QOSResourceLimit in the NODELIST(REASON) field of the squeue command) until additional run slots are available (i.e. one of your currently running jobs completes).

The exact number of this limit is still subject to some tweaking as we try to find the best number to ensure the cluster can be used well by all of our diverse user base. It also changes (with out warning) as demand for resources fluctuates.

My openmpi job complains about CUDA libraries. What does this mean?

Many OpenMPI jobs may see a warning like the following near the start of their Slurm output file:

Example

--------------------------------------------------------------------------
The library attempted to open the following supporting CUDA libraries,
but each of them failed.  CUDA-aware support is disabled.
libcuda.so.1: cannot open shared object file: No such file or directory
/usr/lib64/libcuda.so.1: cannot open shared object file: No such file or directory
If you are not interested in CUDA-aware support, then run with
--mca mpi_cuda_support 0 to suppress this message.  If you are interested
in CUDA-aware support, then try setting LD_LIBRARY_PATH to the location
of libcuda.so.1 to get passed this issue.
--------------------------------------------------------------------------

These are generally harmless warnings. The first message is stating that one of the MPI tasks was not able to open certain CUDA related libraries. As CUDA is an API used for interfacing with GPUs, those libraries are only present on nodes with GPUs. If your job did not request to use GPUs, it likely will have started on nodes without GPUs, and without these libraries, thereby causing Slurm to print this warning. Since you did not request GPUs (and so presumably were not intending to use GPUs), this is a harmless warning.

As stated in the warning message, if you are not planning on using GPUs and you wish to suppress the error message, you can add the flags --mca mpi_cuda_support 0 to your mpirun or equivalent command to turn this warning off. You might also wish to see the question regarding messages about error/warning aggregation.

If you were planning to use GPUs and you get this warning, then there is a problem. Most likely you did not properly request GPUs for your job. If that is not the case, contact systems staff.

Why is my job complaining about not finding the 'module' command ?

This typically happens when you are running a job in a shell which is different than your default login shell. E.g., your default shell is 'tcsh' you are running a 'bash' job script.

In scenarios of the type described above, your initialization dot files do not get invoked before Slurm starts the job script on the first compute node assigned to your job. Therefore, certain initializations which you tend to expect (e.g. the definition of the "module" command) do not occur, resulting in this error.

To resolve this is relatively simple, just explicity invoke the appropriate initialization scripts at the start of your job script (right after the last "#SBATCH" line), e.g.

#!/bin/bash
#SBATCH -t 1:00
#SBATCH -n 8
#SBATCH --mem-per-cpu=4096
#... any other SBATCH directives you need/desire
#
. ~/.profile

#... the rest of your script

See also the discussion of the job script.