Skip to content

FF chevron

FAQ: Problems running jobs

This page lists a number of frequently-asked questions pertaining to running jobs and issues with running jobs. Please review this page before asking HPC staff for help.

My job is spending a lot of time in the queue. Why? When will it start?

HPC clusters have a large number of nodes, and many cores and other resources. However, they also have many users, some of whom submit very large jobs. DIT and the HPC team make no promises about the maximum amount of time a job can be queued. If the system is lightly loaded, most jobs will start within a few minutes of being submitted (there is some small overhead to the batch system, so it is not immediate). When the cluster is heavily used, wait times of hours to a significant fraction of a day are not to be unexpected. This is especially true for jobs requesting resources like GPUs which are often in high demand.

If your job is taking a while to run, first check its status with the squeue command. Pay attention to the NODELIST(REASON) or REASON field. If it shows (Resources) or (Priority), that means that the resources your job requires are already in use or reserved for higher priority jobs (e.g. jobs that we submitted before your job). In other words, these conditions indicate that the cluster is busy, and that your job will start when resources are available and it is your job's turn. You can use the squeue command to get an estimate of when your job will actually start.

If the NODELIST(REASON) field is showing something else, there might be something wrong which is preventing your job from starting.
Typical cases are

  • (AssociationJobLimit): this usually means your allocation account does not have enough SUs to run the job.
  • (QOSResourceLimit): this generally means that you are exceeding a per-user limit on the number of running jobs, or the number of GPUs of a specific type in use at the same time. These limits are imposed to try to allow for more equitable distribution of resources among users. In these cases, your pending jobs should run once some of your running jobs finish, bringing your usage below the threshold.
  • JobHeldUser: this indicates that a "hold" was put on your job, and the job will not run until that hold is "released." You can put holds on your jobs, and system staff may do so from time to time if your job is interfering with the smooth operation of the cluster (e.g. you have many jobs exceeding the SUs in your allocation, clogging the scheduler --- we will generally try to email you in such cases. You can use the scontrol release command to release such jobs; however, if system staff placed a hold on the job(s), please do not release the job until the situation which lead to the job(s) originally being held has changed.

How can I reduce the amount of time my job spends waiting in the queue?

For a full answer, please see the section on job scheduling for more infomation on the scheduling process and the factors affecting scheduling. But for the quick answer, to minimize the amount of time your jobs spend in the queue:

  • For short (under 15 minute) debugging jobs, use the debug partition.
  • Do not use the scavenger partition, as it has the lowest priority of all partitions.
  • Do not request an excessive amount of walltime, as that can prevent your job from taking advantage of backfill scheduling. Your job will be terminated, finished or not, when the walltime runs out, so a bit of padding is recommended, but be reasonable. If your job should finish within 4 hours, 5 hours is probably reasonable, but 8 is probably excessive.
  • While you should always request the resources you expect to need, and maybe with some padding since these are often estimates, do not be excessive. E.g., requesting 8 GiB/core of RAM when in reality only 4 GiB/core are needed can result in your job spending more time in the queue (as well cause more SUs to be charged against your allocation.

My batch job finishes as soon as it starts with no/little output

While these symptoms can have a number of causes, frequently the cause is quite simple --- the last line in your job script is missing a proper Unix end-of-line (EOL) character. This can often happen if you edit the script on Windows and then transfer to Unix, as the standard EOL characters differ on the two OSes.

When your script does not have a proper EOL terminating the last line, the shell ignores/does not execute the last line. As often the last line is the line wherein you issue the command which is going to do all the real work, this often means that your job starts, does some simple housekeeping commands (loading modules, change working directory, etc), then skips the last line (with the missing EOL), which would have done the real work, and exits (successfully) almost immediately afterwards.

While there are various ways to ensure the file ends with a proper EOL character, the easiest solution is to just ensure that your script ends with a couple of blank lines. That does not actually fix the EOL issue, but does mean that if the final EOL somehow is missing, the line that is ignored is blank, and so you do not care if it is ignored.

What does \"(AssocGrpCPUMinsLimit)\" or \"(AssociationJobLimit)\" mean?

See the answer to this question

What does \"(QOSResourceLimit)\" mean?

See the answer to this question

My openmpi job complains about CUDA libraries. What does this mean?

This is likely a harmless warning. See this question and answer for more details.

My OpenMPI job has a warning about 'N more processes have sent' some message. What does that mean?

OpenMPI job output will sometimes include output similar to this:

Example

[compute-b8-48.zaratan.umd.edu:38553] 20 more processes have sent help message help-mpi-common-cuda.txt / dlopen failed
[compute-b8-48.zaratan.umd.edu:38553] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

This message is basically stating that a number of other (20 in this case) MPI tasks have generated the same error. OpenMPI normally assumes you only want to see errors and warnings once per job, not once per MPI task. So if more than one MPI task produces basically the same error message, it prints the error message once, and then prints a message like the one above to let you know that the message occurred multiple times. This is usually what you want --- typically the message occurs on every MPI task in the job, and most people do not want to be inundated with the same error message hundreds or thousands of times.

This message usually can just be ignored, as it is just saying that the previous warning/error message occurred multiple times. Instead, you should focus on previous warning/error message, which might or might not be significant.

As stated in the warning, if you add the flag --mca orte_base_help_aggregate 0 to your mpirun command, OpenMPI will not aggregate messages and you will see the messages from each and every MPI task that generated them.