Skip to content

FF chevron

FAQ: Problems running jobs

This page lists a number of frequently-asked questions pertaining to running job and resource accounting and monitoring. Please review this page before asking HPC staff for help.

What is an SU? Or a kSU?"

Accounting on the HPC clusters is done in units of Service Units, typically abbreviated as SUs. A more complete definition of SU can be found in the glossary, but basically your allocation account will be charged a certain number of SUs for your job. This amount will be the product of the walltime of your job (in hours) and a hourly SU factor depending on the amount of resources requested by the job. At the most basic level this factor is one SU per CPU core requested, but will increase if your job is requesting more than the standard amount of memory per CPU core, and will increase substantially if requesting GPUs.

Note

The hourly SU rate for the job depends on the resources allocated for the job (which generally corresponds to the resources requested for the job), not what was actually used.
This is because the scheduler will reserve the allocated resources for the sole use of your job for the duration of your job; so if you request resources you are not using you will be wasting both the cluster resources and your allocation's SUs. E.g., if you have a single core CPU-only job requesting 8 GiB of RAM but the job only uses 3 GiB, your job will be charged twice as much as if it only requested 4 GiB.

Note

While the hourly SU rate depends on what was requested, the walltime factor is based on the actual runtime of the job (as opposed to the maximum walltime requested), so if you requested 6 hours for a job but it died after 15 minutes, the walltime factor for your job will only be 0.25 hour, not 6 hour.

Both job charges and the funds in allocations are measured in SUs. Because SUs are actually rather small and unwieldy for many purposes, and we often talk in terms of kSUs, with 1 kSU = 1000 SU.

See also:

Which allocations do I have access to?

The sbalance command with no arguments will list the balances for all allocations which you have access to on the cluster you ran the command on.

If you see any allocations which you do not believe you should have access to, please contact us. If you believe you should have access to allocations which are not listed, contact the point of contact/owner of the allocation (typically your faculty advisor) and have them request that you be granted access.

Which allocation should I charge my job against?

If you are asking this question, presumably you have more than one allocation to which you have access. If not, you do not have a choice, and the system will automatically charge against the single allocation you have access to and this question becomes moot.

If the research groups, etc. that you belong to have local policies on which allocation you should charge, those policies should take precedence over the advise in this FAQ. We are just providing some useful guidelines in the absence of any policies of the projects to which you belong.

If you belong to more than one project, then you should choose the project that best fits the work that the job you are submitting. I.e., if you belong to an allocation for Dr. Jones and one for Dr. Smith, work for Dr. Jones should be charged against one of the allocations under Dr. Jones's project and not one of Dr. Smith's allocation(s), and vica versa.

If the project you are working with has multiple allocations, and there is no project level policies on which allocation to use, we generally recommend that you use SUs in purchased allocations (e.g. with names like projectname-paid) first, followed by allocations from departmental or college pools (projectname-dept_or_college_suffix), and lastly from allocations from the AAC (projectname-aac). This is because paid and departmental/college allocations replenish quarterly, with unused SUs vanishing at the end of the quarter, whereas AAC allocations typically are granted for a year.
So the SUs in paid or departmental/college allocations typically will expire before those in the AAC allocation.

The above assumes that there are sufficient SUs in the all of the allocations to run your job; if not, use the first allocation (in the above ordering) found which has sufficient unused resources to run your job. The scheduler will not automatically select an allocation with sufficient resources; you have to track that.

See the knowledge base article on specifying the account to be charged for more information.

Why am I receiving an error re disk quota exceeded?"

To promote fair and efficient use of the disk space provided on the Zaratan cluster, we have quotas on all networked filesystems on the cluster. Due to the different underlying hardware and filesystems being used, the quotas and technical details behind the quota numbers vary depending on the system.

  • Home directory: Home directories have a strict quota of 10 GB per user. We do not increase this. Use the home_quota command to view your home directory quota and usage. More information can be found in the knowledge base article on home directories.
  • Scratch space: Scratch space is a high-performance file system intended for the storage of data pertaining to active jobs (jobs that are running, jobs that are in the queue, or jobs that are planned to be submitted shortly). Data should be deleted when no longer needed, and moved elsewhere (e.g. SHELL) if it should be retained but is no longer needed for active jobs. Use the scratch_quota command to see quota and usage. For more information, see the knowledge base article on scratch space NOTE: Scratch space is not backed up.
  • SHELL space: SHELL space is for the medium term storage of data. Use the shell_quota command to view quota and usage stats. More information can be found in the knowledge base article on the SHELL storage tier. NOTE: SHELL space is not backed up.

If you are receiving errors about exceeding storage quotas, please use the above *_quota commands and the reported file paths to determine which filesystem is the source of the issue. Then delete unneeded files and if needed move files and data elsewhere (e.g. off the cluster, to SHELL, etc) as appropriate. If additional scratch or SHELL space is really needed, you see about obtaining additional storage by:

  • applying to the campus Allocations and Advisory Committee (AAC): you will need to fill out an application. There are limits on the amount which can be awarded to any single faculty member, and the larger the request, the better the justification needs to be. There is no charge for resources granted by the AAC.
  • requesting resources from the manager of the departmental/college pool(s) for departments and/or colleges to which you belong. The unit owning the pool decides on such requests and sets the policies around such. Some units impose cost-recovery fees for resources from their pool.
  • purchasing resources from DIT.

Why do scratch_quota and du -h not agree?

If you are using the Unix du command to check on your scratch disk usage, you might notice that there can be significant discrepencies in the usages reported. This is due to a number of reasons, including (but not limited to) the fact that:

  • the scratch_quota command uses SI (base 10) units like kB, GB, TB whereas the du command usually defaults to powers of 2 based units (kiB, GiB, TiB, etc). The difference between 1 TB and 1 TiB is almost 10%.
  • The actually accounting of usage differs between the du command and the underlying BeeGFS filesystem, including how things like partially filled blocks are accounted for. The actual enforcement of quotas (and the usage reported by "scratch_quota") are based on the accounting used by the underlying filesystem, so the scratch_quota command returns values as used in the enforcement of quotas.

To get more accurate numbers reported by the du command, we recommend using the flags --si --apparent-size to your du command. The former causes du to output using SI units, the latter causes the accounting algorithm used by du to much more closely align to what is used by the underlying filesystem (still not exactly the same, but in much closer agreement).