Skip to content

FF chevron

Partitions on the UMD HPC clusters

The various UMD HPC cluster provide a number of partitions which serve to group nodes by function or to provide special capabilities.
These different partitions may also allow different maximum walltimes for jobs.

The exact partitions available, and the details of those partitions, depend on the cluster being used, namely:

Although the exact details of the partitions are cluster dependent, we try to keep the names and functionality somewhat consistent among clusters:

Standard partition

Every HPC cluster has a "default" partition which will generally be used if the job does not explicitly specify a partition.

On Zaratan, jobs that do not explicitly specify a partition will be defaulted to the gpu partition if one of more GPUs are requested, or the standard partition otherwise.

The standard partition on Zaratan sends jobs to the standard compute nodes of the cluster; these nodes comprise the bulk of the cluster.

GPU partition

The GPU partition consists of all of the GPU-enabled nodes on the cluster, so you should be using this partition if you plan to use GPUs.

The GPU-enabled nodes are placed in a separate partition in order to prevent the CPU resources on these nodes being consumed by non-GPU jobs, potentially preventing the running of GPU jobs on these limited number of nodes.

On the Zaratan cluster, if a job does not explicitly specify a partition, it will default to this GPU partition if the job requests at least one GPU, and the standard partition otherwise.

Bigmem partition

The UMD HPC clusters generally have a small number of compute nodes with very large amounts of RAM. These nodes are significantly more expensive to purchase than the standard partition nodes, but are needed for some jobs with very large memory requirements, especially for non-MPI jobs.

On the Zaratan cluster, the nodes in the bigmem partiton have 2 TiB of memory, compared to the significantly smaller (though still substantial) 512 GiB of memory on the standard partition nodes. Since the nodes on both partitions have 128 CPU cores per node, this amounts to an average of 16 GiB per core on bigmem nodes vs 4 GiB per core on standard nodes.

While we do not currently apply a surcharge to jobs requesting the bigmem nodes, due to the small number of such nodes (on Zaratan there are 6 bigmem nodes compared to about 360 standard nodes) jobs submitted to the bigmem partition will likely spend a significantly longer amount of time waiting in the queue for resources to become available. So we recommend that the bigmem partition only be used when you have jobs which require more memory than is available in the standard parition.

Debug partition

All of the UMD HPC clusters typically have a small number of compute nodes in a "debug" partition. This parition exists so the users can quickly submit and run small test jobs for diagnositic purposes or quick example jobs. To ensure a rapid throughput for jobs in this partition, jobs in the debug partition have a very limited runtime; usually about 15 minutes.

The Zaratan cluster has 4 nodes in its debug partition, with a 15 minute time limit. There are currently no GPUs available in this partition.

Scavenger partition

Jobs in the low priority scavenger partition run at the lowest possible priority, and are preemptible. I.e., even once the job has started (which due to the low priority can take a while), if another non-scavenger job comes along and needs the resources the scavenger job is using, the scavenger partition job will be terminated and put back in the queue.

Your allocation account does not get charged for jobs in the scavenger partition, but in order to make effective use of it your job needs to be able to checkpoint itself so that it can make progress in the slices of time it gets between other jobs.

Limits on the sizes of jobs

The UMD HPC clusters are designed to support research computing in a variety of ways, including both traditional high performance computing (HPC) as well as high throughput computing (HTC). These and other paradigms sometimes at conflict with each other; while the HPC team strives to accommodate as many users as possible, this is not always possible.

Specifically, we often restrict the amount of resouces that can be used by a single job and/or single user in order to improve the performance and fairness of the cluster. In particular:

  • we impose a limit on the maximum number of GPUs of a given type that can be used at the same time by a single user.
  • we limit the maximum number of CPU cores and/or nodes that can be used by a single job
  • users with large numbers of pending jobs requesting GPUs may have the priorities of such jobs reduced to allow the jobs of other users a chance to run.

These restrictions are imposed in order to improve the experience for all users. We have found that individual jobs requesting a significant fraction of the cluster's resources can have serious adverse impact on the scheduling of jobs for the entire cluster. Also, the GPUs are often in very high demand, and due to their limited supply we do not want a large number of jobs from a single user preventing anyone else's jobs requiring GPUs to run for days at a time. With these restrictions we are striving to improve throughput and increase fairness for all users of the cluster.

The exact limits are subject to change without notice, and we will adjust things as needed to improve the overall performance of the cluster. E.g., if we discover that there are idle GPUs of a given type, and the only pending jobs seeking to use those GPUs are blocked due to the per user limit, we will temporarily increase the limit on that GPU type as we do not wish to see GPUs sitting idle for no good reason.

If you have need to run a job requesting a large number of GPU or CPU resources, please contact HPC staff to request access to submit such jobs. We will relay the request to the Allocations and Advisory Committee; please include information about why the use of such large amounts of resources are required. Note that such a request is only for the ability to bypass the restrictions on submitting jobs requesting a significant fraction of the cluster's resources; you still must have enough SUs to "pay" for the job.