HPC Cluster Usage Policies

The HPC clusters are governed by the University of Maryland Policy on Acceptable Use of Information Technology Resources (AUP).

In addition to the university-wide AUP, all users are required to adhere to the following policies and guidelines specific to the the HPC clusters.

Acceptable Use

The HPC cluster is a shared/collaborative environment and a research tool used by the university community. Your cooperation is needed to keep the computing environment useful to everyone.

The HPC clusters use University credentials (Directory ID and Passphrase) for authentication, hence, all university account and passphrase policies apply.
Do not share your login credentials with anyone else. (If your collaborator/research scientist/graduate student/affiliate needs access, please see Getting Access.)
Do not interfere with others' use of the cluster. (See "Prohibited Conduct" in the AUP.)
Do not attempt to gain higher privileges or otherwise "hack" the systems. (Example: Do not run "sudo" commands.)
You are required to promptly comply with all requests from the HPC systems staff. These requests may include, but not limited to, cleaning up your disk usage, cancelling jobs that adversely affect the cluster and other users, and modifying software usage behavior/patterns.
You are required to use and monitor your username@umd.edu or username@terpmail.umd.edu (university provided) email addresses. Do use your UMD email address when contacting the HPC staff.
Do not unsubscribe from the (very low traffic) HPCC-ANNOUNCE mailing list. We use this list to inform users about important issues, upgrades, planned maintenance, etc.

Allocations and Accounts

Accessing the cluster requires a linux account and the user to be a part of an active allocation. If you are a new user, see the Getting Started pages.

Allocations are granted (either by AAC or the College/Department) to faculty members only. Others must be added to a faculty allocation.
If you are the PI or point-of-contact for an allocation, you must remove users who no longer belong to your group/lab from your allocation.
Non-UMD personnel may get access to the HPC cluster provided they are eligible for an affiliate status with the university. The eligibility is determined by the campus UHR.
Your account and its data are treated as belonging to the allocation and the PI. Do not store private data in your account. When you leave the group/lab/university, your data will be made available to the PI (or allocation owner). See the relevant sections in the scratch space or the SHELL storage documentation for procedures and details.
The AAC provided allocations are granted for one year only. They are not automatically renewed.
When your association with the university ceases, your HPC account will be removed from all allocations, and your access to the cluster will be terminated. Copy any data you need before your last day at the university.

The login nodes are provided for users to access the HPC cluster. These nodes are intended for: setting up and submitting jobs, transferring data, compiling code, installing software, editing/manipulating files, etc.

Do not use login nodes to run compute-intensive tasks. (The HPC systems staff will terminate any compute jobs running on the login nodes.) Compiling codes is generally allowed on the login nodes.
We enforce a limit on the number of processes a user can start on the login nodes.
If your software compilation uses parallel tasks, please limit the number to maximum of 8; or compile your code on an CLI interactive session or a GUI interactive desktop.
If you use VScode to connect to the login nodes, you must follow the instructions provided to modify the VScode configs to not leave behind stale processes.

Dedicated, Interactive, and Batch use

The HPC cluster is a shared resource. The primary method to run long tasks is to submit them to the scheduler via sbatch.

The cluster nodes are shared by default. This means your job may run alongside other jobs on the same node; these other jobs might belong to you or to other users. The scheduler will ensure that there are sufficcient resources (CPU, memory, node local disk space, etc) on the node to support all the resources that all of the jobs running on the node have requested.
Do not use the --exclusive option unless absolutely necessary. If you do use this option, your allocation account will be charged for the entire node (128 CPU cores, 512GB memory, and all GPUs if present).
Do not use interactive sessions for long jobs. Example use-cases for interactive sessions include: debugging code for a short time period, setting up code via IDE like Jupyter Notebook or Matlab Desktop for long batch jobs, compiling code in parallel.
Do not "reserve" nodes/CPUs/GPUs by running srun ... bash for long periods of time. Instead, submit your job to the cluster via sbatch. All cluster resources are in high-demand; running interactive bash sessions for long periods is an inefficient use of the in-demand cluster resources. This is especially important for GPU jobs as the GPUs are generally in very high demand.
By policy and design, the compute/GPU nodes do not have access to off-campus internet. You will have to use the login nodes to download any data you need for the jobs.

Job Efficiency

The HPC resources are finite and some resources such as GPUs are in high-demand. It is critical that users do not request resources that will not be used. Doing so hurts all users and increases the wait times across the cluster. It is also wasteful of valuable HPC resources, and could result in the AAC not approving renewal requests or requests for additional resources.

Do not request resources your job cannot use. For instance, do not request GPUs if your job only uses CPUs. Do not request multiple CPU cores if your job is single-threaded. Do not requests CPUs across multiple nodes if your code cannot avail itself of resources on different nodes.
Scale up your job incrementally. Monitor the job to ensure the additional resources are being used. Start with a small set of resources and monitor the job. Only increase the resources once you are confident that all resources you assign to the job will be efficiently used. When requesting additional resources from the AAC, they will wish to see a summary of tests you have conducted on the cluster showing performance vs. the amount of resources requested.
Ensure that interactive jobs are not left idle. If you run an interactive job, either via command-line or Open OnDemand, do not let it sit idle. Long jobs should be submitted to the scheduler to ensure the resources are immediately released when the batch jobs complete.

Data Storage: Home, Scratch, SHELL, and /tmp

The HPC cluster has multiple tiers of storage. The following are only the policies regarding storage on the cluster. Please read our KB article on HPC Storage for other details and important information.

The HPC cluster has not been certified to hold sensitive information. Do not store classified, controlled unclassified, or HIPAA data on the cluster. All data stored on the cluster must not exceed Moderate (Level 2) classification on the UMD Data Classification Table.
The "scratch" and "SHELL" storage tiers are not backed up by the system. You must periodically backup any important data to an off-cluster location (ex: your work laptop/desktop, lab systems, cloud).
The data in your home, scratch, and SHELL filesystems are considered to be owned by the principal investigator(s) of the allocations to which you belong. Do not store personal/private data on the HPC cluster.
The scratch filesystem should not be used for medium or long term storage. Once your jobs conclude, you must remove the data from the scratch filesystem.
An automated file age-out system may be implemented for the scratch storage in the future. If and when that happens, you will be notified via the HPCC-ANNOUNCE email list.
The home directory has a 10G quota. You can go over that soft limit up to the 20G hard limit for a 7-day grace period. These limits cannot be changed.
Do not start jobs from the home directory. Your jobs should not create heavy I/O against the home directory. Use the high-performance scratch filesystem instead.
The /tmp filesystem provides fast I/O, and it is local to each node. The files you store on /tmp will be automatically deleted at the conclusion of your jobs on that particular node.
The SHELL disk space is not available on compute nodes.
The only storage spaces backed up by DIT are the home directories.
The PI will be informed if their project's storage usage exceeds the allocated quota. If the disk usage isn't brought under the quota limits within a resonable time, the PI will be billed for the excess usage.

Software

There are free and open source software packages as well as commercial vendor provided software packages installed on the cluster.

You are responsible for adhering to the license terms & conditions (EULA) of the software packages you install and use.
The HPC systems staff will restrict who can access the restrictively licensed software.
Do not attempt to circumvent license restrictions or to use pirated copies of commercial software.
If you install software that you did not write, you must first read, and adhere to, the license terms and conditions prior to installing/using the software.
Open Source does not necessarily mean a software package has been through a code review process to verify that it is suitable for research purposes or that it does not contain malicious code. You must take care to review and understand the software you use.

Classroom Use

Official UMD classes are eligible to receive an allocation from the AAC. See Allocations page.

Class allocations are provided per class, for the specific semester. Typically we create a single allocation for all sections of the class (including any cross-listings), although we will consider creating distinct allocations by faculty member in some cases.
Class students must use their University credentials.
The data in the allocation space may be deleted at the end of the semester (except for multi-semester courses for which other arrangements have been made with the HPC team before the end of the semester). Please move (and advise your students to move) and data which should be retained off the cluster before the end of the semester.
Class allocations should only be used for work related to the course in question. They are intended to be used for short homework assignments and/or small projects. HPC resources for larger projects should generally come from the faculty advisor's research allocation (see comments and links in the Independent Research item below).
Class allocations will not be awarded for Independent Research courses (typically numbered 398, 399, 498, 499), or Thesis or Dissertation Research courses (typically 799 or 899 courses). If HPC resources are needed for such courses, the resources should come out of the faculty advisor's research allocation (see how to apply for a research allocation from the Allocations and Advisory Committee) (AAC).
Class allocations will typically be awarded 50 kSU (about 50,000 CPU-hours or 1000 GPU-hours on an A100), 100 GB of scratch storage, and 1 TB of SHELL storage. While the AAC will consider requests for additional resources beyond the aforementioned, it is not likely to award much more than double that amount, and even then only with good justification.
The cluster is actively used by researchers around the clock, and often is very heavily utilized. It is not uncommon for jobs to be queued for an hour or longer; GPUs are at times in very heavy demand and wait times for GPUs can often by multiple hours or longer. Please make sure your students are aware of such and plan accordingly; we will not prioritize class jobs over research jobs in the queues. Students are encouraged to make use of the fractional/virtual a100_1g.5gb GPUs when feasible, as these are often more available and often can often support the workloads for the simpler jobs frequently used with classes.