Skip to content

FF chevron

Allocations on the HPC cluster

This page discusses various aspects of allocations on the University of Maryland HPC clusters.

Available HPC Resources

The HPC Cluster has the following resources.

  • CPU/GPU time and Memory usage (measured in kSU)
  • High Performance (scratch) disk space (measured in TB)
  • Longer term storage (SHELL) disk space (measured in TB)

The Cluster is a time-share system. Our goal is to make use of its compute cycles 100% of the time. To achieve this goal, the HPC Allocations and Advisory Committee (AAC) slices up the above resources (time and disk space on the cluster) into smaller chunks of "Allocations" and grants them to the university faculty members as their research needs require.

Types of Allocations available

As a user of the cluster you belong to at least one project, and each project contains one or more allocations, and you will belong to at least one of those. Each allocation represents an allotment of resources on an HPC cluster.

The reason that there may be multiple allocations within a project is because there are different types of allocations, mostly distinguished by having different resource pools from which the corresponding resources came. These allocations can have different expiration dates and replenishment schedules. These are discussed in more detail below.

Allocations are generally named with the following format: "projectname-allocationtype". The projectname often (but not always) ends in "-prj" or "-lab". Common allocation types include:

  • class: These allocations are for specific classes/workshops/etc.. You should only charge jobs related to the corresponding course to that allocation.
    You will be removed from the allocation shortly after the class and/or workshop ends. These are one-time awards of resources that last until the end of the semester/end of the workshop/etc.
  • aac: These allocations are awarded by the Allocations and Advisory Committee (AAC), either small developmental or larger AAC allocations. These are one-time awards of resources that typically last for one year, although they can be renewed/additional resources requested, up to certain limits.
  • paid: The resources in these allocations were purchased from the Division of Information Technology (DIT), usually by your advisor. The compute time in these allocations is typically replenished at the start of every quarter (1 Jan, 1 Apr, 1 Jul, 1 Oct) over the lifetime of the purchase agreement.
  • other: These suffixes generally indicate that the allocation was awarded from a college or departmental pool, with the suffix indicating the unit from which it was granted. The compute time in these allocations is typically replenished at the start of every quarter (1 Jan, 1 Apr, 1 Jul, 1 Oct) until the college or departmental pool expires, or the college/departmental manager of the pool instructs HPC staff to change the allocation.

Allotments of compute time are distinct across allocations.
Each allocation with an compute allotment has its own Slurm allocation account and when submitting a job you can specify which allocation account the job should charge against. All jobs submitted to the cluster have an allocation account which the job charges against; if no such allocation account is explicitly specified (e.g. with the -A flag to sbatch), the user's default allocation will be used. The allocation account will be charged an hourly SU rate depending on the resources requested by the job times the actual runtime of the job in hours (this charge is typically incurred every few seconds while the job is running).

Tip

If you have a choice of allocations within the same project to charge against, we generally recommend that you charge against allocations whose SUs expire or replenish first. Typcially this will be a departmental or paid allocation. This is because any SUs which are unused when the allocation expires or is renewed simply vanish, they do not get rolled over to the next quarter.

If your research group has established other policies regarding which allocations inside the project should be used, those policies should take precedence. And of course, if there are insufficent SUs in the allocation which expires first, then you should charge against a different allocation.

The storage allotments for all of the different allocations within a project are typically summed, individually for each tier, to get the effective storage limits for the entire project (the group of members in the project) on that storage tier. So generally you do not have to assign specific files to specific allocations within a project (although if you belong to multiple projects, you should be storing the file underneath the project specific scratch or SHELL directory structures). Storage allotments are for the duration of the allocation; they do not increase automatically with time. Note that if an allocation expires, the effective limits on storage for the members of the project may be reduced, which could potentially lead to to the disk usage exceeding the limit. In this case, members will be notified of this issue, and given a week to resolve the situation. This can be achieved either by reducing the amount of disk storage used or by obtaining additional storage. To reduce usage, one can delete unneeded files, or move files off the cluster. To obtain additional storage, one can renew an expired allocation, request an increase in the storage allotment on a remaining allocation, or request (perhaps by purchasing) a new allocation with additional storage.

Allocations from the AAC

The Allocations and Advisory Committee (AAC) awards allocations to UMD faculty members (Assistant/Associate/Full Professors and the equivalent) at no cost to the faculty member. These allocations are awarded after the AAC reviews applications submitted by the faculty member (or a student, etc. on their behalf). The AAC is entrusted with ensuring that the valuable HPC resources are being used well, and so reviews the applications carefully; the more resources being requested, the more closely the application is examined. Because the AAC does not oversubscribe resources from the pool available to them, awarding resources to one researcher reduces the amount of resources that can be awarded to others.

  • The AAC grants only one allocation per faculty member (excluding class allocations) regardless of the the number of projects or students. So when applying for an allocation from the AAC, you should submit a single application covering all research projects, etc. that you intend to work on for the upcoming year.
  • The AAC grant limit is 550kSU/year.
  • The grant period is one year--365 days from the date of the grant (not a calendar year as in Jan-1 to Dec-31).
  • All applications must come from a faculty member.
    However, a student may fill out the form on behalf of their faculty advisor as long as the faculty member approves it.
  • Faculty members are eligible for at most one development allocation from the AAC per year. If additional resources are needed beyond what is available in a development allocation, you must apply for a larger allocation.
  • AAC allocations do not renew or replenish automatically. Allocations are only renewed and only receive additional resources as a result of applications that are appoved by the AAC.
  • AAC allocations are granted for one year. You can submit a "renewal" application if:
    • the application has expired
    • you need additional resources from the AAC
  • There is no charge to the faculty member for either a developmental allocation or an allocation renewal from the AAC.
  • If you need more than the AAC's yearly limit, you can look into departmental and paid allocations.
  • The AAC and DIT do not in general provide software licenses for research software packages. Some commercial products (e.g. MATLAB) are available to all users on the cluster due to site licenses. These licenses are purchased either by campus or by individual units who allow access to all users on the cluster. In general, to use any other restrictively licensed software package on the cluster you must provide a license. Always contact HPC staff before purchasing any license you intend to use on the cluster; not all licenses are compatible with the HPC cluster.

Warning

Do not purchase a license for a software package you intend to use on a DIT maintained HPC cluster without first consulting with HPC staff.

Not all software licenses are suitable for use on an HPC cluster, and we do not wish for you to purchase a license which you cannot use.

There are basically two "flavors" of allocations from the AAC:

  • Developmental allocations: This is a small allocation (up to 50 kSU/year, with 100 GB of scratch storage and 1 TB of SHELL storage) and is intended for new users to explore how (or even whether) their research can make use of HPC resources. For small projects, it might even be enough to complete the research project. It also allows access to the cluster so that one can collect data about how your jobs run on the cluster, which the AAC will wish to see in any renewal application for a larger allocation.
  • Larger allocations: This is a larger allocation (up to 550 kSU/year, up to 10 TB of scratch storage and up to 50 TB of SHELL storage). This type of allocation is often sufficient for medium sized projects. Due to the larger size, the AAC requires more detailed research plans in the application, including metrics showing that you are using the HPC resources efficiently; such metrics are typically collected in the previous developmental allocation.

Although it is convenient to discuss the two "flavors" of AAC allocations, the only difference is in the amount of resources awarded and the requirements in the application process. Either flavor of allocation will appear as a Slurm allocation account named something like PROJECT-aac, where PROJECT is the name of the project under which the allocation exists. I.e., you will not see separate developmental and renewal allocations, but a single AAC allocation.

Similarly, the same application form is used for both flavors of allocations. However, the scrutiny with which the application is reviewed differs substantially, and in general the more resources you request, the higher the standard to which the application will be held by the AAC.

Developmental Allocations from the AAC

If you are new to the UMD HPC cluster, even if you have experience with HPC clusters at other institutions, this is the allocation type you should request first. This allocation grant from the AAC includes 50 kSU/year of compute time, 0.1 TB (100 GB) of high performance scratch disk space, and 1 TB of medium term SHELL storage.

As with all allocations and resources awarded by the AAC, there is no charge to the faculty member for this type of allocation.

The developmental allocation allows you to run test jobs on the UMD cluster to see how your applications and parallelizing algorithms work. This allows you to explore whether or not the campus HPC resources will be useful to your research. For small projects, this might even be enough to complete the project.

If the HPC resources appear to be useful to your research, you should be able to start working on your project with the resources awarded in the developmental allocation. While doing so, you should also be collecting some data about how your jobs run on the cluster; the AAC will want to see this data in any subsequent application for additional resources. The data you collect should include:

  • Scaling data on how well the codes being use perform with different amounts of resources, for each type of job you will be running, in order to determine the optimal amount of resources to request for a job. E.g.
    • Performance vs the number of CPU cores
    • Performance vs the number and type of GPUs used
  • If you are requesting more than the standard 4 GiB of RAM per CPU core, you should verify that almost all of your successful jobs are using almost all of the memory allocated to them.
  • The average walltime and SU cost for each type of job you will be running, based on the optimal amount of resources for the job determined above.

This data collection can generally be performed as you are running production jobs to advance your research.

For guidance on applying for developmental allocations from the AAC.

Larger Allocations from the AAC

As many projects will need more HPC resources than are available in a developmental allocation, you can request up to:

within a given year from the AAC. However, you will need to justify all of the resources you request in your application, and the more resources requested the more closely the AAC will be reviewing the application.

Tip

If you are new to the UMD HPC cluster, even if you have experience with HPC clusters at other institutions, you should first request a developmental allocation. Use this to start your research and collect the benchmarks on the cluster which will be needed for any subsequent application for a larger allocation.

If you request a larger allocation without a previous developmental allocation and the required metrics, the AAC will generally not approve you for anything more than developmental allocation anyway.

When applying for a larger allocation, the AAC will want to see:

The AAC expects that the data above was collected while using the resources of the initial, developmental allocation.

For guidance on applying for larger allocations from the AAC.

College/Departmental Allocations

The School of Engineering and the College of Computer, Mathematical and Natural Sciences have made significant contributions towards the purchase of the HPC cluster. As such, their HPC representatives have been given control over some "pools" of HPC resources. The criteria they use to distribute the HPC resources to faculty members are up to the unit "owning" the resource "pool".

Faculty members in the following departments should contact the listed reps for details. Some departments may have internal charge-back methods to recoup their cost of the HPC cluster.

College/Department Contact Person
ENGR -- All departmentsJim Zahniser
CMNS -- College-levelMike Landavere
CMNS -- AOSCKayo Ide
CMNS -- ASTRBenedikt Diemer
CMNS -- BIOLWan Chan
CMNS -- CBMGWan Chan
CMNS -- CHEMCaedmon Walters
CMNS -- CMSCJeanine Worden
CMNS -- ENTMGreg Hess
CMNS -- GEOLPhil Piccoli
CMNS -- IPSTAnil Zenginoglu, Gangadhar Kanamarlapudi
CMNS -- JQIJay Sau
CMNS -- PHYSJay Sau

If your College or Department isn't listed above, that does not mean you can't use the HPC cluster. You can still use the Basic and Need-based allocations from the AAC.

Compute resources (i.e. SUs) are meted out quarterly for college and/or departmental allocations; e.g. if your allocation is for 800 kSU per year, you will normally get 200 kSU per quarter over the 4 quarters of the year. At the start of each quarter (1 Jan, 1 Apr, 1 Jul, 1 Oct), any unused SUs vanish, and the allocation is replenished to have 200 kSU again.

Note

Unused SUs from one quarter do not roll over to the next quarter, they simply disappear.

Users are generally advised to use the SUs from allocations which replenish quarterly in preference to those from allocations (e.g. AAC allocations) which have annual expirations.

Storage (both scratch and SHELL tiers) is also allocated quarterly, but as files generally persist across quarters, this is usually not as noticeable. If one of your allocations is expired or is revoked, this may cause the storage quota for your project to be reduced. This may result in your project being over quota on one or more storage tiers. If this occurs, the PI of the allocation will receive email from HPC staff informing them of the issue, and requesting that the storage be brought under quota in a timely fashion. This can be accomplished either by reducing the storage footprint, or by increasing the storage quota (e.g. getting more quota from the AAC, department/college pools, and/or purchasing from DIT). If this is not done in a reasonable time period, we may be forced to bill the PI for the excessive storage used.

Class Allocations

DIT will make every effort to accomodate reasonable requests for HPC resources for use by classes for small projects and/or homework assignments.
Please assist us by submitting the class allocation request well in advance of the first day of the class. Note that class allocations are generally only granted for a single semester; the instructor will need to request a new class allocation for every class and every semester in which the allocation is needed, preferably about a month or so before the start of the semester..

If you are the instructor of record for the class, please open a ticket with the information requested below:

  • Official UMD Course ID (ex: BIOE 658B), Class Title (ex: ""Introduction to Medical Image Analysis"), and the Semester (ex: 2024-Fall)
  • HPC staff normally create a single allocation for all sections of a course. In some situations, we will consider creating multiple allocations per course, one for each faculty member. If you feel your course deserves such special treatment, please explain why.
  • If it is a multi-semester course, please provide above info for all semesters. You will still be required request the allocation each semester for which it is needed.
  • What is the estimated number of students?
  • What is the total number of resources needed? ( CPU core hours, GPU hours (and type of GPU), scratch and SHELL storage requirements, etc.) Class allocations will typically be awarded 50 kSU (about 50,000 CPU-hours or 1000 GPU-hours on an A100), 100 GB of scratch, and 1 TB of SHELL storage. While additional resources can be requested, it is unlikely that the AAC will award much more than double that amount, and then only with good justification.
  • How do you intend to use the HPC resources?
    • Live demos by the instructor? Do you need to reserve HPC resources? If so, what resources and how frequently?
    • Will students use the HPC resources in a "lab" setting (ie: with instructor and TAs present for assistance)? Will the lab need HPC reservations?
    • Will the students use the cluster for home work? How often and how much resources do you estimate per student and for the entire class?

NOTE: Class allocations will not be awarded for Independent Research, Thesis Research, or Dissertation Research courses. HPC resources for these courses should come from the faculty advisor's research allocation (see how to apply for a research allocation from the Allocations and Advisory Committee) (AAC)). Similarly, larger class projects should generally come from the advisor's research allocation.

Class allocations will expire shortly after the end of the semester for which they were requested. If there are extenuating circumstances which would necessitate that HPC resources be used beyond the end of the semester (e.g. a student has an incomplete, etc), please open a ticket with HPC staff describing the situation (preferably before the end of the semester) and we will attempt to reach a mutually satisfactory solution.

If you are teaching the same or another course which will need HPC resources in another semester, please make a new request for resources each and every semester.

If you need HPC resources for an unofficial class (ex: conference tutorials, on-site demos, etc.) please provide similar information to above. Also include the composition of the student base (current UMD students vs non-UMD individuals), and if possible a URL to a page describing the program. There are additional difficulties associated with setting up access for non-UMD members.

We recommend contacting the HPC team a month or two before the start of the semester for which the class is to start. This will allow us time to set up the class account so that you can login and check the availability and functionality of any software you intend to use early enough that we might be able to assist if there are problems. The HPC team is often quite busy, and software installations/updates/etc can be quite involved, so we might not be able to assist with such in the middle of a semester.

For both official and unofficial classes, the instructor and any TAs, etc. are expected to be familiar with the use of the cluster, and to provide any needed training to the students, workshop attendees, etc. The instructor and TAs, etc. are also expected to be the first line of support for the students, etc. in the courses; while the HPC team is willing to assist with non-trivial issues, the instructors are expected to be teaching and assisting the students with the basic use of the cluster. In particular, the instructors are responsible for:

  • Monitoring and managing the SU and storage usage of their class allocations.
  • Ensuring that the cluster is used efficiently by their students.
  • Instructing the students on basic principals of accessing the cluster
  • Instructing the students on the basics of Unix
  • Instructing students on basic HPC policies and etiquette, including:
  • Instructing students on the basic prinipals of HPC usage, including:
    • use of the batch system
    • submission of jobs
    • monitoring SU usage
  • Instructing students on the different storage tiers on the cluster, including:
    • usage of home vs scratch space
    • monitoring space usage of home and scratch space

For more information, please see policies on class allocations.

There is a bit of excess capacity that's available to be purchased directly from DIT. Funds will need to be transferred from your department to DIT accounts, and an MOU will be signed by both parties the for the exchange of funds for resources.

Resource Unit Price per Quarter Price per Year
Compute 1 kSU $2.32 $9.28
Scratch storage 1 TB $8.28 $33.12
SHELL storage 1 TB $5.57 $22.28
Combo package 1 unit $2.86 $11.44

One unit of the "combo package" consists of 1kSU compute, 15.8 GB scratch storage, and 73.0 GB SHELL storage.

Allocations purchased directly from DIT are governed by the Memorandum of Understanding (MOU) between the purchaser and DIT which was signed at the time of purchase.

The purchased resources are tied to the specific quarter(s) as mentioned in the MOU and any SUs not used within that quarter expire at the end of the time period. At the time of the planning and purchase, you may assign different amounts to each quarter.
For the ease of processing your funds transfers, we prefer to sell the resources on an annual basis. To obtain a "paid allocation" please contact the HPC systems staff. DIT will typically allow some revisions to the MOU after it is signed; e.g. to transfer SUs from a future quarter to the current quarter and similar.

These allocations have an expiration date; typically for one year from the start date, but this is negotiable in the MOU, and shorter terms (down to even a single quarter) are available on request.

Storage (both scratch and SHELL tiers) is also allocated quarterly, but as files generally persist across quarters, this is usually not as noticeable. If one of your allocations is expired or is revoked, this may cause the storage quota for your project to be reduced. This may result in your project being over quota on one or more storage tiers. If this occurs, the PI of the allocation will receive email from HPC staff informing them of the issue, and requesting that the storage be brought under quota in a timely fashion. This can be accomplished either by reducing the storage footprint, or by increasing the storage quota (e.g. getting more quota from the AAC, department/college pools, and/or purchasing from DIT). If this is not done in a reasonable time period, we may be forced to bill the PI for the excessive storage used.

If you are interested in purchasing an amount of resources which would exceed our existing excess capacity, or need specific hardware, etc. not currently included in the cluster, we can work with you on that. Such cases will require more of a dialog. Please open a ticket with HPC staff describing as best you can what you think you need, and we will reach out to you to schedule a meeting to discuss the matter further, including working out exactly what resources are required, etc. As this will generally require the purchase of additional hardware, we will need to obtain quotes from vendors before we can provide you with a price. Your total cost will be the amount for the new hardware, plus some small charges for shared resources consumed like network ports, etc.; in exchange you will receive an amount of resources (SUs, etc.), equal to the amount of resources being added to the cluster, less some percentage to account for downtime and to cover administrative costs. This will all be explained and detailed in the meeting.

How to Apply for an AAC Allocation

Please use the online form to apply for either a Basic/Development allocation, or a Need-based/renewal Allocation (a renewal of the basic allocation).

  • AAC grants only one allocation per faculty member (excluding class allocations) regardless of the the number of projects or students. AAC grant limit is 550kSU/year.
  • The grant period is one year--365 days from the date of the grant (not a calendar year as in Jan-1 to Dec-31).
  • The AAC will only award a single development allocation to a faculty member in the annual grant period. If additional resources are needed beyond what is available in a development allocation, you must apply for a renewal allocation.
  • All applications must come from a faculty member. However, a student may fill out the form on behalf of the faculty member as long as the faculty member approves it.
  • AAC allocations are granted for one year. You can submit a "renewal" application if:
    • the application has expired
    • you need additional resources from the AAC
  • There is no charge to the faculty member for either a developmental or renewal allocation from the AAC.
  • If you need more than the AAC's yearly limit, you can look into departmental and paid allocations.
  • The AAC and DIT do not in general provide software licenses for research software packages. Some commercial products (e.g. MATLAB) are available to all users on the cluster due to site licenses. These licenses are purchased either by campus or by individual units who allow access to all users on the cluster. In general, to use any other restrictively licensed software package on the cluster you must provide a license. Always contact HPC staff before purchasing any license you intend to use on the cluster; not all licenses are compatible with the HPC cluster.

All allocation requests are reviewed by the HPC systems staff first, and then by the AAC. The more resources you request, the more scrutiny the AAC will give your application. The AAC does not over-subscribe the HPC resources, hence, when an allocation is graneted, those resources are no longer available for other campus researchers. This is why the review process is very strict. If clarifications or more explanations/data are required, the application will be sent back to be updated accordingly.

Guidance for a Development Allocation application

UMD faculty members are eligible to a developmental allocation from the AAC for up to 50 kSU per year.

The HPC Allocations and Advisory Committee (AAC) can grant one-time unpaid allocations to faculty and students for small projects, classes, feasibility tests, etc. These allocations are granted out of computing resources purchased by funds from the Provost's Office and the Division of Information Technology.

Faculty members can apply for such allocations by submitting an application to the AAC.
This can be submitted by the faculty member themselves, or by a post-doc or student on behalf of the faculty member (in the latter case, the faculty member will be required to consent to assuming "ownership" of any resulting allocation). The review of the application by the AAC is more rigorous as more resources are requested; faculty members are eligible for up to 50 kSU per year with very little information. With proper justification and benchmarking and approval by the AAC, faculty members can get up to 550 kSU per year (including the aforementioned 50 kSU) for free from the AAC.

Generally, the AAC will not grant more than 50 kSU/year to faculty members without the faculty member having run jobs on the cluster to (and summarizing such in the application):

If the above criteria have not been met, the AAC will generally limit a researcher to an initial grant of 50 kSU/year. That allocation can be used to start the research, and while doing so collect the aforementioned benchmarks which can be used in a renewal request for additional resources (which can be submitted anytime; you do not have to wait for the initial allocation to expire).

The 50 kSU/year allocation size is also useful if you are new to HPC or the cluster. While the form is the same for all allocation sizes, the review of the application for the first 50 kSU/year for a faculty member is relaxed.

In general, when applying, answer all fields to the best of your ability, and HPC staff will get back to you with questions if more information is needed.

Guidance for a "renew" (Need-based Allocation) application

UMD faculty members are eligible to renew existing allocations from the Allocations and Advisory Committee (AAC), either to:

  • extend the expiration date of the allocation that is about to expire
  • increase the resources allocated to an allocation that is not expired/about to exp[ire.

Because the AAC tries not to oversubscribe the available HPC resources in order to try to keep queue times at reasonable levels, requests for HPC resources are carefully considered. The greater the amount of resources requested, the more scrutiny the application will receive by the AAC.

The AAC is responsible for ensuring that the valuable HPC resources are used responsibly and efficiently. As the application renewal process is one of the few opportunities for the AAC to review these aspects of a research project, this renewal process is taken very seriously. With proper justification, benchmarking and approval by the AAC, faculty members can get up to 550 kSU per twelve month period (i.e. year) (including the 50 kSU from any developmental allocations). a

Like all allocations from the AAC, the allocation is awarded from resources purchased by the Provost's Office and the Division of Information Technology, and there are no direct costs to the faculty member.

To apply for such an allocation, you must submit an application to the AAC. This can be submitted by the faculty member themself, or by a post-doc or student on behalf of the faculty member (in the latter case the faculty member will be required to acknowledge that the consent to the application and to assuming "ownership" of any resulting allocation).

Tip

Do not apply for a larger allocation if you or your faculty advisor do not already have a developmental allocation. The AAC will not award more than a developmental allocation in that case.

Tip

Faculty members are only eligible for a single allocation from the AAC, although that allocation can be renewed. If a faculty member is overseeing multiple research projects that will be using HPC resources, there should ideally be a single application requesting resources for all of these projects.

Tip

If a student or post-doc is applying on behalf of their faculty advisor, they should consult with their advisor and colleagues so that there is a single application covering all of the research under the faculty member which will be using HPC resources.

Failure to do so will make the research group appear disorganized, which will not make the AAC more likely to approve your application.

Tip

The AAC will be looking at all allocations of the faculty member, including those from departmental or college pools when determining whether to approve the application and for how much.

If a faculty member has multiple allocations, the application should present a justification for why additional resources are required from the AAC. If resources from the non-AAC allocation(s) are committed to other research projects, you will need to discuss those projects and include the estimated SU usage in the SU Justification section.

E.g., if you have a departmental allocation for 100 kSU/quarter, and your SU justification section only justifies 450 kSU, then the AAC is unlikely to award more than 50 kSU additional ( 450 kSU less 4 quarters * 100 kSU/quarter).

If there mitigating factors, please explain. E.g., if the departmental allocations are expected to be reduced or cancelled in the coming year, discuss that and the AAC will take that into consideration.

In your renewal application, the AAC will expect to see:

Managing Allocations and Projects

This section is for managers of projects and/or allocations on one of the UMD DIT maintained HPC clusters. The processes described below will in general not work for users who are not managers for the relevant project, and the HPC staff generally will not honor requests from non-manager users.

Managing Membership

Designated allocation/project managers can manage the membership of their allocations/projects in either of two ways:

  1. Directly via the Coldfront allocation management portal
  2. By opening a ticket with HPC staff

Tip

Using the ColdFront portal will generally result in faster results. It also allows better insight into your allocations and projects and usage.

Tip

If there are multiple allocations within the project, it is strongly recommended that the membership lists for all such allocations be the same. Membership in any of the allocations for a project grant the user full access to all of the scratch and SHELL storage allotted to the project, and generally it is recommended that users have access to the compute time for all of the allocations belonging to a project as well.

Managing Users in ColdFront

PIs and designated allocation/project managers can view and modify the membership lists of their allocations directly using the Coldfront web-based allocation management portal.

To add a user to your allocations, there are a few steps:

  1. Open up your web browser and log into Coldfront.
  2. Find your project.
  3. Add the user to your project
  4. Find your allocation(s) underneath the project and add the user to your allocation. Repeat the last step for all of the allocations under the project.

When adding multiple users at the same time, we recommend that you add all users to your project first, and then add the users to your allocation(s). The screen for adding users to the allocations allows you to add multiple users at once, and even add all project members not currently in the allocation with a single mouse click.

Note

NOTE You must add someone to both the project and at least one allocation for the person to get access to the HPC cluster. Adding someone to the project only does not do much; basically it only makes them eligible to be added to allocations for the project. It is adding the person to the allocation which actually grants them access to the cluster and allocation resources.

Tip

If there are multiple allocations within the project, it is strongly recommended that the membership lists for all such allocations be the same. Membership in any of the allocations for a project grant the user full access to all of the scratch and SHELL storage allotted to the project, and generally it is recommended that users have access to the compute time for all of the allocations belonging to a project as well.

To remove someone from the allocation and project, the process is essentially just the reverse of the process described above for adding an user:

  1. Open up your web browser and log into Coldfront.
  2. Find your project.
  3. Find your allocation(s) underneath the project and remove the user from your allocation. Repeat the last step for all of the allocations under the project.
  4. Remove the user from your project

Please note that it can take up to two hours from when you add someone to an allocation in ColdFront for the provisioning process to complete, granting the user access to the allocation. The deprovisioning process has some manual components and can take a day or so; please contact HPC staff if the removal of user access is more urgent.

Allocation Accounting

When a job runs, the allocation account against which it charges will get charged a number of SUs based on the amount of resources allocated to the job. The hourly SU rate is based on the maximum of various factors, including:

  • 1 SU/hour/CPU-core times the number of CPU cores
  • 0.25 SU/hour/GiB times the total amount of memory allocated to the job, in GiB in the standard or gpu partitions. Note that this is for the total memory allocated to the job (e.g. memory per CPU core times the number of CPU cores). Jobs in the bigmem partition get charged 0.0625 SU/hour/GiB instead of 0.24 SU/hour/GiB.
  • 144 SU/hour/GPU per H100 GPU allocated to the job
  • 48 SU/hour/GPU per full A100 GPU allocated to the job
  • 7 SU/hour/GPU per a100_1g.5gb GPU (i.e. 1/7 A100)

This hourly rate is then multiplied by the actual walltime of the job to determine the SU cost of the job. In actuality, this is all done in small timeslices; every few minutes or so, the scheduler will deduct a few minutes worth of SUs from the allocations associated with running jobs.

Note

NOTE: The hourly SU rate is based on the amount of resources allocated to your job; if you request resources that your job is not using (or not using effectively), they will still impact the hourly SU rate as these resources are reserved for your job while your job is running. On the other hand, your job is only charged for the actual walltime during which the job is running --- e.g. if you requested 15 hours and the job finished after 10 hours, you are only charged for 10 hours.

Jobs in the scavenger paritition are an exception --- jobs in this low priority allocation are not charged any SUs, but are subject to preemption.

The scheduler keeps track of all jobs running against a given account, and keeps track of how many SUs are required to complete these jobs (using the walltime requirements requested when the jobs were submitted). Before a new job charging against that account is started, the scheduler makes sure that there are sufficient funds to complete it AND all currently running jobs charging against that account. If there are, the job can be started; otherwise, it is left in the pending state with a reason code AssociationJobLimit or similar.

Choosing the Allocation to Use

If you only have a single account (check with the sbalance command), you can skip this section. You only have the one account, so there is nothing to choose.

If you have multiple accounts due to your membership in multiple research groups and/or projects, you may wish to choose which account you use based on your job. I.e., if the job is doing something for group A, you probably should only submit it using one of the group A accounts, even if you also have access to group B accounts. If the research areas of the two groups overlap, you will need to follow what ever group-specific policies may exist (we recommend you consult your colleagues).

If you have access to multiple allocation accounts within the same research group/project, and they all have sufficient funds to run the job in question, then there is a choice to be made. If your research group has group-specific policies about which allocation to use, follow those. Otherwise, you will normally see an allocation from the Allocations and Advisory Committee (AAC) plus one or more allocations from college and/or departmental resource pools, and maybe an allocation purchased from the Division of IT. Again, the sbalance command is an easy way to see this.

Generally, we encourage users to use SUs from college/departmental and paid allocations before using them from allocations from the AAC. This is because the former are allotted quarterly, meaning that at the start of each new quarter year (1 Jan, 1 Apr, 1 Jul, 1 Oct), any unused SUs in that allocation disappear and (unless the allocation expired) the allocation is replenished at its nominal quarterly level. Since the SUs in these allocations typically have the shortest lifespan, you generally want to use those SUs first.

Allocations granted from the AAC, on the other hand, represent an one-time grant of resources, and although these SUs will also expire and vanish at the end of the term of the AAC allocation, this is typical on a timescale of about one year.
Also, AAC allocations do not automatically renew --- your advisor (or you on your advisor's behalf) must apply to the AAC for any renewals, etc.

Therefore, we normally recommend that you set your default allocation to a paid and/or departmental allocation, and normally charge jobs against those allocations. If and when you encounter a situation wherein your workload for a given quarter is exceeding the quarterly allotment from your paid and/or college/departmental allocations, then you can dip into the AAC allocations to make up the difference. Although there are exceptions to this policy, we find that this type of arrangement is likely to maximize your benefit from the allocations.

Another consideration to be accounted for in this decision is the amount of SUs left in the allocation, and the number of running and pending jobs charging against the allocation. The scheduler will not start a job unless it determines that there are sufficient SUs in the allocation to complete the job in question along with all currently running jobs charging against that allocation. In order to estimate the number of SUs needed to complete a job, the scheduler uses the walltime requested for the job when it was submitted.

Example

If you have 5 kSU unused in your allocA allocation, and the job you wish to submit has a walltime of 50 hours and an SU billing rate of 96 SU/hour, the scheduler will assume the job needs 4.8 kSU to complete. If there are no other running jobs charging against the allocA allocation (this includes jobs by other people in your group), the scheduler will consider the job able to start if sufficient compute resources are available. However, if there are 5 jobs already running and charging against the allocA allocation which have an SU billing rate of 50 SU/hour and are halfway through their requested 6 hours of walltime, the scheduler will estimate that each job will run for 3 more hours and so consume 150 SU each, or 0.75 kSU for all 5 such jobs. In that case, the new job will not start because 0.75 kSU for the running jobs plus 4.8 kSU for the new job will exceed the 5 kSU unused in the allocation. This calculation will change over time; e.g. if 4 of those jobs finish right after this calculation (so effectively do not consume any of the 5 kSU unused), the next time the scheduler looks at the job it will see 0.15 kSU needed to finish the remaining job, and 4.8 kSU to finish the new job, or 4.95 kSU total, and the job will be able to start.

Note

The queuing system will not automatically select another account if there are insufficient funds in the account specified for the job. E.g., if you have access to both allocA and allocB and you specify that a job should charge against allocA (either explicitly or via the default account), the scheduler will not change that to allocB if allocA is depleted. The job will just wait in the queue until such time that additional SUs are available in allocA.

Note also that others in your group may have access to the same account, so just because funds were there when you submitted a job, someone else's jobs may have started since then and may reduce the funds in the account.

Instructions on setting the account against which a job charges.

Managing Allocations

This section discusses various topics related to the management of allocations. It is broken into two sections depending on what level of management:

For PIs

Most of the tools for managing allocations, etc. for PIs are through the ColdFront allocation management portal. A full list of these tools can be found at the aforementioned pages, these including:

You can also get a bit of useful information from the followimg commands:

  • The sbalance command can display the amount of compute time allotted to and used by your allocations, and with additional flags it will show this broken down by user.
  • The scratch_quota command can display the amount of scratch space allotted to and used by your projects, and with additional flags it will show this broken down by user.
  • The shell_quota command can display the amount of SHELL space allotted to and used by your projects, and with additional flags it will show this broken down by subvolume (including user subvolumes) in the project.

For College/Department Pool Managers

Certain units may have pools of resources on the cluster that they can allocate to researchers within their units. These pools are typically granted in return for contributions of monies toward the initial construction of the cluster. The college or departmental managers of these pools can then grant allocations out of the resources in the pool to various PIs in their respective unit.

The PI will typically already have a a project containing all of the allocations for the research group, usually including at least one from the AAC, and if you grant the PI an allocation from your pool, that will appear as another allocation under the project. There could be additional allocations as well, if the PI has an allocation from another department or unit, or if they have purchased resources.

The PI could also have multiple projects --- this could be because they have multiple research groups, or more typically if they have a project for a class they are teaching. If they are also a pool manager, the pool will also appear as a separate project.

Generally, all users belonging to allocations within the same project belong to the same group, and scratch and SHELL storage are organized by project.

The compute resources for each allocation in the project will appear as distinct Slurm allocation accounts, and when a job is submitted it will need to specify which account to charge against (or charge against the default allocation account). Storage resources are handled differently --- because it is difficult to classify a file as belonging to one allocation or another, and even messier to have to assign it in such a fashion, we sum up the storage allotments (separately for each storage tier) for all allocations in a project, and use that to set the storage limit for the project's storage directory on that tier.

Pool managers are responsible for allocating the resources in the pool to the individual researchers. Unfortunately, system administrators are not at this time (or in the foreseeable future) able to delegate the actual ability to create/modify allocations to the pool managers, so pool managers will need to send email to hpcc-help@umd.edu to request your changes. This will be handled by people, so you can send multiple requests in a single email.

You are allowed to oversubscribe the compute and/or storage resources in your pool; that means it is permissible for the sum of the compute and/or storage limits (separately for compute and each tier of storage) alloted to each of the suballocations from this pool to exceed the size of these resources in the pool. This can be advantageous if you have a fair number of PIs who have a modest average quarterly usage of compute time, but on rare occasions their needs spike. By oversubscribing these allocations, you can let them have enough SUs to cover those spikes, as long as the average use over all allocations in the pool remains under the pool's limit.

While this has benefits, it needs to used carefully, because to be fair to other users on the cluster, the total usage from suballocations from your pool are restricted to the available compute resources in pool (on a quarterly basis). E.g., if you have a pool of 100 kSU/quarter and you have two suballocations A and B to which you assign 75 kSU/quarter each, then if A uses 75 kSU in a given quarter, B is limited to 25 kSU. So this will work if both A and B only use 50 kSU/quarter on average, and when one uses more than average, the other uses correspondingly less. But clearly there will be complaints if there is a quarter where users of both A and B suballocations want/need to use more than 50 kSU.

You can also oversubscribe storage. This is more problematic, since while SUs are somewhat ephemeral (every quarter the quarterly SU usage is reset), files tend to be more permanent --- once created they remain until someone deletes them. Furthermore, there are various mechanisms by which individual projects can end up going over their filesystem limits. For instance, the project limit is contributed from various allocations, which can expire. E.g., consider a project with a 3 TB limit, with 1 TB coming from an AAC allocation, one from your pool, and one from a project level purchase agreement with DIT, and assume that the project is using 2.9 TB of storage. If one of these allocations expires, suddenly the project is consuming 2.9 TB but only has a 2 TB limit. Assuming the allocation from your pool has not expired, your pool will now be considered to be consuming 1.45 TB from that suballocation (despite your only authorizing 1 TB to that suballocation). It is even more complicated than that, as the enforcement of storage limits has some technical limitations. The Zaratan scratch storage has some delays in the quota enforcement stage, so users can in some cases continue to write data over the limit for a fraction of an hour after exceeding the limit. And the SHELL storage for each project is divided into multiple volumes (at least one per user, and perhaps more), each of which has a size limit, but does not have quotas as such. The sum of these maximum volume sizes can exceed the SHELL storage limit for the project. So it is very possible for individual projects to exceed their storage limits, at which point they will be notified and instructed to rectify the situation, but such can also impact the usage of your pool.

It is recommended that pool managers use care if/when oversubscribing, especially for storage. Ideally, you should avoid oversubscribing initially if possible, and wait until you have several quarters worth of data showing actual utilization of the resources in the pool. If there is a consistent history of under-utilization, then it might be reasonable to allow for some oversubscription if needed, but even then you probably should be a bit conservative and allow some room for fluctuations in the average usage. For storage, you should also remember that storage usage, especially on the SHELL tier, is likely to monotonically increase.

The are several things that pool managers can do in the ColdFront Allocation Management portal:

  • If you wish to delegate/share/designate a backup for managing the pool, you can do so in ColdFront. To authorize someone else to contact the HPC admins to create/delete/alter suballocations in your pool, you can just add the user as a manager of the project associated with the pool or remove their authorization by removing them as a manager. You may wish to inform us of such changes, as while adding them as a manager of the project will cause us to treat them as authorized to give us instructions regarding suballocations, it does not automatically grant them permission to view metrics for the suballocations in ColdFront.

  • You can see various metrics about the pool in ColdFront. To do so, login and click on the project associated with the pool.
    Scroll down, and you will see a section called "Allocations", which displays an allocation listing resources within the pool. Important resources values are:

    • kSU (Allocatable): This gives the number of kSU/year that have been allocated (i.e. the sum of the kSU/year limits for all suballocations of the pool)/ the total number of kSUs/year available in the pool.
    • Storage Allocated (TB) - HPFS: This gives the number of TB of scratch space that has been allocated versus the amount of such storage in your pool.
    • Storage Allocated (TB) - SHELL: This gives the number of TB of SHELL space that has been allocated versus the amount of such storage in your pool.
    • kSU (Quarterly): This gives the number of quarterly kSU that users in the various suballocations have used in the current quarter, versus the total amount available for suballocations of the pool. This total amount is one quarter of the yearly kSUs available in the pool. This limits the usage of all the suballocations of the pool to the size of the pool --- you are able to oversubscribe the pool (i.e. the sum of the SUs allocated to all of the suballocations can exceed the number of kSUs in the pool), but for fairness to other users, the total usage of the suballocations is limited to the size of the pool.
    • Storage Quota (TB) - HPFS: This gives the number of TB of scratch storage that users in the various suballocations are currently using, versus the total amount available for suballocations of the pool. Since the storage limit for each suballocation is the sum of the limits on scratch storage for all allocations in a project, this metric is a bit tricky. We determine what fraction of the limit of scratch storage for the project is being used, then compute what that fraction is of the scratch storage contributed to that limit by each allocation and assign that as the scratch space used by the allocation. This metric is useful if you oversubscribe the scratch storage in your pool. If the used scratch storage exceeds the limit, you will be requested to quickly rectify the problem, either by getting users/research groups to reduce their usage, or by obtaining/purchasing additional storage.
    • Storage Quota (TB) - SHELL: This is similar to Storage Quota (TB) - HPFS, but for SHELL storage. Basically, it displays the number of TB of SHELL storage that users in the various suballocations are currently consuming, versus the total amount available for suballocations of the pool. It is useful if you oversubscribe the SHELL storage in your pool; if the used SHELL storage exceeds the limit, you will be requested to quickly rectify the situation.
    • You can also see the above metrics in more graphical form by going to the allocation associated with the pool. Either click on the associated allocation from your home page, or if you are on the project page, go to the Allocations section and click the icon on the right (under the Actions heading) for the allocation.

    • You can also see information about the various suballocations from your pool. Actually, we do not currently have sufficiently fine grained control to restrict this to only suballocations of the pool you manage, so you will have access to see information about all allocations and projects in ColdFront. Please be respectful of the privacy of projects and allocations not related to the pool you are managing. To see information for a subproject/suballocation of your pool:

    • Log into ColdFront.

    • Select Projects from the menu bar
    • Click on Filter
    • Enter any search criteria (i.e. the PI's name or field of science)
    • IMPORTANT: Click the Show all projects button
    • Click the Search button
    • You will then see a list of projects. Click on the ID number to see more information about the project. When you see the Allocations list for the Project, you can click on the icon under the Action header to see more information about that Allocation.

Monitoring Allocations

You and your research group are responsible for ensuring proper rationing of the funds in your allocation account(s). Excessive use of funds for a non-AAC allocation at the beginning of a quarter could result in no funds being available for jobs at the end of the quarter.

This can be deliberate and beneficial, e.g. if you have important deadlines in the middle of the quarter and are willing to "borrow ahead" to get computations for that completed before the deadlines. This is an advantage of the model used by the UMD HPC clusters; you can use nearly 3 times the power of the computers you purchased in a single month to rush out computations, at the cost of having very limited usage the following two months (but since it is after the deadlines, that might not be important).

But if such a shortage of funds occurs because some junior member of the group is sending an excessive number of very expensive jobs, this can be quite problematic, especially as you might not notice the impact of the errant user until too late.

The Division of Information Technology cannot tell which jobs are important and which are not, nor what is good usage of your allocation funds and what is not. If we notice seriously problematic usage (e.g. a job reserving 10 nodes but only running processes on 1 node), we will do our best to notify and instruct the relevant users. But you are responsible for monitoring your own jobs, and it behooves you to monitor jobs of other users of your allocations. We will provide the necessary tools to do such, but we strongly advise all research groups to have at least one person monitor the usage of their allocations' funds regularly to ensure there are no problems, or at least catch any problems early.

How many SUs are left in my allocation?

The first level of monitoring of your allocations is with the sbalance command which can be used to see how many SUs were allotted to the allocation and how many were used for the current quarter.

More information on the command can be found in the utilities page or by giving the --help option to the sbalance command.

How much scratch storage space do we have?

The scratch_quota command will show the quota and usage for the high-performance or scratch tier of storage on the cluster. Without any arguments, it will display the usage and quotas on the scratch filesystem for all groups to which you belong. This should include a group named zt-PROJECTNAME for each project to which you belong. Currently, the list might also include other groups which do not correspond to projects; these will be shown with no usage or quota. After that, it will show the total usage (across all projects/groups to which you belong) for you.

More information on the command can be found in the utilities page or by giving the --help option to the scratch_quota command.

How much SHELL space do we have?

The shell_quota command will show the quota and usage for the medium term SHELL tier of storage on the cluster. Without any arguments, it will display the usage and quotas on the SHELL filesystem for all projects to which you belong.

More information on the command can be found in the utilities page or by giving the --help option to the shell_quota command.

Seeing job history

The sacct command can be used to view the accounting records of jobs, both past and currently running. It takes some time to run, and can display a fair amount of information (which is documented in its man page). You will almost always wish to restrict it to a time range, using the --starttime argument. For example, to see the usage of allocation account foo-aac for the month of November 2024, one could use a command like

Example

login-1> sacct --format=JobID,State,ReqTRES,AllocTRES,Elapsed \\
-P -X -v -v -v --start 2024-10-01 --end 2024-10-31 -A test-aac 
JobID|State|ReqTRES|AllocTRES|Elapsed
8057991|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:25
8148243|COMPLETED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:04
8148253|CANCELLED by 123456|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:06
8148261|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:00:05
8148381|FAILED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|00:02:07
8148416|COMPLETED|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|billing=96,cpu=24,gres/gpu:a100=2,gres/gpu=2,mem=360G,node=1|02:13:37

More information on the command can be found in the utilities page or by giving the --help option to the sacct command or man sacct. We also discuss this more in the page on managing jobs.

Usage reports

The sbalance command returns the usage for the current quarter, and the scratch_quota, shell_quota, and home_quota commands all report the current usage status. While this is probably what most users are concerned with most of the time (e.g., does my allocation have sufficient resources to run my job now, and I do not care about usage from previous quarters), sometimes one needs information regarding usage over longer time scales. This is especially useful for people who manage "pools" of resources for departments or colleges.

There are a couple of tools available to get more historic information regarding allocation use:

Using ColdFront

The ColdFront allocation management portal is more generally used to manage allocations and projects, especially the membership of such, but you can also view usage summaries for the past year or so. If you view the attributes of an allocation, you will typically see attributes with names like kSU.Q1, scratchTB.Q3, or shellTB.Q2 --- these represent the usage of compute, scratch space, or SHELL space for last completed quarter corresponding to the number after the Q. E.g., if the current date is Aug 21, 2025 (which falls in the quarter 2025Q3), the *.Q2 values will refer to 2025Q2, the *.Q1 values to 2025Q1, the *.Q4 values to 2024Q4, and the *.Q3 values to 2024Q3. On Oct 1, 2025, the 2025Q3 quarter will have been completed, and the *.Q3 values will then refer to 2025Q3. This allows for a quick graphical gauge of the usage over the past year or so.

Using XDMoD

The Zaratan XDMoD website is a web page running the Open XDMoD (Open XD Metrics on Demand) web application. This can present in graphical form many metrics pertaining to the Zaratan cluster. One can see how many kSUs were consumed by a given allocation as a function of time, or what was the average job length for an allocation over the past year. Although some features are available without logging in, for the best experience we recommend that you log into the site using your standard UMD username and password. This allows for more detailed analysis, but generally does not contain any information regarding the allocation limits.

Using sreport

The sreport command can be used to generate a variety of usage reports to show usage by allocation and/or user over specified time periods.

For example, to see the usage of A100 GPUs by users of the hogwarts-aac allocation in March of 2024, one could give the command

Example

    login-1:  sreport --tres=gres/gpu:a100 -t Hour cluster UserUtilizationByAccount Start=2025-03-01 End=2025-03-31 Account=hogwarts-aac
    --------------------------------------------------------------------------------
    Cluster/User/Account Utilization 2025-03-01T00:00:00 - 2025-03-30T23:59:59 (2588400 secs)
    Usage reported in TRES Hours
    --------------------------------------------------------------------------------
      Cluster     Login     Proper Name         Account      TRES Name     Used
    --------- --------- --------------- --------------- -------------- --------
      zaratan  hpotter  Harry Potter        hogwarts-aac  gres/gpu:a100     4328
      zaratan  hermione Hermione Granger    hogwarts-aac  gres/gpu:a100        0
      zaratan  voldemort Tom Riddle         hogwarts-aac  gres/gpu:a100      269
      zaratan  weasleyr Ron Weasley         hogwarts-aac  gres/gpu:a100        0

All usages in the above example are GPU-hours.

More information can be found in the sreport section of the utilities page or the sreport man page (or run "man sreport").