Skip to content

FF chevron

Monitoring Jobs, Allocations, and the Cluster

In order to use the HPC cluster effectively, you need to be able to monitor aspects of your jobs, your allocations and the cluster as a whole.

This page discusses various topics relating to doing such.

  1. Monitoring Jobs and Job Steps
    1. Listing Running/Queued jobs
    2. When will my queued job start?
    3. Details about running/queued jobs?
    4. Listing finished jobs
    5. Showing memory/compute efficiency
    6. Details about jobs that have run
  2. Monitoring Allocations
    1. SU usage this quarter
    2. Current scratch usage
    3. Current SHELL usage
    4. Usage reports
  3. Monitoring the Cluster
    1. Using XDMoD
    2. Using stats page

Monitoring jobs

The Slurm scheduler generally maintains records about jobs in two distinct places, one primarily dealing with jobs which are queued or running, and one primarily useful for jobs which have already completed/terminated, and the commands to view that information basically fall into those two categories as well.

  1. Listing Running/Queued jobs
  2. When will my queued job start?
  3. Details about running/queued jobs?
  4. Listing finished jobs
  5. Showing memory/compute efficiency
  6. Details about jobs that have run

Monitoring allocations

Because all allotments of resources are assigned to allocations and projects, it behooves you to monitor the usage of the allocations and projects you belong to, especially as there are typically multiple users belonging to such.

  1. SU usage this quarter
  2. Current scratch usage
  3. Current SHELL usage
  4. Usage reports

The various tools listed in the Monitoring Jobs section can also be used to monitor the jobs charging against an allocation at a more job-oriented level.

Monitoring the cluster

Although the various commands listed in the Monitoring Jobs and Monitoring Allocations sections also can be used to provide information about the utilization of the cluster, there are some additional web based tools which provide more graphical and more general data on the cluster, namely:

  1. Using XDMoD
  2. Cluster metrics page
  3. Using sview

In addition, notices of scheduled and unscheduled outages or issues on the cluster will be announced on the

Cluster metrics page

We maintain a web page displaying various metrics for the DIT maintained HPC clusters, namely:

Metric data are currently retained for about 2 weeks.

The Zaratan Metrics page offers a collection of dashboards:

  • Kiosk View dashboard: This display a quick "kiosk" view showing donut graphs of current usage on the cluster, etc. A basic high-level overview of current utilization.
  • GPU Aggregate Metrics dashboard: This presents aggregated metrics over all GPUs (e.g. number of GPUs which are active, etc.)
  • GPU Metrics dashboard: This shows various GPU metrics for specific nodes.
  • Server and Node Overview: This shows various CPU and memory related metrics for specific compute nodes.

Most dashboards have a dropdown near the top right which allows you to control the time range displayed. Just to the right of that dropdown is a magnifying glass icon; you can use this to zoom in on a time range --- just click on the magnifying glass icon and then drag the mouse with the left button down to select a time range in one of the graphs.

Many dashboards have a dropdown near the upper left that allow you to restrict the output to a specified compute node