Skip to content

FF chevron

Running Jobs

This page dicusses various topics related to the running of jobs and different types of codes.

When your job starts to execute, the scheduler will execute the script file you submitted on the first node assigned to your job. If your job is to run on multiple cores and/or multiple nodes, it is your script's responsibility to deliver the various tasks to the different cores and/or nodes. How to do this varies with the application, but some common techniques are discussed here:

We also discuss various topics related to evaluating and improving the efficiency of your jobs. Doing so is an important part of using the cluster resources responsibly, will generally improve your throughput, and is required in applications to the Allocations and Advisory Committee (AAC) for additional resources. In particular we discuss:

Running Sequential jobs

Sequential or single-core jobs represent the simplest case. Indeed, when the scheduler sttarts your job, the job submission script will be executed on a core of the first node allocated to the job (and for a sequential job, this should be the only node and core allocated to the job --- if multiple nodes and/or cores are allocated to your sequential job, there is an error in your job request). So there is really nothing special that you need to do; just enter the command for your job and it should run.

You still need to submit jobs to the scheduler to have them run on one of the compute nodes; i.e. you will probably want to submit a batch job script via the sbatch command.

For more information, see our sequential job example.

Running Multithreaded jobs on a single node

Many codes used on HPC are multithreaded, and these jobs will generally see performance benefits by running on multiple CPU cores. Multithreading requires all threads to be able to access the same memory, so all cores must be allocated on the same node; so usually you will want to use the "-c" or "--cpus-per-task" option to the sbatch command.

For more information, see our multithreaded job example.

Running MPI jobs

Many codes used on HPC clusters support MPI as a standardized and portable system for communication between the various tasks of parallized jobs. The MPI mechanism supports jobs which span multiple nodes. A number of different implementations of MPI libraries are available n the Research Software Library (RSWL), and usually several versions of each. Although the MPI interface itself is somewhat standardized, the different versions are not binary compatible. It is important that you match the MPI implementation (and version, and to be safe the compiler and compiler version you use and with which your code was compiled.

Fortunately, the commands to run MPI-enabled codes are generally largely independent of the MPI implementation. Basically, you should use the module command to module load the appropriate compiler and MPI libraries, along with whatever codes you wish to run, and generally then use the MPI command mpirun to launch your code; the mpirun command will launch the requested number of MPI tasks on the nodes allocated to the job.

Tip

MPI codes generally need to be launched using the "mpirun" from the same implementation (and usually same version, down to minor version) of the MPI library with which the code was compiled.

The module command generally tracks the MPI library packages were compiled with, and will try to prevent you from mismatch MPI or compiler versions. But you will need to do this manually for codes you have compiled.

Note

NOTE: Your code must be MPI aware to effectively use MPI. Running a non-MPI code with mpirun might not abort, but you will most likely have every core assigned to your job running the exact same calculations, duplicating each others work, and wasting resources.

Tip

The openmpi installations in the RSWL are CUDA enabled. Although they will work on nodes with or without GPUs, on nodes without GPUs, you are likely to get warnings about CUDA libraries. These are harmless and can be ignored as long as your job was not expecting to use GPUs --- since your job is not using GPUs, it was likely assigned to non-GPU enabled nodes which do not have the hardware specific CUDA libraries installed, thereby causing the error. But since the job is not going to use GPUs, it does not need the CUDA libraries, so the warning can be ignored.

For more information, see our MPI job example.

Running hybrid OpenMP/MPI jobs

Some codes support hybrid parallelization, in which parallelization occurs via both multiple MPI tasks, with each task also doing multithreading. In this case, the resources for each MPI task must be allocated on the same node, although resources for different tasks can be allocated to different nodes.

For more information, see our hybrid (MPI + multithreading) job example.

Ensuring that jobs run efficiently

As an user of the HPC cluster, you should ensure that your jobs are making efficient use of the cluster. Doing so is an important part of using the cluster resources responsibly, will generally improve your throughput, and is required in applications to the Allocations and Advisory Committee (AAC) for additional resources.

Checking memory usage

Memory is a critical HPC resource, and it is important that jobs use it efficiently. This is particularly true for jobs that are requesting more than the default 4 GiB per core or RAM.

When your job requests RAM using the appropriate sbatch flag, that memory is reserved on the allocated nodes for use by your job and only your job, for the duration of your job; i.e. those resources are not available to any other jobs or users. Therefore, the allocation account for the job is charged for this memory. This is true whether the memory is actually used by your job or not.

If your job uses more memory than was allocated to the job (either explicitly via sbatch arguments or from defaults), the job will die with an "Out-of-Memory" (OOM) error. So you want to ensure that the memory requested for the job is a bit more than the job will actually use. However, at the same time, you do not wish to request significantly more memory than the job will need, because such an excessive request will waste valuable resources. So care is needed to properly size your memory requests.

To properly size the memory request for a job, the best course is to examine jobs that ran successfully, and check how much memory was actually used by the job. The easiest way to do such is with the seff command, just provide a job number and it will display the memory requested, the memory used, and the ratio of the two. Your jobs should generally be using at least 80% of the memory requested, and ideally around 90-95%.

If the memory usage of your jobs is stable (or at least predictable), the above is usually sufficient; just find an appropriate amount of memory to request for jobs of each of the expected sizes based on what past jobs were using, and use this reduced value with future jobs.

If the memory usage of your jobs have significant and unpredictable variations, the situation is more difficult. Generally you will want to look for a value for the memory setting which satisfies almost all of the jobs, and use that. Some jobs will fail due to OOM errors, and these jobs will need to be rerun at a higher memory setting. But in many cases, this process is more efficient than running all of the jobs at a higher memory setting.

Example

E.g., let's assume you need to run 100 jobs, all single core, CPU only jobs which run for 20 hours, and 90 of these jobs require more than 10 GB but less than 12 GB of RAM, but the other 10 jobs require between 22 and 24 GB of RAM, but you cannot predict beforehand how much memory a given job will need. If you were to run all 100 jobs requesting 24 GB of RAM (the amount needed by the most memory-hungry job), the hourly SU cost for each job would be 0.25 SU/GB/hour * 24 GB = 6 SU/hour, so the cost of all 100 jobs would be 100 jobs * 6 SU/hour * 20 hour/job = 12 kSU. But if instead we ran all jobs requesting only 12 GB of RAM, this would cost 0.25 SU/GB/hour * 12 GB = 3 GB/hour, and running all 100 jobs would cost 100 jobs * 3 SU/hour * 20 hour/job = 6 kSU. Of course, we would still have 10 jobs that failed and would need to be rerun with 12 GB, costing 10 jobs * 6 SU/hour * 20 hour/job = 1.2 kSU, for a total of 7.2 kSU, or only 60% of the SU cost compared to running all jobs requesting 24 GB.

Checking CPU efficiency

CPU cores are another important HPC resource, and is the basis of the "high-performance" in high-performance computing --- by parallelizing codes and running each task or thread on a different core to achieve performance gains.

Although the seff command also show computation efficiency, this is currently only works well for CPU-only jobs, and even in those cases the efficiency numbers will consider time used for I/O, etc., as inefficient. While ideally your job will want to see the compute efficiency reported by seff as high as possible, that often is limited by the coding of the application and perhaps even the algorithm, neither of which is generally readily improved. So generally we focus on ensuring that the job is requesting the correct CPU configuration.

Typically, the performance of jobs will increase significantly, almost linearly, as additional CPU cores are made available to the job, up to some threshold value. Increasing the number of CPU cores beyond that value yields little if any increase in performance, and indeed in some cases may even degrades performance.
This threshold value is dependent on many factors, including details of the problem being solved, the algorithm being used and the exact coding of algorithm, as well as details of the cluster it is being run on.
Because of this, this threshold is generally best determined empirically.

You should be conducting benchmarks on the cluster to show that you are running your jobs in the optimal range, where performance is still scaling well. E.g., if the job runs almost twice as fast with 16 cores as compared to 8 cores, but only about 20% faster at 32 cores compared to 16 cores, why would you run the job at 32 cores? The AAC will wish to see a detailed summary of these tests in any renewal applications.

For CPU only jobs, this is generally a matter of running jobs (either production jobs or test jobs which are expected to behave similarly to production jobs for this purpose) with different numbers of cores, and look at the observed parallel speedup. You can use the total runtime for the job, or some other suitable metric as appropriate (be sure to explain to the AAC what metric was used). Because there is usually some sequential overhead, you probably want to use jobs which run at least a couple of hours for these benchmarks. You can run the same exact calculation with different cores, although often it is more efficient to run different jobs with different number of cores, as long as you expect the metric to be consistent; that way you can actually get production data while conducting the benchmarks. If you are using different jobs for differ core counts, you should generally run several jobs at each core count and average the results; this will allow you to better judge how much or how little variation there is in the metric just due to performing different calculations.

Applications typically show nearly linear performance gains as CPUs are added, e.g. doubling the number of CPU cores should almost halve the run time. Once you start seeing signicantly less performance gains (e.g. a 25% increase in the number of cores only yielding a few percent gain in performance) you can generally terminate your tests. The goal of this exercise is to determine the optimal number of cores to use to maximize efficiency and throughput.

The above discussion applies well to either purely multithreaded jobs and pure MPI jobs. For hybrid jobs (which do both MPI and multithreading), you generally need to modify the process above to do a two parameter search for the "sweet spot" by varying both the number of cores used per MPI tasks and the number of MPI tasks. (This is assuming that neither are constrained by the nature of the problem being solved).

GPU jobs also will make use of CPU resources, and in many cases changing the number of CPUs being used will affect the performance to some degree, although typically not as strongly as in CPU-only job. Even so, once you determine the optimal GPU configuration for the job, you should conduct additional tests to determine the optimal number of CPU cores to use. The above process is generally applicable, but as GPU costs are typically significantly more than CPU costs and are likely to dominate the job's SU cost, a somewhat different interpretation of the results is called for. For the full A100 GPUs, there are 4 GPUs on a node with 128 cores, so you can use up to 32 cores without increasing the hourly SU cost of the job, and it would be most efficient to use the number of CPU cores (up to 32 cores per GPU) which yields the greatest performance. This is true even if the performance gain would not be considered worthwhile in a purely CPU based job.

E.g., if a job using a single A100 GPU sees a 20% reduction in runtime when using 32 cores vs 16 cores, it still would be worthwhile to use 32 cores. This is because unlike the previous example, where we claimed a doubling of the number of CPU cores for a mere 20% performance gain is not efficient, in the GPU case the 20% performance gain for the doubling of the number of CPU cores will reduce the amount of time the value GPU resource is consumed, and therefore is efficient.

For the H100 GPUs, there are 4 GPUs on a 96 core node, so the corresponding number of CPU cores is 24. Using more than 32 cores for an A100 or 24 cores for an H100 is more problematic, as doing so could prevent the use of the other GPUs on the node, which would generally not be efficient unless you are seeing quite significant performance gains in your code.

Checking GPU efficiency

GPUs are very valuable resources in the HPC cluster, and as such ensuring that they are being used in an efficient manner is quite important.

For all GPU-centric jobs, you should examine all of the GPU types supported by the cluster being used:

-- GPUs available on the Zaratan cluster

and determine which ones are able to support your job. Generally, GPU memory is the determining factor, although in some cases using multiple smaller GPU devises can work. Once you determine which GPUs your jobs can run on, you should run tests to see which one yields the best performance. This is generally best done by running jobs on the different GPUs and comparing performance.

If your application supports using multiple GPUs, you should also determine the optimal number of GPUs to use for your job. Again, this is best done by running test jobs wherein you vary the number of GPUs allocated to the job. Ideally, you should see a performance increase which goes almost linearly with the number of GPUs, e.g. when using two GPUs the code should be almost twice as fast as using a single GPU, and going from 2 to 3 GPUs should see an almost 50% increase in performance.

Performance can be measured in various ways; the simplest is typically to see how long it takes for the same or similar jobs to complete, but other metrics might be suitable in your case (e.g. the number of iterations performed per unit time). When doing test jobs, you should make sure they run at least a couple of hours so that various sequential overheads, etc. do not unduly bias the results. You can conduct the performance tests on full production jobs or test jobs, as long as the test jobs are expected to behave similarly to the production jobs. You can run the same calculation at each GPU count, but often it is more efficient to run different production jobs for each GPU configuration --- this allows you to get production data at the same time that you are conducting benchmarks. If you run different calculations for each GPU configuration, you should run multiple jobs at each configuration and average the results; this also gives you a feel for how much variation the metric shows from calculation to calculation.

Once you determine the optimal GPU configuration, you should run some tests to find the optimal number of CPU cores to use with that GPU configuration.

The AAC will wish to see a detailed summary of these tests in any renewal applications.