Skip to content

FF chevron

Glossary Of HPC Terminology

High Performance and related computing paradigms have their own set of terminology which might be confusing to those new to these fields. This page tries to define the terminology as used throughout this web site in simple language. It also defines and explains some basic computing terms and some terms which might be a bit more specific to the UMD HPC environment.

Terms are listed in alphabetical order.

A

(Hardware) Accelerator

While traditionally in computing the computation takes place on a general purpose CPU, many workloads can obtain great improvements in performance when run on specialized processors like GPUs, field-programmable gate arrays, and other technologies. The general term for these specialized processors is accelerators or hardware accelerators.

These accelerators typically must be placed in a node containing a generalized CPU, and programs must be specifically coded to take advantage of these technologies. These accelerators can often be expensive, with nodes containing the accelerators generally costing several times the cost of a comparable node without the accelerators, but the performance gains for suitable workloads can be very significant. Because of this, typically only a small subset of the nodes have accelerators, and you will need to instruct the scheduler to use those nodes

Access Control List (ACL)

The traditional Unix file permission scheme can cover some common use cases, but it has limitations, especially if you need to grant access to files to users in distinct groups. To cope with these limitations, the concept of access control lists, or ACLs, was created. ACLs supplement or replace the traditional file permissions and allow one to grant specific permissions to specific users or groups.

Different filesystems use different ACL mechanisms; although the basic concepts are essentially the same, the actual commands to manage the ACLs are different. But generally, there are a bunch of permission flags for each file or directory, and you can grant those permissions on a file or directory to specific users or groups. On the UMD HPC clusters, the relevant ACL schemes are:

See also:

Account

The term "account" can have two disparate usages.

The first usage, sometimes distinguished with the terms "user" or "login", refers to the user credentials that you use to log into the cluster, and other attributes, etc. connected with your user identity on the cluster. This sense of the word "account" is similar to the login/user accounts on other computer systems. At UMD, we will try to use the terms user, login, or at least user account or login account when referring to this definition.

The second usage refers to allocation accounts, sometimes simply referred to as allocations. This is the what Slurm and some other HPC utitilities on the system usually mean when they refer to account. We will try to use the terms allocation or allocation account for this sense of the word.

ACL
See Access Control List (ACL)
Allocation

Allocations are the means by which HPC resources are made available to users. Every allocation consists of a set of resources to which HPC members have been granted access; this consists not only of the type of resource, but also the amount of that resource that can be used. E.g., allocations will typically include access to compute resources; this will include the specification of the cluster access is granted to, as well as the number of SUs over a time period (and what time period) that the members of the allocation have access to. Allocations typically will give access to storage resources as well; this will include the specification of which storage resources (e.g. high-performance storage or SHELL storage, and the limits on the amount of data that can be stored there.

Every allocation is contained within a single project which represents the research group the allocation is for. Every member of an allocation must also belong to the containing project.
So every user of the cluster must be a member of a project and a member of an allocation in that project which provides resources on that cluster.

(Job) Array
See job array
Auristor file system

The Auristor file system is the distributed file system used to back the SHELL storage tier on the Zaratan cluster. It is an enhanced version of the AFS file system long used on the campus Glue/TerpConnect Unix clusters.
It uses kerberos based authentication, and provides a globally distributed file system.

See also:

Auristor/AFS Access control lists (ACLs)

The Auristor file system uses its own version of access control lists (ACLs) based on the AFS filesystem from which it descends.

This ACL system applies to directories only; files inherit ACLs and permissions based on the containing directory. User and group ownership is generally ignored, except that the user owner will have permission to change the ACLs. All of the standard Unix file permission bits are ignored except for the user permissions --- these are used to mask the permissions granted by the ACLs on the containing directory at the file level. E.g., if the directory grants read and write permission to the members of a group, if a file has standard file permissions of "r--??????" (the fields marked with question marks "?" are irrelevant), then the group will get read access (because the ACL and the user permission flags both allow it), but not write access (because although the ACL allows it, the user permission flags do not).

The following permission flags are allowed on directories:

  • lookup (l) flag: This allows one to list the files in the directory.
  • insert (i) flag: This allows one to create new files in the directory. That includes copying files into the directory.
  • delete (d) flag: This allows one to remove files and subdirectories from the directory.
  • administer (a) flag: This allows one to modify the directory's ACL.
  • read (r) flag: This allows one to read the file
  • write (w) flag: This allows one to write to the file. Note that while this does not allow one to completely remove a file, it will allow one to replace the content of the file, including basically replacing the file with an empty file.
  • lock (k) flag: This allows the user (or more typically, codes run by the user) to "lock" the file.

The permissions granted to an user or group are represented as a string of the one letter abbreviations of the above permissions, order does not matter. There are also some special "shorthand" permissions:

  • none: This means none of the above permissions
  • all: This means all of the above permissions, i.e. "rlidwka"
  • read: This equivalent to "rl" permissions
  • write: This is equivalent to "rlidwk" (i.e. everything but administer)

Technically, this ACL scheme supports normal and negative ACLs, but we are only discuss (and recommend you only use) the normal ACLs.

You can use the command fs listacl to see the ACLs belonging to a directory. You can use the fs setacl command to set ACLs on a directory.

See also:

B

Backfilling of jobs

Backfilling is a technique used by the scheduler to increase the utilization of the cluster. When enabled, it allows the scheduler to run smaller, shorter jobs from further down the queue when the job at the top of the queue is waiting for resources, provided that the smaller, shorter jobs are expected to complete before the additional resources needed by the larger job at the top of the queue are expected to become available.

For example, consider a small cluster with ten nodes (nodes node-1 to node-10), and five of the nodes (node-1 to node-5) are currently in use by jobs which have another 12 hours of walltime remaining, and the remaining five nodes (node-6 to node-10)
are idle. At the top of the queue is a job (job-1) that needs 8 nodes --- since only five nodes are idle, it is currently pending and the scheduler estimates that it will remain pending for another 12 hours (when some of the jobs currently running finish and at least three of nodes #1 to #5 are expected to become idle). Next in the queue are a bunch of small jobs (jobs #2-6) that only need one node each for no more than 2 hours.

Without backfilling, the scheduler would hold nodes node-6 through node-10 for job #1, and wait until at least three of the nodes node-1 through node-5 become idle (which will happen when the jobs running on them finish), and then allocate those nodes to job-1 and start that job. Only after that happens will the scheduler consider starting job-2 and the others. Since it is expected take about 12 hours for the jobs on node-1 through node-5 to finish, nodes node-6 through node-10 will remain idle for about 12 hours, and jobs job-2 through job-6 will not even be considered for running for all that time.

With backfill, the scheduler sees that jobs job-2 through job-6 are small, and that if it started them on the nodes node-6 through node-10, they could all run and finish within 2 hours (since the scheduler will kill them if they run past 2 hours), and so running those jobs should not delay the start of job-1. So it will start those jobs on the nodes it was reserving for job-1, and job-1 should still start in about 12 hours, but jobs job-2 through job-6 will already have been completed. This allows for more efficient use of resources.

Of course, in reality things are not quite so black and white.
The scheduler knows the walltimes requested by jobs, but a job can finish before its walltime is up, and indeed they usually will (since the scheduler will terminate jobs if they exceed their requested walltimes, users are encouraged to provide a bit of padding in their walltime estimates to ensure the job will finish before then). However, if the walltimes provided by the users are somewhat reasonable estimates of the actual runtime, even if padded by 20 or 30% for safety, this backfill process can help keep the queue moving without introducing much delay into the larger jobs.

BeeGFS file system

BeeGFS is a high-performance, parallel file system (HPFS).
It is used to provide the scratch tier of storage on the Zaratan high-performance computing cluster

See also:

BeeOND: BeeGFS On Demand

BeeOND is an extension to the BeeGFS filesystem which enables one to create temporary shared filesystems from local drives on a node. This is useful in certain cases, e.g. when a job larger than a single node needs to do parallel I/O, or needs a temporary filesystem larger than the node-local /tmp filesystem.

See also:

Bit

A bit (from binary digit) is the smallest unit of information in a classical computer system. It can hold one of two values, typically denoted as 0 or 1 (although other labels like true/false or on/off are equally valid). As such, a bit can be represented by a simple switch which can be in either an on or off position, which in the most elementary sense forms the basis of digital electronics. Although a single bit can only hold either a 0 or 1, it is possible to string multiple bits together to hold arbitrarily large numbers (limited only by the number of bits); a common example is the byte.

See also:

  • Glossary entry for byte
  • Glossary entry for qubit
  • Wikipedia entry for bit
Byte

A byte byte is a collection of 8 bits, and as such it can take on any of 2^8 distinct values, typically expressed as the integers between 0 and 255. Bytes are also commonly used to represent a single character in many language sets, or the instructions in the machine level language for computer codes, etc. Bytes are typically the smallest addressable unit of information, and as such are often used as the unit by which digital information is quantified; i.e. the size of memory used by a program or job, or the size of files stored on disk, etc. --- since a byte is a rather small unit by today's standards, this typically is used with a metric multiplier prefix like k (1000x), M (1,000,000x), G(10^9x), T(10^12x), etc.

See also:

C

Checkpointing

Checkpointing is the process by which a program or job periodically saves its state to disk so that if the code is terminated for some reason, the state information can be used to resume the calculation from when the checkpoint was made.
This is commonly used to provide some resilience to jobs, especially long jobs, from hardware and other failures --- when the job is resubmitted, it can (with appropriate configuration) resume from the last checkpoint rather than from the very beginning.

See also:

Command Line Interface (CLI)

A command line interface or CLI is an interface for interacting with computer systems and/or programs based on textual commands typically typed on a computer keyboard.
This is in contrast to a graphical user interface (GUI) which uses a mouse or similar device to interact with the application.

CLIs can be easily scripted, often by simply copying the commands that would normally be typed into a simple text file. This makes the command line interface a great way to submit batch jobs, etc. to the HPC cluster. Such scripting is quite useful in many production analyses, as such often require the submission of many rather similar calculations, which can often be facilitated by copying a script file and making small edits (or even generating the script file itself from another script). This approach is often less tedious, more reproducible, and less error-prone than repeating a sequence of mouse actions as would typically be required in a GUI situation. For these reasons, CLIs are generally recommended for HPC work.

See also:

compute nodes

A compute node is a node on the cluster which is intended for heavy computation. These are the "work horses" of the cluster, where all the real computation gets done and most of the nodes fall into this category. These nodes can be further specialized into compute nodes with specific features, like nodes with large amounts of RAM (large memory nodes) or nodes which have GPUs or other specialized hardware accelerators(e.g. GPU nodes).

On Zaratan, only the login nodes and data transfer nodes are connected to the public IPv4 internet. In particular, compute nodes are not connected to the public internet.

Constraint
See Feature
Controlled Unclassified Information (CUI)
Controlled Unclassified Information (CUI) is an unmbrella term to describe data which is not classified by the US government but which still has some data sensitivity which requires it to safeguarded. This encompasses various categories of sensitive data which is not formally classified, including personally identifiable information (PII), medical records and similar.
core

A computer core is the basic unit for processing on a computer processor[#cpu] or GPU. Each core is capable of performing an independent calculation at the same time, so in the high performance computing paradigm we typically want to parallelize code with each parallel thread of execution running on a separate core to gain maximum performance.

From a software perspective, each core looks much like a CPU, and the cores are sometimes referred to as CPUs, especially at the level of Slurm commands and such. We will try to use the terms core or CPU core instead of CPU when refering to cores, and will use socket when referring to the actual CPU chip/socket.

See also:

CPU

The acronym CPU stands for the "central processing unit" of the system, traditionally the "brain" behind the computer.
It is where most of the processing traditionally occurs.

The CPU consists of one or more cores (modern CPUs almost always consist of multiple cores, the CPUs on the Zaratan standard compute nodes have 64 cores per CPU socket). The CPU is the physical "chip"; many nodes have multiple CPUs. The CPU is sometimes referred to as a "socket" to more clearly distinguish it from the cores --- from a software perspective the cores look like CPUs, and some places use the term CPU to refer to cores, so the socket terminology is useful. On this web site, we will try to always use the term CPU socket to refer to the chip/socket, and refer to cores as cores or CPU cores. Note that the Slurm scheduler and its utilities (e.g. sbatch, etc. ) tend to use the term CPU to refer to a core, and uses the term socket when refering to the physical processor.

See also:

CUDA

CUDA is a platform for parallel computing, primarily used for doing general purpose computing on GPUs. CUDA can be accessed via specific libraries, the use of compiler directives like OpenACC, and the specialized nvcc compiler.

See also:

CUDA compute capability

Different GPUs support different features and command sets. Generally this is a cumulative hierarchy, so later GPUs support all of the features of earlier GPUs, and have some additional capabilities not supported by the earlier ones. a This hierarchy is encapsulated as the CUDA compute capability, which is a simple version number which specifies the features supported by a given GPU and CUDA driver library.
E.g., double precision floating point operations were introduced in CUDA compute capability 1.3; so GPUs supporting only 1.0, 1.1, or 1.2 do not support double precision, but GPUs supporting 1.3 or higher do.

The following list gives the CUDA compute capability for GPUs on the various UMD clusters:

- K20 GPUs: 3.5 ([Deepthought2 cluster](/home/clusters/deepthought2.md) [**RETIRED**])
- K80 GPUs: 3.7 ([Bluecrab cluster](/home/clusters/bluecrab.md) [**RETIRED**])
- P100 GPUs: 6.0 ([Bluecrab](/home/clusters/bluecrab.md) and [Juggernaut](/home/clusters/juggernaut.md) clusters [**RETIRED**])
- V100 GPUs: 7.0 ([Zaratan cluster](/home/clusters/zaratan.md))
- A100 GPUs: 8.0 ([Zaratan cluster](/home/clusters/zaratan.md))
- H100 GPUs: 9.0 ([Zaratan cluster](/home/clusters/zaratan.md))

See also:

D

data transfer nodes (DTNs)
These are nodes optimized for the transferring of the large amounts of data, both data sets needed for jobs running on the cluster, and moving results of finished jobs off the cluster.
Distributed Memory Parallelism

Distributed memory parallelism is a paradigm for parallel computing in which the parallel processes do not all share a global memory address space. Because of this, communication among the different processes cannot occur only over shared memory. This is in contrast with shared memory parallelism which can use the shared memory space for inter-process communication.

Distributed memory parallelism is typically implemented by using the Message Passing Interface (MPI) for communication between the tasks. Although programming using MPI is generally harder than using shared memory paradigms like OpenMP, it has the advantage that it is not restricted to a single node.

E

Embarrassingly parallel

A workload or problem is embarrassingly parallel (also referred to a perfectly, delightfully, or pleasingly parallel) is a workload such that there is no or only minimal communication required between the parallel tasks. Because of this, such workloads can be parallelized with only mimimal effort.

Grid searches of a parameter space are often a common case of an embarrassingly parallel problem. Consider a case wherein you have an N dimensional grid of parameters, and you need to evaluate some (potentially complicated) function for the parameters at each point in the grid. Typically the evaluation at any point in the grid is independent of any other evaluation, and these evaluations can be perofrmed as a large number of sequential jobs. The sequential jobs can be sequentially or simultaneously or in some combination of the two.

See also:

Exclusive mode

Exclusive mode is a an option with the Slurm scheduler in which a job will be granted exclusive access to any and all nodes on which it is running. Normally, jobs will be given an allotment of resources (CPU cores, memory, GPUs on one or more nodes, and the resources allotted to the job will be reserved for that job, but other jobs (from either the same or different users) might be allotted other resources on the same nodes. In exclusive mode, no other jobs are allowed on the nodes on which the job running in exclusive mode is running, and all of the resources on those nodes are allotted to the exclusive mode job (and your allocation account will be charged for such).

We generally recommend against using exclusive mode, as in most cases it is wasteful of both HPC resources and the SUs in your allocation.

F

Feature/constraint

The Slurm scheduler uses the terms feature or constraint to refer to non-consumable aspects of compute nodes; on the UMD DIT-maintained HPC clusters this is mainly used for inherent properyies of the node which do not get consumed, like the CPU type, the presence of infiniband adapters, or special storage devices like NVMe drives. This is in contrast to GRESes which do get consumed.

See also: - Entry for GRES - List of features on the Zaratan cluster - Requesting features in sbatch

File permissions

The standard, traditional POSIX file permissions is used on Linux and other Unix-like file systems. It breaks users into three classes, with an user belonging to the first group found in the following order:

  • user (u): the owner of the file
  • group (g): members of the group owning the file
  • other (o): all other users

and three classes of operations:

  • read (r): the ability to read a file. On a directory, it means the ability to read the names of files inside the directory, but no other information (e.g. contents, file type, size, permissions, etc.)
  • write (w): the ability to modify a file. Although it does not allow for the deletion of the file, it does allow one to remove all of the content of the file. For a directory, this allows (if the execute permission is also set) the creation, deletion, or renaming of any files in the directory.
  • execute (x): the ability to execute a file. It must be set for executable files in order to run them from the command line. For a directory, this is interpretted as search permission, the ability to access the file contents and metadata if the name is known, but not to list the names of files in the directory (unless read permission is also set.

The file permission consists of three sets of three permission flags; one set for each of the user classes user, group, and other, in that order, with flags for the three operations rwx. This is usually represented by a 9 character string, with the letters 'r', 'w', or 'x' when the permission flag is set, or '-' if the permission is not set. (These are sometimes expressed as three octal digits, each digit representing one set, with the digit representing the three digit binary number with '-' being treating as 0 and any of r,w, or x being treated as 1.)

So a file giving the user full permissions and no permission to anyone else would have the permission string rwx------ (or 700 in octal). If the the everyone (user, group and other) have read and execute permissions only, it would be r-xr-xr-x (or 555). If the user and group both have read and write permission, and others only have read permission, the string would be rw-rw-r-- (or 664).

While this scheme works in many cases (especially when the group is reasonably defined), it does have limitations. These limitations can generally be overcome with the use of ACLs.

See alse:

FLOPS

FLOPS, short for FLoating-point OPerations per Second is a measurement of computer performance, and as the name suggests, basically measures the maximum number of calculations using floating point arithmetic that a computer can perform per second. This is the measure commonly used to compare the performance of HPC clusters, except that we usually talk in terms of teraFLOPS (1 TFLOPS = 10^12 FLOPS) or petaFLOPS (1 PFLOPS = 10^15 FLOPS) or even exaFLOPS (1 EFLOPS = 10^18 FLOPS).

See also:

  • Wikipedia page for FLOPS

G

Gibibyte (GiB)

This is the binary unit corresponding to the decimal unit gigabyte, with 1 GiB = 2^10 MiB = 1024 MiB = 2^30 B = 1024^3 B = 1,073,741,824 B = 1.073 GB. This is approximately the size of half an hour of video.

See also:

Gigabyte (GB)

This is the decimal unit corresponding to one billion (10^9) bytes. This is approximately the size of half an hour of video.

See also:

Graphics Processing Unit (GPU)

Graphics Processing Units (GPUs) are hardware accelerators which were initially designed to facilitate the creation of graphics for output to a display device. However, their highly parallel structure makes them more efficient then general purpose CPUs for certain types of algorithms, and so they are quite useful for processing certain "number crunching" workloads.

High-end GPUs can be expensive, and are rapidly evolving, so only a small subset of nodes typically have GPUs, and if you wish to use them you will need to instruct the scheduler to use those nodes Although a GPU contains a large number of cores, they are not compatible with the standard Intel x86 architecture, and codes need to written (and compiled) especially for these devices, typically using the CUDA or OpenCL](#opencl) platforms.
Some of the applications in the standard Research Software Library (RSWL) support GPUs, but even in those cases you need to use the versions which specifically support CUDA.

See also:

Generic resource (GRES)

The Slurm scheduler uses the term GRES (short for Generic RESource) for resources other than CPU cores, nodes, and memory.
Unlike features, GRESes are consumable resources.

Currently on the DIT managed HPC clusters, the only GRES of note are GPUs.

See also: - Entry on features - Entry on GPU - GRESes available on UMD clusters: - [GRESes on Zaratan]](/home/cluster/zaratan.md#zt-greses) - Requesting GRESes in sbatch

Graphical User Interface (GUI)

A graphical user interface or GUI is an interface for interacting with computer systems and/or programs with a mouse or other "pointing" device and graphical icons, etc.
This is in contrast to command line interfaces (CLI) in which the interface is primarily through textual commands typed in via a keyboard, etc.

While GUIs are often easier to use, at least for new users and/or infrequently exceuted procedures, they are not generally the best way to handle production calculations, in which one typically submits many very similar calculations. In such cases, use of a command line interface, typically in conjunction with scripting (which CLIs readily support) yield greater throughput, increased reproducibility, and reduced errors. Also, CLIs generally have less processing overhead than GUIs, which are often somewhat resource intensive. For these reasons, HPC environments are usually geared mainly towards CLI usage.

There are a few exceptions to this rule, and one of the largest is visualization. Humans can often process large amounts of data most readily when such data is presented in a graphical format, e.g. charts, graphs, etc., and GUIs are much, much better at presenting such than CLIs (although can often produce graphical output to a file, they generally cannot display such output in a good way).

On the UMD HPC clusters, the best way for running GUI versions of applications is through the Interactive apps of the OnDemand Portal, especially the Interactive desktop app.

See also:

H

HDF5
Hierarchical Data Format is a file format designed to store and organize large amounts of data. For more information, see here.
high-performance storage system (HPFS) (aka scratch filesystem)

A high-performance storage system is a disk-like storage system designed to support sustained high level of input/output operations from many processes across many nodes. This is generally achieved by spreading data across many file servers.
In particular, with appropriate settings, the data for a single file may be striped across multiple file servers, thereby potentially allowing the many different tasks of a large parallel job to access different parts of the same file without overwhelming a single file server.
For this reason, HPFS systems are ofter referred to as parallel file systems.

The large number of file servers and high performance makes the HPFS tier of storage more expensive than less performant tiers. As such, HPFS tiers are often considered scratch storage, as they are only intended for the storage of data (either input to or output from) jobs that are currently running or that would be running in the near future. Typically, input data should be downloaded to the HPFS (or copied from medium term or longer tiered storage), the job submitted, the job runs to completion, and then the inputs are deleted (or returned to longer term storage tiers), temporary files from the job are deleted, and precious output is moved to longer term storage. It is the policy on UMD HPC systems that scratch space is only for data related to active jobs on the cluster, i.e. jobs that are running, in the queue, that completed recently, or that you plan to submit in the near future. Please delete all unneeded files in a timely fashion, and move and data off the scratch file system when no longer needed for active jobs.

See also:

home directory

Every user on the cluster has a home directory which is the directory into which you are placed when you first log into the system. This directory is not optimized for high performance, so it should not be used for the storage of input or output files for jobs; in addition it has a rather tight (10 GB) quota which also makes it unsuitable for storage of data files. This is a hard limit which will not be increased.

By default, your home directory is private to you; however please note that all data on the UMD DIT maintained HPC clusters are considered to be related to your research and not of a personal nature, and such data is considered to be "owned" by the PIs for the allocations through which you are granted access to the cluster.

Your home directory is the only storage tier which gets automatically backed up by DIT. Therefore it is good for storing small but precious files. You are still encouraged to manually backup precious data elsewhere as well.

See also:

hybrid parallelism

Hybrid parallelism is a paradigm for parallel computing in which some of the parallelism is done using distributed memory parallelism for some of the parallelism, and shared-memory parallelism for the rest. This is commonly done by using MPI for the distributed-memory type, with each MPI task being multithreaded.

This technique is not common, but typically occurs when the problem can be decomposed for parallelization in two distinct manners, with the code using one paradigm for one decomposition and the other for the other.

Even for codes which support hybrid parallelization, then benefits and the performance gains might be highly dependent on the problem being solved. So it is advisable to benchmark to find the optimal configuration.

See also

High Performance Computing (HPC)

High Performance Computing is a general term for computing with a high level of performance. Generally high performance computing specifically refers to running jobs which are very parallel, often running on hundreds or even thousands of cores.

However, then term is often used more generally to encompass not only traditional high performance computing but also high-throughput computing and various machine learning paradigms. The UMD HPC clusters are generally designed for High Performance Computing, and are referred to as such, but they also support High Throughput Computing as well.

See also:

High Throughput Computing (HTC)

High Throughput Computing is a general term for computing with a high level of throughput, that is getting a large number of calculations performed over the course of days or weeks. Whereas high performance computing is more focussed on the use of computer resources to solve complicated problems, often demanding the use of thousands or more CPU cores across many nodes, high throughput computing typically involves the execution of hundreds or thousands of distinct, smaller jobs.

Although the UMD HPC clusters are primarily designed for high performance computing, they support high throughput computing as well.

See also:

I

Infiniband

Infiniband is a data interconnect used by many high performance computing (HPC) clusters because of its high bandwidth and low latency. These factors are important in HPC systems because the different tasks in an MPI code typically need to exchange significant amounts of data with each other. While more traditional interconnects like ethernet can sometimes be competitive on bandwidth, they typically have significantly more latency, which can degrade performance in many MPI based codes.

See also:

Intel MPI
Intel MPI is an implementation of the MPI library provided by the Intel Corporation, primarily intended for use with their CPUs.

J

Job

A job is a basic unit of execution in the HPC environment. Basically, when you execute a sbatch command (or use the OnDemand portal's Job Composer or Interactive Applications) you create a job in the scheduler.

Jobs have associated with them a number of directives to the scheduler; these include specifications of the resources the job should be allocated (the number of CPU cores, nodes, amount of memory, the number and type of GPUs, etc.), the maximum amount of walltime the resources can be allocated to the job, accounting information (e.g. who submitted the job, and which allocation account it should charge against), and possibly other instructions about the scheduling of the job (e.g. instructing the scheduler not to run the job until another job completes, etc.), and perhaps most importantly of all, a list of commands (or a job script) which is to be run for the job.

Every job is assigned an unique numerical ID; this is how the job is identified within the scheduler, and this ID should be provided to HPC staff if you are requesting help or assistance with a job. The scheduler places the job in a queue until the required resources to run the job are available, at which point it starts the job by running the specified job script (or other commands; e.g. typically a shell for interactive jobs).

Most jobs on the cluster are batch jobs, i.e. a script which runs without any user interaction once the job is started. The OnDemand Interactive Applications and other interactive jobs behave similarly, except that they start up an interactive environment (e.g. a desktop, possibly running some command, or an interactive shell prompt) when the job is started.

Job Array

A job array is mechanism by which one can create many jobs using a single sbatch command. You can create a job array with the --array or -a flag to sbatch, followed by an argument specifying the indices for the array; this indices argument can be a list of non-negative integers separated by commas. You can also give ranges of the form i-j or i-j:k; the former is equivalent to listing all of the integers from i to j (inclusive); the latter is similar but uses k as the interval in the list, e.g. i, i+k, i+2k, ... etc until i+nk is greater then j.

All jobs in the job array will request the same amount of resources (number of CPU cores and nodes, amount of memory, number and type of GPUs, and walltime) and will run the same script. When the job starts, the scheduler will set the environmental variable SLURM_ARRAY_TASK_ID to the index value for that particular job, and you can use that inside your job script to have each index perform a different calculation.

While the ability to submit a large number of jobs with a single sbatch command is a small convenience to the user, it is of great benefit to the scheduler when the number of jobs is large. With the job array mechanism, there is only a single record for the job array inside of the scheduler until one of the indexed jobs starts to run. At that point, the indexed job will receive its own job number and be tracked as a separate entity by the scheduler. But until then, all of the pending jobs inside the job array are treated as a single entity, which greatly improves the performance of the scheduler (i.e. it only needs to check if the job array jobs can run once, rather than doing such perhaps hundreds or thousands of times if you submitted hundreds or thousands of jobs without using a job array).

See also:

  • Section on [submitting job arrays]
Job Priority

Every job submitted to the scheduler has an associated priority which determines the order in which the scheduler will try to run them. While broadly speaking jobs are run in a first-come, first-served order, there are exceptions. Jobs submitted to the debug partition get the highest priority (i.e. the scheduler tries to run those first) --- these jobs have highly restrictive time limits and a small dedicated pool of nodes, so this generally allows users to get these jobs started quickly, to promote rapid debugging of issues.

Most remaining jobs are run at standard priority --- these comprise the bulk of the jobs on the cluster. Basically, unless you specifically request otherwise (by either submitting to the debug partition above, or the scavenger partition below), the job will run at standard priority. Typically jobs submitted at this priority run in a first-come/first-serve fashion. However, if the next job X in line to run is waiting on resources, the scheduler may "backfill" some smaller jobs onto resources reserved for the job X if such jobs would finish before the remaining resources needed to run job X are expected to be ready.

Jobs submitted to the scavenger partition run at the lowest priority, and are subject to preemption, but to compensate for that such jobs do not incur charges to the allocation account.

See also:

Job Step

A job step is the smallest unit of execution within a job in Slurm. A job consists of at least one, typically 3 or 4, job steps. Job steps will have the same job number as the main job, with an suffix consisting of a dot and a string or number, e.g. 11970454.batch, 11970454.extern or 11970454.0.

Job steps are most often seen when using the sacct command, and as they are usually not very interesting, you can supress their display with the "-X" flag to sacct.

K

Kerberos

Kerberos is an authentication system which makes use of tickets to grant access to computer resources on the network. Typcially, you will automatically get Kerberos tickets when you enter your password to log into a Kerberos enabled system. These tickets have expiration dates (typically on the order of a day), although they can be renewed. Kerberos tickets form the basis for obtaining tokens to access the Auristor file system which is used for the SHELL medium term storage tier.

If you have (or install) the Kerberos client utilities on your workstation, you can then configure ssh for passwordless ssh to the cluster using Kerberos. With this setup, once a day or so you issue the kinit command on your workstation (which will prompt you for your campus password), after which you can ssh to the HPC system and other DIT maintained systems without needing to enter your password again (until your Kerberos tickets expire). For best results, you will want to use the campus VPN to avoid needing to perform multifactor authentication.

See also:

kibibyte (kiB)

A kibibyte is 1024 bytes, as distinct from a kilobyte (kB) which is 1000 bytes. Because certain parts of computer hardware (e.g. memory) typically tend to favor numbers which are powers of 2, there is a need for binary units like kibibytes. In the early days of personal computers, usage was sloppy, and many used the term kilobyte to refer to 1024 bytes (even though the metric kilo- prefix means 1000x), but this became problematic as sizes increased, with each successive prefix adding about 2% to the difference between purely decimal and purely binary units. Eventually the s tandards committees settled on this system of binary units.

A kibibyte is about the size of the text of Jabberwocky or a small icon image.

See also:

kilobyte (kB)

A kilobyte is 1000 bytes. Sometimes this term is used for 1024 (2^10) bytes, but that is more properly referred to as a kibibyte (kiB) as the metric prefix k is meant to be a multiplier of exactly 1000.

A kilobyte is about the size of the text of Jabberwocky or a small icon image.

See also:

kSU

One kSU is 1000 SU

See also:

L

local temporary storage
See node-local temporary storage
Login node

These are the nodes you can actually log into to submit jobs, edit files, monitor your jobs, view the results of jobs, etc. They are not intended to do number crunching or real computation. If the cluster has distinct DTNs, then most file transfers (certainly all transfer of large amounts of data) should be done on the DTNs; usually you will need to use . You should not run any computationally intensive processes on the login nodes.

On Zaratan, only the login nodes and data transfer nodes are connected to the public internet. In particular, compute nodes are not connected to the public internet.

Login shell

The login shell is the Unix shell which is run for you when you first log into the system.

See also:

Lustre

Lustre is a high-performance, parallel file system. It was used to provide the scratch tier storage resource on the Deepthought2 high-performance computing cluster, as well as for the Juggernaut cluster.

See also:

M

** Machine Learning**

Machine learning is a field within artificial intelligence which is concerned with the study (and development of) algorithms which can "learn" (i.e. detect patterns) from training data and later apply that "learning" on new, unseen data. The subfields of deep learning and neural networks have made great strides in recent years.

Although recent models have proven quite capable, they also can consume large amounts of computing power, and tend to benefit greatly from the processing power of GPUs.

See also:

Mebibyte (MiB)

This is the binary unit corresponding to the megabyte, with 1 MiB = 2^10 kiB = 1024 kiB = 2^20 B = 1024^2 B = 1,048,576 B. This is approximately the size of the content of a typical Harry Potter novel.

See also:

Medium-term storage (SHELL storage tier)

The SHELL storage tier is a medium-term storage tier provided on the Zaratan cluster Unlike the scratch tier, data on the medium-term storage cluster is not required to pertain to an "active" job (i.e. a job which is running, queued, or to be run in the near future). However, it is not long-term or archival storage --- the storage it not automatically backed up by DIT, nor are there guarantees that the storage will last beyond the lifetime of the Zaratan cluster (5 years or so). It is intended for the storage of large amounts of data related to active research on the cluster, even if the data is not related to jobs which are running or will be run in the near future.

Please note that the SHELL tier is not designed to the same level of performance as the HPFS/scratch tier. To prevent users from overloading the small number of fileservers forming this tier by having many tasks from large parallel jobs do large amounts of I/O to it, the SHELL tier is not accessible from the compute nodes. It is accessible from the login and the and data transfer nodes. Furthermore, since the underyling filesystem is the Auristor file system, it can be securely accessed with appropriate credentials from any system with the appropriate file client, even systems outside cluster.

Megabyte (MB)

A megabyte is 10^6 (one million) bytes. Sometimes this term is used for 1024^2 (2^20) bytes, but such is more properly referred to as a mebibyte, as the metric prefix M is meant to be a multiplier of exactly 10^6. One megabyte is approximately the size of the content of a typical Harry Potter novel.

See also:

Memory

Just like people, when talking about computers memory is the means by which computers and computer programs keep track of things for future use. With computer systems, the term memory typically refers to RAM, which is a specific type/family of memory chips which provide relatively high speed access. Although both memory and disk storage are usually measured in bytes, disk storage is typically not considered memory (just like one normally does not refer to notes in a notebook as a person's memory).

Note that GPUs typically have memory (VRAM) which is distinct from the main system (CPU) memory, usually an order of magnitude or so smaller.

Message Passing Interface (MPI)

The Message Passing Interface (MPI) is a standardized and portable standard for communication between tasks in a distributed memory parallel job. It has become a de-facto standard for programs which can run across multiple nodes.

There are a number of implementations of this interface. UMD HPC clusters typically provide:

  • For GNU compilers, OpenMPI: an open-source MPI implementation
  • For Intel compilers, Intel MPI: Intel's implementation of MPI

Both implementations include wrappers for C, C++, and Fortran compilers, as well as bindings for many scripting languages including Python and R.

See also:

MPI Communicator

MPI jobs group the various processes which comprise the job into objects called communicators. The default communicator is MPI_COMM_WORLD which consists of all of the processes that were in the job when it first started. For many jobs, all of the processes are closely coupled, and that is the only communicator you need, but MPI provides other mechanisms which are needed in some more complicated cases.

Each communicator has a size, which is the number of processes in the communicator. Every process in the communicator has an unique rank, which is an integer between 0 and the size - 1 of the communicator.

See also:

MSU

One MSU is one million (10^6) SU

See also:

Multifactor authentication (MFA)

Multifactor authentication is a process by which one needs more than just a password to access a computer system. Although passwords have a long history of use for authenticating users to computer systems, they can be compromised either through lack of complexity or by being stolen. Multifactor authentication strengthens the security be requiring on two or more distinct authentication mechanisms or factors to be passed before allowing access to the system. This is commonly referred to as the "something you know" (e.g. a password) and "something you have" (e.g. your cellphone with the appropriate MFA client) authentication process.

At UMD we require multifactor authentication to access the cluster. Since the campus VPN requires multifactor authentication, if you are connecting to the cluster via the VPN, your VPN connection is treated as one of the authentication factors, and you only need to provide your password or equivalent for the second factor in order to access to the HPC cluster. If you try to ssh to the cluster without using the VPN, you will be prompted for
a MFA option. Typically your choices will be to:

  • Do a "push" to your smart phone, opening an app to verify it is you.
  • Get a phone call to any of the phone numbers registered to your account

The various web portals, etc. on the cluster require you to use the campus VPN in order to connect.

See also:

Multi-instance GPU (MIG)

Modern GPUs represent a significant amount of computational power, and sometimes jobs cannot fully utilize a full GPU. However, a single GPU cannot be split between jobs, so in this situation the GPU goes underutilized. Certain models of NVidia GPUs support something called multi-instance GPU which allows one physical GPU to be split into multiple, less powerful, virtual GPUs. These virtual GPUs can each be assigned to different jobs, thereby allowing multiple jobs to share one of these powerful GPUs, increasing utilitization of the device.

The A100 and H100 GPUs both support this feature, and each physical GPU can be split into up to 7 smaller virtual devices. We have currently divided the A100s on one of the nodes into seven smaller virtual units, called a100_1g.5gb which is basically about 1/8-1/7 of a full GPU, with 5 GB of GPU RAM, and costing 7 SU/hour/GPU.

N

NetCDF
A set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. For more information, see here.
node

In HPC terminology, we use the term node to refer to a complete computer system, comprising of at least one CPU, memory, power supply, various buses, networking devices and generally some local disk storage. This is the direct analog to your laptop or workstation.

On the clusters, we have differentiated the nodes into various specializations.

See also:

node-local temporary storage

The compute nodes of the cluster have temporary local storage mounted at /tmp. Your jobs are able to use this space. Note that while every node has a /tmp directory, the directories on distinct nodes are distinct; files written to /tmp on one node will not be visible on another node. Also, all data in /tmp belonging to you will be deleted when your last job on the node finishes, so you will need to copy any data stored under /tmp elsewhere (e.g. scratch).

See also:

NVLink

NVLink is a proprietary high-speed interconnect for connecting GPUs to each other or to CPUs providing faster data transfer and lower latency than traditional PCIe interfaces.

See also:

  • Glossary entry on GPU
  • Wikipedia page on NVLink
NVMe (Non-volatile Memory Express)

NVMe is an interface specification typically used for accessing solid-state drives(SSDs). Although the term technically refers to the interface, it is often used to refer to the drive as well. Traditional SSDs use interfaces designed for spinning disks and which are not able to keep up with throughput of the drive. The NVMe interface supports higher throughput rates, making NVMe drives perform better than traditional SSDs. traditional HDDs.

See also:

O

OpenACC

OpenACC is a programming standard designed for parallel computing on heterogeneous systems including CPUs, GPUs, and possibly other accelerators.

See also:

OpenCL

OpenCL is a programming framework for writing code to execute across heterogeneous platforms including CPUs, GPUs, and possibly and other hardware accelerators. It is an open standard.

See also:

OpenMP

OpenMP is an application programming interface for shared memory parallelism using threads. (Should not be confused with OpenMPI.)

OpenMP is implemented in the compiler, adding some pragmas to the language with which you can instruct the compiler which sections of the code are parallelizable and which must be run sequentially.

See also:

OpenMPI
OpenMPI is an open source implementation of the MPI library.

P

Parallel Computing

Parallel computing is a type of computation wherein many calculations are carried out simultaneously. This contrasts with sequential computing wherein only a single calculation occurs at any given time.

Obviously, a parallel computation should be faster than a sequential computation. The speedup of an algorithm when parallelized is the ratio of the runtime of the sequential code to that of the parallelized code.
Although one might naively expect the speedup (compared to the sequential calculation) of a computation parallelized over N CPU cores would be N, this is rarely the case. Typically, there are parts of an algorithm which cannot take advantage of parallelization, causing the speedup to be less than N. Generally, the addition of CPU cores will start having diminishing returns at some point, depending on the algorithm.

See also:

Parallel File System
Please see High-Performance File System.
Partition

In HPC terminology, a partition is a set of compute nodes. Clusters typically divide their nodes into partitions to reflect different hardware availabilities, or to associate priority and quality of service settings to jobs submitted to the partition.

The term queue is sometimes used synonymously with partition (because in some sense the scheduler maintains a distinct queue for each partition).

See also:

Petabyte (PB)

This is the decimal unit corresponding to one quadrillion (10^15) bytes. This is about the size of 2000 years of MP3-encoded music.

See also:

POSIX Access control lists (ACLs)

The BeeGFS file system used for scratch uses POSIX access control lists (ACLs) to supplement the traditional Unix file permissions.

This ACL scheme uses the same read, write, and execute file permissions as the traditional file permissions, with the same meanings, but allows one to apply them to possibly multiple specified users and groups.

The command getfacl can be used to view the ACLs, and the setfacl command to set ACLs.

Directories have two sets of ACLs, "regular" or file ACLs which applies to the directory itself, and "default" ACLs which are used to default the ACLs of any files in the directory which do not have ACLs, including newly created files.

Access controls consist of three or four colon (:) delimitted fields. In the four field case, the first field should be ACL selector; if omitted (the three field case) the control is a regular access ACL, and if the field is "default" (the four field case), the access control is a default ACL.

The next field (the second field in the 4 field case, or the first in the 3 field case) specifies the entity type. This can be any of:

  • user: This access control is for users
  • group: This access control is for groups
  • mask: This access control is the effective rights mask
  • other: This access control is for the other field in the standard Unix file permissions scheme. In this case the name field should be blank.

The next field is the name of the group or user (depending on the entity type). If blank, which is is only allowed for 'user', 'group', or 'other' entity types, this field corresponds to the 'user', 'group', or 'other' fields of the standard Unix file permissions.

The last field is a triplet giving the file permissions as a string of 'rwx' for set permissions and '-' for unset permissions, just as in the standard case.

See also:

Preemption of Jobs

Most jobs on the cluster are submitted to the scheduler, wait in the queue until resources are available, and then once the job is started, it runs until it either completes, crashes, or is cancelled due to exceeding its requested walltime. Even if a higher priority job is submitted afterwards which "wants" the resources being used by the lower priority job which already started, the running job is allowed to run to its successful or unsuccessful completion, and the higher priority job is forced to wait until it finishes (or other resources become available).

An exception to this is jobs submitted to the scavenger partition. These jobs, in addition to be the lowest priority, are also preemptible. This means that even after the scavenger job starts, if another, higher priority job (and every non-scavenger job is higher priority) "wants" the resources being used by the scavenger job, the higher priority job can kick the scavenger job off of the node(s) it is using, and take those node(s) for its own use. This is called preemption. Typically, the preempted scavenger job is put back in the queue, and will eventually start again (it is advisable that such scavenger jobs do checkpointing so that they do not start over from the very beginning but can make some progress towards completion in these intermittent runs).

Again, normal jobs will not get preempted, only scavenger jobs. Scavenger jobs are subject to preemption, but your allocation account does not get charged for scavenger jobs.

See also:

Priority of Jobs
Please see job priority.
Process

A process is an instance of a computing program that is being executed by one or more threads. In addition to the one or more threads belonging to a process, the process also contains the instructions found in the program's code and any resources needed (e.g. intermediate data, file handles, etc.). For a fully sequential application, the application runs in a single thread of a single process, although that thread might spawn off subprocesses (i.e. new child processes with the original process as the parent).

Multithreaded processes will consist of more than one thread. Depending on the resources allocated to the process, these threads might or might not be running in parallel on separate CPU cores or they might be running sequentially on a single core. Typically in an HPC environment, one wants a separate core allocated for each thread so that there is maximal parallelization.

In MPI codes, each MPI task is a represented by a single process, usually with a single thread but possibly with multiple threads in the hybrid case.

See also:

Project

A project is a collection of allocations for a research group, i.e. under a single principal investigator (PI). In theory, a single project can have allocations across multiple clusters. Usually a faculty member will have a single project; although exceptions exist. If the faculty member has requested use of the cluster for a class he/she is teaching, then that class will be in a separate project. Also, faculty members who manage college or departmental pools of HPC resources will see that as a distinct project.

Every user of the cluster is associated with one or more projects, and also with at least one of the allocation accounts for the cluster beneath the project.

Projects usually belong to research groups, although sometimes they belong to a department or college. They provide a means of organizing users and associating them with allocation accounts. Members of the same project also share membership in an Unix group, which can be used to facilitate sharing files (although by default your home and data directories are restricted to your login account only, they are generally group-owned by the project group, and you can use chmod to allow group members to read).

See also:

Public key authentication
See SSH public key authentication

Q

Quarter (of a year)
This represents one fourth of a year. Some resources are meted out on a quarterly basis. This is generally to encourage the spreading out of the use of the resource over time. Quarters are labelled Q1, Q2, Q3, and Q4, starting on 1 Jan, 1 Apr, 1 Jul, and 1 Oct, respectively.
Qubit

A qubit (from quantum bit) is the smallest unit of information in a quantum computer system. When observed/read out at the end of a calculation, it returns a binary value (typically denoted as 0 or 1) just like a classical bit, but during processing it can be a coherent superposition of both of those states simultaneously, thereby containing much more information.

See also:

  • Glossary entry for bit
  • Wikipedia entry for qubit
Queue

In Britain, a queue is a line of people, etc. waiting their turn for something. This definition is borrowed in HPC to refer to the list of jobs that are waiting to be allocated resources and started by the scheduler.

When you submit a job to the cluster and then check its status, unless you are lucky enough to have submitted it when the cluster is very idle, you will see that the job is in a "pending" state --- i.e. the job is waiting in the queue.

The amount of time a job spends in the queue depends on how busy the cluster is, how many other jobs are in the queue, how many resources your job is requesting and for how long, the priority of the job, and many other factors.
A sizable production job requesting a day or more of runtime can easily spend a couple of hours in the queue. Very short jobs (under 15 minutes) should look into the debug partition which generally has much lesser wait times.

In some sense, the different partitions each have their own queue, and so sometimes the term queue is also used to refer to a partition. Some schedulers tend to use the term queue in that fashion more than others.

See also:

R

RAM

RAM is an acronym for Random-access memory. This is particular form of computer memory which can be both read and written, and for which the access times are basically irrespective of the location of the data within the memory. Most typically, this takes the form of dynamic random access memory (DRAM), which is "volatile", meaning that the stored information is lost if power is removed from the chip.

Without further qualification, RAM typically refers to the memory used by the CPUs on a node. GPUs typically have their own internal memory generally referred to as VRAM.

RAM is typically where the machine code for the program typically resides when it is being run, along with all of the data that the program is actiing on, although in both cases this can be supplemented by disk storage as well. For static data like the code, the system just overwrites the memory storing the code when needed, and when the code is needed reads it back from the program file on disk. For general data, the program can write out chunks of memory to a swap area on the disk, overwrite the memory thus freed for a while, and read the memory back from disk when needed. In either case (but especially the latter case), this can significantly degrade performance, as disks are much slower than RAM, and so high performance codes typically wish to avoid swapping to disk.

See also:

ROCm

ROCm is a platform for parallel computing, primarily used for doing general purpose computing on AMD GPUs, basically the AMD version of CUDA.

See also:

  • Glossary entry for CUDA
  • Wikipedia entry for ROCm
rsync
The arsync command (short for Remote Sync) is an Unix utility for transferring and/or synchronizing files between two locations, often on two different computer systems (and therefore involving data going across the network). Rsync is useful because it normally does not blindly copy data, but checks the checks if the source and destination files differ (usually based on modification times and/or file sizes, but optionally due to a checksum) and only copies files that differ. This can greatly reduce the amount of data flowing over the wire (and therefore the amount of time for the transfer) especially if one needs to restart an aborted transfer for any reason.

S

Scavenger partition

The scavenger partition is a special partition that allows users to submit very low priority jobs that are subject to preemption, but do not incur any SU charges against the allocation account.

Because scavenger jobs do not incur charges against the allocation account, these are an useful mechanism to run jobs in excess of your SU allotment. But because of the preemption, you are strongly encouraged to make use of checkpointing in order that your job can make progress during its piecemeal run time.

Scheduler

The scheduler is a critical component of an HPC environment. It is a complex piece of code responsible for allocating resources to the various jobs in the queue in a timely fashion.

The scheduler is in some ways analagous to the host or hostess at a restaurant, with the restaurant customers being like jobs, the tables being like the compute resources, and the waiting list being like the queue. If the restaurant is not busy, a new customer can just come in and the host/hostess will seat them almost immediately, just like when the cluster is idle, the scheduler will allocate resources to a new job almost immediately. When things are busy, the host/hostess will place customers on a waiting list, and customers will be seated in roughly the order they c ame in. But there are exceptions --- some parties are too big for some tables or might have special requests (e.g. indoor vs outdoor seating, etc), so sometimes smaller or less particular parties can jump ahead in the queue.

In the same way, the scheduler tries to allocate resources to jobs roughly in the order they came in, but the order can be modified because the jobs have different sizes (from single cores to thousands of cores) and can request GPUs or large amounts of memory. Also, unlike most restaurants, the scheduler will allow in some cases for multiple jobs to be placed on the same table. As an added twist, jobs are required to specify the maximum amount of time for which they will run, and the scheduler can use this to backfill jobs.

There are many different schedulers available for HPC clusters. At UMD, we have been using the Slurm scheduler since 2014. Slurm is an open source product used at many clusters in the world, including most of the TOP500 systems.

See also:

  • Wikipedia entry for Slurm
scp
See Secure copy protocol
Scratch file system
Please see the High-Performance File system
Secure copy protocol (scp)

The secure copy protocol and command (scp) are a protocol and command for transfering files from one computer system to another over the network. The secure in the name comes from the fact that the authentication and the transferred file data are both encrypted over the wire, thereby protecting the data from wire tapping attacks.

See also:

  • Glossary entry for sftp
  • Glossary entry for ssh
Secure file transfer protocol (sftp)

The secure file transfer protocol and the related command (sftp) are a protocol and command for transfering files from one computer system to another over the network. The secure in the name comes from the fact that the authentication and the transferred file data are both encrypted over the wire (actually using the ssh protocol under the covers), thereby protecting the data from wire tapping attacks.

See also:

  • Glossary entry for scp
  • Glossary entry for ssh
Secure Shell Protocol (SSH)

The Secure Shell Protocol is a cryptographic network protocol for communicating between systems. It is also the name of the most common remote login client application ssh using this protocol. Typically one uses the ssh command to open a command line terminal session on a remote system.

Unix and Mac systems typically have a client application named ssh installed by default; Windows systems usually require a ssh client to be installed (e.g. putty).

See also:

  • Glossary entry for scp
Sequential Job or Process

A sequential job or process is a job or process which does not do any form of parallelism, i.e. the code processes one instruction, then the next, and so on in a sequential fashion. These are also referred to as single core jobs or processes, as they can efficiently run on a single CPU core.

See also:

Service node

Service node is a catch-all phrase for any of the nodes in the cluster which provide various services needed to keep the cluster running.
Generally you will never directly run processes on these nodes, but they provide essential services. E.g. the scheduler, file servers, web servers, etc.

See also:

Service Unit (SU)

A service unit (commonly abbreviated SU) is the basic unit used in HPC clusters to measure an amount of computation. It is nominally defined as 1 SU = 1 walltime hour of use of a single CPU core, although this definition can be affected by other factors. On clusters with many different CPU models, an additional factor might be included to account for the different relative computational power of the different CPU models; e.g. often the above formula only holds for the slowest CPU model on the cluster, and more powerful CPU cores will have an additional factor (>1) applied to normalize. E.g., if a typical job that takes one hour on the slower core (and so consumes 1 SU) only takes 30 minutes on the faster core, the faster core might have a factor of 2 added in the above formula (so the SU cost on that core would be twice the number of CPU hours, or 2 * 1 core * 0.5 hour = 1 SU, so the job costs the same number of SUs on either core).

Similarly, the above formula will be tweaked to account for other resources consumed by the job. E.g., memory is a restricted resource, and so a job which is using twice the average memory per CPU core might be charged as if twice as many cores were in use. GPUs are another (very) valuable resource, typically with each GPU having thousands of cores (e.g. an Nvidia A100 GPU has almost 7000 CUDA cores and over 400 Tensor cores) and costing a substantial amount of money to purchase. Although each GPU core is less powerful than a single CPU core, the sheer number of such cores makes the GPUs very powerful for some alorithms. Currently, unlike the situation with CPUs, we are unable to allow multiple jobs to access the same GPU at the same time, so if a job requests a GPU, it gets the entire GPU (and all of its cores and memory) (The a100_1g.5gb are an exception to this, but in this case the single A100 GPU has been split into seven a100_1g.5gb virtual GPUs at a high level, and each a100_1g.5gb GPU appears as a physical GPU to the system. So while different jobs can be assigned to different a100_1g.5gb GPUs which are part of the same A100 GPU, you cannot have multiple jobs on the same a100_1g.5gb GPU at the same time.). We do not charge 1 SU for each GPU core used for 1 hour, but we do charge significantly more than 1 SU for the use of an entire GPU for one hour. The exact factor depends on the type of GPU being used.

Generally, your job will be charged a different hourly SU rate depending on the partition selected:

  • For jobs in the standard partition, the hourly SU rate will be the maximum of:
    • 1 SU/hour times the number of CPU cores allocated to the job
    • 0.25 SU/hour per GiB times the total the amount of memory allocated to the job (in GiB). This corresponds to 1 SU/hour for 4 GiB of RAM, based on the fact that there are 128 cores and 512 GB of RAM on a compute node in the standard partition.
  • For jobs in the debug partition, the hourly SU rate is the same as for the standard partition.
  • For jobs in the serial partition, the hourly SU rate is the same as for the standard partition.
  • For jobs in the bigmem partition, the hourly SU rate will be the maximum of:
    • 1 SU/hour times the number of CPU cores allocated to the job
    • 0.0625 SU/hour per GiB times the total the amount of memory allocated to the job (in GiB). This corresponds to 1 SU/hour for 16 GiB of RAM, based on the fact that there are 128 cores and 2048 GB of RAM on a compute node in the bigmem partition.
  • For jobs in the gpu-h100 partition, the hourly SU rate will be the maximum of:
    • 144 SU/hour for each H100 GPU allocated to the job. This is based on the cost and performance of these GPUs.
    • 6 SU/hour times the number of CPU cores allocated to the job. This means that a job using a single H100 can use up to (144/6)=24 cores without increasing the SU cost, as each compute node in the gpu-h100 partition has 4 H100s and 96 CPU cores, or 24 cores per GPU.
    • 1.125 SU/hour per GiB times the total the amount of CPU memory allocated to the job (in GiB). This corresponds to 144 SU/hour for 128 GiB of RAM, based on the fact that there are 512 GiB of CPU RAM and 4 GPUs per compute node in the gpu-h100 partition.
  • For jobs in the gpu-a100 partition, the hourly SU rate will be the maximum of: - 48 SU/hour for each A100 GPU allocated to the job. This is based on the cost and performance of these GPUs.
    • 1.5 SU/hour times the number of CPU cores allocated to the job. This means that a job using a single A100 can use up to (48/1.5)=32 cores without increasing the SU cost, as each compute node in the gpu-a100 partition has 4 A100s and 128 CPU cores, or 32 cores per GPU.
    • 0.375 SU/hour per GiB times the total the amount of CPU memory allocated to the job (in GiB). This corresponds to 48 SU/hour for 128 GiB of RAM, based on the fact that there are 512 GiB of CPU RAM and 4 GPUs per compute node in the gpu-h100 partition.
  • For jobs in the gpu-a100_1g_5gb partition, the hourly SU rate will be the maximum of: - 7 SU/hour for each a100_1g.5gb GPU allocated to the job. This is based on the fact there we split a full A100 GPU into seven a100_1g.5gb GPUs for this partition.
    • 1.5 SU/hour times the number of CPU cores allocated to the job. This is based on the fact that each node in the gpu-a100_1g_5gb partition has 128 CPU cores and 28 a100_1g.5g GPUs, so the cost of using all 128 CPU cores would be 128*1.5 SU/hour which is approximately the same as using all 28 GPUs (28 * 7 SU/hour).
    • 0.375 SU/hour per GiB times the total the amount of CPU memory allocated to the job (in GiB). This is based on there being 28 GPUs and 512 GiB of CPU RAM on each compute node in this partition.
  • For jobs in the scavenger partition, there is no SU cost.

For example, a job requesting 6 CPU cores, no GPUs, and 8 GB/CPU-core on the standard partition would be charged an hourly rate of 12 SU/hour (in this case, the total memory allocated would be 6 CPU * 8 GB/CPU = 48 GB, and the maximal value in the list above would be 48 GB * 0.25 SU/hr/GB = 12 SU/hour). Similarly, a job requesting two A100 GPUs (so gpu-a100 partition) along with 64 CPU cores and 4 GB/CPU core would be dominated by the GPU costs, and result in 48 SU/hour/GPU * 2 GPU = 96 SU/hour.

This houly rate is multiplied by the actual walltime the job used (in hours) to calculate the SU cost of the job. Note that the actual walltime is used, not the requested walltime; if your job requested 24 hours of walltime but completed in 20 hours, your job is charged for 20 hours, not 24 hours, as the scheduler will release any resources reserved for your job once the job ends. So you are not charged extra for requesting more time than is needed. But you should try not to request an excessive amount of additional time; request what you think the maximum amount of time the job should possibly run, perhaps with a 10-25% buffer. Requesting excessive amounts of walltime can cause your job to spend extra time in the queue due to an inability to backfill your job, and can result in additional SU costs if your job fails in such a way that it does not terminate properly (e.g. if your code is stuck in loop so that it is not actually doing any useful calculation but it does not terminate; in this case the job will only end when the specified walltime is up, and requesting double or triple what is needed will result in the job uselessly burning SUs for that much longer).

Note however that the hourly SU rate is based on the amount of resources allocated to the job, not the resources actually used. E.g., if you submit a single core job requesting 8 GB or RAM and no GPUs, you will be charged for 8 GB of RAM whether your job uses all 8 GB or not --- these resources are reserved for your job (and therefore unavailable to other users) for the duration of the job and so you will be charged for them. In particular, submitting a job in exclusive mode (which disallows any other jobs from sharing the node with your job) will result in your being charged for all the resource on the node whether you are using them or not.

Compared to the amount of computation done on the cluster, an SU is a fairly small unit, so one often talks in terms of kSU or even MSU, where 1 kSU = 1000 SU and 1 MSU = 1,000,000 SU.

See also:

Shared memory parallelism

Shared memory parallelism is a paradigm for parallel computing in which all of the parallel processes share a global, shared memory address space which they can use to communicate with each other. It contrasts with distributed memory parallelism techniques where no memory space is shared among all of the processes.

Shared memory parallelism is typically implemented by threads using OpenMP or Threading Building Blocks.

Note: in order to use this paradigm, all of the processes using shared memory parallelism must be running on the same node, otherwise they would not be able to share memory.

See also:

sftp
See Secure File Transfer Protocol
shebang (Unix)

The Unix and Linux operating systems support many different interpretters and shells to be used for processing script files. Unlike some operating systems that use the extension of the filename to determine which interpretter should be used to process a script file, Unix and Unix-like systems typically use what is called a shebang. This is the very first line of the script file, starting with the characters '#!' (from which the name shebang derives), followed by the path to the interpretter. The leading '#' usually causes the shebang line to be ignored by the interpretter as a comment.

Commonly seen shebang lines include:

  • #!/bin/bash : process the script using the Bourne-again (bash) shell.
    -#!/bin/sh : process the script using the original Bourne (sh) shell. -#!/bin/tcsh : process the script using the TENEX C (tcsh) shell.
  • #!/bin/bash : process the script using the Bourne-again (bash) shell.
  • #!/usr/bin/python3 : process the script using the python interpretter with path /usr/bin/python3
  • #!/usr/bin/python3 : process the script using the python interpretter with path /usr/bin/python3
  • #!/usr/bin/env python : process the script using the python interpretter found in the PATH of the calling process

The last example uses the /usr/bin/env command is an example of a common usage when the absolute path of the interpretter is not known (or might change depending on which system the script is to be run on). In this case, the /usr/bin/env command will find the named interpretter using the PATH settings of the process which invoked the script.

NOTE: The shebang line *must be the very first line of the script in order to work as expected.

See also:

shell (Unix)

The Unix shell is the main command-line interface for interacting with the operating system on Unix and Unix-like systems. It is both an interactive command interface as well as a scripting language. The shell is just a program, and typically there are multiple flavors of shells you can choose from. Common examples are:

  • bash: This is the default shell on Zaratan, as it is both user friendly and provides a very functional programming interface. It is also the default shell for most Linux distributions, and is what most online documentation expects.
  • tcsh: This was the default shell on the old UMD HPC clusters. It is somewhat user friendly, but is less capable than bash for scripting.

See also:

SHELL file system
Please see Medium-term storage
Slurm Workload Manager

The Slurm Workload Manager is job scheduler used on many of the world's top HPC clusters, and is also used on the DIT maintained HPC clusters at UMD.

See also:

Solid-state drive
Solid-state drives(SSDs) are a form of mass storage which use integrated circuits for the actual storage instead of spinning magnetic media. These devices are of interest in HPC applications primarily because they have much higher input and output speeds and much lower latency than traditional spinning media. Solid state drives can either use traditional hard disk drive (HDD) interfaces like SAS or SATA, or they can also use newer interfaces like NVMe which can better support the parallelism available in modern SSDs and therefore even further improve performance. The downside is that currently SSDs are more expensive per storage unit than traditional HDDs.
Speedup

The speedup of a code when parallelized is a measurement of how much performance improvement one obtains when parallelizing a job. It is given by the ratio of the runtime for a sequential version of the algorithm and the parallelized computation.

I.e., if the sequential code takes T_seq seconds, and the parallel code takes T_par seconds, the speedup is given by S = T_seq/T_par.

See also:

SSD
Please see Solid-state drives (SSDs)
ssh
See Secure Shell Protocol
SSH public key authentication

SSH public key authentication is a process by which one can ssh into a system without needing to enter a password. The security for such is based on any of several asymmetric encryption algorithms which have distinct private and public keys. In such cryptographic systems, data encrypted or signed with the private key can only be decrypted or verified with the public key, and vice versa.

For public key authentication, the private key is kept secure on the source system (e.g. your workstation), and the public key is copied to an appropriate location on the destination system (e.g. the HPC cluster).
When you log into the destination system from the source system using public key authentication, the source system will digitally sign a standard message with the private key and send this and the public key to the destination. The destination sshd daemon will check that the public key is auhorized for login purposes for that account, and then validate the signature of the message. If the signature validates, the source ssh client is in possession of the private key, and the destination system grants it access to the system.

Certain software packages (e.g. Ansys require that public key authentication be enabled among the nodes of the HPC cluster.

SSH public key authentication is unable to provide you the Kerberos tickets and AFS tokens needed to access SHELL storage and for some commands, so for (almost) passwordless login to the cluster, we recommend that users setup up Kerberos on their workstations.

See also:

Striping (of a file)

High-performance file systems (like BeeGFS or Lustre typically achieve their high performance by striping the data in a file across multiple file servers, or at least disk arrays on a file server. I.e., different portions or chunks of a file will get written to different disk arrays, and usually to disk arrays on different file servers. Generally, there is a chunk size and a number of stripes set for a file (either by default or specified by the user), and the system will assign the file to a number of disk arrays and/or file servers equal to the number of stripes, and write the first chunk size amount of data to the first such disk array, and when that is done write the second chunk size amount of data to the second, and so forth. After the chunk is written to the last of the assigned disk arrays, things cycle back around to the first again. This spreads out the I/O load among multiple disks, and usually across multiple servers, so in theory the maximum I/O rate is the sum of the I/O rates for the individual disk arrays. Of course, in practice the data rates do not quite reach the theoretical maximums, but process does allow for substantial increases in I/O throughput.

See also:

SU
Please see service unit
Symbolic links (symlinks)

Symbolic links ("symlinks" for short) are a feature of Linux and other Unix-like operating systems that allow you to reference a file or directory (including other symlinks) by an alternate name.

For example, in your home directory by default you will have a symlink named "scratch". This will generally point to another symlink "scratch.PROJECT" where PROJECT is the name of one of the projects to which you belong. This "scratch.PROJECT" symlink will (unless you changed it) point to the directory "/scratch/zt1/project/PROJECT/user/USERNAME" where USERNAME is your username on the cluster.

If there is a file "foo.txt" under "/scratch/zt1/project/PROJECT/user/USERNAME", you can refer to that file by any of the following paths: - /scratch/zt1/project/PROJECT/user/USERNAME/foo.txt - ~/scratch.PROJECT/foo.txt - ~/scratch/foo.txt

Note that these are all the same file; if you edit (or delete) the file under any of those names, the change will impact the file under the other names as well, as there is only one file.

If you delete a symlink (e.g. "rm ~/scratch") it only deletes the symlink, not the actual directory the symlink points to. NOTE: when the symlink points to a directory, deleting a file inside the symlink (e.g. "rm ~/scratch/foo.txt") will actually delete the file. You can use the "ln -s" command to create symlinks as long as you have write permission to the directory in which the symlink is to be put.

Symlinks are just a convenience which allow you to use shorter names for paths, etc. They are similar to "shortcuts" in Windows.

See also:

T

task

MPI jobs consist of a number of tasks, which is the smallest unit of execution within MPI. All processes, etc. for a given task run on the same node (processes for other tasks can either be on the same or different nodes, but the processes for a given task will not be split across multiple nodes), and each task uses the MPI for communication with the other tasks, so therefore the tasks can all be running on different nodes (although they can also run on the same node).

In a pure MPI job, the tasks are sequential, and so the job will use a number of cores equal to the number of tasks.

Hybrid MPI jobs can also exist, in which each MPI task uses a shared memory parallelism technique (OpenMP or other threading paradigms) for additional parallelism. In this case, each task needs multiple CPU cores on the same node. Although all cores for a task will always be assigned on the same node, the default is one CPU core per task, so you will need to specify the number of CPU cores per task specify the number of CPU cores per task

Tebibyte (TiB)

This is the binary unit corresponding to one terabyte (TB) or roughly one trillion bytes (B). It is equivalent to 1 TiB = 2^40 B = 1024^4 B = 1024^3 kiB = 1099511627776 B ~ 1.1 TB. Note that on this scale, the difference between the binary (TiB) and decimal (TB) units is almost 10%.

The largest consumer hard drives available in 2007 were only about 1 TiB, and even in 2022 most laptops only have drives in the 1/4 to 1 TiB range. The entire printed collection of the U.S. Library of Congress would come to about 10 TiB.

See also:

temporary storage
See node-local temporary storage
Terabyte (TB)

This is the decimal unit corresponding to one trillion (10^12) bytes.

The largest consumer hard drives available in 2007 were only 1 TB, and even in 2022 most laptops only have drives in the 1/4 to 1 TB range. The entire printed collection of the U.S. Library of Congress would come to about 10 TB.

See also:

thread

A thread, often called a "lightweight process", is a thread of execution, which is the smallest sequence of instructions which the operating system can manage.

From an HPC perspective, a multi-threaded process can have multiple threads of execution running simultaneously, assuming that the node the job is running on has available cores for each thread. Technically, a multi-threaded process can run on fewer cores than threads, but in that case you will not get full advantage of the parallelism. Generally for HPC purposes, you want a separate CPU core for each thread, to get the maximum speedup.

Threaded programs typically use either OpenMP or threading building blocks as multithreaded programming models.

See also:

Threading Building Blocks

Threading Building Blocks (aka TBB or oneTBB) is a C++ template library developed by Intel for shared memory parallelism.

See also:

TRES (Trackable RESources)
TRESes (short for Trackable Resourses) is the term used by the Slurm scheduler for the various consumable resources that it tracks. This includes resources like CPU cores, memory, as well as GRESes like GPUs.


Nodes have lists of TRESes which list which resources are configured on the node and which are allocated to jobs.

Jobs have lists of TRESes which list which resources were requested by and allocated to the job.

There is also a special TRES called "billing" which is the hourly SU cost; for a job it is the number of SUs which the job will be charged for every hour of runtime. For a node, it is the hourly cost of the entire node in SUs.

U

Unix shell
See shell (Unix)

V

Video RAM (VRAM)
Video RAM (VRAM) refers to the memory included in GPUs. Code run on GPUs typically only have access to data stored in VRAM, so this becomes a critical resource in GPU-based codes. In particular, the UMD HPC clusters cannot directly allocate VRAM resources; the scheduler only allocates GPUs to jobs, and when a GPU is allocated, all of the VRAM on that GPU is allocated to the job.
Virtual Private Network (VPN)

A virtual private network or more commonly VPN is a network architecture for virtually extending a private network across other, potentially untrusted networks. Basically, the VPN will encrypt all of the information between your workstation and the trusted campus network, essentially securing your network connection.

The campus VPN requires you to login with your standard campus credentials, and requires multifactor authentication. For this reason, connections to the cluster using the VPN are considered to have already presented one authentication factor, and such connections only require one additional authentication factor (e.g. entering the correct password, or presenting a kerberos ticket) to gain access. Web based portals (e.g. the OnDemand portal or the ColdFront Allocation Management portal) are only accessible over the VPN.

See also:

VPN
See Virtual Private Network
VRAM
See Video RAM

W

Walltime
The term walltime refers to the amount of time that a job actually runs or ran. The "wall" in the term means that this is the normal, everyday time you are familiar with, as for example might be measured by a clock on the wall. (This is to distinguish from other, more specialized "times" sometimes used in refering to processes running on a computer system.)
I.e., a job which starts to run at 1:05 PM and finishes at 3:27 PM on the same day will have a walltime of 3:27 - 1:05 = 2:22 or 2.37 hours.

X

Y

Z