Skip to content

FF chevron

Compiling codes

On this page we discuss various topics related to the compilation of codes on the University of Maryland HPC clusters.

Compiling codes

Compiling codes in an HPC environment can be tricky, as they often involve MPI and other external libraries. Furthermore, the various HPC clusters offer multiple compiler familiess, and multiple versions of the compilers within each family, and multiple versions of many libraries. All of this can make this compilation even more complicated.

Note

By design, the compute nodes of the cluster are not able to access the public internet. You should download source codes, etc. from the login or data transfer nodes. In general, we allow the compilation of codes on the login nodes, the only exception to the policy of no compute intensive processes on the login nodes. We do ask that you restrict any compilations to use no more than 8 threads.

We have simplified this a bit with the introduction of "toolchains".
These are collections of compilers and related libraries. The various toolchains are:

  • toolchain/gcc: This is a toolchain consisting of open source packages and libraries based on the GNU Compiler collection This is the toolchain with which most of the system software library has been built. This is usually the best supported compiler for most open source packages. In addition to gcc, it uses openmpi, openblas, fftw, etc.
  • toolchain/intel: This is the toolchain based on the legacy Intel compilers. The Intel compilers aggressively optimize code for running on Intel processors; generally they do not optimize well for AMD processors (which comprise most of the Zaratan cluster, even for optimizations which would work on either Intel or AMD chipsets, and so we generally do not encourage the use of Intel compilers on Zaratan. Note: Intel no longer supports the legacy Intel compilers, so their use on Zaratan is deprecated and will likely be removed in the near future.
  • toolchain/oneapi: This is the toolchain based on the newer clang based Intel compilers. The Intel compilers aggressively optimize code for running on Intel processors; generally they do not optimate well for AMD processors (which comprise most of the Zaratan cluster, even for optimizations which would work on either Intel or AMD chipsets, and so we generally do not encourage the use of Intel compilers on Zaratan.
  • toolchain/aocc: This is the toolchain based on the AMD Optimizing Compilers. These compilers are designed to strongly optimize codes for running on AMD processors; it is not clear how well they optimize for Intel processors.
    This toolchain includes OpenMPI, and AMD optimized math libraries.

Tip

Most of the compute nodes on Zaratan use AMD based CPUs. The Intel compilers generally do not optimize code well for AMD chipsets (even in cases where the same optimization for Intel chipsets would work), and so we generally do not recommend the use of Intel compilers on Zaratan.

Warning

The legacy Intel compilers (icc and ifort) are no longer supported by Intel. The use of such compilers on Zaratan is deprecated, and they will likely be removed in the future.

Once you decide which toolchain you want to use, you should module load it. You can module load the entire toolchain, or individually module load the various components it includes (the toolchain module does not do anything special, it is just a shortcut for loading a bunch of modules).

The following compiler commands are available on the Zaratan HPC cluster:

Compiler Family MPI library C Compiler C++ Compiler Fortran Compiler
GNU none gcc g++ gfortran
GNU OpenMPI mpicc mpic++ mpifort
Legacy Intel none icc icpc ifort
Legacy Intel IntelMPI mpiicc mpiicc mpiifort
New Intel none icx icpx ifx
New Intel IntelMPI mpiicc mpiicc mpiifort

Note

Please note the doubled i in the compiler names with IntelMPI

If you have any external libraries you need to use, you need to module load these as well. Some libraries have specific versions compiled with and for a specific compiler/MPI library combination; in such cases you need to pick a version which matches what you are using; not all combinations exist. You are generally advised to try to use the latest versions available on the system. Fortran90 codes are particularly sensitive to this, and the .mod files from different versions of the same compiler might not be compatible (and are definitely not compatible across compilers).

For packages which are libraries used by other packages (e.g. LAPACK, NETCDF, etc), the module load command may not be enough. The package being compiled needs to know where to find the libraries, and the way to inform the package of that depends on the build system used by the package.

Doing a module load or tap for an external library generally only defines some environmental variables for you, you still need to instruct the compiler where to find any needed include files and where to find the libraries. Generally, "module help MODULENAME" will give a brief usage summary giving the names of these variables.
Typically there is a variable with a name like ending in INC or INCDIR, and another one with a name ending in LIB or LIBDIR E.g. the netcdf package defines NETCDF_INCDIR and NETCDF_LIBDIR. Similarly, the fftw package defines FFTW_INCDIR and FFTW_LIBDIR.

The INCDIR variable provides the location of the include directory for the package. You will generally want to either add arguemnts with these variables preceded by the -I flag to your compilation command. The -I flag takes a single path, so you should repeat it if you have multiple library packages that you are using.

The LIBDIR variable provides the location of the directory containing the static or shared libraries for the package, and they work similarly to the INCDIR variables discuss, except these are needed when the compiler/linker combines the object code files into the executable. For small codes, the compilation and linking usually occur in a single step; larger codes, especially those with makefiles, usually break this into separate steps. Also, the dynamic linker needs to know where the libraries files are when you run the code, so it is usually easiest to set the "rpath" during compilation. (Otherwise you will need to set the environmental variable LD_LIBRARY_PATH each time before you run the code). To tell the compiler where to find the library for linking, you provide the "LIB" variables as arguments to the "-L" flag. To set the rpath, you provide them as arguments to the "-Wl,-rpath," flag. Both the -L and -Wl,-rpath, take a single path, so you should repeat these flags for each library package. You will also need to use the "-l" flag to give the names of library files (with the leading "lib" and any ".a" or ".so" extension removed).

E.g., if you have a code in the file my_netcdf_code.c which uses the netcdf and fftw libraries, you might compile it with a single command doing both the compilation and linking together, producing a my_netcdf_code binary, like:

Example

login-1:~: gcc -I$NETCDF_INCDIR -I$FFTW_INCDIR \\
    -L $NETCDF_LIBDIR -L $FFTW_LIBDIR \\
    -Wl,-rpath,$NETCDF_LIBDIR -Wl,-rpath,$FFTW_LIBDIR \\
    -l netcdf -l fftw3f -o my_netcdf_code my_netcdf_code.c

You could also break this into two steps, first producing the my_netcdf_code.o object file, and then building the my_netcdf_code binary. Note that the first, compilation only, step only requires the INCDIR variables, while the second (linking) steps only requires the LIBDIR variables. This might go something like:

Example

login-1:~: gcc -c -I$NETCDF_INCDIR -I$FFTW_INCDIR my_netcdf_code.c
login-1:~: gcc  -L$NETCDF_LIBDIR -L$FFTW_LIBDIR \\
    -Wl,-rpath,$NETCDF_LIBDIR -Wl,-rpath,$FFTW_LIBDIR \\
    -l netcdf -l fftw3f -o my_netcdf_code my_netcdf_code.c

If your code is using a Makefile for the build, you would probably need to do something like the following in your Makefile (this is assuming a standard Makefile setup; but not all Makefiles are standard so this will not work in all cases)

Example

CFLAGS += -I$(NETCDF_INCDIR) -I$(FFTW_INCDIR)
FFLAGS += -I$(NETCDF_INCDIR) -I$(FFTW_INCDIR)
LDFLAGS += -L$(NETCDF_LIBDIR) -Wl,-rpath,$(NETCDF_LIBDIR) -lnetcdf
LDFLAGS += -L$(FFTW_LIBDIR) -Wl,-rpath,$(FFTW_LIBDIR) -lfftw3f

Note that if the code you are compiling is purely C, you will not need the FFLAGS line. Conversely, if compiling pure Fortran code, you generally will not need the CFLAGS line.

If you omit the "-Wl,-rpath,..." arguments in any of the above, your code should compile and generate an executable, but you will likely get an error about libraries not being found when you attempt to run the binary, unless you first module load netcdf and fftw first (or otherwise set LD_LIBRARY_PATH appropriately.)
This will need to be done in every login shell in which you wish run the code. And if you mistakenly module load a different version of netcdf and/or fftw, the program might or might load, but even if it loads it might cresh with a difficult to debug error at some seemingly arbitrary location (or perhaps even worse, it might run to a seemingly successful conclusion but produce erroneous results).

We strongly recommend that for your own code, or for code that you are compiling, that you use the rpath arguments shown earlier. The LD_LIBRARY_PATH option is, in our opinion, best used when you do not have the option of compiling the code with the rpath settings.

Optimization of Code

Since you are using a High Performance Computing cluster, you most likely have long, compute intensive jobs. These can generally benefit greatly from various optimization techniques. The general topic of code optimization is quite broad, far too large to give more than a cursory discussion here. You are encouraged to look at the myriad of resources on the topic that exist on the web. Here we just discuss some specific details related to the UMD clusters.

A major performance feature of modern processors is the ability to vectorize certain operations. In this mode, an instruction is decoded once and operates on a stream of data. This allows for significant performance boosts. There are various vectorization command sets available on x86 based processors, known by names like SSE and AVX (with numerical suffixes to specify the version of the command sets). Selecting the correct optimization level can be tricky, as different processors support different vectorization command sets. This is further complicated by the mix of Intel and AMD processors on the cluster.

Currently the Zaratan cluster is fairly homogeneous, although we expect that this could change in the future as more nodes are added to the cluster, Most of the compute nodes on Zaratan consist of dual AMD Epyc Zen3 chips with 128 cores per node, which support the various flavors of SSE as well as AVX and AVX2, but not AVX512. These chips are present on all of the standard partition as well as the A100 nodes on the [gpu partition)(/kb/queues.md#gpu) and the login nodes. The H100 nodes on gpu partition are Intel Xeon SapphireRapids chips,, which support AVX512 as well as AVX2, AVX, and the various SSE flags.

Note

The optimization flags discussed below are specifically for the GCC compiler suite except where noted otherwise, but they should also generally work on most modern compilers. (Some compilers might have an equvialent flag with a different name.)

Generally one should enable a reasonable level of optimization during compilation. This is typically done with the -O optimization flag. This flag accepts an argument after the 'O' to specify how aggressively to optimize; typically you will want to use at least "-O2", and maybe even "-O3". The O2 optimizations typically include all performance optimizations which do not cause a significant increase is the size of the binary; the O3 level includes all O2 optimizations as well as some which increase the size of the binary. The "-Ofast" optimization is even more aggressive, but includes optimizations which do not strictly adhere to standards.

Tip

Use one of -O2, -O3, or -Ofast when compiling your code. - -O2 is the safest, and optimizes well, but misses some optimizations - -O3 should usually be safe, and opimizes a bit more than -O2 - -Ofast us the most aggressive, and might cause problems with some codes

You can also enable chip specific optimizations. Here there are two main options (in GCC nomenclature):

  • -march=X: this will generate code which will run on systems with an CPU architecture given by X, but might not run on other systems. Typically the code will run on newer CPUs in the same family, but might or might not run on older CPUs. As a special case, you can give the word 'native' for X and it will generate code for the CPU architecture the compiler is running on. On Intel compilers, the equivalent flag is -xHost
  • -mtune=X: this is a bit weaker optimization-wise than -march. The code produced should run on all CPU architectures in the same family (e.g. x86-64) supported by the compiler, but will be optimized for the architecture given by X.

In general, using -march=X implies the optimizations of -mtune=X as well. This assumption sometimes breaks down for older compiler versions which do not know about (e.g. were released before) the chips being used. For code which is to be run on the AMD nodes, especially for CPU only codes in the standard partition, we recommend using -march=native.

For codes being compiled for use on the Intel nodes, we recommend building with the -march=native or -xHost flag, depending on the compiler being used.

For codes that you plan to run on multiple architectures, we recommend compiling multiple versions of the code optimized for the specific architectures. Although you could produce a single executable that works on both AMD and Intel chips, you would need to lose many of the advanced features of both chips to do so.

Compiling with OpenMP

OpenMP is an API for shared memory parallelization, i.e. for spliting a code into multiple threads running on the same node. Because the parallelization is limited to a single node, it is less powerful than other APIs (e.g. MPI which can span multiple nodes, but is also much easier to code for.

Indeed, OpenMP is implemented by the compiler, and is generally invoked through various compiler directives and/or pragmas.
E.g., if you have a C code with a for loop, you can add a pragma just before the start of the for loop which will tell the compiler to try to split the loop into multiple threads each running in parallel on a different processor core.

If you have a code with OpenMP directives, you need to tell the compiler to compile it with the OpenMP extensions enabled. The exact mechanism is compiler dependent:

Compiler Family Flag to use OpenMP Default number of threads
GNU -fopenmp available cores
Intel -qopenmp available cores
nvhpc -mp 1

The "Default number of threads" column in the above table lists the number of threads which the program will use if not specified otherwise (e.g. not set inside the code and OMP_NUM_THREADS environment variable is not set). When available cores is listed, that means the program will attempt to use all the cores that it can see on the node.

Note

If you are using the Intel compiler suite and the Math Kernel Libraries (MKL), some of the MKL routines might use OpenMP even if you did not request OpenMP in the main compilation. You can set the environmental variable OMP_NUM_THREADS to 1 to effectively disable that if such is really desired.

By default, OpenMP will attempt to use as many threads as there are cores on the system (except with nvhpc cmopilers, which default to one thread). This can be problematic in some cases. At runtime, you can set the environmental variable OMP_NUM_THREADS to an integer to control the maximum number of threads that OpenMP will use.

Tip

Be sure to use the appropriate OpenMP flag for your compiler on all of the compilation and link stages of your code.