Using Conda

Conda is a package and environment management system allowing users to install different versions of binary software packages and their dependencies into their home or other directories to which the user has write permission. These packages are installed into Conda environments which are somewhat isolated from the rest of the computing platform, including system installed packages and other Conda environments.

This isolation provided by Conda environments can either be an advantage or a liability depending on the use case. The isolation means that you are not constrained by the packages (or versions thereof) which the HPC staff have installed on the cluster, but it also makes it very difficult if not impossible to avail yourself of packages installed by HPC staff. Furthermore, as you are relying on an entire environment which you installed, the ability of HPC staff to provide useful assistance in the case of issues will be limited.

Conda is a commonly used for the installation of Python and R modules to enhance the capabilities of those scripting languages. Users on the UMD HPC clusters are free to use Conda environments for installing their own python or R modules, however we recommend that most users look at other means of installing Python and/or R modules as conda installations do not integrate well with the rest of the Research Software Library (RSWL).

Tip

Conda environments can grow rather large. It is recommended that Zaratan users do not install Conda environments in their home directory as such environments may easily exceed the allotted home quotas. It is recommended that users install their Conda environments to the scratch storage tier, or (if using conda-pack) to the SHELL storage tier.

As the SHELL storage tier is not accessible from the compute nodes, placing Conda environments here is only recommended if you are using conda-pack to create a tarball of the environment in scratch to then extract on the compute node's local /tmp filesystem.

Note

Due to potential licensing concerns, DIT staff strongly recommends that users use the Miniforge/Mambaforge conda installers instead of Miniconda or Anaconda.

Tip

It is recommended that users consider installing Python and/or R modules via other methods before trying to install in a Conda environment. This is because the isolation discussed above means that Conda environments do not integrate well with the Research Software Library (RSWL).

Python packages installed with setup.py or virtual environments, or R packages installed with R CMD INSTALL integrate better with the Research Software Library (RSWL), allowing you to avail yourself of python/R modules and other libraries within the RSWL as well as the modules your install.

Conda, Anaconda, and Licensing

Tip

As explained below, Conda licensing and terms of service are a bit complicated.

To avoid any potential licensing issues from using Anaconda or Miniconda and their associated channels, we strongly recommend users at UMD to instead use MiniForge/MambaForge which by default use the "free-to-use" conda-forge channel. The Bioconda channel is also free to use.

The use and functionality of MiniForge/MambaForge is very similar to that of Miniconda, and you will not need to worry about rebuilding your Conda environments should the Anaconda licensing terms change in the future.

Conda licensing and terms of service can get a bit complicated. To avoid any potential current or future licensing issues from the use of Anaconda or Miniconda and their associated channels, we strongly recommend users in the UMD community to use MiniForge/MambaForge instead of either Anaconda or Miniconda. This installer is set up from the start to use the freely-available conda-forge channel.

The Conda Ecosystem

Before trying to explain Conda licensing, we first discuss the Conda Ecosystem. The Conda ecosystem refers to a collection with various components, including:

• the actual conda installer program (or similar installers like mamba or pixi)
• the collection of software packages which the conda command can install. These packages are grouped into channels
• potentially other utilities, pre-installed packages, etc. to assist with package management

Conda is typically installed from one of the following - the Anaconda distribution - the Miniconda installer - the MiniForge installer We will discuss the differences between these below.

The conda installer program is open-source, and is free for use by anyone (as long as the channels from which the software is being installed is also free to use). Anaconda, Miniconda and Miniforge all include the open-source conda installer program. The mamba and pixi programs are alternate, conda-like, installer programs. They are also open-source.

In order for the conda installer to be of any real value, it needs access to a bunch of software packages which it can install. These are arranged into collections called channels. The maintainers of these channels perform an useful service to create the installable packages, ensure things are built consistently and securely, etc. This work of curating the software packages is an added value, and the curators of these packages can impose licensing conditions/terms of service regarding the access to the packages they curate (even if the underlying package is open-source; what they are "licensing" is the access to their curated collections).

There are a number of channels which are in common use: - the Anaconda defaults channel: this is covered by Anaconda's terms of service. Anaconda engineers build, maintain, and curate the packages in this channel to provide maximal security and stability. Licensing fees for the use of this channel help defray the cost of this. This channel is further divided into subchannels, which are covered by the same terms of service: (full list of subchannels of the defaults channel: - main - r - msys2 - the conda-forge channel: this is a channel providing general software packages. Many packages here overlap those found in the defaults channel above. This channel is maintained by volunteer communities, and provides best_effort security. - the Bioconda channel: this is a channel containing many packages related to biomedical research. This channel is also maintained by volunteers.

The Anaconda distribution and the Miniconda installer are configured upon installation to use the Anaconda defaults channel. They both also install a bunch of packages automatically, like python, etc., and these will be installed from the defaults channel. Anaconda is usually referred to as a distribution as opposed to an installer because it will install a lot of (over 100) packages automatically along with the conda program on the initial install; Miniconda just installs python and some minimal dependencies for conda and python.

The MambaForge/MiniForge installer are configured upon installation to use the conda-forge channel. When MambaForge/Miniforge is installed, it will just install the conda program and the various dependencies it needs to run, like python,etc. These will be installed from the conda-forge channel. It does not even install python by default.

Licensing

In 2020, Anaconda changed the terms of service for the Anaconda distribution and the Anaconda defaults channel (which is used by default by the Anaconda distribution and the Miniconda installer). These new terms required organizations with more than 200 employees or contractors to have a Business or Enterprise license in order to use Anaconda. Initially, there was some confusion as to whether that condition applied to educational institutions, but in 2024 Anaconda clarified that it did apply, unless the use of Anaconda is "is solely limited to being used for a curriculum-based course."

The University of Maryland at the current time (Jan 2025) does not have a Business or Enterprise license with Anaconda, and there are no plans we are aware of to obtain such as the cost would be prohibitive. It is not clear to the HPC staff that the use of Anaconda at UMD meets the "curriculum-based course" exemption, and so we are strongly advising all users at UMD to stick to the installers and channels which are not encumbered by these licensing/terms of service restrictions.

In particular, we recommend that users avoid the Anaconda distribution and the Miniconda installer, as installing Conda from either of these will result in the installation of packages from the Anaconda defaults channel, which is covered by the aforementioned terms of service.

Instead, we recommend that users install Conda from the MambaForge/MiniForge installer or a similar unencumbered installer. And that users make sure that they are only using unencumbered channels, e.g.: - the conda-forge channel - the Bioconda channel

In particular, make sure that any encumbered channels are removed, including: - the Anaconda defaults channel - the main subchannel of defaults - the r subchannel of defaults - the msys2 subchannel of defaults - [any other subchannels of defaults]((https://docs.anaconda.com/working-with-conda/reference/default-channels)

Please see the section on channels for instructions on listing and configuring channels.

For more information, see: - Anaconda terms of service - Blog post Is Conda Free? - Bioconda common license - Conda-forge license

Warning

To avoid potential licensing issues, the use of the Anaconda distribution or Miniconda installer should be avoided.

The Division of Information Technology recommends that users should instead install Conda using the MambaForge/MiniForge installer

Tip

Conda environments can grow rather large. It is recommended that Zaratan users do not install Conda environments in their [home directory]/kb/storage.md#home) as such environments may easily exceed the allotted home quotas. It is recommended that users install their Conda environments to the scratch storage tier, or (if using conda-pack) to the SHELL storage tier.

As the SHELL storage tier is not accessible from the compute nodes, placing Conda environments here is only recommended if you are using conda-pack to create a tarball of the environment in scratch to then extract on the compute node's local /tmp filesystem.

To install Conda, we recommend the following:

1. Choose a location for your Conda environments. Because Conda environments can get quite large (~> 10 GB), we recommend that HPC users do not install Conda into their home directory due to quota limitations. Instead, use
   • a directory in scratch, e.g. ~/scratch/conda, or
   • If you will be using conda-pack, you can install your Conda environment on the SHELL storage tier, e.g. ~/SHELL/conda, and then create a conda-pack tarball of it into scratch, and untar that into the compute node's local /tmp filesystem at the start of the job. Create that directory if it does not already exist. We will refer to this as your conda root directory.
2. Download the latest MambaForge/MiniForge installer into your conda root directory.
3. We recommend that you make a copy of your initialization dot files, e.g. cp ~/.bashrc ~/.bashrc.HOLD
4. Change to your conda root directory, and install miniforge/mambaforge: bash Mambaforge.X.Y.Z-A-Linux-x86_64 -p mambaforge-X.Y.Z Replace the X.Y.Z in the above with the version of the installer your downloaded. The above command will install Conda into the directory named after the -p flag in the directory you ran the script from. The script will ask you some questions, generally answering 'yes' to all the questions should work.
5. The above command will update your ~/.bashrc file to add some initialization code for Conda. Although you could leave things like that, doing so would mean Conda would be initialized everytime you start a shell, whether it needs it or not (and potentially causing delays in startup and/or issues with packages in the Research Software Library. We recommend that you copy the modifications the installation process made to your dot files (everything between the # >>> conda initialize >>> and # <<< conda initialize <<<' lines) to a Conda specific initialization (e.g._{bashrc.conda), and then remove those from your}/.bashrcfile. You can then "source" this script in shells which should use Conda (something likesource ~/.bashrc.conda`).

At this point, conda is installed and you can start using it.

Using Conda

The following section discusses some basics of using Conda.

Initializing Conda

By default, running the script to install Conda will add code to initialize Conda to your .bashrc initialization script. While this might be convenient for a personal workstation, it causes Conda to be initialized for every shell, whether it is needed or not. On the UMD HPC systems, this can cause startup delays as well as problems using some packages in the Research Software Library (RSWL) or problems with the Interactive Applications in the OnDemand Portal. As such, we recommend that after running the Conda install script, you cut this initialization code from your ~/.bashrc file (the lines between the # >>> conda initialize >>> and # <<< conda initialize <<< lines) and paste them into a new, Conda-specific initialization file, e.g. ~/.bashrc.conda.

If you have done this, Conda will not automatically be initialized with every shell (which should speed up the startup of the shell, and avoid issues with some packages in the RSWL). When you intend to use Conda, you need to issue a command like

Example

source ~/.bashrc.conda

before you start to use and conda commands. This will need to be done once per command shell in which you plan to use Conda. It will also be required at the start of any Slurm job scripts which will be using Conda.

If you did not follow our advice and kept the Conda initialization lines in your ~/.bashrc, then Conda should have been automatically initialized and nothing further needs to be done.

Environments

Note

You will need to initialize Conda before you can issue the Conda commands below

It is recommended when using Conda to use environments.
Environments allow you to keep the software installed for various projects separate from each other. They can also be used to keep different, incompatible software packages from adversely affecting each other.

The conda command has a number of subcommands for managing environments. Common commands are:

• Creating an environment named myenvironment.

Example

conda create -n myenvironment`

• Listing environments

Example

conda info --envs

• Activating an environment named myenvironment. The various conda, python, etc. commands all act on the currently activated environment.

Example

conda activate myenvironment

• Deactivating the currently active environment. This will cause the currently active environment to no longer considered active, and the base environment will become the new active environment.

Example

conda deactivate

Once you activate an environment, the packages installed in the environment are available for your use.

Installing packages

Note

You will need to initialize Conda before you can issue the Conda commands below. You will probably also wish to activate an environment before installing packages.

To install packages from one of the configured Conda channels, you can use a command like

Example

conda install python

You can also often install Python modules with the pip command (this assumes that the python package has already been installed in the environment). When you are in an activated environment, the python and pip commands will be the ones from the environment, and installing python modules using the pip command will install the package in such a way that it is available for use with the python command from the environment. Python packages available in the channel can also be installed with the conda install command.

Warning

The channel r, a subchannel of the Anaconda defaults channel, is covered by the Anaconda terms of service, and therefore we strongly advise UMD users not to use it.

R modules found in the channel can also be installed with the conda install command. Please do not use the channel r, as it is a subchannel of the Anaconda defaults channel. If R is installed in the environment (e.g.

Example

conda install r

), then you can often also use the R CMD INSTALL command to install R packages.

Package management

Note

You will need to initialize Conda before you can issue the Conda commands below. You will probably also wish to activate an environment before installing packages.

To see what packages (and versions of the packages) are installed in the currently active environment, you can use the command

Example

conda list

To remove a package from the currently active environment, you can use the command

Example

conda remove *PACKAGENAME*

To upgrade a package in the currently active environment, you can use the command

Example

conda upgrade *PACKAGENAME*

Channel Management

There are a number of commonly used channels, namely (the Terms of Service field is 'Anaconda' for channels which are covered by the Anaconda terms of service and 'free' for channels which are freely usable these should generally *not be used by members of the UMD community due to those terms of service):

Channel name	Encumbered?	Description
defaults	Anaconda	The channel Anaconda/Miniconda is configured to use by default
main	Anaconda	A subchannel defaults
r	Anaconda	A subchannel defaults
msys2	Anaconda	A subchannel defaults
conda-forge channel	free	A freely usable general package repository
Bioconda channel2	free	A freely usage repository of biomedical packages

Warning

For licensing reasons, Users at UMD should only be using channels marked as free in the above table, unless you are certain that you are covered by the 'curriculum-based course' exemption.

To view the configured channels, you can use the command

Example

conda config --show channels

You can remove a channel from the list, e.g. the defaults channel with the command

Example

conda config --remove channels defaults

You can ignore any errors about channels not being present in the config (as you are deleting them anyway).

You can add channels, e.g. conda-forge with the command

Example

conda config --add channels conda-forge

For advanced users, the channels configuration can be found in the file .condarc under your Conda installation, although we advise against modifying it.

Using conda-pack

Using conda has some issues on the UMD HPC clusters. Conda environments can get quite large, and it is not uncommon for them to exceed 5 or even 10 GB in size. This makes them difficult if not impossible to install in your home directory which is currently capped at 10 GB. As the conda environment is typically needed for use in jobs running on the compute nodes, the scratch directory is a natural choice.

However, conda environments are typically comprised of many, many small files, and many if not most of these need to be opened and read during the initialization, activation, and use of a conda environment. The scratch directory, however, is optimized for large files, and the opening and reading of numerous small files can lead to a bottleneck, especially if a large number of MPI tasks and/or jobs are doing this at the same time. (This small file bottleneck is an issue with most if not all high performance file systems used in HPC.)

One strategy to overcome this issue is to store the Conda environment in a single file (e.g. a tarball or similar), and then when the job starts extract this tarball to the node local /tmp filesystem and initialize, activate, and use the copy of the environment in /tmp. In this scenario, there is only a single, largish file from the perspecitive of the scratch filesystem (so playing to the strengths of the scratch filesystem and avoiding the small file bottleneck), and the large number of opening and reading files is occuring on the node local /tmp filesystem which can handle that better than scratch.

The above description is a bit simplistic --- Conda environments are not relocatable, and so simply installing a Conda environment somewhere, tarring it up, and then untarring it somewhere else will not work.

Conda-pack is a command-line tool which can handle the packaging of a Conda environment in a relocatable fashion, allowing the above strategy to work.

So the general use case on UMD HPC clusters would be: 1. Install conda-pack if needed. It is usually easiest to do this by: 1. installing Conda, you can do this in your SHELL directory as it should only be needed on login nodes 2. Create a conda-pack environment and activate it. 3. Install the conda-pack package, e.g.

Example

conda install conda-pack

1. Create a Conda environment for the package/project you wish to use on the compute nodes, and install packages into the environment basically as usual.
   • Since you will copying the environment to the node local /tmp filesystem and activating/using it from there on the compute nodes, you can actually install the environment in the less expensive SHELL storage tier.
2. Activate your conda-pack environment, conda activate conda-pack, and then use the conda-pack command to create a tarball or zipfile, etc. from desired Conda environment. This can be done with a command like (assuming the environment you wish to use on the compute nodes is myproject): conda-pack --name myproject This will produce a tarball myproject.tar.gz in the current directory.
   • You can use the --prefix flag instead of --name to specify the environment to pack by path instead of name
   • You can add the --output flag to give a different name to the output tarball
   • You can use the --format flag to specify the output format. zip files and tarballs with various compression schemes are supported. By default, it will try to guess based on the output file extension, or default to tar.gz otherwise.
   • --help will give a full list of options
3. The tarball, etc. created above should be saved in the scratch filesystem as it will need to be accessed from the compute nodes. Move it there if needed.
4. At the beginning of the job script, you will need to create a directory in the node local /tmp filesystem, and extract the tarball/zipfile/etc containing the Conda environment into that directory. If your job is running across multiple nodes, this will need to be done on all nodes used by the job. See below for an example of this.
5. Run your job using the conda environment you extracted to the node-local /tmp filesystem.

When creating a directory under the node-local /tmp filesystem, there are a couple of approaches. In general, we recommend that for multinode jobs you keep the directory path the same on all nodes. Options include: - /tmp/$USER/$SLURM_JOB_ID: This will use a distinct copy of the environment for each job. This is a safe approach, especially if different jobs might use different Conda environments. - /tmp/$USER/*ENVNAME*: This will create a distinct copy of the environment for each environment being used (based on the environment name). This means that multiple jobs running on the same node and using the same environment can share the same extracted tarball.

To help with the process of extracting the tarball created with conda-pack, we have created the script conda-pack-unpacker.sh. This script basically untars the provided tarball, with some extra bells and whistles so that if multiple processes call the script with the same arguments on the same node at more or less the same time (as might happen in an MPI job or if multiple jobs are running on the same node), they will not overwrite each other. The first process called will extract the file, and the others will wait until it is done. If there is an error extracting the tarball, or it is delayed too much, an error will be returned.

For example, if you have an environment named foo in a tarball ~/scratch/foo.tar.gz and your job script is to run the python script myfoojob.py on a single core, you might use a job script like:

Example

#!/bin/bash
#SBATCH -c 1
#SBATCH -t 5:00

#Unpack the Conda environment
#(assuming you have installed conda-pack-unpacker.sh to
#your ~/bin directory)
~/bin/conda-pack-unpacker.sh -f ~/scratch/foo.tar.gz
if [ $? -ne 0 ]; then
    echo "[ERROR] Error unpackaging ~/scratch/foo.tar.gz"
    exit 1
fi

/tmp/$USER/foo/bin/python myfoojob.py

A multithreaded job script would look similar, except you would want to increase the number of cores requested by the job. If myfoojob.sh is designed to use MPI, the script would be a bit more complicated given the need to extract the tarball on each node the job is running on. However, you can use the srun command to do such, and it is not that much more complicated, e.g:

Example

#!/bin/bash
#SBATCH -c 1
#SBATCH -n 20
#SBATCH -t 5:00

#Unpack the Conda environment on each node allocated
#to the job.  Just run conda-pack-unpacker.sh in each
#task, as it can handle multiple calls on the same node
#your ~/bin directory)
srun ~/bin/conda-pack-unpacker.sh -f ~/scratch/foo.tar.gz
if [ $? -ne 0 ]; then
    echo "[ERROR] Error unpackaging ~/scratch/foo.tar.gz"
    exit 1
fi

mpirun /tmp/$USER/foo/bin/python myfoojob.py