python: Python scripting language

Overview of package

Package:	python
Description:	Python scripting language
For more information:	https://www.python.org
Categories:	SoftwareDevelopment
License:	OpenSource (Python Software Foundation)

General Usage

Python is a high-level scripting language.

This module will add the python and related commands to your path.

In case you need to link against this library in your code, the following environmental variables have been defined:

$PYTHON_ROOT has been set to the root of the python installation
$PYTHON_LIBDIR points to the directory containing the libraries
$PYTHON_INCDIR points to the directory containing the header files

You will probably wish to use these by adding the following flags to your compilation command (e.g. to CFLAGS in your Makefile):

-I$PYTHON_INCDIR

and the following flags to your link command (e.g. LDFLAGS in your Makefile):

-L$PYTHON_LIBDIR -Wl,-rpath,$PYTHON_LIBDIR

Available versions by cluster

This section lists the available versions of the package pythonon the UMD HPC clusters.

Versions available on the Zaratan cluster

Version	Module tags	CPU(s) optimized for	GPU ready?
3.8.12	3.8.12 python/3.8.12/gcc/9.4.0/cuda/11.6.2/linux-rhel8-zen2	zen2	Yes
3.8.12	3.8.12 python/3.8.12/gcc/9.4.0/nocuda/linux-rhel8-zen2	zen2	Yes
3.10.10	3.10.10 python/3.10.10/gcc/11.3.0/cuda/12.3.0/linux-rhel8-x86_64	x86_64	Yes
3.10.10	3.10.10 python/3.10.10/gcc/11.3.0/nocuda/linux-rhel8-zen2	zen2	Yes
3.10.10	3.10.10 python/3.10.10/gcc/11.3.0/nocuda/linux-rhel8-x86_64	x86_64	Yes
3.10.10	3.10.10 python/3.10.10/gcc/11.3.0/nocuda/linux-rhel8-icelake	icelake	Yes
3.10.10	3.10.10 python/3.10.10/gcc/11.3.0/cuda/12.3.0/linux-rhel8-icelake	icelake	Yes
3.10.10	3.10.10 python/3.10.10/gcc/11.3.0/cuda/12.3.0/linux-rhel8-zen2	zen2	Yes

System modules

The Research Software Library (RSWL) installation of Python include a large number of Python modules to enhance the capabilities of Python. The table below list those modules and version numbers for the various installations of Python which are made available with the "module load python" or similar commands. (The ones which list a CUDA version are only available if the appropriate CUDA module was loaded before the python module was loaded.)

Packages/modules for Python

The following table lists the various packages/modules/extensions for Python along with their version which are installed on the UMD HPC clusters, by cluster, Pythonversion, and platform. These are the packages/modules/extensions which are enabled by default when you do the "module load python".

Python Version
Compiler
CPU optimized for
CUDA

In addition, a number of other packages, C libraries, applications, etc. provide a Python API, etc. through a module. As these package are more than just Python modules, and indeed in many cases most users of the package are not accessing the Python modules they define, these packages have their own modulefiles. Generally, you can access these by loading their corresponding modulefiles before starting python (normally it should not matter whether you load the package module before or after loading python). These packages are:

Python Modules

This section lists various software packages containing python modules which can be imported in your python code, after the corresponding environmental module is loaded.

Package Name	Description	CPUS optimized for	GPU ready?
adios2	The Adapatble Input/Output System, version 2	icelake, x86_64, zen, zen2	Yes
charm4py	distributed/parallel programing framework for Python	zen, zen2	Yes
charmm	Chemistry at HARvard Macromolecular Mechanics.	zen2	Yes
cupy	a Python array library accelerated with NVIDIA CUDA	zen2	Yes
cython	Compiler for writing C extensions for Python	icelake, x86_64, zen, zen2	No
gdal	Geospatial Data Abstraction Library	icelake, zen2	Yes
jupyter	The Jupyter Notebook application	icelake, x86_64, zen, zen2	No
mercurial	Mercurial source control management system	icelake, x86_64, zen2	No
openmm	A high-performance toolkit for molecular simulation.	icelake, zen2	Yes
paraview	data analysis and visualization application	zen, zen2	No
pyray	Framework for scaling AI and Python applications	zen2	No
pytorch	Open-source machine learning library	icelake, zen2	Yes
scons	SCons open-source software construction tool	icelake, x86_64, zen2	No
tensorflow	Open-source software library for machine intelligence	icelake, zen, zen2	Yes
trilinos	Solvers collection including linear, non-linear, optimization, and transient solvers	zen2	No

Installing modules

Python's capabilities can be significantly enhanced through the addition of modules. Code can import a module to enable its functionality.

The python builds as part of the Research Software Library on the system have a large selection of modules preinstalled. If a module you are interested in is not in that list, you can either install a personal copy of the module for yourself, or request that it be installed site wide. Installation of software packages by HPC staff is done on a "as time allows" basis --- HPC staff have many other duties with higher priorities, but will make reasonable efforts to accomodate such requests as staffing resources allow. We will also consider how widely used the package is expected to be, and whether it is designed for system or individual installation, etc.

Installing modules yourself

There are three general approaches for installing python modules yourself:

Using setup.py
Using virtual environments and/or pip
Using conda environments

The mechanism for installing a module is of course dependent on the module being installed, but most modern python modules support the setup.py mechanism described below. But many packages will support installing via pip into a virtual environmens as well, and that is typically easier.

Using setup.py

Note

Users might wish to look at using virtual environments to install packages first, as that is often easier.

The standard procedure for installing your own copy of a module is:

Module load the apropriate version of python, or omit version number to get the latest version installed, e.g.
```
module load python
```
Create a directory to contain your python module, if not already done. Typically, you will wan a single directory to house all of the modules you are planning to install, so something like mkdir ~/mypython will work. You should also create lib and lib/python directories beneath it.
You will need to tell python where to look for your modules. Assuming you are putting your modules under ~/mypython, something like
```
export PYTHONPATH="${HOME}/mypython/lib/python:${PYTHONPATH}"
```
You might want to add this to your .bashrc
Download and unzip/untar/etc the module sources. Cd to the main module source directory (it should contain the file setup.py
Run python setup.py install --home ~/mypython

If all goes well, the module should now be installed under ~/mypython or wherever you specified. If there are executables associated with it, they should be in ~/mypython/bin. You should be able to import the module in python now (this assumes that PYTHONPATH is set as indicated above).

Of course, not all modules install easily. Unfortunately, the install process can fail in far too many ways than can be reasonably enumerated. If you are comfortable with building modules, you might find reasonable guidance from error messages to assist you in getting the module to build, or by doing a web search on the error message.

Installing modules yourself using virtual environments

Although the standard procedure described above works for most cases, there are cases where more separation is required. Python3 includes a venv module which allows you to create a fully independent virtual python environment, copying the python executables and standard and system libraries to your own directory, and allowing you to add/update/delete from there. This has the advantage that the virtualenv is almost completely isolated; so changes made in the system installation of python are unlikely to impact your virtualenv. This can be important if you have a code or application which requires e.g. version 1.6 of the foo package, but will break if it is upgraded to 1.7 (it appears that when using standard scheme above using PYTHONPATH, the system library directories are ALWAYS searched before PYTHONPATH, meaning that method can be used to add modules, but not to upgrade or downgrade modules).

However, the virtualenv takes up a significant amount of diskspace, and the isolation from the system python can be a negative as well as upgrades and/or new modules added to the system python will NOT be visible --- this is good when as in the example above it breaks something, but most of the time the upgrades are desirable.

To install a package with the virtualenv mechanism, you must first create a virtual python environment.

Module load the apropriate version of python you wish to use in the virtual environment, or omit version number to get the latest version installed, e.g.
```
module load python
```
You should select a directory where the virtual python environment should live. Each virtual environment you create will be a subdirectory of this directory. For the examples below, we will use the my-venv subdirectory of your home directory (e.g. ~/my-venv).
You can create a virtual env with the following command:
```
python -mvenv ~/my-venv
```
on the system have a large selection of modules preinstalled.
1. This will create a virtual environment which is isolated from the system installed python. I.e., your virtual environment python will not be able to access RSWL installed python packages. This is the safest approach, but if you need to access other python modules (even those in the RSWL) you will need to install those modules into your environment.
2. If you add the --system-site-packages flag to the above command, your virtual environment will get access to RSWL installed python packages, e.g. numpy, scipy, matplotlib, etc. This is the easiest way to proceed, but as it is less isolated from the RSWL python installation there can be problems if there are version compatiblility issues.
In order to use this virtual python environment, you must first activate it. This must be done in every process using the virtual environment. You do this by issuing the command:
```
 source ~/my-venv/bin/activate
```
hen you are finished with the environment, the command deactivate will deactivate the virtual environment.

Once the virtual environment is created and activated, installation is usually relatively simple using the pip command. You should just be able to do pip install <var>NameOfPackage</var>. Pip should take care of downloading the package and installing it for you.

Of course, not all modules install easily. Unfortunately, the install process can fail in far too many ways than can be reasonably enumerated. If you are comfortable with building modules, you might find reasonable guidance from error messages to assist you in getting the module to build, but it is probably easiest to just request the module be installed to the system libraries.

Using conda

Note

We generally recommend that you try installing your own Python packages using one of the methods above, i.e. either setup.py or virtual enviroments rather than using conda. That is because conda environments are very isolated and you will lose much ability to use packages in the Research Software Library.

Conda is another option for installing python and other packages, but it does even more separation and isolation than the venv method. This can either be an advantage or a liability depending on the use case. This isolation means that you are not constrained but what system staff have installed on the cluster, but it also means it is very difficult if not impossible to take advantage of any packages system staff have installed on the cluster. Also, since you are relying on an entire environment which you installed, the ability of HPC staff to provide assistance should there be issues is limited. Therefore, we encourage users to try one of the other methods if such is feasible.

Warning

To avoid risking violating the Anaconda terms of service, we strongly recommend users at UMD use the Miniforge/Mambaforge installers for Conda.

For more information, see: - The UMD HPC Conda page - Conda Installation instructions - Installing packages using Conda

Assorted Tips and Tricks

Using matplotlib w/out an X server

By default, the matplotlib package in Python expects to work with a graphical user interface (GUI), which on Unix-like systems means an X server running. This can be problematic if one wishes to use matplotlib in batch jobs (e.g. on an HPC cluster) because typically a display will not be available. The easiest way to do this is to specify a non-interactive backend. There are several ways to do this, but since you probably want to continue using an interactive backend when using python interactively, the best approach is to have your batch code select a non-interactive backend. A common choice for such is Agg (for Anti-Grain Geometry engine) which can produce PNG files, Cairo and Gdk are other options. Use would be something like:

import matplotlib
# This needs to be done *before* importing pyplot or pylab
matplotlib.use('Agg')
import matplotlib.pyplot as plt

#Do your plotting, e.g.
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(range(10))
fig.savefig('test.png')

For more information, see: - Matplotlib Documentation on running without a GUI

Using python with MPI

If you wish to take advantage of the multiple cores and even many nodes available on High Performance Computing (HPC) clusters, it is useful to use the Message Passing Interface (MPI) for coordinate and communicate among the various processes, a standard and ubiquitous programming methodology for distributed memory parallelism.

There is a package mpi4py available on all Pythons included as part of the Research Software Library on UMD HPC clusters which basically makes the various MPI calls available to python code. Because mpi4py basically mimics the function calls in the standard MPI library/API, it makes the task of transcribing algorithms from python to/from C much easier.

When you have python code (e.g. my-mpi4py-script.py) designed to use MPI via mpi4py, you will normally wish to execute the python code using the mpirun command. It is important that you use the mpirun command from the SAME MPI library as was used to build mpi4py for the python version you are running --- typically this will mean using module load to load the correct gcc compiler and openmpi version as used in building the python interpretter and modules, as listed in the package version information table. E.g., a job submission script to launch my-mpi4py-script.py on 200 cores using python/3.10.10 might look like:

#!/bin/bash
#Assume will be finished in no more than 8 hours
#SBATCH -t 8:00:00
#Launch on 200 cores distributed over as many nodes as needed
#SBATCH -n 200
#Assume need 4 GB/core (4000 MB/core)
#SBATCH --mem-per-core=4000

#Make sure module cmd gets defined
source /etc/profile

#Load required modules
module load python/3.10.10
#The module loading of python will load the correct compiler
#We can leave the version of openmpi off and module will load
#the correct version for the loaded compiler/python
module load openmpi

#Normally do not need to give -n 200, as openmpi will 
#determine from Slurm environment variables
mpirun mp-mpi4py-script.py

Although exploring mpi4py is beyond the scope of this document, we do provide some on-line tutorials, etc., to help if you wish to explore mpi4py further: