Skip to content

FF chevron

Installing Your Own Software

While the HPC team has installed many software packages for the use of researcher in the Research Software Library (RSWL), researcher are permitted to install software packages into directories which you can write to. Reasons why you might need or wish to install your own software packages can include:

  • The code is your own
  • You need to make personal modifications to the code, especially if anticipate making repeated modifications that will
  • You need a non-standard version of a code, especially one which would conflict existing libraries, etc.
  • The package is not ameniable to a system wide installation; i.e. it is designed in such a way that really promotes personal installations
  • You desire the package sooner than HPC staff anticipates being able to provide it. HPC staff has a limited amount of manpower, and many responsibilities. Installation of research software will only be performed as time and other responsibilities allow.

Where to Install Software

Warning

Users are not allowed to install software at the system level.

In particular, users are not allowed to use the "su" or "sudo" commands. Those commands are for providing system level access, which you do not have permission for, nor do you need for installing software into directories for which you have write access.

The only exception to this is when you are executing a shell in an apptainer container with the --fakeroot flag. In such a case, you will be able to use the standard binary package installation commands (e.g. rpm, dnf, apt, etc., depending on the flavor of Linux installed in the container), and you will not need the "su" or "sudo" commands.

As you are only permitted to install software to a directory to which you have write permission, that basically limits you choices to:

  • your home directory: as this directory is strictly limited to a 10 GB quota, this location generally does not have enough space for software installations.
  • the scratch storage for one of the projects to which you belong is a good choice. In particular:
    • a subdirectory of your per-user scratch directory for the appropriate project, e.g. ~/scratch* is suitable for software which is for your personal use.
    • a subdirectory of the shared folder of your project's scratch directory is a good choice for software to be used by multiple members of the project
  • as the SHELL storage tier is not accessible from the compute nodes, this is not generally a good choice for software installs.

So generally we recommend installing software to scratch space. As software installations are typically only on the order of a few GB or so, we usually will ignore such when enforcing policies regarding scratch use. However, there are some factors that should be considered:

  • while the scratch tier is optimized for high performance with large files, it can have issues with lots of small files.
    • this can be a serious problem with conda environments; we recommend conda users consider use conda-pack.
    • Source code also can involve lots of small files. Many build systems allow one to build the code in one directory and have it installed in another (often with the use of "--prefix" or similar flags) --- if your package supports that, you should consider downloading the source files to SHELL storage and building the package in SHELL, with the software being installed to your scratch space.
      • As the source code and temporary build files are generally only needed at run time, putting them in SHELL is fine as you can build the code on a login node.
      • Only the installed files are needed at run time, and these will be in scratch which is available to the compute nodes.
      • This saves your more precious scratch space, as well as avoids some of the small file issues on scratch.
  • the scratch tier is not backed up by HPC staff; while HPC staff does not currently purge files from scratch, hardware failures and mishaps do occur.
    • we recommend making backup copies of the configuration and/or similar information to other locations. As this is typical relatively small, it can usually fit in your home directory. You might also wish to copy off cluster as well.
      • For compiled codes, you should generally save the exact configuration and build commands used. I.e., the arguments passed to the "./configure" or "cmake" commands.
      • For conda environments, you might wish to save the output from the conda env export command
    • Should the software installed in scratch be deleted or become corrupted, these files should enable you to quickly reinstall the packages.
    • Source codes are generally available from the web, in which case that typically will not need to be backed up.
      • If you need to make modifications to the source code as part of the build process, you should back such up. The patch command might be useful in this case (tracking just the changes that were made).

Under Construction

This page is still under construction. We hope to add additional content/topics soon.