Advanced Connectivity

This page discusses some more advanced topics related to connecting to the HPC clusters, including password-less login and displaying graphics.

Most users will not need to use any of the items discussed on this page, and new users and/or users not comfortable with Unix/Linux will probably want to skip these topics until they are more familiar with the system.

---

Configuring SSH Public Key Authentication

Warning

NOTE: If you login to the HPC cluster using public-key authentication, you will not have Kerberos tickets or AFS tokens, so certain commands will not work. For example, you will not be able to access the contents of your SHELL directory as the underlying filesystem uses AFS tokens for authentication.
If you do find yourself needing such, you can just issue the renew command (which will prompt you for your password, defeating the goal of "password-less" ssh, etc).

We recommend using kinit on your workstation to achieve a mostly "password-less" ssh capability which will provide you with Kerberos tickets and AFS tokens after ssh-ing into the cluster.

This section discusses how to setup, configure and use SSH with public-key based authentication to allow the use of ssh without entering a password for every ssh session. It can be used to allow passwordless ssh between the various login and compute nodes, which is likely required if you are using ssh to spawn processes on the different nodes allocated to your job, and is required for certain software packages (e.g. ANSYS) to launch jobs over multiple nodes. It can also be used for password-less ssh from your workstation to the login nodes, although we strongly recommend using kinit on your workstation instead, as that is more secure and will provide you with the Kerberos tickets and AFS tokens needed to access to the SHELL storage tier.

Tip

The procedures listed in this section are not required for you to access the system, and you can use normal password based authentication instead.

These procedures are also not generally required for most multinode jobs (using MPI or srun).


If you are new to Unix/Linux, it is recommended that you skip this section and just use password based authentication.

Public-key authentication uses asymmetric encryption for its security. In asymmetric cryptographic systems, there exist distinct private and public keys. Data can be encrypted and/or digitally signed with the private key and can only be decrypted/signature verified with the public key.

For public-key authentication, the private key is securely kept on the local system (the system you are ssh-ing from), and the public key is copied to the remote system (the system you are ssh-ing to). When you ssh to the remote system, the ssh client on local system digitally signs a standard message and sends this to the sshd server process on the remote system along with the appropriate public key. The remote sshd process verifies that the public key is authorized for use with public key authentication for the specified login account, and if so verifies if the digital signature is valid. If the signature is valid, that means the ssh client is in possession of the private key corresponding to the public key. So if the public key is authorized and the signature matches, access is granted to the remote system.

Warning

NOTE: the public-key authentication process grants login access to anyone/any system/etc. that knows your private key. Thus you need to ensure the security of your private key. Protect your private key at least as strongly as you would your password.

Public key authorization within the HPC cluster

In some cases, it might be necessary to enable passwordless ssh between the nodes of the cluster. The most typical such case is if you intend to use the ssh command to launch processes on multiple nodes as part of your job. Most multinode jobs use MPI and/or the srun command, and so do not require this, but some may. Because you cannot feasibly enter a password with a batch script, you need to enable passwordless ssh in such cases. Some software packages (e.g. ANSYS) use such a mechanism internally, and so require public key authentication to be set up between the nodes of the cluster.

Because your home directory is shared among all nodes in the HPC cluster, everything for this process can be done on one of the HPC login nodes. Just log into a login node on the cluster, and then:

1. Generate your host key, if needed. Host keys can have various types depending on the encryption algorithm. Key types include: ecdsa, ecdsa-sk, ed25519, ed25519-sk, and rsa. You can use any recognized type; we recommend that you stick with the default provided by the ssh-keygen command (as that will likely be a secure choice):
   1. Run the command ls -l ~/.ssh/id_* to see if you already have a public/private host key pair. These should have names like id_KEYTYPE and id_KEYTYPE.pub for the private and public keys, resp., where KEYTYPE should be one of the key types listed above:
      • If the ~/.ssh/id_KEYTYPE already exists, you should already have the public host keys and should not need to do anything else and can skip the running the ssh-keygen command, although you should still verify the ownership and permission of the file.
      • If you are having problems with passwordless ssh among the cluster nodes, you can safely regenerate the public host key with the next step. If you arranged for passwordless ssh from the cluster to other systems, you will likely need to fix the authorized_keys file on the other systems if you regenerate the HPC host key.
   2. To (re-)generate the host key, run the ssh-keygen command. You can use -t KEYTYPE to specify a specific key type, or you can just omit the option and the program will choose a reasonable default value. You can accept the default value for the name of the file in which to save the key; the public key will be stored in the same directory, with a .pub extension.
      Generally, you should not enter a passphrase (just hit return at the two passphrase prompts) --- the most common use case is to enable passwordless ssh among nodes in the cluster for use in job scripts, and that will not work if the key has a passphrase.
   3. Repeat the ls -l ~/.ssh/id_KEYTYPE command from above.
      It should now show the existance of the file. Please verify that the file is owned by you, and no one can access it but you. (Permission flags should be -rw-------.).
2. Add the public host key to the authorized_keys file.
   1. If ~/.ssh/authorized_keys already exists, you should append the contents of ~/.ssh/id_KEYTYPE.pub to the file; you can do this with the command cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys.
   2. If the authorized_keys file does not exist, yo can create it with the proper contents with the command: cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys.

Warning

Please ensure that your id_KEYTYPE file is only readable or writable by you. This host key is all that is needed to access any system which has the corresponding public key added to its authorized_keys file.

You should now be able to ssh between nodes in the cluster without entering a password. Note that you do not have access to ssh to compute nodes unless you have a job currently running on the node. Typically this passwordless ssh is only needed to ssh between nodes allocated to a job within your job script.

Passwordless ssh with Kerberos

We recognize that people will often start multiple ssh sessions to the cluster, and that typing in your password for each such session is annoying. Our recommended solution for this is to install a Kerberos client on your workstation and configure your ssh client to use GSSAPI/Kerberos authentication when connecting to the cluster. Then when you log onto your workstation for the day, you enter one command (requiring you to enter your campus password) and then for the rest of the day you can ssh into the cluster without being required to enter your password.

Tip

Once Kerberos is set up, you can also optionally configure your system to directly access SHELL storage.

The configuration process depends on the operating system of your workstation:

• For Windows systems
• For Mac systems
• For Linux systems

Setup Kerberos on Windows systems

Warning

This section is still under construction. The instructions which follow have not been fully tested, but we expect that they should at least mostly work. Please let us know if you experience any difficulties.

On Windows systems, you will need to install the Heimdal Kerberos package in order to get valid Kerberos tickets on your workstation. You also will need to install the Network Identity Manager to allow you to manage and monitor your Kerberos tickets. Both packages are available on TERPware.

1. Download the latest version of the Heimdal Kerberos package from TERPware.
2. If it does not start automatically, open the File Explorer application and go to your Downloads folder, and double click on the file Heimdal-AMD64-full-7-99-25-67.msi.
3. The installer will start with a license agreement. You should read the agreement, and if you have no objections, accept the agreement in order to continue with the installation.
4. On the next page entitled 'Custom Setup', you do not need to change any options, you can just click 'Next'. Then click 'Install' and the package installation should proceed.

Once you have installed Heimdal Kerberos, you should next install the Network Identity Manager.

1. Download the latest version of the Network Identity Manager package from TERPware.
2. If it does not start automatically, open the File Explorer application and go to your Downloads folder, and double click on the file netidmgr-heimdal-AMD64-rel-2_5_6_112.msi.
3. The installer will start with a license agreement. You should read the agreement, and if you have no objections, accept the agreement in order to continue with the installation.
4. On the next page entitled 'Choose Setup Type', select 'Typical'. Then click 'Install' on the page that follows.

Once you have both the Heimdal Kerberos and Network Identity Manager packages installed, you should start the Network Identity Manager by selecting it from the Start menu. It will appear that nothing happens, but you should notice a small box icon appear in the system tray (icons in the lower right corner of your screen.) This box shows the status of your Kerberos tickets, if the box is empty, you have none. If the box is green, you have valid tickets.
If the box is red, your tickets have expired. Note that you won't normally have to start the Network Identity Manager, as it will be started automatically when your system is rebooted.

To obtain new Kerberos tickets, click on the Network Identity Manager box and a window will pop up. In the top left portion of the window click on the icon with the yellow sunburst to obtain new Kerberos tickets. If prompted to enter your Kerberos Principal name, enter your directory ID in the 'Username' box and 'UMD.EDU' in the Realm box.
If you are shown a 'Credential Options' window, you can leave the default values alone and click 'Finish'. You'll then be prompted for your password. Enter your standard UMD directory password here. You should now see a green battery icon next to your Kerberos Principal entry, and you can close the Network Identity Manager Window. You should also notice that the box in the system tray is now green.

Kerberos tickets are generally valid for 7 days, so if your workstation is kept on for longer than that, you'll need to repeat the procedure to obtain new tickets. Also, if you reboot your machine, you will similarly need to obtain new tickets.

Although the above process will obtain Kerberos tickets for you, you still need to configure your ssh client to authenticate to the remote system using these tickets. The steps to accomplish this depends on the specific ssh client you are using.

Tip

Use the campus VPN, which will require you to perform an MFA authentication when you start it up. But after that, when connecting to the cluster you will not be required to perform MFA, and with valid Kerberos credentials you can log in without any prompts.

Setup Kerberos on Mac systems

The Kerberos client should already be installed on recent MacOS systems, so you should not need to do anything to install it.

The process to configure SSH to use Kerberos for authentication is the same on Macs as it is for linux systems.

Setup Kerberos on Linux systems

Many modern linux distributions come with Kerberos clients automatically installed. You can verify this by issuing the command which kinit --- if that returns a path to the kinit command, you should have the required packages already installed.

If the which kinit command returns an error saying something like kinit: Command not found, then a kerberos client is not properly installed on your system. You should use whatever packaging system is appropriate for your distribution (e.g. dnf or yum for RedHat and Fedora-like systems, dpkg, apt or similar commands for Debian, Ubuntu, Mint and related systems) to find and install the appropriate package. The package names are distribution dependent, but typical names something like:

• krb5-user
• krb5-workstation
• heimdal-workstation

The proper package name will usually be something like one of the above; you probably just need one package, and likely will not find all of the above packages. You should not need to install any packages with server in the name for this. The packages starting with krb5 typically use MIT's implementation of Kerberos, and the heimdal named packages use the Heimdal implementation --- for this purpose you can use either implementation, and we recommend using whichever one is best supported for your distribution.

SSH Kerberos/GSSAPI Setup (Windows)

You should first make sure that you have Kerberos set up properly- follow the steps here first, and then proceed below.

For the putty ssh client, do the following:

1. Start putty, and go to the configuration menu.
2. In the configuration menu, expand SSH in the Connection section, expand Auth, and then select GSSAPI. On this pane, make sure the two boxes Attempt GSSAPI authentication and Allow GSSAPI credential delegation are checked.
3. In the Preference order for GSSAPI libraries box, select User-specified GSSAPI DLL and click Up twice to move that entry to the top.
4. In the User-supplied GSSAPI library path, enter the following: C:\Program Files\Heimdal\bin\gssapi64.dll
5. Find the Data section under Connection in the configuration menus, and in the field Auto-login username enter your UMD username.
6. Save the configuration by clicking the Session entry in the left column, highlight Default Settings in the Saved Sessions box, and click 'Save'.

The above ssh client configuration should only need to be done once. After doing that, and assuming you have valid Kerberos tickets, you should be able to ssh into the HPC login node without an additional password prompt (although you will see a multi-factor prompt if not on the campus VPN).

SSH Kerberos/GSSAPI Setup (Linux and Mac)

The process to configure ssh to use Kerberos/GSSAPI for authentication is mostly the same for Linux and Mac systems, and is discussed here.

Once a Kerberos client is installed, you need to configure ssh to send your Kerberos credentials to the login nodes of the cluster. To do that, you can edit (or create if needed) a file named config in the directory .ssh under your home directory and add the following lines:

    Host *.umd.edu
        GSSAPIAuthentication yes
        GSSAPIDelegateCredentials yes
        GSSAPITrustDns yes 

For Macs, you will likely need to omit the GSSAPITrustDns line, as the ssh package shipped with recent MacOSes does not support that option.

The first line Host *.umd.edu specifies which hosts the configuration is restricted to, and the next two lines instruct ssh to attempt to login via the Kerberos tickets on your workstation when connecting to the named hosts. If the attempt to login via Kerberos tickets fails, it will fall back to requesting a password. If it can authenticate using Kerberos, it will also (due to the GSSAPIDelegateCredentials directive) forward your Kerberos tickets to the system you are logging into; this means that your login session on the remote system will have valid Kerberos tickets.

Without the GSSAPITrustDns directive set to 'yes', Kerberos authentication will not work when using an alias like login.zaratan.umd.edu (which will semi-randomly) expand into one of login-1.zaratan.umd.edu or login-2.zaratan.umd.edu, etc. since the hostname given to the ssh command does not match the hostname the sshd daemon is running on. Since on MacOS that option is not allowed, you will need to explicity select one of the login nodes in your ssh command, e.g.

    ssh login-1.zaratan.umd.edu

instead of

    ssh login.zaratan.umd.edu

Warning

You should not forward your Kerberos tickets to untrusted systems; although Kerberos tickets are encrypted and do not contain your password, if a malicious user can access your Kerberos tickets, they can impersonate you until the tickets expire.

Since your Kerberos credentials should not grant you access to any non-UMD system, in the snippet above we restrict ssh to only do Kerberos authentication and only forward tickets when connecting to a system in the umd.edu domain. You could change the string after the Host directive to e.g. tighten it only to a specific HPC cluster, but leaving at *.umd.edu should be safe and will cover all UMD HPC clusters as well as the Glue/TerpConnect/GRACE systems.

If a section for the desired host expression already exists, you can just add the GSSAPIAuthentication and GSSAPIDelegateCredentials lines into the existing section. You might also with to add the lines:

    ServerAliveInterval 60
    ForwardAgent yes
    ForwardX11 yes
    ForwardX11Trusted yes

The first line will help reduce SSH timeouts due to inactivity of the terminal. The various Forward* lines have to do with forwarding X11 graphical connections and similar. None of these lines are required to do "passwordless" authentication, but you might find them useful.

Accessing SHELL storage from outside the cluster

The Auristor filesystem underlying the SHELL storage tier is a globally distributed file system which can securely provide access to files and data based on Kerberos credentials. What this means is that by installing the appropriate client software on your local workstation or laptop, you can access your SHELL space as if it were a local filesystem.

The process to do this depends on the OS running on your local workstation, and we split into different sections for different OSes:

• Accessing your SHELL space from a Windows system
• Accessing your SHELL space from a Mac system
• Accessing your SHELL space from a Linux system

Note

You generally need to have already configured your system to use Kerboros for this to work.

<! the following anchor is linked to from TERPware>

Mounting SHELL on Windows

You should first make sure that you have Kerberos set up properly- follow the steps here first, and then proceed below.

Note

The following procedure is thought to be correct, but has not been extensively tested. As such, there might be some issues with the installation. If you notice any discrepencies between your installation and what is described below, and especially if you need assistance, please contact systems staff.

1. Download the latest version of the Auristor Client Installer from TERPware.
2. If it does not start automatically, open the File Explorer application and go to your Downloads folder, and double click on the file AuriStor-client-en_US-AMD64-0_0_531.msi.
3. The installer will start with a license agreement. You should read the agreement, and if you have no objections, accept the agreement in order to continue with the installation. You are responsible for ensuring your compliance with the license.
4. The next page will ask for some options.
   • Default Cell should be set to shell.umd.edu (for the SHELL storage tier on Zaratan).
   • Integrated logon should be set to Disable.
   • Cache size: keep the default.
5. The next page ("Custom Setup") gives you options for what to install.
   Just use the defaults.
6. The next page is to confirm that you really wish to install. Click the Install button to proceed.
7. The Windows OS might also pop up a confirmation window asking if you wish to install new software. If so, click the Yes button to proceed.
8. The package should be installing, and a window with a progress bar will be displayed. When done, you can click on the Finish button to exit the setup wizard.
9. You will be prompted to restart the system to complete the installation.

Now that you have the Auristor client installed, here's how to access files in your SHELL filesystem:

1. Open File Explorer and click on This PC in the left column.
2. In the bar at the top, click on the three dots ... and from the menu that appears, select Map Network Drive. This should open a Map Network Drive window.
3. There should be a drop down selector labeled Drive. You can use this to select which drive letter the SHELL folder should be mounted as.
4. Underneath that is a text box labeled Folder. This is where you provide the path to the SHELL folder you wish to mount. You should enter the full path to directory, with forward slashes (i.e. '/') being converted to backslashes ('\'), and two backslashes at the start. For example, if you want to mount your personal SHELL directory from project foo, and your username is smith, you would enter \\afs\shell.umd.edu\project\foo\user\smith. Note that you cannot use the symlink ~/SHELL or similar here.
5. You can leave the check boxes unchecked. The Reconnect at login box sounds tempting, but since you will not have Kerberos tickets at login, this will not work.
6. Click the Finish button.

This should result in the SHELL directory specified being mounted at the drive letter specified. You might wish to mount your personal directory and the share directory for the same project to different letters, or if you belong to multiple projects, mount your personal space from each project to a different drive letter.

Mounting SHELL on Mac

Warning

This section is still under construction.

You can Download the OpenAFS Client Installer for Mac from Auristor.

Mounting SHELL on Linux

Warning

This section is still under construction.

You can Download the OpenAFS Client Installer for Linux from Auristor.

Graphics

Although HPC clusters are generally most effectively used from a command-line interface, there are times when it is beneficial to be able to display graphics, either for the visualization of results of some calculation performed on the cluster, or for the use of a GUI for some application. We discuss how to do this here.

The recommended way to do this is to start an Interactive Application using the OnDemand portal. There are interactive apps already defined for a small number of software packages (e.g. Matlab). But even if there is no interactive app already defined for the software package you intend to use, you should be able to use the interactive desktop app for this --- the interactive desktop app will connect you to a GUI desktop session on one of the compute nodes through your web browser. From there you can open a terminal window, use module load or otherwise set up the environment for the software package you intend to run, and then run the package, and if it opens a GUI window inside the desktop appearing in your browser, which you can then interact with as if you were sitting at a workstation.

This is by far the easiest way to get a graphical display on the cluster. The desktop (or other interactive app) is running on a compute node with the requested resources, so you are free to start compute intensive processes in the OnDemand interactive app window. If you start a long running calculation, you can disconnect from the desktop window in your browser and the calculation wil continue to run (as long as you do not delete or kill the job in which the interactive app is running).
Later on, you can go back to the OnDemand portal and reconnect to the interactive desktop or app.

Tip

While running long jobs from an interactive desktop or app is possible and is permitted, and in some cases might even be useful, we generally do not recommend doing so. Generally using non-interactive or batch jobs will be more efficient. Since interactive apps go through the the scheduler, depending on the amount and type of resources being requested it might be hours before your job starts running. This means the onus is on you to be constantly checking for the start of the job in order to start the calculation (and the requested resources will be sitting essentially idle (but charging against your allocation account) until you do so). Similarly, once the calculation is started, you will need to keep checking for when it is finished, so that you can either start a new calculation or terminate the job (and again, resources will be sitting essentially idle but charging against your allocation for all that time).
If you convert the job to a batch job, the batch commands will start automatically immediately after the job starts, automatically move to the next command in the script once the previous one completes, and automatically terminate the job when the script finishes. This removes the need for you to be constantly monitoring the job and improves the efficiency of the job in terms of resource usage. It also generally makes the calculations more reproducible.

Alternatively, you can use the network features of the X11 Windowing System. to run a command from an ssh session on your local workstation connected to a cluster node (typically a login node), and have the graphical windows opened by that command appear on your local workstation. In order to do this, you will need to be running an X11 server on your local workstation (and depending on the OS of your workstation, you may need to install an X server as well). We discuss how to do such below. While this alternative is permitted and the procedures below should work, this procedure is significantly more complicated than the use of the OnDemand interactive apps procedure described above. Also, this typically only allows you to run graphical applications on the login node; combined with the policies about the use of login nodes this means that you are really limited to using it for simple visualization tasks only. And depending on the application being used, perhaps not even that --- we impose a per user limit on the number of processes and/or threads on the login nodes to reduce the adverse effects of users who accidentally launch compute intensive application on the login nodes; some GUI applications are so resource hungry that they will not work within those limits. We strongly recommend that users consider using OnDemand interactive apps which run on the compute nodes and therefore do not have these limitations.

This section discusses running graphical applications on the login nodes of the cluster with the graphics appearing on your desktop using the network features of the In order for this to work, however, you need to be running an X11 server on your desktop, which we discuss below. However, while the procedures described below should still work, we recommend instead that you look into using the Interactive Desktop of the OnDemand portal; this will allow you to start up an interactive graphical job on a compute node with the graphics displaying in a window on your web browser. This is usually significantly easier to use than setting up X11 as described below.

The exact procedure for installing and running an X11 server on your local workstation depends on what your desktop, etc. is, and you might wish to contact the UMD helpdesk for assistance.

• For Linux or Unix/Unix-like systems, X11 is usually available by default. You should check your DISPLAY environmental variable (e.g. echo $DISPLAY; this should return something, usually your hostname followed by a colon and a number). You might need to add a -X to your ssh command to allow ssh to forward X11 connections.
• For Macs running OS X, see this knowledge base article
• Microsoft Windows does not ship with an X11 server, but there are third party servers available, some free and some proprietarily licensed, including:
  • Cygwin/X: an open-source/free implementation
  • MobaXterm: the Home edition is free to use, provided that the user is the one who installed it (shared installations are not allowed with the Home Edition)
  • VcXsrv: an open-source/free implementation
  • Xwin32 (proprietary)
  • MKS X/Server (proprietary)
• There is also at least one java implementation, WeirdX, that is able to run as an applet in a web browser.

Some software packages have their own protocols for remote visualization, including:

• Matlab: The Matlab Parallel Server (MPS) package allows one to run Matlab on one's desktop but have computation occur on the HPC cluster. Among other benefits, you can do plotting on local machine using data generated on the cluster.
• ParaView: Paraview supports a remote visualization mode wherein you can do calculations and even rendering on compute nodes and have graphics sent back to your desktop. More information forthcoming.
• VisIt: VisIt supports a client/server remote visualization mode enabling you to visualize data on the HPC cluster filesystems and/or do rendering on the HPC compute nodes.