visit: interactive, scalable, visualization, animation and analysis tool

• login.deepthought2.umd.edu for the Deepthought2 cluste (RHEL8)
• rhel6.deepthought2.umd.edu for the Deepthought2 cluste (RHEL6)
• login.juggernaut.umd.edu for the Juggernaut cluster

• login-*.deepthought2.umd.edu for Deepthought2 (RHEL8 and RHEL6)
• login-*.juggernaut.umd.edu for Juggernaut

1. serial: this runs ViSit in one process on the login node.
2. parallel (debug partition): this will run ViSit in a job submitted to the debug partition. I.e., a short job, but run at somewhat higher priority for better interactive use.
3. parallel: this will run ViSit in a more generic job. You can specify the number of cores/nodes/etc.

1. Click the Launch parallel engine checkbox.
2. Click the Parallel launch method checkbox and select sbatch/mpirun in the drop down (probably the last entry).
3. For the parallel (debug partition) profile, also click the Parition/PoolQueue checkbox and enter debug in the text box. For the generic parallel profile, you are probably best just leaving this unchecked/blank.
4. You can adjust the Number of processors value to the desired default value. You will be able to adjust this each time you use the profile, but this will be the default value. I recommend a value of 20 for Deepthought2 and 28 for Juggernaut, as this is what is typically available on a single node.
5. For the next 4 items (Number of nodes, Bank/Account, Time limit, and Machine file), if you check the checkbox you can set a default which can be modified each time you use the profile. If left unchecked, you will not be able to modify when using the profile, and it will default to whatever sbatch decides. I would recommend checking the boxes for Number of nodes, Bank/Account and Time Limit, but typically Machine File can be left unchecked.
6. There is also an Advanced subtab just under the Launch parallel engine checkbox. You normally do not need to set anything here, but for some more complicated cases they might be needed.
   • The Use VisIt script to set up parallel envionrment checkbox should be checked (this should be by default).
   • The remaining arguments can typically be left unchecked except in certain cases. I would recommend that if you need to make modifications here, you create (or copy) a new profile for that specific need. E.g., if the default memory requested for the Slurm job is insufficient, you can create a "parallel-large memory" profile with everything copied from the "parallel" profile and then make the needed changes.
     • Launcher arguments: Here you can provide additional arguments to be passed to the sbatch command. E.g., to request 9 GB of RAM per CPU core (instead of the default 6 GB) you could add here something like --mem-per-cpu=9000.
     • Sublauncher arguments: Here you can provide additional arguments to be passed to the mpirun command.
     • Sublauncher pre-mpi command: Here you can provide a command to be run before the mpirun command in the batch script.
     • Sublauncher post-mpi command: Here you can provide a command to be run after the mpirun command in the batch script.

In this section we will briefly describe how to use the profiles. I am assuming you have a Host Profile for one of the Deepthought2 or Juggernaut clusters, with three launch profiles as described above, and that you have access to use the HPC cluster you have the profile for.

NOTE: I believe the version of VisIt that you are running on your workstation must match what is available on the cluster, at least down to the minor (second/middle) version number. If you do not have that on your workstation, you can try running VisIt on the login node with the display going back to your workstation, but having the heavy work being done on the compute nodes using the client/server model discussed here.

In general, using ViSit in client/server starts with opening the data file. Just start to open a file as usual, but in the Host dropdown at the top of the file open dialog, there should be an option for the various host profiles you have defined. Select the appropriate host profile. It will likely prompt you for a password (make sure the username given, if any is correct, and correct it if not) (If no username is given, it assumes your username on the workstation is the same as on the remote cluster). Within a few seconds you should see a file list corresponding to your home directory on the cluster. You can then select a file as usual.

If multiple launch profiles exist for that host, you will be given an option of choosing which profile you wish to use, and what options you wish to use with that launch profile if it supports any. If there is only a single launch profile, you obviously cannot choose a different launch profile, but a pop up will appear if there are any options for that launch profile. Otherwise, visit will just launch the profile with the defaults.

If you just wish to use ViSit on a file that resides on the HPC cluster (without copying the file to your local workstation) but do not need (or cannot use) the parallel capabilities of ViSit, the serial option is the easiest, and does not take additional options. Just select it and hit OK. It may take a couple of seconds to start the remote engine, but then should return and you can visualize your data as if it were local.

To use one of the parallel profiles, just select it after selecting the file. The parallel (debug partition) is good for a short interactive visualization, but is limited in number of processes/nodes and to 15 minutes. However, since it uses the debug partition, it generally will spend less time waiting in the queue. The generic parallel profile is less restrictive, but depends on jobs submitted via sbatch and can have significant wait times before the job starts running.

When you select the profile, you typically will have the opportunity to change the defaults for wall time, number of nodes, and allocation account to which the job is to be charged. NOTE: ViSit seems to assume 8 processes per node by default, so e.g. if you request 20 processes on Deepthought2, it will try to spread them over 3 nodes. I strongly advise manually setting the number of nodes appropriately. Note also that the memory of the node is split evenly over all the ViSit processes on the node, so you might need to adjust the node count to use more than the minimal number of nodes in cases where memory requirements are higher.

When you finish updating options and hit "OK", your ViSit GUI will ssh to the login node for the cluster and submit a batch job requesting the desired number of nodes/cores. Typically you will see a pop up showing that ViSit is awaiting a connection from the compute engines --- this will not occur until after the batch job starts. For batch jobs submitted to the debug partition, this should typically be within a minute or two, but it is likely significantly longer for the general parallel profile.

When the job starts, after 20 seconds or so the connection should be made and the pop up will go away. At this point you can use ViSit as normal.

At some point, the scheduler may terminate your compute engines (e.g. due to exceeding walltime). You should be able to continue using the GUI, and when you try to do something that requires the compute engine, a pop up will appear allowing you to start up a new launch profile.

Example of using parallel visit in client/server mode

Here we provide a quick example of using VisIt with parallel processing in client/server mode. This example assumes that you have already downloaded the appropriate standard profile above and placed it in the proper VisIt configuration directory. We are going to assume you going to run VisIt from the cluster login node, and so put the host profile in your ~/.visit/hosts directory on the cluster.

1. Ssh to the cluster, and module load visit/VERSION.
2. Startup visit with the visit command.
3. From the main control window, under the File tab, select Open File.
   1. This will open the File Open dialog.
   2. On the top line (Host), click on the drop down to the right. You should see an option for UMD CLUSTERNAME Cluster matching the profile you downloaded. If not, you did not download the profile or did not put the profile file in the correct location. Exit VisIt, fix the issue with the profile, and restart VisIt. If you see it select it.
   3. The dialog will gray out for a couple of seconds as VisIt ssh-es to the login node and looks at your home directory. In general, it will ask for a password (and if it does, verify the username is correct), but since you presumably just logged into the login node, your Kerberos tickets should still be valid and it should just ssh in w/out requiring a password. After ssh you in, it should display a file browser for your home directory.
   4. We are going to load one of the VisIt example files, so in the second line, labelled Path, replace the existing text with /software/visit for the Deepthought2 or Juggernaut cluster (for the RHEL6 nodes on Deepthought2, use /cell_root/software/visit). After the file browser window updates, descend down the directory named after the VisIt version and related subdirs (e.g. 2.10.2/osmesa/sys/data on Deepthought2 or 2.13.2/osmesa/linux-rhel7-x86_64/data on Juggernaut (since we are just looking for an example file, the version/builds do not need to exactly match what you are running).
   5. Double click on the multi_ucd3d.silo file. You can try other files if you want, but only the multi_* files are multidomain datasets --- if you open a file not starting with multi_*, it will be a single domain dataset and you will not be able to effectively use the parallelism in VisIt.
   6. You will now get the dialog for selecting options on the cluster. Basically, a list wherein you can select one of the launch profiles for the cluster. Click either parallel (debug partition) or parallel (generally parallel (debug partition) will give you quicker turn around for this simple example). Set the time limit to something reasonable (like 15 minutes), and select the number of cores and nodes you want. The multi_ucd3d.silo file I believe has 36 data domains, so it is probably best to choose a number which divides 36 evenly. Be sure to select an appropriate number of nodes (there are 20 cores/node on Deepthought2, 28-40 on Juggernaut). More than 36 cores is wasteful.
   7. Click OK when satisfied. A window should come up showing that it is waiting for the compute engine to start up. This will cause a batch job to be submitted to the scheduler to run your compute engine. It should take only a few minutes if you are using the debug queue. If it is going on for a long time, you might wish to check if there is a job submitted on your behalf on the cluster (name should be something like visit.USERNAME, and the number should be in the terminal from which you launched visit.
4. Shortly after the job for the compute engine starts, you should be back to the normal VisIt GUI.
5. To finish our example, define a procid scalar.
   1. From Controls menu, select Expressions
   2. Click New, and enter procid in the Name field.
   3. Click Insert function, go to the Miscellaneous submenu item, and click on procid
   4. Click Insert variable, go to the Meshes submenu item, and click on mesh1.
   5. The definition box should now display procid(mesh1). If not, fix it so it does. When done, click Apply and Dismiss.
6. Now in the Plots section, click on the Add button, and in the Pseudocolor submenu select procid. If no procid appears, you did not define the procid scalar properly above.
7. Now in the Plots section, click Draw.
8. You should see a multicolored 3D half cylinder (if you chose the multi_ucd3d.silo file; other multi_* files will be different shapes but still multicolored. Single domain files will be a solid color). The different colors represent the processor id of the processor responsible for rendering that section of the data.
9. Congratulations. You just used VisIt in parallel mode.