Skip to content

FF chevron

Hybrid Shared/Distributed Parallel Job without MPI

This page shows an example of submitting a job that uses a hybrid parallel job which uses both distributed memory parallelization and shared memory parallelization. It uses multithreading for the shared memory parallelization, but unlike the hybrid multithreading+MPI case, the distributed memory parallelization does not use MPI. Basically the job uses the ssh command to launch the different tasks/processes, potentially across multiple nodes.

Most packages which support distributed memory parallelism do such via MPI, as the MPI libraries provide support passing data between tasks, which is needed for all but trivial cases (like this example). However, a small number of packages use some alternative mechanism for distributed memory parallelism, and the following example shows how a job for such a package might be launched. (It does not discuss communication between tasks as such is quite complicated and application specific, and well beyond the scope of this document).

Overview

The example shown is a rather trivial Hello World type program in which each thread of each tasks prints out a "Hello" message identifying the thread and task number (along with hostname and process ID). Although trivial, it shows the basics of submitting such a job, along with various other good practices, including:

  • setting standard sbatch options within the script
  • loading the needed modules within the script
  • printing some useful diagnostic information at start of the script
  • creating a job specific work directory
  • running the code and saving the exit code
  • exiting with the exit code from the main application

Many of the practices above are rather overkill for such a simple job --- indeed, the vast majority of lines are for these "good practices" rather than the running of the intended code, but are included for educational purposes.

The submission script

The submission script can be downloaded as plain text, suitable for you to submit to the scheduler with the sbatch command. We present a copy with line numbers for discussion below.

Source of submit.sh

The line numbers in the listing below are to facilitate a line-by-line discussion pf the script; they should not present in the actual script. Downloaded as plain text version of the script.

Example

1   #!/bin/bash
2   # The line above this is the "shebang" line.  It must be first line in script
3   #-----------------------------------------------------
4   #   Example Submission Script: Hello-UMD, Hybrid Ssh version
5   #   Runs a multithreaded hello-umd command using ssh to span nodes
6   #-----------------------------------------------------
7   #
8   # Slurm sbatch parameters section:
9   #   Request 3 "tasks" with 15 CPU cores each
10  #SBATCH -n 3
11  #SBATCH -c  15
12  #   Request 5 minutes of walltime
13  #SBATCH -t 5
14  #   Request 1 GB of memory per CPU core
15  #SBATCH --mem-per-cpu=1024
16  #   Allow other jobs to run on same node
17  #SBATCH --oversubscribe
18  #       Run on debug partition for rapid turnaround.  You will need
19  #   to change this (remove the line) if walltime > 15 minutes
20  #SBATCH --partition=debug

21  # This job will run the hello-umd with multiple "tasks" each using 
22  # multiple threads.  The tasks are launched using ssh instead of MPI.
23  # We create a directory on parallel filesystem from  where we actually 
24  # will run the job.

25  # We start by defining a function to launch the program on every
26  # node allocated to the job (and for each node, as many times as
27  # the number of tasks allocated to the node).  
28  # The use of this function makes the code clearer, plus allows us
29  # to easily capture the output to a single file.
30  launch_ssh_tasks()
31  {   # First, get the full path to our hello-umd executable.  It is best
32      # to provide the full path of our executable to ssh, etc.
33      MYEXE=`which hello-umd`
34      echo "Using executable $MYEXE"

35      # Now we get a list of hostnames for nodes our job is running
36      # on.  We want each hostname to appear in the list exactly as
37      # many times as the number of tasks allocated to the node.
38      # This information is in the Slurm environmental variables
39      # SLURM_JOB_NODELIST and SLURM_TASKS_PER_NODE, but in an abbreviated
40      # form.  We call a helper script to convert to the format we want.
41      MYNODES=`/software/acigs-utilities/bin/slurm_hostnames_by_tasks \
42          -r ' ' -m hosts_by_tasks`

43      # Keep track of our task number and the pid for each task
44      tasknum=0
45      pidlist=()

46      # We now use ssh to spawn a copy of hello-umd for each task 
47      # for our, using $SLURM_CPUS_PER_TASK threads per task
48      for node in $MYNODES
49      do
50          # We run in background to get parallelism
51          ssh -q $node $MYEXE \
52              -t ${SLURM_CPUS_PER_TASK} \
53              -m "'Hello from task $tasknum'" &
54          # Collect the process id (pid)
55          pidlist[$tasknum]=$!
56          # and increment tasknum
57          tasknum=$(( tasknum + 1 ))
58      done

59      # Wait for all of our background jobs to complete and if any
60      # had non-zero exit codes, warn us and set the main script's 
61      # exit code to non-zero
62      ECODE=0
63      tasknum=0
64      for pid in "${pidlist[@]}"
65      do
66          wait $pid
67          tmpecode=$?
68          if  [ $tmpecode -ne 0 ]; then
69              ECODE=$tmpecode
70              echo >&2 "Task $tasknum had non-zero exit code $ECODE"
71          fi
72          tasknum=$(( tasknum + 1 ))
73      done
74      return $ECODE
75  }

76  # Process our dot files (and in particular define the "module" command)
77  source /etc/profile

78  # Module load section
79  # First clear our module list 
80  module purge
81  # and reload the standard modules
82  module load hpcc/zaratan
83  # Load the desired compiler and package modules
84  module load gcc
85  module load hello-umd

86  # Section to make a scratch directory for this job
87  # Because different tasks might be on different nodes, and will need
88  # access to it, we put it in a parallel file system.  
89  # We include the SLURM jobid in the directory name to avoid interference 
90  # if multiple jobs running at same time.
91  TMPWORKDIR="${HOME}/scratch/job.${SLURM_JOB_ID}"
92  mkdir $TMPWORKDIR
93  cd $TMPWORKDIR

94  # Section to output information identifying the job, etc.
95  echo "Slurm job submit-hello-umd-hybridssh.sh ${SLURM_JOB_ID} running on:"
96  hostname
97  echo "Ntasks=${SLURM_NTASKS}; CPU cores per task=${SLURM_CPUS_PER_TASK}"
98  echo "On ${SLURM_JOB_NUM_NODES} nodes: ${SLURM_JOB_NODELIST}"
99  date
100 pwd
101 echo "Loaded modules are:"
102 module list
103 echo "Job will be run in $TMPWORKDIR"

104 # Launch our code using the previously defined function, saving output
105 # to hello.out
106 launch_ssh_tasks > $TMPWORKDIR/hello.out 2>&1
107 # and save exit code
108 ECODE=$?

109 echo "Job finished with exit code $ECODE.  Work dir is $TMPWORKDIR"
110 date

111 # Exit with the cached exit code
112 exit $ECODE

Discussion of submit.sh

The following is a detailed, more or less line-by-line discussion of the job submission script:

  • Line 1: This shebang line specifies that the script is to be processed using the bash shell.
    For more information on the shebang line.
  • Lines 2-7: These are comment lines, which are ignored by the bash shell.
    They are only present to make the script more readable to humans. For more information on comment lines.
  • Lines 8-20: These lines specify various flags to be passed to the sbatch command. For more information about #SBATCH directives.
  • Lines 10-11: As this is a hybrid job, we request a three MPI takss with each task having 15 threads, for a total of 45 CPU cores. All 15 cores for the same task will be allocated on the same node; the cores for different tasks might or might not be allocated to the same node. For more information about core, task, and node settings
  • Line 13: This is a trivial job, so we only request 5 minutes for it. It should finish in well under 5 minutes. For more information about maximum walltime settings
  • Line 15: This is a trivial job which does not need much RAM, so we only request 1 GiB per CPU core (so 45 GiB in total). Normally jobs needing no more than 4 GiB per CPU core can just omit this and get the default value of 4 GiB per core. For more information about memory settings
  • Line 17: This #SBATCH directive says that we are allowing other jobs to share the same node(s) as our job. This is the default (so this directive can usually be omitted) and is recommended, especially for jobs using only a fraction of the cores available on a node (128 cores per node on the Zaratan cluster), otherwise a large number of the cores on the node will not be used by your job nor will the scheduler allow other jobs to use them. For more information about exclusive/shared mode.
  • Line 20: This tells the scheduler to put our job in the debug partition. Since this is a short tutorial job, the debug partition is fine and generally will allow the job to spend less time in the queue. For more information about selecting partitions.
  • Lines 25-75: These lines define a shell function which will actually handle the launching of the various tasks. (Shell functions are one reason we recommend bash over some other shells (e.g. tcsh) which lack such features.) We use a shell function because:
    • It makes the code easier to read
    • It allows us to have all of the output redirected to a single file These lines just define the function; the code in these lines are not actually run (i.e. the function is not actually invoked) until line 105.
  • Line 33: This line uses the unix which command to get the full path to the hello-umd command. It is best to provide the full path to the ssh command.
  • Line 33: This lines uses the unix which command to get the full
  • Lines 41-42: These lines determine which nodes were allocated to our job. In particular we generate a list of hostnames of the nodes allocated to our job, wherein each node occurs in the list exactly as many times as the number of tasks allocated to that ndoe. This enables us to launch a hello-umd process for every task we requested.

    The information we want is contained in the environment variables set by the Slurm scheduler: SLURM_JOB_NODELIST and SLURM_TASKS_PER_NODE, but Slurm uses an abbreviated form in these variables. So we use a helper script installed at /software/acigs-utilities/bin/slurm_hostnames_by_tasks to do the work of converting the contents of these variables to the more useful form described above. The argument -r ' ' causes the node names to be separated by a space (which makes it easy to use in a bash for loop), and the argument -n hosts_by_tasks causes it to print a list of hostnames with the hostnames occuring once for each task allocated to the node.
    The "\" character at the end of first line is a continuation character --- it allows us to split the command across two lines but still have the shell interpret it as a single command.
  • Line 44: This line initializes the variable tasknum for our loop. This is mainly used so that we can change the default message of hello-umd to show our task number (as we are not using MPI.
  • Line 45: This line initializes the variable pidlist to store the list of process ids (pids) for each of our child processes, which we use later on to collect the exit codes.
  • Lines 48-58: These lines define a loop over the elements of MYNODES, which was defined in line 41 to contain a list of hostnames assigned to the job, with hostnames repeated to indicate the number of tasks assigned to the node. So by looping over the values of MYNODES we are looping over taks, with node being the name of the node the task is allocated to.
  • Lines 51-53: These lines invoke the ssh command to run an instance of hello-umd on the node from the MYNODES list. The hello-umd command is supplied the arguments -t $SLURM_CPUS_PER_TASK to tell that instance of hello-umd to spawn the correct number of threads, and the -m flag to change the default message (in order to indicate thread number, as the hello-umd command cannot determine that since we are not using MPI).

    The "\" character at the end of first two lines is a continuation character --- it allows us to split the command across three lines but still have the shell interpret it as a single command.

    The "&" character at the end of last line causes the command to be run in the background --- this means that the script continues immediately with the command following the ssh command while the ssh command (and the hello-umd command it invoked) are still running. This is needed in order to get the required parallelism; without it, only one instance hello-umd would be running at a time --- the script would use ssh to spawn one instance of hello-umd on the first node, wait for it to complete, and once it is done continue the loop and spawn an instance on the second node, etc. So using the & character to run the ssh commands in the background is needed for parallel rather than sequential operation.
  • Lines 55: This line stores the process id (pid) of the ssh command we just ran in the background, using the special shell variable $!. We need this value so that we can later wait for the process to finish and collect its exit code.
  • Lines 57: This line increments the value of tasknum, which we use in the -m argument to hello-umd in line 53.
  • Line 58: This ends the loop started in line 48.
  • Line 62: This line defaults the ECODE variable to 0 (success).
    We use this to collect the final exit code for the job script.
  • Line 63: This line defaults the tasknum variable to 0. We use this to track the task number for diagnostic messages.
  • Lines 64-73: These lines wait for all of the tasks spawned in the looped ssh command from line 51 and collect their respective exit codes. A non-zero exit code for any task will be stored into ECODE. If this step were omitted, the bash function, and therefore the entire job script, would return almost immediately after they started, and almost certainly before the spawned commands finished. This wil cause the scheduler to interpret the job as as having completed, and revoke the resources allocated to it, terminating the spawned tasks shortly after they started --- not what is wanted.
  • Line 66: This line waits until the process with the identifier stored in the variable pid is finished, setting the exit code of the wait command to that of the process waiting on. If the process had finished before the wait command runs, it does not wait and just sets the exit code.
  • Line 67: This line stores the exit code from the spawned task into the temporary variable tmpecode.
  • Lines 68-71: These lines check if the exit code from the spawned task was zero (indicating success). If not, the script sets the ECODE variable to this error exit code and prints a warning.
  • Line 72: This line increments the tasknum variable
  • Line 74: This line returns from the bash function, with the exit code we stored in ECODE
  • Lines 77-85: These lines set up the environment within which to run your job. These are explained in more detail in the main job examples page, but in this case the lines basically:
    • Ensure the module command is defined
    • Clear out any previously loaded modules (inherited from .bashrc or similar)
    • Load the modules for compiler and hello-umd. No MPI library is loaded because this example does not use MPI.
  • Lines 86-93: These lines set up a work directory for our job, in this case since we are potentially running on multiple node a shared filesystem is appropriate, so we use a work directory in scratch. For more information about setting up work directories.
  • Lines 94-103: These lines print out various pieces of information to identify the job. The output will go into the file "slurm-JOB_NUMBER.out" in the directory from which you issued the sbatch command (where JOB_NUMBER is the Slurm job number). There are options you can pass to sbatch to change the name of the file.
  • Line 106: This line actually runs the hello-umd command, saving all output to the file hello.out in the working directory created in lines 86-93. I.e., this one line is the meat of the job script; everything else basically just exists to get to this point or cleanup afterwards.

    When writing your own job script to do something useful, this is where you will be putting most of your effort, possibly expanding to multiple lines.

    In this simple case, we basically just run the hello-umd command with the -t $SLURM_CPUS_PER_TASK argument. The environmental variable SLURM_CPUS_PER_TASK contains the value of the --cpus-per-task argument from line 14 to ensure the correct number of threads are run in each MPI task. The "> hello.out" part means that the standard output from the command will be saved in the file hello.out in the working directory. The "2>&1" part means that any output to the standard error output stream will be redirected to the standard output. (NOTE: if you wish to save both standard output and standard error to the same file, you need to give these output redirection arguments in that order; redirect standard output first, then redirect standard error to standard output.)
  • Line 108: This line saves the exit code from the hello-umd command so thatt be used when the script ends.
    For more information about exit codes.
  • Lines 109-110: These lines print some additional information just before the job terminates.
    These lines will go in the same file as those from the start of the script, usually "slurm-JOBNUMBER.out".
    This just prints out:
    • The exit code from the hello-umd command (as stashed in the ECODE environment variable)
    • The time when the job finished
  • Line 112: This line actually ends the job script, and terminates the job. The script and job will actually end at the end of the file even without an explicit exit command, but in that case the exit code for the job script will be that of the last command which ran, which typically will be part of the cleanup commands, and not the meat of the script.

    Here we explicitly set the exit code to the exit code from the hello-umd command (which we previously saved in the variable ECODE).

Submitting the example

To submit the script, you can just: - Download the script as plain text and copy to one of the cluster login nodes. Move it to an appropriate directory. - From the directory with the job script, run the command sbatch submit.sh (replacing submit.sh with the name of the script you downloaded). The scheduler should return something like Submitted batch job 11274515 --- the number will vary and is the job number for this job). The job number can be used to reference the job in various Slurm commands, etc. (Please always give the job number(s) when requesting help about a job you submitted).

The job will be queued for partition specified (or defaulted to); in this example we specified the the debug partition so it will be in that partition. It should typically start running within 15 minutes or so, and finish shortly after it starts (as it is a rather trivial job).

When it finishes running, you should be able to find the file "slurm-JOBNUMBER.out" in the directory from which you ran the sbatch command which should contain the output of the diagnostic commands (the job number, when it started, what hosts running on, when it ended and exit code, etc), something like:

Example

Slurm job submit-hello-umd-hybridssh.sh 11567838 running on:
compute-b8-57.zaratan.umd.edu
Ntasks=3; CPU cores per task=15
On 3 nodes: compute-b8-[57-59]
Fri May  2 15:53:01 EDT 2025
/home/payerle/scratch/job.11567838
Loaded modules are:
Currently Loaded Modulefiles:
 1) hpcc/zaratan(default)   3) hello-umd/gcc/11.3.0/nompi/zen2/1.5
 2) gcc/11.3.0
Job will be run in /home/payerle/scratch/job.11567838
Job finished with exit code 0.  Work dir is /home/payerle/scratch/job.11567838
Fri May  2 15:53:04 EDT 2025

Similar, in the working directory of the job (normally "~/scratch/job.JOBNUMBER") the file "hello.out" should contain the output of the actual "hello-umd" command, something like:

Example

Using executable /cvmfs/hpcsw.umd.edu/spack-software/2023.11.20/linux-rhel8-zen2/gcc-11.3.0/hello-umd-1.5-o2uyvdepdbpr6orl56p6qpfcem53357l/bin/hello-umd
hello-umd: Version 1.5
Built for compiler: gcc/11.3.0
Hello from task 0 from thread 4 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 10 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 2 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 8 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 0 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 6 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 14 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 12 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 3 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 7 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 1 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 11 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 13 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 5 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
Hello from task 0 from thread 9 of 15, task 0 of 1 (pid=3026615 on host compute-b8-57.zaratan.umd.edu
hello-umd: Version 1.5
Built for compiler: gcc/11.3.0
Hello from task 1 from thread 0 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 2 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 10 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 13 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 9 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 8 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 11 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 1 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 14 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 3 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 12 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 6 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 5 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 4 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
Hello from task 1 from thread 7 of 15, task 0 of 1 (pid=1596832 on host compute-b8-58.zaratan.umd.edu
hello-umd: Version 1.5
Built for compiler: gcc/11.3.0
Hello from task 2 from thread 0 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 4 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 8 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 11 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 7 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 3 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 5 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 1 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 14 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 6 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 9 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 2 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 13 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 10 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu
Hello from task 2 from thread 12 of 15, task 0 of 1 (pid=2606183 on host compute-b8-59.zaratan.umd.edu

In both cases, some of the specifics (e.g. job numbers, timestamps, pids, and node names) will differ from run to run. Also note that the ordering of the lines from various tasks and threads will vary from run to run.