Skip to content

Quick Start Guide: Command Line

Before we begin

Make sure you have the following prerequisites.

Create an application to run

Let's make a simple python script as an application to print today's day of the week, date and time. Use a text editor to create a file named print_date_time.py in your ~/scratch directory with the following content.

Example

# This python script uses the "datetime" module to 
# print date and time.

# Import the required python modules
import datetime

x = datetime.datetime.now()
print(x.strftime("Today is %A, %b %d, %Y"))
print(x.strftime("Time now is %X"))

Let's test to make sure the python script works.

Example

login-2:~: module load python
Loading python/gcc/11.3.0/nocuda/linux-rhel8-zen2/3.10.10
  Loading requirement: gcc/11.3.0
login-2:~: python print_date_time.py 
Today is Friday, Jan 03, 2025
Time now is 13:19:57

Explanation:

  • The module load python makes the HPC cluster's Research Software Library's default version of python environment available to you. The next two lines are the output from the 'module load' command.
  • python print_date_time.py executes your script within the above python environment.
  • The next two lines are the output from the application, print_date_time.py the python script.

Write a bash script to submit the python app as a job

Now that we know the python script works as expected, we can run it as a job on the cluster. Create a file named first_submit_script.sh, again in the your scratch directory, with the following content.

Example

#!/bin/bash
#SBATCH --time=2:30
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=128

date
hostname
module load python
module list

python print_date_time.py

Explanation:

  • The first line #!/bin/bash tells Slurm that this file is in the bash scripting language. This shebang (#!) line must be the first line of the file.
  • The #SBATCH lines are directives to slurm to request resources. All of the #SBATCH lines must come before any actual bash scripting. While following are the basic descriptions of the directives used, please see man sbatch for the complete set of options.
    • --time=2:30 requests 2 mins 30 seconds
    • --ntasks=2 requests two tasks for the job
    • --cpus-per-task=2 requests two CPU cores per task (a total of four CPU cores for the job). We will show below why this is an excessive request of resources.
    • --mem-per-cpu=128 requests 128MB per CPU core (MB is the default unit)
  • The rest of the lines get run sequentially as bash shell commands. The date here is a shell command; the output will differ from that of the python script.
  • If your login shell is not bash, you should add this line after the #SBATCH lines.
    . ~/.bashrc

Submit the job to the scheduler

Use the sbatch command to submit the job to the scheduler. As a general rule, you should not start jobs from your home directory. Unlike the home directory, the "scratch" disk space is on a high-performance filesystem that is optimized for high I/O.

Example

login-2:~/scratch$ sbatch first_submit_script.sh 
Submitted batch job 9947714

Note the assigned JobID 9947714. Your own JobID will be different; for the rest of the examples, please use your own JobID.

Since we did not specify a filename for errors and output, slurm will place the output in the slurm-<JobID>.out file in the same working directory. We recommend you use this option most of the time so that the output from multiple runs of the same job script will be saved in separate files.

Example

login-2:~/scratch$ cat -n slurm-9947714.out
     1  Fri Jan  3 17:00:34 EST 2025
     2  compute-a7-41.zaratan.umd.edu
     3  Currently Loaded Modulefiles:
     4   1) hpcc/zaratan(default)                              
     5   2) gcc/11.3.0                                         
     6   3) python/gcc/11.3.0/nocuda/linux-rhel8-zen2/3.10.10  
     7  Today is Friday, Jan 03, 2025
     8  Time now is 17:00:34

Explanation:

  • cat -n adds line numbers to the display for this explanation.
  • The first line (#1) of the slurm-9947714.out file is the output from the date command running in the bash shell. Note how it's formatted.
  • The next line (#2) compute-a7-41.zaratan.umd.edu is the hostname of the node where the job was executed. Note that this is not one of the login nodes.
  • The next four lines (#3-6) are the loaded software modules--results of the module list command.
  • The last two lines (#7-8) of the slurm-9947714.out output file are from the python script--output of python print_date_time.py. Note now this date and time display is different from line #1.

Slurm commands to inspect the job

This particular test job only requests a small amount of resources for a very short amount of time, hence, it will probably be completed very quickly. Later, when you submit large jobs, you will be able to use the commands introduced in the slurm command basics page.
For instance, squeue --me would show the queued job waiting in PD (pending) state for the resources to become available. Either during the PD or the R (running) state, you'd be able to run scontrol show job <JobID> to see information about the job.

You can get job accounting information with the sacct command (use the -j option to specify the JobID as shown below).

Example

login-1:~$ sacct -j 9947714 -o JobID%15,Start,Elapsed,State,AveRSS,MaxRSS,AllocTRES%32
          JobID               Start    Elapsed      State     AveRSS     MaxRSS                        AllocTRES 
--------------- ------------------- ---------- ---------- ---------- ---------- -------------------------------- 
        9947714 2025-01-03T17:00:33   00:00:03  COMPLETED                        billing=4,cpu=4,mem=512M,node=2 
  9947714.batch 2025-01-03T17:00:33   00:00:03  COMPLETED          0          0            cpu=2,mem=256M,node=1 
 9947714.extern 2025-01-03T17:00:33   00:00:03  COMPLETED          0          0  billing=4,cpu=4,mem=512M,node=2

The two tasks requested were assigned on two different nodes (node=2; use the NodeList keyword in the -o option to get the actual node names). The job was billed for 4 SU (billing=4) for this single-threaded simple program. Such a single-threaded program only uses one CPU core, so other three were wasted. The job completed in 3 seconds (Elapsed), which was too short a time period to register the memory usage in any meaningful way.

Better resource requests for this job

  • It is a waste to request 4 CPU cores in total for something that just runs on a single core. We should use these directives instead:
    #SBATCH --ntasks=1
    #SBATCH --cpus-per-task=1
  • On the Zaratan cluster, each CPU request is granted 4GB of memory by default. (If you request more than 4GB memory per CPU core, your account will be charged as if you requested an additonal core per each 4 GB. i.e., if you request 16GB you'll be charged for 4 extra cores.)
    So, for this test job, and in most cases, skip the mem-per-cpu line.
  • If you have multiple allocations--Slurm Accounts (use the sbalance commands to check)--you will want to use the #SBATCH -A <account-name> directive to charge your jobs against.

Use the commands shown in monitoring and managing jobs to get more information about your jobs; and use that info to refine your future resource requests.

Be sure to keep an eye on your Slurm account balance by periodically running the sbalance command.

Previous: Slurm Command Basics Next: Web Portal