Examples of SLURM scripts

Here is examples of simple SLURM scripts for running serial, OpenMP and MPI jobs.

Note that a very nice introduction to running SLURM scripts exist at SCIENCE HPC SLURM.

Simple Batch Script For Running Serial Programs

Submission scripts are really just shell scripts (here we use bash syntax) with a few additional variable specifications at the beginning. These are prefixed with "#SBATCH" and otherwise use the same keywords and syntax as the command line options described for the sbatch command. The following script presents a minimal example:

#!/bin/bash
#SBATCH --job-name=isothermal     # shows up in the output of 'squeue'
#SBATCH --partition=astro_devel   # specify the partition to run on
srun  /bin/echo "Hello World!" 

Specifying Mail Notifications And Manage Output And Error Files: You can enable e-Mail notification on various events, this can be specified via the --mail-type option which takes the following values: BEGIN, END, FAIL, REQUEUE and ALL.

#SBATCH --mail-type=FAIL
#SBATCH --output=/astro/username/que/job.%J.out
#SBATCH --error=/astro/username/que/job.%J.err

The standard output and error can be written to specific files with the --output and --error commands. By default, both files are directed to a file of the name slurm-%j.out, where the %j is replaced with the job number.

Simple Batch Script For Running MPI-Parallel Jobs

Note: The following example script assumes that you submit the script from the directory where the code will be executed (the path to that directory is stored in the $SLURM_SUBMIT_DIR environment variable, and is where SLURM will execute the script from).

#!/bin/bash
#
# SLURM resource specifications
# (use an extra '#' in front of SBATCH to comment-out any unused options)
#
#SBATCH --job-name=isothermal   # shows up in the output of 'squeue'
#SBATCH --time=4-23:59:59       # specify the requested wall-time
#SBATCH --partition=astro_long  # specify the partition to run on
#SBATCH --nodes=4               # number of nodes allocated for this job
#SBATCH --ntasks-per-node=20    # number of MPI ranks per node
#SBATCH --cpus-per-task=1       # number of OpenMP threads per MPI rank
##SBATCH --exclude=<node list>  # avoid nodes (e.g. --exclude=node786)

# Load default settings for environment variables
module load astro

# If required, replace specific modules
# module unload intelmpi
# module load mvapich2

# When compiling remember to use the same environment and modules

# Execute the code
srun <executable> [args...]

Simple Batch Script For Jobs Using OpenMP Only

Note: The following example script assumes that you submit the script from the directory where the code will be executed (the path to that directory is stored in the $SLURM_SUBMIT_DIR environment variable).

Note: By choosing --cpus-per-task=40 along with --threads-per-core=2, you have assumed that your program will take advantage of hyper threading. If this is not the case, or you are uncertain, use --cpus-per-task=20 along with --threads-per-core=1, and your program will be executed with 20 threads.

#!/bin/bash
#
# Define SLURM resource specifications
# (use an extra '#' in front of SBATCH to comment-out any unused options)
#
#SBATCH --job-name=isothermal   # shows up in the output of 'squeue'
#SBATCH --time=4-23:59:59       # specify the requested wall-time
#SBATCH --partition=astro_long  # specify the partition to run on
#SBATCH --cpus-per-task=40      # total number of threads
#SBATCH --threads-per-core=2     # threads per core (hyper-threading)
##SBATCH --exclude=<node list>  # avoid nodes (e.g. --exclude=node786)

# Load default settings for environment variables
module load astro

# OpenMP affinity
# no hyperthreading
# export KMP_AFFINITY="granularity=core,scatter,1,0"
# hyperthreading
export KMP_AFFINITY="granularity=thread,scatter,1,0" 

# When compiling remember to use the same environment

# Execute the code
srun --cpu_bind=threads <executable> [args...]

Hybrid MPI + OpenMP Batch Script

Note: The following example script assumes that you submit the script from the directory where the code will be executed (the path to that directory is stored in the $SLURM_SUBMIT_DIR environment variable, and is where SLURM will execute the script from).

#!/bin/bash
#
# Define SLURM resource specifications
# (use an extra '#' in front of SBATCH to comment-out any unused options)
#
#SBATCH --job-name=isothermal   # shows up in the output of 'squeue'
#SBATCH --time=4-23:59:59       # specify the requested wall-time
#SBATCH --partition=astro_long  # specify the partition to run on
#SBATCH --nodes=32              # number of nodes allocated for this job
#SBATCH --ntasks-per-node=8     # lower than the usual 20 for MPI only 
#SBATCH --cpus-per-task=5       # number of CPUs per MPI rank
#SBATCH --threads-per-core=2     # threads per core (hyper-threading)
##SBATCH --exclude=<node list>  # avoid nodes (e.g. --exclude=node786)

# Load default settings for environment variables
module load astro

# OpenMP affinity
export KMP_AFFINITY="granularity=thread,scatter,1,0"

# If required, replace specific modules
# module unload intelmpi
# module load mvapich2

# When compiling remember to use the same environment and modules

# Execute the code
cd $SLURM_SUBMIT_DIR
srun --cpu_bind=threads <executable> [args...]