Tycho - User contributions [en]

Examples of SLURM scripts

2023-11-15T14:47:32Z

Philip:

Here is examples of simple SLURM scripts for running serial, OpenMP and MPI jobs.

Note that a very nice introduction to running SLURM scripts exist at [https://hpc.ku.dk/documentation/slurm.html#readme SCIENCE HPC SLURM].

<h2>
Simple Batch Script For Running Serial Programs
</h2>

Submission scripts are really just shell scripts (here we use bash syntax) with a few additional variable specifications at the beginning. These are prefixed with "#SBATCH" and otherwise use the same keywords and syntax as the command line options described for the sbatch command. The following script presents a minimal example:
<pre>
#!/bin/bash
#SBATCH --job-name=isothermal # shows up in the output of 'squeue'
#SBATCH --partition=astro_devel # specify the partition to run on
srun /bin/echo "Hello World!"
</pre>

Specifying Mail Notifications And Manage Output And Error Files:
You can enable e-Mail notification on various events, this can be specified via the <code>--mail-type</code>
option which takes the following values: <code>BEGIN</code>, <code>END</code>, <code>FAIL</code>, <code>REQUEUE</code> and <code>ALL</code>.

<pre>
#SBATCH --mail-type=FAIL
#SBATCH --output=/astro/username/que/job.%J.out
#SBATCH --error=/astro/username/que/job.%J.err
</pre>

The standard output and error can be written to specific files with the <code>--output</code> and <code>--error</code> commands. By default, both files are directed to a file of the name <code>slurm-%j.out</code>, where the <code>%j</code> is replaced with the job number.

<h2>
Simple Batch Script For Running MPI-Parallel Jobs
</h2>
Note: The following example script assumes that you submit the script from the directory where the code will be executed (the path to that directory is stored in the
<code>$SLURM_SUBMIT_DIR</code> environment variable, and is where SLURM will execute the script from).

<pre>
#!/bin/bash
#
# SLURM resource specifications
# (use an extra '#' in front of SBATCH to comment-out any unused options)
#
#SBATCH --job-name=isothermal # shows up in the output of 'squeue'
#SBATCH --time=4-23:59:59 # specify the requested wall-time
#SBATCH --partition=astro_long # specify the partition to run on
#SBATCH --nodes=4 # number of nodes allocated for this job
#SBATCH --ntasks-per-node=20 # number of MPI ranks per node
#SBATCH --cpus-per-task=1 # number of OpenMP threads per MPI rank
##SBATCH --exclude=<node list> # avoid nodes (e.g. --exclude=node786)

# Load default settings for environment variables
module load astro

# If required, replace specific modules
# module unload intelmpi
# module load mvapich2

# When compiling remember to use the same environment and modules

# Execute the code
srun <executable> [args...]
</pre>

<h2>
Simple Batch Script For Jobs Using OpenMP Only
</h2>
Note: The following example script assumes that you submit the script from the directory where the code will be executed (the path to that directory is stored in the
<code>$SLURM_SUBMIT_DIR</code> environment variable).

Note: By choosing <code>--cpus-per-task=40</code> along with <code>--threads-per-core=2</code>, you have assumed that your program will take advantage of [https://en.wikipedia.org/wiki/Hyper-threading#readme hyper threading]. If this is not the case, or you are uncertain, use <code>--cpus-per-task=20</code> along with <code>--threads-per-core=1</code>, and your program will be executed with 20 threads.

<pre>
#!/bin/bash
#
# Define SLURM resource specifications
# (use an extra '#' in front of SBATCH to comment-out any unused options)
#
#SBATCH --job-name=isothermal # shows up in the output of 'squeue'
#SBATCH --time=4-23:59:59 # specify the requested wall-time
#SBATCH --partition=astro_long # specify the partition to run on
#SBATCH --cpus-per-task=40 # total number of threads
#SBATCH --threads-per-core=2 # threads per core (hyper-threading)
##SBATCH --exclude=<node list> # avoid nodes (e.g. --exclude=node786)

# Load default settings for environment variables
module load astro

# OpenMP affinity
# no hyperthreading
# export KMP_AFFINITY="granularity=core,scatter,1,0"
# hyperthreading
export KMP_AFFINITY="granularity=thread,scatter,1,0"

# When compiling remember to use the same environment

# Execute the code
srun --cpu_bind=threads <executable> [args...]
</pre>

<h2>
Hybrid MPI + OpenMP Batch Script
</h2>
Note: The following example script assumes that you submit the script from the directory where the code will be executed (the path to that directory is stored in the
<code>$SLURM_SUBMIT_DIR</code> environment variable, and is where SLURM will execute the script from).

<pre>
#!/bin/bash
#
# Define SLURM resource specifications
# (use an extra '#' in front of SBATCH to comment-out any unused options)
#
#SBATCH --job-name=isothermal # shows up in the output of 'squeue'
#SBATCH --time=4-23:59:59 # specify the requested wall-time
#SBATCH --partition=astro_long # specify the partition to run on
#SBATCH --nodes=32 # number of nodes allocated for this job
#SBATCH --ntasks-per-node=8 # lower than the usual 20 for MPI only
#SBATCH --cpus-per-task=5 # number of CPUs per MPI rank
#SBATCH --threads-per-core=2 # threads per core (hyper-threading)
##SBATCH --exclude=<node list> # avoid nodes (e.g. --exclude=node786)

# Load default settings for environment variables
module load astro

# OpenMP affinity
export KMP_AFFINITY="granularity=thread,scatter,1,0"

# If required, replace specific modules
# module unload intelmpi
# module load mvapich2

# When compiling remember to use the same environment and modules

# Execute the code
cd $SLURM_SUBMIT_DIR
srun --cpu_bind=threads <executable> [args...]
</pre>

Running batch jobs

2023-11-15T14:45:33Z

Philip:

<h1>
Submit Jobs Via The SLURM Queueing System
</h1>
This is the preferred method of submitting batch jobs to the cluster queueing system and to run jobs interactively.

<h2>
Most Important Queue Commands:
</h2>
Here we list the most commonly used queueing commands. If you are migrating from a different scheduling system, this [https://slurm.schedmd.com/rosetta.pdf#readme cheat sheet] may be useful for you. There also exists a compact [https://slurm.schedmd.com/pdfs/summary.pdf#readme two-page overview] of the most important commands.

<h3>
Use The <code>'Sinfo'</code> Command To Display Information About Available Resources:
</h3>
If you use the command without any options, it will display all available partitions. Use the <code>-p</code> switch to select a specific partition, for instance:

<pre>
astro06:> sinfo -p astro2
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
astro2 up 10-00:00:0 1 down* node458
astro2 up 10-00:00:0 13 alloc node[454-457,459-462,480-481]
astro2 up 10-00:00:0 18 idle node[463-479,482]
</pre>
The command displays how many nodes in the partition are offline (<code>down</code>), are busy (<code>alloc</code>) and how many are still available (<code>idle</code>)). For each sub-category, a <code>NODELIST</code> is displayed. The <code>TIMELIMIT</code> column shows the maximum job duration allowed for the partition in <code>days-hours:minutes:seconds</code> format. You can find more information about how to use the [https://slurm.schedmd.com/sinfo.html#readme sinfo] command on the official [https://slurm.schedmd.com/man_index.html#readme SLURM man pages].

<h3>
Use The 'Squeue' Command To Display Information About Scheduled Jobs:
</h3>
<pre>
astro06:> squeue -astro_long
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
566136 astro_long jobname1 username R 47:22 1 node485
566135 astro_long jobname2 username R 54:22 1 node481
</pre>
The command displays a table with useful information. Use the <code>JOBID</code> of your job to modify or cancel a scheduled or already running job (see below). The status column (<code>ST</code>) shows the state of the queued job, the letters stand for: <code>PD</code> (pending), <code>R</code> (running), <code>CA</code> (cancelled), <code>CG</code> (completing), <code>CD</code> (completed), <code>F</code> (failed), <code>TO</code> (timeout), and <code>NF</code> (node failure).

Useful command line switches for <code>squeue</code> include <code>-u</code> (or <code>--users</code>) for only listing jobs that belong to a specific user. You can find more information about how to use the [https://slurm.schedmd.com/squeue.html#readme squeue] command on the official [https://slurm.schedmd.com/man_index.html#readme SLURM man pages].

<h3>
Use The 'Scancel' Command To Cancel A Scheduled Or Running Job:
</h3>
<pre>
astro06:> scancel 566136
</pre>
You can find more information about how to use the [https://slurm.schedmd.com/scancel.html#readme scancel] command on the official [https://slurm.schedmd.com/man_index.html#readme SLURM man pages].

<h3>
Use The 'Srun' Command To Run Jobs Interactively:
</h3>
You can run serial, openMP- or MPI-parallel code interactively using the <code>srun</code> command. Always make sure to specify the partition to run on via the <code>-p</code> command line switch. When running an MPI job, you can use the <code>-n</code> switch to specify the number of MPI tasks that you require. Command line arguments for your program can be passed at the end.

<pre>
astro06:> srun -p astro_devel -n 20 <executable> [args...]
</pre>
You can find more information about how to use the [https://slurm.schedmd.com/srun.html#readme srun] command on the official [https://slurm.schedmd.com/man_index.html#readme SLURM man pages].

<h3>
Use The 'Sbatch' Command To Queue A Job Via A Submission Script:
</h3>
<pre>
astro06:> sbatch [additional options] job-submission-script.sh
</pre>
You can find more information about how to use the [https://slurm.schedmd.com/sbatch.html#readme sbatch] command on the official [https://slurm.schedmd.com/man_index.html#readme SLURM man pages].

Running batch jobs

2023-11-15T14:39:08Z

Philip: Created page with "<h1> Submit Jobs Via The SLURM Queueing System </h1> This is the preferred method of submitting batch jobs to the cluster queueing system and to run jobs interactively. <h2> Most Important Queue Commands: </h2> Here we list the most commonly used queueing commands. If you are migrating from a different scheduling system, this cheat sheet may be useful for you. There also exists a compact two-page overview of the most important commands. <h3> Use The <code>'Sinfo'</code..."

<h1>
Submit Jobs Via The SLURM Queueing System
</h1>
This is the preferred method of submitting batch jobs to the cluster queueing system and to run jobs interactively.

<h2>
Most Important Queue Commands:
</h2>
Here we list the most commonly used queueing commands. If you are migrating from a different scheduling system, this cheat sheet may be useful for you. There also exists a compact two-page overview of the most important commands.

<h3>
Use The <code>'Sinfo'</code> Command To Display Information About Available Resources:
</h3>
If you use the command without any options, it will display all available partitions. Use the <code>-p</code> switch to select a specific partition, for instance:

<pre>
astro06:> sinfo -p astro2
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
astro2 up 10-00:00:0 1 down* node458
astro2 up 10-00:00:0 13 alloc node[454-457,459-462,480-481]
astro2 up 10-00:00:0 18 idle node[463-479,482]
</pre>
The command displays how many nodes in the partition are offline (<code>down</code>), are busy (<code>alloc</code>) and how many are still available (<code>idle</code>)). For each sub-category, a <code>NODELIST</code> is displayed. The <code>TIMELIMIT</code> column shows the maximum job duration allowed for the partition in <code>days-hours:minutes:seconds</code> format. You can find more information about how to use the <code>sinfo</code> command on the official SLURM main pages.

<h3>
Use The 'Squeue' Command To Display Information About Scheduled Jobs:
</h3>
<pre>
astro06:> squeue -astro_long
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
566136 astro_long jobname1 username R 47:22 1 node485
566135 astro_long jobname2 username R 54:22 1 node481
</pre>
The command displays a table with useful information. Use the <code>JOBID</code> of your job to modify or cancel a scheduled or already running job (see below). The status column (<code>ST</code>) shows the state of the queued job, the letters stand for: <code>PD</code> (pending), <code>R</code> (running), <code>CA</code> (cancelled), <code>CG</code> (completing), <code>CD</code> (completed), <code>F</code> (failed), <code>TO</code> (timeout), and <code>NF</code> (node failure).

Useful command line switches for <code>squeue</code> include <code>-u</code> (or <code>--users</code>) for only listing jobs that belong to a specific user. You can find more information about how to use the <code>squeue</code> command on the official SLURM man pages.

<h3>
Use The 'Scancel' Command To Cancel A Scheduled Or Running Job:
</h3>
<pre>
astro06:> scancel 566136
</pre>
You can find more information about how to use the scancel command on the official SLURM main pages.

<h3>
Use The 'Srun' Command To Run Jobs Interactively:
</h3>
You can run serial, openMP- or MPI-parallel code interactively using the <code>srun</code> command. Always make sure to specify the partition to run on via the <code>-p</code> command line switch. When running an MPI job, you can use the <code>-n</code> switch to specify the number of MPI tasks that you require. Command line arguments for your program can be passed at the end.

<pre>
astro06:> srun -p astro_devel -n 20 <executable> [args...]
</pre>
You can find more information about how to use the srun command on the official SLURM man pages.

<h3>
Use The 'Sbatch' Command To Queue A Job Via A Submission Script:
</h3>
<pre>
astro06:> sbatch [additional options] job-submission-script.sh
</pre>
You can find more information about how to use the sbatch command on the official SLURM man pages.

Tycho Technical Documentation

2023-11-15T14:25:03Z

Philip:

Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.

Tycho contain in total more 10,000 CPU cores and 13 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current peak performance is 443 TFlops from the CPUs and 93 TFlops from the GPUs.

Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.

===First steps===
Please visit the [[first steps]] page to get started

===Overview===
* [[Acknowledging the use of Tycho in articles and presentations]]
* [[Getting Help]]
* [[Mattermost discussion forum]]
* [[Being a good HPC user]]
* [[Accessing Tycho]]
* [[Using Jupyter notebooks on the frontends]]
* [[Visual Studio Remote Development]]
* [[Virtual Desktop]]
* [[Hardware]]
* [[Erda]]
* [[Using GPUs]]
* [[Compilers]]
* [[Debugging and Profiling]]
* [[MPI Libraries]]
* [[Running batch jobs]]
* [[Examples of SLURM scripts]]
* [[Codes]]
* [[Adding a second IP Address]]
* [[Setting up One-Time-Password Access before travelling]]

===Scientific Software===

* [[Module system]]
* [[Running Mathematica on compute nodes]]
* [[GRChombo]]

*************************

Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.

== Getting started ==
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]

Tycho Technical Documentation

2023-11-15T14:24:41Z

Philip:

Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.

Tycho contain in total more 10,000 CPU cores and 13 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current peak performance is 443 TFlops from the CPUs and 93 TFlops from the GPUs.

Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.

===First steps===
Please visit the [[first steps]] page to get started

===Overview===
* [[Acknowledging the use of Tycho in articles and presentations]]
* [[Getting Help]]
* [[Mattermost discussion forum]]
* [[Being a good HPC user]]
* [[Accessing Tycho]]
* [[Using Jupyter notebooks on the frontends]]
* [[Visual Studio Remote Development]]
* [[Virtual Desktop]]
* [[Hardware]]
* [[Erda]]
* [[Using GPUs]]
* [[Compilers]]
* [[Debugging and Profiling]]
* [[MPI Libraries]]
* [Running batch jobs]]
* [[Examples of SLURM scripts]]
* [[Codes]]
* [[Adding a second IP Address]]
* [[Setting up One-Time-Password Access before travelling]]

===Scientific Software===

* [[Module system]]
* [[Running Mathematica on compute nodes]]
* [[GRChombo]]

*************************

Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.

== Getting started ==
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]

Examples of SLURM scripts

2023-11-15T14:18:56Z

Philip: Created page with "Here is examples of simple SLURM scripts for running serial, OpenMP and MPI jobs. Note that a very nice introduction to running SLURM scripts exist at https://hpc.ku.dk/documentation/slurm.html. <h2> Simple Batch Script For Running Serial Programs </h2> Submission scripts are really just shell scripts (here we use bash syntax) with a few additional variable specifications at the beginning. These are prefixed with "#SBATCH" and otherwise use the same keywords and syn..."

Here is examples of simple SLURM scripts for running serial, OpenMP and MPI jobs.

Note that a very nice introduction to running SLURM scripts exist at https://hpc.ku.dk/documentation/slurm.html.

<h2>
Simple Batch Script For Running Serial Programs
</h2>

Submission scripts are really just shell scripts (here we use bash syntax) with a few additional variable specifications at the beginning. These are prefixed with "#SBATCH" and otherwise use the same keywords and syntax as the command line options described for the sbatch command. The following script presents a minimal example:
<pre>
#!/bin/bash
#SBATCH --job-name=isothermal # shows up in the output of 'squeue'
#SBATCH --partition=astro_devel # specify the partition to run on
srun /bin/echo "Hello World!"
</pre>

Specifying Mail Notifications And Manage Output And Error Files:
You can enable e-Mail notification on various events, this can be specified via the <code>--mail-type</code>
option which takes the following values: <code>BEGIN</code>, <code>END</code>, <code>FAIL</code>, <code>REQUEUE</code> and <code>ALL</code>.

<pre>
#SBATCH --mail-type=FAIL
#SBATCH --output=/astro/username/que/job.%J.out
#SBATCH --error=/astro/username/que/job.%J.err
</pre>

The standard output and error can be written to specific files with the <code>--output</code> and <code>--error</code> commands. By default, both files are directed to a file of the name <code>slurm-%j.out</code>, where the <code>%j</code> is replaced with the job number.

<h2>
Simple Batch Script For Running MPI-Parallel Jobs
</h2>
Note: The following example script assumes that you submit the script from the directory where the code will be executed (the path to that directory is stored in the
<code>$SLURM_SUBMIT_DIR</code> environment variable, and is where SLURM will execute the script from).

<pre>
#!/bin/bash
#
# SLURM resource specifications
# (use an extra '#' in front of SBATCH to comment-out any unused options)
#
#SBATCH --job-name=isothermal # shows up in the output of 'squeue'
#SBATCH --time=4-23:59:59 # specify the requested wall-time
#SBATCH --partition=astro_long # specify the partition to run on
#SBATCH --nodes=4 # number of nodes allocated for this job
#SBATCH --ntasks-per-node=20 # number of MPI ranks per node
#SBATCH --cpus-per-task=1 # number of OpenMP threads per MPI rank
##SBATCH --exclude=<node list> # avoid nodes (e.g. --exclude=node786)

# Load default settings for environment variables
module load astro

# If required, replace specific modules
# module unload intelmpi
# module load mvapich2

# When compiling remember to use the same environment and modules

# Execute the code
srun <executable> [args...]
</pre>

<h2>
Simple Batch Script For Jobs Using OpenMP Only
</h2>
Note: The following example script assumes that you submit the script from the directory where the code will be executed (the path to that directory is stored in the
<code>$SLURM_SUBMIT_DIR</code> environment variable).

Note: By choosing <code>--cpus-per-task=40</code> along with <code>--threads-per-core=2</code>, you have assumed that your program will take advantage of hyper threading. If this is not the case, or you are uncertain, use <code>--cpus-per-task=20</code> along with <code>--threads-per-core=1</code>, and your program will be executed with 20 threads.

<pre>
#!/bin/bash
#
# Define SLURM resource specifications
# (use an extra '#' in front of SBATCH to comment-out any unused options)
#
#SBATCH --job-name=isothermal # shows up in the output of 'squeue'
#SBATCH --time=4-23:59:59 # specify the requested wall-time
#SBATCH --partition=astro_long # specify the partition to run on
#SBATCH --cpus-per-task=40 # total number of threads
#SBATCH --threads-per-core=2 # threads per core (hyper-threading)
##SBATCH --exclude=<node list> # avoid nodes (e.g. --exclude=node786)

# Load default settings for environment variables
module load astro

# OpenMP affinity
# no hyperthreading
# export KMP_AFFINITY="granularity=core,scatter,1,0"
# hyperthreading
export KMP_AFFINITY="granularity=thread,scatter,1,0"

# When compiling remember to use the same environment

# Execute the code
srun --cpu_bind=threads <executable> [args...]
</pre>

<h2>
Hybrid MPI + OpenMP Batch Script
</h2>
Note: The following example script assumes that you submit the script from the directory where the code will be executed (the path to that directory is stored in the
<code>$SLURM_SUBMIT_DIR</code> environment variable, and is where SLURM will execute the script from).

<pre>
#!/bin/bash
#
# Define SLURM resource specifications
# (use an extra '#' in front of SBATCH to comment-out any unused options)
#
#SBATCH --job-name=isothermal # shows up in the output of 'squeue'
#SBATCH --time=4-23:59:59 # specify the requested wall-time
#SBATCH --partition=astro_long # specify the partition to run on
#SBATCH --nodes=32 # number of nodes allocated for this job
#SBATCH --ntasks-per-node=8 # lower than the usual 20 for MPI only
#SBATCH --cpus-per-task=5 # number of CPUs per MPI rank
#SBATCH --threads-per-core=2 # threads per core (hyper-threading)
##SBATCH --exclude=<node list> # avoid nodes (e.g. --exclude=node786)

# Load default settings for environment variables
module load astro

# OpenMP affinity
export KMP_AFFINITY="granularity=thread,scatter,1,0"

# If required, replace specific modules
# module unload intelmpi
# module load mvapich2

# When compiling remember to use the same environment and modules

# Execute the code
cd $SLURM_SUBMIT_DIR
srun --cpu_bind=threads <executable> [args...]
</pre>