https://wiki.nbi.ku.dk/w/tycho/api.php?action=feedcontributions&feedformat=atom&user=HaugboelTycho - User contributions [en]2026-04-06T08:51:52ZUser contributionsMediaWiki 1.43.8https://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=238Hardware2025-06-26T07:20:09Z<p>Haugboel: /* CPUs */</p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L1: 64 KB / core, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 TB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L1: 64 KB / core, L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L1: 64 KB / core, L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 2 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L1: 64 KB / core, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===CPUs===<br />
Our cluster nodes use Intel Cascade-lake, AMD Genoa, and AMD Turin for the pure CPU nodes, and AMD Rome and AMD Genoa for the GPU does providing varying real performance per clock, with the newest Genoa and Turin does having the best performance. For a deep dive in to CPU architecture, see:<br />
* Intel Cascade-lake and AMD Rome: https://www.nas.nasa.gov/assets/nas/pdf/papers/NAS_Technical_Report_NAS-2022_01.pdf<br />
* AMD Genoa: https://www.chpc.utah.edu/documentation/white_papers/cpus_may2023_v3.pdf#:~:text=There%20are%20additional%20two%20256%20bit%20vector,the%201x64%20or%201x96%20~105%25%20theoretical%20peak.<br />
* AMD Turin:<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro04, and the different sized virtual GPUs on astro02) to determine how well the workload scales.<br />
<br />
* 8 A100 GPUs are available on the astro01 frontend machine and in the astro2_gpu queue. They are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. The connection between the CPU and the GPU is through PCIe Gen 4 x16 link with 64 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/data-center/a100 .<br />
<br />
* 3 A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. The connection between the CPU and the GPU is through PCIe Gen 4 x16 link with 64 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu .<br />
<br />
* 4 RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. The connection between the CPU and the GPU is through PCIe Gen 4 x16 link with 64 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/rtx-a6000 .<br />
<br />
* 4 H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. The connection between the CPU and the GPU is through PCIe Gen 5 x16 link with 128 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/data-center/h100/ .<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=237Hardware2025-06-26T07:15:23Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L1: 64 KB / core, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 TB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L1: 64 KB / core, L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L1: 64 KB / core, L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 2 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L1: 64 KB / core, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===CPUs===<br />
Our cluster nodes use Intel Cascade-lake, AMD Genoa, and AMD Turin for the pure CPU nodes, and AMD Rome and AMD Genoa for the GPU does providing varying real performance per clock, with the newest Genoa and Turin does having the best performance. For a deep dive in to CPU architecture, and in particular Intel Cascade-lake and AMD Rome see https://www.nas.nasa.gov/assets/nas/pdf/papers/NAS_Technical_Report_NAS-2022_01.pdf<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro04, and the different sized virtual GPUs on astro02) to determine how well the workload scales.<br />
<br />
* 8 A100 GPUs are available on the astro01 frontend machine and in the astro2_gpu queue. They are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. The connection between the CPU and the GPU is through PCIe Gen 4 x16 link with 64 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/data-center/a100 .<br />
<br />
* 3 A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. The connection between the CPU and the GPU is through PCIe Gen 4 x16 link with 64 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu .<br />
<br />
* 4 RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. The connection between the CPU and the GPU is through PCIe Gen 4 x16 link with 64 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/rtx-a6000 .<br />
<br />
* 4 H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. The connection between the CPU and the GPU is through PCIe Gen 5 x16 link with 128 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/data-center/h100/ .<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=236Hardware2025-05-19T13:06:39Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L1: 64 KB / core, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 TB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L1: 64 KB / core, L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L1: 64 KB / core, L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 2 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L1: 64 KB / core, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro04, and the different sized virtual GPUs on astro02) to determine how well the workload scales.<br />
<br />
* 8 A100 GPUs are available on the astro01 frontend machine and in the astro2_gpu queue. They are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. The connection between the CPU and the GPU is through PCIe Gen 4 x16 link with 64 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/data-center/a100 .<br />
<br />
* 3 A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. The connection between the CPU and the GPU is through PCIe Gen 4 x16 link with 64 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu .<br />
<br />
* 4 RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. The connection between the CPU and the GPU is through PCIe Gen 4 x16 link with 64 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/rtx-a6000 .<br />
<br />
* 4 H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. The connection between the CPU and the GPU is through PCIe Gen 5 x16 link with 128 GB/s. You can read more about their specs here https://www.nvidia.com/en-us/data-center/h100/ .<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=235Hardware2025-05-06T11:45:28Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L1: 64 KB / core, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 TB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L1: 64 KB / core, L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L1: 64 KB / core, L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L1: 64 KB / core, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L1: 64 KB / core, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro04, and the different sized virtual GPUs on astro02) to determine how well the workload scales.<br />
<br />
* 8 A100 GPUs are available on the astro01 frontend machine and in the astro2_gpu queue are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. You can read more about their specs here https://www.nvidia.com/en-us/data-center/a100 .<br />
<br />
* 3 A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu .<br />
<br />
* 4 RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/rtx-a6000 .<br />
<br />
* 6 H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. You can read more about their specs here https://www.nvidia.com/en-us/data-center/h100/ .<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=234Hardware2025-03-14T07:03:04Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 TB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro04, and the different sized virtual GPUs on astro02) to determine how well the workload scales.<br />
<br />
* 8 A100 GPUs are available on the astro01 frontend machine and in the astro2_gpu queue are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. You can read more about their specs here https://www.nvidia.com/en-us/data-center/a100 .<br />
<br />
* 3 A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu .<br />
<br />
* 4 RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/rtx-a6000 .<br />
<br />
* 6 H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. You can read more about their specs here https://www.nvidia.com/en-us/data-center/h100/ .<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=233Hardware2025-03-14T07:01:54Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 TB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro04, and the different sized virtual GPUs on astro02) to determine how well the workload scales.<br />
<br />
* 8 A100 GPUs are available on the astro01 frontend machine and in the astro2_gpu queue are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. You can read more about their specs here https://www.nvidia.com/en-us/data-center/a100 .<br />
<br />
* 3 A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu .<br />
<br />
* 4 RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/rtx-a6000 .<br />
<br />
* 6 H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/h100 .<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Tycho_Technical_Documentation&diff=232Tycho Technical Documentation2025-03-14T07:00:59Z<p>Haugboel: </p>
<hr />
<div>Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.<br />
<br />
Tycho contain in total more than 10,000 CPU cores and 21 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current FP64 peak performance is 443 TFlops from the CPUs and 276 TFlops from the GPUs. The GPUs are even more powerful at lower precision having 700 TFlops FP32 performance, and more than 18 PFlops of FP16 Tensor core performance for machine learning workloads.<br />
<br />
Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.<br />
<br />
===First steps===<br />
Please visit the [[first steps]] page to get started<br />
<br />
===Overview===<br />
* [[Acknowledging the use of Tycho in articles and presentations]] <br />
* [[Getting Help]]<br />
* [[Mattermost discussion forum]]<br />
* [[Being a good HPC user]]<br />
* [[Accessing Tycho]]<br />
* [[Using Jupyter notebooks on the frontends]]<br />
* [[Visual Studio Remote Development]]<br />
* [[Virtual Desktop]]<br />
* [[Hardware]]<br />
* [[Erda]]<br />
* [[Using GPUs]]<br />
* [[Compilers]]<br />
* [[Debugging and Profiling]]<br />
* [[MPI Libraries]]<br />
* [[Running batch jobs]]<br />
* [[Examples of SLURM scripts]]<br />
* [[Codes]]<br />
* [[Adding a second IP Address]]<br />
* [[Setting up One-Time-Password Access before travelling]]<br />
* [[FAQs]]<br />
<br />
===Scientific Software===<br />
<br />
* [[Module system]]<br />
* [[Running Mathematica on compute nodes]]<br />
* [[GRChombo]]<br />
<br />
<br />
*************************<br />
<br />
Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.<br />
<br />
== Getting started ==<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=231Hardware2025-03-13T10:51:13Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 TB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro04, and the different sized virtual GPUs on astro02) to determine how well the workload scales.<br />
<br />
* The A100 GPUs available on the astro01 frontend machine and in the astro2_gpu queue are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. You can read more about their specs here https://www.nvidia.com/en-us/data-center/a100 .<br />
<br />
* The three A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu .<br />
<br />
* The four RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/rtx-a6000 .<br />
<br />
* The six H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/h100 .<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=FAQs&diff=230FAQs2025-03-06T09:59:05Z<p>Haugboel: </p>
<hr />
<div>'''How do I access the HPC Cluster?'''<br />
<br />
You can remotely access the frontend machines astro01-09 via SSH to submit jobs or to analyze data (see [[Hardware]] for an up to date list of available frontends). For example, you can login to the astro01 machine entering the following in the command line<br />
<br />
<pre><br />
ssh username@astro01.hpc.ku.dk<br />
</pre><br />
<br />
From which you will be prompted to enter your password (Note: If the connection times out, it's likely that your IP address is not recognised). By entering the correct password, you will arrive at your home directory /groups/astro/username. It may be a good idea to check the load factor after logging in, using "top", and choose a different frontend if the CPU or memory use is already high (use "<" or ">" in top to temporarily change from sorting on CPU to memory / virtual memory).<br />
<br />
You can cut this down to<br />
<br />
<pre><br />
ssh astro01<br />
</pre><br />
<br />
by adding these lines to the file ~/.ssh/config:<br />
<br />
<pre><br />
Host astro01 <br />
User username <br />
HostName astro01.hpc.ku.dk<br />
</pre><br />
<br />
Do not use the ssh option -Y, which "enables trusted X11 forwarding". This means, basically, that you turn off some essential X security features and say "I trust the remote host completely". Forwarding of X should work without any extra options.<br />
<br />
'''How can I setup a passwordless ssh?'''<br />
<br />
You can avoid to type in your password each time for the authenticating of SSH sessions, by using an SSH key with the [http://en.wikipedia.org/wiki/RSA_(cryptosystem)#readme RSA] encryption, in combination with ssh-add and ssh-agent. Therefore, you generate with [http://en.wikipedia.org/wiki/Ssh-keygen#readme ssh-keygen] on your local laptop/computer a pair of a personal (id_rsa) and public key (id_rsa.pub). A widely recommended and secure cipher to use is based on ED25519 encryption<br />
<br />
<pre><br />
ssh-keygen -t ed25519<br />
</pre><br />
<br />
With the option -t you specify the encryption type. Here we use ED25519; many others exist but that is what is recommended across large organizations to keep your key, login, and data secure. You will asked for the filename, you can just press enter and use the default. Then, you will be asked for a passphrase to protect your personal key.<br />
You should NOT under any circumstances use an empty pass phrase; it is not necessary for convenience reasons (see below), and could endanger your access to remote supercomputers. Use instead a "pass phrase"; similar to a password.<br />
There will be stored two keys (id_ed25519 and id_ed25519.pub) in the hidden folder ~/.ssh/ on your client (normally your laptop, but it could also be a server you want to use to login to Tycho). To use the key-pair for logging in you have to copy your public key from your client to Tycho and store it in the file ~/.ssh/authorized_keys. From your client (e.g. the laptop) do:<br />
<br />
<pre><br />
cat ~/.ssh/id_rsa.pub | ssh username@astro01.hpc.ku.dk 'cat >> .ssh/authorized_keys'<br />
</pre><br />
<br />
Next time you login to Tycho with ssh, instead of asking for the password, the server will request your pass phrase from the client (the passphrase you supplied when running the ssh-keygen command). However, rather than writing it every time at the ssh prompt, you can use "ssh-add" to give the pass phrase once and keep the decrypted private key in memory. On Mac and Linux this works transparently, on windows you need to make sure that another program, ssh-agent, is running on the system and can store the decrypted key. You can read some suggestions about how to do that on Windows here: https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement<br />
<br />
If your ssh command is issued in a window inside a VNC session, or in a window on your local laptop, chances are you can just type "ssh-add" to store the password in a process called ssh-agent, which typically is already running (most X-sessions -- such as the ones you start under VNC -- are automatically started as child processes of an ssh-agent).<br />
<br />
If an ssh-agent process is already running (and your ssh command is a descendant of it), it stores the credentials created by your once-per-reboot ssh-add command and automatically answers requests from hosts you try to connect to. If for some reason no ssh-agent is running on an intermediate host, then as an alternative you can forward your ssh-credentials from your laptop, by using "ssh -A" to login to astro0X, and continuing an ssh from there (having stored the id_rsa.pub from your laptop on the remote supercomputer).<br />
<br />
Finally, if for some reason you start out from a place (such a just a command window) where no ssh-agent is running, you can just start one, by doing<br />
<br />
<pre><br />
ssh-agent bash<br />
</pre><br />
<br />
This starts an ssh-agent, which starts bash as a descendant. Then execute "ssh-add", and you're free from typing the pass phrase, for as long as you keep the ssh-agent running.<br />
<br />
'''How do I get my IP white-listed'''<br />
<br />
On any of the astro0X hosts, use the command<br />
<br />
<pre><br />
hpc-setup-firewall.sh<br />
</pre><br />
<br />
You have three personal slots, but other people's slots also work for you. The first time you login from home, or a new place, you may need to login via muon.nbi.dk, or some other NBI host, from which astro0X is already open. To find your IP-number, either use one of the web-services (but beware of spam-ware), or just type "finger -m $user" after logging in to an NBI host.<br />
SSHFS.<br />
<br />
You can locally access remote data on the frontend very conveniently by the SSH file system ([http://en.wikipedia.org/wiki/SSHFS#readme SSHFS]). To use SSHFS you need to install [http://en.wikipedia.org/wiki/Filesystem_in_Userspace#readme FUSE]. For Linux you can install [http://fuse.sourceforge.net/#readme fuse] and for Mac there is [https://osxfuse.github.io/#readem osxfuse]. You need to create an empty file as mount point on your local laptop/computer, e.g.<br />
<br />
<pre><br />
makedir ~/nbi<br />
</pre><br />
<br />
Then you mount your by specifying the host and the mount point.<br />
<br />
<pre><br />
sshfs username@astro01.hpc.ku.dk:/astro/username/ ~/nbi/ -oauto_cache,reconnect,volname=nbi<br />
</pre><br />
<br />
You can unmount the filesystem with<br />
<br />
<pre><br />
fusermount -u ~/nbi<br />
</pre><br />
<br />
SSH tunneling: To create a tunnel for for example display :11 on astro01 using ssh, do<br />
<br />
<pre><br />
ssh astro01 -L 5911:localhost:5911<br />
</pre><br />
<br />
You can do the same without starting a remote shell by doing<br />
<br />
<pre><br />
ssh astro01 -L 5911:localhost:5911 -fN<br />
</pre><br />
<br />
If you succeed in always using display :11 you can add the tunnel configuration as part of the ~/.ssh/config file:<br />
<br />
<pre><br />
Host tunnel <br />
User username <br />
HostName astro01.hpc.ku.dk <br />
LocalForward 5911 <br />
localhost:5911<br />
</pre><br />
<br />
and then start the tunnel with<br />
<br />
<pre><br />
ssh tunnel -fN<br />
</pre><br />
<br />
===GENERAL & SUPPORT===<br />
'''Whom should I ask which questions?'''<br />
# login nodes are unreachable: mailto:support@hpc.ku.dk<br />
# login nodes are alive, but I don't have a homedir: mailto:support@hpc.ku.dk<br />
# I don't have access to the software I need: mailto:support@hpc.ku.dk<br />
# forgot my password: mailto:support@hpc.ku.dk<br />
* Is there a sharing policy?<br />
# cluster queues<br />
# disk space<br />
# analysis servers<br />
* Can I run codes interactively, fx on a front end, or on an analysis server?<br />
<br />
<br />
ENVIRONMENTS<br />
<br />
* What are 'modules' and how do I use them?<br />
* What is my default environment setup?<br />
- MPI <br />
- libraries<br />
- compilers<br />
-...<br />
<br />
* Is there a standard module to load which lays out everything - no sweat?<br />
Yes, access to the astro software is provided by writing<br />
<br />
<pre><br />
module load astro<br />
</pre><br />
<br />
command that makes all astro specific modules available. You can see what is available by writing<br />
<br />
<pre><br />
module load avail<br />
</pre><br />
<br />
and list what you have loaded with<br />
<br />
<pre><br />
module load list<br />
</pre><br />
<br />
To undo the default and start from scratch, use<br />
<br />
<pre><br />
module purge<br />
</pre><br />
<br />
<br />
SOFTWARE<br />
<br />
* What software do I have available?<br />
* Do I have access <br />
* What is SLURM?<br />
* Can I submit SLURM jobs from anywhere?<br />
* How do I find out which queues I can submit to?<br />
<br />
<br />
STORAGE<br />
<br />
* Where should I put data from large simulations?<br />
* How much disk space can I claim?<br />
* How long can I have stuff on disk?<br />
* Are my data backed up, and where?</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=FAQs&diff=229FAQs2025-03-06T09:35:07Z<p>Haugboel: </p>
<hr />
<div>'''How do I access the HPC Cluster?'''<br />
<br />
You can remotely access the frontend machines astro01-09 via SSH to submit jobs or to analyze data (see [[Hardware]] for an up to date list of available frontends). For example, you can login to the astro01 machine entering the following in the command line<br />
<br />
<pre><br />
ssh username@astro01.hpc.ku.dk<br />
</pre><br />
<br />
From which you will be prompted to enter your password (Note: If the connection times out, it's likely that your IP address is not recognised). By entering the correct password, you will arrive at your home directory /groups/astro/username. It may be a good idea to check the load factor after logging in, using "top", and choose a different frontend if the CPU or memory use is already high (use "<" or ">" in top to temporarily change from sorting on CPU to memory / virtual memory).<br />
<br />
You can cut this down to<br />
<br />
<pre><br />
ssh astro01<br />
</pre><br />
<br />
by adding these lines to the file ~/.ssh/config:<br />
<br />
<pre><br />
Host astro01 <br />
User username <br />
HostName astro01.hpc.ku.dk<br />
</pre><br />
<br />
Do not use the ssh option -Y, which "enables trusted X11 forwarding". This means, basically, that you turn off some essential X security features and say "I trust the remote host completely". Forwarding of X should work without any extra options.<br />
<br />
'''How can I setup a passwordless ssh?'''<br />
<br />
You can avoid to type in your password each time for the authenticating of SSH sessions, by using an SSH key with the [http://en.wikipedia.org/wiki/RSA_(cryptosystem)#readme RSA] encryption, in combination with ssh-add and ssh-agent. Therefore, you generate with [http://en.wikipedia.org/wiki/Ssh-keygen#readme ssh-keygen] on your local laptop/computer a pair of a personal (id_rsa) and public key (id_rsa.pub). A widely recommended and secure cipher to use is based on ED25519 encryption<br />
<br />
<pre><br />
ssh-keygen -t ed25519<br />
</pre><br />
<br />
With the option -t you specify the encryption type. Here we use ED25519; many others exist but that is what is recommended across large organizations to keep your key, login, and data secure. You will asked for the filename, you can just press enter and use the default. Then, you will be asked for a passphrase to protect your personal key.<br />
You should NOT under any circumstances use an empty pass phrase; it is not necessary for convenience reasons (see below), and could endanger your access to remote supercomputers. Use instead a "pass phrase"; similar to a password.<br />
There will be stored two keys (id_ed25519 and id_ed25519.pub) in the hidden folder ~/.ssh/ on your client (normally your laptop, but it could also be a server you want to use to login to Tycho). To use the key-pair for logging in you have to copy your public key from your client to Tycho and store it in the file ~/.ssh/authorized_keys. From your client (e.g. the laptop) do:<br />
<br />
<pre><br />
cat ~/.ssh/id_rsa.pub | ssh username@astro01.hpc.ku.dk 'cat >> .ssh/authorized_keys'<br />
</pre><br />
<br />
Next time you login to Tycho with ssh, instead of asking for the password, the server will request your pass phrase from the client (the one where you issued the ssh-keygen command). However, rather than wait for the ssh prompt, you should use "ssh-add" to give the pass phrase, once and for all (e.g. just after rebooting of you client).<br />
If your ssh command is issued in a window inside a VNC session, or in a window on your local laptop, chances are you can just type "ssh-add" to store the password in a process called ssh-agent, which typically is already running (most X-sessions -- such as the ones you start under VNC -- are automatically started as child processes of an ssh-agent).<br />
<br />
If an ssh-agent process is already running (and your ssh command is a descendant of it), it stores the credentials created by your once-per-reboot ssh-add command and automatically answers requests from hosts you try to connect to. If for some reason no ssh-agent is running on an intermediate host, then as an alternative you can forward your ssh-credentials from your laptop, by using "ssh -A" to login to astro0X, and continuing an ssh from there (having stored the id_rsa.pub from your laptop on the remote supercomputer).<br />
Finally, if for some reason you start out from a place (such a just a command window) where no ssh-agent is running, you can just start one, by doing<br />
<br />
<pre><br />
ssh-agent bash<br />
</pre><br />
<br />
This starts an ssh-agent, which starts bash (or tcsh) as a descendant. Then execute "ssh-add", and you're free from typing the pass phrase, for as long as you keep the ssh-agent running.<br />
<br />
'''How do I get my IP white-listed'''<br />
<br />
On any of the astro0X hosts, use the command<br />
<br />
<pre><br />
hpc-setup-firewall.sh<br />
</pre><br />
<br />
You have three personal slots, but other people's slots also work for you. The first time you login from home, or a new place, you may need to login via muon.nbi.dk, or some other NBI host, from which astro0X is already open. To find your IP-number, either use one of the web-services (but beware of spam-ware), or just type "finger -m $user" after logging in to an NBI host.<br />
SSHFS.<br />
<br />
You can locally access remote data on the frontend very conveniently by the SSH file system ([http://en.wikipedia.org/wiki/SSHFS#readme SSHFS]). To use SSHFS you need to install [http://en.wikipedia.org/wiki/Filesystem_in_Userspace#readme FUSE]. For Linux you can install [http://fuse.sourceforge.net/#readme fuse] and for Mac there is [https://osxfuse.github.io/#readem osxfuse]. You need to create an empty file as mount point on your local laptop/computer, e.g.<br />
<br />
<pre><br />
makedir ~/nbi<br />
</pre><br />
<br />
Then you mount your by specifying the host and the mount point.<br />
<br />
<pre><br />
sshfs username@astro06.hpc.ku.dk:/astro/username/ ~/nbi/ -oauto_cache,reconnect,volname=nbi<br />
</pre><br />
<br />
You can unmount the filesystem with<br />
<br />
<pre><br />
fusermount -u ~/nbi<br />
</pre><br />
<br />
SSH tunneling: To create a tunnel for for example display :11 on astro06 using ssh, do<br />
<br />
<pre><br />
ssh astro06 -L 5911:localhost:5911<br />
</pre><br />
<br />
You can do the same without starting a remote shell by doing<br />
<br />
<pre><br />
ssh astro06 -L 5911:localhost:5911 -fN<br />
</pre><br />
<br />
If you succeed in always using display :11 you can add the tunnel configuration as part of the ~/.ssh/config file:<br />
<br />
<pre><br />
Host tunnel <br />
User username <br />
HostName astro06.hpc.ku.dk <br />
LocalForward 5911 <br />
localhost:5911<br />
</pre><br />
<br />
and then start the tunnel with<br />
<br />
<pre><br />
ssh tunnel -fN<br />
</pre><br />
<br />
===GENERAL & SUPPORT===<br />
'''Whom should I ask which questions?'''<br />
# login nodes are unreachable: mailto:support@hpc.ku.dk<br />
# login nodes are alive, but I don't have a homedir: mailto:support@hpc.ku.dk<br />
# I don't have access to the software I need: mailto:support@hpc.ku.dk<br />
# forgot my password: mailto:support@hpc.ku.dk<br />
* Is there a sharing policy?<br />
# cluster queues<br />
# disk space<br />
# analysis servers<br />
* Can I run codes interactively, fx on a front end, or on an analysis server?<br />
<br />
<br />
ENVIRONMENTS<br />
<br />
* What are 'modules' and how do I use them?<br />
* What is my default environment setup?<br />
- MPI <br />
- libraries<br />
- compilers<br />
-...<br />
<br />
* Is there a standard module to load which lays out everything - no sweat?<br />
Yes, several standard modules are loaded by default via the same<br />
<br />
<pre><br />
source<br />
</pre><br />
<br />
command that makes the module command available:<br />
<br />
<pre><br />
source /software/astro/startup.{sh,csh} <br />
</pre><br />
<br />
Use the .sh for bash and .csh for tcsh.<br />
To undo the default and start from scratch, use<br />
<br />
<pre><br />
module purge<br />
</pre><br />
<br />
<br />
SOFTWARE<br />
<br />
* What software do I have available?<br />
* Do I have access <br />
* What is SLURM?<br />
* Can I submit SLURM jobs from anywhere?<br />
* How do I find out which queues I can submit to?<br />
<br />
<br />
STORAGE<br />
<br />
* Where should I put data from large simulations?<br />
* How much disk space can I claim?<br />
* How long can I have stuff on disk?<br />
* Are my data backed up, and where?</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=FAQs&diff=228FAQs2025-03-06T09:08:07Z<p>Haugboel: </p>
<hr />
<div>'''How do I access the HPC Cluster?'''<br />
<br />
You can remotely access the frontend machines astro01-09 via SSH to submit jobs or to analyze data (see [[Hardware]] for an up to date list of available frontends). For example, you can login to the astro06 machine entering the following in the command line<br />
<br />
<pre><br />
ssh username@astro06.hpc.ku.dk<br />
</pre><br />
<br />
From which you will be prompted to enter your password (Note: If the connection times out, it's likely that your IP address is not recognised). By entering the correct password, you will arrive at your home directory /groups/astro/username. It may be a good idea to check the load factor after logging in, using "top", and choose a different frontend if the CPU or memory use is already high (use "<" or ">" in top to temporarily change from sorting on CPU to memory / virtual memory).<br />
<br />
You can cut this down to<br />
<br />
<pre><br />
ssh astro06<br />
</pre><br />
<br />
by adding these lines to the file ~/.ssh/config:<br />
<br />
<pre><br />
Host astro06 <br />
User username <br />
HostName astro06.hpc.ku.dk<br />
</pre><br />
<br />
Do not use the ssh option -Y, which "enables trusted X11 forwarding". This means, basically, that you turn off some essential X security features and say "I trust the remote host completely". Forwarding of X should work without any extra options.<br />
<br />
'''How can I setup a passwordless ssh?'''<br />
<br />
You can avoid to type in your password each time for the authenticating of SSH sessions, by using an SSH key with the [http://en.wikipedia.org/wiki/RSA_(cryptosystem)#readme RSA] or [http://en.wikipedia.org/wiki/Digital_Signature_Algorithm# readme DSA] encryption, in combination with ssh-add and ssh-agent. Therefore, you generate with [http://en.wikipedia.org/wiki/Ssh-keygen#readme ssh-keygen] on your local laptop/computer a pair of a personal (id_rsa) and public key (id_rsa.pub).<br />
<br />
<pre><br />
ssh-keygen -t rsa<br />
</pre><br />
<br />
With the option -t you specify the encryption type. Here we use RSA, but you can also use DSA. You will asked for the filename, you can just press enter and use the default. Then, you will be asked for a passphrase to protect your personal key.<br />
You should NOT under any circumstances use an empty pass phrase; it is not necessary for convenience reasons (see below), and could endanger your access to remote supercomputers. Use instead a "pass phrase"; similar to a password.<br />
There will be stored two keys (id_rsa and id_rsa.pub) in the hidden folder ~/.ssh/ on your client (laptop or remote host you want to be a client of another host). You need to copy your public key to the accepted key list of on your host machine (authorized_keys).<br />
<br />
<pre><br />
cat ~/.ssh/id_rsa.pub | ssh username@astro06.hpc.ku.dk 'cat >> .ssh/authorized_keys'<br />
</pre><br />
<br />
When you connect now, the server will request your pass phrase from the client (the one where you issued the ssh-keygen command). However, rather than wait for the ssh prompt, you should use "ssh-add" to give the pass phrase, once and for all (e.g. just after rebooting of you client).<br />
If your ssh command is issued in a window inside a VNC session, or in a window on your local laptop, chances are you can just type "ssh-add" to store the password in a process called ssh-agent, which typically is already running (most X-sessions -- such as the ones you start under VNC -- are automatically started as child processes of an ssh-agent).<br />
<br />
If an ssh-agent process is already running (and your ssh command is a descendant of it), it stores the credentials created by your once-per-reboot ssh-add command and automatically answers requests from hosts you try to connect to. If for some reason no ssh-agent is running on an intermediate host, then as an alternative you can forward your ssh-credentials from your laptop, by using "ssh -A" to login to astro0X, and continuing an ssh from there (having stored the id_rsa.pub from your laptop on the remote supercomputer).<br />
Finally, if for some reason you start out from a place (such a just a command window) where no ssh-agent is running, you can just start one, by doing<br />
<br />
<pre><br />
ssh-agent bash<br />
</pre><br />
<br />
This starts an ssh-agent, which starts bash (or tcsh) as a descendant. Then execute "ssh-add", and you're free from typing the pass phrase, for as long as you keep the ssh-agent running.<br />
<br />
'''How do I get my IP white-listed'''<br />
<br />
On any of the astro0X hosts, use the command<br />
<br />
<pre><br />
hpc-setup-firewall.sh<br />
</pre><br />
<br />
You have three personal slots, but other people's slots also work for you. The first time you login from home, or a new place, you may need to login via muon.nbi.dk, or some other NBI host, from which astro0X is already open. To find your IP-number, either use one of the web-services (but beware of spam-ware), or just type "finger -m $user" after logging in to an NBI host.<br />
SSHFS.<br />
<br />
You can locally access remote data on the frontend very conveniently by the SSH file system ([http://en.wikipedia.org/wiki/SSHFS#readme SSHFS]). To use SSHFS you need to install [http://en.wikipedia.org/wiki/Filesystem_in_Userspace#readme FUSE]. For Linux you can install [http://fuse.sourceforge.net/#readme fuse] and for Mac there is [https://osxfuse.github.io/#readem osxfuse]. You need to create an empty file as mount point on your local laptop/computer, e.g.<br />
<br />
<pre><br />
makedir ~/nbi<br />
</pre><br />
<br />
Then you mount your by specifying the host and the mount point.<br />
<br />
<pre><br />
sshfs username@astro06.hpc.ku.dk:/astro/username/ ~/nbi/ -oauto_cache,reconnect,volname=nbi<br />
</pre><br />
<br />
You can unmount the filesystem with<br />
<br />
<pre><br />
fusermount -u ~/nbi<br />
</pre><br />
<br />
SSH tunneling: To create a tunnel for for example display :11 on astro06 using ssh, do<br />
<br />
<pre><br />
ssh astro06 -L 5911:localhost:5911<br />
</pre><br />
<br />
You can do the same without starting a remote shell by doing<br />
<br />
<pre><br />
ssh astro06 -L 5911:localhost:5911 -fN<br />
</pre><br />
<br />
If you succeed in always using display :11 you can add the tunnel configuration as part of the ~/.ssh/config file:<br />
<br />
<pre><br />
Host tunnel <br />
User username <br />
HostName astro06.hpc.ku.dk <br />
LocalForward 5911 <br />
localhost:5911<br />
</pre><br />
<br />
and then start the tunnel with<br />
<br />
<pre><br />
ssh tunnel -fN<br />
</pre><br />
<br />
===GENERAL & SUPPORT===<br />
'''Whom should I ask which questions?'''<br />
# login nodes are unreachable: mailto:support@hpc.ku.dk<br />
# login nodes are alive, but I don't have a homedir: mailto:support@hpc.ku.dk<br />
# I don't have access to the software I need: mailto:support@hpc.ku.dk<br />
# forgot my password: mailto:support@hpc.ku.dk<br />
* Is there a sharing policy?<br />
# cluster queues<br />
# disk space<br />
# analysis servers<br />
* Can I run codes interactively, fx on a front end, or on an analysis server?<br />
<br />
<br />
ENVIRONMENTS<br />
<br />
* What are 'modules' and how do I use them?<br />
* What is my default environment setup?<br />
- MPI <br />
- libraries<br />
- compilers<br />
-...<br />
<br />
* Is there a standard module to load which lays out everything - no sweat?<br />
Yes, several standard modules are loaded by default via the same<br />
<br />
<pre><br />
source<br />
</pre><br />
<br />
command that makes the module command available:<br />
<br />
<pre><br />
source /software/astro/startup.{sh,csh} <br />
</pre><br />
<br />
Use the .sh for bash and .csh for tcsh.<br />
To undo the default and start from scratch, use<br />
<br />
<pre><br />
module purge<br />
</pre><br />
<br />
<br />
SOFTWARE<br />
<br />
* What software do I have available?<br />
* Do I have access <br />
* What is SLURM?<br />
* Can I submit SLURM jobs from anywhere?<br />
* How do I find out which queues I can submit to?<br />
<br />
<br />
STORAGE<br />
<br />
* Where should I put data from large simulations?<br />
* How much disk space can I claim?<br />
* How long can I have stuff on disk?<br />
* Are my data backed up, and where?</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=227Hardware2025-03-06T08:44:32Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 GB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro04, and the different sized virtual GPUs on astro02) to determine how well the workload scales.<br />
<br />
* The A100 GPUs available on the astro01 frontend machine and in the astro2_gpu queue are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. You can read more about their specs here https://www.nvidia.com/en-us/data-center/a100 .<br />
<br />
* The three A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu .<br />
<br />
* The four RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/rtx-a6000 .<br />
<br />
* The six H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/h100 .<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=226Hardware2025-03-06T08:43:19Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 GB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro2, and astro04) to determine how well the workload scales.<br />
<br />
* The A100 GPUs available on the astro01 frontend machine and in the astro2_gpu queue are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. You can read more about their specs here https://www.nvidia.com/en-us/data-center/a100 .<br />
<br />
* The three A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu .<br />
<br />
* The four RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/rtx-a6000 .<br />
<br />
* The six H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. You can read more about their specs here https://www.nvidia.com/en-us/design-visualization/h100 .<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=225Hardware2025-03-06T08:42:46Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 GB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro2, and astro04) to determine how well the workload scales.<br />
<br />
* The A100 GPUs available on the astro01 frontend machine and in the astro2_gpu queue are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. You can read more about their specs here [https://www.nvidia.com/en-us/data-center/a100].<br />
<br />
* The three A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. You can read more about their specs here https://www.nvidia.com/en-us/data-center/products/a30-gpu.<br />
<br />
* The four RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. You can read more about their specs here [https://www.nvidia.com/en-us/design-visualization/rtx-a6000].<br />
<br />
* The six H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. You can read more about their specs here [https://www.nvidia.com/en-us/design-visualization/h100].<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=224Hardware2025-03-06T08:42:19Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 GB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro2, and astro04) to determine how well the workload scales.<br />
<br />
* The A100 GPUs available on the astro01 frontend machine and in the astro2_gpu queue are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. You can read more about their specs here [https://www.nvidia.com/en-us/data-center/a100].<br />
<br />
* The three A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. You can read more about their specs here [https://www.nvidia.com/en-us/data-center/products/a30-gpu].<br />
<br />
* The four RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. You can read more about their specs here [https://www.nvidia.com/en-us/design-visualization/rtx-a6000].<br />
<br />
* The six H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. You can read more about their specs here [https://www.nvidia.com/en-us/design-visualization/h100].<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=223Hardware2025-03-06T08:41:15Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 GB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===GPUs===<br />
GPUs are accessible interactively on the astro01, astro02, and astro04 frontends machines and through SLURM in the astro_gpu and astro2_gpu queues. GPUs are small in numbers on Tycho but provide potentially enormous computational value. _Please_ test if your code can efficiently use e.g. a full GPU, or more than one GPU, before running long production jobs on them. In particular machine-learning jobs, or codes that off-load calculations from high-level languages such as Python and Julia may effectively block a full GPU, and sometimes speculatively allocate all GPU memory, without actually making good use of the resources. Therefore, test by either profiling your code, using timers, or simply running on the different machines (astro01, astro2, and astro04) to determine how well the workload scales.<br />
<br />
* The A100 GPUs available on the astro01 frontend machine and in the astro2_gpu queue are equipped with 40 GB of memory. They have full FP64 performance and are well suited for large-scale computations as well as machine learning workloads, but they are not the best ML GPUs available at Tycho. You can read more about their specs here [[https://www.nvidia.com/en-us/data-center/a100]].<br />
<br />
* The three A30 GPUs available on astro02 have 24GB of memory per GPU. They have been split up so that the first GPU is fully available, the second GPU is split in to 2 virtual GPUs (each with 28 SMs), and the third GPU is split in to 4 virtual GPUs (each with 14 SMs). The GPUs on astro02 have full FP64 capabilities, they are smaller than on other machines and very useful for longer running jobs that only require a smaller amount of GPU computing. You can read more about their specs here [[https://www.nvidia.com/en-us/data-center/products/a30-gpu]]. <br />
<br />
* The four RTX A6000 GPUs available on astro04 have 48 GB ram per GPU and very low FP64 performance. They are therefore not well suited for scientific calculations, but provide hardware accelerated remote visualization, and have similar performance for machine learning workloads to the A100 GPUs. You can read more about their specs here [[https://www.nvidia.com/en-us/design-visualization/rtx-a6000]].<br />
<br />
* The six H100 GPUs available through the astro_gpu queue have 94 GB of memory and are our newest GPUs. They have full FP64 performance and very high machine learning performance and are well-suited for both scientific and machine learning jobs. You can read more about their specs here [[https://www.nvidia.com/en-us/design-visualization/h100]].<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Tycho_Technical_Documentation&diff=222Tycho Technical Documentation2025-03-06T08:20:39Z<p>Haugboel: </p>
<hr />
<div>Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.<br />
<br />
Tycho contain in total more than 10,000 CPU cores and 13 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current FP64 peak performance is 443 TFlops from the CPUs and 276 TFlops from the GPUs. The GPUs are even more powerful at lower precision having 700 TFlops FP32 performance, and more than 18 PFlops of FP16 Tensor core performance for machine learning workloads.<br />
<br />
Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.<br />
<br />
===First steps===<br />
Please visit the [[first steps]] page to get started<br />
<br />
===Overview===<br />
* [[Acknowledging the use of Tycho in articles and presentations]] <br />
* [[Getting Help]]<br />
* [[Mattermost discussion forum]]<br />
* [[Being a good HPC user]]<br />
* [[Accessing Tycho]]<br />
* [[Using Jupyter notebooks on the frontends]]<br />
* [[Visual Studio Remote Development]]<br />
* [[Virtual Desktop]]<br />
* [[Hardware]]<br />
* [[Erda]]<br />
* [[Using GPUs]]<br />
* [[Compilers]]<br />
* [[Debugging and Profiling]]<br />
* [[MPI Libraries]]<br />
* [[Running batch jobs]]<br />
* [[Examples of SLURM scripts]]<br />
* [[Codes]]<br />
* [[Adding a second IP Address]]<br />
* [[Setting up One-Time-Password Access before travelling]]<br />
* [[FAQs]]<br />
<br />
===Scientific Software===<br />
<br />
* [[Module system]]<br />
* [[Running Mathematica on compute nodes]]<br />
* [[GRChombo]]<br />
<br />
<br />
*************************<br />
<br />
Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.<br />
<br />
== Getting started ==<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Tycho_Technical_Documentation&diff=221Tycho Technical Documentation2025-03-06T08:10:47Z<p>Haugboel: </p>
<hr />
<div>Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.<br />
<br />
Tycho contain in total more than 10,000 CPU cores and 13 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current peak performance is 443 TFlops from the CPUs and 93 TFlops from the GPUs.<br />
<br />
Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.<br />
<br />
===First steps===<br />
Please visit the [[first steps]] page to get started<br />
<br />
===Overview===<br />
* [[Acknowledging the use of Tycho in articles and presentations]] <br />
* [[Getting Help]]<br />
* [[Mattermost discussion forum]]<br />
* [[Being a good HPC user]]<br />
* [[Accessing Tycho]]<br />
* [[Using Jupyter notebooks on the frontends]]<br />
* [[Visual Studio Remote Development]]<br />
* [[Virtual Desktop]]<br />
* [[Hardware]]<br />
* [[Erda]]<br />
* [[Using GPUs]]<br />
* [[Compilers]]<br />
* [[Debugging and Profiling]]<br />
* [[MPI Libraries]]<br />
* [[Running batch jobs]]<br />
* [[Examples of SLURM scripts]]<br />
* [[Codes]]<br />
* [[Adding a second IP Address]]<br />
* [[Setting up One-Time-Password Access before travelling]]<br />
* [[FAQs]]<br />
<br />
===Scientific Software===<br />
<br />
* [[Module system]]<br />
* [[Running Mathematica on compute nodes]]<br />
* [[GRChombo]]<br />
<br />
<br />
*************************<br />
<br />
Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.<br />
<br />
== Getting started ==<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=220Hardware2025-03-06T08:10:00Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 GB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01, astro02, and astro04 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=219Hardware2025-03-06T08:06:37Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 GB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 3 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01 and astro02 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=218Hardware2025-02-21T12:07:06Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 GB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|-<br />
| astro06.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|-<br />
| astro07.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 2 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01 and astro02 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=217Hardware2025-02-21T12:06:30Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro03.hpc.ku.dk || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 1,5 GB DDR5-4800 MHz - 12 GB / core || 922 GB/s, 7.2 GB/s/core || None || None || L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s to storage<br />
|-<br />
| astro04.hpc.ku.dk || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 9.6 GB/s/core || 4x RTX A6000 || 42 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, 2 x NDR 200 Gbit/s to storage<br />
|-<br />
| astro06.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|-<br />
| astro07.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro_gpu || 2 || 1 x 48 cores Epyc Genoa 9454P @ 2.75 GHz || 768 GB DDR5-4800 MHz - 16 GB / core || 461 GB/s, 9.6 GB/s/core || 2x H100 GPUs, L2: 1 MB / core, L3: 256 MB, AVX-512, HDR 100 Gbit/s<br />
||-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01 and astro02 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Ethernet or Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=216Hardware2024-03-01T07:51:42Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro06.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|-<br />
| astro07.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01 and astro02 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point in Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The backend storage servers for /groups/astro and /lustre/astro are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=215Hardware2024-03-01T07:49:07Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro06.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|-<br />
| astro07.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. The total space (disregarding the transparent compression) is 1300 TB. The default quota on scratch is 5 TB, but if you need more please contact Troels Haugbølle with your supervisor / mentor / sponsor in CC and explain why and how much.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01 and astro02 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The storage server are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=214Hardware2024-03-01T07:43:51Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro06.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|-<br />
| astro07.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. No quotas are enforced and the total space (disregarding the transparent compression) is 1300 TB.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01 and astro02 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The storage server are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Mattermost_discussion_forum&diff=213Mattermost discussion forum2024-01-30T14:29:26Z<p>Haugboel: </p>
<hr />
<div>We have a dedicated user forum for the cluster where you can contact other users, and get more instantaneous community feedback. This is also a good place to discuss usage or installation of scientific software.<br />
<br />
To get an account, please see the sign up link that you received in your welcome e-mail to the HPC system. Signup is restricted based on the e-mail account you submit. Please use your work e-mail. If you are an external user of Tycho (e.g. not from NBI or Globe), please contact Troels Haugbølle (haugboel@nbi.ku.dk)</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=212First steps2024-01-15T08:14:49Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group. '''Remember to sign the rules-of-conduct-form'''<br />
<br />
* If you have received the old welcome e-mail without a link to this wiki, the info is not correct. You should never login to fend0X.hpc.ku.dk, but instead use the Tycho-specific frontends called astro0X.hpc.ku.dk (see [[Hardware]] for an up to date list of available frontends / Analysis hardware). <br />
<br />
* Set up an SSH key pair for secure passwordless login (see [[Accessing Tycho]])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software. You can read more about the <code>module</code> command.<br />
<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.<br />
<br />
* If you are going to travel, but want to continue working on Tycho while you are travelling, you need to set up a Dynamic Firewall '''before''' traveling. You can do so by following the detailed instructions [https://hpc.ku.dk/documentation/otp.html: here]<br />
<br />
* You can set up e.g. Visual Studio Code for remote development for transparent editing of files on the cluster. See [[Visual Studio Remote Development]]<br />
<br />
* Every user has a 50 GB quota for the home folder (<code>/groups/astro/yourusername</code>. Whenever you are going to be working with large amounts of data, considering using the <code>/lustre/astro/yourusername</code> directory. This scratch folder is residing on a ZFS based high performance Lustre filesystem with dedicated hardware for our group. No quotas are enforced and the total space (disregarding the transparent compression) is 1300 TB.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=180First steps2023-11-15T14:37:15Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group. '''Remember to sign the rules-of-conduct-form'''<br />
<br />
* '''BE AWARE THAT THE WELCOME E-MAIL IS NOT CORRECT'''. You should never login to fend0X.hpc.ku.dk, but instead use the Tycho-specific frontends called astro0X.hpc.ku.dk (see [[Hardware]] for an up to date list of available frontends / Analysis hardware).<br />
<br />
* Set up an SSH key pair for secure passwordless login (see [[Accessing Tycho]])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software. You can read more about the <code>module</code> command.<br />
<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.<br />
<br />
* If you are going to travel, but want to continue working on Tycho while you are travelling, you need to set up a Dynamic Firewall '''before''' travelling. You can do so by following the detailed instructions [https://hpc.ku.dk/documentation/otp.html: here]<br />
<br />
* You can set up e.g. Visual Studio Code for remote development for transparent editing of files on the cluster. See [[Visual Studio Remote Development]]<br />
<br />
* Every user has a 50 GB quota for the home folder (<code>/groups/astro/yourusername</code>. Whenever you are going to be working with large amounts of data, considering using the <code>/lustre/astro/yourusername</code> directory. This scratch folder is residing on a ZFS based high performance Lustre filesystem with dedicated hardware for our group. No quotas are enforced and the total space (disregarding the transparent compression) is 1300 TB.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=FAQs&diff=177FAQs2023-11-15T14:28:34Z<p>Haugboel: </p>
<hr />
<div>* How do I access the HPC Cluster?<br />
Enter the following in the command terminal<br />
<br />
<pre><br />
ssh username@astroXX.hpc.ku.dk<br />
</pre><br />
<br />
You can remotely access the frontend machines astro06-09 via SSH to submit jobs or to analyze data. It may be a good idea to check the load factor after logging in, using "top", and choose a different frontend if the CPU or memory use is already high (use "<" or ">" in top to temporarily change from sorting on CPU to memory / virtiual memory).<br />
Login<br />
You can login to the astro06 machine for example by</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Compilers&diff=174Compilers2023-11-15T14:26:03Z<p>Haugboel: </p>
<hr />
<div>We recommend using the Intel ifort, icc, and icpc (fortran, C, C++) compilers for production, because of their superior speed optimization. Compiling for MPI is done with mpiifort, mpiicc, mpiicpc (note the double ii), which are wrapper scripts around the compilers that link in the correct MPI libraries. <br />
<br />
However, using gfortran for development and testing can be a very good idea, especially if / when problems arise, and are difficult to locate. Compiling the same code with different compilers gives a better chance to discover problems, or to exclude a compiler bug as the reason for a problem under investigation. Gfortran's MPI wrapper is called mpif90 or mpifort (note the single i!).</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=170First steps2023-11-15T14:24:09Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group. '''Remember to sign the rules-of-conduct-form'''<br />
<br />
* '''BE AWARE THAT THE WELCOME E-MAIL IS NOT CORRECT'''. You should never login to fend0X.hpc.ku.dk, but instead use the Tycho-specific frontends called astro0X.hpc.ku.dk (see [[Hardware]] for an up to date list of available frontends / Analysis hardware).<br />
<br />
* Set up an SSH key pair for secure passwordless login (see [[Accessing Tycho]])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software. You can read more about the <code>module</code> command.<br />
<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.<br />
<br />
* If you are going to travel, but want to continue working on Tycho while you are travelling, you need to set up a Dynamic Firewall '''before''' travelling. You can do so by following the detailed instructions [https://hpc.ku.dk/documentation/otp.html: here]<br />
<br />
* Consider setting up Visual Studio Code for remote development for transparent editing of files on the cluster. See [[Visual Studio Remote Development]]<br />
<br />
* Whenever you are going to be working with large amounts of data, considering using the <code>/lustre/astro/yourusername</code> directory. The scratch directory is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. No quotas are enforced and the total space (disregarding the transparent compression) is 1300 TB.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Acknowledging_the_use_of_Tycho_in_articles_and_presentations&diff=163Acknowledging the use of Tycho in articles and presentations2023-11-15T14:14:36Z<p>Haugboel: Created page with "Tycho is a group effort and we rely entirely on external funding to maintain the hardware. Users are strongly encouraged to acknowledge Tycho in all publications that describe results obtained using Tycho resources. Users shall use the following wording in such acknowledgement in all papers and other publications. Having a standard formulation helps when assessing the impact of Tycho in e.g. future grant applications, and benefit all of us. ====Acknowledging Tycho in a..."</p>
<hr />
<div>Tycho is a group effort and we rely entirely on external funding to maintain the hardware. Users are strongly encouraged to acknowledge Tycho in all publications that describe results obtained using Tycho resources.<br />
<br />
Users shall use the following wording in such acknowledgement in all papers and other publications. Having a standard formulation helps when assessing the impact of Tycho in e.g. future grant applications, and benefit all of us.<br />
<br />
====Acknowledging Tycho in a publication====<br />
The Tycho supercomputer hosted at the SCIENCE HPC center at the University of Copenhagen was used for supporting this work.<br />
<br />
====Referencing Tycho in a presentation====<br />
You may reference Tycho in your presentations. You may use the Tycho logo and/or select images from the [[media gallery]].</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Tycho_Technical_Documentation&diff=139Tycho Technical Documentation2023-11-15T14:00:45Z<p>Haugboel: </p>
<hr />
<div>Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.<br />
<br />
Tycho contain in total more 10,000 CPU cores and 13 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current peak performance is 443 TFlops from the CPUs and 93 TFlops from the GPUs.<br />
<br />
Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.<br />
<br />
===First steps===<br />
Please visit the [[first steps]] page to get started<br />
<br />
===Overview===<br />
* [[Acknowledging the use of Tycho in articles and presentations]] <br />
* [[Getting Help]]<br />
* [[Mattermost discussion forum]]<br />
* [[Being a good HPC user]]<br />
* [[Accessing Tycho]]<br />
* [[Using Jupyter notebooks on the frontends]]<br />
* [[Visual Studio Remote Development]]<br />
* [[Virtual Desktop]]<br />
* [[Hardware]]<br />
* [[Erda]]<br />
* [[Using GPUs]]<br />
* [[Compilers]]<br />
* [[Debugging and Profiling]]<br />
* [[MPI Libraries]]<br />
* [[Examples of SLURM scripts]]<br />
* [[Codes]]<br />
* [[Adding a second IP Address]]<br />
* [[Setting up One-Time-Password Access before travelling]]<br />
<br />
<br />
===Scientific Software===<br />
<br />
* [[Running Mathematica on compute nodes]]<br />
* [[GRChombo]]<br />
<br />
<br />
*************************<br />
<br />
Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.<br />
<br />
== Getting started ==<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=135First steps2023-11-15T13:58:14Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group. '''Remember to sign the rules-of-conduct-form'''<br />
<br />
* '''BE AWARE THAT THE WELCOME E-MAIL IS NOT CORRECT'''. You should never login to fend0X.hpc.ku.dk, but instead use the Tycho-specific frontends called astro0X.hpc.ku.dk (see [[Hardware]] for an up to date list of available frontends / Analysis hardware).<br />
<br />
* Set up an SSH key pair for secure passwordless login (see [[Accessing Tycho]])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software.<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.<br />
<br />
* If you are going to travel, but want to continue working on Tycho while you are travelling, you need to set up a Dynamic Firewall '''before''' travelling. You can do so by following the detailed instructions [https://hpc.ku.dk/documentation/otp.html: here]<br />
<br />
* Consider setting up Visual Studio Code for remote development for transparent editing of files on the cluster. See [[Visual Studio Remote Development]]</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=128First steps2023-11-15T13:54:07Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group. '''Remember to sign the rules-of-conduct-form'''<br />
<br />
* '''BE AWARE THAT THE WELCOME E-MAIL IS NOT CORRECT'''. You should never login to fend0X.hpc.ku.dk, but instead use the Tycho-specific frontends called astro0X.hpc.ku.dk (see [[Hardware]] for an up to date list of available frontends / Analysis hardware).<br />
<br />
* Set up an SSH key pair for secure passwordless login (see [[Accessing Tycho]])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software.<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Tycho_Technical_Documentation&diff=122Tycho Technical Documentation2023-11-15T13:51:09Z<p>Haugboel: </p>
<hr />
<div>Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.<br />
<br />
Tycho contain in total more 10,000 CPU cores and 13 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current peak performance is 443 TFlops from the CPUs and 93 TFlops from the GPUs.<br />
<br />
Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.<br />
<br />
===First steps===<br />
Please visit the [[first steps]] page to get started<br />
<br />
===Overview===<br />
* [[Getting Help]]<br />
* [[Mattermost discussion forum]]<br />
* [[Being a good HPC user]]<br />
* [[Accessing Tycho]]<br />
* [[Using Jupyter notebooks on the frontends]]<br />
* [[Visual Studio Remote Development]]<br />
* [[Virtual Desktop]]<br />
* [[Hardware]]<br />
* [[Erda]]<br />
* [[Using GPUs]]<br />
* [[Compilers]]<br />
* [[Debugging and Profiling]]<br />
* [[MPI Libraries]]<br />
* [[Examples of SLURM scripts]]<br />
* [[Codes]]<br />
* [[Adding a second IP Address]]<br />
* [[Setting up One-Time-Password Access before travelling]]<br />
<br />
<br />
===Scientific Software===<br />
<br />
* [[Running Mathematica on compute nodes]]<br />
* [[GRChombo]]<br />
<br />
<br />
*************************<br />
<br />
Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.<br />
<br />
== Getting started ==<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=119First steps2023-11-15T13:49:58Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group<br />
<br />
* '''BE AWARE THAT THE WELCOME E-MAIL IS NOT CORRECT'''. You should never login to fend0X.hpc.ku.dk, but instead use the Tycho-specific frontends called astro0X.hpc.ku.dk (see [[Hardware]] for an up to date list of available frontends / Analysis hardware).<br />
<br />
* Set up an SSH key pair for secure passwordless login (see [[Accessing Tycho]])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software.<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=116First steps2023-11-15T13:49:30Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group<br />
<br />
* '''BE AWARE THAT THE WELCOME E-MAIL IS NOT CORRECT'''. You should never login to fend0X.hpc.ku.dk, but instead use the Tycho-specific frontends called astro0X.hpc.ku.dk (see [[Hardware]] for a list of available frontends).<br />
<br />
* Set up an SSH key pair for secure passwordless login (see [[Accessing Tycho]])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software.<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Hardware&diff=115Hardware2023-11-15T13:48:24Z<p>Haugboel: </p>
<hr />
<div>Tycho contains both frontends ("Analysis Hardware") that are accessible from the outside and which can be used for interactive work, such as development and analysis, and compute nodes ("Cluster Hardware") that are only accessible through the SLURM queue systems.<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Analysis Hardware<br />
|-<br />
! Name !! CPUs !! Memory !! Memory Bandwidth !! GPUs !! Scratch !! Notes<br />
|-<br />
| astro01.hpc.ku.dk || 2 x 24 cores Epyc Rome 7F72 @ 3.2 GHz || 1 TB DDR4-3200 MHz - 21 GB / core || 410 GB/s, 8.5 GB/s/core || 4x A100 || 11 TB || L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s to storage<br />
|-<br />
| astro02.hpc.ku.dk || 1 x 64 cores Epyc Genoa 9554P @ 3.1 GHz || 768 GB DDR5-4800 MHz - 12 GB / core || 461 GB/s, 7.2 GB/s/core || 3x A30 || 28 TB || L2: 1 MB / core, L3: 256 MB, AVX-512, EDR 100 Gbit/s to storage<br />
|-<br />
| astro06.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|-<br />
| astro06.hpc.ku.dk || 2 x 14 cores Broadwell E5-2680 v4 @ 2.40GHz || 512 GB DDR4-2400 MHz - 20 GB / core || || || || L2: 512 KB / core, L3: 35 MB / socket, AVX2, QDR 40 Gbit/s to storage<br />
|}<br />
<br />
{| class="wikitable" style="margin:auto"<br />
|+ Cluster Hardware<br />
|-<br />
! Queue Name !! #Nodes !! CPUs !! Memory !! Memory Bandwidth !! Notes<br />
|-<br />
| astro_XX || 16 || 2 x 10 cores Xeon E5-2680v2 @ 2.8GHz || 64 GB DDR3-1866 MHz - 3.2 GB / core || 120 GB/s, 6 GB/s/core || L2: 256 KB / core, L3: 25 MB / socket, AVX2, FDR 56 Gbit/s<br />
|-<br />
| astro2_XX || 70 || 2 x 24 cores Xeon 6248R @ 3.0GHz || 192 GB DDR4-2933 MHz - 4 GB / core || 282 GB/s, 5.9 GB/s/core || L2: 1 MB / core, L3: 35.75 MB / socket, AVX-512, EDR 100 Gbit/s<br />
|-<br />
| astro3_XX || 50 || 2 x 64 cores Epyc Genoa 9554 @ 3.1 GHz || 768 GB DDR5-4800 MHz - 6 GB / core || 922 GB/s, 7.2 GB/s/core || L2: 1 MB / core, L3: 256 MB / socket, AVX-512, 2 x NDR 200 Gbit/s<br />
|-<br />
| astro2_gpu || 1 || 2 x 16 cores Epyc Rome 7302 @ 3.0 GHz || 1 TB DDR4-3200 MHz - 32 GB / core || 410 GB/s, 12.8 GB/s/core || 4x A100 GPUs, L2: 512 KB / core, L3: 192 MB / socket, AVX2, EDR 100 Gbit/s<br />
|}<br />
<br />
===Servers===<br />
The astro_XX nodes are based on a Dell C6220II shoe-box design with dual 10-core ivy-bridge CPUs. The astro2_XX nodes are Huawei Fusion server pro X6000 with dual 24-core cascade-lake CPUs. The astro3_XX nodes are XFusion servers model 1258H V7 with dual 64-core Genoa CPUs.<br />
<br />
===Global Storage===<br />
* The home directory (/groups/astro) is a fully backed up Lustre filesystem. We have a shared 6TB quota and individual quotas of 50 GB per user.<br />
* The scratch directory (/lustre/astro) is a ZFS based high performance Lustre filesystem with dedicated hardware for our group. No quotas are enforced and the total space (disregarding the transparent compression) is 1300 TB.<br />
* Archive are two ZFS filesystems exported as NFS volumes from a storage server connected to the clusters with a 10 Gbit/s network connection. Each archive system can be found under /groups/astro/archive0 and /groups/astro/archive1. These filesystems are old and new users will not get directories on them. They will soon be decommissioned.<br />
<br />
===Scratch storage===<br />
The scratch disks on astro01 and astro02 are RAID0 volumes consisting of a number of locally mounted NVMe disks. They have slightly higher bandwidth than the global filesystem, but can only be accessed from the specific machine. The scratch disks have several orders of magnitudes higher IOPS compared to the global filesystem, and random access I/O or operations that require opening and closing a lot of files will perform a lot faster on the scratch disks. _Space is limited_. Please clean up after use, and remember there are no backups or redundancy on the scratch disks.<br />
<br />
===Networks===<br />
* External connection: The local HPC center is a Tier-1 CERN node and has a direct dual 400 gbit/s connection to the Danish entrance point Lyngby of the European GEANT network. In practice we easily reach 100 MB/s for transfer of larger files, with higher speeds possible by doing parallel transfers.<br />
* The storage server are all inter-connected with 100 Gbit/s HDR Infiniband. This switch has uplinks to the different cluster networks.<br />
* All frontend machines have Infiniband adapters to provide optimal bandwidth to the I/O.<br />
* Astro_XX nodes have FDR (56 Gbit/s) infiniband connected to a single switch.<br />
* Astro2_XX nodes have EDR (100 Gbit/s) with a 2:1 blocking factor and 24 nodes per switch (3 uplink switches, 1 core switch).<br />
* Astro3_XX nodes have two NDR-200 (200 Gbit/s) adapters with one adapter per CPU socket. They are connected directly to a single 128-port NDR-200 switch.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=112First steps2023-11-15T13:47:28Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group<br />
<br />
* '''BE AWARE THAT THE WELCOME E-MAIL IS NOT CORRECT'''. Most important, you should never login to fend0X.hpc.ku.dk, but instead use the Tycho-specific frontends called astro0X.hpc.ku.dk (see [[Hardware]] for a list of available frontends).<br />
<br />
* Set up an SSH key pair for secure passwordless login (see [[Accessing Tycho]])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software.<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=107First steps2023-11-15T13:44:45Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group<br />
<br />
* Set up an SSH key pair for secure passwordless login (see [Accessing Tycho])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software.<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.<br />
<br />
<pre><br />
fdsnkflsdnk<br />
sdfnlkdsf<br />
ksndfds<br />
</pre></div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=101First steps2023-11-15T13:41:41Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html https://hpc.ku.dk/account.html<br />
<br />
You have to select "Astro" as your group<br />
<br />
* Set up an SSH key pair for secure passwordless login (see [Accessing Tycho])<br />
<br />
* Add <code>module load astro</code> to you <code>$HOME/.bashrc</code> file to have access to all the custom installed software.<br />
* Consider changing <code>umask 077</code> to <code>umask 027</code> in <code>$HOME/.bashrc</code> to allow collaborators and/or your supervisor read access to your files when logged in to the cluster.</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Tycho_Technical_Documentation&diff=90Tycho Technical Documentation2023-11-15T13:36:37Z<p>Haugboel: </p>
<hr />
<div>Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.<br />
<br />
Tycho contain in total more 10,000 CPU cores and 13 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current peak performance is 443 TFlops from the CPUs and 93 TFlops from the GPUs.<br />
<br />
Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.<br />
<br />
Please visit the [[first steps]] page to get started<br />
<br />
Overview:<br />
* [[Getting Help]]<br />
* [[Mattermost discussion forum]]<br />
* [[Being a good HPC user]]<br />
* [[Accessing Tycho]]<br />
* [[Using Jupyter notebooks on the frontends]]<br />
* [[Visual Studio Remote Development]]<br />
* [[Virtual Desktop]]<br />
* [[Hardware]]<br />
* [[Using GPUs]]<br />
* [[Compilers]]<br />
* [[Debugging and Profiling]]<br />
* [[MPI Libraries]]<br />
* [[Examples of SLURM scripts]]<br />
* [[Codes]]<br />
* Software:<br />
<br />
* [[Running Mathematica on compute nodes]]<br />
<br />
<br />
*************************<br />
<br />
Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.<br />
<br />
== Getting started ==<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Tycho_Technical_Documentation&diff=89Tycho Technical Documentation2023-11-15T13:36:06Z<p>Haugboel: </p>
<hr />
<div>Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.<br />
<br />
Tycho contain in total more 10,000 CPU cores and 13 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current peak performance is 443 TFlops from the CPUs and 93 TFlops from the GPUs.<br />
<br />
Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.<br />
<br />
Please visit the [[first steps]] page to get started<br />
<br />
Overview:<br />
* [[Getting Help]]<br />
* [[Mattermost discussion forum]]<br />
* [[Being a good HPC user]]<br />
* [[Accessing Tycho]]<br />
* [[Using Jupyter notebooks on the frontends]]<br />
* [[Visual Studio Remote Development]]<br />
* [[Virtual Desktop]]<br />
* [[Hardware]]<br />
* [[Using GPUs]]<br />
* [[Compilers]]<br />
* Software:<br />
* [[Running Mathematica on compute nodes]]<br />
<br />
<br />
* [[Debugging and Profiling]]<br />
* [[MPI Libraries]]<br />
* [[Examples of SLURM scripts]]<br />
* [[Codes]]<br />
<br />
<br />
*************************<br />
<br />
Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.<br />
<br />
== Getting started ==<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=Tycho_Technical_Documentation&diff=87Tycho Technical Documentation2023-11-15T13:35:44Z<p>Haugboel: </p>
<hr />
<div>Welcome to the technical documentation for the Tycho high performance computing resources. See [https://nbi.ku.dk/english/research_infrastructure/tycho-supercomputer/ the Infrastructure page] at NBI for a non-technical overview of the cluster.<br />
<br />
Tycho contain in total more 10,000 CPU cores and 13 data center class GPUs. The cluster is complemented by a 1,300 TB data storage archive and a number of powerful analysis machines used as frontends for the cluster and for pre- and post-prcessing. Current peak performance is 443 TFlops from the CPUs and 93 TFlops from the GPUs.<br />
<br />
Tycho is hosted at the [http://www.dcsc.ku.dk/: High Performance Computing center] at the faculty of SCIENCE, University of Copenhagen.<br />
<br />
Please visit the [[first steps]] page to get started<br />
<br />
Overview:<br />
* [[Getting Help]]<br />
* [[Mattermost discussion forum]]<br />
* [[Being a good HPC user]]<br />
* [[Accessing Tycho]]<br />
* [[Using Jupyter notebooks on the frontends]]<br />
* [[Visual Studio Remote Development]]<br />
* [[Virtual Desktop]]<br />
* [[Hardware]]<br />
* [[Using GPUs]]<br />
* [[Compilers]]<br />
* Software:<br />
* [[Running Mathematica on compute nodes]]<br />
<br />
<br />
* [[Debugging and Profiling]]<br />
* [[MPI Libraries]]<br />
* [[Examples of SLURM scripts]]<br />
* [[Codes]]<br />
<br />
<br />
*************************<br />
<br />
Consult the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents User's Guide] for information on using the wiki software.<br />
<br />
== Getting started ==<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/postorius/lists/mediawiki-announce.lists.wikimedia.org/ MediaWiki release mailing list]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Localisation#Translation_resources Localise MediaWiki for your language]<br />
* [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Combating_spam Learn how to combat spam on your wiki]</div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=79First steps2023-11-15T13:20:57Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
dsfsdfs<br />
<br />
<br />
[https://hpc.ku.dk/account.html https://hpc.ku.dk/account.html]<br />
<br />
You have to select "Astro" as your group<br />
<br />
Set up an SSH key pair for secure passwordless login (see [Accessing Tycho])<br />
<br />
Add <code>module load astro</code> to you <code>.bashrc</code></div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=76First steps2023-11-15T13:18:37Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
dsfsdfs<br />
<br />
<br />
https://hpc.ku.dk/account.html . You have to select "Astro" as your group<br />
<br />
Set up an SSH key pair for secure passwordless login (see [Accessing Tycho])<br />
<br />
Add <code>module load astro</code> to you <code>.bashrc</code></div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=71First steps2023-11-15T13:07:35Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
* Get a user account by contacting a sponsor and signing up at <br />
<br />
dsfsdfs<br />
<br />
<br />
https://hpc.ku.dk/account.html . You have to select "Astro" as your group<br />
<br />
Set up an SSH key pair for secure passwordless login (see [Accessing Tycho])<br />
<br />
Add <code>module load astro</code> to you <code>.bashrc</code></div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=69First steps2023-11-15T13:06:58Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
1. Get a user account by contacting a sponsor and signing up at <br />
<br />
https://hpc.ku.dk/account.html . You have to select "Astro" as your group<br />
<br />
Set up an SSH key pair for secure passwordless login (see [Accessing Tycho])<br />
<br />
Add <code>module load astro</code> to you <code>.bashrc</code></div>Haugboelhttps://wiki.nbi.ku.dk/w/tycho/index.php?title=First_steps&diff=68First steps2023-11-15T13:06:17Z<p>Haugboel: </p>
<hr />
<div>To start using Tycho, please follow these steps<br />
* Get a user account by contacting a sponsor and signing up at https://hpc.ku.dk/account.html . You have to select "Astro" as your group<br />
<br />
Set up an SSH key pair for secure passwordless login (see [Accessing Tycho])<br />
<br />
Add <code>module load astro</code> to you <code>.bashrc</code></div>Haugboel