GRChombo

GRChombo is an open-source, AMR-based code for Numerical Relativity. Here are some instructions on how to set up GRChombo in Tycho.

Step 1. Compile Chombo

Go to Chombo and clone the repository in the cluster:

>>git clone https://github.com/GRTLCollaboration/Chombo.git

Then we need to make a new file that will set up the compilation details, specifically tweaked for our cluster. Run these commands:

>>cd Chombo/lib
>>touch mk/Make.defs.local

Using your favourite editor (vim or nano) copy this into Make.defs.local:

DIM             = 3
DEBUG            = FALSE
CXX = g++ -I${MKLROOT}/include
FC = gfortran -I${MKLROOT}/include
OPT             = TRUE
MPI             = TRUE
OPENMPCC        = TRUE
OPT             = TRUE
USE_64          = TRUE
USE_HDF         = TRUE
MPICXX           = mpicxx -I${MKLROOT}/include
HDFINCFLAGS	= -I/software/astro/hdf5/hdf5-1.8.14-intel/include
HDFLIBFLAGS	= -L/software/astro/hdf5/hdf5-1.8.14-intel/lib -lhdf5 -lz
HDFMPIINCFLAGS	= -I/software/astro/hdf5/hdf5-1.8.14-intel/include
HDFMPILIBFLAGS  = -L/software/astro/hdf5/hdf5-1.8.14-intel/lib -lhdf5 -lz
cxxdbgflags     = -g -Wl,--eh-frame-hdr
cxxoptflags     = -O3
fdbgflags       = -g -Wl,--eh-frame-hdr
foptflags       = -O3
syslibflags      = -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl

Notice that we are NOT compiling the Apparent Horizon finder. That requires some additional work.

Now, load the appropriate modules.

>>module purge
>>module load astro
>>module load hdf5/inte/1.10.4
>>module load gcc/12.2-offload
>>module load intelmpi/18.0.3
>>module load intel/20.1

It is recommended to add these to your ~/.bashrc profile.

Now, do

make all -j 8

And it all should compile in a matter of minutes.

Step 2. Run a Binary BH

Go back to your home directory, and clone the GRChombo repository:

>> cd ~
>> git clone https://github.com/GRTLCollaboration/GRChombo.git]
>> cd GRChombo

Export the variable containing where is the Chombo library

>>export CHOMBO_HOME=~/Chombo/lib/

Let us go to the example of the Binary BH and compile the code:

>>cd GRChombo/Examples/BinaryBH/
>>make all -j 8

Now you can start the run with the following BASH script:

#!/bin/sh                                                  
#SBATCH -J BBH           		# job name                  
#SBATCH -t 6:00:00        		# walltime (dd:hh:mm:ss)    
#SBATCH -p astro2_short

#SBATCH --nodes 8
#SBATCH --ntasks-per-node=24
#SBATCH --cpus-per-task=2
# Output files
#SBATCH -o std_output.txt
#SBATCH -e std_error.txt
#SBATCH -D ./
module load intelmpi/18.0.3

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# Run the program
srun ./Main_BinaryBH3d.Linux.64.mpicxx.gfortran.OPT.MPI.OPENMPCC.ex params.txt

You can check the Wiki for more information on the parameter file params.txt, and how to visualize the output data. Visit is a good option for this.