GRChombo

[GRChombo](https://www.grchombo.org/) is an open-source, AMR-based code for Numerical Relativity. Here are some instructions on how to set up GRChombo in Tycho.

Step 1. Compile Chombo

Go to [Chombo](https://github.com/GRTLCollaboration/Chombo) and clone the repository in the cluster:

>>git clone https://github.com/GRTLCollaboration/Chombo.git

Then we need to make a new file that will set up the compilation details, specifically tweaked for our cluster. Run these commands:

>>cd Chombo/lib
>>touch mk/Make.defs.local

Using your favourite editor (vim or nano) copy this into Make.defs.local:

DIM             = 3
DEBUG            = FALSE
CXX = g++ -I${MKLROOT}/include
FC = gfortran -I${MKLROOT}/include
OPT             = TRUE
MPI             = TRUE
OPENMPCC        = TRUE
OPT             = TRUE
USE_64          = TRUE
USE_HDF         = TRUE
MPICXX           = mpicxx -I${MKLROOT}/include
HDFINCFLAGS	= -I/software/astro/hdf5/hdf5-1.8.14-intel/include
HDFLIBFLAGS	= -L/software/astro/hdf5/hdf5-1.8.14-intel/lib -lhdf5 -lz
HDFMPIINCFLAGS	= -I/software/astro/hdf5/hdf5-1.8.14-intel/include
HDFMPILIBFLAGS  = -L/software/astro/hdf5/hdf5-1.8.14-intel/lib -lhdf5 -lz
cxxdbgflags     = -g -Wl,--eh-frame-hdr
cxxoptflags     = -O3
fdbgflags       = -g -Wl,--eh-frame-hdr
foptflags       = -O3
syslibflags      = -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl

Notice that we are NOT compiling the Apparent Horizon finder. That requires some additional work.

Now, load the appropriate modules.

>>module purge
>>module load astro
>>module load hdf5/inte/1.10.4
>>module load gcc/12.2-offload
>>module load intelmpi/18.0.3
>>module load intel/20.1

It is recommended to add these to your ~/.bashrc profile.

Now, do

make all -j 8

And it all should compile in a matter of minutes.

Step 2. Run a Binary BH

Go back to your home directory, and clone the GRChombo repository:

>> cd ~
>> git clone https://github.com/GRTLCollaboration/GRChombo.git]
>> cd GRChombo

Export the variable containing where is the Chombo library

>>export CHOMBO_HOME=~/Chombo/lib/

Let us go to the example of the Binary BH and compile the code:

>>cd GRChombo/Examples/BinaryBH/
>>make all -j 8

Now you can start the run with the following BASH script:

#!/bin/sh                                                  
#SBATCH -J BBH           		# job name                  
#SBATCH -t 6:00:00        		# walltime (dd:hh:mm:ss)    
#SBATCH -p astro2_short

#SBATCH --nodes 8
#SBATCH --ntasks-per-node=24
#SBATCH --cpus-per-task=2
# Output files
#SBATCH -o std_output.txt
#SBATCH -e std_error.txt
#SBATCH -D ./
module load intelmpi/18.0.3

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# Run the program
srun ./Main_BinaryBH3d.Linux.64.mpicxx.gfortran.OPT.MPI.OPENMPCC.ex params.txt

You can check [the Wiki](https://github.com/GRTLCollaboration/GRChombo/wiki/Running-the-BinaryBH-example) for more information on the parameter file params.txt, and how to visualize the output data. Visit is a good option for this.