Building Dependencies with Spack

The Spack Package Manager may be used to download and build GCHP and all of its required external software depenencies, including CMake, MPI, ESMF, and NetCDF libraries. The only essential prerequisite for using Spack is a local C/C++/Fortran Compiler such as GNU Compiler Collection. You can use this local compiler to later install a different compiler version through Spack. GCHP requires GNU Compilers version ≥ 8.3 or Intel compilers version ≥ 18.0.5. You must install a newer compiler through Spack if your pre-installed compiler does not meet these requirements.

There are three different ways to use Spack to setup GCHP and/or its dependencies:

This page covers each of these options. For any of these options, first follow the instructions below about setting up Spack, configuring compilers in Spack, and specifying dependencies you already have installed which you would like Spack to use for building GCHP or other dependencies (such as ESMF or Slurm).

Setting up Spack, installing new compilers, and specifying preinstalled dependencies

To begin using Spack, clone the latest version by typing git clone https://github.com/spack/spack.git. Execute the following commands to initialize Spack’s environment (replacing /path/to/spack with the path of your spack directory). Add these commands to an environment initialization script for easy reuse.

$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh

If you do not already have a copy of your preferred text editor, you can use Spack to install and load one before proceeding (e.g. spack install emacs; spack load emacs).

Installing compilers

Ensure Spack recognizes any existing compiler on your system by typing spack compilers. You can use this compiler to build a new one.

Setting up and using GNU compilers

GNU compilers are free, open source, and easily installable through Spack. Execute the following at the command prompt to install version 9.3.0 of the GNU Compiler Collection:

$ spack install gcc@9.3.0

The package@VERSION notation is common to all packages in Spack and can be customized to choose different versions.

The installation of gcc may take a long time. Once it is complete, you’ll need to add it to Spack’s list of compilers using the following command:

$ spack compiler find $(spack location -i gcc@9.3.0)

Make sure that spack recognizes your new compiler by typing spack compilers, which should display a list of compilers including GCC 9.3.0.

Setting up and using Intel compilers

If you would like to use Intel compilers, whether pre-installed on your system or which you would like to install through Spack using an existing license, follow the instructions for pre-installed Intel compilers or for Intel compilers you want to install through Spack on the official Spack documentation site. In the instructions below, simply replace %gcc@9.3.0 with %intel to use your Intel compilers to build libraries.

Specifying preinstalled dependencies

Just as Spack can use an existing compiler on your system, it can also use your existing installations of dependencies for GCHP rather than building new copies. This is essential for interfacing your cluster’s Slurm with a Spack-built GCHP and its dependencies. For any preinstalled dependency you want Spack to always use, you must specify its path on your system and that you want Spack to always use this preinstalled package rather than building a new version. The code below shows how to do this by editing $HOME/.spack/packages.yaml (you may need to create this file):

packages:
 slurm:
  paths:
   slurm: /path/to/slurm
  buildable: False

Installing individual dependencies with Spack

This section describes how to use Spack to build GCHP’s individual dependencies. While these dependencies can be used to then install GCHP directly using Spack, this section is mainly intended for those looking to manually download and compile GCHP as described in the User Guide.

Installing basic dependencies

You should now install Git and CMake using Spack:

$ spack install git@2.17.0%gcc@9.3.0
$ spack install cmake@3.16.1%gcc@9.3.0

Installing without Slurm support

If you do not intend to use a job scheduler like Slurm to run GCHP, use the following commands to install MPI and NetCDF-Fortran. Otherwise, scroll down to see necessary modifications you must make to include Slurm support.

OpenMPI

$ spack install openmpi@4.0.4%gcc@9.3.0
$ spack install netcdf-fortran%gcc@9.3.0 ^netcdf-c^hdf5^openmpi@4.0.4

Intel MPI

  $ spack install intel-mpi%gcc@9.3.0
  $ spack install netcdf-fortran%gcc@9.3.0 ^intel-mpi



**MVAPICH2**
$ spack install mvapich2%gcc@9.3.0
$ spack install netcdf-fortran%gcc@9.3.0 ^netcdf-c^hdf5^mvapich2

Installing with Slurm support

OpenMPI

You need to tell Spack to build OpenMPI with Slurm support and to build NetCDF-Fortran with the correct OpenMPI version as a dependency:

$ spack install openmpi@4.0.4%gcc@9.3.0 +pmi schedulers=slurm
$ spack install netcdf-fortran%gcc@9.3.0  ^netcdf-c^hdf5^openmpi@4.0.4+pmi schedulers=slurm

You may run into issues building OpenMPI if your cluster has preexisting versions of PMIx that are newer than OpenMPI’s internal version. OpenMPI will search for and use the newest version of PMIx installed on your system, which will likely cause a crash during build because OpenMPI requires you to build with the same libevent library as was used to build PMIx. This information may not be readily available to you, in which case you can tweak the build arguments for OpenMPI to always use OpenMPI’s internal version of PMIx. Open $SPACK_ROOT/var/spack/repos/builtin/packages/openmpi/package.py and navigate to the configure_args() function. In the body of this function, place the following line:

config_args.append('--with-pmix=internal')

Intel MPI

No build-time tweaks need to be made to install Intel MPI with Slurm support.

$ spack install intel-mpi%gcc@9.3.0
$ spack install netcdf-fortran%gcc@9.3.0 ^intel-mpi

Scroll down to find environment variables you need to set when running GCHP with Intel MPI, including when using Slurm.

MVAPICH2

Like OpenMPI, you must specify that you want to build MVAPICH2 with Slurm support and build NetCDF-Fortran with the correct MVAPICH2 version.

$ spack install mvapich2%gcc@9.3.0 process_managers=slurm
$ spack install netcdf-fortran%gcc@9.3.0 ^netcdf-c^hdf5^mvapich2

Once you’ve installed all of your dependencies, you can follow the GCHP instructions for downloading, compiling, and setting up a run directory in the User Guide section of this Read The Docs site.

One-line install of GCHP dependencies with Spack

Rather than using Spack to install individual dependencies, you can use the spack install --only dependencies gchp command to install every dependency for GCHP in a single command. The --only dependencies option tells Spack to build GCHP’s dependencies without building GCHP itself.

Spack is smart about choosing compatible versions for all of GCHP’s different dependencies. You can further specify which package versions or MPI implementations (OpenMPI, Intel MPI, or MVAPICH2) you wish to use by appending options to spack install --only dependencies gchp, such as ^openmpi@4.0.4 or ^intel-mpi. If you wish to use Slurm with GCHP and want Spack to install a new version of OpenMPI or MVAPICH2, you need to specify +pmi schedulers=slurm (for OpenMPI) or process_managers=slurm (for MVAPICH2). A full install line for all of GCHP’s dependencies, including Slurm-enabled OpenMPI, would look like spack install --only dependencies gchp ^openmpi +pmi schedulers=slurm.

Once you’ve installed all of your dependencies, you can follow the GCHP instructions for downloading, compiling, and setting up a run directory in the User Guide section of this Read The Docs site.

One-line install of GCHP and its dependencies with Spack

You can use Spack to install all of GCHP’s dependencies and GCHP itself in a single line: spack install gchp. Just as when installing only GCHP’s dependencies, you can modify this command with further options for GCHP’s dependencies (and should do so if you intend to use a job scheduler like Slurm).

Spack is smart about choosing compatible versions for all of GCHP’s different dependencies. You can further specify which package versions or MPI implementations (OpenMPI, Intel MPI, or MVAPICH2) you wish to use by appending options to spack install gchp, such as ^openmpi@4.0.4 or ^intel-mpi. If you wish to use Slurm with GCHP and want Spack to install a new version of OpenMPI or MVAPICH2, you need to specify +pmi schedulers=slurm (for OpenMPI) or process_managers=slurm (for MVAPICH2). A full install line for GCHP and all of its dependencies, including Slurm-enabled OpenMPI, would look like spack install gchp ^openmpi +pmi schedulers=slurm.

In addition to specifying options for GCHP’s dependencies, GCHP also has its own options you can specify in your spack install gchp command. The available options (which you can view for yourself using spack info gchp) include:

  • apm - APM Microphysics (Experimental) (Default: off)

  • build_type - Choose CMake build type (Default: RelWithDebInfo)

  • ipo - CMake interprocedural optimization (Default: off)

  • luo - Luo et al 2019 wet deposition scheme (Default: off)

  • omp - OpenMP parallelization (Default: off)

  • real8 - REAL*8 precision (Default: on)

  • rrtmg - RRTMG radiative transfer model (Default: off)

  • tomas - TOMAS Microphysics (Experimental) (Default: off)

To specify any of these options, place it directly after gchp with a + to enable it or a ~ to disable it (e.g. spack install gchp ~real8 +rrtmg).

When you run spack install gchp, Spack will build all of GCHP’s dependencies and then download and build GCHP itself. The overall process may take a very long time if you are installing fresh copies of many dependencies, particularly MPI or ESMF. Once the install is completed, Spack will leave you with a built gchp executable and a copy of GCHP’s source code at spack location -i gchp.

You can use Spack’s included copy of the source code to create a run directory. Navigate to the directory returned by spack location -i gchp, and then cd to source_code/run. Run ./createRunDir.sh to generate a GCHP run directory. Once you’ve created a run directory, follow the instructions on Running GCHP in the User Guide.

You can find information on loading your environment for running GCHP below.

Loading Spack libraries for use with GCHP and/or ESMF

After installing the necessary libraries, place the following in a script that you will run before building/running GCHP (such as $HOME/.bashrc or a separate environment script) to initialize Spack and load requisite packages for building ESMF and/or building/running GCHP.

OpenMPI

export SPACK_ROOT=$HOME/spack #your path to Spack
source $SPACK_ROOT/share/spack/setup-env.sh
if [[ $- = *i* ]] ; then
 echo "Loading Spackages, please wait ..."
fi
#==============================================================================
%%%%% Load Spackages %%%%%
#==============================================================================
# List each Spack package that you want to load
# NOTE: Only needed if you did not install GCHP directly through Spack
pkgs=(gcc@9.3.0            \
 git@2.17.0           \
 netcdf-fortran@4.5.2 \
 cmake@3.16.1         \
 openmpi@4.0.4        \
 esmf@8.0.1           )

# Load each Spack package
for f in ${pkgs[@]}; do
  echo "Loading $f"
  spack load $f
done

# If you installed GCHP directly through Spack,comment out the above code after "Load Spackages"
# and uncomment the following line
#spack load gchp

export MPI_ROOT=$(spack location -i openmpi)

# These lines only needed for building ESMF outside of Spack
export ESMF_COMPILER=gfortran #intel for intel compilers
export ESMF_COMM=openmpi

IntelMPI

export SPACK_ROOT=$HOME/spack #your path to Spack
source $SPACK_ROOT/share/spack/setup-env.sh
if [[ $- = *i* ]] ; then
 echo "Loading Spackages, please wait ..."
fi
#==============================================================================
%%%%% Load Spackages %%%%%
#==============================================================================
# List each Spack package that you want to load
# NOTE: Only needed if you did not install GCHP directly through Spack
pkgs=(gcc@9.3.0            \
 git@2.17.0           \
 netcdf-fortran@4.5.2 \
 cmake@3.16.1         \
 intel-mpi        )

# Load each Spack package
for f in ${pkgs[@]}; do
  echo "Loading $f"
  spack load $f
done

# If you installed GCHP directly through Spack,comment out the above code after "Load Spackages"
# and uncomment the following line
#spack load gchp

# Environment variables only needed for Intel MPI
export I_MPI_CC=gcc #icc for intel compilers
export I_MPI_CXX=g++ #icpc for intel compilers
export I_MPI_FC=gfortran #ifort for intel compilers
export I_MPI_F77=gfortran #ifort for intel compilers
export I_MPI_F90=gfortran #ifort for intel compilers
export MPI_ROOT=$(spack location -i intel-mpi)

export I_MPI_PMI_LIBRARY=/path/to/slurm/libpmi2.so #when using srun through Slurm
#unset I_MPI_PMI_LIBRARY #when using mpirun

# These lines only needed for building ESMF outside of Spack
export ESMF_COMPILER=gfortran #intel for intel compilers
export ESMF_COMM=intelmpi

MVAPICH2

export SPACK_ROOT=$HOME/spack #your path to Spack
source $SPACK_ROOT/share/spack/setup-env.sh
if [[ $- = *i* ]] ; then
 echo "Loading Spackages, please wait ..."
fi
#==============================================================================
%%%%% Load Spackages %%%%%
#==============================================================================
# List each Spack package that you want to load
# NOTE: Only needed if you did not install GCHP directly through Spack
pkgs=(gcc@9.3.0            \
 git@2.17.0           \
 netcdf-fortran@4.5.2 \
 cmake@3.16.1         \
 mvapich2        )

# Load each Spack package
for f in ${pkgs[@]}; do
  echo "Loading $f"
  spack load $f
done

# If you installed GCHP directly through Spack,comment out the above code after "Load Spackages"
# and uncomment the following line
#spack load gchp

export MPI_ROOT=$(spack location -i mvapich2)

# These lines only needed for building ESMF outside of Spack
export ESMF_COMPILER=gfortran #intel for intel compilers
export ESMF_COMM=mvapich2

You can also add other packages you’ve installed with Spack like emacs to the pkgs lists above.

ESMF and your environment file

The following gives some information on building ESMF separately from Spack and provides more environment file examples.

You must load your environment file prior to building and running GCHP.

$ source /home/envs/gchpctm_ifort18.0.5_openmpi4.0.1.env

If you don’t already have ESMF 8.0.0+, you will need to download and build it. You only need to build ESMF once per compiler and MPI configuration (this includes for ALL users on a cluster!). It is therefore worth downloading and building somewhere stable and permanent, as almost no users of GCHP would be expected to need to modify or rebuild ESMF except when adding a new compiler or MPI. ESMF is available through Spack, and will already be installed if you chose the spack install gchp --only dependencies or spack install gchp routes above. Instructions for manually downloading and building ESMF are available at the GCHP wiki.

It is good practice to store your environment setup in a text file for reuse. Below are a couple examples that load libraries and export the necessary environment variables for building and running GCHP. Note that library version information is included in the filename for easy reference. Be sure to use the same libraries that were used to create the ESMF build install directory stored in environment variable ESMF_ROOT.

Environment file example 1

# file: gchpctm_ifort18.0.5_openmpi4.0.1.env

# Start fresh
module --force purge

# Load modules (some include loading other libraries such as netcdf-C and hdf5)
module load intel/18.0.5
module load openmpi/4.0.1
module load netcdf-fortran/4.5.2
module load cmake/3.16.1

# Set environment variables
export CC=gcc
export CXX=g++
export FC=ifort

# Set location of ESMF
export ESMF_ROOT=/n/lab_shared/libraries/ESMF/ESMF_8_0_1/INSTALL_ifort18_openmpi4

Environment file example 2 (Spack libraries built with a pre-installed compiler)

# file: gchpctm_gcc7.4_openmpi.rc

# Start fresh
module --force purge

# Load modules
module load gcc-7.4.0
spack load cmake
spack load openmpi%gcc@7.4.0
spack load hdf5%gcc@7.4.0
spack load netcdf%gcc@7.4.0
spack load netcdf-fortran%gcc@7.4.0

# Set environment variables
export CC=gcc
export CXX=g++
export FC=gfortran

# Set location of ESMF
export ESMF_ROOT=/n/home/ESMFv8/DEFAULTINSTALLDIR