Building Dependencies with Spack

The Spack Package Manager may be used to download and build GCHP and all of its required external software depenencies, including CMake, MPI, ESMF, and NetCDF libraries. The only essential prerequisite for using Spack is a local C/C++/Fortran Compiler such as GNU Compiler Collection. You can use this local compiler to later install a different compiler version through Spack. GCHP requires GNU Compilers version ≥ 8.3 or Intel compilers version ≥ 18.0.5. You must install a newer compiler through Spack if your pre-installed compiler does not meet these requirements.

There are 3 stages to setting up GCHP and/or its dependencies with Spack: setting up Spack and configuring existing libraries, installing GCHP and/or its dependencies, and setting up an environment file for loading installed Spack packages.

All Spack users should first read the section on setting up Spack and configuring Spack to recognize existing dependencies on your system that you would like Spack to use for building GCHP or other dependencies (such as ESMF or Slurm):

You can optionally read about the most important Spack commands and tips on how they work:

Read one of the following sections depending on which portions of the GCHP software stack you want to install using Spack:

Read how to load Spack libraries in an environment file for running or building GCHP or ESMF:

Setting up Spack, installing new compilers, and specifying preinstalled dependencies

To begin using Spack, clone the latest version by typing git clone https://github.com/spack/spack.git. Execute the following commands to initialize Spack’s environment (replacing /path/to/spack with the path of your spack directory). Add these commands to an environment initialization script for easy reuse.

$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh

If you do not already have a copy of your preferred text editor, you can use Spack to install and load one before proceeding (e.g. spack install emacs; spack load emacs).

Installing compilers

Ensure Spack recognizes any existing compiler on your system by typing spack compilers. You can use this compiler to build a new one.

Setting up and using GNU compilers

GNU compilers are free, open source, and easily installable through Spack. Execute the following at the command prompt to install version 9.3.0 of the GNU Compiler Collection:

$ spack install gcc@9.3.0

The installation of gcc may take a long time. Once it is complete, you’ll need to add it to Spack’s list of compilers using the following command:

$ spack compiler find $(spack location -i gcc@9.3.0)

Make sure that spack recognizes your new compiler by typing spack compilers, which should display a list of compilers including GCC 9.3.0.

Setting up and using Intel compilers

If you would like to use Intel compilers, whether pre-installed on your system or which you would like to install through Spack using an existing license, follow the instructions for pre-installed Intel compilers or for Intel compilers you want to install through Spack on the official Spack documentation site. In the instructions below, simply replace %gcc@9.3.0 with %intel to use your Intel compilers to build libraries.

Specifying preinstalled dependencies

Just as Spack can use an existing compiler on your system, it can also use your existing installations of dependencies for GCHP rather than building new copies. This is essential for interfacing your cluster’s Slurm with a Spack-built GCHP and its dependencies. For any preinstalled dependency you want Spack to always use, you must specify its path on your system and that you want Spack to always use this preinstalled package rather than building a new version. The code below shows how to do this by editing $HOME/.spack/packages.yaml (you may need to create this file):

packages:
 slurm:
  paths:
   slurm: /path/to/slurm
  buildable: False

Finding and specifying your system’s Slurm installation

If you plan on submitting jobs through the Slurm job scheduler, you’ll need to specify the location of Slurm on your system for Spack. Spack expects a specific directory format for your external Slurm installation path: it must contain both an include/ directory and a lib64/ directory. Depending on your cluster’s Slurm configuration, these directories may or may not already be in the same parent directory. Additionally, finding these individual directories can prove challenging. Each of the following commands can help lead you to the correct include/ and lib64/ directories on your cluster:

$ whereis slurm
$ whereis libpmi
$ whereis libpmi2
$ whereis srun
$ whereis sinfo

You may or may not receive any output from each of these commands, but hopefully at least one of these commands reveals a high level Slurm directory (e.g. opt/slurm), an include/ directory, and/or a lib64/ directory. You may encounter multiple directories with the name include or lib64; the correct include/ directory should contain .h files like pmi.h and slurm.h, while the lib64/ directory should contain libpmi.so, libpmi2.so, and/or libpmix.so.

If you have a high level Slurm directory that contains correct include/ and lib64/ directories, then you can use the path to that high level directory in your $HOME/.spack/packages.yaml file.

If your revealed include/ and lib64/ directories are not located in the same parent directory, you’ll need to create a new directory (called slurm_hub in this example, though you can name it anything and put it anywhere) with symlinks to your include/ and lib64/ directories:

$ mkdir slurm_hub
$ cd slurm_hub
$ ln -s /path/to/include include
$ ln -s /path/to/lib64 lib64

Put the path to slurm_hub in the correct location of your $HOME/.spack/packages.yaml file, and Spack will consider this directory to be your Slurm directory.

Spack syntax and handy commands

This section describes the most important parts of Spack’s syntax as well as various useful Spack commands.

When describing a Spack package, the @version syntax specifies the package version (e.g. gchp@13.0.2). If you do not specify a specific version of a package during installation, Spack will either install the latest available version of that package or Spack will choose a version that it knows to be compatible with other dependencies in your software stack.

The package1 ^package2 syntax tells Spack that package2 is a dependency of package1. Packages will often have locked dependency requirements such that you cannot add totally new packages as dependencies or avoid using specific packages, but you can still use this syntax to specify information on how to build certain dependencies (e.g. spack install gchp ^netcdf-fortran@4.5.2%gcc@9.3.0). Other dependency requirements can be fulfilled by multiple packages (e.g. you can specify an MPI implementation to use through spack install gchp ^openmpi or spack install gchp ^intel-mpi). If you want to specify build options for a package, make sure that option immediately follows the package’s name in a Spack command (e.g. spack install gchp +ofi ^openmpi +pmi is correct because +ofi is an option for gchp and +pmi is an option for openmpi, but spack install gchp ^openmpi +pmi +ofi will fail because +ofi is not an option for openmpi).

The %compiler syntax specifies which compiler to use to build a package. You can specify the compiler version as well (e.g. gchp%gcc@9.3.0). Spack should build all child dependencies of a package with the same compiler you specify for the parent package, but compiler versions may or may not be consistent if you first specify a compiler version further down the dependency tree (e.g. spack install gchp%gcc@9.3.0 ^openmpi will build all dependencies of GCHP with gcc@9.3.0 including OpenMPI, but spack install gchp ^openmpi%gcc@9.3.0 only requires Spack to build OpenMPI and its dependencies with gcc@9.3.0 so other dependencies of GCHP may be built with another compiler version).

Every build of a package through Spack receives its own unique hash identifier. This hash can be useful when you install multiple copies of the same package that are not readily distinguishable (e.g. they only differ by one dependency version). You can use the -l option in spack find to see the beginning of the hash ID for each package. You only need this short code (rather than the full 32 character hash) to identify a package for Spack. Use /hash_id to specify a package build based on its hash (you don’t need to include the package’s name, e.g. a GCHP installation with the hash 2qmcc6kq3hwm3vasey63y6h4l77pzw2o can be loaded using spack load /2qmcc6).

All of the above syntax examples can be added to any of the commands below for any package_name.

spack info package_name will return information on package package_name, including available versions and build variant options, if this package is available through Spack. This info is pulled from data in your clone of the Spack Git repository. You can get the latest information (as well as any updates to packages) by updating your Spack repo using git pull.

spack spec package_name will return a list of all dependencies (including package versions, compilers, and build variants) that would be installed if you were to use spack install package_name.

spack install package_name is the core Spack installation command. Many examples of install commands are found in other sections on this page. One important option for spack install is --only dependencies, which installs all dependencies for a package without installing the package itself. Note that if you use spack install --only dependencies package_name you will not be able to later load all dependencies for package_name using spack load package_name; you will have to load dependencies individually.

spack uninstall package_name lets you uninstall specific packages. You’ll be prompted with additional options to append after uninstall if the package you’re attempting to remove has dependents remaining on your system. spack uninstall -a pattern can be used to uninstall all packages matching a certain spec pattern (e.g. spack uninstall %gcc@9.3.0 will uninstall all packages that were compiled with gcc@9.3.0).

spack compiler remove compiler_name will remove a compiler from Spack’s list of available compilers.

spack find package_name will show you all installed versions of package_name. spack find will show you all installed packages, and spack find --loaded will show you all loaded packages.

spack location -i package_name will return the installation path of package_name.

spack load package_name will load a package and all of its runtime dependencies.

spack unload package_name will unload a specific package. Note that some environment variable changes will not be undone, and dependencies will not be unloaded. spack unload (without specifying a package) will unload all loaded packages, similar to module purge.

Installing individual dependencies with Spack

This section describes how to use Spack to build GCHP’s individual dependencies. While these dependencies can be used to then install GCHP directly using Spack, this section is mainly intended for those looking to manually download and compile GCHP as described in the User Guide.

Installing basic dependencies

You should now install Git and CMake using Spack:

$ spack install git@2.17.0%gcc@9.3.0
$ spack install cmake@3.16.1%gcc@9.3.0

Installing without Slurm support

If you do not intend to use a job scheduler like Slurm to run GCHP, use the following commands to install MPI and NetCDF-Fortran. Otherwise, scroll down to see necessary modifications you must make to include Slurm support.

OpenMPI

$ spack install openmpi@4.0.4%gcc@9.3.0
$ spack install netcdf-fortran%gcc@9.3.0 ^netcdf-c^hdf5^openmpi@4.0.4

Intel MPI

$ spack install intel-mpi%gcc@9.3.0
$ spack install netcdf-fortran%gcc@9.3.0 ^intel-mpi

MVAPICH2

$ spack install mvapich2%gcc@9.3.0
$ spack install netcdf-fortran%gcc@9.3.0 ^netcdf-c^hdf5^mvapich2

Installing with Slurm support

OpenMPI

You need to tell Spack to build OpenMPI with Slurm support and to build NetCDF-Fortran with the correct OpenMPI version as a dependency:

$ spack install openmpi@4.0.4%gcc@9.3.0 +pmi schedulers=slurm
$ spack install netcdf-fortran%gcc@9.3.0  ^netcdf-c^hdf5^openmpi@4.0.4+pmi schedulers=slurm

You may run into issues building OpenMPI if your cluster has preexisting versions of PMIx that are newer than OpenMPI’s internal version. OpenMPI will search for and use the newest version of PMIx installed on your system, which will likely cause a crash during build because OpenMPI requires you to build with the same libevent library as was used to build PMIx. This information may not be readily available to you, in which case you can tweak the build arguments for OpenMPI to always use OpenMPI’s internal version of PMIx. Open $SPACK_ROOT/var/spack/repos/builtin/packages/openmpi/package.py and navigate to the configure_args() function. In the body of this function, place the following line:

config_args.append('--with-pmix=internal')

Intel MPI

No build-time tweaks need to be made to install Intel MPI with Slurm support.

$ spack install intel-mpi%gcc@9.3.0
$ spack install netcdf-fortran%gcc@9.3.0 ^intel-mpi

Scroll down to find environment variables you need to set when running GCHP with Intel MPI, including when using Slurm.

MVAPICH2

Like OpenMPI, you must specify that you want to build MVAPICH2 with Slurm support and build NetCDF-Fortran with the correct MVAPICH2 version.

$ spack install mvapich2%gcc@9.3.0 process_managers=slurm
$ spack install netcdf-fortran%gcc@9.3.0 ^netcdf-c^hdf5^mvapich2

Once you’ve installed all of your dependencies, you can follow the GCHP instructions for downloading, compiling, and setting up a run directory in the User Guide section of this Read The Docs site.

One-line install of GCHP dependencies with Spack

Rather than using Spack to install individual dependencies, you can use the spack install --only dependencies gchp command to install every dependency for GCHP in a single command. The --only dependencies option tells Spack to build GCHP’s dependencies without building GCHP itself.

Spack is smart about choosing compatible versions for all of GCHP’s different dependencies. You can further specify which package versions or MPI implementations (OpenMPI, Intel MPI, or MVAPICH2) you wish to use by appending options to spack install --only dependencies gchp, such as ^openmpi@4.0.4 or ^intel-mpi. If you wish to use Slurm with GCHP and want Spack to install a new version of OpenMPI or MVAPICH2, you need to specify +pmi schedulers=slurm (for OpenMPI) or process_managers=slurm (for MVAPICH2). A full install line for all of GCHP’s dependencies, including Slurm-enabled OpenMPI, would look like spack install --only dependencies gchp ^openmpi +pmi schedulers=slurm.

Once you’ve installed all of your dependencies, you can follow the GCHP instructions for downloading, compiling, and setting up a run directory in the User Guide section of this Read The Docs site.

One-line install of GCHP and its dependencies with Spack

You can use Spack to install all of GCHP’s dependencies and GCHP itself in a single line: spack install gchp. Just as when installing only GCHP’s dependencies, you can modify this command with further options for GCHP’s dependencies (and should do so if you intend to use a job scheduler like Slurm).

Spack is smart about choosing compatible versions for all of GCHP’s different dependencies. You can further specify which package versions or MPI implementations (OpenMPI, Intel MPI, or MVAPICH2) you wish to use by appending options to spack install gchp, such as ^openmpi@4.0.4 or ^intel-mpi. If you wish to use Slurm with GCHP and want Spack to install a new version of OpenMPI or MVAPICH2, you need to specify +pmi schedulers=slurm (for OpenMPI) or process_managers=slurm (for MVAPICH2). A full install line for GCHP and all of its dependencies, including Slurm-enabled OpenMPI, would look like spack install gchp ^openmpi +pmi schedulers=slurm.

In addition to specifying options for GCHP’s dependencies, GCHP also has its own options you can specify in your spack install gchp command. The available options (which you can view for yourself using spack info gchp) include:

  • apm - APM Microphysics (Experimental) (Default: off)

  • build_type - Choose CMake build type (Default: RelWithDebInfo)

  • ipo - CMake interprocedural optimization (Default: off)

  • luo - Luo et al 2019 wet deposition scheme (Default: off)

  • omp - OpenMP parallelization (Default: off)

  • real8 - REAL*8 precision (Default: on)

  • rrtmg - RRTMG radiative transfer model (Default: off)

  • tomas - TOMAS Microphysics (Experimental) (Default: off)

To specify any of these options, place it directly after gchp with a + to enable it or a ~ to disable it (e.g. spack install gchp ~real8 +rrtmg).

When you run spack install gchp, Spack will build all of GCHP’s dependencies and then download and build GCHP itself. The overall process may take a very long time if you are installing fresh copies of many dependencies, particularly MPI or ESMF. Once the install is completed, Spack will leave you with a built gchp executable and a copy of GCHP’s source code at spack location -i gchp.

You can use Spack’s included copy of the source code to create a run directory. Navigate to the directory returned by spack location -i gchp, and then cd to source_code/run. Run ./createRunDir.sh to generate a GCHP run directory. Once you’ve created a run directory, follow the instructions on Running GCHP in the User Guide.

You can find information on loading your environment for running GCHP below.

Loading Spack libraries for use with GCHP and/or ESMF

After installing the necessary libraries, place the following in a script that you will run before building/running GCHP (such as $HOME/.bashrc or a separate environment script) to initialize Spack and load requisite packages for building ESMF and/or building/running GCHP.

OpenMPI

export SPACK_ROOT=$HOME/spack #your path to Spack
source $SPACK_ROOT/share/spack/setup-env.sh
if [[ $- = *i* ]] ; then
 echo "Loading Spackages, please wait ..."
fi
#==============================================================================
%%%%% Load Spackages %%%%%
#==============================================================================
# List each Spack package that you want to load
# NOTE: Only needed if you did not install GCHP directly through Spack
pkgs=(gcc@9.3.0       \
 git@2.17.0           \
 netcdf-fortran@4.5.2 \
 cmake@3.16.1         \
 openmpi@4.0.4        \
 esmf@8.0.1           )

# Load each Spack package
for f in ${pkgs[@]}; do
  echo "Loading $f"
  spack load $f
done

# If you installed GCHP directly through Spack,comment out the above code after "Load Spackages"
# and uncomment the following line
#spack load gchp

export MPI_ROOT=$(spack location -i openmpi)

# These lines only needed for building ESMF outside of Spack
export ESMF_COMPILER=gfortran #intel for intel compilers
export ESMF_COMM=openmpi

Intel MPI

export SPACK_ROOT=$HOME/spack #your path to Spack
source $SPACK_ROOT/share/spack/setup-env.sh
if [[ $- = *i* ]] ; then
 echo "Loading Spackages, please wait ..."
fi
#==============================================================================
%%%%% Load Spackages %%%%%
#==============================================================================
# List each Spack package that you want to load
# NOTE: Only needed if you did not install GCHP directly through Spack
pkgs=(gcc@9.3.0       \
 git@2.17.0           \
 netcdf-fortran@4.5.2 \
 cmake@3.16.1         \
 intel-mpi            \
 esmf                 )

# Load each Spack package
for f in ${pkgs[@]}; do
  echo "Loading $f"
  spack load $f
done

# If you installed GCHP directly through Spack,comment out the above code after "Load Spackages"
# and uncomment the following line
#spack load gchp

# Environment variables only needed for Intel MPI
export I_MPI_CC=gcc #icc for intel compilers
export I_MPI_CXX=g++ #icpc for intel compilers
export I_MPI_FC=gfortran #ifort for intel compilers
export I_MPI_F77=gfortran #ifort for intel compilers
export I_MPI_F90=gfortran #ifort for intel compilers
export MPI_ROOT=$(spack location -i intel-mpi)

export I_MPI_PMI_LIBRARY=/path/to/slurm/libpmi2.so #when using srun through Slurm
#unset I_MPI_PMI_LIBRARY #when using mpirun

# These lines only needed for building ESMF outside of Spack
export ESMF_COMPILER=gfortran #intel for intel compilers
export ESMF_COMM=intelmpi

MVAPICH2

export SPACK_ROOT=$HOME/spack #your path to Spack
source $SPACK_ROOT/share/spack/setup-env.sh
if [[ $- = *i* ]] ; then
 echo "Loading Spackages, please wait ..."
fi
#==============================================================================
%%%%% Load Spackages %%%%%
#==============================================================================
# List each Spack package that you want to load
# NOTE: Only needed if you did not install GCHP directly through Spack
pkgs=(gcc@9.3.0       \
 git@2.17.0           \
 netcdf-fortran@4.5.2 \
 cmake@3.16.1         \
 mvapich2             \
 esmf                 )

# Load each Spack package
for f in ${pkgs[@]}; do
  echo "Loading $f"
  spack load $f
done

# If you installed GCHP directly through Spack,comment out the above code after "Load Spackages"
# and uncomment the following line
#spack load gchp

export MPI_ROOT=$(spack location -i mvapich2)

# These lines only needed for building ESMF outside of Spack
export ESMF_COMPILER=gfortran #intel for intel compilers
export ESMF_COMM=mvapich2

You can also add other packages you’ve installed with Spack like emacs to the pkgs lists above.

ESMF and your environment file

The following gives some information on building ESMF separately from Spack and provides more environment file examples.

You must load your environment file prior to building and running GCHP.

$ source /home/envs/gchpctm_ifort18.0.5_openmpi4.0.1.env

If you don’t already have ESMF 8.0.0+, you will need to download and build it. You only need to build ESMF once per compiler and MPI configuration (this includes for ALL users on a cluster!). It is therefore worth downloading and building somewhere stable and permanent, as almost no users of GCHP would be expected to need to modify or rebuild ESMF except when adding a new compiler or MPI. ESMF is available through Spack, and will already be installed if you chose the spack install gchp --only dependencies or spack install gchp routes above. Instructions for manually downloading and building ESMF are available at the GCHP wiki.

It is good practice to store your environment setup in a text file for reuse. Below are a couple examples that load libraries and export the necessary environment variables for building and running GCHP. Note that library version information is included in the filename for easy reference. Be sure to use the same libraries that were used to create the ESMF build install directory stored in environment variable ESMF_ROOT.

Environment file example 1

# file: gchpctm_ifort18.0.5_openmpi4.0.1.env

# Start fresh
module --force purge

# Load modules (some include loading other libraries such as netcdf-C and hdf5)
module load intel/18.0.5
module load openmpi/4.0.1
module load netcdf-fortran/4.5.2
module load cmake/3.16.1

# Set environment variables
export CC=gcc
export CXX=g++
export FC=ifort

# Set location of ESMF
export ESMF_ROOT=/n/lab_shared/libraries/ESMF/ESMF_8_0_1/INSTALL_ifort18_openmpi4

Environment file example 2 (Spack libraries built with a pre-installed compiler)

# file: gchpctm_gcc7.4_openmpi.rc

# Start fresh
module --force purge

# Load modules
module load gcc-7.4.0
spack load cmake
spack load openmpi%gcc@7.4.0
spack load hdf5%gcc@7.4.0
spack load netcdf%gcc@7.4.0
spack load netcdf-fortran%gcc@7.4.0

# Set environment variables
export CC=gcc
export CXX=g++
export FC=gfortran

# Set location of ESMF
export ESMF_ROOT=/n/home/ESMFv8/DEFAULTINSTALLDIR