Compile

Note

Compiling GCHP and creating a run directory are independent steps, and their order doesn’t matter. A small exception is the RUNDIR build option, which controls the behaviour of make install which copies the GCHP executable to the run directory; however, this setting can be reconfigured at any time (e.g., after compiling and creating a run directory).

Here in the User Guide we describe compiling GCHP before we describe creating a run directory. This is so that conceptually the instructions have a linear flow. The Quickstart Guide, on the other hand, shows how to make a run directory prior to compiling.

Another resource for GCHP build instructions is our YouTube tutorial. It is for version 13 but the build information is still applicable.

There are three steps to building GCHP. The first is configuring your build, which is done with cmake; the second step is compiling, which is done with make. The third step is install, which is also done with make. In the first step (build configuration), cmake finds GCHP’s software dependencies on your system, and you can set build options like enabling/disabling components (such as RRTMG), setting paths to run directories, picking between debug or speed-optimizing compiler flags, etc. The second step (running make) compiles GCHP according your build configuration. The third step copies GCHP executable to an appropriate location, such as one or more run directories if you specify them.

Important

These instructions assume you have loaded a computing environment that satisfies GCHP’s software requirements. You can find instructions for building GCHP’s dependencies yourself in the Spack instructions.

Create a build directory

A build directory is the working directory for a “build”. Conceptually, a “build” is a case/instance of you compiling GCHP. A build directory stores configuration files and intermediate files related to the build. These files are generated and used by CMake, Make, and compilers. You can think of a build directory like the blueprints for a construction project.

Create a new directory and initialize it as a build directory by running CMake. When you initialize a build directory, the path to the source code is a required argument:

$ cd ~/Code.GCHP     # Navigate to the GCHP source code folder
$ mkdir build        # Create a new directory
$ cd build           # Navigate to the new directory
$ cmake ~/Code.GCHP  # Initialize the current dir as a build dir

-- The Fortran compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The C compiler identification is GNU 12.2.0
-- Check for working Fortran compiler: /usr/bin/f95
-- Check for working Fortran compiler: /usr/bin/f95  -- works
...
-- Configuring done
-- Generating done
-- Build files have been written to: /src/build

If your cmake output is similar to the snippet above, and it says configuring & generating done, then your configuration was successful and you can move on to compiling or modifying build settings. If you got an error, don’t worry, that just means the automatic configuration failed. To fix the error you might need to tweak settings with more cmake commands, or you might need to modify your environment and run cmake again to retry the automatic configuration.

If you want to restart configuring your build from scratch, delete your build directory. Note that the name and location of your build directory doesn’t matter, but a good name is build/, and a good place for it is the top-level of your source code.

Resolving initialization errors

If your last step was successful, skip to this section.

Even if you got a cmake error, your build directory was initialized. This means from now on, you can check if the configuration is fixed by running

$ cd ~/Code.GCHP/build  # Navigate to the build/ folder in the GCHP code dir
$ cmake .               # "." because the current dir is the build dir

To resolve your errors, you might need to modify your environment (e.g., load different software modules), or give CMake a hint about where some software is installed. Once you identify the problem and make the appropriate update, run cmake . to see if the error is fixed.

To start troubleshooting, read the cmake output in full. It is human-readable, and includes important information about how the build was set up on your system, and specifically what error is preventing a successful configuration (e.g., a dependency that wasn’t found, or a compiler that is broken). To begin troubleshooting you should:

Check that the compilers are what you expect (e.g., GNU 12.2, Intel 19.1, etc.)
Check that dependencies like MPI, HDF5, NetCDF, and ESMF were found
Check for obvious errors/incompatibilities in the paths to “Found” dependencies

Note

F2PY and ImageMagick are not required. You can safely ignore warnings about them not being found.

Most errors are caused by one or more of the following issues:

The wrong compilers were chosen. Fix this by explicitly setting the compilers.
The compiler’s version is too old. Fix this by using newer compilers.
A software dependency is missing. Fix this by loading the appropriate software. Some hints:
- If HDF5 is missing, does h5cc -show or h5pcc -show work?
- If NetCDF is missing, do nc-config --all and nf-config --all work?
- If MPI is missing, does mpiexec --help work?
A software dependency is loaded but it wasn’t found automatically. Fix this by pointing CMake to the missing software/files with
```
$ cmake . -DCMAKE_PREFIX_PATH=/path/to/missing/files
```
- If ESMF is missing, point CMake to your ESMF install with -DCMAKE_PREFIX_PATH
Software modules that are not compatible. Fix this by loading compatible modules/dependencies/compilers. Some hints:
- This often shows as an error message saying a compiler is “broken” or “doesn’t work”
- E.g. incompatibility #1: you’re using GNU compilers but HDF5 is built for Intel compilers
- E.g. incompatibility #2: ESMF was compiled for a different compiler, MPI, or HDF5

If you are stumped, don’t hesitate to open an issue on GitHub. Your system administrators might also be able to help. Be sure to include CMakeCache.txt from your build directory, as it contains useful information for troubleshooting.

Note

If you get a CMake error saying “Could not find XXXX” (where XXXX is a dependency like ESMF, NetCDF, HDF5, etc.), the problem is that CMake can’t automatically find where that library is installed. You can add custom paths to CMake’s default search list by setting the CMAKE_PREFIX_PATH variable.

For example, if you got an error saying “Could not find ESMF”, and ESMF is installed to /software/ESMF, you would do

$ cd ~/Code.GCHP/build   # Skip if you are already in the build/ folder

$ cmake . -DCMAKE_PREFIX_PATH=/software/ESMF
...
-- Found ESMF: /software/ESMF/include (found version "8.6.1")
...
-- Configuring done
-- Generating done
-- Build files have been written to: /src/build

See the next section for details on setting variables like CMAKE_PREFIX_PATH.

Note

You can explicitly specify compilers by setting the CC, CXX, and FC environment variables. If the auto-selected compilers are the wrong ones, create a brand new build directory, and set these variables before you initialize it. E.g.:

$ cd ~/Code.GCHP     # Navigate to top-level source code folder
$ rm -rf build       # build dir initialized with wrong compilers
$ mkdir build        # Make a new build directory
$ cd build           # ... and navigate to it
$ export CC=icc      # select "icc" as C compiler
$ export CXX=icpc    # select "icpc" as C++ compiler
$ export FC=ifort    # select "ifort" as Fortran compiler
$ cmake ~/Code.GCHP  # initialize new build dir
-- The Fortran compiler identification is Intel 19.1.0.20191121
-- The CXX compiler identification is Intel 19.1.0.20191121
-- The C compiler identification is Intel 19.1.0.20191121
...

Configure your build

Build settings are controlled by cmake commands like:

$ cmake . -D<NAME>="<VALUE>"

where <NAME> is the name of the setting, and <VALUE> is the value you are assigning it. These settings are persistent and saved in your build directory. You can set multiple variables in the same command, and you can run cmake as many times as needed to configure your desired settings.

Note

The . argument is important. It is the path to your build directory which is . here.

No build settings are required. You can find the complete list of GCHP’s build settings here. The most common setting is RUNDIR, which lets you specify one or more run directories to install GCHP to. Here, “install” refers to copying the compiled executable, and some supplemental files with build settings, to your run directory/directories.

Tip

You can update build settings after you compile GCHP. Simply rerun make and (optionally) make install, and the build system will automatically figure out what needs to be recompiled.

Since there are no required build settings, we will stick with the default settings.

You should notice that when you run cmake it ends with:

...
-- Configuring done
-- Generating done
-- Build files have been written to: /src/build

This tells you that the configuration was successful, and that you are ready to compile.

Compile GCHP

You compile GCHP with:

$ cd ~/Code.GCHP/build  # Skip if you are already in the build/ folder
$ make -j               # -j enables compiling in parallel

Tip

You can add VERBOSE=1 to see all the compiler commands.

Tip

If you run out of memory while compiling, restrict the number of processes that can run concurrently (e.g., use -j20 to restrict to 20 processes).

Compiling GCHP creates ./bin/gchp (the GCHP executable). You can copy this executable to your run directory manually, or if you set the RUNDIR build option, you can do

$ cd ~/Code.GCHP/build  # Skip if you are already in the build/ folder
$ make install          # Requires that RUNDIR build option is set

to copy the executable (and supplemental files) to your run directories.

Now you have compiled GCHP! You can move on to creating a run directory!

Recompiling

You need to recompile GCHP if you update a build setting or modify the source code. With CMake, you do not need to clean before recompiling. The build system automatically figures out which files need to be recompiled (it’s usually a small subset). This is known as incremental compiling.

To recompile GCHP, simply do

$ cd ~/Code.GCHP/build   # Skip if you are already in the build/ folder
$ make -j                # -j enables compiling in parallel

and then optionally, make install.

Tip

GNU compilers recompile GCHP faster than Intel compilers. This is because of how gfortran formats Fortran modules files (*.mod files). Therefore, if you want to be able to recompile quickly, consider using GNU compilers.

GCHP build options

These are persistent build setting that are set with cmake commands like

$ cmake . -D<NAME>="<VALUE>"

where <NAME> is the name of the build setting, and <VALUE> is the value you are assigning it. Below is the list of build settings for GCHP.

RUNDIR

Specfies paths to run directories where make install installs GCHP. Multiple run directories can be specified by a semicolon separated list. A warning is issued if one of these directories does not look like a run directory.

These paths can be relative paths or absolute paths. Relative paths are interpreted as relative to your build directory.

INSTALLCOPY: Similar to RUNDIR, except the directories do not need to be run directories.

CMAKE_BUILD_TYPE

Specifies the type of build. Accepted values are:

Release: Tells CMake to configure GCHP in Release mode. This means that all optimizations will be applied and all debugging options will be disabled. (Default option).

RelWithDebInfo: Tells CMake to configure GCHP in Release mode, but to generate debugging output during the build sequence.

Debug: Tells CMake to configure GCHP in Debug mode. This turns on several runtime error checks. Debug mode makes it easier to find errors but will adversely impact performance. Only use this option if you are actively debugging GCHP.

CMAKE_PREFIX_PATH: Extra directories that CMake will search when it’s looking for dependencies. Directories in CMAKE_PREFIX_PATH have the highest precedence when CMake is searching for dependencies. Multiple directories can be specified with a semicolon-separated list.

GEOSChem_Fortran_FLAGS_<COMPILER_ID>: Compiler options for GEOS-Chem for all build types. Valid values for <COMPILER_ID> are GNU and Intel.

GEOSChem_Fortran_FLAGS_<BUILD_TYPE>_<COMPILER_ID>: Additional compiler options for GEOS-Chem for build type <BUILD_TYPE>.

HEMCO_Fortran_FLAGS_<COMPILER_ID>: Same as GEOSChem_Fortran_FLAGS_<COMPILER_ID>, but for HEMCO.

HEMCO_Fortran_FLAGS_<BUILD_TYPE>_<COMPILER_ID>: Same as GEOSChem_Fortran_FLAGS_<BUILD_TYPE>_<COMPILER_ID>, but for HEMCO.

MECH

Specifies the chemical mechanism that you wish to use with GCHP:

fullchem: Activates the fullchem mechanism. The source code files that define this mechanism are stored in KPP/fullchem. (Default option)

carbon: Activates the carbon mechanism (CH4-CO-CO2-OCS). The source code files that define this mechanism are stored in KPP/carbon.

custom: Activates a custom mechanism defined by the user. The source code files that define this mechanism are stored in KPP/custom.

JACOBIAN

Builds the carbon simulation with CH₄ Jacobian tracers, for use with the Integrated Methane Inversion.

Note

JACOBIAN may only be used with the carbon mechanism. The number of Jacobian tracers is set when the mechanism is built. Use the KPP/carbon/util/expand_carbon_eqn.py script to expand carbon.eqn to the number of CH₄ Jacobian tracers that you need.

Accepted values are:

n: Builds the carbon simulation without Jacobian tracers. (Default option)

y: Builds the carbon simulation with Jacobian tracers.

OMP

Configures GCHP to use OpenMP parallelization within each MPI process (hybrid MPI+OpenMP).

Attention

Hybrid MPI+OpenMP has not been validated by the GCST and is not recommended for production runs. GCHP is normally run as pure MPI, one thread per process. Results for 1 versus N threads are bit-identical, but results from an OMP=y build differ from an OMP=n build at round-off level. See GCHP issue #571 for test results.

If you do enable OpenMP:

Set OMP_STACKSIZE (e.g. 500m) and OMP_NUM_THREADS, and pass both to every MPI process (e.g. mpirun -x OMP_NUM_THREADS -x OMP_STACKSIZE). Without OMP_STACKSIZE the run will segfault in Cloud-J, because OpenMP worker threads default to a 2 MB stack.
Build with MPI_LOAD_BALANCE=n. The load-balanced chemistry loop is not threaded, so with load balancing on, the extra threads stay idle during KPP integration. Running with OMP=y, MPI_LOAD_BALANCE=y and more than one thread per MPI process will produce incorrect outputs.
Expect no speedup over pure MPI at equal core count. Our tests shows that threaded KPP integration time is reduced by about 25%, but transport, convection and emissions slowed with fewer MPI processes, and pure MPI with load balancing on (default) was fastest.

Accepted values are:

n: Deactivates OpenMP parallelization. GCHP will use MPI (Message Passing Interface) for all core-to-core communication. (Default option)

y: Activates OpenMP parallelization. GCHP will use OpenMP to parallelize DO loops marked with !$OMP PARALLEL within each MPI process, and MPI between processes.

RRTMG

Configures GCHP to use the RRTMG radiative transfer model. Accepted values are:

n: Deactivates the RRTMG radiative transfer model. (Default option)

y: Activates the RRTMG radiative transfer model.

TOMAS

Configures GCHP to use the TOMAS aerosol microphysics package. Accepted values are:

n: Deactivate TOMAS microphysics. (Default option)

y: Activate TOMAS microphysics.

TOMAS_BINS

Specifies the number of size-resolved bins for TOMAS. Accepted values are:

15: Use 15 size-resolved bins with TOMAS simulations.

40: Use 40 size-resolved bins with TOMAS simulations.

FASTJX

Configures GEOS-Chem to use the legacy FAST-JX photolysis mechanism instead of Cloud-J.

Attention

This option is deprecated, as FAST-JX is no longer used for the Fullchem simulation.

Accepted values are:

n: Uses Cloud-J for photolysis. (Default option)

y: Uses legacy FAST-JX for photolysis.

KPPSA

Compiles the KPP-Standalone Box Model executable.

n: Will not install the KPP-Standalone Box model in the run directory. (Default option)

y: Will install the KPP-Standalone Box mode in the run directory.

MPI_LOAD_BALANCE

Applies MPI load balancing in chemistry. This is automatically turned off if KPPSA is set to y. Accepted values are: Applies dynamic load balancing of chemistry columns across the MPI processes on each node, using MPI shared memory (see geos-chem PR #3015). Keep this on unless you are running hybrid MPI+OpenMP (see OMP). This is automatically turned off if KPPSA is set to y. Accepted values are:

y: Activates MPI load balancing in chemistry. (Default option)

n: Deactivates MPI load balancing in chemistry.

LUO_WETDEP

Configures GEOS-Chem to use the Luo et al. [2020] wet deposition scheme.

Note

The Luo et al. [2020] wet deposition scheme will eventually become the default wet deposition scheme in GEOS-Chem. We have made it an option for the time being while further evaluation is being done.

Accepted values are:

n: Deactivates the Luo et al., 2020 wet deposition scheme. (Default option)

y: Activates the Luo et al., 2020 wet deposition scheme.

SANITIZE

Activates the AddressSanitizer/LeakSanitizer functionality in GNU Fortran to identify memory leaks. Accepted values are:

n: Deactivates AddressSanitizer/LeakSanitizer (Default option).

y: Activates AddressSanitizer/LeakSanitizer.

USE_REAL8

Sets GEOS-Chem’s flexible precision (fp) to 8-byte floating point.

Attention

GCHP is only validated with USE_REAL8=y, which is why it is the default. Change this setting only for development or testing purposes.

Accepted values are:

y: Flexible precision is 8-byte floating point. (Default option)

n: Flexible precision is 4-byte floating point.