MITgcm-Hg ECCO4 guide

by | Sep 6, 2024 | MITgcm

Model Introduction

This wiki outlines the procedures for running the MIT General Circulation Model (MITgcm) mercury (Hg) simulations on the LONI QB4 system. General information about MITgcm can be found in the MITgcm User’s Manual. A user guide for the QB4 system can be found here: User Guide.

Currently, there is one type of simulation available:

  • A 1° x 1° online simulation using ocean circulation data from ECCOv4 over a global domain with higher spatial resolution over the Arctic Ocean. The grid system is called cubed-sphere LLC90. Please refer to Forget et al. 2015.

Source Code Overview

The numerical model is embedded within an execution environment support wrapper. This wrapper is designed to provide a general framework for grid-point models. MITgcm is a specific numerical model using this framework. Under this structure, the model is divided into execution environment support code and conventional numerical model code. The support code is in the eesupp/ directory, while the grid-point model code is in the model/ directory. Code execution starts in the eesupp/ routines, not in the model routines, which is why the top-level MAIN.F is located in eesupp/src/. Generally, end-users do not need to worry about this level. The top-level routine for the numerical model is in model/src/THE_MODEL_MAIN.F.

Directory Structure

  • doc: Contains brief documentation notes.
  • eesupp: Contains the execution environment source code, subdivided into inc/ and src/ directories.
  • model: Main source code, subdivided into inc/ and src/ directories.
  • pkg: Contains source code for packages, each corresponding to a subdirectory (e.g., gmredi/ for Gent-McWilliams/Redi scheme).
  • tools: Contains various useful tools, such as genmake2 (for generating Makefiles) and the adjoint/ directory with the Tangent Linear and Adjoint Compiler (TAMC) Makefile. It also includes build_options/, which contains ‘optfiles’ for different compilers and machines.
  • utils: Contains various utilities. For example, knudsen2/ contains code and a Makefile for computing coefficients for a polynomial approximation of the Knudsen formula for ocean equations of state. The matlab/ subdirectory contains MATLAB scripts for reading model output, and the scripts/ directory has post-processing C-shell scripts for joining model outputs. The exch2/ subdirectory contains code necessary for the exch2 package.
  • jobs: Contains sample job scripts for running MITgcm.
  • lsopt: Line search code for optimization.
  • optim: Interface between MITgcm and the line search code.

Setting Up a Working Directory

Copy all the files from /work/yxzhang/Group/MITgcm, which contains a template for module simulation:

cp -r /work/yxzhang/Group/MITgcm/ ~/

Compiling the Source Code

Navigate to the MITgcm directory and load the compiling environment:

source env.mitgcm

Then navigate to the build/ directory and build your Makefile:

cd ~/MITgcm/verification/global_hg_llc90/build
premake

The command overrides model source code with files from ../code/. The premake alias, defined in env.mitgcm, replaces the long genmake command:

../../../tools/genmake2 -mods=../code -optfile=../../../tools/build_options/linux_ifort_impi

Once the Makefile is generated, create dependencies using:

make depend

This modifies the Makefile to include a list of dependent files, reducing re-compilation. Occasionally, warnings or errors may occur due to missing headers (e.g., netcdf.inc), which can usually be ignored.

Next, compile the code using:

make

To speed up, use multi-core compilation (e.g., make -j8). After successful compilation, move the generated file to your run directory:

cp mitgcmuv ../run/mitgcmuv

Running the Model

To run the model:

cd ~/MITgcm/verification/global_hg_llc90/run
sbatch < run.mitgcm

If successful, the last line of your STDOUT.0000 file will indicate ‘ENDED NORMALLY’.

Monitoring and Debugging

Check job status with:

squeue

To terminate a job:

skill JOBID

For debugging, check files like STDERR.0000, STDOUT.0000, or omp.error.<job_id>. You may also check the .f files in the build/ directory for issues like missing #include statements. If the problem relates to partitioning, chemistry, or deposition, you can isolate processes in gchem_calc_tendency.F, recompile, and rerun.

More details regarding job management in the SLURM system, please refer to: QB4 User Guide

Changing CPU Core Count

ECCOv4 is configured to run on 48 cores by default. You can increase this number (e.g., 96 cores):

Navigate to the code direcctory:

cd /MITgcm/verification/global_hg_llc90/code

Replace the original SIZE.h file:

cd /MITgcm/verification/global_hg_llc90/code

cp SIZE.h.96np SIZE.h

Recompile and move the executable to your run directory.

In the run directory, update data.exch2 to reflect the modified core count:

cp data.exch2.96np data.exch2

Update the run script run.mitgcm accordingly.

Finally, submit the modified job.