MITgcm 18km Guide

by | Sep 7, 2024 | MITgcm

Create working directory

Copy all the files in /data2/Lab/MITgcm/gud-dev, which contains a template for module simulation.

   cp -r /data2/Lab/MITgcm/gud-dev/ YOUR_FOLDER

Source code compiling

   cd YOUR_FOLDER
   cp /data2/home/ppwu/gud-dev/env.mitgcm.openmpi env.mitgcm
   cd verification

Create a folder for the high resolution model：

   mkdir global_18km   
   cd global_18km

Copy the code and input folders:

   cp -r /data2/Lab/MITgcm/gud-dev/cube92/code .   
   cp -r /data2/Lab/MITgcm/gud-dev/cube92/input/* .

Create the build directory and build your Makefile:

   mkdir build
   cd build
   ../../../tools/genmake2 -mpi -mods=../code -optfile=../../../tools/build_options/linux_ifort_openmpi

If this long command is aliased as “premke” in ~/.bashrc, it can be replaced with “premake” command.

Once a Makefile has been generated, we create the dependencies with the command:

   make depend

This modifies the Makefile by attaching a (usually, long) list of files upon which other files depend. The purpose of this is to reduce re-compilation if and when you start to modify the code. The make depend command also creates links from the model source to this directory. It is important to note that the make depend stage will occasionally produce warnings or errors since the dependency parsing tool is unable to find all of the necessary header files (eg. netcdf.inc). In these circumstances, it is usually OK to ignore the warnings/errors and proceed to the next step.

Next, you compile the code using:

   make -j8

You can speed up the compilation by adding “-j4” or “-j8” (e.g., “make -j8”) to split the compilation amongst 4 or 8 processors.

If the compiling goes well (i.e. no error message), we can move the generated mitgcmuv file to your run directory:

   cp mitgcmuv ../run/mitgcmuv.24np

Here, the postfix 24np means it requires 24 cpu cores to run.

Model running

The model run folder is located here:

   cd ~/MITgcm/gud-dev/verification/global_18km/run

data* files

   vi data

deltaTClock = 3600. indicates the model’s time resolution, in seconds. A value of 3600 means the model calculates once per hour. Typically, deltaTClock = 3600 doesn’t need to be changed, as modifying it may cause instability in the calculations.

nTimeSteps represents the simulation time in the model and is related to deltaTClock. For example, with deltaTClock = 3600, the model starts on January 10, 1992. With a time resolution of 1 hour, the number of time steps for 10 years is 87672, calculated as (366*3 + 365*7) * 24 = 87672. With deltaTClock = 1200, meaning a time resolution of 20 minutes, the number of time steps for 10 years is 263016, calculated as (366*3 + 365*7) * 72 = 263016. First, calculate the total number of days in 10 years, then multiply by 24 for a 1-hour time resolution. The same principle applies for other resolutions.

   vi run.18km

The places that need to be modified are:

  1. BSUB -n 24 # number of tasks in job

mpiexec -np 24 ./mitgcmuv.24np — 24 is the number of CPU cores being used, and mitgcmuv.24np is the executable file copied from the build directory.

BSUB -m "cnode10" specifies running on cnode10. You can omit this by deleting the line.

BSUB -m "cnode3 cnode4" specifies running on both cnode3 and cnode4.

You can use the bhosts command to check the available CPU cores on each node and submit the job to a node with available cores.

submit your job:

   bsub < run.18km

Check running status

After submitting the job, you can check the model’s progress and errors in omp.output, omp.error, STDOUT, and STDERR. The omp.output and omp.error files will only appear after the model run is complete. Pay attention to the timestamps of STDOUT and STDERR. Before submitting a new job, you can delete the previous job’s STD files using:

   rm STD*

After submitting the job with bsub, you can check the job’s running status using:

   bjobs

Stop the model

You can also kill your job if you find out that something is going wrong:

   bkill JOBID (e.g. bkill 4321)

JOBID is a number assigned to your job, and could be found by running bjobs.

Modify the number of CPU cores

   cd ~/MITgcm/gud-dev/verification/global_18km/code
   cp /data2/home/ppwu/gud-dev/verification/global_18km/code/SIZE.h* .

If you want to use 54 cores, replace SIZE.h with SIZE.h.54np.

   cp SIZE.h.54np SIZE.h

Enter the build directory and recompile.