Create work directory
Copy all the files in /data/Lab/MITgcm, which contains a template for module simulation.
cp -r /data/Lab/MITgcm/ .
Source code compiling
Then let’s go to the build/ directory and build your Makefile:
cd ~/MITgcm/verification/global_hg_igsm/build premake
This command line option tells genmake to override model source code with any files in the directory ../code/. I have written an alias called ‘premake’ in my .bashrc file to replace this long genmake command. It’s actually running: ../../../tools/genmake2 -mods=../code -optfile=../../../tools/build_options/linux_ifort_impi
Once a Makefile has been generated, we create the dependencies with the command:
make depend
This modifies the Makefile by attaching a (usually, long) list of files upon which other files depend. The purpose of this is to reduce re-compilation if and when you start to modify the code. The make depend command also creates links from the model source to this directory. It is important to note that the make depend stage will occasionally produce warnings or errors since the dependency parsing tool is unable to find all of the necessary header files (eg. netcdf.inc). In these circumstances, it is usually OK to ignore the warnings/errors and proceed to the next step.
Next, you compile the code using:
make
You can speed up the compilation by adding “-j4” or “-j8” (e.g., “make -j8”) to split the compilation amongst 4 or 8 processors.
If the compiling goes well (i.e. no error message), we can move the generated mitgcmuv file to your run directory:
cp mitgcmuv ../run/mitgcmuv.12np
Here, the postfix 12np means it requires 12 cpu cores to run.
Model running
The model run folder is located here:
cd ~/MITgcm/verification/global_hg_igsm/run
Here I will quickly go over the important files and subfolders in this folder:
- forcing: atmospheric forcing data, e.g. wind speed, precipitation, temperature
- initial: initial conditions for model run
- input_hg: atmospheric Hg concentrations and deposition fluxes.
- input_darwin: ecosystem and biogeochemical parameters from the Darwin simulation.
- control: control files for ECCOv4
- data* files: control files for MITgcm run
Now, we can submit a job to Caramel: bsub < run.mehg.6+2 If your run finishes without any problems, the very last line of your STDOUT.0000 file should indicate the model ‘ENDED NORMALLY’.
data* files
data:
nIter0: Run starting timestep number Be consistent with PTRACERS_Iter0 in data.ptracers.
180 represents 1860.01. If you run from 1950.01, nIter0=194400 (1949.12), and so on.
nTimeSteps: Momentum equation timestep
If you run for 10 years, nTimesteps = 21600.
Check running status
You can most simply enter either of the following for information on your runs:
bjobs
Stop running
You can also kill your job if you find out that something is going wrong:
bkill JOBID (e.g. bkill 4321)
JOBID is a number assigned to your job, and could be found by running bjobs.
Model tuning
1. If your run crashes, check the following files for error messages:
STDERR.0000
STDOUT.0000
omp.error.<job_id> (where <job_id> is the ID number assigned by the queue system)
2. You may also find it is helpful to check the *.f files in your build/ directory. This is what the code looks like at ‘run time’, so if pieces of code are being chopped off or #include statements are missing, this kind of thing will turn up in the *.f compiled files.
3. If you want to isolate if your problem is coming from partitioning, chemistry, deposition, etc, you can comment out individual processes in gchem_calc_tendency.F. Recompile with ‘make’ and run with a limited number of processes turned on.
Modify the number of CPU cores
Here’s how to change the number of cores you need for a job:
cd ~/MITgcm/verification/global_hg_igsm/code vi SIZE.h change sNx = 36 change sNy = 30 change nPx = 4 change nPy = 3
Ensure that sNx*nPx=144,sNy*nPy=90. nPx*nPy is the number of cores you use. Save and close SIZE.h Go to your build/ directory and recompile your code:
make cp mitgcmuv ../run/mitgcmuv.12np
Submit your job.
Recent Comments