Tutorial for CESM2 Workflow Introduction
CESM2 Official Website: https://www.cesm.ucar.edu/models/cesm2/
CESM2 Quick Start Guide: https://escomp.github.io/CESM/versions/cesm2.1/html/
CIME Manual: http://esmci.github.io/cime/versions/master/html/index.html
### Table of Contents
1. Copy the code
2. Check model integrity
3. Set environment variables
4. Modify model configuration
5. Admin cluster configuration
6. Run a test case
7. Technical support
### Copy the Code
The source code of CESM2.1.3 has been downloaded to the server: /data2/cesmdata/cesm2.1.3
First, copy the source code to your home directory:
cd ~ cp -r /data2/cesmdata/cesm2.1.3 ~/$MODEL # Note: $ can be any custom name
You can also download the latest code from the CESM2 official website.
### Check Model Integrity
Go to the `$MODEL` directory and enter:
cd $MODEL/ ./manage_externals/checkout_externals -S
If the model is complete, the output should be:
Processing externals description file: Externals.cfg Processing externals description file: Externals_CLM.cfg Processing externals description file: Externals_POP.cfg Processing externals description file: Externals_CISM.cfg Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam, ./cime ./components/cam ./components/cice ./components/cism ./components/cism/source_cism ./components/clm ./components/clm/src/fates ./components/clm/tools/PTCLM ./components/mosart ./components/pop ./components/pop/externals/CVMix ./components/pop/externals/MARBL ./components/rtm ./components/ww3
If you see something like:
e- ./components/cam
You can download the source code again from the official website or contact the administrator.
### Set Environment Variables
You can first run the following code to copy `.bashrc` to your home directory:
cp /data2/cesmdata/.bashrc ~/
Activate the environment variables:
source ~/.bashrc
Later, you can adjust it as needed.
### Modify Model Configuration
Go to the model configuration directory:
cd ~/$MODEL/cime/config/cesm/machines
There are three files that need modification:
1. `config_batch.xml`
Copy the existing file (recommended):
cp /data2/cesmdata/config_batch.xml .
Or (customize if needed):
Delete the `-n` on line 116, retain `-J`, `-o`, `-e`, and add/replace:
<directive> -q normal </directive> <directive> -W 1000:10 </directive> <directive> -hostfile host </directive> </directives> <queues> <queue nodemin="1" nodemax="400" default="true">normal</queue> </queues> </batch_system>
2. `config_compilers.xml`
Copy the existing file (recommended):
cp /data2/cesmdata/config_compilers.xml .
Or (customize if needed):
Replace lines 246 `<MPICC> mpicc </MPICC>` to 263:
<NETCDF_PATH>/scratch/software/netcdf-4</NETCDF_PATH>
<MPICC> mpicc </MPICC>
<MPICXX> mpicpc </MPICXX>
<MPIFC> mpifort</MPIFC>
<SCC> icc </SCC>
<SCXX> icpc </SCXX>
<SFC> ifort </SFC>
<SLIBS>
<base> -L${NETCDF_PATH}/lib -lnetcdff -lnetcdf </base>
<append MPILIB="impi"> -mkl=cluster </append>
</SLIBS>
<SUPPORTS_CXX>TRUE</SUPPORTS_CXX>
</compiler> 3. config_machines.xml
After line 51 `<config_machines version=”2.0″>`, add:
<machine MACH="$NAME_OF_YOUR_MACHINE"> <DESC>add some descriptions</DESC> <OS>LINUX</OS> <COMPILERS>intel</COMPILERS> <MPILIBS>impi</MPILIBS> <CIME_OUTPUT_ROOT>Absolute path where model output results are stored (recommended to create an output folder in the model directory)</CIME_OUTPUT_ROOT> <DIN_LOC_ROOT>/data4/cesmdata/inputdata (CESM shared input files, do not modify the contents randomly!)</DIN_LOC_ROOT> <DIN_LOC_ROOT_CLMFORC>$DIN_LOC_ROOT/atm/datm7</DIN_LOC_ROOT_CLMFORC> <DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT> <GMAKE>make</GMAKE> <GMAKE_J>8</GMAKE_J> <BATCH_SYSTEM>lsf</BATCH_SYSTEM> <SUPPORTED_BY>YOUR_NAME</SUPPORTED_BY> <MAX_TASKS_PER_NODE>28</MAX_TASKS_PER_NODE> <MAX_MPITASKS_PER_NODE>28</MAX_MPITASKS_PER_NODE> <PROJECT_REQUIRED>FALSE</PROJECT_REQUIRED> <mpirun mpilib="impi"> <executable>mpirun</executable> <arguments></arguments> </mpirun> <module_system type="none"></module_system> <environment_variables> <env name="NETCDF_PATH">/scratch/software/netcdf-4</env> </environment_variables> </machine>
**Note:** Modifying these three files is a step where errors often occur!
### Admin Cluster Configuration
1. Modify `.bashrc` (for Intel compilers and OpenMPI):
# for Intel MPI and MKL compilers source /scratch/software/intel2019u5/compilers_and_libraries_2019.5.281/linux/bin/iccvars.sh intel64 source /scratch/software/intel2019u5/compilers_and_libraries_2019.5.281/linux/bin/ifortvars.sh intel64 source /scratch/software/intel2019u5/compilers_and_libraries_2019.5.281/linux/mkl/bin/mklvars.sh intel64 export INTEL_LICENSE_FILE=/scratch/software/intel2019u5/parallel_studio_xe_2019/licensing/parallel_studio_x.lic # for OpenMPI4 export OPENMPI=/data3/home/pzhang/software/openmpi4/ export PATH=$OPENMPI/bin:$PATH export LD_LIBRARY_PATH=$OPENMPI/lib:$LD_LIBRARY_PATH export MANPATH=$OPENMPI/share/man:$MANPATH
2. The `config_batch.xml` and `config_compilers.xml` files are the same as before.
For `config_machines.xml`, modify the `<MAX_TASKS_PER_NODE>` and `<MAX_MPITASKS_PER_NODE>`:
<MAX_TASKS_PER_NODE>80</MAX_TASKS_PER_NODE> <MAX_MPITASKS_PER_NODE>80</MAX_MPITASKS_PER_NODE>
Add this line to `<environment_variables>`:
<env name="OMP_NUM_THREADS">20</env>
### Run a Test Case
First, create a folder to store the case:
cd ~/$MODEL mkdir mycase
1. Create a new case:
cd ~/$MODEL/cime/scripts ./create_newcase --case ~/$MODEL/mycase/$casename --res f19_g16 --compset X --compiler intel --mach $NAME_OF_YOUR_MACHINE
2. Set the number of cores:
cd ~/$MODEL/mycase/$casename vim host
Press `i` to enter edit mode, then input:
cnode1 28
Press `esc`, type `:wq`, and exit.
Note: `cnodeX` represents the X node of the server, and 28 represents the number of cores for the model run. You can use the `bhosts` command to check available nodes.
3. Run:
./case.setup
4. Run:
./case.build
You will encounter a “File not found” error, stop the build by pressing `ctrl+c`, and enter:
./check_input_data
It will list a series of missing files. Then enter:
./check_input_data --download
The model will automatically download the necessary input data based on the selected compsets (Note: The total size of input data for CESM2 exceeds 10TB, download only the necessary data).
After the input data download
is complete, enter:
./check_input_data
Check if there are any missing files, and if not, run:
./case.build --clean-all ./case.build
Wait for the model to compile. If you see “Successfully,” the build was successful. If you encounter an error, check the error file for the cause.
5. Submit the job:
./case.submit
You can monitor the job status using `bjobs`. If two jobs appear (one in `run` and one in `pend` status), the model is running correctly. Once the `run` job finishes, the `pend` job will continue running. If only one job remains in `pend` status, the model has encountered an error, and you can stop the job using the `bkill` command.
Monitor job status:
cat CaseStatus
Check job output or errors:
cd ~/$MODEL/CIME_OUTPUT_ROOT/run # The output path set in config_machines.xml
### Technical Support
This tutorial uses compset X as a test case. Please choose the required compsets and grids as needed.
The official website provides compsets: https://www.cesm.ucar.edu/models/cesm2/config/2.1.3/compsets.html
Grids: https://www.cesm.ucar.edu/models/cesm2/config/2.1.3/grids.html
Configuration: https://www.cesm.ucar.edu/models/cesm2/settings/2.1.3/
Additionally, CESM2 provides a forum for questions and discussions: https://xenforo.cgd.ucar.edu/cesm/
For any issues with model runs, consult the manuals on the official website or feel free to discuss (pzhang_ebmg@163.com).
Recent Comments