Debug SURFEX-Crocus¶

If you are implementing new developments or if you meet unexpected troubles during a run execution, please check the following instructions before contacting the support team :

Always compile SURFEX with the DEBUG option :

export OPTLEVEL=DEBUG
./configure

Once the development is finished,recompile with O2 optimization level before running your long experiments.

For problems occurring inside the snow scheme, it is helpful to use a special debugging mode for Crocus, please read the following documentation:
http://www.cnrm.meteo.fr/surfex-lab/IMG/pdf/doc_crodebug.pdf

Meteo-France supercomputers
To understand unexpected crashes during belenos runs, two options are possible:

1/ For complex bugs, it is highly recommended to use use the ddt debugger from ARM-FORGE.
A local installation of the debugger is much easier to use from Grenoble than the remote installation on the supercomputers.
Unfortunately Meteo-France DSI does not currently provide any access to the debugger for teleworking.

Local installation procedure:¶

First, check the remote version of DDT on the server :

module avail

Select a version and find the remote installation path for later step:

module show arm/forge/22.1.1

Locally, download the same version of ddt client on your machine at https://developer.arm.com/downloads/-/arm-forge/older-versions-of-remote-client-for-arm-forge
(Since 2023 the ARM forge debugger has a new name, Linaro forge, if the previous link is dead use the following: https://www.linaroforge.com/downloadForge/)
Then untar and run the install on your machine

tar xvf arm-forge-22.1.3-linux-x86_64.tar
arm-forge-22.1.3-linux-x86_64/textinstall.sh

Follow the installation procedure and specify an appropriate path for installation (example: home/username/arm/forge/)
Then start the debugger locally:

/home/username/arm/forge/bin/forge

Below "remote launch" click on "configure", "add", and fill the different fields :

Connection name: belenos
Hostname: username@belenos
Remote installation directory: /opt/softs/arm/22.1.1/bin

and click "ok"

Now, you are ready to use the debugger from your PC.

Run the debugger job on the server¶

Remember, that the binaries must have be compiled in DEBUG mode.
Then, prepare a simple debug job job_debug.sh following this example:

#!/bin/bash
#SBATCH --verbose
#SBATCH --job-name=debug_offline
#SBATCH --nodes=1
#SBATCH --ntasks=80
#SBATCH --ntasks-per-core=1
#SBATCH --time=01:00:00

module load arm/forge/22.1.1
cd /scratch/mtool/username/abort/dump_XXXXXX_reas2m/root/surfex_vortex_task
ulimit -s unlimited
ddt --connect srun --ntasks 80 /home/cnrm_other/cen/mrns/lafaysse/SURFEX/cen_dev/exe/OFFLINE-LXifort-SFX-V8-1-1-MPIAUTO-OMP-DEBUG-X0

Tricky segmentation faults are likely to happen if you forget "ulimit -s unlimited"

Run the job:

sbatch job_debug.sh

or in case of saturation:

sbatch --partition=debug256 --qos=debug job_debug.sh

Once the job is running, a pop-up opens in ddt : you must accept the remote connection in ddt on your PC.
You can start using the debugger normally.

Don't forget to tick the OpenMP box and sectect 1 threads!!

Memory debugging must be disactivated with this version

Full documentation available here: http://intradsi.meteo.fr/spip.php?article2007