Data assimilation of snow observations with CrocO¶

Implementation of a Particle Filter to assimilate spatialized snow observations within SURFEX platform
Development of vortex task to perform the assimilation sequence on beaufix-hendrix, launched through a s2m (snowtools) command.
User documentation : see CrocO user doc

Publication:
Cluzet et al., CrocO : a Particle Filter to assimilate snowpack observations in a spatialised framework (in prep)

Git projects :

Surfex_Git2, branch cen_dev
Vortex, branch cen_assim
snowtools_git, branch cen_assim

Main developments¶

Spatialized PF algorithms developped into SURFEX/SODA in global, k-localized and r-localized configurations
Use of ESCROC ensemblist version of Crocus (Lafaysse et al. 2017) and perturbed meteorological forcings (Charrois et al., 2016)
Use of Vortex (cen_assim branch) /snowtools_git to launch the assimilation sequence on beaufix and store data on hendrix (similarly as described in Run a SURFEX-Crocus experiment with vortex)

Developer name¶

Bertrand Cluzet

Date of start of development¶

January 2018

Commit number before the development¶

Surfex_Git2 : b469f610
Vortex : 4ab65aa3

Current status¶

Surfex_Git2: Phased in SURFEX V8.1 branch cen_dev, tag CrocO_v1.0
Vortex : Phased with recent (late october 2018 (commit 87922088)) cen_assim branch, tag CrocO_v1.0
snowtools_git : commit 53e2e16, cen_assim branch, tag CrocO_v1.0

Key technical points (SURFEX)¶

new diag variable SPECMOD to represent sow reflectance in MODIS bands, sampled from SPEC_ALB
writing of this diag variable into PREP file (mandatory for soda assimilation)
new dimension in PRO file : MODIS_bands
in SODA, use of a SURFEX_LIST (vector of SURFEX_TYPE) to store all members (PREP files) before the assimilation mix
mixing of PREP is done "manually" in PUT_ALL_VAR_PF.F90. Some fields (..._ISBA) are not mixed at the moment (no need for us.)
Need to actualize this routine after any change in the SURFEX types and variable names and for new prognostic/cumulated variables
LFIX_OUTPUT (see below)
reading of obs. in NETCDF.
computing Snow Cover Fraction from SWE in PREP : implementation of a depletion curve (Zaitchik et al., 2009, De Lannoy et al., 2012)

Evaluated against SURFEX test database ?¶

No, in progress.

New test added in the database¶

No, in progress

Changes in namelist¶

&NAM_ASSIM

LASSIM = .TRUE.                       ! Activate Surfex data assimilation  
LCROCO = .TRUE.                       ! Activate CrocO
CASSIM_ISBA = 'PF   '                 ! Particle Filter activation in SODA  (/!\ don't forget the '   ' ) 
CPF_CROCUS = 'GLOBAL' (default)       ! select the PF algo to perform ('GLOBAL', 'RLOCAL', 'KLOCAL')
XDLOC_PF = 1E-8                       ! localisation angle (degrees) for the RLOCAL PF. 
                                      ! Let to default or < 1e-7 to consider only local observations.
NEFF_PF = 1                           ! set the target effective sample size for the PF with inflation.
NPRINTLEV=1,                          ! verbosity
LEXTRAP_SEA=.FALSE.,                  !  |
LEXTRAP_WATER=.FALSE.,                !  |  Prevent SODA 
LEXTRAP_NATURE=.FALSE.,               !  |  from being too fancy
LWATERTG2=.TRUE.,                     !  |

(!) the following option has been deleted in code versions posterior to #5734b28e
NLOC_PF = 0 (default)                 ! set the maximal number of obs (>0) to select in the 'RLOCAL' and 'KLOCAL'.

About the algorithm of the Particle Filter:
- 'GLOBAL' means that all the observations are used globally for all the points of the simulation.
- 'RLOCAL' means that for each point of the simulation, only observations around this point (if any) are used.
- 'KLOCAL' means that for each point of the simulation, a correlation with other points is calculated. Only observations on best correlated points are used.

&NAM_OBS

  
NOBSTYPE = 8                  ! number of different obs. variables (up to 9)
CFILE_FORMAT_OBS="NC    "     ! daily obs. files on NETCDF file format (/!\ don't forget the '    ' ) 
COBS_M = 'B1','B2','B3','B4',\! Names of MODIS bands and Snow cover fraction variables 
         'B5','B6','B7','SCF' ! in the observation files. (/!\ same length as NOBSTYPE)
                              ! other possible values : 'DEP' (snow depth),
                              ! 'R51', R52', 'R53', 'R54' (band ratios)
NNCO = 1,1,1,1,1,1,1,1        ! select which variables to assimilate among COBS_M 
                              ! (/!\ same length as NOBSTYPE)
XERROBS_M = 0.00071, 0.00046,\! default obs.error *variances* for MODIS bands 
            0.00056, 0.00056,\! (charrois PhD manuscript pp 104.,140.).
            0.002, 0.0015,\   ! last one is a percentage of value for SCF. 
            0.00078, 0.05,\   ! For DEP, it's also an error variance [m^2]
XERROBS_FACTOR_M = 1,1,1,1,1,\! If necessary multiply the prescribed errors by some factors
                   1,1,1

&NAM_VAR

NVAR=8                        ! number of different model control variables (up to 9)
CVAR_M='PB1','PB2','PB3',\    ! keywords for the control variables to be read. 
       'PB4','PB5','PB6',\    ! here, 7 MODIS spectral bands 
       'PB7','PSB'            ! + bulk SWE, which is the control variable for SCF obs.
                              ! other possible values : 'DEP' (snow depth),
                              ! 'R51', 'R52', 'R53', 'R54' (band ratios).
NNCV = 1,1,1,1,1,1,1,1        ! select which variables to assimilate among CVAR_M
XTAUSCF =4                    ! value for the tau parameter of the depletion curve 
                              !used to retrieve model SCF from bulk SWE (default=4)
XCEILSCF = 20                 ! value for the ceil SWE of the depletion curve (default=20mm)

&NAM_ENS

NENS = 1                      ! ensemble size (overwritten by snowtools_git argument --nmembers)

&NAM_DIAG_ISBAN

LSPECMOD = .TRUE.  ! write MODIS spectral into PREP files, mandatory for assimilation
LPROBANDS = .TRUE. ! mandatory too for MODIS assimilation

&NAM_DIAG_SURFN

LRAD_BUDGET = .TRUE.

&NAM_WRITE_DIAG_SURFn

LRESETCUMUL = .TRUE.
LSNOWDIMNC = .TRUE. ! mandatory + activate layer/bands dimensions in PRO
CSELECT = 'SPECMOD' ! add MODIS bands to PRO file

&NAM_IO_OFFLINE

LFIX_OUTPUT = .TRUE. !(default : .FALSE.) : force hours of output in PRO at 00h+n*timestep 
                     ! instead of timestart+ n*timestep (/!\ possible edge effects on cumulated diags)

local testing of SODA¶

SODA is the executable performing the assimilation. Provided a set of background PREP files and an observation, it produces a set of analyzed PREP files.
It is possible to test the performance of this algorithm locally, (without the use of vortex, snowtoools_git etc.)
For that, simply execute SODA in a repertory where all files are given (Namelist, PGD, PREP files, observations, etc.)
For more details, ask for support.

Other required changes¶

Namelist¶

Computing of snow reflectance in MODIS bands requires impurities handling options, see Explicit representation of impurities

Forcing files¶

only yearly files are handled at the moment
see Explicit representation of impurities about adding impurities deposition fluxes to the forcings

Spinup files¶

mandatory
/!\ V8.0 files are not compatible with V8.1, but it is possible to upgrade it (ask for support).

New files¶

Assimilation configuration file¶

mandatory (see an example at s2m_postes_12_csv.ini)
named s2m_<region_id>.ini
- region_id is the safran id of the region (for ex. 12 for Grandes-Rousses)
- region_name is the safran name of the region (for ex. grandes_rousses for Grandes-Rousses)
format: python configuration file (text file):
- first line : section header set to DEFAULT:
```
[DEFAULT]
```
- use the assimdates keyword to specify the *assimilation dates. ex:
```
assimdates=2013110106,2013120106
```
- use the members_id keyword to specify the ids of the ESCROC members you'd like to run (if you are using ESCROC E1* ensemble). Otherwise they will be randomly drawn (and written down in the configuration file). This is necessary for twin experiments and reproducibility.ex with 3 members:
```
membersid=505,402,705
```
some additional information on the experiment are appended to it during the run. It is then archived (see below)

Observation files¶

daily files named obs_<scope>.surfex.<geometry>_yyyymmddhh.nc
ex : obs_MODIS_grandes_rousses_20341231.nc
NETCDF format(see attached example)

same geometry as PREP/PGD
variables must be double
variables _FillValue must be set to 1.e+20.

for synthetical observations, <your_sensor> is free to define.

Particle Filter auxiliary output files¶

Output from the assimilation sequence.
One file per assimilation date, named PART_YYYYMMDDHH.txt
This last file give the ids(starting at 1)of the replicated members, separated by spaces. Moreover:
- if DISTRIBUTED assimilation, assimilation per grid point as CPF_CROCUS = 'KLOCAL': one line per grid point
- if SEMI-DISTRIBUTED assimilation, CPF_CROCUS = 'GLOBAL': only one line (for all points, same ids of replicated members)

Additionally, depending on the PF configuration the following files are produced

ALPHA : (NEFF_PF >0) inflation factor of the background correlation matrix for each point. Only one value if CPF_CROCUS = 'GLOBAL'.
BG_CORR : (CPF_CROCUS = 'KLOCAL') background correlation matrix for the assimilation variables (NVAR lines per grid point, variables written in the order of NAM_VAR).
IMASK : (CPF_CROCUS = 'KLOCAL') klocalisation mask (FORTRAN IDs of the selected points) for the different variables (NVAR lines per grid point, variables written in the order of NAM_VAR).

Simulation framework¶

Simulations are performed on beaufix-hendrix through an s2m command (see added options below).
Vortex requires a strict file storing and naming. The following lines are detailing it.

Forcing files¶

stored in a vortex path on hendrix :
<yourhome>/vortex/safran/<region_id>/<xpid_forcing>/mbxxxx/meteo/
- xpid_forcing is a user-defined name (usually the name of the experiment that generated the forcings), given to s2m via -f option (see below)

Spinup files¶

named PGD_<region_name>.nc and PREP_<starting_date>.nc
stored in a vortex path on hendrix (xxx is "pgd" and "prep" respectively):
<yourhome>/vortex/s2m/<region_id>/spinup/<xxx>/

Namelist¶

Stored on beaufix home.

Assimilation Configuration file¶

Stored on beaufix home.

Observation files¶

stored in a vortex path on hendrix :
<yourhome>/vortex/s2m/<region_id>/obs/<your_sensor>/
for synthetical observations : do the same.

Simulation outputs storing¶

If the simulation has succeded :

all files are deleted from beaufix
PREP and PRO files are stored on hendrix :
<your_home>/vortex/s2m/<region_id>/<xpid>/<mbxxxx>/<xxx>
- xpid is specified through s2m -o option, see below
- xxx is bg, an or pro for backgroud PREP, analysis PREP and PRO files respectively
Assimilation configuration file (useful for twin experiments and traceability) is stored in <your_home>/vortex/s2m/<region_id>/conf/
Base namelist is also stored in <your_home>/vortex/s2m/<region_id>/conf/ (name namelist.surfex.foo)
PF sample files are stored in <your_home>/vortex/s2m/<region_id>/workSODA/ (name PART_YYYYMMDDHH.txt.foo)
For post-processing and dev, PREP files and the configuration file can also be fetched to sxcen (in /cnrm/cen/users/NO_SAVE/...), which has no backup and limited space (see the options below)

New s2m options/modifications¶

- w option is useless
- o xpid : output repertory on hendrix
--escroc E1Tartes or E1noTartes : Escroc big ensembles with Tartes only (1944 members) or no Tartes (the rest). nmembers will be randomly drawn from this ensemble between each assimilation date
--croco your_path_to_assimilation_configuration_file : activate CrocO and give the path to your conf file.
--nforcing : numbers of different forcings to use
- f xpid_forcing: xpid for your forcing files on hendrix
--openloop : activate openloop sequence (no assimilation but output at each assimilation date)
--synth mbid : assimilation of synthetic data. Remove and replace the <mbid> member (synthetic truth)
--real : assimilation of real data
--walltime : estimate for execution time on beaufix
--writesx : fetch output data directly to NO_SAVE/ on sxcen.
--sensor your_sensor: specify the name of the sensor used to generate the observations (default :MODIS), or of your synthetical obs.

Files (1)

Project

General

Profile

snowtools_git

Wiki

Data assimilation of snow observations with CrocO¶

Main developments¶

Developer name¶

Date of start of development¶

Commit number before the development¶

Current status¶

Key technical points (SURFEX)¶

Evaluated against SURFEX test database ?¶

New test added in the database¶

Changes in namelist¶

local testing of SODA¶

Other required changes¶

Namelist¶

Forcing files¶

Spinup files¶

New files¶

Assimilation configuration file¶

Observation files¶

Particle Filter auxiliary output files¶

Simulation framework¶

Forcing files¶

Spinup files¶

Namelist¶

Assimilation Configuration file¶

Observation files¶

Simulation outputs storing¶

New s2m options/modifications¶