Writing a CMORizer script for an additional dataset#
ESMValTool is designed to work with CF compliant data and follows the CMOR tables from the CMIP data request, therefore the observational datasets need to be CMORized for usage in ESMValTool. The following steps are necessary to prepare an observational data set for the use in ESMValTool.
Note
CMORization as a fix. As of early 2020, we’ve started implementing cmorization as fixes. As compared to the workflow described below, this has the advantage that the user does not need to store a duplicate (CMORized) copy of the data. Instead, the CMORization is performed ‘on the fly’ when running a recipe. ERA5 is the first dataset for which this ‘CMORization on the fly’ is supported. For more information, see Datasets in native format.
1. Check if your variable is CMOR standard#
Most variables are defined in the CMIP data request and can be found in the
CMOR tables in the folder /esmvalcore/cmor/tables/cmip6/Tables/,
differentiated according to the MIP
they belong to. The tables are a
copy of the PCMDI guidelines. If you find the
variable in one of these tables, you can proceed to the next section.
If your variable is not available in the standard CMOR tables, you need to write a custom CMOR table entry for the variable as outlined below and add it to /esmvalcore/cmor/tables/custom/.
To create a new custom CMOR table you need to follow these guidelines:
Provide the
variable_entry
;Provide the
modeling_realm
;Provide the variable attributes, but leave
standard_name
blank. Necessary variable attributes are:units
,cell_methods
,cell_measures
,long_name
,comment
.Provide some additional variable attributes. Necessary additional variable attributes are:
dimensions
,out_name
,type
. There are also additional variable attributes that can be defined here (see the already available cmorizers).
It is recommended to use an existing custom table as a template, to edit the
content and save it as CMOR_<short_name>.dat
.
2. Edit your configuration file#
Make sure that beside the paths to the model simulations and observations, also
the path to raw observational data to be cmorized (RAWOBS
) is present in
your configuration file.
3. Store your dataset in the right place#
The folder RAWOBS
needs the subdirectories Tier1
, Tier2
and
Tier3
. The different tiers describe the different levels of restrictions
for downloading (e.g. providing contact information, licence agreements)
and using the observations. The unformatted (raw) observations
should then be stored then in the appropriate of these three folders.
For each additional dataset, an entry needs to be made to the file datasets.yml. The dataset entry should contain:
the correct
tier
information;the
source
of the raw data;the
last_access
date;the
info
that explain how to download the data.
Note that these fields should be identical to the content of the header of the cmorizing script (see Section 4. Create a cmorizer for the dataset).
3.1 Downloader script (optional)#
A Python script can be written to download raw observations
from source and store the data in the appropriate tier subdirectory of the
folder RAWOBS
automatically.
There are many downloading scripts available in
/esmvaltool/cmorizers/data/downloaders/datasets/
where several data download mechanisms are provided:
A wget get based downloader for http(s) downloads, with a specific derivation for NASA datasets.
A ftp downloader with a specific derivation for ESACCI datasets available from CEDA.
A Climate Data Store downloader based on cdsapi.
Note that the name of this downloading script has to be identical to the name of the dataset.
Depending on the source server, the downloading script needs to contain paths to
raw observations, filename patterns and various necessary fields to retrieve
the data.
Default start_date
and end_date
can be provided in cases where raw data
are stored in daily, monthly, and yearly files.
The downloading script for the given dataset can be run with:
esmvaltool data download --config_file <config-user.yml> <dataset-name>
The options --start
and --end
can be added to the command above to
restrict the download of raw data to a time range. They will be ignored if a specific dataset
does not support it (i.e. because it is provided as a single file). Valid formats are
YYYY
, YYYYMM
and YYYYMMDD
. By default, already downloaded data are not overwritten
unless the option --overwrite=True
is used.
4. Create a cmorizer for the dataset#
There are many cmorizing scripts available in /esmvaltool/cmorizers/data/formatters/datasets/ where solutions to many kinds of format issues with observational data are addressed. These scripts are either written in Python or in NCL.
Note
NCL support will terminate soon, so new cmorizer scripts should preferably be written in Python.
How much cmorizing an observational data set needs is strongly dependent on the original NetCDF file and how close the original formatting already is to the strict CMOR standard.
In the following two subsections two cmorizing scripts, one written in Python and one written in NCL, are explained in more detail.
4.1 Cmorizer script written in python#
Find here an example of a cmorizing script, written for the MTE
dataset
that is available at the MPI for Biogeochemistry in Jena: mte.py.
All the necessary information about the dataset to write the filename
correctly, and which variable is of interest, is stored in a separate
configuration file: MTE.yml
in the directory ESMValTool/esmvaltool/cmorizers/data/cmor_config/
. Note
that both the name of this configuration file and the cmorizing script have to be
identical to the name of your dataset.
It is recommended that you set project
to OBS6
in the
configuration file. That way, the variables defined in the CMIP6 CMOR table,
augmented with the custom variables described above, are available to your script.
The first part of this configuration file defines the filename of the raw
observations file. The second part defines the common global attributes for
the cmorizer output, e.g. information that is needed to piece together the
final observations file name in the correct structure (see Section 6. Naming convention of the observational data files).
Another global attribute is reference
which includes a doi
related to the dataset.
Please see the section adding references
on how to add reference tags to the reference
section in the configuration file.
If a single dataset has more than one reference,
it is possible to add tags as a list e.g. reference: ['tag1', 'tag2']
.
The third part in the configuration file defines the variables that are supposed to be cmorized.
The actual cmorizing script mte.py
consists of a header with
information on where and how to download the data, and noting the last access
of the data webpage.
The main body of the CMORizer script must contain a function called
def cmorization(in_dir, out_dir, cfg, cfg_user, start_date, end_date):
with this exact call signature. Here, in_dir
corresponds to the input
directory of the raw files, out_dir
to the output directory of final
reformatted data set, cfg
to the dataset-specific configuration file,
cfg_user
to the user configuration file, start_date
to the start
of the period to format, and end_date
to the end of the period to format.
If not needed, the last three arguments can be ignored using underscores.
The return value of this function is ignored. All
the work, i.e. loading of the raw files, processing them and saving the final
output, has to be performed inside its body. To simplify this process, ESMValTool
provides a set of predefined utilities.py, which can be imported into your CMORizer
by
from esmvaltool.cmorizers.data import utilities as utils
Apart from a function to easily save data, this module contains different kinds of small fixes to the data attributes, coordinates, and metadata which are necessary for the data field to be CMOR-compliant.
Note that this specific CMORizer script contains several subroutines in order
to make the code clearer and more readable (we strongly recommend to follow
that code style). For example, the function _get_filepath
converts the raw
filepath to the correct one and the function _extract_variable
extracts and
saves a single variable from the raw data.
4.2 Cmorizer script written in NCL#
Find here an example of a cmorizing script, written for the ESACCI XCH4
dataset that is available on the Copernicus Climate Data Store:
cds_xch4.ncl.
The first part of the script collects all the information about the dataset that are necessary to write the filename correctly and to understand which variable is of interest here. Please make sure to provide the correct information for following key words: DIAG_SCRIPT, VAR, NAME, MIP, FREQ, CMOR_TABLE.
Note: the fields
VAR
,NAME
,MIP
andFREQ
all ask for one or more entries. If more than one entry is provided, make sure that the order of the entries is the same for all four fields! (for example, that the first entry in all four fields describe the variablexch4
that you would like to extract);Note: some functions in the script are NCL-specific and are available through the loading of the script interface.ncl. There are similar functions available for python scripts.
In the second part of the script each variable defined in VAR
is separately
extracted from the original data file and processed. Most parts of the code are
commented, and therefore it should be easy to follow. ESMValTool provides a set
of predefined utilities.ncl, which are imported by default into your CMORizer.
This module contains different kinds of small fixes to the data attributes,
coordinates, and metadata which are necessary for the data field to be
CMOR-compliant.
5. Run the cmorizing script#
The cmorizing script for the given dataset can be run with:
esmvaltool data format --config_file <config-user.yml> <dataset-name>
The options --start
and --end
can be added to the command above to
restrict the formatting of raw data to a time range. They will be ignored if a specific dataset
does not support it (i.e. because it is provided as a single file). Valid formats are
YYYY
, YYYYMM
and YYYYMMDD
.
Note
The output path given in the configuration file is the path where
your cmorized dataset will be stored. The ESMValTool will create a folder
with the correct tier information
(see Section 2. Edit your configuration file) if that tier folder is not
already available, and then a folder named after the dataset.
In this folder the cmorized data set will be stored as a NetCDF file.
The cmorized dataset will be automatically moved to the correct tier
subfolder of your OBS or OBS6 directory if the option
--install=True
is used in the command above and no such directory
was already created.
If your run was successful, one or more NetCDF files are produced in your output directory.
If a downloading script is available for the dataset, the downloading and the cmorizing scripts can be run in a single command with:
esmvaltool data prepare --config_file <config-user.yml> <dataset-name>
Note that options from the `esmvaltool data download
and
esmvaltool data format
commands can be passed to the above command.
6. Naming convention of the observational data files#
For the ESMValTool to be able to read the observations from the NetCDF file, the file name needs a very specific structure and order of information parts (very similar to the naming convention for observations in ESMValTool v1.0). The file name will be automatically correctly created if a cmorizing script has been used to create the netCDF file.
The correct structure of an observational data set is defined in config-developer.yml, and looks like the following:
OBS_[dataset]_[type]_[version]_[mip]_[short_name]_YYYYMM-YYYYMM.nc
For the example of the CDS-XCH4
data set, the correct structure of the
file name looks then like this:
OBS_CDS-XCH4_sat_L3_Amon_xch4_200301-201612.nc
The different parts of the name are explained in more detail here:
OBS: describes what kind of data can be expected in the file, in this case
observations
;CDS-XCH4: that is the name of the dataset. It has been named this way for illustration purposes (so that everybody understands it is the xch4 dataset downloaded from the CDS), but a better name would indeed be
ESACCI-XCH4
since it is a ESA-CCI dataset;sat: describes the source of the data, here we are looking at satellite data (therefore
sat
), could also bereanaly
for reanalyses;L3: describes the version of the dataset:
Amon: is the information in which
mip
the variable is to be expected, and what kind of temporal resolution it has; here we expectxch4
to be part of the atmosphere (A
) and we have the dataset in a monthly resolution (mon
);xch4: Is the name of the variable. Each observational data file is supposed to only include one variable per file;
200301-201812: Is the period the dataset spans with
200301
being the start year and month, and201812
being the end year and month;
Note
There is a different naming convention for obs4MIPs
data (see the exact
specifications for the obs4MIPs data file naming convention in the
config-developer.yml
file).
7. Test the cmorized dataset#
To verify that the cmorized data file is indeed correctly formatted, you can
run a dedicated test recipe, that does not include any diagnostic, but only
reads in the data file and has it processed in the preprocessor. Such a recipe
is called recipes/examples/recipe_check_obs.yml
. You just need to add a
diagnostic for your dataset following the existing entries.
Only the diagnostic of interest needs to be run, the others should be commented
out for testing.