.. _new-cmorizer:
Writing a CMORizer script for an additional dataset
***************************************************
ESMValTool is designed to work with `CF compliant `_
data and follows the CMOR tables from the CMIP data request, therefore
the observational datasets need to be CMORized for usage in ESMValTool.
The following steps are necessary to prepare an observational
data set for the use in ESMValTool.
| `1. Check if your variable is CMOR standard`_
| `2. Edit your configuration file`_
| `3. Store your dataset in the right place`_
| `3.1 Downloader script (optional)`_
| `4. Create a cmorizer for the dataset`_
| `4.1 Cmorizer script written in python`_
| `4.2 Cmorizer script written in NCL`_
| `5. Run the cmorizing script`_
| `6. Naming convention of the observational data files`_
| `7. Test the cmorized dataset`_
.. note::
**CMORization as a fix.** As of early 2020, we've started implementing cmorization as
*fixes*. As compared to the workflow described below, this has the advantage that
the user does not need to store a duplicate (CMORized) copy of the data. Instead, the
CMORization is performed 'on the fly' when running a recipe. **ERA5** is the first dataset
for which this 'CMORization on the fly' is supported. For more information, see
:ref:`inputdata_native_datasets`.
1. Check if your variable is CMOR standard
==========================================
Most variables are defined in the CMIP data request and can be found in the
CMOR tables in the folder `/esmvalcore/cmor/tables/cmip6/Tables/
`_,
differentiated according to the ``MIP`` they belong to. The tables are a
copy of the `PCMDI `_ guidelines. If you find the
variable in one of these tables, you can proceed to the next section.
If your variable is not available in the standard CMOR tables,
you need to write a custom CMOR table entry for the variable
as outlined below and add it to `/esmvalcore/cmor/tables/custom/
`_.
To create a new custom CMOR table you need to follow these
guidelines:
- Provide the ``variable_entry``;
- Provide the ``modeling_realm``;
- Provide the variable attributes, but leave ``standard_name`` blank. Necessary
variable attributes are: ``units``, ``cell_methods``, ``cell_measures``,
``long_name``, ``comment``.
- Provide some additional variable attributes. Necessary additional variable
attributes are: ``dimensions``, ``out_name``, ``type``. There are also
additional variable attributes that can be defined here (see the already
available cmorizers).
It is recommended to use an existing custom table as a template, to edit the
content and save it as ``CMOR_.dat``.
2. Edit your configuration file
===============================
Make sure that beside the paths to the model simulations and observations, also
the path to raw observational data to be cmorized (``RAWOBS``) is present in
your configuration file.
3. Store your dataset in the right place
========================================
The folder ``RAWOBS`` needs the subdirectories ``Tier1``, ``Tier2`` and
``Tier3``. The different tiers describe the different levels of restrictions
for downloading (e.g. providing contact information, licence agreements)
and using the observations. The unformatted (raw) observations
should then be stored in the appropriate of these three folders.
For each additional dataset, an entry needs to be made to the file
`datasets.yml
`_.
The dataset entry should contain:
- the correct ``tier`` information;
- the ``source`` of the raw data;
- the ``last_access`` date;
- the ``info`` that explain how to download the data.
Note that these fields should be identical to the content of the header
of the cmorizing script (see Section `4. Create a cmorizer for the dataset`_).
3.1 Downloader script (optional)
--------------------------------
A Python script can be written to download raw observations
from source and store the data in the appropriate tier subdirectory of the
folder ``RAWOBS`` automatically.
There are many downloading scripts available in
`/esmvaltool/cmorizers/data/downloaders/datasets/
`_
where several data download mechanisms are provided:
- A `wget` get based downloader for http(s) downloads, with a specific derivation for NASA datasets.
- A `ftp` downloader with a specific derivation for ESACCI datasets available from CEDA.
- A Climate Data Store downloader based on `cdsapi`.
Note that the name of this downloading script has to be identical to the
name of the dataset.
Depending on the source server, the downloading script needs to contain paths to
raw observations, filename patterns and various necessary fields to retrieve
the data.
Default ``start_date`` and ``end_date`` can be provided in cases where raw data
are stored in daily, monthly, and yearly files.
The downloading script for the given dataset can be run with:
.. code-block:: console
esmvaltool data download --config_file
The options ``--start`` and ``--end`` can be added to the command above to
restrict the download of raw data to a time range. They will be ignored if a specific dataset
does not support it (i.e. because it is provided as a single file). Valid formats are
``YYYY``, ``YYYYMM`` and ``YYYYMMDD``. By default, already downloaded data are not overwritten
unless the option ``--overwrite=True`` is used.
4. Create a cmorizer for the dataset
====================================
There are many cmorizing scripts available in
`/esmvaltool/cmorizers/data/formatters/datasets/
`_
where solutions to many kinds of format issues with observational data are
addressed. These scripts are either written in Python or in NCL.
.. note::
NCL support will terminate soon, so new cmorizer scripts should preferably be
written in Python.
How much cmorizing an observational data set needs is strongly dependent on
the original NetCDF file and how close the original formatting already is to
the strict CMOR standard.
In the following two subsections two cmorizing scripts, one written in Python
and one written in NCL, are explained in more detail.
4.1 Cmorizer script written in python
-------------------------------------
Find here an example of a cmorizing script, written for the ``MTE`` dataset
that is available at the MPI for Biogeochemistry in Jena: `mte.py
`_.
All the necessary information about the dataset to write the filename
correctly, and which variable is of interest, is stored in a separate
configuration file: `MTE.yml
`_
in the directory ``ESMValTool/esmvaltool/cmorizers/data/cmor_config/``. Note
that both the name of this configuration file and the cmorizing script have to be
identical to the name of your dataset.
It is recommended that you set ``project`` to ``OBS6`` in the
configuration file. That way, the variables defined in the CMIP6 CMOR table,
augmented with the custom variables described above, are available to your script.
The first part of this configuration file defines the filename of the raw
observations file. The second part defines the common global attributes for
the cmorizer output, e.g. information that is needed to piece together the
final observations file name in the correct structure (see Section `6. Naming convention of the observational data files`_).
Another global attribute is ``reference`` which includes a ``doi`` related to the dataset.
Please see the section `adding references
`_
on how to add reference tags to the ``reference`` section in the configuration file.
If a single dataset has more than one reference,
it is possible to add tags as a list e.g. ``reference: ['tag1', 'tag2']``.
The third part in the configuration file defines the variables that are supposed to be cmorized.
The actual cmorizing script ``mte.py`` consists of a header with
information on where and how to download the data, and noting the last access
of the data webpage.
The main body of the CMORizer script must contain a function called
.. code-block:: python
def cmorization(in_dir, out_dir, cfg, cfg_user, start_date, end_date):
with this exact call signature. Here, ``in_dir`` corresponds to the input
directory of the raw files, ``out_dir`` to the output directory of final
reformatted data set, ``cfg`` to the dataset-specific configuration file,
``cfg_user`` to the user configuration file, ``start_date`` to the start
of the period to format, and ``end_date`` to the end of the period to format.
If not needed, the last three arguments can be ignored using underscores.
The return value of this function is ignored. All
the work, i.e. loading of the raw files, processing them and saving the final
output, has to be performed inside its body. To simplify this process, ESMValTool
provides a set of predefined utilities.py_, which can be imported into your CMORizer
by
.. code-block:: python
from esmvaltool.cmorizers.data import utilities as utils
Apart from a function to easily save data, this module contains different kinds
of small fixes to the data attributes, coordinates, and metadata which are
necessary for the data field to be CMOR-compliant.
Note that this specific CMORizer script contains several subroutines in order
to make the code clearer and more readable (we strongly recommend to follow
that code style). For example, the function ``_get_filepath`` converts the raw
filepath to the correct one and the function ``_extract_variable`` extracts and
saves a single variable from the raw data.
.. _utilities.py: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/utilities.py
4.2 Cmorizer script written in NCL
----------------------------------
Find here an example of a cmorizing script, written for the ``ESACCI XCH4``
dataset that is available on the Copernicus Climate Data Store:
`cds_xch4.ncl
`_.
The first part of the script collects all the information about the dataset
that are necessary to write the filename correctly and to understand which
variable is of interest here. Please make sure to provide the correct
information for following key words: DIAG_SCRIPT, VAR, NAME, MIP, FREQ,
CMOR_TABLE.
- **Note:** the fields ``VAR``, ``NAME``, ``MIP`` and ``FREQ`` all ask for one
or more entries. If more than one entry is provided, make sure that the order
of the entries is the same for all four fields! (for example, that the first
entry in all four fields describe the variable ``xch4`` that you would like
to extract);
- **Note:** some functions in the script are NCL-specific and are available
through the loading of the script interface.ncl_. There are similar
functions available for python scripts.
.. _interface.ncl: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/formatters/interface.ncl
.. _utilities.ncl: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/formatters/utilities.ncl
In the second part of the script each variable defined in ``VAR`` is separately
extracted from the original data file and processed. Most parts of the code are
commented, and therefore it should be easy to follow. ESMValTool provides a set
of predefined utilities.ncl_, which are imported by default into your CMORizer.
This module contains different kinds of small fixes to the data attributes,
coordinates, and metadata which are necessary for the data field to be
CMOR-compliant.
5. Run the cmorizing script
===========================
The cmorizing script for the given dataset can be run with:
.. code-block:: console
esmvaltool data format --config_file
The options ``--start`` and ``--end`` can be added to the command above to
restrict the formatting of raw data to a time range. They will be ignored if a specific dataset
does not support it (i.e. because it is provided as a single file). Valid formats are
``YYYY``, ``YYYYMM`` and ``YYYYMMDD``.
.. note::
The output path given in the configuration file is the path where
your cmorized dataset will be stored. The ESMValTool will create a folder
with the correct tier information
(see Section `2. Edit your configuration file`_) if that tier folder is not
already available, and then a folder named after the dataset.
In this folder the cmorized data set will be stored as a NetCDF file.
The cmorized dataset will be automatically moved to the correct tier
subfolder of your OBS or OBS6 directory if the option
``--install=True`` is used in the command above and no such directory
was already created.
If your run was successful, one or more NetCDF files are produced in your
output directory.
If a downloading script is available for the dataset, the downloading and
the cmorizing scripts can be run in a single command with:
.. code-block:: console
esmvaltool data prepare --config_file
Note that options from the ```esmvaltool data download`` and
``esmvaltool data format`` commands can be passed to the above command.
6. Naming convention of the observational data files
====================================================
For the ESMValTool to be able to read the observations from the NetCDF file,
the file name needs a very specific structure and order of information parts
(very similar to the naming convention for observations in ESMValTool
v1.0). The file name will be automatically correctly created if a cmorizing
script has been used to create the netCDF file.
The correct structure of an observational data set is defined in
`config-developer.yml
`_,
and looks like the following:
.. code-block:: console
OBS_[dataset]_[type]_[version]_[mip]_[short_name]_YYYYMM-YYYYMM.nc
For the example of the ``CDS-XCH4`` data set, the correct structure of the
file name looks then like this:
.. code-block:: console
OBS_CDS-XCH4_sat_L3_Amon_xch4_200301-201612.nc
The different parts of the name are explained in more detail here:
- OBS: describes what kind of data can be expected in the file, in this case
``observations``;
- CDS-XCH4: that is the name of the dataset. It has been named this way for
illustration purposes (so that everybody understands it is the xch4 dataset
downloaded from the CDS), but a better name would indeed be ``ESACCI-XCH4``
since it is a ESA-CCI dataset;
- sat: describes the source of the data, here we are looking at satellite data
(therefore ``sat``), could also be ``reanaly`` for reanalyses;
- L3: describes the version of the dataset:
- Amon: is the information in which ``mip`` the variable is to be expected, and
what kind of temporal resolution it has; here we expect ``xch4`` to be part
of the atmosphere (``A``) and we have the dataset in a monthly resolution
(``mon``);
- xch4: Is the name of the variable. Each observational data file is supposed
to only include one variable per file;
- 200301-201812: Is the period the dataset spans with ``200301`` being the
start year and month, and ``201812`` being the end year and month;
.. note::
There is a different naming convention for ``obs4MIPs`` data (see the exact
specifications for the obs4MIPs data file naming convention in the
``config-developer.yml`` file).
7. Test the cmorized dataset
======================================
To verify that the cmorized data file is indeed correctly formatted, you can
run a dedicated test recipe, that does not include any diagnostic, but only
reads in the data file and has it processed in the preprocessor. Such a recipe
is called ``recipes/examples/recipe_check_obs.yml``. You just need to add a
diagnostic for your dataset following the existing entries.
Only the diagnostic of interest needs to be run, the others should be commented
out for testing.