Diagnostic script interfaces¶
In order to communicate with diagnostic scripts, ESMValCore uses YAML files. The YAML files provided by ESMValCore to the diagnostic script tell the diagnostic script the settings that were provided in the recipe and where to find the pre-processed input data. On the other hand, the YAML file provided by the diagnostic script to ESMValCore tells ESMValCore which pre-processed data was used to create what plots. The latter is optional, but needed for recording provenance.
When ESMValCore (the
esmvaltool command) runs a recipe, it will first find all data and run the default preprocessor steps plus any
additional preprocessing steps defined in the recipe. Next it will run the diagnostic script defined in the recipe
and finally it will store provenance information. Provenance information is stored in the
W3C PROV XML format
and also plotted in an SVG file for human inspection. In addition to provenance information, a caption is also added
to the plots.
Information provided by ESMValCore to the diagnostic script¶
To provide the diagnostic script with the information it needs to run (e.g. location of input data, various settings), the ESMValCore creates a YAML file called settings.yml and provides the path to this file as the first command line argument to the diagnostic script.
The most interesting settings provided in this file are
run_dir: /path/to/recipe_output/run/diagnostic_name/script_name work_dir: /path/to/recipe_output/work/diagnostic_name/script_name plot_dir: /path/to/recipe_output/plots/diagnostic_name/script_name input_files: - /path/to/recipe_output/preproc/diagnostic_name/ta/metadata.yml - /path/to/recipe_output/preproc/diagnostic_name/pr/metadata.yml
Custom settings in the script section of the recipe will also be made available in this file.
There are three directories defined:
run_diruse this for storing temporary files
work_diruse this for storing NetCDF files containing the data used to make a plot
plot_diruse this for storing plots
input_files is a list of YAML files, containing a description of the preprocessed data. Each entry in these
YAML files is a path to a preprocessed file in NetCDF format, with a list of various attributes.
An example preprocessor metadata.yml file could look like this:
? /path/to/recipe_output/preproc/diagnostic_name/pr/CMIP5_GFDL-ESM2G_Amon_historical_r1i1p1_T2Ms_pr_2000-2002.nc : alias: GFDL-ESM2G cmor_table: CMIP5 dataset: GFDL-ESM2G diagnostic: diagnostic_name end_year: 2002 ensemble: r1i1p1 exp: historical filename: /path/to/recipe_output/preproc/diagnostic_name/pr/CMIP5_GFDL-ESM2G_Amon_historical_r1i1p1_T2Ms_pr_2000-2002.nc frequency: mon institute: [NOAA-GFDL] long_name: Precipitation mip: Amon modeling_realm: [atmos] preprocessor: preprocessor_name project: CMIP5 recipe_dataset_index: 1 reference_dataset: MPI-ESM-LR short_name: pr standard_name: precipitation_flux start_year: 2000 units: kg m-2 s-1 variable_group: pr ? /path/to/recipe_output/preproc/diagnostic_name/pr/CMIP5_MPI-ESM-LR_Amon_historical_r1i1p1_T2Ms_pr_2000-2002.nc : alias: MPI-ESM-LR cmor_table: CMIP5 dataset: MPI-ESM-LR diagnostic: diagnostic_name end_year: 2002 ensemble: r1i1p1 exp: historical filename: /path/to/recipe_output/preproc/diagnostic1/pr/CMIP5_MPI-ESM-LR_Amon_historical_r1i1p1_T2Ms_pr_2000-2002.nc frequency: mon institute: [MPI-M] long_name: Precipitation mip: Amon modeling_realm: [atmos] preprocessor: preprocessor_name project: CMIP5 recipe_dataset_index: 2 reference_dataset: MPI-ESM-LR short_name: pr standard_name: precipitation_flux start_year: 2000 units: kg m-2 s-1 variable_group: pr
Information provided by the diagnostic script to ESMValCore¶
After the diagnostic script has finished running, ESMValCore will try to store provenance information. In order to
link the produced files to input data, the diagnostic script needs to store a YAML file called
For output file produced by the diagnostic script, there should be an entry in the
The name of each entry should be the path to the output file.
Each file entry should at least contain the following items
ancestorsa list of input files used to create the plot
captiona caption text for the plot
plot_fileif the diagnostic also created a plot file, e.g. in .png format.
Each file entry can also contain items from the categories defined in the file
The short entries will automatically be replaced by their longer equivalent in the final provenance records.
It is possible to add custom provenance information by adding custom items to entries.
diagnostic_provenance.yml file could look like this
? /path/to/recipe_output/work/diagnostic_name/script_name/CMIP5_GFDL-ESM2G_Amon_historical_r1i1p1_T2Ms_pr_2000-2002_mean.nc : ancestors:[/path/to/recipe_output/preproc/diagnostic_name/pr/CMIP5_GFDL-ESM2G_Amon_historical_r1i1p1_T2Ms_pr_2000-2002.nc] authors: [andela_bouwe, righi_mattia] caption: Average Precipitation between 2000 and 2002 according to GFDL-ESM2G. domains: [global] plot_file: /path/to/recipe_output/plots/diagnostic_name/script_name/CMIP5_GFDL ESM2G_Amon_historical_r1i1p1_T2Ms_pr_2000-2002_mean.png plot_type: zonal references: [acknow_project] statistics: [mean] ? /path/to/recipe_output/work/diagnostic_name/script_name/CMIP5_MPI-ESM-LR_Amon_historical_r1i1p1_T2Ms_pr_2000-2002_mean.nc : ancestors: [/path/to/recipe_output/preproc/diagnostic_name/pr/CMIP5_MPI-ESM-LR_Amon_historical_r1i1p1_T2Ms_pr_2000-2002.nc] authors: [andela_bouwe, righi_mattia] caption: Average Precipitation between 2000 and 2002 according to MPI-ESM-LR. domains: [global] plot_file: /path/to/recipe_output/plots/diagnostic_name/script_name/CMIP5_MPI-ESM-LR_Amon_historical_r1i1p1_T2Ms_pr_2000-2002_mean.png plot_type: zonal references: [acknow_project] statistics: [mean]
You can check whether your diagnostic script successfully provided the provenance information to the ESMValCore by verifying that
for each output file in the
work_dir, a file with the same name, but ending with _provenance.xml is created
any NetCDF files created by your diagnostic script contain a ‘provenance’ global attribute
any PNG plots created by your diagnostic script contain the provenance information in the ‘Image History’ attribute
Note that this information is included automatically by ESMValCore if the diagnostic script provides the required