ESMValTool automatically generates a new output directory with every run. The location is determined by the output_dir option in the config-user.yml file, the recipe name, and the date and time, using the the format: YYYYMMDD_HHMMSS.
For instance, a typical output location would be: output_directory/recipe_ocean_amoc_20190118_1027/
This is effectively produced by the combination: output_dir/recipe_name_YYYYMMDD_HHMMSS/
This directory will contain 4 further subdirectories:
Diagnostic output (work): A place for any diagnostic script results that are not plots, e.g. files in NetCDF format (depends on the diagnostics).
Plots: The location for all the plots, split by individual diagnostics and fields.
Run: This directory includes all log files, a copy of the recipe, a summary of the resource usage, and the settings.yml interface files and temporary files created by the diagnostic scripts.
Preprocessed datasets (preproc): This directory contains all the preprocessed netcdfs data and the metadata.yml interface files. Note that by default this directory will be deleted after each run, because most users will only need the results from the diagnostic scripts.
The preprocessed datasets will be stored to the preproc/ directory. Each variable in each diagnostic will have its own the metadata.yml interface files saved in the preproc directory.
If the option
save_intermediary_cubes is set to
true in the
config-user.yml file, then the intermediary cubes will also be saved here.
This option is set to false in the default
If the option
remove_preproc_dir is set to
true in the config-user.yml
file, then the preproc directory will be deleted after the run completes. This
option is set to true in the default
The log files in the run directory are automatically generated by ESMValTool and create a record of the output messages produced by ESMValTool and they are saved in the run directory. They can be helpful for debugging or monitoring the job, but also allow a record of the job output to screen after the job has been completed.
The run directory will also contain a copy of the recipe and the settings.yml file, described below. The run directory is also where the diagnostics are executed, and may also contain several temporary files while diagnostics are running.
The work/ directory will contain all files that are output at the diagnostic stage. Ie, the model data is preprocessed by ESMValTool and stored in the preproc/ directory. These files are opened by the diagnostic script, then some processing is applied. Once the diagnostic level processing has been applied, the results should be saved to the work directory.
The plots directory is where diagnostics save their output figures. These plots are saved in the format requested by the option output_file_type in the config-user.yml file.
The settings.yml file is automatically generated by ESMValCore. For each diagnostic, a unique settings.yml file will be produced.
The settings.yml file passes several global level keys to diagnostic scripts. This includes several flags from the config-user.yml file (such as ‘write_netcdf’, ‘write_plots’, etc…), several paths which are specific to the diagnostic being run (such as ‘plot_dir’ and ‘run_dir’) and the location on disk of the metadata.yml file (described below).
input_files:[[...]recipe_ocean_bgc_20190118_134855/preproc/diag_timeseries_scalars/mfo/metadata.yml] log_level: debug output_file_type: png plot_dir: [...]recipe_ocean_bgc_20190118_134855/plots/diag_timeseries_scalars/Scalar_timeseries profile_diagnostic: false recipe: recipe_ocean_bgc.yml run_dir: [...]recipe_ocean_bgc_20190118_134855/run/diag_timeseries_scalars/Scalar_timeseries script: Scalar_timeseries version: 2.0a1 work_dir: [...]recipe_ocean_bgc_20190118_134855/work/diag_timeseries_scalars/Scalar_timeseries write_netcdf: true write_plots: true
The first item in the settings file will be a list of Metadata.yml files. There is a metadata.yml file generated for each field in each diagnostic.
The metadata.yml files is automatically generated by ESMValTool. Along with the settings.yml file, it passes all the paths, boolean flags, and additional arguments that your diagnostic needs to know in order to run.
The metadata is loaded from cfg as a dictionary object in python diagnostics.
Here is an example metadata.yml file:
? [...]/recipe_ocean_bgc_20190118_134855/preproc/diag_timeseries_scalars/mfo/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_TO0M_mfo_2002-2004.nc : cmor_table: CMIP5 dataset: HadGEM2-ES diagnostic: diag_timeseries_scalars end_year: 2004 ensemble: r1i1p1 exp: historical field: TO0M filename: [...]recipe_ocean_bgc_20190118_134855/preproc/diag_timeseries_scalars/mfo/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_TO0M_mfo_2002-2004.nc frequency: mon institute: [INPE, MOHC] long_name: Sea Water Transport mip: Omon modeling_realm: [ocean] preprocessor: prep_timeseries_scalar project: CMIP5 recipe_dataset_index: 0 short_name: mfo standard_name: sea_water_transport_across_line start_year: 2002 units: kg s-1 variable_group: mfo
As you can see, this is effectively a dictionary with several items including data paths, metadata and other information.
There are several tools available in python which are built to read and parse these files. The tools are avaialbe in the shared directory in the diagnostics directory.