.. _outputdata:

Output
******

ESMValTool automatically generates a new output directory with every run. The
location is determined by the output_dir option  in the config-user.yml file,
the recipe name, and the date and time, using the the format: ``YYYYMMDD_HHMMSS``.

For instance, a typical output location would be:
``output_directory/recipe_ocean_amoc_20190118_1027/``

This is effectively produced by the combination:
``output_dir/recipe_name_YYYYMMDD_HHMMSS/``

This directory will contain 4 further subdirectories:

1. `Diagnostic output`_ (``work``): A place for any diagnostic script results that are not plots, e.g. files in NetCDF format (depends on the diagnostics).

2. `Plots`_ (``plots``): The location for all the plots, split by individual diagnostics and fields.

3. `Run`_ (``run``): This directory includes all log files, a copy of the recipe, a summary of the resource usage, and the `settings.yml`_ interface files and temporary files created by the diagnostic scripts.

4. `Preprocessed datasets`_ (``preproc``): This directory contains all the preprocessed netcdfs data and the `metadata.yml`_ interface files. Note that by default this directory will be deleted after each run, because most users will only need the results from the diagnostic scripts.

A summary of the output is produced in the file:
``index.html``


Preprocessed datasets
=====================

The preprocessed datasets will be stored to the preproc/ directory.
Each variable in each diagnostic will have its own the `metadata.yml`_
interface files saved in the preproc directory.

If the option ``save_intermediary_cubes`` is set to ``true`` in the
config-user.yml file, then the intermediary cubes will also be saved here.
This option is set to false in the default ``config-user.yml`` file.

If the option ``remove_preproc_dir`` is set to ``true`` in the config-user.yml
file, then the preproc directory will be deleted after the run completes. This
option is set to true in the default  ``config-user.yml`` file.


Run
===

The log files in the run directory are automatically generated by ESMValTool
and create a record of the output messages produced by ESMValTool and they are
saved in the run directory. They can be helpful for debugging or monitoring the
job, but also allow a record of the job output to screen after the job has been
completed.

The run directory will also contain a copy of the recipe and the
`settings.yml`_ file, described below.
The run directory is also where the diagnostics are executed, and may also
contain several temporary files while diagnostics are running.

Diagnostic output
=================

The work/ directory will contain all files that are output at the diagnostic
stage. Ie, the model data is preprocessed by ESMValTool and stored in the
preproc/ directory. These files are opened by the diagnostic script, then some
processing is applied. Once the diagnostic level processing has been applied,
the results should be saved to the work directory.


Plots
=====

The plots directory is where diagnostics save their output figures. These
plots are saved in the format requested by the option `output_file_type` in the
config-user.yml file.


Settings.yml
============

The settings.yml file is automatically generated by ESMValTool. Each diagnostic
will produce a unique settings.yml file.

The settings.yml file passes several global level keys to diagnostic scripts.
This includes several flags from the config-user.yml file (such as
'log_level'), several paths which are specific to the
diagnostic being run (such as 'plot_dir' and 'run_dir') and the location on
disk of the metadata.yml file (described below).

.. code-block:: yaml

    input_files:[[...]recipe_ocean_bgc_20190118_134855/preproc/diag_timeseries_scalars/mfo/metadata.yml]
    log_level: debug
    output_file_type: png
    plot_dir: [...]recipe_ocean_bgc_20190118_134855/plots/diag_timeseries_scalars/Scalar_timeseries
    profile_diagnostic: false
    recipe: recipe_ocean_bgc.yml
    run_dir: [...]recipe_ocean_bgc_20190118_134855/run/diag_timeseries_scalars/Scalar_timeseries
    script: Scalar_timeseries
    version: 2.0a1
    work_dir: [...]recipe_ocean_bgc_20190118_134855/work/diag_timeseries_scalars/Scalar_timeseries

The first item in the settings file will be a list of `Metadata.yml`_ files.
There is a metadata.yml file generated for each field in each diagnostic.


.. _metadata_yml:

Metadata.yml
============

The metadata.yml files is automatically generated by ESMValTool. Along with the
settings.yml file, it passes all the paths, boolean flags, and additional
arguments that your diagnostic needs to know in order to run.

The metadata is loaded from cfg as a dictionairy object in python diagnostics.

Here is an example metadata.yml file:

.. code-block:: yaml

  ?
    [...]/recipe_ocean_bgc_20190118_134855/preproc/diag_timeseries_scalars/mfo/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_TO0M_mfo_2002-2004.nc
    : cmor_table: CMIP5
    dataset: HadGEM2-ES
    diagnostic: diag_timeseries_scalars
    end_year: 2004
    ensemble: r1i1p1
    exp: historical
    field: TO0M
    filename: [...]recipe_ocean_bgc_20190118_134855/preproc/diag_timeseries_scalars/mfo/CMIP5_HadGEM2-ES_Omon_historical_r1i1p1_TO0M_mfo_2002-2004.nc
    frequency: mon
    institute: [INPE, MOHC]
    long_name: Sea Water Transport
    mip: Omon
    modeling_realm: [ocean]
    preprocessor: prep_timeseries_scalar
    project: CMIP5
    recipe_dataset_index: 0
    short_name: mfo
    standard_name: sea_water_transport_across_line
    start_year: 2002
    units: kg s-1
    variable_group: mfo


As you can see, this is effectively a dictionary with several items including
data paths, metadata and other information.

There are  several tools available in python which are built to read and parse
these files. The tools are available in the shared directory in the diagnostics
directory.