Running#

The ESMValCore package provides the esmvaltool command line tool, which can be used to run a recipe.

To list the available commands, run

esmvaltool --help

It is also possible to get help on specific commands, e.g.

esmvaltool run --help

will display the help message with all options for the run command.

To run a recipe, call esmvaltool run with the path to the desired recipe:

esmvaltool run recipe_example.yml

The esmvaltool run recipe_example.yml command will first look if recipe_example.yml is the path to an existing file. If this is the case, it will run that recipe. If you have ESMValTool installed, it will look if the name matches one of the recipes in your ESMValTool installation directory, in the subdirectory recipes and run that.

Note

There is no recipe_example.yml shipped with either ESMValCore or ESMValTool. If you would like to try out the command above, replace recipe_example.yml with the path to an existing recipe, e.g. examples/recipe_python.yml if you have the ESMValTool package installed.

To work with installed recipes, the ESMValTool package provides the esmvaltool recipes command, see Available diagnostics and metrics.

By default, ESMValTool searches for configuration files in ~/.config/esmvaltool. If you’d like to use a custom location, you can specify this via the --config_dir command line argument:

esmvaltool run --config_dir /path/to/custom_config recipe_example.yml

It is also possible to explicitly set configuration options with command line arguments:

esmvaltool run --argument_name argument_value recipe_example.yml

To automatically download the files required to run a recipe from ESGF, use the configuration option search_esgf=when_missing (use local files whenever possible) or search_esgf=always (always search ESGF for latest version of files and only use local data if it is the latest version):

esmvaltool run --search_esgf=when_missing recipe_example.yml

or

esmvaltool run --search_esgf=always recipe_example.yml

This feature is available for projects that are hosted on the ESGF, i.e. CMIP3, CMIP5, CMIP6, CORDEX, and obs4MIPs.

To control the strictness of the CMOR checker and the checks during concatenation on auxiliary coordinates, supplementary variables, and derived coordinates, use the flag --check_level:

esmvaltool run --check_level=relaxed recipe_example.yml

Possible values are:

  • ignore: all errors will be reported as warnings. Concatenation will be performed without checks.

  • relaxed: only fail if there are critical errors. Concatenation will be performed without checks.

  • default: fail if there are any errors.

  • strict: fail if there are any warnings.

To reuse pre-processed files from a previous run of the same recipe, you can use

esmvaltool run recipe_example.yml --resume_from ~/esmvaltool_output/recipe_python_20210930_123907

Multiple directories can be specified for reuse, make sure to quote them:

esmvaltool run recipe_example.yml --resume_from "~/esmvaltool_output/recipe_python_20210930_101007 ~/esmvaltool_output/recipe_python_20210930_123907"

The first preprocessor directory containing the required data will be used.

This feature can be useful when developing new diagnostics, because it avoids the need to re-run the preprocessor. Another potential use case is running the preprocessing part of a recipe on one or more machines that have access to a lot of data and then running the diagnostics on a machine without access to data.

To run only the preprocessor tasks from a recipe, use

esmvaltool run recipe_example.yml --remove_preproc_dir=False --run_diagnostic=False

Note

Only preprocessing tasks that completed successfully can be reused with the --resume_from option. Preprocessing tasks that completed successfully, contain a file called metadata.yml in their output directory.

To run a reduced version of the recipe, usually for testing purpose you can use

esmvaltool run --max_datasets=NDATASETS --max_years=NYEARS recipe_example.yml

In this case, the recipe will limit the number of datasets per variable to NDATASETS and the total amount of years loaded to NYEARS. They can also be used separately. Note that diagnostics may require specific combinations of available data, so use the above two flags at your own risk and for testing purposes only.

To run a recipe, even if some datasets are not available, use

esmvaltool run --skip_nonexistent=True recipe_example.yml

It is also possible to select only specific diagnostics to be run. To tun only one, just specify its name. To provide more than one diagnostic to filter use the syntax ‘diag1 diag2/script1’ or ‘(“diag1”, “diag2/script1”)’ and pay attention to the quotes.

esmvaltool run --diagnostics=diagnostic1 recipe_example.yml

Note

ESMValTool command line interface is created using the Fire python package. This package supports the creation of completion scripts for the Bash and Fish shells. Go to https://google.github.io/python-fire/using-cli/#python-fires-flags to learn how to set up them.

This Page