Configuration#

This section describes the config module.

CFG#

Configuration of ESMValCore/Tool is done via CFG object:

>>> from esmvalcore.config import CFG
>>> CFG
Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
        'compress_netcdf': False,
        'config_developer_file': None,
        'drs': {'CMIP5': 'default', 'CMIP6': 'default'},
        'exit_on_warning': False,
        'log_level': 'info',
        'max_parallel_tasks': None,
        'output_dir': PosixPath('/home/user/esmvaltool_output'),
        'output_file_type': 'png',
        'profile_diagnostic': False,
        'remove_preproc_dir': True,
        'rootpath': {'CMIP5': '~/default_inputpath',
                     'CMIP6': '~/default_inputpath',
                     'default': '~/default_inputpath'},
        'save_intermediary_cubes': False)

All configuration parameters are listed here.

CFG is essentially a python dictionary with a few extra functions, similar to matplotlib.rcParams. This means that values can be updated like this:

>>> CFG['output_dir'] = '~/esmvaltool_output'
>>> CFG['output_dir']
PosixPath('/home/user/esmvaltool_output')

Notice that CFG automatically converts the path to an instance of pathlib.Path and expands the home directory. All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:

>>> CFG['output_directory'] = '~/esmvaltool_output'
InvalidConfigParameter: `output_directory` is not a valid config parameter.

Or, if the value entered cannot be converted to the expected type:

>>> CFG['max_parallel_tasks'] = '🐜'
InvalidConfigParameter: Key `max_parallel_tasks`: Could not convert '🐜' to int

CFG is also flexible, so it tries to correct the type of your input if possible:

>>> CFG['max_parallel_tasks'] = '8'  # str
>>> type(CFG['max_parallel_tasks'])
int

By default, the configuration is loaded from YAML files in the user’s home directory at ~/.config/esmvaltool. If set, this can be overwritten with the ESMVALTOOL_CONFIG_DIR environment variable. Defaults for options that are not specified explicitly are listed here. To reload the current configuration object according to these rules, use:

>>> CFG.reload()

To load the configuration object from custom directories, use:

>>> dirs = ['my/default/config', 'my/custom/config']
>>> CFG.load_from_dirs(dirs)

To update the existing configuration object from custom directories, use:

>>> dirs = ['my/default/config', 'my/custom/config']
>>> CFG.update_from_dirs(dirs)

Session#

Recipes and diagnostics will be run in their own directories. This behavior can be controlled via the Session object. A Session must always be initiated from the global CFG object:

>>> session = CFG.start_session(name='my_session')

A Session is very similar to the config. It is also a dictionary, and copies all the keys from the CFG object. At this moment, session is essentially a copy of CFG:

>>> print(session == CFG)
True
>>> session['output_dir'] = '~/my_output_dir'
>>> print(session == CFG)  # False
False

A Session also knows about the directories where the data will stored. The session name is used to prefix the directories.

>>> session.session_dir
/home/user/my_output_dir/my_session_20201203_155821
>>> session.run_dir
/home/user/my_output_dir/my_session_20201203_155821/run
>>> session.work_dir
/home/user/my_output_dir/my_session_20201203_155821/work
>>> session.preproc_dir
/home/user/my_output_dir/my_session_20201203_155821/preproc
>>> session.plot_dir
/home/user/my_output_dir/my_session_20201203_155821/plots

Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from CFG.

API reference#

Configuration module.

esmvalcore.config.CFG#

Global ESMValCore configuration object of type esmvalcore.config.Config.

By default, this will be loaded from YAML files in the user configuration directory (by default ~/.config/esmvaltool, but this can be changed with the ESMVALTOOL_CONFIG_DIR environment variable) similar to the way Dask handles configuration.

Classes:

`Config`(args, *kwargs)	ESMValTool configuration object.
`Session`(config[, name])	Container class for session configuration and directory information.

class esmvalcore.config.Config(*args, **kwargs)[source]#

ESMValTool configuration object.

Do not instantiate this class directly, but use esmvalcore.config.CFG instead.

Methods:

`load_from_dirs`(dirs)	Clear and load configuration object from directories.
`load_from_file`([filename])	Load user configuration from the given file.
`reload`()	Clear and reload the configuration object.
`start_session`(name)	Start a new session from this configuration object.
`update_from_dirs`(dirs)	Update configuration object from directories.

load_from_dirs(dirs: Iterable[str | Path]) → None[source]#

Clear and load configuration object from directories.

This searches for all YAML files within the given directories and merges them together using dask.config.collect(). Nested objects are properly considered; see dask.config.update() for details. Values in the latter directories are preferred to those in the former.

Options that are not explicitly specified via YAML files are set to the default values.

Note

Just like dask.config.collect(), this silently ignores non-existing directories.

Parameters:: dirs (Iterable[str | Path]) – A list of directories to search for YAML configuration files.
Raises:: esmvalcore.exceptions.InvalidConfigParameter – Invalid configuration option given.
Return type:: None

load_from_file(filename: PathLike | str | None = None) → None[source]#

Load user configuration from the given file.

Deprecated since version 2.12.0: This method has been deprecated in ESMValCore version 2.14.0 and is scheduled for removal in version 2.14.0. Please use CFG.load_from_dirs() instead.

Parameters:: filename (PathLike | str | None) – YAML file to load.
Return type:: None

reload() → None[source]#

Clear and reload the configuration object.

This will read all YAML files in the user configuration directory (by default ~/.config/esmvaltool, but this can be changed with the ESMVALTOOL_CONFIG_DIR environment variable) and merges them together using dask.config.collect(). Nested objects are properly considered; see dask.config.update() for details.

Options that are not explicitly specified via YAML files are set to the default values.

Note

If the user configuration directory does not exist, this will be silently ignored.

Raises:: esmvalcore.exceptions.InvalidConfigParameter – Invalid configuration option given.
Return type:: None

start_session(name: str) → Session[source]#

Start a new session from this configuration object.

Parameters:: name (str) – Name of the session.
Return type:: Session

update_from_dirs(dirs: Iterable[str | Path]) → None[source]#

Update configuration object from directories.

This will first search for all YAML files within the given directories and merge them together using dask.config.collect() (if identical values are provided in multiple files, the value from the last file will be used). Then, the current configuration is merged with these new configuration options using dask.config.merge() (new values are preferred over old values). Nested objects are properly considered; see dask.config.update() for details.

Note

Just like dask.config.collect(), this silently ignores non-existing directories.

Parameters:: dirs (Iterable[str | Path]) – A list of directories to search for YAML configuration files.
Raises:: esmvalcore.exceptions.InvalidConfigParameter – Invalid configuration option given.
Return type:: None

class esmvalcore.config.Session(config: dict, name: str = 'session')[source]#

Container class for session configuration and directory information.

Do not instantiate this class directly, but use CFG.start_session instead.

Parameters:

config (dict) – Dictionary with configuration settings.
name (str) – Name of the session to initialize, for example, the name of the recipe (default=’session’).

Attributes:

`cmor_log`	Return CMOR log file.
`config_dir`	Return user config directory.
`main_log`	Return main log file.
`main_log_debug`	Return main log debug file.
`plot_dir`	Return plot directory.
`preproc_dir`	Return preproc directory.
`relative_cmor_log`
`relative_main_log`
`relative_main_log_debug`
`relative_plot_dir`
`relative_preproc_dir`
`relative_run_dir`
`relative_work_dir`
`run_dir`	Return run directory.
`session_dir`	Return session directory.
`work_dir`	Return work directory.

Methods:

set_session_name([name])

Set the name for the session.

property cmor_log#: Return CMOR log file.

property config_dir#: Return user config directory.

Deprecated since version 2.12.0: This attribute has been deprecated in ESMValCore version 2.12.0 and is scheduled for removal in version 2.14.0.

property main_log#: Return main log file.

property main_log_debug#: Return main log debug file.

property plot_dir#: Return plot directory.

property preproc_dir#: Return preproc directory.

relative_cmor_log = PosixPath('run/cmor_log.txt')#

relative_main_log = PosixPath('run/main_log.txt')#

relative_main_log_debug = PosixPath('run/main_log_debug.txt')#

relative_plot_dir = PosixPath('plots')#

relative_preproc_dir = PosixPath('preproc')#

relative_run_dir = PosixPath('run')#

relative_work_dir = PosixPath('work')#

property run_dir#: Return run directory.

property session_dir#: Return session directory.

session_name: str | None#

set_session_name(name: str = 'session')[source]#

Set the name for the session.

The name is used to name the session directory, e.g. session_20201208_132800/. The date is suffixed automatically.

Parameters:: name (str)

property work_dir#: Return work directory.