CMOR functions

CMOR module.

Checking compliance

Module for checking iris cubes against their CMOR definitions.

Classes:

CMORCheck(cube, var_info[, frequency, ...])

Class used to check the CMOR-compliance of the data.

CheckLevels(value)

Level of strictness of the checks.

Exceptions:

CMORCheckError

Exception raised when a cube does not pass the CMORCheck.

Functions:

cmor_check(cube, cmor_table, mip, ...)

Check if cube conforms to variable's CMOR definition.

cmor_check_data(cube, cmor_table, mip, ...)

Check if data conforms to variable's CMOR definition.

cmor_check_metadata(cube, cmor_table, mip, ...)

Check if metadata conforms to variable's CMOR definition.

class esmvalcore.cmor.check.CMORCheck(cube, var_info, frequency=None, fail_on_error=False, check_level=CheckLevels.DEFAULT, automatic_fixes=False)[source]

Bases: object

Class used to check the CMOR-compliance of the data.

It can also fix some minor errors and does some minor data homogeneization:

Parameters
  • cube (iris.cube.Cube:) – Iris cube to check.

  • var_info (variables_info.VariableInfo) – Variable info to check.

  • frequency (str) – Expected frequency for the data.

  • fail_on_error (bool) – If true, CMORCheck stops on the first error. If false, it collects all possible errors before stopping.

  • automatic_fixes (bool) – If True, CMORCheck will try to apply automatic fixes for any detected error, if possible.

  • check_level (CheckLevels) – Level of strictness of the checks.

frequency

Expected frequency for the data.

Type

str

Attributes:

ALTERNATIVE_GENERIC_LEV_COORDS

Methods:

check_data([logger])

Check the cube data.

check_metadata([logger])

Check the cube metadata.

has_debug_messages()

Check if there are reported debug messages.

has_errors()

Check if there are reported errors.

has_warnings()

Check if there are reported warnings.

report(level, message, *args)

Report a message from the checker.

report_critical(message, *args)

Report an error.

report_debug_message(message, *args)

Report a debug message.

report_debug_messages()

Report detected debug messages to the given logger.

report_error(message, *args)

Report a normal error.

report_errors()

Report detected errors.

report_warning(message, *args)

Report a warning level error.

report_warnings()

Report detected warnings to the given logger.

ALTERNATIVE_GENERIC_LEV_COORDS = {'alevel': {'CMIP5': ['alt40', 'plevs'], 'CMIP6': ['alt16', 'plev3'], 'obs4MIPs': ['alt16', 'plev3']}, 'zlevel': {'CMIP3': ['pressure']}}
check_data(logger=None)[source]

Check the cube data.

Performs all the tests that require to have the data in memory. Assumes that metadata is correct, so you must call check_metadata prior to this.

It will also report some warnings in case of minor errors.

Parameters

logger (logging.Logger) – Given logger.

Raises

CMORCheckError – If errors are found. If fail_on_error attribute is set to True, raises as soon as an error is detected. If set to False, it perform all checks and then raises.

check_metadata(logger=None)[source]

Check the cube metadata.

Perform all the tests that do not require to have the data in memory.

It will also report some warnings in case of minor errors and homogenize some data:

  • Equivalent calendars will all default to the same name.

  • Time units will be set to days since 1850-01-01

Parameters

logger (logging.Logger) – Given logger.

Raises

CMORCheckError – If errors are found. If fail_on_error attribute is set to True, raises as soon as an error is detected. If set to False, it perform all checks and then raises.

has_debug_messages()[source]

Check if there are reported debug messages.

Returns

True if there are pending debug messages, False otherwise.

Return type

bool

has_errors()[source]

Check if there are reported errors.

Returns

True if there are pending errors, False otherwise.

Return type

bool

has_warnings()[source]

Check if there are reported warnings.

Returns

True if there are pending warnings, False otherwise.

Return type

bool

report(level, message, *args)[source]

Report a message from the checker.

Parameters
  • level (CheckLevels) – Message level

  • message (str) – Message to report

  • args – String format args for the message

Raises

CMORCheckError – If fail on error is set, it is thrown when registering an error message

report_critical(message, *args)[source]

Report an error.

If fail_on_error is set to True, raises automatically. If fail_on_error is set to False, stores it for later reports.

Parameters
  • message (str: unicode) – Message for the error.

  • *args – arguments to format the message string.

report_debug_message(message, *args)[source]

Report a debug message.

Parameters
  • message (str: unicode) – Message for the debug logger.

  • *args – arguments to format the message string

report_debug_messages()[source]

Report detected debug messages to the given logger.

Parameters

logger (logging.Logger) – Given logger.

report_error(message, *args)[source]

Report a normal error.

Parameters
  • message (str: unicode) – Message for the error.

  • *args – arguments to format the message string.

report_errors()[source]

Report detected errors.

Raises

CMORCheckError – If any errors were reported before calling this method.

report_warning(message, *args)[source]

Report a warning level error.

Parameters
  • message (str: unicode) – Message for the warning.

  • *args – arguments to format the message string.

report_warnings()[source]

Report detected warnings to the given logger.

Parameters

logger (logging.Logger) – Given logger

exception esmvalcore.cmor.check.CMORCheckError[source]

Bases: Exception

Exception raised when a cube does not pass the CMORCheck.

args
with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

class esmvalcore.cmor.check.CheckLevels(value)

Bases: enum.IntEnum

Level of strictness of the checks.

- DEBUG
Type

Report any debug message that the checker wants to communicate.

- STRICT
Type

Fail if there are warnings regarding compliance of CMOR standards.

- DEFAULT
Type

Fail if cubes present any discrepancy with CMOR standards.

- RELAXED
Type

Fail if cubes present severe discrepancies with CMOR standards.

- IGNORE
Type

Do not fail for any discrepancy with CMOR standards.

Attributes:

DEBUG

DEFAULT

IGNORE

RELAXED

STRICT

DEBUG = 1
DEFAULT = 3
IGNORE = 5
RELAXED = 4
STRICT = 2
esmvalcore.cmor.check.cmor_check(cube, cmor_table, mip, short_name, frequency, check_level)[source]

Check if cube conforms to variable’s CMOR definition.

Equivalent to calling cmor_check_metadata and cmor_check_data consecutively.

Parameters
  • cube (iris.cube.Cube) – Data cube to check.

  • cmor_table (str) – CMOR definitions to use.

  • mip – Variable’s mip.

  • short_name (str) – Variable’s short name.

  • frequency (str) – Data frequency.

  • check_level (enum.IntEnum) – Level of strictness of the checks.

esmvalcore.cmor.check.cmor_check_data(cube, cmor_table, mip, short_name, frequency, check_level=CheckLevels.DEFAULT)[source]

Check if data conforms to variable’s CMOR definition.

The checks performed at this step require the data in memory.

Parameters
  • cube (iris.cube.Cube) – Data cube to check.

  • cmor_table (str) – CMOR definitions to use.

  • mip – Variable’s mip.

  • short_name (str) – Variable’s short name

  • frequency (str) – Data frequency

  • check_level (CheckLevels) – Level of strictness of the checks.

esmvalcore.cmor.check.cmor_check_metadata(cube, cmor_table, mip, short_name, frequency, check_level=CheckLevels.DEFAULT)[source]

Check if metadata conforms to variable’s CMOR definition.

None of the checks at this step will force the cube to load the data.

Parameters
  • cube (iris.cube.Cube) – Data cube to check.

  • cmor_table (str) – CMOR definitions to use.

  • mip – Variable’s mip.

  • short_name (str) – Variable’s short name.

  • frequency (str) – Data frequency.

  • check_level (CheckLevels) – Level of strictness of the checks.

Automatically fixing issues

Apply automatic fixes for known errors in cmorized data.

All functions in this module will work even if no fixes are available for the given dataset. Therefore is recommended to apply them to all variables to be sure that all known errors are fixed.

Functions:

fix_data(cube, short_name, project, dataset, mip)

Fix cube data if fixes add present and check it anyway.

fix_file(file, short_name, project, dataset, ...)

Fix files before ESMValTool can load them.

fix_metadata(cubes, short_name, project, ...)

Fix cube metadata if fixes are required and check it anyway.

esmvalcore.cmor.fix.fix_data(cube, short_name, project, dataset, mip, frequency=None, check_level=CheckLevels.DEFAULT, **extra_facets)[source]

Fix cube data if fixes add present and check it anyway.

This method assumes that metadata is already fixed and checked.

This method collects all the relevant fixes for a given variable, applies them and checks resulting cube (or the original if no fixes were needed) metadata to ensure that it complies with the standards of its project CMOR tables.

Parameters
  • cube (iris.cube.Cube) – Cube to fix

  • short_name (str) – Variable’s short name

  • project (str) –

  • dataset (str) –

  • mip (str) – Variable’s MIP

  • frequency (str, optional) – Variable’s data frequency, if available

  • check_level (CheckLevels) – Level of strictness of the checks. Set to default.

  • **extra_facets (dict, optional) – Extra facets are mainly used for data outside of the big projects like CMIP, CORDEX, obs4MIPs. For details, see Extra Facets.

Returns

Fixed and checked cube

Return type

iris.cube.Cube

Raises

CMORCheckError – If the checker detects errors in the data that it can not fix.

esmvalcore.cmor.fix.fix_file(file, short_name, project, dataset, mip, output_dir, **extra_facets)[source]

Fix files before ESMValTool can load them.

This fixes are only for issues that prevent iris from loading the cube or that cannot be fixed after the cube is loaded.

Original files are not overwritten.

Parameters
  • file (str) – Path to the original file

  • short_name (str) – Variable’s short name

  • project (str) –

  • dataset (str) –

  • output_dir (str) – Output directory for fixed files

  • **extra_facets (dict, optional) – Extra facets are mainly used for data outside of the big projects like CMIP, CORDEX, obs4MIPs. For details, see Extra Facets.

Returns

Path to the fixed file

Return type

str

esmvalcore.cmor.fix.fix_metadata(cubes, short_name, project, dataset, mip, frequency=None, check_level=CheckLevels.DEFAULT, **extra_facets)[source]

Fix cube metadata if fixes are required and check it anyway.

This method collects all the relevant fixes for a given variable, applies them and checks the resulting cube (or the original if no fixes were needed) metadata to ensure that it complies with the standards of its project CMOR tables.

Parameters
  • cubes (iris.cube.CubeList) – Cubes to fix

  • short_name (str) – Variable’s short name

  • project (str) –

  • dataset (str) –

  • mip (str) – Variable’s MIP

  • frequency (str, optional) – Variable’s data frequency, if available

  • check_level (CheckLevels) – Level of strictness of the checks. Set to default.

  • **extra_facets (dict, optional) – Extra facets are mainly used for data outside of the big projects like CMIP, CORDEX, obs4MIPs. For details, see Extra Facets.

Returns

Fixed and checked cube

Return type

iris.cube.Cube

Raises

CMORCheckError – If the checker detects errors in the metadata that it can not fix.

Functions for fixing issues

Functions for fixing specific issues with datasets.

Functions:

add_altitude_from_plev(cube)

Add altitude coordinate from pressure level coordinate.

add_plev_from_altitude(cube)

Add pressure level coordinate from altitude coordinate.

esmvalcore.cmor.fixes.add_altitude_from_plev(cube)[source]

Add altitude coordinate from pressure level coordinate.

Parameters

cube (iris.cube.Cube) – Input cube.

Raises

ValueErrorcube does not contain coordinate air_pressure.

esmvalcore.cmor.fixes.add_plev_from_altitude(cube)[source]

Add pressure level coordinate from altitude coordinate.

Parameters

cube (iris.cube.Cube) – Input cube.

Raises

ValueErrorcube does not contain coordinate altitude.

Using CMOR tables

CMOR information reader for ESMValTool.

Read variable information from CMOR 2 and CMOR 3 tables and make it easily available for the other components of ESMValTool

Classes:

CMIP3Info(cmor_tables_path[, default, ...])

Class to read CMIP3-like data request.

CMIP5Info(cmor_tables_path[, default, ...])

Class to read CMIP5-like data request.

CMIP6Info(cmor_tables_path[, default, ...])

Class to read CMIP6-like data request.

CoordinateInfo(name)

Class to read and store coordinate information.

CustomInfo([cmor_tables_path])

Class to read custom var info for ESMVal.

InfoBase(default, alt_names, strict)

Base class for all table info classes.

JsonInfo()

Base class for the info classes.

TableInfo(*args, **kwargs)

Container class for storing a CMOR table.

VariableInfo(table_type, short_name)

Class to read and store variable information.

Data:

CMOR_TABLES

CMOR info objects.

Functions:

get_var_info(project, mip, short_name)

Get variable information.

read_cmor_tables([cfg_developer])

Read cmor tables required in the configuration.

class esmvalcore.cmor.table.CMIP3Info(cmor_tables_path, default=None, alt_names=None, strict=True)[source]

Bases: esmvalcore.cmor.table.CMIP5Info

Class to read CMIP3-like data request.

Parameters
  • cmor_tables_path (str) – Path to the folder containing the Tables folder with the json files

  • default (object) – Default table to look variables on if not found

  • strict (bool) – If False, will look for a variable in other tables if it can not be found in the requested one

Methods:

get_table(table)

Search and return the table info.

get_variable(table_name, short_name[, derived])

Search and return the variable info.

get_table(table)

Search and return the table info.

Parameters

table (str) – Table name

Returns

Return the TableInfo object for the requested table if found, returns None if not

Return type

TableInfo

get_variable(table_name, short_name, derived=False)

Search and return the variable info.

Parameters
  • table_name (str) – Table name

  • short_name (str) – Variable’s short name

  • derived (bool, optional) – Variable is derived. Info retrieval for derived variables always look on the default tables if variable is not find in the requested table

Returns

Return the VariableInfo object for the requested variable if found, returns None if not

Return type

VariableInfo

class esmvalcore.cmor.table.CMIP5Info(cmor_tables_path, default=None, alt_names=None, strict=True)[source]

Bases: esmvalcore.cmor.table.InfoBase

Class to read CMIP5-like data request.

Parameters
  • cmor_tables_path (str) – Path to the folder containing the Tables folder with the json files

  • default (object) – Default table to look variables on if not found

  • strict (bool) – If False, will look for a variable in other tables if it can not be found in the requested one

Methods:

get_table(table)

Search and return the table info.

get_variable(table_name, short_name[, derived])

Search and return the variable info.

get_table(table)[source]

Search and return the table info.

Parameters

table (str) – Table name

Returns

Return the TableInfo object for the requested table if found, returns None if not

Return type

TableInfo

get_variable(table_name, short_name, derived=False)

Search and return the variable info.

Parameters
  • table_name (str) – Table name

  • short_name (str) – Variable’s short name

  • derived (bool, optional) – Variable is derived. Info retrieval for derived variables always look on the default tables if variable is not find in the requested table

Returns

Return the VariableInfo object for the requested variable if found, returns None if not

Return type

VariableInfo

class esmvalcore.cmor.table.CMIP6Info(cmor_tables_path, default=None, alt_names=None, strict=True, default_table_prefix='')[source]

Bases: esmvalcore.cmor.table.InfoBase

Class to read CMIP6-like data request.

This uses CMOR 3 json format

Parameters
  • cmor_tables_path (str) – Path to the folder containing the Tables folder with the json files

  • default (object) – Default table to look variables on if not found

  • strict (bool) – If False, will look for a variable in other tables if it can not be found in the requested one

Methods:

get_table(table)

Search and return the table info.

get_variable(table_name, short_name[, derived])

Search and return the variable info.

get_table(table)[source]

Search and return the table info.

Parameters

table (str) – Table name

Returns

Return the TableInfo object for the requested table if found, returns None if not

Return type

TableInfo

get_variable(table_name, short_name, derived=False)

Search and return the variable info.

Parameters
  • table_name (str) – Table name

  • short_name (str) – Variable’s short name

  • derived (bool, optional) – Variable is derived. Info retrieval for derived variables always look on the default tables if variable is not find in the requested table

Returns

Return the VariableInfo object for the requested variable if found, returns None if not

Return type

VariableInfo

esmvalcore.cmor.table.CMOR_TABLES: Dict[str, Type[esmvalcore.cmor.table.InfoBase]] = {'CMIP3': <esmvalcore.cmor.table.CMIP3Info object>, 'CMIP5': <esmvalcore.cmor.table.CMIP5Info object>, 'CMIP6': <esmvalcore.cmor.table.CMIP6Info object>, 'CORDEX': <esmvalcore.cmor.table.CMIP5Info object>, 'EMAC': <esmvalcore.cmor.table.CMIP6Info object>, 'ICON': <esmvalcore.cmor.table.CMIP6Info object>, 'IPSLCM': <esmvalcore.cmor.table.CMIP6Info object>, 'OBS': <esmvalcore.cmor.table.CMIP5Info object>, 'OBS6': <esmvalcore.cmor.table.CMIP6Info object>, 'ana4mips': <esmvalcore.cmor.table.CMIP5Info object>, 'custom': <esmvalcore.cmor.table.CustomInfo object>, 'native6': <esmvalcore.cmor.table.CMIP6Info object>, 'obs4MIPs': <esmvalcore.cmor.table.CMIP6Info object>}

CMOR info objects.

Type

dict of str, obj

class esmvalcore.cmor.table.CoordinateInfo(name)[source]

Bases: esmvalcore.cmor.table.JsonInfo

Class to read and store coordinate information.

Attributes:

axis

Axis

generic_lev_name

Generic level name

long_name

Long name

must_have_bounds

Whether bounds are required on this dimension

out_name

Out name

requested

Values requested

standard_name

Standard name

stored_direction

Direction in which the coordinate increases

units

Units

valid_max

Maximum allowed value

valid_min

Minimum allowed value

value

Coordinate value

var_name

Short name

Methods:

read_json(json_data)

Read coordinate information from json.

axis

Axis

generic_lev_name

Generic level name

long_name

Long name

must_have_bounds

Whether bounds are required on this dimension

out_name

Out name

This is the name of the variable in the file

read_json(json_data)[source]

Read coordinate information from json.

Non-present options will be set to empty

Parameters

json_data (dict) – dictionary created by the json reader containing coordinate information

requested

Values requested

standard_name

Standard name

stored_direction

Direction in which the coordinate increases

units

Units

valid_max

Maximum allowed value

valid_min

Minimum allowed value

value

Coordinate value

var_name

Short name

class esmvalcore.cmor.table.CustomInfo(cmor_tables_path=None)[source]

Bases: esmvalcore.cmor.table.CMIP5Info

Class to read custom var info for ESMVal.

Parameters

cmor_tables_path (str or None) – Full path to the table or name for the table if it is present in ESMValTool repository

Methods:

get_table(table)

Search and return the table info.

get_variable(table, short_name[, derived])

Search and return the variable info.

get_table(table)

Search and return the table info.

Parameters

table (str) – Table name

Returns

Return the TableInfo object for the requested table if found, returns None if not

Return type

TableInfo

get_variable(table, short_name, derived=False)[source]

Search and return the variable info.

Parameters
  • table (str) – Table name

  • short_name (str) – Variable’s short name

  • derived (bool, optional) – Variable is derived. Info retrieval for derived variables always look on the default tables if variable is not find in the requested table

Returns

Return the VariableInfo object for the requested variable if found, returns None if not

Return type

VariableInfo

class esmvalcore.cmor.table.InfoBase(default, alt_names, strict)[source]

Bases: object

Base class for all table info classes.

This uses CMOR 3 json format

Parameters
  • default (object) – Default table to look variables on if not found

  • alt_names (list[list[str]]) – List of known alternative names for variables

  • strict (bool) – If False, will look for a variable in other tables if it can not be found in the requested one

Methods:

get_table(table)

Search and return the table info.

get_variable(table_name, short_name[, derived])

Search and return the variable info.

get_table(table)[source]

Search and return the table info.

Parameters

table (str) – Table name

Returns

Return the TableInfo object for the requested table if found, returns None if not

Return type

TableInfo

get_variable(table_name, short_name, derived=False)[source]

Search and return the variable info.

Parameters
  • table_name (str) – Table name

  • short_name (str) – Variable’s short name

  • derived (bool, optional) – Variable is derived. Info retrieval for derived variables always look on the default tables if variable is not find in the requested table

Returns

Return the VariableInfo object for the requested variable if found, returns None if not

Return type

VariableInfo

class esmvalcore.cmor.table.JsonInfo[source]

Bases: object

Base class for the info classes.

Provides common utility methods to read json variables

class esmvalcore.cmor.table.TableInfo(*args, **kwargs)[source]

Bases: dict

Container class for storing a CMOR table.

Methods:

clear()

copy()

fromkeys([value])

Create a new dictionary with keys from iterable and values set to value.

get(key[, default])

Return the value for key if key is in the dictionary, else default.

items()

keys()

pop(k[,d])

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

setdefault(key[, default])

Insert key with a value of default if key is not in the dictionary.

update([E, ]**F)

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

clear() None.  Remove all items from D.
copy() a shallow copy of D
fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)

Return the value for key if key is in the dictionary, else default.

items() a set-like object providing a view on D's items
keys() a set-like object providing a view on D's keys
pop(k[, d]) v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) None.  Update D from dict/iterable E and F.

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values() an object providing a view on D's values
class esmvalcore.cmor.table.VariableInfo(table_type, short_name)[source]

Bases: esmvalcore.cmor.table.JsonInfo

Class to read and store variable information.

Attributes:

coordinates

Coordinates

dimensions

List of dimensions

frequency

Data frequency

long_name

Long name

modeling_realm

Modeling realm

positive

Increasing direction

short_name

Short name

standard_name

Standard name

units

Data units

valid_max

Maximum admitted value

valid_min

Minimum admitted value

Methods:

copy()

Return a shallow copy of VariableInfo.

read_json(json_data, default_freq)

Read variable information from json.

coordinates

Coordinates

This is a dict with the names of the dimensions as keys and CoordinateInfo objects as values.

copy()[source]

Return a shallow copy of VariableInfo.

Returns

Shallow copy of this object

Return type

VariableInfo

dimensions

List of dimensions

frequency

Data frequency

long_name

Long name

modeling_realm

Modeling realm

positive

Increasing direction

read_json(json_data, default_freq)[source]

Read variable information from json.

Non-present options will be set to empty

Parameters
  • json_data (dict) – dictionary created by the json reader containing variable information

  • default_freq (str) – Default frequency to use if it is not defined at variable level

short_name

Short name

standard_name

Standard name

units

Data units

valid_max

Maximum admitted value

valid_min

Minimum admitted value

esmvalcore.cmor.table.get_var_info(project, mip, short_name)[source]

Get variable information.

Parameters
  • project (str) – Dataset’s project.

  • mip (str) – Variable’s cmor table.

  • short_name (str) – Variable’s short name.

esmvalcore.cmor.table.read_cmor_tables(cfg_developer=None)[source]

Read cmor tables required in the configuration.

Parameters

cfg_developer (dict of str) – Parsed config-developer file