CMOR functions¶
CMOR module.
Checking compliance¶
Module for checking iris cubes against their CMOR definitions.
Classes
|
Class used to check the CMOR-compliance of the data. |
|
Level of strictness of the checks. |
Exceptions
Exception raised when a cube does not pass the CMORCheck. |
Functions
|
Check if cube conforms to variable’s CMOR definiton. |
|
Check if data conforms to variable’s CMOR definiton. |
|
Check if metadata conforms to variable’s CMOR definiton. |
-
class
esmvalcore.cmor.check.
CMORCheck
(cube, var_info, frequency=None, fail_on_error=False, check_level=<CheckLevels.DEFAULT: 3>, automatic_fixes=False)[source]¶ Bases:
object
Class used to check the CMOR-compliance of the data.
It can also fix some minor errors and does some minor data homogeneization:
- Parameters
cube (iris.cube.Cube:) – Iris cube to check.
var_info (variables_info.VariableInfo) – Variable info to check.
frequency (str) – Expected frequency for the data.
fail_on_error (bool) – If true, CMORCheck stops on the first error. If false, it collects all possible errors before stopping.
automatic_fixes (bool) – If True, CMORCheck will try to apply automatic fixes for any detected error, if possible.
check_level (CheckLevels) – Level of strictness of the checks.
Methods
check_data
([logger])Check the cube data.
check_metadata
([logger])Check the cube metadata.
Check if there are reported debug messages.
Check if there are reported errors.
Check if there are reported warnings.
report
(level, message, *args)Generic method to report a message from the checker
report_critical
(message, *args)Report an error.
report_debug_message
(message, *args)Report a debug message.
Report detected debug messages to the given logger.
report_error
(message, *args)Report a normal error.
Report detected errors.
report_warning
(message, *args)Report a warning level error.
Report detected warnings to the given logger.
-
check_data
(logger=None)[source]¶ Check the cube data.
Performs all the tests that require to have the data in memory. Assumes that metadata is correct, so you must call check_metadata prior to this.
It will also report some warnings in case of minor errors.
- Parameters
logger (logging.Logger) – Given logger.
- Raises
CMORCheckError – If errors are found. If fail_on_error attribute is set to True, raises as soon as an error is detected. If set to False, it perform all checks and then raises.
-
check_metadata
(logger=None)[source]¶ Check the cube metadata.
Perform all the tests that do not require to have the data in memory.
It will also report some warnings in case of minor errors and homogenize some data:
Equivalent calendars will all default to the same name.
Time units will be set to days since 1850-01-01
- Parameters
logger (logging.Logger) – Given logger.
- Raises
CMORCheckError – If errors are found. If fail_on_error attribute is set to True, raises as soon as an error is detected. If set to False, it perform all checks and then raises.
-
has_debug_messages
()[source]¶ Check if there are reported debug messages.
- Returns
True if there are pending debug messages, False otherwise.
- Return type
-
has_errors
()[source]¶ Check if there are reported errors.
- Returns
True if there are pending errors, False otherwise.
- Return type
-
has_warnings
()[source]¶ Check if there are reported warnings.
- Returns
True if there are pending warnings, False otherwise.
- Return type
-
report
(level, message, *args)[source]¶ Generic method to report a message from the checker
- Parameters
level (CheckLevels) – Message level
message (str) – Message to report
args – String format args for the message
- Raises
CMORCheckError – If fail on error is set, it is thrown when registering an error message
-
report_critical
(message, *args)[source]¶ Report an error.
If fail_on_error is set to True, raises automatically. If fail_on_error is set to False, stores it for later reports.
- Parameters
message (str: unicode) – Message for the error.
*args – arguments to format the message string.
-
report_debug_message
(message, *args)[source]¶ Report a debug message.
- Parameters
message (str: unicode) – Message for the debug logger.
*args – arguments to format the message string
-
report_debug_messages
()[source]¶ Report detected debug messages to the given logger.
- Parameters
logger (logging.Logger) – Given logger.
-
report_error
(message, *args)[source]¶ Report a normal error.
- Parameters
message (str: unicode) – Message for the error.
*args – arguments to format the message string.
-
report_errors
()[source]¶ Report detected errors.
- Raises
CMORCheckError – If any errors were reported before calling this method.
-
report_warning
(message, *args)[source]¶ Report a warning level error.
- Parameters
message (str: unicode) – Message for the warning.
*args – arguments to format the message string.
-
report_warnings
()[source]¶ Report detected warnings to the given logger.
- Parameters
logger (logging.Logger) – Given logger
-
exception
esmvalcore.cmor.check.
CMORCheckError
[source]¶ Bases:
Exception
Exception raised when a cube does not pass the CMORCheck.
-
args
¶
-
with_traceback
()¶ Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.
-
-
class
esmvalcore.cmor.check.
CheckLevels
(value)¶ Bases:
enum.IntEnum
Level of strictness of the checks.
Attributes
int([x]) -> integer
int([x]) -> integer
int([x]) -> integer
int([x]) -> integer
int([x]) -> integer
-
- DEBUG
- Type
Report any debug message that the checker wants to communicate.
-
- STRICT
- Type
Fail if there are warnings regarding compliance of CMOR standards.
-
- DEFAULT
- Type
Fail if cubes present any discrepancy with CMOR standards.
-
- RELAXED
- Type
Fail if cubes present severe discrepancies with CMOR standards.
-
- IGNORE
- Type
Do not fail for any discrepancy with CMOR standards.
-
DEBUG
= 1¶
-
DEFAULT
= 3¶
-
IGNORE
= 5¶
-
RELAXED
= 4¶
-
STRICT
= 2¶
-
-
esmvalcore.cmor.check.
cmor_check
(cube, cmor_table, mip, short_name, frequency, check_level)[source]¶ Check if cube conforms to variable’s CMOR definiton.
Equivalent to calling cmor_check_metadata and cmor_check_data consecutively.
- Parameters
cube (iris.cube.Cube) – Data cube to check.
cmor_table (basestring) – CMOR definitions to use.
mip – Variable’s mip.
short_name (basestring) – Variable’s short name.
frequency (basestring) – Data frequency.
check_level (enum.IntEnum) – Level of strictness of the checks.
-
esmvalcore.cmor.check.
cmor_check_data
(cube, cmor_table, mip, short_name, frequency, check_level=<CheckLevels.DEFAULT: 3>)[source]¶ Check if data conforms to variable’s CMOR definiton.
The checks performed at this step require the data in memory.
- Parameters
cube (iris.cube.Cube) – Data cube to check.
cmor_table (basestring) – CMOR definitions to use.
mip – Variable’s mip.
short_name (basestring) – Variable’s short name
frequency (basestring) – Data frequency
check_level (CheckLevels) – Level of strictness of the checks.
-
esmvalcore.cmor.check.
cmor_check_metadata
(cube, cmor_table, mip, short_name, frequency, check_level=<CheckLevels.DEFAULT: 3>)[source]¶ Check if metadata conforms to variable’s CMOR definiton.
None of the checks at this step will force the cube to load the data.
- Parameters
cube (iris.cube.Cube) – Data cube to check.
cmor_table (basestring) – CMOR definitions to use.
mip – Variable’s mip.
short_name (basestring) – Variable’s short name.
frequency (basestring) – Data frequency.
check_level (CheckLevels) – Level of strictness of the checks.
Automatically fixing issues¶
Apply automatic fixes for known errors in cmorized data
All functions in this module will work even if no fixes are available for the given dataset. Therefore is recommended to apply them to all variables to be sure that all known errors are fixed.
Functions
|
Fix cube data if fixes add present and check it anyway. |
|
Fix files before ESMValTool can load them. |
|
Fix cube metadata if fixes are required and check it anyway. |
-
esmvalcore.cmor.fix.
fix_data
(cube, short_name, project, dataset, mip, frequency=None, check_level=<CheckLevels.DEFAULT: 3>)[source]¶ Fix cube data if fixes add present and check it anyway.
This method assumes that metadata is already fixed and checked.
This method collects all the relevant fixes for a given variable, applies them and checks resulting cube (or the original if no fixes were needed) metadata to ensure that it complies with the standards of its project CMOR tables.
- Parameters
cube (iris.cube.Cube) – Cube to fix
short_name (str) – Variable’s short name
project (str) –
dataset (str) –
mip (str) – Variable’s MIP
frequency (str, optional) – Variable’s data frequency, if available
check_level (CheckLevels) – Level of strictness of the checks. Set to default.
- Returns
Fixed and checked cube
- Return type
- Raises
CMORCheckError – If the checker detects errors in the data that it can not fix.
-
esmvalcore.cmor.fix.
fix_file
(file, short_name, project, dataset, mip, output_dir)[source]¶ Fix files before ESMValTool can load them.
This fixes are only for issues that prevent iris from loading the cube or that cannot be fixed after the cube is loaded.
Original files are not overwritten.
-
esmvalcore.cmor.fix.
fix_metadata
(cubes, short_name, project, dataset, mip, frequency=None, check_level=<CheckLevels.DEFAULT: 3>)[source]¶ Fix cube metadata if fixes are required and check it anyway.
This method collects all the relevant fixes for a given variable, applies them and checks the resulting cube (or the original if no fixes were needed) metadata to ensure that it complies with the standards of its project CMOR tables.
- Parameters
cubes (iris.cube.CubeList) – Cubes to fix
short_name (str) – Variable’s short name
project (str) –
dataset (str) –
mip (str) – Variable’s MIP
frequency (str, optional) – Variable’s data frequency, if available
check_level (CheckLevels) – Level of strictness of the checks. Set to default.
- Returns
Fixed and checked cube
- Return type
- Raises
CMORCheckError – If the checker detects errors in the metadata that it can not fix.
Functions for fixing issues¶
Functions for fixing specific issues with datasets.
Functions
|
Add pressure level coordinate from altitude coordinate. |
|
Add factory for |
-
esmvalcore.cmor.fixes.
add_plev_from_altitude
(cube)[source]¶ Add pressure level coordinate from altitude coordinate.
- Parameters
cube (iris.cube.Cube) – Input cube.
- Raises
ValueError –
cube
does not contain coordinatealtitude
.
-
esmvalcore.cmor.fixes.
add_sigma_factory
(cube)[source]¶ Add factory for
atmosphere_sigma_coordinate
.- Parameters
cube (iris.cube.Cube) – Input cube.
- Raises
ValueError –
cube
does not contain coordinateatmosphere_sigma_coordinate
.
Using CMOR tables¶
CMOR information reader for ESMValTool.
Read variable information from CMOR 2 and CMOR 3 tables and make it easily available for the other components of ESMValTool
Classes
|
Class to read CMIP3-like data request. |
|
Class to read CMIP5-like data request. |
|
Class to read CMIP6-like data request. |
|
Class to read and store coordinate information. |
|
Class to read custom var info for ESMVal. |
|
Base class for the info classes. |
|
Container class for storing a CMOR table. |
|
Class to read and store variable information. |
Data
dict of str, obj: CMOR info objects. |
Functions
|
Get variable information. |
|
Read cmor tables required in the configuration. |
-
class
esmvalcore.cmor.table.
CMIP3Info
(cmor_tables_path, default=None, strict=True)[source]¶ Bases:
esmvalcore.cmor.table.CMIP5Info
Class to read CMIP3-like data request.
- Parameters
Methods
get_table
(table)Search and return the table info.
get_variable
(table, short_name[, derived])Search and return the variable info.
-
get_table
(table)¶ Search and return the table info.
- Parameters
table (basestring) – Table name
- Returns
Return the TableInfo object for the requested table if found, returns None if not
- Return type
-
get_variable
(table, short_name, derived=False)¶ Search and return the variable info.
- Parameters
table (basestring) – Table name
short_name (basestring) – Variable’s short name
derived (bool, optional) – Variable is derived. Info retrieval is less strict
- Returns
Return the VariableInfo object for the requested variable if found, returns None if not
- Return type
-
class
esmvalcore.cmor.table.
CMIP5Info
(cmor_tables_path, default=None, strict=True)[source]¶ Bases:
object
Class to read CMIP5-like data request.
- Parameters
Methods
get_table
(table)Search and return the table info.
get_variable
(table, short_name[, derived])Search and return the variable info.
-
get_table
(table)[source]¶ Search and return the table info.
- Parameters
table (basestring) – Table name
- Returns
Return the TableInfo object for the requested table if found, returns None if not
- Return type
-
get_variable
(table, short_name, derived=False)[source]¶ Search and return the variable info.
- Parameters
table (basestring) – Table name
short_name (basestring) – Variable’s short name
derived (bool, optional) – Variable is derived. Info retrieval is less strict
- Returns
Return the VariableInfo object for the requested variable if found, returns None if not
- Return type
-
class
esmvalcore.cmor.table.
CMIP6Info
(cmor_tables_path, default=None, strict=True, default_table_prefix='')[source]¶ Bases:
object
Class to read CMIP6-like data request.
This uses CMOR 3 json format
- Parameters
Methods
get_table
(table)Search and return the table info.
get_variable
(table_name, short_name[, derived])Search and return the variable info.
-
get_table
(table)[source]¶ Search and return the table info.
- Parameters
table (basestring) – Table name
- Returns
Return the TableInfo object for the requested table if found, returns None if not
- Return type
-
get_variable
(table_name, short_name, derived=False)[source]¶ Search and return the variable info.
- Parameters
table_name (basestring) – Table name
short_name (basestring) – Variable’s short name
derived (bool, optional) – Variable is derived. Info retrieval is less strict
- Returns
Return the VariableInfo object for the requested variable if found, returns None if not
- Return type
-
esmvalcore.cmor.table.
CMOR_TABLES
= {'CMIP3': <esmvalcore.cmor.table.CMIP3Info object>, 'CMIP5': <esmvalcore.cmor.table.CMIP5Info object>, 'CMIP6': <esmvalcore.cmor.table.CMIP6Info object>, 'CORDEX': <esmvalcore.cmor.table.CMIP5Info object>, 'EMAC': <esmvalcore.cmor.table.CMIP5Info object>, 'OBS': <esmvalcore.cmor.table.CMIP5Info object>, 'OBS6': <esmvalcore.cmor.table.CMIP6Info object>, 'ana4mips': <esmvalcore.cmor.table.CMIP5Info object>, 'custom': <esmvalcore.cmor.table.CustomInfo object>, 'native6': <esmvalcore.cmor.table.CMIP6Info object>, 'obs4mips': <esmvalcore.cmor.table.CMIP6Info object>}¶ CMOR info objects.
- Type
dict of str, obj
-
class
esmvalcore.cmor.table.
CoordinateInfo
(name)[source]¶ Bases:
esmvalcore.cmor.table.JsonInfo
Class to read and store coordinate information.
Methods
read_json
(json_data)Read coordinate information from json.
Attributes
Axis
Coordinate value
Standard name
Long name
Out name
Short name
Units
Direction in which the coordinate increases
Values requested
Minimum allowed value
Maximum allowed value
Whether bounds are required on this dimension
-
axis
¶ Axis
-
long_name
¶ Long name
-
must_have_bounds
¶ Whether bounds are required on this dimension
-
out_name
¶ Out name
This is the name of the variable in the file
-
read_json
(json_data)[source]¶ Read coordinate information from json.
Non-present options will be set to empty
- Parameters
json_data (dict) – dictionary created by the json reader containing coordinate information
-
requested
¶ Values requested
-
standard_name
¶ Standard name
-
stored_direction
¶ Direction in which the coordinate increases
-
units
¶ Units
-
valid_max
¶ Maximum allowed value
-
valid_min
¶ Minimum allowed value
-
value
¶ Coordinate value
-
var_name
¶ Short name
-
-
class
esmvalcore.cmor.table.
CustomInfo
(cmor_tables_path=None)[source]¶ Bases:
esmvalcore.cmor.table.CMIP5Info
Class to read custom var info for ESMVal.
- Parameters
cmor_tables_path (basestring or None) – Full path to the table or name for the table if it is present in ESMValTool repository
Methods
get_table
(table)Search and return the table info.
get_variable
(table, short_name[, derived])Search and return the variable info.
-
get_table
(table)[source]¶ Search and return the table info.
- Parameters
table (basestring) – Table name
- Returns
Return the TableInfo object for the requested table if found, returns None if not
- Return type
-
get_variable
(table, short_name, derived=False)[source]¶ Search and return the variable info.
- Parameters
table (basestring) – Table name
short_name (basestring) – Variable’s short name
derived (bool, optional) – Variable is derived. Info retrieval is less strict
- Returns
Return the VariableInfo object for the requested variable if found, returns None if not
- Return type
-
class
esmvalcore.cmor.table.
JsonInfo
[source]¶ Bases:
object
Base class for the info classes.
Provides common utility methods to read json variables
-
class
esmvalcore.cmor.table.
TableInfo
(*args, **kwargs)[source]¶ Bases:
dict
Container class for storing a CMOR table.
Methods
clear
()copy
()fromkeys
([value])Create a new dictionary with keys from iterable and values set to value.
get
(key[, default])Return the value for key if key is in the dictionary, else default.
items
()keys
()pop
(k[,d])If key is not found, d is returned if given, otherwise KeyError is raised
popitem
()2-tuple; but raise KeyError if D is empty.
setdefault
(key[, default])Insert key with a value of default if key is not in the dictionary.
update
([E, ]**F)If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
values
()-
clear
() → None. Remove all items from D.¶
-
copy
() → a shallow copy of D¶
-
fromkeys
(value=None, /)¶ Create a new dictionary with keys from iterable and values set to value.
-
get
(key, default=None, /)¶ Return the value for key if key is in the dictionary, else default.
-
items
() → a set-like object providing a view on D’s items¶
-
keys
() → a set-like object providing a view on D’s keys¶
-
pop
(k[, d]) → v, remove specified key and return the corresponding value.¶ If key is not found, d is returned if given, otherwise KeyError is raised
-
popitem
() → (k, v), remove and return some (key, value) pair as a¶ 2-tuple; but raise KeyError if D is empty.
-
setdefault
(key, default=None, /)¶ Insert key with a value of default if key is not in the dictionary.
Return the value for key if key is in the dictionary, else default.
-
update
([E, ]**F) → None. Update D from dict/iterable E and F.¶ If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
-
values
() → an object providing a view on D’s values¶
-
-
class
esmvalcore.cmor.table.
VariableInfo
(table_type, short_name)[source]¶ Bases:
esmvalcore.cmor.table.JsonInfo
Class to read and store variable information.
Methods
copy
()Return a shallow copy of VariableInfo.
read_json
(json_data, default_freq)Read variable information from json.
Attributes
Modeling realm
Short name
Standard name
Long name
Data units
Minimum admitted value
Maximum admitted value
Data frequency
Increasing direction
List of dimensions
Coordinates
-
coordinates
¶ Coordinates
This is a dict with the names of the dimensions as keys and CoordinateInfo objects as values.
-
copy
()[source]¶ Return a shallow copy of VariableInfo.
- Returns
Shallow copy of this object
- Return type
-
dimensions
¶ List of dimensions
-
frequency
¶ Data frequency
-
long_name
¶ Long name
-
modeling_realm
¶ Modeling realm
-
positive
¶ Increasing direction
-
read_json
(json_data, default_freq)[source]¶ Read variable information from json.
Non-present options will be set to empty
-
short_name
¶ Short name
-
standard_name
¶ Standard name
-
units
¶ Data units
-
valid_max
¶ Maximum admitted value
-
valid_min
¶ Minimum admitted value
-