CMOR functions#
CMOR module.
Checking compliance#
Module for checking iris cubes against their CMOR definitions.
Classes:
|
Level of strictness of the checks. |
|
Class used to check the CMOR-compliance of the data. |
Exceptions:
Exception raised when a cube does not pass the CMORCheck. |
Functions:
|
Check if metadata conforms to variable's CMOR definition. |
|
Check if data conforms to variable's CMOR definition. |
|
Check if cube conforms to variable's CMOR definition. |
- class esmvalcore.cmor.check.CheckLevels(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#
Bases:
IntEnum
Level of strictness of the checks.
Attributes:
Report any debug message that the checker wants to communicate.
Fail if there are warnings regarding compliance of CMOR standards.
Fail if cubes present any discrepancy with CMOR standards.
Fail if cubes present severe discrepancies with CMOR standards.
Do not fail for any discrepancy with CMOR standards.
- DEBUG = 1#
Report any debug message that the checker wants to communicate.
- STRICT = 2#
Fail if there are warnings regarding compliance of CMOR standards.
- DEFAULT = 3#
Fail if cubes present any discrepancy with CMOR standards.
- RELAXED = 4#
Fail if cubes present severe discrepancies with CMOR standards.
- IGNORE = 5#
Do not fail for any discrepancy with CMOR standards.
- exception esmvalcore.cmor.check.CMORCheckError[source]#
Bases:
Exception
Exception raised when a cube does not pass the CMORCheck.
- class esmvalcore.cmor.check.CMORCheck(cube, var_info, frequency=None, fail_on_error=False, check_level=CheckLevels.DEFAULT)[source]#
Bases:
object
Class used to check the CMOR-compliance of the data.
- Parameters:
cube (iris.cube.Cube:) – Iris cube to check.
var_info (variables_info.VariableInfo) – Variable info to check.
frequency (str) – Expected frequency for the data. If not given, use the one from the variable information.
fail_on_error (bool) – If true, CMORCheck stops on the first error. If false, it collects all possible errors before stopping.
check_level (CheckLevels) – Level of strictness of the checks.
Methods:
check_metadata
([logger])Check the cube metadata.
check_data
([logger])Check the cube data.
Report detected errors.
Report detected warnings to the given logger.
Report detected debug messages to the given logger.
Check if there are reported errors.
Check if there are reported warnings.
Check if there are reported debug messages.
report
(level, message, *args)Report a message from the checker.
report_critical
(message, *args)Report an error.
report_error
(message, *args)Report a normal error.
report_warning
(message, *args)Report a warning level error.
report_debug_message
(message, *args)Report a debug message.
- check_metadata(logger: Logger | None = None) Cube [source]#
Check the cube metadata.
It will also report some warnings in case of minor errors.
- Parameters:
logger (Logger | None) – Given logger.
- Returns:
Checked cube.
- Return type:
- Raises:
CMORCheckError – If errors are found. If fail_on_error attribute is set to True, raises as soon as an error is detected. If set to False, it perform all checks and then raises.
- check_data(logger: Logger | None = None) Cube [source]#
Check the cube data.
Assumes that metadata is correct, so you must call check_metadata prior to this.
It will also report some warnings in case of minor errors.
- Parameters:
logger (Logger | None) – Given logger.
- Returns:
Checked cube.
- Return type:
- Raises:
CMORCheckError – If errors are found. If fail_on_error attribute is set to True, raises as soon as an error is detected. If set to False, it perform all checks and then raises.
- report_errors()[source]#
Report detected errors.
- Raises:
CMORCheckError – If any errors were reported before calling this method.
- has_errors()[source]#
Check if there are reported errors.
- Returns:
True if there are pending errors, False otherwise.
- Return type:
- has_warnings()[source]#
Check if there are reported warnings.
- Returns:
True if there are pending warnings, False otherwise.
- Return type:
- has_debug_messages()[source]#
Check if there are reported debug messages.
- Returns:
True if there are pending debug messages, False otherwise.
- Return type:
- report(level, message, *args)[source]#
Report a message from the checker.
- Parameters:
level (CheckLevels) – Message level
message (str) – Message to report
args – String format args for the message
- Raises:
CMORCheckError – If fail on error is set, it is thrown when registering an error message
- report_critical(message, *args)[source]#
Report an error.
If fail_on_error is set to True, raises automatically. If fail_on_error is set to False, stores it for later reports.
- Parameters:
message (str: unicode) – Message for the error.
*args – arguments to format the message string.
- report_error(message, *args)[source]#
Report a normal error.
- Parameters:
message (str: unicode) – Message for the error.
*args – arguments to format the message string.
- esmvalcore.cmor.check.cmor_check_metadata(cube: Cube, cmor_table: str, mip: str, short_name: str, frequency: str | None = None, check_level: CheckLevels = CheckLevels.DEFAULT) Cube [source]#
Check if metadata conforms to variable’s CMOR definition.
None of the checks at this step will force the cube to load the data.
- Parameters:
cube (Cube) – Data cube to check.
cmor_table (str) – CMOR definitions to use (i.e., the variable’s project).
mip (str) – Variable’s MIP.
short_name (str) – Variable’s short name.
frequency (str | None) – Data frequency. If not given, use the one from the CMOR table of the variable.
check_level (CheckLevels) – Level of strictness of the checks.
- Returns:
Checked cube.
- Return type:
- esmvalcore.cmor.check.cmor_check_data(cube: Cube, cmor_table: str, mip: str, short_name: str, frequency: str | None = None, check_level: CheckLevels = CheckLevels.DEFAULT) Cube [source]#
Check if data conforms to variable’s CMOR definition.
- Parameters:
cube (Cube) – Data cube to check.
cmor_table (str) – CMOR definitions to use (i.e., the variable’s project).
mip (str) – Variable’s MIP.
short_name (str) – Variable’s short name
frequency (str | None) – Data frequency. If not given, use the one from the CMOR table of the variable.
check_level (CheckLevels) – Level of strictness of the checks.
- Returns:
Checked cube.
- Return type:
- esmvalcore.cmor.check.cmor_check(cube: Cube, cmor_table: str, mip: str, short_name: str, frequency: str | None = None, check_level: CheckLevels = CheckLevels.DEFAULT) Cube [source]#
Check if cube conforms to variable’s CMOR definition.
Equivalent to calling
cmor_check_metadata()
andcmor_check_data()
consecutively.- Parameters:
cube (Cube) – Data cube to check.
cmor_table (str) – CMOR definitions to use (i.e., the variable’s project).
mip (str) – Variable’s MIP.
short_name (str) – Variable’s short name.
frequency (str | None) – Data frequency. If not given, use the one from the CMOR table of the variable.
check_level (CheckLevels) – Level of strictness of the checks.
- Returns:
Checked cube.
- Return type:
Automatically fixing issues#
Apply automatic fixes for known errors in cmorized data.
All functions in this module will work even if no fixes are available for the given dataset. Therefore is recommended to apply them to all variables to be sure that all known errors are fixed.
Functions:
|
Fix cube data if fixes are required. |
|
Fix files before ESMValTool can load them. |
|
Fix cube metadata if fixes are required. |
- esmvalcore.cmor.fix.fix_data(cube: Cube, short_name: str, project: str, dataset: str, mip: str, frequency: str | None = None, session: Session | None = None, **extra_facets) Cube [source]#
Fix cube data if fixes are required.
This method assumes that metadata is already fixed and checked.
This method collects all the relevant fixes (including generic ones) for a given variable and applies them.
- Parameters:
cube (Cube) – Cube to fix.
short_name (str) – Variable’s short name.
project (str) – Project of the dataset.
dataset (str) – Name of the dataset.
mip (str) – Variable’s MIP.
frequency (Optional[str]) – Variable’s data frequency, if available.
session (Optional[Session]) – Current session which includes configuration and directory information.
**extra_facets – Extra facets are mainly used for data outside of the big projects like CMIP, CORDEX, obs4MIPs. For details, see Extra Facets.
- Returns:
Fixed cube.
- Return type:
- esmvalcore.cmor.fix.fix_file(file: Path, short_name: str, project: str, dataset: str, mip: str, output_dir: Path, add_unique_suffix: bool = False, session: Session | None = None, frequency: str | None = None, **extra_facets) str | Path [source]#
Fix files before ESMValTool can load them.
These fixes are only for issues that prevent iris from loading the cube or that cannot be fixed after the cube is loaded.
Original files are not overwritten.
- Parameters:
file (Path) – Path to the original file.
short_name (str) – Variable’s short name.
project (str) – Project of the dataset.
dataset (str) – Name of the dataset.
mip (str) – Variable’s MIP.
output_dir (Path) – Output directory for fixed files.
add_unique_suffix (bool) – Adds a unique suffix to output_dir for thread safety.
session (Optional[Session]) – Current session which includes configuration and directory information.
frequency (Optional[str]) – Variable’s data frequency, if available.
**extra_facets – Extra facets are mainly used for data outside of the big projects like CMIP, CORDEX, obs4MIPs. For details, see Extra Facets.
- Returns:
Path to the fixed file.
- Return type:
str or pathlib.Path
- esmvalcore.cmor.fix.fix_metadata(cubes: Sequence[Cube], short_name: str, project: str, dataset: str, mip: str, frequency: str | None = None, session: Session | None = None, **extra_facets) CubeList [source]#
Fix cube metadata if fixes are required.
This method collects all the relevant fixes (including generic ones) for a given variable and applies them.
- Parameters:
cubes (Sequence[Cube]) – Cubes to fix.
short_name (str) – Variable’s short name.
project (str) – Project of the dataset.
dataset (str) – Name of the dataset.
mip (str) – Variable’s MIP.
frequency (Optional[str]) – Variable’s data frequency, if available.
session (Optional[Session]) – Current session which includes configuration and directory information.
**extra_facets – Extra facets are mainly used for data outside of the big projects like CMIP, CORDEX, obs4MIPs. For details, see Extra Facets.
- Returns:
Fixed cubes.
- Return type:
Functions for fixing issues#
Functions for fixing specific issues with datasets.
Functions:
|
Add altitude coordinate from pressure level coordinate. |
|
Add pressure level coordinate from altitude coordinate. |
|
Get next month and year. |
|
Get bounds for time coordinate. |
- esmvalcore.cmor.fixes.add_altitude_from_plev(cube)[source]#
Add altitude coordinate from pressure level coordinate.
- Parameters:
cube (iris.cube.Cube) – Input cube.
- Raises:
ValueError –
cube
does not contain coordinateair_pressure
.
- esmvalcore.cmor.fixes.add_plev_from_altitude(cube)[source]#
Add pressure level coordinate from altitude coordinate.
- Parameters:
cube (iris.cube.Cube) – Input cube.
- Raises:
ValueError –
cube
does not contain coordinatealtitude
.
- esmvalcore.cmor.fixes.get_next_month(month: int, year: int) tuple[int, int] [source]#
Get next month and year.
- esmvalcore.cmor.fixes.get_time_bounds(time: Coord, freq: str) ndarray [source]#
Get bounds for time coordinate.
For decadal data, use 1 January 5 years before/after the current year. For yearly data, use 1 January of the current year and 1 January of the next year. For monthly data, use the first day of the current month and the first day of the next month. For other frequencies (daily or n-hourly, where n is a divisor of 24), half of the frequency is subtracted/added from the current point in time to get the bounds.
- Parameters:
- Returns:
Time bounds
- Return type:
np.ndarray
- Raises:
NotImplementedError – Non-supported frequency is given.
Using CMOR tables#
CMOR information reader for ESMValTool.
Read variable information from CMOR 2 and CMOR 3 tables and make it easily available for the other components of ESMValTool
Classes:
|
Class to read CMIP3-like data request. |
|
Class to read CMIP5-like data request. |
|
Class to read CMIP6-like data request. |
|
Class to read and store coordinate information. |
|
Class to read custom var info for ESMVal. |
|
Base class for all table info classes. |
|
Base class for the info classes. |
|
Container class for storing a CMOR table. |
|
Class to read and store variable information. |
Data:
CMOR info objects. |
Functions:
|
Get variable information. |
|
Read cmor tables required in the configuration. |
- class esmvalcore.cmor.table.CMIP3Info(cmor_tables_path, default=None, alt_names=None, strict=True)[source]#
Bases:
CMIP5Info
Class to read CMIP3-like data request.
- Parameters:
Methods:
get_table
(table)Search and return the table info.
get_variable
(table_name, short_name[, derived])Search and return the variable information.
- get_table(table)#
Search and return the table info.
- get_variable(table_name: str, short_name: str, derived: bool = False) VariableInfo | None #
Search and return the variable information.
- Parameters:
- Returns:
VariableInfo object for the requested variable if found,
None
otherwise.- Return type:
VariableInfo | None
- class esmvalcore.cmor.table.CMIP5Info(cmor_tables_path, default=None, alt_names=None, strict=True)[source]#
Bases:
InfoBase
Class to read CMIP5-like data request.
- Parameters:
Methods:
get_table
(table)Search and return the table info.
get_variable
(table_name, short_name[, derived])Search and return the variable information.
- get_variable(table_name: str, short_name: str, derived: bool = False) VariableInfo | None #
Search and return the variable information.
- Parameters:
- Returns:
VariableInfo object for the requested variable if found,
None
otherwise.- Return type:
VariableInfo | None
- class esmvalcore.cmor.table.CMIP6Info(cmor_tables_path, default=None, alt_names=None, strict=True, default_table_prefix='')[source]#
Bases:
InfoBase
Class to read CMIP6-like data request.
This uses CMOR 3 json format
- Parameters:
Methods:
get_table
(table)Search and return the table info.
get_variable
(table_name, short_name[, derived])Search and return the variable information.
- get_variable(table_name: str, short_name: str, derived: bool = False) VariableInfo | None #
Search and return the variable information.
- Parameters:
- Returns:
VariableInfo object for the requested variable if found,
None
otherwise.- Return type:
VariableInfo | None
- esmvalcore.cmor.table.CMOR_TABLES: dict[str, CMIP3Info | CMIP5Info | CMIP6Info | CustomInfo] = {'ACCESS': <esmvalcore.cmor.table.CMIP6Info object>, 'CESM': <esmvalcore.cmor.table.CMIP6Info object>, 'CMIP3': <esmvalcore.cmor.table.CMIP3Info object>, 'CMIP5': <esmvalcore.cmor.table.CMIP5Info object>, 'CMIP6': <esmvalcore.cmor.table.CMIP6Info object>, 'CORDEX': <esmvalcore.cmor.table.CMIP5Info object>, 'EMAC': <esmvalcore.cmor.table.CMIP6Info object>, 'ICON': <esmvalcore.cmor.table.CMIP6Info object>, 'IPSLCM': <esmvalcore.cmor.table.CMIP6Info object>, 'OBS': <esmvalcore.cmor.table.CMIP5Info object>, 'OBS6': <esmvalcore.cmor.table.CMIP6Info object>, 'ana4mips': <esmvalcore.cmor.table.CMIP5Info object>, 'custom': <esmvalcore.cmor.table.CustomInfo object>, 'native6': <esmvalcore.cmor.table.CMIP6Info object>, 'obs4MIPs': <esmvalcore.cmor.table.CMIP6Info object>}#
CMOR info objects.
- class esmvalcore.cmor.table.CoordinateInfo(name)[source]#
Bases:
JsonInfo
Class to read and store coordinate information.
Attributes:
Axis
Generic level name
Long name
Whether bounds are required on this dimension
Out name
Values requested
Standard name
Direction in which the coordinate increases
Units
Maximum allowed value
Minimum allowed value
Coordinate value
Short name
Methods:
read_json
(json_data)Read coordinate information from json.
- axis#
Axis
- generic_lev_name#
Generic level name
- long_name#
Long name
- must_have_bounds#
Whether bounds are required on this dimension
- out_name#
Out name
This is the name of the variable in the file
- read_json(json_data)[source]#
Read coordinate information from json.
Non-present options will be set to empty
- Parameters:
json_data (dict) – dictionary created by the json reader containing coordinate information
- requested#
Values requested
- standard_name#
Standard name
- stored_direction#
Direction in which the coordinate increases
- units#
Units
- valid_max#
Maximum allowed value
- valid_min#
Minimum allowed value
- value#
Coordinate value
- var_name#
Short name
- class esmvalcore.cmor.table.CustomInfo(cmor_tables_path: str | Path | None = None)[source]#
Bases:
CMIP5Info
Class to read custom var info for ESMVal.
- Parameters:
cmor_tables_path (Optional[str | Path]) – Full path to the table or name for the table if it is present in ESMValTool repository. If
None
, use default tables from esmvalcore/cmor/tables/custom.
Methods:
get_table
(table)Search and return the table info.
get_variable
(table, short_name[, derived])Search and return the variable info.
- get_table(table)#
Search and return the table info.
- get_variable(table: str, short_name: str, derived: bool = False) VariableInfo | None [source]#
Search and return the variable info.
- Parameters:
- Returns:
VariableInfo object for the requested variable if found, returns None if not.
- Return type:
VariableInfo | None
- class esmvalcore.cmor.table.InfoBase(default, alt_names, strict)[source]#
Bases:
object
Base class for all table info classes.
This uses CMOR 3 json format
- Parameters:
Methods:
get_table
(table)Search and return the table info.
get_variable
(table_name, short_name[, derived])Search and return the variable information.
- get_variable(table_name: str, short_name: str, derived: bool = False) VariableInfo | None [source]#
Search and return the variable information.
- Parameters:
- Returns:
VariableInfo object for the requested variable if found,
None
otherwise.- Return type:
VariableInfo | None
- class esmvalcore.cmor.table.JsonInfo[source]#
Bases:
object
Base class for the info classes.
Provides common utility methods to read json variables
- class esmvalcore.cmor.table.TableInfo(*args, **kwargs)[source]#
Bases:
dict
Container class for storing a CMOR table.
Methods:
clear
()Remove all items from the dict.
copy
()Return a shallow copy of the dict.
fromkeys
([value])Create a new dictionary with keys from iterable and values set to value.
get
(key[, default])Return the value for key if key is in the dictionary, else default.
items
()Return a set-like object providing a view on the dict's items.
keys
()Return a set-like object providing a view on the dict's keys.
pop
(k[,d])If the key is not found, return the default if given; otherwise, raise a KeyError.
popitem
()Remove and return a (key, value) pair as a 2-tuple.
setdefault
(key[, default])Insert key with a value of default if key is not in the dictionary.
update
([E, ]**F)If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
values
()Return an object providing a view on the dict's values.
- clear()#
Remove all items from the dict.
- copy()#
Return a shallow copy of the dict.
- fromkeys(value=None, /)#
Create a new dictionary with keys from iterable and values set to value.
- get(key, default=None, /)#
Return the value for key if key is in the dictionary, else default.
- items()#
Return a set-like object providing a view on the dict’s items.
- keys()#
Return a set-like object providing a view on the dict’s keys.
- pop(k[, d]) v, remove specified key and return the corresponding value. #
If the key is not found, return the default if given; otherwise, raise a KeyError.
- popitem()#
Remove and return a (key, value) pair as a 2-tuple.
Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.
- setdefault(key, default=None, /)#
Insert key with a value of default if key is not in the dictionary.
Return the value for key if key is in the dictionary, else default.
- update([E, ]**F) None. Update D from mapping/iterable E and F. #
If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
- values()#
Return an object providing a view on the dict’s values.
- class esmvalcore.cmor.table.VariableInfo(table_type, short_name)[source]#
Bases:
JsonInfo
Class to read and store variable information.
Attributes:
Coordinates
List of dimensions
Data frequency
Long name
Modeling realm
Increasing direction
Short name
Standard name
Data units
Maximum admitted value
Minimum admitted value
Methods:
copy
()Return a shallow copy of VariableInfo.
has_coord_with_standard_name
(standard_name)Check if a coordinate with a given standard_name exists.
read_json
(json_data, default_freq)Read variable information from json.
- coordinates#
Coordinates
This is a dict with the names of the dimensions as keys and CoordinateInfo objects as values.
- copy()[source]#
Return a shallow copy of VariableInfo.
- Returns:
Shallow copy of this object.
- Return type:
- dimensions#
List of dimensions
- frequency#
Data frequency
- has_coord_with_standard_name(standard_name: str) bool [source]#
Check if a coordinate with a given standard_name exists.
For some coordinates, multiple (slightly different) versions with different dimension names but identical standard_name exist. For example, the CMIP6 tables provide 4 different standard_name=time dimensions: time, time1, time2, and time3. Other examples would be the CMIP6 pressure levels (plev19, plev23, plev27, etc. with standard name air_pressure) and the altitudes (alt16, alt40 with standard name altitude).
This function can be used to check for the existence of a specific coordinate defined by its standard_name, not its dimension name.
- long_name#
Long name
- modeling_realm#
Modeling realm
- positive#
Increasing direction
- read_json(json_data, default_freq)[source]#
Read variable information from json.
Non-present options will be set to empty
- short_name#
Short name
- standard_name#
Standard name
- units#
Data units
- valid_max#
Maximum admitted value
- valid_min#
Minimum admitted value
- esmvalcore.cmor.table.get_var_info(project: str, mip: str, short_name: str) VariableInfo | None [source]#
Get variable information.
Note
If project=CORDEX and the mip ends with ‘hr’, it is cropped to ‘h’ since CORDEX X-hourly tables define the mip as ending in ‘h’ instead of ‘hr’.
- Parameters:
- Returns:
VariableInfo object for the requested variable if found,
None
otherwise.- Return type:
VariableInfo | None
- Raises:
KeyError – No CMOR tables available for project.