Preprocessor functions

By default, preprocessor functions are applied in the order in which they are listed here.

Preprocessor module.

Functions:

amplitude(cube, coords)

Calculate amplitude of cycles by aggregating over coordinates.

annual_statistics(cube[, operator])

Compute annual statistics.

anomalies(cube, period[, reference, standardize])

Compute anomalies using a mean with the specified granularity.

area_statistics(cube, operator[, fx_variables])

Apply a statistical operator in the horizontal direction.

cleanup(files[, remove])

Clean up after running the preprocessor.

climate_statistics(cube[, operator, period])

Compute climate statistics with the specified granularity.

clip(cube[, minimum, maximum])

Clip values at a specified minimum and/or maximum value

cmor_check_data(cube, cmor_table, mip, …)

Check if data conforms to variable’s CMOR definiton.

cmor_check_metadata(cube, cmor_table, mip, …)

Check if metadata conforms to variable’s CMOR definiton.

concatenate(cubes)

Concatenate all cubes after fixing metadata.

convert_units(cube, units)

Convert the units of a cube to new ones.

daily_statistics(cube[, operator])

Compute daily statistics.

decadal_statistics(cube[, operator])

Compute decadal statistics.

depth_integration(cube)

Determine the total sum over the vertical component.

derive(cubes, short_name, long_name, units)

Derive variable.

detrend(cube[, dimension, method])

Detrend data along a given dimension.

download(files, dest_folder)

Download files that are not available locally

extract_levels(cube, levels, scheme[, …])

Perform vertical interpolation.

extract_month(cube, month)

Slice cube to get only the data belonging to a specific month.

extract_named_regions(cube, regions)

Extract a specific named region.

extract_point(cube, latitude, longitude, scheme)

Extract a point, with interpolation

extract_region(cube, start_longitude, …)

Extract a region from a cube.

extract_season(cube, season)

Slice cube to get only the data belonging to a specific season.

extract_shape(cube, shapefile[, method, …])

Extract a region defined by a shapefile.

extract_time(cube, start_year, start_month, …)

Extract a time range from a cube.

extract_trajectory(cube, latitudes, longitudes)

Extract data along a trajectory.

extract_transect(cube[, latitude, longitude])

Extract data along a line of constant latitude or longitude.

extract_volume(cube, z_min, z_max)

Subset a cube based on a range of values in the z-coordinate.

fix_data(cube, short_name, project, dataset, mip)

Fix cube data if fixes add present and check it anyway.

fix_file(file, short_name, project, dataset, …)

Fix files before ESMValTool can load them.

fix_metadata(cubes, short_name, project, …)

Fix cube metadata if fixes are required and check it anyway.

load(file[, callback])

Load iris cubes from files.

mask_above_threshold(cube, threshold)

Mask above a specific threshold value.

mask_below_threshold(cube, threshold)

Mask below a specific threshold value.

mask_fillvalues(products, threshold_fraction)

Compute and apply a multi-dataset fillvalues mask.

mask_glaciated(cube, mask_out)

Mask out glaciated areas.

mask_inside_range(cube, minimum, maximum)

Mask inside a specific threshold range.

mask_landsea(cube, fx_variables, mask_out[, …])

Mask out either land mass or sea (oceans, seas and lakes).

mask_landseaice(cube, fx_variables, mask_out)

Mask out either landsea (combined) or ice.

mask_outside_range(cube, minimum, maximum)

Mask outside a specific threshold range.

meridional_statistics(cube, operator)

Compute meridional statistics.

monthly_statistics(cube[, operator])

Compute monthly statistics.

multi_model_statistics(products, span, …)

Compute multi-model statistics.

regrid(cube, target_grid, scheme[, …])

Perform horizontal regridding.

regrid_time(cube, frequency)

Align time axis for cubes so they can be subtracted.

save(cubes, filename[, optimize_access, …])

Save iris cubes to file.

seasonal_statistics(cube[, operator])

Compute seasonal statistics.

timeseries_filter(cube, window, span[, …])

Apply a timeseries filter.

volume_statistics(cube, operator[, fx_variables])

Apply a statistical operation over a volume.

weighting_landsea_fraction(cube, …)

Weight fields using land or sea fraction.

zonal_statistics(cube, operator)

Compute zonal statistics.
esmvalcore.preprocessor.amplitude(cube, coords)[source]

Calculate amplitude of cycles by aggregating over coordinates.

Note

The amplitude is calculated as peak-to-peak amplitude (difference between maximum and minimum value of the signal). Other amplitude types are currently not supported.

Parameters

  • cube (iris.cube.Cube) – Input data.

  • coords (str or list of str) – Coordinates over which is aggregated. For example, use 'year' to extract the annual cycle amplitude for each year in the data or ['day_of_year', 'year'] to extract the diurnal cycle amplitude for each individual day in the data. If the coordinates are not found in cube, try to add it via iris.coord_categorisation (at the moment, this only works for the temporal coordinates day_of_month, day_of_year, hour, month, month_fullname, month_number, season, season_number, season_year, weekday, weekday_fullname, weekday_number or year.

Returns

Amplitudes.

Return type

iris.cube.Cube

Raises

iris.exceptions.CoordinateNotFoundError – A coordinate is not found in cube and cannot be added via iris.coord_categorisation.

esmvalcore.preprocessor.annual_statistics(cube, operator='mean')[source]

Compute annual statistics.

Note that this function does not weight the annual mean if uneven time periods are present. Ie, all data inside the year are treated equally.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • operator (str, optional) – Select operator to apply. Available operators: ‘mean’, ‘median’, ‘std_dev’, ‘sum’, ‘min’, ‘max’, ‘rms’

Returns

Annual statistics cube

Return type

iris.cube.Cube

esmvalcore.preprocessor.anomalies(cube, period, reference=None, standardize=False)[source]

Compute anomalies using a mean with the specified granularity.

Computes anomalies based on daily, monthly, seasonal or yearly means for the full available period

Parameters

  • cube (iris.cube.Cube) – input cube.

  • period (str) – Period to compute the statistic over. Available periods: ‘full’, ‘season’, ‘seasonal’, ‘monthly’, ‘month’, ‘mon’, ‘daily’, ‘day’

  • reference (list int, optional, default: None) – Period of time to use a reference, as needed for the ‘extract_time’ preprocessor function If None, all available data is used as a reference

  • standardize (bool, optional) – If True standardized anomalies are calculated

Returns

Anomalies cube

Return type

iris.cube.Cube

esmvalcore.preprocessor.area_statistics(cube, operator, fx_variables=None)[source]

Apply a statistical operator in the horizontal direction.

The average in the horizontal direction. We assume that the horizontal directions are [‘longitude’, ‘latutude’].

This function can be used to apply several different operations in the horizontal plane: mean, standard deviation, median variance, minimum and maximum. These options are specified using the operator argument and the following key word arguments:

mean

Area weighted mean.

median

Median (not area weighted)

std_dev

Standard Deviation (not area weighted)

sum

Area weighted sum.

variance

Variance (not area weighted)

min:

Minimum value

max

Maximum value

rms

Area weighted root mean square.
Parameters

  • cube (iris.cube.Cube) – Input cube.

  • operator (str) – The operation, options: mean, median, min, max, std_dev, sum, variance, rms.

  • fx_variables (dict) – dictionary of field:filename for the fx_variables

Returns

collapsed cube.

Return type

iris.cube.Cube

Raises

  • iris.exceptions.CoordinateMultiDimError – Exception for latitude axis with dim > 2.

  • ValueError – if input data cube has different shape than grid area weights

esmvalcore.preprocessor.cleanup(files, remove=None)[source]

Clean up after running the preprocessor.

esmvalcore.preprocessor.climate_statistics(cube, operator='mean', period='full')[source]

Compute climate statistics with the specified granularity.

Computes statistics for the whole dataset. It is possible to get them for the full period or with the data grouped by day, month or season

Parameters

  • cube (iris.cube.Cube) – input cube.

  • operator (str, optional) – Select operator to apply. Available operators: ‘mean’, ‘median’, ‘std_dev’, ‘sum’, ‘min’, ‘max’, ‘rms’

  • period (str, optional) – Period to compute the statistic over. Available periods: ‘full’, ‘season’, ‘seasonal’, ‘monthly’, ‘month’, ‘mon’, ‘daily’, ‘day’

Returns

Monthly statistics cube

Return type

iris.cube.Cube

esmvalcore.preprocessor.clip(cube, minimum=None, maximum=None)[source]

Clip values at a specified minimum and/or maximum value

Values lower than minimum are set to minimum and values higher than maximum are set to maximum.

Parameters

  • cube (iris.cube.Cube) – iris cube to be clipped

  • minimum (float) – lower threshold to be applied on input cube data.

  • maximum (float) – upper threshold to be applied on input cube data.

Returns

clipped cube.

Return type

iris.cube.Cube

esmvalcore.preprocessor.cmor_check_data(cube, cmor_table, mip, short_name, frequency, check_level=<CheckLevels.DEFAULT: 3>)[source]

Check if data conforms to variable’s CMOR definiton.

The checks performed at this step require the data in memory.

Parameters

  • cube (iris.cube.Cube) – Data cube to check.

  • cmor_table (basestring) – CMOR definitions to use.

  • mip – Variable’s mip.

  • short_name (basestring) – Variable’s short name

  • frequency (basestring) – Data frequency

  • check_level (CheckLevels) – Level of strictness of the checks.

esmvalcore.preprocessor.cmor_check_metadata(cube, cmor_table, mip, short_name, frequency, check_level=<CheckLevels.DEFAULT: 3>)[source]

Check if metadata conforms to variable’s CMOR definiton.

None of the checks at this step will force the cube to load the data.

Parameters

  • cube (iris.cube.Cube) – Data cube to check.

  • cmor_table (basestring) – CMOR definitions to use.

  • mip – Variable’s mip.

  • short_name (basestring) – Variable’s short name.

  • frequency (basestring) – Data frequency.

  • check_level (CheckLevels) – Level of strictness of the checks.

esmvalcore.preprocessor.concatenate(cubes)[source]

Concatenate all cubes after fixing metadata.

esmvalcore.preprocessor.convert_units(cube, units)[source]

Convert the units of a cube to new ones.

This converts units of a cube.

Parameters

  • cube (iris.cube.Cube) – input cube

  • units (str) – new units in udunits form

Returns

converted cube.

Return type

iris.cube.Cube

esmvalcore.preprocessor.daily_statistics(cube, operator='mean')[source]

Compute daily statistics.

Chunks time in daily periods and computes statistics over them;

Parameters

  • cube (iris.cube.Cube) – input cube.

  • operator (str, optional) – Select operator to apply. Available operators: ‘mean’, ‘median’, ‘std_dev’, ‘sum’, ‘min’, ‘max’, ‘rms’

Returns

Daily statistics cube

Return type

iris.cube.Cube

esmvalcore.preprocessor.decadal_statistics(cube, operator='mean')[source]

Compute decadal statistics.

Note that this function does not weight the decadal mean if uneven time periods are present. Ie, all data inside the decade are treated equally.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • operator (str, optional) – Select operator to apply. Available operators: ‘mean’, ‘median’, ‘std_dev’, ‘sum’, ‘min’, ‘max’, ‘rms’

Returns

Decadal statistics cube

Return type

iris.cube.Cube

esmvalcore.preprocessor.depth_integration(cube)[source]

Determine the total sum over the vertical component.

Requires a 3D cube. The z-coordinate integration is calculated by taking the sum in the z direction of the cell contents multiplied by the cell thickness.

Parameters

cube (iris.cube.Cube) – input cube.

Returns

collapsed cube.

Return type

iris.cube.Cube

esmvalcore.preprocessor.derive(cubes, short_name, long_name, units, standard_name=None)[source]

Derive variable.

Parameters

  • cubes (iris.cube.CubeList) – Includes all the needed variables for derivation defined in get_required().

  • short_name (str) – short_name

  • long_name (str) – long_name

  • units (str) – units

  • standard_name (str, optional) – standard_name

Returns

The new derived variable.

Return type

iris.cube.Cube

esmvalcore.preprocessor.detrend(cube, dimension='time', method='linear')[source]

Detrend data along a given dimension.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • dimension (str) – Dimension to detrend

  • method (str) – Method to detrend. Available: linear, constant. See documentation of ‘scipy.signal.detrend’ for details

Returns

Detrended cube

Return type

iris.cube.Cube

esmvalcore.preprocessor.download(files, dest_folder)[source]

Download files that are not available locally

esmvalcore.preprocessor.extract_levels(cube, levels, scheme, coordinate=None)[source]

Perform vertical interpolation.

Parameters

  • cube (cube) – The source cube to be vertically interpolated.

  • levels (array) – One or more target levels for the vertical interpolation. Assumed to be in the same S.I. units of the source cube vertical dimension coordinate.

  • scheme (str) – The vertical interpolation scheme to use. Choose from ‘linear’, ‘nearest’, ‘nearest_horizontal_extrapolate_vertical’, ‘linear_horizontal_extrapolate_vertical’.

  • coordinate (optional str) – The coordinate to interpolate

Returns

Return type

cube

See also

regrid

Perform horizontal regridding.

esmvalcore.preprocessor.extract_month(cube, month)[source]

Slice cube to get only the data belonging to a specific month.

Parameters

  • cube (iris.cube.Cube) – Original data

  • month (int) – Month to extract as a number from 1 to 12

Returns

data cube for specified month.

Return type

iris.cube.Cube

esmvalcore.preprocessor.extract_named_regions(cube, regions)[source]

Extract a specific named region.

The region coordinate exist in certain CMIP datasets. This preprocessor allows a specific named regions to be extracted.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • regions (str, list) – A region or list of regions to extract.

Returns

collapsed cube.

Return type

iris.cube.Cube

Raises
esmvalcore.preprocessor.extract_point(cube, latitude, longitude, scheme)[source]

Extract a point, with interpolation

Extracts a single latitude/longitude point from a cube, according to the interpolation scheme scheme.

Multiple points can also be extracted, by supplying an array of latitude and/or longitude coordinates. The resulting point cube will match the respective latitude and longitude coordinate to those of the input coordinates. If the input coordinate is a scalar, the dimension will be missing in the output cube (that is, it will be a scalar).

Parameters

  • cube (cube) – The source cube to extract a point from.

  • latitude (float, or array of float) – The latitude and longitude of the point.

  • longitude (float, or array of float) – The latitude and longitude of the point.

  • scheme (str) – The interpolation scheme. ‘linear’ or ‘nearest’. No default.

Returns

  • Returns a cube with the extracted point(s), and with adjusted

  • latitude and longitude coordinates (see above).

Examples

With a cube that has the coordinates

  • latitude: [1, 2, 3, 4]

  • longitude: [1, 2, 3, 4]

  • data values: [[[1, 2, 3, 4], [5, 6, …], […], […],

    … ]]]

>>> point = extract_point(cube, 2.5, 2.5, 'linear')  
>>> point.data  
array([ 8.5, 24.5, 40.5, 56.5])

Extraction of multiple points at once, with a nearest matching scheme. The values for 0.1 will result in masked values, since this lies outside the cube grid.

>>> point = extract_point(cube, [1.4, 2.1], [0.1, 1.1],
...                       'nearest')  
>>> point.data.shape  
(4, 2, 2)
>>> # x, y, z indices of masked values
>>> np.where(~point.data.mask)     
(array([0, 0, 1, 1, 2, 2, 3, 3]), array([0, 1, 0, 1, 0, 1, 0, 1]),
array([1, 1, 1, 1, 1, 1, 1, 1]))
>>> point.data[~point.data.mask].data  
array([ 1,  5, 17, 21, 33, 37, 49, 53])
esmvalcore.preprocessor.extract_region(cube, start_longitude, end_longitude, start_latitude, end_latitude)[source]

Extract a region from a cube.

Function that subsets a cube on a box (start_longitude, end_longitude, start_latitude, end_latitude) This function is a restriction of masked_cube_lonlat().

Parameters

  • cube (iris.cube.Cube) – input data cube.

  • start_longitude (float) – Western boundary longitude.

  • end_longitude (float) – Eastern boundary longitude.

  • start_latitude (float) – Southern Boundary latitude.

  • end_latitude (float) – Northern Boundary Latitude.

Returns

smaller cube.

Return type

iris.cube.Cube

esmvalcore.preprocessor.extract_season(cube, season)[source]

Slice cube to get only the data belonging to a specific season.

Parameters

  • cube (iris.cube.Cube) – Original data

  • season (str) – Season to extract. Available: DJF, MAM, JJA, SON

Returns

data cube for specified season.

Return type

iris.cube.Cube

esmvalcore.preprocessor.extract_shape(cube, shapefile, method='contains', crop=True, decomposed=False)[source]

Extract a region defined by a shapefile.

Note that this function does not work for shapes crossing the prime meridian or poles.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • shapefile (str) – A shapefile defining the region(s) to extract.

  • method (str, optional) – Select all points contained by the shape or select a single representative point. Choose either ‘contains’ or ‘representative’. If ‘contains’ is used, but not a single grid point is contained by the shape, a representative point will selected.

  • crop (bool, optional) – Crop the resulting cube using extract_region(). Note that data on irregular grids will not be cropped.

  • decomposed (bool, optional) – Whether or not to retain the sub shapes of the shapefile in the output. If this is set to True, the output cube has a dimension for the sub shapes.

Returns

Cube containing the extracted region.

Return type

iris.cube.Cube

See also

extract_region

Extract a region from a cube.

esmvalcore.preprocessor.extract_time(cube, start_year, start_month, start_day, end_year, end_month, end_day)[source]

Extract a time range from a cube.

Given a time range passed in as a series of years, months and days, it returns a time-extracted cube with data only within the specified time range.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • start_year (int) – start year

  • start_month (int) – start month

  • start_day (int) – start day

  • end_year (int) – end year

  • end_month (int) – end month

  • end_day (int) – end day

Returns

Sliced cube.

Return type

iris.cube.Cube

Raises

ValueError – if time ranges are outside the cube time limits

esmvalcore.preprocessor.extract_trajectory(cube, latitudes, longitudes, number_points=2)[source]

Extract data along a trajectory.

latitudes and longitudes are the pairs of coordinates for two points. number_points is the number of points between the two points.

This version uses the expensive interpolate method, but it may be necceasiry for irregular grids.

If only two latitude and longitude coordinates are given, extract_trajectory will produce a cube will extrapolate along a line between those two points, and will add number_points points between the two corners.

If more than two points are provided, then extract_trajectory will produce a cube which has extrapolated the data of the cube to those points, and number_points is not needed.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • latitudes (list) – list of latitude coordinates (floats).

  • longitudes (list) – list of longitude coordinates (floats).

  • number_points (int) – number of points to extrapolate (optional).

Returns

collapsed cube.

Return type

iris.cube.Cube

Raises

ValueError – if latitude and longitude have different dimensions.

esmvalcore.preprocessor.extract_transect(cube, latitude=None, longitude=None)[source]

Extract data along a line of constant latitude or longitude.

Both arguments, latitude and longitude, are treated identically. Either argument can be a single float, or a pair of floats, or can be left empty. The single float indicates the latitude or longitude along which the transect should be extracted. A pair of floats indicate the range that the transect should be extracted along the secondairy axis.

For instance ‘extract_transect(cube, longitude=-28)’ will produce a transect along 28 West.

Also, ‘extract_transect(cube, longitude=-28, latitude=[-50, 50])’ will produce a transect along 28 West between 50 south and 50 North.

This function is not yet implemented for irregular arrays - instead try the extract_trajectory function, but note that it is currently very slow. Alternatively, use the regrid preprocessor to regrid along a regular grid and then extract the transect.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • latitude (None, float or [float, float], optional) – transect latiude or range.

  • longitude (None, float or [float, float], optional) – transect longitude or range.

Returns

collapsed cube.

Return type

iris.cube.Cube

Raises

  • ValueError – slice extraction not implemented for irregular grids.

  • ValueError – latitude and longitude are both floats or lists; not allowed to slice on both axes at the same time.

esmvalcore.preprocessor.extract_volume(cube, z_min, z_max)[source]

Subset a cube based on a range of values in the z-coordinate.

Function that subsets a cube on a box (z_min, z_max) This function is a restriction of masked_cube_lonlat(); Note that this requires the requested z-coordinate range to be the same sign as the iris cube. ie, if the cube has z-coordinate as negative, then z_min and z_max need to be negative numbers.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • z_min (float) – minimum depth to extract.

  • z_max (float) – maximum depth to extract.

Returns

z-coord extracted cube.

Return type

iris.cube.Cube

esmvalcore.preprocessor.fix_data(cube, short_name, project, dataset, mip, frequency=None, check_level=<CheckLevels.DEFAULT: 3>)[source]

Fix cube data if fixes add present and check it anyway.

This method assumes that metadata is already fixed and checked.

This method collects all the relevant fixes for a given variable, applies them and checks resulting cube (or the original if no fixes were needed) metadata to ensure that it complies with the standards of its project CMOR tables.

Parameters

  • cube (iris.cube.Cube) – Cube to fix

  • short_name (str) – Variable’s short name

  • project (str) –

  • dataset (str) –

  • mip (str) – Variable’s MIP

  • frequency (str, optional) – Variable’s data frequency, if available

  • check_level (CheckLevels) – Level of strictness of the checks. Set to default.

Returns

Fixed and checked cube

Return type

iris.cube.Cube

Raises

CMORCheckError – If the checker detects errors in the data that it can not fix.

esmvalcore.preprocessor.fix_file(file, short_name, project, dataset, mip, output_dir)[source]

Fix files before ESMValTool can load them.

This fixes are only for issues that prevent iris from loading the cube or that cannot be fixed after the cube is loaded.

Original files are not overwritten.

Parameters

  • file (str) – Path to the original file

  • short_name (str) – Variable’s short name

  • project (str) –

  • dataset (str) –

  • output_dir (str) – Output directory for fixed files

Returns

Path to the fixed file

Return type

str

esmvalcore.preprocessor.fix_metadata(cubes, short_name, project, dataset, mip, frequency=None, check_level=<CheckLevels.DEFAULT: 3>)[source]

Fix cube metadata if fixes are required and check it anyway.

This method collects all the relevant fixes for a given variable, applies them and checks the resulting cube (or the original if no fixes were needed) metadata to ensure that it complies with the standards of its project CMOR tables.

Parameters

  • cubes (iris.cube.CubeList) – Cubes to fix

  • short_name (str) – Variable’s short name

  • project (str) –

  • dataset (str) –

  • mip (str) – Variable’s MIP

  • frequency (str, optional) – Variable’s data frequency, if available

  • check_level (CheckLevels) – Level of strictness of the checks. Set to default.

Returns

Fixed and checked cube

Return type

iris.cube.Cube

Raises

CMORCheckError – If the checker detects errors in the metadata that it can not fix.

esmvalcore.preprocessor.load(file, callback=None)[source]

Load iris cubes from files.

esmvalcore.preprocessor.mask_above_threshold(cube, threshold)[source]

Mask above a specific threshold value.

Takes a value ‘threshold’ and masks off anything that is above it in the cube data. Values equal to the threshold are not masked.

Parameters

  • cube (iris.cube.Cube) – iris cube to be thresholded.

  • threshold (float) – threshold to be applied on input cube data.

Returns

thresholded cube.

Return type

iris.cube.Cube

esmvalcore.preprocessor.mask_below_threshold(cube, threshold)[source]

Mask below a specific threshold value.

Takes a value ‘threshold’ and masks off anything that is below it in the cube data. Values equal to the threshold are not masked.

Parameters

  • cube (iris.cube.Cube) – iris cube to be thresholded

  • threshold (float) – threshold to be applied on input cube data.

Returns

thresholded cube.

Return type

iris.cube.Cube

esmvalcore.preprocessor.mask_fillvalues(products, threshold_fraction, min_value=None, time_window=1)[source]

Compute and apply a multi-dataset fillvalues mask.

Construct the mask that fills a certain time window with missing values if the number of values in that specific window is less than a given fractional threshold. This function is the extension of _get_fillvalues_mask and performs the combination of missing values masks from each model (of multimodels) into a single fillvalues mask to be applied to each model.

Parameters

  • products (iris.cube.Cube) – data products to be masked.

  • threshold_fraction (float) – fractional threshold to be used as argument for Aggregator. Must be between 0 and 1.

  • min_value (float) – minimum value threshold; default None If default, no thresholding applied so the full mask will be selected.

  • time_window (float) – time window to compute missing data counts; default set to 1.

Returns

Masked iris cubes.

Return type

iris.cube.Cube

Raises

NotImplementedError – Implementation missing for data with higher dimensionality than 4.

esmvalcore.preprocessor.mask_glaciated(cube, mask_out)[source]

Mask out glaciated areas.

It applies a Natural Earth mask. Note that for computational reasons only the 10 largest polygons are used for masking.

Parameters

  • cube (iris.cube.Cube) – data cube to be masked.

  • mask_out (str) – “glaciated” to mask out glaciated areas

Returns

Returns the masked iris cube.

Return type

iris.cube.Cube

Raises

ValueError – Error raised if masking on irregular grids is attempted or if mask_out has a wrong value.

esmvalcore.preprocessor.mask_inside_range(cube, minimum, maximum)[source]

Mask inside a specific threshold range.

Takes a MINIMUM and a MAXIMUM value for the range, and masks off anything that’s between the two in the cube data.

Parameters

  • cube (iris.cube.Cube) – iris cube to be thresholded

  • minimum (float) – lower threshold to be applied on input cube data.

  • maximum (float) – upper threshold to be applied on input cube data.

Returns

thresholded cube.

Return type

iris.cube.Cube

esmvalcore.preprocessor.mask_landsea(cube, fx_variables, mask_out, always_use_ne_mask=False)[source]

Mask out either land mass or sea (oceans, seas and lakes).

It uses dedicated fx files (sftlf or sftof) or, in their absence, it applies a Natural Earth mask (land or ocean contours). Note that the Natural Earth masks have different resolutions: 10m for land, and 50m for seas; these are more than enough for ESMValTool puprpose.

Parameters

  • cube (iris.cube.Cube) – data cube to be masked.

  • fx_variables (dict) – dict: keys: fx variables, values: full paths to fx files.

  • mask_out (str) – either “land” to mask out land mass or “sea” to mask out seas.

  • always_use_ne_mask (bool, optional (default: False)) – always apply Natural Earths mask, regardless if fx files are available or not.

Returns

Returns the masked iris cube.

Return type

iris.cube.Cube

Raises

ValueError – Error raised if masking on irregular grids is attempted. Irregular grids are not currently supported for masking with Natural Earth shapefile masks.

esmvalcore.preprocessor.mask_landseaice(cube, fx_variables, mask_out)[source]

Mask out either landsea (combined) or ice.

Function that masks out either landsea (land and seas) or ice (Antarctica and Greenland and some wee glaciers). It uses dedicated fx files (sftgif).

Parameters

  • cube (iris.cube.Cube) – data cube to be masked.

  • fx_variables (dict) – dict: keys: fx variables, values: full paths to fx files.

  • mask_out (str) – either “landsea” to mask out landsea or “ice” to mask out ice.

Returns

Returns the masked iris cube with either land or ice masked out.

Return type

iris.cube.Cube

Raises

  • ValueError – Error raised if fx mask and data have different dimensions.

  • ValueError – Error raised if fx files list is empty.

esmvalcore.preprocessor.mask_outside_range(cube, minimum, maximum)[source]

Mask outside a specific threshold range.

Takes a MINIMUM and a MAXIMUM value for the range, and masks off anything that’s outside the two in the cube data.

Parameters

  • cube (iris.cube.Cube) – iris cube to be thresholded

  • minimum (float) – lower threshold to be applied on input cube data.

  • maximum (float) – upper threshold to be applied on input cube data.

Returns

thresholded cube.

Return type

iris.cube.Cube

esmvalcore.preprocessor.meridional_statistics(cube, operator)[source]

Compute meridional statistics.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • operator (str, optional) – Select operator to apply. Available operators: ‘mean’, ‘median’, ‘std_dev’, ‘sum’, ‘min’, ‘max’, ‘rms’.

Returns

Meridional statistics cube.

Return type

iris.cube.Cube

Raises

ValueError – Error raised if computation on irregular grids is attempted. Zonal statistics not yet implemented for irregular grids.

esmvalcore.preprocessor.monthly_statistics(cube, operator='mean')[source]

Compute monthly statistics.

Chunks time in monthly periods and computes statistics over them;

Parameters

  • cube (iris.cube.Cube) – input cube.

  • operator (str, optional) – Select operator to apply. Available operators: ‘mean’, ‘median’, ‘std_dev’, ‘sum’, ‘min’, ‘max’, ‘rms’

Returns

Monthly statistics cube

Return type

iris.cube.Cube

esmvalcore.preprocessor.multi_model_statistics(products, span, statistics, output_products=None)[source]

Compute multi-model statistics.

Multimodel statistics computed along the time axis. Can be computed across a common overlap in time (set span: overlap) or across the full length in time of each model (set span: full). Restrictive computation is also available by excluding any set of models that the user will not want to include in the statistics (set exclude: [excluded models list]).

Restrictions needed by the input data: - model datasets must have consistent shapes, - higher dimensional data is not supported (ie dims higher than four: time, vertical axis, two horizontal axes).

Parameters

  • products (list) – list of data products or cubes to be used in multimodel stat computation; cube attribute of product is the data cube for computing the stats.

  • span (str) – overlap or full; if overlap, statitsticss are computed on common time- span; if full, statistics are computed on full time spans, ignoring missing data.

  • output_products (dict) – dictionary of output products.

  • statistics (str) – statistical measure to be computed. Available options: mean, median, max, min, std, or pXX.YY (for percentile XX.YY; decimal part optional).

Returns

list of data products or cubes containing the multimodel stats computed.

Return type

list

Raises

ValueError – If span is neither overlap nor full.

esmvalcore.preprocessor.regrid(cube, target_grid, scheme, lat_offset=True, lon_offset=True)[source]

Perform horizontal regridding.

Parameters

  • cube (cube) – The source cube to be regridded.

  • target_grid (cube or str) – The cube that specifies the target or reference grid for the regridding operation. Alternatively, a string cell specification may be provided, of the form ‘MxN’, which specifies the extent of the cell, longitude by latitude (degrees) for a global, regular target grid.

  • scheme (str) – The regridding scheme to perform, choose from ‘linear’, ‘linear_extrapolate’, ‘nearest’, ‘area_weighted’, ‘unstructured_nearest’.

  • lat_offset (bool) – Offset the grid centers of the latitude coordinate w.r.t. the pole by half a grid step. This argument is ignored if target_grid is a cube or file.

  • lon_offset (bool) – Offset the grid centers of the longitude coordinate w.r.t. Greenwich meridian by half a grid step. This argument is ignored if target_grid is a cube or file.

Returns

Return type

cube

See also

extract_levels

Perform vertical regridding.

esmvalcore.preprocessor.regrid_time(cube, frequency)[source]

Align time axis for cubes so they can be subtracted.

Operations on time units, time points and auxiliary coordinates so that any cube from cubes can be subtracted from any other cube from cubes. Currently this function supports yearly (frequency=yr), monthly (frequency=mon), daily (frequency=day), 6-hourly (frequency=6hr), 3-hourly (frequency=3hr) and hourly (frequency=1hr) data time frequencies.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • frequency (str) – data frequency: mon, day, 1hr, 3hr or 6hr

Returns

cube with converted time axis and units.

Return type

iris.cube.Cube

esmvalcore.preprocessor.save(cubes, filename, optimize_access='', compress=False, alias='', **kwargs)[source]

Save iris cubes to file.

Parameters

  • cubes (iterable of iris.cube.Cube) – Data cubes to be saved

  • filename (str) – Name of target file

  • optimize_access (str) –

    Set internal NetCDF chunking to favour a reading scheme

    Values can be map or timeseries, which improve performance when reading the file one map or time series at a time. Users can also provide a coordinate or a list of coordinates. In that case the better performance will be avhieved by loading all the values in that coordinate at a time

  • compress (bool, optional) – Use NetCDF internal compression.

Returns

filename

Return type

str

esmvalcore.preprocessor.seasonal_statistics(cube, operator='mean')[source]

Compute seasonal statistics.

Chunks time in 3-month periods and computes statistics over them;

Parameters

  • cube (iris.cube.Cube) – input cube.

  • operator (str, optional) – Select operator to apply. Available operators: ‘mean’, ‘median’, ‘std_dev’, ‘sum’, ‘min’, ‘max’, ‘rms’

Returns

Seasonal statistic cube

Return type

iris.cube.Cube

esmvalcore.preprocessor.timeseries_filter(cube, window, span, filter_type='lowpass', filter_stats='sum')[source]

Apply a timeseries filter.

Method borrowed from iris example

Apply each filter using the rolling_window method used with the weights keyword argument. A weighted sum is required because the magnitude of the weights are just as important as their relative sizes.

See also the iris rolling window

Parameters

  • cube (iris.cube.Cube) – input cube.

  • window (int) – The length of the filter window (in units of cube time coordinate).

  • span (int) – Number of months/days (depending on data frequency) on which weights should be computed e.g. 2-yearly: span = 24 (2 x 12 months). Span should have same units as cube time coordinate.

  • filter_type (str, optional) – Type of filter to be applied; default ‘lowpass’. Available types: ‘lowpass’.

  • filter_stats (str, optional) – Type of statistic to aggregate on the rolling window; default ‘sum’. Available operators: ‘mean’, ‘median’, ‘std_dev’, ‘sum’, ‘min’, ‘max’, ‘rms’

Returns

cube time-filtered using ‘rolling_window’.

Return type

iris.cube.Cube

Raises

  • iris.exceptions.CoordinateNotFoundError: – Cube does not have time coordinate.

  • NotImplementedError: – If filter_type is not implemented.

esmvalcore.preprocessor.volume_statistics(cube, operator, fx_variables=None)[source]

Apply a statistical operation over a volume.

The volume average is weighted according to the cell volume. Cell volume is calculated from iris’s cartography tool multiplied by the cell thickness.

Parameters

  • cube (iris.cube.Cube) – Input cube.

  • operator (str) – The operation to apply to the cube, options are: ‘mean’.

  • fx_variables (dict) – dictionary of field:filename for the fx_variables

Returns

collapsed cube.

Return type

iris.cube.Cube

Raises

ValueError – if input cube shape differs from grid volume cube shape.

esmvalcore.preprocessor.weighting_landsea_fraction(cube, fx_variables, area_type)[source]

Weight fields using land or sea fraction.

This preprocessor function weights a field with its corresponding land or sea area fraction (value between 0 and 1). The application of this is important for most carbon cycle variables (and other land-surface outputs), which are e.g. reported in units of kgC m-2. This actually refers to ‘per square meter of land/sea’ and NOT ‘per square meter of gridbox’. So in order to integrate these globally or regionally one has to both area-weight the quantity but also weight by the land/sea fraction.

Parameters

  • cube (iris.cube.Cube) – Data cube to be weighted.

  • fx_variables (dict) – Dictionary holding var_name (keys) and full paths (values) to the fx files as str or empty list (if not available).

  • area_type (str) – Use land ('land') or sea ('sea') fraction for weighting.

Returns

Land/sea fraction weighted cube.

Return type

iris.cube.Cube

Raises

  • TypeError – area_type is not 'land' or 'sea'.

  • ValueError – Land/sea fraction variables sftlf or sftof not found or shape of them is not broadcastable to cube.

esmvalcore.preprocessor.zonal_statistics(cube, operator)[source]

Compute zonal statistics.

Parameters

  • cube (iris.cube.Cube) – input cube.

  • operator (str, optional) – Select operator to apply. Available operators: ‘mean’, ‘median’, ‘std_dev’, ‘sum’, ‘min’, ‘max’, ‘rms’.

Returns

Zonal statistics cube.

Return type

iris.cube.Cube

Raises

ValueError – Error raised if computation on irregular grids is attempted. Zonal statistics not yet implemented for irregular grids.