Changelog#
v2.10.0#
Highlights
All statistics preprocessors support the same operators and have a common documentation. In addition, arbitrary keyword arguments for the statistical operation can be directly given to the preprocessor.
The output webpage generated by the tool now looks better and provides methods to select and filter the output.
Improved computational efficiency:
Automatic rechunking between preprocessor steps to keep the graph size smaller and the chunk size optimal.
Reduce the size of the dask graph created by
esmvalcore.preprocessor.anomalies()
.Preprocessors
esmvalcore.preprocessor.mask_above_threshold()
,esmvalcore.preprocessor.mask_below_threshold()
,esmvalcore.preprocessor.mask_inside_range()
,esmvalcore.preprocessor.mask_outside_range()
are now lazy.Lazy coordinates bounds are no longer loaded into memory by the CMOR checks and fixes.
This release includes
Backwards incompatible changes#
Remove the deprecated option
use_legacy_supplementaries
(Pull request #2202) by @bouweandelaThe recommended upgrade procedure is to remove
use_legacy_supplementaries
from config-user.yml (if it was there) and remove any mention offx_variables
from the recipe. If automatically defining the required supplementary variables does not work, define them in the variable or (additional_
)datasets
section as described in Defining supplementary variables (ancillary variables and cell measures).
Use smarter (units-aware) weights (Pull request #2139) by @schlunma
Some preprocessors handle units better. For details, see the pull request.
Removed deprecated configuration option
offline
(Pull request #2213) by @schlunmaFix issue with CORDEX datasets requiring different dataset tags for downloads and fixes (Pull request #2066) by @ljoakim
Due to the different facets for CORDEX datasets, there was an inconsistency in the fixing mechanism. This change requires changes to existing recipes that use CORDEX datasets. Please refer to the pull request for detailed update instructions.
For the following changes, no user change is necessary
Remove deprecated way of calling
read_cmor_tables()
(Pull request #2201) by @bouweandelaRemove deprecated callback argument from preprocessor
load
function (Pull request #2207) by @bouweandelaRemove deprecated preprocessor function cleanup (Pull request #2215) by @bouweandela
Deprecations#
Clearly separate fixes and CMOR checks (Pull request #2157) by @schlunma
Added new operators for statistics preprocessor (e.g.,
'percentile'
) and allowed passing additional arguments (Pull request #2191) by @schlunmaThis harmonizes the operators for all statistics preprocessors. From this version, the new names can be used; the old arguments will stop working from version 2.12.0. Please refer to Statistical preprocessors for a detailed description.
Bug fixes#
Re-add correctly region-extracted cell measures and ancillary variables after extract_region (Pull request #2166) by @valeriupredoi, @schlunma
Fix sorting of datasets
Fix sorting of ensemble members in
datasets_to_recipe()
(Pull request #2095) by @bouweandelaFix a problem with sorting datasets that have a mix of facet types (Pull request #2238) by @bouweandela
Avoid a crash if dataset has supplementary variables (Pull request #2198) by @bouweandela
CMOR standard#
ERA5 on-the-fly CMORizer: changed sign of variables
evspsbl
andevspsblpot
(Pull request #2115) by @katjaweigelAdd
ch4
surface custom cmor table entry (Pull request #2168) by @hb326Add CMIP3 institutes names used at NCI (Pull request #2152) by @rbeucher
Added
get_time_bounds()
andget_next_month()
to public API (Pull request #2214) by @schlunmaImprove concatenation checks
Relax concatenation checks for
--check_level=relax
and--check_level=ignore
(Pull request #2144) by @sloosvelFix
concatenate
preprocessor function (Pull request #2240) by @bouweandelaFix time overlap handling in concatenation (Pull request #2247) by @zklaus
Computational performance improvements#
Make Minimum, maximum and interval masking preprocessors lazy (Pull request #2169) by @joergbenke
Restored usage of numpy in _mask_with_shp (Pull request #2209) by @joergbenke
Do not realize lazy coordinate bounds in CMOR check (Pull request #2146) by @sloosvel
Rechunk between preprocessor steps (Pull request #2205) by @bouweandela
Reduce the size of the dask graph created by the
anomalies
preprocessor function (Pull request #2200) by @bouweandela
Documentation#
Add reference to release v2.9.0 in the changelog (Pull request #2130) by @remi-kazeroni
Add merge instructions to release instructions (Pull request #2131) by @zklaus
Update mamba before building environment during Readthedocs build (Pull request #2149) by @valeriupredoi
Ensure compatible zstandard and zstd versions for .conda support (Pull request #2204) by @zklaus
Remove outdated documentation (Pull request #2210) by @bouweandela
Remove meercode badge from README because their API is broken (Pull request #2224) by @valeriupredoi
Correct usage help text of version command (Pull request #2232) by @jfrost-mo
Add
navigation_with_keys: False
tohtml_theme_options
in Readthedocsconf.py
(Pull request #2245) by @valeriupredoiReplace squarey badge with roundy shield for Anaconda sticker in README (Pull request #2233, Pull request #2260) by @valeriupredoi
Fixes for datasets#
Updated doc about fixes and added type hints to fix functions (Pull request #2160) by @schlunma
Installation#
Clean-up how pins are written in conda environment file (Pull request #2125) by @valeriupredoi
Use importlib.metadata instead of deprecated pkg_resources (Pull request #2096) by @bouweandela
Pin shapely to >=2.0 (Pull request #2075) by @valeriupredoi
Pin Python to <3.12 in conda environment (Pull request #2272) by @bouweandela
Preprocessor#
Improve preprocessor output sorting code (Pull request #2111) by @bouweandela
Preprocess datasets in the same order as they are listed in the recipe (Pull request #2103) by @bouweandela
Automatic testing#
[Github Actions] Compress all bash shell setters into one default option per workflow (Pull request #2126) by @valeriupredoi
[Github Actions] Fix Monitor Tests Github Action (Pull request #2135) by @valeriupredoi
[condalock] update conda lock file (Pull request #2141) by @valeriupredoi
[Condalock] make sure mamba/conda are at latest version by forcing a pinned mamba install (Pull request #2136) by @valeriupredoi
Update code coverage orbs (Pull request #2206) by @bouweandela
Revisit the comment-triggered Github Actions test (Pull request #2243) by @valeriupredoi
Remove workflow that runs Github Actions tests from PR comment (Pull request #2244) by @valeriupredoi
Improvements#
Merge v2.9.x into main (Pull request #2128) by @schlunma
Fix typo in citation file (Pull request #2182) by @bouweandela
Cleaned and extended function that extracts datetimes from paths (Pull request #2181) by @schlunma
Add file encoding (and some read modes) at open file step (Pull request #2219) by @valeriupredoi
Check type of argument passed to
read_cmor_tables()
(Pull request #2217) by @valeriupredoiDynamic HTML output for monitoring (Pull request #2062) by @bsolino
Use PyPI’s trusted publishers authentication (Pull request #2269) by @valeriupredoi
v2.9.0#
Highlights#
It is now possible to use the Dask distributed scheduler, which can significantly reduce the run-time of recipes. Configuration examples and advice are available in our documentation. More work on improving the computational performance is planned, so please share your experiences, good and bad, with this new feature in Discussion #1763.
This release includes
Backwards incompatible changes#
Remove deprecated configuration options (Pull request #2056) by @bouweandela
The module
esmvalcore.experimental.config
has been removed. To upgrade, import the module fromesmvalcore.config
instead.The module
esmvalcore._config
has been removed. To upgrade, useesmvalcore.config
instead.The methods
esmvalcore.config.Session.to_config_user
andesmvalcore.config.Session.from_config_user
have been removed. To upgrade, useesmvalcore.config.Session
to access the configuration values directly.
Bug fixes#
Respect
ignore_warnings
settings from the project configuration inesmvalcore.dataset.Dataset.load()
(Pull request #2046) by @schlunmaFixed usage of custom location for custom CMOR tables (Pull request #2052) by @schlunma
Fix issue with writing index.html when running a recipe with
--resume-from
(Pull request #2055) by @bouweandelaFixed bug in ICON CMORizer that lead to shifted time coordinates (Pull request #2038) by @schlunma
Include
-
in allowed characters for bibtex references (Pull request #2097) by @alistairsellarDo not raise an exception if the requested version of a file is not available for all matching files on ESGF (Pull request #2105) by @bouweandela
Computational performance improvements#
Add support for configuring Dask distributed (Pull request #2049, Pull request #2122) by @bouweandela
Make
esmvalcore.preprocessor.extract_levels()
lazy (Pull request #1761) by @bouweandelaLazy implementation of
esmvalcore.preprocessor.multi_model_statistics()
andesmvalcore.preprocessor.ensemble_statistics()
(Pull request #968 and Pull request #2087) by @Peter9192Avoid realizing data in preprocessor function
esmvalcore.preprocessor.concatenate()
when cubes overlap (Pull request #2109) by @bouweandela
Documentation#
Remove unneeded sphinxcontrib extension (Pull request #2047) by @valeriupredoi
Show ESMValTool logo on PyPI webpage (Pull request #2065) by @valeriupredoi
Fix gitter badge in README (Pull request #2118) by @remi-kazeroni
Add changelog for v2.9.0 (Pull request #2088 and Pull request #2123) by @bouweandela
Fixes for datasets#
Pass the
esmvalcore.config.Session
to fixes (Pull request #1988) by @schlunmaICON: Allowed specifying vertical grid information in recipe (Pull request #2067) by @schlunma
Allow specifying
raw_units
for CESM2, EMAC, and ICON CMORizers (Pull request #2043) by @schlunmaICON: allow specifying horizontal grid file in recipe/extra facets (Pull request #2078) by @schlunma
Fix tas/tos CMIP6: FIO, KACE, MIROC, IITM (Pull request #2061) by @pepcos
Add fix for EC-Earth3-Veg tos calendar (Pull request #2100) by @bouweandela
Correct GISS-E2-1-G
tos
units (Pull request #2099) by @bouweandela
Installation#
Drop support for Python 3.8 (Pull request #2053) by @bouweandela
Add python 3.11 to Github Actions package (conda and PyPI) installation tests (Pull request #2083) by @valeriupredoi
Remove
with_mypy
orwith-mypy
optional tool for prospector (Pull request #2108) by @valeriupredoi
Preprocessor#
Added
period='hourly'
foresmvalcore.preprocessor.climate_statistics()
andesmvalcore.preprocessor.anomalies()
(Pull request #2068) by @schlunmaSupport IPCC AR6 regions in
esmvalcore.preprocessor.extract_shape()
(Pull request #2008) by @schlunma
v2.8.1#
Highlights#
This release adds support for Python 3.11 and includes several bugfixes.
This release includes:
Bug fixes#
Pin numpy !=1.24.3 (Pull request #2011) by @valeriupredoi
Fix a bug in recording provenance for the
mask_multimodel
preprocessor (Pull request #1984) by @schlunmaFix ICON hourly data rounding issues (Pull request #2022) by @BauerJul
Use the default SSL context when using the
extract_location
preprocessor (Pull request #2023) by @ehoganMake time-related CMOR fixes work with time dimensions time1, time2, time3 (Pull request #1971) by @schlunma
Always create a cache directory for storing ICON grid files (Pull request #2030) by @schlunma
Fixed altitude <–> pressure level conversion for masked arrays in the
extract_levels
preprocessor (Pull request #1999) by @schlunmaAllowed ignoring of scalar time coordinates in the
multi_model_statistics
preprocessor (Pull request #1961) by @schlunma
Fixes for datasets#
Add support for hourly ICON data (Pull request #1990) by @BauerJul
Fix areacello in BCC-CSM2-MR (Pull request #1993) by @remi-kazeroni
Installation#
Add support for Python=3.11 (Pull request #1832) by @valeriupredoi
Modernize conda lock file creation workflow with mamba, Mambaforge etc (Pull request #2027) by @valeriupredoi
Pin libnetcdf!=4.9.1 (Pull request #2072) by @remi-kazeroni
Documentation#
Add changelog for v2.8.1 (Pull request #2079) by @bouweandela
Automatic testing#
Use mocked geopy.geocoders.Nominatim to avoid ReadTimeoutError (Pull request #2005) by @schlunma
Update pre-commit hooks (Pull request #2020) by @bouweandela
v2.8.0#
Highlights#
ESMValCore now supports wildcards in recipes and offers improved support for ancillary variables and dataset versioning thanks to contributions by @bouweandela. For details, see Automatically populating a recipe with all available datasets and Defining supplementary variables.
Support for CORDEX datasets in a rotated pole coordinate system has been added by @sloosvel.
Native ICON output is now made UGRID-compliant on-the-fly to unlock the use of more sophisticated regridding algorithms, thanks to @schlunma.
The Python API has been extended with the addition of three modules:
esmvalcore.config
,esmvalcore.dataset
, andesmvalcore.local
, all these features courtesy of @bouweandela. For details, see our new example Example notebooks.The preprocessor
multi_model_statistics()
has been extended to support more use-cases thanks to contributions by @schlunma. For details, see Multi-model statistics.
This release includes:
Backwards incompatible changes#
Please read the descriptions of the linked pull requests for detailed upgrade instructions.
The algorithm for automatically defining the ancillary variables and cell measures has been improved (Pull request #1609) by @bouweandela. If this does not work as expected, more examples of how to adapt your recipes are given here and in the corresponding sections of the recipe documentation and the preprocessor documentation.
Remove deprecated features scheduled for removal in v2.8.0 or earlier (Pull request #1826) by @schlunma. Removed
esmvalcore.iris_helpers.var_name_constraint
(has been deprecated in v2.6.0; please useiris.NameConstraint
with the keyword argumentvar_name
instead) and the optionalways_use_ne_mask
foresmvalcore.preprocessor.mask_landsea()
(has been deprecated in v2.5.0; the same behavior can now be achieved by specifyingsupplementary_variables
.No files will be found if a non-existent version of a dataset is specified (Pull request #1835) by @bouweandela. If a
version
of a dataset is specified in the recipe, the tool will now search for exactly that version, instead of simply using the latest version. Therefore, it is necessary to make sure that the version number in the directory tree matches with the version number in the recipe to find the files.The default filename template for obs4MIPs has been updated to better match filenames used in this project in (Pull request #1866) by @bouweandela. This may cause issues if you are storing all the files for obs4MIPs in a directory with no subdirectories per dataset.
Deprecations#
Please read the descriptions of the linked pull requests for detailed upgrade instructions.
Various configuration related options that are now available through
esmvalcore.config
have been deprecated (Pull request #1769) by @bouweandela.The
fx_variables
preprocessor argument and related features have been deprecated (Pull request #1609) by @bouweandela. See Pull request #1609#Deprecations for more information.Combined
offline
andalways_search_esgf
into a single optionsearch_esgf
(Pull request #1935) @schlunma. The configuration option/command line argumentoffline
has been deprecated in favor ofsearch_esgf
. The previousoffline: true
is nowsearch_esgf: never
(the default); the previousoffline: false
is nowsearch_esgf: when_missing
. More details on how to adapt your workflow regarding these new options are given in Pull request #1935 and the documentation.esmvalcore.preprocessor.cleanup()
has been deprecated (Pull request #1949) @schlunma. Please do not use this anymore in the recipe (it is not necessary).
Python API#
Support searching ESGF for a specific version of a file and add
esmvalcore.esgf.ESGFFile.facets
(Pull request #1822) by @bouweandelaFix issues with searching for files on ESGF (Pull request #1863) by @bouweandela
Move the
esmvalcore.experimental.config
module toesmvalcore.config
(Pull request #1769) by @bouweandelaAdd
esmvalcore.local
, a module to search data on the local filesystem (Pull request ##1835) by @bouweandelaAdd
esmvalcore.dataset
module (Pull request #1877) by @bouweandela
Bug fixes#
Import from
esmvalcore.config
in theesmvalcore.experimental
module (Pull request #1816) by @bouweandelaAdded scalar coords of input cubes to output of esmpy_regrid (Pull request #1811) by @schlunma
Fix severe bug in
esmvalcore.preprocessor.mask_fillvalues()
(Pull request #1823) by @schlunmaFix LWP of ICON on-the-fly CMORizer (Pull request #1839) by @schlunma
Fixed issue in irregular regridding regarding scalar coordinates (Pull request #1845) by @schlunma
Update product attributes and metadata.yml with cube metadata before saving files (Pull request #1837) by @schlunma
Remove an extra space character from a filename (Pull request #1883) by @bouweandela
Improve resilience of ESGF search (Pull request #1869) by @bouweandela
Fix issue with no files found if timerange start/end differs in length (Pull request #1880) by @bouweandela
Add driver and sub_experiment tags to generate dataset aliases (Pull request #1886) by @sloosvel
Fixed time points of native CESM2 output (Pull request #1772) by @schlunma
Fix type hints for Python versions < 3.10 (Pull request #1897) by @bouweandela
Fixed set_range_in_0_360 for dask arrays (Pull request #1919) by @schlunma
Made equalized attributes in concatenated cubes consistent across runs (Pull request #1783) by @schlunma
Fix issue with reading dates from files (Pull request #1936) by @bouweandela
Add institute name used on ESGF for CMIP5 CanAM4, CanCM4, and CanESM2 (Pull request #1937) by @bouweandela
Fix issue where data was not loaded and saved (Pull request #1962) by @bouweandela
Fix type hints for Python 3.8 (Pull request #1795) by @bouweandela
Update the institute facet of the CSIRO-Mk3L-1-2 model (Pull request #1966) by @remi-kazeroni
Fixed race condition that may result in errors in
esmvalcore.preprocessor.cleanup()
(Pull request #1949) by @schlunmaUpdate notebook so it uses supplementaries instead of ancillaries (Pull request #1945) by @bouweandela
Documentation#
Fix anaconda badge in README (Pull request #1759) by @valeriupredoi
Fix mistake in the documentation of
esmvalcore.esgf.find_files
(Pull request #1784) by @bouweandelaSupport linking to “stable” ESMValTool version on readthedocs (Pull request #1608) by @bouweandela
Updated ICON doc with information on usage of extract_levels preprocessor (Pull request #1903) by @schlunma
Add changelog for latest released version v2.7.1 (Pull request #1905) by @valeriupredoi
Update preprocessor.rst due to renaming of NCEP dataset to NCEP-NCAR-R1 (Pull request #1908) by @hb326
Replace timerange nested lists in docs with overview table (Pull request #1940) by @zklaus
Updated section “backward compatibility” in contributing.rst (Pull request #1918) by @axel-lauer
Add link to ESMValTool release procedure steps (Pull request #1957) by @remi-kazeroni
Synchronize documentation table of contents with ESMValTool (Pull request #1958) by @bouweandela
Improvements#
Support wildcards in the recipe and improve support for ancillary variables and dataset versioning (Pull request #1609) by @bouweandela. More details on how to adapt your recipes are given in the corresponding pull request description and in the corresponding sections of the recipe documentation and the preprocessor documentation.
Create a session directory with suffix “-1”, “-2”, etc if it already exists (Pull request #1818) by @bouweandela
Message for users when they use esmvaltool executable from esmvalcore only (Pull request #1831) by @valeriupredoi
Order recipe output in index.html (Pull request #1899) by @bouweandela
Improve reading facets from ESGF search results (Pull request #1920) by @bouweandela
Fixes for datasets#
Fix rotated coordinate grids and tas and pr for CORDEX datasets (Pull request #1765) by @sloosvel
Made ICON output UGRID-compliant (on-the-fly) (Pull request #1664) by @schlunma
Fix automatic download of ICON grid file and make ICON UGRIDization optional (default: true) (Pull request #1922) by @schlunma
Add siconc fixes for EC-Earth3-Veg and EC-Earth3-Veg-LR models (Pull request #1771) by @egalytska
Fix siconc in KIOST-ESM (Pull request #1829) by @LisaBock
Extension of ERA5 CMORizer (variable cl) (Pull request #1850) by @axel-lauer
Add standard variable names for EMAC (Pull request #1853) by @FranziskaWinterstein
Fix for FGOALS-f3-L clt (Pull request #1928) by @LisaBock
Installation#
Add all deps to the conda-forge environment and suppress installing and reinstalling deps with pip at readthedocs builds (Pull request #1786) by @valeriupredoi
Pin netCDF4<1.6.1 (Pull request #1805) by @bouweandela
Unpin NetCF4 (Pull request #1814) by @valeriupredoi
Unpin flake8 (Pull request #1820) by @valeriupredoi
Add iris-esmf-regrid as a dependency (Pull request #1809) by @sloosvel
Pin esmpy<8.4 (Pull request #1871) by @zklaus
Update esmpy import for ESMF v8.4.0 (Pull request #1876) by @bouweandela
Preprocessor#
Allow
esmvalcore.preprocessor.multi_model_statistics()
on cubes with arbitrary dimensions (Pull request #1808) by @schlunmaSmarter removal of coordinate metadata in
esmvalcore.preprocessor.multi_model_statistics()
preprocessor (Pull request #1813) by @schlunmaAllowed usage of
esmvalcore.preprocessor.multi_model_statistics()
on single cubes/products (Pull request #1849) by @schlunmaAllowed usage of
esmvalcore.preprocessor.multi_model_statistics()
on cubes with identicalname()
andunits
(but e.g. different long_name) (Pull request #1921) by @schlunmaAllowed ignoring scalar coordinates in
esmvalcore.preprocessor.multi_model_statistics()
(Pull request #1934) by @schlunmaRefactored
esmvalcore.preprocessor.regrid()
and removed unnecessary code not needed anymore due to new iris version (Pull request #1898) by @schlunmaDo not realise coordinates during CMOR check (Pull request #1912) by @sloosvel
Make
esmvalcore.preprocessor.extract_volume()
work with closed and mixed intervals and allow nearest value selection (Pull request #1930) by @sloosvel
Release#
Changelog for v2.8.0rc1 (Pull request #1952) by @remi-kazeroni
Increase version number for ESMValCore v2.8.0rc1 (Pull request #1955) by @remi-kazeroni
Changelog for v2.8.0rc2 (Pull request #1959) by @remi-kazeroni
Increase version number for ESMValCore v2.8.0rc2 (Pull request #1973) by @remi-kazeroni
Changelog for v2.8.0 (Pull request #1978) by @remi-kazeroni
Increase version number for ESMValCore v2.8.0 (Pull request #1983) by @remi-kazeroni
Automatic testing#
Set implicit optional to true in mypy config to avert side effects and test fails from new mypy version (Pull request #1790) by @valeriupredoi
Remove duplicate implicit_optional = True line in
setup.cfg
(Pull request #1791) by @valeriupredoiFix failing test due to missing sample data (Pull request #1797) by @bouweandela
Remove outdated cmor_table facet from data finder tests (Pull request #1798) by @bouweandela
Modernize tests for
esmvalcore.preprocessor.save()
(Pull request #1799) by @bouweandelaNo more sequential tests since SegFaults were not noticed anymore (Pull request #1819) by @valeriupredoi
Update pre-commit configuration (Pull request #1821) by @bouweandela
Updated URL of ICON grid file used for testing (Pull request #1914) by @schlunma
Variable Derivation#
Add derivation of sea ice extent (Pull request #1695) by @sloosvel
v2.7.1#
Highlights#
This is a bugfix release where we unpin cf-units to allow the latest iris=3.4.0 to be installed. It also includes an update to the default configuration used when searching the ESGF for files, to account for a recent change of the CEDA ESGF index node hostname. The changelog contains only changes that were made to the main
branch.
Installation#
Set the version number on the development branches to one minor version more than the previous release (Pull request #1854) by @bouweandela
Unpin cf-units (Pull request #1770) by @bouweandela
Bug fixes#
Improve error handling if an esgf index node is offline (Pull request #1834) by @bouweandela
Automatic testing#
Removed unnecessary test that fails with iris 3.4.0 (Pull request #1846) by @schlunma
Update CEDA ESGF index node hostname (Pull request #1838) by @valeriupredoi
v2.7.0#
Highlights#
We have a new preprocessor function called ‘rolling_window_statistics’ implemented by @malininae
We have improved the support for native models, refactored native model fixes by adding common base class NativeDatasetFix, changed default DRS for reading native ICON output, and added tests for input/output filenames for ICON and EMAC on-the-fly CMORizer, all these features courtesy of @schlunma
Performance of preprocessor functions that use time dimensions has been sped up by two orders of magnitude thanks to contributions by @bouweandela
This release includes:
Backwards incompatible changes#
Change default DRS for reading native ICON output (Pull request #1705) by @schlunma
Bug fixes#
Add support for regions stored as MultiPolygon to extract_shape preprocessor (Pull request #1670) by @bouweandela
Fixed type annotations for Python 3.8 (Pull request #1700) by @schlunma
Core _io.concatenate() may fail due to case when one of the cubes is scalar - this fixes that (Pull request #1715) by @valeriupredoi
Pick up esmvalcore badge instead of esmvaltool one in README (Pull request #1749) by @valeriupredoi
Restore support for scalar cubes to time selection preprocessor functions (Pull request #1750) by @bouweandela
Fix calculation of precipitation flux in EMAC on-the-fly CMORizer (Pull request #1755) by @schlunma
Deprecations#
Remove deprecation warning for regrid schemes already deprecated for v2.7.0 (Pull request #1753) by @valeriupredoi
Documentation#
Add Met Office Installation Method (Pull request #1692) by @mo-tgeddes
Add MO-paths to config file (Pull request #1709) by @mo-tgeddes
Update MO obs4MIPs paths in the user configuration file (Pull request #1734) by @mo-tgeddes
Update Making a release section of the documentation (Pull request #1689) by @sloosvel
Added changelog for v2.7.0 (Pull request #1746) by @valeriupredoi
update CITATION.cff file with 2.7.0 release info (Pull request #1757) by @valeriupredoi
Improvements#
New preprocessor function ‘rolling_window_statistics’ (Pull request #1702) by @malininae
Remove pytest_flake8 plugin and use flake8 instead (Pull request #1722) by @valeriupredoi
Added CESM2 CMORizer (Pull request #1678) by @schlunma
Speed up functions that use time dimension (Pull request #1713) by @bouweandela
Modernize and minimize pylint configuration (Pull request #1726) by @bouweandela
Fixes for datasets#
Refactored native model fixes by adding common base class NativeDatasetFix (Pull request #1694) by @schlunma
Installation#
Pin netCDF4 != 1.6.1 since that seems to throw a flurry of Segmentation Faults (Pull request #1724) by @valeriupredoi
Automatic testing#
Pin flake8<5.0.0 since Circle CI tests are failing copiously (Pull request #1698) by @valeriupredoi
Added tests for input/output filenames for ICON and EMAC on-the-fly CMORizer (Pull request #1718) by @schlunma
Fix failed tests for Python<3.10 resulting from typing (Pull request #1748) by @schlunma
v2.6.0#
Highlights#
A new set of CMOR fixes is now available in order to load native EMAC model output and CMORize it on the fly. For details, see Supported native models: EMAC.
The version number of ESMValCore is now automatically generated using setuptools_scm, which extracts Python package versions from git metadata.
This release includes
Deprecations#
Deprecate the function esmvalcore.var_name_constraint (Pull request #1592) by @schlunma. This function is scheduled for removal in v2.8.0. Please use
iris.NameConstraint
with the keyword argument var_name instead: this is an exact replacement.
Bug fixes#
Added start_year and end_year attributes to derived variables (Pull request #1547) by @schlunma
Show all results on recipe results webpage (Pull request #1560) by @bouweandela
Regridding regular grids with similar coordinates (Pull request #1567) by @tomaslovato
Fix timerange wildcard search when deriving variables or downloading files (Pull request #1562) by @sloosvel
Fix force_derivation bug (Pull request #1627) by @sloosvel
Correct build-and-deploy-on-pypi action (Pull request #1634) by @sloosvel
Apply clip_timerange to time dependent fx variables (Pull request #1603) by @sloosvel
Correctly handle requests.exceptions.ConnectTimeout when an ESGF index node is offline (Pull request #1638) by @bouweandela
CMOR standard#
Added custom CMOR tables used for EMAC CMORizer (Pull request #1599) by @schlunma
Extended ICON CMORizer (Pull request #1549) by @schlunma
Add CMOR check exception for a basin coord named sector (Pull request #1612) by @dhohn
Custom user-defined location for custom CMOR tables (Pull request #1625) by @schlunma
Containerization#
Remove update command in Dockerfile (Pull request #1630) by @sloosvel
Community#
Add David Hohn to contributors’ list (Pull request #1586) by @valeriupredoi
Documentation#
[Github Actions Docs] Full explanation on how to use the GA test triggered by PR comment and added docs link for GA hosted runners (Pull request #1553) by @valeriupredoi
Update the command for building the documentation (Pull request #1556) by @bouweandela
Update documentation on running the tool (Pull request #1400) by @bouweandela
Add support for DKRZ-Levante (Pull request #1558) by @remi-kazeroni
Improved documentation on native dataset support (Pull request #1559) by @schlunma
Tweak extract_point preprocessor: explain what it returns if one point coord outside cube and add explicit test (Pull request #1584) by @valeriupredoi
Update CircleCI, readthedocs, and Docker configuration (Pull request #1588) by @bouweandela
Remove support for Mistral in config-user.yml (Pull request #1620) by @remi-kazeroni
Add changelog for v2.6.0rc1 (Pull request #1633) by @sloosvel
Add a note on transferring permissions to the release manager (Pull request #1645) by @bouweandela
Add documentation on building and uploading Docker images (Pull request #1644) by @bouweandela
Update documentation on ESMValTool module at DKRZ (Pull request #1647) by @remi-kazeroni
Expanded information on deprecations in changelog (Pull request #1658) by @schlunma
Improvements#
Removed trailing whitespace in custom CMOR tables (Pull request #1564) by @schlunma
Try searching multiple ESGF index nodes (Pull request #1561) by @bouweandela
Add CMIP6 amoc derivation case and add a test (Pull request #1577) by @valeriupredoi
Added EMAC CMORizer (Pull request #1554) by @schlunma
Improve performance of volume_statistics (Pull request #1545) by @sloosvel
Fixes for datasets#
Fixes of ocean variables in multiple CMIP6 datasets (Pull request #1566) by @tomaslovato
Ensure lat/lon bounds in FGOALS-l3 atmos variables are contiguous (Pull request #1571) by @sloosvel
Added AllVars fix for CMIP6’s ICON-ESM-LR (Pull request #1582) by @schlunma
Installation#
Removed package/meta.yml (Pull request #1540) by @schlunma
Pinned iris>=3.2.1 (Pull request #1552) by @schlunma
Use setuptools-scm to automatically generate the version number (Pull request #1578) by @bouweandela
Pin cf-units to lower than 3.1.0 to temporarily avoid changes within new version related to calendars (Pull request #1659) by @valeriupredoi
Preprocessor#
Allowed special case for unit conversion of precipitation (kg m-2 s-1 <–> mm day-1) (Pull request #1574) by @schlunma
Add general extract_coordinate_points preprocessor (Pull request #1581) by @sloosvel
Add preprocessor accumulate_coordinate (Pull request #1281) by @jvegreg
Add axis_statistics and improve depth_integration (Pull request #1589) by @sloosvel
Release#
Increase version number for ESMValCore v2.6.0rc1 (Pull request #1632) by @sloosvel
Update changelog and version for 2.6rc3 (Pull request #1646) by @sloosvel
Add changelog for rc4 (Pull request #1662) by @sloosvel
Automatic testing#
Refresh CircleCI cache weekly (Pull request #1597) by @bouweandela
Use correct cache restore key on CircleCI (Pull request #1598) by @bouweandela
Install git and ssh before checking out code on CircleCI (Pull request #1601) by @bouweandela
Fetch all history in Github Action tests (Pull request #1622) by @sloosvel
Test Github Actions dashboard badge from meercode.io (Pull request #1640) by @valeriupredoi
Improve esmvalcore.esgf unit test (Pull request #1650) by @bouweandela
Variable Derivation#
Added derivation of hfns (Pull request #1594) by @schlunma
v2.5.0#
Highlights#
The new preprocessor
extract_location()
can extract arbitrary locations on the Earth using the geopy package that connects to OpenStreetMap. For details, see Extract location.Time ranges can now be extracted using the ISO 8601 format. In addition, wildcards are allowed, which makes the time selection much more flexible. For details, see Recipe section: Datasets.
The new preprocessor
ensemble_statistics()
can calculate arbitrary statistics over all ensemble members of a simulation. In addition, the preprocessormulti_model_statistics()
now accepts the keywordgroupy
, which allows the calculation of multi-model statistics over arbitrary multi-model ensembles. For details, see Ensemble statistics and Multi-model statistics.
This release includes
Backwards incompatible changes#
Update Cordex section in config-developer.yml (Pull request #1303) by @francesco-cmcc. This changes the naming convention of ESMValCore’s output files from CORDEX dataset. This only affects recipes that use CORDEX data. Most likely, no changes in diagnostics are necessary; however, if code relies on the specific naming convention of files, it might need to be adapted.
Dropped Python 3.7 (Pull request #1530) by @schlunma. ESMValCore v2.5.0 dropped support for Python 3.7. From now on Python >=3.8 is required to install ESMValCore. The main reason for this is that conda-forge dropped support for Python 3.7 for OSX and arm64 (more details are given here).
Bug fixes#
Fix extract_shape when fx vars are present (Pull request #1403) by @sloosvel
Added support of extra_facets to fx variables added by the preprocessor (Pull request #1399) by @schlunma
Augmented input for derived variables with extra_facets (Pull request #1412) by @schlunma
Correctly use masked arrays after unstructured_nearest regridding (Pull request #1414) by @schlunma
Fixing the broken derivation script for XCH4 (and XCO2) (Pull request #1428) by @hb326
Ignore .pymon-journal file in test discovery (Pull request #1436) by @valeriupredoi
Fixed bug that caused automatic download to fail in rare cases (Pull request #1442) by @schlunma
Add new JULIA_LOAD_PATH to diagnostic task test (Pull request #1444) by @valeriupredoi
Fix provenance file permissions (Pull request #1468) by @bouweandela
Fixed usage of statistics=std_dev option in multi-model statistics preprocessors (Pull request #1478) by @schlunma
Removed scalar coordinates p0 and ptop prior to merge in multi_model_statistics (Pull request #1471) by @axel-lauer
Added dataset and alias attributes to multi_model_statistics output (Pull request #1483) by @schlunma
Fixed issues with multi-model-statistics timeranges (Pull request #1486) by @schlunma
Fixed output messages for CMOR logging (Pull request #1494) by @schlunma
Fixed clip_timerange if only a single time point is extracted (Pull request #1497) by @schlunma
Fixed chunking in multi_model_statistics (Pull request #1500) by @schlunma
Fixed renaming of auxiliary coordinates in multi_model_statistics if coordinates are equal (Pull request #1502) by @schlunma
Fixed timerange selection for automatic downloads (Pull request #1517) by @schlunma
Fixed chunking in multi_model_statistics (Pull request #1524) by @schlunma
Deprecations#
Renamed vertical regridding schemes (Pull request #1429) by @schlunma. Old regridding schemes are supported until v2.7.0. For details, see Vertical interpolation schemes.
Documentation#
Remove duplicate entries in changelog (Pull request #1391) by @zklaus
Documentation on how to use HPC central installations (Pull request #1409) by @valeriupredoi
Correct brackets in preprocessor documentation for list of seasons (Pull request #1420) by @bouweandela
Add Python=3.10 to package info, update Circle CI auto install and documentation for Python=3.10 (Pull request #1432) by @valeriupredoi
Reverted unintentional change in .zenodo.json (Pull request #1452) by @schlunma
Synchronized config-user.yml with version from ESMValTool (Pull request #1453) by @schlunma
Solved issues in configuration files (Pull request #1457) by @schlunma
Add direct link to download conda lock file in the install documentation (Pull request #1462) by @valeriupredoi
CITATION.cff fix and automatic validation of citation metadata (Pull request #1467) by @valeriupredoi
Updated documentation on how to deprecate features (Pull request #1426) by @schlunma
Added reference hook to conda lock in documentation install section (Pull request #1473) by @valeriupredoi
Increased ESMValCore version to 2.5.0rc1 (Pull request #1477) by @schlunma
Added changelog for v2.5.0 release (Pull request #1476) by @schlunma
Increased ESMValCore version to 2.5.0rc2 (Pull request #1487) by @schlunma
Added some authors to citation and zenodo files (Pull request #1488) by @SarahAlidoost
Restored scipy intersphinx mapping (Pull request #1491) by @schlunma
Increased ESMValCore version to 2.5.0rc3 (Pull request #1504) by @schlunma
Fix download instructions for the MSWEP dataset (Pull request #1506) by @remi-kazeroni
Documentation updated for the new cmorizer framework (Pull request #1417) by @remi-kazeroni
Added tests for duplicates in changelog and removed duplicates (Pull request #1508) by @schlunma
Increased ESMValCore version to 2.5.0rc4 (Pull request #1519) by @schlunma
Add Github Actions Test badge in README (Pull request #1526) by @valeriupredoi
Increased ESMValCore version to 2.5.0rc5 (Pull request #1529) by @schlunma
Increased ESMValCore version to 2.5.0rc6 (Pull request #1532) by @schlunma
Fixes for datasets#
Added fix for AIRS v2.1 (obs4mips) (Pull request #1472) by @axel-lauer
Preprocessor#
Added bias preprocessor (Pull request #1406) by @schlunma
Improve error messages when a preprocessor is failing (Pull request #1408) by @schlunma
Added option to explicitly not use fx variables in preprocessors (Pull request #1416) by @schlunma
Add extract_location preprocessor to extract town, city, mountains etc - anything specifiable by a location (Pull request #1251) by @jvegreg
Add ensemble statistics preprocessor and ‘groupby’ option for multimodel (Pull request #673) by @sloosvel
Generic regridding preprocessor (Pull request #1448) by @zklaus
Automatic testing#
Add pandas as dependency :panda_face: (Pull request #1402) by @valeriupredoi
Fixed tests for python 3.7 (Pull request #1410) by @schlunma
Remove accessing .xml() cube method from test (Pull request #1419) by @valeriupredoi
Remove flag to use pip 2020 solver from Github Action pip install command on OSX (Pull request #1357) by @valeriupredoi
Add Python=3.10 to Github Actions and switch to Python=3.10 for the Github Action that builds the PyPi package (Pull request #1430) by @valeriupredoi
Pin flake8<4 to keep getting relevant error traces when tests fail with FLAKE8 issues (Pull request #1434) by @valeriupredoi
Implementing conda lock (Pull request #1164) by @valeriupredoi
Relocate pytest-monitor outputted database .pymon so .pymon-journal file should not be looked for by pytest (Pull request #1441) by @valeriupredoi
Switch to Mambaforge in Github Actions tests (Pull request #1438) by @valeriupredoi
Turn off conda lock file creation on any push on main branch from Github Action test (Pull request #1489) by @valeriupredoi
Add DRS path test for IPSLCM files (Pull request #1490) by @senesis
Add a test module that runs tests of iris I/O every time we notice serious bugs there (Pull request #1510) by @valeriupredoi
[Github Actions] Trigger Github Actions tests (run-tests.yml workflow) from a comment in a PR (Pull request #1520) by @valeriupredoi
Update Linux condalock file (various pull requests) github-actions[bot]
Installation#
Move nested-lookup dependency to environment.yml to be installed from conda-forge instead of PyPi (Pull request #1481) by @valeriupredoi
Pinned iris (Pull request #1511) by @schlunma
Updated dependencies (Pull request #1521) by @schlunma
Pinned iris<3.2.0 (Pull request #1525) by @schlunma
Improvements#
Allow to load all files, first X years or last X years in an experiment (Pull request #1133) by @sloosvel
Filter tasks earlier (Pull request #1264) by @jvegreg
Added earlier validation for command line arguments (Pull request #1435) by @schlunma
Remove profile_diagnostic from diagnostic settings and increase test coverage of _task.py (Pull request #1404) by @valeriupredoi
Add output2 to the product extra facet of CMIP5 data (Pull request #1514) by @remi-kazeroni
Speed up ESGF search (Pull request #1512) by @bouweandela
v2.4.0#
Highlights#
ESMValCore now has the ability to automatically download missing data from ESGF. For details, see Data Retrieval.
ESMValCore now also can resume an earlier run. This is useful to re-use expensive preprocessor results. For details, see Running.
This release includes
Bug fixes#
Crop on the ID-selected region(s) and not on the whole shapefile (Pull request #1151) by @stefsmeets
Add ‘comment’ to list of removed attributes (Pull request #1244) by @Peter9192
Speed up multimodel statistics and fix bug in peak computation (Pull request #1301) by @bouweandela
No longer make plots of provenance (Pull request #1307) by @bouweandela
No longer embed provenance in output files (Pull request #1306) by @bouweandela
Removed automatic addition of areacello to obs4mips datasets (Pull request #1316) by @schlunma
Pin docutils <0.17 to fix bullet lists on readthedocs (Pull request #1320) by @zklaus
Fix obs4MIPs capitalization (Pull request #1328) by @bouweandela
Fix Python 3.7 tests (Pull request #1330) by @bouweandela
Handle fx variables in extract_levels and some time operations (Pull request #1269) by @sloosvel
Refactored mask regridding for irregular grids (fixes #772) (Pull request #865) by @zklaus
Fix da.broadcast_to call when the fx cube has different shape than target data cube (Pull request #1350) by @valeriupredoi
Add tests for _aggregate_time_fx (Pull request #1354) by @sloosvel
Fix extra facets (Pull request #1360) by @bouweandela
Pin pip!=21.3 to avoid pypa/pip#10573 with editable installs (Pull request #1359) by @zklaus
Add a custom date2num function to deal with changes in cftime (Pull request #1373) by @zklaus
Removed custom version of AtmosphereSigmaFactory (Pull request #1382) by @schlunma
Deprecations#
Remove write_netcdf and write_plots from config-user.yml (Pull request #1300) by @bouweandela
Documentation#
Add link to plot directory in index.html (Pull request #1256) by @stefsmeets
Work around issue with yapf not following PEP8 (Pull request #1277) by @bouweandela
Update the core development team (Pull request #1278) by @bouweandela
Update the documentation of the provenance interface (Pull request #1305) by @bouweandela
Update version number to first release candidate 2.4.0rc1 (Pull request #1363) by @zklaus
Update to new ESMValTool logo (Pull request #1374) by @zklaus
Update version number for third release candidate 2.4.0rc3 (Pull request #1384) by @zklaus
Update changelog for 2.4.0rc3 (Pull request #1385) by @zklaus
Update version number to final 2.4.0 release (Pull request #1389) by @zklaus
Update changelog for 2.4.0 (Pull request #1366) by @zklaus
Fixes for datasets#
Add fix for differing latitude coordinate between historical and ssp585 in MPI-ESM1-2-HR r2i1p1f1 (Pull request #1292) by @bouweandela
Add fixes for time and latitude coordinate of EC-Earth3 r3i1p1f1 (Pull request #1290) by @bouweandela
Apply latitude fix to all CCSM4 variables (Pull request #1295) by @bouweandela
Fix lat and lon bounds for FGOALS-g3 mrsos (Pull request #1289) by @thomascrocker
Add grid fix for tos in fgoals-f3-l (Pull request #1326) by @sloosvel
Add fix for CIESM pr (Pull request #1344) by @bouweandela
Fix DRS for IPSLCM : split attribute ‘freq’ into : ‘out’ and ‘freq’ (Pull request #1304) by @senesis
CMOR standard#
Remove history attribute from coords (Pull request #1276) by @jvegreg
Increased flexibility of CMOR checks for datasets with generic alevel coordinates (Pull request #1032) by @schlunma
Automatically fix small deviations in vertical levels (Pull request #1177) by @bouweandela
Adding standard names to the custom tables of the rlns and rsns variables (Pull request #1386) by @remi-kazeroni
Preprocessor#
Implemented fully lazy climate_statistics (Pull request #1194) by @schlunma
Run the multimodel statistics preprocessor last (Pull request #1299) by @bouweandela
Automatic testing#
Improving test coverage for _task.py (Pull request #514) by @valeriupredoi
Upload coverage to codecov (Pull request #1190) by @bouweandela
Improve codecov status checks (Pull request #1195) by @bouweandela
Fix curl install in CircleCI (Pull request #1228) by @jvegreg
Drop support for Python 3.6 (Pull request #1200) by @valeriupredoi
Allow more recent version of scipy (Pull request #1182) by @schlunma
Speed up conda build conda_build Circle test by using mamba solver via boa (and use it for Github Actions test too) (Pull request #1243) by @valeriupredoi
Fix numpy deprecation warnings (Pull request #1274) by @bouweandela
Unpin upper bound for iris (previously was at <3.0.4) (Pull request #1275) by @valeriupredoi
Modernize conda_install test on Circle CI by installing from conda-forge with Python 3.9 and change install instructions in documentation (Pull request #1280) by @valeriupredoi
Run a nightly Github Actions workflow to monitor tests memory per test (configurable for other metrics too) (Pull request #1284) by @valeriupredoi
Speed up tests of tasks (Pull request #1302) by @bouweandela
Fix upper case to lower case variables and functions for flake compliance in tests/unit/preprocessor/_regrid/test_extract_levels.py (Pull request #1347) by @valeriupredoi
Cleaned up a bit Github Actions workflows (Pull request #1345) by @valeriupredoi
Update circleci jobs: renaming tests to more descriptive names and removing conda build test (Pull request #1351) by @zklaus
Pin iris to latest >=3.1.0 (Pull request #1341) by @valeriupredoi
Installation#
Pin esmpy to anything but 8.1.0 since that particular one changes the CPU affinity (Pull request #1310) by @valeriupredoi
Improvements#
Add a more friendly and useful message when using default config file (Pull request #1233) by @valeriupredoi
Replace os.walk by glob.glob in data finder (only look for data in the specified locations) (Pull request #1261) by @bouweandela
Machine-specific directories for auxiliary data in the config-user.yml file (Pull request #1268) by @remi-kazeroni
Add an option to download missing data from ESGF (Pull request #1217) by @bouweandela
Speed up provenance recording (Pull request #1327) by @bouweandela
Improve results web page (Pull request #1332) by @bouweandela
Move institutes from config-developer.yml to default extra facets config and add wildcard support for extra facets (Pull request #1259) by @bouweandela
Add support for re-using preprocessor output from previous runs (Pull request #1321) by @bouweandela
Log fewer messages to screen and hide stack trace for known recipe errors (Pull request #1296) by @bouweandela
Log ESMValCore and ESMValTool versions when running (Pull request #1263) by @jvegreg
Add “grid” as a tag to the output file template for CMIP6 (Pull request #1356) by @zklaus
Implemented ICON project to read native ICON model output (Pull request #1079) by @bsolino
v2.3.1#
This release includes
Bug fixes#
Update config-user.yml template with correct drs entries for CEDA-JASMIN (Pull request #1184) by @valeriupredoi
Enhancing MIROC5 fix for hfls and evspsbl (Pull request #1192) by @katjaweigel
Fix alignment of daily data with inconsistent calendars in multimodel statistics (Pull request #1212) by @Peter9192
Pin cf-units, remove github actions test for Python 3.6 and fix test_access1_0 and test_access1_3 to use cf-units for comparisons (Pull request #1197) by @valeriupredoi
Fixed search for fx files when no
mip
is given (Pull request #1216) by @schlunmaMake sure climate statistics always returns original dtype (Pull request #1237) by @zklaus
Bugfix for regional regridding when non-integer range is passed (Pull request #1231) by @stefsmeets
Make sure area_statistics preprocessor always returns original dtype (Pull request #1239) by @zklaus
Add “.” (dot) as allowed separation character for the time range group (Pull request #1248) by @zklaus
Documentation#
Add a link to the instructions to use pre-installed versions on HPC clusters (Pull request #1186) by @remi-kazeroni
Bugfix release: set version to 2.3.1 (Pull request #1253) by @zklaus
Fixes for datasets#
Set circular attribute in MCM-UA-1-0 fix (Pull request #1178) by @sloosvel
Fixed time coordinate of MIROC-ESM (Pull request #1188) by @schlunma
Preprocessor#
Filter warnings about collapsing multi-model dimension in multimodel statistics preprocessor function (Pull request #1215) by @bouweandela
Remove fx variables before computing multimodel statistics (Pull request #1220) by @sloosvel
Installation#
Pin lower bound for iris to 3.0.2 (Pull request #1206) by @valeriupredoi
Pin iris<3.0.4 to ensure we still (sort of) support Python 3.6 (Pull request #1252) by @valeriupredoi
Improvements#
Add test to verify behaviour for scalar height coord for tas in multi-model (Pull request #1209) by @Peter9192
Sort missing years in “No input data available for years” message (Pull request #1225) by @ledm
v2.3.0#
This release includes
Bug fixes#
Extend preprocessor multi_model_statistics to handle data with “altitude” coordinate (Pull request #1010) by @axel-lauer
Remove scripts included with CMOR tables (Pull request #1011) by @bouweandela
Avoid side effects in extract_season (Pull request #1019) by @jvegreg
Use nearest scheme to avoid interpolation errors with masked data in regression test (Pull request #1021) by @stefsmeets
Move _get_time_bounds from preprocessor._time to cmor.check to avoid circular import with cmor module (Pull request #1037) by @valeriupredoi
Fix test that makes conda build fail (Pull request #1046) by @valeriupredoi
Fix ‘positive’ attribute for rsns/rlns variables (Pull request #1051) by @lukasbrunner
Added preprocessor mask_multimodel (Pull request #767) by @schlunma
Fix bug when fixing bounds after fixing longitude values (Pull request #1057) by @sloosvel
Run conda build parallel AND sequential tests (Pull request #1065) by @valeriupredoi
Add key to id_prop (Pull request #1071) by @lukasbrunner
Fix bounds after reversing coordinate values (Pull request #1061) by @sloosvel
Fixed –skip-nonexistent option (Pull request #1093) by @schlunma
Do not consider CMIP5 variable sit to be the same as sithick from CMIP6 (Pull request #1033) by @bouweandela
Improve finding date range in filenames (enforces separators) (Pull request #1145) by @senesis
Review fx handling (Pull request #1147) by @sloosvel
Fix lru cache decorator with explicit call to method (Pull request #1172) by @valeriupredoi
Update _volume.py (Pull request #1174) by @ledm
Deprecations#
Documentation#
Final changelog for 2.3.0 (Pull request #1163) by @zklaus
Set version to 2.3.0 (Pull request #1162) by @zklaus
Fix documentation build (Pull request #1006) by @bouweandela
Add labels required for linking from ESMValTool docs (Pull request #1038) by @bouweandela
Update contribution guidelines (Pull request #1047) by @bouweandela
Fix basestring references in documentation (Pull request #1106) by @jvegreg
Updated references master to main (Pull request #1132) by @axel-lauer
Add instructions how to use the central installation at DKRZ-Mistral (Pull request #1155) by @remi-kazeroni
Fixes for datasets#
Added fixes for various CMIP5 datasets, variable cl (3-dim cloud fraction) (Pull request #1017) by @axel-lauer
Added fixes for hybrid level coordinates of CESM2 models (Pull request #882) by @schlunma
Extending LWP fix for CMIP6 models (Pull request #1049) by @axel-lauer
Add fixes for the net & up radiation variables from ERA5 (Pull request #1052) by @lukasbrunner
Add derived variable rsus (Pull request #1053) by @lukasbrunner
Supported mip-level fixes (Pull request #1095) by @schlunma
Fix erroneous use of grid_latitude and grid_longitude and cleaned ocean grid fixes (Pull request #1092) by @schlunma
Fix for pr of miroc5 (Pull request #1110) by @remi-kazeroni
Ocean depth fix for cnrm_esm2_1, gfdl_esm4, ipsl_cm6a_lr datasets + mcm_ua_1_0 (Pull request #1098) by @tomaslovato
Fix for uas variable of the MCM_UA_1_0 dataset (Pull request #1102) by @remi-kazeroni
Fixes for sos and siconc of BCC models (Pull request #1090) by @remi-kazeroni
Run fgco2 fix for all CESM2 models (Pull request #1108) by @LisaBock
Fixes for the siconc variable of CMIP6 models (Pull request #1105) by @remi-kazeroni
Fix wrong sign for land surface flux (Pull request #1113) by @LisaBock
Fix for pr of EC_EARTH (Pull request #1116) by @remi-kazeroni
CMOR standard#
Format cmor related files (Pull request #976) by @jvegreg
Check presence of time bounds and guess them if needed (Pull request #849) by @sloosvel
Add custom variable “tasaga” (Pull request #1118) by @LisaBock
Find files for CMIP6 DCPP startdates (Pull request #771) by @sloosvel
Preprocessor#
Update tests for multimodel statistics preprocessor (Pull request #1023) by @stefsmeets
Raise in extract_season and extract_month if result is None (Pull request #1041) by @jvegreg
Allow selection of shapes in extract_shape (Pull request #764) by @jvegreg
Add option for regional regridding to regrid preprocessor (Pull request #1034) by @stefsmeets
Load fx variables as cube cell measures / ancillary variables (Pull request #999) by @sloosvel
Check horizontal grid before regridding (Pull request #507) by @BenMGeo
Clip irregular grids (Pull request #245) by @bouweandela
Use native iris functions in multi-model statistics (Pull request #1150) by @Peter9192
Notebook API (experimental)#
Automatic testing#
Report coverage for tests that run on any pull request (Pull request #994) by @bouweandela
Install ESMValTool sample data from PyPI (Pull request #998) by @jvegreg
Fix tests for multi-processing with spawn method (i.e. macOSX with Python>3.8) (Pull request #1003) by @bvreede
Switch to running the Github Action test workflow every 3 hours in single thread mode to observe if Segmentation Faults occur (Pull request #1022) by @valeriupredoi
Revert to original Github Actions test workflow removing the 3-hourly test run with -n 1 (Pull request #1025) by @valeriupredoi
Avoid stale cache for multimodel statistics regression tests (Pull request #1030) by @bouweandela
Add newer Python versions in OSX to Github Actions (Pull request #1035) by @bvreede
Add tests for type annotations with mypy (Pull request #1042) by @stefsmeets
Run problematic cmor tests sequentially to avoid segmentation faults on CircleCI (Pull request #1064) by @valeriupredoi
Test installation of esmvalcore from conda-forge (Pull request #1075) by @valeriupredoi
Added additional test cases for integration tests of data_finder.py (Pull request #1087) by @schlunma
Pin cf-units and fix tests (cf-units>=2.1.5) (Pull request #1140) by @valeriupredoi
Fix failing CircleCI tests (Pull request #1167) by @bouweandela
Fix test failing due to fx files chosen differently on different OS’s (Pull request #1169) by @valeriupredoi
Compare datetimes instead of strings in _fixes/cmip5/test_access1_X.py (Pull request #1173) by @valeriupredoi
Pin Python to 3.9 in environment.yml on CircleCI and skip mypy tests in conda build (Pull request #1176) by @bouweandela
Installation#
Update yamale to version 3 (Pull request #1059) by @zklaus
Improvements#
Refactor diagnostics / tags management (Pull request #939) by @stefsmeets
Support multiple paths in input_dir (Pull request #1000) by @jvegreg
Generate HTML report with recipe output (Pull request #991) by @stefsmeets
Add timeout to requests.get in _citation.py (Pull request #1091) by @SarahAlidoost
Add SYNDA drs for CMIP5 and CMIP6 (closes #582) (Pull request #583) by @zklaus
Add basic support for variable mappings (Pull request #1124) by @zklaus
Handle IPSL-CM6 (Pull request #1153) by @senesis
v2.2.0#
Highlights#
ESMValCore is now using the recently released Iris 3. We acknowledge that this change may impact your work, as Iris 3 introduces several changes that are not backward-compatible, but we think that moving forward is the best decision for the tool in the long term.
This release is also the first one including support for downloading CMIP6 data using Synda and we have also started supporting Python 3.9. Give it a try!
This release includes
Bug fixes#
Fix path settings for DKRZ/Mistral (Pull request #852) by @bouweandela
Change logic for calling the diagnostic script to avoid problems with scripts where the executable bit is accidentally set (Pull request #877) by @bouweandela
Fix overwriting in generic level check (Pull request #886) by @sloosvel
Add double quotes to script args in rerun screen message when using vprof profiling (Pull request #897) by @valeriupredoi
Simplify time handling in multi-model statistics preprocessor (Pull request #685) by @Peter9192
Fix links to Iris documentation (Pull request #966) by @jvegreg
Bugfix: Fix units for MSWEP data (Pull request #986) by @stefsmeets
Deprecations#
Deprecate defining write_plots and write_netcdf in config-user file (Pull request #808) by @bouweandela
Documentation#
Fix numbering of steps in release instructions (Pull request #838) by @bouweandela
Add labels to changelogs of individual versions for easy reference (Pull request #899) by @zklaus
Make CircleCI badge specific to main branch (Pull request #902) by @bouweandela
Fix docker build badge url (Pull request #906) by @stefsmeets
Update github PR template (Pull request #909) by @stefsmeets
Refer to ESMValTool GitHub discussions page in the error message (Pull request #900) by @bouweandela
Support automatically closing issues (Pull request #922) by @bouweandela
Fix checkboxes in PR template (Pull request #931) by @stefsmeets
Change in config-user defaults and documentation with new location for esmeval OBS data on JASMIN (Pull request #958) by @valeriupredoi
Update Core Team info (Pull request #942) by @axel-lauer
Update iris documentation URL for sphinx (Pull request #964) by @bouweandela
Set version to 2.2.0 (Pull request #977) by @jvegreg
Add first draft of v2.2.0 changelog (Pull request #983) by @jvegreg
Add checkbox in PR template to assign labels (Pull request #985) by @jvegreg
Update install.rst (Pull request #848) by @bascrezee
Change the order of the publication steps (Pull request #984) by @jvegreg
Add instructions how to use esmvaltool from HPC central installations (Pull request #841) by @valeriupredoi
Fixes for datasets#
Fixing unit for derived variable rsnstcsnorm to prevent overcorrection2 (Pull request #846) by @katjaweigel
Cmip6 fix awi cm 1 1 mr (Pull request #822) by @mwjury
Cmip6 fix ec earth3 veg (Pull request #836) by @mwjury
Changed latitude longitude fix from Tas to AllVars. (Pull request #916) by @katjaweigel
Fix for precipitation (pr) to use ERA5-Land cmorizer (Pull request #879) by @katjaweigel
Cmip6 fix ec earth3 (Pull request #837) by @mwjury
Cmip6_fix_fgoals_f3_l_Amon_time_bnds (Pull request #831) by @mwjury
Fix for FGOALS-f3-L sftlf (Pull request #667) by @mwjury
Improve ACCESS-CM2 and ACCESS-ESM1-5 fixes and add CIESM and CESM2-WACCM-FV2 fixes for cl, clw and cli (Pull request #635) by @axel-lauer
Add fixes for cl, cli, clw and tas for several CMIP6 models (Pull request #955) by @schlunma
Dataset fixes for MSWEP (Pull request #969) by @stefsmeets
Dataset fixes for: ACCESS-ESM1-5, CanESM5, CanESM5 for carbon cycle (Pull request #947) by @bettina-gier
Fixes for KIOST-ESM (CMIP6) (Pull request #904) by @remi-kazeroni
Fixes for AWI-ESM-1-1-LR (CMIP6, piControl) (Pull request #911) by @remi-kazeroni
CMOR standard#
CMOR check generic level coordinates in CMIP6 (Pull request #598) by @sloosvel
Update CMIP6 tables to 6.9.33 (Pull request #919) by @jvegreg
Adding custom variables for tas uncertainty (Pull request #924) by @LisaBock
Remove monotonicity coordinate check for unstructured grids (Pull request #965) by @jvegreg
Preprocessor#
Added clip_start_end_year preprocessor (Pull request #796) by @schlunma
Add support for downloading CMIP6 data with Synda (Pull request #699) by @bouweandela
Add multimodel tests using real data (Pull request #856) by @stefsmeets
Add plev/altitude conversion to extract_levels (Pull request #892) by @axel-lauer
Add possibility of custom season extraction. (Pull request #247) by @mwjury
Adding the ability to derive xch4 (Pull request #783) by @hb326
Add preprocessor function to resample time and compute x-hourly statistics (Pull request #696) by @jvegreg
Fix duplication in preprocessors DEFAULT_ORDER introduced in #696 (Pull request #973) by @jvegreg
Use consistent precision in multi-model statistics calculation and update reference data for tests (Pull request #941) by @Peter9192
Refactor multi-model statistics code to facilitate ensemble stats and lazy evaluation (Pull request #949) by @Peter9192
Add option to exclude input cubes in output of multimodel statistics to solve an issue introduced by #949 (Pull request #978) by @Peter9192
Automatic testing#
Pin cftime>=1.3.0 to have newer string formatting in and fix two tests (Pull request #878) by @valeriupredoi
Switched miniconda conda setup hooks for Github Actions workflows (Pull request #873) by @valeriupredoi
Add test for latest version resolver (Pull request #874) by @stefsmeets
Update codacy coverage reporter to fix coverage (Pull request #905) by @nielsdrost
Avoid hardcoded year in tests and add improvement to plev test case (Pull request #921) by @bouweandela
Pin scipy to less than 1.6.0 until Issue #927 gets resolved (Pull request #928) by @valeriupredoi
Github Actions: change time when conda install test runs (Pull request #930) by @valeriupredoi
Remove redundant test line from test_utils.py (Pull request #935) by @valeriupredoi
Removed netCDF4 package from integration tests of fixes (Pull request #938) by @schlunma
Use new conda environment for installing ESMValCore in Docker containers (Pull request #951) by @bouweandela
Notebook API (experimental)#
Implement importable config object in experimental API submodule (Pull request #868) by @stefsmeets
Add loading and running recipes to the notebook API (Pull request #907) by @stefsmeets
Add displaying and loading of recipe output to the notebook API (Pull request #957) by @stefsmeets
Add functionality to run single diagnostic task to notebook API (Pull request #962) by @stefsmeets
Improvements#
Create CODEOWNERS file (Pull request #809) by @jvegreg
Remove code needed for Python <3.6 (Pull request #844) by @bouweandela
Add requests as a dependency (Pull request #850) by @bouweandela
Pin Python to less than 3.9 (Pull request #870) by @valeriupredoi
Remove numba dependency (Pull request #880) by @schlunma
Add Listing and finding recipes to the experimental notebook API (Pull request #901) by @stefsmeets
Skip variables that don’t have dataset or additional_dataset keys (Pull request #860) by @valeriupredoi
Refactor logging configuration (Pull request #933) by @stefsmeets
Xco2 derivation (Pull request #913) by @bettina-gier
Working environment for Python 3.9 (pin to !=3.9.0) (Pull request #885) by @valeriupredoi
Print source file when using config get_config_user command (Pull request #960) by @valeriupredoi
Switch to Iris 3 (Pull request #819) by @stefsmeets
Refactor tasks (Pull request #959) by @stefsmeets
Restore task summary in debug log after #959 (Pull request #981) by @bouweandela
Pin pre-commit hooks (Pull request #974) by @stefsmeets
Improve error messages when data is missing (Pull request #917) by @jvegreg
Set remove_preproc_dir to false in default config-user (Pull request #979) by @valeriupredoi
Move fiona to be installed from conda forge (Pull request #987) by @valeriupredoi
Re-added fiona in setup.py (Pull request #990) by @valeriupredoi
v2.1.0#
This release includes
Bug fixes#
Set unit=1 if anomalies are standardized (Pull request #727) by @bascrezee
Fix crash for FGOALS-g2 variables without longitude coordinate (Pull request #729) by @bouweandela
Improve variable alias management (Pull request #595) by @jvegreg
Fix area_statistics fx files loading (Pull request #798) by @jvegreg
Fix units after derivation (Pull request #754) by @schlunma
Documentation#
Update v2.0.0 release notes with final additions (Pull request #722) by @bouweandela
Update package description in setup.py (Pull request #725) by @mattiarighi
Add installation instructions for pip installation (Pull request #735) by @bouweandela
Improve config-user documentation (Pull request #740) by @bouweandela
Update the zenodo file with contributors (Pull request #807) by @valeriupredoi
Improve command line run documentation (Pull request #721) by @jvegreg
Update the zenodo file with contributors (continued) (Pull request #810) by @valeriupredoi
Improvements#
Reduce size of docker image (Pull request #723) by @jvegreg
Add ‘test’ extra to installation, used by docker development tag (Pull request #733) by @bouweandela
Correct dockerhub link (Pull request #736) by @bouweandela
Create action-install-from-pypi.yml (Pull request #734) by @valeriupredoi
Add pre-commit for linting/formatting (Pull request #766) by @stefsmeets
Run tests in parallel and when building conda package (Pull request #745) by @bouweandela
Readable exclude pattern for pre-commit (Pull request #770) by @stefsmeets
Github Actions Tests (Pull request #732) by @valeriupredoi
Remove isort setup to fix formatting conflict with yapf (Pull request #778) by @stefsmeets
Fix yapf-isort import formatting conflict (Fixes #777) (Pull request #784) by @stefsmeets
Sorted output for esmvaltool recipes list (Pull request #790) by @stefsmeets
Replace vmprof with vprof (Pull request #780) by @valeriupredoi
Update CMIP6 tables to 6.9.32 (Pull request #706) by @jvegreg
Default config-user path now set in config-user read function (Pull request #791) by @jvegreg
Add custom variable lweGrace (Pull request #692) by @bascrezee
Create Github Actions workflow to build and deploy on Test PyPi and PyPi (Pull request #820) by @valeriupredoi
Build and publish the esmvalcore package to conda via Github Actions workflow (Pull request #825) by @valeriupredoi
Fixes for datasets#
Fix cmip6 models (Pull request #629) by @npgillett
Fix siconca variable in EC-Earth3 and EC-Earth3-Veg models in amip simulation (Pull request #702) by @egalytska
Preprocessor#
Move cmor_check_data to early in preprocessing chain (Pull request #743) by @bouweandela
Add RMS iris analysis operator to statistics preprocessor functions (Pull request #747) by @pcosbsc
Add surface chlorophyll concentration as a derived variable (Pull request #720) by @sloosvel
Use dask to reduce memory consumption of extract_levels for masked data (Pull request #776) by @valeriupredoi
v2.0.0#
This release includes
Bug fixes#
Fixed derivation of co2s (Pull request #594) by @schlunma
Padding while cropping needs to stay within sane bounds for shapefiles that span the whole Earth (Pull request #626) by @valeriupredoi
Fix concatenation of a single cube (Pull request #655) by @bouweandela
Fix mask fx dict handling not to fail if empty list in values (Pull request #661) by @valeriupredoi
Preserve metadata during anomalies computation when using iris cubes difference (Pull request #652) by @valeriupredoi
Avoid crashing when there is directory ‘esmvaltool’ in the current working directory (Pull request #672) by @valeriupredoi
Solve bug in ACCESS1 dataset fix for calendar. (Pull request #671) by @Peter9192
Fix the syntax for adding multiple ensemble members from the same dataset (Pull request #678) by @SarahAlidoost
Fix bug that made preprocessor with fx files fail in rare cases (Pull request #670) by @schlunma
Add support for string coordinates (Pull request #657) by @jvegreg
Fixed the shape extraction to account for wraparound shapefile coords (Pull request #319) by @valeriupredoi
Fixed bug in time weights calculation (Pull request #695) by @schlunma
Fix diagnostic filter (Pull request #713) by @jvegreg
Documentation#
Add pandas as a requirement for building the documentation (Pull request #607) by @bouweandela
Document default order in which preprocessor functions are applied (Pull request #633) by @bouweandela
Add pointers about data loading and CF standards to documentation (Pull request #571) by @valeriupredoi
Config file populated with site-specific data paths examples (Pull request #619) by @valeriupredoi
Update Codacy badges (Pull request #643) by @bouweandela
Update copyright info on readthedocs (Pull request #668) by @bouweandela
Updated references to documentation (now docs.esmvaltool.org) (Pull request #675) by @axel-lauer
Add all European grants to Zenodo (Pull request #680) by @bouweandela
Update Sphinx to v3 or later (Pull request #683) by @bouweandela
Increase version to 2.0.0 and add release notes (Pull request #691) by @bouweandela
Update setup.py and README.md for use on PyPI (Pull request #693) by @bouweandela
Suggested Documentation changes (Pull request #690) by @ssmithClimate
Improvements#
Reduce the size of conda package (Pull request #606) by @bouweandela
Add a few unit tests for DiagnosticTask (Pull request #613) by @bouweandela
Make ncl or R tests not fail if package not installed (Pull request #610) by @valeriupredoi
Pin flake8<3.8.0 (Pull request #623) by @valeriupredoi
Log warnings for likely errors in provenance record (Pull request #592) by @bouweandela
Unpin flake8 (Pull request #646) by @bouweandela
More flexible native6 default DRS (Pull request #645) by @bouweandela
Try to use the same python for running diagnostics as for esmvaltool (Pull request #656) by @bouweandela
Fix test for lower python version and add note on lxml (Pull request #659) by @valeriupredoi
Added 1m deep average soil moisture variable (Pull request #664) by @bascrezee
Update docker recipe (Pull request #603) by @jvegreg
Improve command line interface (Pull request #605) by @jvegreg
Remove utils directory (Pull request #697) by @bouweandela
Avoid pytest version that crashes (Pull request #707) by @bouweandela
Options arg in read_config_user_file now optional (Pull request #716) by @jvegreg
Produce a readable warning if ancestors are a string instead of a list. (Pull request #711) by @katjaweigel
Pin Yamale to v2 (Pull request #718) by @bouweandela
Expanded cmor public API (Pull request #714) by @schlunma
Fixes for datasets#
Added various fixes for hybrid height coordinates (Pull request #562) by @schlunma
Extended fix for cl-like variables of CESM2 models (Pull request #604) by @schlunma
Added fix to convert “geopotential” to “geopotential height” for ERA5 (Pull request #640) by @egalytska
Do not fix longitude values if they are too far from valid range (Pull request #636) by @jvegreg
Preprocessor#
Implemented concatenation of cubes with derived coordinates (Pull request #546) by @schlunma
Fix derived variable ctotal calculation depending on project and standard name (Pull request #620) by @valeriupredoi
State of the art FX variables handling without preprocessing (Pull request #557) by @valeriupredoi
Add max, min and std operators to multimodel (Pull request #602) by @jvegreg
Added preprocessor to extract amplitude of cycles (Pull request #597) by @schlunma
Overhaul concatenation and allow for correct concatenation of multiple overlapping datasets (Pull request #615) by @valeriupredoi
Change volume stats to handle and output masked array result (Pull request #618) by @valeriupredoi
Area_weights for cordex in area_statistics (Pull request #631) by @mwjury
Accept cubes as input in multimodel (Pull request #637) by @sloosvel
Make multimodel work correctly with yearly data (Pull request #677) by @valeriupredoi
Optimize time weights in time preprocessor for climate statistics (Pull request #684) by @valeriupredoi
Add percentiles to multi-model stats (Pull request #679) by @Peter9192
v2.0.0b9#
This release includes
Bug fixes#
Cast dtype float32 to output from zonal and meridional area preprocessors (Pull request #581) by @valeriupredoi
Improvements#
Unpin on Python<3.8 for conda package (run) (Pull request #570) by @valeriupredoi
Update pytest installation marker (Pull request #572) by @bouweandela
Remove vmrh2o (Pull request #573) by @mattiarighi
Restructure documentation (Pull request #575) by @bouweandela
Fix mask in land variables for CCSM4 (Pull request #579) by @zklaus
Fix derive scripts wrt required method (Pull request #585) by @zklaus
Check coordinates do not have repeated standard names (Pull request #558) by @jvegreg
Added derivation script for co2s (Pull request #587) by @schlunma
Adapted custom co2s table to match CMIP6 version (Pull request #588) by @schlunma
Increase version to v2.0.0b9 (Pull request #593) by @bouweandela
Add a method to save citation information (Pull request #402) by @SarahAlidoost
For older releases, see the release notes on ESMValGroup/ESMValCore.