Changelog

v2.3.1

This release includes

Bug fixes

• Update config-user.yml template with correct drs entries for CEDA-JASMIN (#1184) Valeriu Predoi

• Enhancing MIROC5 fix for hfls and evspsbl (#1192) katjaweigel

• Fix alignment of daily data with inconsistent calendars in multimodel statistics (#1212) Peter Kalverla

• Pin cf-units, remove github actions test for Python 3.6 and fix test_access1_0 and test_access1_3 to use cf-units for comparisons (#1197) Valeriu Predoi

• Fixed search for fx files when no mip is given (#1216) Manuel Schlund

• Make sure climate statistics always returns original dtype (#1237) Klaus Zimmermann

• Bugfix for regional regridding when non-integer range is passed (#1231) Stef Smeets

• Make sure area_statistics preprocessor always returns original dtype (#1239) Klaus Zimmermann

• Add “.” (dot) as allowed separation character for the time range group (#1248) Klaus Zimmermann

Documentation

• Add a link to the instructions to use pre-installed versions on HPC clusters (#1186) Rémi Kazeroni

• Bugfix release: set version to 2.3.1 (#1253) Klaus Zimmermann

Fixes for datasets

• Set circular attribute in MCM-UA-1-0 fix (#1178) sloosvel

• Fixed time coordinate of MIROC-ESM (#1188) Manuel Schlund

Preprocessor

• Filter warnings about collapsing multi-model dimension in multimodel statistics preprocessor function (#1215) Bouwe Andela

• Remove fx variables before computing multimodel statistics (#1220) sloosvel

Automatic testing

• Pin Python to 3.9 in environment.yml on CircleCI and skip mypy tests in conda build (#1176) Bouwe Andela

Installation

• Pin lower bound for iris to 3.0.2 (#1206) Valeriu Predoi

• Pin iris<3.0.4 to ensure we still (sort of) support Python 3.6 (#1252) Valeriu Predoi

Improvements

• Add test to verify behaviour for scalar height coord for tas in multi-model (#1209) Peter Kalverla

• Sort missing years in “No input data available for years” message (#1225) Lee de Mora

v2.3.0

This release includes

Bug fixes

• Extend preprocessor multi_model_statistics to handle data with “altitude” coordinate (#1010) Axel Lauer

• Remove scripts included with CMOR tables (#1011) Bouwe Andela

• Avoid side effects in extract_season (#1019) Javier Vegas-Regidor

• Use nearest scheme to avoid interpolation errors with masked data in regression test (#1021) Stef Smeets

• Move _get_time_bounds from preprocessor._time to cmor.check to avoid circular import with cmor module (#1037) Valeriu Predoi

• Fix test that makes conda build fail (#1046) Valeriu Predoi

• Fix ‘positive’ attribute for rsns/rlns variables (#1051) Lukas Brunner

• Added preprocessor mask_multimodel (#767) Manuel Schlund

• Fix bug when fixing bounds after fixing longitude values (#1057) sloosvel

• Run conda build parallel AND sequential tests (#1065) Valeriu Predoi

• Add key to id_prop (#1071) Lukas Brunner

• Fix bounds after reversing coordinate values (#1061) sloosvel

• Fixed –skip-nonexistent option (#1093) Manuel Schlund

• Do not consider CMIP5 variable sit to be the same as sithick from CMIP6 (#1033) Bouwe Andela

• Improve finding date range in filenames (enforces separators) (#1145) Stéphane Sénési - work

• Review fx handling (#1147) sloosvel

• Fix lru cache decorator with explicit call to method (#1172) Valeriu Predoi

• Update _volume.py (#1174) Lee de Mora

Deprecations

Documentation

• Final changelog for 2.3.0 (#1163) Klaus Zimmermann

• Set version to 2.3.0 (#1162) Klaus Zimmermann

• Fix documentation build (#1006) Bouwe Andela

• Add labels required for linking from ESMValTool docs (#1038) Bouwe Andela

• Update contribution guidelines (#1047) Bouwe Andela

• Fix basestring references in documentation (#1106) Javier Vegas-Regidor

• Updated references master to main (#1132) Axel Lauer

• Add instructions how to use the central installation at DKRZ-Mistral (#1155) Rémi Kazeroni

Fixes for datasets

• Added fixes for various CMIP5 datasets, variable cl (3-dim cloud fraction) (#1017) Axel Lauer

• Added fixes for hybrid level coordinates of CESM2 models (#882) Manuel Schlund

• Extending LWP fix for CMIP6 models (#1049) Axel Lauer

• Add fixes for the net & up radiation variables from ERA5 (#1052) Lukas Brunner

• Add derived variable rsus (#1053) Lukas Brunner

• Supported mip-level fixes (#1095) Manuel Schlund

• Fix erroneous use of grid_latitude and grid_longitude and cleaned ocean grid fixes (#1092) Manuel Schlund

• Fix for pr of miroc5 (#1110) Rémi Kazeroni

• Ocean depth fix for cnrm_esm2_1, gfdl_esm4, ipsl_cm6a_lr datasets + mcm_ua_1_0 (#1098) Tomas Lovato

• Fix for uas variable of the MCM_UA_1_0 dataset (#1102) Rémi Kazeroni

• Fixes for sos and siconc of BCC models (#1090) Rémi Kazeroni

• Run fgco2 fix for all CESM2 models (#1108) Lisa Bock

• Fixes for the siconc variable of CMIP6 models (#1105) Rémi Kazeroni

• Fix wrong sign for land surface flux (#1113) Lisa Bock

• Fix for pr of EC_EARTH (#1116) Rémi Kazeroni

CMOR standard

• Format cmor related files (#976) Javier Vegas-Regidor

• Check presence of time bounds and guess them if needed (#849) sloosvel

• Add custom variable “tasaga” (#1118) Lisa Bock

• Find files for CMIP6 DCPP startdates (#771) sloosvel

Preprocessor

• Update tests for multimodel statistics preprocessor (#1023) Stef Smeets

• Raise in extract_season and extract_month if result is None (#1041) Javier Vegas-Regidor

• Allow selection of shapes in extract_shape (#764) Javier Vegas-Regidor

• Add option for regional regridding to regrid preprocessor (#1034) Stef Smeets

• Load fx variables as cube cell measures / ancillary variables (#999) sloosvel

• Check horizontal grid before regridding (#507) Benjamin Müller

• Clip irregular grids (#245) Bouwe Andela

• Use native iris functions in multi-model statistics (#1150) Peter Kalverla

Notebook API (experimental)

Automatic testing

• Report coverage for tests that run on any pull request (#994) Bouwe Andela

• Install ESMValTool sample data from PyPI (#998) Javier Vegas-Regidor

• Fix tests for multi-processing with spawn method (i.e. macOSX with Python>3.8) (#1003) Barbara Vreede

• Switch to running the Github Action test workflow every 3 hours in single thread mode to observe if Sementation Faults occur (#1022) Valeriu Predoi

• Revert to original Github Actions test workflow removing the 3-hourly test run with -n 1 (#1025) Valeriu Predoi

• Avoid stale cache for multimodel statistics regression tests (#1030) Bouwe Andela

• Add newer Python versions in OSX to Github Actions (#1035) Barbara Vreede

• Add tests for type annotations with mypy (#1042) Stef Smeets

• Run problematic cmor tests sequentially to avoid segmentation faults on CircleCI (#1064) Valeriu Predoi

• Test installation of esmvalcore from conda-forge (#1075) Valeriu Predoi

• Added additional test cases for integration tests of data_finder.py (#1087) Manuel Schlund

• Pin cf-units and fix tests (cf-units>=2.1.5) (#1140) Valeriu Predoi

• Fix failing CircleCI tests (#1167) Bouwe Andela

• Fix test failing due to fx files chosen differently on different OS’s (#1169) Valeriu Predoi

• Compare datetimes instead of strings in _fixes/cmip5/test_access1_X.py (#1173) Valeriu Predoi

• Pin Python to 3.9 in environment.yml on CircleCI and skip mypy tests in conda build (#1176) Bouwe Andela

Installation

• Update yamale to version 3 (#1059) Klaus Zimmermann

Improvements

• Refactor diagnostics / tags management (#939) Stef Smeets

• Support multiple paths in input_dir (#1000) Javier Vegas-Regidor

• Generate HTML report with recipe output (#991) Stef Smeets

• Add timeout to requests.get in _citation.py (#1091) SarahAlidoost

• Add SYNDA drs for CMIP5 and CMIP6 (closes #582) (#583) Klaus Zimmermann

• Add basic support for variable mappings (#1124) Klaus Zimmermann

• Handle IPSL-CM6 (#1153) Stéphane Sénési - work

v2.2.0

Highlights

ESMValCore is now using the recently released Iris 3. We acknowledge that this change may impact your work, as Iris 3 introduces several changes that are not backward-compatible, but we think that moving forward is the best decision for the tool in the long term.

This release is also the first one including support for downloading CMIP6 data using Synda and we have also started supporting Python 3.9. Give it a try!

This release includes

Bug fixes

• Fix path settings for DKRZ/Mistral (#852) Bouwe Andela

• Change logic for calling the diagnostic script to avoid problems with scripts where the executable bit is accidentally set (#877) Bouwe Andela

• Fix overwriting in generic level check (#886) sloosvel

• Add double quotes to script args in rerun screen message when using vprof profiling (#897) Valeriu Predoi

• Simplify time handling in multi-model statistics preprocessor (#685) Peter Kalverla

• Fix links to Iris documentation (#966) Javier Vegas-Regidor

• Bugfix: Fix units for MSWEP data (#986) Stef Smeets

Deprecations

• Deprecate defining write_plots and write_netcdf in config-user file (#808) Bouwe Andela

Documentation

• Fix numbering of steps in release instructions (#838) Bouwe Andela

• Add labels to changelogs of individual versions for easy reference (#899) Klaus Zimmermann

• Make CircleCI badge specific to main branch (#902) Bouwe Andela

• Fix docker build badge url (#906) Stef Smeets

• Update github PR template (#909) Stef Smeets

• Refer to ESMValTool GitHub discussions page in the error message (#900) Bouwe Andela

• Support automatically closing issues (#922) Bouwe Andela

• Fix checkboxes in PR template (#931) Stef Smeets

• Change in config-user defaults and documentation with new location for esmeval OBS data on JASMIN (#958) Valeriu Predoi

• Update Core Team info (#942) Axel Lauer

• Update iris documentation URL for sphinx (#964) Bouwe Andela

• Set version to 2.2.0 (#977) Javier Vegas-Regidor

• Add first draft of v2.2.0 changelog (#983) Javier Vegas-Regidor

• Add checkbox in PR template to assign labels (#985) Javier Vegas-Regidor

• Update install.rst (#848) bascrezee

• Change the order of the publication steps (#984) Javier Vegas-Regidor

• Add instructions how to use esmvaltool from HPC central installations (#841) Valeriu Predoi

Fixes for datasets

• Fixing unit for derived variable rsnstcsnorm to prevent overcorrection2 (#846) katjaweigel

• Cmip6 fix awi cm 1 1 mr (#822) mwjury

• Cmip6 fix ec earth3 veg (#836) mwjury

• Changed latitude longitude fix from Tas to AllVars. (#916) katjaweigel

• Fix for precipitation (pr) to use ERA5-Land cmorizer (#879) katjaweigel

• Cmip6 fix ec earth3 (#837) mwjury

• Cmip6_fix_fgoals_f3_l_Amon_time_bnds (#831) mwjury

• Fix for FGOALS-f3-L sftlf (#667) mwjury

• Improve ACCESS-CM2 and ACCESS-ESM1-5 fixes and add CIESM and CESM2-WACCM-FV2 fixes for cl, clw and cli (#635) Axel Lauer

• Add fixes for cl, cli, clw and tas for several CMIP6 models (#955) Manuel Schlund

• Dataset fixes for MSWEP (#969) Stef Smeets

• Dataset fixes for: ACCESS-ESM1-5, CanESM5, CanESM5 for carbon cycle (#947) Bettina Gier

• Fixes for KIOST-ESM (CMIP6) (#904) Rémi Kazeroni

• Fixes for AWI-ESM-1-1-LR (CMIP6, piControl) (#911) Rémi Kazeroni

CMOR standard

• CMOR check generic level coordinates in CMIP6 (#598) sloosvel

• Update CMIP6 tables to 6.9.33 (#919) Javier Vegas-Regidor

• Adding custom variables for tas uncertainty (#924) Lisa Bock

• Remove monotonicity coordinate check for unstructured grids (#965) Javier Vegas-Regidor

Preprocessor

• Added clip_start_end_year preprocessor (#796) Manuel Schlund

• Add support for downloading CMIP6 data with Synda (#699) Bouwe Andela

• Add multimodel tests using real data (#856) Stef Smeets

• Add plev/altitude conversion to extract_levels (#892) Axel Lauer

• Add possibility of custom season extraction. (#247) mwjury

• Adding the ability to derive xch4 (#783) Birgit Hassler

• Add preprocessor function to resample time and compute x-hourly statistics (#696) Javier Vegas-Regidor

• Fix duplication in preprocessors DEFAULT_ORDER introduced in #696 (#973) Javier Vegas-Regidor

• Use consistent precision in multi-model statistics calculation and update reference data for tests (#941) Peter Kalverla

• Refactor multi-model statistics code to facilitate ensemble stats and lazy evaluation (#949) Peter Kalverla

• Add option to exclude input cubes in output of multimodel statistics to solve an issue introduced by #949 (#978) Peter Kalverla

Automatic testing

• Pin cftime>=1.3.0 to have newer string formatting in and fix two tests (#878) Valeriu Predoi

• Switched miniconda conda setup hooks for Github Actions workflows (#873) Valeriu Predoi

• Add test for latest version resolver (#874) Stef Smeets

• Update codacy coverage reporter to fix coverage (#905) Niels Drost

• Avoid hardcoded year in tests and add improvement to plev test case (#921) Bouwe Andela

• Pin scipy to less than 1.6.0 until ESMValGroup/ESMValCore/issues/927 gets resolved (#928) Valeriu Predoi

• Github Actions: change time when conda install test runs (#930) Valeriu Predoi

• Remove redundant test line from test_utils.py (#935) Valeriu Predoi

• Removed netCDF4 package from integration tests of fixes (#938) Manuel Schlund

• Use new conda environment for installing ESMValCore in Docker containers (#951) Bouwe Andela

Notebook API (experimental)

• Implement importable config object in experimental API submodule (#868) Stef Smeets

• Add loading and running recipes to the notebook API (#907) Stef Smeets

• Add displaying and loading of recipe output to the notebook API (#957) Stef Smeets

• Add functionality to run single diagnostic task to notebook API (#962) Stef Smeets

Improvements

• Create CODEOWNERS file (#809) Javier Vegas-Regidor

• Remove code needed for Python <3.6 (#844) Bouwe Andela

• Add requests as a dependency (#850) Bouwe Andela

• Pin Python to less than 3.9 (#870) Valeriu Predoi

• Remove numba dependency (#880) Manuel Schlund

• Add Listing and finding recipes to the experimental notebook API (#901) Stef Smeets

• Skip variables that don’t have dataset or additional_dataset keys (#860) Valeriu Predoi

• Refactor logging configuration (#933) Stef Smeets

• Xco2 derivation (#913) Bettina Gier

• Working environment for Python 3.9 (pin to !=3.9.0) (#885) Valeriu Predoi

• Print source file when using config get_config_user command (#960) Valeriu Predoi

• Switch to Iris 3 (#819) Stef Smeets

• Refactor tasks (#959) Stef Smeets

• Restore task summary in debug log after #959 (#981) Bouwe Andela

• Pin pre-commit hooks (#974) Stef Smeets

• Improve error messages when data is missing (#917) Javier Vegas-Regidor

• Set remove_preproc_dir to false in default config-user (#979) Valeriu Predoi

• Move fiona to be installed from conda forge (#987) Valeriu Predoi

• Re-added fiona in setup.py (#990) Valeriu Predoi

v2.1.0

This release includes

Bug fixes

• Set unit=1 if anomalies are standardized (#727) bascrezee

• Fix crash for FGOALS-g2 variables without longitude coordinate (#729) Bouwe Andela

• Improve variable alias management (#595) Javier Vegas-Regidor

• Fix area_statistics fx files loading (#798) Javier Vegas-Regidor

• Fix units after derivation (#754) Manuel Schlund

Documentation

• Update v2.0.0 release notes with final additions (#722) Bouwe Andela

• Update package description in setup.py (#725) Mattia Righi

• Add installation instructions for pip installation (#735) Bouwe Andela

• Improve config-user documentation (#740) Bouwe Andela

• Update the zenodo file with contributors (#807) Valeriu Predoi

• Improve command line run documentation (#721) Javier Vegas-Regidor

• Update the zenodo file with contributors (continued) (#810) Valeriu Predoi

Improvements

• Reduce size of docker image (#723) Javier Vegas-Regidor

• Add ‘test’ extra to installation, used by docker development tag (#733) Bouwe Andela

• Correct dockerhub link (#736) Bouwe Andela

• Create action-install-from-pypi.yml (#734) Valeriu Predoi

• Add pre-commit for linting/formatting (#766) Stef Smeets

• Run tests in parallel and when building conda package (#745) Bouwe Andela

• Readable exclude pattern for pre-commit (#770) Stef Smeets

• Github Actions Tests (#732) Valeriu Predoi

• Remove isort setup to fix formatting conflict with yapf (#778) Stef Smeets

• Fix yapf-isort import formatting conflict (Fixes #777) (#784) Stef Smeets

• Sorted output for esmvaltool recipes list (#790) Stef Smeets

• Replace vmprof with vprof (#780) Valeriu Predoi

• Update CMIP6 tables to 6.9.32 (#706) Javier Vegas-Regidor

• Default config-user path now set in config-user read function (#791) Javier Vegas-Regidor

• Add custom variable lweGrace (#692) bascrezee

• Create Github Actions workflow to build and deploy on Test PyPi and PyPi (#820) Valeriu Predoi

• Build and publish the esmvalcore package to conda via Github Actions workflow (#825) Valeriu Predoi

Fixes for datasets

• Fix cmip6 models (#629) npgillett

• Fix siconca variable in EC-Earth3 and EC-Earth3-Veg models in amip simulation (#702) Evgenia Galytska

Preprocessor

• Move cmor_check_data to early in preprocessing chain (#743) Bouwe Andela

• Add RMS iris analysis operator to statistics preprocessor functions (#747) Pep Cos

• Add surface chlorophyll concentration as a derived variable (#720) sloosvel

• Use dask to reduce memory consumption of extract_levels for masked data (#776) Valeriu Predoi

v2.0.0

This release includes

Bug fixes

• Fixed derivation of co2s (#594) Manuel Schlund

• Padding while cropping needs to stay within sane bounds for shapefiles that span the whole Earth (#626) Valeriu Predoi

• Fix concatenation of a single cube (#655) Bouwe Andela

• Fix mask fx dict handling not to fail if empty list in values (#661) Valeriu Predoi

• Preserve metadata during anomalies computation when using iris cubes difference (#652) Valeriu Predoi

• Avoid crashing when there is directory ‘esmvaltool’ in the current working directory (#672) Valeriu Predoi

• Solve bug in ACCESS1 dataset fix for calendar. (#671) Peter Kalverla

• Fix the syntax for adding multiple ensemble members from the same dataset (#678) SarahAlidoost

• Fix bug that made preprocessor with fx files fail in rare cases (#670) Manuel Schlund

• Add support for string coordinates (#657) Javier Vegas-Regidor

• Fixed the shape extraction to account for wraparound shapefile coords (#319) Valeriu Predoi

• Fixed bug in time weights calculation (#695) Manuel Schlund

• Fix diagnostic filter (#713) Javier Vegas-Regidor

Documentation

• Add pandas as a requirement for building the documentation (#607) Bouwe Andela

• Document default order in which preprocessor functions are applied (#633) Bouwe Andela

• Add pointers about data loading and CF standards to documentation (#571) Valeriu Predoi

• Config file populated with site-specific data paths examples (#619) Valeriu Predoi

• Update Codacy badges (#643) Bouwe Andela

• Update copyright info on readthedocs (#668) Bouwe Andela

• Updated references to documentation (now docs.esmvaltool.org) (#675) Axel Lauer

• Add all European grants to Zenodo (#680) Bouwe Andela

• Update Sphinx to v3 or later (#683) Bouwe Andela

• Increase version to 2.0.0 and add release notes (#691) Bouwe Andela

• Update setup.py and README.md for use on PyPI (#693) Bouwe Andela

• Suggested Documentation changes (#690) Steve Smith

Improvements

• Reduce the size of conda package (#606) Bouwe Andela

• Add a few unit tests for DiagnosticTask (#613) Bouwe Andela

• Make ncl or R tests not fail if package not installed (#610) Valeriu Predoi

• Pin flake8<3.8.0 (#623) Valeriu Predoi

• Log warnings for likely errors in provenance record (#592) Bouwe Andela

• Unpin flake8 (#646) Bouwe Andela

• More flexible native6 default DRS (#645) Bouwe Andela

• Try to use the same python for running diagnostics as for esmvaltool (#656) Bouwe Andela

• Fix test for lower python version and add note on lxml (#659) Valeriu Predoi

• Added 1m deep average soil moisture variable (#664) bascrezee

• Update docker recipe (#603) Javier Vegas-Regidor

• Improve command line interface (#605) Javier Vegas-Regidor

• Remove utils directory (#697) Bouwe Andela

• Avoid pytest version that crashes (#707) Bouwe Andela

• Options arg in read_config_user_file now optional (#716) Javier Vegas-Regidor

• Produce a readable warning if ancestors are a string instead of a list. (#711) katjaweigel

• Pin Yamale to v2 (#718) Bouwe Andela

• Expanded cmor public API (#714) Manuel Schlund

Fixes for datasets

• Added various fixes for hybrid height coordinates (#562) Manuel Schlund

• Extended fix for cl-like variables of CESM2 models (#604) Manuel Schlund

• Added fix to convert “geopotential” to “geopotential height” for ERA5 (#640) Evgenia Galytska

• Do not fix longitude values if they are too far from valid range (#636) Javier Vegas-Regidor

Preprocessor

• Implemented concatenation of cubes with derived coordinates (#546) Manuel Schlund

• Fix derived variable ctotal calculation depending on project and standard name (#620) Valeriu Predoi

• State of the art FX variables handling without preprocessing (#557) Valeriu Predoi

• Add max, min and std operators to multimodel (#602) Javier Vegas-Regidor

• Added preprocessor to extract amplitude of cycles (#597) Manuel Schlund

• Overhaul concatenation and allow for correct concatenation of multiple overlapping datasets (#615) Valeriu Predoi

• Change volume stats to handle and output masked array result (#618) Valeriu Predoi

• Area_weights for cordex in area_statistics (#631) mwjury

• Accept cubes as input in multimodel (#637) sloosvel

• Make multimodel work correctly with yearly data (#677) Valeriu Predoi

• Optimize time weights in time preprocessor for climate statistics (#684) Valeriu Predoi

• Add percentiles to multi-model stats (#679) Peter Kalverla

v2.0.0b9

This release includes

Bug fixes

• Cast dtype float32 to output from zonal and meridional area preprocessors (#581) Valeriu Predoi

Improvements

• Unpin on Python<3.8 for conda package (run) (#570) Valeriu Predoi

• Update pytest installation marker (#572) Bouwe Andela

• Remove vmrh2o (#573) Mattia Righi

• Restructure documentation (#575) Bouwe Andela

• Fix mask in land variables for CCSM4 (#579) Klaus Zimmermann

• Fix derive scripts wrt required method (#585) Klaus Zimmermann

• Check coordinates do not have repeated standard names (#558) Javier Vegas-Regidor

• Added derivation script for co2s (#587) Manuel Schlund

• Adapted custom co2s table to match CMIP6 version (#588) Manuel Schlund

• Increase version to v2.0.0b9 (#593) Bouwe Andela

• Add a method to save citation information (#402) SarahAlidoost

For older releases, see the release notes on https://github.com/ESMValGroup/ESMValCore/releases.