Contributions are very welcome#
We greatly value contributions of any kind. Contributions could include, but are not limited to documentation improvements, bug reports, new or improved code, scientific and technical code reviews, infrastructure improvements, mailing list and chat participation, community help/building, education and outreach. We value the time you invest in contributing and strive to make the process as easy as possible. If you have suggestions for improving the process of contributing, please do not hesitate to propose them.
If you have a bug or other issue to report or just need help, please open an issue on the issues tab on the ESMValCore github repository.
If you would like to contribute a new preprocessor function, derived variable, fix for a dataset, or another new feature, please discuss your idea with the development team before getting started, to avoid double work and/or disappointment later. A good way to do this is to open an issue on GitHub.
See Installation from source for instructions on how to set up a development installation.
New development should preferably be done in the
The default git branch is
Use this branch to create a new feature branch from and make a pull request
offers a good introduction to git branches, but it was written for
BitBucket while we use GitHub, so replace the word BitBucket by GitHub
whenever you read it.
It is recommended that you open a draft pull request early, as this will cause CircleCI to run the unit tests, Codacy to analyse your code, and readthedocs to build the documentation. It’s also easier to get help from other developers if your code is visible in a pull request.
Make small pull requests, the ideal pull requests changes just a few files and adds/changes no more than 100 lines of production code. The amount of test code added can be more extensive, but changes to existing test code should be made sparingly.
When making changes, try to respect the current structure of the program. If you need to make major changes to the structure of program to add a feature, chances are that you have either not come up with the most optimal design or the feature is not a very good fit for the tool. Discuss your feature with the @ESMValGroup/esmvaltool-coreteam in an issue to find a solution.
Please keep the following considerations in mind when programming:
Changes should preferably be backward compatible.
Apply changes gradually and change no more than a few files in a single pull request, but do make sure every pull request in itself brings a meaningful improvement. This reduces the risk of breaking existing functionality and making backward incompatible changes, because it helps you as well as the reviewers of your pull request to better understand what exactly is being changed.
Preprocessor functions are Python functions (and not classes) so they are easy to understand and implement for scientific contributors.
No additional CMOR checks should be implemented inside preprocessor functions. The input cube is fixed and confirmed to follow the specification in esmvalcore/cmor/tables before applying any other preprocessor functions. This design helps to keep the preprocessor functions and diagnostics scripts that use the preprocessed data from the tool simple and reliable. See Project CMOR table configuration for the mapping from
projectin the recipe to the relevant CMOR table.
The ESMValCore package is based on iris. Preprocessor functions should preferably be small and just call the relevant iris code. Code that is more involved and more broadly applicable than just in the ESMValCore, should be implemented in iris instead.
Any settings in the recipe that can be checked before loading the data should be checked at the task creation stage. This avoids that users run a recipe for several hours before finding out they made a mistake in the recipe. No data should be processed or files written while creating the tasks.
CMOR checks should provide a good balance between reliability of the tool and ease of use. Several levels of strictness of the checks are available to facilitate this.
Keep your code short and simple: we would like to make contributing as easy as possible. For example, avoid implementing complicated class inheritance structures and boilerplate code.
If you find yourself copy-pasting a piece of code and making minor changes to every copy, instead put the repeated bit of code in a function that you can re-use, and provide the changed bits as function arguments.
Be careful when changing existing unit tests to make your new feature work. You might be breaking existing features if you have to change existing tests.
Finally, if you would like to improve the design of the tool, discuss your plans with the @ESMValGroup/esmvaltool-coreteam to make sure you understand the current functionality and you all agree on the new design.
Checklist for pull requests#
To clearly communicate up front what is expected from a pull request, we have the following checklist. Please try to do everything on the list before requesting a review. If you are unsure about something on the list, please ask the @ESMValGroup/tech-reviewers or @ESMValGroup/science-reviewers for help by commenting on your (draft) pull request or by starting a new discussion.
In the ESMValTool community we use pull request reviews to ensure all code and documentation contributions are of good quality. The icons indicate whether the item will be checked during the 🛠 Technical review or 🧪 Scientific review.
🧪 The new functionality is relevant and scientifically sound
🛠 Code is written according to the code quality guidelines
🧪 and 🛠 Documentation is available
🛠 Unit tests have been added
🛠 Changes are backward compatible
🛠 The list of authors is up to date
🛠 The checks shown below the pull request are successful
Pull requests introducing a change that causes a recipe to no longer run successfully (breaking change), or which results in scientifically significant changes in results (science change) require additional items to be reviewed defined in the backward compatibility policy. These include in particular:
🛠 Instructions for the release notes to assist recipe developers to adapt their recipe in light of the backward-incompatible change available.
🛠 General instructions for recipe developers working on user recipes to enable them to adapt their code related to backward-incompatible changes available (see ESMValTool_Tutorial: issue #263).
🛠 Core development team tagged to notify them of the backward-incompatible change, and give at least 2 weeks for objections to be raised before merging to the main branch. If a strong objection is raised the backward-incompatible change should not be merged until the objection is resolved.
🛠 Information required for the “backward-incompatible changes” section in the PR that introduces the backward-incompatible change available.
The proposed changes should be relevant for the larger scientific community. The implementation of new features should be scientifically sound; e.g. the formulas used in new preprocesssor functions should be accompanied by the relevant references and checked for correctness by the scientific reviewer. The CF Conventions as well as additional standards imposed by CMIP should be followed whenever possible.
Pull request title and label#
The title of a pull request should clearly describe what the pull request changes. If you need more text to describe what the pull request does, please add it in the description. Add one or more labels to your pull request to indicate the type of change. At least one of the following labels should be used: bug, deprecated feature, fix for dataset, preprocessor, cmor, api, testing, documentation or enhancement.
The titles and labels of pull requests are used to compile the Changelog, therefore it is important that they are easy to understand for people who are not familiar with the code or people in the project. Descriptive pull request titles also makes it easier to find back what was changed when, which is useful in case a bug was introduced.
To increase the readability and maintainability or the ESMValCore source code, we aim to adhere to best practices and coding standards.
We include checks for Python and yaml files, most of which are described in more detail in the sections below. This includes checks for invalid syntax and formatting errors. Pre-commit is a handy tool that can run all of these checks automatically just before you commit your code. It knows knows which tool to run for each filetype, and therefore provides a convenient way to check your code.
The standard document on best practices for Python code is PEP8 and there is PEP257 for code documentation. We make use of numpy style docstrings to document Python functions that are visible on readthedocs.
To check if your code adheres to the standard, go to the directory where
the repository is cloned, e.g.
cd ESMValCore, and run prospector
In addition to prospector, we use flake8 to automatically check for bugs and formatting mistakes and mypy for checking that type hints are correct. Note that type hints are completely optional, but if you do choose to add them, they should be correct.
When you make a pull request, adherence to the Python development best practices is checked in two ways:
As part of the unit tests, flake8 and mypy are run by CircleCI, see the section on Tests for more information.
Codacy is a service that runs prospector (and other code quality tools) on changed files and reports the results. Click the ‘Details’ link behind the Codacy check entry and then click ‘View more details on Codacy Production’ to see the results of the static code analysis done by Codacy. If you need to log in, you can do so using your GitHub account.
The automatic code quality checks by prospector are really helpful to improve the quality of your code, but they are not flawless. If you suspect prospector or Codacy may be wrong, please ask the @ESMValGroup/tech-reviewers by commenting on your pull request.
Note that running prospector locally will give you quicker and sometimes more accurate results than waiting for Codacy.
Most formatting issues in Python code can be fixed automatically by running the commands
to sort the imports in the standard way using isort and
yapf -i some_file.py
to add/remove whitespace as required by the standard using yapf,
docformatter -i some_file.py
to run docformatter which helps formatting the docstrings (such as line length, spaces).
Please use yamllint to check that your
YAML files do not contain mistakes.
yamllint checks for valid syntax, common mistakes like key repetition and
cosmetic problems such as line length, trailing spaces, wrong indentation, etc.
Any text file#
A generic tool to check for common spelling mistakes is codespell.
The documentation lives on docs.esmvaltool.org.
The documentation is built by readthedocs using Sphinx. There are two main ways of adding documentation:
As written text in the directory doc. When writing reStructuredText (
.rst) files, please try to limit the line length to 80 characters and always start a sentence on a new line. This makes it easier to review changes to documentation on GitHub.
As docstrings or comments in code. For Python code, only the docstrings of Python modules, classes, and functions that are mentioned in doc/api are used to generate the online documentation. This results in the ESMValCore API Reference. The standard document with best practices on writing docstrings is PEP257. For the API documentation, we make use of numpy style docstrings.
What should be documented#
Functionality that is visible to users should be documented. Any documentation that is visible on readthedocs should be well written and adhere to the standards for documentation. Examples of this include:
Preprocessor functions and their use from the recipe
For functions that compute scientific results, comments with references to papers and/or other resources as well as formula numbers should be included.
When making changes to/introducing a new preprocessor function, also update the preprocessor documentation.
There is no need to write complete numpy style documentation for functions that are not visible in the ESMValCore API Reference chapter on readthedocs. However, adding a docstring describing what a function does is always a good idea. For short functions, a one-line docstring is usually sufficient, but more complex functions might require slightly more extensive documentation.
When reviewing a pull request, always check that documentation is easy to understand and available in all expected places.
How to build and view the documentation#
Whenever you make a pull request or push new commits to an existing pull
request, readthedocs will automatically build the documentation.
The link to the documentation will be shown in the list of checks below your
Click ‘Details’ behind the check
docs/readthedocs.org:esmvalcore to preview
If all checks were successful, you may need to click ‘Show all checks’ to see
the individual checks.
To build the documentation on your own computer, go to the directory where the repository was cloned and run
sphinx-build doc doc/build
sphinx-build -Ea doc doc/build
to build it from scratch.
Make sure that your newly added documentation builds without warnings or errors and looks correctly formatted. CircleCI will build the documentation with the command:
sphinx-build -W doc doc/build
This will catch mistakes that can be detected automatically.
The configuration file for Sphinx is doc/shinx/source/conf.py.
See Integration with the ESMValCore documentation for information on how the ESMValCore documentation is integrated into the complete ESMValTool project documentation on readthedocs.
When reviewing a pull request, always check that the documentation checks shown below the pull request were successful.
To check that the code works correctly, there tests available in the tests directory. We use pytest to write and run our tests.
Contributions to ESMValCore should be
covered by unit tests.
Have a look at the existing tests in the
tests directory for inspiration on
how to write your own tests.
If you do not know how to start with writing unit tests, ask the
@ESMValGroup/tech-reviewers for help by commenting on the pull request and
they will try to help you.
It is also recommended that you have a look at the pytest documentation at some
point when you start writing your own tests.
To run the tests on your own computer, go to the directory where the repository is cloned and run the command
Optionally you can skip tests which require additional dependencies for
supported diagnostic script languages by adding
-m 'not installation' to the
previous command. To only run tests from a single file, run the command
If you would like to avoid loading the default pytest configuration from setup.cfg because this can be a bit slow for running just a few tests, use
pytest -c /dev/null tests/unit/test_some_file.py
for more information on the available commands.
Whenever you make a pull request or push new commits to an existing pull
request, the tests in the
tests directory of the branch associated with the
pull request will be run automatically on CircleCI.
The results appear at the bottom of the pull request.
Click on ‘Details’ for more information on a specific test job.
When reviewing a pull request, always check that all test jobs on CircleCI were successful.
To check which parts of your code are covered by unit tests, open the file
test-reports/coverage_html/index.html (available after running a
command) and browse to the relevant file.
CircleCI will upload the coverage results from running the tests to codecov and
codecov is a service
that will comment on pull requests with a summary of the test coverage.
If codecov reports that the coverage has decreased, check the report and add
Alternatively, it is also possible to view code coverage on Codacy (click the
Files tab) and CircleCI (open the
tests job and click the ARTIFACTS tab).
To see some of the results on CircleCI, Codacy, or codecov, you may need to log
in; you can do so using your GitHub account.
When reviewing a pull request, always check that new code is covered by unit tests and codecov reports an increased coverage.
New or modified preprocessor functions should preferably also be tested using
the sample data.
These tests are located in
Please mark new tests that use the sample data with the
The ESMValTool_sample_data repository contains samples of CMIP6 data for testing ESMValCore. The ESMValTool-sample-data package is installed as part of the developer dependencies. The size of the package is relatively small (~ 100 MB), so it can be easily downloaded and distributed.
Preprocessing the sample data can be time-consuming, so some intermediate results are cached by pytest to make the tests run faster. If you suspect the tests are failing because the cache is invalid, clear it by running
To avoid running the time consuming tests that use sample data altogether, run
pytest -m "not use_sample_data"
Whenever you make a pull request or push new commits to an existing pull
request, the tests in the
tests of the branch associated with the
pull request will be run automatically on CircleCI.
Every night, more extensive tests are run to make sure that problems with the installation of the tool are discovered by the development team before users encounter them. These nightly tests have been designed to follow the installation procedures described in the documentation, e.g. in the Installation chapter. The nightly tests are run using both CircleCI and GitHub Actions. The result of the tests ran by CircleCI can be seen on the CircleCI project page and the result of the tests ran by GitHub Actions can be viewed on the Actions tab of the repository (to learn more about the Github-hosted runners, please have a look the documentation).
When opening a pull request, if you wish to run the Github Actions Test test, you can activate it via a simple comment containing the @runGAtests tag (e.g. “@runGAtests” or “@runGAtests please run” - in effect, tagging the runGAtests bot that will start the test automatically). This is useful to check if a certain feature that you included in the Pull Request, and can be tested for via the test suite, works across the supported Python versions, and both on Linux and OSX. The test is currently deactivated, so before triggering the test via comment, make sure you activate the test in the main Actions page (click on Test via PR Comment and activate it); also and be sure to deactivate it afterwards (the Github API still needs a bit more development, and currently it triggers the test for each comment irrespective of PR, that’s why this needs to be activated/decativated).
The configuration of the tests run by CircleCI can be found in the directory .circleci, while the configuration of the tests run by GitHub Actions can be found in the directory .github/workflows.
The ESMValCore package is used by many people to run their recipes. Many of these recipes are maintained in the public ESMValTool repository, but there are also users who choose not to share their work there. While our commitment is first and foremost to users who do share their recipes in the ESMValTool repository, we still try to be nice to all of the ESMValCore users.
The backward compatibility policy outlines the key principles on backward compatibility and additional guidance on handling backward-incompatible changes. This policy applies to both, ESMValCore and ESMValTool.
When making changes, e.g. to the recipe format, the diagnostic script interface, the public Python API, or the configuration file format, keep in mind that this may affect many users. To keep the tool user friendly, try to avoid making changes that are not backward compatible, i.e. changes that require users to change their existing recipes, diagnostics, configuration files, or scripts.
If you really must change the public interfaces of the tool, always discuss
this with the @ESMValGroup/esmvaltool-coreteam. Try to deprecate the
feature first by issuing an
ESMValCoreDeprecationWarning using the
warnings module and schedule it for removal two minor versions from the upcoming release. For example, when you
deprecate a feature in a pull request that will be included in version 2.5,
that feature should be removed in version 2.7:
import warnings from esmvalcore.exceptions import ESMValCoreDeprecationWarning # Other code def func(x, deprecated_option=None): """Deprecate deprecated_option.""" if deprecated_option is not None: deprecation_msg = ( "The option ``deprecated_option`` has been deprecated in " "ESMValCore version 2.5 and is scheduled for removal in " "version 2.7. Add additional text (e.g., description of " "alternatives) here.") warnings.warn(deprecation_msg, ESMValCoreDeprecationWarning) # Other code
Mention the version in which the feature will be removed in the deprecation message. Label the pull request with the deprecated feature label. When deprecating a feature, please follow up by actually removing the feature in due course.
If you must make backward incompatible changes, you need to update the available recipes in ESMValTool and link the ESMValTool pull request(s) in the ESMValCore pull request description. You can ask the @ESMValGroup/esmvaltool-recipe-maintainers for help with updating existing recipes, but please be considerate of their time. You should tag the @ESMValGroup/esmvaltool-coreteam to notify them of the backward-incompatible change, and give at least 2 weeks for objections to be raised before merging to the main branch. If a strong objection is raised the backwards-incompatible change should not be merged until the objection is resolved.
When reviewing a pull request, always check for backward incompatible changes and make sure they are needed and have been discussed with the @ESMValGroup/esmvaltool-coreteam. Also, make sure the author of the pull request has created the accompanying pull request(s) to update the ESMValTool, before merging the ESMValCore pull request.
Before considering adding a new dependency, carefully check that the license of the dependency you want to add and any of its dependencies are compatible with the Apache 2.0 license that applies to the ESMValCore. Note that GPL version 2 license is considered incompatible with the Apache 2.0 license, while the compatibility of GPL version 3 license with the Apache 2.0 license is questionable. See this statement by the authors of the Apache 2.0 license for more information.
When adding or removing dependencies, please consider applying the changes in the following files:
environment.ymlcontains all the development dependencies; these are all from conda-forge
setup.pycontains all Python dependencies, regardless of their installation source
Note that packages may have a different name on conda-forge than on PyPI.
Several test jobs on CircleCI related to the installation of the tool will only run if you change the dependencies. These will be skipped for most pull requests.
When reviewing a pull request where dependencies are added or removed, always check that the changes have been applied in all relevant files.
Pull request checks#
To check that a pull request is up to standard, several automatic checks are run when you make a pull request. Read more about it in the Tests and Documentation sections. Successful checks have a green ✓ in front, a ❌ means the check failed.
If you need help with the checks, please ask the technical reviewer of your pull request for help. Ask @ESMValGroup/tech-reviewers if you do not have a technical reviewer yet.
If the checks are broken because of something unrelated to the current
pull request, please check if there is an open issue that reports the problem.
Create one if there is no issue yet.
You can attract the attention of the @ESMValGroup/esmvaltool-coreteam by
mentioning them in the issue if it looks like no-one is working on solving the
The issue needs to be fixed in a separate pull request first.
After that has been merged into the
main branch and all checks on this
branch are green again, merge it into your own branch to get the tests to pass.
When reviewing a pull request, always make sure that all checks were successful. If the Codacy check keeps failing, please run prospector locally. If necessary, ask the pull request author to do the same and to address the reported issues. See the section on code_quality for more information. Never merge a pull request with failing CircleCI or readthedocs checks.
Making a release#
The release manager makes the release, assisted by the release manager of the previous release, or if that person is not available, another previous release manager. Perform the steps listed below with two persons, to reduce the risk of error.
The previous release manager ensures the current release manager has the required administrative permissions to make the release. Consider the following services: conda-forge, DockerHub, PyPI, and readthedocs.
The release of ESMValCore is tied to the release of ESMValTool. The detailed steps can be found in the ESMValTool documentation. To start the procedure, ESMValCore gets released as a release candidate to test the recipes in ESMValTool. If bugs are found during the testing phase of the release candidate, make as many release candidates for ESMValCore as needed in order to fix them.
To make a new release of the package, be it a release candidate or the final release, follow these steps:
1. Check that all tests and builds work#
Check that the
nightlytest run on CircleCI was successful.
Check that the GitHub Actions test runs were successful.
Check that the documentation builds successfully on readthedocs.
Check that the Docker images are building successfully.
All tests should pass before making a release (branch).
2. Create a release branch#
Create a branch off the
main branch and push it to GitHub.
Ask someone with administrative permissions to set up branch protection rules
for it so only you and the person helping you with the release can push to it.
3. Increase the version number#
The version number is automatically generated from the information provided by
git using [setuptools-scm](https://pypi.org/project/setuptools-scm/), but a
static version number is stored in
Make sure to update the version number and release date in
See https://semver.org for more information on choosing a version number.
Make a pull request and get it merged into
main and cherry pick it into
the release branch.
4. Add release notes#
Use the script
to create create a draft of the release notes.
This script uses the titles and labels of merged pull requests since the
Open a discussion to allow members of the development team to nominate pull
requests as highlights. Add the most voted pull requests as highlights at the
beginning of changelog. After the highlights section, list any backward
incompatible changes that the release may include. The
backward compatibility policy.
lists the information that should be provided by the developer of any backward
incompatible change. Make sure to also list any deprecations that the release
may include, as well as a brief description on how to upgrade a deprecated feature.
Review the results, and if anything needs changing, change it on GitHub and
re-run the script until the changelog looks acceptable.
Copy the result to the file
Make a pull request and get it merged into
main and cherry pick it into
the release branch.
5. Make the (pre-)release on GitHub#
Do a final check that all tests on CircleCI and GitHub Actions completed
Then click the
and create the new release from the release branch (i.e. not from
Create a tag and tick the This is a pre-release box if working with a release candidate.
6. Create and upload the PyPI package#
The package is automatically uploaded to the PyPI by a GitHub action. If has failed for some reason, build and upload the package manually by following the instructions below.
Follow these steps to create a new Python package:
Check out the tag corresponding to the release, e.g.
git checkout tags/v2.1.0
Make sure your current working directory is clean by checking the output of
git statusand by running
git clean -xdfto remove any files ignored by git.
Install the required packages:
python3 -m pip install --upgrade pep517 twine
Build the package:
python3 -m pep517.build --source --binary --out-dir dist/ .This command should generate two files in the
Upload the package:
python3 -m twine upload dist/*You will be prompted for an API token if you have not set this up before, see here for more information.
You can read more about this in Packaging Python Projects.
7. Create the Conda package#
esmvalcore package is published on the conda-forge conda channel.
This is done via a pull request on the esmvalcore-feedstock repository.
To publish a release candidate, you have to open a pull request yourself. An example for this can be found here. Make sure to use the rc branch as the target branch for your pull request and follow all instructions given by the linter bot. The testing of ESMValTool will be performed with the published release candidate.
For the final release, this pull request is automatically opened by a bot. An example pull request can be found here. Follow the instructions by the bot to finalize the pull request. This step mostly contains updating dependencies that have been changed during the last release cycle. Once approved by the feedstock maintainers they will merge the pull request, which will in turn publish the package on conda-forge some time later. Contact the feedstock maintainers if you want to become a maintainer yourself.
8. Check the Docker images#
There are two main Docker container images available for ESMValCore on Dockerhub:
esmvalgroup/esmvalcore:stable, built from docker/Dockerfile, this is a tag that is always the same as the latest released version. This image is only built by Dockerhub when a new release is created.
esmvalgroup/esmvalcore:development, built from docker/Dockerfile.dev, this is a tag that always contains the latest conda environment for ESMValCore, including any test dependencies. It is used by CircleCI to run the unit tests. This speeds up running the tests, as it avoids the need to build the conda environment for every test run. This image is built by Dockerhub every time there is a new commit to the
mainbranch on Github.
In addition to the two images mentioned above, there is an image available
for every release (e.g.
When working on the Docker images, always try to follow the
After making the release, check that the Docker image for that release has been built correctly by
checking that the version tag is available on Dockerhub and the
stabletag has been updated,
running some recipes with the
stabletag Docker container, for example one recipe for Python, NCL, R, and Julia,
running a recipe with a Singularity container built from the
If there is a problem with the automatically built container image, you can fix the problem and build a new image locally. For example, to build and upload the container image for v2.5.0 of the tool run:
git checkout v2.5.0 git clean -x docker build -t esmvalgroup/esmvalcore:v2.5.0 . -f docker/Dockerfile docker push esmvalgroup/esmvalcore:v2.5.0
(when making updates, you may want to add .post0, .post1, .. to the version number to avoid overwriting an older tag) and if it is the latest release that you are updating, also run
docker tag esmvalgroup/esmvalcore:v2.5.0 esmvalgroup/esmvalcore:stable docker push esmvalgroup/esmvalcore:stable
Note that the
docker push command will overwrite the existing tags on
Dockerhub, but the previous container image will remain available as an