Composing recipes#
This notebooks shows how to fill the datasets section in a recipe.
[1]:
from esmvalcore.config import CFG
from esmvalcore.dataset import Dataset, datasets_to_recipe
import yaml
Configure ESMValCore so it always searches the ESGF for data
[2]:
CFG['search_esgf'] = 'always'
Here is a small example recipe, that uses the datasets_to_recipe
function to convert a list of datasets to a recipe:
[3]:
tas = Dataset(
short_name='tas',
mip='Amon',
project='CMIP6',
dataset='CanESM5-1',
ensemble='r1i1p1f1',
exp='historical',
grid='gn',
timerange='2000/2002',
)
tas['diagnostic'] = 'diagnostic_name'
pr = tas.copy(short_name='pr')
print(yaml.safe_dump(datasets_to_recipe([tas, pr])))
datasets:
- dataset: CanESM5-1
diagnostics:
diagnostic_name:
variables:
pr:
ensemble: r1i1p1f1
exp: historical
grid: gn
mip: Amon
project: CMIP6
timerange: 2000/2002
tas:
ensemble: r1i1p1f1
exp: historical
grid: gn
mip: Amon
project: CMIP6
timerange: 2000/2002
A more ambitious recipe might want to use all data that is available on ESGF. We can define a dataset template with a facet value of *
where any value can be used. This can then be expanded to a list of datasets using the from_files()
method.
[4]:
dataset_template = Dataset(
short_name='tas',
mip='Amon',
project='CMIP6',
exp='historical',
dataset='*',
institute='*',
ensemble='*',
grid='*',
)
datasets = list(dataset_template.from_files())
len(datasets)
[4]:
778
This results in the following recipe:
[5]:
for dataset in datasets:
dataset.facets['diagnostic'] = 'diagnostic_name'
print(yaml.safe_dump(datasets_to_recipe(datasets)))
datasets:
- dataset: TaiESM1
ensemble: r(1:2)i1p1f1
grid: gn
institute: AS-RCEC
- dataset: AWI-CM-1-1-MR
ensemble: r(1:5)i1p1f1
grid: gn
institute: AWI
- dataset: AWI-ESM-1-1-LR
ensemble: r1i1p1f1
grid: gn
institute: AWI
- dataset: BCC-CSM2-MR
ensemble: r(1:3)i1p1f1
grid: gn
institute: BCC
- dataset: BCC-ESM1
ensemble: r(1:3)i1p1f1
grid: gn
institute: BCC
- dataset: CAMS-CSM1-0
ensemble: r1i1p1f2
grid: gn
institute: CAMS
- dataset: CAMS-CSM1-0
ensemble: r(1:2)i1p1f1
grid: gn
institute: CAMS
- dataset: CAS-ESM2-0
ensemble: r(1:4)i1p1f1
grid: gn
institute: CAS
- dataset: FGOALS-f3-L
ensemble: r(1:3)i1p1f1
grid: gr
institute: CAS
- dataset: FGOALS-g3
ensemble: r(1:6)i1p1f1
grid: gn
institute: CAS
- dataset: IITM-ESM
ensemble: r1i1p1f1
grid: gn
institute: CCCR-IITM
- dataset: CanESM5-1
ensemble: r(1:20)i1p1f1
grid: gn
institute: CCCma
- dataset: CanESM5-1
ensemble: r(1:25)i1p2f1
grid: gn
institute: CCCma
- dataset: CanESM5-1
ensemble: r22i1p1f1
grid: gn
institute: CCCma
- dataset: CanESM5-1
ensemble: r(24:39)i1p1f1
grid: gn
institute: CCCma
- dataset: CanESM5-1
ensemble: r(41:50)i1p1f1
grid: gn
institute: CCCma
- dataset: CanESM5-CanOE
ensemble: r(1:3)i1p2f1
grid: gn
institute: CCCma
- dataset: CanESM5
ensemble: r(1:25)i1p1f1
grid: gn
institute: CCCma
- dataset: CanESM5
ensemble: r(1:40)i1p2f1
grid: gn
institute: CCCma
- dataset: CMCC-CM2-HR4
ensemble: r1i1p1f1
grid: gn
institute: CMCC
- dataset: CMCC-CM2-SR5
ensemble: r1i1p1f1
grid: gn
institute: CMCC
- dataset: CMCC-CM2-SR5
ensemble: r(2:11)i1p2f1
grid: gn
institute: CMCC
- dataset: CMCC-ESM2
ensemble: r1i1p1f1
grid: gn
institute: CMCC
- dataset: CNRM-CM6-1-HR
ensemble: r1i1p1f2
grid: gr
institute: CNRM-CERFACS
- dataset: CNRM-CM6-1
ensemble: r(1:30)i1p1f2
grid: gr
institute: CNRM-CERFACS
- dataset: CNRM-ESM2-1
ensemble: r(1:11)i1p1f2
grid: gr
institute: CNRM-CERFACS
- dataset: ACCESS-CM2
ensemble: r(1:10)i1p1f1
grid: gn
institute: CSIRO-ARCCSS
- dataset: ACCESS-ESM1-5
ensemble: r(1:40)i1p1f1
grid: gn
institute: CSIRO
- dataset: E3SM-1-0
ensemble: r(1:5)i1p1f1
grid: gr
institute: E3SM-Project
- dataset: E3SM-1-1-ECA
ensemble: r1i1p1f1
grid: gr
institute: E3SM-Project
- dataset: E3SM-1-1
ensemble: r1i1p1f1
grid: gr
institute: E3SM-Project
- dataset: E3SM-2-0
ensemble: r(1:5)i1p1f1
grid: gr
institute: E3SM-Project
- dataset: EC-Earth3-AerChem
ensemble: r1i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3-AerChem
ensemble: r(3:4)i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3-CC
ensemble: r1i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3-CC
ensemble: r4i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3-CC
ensemble: r(6:13)i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3-Veg-LR
ensemble: r(1:3)i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3-Veg
ensemble: r(1:6)i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3-Veg
ensemble: r10i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3-Veg
ensemble: r12i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3-Veg
ensemble: r14i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3
ensemble: r(1:7)i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3
ensemble: r(9:25)i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: EC-Earth3
ensemble: r(101:150)i1p1f1
grid: gr
institute: EC-Earth-Consortium
- dataset: FIO-ESM-2-0
ensemble: r(1:3)i1p1f1
grid: gn
institute: FIO-QLNM
- dataset: MPI-ESM-1-2-HAM
ensemble: r(1:3)i1p1f1
grid: gn
institute: HAMMOZ-Consortium
- dataset: INM-CM4-8
ensemble: r1i1p1f1
grid: gr1
institute: INM
- dataset: INM-CM5-0
ensemble: r(1:10)i1p1f1
grid: gr1
institute: INM
- dataset: IPSL-CM5A2-INCA
ensemble: r1i1p1f1
grid: gr
institute: IPSL
- dataset: IPSL-CM6A-LR-INCA
ensemble: r1i1p1f1
grid: gr
institute: IPSL
- dataset: IPSL-CM6A-LR
ensemble: r(1:33)i1p1f1
grid: gr
institute: IPSL
- dataset: KIOST-ESM
ensemble: r1i1p1f1
grid: gr1
institute: KIOST
- dataset: MIROC-ES2H
ensemble: r1i1p(1:3)f2
grid: gn
institute: MIROC
- dataset: MIROC-ES2H
ensemble: r(1:3)i1p4f2
grid: gn
institute: MIROC
- dataset: MIROC-ES2L
ensemble: r1i1000p1f2
grid: gn
institute: MIROC
- dataset: MIROC-ES2L
ensemble: r(1:30)i1p1f2
grid: gn
institute: MIROC
- dataset: MIROC6
ensemble: r(1:50)i1p1f1
grid: gn
institute: MIROC
- dataset: HadGEM3-GC31-LL
ensemble: r(1:5)i1p1f3
grid: gn
institute: MOHC
- dataset: HadGEM3-GC31-MM
ensemble: r(1:4)i1p1f3
grid: gn
institute: MOHC
- dataset: UKESM1-0-LL
ensemble: r(1:4)i1p1f2
grid: gn
institute: MOHC
- dataset: UKESM1-0-LL
ensemble: r(5:7)i1p1f3
grid: gn
institute: MOHC
- dataset: UKESM1-0-LL
ensemble: r(8:12)i1p1f2
grid: gn
institute: MOHC
- dataset: UKESM1-0-LL
ensemble: r(16:19)i1p1f2
grid: gn
institute: MOHC
- dataset: UKESM1-1-LL
ensemble: r1i1p1f2
grid: gn
institute: MOHC
- dataset: ICON-ESM-LR
ensemble: r(1:5)i1p1f1
grid: gn
institute: MPI-M
- dataset: MPI-ESM1-2-HR
ensemble: r(1:10)i1p1f1
grid: gn
institute: MPI-M
- dataset: MPI-ESM1-2-LR
ensemble: r1i2000p1f1
grid: gn
institute: MPI-M
- dataset: MPI-ESM1-2-LR
ensemble: r(1:30)i1p1f1
grid: gn
institute: MPI-M
- dataset: MRI-ESM2-0
ensemble: r1i2p1f1
grid: gn
institute: MRI
- dataset: MRI-ESM2-0
ensemble: r1i1000p1f1
grid: gn
institute: MRI
- dataset: MRI-ESM2-0
ensemble: r(1:10)i1p1f1
grid: gn
institute: MRI
- dataset: GISS-E2-1-G-CC
ensemble: r1i1p1f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-G
ensemble: r(1:4)i1p5f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-G
ensemble: r(1:5)i1p1f3
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-G
ensemble: r(1:10)i1p1f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-G
ensemble: r(1:10)i1p3f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-G
ensemble: r(1:11)i1p1f2
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-G
ensemble: r(6:10)i1p5f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-G
ensemble: r(101:102)i1p1f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-H
ensemble: r(1:5)i1p1f2
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-H
ensemble: r(1:5)i1p3f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-H
ensemble: r(1:5)i1p5f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-1-H
ensemble: r(1:10)i1p1f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-2-G
ensemble: r(1:5)i1p3f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-2-G
ensemble: r(1:6)i1p1f1
grid: gn
institute: NASA-GISS
- dataset: GISS-E2-2-H
ensemble: r(1:5)i1p1f1
grid: gn
institute: NASA-GISS
- dataset: CESM2-FV2
ensemble: r1i2p2f1
grid: gn
institute: NCAR
- dataset: CESM2-FV2
ensemble: r(1:3)i1p1f1
grid: gn
institute: NCAR
- dataset: CESM2-WACCM-FV2
ensemble: r(1:3)i1p1f1
grid: gn
institute: NCAR
- dataset: CESM2-WACCM
ensemble: r(1:3)i1p1f1
grid: gn
institute: NCAR
- dataset: CESM2
ensemble: r(1:11)i1p1f1
grid: gn
institute: NCAR
- dataset: NorCPM1
ensemble: r(1:30)i1p1f1
grid: gn
institute: NCC
- dataset: NorESM2-LM
ensemble: r(1:3)i1p1f1
grid: gn
institute: NCC
- dataset: NorESM2-MM
ensemble: r(1:3)i1p1f1
grid: gn
institute: NCC
- dataset: KACE-1-0-G
ensemble: r(1:3)i1p1f1
grid: gr
institute: NIMS-KMA
- dataset: UKESM1-0-LL
ensemble: r(13:15)i1p1f2
grid: gn
institute: NIMS-KMA
- dataset: GFDL-CM4
ensemble: r1i1p1f1
grid: gr1
institute: NOAA-GFDL
- dataset: GFDL-ESM4
ensemble: r(1:3)i1p1f1
grid: gr1
institute: NOAA-GFDL
- dataset: NESM3
ensemble: r(1:5)i1p1f1
grid: gn
institute: NUIST
- dataset: SAM0-UNICON
ensemble: r1i1p1f1
grid: gn
institute: SNU
- dataset: CIESM
ensemble: r(1:3)i1p1f1
grid: gr
institute: THU
- dataset: MCM-UA-1-0
ensemble: r1i1p1f(1:2)
grid: gn
institute: UA
diagnostics:
diagnostic_name:
variables:
tas:
exp: historical
mip: Amon
project: CMIP6