Dataset#
Classes and functions for defining, finding, and loading data.
Classes:
|
Define datasets, find the related files, and load them. |
Data:
Inherited facets. |
Functions:
|
Create or update a recipe from datasets. |
- class esmvalcore.dataset.Dataset(**facets: str | Sequence[str] | Number)[source]#
Define datasets, find the related files, and load them.
- Parameters:
**facets – Facets describing the dataset. See
esmvalcore.esgf.facets.FACETS
for the mapping between the facet names used by ESMValCore and those used on ESGF.
- facets#
Facets describing the dataset.
- Type:
Methods:
add_supplementary
(**facets)Add an supplementary dataset.
Add extra facets.
copy
(**facets)Create a copy.
Find files.
Create datasets based on the available files.
Create a list of datasets from short notations.
from_recipe
(recipe, session)Read datasets from a recipe.
load
()Load dataset.
set_facet
(key, value[, persist])Set facet.
Set the
'version'
facet based on the available data.summary
([shorten])Summarize the content of dataset.
Attributes:
The files associated with this dataset.
Return a dictionary with the persistent facets.
A
esmvalcore.config.Session
associated with the dataset.- add_supplementary(**facets: str | Sequence[str] | Number) None [source]#
Add an supplementary dataset.
This is a convenience function that will create a copy of the current dataset, update its facets with the values specified in
**facets
, and append it toDataset.supplementaries
. For more control over the creation of the supplementary dataset, first create a newDataset
describing the supplementary dataset and then append it toDataset.supplementaries
.- Parameters:
**facets – Facets describing the supplementary variable.
- augment_facets() None [source]#
Add extra facets.
This function will update the dataset with additional facets from various sources.
- copy(**facets: str | Sequence[str] | Number) Dataset [source]#
Create a copy.
- Parameters:
**facets – Update these facets in the copy. Note that for supplementary datasets attached to the dataset, the
'short_name'
and'mip'
facets will not be updated with these values.- Returns:
A copy of the dataset.
- Return type:
- find_files() None [source]#
Find files.
Look for files and populate the
Dataset.files
property of the dataset and its supplementary datasets.
- from_files() Iterator[Dataset] [source]#
Create datasets based on the available files.
The facet values for local files are retrieved from the directory tree where the directories represent the facets values. Reading facet values from file names is not yet supported. See CMIP data for more information on this kind of file organization.
glob.glob()
patterns can be used as facet values to select multiple datasets. If for some of the datasets not all glob patterns can be expanded (e.g. because the required facet values cannot be inferred from the directory names), these datasets will be ignored, unless this happens to be all datasets.If
glob.glob()
patterns are used in supplementary variables and multiple matching datasets are found, only the supplementary dataset that has most facets in common with the main dataset will be attached.Supplementary datasets will in inherit the facet values from the main dataset for those facets listed in
INHERITED_FACETS
.Examples
See Discovering data for example use cases.
- Yields:
Dataset – Datasets representing the available files.
- from_ranges() list['Dataset'] [source]#
Create a list of datasets from short notations.
This expands the
'ensemble'
and'sub_experiment'
facets in the dataset definition if they are ranges.For example
'ensemble'='r(1:3)i1p1f1'
will be expanded to three datasets, with'ensemble'
values'r1i1p1f1'
,'r2i1p1f1'
,'r3i1p1f1'
.
- static from_recipe(recipe: Path | str | dict, session: Session) list['Dataset'] [source]#
Read datasets from a recipe.
- Parameters:
- Returns:
A list of datasets.
- Return type:
- load() Cube [source]#
Load dataset.
- Raises:
InputFilesNotFound – When no files were found.
- Returns:
An
iris
cube with the data corresponding the the dataset.- Return type:
- property minimal_facets: Dict[str, str | Sequence[str] | Number]#
Return a dictionary with the persistent facets.
- property session: Session#
A
esmvalcore.config.Session
associated with the dataset.
- esmvalcore.dataset.INHERITED_FACETS: list[str] = ['dataset', 'domain', 'driver', 'grid', 'project', 'timerange']#
Inherited facets.
Supplementary datasets created based on the available files using the
Dataset.from_files()
method will inherit the values of these facets from the main dataset.
- esmvalcore.dataset.datasets_to_recipe(datasets: Iterable[Dataset], recipe: Path | str | dict[str, Any] | None = None) dict [source]#
Create or update a recipe from datasets.
- Parameters:
Examples
See Composing recipes for example use cases.
- Returns:
The recipe with the datasets. To convert the
dict
to a recipe, use e.g.yaml.safe_dump()
.- Return type:
- Raises:
RecipeError – Raised when a dataset is missing the
diagnostic
facet.