anemoi-datasets 0.5.0__tar.gz → 0.5.5__tar.gz

anemoi_datasets-0.5.5/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,40 @@
+---
+name: Bug report
+about: Create a report to help us improve. **❗ IMPORTANT:** If you have difficulties creating a dataset from any source, please provide us with a sample data file or a URL to the source data.
+title: ''
+labels: ''
+assignees: ''
+
+---
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**URL to sample input data**
+Provide a URL to a sample input data, or attach a file to that report if it is small enough.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/.gitignore

@@ -194,3 +194,4 @@ _dev/
 test.ipynb
 *tmp_data/
 tempCodeRunnerFile.python
+Untitled-*.py

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/.pre-commit-config.yaml

@@ -44,7 +44,7 @@ repos:
   hooks:
   - id: ruff
     # Next line if for documenation cod snippets
-    exclude: '^[^_].*_\.py$'
+    exclude: '.*/[^_].*_\.py$'
     args:
     - --line-length=120
     - --fix

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/CHANGELOG.md

@@ -8,11 +8,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 Please add your functional changes to the appropriate section in the PR.
 Keep it human-readable, your future self will thank you!
 
-## [Unreleased]
+
+## [Unreleased](https://github.com/ecmwf/anemoi-datasets/compare/0.5.4...HEAD)
+
+## [Add support for (semi-)constant fields](https://github.com/ecmwf/anemoi-datasets/compare/0.5.1...0.5.2) - 2024-10-03
+
+### Changed
+
+- Fix failing zarr lat/lon tests
+
+## [Bug fixes](https://github.com/ecmwf/anemoi-datasets/compare/0.5.0...0.5.1) - 2024-10-01
+
+### Added
+
+- Adding the user recipe in the dataset PR #59.
+- Add `repeated_dates` action in create.
+
+### Changed
+
+- Bug fix in create/rename
+
+## [0.5.0 - Incremental builds and Rescaling](https://github.com/ecmwf/anemoi-datasets/compare/0.4.5...0.5.0) - 2024-09-25
 
 ### Added
 
 - New `rescale` keyword in `open_dataset` to change units of variables #36
+- Add support for constant fields when creating datasets
 - Simplify imports
 
 ### Changed
@@ -25,9 +46,10 @@ Keep it human-readable, your future self will thank you!
 - add paths-ignore to ci workflow
 
 ### Removed
+
 - pytest for notebooks
 
-## [0.4.5]
+## [0.4.5](https://github.com/ecmwf/anemoi-datasets/compare/0.4.4...0.4.5)
 
 ### Added
 
@@ -47,48 +69,48 @@ Keep it human-readable, your future self will thank you!
 
 ### Removed
 
-## [0.4.4] Bug fixes
+## [0.4.4](https://github.com/ecmwf/anemoi-datasets/compare/0.4.0...0.4.4) Bug fixes
 
 - Bug fix in accumulations() that did not work with webmars
 - Bug fix when using hindcasts input (missing dates on leap years)
 
-## [0.4.0] Minor Release
+## [0.4.0](https://github.com/ecmwf/anemoi-datasets/compare/0.3.0...0.4.0) Minor Release
 
 ### Added
+
 - earthkit-data replaces climetlab
 
 ### Removed
+
 - climetlab
 
-## [0.3.0] Minor Release
+## [0.3.0](https://github.com/ecmwf/anemoi-datasets/compare/0.2.0...0.3.0) Minor Release
 
 ### Added
+
 - hindcast source
 
 ### Changed
+
 - updated documentation
 
-## [0.2.0] Minor Release
+## [0.2.0](https://github.com/ecmwf/anemoi-datasets/compare/0.1.0...0.2.0) Minor Release
 
 ### Added
+
 - statistics tendencies
 
 ### Removed
+
 - CubesFilter
 
-## [0.1.0] Initial Release
+## [0.1.0](https://github.com/ecmwf/anemoi-models/releases/tag/0.1.0) Initial Release
 
 ### Added
+
 - Documentation
 - Initial code release for anemoi-datasets: create datasets for data-driven weather-models
 - open datasets
 - combine datasets
 
 ## Git Diffs:
-[Unreleased]: https://github.com/ecmwf/anemoi-datasets/compare/0.4.5...HEAD
-[0.4.5]: https://github.com/ecmwf/anemoi-datasets/compare/0.4.4...0.4.5
-[0.4.4]: https://github.com/ecmwf/anemoi-datasets/compare/0.4.0...0.4.4
-[0.4.0]: https://github.com/ecmwf/anemoi-datasets/compare/0.3.0...0.4.0
-[0.3.0]: https://github.com/ecmwf/anemoi-datasets/compare/0.2.0...0.3.0
-[0.2.0]: https://github.com/ecmwf/anemoi-datasets/compare/0.1.0...0.2.0
-[0.1.0]: https://github.com/ecmwf/anemoi-models/releases/tag/0.1.0

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: anemoi-datasets
-Version: 0.5.0
+Version: 0.5.5
 Summary: A package to hold various functions to support training of ML models on ECMWF data.
 Author-email: "European Centre for Medium-Range Weather Forecasts (ECMWF)" <software.support@ecmwf.int>
 License:                                  Apache License

anemoi_datasets-0.5.5/docs/building/filters/rename.rst

@@ -0,0 +1,6 @@
+########
+ rename
+########
+
+.. literalinclude:: yaml/rename.yaml
+   :language: yaml

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/building/sources/accumulations.rst

@@ -54,12 +54,12 @@ accumulation between step 1h and step 7h if it is the most appropriate
 data available, while in the second case, the source will always return
 the accumulation between step 0h and step 6h, if available.
 
-.. literalinclude:: accumulations1.yaml
+.. literalinclude:: yaml/accumulations1.yaml
    :language: yaml
 
 or:
 
-.. literalinclude:: accumulations2.yaml
+.. literalinclude:: yaml/accumulations2.yaml
    :language: yaml
 
 .. [1]

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/building/sources/forcings.rst

@@ -11,7 +11,7 @@ Because the source needs to generate fields on the same grids as the
 other, it requires a template field. This is provided in the recipe with
 the ``template`` keyword:
 
-.. literalinclude:: forcings.yaml
+.. literalinclude:: yaml/forcings.yaml
    :language: yaml
 
 .. _yaml-reference:

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/building/sources/mars.rst

@@ -13,13 +13,13 @@ specification`_, with the exception of the ``date``, ``time``` and
 The missing keys will be filled with the default values, as defined in
 the MARS language specification.
 
-.. literalinclude:: mars1.yaml
+.. literalinclude:: yaml/mars1.yaml
    :language: yaml
 
 Data from several levels types must be requested in separate requests,
 with the ``join`` command.
 
-.. literalinclude:: mars2.yaml
+.. literalinclude:: yaml/mars2.yaml
    :language: yaml
 
 See :ref:`naming-variables` for information on how to name the variables

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/building/sources/netcdf.rst

@@ -2,5 +2,5 @@
  netcdf
 ########
 
-.. literalinclude:: netcdf.yaml
+.. literalinclude:: yaml/netcdf.yaml
    :language: yaml

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/building/sources/opendap.rst

@@ -2,5 +2,5 @@
  opendap
 #########
 
-.. literalinclude:: opendap.yaml
+.. literalinclude:: yaml/opendap.yaml
    :language: yaml

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/building/sources/recentre.rst

@@ -19,7 +19,7 @@ formula:
 
 Additionally, some variables must be non-negative to have a physical
 meaning (e.g. accumulated variables or `specific humidity`). To ensure
-this, positive clipping is performed using the alternative fomula :
+this, positive clipping is performed using the alternative formula:
 
 .. math::
 

anemoi_datasets-0.5.5/docs/building/sources/repeated_dates.rst

@@ -0,0 +1,29 @@
+################
+ repeated_dates
+################
+
+The generale format of the `repeated_dates` source is:
+
+.. literalinclude:: yaml/repeated_dates1.yaml
+   :language: yaml
+
+**********
+ constant
+**********
+
+.. literalinclude:: yaml/repeated_dates2.yaml
+   :language: yaml
+
+*************
+ climatology
+*************
+
+.. literalinclude:: yaml/repeated_dates3.yaml
+   :language: yaml
+
+*********
+ closest
+*********
+
+.. literalinclude:: yaml/repeated_dates4.yaml
+   :language: yaml

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/building/sources/xarray-zarr.rst

@@ -2,5 +2,5 @@
  xarray-zarr
 #############
 
-.. literalinclude:: xarray-zarr.yaml
+.. literalinclude:: yaml/xarray-zarr.yaml
    :language: yaml

anemoi_datasets-0.5.5/docs/building/sources/yaml/hindcasts.yaml

@@ -0,0 +1,3 @@
+---
+input:
+   hincasts:

anemoi_datasets-0.5.5/docs/building/sources/yaml/repeated_dates1.yaml

@@ -0,0 +1,6 @@
+
+repeated_dates:
+    mode: mode
+    # ... parameters related to the mode ...
+    source:
+      # ... a source definition ...

anemoi_datasets-0.5.5/docs/building/sources/yaml/repeated_dates2.yaml

@@ -0,0 +1,6 @@
+repeated_dates:
+  mode: constant
+  source:
+      xarray-zarr:
+        url: dem.zarr
+        variable: dem

anemoi_datasets-0.5.5/docs/building/sources/yaml/repeated_dates3.yaml

@@ -0,0 +1,8 @@
+repeated_dates:
+    mode: climatology
+    year: 2019
+    day: 15
+    source:
+        grib:
+          path: some/path/to/data.grib
+          param: [some_param]

anemoi_datasets-0.5.5/docs/building/sources/yaml/repeated_dates4.yaml

@@ -0,0 +1,9 @@
+repeated_dates:
+    mode: closest
+    frequency: 24h
+    maximum: 30d
+    skip_all_nans: true
+    source:
+      grib:
+          path: path/to/data.grib
+          param: [some_param]

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/building/sources/zenodo.rst

@@ -4,5 +4,5 @@
 
 https://zenodo.org/records/6470725
 
-.. literalinclude:: zenodo.yaml
+.. literalinclude:: yaml/zenodo.yaml
    :language: yaml

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/building/sources.rst

@@ -28,6 +28,7 @@ The following `sources` are currently available:
    sources/netcdf
    sources/opendap
    sources/recentre
+   sources/repeated_dates
    sources/xarray-kerchunk
    sources/xarray-zarr
    sources/zenodo

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/docs/index.rst

@@ -1,8 +1,8 @@
 .. _index-page:
 
-####################################
- Welcome to Anemoi's documentation!
-####################################
+#############################################
+ Welcome to `anemoi-datasets` documentation!
+#############################################
 
 .. warning::
 

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/_version.py

@@ -12,5 +12,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.5.0'
-__version_tuple__ = version_tuple = (0, 5, 0)
+__version__ = version = '0.5.5'
+__version_tuple__ = version_tuple = (0, 5, 5)

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/commands/inspect.py

@@ -311,7 +311,7 @@ class Version:
                 print(f"🕰️  Dataset initialized {when(start)}.")
                 if built and latest:
                     speed = (latest - start) / built
-                    eta = datetime.datetime.utcnow() + speed * (total - built)
+                    eta = datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None) + speed * (total - built)
                     print(f"🏁 ETA {when(eta)}.")
         else:
             if latest:

anemoi_datasets-0.5.5/src/anemoi/datasets/commands/publish.py

@@ -0,0 +1,30 @@
+import logging
+
+from . import Command
+
+LOG = logging.getLogger(__name__)
+
+
+class Publish(Command):
+    """Publish a dataset."""
+
+    # This is a command that is used to publish a dataset.
+    # it is a class, inheriting from Command.
+
+    internal = True
+    timestamp = True
+
+    def add_arguments(self, parser):
+        parser.add_argument("path", help="Path of the dataset to publish.")
+
+    def run(self, args):
+        try:
+            from anemoi.registry import publish_dataset
+        except ImportError:
+            LOG.error("anemoi-registry is not installed. Please install it to use this command.")
+            return
+
+        publish_dataset(args.path)
+
+
+command = Publish

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/create/__init__.py

@@ -14,6 +14,7 @@ import os
 import time
 import uuid
 import warnings
+from copy import deepcopy
 from functools import cached_property
 
 import numpy as np
@@ -24,9 +25,11 @@ from anemoi.utils.dates import frequency_to_string
 from anemoi.utils.dates import frequency_to_timedelta
 from anemoi.utils.humanize import compress_dates
 from anemoi.utils.humanize import seconds_to_human
+from earthkit.data.core.order import build_remapping
 
 from anemoi.datasets import MissingDateError
 from anemoi.datasets import open_dataset
+from anemoi.datasets.create.input.trace import enable_trace
 from anemoi.datasets.create.persistent import build_storage
 from anemoi.datasets.data.misc import as_first_date
 from anemoi.datasets.data.misc import as_last_date
@@ -308,7 +311,6 @@ class HasElementForDataMixin:
 
 
 def build_input_(main_config, output_config):
-    from earthkit.data.core.order import build_remapping
 
     builder = build_input(
         main_config.input,
@@ -323,6 +325,43 @@ def build_input_(main_config, output_config):
     return builder
 
 
+def tidy_recipe(config: object):
+    """Remove potentially private information in the config"""
+    config = deepcopy(config)
+    if isinstance(config, (tuple, list)):
+        return [tidy_recipe(_) for _ in config]
+    if isinstance(config, (dict, DotDict)):
+        for k, v in config.items():
+            if k.startswith("_"):
+                config[k] = "*** REMOVED FOR SECURITY ***"
+            else:
+                config[k] = tidy_recipe(v)
+    if isinstance(config, str):
+        if config.startswith("_"):
+            return "*** REMOVED FOR SECURITY ***"
+        if config.startswith("s3://"):
+            return "*** REMOVED FOR SECURITY ***"
+        if config.startswith("gs://"):
+            return "*** REMOVED FOR SECURITY ***"
+        if config.startswith("http"):
+            return "*** REMOVED FOR SECURITY ***"
+        if config.startswith("ftp"):
+            return "*** REMOVED FOR SECURITY ***"
+        if config.startswith("file"):
+            return "*** REMOVED FOR SECURITY ***"
+        if config.startswith("ssh"):
+            return "*** REMOVED FOR SECURITY ***"
+        if config.startswith("scp"):
+            return "*** REMOVED FOR SECURITY ***"
+        if config.startswith("rsync"):
+            return "*** REMOVED FOR SECURITY ***"
+        if config.startswith("/"):
+            return "*** REMOVED FOR SECURITY ***"
+        if "@" in config:
+            return "*** REMOVED FOR SECURITY ***"
+    return config
+
+
 class Init(Actor, HasRegistryMixin, HasStatisticTempMixin, HasElementForDataMixin):
     dataset_class = NewDataset
     def __init__(self, path, config, check_name=False, overwrite=False, use_threads=False, statistics_temp_dir=None, progress=None, test=False, cache=None, **kwargs):  # fmt: skip
@@ -409,6 +448,7 @@ class Init(Actor, HasRegistryMixin, HasStatisticTempMixin, HasElementForDataMixi
         metadata.update(self.main_config.get("add_metadata", {}))
 
         metadata["_create_yaml_config"] = self.main_config.get_serialisable_dict()
+        metadata["recipe"] = tidy_recipe(self.main_config.get_serialisable_dict())
 
         metadata["description"] = self.main_config.description
         metadata["licence"] = self.main_config["licence"]
@@ -524,7 +564,7 @@ class Load(Actor, HasRegistryMixin, HasStatisticTempMixin, HasElementForDataMixi
             # assert isinstance(group[0], datetime.datetime), type(group[0])
             LOG.debug(f"Building data for group {igroup}/{self.n_groups}")
 
-            result = self.input.select(dates=group)
+            result = self.input.select(group_of_dates=group)
             assert result.group_of_dates == group, (len(result.group_of_dates), len(group), group)
 
             # There are several groups.
@@ -992,7 +1032,6 @@ def chain(tasks):
 
 def creator_factory(name, trace=None, **kwargs):
     if trace:
-        from anemoi.datasets.create.trace import enable_trace
 
         enable_trace(trace)
 

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/create/check.py

@@ -140,9 +140,15 @@ class StatisticsValueError(ValueError):
 
 def check_data_values(arr, *, name: str, log=[], allow_nans=False):
 
+    shape = arr.shape
+
     if (isinstance(allow_nans, (set, list, tuple, dict)) and name in allow_nans) or allow_nans:
         arr = arr[~np.isnan(arr)]
 
+    if arr.size == 0:
+        warnings.warn(f"Empty array for {name} ({shape})")
+        return
+
     assert arr.size > 0, (name, *log)
 
     min, max = arr.min(), arr.max()

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/create/functions/filters/rename.py

@@ -32,7 +32,7 @@ class RenamedFieldMapping:
 
         value = self.field.metadata(key, **kwargs)
         if key == self.what:
-            return self.renaming.get(value, value)
+            return self.renaming.get(self.what, {}).get(value, value)
 
         return value
 
@@ -68,8 +68,7 @@ class RenamedFieldFormat:
 
 
 def execute(context, input, what="param", **kwargs):
-    # print('🍍🍍🍍🍍🍍🍍🍍🍍🍍🍍🍍🍍🍍 ==========', kwargs)
-    if what in kwargs:
+    if what in kwargs and isinstance(kwargs[what], str):
         return FieldArray([RenamedFieldFormat(fs, kwargs[what]) for fs in input])
 
     return FieldArray([RenamedFieldMapping(fs, what, kwargs) for fs in input])

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/create/functions/sources/__init__.py

@@ -16,6 +16,10 @@ LOG = logging.getLogger(__name__)
 
 
 def _expand(paths):
+
+    if not isinstance(paths, list):
+        paths = [paths]
+
     for path in paths:
         if path.startswith("file://"):
             path = path[7:]
@@ -40,8 +44,10 @@ def iterate_patterns(path, dates, **kwargs):
     given_paths = path if isinstance(path, list) else [path]
 
     dates = [d.isoformat() for d in dates]
+    if len(dates) > 0:
+        kwargs["date"] = dates
 
     for path in given_paths:
-        paths = Pattern(path, ignore_missing_keys=True).substitute(date=dates, **kwargs)
+        paths = Pattern(path, ignore_missing_keys=True).substitute(**kwargs)
         for path in _expand(paths):
             yield path, dates

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/create/functions/sources/accumulations.py

@@ -375,6 +375,8 @@ def accumulations(context, dates, **request):
         ("od", "elda"): dict(base_times=(6, 18)),
         ("ea", "oper"): dict(data_accumulation_period=1, base_times=(6, 18)),
         ("ea", "enda"): dict(data_accumulation_period=3, base_times=(6, 18)),
+        ("rr", "oper"): dict(data_accumulation_period=3, base_times=(0, 3, 6, 9, 12, 15, 18, 21)),
+        ("l5", "oper"): dict(data_accumulation_period=1, base_times=(0,)),
     }
 
     kwargs = KWARGS.get((class_, stream), {})

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/create/functions/sources/grib.py

@@ -135,7 +135,7 @@ def execute(context, dates, path, latitudes=None, longitudes=None, *args, **kwar
             s = s.sel(valid_datetime=dates, **kwargs)
             ds = ds + s
 
-    if kwargs:
+    if kwargs and not context.partial_ok:
         check(ds, given_paths, valid_datetime=dates, **kwargs)
 
     if geography is not None:

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/create/functions/sources/xarray/__init__.py

@@ -12,6 +12,7 @@ import logging
 from earthkit.data.core.fieldlist import MultiFieldList
 
 from anemoi.datasets.data.stores import name_to_zarr_store
+from anemoi.datasets.utils.fields import NewMetadataField as NewMetadataField
 
 from .. import iterate_patterns
 from .fieldlist import XarrayFieldList
@@ -49,7 +50,11 @@ def load_one(emoji, context, dates, dataset, options={}, flavour=None, **kwargs)
         data = xr.open_dataset(dataset, **options)
 
     fs = XarrayFieldList.from_xarray(data, flavour)
-    result = MultiFieldList([fs.sel(valid_datetime=date, **kwargs) for date in dates])
+
+    if len(dates) == 0:
+        return fs.sel(**kwargs)
+    else:
+        result = MultiFieldList([fs.sel(valid_datetime=date, **kwargs) for date in dates])
 
     if len(result) == 0:
         LOG.warning(f"No data found for {dataset} and dates {dates} and {kwargs}")

{anemoi_datasets-0.5.0 → anemoi_datasets-0.5.5}/src/anemoi/datasets/create/functions/sources/xarray/coordinates.py

@@ -56,6 +56,8 @@ class Coordinate:
     is_step = False
     is_date = False
     is_member = False
+    is_x = False
+    is_y = False
 
     def __init__(self, variable):
         self.variable = variable
@@ -66,10 +68,11 @@ class Coordinate:
         return 1 if self.scalar else len(self.variable)
 
     def __repr__(self):
-        return "%s[name=%s,values=%s]" % (
+        return "%s[name=%s,values=%s,shape=%s]" % (
             self.__class__.__name__,
             self.variable.name,
             self.variable.values if self.scalar else len(self),
+            self.variable.shape,
         )
 
     def reduced(self, i):
@@ -225,11 +228,13 @@ class LatitudeCoordinate(Coordinate):
 
 class XCoordinate(Coordinate):
     is_grid = True
+    is_x = True
     mars_names = ("x",)
 
 
 class YCoordinate(Coordinate):
     is_grid = True
+    is_y = True
     mars_names = ("y",)