anemoi-datasets 0.5.24__tar.gz → 0.5.26__tar.gz

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/.github/pull_request_template.md

@@ -11,3 +11,5 @@
 <!-- Include any additional information, caveats, or considerations that the reviewer should be aware of. -->
 
 ***As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting  (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/***
+
+By opening this pull request, I affirm that all authors agree to the [Contributor License Agreement.](https://github.com/ecmwf/codex/blob/main/Legal/contributor_license_agreement.md)

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/.pre-commit-config.yaml

@@ -41,7 +41,7 @@ repos:
     - --profile black
     - --project anemoi
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.11.4
+  rev: v0.11.12
   hooks:
   - id: ruff
     args:
@@ -65,7 +65,7 @@ repos:
   - id: docconvert
     args: ["numpy"]
 - repo: https://github.com/tox-dev/pyproject-fmt
-  rev: "v2.5.1"
+  rev: "v2.6.0"
   hooks:
   - id: pyproject-fmt
 -   repo: https://github.com/jshwi/docsig # Check docstrings against function sig

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/.release-please-config.json

@@ -10,11 +10,16 @@
   "draft-pull-request": true,
   "pull-request-title-pattern": "chore${scope}: Release${component} ${version}",
   "pull-request-header": ":robot: Automated Release PR\n\nThis PR was created by `release-please` to prepare the next release. Once merged:\n\n1. A new version tag will be created\n2. A GitHub release will be published\n3. The changelog will be updated\n\nChanges to be included in the next release:",
-  "pull-request-footer": "> [!IMPORTANT]\n> :warning: Merging this PR will:\n> - Create a new release\n> - Trigger deployment pipelines\n> - Update package versions\n\n **Before merging:**\n - Ensure all tests pass\n - Review the changelog carefully\n - Get required approvals\n\n [Release-please documentation](https://github.com/googleapis/release-please)",
+  "pull-request-footer": "> [!IMPORTANT]\n> Please do not change the PR title, manifest file, or any other automatically generated content in this PR unless you understand the implications. Changes here can break the release process.\n> :warning: Merging this PR will:\n> - Create a new release\n> - Trigger deployment pipelines\n> - Update package versions\n\n **Before merging:**\n - Ensure all tests pass\n - Review the changelog carefully\n - Get required approvals\n\n [Release-please documentation](https://github.com/googleapis/release-please)",
   "packages": {
     ".": {
         "package-name": "anemoi-datasets"
     }
   },
+  "plugins": [
+    {
+      "type": "sentence-case"
+    }
+  ],
   "$schema": "https://raw.githubusercontent.com/googleapis/release-please/main/schemas/config.json"
 }

anemoi_datasets-0.5.26/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+    ".": "0.5.26"
+}

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/CHANGELOG.md

@@ -8,6 +8,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 Please add your functional changes to the appropriate section in the PR.
 Keep it human-readable, your future self will thank you!
 
+## [0.5.26](https://github.com/ecmwf/anemoi-datasets/compare/0.5.25...0.5.26) (2025-07-11)
+
+
+### Features
+
+* Add planetary planetary source ([#368](https://github.com/ecmwf/anemoi-datasets/issues/368)) ([c85337f](https://github.com/ecmwf/anemoi-datasets/commit/c85337f4b08381dc1e2e8ba71134c998c6258c2d))
+* Extend xarray coordinate guesser to include pressure as a long_name ([#335](https://github.com/ecmwf/anemoi-datasets/issues/335)) ([bcbffaa](https://github.com/ecmwf/anemoi-datasets/commit/bcbffaa0b0ad011db5b1fd5879f9829f0d2e68af))
+* Extented complement nearest functionality to support k>1 ([#371](https://github.com/ecmwf/anemoi-datasets/issues/371)) ([a2c5f92](https://github.com/ecmwf/anemoi-datasets/commit/a2c5f921f57fce60bd3fe728c21dc5e5b7d3a4a1))
+* Point data ([#358](https://github.com/ecmwf/anemoi-datasets/issues/358)) ([78c4f85](https://github.com/ecmwf/anemoi-datasets/commit/78c4f85bf9cd4d6cc7f17268c8c526da32158220))
+* Relaxed compatibility check for horizontal grid. ([#339](https://github.com/ecmwf/anemoi-datasets/issues/339)) ([a7dc859](https://github.com/ecmwf/anemoi-datasets/commit/a7dc859b6ec2fb0bad2ed7b7b8ee3c76034a258b))
+* Sort xarray coordinate with patch ([#356](https://github.com/ecmwf/anemoi-datasets/issues/356)) ([638cbdb](https://github.com/ecmwf/anemoi-datasets/commit/638cbdb013b94b87748654ae338e0d02e69c1900))
+* Use pytest-xdist to run tests in parallel. ([#346](https://github.com/ecmwf/anemoi-datasets/issues/346)) ([1d656ee](https://github.com/ecmwf/anemoi-datasets/commit/1d656ee62c51ee0cbe043943581cef16b2f7d7db))
+
+
+### Bug Fixes
+
+* Deprecated filters in anemoi datasets ([#350](https://github.com/ecmwf/anemoi-datasets/issues/350)) ([d08c6f5](https://github.com/ecmwf/anemoi-datasets/commit/d08c6f55c3d983c930480bd777e74fd5a5bea6eb))
+* **grib-index:** Match against filename, not full path ([#374](https://github.com/ecmwf/anemoi-datasets/issues/374)) ([e71a952](https://github.com/ecmwf/anemoi-datasets/commit/e71a95221493a703515d7a14a17f3dead06c8430))
+* Missing method in xarray flavour ([#369](https://github.com/ecmwf/anemoi-datasets/issues/369)) ([d0b9588](https://github.com/ecmwf/anemoi-datasets/commit/d0b9588f0bff626ae751a74cf4cbbc373a2f67f4))
+* Remove trailing slash from zarr paths.  ([#362](https://github.com/ecmwf/anemoi-datasets/issues/362)) ([9a9bfe5](https://github.com/ecmwf/anemoi-datasets/commit/9a9bfe50406d299f44f7cce7e3befcabef6fefaf))
+
+
+### Documentation
+
+* Fix pre-commit ([#367](https://github.com/ecmwf/anemoi-datasets/issues/367)) ([6e6ebe2](https://github.com/ecmwf/anemoi-datasets/commit/6e6ebe2e81fc735a2af422b8c222b0a2f229d8dc))
+* Zarr dataset creation ([#365](https://github.com/ecmwf/anemoi-datasets/issues/365)) ([622c261](https://github.com/ecmwf/anemoi-datasets/commit/622c26156de44ae4eaa3359fef836741b1d81d67))
+
+## [0.5.25](https://github.com/ecmwf/anemoi-datasets/compare/0.5.24...0.5.25) (2025-06-11)
+
+
+### Features
+
+* Integrating non-regular datasets in anemoi for observations. ([#306](https://github.com/ecmwf/anemoi-datasets/issues/306)) ([95a0fe4](https://github.com/ecmwf/anemoi-datasets/commit/95a0fe4bb10dc48469c0be0efad94f4d5e2a9fe8))
+
+
+### Bug Fixes
+
+* Incremental dataset build tasks called regardless of presence of debug flag in CLI code ([#294](https://github.com/ecmwf/anemoi-datasets/issues/294)) ([37afc0d](https://github.com/ecmwf/anemoi-datasets/commit/37afc0d6489f2d6c4b3ce3f9901c40e4cec5c4eb))
+* Regression in accumulations [#354](https://github.com/ecmwf/anemoi-datasets/issues/354) ([#355](https://github.com/ecmwf/anemoi-datasets/issues/355)) ([f9769d7](https://github.com/ecmwf/anemoi-datasets/commit/f9769d7944738ecbedb6b3cc1f78cd26de36a73f))
+* Remove 2 layers of build function ([#348](https://github.com/ecmwf/anemoi-datasets/issues/348)) ([7a904c4](https://github.com/ecmwf/anemoi-datasets/commit/7a904c451772089f120419a9d39bff746e0aeebb))
+
 ## [0.5.24](https://github.com/ecmwf/anemoi-datasets/compare/0.5.23...0.5.24) (2025-05-23)
 
 

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: anemoi-datasets
-Version: 0.5.24
+Version: 0.5.26
 Summary: A package to hold various functions to support training of ML models on ECMWF data.
 Author-email: "European Centre for Medium-Range Weather Forecasts (ECMWF)" <software.support@ecmwf.int>
 License:                                  Apache License
@@ -226,7 +226,7 @@ Classifier: Programming Language :: Python :: Implementation :: PyPy
 Requires-Python: >=3.9
 License-File: LICENSE
 Requires-Dist: anemoi-transform>=0.1.10
-Requires-Dist: anemoi-utils[provenance]>=0.4.21
+Requires-Dist: anemoi-utils[provenance]>=0.4.26
 Requires-Dist: cfunits
 Requires-Dist: numcodecs<0.16
 Requires-Dist: numpy
@@ -262,6 +262,7 @@ Requires-Dist: requests; extra == "remote"
 Provides-Extra: tests
 Requires-Dist: anemoi-datasets[xarray]; extra == "tests"
 Requires-Dist: pytest; extra == "tests"
+Requires-Dist: pytest-xdist; extra == "tests"
 Provides-Extra: xarray
 Requires-Dist: adlfs; extra == "xarray"
 Requires-Dist: gcsfs; extra == "xarray"

anemoi_datasets-0.5.26/docs/adr/adr-1.md

@@ -0,0 +1,63 @@
+# Support irregular observations datasets
+
+## Status
+
+<!--What is the status? -->
+
+Proposed - 30/04/2025
+
+## Context
+
+<!--What is the issue that we are seeing that is motivating this decision or change?-->
+
+The objective of this change is to support observations data which is not regular.
+
+In contrast with the fields data where each date contain the same number of points,
+in the observations data, the number of points can change for every time window.
+
+The Zarr format fits well the fields data, but does not fit the observations data.
+
+To allow storing data with irregular shape, we need to use another format than the zarr used for fields.
+An experimental implementation using xarray-zarr has been developed and is not optimised for ML training.
+
+## Decision
+
+<!--Describe the change that you are proposing.-->
+
+Add a functionality in anemoi-datasets to read observations datasets and provide the data as dictionary/mapper of numpy arrays.
+
+Mimic as much as possible what is done for field datasets :
+
+`ds = open_dataset(....)`
+`ds[i]` -> provides the data for a given time window, related to a given reference date. As a dictionary-like object.
+`ds.dates` -> list of reference date, `ds.dates[i]` is the reference date for the data provided in `ds[i]`
+
+Also expose the latitudes, longitudes in a sensible way (as `ds.latitudes` and `ds.longitudes` now depend on the dates) and name_to_index and statistics and metadata, etc.
+
+These API choices need to be made on an actual training use case.
+
+Step to achieve this:
+- Implement now a prototype format to allow developing ML training code on observation data.
+- Performing extensive benchmarking with various formats (explore parquet, and other).
+- As the final format is not defined yet, ensure a flexible architecture to allow switching (this will help for benchmarking).
+
+
+## Scope of Change
+
+<!--Specify which Anemoi packages/modules will be affected by this decision.-->
+- anemoi-datasets
+Not a breaking change, this only add functionality to read observations datasets.
+
+Must be in line with the change related to multi-datasets.
+
+## Consequences
+
+<!--Discuss the impact of this decision, including benefits, trade-offs, and potential technical debt.-->
+
+## Alternatives Considered [Optional]
+
+<!--List alternative solutions and why they were not chosen.-->
+
+## References [Optional]
+
+<!--Links to relevant discussions, documentation, or external resources.-->

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/docs/datasets/building/sources/anemoi-dataset.rst

@@ -17,3 +17,9 @@ An anemoi-dataset can be a source for a dataset:
 The parameters are the same as those used in the ``open_dataset``
 function, which allows you to subset and combine datasets. See
 :ref:`opening-datasets` for more information.
+
+In particular, this is how local zarr datasets created with anemoi in a
+can be used as a source, contrary to :ref:`xarray-zarr` :
+
+.. literalinclude:: yaml/anemoi-zarr-dataset.yaml
+   :language: yaml

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/docs/datasets/building/sources/xarray-zarr.rst

@@ -1,3 +1,5 @@
+.. _xarray-zarr:
+
 #############
  xarray-zarr
 #############
@@ -17,4 +19,8 @@ it is necessary to use the :ref:`join <building-join>` operation to join
 separate lists containing 2D variables and 3D variables. If all vertical
 levels are desired, then it is acceptable to specify a single source.
 
+Also, an ``xarray-zarr`` source uses the ``url`` keyword, and cannot be
+used for accessing local datasets. For using local zarr datasets as
+sources, use instead :ref:`anemoi-dataset_source`.
+
 See :ref:`create-cf-data` for more information.

anemoi_datasets-0.5.26/docs/datasets/building/sources/yaml/anemoi-zarr-dataset.yaml

@@ -0,0 +1,3 @@
+input:
+  anemoi-dataset:
+    dataset: path/to/dataset.zarr

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/docs/datasets/using/combining.rst

@@ -235,13 +235,15 @@ variables of `dataset1` and return the result.
        source=dataset2,
        what="variables",
        interpolate="nearest",
+       k=1,
    )
 
 Currently ``what`` can only be ``variables`` and can be omitted.
 
 The value for ``interpolate`` can be one of ``none`` (default) or
 ``nearest``. In the case of ``none``, the grids of the two datasets must
-match.
+match. In case of ``interpolate``, an additional parameter ``k`` can be
+set to specify the number of nearest neighbors to use.
 
 This feature was originally designed to be used in conjunction with
 ``cutout``, where `dataset1` is the lam, and `dataset2` is the global

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/docs/howtos/create/02-cf-data.rst

@@ -46,9 +46,18 @@ can contain patterns. See :ref:`file-pattern` for more information.
  Zarr
 ******
 
+For using remote hosted zarr datasets as sources, use
+:ref:`xarray-zarr`.
+
 .. literalinclude:: yaml/zarr1.yaml
    :language: yaml
 
+For using local zarr datasets (such as anemoi-generated datasets), use
+:ref:`anemoi-dataset_source`.
+
+.. literalinclude:: yaml/zarr2.yaml
+   :language: yaml
+
 *********************************************
  Handling data that is not 100% CF-compliant
 *********************************************

anemoi_datasets-0.5.26/docs/howtos/create/yaml/zarr2.yaml

@@ -0,0 +1,8 @@
+dates:
+  start: 2023-01-01T00:00:00
+  end: 2023-01-02T18:00:00
+  frequency: 6h
+
+input:
+  anemoi-dataset:
+    dataset: /path/to/input.zarr

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/docs/howtos/usage/code/cutout-complement1.py

@@ -14,4 +14,5 @@ ds = open_dataset(
     },
     source="global-dataset",
     interpolation="nearest",
+    k=1,
 )

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/docs/howtos/usage/yaml/cutout-complement1.yaml

@@ -9,3 +9,4 @@ dataset:
          adjust: dates
    source: global-dataset
    interpolation: nearest
+   k: 1

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/pyproject.toml

@@ -50,7 +50,7 @@ dynamic = [
 ]
 dependencies = [
   "anemoi-transform>=0.1.10",
-  "anemoi-utils[provenance]>=0.4.21",
+  "anemoi-utils[provenance]>=0.4.26",
   "cfunits",
   "numcodecs<0.16",                   # Until we move to zarr3
   "numpy",
@@ -100,6 +100,7 @@ optional-dependencies.remote = [
 optional-dependencies.tests = [
   "anemoi-datasets[xarray]",
   "pytest",
+  "pytest-xdist",
 ]
 
 optional-dependencies.xarray = [
@@ -131,6 +132,13 @@ version_file = "src/anemoi/datasets/_version.py"
 [tool.isort]
 profile = "black"
 
+[tool.pytest.ini_options]
+testpaths = "tests"
+addopts = [
+  "--numprocesses=auto",
+  "--strict-config",
+]
+
 [tool.mypy]
 strict = false
 exclude = [

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/_version.py

@@ -17,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.5.24'
-__version_tuple__ = version_tuple = (0, 5, 24)
+__version__ = version = '0.5.26'
+__version_tuple__ = version_tuple = (0, 5, 26)

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/commands/finalise-additions.py

@@ -61,7 +61,8 @@ class FinaliseAdditions(Command):
 
         if "debug" in options:
             options.pop("debug")
-            task(step, options)
+
+        task(step, options)
 
         LOG.info(f"Create step '{step}' completed in {seconds_to_human(time.time()-now)}")
 

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/commands/finalise.py

@@ -55,7 +55,8 @@ class Finalise(Command):
 
         if "debug" in options:
             options.pop("debug")
-            task(step, options)
+
+        task(step, options)
 
         LOG.info(f"Create step '{step}' completed in {seconds_to_human(time.time()-now)}")
 

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/commands/grib-index.py

@@ -81,7 +81,7 @@ class GribIndexCmd(Command):
             bool
                 True if the path matches, False otherwise.
             """
-            return fnmatch.fnmatch(path, args.match)
+            return fnmatch.fnmatch(os.path.basename(path), args.match)
 
         from anemoi.datasets.create.sources.grib_index import GribIndex
 

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/commands/init-additions.py

@@ -61,7 +61,8 @@ class InitAdditions(Command):
 
         if "debug" in options:
             options.pop("debug")
-            task(step, options)
+
+        task(step, options)
 
         LOG.info(f"Create step '{step}' completed in {seconds_to_human(time.time()-now)}")
 

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/commands/load-additions.py

@@ -62,7 +62,8 @@ class LoadAdditions(Command):
 
         if "debug" in options:
             options.pop("debug")
-            task(step, options)
+
+        task(step, options)
 
         LOG.info(f"Create step '{step}' completed in {seconds_to_human(time.time()-now)}")
 

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/commands/load.py

@@ -62,7 +62,8 @@ class Load(Command):
 
         if "debug" in options:
             options.pop("debug")
-            task(step, options)
+
+        task(step, options)
 
         LOG.info(f"Create step '{step}' completed in {seconds_to_human(time.time()-now)}")
 

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/create/__init__.py

@@ -44,7 +44,7 @@ from .check import check_data_values
 from .chunks import ChunkFilter
 from .config import build_output
 from .config import loader_config
-from .input import build_input
+from .input import InputBuilder
 from .statistics import Summary
 from .statistics import TmpStatistics
 from .statistics import check_variance
@@ -101,7 +101,9 @@ def json_tidy(o: Any) -> Any:
 
 
 def build_statistics_dates(
-    dates: list[datetime.datetime], start: Optional[datetime.datetime], end: Optional[datetime.datetime]
+    dates: list[datetime.datetime],
+    start: Optional[datetime.datetime],
+    end: Optional[datetime.datetime],
 ) -> tuple[str, str]:
     """Compute the start and end dates for the statistics.
 
@@ -551,36 +553,16 @@ class HasElementForDataMixin:
 
         self.output = build_output(config.output, parent=self)
 
-        self.input = build_input_(main_config=config, output_config=self.output)
-        # LOG.info("%s", self.input)
-
-
-def build_input_(main_config: Any, output_config: Any) -> Any:
-    """Build the input for the dataset.
-
-    Parameters
-    ----------
-    main_config : Any
-        The main configuration.
-    output_config : Any
-        The output configuration.
-
-    Returns
-    -------
-    Any
-        The input builder.
-    """
-    builder = build_input(
-        main_config.input,
-        data_sources=main_config.get("data_sources", {}),
-        order_by=output_config.order_by,
-        flatten_grid=output_config.flatten_grid,
-        remapping=build_remapping(output_config.remapping),
-        use_grib_paramid=main_config.build.use_grib_paramid,
-    )
-    LOG.debug("✅ INPUT_BUILDER")
-    LOG.debug(builder)
-    return builder
+        self.input = InputBuilder(
+            config.input,
+            data_sources=config.get("data_sources", {}),
+            order_by=self.output.order_by,
+            flatten_grid=self.output.flatten_grid,
+            remapping=build_remapping(self.output.remapping),
+            use_grib_paramid=config.build.use_grib_paramid,
+        )
+        LOG.debug("✅ INPUT_BUILDER")
+        LOG.debug(self.input)
 
 
 class Init(Actor, HasRegistryMixin, HasStatisticTempMixin, HasElementForDataMixin):
@@ -1541,7 +1523,16 @@ class Statistics(Actor, HasStatisticTempMixin, HasRegistryMixin):
         if not all(self.registry.get_flags(sync=False)):
             raise Exception(f"❗Zarr {self.path} is not fully built, not writing statistics into dataset.")
 
-        for k in ["mean", "stdev", "minimum", "maximum", "sums", "squares", "count", "has_nans"]:
+        for k in [
+            "mean",
+            "stdev",
+            "minimum",
+            "maximum",
+            "sums",
+            "squares",
+            "count",
+            "has_nans",
+        ]:
             self.dataset.add_dataset(name=k, array=stats[k], dimensions=("variable",))
 
         self.registry.add_to_history("compute_statistics_end")

anemoi_datasets-0.5.24/src/anemoi/datasets/create/filters/transform.py → anemoi_datasets-0.5.26/src/anemoi/datasets/create/filter.py

@@ -1,4 +1,4 @@
-# (C) Copyright 2025  Anemoi contributors.
+# (C) Copyright 2025- Anemoi contributors.
 #
 # This software is licensed under the terms of the Apache Licence Version 2.0
 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
@@ -12,10 +12,8 @@ from typing import Dict
 
 import earthkit.data as ekd
 
-from ..filter import Filter
 
-
-class TransformFilter(Filter):
+class TransformFilter:
     """Calls filters from anemoi.transform.filters
 
     Parameters
@@ -29,7 +27,6 @@ class TransformFilter(Filter):
     """
 
     def __init__(self, context: Any, name: str, config: Dict[str, Any]) -> None:
-
         from anemoi.transform.filters import create_filter
 
         self.name = name

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/create/input/__init__.py

@@ -104,23 +104,3 @@ class InputBuilder:
             Trace string.
         """
         return f"InputBuilder({group_of_dates})"
-
-
-def build_input(config: dict, data_sources: Union[dict, list], **kwargs: Any) -> InputBuilder:
-    """Build an InputBuilder instance.
-
-    Parameters
-    ----------
-    config : dict
-        Configuration dictionary.
-    data_sources : Union[dict, list]
-        Data sources.
-    **kwargs : Any
-        Additional keyword arguments.
-
-    Returns
-    -------
-    InputBuilder
-        An instance of InputBuilder.
-    """
-    return InputBuilder(config, data_sources, **kwargs)

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/create/input/step.py

@@ -8,7 +8,6 @@
 # nor does it submit to any jurisdiction.
 
 import logging
-import warnings
 from copy import deepcopy
 from typing import Any
 from typing import Dict
@@ -165,24 +164,11 @@ def step_factory(config: Dict[str, Any], context: ActionContext, action_path: Li
     if cls is not None:
         return cls(context, action_path, previous_step, *args, **kwargs)
 
-    # Try filters from datasets filter registry
+    # Try filters from transform filter registry
     from anemoi.transform.filters import filter_registry as transform_filter_registry
 
-    from ..filters import create_filter as create_datasets_filter
-    from ..filters import filter_registry as datasets_filter_registry
-
-    if datasets_filter_registry.is_registered(key):
-
-        if transform_filter_registry.is_registered(key):
-            warnings.warn(f"Filter `{key}` is registered in both datasets and transform filter registries")
-
-        filter = create_datasets_filter(None, config)
-        return FunctionStepAction(context, action_path + [key], previous_step, key, filter)
-
-    # Use filters from transform registry
-
     if transform_filter_registry.is_registered(key):
-        from ..filters.transform import TransformFilter
+        from ..filter import TransformFilter
 
         return FunctionStepAction(
             context, action_path + [key], previous_step, key, TransformFilter(context, key, config)

{anemoi_datasets-0.5.24 → anemoi_datasets-0.5.26}/src/anemoi/datasets/create/sources/accumulations.py

@@ -459,12 +459,13 @@ class AccumulationFromStart(Accumulation):
             A tuple representing the MARS date-time step.
         """
         assert user_date is None, user_date
-        assert not frequency, frequency
 
         steps = (step1 + add_step, step2 + add_step)
         if steps[0] == 0:
             steps = (steps[1],)
 
+        assert frequency == 0 or frequency == (step2 - step1), frequency
+
         return (
             base_date.year * 10000 + base_date.month * 100 + base_date.day,
             base_date.hour * 100 + base_date.minute,
@@ -824,6 +825,11 @@ def _compute_accumulations(
     step1, step2 = user_accumulation_period
     assert step1 < step2, user_accumulation_period
 
+    if accumulations_reset_frequency is not None:
+        AccumulationClass = AccumulationFromLastReset
+    else:
+        AccumulationClass = AccumulationFromStart if data_accumulation_period in (0, None) else AccumulationFromLastStep
+
     if data_accumulation_period is None:
         data_accumulation_period = user_accumulation_period[1] - user_accumulation_period[0]
 
@@ -838,11 +844,6 @@ def _compute_accumulations(
 
     base_times = [t // 100 if t > 100 else t for t in base_times]
 
-    if accumulations_reset_frequency is not None:
-        AccumulationClass = AccumulationFromLastReset
-    else:
-        AccumulationClass = AccumulationFromStart if data_accumulation_period in (0, None) else AccumulationFromLastStep
-
     mars_date_time_steps = AccumulationClass.mars_date_time_steps(
         dates=dates,
         step1=step1,

anemoi_datasets-0.5.26/src/anemoi/datasets/create/sources/planetary_computer.py

@@ -0,0 +1,44 @@
+# (C) Copyright 2024 Anemoi contributors.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+#
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+
+
+from . import source_registry
+from .xarray import XarraySourceBase
+
+
+@source_registry.register("planetary_computer")
+class PlanetaryComputerSource(XarraySourceBase):
+    """An Xarray data source for the planetary_computer."""
+
+    emoji = "🪐"
+
+    def __init__(self, context, data_catalog_id, version="v1", *args, **kwargs: dict):
+
+        import planetary_computer
+        import pystac_client
+
+        self.data_catalog_id = data_catalog_id
+        self.flavour = kwargs.pop("flavour", None)
+        self.patch = kwargs.pop("patch", None)
+        self.options = kwargs.pop("options", {})
+
+        catalog = pystac_client.Client.open(
+            f"https://planetarycomputer.microsoft.com/api/stac/{version}/",
+            modifier=planetary_computer.sign_inplace,
+        )
+        collection = catalog.get_collection(self.data_catalog_id)
+
+        asset = collection.assets["zarr-abfs"]
+
+        if "xarray:storage_options" in asset.extra_fields:
+            self.options["storage_options"] = asset.extra_fields["xarray:storage_options"]
+
+        self.options.update(asset.extra_fields["xarray:open_kwargs"])
+
+        super().__init__(context, url=asset.href, *args, **kwargs)