metadata-crawler 2509.0.0__py3-none-any.whl

metadata_crawler-2509.0.0.dist-info/METADATA

@@ -0,0 +1,399 @@
+Metadata-Version: 2.4
+Name: metadata-crawler
+Version: 2509.0.0
+Summary: Crawl, extract and push climate metadata for indexing.
+Author-email: "DKRZ, Clint" <freva@dkrz.de>
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+License-File: LICENSE
+Requires-Dist: aiohttp
+Requires-Dist: appdirs
+Requires-Dist: anyio
+Requires-Dist: ciso8601
+Requires-Dist: fsspec
+Requires-Dist: diskcache
+Requires-Dist: s3fs
+Requires-Dist: jinja2
+Requires-Dist: intake
+Requires-Dist: intake-xarray
+Requires-Dist: intake-esm
+Requires-Dist: pandas
+Requires-Dist: python-dateutil
+Requires-Dist: numpy
+Requires-Dist: orjson
+Requires-Dist: pyarrow
+Requires-Dist: h5netcdf
+Requires-Dist: pydantic
+Requires-Dist: pyarrow
+Requires-Dist: rich
+Requires-Dist: rich-argparse
+Requires-Dist: tomli
+Requires-Dist: tomlkit
+Requires-Dist: typing_extensions
+Requires-Dist: zarr
+Requires-Dist: xarray
+Requires-Dist: httpx
+Requires-Dist: uvloop
+Requires-Dist: motor
+Requires-Dist: flit ; extra == "dev"
+Requires-Dist: tox ; extra == "dev"
+Requires-Dist: codespell ; extra == "doc"
+Requires-Dist: blacken-docs ; extra == "doc"
+Requires-Dist: numpydoc ; extra == "doc"
+Requires-Dist: sphinx ; extra == "doc"
+Requires-Dist: sphinxcontrib_github_alt ; extra == "doc"
+Requires-Dist: sphinx-execute-code-python3 ; extra == "doc"
+Requires-Dist: sphinx-copybutton ; extra == "doc"
+Requires-Dist: sphinx-sitemap ; extra == "doc"
+Requires-Dist: sphinx-togglebutton ; extra == "doc"
+Requires-Dist: sphinxext-opengraph[social-cards] ; extra == "doc"
+Requires-Dist: pydata-sphinx-theme ; extra == "doc"
+Requires-Dist: myst-parser ; extra == "doc"
+Requires-Dist: mkdocs ; extra == "mkdoc"
+Requires-Dist: mkdocs-material ; extra == "mkdoc"
+Requires-Dist: mkdocstrings[python] ; extra == "mkdoc"
+Requires-Dist: mkdocs-macros-plugin ; extra == "mkdoc"
+Requires-Dist: mkdocs-minify-plugin ; extra == "mkdoc"
+Requires-Dist: mkdocs-redirects ; extra == "mkdoc"
+Requires-Dist: mike ; extra == "mkdoc"
+Requires-Dist: codespell ; extra == "tests"
+Requires-Dist: pydocstyle ; extra == "tests"
+Requires-Dist: types-appdirs ; extra == "tests"
+Requires-Dist: black ; extra == "tests"
+Requires-Dist: isort ; extra == "tests"
+Requires-Dist: mock ; extra == "tests"
+Requires-Dist: mypy ; extra == "tests"
+Requires-Dist: netcdf4 ; extra == "tests"
+Requires-Dist: pandas ; extra == "tests"
+Requires-Dist: intake-parquet ; extra == "tests"
+Requires-Dist: pytest-asyncio ; extra == "tests"
+Requires-Dist: pytest-cov ; extra == "tests"
+Requires-Dist: pytest-env ; extra == "tests"
+Requires-Dist: requests ; extra == "tests"
+Requires-Dist: pre-commit ; extra == "tests"
+Requires-Dist: toml ; extra == "tests"
+Project-URL: Documentation, https://github.com/freva-org/freva-admin
+Project-URL: Home, https://github.com/freva-org/freva-admin
+Project-URL: Issues, https://github.com/freva-org/freva-admin/issues
+Project-URL: Source, https://github.com/freva-org/freva-admin
+Provides-Extra: dev
+Provides-Extra: doc
+Provides-Extra: mkdoc
+Provides-Extra: tests
+
+# metadata-crawler
+
+[![License](https://img.shields.io/badge/License-BSD-purple.svg)](LICENSE)
+[![PyPI](https://img.shields.io/pypi/pyversions/freva-client.svg)](https://pypi.org/project/metadata-crawler/)
+[![Docs](https://readthedocs.org/projects/metadata-crawler/badge/?version=latest)](https://metadata-crawler.readthedocs.io/en/latest/?badge=latest)
+[![Tests](https://github.com/freva-org/metadata-crawler/actions/workflows/ci_job.yml/badge.svg)](https://github.com/freva-org/metadata-crawler/actions)
+[![Test-Coverage](https://codecov.io/gh/freva-org/metadata-crawler/graph/badge.svg?token=W2YziDnh2N)](https://codecov.io/gh/freva-org/metadata-crawler)
+
+Harvest, normalise, and index climate / earth-system metadata from **POSIX**,
+**S3/MinIO**, and **OpenStack Swift** using configurable **DRS dialects**
+(CMIP6, CMIP5, CORDEX, …). Output to a temporary **catalogue** (DuckDB or
+JSONLines) and then **index** into systems such as **Solr** or **MongoDB**.
+Configuration is **TOML** with inheritance, templating, and computed rules.
+
+> [!TIP]
+>   **TL;DR**
+>   - Define datasets + dialects in ``drs_config.toml``
+>   - ``mdc crawl`` → write a temporary catalogue (``jsonl.gz`` or **DuckDB**)
+>   - ``mdc config`` → inspect a the (merged) crawler config.
+>   - ``mdc walk-intake`` → inspect the content of an intake catalogue.
+>   - ``mdc <backend> index`` → push records from catalogue into your index backend
+>   - ``mdc <backend> delete`` → remove records by facet match
+
+## Features
+
+- **Multi-backend discovery**: POSIX, S3/MinIO, Swift (async REST), Intake
+- **Two-stage pipeline**: *crawl → catalogue* then *catalogue → index*
+- **Schema driven**: strong types (e.g. ``string``, ``datetime[2]``,
+  ``float[4]``, ``string[]``)
+- **DRS dialects**: packaged CMIP6/CMIP5/CORDEX; build your own via inheritance
+- **Path specs & data specs**: parse directory/filename parts and/or read
+  dataset attributes/vars
+- **Special rules**: conditionals, cache lookups and function calls (e.g. CMIP6 realm,
+  time aggregation)
+- **Index backends**: DuckDB (local/S3), MongoDB (Motor), Solr
+- **Sync + Async APIs** and a clean CLI
+- **Docs**: Sphinx with ``pydata_sphinx_theme``
+
+## Install
+
+```console
+
+   pip install metadata-crawler
+   conda install -c conda-forge metadata-crawler
+```
+
+## Quickstart (CLI)
+
+```console
+
+   # 1) Crawl → write catalogue
+   mdc crawl \
+     cat.yaml \
+     --config-file drs_config.toml \
+     --dataset cmip6-fs,obs-fs \
+     --threads 4 --batch-size 100
+
+   # 2) Index from catalogue → Solr (or Mongo/DuckDB)
+   mdc soler index \
+     cat.yaml \
+     --server localhot:8983
+
+   # 3) Delete by facets (supports globs on values)
+   mdc delete \
+     --server localhost:8983 \
+     --facets "file *.nc" --facets "project CMIP6"
+```
+
+> [!NOTE]
+>   The CLI is a **custom framework** inspired by Typer (not Typer itself).
+>   Use ``--help`` on any subcommand to see all options.
+
+## Minimal config (``drs_config.toml``)
+
+```toml
+
+   # === Canonical schema ===
+   [drs_settings.schema.file]
+   key      = "file"
+   type     = "path"
+   required = true
+   indexed  = true
+   unique   = true
+
+   [drs_settings.schema.uri]
+   key      = "uri"
+   type     = "uri"
+   required = true
+   indexed  = true
+
+   [drs_settings.schema.variable]
+   key          = "variable"
+   type         = "string[]"
+   multi_valued = true
+   indexed      = true
+
+   [drs_settings.schema.time]
+   key     = "time"
+   type    = "datetime[2]"     # [start, end]
+   indexed = true
+   default = []
+
+   [drs_settings.schema.bbox]
+   key     = "bbox"
+   type    = "float[4]"        # [W,E,S,N]
+   default = [0, 360, -90, 90]
+
+   # === Dialect: CMIP6 (example) ===
+   [drs_settings.dialect.cmip6]
+   sources   = ["path","data"]         # path | data | storage
+   defaults.grid_label = "gn"
+   specs_dir  = ["mip_era","activity_id","institution_id","source_id","experiment_id","member_id","table_id","variable_id","grid_label","version"]
+   specs_file = ["variable_id","table_id","source_id","experiment_id","member_id","grid_label","time"]
+
+   [drs_settings.dialect.cmip6.special.realm]
+   type   = "method"
+   method = "_get_realm"
+   args   = ["table_id","variable_id","__file_name__"]
+
+   [drs_settings.dialect.cmip6.special.time_aggregation]
+   type   = "method"
+   method = "_get_aggregation"
+   args   = ["table_id","variable_id","__file_name__"]
+
+   # === Dialect: CORDEX (bbox by domain) ===
+   [drs_settings.dialect.cordex]
+   sources   = ["path","data"]
+   specs_dir = ["project","product","domain","institution","driving_model","experiment","ensemble","rcm_name","rcm_version","time_frequency","variable","version"]
+   specs_file= ["variable","domain","driving_model","experiment","ensemble","rcm_name","rcm_version","time_frequency","time"]
+
+   [drs_settings.dialect.cordex.special.bbox]
+   type   = "call"
+   method = "dialect['cordex']['domains'].get('{{domain | upper }}', [0,360,-90,90])"
+
+   [drs_settings.dialect.cordex.domains]
+   EUR-11 = [-44.14, 64.40, 22.20, 72.42]
+   AFR-44 = [-24.64, 60.28, -45.76, 42.24]
+
+   # === Datasets ===
+   [cmip6-fs]
+   root_path  = "/data/model/global/cmip6"
+   drs_format = "cmip6"             # dialect name
+   fs_type    = "posix"
+
+   [cmip6-s3]
+   root_path        = "s3://test-bucket/data/model/global/cmip6"
+   drs_format       = "cmip6"
+   fs_type          = "s3"
+   storage_options.endpoint_url = "http://127.0.0.1:9000"
+   storage_options.aws_access_key_id = "minioadmin"
+   storage_options.aws_secret_access_key = "minioadmin"
+   storage_options.region_name = "us-east-1"
+   storage_options.url_style   = "path"
+   storage_options.use_ssl     = false
+
+   [obs-fs]
+   root_path  = "/arch/observations"
+   drs_format = "custom"
+   # define your specs_dir/specs_file or inherit from another dialect
+```
+
+## Concepts
+
+### Schema (facet definitions)
+
+Each canonical facet describes:
+
+- ``key``: where to read value (``"project"``, ``"variable"``,)
+- ``type``: ``string``, ``integer``, ``float``, ``datetime``, with arrays like
+  ``float[4]``, ``string[]``, ``datetime[2]``, or special types like ``file``,
+  ``uri``, ``fs_type``, ``dataset``, ``fmt``
+- ``required``, ``default``, ``indexed``, ``unique``, ``multi_valued``
+
+### Dialects
+
+A dialect tells the crawler how to **interpret paths** and **read data**:
+
+- ``sources``: which sources to consult (``path``, ``data``, ``storage``) in priority
+- ``specs_dir`` / ``specs_file``: ordered facet names encoded in directory and file names
+- ``data_specs``: pull values from dataset content (attrs/variables); supports
+  ``__variable__`` and templated specs
+- ``special``: computed fields (``conditional`` | ``method`` | ``function``)
+- Optional lookups (e.g., CORDEX ``domains`` for bbox)
+
+### Path specs vs data specs
+
+- **Path specs** parse segments from the path, e.g.:
+  ``/project/product/institute/model/experiment/.../variable_time.nc``
+- **Data specs** read from the dataset itself (e.g., xarray/global attribute, variable
+  attributes, per-var stats). Example: gather all variables ``__variable__``, then
+  their units with a templated selector.
+
+### Inheritance
+
+Create new dialects/datasets by inheriting:
+
+```toml
+
+   [drs_settings.dialect.reana]
+   inherits_from = "cmip5"
+   sources       = ["path","data"]
+   [drs_settings.dialect.reana.data_specs.read_kws]
+   engine = "h5netcdf"
+```
+
+## Python API
+
+### Async
+
+```python
+
+   import asyncio
+   from metadata_crawler.run import async_add, async_index, async_delete
+
+   async def main():
+       # crawl → catalogue
+       await async_add(
+           "cat.yaml",
+           config_file="drs_config.toml",
+           dataset_names=["cmip6-fs"],
+           threads=4,
+           batch_size=100,
+       )
+       # index → backend
+       await async_index(
+           "solr",
+           "cat.yaml",
+           config_file="drs_config.toml",
+           server="localhost:8983",
+       )
+       # delete by facets
+       await async_delete(
+           config_path="drs_config.toml",
+           index_store="solr",
+           facets=[("file", "*.nc")],
+       )
+
+   asyncio.run(main())
+```
+
+
+### Sync (simple wrapper)
+
+```python
+
+   import asyncio
+   from metadata_crawler import add
+
+   add(
+       store="cat.yaml",
+       config_file="drs_config.toml",
+       dataset_names=["cmip6-fs"],
+   )
+```
+
+## Index backends
+
+- **MongoDB** (Motor): upserts by unique facet (e.g., ``file``), bulk deletes (glob → regex)
+- **Solr**: fields align with managed schema; supports multi-valued facets
+
+
+## Contributing
+
+Development install:
+
+```console
+
+   git clone https://github.com/freva-org/metadata-crawler.git
+   cd metadata-crawler
+   pip install -e .
+
+```
+
+PRs and issues welcome. Please add tests and keep examples minimal & reproducible
+(use the MinIO compose stack). Run:
+
+
+```console
+   python -m pip install tox
+   tox -e test lint types
+```
+
+### Benchmarks
+For benchmarking you can create a directory tree with roughly 1.5 M files by
+calling the ``create-cordex.sh`` script in the ``dev-env`` folder:
+
+```console
+./dev-env/create-cordex.sh
+python dev-env/benchmark.py --max-files 20000
+```
+
+
+See ``code-of-conduct.rst`` and ``whatsnew.rst`` for guidelines and changelog.
+
+Use MinIO or LocalStack via ``docker-compose`` and seed a bucket (e.g., ``test-bucket``).
+Then point a dataset’s ``fs_type = "s3"`` and set ``storage_options``.
+
+### Documentation
+
+Built with Sphinx + ``pydata_sphinx_theme``. Build locally:
+
+```console
+   tox -e docs
+```
+

metadata_crawler-2509.0.0.dist-info/RECORD

@@ -0,0 +1,34 @@
+metadata_crawler/__init__.py,sha256=7gEpJjS9FpR6MHRY_Ztk8ORJ8JQ7WZUTV2TfLkaYgqs,6741
+metadata_crawler/__main__.py,sha256=4m56VOh7bb5xmZqb09fFbquke8g6KZfMbb3CUdBA60M,163
+metadata_crawler/_version.py,sha256=Z6_4SgU9Dpc127xJlyvGKjeWd_Q1ONlOHQO123XGv30,25
+metadata_crawler/cli.py,sha256=meY5ZfR5VEW5ZorOPWO_b4MyIIQy0wTTPs9OkJ1WnfA,17180
+metadata_crawler/data_collector.py,sha256=9CVr4arKJspyLNLuF2MfkmY_r8x74Mw8hAaDSMouQUA,8372
+metadata_crawler/logger.py,sha256=5Lc0KdzH2HdWkidW-MASW8Pfy7vTMnzPv1-e2V3Any0,4407
+metadata_crawler/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+metadata_crawler/run.py,sha256=w1kV4D63dS3mdgDTQj2ngzeSCjZPphWg1HwIJeJ6ATE,11345
+metadata_crawler/utils.py,sha256=QNr_9jZkuuQOrkuO46PrFhUfwLmfCJCq9gWUwwARfyM,11580
+metadata_crawler/api/__init__.py,sha256=UUF0_FKgfqgcXYmknxB0Wt1jaLNaf-w_q0tWVJhgV0M,28
+metadata_crawler/api/cli.py,sha256=pgj3iB_Irt74VbG3ZKStLRHKYY_I4bZpbOW1famKDnQ,1498
+metadata_crawler/api/config.py,sha256=j__JDKYTOR8kYC--HaHlYXfz38rzEhtUvHdO5Bh_j2E,28250
+metadata_crawler/api/drs_config.toml,sha256=90lQaSC2VdJ8OUoc6j27kg6d2OnfxR5a_KZH3W-FZV4,10603
+metadata_crawler/api/index.py,sha256=8g5HdSxluKtCwU45P0w_7LDIaSf200JbB-ekGJiI18c,4130
+metadata_crawler/api/metadata_stores.py,sha256=oWewL6XRmNZ6i5WxYI8Lm2jfpwLqBCGP2p4j3wLLNpQ,23735
+metadata_crawler/api/storage_backend.py,sha256=jdZZ_3SZcP3gJgw_NmPPdpDEx4D7qfLJDABfupTH9p0,7803
+metadata_crawler/api/mixin/__init__.py,sha256=4Y0T1eM4vLlgFazuC1q2briqx67LyfeCpY_pCICTnjk,197
+metadata_crawler/api/mixin/lookup_mixin.py,sha256=WxJ-ZNs8DcIXS9ThSoIZiepD07jfmLlzyTp65-Z1fLc,3558
+metadata_crawler/api/mixin/lookup_tables.py,sha256=za63xfZB0EvAm66uTTYo52zC0z7Y6VL8DUrP6CJ-DnQ,308683
+metadata_crawler/api/mixin/path_mixin.py,sha256=WKpesEjlwVSJ-VdoYYLEY5oBSAQTsvuv1B38ragAVIM,1247
+metadata_crawler/api/mixin/template_mixin.py,sha256=_qDp5n_CPnSYPMBsTia44b1ybBqrJEi-M1NaRkQ0z3U,5106
+metadata_crawler/backends/__init__.py,sha256=yrk1L00ubQlMj3yXI73PPbhAahDKp792PJB-xcXUJIM,35
+metadata_crawler/backends/intake.py,sha256=TkvzBU8Rk49L0Y8e7Exz2nE3iLSWrBAwZnpEJtdlNR8,6595
+metadata_crawler/backends/posix.py,sha256=6sjAoCQHiOOjp_Hvwxn247wHBnoAJYUGequqphyZWaA,3409
+metadata_crawler/backends/s3.py,sha256=DPz_bOyOlUveCwkSLVatwU_mcxUbFvygU_Id1AZVIMA,4455
+metadata_crawler/backends/swift.py,sha256=az3ctF_npadjzAybX65CQbDLGoxRnk0ZR7vByo6lQOM,10954
+metadata_crawler/ingester/__init__.py,sha256=Y-c9VkQWMHDLb9WagwITCaEODlYa4p8xW-BkzzSRZXw,55
+metadata_crawler/ingester/mongo.py,sha256=lpWIZ8mo6S8oY887uz2l6Y9pir0sUVEkfgOdDxrjIMM,6142
+metadata_crawler/ingester/solr.py,sha256=EoKS3kFeDTLf9zP22s2DhQGP81T6rTXVWDNT2wWKFkk,5242
+metadata_crawler-2509.0.0.dist-info/entry_points.txt,sha256=4LzS7pbqwUPTD6C-iW42vuhXdtsOJmKXqFZpdpaKwF8,428
+metadata_crawler-2509.0.0.dist-info/licenses/LICENSE,sha256=GAUualebvSlegSVqb86FUqHrHM8WyM145__Nm2r_dfA,1496
+metadata_crawler-2509.0.0.dist-info/WHEEL,sha256=G2gURzTEtmeR8nrdXUJfNiB3VYVxigPQ-bEQujpNiNs,82
+metadata_crawler-2509.0.0.dist-info/METADATA,sha256=Dk0trqXYleepz1L8HXwKF-vAdSQww1zBm4Q014G4aOU,12938
+metadata_crawler-2509.0.0.dist-info/RECORD,,

metadata_crawler-2509.0.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: flit 3.12.0
+Root-Is-Purelib: true
+Tag: py3-none-any

metadata_crawler-2509.0.0.dist-info/entry_points.txt

@@ -0,0 +1,14 @@
+[console_scripts]
+mdc=metadata_crawler.cli:cli
+metadata-crawler=metadata_crawler.cli:cli
+
+[metadata_crawler.ingester]
+mongo=metadata_crawler.ingester.mongo:MongoIndex
+solr=metadata_crawler.ingester.solr:SolrIndex
+
+[metadata_crawler.storage]
+intake=metadata_crawler.backends.intake:IntakePath
+posix=metadata_crawler.backends.posix:PosixPath
+s3=metadata_crawler.backends.s3:S3Path
+swift=metadata_crawler.backends.swift:SwiftPath
+

metadata_crawler-2509.0.0.dist-info/licenses/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2025, freva-org
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.