marinerg-data 0.1.0__tar.gz

marinerg_data-0.1.0/PKG-INFO

@@ -0,0 +1,87 @@
+Metadata-Version: 2.4
+Name: marinerg-data
+Version: 0.1.0
+Summary: Facility-side MARINERG-i data client: inspect, convert, validate, publish, and fetch marine test-facility datasets.
+Author-email: Irish Centre for High End Computing <james.grogan@ichec.ie>
+License: AGPL-3.0-or-later
+Project-URL: Repository, https://git.ichec.ie/marinerg-i/marinerg-data
+Project-URL: Homepage, https://git.ichec.ie/marinerg-i/marinerg-data
+Keywords: Marine Renewable Energy,Data,EOSC,Zenodo,CKAN,kerchunk
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.14
+Description-Content-Type: text/markdown
+Requires-Dist: jsonschema>=4
+Requires-Dist: requests
+Requires-Dist: PyYAML
+Requires-Dist: pydantic>=2
+Provides-Extra: mock
+Requires-Dist: numpy; extra == "mock"
+Provides-Extra: kerchunk
+Requires-Dist: kerchunk>=0.2.6; extra == "kerchunk"
+Requires-Dist: h5py>=3.10; extra == "kerchunk"
+Requires-Dist: fsspec>=2024.2; extra == "kerchunk"
+Requires-Dist: zarr>=3; extra == "kerchunk"
+Requires-Dist: xarray>=2024.1; extra == "kerchunk"
+Provides-Extra: types
+Requires-Dist: types-requests; extra == "types"
+Requires-Dist: types-PyYAML; extra == "types"
+
+# marinerg-data
+
+Facility-side data client for the MARINERG-i e-infrastructure: **inspect →
+convert → validate → publish**, plus VRE consumption (search, resolve, fetch,
+lazy streaming). Django-free and pip-installable, so it runs both at a test
+facility and inside a Blue-Cloud / D4Science VRE Jupyter image.
+
+The invariant that shapes everything: **bulk data never transits the
+e-infrastructure.** Facilities publish to their own Zenodo; this tool registers
+the record in CKAN; the server only re-validates metadata.
+
+## Install
+
+```sh
+pip install marinerg-data                 # core client (inspect/validate/publish)
+pip install "marinerg-data[kerchunk]"     # + kerchunk reference sidecars
+pip install "marinerg-data[mock]"         # + synthetic scenario generator
+pip install marinerg-wave-tank            # optional wave-tank converter family
+```
+
+## Commands
+
+| Command | Purpose |
+|---|---|
+| `inspect <dir>` | Sniff capture formats, report converter families; `--draft-manifest` writes a skeleton |
+| `convert <run-dir>` | Run a converter family (`--family wave-tank`, else auto-detect); writes open-format outputs + `<run>.manifest.json`, then validates. With `[kerchunk]`, also emits a `<file>.nc.kerchunk.json` reference sidecar per NetCDF |
+| `validate <manifest-or-dir>` | JSON Schema (run-manifest v0.1) + referential integrity + file existence; `--verify-checksums`, `--no-files` |
+| `publish <bundle-dir>` | Facility Zenodo record + CKAN registration; `--sandbox`, `--new-version`, `--dry-run` |
+| `mock --scenario <name>` | Synthetic NetCDF + manifest for fixtures/seeding/demos (`--scenario list`) |
+
+Environment: `ZENODO_TOKEN` (a sandbox token with `--sandbox`), `CKAN_URL`,
+`CKAN_TOKEN`.
+
+## Cloud-optimised access (VRE)
+
+With the `[kerchunk]` extra, `convert` emits a kerchunk reference sidecar per
+NetCDF. The sidecar lets `xarray` lazily stream individual chunks straight from
+Zenodo over HTTP range requests — open a 10 GB campaign, plot five minutes of one
+gauge, transfer megabytes not gigabytes. The reference URL is templated (`{{u}}`)
+so the sidecar is portable; it is retargeted to the Zenodo file URL at publish.
+
+## Develop
+
+```sh
+uv sync --group dev
+uv run pytest          # Django-free, no network (HTTP is faked)
+uv run ruff check src tests
+```
+
+Python 3.14, uv-native toolchain. See `CLAUDE.md` for architecture pointers and
+the token-free release flow.
+
+## Licence
+
+Copyright ICHEC. GNU AGPL v3 or later. Exemptions available for MARINERG-i
+project partners.

marinerg_data-0.1.0/README.md

@@ -0,0 +1,56 @@
+# marinerg-data
+
+Facility-side data client for the MARINERG-i e-infrastructure: **inspect →
+convert → validate → publish**, plus VRE consumption (search, resolve, fetch,
+lazy streaming). Django-free and pip-installable, so it runs both at a test
+facility and inside a Blue-Cloud / D4Science VRE Jupyter image.
+
+The invariant that shapes everything: **bulk data never transits the
+e-infrastructure.** Facilities publish to their own Zenodo; this tool registers
+the record in CKAN; the server only re-validates metadata.
+
+## Install
+
+```sh
+pip install marinerg-data                 # core client (inspect/validate/publish)
+pip install "marinerg-data[kerchunk]"     # + kerchunk reference sidecars
+pip install "marinerg-data[mock]"         # + synthetic scenario generator
+pip install marinerg-wave-tank            # optional wave-tank converter family
+```
+
+## Commands
+
+| Command | Purpose |
+|---|---|
+| `inspect <dir>` | Sniff capture formats, report converter families; `--draft-manifest` writes a skeleton |
+| `convert <run-dir>` | Run a converter family (`--family wave-tank`, else auto-detect); writes open-format outputs + `<run>.manifest.json`, then validates. With `[kerchunk]`, also emits a `<file>.nc.kerchunk.json` reference sidecar per NetCDF |
+| `validate <manifest-or-dir>` | JSON Schema (run-manifest v0.1) + referential integrity + file existence; `--verify-checksums`, `--no-files` |
+| `publish <bundle-dir>` | Facility Zenodo record + CKAN registration; `--sandbox`, `--new-version`, `--dry-run` |
+| `mock --scenario <name>` | Synthetic NetCDF + manifest for fixtures/seeding/demos (`--scenario list`) |
+
+Environment: `ZENODO_TOKEN` (a sandbox token with `--sandbox`), `CKAN_URL`,
+`CKAN_TOKEN`.
+
+## Cloud-optimised access (VRE)
+
+With the `[kerchunk]` extra, `convert` emits a kerchunk reference sidecar per
+NetCDF. The sidecar lets `xarray` lazily stream individual chunks straight from
+Zenodo over HTTP range requests — open a 10 GB campaign, plot five minutes of one
+gauge, transfer megabytes not gigabytes. The reference URL is templated (`{{u}}`)
+so the sidecar is portable; it is retargeted to the Zenodo file URL at publish.
+
+## Develop
+
+```sh
+uv sync --group dev
+uv run pytest          # Django-free, no network (HTTP is faked)
+uv run ruff check src tests
+```
+
+Python 3.14, uv-native toolchain. See `CLAUDE.md` for architecture pointers and
+the token-free release flow.
+
+## Licence
+
+Copyright ICHEC. GNU AGPL v3 or later. Exemptions available for MARINERG-i
+project partners.

marinerg_data-0.1.0/pyproject.toml

@@ -0,0 +1,111 @@
+[project]
+name = "marinerg-data"
+dynamic = ["version"]
+authors = [
+  { name = "Irish Centre for High End Computing", email = "james.grogan@ichec.ie" },
+]
+description = "Facility-side MARINERG-i data client: inspect, convert, validate, publish, and fetch marine test-facility datasets."
+readme = "README.md"
+requires-python = ">=3.14"
+license = { text = "AGPL-3.0-or-later" }
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.14",
+    "Operating System :: OS Independent",
+]
+keywords = ["Marine Renewable Energy", "Data", "EOSC", "Zenodo", "CKAN", "kerchunk"]
+
+# Django-free client. Core deps are what the CLI itself needs; heavier,
+# workflow-specific stacks are optional extras.
+dependencies = [
+    "jsonschema>=4",
+    "requests",
+    "PyYAML",
+    "pydantic>=2",
+]
+
+[project.urls]
+Repository = "https://git.ichec.ie/marinerg-i/marinerg-data"
+Homepage = "https://git.ichec.ie/marinerg-i/marinerg-data"
+
+[project.optional-dependencies]
+# `marinerg-data mock` generates synthetic scenario datasets.
+mock = ["numpy"]
+
+# Cloud-optimised access: emit kerchunk reference sidecars per NetCDF for lazy
+# HTTP-range streaming from Zenodo (see the converter / VRE docs).
+kerchunk = [
+    "kerchunk>=0.2.6",
+    "h5py>=3.10",
+    "fsspec>=2024.2",
+    "zarr>=3",
+    "xarray>=2024.1",
+]
+
+types = ["types-requests", "types-PyYAML"]
+
+[project.scripts]
+marinerg-data = "marinerg_data.cli:main"
+
+[build-system]
+requires = ["setuptools>=64", "setuptools-scm>=8", "wheel"]
+build-backend = "setuptools.build_meta"
+
+# Version derived from the git tag (setuptools-scm); fallback keeps non-tag
+# builds importable. Token-free release: compute-version stamps
+# SETUPTOOLS_SCM_PRETEND_VERSION, create-release makes the tag (planning#132).
+[tool.setuptools_scm]
+fallback_version = "0.0.0"
+
+# Development toolchain (uv-native, PEP 735). CI installs it with
+# `uv sync --group dev`. Toolchain set canonical (see ichec-platform-core).
+[dependency-groups]
+dev = [
+    "ruff",
+    "mypy",
+    "pytest",
+    "pytest-cov",
+    "jsonschema>=4",
+    "requests",
+    "types-requests",
+    "PyYAML",
+    "types-PyYAML",
+    "numpy",
+    # kerchunk sidecar tests (round-trip lazy open); mirror the [kerchunk] extra.
+    "kerchunk>=0.2.6",
+    "h5py>=3.10",
+    "fsspec>=2024.2",
+    "zarr>=3",
+    "xarray>=2024.1",
+    "h5netcdf>=1.3",
+]
+
+[tool.mypy]
+ignore_missing_imports = true
+# strict baseline (target — enable as the codebase reaches it):
+# warn_return_any = true
+# disallow_untyped_defs = true
+
+[tool.setuptools.package-data]
+"marinerg_data.data" = ["run_manifest.schema.json", "capture_formats.yaml"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+# --- ruff: format + lint (supersedes black, isort, flake8, pylint) ---
+[tool.ruff]
+line-length = 88
+target-version = "py314"
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "N", "UP", "B"]
+
+[tool.ruff.lint.isort]
+known-first-party = ["marinerg_data"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+log_cli_level = "warning"
+addopts = "--cov=marinerg_data --cov-report term --cov-report xml:coverage.xml --cov-report html"

marinerg_data-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

marinerg_data-0.1.0/src/marinerg_data/__init__.py

@@ -0,0 +1,14 @@
+"""marinerg-data: facility-side convert / validate / publish tool.
+
+Design: docs/domain-metadata-roadmap.md "Submission & ingest architecture" in
+the data-access-service repo. Validation runs locally at the facility before any
+upload; publication is thin orchestration over the facility's own Zenodo account
+plus CKAN registration. Bulk data never transits the e-infrastructure.
+"""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("marinerg-data")
+except PackageNotFoundError:  # not installed (e.g. running from a source tree)
+    __version__ = "0.0.0"

marinerg_data-0.1.0/src/marinerg_data/bundle.py

@@ -0,0 +1,159 @@
+"""A publishable bundle: one campaign directory.
+
+Layout expected by ``marinerg-data publish <dir>``:
+
+    <dir>/dataset.yaml           campaign-level metadata (this module's model)
+    <dir>/*.manifest.json        one run manifest per run (schema v0.1)
+    <dir>/...                    data files referenced by the manifests
+
+The CKAN dataset represents the campaign; runs are structured resources
+within it (domain-metadata-roadmap.md design principle 6).
+
+Publish state (Zenodo record id + DOIs) is written to
+``<dir>/.marinerg-publish.json`` so ``publish --new-version`` knows which
+record to version. The state file is local bookkeeping, not metadata — it
+never gets uploaded.
+"""
+
+from __future__ import annotations
+
+import json
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any
+
+import yaml
+from pydantic import BaseModel, ConfigDict, Field
+
+from .manifest import (
+    ManifestError,
+    find_dataset_metadata,
+    find_manifests,
+    load_manifest,
+)
+
+STATE_FILE_NAME = ".marinerg-publish.json"
+
+
+class Creator(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    name: str
+    affiliation: str | None = None
+    orcid: str | None = None
+
+
+class DatasetMetadata(BaseModel):
+    """Campaign-level metadata; the manifest carries per-run detail."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    name: str = Field(description="CKAN dataset slug, e.g. 'gwk-floatx-2026'")
+    title: str
+    description: str
+    creators: list[Creator] = Field(min_length=1)
+    license: str = "CC-BY-4.0"
+    keywords: list[str] = Field(default_factory=list)
+    visibility: str = "public"
+    community: str = "marinerg-i"
+
+    facility_ref: str | None = None
+    facility_name: str | None = None
+    equipment_name: str | None = None
+    owner_org: str | None = None
+
+    site: str | None = None
+    feature_type: str | None = None
+    processing_level: str | None = None
+    coordinate_reference_system: str | None = None
+    data_mode: str | None = None
+
+    # Test Context (domain-metadata-roadmap short term). Passed to CKAN as
+    # extras until the scheming field group lands, then as top-level fields.
+    test_type: str | None = None
+    device_type: str | None = None
+    device_scale: str | None = None
+    campaign_name: str | None = None
+    test_report_ref: str | None = None
+
+
+class PublishState(BaseModel):
+    model_config = ConfigDict(extra="allow")
+
+    record_id: str
+    doi: str
+    concept_doi: str | None = None
+    ckan_id: str | None = None
+    published_at: str | None = None
+    sandbox: bool = False
+
+
+class BundleError(Exception):
+    pass
+
+
+class Bundle:
+    def __init__(
+        self,
+        directory: Path,
+        metadata: DatasetMetadata,
+        manifests: dict[Path, dict[str, Any]],
+    ):
+        self.directory = directory
+        self.metadata = metadata
+        self.manifests = manifests
+
+    @classmethod
+    def load(cls, directory: Path) -> Bundle:
+        if not directory.is_dir():
+            raise BundleError(f"Not a directory: {directory}")
+        metadata_path = find_dataset_metadata(directory)
+        if metadata_path is None:
+            raise BundleError(
+                f"No dataset.yaml in {directory} — the bundle needs "
+                "campaign-level metadata (title, creators, license...)"
+            )
+        raw = yaml.safe_load(metadata_path.read_text(encoding="utf-8")) or {}
+        metadata = DatasetMetadata(**raw)
+        manifests = {}
+        try:
+            for manifest_path in find_manifests(directory):
+                manifests[manifest_path] = load_manifest(manifest_path)
+        except ManifestError as exc:
+            raise BundleError(str(exc)) from exc
+        if not manifests:
+            raise BundleError(f"No *.manifest.json files in {directory}")
+        return cls(directory, metadata, manifests)
+
+    def upload_files(self) -> list[Path]:
+        """Files that belong in the Zenodo record: every manifest, plus all
+        zenodo-hosted resources, media, documents, and reference sidecars they
+        reference."""
+        files: list[Path] = sorted(self.manifests)
+        seen = set(files)
+        for manifest_path, manifest in sorted(self.manifests.items()):
+            base = manifest_path.parent
+            for section in ("resources", "media", "documents", "references"):
+                for entry in manifest.get(section, []):
+                    if entry.get("hosted", "zenodo") != "zenodo":
+                        continue
+                    local = base / entry["path"]
+                    if local not in seen:
+                        files.append(local)
+                        seen.add(local)
+        return files
+
+    @property
+    def state_path(self) -> Path:
+        return self.directory / STATE_FILE_NAME
+
+    def read_state(self) -> PublishState | None:
+        if not self.state_path.is_file():
+            return None
+        return PublishState(**json.loads(self.state_path.read_text(encoding="utf-8")))
+
+    def write_state(self, state: PublishState) -> None:
+        state.published_at = datetime.now(UTC).isoformat()
+        self.state_path.write_text(
+            json.dumps(state.model_dump(), indent=2) + "\n", encoding="utf-8"
+        )