ckanext-marinerg 0.1.0__tar.gz

ckanext_marinerg-0.1.0/PKG-INFO

@@ -0,0 +1,200 @@
+Metadata-Version: 2.4
+Name: ckanext-marinerg
+Version: 0.1.0
+Summary: CKAN extension for the MARINERG-i data catalogue: marine metadata schema, Zenodo prefill, facility/equipment picker, and DCAT-AP export.
+Author-email: Irish Centre for High End Computing <james.grogan@ichec.ie>
+License: AGPL-3.0-or-later
+Project-URL: Repository, https://git.ichec.ie/marinerg-i/ckanext-marinerg
+Project-URL: Homepage, https://git.ichec.ie/marinerg-i/ckanext-marinerg
+Keywords: CKAN,Marine Renewable Energy,Data Catalogue,EOSC,DCAT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Paste
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: ckanext-scheming
+Requires-Dist: PyYAML
+Requires-Dist: requests
+Requires-Dist: pydantic>=2.0
+Requires-Dist: prometheus-client>=0.20
+
+# ckanext-marinerg
+
+CKAN extension for the MARINERG-i data catalogue. Part of the MARINERG-i e-infrastructure — a Distributed Research Infrastructure supporting Marine Renewable Energy (MRE) test facilities across the EU.
+
+Companion to the [data-access-service](https://git.ichec.ie/marinerg-i/data-access-service) repo, which contains the Docker compose stack, Django facility service, and planning documents.
+
+---
+
+## Features
+
+### Marine metadata schema
+
+Custom CKAN scheming fields on top of the standard dataset form:
+
+| Field | Description |
+|---|---|
+| `zenodo_doi` | DOI entry with "Prefill from Zenodo" button |
+| `publication_date` | `YYYY-MM-DD` |
+| `visibility` | `public` / `private` / `restricted` — controlled vocab |
+| `access_scope` | `open` / `org_members` / `on_request` — controlled vocab |
+| `data_mode` | `Raw` / `Provisional` / `Delayed-mode` / `Mixed` — MARINERG-i vocab |
+| `source_kind` | `zenodo` / `datacite` / `manual` / `synthetic` — MARINERG-i vocab |
+| `feature_type` | CF DSG feature type — MARINERG-i SKOS vocab |
+| `processing_level` | EMODnet EPL level (L0–L4) — NERC NVS EPL vocab |
+| `coordinate_reference_system` | EPSG code or `CUSTOM` — NERC NVS L10 vocab |
+| `site` | Named test site / location |
+| `time_coverage_start` / `time_coverage_end` | Data collection period |
+| `facility_ref` | Stable facility PID — ROR URI preferred, portal URL fallback |
+| `facility_name` | Display name of selected facility |
+| `equipment_name` | Name of selected equipment |
+
+### Zenodo prefill
+
+Server-side CKAN action (`marinerg_zenodo_prefill`) triggered by DOI entry. Fetches DataCite/InvenioRDM metadata from Zenodo using an optional API key held in CKAN server config and pre-populates the creation form. Handles both the current InvenioRDM format and the legacy Zenodo API format.
+
+### Facility/equipment picker
+
+Two-dropdown form widget that calls the Django facility service to populate contextual selections. Resolves each facility to a stable PID (ROR URI preferred; portal URL fallback) — internal Django API URLs are never written to dataset records.
+
+### Controlled vocabulary validators
+
+All controlled fields are validated at submission time against `registry/controlled-metadata.yaml`, which is auto-generated from the LinkML schema (`schema/marinerg_dataset.linkml.yaml`). Values not in the registry are rejected. Each code has a corresponding concept URI emitted in RDF export.
+
+Vocabulary sources:
+- `processing_level` → NERC NVS EPL (cached in `registry/nerc_cache/EPL.yaml`)
+- `coordinate_reference_system` → NERC NVS L10 (cached in `registry/nerc_cache/L10.yaml`)
+- `feature_type`, `access_scope`, `data_mode`, `source_kind`, `visibility` → MARINERG-i SKOS vocab
+- `euroscivoc` → EuroSciVoc (EU Science Vocabulary; cached in `registry/euroscivoc_cache.yaml`)
+- `gcmd_keyword` → NASA GCMD Science Keywords (cached in `registry/gcmd_keyword_cache.yaml`)
+- `source_repository` → re3data registry (cached in `registry/source_repository_cache.yaml`)
+- `license_id` → SPDX 3.x (cached in `registry/spdx_cache.yaml`; served via `licenses_group_url`)
+
+The EOSC-vocabulary label caches are refreshed from their authorities with
+`python scripts/fetch_eosc_vocabs.py` (URIs stay single-sourced in the LinkML schema).
+
+### DCAT-AP 3.0 RDF export
+
+`ckanext-dcat` is installed; `MarinergDCATProfile` stacks on `euro_dcat_ap_3`. Custom mappings:
+
+- `visibility` → `dct:accessRights` (EU Publications Office vocab)
+- `facility_ref` / `facility_name` → `dct:publisher` (ROR URI as `foaf:homepage`)
+- `author` → `dct:creator`; `author_email` → `dcat:contactPoint` (restricted datasets only)
+- `zenodo_doi` → `dct:identifier` + `adms:identifier` (the DOI is the dataset's sole identity; no `owl:sameAs`)
+- `publication_date` → `dct:issued`
+- `feature_type` → `dcat:theme` (MARINERG-i concept URI)
+- `euroscivoc` → `dcat:theme` (EuroSciVoc concept URIs); `gcmd_keyword` → `dct:subject` (GCMD concept URIs)
+- `source_repository` → `dct:source` (re3data repository URI)
+- `processing_level` → `dqv:hasQualityAnnotation` with NERC EPL concept URI
+- `coordinate_reference_system` → `dct:conformsTo` (NERC L10 URI or Literal)
+- `site` → `dct:spatial`; `time_coverage_start/end` → `dct:temporal`
+
+Endpoints: `/catalog.rdf`, `/catalog.ttl`, `/catalog.n3`, `/catalog.jsonld` and per-dataset equivalents.
+
+### FAIR Signposting
+
+Dataset landing pages (`/dataset/<name>`) carry a typed HTTP `Link` header (FAIR
+Signposting Profile), added by an `IBlueprint` app-wide `after_request` hook
+(`signposting.py`):
+
+- `cite-as` → the DOI (canonical citation)
+- `describedby` → the `.ttl` / `.jsonld` / `.rdf` DCAT serialisations (with media types)
+- `type` → `dcat:Dataset` + `schema.org/Dataset`
+- `author` → ORCiD URI (when recorded), `license` → licence URL, `item` → each distribution
+
+Machine-navigable without OAI-PMH or SPARQL; assessed by F-UJI and expected by EOSC
+harvesters. Ref: `../data-access-service/docs/eosc-roadmap.md` §2.9.
+
+### CKAN theming
+
+Custom header (MARINERG-i logo), colour scheme (`--marinerg-primary: #003964`), favicon, background imagery.
+
+---
+
+## Planned
+
+- **Keycloak SSO** — `ckanext-oidc-pkce` against ICHEC Keycloak; JIT user creation; group membership from JWT claims
+- **NERC P01 parameters** — discipline-filtered allowlist; instrument/parameter linking
+- **SeaDataNet device categories (L05)** — upgrade `equipment_name` from free text to controlled picker
+- **DataCite export** — same schema and vocab registry; DOI registration
+- **OAI-PMH endpoint** — for EOSC/B2FIND harvesting
+- **MARINERG-i VO** — EOSC Virtual Organisation for consortium-level access policies
+
+---
+
+## Local development
+
+This repo is a sibling of `data-access-service/`. The CKAN dev stack lives there:
+
+```sh
+cd ../data-access-service/infra/ckan
+docker compose -f compose.yml -f compose.local.yml up -d --build
+# CKAN at http://localhost:5001  (port 5000 is owned by Docker Desktop's gvproxy)
+# Login: ckan_admin / ckan_admin
+```
+
+`compose.local.yml` mounts this repo into the container at `/srv/app/src/ckanext-marinerg`. Python and template changes are live immediately. Only dependency changes need a rebuild.
+
+### Facility service
+
+The facility/equipment picker calls the Django facility service. For local dev it should be running on port 8000. The CKAN image has `http://host.docker.internal:8000` baked in at build time — Docker resolves this to the Mac host from inside the container.
+
+### Running tests
+
+```sh
+python -m venv .venv
+.venv/bin/pip install -e ".[test]"
+.venv/bin/python -m pytest tests/ -v
+```
+
+All tests run without a CKAN installation — `conftest.py` stubs `ckan.plugins.toolkit` so pure action/model/validator logic can be tested directly.
+
+### Linting and formatting
+
+```sh
+tox -e format_check   # check (CI)
+tox -e format_apply   # apply
+tox -e style          # flake8
+tox -e type           # mypy
+tox -e registry       # check controlled-metadata.yaml is in sync with the LinkML schema
+```
+
+---
+
+## Vocabulary cache scripts
+
+Run these manually (or on a schedule in CI) to update pinned snapshots. Commit the diff after reviewing.
+
+```sh
+# NERC NVS collections (EPL processing levels, L10 CRS)
+python scripts/fetch_nerc_vocabs.py
+
+# SPDX license list
+python scripts/fetch_spdx_licenses.py
+
+# Regenerate controlled-metadata.yaml from the LinkML schema (also run by `tox -e registry`)
+python scripts/generate_registry.py
+```
+
+---
+
+## CKAN config keys
+
+Baked into `ckan.ini` at image build time via `RUN ckan config-tool` in `data-access-service/infra/ckan/Dockerfile`. Do not rely on runtime env vars — `prerun.py` only writes `ckan.plugins`.
+
+| Key | Default | Purpose |
+|---|---|---|
+| `scheming.dataset_schemas` | — | Path to the marine dataset schema |
+| `ckanext.dcat.rdf.profiles` | `euro_dcat_ap_3 marinerg_dcat` | RDF profile stack |
+| `marinerg.zenodo_api_key` | `""` | Zenodo API key for prefill (optional) |
+| `marinerg.facility_api_url` | `http://host.docker.internal:8000` | Django facility service base URL |
+| `marinerg.facility_api_token` | `""` | Token for authenticated facility API access |
+| `marinerg.portal_base_url` | `http://localhost:5001` | Fallback facility PID base when no ROR ID |
+
+---
+
+## Licence
+
+Copyright ICHEC. GNU AGPL v3 or later.

ckanext_marinerg-0.1.0/README.md

@@ -0,0 +1,178 @@
+# ckanext-marinerg
+
+CKAN extension for the MARINERG-i data catalogue. Part of the MARINERG-i e-infrastructure — a Distributed Research Infrastructure supporting Marine Renewable Energy (MRE) test facilities across the EU.
+
+Companion to the [data-access-service](https://git.ichec.ie/marinerg-i/data-access-service) repo, which contains the Docker compose stack, Django facility service, and planning documents.
+
+---
+
+## Features
+
+### Marine metadata schema
+
+Custom CKAN scheming fields on top of the standard dataset form:
+
+| Field | Description |
+|---|---|
+| `zenodo_doi` | DOI entry with "Prefill from Zenodo" button |
+| `publication_date` | `YYYY-MM-DD` |
+| `visibility` | `public` / `private` / `restricted` — controlled vocab |
+| `access_scope` | `open` / `org_members` / `on_request` — controlled vocab |
+| `data_mode` | `Raw` / `Provisional` / `Delayed-mode` / `Mixed` — MARINERG-i vocab |
+| `source_kind` | `zenodo` / `datacite` / `manual` / `synthetic` — MARINERG-i vocab |
+| `feature_type` | CF DSG feature type — MARINERG-i SKOS vocab |
+| `processing_level` | EMODnet EPL level (L0–L4) — NERC NVS EPL vocab |
+| `coordinate_reference_system` | EPSG code or `CUSTOM` — NERC NVS L10 vocab |
+| `site` | Named test site / location |
+| `time_coverage_start` / `time_coverage_end` | Data collection period |
+| `facility_ref` | Stable facility PID — ROR URI preferred, portal URL fallback |
+| `facility_name` | Display name of selected facility |
+| `equipment_name` | Name of selected equipment |
+
+### Zenodo prefill
+
+Server-side CKAN action (`marinerg_zenodo_prefill`) triggered by DOI entry. Fetches DataCite/InvenioRDM metadata from Zenodo using an optional API key held in CKAN server config and pre-populates the creation form. Handles both the current InvenioRDM format and the legacy Zenodo API format.
+
+### Facility/equipment picker
+
+Two-dropdown form widget that calls the Django facility service to populate contextual selections. Resolves each facility to a stable PID (ROR URI preferred; portal URL fallback) — internal Django API URLs are never written to dataset records.
+
+### Controlled vocabulary validators
+
+All controlled fields are validated at submission time against `registry/controlled-metadata.yaml`, which is auto-generated from the LinkML schema (`schema/marinerg_dataset.linkml.yaml`). Values not in the registry are rejected. Each code has a corresponding concept URI emitted in RDF export.
+
+Vocabulary sources:
+- `processing_level` → NERC NVS EPL (cached in `registry/nerc_cache/EPL.yaml`)
+- `coordinate_reference_system` → NERC NVS L10 (cached in `registry/nerc_cache/L10.yaml`)
+- `feature_type`, `access_scope`, `data_mode`, `source_kind`, `visibility` → MARINERG-i SKOS vocab
+- `euroscivoc` → EuroSciVoc (EU Science Vocabulary; cached in `registry/euroscivoc_cache.yaml`)
+- `gcmd_keyword` → NASA GCMD Science Keywords (cached in `registry/gcmd_keyword_cache.yaml`)
+- `source_repository` → re3data registry (cached in `registry/source_repository_cache.yaml`)
+- `license_id` → SPDX 3.x (cached in `registry/spdx_cache.yaml`; served via `licenses_group_url`)
+
+The EOSC-vocabulary label caches are refreshed from their authorities with
+`python scripts/fetch_eosc_vocabs.py` (URIs stay single-sourced in the LinkML schema).
+
+### DCAT-AP 3.0 RDF export
+
+`ckanext-dcat` is installed; `MarinergDCATProfile` stacks on `euro_dcat_ap_3`. Custom mappings:
+
+- `visibility` → `dct:accessRights` (EU Publications Office vocab)
+- `facility_ref` / `facility_name` → `dct:publisher` (ROR URI as `foaf:homepage`)
+- `author` → `dct:creator`; `author_email` → `dcat:contactPoint` (restricted datasets only)
+- `zenodo_doi` → `dct:identifier` + `adms:identifier` (the DOI is the dataset's sole identity; no `owl:sameAs`)
+- `publication_date` → `dct:issued`
+- `feature_type` → `dcat:theme` (MARINERG-i concept URI)
+- `euroscivoc` → `dcat:theme` (EuroSciVoc concept URIs); `gcmd_keyword` → `dct:subject` (GCMD concept URIs)
+- `source_repository` → `dct:source` (re3data repository URI)
+- `processing_level` → `dqv:hasQualityAnnotation` with NERC EPL concept URI
+- `coordinate_reference_system` → `dct:conformsTo` (NERC L10 URI or Literal)
+- `site` → `dct:spatial`; `time_coverage_start/end` → `dct:temporal`
+
+Endpoints: `/catalog.rdf`, `/catalog.ttl`, `/catalog.n3`, `/catalog.jsonld` and per-dataset equivalents.
+
+### FAIR Signposting
+
+Dataset landing pages (`/dataset/<name>`) carry a typed HTTP `Link` header (FAIR
+Signposting Profile), added by an `IBlueprint` app-wide `after_request` hook
+(`signposting.py`):
+
+- `cite-as` → the DOI (canonical citation)
+- `describedby` → the `.ttl` / `.jsonld` / `.rdf` DCAT serialisations (with media types)
+- `type` → `dcat:Dataset` + `schema.org/Dataset`
+- `author` → ORCiD URI (when recorded), `license` → licence URL, `item` → each distribution
+
+Machine-navigable without OAI-PMH or SPARQL; assessed by F-UJI and expected by EOSC
+harvesters. Ref: `../data-access-service/docs/eosc-roadmap.md` §2.9.
+
+### CKAN theming
+
+Custom header (MARINERG-i logo), colour scheme (`--marinerg-primary: #003964`), favicon, background imagery.
+
+---
+
+## Planned
+
+- **Keycloak SSO** — `ckanext-oidc-pkce` against ICHEC Keycloak; JIT user creation; group membership from JWT claims
+- **NERC P01 parameters** — discipline-filtered allowlist; instrument/parameter linking
+- **SeaDataNet device categories (L05)** — upgrade `equipment_name` from free text to controlled picker
+- **DataCite export** — same schema and vocab registry; DOI registration
+- **OAI-PMH endpoint** — for EOSC/B2FIND harvesting
+- **MARINERG-i VO** — EOSC Virtual Organisation for consortium-level access policies
+
+---
+
+## Local development
+
+This repo is a sibling of `data-access-service/`. The CKAN dev stack lives there:
+
+```sh
+cd ../data-access-service/infra/ckan
+docker compose -f compose.yml -f compose.local.yml up -d --build
+# CKAN at http://localhost:5001  (port 5000 is owned by Docker Desktop's gvproxy)
+# Login: ckan_admin / ckan_admin
+```
+
+`compose.local.yml` mounts this repo into the container at `/srv/app/src/ckanext-marinerg`. Python and template changes are live immediately. Only dependency changes need a rebuild.
+
+### Facility service
+
+The facility/equipment picker calls the Django facility service. For local dev it should be running on port 8000. The CKAN image has `http://host.docker.internal:8000` baked in at build time — Docker resolves this to the Mac host from inside the container.
+
+### Running tests
+
+```sh
+python -m venv .venv
+.venv/bin/pip install -e ".[test]"
+.venv/bin/python -m pytest tests/ -v
+```
+
+All tests run without a CKAN installation — `conftest.py` stubs `ckan.plugins.toolkit` so pure action/model/validator logic can be tested directly.
+
+### Linting and formatting
+
+```sh
+tox -e format_check   # check (CI)
+tox -e format_apply   # apply
+tox -e style          # flake8
+tox -e type           # mypy
+tox -e registry       # check controlled-metadata.yaml is in sync with the LinkML schema
+```
+
+---
+
+## Vocabulary cache scripts
+
+Run these manually (or on a schedule in CI) to update pinned snapshots. Commit the diff after reviewing.
+
+```sh
+# NERC NVS collections (EPL processing levels, L10 CRS)
+python scripts/fetch_nerc_vocabs.py
+
+# SPDX license list
+python scripts/fetch_spdx_licenses.py
+
+# Regenerate controlled-metadata.yaml from the LinkML schema (also run by `tox -e registry`)
+python scripts/generate_registry.py
+```
+
+---
+
+## CKAN config keys
+
+Baked into `ckan.ini` at image build time via `RUN ckan config-tool` in `data-access-service/infra/ckan/Dockerfile`. Do not rely on runtime env vars — `prerun.py` only writes `ckan.plugins`.
+
+| Key | Default | Purpose |
+|---|---|---|
+| `scheming.dataset_schemas` | — | Path to the marine dataset schema |
+| `ckanext.dcat.rdf.profiles` | `euro_dcat_ap_3 marinerg_dcat` | RDF profile stack |
+| `marinerg.zenodo_api_key` | `""` | Zenodo API key for prefill (optional) |
+| `marinerg.facility_api_url` | `http://host.docker.internal:8000` | Django facility service base URL |
+| `marinerg.facility_api_token` | `""` | Token for authenticated facility API access |
+| `marinerg.portal_base_url` | `http://localhost:5001` | Fallback facility PID base when no ROR ID |
+
+---
+
+## Licence
+
+Copyright ICHEC. GNU AGPL v3 or later.

ckanext_marinerg-0.1.0/ckanext/__init__.py

@@ -0,0 +1 @@
+__import__("pkg_resources").declare_namespace(__name__)

ckanext_marinerg-0.1.0/ckanext/marinerg/__init__.py

@@ -0,0 +1 @@
+# ckanext-marinerg: MARINERG-i CKAN extension