envlib 0.1.0__tar.gz → 0.1.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {envlib-0.1.0 → envlib-0.1.1}/.gitignore +5 -1
- envlib-0.1.1/PKG-INFO +123 -0
- envlib-0.1.1/README.md +105 -0
- {envlib-0.1.0 → envlib-0.1.1}/envlib/__init__.py +1 -1
- {envlib-0.1.0 → envlib-0.1.1}/envlib/catalogue.py +56 -69
- {envlib-0.1.0 → envlib-0.1.1}/envlib/metadata.py +2 -1
- {envlib-0.1.0 → envlib-0.1.1}/envlib/vocabularies/__init__.py +1 -1
- {envlib-0.1.0 → envlib-0.1.1}/pyproject.toml +6 -5
- envlib-0.1.0/PKG-INFO +0 -75
- envlib-0.1.0/README.md +0 -57
- {envlib-0.1.0 → envlib-0.1.1}/LICENSE +0 -0
- {envlib-0.1.0 → envlib-0.1.1}/envlib/vocabularies/aggregation_statistic.json +0 -0
- {envlib-0.1.0 → envlib-0.1.1}/envlib/vocabularies/feature.json +0 -0
- {envlib-0.1.0 → envlib-0.1.1}/envlib/vocabularies/frequency_interval.json +0 -0
- {envlib-0.1.0 → envlib-0.1.1}/envlib/vocabularies/license.json +0 -0
- {envlib-0.1.0 → envlib-0.1.1}/envlib/vocabularies/method.json +0 -0
- {envlib-0.1.0 → envlib-0.1.1}/envlib/vocabularies/processing_level.json +0 -0
- {envlib-0.1.0 → envlib-0.1.1}/envlib/vocabularies/standard_name.json +0 -0
- {envlib-0.1.0 → envlib-0.1.1}/envlib/vocabularies/variable.json +0 -0
envlib-0.1.1/PKG-INFO
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: envlib
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: The environmental library
|
|
5
|
+
Project-URL: Documentation, https://envlib.github.io/envlib/
|
|
6
|
+
Project-URL: Source, https://github.com/envlib/envlib
|
|
7
|
+
Author-email: mullenkamp <mullenkamp1@gmail.com>
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
10
|
+
Requires-Python: >=3.10
|
|
11
|
+
Requires-Dist: booklet>=0.12.6
|
|
12
|
+
Requires-Dist: cfdb>=0.9.1
|
|
13
|
+
Requires-Dist: ebooklet>=0.10.0
|
|
14
|
+
Requires-Dist: pyproj
|
|
15
|
+
Requires-Dist: shapely>=2.0
|
|
16
|
+
Requires-Dist: urllib3
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
|
|
19
|
+
# envlib
|
|
20
|
+
|
|
21
|
+
<p align="center">
|
|
22
|
+
<em>A distributed catalogue for environmental data</em>
|
|
23
|
+
</p>
|
|
24
|
+
|
|
25
|
+
[](https://github.com/envlib/envlib/actions)
|
|
26
|
+
[](https://codecov.io/gh/envlib/envlib)
|
|
27
|
+
[](https://badge.fury.io/py/envlib)
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
**Documentation**: <a href="https://envlib.github.io/envlib/" target="_blank">https://envlib.github.io/envlib/</a>
|
|
32
|
+
|
|
33
|
+
**Source Code**: <a href="https://github.com/envlib/envlib" target="_blank">https://github.com/envlib/envlib</a>
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Overview
|
|
38
|
+
|
|
39
|
+
envlib is a distributed database and catalogue for environmental datasets — gridded model output and station time series alike. Datasets are stored as [cfdb](https://github.com/mullenkamp/cfdb) files on S3-compatible object storage, hosted by whoever owns the data; envlib provides the shared layer on top: standardized metadata, controlled vocabularies, deterministic identifiers, and a catalogue you can query to discover and open any registered dataset.
|
|
40
|
+
|
|
41
|
+
## Key features
|
|
42
|
+
|
|
43
|
+
- **Distributed by design** — the catalogue is an index, not a data silo: each dataset stays on its owner's storage, and one catalogue can span many owners.
|
|
44
|
+
- **Standardized metadata** — eleven identity fields (drawing on [ODM2](http://vocabulary.odm2.org/), [CF conventions](https://cfconventions.org/), and lessons from tethys) describe what every dataset *is*, validated against controlled vocabularies at creation time.
|
|
45
|
+
- **Deterministic, permanent identifiers** — a dataset's identity metadata hashes to stable ids (`dataset_id`, `dataset_version_id`, `station_id`), so the same data gets the same id everywhere, forever.
|
|
46
|
+
- **Queryable catalogue** — filter by any identity field, spatial extent (including across the antimeridian), and time range; browse what a catalogue holds with `cat.variables`, `cat.owners`, and friends.
|
|
47
|
+
- **CF standard names derived for you** — envlib curates the mapping from its variables to CF `standard_name`s and applies it automatically at registration.
|
|
48
|
+
- **cfdb storage** — every dataset version is a [cfdb](https://github.com/mullenkamp/cfdb) file: chunked, compressed, partially readable, S3-syncable.
|
|
49
|
+
|
|
50
|
+
## Installation
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
pip install envlib
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Quick example
|
|
57
|
+
|
|
58
|
+
```python
|
|
59
|
+
import envlib
|
|
60
|
+
|
|
61
|
+
# The public envlib catalogue — zero config, no credentials
|
|
62
|
+
# (until it's hosted: Catalogue(remotes=['https://.../catalogue.rcg']))
|
|
63
|
+
cat = envlib.Catalogue()
|
|
64
|
+
|
|
65
|
+
# What's in it?
|
|
66
|
+
cat.variables # ['precipitation', 'streamflow', 'temperature']
|
|
67
|
+
cat.owners # ['ecan', 'ecmwf', 'niwa']
|
|
68
|
+
|
|
69
|
+
# Find the latest version of each matching dataset
|
|
70
|
+
results = cat.query(
|
|
71
|
+
variable='temperature',
|
|
72
|
+
feature='atmosphere',
|
|
73
|
+
bbox=[166, -47, 179, -34],
|
|
74
|
+
start_date='2020-01-01',
|
|
75
|
+
)
|
|
76
|
+
|
|
77
|
+
# Open one as a cfdb dataset and read it
|
|
78
|
+
ds = results[0].open()
|
|
79
|
+
temp = ds['temperature']
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Producing data is the same library in the other direction: build a cfdb file, attach `envlib.Metadata`, and `cat.publish(...)` it — see the [documentation](https://envlib.github.io/envlib/) for the full guide.
|
|
83
|
+
|
|
84
|
+
Changes between releases are tracked in the [changelog](https://envlib.github.io/envlib/changelog/).
|
|
85
|
+
|
|
86
|
+
## Development
|
|
87
|
+
|
|
88
|
+
### Setup environment
|
|
89
|
+
|
|
90
|
+
We use [UV](https://docs.astral.sh/uv/) to manage the development environment and production build.
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
uv sync
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
### Run unit tests
|
|
97
|
+
|
|
98
|
+
You can run all the tests with:
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
uv run pytest
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### Format the code
|
|
105
|
+
|
|
106
|
+
Execute the following commands to apply linting and check typing:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
uv run ruff check .
|
|
110
|
+
uv run black --check --diff .
|
|
111
|
+
uv run mypy --install-types --non-interactive envlib
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
To auto-format:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
uv run black .
|
|
118
|
+
uv run ruff check --fix .
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
## License
|
|
122
|
+
|
|
123
|
+
This project is licensed under the terms of the Apache Software License 2.0.
|
envlib-0.1.1/README.md
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# envlib
|
|
2
|
+
|
|
3
|
+
<p align="center">
|
|
4
|
+
<em>A distributed catalogue for environmental data</em>
|
|
5
|
+
</p>
|
|
6
|
+
|
|
7
|
+
[](https://github.com/envlib/envlib/actions)
|
|
8
|
+
[](https://codecov.io/gh/envlib/envlib)
|
|
9
|
+
[](https://badge.fury.io/py/envlib)
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
**Documentation**: <a href="https://envlib.github.io/envlib/" target="_blank">https://envlib.github.io/envlib/</a>
|
|
14
|
+
|
|
15
|
+
**Source Code**: <a href="https://github.com/envlib/envlib" target="_blank">https://github.com/envlib/envlib</a>
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Overview
|
|
20
|
+
|
|
21
|
+
envlib is a distributed database and catalogue for environmental datasets — gridded model output and station time series alike. Datasets are stored as [cfdb](https://github.com/mullenkamp/cfdb) files on S3-compatible object storage, hosted by whoever owns the data; envlib provides the shared layer on top: standardized metadata, controlled vocabularies, deterministic identifiers, and a catalogue you can query to discover and open any registered dataset.
|
|
22
|
+
|
|
23
|
+
## Key features
|
|
24
|
+
|
|
25
|
+
- **Distributed by design** — the catalogue is an index, not a data silo: each dataset stays on its owner's storage, and one catalogue can span many owners.
|
|
26
|
+
- **Standardized metadata** — eleven identity fields (drawing on [ODM2](http://vocabulary.odm2.org/), [CF conventions](https://cfconventions.org/), and lessons from tethys) describe what every dataset *is*, validated against controlled vocabularies at creation time.
|
|
27
|
+
- **Deterministic, permanent identifiers** — a dataset's identity metadata hashes to stable ids (`dataset_id`, `dataset_version_id`, `station_id`), so the same data gets the same id everywhere, forever.
|
|
28
|
+
- **Queryable catalogue** — filter by any identity field, spatial extent (including across the antimeridian), and time range; browse what a catalogue holds with `cat.variables`, `cat.owners`, and friends.
|
|
29
|
+
- **CF standard names derived for you** — envlib curates the mapping from its variables to CF `standard_name`s and applies it automatically at registration.
|
|
30
|
+
- **cfdb storage** — every dataset version is a [cfdb](https://github.com/mullenkamp/cfdb) file: chunked, compressed, partially readable, S3-syncable.
|
|
31
|
+
|
|
32
|
+
## Installation
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
pip install envlib
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Quick example
|
|
39
|
+
|
|
40
|
+
```python
|
|
41
|
+
import envlib
|
|
42
|
+
|
|
43
|
+
# The public envlib catalogue — zero config, no credentials
|
|
44
|
+
# (until it's hosted: Catalogue(remotes=['https://.../catalogue.rcg']))
|
|
45
|
+
cat = envlib.Catalogue()
|
|
46
|
+
|
|
47
|
+
# What's in it?
|
|
48
|
+
cat.variables # ['precipitation', 'streamflow', 'temperature']
|
|
49
|
+
cat.owners # ['ecan', 'ecmwf', 'niwa']
|
|
50
|
+
|
|
51
|
+
# Find the latest version of each matching dataset
|
|
52
|
+
results = cat.query(
|
|
53
|
+
variable='temperature',
|
|
54
|
+
feature='atmosphere',
|
|
55
|
+
bbox=[166, -47, 179, -34],
|
|
56
|
+
start_date='2020-01-01',
|
|
57
|
+
)
|
|
58
|
+
|
|
59
|
+
# Open one as a cfdb dataset and read it
|
|
60
|
+
ds = results[0].open()
|
|
61
|
+
temp = ds['temperature']
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Producing data is the same library in the other direction: build a cfdb file, attach `envlib.Metadata`, and `cat.publish(...)` it — see the [documentation](https://envlib.github.io/envlib/) for the full guide.
|
|
65
|
+
|
|
66
|
+
Changes between releases are tracked in the [changelog](https://envlib.github.io/envlib/changelog/).
|
|
67
|
+
|
|
68
|
+
## Development
|
|
69
|
+
|
|
70
|
+
### Setup environment
|
|
71
|
+
|
|
72
|
+
We use [UV](https://docs.astral.sh/uv/) to manage the development environment and production build.
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
uv sync
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
### Run unit tests
|
|
79
|
+
|
|
80
|
+
You can run all the tests with:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
uv run pytest
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
### Format the code
|
|
87
|
+
|
|
88
|
+
Execute the following commands to apply linting and check typing:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
uv run ruff check .
|
|
92
|
+
uv run black --check --diff .
|
|
93
|
+
uv run mypy --install-types --non-interactive envlib
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
To auto-format:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
uv run black .
|
|
100
|
+
uv run ruff check --fix .
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## License
|
|
104
|
+
|
|
105
|
+
This project is licensed under the terms of the Apache Software License 2.0.
|
|
@@ -4,6 +4,6 @@ from envlib import vocabularies
|
|
|
4
4
|
from envlib.catalogue import Catalogue, DatasetRef
|
|
5
5
|
from envlib.metadata import Metadata, ValidationError, compute_station_id
|
|
6
6
|
|
|
7
|
-
__version__ = '0.1.
|
|
7
|
+
__version__ = '0.1.1'
|
|
8
8
|
|
|
9
9
|
__all__ = ['Catalogue', 'DatasetRef', 'Metadata', 'ValidationError', 'compute_station_id', 'vocabularies']
|
|
@@ -10,7 +10,6 @@ is the authoritative metadata store; RCG entries are a derived index.
|
|
|
10
10
|
from __future__ import annotations
|
|
11
11
|
|
|
12
12
|
import datetime
|
|
13
|
-
import json
|
|
14
13
|
import os
|
|
15
14
|
import pathlib
|
|
16
15
|
import re
|
|
@@ -18,12 +17,10 @@ import urllib.parse
|
|
|
18
17
|
import warnings
|
|
19
18
|
from hashlib import blake2b
|
|
20
19
|
|
|
21
|
-
import booklet
|
|
22
20
|
import cfdb
|
|
23
21
|
import ebooklet
|
|
24
22
|
import pyproj
|
|
25
23
|
import shapely
|
|
26
|
-
import urllib3
|
|
27
24
|
|
|
28
25
|
from envlib import vocabularies
|
|
29
26
|
from envlib.metadata import (
|
|
@@ -48,10 +45,20 @@ _FULL_CIRCLE_DEG = 360.0
|
|
|
48
45
|
|
|
49
46
|
_QUERYABLE_FIELDS = frozenset(IDENTITY_FIELDS) | {'license', 'dataset_type', 'dataset_version_id', 'dataset_id'}
|
|
50
47
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
48
|
+
|
|
49
|
+
def _raise_on_push_failure(result, context: str):
|
|
50
|
+
"""ebooklet's push() returns a PushResult; result.failures maps failed
|
|
51
|
+
keys to error strings (the pending changes are retained by ebooklet for
|
|
52
|
+
retry). A partial failure must never pass silently: publish/deregister
|
|
53
|
+
would claim success while the remote is not fully updated."""
|
|
54
|
+
if result.failures:
|
|
55
|
+
msg = (
|
|
56
|
+
f'{context}: the push partially failed and the remote was NOT fully updated. '
|
|
57
|
+
f'Failed keys: {result.failures}. The pending changes are retained - fix the '
|
|
58
|
+
'cause (see the ebooklet operations guide, docs/ops.md, for the recovery '
|
|
59
|
+
'recipes) and retry.'
|
|
60
|
+
)
|
|
61
|
+
raise RuntimeError(msg)
|
|
55
62
|
|
|
56
63
|
|
|
57
64
|
def _public_rcg_url() -> str | None:
|
|
@@ -200,12 +207,6 @@ def _bbox_within_radius(stored, lon: float, lat: float, radius_km: float) -> boo
|
|
|
200
207
|
# Dataset validation + State Metadata extraction
|
|
201
208
|
|
|
202
209
|
|
|
203
|
-
def _sys_dataset_type(ds) -> str:
|
|
204
|
-
# cfdb 0.9.0 has no public dataset_type property (and open_edataset always
|
|
205
|
-
# returns the EGrid class), so the sys-metadata slot is the reliable source.
|
|
206
|
-
return ds._sys_meta.dataset_type.value
|
|
207
|
-
|
|
208
|
-
|
|
209
210
|
def _coord_by_axis(ds, axis: str):
|
|
210
211
|
for coord in ds.coords:
|
|
211
212
|
ax = coord.axis
|
|
@@ -227,7 +228,7 @@ def _reproject_points(points, crs) -> list:
|
|
|
227
228
|
|
|
228
229
|
def _extract_state(ds, meta: Metadata) -> dict:
|
|
229
230
|
"""Extract State Metadata (bbox, time range, steps, dataset_type) from an open dataset."""
|
|
230
|
-
dataset_type =
|
|
231
|
+
dataset_type = ds.dataset_type
|
|
231
232
|
crs = ds.crs
|
|
232
233
|
|
|
233
234
|
time_coord = ds.get('time') if 'time' in ds.coord_names else None
|
|
@@ -368,7 +369,7 @@ def _decide_standard_name(ds, meta: Metadata) -> dict:
|
|
|
368
369
|
|
|
369
370
|
|
|
370
371
|
def _validate_dataset(ds, *, validate_cv: bool) -> dict:
|
|
371
|
-
"""Validate an open cfdb
|
|
372
|
+
"""Validate an open cfdb file against envlib requirements; extract everything.
|
|
372
373
|
|
|
373
374
|
Returns {'metadata', 'dataset_version_id', 'dataset_id', 'state', 'standard_name'}.
|
|
374
375
|
Raises ValidationError on any failure. Never modifies the dataset.
|
|
@@ -402,7 +403,7 @@ def _validate_dataset(ds, *, validate_cv: bool) -> dict:
|
|
|
402
403
|
msg = f'declared ancillary variable {name!r} is not a data variable in the dataset.'
|
|
403
404
|
raise ValidationError(msg)
|
|
404
405
|
|
|
405
|
-
if
|
|
406
|
+
if ds.dataset_type == 'ts_ortho':
|
|
406
407
|
_check_stations(ds)
|
|
407
408
|
|
|
408
409
|
state = _extract_state(ds, meta)
|
|
@@ -443,7 +444,7 @@ def _apply_derived_attrs(ds, result: dict) -> bool:
|
|
|
443
444
|
|
|
444
445
|
|
|
445
446
|
class DatasetRef:
|
|
446
|
-
"""A catalogue entry: metadata plus how to open
|
|
447
|
+
"""A catalogue entry: one dataset version's metadata, plus how to open its cfdb file."""
|
|
447
448
|
|
|
448
449
|
def __init__(self, dataset_version_id: str, entry: dict, cache_dir: pathlib.Path):
|
|
449
450
|
self._dataset_id = dataset_version_id
|
|
@@ -473,7 +474,7 @@ class DatasetRef:
|
|
|
473
474
|
return repr(self.metadata)
|
|
474
475
|
|
|
475
476
|
def open(self, file_path=None, access_key_id=None, access_key=None):
|
|
476
|
-
"""Open
|
|
477
|
+
"""Open this dataset version's cfdb file as a read-only EDataset.
|
|
477
478
|
|
|
478
479
|
Entries never store credentials. Public-HTTPS datasets open via their
|
|
479
480
|
``data_url`` with no credentials; for private buckets inject
|
|
@@ -516,6 +517,13 @@ class Catalogue:
|
|
|
516
517
|
to re-pull after new registrations. When a remote is unreachable and a
|
|
517
518
|
previously pulled local index exists, the catalogue degrades to the cached
|
|
518
519
|
copy (with a warning) instead of failing.
|
|
520
|
+
|
|
521
|
+
Example:
|
|
522
|
+
>>> cat = Catalogue(remotes=['https://s3.example.com/bucket/catalogue.rcg'])
|
|
523
|
+
>>> cat.variables
|
|
524
|
+
['streamflow', 'temperature']
|
|
525
|
+
>>> refs = cat.query(variable='temperature', feature='atmosphere')
|
|
526
|
+
>>> ds = refs[0].open()
|
|
519
527
|
"""
|
|
520
528
|
|
|
521
529
|
def __init__(self, remotes=None, cache: str = DEFAULT_CACHE_DIR, *, include_public: bool = False):
|
|
@@ -558,23 +566,34 @@ class Catalogue:
|
|
|
558
566
|
for source in self._sources:
|
|
559
567
|
path = self._rcg_cache_path(source)
|
|
560
568
|
try:
|
|
561
|
-
|
|
569
|
+
# offline='auto': ebooklet falls back to the cached local index
|
|
570
|
+
# (with its own UserWarning) ONLY on transport-level
|
|
571
|
+
# unreachability. Everything else raises typed: a broken store
|
|
572
|
+
# (RemoteIntegrityError), an incompatible format
|
|
573
|
+
# (UnsupportedFormatError - pre-0.10 this was silently
|
|
574
|
+
# mislabeled "not readable yet" by a blanket ValueError catch),
|
|
575
|
+
# and unreachable-with-no-cache (OfflineError) all propagate.
|
|
576
|
+
with ebooklet.open_rcg(source, path, flag='r', offline='auto') as rcg:
|
|
562
577
|
source_entries = {k: v for k, v in rcg.items() if _HEX24_RE.fullmatch(str(k))}
|
|
563
|
-
except
|
|
564
|
-
#
|
|
565
|
-
# remote
|
|
566
|
-
#
|
|
567
|
-
#
|
|
568
|
-
warnings.warn(f'RCG source not readable yet ({err}); treating as empty.', stacklevel=2)
|
|
569
|
-
source_entries = {}
|
|
570
|
-
except _OFFLINE_ERRORS as err:
|
|
571
|
-
if not path.exists():
|
|
572
|
-
raise
|
|
578
|
+
except ebooklet.UUIDMismatchError as err:
|
|
579
|
+
# The local cache identifies a DIFFERENT database than the
|
|
580
|
+
# remote - the remote was likely deleted and recreated.
|
|
581
|
+
# Distinct from the bootstrap case: the fix is deleting the
|
|
582
|
+
# stale cache file, so say so instead of a generic warning.
|
|
573
583
|
warnings.warn(
|
|
574
|
-
f'RCG
|
|
584
|
+
f'RCG source identity mismatch ({err}); the remote was likely deleted and '
|
|
585
|
+
f'recreated. Delete the stale cache at {path} to adopt the new remote. '
|
|
586
|
+
'Treating this source as empty for now.',
|
|
575
587
|
stacklevel=2,
|
|
576
588
|
)
|
|
577
|
-
source_entries =
|
|
589
|
+
source_entries = {}
|
|
590
|
+
except ebooklet.RemoteMissingError as err:
|
|
591
|
+
# The RCG does not exist on the remote yet (and no local copy
|
|
592
|
+
# exists) — the bootstrap case: a producer constructs the
|
|
593
|
+
# Catalogue before the first publish creates the RCG. Treat as
|
|
594
|
+
# an empty source, loudly.
|
|
595
|
+
warnings.warn(f'RCG source not readable yet ({err}); treating as empty.', stacklevel=2)
|
|
596
|
+
source_entries = {}
|
|
578
597
|
for key, entry in source_entries.items():
|
|
579
598
|
entries.setdefault(key, entry) # first configured source wins on duplicates
|
|
580
599
|
self._entries = entries
|
|
@@ -770,13 +789,13 @@ class Catalogue:
|
|
|
770
789
|
first_time = eds.attrs.get('envlib_dataset_version_id') is None
|
|
771
790
|
result = _validate_dataset(eds, validate_cv=first_time)
|
|
772
791
|
_apply_derived_attrs(eds, result)
|
|
773
|
-
eds.push()
|
|
792
|
+
_raise_on_push_failure(eds.push(), 'publish: pushing the dataset data')
|
|
774
793
|
|
|
775
794
|
self._upsert_entry(rcg_remote_conn, member_conn, result)
|
|
776
795
|
return result
|
|
777
796
|
|
|
778
797
|
def register(self, remote_conn, rcg_remote_conn, **open_kwargs) -> dict:
|
|
779
|
-
"""Register an already-remote cfdb
|
|
798
|
+
"""Register an already-remote cfdb file in the catalogue (no data push).
|
|
780
799
|
|
|
781
800
|
``remote_conn`` must be writable (credentials): first registration
|
|
782
801
|
writes the self-identification attrs (and any auto-populated
|
|
@@ -788,7 +807,7 @@ class Catalogue:
|
|
|
788
807
|
first_time = eds.attrs.get('envlib_dataset_version_id') is None
|
|
789
808
|
result = _validate_dataset(eds, validate_cv=first_time)
|
|
790
809
|
if _apply_derived_attrs(eds, result):
|
|
791
|
-
eds.push()
|
|
810
|
+
_raise_on_push_failure(eds.push(), 'register: pushing the self-identification attrs')
|
|
792
811
|
|
|
793
812
|
self._upsert_entry(rcg_remote_conn, member_conn, result)
|
|
794
813
|
return result
|
|
@@ -820,8 +839,7 @@ class Catalogue:
|
|
|
820
839
|
target_conn = entry.get('remote_conn') or {}
|
|
821
840
|
target = (target_conn.get('endpoint_url'), target_conn.get('bucket'), target_conn.get('db_key'))
|
|
822
841
|
# list() first: fetching entries while iterating keys() would
|
|
823
|
-
# deadlock on the underlying booklet thread lock
|
|
824
|
-
# _read_cached_index).
|
|
842
|
+
# deadlock on the underlying booklet thread lock.
|
|
825
843
|
for other_key in list(rcg.keys()):
|
|
826
844
|
if other_key == dataset_version_id or not _HEX24_RE.fullmatch(str(other_key)):
|
|
827
845
|
continue
|
|
@@ -847,7 +865,7 @@ class Catalogue:
|
|
|
847
865
|
with member_conn.open('w') as session:
|
|
848
866
|
session.delete_remote()
|
|
849
867
|
del rcg[dataset_version_id]
|
|
850
|
-
rcg.changes().push()
|
|
868
|
+
_raise_on_push_failure(rcg.changes().push(), 'deregister: pushing the catalogue entry removal')
|
|
851
869
|
self.refresh()
|
|
852
870
|
|
|
853
871
|
# -- entry construction ----------------------------------------------------
|
|
@@ -871,7 +889,7 @@ class Catalogue:
|
|
|
871
889
|
|
|
872
890
|
user_meta['modified_at'] = now
|
|
873
891
|
rcg.add(member_conn, key=dataset_version_id, user_meta=user_meta)
|
|
874
|
-
rcg.changes().push()
|
|
892
|
+
_raise_on_push_failure(rcg.changes().push(), 'publish/register: pushing the catalogue entry')
|
|
875
893
|
self.refresh()
|
|
876
894
|
|
|
877
895
|
|
|
@@ -937,34 +955,3 @@ def _created_at(ref: DatasetRef) -> datetime.datetime:
|
|
|
937
955
|
if value is None:
|
|
938
956
|
return datetime.datetime.min.replace(tzinfo=datetime.timezone.utc)
|
|
939
957
|
return _parse_iso(value)
|
|
940
|
-
|
|
941
|
-
|
|
942
|
-
def _read_cached_index(path: pathlib.Path) -> dict:
|
|
943
|
-
"""Offline fallback: read a previously pulled RCG index directly with booklet.
|
|
944
|
-
|
|
945
|
-
Booklet files are self-describing (serializers stored in-file), so entries
|
|
946
|
-
decode without parameters. envlib entries are keyed by 24-hex dataset_ids;
|
|
947
|
-
everything else (ebooklet-internal keys) is skipped. Values that fail to
|
|
948
|
-
decode as entry dicts are skipped too.
|
|
949
|
-
"""
|
|
950
|
-
entries: dict = {}
|
|
951
|
-
with booklet.open(path, 'r') as blt:
|
|
952
|
-
# materialize keys() BEFORE fetching: booklet's keys() generator holds
|
|
953
|
-
# the file's thread lock across yields, so a get() inside the loop
|
|
954
|
-
# deadlocks (single-threaded) — found via the hung offline test.
|
|
955
|
-
for key in list(blt.keys()):
|
|
956
|
-
key_str = key.decode() if isinstance(key, bytes) else str(key)
|
|
957
|
-
if not _HEX24_RE.fullmatch(key_str):
|
|
958
|
-
continue
|
|
959
|
-
try:
|
|
960
|
-
value = blt[key]
|
|
961
|
-
except (KeyError, ValueError):
|
|
962
|
-
continue
|
|
963
|
-
if isinstance(value, bytes):
|
|
964
|
-
try:
|
|
965
|
-
value = json.loads(value)
|
|
966
|
-
except ValueError:
|
|
967
|
-
continue
|
|
968
|
-
if isinstance(value, dict) and 'remote_conn' in value:
|
|
969
|
-
entries[key_str] = value
|
|
970
|
-
return entries
|
|
@@ -12,6 +12,7 @@ from __future__ import annotations
|
|
|
12
12
|
|
|
13
13
|
import math
|
|
14
14
|
import re
|
|
15
|
+
from collections.abc import Mapping
|
|
15
16
|
from hashlib import blake2b
|
|
16
17
|
from typing import cast
|
|
17
18
|
|
|
@@ -503,7 +504,7 @@ class Metadata:
|
|
|
503
504
|
return out
|
|
504
505
|
|
|
505
506
|
@classmethod
|
|
506
|
-
def from_attrs(cls, attrs, *, validate_cv: bool = True) -> Metadata:
|
|
507
|
+
def from_attrs(cls, attrs: Mapping, *, validate_cv: bool = True) -> Metadata:
|
|
507
508
|
"""Rebuild Metadata from ``envlib_``-prefixed attrs (``ds.attrs`` or a dict).
|
|
508
509
|
|
|
509
510
|
Non-envlib keys are ignored. When ``envlib_dataset_version_id`` /
|
|
@@ -293,7 +293,7 @@ def _refresh_variable(target_dir: pathlib.Path) -> dict:
|
|
|
293
293
|
}
|
|
294
294
|
|
|
295
295
|
|
|
296
|
-
def refresh(field: str | None = None, _target_dir=None) -> dict:
|
|
296
|
+
def refresh(field: str | None = None, _target_dir: str | pathlib.Path | None = None) -> dict:
|
|
297
297
|
"""Refresh upstream-sourced vocabularies from their APIs.
|
|
298
298
|
|
|
299
299
|
Fetches the ODM2 variablename list and/or the current CF standard-name table
|
|
@@ -11,9 +11,9 @@ classifiers = [
|
|
|
11
11
|
]
|
|
12
12
|
requires-python = ">=3.10"
|
|
13
13
|
dependencies = [
|
|
14
|
-
"cfdb>=0.9.
|
|
15
|
-
"ebooklet>=0.
|
|
16
|
-
"booklet>=0.12.
|
|
14
|
+
"cfdb>=0.9.1",
|
|
15
|
+
"ebooklet>=0.10.0",
|
|
16
|
+
"booklet>=0.12.6",
|
|
17
17
|
"pyproj",
|
|
18
18
|
"shapely>=2.0",
|
|
19
19
|
"urllib3",
|
|
@@ -31,6 +31,7 @@ dev = [
|
|
|
31
31
|
"mkdocstrings[python]",
|
|
32
32
|
"types-shapely>=2.1.0.20260630",
|
|
33
33
|
"numpy>=2.2.6",
|
|
34
|
+
"tomli>=2.4.1 ; python_full_version < '3.11'",
|
|
34
35
|
]
|
|
35
36
|
|
|
36
37
|
[tool.pytest.ini_options]
|
|
@@ -130,8 +131,8 @@ requires = ["hatchling>=1.26.1"]
|
|
|
130
131
|
build-backend = "hatchling.build"
|
|
131
132
|
|
|
132
133
|
[project.urls]
|
|
133
|
-
Documentation = "https://
|
|
134
|
-
Source = "https://github.com/envlib"
|
|
134
|
+
Documentation = "https://envlib.github.io/envlib/"
|
|
135
|
+
Source = "https://github.com/envlib/envlib"
|
|
135
136
|
|
|
136
137
|
[tool.hatch.build.targets.sdist]
|
|
137
138
|
include = [
|
envlib-0.1.0/PKG-INFO
DELETED
|
@@ -1,75 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: envlib
|
|
3
|
-
Version: 0.1.0
|
|
4
|
-
Summary: The environmental library
|
|
5
|
-
Project-URL: Documentation, https://mullenkamp.github.io/envlib/
|
|
6
|
-
Project-URL: Source, https://github.com/envlib
|
|
7
|
-
Author-email: mullenkamp <mullenkamp1@gmail.com>
|
|
8
|
-
License-File: LICENSE
|
|
9
|
-
Classifier: Programming Language :: Python :: 3 :: Only
|
|
10
|
-
Requires-Python: >=3.10
|
|
11
|
-
Requires-Dist: booklet>=0.12.5
|
|
12
|
-
Requires-Dist: cfdb>=0.9.0
|
|
13
|
-
Requires-Dist: ebooklet>=0.9.1
|
|
14
|
-
Requires-Dist: pyproj
|
|
15
|
-
Requires-Dist: shapely>=2.0
|
|
16
|
-
Requires-Dist: urllib3
|
|
17
|
-
Description-Content-Type: text/markdown
|
|
18
|
-
|
|
19
|
-
# envlib
|
|
20
|
-
|
|
21
|
-
<p align="center">
|
|
22
|
-
<em>The environmental library</em>
|
|
23
|
-
</p>
|
|
24
|
-
|
|
25
|
-
[](https://github.com/envlib/actions)
|
|
26
|
-
[](https://codecov.io/gh/mullenkamp/envlib)
|
|
27
|
-
[](https://badge.fury.io/py/envlib)
|
|
28
|
-
|
|
29
|
-
---
|
|
30
|
-
|
|
31
|
-
**Source Code**: <a href="https://github.com/envlib" target="_blank">https://github.com/envlib</a>
|
|
32
|
-
|
|
33
|
-
---
|
|
34
|
-
## Overview
|
|
35
|
-
envlib is a distributed database and catalogue for environmental data. It uses controlled vocabulary and standarized metadata to make it easy for users to query and access data. It uses cfdb as the backend. The metadata structure is based on tethysts.
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
## Development
|
|
39
|
-
|
|
40
|
-
### Setup environment
|
|
41
|
-
|
|
42
|
-
We use [UV](https://docs.astral.sh/uv/) to manage the development environment and production build.
|
|
43
|
-
|
|
44
|
-
```bash
|
|
45
|
-
uv sync
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
### Run unit tests
|
|
49
|
-
|
|
50
|
-
You can run all the tests with:
|
|
51
|
-
|
|
52
|
-
```bash
|
|
53
|
-
uv run pytest
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
### Format the code
|
|
57
|
-
|
|
58
|
-
Execute the following commands to apply linting and check typing:
|
|
59
|
-
|
|
60
|
-
```bash
|
|
61
|
-
uv run ruff check .
|
|
62
|
-
uv run black --check --diff .
|
|
63
|
-
uv run mypy --install-types --non-interactive envlib
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
To auto-format:
|
|
67
|
-
|
|
68
|
-
```bash
|
|
69
|
-
uv run black .
|
|
70
|
-
uv run ruff check --fix .
|
|
71
|
-
```
|
|
72
|
-
|
|
73
|
-
## License
|
|
74
|
-
|
|
75
|
-
This project is licensed under the terms of the Apache Software License 2.0.
|
envlib-0.1.0/README.md
DELETED
|
@@ -1,57 +0,0 @@
|
|
|
1
|
-
# envlib
|
|
2
|
-
|
|
3
|
-
<p align="center">
|
|
4
|
-
<em>The environmental library</em>
|
|
5
|
-
</p>
|
|
6
|
-
|
|
7
|
-
[](https://github.com/envlib/actions)
|
|
8
|
-
[](https://codecov.io/gh/mullenkamp/envlib)
|
|
9
|
-
[](https://badge.fury.io/py/envlib)
|
|
10
|
-
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
**Source Code**: <a href="https://github.com/envlib" target="_blank">https://github.com/envlib</a>
|
|
14
|
-
|
|
15
|
-
---
|
|
16
|
-
## Overview
|
|
17
|
-
envlib is a distributed database and catalogue for environmental data. It uses controlled vocabulary and standarized metadata to make it easy for users to query and access data. It uses cfdb as the backend. The metadata structure is based on tethysts.
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
## Development
|
|
21
|
-
|
|
22
|
-
### Setup environment
|
|
23
|
-
|
|
24
|
-
We use [UV](https://docs.astral.sh/uv/) to manage the development environment and production build.
|
|
25
|
-
|
|
26
|
-
```bash
|
|
27
|
-
uv sync
|
|
28
|
-
```
|
|
29
|
-
|
|
30
|
-
### Run unit tests
|
|
31
|
-
|
|
32
|
-
You can run all the tests with:
|
|
33
|
-
|
|
34
|
-
```bash
|
|
35
|
-
uv run pytest
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
### Format the code
|
|
39
|
-
|
|
40
|
-
Execute the following commands to apply linting and check typing:
|
|
41
|
-
|
|
42
|
-
```bash
|
|
43
|
-
uv run ruff check .
|
|
44
|
-
uv run black --check --diff .
|
|
45
|
-
uv run mypy --install-types --non-interactive envlib
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
To auto-format:
|
|
49
|
-
|
|
50
|
-
```bash
|
|
51
|
-
uv run black .
|
|
52
|
-
uv run ruff check --fix .
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
## License
|
|
56
|
-
|
|
57
|
-
This project is licensed under the terms of the Apache Software License 2.0.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|