envlib 0.1.0__tar.gz → 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {envlib-0.1.0 → envlib-0.1.2}/.gitignore +5 -1
- envlib-0.1.2/PKG-INFO +123 -0
- envlib-0.1.2/README.md +105 -0
- {envlib-0.1.0 → envlib-0.1.2}/envlib/__init__.py +1 -1
- {envlib-0.1.0 → envlib-0.1.2}/envlib/catalogue.py +61 -73
- {envlib-0.1.0 → envlib-0.1.2}/envlib/metadata.py +2 -1
- {envlib-0.1.0 → envlib-0.1.2}/envlib/vocabularies/__init__.py +1 -1
- {envlib-0.1.0 → envlib-0.1.2}/pyproject.toml +6 -5
- envlib-0.1.0/PKG-INFO +0 -75
- envlib-0.1.0/README.md +0 -57
- {envlib-0.1.0 → envlib-0.1.2}/LICENSE +0 -0
- {envlib-0.1.0 → envlib-0.1.2}/envlib/vocabularies/aggregation_statistic.json +0 -0
- {envlib-0.1.0 → envlib-0.1.2}/envlib/vocabularies/feature.json +0 -0
- {envlib-0.1.0 → envlib-0.1.2}/envlib/vocabularies/frequency_interval.json +0 -0
- {envlib-0.1.0 → envlib-0.1.2}/envlib/vocabularies/license.json +0 -0
- {envlib-0.1.0 → envlib-0.1.2}/envlib/vocabularies/method.json +0 -0
- {envlib-0.1.0 → envlib-0.1.2}/envlib/vocabularies/processing_level.json +0 -0
- {envlib-0.1.0 → envlib-0.1.2}/envlib/vocabularies/standard_name.json +0 -0
- {envlib-0.1.0 → envlib-0.1.2}/envlib/vocabularies/variable.json +0 -0
envlib-0.1.2/PKG-INFO
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: envlib
|
|
3
|
+
Version: 0.1.2
|
|
4
|
+
Summary: The environmental library
|
|
5
|
+
Project-URL: Documentation, https://envlib.github.io/envlib/
|
|
6
|
+
Project-URL: Source, https://github.com/envlib/envlib
|
|
7
|
+
Author-email: mullenkamp <mullenkamp1@gmail.com>
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
10
|
+
Requires-Python: >=3.10
|
|
11
|
+
Requires-Dist: booklet>=0.12.6
|
|
12
|
+
Requires-Dist: cfdb>=0.9.1
|
|
13
|
+
Requires-Dist: ebooklet>=0.10.0
|
|
14
|
+
Requires-Dist: pyproj
|
|
15
|
+
Requires-Dist: shapely>=2.0
|
|
16
|
+
Requires-Dist: urllib3
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
|
|
19
|
+
# envlib
|
|
20
|
+
|
|
21
|
+
<p align="center">
|
|
22
|
+
<em>A distributed catalogue for environmental data</em>
|
|
23
|
+
</p>
|
|
24
|
+
|
|
25
|
+
[](https://github.com/envlib/envlib/actions)
|
|
26
|
+
[](https://codecov.io/gh/envlib/envlib)
|
|
27
|
+
[](https://badge.fury.io/py/envlib)
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
**Documentation**: <a href="https://envlib.github.io/envlib/" target="_blank">https://envlib.github.io/envlib/</a>
|
|
32
|
+
|
|
33
|
+
**Source Code**: <a href="https://github.com/envlib/envlib" target="_blank">https://github.com/envlib/envlib</a>
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Overview
|
|
38
|
+
|
|
39
|
+
envlib is a distributed database and catalogue for environmental datasets — gridded model output and station time series alike. Datasets are stored as [cfdb](https://github.com/mullenkamp/cfdb) files on S3-compatible object storage, hosted by whoever owns the data; envlib provides the shared layer on top: standardized metadata, controlled vocabularies, deterministic identifiers, and a catalogue you can query to discover and open any registered dataset.
|
|
40
|
+
|
|
41
|
+
## Key features
|
|
42
|
+
|
|
43
|
+
- **Distributed by design** — the catalogue is an index, not a data silo: each dataset stays on its owner's storage, and one catalogue can span many owners.
|
|
44
|
+
- **Standardized metadata** — eleven identity fields (drawing on [ODM2](http://vocabulary.odm2.org/), [CF conventions](https://cfconventions.org/), and lessons from tethys) describe what every dataset *is*, validated against controlled vocabularies at creation time.
|
|
45
|
+
- **Deterministic, permanent identifiers** — a dataset's identity metadata hashes to stable ids (`dataset_id`, `dataset_version_id`, `station_id`), so the same data gets the same id everywhere, forever.
|
|
46
|
+
- **Queryable catalogue** — filter by any identity field, spatial extent (including across the antimeridian), and time range; browse what a catalogue holds with `cat.variables`, `cat.owners`, and friends.
|
|
47
|
+
- **CF standard names derived for you** — envlib curates the mapping from its variables to CF `standard_name`s and applies it automatically at registration.
|
|
48
|
+
- **cfdb storage** — every dataset version is a [cfdb](https://github.com/mullenkamp/cfdb) file: chunked, compressed, partially readable, S3-syncable.
|
|
49
|
+
|
|
50
|
+
## Installation
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
pip install envlib
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Quick example
|
|
57
|
+
|
|
58
|
+
```python
|
|
59
|
+
import envlib
|
|
60
|
+
|
|
61
|
+
# The public envlib catalogue — zero config, no credentials
|
|
62
|
+
# (until it's hosted: Catalogue(remotes=['https://.../catalogue.rcg']))
|
|
63
|
+
cat = envlib.Catalogue()
|
|
64
|
+
|
|
65
|
+
# What's in it?
|
|
66
|
+
cat.variables # ['precipitation', 'streamflow', 'temperature']
|
|
67
|
+
cat.owners # ['ecan', 'ecmwf', 'niwa']
|
|
68
|
+
|
|
69
|
+
# Find the latest version of each matching dataset
|
|
70
|
+
results = cat.query(
|
|
71
|
+
variable='temperature',
|
|
72
|
+
feature='atmosphere',
|
|
73
|
+
bbox=[166, -47, 179, -34],
|
|
74
|
+
start_date='2020-01-01',
|
|
75
|
+
)
|
|
76
|
+
|
|
77
|
+
# Open one as a cfdb dataset and read it
|
|
78
|
+
ds = results[0].open()
|
|
79
|
+
temp = ds['temperature']
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Producing data is the same library in the other direction: build a cfdb file, attach `envlib.Metadata`, and `cat.publish(...)` it — see the [documentation](https://envlib.github.io/envlib/) for the full guide.
|
|
83
|
+
|
|
84
|
+
Changes between releases are tracked in the [changelog](https://envlib.github.io/envlib/changelog/).
|
|
85
|
+
|
|
86
|
+
## Development
|
|
87
|
+
|
|
88
|
+
### Setup environment
|
|
89
|
+
|
|
90
|
+
We use [UV](https://docs.astral.sh/uv/) to manage the development environment and production build.
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
uv sync
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
### Run unit tests
|
|
97
|
+
|
|
98
|
+
You can run all the tests with:
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
uv run pytest
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### Format the code
|
|
105
|
+
|
|
106
|
+
Execute the following commands to apply linting and check typing:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
uv run ruff check .
|
|
110
|
+
uv run black --check --diff .
|
|
111
|
+
uv run mypy --install-types --non-interactive envlib
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
To auto-format:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
uv run black .
|
|
118
|
+
uv run ruff check --fix .
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
## License
|
|
122
|
+
|
|
123
|
+
This project is licensed under the terms of the Apache Software License 2.0.
|
envlib-0.1.2/README.md
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# envlib
|
|
2
|
+
|
|
3
|
+
<p align="center">
|
|
4
|
+
<em>A distributed catalogue for environmental data</em>
|
|
5
|
+
</p>
|
|
6
|
+
|
|
7
|
+
[](https://github.com/envlib/envlib/actions)
|
|
8
|
+
[](https://codecov.io/gh/envlib/envlib)
|
|
9
|
+
[](https://badge.fury.io/py/envlib)
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
**Documentation**: <a href="https://envlib.github.io/envlib/" target="_blank">https://envlib.github.io/envlib/</a>
|
|
14
|
+
|
|
15
|
+
**Source Code**: <a href="https://github.com/envlib/envlib" target="_blank">https://github.com/envlib/envlib</a>
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Overview
|
|
20
|
+
|
|
21
|
+
envlib is a distributed database and catalogue for environmental datasets — gridded model output and station time series alike. Datasets are stored as [cfdb](https://github.com/mullenkamp/cfdb) files on S3-compatible object storage, hosted by whoever owns the data; envlib provides the shared layer on top: standardized metadata, controlled vocabularies, deterministic identifiers, and a catalogue you can query to discover and open any registered dataset.
|
|
22
|
+
|
|
23
|
+
## Key features
|
|
24
|
+
|
|
25
|
+
- **Distributed by design** — the catalogue is an index, not a data silo: each dataset stays on its owner's storage, and one catalogue can span many owners.
|
|
26
|
+
- **Standardized metadata** — eleven identity fields (drawing on [ODM2](http://vocabulary.odm2.org/), [CF conventions](https://cfconventions.org/), and lessons from tethys) describe what every dataset *is*, validated against controlled vocabularies at creation time.
|
|
27
|
+
- **Deterministic, permanent identifiers** — a dataset's identity metadata hashes to stable ids (`dataset_id`, `dataset_version_id`, `station_id`), so the same data gets the same id everywhere, forever.
|
|
28
|
+
- **Queryable catalogue** — filter by any identity field, spatial extent (including across the antimeridian), and time range; browse what a catalogue holds with `cat.variables`, `cat.owners`, and friends.
|
|
29
|
+
- **CF standard names derived for you** — envlib curates the mapping from its variables to CF `standard_name`s and applies it automatically at registration.
|
|
30
|
+
- **cfdb storage** — every dataset version is a [cfdb](https://github.com/mullenkamp/cfdb) file: chunked, compressed, partially readable, S3-syncable.
|
|
31
|
+
|
|
32
|
+
## Installation
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
pip install envlib
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Quick example
|
|
39
|
+
|
|
40
|
+
```python
|
|
41
|
+
import envlib
|
|
42
|
+
|
|
43
|
+
# The public envlib catalogue — zero config, no credentials
|
|
44
|
+
# (until it's hosted: Catalogue(remotes=['https://.../catalogue.rcg']))
|
|
45
|
+
cat = envlib.Catalogue()
|
|
46
|
+
|
|
47
|
+
# What's in it?
|
|
48
|
+
cat.variables # ['precipitation', 'streamflow', 'temperature']
|
|
49
|
+
cat.owners # ['ecan', 'ecmwf', 'niwa']
|
|
50
|
+
|
|
51
|
+
# Find the latest version of each matching dataset
|
|
52
|
+
results = cat.query(
|
|
53
|
+
variable='temperature',
|
|
54
|
+
feature='atmosphere',
|
|
55
|
+
bbox=[166, -47, 179, -34],
|
|
56
|
+
start_date='2020-01-01',
|
|
57
|
+
)
|
|
58
|
+
|
|
59
|
+
# Open one as a cfdb dataset and read it
|
|
60
|
+
ds = results[0].open()
|
|
61
|
+
temp = ds['temperature']
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Producing data is the same library in the other direction: build a cfdb file, attach `envlib.Metadata`, and `cat.publish(...)` it — see the [documentation](https://envlib.github.io/envlib/) for the full guide.
|
|
65
|
+
|
|
66
|
+
Changes between releases are tracked in the [changelog](https://envlib.github.io/envlib/changelog/).
|
|
67
|
+
|
|
68
|
+
## Development
|
|
69
|
+
|
|
70
|
+
### Setup environment
|
|
71
|
+
|
|
72
|
+
We use [UV](https://docs.astral.sh/uv/) to manage the development environment and production build.
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
uv sync
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
### Run unit tests
|
|
79
|
+
|
|
80
|
+
You can run all the tests with:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
uv run pytest
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
### Format the code
|
|
87
|
+
|
|
88
|
+
Execute the following commands to apply linting and check typing:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
uv run ruff check .
|
|
92
|
+
uv run black --check --diff .
|
|
93
|
+
uv run mypy --install-types --non-interactive envlib
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
To auto-format:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
uv run black .
|
|
100
|
+
uv run ruff check --fix .
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## License
|
|
104
|
+
|
|
105
|
+
This project is licensed under the terms of the Apache Software License 2.0.
|
|
@@ -4,6 +4,6 @@ from envlib import vocabularies
|
|
|
4
4
|
from envlib.catalogue import Catalogue, DatasetRef
|
|
5
5
|
from envlib.metadata import Metadata, ValidationError, compute_station_id
|
|
6
6
|
|
|
7
|
-
__version__ = '0.1.
|
|
7
|
+
__version__ = '0.1.2'
|
|
8
8
|
|
|
9
9
|
__all__ = ['Catalogue', 'DatasetRef', 'Metadata', 'ValidationError', 'compute_station_id', 'vocabularies']
|
|
@@ -10,7 +10,6 @@ is the authoritative metadata store; RCG entries are a derived index.
|
|
|
10
10
|
from __future__ import annotations
|
|
11
11
|
|
|
12
12
|
import datetime
|
|
13
|
-
import json
|
|
14
13
|
import os
|
|
15
14
|
import pathlib
|
|
16
15
|
import re
|
|
@@ -18,12 +17,10 @@ import urllib.parse
|
|
|
18
17
|
import warnings
|
|
19
18
|
from hashlib import blake2b
|
|
20
19
|
|
|
21
|
-
import booklet
|
|
22
20
|
import cfdb
|
|
23
21
|
import ebooklet
|
|
24
22
|
import pyproj
|
|
25
23
|
import shapely
|
|
26
|
-
import urllib3
|
|
27
24
|
|
|
28
25
|
from envlib import vocabularies
|
|
29
26
|
from envlib.metadata import (
|
|
@@ -34,9 +31,10 @@ from envlib.metadata import (
|
|
|
34
31
|
compute_station_id,
|
|
35
32
|
)
|
|
36
33
|
|
|
37
|
-
# The
|
|
38
|
-
#
|
|
39
|
-
|
|
34
|
+
# The public envlib commons catalogue (read-only, credential-less HTTPS; a
|
|
35
|
+
# custom domain fronting the hosting bucket, so the URL outlives any bucket
|
|
36
|
+
# move). The env var below overrides it (stand-ins, testing, mirrors).
|
|
37
|
+
PUBLIC_RCG_URL: str | None = 'https://b2.envlib.xyz/file/envlib/envlib-commons/catalogue'
|
|
40
38
|
PUBLIC_RCG_ENV_VAR = 'ENVLIB_PUBLIC_RCG_URL'
|
|
41
39
|
DEFAULT_CACHE_DIR = '~/.envlib/cache'
|
|
42
40
|
|
|
@@ -48,10 +46,20 @@ _FULL_CIRCLE_DEG = 360.0
|
|
|
48
46
|
|
|
49
47
|
_QUERYABLE_FIELDS = frozenset(IDENTITY_FIELDS) | {'license', 'dataset_type', 'dataset_version_id', 'dataset_id'}
|
|
50
48
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
49
|
+
|
|
50
|
+
def _raise_on_push_failure(result, context: str):
|
|
51
|
+
"""ebooklet's push() returns a PushResult; result.failures maps failed
|
|
52
|
+
keys to error strings (the pending changes are retained by ebooklet for
|
|
53
|
+
retry). A partial failure must never pass silently: publish/deregister
|
|
54
|
+
would claim success while the remote is not fully updated."""
|
|
55
|
+
if result.failures:
|
|
56
|
+
msg = (
|
|
57
|
+
f'{context}: the push partially failed and the remote was NOT fully updated. '
|
|
58
|
+
f'Failed keys: {result.failures}. The pending changes are retained - fix the '
|
|
59
|
+
'cause (see the ebooklet operations guide, docs/ops.md, for the recovery '
|
|
60
|
+
'recipes) and retry.'
|
|
61
|
+
)
|
|
62
|
+
raise RuntimeError(msg)
|
|
55
63
|
|
|
56
64
|
|
|
57
65
|
def _public_rcg_url() -> str | None:
|
|
@@ -200,12 +208,6 @@ def _bbox_within_radius(stored, lon: float, lat: float, radius_km: float) -> boo
|
|
|
200
208
|
# Dataset validation + State Metadata extraction
|
|
201
209
|
|
|
202
210
|
|
|
203
|
-
def _sys_dataset_type(ds) -> str:
|
|
204
|
-
# cfdb 0.9.0 has no public dataset_type property (and open_edataset always
|
|
205
|
-
# returns the EGrid class), so the sys-metadata slot is the reliable source.
|
|
206
|
-
return ds._sys_meta.dataset_type.value
|
|
207
|
-
|
|
208
|
-
|
|
209
211
|
def _coord_by_axis(ds, axis: str):
|
|
210
212
|
for coord in ds.coords:
|
|
211
213
|
ax = coord.axis
|
|
@@ -227,7 +229,7 @@ def _reproject_points(points, crs) -> list:
|
|
|
227
229
|
|
|
228
230
|
def _extract_state(ds, meta: Metadata) -> dict:
|
|
229
231
|
"""Extract State Metadata (bbox, time range, steps, dataset_type) from an open dataset."""
|
|
230
|
-
dataset_type =
|
|
232
|
+
dataset_type = ds.dataset_type
|
|
231
233
|
crs = ds.crs
|
|
232
234
|
|
|
233
235
|
time_coord = ds.get('time') if 'time' in ds.coord_names else None
|
|
@@ -368,7 +370,7 @@ def _decide_standard_name(ds, meta: Metadata) -> dict:
|
|
|
368
370
|
|
|
369
371
|
|
|
370
372
|
def _validate_dataset(ds, *, validate_cv: bool) -> dict:
|
|
371
|
-
"""Validate an open cfdb
|
|
373
|
+
"""Validate an open cfdb file against envlib requirements; extract everything.
|
|
372
374
|
|
|
373
375
|
Returns {'metadata', 'dataset_version_id', 'dataset_id', 'state', 'standard_name'}.
|
|
374
376
|
Raises ValidationError on any failure. Never modifies the dataset.
|
|
@@ -402,7 +404,7 @@ def _validate_dataset(ds, *, validate_cv: bool) -> dict:
|
|
|
402
404
|
msg = f'declared ancillary variable {name!r} is not a data variable in the dataset.'
|
|
403
405
|
raise ValidationError(msg)
|
|
404
406
|
|
|
405
|
-
if
|
|
407
|
+
if ds.dataset_type == 'ts_ortho':
|
|
406
408
|
_check_stations(ds)
|
|
407
409
|
|
|
408
410
|
state = _extract_state(ds, meta)
|
|
@@ -443,7 +445,7 @@ def _apply_derived_attrs(ds, result: dict) -> bool:
|
|
|
443
445
|
|
|
444
446
|
|
|
445
447
|
class DatasetRef:
|
|
446
|
-
"""A catalogue entry: metadata plus how to open
|
|
448
|
+
"""A catalogue entry: one dataset version's metadata, plus how to open its cfdb file."""
|
|
447
449
|
|
|
448
450
|
def __init__(self, dataset_version_id: str, entry: dict, cache_dir: pathlib.Path):
|
|
449
451
|
self._dataset_id = dataset_version_id
|
|
@@ -473,7 +475,7 @@ class DatasetRef:
|
|
|
473
475
|
return repr(self.metadata)
|
|
474
476
|
|
|
475
477
|
def open(self, file_path=None, access_key_id=None, access_key=None):
|
|
476
|
-
"""Open
|
|
478
|
+
"""Open this dataset version's cfdb file as a read-only EDataset.
|
|
477
479
|
|
|
478
480
|
Entries never store credentials. Public-HTTPS datasets open via their
|
|
479
481
|
``data_url`` with no credentials; for private buckets inject
|
|
@@ -516,6 +518,13 @@ class Catalogue:
|
|
|
516
518
|
to re-pull after new registrations. When a remote is unreachable and a
|
|
517
519
|
previously pulled local index exists, the catalogue degrades to the cached
|
|
518
520
|
copy (with a warning) instead of failing.
|
|
521
|
+
|
|
522
|
+
Example:
|
|
523
|
+
>>> cat = Catalogue(remotes=['https://s3.example.com/bucket/catalogue.rcg'])
|
|
524
|
+
>>> cat.variables
|
|
525
|
+
['streamflow', 'temperature']
|
|
526
|
+
>>> refs = cat.query(variable='temperature', feature='atmosphere')
|
|
527
|
+
>>> ds = refs[0].open()
|
|
519
528
|
"""
|
|
520
529
|
|
|
521
530
|
def __init__(self, remotes=None, cache: str = DEFAULT_CACHE_DIR, *, include_public: bool = False):
|
|
@@ -527,7 +536,7 @@ class Catalogue:
|
|
|
527
536
|
public = _public_rcg_url()
|
|
528
537
|
if public is None:
|
|
529
538
|
msg = (
|
|
530
|
-
'
|
|
539
|
+
'no public envlib RCG is configured — pass remotes=[...] '
|
|
531
540
|
f'or set the {PUBLIC_RCG_ENV_VAR} environment variable.'
|
|
532
541
|
)
|
|
533
542
|
raise ValueError(msg)
|
|
@@ -558,23 +567,34 @@ class Catalogue:
|
|
|
558
567
|
for source in self._sources:
|
|
559
568
|
path = self._rcg_cache_path(source)
|
|
560
569
|
try:
|
|
561
|
-
|
|
570
|
+
# offline='auto': ebooklet falls back to the cached local index
|
|
571
|
+
# (with its own UserWarning) ONLY on transport-level
|
|
572
|
+
# unreachability. Everything else raises typed: a broken store
|
|
573
|
+
# (RemoteIntegrityError), an incompatible format
|
|
574
|
+
# (UnsupportedFormatError - pre-0.10 this was silently
|
|
575
|
+
# mislabeled "not readable yet" by a blanket ValueError catch),
|
|
576
|
+
# and unreachable-with-no-cache (OfflineError) all propagate.
|
|
577
|
+
with ebooklet.open_rcg(source, path, flag='r', offline='auto') as rcg:
|
|
562
578
|
source_entries = {k: v for k, v in rcg.items() if _HEX24_RE.fullmatch(str(k))}
|
|
563
|
-
except
|
|
564
|
-
#
|
|
565
|
-
# remote
|
|
566
|
-
#
|
|
567
|
-
#
|
|
568
|
-
warnings.warn(f'RCG source not readable yet ({err}); treating as empty.', stacklevel=2)
|
|
569
|
-
source_entries = {}
|
|
570
|
-
except _OFFLINE_ERRORS as err:
|
|
571
|
-
if not path.exists():
|
|
572
|
-
raise
|
|
579
|
+
except ebooklet.UUIDMismatchError as err:
|
|
580
|
+
# The local cache identifies a DIFFERENT database than the
|
|
581
|
+
# remote - the remote was likely deleted and recreated.
|
|
582
|
+
# Distinct from the bootstrap case: the fix is deleting the
|
|
583
|
+
# stale cache file, so say so instead of a generic warning.
|
|
573
584
|
warnings.warn(
|
|
574
|
-
f'RCG
|
|
585
|
+
f'RCG source identity mismatch ({err}); the remote was likely deleted and '
|
|
586
|
+
f'recreated. Delete the stale cache at {path} to adopt the new remote. '
|
|
587
|
+
'Treating this source as empty for now.',
|
|
575
588
|
stacklevel=2,
|
|
576
589
|
)
|
|
577
|
-
source_entries =
|
|
590
|
+
source_entries = {}
|
|
591
|
+
except ebooklet.RemoteMissingError as err:
|
|
592
|
+
# The RCG does not exist on the remote yet (and no local copy
|
|
593
|
+
# exists) — the bootstrap case: a producer constructs the
|
|
594
|
+
# Catalogue before the first publish creates the RCG. Treat as
|
|
595
|
+
# an empty source, loudly.
|
|
596
|
+
warnings.warn(f'RCG source not readable yet ({err}); treating as empty.', stacklevel=2)
|
|
597
|
+
source_entries = {}
|
|
578
598
|
for key, entry in source_entries.items():
|
|
579
599
|
entries.setdefault(key, entry) # first configured source wins on duplicates
|
|
580
600
|
self._entries = entries
|
|
@@ -770,13 +790,13 @@ class Catalogue:
|
|
|
770
790
|
first_time = eds.attrs.get('envlib_dataset_version_id') is None
|
|
771
791
|
result = _validate_dataset(eds, validate_cv=first_time)
|
|
772
792
|
_apply_derived_attrs(eds, result)
|
|
773
|
-
eds.push()
|
|
793
|
+
_raise_on_push_failure(eds.push(), 'publish: pushing the dataset data')
|
|
774
794
|
|
|
775
795
|
self._upsert_entry(rcg_remote_conn, member_conn, result)
|
|
776
796
|
return result
|
|
777
797
|
|
|
778
798
|
def register(self, remote_conn, rcg_remote_conn, **open_kwargs) -> dict:
|
|
779
|
-
"""Register an already-remote cfdb
|
|
799
|
+
"""Register an already-remote cfdb file in the catalogue (no data push).
|
|
780
800
|
|
|
781
801
|
``remote_conn`` must be writable (credentials): first registration
|
|
782
802
|
writes the self-identification attrs (and any auto-populated
|
|
@@ -788,7 +808,7 @@ class Catalogue:
|
|
|
788
808
|
first_time = eds.attrs.get('envlib_dataset_version_id') is None
|
|
789
809
|
result = _validate_dataset(eds, validate_cv=first_time)
|
|
790
810
|
if _apply_derived_attrs(eds, result):
|
|
791
|
-
eds.push()
|
|
811
|
+
_raise_on_push_failure(eds.push(), 'register: pushing the self-identification attrs')
|
|
792
812
|
|
|
793
813
|
self._upsert_entry(rcg_remote_conn, member_conn, result)
|
|
794
814
|
return result
|
|
@@ -820,8 +840,7 @@ class Catalogue:
|
|
|
820
840
|
target_conn = entry.get('remote_conn') or {}
|
|
821
841
|
target = (target_conn.get('endpoint_url'), target_conn.get('bucket'), target_conn.get('db_key'))
|
|
822
842
|
# list() first: fetching entries while iterating keys() would
|
|
823
|
-
# deadlock on the underlying booklet thread lock
|
|
824
|
-
# _read_cached_index).
|
|
843
|
+
# deadlock on the underlying booklet thread lock.
|
|
825
844
|
for other_key in list(rcg.keys()):
|
|
826
845
|
if other_key == dataset_version_id or not _HEX24_RE.fullmatch(str(other_key)):
|
|
827
846
|
continue
|
|
@@ -847,7 +866,7 @@ class Catalogue:
|
|
|
847
866
|
with member_conn.open('w') as session:
|
|
848
867
|
session.delete_remote()
|
|
849
868
|
del rcg[dataset_version_id]
|
|
850
|
-
rcg.changes().push()
|
|
869
|
+
_raise_on_push_failure(rcg.changes().push(), 'deregister: pushing the catalogue entry removal')
|
|
851
870
|
self.refresh()
|
|
852
871
|
|
|
853
872
|
# -- entry construction ----------------------------------------------------
|
|
@@ -871,7 +890,7 @@ class Catalogue:
|
|
|
871
890
|
|
|
872
891
|
user_meta['modified_at'] = now
|
|
873
892
|
rcg.add(member_conn, key=dataset_version_id, user_meta=user_meta)
|
|
874
|
-
rcg.changes().push()
|
|
893
|
+
_raise_on_push_failure(rcg.changes().push(), 'publish/register: pushing the catalogue entry')
|
|
875
894
|
self.refresh()
|
|
876
895
|
|
|
877
896
|
|
|
@@ -937,34 +956,3 @@ def _created_at(ref: DatasetRef) -> datetime.datetime:
|
|
|
937
956
|
if value is None:
|
|
938
957
|
return datetime.datetime.min.replace(tzinfo=datetime.timezone.utc)
|
|
939
958
|
return _parse_iso(value)
|
|
940
|
-
|
|
941
|
-
|
|
942
|
-
def _read_cached_index(path: pathlib.Path) -> dict:
|
|
943
|
-
"""Offline fallback: read a previously pulled RCG index directly with booklet.
|
|
944
|
-
|
|
945
|
-
Booklet files are self-describing (serializers stored in-file), so entries
|
|
946
|
-
decode without parameters. envlib entries are keyed by 24-hex dataset_ids;
|
|
947
|
-
everything else (ebooklet-internal keys) is skipped. Values that fail to
|
|
948
|
-
decode as entry dicts are skipped too.
|
|
949
|
-
"""
|
|
950
|
-
entries: dict = {}
|
|
951
|
-
with booklet.open(path, 'r') as blt:
|
|
952
|
-
# materialize keys() BEFORE fetching: booklet's keys() generator holds
|
|
953
|
-
# the file's thread lock across yields, so a get() inside the loop
|
|
954
|
-
# deadlocks (single-threaded) — found via the hung offline test.
|
|
955
|
-
for key in list(blt.keys()):
|
|
956
|
-
key_str = key.decode() if isinstance(key, bytes) else str(key)
|
|
957
|
-
if not _HEX24_RE.fullmatch(key_str):
|
|
958
|
-
continue
|
|
959
|
-
try:
|
|
960
|
-
value = blt[key]
|
|
961
|
-
except (KeyError, ValueError):
|
|
962
|
-
continue
|
|
963
|
-
if isinstance(value, bytes):
|
|
964
|
-
try:
|
|
965
|
-
value = json.loads(value)
|
|
966
|
-
except ValueError:
|
|
967
|
-
continue
|
|
968
|
-
if isinstance(value, dict) and 'remote_conn' in value:
|
|
969
|
-
entries[key_str] = value
|
|
970
|
-
return entries
|
|
@@ -12,6 +12,7 @@ from __future__ import annotations
|
|
|
12
12
|
|
|
13
13
|
import math
|
|
14
14
|
import re
|
|
15
|
+
from collections.abc import Mapping
|
|
15
16
|
from hashlib import blake2b
|
|
16
17
|
from typing import cast
|
|
17
18
|
|
|
@@ -503,7 +504,7 @@ class Metadata:
|
|
|
503
504
|
return out
|
|
504
505
|
|
|
505
506
|
@classmethod
|
|
506
|
-
def from_attrs(cls, attrs, *, validate_cv: bool = True) -> Metadata:
|
|
507
|
+
def from_attrs(cls, attrs: Mapping, *, validate_cv: bool = True) -> Metadata:
|
|
507
508
|
"""Rebuild Metadata from ``envlib_``-prefixed attrs (``ds.attrs`` or a dict).
|
|
508
509
|
|
|
509
510
|
Non-envlib keys are ignored. When ``envlib_dataset_version_id`` /
|
|
@@ -293,7 +293,7 @@ def _refresh_variable(target_dir: pathlib.Path) -> dict:
|
|
|
293
293
|
}
|
|
294
294
|
|
|
295
295
|
|
|
296
|
-
def refresh(field: str | None = None, _target_dir=None) -> dict:
|
|
296
|
+
def refresh(field: str | None = None, _target_dir: str | pathlib.Path | None = None) -> dict:
|
|
297
297
|
"""Refresh upstream-sourced vocabularies from their APIs.
|
|
298
298
|
|
|
299
299
|
Fetches the ODM2 variablename list and/or the current CF standard-name table
|
|
@@ -11,9 +11,9 @@ classifiers = [
|
|
|
11
11
|
]
|
|
12
12
|
requires-python = ">=3.10"
|
|
13
13
|
dependencies = [
|
|
14
|
-
"cfdb>=0.9.
|
|
15
|
-
"ebooklet>=0.
|
|
16
|
-
"booklet>=0.12.
|
|
14
|
+
"cfdb>=0.9.1",
|
|
15
|
+
"ebooklet>=0.10.0",
|
|
16
|
+
"booklet>=0.12.6",
|
|
17
17
|
"pyproj",
|
|
18
18
|
"shapely>=2.0",
|
|
19
19
|
"urllib3",
|
|
@@ -31,6 +31,7 @@ dev = [
|
|
|
31
31
|
"mkdocstrings[python]",
|
|
32
32
|
"types-shapely>=2.1.0.20260630",
|
|
33
33
|
"numpy>=2.2.6",
|
|
34
|
+
"tomli>=2.4.1 ; python_full_version < '3.11'",
|
|
34
35
|
]
|
|
35
36
|
|
|
36
37
|
[tool.pytest.ini_options]
|
|
@@ -130,8 +131,8 @@ requires = ["hatchling>=1.26.1"]
|
|
|
130
131
|
build-backend = "hatchling.build"
|
|
131
132
|
|
|
132
133
|
[project.urls]
|
|
133
|
-
Documentation = "https://
|
|
134
|
-
Source = "https://github.com/envlib"
|
|
134
|
+
Documentation = "https://envlib.github.io/envlib/"
|
|
135
|
+
Source = "https://github.com/envlib/envlib"
|
|
135
136
|
|
|
136
137
|
[tool.hatch.build.targets.sdist]
|
|
137
138
|
include = [
|
envlib-0.1.0/PKG-INFO
DELETED
|
@@ -1,75 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: envlib
|
|
3
|
-
Version: 0.1.0
|
|
4
|
-
Summary: The environmental library
|
|
5
|
-
Project-URL: Documentation, https://mullenkamp.github.io/envlib/
|
|
6
|
-
Project-URL: Source, https://github.com/envlib
|
|
7
|
-
Author-email: mullenkamp <mullenkamp1@gmail.com>
|
|
8
|
-
License-File: LICENSE
|
|
9
|
-
Classifier: Programming Language :: Python :: 3 :: Only
|
|
10
|
-
Requires-Python: >=3.10
|
|
11
|
-
Requires-Dist: booklet>=0.12.5
|
|
12
|
-
Requires-Dist: cfdb>=0.9.0
|
|
13
|
-
Requires-Dist: ebooklet>=0.9.1
|
|
14
|
-
Requires-Dist: pyproj
|
|
15
|
-
Requires-Dist: shapely>=2.0
|
|
16
|
-
Requires-Dist: urllib3
|
|
17
|
-
Description-Content-Type: text/markdown
|
|
18
|
-
|
|
19
|
-
# envlib
|
|
20
|
-
|
|
21
|
-
<p align="center">
|
|
22
|
-
<em>The environmental library</em>
|
|
23
|
-
</p>
|
|
24
|
-
|
|
25
|
-
[](https://github.com/envlib/actions)
|
|
26
|
-
[](https://codecov.io/gh/mullenkamp/envlib)
|
|
27
|
-
[](https://badge.fury.io/py/envlib)
|
|
28
|
-
|
|
29
|
-
---
|
|
30
|
-
|
|
31
|
-
**Source Code**: <a href="https://github.com/envlib" target="_blank">https://github.com/envlib</a>
|
|
32
|
-
|
|
33
|
-
---
|
|
34
|
-
## Overview
|
|
35
|
-
envlib is a distributed database and catalogue for environmental data. It uses controlled vocabulary and standarized metadata to make it easy for users to query and access data. It uses cfdb as the backend. The metadata structure is based on tethysts.
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
## Development
|
|
39
|
-
|
|
40
|
-
### Setup environment
|
|
41
|
-
|
|
42
|
-
We use [UV](https://docs.astral.sh/uv/) to manage the development environment and production build.
|
|
43
|
-
|
|
44
|
-
```bash
|
|
45
|
-
uv sync
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
### Run unit tests
|
|
49
|
-
|
|
50
|
-
You can run all the tests with:
|
|
51
|
-
|
|
52
|
-
```bash
|
|
53
|
-
uv run pytest
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
### Format the code
|
|
57
|
-
|
|
58
|
-
Execute the following commands to apply linting and check typing:
|
|
59
|
-
|
|
60
|
-
```bash
|
|
61
|
-
uv run ruff check .
|
|
62
|
-
uv run black --check --diff .
|
|
63
|
-
uv run mypy --install-types --non-interactive envlib
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
To auto-format:
|
|
67
|
-
|
|
68
|
-
```bash
|
|
69
|
-
uv run black .
|
|
70
|
-
uv run ruff check --fix .
|
|
71
|
-
```
|
|
72
|
-
|
|
73
|
-
## License
|
|
74
|
-
|
|
75
|
-
This project is licensed under the terms of the Apache Software License 2.0.
|
envlib-0.1.0/README.md
DELETED
|
@@ -1,57 +0,0 @@
|
|
|
1
|
-
# envlib
|
|
2
|
-
|
|
3
|
-
<p align="center">
|
|
4
|
-
<em>The environmental library</em>
|
|
5
|
-
</p>
|
|
6
|
-
|
|
7
|
-
[](https://github.com/envlib/actions)
|
|
8
|
-
[](https://codecov.io/gh/mullenkamp/envlib)
|
|
9
|
-
[](https://badge.fury.io/py/envlib)
|
|
10
|
-
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
**Source Code**: <a href="https://github.com/envlib" target="_blank">https://github.com/envlib</a>
|
|
14
|
-
|
|
15
|
-
---
|
|
16
|
-
## Overview
|
|
17
|
-
envlib is a distributed database and catalogue for environmental data. It uses controlled vocabulary and standarized metadata to make it easy for users to query and access data. It uses cfdb as the backend. The metadata structure is based on tethysts.
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
## Development
|
|
21
|
-
|
|
22
|
-
### Setup environment
|
|
23
|
-
|
|
24
|
-
We use [UV](https://docs.astral.sh/uv/) to manage the development environment and production build.
|
|
25
|
-
|
|
26
|
-
```bash
|
|
27
|
-
uv sync
|
|
28
|
-
```
|
|
29
|
-
|
|
30
|
-
### Run unit tests
|
|
31
|
-
|
|
32
|
-
You can run all the tests with:
|
|
33
|
-
|
|
34
|
-
```bash
|
|
35
|
-
uv run pytest
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
### Format the code
|
|
39
|
-
|
|
40
|
-
Execute the following commands to apply linting and check typing:
|
|
41
|
-
|
|
42
|
-
```bash
|
|
43
|
-
uv run ruff check .
|
|
44
|
-
uv run black --check --diff .
|
|
45
|
-
uv run mypy --install-types --non-interactive envlib
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
To auto-format:
|
|
49
|
-
|
|
50
|
-
```bash
|
|
51
|
-
uv run black .
|
|
52
|
-
uv run ruff check --fix .
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
## License
|
|
56
|
-
|
|
57
|
-
This project is licensed under the terms of the Apache Software License 2.0.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|