usdata 0.14.0__tar.gz → 0.16.0__tar.gz

usdata-0.16.0/PKG-INFO

@@ -0,0 +1,122 @@
+Metadata-Version: 2.4
+Name: usdata
+Version: 0.16.0
+Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
+Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
+Author: Jake Van Slyke
+Author-email: Jake Van Slyke <jakervanslyke@gmail.com>
+License-Expression: Apache-2.0
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: pydantic>=2.7
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: typer>=0.18
+Requires-Dist: eccodes>=2.48 ; extra == 'grib'
+Requires-Dist: eccodeslib>=2.48 ; sys_platform == 'linux' and extra == 'grib'
+Requires-Dist: xarray>=2025.1 ; extra == 'grib'
+Requires-Dist: numpy>=2 ; extra == 'grib'
+Requires-Dist: xarray>=2025.1 ; extra == 'netcdf'
+Requires-Dist: h5netcdf[h5py]>=1.8.1 ; extra == 'netcdf'
+Requires-Dist: pandas>=2.2 ; extra == 'pandas'
+Requires-Dist: xradar>=0.12.0 ; extra == 'radar'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://usdata.dev/
+Project-URL: Documentation, https://docs.usdata.dev/
+Project-URL: Repository, https://github.com/jakeryderv/usdata
+Provides-Extra: grib
+Provides-Extra: netcdf
+Provides-Extra: pandas
+Provides-Extra: radar
+Description-Content-Type: text/markdown
+
+# usdata
+
+Discover U.S. scientific datasets, fetch their files, and keep a reproducible
+record of where every input came from. One Python SDK and CLI across supported
+NOAA and USGS datasets, with provenance sidecars and lockable manifests.
+
+**Pre-alpha.** Features marked **Unreleased** in the documentation require a
+source installation; the [changelog](CHANGELOG.md) lists what each published
+version contains.
+
+## Documentation
+
+| Site | For |
+| --- | --- |
+| [usdata.dev](https://usdata.dev/) | What the package is for: the dataset browser and worked examples with saved results |
+| [docs.usdata.dev](https://docs.usdata.dev/) | Using the package: install, first dataset, guides, dataset notes, and reference |
+| This repository | Changing the project: everything below, plus [contributing](CONTRIBUTING.md), [architecture](docs/architecture.md), [testing](docs/testing.md), [decisions](docs/adr/README.md), [roadmap](docs/roadmap.md), and [versioning](docs/versioning.md) |
+
+```sh
+pip install "usdata[pandas]"
+usdata search precipitation --location Oklahoma
+usdata info noaa:ghcn-daily
+```
+
+## Development
+
+Requires a source checkout, [uv](https://docs.astral.sh/uv/), and
+[just](https://just.systems/):
+
+```sh
+git clone https://github.com/jakeryderv/usdata && cd usdata
+just setup     # install toolchain and dependencies
+just check     # format, lint, typecheck, offline tests, generated docs, release notices
+```
+
+`just setup` uses the tested Python 3.14.7 pin in `.python-version`. Older Linux
+uv Python 3.14 builds can crash during NumPy array operations; see
+[the upstream fix](https://github.com/astral-sh/python-build-standalone/issues/991).
+
+```sh
+just hooks-install # optional: install fast staged-file checks
+just test      # all offline tests
+just check-pandas  # install the CSV extra and run the same checks
+just check-radar   # install the radar extra and run the same checks
+just check-netcdf  # install the NetCDF4 extra and run the same checks
+just check-grib    # install the GRIB2 extra and run the same checks (Linux, Windows <= 3.13)
+just notebooks    # launch the optional Jupyter examples environment
+just run-notebooks # execute notebooks live in fresh kernels and temporary caches
+just docs-serve # build and preview the documentation locally, with reload
+just check-docs # validate generated content and build the site strictly
+just build     # build wheel and sdist
+just smoke     # exercise core and pandas wheel installations outside the checkout
+just changes   # preview upcoming release notes
+just cleanup 123 # from main: remove PR #123's merged branch and clean worktree
+just run search radar
+```
+
+Commit hooks fix and format staged Python files using the locked Ruff version,
+and check configuration, merge conflicts, and GitHub Actions workflows. Full
+validation remains `just check`; `just check-hooks` runs the non-mutating config
+and workflow checks alone. Hooks do not rewrite notebooks or raw data fixtures.
+The first hook run downloads its isolated tools and requires network access.
+
+Offline tests mechanically block network connections. Tests that hit live
+services run with `just test-live`; see [testing levels and organization](docs/testing.md).
+CI checks Python 3.11 and 3.14 on Linux with core-only, pandas, radar, NetCDF,
+and GRIB2 dependency profiles. A further Python 3.11 profile replaces the locked
+pandas with the oldest release the project declares, so the `>=` floor is
+executed rather than assumed; the number comes from `pyproject.toml` through
+`scripts/lowest_version.py`. Installed-wheel checks cover core, pandas, radar,
+and NetCDF on Linux, macOS, and Windows. The full offline and live-service
+suites run on Linux. `just setup` restores a core-only development environment;
+the `check-*` commands install their respective extras.
+
+Releases: `just release minor` prepares a release branch. Update release
+notices, then `just release-pr` validates and opens a draft for review. Merging
+the reviewed PR publishes to PyPI and creates the tag and GitHub release. See
+[versioning](docs/versioning.md).
+
+The main way the project grows is a new dataset: one registry entry and one
+adapter, following [adding a dataset](docs/guides/adding-a-dataset.md). The two
+websites deploy automatically from `main`; see
+[website operations](docs/guides/website-operations.md) and
+[maintaining documentation](docs/guides/documentation.md).
+
+## License
+
+[Apache-2.0](LICENSE).

usdata-0.16.0/README.md

@@ -0,0 +1,88 @@
+# usdata
+
+Discover U.S. scientific datasets, fetch their files, and keep a reproducible
+record of where every input came from. One Python SDK and CLI across supported
+NOAA and USGS datasets, with provenance sidecars and lockable manifests.
+
+**Pre-alpha.** Features marked **Unreleased** in the documentation require a
+source installation; the [changelog](CHANGELOG.md) lists what each published
+version contains.
+
+## Documentation
+
+| Site | For |
+| --- | --- |
+| [usdata.dev](https://usdata.dev/) | What the package is for: the dataset browser and worked examples with saved results |
+| [docs.usdata.dev](https://docs.usdata.dev/) | Using the package: install, first dataset, guides, dataset notes, and reference |
+| This repository | Changing the project: everything below, plus [contributing](CONTRIBUTING.md), [architecture](docs/architecture.md), [testing](docs/testing.md), [decisions](docs/adr/README.md), [roadmap](docs/roadmap.md), and [versioning](docs/versioning.md) |
+
+```sh
+pip install "usdata[pandas]"
+usdata search precipitation --location Oklahoma
+usdata info noaa:ghcn-daily
+```
+
+## Development
+
+Requires a source checkout, [uv](https://docs.astral.sh/uv/), and
+[just](https://just.systems/):
+
+```sh
+git clone https://github.com/jakeryderv/usdata && cd usdata
+just setup     # install toolchain and dependencies
+just check     # format, lint, typecheck, offline tests, generated docs, release notices
+```
+
+`just setup` uses the tested Python 3.14.7 pin in `.python-version`. Older Linux
+uv Python 3.14 builds can crash during NumPy array operations; see
+[the upstream fix](https://github.com/astral-sh/python-build-standalone/issues/991).
+
+```sh
+just hooks-install # optional: install fast staged-file checks
+just test      # all offline tests
+just check-pandas  # install the CSV extra and run the same checks
+just check-radar   # install the radar extra and run the same checks
+just check-netcdf  # install the NetCDF4 extra and run the same checks
+just check-grib    # install the GRIB2 extra and run the same checks (Linux, Windows <= 3.13)
+just notebooks    # launch the optional Jupyter examples environment
+just run-notebooks # execute notebooks live in fresh kernels and temporary caches
+just docs-serve # build and preview the documentation locally, with reload
+just check-docs # validate generated content and build the site strictly
+just build     # build wheel and sdist
+just smoke     # exercise core and pandas wheel installations outside the checkout
+just changes   # preview upcoming release notes
+just cleanup 123 # from main: remove PR #123's merged branch and clean worktree
+just run search radar
+```
+
+Commit hooks fix and format staged Python files using the locked Ruff version,
+and check configuration, merge conflicts, and GitHub Actions workflows. Full
+validation remains `just check`; `just check-hooks` runs the non-mutating config
+and workflow checks alone. Hooks do not rewrite notebooks or raw data fixtures.
+The first hook run downloads its isolated tools and requires network access.
+
+Offline tests mechanically block network connections. Tests that hit live
+services run with `just test-live`; see [testing levels and organization](docs/testing.md).
+CI checks Python 3.11 and 3.14 on Linux with core-only, pandas, radar, NetCDF,
+and GRIB2 dependency profiles. A further Python 3.11 profile replaces the locked
+pandas with the oldest release the project declares, so the `>=` floor is
+executed rather than assumed; the number comes from `pyproject.toml` through
+`scripts/lowest_version.py`. Installed-wheel checks cover core, pandas, radar,
+and NetCDF on Linux, macOS, and Windows. The full offline and live-service
+suites run on Linux. `just setup` restores a core-only development environment;
+the `check-*` commands install their respective extras.
+
+Releases: `just release minor` prepares a release branch. Update release
+notices, then `just release-pr` validates and opens a draft for review. Merging
+the reviewed PR publishes to PyPI and creates the tag and GitHub release. See
+[versioning](docs/versioning.md).
+
+The main way the project grows is a new dataset: one registry entry and one
+adapter, following [adding a dataset](docs/guides/adding-a-dataset.md). The two
+websites deploy automatically from `main`; see
+[website operations](docs/guides/website-operations.md) and
+[maintaining documentation](docs/guides/documentation.md).
+
+## License
+
+[Apache-2.0](LICENSE).

{usdata-0.14.0 → usdata-0.16.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.14.0"
+version = "0.16.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -37,6 +37,12 @@ netcdf = [
     "xarray>=2025.1",
     "h5netcdf[h5py]>=1.8.1",
 ]
+grib = [
+    "eccodes>=2.48",
+    "eccodeslib>=2.48; sys_platform == 'linux'",
+    "xarray>=2025.1",
+    "numpy>=2",
+]
 
 [project.urls]
 Homepage = "https://usdata.dev/"
@@ -129,6 +135,7 @@ markers = [
     "pandas: optional CSV reader coverage",
     "radar: optional radar reader coverage",
     "netcdf: optional NetCDF reader coverage",
+    "grib: optional GRIB2 reader coverage",
 ]
 addopts = "-ra --strict-markers"
 

{usdata-0.14.0 → usdata-0.16.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.14.0"
+version = "0.16.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -26,6 +26,13 @@ dependencies = [
 pandas = ["pandas>=2.2"]
 radar = ["xradar>=0.12.0"]
 netcdf = ["xarray>=2025.1", "h5netcdf[h5py]>=1.8.1"]
+grib = [
+    "eccodes>=2.48",
+    # The eccodes sdist metadata omits the library wheel that its own wheel declares.
+    "eccodeslib>=2.48; sys_platform == 'linux'",
+    "xarray>=2025.1",
+    "numpy>=2",
+]
 
 [project.urls]
 Homepage = "https://usdata.dev/"
@@ -101,6 +108,7 @@ markers = [
     "pandas: optional CSV reader coverage",
     "radar: optional radar reader coverage",
     "netcdf: optional NetCDF reader coverage",
+    "grib: optional GRIB2 reader coverage",
 ]
 addopts = "-ra --strict-markers"
 

{usdata-0.14.0 → usdata-0.16.0}/src/usdata/__init__.py

@@ -10,6 +10,7 @@ try:
 except PackageNotFoundError:  # running from a source tree without an install
     __version__ = "0.0.0"
 
+from usdata._fetch import ChecksumMismatch, FetchedAsset, fetch, fetch_asset
 from usdata.models import Asset, BBox, Dataset, Provenance, Query, TemporalSelection, TimeRange
 from usdata.pull import pull, verify
 from usdata.query import build_query
@@ -19,8 +20,10 @@ from usdata.selection import select_by_time
 __all__ = [
     "Asset",
     "BBox",
+    "ChecksumMismatch",
     "Dataset",
     "DatasetNotFound",
+    "FetchedAsset",
     "Provenance",
     "Query",
     "Registry",
@@ -30,6 +33,8 @@ __all__ = [
     "__version__",
     "build_query",
     "default_registry",
+    "fetch",
+    "fetch_asset",
     "get",
     "pull",
     "search",

usdata-0.14.0/src/usdata/fetch.py → usdata-0.16.0/src/usdata/_fetch.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+from collections.abc import Mapping
 from pathlib import Path
 from typing import Any
 
@@ -35,15 +36,17 @@ class FetchedAsset(BaseModel):
         usecols: list[str] | None = None,
         nrows: int | None = None,
         sweep: int | list[int] | None = None,
+        select: Mapping[str, Any] | None = None,
     ) -> Any:
-        """Open local data with an optional ``pandas``, ``radar``, or ``netcdf`` reader.
+        """Open local data with an optional ``pandas``, ``radar``, ``netcdf``, or ``grib`` reader.
 
         ERDDAP units are kept in ``frame.attrs["units"]`` and source provenance
         in ``frame.attrs["usdata"]``. NEXRAD returns a xarray DataTree with provenance
-        in ``radar.attrs["usdata"]``. NetCDF4 returns a loaded xarray Dataset with
-        matching provenance in its attributes. See ``usdata.readers.open_asset`` for options.
-        Use ``sweep=0`` or ``sweep=[0, 2]`` to load selected zero-based radar sweeps.
-        Cached files and provenance sidecars are never changed.
+        in ``radar.attrs["usdata"]``. NetCDF4 and GRIB2 return a loaded xarray Dataset
+        with matching provenance in its attributes. See ``usdata.readers.open_asset``
+        for options. Use ``sweep=0`` or ``sweep=[0, 2]`` to load selected zero-based
+        radar sweeps, and ``select={"shortName": "cape", "typeOfLevel": "surface"}``
+        to choose GRIB2 messages. Cached files and provenance sidecars are never changed.
         """
         from usdata.readers import open_asset
 
@@ -55,9 +58,24 @@ class FetchedAsset(BaseModel):
             usecols=usecols,
             nrows=nrows,
             sweep=sweep,
+            select=select,
         )
 
 
+def _sidecar_not_older(path: Path) -> bool:
+    """Whether ``path`` still carries the mtime it had when its sidecar was written.
+
+    The sidecar is always written after the data file lands, so a data file that
+    is newer than its sidecar was touched afterwards and the record no longer
+    implies its bytes. See ADR 0024.
+    """
+    try:
+        sidecar = provenance.sidecar_path(path).stat()
+        return path.stat().st_mtime_ns <= sidecar.st_mtime_ns
+    except OSError:
+        return False
+
+
 def _fetch_asset(
     dataset: Dataset,
     asset: Asset,
@@ -82,7 +100,8 @@ def _fetch_asset(
             and prov.source_url == asset.href
             and prov.size == path.stat().st_size
             and (asset.checksum is None or prov.checksum == asset.checksum)
-            and sha256_file(path) == prov.checksum
+            # Trust an untouched cached file; hash whenever anything is unclear.
+            and (_sidecar_not_older(path) or sha256_file(path) == prov.checksum)
         ):
             _progress.emit(_progress.AssetProgress(asset.id, "cached", prov.size))
             return FetchedAsset(asset=asset, path=path, provenance=prov, from_cache=True)
@@ -100,16 +119,28 @@ def _fetch_asset(
 def fetch_asset(
     dataset: Dataset, asset: Asset, *, root: Path | None = None, force: bool = False
 ) -> FetchedAsset:
-    """Fetch one asset, reusing the cache only when bytes and provenance agree."""
+    """Fetch one asset, reusing the cache only when its provenance still describes it."""
     with load_adapter(dataset) as adapter:
         return _fetch_asset(dataset, asset, adapter, root=root, force=force)
 
 
+def _fetch_with(
+    adapter: Provider,
+    dataset: Dataset,
+    query: Query,
+    *,
+    root: Path | None = None,
+    force: bool = False,
+) -> list[FetchedAsset]:
+    """Run the loop on an adapter the caller opened, so one adapter can serve many queries."""
+    assets = adapter.list_assets(query)
+    _progress.batch([asset.size for asset in assets])
+    return [_fetch_asset(dataset, a, adapter, root=root, force=force) for a in assets]
+
+
 def fetch(
     dataset: Dataset, query: Query, *, root: Path | None = None, force: bool = False
 ) -> list[FetchedAsset]:
     """Resolve and fetch a query, sharing one adapter and closing its owned resources."""
     with load_adapter(dataset) as adapter:
-        assets = adapter.list_assets(query)
-        _progress.batch([asset.size for asset in assets])
-        return [_fetch_asset(dataset, a, adapter, root=root, force=force) for a in assets]
+        return _fetch_with(adapter, dataset, query, root=root, force=force)