usdata 0.3.0__tar.gz → 0.5.0__tar.gz

{usdata-0.3.0 → usdata-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.3.0
+Version: 0.5.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -24,24 +24,25 @@ Description-Content-Type: text/markdown
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. `noaa:ghcn-daily` and `noaa:nexrad-level2` fetch real
-> data with provenance; other registry entries are stubs.
+> Status: pre-alpha. v0.5 supports GHCN-Daily, NEXRAD Level II, USGS daily
+> values, and CoastWatch SST subsets with provenance, plus Census state/county
+> lookup. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
 
 <!-- registry:start -->
-| Provider | Available | Stub | Planned | Next up (0.3) | Datasets |
+| Provider | Available | Stub | Planned | Next up (0.6) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | `gfs`, `hrrr`, `oisst`, `etopo` | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
 | [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
-| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | `gpm-imerg` | +1 planned |
 | [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
-| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | — | +3 planned |
 
-Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
+Available datasets are in `code`, stubs in _italics_; planned ones are counted. Available means implemented in this source checkout; consult the [releases](https://github.com/jakeryderv/usdata/releases) for published support. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
 <!-- registry:end -->
 
 ## Install
@@ -74,6 +75,7 @@ for item in fetch(ds, query):
 
 ```sh
 usdata search "tornado radar" --state OK
+usdata search precipitation --location "Cleveland County, OK"
 usdata info noaa:ghcn-daily
 usdata fetch noaa:ghcn-daily --lat 35.39 --lon -97.60 --radius-km 15 \
     --start 2024-05-06 --end 2024-05-07 --vars PRCP,TMAX
@@ -81,6 +83,10 @@ usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 20
 usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
     --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
 usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
+usdata fetch usgs:water-daily -p sites=07164500 --vars 00060 \
+    --start 2024-05-06 --end 2024-05-07
+usdata fetch noaa:coastwatch-sst --bbox=-80.08,30.02,-80.02,30.08 \
+    --start 2024-05-06T12:00Z --end 2024-05-06T12:00Z    # four grid cells
 usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
 usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
@@ -89,12 +95,27 @@ Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
+Locations accept state names/postal codes, county/state names, and quoted FIPS
+codes. These select bounding rectangles; see [place lookup](docs/reference/places.md)
+for coverage, ambiguity, and antimeridian limits. CoastWatch CSV includes a
+second header row containing units; see its [access notes](docs/providers/noaa.md#coastwatch-sst).
+
+Manifest and source fields are validated strictly; unknown fields are errors.
+Provider-specific options belong under `params`.
+
 A manifest declares every input a project needs. `pull` resolves each source,
 fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
 and provenance. A second `pull` restores exactly what the lockfile pins without
 re-querying upstream, so the inputs stay reproducible even if the source
-changes. `verify` re-hashes the cached files against the lockfile. Editing the
-manifest after locking requires `pull --force` to re-resolve.
+changes. `verify` checks the manifest checksum and re-hashes cached files
+against the lockfile. Editing the manifest after locking requires `pull --force`
+to re-resolve. A required source matching no assets fails the pull; set
+`allow_empty: true` on a source only when an empty result is intentional.
+
+Checksums detect upstream changes; they cannot recover historical bytes that
+are no longer available. Preserve the cache for long-lived reproducibility.
+See the [manifest reference](docs/reference/manifests.md) and the small
+[NOAA/USGS example](examples/weather-and-streamflow/README.md).
 
 ```yaml
 name: tornado-environment
@@ -117,11 +138,16 @@ Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
-just check     # format, lint, typecheck, tests (what CI runs)
+just check     # format, lint, typecheck, offline tests, generated docs
+just build     # build wheel and sdist
+just smoke     # install and exercise the built wheel outside the checkout
 just run search radar
 ```
 
-Integration tests that hit live services run with `just test-integration`.
+Unit tests mechanically block network connections. Integration tests that hit
+live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
+Linux and smoke-tests the installed wheel on Linux, macOS, and Windows. The
+full unit and live-service suites currently run on Linux.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.3.0 → usdata-0.5.0}/README.md

@@ -3,24 +3,25 @@
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. `noaa:ghcn-daily` and `noaa:nexrad-level2` fetch real
-> data with provenance; other registry entries are stubs.
+> Status: pre-alpha. v0.5 supports GHCN-Daily, NEXRAD Level II, USGS daily
+> values, and CoastWatch SST subsets with provenance, plus Census state/county
+> lookup. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
 
 <!-- registry:start -->
-| Provider | Available | Stub | Planned | Next up (0.3) | Datasets |
+| Provider | Available | Stub | Planned | Next up (0.6) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | `gfs`, `hrrr`, `oisst`, `etopo` | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
 | [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
-| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | `gpm-imerg` | +1 planned |
 | [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
-| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | — | +3 planned |
 
-Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
+Available datasets are in `code`, stubs in _italics_; planned ones are counted. Available means implemented in this source checkout; consult the [releases](https://github.com/jakeryderv/usdata/releases) for published support. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
 <!-- registry:end -->
 
 ## Install
@@ -53,6 +54,7 @@ for item in fetch(ds, query):
 
 ```sh
 usdata search "tornado radar" --state OK
+usdata search precipitation --location "Cleveland County, OK"
 usdata info noaa:ghcn-daily
 usdata fetch noaa:ghcn-daily --lat 35.39 --lon -97.60 --radius-km 15 \
     --start 2024-05-06 --end 2024-05-07 --vars PRCP,TMAX
@@ -60,6 +62,10 @@ usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 20
 usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
     --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
 usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
+usdata fetch usgs:water-daily -p sites=07164500 --vars 00060 \
+    --start 2024-05-06 --end 2024-05-07
+usdata fetch noaa:coastwatch-sst --bbox=-80.08,30.02,-80.02,30.08 \
+    --start 2024-05-06T12:00Z --end 2024-05-06T12:00Z    # four grid cells
 usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
 usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
@@ -68,12 +74,27 @@ Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
+Locations accept state names/postal codes, county/state names, and quoted FIPS
+codes. These select bounding rectangles; see [place lookup](docs/reference/places.md)
+for coverage, ambiguity, and antimeridian limits. CoastWatch CSV includes a
+second header row containing units; see its [access notes](docs/providers/noaa.md#coastwatch-sst).
+
+Manifest and source fields are validated strictly; unknown fields are errors.
+Provider-specific options belong under `params`.
+
 A manifest declares every input a project needs. `pull` resolves each source,
 fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
 and provenance. A second `pull` restores exactly what the lockfile pins without
 re-querying upstream, so the inputs stay reproducible even if the source
-changes. `verify` re-hashes the cached files against the lockfile. Editing the
-manifest after locking requires `pull --force` to re-resolve.
+changes. `verify` checks the manifest checksum and re-hashes cached files
+against the lockfile. Editing the manifest after locking requires `pull --force`
+to re-resolve. A required source matching no assets fails the pull; set
+`allow_empty: true` on a source only when an empty result is intentional.
+
+Checksums detect upstream changes; they cannot recover historical bytes that
+are no longer available. Preserve the cache for long-lived reproducibility.
+See the [manifest reference](docs/reference/manifests.md) and the small
+[NOAA/USGS example](examples/weather-and-streamflow/README.md).
 
 ```yaml
 name: tornado-environment
@@ -96,11 +117,16 @@ Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
-just check     # format, lint, typecheck, tests (what CI runs)
+just check     # format, lint, typecheck, offline tests, generated docs
+just build     # build wheel and sdist
+just smoke     # install and exercise the built wheel outside the checkout
 just run search radar
 ```
 
-Integration tests that hit live services run with `just test-integration`.
+Unit tests mechanically block network connections. Integration tests that hit
+live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
+Linux and smoke-tests the installed wheel on Linux, macOS, and Windows. The
+full unit and live-service suites currently run on Linux.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.3.0 → usdata-0.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.3.0"
+version = "0.5.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

{usdata-0.3.0 → usdata-0.5.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.3.0"
+version = "0.5.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

usdata-0.5.0/src/usdata/_files.py

@@ -0,0 +1,29 @@
+"""Atomic replacement helpers for downloads and JSON records."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from contextlib import contextmanager
+from pathlib import Path
+from tempfile import NamedTemporaryFile
+
+
+@contextmanager
+def staged_path(dest: Path) -> Iterator[Path]:
+    """Replace dest only after successful work in a unique sibling temporary file."""
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    with NamedTemporaryFile(
+        prefix=f".{dest.name}.", suffix=".part", dir=dest.parent, delete=False
+    ) as f:
+        tmp = Path(f.name)
+    try:
+        yield tmp
+        tmp.replace(dest)
+    finally:
+        tmp.unlink(missing_ok=True)
+
+
+def atomic_write_text(dest: Path, text: str) -> None:
+    """Write UTF-8 text without exposing an incomplete record to readers."""
+    with staged_path(dest) as tmp:
+        tmp.write_text(text, encoding="utf-8")

{usdata-0.3.0 → usdata-0.5.0}/src/usdata/cache.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import hashlib
 import os
+import re
 from pathlib import Path
 
 from usdata.models import Asset
@@ -22,9 +23,20 @@ def cache_dir() -> Path:
 
 def asset_path(asset: Asset, root: Path | None = None) -> Path:
     """Where an asset lives in the cache: <root>/<provider>/<dataset>/<asset id>."""
-    provider, name = asset.dataset_id.split(":", 1)
+    provider, sep, name = asset.dataset_id.partition(":")
+    if not sep or any(
+        not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*", part) for part in (provider, name)
+    ):
+        raise ValueError(f"unsafe dataset id: {asset.dataset_id!r}")
     safe_id = asset.id.replace("/", "_")
-    return (root or cache_dir()) / provider / name / safe_id
+    if not safe_id or safe_id in {".", ".."} or "\\" in safe_id or "\x00" in safe_id:
+        raise ValueError(f"unsafe asset id: {asset.id!r}")
+    base = (root or cache_dir()).expanduser().resolve()
+    path = base / provider / name / safe_id
+    for candidate in (path, path.with_name(path.name + ".provenance.json")):
+        if not candidate.resolve().is_relative_to(base):
+            raise ValueError(f"cache path escapes root: {candidate}")
+    return path
 
 
 def sha256_file(path: Path, chunk_size: int = 1 << 20) -> str:

{usdata-0.3.0 → usdata-0.5.0}/src/usdata/cli/app.py

@@ -9,11 +9,12 @@ import httpx
 import typer
 
 from usdata import __version__, build_query, default_registry
-from usdata.fetch import ChecksumMismatch, fetch_asset
+from usdata.fetch import ChecksumMismatch
+from usdata.fetch import fetch as fetch_query
 from usdata.manifest import lockfile_path
 from usdata.providers import load_adapter
 from usdata.providers.base import NotImplementedProvider
-from usdata.pull import ManifestChanged, UnknownDatasets
+from usdata.pull import EmptySource, ManifestChanged, UnknownDatasets
 from usdata.pull import pull as pull_manifest
 from usdata.pull import verify as verify_manifest
 from usdata.query import UnknownPlace
@@ -49,7 +50,10 @@ def search(
     provider: Annotated[
         str | None, typer.Option(help="Restrict to one provider, e.g. noaa.")
     ] = None,
-    state: Annotated[str | None, typer.Option(help="State name or postal code.")] = None,
+    state: Annotated[
+        str | None,
+        typer.Option("--location", "--state", help="State, 'County, ST', or quoted FIPS code."),
+    ] = None,
     start: Annotated[str | None, typer.Option(help="ISO date or datetime.")] = None,
     end: Annotated[str | None, typer.Option(help="ISO date or datetime.")] = None,
     planned: Annotated[
@@ -59,8 +63,8 @@ def search(
     """Search the curated dataset registry."""
     try:
         query = build_query(text, provider=provider, location=state, start=start, end=end)
-    except UnknownPlace as e:
-        typer.secho(f"Unknown place: {e}", err=True, fg="red")
+    except ValueError as e:
+        typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
     results = default_registry().search(query, include_planned=planned)
     if not results:
@@ -103,7 +107,10 @@ def info(
 @app.command()
 def fetch(
     dataset_id: Annotated[str, typer.Argument(help="Dataset id, e.g. noaa:ghcn-daily")],
-    state: Annotated[str | None, typer.Option(help="State name or postal code.")] = None,
+    state: Annotated[
+        str | None,
+        typer.Option("--location", "--state", help="State, 'County, ST', or quoted FIPS code."),
+    ] = None,
     bbox: Annotated[str | None, typer.Option(help="west,south,east,north in degrees.")] = None,
     lat: Annotated[float | None, typer.Option()] = None,
     lon: Annotated[float | None, typer.Option()] = None,
@@ -152,21 +159,21 @@ def fetch(
             variables=[v.strip() for v in variables.split(",")] if variables else None,
             **params,
         )
-        adapter = load_adapter(ds)
-        assets = adapter.list_assets(query)
         if dry_run:
+            with load_adapter(ds) as adapter:
+                assets = adapter.list_assets(query)
             for a in assets:
                 typer.echo(f"{a.id}\t{a.href}")
             typer.echo(f"{len(assets)} asset(s) matched", err=True)
             return
-        fetched = [fetch_asset(ds, a, root=cache_dir, force=force) for a in assets]
+        fetched = fetch_query(ds, query, root=cache_dir, force=force)
     except (DatasetNotFound, UnknownPlace, ValueError) as e:
         typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
     except NotImplementedProvider as e:
         typer.secho(str(e), err=True, fg="yellow")
         raise typer.Exit(code=3) from None
-    except httpx.HTTPError as e:
+    except (httpx.HTTPError, ChecksumMismatch) as e:
         typer.secho(f"request failed: {e}", err=True, fg="red")
         raise typer.Exit(code=4) from None
     if not fetched:
@@ -189,7 +196,10 @@ def pull(
     """Fetch every source in a manifest and write (or restore from) its lockfile."""
     try:
         result = pull_manifest(manifest, root=cache_dir, force=force)
-    except (UnknownDatasets, ManifestChanged, UnknownPlace, ValueError) as e:
+    except EmptySource as e:
+        typer.secho(str(e), err=True, fg="yellow")
+        raise typer.Exit(code=1) from None
+    except (DatasetNotFound, UnknownDatasets, ManifestChanged, UnknownPlace, ValueError) as e:
         typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
     except NotImplementedProvider as e:
@@ -215,7 +225,11 @@ def verify(
     if not lock.exists():
         typer.secho(f"no lockfile at {lock}; run pull first", err=True, fg="red")
         raise typer.Exit(code=2)
-    drift = verify_manifest(manifest, root=cache_dir)
+    try:
+        drift = verify_manifest(manifest, root=cache_dir)
+    except (ManifestChanged, ValueError, OSError) as e:
+        typer.secho(str(e), err=True, fg="red")
+        raise typer.Exit(code=2) from None
     for d in drift:
         typer.echo(f"{d.asset_id}\t{d.problem}\t{d.path}")
     if drift: