usdata 0.2.1__tar.gz → 0.4.0__tar.gz

{usdata-0.2.1 → usdata-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.2.1
+Version: 0.4.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -28,6 +28,22 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 > data with provenance; other registry entries are stubs.
 > See [docs/roadmap.md](docs/roadmap.md).
 
+## Providers
+
+<!-- registry:start -->
+| Provider | Available | Stub | Planned | Next up (0.5) | Datasets |
+|---|---:|---:|---:|---|---|
+| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `gsom`, `gsoy`, `storm-events`, `mrms`, `goes-abi`, `hurdat2`, `ibtracs`, `climate-normals`, `coops-water-levels`, `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
+| [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
+| [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
+| [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
+| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | `water-daily` | +3 planned |
+
+Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
+<!-- registry:end -->
+
 ## Install
 
 ```sh
@@ -65,15 +81,23 @@ usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 20
 usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
     --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
 usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
-usdata pull dataset.yaml
+usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
+usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
-A manifest declares every input a project needs; `pull` fetches them and writes
-a lockfile with checksums and provenance so the inputs can be reproduced:
+Manifest and source fields are validated strictly; unknown fields are errors.
+Provider-specific options belong under `params`.
+
+A manifest declares every input a project needs. `pull` resolves each source,
+fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
+and provenance. A second `pull` restores exactly what the lockfile pins without
+re-querying upstream, so the inputs stay reproducible even if the source
+changes. `verify` re-hashes the cached files against the lockfile. Editing the
+manifest after locking requires `pull --force` to re-resolve.
 
 ```yaml
 name: tornado-environment
@@ -106,7 +130,8 @@ Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See
 [docs/versioning.md](docs/versioning.md).
 
-See [docs/architecture.md](docs/architecture.md) for how the pieces fit,
+See [docs/providers/](docs/providers/) for per-provider access notes,
+[docs/architecture.md](docs/architecture.md) for how the pieces fit,
 [docs/adr/](docs/adr/) for why, and [CONTRIBUTING.md](CONTRIBUTING.md) to add
 a dataset.
 

{usdata-0.2.1 → usdata-0.4.0}/README.md

@@ -7,6 +7,22 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 > data with provenance; other registry entries are stubs.
 > See [docs/roadmap.md](docs/roadmap.md).
 
+## Providers
+
+<!-- registry:start -->
+| Provider | Available | Stub | Planned | Next up (0.5) | Datasets |
+|---|---:|---:|---:|---|---|
+| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `gsom`, `gsoy`, `storm-events`, `mrms`, `goes-abi`, `hurdat2`, `ibtracs`, `climate-normals`, `coops-water-levels`, `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
+| [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
+| [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
+| [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
+| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | `water-daily` | +3 planned |
+
+Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
+<!-- registry:end -->
+
 ## Install
 
 ```sh
@@ -44,15 +60,23 @@ usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 20
 usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
     --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
 usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
-usdata pull dataset.yaml
+usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
+usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
-A manifest declares every input a project needs; `pull` fetches them and writes
-a lockfile with checksums and provenance so the inputs can be reproduced:
+Manifest and source fields are validated strictly; unknown fields are errors.
+Provider-specific options belong under `params`.
+
+A manifest declares every input a project needs. `pull` resolves each source,
+fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
+and provenance. A second `pull` restores exactly what the lockfile pins without
+re-querying upstream, so the inputs stay reproducible even if the source
+changes. `verify` re-hashes the cached files against the lockfile. Editing the
+manifest after locking requires `pull --force` to re-resolve.
 
 ```yaml
 name: tornado-environment
@@ -85,7 +109,8 @@ Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See
 [docs/versioning.md](docs/versioning.md).
 
-See [docs/architecture.md](docs/architecture.md) for how the pieces fit,
+See [docs/providers/](docs/providers/) for per-provider access notes,
+[docs/architecture.md](docs/architecture.md) for how the pieces fit,
 [docs/adr/](docs/adr/) for why, and [CONTRIBUTING.md](CONTRIBUTING.md) to add
 a dataset.
 

{usdata-0.2.1 → usdata-0.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.2.1"
+version = "0.4.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

{usdata-0.2.1 → usdata-0.4.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.2.1"
+version = "0.4.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

{usdata-0.2.1 → usdata-0.4.0}/src/usdata/__init__.py

@@ -11,6 +11,7 @@ except PackageNotFoundError:  # running from a source tree without an install
     __version__ = "0.0.0"
 
 from usdata.models import Asset, BBox, Dataset, Provenance, Query, TimeRange
+from usdata.pull import pull, verify
 from usdata.query import build_query
 from usdata.registry import DatasetNotFound, Registry, SearchResult, default_registry
 
@@ -28,13 +29,17 @@ __all__ = [
     "build_query",
     "default_registry",
     "get",
+    "pull",
     "search",
+    "verify",
 ]
 
 
-def search(text: str | None = None, **kwargs: Any) -> list[SearchResult]:
+def search(
+    text: str | None = None, *, include_planned: bool = False, **kwargs: Any
+) -> list[SearchResult]:
     """Search the curated registry. Keyword arguments match ``build_query``."""
-    return default_registry().search(build_query(text, **kwargs))
+    return default_registry().search(build_query(text, **kwargs), include_planned=include_planned)
 
 
 def get(dataset_id: str) -> Dataset:

usdata-0.4.0/src/usdata/_files.py

@@ -0,0 +1,29 @@
+"""Atomic replacement helpers for downloads and JSON records."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from contextlib import contextmanager
+from pathlib import Path
+from tempfile import NamedTemporaryFile
+
+
+@contextmanager
+def staged_path(dest: Path) -> Iterator[Path]:
+    """Replace dest only after successful work in a unique sibling temporary file."""
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    with NamedTemporaryFile(
+        prefix=f".{dest.name}.", suffix=".part", dir=dest.parent, delete=False
+    ) as f:
+        tmp = Path(f.name)
+    try:
+        yield tmp
+        tmp.replace(dest)
+    finally:
+        tmp.unlink(missing_ok=True)
+
+
+def atomic_write_text(dest: Path, text: str) -> None:
+    """Write UTF-8 text without exposing an incomplete record to readers."""
+    with staged_path(dest) as tmp:
+        tmp.write_text(text, encoding="utf-8")

{usdata-0.2.1 → usdata-0.4.0}/src/usdata/cache.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import hashlib
 import os
+import re
 from pathlib import Path
 
 from usdata.models import Asset
@@ -22,9 +23,20 @@ def cache_dir() -> Path:
 
 def asset_path(asset: Asset, root: Path | None = None) -> Path:
     """Where an asset lives in the cache: <root>/<provider>/<dataset>/<asset id>."""
-    provider, name = asset.dataset_id.split(":", 1)
+    provider, sep, name = asset.dataset_id.partition(":")
+    if not sep or any(
+        not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*", part) for part in (provider, name)
+    ):
+        raise ValueError(f"unsafe dataset id: {asset.dataset_id!r}")
     safe_id = asset.id.replace("/", "_")
-    return (root or cache_dir()) / provider / name / safe_id
+    if not safe_id or safe_id in {".", ".."} or "\\" in safe_id or "\x00" in safe_id:
+        raise ValueError(f"unsafe asset id: {asset.id!r}")
+    base = (root or cache_dir()).expanduser().resolve()
+    path = base / provider / name / safe_id
+    for candidate in (path, path.with_name(path.name + ".provenance.json")):
+        if not candidate.resolve().is_relative_to(base):
+            raise ValueError(f"cache path escapes root: {candidate}")
+    return path
 
 
 def sha256_file(path: Path, chunk_size: int = 1 << 20) -> str:

{usdata-0.2.1 → usdata-0.4.0}/src/usdata/cli/app.py

@@ -9,10 +9,14 @@ import httpx
 import typer
 
 from usdata import __version__, build_query, default_registry
-from usdata.fetch import fetch_asset
-from usdata.manifest import Manifest
+from usdata.fetch import ChecksumMismatch
+from usdata.fetch import fetch as fetch_query
+from usdata.manifest import lockfile_path
 from usdata.providers import load_adapter
 from usdata.providers.base import NotImplementedProvider
+from usdata.pull import ManifestChanged, UnknownDatasets
+from usdata.pull import pull as pull_manifest
+from usdata.pull import verify as verify_manifest
 from usdata.query import UnknownPlace
 from usdata.registry import DatasetNotFound
 
@@ -49,20 +53,24 @@ def search(
     state: Annotated[str | None, typer.Option(help="State name or postal code.")] = None,
     start: Annotated[str | None, typer.Option(help="ISO date or datetime.")] = None,
     end: Annotated[str | None, typer.Option(help="ISO date or datetime.")] = None,
+    planned: Annotated[
+        bool, typer.Option("--planned", help="Include planned datasets that have no adapter yet.")
+    ] = False,
 ) -> None:
     """Search the curated dataset registry."""
     try:
         query = build_query(text, provider=provider, location=state, start=start, end=end)
-    except UnknownPlace as e:
-        typer.secho(f"Unknown place: {e}", err=True, fg="red")
+    except ValueError as e:
+        typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
-    results = default_registry().search(query)
+    results = default_registry().search(query, include_planned=planned)
     if not results:
         typer.echo("No datasets matched.")
         raise typer.Exit(code=1)
     width = max(len(r.dataset.id) for r in results)
     for r in results:
-        typer.echo(f"{r.dataset.id:<{width}}  {r.dataset.title}")
+        ds = r.dataset
+        typer.echo(f"{ds.id:<{width}}  {ds.status.value:<9}  {ds.version_label:<12}  {ds.title}")
 
 
 @app.command()
@@ -77,6 +85,8 @@ def info(
         raise typer.Exit(code=2) from None
     typer.echo(f"{ds.id}\n  {ds.title}\n")
     typer.echo(f"  {ds.description.strip()}\n")
+    typer.echo(f"  status:    {ds.status.value} ({ds.version_label})")
+    typer.echo(f"  domain:    {ds.domain}")
     typer.echo(f"  provider:  {ds.provider}")
     typer.echo(f"  protocol:  {ds.protocol.value}")
     typer.echo(f"  license:   {ds.license or 'unknown'}")
@@ -143,21 +153,21 @@ def fetch(
             variables=[v.strip() for v in variables.split(",")] if variables else None,
             **params,
         )
-        adapter = load_adapter(ds)
-        assets = adapter.list_assets(query)
         if dry_run:
+            with load_adapter(ds) as adapter:
+                assets = adapter.list_assets(query)
             for a in assets:
                 typer.echo(f"{a.id}\t{a.href}")
             typer.echo(f"{len(assets)} asset(s) matched", err=True)
             return
-        fetched = [fetch_asset(ds, a, root=cache_dir, force=force) for a in assets]
+        fetched = fetch_query(ds, query, root=cache_dir, force=force)
     except (DatasetNotFound, UnknownPlace, ValueError) as e:
         typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
     except NotImplementedProvider as e:
         typer.secho(str(e), err=True, fg="yellow")
         raise typer.Exit(code=3) from None
-    except httpx.HTTPError as e:
+    except (httpx.HTTPError, ChecksumMismatch) as e:
         typer.secho(f"request failed: {e}", err=True, fg="red")
         raise typer.Exit(code=4) from None
     if not fetched:
@@ -169,13 +179,51 @@ def fetch(
 
 
 @app.command()
-def pull(manifest: Annotated[Path, typer.Argument(exists=True, dir_okay=False)]) -> None:
-    """Fetch every source in a manifest and write a lockfile."""
-    m = Manifest.load(manifest)
-    missing = m.validate_against()
-    if missing:
-        typer.secho(f"Unknown datasets in manifest: {', '.join(missing)}", err=True, fg="red")
+def pull(
+    manifest: Annotated[Path, typer.Argument(exists=True, dir_okay=False)],
+    cache_dir: Annotated[Path | None, typer.Option(help="Override the cache directory.")] = None,
+    force: Annotated[
+        bool,
+        typer.Option(help="Ignore an existing lockfile: re-resolve every source and rewrite it."),
+    ] = False,
+) -> None:
+    """Fetch every source in a manifest and write (or restore from) its lockfile."""
+    try:
+        result = pull_manifest(manifest, root=cache_dir, force=force)
+    except (DatasetNotFound, UnknownDatasets, ManifestChanged, UnknownPlace, ValueError) as e:
+        typer.secho(str(e), err=True, fg="red")
+        raise typer.Exit(code=2) from None
+    except NotImplementedProvider as e:
+        typer.secho(str(e), err=True, fg="yellow")
+        raise typer.Exit(code=3) from None
+    except (httpx.HTTPError, ChecksumMismatch) as e:
+        typer.secho(f"fetch failed: {e}", err=True, fg="red")
+        raise typer.Exit(code=4) from None
+    for f in result.fetched:
+        tag = "cached" if f.from_cache else "fetched"
+        typer.echo(f"{f.path}\t{tag}\t{f.provenance.size} bytes")
+    mode = "restored from" if result.from_lockfile else "wrote"
+    typer.echo(f"{len(result.fetched)} asset(s); {mode} {result.lockfile_path}", err=True)
+
+
+@app.command()
+def verify(
+    manifest: Annotated[Path, typer.Argument(exists=True, dir_okay=False)],
+    cache_dir: Annotated[Path | None, typer.Option(help="Override the cache directory.")] = None,
+) -> None:
+    """Check cached files against a manifest's lockfile. Exit 1 on any drift."""
+    lock = lockfile_path(manifest)
+    if not lock.exists():
+        typer.secho(f"no lockfile at {lock}; run pull first", err=True, fg="red")
         raise typer.Exit(code=2)
-    typer.echo(f"{m.name} v{m.version}: {len(m.sources)} source(s) validated")
-    typer.secho("pull is not implemented yet; no data was fetched.", err=True, fg="yellow")
-    raise typer.Exit(code=3)
+    try:
+        drift = verify_manifest(manifest, root=cache_dir)
+    except (ValueError, OSError) as e:
+        typer.secho(str(e), err=True, fg="red")
+        raise typer.Exit(code=2) from None
+    for d in drift:
+        typer.echo(f"{d.asset_id}\t{d.problem}\t{d.path}")
+    if drift:
+        typer.secho(f"{len(drift)} asset(s) drifted from {lock.name}", err=True, fg="red")
+        raise typer.Exit(code=1)
+    typer.echo(f"all assets match {lock.name}", err=True)