usdata 0.2.1__tar.gz → 0.3.0__tar.gz

{usdata-0.2.1 → usdata-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.2.1
+Version: 0.3.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -28,6 +28,22 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 > data with provenance; other registry entries are stubs.
 > See [docs/roadmap.md](docs/roadmap.md).
 
+## Providers
+
+<!-- registry:start -->
+| Provider | Available | Stub | Planned | Next up (0.3) | Datasets |
+|---|---:|---:|---:|---|---|
+| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
+| [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
+| [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
+| [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
+| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | — | +3 planned |
+
+Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
+<!-- registry:end -->
+
 ## Install
 
 ```sh
@@ -65,15 +81,20 @@ usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 20
 usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
     --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
 usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
-usdata pull dataset.yaml
+usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
+usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
-A manifest declares every input a project needs; `pull` fetches them and writes
-a lockfile with checksums and provenance so the inputs can be reproduced:
+A manifest declares every input a project needs. `pull` resolves each source,
+fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
+and provenance. A second `pull` restores exactly what the lockfile pins without
+re-querying upstream, so the inputs stay reproducible even if the source
+changes. `verify` re-hashes the cached files against the lockfile. Editing the
+manifest after locking requires `pull --force` to re-resolve.
 
 ```yaml
 name: tornado-environment
@@ -106,7 +127,8 @@ Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See
 [docs/versioning.md](docs/versioning.md).
 
-See [docs/architecture.md](docs/architecture.md) for how the pieces fit,
+See [docs/providers/](docs/providers/) for per-provider access notes,
+[docs/architecture.md](docs/architecture.md) for how the pieces fit,
 [docs/adr/](docs/adr/) for why, and [CONTRIBUTING.md](CONTRIBUTING.md) to add
 a dataset.
 

{usdata-0.2.1 → usdata-0.3.0}/README.md

@@ -7,6 +7,22 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 > data with provenance; other registry entries are stubs.
 > See [docs/roadmap.md](docs/roadmap.md).
 
+## Providers
+
+<!-- registry:start -->
+| Provider | Available | Stub | Planned | Next up (0.3) | Datasets |
+|---|---:|---:|---:|---|---|
+| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
+| [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
+| [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
+| [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
+| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | — | +3 planned |
+
+Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
+<!-- registry:end -->
+
 ## Install
 
 ```sh
@@ -44,15 +60,20 @@ usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 20
 usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
     --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
 usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
-usdata pull dataset.yaml
+usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
+usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
-A manifest declares every input a project needs; `pull` fetches them and writes
-a lockfile with checksums and provenance so the inputs can be reproduced:
+A manifest declares every input a project needs. `pull` resolves each source,
+fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
+and provenance. A second `pull` restores exactly what the lockfile pins without
+re-querying upstream, so the inputs stay reproducible even if the source
+changes. `verify` re-hashes the cached files against the lockfile. Editing the
+manifest after locking requires `pull --force` to re-resolve.
 
 ```yaml
 name: tornado-environment
@@ -85,7 +106,8 @@ Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See
 [docs/versioning.md](docs/versioning.md).
 
-See [docs/architecture.md](docs/architecture.md) for how the pieces fit,
+See [docs/providers/](docs/providers/) for per-provider access notes,
+[docs/architecture.md](docs/architecture.md) for how the pieces fit,
 [docs/adr/](docs/adr/) for why, and [CONTRIBUTING.md](CONTRIBUTING.md) to add
 a dataset.
 

{usdata-0.2.1 → usdata-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.2.1"
+version = "0.3.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

{usdata-0.2.1 → usdata-0.3.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.2.1"
+version = "0.3.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

{usdata-0.2.1 → usdata-0.3.0}/src/usdata/__init__.py

@@ -11,6 +11,7 @@ except PackageNotFoundError:  # running from a source tree without an install
     __version__ = "0.0.0"
 
 from usdata.models import Asset, BBox, Dataset, Provenance, Query, TimeRange
+from usdata.pull import pull, verify
 from usdata.query import build_query
 from usdata.registry import DatasetNotFound, Registry, SearchResult, default_registry
 
@@ -28,13 +29,17 @@ __all__ = [
     "build_query",
     "default_registry",
     "get",
+    "pull",
     "search",
+    "verify",
 ]
 
 
-def search(text: str | None = None, **kwargs: Any) -> list[SearchResult]:
+def search(
+    text: str | None = None, *, include_planned: bool = False, **kwargs: Any
+) -> list[SearchResult]:
     """Search the curated registry. Keyword arguments match ``build_query``."""
-    return default_registry().search(build_query(text, **kwargs))
+    return default_registry().search(build_query(text, **kwargs), include_planned=include_planned)
 
 
 def get(dataset_id: str) -> Dataset:

{usdata-0.2.1 → usdata-0.3.0}/src/usdata/cli/app.py

@@ -9,10 +9,13 @@ import httpx
 import typer
 
 from usdata import __version__, build_query, default_registry
-from usdata.fetch import fetch_asset
-from usdata.manifest import Manifest
+from usdata.fetch import ChecksumMismatch, fetch_asset
+from usdata.manifest import lockfile_path
 from usdata.providers import load_adapter
 from usdata.providers.base import NotImplementedProvider
+from usdata.pull import ManifestChanged, UnknownDatasets
+from usdata.pull import pull as pull_manifest
+from usdata.pull import verify as verify_manifest
 from usdata.query import UnknownPlace
 from usdata.registry import DatasetNotFound
 
@@ -49,6 +52,9 @@ def search(
     state: Annotated[str | None, typer.Option(help="State name or postal code.")] = None,
     start: Annotated[str | None, typer.Option(help="ISO date or datetime.")] = None,
     end: Annotated[str | None, typer.Option(help="ISO date or datetime.")] = None,
+    planned: Annotated[
+        bool, typer.Option("--planned", help="Include planned datasets that have no adapter yet.")
+    ] = False,
 ) -> None:
     """Search the curated dataset registry."""
     try:
@@ -56,13 +62,14 @@ def search(
     except UnknownPlace as e:
         typer.secho(f"Unknown place: {e}", err=True, fg="red")
         raise typer.Exit(code=2) from None
-    results = default_registry().search(query)
+    results = default_registry().search(query, include_planned=planned)
     if not results:
         typer.echo("No datasets matched.")
         raise typer.Exit(code=1)
     width = max(len(r.dataset.id) for r in results)
     for r in results:
-        typer.echo(f"{r.dataset.id:<{width}}  {r.dataset.title}")
+        ds = r.dataset
+        typer.echo(f"{ds.id:<{width}}  {ds.status.value:<9}  {ds.version_label:<12}  {ds.title}")
 
 
 @app.command()
@@ -77,6 +84,8 @@ def info(
         raise typer.Exit(code=2) from None
     typer.echo(f"{ds.id}\n  {ds.title}\n")
     typer.echo(f"  {ds.description.strip()}\n")
+    typer.echo(f"  status:    {ds.status.value} ({ds.version_label})")
+    typer.echo(f"  domain:    {ds.domain}")
     typer.echo(f"  provider:  {ds.provider}")
     typer.echo(f"  protocol:  {ds.protocol.value}")
     typer.echo(f"  license:   {ds.license or 'unknown'}")
@@ -169,13 +178,47 @@ def fetch(
 
 
 @app.command()
-def pull(manifest: Annotated[Path, typer.Argument(exists=True, dir_okay=False)]) -> None:
-    """Fetch every source in a manifest and write a lockfile."""
-    m = Manifest.load(manifest)
-    missing = m.validate_against()
-    if missing:
-        typer.secho(f"Unknown datasets in manifest: {', '.join(missing)}", err=True, fg="red")
+def pull(
+    manifest: Annotated[Path, typer.Argument(exists=True, dir_okay=False)],
+    cache_dir: Annotated[Path | None, typer.Option(help="Override the cache directory.")] = None,
+    force: Annotated[
+        bool,
+        typer.Option(help="Ignore an existing lockfile: re-resolve every source and rewrite it."),
+    ] = False,
+) -> None:
+    """Fetch every source in a manifest and write (or restore from) its lockfile."""
+    try:
+        result = pull_manifest(manifest, root=cache_dir, force=force)
+    except (UnknownDatasets, ManifestChanged, UnknownPlace, ValueError) as e:
+        typer.secho(str(e), err=True, fg="red")
+        raise typer.Exit(code=2) from None
+    except NotImplementedProvider as e:
+        typer.secho(str(e), err=True, fg="yellow")
+        raise typer.Exit(code=3) from None
+    except (httpx.HTTPError, ChecksumMismatch) as e:
+        typer.secho(f"fetch failed: {e}", err=True, fg="red")
+        raise typer.Exit(code=4) from None
+    for f in result.fetched:
+        tag = "cached" if f.from_cache else "fetched"
+        typer.echo(f"{f.path}\t{tag}\t{f.provenance.size} bytes")
+    mode = "restored from" if result.from_lockfile else "wrote"
+    typer.echo(f"{len(result.fetched)} asset(s); {mode} {result.lockfile_path}", err=True)
+
+
+@app.command()
+def verify(
+    manifest: Annotated[Path, typer.Argument(exists=True, dir_okay=False)],
+    cache_dir: Annotated[Path | None, typer.Option(help="Override the cache directory.")] = None,
+) -> None:
+    """Check cached files against a manifest's lockfile. Exit 1 on any drift."""
+    lock = lockfile_path(manifest)
+    if not lock.exists():
+        typer.secho(f"no lockfile at {lock}; run pull first", err=True, fg="red")
         raise typer.Exit(code=2)
-    typer.echo(f"{m.name} v{m.version}: {len(m.sources)} source(s) validated")
-    typer.secho("pull is not implemented yet; no data was fetched.", err=True, fg="yellow")
-    raise typer.Exit(code=3)
+    drift = verify_manifest(manifest, root=cache_dir)
+    for d in drift:
+        typer.echo(f"{d.asset_id}\t{d.problem}\t{d.path}")
+    if drift:
+        typer.secho(f"{len(drift)} asset(s) drifted from {lock.name}", err=True, fg="red")
+        raise typer.Exit(code=1)
+    typer.echo(f"all assets match {lock.name}", err=True)