usdata 0.2.0__tar.gz → 0.3.0__tar.gz

{usdata-0.2.0 → usdata-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.2.0
+Version: 0.3.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -28,6 +28,22 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 > data with provenance; other registry entries are stubs.
 > See [docs/roadmap.md](docs/roadmap.md).
 
+## Providers
+
+<!-- registry:start -->
+| Provider | Available | Stub | Planned | Next up (0.3) | Datasets |
+|---|---:|---:|---:|---|---|
+| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
+| [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
+| [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
+| [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
+| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | — | +3 planned |
+
+Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
+<!-- registry:end -->
+
 ## Install
 
 ```sh
@@ -65,15 +81,20 @@ usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 20
 usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
     --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
 usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
-usdata pull dataset.yaml
+usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
+usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
-A manifest declares every input a project needs; `pull` fetches them and writes
-a lockfile with checksums and provenance so the inputs can be reproduced:
+A manifest declares every input a project needs. `pull` resolves each source,
+fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
+and provenance. A second `pull` restores exactly what the lockfile pins without
+re-querying upstream, so the inputs stay reproducible even if the source
+changes. `verify` re-hashes the cached files against the lockfile. Editing the
+manifest after locking requires `pull --force` to re-resolve.
 
 ```yaml
 name: tornado-environment
@@ -102,11 +123,14 @@ just run search radar
 
 Integration tests that hit live services run with `just test-integration`.
 
-Releases are published to PyPI by tagging: create a GitHub release and the
-`publish.yml` workflow uploads via trusted publishing.
+Releases: `just release minor` opens a version-bump PR; merging it publishes
+to PyPI and creates the tag and GitHub release. See
+[docs/versioning.md](docs/versioning.md).
 
-See [docs/architecture.md](docs/architecture.md) for how the pieces fit and
-[docs/adr/](docs/adr/) for why.
+See [docs/providers/](docs/providers/) for per-provider access notes,
+[docs/architecture.md](docs/architecture.md) for how the pieces fit,
+[docs/adr/](docs/adr/) for why, and [CONTRIBUTING.md](CONTRIBUTING.md) to add
+a dataset.
 
 ## License
 

{usdata-0.2.0 → usdata-0.3.0}/README.md

@@ -7,6 +7,22 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 > data with provenance; other registry entries are stubs.
 > See [docs/roadmap.md](docs/roadmap.md).
 
+## Providers
+
+<!-- registry:start -->
+| Provider | Available | Stub | Planned | Next up (0.3) | Datasets |
+|---|---:|---:|---:|---|---|
+| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
+| [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
+| [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
+| [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
+| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | — | +3 planned |
+
+Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
+<!-- registry:end -->
+
 ## Install
 
 ```sh
@@ -44,15 +60,20 @@ usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 20
 usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
     --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
 usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
-usdata pull dataset.yaml
+usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
+usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
-A manifest declares every input a project needs; `pull` fetches them and writes
-a lockfile with checksums and provenance so the inputs can be reproduced:
+A manifest declares every input a project needs. `pull` resolves each source,
+fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
+and provenance. A second `pull` restores exactly what the lockfile pins without
+re-querying upstream, so the inputs stay reproducible even if the source
+changes. `verify` re-hashes the cached files against the lockfile. Editing the
+manifest after locking requires `pull --force` to re-resolve.
 
 ```yaml
 name: tornado-environment
@@ -81,11 +102,14 @@ just run search radar
 
 Integration tests that hit live services run with `just test-integration`.
 
-Releases are published to PyPI by tagging: create a GitHub release and the
-`publish.yml` workflow uploads via trusted publishing.
+Releases: `just release minor` opens a version-bump PR; merging it publishes
+to PyPI and creates the tag and GitHub release. See
+[docs/versioning.md](docs/versioning.md).
 
-See [docs/architecture.md](docs/architecture.md) for how the pieces fit and
-[docs/adr/](docs/adr/) for why.
+See [docs/providers/](docs/providers/) for per-provider access notes,
+[docs/architecture.md](docs/architecture.md) for how the pieces fit,
+[docs/adr/](docs/adr/) for why, and [CONTRIBUTING.md](CONTRIBUTING.md) to add
+a dataset.
 
 ## License
 

{usdata-0.2.0 → usdata-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.2.0"
+version = "0.3.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -41,6 +41,7 @@ usdata = "usdata.cli:app"
 dev = [
     "pyright>=1.1.380",
     "pytest>=8.0",
+    "pytest-cov>=7.1.0",
     "respx>=0.23.1",
     "ruff>=0.6",
 ]
@@ -66,7 +67,19 @@ select = [
     "B",
     "SIM",
     "RUF",
+    "D",
 ]
+ignore = [
+    "D105",
+    "D107",
+]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["D"]
+"scripts/**" = ["D"]
 
 [tool.pyright]
 include = [

{usdata-0.2.0 → usdata-0.3.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.2.0"
+version = "0.3.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -33,6 +33,7 @@ usdata = "usdata.cli:app"
 dev = [
     "pyright>=1.1.380",
     "pytest>=8.0",
+    "pytest-cov>=7.1.0",
     "respx>=0.23.1",
     "ruff>=0.6",
 ]
@@ -47,7 +48,16 @@ target-version = "py311"
 src = ["src", "tests"]
 
 [tool.ruff.lint]
-select = ["E", "F", "I", "UP", "B", "SIM", "RUF"]
+select = ["E", "F", "I", "UP", "B", "SIM", "RUF", "D"]
+# Docstrings are required on the public API only; magic methods and __init__ are exempt.
+ignore = ["D105", "D107"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["D"]
+"scripts/**" = ["D"]
 
 [tool.pyright]
 include = ["src", "tests"]

{usdata-0.2.0 → usdata-0.3.0}/src/usdata/__init__.py

@@ -11,6 +11,7 @@ except PackageNotFoundError:  # running from a source tree without an install
     __version__ = "0.0.0"
 
 from usdata.models import Asset, BBox, Dataset, Provenance, Query, TimeRange
+from usdata.pull import pull, verify
 from usdata.query import build_query
 from usdata.registry import DatasetNotFound, Registry, SearchResult, default_registry
 
@@ -28,13 +29,17 @@ __all__ = [
     "build_query",
     "default_registry",
     "get",
+    "pull",
     "search",
+    "verify",
 ]
 
 
-def search(text: str | None = None, **kwargs: Any) -> list[SearchResult]:
+def search(
+    text: str | None = None, *, include_planned: bool = False, **kwargs: Any
+) -> list[SearchResult]:
     """Search the curated registry. Keyword arguments match ``build_query``."""
-    return default_registry().search(build_query(text, **kwargs))
+    return default_registry().search(build_query(text, **kwargs), include_planned=include_planned)
 
 
 def get(dataset_id: str) -> Dataset:

{usdata-0.2.0 → usdata-0.3.0}/src/usdata/cache.py

@@ -12,6 +12,7 @@ ENV_VAR = "USDATA_CACHE_DIR"
 
 
 def cache_dir() -> Path:
+    """Cache root: $USDATA_CACHE_DIR, else $XDG_CACHE_HOME/usdata, else ~/.cache/usdata."""
     if override := os.environ.get(ENV_VAR):
         return Path(override).expanduser()
     xdg = os.environ.get("XDG_CACHE_HOME")
@@ -20,12 +21,14 @@ def cache_dir() -> Path:
 
 
 def asset_path(asset: Asset, root: Path | None = None) -> Path:
+    """Where an asset lives in the cache: <root>/<provider>/<dataset>/<asset id>."""
     provider, name = asset.dataset_id.split(":", 1)
     safe_id = asset.id.replace("/", "_")
     return (root or cache_dir()) / provider / name / safe_id
 
 
 def sha256_file(path: Path, chunk_size: int = 1 << 20) -> str:
+    """Hex sha256 of a file, prefixed 'sha256:' to match Asset.checksum."""
     h = hashlib.sha256()
     with path.open("rb") as f:
         while chunk := f.read(chunk_size):

usdata-0.3.0/src/usdata/cli/__init__.py

@@ -0,0 +1,5 @@
+"""Command-line interface. The Typer app lives in ``usdata.cli.app``."""
+
+from usdata.cli.app import app
+
+__all__ = ["app"]

{usdata-0.2.0 → usdata-0.3.0}/src/usdata/cli/app.py

@@ -1,3 +1,5 @@
+"""Typer application: argument parsing and exit codes only; logic lives in the library."""
+
 from __future__ import annotations
 
 from pathlib import Path
@@ -7,10 +9,13 @@ import httpx
 import typer
 
 from usdata import __version__, build_query, default_registry
-from usdata.fetch import fetch_asset
-from usdata.manifest import Manifest
+from usdata.fetch import ChecksumMismatch, fetch_asset
+from usdata.manifest import lockfile_path
 from usdata.providers import load_adapter
 from usdata.providers.base import NotImplementedProvider
+from usdata.pull import ManifestChanged, UnknownDatasets
+from usdata.pull import pull as pull_manifest
+from usdata.pull import verify as verify_manifest
 from usdata.query import UnknownPlace
 from usdata.registry import DatasetNotFound
 
@@ -34,6 +39,7 @@ def main(
         typer.Option("--version", callback=_version_callback, is_eager=True, help="Show version."),
     ] = None,
 ) -> None:
+    """usdata: discover, fetch, and track provenance of U.S. public scientific data."""
     pass
 
 
@@ -46,6 +52,9 @@ def search(
     state: Annotated[str | None, typer.Option(help="State name or postal code.")] = None,
     start: Annotated[str | None, typer.Option(help="ISO date or datetime.")] = None,
     end: Annotated[str | None, typer.Option(help="ISO date or datetime.")] = None,
+    planned: Annotated[
+        bool, typer.Option("--planned", help="Include planned datasets that have no adapter yet.")
+    ] = False,
 ) -> None:
     """Search the curated dataset registry."""
     try:
@@ -53,13 +62,14 @@ def search(
     except UnknownPlace as e:
         typer.secho(f"Unknown place: {e}", err=True, fg="red")
         raise typer.Exit(code=2) from None
-    results = default_registry().search(query)
+    results = default_registry().search(query, include_planned=planned)
     if not results:
         typer.echo("No datasets matched.")
         raise typer.Exit(code=1)
     width = max(len(r.dataset.id) for r in results)
     for r in results:
-        typer.echo(f"{r.dataset.id:<{width}}  {r.dataset.title}")
+        ds = r.dataset
+        typer.echo(f"{ds.id:<{width}}  {ds.status.value:<9}  {ds.version_label:<12}  {ds.title}")
 
 
 @app.command()
@@ -74,6 +84,8 @@ def info(
         raise typer.Exit(code=2) from None
     typer.echo(f"{ds.id}\n  {ds.title}\n")
     typer.echo(f"  {ds.description.strip()}\n")
+    typer.echo(f"  status:    {ds.status.value} ({ds.version_label})")
+    typer.echo(f"  domain:    {ds.domain}")
     typer.echo(f"  provider:  {ds.provider}")
     typer.echo(f"  protocol:  {ds.protocol.value}")
     typer.echo(f"  license:   {ds.license or 'unknown'}")
@@ -166,13 +178,47 @@ def fetch(
 
 
 @app.command()
-def pull(manifest: Annotated[Path, typer.Argument(exists=True, dir_okay=False)]) -> None:
-    """Fetch every source in a manifest and write a lockfile."""
-    m = Manifest.load(manifest)
-    missing = m.validate_against()
-    if missing:
-        typer.secho(f"Unknown datasets in manifest: {', '.join(missing)}", err=True, fg="red")
+def pull(
+    manifest: Annotated[Path, typer.Argument(exists=True, dir_okay=False)],
+    cache_dir: Annotated[Path | None, typer.Option(help="Override the cache directory.")] = None,
+    force: Annotated[
+        bool,
+        typer.Option(help="Ignore an existing lockfile: re-resolve every source and rewrite it."),
+    ] = False,
+) -> None:
+    """Fetch every source in a manifest and write (or restore from) its lockfile."""
+    try:
+        result = pull_manifest(manifest, root=cache_dir, force=force)
+    except (UnknownDatasets, ManifestChanged, UnknownPlace, ValueError) as e:
+        typer.secho(str(e), err=True, fg="red")
+        raise typer.Exit(code=2) from None
+    except NotImplementedProvider as e:
+        typer.secho(str(e), err=True, fg="yellow")
+        raise typer.Exit(code=3) from None
+    except (httpx.HTTPError, ChecksumMismatch) as e:
+        typer.secho(f"fetch failed: {e}", err=True, fg="red")
+        raise typer.Exit(code=4) from None
+    for f in result.fetched:
+        tag = "cached" if f.from_cache else "fetched"
+        typer.echo(f"{f.path}\t{tag}\t{f.provenance.size} bytes")
+    mode = "restored from" if result.from_lockfile else "wrote"
+    typer.echo(f"{len(result.fetched)} asset(s); {mode} {result.lockfile_path}", err=True)
+
+
+@app.command()
+def verify(
+    manifest: Annotated[Path, typer.Argument(exists=True, dir_okay=False)],
+    cache_dir: Annotated[Path | None, typer.Option(help="Override the cache directory.")] = None,
+) -> None:
+    """Check cached files against a manifest's lockfile. Exit 1 on any drift."""
+    lock = lockfile_path(manifest)
+    if not lock.exists():
+        typer.secho(f"no lockfile at {lock}; run pull first", err=True, fg="red")
         raise typer.Exit(code=2)
-    typer.echo(f"{m.name} v{m.version}: {len(m.sources)} source(s) validated")
-    typer.secho("pull is not implemented yet; no data was fetched.", err=True, fg="yellow")
-    raise typer.Exit(code=3)
+    drift = verify_manifest(manifest, root=cache_dir)
+    for d in drift:
+        typer.echo(f"{d.asset_id}\t{d.problem}\t{d.path}")
+    if drift:
+        typer.secho(f"{len(drift)} asset(s) drifted from {lock.name}", err=True, fg="red")
+        raise typer.Exit(code=1)
+    typer.echo(f"all assets match {lock.name}", err=True)