usdata 0.3.0__tar.gz → 0.4.0__tar.gz

{usdata-0.3.0 → usdata-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.3.0
+Version: 0.4.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -31,15 +31,15 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 ## Providers
 
 <!-- registry:start -->
-| Provider | Available | Stub | Planned | Next up (0.3) | Datasets |
+| Provider | Available | Stub | Planned | Next up (0.5) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `gsom`, `gsoy`, `storm-events`, `mrms`, `goes-abi`, `hurdat2`, `ibtracs`, `climate-normals`, `coops-water-levels`, `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
 | [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
 | [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
 | [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
-| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | — | +3 planned |
+| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | `water-daily` | +3 planned |
 
 Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
 <!-- registry:end -->
@@ -89,6 +89,9 @@ Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
+Manifest and source fields are validated strictly; unknown fields are errors.
+Provider-specific options belong under `params`.
+
 A manifest declares every input a project needs. `pull` resolves each source,
 fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
 and provenance. A second `pull` restores exactly what the lockfile pins without

{usdata-0.3.0 → usdata-0.4.0}/README.md

@@ -10,15 +10,15 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 ## Providers
 
 <!-- registry:start -->
-| Provider | Available | Stub | Planned | Next up (0.3) | Datasets |
+| Provider | Available | Stub | Planned | Next up (0.5) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 2 | 1 | 26 | `gsom`, `gsoy`, `storm-events`, `mrms`, `goes-abi`, `hurdat2`, `ibtracs`, `climate-normals`, `coops-water-levels`, `coastwatch-sst` | `ghcn-daily`, `nexrad-level2`, _coastwatch-sst_, +26 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
 | [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
 | [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
 | [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
-| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | — | +3 planned |
+| [USGS](docs/providers/usgs.md) | 0 | 0 | 3 | `water-daily` | +3 planned |
 
 Available datasets are in `code`, stubs in _italics_; planned ones are counted. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
 <!-- registry:end -->
@@ -68,6 +68,9 @@ Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
 
+Manifest and source fields are validated strictly; unknown fields are errors.
+Provider-specific options belong under `params`.
+
 A manifest declares every input a project needs. `pull` resolves each source,
 fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
 and provenance. A second `pull` restores exactly what the lockfile pins without

{usdata-0.3.0 → usdata-0.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.3.0"
+version = "0.4.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

{usdata-0.3.0 → usdata-0.4.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.3.0"
+version = "0.4.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

usdata-0.4.0/src/usdata/_files.py

@@ -0,0 +1,29 @@
+"""Atomic replacement helpers for downloads and JSON records."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from contextlib import contextmanager
+from pathlib import Path
+from tempfile import NamedTemporaryFile
+
+
+@contextmanager
+def staged_path(dest: Path) -> Iterator[Path]:
+    """Replace dest only after successful work in a unique sibling temporary file."""
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    with NamedTemporaryFile(
+        prefix=f".{dest.name}.", suffix=".part", dir=dest.parent, delete=False
+    ) as f:
+        tmp = Path(f.name)
+    try:
+        yield tmp
+        tmp.replace(dest)
+    finally:
+        tmp.unlink(missing_ok=True)
+
+
+def atomic_write_text(dest: Path, text: str) -> None:
+    """Write UTF-8 text without exposing an incomplete record to readers."""
+    with staged_path(dest) as tmp:
+        tmp.write_text(text, encoding="utf-8")

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/cache.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import hashlib
 import os
+import re
 from pathlib import Path
 
 from usdata.models import Asset
@@ -22,9 +23,20 @@ def cache_dir() -> Path:
 
 def asset_path(asset: Asset, root: Path | None = None) -> Path:
     """Where an asset lives in the cache: <root>/<provider>/<dataset>/<asset id>."""
-    provider, name = asset.dataset_id.split(":", 1)
+    provider, sep, name = asset.dataset_id.partition(":")
+    if not sep or any(
+        not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*", part) for part in (provider, name)
+    ):
+        raise ValueError(f"unsafe dataset id: {asset.dataset_id!r}")
     safe_id = asset.id.replace("/", "_")
-    return (root or cache_dir()) / provider / name / safe_id
+    if not safe_id or safe_id in {".", ".."} or "\\" in safe_id or "\x00" in safe_id:
+        raise ValueError(f"unsafe asset id: {asset.id!r}")
+    base = (root or cache_dir()).expanduser().resolve()
+    path = base / provider / name / safe_id
+    for candidate in (path, path.with_name(path.name + ".provenance.json")):
+        if not candidate.resolve().is_relative_to(base):
+            raise ValueError(f"cache path escapes root: {candidate}")
+    return path
 
 
 def sha256_file(path: Path, chunk_size: int = 1 << 20) -> str:

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/cli/app.py

@@ -9,7 +9,8 @@ import httpx
 import typer
 
 from usdata import __version__, build_query, default_registry
-from usdata.fetch import ChecksumMismatch, fetch_asset
+from usdata.fetch import ChecksumMismatch
+from usdata.fetch import fetch as fetch_query
 from usdata.manifest import lockfile_path
 from usdata.providers import load_adapter
 from usdata.providers.base import NotImplementedProvider
@@ -59,8 +60,8 @@ def search(
     """Search the curated dataset registry."""
     try:
         query = build_query(text, provider=provider, location=state, start=start, end=end)
-    except UnknownPlace as e:
-        typer.secho(f"Unknown place: {e}", err=True, fg="red")
+    except ValueError as e:
+        typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
     results = default_registry().search(query, include_planned=planned)
     if not results:
@@ -152,21 +153,21 @@ def fetch(
             variables=[v.strip() for v in variables.split(",")] if variables else None,
             **params,
         )
-        adapter = load_adapter(ds)
-        assets = adapter.list_assets(query)
         if dry_run:
+            with load_adapter(ds) as adapter:
+                assets = adapter.list_assets(query)
             for a in assets:
                 typer.echo(f"{a.id}\t{a.href}")
             typer.echo(f"{len(assets)} asset(s) matched", err=True)
             return
-        fetched = [fetch_asset(ds, a, root=cache_dir, force=force) for a in assets]
+        fetched = fetch_query(ds, query, root=cache_dir, force=force)
     except (DatasetNotFound, UnknownPlace, ValueError) as e:
         typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
     except NotImplementedProvider as e:
         typer.secho(str(e), err=True, fg="yellow")
         raise typer.Exit(code=3) from None
-    except httpx.HTTPError as e:
+    except (httpx.HTTPError, ChecksumMismatch) as e:
         typer.secho(f"request failed: {e}", err=True, fg="red")
         raise typer.Exit(code=4) from None
     if not fetched:
@@ -189,7 +190,7 @@ def pull(
     """Fetch every source in a manifest and write (or restore from) its lockfile."""
     try:
         result = pull_manifest(manifest, root=cache_dir, force=force)
-    except (UnknownDatasets, ManifestChanged, UnknownPlace, ValueError) as e:
+    except (DatasetNotFound, UnknownDatasets, ManifestChanged, UnknownPlace, ValueError) as e:
         typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
     except NotImplementedProvider as e:
@@ -215,7 +216,11 @@ def verify(
     if not lock.exists():
         typer.secho(f"no lockfile at {lock}; run pull first", err=True, fg="red")
         raise typer.Exit(code=2)
-    drift = verify_manifest(manifest, root=cache_dir)
+    try:
+        drift = verify_manifest(manifest, root=cache_dir)
+    except (ValueError, OSError) as e:
+        typer.secho(str(e), err=True, fg="red")
+        raise typer.Exit(code=2) from None
     for d in drift:
         typer.echo(f"{d.asset_id}\t{d.problem}\t{d.path}")
     if drift:

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/data/registry.yaml

@@ -87,7 +87,7 @@ datasets:
     provider: noaa
     status: stub
     domain: satellite-oceanography
-    target: "0.3"
+    target: "0.5"
     title: CoastWatch Blended Sea Surface Temperature
     description: >-
       NOAA geo-polar blended daily SST analysis (day and night) on a global
@@ -105,7 +105,7 @@ datasets:
     provider: noaa
     status: planned
     domain: weather-satellites
-    target: "0.4"
+    target: "0.5"
     title: GOES-R ABI Satellite Imagery
     description: >-
       Advanced Baseline Imager products from GOES-16, 18, and 19 in the public
@@ -139,7 +139,7 @@ datasets:
     provider: noaa
     status: planned
     domain: severe-weather
-    target: "0.4"
+    target: "0.5"
     title: Storm Events Database
     description: >-
       NCEI's record of significant weather events since 1950 (tornadoes, hail,
@@ -157,7 +157,7 @@ datasets:
     provider: noaa
     status: planned
     domain: tropical-cyclones
-    target: "0.4"
+    target: "0.5"
     title: HURDAT2 Atlantic and Pacific Best Tracks
     description: >-
       National Hurricane Center best-track database: six-hourly position,
@@ -175,7 +175,7 @@ datasets:
     provider: noaa
     status: planned
     domain: tropical-cyclones
-    target: "0.4"
+    target: "0.5"
     title: IBTrACS Global Tropical Cyclone Tracks
     description: >-
       International Best Track Archive for Climate Stewardship: merged best
@@ -538,7 +538,7 @@ datasets:
     provider: usgs
     status: planned
     domain: water-resources
-    target: "0.4"
+    target: "0.5"
     title: Streamflow and Water Daily Values
     description: >-
       Daily statistics (streamflow, gage height, temperature) for USGS
@@ -587,7 +587,7 @@ datasets:
     provider: nasa
     status: planned
     domain: weather-satellites
-    target: "0.4"
+    target: "0.6"
     title: GPM IMERG Precipitation
     description: >-
       Global half-hourly and daily merged satellite precipitation estimates.

usdata-0.4.0/src/usdata/fetch.py

@@ -0,0 +1,79 @@
+"""Core fetch loop: resolve a query, download assets through the cache, record provenance."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from pydantic import BaseModel
+
+from usdata import provenance
+from usdata._files import staged_path
+from usdata.cache import asset_path, sha256_file
+from usdata.models import Asset, Dataset, Provenance, Query
+from usdata.providers import Provider, load_adapter
+
+
+class ChecksumMismatch(RuntimeError):
+    """A fetched file's sha256 did not match the checksum the adapter declared."""
+
+
+class FetchedAsset(BaseModel):
+    """One asset on disk with its provenance and whether the cache satisfied it."""
+
+    asset: Asset
+    path: Path
+    provenance: Provenance
+    from_cache: bool
+
+
+def _fetch_asset(
+    dataset: Dataset,
+    asset: Asset,
+    adapter: Provider,
+    *,
+    root: Path | None = None,
+    force: bool = False,
+) -> FetchedAsset:
+    if asset.dataset_id != dataset.id:
+        raise ValueError(f"asset dataset {asset.dataset_id!r} does not match {dataset.id!r}")
+    path = asset_path(asset, root)
+    if not force and path.is_file():
+        try:
+            prov = provenance.read(path)
+        except (ValueError, OSError):
+            prov = None
+        if (
+            prov is not None
+            and prov.dataset_id == dataset.id
+            and prov.provider == dataset.provider
+            and prov.source_url == asset.href
+            and prov.size == path.stat().st_size
+            and (asset.checksum is None or prov.checksum == asset.checksum)
+            and sha256_file(path) == prov.checksum
+        ):
+            return FetchedAsset(asset=asset, path=path, provenance=prov, from_cache=True)
+    with staged_path(path) as tmp:
+        adapter.fetch(asset, tmp)
+        prov = provenance.record(dataset, asset, tmp)
+        if asset.checksum and prov.checksum != asset.checksum:
+            raise ChecksumMismatch(f"{asset.id}: expected {asset.checksum}, got {prov.checksum}")
+    # A crash between replacements leaves a detectable mismatch, never a trusted partial file.
+    provenance.write(prov, path)
+    return FetchedAsset(asset=asset, path=path, provenance=prov, from_cache=False)
+
+
+def fetch_asset(
+    dataset: Dataset, asset: Asset, *, root: Path | None = None, force: bool = False
+) -> FetchedAsset:
+    """Fetch one asset, reusing the cache only when bytes and provenance agree."""
+    with load_adapter(dataset) as adapter:
+        return _fetch_asset(dataset, asset, adapter, root=root, force=force)
+
+
+def fetch(
+    dataset: Dataset, query: Query, *, root: Path | None = None, force: bool = False
+) -> list[FetchedAsset]:
+    """Resolve and fetch a query, sharing one adapter and closing its owned resources."""
+    with load_adapter(dataset) as adapter:
+        assets = adapter.list_assets(query)
+        return [_fetch_asset(dataset, a, adapter, root=root, force=force) for a in assets]

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/manifest.py

@@ -7,8 +7,9 @@ from pathlib import Path
 from typing import Any
 
 import yaml
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, ConfigDict, Field, model_validator
 
+from usdata._files import atomic_write_text
 from usdata.models import Asset, BBox, Provenance, Query
 from usdata.query import build_query
 from usdata.registry import Registry, default_registry
@@ -17,6 +18,8 @@ from usdata.registry import Registry, default_registry
 class SourceSpec(BaseModel):
     """One entry under ``sources:`` in a manifest."""
 
+    model_config = ConfigDict(extra="forbid")
+
     dataset: str
     location: str | None = None
     bbox: BBox | None = None
@@ -25,6 +28,13 @@ class SourceSpec(BaseModel):
     variables: list[str] = Field(default_factory=list)
     params: dict[str, Any] = Field(default_factory=dict)
 
+    @model_validator(mode="after")
+    def _reserved_params(self) -> SourceSpec:
+        reserved = {"location", "bbox", "start", "end", "variables"}
+        if overlap := reserved.intersection(self.params):
+            raise ValueError(f"params contains reserved query fields: {', '.join(sorted(overlap))}")
+        return self
+
     def to_query(self) -> Query:
         """Build the Query this source resolves to."""
         return build_query(
@@ -40,6 +50,8 @@ class SourceSpec(BaseModel):
 class Manifest(BaseModel):
     """A declarative list of inputs a project needs: usdata pull fetches them."""
 
+    model_config = ConfigDict(extra="forbid")
+
     name: str
     version: str = "1.0"
     sources: list[SourceSpec] = Field(min_length=1)
@@ -47,7 +59,11 @@ class Manifest(BaseModel):
     @classmethod
     def load(cls, path: Path) -> Manifest:
         """Parse a manifest YAML file."""
-        return cls.model_validate(yaml.safe_load(path.read_text()) or {})
+        try:
+            raw = yaml.safe_load(path.read_text())
+        except yaml.YAMLError as e:
+            raise ValueError(f"invalid manifest YAML in {path}: {e}") from e
+        return cls.model_validate(raw or {})
 
     def validate_against(self, registry: Registry | None = None) -> list[str]:
         """Return the dataset ids referenced by this manifest that the registry lacks."""
@@ -61,6 +77,17 @@ class LockedAsset(BaseModel):
     asset: Asset
     provenance: Provenance
 
+    @model_validator(mode="after")
+    def _consistent(self) -> LockedAsset:
+        if (
+            self.asset.dataset_id != self.provenance.dataset_id
+            or self.asset.href != self.provenance.source_url
+        ):
+            raise ValueError("locked asset and provenance must identify the same source")
+        if self.asset.checksum is not None and self.asset.checksum != self.provenance.checksum:
+            raise ValueError("locked asset and provenance checksums must agree")
+        return self
+
 
 class Lockfile(BaseModel):
     """Exactly what a manifest resolved to, with checksums, so it can be reproduced."""
@@ -78,7 +105,7 @@ class Lockfile(BaseModel):
 
     def save(self, path: Path) -> None:
         """Write the lockfile as indented JSON."""
-        path.write_text(self.model_dump_json(indent=2))
+        atomic_write_text(path, self.model_dump_json(indent=2))
 
 
 def lockfile_path(manifest_path: Path) -> Path:

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/protocols/http.py

@@ -8,6 +8,7 @@ from typing import Any
 import httpx
 
 from usdata import __version__
+from usdata._files import staged_path
 
 USER_AGENT = f"usdata/{__version__} (+https://github.com/jakeryderv/usdata)"
 DEFAULT_TIMEOUT = httpx.Timeout(10.0, read=120.0)
@@ -23,20 +24,14 @@ def client(**kwargs: Any) -> httpx.Client:
 
 def download(url: str, dest: Path, http: httpx.Client | None = None) -> Path:
     """Stream ``url`` to ``dest`` atomically. Raises ``httpx.HTTPStatusError`` on 4xx/5xx."""
-    dest.parent.mkdir(parents=True, exist_ok=True)
-    tmp = dest.with_name(dest.name + ".part")
     own = http is None
     http = http or client()
     try:
-        with http.stream("GET", url) as resp:
+        with staged_path(dest) as tmp, http.stream("GET", url) as resp:
             resp.raise_for_status()
             with tmp.open("wb") as f:
                 for chunk in resp.iter_bytes():
                     f.write(chunk)
-        tmp.replace(dest)
-    except BaseException:
-        tmp.unlink(missing_ok=True)
-        raise
     finally:
         if own:
             http.close()

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/provenance.py

@@ -6,6 +6,7 @@ from datetime import UTC, datetime
 from pathlib import Path
 
 from usdata import __version__
+from usdata._files import atomic_write_text
 from usdata.cache import sha256_file
 from usdata.models import Asset, Dataset, Provenance
 
@@ -34,7 +35,7 @@ def sidecar_path(path: Path) -> Path:
 def write(prov: Provenance, path: Path) -> Path:
     """Write a provenance record beside ``path`` and return the sidecar path."""
     out = sidecar_path(path)
-    out.write_text(prov.model_dump_json(indent=2))
+    atomic_write_text(out, prov.model_dump_json(indent=2))
     return out
 
 

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/providers/base.py

@@ -5,6 +5,8 @@ from __future__ import annotations
 import importlib
 from abc import ABC, abstractmethod
 from pathlib import Path
+from types import TracebackType
+from typing import Self
 
 from usdata.models import Asset, Dataset, Query
 
@@ -23,6 +25,21 @@ class Provider(ABC):
     def __init__(self, dataset: Dataset) -> None:
         self.dataset = dataset
 
+    def __enter__(self) -> Self:
+        return self
+
+    def __exit__(
+        self,
+        exc_type: type[BaseException] | None,
+        exc: BaseException | None,
+        traceback: TracebackType | None,
+    ) -> None:
+        self.close()
+
+    def close(self) -> None:
+        """Release owned resources. Providers with resources override this method."""
+        return None
+
     @abstractmethod
     def list_assets(self, query: Query) -> list[Asset]:
         """Resolve a query to the concrete objects that satisfy it, without downloading."""

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/providers/noaa/coastwatch.py

@@ -1,4 +1,4 @@
-"""CoastWatch SST via ERDDAP. Registered in the registry; implementation is planned for v0.1."""
+"""CoastWatch SST via ERDDAP. Registered in the registry; implementation is planned for v0.5."""
 
 from __future__ import annotations
 

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/providers/noaa/ghcnd.py

@@ -44,6 +44,13 @@ class GhcnDaily(Provider):
     def __init__(self, dataset: Dataset, client: httpx.Client | None = None) -> None:
         super().__init__(dataset)
         self._client = client
+        self._owns_client = client is None
+
+    def close(self) -> None:
+        """Close an internally created HTTP client; injected clients belong to the caller."""
+        if self._owns_client and self._client is not None:
+            self._client.close()
+            self._client = None
 
     def _http(self) -> httpx.Client:
         if self._client is None:

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/providers/noaa/nexrad.py

@@ -47,6 +47,13 @@ class NexradLevel2(Provider):
     def __init__(self, dataset: Dataset, client: httpx.Client | None = None) -> None:
         super().__init__(dataset)
         self._client = client
+        self._owns_client = client is None
+
+    def close(self) -> None:
+        """Close an internally created HTTP client; injected clients belong to the caller."""
+        if self._owns_client and self._client is not None:
+            self._client.close()
+            self._client = None
 
     def _http(self) -> httpx.Client:
         if self._client is None:
@@ -59,7 +66,10 @@ class NexradLevel2(Provider):
         if raw is not None:
             ids = _sites_param(raw)
             for sid in ids:
-                sites.get_site(sid)  # validate
+                try:
+                    sites.get_site(sid)
+                except KeyError as e:
+                    raise QueryError(str(e)) from e
             return ids
         if query.bbox is None:
             raise QueryError(f"{self.dataset.id} needs a location, bbox, lat/lon, or site=...")

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/pull.py

@@ -10,6 +10,7 @@ fetching anything.
 
 from __future__ import annotations
 
+from contextlib import ExitStack
 from datetime import UTC, datetime
 from pathlib import Path
 
@@ -17,9 +18,9 @@ from pydantic import BaseModel
 
 from usdata import __version__, provenance
 from usdata.cache import asset_path, sha256_file
-from usdata.fetch import ChecksumMismatch, FetchedAsset, fetch_asset
+from usdata.fetch import ChecksumMismatch, FetchedAsset, _fetch_asset, fetch
 from usdata.manifest import LockedAsset, Lockfile, Manifest, lockfile_path
-from usdata.providers import load_adapter
+from usdata.providers import Provider, load_adapter
 from usdata.registry import Registry, default_registry
 
 
@@ -67,10 +68,9 @@ def resolve(
     locked: list[LockedAsset] = []
     for source in manifest.sources:
         dataset = reg.get(source.dataset)
-        for asset in load_adapter(dataset).list_assets(source.to_query()):
-            item = fetch_asset(dataset, asset, root=root)
+        for item in fetch(dataset, source.to_query(), root=root):
             fetched.append(item)
-            pinned = asset.model_copy(update={"checksum": item.provenance.checksum})
+            pinned = item.asset.model_copy(update={"checksum": item.provenance.checksum})
             locked.append(LockedAsset(asset=pinned, provenance=item.provenance))
     lock = Lockfile(
         manifest=manifest.name,
@@ -97,18 +97,25 @@ def restore(
             "pull with force to re-resolve"
         )
     fetched: list[FetchedAsset] = []
-    for entry in lock.assets:
-        dataset = reg.get(entry.asset.dataset_id)
-        path = asset_path(entry.asset, root)
-        if path.exists() and sha256_file(path) == entry.provenance.checksum:
-            fetched.append(
-                FetchedAsset(
-                    asset=entry.asset, path=path, provenance=entry.provenance, from_cache=True
+    adapters: dict[str, Provider] = {}
+    with ExitStack() as stack:
+        for entry in lock.assets:
+            dataset = reg.get(entry.asset.dataset_id)
+            path = asset_path(entry.asset, root)
+            if path.is_file() and sha256_file(path) == entry.provenance.checksum:
+                provenance.write(entry.provenance, path)
+                fetched.append(
+                    FetchedAsset(
+                        asset=entry.asset, path=path, provenance=entry.provenance, from_cache=True
+                    )
                 )
+                continue
+            if dataset.id not in adapters:
+                adapters[dataset.id] = stack.enter_context(load_adapter(dataset))
+            pinned = entry.asset.model_copy(update={"checksum": entry.provenance.checksum})
+            fetched.append(
+                _fetch_asset(dataset, pinned, adapters[dataset.id], root=root, force=True)
             )
-            continue
-        # fetch_asset verifies against asset.checksum, which resolve() pinned.
-        fetched.append(fetch_asset(dataset, entry.asset, root=root, force=True))
     return PullResult(lockfile=lock, lockfile_path=lock_path, fetched=fetched, from_lockfile=True)
 
 

{usdata-0.3.0 → usdata-0.4.0}/src/usdata/registry.py

@@ -82,11 +82,12 @@ class Registry:
             self._by_id[ds.id] = ds
         self._providers = {p.id: p for p in providers}
         self._domains = {d.id: d for d in domains}
+        explicit_domains = bool(self._domains)
         for ds in self._by_id.values():
             self._providers.setdefault(
                 ds.provider, ProviderInfo(id=ds.provider, name=ds.provider.upper())
             )
-            if self._domains and ds.domain not in self._domains:
+            if explicit_domains and ds.domain not in self._domains:
                 raise ValueError(f"{ds.id}: unknown domain {ds.domain!r}")
             self._domains.setdefault(ds.domain, DomainInfo(id=ds.domain, name=ds.domain))
 

usdata-0.3.0/src/usdata/fetch.py

@@ -1,54 +0,0 @@
-"""Core fetch loop: resolve a query, download assets through the cache, record provenance."""
-
-from __future__ import annotations
-
-from pathlib import Path
-
-from pydantic import BaseModel
-
-from usdata import provenance
-from usdata.cache import asset_path, sha256_file
-from usdata.models import Asset, Dataset, Provenance, Query
-from usdata.providers import load_adapter
-
-
-class ChecksumMismatch(RuntimeError):
-    """A fetched file's sha256 did not match the checksum the adapter declared."""
-
-    pass
-
-
-class FetchedAsset(BaseModel):
-    """One asset on disk with its provenance and whether the cache satisfied it."""
-
-    asset: Asset
-    path: Path
-    provenance: Provenance
-    from_cache: bool
-
-
-def fetch_asset(
-    dataset: Dataset, asset: Asset, *, root: Path | None = None, force: bool = False
-) -> FetchedAsset:
-    """Fetch one asset via its provider unless it is already cached with a provenance sidecar."""
-    path = asset_path(asset, root)
-    if not force and path.exists() and provenance.sidecar_path(path).exists():
-        return FetchedAsset(
-            asset=asset, path=path, provenance=provenance.read(path), from_cache=True
-        )
-    path.parent.mkdir(parents=True, exist_ok=True)
-    load_adapter(dataset).fetch(asset, path)
-    if asset.checksum and (got := sha256_file(path)) != asset.checksum:
-        path.unlink(missing_ok=True)
-        raise ChecksumMismatch(f"{asset.id}: expected {asset.checksum}, got {got}")
-    prov = provenance.record(dataset, asset, path)
-    provenance.write(prov, path)
-    return FetchedAsset(asset=asset, path=path, provenance=prov, from_cache=False)
-
-
-def fetch(
-    dataset: Dataset, query: Query, *, root: Path | None = None, force: bool = False
-) -> list[FetchedAsset]:
-    """Resolve ``query`` against ``dataset`` and fetch everything it matches."""
-    assets = load_adapter(dataset).list_assets(query)
-    return [fetch_asset(dataset, a, root=root, force=force) for a in assets]