usdata 0.5.0__tar.gz → 0.7.0__tar.gz

{usdata-0.5.0 → usdata-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.5.0
+Version: 0.7.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -14,9 +14,11 @@ Requires-Dist: httpx>=0.28.1
 Requires-Dist: pydantic>=2.7
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: typer>=0.12
+Requires-Dist: pandas>=3.0 ; extra == 'pandas'
 Requires-Python: >=3.11
 Project-URL: Homepage, https://github.com/jakeryderv/usdata
 Project-URL: Repository, https://github.com/jakeryderv/usdata
+Provides-Extra: pandas
 Description-Content-Type: text/markdown
 
 # usdata
@@ -24,22 +26,23 @@ Description-Content-Type: text/markdown
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.5 supports GHCN-Daily, NEXRAD Level II, USGS daily
-> values, and CoastWatch SST subsets with provenance, plus Census state/county
-> lookup. Other datasets are planned.
+> Status: pre-alpha. v0.7 supports GHCN-Daily, GSOM monthly summaries,
+> NEXRAD Level II, USGS daily values, and CoastWatch SST subsets with provenance,
+> plus Census state/county lookup, optional pandas CSV readers, and terminal
+> download progress. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
 
 <!-- registry:start -->
-| Provider | Available | Stub | Planned | Next up (0.6) | Datasets |
+| Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | `gfs`, `hrrr`, `oisst`, `etopo` | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 4 | 0 | 25 | — | `ghcn-daily`, `gsom`, `nexrad-level2`, `coastwatch-sst`, +25 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
 | [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
-| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | `gpm-imerg` | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
 | [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
 
 Available datasets are in `code`, stubs in _italics_; planned ones are counted. Available means implemented in this source checkout; consult the [releases](https://github.com/jakeryderv/usdata/releases) for published support. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
@@ -130,6 +133,24 @@ sources:
     end: 2024-05-31
 ```
 
+Terminal progress is available since v0.7. On a terminal, `fetch` and
+`pull` show progress on stderr: resolved asset counts,
+known bytes and unknown sizes, HTTP download bytes for the current attempt, and
+validated cache hits. `fetch --dry-run` also summarizes known sizes. Asset totals
+include possible cache hits; each manifest source is resolved separately. Bytes
+from a failed HTTP attempt reset on retry; encoded responses have unknown decoded
+size. Adapters that assemble files from metadata requests show asset-level progress.
+Use `--no-progress` to disable it. Progress is automatically disabled when either
+stdout or stderr is redirected; existing output lines and exit codes are unchanged.
+
+## Opening CSV data
+
+`FetchedAsset.open()` is available since v0.6 with the optional pandas
+extra (`pip install "usdata[pandas]"`). It reads cached CSV into a DataFrame,
+preserves identifier strings, and keeps CoastWatch units as metadata.
+See the [reader reference](docs/reference/readers.md)
+and [fetch → open → analyze example](examples/sst-analysis/README.md).
+
 ## Development
 
 Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
@@ -138,16 +159,18 @@ Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
-just check     # format, lint, typecheck, offline tests, generated docs
+just check     # format, lint, typecheck, offline tests, generated docs, release notices
+just check-pandas  # install the CSV extra and run the same checks
 just build     # build wheel and sdist
-just smoke     # install and exercise the built wheel outside the checkout
+just smoke     # exercise core and pandas wheel installations outside the checkout
 just run search radar
 ```
 
 Unit tests mechanically block network connections. Integration tests that hit
 live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
-Linux and smoke-tests the installed wheel on Linux, macOS, and Windows. The
-full unit and live-service suites currently run on Linux.
+Linux, both with and without pandas, and smoke-tests both installed-wheel
+profiles on Linux, macOS, and Windows. The full unit and live-service suites currently run on Linux. `just setup` restores
+a core-only development environment; `just check-pandas` installs the extra.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.5.0 → usdata-0.7.0}/README.md

@@ -3,22 +3,23 @@
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.5 supports GHCN-Daily, NEXRAD Level II, USGS daily
-> values, and CoastWatch SST subsets with provenance, plus Census state/county
-> lookup. Other datasets are planned.
+> Status: pre-alpha. v0.7 supports GHCN-Daily, GSOM monthly summaries,
+> NEXRAD Level II, USGS daily values, and CoastWatch SST subsets with provenance,
+> plus Census state/county lookup, optional pandas CSV readers, and terminal
+> download progress. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
 
 <!-- registry:start -->
-| Provider | Available | Stub | Planned | Next up (0.6) | Datasets |
+| Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | `gfs`, `hrrr`, `oisst`, `etopo` | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 4 | 0 | 25 | — | `ghcn-daily`, `gsom`, `nexrad-level2`, `coastwatch-sst`, +25 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
 | [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
-| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | `gpm-imerg` | +1 planned |
+| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
 | [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |
 
 Available datasets are in `code`, stubs in _italics_; planned ones are counted. Available means implemented in this source checkout; consult the [releases](https://github.com/jakeryderv/usdata/releases) for published support. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
@@ -109,6 +110,24 @@ sources:
     end: 2024-05-31
 ```
 
+Terminal progress is available since v0.7. On a terminal, `fetch` and
+`pull` show progress on stderr: resolved asset counts,
+known bytes and unknown sizes, HTTP download bytes for the current attempt, and
+validated cache hits. `fetch --dry-run` also summarizes known sizes. Asset totals
+include possible cache hits; each manifest source is resolved separately. Bytes
+from a failed HTTP attempt reset on retry; encoded responses have unknown decoded
+size. Adapters that assemble files from metadata requests show asset-level progress.
+Use `--no-progress` to disable it. Progress is automatically disabled when either
+stdout or stderr is redirected; existing output lines and exit codes are unchanged.
+
+## Opening CSV data
+
+`FetchedAsset.open()` is available since v0.6 with the optional pandas
+extra (`pip install "usdata[pandas]"`). It reads cached CSV into a DataFrame,
+preserves identifier strings, and keeps CoastWatch units as metadata.
+See the [reader reference](docs/reference/readers.md)
+and [fetch → open → analyze example](examples/sst-analysis/README.md).
+
 ## Development
 
 Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
@@ -117,16 +136,18 @@ Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
-just check     # format, lint, typecheck, offline tests, generated docs
+just check     # format, lint, typecheck, offline tests, generated docs, release notices
+just check-pandas  # install the CSV extra and run the same checks
 just build     # build wheel and sdist
-just smoke     # install and exercise the built wheel outside the checkout
+just smoke     # exercise core and pandas wheel installations outside the checkout
 just run search radar
 ```
 
 Unit tests mechanically block network connections. Integration tests that hit
 live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
-Linux and smoke-tests the installed wheel on Linux, macOS, and Windows. The
-full unit and live-service suites currently run on Linux.
+Linux, both with and without pandas, and smoke-tests both installed-wheel
+profiles on Linux, macOS, and Windows. The full unit and live-service suites currently run on Linux. `just setup` restores
+a core-only development environment; `just check-pandas` installs the extra.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.5.0 → usdata-0.7.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.5.0"
+version = "0.7.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -30,6 +30,9 @@ dependencies = [
 name = "Jake Van Slyke"
 email = "jakervanslyke@gmail.com"
 
+[project.optional-dependencies]
+pandas = ["pandas>=3.0"]
+
 [project.urls]
 Homepage = "https://github.com/jakeryderv/usdata"
 Repository = "https://github.com/jakeryderv/usdata"

{usdata-0.5.0 → usdata-0.7.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.5.0"
+version = "0.7.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -22,6 +22,9 @@ dependencies = [
     "typer>=0.12",
 ]
 
+[project.optional-dependencies]
+pandas = ["pandas>=3.0"]
+
 [project.urls]
 Homepage = "https://github.com/jakeryderv/usdata"
 Repository = "https://github.com/jakeryderv/usdata"

usdata-0.7.0/src/usdata/_progress.py

@@ -0,0 +1,62 @@
+"""Internal synchronous progress events, scoped to one CLI operation."""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Iterator, Sequence
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass
+from typing import Literal
+
+
+@dataclass(frozen=True)
+class Batch:
+    """A resolved group; sizes describe assets, including possible cache hits."""
+
+    count: int
+    known_bytes: int
+    unknown_sizes: int
+
+
+@dataclass(frozen=True)
+class AssetProgress:
+    """Start or validated completion of an asset."""
+
+    asset_id: str
+    state: Literal["start", "cached", "fetched"]
+    size: int | None
+
+
+@dataclass(frozen=True)
+class TransferProgress:
+    """Bytes written in the current HTTP attempt, reset on every retry."""
+
+    completed: int
+    total: int | None
+    attempt: int
+
+
+Event = Batch | AssetProgress | TransferProgress
+_observer: ContextVar[Callable[[Event], None] | None] = ContextVar("progress", default=None)
+
+
+def emit(event: Event) -> None:
+    """Notify the active observer without importing CLI code."""
+    observer = _observer.get()
+    if observer is not None:
+        observer(event)
+
+
+def batch(sizes: Sequence[int | None]) -> None:
+    """Report known and unknown sizes separately; never guess a total."""
+    emit(Batch(len(sizes), sum(size for size in sizes if size is not None), sizes.count(None)))
+
+
+@contextmanager
+def observe(callback: Callable[[Event], None]) -> Iterator[None]:
+    """Observe an operation and restore the previous observer on every exit."""
+    token = _observer.set(callback)
+    try:
+        yield
+    finally:
+        _observer.reset(token)

{usdata-0.5.0 → usdata-0.7.0}/src/usdata/cli/app.py

@@ -9,6 +9,8 @@ import httpx
 import typer
 
 from usdata import __version__, build_query, default_registry
+from usdata._progress import batch
+from usdata.cli.progress import progress
 from usdata.fetch import ChecksumMismatch
 from usdata.fetch import fetch as fetch_query
 from usdata.manifest import lockfile_path
@@ -126,6 +128,7 @@ def fetch(
     ] = None,
     cache_dir: Annotated[Path | None, typer.Option(help="Override the cache directory.")] = None,
     force: Annotated[bool, typer.Option(help="Re-download even if cached.")] = False,
+    no_progress: Annotated[bool, typer.Option(help="Disable terminal progress.")] = False,
     dry_run: Annotated[
         bool, typer.Option(help="List matching assets without downloading.")
     ] = False,
@@ -165,8 +168,11 @@ def fetch(
             for a in assets:
                 typer.echo(f"{a.id}\t{a.href}")
             typer.echo(f"{len(assets)} asset(s) matched", err=True)
+            with progress(disabled=no_progress):
+                batch([asset.size for asset in assets])
             return
-        fetched = fetch_query(ds, query, root=cache_dir, force=force)
+        with progress(disabled=no_progress):
+            fetched = fetch_query(ds, query, root=cache_dir, force=force)
     except (DatasetNotFound, UnknownPlace, ValueError) as e:
         typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
@@ -192,10 +198,12 @@ def pull(
         bool,
         typer.Option(help="Ignore an existing lockfile: re-resolve every source and rewrite it."),
     ] = False,
+    no_progress: Annotated[bool, typer.Option(help="Disable terminal progress.")] = False,
 ) -> None:
     """Fetch every source in a manifest and write (or restore from) its lockfile."""
     try:
-        result = pull_manifest(manifest, root=cache_dir, force=force)
+        with progress(disabled=no_progress):
+            result = pull_manifest(manifest, root=cache_dir, force=force)
     except EmptySource as e:
         typer.secho(str(e), err=True, fg="yellow")
         raise typer.Exit(code=1) from None

usdata-0.7.0/src/usdata/cli/progress.py

@@ -0,0 +1,89 @@
+"""Terminal-only progress rendering; normal CLI output remains machine readable."""
+
+from __future__ import annotations
+
+import shutil
+import sys
+from collections.abc import Iterator
+from contextlib import contextmanager
+from time import monotonic
+from typing import TextIO
+
+from usdata._progress import AssetProgress, Batch, Event, observe
+
+
+def _interactive() -> bool:
+    return sys.stdout.isatty() and sys.stderr.isatty()
+
+
+class _Display:
+    def __init__(self, stream: TextIO) -> None:
+        self.stream = stream
+        self.width = 0
+        self.updated = 0.0
+        self.asset = ""
+        self.count = 0
+        self.done = 0
+        self.cached = 0
+
+    def clear(self) -> None:
+        if self.width:
+            self.stream.write("\r" + " " * self.width + "\r")
+            self.stream.flush()
+            self.width = 0
+
+    def line(self, text: str, *, final: bool = False) -> None:
+        self.clear()
+        # IDs come from providers: keep control characters out of the terminal.
+        text = "".join(c if c.isprintable() else "?" for c in text)
+        if final:
+            self.stream.write(text + "\n")
+        else:
+            text = text[: max(1, shutil.get_terminal_size().columns - 1)]
+            self.stream.write(text)
+            self.width = len(text)
+        self.stream.flush()
+        self.updated = monotonic()
+
+    def __call__(self, event: Event) -> None:
+        if isinstance(event, Batch):
+            self.count, self.done, self.cached = event.count, 0, 0
+            sizes = f"{event.known_bytes:,} known bytes"
+            if event.unknown_sizes:
+                sizes += f"; {event.unknown_sizes} size(s) unknown"
+            self.line(f"{event.count} asset(s) resolved; {sizes} (before cache checks)", final=True)
+        elif isinstance(event, AssetProgress):
+            self.asset = event.asset_id
+            if event.state == "start":
+                size = f"{event.size:,} bytes" if event.size is not None else "size unknown"
+                self.line(f"[{self.done}/{self.count}] checking cache ({size}) | {self.asset}")
+            else:
+                self.done += 1
+                self.cached += event.state == "cached"
+                self.line(
+                    f"[{self.done}/{self.count}] {event.state} "
+                    f"({event.size:,} bytes; {self.cached} cached) | {self.asset}",
+                    final=self.done == self.count,
+                )
+        else:
+            if event.completed and monotonic() - self.updated < 0.1:
+                return
+            size = f"{event.total:,}" if event.total is not None else "unknown"
+            self.line(
+                f"[{self.done}/{self.count}] {event.completed:,}/{size} bytes "
+                f"(attempt {event.attempt}) | {self.asset}"
+            )
+
+
+@contextmanager
+def progress(*, disabled: bool = False) -> Iterator[None]:
+    """Render progress on stderr only when both output streams are terminals."""
+    if disabled or not _interactive():
+        yield
+        return
+    display = _Display(sys.stderr)
+    with observe(display):
+        try:
+            yield
+        finally:
+            display.clear()

{usdata-0.5.0 → usdata-0.7.0}/src/usdata/data/registry.yaml

@@ -209,19 +209,22 @@ datasets:
 
   - id: noaa:gsom
     provider: noaa
-    status: planned
+    status: available
     domain: surface-weather
-    target: later
+    since: "0.7"
     title: Global Summary of the Month
     description: >-
       Monthly station summaries derived from GHCN-Daily (means, extremes,
       totals) via the NCEI Access Data Service dataset global-summary-of-the-month.
-      Shares the GHCN-Daily client and station search.
+      Selects whole calendar months and explicit stations, or discovers stations
+      through the companion search service.
     keywords: [climate, monthly, stations, temperature, precipitation, gsom, ncei]
     protocol: http
     homepage: https://www.ncei.noaa.gov/access/search/data-search/global-summary-of-the-month
     license: US Government Work (public domain)
     capabilities: { spatial_subset: false, temporal_subset: true, variable_subset: true }
+    spatial_extent: { west: -180.0, south: -90.0, east: 180.0, north: 90.0 }
+    adapter: usdata.providers.noaa.gsom:GlobalSummaryMonthly
 
   - id: noaa:gsoy
     provider: noaa
@@ -293,7 +296,7 @@ datasets:
     provider: noaa
     status: planned
     domain: weather-models
-    target: "0.6"
+    target: later
     title: HRRR Forecast Model Output
     description: >-
       High-Resolution Rapid Refresh 3 km hourly forecasts in GRIB2 from the
@@ -310,7 +313,7 @@ datasets:
     provider: noaa
     status: planned
     domain: weather-models
-    target: "0.6"
+    target: later
     title: GFS Forecast Model Output
     description: >-
       Global Forecast System output in GRIB2 from the public noaa-gfs-bdp-pds
@@ -342,7 +345,7 @@ datasets:
     provider: noaa
     status: planned
     domain: ocean-physics
-    target: "0.6"
+    target: later
     title: OISST Daily Sea Surface Temperature
     description: >-
       Optimum Interpolation SST v2.1: daily global 0.25 degree analysis since
@@ -375,7 +378,7 @@ datasets:
     provider: noaa
     status: planned
     domain: bathymetry
-    target: "0.6"
+    target: later
     title: ETOPO 2022 Global Relief
     description: >-
       Global topography and bathymetry at 15, 30, and 60 arc-seconds as
@@ -591,7 +594,7 @@ datasets:
     provider: nasa
     status: planned
     domain: weather-satellites
-    target: "0.6"
+    target: later
     title: GPM IMERG Precipitation
     description: >-
       Global half-hourly and daily merged satellite precipitation estimates.

{usdata-0.5.0 → usdata-0.7.0}/src/usdata/fetch.py

@@ -3,10 +3,11 @@
 from __future__ import annotations
 
 from pathlib import Path
+from typing import Any
 
 from pydantic import BaseModel
 
-from usdata import provenance
+from usdata import _progress, provenance
 from usdata._files import staged_path
 from usdata.cache import asset_path, sha256_file
 from usdata.models import Asset, Dataset, Provenance, Query
@@ -25,6 +26,27 @@ class FetchedAsset(BaseModel):
     provenance: Provenance
     from_cache: bool
 
+    def open(
+        self,
+        *,
+        reader: str | None = None,
+        dtype: dict[str, str] | None = None,
+        parse_dates: list[str] | None = None,
+        usecols: list[str] | None = None,
+        nrows: int | None = None,
+    ) -> Any:
+        """Open this local CSV as a DataFrame; requires the ``pandas`` extra.
+
+        ERDDAP units are kept in ``frame.attrs["units"]`` and source provenance
+        in ``frame.attrs["usdata"]``. See ``usdata.readers.open_asset`` for options.
+        Cached files and provenance sidecars are never changed.
+        """
+        from usdata.readers import open_asset
+
+        return open_asset(
+            self, reader=reader, dtype=dtype, parse_dates=parse_dates, usecols=usecols, nrows=nrows
+        )
+
 
 def _fetch_asset(
     dataset: Dataset,
@@ -36,6 +58,7 @@ def _fetch_asset(
 ) -> FetchedAsset:
     if asset.dataset_id != dataset.id:
         raise ValueError(f"asset dataset {asset.dataset_id!r} does not match {dataset.id!r}")
+    _progress.emit(_progress.AssetProgress(asset.id, "start", asset.size))
     path = asset_path(asset, root)
     if not force and path.is_file():
         try:
@@ -51,6 +74,7 @@ def _fetch_asset(
             and (asset.checksum is None or prov.checksum == asset.checksum)
             and sha256_file(path) == prov.checksum
         ):
+            _progress.emit(_progress.AssetProgress(asset.id, "cached", prov.size))
             return FetchedAsset(asset=asset, path=path, provenance=prov, from_cache=True)
     with staged_path(path) as tmp:
         adapter.fetch(asset, tmp)
@@ -59,6 +83,7 @@ def _fetch_asset(
             raise ChecksumMismatch(f"{asset.id}: expected {asset.checksum}, got {prov.checksum}")
     # A crash between replacements leaves a detectable mismatch, never a trusted partial file.
     provenance.write(prov, path)
+    _progress.emit(_progress.AssetProgress(asset.id, "fetched", prov.size))
     return FetchedAsset(asset=asset, path=path, provenance=prov, from_cache=False)
 
 
@@ -76,4 +101,5 @@ def fetch(
     """Resolve and fetch a query, sharing one adapter and closing its owned resources."""
     with load_adapter(dataset) as adapter:
         assets = adapter.list_assets(query)
+        _progress.batch([asset.size for asset in assets])
         return [_fetch_asset(dataset, a, adapter, root=root, force=force) for a in assets]

{usdata-0.5.0 → usdata-0.7.0}/src/usdata/protocols/http.py

@@ -11,7 +11,7 @@ from typing import Any, TypeVar
 
 import httpx
 
-from usdata import __version__
+from usdata import __version__, _progress
 from usdata._files import staged_path
 
 USER_AGENT = f"usdata/{__version__} (+https://github.com/jakeryderv/usdata)"
@@ -80,13 +80,30 @@ def download(url: str, dest: Path, http: httpx.Client | None = None) -> Path:
     """Download atomically, restarting interrupted GETs up to three total attempts."""
     own = http is None
     active = http or client()
+    attempt = 0
 
     def request() -> Path:
+        nonlocal attempt
+        attempt += 1
+        _progress.emit(_progress.TransferProgress(0, None, attempt))
         with staged_path(dest) as tmp, active.stream("GET", url) as resp:
             resp.raise_for_status()
+            length = resp.headers.get("Content-Length", "")
+            # iter_bytes writes decoded bytes; an encoded length is not comparable.
+            total = (
+                int(length)
+                if length.isascii()
+                and length.isdigit()
+                and resp.headers.get("Content-Encoding", "identity").lower() == "identity"
+                else None
+            )
+            completed = 0
+            _progress.emit(_progress.TransferProgress(completed, total, attempt))
             with tmp.open("wb") as f:
                 for chunk in resp.iter_bytes():
                     f.write(chunk)
+                    completed += len(chunk)
+                    _progress.emit(_progress.TransferProgress(completed, total, attempt))
         return dest
 
     try:

{usdata-0.5.0 → usdata-0.7.0}/src/usdata/providers/noaa/ghcnd.py

@@ -12,6 +12,7 @@ always goes through search first. Stations are chunked so URLs stay short.
 from __future__ import annotations
 
 import hashlib
+import logging
 from pathlib import Path
 from typing import Any
 
@@ -26,6 +27,7 @@ DATA_URL = "https://www.ncei.noaa.gov/access/services/data/v1"
 NCEI_DATASET = "daily-summaries"
 SEARCH_PAGE_SIZE = 1000
 STATIONS_PER_ASSET = 50
+logger = logging.getLogger(__name__)
 
 
 def _date(value: Any) -> str:
@@ -46,6 +48,8 @@ def _stations_param(raw: Any) -> list[str]:
 class GhcnDaily(Provider):
     """GHCN-Daily adapter. Params: ``stations`` (list or comma string), ``units``."""
 
+    ncei_dataset = NCEI_DATASET
+
     def __init__(self, dataset: Dataset, client: httpx.Client | None = None) -> None:
         super().__init__(dataset)
         self._client = client
@@ -68,7 +72,7 @@ class GhcnDaily(Provider):
             raise QueryError("station search needs a bounding box and a time range")
         b = query.bbox
         params: dict[str, Any] = {
-            "dataset": NCEI_DATASET,
+            "dataset": self.ncei_dataset,
             "bbox": f"{b.north},{b.west},{b.south},{b.east}",
             "startDate": _date(query.time.start),
             "endDate": _date(query.time.end),
@@ -84,12 +88,29 @@ class GhcnDaily(Provider):
             resp.raise_for_status()
             body = resp.json()
             results = body.get("results", [])
+            before = len(found)
             for result in results:
                 for station in result.get("stations", []):
                     sid = station.get("id")
                     if sid and sid not in seen:
                         seen.add(sid)
                         found.append(sid)
+            logger.debug(
+                "NCEI station search: url=%s status=%s content_type=%s "
+                "count=%r totalCount=%r results=%s new_stations=%s station_sample=%r",
+                resp.request.url,
+                resp.status_code,
+                resp.headers.get("content-type"),
+                body.get("count"),
+                body.get("totalCount"),
+                len(results),
+                len(found) - before,
+                found[before : before + 5],
+            )
+            if len(found) == before and logger.isEnabledFor(logging.DEBUG):
+                logger.debug(
+                    "NCEI search page yielded no new stations; response_prefix=%r", resp.text[:512]
+                )
             # "count" is the number matching this query; "totalCount" is dataset-wide.
             params["offset"] += SEARCH_PAGE_SIZE
             if not results or params["offset"] >= int(body.get("count", 0)):
@@ -101,7 +122,7 @@ class GhcnDaily(Provider):
         if query.time is None or query.time.start is None or query.time.end is None:
             raise QueryError(f"{self.dataset.id} requires both start and end dates")
         if unknown := set(query.params) - {"stations", "units"}:
-            raise QueryError(f"unsupported GHCN params: {', '.join(sorted(unknown))}")
+            raise QueryError(f"unsupported {self.dataset.id} params: {', '.join(sorted(unknown))}")
         if query.params.get("units", "metric") not in ("metric", "standard"):
             raise QueryError("units must be metric or standard")
         if "stations" in query.params:
@@ -118,7 +139,7 @@ class GhcnDaily(Provider):
         for i in range(0, len(stations), STATIONS_PER_ASSET):
             chunk = stations[i : i + STATIONS_PER_ASSET]
             params: dict[str, Any] = {
-                "dataset": NCEI_DATASET,
+                "dataset": self.ncei_dataset,
                 "stations": ",".join(chunk),
                 "startDate": start,
                 "endDate": end,
@@ -132,7 +153,7 @@ class GhcnDaily(Provider):
             digest = hashlib.sha1(url.encode()).hexdigest()[:12]
             assets.append(
                 Asset(
-                    id=f"{NCEI_DATASET}_{start}_{end}_{digest}.csv",
+                    id=f"{self.ncei_dataset}_{start}_{end}_{digest}.csv",
                     dataset_id=self.dataset.id,
                     href=url,
                     protocol=Protocol.HTTP,

usdata-0.7.0/src/usdata/providers/noaa/gsom.py

@@ -0,0 +1,49 @@
+"""Monthly station summaries through the NCEI Access Data Service.
+
+Params: ``stations`` (list or comma string) and ``units`` (metric or standard).
+Both dates are required; every UTC calendar month touched by the interval is
+selected in full. Geographic discovery uses the same normalized month bounds.
+"""
+
+from calendar import monthrange
+from datetime import UTC
+
+from usdata.models import Asset, Query, TimeRange
+from usdata.providers.base import QueryError
+from usdata.providers.noaa.ghcnd import GhcnDaily
+
+
+class GlobalSummaryMonthly(GhcnDaily):
+    """GSOM CSV subsets, sharing NCEI station discovery and transport with GHCN."""
+
+    ncei_dataset = "global-summary-of-the-month"
+
+    def _monthly_query(self, query: Query) -> Query:
+        if query.time is None or query.time.start is None or query.time.end is None:
+            raise QueryError(f"{self.dataset.id} requires both start and end dates")
+        start = query.time.start.replace(tzinfo=query.time.start.tzinfo or UTC).astimezone(UTC)
+        end = query.time.end.replace(tzinfo=query.time.end.tzinfo or UTC).astimezone(UTC)
+        return query.model_copy(
+            update={
+                "time": TimeRange(
+                    start=start.replace(day=1, hour=0, minute=0, second=0, microsecond=0),
+                    end=end.replace(
+                        day=monthrange(end.year, end.month)[1],
+                        hour=23,
+                        minute=59,
+                        second=59,
+                        microsecond=999999,
+                    ),
+                )
+            }
+        )
+
+    def find_stations(self, query: Query) -> list[str]:
+        """Find stations overlapping the selected complete calendar months."""
+        return super().find_stations(self._monthly_query(query))
+
+    def list_assets(self, query: Query) -> list[Asset]:
+        """Resolve monthly CSVs with stable URLs and complete-month asset bounds."""
+        if "stations" in query.params and query.bbox is not None:
+            raise QueryError("pass stations or a location/bbox, not both")
+        return super().list_assets(self._monthly_query(query))

{usdata-0.5.0 → usdata-0.7.0}/src/usdata/pull.py

@@ -16,7 +16,7 @@ from pathlib import Path
 
 from pydantic import BaseModel
 
-from usdata import __version__, provenance
+from usdata import __version__, _progress, provenance
 from usdata.cache import asset_path, sha256_file
 from usdata.fetch import ChecksumMismatch, FetchedAsset, _fetch_asset, fetch
 from usdata.manifest import LockedAsset, Lockfile, Manifest, lockfile_path
@@ -111,13 +111,21 @@ def restore(
     lock = Lockfile.load(lock_path)
     _check_manifest(manifest_path, lock)
     fetched: list[FetchedAsset] = []
+    _progress.batch([entry.provenance.size for entry in lock.assets])
     adapters: dict[str, Provider] = {}
     with ExitStack() as stack:
         for entry in lock.assets:
             dataset = reg.get(entry.asset.dataset_id)
             path = asset_path(entry.asset, root)
+            if path.is_file():
+                _progress.emit(
+                    _progress.AssetProgress(entry.asset.id, "start", entry.provenance.size)
+                )
             if path.is_file() and sha256_file(path) == entry.provenance.checksum:
                 provenance.write(entry.provenance, path)
+                _progress.emit(
+                    _progress.AssetProgress(entry.asset.id, "cached", entry.provenance.size)
+                )
                 fetched.append(
                     FetchedAsset(
                         asset=entry.asset, path=path, provenance=entry.provenance, from_cache=True

usdata-0.7.0/src/usdata/readers.py

@@ -0,0 +1,103 @@
+"""Optional readers for local fetched files; never fetch or modify cached bytes."""
+
+from __future__ import annotations
+
+import csv
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+from usdata.models import Protocol
+
+if TYPE_CHECKING:
+    from usdata.fetch import FetchedAsset
+
+CSV_MEDIA_TYPES = {"text/csv", "application/csv"}
+IDENTIFIER_COLUMNS = {
+    "station",
+    "station_id",
+    "site_no",
+    "monitoring_location_id",
+    "parameter_code",
+    "statistic_id",
+}
+
+
+class MissingReaderDependency(ImportError):
+    """The optional dependency required to open an asset is not installed."""
+
+
+class UnsupportedFormat(ValueError):
+    """No reader is implemented for this asset's format."""
+
+
+def open_asset(
+    fetched: FetchedAsset,
+    *,
+    reader: str | None = None,
+    dtype: dict[str, str] | None = None,
+    parse_dates: list[str] | None = None,
+    usecols: list[str] | None = None,
+    nrows: int | None = None,
+) -> Any:
+    """Read a local CSV into a pandas DataFrame, retaining units and provenance.
+
+    Infer ``csv`` or ``erddap-csv`` from media type and protocol, or use an
+    explicit reader for ambiguous metadata. Identifier columns default to pandas
+    strings; explicit dtype entries override those defaults. Dates remain strings
+    unless named in parse_dates. No checksum verification or downloading occurs.
+    """
+    if reader is None:
+        media_type = (fetched.asset.media_type or "").split(";", 1)[0].strip().lower()
+        if media_type not in CSV_MEDIA_TYPES:
+            raise UnsupportedFormat(
+                f"no reader for {fetched.asset.media_type!r}; supported formats are CSV and "
+                "ERDDAP CSV. For a known CSV with ambiguous metadata, pass reader='csv' "
+                "or reader='erddap-csv'; otherwise use fetched.path with a format-specific reader"
+            )
+        reader = "erddap-csv" if fetched.asset.protocol is Protocol.ERDDAP else "csv"
+    if reader not in {"csv", "erddap-csv"}:
+        raise UnsupportedFormat(f"unsupported reader {reader!r}; use 'csv' or 'erddap-csv'")
+    try:
+        pandas = import_module("pandas")
+    except ModuleNotFoundError as error:
+        if error.name != "pandas":
+            raise
+        raise MissingReaderDependency(
+            'CSV reading requires pandas; install it with: pip install "usdata[pandas]" '
+            '(or uv add "usdata[pandas]")'
+        ) from error
+
+    # Pass a file object to pandas: reading a fetched asset is strictly local.
+    with fetched.path.open(encoding="utf-8-sig", newline="") as stream:
+        records = csv.reader(stream)
+        columns = next(records, [])
+        if (
+            not columns
+            or any(not column for column in columns)
+            or len(set(columns)) != len(columns)
+        ):
+            raise ValueError("CSV must have a non-empty header with unique column names")
+        units = {}
+        if reader == "erddap-csv":
+            values = next(records, [])
+            if len(values) != len(columns):
+                raise ValueError("ERDDAP CSV must have a units row matching the header")
+            units = dict(zip(columns, values, strict=True))
+        types = {name: "string" for name in columns if name.casefold() in IDENTIFIER_COLUMNS}
+        types.update(dtype or {})
+        frame = pandas.read_csv(
+            stream,
+            header=None,
+            names=columns,
+            dtype=types,
+            parse_dates=parse_dates,
+            usecols=usecols,
+            nrows=nrows,
+        )
+    if units:
+        frame.attrs["units"] = {name: units[name] for name in frame.columns}
+    frame.attrs["usdata"] = {
+        "asset_id": fetched.asset.id,
+        "provenance": fetched.provenance.model_dump(mode="json"),
+    }
+    return frame