usdata 0.6.0__tar.gz → 0.8.0__tar.gz

{usdata-0.6.0 → usdata-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.6.0
+Version: 0.8.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -14,11 +14,16 @@ Requires-Dist: httpx>=0.28.1
 Requires-Dist: pydantic>=2.7
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: typer>=0.12
+Requires-Dist: xarray>=2025.1 ; extra == 'netcdf'
+Requires-Dist: h5netcdf[h5py]>=1.8.1 ; extra == 'netcdf'
 Requires-Dist: pandas>=3.0 ; extra == 'pandas'
+Requires-Dist: xradar>=0.12.0 ; extra == 'radar'
 Requires-Python: >=3.11
 Project-URL: Homepage, https://github.com/jakeryderv/usdata
 Project-URL: Repository, https://github.com/jakeryderv/usdata
+Provides-Extra: netcdf
 Provides-Extra: pandas
+Provides-Extra: radar
 Description-Content-Type: text/markdown
 
 # usdata
@@ -26,9 +31,11 @@ Description-Content-Type: text/markdown
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.6 supports GHCN-Daily, NEXRAD Level II, USGS daily
-> values, and CoastWatch SST subsets with provenance, plus Census state/county
-> lookup and optional pandas CSV readers. Other datasets are planned.
+> Status: pre-alpha. v0.8 supports GHCN-Daily, GSOM monthly summaries,
+> NEXRAD Level II, GOES ABI CONUS imagery, Storm Events annual archives,
+> USGS daily values, and CoastWatch SST subsets with provenance. It includes
+> optional CSV, radar, and NetCDF4 readers, six executed notebooks, Census
+> state/county lookup, and terminal download progress. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
@@ -36,7 +43,7 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 <!-- registry:start -->
 | Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | — | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 6 | 0 | 23 | — | `ghcn-daily`, `gsom`, `storm-events`, `nexrad-level2`, `goes-abi`, `coastwatch-sst`, +23 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
@@ -93,6 +100,17 @@ usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
 usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
+Storm Events bulk access is available since v0.8. Dates select complete
+annual details archives; filter rows locally after opening the gzip CSV. For example:
+
+```sh
+usdata fetch noaa:storm-events --start 2024-05-01 --end 2024-05-31 --dry-run
+```
+
+This lists the entire 2024 archive. Location and variable filters are rejected;
+see the [executed Storm Events notebook](examples/storm-events/example.ipynb)
+for local filtering and reporting limitations.
+
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
@@ -132,6 +150,25 @@ sources:
     end: 2024-05-31
 ```
 
+Terminal progress is available since v0.7. On a terminal, `fetch` and
+`pull` show progress on stderr: resolved asset counts,
+known bytes and unknown sizes, HTTP download bytes for the current attempt, and
+validated cache hits. `fetch --dry-run` also summarizes known sizes. Asset totals
+include possible cache hits; each manifest source is resolved separately. Bytes
+from a failed HTTP attempt reset on retry; encoded responses have unknown decoded
+size. Adapters that assemble files from metadata requests show asset-level progress.
+Use `--no-progress` to disable it. Progress is automatically disabled when either
+stdout or stderr is redirected; existing output lines and exit codes are unchanged.
+
+For single-channel GOES CONUS imagery (available since v0.8):
+
+```sh
+usdata fetch noaa:goes-abi --start 2024-05-06T12:01:18.1Z --end 2024-05-06T12:01:18.1Z -p satellite=18 -p channel=6
+```
+
+The download is a whole NetCDF scene. See [GOES access notes](docs/providers/noaa.md#goes-abi-conus-imagery)
+for supported selectors and scan-start time semantics.
+
 ## Opening CSV data
 
 `FetchedAsset.open()` is available since v0.6 with the optional pandas
@@ -140,16 +177,32 @@ preserves identifier strings, and keeps CoastWatch units as metadata.
 See the [reader reference](docs/reference/readers.md)
 and [fetch → open → analyze example](examples/sst-analysis/README.md).
 
+The [examples directory](examples/README.md) contains executed Jupyter notebooks
+with saved data previews, small plots, and source provenance. Start with weather
+and streamflow for manifest workflows, SST for gridded CSV reading, or monthly
+climate for GSOM observations.
+
+NetCDF4 scene opening is available since v0.8 with `usdata[netcdf]`.
+See the executed [GOES infrared notebook](examples/goes-imagery/example.ipynb).
+
 ## Development
 
 Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 
+`just setup` uses the tested Python 3.14.7 pin in `.python-version`. Older Linux
+uv Python 3.14 builds can crash during NumPy array operations; see
+[the upstream fix](https://github.com/astral-sh/python-build-standalone/issues/991).
+
 ```sh
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
 just check     # format, lint, typecheck, offline tests, generated docs, release notices
 just check-pandas  # install the CSV extra and run the same checks
+just check-radar   # install the radar extra and run the same checks
+just check-netcdf  # install the NetCDF4 extra and run the same checks
+just notebooks    # launch the optional Jupyter examples environment
+just run-notebooks # execute notebooks live in fresh kernels and temporary caches
 just build     # build wheel and sdist
 just smoke     # exercise core and pandas wheel installations outside the checkout
 just run search radar
@@ -157,9 +210,11 @@ just run search radar
 
 Unit tests mechanically block network connections. Integration tests that hit
 live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
-Linux, both with and without pandas, and smoke-tests both installed-wheel
-profiles on Linux, macOS, and Windows. The full unit and live-service suites currently run on Linux. `just setup` restores
-a core-only development environment; `just check-pandas` installs the extra.
+Linux with core-only, pandas, radar, and NetCDF dependency profiles. Installed-wheel
+checks cover all four profiles on Linux, macOS, and Windows. The full unit and
+live-service suites run on Linux. `just setup` restores a core-only development
+environment; the `check-pandas`, `check-radar`, and `check-netcdf` commands install
+their respective extras.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.6.0 → usdata-0.8.0}/README.md

@@ -3,9 +3,11 @@
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.6 supports GHCN-Daily, NEXRAD Level II, USGS daily
-> values, and CoastWatch SST subsets with provenance, plus Census state/county
-> lookup and optional pandas CSV readers. Other datasets are planned.
+> Status: pre-alpha. v0.8 supports GHCN-Daily, GSOM monthly summaries,
+> NEXRAD Level II, GOES ABI CONUS imagery, Storm Events annual archives,
+> USGS daily values, and CoastWatch SST subsets with provenance. It includes
+> optional CSV, radar, and NetCDF4 readers, six executed notebooks, Census
+> state/county lookup, and terminal download progress. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
@@ -13,7 +15,7 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 <!-- registry:start -->
 | Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | — | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 6 | 0 | 23 | — | `ghcn-daily`, `gsom`, `storm-events`, `nexrad-level2`, `goes-abi`, `coastwatch-sst`, +23 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
@@ -70,6 +72,17 @@ usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
 usdata verify dataset.yaml          # exit 1 if any cached input drifted
 ```
 
+Storm Events bulk access is available since v0.8. Dates select complete
+annual details archives; filter rows locally after opening the gzip CSV. For example:
+
+```sh
+usdata fetch noaa:storm-events --start 2024-05-01 --end 2024-05-31 --dry-run
+```
+
+This lists the entire 2024 archive. Location and variable filters are rejected;
+see the [executed Storm Events notebook](examples/storm-events/example.ipynb)
+for local filtering and reporting limitations.
+
 Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
 `USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
 recording source URL, retrieval time, checksum, size, and license.
@@ -109,6 +122,25 @@ sources:
     end: 2024-05-31
 ```
 
+Terminal progress is available since v0.7. On a terminal, `fetch` and
+`pull` show progress on stderr: resolved asset counts,
+known bytes and unknown sizes, HTTP download bytes for the current attempt, and
+validated cache hits. `fetch --dry-run` also summarizes known sizes. Asset totals
+include possible cache hits; each manifest source is resolved separately. Bytes
+from a failed HTTP attempt reset on retry; encoded responses have unknown decoded
+size. Adapters that assemble files from metadata requests show asset-level progress.
+Use `--no-progress` to disable it. Progress is automatically disabled when either
+stdout or stderr is redirected; existing output lines and exit codes are unchanged.
+
+For single-channel GOES CONUS imagery (available since v0.8):
+
+```sh
+usdata fetch noaa:goes-abi --start 2024-05-06T12:01:18.1Z --end 2024-05-06T12:01:18.1Z -p satellite=18 -p channel=6
+```
+
+The download is a whole NetCDF scene. See [GOES access notes](docs/providers/noaa.md#goes-abi-conus-imagery)
+for supported selectors and scan-start time semantics.
+
 ## Opening CSV data
 
 `FetchedAsset.open()` is available since v0.6 with the optional pandas
@@ -117,16 +149,32 @@ preserves identifier strings, and keeps CoastWatch units as metadata.
 See the [reader reference](docs/reference/readers.md)
 and [fetch → open → analyze example](examples/sst-analysis/README.md).
 
+The [examples directory](examples/README.md) contains executed Jupyter notebooks
+with saved data previews, small plots, and source provenance. Start with weather
+and streamflow for manifest workflows, SST for gridded CSV reading, or monthly
+climate for GSOM observations.
+
+NetCDF4 scene opening is available since v0.8 with `usdata[netcdf]`.
+See the executed [GOES infrared notebook](examples/goes-imagery/example.ipynb).
+
 ## Development
 
 Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).
 
+`just setup` uses the tested Python 3.14.7 pin in `.python-version`. Older Linux
+uv Python 3.14 builds can crash during NumPy array operations; see
+[the upstream fix](https://github.com/astral-sh/python-build-standalone/issues/991).
+
 ```sh
 git clone https://github.com/jakeryderv/usdata && cd usdata
 just setup     # install toolchain and dependencies
 just test      # unit tests
 just check     # format, lint, typecheck, offline tests, generated docs, release notices
 just check-pandas  # install the CSV extra and run the same checks
+just check-radar   # install the radar extra and run the same checks
+just check-netcdf  # install the NetCDF4 extra and run the same checks
+just notebooks    # launch the optional Jupyter examples environment
+just run-notebooks # execute notebooks live in fresh kernels and temporary caches
 just build     # build wheel and sdist
 just smoke     # exercise core and pandas wheel installations outside the checkout
 just run search radar
@@ -134,9 +182,11 @@ just run search radar
 
 Unit tests mechanically block network connections. Integration tests that hit
 live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
-Linux, both with and without pandas, and smoke-tests both installed-wheel
-profiles on Linux, macOS, and Windows. The full unit and live-service suites currently run on Linux. `just setup` restores
-a core-only development environment; `just check-pandas` installs the extra.
+Linux with core-only, pandas, radar, and NetCDF dependency profiles. Installed-wheel
+checks cover all four profiles on Linux, macOS, and Windows. The full unit and
+live-service suites run on Linux. `just setup` restores a core-only development
+environment; the `check-pandas`, `check-radar`, and `check-netcdf` commands install
+their respective extras.
 
 Releases: `just release minor` opens a version-bump PR; merging it publishes
 to PyPI and creates the tag and GitHub release. See

{usdata-0.6.0 → usdata-0.8.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.6.0"
+version = "0.8.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -32,6 +32,11 @@ email = "jakervanslyke@gmail.com"
 
 [project.optional-dependencies]
 pandas = ["pandas>=3.0"]
+radar = ["xradar>=0.12.0"]
+netcdf = [
+    "xarray>=2025.1",
+    "h5netcdf[h5py]>=1.8.1",
+]
 
 [project.urls]
 Homepage = "https://github.com/jakeryderv/usdata"
@@ -48,6 +53,13 @@ dev = [
     "respx>=0.23.1",
     "ruff>=0.6",
 ]
+examples = [
+    "ipykernel>=6.29",
+    "jupyterlab>=4.3",
+    "matplotlib>=3.9",
+    "nbclient>=0.10",
+    "pandas>=3.0",
+]
 
 [build-system]
 requires = ["uv_build>=0.12.5,<0.13.0"]

{usdata-0.6.0 → usdata-0.8.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.6.0"
+version = "0.8.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"
@@ -24,6 +24,8 @@ dependencies = [
 
 [project.optional-dependencies]
 pandas = ["pandas>=3.0"]
+radar = ["xradar>=0.12.0"]
+netcdf = ["xarray>=2025.1", "h5netcdf[h5py]>=1.8.1"]
 
 [project.urls]
 Homepage = "https://github.com/jakeryderv/usdata"
@@ -40,6 +42,13 @@ dev = [
     "respx>=0.23.1",
     "ruff>=0.6",
 ]
+examples = [
+    "ipykernel>=6.29",
+    "jupyterlab>=4.3",
+    "matplotlib>=3.9",
+    "nbclient>=0.10",
+    "pandas>=3.0",
+]
 
 [build-system]
 requires = ["uv_build>=0.12.5,<0.13.0"]

usdata-0.8.0/src/usdata/_netcdf.py

@@ -0,0 +1,36 @@
+"""Local, eagerly loaded NetCDF4 reading behind the netcdf extra."""
+
+from __future__ import annotations
+
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+from usdata.readers import MissingReaderDependency
+
+if TYPE_CHECKING:
+    from usdata.fetch import FetchedAsset
+
+
+def open_netcdf(fetched: FetchedAsset) -> Any:
+    """Load a NetCDF4 root Dataset, then close every source file handle."""
+    try:
+        xarray = import_module("xarray")
+        import_module("h5netcdf")
+        import_module("h5py")
+    except ModuleNotFoundError as error:
+        if error.name not in {"xarray", "h5netcdf", "h5py"}:
+            raise
+        raise MissingReaderDependency(
+            'NetCDF4 reading requires xarray and h5netcdf; install: pip install "usdata[netcdf]"'
+        ) from error
+    # A local file object and fixed engine prevent interpretation as an OPeNDAP URL.
+    with (
+        fetched.path.open("rb") as stream,
+        xarray.open_dataset(stream, engine="h5netcdf", chunks=None) as dataset,
+    ):
+        dataset.load()
+    dataset.attrs["usdata"] = {
+        "asset_id": fetched.asset.id,
+        "provenance": fetched.provenance.model_dump(mode="json"),
+    }
+    return dataset

usdata-0.8.0/src/usdata/_progress.py

@@ -0,0 +1,62 @@
+"""Internal synchronous progress events, scoped to one CLI operation."""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Iterator, Sequence
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass
+from typing import Literal
+
+
+@dataclass(frozen=True)
+class Batch:
+    """A resolved group; sizes describe assets, including possible cache hits."""
+
+    count: int
+    known_bytes: int
+    unknown_sizes: int
+
+
+@dataclass(frozen=True)
+class AssetProgress:
+    """Start or validated completion of an asset."""
+
+    asset_id: str
+    state: Literal["start", "cached", "fetched"]
+    size: int | None
+
+
+@dataclass(frozen=True)
+class TransferProgress:
+    """Bytes written in the current HTTP attempt, reset on every retry."""
+
+    completed: int
+    total: int | None
+    attempt: int
+
+
+Event = Batch | AssetProgress | TransferProgress
+_observer: ContextVar[Callable[[Event], None] | None] = ContextVar("progress", default=None)
+
+
+def emit(event: Event) -> None:
+    """Notify the active observer without importing CLI code."""
+    observer = _observer.get()
+    if observer is not None:
+        observer(event)
+
+
+def batch(sizes: Sequence[int | None]) -> None:
+    """Report known and unknown sizes separately; never guess a total."""
+    emit(Batch(len(sizes), sum(size for size in sizes if size is not None), sizes.count(None)))
+
+
+@contextmanager
+def observe(callback: Callable[[Event], None]) -> Iterator[None]:
+    """Observe an operation and restore the previous observer on every exit."""
+    token = _observer.set(callback)
+    try:
+        yield
+    finally:
+        _observer.reset(token)

usdata-0.8.0/src/usdata/_radar.py

@@ -0,0 +1,73 @@
+"""Local NEXRAD Level II decoding behind the radar extra."""
+
+from __future__ import annotations
+
+import bz2
+import gzip
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+from usdata.readers import MissingReaderDependency
+
+# NOAA RDA/RPG ICD 2620002Y, Table XVII-I notes 21 and 30.
+MOMENT_FLAG_COUNTS = {
+    "DBZH": 2,
+    "VRADH": 2,
+    "WRADH": 2,
+    "ZDR": 2,
+    "PHIDP": 2,
+    "RHOHV": 2,
+    "CCORH": 8,
+}
+
+if TYPE_CHECKING:
+    from usdata.fetch import FetchedAsset
+
+
+def open_nexrad(fetched: FetchedAsset) -> Any:
+    """Decode a local volume into a fully loaded xarray DataTree."""
+    try:
+        xradar = import_module("xradar")
+    except ModuleNotFoundError as error:
+        if error.name != "xradar":
+            raise
+        raise MissingReaderDependency(
+            'NEXRAD reading requires xradar; install it with: pip install "usdata[radar]" '
+            '(or uv add "usdata[radar]")'
+        ) from error
+
+    # Bytes prevent remote URL interpretation and work across the backend's
+    # repeated sweep reads. Compressed source files stay unchanged in the cache.
+    content = fetched.path.read_bytes()
+    if content.startswith(b"\x1f\x8b"):
+        content = gzip.decompress(content)
+    elif content.startswith(b"BZh"):
+        content = bz2.decompress(content)
+    radar = xradar.io.open_nexradlevel2_datatree(content, incomplete_sweep="pad")
+    try:
+        radar.load()
+    finally:
+        radar.close()
+    for node in radar.subtree:
+        for name, variable in node.ds.variables.items():
+            # The backend records the entire input byte string as `source`.
+            # Provenance below is the durable reference, not that decoder buffer.
+            variable.encoding.pop("source", None)
+            if name in MOMENT_FLAG_COUNTS and "range" in variable.dims:
+                scale = variable.encoding.get("scale_factor")
+                offset = variable.encoding.get("add_offset")
+                if scale is not None and offset is not None:
+                    # xradar 0.12 does not supply _FillValue for NEXRAD flags.
+                    # Compare using each moment's native scale, not fixed units.
+                    data = node[name]
+                    valid = data.notnull()
+                    for code in range(MOMENT_FLAG_COUNTS[name]):
+                        valid = valid & (data != offset + code * scale)
+                    masked = data.where(valid)
+                    masked.encoding = data.encoding.copy()
+                    node[name] = masked
+    radar.attrs["usdata"] = {
+        "asset_id": fetched.asset.id,
+        "provenance": fetched.provenance.model_dump(mode="json"),
+    }
+    return radar

{usdata-0.6.0 → usdata-0.8.0}/src/usdata/cli/app.py

@@ -9,6 +9,8 @@ import httpx
 import typer
 
 from usdata import __version__, build_query, default_registry
+from usdata._progress import batch
+from usdata.cli.progress import progress
 from usdata.fetch import ChecksumMismatch
 from usdata.fetch import fetch as fetch_query
 from usdata.manifest import lockfile_path
@@ -126,6 +128,7 @@ def fetch(
     ] = None,
     cache_dir: Annotated[Path | None, typer.Option(help="Override the cache directory.")] = None,
     force: Annotated[bool, typer.Option(help="Re-download even if cached.")] = False,
+    no_progress: Annotated[bool, typer.Option(help="Disable terminal progress.")] = False,
     dry_run: Annotated[
         bool, typer.Option(help="List matching assets without downloading.")
     ] = False,
@@ -165,8 +168,11 @@ def fetch(
             for a in assets:
                 typer.echo(f"{a.id}\t{a.href}")
             typer.echo(f"{len(assets)} asset(s) matched", err=True)
+            with progress(disabled=no_progress):
+                batch([asset.size for asset in assets])
             return
-        fetched = fetch_query(ds, query, root=cache_dir, force=force)
+        with progress(disabled=no_progress):
+            fetched = fetch_query(ds, query, root=cache_dir, force=force)
     except (DatasetNotFound, UnknownPlace, ValueError) as e:
         typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
@@ -192,10 +198,12 @@ def pull(
         bool,
         typer.Option(help="Ignore an existing lockfile: re-resolve every source and rewrite it."),
     ] = False,
+    no_progress: Annotated[bool, typer.Option(help="Disable terminal progress.")] = False,
 ) -> None:
     """Fetch every source in a manifest and write (or restore from) its lockfile."""
     try:
-        result = pull_manifest(manifest, root=cache_dir, force=force)
+        with progress(disabled=no_progress):
+            result = pull_manifest(manifest, root=cache_dir, force=force)
     except EmptySource as e:
         typer.secho(str(e), err=True, fg="yellow")
         raise typer.Exit(code=1) from None

usdata-0.8.0/src/usdata/cli/progress.py

@@ -0,0 +1,89 @@
+"""Terminal-only progress rendering; normal CLI output remains machine readable."""
+
+from __future__ import annotations
+
+import shutil
+import sys
+from collections.abc import Iterator
+from contextlib import contextmanager
+from time import monotonic
+from typing import TextIO
+
+from usdata._progress import AssetProgress, Batch, Event, observe
+
+
+def _interactive() -> bool:
+    return sys.stdout.isatty() and sys.stderr.isatty()
+
+
+class _Display:
+    def __init__(self, stream: TextIO) -> None:
+        self.stream = stream
+        self.width = 0
+        self.updated = 0.0
+        self.asset = ""
+        self.count = 0
+        self.done = 0
+        self.cached = 0
+
+    def clear(self) -> None:
+        if self.width:
+            self.stream.write("\r" + " " * self.width + "\r")
+            self.stream.flush()
+            self.width = 0
+
+    def line(self, text: str, *, final: bool = False) -> None:
+        self.clear()
+        # IDs come from providers: keep control characters out of the terminal.
+        text = "".join(c if c.isprintable() else "?" for c in text)
+        if final:
+            self.stream.write(text + "\n")
+        else:
+            text = text[: max(1, shutil.get_terminal_size().columns - 1)]
+            self.stream.write(text)
+            self.width = len(text)
+        self.stream.flush()
+        self.updated = monotonic()
+
+    def __call__(self, event: Event) -> None:
+        if isinstance(event, Batch):
+            self.count, self.done, self.cached = event.count, 0, 0
+            sizes = f"{event.known_bytes:,} known bytes"
+            if event.unknown_sizes:
+                sizes += f"; {event.unknown_sizes} size(s) unknown"
+            self.line(f"{event.count} asset(s) resolved; {sizes} (before cache checks)", final=True)
+        elif isinstance(event, AssetProgress):
+            self.asset = event.asset_id
+            if event.state == "start":
+                size = f"{event.size:,} bytes" if event.size is not None else "size unknown"
+                self.line(f"[{self.done}/{self.count}] checking cache ({size}) | {self.asset}")
+            else:
+                self.done += 1
+                self.cached += event.state == "cached"
+                self.line(
+                    f"[{self.done}/{self.count}] {event.state} "
+                    f"({event.size:,} bytes; {self.cached} cached) | {self.asset}",
+                    final=self.done == self.count,
+                )
+        else:
+            if event.completed and monotonic() - self.updated < 0.1:
+                return
+            size = f"{event.total:,}" if event.total is not None else "unknown"
+            self.line(
+                f"[{self.done}/{self.count}] {event.completed:,}/{size} bytes "
+                f"(attempt {event.attempt}) | {self.asset}"
+            )
+
+
+@contextmanager
+def progress(*, disabled: bool = False) -> Iterator[None]:
+    """Render progress on stderr only when both output streams are terminals."""
+    if disabled or not _interactive():
+        yield
+        return
+    display = _Display(sys.stderr)
+    with observe(display):
+        try:
+            yield
+        finally:
+            display.clear()