usdata 0.6.0__tar.gz → 0.7.0__tar.gz

{usdata-0.6.0 → usdata-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usdata
-Version: 0.6.0
+Version: 0.7.0
 Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
 Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
 Author: Jake Van Slyke
@@ -26,9 +26,10 @@ Description-Content-Type: text/markdown
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.6 supports GHCN-Daily, NEXRAD Level II, USGS daily
-> values, and CoastWatch SST subsets with provenance, plus Census state/county
-> lookup and optional pandas CSV readers. Other datasets are planned.
+> Status: pre-alpha. v0.7 supports GHCN-Daily, GSOM monthly summaries,
+> NEXRAD Level II, USGS daily values, and CoastWatch SST subsets with provenance,
+> plus Census state/county lookup, optional pandas CSV readers, and terminal
+> download progress. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
@@ -36,7 +37,7 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 <!-- registry:start -->
 | Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | — | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 4 | 0 | 25 | — | `ghcn-daily`, `gsom`, `nexrad-level2`, `coastwatch-sst`, +25 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
@@ -132,6 +133,16 @@ sources:
     end: 2024-05-31
 ```
 
+Terminal progress is available since v0.7. On a terminal, `fetch` and
+`pull` show progress on stderr: resolved asset counts,
+known bytes and unknown sizes, HTTP download bytes for the current attempt, and
+validated cache hits. `fetch --dry-run` also summarizes known sizes. Asset totals
+include possible cache hits; each manifest source is resolved separately. Bytes
+from a failed HTTP attempt reset on retry; encoded responses have unknown decoded
+size. Adapters that assemble files from metadata requests show asset-level progress.
+Use `--no-progress` to disable it. Progress is automatically disabled when either
+stdout or stderr is redirected; existing output lines and exit codes are unchanged.
+
 ## Opening CSV data
 
 `FetchedAsset.open()` is available since v0.6 with the optional pandas

{usdata-0.6.0 → usdata-0.7.0}/README.md

@@ -3,9 +3,10 @@
 Unified Python SDK and CLI for discovering, fetching, and tracking the
 provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 
-> Status: pre-alpha. v0.6 supports GHCN-Daily, NEXRAD Level II, USGS daily
-> values, and CoastWatch SST subsets with provenance, plus Census state/county
-> lookup and optional pandas CSV readers. Other datasets are planned.
+> Status: pre-alpha. v0.7 supports GHCN-Daily, GSOM monthly summaries,
+> NEXRAD Level II, USGS daily values, and CoastWatch SST subsets with provenance,
+> plus Census state/county lookup, optional pandas CSV readers, and terminal
+> download progress. Other datasets are planned.
 > See [docs/roadmap.md](docs/roadmap.md).
 
 ## Providers
@@ -13,7 +14,7 @@ provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).
 <!-- registry:start -->
 | Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
 |---|---:|---:|---:|---|---|
-| [NOAA](docs/providers/noaa.md) | 3 | 0 | 26 | — | `ghcn-daily`, `nexrad-level2`, `coastwatch-sst`, +26 planned |
+| [NOAA](docs/providers/noaa.md) | 4 | 0 | 25 | — | `ghcn-daily`, `gsom`, `nexrad-level2`, `coastwatch-sst`, +25 planned |
 | [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
 | [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
 | [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
@@ -109,6 +110,16 @@ sources:
     end: 2024-05-31
 ```
 
+Terminal progress is available since v0.7. On a terminal, `fetch` and
+`pull` show progress on stderr: resolved asset counts,
+known bytes and unknown sizes, HTTP download bytes for the current attempt, and
+validated cache hits. `fetch --dry-run` also summarizes known sizes. Asset totals
+include possible cache hits; each manifest source is resolved separately. Bytes
+from a failed HTTP attempt reset on retry; encoded responses have unknown decoded
+size. Adapters that assemble files from metadata requests show asset-level progress.
+Use `--no-progress` to disable it. Progress is automatically disabled when either
+stdout or stderr is redirected; existing output lines and exit codes are unchanged.
+
 ## Opening CSV data
 
 `FetchedAsset.open()` is available since v0.6 with the optional pandas

{usdata-0.6.0 → usdata-0.7.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.6.0"
+version = "0.7.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

{usdata-0.6.0 → usdata-0.7.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "usdata"
-version = "0.6.0"
+version = "0.7.0"
 description = "Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data"
 readme = "README.md"
 license = "Apache-2.0"

usdata-0.7.0/src/usdata/_progress.py

@@ -0,0 +1,62 @@
+"""Internal synchronous progress events, scoped to one CLI operation."""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Iterator, Sequence
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass
+from typing import Literal
+
+
+@dataclass(frozen=True)
+class Batch:
+    """A resolved group; sizes describe assets, including possible cache hits."""
+
+    count: int
+    known_bytes: int
+    unknown_sizes: int
+
+
+@dataclass(frozen=True)
+class AssetProgress:
+    """Start or validated completion of an asset."""
+
+    asset_id: str
+    state: Literal["start", "cached", "fetched"]
+    size: int | None
+
+
+@dataclass(frozen=True)
+class TransferProgress:
+    """Bytes written in the current HTTP attempt, reset on every retry."""
+
+    completed: int
+    total: int | None
+    attempt: int
+
+
+Event = Batch | AssetProgress | TransferProgress
+_observer: ContextVar[Callable[[Event], None] | None] = ContextVar("progress", default=None)
+
+
+def emit(event: Event) -> None:
+    """Notify the active observer without importing CLI code."""
+    observer = _observer.get()
+    if observer is not None:
+        observer(event)
+
+
+def batch(sizes: Sequence[int | None]) -> None:
+    """Report known and unknown sizes separately; never guess a total."""
+    emit(Batch(len(sizes), sum(size for size in sizes if size is not None), sizes.count(None)))
+
+
+@contextmanager
+def observe(callback: Callable[[Event], None]) -> Iterator[None]:
+    """Observe an operation and restore the previous observer on every exit."""
+    token = _observer.set(callback)
+    try:
+        yield
+    finally:
+        _observer.reset(token)

{usdata-0.6.0 → usdata-0.7.0}/src/usdata/cli/app.py

@@ -9,6 +9,8 @@ import httpx
 import typer
 
 from usdata import __version__, build_query, default_registry
+from usdata._progress import batch
+from usdata.cli.progress import progress
 from usdata.fetch import ChecksumMismatch
 from usdata.fetch import fetch as fetch_query
 from usdata.manifest import lockfile_path
@@ -126,6 +128,7 @@ def fetch(
     ] = None,
     cache_dir: Annotated[Path | None, typer.Option(help="Override the cache directory.")] = None,
     force: Annotated[bool, typer.Option(help="Re-download even if cached.")] = False,
+    no_progress: Annotated[bool, typer.Option(help="Disable terminal progress.")] = False,
     dry_run: Annotated[
         bool, typer.Option(help="List matching assets without downloading.")
     ] = False,
@@ -165,8 +168,11 @@ def fetch(
             for a in assets:
                 typer.echo(f"{a.id}\t{a.href}")
             typer.echo(f"{len(assets)} asset(s) matched", err=True)
+            with progress(disabled=no_progress):
+                batch([asset.size for asset in assets])
             return
-        fetched = fetch_query(ds, query, root=cache_dir, force=force)
+        with progress(disabled=no_progress):
+            fetched = fetch_query(ds, query, root=cache_dir, force=force)
     except (DatasetNotFound, UnknownPlace, ValueError) as e:
         typer.secho(str(e), err=True, fg="red")
         raise typer.Exit(code=2) from None
@@ -192,10 +198,12 @@ def pull(
         bool,
         typer.Option(help="Ignore an existing lockfile: re-resolve every source and rewrite it."),
     ] = False,
+    no_progress: Annotated[bool, typer.Option(help="Disable terminal progress.")] = False,
 ) -> None:
     """Fetch every source in a manifest and write (or restore from) its lockfile."""
     try:
-        result = pull_manifest(manifest, root=cache_dir, force=force)
+        with progress(disabled=no_progress):
+            result = pull_manifest(manifest, root=cache_dir, force=force)
     except EmptySource as e:
         typer.secho(str(e), err=True, fg="yellow")
         raise typer.Exit(code=1) from None

usdata-0.7.0/src/usdata/cli/progress.py

@@ -0,0 +1,89 @@
+"""Terminal-only progress rendering; normal CLI output remains machine readable."""
+
+from __future__ import annotations
+
+import shutil
+import sys
+from collections.abc import Iterator
+from contextlib import contextmanager
+from time import monotonic
+from typing import TextIO
+
+from usdata._progress import AssetProgress, Batch, Event, observe
+
+
+def _interactive() -> bool:
+    return sys.stdout.isatty() and sys.stderr.isatty()
+
+
+class _Display:
+    def __init__(self, stream: TextIO) -> None:
+        self.stream = stream
+        self.width = 0
+        self.updated = 0.0
+        self.asset = ""
+        self.count = 0
+        self.done = 0
+        self.cached = 0
+
+    def clear(self) -> None:
+        if self.width:
+            self.stream.write("\r" + " " * self.width + "\r")
+            self.stream.flush()
+            self.width = 0
+
+    def line(self, text: str, *, final: bool = False) -> None:
+        self.clear()
+        # IDs come from providers: keep control characters out of the terminal.
+        text = "".join(c if c.isprintable() else "?" for c in text)
+        if final:
+            self.stream.write(text + "\n")
+        else:
+            text = text[: max(1, shutil.get_terminal_size().columns - 1)]
+            self.stream.write(text)
+            self.width = len(text)
+        self.stream.flush()
+        self.updated = monotonic()
+
+    def __call__(self, event: Event) -> None:
+        if isinstance(event, Batch):
+            self.count, self.done, self.cached = event.count, 0, 0
+            sizes = f"{event.known_bytes:,} known bytes"
+            if event.unknown_sizes:
+                sizes += f"; {event.unknown_sizes} size(s) unknown"
+            self.line(f"{event.count} asset(s) resolved; {sizes} (before cache checks)", final=True)
+        elif isinstance(event, AssetProgress):
+            self.asset = event.asset_id
+            if event.state == "start":
+                size = f"{event.size:,} bytes" if event.size is not None else "size unknown"
+                self.line(f"[{self.done}/{self.count}] checking cache ({size}) | {self.asset}")
+            else:
+                self.done += 1
+                self.cached += event.state == "cached"
+                self.line(
+                    f"[{self.done}/{self.count}] {event.state} "
+                    f"({event.size:,} bytes; {self.cached} cached) | {self.asset}",
+                    final=self.done == self.count,
+                )
+        else:
+            if event.completed and monotonic() - self.updated < 0.1:
+                return
+            size = f"{event.total:,}" if event.total is not None else "unknown"
+            self.line(
+                f"[{self.done}/{self.count}] {event.completed:,}/{size} bytes "
+                f"(attempt {event.attempt}) | {self.asset}"
+            )
+
+
+@contextmanager
+def progress(*, disabled: bool = False) -> Iterator[None]:
+    """Render progress on stderr only when both output streams are terminals."""
+    if disabled or not _interactive():
+        yield
+        return
+    display = _Display(sys.stderr)
+    with observe(display):
+        try:
+            yield
+        finally:
+            display.clear()

{usdata-0.6.0 → usdata-0.7.0}/src/usdata/data/registry.yaml

@@ -209,19 +209,22 @@ datasets:
 
   - id: noaa:gsom
     provider: noaa
-    status: planned
+    status: available
     domain: surface-weather
-    target: later
+    since: "0.7"
     title: Global Summary of the Month
     description: >-
       Monthly station summaries derived from GHCN-Daily (means, extremes,
       totals) via the NCEI Access Data Service dataset global-summary-of-the-month.
-      Shares the GHCN-Daily client and station search.
+      Selects whole calendar months and explicit stations, or discovers stations
+      through the companion search service.
     keywords: [climate, monthly, stations, temperature, precipitation, gsom, ncei]
     protocol: http
     homepage: https://www.ncei.noaa.gov/access/search/data-search/global-summary-of-the-month
     license: US Government Work (public domain)
     capabilities: { spatial_subset: false, temporal_subset: true, variable_subset: true }
+    spatial_extent: { west: -180.0, south: -90.0, east: 180.0, north: 90.0 }
+    adapter: usdata.providers.noaa.gsom:GlobalSummaryMonthly
 
   - id: noaa:gsoy
     provider: noaa

{usdata-0.6.0 → usdata-0.7.0}/src/usdata/fetch.py

@@ -7,7 +7,7 @@ from typing import Any
 
 from pydantic import BaseModel
 
-from usdata import provenance
+from usdata import _progress, provenance
 from usdata._files import staged_path
 from usdata.cache import asset_path, sha256_file
 from usdata.models import Asset, Dataset, Provenance, Query
@@ -58,6 +58,7 @@ def _fetch_asset(
 ) -> FetchedAsset:
     if asset.dataset_id != dataset.id:
         raise ValueError(f"asset dataset {asset.dataset_id!r} does not match {dataset.id!r}")
+    _progress.emit(_progress.AssetProgress(asset.id, "start", asset.size))
     path = asset_path(asset, root)
     if not force and path.is_file():
         try:
@@ -73,6 +74,7 @@ def _fetch_asset(
             and (asset.checksum is None or prov.checksum == asset.checksum)
             and sha256_file(path) == prov.checksum
         ):
+            _progress.emit(_progress.AssetProgress(asset.id, "cached", prov.size))
             return FetchedAsset(asset=asset, path=path, provenance=prov, from_cache=True)
     with staged_path(path) as tmp:
         adapter.fetch(asset, tmp)
@@ -81,6 +83,7 @@ def _fetch_asset(
             raise ChecksumMismatch(f"{asset.id}: expected {asset.checksum}, got {prov.checksum}")
     # A crash between replacements leaves a detectable mismatch, never a trusted partial file.
     provenance.write(prov, path)
+    _progress.emit(_progress.AssetProgress(asset.id, "fetched", prov.size))
     return FetchedAsset(asset=asset, path=path, provenance=prov, from_cache=False)
 
 
@@ -98,4 +101,5 @@ def fetch(
     """Resolve and fetch a query, sharing one adapter and closing its owned resources."""
     with load_adapter(dataset) as adapter:
         assets = adapter.list_assets(query)
+        _progress.batch([asset.size for asset in assets])
         return [_fetch_asset(dataset, a, adapter, root=root, force=force) for a in assets]

{usdata-0.6.0 → usdata-0.7.0}/src/usdata/protocols/http.py

@@ -11,7 +11,7 @@ from typing import Any, TypeVar
 
 import httpx
 
-from usdata import __version__
+from usdata import __version__, _progress
 from usdata._files import staged_path
 
 USER_AGENT = f"usdata/{__version__} (+https://github.com/jakeryderv/usdata)"
@@ -80,13 +80,30 @@ def download(url: str, dest: Path, http: httpx.Client | None = None) -> Path:
     """Download atomically, restarting interrupted GETs up to three total attempts."""
     own = http is None
     active = http or client()
+    attempt = 0
 
     def request() -> Path:
+        nonlocal attempt
+        attempt += 1
+        _progress.emit(_progress.TransferProgress(0, None, attempt))
         with staged_path(dest) as tmp, active.stream("GET", url) as resp:
             resp.raise_for_status()
+            length = resp.headers.get("Content-Length", "")
+            # iter_bytes writes decoded bytes; an encoded length is not comparable.
+            total = (
+                int(length)
+                if length.isascii()
+                and length.isdigit()
+                and resp.headers.get("Content-Encoding", "identity").lower() == "identity"
+                else None
+            )
+            completed = 0
+            _progress.emit(_progress.TransferProgress(completed, total, attempt))
             with tmp.open("wb") as f:
                 for chunk in resp.iter_bytes():
                     f.write(chunk)
+                    completed += len(chunk)
+                    _progress.emit(_progress.TransferProgress(completed, total, attempt))
         return dest
 
     try:

{usdata-0.6.0 → usdata-0.7.0}/src/usdata/providers/noaa/ghcnd.py

@@ -12,6 +12,7 @@ always goes through search first. Stations are chunked so URLs stay short.
 from __future__ import annotations
 
 import hashlib
+import logging
 from pathlib import Path
 from typing import Any
 
@@ -26,6 +27,7 @@ DATA_URL = "https://www.ncei.noaa.gov/access/services/data/v1"
 NCEI_DATASET = "daily-summaries"
 SEARCH_PAGE_SIZE = 1000
 STATIONS_PER_ASSET = 50
+logger = logging.getLogger(__name__)
 
 
 def _date(value: Any) -> str:
@@ -46,6 +48,8 @@ def _stations_param(raw: Any) -> list[str]:
 class GhcnDaily(Provider):
     """GHCN-Daily adapter. Params: ``stations`` (list or comma string), ``units``."""
 
+    ncei_dataset = NCEI_DATASET
+
     def __init__(self, dataset: Dataset, client: httpx.Client | None = None) -> None:
         super().__init__(dataset)
         self._client = client
@@ -68,7 +72,7 @@ class GhcnDaily(Provider):
             raise QueryError("station search needs a bounding box and a time range")
         b = query.bbox
         params: dict[str, Any] = {
-            "dataset": NCEI_DATASET,
+            "dataset": self.ncei_dataset,
             "bbox": f"{b.north},{b.west},{b.south},{b.east}",
             "startDate": _date(query.time.start),
             "endDate": _date(query.time.end),
@@ -84,12 +88,29 @@ class GhcnDaily(Provider):
             resp.raise_for_status()
             body = resp.json()
             results = body.get("results", [])
+            before = len(found)
             for result in results:
                 for station in result.get("stations", []):
                     sid = station.get("id")
                     if sid and sid not in seen:
                         seen.add(sid)
                         found.append(sid)
+            logger.debug(
+                "NCEI station search: url=%s status=%s content_type=%s "
+                "count=%r totalCount=%r results=%s new_stations=%s station_sample=%r",
+                resp.request.url,
+                resp.status_code,
+                resp.headers.get("content-type"),
+                body.get("count"),
+                body.get("totalCount"),
+                len(results),
+                len(found) - before,
+                found[before : before + 5],
+            )
+            if len(found) == before and logger.isEnabledFor(logging.DEBUG):
+                logger.debug(
+                    "NCEI search page yielded no new stations; response_prefix=%r", resp.text[:512]
+                )
             # "count" is the number matching this query; "totalCount" is dataset-wide.
             params["offset"] += SEARCH_PAGE_SIZE
             if not results or params["offset"] >= int(body.get("count", 0)):
@@ -101,7 +122,7 @@ class GhcnDaily(Provider):
         if query.time is None or query.time.start is None or query.time.end is None:
             raise QueryError(f"{self.dataset.id} requires both start and end dates")
         if unknown := set(query.params) - {"stations", "units"}:
-            raise QueryError(f"unsupported GHCN params: {', '.join(sorted(unknown))}")
+            raise QueryError(f"unsupported {self.dataset.id} params: {', '.join(sorted(unknown))}")
         if query.params.get("units", "metric") not in ("metric", "standard"):
             raise QueryError("units must be metric or standard")
         if "stations" in query.params:
@@ -118,7 +139,7 @@ class GhcnDaily(Provider):
         for i in range(0, len(stations), STATIONS_PER_ASSET):
             chunk = stations[i : i + STATIONS_PER_ASSET]
             params: dict[str, Any] = {
-                "dataset": NCEI_DATASET,
+                "dataset": self.ncei_dataset,
                 "stations": ",".join(chunk),
                 "startDate": start,
                 "endDate": end,
@@ -132,7 +153,7 @@ class GhcnDaily(Provider):
             digest = hashlib.sha1(url.encode()).hexdigest()[:12]
             assets.append(
                 Asset(
-                    id=f"{NCEI_DATASET}_{start}_{end}_{digest}.csv",
+                    id=f"{self.ncei_dataset}_{start}_{end}_{digest}.csv",
                     dataset_id=self.dataset.id,
                     href=url,
                     protocol=Protocol.HTTP,

usdata-0.7.0/src/usdata/providers/noaa/gsom.py

@@ -0,0 +1,49 @@
+"""Monthly station summaries through the NCEI Access Data Service.
+
+Params: ``stations`` (list or comma string) and ``units`` (metric or standard).
+Both dates are required; every UTC calendar month touched by the interval is
+selected in full. Geographic discovery uses the same normalized month bounds.
+"""
+
+from calendar import monthrange
+from datetime import UTC
+
+from usdata.models import Asset, Query, TimeRange
+from usdata.providers.base import QueryError
+from usdata.providers.noaa.ghcnd import GhcnDaily
+
+
+class GlobalSummaryMonthly(GhcnDaily):
+    """GSOM CSV subsets, sharing NCEI station discovery and transport with GHCN."""
+
+    ncei_dataset = "global-summary-of-the-month"
+
+    def _monthly_query(self, query: Query) -> Query:
+        if query.time is None or query.time.start is None or query.time.end is None:
+            raise QueryError(f"{self.dataset.id} requires both start and end dates")
+        start = query.time.start.replace(tzinfo=query.time.start.tzinfo or UTC).astimezone(UTC)
+        end = query.time.end.replace(tzinfo=query.time.end.tzinfo or UTC).astimezone(UTC)
+        return query.model_copy(
+            update={
+                "time": TimeRange(
+                    start=start.replace(day=1, hour=0, minute=0, second=0, microsecond=0),
+                    end=end.replace(
+                        day=monthrange(end.year, end.month)[1],
+                        hour=23,
+                        minute=59,
+                        second=59,
+                        microsecond=999999,
+                    ),
+                )
+            }
+        )
+
+    def find_stations(self, query: Query) -> list[str]:
+        """Find stations overlapping the selected complete calendar months."""
+        return super().find_stations(self._monthly_query(query))
+
+    def list_assets(self, query: Query) -> list[Asset]:
+        """Resolve monthly CSVs with stable URLs and complete-month asset bounds."""
+        if "stations" in query.params and query.bbox is not None:
+            raise QueryError("pass stations or a location/bbox, not both")
+        return super().list_assets(self._monthly_query(query))

{usdata-0.6.0 → usdata-0.7.0}/src/usdata/pull.py

@@ -16,7 +16,7 @@ from pathlib import Path
 
 from pydantic import BaseModel
 
-from usdata import __version__, provenance
+from usdata import __version__, _progress, provenance
 from usdata.cache import asset_path, sha256_file
 from usdata.fetch import ChecksumMismatch, FetchedAsset, _fetch_asset, fetch
 from usdata.manifest import LockedAsset, Lockfile, Manifest, lockfile_path
@@ -111,13 +111,21 @@ def restore(
     lock = Lockfile.load(lock_path)
     _check_manifest(manifest_path, lock)
     fetched: list[FetchedAsset] = []
+    _progress.batch([entry.provenance.size for entry in lock.assets])
     adapters: dict[str, Provider] = {}
     with ExitStack() as stack:
         for entry in lock.assets:
             dataset = reg.get(entry.asset.dataset_id)
             path = asset_path(entry.asset, root)
+            if path.is_file():
+                _progress.emit(
+                    _progress.AssetProgress(entry.asset.id, "start", entry.provenance.size)
+                )
             if path.is_file() and sha256_file(path) == entry.provenance.checksum:
                 provenance.write(entry.provenance, path)
+                _progress.emit(
+                    _progress.AssetProgress(entry.asset.id, "cached", entry.provenance.size)
+                )
                 fetched.append(
                     FetchedAsset(
                         asset=entry.asset, path=path, provenance=entry.provenance, from_cache=True