mostlyright-data 0.30.2__tar.gz → 0.30.4__tar.gz

{mostlyright_data-0.30.2 → mostlyright_data-0.30.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.30.2
+Version: 0.30.4
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.30.2 → mostlyright_data-0.30.4}/skills/mr-data-build/SKILL.md

@@ -44,6 +44,7 @@ session finds it. Then choose the source shape that fits.
 | One request per row of another source | a per-row `request`, with `batch` when a `GET` takes a list of ids | the gap | [per-row](references/patterns.md#one-request-per-row-many-ids-per-request) |
 | Bulk dump beside a windowed source | `reference` | `recheck_seconds` | [reference](references/patterns.md#bulk-dump-beside-a-windowed-source) |
 | Live `wss://` feed | a recorded stream | see [Streams](references/recording-a-stream-venue.md) | none |
+| A join of tables this workspace already built, rebuilt when one promotes | `table.version@1.0.0` sources with `refresh: upstream_promoted` and `rebuild` (Studio 4.38) | one shared `data_classification`, `debounce_seconds` | [table sources](references/sources.md#reading-another-table-of-this-workspace) |
 
 [Patterns](references/patterns.md) has one complete recipe per row, with the pace set.
 [Sources](references/sources.md#pace-and-width) has the pace rules and the limits of parallel
@@ -152,6 +153,9 @@ step 6. The stage 3 reference holds its rule. Gate 4 is step 6.
    quality checks.
 7. Follow the returned run with `mr-data watch RUN_ID --json`. Its `source_rates` and
    `mr-data status RUN_ID --json` on a live run give each collection source's rate and time left.
+   On a queued or running run, `status` also gives `progress`: the current attempt, how many of
+   its sources it fetched of how many, the source it is on with its pages, and how the attempt
+   before it ended. A handed-back run's new attempt counts from zero.
    Quote those numbers rather than doing the arithmetic. Read `run.status`: returning an existing
    run does not mean new work was queued. Resume the known run after an interruption, using the
    saved cursor where available. Read [run lifecycle](references/live-run.md) for holds and

{mostlyright_data-0.30.2 → mostlyright_data-0.30.4}/skills/mr-data-build/references/sources.md

@@ -383,3 +383,30 @@ with `epoch_millis_value` on the raw millisecond time column, and size the reque
 for all slices and pages. Do not substitute a whole-day query:
 1,440 one-minute bars exceed Binance's 1,000-row response cap. Existing credential-free
 `public.https` file fetches and `public.https.collection@2.0.0` keep their own contracts.
+
+### Reading another table of this workspace
+
+Needs Studio 4.38. When the user wants a table that joins or reshapes tables this workspace has
+already built, do not fetch the upstream data again. Read each upstream as a source:
+
+- `source_class: mostlyright_table`, and a `table` locator with `{table_id, version:
+  latest_promoted}`.
+- Adapter `table.version@1.0.0`, with `credential_mode: none` and the Reader pin
+  `parquet.tabular` `1.0.0`, decode options `{}`.
+- `limits.max_source_bytes` above the upstream's real bytes. Take them from `mr-data table show`;
+  Studio refuses a run whose pinned upstream version is larger.
+
+To rebuild the join whenever an upstream promotes, declare `rebuild` and add `refresh: {trigger:
+upstream_promoted, debounce_seconds: N}`. Pick N so upstreams that promote together start one
+rebuild; 900 is a sound default.
+
+Three rules cost a round trip when they are missed:
+
+- Every source states the same `data_classification`. Use the strictest upstream's class.
+- The upstreams must be in this workspace, promoted, not archived, and not this table. No edge may
+  close a cycle.
+- The trigger runs only after the user chose updates with `mr-data promote --cadence`. That
+  cadence stays the fallback schedule.
+
+The full member list is in `docs/RECIPE-DOCUMENT.md`, "Needs Studio 4.38".
+

{mostlyright_data-0.30.2 → mostlyright_data-0.30.4}/src/mostlyright/data_harness/page_coverage.py

@@ -20,6 +20,7 @@ __all__ = [
     "PAGE_BUDGET_KINDS",
     "PAGE_COVERAGE_COUNT_FIELDS",
     "PAGE_COVERAGE_FIELDS",
+    "SOURCE_ROWS_INCOMPLETE",
     "page_coverage_sentence",
 ]
 
@@ -42,6 +43,9 @@ __all__ = [
 #: predecessor rows, but current collection refresh is refused before acquisition, except a
 #: per-row request collection's, which refreshes with the direct request window it inherits and
 #: asks again for a failed row whenever a refresh covers that row's partition.
+#: A ``source_rows`` collection also states it on a full run, as ``SOURCE_ROWS_INCOMPLETE``, when
+#: the upstream relation it lists from is incomplete: that is not a listing anybody failed to read
+#: but a list that may be short by what the upstream source is missing, and the sentence says so.
 PAGE_COVERAGE_FIELDS: tuple[str, ...] = (
     "discovered",
     "discovery_requests",
@@ -84,6 +88,46 @@ PAGE_COVERAGE_COUNT_FIELDS: tuple[str, ...] = (
 )
 
 
+#: The discovery failure a ``source_rows`` collection states when the upstream relation it lists
+#: from is itself incomplete (failed, pending or unlisted members upstream). Nothing failed to be
+#: LISTED: the list is built from rows another source did not finish delivering, so it may be
+#: short by exactly what that source is missing. Said as what it is, never as an unreadable
+#: listing, because a reader told "the listing could not be read" goes looking for an outage.
+SOURCE_ROWS_INCOMPLETE = "SOURCE_ROWS_INCOMPLETE"
+
+
+def _count(block: Mapping[str, Any], member: str) -> int | None:
+    value = block.get(member)
+    if isinstance(value, bool) or not isinstance(value, int) or value < 0:
+        return None
+    return value
+
+
+def _pages_held(block: Mapping[str, Any]) -> int | None:
+    """How many of the collection's members hold an answer, or ``None`` when the block cannot say.
+
+    MEMBERS, NOT CALLS. ``fetched_this_run`` counts the requests this run made: a per-row request
+    collection asks once for every member that renders the same request, a batch answers up to a
+    hundred members in one call, and a page kept from an earlier capture is not fetched again. So
+    ``fetched_this_run`` over ``discovered`` divides calls by members, and a run that answered
+    every member it listed reads as half done (food run 23161446: "459,156 of 1,005,223" for a
+    run with 1,004,768 members answered and 455 failed). The members that hold an answer are the
+    ones the ledger knows that are neither still owed, nor failed, nor skipped -- the same count
+    the dataset page's sources card prints as fetched.
+
+    Read with ``get``: a record written before these members existed carries only the four the
+    sentence always needed, and says what it always said.
+    """
+
+    known = _count(block, "known")
+    pending = _count(block, "pending")
+    if known is None or pending is None:
+        return None
+    failed = _count(block, "failed") or 0
+    skipped = _count(block, "skipped") or 0
+    return max(known - pending - failed - skipped, 0)
+
+
 def page_coverage_sentence(block: Mapping[str, Any]) -> str:
     """The one sentence a person reads off a page collection's coverage.
 
@@ -96,13 +140,29 @@ def page_coverage_sentence(block: Mapping[str, Any]) -> str:
     failed = int(block["failed"])
     if block["complete"]:
         return f"all {discovered:,} pages fetched"
-    parts = [f"{fetched:,} of {discovered:,} pages fetched"]
+    held = _pages_held(block)
+    if held is None:
+        parts = [f"{fetched:,} of {discovered:,} pages fetched"]
+    else:
+        # A ledger can know pages its latest listing no longer names (a refresh keeps them), so
+        # the whole is the larger of the two and the sentence never says "more of fewer".
+        whole = max(discovered, _count(block, "known") or 0)
+        lead = f"{held:,} of {whole:,} pages fetched"
+        if fetched != held:
+            lead += f" ({fetched:,} requests this run)"
+        parts = [lead]
     # READ WITH ``get``, AND ONLY THIS MEMBER. The block a run produces always carries it, and the
     # block Studio hands back may not: it is optional on the run record, so a record written before
     # the member existed is still one sentence rather than a KeyError this lane swallows whole.
     listing = block.get("discovery_failure")
     if isinstance(listing, Mapping):
-        parts.append(f"the page listing could not be read ({listing['code']})")
+        if listing.get("code") == SOURCE_ROWS_INCOMPLETE:
+            parts.append(
+                "a page list built from another source's rows may be short, because that source "
+                f"is incomplete ({SOURCE_ROWS_INCOMPLETE})"
+            )
+        else:
+            parts.append(f"the page listing could not be read ({listing['code']})")
     if failed:
         parts.append(f"{failed:,} failed")
     # The block does not say which kind of collection it counts, so the remedy names both. A

{mostlyright_data-0.30.2 → mostlyright_data-0.30.4}/src/mostlyright/data_harness/thin/recipe_lint.py

@@ -231,6 +231,8 @@ def lint_document(document: Mapping[str, Any]) -> list[Finding]:
     findings.extend(_columns(document.get("table")))
     findings.extend(_identifiers(document))
     findings.extend(_sources(document.get("sources")))
+    findings.extend(_table_sources(document.get("sources")))
+    findings.extend(_refresh_trigger(document))
     findings.extend(_row_filters(document.get("sources")))
     findings.extend(_row_request_credentials(document.get("sources")))
     findings.extend(_row_request_templates(document.get("sources")))
@@ -551,6 +553,272 @@ def _sources(sources: Any) -> list[Finding]:
     return findings
 
 
+#: ``recipe.schema.json``'s Mostly Right table source (Studio 4.38.0, ADR 0100): the one adapter
+#: that reads another table of this workspace, its source class and its locator kind.
+TABLE_SOURCE_ADAPTER_ID = "table.version@1.0.0"
+TABLE_SOURCE_CLASS = "mostlyright_table"
+TABLE_LOCATOR_KIND = "table"
+
+#: ``#/$defs/table_source_locator/properties/version``: the one selector v1 admits.
+TABLE_SOURCE_VERSIONS: frozenset[str] = frozenset({"latest_promoted"})
+
+#: Members a table source never carries: it is read whole, presents no credential and names no
+#: origin, so the schema refuses each of these beside it.
+TABLE_SOURCE_FORBIDDEN: tuple[str, ...] = (
+    "credential",
+    "window",
+    "collection",
+    "closed",
+    "reference",
+    "document_archive",
+)
+TABLE_CONNECTOR_FORBIDDEN: tuple[str, ...] = (
+    "origin",
+    "generic_api",
+    "stream_connector_document_digest",
+)
+
+#: ``#/$defs/table_refresh_trigger``.
+REFRESH_TRIGGERS: frozenset[str] = frozenset({"upstream_promoted"})
+MAX_REFRESH_DEBOUNCE_SECONDS = 86_400
+
+_UUID = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
+
+
+def is_table_source(source: Any) -> bool:
+    """Whether a source is a Mostly Right table source by any of the three marks the schema reads.
+
+    The schema applies the table-source rules when the adapter, the locator kind OR the source
+    class says so, so a source that states only one of them is held to all of them here too.
+    """
+
+    if not isinstance(source, Mapping):
+        return False
+    connector = source.get("connector")
+    locator = source.get("locator")
+    return (
+        (isinstance(connector, Mapping) and connector.get("adapter_id") == TABLE_SOURCE_ADAPTER_ID)
+        or (isinstance(locator, Mapping) and locator.get("kind") == TABLE_LOCATOR_KIND)
+        or source.get("source_class") == TABLE_SOURCE_CLASS
+    )
+
+
+def _table_sources(sources: Any) -> list[Finding]:
+    """A Mostly Right table source's shape, which the schema refuses at the wire (Studio 4.38.0).
+
+    It reads the latest promoted version of another table in this workspace: source class
+    ``mostlyright_table``, a ``table`` locator naming the upstream ``table_id`` and
+    ``version: latest_promoted``, adapter ``table.version@1.0.0`` with ``credential_mode: none``
+    and the Reader pin that decodes the upstream's Parquet parts. Which table it names, whether that
+    table is in this workspace, promoted, archived, the table itself or closes a cycle, is Studio's
+    to judge at registration; nothing here can see the workspace.
+    """
+
+    findings: list[Finding] = []
+    for index, source in enumerate(sources if isinstance(sources, list) else ()):
+        if not is_table_source(source):
+            continue
+        pointer = f"/sources/{index}"
+        if source.get("source_class") != TABLE_SOURCE_CLASS:
+            findings.append(
+                Finding(
+                    DOCUMENT_INVALID,
+                    f"{pointer}/source_class",
+                    f"must be {TABLE_SOURCE_CLASS!r} for a source that reads another table",
+                )
+            )
+        locator = source.get("locator")
+        locator = locator if isinstance(locator, Mapping) else {}
+        if locator.get("kind") != TABLE_LOCATOR_KIND:
+            findings.append(
+                Finding(
+                    DOCUMENT_INVALID,
+                    f"{pointer}/locator/kind",
+                    f"must be {TABLE_LOCATOR_KIND!r} for a source that reads another table",
+                )
+            )
+        table = locator.get("table")
+        if not isinstance(table, Mapping):
+            findings.append(
+                Finding(
+                    DOCUMENT_INVALID,
+                    f"{pointer}/locator/table",
+                    "is absent; it names the upstream table as {table_id, version: "
+                    "latest_promoted}",
+                )
+            )
+        else:
+            table_id = table.get("table_id")
+            if not isinstance(table_id, str) or _UUID.match(table_id) is None:
+                findings.append(
+                    Finding(
+                        DOCUMENT_INVALID,
+                        f"{pointer}/locator/table/table_id",
+                        "is not a lowercase UUID naming a table of this workspace",
+                    )
+                )
+            if _outside(table.get("version"), TABLE_SOURCE_VERSIONS):
+                findings.append(
+                    Finding(
+                        DOCUMENT_INVALID,
+                        f"{pointer}/locator/table/version",
+                        "must be 'latest_promoted', the one version selector Studio admits",
+                    )
+                )
+            for member in sorted(set(table) - {"table_id", "version"}):
+                findings.append(
+                    Finding(
+                        DOCUMENT_INVALID,
+                        f"{pointer}/locator/table/{_escape(str(member))}",
+                        "is not a member of a table locator",
+                    )
+                )
+        connector = source.get("connector")
+        connector = connector if isinstance(connector, Mapping) else {}
+        if connector.get("adapter_id") != TABLE_SOURCE_ADAPTER_ID:
+            findings.append(
+                Finding(
+                    DOCUMENT_INVALID,
+                    f"{pointer}/connector/adapter_id",
+                    f"must be {TABLE_SOURCE_ADAPTER_ID!r} for a source that reads another table",
+                )
+            )
+        if connector.get("credential_mode") != "none":
+            findings.append(
+                Finding(
+                    DOCUMENT_INVALID,
+                    f"{pointer}/connector/credential_mode",
+                    "must be 'none': a table source reads this workspace's own sealed parts",
+                )
+            )
+        parameters = connector.get("parameters")
+        if not isinstance(parameters, list) or len(parameters) < 2:
+            findings.append(
+                Finding(
+                    DOCUMENT_INVALID,
+                    f"{pointer}/connector/parameters",
+                    "must pin the Reader that decodes the upstream's Parquet parts "
+                    "(reader.family_id parquet.tabular, reader.family_version, "
+                    "reader.decode_options)",
+                )
+            )
+        for member in TABLE_CONNECTOR_FORBIDDEN:
+            if member in connector:
+                findings.append(
+                    Finding(
+                        DOCUMENT_INVALID,
+                        f"{pointer}/connector/{member}",
+                        "is not stated on a table source, which reaches no origin",
+                    )
+                )
+        for member in TABLE_SOURCE_FORBIDDEN:
+            if member in source:
+                findings.append(
+                    Finding(
+                        DOCUMENT_INVALID,
+                        f"{pointer}/{member}",
+                        "is not stated on a table source, which is read whole and presents no "
+                        "credential",
+                    )
+                )
+    return findings
+
+
+def _refresh_trigger(document: Mapping[str, Any]) -> list[Finding]:
+    """``refresh: {trigger: upstream_promoted, debounce_seconds}`` and what it requires.
+
+    The trigger asks Studio to rebuild this table when an upstream table it reads promotes a new
+    version, so the schema requires ``rebuild`` beside it (the rebuild is what answers the request,
+    under its own interval and byte bounds) and at least one ``table.version@1.0.0`` source.
+    """
+
+    if "refresh" not in document:
+        return []
+    refresh = document.get("refresh")
+    if not isinstance(refresh, Mapping):
+        return [Finding(DOCUMENT_INVALID, "/refresh", "is not one JSON object")]
+    findings: list[Finding] = []
+    if _outside(refresh.get("trigger"), REFRESH_TRIGGERS):
+        findings.append(
+            Finding(DOCUMENT_INVALID, "/refresh/trigger", "must be 'upstream_promoted'")
+        )
+    debounce = refresh.get("debounce_seconds")
+    if (
+        isinstance(debounce, bool)
+        or not isinstance(debounce, int)
+        or not 0 <= debounce <= MAX_REFRESH_DEBOUNCE_SECONDS
+    ):
+        findings.append(
+            Finding(
+                DOCUMENT_INVALID,
+                "/refresh/debounce_seconds",
+                f"must be a whole number of seconds from 0 to {MAX_REFRESH_DEBOUNCE_SECONDS}",
+            )
+        )
+    for member in sorted(set(refresh) - {"trigger", "debounce_seconds"}):
+        findings.append(
+            Finding(
+                DOCUMENT_INVALID,
+                f"/refresh/{_escape(str(member))}",
+                "is not a member of refresh",
+            )
+        )
+    if "rebuild" not in document:
+        findings.append(
+            Finding(
+                DOCUMENT_INVALID,
+                "/rebuild",
+                "is absent, and refresh requires it: the declared whole-table rebuild is what "
+                "answers an upstream promotion",
+            )
+        )
+    sources = document.get("sources")
+    if not any(
+        isinstance(source, Mapping)
+        and isinstance(source.get("connector"), Mapping)
+        and source["connector"].get("adapter_id") == TABLE_SOURCE_ADAPTER_ID
+        for source in (sources if isinstance(sources, list) else ())
+    ):
+        findings.append(
+            Finding(
+                DOCUMENT_INVALID,
+                "/refresh",
+                f"needs at least one {TABLE_SOURCE_ADAPTER_ID} source: only an upstream table "
+                "this recipe reads can promote",
+            )
+        )
+    return findings
+
+
+def _classification_coherence(sources: Any) -> list[Finding]:
+    """Every source of one run declares one ``data_classification``.
+
+    The worker's governance refuses a run whose sources declare two classifications
+    (``RUN_RECIPE_INVALID`` at acquire), after the document registered and a run was queued, so
+    it is named here. It matters most for a join of Mostly Right tables, whose sources inherit
+    each upstream's classification and must still agree.
+    """
+
+    stated = [
+        (index, source.get("data_classification"))
+        for index, source in enumerate(sources if isinstance(sources, list) else ())
+        if isinstance(source, Mapping) and isinstance(source.get("data_classification"), str)
+    ]
+    if len({value for _index, value in stated}) <= 1:
+        return []
+    first = stated[0][1]
+    return [
+        Finding(
+            DOCUMENT_INCOHERENT,
+            f"/sources/{index}/data_classification",
+            f"is {value!r} while /sources/{stated[0][0]} is {first!r}; the worker refuses a run "
+            "whose sources declare more than one classification, at acquire",
+        )
+        for index, value in stated
+        if value != first
+    ]
+
+
 def _row_filters(sources: Any) -> list[Finding]:
     """``collection.discovery.row_filter``: where one may stand, and what it must be.
 
@@ -905,6 +1173,7 @@ def _coherence(document: Mapping[str, Any], *, declared: Sequence[str]) -> list[
     findings.extend(_table_partition_coherence(document, known=known))
     findings.extend(_reference_coherence(document.get("sources")))
     findings.extend(_rebuild_coherence(document))
+    findings.extend(_classification_coherence(document.get("sources")))
     return findings
 
 

{mostlyright_data-0.30.2 → mostlyright_data-0.30.4}/src/mostlyright/data_harness/thin/table_search_contract.json

@@ -1,6 +1,6 @@
 {
  "schema_version": "mostlyright-thin-table-search-contract.v1",
- "source_commit": "b308bc2e9ebb424b4cfe1130438ab9f7d0b6736d",
+ "source_commit": "f61f08e29fc94e2fb1c2d16e0de2e08522a9f993",
  "recipe_search_definition": "spec",
  "$defs": {
   "evaluation_case": {

{mostlyright_data-0.30.2 → mostlyright_data-0.30.4}/src/mostlyright/data_harness/thin/v4.py

@@ -148,8 +148,8 @@ from mostlyright.data_harness.thin.transport import ThinLaneError
 #: ``JOB_INVALID`` outright, which in production was every refresh a collection epoch was offered
 #: for rather than only the collection runs. This package must not reach a Studio older than the
 #: commit it pins; ``docs/V4-WORKER-PROTOCOL.md`` states it beside the layout's own ordering rule.
-PINNED_V4_OPENAPI_SOURCE_SHA256 = "001a5e132e8af9872ac61168e41df4465627f34ac062066b212f1b48a88385f4"
-PINNED_V4_CONTRACT_VERSION = "4.35.0"
+PINNED_V4_OPENAPI_SOURCE_SHA256 = "67cc1e66ae3610d9ae5c8b5469bb05e51a8519e55ecef502387fdbb4185b084e"
+PINNED_V4_CONTRACT_VERSION = "4.38.0"
 
 #: The ``schema_version`` every command body that predates 4.14.0 stamps: ``dataset activity``,
 #: ``table rename``, ``table archive`` and ``dataset archive``. It is the number the 0.25 line

{mostlyright_data-0.30.2 → mostlyright_data-0.30.4}/src/mostlyright/data_harness/thin/v4_runs.py

@@ -2355,6 +2355,8 @@ def status(args: argparse.Namespace, *, client: StudioV4RunClient | None = None)
         held = {member: run[member] for member in _PROJECTION_MEMBERS[1:] if member in run}
         if held:
             payload["projection"] = held
+    if state in ATTEMPTING_RUN_STATUSES:
+        payload.update(attempt_progress_members(selected, run_id))
     if getattr(args, "rate", False) and state not in TERMINAL_RUN_STATUSES:
         # Imported here: progress_rate reads the event stream, whose module imports this one.
         from mostlyright.data_harness.thin import progress_rate
@@ -2378,6 +2380,112 @@ def status(args: argparse.Namespace, *, client: StudioV4RunClient | None = None)
     return payload
 
 
+#: The states a run holds while it has, or is between, producer attempts. A run held at a spend
+#: or sample gate has never been claimed, so there is no attempt to read for it.
+ATTEMPTING_RUN_STATUSES: frozenset[str] = frozenset({"queued", "running"})
+
+#: How each way an attempt can end reads in the progress sentence. The keys are
+#: ``run-attempts.schema.json``'s closed ``end`` vocabulary.
+_ATTEMPT_ENDS: dict[str, str] = {
+    "running": "is still running",
+    "completed": "completed",
+    "requeued_by_worker": "was handed back by its own worker",
+    "handed_back": "was handed back after its worker went silent",
+    "failed": "failed",
+    "lost": "went silent",
+}
+
+
+def attempt_progress_members(client: StudioV4RunClient, run_id: str) -> dict[str, Any]:
+    """The ``status`` members for a live run's current attempt and its sources, or why not.
+
+    Read off the attempt ledger (``GET /v4/runs/{run_id}/attempts``), which Studio folds from
+    each progress tick as it arrives and never from the run log. So, unlike the rate, this does
+    not depend on replaying the log in time, and an attempt that has run for hours still says
+    how many of its sources it fetched. Counts are the current attempt's own: a run handed back
+    walks its sources again from the first.
+
+    A Studio before 4.37.0 serves no per-source member, and then the attempt and how the one
+    before it ended are still stated. A refused read is reported, never fatal: the run record
+    has already been read.
+    """
+
+    try:
+        answer = client.run_attempts(run_id)
+    except ThinLaneError as refusal:
+        return {"progress_unavailable": f"{refusal.code}: {refusal.detail}"}
+    rows = answer.get("attempts") if isinstance(answer, Mapping) else None
+    rows = [
+        row for row in rows or () if isinstance(row, Mapping) and type(row.get("generation")) is int
+    ]
+    if not rows:
+        return {"progress_unavailable": "Studio lists no attempt of this run yet"}
+    rows.sort(key=lambda row: row["generation"])
+    current = rows[-1]
+    sources = current.get("sources") if isinstance(current.get("sources"), Mapping) else None
+    named = sources.get("current") if sources is not None else None
+    progress: dict[str, Any] = {
+        "attempt": current["generation"],
+        "attempts": len(rows),
+        "source_count": sources.get("source_count") if sources is not None else None,
+        "sources_fetched": sources.get("fetched") if sources is not None else None,
+        "sources_refused": sources.get("refused") if sources is not None else None,
+        "counted_from_first_tick": (
+            sources.get("counted_from_first_tick") if sources is not None else None
+        ),
+        "current_source": dict(named) if isinstance(named, Mapping) else None,
+        "previous_attempt": None,
+    }
+    if len(rows) > 1:
+        previous = rows[-2]
+        loss = previous.get("loss") if isinstance(previous.get("loss"), Mapping) else {}
+        progress["previous_attempt"] = {
+            "attempt": previous["generation"],
+            "end": previous.get("end"),
+            "ended_at": previous.get("ended_at"),
+            "cause": loss.get("cause"),
+            "cause_inferred": loss.get("inferred"),
+        }
+    return {"progress": progress, "progress_sentence": attempt_progress_sentence(progress)}
+
+
+def attempt_progress_sentence(progress: Mapping[str, Any]) -> str:
+    """``Attempt 4 of 4 has fetched 27 of 86 sources and is on ...``, from the members above."""
+
+    said = f"Attempt {progress['attempt']} of {progress['attempts']}"
+    fetched, count = progress.get("sources_fetched"), progress.get("source_count")
+    if fetched is None:
+        said += "; this Studio reports no per-source progress for it."
+    else:
+        said += f" has fetched {fetched:,}"
+        said += f" of {count:,} sources" if count is not None else " sources"
+        if progress.get("counted_from_first_tick") is False:
+            said += " (at least: counting began after the attempt started)"
+        named = progress.get("current_source")
+        if isinstance(named, Mapping):
+            said += f" and is on {named.get('source_name')}"
+            if named.get("source_index") is not None and count is not None:
+                said += f", source {named['source_index']:,} of {count:,}"
+            done, planned = named.get("pages_processed"), named.get("pages_planned")
+            unit = "members" if named.get("pages_unit") == "member" else "pages"
+            if done is not None and planned is not None and named.get("discovery_complete"):
+                said += f": {done:,} of {planned:,} {unit}"
+            elif done is not None:
+                said += f": {done:,} {unit} so far"
+        said += "."
+    previous = progress.get("previous_attempt")
+    if isinstance(previous, Mapping):
+        end = _ATTEMPT_ENDS.get(str(previous.get("end")), f"ended {previous.get('end')}")
+        said += f" Attempt {previous['attempt']} {end}"
+        if previous.get("cause") is not None:
+            said += (
+                f"; {'inferred ' if previous.get('cause_inferred') else ''}cause: "
+                f"{previous['cause']}"
+            )
+        said += "."
+    return said
+
+
 def status_exit_code(payload: Mapping[str, Any]) -> int:
     """A failed run exits non-zero, so a script gating on a build cannot read it as a success."""