mostlyright-data 0.30.4__tar.gz → 0.30.6__tar.gz

{mostlyright_data-0.30.4 → mostlyright_data-0.30.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.30.4
+Version: 0.30.6
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.30.4 → mostlyright_data-0.30.6}/skills/mr-data-build/references/6-build-one-run-sized-to-acquire-every-measured-source-whole.md

@@ -146,19 +146,44 @@ projected to a few columns decodes to a few hundred kilobytes; an 8.6 MB table r
 to 8.6 MB. The fetched size predicts nothing on its own, and a source over either bound is refused
 `READER_BUDGET`, which names both.
 
-**How far a source may fetch depends on how it will be read.** A Reader-pinned source, and an
-unpinned `csv` one, fetch up to 256 MiB. An unpinned `ndjson` source fetches up to what one
-confined request carries, about 36 MB. An unpinned `json` or `parquet` source keeps 16 MiB,
-because neither has a row-at-a-time reader and both are read whole inside the clean room —
-pinning a Reader is the way past that. What a source may DECODE to is its Reader family's output
-budget, narrowed to at most 1 GiB, because the whole decoded relation is held in memory while it
-is classified and sealed. A wide source is still best narrowed by projection, which is what
-shrinks the decoded relation; a long source can still be split.
+**How far a source may fetch depends on whether it is staged, and then on how it will be read.**
+Studio collects the bytes of a `public.https` source at an `https_url` that names no credential,
+has a `public` classification and states the rights basis `public_domain_asserted`. A Generic API
+or a page collection is not one. Studio writes a collected object to the run's scratch disk and
+decodes it from that file, instead of holding it in memory, when the source is also read whole,
+with no `window`, and is either unpinned or pinned to a Reader with a file decoder. That is a
+staged source. Delimited text and Parquet have a file decoder, and so do the gzip, tar and zip
+archive Readers (not `archive.tar@2.0.0` or `archive.zip@2.0.0`), `meteorology.grib`,
+`meteorology.bufr` and the statistical microdata Readers; `json.tabular` has one in 1.1.0 and
+1.2.0, and only when it reads `ndjson`. The spreadsheet, XML, HTML, NetCDF and `weather.grib2`
+Readers have none, and a pin to one of them is fetched whole. A staged source is read up to 4 GiB,
+bounded by the `limits.max_source_bytes` you declare, with no other member to set. A sample that
+truncates rows (`--sample` with `--max-rows`) is never staged: a source that would be staged is
+read from memory instead, under the cap below.
+
+Every other `public.https` source is held in memory as it is fetched, and a Reader-pinned one, or an
+unpinned `csv` one, fetches up to 256 MiB. An unpinned `ndjson` source fetches up to what one
+confined request carries, about 36 MB. An unpinned `json` or `parquet` source keeps 16 MiB on either
+route, because neither has a row-at-a-time reader and both are read whole inside the clean room —
+pinning a Reader is the way past that. The exception is a source pinned to `parquet.tabular@1.1.0`,
+which fetches by range, up to 1 GiB.
+
+The rights basis is a statement about your right to use the source, and it is recorded with the
+run. It is not a size setting: state the one that is true, and let the ceiling follow.
+
+What a source that is not staged may DECODE to is its Reader family's output budget, narrowed to at
+most 1 GiB, because the whole decoded relation is held in memory while it is classified and sealed.
+A staged source is decoded to a file and is not narrowed to that 1 GiB. On either route the decoded
+table is also held to the source's own `limits.max_source_bytes`, so when a source inflates, as a
+compressed archive does, declare that at the decoded size and not at the size of the download. A
+wide source is still best narrowed by projection, which is what shrinks the decoded relation; a
+long source can still be split.
 
 `limits.max_source_bytes` states where fetching STOPS rather than how big the source is, so the
 preflight does not refuse a large one. Registration puts one sentence on the receipt under
-`warnings` when the declared number is larger than a fetch can reach, and the sentence names the
-formats that stop earlier. Read that as "go and check this one", not as a failure.
+`warnings` when the declared number is larger than the route its source takes can reach, and the
+sentence names the route, the ceiling it used and the formats that stop earlier. Read that as "go
+and check this one", not as a failure.
 
 **Which remedy applies depends on which way the source is too big, and lowering the ceiling is
 never it** — that truncates the preview or refuses the full build rather than fetching less. A WIDE

{mostlyright_data-0.30.4 → mostlyright_data-0.30.6}/skills/mr-data-build/references/making-a-table-incremental.md

@@ -209,6 +209,7 @@ source and the pointer to change.
 | `REFRESH_WORKER_UNAVAILABLE` | This Studio has no refresh worker for the plan. | `wait_for_platform`: report "not available on this Studio yet". |
 | `REBUILD_DISCARDS_SNAPSHOT_HISTORY` | A table declares `rebuild` beside a snapshot source. | `split_table`: keep the snapshot source in a table with no `rebuild`. |
 | `ROW_REQUEST_ANSWERS_NEVER_EXPIRE` | Advisory. A per-row request names no date parameter and its answers are reused with no bound. | `declare_revisit_interval` or `read_answers_fresh`, both in [Needs Studio 4.14](#needs-studio-414). |
+| `PREDECESSOR_TRUNCATED` | The live version is a `sample` that a row or window clamp cut short, so a refresh would seal over rows it never fetched. Needs Studio 4.33. | `resync_once`: run one full build. |
 
 A `permanence` of `predecessor` clears after one full. `estate` is the platform's state, not the
 recipe's. Report it and do not revise the document for it. `document` needs a revision.

{mostlyright_data-0.30.4 → mostlyright_data-0.30.6}/skills/mr-data-build/references/recipe-test.md

@@ -24,7 +24,14 @@ What it reads:
 - A per-row source that presents a credential gets no requests unless `--allow-paid` is set. It
   is reported as skipped, and its relation reaches the transform empty. Its provider's
   per-request price is the reason, so forecast it and ask the user before adding the flag.
-- Generic API walks and whole-file sources are bounded only by the 256 MiB acquisition clamp.
+- Generic API walks and whole-file sources are bounded only by the test's own clamp of 256 MiB
+  on source bytes. A whole file larger than that is refused by the test, although the full reads a
+  staged source up to 4 GiB
+  ([how far a source may fetch](6-build-one-run-sized-to-acquire-every-measured-source-whole.md)).
+  `mr-data run --full` runs this test first, so a staged source over 256 MiB stops it at
+  `run_stopped_by_test`. That stop comes from the clamp and not from anything wrong in the recipe;
+  `--skip-test`, below, is how the user starts that full once they have decided to. A whole file
+  that is not staged is refused by the full as well, and `--skip-test` does not help it.
 
 Read `status`:
 

{mostlyright_data-0.30.4 → mostlyright_data-0.30.6}/src/mostlyright/data_harness/thin/recipe.py

@@ -69,7 +69,7 @@ from mostlyright.data_harness.thin.session import (
     open_studio_session,
     sessions_for,
 )
-from mostlyright.data_harness.thin.transport import ThinLaneError
+from mostlyright.data_harness.thin.transport import ThinLaneError, UrlLibTransport
 from mostlyright.data_harness.thin.v4 import identifier
 from mostlyright.data_harness.thin.v4_datasets import (
     StudioV4DatasetClient,
@@ -114,7 +114,9 @@ REQUIRED_DOCUMENT_MEMBERS: tuple[str, ...] = (
 
 #: The declared ``limits.max_source_bytes`` above which one source is worth a word, in bytes --
 #: for a source that is read through descriptors, which is a Reader-pinned one or an unpinned
-#: ``csv`` one. That is the hosted fetch cap.
+#: ``csv`` one -- and that is not STAGED (:func:`_is_staged`). That is the hosted whole-read cap,
+#: ``hosted_crawler.MAX_RAW_RESPONSE_BYTES``: the most a fetch that holds the body in memory may
+#: take.
 READER_SOURCE_BYTE_CEILING = 268_435_456
 
 #: The same ceiling for an unpinned ``ndjson`` source. It has an inline streaming pass and no
@@ -127,6 +129,71 @@ NDJSON_SOURCE_BYTE_CEILING = 37_745_664
 #: Pinning a Reader is what moves a source off this ceiling.
 DRAINED_SOURCE_BYTE_CEILING = 16_777_216
 
+#: A source pinned to ``parquet.tabular@1.1.0`` is read by range, and its declared
+#: ``max_source_bytes`` bounds the bytes that read fetches -- the footer and the selected column
+#: chunks -- rather than the object, which may be far larger. Its ceiling is the ranged read's own
+#: (``readers.budgets.MAX_RANGED_READ_BYTES``), not the hosted cap, and the coordinate is
+#: ``readers.parquet_tabular.RANGED_PARQUET_COORDINATE``. Stated here for the reason below.
+RANGED_READ_SOURCE_BYTE_CEILING = 1_073_741_824
+RANGED_READER_COORDINATE = ("parquet.tabular", "1.1.0")
+
+#: The same threshold for a STAGED source: one that Studio collects, reads whole and decodes from a
+#: file (:func:`_is_staged`). That object is written to the run's scratch disk instead of being
+#: held, so the whole-read cap does not bind it (Studio #1989, ADR 0099);
+#: ``hosted_crawler.MAX_COLLECTED_FILE_SOURCE_BYTES`` does.
+#:
+#: ⚠ STATED HERE, NOT IMPORTED FROM STUDIO, AND HELD EQUAL TO IT BY A TEST. This client is
+#: standard-library only and the crawler is part of the engine it must never carry, so it cannot
+#: read the ceilings off the module that enforces them. The cross-repository suite
+#: (``tests/test_thin_worker_contract_constants.py``) reads that module's source and fails when
+#: either ceiling here, the ranged one or the table below stops being Studio's. It needs a Studio
+#: checkout and so does not run in CI: run ``make verify-cross-repo`` after a change to any of
+#: them. The sentence that said every Reader-pinned source stops at 256 MiB went on being printed
+#: for days after ADR 0099, because the guard compared the figure with the wrong constant
+#: (mostlyrightmd/mostlyright-planning#1576).
+STAGED_SOURCE_BYTE_CEILING = 4 * 1024**3
+
+#: Under this execution policy a staged read has no fixed ceiling: ``limits.max_source_bytes``
+#: bounds it, narrowed by what the run's capacity admits (ADR 0099).
+_BOUNDED_STREAMING_POLICY = "bounded-streaming.v1"
+
+#: The Reader coordinates a staged file can be decoded by. A pin to any other is decoded from bytes
+#: held in memory, however the source is collected. Restated from ``acquisition.file_decoding``
+#: (the keys of ``_FILE_FAMILIES`` joined with those of ``_SCRATCH_FILE_FAMILIES``) for the reason
+#: above, and held equal to them by a test in both directions: Studio #2050 added
+#: ``archive.zip@3.0.0`` to the table while the change that first restated it was in review.
+#: ``json.tabular`` has a decoder only for ``ndjson`` (:func:`_reads_ndjson`). Among the certified
+#: coordinates without one are ``spreadsheet.xlsx``, ``xml.tabular``, ``html.web_extract``,
+#: ``scientific.netcdf`` and ``weather.grib2``.
+FILE_DECODER_COORDINATES: frozenset[tuple[str, str]] = frozenset(
+    (family, version)
+    for family, versions in {
+        "archive.gzip": ("1.0.0", "1.1.0", "2.0.0"),
+        "archive.tar": ("1.0.0", "1.1.0", "3.0.0"),
+        "archive.zip": ("1.0.0", "1.1.0", "1.2.0", "1.3.0", "1.4.0", "1.5.0", "3.0.0"),
+        "delimited_text": ("1.0.0", "1.1.0", "2.0.0"),
+        "geoparquet.vector": ("1.0.0",),
+        "json.tabular": ("1.1.0", "1.2.0"),
+        "meteorology.bufr": ("1.0.0",),
+        "meteorology.grib": ("1.0.0",),
+        "microdata.sas_xpt": ("1.0.0",),
+        "microdata.spss_sav": ("1.0.0",),
+        "microdata.stata_dta": ("1.0.0",),
+        "official.pcaxis_px": ("1.0.0",),
+        "parquet.tabular": ("1.0.0", "1.1.0"),
+        "raster.geospatial": ("1.0.0",),
+        "satellite.himawari_hsd_complete": ("1.0.0",),
+        "satellite.msg_seviri_l15": ("1.0.0",),
+        "satellite.sentinel_safe": ("1.0.0",),
+        "satellite.xrit_image": ("1.0.0",),
+        "scientific.fits": ("1.0.0",),
+        "scientific.hdf5": ("1.0.0",),
+        "scientific.zarr": ("1.0.0",),
+        "xbrl.facts": ("1.0.0",),
+    }.items()
+    for version in versions
+)
+
 #: How many undescribed columns the warning names before it counts the rest. A table plan admits
 #: 512 columns, and a sentence that spelt every one of them out would be the wall the `numbered`
 #: receipt member exists to avoid. The receipt carries the whole count as a number beside it, so
@@ -159,10 +226,10 @@ PROPOSE = "propose"
 EXPLAIN = recipe_explain.EXPLAIN
 FORECAST = "forecast"
 TEST = recipe_test.TEST
-# Studio evaluates the exact persisted admission proof for each recipe revision. Keep this
-# inventory page small enough that a normal production workspace completes each bounded request
-# inside the thin client's request deadline, while preserving the established 40,000-item
-# inventory bound through cursor aggregation.
+# Studio evaluates the exact persisted admission proof for each recipe revision. A page of 25 is
+# quick on most workspaces; where one revision takes minutes the walk halves the page down to a
+# single row under READINESS_REQUEST_TIMEOUT_SECONDS, at the cost of the timed-out attempts first.
+# Cursor aggregation keeps the established 40,000-item inventory bound.
 READINESS_PAGE_SIZE = 25
 MAX_READINESS_ITEMS = 40_000
 MAX_READINESS_PAGES = MAX_READINESS_ITEMS // READINESS_PAGE_SIZE
@@ -175,6 +242,11 @@ MAX_READINESS_HALVINGS = 5
 #: (the transport's deadline), or a gateway that gave up waiting for Studio.
 READINESS_TIMEOUT_STATUSES = frozenset({502, 503, 504})
 
+#: How long one readiness page may take. Studio evaluates each revision's persisted admission
+#: proof, and on a large production workspace a single revision has taken about 110 seconds, past
+#: the thirty seconds an ordinary request gets: no page, however halved, could come back.
+READINESS_REQUEST_TIMEOUT_SECONDS = 300.0
+
 #: How long the walk waits before it asks a page again after one of those statuses, doubling to the
 #: ceiling, each wait jittered over the upper half of its step. A gateway that answered at once
 #: names an overloaded Studio, and every client in the walk asking straight back adds to the load.
@@ -419,7 +491,7 @@ def check_document(document: dict[str, Any]) -> None:
     check_dataset_binding(document["dataset"])
 
 
-def source_byte_warnings(sources: Any) -> list[str]:
+def source_byte_warnings(sources: Any, *, execution_policy: Any = None) -> list[str]:
     """One sentence per source whose declared ceiling is larger than the hosted lane fetches.
 
     ⚠ WARNINGS, AND NOT A GATE, FOR THE REASON :data:`READER_SOURCE_BYTE_CEILING` GIVES: the
@@ -428,6 +500,18 @@ def source_byte_warnings(sources: Any) -> list[str]:
     decides anything; it tells the agent that wrote the number that the hosted lane will not honour
     it.
 
+    ⚠ "THE HOSTED LANE" IS SEVERAL ROUTES, AND A SOURCE IS HELD TO THE CEILING OF THE ONE IT TAKES.
+    A staged source is written to disk and read to :data:`STAGED_SOURCE_BYTE_CEILING`, a ranged
+    Parquet read has its own, and every other source is held to
+    :data:`READER_SOURCE_BYTE_CEILING` (:func:`_source_byte_ceiling`). Telling a staged source that
+    its fetch stops at 256 MiB sent authors off to shrink sources that read in full
+    (mostlyrightmd/mostlyright-planning#1574), and this client's build skill said the same
+    (mostlyrightmd/mostlyright-planning#1576).
+
+    ⚠ UNDER ``bounded-streaming.v1`` A STAGED READ HAS NO CEILING TO OVERSTATE, so it is not
+    warned about: the declared number alone bounds it. That policy is a member of the document, not
+    of a source, which is why it is passed in beside the sources.
+
     ⚠ THE LARGEST SOURCE DECIDES, NOT THE SUM. One source is fetched at a time, so the sentence is
     about the one that overstates its ceiling rather than about the total.
 
@@ -448,7 +532,10 @@ def source_byte_warnings(sources: Any) -> list[str]:
         declared = limits.get("max_source_bytes")
         if type(declared) is not int:
             continue
-        ceiling, why = _source_byte_ceiling(source)
+        held = _source_byte_ceiling(source, execution_policy=execution_policy)
+        if held is None:
+            continue
+        ceiling, why, sooner = held
         if declared <= ceiling:
             continue
         named = source.get("name")
@@ -457,10 +544,7 @@ def source_byte_warnings(sources: Any) -> list[str]:
             f"source {where} declares limits.max_source_bytes of {declared:,} bytes, and {why} "
             f"stops at {ceiling:,}. That is a ceiling on fetching rather than a measurement of "
             "the source, so this registers and runs, and the fetch stops there whatever the "
-            "document says. An unpinned source that answers ndjson stops at "
-            f"{NDJSON_SOURCE_BYTE_CEILING:,} and one that answers json or parquet at "
-            f"{DRAINED_SOURCE_BYTE_CEILING:,}, and pinning a Reader is what moves it off "
-            "those. Declare what the source actually weighs"
+            f"document says. {sooner}. Declare what the source actually weighs"
         )
     return said
 
@@ -1013,24 +1097,156 @@ def sample_span_warnings(sources: Any) -> list[str]:
     return [". ".join(said)]
 
 
-def _source_byte_ceiling(source: Mapping[str, Any]) -> tuple[int, str]:
-    """The fetch ceiling one declared source is held to, and the phrase that names it.
+def _stated_parameters(source: Mapping[str, Any]) -> dict[str, Any]:
+    """The ``connector.parameters`` one source states, by name.
 
-    A document declares a Reader pin on ``connector.parameters`` and declares no wire format at
-    all: the format is what the response turns out to be. So the one thing this client can read
-    off the document is whether the source is pinned. A pinned source fetches to the hosted cap
-    whatever its wire format. An unpinned source is held to the same cap here, and the sentence
-    says which formats stop earlier, because that is decided by bytes this client has not seen.
+    A parameter whose name is not text names nothing, so it is left out and never hashed: the
+    schema refuses it and the server says so, and this runs before the server has been asked
+    (``--no-lint`` lets a document that fails the schema reach it).
     """
 
     connector = source.get("connector")
     parameters = connector.get("parameters") if isinstance(connector, Mapping) else None
-    if isinstance(parameters, list) and any(
-        isinstance(parameter, Mapping) and parameter.get("name") == "reader.family_id"
+    if not isinstance(parameters, list):
+        return {}
+    return {
+        parameter["name"]: parameter.get("value")
         for parameter in parameters
-    ):
-        return READER_SOURCE_BYTE_CEILING, "a Reader-pinned source's fetch"
-    return READER_SOURCE_BYTE_CEILING, "an unpinned source's fetch"
+        if isinstance(parameter, Mapping) and isinstance(parameter.get("name"), str)
+    }
+
+
+def _is_collected(source: Mapping[str, Any]) -> bool:
+    """Whether Studio's shared fetch lane collects this source's bytes.
+
+    Five members of the document decide it: the adapter is exactly ``public.https``, the connector
+    names no credential, the locator is an https URL, the classification is ``public`` and the
+    rights basis is ``public_domain_asserted``. The worker reads these five
+    (``hosted_run_worker._uses_collected_source_fetch``, beside a ``window`` and what the job
+    delegated) and so does Studio's route matrix. A ``public.https`` source that fails one of them
+    is not collected. Collected is not staged: that is :func:`_is_staged`'s question.
+
+    ⚠ THE FIVE ARE HELD TO THE WORKER'S, not merely to this docstring:
+    ``tests/test_thin_worker_contract_constants.py`` reads the members that function compares and
+    fails when they change.
+    """
+
+    connector = source.get("connector")
+    locator = source.get("locator")
+    rights = source.get("rights_claim")
+    return (
+        isinstance(connector, Mapping)
+        and connector.get("adapter_id") == "public.https"
+        and connector.get("credential_mode") == "none"
+        and isinstance(locator, Mapping)
+        and locator.get("kind") == "https_url"
+        and source.get("data_classification") == "public"
+        and isinstance(rights, Mapping)
+        and rights.get("claimed_basis") == "public_domain_asserted"
+    )
+
+
+def _reads_ndjson(options: Any) -> bool:
+    """Whether a ``reader.decode_options`` value, a JSON text, says ``document_format: "ndjson"``.
+
+    ``json.tabular`` has a file decoder only for newline-delimited documents
+    (``acquisition.file_decoding.file_decode_supported``), and its ``document_format`` defaults to
+    ``json``. Text that is not canonical JSON, or not an object, reads as not ndjson: the server
+    validates the options, and this only decides which ceiling to name.
+
+    ⚠ IT MUST NOT RAISE, because it runs before the server has been asked and the preflight lets
+    a long, hostile-looking pin through: the schema admits 262,144 characters. The parser refuses
+    most bad text with its own error, but text nested a few thousand levels deep escapes it as a
+    ``RecursionError`` and an integer of more than 4,300 digits as a ``ValueError``, and either
+    would turn a registration into ``THIN_COMMAND_FAILED`` before Studio could say what is wrong.
+    """
+
+    if not isinstance(options, str):
+        return False
+    try:
+        stated = parse_json(options)
+    except (RecursionError, ValueError):
+        # `CanonicalJSONError` is a `ValueError`, so this is the parser's own refusal as well.
+        return False
+    return isinstance(stated, Mapping) and stated.get("document_format") == "ndjson"
+
+
+def _is_staged(source: Mapping[str, Any], stated: Mapping[Any, Any]) -> bool:
+    """Whether a run reads this source from a file staged on the scratch disk, not from memory.
+
+    ADR 0099 stages a COLLECTED object (:func:`_is_collected`) and decodes it from that file, on
+    two more conditions the document states. The source is read whole, so it has no ``window``
+    member: a windowed read is fetched a window at a time. And it is either unpinned -- ``csv`` and
+    ``ndjson`` stage -- or pinned to a Reader with a file decoder (:data:`FILE_DECODER_COORDINATES`,
+    and ``json.tabular`` only when it reads ``ndjson``). A pin to any other Reader is decoded from
+    bytes held in memory, however the source is collected. The table also holds the ranged Parquet
+    coordinate; :func:`_source_byte_ceiling` has already given that pin its own ceiling.
+
+    ⚠ A DOCUMENT-LEVEL ANSWER, FOR THE FULL READ. What a run holds is not in the document: a
+    scratch disk, and a truncating row clamp (a sample), which is read whole and never staged. A
+    source this calls staged is staged by a full run, and that is the read the ceiling is warned
+    against.
+    """
+
+    if not _is_collected(source) or source.get("window") is not None:
+        return False
+    if "reader.family_id" not in stated:
+        return True
+    family = stated.get("reader.family_id")
+    version = stated.get("reader.family_version")
+    # A value the schema refuses (a list, a number) is not a coordinate: it must not reach the
+    # table, where an unhashable one would raise before the server could answer with its 422.
+    if not isinstance(family, str) or not isinstance(version, str):
+        return False
+    if (family, version) not in FILE_DECODER_COORDINATES:
+        return False
+    return family != "json.tabular" or _reads_ndjson(stated.get("reader.decode_options"))
+
+
+def _source_byte_ceiling(
+    source: Mapping[str, Any], *, execution_policy: Any = None
+) -> tuple[int, str, str] | None:
+    """Where fetching one declared source stops, the phrase that names it, and what stops sooner.
+
+    ``None`` when nothing but the declared number does: a staged read under
+    ``bounded-streaming.v1``.
+
+    A document declares a Reader pin on ``connector.parameters`` and declares no wire format at
+    all: the format is what the response turns out to be. So what this client reads off the
+    document is the ROUTE a source takes, and whether it is pinned, and to what. A ranged Parquet
+    pin bounds the bytes its ranged read fetches, at the ranged ceiling; that read is never
+    staged, so it is checked first. A staged source is read to the staged ceiling. Every other
+    source is held to the whole-read cap here, pinned or not, and the sentence says which unpinned
+    formats stop earlier, because that is decided by bytes this client has not seen.
+
+    The third member is that sentence, because which formats stop sooner depends on the route: a
+    staged unpinned ``csv`` or ``ndjson`` source is read from the file, so only ``json`` and
+    ``parquet`` stop early; fetched whole, ``ndjson`` stops early as well.
+    """
+
+    stated = _stated_parameters(source)
+    whole_read_sooner = (
+        f"An unpinned source that answers ndjson stops at {NDJSON_SOURCE_BYTE_CEILING:,} and one "
+        f"that answers json or parquet at {DRAINED_SOURCE_BYTE_CEILING:,}, and pinning a Reader "
+        "is what moves it off those"
+    )
+    if (
+        stated.get("reader.family_id"),
+        stated.get("reader.family_version"),
+    ) == RANGED_READER_COORDINATE:
+        return RANGED_READ_SOURCE_BYTE_CEILING, "a ranged Parquet read", whole_read_sooner
+    if _is_staged(source, stated):
+        if execution_policy == _BOUNDED_STREAMING_POLICY:
+            return None
+        return (
+            STAGED_SOURCE_BYTE_CEILING,
+            "a collected source's staged read",
+            "An unpinned source that answers json or parquet stops at "
+            f"{DRAINED_SOURCE_BYTE_CEILING:,}, and pinning a Reader is what moves it off that",
+        )
+    if "reader.family_id" in stated:
+        return READER_SOURCE_BYTE_CEILING, "a Reader-pinned source's fetch", whole_read_sooner
+    return READER_SOURCE_BYTE_CEILING, "an unpinned source's fetch", whole_read_sooner
 
 
 def columns_without_description(table: Any) -> list[str]:
@@ -1253,7 +1469,9 @@ def register(
     undescribed = columns_without_description(document.get("table"))
     uncontinued = sources_without_continuation(document.get("sources"))
     declared_sources = document.get("sources")
-    warnings = source_byte_warnings(declared_sources)
+    warnings = source_byte_warnings(
+        declared_sources, execution_policy=document.get("execution_policy")
+    )
     warnings += continuation_warnings(
         uncontinued, len(declared_sources) if isinstance(declared_sources, list) else 0
     )
@@ -1614,7 +1832,9 @@ def readiness(
         recipe_id = identifier(recipe_id, "recipe identifier")
     if client is None:
         sessions = sessions or open_renewable_session(resolve_cloud_credentials())
-        selected = StudioV4DatasetClient(sessions.session)
+        selected = StudioV4DatasetClient(
+            sessions.session, transport=UrlLibTransport(timeout=READINESS_REQUEST_TIMEOUT_SECONDS)
+        )
     else:
         selected = client
         sessions = sessions or sessions_for(selected.session)
@@ -2015,6 +2235,7 @@ __all__ = [
     "COMMAND_HELP",
     "DRAINED_SOURCE_BYTE_CEILING",
     "EXPLAIN",
+    "FILE_DECODER_COORDINATES",
     "FIRST_READINESS_RETRY_SECONDS",
     "FORECAST",
     "MAX_DOCUMENT_BYTES",
@@ -2028,6 +2249,8 @@ __all__ = [
     "NDJSON_SOURCE_BYTE_CEILING",
     "PROPOSE",
     "PROPOSE_SCHEMA",
+    "RANGED_READER_COORDINATE",
+    "RANGED_READ_SOURCE_BYTE_CEILING",
     "READER_SOURCE_BYTE_CEILING",
     "READINESS",
     "READINESS_FILTERS_SINCE",
@@ -2037,6 +2260,7 @@ __all__ = [
     "REQUIRED_DOCUMENT_MEMBERS",
     "SHOW",
     "SHOW_SCHEMA",
+    "STAGED_SOURCE_BYTE_CEILING",
     "TEST",
     "UNREFRESHABLE_PLAN_TAIL",
     "SampleSpan",

{mostlyright_data-0.30.4 → mostlyright_data-0.30.6}/src/mostlyright/data_harness/thin/recipe_explain.py

@@ -81,6 +81,9 @@ RECORDING_CONTINUATION_PENDING = "RECORDING_CONTINUATION_PENDING"
 REFRESH_WORKER_UNAVAILABLE = "REFRESH_WORKER_UNAVAILABLE"
 REBUILD_DISCARDS_SNAPSHOT_HISTORY = "REBUILD_DISCARDS_SNAPSHOT_HISTORY"
 ROW_REQUEST_ANSWERS_NEVER_EXPIRE = "ROW_REQUEST_ANSWERS_NEVER_EXPIRE"
+#: Contract 4.33.0 (planning#1337): a refresh whose predecessor is a sample a row or window clamp
+#: cut short. It needs a sealed predecessor, so the mirror never claims it.
+PREDECESSOR_TRUNCATED = "PREDECESSOR_TRUNCATED"
 
 RULE_CODES: tuple[str, ...] = (
     NO_SEALED_PREDECESSOR,
@@ -98,10 +101,11 @@ RULE_CODES: tuple[str, ...] = (
     REFRESH_WORKER_UNAVAILABLE,
     REBUILD_DISCARDS_SNAPSHOT_HISTORY,
     ROW_REQUEST_ANSWERS_NEVER_EXPIRE,
+    PREDECESSOR_TRUNCATED,
 )
 
 #: The codes the mirror can observe from a document alone, and so the codes a disagreement is
-#: judged over. The other six need a predecessor or the deployment (a sealed version, a table
+#: judged over. The other seven need a predecessor or the deployment (a sealed version, a table
 #: format, a certified worker, a recording bookmark), and the mirror never claims them.
 #: ``REFRESH_WORKER_UNAVAILABLE`` is judged only where it names sources: the snapshot fence, which
 #: ``capabilities.snapshot_refresh`` states; the certified-worker case names none.
@@ -1249,8 +1253,11 @@ def explain_document_at_studio(
     except ThinLaneError as refusal:
         if route_unavailable(refusal):
             return None
-        keyless = [pointer for pointer, capability in gated_grammar(document)
-                   if capability == "keyless_window"]
+        keyless = [
+            pointer
+            for pointer, capability in gated_grammar(document)
+            if capability == "keyless_window"
+        ]
         if keyless and getattr(refusal, "http_status", None) == 422:
             # Older API schemas reject the document before they can answer capabilities.
             # A schema rejection is never evidence that this grammar is served.
@@ -1393,9 +1400,7 @@ def registration_gate(client: StudioV4Client, document: Mapping[str, Any]) -> li
     return grammar_gate(document, answer)
 
 
-def grammar_gate(
-    document: Mapping[str, Any], answer: Mapping[str, Any] | None
-) -> list[str]:
+def grammar_gate(document: Mapping[str, Any], answer: Mapping[str, Any] | None) -> list[str]:
     """Admit only grammar explicitly served by an explanation; perform no writes."""
 
     gated = gated_grammar(document)

{mostlyright_data-0.30.4 → mostlyright_data-0.30.6}/src/mostlyright/data_harness/thin/transport.py

@@ -105,7 +105,9 @@ class UrlLibTransport:
     under a two-second clock for the same reason. Two clients pass a LONGER one, because each calls
     a Cloud route that waits on Studio before it answers: ``v4_public.PublicQueryClient`` (a public
     query) and ``v4_artifacts.CloudArtifactSessionClient`` (the metered signed-read mints, where
-    Cloud allows Studio sixty seconds). Nothing else passes it.
+    Cloud allows Studio sixty seconds), and ``recipe readiness`` waits five minutes for a page
+    Studio evaluates revision by revision. Others pass their own clocks too; read each caller's
+    constant.
 
     ⚠ AND IT MAY BE A CALLABLE, WHICH IS THE SAME PARAMETER SAYING A DIFFERENT KIND OF THING. A
     constant bounds each request separately, so N requests cost N times the clock; a callable is

{mostlyright_data-0.30.4 → mostlyright_data-0.30.6}/src/mostlyright/data_harness/thin/v4_stream.py

@@ -255,8 +255,9 @@ def watch(
     rates = RateTracker()
 
     def _observe(body: Mapping[str, Any], frame_name: str) -> bool:
-        nonlocal seen_any
+        nonlocal seen_any, quiet_since
         seen_any = True
+        quiet_since = clock()
         if announce:
             print(render_v4_frame(body, frame_name), flush=True)
         moved = rates.observe(body)
@@ -271,10 +272,20 @@ def watch(
 
     def _quiet() -> bool:
         # Asked on every heartbeat of a quiet stream. Never stops the watch; it only says, once
-        # per quiet stretch, that nothing has arrived yet and why that is the ordinary case.
+        # per quiet interval, what was (and was not) observed. Heartbeats prove only
+        # stream connectivity, never worker liveness or progress.
         nonlocal quiet_since
         now = clock()
-        if announce and not seen_any and now - quiet_since >= QUIET_NOTE_SECONDS:
+        if announce and now - quiet_since >= QUIET_NOTE_SECONDS:
+            if seen_any:
+                print(
+                    f"no new progress report for {int(now - quiet_since)} seconds: "
+                    "work may still be running; stream connectivity is not worker progress; "
+                    "ETA unknown",
+                    flush=True,
+                )
+                quiet_since = now
+                return False
             waited = int(now - started)
             print(
                 f"nothing reported yet after {waited} seconds: the run was {run.get('status')} "