mostlyright-data 0.24.0__tar.gz → 0.25.0__tar.gz

{mostlyright_data-0.24.0 → mostlyright_data-0.25.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.24.0
+Version: 0.25.0
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
@@ -112,14 +112,17 @@ its raw inputs still retained. Replay compares against that run and never become
 Studio returns a typed refusal when replay is unavailable; the CLI does not fetch sources locally.
 
 The hosted engine executes sample, full and refresh runs. Studio's action classifier can name
-closed-source reuse, request-window or collection acquisition, and recorded stream input, but the
-current worker has only two executable normal-refresh materializers: an exact single direct
-partition request-window plan with a compatible predecessor, and an all-recorded-stream plan.
-Closed-only, collection, and mixed plans return `RESYNC_REQUIRED` before acquisition, as does a
-mutable snapshot or other unwindowed source. Nothing is revalidated or re-acquired whole as an
-implicit fallback. Use `mr-data table resync TABLE_ID --request-id UUID` when a deliberate full
-source reread is intended, retaining that UUID through an uncertain response. URL windows may use
-declared query parameters or path placeholders; a fixed date URL does not advance automatically.
+closed-source reuse, request-window or collection acquisition, and recorded stream input, but
+normal refresh admits only two families of plan: every source a recorded stream, and at least one
+direct partition request window with a compatible predecessor, with `closed` reuse permitted
+beside it. Once such a plan holds more than one source, every window in it must agree on one
+`merge.materialization`, `merge.partition.column` and `merge.partition.key`, and that column must
+be one the table declares. Closed-only plans, any plan carrying a collection, a window beside a
+recorded stream, a mutable snapshot and any other unwindowed source return `RESYNC_REQUIRED`
+before acquisition. Nothing is revalidated or re-acquired whole as an implicit fallback. Use
+`mr-data table resync TABLE_ID --request-id UUID` when a deliberate full source reread is
+intended, retaining that UUID through an uncertain response. URL windows may use declared query
+parameters or path placeholders; a fixed date URL does not advance automatically.
 Configure a correction lookback where the publisher can revise earlier observations.
 
 A source exposing only its current snapshot cannot supply a historical delta. It is supported only

{mostlyright_data-0.24.0 → mostlyright_data-0.25.0}/README.md

@@ -100,14 +100,17 @@ its raw inputs still retained. Replay compares against that run and never become
 Studio returns a typed refusal when replay is unavailable; the CLI does not fetch sources locally.
 
 The hosted engine executes sample, full and refresh runs. Studio's action classifier can name
-closed-source reuse, request-window or collection acquisition, and recorded stream input, but the
-current worker has only two executable normal-refresh materializers: an exact single direct
-partition request-window plan with a compatible predecessor, and an all-recorded-stream plan.
-Closed-only, collection, and mixed plans return `RESYNC_REQUIRED` before acquisition, as does a
-mutable snapshot or other unwindowed source. Nothing is revalidated or re-acquired whole as an
-implicit fallback. Use `mr-data table resync TABLE_ID --request-id UUID` when a deliberate full
-source reread is intended, retaining that UUID through an uncertain response. URL windows may use
-declared query parameters or path placeholders; a fixed date URL does not advance automatically.
+closed-source reuse, request-window or collection acquisition, and recorded stream input, but
+normal refresh admits only two families of plan: every source a recorded stream, and at least one
+direct partition request window with a compatible predecessor, with `closed` reuse permitted
+beside it. Once such a plan holds more than one source, every window in it must agree on one
+`merge.materialization`, `merge.partition.column` and `merge.partition.key`, and that column must
+be one the table declares. Closed-only plans, any plan carrying a collection, a window beside a
+recorded stream, a mutable snapshot and any other unwindowed source return `RESYNC_REQUIRED`
+before acquisition. Nothing is revalidated or re-acquired whole as an implicit fallback. Use
+`mr-data table resync TABLE_ID --request-id UUID` when a deliberate full source reread is
+intended, retaining that UUID through an uncertain response. URL windows may use declared query
+parameters or path placeholders; a fixed date URL does not advance automatically.
 Configure a correction lookback where the publisher can revise earlier observations.
 
 A source exposing only its current snapshot cannot supply a historical delta. It is supported only

{mostlyright_data-0.24.0 → mostlyright_data-0.25.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.24.0"
+version = "0.25.0"
 description = "Mostly Right hosted CLI for reviewed datasets"
 readme = "README.md"
 requires-python = ">=3.11"

{mostlyright_data-0.24.0 → mostlyright_data-0.25.0}/skills/mr-data-build/SKILL.md

@@ -37,6 +37,24 @@ references. Load the reference for the current stage only, not the entire librar
    Bind `dataset.id`, describe columns, declare checks and units, and register with
    `mr-data recipe RECIPE.json --json`. Repair all lint findings before queuing work.
    The server returns the recipe digest; do not compute or invent it.
+   Decide each source's refresh continuation before you register it: `closed: true` where the
+   address names a range that has demonstrably ended, a `window` with **both** `request`
+   endpoints where the address carries a date range, a `collection` where the corpus is an index
+   of pages, and nothing where the source is mutable with no changed-since contract. A source
+   declaring none of these is `resync_only`, and one `resync_only` source makes the entire table
+   resync-only: it will never refresh on a schedule. Declaring every source is only the first
+   gate; the shape those declarations form is the second, and Studio admits two families only:
+   every source a recorded stream, or at least one `window` with a `request`, with `closed`
+   siblings permitted beside it. A closed-only plan, any plan carrying a `collection`, and a
+   `window` beside a recorded stream are each refused, so `closed: true` alone does not make a
+   table refresh. The second family is necessary and not sufficient: as soon as the plan holds
+   more than one source, every window in it must agree on one `merge.materialization`,
+   `merge.partition.column` and `merge.partition.key`, and that column must be a name
+   `table.columns` declares. Adding one `closed` sibling beside a lone window is what turns that
+   requirement on, so a window partitioning on the publisher's own header under a name the table
+   renames refreshes alone and stops the moment a second source joins it. Declare only what the
+   source's own address proves, and name the `resync_only` sources and the reason in the build
+   message.
 6. Read [build sizing](references/6-build-one-run-sized-to-acquire-every-measured-source-whole.md).
    Choose bounds from measured source size; row and byte clamps apply per source. Use a bounded
    preview for unmeasured sources. Preserve the agreed semantics and quality checks.
@@ -72,7 +90,12 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
 8. Read `mr-data checks RUN_ID --json`, `mr-data receipt RUN_ID --json` and
    `mr-data peek RUN_ID --json`; use bounded `mr-data query` for agreed quality questions.
    Verify row grain, joins, requested fields, coverage and every declared check. Successful
-   execution alone is not a verified dataset. Read
+   execution alone is not a verified dataset. `receipt` carries one coverage entry per source --
+   `bytes_fetched`, `rows_kept`, `rows_available`, `truncated` and the window -- where the run
+   record folds all sources into four numbers. That is what each source ACTUALLY weighed and
+   carried, and it is the evidence the refresh decision belongs on rather than the ceiling you
+   typed in `limits.max_source_bytes` before anything was fetched. Carry the correction into
+   step 10. Read
    [verification](references/7-interrogate-ask-the-run-what-it-actually-delivered.md).
 9. Any truncated source means a preview, not the requested complete dataset. Inspect coverage
    even when the run succeeded. A null coverage window does not mean all history was acquired.
@@ -80,6 +103,18 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
    Record cadence only when authorized and `mr-data recipe readiness --json` reports the current
    revision refresh-ready. A blocked mutable snapshot requires an explicit table resync, not a
    scheduled whole-source fallback. Do not promise freshness without evidence.
+   A recipe that is not refresh-ready is a design fault to repair, not a state to wait out: read
+   `blocking_source_names` and every source's classification in the readiness answer. That list
+   is all-or-nothing: when one source states no continuation Studio cannot derive a plan at all,
+   so it classifies EVERY source `resync_only` and names every one of them, and a twenty-source
+   recipe with one bare address comes back with twenty names rather than the one to repair. Read
+   the document to find which of them actually declares nothing rather than revising nineteen
+   truthful windows. An EMPTY list beside `refresh_ready: false` is the other answer: every source
+   is declared and the plan's SHAPE or its predecessor state is what refuses. Revise against what
+   this run measured, register the revision and resync once. The
+   revision changes the recipe digest, so the sealed predecessor stops matching and one full
+   resync is the price of the repair -- which is why the decision belongs in step 5. A table left
+   not refresh-ready never refreshes; it does not become ready later.
    Download and verify artifacts when the user requested bytes.
 10. Report rows and coverage actually verified, checks, limitations and the stable dataset link.
     Read [delivery](references/9-present-only-what-survived-inspection-with-caveats.md) for details.

{mostlyright_data-0.24.0 → mostlyright_data-0.25.0}/skills/mr-data-build/references/5-draft-one-recipe-document-one-call.md

@@ -292,6 +292,94 @@ names.
 Leaving it out is a defect in the same way a one-line dataset description is: the source is then
 drawn with its endpoint alone, and a reader is left to guess from a query string.
 
+### Say how each source refreshes
+
+Studio reads one refresh strategy per source out of this document, and a source that states none
+is `resync_only`. One `resync_only` source makes the whole table resync-only: it never refreshes
+on a schedule, however truthful its siblings are.
+
+| what the source declares | classification | next action |
+| --- | --- | --- |
+| `source_class: "stream"` | `recorded` | `recorded` |
+| `closed: true` | `closed` | `reuse_predecessor` |
+| a `collection` | `collection` | `acquire_incremental` |
+| a `window` carrying a `request` | `window` | `acquire_incremental` |
+| none of those | `resync_only` | `resync_required` |
+
+The rows are in Studio's own precedence order -- stream, then `closed`, then `collection`, then
+`window` -- because a source may declare more than one of those members and the first match wins.
+A source carrying both `closed: true` and a request window is `closed`, and its window is never
+read. A recorded stream that also says `closed: true` is `recorded`, not `closed`.
+
+`closed: true` says the range at this address is over and a refresh may reuse the bytes the
+predecessor sealed rather than ask again. Use it for a range that has demonstrably ended, and
+understand what you are accepting: if the publisher later restates that period, the table keeps
+serving the old bytes, no run fails, and only an explicit resync repairs it. Never mark the
+current period closed -- that drops new data outright, which is a different and worse mistake.
+
+A `window` is the better answer wherever the address carries time, because it re-requests its
+recent range on every refresh and so picks up corrections inside `lookback_seconds` on its own.
+It needs `start_at`, `granularity`, `timezone`, `lookback_seconds`, `max_span_seconds` and
+`merge`, and a `request` naming BOTH ends -- an address that names only one side of a range
+cannot be narrowed. Each grammar below is closed; a spelling outside it belongs to a later
+version and is refused.
+
+- `request.start` and `request.end` take `encoding`: `date_parts` (names three parameters and
+  takes `pad`), `iso_date` or `epoch_seconds` (each names one parameter and takes no `pad`).
+- `bound` is `inclusive` or `exclusive`. The engine's own window is half-open
+  `[start_inclusive, end_exclusive)` in UTC; start/inclusive and end/exclusive pass both instants
+  through unchanged, and the other two spellings shift by a day. **Probe the publisher rather
+  than assuming**: both spellings validate against the schema, and only the start is checked at
+  registration, so a wrong `end` silently fetches a different range than the address states.
+- `merge.materialization` is `partition_replace` only.
+- `merge.partition.column` names a column of the ACQUIRED relation -- the header the publisher
+  sent, not the name your statement gives it. **That holds only while the window is the recipe's
+  one source.** In a plan of more than one source the column must ALSO be a name `table.columns`
+  declares, so a renamed field is refused at admission and has to take the one-source route. If
+  this recipe has, or will have, a second source of any kind -- a `closed` sibling is enough --
+  partition on a name the table itself declares.
+- `merge.partition.key` is `iso_date_prefix` (first ten characters of an ISO value),
+  `iso_date_value` (a complete ISO date) or `compact_utc_hour` (a real `YYYYMMDDHH`).
+- `merge.row_identity` names the columns the merged relation must be unique on.
+
+`start_at` is a UTC midnight, and the start parameters already written in the address must render
+exactly that instant. If the address says `year1=2026&month1=5&day1=31`, `start_at` is
+`2026-05-31T00:00:00Z`.
+
+Declaring nothing is the honest answer for a mutable address with no changed-since contract, and
+pagination is not one. Say which sources you left `resync_only` and why, because a recipe that
+never asked the question and one that asked and answered `resync_only` register identically.
+
+**Declaring every source is the FIRST gate, and the SHAPE of the plan is the second.** Once Studio
+has one action per source it asks whether a worker can materialize the plan those actions make,
+and admits two families only: every source a recorded stream, which appends sealed batches, or at
+least one `window` carrying a `request`, with `closed` siblings permitted beside it, which replaces
+the bounded partitions the window asked for. A closed-only plan refreshes nothing -- it restores
+the predecessor's bytes and leaves no partition to replace. Any `collection` in the plan refuses
+the whole table, alone or beside a window. A `window` beside a recorded stream refuses it too. Each
+of those is `RESYNC_REQUIRED` before acquisition: the table registers, builds and serves, and only
+an explicit resync ever moves it again. So `closed: true` alone does NOT make a table refresh, and
+marking every source closed to clear the first warning buys a table that is just as dead and warns
+about nothing.
+
+**The second family is NECESSARY and not sufficient, and the count is of sources.** A plan holding
+more than one source -- one `window` and one `closed` sibling is already two -- is materialized by
+the several-source lane, which writes ONE table manifest. Every window in such a plan must
+therefore agree with the others on `merge.materialization`, `merge.partition.column` and
+`merge.partition.key`, and each partition column must be a name `table.columns` declares.
+Disagreeing windows, and a partition column the table does not declare, are both
+`RESYNC_REQUIRED`. So adding a `closed` sibling beside a lone window is not free: it is what turns
+the declared-column requirement on.
+
+**Decide it from MEASUREMENT, not from the ceiling you typed.** `limits.max_source_bytes` is a
+bound written before anything was fetched, and it is wrong in both directions: one live publisher
+came in an order of magnitude under the ceiling declared for it, and another came in over. The
+run's own coverage is the other thing entirely -- `mr-data receipt RUN --json` carries one entry
+per source with `bytes_fetched`, `rows_kept`, `rows_available` and `truncated`, which is what that
+source actually weighed and how many rows it actually carried. Build once, read that, and correct
+the declarations against it before you record a cadence. A revision changes the recipe digest and
+costs one full resync, so the cheapest time to be right is here.
+
 ### Say what each column is, and what it shows
 
 **Every column carries a `description`, and every column that measures a physical quantity carries

{mostlyright_data-0.24.0 → mostlyright_data-0.25.0}/skills/mr-data-build/references/6-build-one-run-sized-to-acquire-every-measured-source-whole.md

@@ -183,11 +183,15 @@ settle it. `mr-data run --cancel RUN_ID` stops a run that is queued or running.
 
 Do not treat the accepted command vocabulary as execution evidence. Before proposing a schedule,
 classify every source under one truthful continuation strategy and record the supporting evidence.
-Current normal refresh materializes only an exact single direct `window.request` source with a
-compatible partitioned predecessor, or an all-recorded-stream recipe. The classifier may name
-closed reuse or collection continuation, but closed-only, collection-only, and mixed plans return
-`RESYNC_REQUIRED` before acquisition because their bounded table materializers do not exist yet. A
-`window.snapshot` or unwindowed mutable source is likewise resync-only. Explicit resync is full,
+Normal refresh materializes an all-recorded-stream recipe, or a plan holding at least one direct
+`window.request` source with a compatible partitioned predecessor, `closed` reuse permitted beside
+it. Beside it is not free: as soon as the plan holds more than one source, every window in it must
+agree on one `merge.materialization`, `merge.partition.column` and `merge.partition.key`, and that
+column must be a name the table declares. The classifier may name closed reuse or collection
+continuation, but closed-only plans, any plan carrying a collection, and a window beside a
+recorded stream return `RESYNC_REQUIRED` before acquisition because their bounded table
+materializers do not exist yet. A `window.snapshot` or unwindowed mutable source is likewise
+resync-only. Explicit resync is full,
 has no predecessor, and a collection starts from the beginning. Never hide one behind conditional
 revalidation, generic pagination, or a whole-source comparison. Research publisher cursors,
 revision identities, listings, corrections, and deletions, but use only deployed recipe grammar

{mostlyright_data-0.24.0 → mostlyright_data-0.25.0}/skills/mr-data-build/references/commands.md

@@ -14,7 +14,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --topics "a,b,c" --license ID --description-file F` writes the title, the descriptive tags, the SPDX licence and the description under the version it was read at, retrying once if somebody else wrote first, and an empty `--topics` or `--license` takes that value off the page; a saved write whose public sync fails exits 2 and reports `public_projection_synced: false` — use `dataset sync ID` to retry that sync without rewriting Studio; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset publish ID [--mode public|link|private]` says who can read the dataset — `public` lists it in the public directory and serves it at an address anybody can read, `link` serves it at an unlisted address, `private` takes it back to the workspace — and `dataset publish ID --show` reads that back without changing it; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. The document is read first and every fault comes back in one refusal, each with the JSON pointer that names it; `--no-lint` sends it as written instead. Registration refuses `THIN_BRIEF_MISSING` when the dataset's record carries no answered question and no delegation, and `THIN_RECIPE_DATASET_UNBOUND` when the document names no `dataset.id`; `--delegated "their words"` writes the delegation onto the dataset and registers in the same command, and clears the first of those two and never the second. `mr-data recipe show ID` reads one back. |
 | `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
-| `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The mode is one of six: `sample`, `full`, `refresh`, `backfill`, `compact` and `replay`. Four have a shorthand flag (`--sample`, `--full`, `--refresh`, `--backfill`) and two do not, so write `--mode MODE`, which is accepted for every one of them. A refresh executes only Studio's persisted strict source-action plan. Although the classifier may name closed reuse, a request-window or collection delta, or recorded input, current normal refresh materializes only an exact single direct partition request-window plan with a compatible predecessor, or an all-recorded-stream plan. Closed-only, collection, mixed, snapshot, and unwindowed plans return `RESYNC_REQUIRED` before acquisition. Use `mr-data table resync TABLE_ID --request-id UUID` for an explicit full source reread, retaining that UUID if the response is uncertain. `--backfill` states the exact window with `--window START END`. `--mode replay --sources-from RUN_ID` asks Studio to run the registered revision against the RETAINED RAW INPUTS of one named successful run of the same table: nothing is fetched from this computer, nothing is acquired again, the result is compared against that run and never becomes the live version, and it neither asks for nor records an approval. Where Studio has replay switched off, or the named run is not successful, not the same table, or no longer retains its inputs, it answers a typed refusal — report the code rather than retrying in another mode. `--sources-from` on any other mode is refused `THIN_ARGUMENT_INVALID` before anything is sent. `--max-rows` and `--max-source-bytes` bound ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles and whose printed `confirm_command` re-states every argument the request carried. `--approve-full RUN_ID` releases the full a sample-first pair is holding once its preview has succeeded, naming either half of the pair; where the deployment still wants a person at a browser it answers `THIN_INTERACTIVE_HUMAN_REQUIRED` and names the run page. `--retry RUN_ID` tries one failed run again when its triple carries `room_fault: true`, re-stating that run's own coordinate under a byte ceiling that cannot narrow it, and refuses with the triple when the recipe was at fault. `--cancel RUN` stops one. |
+| `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The mode is one of six: `sample`, `full`, `refresh`, `backfill`, `compact` and `replay`. Four have a shorthand flag (`--sample`, `--full`, `--refresh`, `--backfill`) and two do not, so write `--mode MODE`, which is accepted for every one of them. A refresh executes only Studio's persisted strict source-action plan. Although the classifier may name closed reuse, a request-window or collection delta, or recorded input, current normal refresh materializes only a plan carrying at least one direct `partition_replace` request window with a compatible predecessor — `closed` siblings may reuse their sealed bytes beside it, a lone window needs a retained source-history record, and several must agree on materialization and partition layout — or an all-recorded-stream plan. Closed-only, any plan carrying a collection, a window beside a recorded stream, snapshot and unwindowed plans return `RESYNC_REQUIRED` before acquisition. Use `mr-data table resync TABLE_ID --request-id UUID` for an explicit full source reread, retaining that UUID if the response is uncertain. `--backfill` states the exact window with `--window START END`. `--mode replay --sources-from RUN_ID` asks Studio to run the registered revision against the RETAINED RAW INPUTS of one named successful run of the same table: nothing is fetched from this computer, nothing is acquired again, the result is compared against that run and never becomes the live version, and it neither asks for nor records an approval. Where Studio has replay switched off, or the named run is not successful, not the same table, or no longer retains its inputs, it answers a typed refusal — report the code rather than retrying in another mode. `--sources-from` on any other mode is refused `THIN_ARGUMENT_INVALID` before anything is sent. `--max-rows` and `--max-source-bytes` bound ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles and whose printed `confirm_command` re-states every argument the request carried. `--approve-full RUN_ID` releases the full a sample-first pair is holding once its preview has succeeded, naming either half of the pair; where the deployment still wants a person at a browser it answers `THIN_INTERACTIVE_HUMAN_REQUIRED` and names the run page. `--retry RUN_ID` tries one failed run again when its triple carries `room_fault: true`, re-stating that run's own coordinate under a byte ceiling that cannot narrow it, and refuses with the triple when the recipe was at fault. `--cancel RUN` stops one. |
 | `mr-data status` | Report which of the seven states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed, where, and whether the fault was the execution room's rather than the recipe's. Exits non-zero on a failed run. |
 | `mr-data runs` | Report this workspace's own runs: identifier, mode, state and creation time, plus the failure triple of any that failed and whether that failure was the execution room's. `--status` and `--mode` narrow it and are refused `THIN_ARGUMENT_INVALID` for a word outside the seven states or the six modes; `--limit` says how many, and pages are followed to reach it. A narrow filter over a workspace of thousands of runs reads a long way to find its matches, and a listing that does not end says how many were read and suggests a smaller `--limit`. |
 | `mr-data watch` | Stream one run's live progress under the durable event type of each stage, resuming across stream cuts. Exits non-zero when the run failed. |
@@ -46,9 +46,12 @@ action, and blocking source names. It reads persisted facts; it does not authori
 
 A refresh executes only Studio's persisted source-action plan. The classifier may name closed
 reuse, a request window or collection delta, or recorded input, but current normal refresh
-materializes only an exact single direct partition request-window plan with a compatible
-predecessor, or an all-recorded-stream plan. Closed-only, collection, mixed, snapshot, and
-unwindowed plans return `RESYNC_REQUIRED` before acquisition. `mr-data table resync TABLE --request-id UUID` is the explicit
+materializes only a plan carrying at least one direct `partition_replace` request window with a
+compatible predecessor, or an all-recorded-stream plan. `closed` siblings may reuse their sealed
+bytes beside a window; a lone window needs a retained source-history record, and several windows
+must agree on materialization and partition layout. Closed-only, any plan carrying a collection, a
+window beside a recorded stream, snapshot and unwindowed plans return `RESYNC_REQUIRED` before
+acquisition. `mr-data table resync TABLE --request-id UUID` is the explicit
 full reread; retain and reuse the caller-generated UUID after a lost response. A held resync receipt
 prints `mr-data run --confirm-held RUN_ID` for that exact run. A sample-first resync instead uses
 `--approve-full` only after its preview succeeds.

{mostlyright_data-0.24.0 → mostlyright_data-0.25.0}/skills/mr-data-build/references/promote.md

@@ -47,13 +47,16 @@ the cadence itself, and it belongs to the user.
 This call records a schedule; it is not proof of data continuation. Schedule only a current recipe
 that `mr-data recipe readiness --json` reports as refresh-ready. Studio must have a persisted bounded
 action for every source. The classifier may name closed reuse, a request-window or collection
-delta, or recorded stream input, but current normal refresh executes only an exact single direct
-partition request-window plan with a compatible predecessor, or an all-recorded-stream plan.
-Closed-only, collection, mixed, snapshot, and unwindowed plans return `RESYNC_REQUIRED` before
-acquisition. Use `mr-data table resync TABLE_ID --request-id UUID` when a full reread is
-genuinely needed; it is an explicit run, not a schedule fallback. Only returned state and durable
-run evidence support a claim that the table is caught up or refreshing. If that evidence is absent,
-report that a schedule was recorded and that ongoing freshness is unavailable.
+delta, or recorded stream input, but normal refresh admits only an all-recorded-stream plan, or a
+plan holding at least one direct partition request window with a compatible predecessor, `closed`
+reuse permitted beside it. A plan of more than one source is admitted only when its windows agree
+on one `merge.materialization`, `merge.partition.column` and `merge.partition.key`, and that
+column is a name the table declares. Closed-only, collection, snapshot, and unwindowed plans, and
+a window beside a recorded stream, return `RESYNC_REQUIRED` before acquisition. Use `mr-data
+table resync TABLE_ID --request-id UUID` when a full reread is genuinely needed; it is an
+explicit run, not a schedule fallback. Only returned state and durable run evidence support a
+claim that the table is caught up or refreshing. If that evidence is absent, report that a
+schedule was recorded and that ongoing freshness is unavailable.
 
 `mr-data pin TABLE_ID --version VERSION_ID` freezes the pointer on an exact version for a rollback
 or a hold, `mr-data unpin TABLE_ID` resumes tracking, and `mr-data demote TABLE_ID` detaches the

{mostlyright_data-0.24.0 → mostlyright_data-0.25.0}/src/mostlyright/data_harness/thin/recipe.py

@@ -31,7 +31,7 @@ which is the point of having two.
 from __future__ import annotations
 
 import argparse
-from collections.abc import Mapping, Sequence
+from collections.abc import Collection, Mapping, Sequence
 from pathlib import Path
 from typing import Any
 from uuid import UUID
@@ -109,6 +109,19 @@ DRAINED_SOURCE_BYTE_CEILING = 16_777_216
 #: nothing is lost by naming a dozen and saying how many more there are.
 NAMED_UNDESCRIBED_COLUMNS = 12
 
+#: How many sources stating no bounded continuation the warning names before it counts the rest.
+#: The same dozen as the columns above and for the same reason, against a different ceiling: a
+#: recipe may declare many sources, and a sentence that spelt every undeclared one of them out
+#: would be the wall the `numbered` receipt member exists to avoid.
+NAMED_UNCONTINUED_SOURCES = 12
+
+#: The fact every plan-shape sentence ends on. The refusal is about the SCHEDULE and not about the
+#: registration, so each of them says what is still true -- the table registers, builds and serves
+#: -- and names the one operation that moves it afterwards.
+UNREFRESHABLE_PLAN_TAIL = (
+    "This table still registers and builds; it refreshes only by an explicit resync"
+)
+
 #: The word that selects the read-back form. A positional rather than a flag because
 #: `mr-data recipe show ID` is how a person says it out loud.
 SHOW = "show"
@@ -292,6 +305,265 @@ def source_byte_warnings(sources: Any) -> list[str]:
     return said
 
 
+def sources_without_continuation(sources: Any) -> list[str]:
+    """The sources a scheduled refresh cannot continue, named in document order.
+
+    The branch order is Studio's own, from ``refresh_plan.action_plan``: a stream is ``recorded``,
+    ``closed`` is ``reuse_predecessor``, a ``collection`` and a ``window`` carrying a ``request``
+    are ``acquire_incremental``, and everything else is ``resync_only``. A ``window`` with only a
+    ``snapshot`` is counted here deliberately: it states an address, not a bounded continuation.
+
+    One walk, because the receipt carries both the count and the sentence and the pair must not be
+    able to disagree -- the same rule :func:`columns_without_description` follows.
+    """
+
+    if not isinstance(sources, list):
+        return []
+    named: list[str] = []
+    for index, source in enumerate(sources):
+        if not isinstance(source, Mapping):
+            continue
+        if source.get("source_class") == "stream" or source.get("closed") is True:
+            continue
+        if isinstance(source.get("collection"), Mapping):
+            continue
+        window = source.get("window")
+        if isinstance(window, Mapping) and isinstance(window.get("request"), Mapping):
+            continue
+        name = source.get("name")
+        named.append(name if isinstance(name, str) and name else f"sources[{index}]")
+    return named
+
+
+def _declared_column_names(table: Any) -> set[str] | None:
+    """The names ``table.columns`` declares, or ``None`` when this document states none readably.
+
+    ⚠ AN EMPTY SET AND ``None`` ARE DIFFERENT ANSWERS, which is why this does not just return a
+    set. Studio builds the same set off the same member and refuses a partition column that is not
+    in it, so a table declaring no columns really does refuse every window. A table this client
+    could not read is a question it cannot answer, and it says so with ``None`` rather than with an
+    empty set that would refuse the same plans for a reason it never established.
+
+    ⚠ AND ``None`` IS NOT THE REGISTRATION PATH. :func:`check_document` has already refused a
+    document carrying no ``table`` by the time this is read, so the only documents that reach the
+    silent half are ones whose ``table`` is present and not an object, or whose ``columns`` is not
+    a list -- both refused by the schema. The `window` beside `closed` plan, which is the shape
+    this check exists for, always arrives with a real column list.
+    """
+
+    if not isinstance(table, Mapping):
+        return None
+    columns = table.get("columns")
+    if not isinstance(columns, list):
+        return None
+    return {
+        column["name"]
+        for column in columns
+        if isinstance(column, Mapping) and isinstance(column.get("name"), str)
+    }
+
+
+def _window_layout_refusal(
+    sources: Sequence[Any], *, declared_columns: Collection[str] | None
+) -> str | None:
+    """The sentence a several-source windowed plan is refused with, or ``None``.
+
+    Studio's ``_windowed_table_layout_is_consistent``, read the way it reads it: for every direct
+    window in the plan, ``merge.materialization``, ``merge.partition.column`` and
+    ``merge.partition.key`` must all be stated as strings, the column must be one the table itself
+    declares, and the three together must be identical across the windows. Closed siblings are
+    skipped because they are reuse rather than a window; collections and recorded streams never
+    reach this branch, which the caller has already refused them on.
+
+    ``declared_columns`` of ``None`` answers only the half that does not need it. The
+    agreement and completeness checks read nothing but the sources.
+    """
+
+    layouts: set[tuple[str, str, str]] = set()
+    for index, source in enumerate(sources):
+        if source.get("source_class") == "stream" or source.get("closed") is True:
+            continue
+        if isinstance(source.get("collection"), Mapping):
+            continue
+        stated = source.get("name")
+        named = repr(stated) if isinstance(stated, str) and stated else f"sources[{index}]"
+        window = source.get("window")
+        merge = window.get("merge") if isinstance(window, Mapping) else None
+        partition = merge.get("partition") if isinstance(merge, Mapping) else None
+        materialization = merge.get("materialization") if isinstance(merge, Mapping) else None
+        column = partition.get("column") if isinstance(partition, Mapping) else None
+        key = partition.get("key") if isinstance(partition, Mapping) else None
+        if (
+            not isinstance(materialization, str)
+            or not isinstance(column, str)
+            or not isinstance(key, str)
+        ):
+            return (
+                "Every source states a continuation, but this plan holds more than one source and "
+                f"the window on {named} states no complete `merge.partition`: a plan of several "
+                "sources is materialized by the generic delta worker, which writes ONE table "
+                "manifest, so Studio reads `merge.materialization`, `merge.partition.column` and "
+                "`merge.partition.key` off every window in the plan and refuses it when one of "
+                f"them is missing. {UNREFRESHABLE_PLAN_TAIL}"
+            )
+        if declared_columns is not None and column not in declared_columns:
+            return (
+                "Every source states a continuation, but this plan holds more than one source and "
+                f"the window on {named} partitions on {column!r}, which `table.columns` does not "
+                "declare: the generic delta worker carries ONE identifier across the source, the "
+                "transform and the table manifest, so a partition column the statement renames on "
+                "the way through is refused rather than localized. Partition on the name the "
+                "TABLE declares, or leave this window the only one in the plan, which is the "
+                f"separately bounded single-source route. {UNREFRESHABLE_PLAN_TAIL}"
+            )
+        layouts.add((materialization, column, key))
+    if len(layouts) > 1:
+        disagreement = ", ".join(sorted(f"`{column}`/`{key}`" for _, column, key in layouts))
+        return (
+            "Every source states a continuation, but this plan's windows disagree on the table "
+            f"partition layout: {disagreement}. Several windows share one refresh only when their "
+            "`merge.materialization`, `merge.partition.column` and `merge.partition.key` are "
+            "identical, because the generic delta worker writes ONE table manifest and cannot "
+            "replace two different partitionings in one pass. Give every window the same layout, "
+            f"or build them as separate tables. {UNREFRESHABLE_PLAN_TAIL}"
+        )
+    return None
+
+
+def unrefreshable_plan_shape(
+    sources: Any, *, declared_columns: Collection[str] | None = None
+) -> str | None:
+    """The one sentence saying a FULLY DECLARED recipe still cannot refresh, or ``None``.
+
+    ⚠ DECLARING EVERY SOURCE IS THE FIRST GATE, NOT THE ONLY ONE.
+    :func:`sources_without_continuation` answers whether Studio can derive one action per source.
+    ``assess_refresh_readiness`` then asks a second question of the plan those actions FORM, and
+    admits two families: every source a recorded stream, or at least one direct ``window`` with no
+    ``collection`` anywhere in the plan and no recorded stream beside it. A closed-only plan
+    restores the predecessor's bytes and leaves no bounded partition to replace; a collection
+    carries a ledger that cannot join a replacement-partition proof; a window beside a recorded
+    stream asks one worker to append and replace at once. Each is `409 RESYNC_REQUIRED`.
+
+    ⚠ AND THE SECOND FAMILY IS NECESSARY RATHER THAN SUFFICIENT. A plan of MORE THAN ONE source is
+    materialized by the generic delta worker, which writes one table manifest, so Studio holds
+    every direct window in such a plan to one layout: identical ``merge.materialization``,
+    ``merge.partition.column`` and ``merge.partition.key``, each stated, and the column among the
+    names ``table.columns`` declares. That is why ``declared_columns`` is taken here. Studio reads
+    it off the document being registered, and a caller that does not have it gets the half of the
+    answer that does not need it rather than a refusal invented from a member nobody read. The
+    count is of SOURCES and not of windows: one window beside one closed source is already a plan
+    of two, and it is held to this same gate.
+
+    ⚠ WITHOUT THIS, THE REMEDY FOR THE FIRST WARNING BECOMES THE NEXT SILENT FAILURE. ``closed:
+    true`` is the cheapest declaration that makes a source continuable, so an author who reads only
+    the first sentence marks twelve sources closed, is warned about NOTHING, and still holds a
+    table that never refreshes. `closed: true` alone does not make a table refresh.
+
+    The branch order is Studio's own twice over -- ``refresh_plan.action_plan`` and the readiness
+    inventory both read ``closed`` BEFORE ``window`` -- so a source carrying both is ``closed``
+    here exactly as it is there. That is the ordering that bites: a closed source beside a window
+    is reuse rather than a second window, and a recipe whose every source says ``closed: true``
+    is a closed-only plan however many windows it also wrote.
+
+    Answered only once the first gate passes. With an undeclared source in the document the plan
+    has no settled shape to judge, and two sentences about one repair read as two faults.
+    """
+
+    if not isinstance(sources, list) or not sources:
+        return None
+    if sources_without_continuation(sources):
+        return None
+    held: set[str] = set()
+    for source in sources:
+        if not isinstance(source, Mapping):
+            return None
+        if source.get("source_class") == "stream":
+            held.add("recorded")
+        elif source.get("closed") is True:
+            held.add("closed")
+        elif isinstance(source.get("collection"), Mapping):
+            held.add("collection")
+        else:
+            held.add("window")
+    if held == {"recorded"}:
+        return None
+    if "window" in held and "collection" not in held and "recorded" not in held:
+        # One source is the single-window lane, which continues from its own retained source
+        # history and is never held to this. Two or more is the generic delta worker's one
+        # manifest, and that is where the windows have to agree with each other and with the table.
+        if len(sources) == 1:
+            return None
+        return _window_layout_refusal(sources, declared_columns=declared_columns)
+    if "collection" in held:
+        return (
+            "Every source states a continuation, but this plan carries a page collection and "
+            "Studio admits no bounded incremental materialization for one: a collection restores "
+            "its ledger from the raw snapshot and cannot join the replacement-partition proof a "
+            "refresh writes, so ANY collection in the plan refuses the whole table. "
+            f"{UNREFRESHABLE_PLAN_TAIL}"
+        )
+    if "window" in held:
+        return (
+            "Every source states a continuation, but this plan puts a `window` beside a recorded "
+            "stream, and no worker appends recorded batches and replaces bounded partitions in "
+            f"one pass: the two continuations cannot share a refresh. {UNREFRESHABLE_PLAN_TAIL}"
+        )
+    if "recorded" in held:
+        return (
+            "Every source states a continuation, but this plan is `closed: true` beside a "
+            "recorded stream, and only an ALL-recorded plan appends: a closed source restores "
+            "bytes and leaves the refresh no bounded partition to replace. "
+            f"{UNREFRESHABLE_PLAN_TAIL}"
+        )
+    return (
+        "Every source states a continuation, but every one of them is `closed: true`, and a "
+        "closed-only plan refreshes nothing: it restores the predecessor's bytes and has no "
+        "bounded partition to replace, so Studio refuses it before it can spend a slot. If these "
+        "sources really are finished, an explicit resync is the operation that moves this table "
+        "and there is nothing here to repair. Where one of them is not, a `window` carrying both "
+        "`request` endpoints is the declaration that makes a table refresh on a schedule -- but "
+        "write one only where the publisher honours it, because a window a publisher does not "
+        f"honour serves the wrong range and nothing says so. {UNREFRESHABLE_PLAN_TAIL}"
+    )
+
+
+def continuation_warnings(absent: list[str], total: int) -> list[str]:
+    """One sentence naming the sources a scheduled refresh will refuse to continue.
+
+    ⚠ A WARNING RATHER THAN A FINDING, because ``resync_only`` is a TRUTHFUL answer. A mutable
+    address with no changed-since contract has no bounded continuation, and saying so is what the
+    document is for; refusing to register it would push an author toward declaring a window the
+    publisher does not honour, which is the one error that silently serves wrong data.
+
+    ⚠ AND IT IS ABOUT THE WHOLE TABLE, NOT THE SOURCE. Studio derives one action per source, and a
+    plan holding a single undeclared source refuses the ENTIRE table's refresh -- a twenty-source
+    recipe with nineteen truthful windows and one bare address refreshes nothing, ever. An author
+    who never learns that ships a table which looks finished and never moves again.
+    """
+
+    if not absent:
+        return []
+    named = ", ".join(repr(name) for name in absent[:NAMED_UNCONTINUED_SOURCES])
+    beyond = len(absent) - NAMED_UNCONTINUED_SOURCES
+    if beyond > 0:
+        named += f", and {beyond} more"
+    return [
+        f"{len(absent)} of {total} sources state no bounded continuation: {named}. Studio "
+        "classifies these as `resync_only` -- and marks every other source `resync_only` too, so "
+        "`mr-data recipe readiness` names them all. ONE of them refuses this whole table's "
+        "scheduled refresh -- the other sources' windows do not run without it. Declaring every "
+        "source is only the FIRST gate; the plan's SHAPE is the second, and Studio admits just "
+        "two families: every source a recorded stream, or at least one `window` carrying both "
+        "`request` endpoints with no collection and no recorded stream beside it. The second is "
+        "NECESSARY and not sufficient: once the plan holds more than one source, every window in "
+        "it must also partition on a column `table.columns` declares, and on the SAME one. So "
+        "`closed: true` alone does NOT make a table refresh. Where the source is genuinely "
+        "mutable with no changed-since contract, leaving it undeclared is the honest answer and "
+        "this table then refreshes only by an explicit resync; say which sources those are and "
+        "why"
+    ]
+
+
 def _source_byte_ceiling(source: Mapping[str, Any]) -> tuple[int, str]:
     """The fetch ceiling one declared source is held to, and the phrase that names it.
 
@@ -509,7 +781,20 @@ def register(
     # written, so they do not depend on the answer; putting them on the receipt rather than in
     # front of the registration is what makes them a warning rather than a gate.
     undescribed = columns_without_description(document.get("table"))
-    warnings = source_byte_warnings(document.get("sources"))
+    uncontinued = sources_without_continuation(document.get("sources"))
+    declared_sources = document.get("sources")
+    warnings = source_byte_warnings(declared_sources)
+    warnings += continuation_warnings(
+        uncontinued, len(declared_sources) if isinstance(declared_sources, list) else 0
+    )
+    # THE SECOND GATE, in the same place as the first. A recipe that passed the first one can
+    # still hold a plan Studio refuses outright, and an author who repaired the first warning by
+    # declaring `closed` everywhere would otherwise be answered with silence.
+    unrefreshable = unrefreshable_plan_shape(
+        declared_sources, declared_columns=_declared_column_names(document.get("table"))
+    )
+    if unrefreshable is not None:
+        warnings.append(unrefreshable)
     warnings += column_description_warnings(undescribed)
     # ⚠ EVERYTHING THIS COMMAND CAN SETTLE ON ITS OWN IS SETTLED BEFORE A CREDENTIAL IS RESOLVED,
     # which is the rule every other lane here follows: a document naming no dataset, and a
@@ -573,6 +858,9 @@ def register(
         # count out of a paragraph and should not have to parse one to learn how bad it is.
         # Omitted at zero, which is this receipt's own rule for `warnings`: absent means none.
         **({"columns_without_description": len(undescribed)} if undescribed else {}),
+        # THE SAME FACT AS ITS WARNING, AS A NUMBER, for whatever reads this receipt as JSON.
+        # Omitted at zero, which is this receipt's own rule for `warnings`.
+        **({"sources_without_continuation": len(uncontinued)} if uncontinued else {}),
         # OMITTED WHEN THERE IS NOTHING TO SAY, rather than an empty list on every clean
         # registration. `Warnings: (none)` under every successful recipe is the kind of line a
         # reader learns to skip, and a reader who skips it will skip the one that matters.
@@ -819,6 +1107,7 @@ __all__ = [
     "MAX_DOCUMENT_BYTES",
     "MAX_READINESS_ITEMS",
     "MAX_READINESS_PAGES",
+    "NAMED_UNCONTINUED_SOURCES",
     "NAMED_UNDESCRIBED_COLUMNS",
     "NDJSON_SOURCE_BYTE_CEILING",
     "READER_SOURCE_BYTE_CEILING",
@@ -829,11 +1118,13 @@ __all__ = [
     "REQUIRED_DOCUMENT_MEMBERS",
     "SHOW",
     "SHOW_SCHEMA",
+    "UNREFRESHABLE_PLAN_TAIL",
     "bound_dataset",
     "check_dataset_binding",
     "check_document",
     "column_description_warnings",
     "columns_without_description",
+    "continuation_warnings",
     "declare_arguments",
     "read_document",
     "readiness",
@@ -842,4 +1133,6 @@ __all__ = [
     "register",
     "show",
     "source_byte_warnings",
+    "sources_without_continuation",
+    "unrefreshable_plan_shape",
 ]