mostlyright-data 0.23.0__tar.gz → 0.24.0__tar.gz

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.23.0
+Version: 0.24.0
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
@@ -89,6 +89,17 @@ can reissue supported platform execution failures without changing the recipe.
 
 Use the identifiers returned by registration and run submission. `mr-data dataset create`
 creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
+A scheduled refresh runs only Studio's persisted bounded source-action plan; it never silently
+falls back to a mutable whole-source fetch. `mr-data table resync TABLE_ID --request-id UUID`
+explicitly requests a full source reread when that is needed. Choose and retain the UUID before
+submitting: if the response is lost, repeat that exact command with the same UUID rather than
+starting another reread. A persisted spend hold is still an accepted resync: its receipt names
+the run, projection and exact `mr-data run --confirm-held RUN_ID` action. If it is a sample-first
+preview, confirm that preview first, then release its linked full
+only after the preview succeeds. `mr-data recipe readiness` reads Studio's paginated,
+immutable-revision inventory before a production-wide schedule sweep: predecessor,
+incremental-materialization and differential-proof readiness, together with each source's next
+action. It reports Studio's persisted facts; it does not authorize a schedule.
 
 To request an offline replay of retained inputs with a registered revision:
 
@@ -100,16 +111,20 @@ Studio must have replay enabled and the named successful run must belong to the
 its raw inputs still retained. Replay compares against that run and never becomes the live version.
 Studio returns a typed refusal when replay is unavailable; the CLI does not fetch sources locally.
 
-The hosted engine executes sample, full and refresh runs. A refresh uses each source's declared
-update behavior: bounded source windows merge with retained history; snapshot sources are
-revalidated or acquired again. The table's transformations and checks run over the resulting
-source relations. URL windows may use declared query parameters or path placeholders; a fixed
-date URL does not advance automatically. Configure a correction lookback where the publisher
-can revise earlier observations.
-
-A source exposing only its current snapshot cannot supply a historical delta. It remains a
-supported source, with snapshot comparison and recomputation rather than a claim that only new
-rows were fetched. Recorded streams use their registered capture and continuation semantics.
+The hosted engine executes sample, full and refresh runs. Studio's action classifier can name
+closed-source reuse, request-window or collection acquisition, and recorded stream input, but the
+current worker has only two executable normal-refresh materializers: an exact single direct
+partition request-window plan with a compatible predecessor, and an all-recorded-stream plan.
+Closed-only, collection, and mixed plans return `RESYNC_REQUIRED` before acquisition, as does a
+mutable snapshot or other unwindowed source. Nothing is revalidated or re-acquired whole as an
+implicit fallback. Use `mr-data table resync TABLE_ID --request-id UUID` when a deliberate full
+source reread is intended, retaining that UUID through an uncertain response. URL windows may use
+declared query parameters or path placeholders; a fixed date URL does not advance automatically.
+Configure a correction lookback where the publisher can revise earlier observations.
+
+A source exposing only its current snapshot cannot supply a historical delta. It is supported only
+through an explicit table resync; refresh does not turn its whole current result into an
+incremental update. Recorded streams use their registered capture and continuation semantics.
 
 See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
 A table's first succeeded run goes live on its own, whatever mode it was; `mr-data promote
@@ -135,7 +150,8 @@ worker executables or image publisher.
 - [Recipe examples](https://mostlyright.md/docs/recipes/)
 - [Join market settlements](https://mostlyright.md/docs/recipes/market-settlement-join/)
 - [Compare a forecast with observations](https://mostlyright.md/docs/recipes/forecast-vs-observation/)
-- [Replace a snapshot on refresh](https://mostlyright.md/docs/recipes/snapshot-window/)
+- [Model a snapshot source](https://mostlyright.md/docs/recipes/snapshot-window/) — normal refresh
+  is unavailable until its bounded materializer ships; use explicit table resync today
 - [Union many weather stations](https://mostlyright.md/docs/recipes/many-station-weather/)
 - [Aggregate a stream into bars](https://mostlyright.md/docs/recipes/stream-to-bars/)
 - [Use a public dataset](https://mostlyright.md/docs/guides/use-public-datasets/)

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/README.md

@@ -77,6 +77,17 @@ can reissue supported platform execution failures without changing the recipe.
 
 Use the identifiers returned by registration and run submission. `mr-data dataset create`
 creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
+A scheduled refresh runs only Studio's persisted bounded source-action plan; it never silently
+falls back to a mutable whole-source fetch. `mr-data table resync TABLE_ID --request-id UUID`
+explicitly requests a full source reread when that is needed. Choose and retain the UUID before
+submitting: if the response is lost, repeat that exact command with the same UUID rather than
+starting another reread. A persisted spend hold is still an accepted resync: its receipt names
+the run, projection and exact `mr-data run --confirm-held RUN_ID` action. If it is a sample-first
+preview, confirm that preview first, then release its linked full
+only after the preview succeeds. `mr-data recipe readiness` reads Studio's paginated,
+immutable-revision inventory before a production-wide schedule sweep: predecessor,
+incremental-materialization and differential-proof readiness, together with each source's next
+action. It reports Studio's persisted facts; it does not authorize a schedule.
 
 To request an offline replay of retained inputs with a registered revision:
 
@@ -88,16 +99,20 @@ Studio must have replay enabled and the named successful run must belong to the
 its raw inputs still retained. Replay compares against that run and never becomes the live version.
 Studio returns a typed refusal when replay is unavailable; the CLI does not fetch sources locally.
 
-The hosted engine executes sample, full and refresh runs. A refresh uses each source's declared
-update behavior: bounded source windows merge with retained history; snapshot sources are
-revalidated or acquired again. The table's transformations and checks run over the resulting
-source relations. URL windows may use declared query parameters or path placeholders; a fixed
-date URL does not advance automatically. Configure a correction lookback where the publisher
-can revise earlier observations.
-
-A source exposing only its current snapshot cannot supply a historical delta. It remains a
-supported source, with snapshot comparison and recomputation rather than a claim that only new
-rows were fetched. Recorded streams use their registered capture and continuation semantics.
+The hosted engine executes sample, full and refresh runs. Studio's action classifier can name
+closed-source reuse, request-window or collection acquisition, and recorded stream input, but the
+current worker has only two executable normal-refresh materializers: an exact single direct
+partition request-window plan with a compatible predecessor, and an all-recorded-stream plan.
+Closed-only, collection, and mixed plans return `RESYNC_REQUIRED` before acquisition, as does a
+mutable snapshot or other unwindowed source. Nothing is revalidated or re-acquired whole as an
+implicit fallback. Use `mr-data table resync TABLE_ID --request-id UUID` when a deliberate full
+source reread is intended, retaining that UUID through an uncertain response. URL windows may use
+declared query parameters or path placeholders; a fixed date URL does not advance automatically.
+Configure a correction lookback where the publisher can revise earlier observations.
+
+A source exposing only its current snapshot cannot supply a historical delta. It is supported only
+through an explicit table resync; refresh does not turn its whole current result into an
+incremental update. Recorded streams use their registered capture and continuation semantics.
 
 See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
 A table's first succeeded run goes live on its own, whatever mode it was; `mr-data promote
@@ -123,7 +138,8 @@ worker executables or image publisher.
 - [Recipe examples](https://mostlyright.md/docs/recipes/)
 - [Join market settlements](https://mostlyright.md/docs/recipes/market-settlement-join/)
 - [Compare a forecast with observations](https://mostlyright.md/docs/recipes/forecast-vs-observation/)
-- [Replace a snapshot on refresh](https://mostlyright.md/docs/recipes/snapshot-window/)
+- [Model a snapshot source](https://mostlyright.md/docs/recipes/snapshot-window/) — normal refresh
+  is unavailable until its bounded materializer ships; use explicit table resync today
 - [Union many weather stations](https://mostlyright.md/docs/recipes/many-station-weather/)
 - [Aggregate a stream into bars](https://mostlyright.md/docs/recipes/stream-to-bars/)
 - [Use a public dataset](https://mostlyright.md/docs/guides/use-public-datasets/)

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.23.0"
+version = "0.24.0"
 description = "Mostly Right hosted CLI for reviewed datasets"
 readme = "README.md"
 requires-python = ">=3.11"

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/skills/mr-data-build/SKILL.md

@@ -77,7 +77,9 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
 9. Any truncated source means a preview, not the requested complete dataset. Inspect coverage
    even when the run succeeded. A null coverage window does not mean all history was acquired.
    Repair only evidenced faults on the same dataset/table and verify the revision again.
-   Record cadence only when authorized; do not promise freshness without evidence.
+   Record cadence only when authorized and `mr-data recipe readiness --json` reports the current
+   revision refresh-ready. A blocked mutable snapshot requires an explicit table resync, not a
+   scheduled whole-source fallback. Do not promise freshness without evidence.
    Download and verify artifacts when the user requested bytes.
 10. Report rows and coverage actually verified, checks, limitations and the stable dataset link.
     Read [delivery](references/9-present-only-what-survived-inspection-with-caveats.md) for details.

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/skills/mr-data-build/references/6-build-one-run-sized-to-acquire-every-measured-source-whole.md

@@ -181,11 +181,17 @@ the record — `mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --c
 held answer prints back as `confirm_command`. Any authenticated surface of the paying workspace may
 settle it. `mr-data run --cancel RUN_ID` stops a run that is queued or running.
 
-Do not treat the accepted command vocabulary as execution evidence. Author a source window only
-when the publisher has a truthful bounded date endpoint: use query names by default or exact
-`{parameter}` markers in its path, retain a correction lookback, and keep historical sources
-revalidatable unless an explicit closure contract says they cannot change. A refresh merges each
-windowed source slice into retained source state and recomputes the complete transform; unrelated
-snapshot sources are conditionally revalidated or re-acquired as needed. Cursor, pagination and
-new-file discovery need their own supported source contract and must not be represented as a date
-window. A `backfill` may be refused before acquisition; the refusal says so.
+Do not treat the accepted command vocabulary as execution evidence. Before proposing a schedule,
+classify every source under one truthful continuation strategy and record the supporting evidence.
+Current normal refresh materializes only an exact single direct `window.request` source with a
+compatible partitioned predecessor, or an all-recorded-stream recipe. The classifier may name
+closed reuse or collection continuation, but closed-only, collection-only, and mixed plans return
+`RESYNC_REQUIRED` before acquisition because their bounded table materializers do not exist yet. A
+`window.snapshot` or unwindowed mutable source is likewise resync-only. Explicit resync is full,
+has no predecessor, and a collection starts from the beginning. Never hide one behind conditional
+revalidation, generic pagination, or a whole-source comparison. Research publisher cursors,
+revision identities, listings, corrections, and deletions, but use only deployed recipe grammar
+that represents the proven strategy. For each incremental source, record the affected key or
+partition and a fixture proving
+that applying its delta to the predecessor produces the same rows as a full build over
+predecessor-plus-delta. A `backfill` may be refused before acquisition; the refusal says so.

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/skills/mr-data-build/references/commands.md

@@ -14,7 +14,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --topics "a,b,c" --license ID --description-file F` writes the title, the descriptive tags, the SPDX licence and the description under the version it was read at, retrying once if somebody else wrote first, and an empty `--topics` or `--license` takes that value off the page; a saved write whose public sync fails exits 2 and reports `public_projection_synced: false` — use `dataset sync ID` to retry that sync without rewriting Studio; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset publish ID [--mode public|link|private]` says who can read the dataset — `public` lists it in the public directory and serves it at an address anybody can read, `link` serves it at an unlisted address, `private` takes it back to the workspace — and `dataset publish ID --show` reads that back without changing it; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. The document is read first and every fault comes back in one refusal, each with the JSON pointer that names it; `--no-lint` sends it as written instead. Registration refuses `THIN_BRIEF_MISSING` when the dataset's record carries no answered question and no delegation, and `THIN_RECIPE_DATASET_UNBOUND` when the document names no `dataset.id`; `--delegated "their words"` writes the delegation onto the dataset and registers in the same command, and clears the first of those two and never the second. `mr-data recipe show ID` reads one back. |
 | `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
-| `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The mode is one of six: `sample`, `full`, `refresh`, `backfill`, `compact` and `replay`. Four have a shorthand flag (`--sample`, `--full`, `--refresh`, `--backfill`) and two do not, so write `--mode MODE`, which is accepted for every one of them. A refresh narrows each source that declares a supported address window and merges its returned relation into retained source state; unwindowed sources are conditionally revalidated or re-acquired whole, then the full declared transform runs. `--backfill` states the exact window with `--window START END`. `--mode replay --sources-from RUN_ID` asks Studio to run the registered revision against the RETAINED RAW INPUTS of one named successful run of the same table: nothing is fetched from this computer, nothing is acquired again, the result is compared against that run and never becomes the live version, and it neither asks for nor records an approval. Where Studio has replay switched off, or the named run is not successful, not the same table, or no longer retains its inputs, it answers a typed refusal — report the code rather than retrying in another mode. `--sources-from` on any other mode is refused `THIN_ARGUMENT_INVALID` before anything is sent. `--max-rows` and `--max-source-bytes` bound ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles and whose printed `confirm_command` re-states every argument the request carried. `--approve-full RUN_ID` releases the full a sample-first pair is holding once its preview has succeeded, naming either half of the pair; where the deployment still wants a person at a browser it answers `THIN_INTERACTIVE_HUMAN_REQUIRED` and names the run page. `--retry RUN_ID` tries one failed run again when its triple carries `room_fault: true`, re-stating that run's own coordinate under a byte ceiling that cannot narrow it, and refuses with the triple when the recipe was at fault. `--cancel RUN` stops one. |
+| `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The mode is one of six: `sample`, `full`, `refresh`, `backfill`, `compact` and `replay`. Four have a shorthand flag (`--sample`, `--full`, `--refresh`, `--backfill`) and two do not, so write `--mode MODE`, which is accepted for every one of them. A refresh executes only Studio's persisted strict source-action plan. Although the classifier may name closed reuse, a request-window or collection delta, or recorded input, current normal refresh materializes only an exact single direct partition request-window plan with a compatible predecessor, or an all-recorded-stream plan. Closed-only, collection, mixed, snapshot, and unwindowed plans return `RESYNC_REQUIRED` before acquisition. Use `mr-data table resync TABLE_ID --request-id UUID` for an explicit full source reread, retaining that UUID if the response is uncertain. `--backfill` states the exact window with `--window START END`. `--mode replay --sources-from RUN_ID` asks Studio to run the registered revision against the RETAINED RAW INPUTS of one named successful run of the same table: nothing is fetched from this computer, nothing is acquired again, the result is compared against that run and never becomes the live version, and it neither asks for nor records an approval. Where Studio has replay switched off, or the named run is not successful, not the same table, or no longer retains its inputs, it answers a typed refusal — report the code rather than retrying in another mode. `--sources-from` on any other mode is refused `THIN_ARGUMENT_INVALID` before anything is sent. `--max-rows` and `--max-source-bytes` bound ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles and whose printed `confirm_command` re-states every argument the request carried. `--approve-full RUN_ID` releases the full a sample-first pair is holding once its preview has succeeded, naming either half of the pair; where the deployment still wants a person at a browser it answers `THIN_INTERACTIVE_HUMAN_REQUIRED` and names the run page. `--retry RUN_ID` tries one failed run again when its triple carries `room_fault: true`, re-stating that run's own coordinate under a byte ceiling that cannot narrow it, and refuses with the triple when the recipe was at fault. `--cancel RUN` stops one. |
 | `mr-data status` | Report which of the seven states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed, where, and whether the fault was the execution room's rather than the recipe's. Exits non-zero on a failed run. |
 | `mr-data runs` | Report this workspace's own runs: identifier, mode, state and creation time, plus the failure triple of any that failed and whether that failure was the execution room's. `--status` and `--mode` narrow it and are refused `THIN_ARGUMENT_INVALID` for a word outside the seven states or the six modes; `--limit` says how many, and pages are followed to reach it. A narrow filter over a workspace of thousands of runs reads a long way to find its matches, and a listing that does not end says how many were read and suggests a smaller `--limit`. |
 | `mr-data watch` | Stream one run's live progress under the durable event type of each stage, resuming across stream cuts. Exits non-zero when the run failed. |
@@ -29,7 +29,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data diff` | Compare two runs and say what changed. |
 | `mr-data connections` | List saved workspace connections and their current recipe coordinates; `--dataset ID` filters to connections granted to that dataset. No credential values are returned. |
 | `mr-data keys` | Enrol a source credential by name, list the names with their created and rotated times, and delete one. `keys set` reads the value from a file or from standard input and never from the command line; no command prints a value back. |
-| `mr-data stream` | Record a public `wss://` venue so a build can read it. `stream document register --file F` admits the connector document, `stream registry create --digest D` admits a set of them, `stream probe` listens briefly and seals nothing, `stream capture --seconds N` records a bounded window, `stream subscribe` keeps recording, `stream subscription {show|pause|resume|stop} ID` moves one, `stream status ID` reads a capture or a probe, and `stream recordings --document-digest D` names what a build could read and until when. `--seconds` is at most 600. `mr-data run` takes no duration: a build reads sealed batches and is bounded by them. |
+| `mr-data stream` | Record a public `wss://` venue so a build can read it. `stream document register --file F` admits the connector document, `stream registry create --digest D` admits a set of them, `stream probe` listens briefly and seals nothing, `stream capture --seconds N` records a bounded window, and `stream subscribe` keeps recording. `stream subscription {show|pause|resume|stop} ID` retains the legacy full-record interface. `stream subscription quiescence ID` reads the bounded server-computed proof; `pause-cas` and `resume-cas` read its ETag and make one conditional bounded write. An ambiguous CAS failure is not retried until a fresh invocation observes the resource again. `stream status ID` reads a capture or probe, and `stream recordings --document-digest D` names what a build could read and until when. `--seconds` is at most 600. `mr-data run` takes no duration: a build reads sealed batches and is bounded by them. |
 | `mr-data open` | Mint a single-use, short-lived address that continues this session in a browser at one dashboard path, and print it. `--open` hands it to the platform opener. The credential decides who mints: a device credential — what `mr-data login` stores — mints at Cloud, and a personal access token mints at Studio. |
 | `mr-data promote` | Record the cadence one table refreshes on and the reasoning behind it. A table's first succeeded run goes live on its own, so this is not what makes it readable; it is idempotent on a table that is already live, and it is how a withdrawn table is put back. The Harness worker does not perform the catch-up or scheduled refresh; report those only when another deployed component returns durable evidence that it did. |
 | `mr-data reschedule` | Change how often one live table refreshes, without the demote and promote that would reset its bookkeeping and buy a catch-up run. `--cadence` takes a cron expression or an interval such as `every 6h`; `--why` records the reasoning. It starts no run. `--lock` freezes the schedule so Studio stops adjusting it; `--unlock` lets it follow the source again. |
@@ -38,6 +38,21 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data unpin` | Ask Studio to resume pointer tracking; read the returned state before claiming it resumed. |
 | `mr-data demote` | Ask Studio to withdraw the pointer and any schedule it owns, for one table or for as many as you name: `demote TABLE [TABLE ...]` withdraws them one after another, attempts every one of them whatever the one before it answered, prints a line for each, and exits non-zero if any is still live. Pair it with `table archive` when you are retiring a set: a live table cannot be archived, so it is withdraw-then-archive, two commands rather than a loop. |
 
+### Strict refresh controls
+
+`mr-data recipe readiness` pages Studio's immutable-revision inventory and reports predecessor,
+incremental-materialization, and differential-proof readiness, every source's classified next
+action, and blocking source names. It reads persisted facts; it does not authorize a schedule.
+
+A refresh executes only Studio's persisted source-action plan. The classifier may name closed
+reuse, a request window or collection delta, or recorded input, but current normal refresh
+materializes only an exact single direct partition request-window plan with a compatible
+predecessor, or an all-recorded-stream plan. Closed-only, collection, mixed, snapshot, and
+unwindowed plans return `RESYNC_REQUIRED` before acquisition. `mr-data table resync TABLE --request-id UUID` is the explicit
+full reread; retain and reuse the caller-generated UUID after a lost response. A held resync receipt
+prints `mr-data run --confirm-held RUN_ID` for that exact run. A sample-first resync instead uses
+`--approve-full` only after its preview succeeds.
+
 `auth`, `login`, `whoami` and `clarify` are the same implementation in both profiles, and
 `clarify` alone reaches nothing at all. The other twenty-eight answer from Studio. Twelve only
 read: `status`, `runs`, `watch`, `peek`, `receipt`, `checks`, `download`, `verify`, `parts`,

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/skills/mr-data-build/references/live-run.md

@@ -57,6 +57,14 @@ full of a sample-first pair: its bounded preview must seal a downloadable table
 though it could. Keep the raw state internal: say what product decision is needed and give the
 person the single action that lets the build continue.
 
+**`build_scope` says what the build covers, not what it is doing.** A `failed` or `cancelled` run
+covers nothing and says so. A `succeeded` one reports what `coverage.truncated` recorded. Anything
+else states the bound it ran under — a `sample` its ceilings, a `full` the whole dataset, and a
+`full` at `awaiting_sample_approval` the whole dataset plus the preview it waits on. The mode
+alone never establishes a truncation, which is the rule reference 6 states: a sample sized above
+every measured source cut nothing and is the build. It is absent on `refresh`, `backfill`,
+`compact` and `replay`, and it is never evidence that a run is executing; `status` is.
+
 **A failed run always carries the same three members** — `failure_code`, `failure_detail` and
 `failed_stage` — on three surfaces: the run record, the terminal event on the stream, and every
 refusal a command raises about it. `mr-data status` and `mr-data runs` add a fourth reading beside

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/skills/mr-data-build/references/promote.md

@@ -44,13 +44,16 @@ Studio stops adjusting it, which is how a person overrides the evidence; `--unlo
 the source again. Do not lock a schedule on your own judgement — it is the same kind of decision as
 the cadence itself, and it belongs to the user.
 
-This call records a schedule; it is not proof of data continuation. The current Harness worker
-can execute a supported refresh, including declared source windows, but it does not make a frozen
-date, a generic cursor or a file listing advance by inference. It also does not attach or execute a
-warehouse schedule. Another deployed Studio or warehouse component may do that work, but only its
-returned state and durable run evidence support a claim that the table is live, caught up, or
-refreshing. If that evidence is absent, report that a schedule was recorded and that ongoing
-freshness is unavailable; do not imply that recording it filled the gap.
+This call records a schedule; it is not proof of data continuation. Schedule only a current recipe
+that `mr-data recipe readiness --json` reports as refresh-ready. Studio must have a persisted bounded
+action for every source. The classifier may name closed reuse, a request-window or collection
+delta, or recorded stream input, but current normal refresh executes only an exact single direct
+partition request-window plan with a compatible predecessor, or an all-recorded-stream plan.
+Closed-only, collection, mixed, snapshot, and unwindowed plans return `RESYNC_REQUIRED` before
+acquisition. Use `mr-data table resync TABLE_ID --request-id UUID` when a full reread is
+genuinely needed; it is an explicit run, not a schedule fallback. Only returned state and durable
+run evidence support a claim that the table is caught up or refreshing. If that evidence is absent,
+report that a schedule was recorded and that ongoing freshness is unavailable.
 
 `mr-data pin TABLE_ID --version VERSION_ID` freezes the pointer on an exact version for a rollback
 or a hold, `mr-data unpin TABLE_ID` resumes tracking, and `mr-data demote TABLE_ID` detaches the

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/skills/mr-data-build/references/reference-pages.md

@@ -24,8 +24,9 @@ https://mostlyright.md/docs/recipes/ lists all thirteen: `city-temperatures`, `c
 CSV file, one table), `json-api` (records pointer and pagination), `html-collection` (many HTML
 pages, one table), `document-extraction` (PDF projection), `weather-grib` (GRIB2 and the scientific
 Readers), `websocket-stream` (a `wss://` venue), `multi-source-join` (two sources joined, with
-checks), `market-settlement-join`, `forecast-vs-observation`, `snapshot-window` (a listing replaced
-on refresh), `many-station-weather` (twenty station feeds unioned) and `stream-to-bars`.
+checks), `market-settlement-join`, `forecast-vs-observation`, `snapshot-window` (a point-in-time
+source whose normal refresh currently requires explicit resync), `many-station-weather` (twenty
+station feeds unioned) and `stream-to-bars`.
 
 The build guides run one per stage, at https://mostlyright.md/docs/build/NAME/ where NAME is
 `probe-sources`, `write-a-recipe`, `run-and-inspect`, `publish-and-refresh`, `credentials`,

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/skills/mr-data-build/references/sources.md

@@ -20,15 +20,23 @@
 - Preserve event time, available time, acquisition time, source revision, and timezone. Do not use
   post-cutoff information in targets, labels, features, joins, or validation decisions.
 - When the question is **what changed over time**, or the table needs point-in-time features, and
-  the source answers only with the present, declare a `snapshot` window on that source: Studio's
-  worker stamps every row with the UTC day it acquired the source on and keeps one partition per
-  day. Compute a transition with
+  the source answers only with the present, a `snapshot` window expresses the intended future
+  daily partition model. It is not executable as normal refresh in this release: Studio returns
+  `RESYNC_REQUIRED` before acquisition. Explicit resync is full with no predecessor and replaces
+  the table with the current observation; it does not retain earlier days. Once the bounded
+  snapshot materializer ships, compute a transition with
   `lag(column) over (partition by <identity> order by <snapshot column>)` and keep
   the rows where the two are `is distinct from` each other. Never diff two sealed versions by hand;
   a version is not a date, and nothing outside the seal can be replayed.
 
 ### Authoring a collection source
 
+Current execution boundary: an initial full or explicit table resync may acquire a collection,
+but collection-only and mixed normal refreshes return `RESYNC_REQUIRED` before acquisition. An
+explicit resync is full, has no predecessor, and starts collection discovery from the beginning.
+The ledger continuation rules below describe historical receipts and a future bounded materializer,
+not a current normal-refresh action.
+
 A corpus published as an index plus many detail pages is **one** source through
 `public.https.collection@2.0.0`, not one source per page. Source count and publisher count are
 unaffected by how many pages sit behind it: 2,819 pages is one source, one publisher, one line on
@@ -79,17 +87,16 @@ column reaches the transform as `VARCHAR`; selecting `page_ordinal` bare while d
 `TIMESTAMP WITH TIME ZONE` and nothing else.
 
 **A listing that could not be read says so.** The coverage block carries `discovery_failure`:
-`null`, or the code and one sentence for the first listing request the run was refused. A refresh
-that already holds pages keeps every row and succeeds; a FIRST run whose listing is refused has
-nothing to continue from and fails with `COLLECTION_DISCOVERY_FAILED` rather than sealing an empty
-corpus. If a run reports zero discovered, read that member before concluding the publisher's index
-is empty.
+`null`, or the code and one sentence for the first listing request the run was refused. A full
+resync whose listing is refused has nothing to continue from and fails with
+`COLLECTION_DISCOVERY_FAILED` rather than sealing an empty corpus. If a run reports zero
+discovered, read that member before concluding the publisher's index is empty.
 
-**Expect the first run not to finish, and say so.** A run that spends its fetch budget still
-SUCCEEDS; its coverage block reports `complete: false` and names the budget that ended it. The
-next refresh continues from the ledger — new pages first, then failures oldest first, then
-revisits — and every row of a page this run did not fetch survives byte for byte. Report the
-shortfall as what it is, a backfill in progress, not as a failure and not as a complete corpus.
+**A partial full/resync result must be reported as partial.** A run that spends its fetch budget
+still SUCCEEDS; its coverage block reports `complete: false` and names the budget that ended it.
+Do not call that a backfill in progress: no current normal refresh can continue the ledger, and a
+later explicit resync starts discovery over. Report it as an incomplete corpus and either accept
+that limitation or revise the bounded recipe before another explicit resync.
 
 A page that disappears from the index deletes nothing. A page answering with different content
 replaces exactly its own rows, increments `page_revision`, and keeps the previous digest: that is

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/src/mostlyright/data_harness/page_coverage.py

@@ -37,9 +37,9 @@ __all__ = [
 #: ``{"code", "detail"}`` of the first listing request this run could not read. It is stated
 #: because a discovery that failed and a corpus that is empty produce the same counts, and a
 #: reader with only the counts cannot tell an unreachable index from a publisher who has stopped
-#: publishing. A run that had nothing to continue from does not report it -- it fails, under
-#: ``COLLECTION_DISCOVERY_FAILED`` -- so a block carrying it is always a refresh that kept its
-#: rows.
+#: publishing. A full/resync run with no listing and no rows fails under
+#: ``COLLECTION_DISCOVERY_FAILED``. Historical refresh receipts may carry the member after keeping
+#: predecessor rows, but current collection refresh is refused before acquisition.
 PAGE_COVERAGE_FIELDS: tuple[str, ...] = (
     "discovered",
     "discovery_requests",
@@ -103,5 +103,5 @@ def page_coverage_sentence(block: Mapping[str, Any]) -> str:
         parts.append(f"the page listing could not be read ({listing['code']})")
     if failed:
         parts.append(f"{failed:,} failed")
-    parts.append("continues on the next refresh")
+    parts.append("requires another explicit resync, which starts from the beginning")
     return "; ".join(parts)

{mostlyright_data-0.23.0 → mostlyright_data-0.24.0}/src/mostlyright/data_harness/thin/recipe.py

@@ -61,6 +61,7 @@ COMMAND_HELP = "register one recipe document and print the identifiers the serve
 
 REGISTER_SCHEMA = f"{THIN_SCHEMA_PREFIX}-recipe.v1"
 SHOW_SCHEMA = f"{THIN_SCHEMA_PREFIX}-recipe-read.v1"
+READINESS_SCHEMA = f"{THIN_SCHEMA_PREFIX}-recipe-readiness.v1"
 
 #: The bound on the document. The same number the authoring lane bounded one of its six documents
 #: by -- four times the contract's own 262144-byte ceiling on a canonical recipe -- rather than a
@@ -111,6 +112,14 @@ NAMED_UNDESCRIBED_COLUMNS = 12
 #: The word that selects the read-back form. A positional rather than a flag because
 #: `mr-data recipe show ID` is how a person says it out loud.
 SHOW = "show"
+READINESS = "readiness"
+# Studio evaluates the exact persisted admission proof for each recipe revision. Keep this
+# inventory page small enough that a normal production workspace completes each bounded request
+# inside the thin client's request deadline, while preserving the established 40,000-item
+# inventory bound through cursor aggregation.
+READINESS_PAGE_SIZE = 25
+MAX_READINESS_ITEMS = 40_000
+MAX_READINESS_PAGES = MAX_READINESS_ITEMS // READINESS_PAGE_SIZE
 
 
 def declare_arguments(parser: argparse.ArgumentParser) -> None:
@@ -125,8 +134,9 @@ def declare_arguments(parser: argparse.ArgumentParser) -> None:
         "target",
         metavar="FILE",
         help=(
-            "the recipe document to register, as a path to a JSON file; or the word `show`, "
-            "followed by a recipe identifier, to read one back"
+            "the recipe document to register, as a path to a JSON file; the word `show`, "
+            "followed by a recipe identifier, to read one back; or `readiness` to inventory "
+            "whether current tables may schedule a bounded incremental refresh"
         ),
     )
     parser.add_argument(
@@ -618,6 +628,149 @@ def show(
     }
 
 
+def _readiness_row(record: Any) -> dict[str, Any]:
+    """One immutable recipe revision, checked before an audit receipt repeats it."""
+
+    if not isinstance(record, Mapping):
+        raise ThinLaneError(
+            "THIN_RESPONSE_INVALID", "Studio returned a non-object readiness record"
+        )
+    recipe_id = identifier(record.get("recipe_id"), "recipe identifier")
+    dataset_id = identifier(record.get("dataset_id"), "dataset identifier")
+    table_id = identifier(record.get("table_id"), "table identifier")
+    digest = record.get("recipe_digest")
+    if not isinstance(digest, str) or not digest:
+        raise ThinLaneError(
+            "THIN_RESPONSE_INVALID", "Studio returned readiness without recipe_digest"
+        )
+    current = record.get("is_current")
+    predecessor_ready = record.get("predecessor_ready")
+    incremental_materialization_ready = record.get("incremental_materialization_ready")
+    differential_proof_ready = record.get("differential_proof_ready")
+    ready = record.get("refresh_ready")
+    table_status = record.get("table_status")
+    if not all(
+        isinstance(value, bool)
+        for value in (
+            current,
+            predecessor_ready,
+            incremental_materialization_ready,
+            differential_proof_ready,
+            ready,
+        )
+    ):
+        raise ThinLaneError(
+            "THIN_RESPONSE_INVALID",
+            "Studio returned readiness without its required boolean readiness facts",
+        )
+    if table_status not in {"live", "not_live"}:
+        raise ThinLaneError("THIN_RESPONSE_INVALID", "Studio returned an unknown table_status")
+    sources = record.get("sources")
+    blocking = record.get("blocking_source_names")
+    if not isinstance(sources, list) or not isinstance(blocking, list):
+        raise ThinLaneError(
+            "THIN_RESPONSE_INVALID",
+            "Studio returned readiness without source classifications and blocking source names",
+        )
+    parsed_sources: list[dict[str, str]] = []
+    for source in sources:
+        if not isinstance(source, Mapping):
+            raise ThinLaneError(
+                "THIN_RESPONSE_INVALID", "Studio returned a non-object source readiness"
+            )
+        name = source.get("source_name")
+        classification = source.get("classification")
+        next_action = source.get("next_action")
+        if (
+            not isinstance(name, str)
+            or not name
+            or classification not in {"closed", "window", "collection", "recorded", "resync_only"}
+            or next_action
+            not in {"reuse_predecessor", "acquire_incremental", "recorded", "resync_required"}
+        ):
+            raise ThinLaneError(
+                "THIN_RESPONSE_INVALID", "Studio returned an invalid source readiness"
+            )
+        parsed_sources.append(
+            {"source_name": name, "classification": classification, "next_action": next_action}
+        )
+    if any(not isinstance(name, str) or not name for name in blocking):
+        raise ThinLaneError(
+            "THIN_RESPONSE_INVALID", "Studio returned an invalid blocking source name"
+        )
+    return {
+        "recipe_id": recipe_id,
+        "recipe_digest": digest,
+        "dataset_id": dataset_id,
+        "table_id": table_id,
+        "is_current": current,
+        "table_status": table_status,
+        "predecessor_ready": predecessor_ready,
+        "incremental_materialization_ready": incremental_materialization_ready,
+        "differential_proof_ready": differential_proof_ready,
+        "refresh_ready": ready,
+        "sources": parsed_sources,
+        "blocking_source_names": list(blocking),
+    }
+
+
+def readiness(
+    args: argparse.Namespace, *, client: StudioV4DatasetClient | None = None
+) -> dict[str, Any]:
+    """Inventory every recipe revision's deterministic incremental-refresh readiness."""
+
+    selected = client or _client(args)
+    rows: list[dict[str, Any]] = []
+    cursor: str | None = None
+    seen_cursors: set[str] = set()
+    pages = 0
+    while True:
+        pages += 1
+        if pages > MAX_READINESS_PAGES:
+            raise ThinLaneError(
+                "THIN_RESPONSE_INVALID",
+                f"Studio's readiness inventory did not end within {MAX_READINESS_PAGES} pages",
+            )
+        page = selected.recipe_readiness(cursor=cursor, limit=READINESS_PAGE_SIZE)
+        if page.get("workspace_id") != str(selected.session.workspace_id):
+            raise ThinLaneError(
+                "THIN_RESPONSE_INVALID",
+                "Studio returned a readiness inventory for a different workspace",
+            )
+        listed = page.get("recipes")
+        if not isinstance(listed, list):
+            raise ThinLaneError(
+                "THIN_RESPONSE_INVALID", "Studio returned readiness without recipes"
+            )
+        rows.extend(_readiness_row(item) for item in listed)
+        next_cursor = page.get("next_cursor")
+        if next_cursor is None:
+            break
+        cursor = identifier(next_cursor, "next recipe cursor")
+        if cursor in seen_cursors:
+            raise ThinLaneError(
+                "THIN_RESPONSE_INVALID",
+                "Studio repeated a readiness cursor, so the inventory cannot be completed safely",
+            )
+        seen_cursors.add(cursor)
+    current = [row for row in rows if row["is_current"]]
+    blocked = [row for row in current if not row["refresh_ready"]]
+    return {
+        "schema_version": READINESS_SCHEMA,
+        "status": "recipe_readiness_listed",
+        "lane": "hosted",
+        "recipes": numbered("recipe", rows),
+        "count": len(rows),
+        "current_count": len(current),
+        "current_not_ready_count": len(blocked),
+        "pages_read": pages,
+        "note": (
+            "Only Studio's persisted readiness classification may authorize a scheduled refresh. "
+            "A resync-only source requires table resync; it is never widened by a schedule."
+        ),
+    }
+
+
 def recipe(
     args: argparse.Namespace, *, client: StudioV4DatasetClient | None = None
 ) -> dict[str, Any]:
@@ -630,7 +783,7 @@ def recipe(
     with a directory in front of it.
     """
 
-    if args.target == SHOW:
+    if args.target in {SHOW, READINESS}:
         given = [
             flag
             for destination, flag in _REGISTER_ONLY_FLAGS
@@ -639,14 +792,22 @@ def recipe(
         if given:
             raise ThinLaneError(
                 "THIN_REQUEST_INVALID",
-                f"mr-data recipe show reads one recipe back and registers nothing, so "
+                f"mr-data recipe {args.target} is read-only and registers nothing, so "
                 f"{', '.join(given)} cannot mean anything here",
             )
-        return show(args, client=client)
+        if args.target == SHOW:
+            return show(args, client=client)
+        if args.recipe_id:
+            raise ThinLaneError(
+                "THIN_REQUEST_INVALID",
+                "mr-data recipe readiness takes no recipe identifier",
+            )
+        return readiness(args, client=client)
     if args.recipe_id:
         raise ThinLaneError(
             "THIN_REQUEST_INVALID",
-            f"mr-data recipe takes one document to register, or `show` and a recipe identifier; "
+            f"mr-data recipe takes one document to register, `show` and a recipe identifier, "
+            f"or `readiness`; "
             f"{args.target!r} is neither",
         )
     return register(args, client=client)
@@ -656,9 +817,14 @@ __all__ = [
     "COMMAND_HELP",
     "DRAINED_SOURCE_BYTE_CEILING",
     "MAX_DOCUMENT_BYTES",
+    "MAX_READINESS_ITEMS",
+    "MAX_READINESS_PAGES",
     "NAMED_UNDESCRIBED_COLUMNS",
     "NDJSON_SOURCE_BYTE_CEILING",
     "READER_SOURCE_BYTE_CEILING",
+    "READINESS",
+    "READINESS_PAGE_SIZE",
+    "READINESS_SCHEMA",
     "REGISTER_SCHEMA",
     "REQUIRED_DOCUMENT_MEMBERS",
     "SHOW",
@@ -670,6 +836,7 @@ __all__ = [
     "columns_without_description",
     "declare_arguments",
     "read_document",
+    "readiness",
     "recipe",
     "record_delegation",
     "register",