mostlyright-data 0.26.1__tar.gz → 0.28.0__tar.gz

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.26.1
+Version: 0.28.0
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.26.1"
+version = "0.28.0"
 description = "Mostly Right hosted CLI for reviewed datasets"
 readme = "README.md"
 requires-python = ">=3.11"

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/SKILL.md

@@ -14,10 +14,16 @@ references. Load the reference for the current stage only, not the entire librar
 1. Run `mr-data whoami --json` once. If authentication needs repair, read
    [authentication](references/cloud-authentication-preflight.md) and follow the reported action.
    Never print credentials or bypass their storage protection.
-2. Create the dataset once with `mr-data dataset create --name "Working title" --handoff --json`.
-   Keep its `dataset_id`. Link the stable `dashboard_url` in the first build message; open the
-   separate `navigation.url` in the host's visible browser if available. A missing handoff does
-   not undo creation: do not create a duplicate dataset.
+2. If the user handed you a `dataset_id` with the request, that dataset is the one: its page is
+   already open in their browser. Do not create another, do not run `mr-data open`, and put that
+   id in the recipe's `dataset.id`. Otherwise create the dataset once with
+   `mr-data dataset create --name "Working title" --handoff --json` and keep its `dataset_id`.
+   A create that answers `open_request_reused` made nothing: it found the dataset the user already
+   opened for this request, so build into that `dataset_id` and do not open it again; use
+   `--separate` only when the user asked for another dataset. After a real create, link the stable
+   `dashboard_url` in the first build message and open the separate `navigation.url` in the
+   host's visible browser if available. A missing handoff does not undo creation: do not create a
+   duplicate dataset.
 3. Settle purpose, row grain, tables, coverage, required sources and update cadence before
    dependent work. In an attended build, ask unresolved material questions in one concise message
    with recommended choices; recording an assumption is not asking the user. Default to one joined,
@@ -46,16 +52,25 @@ references. Load the reference for the current stage only, not the entire librar
    Decide how each source continues before you register it, and whether the table needs a
    declared `rebuild`: read [making a table incremental](references/making-a-table-incremental.md).
    Run `mr-data recipe explain RECIPE.json --json` first. It names every reason the table could
-   not refresh, not only the first. Before any `--full` of a paid collection, run
-   `mr-data recipe forecast RECIPE_ID --digest DIGEST --json` and cap `pages.max_fetches_per_run`
-   from it. Declare only what the source's own address proves, and name the `resync_only`
-   sources and the reason in the build message.
+   not refresh, not only the first. Its `fast_features` names each fast feature the recipe
+   misses, with a one-line fix. Write `table.partition` for any table no window partitions that
+   has a date-like column, `batch` for a per-row GET whose API takes a list of ids, and
+   `reference` for a bulk dump beside a window
+   ([fast builds](references/making-a-table-incremental.md#fast-builds-tablepartition-batch-and-reference)).
+   Put them in the first revision, because each later change costs one resync. Adopting
+   `table.partition` on an existing table costs one full resync, so for a paid source ask the
+   user before that resync, as the brief and forecast rules require. Before any
+   `--full` of a paid collection, run `mr-data recipe forecast RECIPE_ID --digest DIGEST --json`
+   and cap `pages.max_fetches_per_run` from it. Declare only what the source's own address
+   proves, and name the `resync_only` sources and the reason in the build message.
 6. Read [build sizing](references/6-build-one-run-sized-to-acquire-every-measured-source-whole.md).
    Use `--full` for the requested build: it is progressive by default, with a five-minute
    inspection checkpoint while the same acquisition continues. Choose bounds from measured
    source size; row and byte clamps apply per source, not elapsed time. Use explicit `--sample`
    only for a deliberate standalone inspection. Preserve the agreed semantics and quality checks.
-7. Follow the returned run with `mr-data watch RUN_ID --json`. Read `run.status`: returning an
+7. Follow the returned run with `mr-data watch RUN_ID --json`. Its `source_rates` and
+   `mr-data status RUN_ID --json` on a live run give each collection source's rate and time left.
+   Quote those numbers rather than doing the arithmetic. Read `run.status`: returning an
    existing run does not mean new work was queued. Resume the known run after an interruption,
    using the saved cursor where available. Read [run lifecycle](references/live-run.md) for
    holds and continuation commands. Confirm projected spend before acquisition when required;

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/1-open-the-page-and-the-link-to-it-in-the-first-message.md

@@ -1,5 +1,12 @@
 ### 1. Open — the page, and the link to it, in the first message
 
+**If the user handed you a `dataset_id` with the request, the page already exists and is open in
+their browser.** Skip this stage's create and do not open anything: hold that `dataset_id`, link its
+page once in the first message, and bind the recipe to it as `dataset.id`. The same holds when
+`dataset create` answers `open_request_reused`: it made nothing, its `navigation.url` is the plain
+page rather than a handoff, and `navigation.open_before_research` is false. If the quoted request is
+a different dataset from the one you are building, run `dataset create --separate` instead.
+
 Create the dataset from a working title and put it in front of whoever is watching, **before any
 source lookup**. This is the first command after the preflight — nothing reads a note, searches a
 catalog, opens a provider page or fetches a URL until it has returned:

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/3-probe-read-a-source-before-committing-to-it.md

@@ -17,10 +17,14 @@ mr-data dataset activity DATASET_ID --phase drafting \
 The message is one plain line about what you are doing now, and the facts are counts you already
 hold. A fact you have not counted is a fact to leave out.
 
-**Then stop writing it by hand.** Every later phase comes from the commands themselves:
-registering a recipe, starting a run, reading a finished one, asking a question and promoting a
-table each move the page on their own. Stage 3 begins before anything is submitted, which is why
-it is the one stretch that has to say so by hand.
+**Keep writing it through the whole build.** The line is yours. Update it when your task changes
+and renew it as the [agent presence lifecycle](writing-a-decision-record.md#agent-presence-lifecycle)
+says. One command adds a line of its own: registering a recipe tells the page that the recipe is
+being written, and then that it is registered. Each of those is held back while the page shows a
+current line you set, checked just before sending. Apart from `mr-data clarify --dataset`, which
+puts your pending question on the page, no other command tells the page anything. The page shows
+a run and a table going live from Studio's own record of them, and reading a finished run changes
+nothing on it.
 
 **Judge the sources.** This stays with the external agent and is the substance of the task. Search
 every discovery lane with no lane priority — the sealed catalog, host public-web search, official
@@ -79,7 +83,7 @@ reason codes, only once the discovery lanes and the comparisons are complete —
 --dataset DATASET_ID` takes two identifiers rather than a question in words, and no v4 registration
 receipt returns a question identifier; it also rides the frozen `/v3/sessions` routes, which a
 deployment may have switched off. So the reading path on a hosted workspace is the ordinary loop,
-run early: register the recipe and take one unwindowed run.
+run early: register the recipe and take one run without `--window`.
 
 **It obeys the ordinary gate.** Registering needs the brief answered or delegated, so this run
 happens AFTER stage 2's answers arrive — or at once under a delegation. What continues while you

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/5-draft-one-recipe-document-one-call.md

@@ -29,9 +29,9 @@ watching; without it there is no page whose record could carry the brief, and th
 **A document that carries `dataset.id` leaves `dataset.description` out.** Registration FILLS an
 empty page's description and never replaces one, so this destroys nothing; what it does is write
 prose composed for the recipe onto the page as its documentation, from the wrong writer. The page's
-description is written once, below, with `mr-data dataset set DATASET_ID --description-file F`
-under the version it read, and the preflight refuses a document carrying both. Leave `dataset.name`
-in: the title is the recipe's own subject, settled below.
+description is written once, below, in the one `mr-data dataset set` call that carries the title
+and the tags with it, under the version it read. The preflight refuses a document carrying both.
+Leave `dataset.name` in: the title is the recipe's own subject, settled below.
 
 **Registration refuses a dataset nobody was asked about**, with `THIN_BRIEF_MISSING`. The record
 has to carry either an answered `clarification` block or a delegation `decision` block first. Where
@@ -66,7 +66,23 @@ of the rights basis the sources actually grant -- for example `CC-BY-4.0`, `ODbL
 where the sources' terms are not an open licence, leave `--license` off and say so under "Source and
 rights" instead. Never invent a licence.
 
-While composing, send bounded snapshots with `mr-data dataset draft DATASET_ID --description-file DRAFT.md --expected-version VERSION --json`. Chain the returned version into the next snapshot, writing a meaningful paragraph at a time. These snapshots are temporary and do not replace the saved description. Commit the complete documentation with `mr-data dataset set DATASET_ID --description-file DESCRIPTION.md --expected-version VERSION --json`. If another edit moved that version, stop the draft and reconcile it with the user’s text; never retry by blindly advancing the version. Do not send fabricated partial text or delay completed text to simulate writing.
+While composing, send bounded snapshots with `mr-data dataset draft DATASET_ID --description-file DRAFT.md --expected-version VERSION --json`. Chain the returned version into the next snapshot, writing a meaningful paragraph at a time. These snapshots are temporary and do not replace the saved description.
+
+**Commit the title, the tags and the description in ONE call.** A description sent on its own can
+make Studio write a new title and new tags over the ones you chose, so the final commit carries
+all three:
+
+```sh
+mr-data dataset set DATASET_ID --name "Prague airport weather history (LKPR), 2000-2026" \
+  --topics "weather,prague,czech republic,metar,iowa environmental mesonet" \
+  --description-file DESCRIPTION.md --expected-version VERSION --json
+```
+
+Read the receipt. `fields` is what the command sent. `changed` is every field the write moved.
+`written_by_studio` names each field Studio wrote itself, with its new value. If a title or tags
+you chose are in it, send the three together again.
+
+If another edit moved that version, stop the draft and reconcile it with the user’s text; never retry by blindly advancing the version. Do not send fabricated partial text or delay completed text to simulate writing.
 
 **What search engines and answer engines read.** These are the three fields search engines and
 answer engines index. A search engine (Google, Bing) ranks the page by the words a person types.

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/6-build-one-run-sized-to-acquire-every-measured-source-whole.md

@@ -29,6 +29,25 @@ coverage and the full build's proposed coverage before requesting any missing au
 > end of the feed; the other three sources came back whole. The full build would cover 2000 to 2026
 > for all four. Run it?
 
+**A sample of a source with a request window reads its latest days only.** The run asks that
+source for the latest `lookback_seconds`: rounded up to whole days, at least one day, ending at the
+end of the current UTC day, and never starting before `start_at`. With `lookback_seconds` of
+604800 that is the last seven days, whatever `--max-rows` says. A sample that starts after
+`start_at` records `truncated: true` on that source. A check on the whole table, such as a
+`row_expectation` floor, can then fail with `CHECK_FAILED` although the recipe and the source are
+sound. Registration names each such source and its span under `warnings`. A full run is what
+verifies a check on the whole table, once the full build is authorized.
+
+**A source that also declares `refresh_readiness` is read only up to its last completed day.**
+That leaves out the current UTC day, and the day before it until `additional_lag_seconds` of the
+current day have passed. With `lookback_seconds` of 604800 and no lag a sample reads six days.
+When no completed day is left, which is always so for `lookback_seconds` of 86400 or less, a
+sample without `--window` does not run: it is refused `WINDOW_NOT_READY`, or waits in the queue
+until Studio fails it. Registration says so under `warnings`, so read them before you start a
+sample. A full run reads such a source from its `start_at`. A source whose `start_at` is the
+current UTC day has no completed day until that day is over, so neither a sample nor a full run of
+it reads anything before then.
+
 If existing authorization or a stage-2 delegation covers this full build and its spend, do not
 ask again: state the projection and run it. A narrower delegation does not authorize the full run.
 

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/7-interrogate-ask-the-run-what-it-actually-delivered.md

@@ -48,6 +48,14 @@ question actually asked: `mr-data query` opens a session for the run's own datas
 is warm, waits, asks, and closes the session behind the answer. Pass `--no-session` to be refused
 instead, and wait through the window the refusal names rather than retrying in a tight loop.
 
+**A query sent straight after another may find no worker to start.** The worker behind the query
+before is still winding down, and Studio fails the new session at once with
+`RESEARCH_SESSION_WORKER_UNAVAILABLE`. `mr-data query` waits 8 seconds and opens one more session,
+once. The answer then names the first session under `session.retried_after`. If the second has no
+worker either, the command refuses `THIN_RESEARCH_SESSION_ENDED` and names both sessions, each with
+its state, close reason and failure code. Nothing was queued, so running the same command again
+repeats nothing.
+
 **The statement gate is a word-boundary scan over a masked copy of the statement and does not read
 SQL.** A column literally named `system`, `copy`, `set`, `load`, `call`, `glob`, `import` or
 `export` is refused by name even in an obviously harmless statement; quote the identifier or

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/commands.md

@@ -9,9 +9,9 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data login` | Complete device approval and store a device credential. `login --force` is the compatibility alias for safe `auth rotate`, never an in-place truncate. |
 | `mr-data whoami` | Use the compatibility alias for the remotely validated `auth status` answer. |
 | `mr-data clarify` | Check whether clarification is allowed. Use `--attended` for an active user chat and `--unattended` for a scheduled invocation; an explicit operator veto such as `MOSTLYRIGHT_ATTENDED=0` still wins. A non-TTY subprocess does not prove the user is absent. The command does not ask the person in chat or wait for a reply; the agent must do that. It exits non-zero when nobody can answer or the recipe is already registered. |
-| `mr-data probe` | `probe SOURCE_ID QUESTION_ID --dataset DATASET_ID [--kind source_inspect\|sample_rows\|profile_columns\|evaluate_expression]` asks one already-registered source one question and prints the answer. **Do not plan a source inspection around it.** Both positionals are identifiers — `QUESTION_ID` is a question identifier, which no v4 registration receipt returns — and the command rides the frozen `/v3/sessions` routes, which a deployment may have switched off. Read a source instead by registering the recipe and taking one unwindowed run under `--max-rows`, then `peek`, `query` and `receipt`. |
+| `mr-data probe` | `probe SOURCE_ID QUESTION_ID --dataset DATASET_ID [--kind source_inspect\|sample_rows\|profile_columns\|evaluate_expression]` asks one already-registered source one question and prints the answer. **Do not plan a source inspection around it.** Both positionals are identifiers — `QUESTION_ID` is a question identifier, which no v4 registration receipt returns — and the command rides the frozen `/v3/sessions` routes, which a deployment may have switched off. Read a source instead by registering the recipe and taking one run without `--window` under `--max-rows`, then `peek`, `query` and `receipt`. |
 | `mr-data catalog` | `catalog search "QUESTION"` asks the sealed public-source catalogue which of its entries might answer a question and ranks them best first. Every ranked entry comes back with the disposition its own facts earned — `admitted`, `human_escalation_required` or `refused` — because an entry the catalogue could not vouch for is still a finding. `--format csv` states the one data format the question requires — one token per question, never repeated — and it is not a filter: an entry that does not declare that format still comes back ranked, with disposition `refused` and `filters_match` false, so nothing is held back; `--limit N` ranks at most N, up to 25. It fetches nothing and registers nothing. The catalogue holds one provider (Data.gov) and only part of it, so it is never exhaustive and an empty answer is not evidence that no such source exists. When the answer is `catalog_unavailable`, this deployment has no catalogue to search: record the lane and carry on. |
-| `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --topics "a,b,c" --license ID --description-file F` writes the title, the descriptive tags, the SPDX licence and the description under the version it was read at, retrying once if somebody else wrote first, and an empty `--topics` or `--license` takes that value off the page; a saved write whose public sync fails exits 2 and reports `public_projection_synced: false` — use `dataset sync ID` to retry that sync without rewriting Studio; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset publish ID [--mode public|link|private]` says who can read the dataset — `public` lists it in the public directory and serves it at an address anybody can read, `link` serves it at an unlisted address, `private` takes it back to the workspace — and `dataset publish ID --show` reads that back without changing it; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
+| `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`, unless the user's own request already has an empty dataset open, in which case it answers `open_request_reused` with that `dataset_id` and mints nothing (`--separate` mints a new one for a different dataset); `dataset show ID` reads it back; `dataset set ID --name TEXT --topics "a,b,c" --license ID --description-file F` writes the title, the descriptive tags, the SPDX licence and the description under the version it was read at, retrying once if somebody else wrote first, and an empty `--topics` or `--license` takes that value off the page; send `--name` and `--topics` whenever you send `--description-file`, because a description sent on its own can make Studio write a new title and new tags, and read `changed` and `written_by_studio` on the receipt for every field the write moved; a saved write whose public sync fails exits 2 and reports `public_projection_synced: false` — use `dataset sync ID` to retry that sync without rewriting Studio; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset publish ID [--mode public|link|private]` says who can read the dataset — `public` lists it in the public directory and serves it at an address anybody can read, `link` serves it at an unlisted address, `private` takes it back to the workspace — and `dataset publish ID --show` reads that back without changing it; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. The document is read first and every fault comes back in one refusal, each with the JSON pointer that names it; `--no-lint` sends it as written instead. Registration refuses `THIN_BRIEF_MISSING` when the dataset's record carries no answered question and no delegation, and `THIN_RECIPE_DATASET_UNBOUND` when the document names no `dataset.id`; `--delegated "their words"` writes the delegation onto the dataset and registers in the same command, and clears the first of those two and never the second. `mr-data recipe show ID` reads one back. `mr-data recipe explain FILE` (or a recipe identifier) lists every reason its table can or cannot refresh, each with a rule code and Studio's suggested repair, and exits 2 when one blocks it. `mr-data recipe forecast RECIPE_ID --digest DIGEST` says what one run would ask of each source before it runs, and exits 3 when the run would overrun today's platform quota or one attempt. `mr-data recipe readiness --table TABLE_ID --current-only` reads the revision one table runs now. [Making a table incremental](making-a-table-incremental.md#the-loop-explain-forecast-run-diagnose) says when to use each. |
 | `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
 | `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The mode is one of six: `sample`, `full`, `refresh`, `backfill`, `compact` and `replay`. Four have a shorthand flag (`--sample`, `--full`, `--refresh`, `--backfill`) and two do not, so write `--mode MODE`, which is accepted for every one of them. A refresh executes only Studio's persisted source-action plan, and a plan shape it does not refresh returns `RESYNC_REQUIRED` before acquisition; [which plans refresh](making-a-table-incremental.md#which-plans-refresh) lists them, and `mr-data recipe explain` says which one a table is. Use `mr-data table resync TABLE_ID --request-id UUID` for an explicit full source reread, retaining that UUID if the response is uncertain. `--backfill` states the exact window with `--window START END`. `--mode replay --sources-from RUN_ID` asks Studio to run the registered revision against the RETAINED RAW INPUTS of one named successful run of the same table: nothing is fetched from this computer, nothing is acquired again, the result is compared against that run and never becomes the live version, and it neither asks for nor records an approval. Where Studio has replay switched off, or the named run is not successful, not the same table, or no longer retains its inputs, it answers a typed refusal — report the code rather than retrying in another mode. `--sources-from` on any other mode is refused `THIN_ARGUMENT_INVALID` before anything is sent. `--max-rows` and `--max-source-bytes` bound ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles and whose printed `confirm_command` re-states every argument the request carried. `--approve-full RUN_ID` releases the full an existing legacy sample-first pair is holding once its preview has succeeded, naming either half of the pair; where the deployment still wants a person at a browser it answers `THIN_INTERACTIVE_HUMAN_REQUIRED` and names the run page. `--retry RUN_ID` tries one failed run again when its triple carries `room_fault: true`, re-stating that run's own coordinate under a byte ceiling that cannot narrow it, and refuses with the triple when the recipe was at fault. `--cancel RUN` stops one. |
@@ -83,6 +83,12 @@ the same code and starts no run. Only time clears it, and the refused run's rang
 new run is needed once the day is over. Report the code and wait rather than retrying or revising
 the recipe.
 
+**A sample of such a source can find no completed day too.** It reads only the completed days
+inside its lookback, and with `lookback_seconds` of 86400 or less there is never one. A sample
+without `--window` is then refused `WINDOW_NOT_READY`, or waits in the queue until Studio fails
+it. Registration names the source under `warnings`, and
+[stage 6](6-build-one-run-sized-to-acquire-every-measured-source-whole.md) has the rule.
+
 **A refresh on the one-windowed-source bounded route seals no `receipt`.** `mr-data receipt`
 answers `THIN_NO_MATCHING_ARTIFACT` and names what the run did seal, which is the ordinary shape
 of that route rather than a fault. Read `mr-data status RUN_ID --json` instead: its `coverage`

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/live-run.md

@@ -72,8 +72,10 @@ covers nothing and says so. A `succeeded` one reports what `coverage.truncated`
 else states the bound it ran under — a `sample` its ceilings, a `full` the whole dataset, and a
 `full` at `awaiting_sample_approval` the whole dataset plus the preview it waits on. The mode
 alone never establishes a truncation, which is the rule reference 6 states: a sample sized above
-every measured source cut nothing and is the build. It is absent on `refresh`, `backfill`,
-`compact` and `replay`, and it is never evidence that a run is executing; `status` is.
+every measured source cut nothing and is the build. A source with a request window is the
+exception: a sample reads its latest lookback span only, and the sentence says so. It is absent
+on `refresh`, `backfill`, `compact` and `replay`, and it is never evidence that a run is
+executing; `status` is.
 
 **A failed run always carries the same three members** — `failure_code`, `failure_detail` and
 `failed_stage` — on three surfaces: the run record, the terminal event on the stream, and every

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/making-a-table-incremental.md

@@ -20,15 +20,26 @@ after every revision.
    `suggestions`. After the first full, run `mr-data recipe explain RECIPE_ID --json` again: the
    table route also judges the predecessor. The command exits 0 when the table is ready to refresh
    and 2 when a finding blocks it. An advisory finding (`blocks_refresh: false`) never changes the
-   exit code.
+   exit code. The receipt's `fast_features` says which of `table.partition`, `batch` and
+   `reference` the recipe uses or misses, and which sources every run reads whole. Each missed
+   feature carries a one-line fix. `fast_features_missed` lists them for a script. See
+   [Fast builds](#fast-builds-tablepartition-batch-and-reference).
 2. `mr-data recipe forecast RECIPE_ID --digest DIGEST --json` before any `--full` that pays for
    calls. It sends nothing. Per source, it reports the request ceiling the document allows, the
    count the last run of the same source definition actually sent, the pace, and the runtime at
    the ceiling. For a platform credential it also reports today's remaining quota. It exits 3 on
    `PLATFORM_SPEND_EXCEEDS_QUOTA` or `RUNTIME_EXCEEDS_ONE_ATTEMPT`. Stop there and resize the
-   collection rather than start the run.
+   collection rather than start the run. The forecast also reads observed limits. Each source
+   with a history run gets `observed`: the rate that run actually reached and the runtime of the
+   expected requests at that rate. A line flags a run that went under half its declared pace,
+   because some other limit held it. On a Studio without the forecast route, a platform source is
+   paced at its binding's stated rate and provider limits. `--no-observed` skips both reads.
 3. `mr-data run --recipe RECIPE_ID --digest DIGEST --full --json`, followed as
-   [the run lifecycle](live-run.md) describes.
+   [the run lifecycle](live-run.md) describes. `mr-data watch RUN_ID` prints a `rate` line for
+   each collection source about once a minute: members done, members a minute, and the time left
+   once discovery has listed everything. `mr-data status RUN_ID --json` on a live run reports the
+   same under `source_rates`. The rate counts saved answers replayed, so it starts fast and
+   settles on the sending rate.
 4. `mr-data status RUN_ID --diagnose --json` when a run failed, stalled or cost more than the
    forecast said. It combines the run, its progress, its checkpoints and every attempt Studio
    recorded, and a per-row source's plan counts: upstream rows, distinct requests, answers
@@ -62,6 +73,10 @@ primitives, plus `rebuild` for a table where none of them applies.
    bytes the predecessor sealed. If the publisher later restates that period, the table keeps
    serving the old bytes, no run fails, and only an explicit resync repairs it. Never mark the
    current period closed. That drops new data outright, which is a worse mistake.
+   **Is it a bulk dump the publisher republishes, beside a request window?** State
+   `reference: {"recheck_seconds": N}` instead. A refresh reuses it like a closed source, and the
+   first scheduled refresh N seconds after its last read reads it whole again. See
+   [reference](#reference-a-bulk-dump-beside-a-window).
 3. **Does the address carry a date range, with both ends?** Declare a `window` with a `request`.
    A refresh asks again for the recent range inside `lookback_seconds`, so it picks up corrections
    on its own. This is the one continuation that makes a table refresh on its own.
@@ -85,6 +100,7 @@ Studio records the answer as one classification per source:
 | --- | --- | --- |
 | `source_class: "stream"` | `recorded` | `recorded` |
 | `closed: true` | `closed` | `reuse_predecessor` |
+| `reference` | `closed` | `reuse_predecessor`, and a whole read when its recheck is due |
 | a `collection` | `collection` | `acquire_incremental` |
 | a `window` carrying a `request` | `window` | `acquire_incremental` |
 | none of those | `resync_only` | `resync_required` |
@@ -144,9 +160,10 @@ Declaring every source is the first gate. The shape the declarations form is the
 
 - **Every source a recorded stream.** The refresh appends sealed batches.
 - **At least one `window` with a `request`.** The refresh replaces the bounded partitions the
-  window asked for. Beside it the plan may hold `closed` siblings, which reuse their sealed bytes.
-  Or it may hold per-row request collections that carry the window's partition column. It may not
-  hold both.
+  window asked for. Beside it the plan may hold `closed` or `reference` siblings, which reuse their
+  sealed bytes. It may also hold per-row request collections that carry the window's partition
+  column. Since ADR 0061 it may hold both. No per-row collection may derive its rows
+  from a `closed` or `reference` source.
 
 The second family is necessary, not sufficient. A plan of more than one source (one window and one
 `closed` sibling is already two) writes one table manifest. So every window must agree on
@@ -174,7 +191,7 @@ source and the pointer to change.
 | `TABLE_FORMAT_UNAVAILABLE` | The workspace writes an older table format. | `wait_for_platform`. |
 | `SOURCE_READS_WHOLE` | A source states no continuation, so a refresh would read it whole. | `declare_request_window` (its `candidate_parameters` name the address's date pairs), `mark_closed`, or `declare_rebuild`. |
 | `COLLECTION_UPSTREAM_NOT_DIRECT_WINDOW` | A collection is not a per-row request over a direct request window. | `carry_partition_column`, `make_upstream_a_request_window`, `split_table`, or `declare_rebuild`. |
-| `CLOSED_SIBLING_OF_DERIVED_COLLECTION` | A `closed` source sits beside a per-row collection. | `split_table`, or `declare_rebuild`. |
+| `CLOSED_SIBLING_OF_DERIVED_COLLECTION` | No longer emitted. Since ADR 0061 Studio restores a closed sibling beside a per-row collection. Never split a table over it. An older Studio may still name it. | None. Upgrade the Studio rather than the recipe. |
 | `RECORDED_MIXED_WITH_WINDOW` | A recorded stream sits beside a window. | `split_table`: keep the stream in its own table. |
 | `NO_PARTITION_DELTA` | No source is a window, so there is no partition to replace. | `platform_gap`, the collection repairs above, or `declare_rebuild`. |
 | `DIRECT_WINDOW_HISTORY_PROOF_MISSING` | A lone window's predecessor holds no source-history record. | `resync_once`. |
@@ -233,6 +250,7 @@ prints that too ("retry 1/3 at T"). `mr-data reschedule` on a table that cannot
 | --- | --- |
 | recorded stream | nothing: it reads sealed batches the recorder already paid for |
 | `closed` | nothing: it reuses the predecessor's sealed bytes |
+| `reference` | nothing on most refreshes, and one whole read once `recheck_seconds` have passed |
 | `window.request` | the new days plus `lookback_seconds`, per source |
 | per-row request collection | one request per upstream row in the refreshed partitions whose answer is not already held |
 | `collection` | today, only a full or resync reads it, and it reads the whole corpus |
@@ -250,6 +268,76 @@ plus a quarter. A cap near the real count makes a corpus that suddenly doubles f
 `pages.retries`. With no history the forecast says `EXPECTED_UNKNOWN`. Build a small `--sample`
 first, read its plan counts with `mr-data status RUN_ID --diagnose --json`, and size from them.
 
+### Fast builds: `table.partition`, `batch` and `reference`
+
+`mr-data recipe explain` lists each of these under `fast_features`. Each is a recipe member, so
+adding one changes the digest and the table's next full is a resync. Make every change in one
+revision so the table pays once. Adopting `table.partition` on an existing table costs one full
+resync. For a paid source, ask the user before that resync, as the brief and forecast rules
+require. All three need Studio 4.17. An older Studio that serves the explanation route refuses
+each of them at explain and registration with a 422. A Studio before 4.14 has no explanation
+route, and `mr-data recipe` refuses `batch` locally with `THIN_GRAMMAR_NOT_SERVED`.
+
+#### `table.partition`: keep unchanged partitions
+
+Write `table.partition` for any table that no window partitions and that has a date-like column.
+Collection tables, per-row tables and closed-only tables are the usual cases.
+
+```json
+"partition": {"column": "race_date", "encoding": "iso_date_prefix"}
+```
+
+It sits in `table`, beside `columns`.
+
+- `column` is a declared table column, and every row needs a value in it. Check
+  `count(*) where race_date is null` is 0 before you write it.
+- `encoding` is `iso_date_value` for a `date` column, `iso_date_prefix` for a timestamp or an
+  ISO string, `iso_month_prefix` for a calendar month, or `value` for a low-cardinality key.
+- A table holds at most 4,096 parts. A daily key over more than about 11 years overflows it. Use
+  `iso_month_prefix` for a longer span.
+- A full run keeps each partition whose bytes did not change by reference, and uploads only the
+  rest. The first full after you add it is the one resync, and it uploads everything once.
+- It does not make a refresh admissible. A table that could not refresh still cannot.
+- A window already partitions its table. Beside a window, `table.partition` must state the same
+  column and key, so leave it out. A table a stream writes cannot declare one.
+
+#### `batch`: many rows in one request
+
+A per-row GET whose API takes a list of ids in one parameter can send up to 100 members in one
+call:
+
+```json
+"request": {
+  "method": "GET",
+  "static_query": [{"name": "ids", "value": "{id}"}],
+  "parameters": [{"name": "id", "type": "string", "required": true}],
+  "batch": {"parameter": "id", "separator": ",", "max_rows": 50, "key_column": "movie_id"}
+}
+```
+
+- `parameter` is declared with `required: true`. `{id}` is the whole of one `static_query` value
+  and appears nowhere else.
+- `key_column` is a column the pinned Reader declares. The worker splits the answer back to
+  members by that column.
+- `batch` refuses a POST, a `body_template`, a `platform` credential and
+  `pages.max_unavailable_members` above 0. The preflight names each before anything is sent.
+- Member ids and saved answers do not change, so turning `batch` on keeps every paid answer.
+- `absent: "failed"` fails a member the answer does not name, so the next run asks again. The
+  default, `empty`, records it with no rows.
+
+#### `reference`: a bulk dump beside a window
+
+```json
+{"name": "genres", "reference": {"recheck_seconds": 86400}, "limits": {"max_source_bytes": 50000000}}
+```
+
+- `recheck_seconds` is from 3,600 to 31,536,000.
+- A reference is never `closed`, never carries a `window` or a `collection`, and is never a
+  stream. No per-row collection may derive its rows from it.
+- A refresh reuses its sealed bytes. Once `recheck_seconds` have passed, one scheduled refresh
+  reads it whole. The same digest changes nothing. A new digest makes that refresh recompute the
+  whole table, so every version joins exactly one edition of the dump.
+
 ### Probing a request window without an executing probe
 
 `mr-data recipe explain` suggests `declare_request_window` for a source that reads whole. Its

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/readers.md

@@ -38,8 +38,8 @@ or none: there is no other place a Reader is named. Read the pin grammar and the
 **No command tries a Reader against a URL.** `mr-data peek` reads the preview one hosted run sealed,
 and its `--reader`, `--reader-options`, `--format` and `--json-*` flags are accepted and print `no
 effect here; the hosted preview arrives already decoded`. A Reader is settled the way every other
-part of the document is settled: pin the coordinate in the recipe, register it, take an unwindowed
-run under `--max-rows`, and read the columns back with `peek` and the decode evidence with
+part of the document is settled: pin the coordinate in the recipe, register it, take a run without
+`--window` under `--max-rows`, and read the columns back with `peek` and the decode evidence with
 `receipt`. A wrong coordinate or a wrong settings object is a typed refusal on that run naming the
 source, which is the answer — change the pin and register the revision.
 

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/receipts.md

@@ -54,8 +54,11 @@ double quotes.
 
 Two things are currently broken here and are not worth diagnosing again:
 
-- Aggregating a timestamp column fails: `min(obs_time)` answers `Required module 'pytz' failed to
-  import`, because the query image lacks it. Aggregate the rendered value instead —
-  `min(CAST(obs_time AS VARCHAR))`.
+- Any answer that carries a timestamp with a time zone fails, aggregate or not. `select obs_time`,
+  `min(obs_time)` and `max(obs_time)` all answer `Required module 'pytz' failed to import`,
+  because the query image lacks it. Cast the result to text in the statement:
+  `CAST(max(obs_time) AS VARCHAR)`. Never write `max(CAST(obs_time AS VARCHAR))`, which takes the
+  maximum of text and compares characters, not moments. Studio may answer an identical statement
+  with its stored failure. The cast makes a new statement, so it always runs.
 - `mr-data receipt` on a run that succeeded answers `THIN_STUDIO_SCHEMA_VALIDATION_FAILED`. Read
   `mr-data status --json` for coverage and `mr-data download --kind receipt` for the sealed record.

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/required-protocol.md

@@ -21,7 +21,9 @@ are the first two acts of the session after the preflight — before the durable
 before `mr-data runs`, before a catalog is searched, before a provider's page is opened and before
 any URL is fetched. Research is what fills the page, and a page created after the research already
 happened is the assertion this stage exists to prevent. Nothing in stage 1 depends on knowing the
-sources: the title is a working one and stage 5 settles it.
+sources: the title is a working one and stage 5 settles it. A `dataset_id` handed over with the
+request, or a create that answers `open_request_reused`, is that page already open: build into it
+and open nothing.
 
 **Then establish whether anybody can be asked.** When the host provides an active user
 conversation, pass `--attended`; a tool subprocess without a terminal is not proof that the user

{mostlyright_data-0.26.1 → mostlyright_data-0.28.0}/skills/mr-data-build/references/sources.md

@@ -56,9 +56,11 @@ Write it as a source whose connector is that coordinate plus a `collection` memb
 - `collection.pages` says what may be fetched and how much. It names `allowed_origins`, which
   must include the discovery address's own origin. It may state `max_pages`, `max_fetches_per_run`
   (up to 50 000), `concurrency` (up to 10 000), `request_timeout_seconds`, `retries` and a
-  `revisit` rule. Its pace is `min_interval_milliseconds` in whole milliseconds, 16 or more and
-  1000 when left out, or the legacy whole-second `min_interval_seconds`. Only `allowed_origins` is
-  required. Every other member has a stated meaning when it is left out.
+  `revisit` rule. `retries` is 0 to 3 and 2 when left out. Registration refuses 4 or more. Its pace
+  is `min_interval_milliseconds` in whole milliseconds, 16 or more and 1000 when left out, or the
+  legacy whole-second `min_interval_seconds`. Leaving the pace out means one request a second. State
+  the pace the publisher allows. Only `allowed_origins` is required. Every other member has a
+  stated meaning when it is left out.
 - `limits.max_requests` is REQUIRED and bounds every request the run makes, discovery and detail
   pages together. `limits.max_source_bytes` bounds the fetched bytes; `limits.max_rows` clamps the
   merged relation.
@@ -235,6 +237,11 @@ with every byte the source has left, up to the 16 MiB one answer may take. With
 at 0 there is no second attempt, so the first answer too large fails the run. Raise
 `limits.max_source_bytes`, or `pages.retries` when it is 0. Neither helps an answer over 16 MiB.
 
+If the API takes a list of ids in one query parameter, add `request.batch` and send up to 100
+members in one call. It keeps every member id and saved answer. It refuses a POST, a
+`body_template`, a `platform` credential and a failure budget above 0.
+[`batch`](making-a-table-incremental.md#batch-many-rows-in-one-request) has the grammar.
+
 A collection whose calls are paid for is sized from evidence, not from its ceiling. Run
 `mr-data recipe forecast RECIPE_ID --digest DIGEST --json` before any `--full`, and cap
 `pages.max_fetches_per_run` at the suggestion it prints. A cap near the real count refuses a corpus
@@ -260,7 +267,7 @@ sends requests only for rows in the refreshed partitions. A request the predeces
 successfully is never sent again, whatever the row's carried cells or partition now are. A full
 build leaves out an upstream row whose partition cell does not read. Read this source alone in the
 transform, and take upstream values from the carried columns. A transform that joins it back to
-its upstream cannot refresh incrementally. Keep `closed` sources out of the same table. A plan with
-a per-row request collection beside a closed source is `RESYNC_REQUIRED`, although the table still
-registers and builds. [Which plans refresh](making-a-table-incremental.md#which-plans-refresh) has
-the rest, and the YC worked example there splits such a table in two.
+its upstream cannot refresh incrementally. A `closed` or `reference` source may sit in the same
+table, and a refresh restores it from the predecessor (ADR 0061). Do not split a table
+to separate them. No per-row collection may derive its rows from a `closed` or `reference` source.
+[Which plans refresh](making-a-table-incremental.md#which-plans-refresh) has the rest.