mostlyright-data 0.29.10__tar.gz → 0.29.13__tar.gz

{mostlyright_data-0.29.10 → mostlyright_data-0.29.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.29.10
+Version: 0.29.13
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
@@ -39,6 +39,10 @@ places it in your agent's skill directory; there is nothing to install by hand.
 edited yourself is never replaced, and `MOSTLYRIGHT_SKILL_AUTOINSTALL=0` switches the placement
 off.
 
+Once a day, after a command has printed its answer, `mr-data` asks the Python package index whether
+a newer release is published and says so in one line on stderr. `MOSTLYRIGHT_RELEASE_CHECK=0`
+switches that off.
+
 ## Sign in
 
 ```bash

{mostlyright_data-0.29.10 → mostlyright_data-0.29.13}/README.md

@@ -27,6 +27,10 @@ places it in your agent's skill directory; there is nothing to install by hand.
 edited yourself is never replaced, and `MOSTLYRIGHT_SKILL_AUTOINSTALL=0` switches the placement
 off.
 
+Once a day, after a command has printed its answer, `mr-data` asks the Python package index whether
+a newer release is published and says so in one line on stderr. `MOSTLYRIGHT_RELEASE_CHECK=0`
+switches that off.
+
 ## Sign in
 
 ```bash

{mostlyright_data-0.29.10 → mostlyright_data-0.29.13}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.29.10"
+version = "0.29.13"
 description = "Mostly Right hosted CLI for reviewed datasets"
 readme = "README.md"
 requires-python = ">=3.11"

{mostlyright_data-0.29.10 → mostlyright_data-0.29.13}/skills/mr-data-build/SKILL.md

@@ -9,6 +9,68 @@ Build the dataset the user requested, verify its contents, and report its actual
 Use hosted commands with `--json` and returned identifiers. This core workflow governs the
 references. Load the reference for the current stage only, not the entire library before starting.
 
+## Four gates
+
+Every build passes these four gates in order. Do not skip one. The numbered steps below carry
+the detail.
+
+**Gate 1. Read what Studio serves and what the recipe misses.** Draft the recipe. Run
+`mr-data recipe explain RECIPE.json --json` before you register. Read `studio_capabilities`: a
+capability that is not literal `true` is unavailable on this Studio. Read `fast_features`. On a
+new table, fix every entry marked `missed`, or say in the build message why it stays.
+On a table that is already live, list the missed entries and ask the user first. Each fix changes
+the recipe digest, so the next build is a full resync. A paid source is called again for every
+per-row request whose saved answer no longer applies.
+
+**Gate 2. State a pace and leave concurrency out.** A page collection that states no pace sends
+one request a second. State `pages.min_interval_milliseconds` as the gap the publisher documents.
+No documented limit: state 1000 ms and say so. That is a valid reason. Registration refuses a gap
+under 16 ms. That is a platform bound, never a target. Where the deployment spreads requests over
+its egress addresses, the gap is kept per address, so the publisher can see a multiple of the
+stated rate. No command prints how many addresses there are. So when the publisher's limit is
+per key or per account, state 1000 ms unless the user confirms the deployment does not spread
+requests. Leave `pages.concurrency` out: the worker then sizes the width, and a stated value is
+only a ceiling. Keep a stated one only when a run's refusal told you to lower it or the publisher
+documents a limit on concurrent connections, and record the reason in a dataset note so the next
+session finds it. Then choose the source shape that fits.
+
+| Source shape | Use | Always state | Recipe |
+| --- | --- | --- | --- |
+| One public file that never changes | `public.https` with a Reader pin | `closed: true` | [one file](references/patterns.md#one-public-file-that-never-changes) |
+| One large public Parquet file whose publisher answers byte ranges | a `parquet.tabular@1.1.0` pin | `columns` | [ranged Parquet](references/patterns.md#one-large-public-parquet-file-read-by-range) |
+| Keyed JSON API paged by page number, whose first page states the page count | `authenticated.https.api_key@2.1.0` with the counted `pagination.parallel` | `total_pages_pointer`, `max_page_bytes`, the gap | [keyed API](references/patterns.md#keyed-json-api-with-pages-fetched-together) |
+| API that takes a date range | `window.request` | `lookback_seconds` | [date range](references/patterns.md#api-that-takes-a-date-range) |
+| Many pages behind one index | `public.https.collection@2.0.0` | the gap | [collection](references/patterns.md#many-pages-behind-one-index) |
+| One request per row of another source | a per-row `request`, with `batch` when a `GET` takes a list of ids | the gap | [per-row](references/patterns.md#one-request-per-row-many-ids-per-request) |
+| Bulk dump beside a windowed source | `reference` | `recheck_seconds` | [reference](references/patterns.md#bulk-dump-beside-a-windowed-source) |
+| Live `wss://` feed | a recorded stream | see [Streams](references/recording-a-stream-venue.md) | none |
+
+[Patterns](references/patterns.md) has one complete recipe per row, with the pace set.
+[Sources](references/sources.md#pace-and-width) has the pace rules and the limits of parallel
+pages. [Readers](references/readers.md) has the Reader pins and the limits of a ranged read.
+
+**Gate 3. Run a bounded test on a scratch table.** Never test on the real table: its first
+succeeded run goes live, truncated or not. For a public CSV that needs no key, run
+`mr-data source inspect`. For anything else follow [the probe rule](references/3-probe-read-a-source-before-committing-to-it.md#the-probe-rule-test-on-a-scratch-table-never-on-the-real-table): copy the drafted
+recipe under another `table.name`, drop `table.search` from the copy, write small budgets into
+it, and take one bounded run.
+A sample's `--max-rows` cuts rows after the fetch. It does not shorten a collection walk, so the
+scratch recipe states a small `pages.max_fetches_per_run`. A per-row request collection also
+states `pages.stop_at_budget: true`, or it fails at that budget. A source whose provider bills
+per call spends real calls on the test, and the real table cannot reuse them. Forecast it and get
+the user's decision before the test. Retire the scratch table with `mr-data demote TABLE_ID` and
+then `mr-data table archive TABLE_ID --confirm DELETE`.
+
+**Gate 4. Forecast, then one full run.** Run
+`mr-data recipe forecast RECIPE_ID --digest DIGEST --json`. Exit 3 means resize before you run.
+Read the registration `warnings`. Then run `--full` once and follow it with `mr-data watch`.
+
+The steps below are numbered 1 to 10. The reference files are numbered by stage, 1 to 9. Steps 1
+and 2 are stage 1. Step 3 is stage 2. Step 4 is stage 3. Step 5 is stages 4 and 5. Steps 6 and 7
+are stage 6. Step 8 is stage 7. Step 9 is stage 8. Step 10 is stage 9. Gates 1 and 2 belong to
+step 5, where the recipe is drafted. Gate 3 needs that draft, so it runs after step 5 and before
+step 6. The stage 3 reference holds its rule. Gate 4 is step 6.
+
 ## Start and agree the brief
 
 1. Run `mr-data whoami --json` once. If authentication needs repair, read
@@ -27,7 +89,10 @@ references. Load the reference for the current stage only, not the entire librar
 3. Settle purpose, row grain, tables, coverage, required sources and update cadence before
    dependent work. In an attended build, ask unresolved material questions in one concise message
    with recommended choices; recording an assumption is not asking the user. Default to one joined,
-   ML-ready table at the agreed grain unless the user requests otherwise. Validate join cardinality
+   ML-ready table at the agreed grain unless the user requests otherwise. Define membership and
+   relationship evidence for the intended use; resolve consequential proxies (for example,
+   institution location versus a person’s origin) rather than silently treating them as equivalent.
+   Validate join cardinality
    and point-in-time correctness; surface incompatible grains rather than silently multiplying rows.
    Ask whether this is a fixed snapshot or an updating dataset, how often updates are needed, and
    whether automatic cadence adjustment is wanted. Existing answers and explicit delegation remain
@@ -60,12 +125,20 @@ references. Load the reference for the current stage only, not the entire librar
    Decide how each source continues before you register it, and whether the table needs a
    declared `rebuild`: read [making a table incremental](references/making-a-table-incremental.md).
    Run `mr-data recipe explain RECIPE.json --json` first. It names every reason the table could
-   not refresh, not only the first. Its `fast_features` names each fast feature the recipe
-   misses, with a one-line fix. Write `table.partition` for any table no window partitions that
+   not refresh, not only the first. Its `fast_features` has eight entries, and each missed one
+   carries a one-line fix. Write `table.partition` for any table no window partitions that
    has a date-like column, `batch` for a per-row GET whose API takes a list of ids, and
    `reference` for a bulk dump beside a window
    ([fast builds](references/making-a-table-incremental.md#fast-builds-tablepartition-batch-and-reference)).
-   Put them in the first revision, because each later change costs one resync. Adopting
+   State the gap, drop a stated `pages.concurrency`, page a keyed API in parallel and pin the
+   ranged Parquet Reader where the entries `pace`, `concurrency`, `parallel_pagination` and
+   `ranged_parquet` say so and the source meets each condition
+   ([fetching fast](references/making-a-table-incremental.md#fetching-fast-pace-width-parallel-pages-and-ranged-parquet)).
+   Put them in the first revision. Registering a revision is free, but building one is a full
+   resync, so each later change costs one resync.
+   On a table that is already live, list the missed entries and ask the user first. Each fix
+   changes the recipe digest, so the next build is a full resync. A paid source is called again
+   for every per-row request whose saved answer no longer applies. Adopting
    `table.partition` on an existing table costs one full resync, so for a paid source ask the
    user before that resync, as the brief and forecast rules require. Before any
    `--full` of a paid collection, run `mr-data recipe forecast RECIPE_ID --digest DIGEST --json`
@@ -75,7 +148,8 @@ references. Load the reference for the current stage only, not the entire librar
    Use `--full` for the requested build: it is progressive by default, with a five-minute
    inspection checkpoint while the same acquisition continues. Choose bounds from measured
    source size; row and byte clamps apply per source, not elapsed time. Use explicit `--sample`
-   only for a deliberate standalone inspection. Preserve the agreed semantics and quality checks.
+   only for a bounded test on a scratch table, as gate 3 says. Preserve the agreed semantics and
+   quality checks.
 7. Follow the returned run with `mr-data watch RUN_ID --json`. Its `source_rates` and
    `mr-data status RUN_ID --json` on a live run give each collection source's rate and time left.
    Quote those numbers rather than doing the arithmetic. Read `run.status`: returning an existing
@@ -119,8 +193,10 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
 
 8. Read `mr-data checks RUN_ID --json`, `mr-data receipt RUN_ID --json` and
    `mr-data peek RUN_ID --json`; use bounded `mr-data query` for agreed quality questions.
-   Verify row grain, joins, requested fields, coverage and every declared check. Successful
-   execution alone is not a verified dataset. `receipt` carries one coverage entry per source --
+   Verify row grain, joins, requested fields, coverage and every declared check. Inspect representative
+   and suspicious rows against source evidence and the agreed definitions; schema and count checks
+   alone cannot establish that the rows mean what the dataset claims. Successful execution alone
+   is not a verified dataset. `receipt` carries one coverage entry per source --
    `bytes_fetched`, `rows_kept`, `rows_available`, `truncated` and the window -- where the run
    record folds all sources into four numbers. That is what each source ACTUALLY weighed and
    carried, and it is the evidence the refresh decision belongs on rather than the ceiling you
@@ -131,7 +207,8 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
    correction into step 10. Read
    [verification](references/7-interrogate-ask-the-run-what-it-actually-delivered.md).
 9. Any truncated source means a preview, not the requested complete dataset. Inspect coverage
-   even when the run succeeded. A null coverage window does not mean all history was acquired.
+   even when the run succeeded. Untruncated acquisition does not prove publisher pagination, nested
+   records or population coverage is complete. A null coverage window does not mean all history was acquired.
    Repair only evidenced faults on the same dataset/table and verify the revision again.
    Record cadence only when authorized and `mr-data recipe readiness --json` reports the current
    revision refresh-ready. A blocked mutable snapshot requires an explicit table resync, not a
@@ -164,6 +241,7 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
 | Live stream recording | [Streams](references/recording-a-stream-venue.md) |
 | Prediction target or leakage | [Prediction labels](references/prediction-labels.md) |
 | How each source continues, `rebuild`, and the cost of a refresh | [Making a table incremental](references/making-a-table-incremental.md) |
+| Testing a recipe before its full, and its first refresh before the schedule | [Recipe test](references/recipe-test.md) |
 | Refresh cadence | [Promotion and cadence](references/promote.md) |
 | Dataset notes and evidence blocks | [Decision records](references/writing-a-decision-record.md) |
 | Search configuration, example queries and readiness | [Search preparation](references/preparing-table-search.md) |

{mostlyright_data-0.29.10 → mostlyright_data-0.29.13}/skills/mr-data-build/references/2-brief-two-to-four-questions-each-with-a-recommended-answer.md

@@ -17,7 +17,9 @@ user already supplied. Always include update behavior when it remains unresolved
   separate tables as an equal default. Check join keys, cardinality, unmatched rows and, for
   prediction, feature availability at prediction time. If a join would mix incompatible grains,
   duplicate examples or leak future information, explain the tradeoff and resolve it before building.
-- **Coverage.** Which entities, geography and date range?
+- **Coverage.** Which entities, geography and date range? Resolve definitions that materially change
+  membership here: a person’s institutional country, origin and current residence are different
+  populations. Ask about the intended population, not which convenient API filter to use.
 - **Updates.** A fixed historical snapshot or an updating dataset? If updating, how often is new
   data needed, and should that schedule stay fixed or adapt to source publication? Recommend a
   cadence from the use case and available source evidence, mark an unverified recommendation as
@@ -27,10 +29,23 @@ user already supplied. Always include update behavior when it remains unresolved
   the agent recommend them? Ask about the consequential fork — official but delayed against broader
   and more recent, or a source that needs a credential — and research the options yourself.
 
-Fields and definitions, quality tradeoffs and delivery constraints are follow-ups, not openers:
+Other field details, quality tradeoffs and delivery constraints are follow-ups:
 ask one when an answer or a source finding exposes it. There is no second brief — this message is
 the one that decides the shape.
 
+**Turn the brief into criteria the result can be checked against.** Record what one row claims,
+what qualifies an entity or relationship, and how repeated observations or versions count. Use the
+existing decision notes and recipe checks; this does not introduce a new approval or schema.
+Resolve only consequential ambiguity and honor supplied answers. For a human collaboration table,
+a software agent credited on a release is not evidence of collaboration with that agent’s vendor;
+the user need not specify this exclusion. Whether software contributions count at all depends on
+their purpose. A release-history dataset can legitimately retain every version.
+
+Use a few included and excluded examples when they help expose a boundary. Carry unresolved
+source limitations into the source comparison rather than redefining the goal around available
+fields. Do not infer a person’s origin from their name or silently replace it with institutional
+country. If the intended population cannot be evidenced, explain the gap and resolve the scope.
+
 **Every question is a pending `clarification` block on the dataset and the same words in chat.**
 The block carries `question` and no `answer`, so the page shows it open; the chat asks it in the
 same words, so the two readers see one question rather than two. The answer REVISES that cell —

{mostlyright_data-0.29.10 → mostlyright_data-0.29.13}/skills/mr-data-build/references/3-probe-read-a-source-before-committing-to-it.md

@@ -79,11 +79,12 @@ refusal to the provider and never bypass it.
 reason codes, only once the discovery lanes and the comparisons are complete — and claim
 `unsupported` only after every typed-refusal chain ends `lawful_routes_exhausted`.
 
-**Reading the bytes is a bounded run, not `mr-data probe`.** `probe SOURCE_ID QUESTION_ID
+**Reading the bytes is a bounded test run, not `mr-data probe`.** `probe SOURCE_ID QUESTION_ID
 --dataset DATASET_ID` takes two identifiers rather than a question in words, and no v4 registration
 receipt returns a question identifier; it also rides the frozen `/v3/sessions` routes, which a
-deployment may have switched off. So the reading path on a hosted workspace is the ordinary loop,
-run early: register the recipe and take one run without `--window`.
+deployment may have switched off. So the reading path on a hosted workspace is
+[the probe rule](#the-probe-rule-test-on-a-scratch-table-never-on-the-real-table) below: one
+bounded run of a scratch table.
 
 **A public CSV can also be looked at before anything is registered.** `mr-data source inspect` reads
 the start of one HTTPS file that needs no key and answers with what it saw, with no recipe and no
@@ -107,7 +108,7 @@ start of the file and not the file. Its `evidence` carries the `format`, the `co
 not an answer yet, so read it back again later. A status of `truncated` is an answer that stopped at
 a limit: keep its limits and its evidence with whatever you record, and never call it complete. The
 command takes no key and no Reader options, so a source that needs either, or that is not a CSV, is
-read by the run below instead. So is any source that answers `capability_unavailable`, which means
+read by the test run below instead. So is any source that answers `capability_unavailable`, which means
 this deployment cannot inspect it: do not retry it, and do not guess in its place.
 
 **Write down what each look found.** An inspection is not a run, and nothing on the page shows what
@@ -128,33 +129,79 @@ observed_at, stage_m and quality_flag.
 EOF
 ```
 
-**Both obey the ordinary gate.** Registering needs the brief answered or delegated, so the run
-happens AFTER stage 2's answers arrive — or at once under a delegation — and an inspection, which
-writes a question and a source onto the dataset, waits for the same answers. What continues while
+**Both obey the ordinary gate.** Registering needs the brief answered or delegated. So the test
+run happens after stage 2's answers arrive, or at once under a delegation. An inspection writes a
+question and a source onto the dataset, so it waits for the same answers. What continues while
 you wait is research that submits nothing: documentation, publisher comparison, catalog.
 
+#### The probe rule: test on a scratch table, never on the real table
+
+Test a recipe on a scratch table with small budgets. Never test on the real table. A table's
+first succeeded run goes live, truncated or not. A test run on the real table would serve a cut
+table on the dataset page. This stage starts no full build of the real table. Stage 6 starts
+that one build. The test needs a drafted recipe, so run it once stage 5 has a draft and before
+stage 6.
+
+1. Copy the recipe document. Give the copy its own `table.name`. Keep the same `dataset.id`.
+   Drop `table.search` from the copy, so the test prepares no search index.
+2. Write small budgets into the copy, as the list below says.
+3. Register the copy with `mr-data recipe SCRATCH.json --json`. It returns a recipe id, a digest
+   and a table id of its own.
+4. Take one bounded run and read it back.
+5. Retire the scratch table. A scratch table goes live too, so withdraw it first.
+
 ```sh
-mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json
-mr-data status RUN --inspection --json  # inspect while the same full acquisition continues
+mr-data run --recipe SCRATCH_RECIPE_ID --digest SCRATCH_DIGEST --sample --max-rows 2000 --json
 mr-data peek RUN --json      # the columns and types the run sealed
 mr-data query RUN "select count(*) from run_table" --json
 mr-data receipt RUN --json   # what was fetched and what those bytes hashed to
+mr-data status RUN --diagnose --json  # progress, plan counts and how each attempt ended
+mr-data demote SCRATCH_TABLE_ID --json
+mr-data table archive SCRATCH_TABLE_ID --confirm DELETE --json
 ```
 
-**Inspect the full build while it runs.** Its progressive checkpoint appears after about five
-minutes without stopping acquisition. Use an explicit `--sample` only for a deliberate standalone
-experiment: a row ceiling can truncate output after every collection page was fetched and does
-not promise a short acquisition. Never derive source size from a truncated sample's ceiling.
-
-**A reading run that came back whole IS the build, so stage 6 adopts it rather than repeating
-it.** Untruncated everywhere is the build, whichever stage started the run: keep this `run_id`.
-The run meets Studio's ordinary size confirmation before acquisition when it is large (a size
-check, not a charge); its checkpoint requires no
-second approval or second full build.
-
-That is stages 5 through 7 run once to settle the shape. Where the deployment does answer a source
-inspection or a probe, keep its status, reason code and exit value internal and describe only what
-it settled.
+**Know what a sample bounds and what it does not.**
+
+- `--max-rows` cuts each source's rows after that source is fetched and decoded. It does not
+  shorten a collection walk. Every page the budgets admit is still fetched.
+- A collection's walk is bounded by `pages.max_fetches_per_run` and `discovery.max_requests`.
+  Write both small in the scratch recipe. A listing collection that spends its fetch budget
+  still succeeds and reports `complete: false`.
+- A per-row request collection fails with `ROW_REQUESTS_EXCEED_BUDGET` at its fetch budget.
+  State `pages.stop_at_budget: true` beside the small budget in the scratch recipe. The run
+  then seals the rows it answered and reports `complete: false`.
+- A row clamp does not truncate a per-row request collection. Answers past the clamp fail the
+  run with `ROW_REQUESTS_EXCEED_BUDGET`. Keep `--max-rows` above the rows its budget can answer.
+- A document projection source cannot run under a row clamp. The run fails with
+  `CLAMP_UNSUPPORTED_FORMAT`. Test it with a full run of the scratch table:
+  `mr-data run --recipe SCRATCH_RECIPE_ID --digest SCRATCH_DIGEST --full --json`.
+- A source with a request window is read for its latest `lookback_seconds` only.
+  [Stage 6](6-build-one-run-sized-to-acquire-every-measured-source-whole.md) has the exact days.
+- `--max-source-bytes` and `limits.max_source_bytes` refuse in every mode. They never truncate.
+- A sample must state a clamp. Studio refuses a sample with none as `CLAMPS_REQUIRED`. This
+  client refuses it first as `THIN_REQUEST_INVALID`. Give `--max-rows`, `--max-source-bytes` or
+  `--window`.
+
+**A paid source spends real calls on the test.** A per-row request whose provider bills per call
+makes a paid call for every request the scratch run sends. The real table cannot reuse those answers,
+because saved answers belong to the table that holds them. Run `mr-data recipe forecast` on the
+scratch recipe and get the user's decision before the test.
+
+**Retire a scratch table by what happened to it.** A scratch table whose run succeeded is live:
+`demote` it, then `table archive` it. One whose run failed never went live, and `demote` answers
+`TABLE_NOT_PROMOTED` for it. Go straight to `table archive`. To test a revision, register it
+under the same scratch `table.name` before you archive. After the archive the name is free
+again, and registering it makes a new table.
+
+**Keep the activity line truthful during the test.** The page shows the scratch table going live
+as a table event. That event does not write your activity line. Say in the line that you are
+testing on a scratch table, and do not report `--phase done` for it.
+
+Never derive a source's size from a truncated sample's ceiling. A check on the whole table, such
+as a row floor, can fail on a sample of a sound recipe. Read the registration `warnings` first.
+
+Where the deployment does answer a source inspection or a probe, keep its status, reason code and
+exit value internal and describe only what it settled.
 
 **An acquisition's answer is evidence, and cheap to reuse.** Acquisitions are cacheable by
 source, query digest and window, so a fetch feeds a later build in the same warm session without

{mostlyright_data-0.29.10 → mostlyright_data-0.29.13}/skills/mr-data-build/references/6-build-one-run-sized-to-acquire-every-measured-source-whole.md

@@ -1,13 +1,17 @@
 ### 6. Build — one run, sized to acquire every measured source whole
 
-**Stage 3's reading run is usually already this run.** Where it came back with
-`coverage.truncated` false everywhere, that run is the build: keep it, present it at stage 7, start
-nothing here. Re-running the coordinate replays it; a different ceiling buys only another
-acquisition.
-
-**Otherwise start one progressive full build.** Use the registered recipe and measured bounds
+**Stage 3 tested the recipe on a scratch table. This stage starts the one full build of the real
+table.** Do not test on the real table: its first succeeded run goes live, truncated or not.
+[The probe rule](3-probe-read-a-source-before-committing-to-it.md#the-probe-rule-test-on-a-scratch-table-never-on-the-real-table)
+has the scratch-table steps. Before the full build, run
+`mr-data recipe forecast RECIPE_ID --digest RECIPE_DIGEST --json`. Exit 3 means the run as declared
+would outlast one attempt or overrun a quota, so resize it first. Read the registration
+`warnings` too.
+
+**Then start one progressive full build.** Use the registered recipe and measured bounds
 for the requested coverage. Every full run is progressive by default; agents do not choose
-between progressive and sample-first acquisition.
+between progressive and sample-first acquisition. Re-running the same coordinate replays its
+stored result. A different ceiling buys only another acquisition.
 
 ```sh
 mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json
@@ -19,11 +23,12 @@ limit. A short build can finish before a checkpoint is needed. Do not submit ano
 inspecting it. A required size confirmation happens before acquisition; it is not a charge. The
 older `--progressive` flag is accepted for compatibility but is unnecessary.
 
-**A standalone sample is an explicit inspection choice.** Use `--sample --max-rows 2000` only
-when a separate bounded experiment is needed. Rows are clamped after acquisition, so a collection
-may still fetch every page and take longer than five minutes. For an already authorized full
-build, use the progressive checkpoint. A truncated sample is a preview: report its actual
-coverage and the full build's proposed coverage before requesting any missing authorization.
+**A standalone sample belongs on a scratch table.** Use `--sample --max-rows 2000` only for a
+separate bounded experiment, and run it as the probe rule says. Rows are clamped after
+acquisition, so a collection still fetches every page its budgets admit and can take longer than
+five minutes. For an already authorized full build, use the progressive checkpoint. A truncated
+sample is a preview: report its actual coverage and the full build's proposed coverage before
+requesting any missing authorization.
 
 > The preview stopped at 2,000 rows on `asos_observations`, which is the ceiling rather than the
 > end of the feed; the other three sources came back whole. The full build would cover 2000 to 2026
@@ -208,6 +213,18 @@ held answer prints back as `confirm_command`. Any authenticated surface of the s
 may settle it. Never present it to the user as a cost or a payment. `mr-data run --cancel RUN_ID`
 stops a run that is queued or running.
 
+**A full tests its recipe first.** `mr-data run --full` runs `mr-data recipe test` with Studio's
+default bounds, waits up to five minutes for its report, and then starts the full resuming from
+the test's pages, so nothing is fetched twice. A blocking problem, a failed or held test, or a test
+still running after five minutes keeps the full from starting: the status is
+`run_stopped_by_test`, `run_test_awaiting_confirmation` or `run_test_still_running`, `run_id` names
+the test, and `next` ends with the same full plus `--skip-test`. Repair the recipe rather than
+skipping the test, unless the user decides the problem does not matter; a test still running is
+picked back up by the same command. A test that failed in the room (`room_fault: true`) is tested
+again by the same command. Studio admits the test as the full it stands in front of, so a large
+full can be held for a size confirmation twice, once for the test and once for the full. See
+[recipe test](recipe-test.md).
+
 Do not treat the accepted command vocabulary as execution evidence. Before proposing a schedule,
 classify every source under one truthful continuation strategy and record the supporting evidence,
 using [making a table incremental](making-a-table-incremental.md).

{mostlyright_data-0.29.10 → mostlyright_data-0.29.13}/skills/mr-data-build/references/7-interrogate-ask-the-run-what-it-actually-delivered.md

@@ -9,6 +9,35 @@ mr-data checks RUN --json                     # every check the recipe declared
 mr-data receipt RUN --json                    # sources, digests, coverage, snapshot members
 ```
 
+#### Check meaning against the brief
+
+Inspect actual rows beyond the first preview: sample across material source/entity/type groups
+and inspect anomalies such as unusually prolific pairs, repeated titles or concentrated counts.
+Trace selected inclusions and suspected exclusions to source records. Record the version, sample
+selection, rows inspected, evidence and unresolved limitations in the dataset notes. A sample is
+not proof of population-wide accuracy; passing declared checks only proves their assertions.
+
+Choose checks from the dataset’s claims. For a human co-authorship table, inspect counterpart
+identity and affiliation evidence, distinct people rather than self-edges, work types, and repeated
+release families. A bot credit does not establish a human relationship with its vendor. Use
+source-supported identities or narrowly justified matches; a name substring is not a reliable bot
+classifier. Preserve legitimate human names. Treat software releases and version deduplication
+according to the agreed grain: do not ban software or collapse versions in a release-history task.
+Encode evidenced, repeatable failures as supported recipe checks where possible, then verify the
+repair. Do not relax checks or prune legitimate rows to make them pass.
+
+Separate acquisition limits from source and population coverage. Read publisher totals and
+continuation tokens, nested-list caps or completeness flags, and join losses where applicable.
+`coverage.truncated: false` does not prove all publisher pages or all authors inside each record
+were returned. Missing completeness evidence means unknown. A nested cap calls for another
+supported acquisition path or a stated limitation, not silent exclusion of every large record.
+
+Known examples can reveal omissions, but compare their eligibility against the same population,
+dates and relationship definition first. A broader benchmark is useful for discussing scope; its
+size is not a valid recall denominator for a narrower dataset. Investigate eligible misses and
+preserve out-of-scope examples as such. Do not manufacture a recall or accuracy percentage from
+an unvalidated list or an anomaly-focused sample.
+
 **The last of those answers `THIN_NO_MATCHING_ARTIFACT` on a run that sealed no receipt**, naming
 what it sealed instead. That is the ordinary shape of a recipe with one `request`-windowed source
 rather than a fault. Read `mr-data status RUN --json` for its coverage, as
@@ -57,7 +86,7 @@ instead, and wait through the window the refusal names rather than retrying in a
 
 **A query sent straight after another may find no worker to start.** The worker behind the query
 before is still winding down, and Studio fails the new session at once with
-`RESEARCH_SESSION_WORKER_UNAVAILABLE`. `mr-data query` waits 8 seconds and opens one more session,
+`RESEARCH_SESSION_WORKER_UNAVAILABLE`. `mr-data query` waits 8 to 12 seconds and opens one more session,
 once. The answer then names the first session under `session.retried_after`. If the second has no
 worker either, the command refuses `THIN_RESEARCH_SESSION_ENDED` and names both sessions, each with
 its state, close reason and failure code. Nothing was queued, so running the same command again

mostlyright_data-0.29.13/skills/mr-data-build/references/8-fix-revise-the-document-and-register-it-again.md

@@ -0,0 +1,16 @@
+### 8. Fix — revise the document and register it again
+
+Change the document and send it again. Same dataset, same table, new digest:
+
+```sh
+mr-data recipe RECIPE.json --json
+```
+
+**Registering a revision is free and abandons nothing. Building it costs one resync.**
+Registration starts no run. Table identity is stable across revisions: no `_v2.._vN` renaming, no
+proposal wedged on an id somebody has to reclaim, no cleanup after a failed run. The revision has
+a new digest, so the sealed predecessor no longer matches it. Its next build reads every source
+again. A per-row request collection still reuses the answers the live version holds while its
+request, Reader and binding stay the same. Put every change in one revision so the table pays
+once. Forecast a source whose provider bills per call, and ask the user before that resync.
+Build again, interrogate again, until the checks and coverage say what you need.

{mostlyright_data-0.29.10 → mostlyright_data-0.29.13}/skills/mr-data-build/references/9-present-only-what-survived-inspection-with-caveats.md

@@ -1,23 +1,25 @@
 ### 9. Present — only what survived inspection, with caveats
 
 Report what you inspected, and nothing you did not. Nothing else checks this run: `verified` is
-what you established by reading it, not a word another worker said. Three facts carry whatever
-you say and all three are on the run: how many rows you looked at, what the declared checks
-reported, and whether `coverage.truncated` was true anywhere. A null coverage window means the
-acquisition was unwindowed rather than that it covered all time. Never present a sampled row as a
+what you established by reading it, not a word another worker said. Report how many rows you
+inspected and how you selected them, what the declared checks reported, and whether `coverage.truncated` was true anywhere. Include the source evidence and
+limitations behind membership, relationships and population coverage. A null coverage window means
+the acquisition was unwindowed rather than that it covered all time. Never present a sampled row as a
 whole-population value, never call a run verified because it succeeded, and never report coverage
-you did not read off `coverage`.
+you did not establish from the run and source evidence.
 
 **Read `truncated` first, because it decides which of two endings this is.** When it is false on
-every coverage entry, nothing was cut and the table this run sealed is the table a full run would
-have sealed. When it is true anywhere, what you have is a shape rather than an answer.
+every coverage entry, the acquisition reports no transport truncation. This alone does not
+establish publisher pagination, nested-list completeness or coverage of the intended population.
+When it is true anywhere, what you have is a shape rather than an answer.
 
-**Untruncated everywhere is the build. Present it as one.** The mode a run started in is not what
-makes a table finished, and calling a complete table "a preview" tells somebody their dataset does
-not exist yet when it does. Say what was built, and link the page:
+**The real table's full run, untruncated everywhere, is the build. Present it as one.** A
+scratch-table test is never the build, whatever its coverage says. Calling a complete table "a
+preview" tells somebody their dataset does not exist yet when it does. Say what was built, and link the page:
 
-> Built 43,812 rows from four sources, every source acquired whole and nothing truncated. All four
-> declared checks passed. Live at <link>. It refreshes every 6 hours, because the publisher posts
+> Built 43,812 rows from four sources with no reported acquisition truncation. All four declared
+> checks passed. Inspected 40 rows across the four sources; publisher totals match acquired records.
+> Live at <link>. It refreshes every 6 hours, because the publisher posts
 > hourly and the bytes appear about forty minutes late.
 
 Say plainly that it is live: the first succeeded run of a table goes live on its own, so there is
@@ -39,20 +41,24 @@ refresh delegation described above. Stage 6 carries the run commands.
 the dataset page, so before acting on this recipe again read `mr-data runs --json --mode full` and,
 for any run newer than the preview that you did not start, `mr-data status RUN_ID --json` to see
 whether it names this recipe and digest. One that does is theirs, and its state says what to do.
-`queued`, `running` or `succeeded` is the full build: adopt it as if you had started it — write its
-run cell, `mr-data watch RUN_ID --json` if it has not ended, then stage 7. `awaiting_confirmation`
+`queued`, `running` or `succeeded` is the full build of the real table: adopt it as if you had
+started it. Write its run cell, `mr-data watch RUN_ID --json` if it has not ended, then stage 7. A
+run of a scratch table is never adopted as the build. `awaiting_confirmation`
 is the size gate (not a charge), which the same command settles with `--confirm`.
 `awaiting_sample_approval` is stage 6's held full: read its `sample_run_id`, interrogate that
 preview, and release it with `mr-data run --approve-full`. `failed` is the full build too: read its
 triple and structured recovery. A known retry case can use `mr-data run --retry RUN_ID`; otherwise
 preserve the document until the evidence identifies a repair. A bare run of the same coordinate
 replays that failure. `cancelled` means they withdrew the decision, so say so and ask nothing
-further unless they raise it. Ask, then give the turn back; do not hold a stream open waiting for
+further unless they raise it. The exception is a run whose `cancel_reason` is
+`preflight_complete`, or that carries `preflight`: that is a recipe test, which always ends
+`cancelled` with its report and never becomes a version. It is not the full build and not a
+withdrawn decision; read its `preflight_report` as [recipe test](recipe-test.md) describes. Ask, then give the turn back; do not hold a stream open waiting for
 either answer.
 
-Then interrogate that run as stage 7 says, and present it with the preview's caveats removed and
-its own read off `coverage`. Do not start a full build because the preview looked good or because a
-dataset was asked for — the decision is whether THIS is the recipe to build.
+Then interrogate that run as stage 7 says. Remove only the preview caveats the new evidence
+resolves, and retain unresolved source limitations. Do not start a full build because the preview
+looked good or because a dataset was asked for — the decision is whether THIS is the recipe to build.
 
 **Either way, offer the bytes.** A sealed table is downloadable whatever mode produced it —
 `mr-data download RUN --output DIR --json` — so somebody who wants to look before agreeing to