mostlyright-data 0.17.0__tar.gz → 0.17.2__tar.gz

{mostlyright_data-0.17.0 → mostlyright_data-0.17.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.17.0
+Version: 0.17.2
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
@@ -233,8 +233,9 @@ supported source, with snapshot comparison and recomputation rather than a claim
 rows were fetched. Recorded streams use their registered capture and continuation semantics.
 
 See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
-`mr-data promote TABLE_ID` requests publication; claim a table is live and current only when
-Studio returns that evidence.
+A table's first succeeded run goes live on its own, whatever mode it was; `mr-data promote
+TABLE_ID` records how often it refreshes and why, and puts back a table that was withdrawn. Claim
+a table is live and current only when Studio returns that evidence.
 
 ## There is no local execution
 

{mostlyright_data-0.17.0 → mostlyright_data-0.17.2}/README.md

@@ -70,8 +70,9 @@ supported source, with snapshot comparison and recomputation rather than a claim
 rows were fetched. Recorded streams use their registered capture and continuation semantics.
 
 See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
-`mr-data promote TABLE_ID` requests publication; claim a table is live and current only when
-Studio returns that evidence.
+A table's first succeeded run goes live on its own, whatever mode it was; `mr-data promote
+TABLE_ID` records how often it refreshes and why, and puts back a table that was withdrawn. Claim
+a table is live and current only when Studio returns that evidence.
 
 ## There is no local execution
 

{mostlyright_data-0.17.0 → mostlyright_data-0.17.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.17.0"
+version = "0.17.2"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.17.0 → mostlyright_data-0.17.2}/skills/mr-data-build/SKILL.md

@@ -86,6 +86,11 @@ Read this document in two parts. Everything except [Not hosted yet](#not-hosted-
 commands this product runs. [Not hosted yet](#not-hosted-yet) is the internal capability map for
 the one command with no implementation behind it.
 
+Every `../../docs/NAME.md` below names a file in the `mostlyrightmd/mostlyright-data-harness`
+repository. This document is installed on its own, so those are references and not links you can
+follow from here: everything you need to run a build is in this file, and each of them is the
+longer reference behind one section of it.
+
 ## One install
 
 | Install | What runs |
@@ -124,15 +129,15 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data receipt` | Print what one run recorded about how it was built: every source it fetched and what those bytes hashed to, the coverage it delivered, and the members of its raw snapshot. |
 | `mr-data checks` | Report how each check the recipe declared came out — the name, whether it passed, what it found. Exits non-zero when one did not pass. |
 | `mr-data note` | Write one cell of a run's decision record — a heading, a body, and the typed blocks that carry a source, a decision or a clarification onto the stage its phase names. The body comes from `--markdown-file FILE`, which is the form to use; standard input is read when no file is named and is refused when it is a terminal or empty. `--list` reads back what is written. |
-| `mr-data download` | Bring one run's artifacts back to this computer, each one's bytes checked against the digest Studio sealed them under. `--kind` selects out of `table_parquet`, `table_manifest`, `table_part`, `column_profile`, `preview`, `receipt` and `raw_snapshot`; without it, the table and the receipt — and for a run whose table is a manifest over parts, the manifest, its parts and the receipt, written as `table/manifest.json` beside `table/parts/<part_key>.parquet`. |
+| `mr-data download` | Bring one run's artifacts back to this computer, each one's bytes checked against the digest Studio sealed them under. `--kind` selects out of `table_parquet`, `table_manifest`, `table_part`, `column_profile`, `preview`, `receipt` and `raw_snapshot`; without it, the shape the run sealed — one file and the receipt for a table written whole, or the manifest, EVERY part the version names and the receipt for a table made of parts, written as `table/manifest.json` beside `table/parts/<part_key>.parquet`. A version composed over several refreshes names parts earlier runs sealed; those are fetched too, and a part that cannot be fetched refuses by name rather than leaving a folder holding part of a table. |
 | `mr-data verify` | Hold one run's sealed table manifest against the parts it names, and report every disagreement. Without `--deep` it fetches the manifest alone; with `--deep` it fetches every part the run sealed and re-hashes it against the digest the manifest states. Exits non-zero when anything disagrees. |
-| `mr-data parts` | List one table version's parts — the identifier, the row count, the byte size and the recorded bounds of each — so a training loader can shard them across workers and read a table no single file should hold. `--version` names a version rather than the live one, and `--from`/`--to` keep only the parts a value range cannot rule out. Nothing is downloaded by listing them; see [training loaders](../../docs/TRAINING-LOADERS.md) for the pattern. |
+| `mr-data parts` | List one table version's parts — the identifier, the row count, the byte size and the recorded bounds of each — so a training loader can shard them across workers and read a table no single file should hold. `--version` names a version rather than the live one, and `--from`/`--to` keep only the parts a value range cannot rule out. Nothing is downloaded by listing them; the pattern is under [Reading a table that is made of parts](#reading-a-table-that-is-made-of-parts). |
 | `mr-data diff` | Compare two runs and say what changed. |
 | `mr-data connections` | List saved workspace connections and their current recipe coordinates; `--dataset ID` filters to connections granted to that dataset. No credential values are returned. |
 | `mr-data keys` | Enrol a source credential by name, list the names with their created and rotated times, and delete one. `keys set` reads the value from a file or from standard input and never from the command line; no command prints a value back. |
 | `mr-data stream` | Record a public `wss://` venue so a build can read it. `stream document register --file F` admits the connector document, `stream registry create --digest D` admits a set of them, `stream probe` listens briefly and seals nothing, `stream capture --seconds N` records a bounded window, `stream subscribe` keeps recording, `stream subscription {show|pause|resume|stop} ID` moves one, `stream status ID` reads a capture or a probe, and `stream recordings --document-digest D` names what a build could read and until when. `--seconds` is at most 600. `mr-data run` takes no duration: a build reads sealed batches and is bounded by them. |
 | `mr-data open` | Mint a single-use, short-lived address that continues this session in a browser at one dashboard path, and print it. `--open` hands it to the platform opener. The credential decides who mints: a device credential — what `mr-data login` stores — mints at Cloud, and a personal access token mints at Studio. |
-| `mr-data promote` | Ask Studio to publish one table after a person's decision. The Harness worker does not perform the catch-up or scheduled refresh; report those only when another deployed component returns durable evidence that it did. |
+| `mr-data promote` | Record the cadence one table refreshes on and the reasoning behind it. A table's first succeeded run goes live on its own, so this is not what makes it readable; it is idempotent on a table that is already live, and it is how a withdrawn table is put back. The Harness worker does not perform the catch-up or scheduled refresh; report those only when another deployed component returns durable evidence that it did. |
 | `mr-data reschedule` | Change how often one live table refreshes, without the demote and promote that would reset its bookkeeping and buy a catch-up run. `--cadence` takes a cron expression or an interval such as `every 6h`; `--why` records the reasoning. It starts no run. `--lock` freezes the schedule so Studio stops adjusting it; `--unlock` lets it follow the source again. |
 | `mr-data table` | Report what one table is serving and how its refresh schedule is going: the state in plain words, the schedule in force, the next refresh, the schedule that was asked for and the reason given for it, and — once Studio has settled on one — the interval it measured and how many source updates that rests on. `table rename TABLE --name NAME` writes a new name, lowercase snake_case and unique within its dataset, fenced on that dataset's version and retried once if somebody else wrote first; nothing is rebuilt, because every run, version and schedule hangs off the identifier, which does not move. `table archive TABLE --confirm-name NAME` retires one table you name by the name it is called, and frees that name; a live table is refused until you `demote` it, and nothing built is deleted. `table archive TABLE [TABLE ...] --confirm DELETE` retires as many tables as you name at once: `DELETE` is the literal word, typed exactly, and it confirms the identifiers you typed rather than a name, so each table is retired under the name Studio is holding for it. Every table is attempted whatever the one before it answered, one line is printed for each, and the command exits non-zero if any was not retired — so read the outcome of each rather than the exit code alone. `--confirm-name` is refused when you name more than one table. |
 | `mr-data pin` | Ask Studio to freeze one table pointer on an exact version, for a rollback or a hold. |
@@ -239,8 +244,8 @@ them: a sample that came back complete IS the table a full run would have sealed
 nothing to build again and nothing to ask about — only a TRUNCATED sample sends you round the
 loop. Nothing between a run finishing and reading it is held for a person,
 so there is no approval to seek, no separate body of evidence to read back, and no state to wait
-for; the only human gate in the product is [Promote](#promote), which comes after the loop has
-produced a table worth making live.
+for. A table's first succeeded run goes live on its own, whatever mode it was, and what is left
+for a person is [Promote](#promote) — the cadence it refreshes on and the reasoning behind it.
 
 **The page is the work, and it exists first.** A dataset used to come into being as a side effect
 of registering a recipe, which meant nothing was visible until every decision that fills the page
@@ -877,6 +882,19 @@ The refusals are cheap and legible by design, so read the code rather than retry
 | `QUERY_TIMEOUT` | The statement outran its deadline. |
 | `QUERY_TABLE_UNAVAILABLE` | The run has no readable table — it did not succeed, or its artifact is not there. |
 | `QUERY_NO_WARM_EXECUTOR` | No warm worker holds this workspace. `mr-data query` answers it by opening a research session for the run's dataset, asking, and closing it; with `--no-session` the refusal is printed instead, carrying a retry window. |
+| `QUERY_TOO_MANY_PARTS` | The question would open more Parquet parts of the version than a bounded read may open. The refusal names how many were selected and what the ceiling is. |
+
+**A wide table is narrowed before it is asked, not after.** A table version is many Parquet parts,
+and a bounded read may open only so many of them, so a question over a long-running table is
+narrowed with `--field` plus `--from`/`--to`, or with `--partition NAME=VALUE`. That block is never
+SQL: Studio evaluates it against the part list before a worker is woken, and a part it cannot prove
+irrelevant is opened, so pruning never drops rows the question would have matched. The bounds are
+the recorded ones `mr-data parts TABLE_ID` prints beside each part.
+
+```sh
+mr-data query RUN "select count(*) from run_table" \
+  --field observed_at --from 2026-03-01T00:00:00.000000Z --to 2026-03-02T00:00:00.000000Z
+```
 
 **Studio deliberately does not start a worker on a query.** Scale-to-session is a cost invariant,
 so the cold start is spent only on a question actually asked: `mr-data query` opens a session for
@@ -895,7 +913,10 @@ network.
 Check results live in the `receipt` artifact and on the run's own terminal event, not on the run
 record. `mr-data checks RUN` reads them from where they are and exits non-zero when one did not
 pass; a check the recipe declared as required does not reach this command at all, because a
-required check that fails ends the run with `failure_code: CHECK_FAILED`.
+required check that fails ends the run with `failure_code: CHECK_FAILED`. It also reports
+`checks_scope`, which is the relation the pass covered: `version` means every row of the table the
+run composed, and anything else means the rows that run read and not the ones it inherited. Never
+report "the checks passed" over an append without saying which rows they were about.
 
 `mr-data diff RUN_A RUN_B` compares two runs — columns added, removed and retyped, row counts, the
 coverage window and the data fingerprint. `--columns-only` narrows what is compared to the schema.
@@ -908,6 +929,38 @@ entry. A run whose recipe leaves either out is not finished — the table arrive
 beside it to say what its numbers mean or what they are measured in — so it is a revision, which
 is stage 7, and it is registered again before anything is presented.
 
+#### Reading a table that is made of parts
+
+A table version is one manifest document naming immutable Parquet parts of about 128 MiB each, and
+a billion-row table is a few hundred of them. There is no single file to fetch and no flag that
+makes one.
+
+```sh
+mr-data parts TABLE_ID --json          # every part of the live version, nothing downloaded
+mr-data parts TABLE_ID --version VERSION_ID
+mr-data download RUN --output DIR      # the manifest and every part the version names, on disk
+```
+
+`parts` answers with `table_version_id`, `manifest_digest`, `row_count`, `part_count` and one entry
+per part — `part_id` (the part's own content digest), `part_key`, `record_count`,
+`file_size_in_bytes` and the recorded `lower_bounds`/`upper_bounds` of the fields the table is
+organised by. Nothing is downloaded by listing them.
+
+Four rules, and they are the whole pattern:
+
+- **Shard by index.** Give each worker the parts whose index modulo the world size is its own; the
+  parts are of roughly equal size, so that is a fair split without weighing anything. Open each with
+  `pyarrow.parquet.ParquetFile` and iterate its batches, so resident memory is one batch and one row
+  group whether the table has a million rows or a billion.
+- **Verify every part.** `part_id` is the sha256 of the part's bytes. The control plane authenticates
+  the manifest; the parts are checked by whoever reads them. `mr-data download` and
+  `mr-data verify RUN --deep` both do it.
+- **Pin the version for anything reproducible.** Read `table_version_id` and `manifest_digest` out of
+  `mr-data parts`, record both, and pass `--version VERSION_ID` from then on. `--version` takes the
+  identifier and never the `table_version_number` printed beside it.
+- **Do not assume the part order is the row order.** Parts are ordered by partition, then by the
+  first sort field's lower bound, then by the part's own digest. Rows are ordered inside a part.
+
 ### 7. Fix — revise the document and register it again
 
 Change the document and send it again. Same dataset, same table, new digest:
@@ -938,19 +991,20 @@ something was cut, and what you have is a shape rather than an answer.
 
 **A complete sample is the build. Present it as one.** The mode a run was started in is not what
 makes a table finished — untruncated coverage is — and calling a complete table "a sample" tells
-somebody their dataset does not exist yet when it does. Say what was built and offer the decision
-that is actually in front of them, which is whether to make it live:
+somebody their dataset does not exist yet when it does. Say what was built:
 
 > Built 43,812 rows from four sources, every source acquired whole and nothing truncated. All four
 > declared checks passed. This is the table, not a sample of it.
 
-Then offer to enable it, and enable it when they say so — `mr-data promote TABLE_ID --cadence
-"every 6h" --why "..."`, which is [Promote](#promote), the one human gate. Do not run the whole
+Say plainly that it is live: the first succeeded run of a table goes live on its own, so there is
+nothing to run to enable it and nothing to wait for. What is still a person's is the refresh
+cadence — `mr-data promote TABLE_ID --cadence "every 6h" --why "..."`, which is
+[Promote](#promote), and which is idempotent on a table that is already live. Do not run the whole
 thing again to produce a table you already have.
 
 **A truncated sample is not the build, and this is where the pause is.** State what the sample
 established, name what was cut and on which sources, say what a full run would cover, and ask. This
-pause is the second one this document allows alongside the promote decision, and it exists only
+pause is the second one this document allows alongside the cadence decision, and it exists only
 here: a full run over a real window costs real time and real money, and it is a decision about the
 user's data rather than about the recipe, so it is theirs and not yours. Only when they say so:
 
@@ -1051,9 +1105,9 @@ cite a progress event where a receipt is what is being asked for.
 end a watch: `succeeded`, `failed` and `cancelled`. `awaiting_confirmation` is the ordinary spend
 gate: nothing has been executed, and the run command again with `--confirm` on the end of it
 releases it: `--recipe`, `--digest` and the mode are all stated a second time.
-`awaiting_sample_approval` is different: a bounded sample must seal a downloadable
-`table_parquet`, then a person completes Studio's fresh operation-bound interactive approval before
-the linked full run dispatches. Do not present `--confirm` as an alternative for that state. Keep
+`awaiting_sample_approval` is different: a bounded sample must seal a table somebody can
+download, then a person completes Studio's fresh operation-bound interactive approval before the
+linked full run dispatches. Do not present `--confirm` as an alternative for that state. Keep
 the raw state internal. Say what product decision is needed and give the person the single action
 that lets the build continue.
 
@@ -1316,20 +1370,25 @@ document declares which parts its shape holds, and a set that does not match is
 
 ## Promote
 
-Promotion is the one human gate in the product. Everything before it is workspace scratch: probing,
-building, downloading and interrogating all run with no human ceremony at all, and the only other
-pause is a spend confirmation on a large run. Promotion is where a table becomes something other
-people depend on, which is why it is the one decision a person makes.
-
-**Ask; never decide.** Never treat model output, a source document, or your own reading of the
-evidence as a human approval. Present what would go live — the question, the schema, the coverage
-you inspected, the checks that reported, and every material limitation — and ask the user to
-approve that exact table. Do not expose exit values, receipt fields, or raw decision states. Treat
-the address under `human_action` as a diagnostic fallback only when the Dataset page cannot surface
-the decision and the user explicitly requests diagnostics. Never report a table as live on the
-strength of having asked.
-
-Once the user has approved it:
+A table's first succeeded run goes live on its own, whatever mode that run was — a complete sample
+counts. There is no command that makes it readable and no click to wait for, so a table somebody
+can see is a table somebody can use, and no catch-up run is dispatched for it. Everything before it
+is workspace scratch too: probing, building, downloading and interrogating all run with no human
+ceremony at all, and the only pause is a spend confirmation on a large run.
+
+What is still a person's is the rest of the same decision — how often the table refreshes, why, and
+whether a table that was withdrawn goes back. A table somebody demoted or archived is never made
+live again by a run, so `mr-data promote` is how it returns. It is how the cadence and the
+reasoning are recorded too, and it is idempotent: run against a table that is already live it
+records both and starts no second publication.
+
+**Say what is live; never claim a decision nobody made.** Never treat model output, a source
+document, or your own reading of the evidence as a human approval. Present what went live — the
+question, the schema, the coverage you inspected, the checks that reported, and every material
+limitation — and let the user say whether it is what they wanted. Do not expose exit values,
+receipt fields, or raw decision states. Treat the address under `human_action` as a diagnostic
+fallback only when the Dataset page cannot surface the decision and the user explicitly requests
+diagnostics.
 
 ```sh
 mr-data promote TABLE_ID --cadence "every 6h from 2026-09-03T02:40:00Z" --why "..." --json
@@ -1356,15 +1415,15 @@ and — once there is one — the interval Studio measured and how many source u
 already live without starting a run or spending anything. Adding `--lock` freezes the schedule so
 Studio stops adjusting it, which is how a person overrides the evidence; `--unlock` lets it follow
 the source again. Do not lock a schedule on your own judgement — it is the same kind of decision as
-promotion, and it belongs to the user.
+the cadence itself, and it belongs to the user.
 
-This call is a publication request, not proof of data continuation. The current Harness worker
+This call records a schedule; it is not proof of data continuation. The current Harness worker
 can execute a supported refresh, including declared source windows, but it does not make a frozen
 date, a generic cursor or a file listing advance by inference. It also does not attach or execute a
 warehouse schedule. Another deployed Studio or warehouse component may do that work, but only its
 returned state and durable run evidence support a claim that the table is live, caught up, or
-refreshing. If that evidence is absent, report that publication was requested and that ongoing
-freshness is unavailable; do not imply that promotion filled the gap.
+refreshing. If that evidence is absent, report that a schedule was recorded and that ongoing
+freshness is unavailable; do not imply that recording it filled the gap.
 
 `mr-data pin TABLE_ID --version VERSION_ID` freezes the pointer on an exact version for a rollback
 or a hold, `mr-data unpin TABLE_ID` resumes tracking, and `mr-data demote TABLE_ID` detaches the
@@ -1385,10 +1444,10 @@ run has finished; neither refusal deletes anything, and neither ends the command
 ## Autonomous delivery
 
 Own the run until the requested outcome is delivered: a run that succeeded, its declared checks
-read, its coverage inspected, and a publication request made when the user asked for it. Promotion
-is complete only when Studio returns the resulting live state; catch-up and continuing freshness
-require separate durable evidence from the deployed component that owns them. The allowed user pauses
-are the promote decision, the decision to start a full run after a TRUNCATED sample, and a spend
+read, its coverage inspected, and its refresh cadence recorded when the user has said what it should
+be. What is live is what Studio returns as live; catch-up and continuing freshness require separate
+durable evidence from the deployed component that owns them. The allowed user pauses
+are the cadence decision, the decision to start a full run after a TRUNCATED sample, and a spend
 confirmation on a run over the threshold. Do not ask the user to choose sources, Readers, engines,
 formats, workspaces, retry tactics, cleaning operations, or validation repairs. Make those
 operational decisions from the question, observed evidence, registered capabilities, and
@@ -1442,10 +1501,10 @@ population, grain, fields, time semantics, and quality target when possible. Whe
 is not supportable, propose the closest useful `supportable_with_limits` contract, state every
 material limitation, and ask the user only to approve or reject that exact contract.
 
-If recovery changes the recipe digest, say so before promoting: what the user approved was a table
-built from an exact document, and a different document is a different table to approve. Register
-the revision, sample it, interrogate it, show the material differences and limitations, and ask for
-the promote decision again.
+If recovery changes the recipe digest, say so: what the user saw was a table built from an exact
+document, and a different document is a different table. Register the revision, sample it,
+interrogate it, and show the material differences and limitations before anyone reads it as the
+table they asked for.
 
 Never weaken required semantics, checks, rights, security, or authority boundaries to make a run
 pass. Never bypass a Harness refusal with ambient fetching or custom computation. An unavailable
@@ -1787,7 +1846,7 @@ Translate procedure into expertise:
 
 ## Writing a decision record
 
-Whether the record is the reply to the user, the table presented for the promote decision, or a
+Whether the record is the reply to the user, the table presented for the cadence decision, or a
 research notebook on a full install, the writing rules are the same.
 
 Research markdown is an operator-facing decision record: a technical record, not a tutorial. Give
@@ -1817,6 +1876,6 @@ themselves. Do not present a sequence of unlabeled paragraphs or default to a bu
 
 The floating pill represents only your active work. Start `dataset activity` immediately after creation, update the sentence when your task changes, and renew at least once a minute while you are actively working. The default lease is two minutes. Do not leave a detached heartbeat running after your turn ends. Worker ingestion progress has its own table surface and is not proof that you are working.
 
-When recovering, report what you are trying now; an earlier failed attempt belongs in the record. Before every final handoff, cancellation, or exhausted stop, report `--phase done` with a truthful final sentence (for example, “Dataset ready to explore” or “Stopped before the build completed”). Do this even when tables are not enabled. Only use `waiting_on_you` when you have actually asked for an answer. Promotion does not finish your agent session.
+When recovering, report what you are trying now; an earlier failed attempt belongs in the record. Before every final handoff, cancellation, or exhausted stop, report `--phase done` with a truthful final sentence (for example, “Dataset ready to explore” or “Stopped before the build completed”). Do this even when tables are not enabled. Only use `waiting_on_you` when you have actually asked for an answer. A table going live does not finish your agent session.
 
 The host opens the dataset before research begins and keeps the same tab. Enable its optional Follow agent control for a watch-along experience; manual scrolling pauses it and must never be overridden.