mostlyright-data 0.19.0__tar.gz → 0.19.2__tar.gz

{mostlyright_data-0.19.0 → mostlyright_data-0.19.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.19.0
+Version: 0.19.2
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
@@ -111,10 +111,12 @@ worker executables or image publisher.
 
 ## Documentation
 
-- [CLI guide](https://mostlyright.md/docs/guides/cli/)
-- [Hosted research sessions](https://mostlyright.md/docs/guides/hosted-research/)
-- [Credentials](https://mostlyright.md/docs/guides/credentials/)
-- [Public datasets](https://mostlyright.md/docs/guides/use-public-datasets/)
+- [Install and sign in](https://mostlyright.md/docs/start/install/)
+- [Build your first dataset](https://mostlyright.md/docs/start/first-dataset/)
+- [CLI reference](https://mostlyright.md/docs/reference/cli/)
+- [Recipe reference](https://mostlyright.md/docs/reference/recipe/)
+- [Recipe examples](https://mostlyright.md/docs/recipes/)
+- [Use a public dataset](https://mostlyright.md/docs/guides/use-public-datasets/)
 - [Certified document extraction](docs/DOCUMENT-EXTRACTION.md)
 
 Use `mr-data --help` for the full command list and options. Commands that support

{mostlyright_data-0.19.0 → mostlyright_data-0.19.2}/README.md

@@ -99,10 +99,12 @@ worker executables or image publisher.
 
 ## Documentation
 
-- [CLI guide](https://mostlyright.md/docs/guides/cli/)
-- [Hosted research sessions](https://mostlyright.md/docs/guides/hosted-research/)
-- [Credentials](https://mostlyright.md/docs/guides/credentials/)
-- [Public datasets](https://mostlyright.md/docs/guides/use-public-datasets/)
+- [Install and sign in](https://mostlyright.md/docs/start/install/)
+- [Build your first dataset](https://mostlyright.md/docs/start/first-dataset/)
+- [CLI reference](https://mostlyright.md/docs/reference/cli/)
+- [Recipe reference](https://mostlyright.md/docs/reference/recipe/)
+- [Recipe examples](https://mostlyright.md/docs/recipes/)
+- [Use a public dataset](https://mostlyright.md/docs/guides/use-public-datasets/)
 - [Certified document extraction](docs/DOCUMENT-EXTRACTION.md)
 
 Use `mr-data --help` for the full command list and options. Commands that support

{mostlyright_data-0.19.0 → mostlyright_data-0.19.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.19.0"
+version = "0.19.2"
 description = "Mostly Right hosted CLI for reviewed datasets"
 readme = "README.md"
 requires-python = ">=3.13"

{mostlyright_data-0.19.0 → mostlyright_data-0.19.2}/skills/mr-data-build/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: mr-data-build
-description: Build and verify a reproducible Parquet dataset from a data question and supported sources on the hosted Mostly Right backend. Use whenever a user asks to use, test, smoke-test, demonstrate, or debug the Mostly Right Harness or mr-data CLI by producing, reviewing, or verifying a dataset.
+description: Build and verify a reproducible Parquet dataset from a data question and supported sources on the hosted Mostly Right backend. Use whenever a user asks to use, test, smoke-test, demonstrate, or debug the Mostly Right Harness or mr-data CLI by producing, reviewing, or verifying a dataset. The published reference at https://mostlyright.md/docs/ is the contract.
 ---
 
 # Mostly Right Data Build
@@ -81,16 +81,49 @@ work, follow it, and read what the backend proved. Studio is the hosted release
 
 The product is hosted-only and the client is thin. The engine runs on the backend; `mr-data` is a
 pure-Python client that submits work and reads the records Studio holds.
-[Hosted parity](../../docs/HOSTED-PARITY.md) is the ledger of which name runs where.
+[Install and sign in](https://mostlyright.md/docs/start/install/) states that one install and where
+a command's work actually runs.
 
 Read this document in two parts. Everything except [Not hosted yet](#not-hosted-yet) uses only
 commands this product runs. [Not hosted yet](#not-hosted-yet) is the internal capability map for
 the one command with no implementation behind it.
 
-Every `../../docs/NAME.md` below names a file in the `mostlyrightmd/mostlyright-data-harness`
-repository. This document is installed on its own, so those are references and not links you can
-follow from here: everything you need to run a build is in this file, and each of them is the
-longer reference behind one section of it.
+Every link below is a page of the published reference at https://mostlyright.md/docs/, and every
+one of them can be fetched from here. This document is installed on its own and remains the whole
+of what a build needs; each page is the longer, current reference behind one section of it.
+
+## Reference pages
+
+The published reference at https://mostlyright.md/docs/ carries the current contract for every
+field, flag, code and ceiling this document names. Fetch the page for the shape in front of you
+BEFORE drafting it, never guess a field name, and where a page disagrees with what you remember of
+an older version, the page is right.
+
+| What you are about to do | Fetch |
+| --- | --- |
+| Write a recipe document | https://mostlyright.md/docs/reference/recipe/ |
+| Declare a check | https://mostlyright.md/docs/reference/checks/ |
+| Plan a transform step | https://mostlyright.md/docs/reference/transforms/ |
+| Name a unit | https://mostlyright.md/docs/reference/units/ |
+| Choose a connector, a source kind or a connector parameter | https://mostlyright.md/docs/reference/sources/ |
+| Pin a Reader | https://mostlyright.md/docs/reference/readers/ |
+| Read any command's flags and `--json` shape | https://mostlyright.md/docs/reference/cli/ , then the group page: `auth`, `dataset`, `recipe`, `run`, `read`, `table`, `stream` under `/docs/reference/cli/` |
+| Read an error code, a refusal or a run state | https://mostlyright.md/docs/reference/run-states-and-errors/ |
+| Check a ceiling before starting a run | https://mostlyright.md/docs/reference/limits/ |
+| Follow the whole lifecycle end to end | https://mostlyright.md/docs/start/how-it-works/ |
+| Install the client and sign in | https://mostlyright.md/docs/start/install/ |
+
+Read the worked recipe closest to the source in hand before writing your own.
+https://mostlyright.md/docs/recipes/ lists all eight: `city-temperatures` (daily city
+temperatures), `csv-snapshot` (one CSV file, one table), `json-api` (records pointer and
+pagination), `html-collection` (many HTML pages, one table), `document-extraction` (PDF
+projection), `weather-grib` (GRIB2 and the scientific Readers), `websocket-stream` (a `wss://`
+venue), `multi-source-join` (two sources joined, with checks).
+
+The build guides run one per stage of the loop below, each at https://mostlyright.md/docs/build/NAME/
+where NAME is `probe-sources`, `write-a-recipe`, `run-and-inspect`, `publish-and-refresh`,
+`credentials`, `live-streams`, `collections`, `documents` or `troubleshooting`; the list itself is
+at https://mostlyright.md/docs/build/ and the reference index at https://mostlyright.md/docs/reference/.
 
 ## One install
 
@@ -117,7 +150,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data login` | Complete device approval and store a device credential. `login --force` is the compatibility alias for safe `auth rotate`, never an in-place truncate. |
 | `mr-data whoami` | Use the compatibility alias for the remotely validated `auth status` answer. |
 | `mr-data clarify` | Say whether one clarifying question may be put to the user right now, and compose the record of their answer. It reaches nothing, asks nobody and waits for nothing. Exits non-zero when no person can answer or when a recipe is already registered. |
-| `mr-data probe` | Ask one registered source one question and print the answer. **Currently unusable:** it reaches `/v3/sessions`, which Studio refuses, and it takes a question identifier that no v4 receipt returns. Do not plan a source inspection around it. |
+| `mr-data probe` | `probe SOURCE_ID QUESTION_ID --dataset DATASET_ID [--kind source_inspect\|sample_rows\|profile_columns\|evaluate_expression]` asks one already-registered source one question and prints the answer. **Do not plan a source inspection around it.** Both positionals are identifiers — `QUESTION_ID` is a question identifier, which no v4 registration receipt returns — and the command rides the frozen `/v3/sessions` routes, which a deployment may have switched off. Read a source instead by registering the recipe and taking an unwindowed sample under `--max-rows`, then `peek`, `query` and `receipt`. |
 | `mr-data catalog` | `catalog search "QUESTION"` asks the sealed public-source catalogue which of its entries might answer a question and ranks them best first. Every ranked entry comes back with the disposition its own facts earned — `admitted`, `human_escalation_required` or `refused` — because an entry the catalogue could not vouch for is still a finding. `--format csv` states the one data format the question requires — one token per question, never repeated — and it is not a filter: an entry that does not declare that format still comes back ranked, with disposition `refused` and `filters_match` false, so nothing is held back; `--limit N` ranks at most N, up to 25. It fetches nothing and registers nothing. The catalogue holds one provider (Data.gov) and only part of it, so it is never exhaustive and an empty answer is not evidence that no such source exists. When the answer is `catalog_unavailable`, this deployment has no catalogue to search: record the lane and carry on. |
 | `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --topics "a,b,c" --license ID --description-file F` writes the title, the descriptive tags, the SPDX licence and the description under the version it was read at, retrying once if somebody else wrote first, and an empty `--topics` or `--license` takes that value off the page; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset publish ID [--mode public|link|private]` says who can read the dataset — `public` lists it in the public directory and serves it at an address anybody can read, `link` serves it at an unlisted address, `private` takes it back to the workspace — and `dataset publish ID --show` reads that back without changing it; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. `mr-data recipe show ID` reads one back. |
@@ -160,7 +193,7 @@ step between a run finishing and reading it: `download`, `receipt`, `checks`, `p
 answer about a run that is still workspace scratch exactly as they answer about one somebody has
 since made live.
 
-[The recipe document](../../docs/RECIPE-DOCUMENT.md) is the reference for what
+[The recipe document](https://mostlyright.md/docs/reference/recipe/) is the reference for what
 `mr-data recipe FILE` sends. Two rules in it are worth knowing before you write one: no fractional
 number appears anywhere in the document, and you neither compute nor state a digest — the server
 canonicalizes the document, digests it, and answers with every identifier it derived. Sending the
@@ -172,8 +205,9 @@ signed something on a person's own computer, or they were named for a step the p
 performs — asking a person before the work starts, reading back a separate body of evidence,
 opening a session before a question could be asked. For a while each still typed and answered with
 where its work went; that ended when the refusing names outnumbered the working ones two to one.
-[Hosted parity](../../docs/HOSTED-PARITY.md) lists every one of them and what replaced it, and is
-the only place they are still written down.
+[Install and sign in](https://mostlyright.md/docs/start/install/) states the one install and the
+whole surface it carries; the deleted names and what replaced each of them are written down only in
+the Harness repository's own `docs/HOSTED-PARITY.md`, which is not installed beside this document.
 
 Most of the survivors take a run identifier where the deleted local twin took a folder. That is
 the whole difference in the vocabulary: a hosted build has no folder, and the run identifier is
@@ -216,7 +250,7 @@ machine that is only using the product installs from `pip` instead.
 ## Cloud authentication preflight
 
 Authentication begins only when the next command will contact the Mostly Right cloud. Immediately
-before hosted `mr-data probe` or `mr-data recipe`, run
+before hosted `mr-data dataset create` or `mr-data recipe`, run
 `mr-data whoami --json` and read its `status`, and it intentionally does not report
 `MOSTLYRIGHT_API_KEY`: when the variable is set, `whoami` validates that effective environment
 credential and reports stored-device metadata separately. It never prints the environment value and
@@ -394,23 +428,30 @@ provider or bypass it.
 reason codes only after the discovery lanes and source comparisons are complete. Claim
 `unsupported` only after every typed-refusal chain ends `lawful_routes_exhausted`.
 
-Reading a registered source's bytes before you commit to it is one command:
+**Reading the bytes is the sample run, not `mr-data probe`.** `probe SOURCE_ID QUESTION_ID
+--dataset DATASET_ID` takes two identifiers rather than a question in words, and the question
+identifier is one no v4 registration receipt returns; the command also rides the frozen
+`/v3/sessions` routes, which a deployment may have switched off. So the reading path on a hosted
+workspace is the ordinary loop, one stage early: register the recipe, start an unwindowed sample
+under a row ceiling, and read what came back.
 
 ```sh
-mr-data probe SOURCE_ID "does this feed carry city grain before 2021" --json
+mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --mode sample --max-rows 500 --json
+mr-data peek RUN --json      # the columns and types the run sealed
+mr-data query RUN "select count(*) from run_table" --json
+mr-data receipt RUN --json   # what was fetched and what those bytes hashed to
 ```
 
-The session is plumbing and is implicit: it is opened or reused for that source, asked, and closed
-when the command returns, on the failure path as well as the success path, so a refused probe
-leaves no warm worker running. Keep its status, reason code, and exit value internal; when a probe
-result affects a source decision, describe only what it settled about the data.
+That is stages 4 through 6 run once to settle the shape, and a sample that already meets the
+requirements is the table a full run would seal. Where the deployment does answer a probe, keep its
+status, reason code and exit value internal; when a probe result affects a source decision, describe
+only what it settled about the data.
 
-**A probe answer is evidence, and it is cheap to reuse.** The v3 doctrine that a probe was never
-evidence is repealed. Every acquisition — a probe's or a run's — produces a receipt, and
-acquisitions are cacheable by source, query digest and window, so a probe's fetch feeds a sample
-built in the same warm session without paying for the bytes twice. In v4.0 that cache is
-session-scoped by owner ruling: it lives for the session's lifetime and no longer, so do not plan
-around a probe from an hour ago still being warm.
+**An acquisition's answer is evidence, and it is cheap to reuse.** The v3 doctrine that a probe was
+never evidence is repealed. Every acquisition produces a receipt, and acquisitions are cacheable by
+source, query digest and window, so a fetch feeds a later build in the same warm session without
+paying for the bytes twice. In v4.0 that cache is session-scoped by owner ruling: it lives for the
+session's lifetime and no longer, so do not plan around a fetch from an hour ago still being warm.
 
 Where a source needs a credential, enrol it by name first — see
 [Source credentials](#source-credentials) — and reference the name from the recipe document. Where
@@ -480,7 +521,7 @@ mr-data recipe RECIPE.json --json
 One call. The server canonicalizes the bytes, computes the digest, and registers the dataset, the
 question, the sources, the connector configs and the table plan in one transaction, answering with
 `recipe_id`, `recipe_digest`, `dataset_id`, `table_id` and `source_ids`. Read
-[The recipe document](../../docs/RECIPE-DOCUMENT.md) for the shape of every member.
+[The recipe document](https://mostlyright.md/docs/reference/recipe/) for the shape of every member.
 
 **Name the dataset the recipe binds to.** The document's `dataset` member takes an optional `id`,
 and it is the identifier stage 1 printed. With it, the registration attaches to the page somebody
@@ -596,7 +637,8 @@ seconds -- not a paragraph, and not a report. Use exactly this shape, in this or
 section out only when there is genuinely nothing to say (shown indented here; the document
 itself carries the headings at column one):
 
-    Two plain sentences: what each row is, where it comes from, and the period it covers.
+    Two or three plain sentences: what the data is about, why it is worth a look, and what a
+    reader can see or do with it.
 
     ## Coverage
     - **Window:** the exact dates and the time zone.
@@ -618,17 +660,45 @@ itself carries the headings at column one):
 
 That is six sections: the opening paragraph, and the five headings under it.
 
-**Opening paragraph rules** (everything before the first `##` heading)
-1. Two sentences, 150 to 240 characters together. The first sentence ends before character 200
-   and is complete on its own.
-2. Sentence one says what each row is, for where, over what window, from which publisher:
-   `Each row is one hourly METAR observation at Prague Václav Havel Airport (LKPR), January 2000
-   to August 2026, from the Iowa Environmental Mesonet ASOS archive.`
-3. Sentence two gives the size and the measured quantities, as numbers and nouns:
-   `43,812 rows of temperature, dew point, wind, pressure and visibility.`
-4. Full names on first use with the abbreviation in parentheses once. Never "this dataset",
-   "we", "it", "here".
-5. No Markdown in the opening paragraph: no bold, no links, no bullets, no code.
+**Opening paragraph** (everything before the first `##` heading)
+
+Two or three short sentences, 120 to 280 characters, written for somebody who has never heard of
+the source, the venue or the format. Sentence one says what the data is about and what makes it
+worth a look, in at most 110 characters. The rest says, in the same plain words, what a reader can
+see or do with it.
+
+1. No tickers, station codes, series codes, format names, publisher names, row definitions or row
+   counts here. Every one of those facts is in the sections below, which is where a reader who
+   wants it will look.
+2. No Markdown: no bold, no links, no bullets, no code.
+3. A hook is an interesting fact stated plainly, never a pitch. Banned outright: comprehensive,
+   rich, valuable, powerful, unlock, explore, dive into, perfect for, insights, "this dataset", a
+   rhetorical question, an exclamation mark.
+4. Every number in it must be one the run established. "About every half hour" and "thousands of
+   markets" are claims, and a claim that is not on the run does not go here.
+5. The stranger test, before anything else is written: read sentence one alone under the title. If
+   a curious non-specialist could not say what the data is about and would not want the next line,
+   rewrite it.
+
+One airport weather table, the shape to avoid and the shape to write. To avoid:
+
+    Each row is one METAR or SPECI weather report from Beijing (ZBAA), since January 2020, via
+    Iowa Environmental Mesonet. 117,876 reports of temperature, wind, pressure, visibility with
+    the day's highest and lowest reading.
+
+To write:
+
+    Beijing's weather about every half hour since 2020, with each day's official high and low on
+    the same row. Temperature, wind, pressure and visibility as the airport recorded them, ready
+    to test a forecast against what actually happened.
+
+The station code, the report types, the publisher and the count all moved down to Coverage and to
+Source and rights, where a reader who wants them looks. "About every half hour" stays only because
+Coverage puts a month at 1,302 to 1,591 rows, which is one row every 27 to 33 minutes. "On the same
+row" stays because it says what a reader can see; the row definition rule 1 bans is the sentence
+that begins `Each row is`. Two ways to fail: the pitch (`Unlock real-time insights into how
+prediction markets price the weather!`) and the bare subject with no reason to care (`Beijing
+weather data.`).
 
 **Body rules** (the five sections under the headings)
 1. Every sentence must stay true and complete when quoted alone, away from the page. Never
@@ -644,14 +714,16 @@ That is six sections: the opening paragraph, and the five headings under it.
 Rules that keep it readable: short sentences; no sentence over about twenty-five words; bullets
 rather than comma lists; no tables, no code blocks, no links to internal tools, no headings beyond
 the ones above. Spell out the names a reader would search for -- the place, the identifiers in
-every common form, the programme and the publisher -- once each, in the opening sentences and
-never as a keyword list. A one-line description is a defect to fix in the revision, not a style
-choice. Give `table.description` the grain and the window.
+every common form, the programme and the publisher -- once each, under Coverage and under
+Source and rights, and never as a keyword list. A one-line description is a defect to fix in the
+revision, not a style choice. Give `table.description` the grain and the window.
 
 **Check before writing.** Fix and re-check until every line passes:
 - title: at most 60 characters, contains the place, contains no colon, pipe or exclamation mark,
   no brand, no "dataset" or "data", no version number, no run date
-- opening paragraph: 150 to 240 characters, first full stop before character 200, no Markdown
+- opening paragraph: two or three sentences, 120 to 280 characters, first full stop before
+  character 110, no Markdown, no identifier, publisher name, row count or row definition, no
+  banned word
 - category: exactly one fixed ID saved and verified before recipe registration
 - topics: 3 to 8 descriptive tags, no duplicates, lowercase, at most 40 characters each
 - licence: an SPDX identifier the sources actually grant, or `--license` left off
@@ -737,25 +809,36 @@ contradicts what the column declared. A `bucket` the span outgrew is not one of
 is widened and the chart stays yours. So `none` is the right answer to write for a column you
 already know has no shape, and a wide date range is worth a thought before you promise a timeline.
 
-[The recipe document](../../docs/RECIPE-DOCUMENT.md) carries which types each chart admits, what
-the defaults are, and the two members only some charts take.
+[The recipe document](https://mostlyright.md/docs/reference/recipe/) carries which types each
+chart admits, what the defaults are, and the two members only some charts take.
 
 ### What the statement must actually return
 
-Every acquired source is bound as all-text CSV — `read_csv(header = true, auto_detect = false,
-delim = ',')` with every column `VARCHAR` — so each declared type is an explicit cast in the
-statement, and the cast is checked against what the engine returns:
+The last step's relation is projected into the declared columns: every declared column must come
+back BY NAME, in an engine type that narrows into the declared one. A returned column the table does
+not declare is dropped; a declared column no statement returns is `TRANSFORM_PLAN_INVALID` before a
+row exists.
+
+A source's `binding` decides what its columns are typed as. `text` — the default, and what leaving
+the member out means — binds every column `VARCHAR` whatever the file said, so each declared type is
+an explicit cast you write. `typed` keeps the types the bytes carry. Under either one the cast is
+checked against what the engine returns:
 
 | Declared | The engine type the statement must return |
 | --- | --- |
 | `string` | `VARCHAR` |
 | `integer` | `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT` — not the widest |
-| `decimal` | `DECIMAL(p,s)` — never `FLOAT` or `DOUBLE` |
+| `decimal` | `DECIMAL(p,s)` with `p` at most 38 — never `FLOAT` or `DOUBLE` |
+| `float` | `DOUBLE`, `FLOAT` |
 | `boolean` | `BOOLEAN` |
 | `date` | `DATE` |
 | `timestamp` | `TIMESTAMP WITH TIME ZONE` |
 | `json` | `JSON` or `VARCHAR` |
 
+`decimal` refusing the floating types is the point of that split: an average or a division over
+exact decimals comes back `DOUBLE`, so end the expression in `CAST(… AS DECIMAL(p,s))` where the
+column is exact, and declare `float` where it is genuinely approximate.
+
 `TRANSFORM_COLUMN_TYPE_MISMATCH: column obs_time is declared timestamp and the statement returns
 TIMESTAMP` is the time-zone row, and it names the same word twice because the difference is the zone
 rather than the type. Build one from an epoch with `to_timestamp(CAST(x AS BIGINT))`, or from text by
@@ -792,12 +875,22 @@ ones, so a caller cannot assert a digest for its own bytes even by trying; the d
 is the server's reading of what it canonicalized. Registration is idempotent by that digest:
 identical bytes register one recipe rather than two.
 
-Before proposing a step, read [Transforms](#transforms); before proposing a future outcome, read
-[Prediction labels](#prediction-labels). Declare what each measured column measures as a unit code
-from the accepted subset — `cm`, `ug/m3`, `km/h` and the rest of it are written out in
-[Units](../../docs/UNITS.md). Do not invent a unit name: a code outside the subset is refused with
-the construct that put it there. Say `none` where a column carries no physical unit. Remember that
-the two temperature scales are offsets rather than multipliers.
+Before writing a step, read [Transforms](#transforms); before proposing a future outcome, read
+[Prediction labels](#prediction-labels). Declare what each measured column measures as a UCUM code
+from the accepted subset — `Cel`, `ug/m3`, `km/h`, `hPa`, `{count}` and the rest of it are written
+out in [Units](https://mostlyright.md/docs/reference/units/). Write the code rather than a name:
+`ug/m3` composes and `microgram_per_cubic_meter` does not.
+
+**Nothing resolves the code, so there is no `UNIT_*` refusal to lean on.** The recipe schema bounds
+the length of `unit` and nothing else; registration checks only that `units[].column` names a
+declared column and answers `422 RECIPE_DOCUMENT_INCOHERENT` at `/units/N/column` when it does not;
+the run never reads `units` at all. A code outside the subset therefore costs nothing at build time
+and costs every reader afterwards, and a column declared in metres that holds feet is caught by
+nothing. Convert in the statement, then declare the unit of what the column ends up holding. Leave a
+column out of `units` rather than declare something approximate — a density, a rate, anything whose
+unit is genuinely new. `none` is not a unit and does not become one. Remember that the two
+temperature scales are offsets rather than multipliers, so `Cel` composes with nothing: `Cel/m` and
+`mCel` are both outside the subset.
 
 ### 5. Build — take an unwindowed sample first
 
@@ -855,7 +948,7 @@ one", not as a failure: a document declaring 8 MiB over a feed that answers 40 K
 ceiling on that source:** that does not fetch less data, it truncates the sample or refuses the
 full run. Split the window across several sources — one per month, per region, or per whatever the
 publisher paginates on — and union them in a first transform step. A recipe names up to 256
-sources, which is a great many windows. [The recipe document](../../docs/RECIPE-DOCUMENT.md)
+sources, which is a great many windows. [The recipe document](https://mostlyright.md/docs/reference/recipe/)
 carries the measured numbers behind the ceiling.
 
 **A corpus of many pages behind one index is a COLLECTION, not many sources.** See
@@ -1321,9 +1414,9 @@ access is denied, say the current account cannot view the build evidence and ask
 to an authorized account or request access.
 
 A flag that cannot mean what it means locally is accepted and reported, never refused.
-`mr-data runs --depth 3` runs, lists the workspace's runs, and reports under `flags_without_effect`
-that folder depth described nothing here. Read that key before repeating an argument that did
-nothing. Only flags actually passed are reported.
+`mr-data status RUN_ID --receipts` runs, reports the run, and reports under `flags_without_effect`
+that a run's digests, sources, timings and checks are already on the record. Read that key before
+repeating an argument that did nothing. Only flags actually passed are reported.
 
 ### Asking the table a question
 
@@ -1351,7 +1444,7 @@ Two things are currently broken here and are not worth diagnosing again:
 version digest in `source.connection`, with the query in `parameter_values` and a valid response
 reader; set `dataset.id` to the granted dataset. Do not ask the user to re-enter an already saved
 credential under Keys. If it is not granted, explain that an owner or administrator must allow
-the dataset in Settings → Connections & secrets. See [Saved connections](../../docs/RECIPE-CONNECTIONS.md)
+the dataset in Settings → Connections & secrets. See [Saved connections](https://mostlyright.md/docs/build/credentials/)
 for the proposal shape. Studio resolves it into the canonical recipe and enforces access during
 execution. Named secrets below remain available for sources without saved connections.
 
@@ -1615,8 +1708,9 @@ or network search. If every registered, lawful alternative is deterministically
 record and report the typed impossibility. Do not loop forever, fabricate a dataset, or turn a
 refusal into permission.
 
-Never download source bytes directly. Never write custom Python, SQL, or shell transformations.
-Never create Parquet outside a Harness Build. Never report a standalone Parquet file as a Build.
+Never download source bytes directly. Never transform data on this computer: the transform is the
+recipe's own `duckdb_sql` steps, run on the backend, and Python, shell or a local SQL engine reading
+the same bytes is not that. Never create Parquet outside a Harness Build. Never report a standalone Parquet file as a Build.
 Do not report completion unless the run succeeded, its declared checks were read, and any artifacts
 the user asked for came back with their digests checked.
 
@@ -1743,20 +1837,24 @@ the records pointer names an array, and whether `discovery.max_requests` plus
 they are the four that are cheapest to get right before a run spends anybody's rate limit.
 
 The full contract, including every bound and the coverage block's members, is
-[Source collections](../../docs/SOURCE-COLLECTIONS.md).
+[Many-page sources](https://mostlyright.md/docs/build/collections/), with the connector coordinate
+and its parameters under
+[Source kinds and connectors](https://mostlyright.md/docs/reference/sources/).
 
 ## Readers
 
-`docs/READERS.md` defines Reader behavior. Certified Readers may be pinned on an exact
-`ArtifactSourceInput` for `user.file` or on a credential-free public HTTPS source. A Recipe pins the
-exact Reader name and version. Unsupported versions refuse by name. A version change is a recipe
-revision.
+[Readers](https://mostlyright.md/docs/reference/readers/) defines Reader behavior. Certified
+Readers may be pinned on a hash-pinned
+`user.file` artifact, on a credential-free public HTTPS source, or on a bounded ordered public HTTPS
+collection, which applies one exact pin to every member. A Recipe pins the exact Reader name and
+version. Unsupported versions refuse by name. A version change is a recipe revision.
 
 **Direct CSV supports publisher CSV aliases, including Eurostat SDMX-CSV.**
 Accepted labels include `text/csv`, `text/plain`, `application/csv`, `application/x-csv`,
 `text/x-csv`, and `application/vnd.sdmx.data+csv`. For title rows, alternate encodings,
 headerless files, or padded columns, pin `delimited_text@2.0.0` and configure the layout; see
-`docs/READERS.md`. A publisher needing another format can use its matching Reader.
+[Readers](https://mostlyright.md/docs/reference/readers/). A publisher needing another format can
+use its matching Reader.
 
 **A v4 recipe pins a Reader through `connector.parameters`, under three reserved names.**
 `declared_source` is closed and has no member for a decoder, so the pin rides on the parameter array
@@ -1776,38 +1874,35 @@ field and finding none does not mean v4 cannot pin one.
 }
 ```
 
-A parameter under `reader.` that is not one of those three is refused rather than ignored.
+A parameter under `reader.` that is not one of those three is refused rather than ignored. All three
+or none: there is no other place a Reader is named. Read the pin grammar and the closed registry of
+48 coordinates at [Readers](https://mostlyright.md/docs/reference/readers/).
+
+**No command tries a Reader against a URL.** `mr-data peek` reads the preview one hosted run sealed,
+and its `--reader`, `--reader-options`, `--format` and `--json-*` flags are accepted and print `no
+effect here; the hosted preview arrives already decoded`. A Reader is settled the way every other
+part of the document is settled: pin the coordinate in the recipe, register it, take an unwindowed
+sample under `--max-rows`, and read the columns back with `peek` and the decode evidence with
+`receipt`. A wrong coordinate or a wrong settings object is a typed refusal on that run naming the
+source, which is the answer — change the pin and register the revision.
 
 **`reader.decode_options` is the canonical JSON text of the options the family itself admits**, with
 every family-defaulted key filled in. A stated subset digests differently from the pin the worker
-reconstructs and is refused. Derive the value rather than writing it:
-
-```python
-from mostlyright.data_harness.readers.json_tabular import JsonTabularReaderV1_1
-from mostlyright.data_harness.canonical import canonical_json_bytes
-admitted = JsonTabularReaderV1_1().validate_options(
-    {"columns": [{"name": "icao_id", "pointer": "/icaoId"}]}
-)
-value = canonical_json_bytes(admitted).decode()
-```
+reconstructs and is refused. This client ships no Reader code to derive the value with: copy the
+family's complete default object from [Readers](https://mostlyright.md/docs/reference/readers/),
+change only the keys the source needs, keep every other key at its default, and write it as
+canonical JSON — keys sorted, no whitespace.
 
 A column that may be absent from a record must state `"required": false`. Publishers routinely omit
 a key rather than sending null, and a required column that is missing refuses the record.
 
-**Open defect: an empty string in the admitted options is refused at the acquisition boundary with
-`TEXT`, and the refusal names no source.** `records_pointer: ""` is the RFC 6901 pointer for a
-root-level array, which is what `json.tabular` admits by default and what every API returning a bare
-JSON array needs, so this blocks that whole class of source. The request-side screen applies the
-identifier rule — where an empty string is a missing one — to JSON values, where it is not. Fixed by
-harness PR #503; until that reaches the deployed worker image, a root-array JSON source cannot be
-pinned. Confirm the deployed worker carries the fix before planning a build around one.
-
 When you pin a Reader for a source nobody has approved yet, name the family's highest certified
 version rather than its base version. The base version refuses a publisher that merely declined
 to describe its file, and a weak-label widening lands as a new coordinate precisely so an
 approved recipe is not moved under review's feet -- which makes the base version the wrong
-default for a source that has no approval to protect. `ReaderRegistry.widest_version` answers
-which coordinate that is. For a family whose major version has moved, the highest coordinate is
+default for a source that has no approval to protect. The registry table at
+[Readers](https://mostlyright.md/docs/reference/readers/) lists every certified coordinate of a
+family; take the highest within the major you want. For a family whose major version has moved, the highest coordinate is
 a different output contract rather than a wider door onto the same one, so choose it
 deliberately: `weather.grib2@2.0.0` emits columns `1.0.0` does not.
 
@@ -1887,78 +1982,117 @@ response matches the family's exact media-type, settings, and budget contract.
 
 ## Transforms
 
-The graph operations are a closed set, and what gets into that set is decided by one written test:
-a step is cleaning when the sources plus the declared shape determine it, and it is a modeling
-choice when it needs an assumption about what the analyst intends to model. Read
-[Transforms a plan may perform](../../docs/TRANSFORMS.md) before proposing a step that is not
-already in the vocabulary; do not work around a refused operation with a chain that reproduces it.
-
-Four points recur in plans:
-
-- Division is an expression inside `derive`. Numerator first, denominator second, both operands in
-  one exact numeric type, and the result is always the exact decimal type — dividing two whole
-  numbers never rounds down or towards zero. Declare `zero_denominator_policy` and
-  `null_denominator_policy` on any node that divides, and on no other node. A zero denominator
-  stops the run whether or not the numerator is absent. For a rate per hundred thousand, multiply
-  inside the division rather than after it. Write a conversion with the division last —
-  `(x * 9 + 160) / 5` rather than `x * 9 / 5 + 32` — because every operation takes operands of one
-  exact numeric type and a quotient lands in the decimal type.
-- `unpivot` turns one column per period into rows. Name the value-bearing columns in the plan,
-  name the two output columns, declare `value_type` for the long column, and set `null_policy` to
-  `retain`; drop absent cells with a separate `filter` step if the contract calls for it. Every
-  named column must already be the declared type.
-- Join kinds are `left`, `inner`, and `anti`, and all three are graph kinds. Use `anti` for "which
-  of these has none of those"; do not join and then filter on an absent column, which cannot tell a
-  missing counterpart from a legitimately absent value. An anti join emits left columns only, takes
-  `unmatched_policy` `preserve`, and has its declared cardinality checked on the left side. The
-  older plan version's single join is `left` and stays `left` — a settled ruling, not a gap — so a
-  question needing `inner` or `anti` is written as a graph.
-- `date_add_days` is the one fixed calendar availability correction. It takes a declared `date`
-  column, a new output column, and an integer `days` from -3650 to 3650; it keeps nulls null and
-  rejects a result outside the supported date range. Use it for a source-established statement
-  such as `information_available_date = rate_date + 1`; it never reads another row and is not a
-  lag, lead, target, or replacement for `prediction_label`. Date-only arithmetic remains a
-  calendar-day operation at DST boundaries.
-- `trim` and `case_fold` are the whole text-normalization surface, and there is no pattern
-  matching. `trim` removes six named characters; `case_fold` lowers twenty-six letters and touches
-  nothing else. Text needing more than that is normalized before it reaches the plan.
-
-Declare what each measured source column is in, as `column_units` on the source node, and what
-unit a computed column is in, as `output_unit` on its expression. Both are optional and both are
-what the two unit checks read: a merge whose declared units disagree is refused, and a
-conversion that is not the arithmetic those two units imply is refused with the arithmetic they
-do imply. A column you leave undeclared is left alone by both, and a column with no physical
-unit is left out rather than declared `none`. Declare `output_unit` only where the expression
-converts one declared column, or where its operands settle the answer, or where nothing it
-reads is declared; a computed column whose unit is genuinely new -- a density, an area, a rate
--- declares nothing, because nothing can check it.
+**A v4 transform is SQL, not an operation vocabulary.** The member is
+`{"engine": "duckdb_sql", "steps": [...]}`, and each step is `{"step_id": "…", "sql": "…"}` with an
+optional `description`: at most 64 steps, each statement at most 65536 characters. There is no node
+graph, no closed operation set and nothing to work around — the statement is the plan. The schema
+also admits `engine: "none"`, and that document registers and then refuses its first run
+`TRANSFORM_PLAN_INVALID`, as an empty `steps` array does, because neither says what a build
+produces. Fetch [Transforms](https://mostlyright.md/docs/reference/transforms/) before writing one:
+it carries the admitted function table, the worked shapes and every refusal code.
+
+**What a step reads.** One relation per source, under the source's own `name`, and one relation per
+earlier step, under its `step_id`. Steps run in array order, never sorted, so a later step reads an
+earlier one by writing its id. A `step_id` equal to a source `name` is refused
+`TRANSFORM_STEP_SHADOWS_SOURCE` and two steps sharing an id are `TRANSFORM_STEP_DUPLICATE`. Quote a
+name that is also a SQL keyword.
+
+**What the columns are typed as.** The relation carries the ordered header acquisition recorded, and
+the source's `binding` decides the types: `text` — the default, and what leaving the member out
+means — binds every column `VARCHAR`, and `typed` keeps what the bytes carry. Under `text` every
+declared type is a cast you write, which keeps casting an act of the recipe. Two relations are not
+declared anywhere: a recorded stream source presents one of two fixed text headers — eight
+columns `epoch`, `event_id`, `event_time`, `message`, `message_class`, `message_type`,
+`received_at`, `sequence`, or those plus `member` after `event_time` when the connector declares a
+member field — and a collection source presents the pinned Reader's columns plus seven reserved
+`page_*` columns.
+
+**What the last step must return** is under
+[What the statement must actually return](#what-the-statement-must-actually-return). Row order is
+the table's `grain` and then every remaining declared column in declaration order, so an `order by`
+you write is not the sealed order.
+
+```json
+"transform": {
+  "engine": "duckdb_sql",
+  "steps": [
+    {
+      "step_id": "readings",
+      "sql": "select station, cast(concat(valid, ':00+00') as timestamp with time zone) as observed_at, cast((try_cast(tmpf as decimal(6,2)) - 32) * 5 / 9 as decimal(6,2)) as air_temp_c from asos_observations where valid is not null and valid <> ''"
+    },
+    {
+      "step_id": "hourly",
+      "sql": "select station, time_bucket(interval 1 hour, observed_at) as hour_start, cast(avg(air_temp_c) as decimal(6,2)) as mean_air_temp_c, count(*) as reading_count from readings group by station, time_bucket(interval 1 hour, observed_at)"
+    }
+  ]
+}
+```
+
+`asos_observations` is a source's `name`; `readings` is the first step read by the second. The outer
+`cast` on each measured column is what makes it a `DECIMAL(6,2)` — without it the division and the
+`avg` both return `DOUBLE` and the column is refused.
+
+**One read-only `SELECT` per step.** The engine classifies the statement: more than one is
+`TRANSFORM_STATEMENT_NOT_SINGLE`, and anything that is not a `SELECT` is
+`TRANSFORM_STATEMENT_NOT_SELECT` — so `ATTACH`, `COPY`, `INSTALL`, `LOAD`, `PRAGMA`, `CREATE`,
+`INSERT`, `UPDATE` and `DELETE` are all refused. Common table expressions, `UNION`, `QUALIFY`,
+`UNPIVOT`, `VALUES` and `SELECT DISTINCT` are each one `SELECT` and are admitted, which is how a
+long-format reshape, a join, a deduplication and a time bucket are written.
+
+**The result has to re-derive, and the engine decides that rather than a word list.** A sealed table
+is rebuilt and compared byte for byte, so a step may read nothing the recipe and the acquired bytes
+do not fix. The engine's own function catalog is asked about every name the statement references:
+anything it does not report as re-derivable, anything with side effects, and anything it does not
+carry at all is refused `TRANSFORM_PLAN_INVALID` before the step runs. That catches `current_date`,
+`current_timestamp`, `now()`, `today()`, `random()`, `gen_random_uuid()` and anything written in
+terms of them. The date-granular ones are the dangerous ones: both executions inside one build agree
+and the table seals, and tomorrow's refresh can never reproduce it. Reach a column a publisher
+happened to spell like one of those names by qualifying it — `observations.current_date`.
+
+**The engine reaches nothing.** It is given this run's own source files and scratch directory, then
+external access is switched off and the settings are locked, so it can open no file, fetch no URL
+and load no extension. There is no denylist of names in any of it, by design: a source is reached by
+naming its relation and never by opening its address again. Where that confinement cannot be
+established the code is `TRANSFORM_ENGINE_UNCONFINED` and no step was attempted.
+
+**Every table is built in `UTC`**, established before a statement is read. A recipe declaring
+another `timezone` registers and then refuses its first run `RUN_RECIPE_INVALID`. `ICU` is not
+loaded, so `AT TIME ZONE` is unavailable; write the conversion into the statement.
+
+A step the engine itself refused comes back as `RUN_EXECUTION_FAILED` carrying DuckDB's own code and
+words — `step daily_mean would not run: BINDER_ERROR: …`. Read the detail, not the code. Every
+transform code reports `failed_stage: "transform"`.
+
+`units` is declared beside the table, and nothing in the transform reads it or checks the arithmetic
+against it. Convert in the statement, then declare the unit of what the column ends up holding, per
+the paragraph under [What each check kind requires](#what-each-check-kind-requires).
 
 ## Prediction labels
 
-`prediction_label@1.0.0` is retired. Do not put it in a plan, and do not offer a future outcome as
-something the harness can construct. Authoring, plan checking, validation, export and approval all
-refuse a graph naming it, with a typed `OPERATION_RETIRED` refusal.
+There is no prediction-label member in a v4 recipe document and no operation that produces one. Do
+not offer a future outcome as something the harness can construct.
 
-The reason is the admission contract above: the horizon such a label needs is a statement about what
-somebody intends to model, and nothing in the sources says what it should be. A target belongs to
-the modeling step that consumes the dataset, not to the dataset. There is no replacement operation,
-and none is owed.
+The horizon such a label needs is a statement about what somebody intends to model, and nothing in
+the sources says what it should be. A target belongs to the step that consumes the dataset, not to
+the dataset. There is no replacement, and none is owed.
 
-Do not reconstruct it either. `lag`, `lead`, `rolling` and `window_target` are refused by name; a
-derive expression, a self join, or custom code that reproduces the same shift is the same thing
-this contract refuses, whether or not any single check catches it. If a contract asks for a future
-outcome, say that the harness builds the observed table and that the horizon is chosen downstream.
-Set requirements `target_policy` to `none`.
+Do not reconstruct it in the statement either. A window function that shifts a measured value
+forward in time and a self join that reads a later row are the same claim under another spelling,
+and no check catches either — the transform engine refuses what cannot re-derive, not what is
+modelled. Where a request asks for a future outcome, say that the build seals the observed table and
+that the horizon is chosen downstream.
 
-Work sealed or approved before the retirement is unaffected: such a Build still verifies and
-replays, and such a Recipe still runs. [Prediction labels](../../docs/PREDICTION-LABELS.md) is the
-contract for reading it.
+Work sealed before the retirement is unaffected: such a table still verifies and replays, and such a
+Recipe still runs.
+[Units and vocabulary](https://mostlyright.md/docs/reference/units/) records the withdrawal and what
+a reader of such a Build should expect from it.
 
 ## Boundaries
 
-- Keep plans and Recipes declarative. Do not embed Python, SQL, shell, arbitrary expressions,
-  plugins, redirects, or callables.
-- Use only contract-supported operations, joins, adapters, checks, and resource bounds.
+- The transform's `duckdb_sql` steps are the only executable text a Recipe carries, and they run on
+  the backend under the confinement above. Everything else in the document stays declarative: no
+  Python, no shell, no plugins, no redirects, no callables.
+- Use only contract-supported connectors, Readers, checks and resource bounds.
 - Do not claim that recipe registration fetched a source, that a declared check released a table
   version, or that agent output executed a transformation.
 - Preserve the cause and acceptance condition for blocker, critical, and high findings. This

{mostlyright_data-0.19.0 → mostlyright_data-0.19.2}/skills/mr-data-build/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "Mostly Right Data Build"
   short_description: "Build and verify reviewed datasets"
-  default_prompt: "Use $mr-data-build to autonomously deliver the requested dataset outcome before inspecting CLI help, documentation, schemas, fixtures, examples, prior runs, or installed package details whenever the request uses, tests, demonstrates, or debugs the Harness. If the host requires a skill-use disclosure, send only one outcome-specific sentence naming $mr-data-build before the first tool call; otherwise start silently. Never narrate preparation, skill loading, CLI discovery, authentication, document or contract lookup, example searches, package versions, command batches, or acquisition-document authoring. Never narrate skill instructions, commands, receipts, events, protocols, status codes, service behavior, or notebook mechanics. Open the request's canonical Dataset page first when that address can be minted, and carry on without it rather than blocking. Keep routine work silent unless the host requires periodic status; then send concise outcome-oriented updates without mechanics or internal language, grounded in domain-specific observed evidence, a decision, required user action, or the verified outcome."
+  default_prompt: "Use $mr-data-build to autonomously deliver the requested dataset outcome before inspecting CLI help, documentation, schemas, fixtures, examples, prior runs, or installed package details whenever the request uses, tests, demonstrates, or debugs the Harness. If the host requires a skill-use disclosure, send only one outcome-specific sentence naming $mr-data-build before the first tool call; otherwise start silently. Never narrate preparation, skill loading, CLI discovery, authentication, document or contract lookup, example searches, package versions, command batches, or acquisition-document authoring. Never narrate skill instructions, commands, receipts, events, protocols, status codes, service behavior, or notebook mechanics. Open the request's canonical Dataset page first when that address can be minted, and carry on without it rather than blocking. Keep routine work silent unless the host requires periodic status; then send concise outcome-oriented updates without mechanics or internal language, grounded in domain-specific observed evidence, a decision, required user action, or the verified outcome. Fetch the published reference at https://mostlyright.md/docs/ for the recipe field, connector, Reader, transform, check, unit, command flag, error code, ceiling or worked example in front of you before drafting it; it is the contract, it wins over any memory of an older version, and fetching it is preparation you never narrate."