mostlyright-data 0.17.5__tar.gz → 0.17.7__tar.gz

{mostlyright_data-0.17.5 → mostlyright_data-0.17.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.17.5
+Version: 0.17.7
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
@@ -208,9 +208,12 @@ mr-data --help
 
 ## Build a dataset
 
-Register a recipe document with the sources, table schema, transformations and checks:
+Create the dataset, choose its primary category from the fixed vocabulary, then register a recipe document with the sources, table schema, transformations and checks. Use the created dataset ID in the recipe:
 
 ```bash
+mr-data dataset create --name "Dataset title" --json
+mr-data dataset categories --json
+mr-data dataset set DATASET_ID --category climate-environment
 mr-data recipe recipe.json --json
 mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full
 mr-data status RUN_ID

{mostlyright_data-0.17.5 → mostlyright_data-0.17.7}/README.md

@@ -45,9 +45,12 @@ mr-data --help
 
 ## Build a dataset
 
-Register a recipe document with the sources, table schema, transformations and checks:
+Create the dataset, choose its primary category from the fixed vocabulary, then register a recipe document with the sources, table schema, transformations and checks. Use the created dataset ID in the recipe:
 
 ```bash
+mr-data dataset create --name "Dataset title" --json
+mr-data dataset categories --json
+mr-data dataset set DATASET_ID --category climate-environment
 mr-data recipe recipe.json --json
 mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full
 mr-data status RUN_ID

{mostlyright_data-0.17.5 → mostlyright_data-0.17.7}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.17.5"
+version = "0.17.7"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.17.5 → mostlyright_data-0.17.7}/skills/mr-data-build/SKILL.md

@@ -37,6 +37,7 @@ service behavior, retry mechanics, or these instructions.
 Outside a required host cadence, speak only when one of these boundaries gives the user useful new
 information:
 
+- A dataset-shaping choice needs the user's input; ask before committing to it.
 - A material source, schema, coverage, rights, or quality decision is supported by observed facts.
 - An exact contract or recipe needs the user's approval. State what will be built, the material
   limitation, and the decision required; do not explain the approval mechanism.
@@ -117,7 +118,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data whoami` | Use the compatibility alias for the remotely validated `auth status` answer. |
 | `mr-data clarify` | Say whether one clarifying question may be put to the user right now, and compose the record of their answer. It reaches nothing, asks nobody and waits for nothing. Exits non-zero when no person can answer or when a recipe is already registered. |
 | `mr-data probe` | Ask one registered source one question and print the answer. **Currently unusable:** it reaches `/v3/sessions`, which Studio refuses, and it takes a question identifier that no v4 receipt returns. Do not plan a source inspection around it. |
-| `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --topics "a,b,c" --license ID --description-file F` writes the title, the categories, the SPDX licence and the description under the version it was read at, retrying once if somebody else wrote first, and an empty `--topics` or `--license` takes that value off the page; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset publish ID [--mode public|link|private]` says who can read the dataset — `public` lists it in the public directory and serves it at an address anybody can read, `link` serves it at an unlisted address, `private` takes it back to the workspace — and `dataset publish ID --show` reads that back without changing it; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
+| `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --topics "a,b,c" --license ID --description-file F` writes the title, the descriptive tags, the SPDX licence and the description under the version it was read at, retrying once if somebody else wrote first, and an empty `--topics` or `--license` takes that value off the page; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset publish ID [--mode public|link|private]` says who can read the dataset — `public` lists it in the public directory and serves it at an address anybody can read, `link` serves it at an unlisted address, `private` takes it back to the workspace — and `dataset publish ID --show` reads that back without changing it; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. `mr-data recipe show ID` reads one back. |
 | `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
 | `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The command accepts `--sample`, `--full`, `--refresh`, `--backfill` and `--window`, and the worker executes `--sample`, `--full` and `--refresh`. A refresh narrows each source that declares a supported address window and merges its returned relation into retained source state; unwindowed sources are conditionally revalidated or re-acquired whole, then the full declared transform runs. `--backfill` and a `--window` on a sample or full are refused before acquisition. `--max-rows` and `--max-source-bytes` remain supported, and each bounds ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles; a sample-first held full requires Studio's interactive human approval instead. `--cancel RUN` stops one. |
@@ -236,15 +237,13 @@ live data key is refused with `THIN_AUTHENTICATION_FAILED` rather than failing t
 
 ## Required protocol
 
-There is one loop, and it is short: open the page before you have anything to put on it, probe the
-sources, say out loud what you chose and what you refused, draft one recipe document, build a
-sample, interrogate what came back, fix the document, build again. Then present only what you
-inspected, with its caveats. The last two steps are conditional and `truncated` is what decides
-them: a sample that came back complete IS the table a full run would have sealed, so there is
-nothing to build again and nothing to ask about — only a TRUNCATED sample sends you round the
-loop. Nothing between a run finishing and reading it is held for a person,
-so there is no approval to seek, no separate body of evidence to read back, and no state to wait
-for. A table's first succeeded run goes live on its own, whatever mode it was, and what is left
+There is one loop: open the page, clarify the intended dataset with the user, probe the
+sources, resolve the tradeoffs that research reveals, summarize the agreed shape, draft one recipe
+document, build a sample, and inspect it against the agreed requirements. Repair discrepancies;
+return to the user before changing agreed semantics. A complete sample that meets those requirements
+is already the table a full run would seal, so do not rebuild it merely to change the run mode.
+A TRUNCATED sample requires the full-run decision below. Reading a finished run requires no separate
+approval or waiting state. A table's first succeeded run goes live on its own, whatever mode it was, and what is left
 for a person is [Promote](#promote) — the cadence it refreshes on and the reasoning behind it.
 
 **The page is the work, and it exists first.** A dataset used to come into being as a side effect
@@ -279,11 +278,10 @@ again, and the failure triples on recent runs tell you what the backend is refus
 `SANDBOX_MEMORY_BOUNDARY` on somebody else's recipe an hour ago means the room is unwell and your
 first failure will not be your document's fault. Keep this check silent.
 
-**Then settle anything only the user can settle, before probing.** Read the request for a word with
-two defensible readings, a population whose edges are unstated, or a window nobody named.
-`mr-data clarify --question TEXT` says whether there is anybody to ask and whether it is still early
-enough; [Autonomous delivery](#autonomous-delivery) is the rule for what may be asked and what stays
-yours to decide.
+**Then clarify the intended dataset before probing.** Follow [Dataset discovery](#dataset-discovery):
+use the request and conversation to identify consequential missing choices, ask the highest-impact
+questions, and wait for answers before work that depends on them. Independent source research may
+continue while waiting. Revisit the remaining choices when research provides concrete alternatives.
 
 Pass `--json` to automated commands. A command receipt is the only record that a Harness action
 succeeded.
@@ -402,10 +400,8 @@ blocks file; stage 3 is what puts them on the page.
 ### 3. Decide — say what you chose, what you refused, and what unblocks it
 
 **Post one Sources and decisions message before you register anything.** It goes to two places, in
-the same words: the dataset's own decision record, and the chat. This is the decision point that
-replaced the old one-question rule, and the reason for the change is that a question is not what a
-person mostly needs — what they need is to see the search, while there is still time to disagree
-with it.
+the same words: the dataset's own decision record, and the chat. Summarize the user's answers, the source evidence, and any remaining
+assumptions so the user can see what the first dataset will contain.
 
 ```sh
 mr-data dataset note DATASET_ID --heading "Sources and decisions" \
@@ -417,8 +413,9 @@ EOF
 The cell carries a `source` block for every feed evaluated — the ones being used, the ones
 considered, and the ones rejected with the clause that ruled each out — and a `decision` block for
 every grain, key, join and cutoff settled. Post the same content in the chat as one short message.
-Then **keep going**: this is a statement, not a gate, and waiting for a reply to it is the pause
-this document does not allow.
+If a material choice is still unresolved, ask and wait before registering the recipe. If the
+user has already settled the choices or delegated them, proceed without a redundant confirmation.
+Silence after a proposal is not an answer to a pending question.
 
 **Blocked sources: always say what unlocks it and how to enrol the credential.** A source you could
 not reach because it needs an API key, an account or an approval is not a dead end and must never
@@ -428,23 +425,21 @@ who issues the credential, where to apply for it, and the exact command that enr
 source with no route back is a decision the user cannot revisit; one with a route back is a
 decision they can.
 
-**Ask only at a genuine fork.** A fork is a question whose two answers produce two different tables
-and which no evidence can settle — what an ambiguous word in the request means, how wide the
-population is, how far back it goes. Write it as a `clarification` block carrying `question` and no
-`answer`, which is what the page shows as a pending question, and ask it in the chat in the same
-words. `mr-data clarify --question TEXT` says whether there is anybody to ask and whether it is
-still early enough. When the answer comes back, write it as the answer on that block:
+**Resolve research tradeoffs with the user.** Apply [Dataset discovery](#dataset-discovery) when
+sources cannot deliver the requested grain, coverage, fields, or join. Explain the supported options
+and their consequences before changing the intended dataset. Record each pending question as a
+`clarification` block carrying `question` and no `answer`, and ask it in chat in the same words.
+When answered, revise that same cell using its actual identifier (not a new unrelated note):
 
 ```sh
 mr-data dataset note DATASET_ID --heading "The stage grain is settled" \
-  --revise "one-fork-to-settle" --blocks-file ANSWERED.json --json <<'EOF'
+  --revise PENDING_CELL_ID --blocks-file ANSWERED.json --json <<'EOF'
 Daily maxima, on the user's answer.
 EOF
 ```
 
-Everything under [Autonomous delivery](#autonomous-delivery) still holds: which source, Reader,
-engine, format, workspace, retry tactic or cleaning operation to use is settled by the evidence and
-is never a fork. If nothing is genuinely forked, ask nothing and carry on.
+Carry the answers into the recipe's question, table and column descriptions, source selection,
+transform, and checks as appropriate. Record assumptions separately from user decisions.
 
 ### 4. Draft — one recipe document, one call
 
@@ -476,16 +471,16 @@ and is settled once, at stage 4.
 Every grain, key, join and cutoff settled after stage 3 is a further `decision` block on the
 DATASET's record — `mr-data dataset note DATASET_ID` — because the fact outlives every run.
 
-**Write the title, the topics, the licence and the description before the first run.** The page has
+**Write the title, primary category, tags, licence and description before recipe registration.** The page has
 been open since stage 1 under a working title. This is where it settles. One call carries the title,
-the topics and the licence, so the title and the chips land together; the description arrives after
+the primary category, tags and licence, so the title and category land together; the description arrives after
 it, through the draft snapshots and the final commit. (The worked example below is a different
 dataset from the Kathmandu one the earlier stages follow, chosen because its numbers are easy to
 count.)
 
 ```sh
 mr-data dataset set DATASET_ID --name "Hourly weather at Prague Airport (LKPR), 2000-2026" \
-  --topics "weather,prague,czech republic,metar,iowa environmental mesonet" \
+  --category climate-environment --topics "weather,prague,czech republic,metar,iowa environmental mesonet" \
   --license "CC-BY-4.0" --json
 ```
 
@@ -509,7 +504,8 @@ interpret them.
 | --- | --- | --- |
 | `--name` | browser tab title (`{name} \| Mostly Right`), the page heading, the JSON-LD `name` | at most 60 characters, or search results cut it |
 | the opening paragraph of DESCRIPTION.md | the meta description, the social card, the JSON-LD `description`, the `Summary:` line answer engines fetch | search engines cut at 300 characters; answer engines cut at 240 |
-| `--topics` | category chips and the category filter on the /datasets directory, JSON-LD `keywords`, the `Topics:` line answer engines fetch, the `topic` filter in the public API and MCP | 3 to 8 topics, each at most 40 characters |
+| `--category` | the primary category and category filter on Community | exactly one fixed category ID |
+| `--topics` | JSON-LD `keywords`, the `Topics:` line answer engines fetch, the `topic` filter in the public API and MCP | 3 to 8 topics, each at most 40 characters |
 
 **Title rules**
 1. Shape: subject, place, period. `Hourly weather at Prague Airport (LKPR), 2000-2026`.
@@ -520,19 +516,51 @@ interpret them.
    number, a run date.
 5. Sentence case. No trailing full stop.
 
+**Primary category — required before recipe registration**
+Run `mr-data dataset categories --json` to read the fixed vocabulary and its definitions,
+exclusions and examples. Choose exactly one primary category by interpreting the dataset purpose,
+tables and measured entities. Save it with `mr-data dataset set DATASET_ID --category CATEGORY_ID`
+and verify `category_id` in `dataset show`. Put this dataset’s stable ID in
+`dataset.id` in the recipe file (the HTTP command wraps it under `document`) when registering; enforced builds cannot bind by name alone. The updated recipe client requests server enforcement
+of this saved category; it never guesses from tags. Draft pages may remain unassigned during research.
+
+The allowed IDs and labels are:
+
+| ID | Category |
+| --- | --- |
+| `climate-environment` | Climate & Environment |
+| `crypto` | Crypto |
+| `finance` | Finance |
+| `economics-business` | Economics & Business |
+| `government-politics` | Government & Politics |
+| `people-society` | People & Society |
+| `health-medicine` | Health & Medicine |
+| `science-research` | Science & Research |
+| `technology` | Technology |
+| `energy` | Energy |
+| `agriculture-food` | Agriculture & Food |
+| `transport-mobility` | Transport & Mobility |
+| `places-infrastructure` | Places & Infrastructure |
+| `education` | Education |
+| `sports` | Sports |
+| `arts-media-entertainment` | Arts, Media & Entertainment |
+| `law-public-safety` | Law & Public Safety |
+| `other` | Other |
+
+Classify the main subject, not the source, cadence, file format or secondary variables. Bitcoin
+price markets are Crypto; airport weather observations are Climate & Environment; crop yields
+with weather covariates are Agriculture & Food; Steam player counts are Arts, Media & Entertainment.
+For mixed datasets, use their principal analytical purpose. If none fits, choose Other and record
+why the listed definitions exclude the subject. Uncertainty alone is not a reason to use Other.
+Preserve the saved category on routine refreshes and recipe revisions. Reconsider it only when
+the dataset's principal subject changes. Never invent, combine or subdivide category IDs.
+
 **Topic rules**
-1. 3 to 8 topics, comma-separated, no duplicates.
-2. Topic one is the category, chosen from this list and no other:
-   `weather, climate, air quality, hydrology, energy, transport, aviation, public health,
-   economics, finance, elections, sports, prediction markets, demographics, geospatial,
-   agriculture`.
-3. Then, in this order, one each where it applies: the city or region, the country, the
-   phenomenon or instrument, the publisher or programme. After those, up to three more, each a
-   distinct thing a person would type that none of the earlier topics already names.
-4. Form: lowercase; ASCII letters, digits, spaces and hyphens only; one to three words; at most
-   40 characters.
-5. Never: a column name, a unit, a bare year, a second word from the category list, a market or
-   venue name unless the dataset is about that market.
+1. Write 3 to 8 useful descriptive tags, comma-separated, without duplicates.
+2. Tags are search keywords, not categories. No tag position determines the primary category.
+3. Use relevant subject, place, phenomenon, instrument or programme names.
+4. Use lowercase ASCII letters, digits, spaces and hyphens; at most 40 characters per tag.
+5. Omit column names, units, bare years and redundant terms.
 
 **The dataset description is the documentation, and it is read by people and by search engines.**
 `dataset.description` is Markdown, rendered on the dataset page with headings and lists, and it
@@ -597,9 +625,8 @@ choice. Give `table.description` the grain and the window.
 - title: at most 60 characters, contains the place, contains no colon, pipe or exclamation mark,
   no brand, no "dataset" or "data", no version number, no run date
 - opening paragraph: 150 to 240 characters, first full stop before character 200, no Markdown
-- topics: 3 to 8, no duplicates, first one is on the category list, all lowercase, each at most
-  40 characters, none is a column name, a unit, a year, a second word from the category list, or
-  a market or venue name unless the dataset is about that market
+- category: exactly one fixed ID saved and verified before recipe registration
+- topics: 3 to 8 descriptive tags, no duplicates, lowercase, at most 40 characters each
 - licence: an SPDX identifier the sources actually grant, or `--license` left off
 - every `##` heading in DESCRIPTION.md is one of the five, in that relative order, none repeated
 
@@ -1191,14 +1218,16 @@ changelog entry, and the run's own log is already that.
 
 **A `clarification` block with a `question` and no `answer` is a PENDING question**, and it is the
 one shape here that is a way to ask: the Dataset page shows it, and the chat asks it in the same
-words. Write one only at a genuine fork, as stage 3 defines one. The answer is a REVISION of that
+words. Write one for each question [Dataset discovery](#dataset-discovery) has you ask — a
+dataset-shaping choice the evidence cannot settle. The answer is a REVISION of that
 cell — `--revise CELL_ID`, the same blocks again with `answer` and `answered_at` filled in — so the
 page shows the question settled rather than a second cell nothing ties to the first. `answered_at`
 is admitted only beside `answer`, because a time with no answer records that the question was
 settled by nobody, and is a UTC timestamp ending in `Z`, as `2026-09-01T10:00:00Z`. Everything
-under [Autonomous delivery](#autonomous-delivery) still holds — do not put a source, Reader,
-engine, format, workspace, retry tactic, cleaning operation or repair to the user, in a block or
-anywhere else.
+under [Autonomous delivery](#autonomous-delivery) still holds — do not put a Reader, engine,
+parsing format, workspace, retry tactic or semantics-preserving repair to the user, in a block or
+anywhere else. Source preferences and cleaning choices that change the meaning are the user's,
+and go in a block.
 
 `--phase` binds the cell to one build stage — `acquire`, `parse`, `transform`, `check`, `persist` —
 so a reader can group a run's record by stage. `--checkpoint` places the cell beside the event it
@@ -1444,41 +1473,79 @@ each table rather than what the exit code said about the set. A table that is li
 `archive` until it is withdrawn, and a table a run is still writing to is refused by both until the
 run has finished; neither refusal deletes anything, and neither ends the command.
 
+## Dataset discovery
+
+Make the user a participant in shaping the first dataset. Before registering a recipe, review the
+request and prior answers for choices whose alternatives would materially change the result. Lead
+with the unresolved choices most likely to make the first dataset wrong. The agent chooses the
+questions and follow-ups; this is a decision guide, not a form to recite:
+
+- **Purpose:** What analysis, decision, or downstream use should the dataset support? Ask when it
+  helps choose between plausible definitions or shapes, not for background already supplied.
+- **Table structure and joins:** One joined table, separate related tables, or both? If joining,
+  what should match, and should unmatched records remain? Explain when a join could duplicate rows
+  or lose records; do not require the user to write SQL or select a join algorithm.
+- **Granularity:** What does one row represent — an event, a company, a station-hour, a country-year?
+  Resolve raw observations versus aggregates, time frequency, and geographic detail where relevant.
+- **Sources:** Are particular publishers, uploaded files, or sources required or excluded, or should
+  the agent recommend them? Research concrete source options yourself. Ask about consequential choices
+  such as official but delayed data versus broader recent coverage, or a source requiring access.
+- **Coverage:** Which entities, geography, date range, and inclusion/exclusion rules matter? For
+  changing data, distinguish a fixed historical snapshot from continuing updates.
+- **Fields and definitions:** Which measures and identifiers are essential? Resolve ambiguous
+  metrics, units, currencies, time zones, or adjusted versus raw values when they affect the use.
+- **Quality and tradeoffs:** Is incomplete coverage acceptable? Ask before dropping unmatched rows,
+  filling missing values, estimating data, or substituting a proxy when that changes the meaning.
+- **Delivery constraints:** Ask about freshness, destination, format, or scale/budget only when
+  relevant and not already specified. Existing spend and cadence decisions still apply.
+
+For every discovery question, including the initial questions before probing, check attendance with
+`mr-data clarify --attended --question TEXT` when a person is in chat. Record a pending
+`clarification` block on the dataset and ask the same wording in chat. Record the answer by revising
+that same cell, as in stage 3; a plain narrative note alone does not mark a question as pending.
+The command checks whether asking is allowed; the host conversation asks and waits for the answer.
+
+Usually ask one to three related questions per turn, with short bold labels such as **Table shape**,
+**Sources**, and **Granularity**. Use plain language and domain-specific examples. Offer a recommended
+option and explain its consequence when evidence supports it; allow a free-text answer or "you
+choose." Do not overwhelm the user with the entire list, repeatedly ask answered questions, or ask
+about facts you can determine from source inspection. There is no total question quota: follow up
+when an answer or source finding exposes another consequential decision.
+
+Wait for answers to the questions you ask before committing dependent work. Continue independent
+research where useful, but do not register or run the first recipe with unanswered material
+questions. An elapsed wait is not permission to assume an answer. If the request is already specific
+enough, summarize its shape and proceed. If the user explicitly delegates choices or asks for no
+questions, use evidence-backed defaults and state them without forcing another confirmation.
+
+Before the first build, give a compact recap of purpose, table shape, row grain, sources, coverage,
+and important fields or limitations. Highlight unresolved decisions as questions; otherwise proceed
+on the answers already given. During verification compare the sample with this agreed shape, not
+just whether the run succeeded. If research or the sample requires a material semantic change,
+return to the user with concrete alternatives before building a revised recipe; use the existing
+contract revision/approval path once a recipe has been registered.
+
 ## Autonomous delivery
 
 Own the run until the requested outcome is delivered: a run that succeeded, its declared checks
-read, its coverage inspected, and its refresh cadence recorded when the user has said what it should
-be. What is live is what Studio returns as live; catch-up and continuing freshness require separate
-durable evidence from the deployed component that owns them. The allowed user pauses
-are the cadence decision, the decision to start a full run after a TRUNCATED sample, and a spend
-confirmation on a run over the threshold. Do not ask the user to choose sources, Readers, engines,
-formats, workspaces, retry tactics, cleaning operations, or validation repairs. Make those
-operational decisions from the question, observed evidence, registered capabilities, and
-deterministic receipts.
-
-**The decision point replaces the one-question rule.** There used to be a budget of exactly one
-clarifying question, and it was the wrong instrument: it made the agent hoard its one question
-against a fork that might arrive later, and it left the user with no view of the search at all
-until the page was finished. What stage 3 asks for instead is a STATEMENT — sources chosen, sources
-rejected or blocked with the concrete unblock step, the assumption taken about window and grain —
-posted to the dataset's record and to the chat, after which the build continues without waiting.
-A person who disagrees with it says so; a person who does not is not made to answer anything.
-
-A question is still asked, and it is asked at a genuine FORK: two answers that produce two
-different tables, and no evidence that can settle it. `Throughput: movements or passengers?` is
-such a fork. Which Reader parses the feed is not, and neither is anything else named above; those
-stay banned because the evidence settles them. There is no budget of one — a build with two real
-forks asks about both — but a fork is rare, and a question that the evidence could have answered is
-still the failure this section is about. Ask before registering, while nothing is running and
-nothing has been spent, write it as a pending `clarification` block on the dataset, and carry the
-answer back as a revision of that cell.
+read, its coverage inspected against the agreed dataset shape, and its refresh cadence recorded
+when the user has said what it should be. What is live is what Studio returns as live; catch-up and
+continuing freshness require separate durable evidence from the deployed component that owns them.
+Dataset discovery questions, material semantic revisions, cadence decisions, full runs after a
+TRUNCATED sample, and spend confirmations are valid user pauses. Own the operational choices:
+Readers, engines, parsing formats, workspace plumbing, retry tactics, and repairs that preserve the
+agreed semantics. Source preferences and cleaning choices that change the meaning belong in the
+user conversation, even when the agent could technically choose for them.
 
 `mr-data clarify --question TEXT` says whether to ask at all. It exits non-zero when no person can
 answer, which is what a scheduled or headless session reports, and when `--recipe RECIPE_ID` names
 a registered recipe, because the answer to a question asked after the contract is a revision and
-goes through the contract approval instead. On a non-zero exit ask nothing: decide from the
-evidence, write the assumption into the recipe document, and state it as a limitation in the final
-report. `mr-data clarify --question TEXT --answer TEXT` composes the exchange as one `clarification`
+goes through the contract approval instead. When no person can answer, do not wait: use the
+existing brief and evidence to make choices within the requested scope, record assumptions, and
+report limitations. Do not silently replace explicit requirements. When the recipe is already
+registered, use the revision/approval path for material changes rather than treating the refusal
+as permission to choose new semantics. Other command failures are not evidence that the user is
+unavailable. `mr-data clarify --question TEXT --answer TEXT` composes the exchange as one `clarification`
 block. Carry its wording into the recipe document the answer settles — the question text, the table
 description, the column description — so the assumption the table rests on reads the same in the
 document, in the Dataset, and in the report.
@@ -1596,11 +1663,34 @@ Write it as a source whose connector is that coordinate plus a `collection` memb
   merged relation.
 - The Reader pin is required, must be `html.web_extract`, and is the SAME settings for every page.
 
+**`limits.max_requests` is charged per HTTP request, redirect hops and retries included** — a page
+that redirects once costs two. Registration only checks that
+`discovery.max_requests + pages.max_fetches_per_run` fits inside it, which is necessary and NOT
+sufficient: a budget written to exactly that sum runs out early on any publisher that redirects,
+and the run stops with `budget_exhausted: "max_requests"` short of the pages it was allowed. Where
+redirects are likely — trailing slashes, `http`→`https`, `www`, which is most publishers — write
+`limits.max_requests >= discovery.max_requests + 2 * max_fetches_per_run`. A budget larger than the
+corpus can absorb costs nothing.
+
 The relation the transform reads is the reader's columns plus `page_id`, `page_url`,
 `page_fetched_at`, `page_content_sha256`, `page_revision`, `page_discovered_at` and `page_ordinal`.
 Those names are reserved: a reader that declares one is a registration refusal. Select them in the
 transform so every row on the table says which page and which revision produced it.
 
+**All seven arrive as text and must be CAST.** The merged relation is sealed as CSV, so every
+column reaches the transform as `VARCHAR`; selecting `page_ordinal` bare while declaring the column
+`integer` fails the build with `TRANSFORM_COLUMN_TYPE_MISMATCH`. Write
+`cast(page_ordinal as integer)`, `cast(page_revision as integer)` and
+`cast(page_fetched_at as timestamp with time zone)` — a declared `timestamp` column admits
+`TIMESTAMP WITH TIME ZONE` and nothing else.
+
+**A listing that could not be read says so.** The coverage block carries `discovery_failure`:
+`null`, or the code and one sentence for the first listing request the run was refused. A refresh
+that already holds pages keeps every row and succeeds; a FIRST run whose listing is refused has
+nothing to continue from and fails with `COLLECTION_DISCOVERY_FAILED` rather than sealing an empty
+corpus. If a run reports zero discovered, read that member before concluding the publisher's index
+is empty.
+
 **Expect the first run not to finish, and say so.** A run that spends its fetch budget still
 SUCCEEDS; its coverage block reports `complete: false` and names the budget that ended it. The
 next refresh continues from the ledger — new pages first, then failures oldest first, then

{mostlyright_data-0.17.5 → mostlyright_data-0.17.7}/src/mostlyright/data_harness/hosted_crawler_fetch.py

@@ -10,7 +10,7 @@ from collections.abc import Callable, Iterator
 from concurrent.futures import ThreadPoolExecutor
 from contextlib import contextmanager
 from contextvars import ContextVar
-from dataclasses import dataclass
+from dataclasses import dataclass, replace
 from urllib.parse import urlsplit
 
 from mostlyright.data_harness.acquisition.http import (
@@ -271,6 +271,7 @@ def _public_https_retriever(
     transport: StdlibPinnedTransport | None = None,
     connect_proxy: ConnectProxyMember | None = None,
     connect_proxy_config_digest: str | None = None,
+    allowed_media_types: tuple[str, ...] | None = None,
 ) -> PinnedHttpsRetriever:
     """The one reviewed public-HTTPS retriever construction.
 
@@ -279,6 +280,13 @@ def _public_https_retriever(
     limiter -- and differ only in the three explicit caps a caller names: the byte ceiling
     and the two stall timeouts.  Stated once so the ranged path cannot drift into a second,
     subtly different transport policy.
+
+    ``allowed_media_types`` NARROWS THAT ONE TERM AND NOTHING ELSE, for a caller whose response
+    is not the one the query's own pin describes.  A page collection's LISTING request is the
+    case: the source is pinned to an HTML Reader because its detail pages are HTML, and the index
+    that names those pages is JSON or a sitemap, read inertly by the driver and handed to no
+    Reader at all.  Every other term above is the reviewed one, so what the narrowing changes is
+    which representations the origin may answer with and not how the request is made.
     """
 
     return PinnedHttpsRetriever(
@@ -288,17 +296,24 @@ def _public_https_retriever(
             allowed_ports=frozenset({443}),
         ),
         transport=transport or StdlibPinnedTransport(),
-        limits=RetrievalLimits(
-            max_response_bytes=max_response_bytes,
-            max_redirects=3,
-            max_requests=4,
-            connect_timeout_seconds=5.0,
-            read_timeout_seconds=read_timeout_seconds,
-            total_timeout_seconds=total_timeout_seconds,
-            max_concurrency=PUBLIC_HTTPS_MAX_CONCURRENCY,
-            min_interval_seconds=1.0,
-            allowed_media_types=_public_expected_media_types(request.query),
-            user_agent=PUBLIC_HTTPS_USER_AGENT,
+        limits=replace(
+            RetrievalLimits(
+                max_response_bytes=max_response_bytes,
+                max_redirects=3,
+                max_requests=4,
+                connect_timeout_seconds=5.0,
+                read_timeout_seconds=read_timeout_seconds,
+                total_timeout_seconds=total_timeout_seconds,
+                max_concurrency=PUBLIC_HTTPS_MAX_CONCURRENCY,
+                min_interval_seconds=1.0,
+                allowed_media_types=_public_expected_media_types(request.query),
+                user_agent=PUBLIC_HTTPS_USER_AGENT,
+            ),
+            **(
+                {}
+                if allowed_media_types is None
+                else {"allowed_media_types": tuple(allowed_media_types)}
+            ),
         ),
         limiter=_PUBLIC_HTTPS_LIMITER,
         connect_proxy=connect_proxy,
@@ -407,12 +422,16 @@ def fetch_crawler_source(
     *,
     expected_egress_policy_attestation: str,
     retriever: PinnedHttpsRetriever | None = None,
+    allowed_media_types: tuple[str, ...] | None = None,
 ) -> RetrievedBytes:
     """Fetch one exact reviewed URL for the crawler child to normalize.
 
     The retrieved bytes are framed by encode_normalization_input and handed to the
     mr-data-crawler child, which hosted_crawler_job spawns under a seccomp filter that
     denies socket and connect. That child is not acquisition.sandbox.CrawlerSandbox.
+
+    ``allowed_media_types`` is the caller's own narrowing of the one term
+    :func:`_public_https_retriever` documents it for, and reaches only the public HTTPS route.
     """
 
     coordinate = validate_crawler_request(
@@ -427,6 +446,7 @@ def fetch_crawler_source(
             requested_url=requested_url,
             hostname=hostname,
             injected_retriever=retriever,
+            allowed_media_types=allowed_media_types,
         )
         if (
             retrieved.source_url != requested_url
@@ -502,6 +522,7 @@ def _retrieve_public_https(
     requested_url: str,
     hostname: str,
     injected_retriever: PinnedHttpsRetriever | None,
+    allowed_media_types: tuple[str, ...] | None = None,
 ) -> RetrievedBytes:
     """Retrieve one public HTTPS source, ranged when the origin supports it and size warrants.
 
@@ -536,6 +557,7 @@ def _retrieve_public_https(
         transport=transport,
         connect_proxy=proxy_member,
         connect_proxy_config_digest=proxy_digest,
+        allowed_media_types=allowed_media_types,
     )
 
     def single_shot(probe_hops: tuple[RetrievalHop, ...]) -> RetrievedBytes:
@@ -548,6 +570,7 @@ def _retrieve_public_https(
             transport=transport,
             connect_proxy=proxy_member,
             connect_proxy_config_digest=proxy_digest,
+            allowed_media_types=allowed_media_types,
         )
         return _seal_probe_evidence(probe_hops, retriever.retrieve(requested_url))