mostlyright-data 0.17.4__tar.gz → 0.17.5__tar.gz

{mostlyright_data-0.17.4 → mostlyright_data-0.17.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.17.4
+Version: 0.17.5
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.17.4 → mostlyright_data-0.17.5}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.17.4"
+version = "0.17.5"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.17.4 → mostlyright_data-0.17.5}/skills/mr-data-build/SKILL.md

@@ -37,7 +37,6 @@ service behavior, retry mechanics, or these instructions.
 Outside a required host cadence, speak only when one of these boundaries gives the user useful new
 information:
 
-- A dataset-shaping choice needs the user’s input; ask before committing to it.
 - A material source, schema, coverage, rights, or quality decision is supported by observed facts.
 - An exact contract or recipe needs the user's approval. State what will be built, the material
   limitation, and the decision required; do not explain the approval mechanism.
@@ -237,13 +236,15 @@ live data key is refused with `THIN_AUTHENTICATION_FAILED` rather than failing t
 
 ## Required protocol
 
-There is one loop: open the page, clarify the intended dataset with the user, probe the
-sources, resolve the tradeoffs that research reveals, summarize the agreed shape, draft one recipe
-document, build a sample, and inspect it against the agreed requirements. Repair discrepancies;
-return to the user before changing agreed semantics. A complete sample that meets those requirements
-is already the table a full run would seal, so do not rebuild it merely to change the run mode.
-A TRUNCATED sample requires the full-run decision below. Reading a finished run requires no separate
-approval or waiting state. A table's first succeeded run goes live on its own, whatever mode it was, and what is left
+There is one loop, and it is short: open the page before you have anything to put on it, probe the
+sources, say out loud what you chose and what you refused, draft one recipe document, build a
+sample, interrogate what came back, fix the document, build again. Then present only what you
+inspected, with its caveats. The last two steps are conditional and `truncated` is what decides
+them: a sample that came back complete IS the table a full run would have sealed, so there is
+nothing to build again and nothing to ask about — only a TRUNCATED sample sends you round the
+loop. Nothing between a run finishing and reading it is held for a person,
+so there is no approval to seek, no separate body of evidence to read back, and no state to wait
+for. A table's first succeeded run goes live on its own, whatever mode it was, and what is left
 for a person is [Promote](#promote) — the cadence it refreshes on and the reasoning behind it.
 
 **The page is the work, and it exists first.** A dataset used to come into being as a side effect
@@ -278,10 +279,11 @@ again, and the failure triples on recent runs tell you what the backend is refus
 `SANDBOX_MEMORY_BOUNDARY` on somebody else's recipe an hour ago means the room is unwell and your
 first failure will not be your document's fault. Keep this check silent.
 
-**Then clarify the intended dataset before probing.** Follow [Dataset discovery](#dataset-discovery):
-use the request and conversation to identify consequential missing choices, ask the highest-impact
-questions, and wait for answers before work that depends on them. Independent source research may
-continue while waiting. Revisit the remaining choices when research provides concrete alternatives.
+**Then settle anything only the user can settle, before probing.** Read the request for a word with
+two defensible readings, a population whose edges are unstated, or a window nobody named.
+`mr-data clarify --question TEXT` says whether there is anybody to ask and whether it is still early
+enough; [Autonomous delivery](#autonomous-delivery) is the rule for what may be asked and what stays
+yours to decide.
 
 Pass `--json` to automated commands. A command receipt is the only record that a Harness action
 succeeded.
@@ -387,10 +389,10 @@ around a probe from an hour ago still being warm.
 
 Where a source needs a credential, enrol it by name first — see
 [Source credentials](#source-credentials) — and reference the name from the recipe document. Where
-neither a probe nor a run can reach the bytes, because the source is a bounded collection or is
-not registered in the workspace, establish shape from the provider's own documentation and
-published schema, say plainly that the shape is documented rather than observed, and let the hosted
-Clean room be the first thing that opens the bytes. Named messages out of one large object are
+neither a probe nor a run can reach the bytes, because the source is not registered in the
+workspace, establish shape from the provider's own documentation and published schema, say
+plainly that the shape is documented rather than observed, and let the hosted Clean room be the
+first thing that opens the bytes. Named messages out of one large object are
 [`acquire-slices`](#not-hosted-yet), which is not hosted yet.
 
 Hold the comparison in the shape [Narrating the run](#narrating-the-run) takes: one `source` block
@@ -400,8 +402,10 @@ blocks file; stage 3 is what puts them on the page.
 ### 3. Decide — say what you chose, what you refused, and what unblocks it
 
 **Post one Sources and decisions message before you register anything.** It goes to two places, in
-the same words: the dataset's own decision record, and the chat. Summarize the user’s answers, the source evidence, and any remaining
-assumptions so the user can see what the first dataset will contain.
+the same words: the dataset's own decision record, and the chat. This is the decision point that
+replaced the old one-question rule, and the reason for the change is that a question is not what a
+person mostly needs — what they need is to see the search, while there is still time to disagree
+with it.
 
 ```sh
 mr-data dataset note DATASET_ID --heading "Sources and decisions" \
@@ -413,9 +417,8 @@ EOF
 The cell carries a `source` block for every feed evaluated — the ones being used, the ones
 considered, and the ones rejected with the clause that ruled each out — and a `decision` block for
 every grain, key, join and cutoff settled. Post the same content in the chat as one short message.
-If a material choice is still unresolved, ask and wait before registering the recipe. If the
-user has already settled the choices or delegated them, proceed without a redundant confirmation.
-Silence after a proposal is not an answer to a pending question.
+Then **keep going**: this is a statement, not a gate, and waiting for a reply to it is the pause
+this document does not allow.
 
 **Blocked sources: always say what unlocks it and how to enrol the credential.** A source you could
 not reach because it needs an API key, an account or an approval is not a dead end and must never
@@ -425,21 +428,23 @@ who issues the credential, where to apply for it, and the exact command that enr
 source with no route back is a decision the user cannot revisit; one with a route back is a
 decision they can.
 
-**Resolve research tradeoffs with the user.** Apply [Dataset discovery](#dataset-discovery) when
-sources cannot deliver the requested grain, coverage, fields, or join. Explain the supported options
-and their consequences before changing the intended dataset. Record each pending question as a
-`clarification` block carrying `question` and no `answer`, and ask it in chat in the same words.
-When answered, revise that same cell using its actual identifier (not a new unrelated note):
+**Ask only at a genuine fork.** A fork is a question whose two answers produce two different tables
+and which no evidence can settle — what an ambiguous word in the request means, how wide the
+population is, how far back it goes. Write it as a `clarification` block carrying `question` and no
+`answer`, which is what the page shows as a pending question, and ask it in the chat in the same
+words. `mr-data clarify --question TEXT` says whether there is anybody to ask and whether it is
+still early enough. When the answer comes back, write it as the answer on that block:
 
 ```sh
 mr-data dataset note DATASET_ID --heading "The stage grain is settled" \
-  --revise PENDING_CELL_ID --blocks-file ANSWERED.json --json <<'EOF'
+  --revise "one-fork-to-settle" --blocks-file ANSWERED.json --json <<'EOF'
 Daily maxima, on the user's answer.
 EOF
 ```
 
-Carry the answers into the recipe’s question, table and column descriptions, source selection,
-transform, and checks as appropriate. Record assumptions separately from user decisions.
+Everything under [Autonomous delivery](#autonomous-delivery) still holds: which source, Reader,
+engine, format, workspace, retry tactic or cleaning operation to use is settled by the evidence and
+is never a fork. If nothing is genuinely forked, ask nothing and carry on.
 
 ### 4. Draft — one recipe document, one call
 
@@ -794,9 +799,12 @@ one", not as a failure: a document declaring 8 MiB over a feed that answers 40 K
 **If a source really does deliver more than the Reader holds, do not answer it by lowering the
 ceiling on that source:** that does not fetch less data, it truncates the sample or refuses the
 full run. Split the window across several sources — one per month, per region, or per whatever the
-publisher paginates on — and union them in a first transform step. A recipe names up to 64 sources,
-which is a great many windows. [The recipe document](../../docs/RECIPE-DOCUMENT.md) carries the
-measured numbers behind the ceiling.
+publisher paginates on — and union them in a first transform step. A recipe names up to 256
+sources, which is a great many windows. [The recipe document](../../docs/RECIPE-DOCUMENT.md)
+carries the measured numbers behind the ceiling.
+
+**A corpus of many pages behind one index is a COLLECTION, not many sources.** See
+[Authoring a collection source](#authoring-a-collection-source).
 
 **The dataset's record is the primary one; a run's own record is for what happened inside that
 run.** The sources, the decisions and the questions went onto the dataset in stage 3, where they
@@ -1436,79 +1444,41 @@ each table rather than what the exit code said about the set. A table that is li
 `archive` until it is withdrawn, and a table a run is still writing to is refused by both until the
 run has finished; neither refusal deletes anything, and neither ends the command.
 
-## Dataset discovery
-
-Make the user a participant in shaping the first dataset. Before registering a recipe, review the
-request and prior answers for choices whose alternatives would materially change the result. Lead
-with the unresolved choices most likely to make the first dataset wrong. The agent chooses the
-questions and follow-ups; this is a decision guide, not a form to recite:
-
-- **Purpose:** What analysis, decision, or downstream use should the dataset support? Ask when it
-  helps choose between plausible definitions or shapes, not for background already supplied.
-- **Table structure and joins:** One joined table, separate related tables, or both? If joining,
-  what should match, and should unmatched records remain? Explain when a join could duplicate rows
-  or lose records; do not require the user to write SQL or select a join algorithm.
-- **Granularity:** What does one row represent — an event, a company, a station-hour, a country-year?
-  Resolve raw observations versus aggregates, time frequency, and geographic detail where relevant.
-- **Sources:** Are particular publishers, uploaded files, or sources required or excluded, or should
-  the agent recommend them? Research concrete source options yourself. Ask about consequential choices
-  such as official but delayed data versus broader recent coverage, or a source requiring access.
-- **Coverage:** Which entities, geography, date range, and inclusion/exclusion rules matter? For
-  changing data, distinguish a fixed historical snapshot from continuing updates.
-- **Fields and definitions:** Which measures and identifiers are essential? Resolve ambiguous
-  metrics, units, currencies, time zones, or adjusted versus raw values when they affect the use.
-- **Quality and tradeoffs:** Is incomplete coverage acceptable? Ask before dropping unmatched rows,
-  filling missing values, estimating data, or substituting a proxy when that changes the meaning.
-- **Delivery constraints:** Ask about freshness, destination, format, or scale/budget only when
-  relevant and not already specified. Existing spend and cadence decisions still apply.
-
-For every discovery question, including the initial questions before probing, check attendance with
-`mr-data clarify --attended --question TEXT` when a person is in chat. Record a pending
-`clarification` block on the dataset and ask the same wording in chat. Record the answer by revising
-that same cell, as in stage 3; a plain narrative note alone does not mark a question as pending.
-The command checks whether asking is allowed; the host conversation asks and waits for the answer.
-
-Usually ask one to three related questions per turn, with short bold labels such as **Table shape**,
-**Sources**, and **Granularity**. Use plain language and domain-specific examples. Offer a recommended
-option and explain its consequence when evidence supports it; allow a free-text answer or “you
-choose.” Do not overwhelm the user with the entire list, repeatedly ask answered questions, or ask
-about facts you can determine from source inspection. There is no total question quota: follow up
-when an answer or source finding exposes another consequential decision.
-
-Wait for answers to the questions you ask before committing dependent work. Continue independent
-research where useful, but do not register or run the first recipe with unanswered material
-questions. An elapsed wait is not permission to assume an answer. If the request is already specific
-enough, summarize its shape and proceed. If the user explicitly delegates choices or asks for no
-questions, use evidence-backed defaults and state them without forcing another confirmation.
-
-Before the first build, give a compact recap of purpose, table shape, row grain, sources, coverage,
-and important fields or limitations. Highlight unresolved decisions as questions; otherwise proceed
-on the answers already given. During verification compare the sample with this agreed shape, not
-just whether the run succeeded. If research or the sample requires a material semantic change,
-return to the user with concrete alternatives before building a revised recipe; use the existing
-contract revision/approval path once a recipe has been registered.
-
 ## Autonomous delivery
 
 Own the run until the requested outcome is delivered: a run that succeeded, its declared checks
-read, its coverage inspected against the agreed dataset shape, and its refresh cadence recorded
-when the user has said what it should be. What is live is what Studio returns as live; catch-up and
-continuing freshness require separate durable evidence from the deployed component that owns them.
-Dataset discovery questions, material semantic revisions, cadence decisions, full runs after a
-TRUNCATED sample, and spend confirmations are valid user pauses. Own the operational choices:
-Readers, engines, parsing formats, workspace plumbing, retry tactics, and repairs that preserve the
-agreed semantics. Source preferences and cleaning choices that change the meaning belong in the
-user conversation, even when the agent could technically choose for them.
+read, its coverage inspected, and its refresh cadence recorded when the user has said what it should
+be. What is live is what Studio returns as live; catch-up and continuing freshness require separate
+durable evidence from the deployed component that owns them. The allowed user pauses
+are the cadence decision, the decision to start a full run after a TRUNCATED sample, and a spend
+confirmation on a run over the threshold. Do not ask the user to choose sources, Readers, engines,
+formats, workspaces, retry tactics, cleaning operations, or validation repairs. Make those
+operational decisions from the question, observed evidence, registered capabilities, and
+deterministic receipts.
+
+**The decision point replaces the one-question rule.** There used to be a budget of exactly one
+clarifying question, and it was the wrong instrument: it made the agent hoard its one question
+against a fork that might arrive later, and it left the user with no view of the search at all
+until the page was finished. What stage 3 asks for instead is a STATEMENT — sources chosen, sources
+rejected or blocked with the concrete unblock step, the assumption taken about window and grain —
+posted to the dataset's record and to the chat, after which the build continues without waiting.
+A person who disagrees with it says so; a person who does not is not made to answer anything.
+
+A question is still asked, and it is asked at a genuine FORK: two answers that produce two
+different tables, and no evidence that can settle it. `Throughput: movements or passengers?` is
+such a fork. Which Reader parses the feed is not, and neither is anything else named above; those
+stay banned because the evidence settles them. There is no budget of one — a build with two real
+forks asks about both — but a fork is rare, and a question that the evidence could have answered is
+still the failure this section is about. Ask before registering, while nothing is running and
+nothing has been spent, write it as a pending `clarification` block on the dataset, and carry the
+answer back as a revision of that cell.
 
 `mr-data clarify --question TEXT` says whether to ask at all. It exits non-zero when no person can
 answer, which is what a scheduled or headless session reports, and when `--recipe RECIPE_ID` names
 a registered recipe, because the answer to a question asked after the contract is a revision and
-goes through the contract approval instead. When no person can answer, do not wait: use the
-existing brief and evidence to make choices within the requested scope, record assumptions, and
-report limitations. Do not silently replace explicit requirements. When the recipe is already
-registered, use the revision/approval path for material changes rather than treating the refusal
-as permission to choose new semantics. Other command failures are not evidence that the user is
-unavailable. `mr-data clarify --question TEXT --answer TEXT` composes the exchange as one `clarification`
+goes through the contract approval instead. On a non-zero exit ask nothing: decide from the
+evidence, write the assumption into the recipe document, and state it as a limitation in the final
+report. `mr-data clarify --question TEXT --answer TEXT` composes the exchange as one `clarification`
 block. Carry its wording into the recipe document the answer settles — the question text, the table
 description, the column description — so the assumption the table rests on reads the same in the
 document, in the Dataset, and in the report.
@@ -1603,6 +1573,53 @@ What it costs the protocol, and what to do instead:
 - Preserve event time, available time, acquisition time, source revision, and timezone. Do not use
   post-cutoff information in targets, labels, features, joins, or validation decisions.
 
+### Authoring a collection source
+
+A corpus published as an index plus many detail pages is **one** source through
+`public.https.collection@2.0.0`, not one source per page. Source count and publisher count are
+unaffected by how many pages sit behind it: 2,819 pages is one source, one publisher, one line on
+the dataset's sources card.
+
+Write it as a source whose connector is that coordinate plus a `collection` member:
+
+- `collection.discovery` says how the corpus is LISTED — `json_api` with RFC 6901 pointers,
+  `html_index` with the reader's closed selectors, `sitemap`, or an `explicit` member list — and
+  which pagination grammar the listing uses. The run follows only that grammar and never a link
+  found inside a detail page.
+- `collection.pages` says what may be fetched and how much: `allowed_origins` (which must include
+  the discovery address's own origin), `max_pages`, `max_fetches_per_run`, `concurrency` (1 or 2),
+  `min_interval_seconds` as a whole number of seconds (one or more), `request_timeout_seconds`,
+  `retries`, and a `revisit` rule. Only `allowed_origins` is required; every other member has a
+  stated meaning when it is left out.
+- `limits.max_requests` is REQUIRED and bounds every request the run makes, discovery and detail
+  pages together. `limits.max_source_bytes` bounds the fetched bytes; `limits.max_rows` clamps the
+  merged relation.
+- The Reader pin is required, must be `html.web_extract`, and is the SAME settings for every page.
+
+The relation the transform reads is the reader's columns plus `page_id`, `page_url`,
+`page_fetched_at`, `page_content_sha256`, `page_revision`, `page_discovered_at` and `page_ordinal`.
+Those names are reserved: a reader that declares one is a registration refusal. Select them in the
+transform so every row on the table says which page and which revision produced it.
+
+**Expect the first run not to finish, and say so.** A run that spends its fetch budget still
+SUCCEEDS; its coverage block reports `complete: false` and names the budget that ended it. The
+next refresh continues from the ledger — new pages first, then failures oldest first, then
+revisits — and every row of a page this run did not fetch survives byte for byte. Report the
+shortfall as what it is, a backfill in progress, not as a failure and not as a complete corpus.
+
+A page that disappears from the index deletes nothing. A page answering with different content
+replaces exactly its own rows, increments `page_revision`, and keeps the previous digest: that is
+a correction, and it is worth saying on the run record when it happens.
+
+Before registering it, read the discovery block back the way a run will: the address the first
+request goes to, whether the pagination parameter is already written into that address, whether
+the records pointer names an array, and whether `discovery.max_requests` plus
+`pages.max_fetches_per_run` fits inside `limits.max_requests`. Registration refuses all four, and
+they are the four that are cheapest to get right before a run spends anybody's rate limit.
+
+The full contract, including every bound and the coverage block's members, is
+[Source collections](../../docs/SOURCE-COLLECTIONS.md).
+
 ## Readers
 
 `docs/READERS.md` defines Reader behavior. Certified Readers may be pinned on an exact
@@ -1697,6 +1714,7 @@ decoder. Zero matches and multiple matches both refuse.
 | `xml.tabular@1.0.0` | Absolute record element path, declared namespace prefixes, and relative field selectors ending in `text()` or `@attribute`; fields are required unless `required: false`. |
 | `html.tabular@1.0.0` | Absolute record element path and relative text/attribute field selectors; fields are required unless `required: false`. |
 | `html.web_extract@1.0.0` | `mode: dom` with closed CSS-like record/field selectors, or `mode: json` with an inert JSON script selector and RFC 6901 field pointers. |
+| `html.web_extract@1.1.0` | Same settings as 1.0.0; extracted text follows DOM order, with one space at block-element boundaries and `<br>` and none between inline runs. |
 | `protobuf.tabular@1.0.0` | Digest-pinned base64 FileDescriptorSet, fully-qualified message type, `single` or `length_delimited` framing, and fields with an explicit repeated `explode` or `refuse` choice. |
 | `parquet.tabular@1.0.0` | Optional `columns` selects a bounded ordered Parquet projection. |
 | `xbrl.facts@1.0.0` | `document_kind` (`xbrl` or constrained XML `ixbrl`) and exact namespace/name fact projection; contexts, units, and metadata remain evidence fields, not joins. |

{mostlyright_data-0.17.4 → mostlyright_data-0.17.5}/src/mostlyright/data_harness/cli.py

@@ -575,6 +575,7 @@ def _dispatch(
         or selected_command == "propose"
         or selected_command == "acquire-cancel"
         or selected_command == "clarify"
+        or selected_command == "collection-plan"
         or selected_command in _RESEARCH_COMMANDS
     ):
         # `note`, `propose` and the research commands are the hosted lane's own implementation, run
@@ -586,6 +587,8 @@ def _dispatch(
         # nothing and is identical in both profiles. It refuses in the thin lane's typed vocabulary
         # because that is the vocabulary the agent driving it already branches on, and a fifth
         # refusal class for four sentences would be a fifth remediation family for no reader.
+        # `collection-plan` is the same shape again: it reaches nothing, and it refuses in that
+        # same vocabulary because the agent authoring a collection is already reading it.
         from mostlyright.data_harness.thin.transport import ThinLaneError
 
         handled_errors += (ThinLaneError,)
@@ -754,6 +757,10 @@ def _dispatch(
             if args.command == "crosswalk-inspect":
                 payload["rows"] = [row.to_dict() for row in crosswalk.rows]
             _emit(payload)
+        elif args.command == "collection-plan":
+            from mostlyright.data_harness.thin import collection_plan as _collection_plan
+
+            _emit(_collection_plan.collection_plan(args))
         elif args.command == "recipe-validate":
             recipe = parse_fresh_recipe(_read_json(args.recipe))
             _emit(
@@ -1694,6 +1701,20 @@ def _parser(
     )
     recipe_validate.add_argument("--recipe", type=Path, required=True, help=_RECIPE_FILE)
 
+    # DECLARED BY THE MODULE THAT IMPLEMENTS IT, which is the rule `probe` and `recipe` already
+    # follow. It is registered HERE and not in the hosted lane, for the reason that module's own
+    # header states: rendering the plan parses the recipe with the engine's parser, and `mr-data`
+    # installs without it. This is where `recipe-validate` -- the other command that reads a
+    # recipe and reaches nothing -- already lives.
+    from mostlyright.data_harness.thin import collection_plan as _collection_plan
+
+    _collection_plan.declare_arguments(
+        _add(
+            "collection-plan",
+            help="show what a recipe's collection source would discover, fetch and seal",
+        )
+    )
+
     crosswalk_validate = _add(
         "crosswalk-validate",
         help="validate a canonical authored crosswalk and print its digest",

{mostlyright_data-0.17.4 → mostlyright_data-0.17.5}/src/mostlyright/data_harness/hosted_deploy.py

@@ -30,6 +30,7 @@ from mostlyright.data_harness.hosted_crawler_protocol import (
     PUBLIC_HTTPS_EGRESS_POLICY_ATTESTATION,
 )
 from mostlyright.data_harness.studio_boundary import StudioBoundaryError, client_contract_identity
+from mostlyright.data_harness.thin.user_agent import with_user_agent
 from mostlyright.data_harness.ux.credentials import ResolvedCloudCredentials
 
 CLOUD_TOKEN_PATH = "/api/cli/studio-token"
@@ -178,7 +179,7 @@ class UrlLibTokenExchangeTransport:
         body: bytes | None,
         maximum: int,
     ) -> tuple[int, bytes, Mapping[str, str]]:
-        request = Request(url, data=body, headers=dict(headers), method=method)
+        request = Request(url, data=body, headers=with_user_agent(headers), method=method)
         try:
             response = self._opener.open(request, timeout=REQUEST_TIMEOUT_SECONDS)
         except HTTPError as error: