mostlyright-data 0.17.5__tar.gz → 0.17.6__tar.gz

{mostlyright_data-0.17.5 → mostlyright_data-0.17.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.17.5
+Version: 0.17.6
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.17.5 → mostlyright_data-0.17.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.17.5"
+version = "0.17.6"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.17.5 → mostlyright_data-0.17.6}/skills/mr-data-build/SKILL.md

@@ -37,6 +37,7 @@ service behavior, retry mechanics, or these instructions.
 Outside a required host cadence, speak only when one of these boundaries gives the user useful new
 information:
 
+- A dataset-shaping choice needs the user's input; ask before committing to it.
 - A material source, schema, coverage, rights, or quality decision is supported by observed facts.
 - An exact contract or recipe needs the user's approval. State what will be built, the material
   limitation, and the decision required; do not explain the approval mechanism.
@@ -236,15 +237,13 @@ live data key is refused with `THIN_AUTHENTICATION_FAILED` rather than failing t
 
 ## Required protocol
 
-There is one loop, and it is short: open the page before you have anything to put on it, probe the
-sources, say out loud what you chose and what you refused, draft one recipe document, build a
-sample, interrogate what came back, fix the document, build again. Then present only what you
-inspected, with its caveats. The last two steps are conditional and `truncated` is what decides
-them: a sample that came back complete IS the table a full run would have sealed, so there is
-nothing to build again and nothing to ask about — only a TRUNCATED sample sends you round the
-loop. Nothing between a run finishing and reading it is held for a person,
-so there is no approval to seek, no separate body of evidence to read back, and no state to wait
-for. A table's first succeeded run goes live on its own, whatever mode it was, and what is left
+There is one loop: open the page, clarify the intended dataset with the user, probe the
+sources, resolve the tradeoffs that research reveals, summarize the agreed shape, draft one recipe
+document, build a sample, and inspect it against the agreed requirements. Repair discrepancies;
+return to the user before changing agreed semantics. A complete sample that meets those requirements
+is already the table a full run would seal, so do not rebuild it merely to change the run mode.
+A TRUNCATED sample requires the full-run decision below. Reading a finished run requires no separate
+approval or waiting state. A table's first succeeded run goes live on its own, whatever mode it was, and what is left
 for a person is [Promote](#promote) — the cadence it refreshes on and the reasoning behind it.
 
 **The page is the work, and it exists first.** A dataset used to come into being as a side effect
@@ -279,11 +278,10 @@ again, and the failure triples on recent runs tell you what the backend is refus
 `SANDBOX_MEMORY_BOUNDARY` on somebody else's recipe an hour ago means the room is unwell and your
 first failure will not be your document's fault. Keep this check silent.
 
-**Then settle anything only the user can settle, before probing.** Read the request for a word with
-two defensible readings, a population whose edges are unstated, or a window nobody named.
-`mr-data clarify --question TEXT` says whether there is anybody to ask and whether it is still early
-enough; [Autonomous delivery](#autonomous-delivery) is the rule for what may be asked and what stays
-yours to decide.
+**Then clarify the intended dataset before probing.** Follow [Dataset discovery](#dataset-discovery):
+use the request and conversation to identify consequential missing choices, ask the highest-impact
+questions, and wait for answers before work that depends on them. Independent source research may
+continue while waiting. Revisit the remaining choices when research provides concrete alternatives.
 
 Pass `--json` to automated commands. A command receipt is the only record that a Harness action
 succeeded.
@@ -402,10 +400,8 @@ blocks file; stage 3 is what puts them on the page.
 ### 3. Decide — say what you chose, what you refused, and what unblocks it
 
 **Post one Sources and decisions message before you register anything.** It goes to two places, in
-the same words: the dataset's own decision record, and the chat. This is the decision point that
-replaced the old one-question rule, and the reason for the change is that a question is not what a
-person mostly needs — what they need is to see the search, while there is still time to disagree
-with it.
+the same words: the dataset's own decision record, and the chat. Summarize the user's answers, the source evidence, and any remaining
+assumptions so the user can see what the first dataset will contain.
 
 ```sh
 mr-data dataset note DATASET_ID --heading "Sources and decisions" \
@@ -417,8 +413,9 @@ EOF
 The cell carries a `source` block for every feed evaluated — the ones being used, the ones
 considered, and the ones rejected with the clause that ruled each out — and a `decision` block for
 every grain, key, join and cutoff settled. Post the same content in the chat as one short message.
-Then **keep going**: this is a statement, not a gate, and waiting for a reply to it is the pause
-this document does not allow.
+If a material choice is still unresolved, ask and wait before registering the recipe. If the
+user has already settled the choices or delegated them, proceed without a redundant confirmation.
+Silence after a proposal is not an answer to a pending question.
 
 **Blocked sources: always say what unlocks it and how to enrol the credential.** A source you could
 not reach because it needs an API key, an account or an approval is not a dead end and must never
@@ -428,23 +425,21 @@ who issues the credential, where to apply for it, and the exact command that enr
 source with no route back is a decision the user cannot revisit; one with a route back is a
 decision they can.
 
-**Ask only at a genuine fork.** A fork is a question whose two answers produce two different tables
-and which no evidence can settle — what an ambiguous word in the request means, how wide the
-population is, how far back it goes. Write it as a `clarification` block carrying `question` and no
-`answer`, which is what the page shows as a pending question, and ask it in the chat in the same
-words. `mr-data clarify --question TEXT` says whether there is anybody to ask and whether it is
-still early enough. When the answer comes back, write it as the answer on that block:
+**Resolve research tradeoffs with the user.** Apply [Dataset discovery](#dataset-discovery) when
+sources cannot deliver the requested grain, coverage, fields, or join. Explain the supported options
+and their consequences before changing the intended dataset. Record each pending question as a
+`clarification` block carrying `question` and no `answer`, and ask it in chat in the same words.
+When answered, revise that same cell using its actual identifier (not a new unrelated note):
 
 ```sh
 mr-data dataset note DATASET_ID --heading "The stage grain is settled" \
-  --revise "one-fork-to-settle" --blocks-file ANSWERED.json --json <<'EOF'
+  --revise PENDING_CELL_ID --blocks-file ANSWERED.json --json <<'EOF'
 Daily maxima, on the user's answer.
 EOF
 ```
 
-Everything under [Autonomous delivery](#autonomous-delivery) still holds: which source, Reader,
-engine, format, workspace, retry tactic or cleaning operation to use is settled by the evidence and
-is never a fork. If nothing is genuinely forked, ask nothing and carry on.
+Carry the answers into the recipe's question, table and column descriptions, source selection,
+transform, and checks as appropriate. Record assumptions separately from user decisions.
 
 ### 4. Draft — one recipe document, one call
 
@@ -1191,14 +1186,16 @@ changelog entry, and the run's own log is already that.
 
 **A `clarification` block with a `question` and no `answer` is a PENDING question**, and it is the
 one shape here that is a way to ask: the Dataset page shows it, and the chat asks it in the same
-words. Write one only at a genuine fork, as stage 3 defines one. The answer is a REVISION of that
+words. Write one for each question [Dataset discovery](#dataset-discovery) has you ask — a
+dataset-shaping choice the evidence cannot settle. The answer is a REVISION of that
 cell — `--revise CELL_ID`, the same blocks again with `answer` and `answered_at` filled in — so the
 page shows the question settled rather than a second cell nothing ties to the first. `answered_at`
 is admitted only beside `answer`, because a time with no answer records that the question was
 settled by nobody, and is a UTC timestamp ending in `Z`, as `2026-09-01T10:00:00Z`. Everything
-under [Autonomous delivery](#autonomous-delivery) still holds — do not put a source, Reader,
-engine, format, workspace, retry tactic, cleaning operation or repair to the user, in a block or
-anywhere else.
+under [Autonomous delivery](#autonomous-delivery) still holds — do not put a Reader, engine,
+parsing format, workspace, retry tactic or semantics-preserving repair to the user, in a block or
+anywhere else. Source preferences and cleaning choices that change the meaning are the user's,
+and go in a block.
 
 `--phase` binds the cell to one build stage — `acquire`, `parse`, `transform`, `check`, `persist` —
 so a reader can group a run's record by stage. `--checkpoint` places the cell beside the event it
@@ -1444,41 +1441,79 @@ each table rather than what the exit code said about the set. A table that is li
 `archive` until it is withdrawn, and a table a run is still writing to is refused by both until the
 run has finished; neither refusal deletes anything, and neither ends the command.
 
+## Dataset discovery
+
+Make the user a participant in shaping the first dataset. Before registering a recipe, review the
+request and prior answers for choices whose alternatives would materially change the result. Lead
+with the unresolved choices most likely to make the first dataset wrong. The agent chooses the
+questions and follow-ups; this is a decision guide, not a form to recite:
+
+- **Purpose:** What analysis, decision, or downstream use should the dataset support? Ask when it
+  helps choose between plausible definitions or shapes, not for background already supplied.
+- **Table structure and joins:** One joined table, separate related tables, or both? If joining,
+  what should match, and should unmatched records remain? Explain when a join could duplicate rows
+  or lose records; do not require the user to write SQL or select a join algorithm.
+- **Granularity:** What does one row represent — an event, a company, a station-hour, a country-year?
+  Resolve raw observations versus aggregates, time frequency, and geographic detail where relevant.
+- **Sources:** Are particular publishers, uploaded files, or sources required or excluded, or should
+  the agent recommend them? Research concrete source options yourself. Ask about consequential choices
+  such as official but delayed data versus broader recent coverage, or a source requiring access.
+- **Coverage:** Which entities, geography, date range, and inclusion/exclusion rules matter? For
+  changing data, distinguish a fixed historical snapshot from continuing updates.
+- **Fields and definitions:** Which measures and identifiers are essential? Resolve ambiguous
+  metrics, units, currencies, time zones, or adjusted versus raw values when they affect the use.
+- **Quality and tradeoffs:** Is incomplete coverage acceptable? Ask before dropping unmatched rows,
+  filling missing values, estimating data, or substituting a proxy when that changes the meaning.
+- **Delivery constraints:** Ask about freshness, destination, format, or scale/budget only when
+  relevant and not already specified. Existing spend and cadence decisions still apply.
+
+For every discovery question, including the initial questions before probing, check attendance with
+`mr-data clarify --attended --question TEXT` when a person is in chat. Record a pending
+`clarification` block on the dataset and ask the same wording in chat. Record the answer by revising
+that same cell, as in stage 3; a plain narrative note alone does not mark a question as pending.
+The command checks whether asking is allowed; the host conversation asks and waits for the answer.
+
+Usually ask one to three related questions per turn, with short bold labels such as **Table shape**,
+**Sources**, and **Granularity**. Use plain language and domain-specific examples. Offer a recommended
+option and explain its consequence when evidence supports it; allow a free-text answer or "you
+choose." Do not overwhelm the user with the entire list, repeatedly ask answered questions, or ask
+about facts you can determine from source inspection. There is no total question quota: follow up
+when an answer or source finding exposes another consequential decision.
+
+Wait for answers to the questions you ask before committing dependent work. Continue independent
+research where useful, but do not register or run the first recipe with unanswered material
+questions. An elapsed wait is not permission to assume an answer. If the request is already specific
+enough, summarize its shape and proceed. If the user explicitly delegates choices or asks for no
+questions, use evidence-backed defaults and state them without forcing another confirmation.
+
+Before the first build, give a compact recap of purpose, table shape, row grain, sources, coverage,
+and important fields or limitations. Highlight unresolved decisions as questions; otherwise proceed
+on the answers already given. During verification compare the sample with this agreed shape, not
+just whether the run succeeded. If research or the sample requires a material semantic change,
+return to the user with concrete alternatives before building a revised recipe; use the existing
+contract revision/approval path once a recipe has been registered.
+
 ## Autonomous delivery
 
 Own the run until the requested outcome is delivered: a run that succeeded, its declared checks
-read, its coverage inspected, and its refresh cadence recorded when the user has said what it should
-be. What is live is what Studio returns as live; catch-up and continuing freshness require separate
-durable evidence from the deployed component that owns them. The allowed user pauses
-are the cadence decision, the decision to start a full run after a TRUNCATED sample, and a spend
-confirmation on a run over the threshold. Do not ask the user to choose sources, Readers, engines,
-formats, workspaces, retry tactics, cleaning operations, or validation repairs. Make those
-operational decisions from the question, observed evidence, registered capabilities, and
-deterministic receipts.
-
-**The decision point replaces the one-question rule.** There used to be a budget of exactly one
-clarifying question, and it was the wrong instrument: it made the agent hoard its one question
-against a fork that might arrive later, and it left the user with no view of the search at all
-until the page was finished. What stage 3 asks for instead is a STATEMENT — sources chosen, sources
-rejected or blocked with the concrete unblock step, the assumption taken about window and grain —
-posted to the dataset's record and to the chat, after which the build continues without waiting.
-A person who disagrees with it says so; a person who does not is not made to answer anything.
-
-A question is still asked, and it is asked at a genuine FORK: two answers that produce two
-different tables, and no evidence that can settle it. `Throughput: movements or passengers?` is
-such a fork. Which Reader parses the feed is not, and neither is anything else named above; those
-stay banned because the evidence settles them. There is no budget of one — a build with two real
-forks asks about both — but a fork is rare, and a question that the evidence could have answered is
-still the failure this section is about. Ask before registering, while nothing is running and
-nothing has been spent, write it as a pending `clarification` block on the dataset, and carry the
-answer back as a revision of that cell.
+read, its coverage inspected against the agreed dataset shape, and its refresh cadence recorded
+when the user has said what it should be. What is live is what Studio returns as live; catch-up and
+continuing freshness require separate durable evidence from the deployed component that owns them.
+Dataset discovery questions, material semantic revisions, cadence decisions, full runs after a
+TRUNCATED sample, and spend confirmations are valid user pauses. Own the operational choices:
+Readers, engines, parsing formats, workspace plumbing, retry tactics, and repairs that preserve the
+agreed semantics. Source preferences and cleaning choices that change the meaning belong in the
+user conversation, even when the agent could technically choose for them.
 
 `mr-data clarify --question TEXT` says whether to ask at all. It exits non-zero when no person can
 answer, which is what a scheduled or headless session reports, and when `--recipe RECIPE_ID` names
 a registered recipe, because the answer to a question asked after the contract is a revision and
-goes through the contract approval instead. On a non-zero exit ask nothing: decide from the
-evidence, write the assumption into the recipe document, and state it as a limitation in the final
-report. `mr-data clarify --question TEXT --answer TEXT` composes the exchange as one `clarification`
+goes through the contract approval instead. When no person can answer, do not wait: use the
+existing brief and evidence to make choices within the requested scope, record assumptions, and
+report limitations. Do not silently replace explicit requirements. When the recipe is already
+registered, use the revision/approval path for material changes rather than treating the refusal
+as permission to choose new semantics. Other command failures are not evidence that the user is
+unavailable. `mr-data clarify --question TEXT --answer TEXT` composes the exchange as one `clarification`
 block. Carry its wording into the recipe document the answer settles — the question text, the table
 description, the column description — so the assumption the table rests on reads the same in the
 document, in the Dataset, and in the report.
@@ -1596,11 +1631,34 @@ Write it as a source whose connector is that coordinate plus a `collection` memb
   merged relation.
 - The Reader pin is required, must be `html.web_extract`, and is the SAME settings for every page.
 
+**`limits.max_requests` is charged per HTTP request, redirect hops and retries included** — a page
+that redirects once costs two. Registration only checks that
+`discovery.max_requests + pages.max_fetches_per_run` fits inside it, which is necessary and NOT
+sufficient: a budget written to exactly that sum runs out early on any publisher that redirects,
+and the run stops with `budget_exhausted: "max_requests"` short of the pages it was allowed. Where
+redirects are likely — trailing slashes, `http`→`https`, `www`, which is most publishers — write
+`limits.max_requests >= discovery.max_requests + 2 * max_fetches_per_run`. A budget larger than the
+corpus can absorb costs nothing.
+
 The relation the transform reads is the reader's columns plus `page_id`, `page_url`,
 `page_fetched_at`, `page_content_sha256`, `page_revision`, `page_discovered_at` and `page_ordinal`.
 Those names are reserved: a reader that declares one is a registration refusal. Select them in the
 transform so every row on the table says which page and which revision produced it.
 
+**All seven arrive as text and must be CAST.** The merged relation is sealed as CSV, so every
+column reaches the transform as `VARCHAR`; selecting `page_ordinal` bare while declaring the column
+`integer` fails the build with `TRANSFORM_COLUMN_TYPE_MISMATCH`. Write
+`cast(page_ordinal as integer)`, `cast(page_revision as integer)` and
+`cast(page_fetched_at as timestamp with time zone)` — a declared `timestamp` column admits
+`TIMESTAMP WITH TIME ZONE` and nothing else.
+
+**A listing that could not be read says so.** The coverage block carries `discovery_failure`:
+`null`, or the code and one sentence for the first listing request the run was refused. A refresh
+that already holds pages keeps every row and succeeds; a FIRST run whose listing is refused has
+nothing to continue from and fails with `COLLECTION_DISCOVERY_FAILED` rather than sealing an empty
+corpus. If a run reports zero discovered, read that member before concluding the publisher's index
+is empty.
+
 **Expect the first run not to finish, and say so.** A run that spends its fetch budget still
 SUCCEEDS; its coverage block reports `complete: false` and names the budget that ended it. The
 next refresh continues from the ledger — new pages first, then failures oldest first, then

{mostlyright_data-0.17.5 → mostlyright_data-0.17.6}/src/mostlyright/data_harness/hosted_crawler_fetch.py

@@ -10,7 +10,7 @@ from collections.abc import Callable, Iterator
 from concurrent.futures import ThreadPoolExecutor
 from contextlib import contextmanager
 from contextvars import ContextVar
-from dataclasses import dataclass
+from dataclasses import dataclass, replace
 from urllib.parse import urlsplit
 
 from mostlyright.data_harness.acquisition.http import (
@@ -271,6 +271,7 @@ def _public_https_retriever(
     transport: StdlibPinnedTransport | None = None,
     connect_proxy: ConnectProxyMember | None = None,
     connect_proxy_config_digest: str | None = None,
+    allowed_media_types: tuple[str, ...] | None = None,
 ) -> PinnedHttpsRetriever:
     """The one reviewed public-HTTPS retriever construction.
 
@@ -279,6 +280,13 @@ def _public_https_retriever(
     limiter -- and differ only in the three explicit caps a caller names: the byte ceiling
     and the two stall timeouts.  Stated once so the ranged path cannot drift into a second,
     subtly different transport policy.
+
+    ``allowed_media_types`` NARROWS THAT ONE TERM AND NOTHING ELSE, for a caller whose response
+    is not the one the query's own pin describes.  A page collection's LISTING request is the
+    case: the source is pinned to an HTML Reader because its detail pages are HTML, and the index
+    that names those pages is JSON or a sitemap, read inertly by the driver and handed to no
+    Reader at all.  Every other term above is the reviewed one, so what the narrowing changes is
+    which representations the origin may answer with and not how the request is made.
     """
 
     return PinnedHttpsRetriever(
@@ -288,17 +296,24 @@ def _public_https_retriever(
             allowed_ports=frozenset({443}),
         ),
         transport=transport or StdlibPinnedTransport(),
-        limits=RetrievalLimits(
-            max_response_bytes=max_response_bytes,
-            max_redirects=3,
-            max_requests=4,
-            connect_timeout_seconds=5.0,
-            read_timeout_seconds=read_timeout_seconds,
-            total_timeout_seconds=total_timeout_seconds,
-            max_concurrency=PUBLIC_HTTPS_MAX_CONCURRENCY,
-            min_interval_seconds=1.0,
-            allowed_media_types=_public_expected_media_types(request.query),
-            user_agent=PUBLIC_HTTPS_USER_AGENT,
+        limits=replace(
+            RetrievalLimits(
+                max_response_bytes=max_response_bytes,
+                max_redirects=3,
+                max_requests=4,
+                connect_timeout_seconds=5.0,
+                read_timeout_seconds=read_timeout_seconds,
+                total_timeout_seconds=total_timeout_seconds,
+                max_concurrency=PUBLIC_HTTPS_MAX_CONCURRENCY,
+                min_interval_seconds=1.0,
+                allowed_media_types=_public_expected_media_types(request.query),
+                user_agent=PUBLIC_HTTPS_USER_AGENT,
+            ),
+            **(
+                {}
+                if allowed_media_types is None
+                else {"allowed_media_types": tuple(allowed_media_types)}
+            ),
         ),
         limiter=_PUBLIC_HTTPS_LIMITER,
         connect_proxy=connect_proxy,
@@ -407,12 +422,16 @@ def fetch_crawler_source(
     *,
     expected_egress_policy_attestation: str,
     retriever: PinnedHttpsRetriever | None = None,
+    allowed_media_types: tuple[str, ...] | None = None,
 ) -> RetrievedBytes:
     """Fetch one exact reviewed URL for the crawler child to normalize.
 
     The retrieved bytes are framed by encode_normalization_input and handed to the
     mr-data-crawler child, which hosted_crawler_job spawns under a seccomp filter that
     denies socket and connect. That child is not acquisition.sandbox.CrawlerSandbox.
+
+    ``allowed_media_types`` is the caller's own narrowing of the one term
+    :func:`_public_https_retriever` documents it for, and reaches only the public HTTPS route.
     """
 
     coordinate = validate_crawler_request(
@@ -427,6 +446,7 @@ def fetch_crawler_source(
             requested_url=requested_url,
             hostname=hostname,
             injected_retriever=retriever,
+            allowed_media_types=allowed_media_types,
         )
         if (
             retrieved.source_url != requested_url
@@ -502,6 +522,7 @@ def _retrieve_public_https(
     requested_url: str,
     hostname: str,
     injected_retriever: PinnedHttpsRetriever | None,
+    allowed_media_types: tuple[str, ...] | None = None,
 ) -> RetrievedBytes:
     """Retrieve one public HTTPS source, ranged when the origin supports it and size warrants.
 
@@ -536,6 +557,7 @@ def _retrieve_public_https(
         transport=transport,
         connect_proxy=proxy_member,
         connect_proxy_config_digest=proxy_digest,
+        allowed_media_types=allowed_media_types,
     )
 
     def single_shot(probe_hops: tuple[RetrievalHop, ...]) -> RetrievedBytes:
@@ -548,6 +570,7 @@ def _retrieve_public_https(
             transport=transport,
             connect_proxy=proxy_member,
             connect_proxy_config_digest=proxy_digest,
+            allowed_media_types=allowed_media_types,
         )
         return _seal_probe_evidence(probe_hops, retriever.retrieve(requested_url))