mostlyright-data 0.17.4__tar.gz → 0.17.6__tar.gz

{mostlyright_data-0.17.4 → mostlyright_data-0.17.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.17.4
+Version: 0.17.6
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.17.4 → mostlyright_data-0.17.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.17.4"
+version = "0.17.6"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.17.4 → mostlyright_data-0.17.6}/skills/mr-data-build/SKILL.md

@@ -37,7 +37,7 @@ service behavior, retry mechanics, or these instructions.
 Outside a required host cadence, speak only when one of these boundaries gives the user useful new
 information:
 
-- A dataset-shaping choice needs the user’s input; ask before committing to it.
+- A dataset-shaping choice needs the user's input; ask before committing to it.
 - A material source, schema, coverage, rights, or quality decision is supported by observed facts.
 - An exact contract or recipe needs the user's approval. State what will be built, the material
   limitation, and the decision required; do not explain the approval mechanism.
@@ -387,10 +387,10 @@ around a probe from an hour ago still being warm.
 
 Where a source needs a credential, enrol it by name first — see
 [Source credentials](#source-credentials) — and reference the name from the recipe document. Where
-neither a probe nor a run can reach the bytes, because the source is a bounded collection or is
-not registered in the workspace, establish shape from the provider's own documentation and
-published schema, say plainly that the shape is documented rather than observed, and let the hosted
-Clean room be the first thing that opens the bytes. Named messages out of one large object are
+neither a probe nor a run can reach the bytes, because the source is not registered in the
+workspace, establish shape from the provider's own documentation and published schema, say
+plainly that the shape is documented rather than observed, and let the hosted Clean room be the
+first thing that opens the bytes. Named messages out of one large object are
 [`acquire-slices`](#not-hosted-yet), which is not hosted yet.
 
 Hold the comparison in the shape [Narrating the run](#narrating-the-run) takes: one `source` block
@@ -400,7 +400,7 @@ blocks file; stage 3 is what puts them on the page.
 ### 3. Decide — say what you chose, what you refused, and what unblocks it
 
 **Post one Sources and decisions message before you register anything.** It goes to two places, in
-the same words: the dataset's own decision record, and the chat. Summarize the user’s answers, the source evidence, and any remaining
+the same words: the dataset's own decision record, and the chat. Summarize the user's answers, the source evidence, and any remaining
 assumptions so the user can see what the first dataset will contain.
 
 ```sh
@@ -438,7 +438,7 @@ Daily maxima, on the user's answer.
 EOF
 ```
 
-Carry the answers into the recipe’s question, table and column descriptions, source selection,
+Carry the answers into the recipe's question, table and column descriptions, source selection,
 transform, and checks as appropriate. Record assumptions separately from user decisions.
 
 ### 4. Draft — one recipe document, one call
@@ -794,9 +794,12 @@ one", not as a failure: a document declaring 8 MiB over a feed that answers 40 K
 **If a source really does deliver more than the Reader holds, do not answer it by lowering the
 ceiling on that source:** that does not fetch less data, it truncates the sample or refuses the
 full run. Split the window across several sources — one per month, per region, or per whatever the
-publisher paginates on — and union them in a first transform step. A recipe names up to 64 sources,
-which is a great many windows. [The recipe document](../../docs/RECIPE-DOCUMENT.md) carries the
-measured numbers behind the ceiling.
+publisher paginates on — and union them in a first transform step. A recipe names up to 256
+sources, which is a great many windows. [The recipe document](../../docs/RECIPE-DOCUMENT.md)
+carries the measured numbers behind the ceiling.
+
+**A corpus of many pages behind one index is a COLLECTION, not many sources.** See
+[Authoring a collection source](#authoring-a-collection-source).
 
 **The dataset's record is the primary one; a run's own record is for what happened inside that
 run.** The sources, the decisions and the questions went onto the dataset in stage 3, where they
@@ -1183,14 +1186,16 @@ changelog entry, and the run's own log is already that.
 
 **A `clarification` block with a `question` and no `answer` is a PENDING question**, and it is the
 one shape here that is a way to ask: the Dataset page shows it, and the chat asks it in the same
-words. Write one only at a genuine fork, as stage 3 defines one. The answer is a REVISION of that
+words. Write one for each question [Dataset discovery](#dataset-discovery) has you ask — a
+dataset-shaping choice the evidence cannot settle. The answer is a REVISION of that
 cell — `--revise CELL_ID`, the same blocks again with `answer` and `answered_at` filled in — so the
 page shows the question settled rather than a second cell nothing ties to the first. `answered_at`
 is admitted only beside `answer`, because a time with no answer records that the question was
 settled by nobody, and is a UTC timestamp ending in `Z`, as `2026-09-01T10:00:00Z`. Everything
-under [Autonomous delivery](#autonomous-delivery) still holds — do not put a source, Reader,
-engine, format, workspace, retry tactic, cleaning operation or repair to the user, in a block or
-anywhere else.
+under [Autonomous delivery](#autonomous-delivery) still holds — do not put a Reader, engine,
+parsing format, workspace, retry tactic or semantics-preserving repair to the user, in a block or
+anywhere else. Source preferences and cleaning choices that change the meaning are the user's,
+and go in a block.
 
 `--phase` binds the cell to one build stage — `acquire`, `parse`, `transform`, `check`, `persist` —
 so a reader can group a run's record by stage. `--checkpoint` places the cell beside the event it
@@ -1470,8 +1475,8 @@ The command checks whether asking is allowed; the host conversation asks and wai
 
 Usually ask one to three related questions per turn, with short bold labels such as **Table shape**,
 **Sources**, and **Granularity**. Use plain language and domain-specific examples. Offer a recommended
-option and explain its consequence when evidence supports it; allow a free-text answer or “you
-choose.” Do not overwhelm the user with the entire list, repeatedly ask answered questions, or ask
+option and explain its consequence when evidence supports it; allow a free-text answer or "you
+choose." Do not overwhelm the user with the entire list, repeatedly ask answered questions, or ask
 about facts you can determine from source inspection. There is no total question quota: follow up
 when an answer or source finding exposes another consequential decision.
 
@@ -1603,6 +1608,76 @@ What it costs the protocol, and what to do instead:
 - Preserve event time, available time, acquisition time, source revision, and timezone. Do not use
   post-cutoff information in targets, labels, features, joins, or validation decisions.
 
+### Authoring a collection source
+
+A corpus published as an index plus many detail pages is **one** source through
+`public.https.collection@2.0.0`, not one source per page. Source count and publisher count are
+unaffected by how many pages sit behind it: 2,819 pages is one source, one publisher, one line on
+the dataset's sources card.
+
+Write it as a source whose connector is that coordinate plus a `collection` member:
+
+- `collection.discovery` says how the corpus is LISTED — `json_api` with RFC 6901 pointers,
+  `html_index` with the reader's closed selectors, `sitemap`, or an `explicit` member list — and
+  which pagination grammar the listing uses. The run follows only that grammar and never a link
+  found inside a detail page.
+- `collection.pages` says what may be fetched and how much: `allowed_origins` (which must include
+  the discovery address's own origin), `max_pages`, `max_fetches_per_run`, `concurrency` (1 or 2),
+  `min_interval_seconds` as a whole number of seconds (one or more), `request_timeout_seconds`,
+  `retries`, and a `revisit` rule. Only `allowed_origins` is required; every other member has a
+  stated meaning when it is left out.
+- `limits.max_requests` is REQUIRED and bounds every request the run makes, discovery and detail
+  pages together. `limits.max_source_bytes` bounds the fetched bytes; `limits.max_rows` clamps the
+  merged relation.
+- The Reader pin is required, must be `html.web_extract`, and is the SAME settings for every page.
+
+**`limits.max_requests` is charged per HTTP request, redirect hops and retries included** — a page
+that redirects once costs two. Registration only checks that
+`discovery.max_requests + pages.max_fetches_per_run` fits inside it, which is necessary and NOT
+sufficient: a budget written to exactly that sum runs out early on any publisher that redirects,
+and the run stops with `budget_exhausted: "max_requests"` short of the pages it was allowed. Where
+redirects are likely — trailing slashes, `http`→`https`, `www`, which is most publishers — write
+`limits.max_requests >= discovery.max_requests + 2 * max_fetches_per_run`. A budget larger than the
+corpus can absorb costs nothing.
+
+The relation the transform reads is the reader's columns plus `page_id`, `page_url`,
+`page_fetched_at`, `page_content_sha256`, `page_revision`, `page_discovered_at` and `page_ordinal`.
+Those names are reserved: a reader that declares one is a registration refusal. Select them in the
+transform so every row on the table says which page and which revision produced it.
+
+**All seven arrive as text and must be CAST.** The merged relation is sealed as CSV, so every
+column reaches the transform as `VARCHAR`; selecting `page_ordinal` bare while declaring the column
+`integer` fails the build with `TRANSFORM_COLUMN_TYPE_MISMATCH`. Write
+`cast(page_ordinal as integer)`, `cast(page_revision as integer)` and
+`cast(page_fetched_at as timestamp with time zone)` — a declared `timestamp` column admits
+`TIMESTAMP WITH TIME ZONE` and nothing else.
+
+**A listing that could not be read says so.** The coverage block carries `discovery_failure`:
+`null`, or the code and one sentence for the first listing request the run was refused. A refresh
+that already holds pages keeps every row and succeeds; a FIRST run whose listing is refused has
+nothing to continue from and fails with `COLLECTION_DISCOVERY_FAILED` rather than sealing an empty
+corpus. If a run reports zero discovered, read that member before concluding the publisher's index
+is empty.
+
+**Expect the first run not to finish, and say so.** A run that spends its fetch budget still
+SUCCEEDS; its coverage block reports `complete: false` and names the budget that ended it. The
+next refresh continues from the ledger — new pages first, then failures oldest first, then
+revisits — and every row of a page this run did not fetch survives byte for byte. Report the
+shortfall as what it is, a backfill in progress, not as a failure and not as a complete corpus.
+
+A page that disappears from the index deletes nothing. A page answering with different content
+replaces exactly its own rows, increments `page_revision`, and keeps the previous digest: that is
+a correction, and it is worth saying on the run record when it happens.
+
+Before registering it, read the discovery block back the way a run will: the address the first
+request goes to, whether the pagination parameter is already written into that address, whether
+the records pointer names an array, and whether `discovery.max_requests` plus
+`pages.max_fetches_per_run` fits inside `limits.max_requests`. Registration refuses all four, and
+they are the four that are cheapest to get right before a run spends anybody's rate limit.
+
+The full contract, including every bound and the coverage block's members, is
+[Source collections](../../docs/SOURCE-COLLECTIONS.md).
+
 ## Readers
 
 `docs/READERS.md` defines Reader behavior. Certified Readers may be pinned on an exact
@@ -1697,6 +1772,7 @@ decoder. Zero matches and multiple matches both refuse.
 | `xml.tabular@1.0.0` | Absolute record element path, declared namespace prefixes, and relative field selectors ending in `text()` or `@attribute`; fields are required unless `required: false`. |
 | `html.tabular@1.0.0` | Absolute record element path and relative text/attribute field selectors; fields are required unless `required: false`. |
 | `html.web_extract@1.0.0` | `mode: dom` with closed CSS-like record/field selectors, or `mode: json` with an inert JSON script selector and RFC 6901 field pointers. |
+| `html.web_extract@1.1.0` | Same settings as 1.0.0; extracted text follows DOM order, with one space at block-element boundaries and `<br>` and none between inline runs. |
 | `protobuf.tabular@1.0.0` | Digest-pinned base64 FileDescriptorSet, fully-qualified message type, `single` or `length_delimited` framing, and fields with an explicit repeated `explode` or `refuse` choice. |
 | `parquet.tabular@1.0.0` | Optional `columns` selects a bounded ordered Parquet projection. |
 | `xbrl.facts@1.0.0` | `document_kind` (`xbrl` or constrained XML `ixbrl`) and exact namespace/name fact projection; contexts, units, and metadata remain evidence fields, not joins. |

{mostlyright_data-0.17.4 → mostlyright_data-0.17.6}/src/mostlyright/data_harness/cli.py

@@ -575,6 +575,7 @@ def _dispatch(
         or selected_command == "propose"
         or selected_command == "acquire-cancel"
         or selected_command == "clarify"
+        or selected_command == "collection-plan"
         or selected_command in _RESEARCH_COMMANDS
     ):
         # `note`, `propose` and the research commands are the hosted lane's own implementation, run
@@ -586,6 +587,8 @@ def _dispatch(
         # nothing and is identical in both profiles. It refuses in the thin lane's typed vocabulary
         # because that is the vocabulary the agent driving it already branches on, and a fifth
         # refusal class for four sentences would be a fifth remediation family for no reader.
+        # `collection-plan` is the same shape again: it reaches nothing, and it refuses in that
+        # same vocabulary because the agent authoring a collection is already reading it.
         from mostlyright.data_harness.thin.transport import ThinLaneError
 
         handled_errors += (ThinLaneError,)
@@ -754,6 +757,10 @@ def _dispatch(
             if args.command == "crosswalk-inspect":
                 payload["rows"] = [row.to_dict() for row in crosswalk.rows]
             _emit(payload)
+        elif args.command == "collection-plan":
+            from mostlyright.data_harness.thin import collection_plan as _collection_plan
+
+            _emit(_collection_plan.collection_plan(args))
         elif args.command == "recipe-validate":
             recipe = parse_fresh_recipe(_read_json(args.recipe))
             _emit(
@@ -1694,6 +1701,20 @@ def _parser(
     )
     recipe_validate.add_argument("--recipe", type=Path, required=True, help=_RECIPE_FILE)
 
+    # DECLARED BY THE MODULE THAT IMPLEMENTS IT, which is the rule `probe` and `recipe` already
+    # follow. It is registered HERE and not in the hosted lane, for the reason that module's own
+    # header states: rendering the plan parses the recipe with the engine's parser, and `mr-data`
+    # installs without it. This is where `recipe-validate` -- the other command that reads a
+    # recipe and reaches nothing -- already lives.
+    from mostlyright.data_harness.thin import collection_plan as _collection_plan
+
+    _collection_plan.declare_arguments(
+        _add(
+            "collection-plan",
+            help="show what a recipe's collection source would discover, fetch and seal",
+        )
+    )
+
     crosswalk_validate = _add(
         "crosswalk-validate",
         help="validate a canonical authored crosswalk and print its digest",

{mostlyright_data-0.17.4 → mostlyright_data-0.17.6}/src/mostlyright/data_harness/hosted_crawler_fetch.py

@@ -10,7 +10,7 @@ from collections.abc import Callable, Iterator
 from concurrent.futures import ThreadPoolExecutor
 from contextlib import contextmanager
 from contextvars import ContextVar
-from dataclasses import dataclass
+from dataclasses import dataclass, replace
 from urllib.parse import urlsplit
 
 from mostlyright.data_harness.acquisition.http import (
@@ -271,6 +271,7 @@ def _public_https_retriever(
     transport: StdlibPinnedTransport | None = None,
     connect_proxy: ConnectProxyMember | None = None,
     connect_proxy_config_digest: str | None = None,
+    allowed_media_types: tuple[str, ...] | None = None,
 ) -> PinnedHttpsRetriever:
     """The one reviewed public-HTTPS retriever construction.
 
@@ -279,6 +280,13 @@ def _public_https_retriever(
     limiter -- and differ only in the three explicit caps a caller names: the byte ceiling
     and the two stall timeouts.  Stated once so the ranged path cannot drift into a second,
     subtly different transport policy.
+
+    ``allowed_media_types`` NARROWS THAT ONE TERM AND NOTHING ELSE, for a caller whose response
+    is not the one the query's own pin describes.  A page collection's LISTING request is the
+    case: the source is pinned to an HTML Reader because its detail pages are HTML, and the index
+    that names those pages is JSON or a sitemap, read inertly by the driver and handed to no
+    Reader at all.  Every other term above is the reviewed one, so what the narrowing changes is
+    which representations the origin may answer with and not how the request is made.
     """
 
     return PinnedHttpsRetriever(
@@ -288,17 +296,24 @@ def _public_https_retriever(
             allowed_ports=frozenset({443}),
         ),
         transport=transport or StdlibPinnedTransport(),
-        limits=RetrievalLimits(
-            max_response_bytes=max_response_bytes,
-            max_redirects=3,
-            max_requests=4,
-            connect_timeout_seconds=5.0,
-            read_timeout_seconds=read_timeout_seconds,
-            total_timeout_seconds=total_timeout_seconds,
-            max_concurrency=PUBLIC_HTTPS_MAX_CONCURRENCY,
-            min_interval_seconds=1.0,
-            allowed_media_types=_public_expected_media_types(request.query),
-            user_agent=PUBLIC_HTTPS_USER_AGENT,
+        limits=replace(
+            RetrievalLimits(
+                max_response_bytes=max_response_bytes,
+                max_redirects=3,
+                max_requests=4,
+                connect_timeout_seconds=5.0,
+                read_timeout_seconds=read_timeout_seconds,
+                total_timeout_seconds=total_timeout_seconds,
+                max_concurrency=PUBLIC_HTTPS_MAX_CONCURRENCY,
+                min_interval_seconds=1.0,
+                allowed_media_types=_public_expected_media_types(request.query),
+                user_agent=PUBLIC_HTTPS_USER_AGENT,
+            ),
+            **(
+                {}
+                if allowed_media_types is None
+                else {"allowed_media_types": tuple(allowed_media_types)}
+            ),
         ),
         limiter=_PUBLIC_HTTPS_LIMITER,
         connect_proxy=connect_proxy,
@@ -407,12 +422,16 @@ def fetch_crawler_source(
     *,
     expected_egress_policy_attestation: str,
     retriever: PinnedHttpsRetriever | None = None,
+    allowed_media_types: tuple[str, ...] | None = None,
 ) -> RetrievedBytes:
     """Fetch one exact reviewed URL for the crawler child to normalize.
 
     The retrieved bytes are framed by encode_normalization_input and handed to the
     mr-data-crawler child, which hosted_crawler_job spawns under a seccomp filter that
     denies socket and connect. That child is not acquisition.sandbox.CrawlerSandbox.
+
+    ``allowed_media_types`` is the caller's own narrowing of the one term
+    :func:`_public_https_retriever` documents it for, and reaches only the public HTTPS route.
     """
 
     coordinate = validate_crawler_request(
@@ -427,6 +446,7 @@ def fetch_crawler_source(
             requested_url=requested_url,
             hostname=hostname,
             injected_retriever=retriever,
+            allowed_media_types=allowed_media_types,
         )
         if (
             retrieved.source_url != requested_url
@@ -502,6 +522,7 @@ def _retrieve_public_https(
     requested_url: str,
     hostname: str,
     injected_retriever: PinnedHttpsRetriever | None,
+    allowed_media_types: tuple[str, ...] | None = None,
 ) -> RetrievedBytes:
     """Retrieve one public HTTPS source, ranged when the origin supports it and size warrants.
 
@@ -536,6 +557,7 @@ def _retrieve_public_https(
         transport=transport,
         connect_proxy=proxy_member,
         connect_proxy_config_digest=proxy_digest,
+        allowed_media_types=allowed_media_types,
     )
 
     def single_shot(probe_hops: tuple[RetrievalHop, ...]) -> RetrievedBytes:
@@ -548,6 +570,7 @@ def _retrieve_public_https(
             transport=transport,
             connect_proxy=proxy_member,
             connect_proxy_config_digest=proxy_digest,
+            allowed_media_types=allowed_media_types,
         )
         return _seal_probe_evidence(probe_hops, retriever.retrieve(requested_url))
 

{mostlyright_data-0.17.4 → mostlyright_data-0.17.6}/src/mostlyright/data_harness/hosted_deploy.py

@@ -30,6 +30,7 @@ from mostlyright.data_harness.hosted_crawler_protocol import (
     PUBLIC_HTTPS_EGRESS_POLICY_ATTESTATION,
 )
 from mostlyright.data_harness.studio_boundary import StudioBoundaryError, client_contract_identity
+from mostlyright.data_harness.thin.user_agent import with_user_agent
 from mostlyright.data_harness.ux.credentials import ResolvedCloudCredentials
 
 CLOUD_TOKEN_PATH = "/api/cli/studio-token"
@@ -178,7 +179,7 @@ class UrlLibTokenExchangeTransport:
         body: bytes | None,
         maximum: int,
     ) -> tuple[int, bytes, Mapping[str, str]]:
-        request = Request(url, data=body, headers=dict(headers), method=method)
+        request = Request(url, data=body, headers=with_user_agent(headers), method=method)
         try:
             response = self._opener.open(request, timeout=REQUEST_TIMEOUT_SECONDS)
         except HTTPError as error: