mostlyright-data 0.12.0__tar.gz → 0.13.0__tar.gz

{mostlyright_data-0.12.0 → mostlyright_data-0.13.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.12.0
+Version: 0.13.0
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.12.0 → mostlyright_data-0.13.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.12.0"
+version = "0.13.0"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.12.0 → mostlyright_data-0.13.0}/skills/mr-data-build/SKILL.md

@@ -59,6 +59,13 @@ not the other: silent work stays silent in both, and a message that is not worth
 not worth sending. This is a rule about the messages this contract already allows — it does not
 license one extra update.
 
+**Activity is not narration.** Setting the dataset's activity is a routine, silent act like any
+other command: it never produces a chat message and never a cell. The rule above governs what is
+said; activity governs what is happening. They answer different questions and never stand in for
+each other — a phase is not a message, and a message is not a substitute for the phase. So
+`mr-data dataset activity` is written wherever the stages below say to write it, in silence, under
+the same rule that keeps authentication checks and receipt parsing silent.
+
 For hosted work, register the Dataset first and open its canonical page immediately in the host's
 in-app browser: `mr-data open /datasets/DATASET_ID` mints a single-use address for it, and
 when that address cannot be minted, carry on without it rather than blocking.
@@ -105,7 +112,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data whoami` | Use the compatibility alias for the remotely validated `auth status` answer. |
 | `mr-data clarify` | Say whether one clarifying question may be put to the user right now, and compose the record of their answer. It reaches nothing, asks nobody and waits for nothing. Exits non-zero when no person can answer or when a recipe is already registered. |
 | `mr-data probe` | Ask one registered source one question and print the answer. **Currently unusable:** it reaches `/v3/sessions`, which Studio refuses, and it takes a question identifier that no v4 receipt returns. Do not plan a source inspection around it. |
-| `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --description-file F` writes the title and the description under the version it was read at, retrying once if somebody else wrote first; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream. |
+| `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --description-file F` writes the title and the description under the version it was read at, retrying once if somebody else wrote first; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. `mr-data recipe show ID` reads one back. |
 | `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
 | `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The command accepts `--sample`, `--full`, `--refresh`, `--backfill` and `--window`, and the worker executes `--sample`, `--full` and `--refresh` (a refresh re-acquires the declared addresses over the previous version and claims no window); `--backfill` and a `--window` on a sample or full are refused before acquisition. `--max-rows` and `--max-source-bytes` remain supported, and each bounds ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles; a sample-first held full requires Studio's interactive human approval instead. `--cancel RUN` stops one. |
@@ -124,16 +131,18 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data open` | Mint a single-use, short-lived address that continues this session in a browser at one dashboard path, and print it. `--open` hands it to the platform opener. The credential decides who mints: a device credential — what `mr-data login` stores — mints at Cloud, and a personal access token mints at Studio. |
 | `mr-data promote` | Ask Studio to publish one table after a person's decision. The Harness worker does not perform the catch-up or scheduled refresh; report those only when another deployed component returns durable evidence that it did. |
 | `mr-data reschedule` | Change how often one live table refreshes, without the demote and promote that would reset its bookkeeping and buy a catch-up run. `--cadence` takes a cron expression or an interval such as `every 6h`; `--why` records the reasoning. It starts no run. `--lock` freezes the schedule so Studio stops adjusting it; `--unlock` lets it follow the source again. |
-| `mr-data table` | Report what one table is serving and how its refresh schedule is going: the state in plain words, the schedule in force, the next refresh, the schedule that was asked for and the reason given for it, and — once Studio has settled on one — the interval it measured and how many source updates that rests on. |
+| `mr-data table` | Report what one table is serving and how its refresh schedule is going: the state in plain words, the schedule in force, the next refresh, the schedule that was asked for and the reason given for it, and — once Studio has settled on one — the interval it measured and how many source updates that rests on. `table rename TABLE --name NAME` writes a new name, lowercase snake_case and unique within its dataset, fenced on that dataset's version and retried once if somebody else wrote first; nothing is rebuilt, because every run, version and schedule hangs off the identifier, which does not move. `table archive TABLE --confirm-name NAME` retires the table and frees its name; a live table is refused until you `demote` it, and nothing built is deleted. |
 | `mr-data pin` | Ask Studio to freeze one table pointer on an exact version, for a rollback or a hold. |
 | `mr-data unpin` | Ask Studio to resume pointer tracking; read the returned state before claiming it resumed. |
 | `mr-data demote` | Ask Studio to withdraw the pointer and any schedule it owns. |
 
 `auth`, `login`, `whoami` and `clarify` are the same implementation in both profiles, and
-`clarify` alone reaches nothing at all. The other twenty-four answer from Studio. Nine only
-read: `status`, `runs`, `watch`, `peek`, `receipt`, `checks`, `download`, `diff` and `table`.
-Fifteen start or change something: `dataset`, `recipe`, `cover`, `run`, `query`, `probe`, `keys`,
-`stream`, `open`, `note`, `promote`, `reschedule`, `pin`, `unpin` and `demote`.
+`clarify` alone reaches nothing at all. The other twenty-four answer from Studio. Eight only
+read: `status`, `runs`, `watch`, `peek`, `receipt`, `checks`, `download` and `diff`.
+Sixteen start or change something: `dataset`, `recipe`, `cover`, `run`, `query`, `probe`, `keys`,
+`stream`, `open`, `note`, `promote`, `reschedule`, `table`, `pin`, `unpin` and `demote`. The split
+is over names, not over invocations: `table TABLE` reads and `table rename` writes, and the name
+sits on the side of whichever half changes something.
 
 **Any successful run is readable straight away.** There is no approval, no admission and no release
 step between a run finishing and reading it: `download`, `receipt`, `checks`, `peek` and `query`
@@ -221,7 +230,10 @@ live data key is refused with `THIN_AUTHENTICATION_FAILED` rather than failing t
 There is one loop, and it is short: open the page before you have anything to put on it, probe the
 sources, say out loud what you chose and what you refused, draft one recipe document, build a
 sample, interrogate what came back, fix the document, build again. Then present only what you
-inspected, with its caveats. Nothing between a run finishing and reading it is held for a person,
+inspected, with its caveats. The last two steps are conditional and `truncated` is what decides
+them: a sample that came back complete IS the table a full run would have sealed, so there is
+nothing to build again and nothing to ask about — only a TRUNCATED sample sends you round the
+loop. Nothing between a run finishing and reading it is held for a person,
 so there is no approval to seek, no separate body of evidence to read back, and no state to wait
 for; the only human gate in the product is [Promote](#promote), which comes after the loop has
 produced a table worth making live.
@@ -296,6 +308,30 @@ Hold the `dataset_id` for the rest of the session: stage 3 writes the decision r
 
 ### 2. Probe — read a source before committing to it
 
+**Say what is happening, three times, silently.** The dataset's activity is the state the page's
+own indicator reads. It is not a message and not a cell: write it and say nothing, exactly as you
+run every other command. Write it as the search starts, again once the shortlist is in hand, and
+again when the document is being written:
+
+```sh
+mr-data dataset activity DATASET_ID --phase researching \
+  --message "Looking for feeds that carry hourly stage for the Bagmati" --json
+mr-data dataset activity DATASET_ID --phase comparing \
+  --message "Comparing four candidate gauges against the question" \
+  --fact candidates=4 --fact sources_probed=2 --json
+mr-data dataset activity DATASET_ID --phase drafting \
+  --message "Writing the recipe: three gauges, one hourly table" --json
+```
+
+The message is one plain line about what you are doing right now, and the facts are counts you
+already hold — how many sources the discovery lanes shortlisted, how many of them you actually
+opened. Do not invent either; a fact you have not counted is a fact to leave out.
+
+**Then stop writing it by hand.** Every later phase comes from the commands themselves:
+registering a recipe, starting a run, reading a finished one, asking the user a question and
+promoting a table each move the page on their own. Stage 2 is the one stretch of the build where
+nothing has been submitted yet, which is why it is the one stretch that has to say so.
+
 **Judge the sources.** This stays with the external agent and is the substance of the task. Search
 every available discovery lane with no lane priority: a configured sealed catalog, host public-web
 search, official provider catalogs/APIs/download indexes/documentation, and user-supplied sources.
@@ -558,6 +594,11 @@ window into any source request, so any stated `--window` is refused before acqui
 mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --sample --max-rows 2000 --json
 ```
 
+Two thousand is the number to use before you have measured anything: it exercises every column,
+every cast and every declared check, and it comes back while the person who asked is still
+watching. Once stage 2 has measured the sources, raise it above the largest of them — the
+paragraphs below say why a ceiling under the real size costs more than the run it saved.
+
 **Both flags are required and neither is a positional.** `mr-data run RECIPE_ID` is refused:
 `--recipe` names the registered recipe and `--digest` states which bytes of it you mean to run,
 as the bare lowercase hex `recipe_digest` the registration receipt printed — no `sha256:` prefix.
@@ -566,16 +607,40 @@ a revision nobody read, which is why the client will not derive it for you. Hold
 from the registration receipt: every run of that recipe needs the pair.
 
 **`--max-rows` is a per-source ceiling, not a table size.** Each acquired source is truncated at
-that many rows, so a recipe with eight sources delivers up to eight times the number you asked for
-— 2,000 becomes 16,000. Read the delivered size off the run's `coverage` and say the number the run
-reports, never the number you passed.
-
-**Two thousand rows, and the first run is always this one.** A sample is for establishing shape,
-and a usable one has to arrive within minutes: two thousand rows a source exercises every column,
-every cast and every declared check, and it comes back while the person who asked is still watching
-the page. A larger first sample buys nothing the checks do not already report and costs the one
-thing the page cannot get back, which is the first few minutes. The full run comes later, in stage
-8, and only when the user says so.
+that many rows on its own, so a recipe with eight sources delivers up to eight times the number you
+asked for — 2,000 becomes 16,000. Read the delivered size off the run's `coverage` and say the
+number the run reports, never the number you passed.
+
+**And because it is per source, a low ceiling breaks a multi-source recipe's own checks.** Eight
+sources each cut at 2,000 rows are eight partial spans stitched into one table, and the seams are
+gaps that exist nowhere in the data: a continuity or completeness check that would pass over the
+whole thing fails over the sample, and it fails for a reason that has nothing to do with the
+recipe. Debugging that costs more than the run it saved. The ceiling is there to stop a source you
+have not measured from running away with the afternoon, not to make a small table.
+
+**So: when you have measured the sources, set the ceiling above the largest one.** Stage 2 opened
+these feeds and knows roughly how many rows each carries. Pick a number comfortably above the
+biggest of them and the sample acquires every source whole — no artificial seams, every declared
+check meaningful, and `coverage` untruncated on every entry, which is what makes the sample
+finished work rather than a shape test. Where a source was never opened, and its size is therefore
+a guess, two thousand rows a source is the fallback: it exercises every column, every cast and
+every declared check, and it comes back while the person who asked is still watching the page.
+
+**One Reader holds about 3 MiB of one source at a time, and that bounds what a source DELIVERS.**
+It is the largest single source rather than the total — one Reader opens one source at a time — and
+a source that really delivers more than that dies mid-run with `SANDBOX_MEMORY_LIMIT`. This is
+yours to apply to the sizes stage 2 measured, not something the command decides for you:
+`limits.max_source_bytes` states where fetching STOPS rather than how big the source is, so
+`mr-data recipe` does not refuse a large one. It registers the document and puts one sentence on
+the receipt under `warnings`, naming the source and both numbers. Read that as "go and check this
+one", not as a failure: a document declaring 8 MiB over a feed that answers 40 KB is fine.
+
+**If a source really does deliver more than the Reader holds, do not answer it by lowering the
+ceiling on that source:** that does not fetch less data, it truncates the sample or refuses the
+full run. Split the window across several sources — one per month, per region, or per whatever the
+publisher paginates on — and union them in a first transform step. A recipe names up to 64 sources,
+which is a great many windows. [The recipe document](../../docs/RECIPE-DOCUMENT.md) carries the
+measured numbers behind the ceiling.
 
 **The dataset's record is the primary one; a run's own record is for what happened inside that
 run.** The sources, the decisions and the questions went onto the dataset in stage 3, where they
@@ -692,38 +757,71 @@ the checks and the coverage say what you need them to say.
 ### 8. Present — only what survived inspection, with caveats
 
 Report what you inspected, and nothing you did not. Nothing else checks this run: `verified` is not
-a word some other worker said about it, it is what you established by reading it. So the honest
-sentence names the reading, and it says the word **sample** out loud:
-
-> Inspected 2,000 rows from the acquired source snapshot — a sample. All four declared checks
-> passed. Coverage was truncated at the row ceiling, and no run window was applied, so this is not
-> evidence for a stated calendar interval.
-
-Three facts carry that sentence and all three are on the run: how many rows you actually looked at,
-what the declared checks reported, and whether `coverage.truncated` was true. A null coverage
+a word some other worker said about it, it is what you established by reading it. Three facts carry
+whatever you say and all three are on the run: how many rows you actually looked at, what the
+declared checks reported, and whether `coverage.truncated` was true on any entry. A null coverage
 window means the acquisition was unwindowed, not that it covered all time. Never present a sampled
 row as a whole-population value, never call a run verified because it succeeded, and never report
 coverage you did not read off `coverage`.
 
-**Offer the sample's own bytes.** The sample is a sealed table like any other and its Parquet is
-downloadable — `mr-data download RUN --output DIR --json` — so somebody who wants to look at it
-before agreeing to a full run can. Say so.
+**Read `truncated` first, because it decides which of two endings this is.** Every coverage entry
+carries it. When it is false on every source, nothing was cut: each source was acquired whole, and
+the table this run sealed is the table a full run would have sealed. When it is true anywhere,
+something was cut, and what you have is a shape rather than an answer.
+
+**A complete sample is the build. Present it as one.** The mode a run was started in is not what
+makes a table finished — untruncated coverage is — and calling a complete table "a sample" tells
+somebody their dataset does not exist yet when it does. Say what was built and offer the decision
+that is actually in front of them, which is whether to make it live:
 
-**Then stop, and ask whether to run the full build.** This is a pause, and it is the second one
-this document allows alongside the promote decision. A full run over a real window costs real time
-and real money, and it is a decision about the user's data rather than about the recipe, so it is
-theirs and not yours. State what the sample established, what the full run would cover, and ask.
-Only when they say so:
+> Built 43,812 rows from four sources, every source acquired whole and nothing truncated. All four
+> declared checks passed. This is the table, not a sample of it.
+
+Then offer to enable it, and enable it when they say so — `mr-data promote TABLE_ID --cadence
+"every 6h" --why "..."`, which is [Promote](#promote), the one human gate. Do not run the whole
+thing again to produce a table you already have.
+
+**A truncated sample is not the build, and this is where the pause is.** State what the sample
+established, name what was cut and on which sources, say what a full run would cover, and ask. This
+pause is the second one this document allows alongside the promote decision, and it exists only
+here: a full run over a real window costs real time and real money, and it is a decision about the
+user's data rather than about the recipe, so it is theirs and not yours. Only when they say so:
 
 ```sh
 mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --json
 ```
 
+**The same decision is a button on the dataset page.** A truncated sample puts "Run the full
+build" on the agent pill, and pressing it starts the full run through the same spend gate the
+command does, bare — the exact coordinate the command above names — so the answer to this pause
+can arrive as words here or as a run you did not start. Ask, then give the turn back; do not hold
+a stream open waiting for either. Before acting on this recipe again — on their answer, or on your
+next turn — read `mr-data runs --json --mode full` and, for any run newer than the sample that you
+did not start, `mr-data status RUN_ID --json` to see whether it names this recipe and digest. One
+that does, with empty `clamps` and the default `resource_class`, is the pill's run, and its state
+says what to do. `queued`, `running` or `succeeded` is the full run, so adopt it as if the command
+above had started it — write its stage 5 cell, `mr-data watch RUN_ID --json` if it has not ended,
+then stage 6. `awaiting_confirmation` is the spend gate, which they settle from the page or by
+telling you to run the command again with `--confirm`. `awaiting_sample_approval` means Studio is
+sampling first: read its `sample_run_id` off that status and treat that sample as another stage 6
+while the full waits on their approval on the page, and if the sample is itself
+`awaiting_confirmation` that is the spend gate above, on the sample. `failed` is the full run too:
+read its triple as stage 6 says and report it, and do not start another, because a bare run of
+this coordinate replays that failure and a new attempt is a revision (stage 7). `cancelled` means
+they withdrew the decision, so say so and ask nothing further unless they raise it. Start a full
+run yourself only when they said so here and that list shows none, and start it bare — no
+ceiling, no resource class — because one coordinate is one run and a clamp is what would make a
+second.
+
 Then interrogate that run as stage 6 says, and present it with the sample's caveats removed and
 its own read off `coverage`. Do not start a full run because the sample looked good, because the
 checks passed, or because the user asked for the dataset in the first place — the request was for a
 dataset, and the decision in front of them is whether THIS is the recipe to spend a full run on.
 
+**Either way, offer the bytes.** A sealed table is downloadable whatever mode produced it —
+`mr-data download RUN --output DIR --json` — so somebody who wants to look before agreeing to
+anything can. Say so.
+
 ## Live run
 
 `mr-data run` answers with the run identifier and Studio's own run record. These are machine-facing
@@ -1102,7 +1200,7 @@ Own the run until the requested outcome is delivered: a run that succeeded, its
 read, its coverage inspected, and a publication request made when the user asked for it. Promotion
 is complete only when Studio returns the resulting live state; catch-up and continuing freshness
 require separate durable evidence from the deployed component that owns them. The allowed user pauses
-are the promote decision, the decision to start a full run after the sample, and a spend
+are the promote decision, the decision to start a full run after a TRUNCATED sample, and a spend
 confirmation on a run over the threshold. Do not ask the user to choose sources, Readers, engines,
 formats, workspaces, retry tactics, cleaning operations, or validation repairs. Make those
 operational decisions from the question, observed evidence, registered capabilities, and

{mostlyright_data-0.12.0 → mostlyright_data-0.13.0}/src/mostlyright/data_harness/cli.py

@@ -1371,6 +1371,11 @@ def _parser(
     parser.add_argument("--version", action="version", version=f"%(prog)s {package_version()}")
     subparsers = parser.add_subparsers(dest="command", required=True)
 
+    # ⚠ IMPORTED HERE RATHER THAN AT MODULE SCOPE, the rule every thin-lane import in this file
+    # follows: `cli` is the local front door and the hosted lane is loaded when a parser is built,
+    # not when this module is.
+    from mostlyright.data_harness.thin import activity as _thin_activity
+
     # `--json` lives on the subcommands, never on the top-level parser: one dest declared at both
     # levels lets the subparser default silently overwrite a value the caller gave up front.
     json_parent = argparse.ArgumentParser(add_help=False)
@@ -1381,9 +1386,17 @@ def _parser(
     )
 
     def _add(*args: Any, **kwargs: Any) -> argparse.ArgumentParser:
-        """Add one subcommand. Every subcommand takes `--json`; that is not optional."""
+        """Add one subcommand. Every subcommand takes `--json`; that is not optional.
+
+        And the commands that tell the dataset page what they are doing take `--no-activity`,
+        through the same function `thin.router` calls: every gate in the suite holds the two front
+        doors' argument surfaces equal command for command, so a flag one of them grows is a flag
+        the other grows in the same commit or not at all.
+        """
 
-        return subparsers.add_parser(*args, parents=[json_parent], **kwargs)
+        added = subparsers.add_parser(*args, parents=[json_parent], **kwargs)
+        _thin_activity.declare_opt_out(str(args[0]) if args else "", added)
+        return added
 
     # The arguments several commands share, written once. `--help` is the first thing anyone reads
     # and often the only thing, so an argument that says nothing is an argument someone has to go

{mostlyright_data-0.12.0 → mostlyright_data-0.13.0}/src/mostlyright/data_harness/hosted_stream_recorder_job.py

@@ -137,7 +137,7 @@ from mostlyright.data_harness.sources.stream_registry import (
 )
 from mostlyright.data_harness.sources.stream_rotation import run_series_rotation
 from mostlyright.data_harness.sources.stream_series import (
-    STREAM_SERIES_VERSION,
+    STREAM_SERIES_VERSIONS,
     load_stream_series,
     series_gap_report,
     series_windows,
@@ -1354,7 +1354,7 @@ def _sort_series_documents(registry_root: Path, series_root: Path) -> None:
             document = parse_json(path.read_bytes())
         except (CanonicalJSONError, OSError, TypeError, ValueError):
             continue
-        if isinstance(document, dict) and document.get("schema_version") == STREAM_SERIES_VERSION:
+        if isinstance(document, dict) and document.get("schema_version") in STREAM_SERIES_VERSIONS:
             os.rename(path, series_root / name)
 
 

{mostlyright_data-0.12.0 → mostlyright_data-0.13.0}/src/mostlyright/data_harness/hosted_stream_subscription_worker.py

@@ -29,6 +29,12 @@ from pathlib import Path
 from typing import Any, Protocol
 from urllib.parse import urlsplit
 
+from mostlyright.data_harness.acquisition.http import (
+    PinnedHttpsRetriever,
+    RetrievalLimits,
+    StdlibPinnedTransport,
+)
+from mostlyright.data_harness.acquisition.url_policy import EgressPolicy, SystemResolver
 from mostlyright.data_harness.canonical import sha256_bytes
 from mostlyright.data_harness.hosted_bootstrap import (
     BootstrapTransport,
@@ -38,6 +44,9 @@ from mostlyright.data_harness.hosted_bootstrap import (
 from mostlyright.data_harness.hosted_stream_recorder_job import (
     _production_prelude_transport as _recorder_prelude_transport,
 )
+from mostlyright.data_harness.hosted_stream_recorder_job import (
+    _sort_series_documents,
+)
 from mostlyright.data_harness.ingestion.gcs_store import (
     GCSObjectTransport,
     GoogleCloudStorageJSONTransport,
@@ -55,6 +64,12 @@ from mostlyright.data_harness.sources.stream_registry import (
     StreamConnectorEntry,
     load_stream_connector_registry,
 )
+from mostlyright.data_harness.sources.stream_series import (
+    MAX_DISCOVERY_RESPONSE_BYTES,
+    SERIES_DIRECTORY_NAME,
+    StreamSeriesConfig,
+    load_stream_series_documents,
+)
 from mostlyright.data_harness.stream_subscription_feed import (
     ConnectorFrameDecoder,
     ThreadedConnectorFeed,
@@ -64,9 +79,12 @@ from mostlyright.data_harness.stream_subscription_worker import (
     MAX_ACTIVE_SUBSCRIPTIONS,
     ClaimedSubscription,
     ContinuousSubscriptionWorker,
+    DiscoveryTransport,
+    RosterPlanner,
     StreamSubscriptionError,
     SubscriptionFeed,
     install_shutdown_handlers,
+    utc_now,
 )
 from mostlyright.data_harness.stream_venue import StreamVenueError, materialize_connector_registry
 from mostlyright.data_harness.studio_subscription_control import StudioSubscriptionControl
@@ -302,6 +320,154 @@ def _claim_text(value: str | None, label: str) -> str:
     return value
 
 
+class SubscriptionRosterResolver(Protocol):
+    """Build the roster planner one series-following claim needs."""
+
+    def create(self, claim: ClaimedSubscription) -> RosterPlanner: ...
+
+
+class ProductionSubscriptionRosterResolver:
+    """Resolve a claim's pinned series document into the planner that walks its roster.
+
+    The series arrives the way the connector does: as bytes inside the registry the claim pinned,
+    materialized into one private temporary directory, checked against the digest the claim named,
+    and parsed by the harness grammar that owns it.  Nothing about what a series SAYS is decided
+    here, and no second document reader exists: the recorder Job and this worker sort a flat
+    registry into its two loaders by the same rule.
+
+    The discovery leg is a pinned-egress HTTPS retriever narrowed to the one host the document
+    names, no redirects, JSON only, one bounded answer per page.  It is built per claim so a
+    series can never reach a host its own document did not write down.
+    """
+
+    def __init__(
+        self,
+        *,
+        work_root: Path,
+        objects: GCSObjectTransport | None = None,
+        discovery_transport: Callable[[StreamSeriesConfig], DiscoveryTransport] | None = None,
+        utc_clock: Callable[[], str] | None = None,
+    ) -> None:
+        self.work_root = work_root
+        self.objects = GoogleCloudStorageJSONTransport() if objects is None else objects
+        self.discovery_transport = (
+            _production_discovery_transport if discovery_transport is None else discovery_transport
+        )
+        self.utc_clock = utc_now if utc_clock is None else utc_clock
+
+    def create(self, claim: ClaimedSubscription) -> RosterPlanner:
+        series = claim.series
+        if series is None:
+            raise StreamSubscriptionError(
+                "STREAM_SUBSCRIPTION_SERIES_INVALID", "claim follows no series"
+            )
+        config = _materialized_claim_series(claim, self.objects, self.work_root)
+        if series.selection_rule is not None and config.selection_rule != series.selection_rule:
+            # Checked only where the claim states it: the document is the authority for the rule,
+            # and a claim that restates it and disagrees is a claim about another document.
+            raise StreamSubscriptionError(
+                "STREAM_SUBSCRIPTION_SERIES_INVALID", "materialized series selection rule"
+            )
+        return RosterPlanner(
+            config=config,
+            now=self.utc_clock,
+            discover=self.discovery_transport(config) if config.discovered else None,
+        )
+
+
+def _materialized_claim_series(
+    claim: ClaimedSubscription, objects: GCSObjectTransport, work_root: Path
+) -> StreamSeriesConfig:
+    """Materialize and verify precisely the series document this lease was granted for."""
+
+    series = claim.series
+    assert series is not None  # the caller refuses a claim that follows none
+    root = Path(tempfile.mkdtemp(prefix="stream-subscription-series-", dir=work_root))
+    try:
+        os.chmod(root, 0o700)
+        registry_root = root / "registry"
+        registry_root.mkdir(mode=0o700)
+        series_root = registry_root / SERIES_DIRECTORY_NAME
+        series_root.mkdir(mode=0o700)
+        materialize_connector_registry(
+            transport=objects,
+            bucket=_claim_text(claim.stream_venue_bucket, "stream venue bucket"),
+            registry_prefix=_claim_text(claim.connector_registry_prefix, "registry prefix"),
+            registry_digest=_claim_text(claim.connector_registry_digest, "registry digest"),
+            root=registry_root,
+        )
+        _sort_series_documents(registry_root, series_root)
+        documents = load_stream_series_documents(registry_root)
+        held = documents.get(series.series_id)
+        if held is None:
+            raise StreamSubscriptionError(
+                "STREAM_SUBSCRIPTION_SERIES_INVALID", "series is not in the pinned registry"
+            )
+        if held.document_digest != series.series_document_digest:
+            raise StreamSubscriptionError(
+                "STREAM_SUBSCRIPTION_SERIES_INVALID", "materialized series document digest"
+            )
+        if held.config.connector_id != claim.connector_id:
+            raise StreamSubscriptionError(
+                "STREAM_SUBSCRIPTION_SERIES_INVALID", "series rotates another connector"
+            )
+        return held.config
+    except StreamSubscriptionError:
+        raise
+    except (OSError, SourceContractError, StreamVenueError) as error:
+        raise StreamSubscriptionError(
+            "STREAM_SUBSCRIPTION_SERIES_INVALID", type(error).__name__
+        ) from error
+    finally:
+        shutil.rmtree(root, ignore_errors=True)
+
+
+def _production_discovery_transport(config: StreamSeriesConfig) -> DiscoveryTransport:
+    """The pinned HTTPS leg one discovered series reads its listing over.
+
+    This worker builds no HTTP client of its own: the retriever is the same pinned-egress one the
+    keyed acquisition lane uses, narrowed here to the one host the series document names, port
+    443, no redirects, JSON only, and one bounded answer per request.  A roster is read from a
+    literal address a reviewer approved, or it is not read.
+    """
+
+    discovery = config.discovery
+    if discovery is None:  # pragma: no cover - only a discovered series reaches this
+        raise StreamSubscriptionError(
+            "STREAM_SUBSCRIPTION_SERIES_INVALID", "series declares no discovery"
+        )
+    hostname = urlsplit(discovery.url).hostname
+    if not hostname:
+        raise StreamSubscriptionError(
+            "STREAM_SUBSCRIPTION_DISCOVERY_INVALID", "discovery address names no host"
+        )
+    retriever = PinnedHttpsRetriever(
+        resolver=SystemResolver(),
+        egress_policy=EgressPolicy(allowed_hostnames=(hostname,), allowed_ports=frozenset({443})),
+        transport=StdlibPinnedTransport(),
+        limits=RetrievalLimits(
+            max_response_bytes=MAX_DISCOVERY_RESPONSE_BYTES,
+            max_aggregate_response_bytes=MAX_DISCOVERY_RESPONSE_BYTES,
+            max_redirects=0,
+            # More than one request per page so a host answering on several addresses can be
+            # reached at all: the budget is spent walking the approved peer set, not following
+            # redirects, and a single-attempt budget dies on the first unroutable address.
+            max_requests=4,
+            max_ranges=1,
+            connect_timeout_seconds=5.0,
+            read_timeout_seconds=30.0,
+            total_timeout_seconds=45.0,
+            max_concurrency=1,
+            allowed_media_types=("application/json",),
+        ),
+    )
+
+    def fetch(url: str) -> bytes:
+        return retriever.retrieve(url).content
+
+    return fetch
+
+
 @dataclass(frozen=True)
 class SubscriptionWorkerRuntimeConfig:
     """Bounded deployment inputs; provider policy remains in the claimed subscription."""
@@ -381,6 +547,7 @@ def build_worker(
     sealer: GCSSubscriptionSealer | None = None,
     feed_resolver: SubscriptionFeedResolver | None = None,
     credential_resolver: SubscriptionCredentialResolver | None = None,
+    roster_resolver: SubscriptionRosterResolver | None = None,
     object_transport: GCSObjectTransport | None = None,
     stream_connect: StreamConnect | None = None,
     prelude_transport: Callable[[StreamConnectorEntry], PreludeTransport | None] | None = None,
@@ -420,12 +587,21 @@ def build_worker(
         if feed_resolver is None
         else feed_resolver
     )
+    active_rosters = (
+        ProductionSubscriptionRosterResolver(
+            work_root=config.work_root,
+            objects=active_objects,
+        )
+        if roster_resolver is None
+        else roster_resolver
+    )
     return ContinuousSubscriptionWorker(
         control=active_control,
         sealer=active_sealer,
         feed_factory=lambda claim: active_feeds.create(
             claim, credential_resolver=active_credentials
         ),
+        roster_factory=active_rosters.create,
         max_active_subscriptions=config.max_active_subscriptions,
         heartbeat_seconds=config.heartbeat_seconds,
         idle_poll_seconds=config.idle_poll_seconds,
@@ -540,8 +716,10 @@ def _bounded_float(
 __all__ = [
     "DEFAULT_WORK_ROOT",
     "HostedStreamSubscriptionWorkerError",
+    "ProductionSubscriptionRosterResolver",
     "SubscriptionCredentialResolver",
     "SubscriptionFeedResolver",
+    "SubscriptionRosterResolver",
     "SubscriptionWorkerRuntimeConfig",
     "build_control",
     "build_worker",