mostlyright-data 0.10.4__tar.gz → 0.10.5__tar.gz

{mostlyright_data-0.10.4 → mostlyright_data-0.10.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.10.4
+Version: 0.10.5
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.10.4 → mostlyright_data-0.10.5}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.10.4"
+version = "0.10.5"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.10.4 → mostlyright_data-0.10.5}/skills/mr-data-build/SKILL.md

@@ -85,7 +85,7 @@ absence as a defect.
 
 ## Commands
 
-These twenty-four are the whole surface, alongside `acquire-slices`, which names a capability
+These twenty-five are the whole surface, alongside `acquire-slices`, which names a capability
 gap rather than doing anything, and `export-hosted-candidate`, which is a backend operator step.
 
 | Command | What it does |
@@ -97,7 +97,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data probe` | Ask one registered source one question and print the answer. **Currently unusable:** it reaches `/v3/sessions`, which Studio refuses, and it takes a question identifier that no v4 receipt returns. Do not plan a source inspection around it. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. `mr-data recipe show ID` reads one back. |
 | `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
-| `mr-data run` | Start one run against a registered recipe. The command accepts `--sample`, `--full`, `--refresh`, `--backfill` and `--window`, but the current worker executes only unwindowed `--sample` and `--full`; the other coordinates are refused before acquisition. `--max-rows` and `--max-source-bytes` remain supported. A large run is held for a spend confirmation, which `--confirm` settles; a sample-first held full requires Studio's interactive human approval instead. `--cancel RUN` stops one. |
+| `mr-data run` | Start one run against a registered recipe. The command accepts `--sample`, `--full`, `--refresh`, `--backfill` and `--window`, and the worker executes `--sample`, `--full` and `--refresh` (a refresh re-acquires the declared addresses over the previous version and claims no window); `--backfill` and a `--window` on a sample or full are refused before acquisition. `--max-rows` and `--max-source-bytes` remain supported. A large run is held for a spend confirmation, which `--confirm` settles; a sample-first held full requires Studio's interactive human approval instead. `--cancel RUN` stops one. |
 | `mr-data status` | Report which of the seven states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed and where. Exits non-zero on a failed run. |
 | `mr-data runs` | Report this workspace's own runs: identifier, mode, state and creation time, plus the failure triple of any that failed. `--status` and `--mode` narrow it and are refused for a word outside the seven or the four; `--limit` says how many, and pages are followed to reach it. |
 | `mr-data watch` | Stream one run's live progress under the durable event type of each stage, resuming across stream cuts. Exits non-zero when the run failed. |
@@ -109,6 +109,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data download` | Bring one run's artifacts back to this computer, each one's bytes checked against the digest Studio sealed them under. `--kind` selects out of `table_parquet`, `column_profile`, `preview`, `receipt` and `raw_snapshot`; without it, the table and the receipt. |
 | `mr-data diff` | Compare two runs and say what changed. |
 | `mr-data keys` | Enrol a source credential by name, list the names with their created and rotated times, and delete one. `keys set` reads the value from a file or from standard input and never from the command line; no command prints a value back. |
+| `mr-data stream` | Record a public `wss://` venue so a build can read it. `stream document register --file F` admits the connector document, `stream registry create --digest D` admits a set of them, `stream probe` listens briefly and seals nothing, `stream capture --seconds N` records a bounded window, `stream subscribe` keeps recording, `stream subscription {show|pause|resume|stop} ID` moves one, `stream status ID` reads a capture or a probe, and `stream recordings --document-digest D` names what a build could read and until when. `--seconds` is at most 600. `mr-data run` takes no duration: a build reads sealed batches and is bounded by them. |
 | `mr-data open` | Mint a single-use, short-lived address that continues this session in a browser at one dashboard path, and print it. `--open` hands it to the platform opener. A handoff is minted only for a personal access token; a device credential is refused with `THIN_STUDIO_HANDOFF_REQUIRES_PERSONAL_ACCESS_TOKEN`, so give the user the plain dashboard address instead. |
 | `mr-data promote` | Ask Studio to publish one table after a person's decision. The Harness worker does not perform the catch-up or scheduled refresh; report those only when another deployed component returns durable evidence that it did. |
 | `mr-data pin` | Ask Studio to freeze one table pointer on an exact version, for a rollback or a hold. |
@@ -116,9 +117,9 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data demote` | Ask Studio to withdraw the pointer and any schedule it owns. |
 
 `auth`, `login`, `whoami` and `clarify` are the same implementation in both profiles, and
-`clarify` alone reaches nothing at all. The other twenty answer from Studio. Eight only
-read: `status`, `runs`, `watch`, `peek`, `receipt`, `checks`, `download` and `diff`. Twelve
-start or change something: `recipe`, `cover`, `run`, `query`, `probe`, `keys`, `open`,
+`clarify` alone reaches nothing at all. The other twenty-one answer from Studio. Eight only
+read: `status`, `runs`, `watch`, `peek`, `receipt`, `checks`, `download` and `diff`. Thirteen
+start or change something: `recipe`, `cover`, `run`, `query`, `probe`, `keys`, `stream`, `open`,
 `note`, `promote`, `pin`, `unpin` and `demote`.
 
 **Any successful run is readable straight away.** There is no approval, no admission and no release
@@ -461,7 +462,7 @@ still going. [Narrating the run](#narrating-the-run) is the reference for what a
 protocol, on `--sample`, `max_rows` and `window` truncate: the run succeeds at the clamp boundary
 and reports the coverage it actually delivered. `max_source_bytes` refuses in every mode, sample
 included, and on `full`, `refresh` and `backfill`, hitting any clamp is a refusal with
-`CLAMP_EXCEEDED`. The current worker admits only unwindowed `sample` and `full` before applying
+`CLAMP_EXCEEDED`. The current worker admits `sample`, `full` and `refresh` (unwindowed in effect) and refuses `backfill` before applying
 those dispositions. Therefore an executable sample may truncate on `max_rows`; an executable full
 refuses a crossed row or byte ceiling; every stated window, refresh, and backfill is refused before
 a byte is fetched.
@@ -475,9 +476,11 @@ cost, and waits; `mr-data run --confirm RUN` settles it against the number alrea
 Any authenticated surface of the paying workspace may settle it. `--cancel RUN` stops a run that
 is queued or running.
 
-Do not treat the accepted command vocabulary as execution evidence. `refresh` and `backfill` are
-defined as parameterized full runs over half-open windows, but this worker cannot apply those
-windows or restore the predecessor and therefore executes neither mode.
+Do not treat the accepted command vocabulary as execution evidence. A `refresh` restores the
+previous version, re-acquires the declared addresses in full and seals the next version, claiming
+no coverage window -- it is how an enabled table's catch-up and its scheduled refresh run. A
+`backfill` is defined by the exact window it covers, which this worker cannot narrow a source
+request to, and is refused before acquisition.
 
 ### 4. Interrogate — ask the run what it actually delivered
 
@@ -776,6 +779,60 @@ or this command line.
 If a source refuses for want of a credential, that is a product-level fact worth one sentence to
 the user — the source needs a credential and which one — and nothing about the storage mechanism.
 
+## Recording a stream venue
+
+Some sources have no file to fetch: a `wss://` venue emits messages only while a socket is open, so
+the bytes a build reads have to be recorded first. `mr-data stream` is that recording, and it is
+four steps in front of the ordinary build loop.
+
+```sh
+mr-data stream document register --file connector.json --name coinbase --json
+mr-data stream registry create --digest DOCUMENT_DIGEST --json
+mr-data stream probe --document-digest DOCUMENT_DIGEST --registry REGISTRY_ID \
+  --member BTC-USD --json
+mr-data stream capture --document-digest DOCUMENT_DIGEST --registry REGISTRY_ID \
+  --member BTC-USD --seconds 300 --watch --json
+```
+
+1. **Register the connector document.** It says where the venue is, how a message names its own
+   type and event time, and what a subscription frame looks like. Registration prints the
+   fingerprint the recipe will name; `mr-data stream document show ID` reads one back.
+2. **Admit it into a registry.** A registry is the reviewed set a recording may name. Pass every
+   fingerprint the recording will use.
+3. **Probe before you capture.** A probe connects briefly, reports which message types the document
+   classified and which it did not, and seals nothing. Read it with
+   `mr-data stream status ID --kind probe`. A document that classifies nothing is a document to fix
+   before spending a capture window on it.
+4. **Capture, or subscribe.** `capture --seconds N` records one bounded window and seals it;
+   `--seconds` is at most 600, and `--max-events` and `--max-bytes` bound it further. `subscribe`
+   keeps recording until `mr-data stream subscription stop ID`, checkpointing as it goes.
+   `--watch` on a capture reads the record back until it stops and prints what stopped it.
+
+The recipe then names the venue by its fingerprint, as `stream_connector_document_digest` on the
+source, and `mr-data run` builds from the sealed messages. **`run` has no duration flag and will
+not grow one.** A build reads sealed batches and is bounded by them, which is what makes it
+reproducible: the bytes were fixed when they were recorded, not when the build was started. If a
+build refuses for want of a recording, run `mr-data stream recordings --document-digest D` — it
+reports every batch a build could read for that venue, the Bookmark each one carries, and when each
+expires.
+
+**A missed window is a permanent hole.** Recordings are kept for thirty days and a refresh carries
+on from the Bookmark the last build reached; neither of those goes back for messages nobody was
+listening for. If a venue was not being recorded over a period the user cares about, say so as a
+gap in the series rather than promising a backfill: there is nothing to fetch.
+
+**A keyed venue needs a shaped credential.** Store it once, by part, and name it on the recording:
+
+```sh
+mr-data keys set VENUE_KEY --part key_id=ABC123 --part private_key_pem=@signing.pem --json
+mr-data stream capture --document-digest D --registry R --member M --seconds 300 \
+  --secret VENUE_KEY --json
+```
+
+`--part NAME=@PATH` reads that part off disk, which is how a signing key arrives; the parts are
+composed into one envelope here and the value reaches nothing but the request body. The venue's
+document declares which parts its shape holds, and a set that does not match is refused by name.
+
 ## Promote
 
 Promotion is the one human gate in the product. Everything before it is workspace scratch: probing,

{mostlyright_data-0.10.4 → mostlyright_data-0.10.5}/src/mostlyright/data_harness/hosted_run_worker.py

@@ -127,6 +127,7 @@ from mostlyright.data_harness.run_execution_v4 import (
 )
 from mostlyright.data_harness.sources.contracts import SourceContractError
 from mostlyright.data_harness.v4_acquisition import (
+    STREAM_RELATION_COLUMNS,
     RelationInput,
     bind_v4_source,
     connector_authority,
@@ -1025,6 +1026,14 @@ def _receipt_document(
                 for item in coverage
             ],
             "coverage": [item.to_dict() for item in coverage],
+            # WHAT EACH STREAM SOURCE ACTUALLY READ, and the member Studio persists
+            # `bookmark_after` from. It is `{}` for a run that declares none. No transfer
+            # capability appears in it: a signed session is a short-lived bearer credential and
+            # belongs in no durable record. The digest `run:succeeded.receipt_digest` names is
+            # taken over these bytes, so listing the block here is what binds it.
+            "recordings": {
+                name: job.recordings[name].sealed_recording() for name in sorted(job.recordings)
+            },
             "raw_snapshot": {
                 "media_type": RAW_SNAPSHOT_MEDIA_TYPE,
                 "members": [dict(member) for member in raw_members],
@@ -1038,6 +1047,7 @@ def v4_artifact_payloads(
     candidate_run_dir: Path,
     *,
     coverage: Sequence[V4AcquisitionCoverage] = (),
+    predecessor_run_dir: Path | None = None,
 ) -> tuple[pipeline.VerifiedCandidate, list[ArtifactPayload]]:
     """Derive all five v4 artifacts from the sealed candidate tree, in layout order.
 
@@ -1075,7 +1085,18 @@ def v4_artifact_payloads(
             stage="persist",
         )
     verified, snapshot = _candidate_snapshot(candidate_run_dir)
-    run_required_controls(job, verified=verified, snapshot=snapshot, coverage=coverage)
+    # The restored predecessor tree, for the drift control a refresh is held to. Read through the
+    # same replay-verifying door as the candidate: what is compared is what was sealed.
+    predecessor_snapshot = (
+        None if predecessor_run_dir is None else _candidate_snapshot(predecessor_run_dir)[1]
+    )
+    run_required_controls(
+        job,
+        verified=verified,
+        snapshot=snapshot,
+        coverage=coverage,
+        predecessor_snapshot=predecessor_snapshot,
+    )
     candidate_digest = "sha256:" + verified.candidate_digest
     # The tar is built BEFORE the receipt, because the receipt states the tar's members: the
     # container is one artifact and the per-member digests are the only thing that keeps it
@@ -1349,6 +1370,7 @@ def _composed_registry(
         DocumentProjectionAdapter,
     )
     from mostlyright.data_harness.sources.registry import AdapterRegistry
+    from mostlyright.data_harness.v4_stream_recording import RecordedStreamAdapter
 
     limits = [compose_source_limits(ceilings, job) for ceilings in source_ceilings(job.recipe)]
     if not limits:
@@ -1359,15 +1381,22 @@ def _composed_registry(
     disposition = limits[0].row_clamp_disposition
     origins = {item.row_limit_origin for item in limits}
     origin = "run_clamp" if origins == {"run_clamp"} else "registration"
-    window = job.clamps.window
+    # A refresh's stated window is Studio's tick coordinate, not a range this lane applies:
+    # no request is narrowed to it and no coverage entry claims it. See EXECUTABLE_RUN_MODES.
+    window = None if job.mode == "refresh" else job.clamps.window
     # Structural rather than admitted, the same way the Reader pin is read: this asks only
     # WHETHER a source names a coordinate, and the adapter is what verifies it against the
     # capture store. Resolving twice would be two readings of one document that could disagree.
     document_sources = tuple(
         source for source in job.recipe.sources if document_projection_parameters(source)
     )
+    # A source read from a recording reaches no address, so it is routed away from the network
+    # loop below rather than falling through its connector filter unrouted.
+    stream_sources = tuple(source for source in job.recipe.sources if source.name in job.recordings)
     network_sources = tuple(
-        source for source in job.recipe.sources if source not in document_sources
+        source
+        for source in job.recipe.sources
+        if source not in document_sources and source not in stream_sources
     )
     if document_sources and disposition == "truncate":
         raise RunExecutionError(
@@ -1382,7 +1411,9 @@ def _composed_registry(
         for source in network_sources
         if source.connector.adapter_id == GENERIC_API_ADAPTER_ID
     }
-    redeemed = {source.name for source in document_sources}
+    # Both lanes order exactly one confined `parse` over bytes the operator already holds, and
+    # neither orders a decode, so both pin the narrow policy rather than the composed one.
+    redeemed = {source.name for source in (*document_sources, *stream_sources)}
     adapters: list[Any] = []
     if network_sources:
         # `fetch` is the adapter's own declared seam (`InProcessPublicHttpsAdapter.fetch`),
@@ -1484,6 +1515,38 @@ def _composed_registry(
                 adapters=MappingProxyType(document_adapters),
             )
         )
+    if stream_sources:
+        if transfer is None or not base_url:
+            raise RunExecutionError(
+                "RUN_RECIPE_INVALID",
+                failure_detail(
+                    "reading a sealed recording requires a Studio origin and a transfer client"
+                ),
+                stage="acquire",
+            )
+        hosts = signed_storage_hosts(studio_origin=base_url, environ=os.environ)
+        stream_adapters = {
+            source.name: _CoveredRecordedStreamAdapter(
+                adapter=RecordedStreamAdapter(
+                    recording=job.recordings[source.name],
+                    sandbox=sandbox,
+                    governance=recipe_governance(
+                        job, requested_at=trusted_clock, sources=(source,)
+                    ),
+                    transfer=transfer,
+                    permitted_hosts=hosts,
+                ),
+                coverage_log=coverage_log,
+            )
+            for source in stream_sources
+        }
+        first_stream = next(iter(stream_adapters.values()))
+        adapters.append(
+            _PerSourceAdapterRouter(
+                descriptor=first_stream.descriptor,
+                adapters=MappingProxyType(stream_adapters),
+            )
+        )
     registry = AdapterRegistry(tuple(adapters))
     contexts = {
         item.source_id: item.acquisition_context(
@@ -1599,6 +1662,51 @@ class _CoveredDocumentProjectionAdapter:
         return output
 
 
+@dataclass(frozen=True)
+class _CoveredRecordedStreamAdapter:
+    """Add v4 receipt coverage to a replayed recording, on the projection wrapper's own terms.
+
+    A recording reaches no venue and uses no session cache, so ``miss`` means no cached adapter
+    output was used and the byte count is the concatenated sealed batches this run actually read.
+    The URL is the content URN of those bytes and never a capability address.
+    """
+
+    adapter: Any
+    coverage_log: V4CoverageLog
+
+    @property
+    def descriptor(self) -> Any:
+        return self.adapter.descriptor
+
+    def acquire(self, request: Any, context: Any) -> Any:
+        output = self.adapter.acquire(request, context)
+        rows = output.parsed_row_count
+        if rows is None:
+            raise RunExecutionError(
+                "RUN_EXECUTION_FAILED",
+                "the replayed recording did not report parsed row facts",
+                stage="parse",
+            )
+        self.coverage_log.record(
+            V4AcquisitionCoverage(
+                source_id=request.source_id,
+                query_digest=request.query_digest,
+                window=None,
+                cache=CacheDisposition.MISS,
+                url=output.receipt.final_uri,
+                fetched_at=output.receipt.observed_at,
+                snapshot_sha256=output.receipt.snapshot.content_sha256,
+                snapshot_bytes=output.receipt.snapshot.size_bytes,
+                fetched_content_sha256=output.receipt.snapshot.content_sha256,
+                bytes_fetched=output.receipt.snapshot.size_bytes,
+                rows_kept=rows,
+                rows_available=rows,
+                truncated=False,
+            )
+        )
+        return output
+
+
 def _allowed_hostnames_by_source(job: RunExecution) -> Mapping[str, tuple[str, ...]]:
     """The exact host allowlist this run's own recipe states, PER SOURCE and nothing wider.
 
@@ -2069,6 +2177,7 @@ def acquire_v4_relations(
                     request_id=f"run.{job.run_id}.{job.attempt}.{source.name}",
                     requested_at=requested_at,
                     credential_handle=job.credential_handle_for(source.name),
+                    recording=job.recordings.get(source.name),
                 ),
             )
         )
@@ -2091,7 +2200,15 @@ def acquire_v4_relations(
         relation = relation_input(
             output,
             relation_name=source.name,
-            columns=output.parsed_columns or (),
+            # A STREAM SOURCE'S HEADER IS DERIVED AND EVERY OTHER SOURCE'S IS REPORTED. The
+            # Recorder's sealed line fixes it, the same for every stream source of every recipe,
+            # and the adapter has already refused a parse that reported anything else -- so
+            # taking it from the constant rather than from the parse says where it comes from.
+            columns=(
+                STREAM_RELATION_COLUMNS
+                if source.name in job.recordings
+                else output.parsed_columns or ()
+            ),
             coverage=_coverage_for(coverage_log, source.name, output),
             limits=limits,
         )
@@ -2267,25 +2384,27 @@ def _restore_termination_handlers(installed: Sequence[tuple[int, Any]]) -> None:
         signal.signal(number, previous)
 
 
-#: The run modes this worker executes, and the two it refuses rather than half-performing.
-#:
-#: A ``refresh`` and a ``backfill`` are both defined AGAINST a predecessor and a window, and this
-#: worker can establish neither. The window narrows nothing on this lane -- ``bind_v4_source``
-#: composes the exact three members ``public.https`` admits from the address the document states,
-#: and the fetch takes whatever that address returns -- so the two modes whose whole meaning is
-#: "cover this range and no other" would cover whatever arrived while their sealed execution and
-#: their run record said otherwise.
+#: The run modes this worker executes, and the one it refuses rather than half-performing.
 #:
-#: THAT IS WHY THIS IS A REFUSAL RATHER THAN A GAP. ``docs/V4-WORKER-PROTOCOL.md`` has Studio fold
-#: the per-source windows by intersection and compute a later refresh's resumption point from the
-#: result, so a window this worker states is an input to somebody else's correctness. A mode that
-#: half-works and seals a false range is strictly worse than one that refuses: the refusal is
-#: visible on the run record at the moment it happens, and the false range is discovered by the
-#: run that trusted it.
+#: A ``refresh`` on this lane is an unwindowed re-acquisition over the restored predecessor: every
+#: registered source is unwatermarked (``run_execution_v4._refuse_unbindable_watermark``), the
+#: fetch takes whatever the registered address returns, and the engine's ``refresh`` mode asks
+#: for exactly a predecessor and no window. So a refresh can be honest here -- it seals the rows
+#: it fetched as the next version of the same table and claims NO coverage window, so a later
+#: refresh resumes from the run's completion rather than from a range this lane never applied.
+#: Studio states ``clamps.window`` on a refresh to keep consecutive ticks distinct runs; that
+#: window is not applied to any request and is not recorded as covered.
 #:
-#: The refusal is lifted by narrowing the fetch, not by restating the clamp. Until then a run in
-#: either mode is refused before a byte is fetched or a predecessor is restored.
-EXECUTABLE_RUN_MODES: tuple[str, ...] = ("sample", "full")
+#: A ``backfill`` is different in kind and stays refused: the engine defines it BY its exact
+#: window -- it refuses a partition holding an event outside that range -- and this lane cannot
+#: narrow a source request to a range, so a backfill that fetched the whole address would either
+#: refuse at the engine or seal a range it did not cover. The refusal is visible on the run record
+#: at the moment it happens; a false range is discovered by the run that trusted it.
+EXECUTABLE_RUN_MODES: tuple[str, ...] = ("sample", "full", "refresh")
+
+#: The modes on which a stated ``clamps.window`` is a contradiction rather than a request: a
+#: sample or a full with a window would be a windowed run this lane cannot perform.
+_WINDOW_REFUSED_MODES: frozenset[str] = frozenset({"sample", "full"})
 
 
 def _admit_executable_mode(job: RunExecution) -> None:
@@ -2295,13 +2414,13 @@ def _admit_executable_mode(job: RunExecution) -> None:
         raise RunExecutionError(
             "RUN_MODE_INVALID",
             failure_detail(
-                f"this worker executes a run in {' or '.join(EXECUTABLE_RUN_MODES)} mode, and a "
-                f"{job.mode} run is defined against a query window it would not apply, so its "
-                f"sealed record would state a range it did not cover"
+                f"this worker executes a run in {', '.join(EXECUTABLE_RUN_MODES[:-1])} or "
+                f"{EXECUTABLE_RUN_MODES[-1]} mode, and a {job.mode} is defined by the exact "
+                "window it covers, which this worker cannot narrow a source request to"
             ),
             stage="acquire",
         )
-    if job.clamps.window is not None:
+    if job.clamps.window is not None and job.mode in _WINDOW_REFUSED_MODES:
         raise RunExecutionError(
             "RUN_CLAMPS_INVALID",
             failure_detail(
@@ -2597,7 +2716,10 @@ def run_v4(
 
             stage.set("persist")
             verified, payloads = v4_artifact_payloads(
-                job, output_dir, coverage=coverage_log.entries()
+                job,
+                output_dir,
+                coverage=coverage_log.entries(),
+                predecessor_run_dir=previous_run_dir,
             )
             _upload_all(
                 job,