mostlyright-data 0.11.1__tar.gz → 0.12.0__tar.gz

{mostlyright_data-0.11.1 → mostlyright_data-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.11.1
+Version: 0.12.0
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.11.1 → mostlyright_data-0.12.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.11.1"
+version = "0.12.0"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.11.1 → mostlyright_data-0.12.0}/skills/mr-data-build/SKILL.md

@@ -88,14 +88,14 @@ the one command with no implementation behind it.
 There is no second profile and no lane to choose. `--hosted` is accepted and does nothing, because
 the lane it names is the only one; a script that already carries it keeps working.
 
-`mr-data --help` lists every command this product has — twenty-six of them, and the page is the
+`mr-data --help` lists every command this product has — twenty-eight of them, and the page is the
 whole list. A name that is not on it is not a command: typing one answers `THIN_UNKNOWN_COMMAND`,
 exactly as a misspelling does. Do not retry it under a different install, and do not report its
 absence as a defect.
 
 ## Commands
 
-These twenty-six are the whole surface, alongside `acquire-slices`, which names a capability
+These twenty-eight are the whole surface, alongside `acquire-slices`, which names a capability
 gap rather than doing anything, and `export-hosted-candidate`, which is a backend operator step.
 
 | Command | What it does |
@@ -123,15 +123,17 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data stream` | Record a public `wss://` venue so a build can read it. `stream document register --file F` admits the connector document, `stream registry create --digest D` admits a set of them, `stream probe` listens briefly and seals nothing, `stream capture --seconds N` records a bounded window, `stream subscribe` keeps recording, `stream subscription {show|pause|resume|stop} ID` moves one, `stream status ID` reads a capture or a probe, and `stream recordings --document-digest D` names what a build could read and until when. `--seconds` is at most 600. `mr-data run` takes no duration: a build reads sealed batches and is bounded by them. |
 | `mr-data open` | Mint a single-use, short-lived address that continues this session in a browser at one dashboard path, and print it. `--open` hands it to the platform opener. The credential decides who mints: a device credential — what `mr-data login` stores — mints at Cloud, and a personal access token mints at Studio. |
 | `mr-data promote` | Ask Studio to publish one table after a person's decision. The Harness worker does not perform the catch-up or scheduled refresh; report those only when another deployed component returns durable evidence that it did. |
+| `mr-data reschedule` | Change how often one live table refreshes, without the demote and promote that would reset its bookkeeping and buy a catch-up run. `--cadence` takes a cron expression or an interval such as `every 6h`; `--why` records the reasoning. It starts no run. `--lock` freezes the schedule so Studio stops adjusting it; `--unlock` lets it follow the source again. |
+| `mr-data table` | Report what one table is serving and how its refresh schedule is going: the state in plain words, the schedule in force, the next refresh, the schedule that was asked for and the reason given for it, and — once Studio has settled on one — the interval it measured and how many source updates that rests on. |
 | `mr-data pin` | Ask Studio to freeze one table pointer on an exact version, for a rollback or a hold. |
 | `mr-data unpin` | Ask Studio to resume pointer tracking; read the returned state before claiming it resumed. |
 | `mr-data demote` | Ask Studio to withdraw the pointer and any schedule it owns. |
 
 `auth`, `login`, `whoami` and `clarify` are the same implementation in both profiles, and
-`clarify` alone reaches nothing at all. The other twenty-two answer from Studio. Eight only
-read: `status`, `runs`, `watch`, `peek`, `receipt`, `checks`, `download` and `diff`. Fourteen
-start or change something: `dataset`, `recipe`, `cover`, `run`, `query`, `probe`, `keys`,
-`stream`, `open`, `note`, `promote`, `pin`, `unpin` and `demote`.
+`clarify` alone reaches nothing at all. The other twenty-four answer from Studio. Nine only
+read: `status`, `runs`, `watch`, `peek`, `receipt`, `checks`, `download`, `diff` and `table`.
+Fifteen start or change something: `dataset`, `recipe`, `cover`, `run`, `query`, `probe`, `keys`,
+`stream`, `open`, `note`, `promote`, `reschedule`, `pin`, `unpin` and `demote`.
 
 **Any successful run is readable straight away.** There is no approval, no admission and no release
 step between a run finishing and reading it: `download`, `receipt`, `checks`, `peek` and `query`
@@ -629,7 +631,7 @@ Four reads, and none of them waits on anything:
 
 ```sh
 mr-data peek RUN --json                       # columns, logical types, first rows
-mr-data query RUN "select count(*) from t" --json
+mr-data query RUN "select count(*) from run_table" --json
 mr-data checks RUN --json                     # every check the recipe declared
 mr-data receipt RUN --json                    # sources, digests, coverage, snapshot members
 ```
@@ -650,11 +652,13 @@ The refusals are cheap and legible by design, so read the code rather than retry
 | `QUERY_RESULT_TOO_LARGE` | The answer was over the byte ceiling. It is a refusal, never a truncation: a short answer that looked complete would be worse than none. |
 | `QUERY_TIMEOUT` | The statement outran its deadline. |
 | `QUERY_TABLE_UNAVAILABLE` | The run has no readable table — it did not succeed, or its artifact is not there. |
-| `QUERY_NO_WARM_EXECUTOR` | No warm worker holds this workspace. The refusal carries a retry window. |
+| `QUERY_NO_WARM_EXECUTOR` | No warm worker holds this workspace. `mr-data query` answers it by opening a research session for the run's dataset, asking, and closing it; with `--no-session` the refusal is printed instead, carrying a retry window. |
 
 **Studio deliberately does not start a worker on a query.** Scale-to-session is a cost invariant,
-so retrying `QUERY_NO_WARM_EXECUTOR` in a tight loop spends time against a wall. Wait through the
-window it names, or take a sample run first — a run warms the session that a query then reuses.
+so the cold start is spent only on a question actually asked: `mr-data query` opens a session for
+the run's own dataset when no worker is warm, waits for its worker (the states are printed as they
+happen), asks, and closes the session behind the answer. Pass `--no-session` to be refused instead,
+and then wait through the window the refusal names rather than retrying in a tight loop.
 
 **The statement gate is a word-boundary scan over a masked copy of the statement, and it does not
 read SQL.**
@@ -1053,9 +1057,32 @@ strength of having asked.
 Once the user has approved it:
 
 ```sh
-mr-data promote TABLE_ID --cadence source --json
+mr-data promote TABLE_ID --cadence "every 6h from 2026-09-03T02:40:00Z" --why "..." --json
 ```
 
+**Reason about the refresh schedule; do not reach for a default.** You have already read the
+source's own documentation and watched it answer, so you are the one party who knows anything about
+when it publishes. Say what you know: the times of day it posts, how often it changes, and how long
+after the stated publication time the bytes actually appear. `--cadence` takes a five-field cron
+expression, an interval such as `every 6h` or `every 30m from 2026-09-03T12:20:00Z`, or the word
+`source` to decline the question and let Studio choose. `--why` is one paragraph recording the
+reasoning, and it is required unless you asked for `source`; it is kept beside the table and read by
+whoever looks next, so write it for them rather than for yourself.
+
+**What you give is a seed, not a setting.** Studio watches what each source actually does on every
+refresh and moves the schedule to match, so a seed that is roughly right is worth far more than a
+safe guess, and one that is wrong is corrected rather than obeyed. Until the table reports its
+schedule as settled, the schedule is still being worked out: `mr-data table TABLE_ID` says which it
+is, in plain words, along with the schedule in force, the next refresh, what you asked for and why,
+and — once there is one — the interval Studio measured and how many source updates it rests on.
+**Never describe a schedule as learned, settled or confirmed while that command says otherwise.**
+
+`mr-data reschedule TABLE_ID --cadence ... --why "..."` changes the schedule of a table that is
+already live without starting a run or spending anything. Adding `--lock` freezes the schedule so
+Studio stops adjusting it, which is how a person overrides the evidence; `--unlock` lets it follow
+the source again. Do not lock a schedule on your own judgement — it is the same kind of decision as
+promotion, and it belongs to the user.
+
 This call is a publication request, not proof of data continuation. The current Harness worker
 cannot execute the catch-up backfill, refresh mode, or windowed acquisition needed to close the
 idle interval after the last successful run. It also does not attach or execute a warehouse

{mostlyright_data-0.11.1 → mostlyright_data-0.12.0}/src/mostlyright/data_harness/hosted_run_worker.py

@@ -39,7 +39,6 @@ TWO SEAMS THIS REPOSITORY DOES NOT OWN, both named here so a reader does not go
 from __future__ import annotations
 
 import argparse
-import contextlib
 import hashlib
 import io
 import json
@@ -57,6 +56,7 @@ from dataclasses import dataclass, field, replace
 from pathlib import Path
 from types import FrameType, MappingProxyType
 from typing import Any, ClassVar, NoReturn, Protocol
+from urllib.parse import urlsplit
 from uuid import UUID, uuid5
 
 from mostlyright.data_harness import pipeline, progress_events
@@ -109,19 +109,23 @@ from mostlyright.data_harness.recipe import RecipeError, parse_recipe_execution
 from mostlyright.data_harness.run_execution_v4 import (
     DOCUMENT_STAGE,
     FAILED_STAGES,
+    UNCHANGED_OUTCOME_MODES,
     RunCallbacks,
     RunExecution,
     RunExecutionError,
+    SourceChange,
     artifact_uploaded_body,
     compose_resource_limits,
     compose_source_limits,
     coverage_entry,
     engine_recipe,
+    every_source_unchanged,
     failed_body_for_unparsed,
     failed_body_from_error,
     failure_detail,
     parse_run_execution_v4,
     progress_body,
+    source_change_report,
     succeeded_body,
     synthesize_execution_lock,
 )
@@ -702,7 +706,12 @@ def permitted_artifact_hosts(
 
 
 def _download_signed_bytes(
-    transfer: Any, session: Any, *, maximum: int, permitted_hosts: frozenset[str]
+    transfer: Any,
+    session: Any,
+    *,
+    maximum: int,
+    permitted_hosts: frozenset[str],
+    subject: str = "the predecessor download session",
 ) -> bytes:
     """GET one Studio-issued signed download session, bounded, into memory.
 
@@ -716,6 +725,10 @@ def _download_signed_bytes(
     :func:`restore_predecessor_record` before the receipt is read.
 
     BOTH ADDRESS CHECKS RUN BEFORE THE STREAM OPENS, so a refused session costs no bytes.
+
+    ``subject`` is how a refusal names the session. The predecessor lane wrote this helper and
+    names it as its own; a lane that restates the refusal in its own words hands over the name
+    it wants read back, so its sentence does not send a reader after a predecessor.
     """
 
     # ONE POLICY FOR BOTH HALVES. `permitted_hosts` has already been proved to be a host this
@@ -728,7 +741,7 @@ def _download_signed_bytes(
     # that WAS permitted -- Studio's own storage origin, which is not a secret -- and stops there.
     reason = signed_storage_refusal(session.signed_url, permitted=permitted_hosts)
     if reason is not None:
-        raise _host_refusal(f"the predecessor download session is refused: {reason}")
+        raise _host_refusal(f"{subject} is refused: {reason}")
     chunks: list[bytes] = []
     total = 0
     # THE SIGNED URL REACHES NO LOG RECORD, which is the other half of the discipline the host
@@ -737,14 +750,12 @@ def _download_signed_bytes(
     # stream because the request line is written when the response arrives, not when it is sent.
     with _quiet_signed_transfer(), transfer.stream("GET", session.signed_url) as response:
         if int(response.status_code) != 200:
-            raise _predecessor_refusal(
-                f"the predecessor download session answered HTTP {response.status_code}"
-            )
+            raise _predecessor_refusal(f"{subject} answered HTTP {response.status_code}")
         for chunk in response.iter_bytes():
             total += len(chunk)
             if total > maximum:
                 raise _predecessor_refusal(
-                    "the predecessor artifact set exceeds the candidate envelope byte ceiling"
+                    f"the bytes behind {subject} exceed the candidate envelope byte ceiling"
                 )
             chunks.append(chunk)
     return b"".join(chunks)
@@ -1905,11 +1916,23 @@ def _evidence_uri(source: V4Source) -> str:
     So the address is sealed when the source has one, and the origin is what is sealed for a source
     addressed some other way, which has no requested URL to name. It is still a statement the
     DOCUMENT made: neither member is learned by fetching.
+
+    A STREAM SOURCE NAMES A SOCKET, AND THE EVIDENCE CONTRACT ADMITS ONLY HTTPS. Its recording
+    was made at a ``wss://`` address that no request in this run is made to, and the reference
+    the receipt seals must be an HTTPS URI. When the document states no origin, the venue's host
+    is sealed under ``https`` -- the same host, and still nothing learned by fetching -- rather
+    than the socket address the contract would refuse.
     """
 
     if source.locator.kind == "https_url":
         return source.locator.display_locator
-    return source.connector.origin or source.locator.display_locator
+    if source.connector.origin:
+        return source.connector.origin
+    if source.locator.kind == "stream":
+        host = urlsplit(source.locator.display_locator).hostname
+        if host:
+            return f"https://{host}"
+    return source.locator.display_locator
 
 
 def _workspace_document_cas_root(job: RunExecution) -> Path:
@@ -2514,6 +2537,7 @@ def build_v4_candidate(
     output_dir: Path,
     cursor: _StageCursor | None = None,
     narrate: _ItemNarrator | None = None,
+    halt_before_transform: Callable[[Mapping[str, str]], bool] | None = None,
 ) -> Any:
     """Acquire, transform and seal one registered document into a candidate.
 
@@ -2527,6 +2551,15 @@ def build_v4_candidate(
     the build and the replay are one function of the plan and the acquired bytes. That is why
     nothing here runs the statements first and hands the seal a result -- a result computed twice by
     two callers is a result two callers can disagree about.
+
+    ``halt_before_transform`` IS ASKED EXACTLY ONCE, AFTER EVERY SOURCE IS ACQUIRED AND BEFORE ANY
+    STATEMENT IS RUN, and answering true returns ``None`` instead of a candidate. It is handed the
+    digest of the bytes each source was read as, keyed by the source's own name -- the member the
+    sealed receipt states for that source, so a caller comparing against a predecessor compares
+    the same member on both sides. This is the only place that decision can be taken: the
+    acquisition is what produces the digests, and the transform is what a run with nothing new to
+    seal must not spend. Nothing is narrated as a transform on that path, and no cursor moves off
+    ``acquire``, because neither happened.
     """
 
     relations = acquire_v4_relations(
@@ -2538,6 +2571,13 @@ def build_v4_candidate(
         input_root=input_root,
         narrate=narrate,
     )
+    if halt_before_transform is not None and halt_before_transform(
+        # `content_sha256` is the coverage record's own `snapshot_sha256`; `relation_input`
+        # refuses a relation whose two spellings disagree, so reading it off the relation here
+        # and off the receipt there is one member read twice rather than two members.
+        {relation.relation_name: relation.content_sha256 for relation in relations}
+    ):
+        return None
     if cursor is not None:
         cursor.set("transform")
     # ⚠ THE STEPS ARE NARRATED FROM THE DOCUMENT, NOT FROM THE ENGINE, AND THAT IS WHY THEY ARE
@@ -2820,6 +2860,21 @@ def _as_run_failure(error: BaseException, *, stage: str) -> RunExecutionError:
     )
 
 
+def _reads_signed_storage(job: RunExecution) -> bool:
+    """Whether acquisition redeems a Studio-signed download: a snapshot input or a sealed recording.
+
+    Both are read through the same unauthenticated transfer client, and the driver opens that
+    client for itself when the caller supplied none. The predicate is the union of the two source
+    shapes that need it, because a stream-only recipe used to reach the recording adapter with no
+    client at all and was refused before its first progress report.
+    """
+
+    return bool(job.recordings) or any(
+        source.connector.adapter_id.partition("@")[0] == "snapshot.artifact"
+        for source in job.recipe.sources
+    )
+
+
 def run_v4(
     job: RunExecution,
     token: str,
@@ -2924,10 +2979,7 @@ def run_v4(
             snapshot_root = root / "snapshots"
             snapshot_root.mkdir(mode=0o700)
             active_sandbox = sandbox if sandbox is not None else _default_sandbox(root)
-            if transfer is None and any(
-                source.connector.adapter_id.partition("@")[0] == "snapshot.artifact"
-                for source in job.recipe.sources
-            ):
+            if transfer is None and _reads_signed_storage(job):
                 transfer = stack.enter_context(_closing(_new_transfer_client()))
             registry, contexts = _composed_registry(
                 job,
@@ -2972,13 +3024,42 @@ def run_v4(
                     ),
                     stage="transform",
                 )
+            # WHAT EACH SOURCE CARRIED THIS RUN, decided at the one moment every source has been
+            # acquired and nothing has been transformed. The report is this run's own; the halt is
+            # the question of whether the table has anything to seal.
+            changes: tuple[SourceChange, ...] = ()
+
+            def _halt(acquired: Mapping[str, str]) -> bool:
+                nonlocal changes
+                changes = source_change_report(
+                    mode=job.mode,
+                    acquired=acquired,
+                    predecessor_source_states=source_states,
+                )
+                # THE REPORT IS BUILT FOR EVERY MODE AND THE EARLY STOP IS NOT. A backfill has a
+                # predecessor to compare against, so its per-source answer is honest and worth
+                # reporting -- but it was dispatched to fill a named window, and matching bytes are
+                # no reason to decline that. The one mode that stops is the one that exists to ask.
+                if job.mode not in UNCHANGED_OUTCOME_MODES:
+                    return False
+                # ⚠ UNCHANGED BYTES ARE NOT AN UNCHANGED TABLE WHEN THE DOCUMENT ITSELF MOVED. The
+                # predecessor is resolved as the table's latest succeeded run, not as a run of this
+                # document, so a refresh dispatched against a REVISED recipe -- a dropped source, a
+                # rewritten transform, a changed check -- can read byte-identical sources and still
+                # owe a new version. Only the sources are compared above, so the revision would be
+                # swallowed silently and never reach the table. The digests the two runs sealed are
+                # what say whether the document is the same one.
+                if predecessor is None or predecessor.recipe_digest != job.recipe_digest:
+                    return False
+                return every_source_unchanged(changes)
+
             with progress_events.progress_sink(narration), progress_events.bridge_build_events():
                 # SERIAL by decision, and the decision is not re-taken here. The rationale
                 # ``hosted_worker`` records for a serial acquisition loop changes under in-process
                 # acquisition -- there is no job dispatch to serialize against any more -- but
                 # re-deciding it is a separate reviewed change, not a side effect of composing the
                 # primitive.
-                build_v4_candidate(
+                candidate = build_v4_candidate(
                     job,
                     engine,
                     approval,
@@ -2990,7 +3071,38 @@ def run_v4(
                     output_dir=output_dir,
                     cursor=stage,
                     narrate=narrate,
+                    halt_before_transform=_halt,
+                )
+
+            if candidate is None:
+                # NOTHING THIS TABLE HAS NOT ALREADY SEALED, so nothing after acquisition ran: no
+                # transform, no check pass, no artifact, no upload, and no version. The run
+                # SUCCEEDED -- it did what a refresh is for, which is to find out -- and the
+                # per-source report below is what the schedule that dispatched it reasons from.
+                #
+                # Neither the checks nor the coverage are reported. The checks are evaluated over a
+                # sealed candidate and this run sealed none, so naming them with an invented
+                # outcome would put a result on the record that no pass produced; coverage is what
+                # a run that DELIVERED something reports, and the contract refuses it here.
+                reported.succeeded(
+                    succeeded_body(
+                        job,
+                        receipt_digest=None,
+                        check_results=[],
+                        coverage=[],
+                        outcome="unchanged",
+                        sources=changes,
+                    )
                 )
+                return {
+                    "status": "succeeded",
+                    "run_id": str(job.run_id),
+                    "attempt": job.attempt,
+                    "mode": job.mode,
+                    "outcome": "unchanged",
+                    "sources": [item.to_wire() for item in changes],
+                    "check_results": [],
+                }
 
             stage.set("check")
             # The checks were PARSED before acquisition; this is the half that needs rows.
@@ -3065,6 +3177,8 @@ def run_v4(
                     receipt_digest=receipt.content_digest,
                     check_results=[item.to_wire() for item in results],
                     coverage=coverage,
+                    outcome="version",
+                    sources=changes,
                 )
             )
             return {
@@ -3072,6 +3186,8 @@ def run_v4(
                 "run_id": str(job.run_id),
                 "attempt": job.attempt,
                 "mode": job.mode,
+                "outcome": "version",
+                "sources": [item.to_wire() for item in changes],
                 "candidate_digest": "sha256:" + verified.candidate_digest,
                 "receipt_digest": receipt.content_digest,
                 "artifact_ids": [str(item.artifact_id) for item in payloads],
@@ -3100,8 +3216,8 @@ def run_v4(
                 traceback.print_exc(file=sys.stderr)
             try:
                 reported.failed(failed_body_from_error(job, refusal))
-            except Exception:
-                pass
+            except Exception as unreported:
+                _note_unreported_failure(unreported)
             raise refusal from error
 
 
@@ -3296,6 +3412,28 @@ def _fatal(error: BaseException) -> NoReturn:
     raise SystemExit(1)
 
 
+def _note_unreported_failure(error: BaseException) -> None:
+    """One stderr line saying the ``run:failed`` report itself was refused.
+
+    The report is best effort and its refusal must not become a second death, but a refusal that
+    was swallowed left a run record still saying ``running`` beside a container log that carried
+    the cause, with nothing connecting the two. This line is that connection: it precedes the
+    operator line :func:`_fatal` writes and names the refusal in the same three-member shape.
+    """
+
+    code, detail, _stage = _failure_triple(error)
+    line = json.dumps(
+        {"status": "unreported", "code": code, "detail": detail},
+        sort_keys=True,
+        separators=(",", ":"),
+    )
+    # A stderr that cannot be written must not displace the refusal the caller is about to raise.
+    try:
+        sys.stderr.write(line + "\n")
+    except OSError:
+        pass
+
+
 def _batch_studio_origin(environ: Mapping[str, str]) -> str:
     """The Studio origin the batch shape reports through, REQUIRED and HTTPS.
 
@@ -3347,9 +3485,10 @@ def _report_then_fatal(
     comes off the RAW mapping, because the reason the parse failed may be one of the three envelope
     members.
 
-    ⚠ AND THE REPORT IS BEST EFFORT, SUPPRESSED, exactly as ``run_v4``'s own is. A report that
-    fails must not become a second death: the caller is already dying, and the operator line
-    :func:`_fatal` writes is what an operator reads either way.
+    ⚠ AND THE REPORT IS BEST EFFORT, exactly as ``run_v4``'s own is. A report that fails must
+    not become a second death: the caller is already dying, and the operator line :func:`_fatal`
+    writes is what an operator reads either way. A refused report is noted on stderr first, so
+    the two lines together say both what killed the run and that the record never learned it.
     """
 
     if callbacks is not None:
@@ -3363,8 +3502,10 @@ def _report_then_fatal(
             failed_stage=stage,
         )
         if body is not None:
-            with contextlib.suppress(Exception):
+            try:
                 callbacks.failed(body)
+            except Exception as unreported:
+                _note_unreported_failure(unreported)
     _fatal(error)