mostlyright-data 0.9.0__py3-none-any.whl

mostlyright/data_harness/progress_events.py

@@ -0,0 +1,534 @@
+"""Unsealed run-progress events for the live hosted timeline (ADR 0021).
+
+WHAT THIS IS NOT. This is not the Build event vocabulary. `events` holds that one, and it holds an
+invariant this module deliberately does not: every BUILD name there must be re-derivable from
+sealed bytes by `events.project_sealed_run`, which is why a time-throttled heartbeat is refused
+there and why the module says so in its own docstring. Widening that vocabulary to move a progress
+bar would trade a proof for a spinner. So progress got its own vocabulary instead, modelled on the
+one exception that vocabulary already carries -- the post-seal notebook sidecar, three names that
+narrate an unsealed convenience write and are separated so a consumer cannot mistake them for Build
+evidence.
+
+THE BOUNDARY, stated as the five things a reader has to be able to rely on:
+
+* **Disjoint names.** `PROGRESS_EVENT_NAMES` shares not one member with `events.EVENT_NAMES` or
+  `events.RESERVED_EVENT_NAMES`. Every progress name that mirrors a Build name carries the
+  ``build_`` prefix precisely so the two can never be confused in a log line, a dashboard row or a
+  grep. A test asserts the disjointness rather than trusting the prefix.
+* **Blind to projection.** `events.project_sealed_run` has no branch for any name here and must
+  never grow one. A projection of a sealed Build is the same list of records whether or not a
+  progress stream existed, because a progress record is not derived from sealed bytes and could
+  not be re-derived from them afterwards. A test projects a real Build and asserts the projection
+  names are exactly the sealed vocabulary.
+* **Outside attestation.** No progress record reaches a candidate member, a manifest, a digest that
+  anything is verified against, or a receipt. The only digest a progress record produces is
+  `progress_payload_digest`, which is a content address for the record's OWN payload on the wire --
+  it attests the payload has not been rewritten in transit, and it attests nothing about the Build.
+  A verifier never reads this stream, and the verifier lane emits nothing here: it holds a signing
+  key and stays job-per-run (ADR 0021).
+* **Never load-bearing.** Emission is best effort by contract. `emit` with no sink armed is a
+  ContextVar read and a return. A sink that raises is caught by the caller of record and the run
+  proceeds. A run that emitted zero progress events and a run that emitted all of them produce
+  byte-identical Builds.
+* **Bounded, not chatty.** The vocabulary covers stage boundaries and per-source, per-artifact and
+  per-stage milestones. It has no per-row, per-byte, per-chunk or wall-clock heartbeat name, and
+  `MAX_PROGRESS_EVENTS_PER_ATTEMPT` is a hard per-attempt ceiling on top of that, so the stream
+  stays bounded by the shape of one run even if a caller loops.
+
+THE WIRE. Studio's V3 producer append path (`POST /internal/v3/producer/attempts/{id}/events`,
+`ProducerEventCommand`) carries `event_type` from a CLOSED six-value enum plus a `payload_digest`.
+`STUDIO_EVENT_TYPE` is therefore part of this vocabulary and not an implementation detail: it is
+the declared, total mapping from a progress name to the contract enum value it travels as, and it
+lands inside `UNSEALED_STUDIO_EVENT_TYPES` only. The two values the hosted producer's sealed bulk
+appends already own are reserved against exactly this vocabulary, so a consumer reading the wire
+can still tell a proof from a progress tick with no new field.
+
+The wire ALSO carries the payload body now, in the additive optional `payload` object Studio added
+beside `payload_digest`. Studio stores it as the UNSEALED adjunct of the log record -- beside the
+sealed record, never inside it, out of every sealed-bytes projection and every attestation -- and
+refuses the append unless the payload's canonical-JSON SHA-256 equals `payload_digest`, so the
+digest still binds exactly the bytes the worker claims. `progress_payload_and_digest` is what the
+emitter calls: it returns the body and its content address from ONE canonicalization, so the two
+cannot disagree at the call site. Sending the body is what makes the fact values below reach a
+timeline instead of dying in the worker; a consumer that reads only `payload_digest` sees exactly
+what it saw before.
+
+A PROBE IS NOT AN ACQUISITION, which is why the pair below is its own two names rather than the
+`source_acquire_*` ones a session worker first borrowed. Those two carry `index` and `total`,
+because a Build acquires source i of a set the recipe fixes and a progress rail can render the
+fraction. A session's probes are a conversation: how many there will be is not knowable when any of
+them runs, so a worker reusing that shape had to send `index == total` and every probe read "1 of
+1", "2 of 2", "3 of 3" -- a fraction that is always whole, which is worse than no fraction because
+a rail will draw it. `source_probe_started` and `source_probe_settled` carry Studio's own `ordinal`
+instead: a counter, monotonic and meaningful on its own, and no denominator that would have to be
+invented.
+
+TERMINAL RECORDS, and why a failure is a COMPLETION here rather than a diagnostic. A watcher on
+this stream pairs a start with an end; a start with no end is indistinguishable from a probe still
+running, forever. So every terminal outcome of a probe emits `source_probe_settled`, and it travels
+as `source_probe_completed` -- the same contract enum value a successful one travels as -- because
+that enum value means "a probe settled" and a refused probe settled. Routing failures to
+`diagnostic_emitted` instead was the tempting reading of the two names, and it is the wrong one: a
+consumer would then have to know two event types to answer "did this probe finish", and one of them
+is the type an attempt-level narration also uses (`attempt_failed`), so counting probe endings would
+have counted attempt diagnostics too. `diagnostic_emitted` stays what it already was -- narration
+ABOUT a failure -- and the outcome lives in a typed fact on the completion, where a watcher reads it
+without parsing a second stream. See `PROBE_OUTCOMES` for the values and what separates them.
+
+THE BRIDGE. `bridge_build_events` watches the sealed feed through `events.observer` and republishes
+a bounded subset of Build records as progress records. It exists because the hosted producer calls
+`execute_recipe` as one opaque step: without it the timeline is silent for the whole build. It is a
+one-way read -- sealed to progress, never the reverse -- with a hand-written translation table
+(`SEALED_BUILD_EVENT_BRIDGE`), a per-event allowlist of the fact keys that cross
+(`BRIDGED_FACT_KEYS`), and a value guard (`_safe_value`) that drops any record carrying a value
+outside integers, booleans and short token strings. The guard is what keeps this side placeless:
+the sealed feed redacts locations in its own writer, downstream of `emit`, so a watcher sees raw
+facts and must do its own refusing.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import re
+from collections.abc import Callable, Iterator, Mapping
+from contextvars import ContextVar
+from typing import Any
+
+from mostlyright.data_harness import events
+from mostlyright.data_harness.canonical import canonical_json_bytes, sha256_bytes
+
+PROGRESS_SCHEMA_VERSION = "mr-progress-events.v1"
+
+# A hard per-attempt ceiling, enforced by the emitter rather than by hope. The vocabulary alone
+# already bounds the stream by the shape of one run -- one record per source, per artifact and per
+# stage -- so reaching this number means a caller is looping, and a looping caller must cost the
+# run nothing rather than cost Studio an unbounded append.
+MAX_PROGRESS_EVENTS_PER_ATTEMPT = 512
+
+# The ordered producer stages a live timeline draws its skeleton from. `stage_started` and
+# `stage_completed` carry an index into this tuple, so a dashboard can render the whole shape of a
+# run from its first event instead of discovering stages as they arrive.
+PRODUCER_STAGES: tuple[str, ...] = ("acquire", "build", "upload", "submit", "handoff")
+
+# The six values Studio's `ProducerEventCommand.event_type` enum admits, spelled here as LITERALS
+# on the `events._RECIPE_MANIFEST_VERSION` precedent: this module imports no generated client, so
+# it stays loadable wherever the vocabulary is needed, and a test asserts the set equals the
+# generated `ContractProducerEventCommandEventType` so the two spellings cannot drift.
+STUDIO_EVENT_TYPES = frozenset(
+    {
+        "attempt_progress",
+        "source_probe_completed",
+        "artifact_upload_completed",
+        "candidate_build_completed",
+        "diagnostic_emitted",
+        "run_stage_reported",
+    }
+)
+
+# Two of those six belong to the hosted producer's SEALED bulk appends and are off limits here.
+# The wire carries no sealed/unsealed flag -- one enum value plus one opaque digest -- so if a
+# progress record travelled as `artifact_upload_completed`, a consumer counting artifacts would see
+# nine completions for eight artifacts and have no way to tell which one was evidence. Reserving
+# these two is what keeps the wire unambiguous without asking Studio for a new field.
+SEALED_STUDIO_EVENT_TYPES = frozenset({"artifact_upload_completed", "candidate_build_completed"})
+UNSEALED_STUDIO_EVENT_TYPES = STUDIO_EVENT_TYPES - SEALED_STUDIO_EVENT_TYPES
+
+# The authoritative declaration of what each progress event asserts, in the shape `events`
+# declares its own: a name maps to its exact fact keys, and a producer that drifts is refused
+# rather than emitted. Facts are numbers and short tokens only -- see `_safe_value`.
+#
+# The `build_*` half is written by the bridge and never by hand; the rest is written by the hosted
+# producer. Both halves are one vocabulary because both are unsealed and both travel the same wire.
+PROGRESS_EVENT_FACTS: Mapping[str, tuple[str, ...]] = {
+    # -- producer lifecycle ----------------------------------------------------------------
+    "attempt_started": ("role", "stage_total"),
+    "stage_started": ("stage", "index", "total"),
+    "stage_completed": ("stage", "index", "total"),
+    "source_acquire_started": ("source_id", "index", "total"),
+    "source_acquire_completed": ("source_id", "index", "total", "bytes"),
+    "source_probe_started": ("source_id", "probe_kind", "ordinal"),
+    "source_probe_settled": (
+        "source_id",
+        "probe_kind",
+        "ordinal",
+        "outcome",
+        "code",
+        "display_code",
+        "retry_after_seconds",
+        "bytes",
+    ),
+    "artifact_upload_started": ("artifact_id", "kind", "index", "total", "bytes"),
+    "artifact_upload_sealed": ("artifact_id", "kind", "index", "total"),
+    "candidate_submitted": ("candidate_digest", "artifact_count"),
+    "verify_handoff": ("candidate_digest", "admission_id"),
+    "attempt_finished": ("outcome",),
+    "attempt_failed": ("code",),
+    # -- bridged from the sealed Build feed ------------------------------------------------
+    "build_sources_read_started": ("total",),
+    "build_source_read": ("source_id", "bytes"),
+    "build_sources_parse_started": ("total",),
+    "build_source_parsed": ("source_id", "rows", "columns"),
+    "build_cleaning_started": ("total",),
+    "build_cleaning_applied": ("step_index", "step_total", "source_id", "operation"),
+    "build_join_completed": ("left_rows", "right_rows", "output_rows"),
+    "build_rows_selected": ("rows", "columns"),
+    "build_checks_started": ("total",),
+    "build_checks_completed": ("passed", "total"),
+    "build_table_written": ("rows", "columns"),
+    "build_candidate_sealed": ("candidate_digest", "table_sha256", "row_count"),
+}
+
+PROGRESS_EVENT_NAMES = frozenset(PROGRESS_EVENT_FACTS)
+
+# How a settled probe ended, as a CLOSED set rather than as whatever token the emitter happened to
+# hold. A watcher branches on this value -- it is the whole reason `source_probe_settled` exists --
+# and a value it has never seen is worse than no record, because the record still terminates the
+# probe on the timeline while saying nothing about how.
+#
+# These four REFINE the two Studio's `ContractProbeResultCommand.outcome` enum admits; they do not
+# contradict them. `refused` and `timed_out` are both `failed` on the wire, because the wire is
+# Studio's enum and widening it would be a contract change. The refinement lives here because this
+# record is unsealed and harness-owned, which is exactly the licence this vocabulary was given:
+#
+# * `succeeded` -- the probe was answered and the answer was reported.
+# * `refused`   -- the probe was refused rather than answered: its request was outside the probe
+#                  vocabulary, or named a source the session may not look at or a column the source
+#                  does not have, or the source channel would not serve it at all. Re-asking the
+#                  same probe refuses identically; the fix is a different probe, or a channel.
+# * `timed_out` -- the probe was attempted and did not finish inside a ceiling. A post-acquisition
+#                  `SANDBOX_TIMEOUT` may yield to a smaller scan; a source-acquisition timeout
+#                  does not, because bytes never arrived.
+# * `throttled` -- an acquisition admission limit, not a source or probe refusal. The worker keeps
+#                  its wire code, emits a display code, and carries Studio's retry window only
+#                  when Studio supplied one; it never guesses a wait. This is unrelated to model
+#                  tokens, billing, authentication, or source credentials.
+# * `failed`    -- attempted, and broke for a reason that is none of the above.
+PROBE_OUTCOMES: tuple[str, ...] = ("succeeded", "refused", "throttled", "failed", "timed_out")
+
+# The `code` fact is a failure code, and a succeeded probe has none. It carries this sentinel rather
+# than being absent, because a fact set is exact here and an optional key would be the one thing
+# `validate_progress_facts` exists to refuse. `OK` is spelled in the code namespace's own case so a
+# reader grepping a timeline for a code sees one value per record and never an empty column.
+PROBE_OUTCOME_OK_CODE = "OK"
+
+# Fact values that are a closed enumeration rather than an open token. `_safe_value` bounds the
+# SHAPE of every fact; this bounds the VALUE of the few that are a decision a consumer branches on.
+# Keyed by (name, fact) so adding one is a two-line edit and forgetting to declare one is not
+# silently permissive -- a fact absent from here is an ordinary token, which is the honest default.
+PROGRESS_ENUM_FACT_VALUES: Mapping[tuple[str, str], tuple[str, ...]] = {
+    ("source_probe_settled", "outcome"): PROBE_OUTCOMES,
+}
+
+# Which contract enum value each progress name travels as. TOTAL over the vocabulary -- a name
+# without an entry cannot be emitted, which is checked at import rather than at the first send, so
+# a new name that nobody mapped fails the test run and never the production attempt.
+STUDIO_EVENT_TYPE: Mapping[str, str] = {
+    "attempt_started": "run_stage_reported",
+    "stage_started": "run_stage_reported",
+    "stage_completed": "run_stage_reported",
+    "source_acquire_started": "attempt_progress",
+    "source_acquire_completed": "source_probe_completed",
+    "source_probe_started": "attempt_progress",
+    "source_probe_settled": "source_probe_completed",
+    "artifact_upload_started": "attempt_progress",
+    "artifact_upload_sealed": "attempt_progress",
+    "candidate_submitted": "run_stage_reported",
+    "verify_handoff": "run_stage_reported",
+    "attempt_finished": "run_stage_reported",
+    "attempt_failed": "diagnostic_emitted",
+    "build_sources_read_started": "run_stage_reported",
+    "build_source_read": "attempt_progress",
+    "build_sources_parse_started": "run_stage_reported",
+    "build_source_parsed": "attempt_progress",
+    "build_cleaning_started": "run_stage_reported",
+    "build_cleaning_applied": "attempt_progress",
+    "build_join_completed": "attempt_progress",
+    "build_rows_selected": "attempt_progress",
+    "build_checks_started": "run_stage_reported",
+    "build_checks_completed": "attempt_progress",
+    "build_table_written": "attempt_progress",
+    "build_candidate_sealed": "attempt_progress",
+}
+
+# Sealed Build name -> progress name. Hand written, and deliberately PARTIAL. What is left out and
+# why:
+#
+# * `member_sealed`, `member_verified` and `check_completed` are per-item over sets whose size the
+#   plan fixes but does not bound tightly; a timeline gains a spinner from them and nothing else.
+# * `run_started`, `rights_checked`, `evidence_matched`, `sources_profiled`,
+#   `members_sealed_started`, `snapshot_verify_started`, `snapshot_verified` and
+#   `candidate_installed` are Build evidence whose value is that they are PROVABLE. A live copy of
+#   a provable record is the weaker of the two and invites a reader to cite the copy.
+# * `build_failed` and `run_interrupted` are terminal Build records. The hosted producer narrates
+#   its own failure through `attempt_failed`, from the error it is about to raise, with a code it
+#   controls -- rather than through a mirrored record whose `message` fact is free prose.
+SEALED_BUILD_EVENT_BRIDGE: Mapping[str, str] = {
+    "sources_read_started": "build_sources_read_started",
+    "source_read": "build_source_read",
+    "sources_parsed_started": "build_sources_parse_started",
+    "source_parsed": "build_source_parsed",
+    "cleaning_started": "build_cleaning_started",
+    "cleaning_applied": "build_cleaning_applied",
+    "join_completed": "build_join_completed",
+    "rows_selected": "build_rows_selected",
+    "checks_started": "build_checks_started",
+    "checks_completed": "build_checks_completed",
+    "table_written": "build_table_written",
+    "build_sealed": "build_candidate_sealed",
+}
+
+# The exact sealed fact keys that cross the bridge, per event. Everything else is dropped, including
+# `evidence_member`: a member path is a Build pointer, and a progress record that carried one would
+# be inviting a reader to resolve it against a candidate that may not exist yet.
+BRIDGED_FACT_KEYS: Mapping[str, tuple[str, ...]] = {
+    "sources_read_started": ("total",),
+    "source_read": ("source_id", "bytes"),
+    "sources_parsed_started": ("total",),
+    "source_parsed": ("source_id", "rows", "columns"),
+    "cleaning_started": ("total",),
+    "cleaning_applied": ("step_index", "step_total", "source_id", "operation"),
+    "join_completed": ("left_rows", "right_rows", "output_rows"),
+    "rows_selected": ("rows", "columns"),
+    "checks_started": ("total",),
+    "checks_completed": ("passed", "total"),
+    "table_written": ("rows", "columns"),
+    "build_sealed": ("candidate_digest", "table_sha256", "row_count"),
+}
+
+# A fact string is a short token: identifiers, digests, enum words and dotted names. It is NOT
+# prose and NOT a path. The class excludes the space, the backslash and every quote, so a sentence,
+# a Windows drive prefix and a rooted POSIX path all fail it, and the record carrying one is dropped
+# whole rather than half redacted.
+_TOKEN_RE = re.compile(r"\A[A-Za-z0-9_.:+-]{1,128}\Z")
+
+_MAX_FACT_INT = 2**53 - 1
+
+
+class ProgressVocabularyError(ValueError):
+    """A progress record does not match its declared name, fact set or value domain."""
+
+
+def _safe_value(value: Any) -> bool:
+    """Return whether one fact value is inside the progress value domain.
+
+    ``bool`` is tested before ``int`` on purpose: in Python a bool IS an int, and a magnitude bound
+    applied to ``True`` reads as an accident rather than as a rule.
+    """
+
+    if isinstance(value, bool):
+        return True
+    if isinstance(value, int):
+        return -_MAX_FACT_INT <= value <= _MAX_FACT_INT
+    if isinstance(value, str):
+        return _TOKEN_RE.match(value) is not None
+    return False
+
+
+def is_progress_fact_value(value: Any) -> bool:
+    """Return whether ``value`` may travel as a progress fact, for a caller building one.
+
+    The public view of `_safe_value`, and the only supported way to ask the question outside this
+    module. An emitter that has a value from somewhere else -- a failure code off an exception, say
+    -- can check it here and substitute a known-good one, rather than handing over a value that
+    makes the whole record get dropped at `emit`.
+    """
+
+    return _safe_value(value)
+
+
+def validate_progress_facts(name: str, facts: Mapping[str, Any]) -> None:
+    """Enforce the declared name, fact set and value domain; raise naming the difference.
+
+    Raises ``ProgressVocabularyError``. Callers on the emission path never let it escape -- the
+    point of raising is that a drifted producer fails a test run, not a hosted attempt.
+    """
+
+    declared = PROGRESS_EVENT_FACTS.get(name)
+    if declared is None:
+        raise ProgressVocabularyError(f"progress name is outside the closed vocabulary: {name!r}")
+    if not isinstance(facts, Mapping):
+        raise ProgressVocabularyError(f"facts for {name!r} is not a mapping")
+    present = set(facts)
+    expected = set(declared)
+    if present != expected:
+        missing = sorted(expected - present)
+        extra = sorted(present - expected)
+        raise ProgressVocabularyError(
+            f"facts for {name!r} do not match the declared set: "
+            f"missing={missing} extra={extra} declared={list(declared)}"
+        )
+    for key, value in facts.items():
+        if not _safe_value(value):
+            raise ProgressVocabularyError(
+                f"fact {key!r} of {name!r} is outside the progress value domain: {type(value)}"
+            )
+        admitted = PROGRESS_ENUM_FACT_VALUES.get((name, key))
+        if admitted is not None and value not in admitted:
+            raise ProgressVocabularyError(
+                f"fact {key!r} of {name!r} is not one of {list(admitted)}: {value!r}"
+            )
+
+
+def progress_payload(name: str, facts: Mapping[str, Any]) -> dict[str, Any]:
+    """Return the canonical payload one progress record commits to.
+
+    ``unsealed`` is a literal field rather than a comment. A payload that ever reaches a store
+    beside sealed evidence says what it is in its own bytes, so a reader that has only the payload
+    still cannot mistake it for Build evidence.
+    """
+
+    validate_progress_facts(name, facts)
+    return {
+        "schema_version": PROGRESS_SCHEMA_VERSION,
+        "event": name,
+        "unsealed": True,
+        "studio_event_type": STUDIO_EVENT_TYPE[name],
+        "facts": {key: facts[key] for key in PROGRESS_EVENT_FACTS[name]},
+    }
+
+
+def progress_payload_and_digest(name: str, facts: Mapping[str, Any]) -> tuple[dict[str, Any], str]:
+    """Return one progress payload and its content address, from a single canonicalization.
+
+    The wire carries BOTH halves -- the body in ``payload`` and the address in ``payload_digest``
+    -- and Studio refuses the append when they disagree. Deriving them separately at a call site is
+    therefore the one mistake worth designing out, so this is the only place that pairs them, and
+    :func:`progress_payload_digest` is the digest-only view of it rather than a second formula.
+    """
+
+    payload = progress_payload(name, facts)
+    return payload, "sha256:" + sha256_bytes(canonical_json_bytes(payload))
+
+
+def progress_payload_digest(name: str, facts: Mapping[str, Any]) -> str:
+    """Return the ``sha256:``-prefixed content address of one progress payload.
+
+    This is the ONLY digest this module produces and it attests one thing: the payload bytes. It is
+    not evidence about a Build, it is not an input to any manifest, and nothing verifies a candidate
+    against it.
+    """
+
+    return progress_payload_and_digest(name, facts)[1]
+
+
+ProgressSink = Callable[[str, Mapping[str, Any]], None]
+
+# A ContextVar for the same reason `events._EVENTS` is one, and a SEPARATE one so that arming or
+# disarming either stream can never touch the other. A thread starts from an empty context, so one
+# hosted attempt can never emit into another's stream.
+_PROGRESS: ContextVar[ProgressSink | None] = ContextVar("mostlyright_run_progress", default=None)
+
+
+def emit(name: str, facts: Mapping[str, Any]) -> None:
+    """Emit one progress record. Returns nothing and, with no sink armed, does nothing.
+
+    On ANY exception from the sink the sink is DISARMED for the rest of the arming context rather
+    than retried, exactly as `events.emit` does: a progress stream missing its tail is a loud
+    failure, and a progress stream that fails a Build is not a tradeoff anyone agreed to.
+    """
+
+    sink = _PROGRESS.get()
+    if sink is None:
+        return None
+    try:
+        sink(name, facts)
+    except Exception:
+        _PROGRESS.set(None)
+    return None
+
+
+@contextlib.contextmanager
+def progress_sink(sink: ProgressSink) -> Iterator[None]:
+    """Arm ``sink`` for the duration of the block and restore the previous one on exit."""
+
+    token = _PROGRESS.set(sink)
+    try:
+        yield
+    finally:
+        _PROGRESS.reset(token)
+
+
+def translate_build_event(
+    event: str, facts: Mapping[str, Any]
+) -> tuple[str, dict[str, Any]] | None:
+    """Return the progress record for one sealed Build record, or ``None`` to drop it.
+
+    PURE. Four reasons to return ``None``, all of them silent by design -- a bridge that raised
+    would surface a sealed-feed detail as a progress-stream failure:
+
+    * the Build name is not in `SEALED_BUILD_EVENT_BRIDGE`;
+    * a fact the translation needs is absent (a sealed producer that drifted);
+    * a fact value is outside the progress value domain, which is the guard that keeps a path or a
+      free-prose message from crossing;
+    * the translated record does not validate, which cannot happen while the tables agree and is
+      checked anyway because the tables are hand written.
+    """
+
+    name = SEALED_BUILD_EVENT_BRIDGE.get(event)
+    if name is None:
+        return None
+    carried = BRIDGED_FACT_KEYS[event]
+    if any(key not in facts for key in carried):
+        return None
+    translated = {key: facts[key] for key in carried}
+    try:
+        validate_progress_facts(name, translated)
+    except ProgressVocabularyError:
+        return None
+    return name, translated
+
+
+@contextlib.contextmanager
+def bridge_build_events() -> Iterator[None]:
+    """Republish the bridged subset of live Build records onto the armed progress sink.
+
+    One way only. This watches `events` and writes to `_PROGRESS`; nothing here can write a record
+    into the Build feed, and `events.observer` calls the previously armed Build sink first, so the
+    feed is unaffected whether or not this is installed.
+    """
+
+    def _watch(
+        event: str, facts: Mapping[str, Any], evidence: Mapping[str, Any] | None = None
+    ) -> None:
+        translated = translate_build_event(event, facts)
+        if translated is not None:
+            emit(*translated)
+
+    with events.observer(_watch):
+        yield
+
+
+def _check_tables() -> None:
+    """Refuse to import a module whose hand-written tables disagree.
+
+    Every one of these is a table an author edits by hand, and every one of them is cheap to check
+    once at import. The alternative -- discovering a missing `STUDIO_EVENT_TYPE` entry when a hosted
+    attempt tries to send it -- turns an editing mistake into a production one.
+    """
+
+    if set(STUDIO_EVENT_TYPE) != PROGRESS_EVENT_NAMES:
+        raise ProgressVocabularyError("STUDIO_EVENT_TYPE is not total over the progress vocabulary")
+    if not UNSEALED_STUDIO_EVENT_TYPES.issuperset(STUDIO_EVENT_TYPE.values()):
+        raise ProgressVocabularyError(
+            "a progress name maps outside the unsealed half of the Studio event-type enum"
+        )
+    for enum_name, enum_fact in PROGRESS_ENUM_FACT_VALUES:
+        if enum_fact not in PROGRESS_EVENT_FACTS.get(enum_name, ()):
+            raise ProgressVocabularyError(
+                f"PROGRESS_ENUM_FACT_VALUES names {enum_fact!r} of {enum_name!r}, "
+                "which is not a declared fact of a declared name"
+            )
+    for admitted in PROGRESS_ENUM_FACT_VALUES.values():
+        for value in admitted:
+            if not _safe_value(value):
+                raise ProgressVocabularyError(
+                    f"an admitted enum value is outside the progress value domain: {value!r}"
+                )
+    if set(SEALED_BUILD_EVENT_BRIDGE) != set(BRIDGED_FACT_KEYS):
+        raise ProgressVocabularyError("the bridge and its fact allowlist cover different events")
+    for event, name in SEALED_BUILD_EVENT_BRIDGE.items():
+        if set(BRIDGED_FACT_KEYS[event]) != set(PROGRESS_EVENT_FACTS[name]):
+            raise ProgressVocabularyError(f"bridged fact keys for {event!r} are not {name!r}'s set")
+
+
+_check_tables()

mostlyright/data_harness/readers/__init__.py

@@ -0,0 +1,46 @@
+"""The Toolbox: the closed set of certified Readers this product ships with.
+
+A Reader is the code that turns fetched bytes into rows.  ``docs/VOCABULARY.md`` states
+what this package is in one sentence -- "the Toolbox, the set of certified Readers this
+product ships with.  A table in the software; nothing is uploaded to it" -- and that
+sentence is the whole design.  The table is closed: membership is exactly the family
+instances written into ``registry.py``, resolution is on an exact ``(family_id,
+family_version)`` key with no fallback to a nearby version, and there is no plugin
+discovery, no import-by-name, and no environment override.  Widening the table is a source
+edit that goes through review, and two versions of one family ship side by side so that a
+sealed recipe keeps resolving its own pin forever.
+
+Import discipline
+-----------------
+This package may import ``formats``, ``canonical``, the standard library, and a family's
+own pinned decoder library.  It must not import ``acquisition``, ``sources``, ``recipe``,
+or ``pipeline``.  Those layers import this one -- the sandbox worker resolves a Reader
+family from this table -- and a back-edge would close an import cycle whose failure is
+order-dependent, which is worse than a hard error: a test run that imported ``readers``
+first would fail while the application, which imports the other way round, stayed green.
+
+The rule that keeps the discipline enforceable is that anything both ``acquisition`` and
+``readers`` need lives in ``formats``, which is a true leaf and says so in its own
+docstring.  Never satisfy a shared need with an import from ``acquisition``.  A test walks
+this package and fails on any import of a layer above it.
+"""
+
+from mostlyright.data_harness.readers.contracts import (
+    DECODE_FLAGS,
+    READER_ERROR_CODES,
+    ReaderBudgets,
+    ReaderError,
+    ReaderFamily,
+    ReaderPin,
+    ReaderResult,
+)
+
+__all__ = [
+    "DECODE_FLAGS",
+    "READER_ERROR_CODES",
+    "ReaderBudgets",
+    "ReaderError",
+    "ReaderFamily",
+    "ReaderPin",
+    "ReaderResult",
+]