mostlyright-data 0.9.7__tar.gz → 0.9.9__tar.gz

{mostlyright_data-0.9.7 → mostlyright_data-0.9.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.9.7
+Version: 0.9.9
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
@@ -126,7 +126,10 @@ The local execution lane runs only on Linux:
 python -m pip install "mostlyright-data[local]"
 ```
 
-It builds and checks worker inputs. It is not required for the hosted client.
+It is not required for the hosted client, and it is off by default: `mr-data` sends that work to
+the backend. What still runs on this computer is catalog authoring, the search index, and the
+commands that check a document or a run folder. A command with no backend lane refuses by name and
+says why.
 
 ## Documentation
 
@@ -138,11 +141,12 @@ It builds and checks worker inputs. It is not required for the hosted client.
 Use `mr-data --help` for the full command list and options. Commands that support
 `--json` write one JSON object.
 
-For local builds, `--input-root` and `--output` may use relative paths.
+`--input-root` and `--output` may use relative paths.
 
 ## Behavior and limits
 
-- `mr-data review --local RUN_DIR` exits 1 when a check fails.
+- `mr-data review RUN_DIR --local` runs the deterministic checks and writes no review decision. It exits 1 when a check fails. Write the flag after the command name; before it, `--local` is the lane selector.
+- `mr-data review RUN_DIR --report REPORT` seals a review decision from signed reviewer reports.
 - The weather reader accepts one extracted GRIB2 record. It refuses a multi-record GFS or HRRR file.
 - Hosted credential-free acquisition supports one single-address public HTTPS object. Bounded collections require both local attestations.
 

{mostlyright_data-0.9.7 → mostlyright_data-0.9.9}/README.md

@@ -45,7 +45,10 @@ The local execution lane runs only on Linux:
 python -m pip install "mostlyright-data[local]"
 ```
 
-It builds and checks worker inputs. It is not required for the hosted client.
+It is not required for the hosted client, and it is off by default: `mr-data` sends that work to
+the backend. What still runs on this computer is catalog authoring, the search index, and the
+commands that check a document or a run folder. A command with no backend lane refuses by name and
+says why.
 
 ## Documentation
 
@@ -57,11 +60,12 @@ It builds and checks worker inputs. It is not required for the hosted client.
 Use `mr-data --help` for the full command list and options. Commands that support
 `--json` write one JSON object.
 
-For local builds, `--input-root` and `--output` may use relative paths.
+`--input-root` and `--output` may use relative paths.
 
 ## Behavior and limits
 
-- `mr-data review --local RUN_DIR` exits 1 when a check fails.
+- `mr-data review RUN_DIR --local` runs the deterministic checks and writes no review decision. It exits 1 when a check fails. Write the flag after the command name; before it, `--local` is the lane selector.
+- `mr-data review RUN_DIR --report REPORT` seals a review decision from signed reviewer reports.
 - The weather reader accepts one extracted GRIB2 record. It refuses a multi-record GFS or HRRR file.
 - Hosted credential-free acquisition supports one single-address public HTTPS object. Bounded collections require both local attestations.
 

{mostlyright_data-0.9.7 → mostlyright_data-0.9.9}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.9.7"
+version = "0.9.9"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"
@@ -54,8 +54,10 @@ mostlyright-studio-client-v3 = { path = "vendor/wheels/mostlyright_studio_client
 [project.scripts]
 # ⚠ NOT `cli:main`. `cli` imports `hosted_worker`, which imports pyarrow and cryptography at module
 # scope, so `cli` cannot even be IMPORTED on a thin install. The router is the one module that can
-# run on a bare interpreter and decide which lane this invocation belongs to; with the `local`
-# extra present it hands every ordinary invocation straight to `cli.main` unchanged.
+# run on a bare interpreter and decide which lane this invocation belongs to. The local product
+# lanes are off by default, so with the `local` extra present it hands the names that still run on
+# this computer to `cli.main` unchanged, and every other name is answered by the hosted lane or
+# told, by name, that this computer no longer runs it.
 mr-data = "mostlyright.data_harness.thin.router:main"
 # ⚠ AND NOT THE MODULES THEMSELVES, for the same reason. Each of these seven is defined in a
 # module that imports pyarrow or cryptography at module scope -- or, for the catalogue job, reaches

{mostlyright_data-0.9.7 → mostlyright_data-0.9.9}/src/mostlyright/data_harness/cli.py

@@ -2260,7 +2260,29 @@ def _require_verify_url(value: str) -> str:
 
 
 def _agent_workflow_surface() -> dict[str, Any]:
-    """Return the versioned discovery surface for external data agents."""
+    """Return the versioned discovery surface for external data agents.
+
+    ``implementation_status`` says where a stage runs, and the three values partition the stages
+    rather than describing them: exactly one is true of any stage, and which one is decided by the
+    surface beside it rather than by how the stage feels.
+
+    * ``available`` -- every command the surface names runs from this command line, with no
+      backend: the kept lane-``local`` names and the portable ones.
+    * ``hosted_only`` -- the surface names only hosted-lane commands. The stage still happens, and
+      the named commands are how it is asked for and reported on; what is not true of it is that
+      this computer performs it. A hosted-lane command is not a skipped stage.
+    * ``backend_internal`` -- a backend operator step: it has a name, and nobody runs it from here.
+
+    ⚠ THE STATUS IS THE PROMISE, SO IT IS NOT DECORATION. An agent chooses its route from this
+    payload, and a stage that says ``available`` about a command which exits 2 sends the agent to
+    a refusal it will read as a defect and retry. The word has to be the answer the command would
+    actually give.
+
+    ⚠ AND A SURFACE NAMED TWICE IS CLASSIFIED ONCE. ``view`` appears under two stages; both say
+    ``hosted_only``, because the same command cannot run here in one row and on the backend in the
+    next. ``tests/test_agent_workflow_cli.py`` derives both halves of the partition from the live
+    lane table, so a command that changes lane fails there rather than drifting.
+    """
 
     return {
         "schema_version": "mr-data-agent-workflow.v4",
@@ -2269,9 +2291,9 @@ def _agent_workflow_surface() -> dict[str, Any]:
         "stages": [
             {
                 "stage": "viewer_observability",
-                "implementation_status": "available",
+                "implementation_status": "hosted_only",
                 "judgment_owner": "external_agent",
-                "deterministic_surface": "mr-data view",
+                "deterministic_surface": "mr-data watch/view",
             },
             {
                 "stage": "question_and_requirements",
@@ -2299,15 +2321,15 @@ def _agent_workflow_surface() -> dict[str, Any]:
             },
             {
                 "stage": "recipe_approval",
-                "implementation_status": "available",
+                "implementation_status": "hosted_only",
                 "judgment_owner": "human",
                 "deterministic_surface": "mr-data recipe-approve",
             },
             {
                 "stage": "acquire_clean_join_validate_package",
-                "implementation_status": "available",
+                "implementation_status": "hosted_only",
                 "judgment_owner": "deterministic_worker",
-                "deterministic_surface": "mr-data recipe-run/recipe-verify",
+                "deterministic_surface": "mr-data run/status",
             },
             {
                 "stage": "independent_review",
@@ -2317,19 +2339,19 @@ def _agent_workflow_surface() -> dict[str, Any]:
             },
             {
                 "stage": "independent_verification",
-                "implementation_status": "available",
+                "implementation_status": "hosted_only",
                 "judgment_owner": "deterministic_verifier",
                 "deterministic_surface": "mr-data verify/status/inspect",
             },
             {
                 "stage": "handoff",
-                "implementation_status": "available",
+                "implementation_status": "hosted_only",
                 "judgment_owner": "external_agent",
                 "deterministic_surface": "mr-data notebook/view",
             },
             {
                 "stage": "hosted_export",
-                "implementation_status": "available",
+                "implementation_status": "backend_internal",
                 "judgment_owner": "external_agent",
                 "deterministic_surface": "mr-data export-hosted-candidate",
             },

{mostlyright_data-0.9.7 → mostlyright_data-0.9.9}/src/mostlyright/data_harness/thin/download.py

@@ -70,6 +70,11 @@ class DownloadedArtifact:
     media_type: str
     size_bytes: int
     content_digest: str
+    #: False when the bytes were already on disk, carried the digest the signed session declared,
+    #: and were adopted rather than fetched again. Reported so that "it was already here" can
+    #: never be read as "it was fetched and checked just now" -- the same reason ``verify`` prints
+    #: ``checked_by`` and ``acquire`` prints ``reader_binding``.
+    transferred: bool = True
 
     def to_receipt(self) -> dict[str, Any]:
         return {
@@ -78,6 +83,7 @@ class DownloadedArtifact:
             "media_type": self.media_type,
             "size_bytes": self.size_bytes,
             "content_digest": self.content_digest,
+            "transferred": self.transferred,
         }
 
 
@@ -103,6 +109,48 @@ def _required_headers(session: Mapping[str, Any]) -> dict[str, str]:
     return headers
 
 
+def _adopted(
+    destination: Path,
+    *,
+    expected_digest: Any,
+    expected_preimage: ExpectedPreimage | None,
+) -> tuple[bool, str, int]:
+    """Whether the file already on disk IS this artifact, by the digest the session declared.
+
+    ⚠ A DIGEST SKIP, NOT AN OVERWRITE. `deploy-dataset` fetches one artifact per kind in a loop,
+    and every one was created ``O_EXCL``, so a transfer interrupted after the first of two left a
+    folder holding one complete, verified artifact and no way to finish the job except a fresh
+    folder that re-downloaded the first as well. On a released dataset of several gigabytes the
+    cost of a dropped connection was paid twice (#438).
+
+    The skip is decided by the SAME fact the download would be verified against, so adopting
+    cannot be weaker than fetching: the bytes on disk must hash to the digest the signed session
+    declares, in whichever of the two authorities this download uses. Anything present that does
+    not match stays ``THIN_DESTINATION_EXISTS``: that file is not this artifact, and replacing it
+    silently is exactly what that refusal exists to prevent.
+    """
+
+    size = destination.stat().st_size
+    if size > MAX_ARTIFACT_BYTES:
+        return False, "", size
+    raw = bytearray()
+    observed = hashlib.sha256()
+    with destination.open("rb") as handle:
+        while chunk := handle.read(DOWNLOAD_CHUNK_BYTES):
+            observed.update(chunk)
+            if expected_preimage is not None:
+                raw.extend(chunk)
+    digest = observed.hexdigest()
+    if expected_preimage is None:
+        return f"sha256:{digest}" == expected_digest, f"sha256:{digest}", size
+    # The bundle seals its own preimage, so the authority is the excised hash, not the raw one.
+    body = bytes(raw)
+    if body.count(expected_preimage.excised_field) != 1:
+        return False, "", size
+    preimage = hashlib.sha256(body.replace(expected_preimage.excised_field, b"", 1)).hexdigest()
+    return preimage == expected_preimage.digest, f"sha256:{digest}", size
+
+
 def download_signed_artifact(
     session: Mapping[str, Any],
     destination: Path,
@@ -153,6 +201,30 @@ def download_signed_artifact(
 
     selected = transport or UrlLibTransport()
     destination.parent.mkdir(parents=True, exist_ok=True)
+    # ⚠ A SYMLINK IS NEVER ADOPTED. The write path opens with ``O_NOFOLLOW`` precisely so a
+    # symlinked destination is refused rather than followed, and adopting through one would walk
+    # around that. Only a regular file in the place the artifact belongs can stand in for it; a
+    # symlink falls through to the open below and is refused there, as before.
+    #
+    # The remaining race is benign: a file swapped between this check and the read would have to
+    # hash to the sealed digest to be adopted, and a file that does is the artifact.
+    if destination.is_file() and not destination.is_symlink():
+        matched, observed_digest, size = _adopted(
+            destination, expected_digest=expected_digest, expected_preimage=expected_preimage
+        )
+        if matched:
+            media_type = session.get("media_type")
+            artifact_id = session.get("artifact_id")
+            return DownloadedArtifact(
+                path=destination,
+                artifact_id=artifact_id if isinstance(artifact_id, str) else "",
+                media_type=(
+                    media_type if isinstance(media_type, str) else "application/octet-stream"
+                ),
+                size_bytes=size,
+                content_digest=(expected_digest if expected_preimage is None else observed_digest),
+                transferred=False,
+            )
     flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL
     if hasattr(os, "O_NOFOLLOW"):
         flags |= os.O_NOFOLLOW
@@ -165,7 +237,8 @@ def download_signed_artifact(
     except FileExistsError as error:
         raise ThinLaneError(
             "THIN_DESTINATION_EXISTS",
-            f"{destination} already exists; a hosted download never overwrites a local file",
+            f"{destination} already exists and does not carry this artifact's sealed digest, so "
+            "it is a different file; a hosted download never overwrites a local file",
         ) from error
 
     try:

{mostlyright_data-0.9.7 → mostlyright_data-0.9.9}/src/mostlyright/data_harness/thin/propose.py

@@ -2099,6 +2099,7 @@ def propose(
     recipe = bundle.recipe
     assert recipe is not None
     _check_recipe_sources(recipe, source_ids=source_ids, configurations=configurations)
+    _check_recipe_query_templates(recipe, configurations=configurations)
     research_run_id: str | None = None
     if research_session_id is not None:
         record = _lookup_session(
@@ -2359,6 +2360,149 @@ def _refuse_a_registered_source_the_bundle_no_longer_names(
         )
 
 
+#: The Reader families a thin install can canonicalize in process. The other five certified
+#: coordinates -- ``spreadsheet.xlsx`` (two), ``parquet.tabular``, ``weather.grib2`` (two) --
+#: import openpyxl, pyarrow or mostlyright-grib at module scope, which the thin profile does not
+#: install. ``propose`` therefore refuses such a pin by name rather than sealing one it cannot
+#: check; see :func:`_admitted_reader_pin`.
+_READER_CANONICALIZATION_UNAVAILABLE = (
+    "this Reader family needs a native wheel the hosted profile does not install, so propose "
+    "cannot admit its options here; author the Recipe from an install that has it "
+    '(pip install "mostlyright-data[local]") so the sealed pin is the admitted one'
+)
+
+
+def _admitted_reader_pin(pin: Any, *, name: str, path: str) -> Any:
+    """One Reader pin in the form the CRAWLER will recompute, or a refusal naming why not.
+
+    ⚠ WHY AUTHORING HAS TO DO THIS. A pin is digested, and the digest is over the ADMITTED options
+    -- ``family.validate_options`` with every family-defaulted key filled in. An author who writes
+    ``{"delimiter": ","}`` and omits ``encoding``, ``header``, ``expand`` or ``required`` seals a
+    document that digests differently from the one the crawler will reconstruct, and every crawl
+    then fails ``reader_identity`` with nothing naming the cause. That is what killed r13 and r14
+    (#411). Every gate accepted those Recipes -- propose, Studio admission, approval -- and only
+    the crawl refused, unnamed.
+
+    The principle the issue states is the one applied here: an agent authors against ``propose``,
+    so anything Studio or the crawler will refuse at run time must be refused HERE, by name,
+    before a person is asked to approve it.
+
+    ⚠ THE THIN PROFILE CAN DO THIS FOR MOST FAMILIES, AND SAYS SO WHEN IT CANNOT. Importing
+    ``readers.registry`` is stdlib-only, and the registry loads a family lazily, so the ten
+    delimited/JSON/archive coordinates -- ``delimited_text`` among them, which is the family the
+    failing Recipes used -- canonicalize on a bare interpreter. The five that need a native wheel
+    are refused rather than passed through unchecked.
+    """
+
+    if pin is None:
+        return None
+    if not isinstance(pin, dict):
+        raise _invalid(name, path, "must be null or a Reader pin object")
+    _keys(pin, name, path, ("family_id", "family_version", "decode_options"))
+    family_id = pin["family_id"]
+    family_version = pin["family_version"]
+    options = pin["decode_options"]
+    if not isinstance(family_id, str) or not isinstance(family_version, str):
+        raise _invalid(name, path, "family_id and family_version must be strings")
+    if not isinstance(options, dict):
+        raise _invalid(name, f"{path}.decode_options", "must be an object")
+
+    # Imported here rather than at module scope: `propose` is reached on a bare interpreter, and
+    # the registry is only needed when a Recipe actually pins a Reader.
+    from mostlyright.data_harness.readers.registry import TOOLBOX
+
+    try:
+        family = TOOLBOX.resolve(family_id, family_version)
+    except ImportError as error:  # the family's native wheel is absent on this install
+        raise ProposeError(
+            "THIN_PROPOSE_READER_PIN",
+            f"{name} {path} pins {family_id}@{family_version}: "
+            f"{_READER_CANONICALIZATION_UNAVAILABLE} ({error})",
+        ) from error
+    except Exception as error:  # unknown family, unknown version, or a dependency refusal
+        raise ProposeError(
+            "THIN_PROPOSE_READER_PIN",
+            f"{name} {path} pins {family_id}@{family_version}, which is not a certified Reader "
+            f"coordinate this Harness can admit: {error}",
+        ) from error
+    try:
+        admitted = dict(family.validate_options(options))
+    except Exception as error:
+        raise ProposeError(
+            "THIN_PROPOSE_READER_PIN",
+            f"{name} {path} pins Reader options {family_id}@{family_version} refuses: {error}",
+        ) from error
+    return {
+        "family_id": family_id,
+        "family_version": family_version,
+        "decode_options": admitted,
+    }
+
+
+def _reader_pin_of(query: Any) -> Any:
+    return query.get("reader_pin") if isinstance(query, dict) else None
+
+
+def _check_recipe_query_templates(
+    recipe: Mapping[str, Any],
+    *,
+    configurations: Mapping[str, Mapping[str, Any]],
+) -> None:
+    """Every recipe query is the query its registered connector holds, in the admitted form.
+
+    ⚠ TWO REFUSALS, BECAUSE THEY ARE TWO MISTAKES. Studio binds ``query_template`` byte-for-byte
+    to the registered connector configuration, so a Recipe whose query has drifted from its source
+    is refused at run time with nothing naming the field. And a raw (non-admitted) Reader pin
+    digests differently from the one the crawler rebuilds, failing ``reader_identity`` on every
+    crawl. Both were accepted by every authoring gate before this (#411); both are named here,
+    with the exact value to write.
+
+    Sources whose adapter carries no connector query -- ``user.file`` snapshots -- are skipped:
+    there is nothing registered to compare against.
+    """
+
+    for source in recipe["sources"]:
+        source_id = source["source_id"]
+        template = source.get("query_template")
+        registered = configurations.get(source_id, {}).get("query")
+        if template is None or registered is None:
+            continue
+        if not isinstance(template, dict):
+            raise _invalid("recipe.json", f"sources[{source_id}].query_template", "must be object")
+
+        admitted_pin = _admitted_reader_pin(
+            _reader_pin_of(template),
+            name="recipe.json",
+            path=f"sources[{source_id}].query_template.reader_pin",
+        )
+        if admitted_pin != _reader_pin_of(template):
+            raise ProposeError(
+                "THIN_PROPOSE_READER_PIN",
+                f"recipe.json source {source_id} pins Reader options that are not the admitted "
+                "form, so its sealed digest is not the one every crawl will recompute and "
+                "reader_identity would fail on each run; write the pin exactly as: "
+                f"{json.dumps(admitted_pin, sort_keys=True)}",
+            )
+
+        differing = sorted(
+            key
+            for key in set(template) | set(registered)
+            if canonical_json_bytes(template.get(key)) != canonical_json_bytes(registered.get(key))
+        )
+        if differing:
+            detail = "; ".join(
+                f"{key}: recipe.json has {json.dumps(template.get(key), sort_keys=True)}, "
+                f"the registered connector has {json.dumps(registered.get(key), sort_keys=True)}"
+                for key in differing
+            )
+            raise ProposeError(
+                "THIN_PROPOSE_SOURCE_QUERY",
+                f"recipe.json source {source_id} carries a query its registered connector "
+                f"configuration does not hold, and Studio binds the two byte-for-byte, so this "
+                f"Recipe would be refused when it ran -- {detail}",
+            )
+
+
 def _check_recipe_sources(
     recipe: Mapping[str, Any],
     *,
@@ -2560,8 +2704,8 @@ def _proposal_receipt(
     and ``mr-data build --hosted`` queues one; it is approved and the confirmation STARTED the
     Build in the same act -- Studio #124 -- and the run is named here because ``mr-data build``
     would refuse it rather than replay it; or it was declined (exit 2). Both commands carry
-    ``--hosted``, because on a full install the bare names are the local lane's. Nothing here
-    decides anything.
+    ``--hosted`` because it names the lane explicitly and remains valid: the bare names reach the
+    backend on every install. Nothing here decides anything.
     """
 
     payload = _common(client, journal, stage=stage)