mostlyright-data 0.10.1__tar.gz → 0.10.2__tar.gz

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.10.1
+Version: 0.10.2
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.10.1"
+version = "0.10.2"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/skills/mr-data-build/SKILL.md

@@ -92,7 +92,7 @@ here, then the ones that do not, grouped by why.
 
 ## Commands
 
-These twenty-one are the whole surface a hosted install runs. Every other name in
+These twenty-two are the whole surface a hosted install runs. Every other name in
 [Hosted parity](../../docs/HOSTED-PARITY.md) refuses here.
 
 | Command | What it does |
@@ -102,9 +102,10 @@ These twenty-one are the whole surface a hosted install runs. Every other name i
 | `mr-data whoami` | Use the compatibility alias for the remotely validated `auth status` answer. |
 | `mr-data probe` | Ask one registered source one question and print the answer. **Currently unusable:** it reaches `/v3/sessions`, which Studio refuses, and it takes a question identifier that no v4 receipt returns. Do not plan a source inspection around it. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. `mr-data recipe show ID` reads one back. |
-| `mr-data run` | Start one run against a registered recipe: `--sample`, `--full`, `--refresh` or `--backfill`, under `--max-rows`, `--max-source-bytes` and `--window`. A large run is held for a spend confirmation, which `--confirm` settles; `--cancel RUN` stops one. |
-| `mr-data status` | Report which of the six states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed and where. Exits non-zero on a failed run. |
-| `mr-data runs` | Report this workspace's own runs: identifier, mode, state and creation time, plus the failure triple of any that failed. `--status` and `--mode` narrow it and are refused for a word outside the six or the four; `--limit` says how many, and pages are followed to reach it. |
+| `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
+| `mr-data run` | Start one run against a registered recipe: `--sample`, `--full`, `--refresh` or `--backfill`, under `--max-rows`, `--max-source-bytes` and `--window`. A large run is held for a spend confirmation, which `--confirm` settles; a sample-first held full requires Studio's interactive human approval instead. `--cancel RUN` stops one. |
+| `mr-data status` | Report which of the seven states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed and where. Exits non-zero on a failed run. |
+| `mr-data runs` | Report this workspace's own runs: identifier, mode, state and creation time, plus the failure triple of any that failed. `--status` and `--mode` narrow it and are refused for a word outside the seven or the four; `--limit` says how many, and pages are followed to reach it. |
 | `mr-data watch` | Stream one run's live progress under the durable event type of each stage, resuming across stream cuts. Exits non-zero when the run failed. |
 | `mr-data query` | Run one bounded read-only SQL statement over a finished run's own table and print the rows. Bounded on rows, on answer bytes and on time, and refused by name when a bound is crossed. |
 | `mr-data peek` | Print the columns, types and first rows of one run's table. |
@@ -119,11 +120,10 @@ These twenty-one are the whole surface a hosted install runs. Every other name i
 | `mr-data unpin` | Resume tracking the latest passing version. |
 | `mr-data demote` | Detach the schedule and withdraw the live pointer. |
 
-`auth`, `login` and `whoami` are the same implementation in both profiles. The other eighteen
-answer from Studio, and ten of them only read: `status`, `runs`, `watch`, `peek`, `receipt`,
-`checks`, `download`, `diff`, and — after the state they report — `unpin` and `demote` do not.
-Eight of the eighteen change something: `recipe`, `run`, `query`, `probe`, `keys`, `promote`,
-`pin`, `unpin` and `demote`.
+`auth`, `login` and `whoami` are the same implementation in both profiles. The other nineteen
+answer from Studio. Eight only read: `status`, `runs`, `watch`, `peek`, `receipt`, `checks`,
+`download` and `diff`. Eleven start or change something: `recipe`, `cover`, `run`, `query`,
+`probe`, `keys`, `open`, `promote`, `pin`, `unpin` and `demote`.
 
 **Any successful run is readable straight away.** There is no approval, no admission and no release
 step between a run finishing and reading it: `download`, `receipt`, `checks`, `peek` and `query`
@@ -528,12 +528,15 @@ proved, and a run that emitted none produces the same build as a run that emitte
 Never turn a progress count into a percentage, never infer activity from elapsed time, and never
 cite a progress event where a receipt is what is being asked for.
 
-**Six states, one machine.** A run is `queued`, `running`, `awaiting_confirmation`, `succeeded`,
-`failed` or `cancelled`. Three of them are terminal and end a watch: `succeeded`, `failed` and
-`cancelled`. `awaiting_confirmation` is the spend gate and the one state that is waiting on a
-person: nothing has been executed, and `mr-data run --confirm RUN` releases it. Keep the raw state
-internal. Say what product decision is needed and give the person the single action that lets the
-build continue.
+**Seven states, one machine.** A run is `queued`, `running`, `awaiting_confirmation`,
+`awaiting_sample_approval`, `succeeded`, `failed` or `cancelled`. Three of them are terminal and
+end a watch: `succeeded`, `failed` and `cancelled`. `awaiting_confirmation` is the ordinary spend
+gate: nothing has been executed, and `mr-data run --confirm RUN` releases it.
+`awaiting_sample_approval` is different: a bounded sample must seal a downloadable
+`table_parquet`, then a person completes Studio's fresh operation-bound interactive approval before
+the linked full run dispatches. Do not present `--confirm` as an alternative for that state. Keep
+the raw state internal. Say what product decision is needed and give the person the single action
+that lets the build continue.
 
 **A failed run always carries the same three members** — `failure_code`, `failure_detail` and
 `failed_stage` — and carries them on three surfaces: the run record, the terminal event on the

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/src/mostlyright/data_harness/acquisition/sandbox.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import base64
 import contextlib
+import errno
 import fcntl
 import ipaddress
 import json
@@ -74,7 +75,6 @@ from mostlyright.data_harness.formats import (
 )
 from mostlyright.data_harness.linux_process_boundary import (
     _close_landlock_runtime_roots,
-    _disable_process_dumpability,
     _install_landlock_filesystem_confinement,
     _install_landlock_filesystem_confinement_from_coordinates,
     _install_networkless_seccomp,
@@ -566,38 +566,18 @@ class CrawlerSandbox:
         if external_network_policy_attestation is not None:
             _require_digest(external_network_policy_attestation, "external network policy")
         self._external_network_policy_attestation = external_network_policy_attestation
-        if memory_cgroup_root_fd is not None and hosted_session_container_memory_bytes is not None:
+        if child_address_space_bytes is None:
+            child_address_space_bytes = READER_WORKER_MEMORY_BYTES
+        if (
+            type(child_address_space_bytes) is not int
+            or not MIN_CHILD_ADDRESS_SPACE_BYTES
+            <= child_address_space_bytes
+            <= MAX_CHILD_ADDRESS_SPACE_BYTES
+        ):
             raise AcquisitionSecurityError(
                 "SANDBOX_MEMORY_BOUNDARY",
-                "delegated and hosted-session memory boundaries are mutually exclusive",
+                "the child address-space ceiling is outside the reviewed range",
             )
-        if memory_cgroup_root_fd is not None:
-            _require_bounded_cgroup_root(memory_cgroup_root_fd)
-        if hosted_session_container_memory_bytes is not None:
-            try:
-                _disable_process_dumpability()
-            except OSError as error:
-                raise AcquisitionSecurityError(
-                    "SANDBOX_OS_BOUNDARY",
-                    "hosted-session coordinator process hardening is unavailable",
-                ) from error
-            _require_hosted_session_container_boundary(hosted_session_container_memory_bytes)
-        if child_address_space_bytes is not None:
-            if memory_cgroup_root_fd is not None:
-                raise AcquisitionSecurityError(
-                    "SANDBOX_MEMORY_BOUNDARY",
-                    "delegated and address-space memory boundaries are mutually exclusive",
-                )
-            if (
-                type(child_address_space_bytes) is not int
-                or not MIN_CHILD_ADDRESS_SPACE_BYTES
-                <= child_address_space_bytes
-                <= MAX_CHILD_ADDRESS_SPACE_BYTES
-            ):
-                raise AcquisitionSecurityError(
-                    "SANDBOX_MEMORY_BOUNDARY",
-                    "the child address-space ceiling is outside the reviewed range",
-                )
         self._memory_cgroup_root_fd = memory_cgroup_root_fd
         self._hosted_session_container_memory_bytes = hosted_session_container_memory_bytes
         self._child_address_space_bytes = child_address_space_bytes
@@ -1163,18 +1143,6 @@ class CrawlerSandbox:
         # ceiling says a decode cannot take the coordinator down with it. Until an address-space
         # bound existed there was no way to state the second without a delegated cgroup, which is
         # why this arm could not be written before.
-        if (
-            self._external_network_policy_attestation is None
-            and self._hosted_session_container_memory_bytes is None
-            and self._child_address_space_bytes is None
-        ):
-            raise AcquisitionSecurityError(
-                "SANDBOX_OS_BOUNDARY",
-                (
-                    "crawler execution requires an explicit externally enforced "
-                    "deny-default network policy attestation"
-                ),
-            )
         return _WorkerLaunch(
             command=tuple(command),
             cwd=self._staging_root,
@@ -1185,18 +1153,20 @@ class CrawlerSandbox:
     def _operation_refusal(self, operation: str) -> AcquisitionSecurityError | None:
         """The refusals that depend on WHICH operation this confinement is asked for.
 
-        Both stand on the same fact and neither is a policy preference: an operation that runs a
-        certified Reader needs a per-child memory ceiling, and there are two confinements that
-        cannot give it one.  Under the hosted-session container boundary ``_prepare_worker`` builds
-        no cgroup job at all, so the only ceiling is the container the coordinator itself lives in
-        and a decoder that overran it would take the coordinator and the run's own reporting with
-        it.  Off Linux there is no cgroup to build a job from in the first place.
+        Linux is the sole clean-room implementation. It is where a child gets both its
+        ``RLIMIT_AS`` ceiling and seccomp boundary; every other launcher would be an ordinary
+        subprocess, regardless of whether a caller supplied an egress attestation.
 
         Returned rather than raised so the same decision can be REPORTED without being taken --
         see :meth:`confinement_refusal`.  A caller that has to know whether a decode is admissible
         before it spends a fetch cannot find out by attempting one.
         """
 
+        if sys.platform != "linux":
+            return AcquisitionSecurityError(
+                "SANDBOX_OS_BOUNDARY",
+                "clean-room execution requires the Linux RLIMIT_AS and seccomp boundaries",
+            )
         # THE ADDRESS-SPACE CEILING ADMITS A DECODE; THE HOSTED-SESSION CONTAINER DOES NOT, and
         # the difference is which process dies. Under the container bound the child has no ceiling
         # of its own, so a runaway decode takes the coordinator with it and the run reports
@@ -1205,24 +1175,11 @@ class CrawlerSandbox:
         # reason a per-child bound is required for a Reader and a container limit is not enough.
         if (
             self._hosted_session_container_memory_bytes is not None
-            and self._child_address_space_bytes is None
-            and operation not in HOSTED_SESSION_CONTAINER_OPERATIONS
+            and operation in NETWORK_PERFORMING_OPERATIONS
         ):
             return AcquisitionSecurityError(
                 "SANDBOX_OS_BOUNDARY",
-                "the hosted-session container gives a child no memory ceiling of its own, so this "
-                "confinement admits no-network parsing and nothing else",
-            )
-        # Was darwin-only, and is now every host that is not Linux. The refusal never really
-        # was about macOS: it was about there being no cgroup to bound a Reader's memory with,
-        # which is true of every host without cgroups v2. Seatbelt used to make macOS the one
-        # non-Linux host that got this far with a boundary of its own; with it gone (ADR 0021
-        # ADR 0021) there is no reason to keep naming one platform. The typed code is unchanged,
-        # because ``SANDBOX_MEMORY_BOUNDARY`` is what callers and receipts already read.
-        if operation in READER_PERFORMING_OPERATIONS and sys.platform != "linux":
-            return AcquisitionSecurityError(
-                "SANDBOX_MEMORY_BOUNDARY",
-                "Clean room Reader execution has no enforceable memory boundary off Linux",
+                "the hosted-session clean room is networkless and cannot perform retrieval",
             )
         return None
 
@@ -1241,16 +1198,6 @@ class CrawlerSandbox:
     def _memory_boundary_refusal(self) -> AcquisitionSecurityError | None:
         """Whether this confinement can bound a child's memory at all, for any operation."""
 
-        if (
-            sys.platform == "linux"
-            and self._memory_cgroup_root_fd is None
-            and self._hosted_session_container_memory_bytes is None
-            and self._child_address_space_bytes is None
-        ):
-            return AcquisitionSecurityError(
-                "SANDBOX_MEMORY_BOUNDARY",
-                "Linux Clean room execution requires a trusted bounded cgroup root",
-            )
         return None
 
     def confinement_refusal(self, operation: str) -> AcquisitionSecurityError | None:
@@ -1350,7 +1297,10 @@ class CrawlerSandbox:
                 prepared,
                 request_bytes=request_bytes,
                 timeout_seconds=self._timeout_seconds,
-                terminate_process_group=(self._hosted_session_container_memory_bytes is not None),
+                terminate_process_group=(
+                    self._hosted_session_container_memory_bytes is not None
+                    or self._child_is_networkless()
+                ),
             )
         if returncode != 0:
             refusal = _bounded_worker_error(stderr, returncode=returncode)
@@ -1488,6 +1438,27 @@ def worker_main(*, prewarm: bool = False) -> int:
     except (AcquisitionSecurityError, ValueError, TypeError) as exc:
         code = getattr(exc, "code", "SANDBOX_REQUEST")
         return _worker_error(code, str(exc))
+    # ``RLIMIT_AS`` is the production per-child memory boundary on Cloud Run: the
+    # platform supplies no writable cgroup subtree for a batch parser.  An
+    # allocation refused by that boundary is an expected, safely actionable
+    # terminal condition, not an unmapped implementation defect.  Preserve no
+    # exception text -- allocation errors can arise deep in a native decoder --
+    # but give Studio the stable code it needs to report the failure truthfully.
+    except MemoryError:
+        return _worker_error(
+            "SANDBOX_MEMORY_LIMIT",
+            "the worker address-space limit refused an allocation",
+        )
+    except OSError as exc:
+        if exc.errno == errno.ENOMEM:
+            return _worker_error(
+                "SANDBOX_MEMORY_LIMIT",
+                "the worker address-space limit refused an allocation",
+            )
+        return _worker_error(
+            "SANDBOX_UNEXPECTED",
+            f"the worker raised {type(exc).__name__}, which no refusal maps",
+        )
     except Exception as exc:
         return _worker_error(
             "SANDBOX_UNEXPECTED",
@@ -3469,7 +3440,15 @@ def _probe_network_denials(
     results: list[tuple[str, int]] = []
     for label, ip, port in targets:
         family = socket.AF_INET6 if ":" in ip else socket.AF_INET
-        probe = socket.socket(family, socket.SOCK_STREAM)
+        # A networkless seccomp profile may reject the socket syscall itself rather than letting
+        # it create a descriptor and rejecting connect(2). Both outcomes prove the same boundary;
+        # socket construction therefore belongs in the typed denial path too. Leaving it outside
+        # the try turned one valid kernel posture into SANDBOX_UNEXPECTED.
+        try:
+            probe = socket.socket(family, socket.SOCK_STREAM)
+        except OSError as exc:
+            results.append((label, exc.errno or 1))
+            continue
         try:
             probe.settimeout(0.25)
             error = probe.connect_ex((ip, port))
@@ -3890,8 +3869,8 @@ def _drain_worker_ipc(
                         "The Clean room process returned more data than this job allows",
                     )
 
-        # Hosted-session children may fork bounded parser helpers, but no helper may outlive the
-        # one request. Signal the still-owned process group before reaping its leader: this covers
+        # Networkless children may fork bounded parser helpers, but no helper may outlive the one
+        # request. Signal the still-owned process group before reaping its leader: this covers
         # the success path, catches descendants that deliberately closed all inherited pipes, and
         # avoids signalling a numeric group id after the leader pid could have been reused.
         if terminate_process_group:
@@ -4920,10 +4899,10 @@ def _prepare_worker(
             raise error
         if isinstance(error, AcquisitionSecurityError):
             raise
-        if hosted_session_container_memory_bytes is not None:
+        if hosted_session_container_memory_bytes is not None or networkless_child:
             raise AcquisitionSecurityError(
                 "SANDBOX_OS_BOUNDARY",
-                "hosted-session networkless process boundary could not be established before exec",
+                "networkless process boundary could not be established before exec",
             ) from error
         raise AcquisitionSecurityError(
             "SANDBOX_MEMORY_BOUNDARY",
@@ -5190,7 +5169,9 @@ def _run_worker_command(
         prepared,
         request_bytes=request_bytes,
         timeout_seconds=timeout_seconds,
-        terminate_process_group=hosted_session_container_memory_bytes is not None,
+        terminate_process_group=(
+            hosted_session_container_memory_bytes is not None or networkless_child
+        ),
     )
 
 

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/src/mostlyright/data_harness/cli.py

@@ -2029,6 +2029,10 @@ def _parser(
 
     _recipe.declare_arguments(_add("recipe", help=_recipe.COMMAND_HELP))
 
+    from mostlyright.data_harness.thin import v4_dataset_covers as _v4_dataset_covers
+
+    _v4_dataset_covers.declare_arguments(_add("cover", help=_v4_dataset_covers.COMMAND_HELP))
+
     # The commands that read a run back once it has finished, declared the same way and for the
     # same reason: one declaration, two front doors, no argument that can differ between them.
     from mostlyright.data_harness.thin import v4_artifacts as _v4_artifacts

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/src/mostlyright/data_harness/formats.py

@@ -46,6 +46,7 @@ __all__ = [
     "SNAPSHOT_SUFFIXES",
     "SNAPSHOT_SUFFIX_PATTERN",
     "WEAK_TEXT_MEDIA_TYPES",
+    "resolve_direct_data_format",
 ]
 
 # The label of the Reader contract a recipe pins itself to.  A Reader family is the code
@@ -153,32 +154,17 @@ DIRECT_FETCH_MEDIA_TYPES: Mapping[str, frozenset[str]] = MappingProxyType(
     }
 )
 
-# Which wire formats a FETCHED media type may be read back into, and why the set is this narrow.
-#
-# A registered acquisition document can say where bytes are without saying how they are encoded,
-# and every acquisition here requires an encoding.  The honest answer is to read it back out of
-# what the response actually carried, through the inverse of the map above.  That inverse is only
-# a function over the set below.
-#
-# ``text/plain`` is accepted for every text-shaped format, so an inverse taken over ``csv``,
-# ``json`` and ``ndjson`` at once would have to answer one weak label with three formats and pick
-# between them -- which is a guess wearing a table's clothes.  Narrowing the domain is what makes
-# the inverse total and unambiguous instead of merely plausible.
-#
-# One member today, and that is three independent constraints rather than a preference: a
-# transform plan admits exactly one input encoding, the only bridge from any other encoding into
-# it is a Reader family, and the in-process acquisition path refuses a Reader-pinned source
-# outright.  A second member becomes admissible only when all three of those move, and the
-# derivation below REFUSES rather than choosing if one is added while a label stays shared.
-INFERRED_DATA_FORMATS: frozenset[str] = frozenset({"csv"})
+# Formats that direct, unpinned fetches can resolve after receiving their response.  This is
+# deliberately a literal closed set: adding a parser format is not enough to make it a native
+# direct format; its fetch resolution and DuckDB dispatch must move with it.
+INFERRED_DATA_FORMATS: frozenset[str] = frozenset({"csv", "json", "ndjson", "parquet"})
 
 
 def _inverted_media_types(data_formats: frozenset[str]) -> dict[str, str]:
-    """Invert ``DIRECT_FETCH_MEDIA_TYPES`` over ``data_formats``, refusing a shared label.
+    """Invert the strong media-type table over ``data_formats``, refusing a shared label.
 
-    DERIVED rather than re-typed, so a widening on one side cannot leave the other behind: a media
-    type added to a format's accepted set becomes readable-back the same moment it becomes
-    fetchable, and the two directions cannot come to disagree about one label.
+    ``text/plain`` is a weak label shared by text formats.  It therefore cannot appear in this
+    table: it is resolved only from a bounded content prefix after fetch.
 
     Taking the set as an argument rather than reading the constant is what lets the shared-label
     refusal be exercised: it is unreachable while the domain has one member, and a guard nothing
@@ -187,7 +173,7 @@ def _inverted_media_types(data_formats: frozenset[str]) -> dict[str, str]:
 
     inverse: dict[str, str] = {}
     for data_format in sorted(data_formats):
-        accepted = DIRECT_FETCH_MEDIA_TYPES.get(data_format)
+        accepted = FORMAT_MEDIA_TYPES.get(data_format)
         if accepted is None:
             raise ValueError(
                 f"{data_format} has no accepted media types, so nothing can be read back into it"
@@ -212,6 +198,35 @@ MEDIA_TYPE_DATA_FORMATS: Mapping[str, str] = MappingProxyType(
     _inverted_media_types(INFERRED_DATA_FORMATS)
 )
 
+
+def resolve_direct_data_format(media_type: str, content: bytes) -> str | None:
+    """Resolve an unpinned direct response without guessing from an ambiguous weak label."""
+
+    normalized = str(media_type).split(";", 1)[0].strip().lower()
+    for data_format, accepted in FORMAT_MEDIA_TYPES.items():
+        if normalized in accepted:
+            return data_format
+    if normalized not in WEAK_TEXT_MEDIA_TYPES:
+        return None
+
+    sample = content[: 64 * 1024].lstrip()
+    if sample.startswith(b"["):
+        return "json"
+    if sample.startswith(b"{"):
+        return "ndjson"
+    try:
+        lines = sample.decode("utf-8").splitlines()
+    except UnicodeDecodeError:
+        return None
+    nonempty = [line for line in lines if line.strip()]
+    if len(nonempty) < 2:
+        return None
+    header, first_row = nonempty[:2]
+    if header.count(",") > 0 and header.count(",") == first_row.count(","):
+        return "csv"
+    return None
+
+
 # Filename suffixes accepted for each wire format.  A format may carry more than one.
 FORMAT_SUFFIXES: Mapping[str, frozenset[str]] = MappingProxyType(
     {

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/src/mostlyright/data_harness/hosted_bootstrap.py

@@ -850,6 +850,24 @@ def _validate_v4_bootstrap_response(value: dict[str, Any]) -> tuple[dict[str, by
     if len(capability_raw) > MAX_CAPABILITY_BYTES:
         raise HostedWorkerError("BOOTSTRAP_TOO_LARGE", "capability exceeds its byte limit")
     parse_capability(capability_raw)
+    attempt_id = value["attempt_id"]
+    if not isinstance(attempt_id, str):
+        raise HostedWorkerError(
+            "BOOTSTRAP_RESPONSE_INVALID",
+            "v4 bootstrap attempt_id must be a canonical UUID",
+        )
+    try:
+        parsed_attempt_id = UUID(attempt_id)
+    except ValueError as error:
+        raise HostedWorkerError(
+            "BOOTSTRAP_RESPONSE_INVALID",
+            "v4 bootstrap attempt_id must be a canonical UUID",
+        ) from error
+    if str(parsed_attempt_id) != attempt_id:
+        raise HostedWorkerError(
+            "BOOTSTRAP_RESPONSE_INVALID",
+            "v4 bootstrap attempt_id must be a canonical UUID",
+        )
     job_raw = canonical_json_bytes(dict(job_wire))
     if len(job_raw) > MAX_DESCRIPTOR_BYTES:
         raise HostedWorkerError("BOOTSTRAP_TOO_LARGE", "worker job exceeds its byte limit")
@@ -866,7 +884,11 @@ def _validate_v4_bootstrap_response(value: dict[str, Any]) -> tuple[dict[str, by
             "BOOTSTRAP_RESPONSE_INVALID",
             "v4 job carries no canonical workspace id",
         )
-    return {"job": job_raw, "capability": capability_raw}, _V4AdmissionJob(workspace_id)
+    return {
+        "job": job_raw,
+        "capability": capability_raw,
+        "attempt_id": attempt_id.encode("ascii"),
+    }, _V4AdmissionJob(workspace_id)
 
 
 def _anonymous_payload(raw: bytes, label: str) -> _AnonymousPayload:
@@ -988,6 +1010,8 @@ def _worker_argv(
         "--capability-fd",
         str(payloads["capability"].descriptor),
     ]
+    if "attempt_id" in payloads:
+        arguments.extend(("--attempt-id-fd", str(payloads["attempt_id"].descriptor)))
     return arguments
 
 
@@ -1357,7 +1381,18 @@ def main(argv: Sequence[str] | None = None) -> int:
             )
         )
     except HostedWorkerError as error:
-        print(f"{error.code}: {error.detail}", file=sys.stderr)
+        sys.stderr.buffer.write(
+            canonical_json_bytes(
+                {
+                    "status": "failed",
+                    "code": error.code,
+                    "detail": error.detail,
+                    "stage": "bootstrap",
+                }
+            )
+            + b"\n"
+        )
+        sys.stderr.flush()
         return 1
     return 0
 

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/src/mostlyright/data_harness/hosted_crawler.py

@@ -34,7 +34,12 @@ from mostlyright.data_harness.canonical import (
     parse_canonical_json,
     sha256_bytes,
 )
-from mostlyright.data_harness.formats import DATA_FORMATS
+from mostlyright.data_harness.formats import (
+    DATA_FORMATS,
+    DIRECT_FETCH_MEDIA_TYPES,
+    FORMAT_SUFFIXES,
+    resolve_direct_data_format,
+)
 from mostlyright.data_harness.hosted_crawler_protocol import (
     CRAWLER_REQUEST_SCHEMA_V3,
     CRAWLER_RESULT_SCHEMA_V1,
@@ -480,7 +485,7 @@ def _public_https_query(value: Any) -> dict[str, Any]:
     _canonical_url_host(url)
     data_format = value["data_format"]
     filename = value["filename"]
-    if data_format not in DATA_FORMATS:
+    if data_format not in DATA_FORMATS | {"auto"}:
         raise HostedCrawlerError("CRAWLER_QUERY_INVALID", "public HTTPS format is unsupported")
     if (
         not isinstance(filename, str)
@@ -519,6 +524,10 @@ def _public_https_query(value: Any) -> dict[str, Any]:
                 "CRAWLER_QUERY_INVALID", "public HTTPS max_nesting_depth must be exactly 1"
             )
     if reader is not None:
+        if data_format == "auto":
+            raise HostedCrawlerError(
+                "CRAWLER_QUERY_INVALID", "a Reader pin must declare its output format"
+            )
         if not isinstance(reader, dict) or set(reader) != {
             "family_id",
             "family_version",
@@ -560,13 +569,25 @@ def _public_expected_media_types(query: dict[str, Any]) -> tuple[str, ...]:
         # admitted set from the pinned family, and a fetch with no pin has no recipe to
         # protect, so the weak text label a public agency serves for CSV or JSON is
         # admitted here or nowhere.
-        from mostlyright.data_harness.formats import DIRECT_FETCH_MEDIA_TYPES
-
+        if query["data_format"] == "auto":
+            return tuple(sorted(set().union(*DIRECT_FETCH_MEDIA_TYPES.values())))
         return tuple(sorted(DIRECT_FETCH_MEDIA_TYPES[query["data_format"]]))
     family = TOOLBOX.resolve(reader["family_id"], reader["family_version"])
     return family.accepted_media_types
 
 
+def _resolve_direct_public_format(media_type: str, content: bytes) -> str | None:
+    return resolve_direct_data_format(media_type, content)
+
+
+def _normalized_filename(filename: str, data_format: str) -> str:
+    suffixes = FORMAT_SUFFIXES[data_format]
+    if filename.endswith(tuple(suffixes)):
+        return filename
+    stem = filename.rsplit(".", 1)[0] if "." in filename else filename
+    return f"{stem}{sorted(suffixes)[0]}"
+
+
 def _normalize_public_https_response(
     request: HostedCrawlerRequest,
     *,
@@ -593,9 +614,14 @@ def _normalize_public_https_response(
     reader = query["reader_pin"]
     if reader is None:
         normalized = raw
-        normalized_format = query["data_format"]
+        normalized_format = _resolve_direct_public_format(media_type, raw)
+        if normalized_format is None:
+            raise HostedCrawlerError(
+                "CRAWLER_FORMAT_UNRESOLVED",
+                "public HTTPS response did not provide a deterministic tabular format",
+            )
         normalized_media_type = media_type
-        normalized_filename = query["filename"]
+        normalized_filename = _normalized_filename(query["filename"], normalized_format)
         family_id = family_version = options_digest = None
         decode_flags: list[str] = []
         reader_budgets = None

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/src/mostlyright/data_harness/hosted_crawler_job.py

@@ -1037,12 +1037,20 @@ def _validate_public_https_result(
     query = request.query
     reader = query.get("reader_pin")
     if reader is None:
+        from mostlyright.data_harness.hosted_crawler import (
+            _normalized_filename,
+            _resolve_direct_public_format,
+        )
+
         reader_identity = (None, None, None)
         expected_resource_caps = None
         expected_reader_budgets = None
+        resolved_format = _resolve_direct_public_format(retrieved.media_type, retrieved.content)
+        if resolved_format is None:
+            _fail("CRAWLER_CHILD_INVALID", "public HTTPS format could not be resolved")
         direct_identity = (
-            query.get("data_format"),
-            query.get("filename"),
+            resolved_format,
+            _normalized_filename(query.get("filename", ""), resolved_format),
             retrieved.media_type,
         )
     else:

{mostlyright_data-0.10.1 → mostlyright_data-0.10.2}/src/mostlyright/data_harness/hosted_ingestion_job.py

@@ -322,7 +322,7 @@ def ingestion_job_main(argv: Sequence[str] | None = None) -> int:
     try:
         dispatch = parse_ingestion_dispatch(arguments)
     except HostedIngestionJobError as error:
-        sys.stderr.write(f"{error.code}: {error.detail}\n")
+        _write_terminal_failure(error)
         return 2
     try:
         config = IngestionJobConfig.from_environment()
@@ -331,15 +331,34 @@ def ingestion_job_main(argv: Sequence[str] | None = None) -> int:
     except HostedIngestionJobError as error:
         if error.code in CONSUMED_INGESTION_DISPATCH_CODES:
             return 0
-        sys.stderr.write(f"{error.code}: {error.detail}\n")
+        _write_terminal_failure(error)
         return 1
     # The deliberately closed dispatch contract has no source URL, generic callback endpoint or
     # credential.  A later reviewed connector/callback slice must call run_hosted_ingestion_delivery
     # instead of making this entrypoint silently invent one of those authorities.
-    sys.stderr.write("INGESTION_DELIVERY_PROCESSOR_UNWIRED: no reviewed connector is installed\n")
+    _write_terminal_failure(
+        HostedIngestionJobError(
+            "INGESTION_DELIVERY_PROCESSOR_UNWIRED", "no reviewed connector is installed"
+        )
+    )
     return 1
 
 
+def _write_terminal_failure(error: HostedIngestionJobError) -> None:
+    sys.stderr.buffer.write(
+        canonical_json_bytes(
+            {
+                "status": "failed",
+                "code": error.code,
+                "detail": error.detail,
+                "stage": "ingestion",
+            }
+        )
+        + b"\n"
+    )
+    sys.stderr.flush()
+
+
 __all__ = [
     "CHECKPOINT_RESERVE_SECONDS",
     "CONSUMED_INGESTION_DISPATCH_CODES",