mostlyright-data 0.17.3__tar.gz → 0.17.5__tar.gz

{mostlyright_data-0.17.3 → mostlyright_data-0.17.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.17.3
+Version: 0.17.5
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.17.3 → mostlyright_data-0.17.5}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.17.3"
+version = "0.17.5"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.17.3 → mostlyright_data-0.17.5}/skills/mr-data-build/SKILL.md

@@ -389,10 +389,10 @@ around a probe from an hour ago still being warm.
 
 Where a source needs a credential, enrol it by name first — see
 [Source credentials](#source-credentials) — and reference the name from the recipe document. Where
-neither a probe nor a run can reach the bytes, because the source is a bounded collection or is
-not registered in the workspace, establish shape from the provider's own documentation and
-published schema, say plainly that the shape is documented rather than observed, and let the hosted
-Clean room be the first thing that opens the bytes. Named messages out of one large object are
+neither a probe nor a run can reach the bytes, because the source is not registered in the
+workspace, establish shape from the provider's own documentation and published schema, say
+plainly that the shape is documented rather than observed, and let the hosted Clean room be the
+first thing that opens the bytes. Named messages out of one large object are
 [`acquire-slices`](#not-hosted-yet), which is not hosted yet.
 
 Hold the comparison in the shape [Narrating the run](#narrating-the-run) takes: one `source` block
@@ -799,9 +799,12 @@ one", not as a failure: a document declaring 8 MiB over a feed that answers 40 K
 **If a source really does deliver more than the Reader holds, do not answer it by lowering the
 ceiling on that source:** that does not fetch less data, it truncates the sample or refuses the
 full run. Split the window across several sources — one per month, per region, or per whatever the
-publisher paginates on — and union them in a first transform step. A recipe names up to 64 sources,
-which is a great many windows. [The recipe document](../../docs/RECIPE-DOCUMENT.md) carries the
-measured numbers behind the ceiling.
+publisher paginates on — and union them in a first transform step. A recipe names up to 256
+sources, which is a great many windows. [The recipe document](../../docs/RECIPE-DOCUMENT.md)
+carries the measured numbers behind the ceiling.
+
+**A corpus of many pages behind one index is a COLLECTION, not many sources.** See
+[Authoring a collection source](#authoring-a-collection-source).
 
 **The dataset's record is the primary one; a run's own record is for what happened inside that
 run.** The sources, the decisions and the questions went onto the dataset in stage 3, where they
@@ -1570,6 +1573,53 @@ What it costs the protocol, and what to do instead:
 - Preserve event time, available time, acquisition time, source revision, and timezone. Do not use
   post-cutoff information in targets, labels, features, joins, or validation decisions.
 
+### Authoring a collection source
+
+A corpus published as an index plus many detail pages is **one** source through
+`public.https.collection@2.0.0`, not one source per page. Source count and publisher count are
+unaffected by how many pages sit behind it: 2,819 pages is one source, one publisher, one line on
+the dataset's sources card.
+
+Write it as a source whose connector is that coordinate plus a `collection` member:
+
+- `collection.discovery` says how the corpus is LISTED — `json_api` with RFC 6901 pointers,
+  `html_index` with the reader's closed selectors, `sitemap`, or an `explicit` member list — and
+  which pagination grammar the listing uses. The run follows only that grammar and never a link
+  found inside a detail page.
+- `collection.pages` says what may be fetched and how much: `allowed_origins` (which must include
+  the discovery address's own origin), `max_pages`, `max_fetches_per_run`, `concurrency` (1 or 2),
+  `min_interval_seconds` as a whole number of seconds (one or more), `request_timeout_seconds`,
+  `retries`, and a `revisit` rule. Only `allowed_origins` is required; every other member has a
+  stated meaning when it is left out.
+- `limits.max_requests` is REQUIRED and bounds every request the run makes, discovery and detail
+  pages together. `limits.max_source_bytes` bounds the fetched bytes; `limits.max_rows` clamps the
+  merged relation.
+- The Reader pin is required, must be `html.web_extract`, and is the SAME settings for every page.
+
+The relation the transform reads is the reader's columns plus `page_id`, `page_url`,
+`page_fetched_at`, `page_content_sha256`, `page_revision`, `page_discovered_at` and `page_ordinal`.
+Those names are reserved: a reader that declares one is a registration refusal. Select them in the
+transform so every row on the table says which page and which revision produced it.
+
+**Expect the first run not to finish, and say so.** A run that spends its fetch budget still
+SUCCEEDS; its coverage block reports `complete: false` and names the budget that ended it. The
+next refresh continues from the ledger — new pages first, then failures oldest first, then
+revisits — and every row of a page this run did not fetch survives byte for byte. Report the
+shortfall as what it is, a backfill in progress, not as a failure and not as a complete corpus.
+
+A page that disappears from the index deletes nothing. A page answering with different content
+replaces exactly its own rows, increments `page_revision`, and keeps the previous digest: that is
+a correction, and it is worth saying on the run record when it happens.
+
+Before registering it, read the discovery block back the way a run will: the address the first
+request goes to, whether the pagination parameter is already written into that address, whether
+the records pointer names an array, and whether `discovery.max_requests` plus
+`pages.max_fetches_per_run` fits inside `limits.max_requests`. Registration refuses all four, and
+they are the four that are cheapest to get right before a run spends anybody's rate limit.
+
+The full contract, including every bound and the coverage block's members, is
+[Source collections](../../docs/SOURCE-COLLECTIONS.md).
+
 ## Readers
 
 `docs/READERS.md` defines Reader behavior. Certified Readers may be pinned on an exact
@@ -1577,10 +1627,11 @@ What it costs the protocol, and what to do instead:
 exact Reader name and version. Unsupported versions refuse by name. A version change is a recipe
 revision.
 
-**Without a pin a source must answer `text/csv` or `text/plain`, and both are parsed as CSV.**
-Anything else is refused `MEDIA_TYPE` before a row is read. So a publisher that serves only JSON or
-XML needs a pin; it is not unbuildable, and it is not a reason to go looking for a different
-publisher.
+**Direct CSV supports publisher CSV aliases, including Eurostat SDMX-CSV.**
+Accepted labels include `text/csv`, `text/plain`, `application/csv`, `application/x-csv`,
+`text/x-csv`, and `application/vnd.sdmx.data+csv`. For title rows, alternate encodings,
+headerless files, or padded columns, pin `delimited_text@2.0.0` and configure the layout; see
+`docs/READERS.md`. A publisher needing another format can use its matching Reader.
 
 **A v4 recipe pins a Reader through `connector.parameters`, under three reserved names.**
 `declared_source` is closed and has no member for a decoder, so the pin rides on the parameter array
@@ -1657,11 +1708,13 @@ decoder. Zero matches and multiple matches both refuse.
 | `spreadsheet.xlsx@1.1.0` | Same; also accepts a response served as `application/octet-stream`. |
 | `delimited_text@1.0.0` | `delimiter` (comma or tab); optional `header: true`; optional `encoding: utf-8`. |
 | `delimited_text@1.1.0` | Same; also accepts a response served as `text/plain` (common for public agency CSV such as IEM). |
+| `delimited_text@2.0.0` | Explicit delimiter, encoding, quoting, header presence, skip_rows, skip_blank_rows, and indexed columns with output names and optional expected_header; unselected_columns requires empty cells unless explicitly set to allow. |
 | `json.tabular@1.0.0` | `columns` with output names and RFC 6901 pointers; optional `document_format` (`json` or `ndjson`), `records_pointer`, and sequential `expand` pointers. Columns are required unless they state `required: false`. |
 | `json.tabular@1.1.0` | Same; also accepts a response served as `text/plain` (common for public agency JSON and NDJSON). |
 | `xml.tabular@1.0.0` | Absolute record element path, declared namespace prefixes, and relative field selectors ending in `text()` or `@attribute`; fields are required unless `required: false`. |
 | `html.tabular@1.0.0` | Absolute record element path and relative text/attribute field selectors; fields are required unless `required: false`. |
 | `html.web_extract@1.0.0` | `mode: dom` with closed CSS-like record/field selectors, or `mode: json` with an inert JSON script selector and RFC 6901 field pointers. |
+| `html.web_extract@1.1.0` | Same settings as 1.0.0; extracted text follows DOM order, with one space at block-element boundaries and `<br>` and none between inline runs. |
 | `protobuf.tabular@1.0.0` | Digest-pinned base64 FileDescriptorSet, fully-qualified message type, `single` or `length_delimited` framing, and fields with an explicit repeated `explode` or `refuse` choice. |
 | `parquet.tabular@1.0.0` | Optional `columns` selects a bounded ordered Parquet projection. |
 | `xbrl.facts@1.0.0` | `document_kind` (`xbrl` or constrained XML `ixbrl`) and exact namespace/name fact projection; contexts, units, and metadata remain evidence fields, not joins. |

{mostlyright_data-0.17.3 → mostlyright_data-0.17.5}/src/mostlyright/data_harness/acquisition/http.py

@@ -3087,6 +3087,10 @@ class _PinnedProxyHTTPSConnection(http.client.HTTPSConnection):
             )
             response = http.client.HTTPResponse(raw_socket)
             response.begin()
+            if response.status == 503:
+                raise AcquisitionSecurityError(
+                    "CONNECT_PROXY_BUSY", "gateway capacity is temporarily unavailable"
+                )
             if response.status != 200:
                 raise AcquisitionSecurityError(
                     "CONNECT_PROXY_STATUS", "gateway refused the pinned tunnel"

{mostlyright_data-0.17.3 → mostlyright_data-0.17.5}/src/mostlyright/data_harness/acquisition/sandbox.py

@@ -279,6 +279,17 @@ MAX_ANY_REQUEST_BYTES = max((MAX_REQUEST_BYTES, *MAX_REQUEST_BYTES_BY_OPERATION.
 # ``max_carried_source_bytes`` subtracts it before dividing out the base64 expansion, which is
 # what keeps a fetch ceiling derived from a request budget from being an over-estimate.
 MAX_REQUEST_ENVELOPE_BYTES = 4 * 1024
+
+
+# JSON escapes each admitted header character into at most six bytes. Include
+# the actual sealed configuration as well as bounded header/pointer/control overhead.
+def max_generic_api_page_source_bytes(configuration: Mapping[str, Any]) -> int:
+    envelope = len(_ipc_json_bytes(dict(configuration))) + 128 * 1024
+    return max(0, (_request_byte_budget("generic_api_page") - envelope) // 4 * 3)
+
+
+GENERIC_API_PAGE_OUTPUT_BYTES = (MAX_RESPONSE_BYTES - 128 * 1024) // 4 * 3
+
 MAX_WORKER_STDERR_BYTES = 64 * 1024
 # The most rows a response may state, whether it carries them or only their count. One value for
 # both directions on purpose: a table too long to hand back is also a table too long to have been
@@ -409,6 +420,11 @@ _ALLOWED_ENVIRONMENT = {
     "PYTHONDONTWRITEBYTECODE": "1",
     "PYTHONIOENCODING": "utf-8",
     "PYTHONNOUSERSITE": "1",
+    # Native math libraries otherwise reserve one thread stack per host CPU on
+    # import. Arrow's NumPy dependency can then exhaust RLIMIT_AS before a
+    # request is read, even though a single-thread import fits the same bound.
+    "OPENBLAS_NUM_THREADS": "1",
+    "OMP_NUM_THREADS": "1",
 }
 _FORBIDDEN_EGRESS_HOSTS = frozenset(
     {
@@ -1220,31 +1236,83 @@ class CrawlerSandbox:
         narrow the family's own defaults; it can never buy more room than the family allows.
         """
 
-        return self._invoke(
-            {
-                "schema_version": SANDBOX_PROTOCOL_VERSION,
-                "request_id": request_id,
-                "operation": "decode_and_parse",
-                "content_base64": base64.b64encode(content).decode("ascii"),
-                "data_format": output_format,
-                # The media type and the filename of the parsed artifact are the Reader
-                # family's declarations, not the coordinator's choice, so they travel back on
-                # the response rather than out on the request.
-                "media_type": None,
-                "filename": None,
-                "parse_limits": _parse_limits_dict(limits),
-                "url": None,
-                "allowed_hostnames": [],
-                "approved_endpoints": [],
-                "retrieval_limits": None,
-                "network_probe_targets": [],
-                "reader_pin": dict(reader_pin),
-                "reader_budgets": None if reader_budgets is None else dict(reader_budgets),
-                "range_plan": None,
-                "cycle_survey": None,
-            },
-            approved_endpoints=(),
-        )
+        request = {
+            "schema_version": SANDBOX_PROTOCOL_VERSION,
+            "request_id": request_id,
+            "operation": "decode_and_parse",
+            "content_base64": None,
+            "data_format": output_format,
+            # The media type and the filename of the parsed artifact are the Reader
+            # family's declarations, not the coordinator's choice, so they travel back on
+            # the response rather than out on the request.
+            "media_type": None,
+            "filename": None,
+            "parse_limits": _parse_limits_dict(limits),
+            "url": None,
+            "allowed_hostnames": [],
+            "approved_endpoints": [],
+            "retrieval_limits": None,
+            "network_probe_targets": [],
+            "reader_pin": dict(reader_pin),
+            "reader_budgets": None if reader_budgets is None else dict(reader_budgets),
+            "range_plan": None,
+            "cycle_survey": None,
+        }
+        # Include optional fields _invoke adds, and base64 padding, before allocating a copy.
+        inline_size = len(_ipc_json_bytes(request)) + 4 * ((len(content) + 2) // 3)
+        if inline_size + MAX_REQUEST_ENVELOPE_BYTES <= _request_byte_budget("decode_and_parse"):
+            request["content_base64"] = base64.b64encode(content).decode("ascii")
+            return self._invoke(request, approved_endpoints=())
+        # Keep bulk source bytes outside the control channel. The child receives one
+        # read-only, unlinked regular file; neither a path nor a writable descriptor.
+        pin = _reader_pin_from_request(dict(reader_pin))
+        family = TOOLBOX.resolve(pin.family_id, pin.family_version)
+        budget = family.default_budgets.narrowed_by(reader_budgets)
+        if len(content) > budget.max_input_bytes:
+            raise AcquisitionSecurityError("READER_BUDGET", "decode input exceeds Reader budget")
+        with self._reader_input_descriptor(content) as descriptor:
+            request["content_base64"] = None
+            request["file_input_descriptor"] = descriptor
+            request["file_output_descriptor"] = None
+            return self._invoke(request, approved_endpoints=(), inherited_fds=(descriptor,))
+
+    @contextlib.contextmanager
+    def _reader_input_descriptor(self, content: bytes) -> Iterator[int]:
+        root_fd = os.open(self._staging_root, os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC)
+        name = "reader-" + secrets.token_hex(16)
+        write_fd = read_fd = -1
+        created = False
+        try:
+            write_fd = os.open(
+                name,
+                os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW | os.O_CLOEXEC,
+                0o600,
+                dir_fd=root_fd,
+            )
+            created = True
+            view = memoryview(content)
+            while view:
+                count = os.write(write_fd, view)
+                if count <= 0:
+                    raise OSError("short Reader input write")
+                view = view[count:]
+            read_fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC, dir_fd=root_fd)
+            written, opened = os.fstat(write_fd), os.fstat(read_fd)
+            if (written.st_dev, written.st_ino) != (opened.st_dev, opened.st_ino):
+                raise AcquisitionSecurityError("SANDBOX_FILE", "Reader input identity changed")
+            os.unlink(name, dir_fd=root_fd)
+            created = False
+            os.close(write_fd)
+            write_fd = -1
+            yield read_fd
+        finally:
+            if created:
+                with contextlib.suppress(OSError):
+                    os.unlink(name, dir_fd=root_fd)
+            for descriptor in (write_fd, read_fd, root_fd):
+                if descriptor >= 0:
+                    with contextlib.suppress(OSError):
+                        os.close(descriptor)
 
     def retrieve_and_parse(
         self,
@@ -1562,6 +1630,7 @@ class CrawlerSandbox:
         prewarm: bool,
         resource_profile: _WorkerResourceProfile = _ORDINARY_WORKER_RESOURCE_PROFILE,
         descriptor_window_fds: tuple[int, int] | None = None,
+        descriptor_reader_fd: int | None = None,
     ) -> _WorkerLaunch:
         """Build the exact confinement one child is started in.
 
@@ -1578,7 +1647,16 @@ class CrawlerSandbox:
             "mostlyright.data_harness.acquisition.sandbox",
             "--worker",
         ]
-        if descriptor_window_fds is not None:
+        if descriptor_reader_fd is not None:
+            command = [
+                self._python,
+                "-I",
+                "-m",
+                "mostlyright.data_harness.acquisition.sandbox",
+                "--descriptor-reader-worker",
+                str(descriptor_reader_fd),
+            ]
+        elif descriptor_window_fds is not None:
             source_fd, output_fd = descriptor_window_fds
             command = [
                 self._python,
@@ -1761,17 +1839,32 @@ class CrawlerSandbox:
             and self._stream_timeout_seconds is not None
             else self._timeout_seconds
         )
+        descriptor_reader = (
+            operation == "decode_and_parse"
+            and bool(inherited_fds)
+            and isinstance(request.get("reader_pin"), dict)
+            and request["reader_pin"].get("family_id") == "meteorology.grib"
+            and request["reader_pin"].get("family_version") == "1.0.0"
+        )
         launch = (
             self._worker_launch(
                 prewarm=False,
                 resource_profile=resource_profile,
-                descriptor_window_fds=(
-                    request["file_input_descriptor"],
-                    request["file_output_descriptor"],
-                ),
+                descriptor_reader_fd=request["file_input_descriptor"],
+            )
+            if descriptor_reader
+            else (
+                self._worker_launch(
+                    prewarm=False,
+                    resource_profile=resource_profile,
+                    descriptor_window_fds=(
+                        request["file_input_descriptor"],
+                        request["file_output_descriptor"],
+                    ),
+                )
+                if operation == "parse_streaming_csv_window"
+                else self._worker_launch(prewarm=False)
             )
-            if operation == "parse_streaming_csv_window"
-            else self._worker_launch(prewarm=False)
         )
         # A warm child is only ever taken for a confinement identical to the one this
         # invocation would have spawned for itself, and only when no socket is in play. Taking
@@ -1795,9 +1888,11 @@ class CrawlerSandbox:
                 "networkless_child": self._child_is_networkless(),
                 "inherited_fds": inherited_fds,
                 "on_output_progress": on_output_progress,
-                "descriptor_window": operation == "parse_streaming_csv_window",
+                # Both closed descriptor entrypoints lock down path opens after trusted
+                # prewarm. Other inherited-Reader lanes retain their Landlock boundary.
+                "descriptor_window": operation == "parse_streaming_csv_window" or descriptor_reader,
             }
-            if operation == "parse_streaming_csv_window":
+            if operation == "parse_streaming_csv_window" or descriptor_reader:
                 worker_arguments["resource_profile"] = resource_profile
             stdout, stderr, returncode = _run_worker_command(launch.argv(), **worker_arguments)
         else:
@@ -1940,8 +2035,88 @@ def _prime_descriptor_window_csv() -> None:
         raise RuntimeError("trusted CSV window prewarm did not produce its fixed result")
 
 
+def _prime_descriptor_grib_reader() -> None:
+    """Resolve only bundled, trusted GRIB definitions before path access is denied."""
+    from mostlyright.data_harness.readers.grib_eccodes import MeteorologyGribReader, _eccodes
+
+    module = _eccodes()
+    family = MeteorologyGribReader()
+    for sample in ("regular_ll_sfc_grib1", "regular_ll_sfc_grib2"):
+        handle = module.codes_grib_new_from_samples(sample)
+        try:
+            content = module.codes_get_message(handle)
+            names = (
+                ("edition", "edition"),
+                ("param_id", "paramId"),
+                ("type_of_level", "typeOfLevel"),
+                ("level", "level"),
+                ("data_date", "dataDate"),
+                ("data_time", "dataTime"),
+                ("step_range", "stepRange"),
+            )
+            selector = {name: module.codes_get(handle, key) for name, key in names}
+            selector["step_range"] = str(selector["step_range"])
+        finally:
+            module.codes_release(handle)
+        pin = _reader_pin_from_request(
+            {
+                "family_id": family.family_id,
+                "family_version": family.family_version,
+                "decode_options": {
+                    "profile_version": 1,
+                    "selector": selector,
+                    "points": [{"id": "origin", "latitude": "0", "longitude": "0"}],
+                },
+            }
+        )
+        result = family.decode(content, pin, family.default_budgets)
+        parsed = parse_tabular_bytes(
+            result.content,
+            data_format=result.data_format,
+            media_type=result.media_type,
+            filename=result.filename,
+            limits=ParseLimits(),
+        )
+        if result.row_count != 1 or len(parsed.rows) != 1:
+            raise RuntimeError("trusted GRIB prewarm did not produce its fixed result")
+
+
+def _validate_descriptor_reader_fd(descriptor: int) -> None:
+    if type(descriptor) is not int or descriptor < 3:
+        raise OSError(errno.EINVAL, "invalid Reader descriptor")
+    info = os.fstat(descriptor)
+    mode = fcntl.fcntl(descriptor, fcntl.F_GETFL) & os.O_ACCMODE
+    if not stat.S_ISREG(info.st_mode) or mode != os.O_RDONLY or info.st_nlink != 0:
+        raise OSError(errno.EINVAL, "Reader descriptor must be private and read-only")
+
+
+def _validate_descriptor_reader_single_thread() -> None:
+    # prctl installs this filter on the calling thread. A native import that left a
+    # helper thread alive must refuse, rather than leave an unfiltered parser executor.
+    if len(list(Path("/proc/self/task").iterdir())) != 1:
+        raise OSError(errno.EPERM, "Reader prewarm left another thread active")
+
+
+def _validate_descriptor_reader_request(request: dict[str, Any], descriptor: int) -> None:
+    pin = request.get("reader_pin")
+    if (
+        request.get("operation") != "decode_and_parse"
+        or type(request.get("file_input_descriptor")) is not int
+        or request["file_input_descriptor"] != descriptor
+        or request.get("file_output_descriptor") is not None
+        or request.get("content_base64") is not None
+        or not isinstance(pin, dict)
+        or pin.get("family_id") != "meteorology.grib"
+        or pin.get("family_version") != "1.0.0"
+    ):
+        raise AcquisitionSecurityError("SANDBOX_FIELDS", "request differs from locked Reader lane")
+
+
 def worker_main(
-    *, prewarm: bool = False, descriptor_window_fds: tuple[int, int] | None = None
+    *,
+    prewarm: bool = False,
+    descriptor_window_fds: tuple[int, int] | None = None,
+    descriptor_reader_fd: int | None = None,
 ) -> int:
     """Read one bounded request from stdin and emit one canonical response.
 
@@ -1966,12 +2141,14 @@ def worker_main(
     recipe can repair, and the code says so.
     """
 
-    if prewarm or descriptor_window_fds is not None:
+    if prewarm or descriptor_window_fds is not None or descriptor_reader_fd is not None:
         # Before the request, never after it: the point of priming is that the import cost is
         # paid by an idle child rather than by a bound one. A failure here is a startup
         # failure and is reported on the same channel every other worker refusal uses.
         try:
-            if descriptor_window_fds is None:
+            if descriptor_reader_fd is not None:
+                _prime_descriptor_grib_reader()
+            elif descriptor_window_fds is None:
                 prime_parquet_support()
             else:
                 _prime_descriptor_window_csv()
@@ -1980,13 +2157,24 @@ def worker_main(
                 "SANDBOX_UNEXPECTED",
                 f"pre-warm raised {type(exc).__name__}, which no refusal maps",
             )
-        if descriptor_window_fds is not None:
+        if descriptor_reader_fd is not None or descriptor_window_fds is not None:
             try:
                 # Imports may open shared-object/cache descriptors. They are no longer needed
                 # once trusted prewarm finishes, so prove the locked phase retains only the
                 # stdio pipes and the two coordinator-selected file capabilities.
-                _close_child_descriptors_except({0, 1, 2, *descriptor_window_fds})
-                _validate_descriptor_window_fds(*descriptor_window_fds)
+                retained = (
+                    (descriptor_reader_fd,)
+                    if descriptor_reader_fd is not None
+                    else descriptor_window_fds
+                )
+                assert retained is not None
+                _close_child_descriptors_except({0, 1, 2, *retained})
+                if descriptor_reader_fd is not None:
+                    _validate_descriptor_reader_single_thread()
+                    _validate_descriptor_reader_fd(descriptor_reader_fd)
+                else:
+                    assert descriptor_window_fds is not None
+                    _validate_descriptor_window_fds(*descriptor_window_fds)
                 install_descriptor_window_seccomp()
             except OSError:
                 return _worker_error(
@@ -1995,7 +2183,7 @@ def worker_main(
                 )
         marker = (
             DESCRIPTOR_WINDOW_LOCKED_MARKER
-            if descriptor_window_fds is not None
+            if descriptor_window_fds is not None or descriptor_reader_fd is not None
             else WORKER_READY_MARKER
         )
         sys.stdout.buffer.write(marker)
@@ -2017,6 +2205,8 @@ def worker_main(
                 "SANDBOX_IPC_LIMIT",
                 "request exceeds the IPC byte budget for its operation",
             )
+        if descriptor_reader_fd is not None:
+            _validate_descriptor_reader_request(request, descriptor_reader_fd)
         response = _execute_worker_request(request)
         encoded = _ipc_json_bytes(response)
         if len(encoded) > MAX_RESPONSE_BYTES:
@@ -2063,7 +2253,11 @@ def _execute_worker_request(request: dict[str, Any]) -> dict[str, Any]:
         request = {**request, "generic_api_page": None}
     if request.get("operation") != "parse_streaming_csv_window" and "window_stream" not in request:
         request = {**request, "window_stream": None}
-    if request.get("operation") not in {"parse_evidence_file", "parse_streaming_csv_window"}:
+    if request.get("operation") not in {
+        "parse_evidence_file",
+        "parse_streaming_csv_window",
+        "decode_and_parse",
+    }:
         if (
             request.get("file_input_descriptor") is not None
             or request.get("file_output_descriptor") is not None
@@ -2076,6 +2270,10 @@ def _execute_worker_request(request: dict[str, Any]) -> dict[str, Any]:
             "file_input_descriptor": None,
             "file_output_descriptor": None,
         }
+    if request.get("operation") == "decode_and_parse":
+        request = {"file_input_descriptor": None, "file_output_descriptor": None, **request}
+        if request["file_output_descriptor"] is not None:
+            raise AcquisitionSecurityError("SANDBOX_FIELDS", "decode cannot carry output authority")
     expected = {
         "schema_version",
         "request_id",
@@ -2253,6 +2451,7 @@ def _execute_worker_request(request: dict[str, Any]) -> dict[str, Any]:
                     pagination=configuration.pagination,
                     limits=configuration.limits,
                     records_pointer=page.get("records_pointer"),
+                    max_output_bytes=GENERIC_API_PAGE_OUTPUT_BYTES,
                 )
                 if v3
                 else interpret_generic_api_page(
@@ -2260,6 +2459,7 @@ def _execute_worker_request(request: dict[str, Any]) -> dict[str, Any]:
                     headers,
                     response=configuration.response,
                     pagination=configuration.pagination,
+                    max_output_bytes=GENERIC_API_PAGE_OUTPUT_BYTES,
                 )
             )
         except Exception as error:
@@ -2653,7 +2853,7 @@ def _decode_in_worker(
             "SANDBOX_FIELDS",
             "decode request cannot choose the media type or filename the Reader declares",
         )
-    content = _request_content(request)
+    content = _reader_request_content(request)
     return _decode_content_in_worker(
         request,
         base,
@@ -2663,6 +2863,53 @@ def _decode_in_worker(
     )
 
 
+def _reader_request_content(request: dict[str, Any]) -> bytes:
+    descriptor = request.get("file_input_descriptor")
+    if descriptor is None:
+        return _request_content(request)
+    if request["content_base64"] is not None or type(descriptor) is not int or descriptor < 3:
+        raise AcquisitionSecurityError("SANDBOX_FILE", "invalid Reader input authority")
+    pin = _reader_pin_from_request(request["reader_pin"])
+    family = TOOLBOX.resolve(pin.family_id, pin.family_version)
+    budget = family.default_budgets.narrowed_by(request["reader_budgets"])
+    try:
+        info = os.fstat(descriptor)
+        mode = fcntl.fcntl(descriptor, fcntl.F_GETFL) & os.O_ACCMODE
+        if not stat.S_ISREG(info.st_mode) or mode != os.O_RDONLY or info.st_nlink != 0:
+            raise AcquisitionSecurityError(
+                "SANDBOX_FILE", "Reader input must be private read-only file"
+            )
+        if info.st_size > budget.max_input_bytes:
+            raise AcquisitionSecurityError("READER_BUDGET", "decode input exceeds Reader budget")
+        # CPython duplicates a descriptor with F_DUPFD_CLOEXEC, which the closed
+        # descriptor filter intentionally denies. Read the retained capability directly
+        # using already permitted syscalls, without adding another descriptor or authority.
+        offset = os.lseek(descriptor, 0, os.SEEK_CUR)
+        chunks: list[bytes] = []
+        remaining = info.st_size + 1
+        try:
+            os.lseek(descriptor, 0, os.SEEK_SET)
+            while remaining:
+                chunk = os.read(descriptor, min(remaining, 1024 * 1024))
+                if not chunk:
+                    break
+                chunks.append(chunk)
+                remaining -= len(chunk)
+        finally:
+            os.lseek(descriptor, offset, os.SEEK_SET)
+        content = b"".join(chunks)
+        after = os.fstat(descriptor)
+
+        def identity(value: os.stat_result) -> tuple[int, int, int, int, int]:
+            return (value.st_dev, value.st_ino, value.st_size, value.st_mtime_ns, value.st_ctime_ns)
+
+        if len(content) != info.st_size or identity(after) != identity(info):
+            raise AcquisitionSecurityError("SANDBOX_FILE", "Reader input changed during decode")
+        return content
+    except OSError as error:
+        raise AcquisitionSecurityError("SANDBOX_FILE", "Reader input unavailable") from error
+
+
 def _retrieve_decode_in_worker(
     request: dict[str, Any],
     base: dict[str, Any],
@@ -4585,6 +4832,11 @@ def _policy_document(operation: str) -> dict[str, Any]:
     # of that authority, which is the thing an attestation should be bound to.
     if operation in READER_PERFORMING_OPERATIONS:
         policy["reader_contract"] = READER_CONTRACT_VERSION
+    if operation == "decode_and_parse":
+        policy["reader_input"] = "inline-or-private-readonly-unlinked-descriptor-v1"
+        policy["reader_descriptor_boundary"] = {
+            "meteorology.grib@1.0.0": "trusted-prewarm-descriptor-only-seccomp-v1"
+        }
     return policy
 
 
@@ -7156,4 +7408,10 @@ if __name__ == "__main__":
         except ValueError:
             raise SystemExit(2) from None
         raise SystemExit(worker_main(descriptor_window_fds=descriptors))
+    if len(sys.argv) == 3 and sys.argv[1] == "--descriptor-reader-worker":
+        try:
+            descriptor = int(sys.argv[2], 10)
+        except ValueError:
+            raise SystemExit(2) from None
+        raise SystemExit(worker_main(descriptor_reader_fd=descriptor))
     raise SystemExit(2)