chtypes 2.0.0.dev11__tar.gz → 2.0.0.dev12__tar.gz

{chtypes-2.0.0.dev11 → chtypes-2.0.0.dev12}/CHANGELOG.md

@@ -8,7 +8,9 @@ The four bindings in this repository are released together and give one answer,
 
 ### Added
 
+- Tests: the shared revocation-statement table (`tests/fixtures/revocation-statement/cases.json`) runs against `verify_revocation` and `check_installed`: `superseded_by` must be a string (`""` = no fix), and the first verifying revocation in a listing wins. No behavior changed.
 - `column_defaults` is covered by unit tests (ABI v2): the exact `insert_options` JSON for a UTF-8 value, a non-UTF-8 value (`value_b64`), an empty value (sent as `""`) and two entries in order, and the decode of a `default_override` cell (`Source.DEFAULT_OVERRIDE`, stored). The batches guide gains a usage section. No behavior changed.
+- Docs: `docs/guides/batches.md` shows how a `column_defaults` value is read, in the body format's own field encoding (CSV quoting, TSV escapes, a JSON string or number), with a value that doesn't read refused at the call before any body byte.
 - `CODE_DECLINED` (ABI v2, -2): the code a declined row, outcome or verdict carries, generated from the description and exported from `chtypes`, as `bindings-v1.md` section 3 promises.
 - `CallError.reason` (ABI v2): the error's catalog id, read through `chs_error_reason` and passed through verbatim (ASCII, empty if none), on all four call classes.
 - Spec batch 2, the options documents (ABI v2; moves the dev fingerprint): keyword-only `insertable_columns`, `column_defaults` (a sequence of `ColumnDefault`), `verdict_basis` (`VerdictBasis`: `INPUT`, `EXPORT`) and `defer_defaults` (`DeferDefaults`: `NONE`, `SERVER`) build `insert_options` for `Schema.row`, `Schema.rows`, `Schema.parse_block` and `Filter.rows`; `param_encoding` (`ParamEncoding`: `ESCAPED`, `RAW`) builds `filter_options` for `Schema.compile_filter`; `posture` (`Posture`: `CREATE`, `LOAD`) builds `schema_options` for `Library.compile_table`. `insert_options_extra`, `filter_options_extra` and `schema_options_extra` pass a member the binding has no name for through as JSON text, so the library, which refuses an unknown key (rule r1), is the only judge. A document with no member set is not sent. `Filter.rows` takes `columns` (`chs_filter_eval_body` takes the column list).
@@ -34,7 +36,8 @@ The four bindings in this repository are released together and give one answer,
 
 ### Changed
 
-- **Breaking (ABI v2 dev): `BatchResult.spans` is a tuple of `Span | None`**, one entry per `rows` entry and index-aligned with it: `row_spans` carries `null` for a row the export did not carry (a skipped row, or a row a filter cut), which reads as `None`. It was the exported rows' spans only.
+- **Breaking (vs the 2.0.0-dev releases):** the batch preview's default `doc_flags` is now 0 (the lean result): `Schema.rows` without `doc_flags=` returns no per-value documents. Pass `doc_flags=DocFlags.ALL` explicitly to get every column's per-value documents.
+- **Breaking (ABI v2 dev): `BatchResult.spans` is a tuple of `Span | None`**, one entry per `rows` entry and index-aligned with it: `row_spans` carries `null` for a row the export did not carry (a skipped row, or a row a filter cut), which reads as `None`. It was the exported rows' spans only. Code that reads `span.off` from every entry raises `AttributeError` on a `None` entry: check each entry before reading it.
 - **Breaking (ABI v2 dev): `AtMergeEntry.row` is `AtMergeEntry.part_row`**, the document's `part_row`.
 - A document's `code` can be -2 (`CHS_CODE_DECLINED`) for a row, batch, filter result or verdict this build declined, and every decline names its `reason`.
 - The refusal of a settings, query-parameter, default or server-settings value that is not valid UTF-8 is gone (it is the `{"b64": ...}` form now); a key that is not valid UTF-8, a macro and a server timezone are still refused, as a `UsageError` naming the input.
@@ -79,6 +82,7 @@ The four bindings in this repository are released together and give one answer,
 - **Enforcement gate lifted** for `26.8` on `darwin-arm64`.
 - **Enforcement gate lifted** for `26.9` on `darwin-arm64`.
 - `docs/limitations.md` is re-checked against artifact build `20261012.142240` (v2-dev.13), the library half measured on that build: functions that need a server, in a column expression, and a body the server may read serially or in parallel under a non-zero error budget are listed under "Declined by design"; the known divergence for a column expression that calls a function needing a server is retired, since that build declines those schemas (on 26.8 and 26.9 the `AI*` and `ai*` functions compile, as the page says); and the `lossy` raw-NUL TSV gap is marked fixed on that build and still present in the 1.0 libraries. `docs/guides/filters.md` gains a section on subcolumn filters and a JSON value typed by its batch.
+- The ABI v2 spec now states the INSERT-time settings layering the library already uses, since artifact build `20261021.025722` (#632): at the INSERT, the schema's own settings (the compile profile) reach only the parse, minus a type gate, and every expression the INSERT evaluates and its limits (`max_partitions_per_insert_block` among them) run under the call's own settings, the server profile and the setup defaults only. A setting your INSERT session carries for those goes in the call's settings or the server profile (`docs/guides/settings.md`). No API change.
 
 ### Removed
 
@@ -87,6 +91,8 @@ The four bindings in this repository are released together and give one answer,
 
 ### Fixed
 
+- An offline lookup no longer reads every cached blob once per pre-seeded entry (#690). For each `index.json` entry with no record, the lookup read every blob of the layout, layers included, to find its signature, so entries it could not install (a signature not present, or one no trusted key verifies) cost entries × cache bytes on every lookup: about 4.9 s for 27 such entries over 1.27 GB (`measured`, darwin-arm64). The signature referrers are now read once per lookup, and never a layer, because a blob larger than `limits.manifest_bytes` cannot be a manifest. Installed entries were already skipped. The build that is installed is verified exactly as before: its manifest against its descriptor, its whole layer against its digest, and its signature, platform, request and fingerprint. A unit test counts the blob reads of one lookup that installs nothing and holds them to at most two per blob and five per entry, with no layer read, at 2 and at 20 entries alike.
+- Docs: `docs/guides/settings.md` names the ABI v2 admission time budget `chtypes_default_eval_cpu_nanos`, the evaluating thread's CPU time. Its examples used the 1.x name `chtypes_default_eval_wall_nanos`, which v2 refuses with `CHS_INVALID_ARGUMENT`.
 - A response body cut off part way (shorter than its Content-Length, a reset, a TLS close) is retried like a 5xx and, once the retry table is spent, is `CHTYPES_SOURCE_UNREACHABLE`; only an oversize body is `CHTYPES_ARTIFACT_CORRUPT`, and it is never retried (#641). Python took a body that ended before its Content-Length as the whole body and failed its digest check as corrupt without a retry; now a short body, or `http.client.IncompleteRead`, is retried and ends as unreachable. The rule is `docs/guides/fetch-v1.md` section 7, and the new `body-cut-off-then-ok` and `body-cut-off-exhausted` conformance cases hold every binding to it.
 - A chunked response body cut off before its terminating zero-length chunk is held to the same rule (#644): the new `body-cut-off-chunked-then-ok` and `body-cut-off-chunked-exhausted` conformance cases cut one on the layer blob GET, and this binding already retried it and ended as `CHTYPES_SOURCE_UNREACHABLE` once the retry table was spent. No behavior change; the case now holds it.
 - A settings or query-parameter key or value that is not valid UTF-8 is refused as a usage error; it was silently replaced with U+FFFD, so the library saw a different value. A string holding a lone surrogate (what `surrogateescape` makes of undecodable bytes) was sent as a `\udXXX` escape; the same inputs are refused now, for the per-call zone, the setup defaults and the server profile too. Valid UTF-8 serializes as before.

{chtypes-2.0.0.dev11 → chtypes-2.0.0.dev12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: chtypes
-Version: 2.0.0.dev11
+Version: 2.0.0.dev12
 Summary: Python binding for the chtypes artifacts: ClickHouse's own type system, per version, behind the frozen chs_* C ABI
 Project-URL: Homepage, https://github.com/wave-rf/chtypes
 Project-URL: Repository, https://github.com/wave-rf/chtypes
@@ -43,7 +43,7 @@ uv add "chtypes>=2.0.0.dev0"            # or: pip install "chtypes>=2.0.0.dev0"
 
 ```python
 import chtypes
-from chtypes import Format, Registry
+from chtypes import DocFlags, Format, Registry
 
 chtypes.setup(timezone="UTC")  # once, before the first open (optional)
 registry = Registry(autofetch=True)
@@ -51,7 +51,7 @@ library = registry.for_version("26.8")  # a release line or an exact version
 
 ddl = b"CREATE TABLE t (x UInt8, ts DateTime DEFAULT now()) ENGINE = Memory"
 with library.compile_table(ddl) as schema:
-    batch = schema.rows(Format.JSON_EACH_ROW, b'{"x":256}\n')
+    batch = schema.rows(Format.JSON_EACH_ROW, b'{"x":256}\n', doc_flags=DocFlags.ALL)
 
 row = batch.rows[0]
 print(batch.outcome)  # accepted

{chtypes-2.0.0.dev11 → chtypes-2.0.0.dev12}/README.md

@@ -24,7 +24,7 @@ uv add "chtypes>=2.0.0.dev0"            # or: pip install "chtypes>=2.0.0.dev0"
 
 ```python
 import chtypes
-from chtypes import Format, Registry
+from chtypes import DocFlags, Format, Registry
 
 chtypes.setup(timezone="UTC")  # once, before the first open (optional)
 registry = Registry(autofetch=True)
@@ -32,7 +32,7 @@ library = registry.for_version("26.8")  # a release line or an exact version
 
 ddl = b"CREATE TABLE t (x UInt8, ts DateTime DEFAULT now()) ENGINE = Memory"
 with library.compile_table(ddl) as schema:
-    batch = schema.rows(Format.JSON_EACH_ROW, b'{"x":256}\n')
+    batch = schema.rows(Format.JSON_EACH_ROW, b'{"x":256}\n', doc_flags=DocFlags.ALL)
 
 row = batch.rows[0]
 print(batch.outcome)  # accepted

{chtypes-2.0.0.dev11 → chtypes-2.0.0.dev12}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "chtypes"
-version = "2.0.0.dev11"
+version = "2.0.0.dev12"
 description = "Python binding for the chtypes artifacts: ClickHouse's own type system, per version, behind the frozen chs_* C ABI"
 readme = "README.md"
 license = "Apache-2.0"

{chtypes-2.0.0.dev11 → chtypes-2.0.0.dev12}/src/chtypes/__init__.py

@@ -13,14 +13,15 @@ C ABI. Nothing here reimplements a coercion rule, which is why the answers are
 exact by construction rather than approximately right.
 
     import chtypes
-    from chtypes import Format, Registry
+    from chtypes import DocFlags, Format, Registry
 
     chtypes.setup(timezone="UTC")                # once, before the first open
     registry = Registry(autofetch=True)
     library = registry.for_version("26.8")       # a release line or an exact version
     ddl = b"CREATE TABLE t (ts DateTime, seq UInt8) ENGINE = Memory"
     with library.compile_table(ddl) as schema:
-        batch = schema.rows(Format.JSON_EACH_ROW, b'{"ts":"2026-01-15 10:30:00","seq":256}\\n')
+        body = b'{"ts":"2026-01-15 10:30:00","seq":256}\\n'
+        batch = schema.rows(Format.JSON_EACH_ROW, body, doc_flags=DocFlags.ALL)
 
     batch.outcome        # Outcome.ACCEPTED: the insert would succeed
     batch.transformed    # the changes ClickHouse makes and reports as success

{chtypes-2.0.0.dev11 → chtypes-2.0.0.dev12}/src/chtypes/_ocifetch/_ensure.py

@@ -1296,6 +1296,48 @@ def _ensure(
         raise _translate_transport_error(e) from e
 
 
+def _local_bundle_referrers(
+    blobs_dir: Path, skip: Collection[str], known: dict[str, bytes] | None = None
+) -> dict[str, list[tuple[str, dict]]]:
+    """Each subject manifest digest's signature referrer manifests among
+    `blobs_dir`'s blobs, as (blob name, parsed manifest), in listing order.
+    A referrer need not be listed in `index.json`, so every blob that could
+    be one is read, once per lookup rather than once per entry. A blob named
+    in `skip` (the caller's candidate layers) and a blob larger than
+    `limits.manifest_bytes`, which cannot be a manifest, are never read, so
+    no layer is read here, and a blob in `known` (bytes the caller already
+    read, by name) is not read again (public issue #690)."""
+    out: dict[str, list[tuple[str, dict]]] = {}
+    try:
+        blob_paths = list(blobs_dir.iterdir())
+    except OSError:
+        return out
+    for blob_path in blob_paths:
+        if blob_path.name in skip:
+            continue
+        try:
+            body = known.get(blob_path.name) if known is not None else None
+            if body is None:
+                if blob_path.stat().st_size > C.MANIFEST_MAX_BYTES:
+                    continue
+                body = blob_path.read_bytes()
+            doc = json.loads(body)
+        except (ValueError, OSError, UnicodeDecodeError):
+            continue
+        if not isinstance(doc, dict) or doc.get("mediaType") != C.MEDIA_TYPE_MANIFEST:
+            continue
+        subject = doc.get("subject") or {}
+        subject_digest = subject.get("digest") if isinstance(subject, dict) else None
+        if not isinstance(subject_digest, str):
+            continue
+        # A signature referrer only: a revocation of the manifest carries the same
+        # bundle media type, and is not the signature of the artifact.
+        if doc.get("artifactType") != C.MEDIA_TYPE_BUNDLE:
+            continue
+        out.setdefault(subject_digest, []).append((blob_path.name, doc))
+    return out
+
+
 def _verify_and_install_from_local_blobs(
     *,
     source_root: Path,
@@ -1303,6 +1345,7 @@ def _verify_and_install_from_local_blobs(
     cache_write_root: Path,
     trusted_keys: tuple[TrustedKey, ...],
     visible_only: bool = False,
+    referrers: dict[str, list[tuple[str, dict]]] | None = None,
 ) -> VerifiedRecord | None:
     """docs/guides/fetch-v1.md §1/§6: "A pre-seeded layout has entries in
     `index.json` with no corresponding `unpacked/` directory yet; the first
@@ -1328,23 +1371,12 @@ def _verify_and_install_from_local_blobs(
     except (ValueError, OSError, FetchError):
         return None
 
+    if referrers is None:
+        referrers = _local_bundle_referrers(blobs_dir, {parse_digest(layer_desc.digest)})
     verified = None
     bundle_digest = None
     bundle_manifest_digest = None
-    for blob_path in blobs_dir.iterdir():
-        try:
-            doc = json.loads(blob_path.read_bytes())
-        except (ValueError, OSError, UnicodeDecodeError):
-            continue
-        if not isinstance(doc, dict) or doc.get("mediaType") != C.MEDIA_TYPE_MANIFEST:
-            continue
-        subject = doc.get("subject") or {}
-        if subject.get("digest") != manifest_digest:
-            continue
-        # A signature referrer only: a revocation of the manifest carries the same
-        # bundle media type, and is not the signature of the artifact.
-        if doc.get("artifactType") != C.MEDIA_TYPE_BUNDLE:
-            continue
+    for referrer_name, doc in referrers.get(manifest_digest, ()):
         try:
             referrer_layer = manifest_single_layer(doc, expected_media_type=C.MEDIA_TYPE_BUNDLE)
             bundle_bytes = (blobs_dir / parse_digest(referrer_layer.digest)).read_bytes()
@@ -1354,7 +1386,7 @@ def _verify_and_install_from_local_blobs(
         if candidate is not None:
             verified = candidate
             bundle_digest = referrer_layer.digest
-            bundle_manifest_digest = "sha256:" + blob_path.name
+            bundle_manifest_digest = "sha256:" + referrer_name
             break
     if verified is None:
         return None
@@ -1422,7 +1454,9 @@ def _verify_preseeded_entries(roots: Sequence[Path], trusted_keys: tuple[Trusted
     anywhere yet, and verifies+unpacks it from that root's OWN local blobs,
     writing the result into `roots[0]` (the writable cache) regardless of
     which root — including a read-only system directory — the blobs came
-    from."""
+    from. An entry that already has a record is passed over without reading
+    a blob, and each root's signature referrers are read once, never by
+    reading a candidate's layer (public issue #690)."""
     cache_write_root = roots[0]
     already_verified = {record.manifest for _dir, record in list_verified_records(roots)}
     for root in roots:
@@ -1431,16 +1465,36 @@ def _verify_preseeded_entries(roots: Sequence[Path], trusted_keys: tuple[Trusted
             index_doc = json.loads(index_path.read_text(encoding="utf-8"))
         except (OSError, ValueError):
             continue
+        blobs_dir = root / "blobs" / "sha256"
+        candidates: list[str] = []
+        candidate_layers: set[str] = set()
+        manifests_read: dict[str, bytes] = {}
         for m in index_doc.get("manifests") or []:
             digest = m.get("digest")
             if not digest or digest in already_verified:
                 continue
+            candidates.append(digest)
+            try:
+                manifest_hex = parse_digest(digest)
+                manifest_bytes = (blobs_dir / manifest_hex).read_bytes()
+                manifests_read[manifest_hex] = manifest_bytes
+                layer = manifest_layer_descriptor(json.loads(manifest_bytes))
+                candidate_layers.add(parse_digest(layer.digest))
+            except (ValueError, OSError, FetchError):
+                continue
+        if not candidates:
+            continue
+        referrers = _local_bundle_referrers(blobs_dir, candidate_layers, manifests_read)
+        for digest in candidates:
+            if digest in already_verified:
+                continue
             record = _verify_and_install_from_local_blobs(
                 source_root=root,
                 manifest_digest=digest,
                 cache_write_root=cache_write_root,
                 trusted_keys=trusted_keys,
                 visible_only=True,
+                referrers=referrers,
             )
             if record is not None:
                 already_verified.add(digest)

{chtypes-2.0.0.dev11 → chtypes-2.0.0.dev12}/src/chtypes/library.py

@@ -54,6 +54,8 @@ __all__ = ["Block", "Filter", "Library", "Schema", "Server", "open_unverified"]
 # One image per file, process-wide: keyed on the realpath's device and inode, so
 # two registries, two spellings or a hardlink of one artifact share one image
 # and one Library. Guarded by the setup lock, which an open holds throughout.
+# The batch preview's default `doc_flags`: 0, the lean result (no per-value documents).
+_LEAN_DOC_FLAGS = DocFlags(0)
 _IMAGES: dict[tuple[int, int], Library] = {}
 
 
@@ -212,7 +214,8 @@ class Filter(_Handle):
         insert_options_extra: Mapping[str, str] | None = None,
     ) -> FilterResult:
         """Evaluate over a body. `settings` and `session_timezone` are the body's
-        PARSE settings only: the filter's own zone was fixed when it compiled.
+        INSERT settings: they govern its parse and the expressions the INSERT
+        evaluates, never the WHERE, whose zone was fixed when the filter compiled.
         `columns` is the INSERT column list and the keywords after it the
         `insert_options` members, as for `Schema.rows`."""
         payload = body_bytes(body)
@@ -304,11 +307,14 @@ class Schema(_Handle):
         insert_options_extra: Mapping[str, str] | None = None,
         row_filter: Filter | None = None,
         export: Format | int | None = None,
-        doc_flags: DocFlags = DocFlags.ALL,
+        doc_flags: DocFlags = _LEAN_DOC_FLAGS,
     ) -> BatchResult:
         """A whole body. `export` asks for the accepted rows serialized in that
         format (`payload`, and `spans`); to insert a body whose rows carry a
-        `default_generated` column, insert `payload`, never the original input."""
+        `default_generated` column, insert `payload`, never the original input.
+
+        `doc_flags` chooses the document groups; absent means 0, the lean result
+        (no per-value documents). Pass `DocFlags.ALL` for every group."""
         payload = body_bytes(body)
         settings_bytes = settings_json(settings, session_timezone)
         columns_bytes = columns_json(columns)

{chtypes-2.0.0.dev11 → chtypes-2.0.0.dev12}/tests/abi2/test_public_library.py

@@ -158,10 +158,27 @@ def test_rows_marshals_columns_filter_export_and_doc_flags(lib, monkeypatch) ->
     assert fmt == 0 and body == b'{"a":1}\n' and settings is None and options is None
     assert json.loads(columns) == [{"name": "a"}, {"name_b64": "/w=="}]
     assert handle is not None and export == int(Format.CSV) and flags == 3
-    assert plain[5] is None and plain[6] == -1 and plain[7] == 7  # no export, all groups
+    assert plain[5] is None and plain[6] == -1 and plain[7] == 0  # no export, lean (0)
     assert plain[2] == b"1\n"  # a bytes-like body is copied to bytes
 
 
+def test_rows_doc_flags_default_is_lean_and_explicit_all_reaches_the_call(lib, monkeypatch) -> None:
+    calls: list[tuple] = []
+    real = lib._api.preview_batch
+
+    def spy(*args):
+        calls.append(args)
+        return real(*args)
+
+    monkeypatch.setattr(lib._api, "preview_batch", spy)
+    with lib.compile_table(CREATE) as schema:
+        schema.rows(Format.CSV, b"1\n")
+        schema.rows(Format.CSV, b"1\n", export=Format.CSV)
+        schema.rows(Format.CSV, b"1\n", doc_flags=DocFlags.ALL)
+    # Argument 7 of chs_preview_batch is doc_flags, as the real stub call received it.
+    assert [c[7] for c in calls] == [0, 0, int(DocFlags.ALL)]
+
+
 def test_no_export_means_no_payload(lib) -> None:
     with lib.compile_table(CREATE) as schema:
         assert schema.rows(Format.CSV, b"1\n").payload is None

chtypes-2.0.0.dev12/tests/ocifetch/test_preseed_scan.py

@@ -0,0 +1,161 @@
+"""Public issue #690: an offline lookup reads a bounded number of blobs,
+whatever the number of builds the cache holds.
+
+The verdicts come from the blob reads the lookup itself made (every
+`Path.read_bytes` under the layout's `blobs/sha256/`), never from a timing.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import shutil
+from collections import Counter
+from collections.abc import Callable
+from pathlib import Path
+
+import pytest
+
+from chtypes._ocifetch._dsse import TrustedKey
+from chtypes._ocifetch._ensure import Options, Request, resolve_installed
+
+FIXTURES = Path(__file__).resolve().parents[3] / "tests" / "fixtures" / "fetch-v1"
+LAYOUT = FIXTURES / "layouts" / "offline-hit"
+# The offline-hit layout's one build: 26.1.1.1 on linux-arm64.
+BUILD_MANIFEST = "sha256:b461315c30649fec6d9cc988330ed3081ffd6a2a45aa21acda6ff212413f8637"
+BUILD_LAYER_HEX = "df72d6f06f8ed738ecdd8e080dbd71244b66024daee20c85f53d3f822a49463b"
+BUNDLE_REF_HEX = "887b88511be68e82d653c734f57fc0e2c9c2e64bda4f23f25fe0ae1648f6244b"
+
+
+def _key(name: str) -> TrustedKey:
+    public = bytes.fromhex((FIXTURES / name / "public.hex").read_text().strip())
+    return TrustedKey(keyid=hashlib.sha256(public).hexdigest()[:16], public_key=public)
+
+
+def _seed_clones(tmp_path: Path, n: int) -> tuple[Path, list[str]]:
+    """The offline-hit layout plus n-1 more pre-seeded entries of the same
+    signed build: each its own manifest (the original plus one annotation),
+    its own signature referrer manifest naming it, and its own copy of the
+    bundle (the original bytes plus trailing whitespace, which the signature
+    does not cover). The layer stays the one signed layer."""
+    if not LAYOUT.is_dir():
+        pytest.skip(f"fetch fixtures not available at {LAYOUT}")
+    cache = tmp_path / f"cache-{n}"
+    shutil.copytree(LAYOUT, cache)
+    blobs = cache / "blobs" / "sha256"
+
+    def put(b: bytes) -> tuple[str, int]:
+        h = hashlib.sha256(b).hexdigest()
+        (blobs / h).write_bytes(b)
+        return "sha256:" + h, len(b)
+
+    index = json.loads((cache / "index.json").read_text())
+    first = index["manifests"][0]
+    digests = [first["digest"]]
+    manifest = json.loads((blobs / BUILD_MANIFEST.removeprefix("sha256:")).read_bytes())
+    ref = json.loads((blobs / BUNDLE_REF_HEX).read_bytes())
+    bundle_desc = ref["layers"][0]
+    bundle = (blobs / bundle_desc["digest"].removeprefix("sha256:")).read_bytes()
+    for i in range(1, n):
+        manifest["annotations"] = {"test.clone": str(i)}
+        md, msize = put(json.dumps(manifest).encode())
+        bd, bsize = put(bundle + b" " * i)
+        bundle_desc["digest"], bundle_desc["size"] = bd, bsize
+        ref["subject"] = {
+            "mediaType": "application/vnd.oci.image.manifest.v1+json",
+            "digest": md,
+            "size": msize,
+        }
+        put(json.dumps(ref).encode())
+        entry = {k: v for k, v in first.items() if k != "annotations"}
+        entry["digest"], entry["size"] = md, msize
+        index["manifests"].append(entry)
+        digests.append(md)
+    (cache / "index.json").write_text(json.dumps(index))
+    return cache, digests
+
+
+def _blob_reads(monkeypatch: pytest.MonkeyPatch, cache: Path, fn: Callable[[], object]) -> Counter:
+    """Every blob of `cache` read while fn runs, by name."""
+    reads: Counter = Counter()
+    blobs = cache / "blobs" / "sha256"
+    real = Path.read_bytes
+
+    def counting(self: Path) -> bytes:
+        data = real(self)
+        if self.parent == blobs:
+            reads[self.name] += 1
+        return data
+
+    with monkeypatch.context() as m:
+        m.setattr(Path, "read_bytes", counting)
+        fn()
+    return reads
+
+
+def _options(cache: Path, key: str) -> Options:
+    return Options(cache_dir=cache, system_dirs=(), trusted_keys=(_key(key),))
+
+
+@pytest.mark.parametrize("n", [2, 20])
+def test_lookup_over_unverifiable_entries_reads_no_layer(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch, n: int
+) -> None:
+    """N pre-seeded entries that no lookup installs (their signature verifies
+    under no trusted key), so every lookup meets them again: no layer is
+    read, no blob more than twice, and at most 5 reads per entry plus 2, at
+    N=2 and N=20 alike. The old scan read every blob of the layout, the layer
+    included, once per entry."""
+    cache, _ = _seed_clones(tmp_path, n)
+    options = _options(cache, "other-key")
+    results: list[object] = []
+    reads = _blob_reads(
+        monkeypatch,
+        cache,
+        lambda: results.append(resolve_installed(Request("26.1"), "linux-arm64", options)),
+    )
+    assert results == [None]
+    total, most = sum(reads.values()), max(reads.values(), default=0)
+    print(
+        f"N={n}: {total} blob reads, {len(reads)} distinct, most {most}, "
+        f"layer {reads[BUILD_LAYER_HEX]}"
+    )
+    assert reads[BUILD_LAYER_HEX] == 0, f"N={n}: the lookup read the layer"
+    assert most <= 2, f"N={n}: one blob was read {most} times in one lookup"
+    assert total <= 5 * n + 2, f"N={n}: {total} blob reads in one lookup"
+
+
+@pytest.mark.parametrize("n", [2, 20])
+def test_miss_over_installed_builds_reads_no_blob(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch, n: int
+) -> None:
+    """The consumer's case: every entry is installed, so none is pre-seeded
+    (fetch-v1.md §6), and a miss reads no blob at all."""
+    cache, digests = _seed_clones(tmp_path, n)
+    options = _options(cache, "test-key")
+    # The first lookup installs every pre-seeded entry of the build.
+    first = resolve_installed(Request("26.1"), "linux-arm64", options)
+    assert first is not None
+    assert all(
+        (cache / "unpacked" / "sha256" / d.removeprefix("sha256:")).is_dir() for d in digests
+    )
+    results: list[object] = []
+    reads = _blob_reads(
+        monkeypatch,
+        cache,
+        lambda: results.append(resolve_installed(Request("26.3"), "linux-arm64", options)),
+    )
+    assert results == [None]
+    assert sum(reads.values()) == 0, f"N={n}: the miss read {dict(reads)}"
+
+
+def test_preseeded_install_still_verifies_the_layer(tmp_path: Path) -> None:
+    """The install is as strict as before: with the layer's bytes altered,
+    nothing is installed."""
+    cache, digests = _seed_clones(tmp_path, 20)
+    layer = cache / "blobs" / "sha256" / BUILD_LAYER_HEX
+    b = bytearray(layer.read_bytes())
+    b[-1] ^= 0xFF
+    layer.write_bytes(bytes(b))
+    assert resolve_installed(Request("26.1"), "linux-arm64", _options(cache, "test-key")) is None
+    assert not (cache / "unpacked" / "sha256" / digests[0].removeprefix("sha256:")).exists()

chtypes-2.0.0.dev12/tests/ocifetch/test_revocation_statement.py

@@ -0,0 +1,240 @@
+"""The shared revocation-statement table (tests/fixtures/revocation-statement/
+cases.json), generated by the delivery side's registry tests and read by all four
+bindings: which signed revocation statements count (docs/guides/fetch-v1.md,
+"Build-level revocation", rule 5), and that the first verifying revocation in a
+listing wins. The statement rows call `verify_revocation`; the listing rows drive
+the real candidate loop through `check_installed` against a local registry."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import threading
+from collections.abc import Iterator
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+
+import pytest
+
+from chtypes._ocifetch import _channel
+from chtypes._ocifetch import _constants as C
+from chtypes._ocifetch._dsse import TrustedKey
+from chtypes._ocifetch._ensure import (
+    Options,
+    Resolved,
+    check_installed,
+    reset_revocation_memo_for_tests,
+)
+from chtypes._ocifetch._errors import ArtifactRevokedError
+from chtypes._ocifetch._layout import resolve_cache_root
+from chtypes._ocifetch._revocation import Subject, revoked_text, verify_revocation
+
+DIR = Path(__file__).resolve().parents[3] / "tests" / "fixtures" / "revocation-statement"
+TABLE = DIR / "cases.json"
+FIELDS = ("id", "url", "class", "summary", "superseded_by", "revoked_at")
+
+if not TABLE.is_file():
+    pytest.skip(
+        f"SKIPPED: the shared revocation-statement table is not beside this checkout ({TABLE})",
+        allow_module_level=True,
+    )
+
+_DOC = json.loads(TABLE.read_text(encoding="utf-8"))
+_BUILD = json.loads((DIR / "build.predicate.json").read_text(encoding="utf-8"))
+_KEY = bytes.fromhex((DIR / "key.pub.hex").read_text().strip())
+_KEYS = (TrustedKey(keyid=hashlib.sha256(_KEY).hexdigest()[:16], public_key=_KEY),)
+
+
+def _advisory(a) -> dict:  # noqa: ANN001
+    return {
+        "id": a.id,
+        "url": a.url,
+        "class": a.class_,
+        "summary": a.summary,
+        "superseded_by": a.superseded_by,
+        "revoked_at": a.revoked_at,
+    }
+
+
+def _digest(data: bytes) -> str:
+    return "sha256:" + hashlib.sha256(data).hexdigest()
+
+
+@pytest.fixture
+def prod_v2() -> Iterator[None]:
+    restore = _channel.use_prod_v2_for_tests()
+    reset_revocation_memo_for_tests()
+    try:
+        yield
+    finally:
+        reset_revocation_memo_for_tests()
+        restore()
+
+
+def test_the_table_has_rows() -> None:
+    assert _DOC["cases"] and _DOC["listings"], "an empty table would check nothing"
+    assert any(c["expect"] == "verifies" for c in _DOC["cases"])
+    assert any(r["expect"] == "revoked" for r in _DOC["listings"])
+
+
+@pytest.mark.parametrize("case", _DOC["cases"], ids=lambda c: c["name"])
+def test_statement_case(case: dict) -> None:
+    sub = Subject(
+        manifest=_BUILD["manifest"],
+        layer=_BUILD["layer"],
+        predicate=_BUILD["predicate"],
+        version="26.8.15.10",
+        platform="linux-amd64",
+        build="20260101.000000",
+        request="26.8",
+        dir=None,
+        writable=False,
+    )
+    rev, reason = verify_revocation((DIR / case["bundle"]).read_bytes(), _KEYS, sub)
+    if case["expect"] == "ignored":
+        assert rev is None, f"{case['name']} verifies, want it refused"
+        assert reason, "refused with no reason"
+        return
+    assert case["expect"] == "verifies"
+    assert rev is not None, f"{case['name']} does not verify ({reason})"
+    assert len(rev.advisories) == 1
+    assert _advisory(rev.advisories[0]) == case["expect_advisory"]
+    has_fix = case["expect_advisory"]["superseded_by"] != ""
+    assert has_fix == ("; superseded by build " in revoked_text(sub, rev)), revoked_text(sub, rev)
+
+
+class _Registry:
+    """Serves one build's referrers listing in order, each referrer's manifest and
+    its bundle; `served` maps a bundle file to the manifest digest it is served
+    under."""
+
+    def __init__(self, referrers: list[str]) -> None:
+        objects: dict[str, bytes] = {}
+        self.served: dict[str, str] = {}
+        manifests = [
+            {
+                "mediaType": C.MEDIA_TYPE_MANIFEST,
+                "digest": _digest(b"signature"),
+                "size": 9,
+                "artifactType": C.MEDIA_TYPE_BUNDLE,  # the listing must name a signature
+            }
+        ]
+        for name in referrers:
+            bundle = (DIR / name).read_bytes()
+            layer = {
+                "mediaType": C.MEDIA_TYPE_BUNDLE,
+                "digest": _digest(bundle),
+                "size": len(bundle),
+            }
+            body = json.dumps(
+                {
+                    "schemaVersion": 2,
+                    "mediaType": C.MEDIA_TYPE_MANIFEST,
+                    "artifactType": C.REVOCATION_ARTIFACT_TYPE,
+                    "config": {
+                        "mediaType": "application/vnd.oci.empty.v1+json",
+                        "digest": _digest(b"{}"),
+                        "size": 2,
+                    },
+                    "layers": [layer],
+                }
+            ).encode()
+            digest = _digest(body)
+            self.served[name] = digest
+            objects["/manifests/" + digest] = body
+            objects["/blobs/" + layer["digest"]] = bundle
+            manifests.append(
+                {
+                    "mediaType": C.MEDIA_TYPE_MANIFEST,
+                    "digest": digest,
+                    "size": len(body),
+                    "artifactType": C.REVOCATION_ARTIFACT_TYPE,
+                }
+            )
+        objects["/referrers/" + _BUILD["manifest"]] = json.dumps(
+            {"schemaVersion": 2, "mediaType": C.MEDIA_TYPE_INDEX, "manifests": manifests}
+        ).encode()
+
+        class Handler(BaseHTTPRequestHandler):
+            def log_message(self, *args) -> None:  # noqa: D102
+                pass
+
+            def do_GET(self) -> None:  # noqa: N802
+                for suffix, body in objects.items():
+                    if self.path.endswith(suffix):
+                        self.send_response(200)
+                        self.send_header("Content-Length", str(len(body)))
+                        self.end_headers()
+                        self.wfile.write(body)
+                        return
+                self.send_response(404)
+                self.send_header("Content-Length", "0")
+                self.end_headers()
+
+        self._httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+        self._httpd.daemon_threads = True
+        threading.Thread(target=self._httpd.serve_forever, daemon=True).start()
+
+    @property
+    def base(self) -> str:
+        host, port = self._httpd.server_address[:2]
+        return f"http://{host}:{port}/chtypes/v2"
+
+    def stop(self) -> None:
+        self._httpd.shutdown()
+        self._httpd.server_close()
+
+
+@pytest.mark.parametrize("listing", _DOC["listings"], ids=lambda r: r["name"])
+def test_listing_case(prod_v2: None, listing: dict, tmp_path: Path) -> None:
+    registry = _Registry(listing["referrers"])
+    try:
+        manifest = _BUILD["manifest"]
+        entry = resolve_cache_root(tmp_path) / "unpacked" / "sha256" / manifest.split(":")[1]
+        entry.mkdir(parents=True)
+        res = Resolved(
+            abi_generation=2,
+            platform="linux-amd64",
+            request="26.8",
+            version="26.8.15.10",
+            channel=None,
+            build="20260101.000000",
+            library_path=entry / "libchtypes.so",
+            dir=entry,
+            digests={"manifest": manifest, "layer": _BUILD["layer"], "index": None, "bundle": None},
+            predicate=_BUILD["predicate"],
+            signed_by="",
+            source="cache",
+            already_installed=True,
+        )
+        opts = Options(
+            bases=(registry.base,), cache_dir=tmp_path, system_dirs=(), trusted_keys=_KEYS
+        )
+        warnings: list[str] = []
+        revoked: ArtifactRevokedError | None = None
+        try:
+            warnings = check_installed(res, opts)
+        except ArtifactRevokedError as exc:
+            revoked = exc
+            warnings = list(getattr(exc, "warnings", ()))
+    finally:
+        registry.stop()
+
+    ignored = [registry.served[f] for f in listing["ignored"]]
+    dont_care = [registry.served[f] for f in listing.get("dont_care", [])]
+    for digest in ignored:
+        assert any(digest in w and "does not verify and is ignored" in w for w in warnings), (
+            f"no warning names ignored referrer {digest}: {warnings}"
+        )
+    for w in warnings:
+        assert "does not verify and is ignored" in w, f"unexpected warning {w!r}"
+        assert any(d in w for d in ignored + dont_care), f"warning names no listed referrer: {w!r}"
+
+    if listing["expect"] == "revoked":
+        assert revoked is not None, "the build was allowed, want CHTYPES_ARTIFACT_REVOKED"
+        rev = revoked.revocation
+        assert rev is not None and rev.signed and len(rev.advisories) == 1
+        assert _advisory(rev.advisories[0]) == listing["expect_advisory"]
+    else:
+        assert listing["expect"] == "fail-open"
+        assert revoked is None, "the build was refused, want it allowed"

{chtypes-2.0.0.dev11 → chtypes-2.0.0.dev12}/uv.lock

@@ -74,7 +74,7 @@ wheels = [
 
 [[package]]
 name = "chtypes"
-version = "2.0.0.dev11"
+version = "2.0.0.dev12"
 source = { editable = "." }
 dependencies = [
     { name = "backports-zstd", marker = "python_full_version < '3.14'" },