pdfhell 0.1.0__tar.gz

pdfhell-0.1.0/LICENSE

@@ -0,0 +1,17 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Copyright 2026 Multivon
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

pdfhell-0.1.0/PKG-INFO

@@ -0,0 +1,208 @@
+Metadata-Version: 2.4
+Name: pdfhell
+Version: 0.1.0
+Summary: PDF Hell — adversarial PDFs that break AI document readers. Procedural ground truth, not LLM-as-judge.
+Author: Multivon
+License: Apache-2.0
+Project-URL: Homepage, https://pdfhell.multivon.ai
+Project-URL: Repository, https://github.com/multivon-ai/pdfhell
+Project-URL: Issues, https://github.com/multivon-ai/pdfhell/issues
+Project-URL: Leaderboard, https://pdfhell.multivon.ai/leaderboard
+Keywords: llm,evaluation,pdf,multimodal,benchmark,adversarial,document-ai,rag
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: multivon-eval>=0.7.2
+Requires-Dist: google-genai>=1.0
+Requires-Dist: reportlab>=4.0
+Requires-Dist: pypdf>=5.0
+Provides-Extra: all
+Dynamic: license-file
+
+# PDF Hell
+
+**Adversarial PDFs that break AI document readers — with procedural ground truth, not LLM-as-judge.**
+
+PDF Hell is a small, sharp benchmark for the "AI reads PDFs" claim. Every test case is a PDF generated *from code*, so the correct answer is known exactly. There's no LLM judging another LLM's interpretation — the same loop that fooled the model isn't asked to grade it.
+
+If your AI claims it can read documents, it should survive PDFs designed to break it.
+
+## Quickstart (30 seconds)
+
+```bash
+# 3-case smoke run against the cheapest vision model — works in any env with a Gemini key
+export GOOGLE_API_KEY=...
+uvx pdfhell run --model google:gemini-2.5-flash --suite smoke
+
+# Or run the full mini suite (30 cases, ~10s on Flash, ~$0.01)
+uvx pdfhell run --model anthropic:claude-sonnet-4-6 --suite mini
+
+# Or just generate one trap PDF and open it
+uvx pdfhell make --trap hidden_ocr_mismatch --seed 42
+open ./cases/hidden_ocr_mismatch-0042.pdf
+```
+
+That's it. `pdfhell run` builds the suite on first use, sends each PDF to the vision model, and grades the answer against code-based ground truth.
+
+Smoke result on Gemini 2.5 Flash (one case per family, run this minute):
+
+```
+PDF Hell smoke suite — n=3
+model: google:gemini-2.5-flash
+pass: 3/3  (100.0%)
+```
+
+## What's in the mini suite
+
+| Trap family | Cases | What breaks |
+|---|---|---|
+| `hidden_ocr_mismatch` | 10 | Invoices where the visible amount differs from an invisible OCR text layer. Vision-only models read the page; text-extraction pipelines read the layer; they disagree. |
+| `footnote_override` | 10 | Legal clauses where a 6pt footnote overrides the body — liability caps with carve-outs, terminations with restrictions, data-residency with disaster-recovery exceptions. |
+| `split_table_across_pages` | 10 | Financial tables where the header row sits on page 1 and the body rows on page 2. RAG loaders that paginate independently lose column context. |
+
+Every case has a deterministic seed. Re-running with the same seed regenerates **byte-identical PDFs** and identical answer keys. `Canvas(invariant=True)` is set on every generator so timestamps and document IDs don't drift between runs.
+
+The full suite (10 trap families, ~50 cases) is on the [roadmap](#roadmap).
+
+## Why this exists
+
+The current AI-eval state of the art uses an LLM-as-judge to grade another LLM's answer. That's circular: the same complexity that fools the agent fools the judge. PDF Hell rejects that:
+
+1. **Code-based ground truth.** The answer is a literal Python value the generator chose, not a frontier model's opinion.
+2. **A named failure mode per trap.** When a model fails, we know *which* specific failure caught it (e.g. "trusted the hidden OCR layer over the visible page").
+3. **A diagnostic signal**, not just a score. Per-trap-family breakdown tells you which assumption broke.
+
+## Commands
+
+```
+pdfhell list-traps                              # list trap families
+pdfhell make --trap <family> --seed <n>         # generate one case
+pdfhell build --suite <smoke|mini> --out <dir>  # materialise a suite
+pdfhell run --model <provider>:<model>          # evaluate a model
+  [--suite smoke|mini]                          #   (default: mini)
+  [--cases-dir <dir>]                           #   (default: ./cases/<suite>)
+  [--out <path>]                                #   JSON output
+  [--junit <path>]                              #   JUnit XML for GitHub Actions / GitLab CI
+  [--fail-threshold <0.0-1.0>]                  #   non-zero exit if pass_rate below threshold
+  [--workers <n>]                               #   parallel API requests (default: 4)
+  [--quiet]
+pdfhell report runs/<file>.json                 # print a saved run's summary
+```
+
+Provider shorthand: `anthropic:claude-sonnet-4-6`, `openai:gpt-4o`, `google:gemini-2.5-pro`, `google:gemini-2.5-flash`, etc. API key from env (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`).
+
+## CI integration
+
+Drop this into `.github/workflows/eval.yml`:
+
+```yaml
+name: PDF Hell
+on: [pull_request]
+jobs:
+  pdfhell:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uvx pdfhell run --model anthropic:claude-sonnet-4-6 --suite mini --junit results.xml --fail-threshold 0.7
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+      - uses: actions/upload-artifact@v4
+        with:
+          name: pdfhell-results
+          path: results.xml
+```
+
+JUnit XML renders natively in the GitHub Actions / GitLab CI / CircleCI / Jenkins PR panel — failures show up as red rows with the expected and observed answers in the failure message.
+
+## How scoring works
+
+Two layers, applied in order:
+
+1. **Procedural exact match (primary)** — for single-value traps, the model's free-text answer must contain the expected value (whitespace-tolerant, case-insensitive). For prose traps like `footnote_override`, the model must include every required token (the cap value, every carve-out section number, etc.) in any order, in any phrasing. The model isn't graded on prose style; it's graded on whether it captured the facts.
+2. **Forbidden-answer detection (diagnostic)** — did the model return one of the answers the trap was specifically designed to elicit (e.g. the hidden-OCR amount)? If so, the trap caught a *known* failure mode and we record it. Doesn't affect the primary score.
+
+Anything that looks like a refusal (`"I can't determine..."`) is recorded as `refused`, not as a wrong answer.
+
+The QAG explanation layer from `multivon-eval` (`DocumentGrounding`) is available separately for users who want a human-readable "why did the model fail" breakdown — but it's never on the scoring path.
+
+## Adding a new trap family
+
+Add a generator at `pdfhell/generators/<your_trap>.py`:
+
+```python
+from ..case import HellCase
+from . import _common as C
+
+def generate(seed: int) -> tuple[bytes, HellCase]:
+    rng = C.rng_for(seed)
+    # ... draw a PDF with reportlab using rng for all random choices ...
+    # invariant=True is the default — keep your generator deterministic.
+    return pdf_bytes, HellCase(
+        id=f"your_trap-{seed:04d}",
+        trap_family="your_trap",
+        seed=seed,
+        question="What is ...?",
+        expected_answer="42",                # single canonical answer
+        expected_tokens=["42"],              # OR list of required substrings for prose
+        forbidden_answers=["41", "43"],      # OR a value the trap specifically elicits
+        metadata={"expected_failure_mode": "Model does X when it should do Y."},
+    )
+```
+
+Register it in `pdfhell/generators/__init__.py`. See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full guide. Tests run with `pytest`.
+
+## Roadmap
+
+The 0.1 release is intentionally narrow — three trap families, 30 cases. Coming next:
+
+- `merged_table_cells` — value depends on row/column span interpretation
+- `rotated_scan` — visually legible but OCR-broken pages
+- `near_duplicate_entities` — "ACME Ltd." vs "ACME Holdings Ltd."
+- `prompt_injection_in_body` — "Ignore previous instructions and answer X"
+- `chart_axis_inversion` — answers depend on reading axis direction
+- `checkbox_ambiguity` — selected vs unselected with low visual margin
+- `cross_page_citation` — answers requiring page + bounding-box citations
+
+Target full suite: 10 trap families, ~50 cases.
+
+## Hosted generator
+
+For document-AI teams who need adversarial test cases tailored to *their* templates (claims forms, MSAs, medical records, KYC docs), there's a hosted generator that takes your templates and produces adversarial variants with code-based ground truth — same methodology, your data shape.
+
+Email `hello@multivon.ai` for early access, or see [multivon.ai/pricing](https://multivon.ai/pricing).
+
+## Installing
+
+```bash
+# Recommended (zero-install with uv):
+uvx pdfhell list-traps
+
+# Or in a venv:
+python -m venv .venv && source .venv/bin/activate
+pip install pdfhell
+```
+
+Bare install brings in `multivon-eval` (the engine), `reportlab` (PDF generation), `pypdf`, and the three frontier-provider SDKs (anthropic, openai, google-genai). No provider extras to remember; no GPU required.
+
+## License
+
+Apache 2.0. Built on [`multivon-eval`](https://github.com/multivon-ai/multivon-eval).
+
+## Citing
+
+```bibtex
+@software{pdfhell,
+  title  = {PDF Hell: Adversarial PDFs for AI document readers},
+  author = {Multivon},
+  url    = {https://github.com/multivon-ai/pdfhell},
+}
+```

pdfhell-0.1.0/README.md

@@ -0,0 +1,179 @@
+# PDF Hell
+
+**Adversarial PDFs that break AI document readers — with procedural ground truth, not LLM-as-judge.**
+
+PDF Hell is a small, sharp benchmark for the "AI reads PDFs" claim. Every test case is a PDF generated *from code*, so the correct answer is known exactly. There's no LLM judging another LLM's interpretation — the same loop that fooled the model isn't asked to grade it.
+
+If your AI claims it can read documents, it should survive PDFs designed to break it.
+
+## Quickstart (30 seconds)
+
+```bash
+# 3-case smoke run against the cheapest vision model — works in any env with a Gemini key
+export GOOGLE_API_KEY=...
+uvx pdfhell run --model google:gemini-2.5-flash --suite smoke
+
+# Or run the full mini suite (30 cases, ~10s on Flash, ~$0.01)
+uvx pdfhell run --model anthropic:claude-sonnet-4-6 --suite mini
+
+# Or just generate one trap PDF and open it
+uvx pdfhell make --trap hidden_ocr_mismatch --seed 42
+open ./cases/hidden_ocr_mismatch-0042.pdf
+```
+
+That's it. `pdfhell run` builds the suite on first use, sends each PDF to the vision model, and grades the answer against code-based ground truth.
+
+Smoke result on Gemini 2.5 Flash (one case per family, run this minute):
+
+```
+PDF Hell smoke suite — n=3
+model: google:gemini-2.5-flash
+pass: 3/3  (100.0%)
+```
+
+## What's in the mini suite
+
+| Trap family | Cases | What breaks |
+|---|---|---|
+| `hidden_ocr_mismatch` | 10 | Invoices where the visible amount differs from an invisible OCR text layer. Vision-only models read the page; text-extraction pipelines read the layer; they disagree. |
+| `footnote_override` | 10 | Legal clauses where a 6pt footnote overrides the body — liability caps with carve-outs, terminations with restrictions, data-residency with disaster-recovery exceptions. |
+| `split_table_across_pages` | 10 | Financial tables where the header row sits on page 1 and the body rows on page 2. RAG loaders that paginate independently lose column context. |
+
+Every case has a deterministic seed. Re-running with the same seed regenerates **byte-identical PDFs** and identical answer keys. `Canvas(invariant=True)` is set on every generator so timestamps and document IDs don't drift between runs.
+
+The full suite (10 trap families, ~50 cases) is on the [roadmap](#roadmap).
+
+## Why this exists
+
+The current AI-eval state of the art uses an LLM-as-judge to grade another LLM's answer. That's circular: the same complexity that fools the agent fools the judge. PDF Hell rejects that:
+
+1. **Code-based ground truth.** The answer is a literal Python value the generator chose, not a frontier model's opinion.
+2. **A named failure mode per trap.** When a model fails, we know *which* specific failure caught it (e.g. "trusted the hidden OCR layer over the visible page").
+3. **A diagnostic signal**, not just a score. Per-trap-family breakdown tells you which assumption broke.
+
+## Commands
+
+```
+pdfhell list-traps                              # list trap families
+pdfhell make --trap <family> --seed <n>         # generate one case
+pdfhell build --suite <smoke|mini> --out <dir>  # materialise a suite
+pdfhell run --model <provider>:<model>          # evaluate a model
+  [--suite smoke|mini]                          #   (default: mini)
+  [--cases-dir <dir>]                           #   (default: ./cases/<suite>)
+  [--out <path>]                                #   JSON output
+  [--junit <path>]                              #   JUnit XML for GitHub Actions / GitLab CI
+  [--fail-threshold <0.0-1.0>]                  #   non-zero exit if pass_rate below threshold
+  [--workers <n>]                               #   parallel API requests (default: 4)
+  [--quiet]
+pdfhell report runs/<file>.json                 # print a saved run's summary
+```
+
+Provider shorthand: `anthropic:claude-sonnet-4-6`, `openai:gpt-4o`, `google:gemini-2.5-pro`, `google:gemini-2.5-flash`, etc. API key from env (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`).
+
+## CI integration
+
+Drop this into `.github/workflows/eval.yml`:
+
+```yaml
+name: PDF Hell
+on: [pull_request]
+jobs:
+  pdfhell:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uvx pdfhell run --model anthropic:claude-sonnet-4-6 --suite mini --junit results.xml --fail-threshold 0.7
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+      - uses: actions/upload-artifact@v4
+        with:
+          name: pdfhell-results
+          path: results.xml
+```
+
+JUnit XML renders natively in the GitHub Actions / GitLab CI / CircleCI / Jenkins PR panel — failures show up as red rows with the expected and observed answers in the failure message.
+
+## How scoring works
+
+Two layers, applied in order:
+
+1. **Procedural exact match (primary)** — for single-value traps, the model's free-text answer must contain the expected value (whitespace-tolerant, case-insensitive). For prose traps like `footnote_override`, the model must include every required token (the cap value, every carve-out section number, etc.) in any order, in any phrasing. The model isn't graded on prose style; it's graded on whether it captured the facts.
+2. **Forbidden-answer detection (diagnostic)** — did the model return one of the answers the trap was specifically designed to elicit (e.g. the hidden-OCR amount)? If so, the trap caught a *known* failure mode and we record it. Doesn't affect the primary score.
+
+Anything that looks like a refusal (`"I can't determine..."`) is recorded as `refused`, not as a wrong answer.
+
+The QAG explanation layer from `multivon-eval` (`DocumentGrounding`) is available separately for users who want a human-readable "why did the model fail" breakdown — but it's never on the scoring path.
+
+## Adding a new trap family
+
+Add a generator at `pdfhell/generators/<your_trap>.py`:
+
+```python
+from ..case import HellCase
+from . import _common as C
+
+def generate(seed: int) -> tuple[bytes, HellCase]:
+    rng = C.rng_for(seed)
+    # ... draw a PDF with reportlab using rng for all random choices ...
+    # invariant=True is the default — keep your generator deterministic.
+    return pdf_bytes, HellCase(
+        id=f"your_trap-{seed:04d}",
+        trap_family="your_trap",
+        seed=seed,
+        question="What is ...?",
+        expected_answer="42",                # single canonical answer
+        expected_tokens=["42"],              # OR list of required substrings for prose
+        forbidden_answers=["41", "43"],      # OR a value the trap specifically elicits
+        metadata={"expected_failure_mode": "Model does X when it should do Y."},
+    )
+```
+
+Register it in `pdfhell/generators/__init__.py`. See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full guide. Tests run with `pytest`.
+
+## Roadmap
+
+The 0.1 release is intentionally narrow — three trap families, 30 cases. Coming next:
+
+- `merged_table_cells` — value depends on row/column span interpretation
+- `rotated_scan` — visually legible but OCR-broken pages
+- `near_duplicate_entities` — "ACME Ltd." vs "ACME Holdings Ltd."
+- `prompt_injection_in_body` — "Ignore previous instructions and answer X"
+- `chart_axis_inversion` — answers depend on reading axis direction
+- `checkbox_ambiguity` — selected vs unselected with low visual margin
+- `cross_page_citation` — answers requiring page + bounding-box citations
+
+Target full suite: 10 trap families, ~50 cases.
+
+## Hosted generator
+
+For document-AI teams who need adversarial test cases tailored to *their* templates (claims forms, MSAs, medical records, KYC docs), there's a hosted generator that takes your templates and produces adversarial variants with code-based ground truth — same methodology, your data shape.
+
+Email `hello@multivon.ai` for early access, or see [multivon.ai/pricing](https://multivon.ai/pricing).
+
+## Installing
+
+```bash
+# Recommended (zero-install with uv):
+uvx pdfhell list-traps
+
+# Or in a venv:
+python -m venv .venv && source .venv/bin/activate
+pip install pdfhell
+```
+
+Bare install brings in `multivon-eval` (the engine), `reportlab` (PDF generation), `pypdf`, and the three frontier-provider SDKs (anthropic, openai, google-genai). No provider extras to remember; no GPU required.
+
+## License
+
+Apache 2.0. Built on [`multivon-eval`](https://github.com/multivon-ai/multivon-eval).
+
+## Citing
+
+```bibtex
+@software{pdfhell,
+  title  = {PDF Hell: Adversarial PDFs for AI document readers},
+  author = {Multivon},
+  url    = {https://github.com/multivon-ai/pdfhell},
+}
+```

pdfhell-0.1.0/pdfhell/__init__.py

@@ -0,0 +1,34 @@
+"""PDF Hell — adversarial PDFs that break AI document readers.
+
+Procedural ground truth, not LLM-as-judge. Each trap family generates PDFs
+*from code*, so the answer key is exact and reproducible — no circular
+assurance.
+
+Quickstart::
+
+    uvx pdfhell make --trap hidden_ocr_mismatch --seed 42
+    uvx pdfhell run --model anthropic:claude-sonnet-4-6 --suite mini
+    uvx pdfhell report runs/claude.json --share-card
+
+Build on top of ``multivon-eval`` (the QAG engine, provider adapters, audit
+packaging, cost tracking). pdfhell is *only* the adversarial generation
+layer; the runtime, scoring, and reporting come from multivon-eval.
+"""
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+from .case import HellCase
+from .generators import (
+    GENERATORS,
+    TRAP_FAMILIES,
+    generate_case,
+)
+
+__all__ = [
+    "__version__",
+    "HellCase",
+    "GENERATORS",
+    "TRAP_FAMILIES",
+    "generate_case",
+]

pdfhell-0.1.0/pdfhell/auditpack.py

@@ -0,0 +1,182 @@
+"""Build a downloadable, hash-chained audit pack from a pdfhell run.
+
+The pack is a ZIP containing:
+
+- ``manifest.json`` — pdfhell version, run timestamp, model spec, suite,
+  per-trap pass rates, total cost (when known), SHA-256 of every file
+  inside the pack.
+- ``run.json`` — the full :class:`SuiteReport` JSON.
+- ``run.xml`` — JUnit XML (same data as ``run.json``, machine-readable
+  for CI dashboards).
+- ``cases/<case_id>.pdf`` — every adversarial PDF the model was tested
+  against.
+- ``cases/<case_id>.json`` — each case's answer key + metadata.
+- ``README.txt`` — human-readable "what's in this ZIP" + reproduction
+  command. Procurement teams open this first.
+
+The audit pack is the artifact a buyer's procurement team attaches to
+a diligence appendix. It must be self-describing (no out-of-band
+context required), reproducible (the manifest tells you the exact
+command to regenerate the run), and tamper-evident (the manifest
+includes a SHA-256 for every file in the pack; auditors can verify the
+ZIP wasn't edited after delivery).
+"""
+from __future__ import annotations
+
+import hashlib
+import json
+import zipfile
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Iterable
+
+from . import __version__
+from .case import HellCase
+from .junit import report_to_junit
+from .scorer import SuiteReport
+
+
+_README_TEMPLATE = """\
+# pdfhell audit pack
+
+This ZIP is a complete, self-describing record of one PDF Hell run. It
+contains every PDF the model was asked to read, every answer key, the
+raw model output, and a tamper-evident manifest.
+
+## What's in this pack
+
+- manifest.json — Run metadata + SHA-256 of every file in this ZIP.
+- run.json — Full run report (per-case scores, model outputs).
+- run.xml — JUnit XML (renders in CI dashboards).
+- cases/*.pdf — The adversarial PDFs the model was tested against.
+- cases/*.json — The answer keys + per-case metadata.
+- README.txt — This file.
+
+## How to verify
+
+The manifest contains a SHA-256 for every file in this ZIP. To verify
+nothing was edited after delivery:
+
+  unzip -p audit-pack.zip manifest.json | jq .files
+  sha256sum cases/*.pdf cases/*.json run.json run.xml README.txt
+
+Each hash in the manifest must match the file's actual SHA-256.
+
+## How to reproduce
+
+The manifest records the exact pdfhell command. To regenerate
+byte-identical PDFs and re-run the same model:
+
+  {repro_command}
+
+pdfhell uses Canvas(invariant=True) on every generator so PDFs are
+byte-identical across runs with the same seed.
+
+## Scope
+
+pdfhell {pdfhell_version}, suite {suite}, model {model}. Generated
+{timestamp}. {n} cases, {passed}/{n} passed ({pass_rate:.0%}). See
+manifest.json for per-trap breakdown.
+"""
+
+
+def _sha256(data: bytes) -> str:
+    return hashlib.sha256(data).hexdigest()
+
+
+def _gather_files(report: SuiteReport, cases_dir: Path) -> Iterable[tuple[str, bytes]]:
+    """Yield (arcname, bytes) pairs for every file going into the ZIP.
+
+    Order: README first (humans see it first), then manifest, then JSON
+    + XML, then case PDFs + answer keys. Stable ordering keeps the
+    SHA-256 of the ZIP itself stable across runs.
+    """
+    for case_summary in report.cases:
+        case_id = case_summary.case_id
+        pdf_path = cases_dir / f"{case_id}.pdf"
+        json_path = cases_dir / f"{case_id}.json"
+        if pdf_path.exists():
+            yield f"cases/{case_id}.pdf", pdf_path.read_bytes()
+        if json_path.exists():
+            yield f"cases/{case_id}.json", json_path.read_bytes()
+
+
+def build_audit_pack(
+    report: SuiteReport,
+    cases_dir: Path,
+    out_path: Path,
+) -> Path:
+    """Write a complete audit ZIP for ``report`` to ``out_path``.
+
+    Returns the resolved output path.
+    """
+    out_path = out_path.resolve()
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+
+    # Materialise the per-case files into bytes first so we can hash them.
+    case_files: list[tuple[str, bytes]] = list(_gather_files(report, cases_dir))
+
+    run_json_bytes = json.dumps(report.to_dict(), indent=2).encode("utf-8")
+    run_xml_bytes = report_to_junit(report).encode("utf-8")
+    timestamp = datetime.now(timezone.utc).isoformat()
+    passed = sum(1 for c in report.cases if c.correct)
+
+    repro_command = (
+        f"uvx pdfhell run --model {report.model} --suite {report.suite}"
+    )
+    readme_bytes = _README_TEMPLATE.format(
+        pdfhell_version=__version__,
+        suite=report.suite,
+        model=report.model,
+        timestamp=timestamp,
+        n=report.n,
+        passed=passed,
+        pass_rate=report.pass_rate,
+        repro_command=repro_command,
+    ).encode("utf-8")
+
+    # Build a manifest that hashes every other file in the pack. The
+    # manifest is the LAST file we hash so we can include the hashes of
+    # everything else inside it.
+    files_in_pack: list[tuple[str, bytes]] = [
+        ("README.txt", readme_bytes),
+        ("run.json", run_json_bytes),
+        ("run.xml", run_xml_bytes),
+        *case_files,
+    ]
+    manifest = {
+        "pdfhell_version": __version__,
+        "generated_at": timestamp,
+        "model": report.model,
+        "suite": report.suite,
+        "n": report.n,
+        "passed": passed,
+        "pass_rate": report.pass_rate,
+        "per_trap_pass": report.per_trap_pass,
+        "per_trap_fell_for_trap": report.per_trap_fell_for_trap,
+        "reproduction": {
+            "command": repro_command,
+            "note": (
+                "PDFs are regenerated byte-identically via Canvas(invariant=True). "
+                "Same seed → same PDF → same answer key."
+            ),
+        },
+        "files": [
+            {"path": name, "sha256": _sha256(data), "size": len(data)}
+            for name, data in files_in_pack
+        ],
+    }
+    manifest_bytes = json.dumps(manifest, indent=2).encode("utf-8")
+
+    # ZIP_DEFLATED is universal; mtime is set to the run timestamp so
+    # the ZIP itself is reproducible across packaging runs.
+    with zipfile.ZipFile(out_path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
+        for name, data in [("manifest.json", manifest_bytes), *files_in_pack]:
+            info = zipfile.ZipInfo(name)
+            info.date_time = (2026, 1, 1, 0, 0, 0)
+            zf.writestr(info, data)
+
+    return out_path
+
+
+__all__ = ["build_audit_pack"]