athena-python-docx 0.1.5__tar.gz → 0.1.6__tar.gz

{athena_python_docx-0.1.5 → athena_python_docx-0.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: athena-python-docx
-Version: 0.1.5
+Version: 0.1.6
 Summary: Drop-in replacement for python-docx that connects to Athena's Superdoc/Keryx collaborative document stack
 Project-URL: Homepage, https://athenaintelligence.ai
 Author-email: Athena Intelligence <engineering@athenaintelligence.ai>

{athena_python_docx-0.1.5 → athena_python_docx-0.1.6}/docx/__init__.py

@@ -6,7 +6,7 @@ See CLAUDE.md for the API parity contract.
 
 from __future__ import annotations
 
-__version__ = "0.1.5"
+__version__ = "0.1.6"
 
 from docx.api import Document
 

{athena_python_docx-0.1.5 → athena_python_docx-0.1.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "athena-python-docx"
-version = "0.1.5"
+version = "0.1.6"
 description = "Drop-in replacement for python-docx that connects to Athena's Superdoc/Keryx collaborative document stack"
 readme = "README.md"
 license = "MIT"

athena_python_docx-0.1.6/tests/fidelity/README.md

@@ -0,0 +1,71 @@
+# Fidelity tests
+
+End-to-end comparison of `athena-python-docx` against stock `python-docx`.
+For each test case (a small script using the python-docx API), the runner:
+
+1. Executes the script via **stock `python-docx`** (installed to an
+   isolated site-packages at `/tmp/docx-fidelity-stock-site-packages`)
+   and saves the resulting `.docx` to `/tmp/docx-fidelity-artifacts/`.
+2. Executes the same script via **`athena-python-docx`** inside a
+   Daytona sandbox (the `document-exec:v<N>` image). The resulting Y.Doc
+   is exported to `.docx` by Superdoc and downloaded back to
+   `/tmp/docx-fidelity-artifacts/`.
+3. Reads both `.docx` files back with stock `python-docx` and extracts a
+   normalized feature set (paragraph text + style, runs + bold/italic/
+   underline/color, table row/col counts + cell texts).
+4. Compares the feature sets and prints a scorecard.
+
+Stylistic differences that stem from the two libraries' different default
+templates (font, margins, theme colors) are intentionally ignored — the
+focus is on the content-level operations an agent actually cares about.
+
+## Running
+
+```bash
+cd agora && doppler run --project agora --config dev -- \
+    uv run python \
+    "$(git rev-parse --show-toplevel)/docx-studio/python-sdk/tests/fidelity/runner.py"
+```
+
+Configuration via env vars:
+
+| Var | Default | Purpose |
+|-----|---------|---------|
+| `ATHENA_DOCX_FIDELITY_ASSET_ID` | `asset_de593a96-…-2b16ad` (existing staging test doc) | SuperDoc asset to write against — contents are cleared via `doc.clear_content()` between cases |
+| `DAYTONA_DOCUMENT_EXEC_SNAPSHOT` | `document-exec:v1` (from agora settings) | Snapshot to spin up |
+
+## Interpreting results
+
+| Status | Meaning |
+|---|---|
+| `✅ OK` | Athena script succeeded and content matches stock python-docx |
+| `⚠️ DIFF` | Athena script succeeded but content differs — see listed diffs |
+| `🟡 STUB_HIT` | Case hit an expected `NotImplementedError` stub (a known Phase-2 gap) |
+| `🟠 UNEXPECTED_PASS` | Case was expected to raise a stub error, but didn't — probably means the Phase-2 gap has been filled; re-classify the case |
+| `🔴 WRONG_EXC` | Case raised, but the wrong exception type |
+| `❌ SCRIPT_ERROR` | Athena side crashed with an unexpected exception |
+| `❌ SETUP_ERROR` | Pre-flight `doc.clear_content()` failed |
+| `❌ EXPORT_ERROR` | Superdoc couldn't save the Y.Doc to `.docx` |
+| `❌ DOWNLOAD_ERROR` | `fs.download_file` on the Daytona sandbox failed |
+
+## Adding cases
+
+Edit `cases.py` and add a `Case(...)` entry. Keep scripts minimal (one
+surface per case where possible). If the operation is a known Phase-2
+stub, set `expected_athena_exc="NotImplementedError"` so the case is
+scored as `STUB_HIT` rather than as a failure.
+
+## Known caveats
+
+- The runner shares **one** SuperDoc asset across all cases, clearing
+  content between each via `doc.clear_content()`. If `clear_content`
+  ever regresses, every case after the first will be corrupted. The
+  runner reports that as `SETUP_ERROR`.
+- Stock python-docx's default template pads the document with a trailing
+  empty paragraph. The comparator filters empty paragraphs before
+  counting, but paragraph indices in diff output are 0-based on the
+  non-empty subset only.
+- `skip_content_diff=True` on a case tells the runner to only check that
+  the athena side didn't raise; useful for structural ops
+  (`add_page_break`) where the exported OOXML legitimately differs from
+  the stock output.

athena_python_docx-0.1.6/tests/fidelity/__init__.py

@@ -0,0 +1,14 @@
+"""Fidelity tests for athena-python-docx vs stock python-docx.
+
+Each test case is a small script that uses the python-docx `docx` namespace.
+The same script runs against:
+  - stock `python-docx` (in an isolated venv)      -> produces .docx A
+  - `athena-python-docx` via Daytona + Superdoc    -> produces .docx B
+
+Both .docx files are then read back through stock python-docx for feature
+extraction, and the extracted structures are compared. Differences that
+matter (missing text, wrong style, different cell count) are scored as
+FAIL; stylistic differences (default font, theme color) are tolerated.
+
+See `README.md` for running.
+"""

athena_python_docx-0.1.6/tests/fidelity/cases.py

@@ -0,0 +1,155 @@
+"""Fidelity test cases.
+
+Each case is a snippet that operates on a pre-bound ``doc`` (the Document
+instance). The same snippet runs against stock python-docx and against
+athena-python-docx via Daytona.
+
+Conventions
+-----------
+- Use only the python-docx public API — no private `_element` access, no
+  imports beyond what's in the preamble.
+- Keep cases minimal (one surface per case where possible) so a failure
+  pinpoints the SDK gap.
+- Mark known-stubbed operations with ``expected_exc`` — the case passes
+  if the stub actually raises (regression detection).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+
+@dataclass(frozen=True)
+class Case:
+    name: str
+    description: str
+    script: str
+    # When set, the athena-python-docx run is expected to raise this exception
+    # type. The local python-docx run is never expected to raise — if it does,
+    # the case is broken and reported as an infrastructure error.
+    expected_athena_exc: str | None = None
+    # If True, the athena side is not compared against stock — only asserted
+    # to succeed. Use when the op has no content-level effect worth diffing
+    # (e.g. add_page_break), or when the stock and Superdoc template diverge
+    # too much for a meaningful diff.
+    skip_content_diff: bool = False
+    # Optional tags for filtering.
+    tags: tuple[str, ...] = field(default_factory=tuple)
+
+
+CASES: list[Case] = [
+    # ---- Core append ops -----------------------------------------------------
+    Case(
+        name="add_paragraph_with_text",
+        description="One plain paragraph containing a short string.",
+        script='doc.add_paragraph("hello world")',
+        tags=("paragraph",),
+    ),
+    Case(
+        name="add_paragraph_empty",
+        description="A paragraph with no text (docx.add_paragraph()).",
+        script="doc.add_paragraph()",
+        tags=("paragraph",),
+    ),
+    Case(
+        name="add_heading_level_1",
+        description="Heading 1 with text.",
+        script='doc.add_heading("H1 heading", level=1)',
+        tags=("heading",),
+    ),
+    Case(
+        name="add_heading_level_2",
+        description="Heading 2 with text.",
+        script='doc.add_heading("H2 heading", level=2)',
+        tags=("heading",),
+    ),
+    Case(
+        name="add_heading_level_3",
+        description="Heading 3 with text.",
+        script='doc.add_heading("H3 heading", level=3)',
+        tags=("heading",),
+    ),
+    # ---- Runs + inline formatting --------------------------------------------
+    Case(
+        name="add_run_bold_italic",
+        description="Paragraph with two runs, bold + italic.",
+        script=(
+            "p = doc.add_paragraph()\n"
+            'r1 = p.add_run("bold run")\n'
+            "r1.bold = True\n"
+            'r2 = p.add_run("italic run")\n'
+            "r2.italic = True"
+        ),
+        tags=("run", "formatting"),
+    ),
+    Case(
+        name="add_run_colored",
+        description="Paragraph with a run colored green via RGBColor.",
+        script=(
+            "from docx.shared import RGBColor\n"
+            "p = doc.add_paragraph()\n"
+            'r = p.add_run("green text")\n'
+            "r.font.color.rgb = RGBColor(0x00, 0x66, 0x00)"
+        ),
+        tags=("run", "formatting", "color"),
+    ),
+    Case(
+        name="add_run_font_size",
+        description="Paragraph with a run set to 18pt.",
+        script=(
+            "from docx.shared import Pt\n"
+            "p = doc.add_paragraph()\n"
+            'r = p.add_run("sized 18pt")\n'
+            "r.font.size = Pt(18)"
+        ),
+        tags=("run", "formatting", "font"),
+    ),
+    # ---- Tables --------------------------------------------------------------
+    Case(
+        name="add_table_2x2_empty",
+        description="Create an empty 2x2 table; verify row/col count.",
+        script="doc.add_table(rows=2, cols=2)",
+        tags=("table",),
+    ),
+    Case(
+        name="add_table_3x3_with_style",
+        description="Create a 3x3 TableGrid table.",
+        script='doc.add_table(rows=3, cols=3, style="TableGrid")',
+        tags=("table", "style"),
+    ),
+    Case(
+        name="cell_text_setter",
+        description="Set cell(0,0).text on a 2x2 table — known NotImplementedError in 0.1.5.",
+        script=(
+            "t = doc.add_table(rows=2, cols=2)\n"
+            't.cell(0, 0).text = "A1"'
+        ),
+        expected_athena_exc="NotImplementedError",
+        tags=("table", "stub"),
+    ),
+    # ---- Structural ops ------------------------------------------------------
+    Case(
+        name="add_page_break",
+        description="Append a page break; content-diff skipped (template differs).",
+        script="doc.add_page_break()",
+        skip_content_diff=True,
+        tags=("structural",),
+    ),
+    # ---- Multi-op combo ------------------------------------------------------
+    Case(
+        name="combo_heading_paragraph_table",
+        description="Heading + paragraph with formatted runs + 2x2 table (no cell text).",
+        script=(
+            "from docx.shared import RGBColor\n"
+            'doc.add_heading("Report", level=1)\n'
+            "p = doc.add_paragraph()\n"
+            'r1 = p.add_run("Revenue grew ")\n'
+            'r2 = p.add_run("12%")\n'
+            "r2.bold = True\n"
+            "r2.font.color.rgb = RGBColor(0x00, 0x66, 0x00)\n"
+            'p.add_run(" YoY.")\n'
+            'doc.add_table(rows=2, cols=2, style="TableGrid")'
+        ),
+        tags=("combo",),
+    ),
+]

athena_python_docx-0.1.6/tests/fidelity/extract.py

@@ -0,0 +1,259 @@
+"""Feature extractor — reads a .docx (produced by either SDK) with stock
+python-docx and returns a normalized structure for comparison.
+
+The extractor focuses on features that SHOULD round-trip regardless of
+template differences between stock python-docx and Superdoc:
+  - paragraph text content and style name
+  - runs within each paragraph: text + bold + italic + underline + color
+  - table row/col counts + cell texts
+
+It does NOT try to compare defaults, fonts, margins, or theme colors —
+those legitimately differ between a blank python-docx template and the
+Superdoc SuperDoc-default template, and would produce false positives.
+"""
+
+from __future__ import annotations
+
+import importlib
+import sys
+import types
+from pathlib import Path
+from typing import TypedDict
+
+
+# ── isolated python-docx loader ────────────────────────────────────────
+# We run this extractor in a process that has the athena-python-docx
+# `docx` namespace installed. To read real .docx files we need STOCK
+# python-docx. Load it from an isolated path and return a module handle.
+
+_STOCK_CACHE: types.ModuleType | None = None
+
+
+def _load_stock_python_docx(stock_site_packages: str) -> types.ModuleType:
+    """Import the stock python-docx module from an isolated site-packages.
+
+    Mirrors the pattern from pptx-studio's
+    test_python_pptx_api_parity.py::_load_standard_module.
+    """
+    global _STOCK_CACHE
+    if _STOCK_CACHE is not None:
+        return _STOCK_CACHE
+
+    original_path = sys.path.copy()
+    original_modules = {k: v for k, v in sys.modules.items() if k.startswith("docx")}
+    for key in list(sys.modules):
+        if key.startswith("docx"):
+            del sys.modules[key]
+
+    sys.path.insert(0, stock_site_packages)
+    try:
+        mod = importlib.import_module("docx")
+        _STOCK_CACHE = mod
+        return mod
+    finally:
+        sys.path[:] = original_path
+        # Restore our own `docx.*` modules so later imports of the Athena
+        # SDK work. (The cached stock module stays importable as long as
+        # _STOCK_CACHE holds a reference.)
+        for key in list(sys.modules):
+            if key.startswith("docx"):
+                del sys.modules[key]
+        sys.modules.update(original_modules)
+
+
+# ── feature types ──────────────────────────────────────────────────────
+
+class RunFeature(TypedDict, total=False):
+    text: str
+    bold: bool
+    italic: bool
+    underline: bool
+    color: str | None  # hex string, e.g. "006600", or None
+
+
+class ParaFeature(TypedDict, total=False):
+    text: str
+    style: str
+    runs: list[RunFeature]
+
+
+class CellFeature(TypedDict):
+    text: str
+
+
+class TableFeature(TypedDict):
+    rows: int
+    cols: int
+    cells: list[list[CellFeature]]  # rows × cols
+
+
+class DocxFeatures(TypedDict):
+    paragraphs: list[ParaFeature]
+    tables: list[TableFeature]
+    paragraph_count: int
+    table_count: int
+
+
+# ── the extractor ──────────────────────────────────────────────────────
+
+
+def _ensure_styles_with_effects(docx_path: Path) -> Path:
+    """Patch a .docx to contain a stub `word/stylesWithEffects.xml`.
+
+    New python-docx (≥0.8.10) requires that part when `word/_rels/
+    document.xml.rels` references it (older Office template legacy).
+    Superdoc's export doesn't include it, which makes python-docx raise
+    "There is no item named 'word/stylesWithEffects.xml' in the archive"
+    the first time a style is dereferenced.
+
+    We add a minimal valid-but-empty styles document. Mutation happens in
+    a sibling tmpfile so the original export stays intact for inspection.
+    """
+    import shutil
+    import zipfile
+
+    import tempfile
+    with zipfile.ZipFile(docx_path, "r") as z:
+        names = set(z.namelist())
+    if "word/stylesWithEffects.xml" in names:
+        return docx_path
+    patched = Path(tempfile.mkstemp(prefix="patched_", suffix=".docx")[1])
+    shutil.copy2(docx_path, patched)
+    stub = (
+        b'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
+        b'<w:styles xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">'
+        b'</w:styles>'
+    )
+    with zipfile.ZipFile(patched, "a") as z:
+        if "word/stylesWithEffects.xml" not in set(z.namelist()):
+            z.writestr("word/stylesWithEffects.xml", stub)
+    return patched
+
+
+def extract_features(docx_path: Path, stock_site_packages: str) -> DocxFeatures:
+    """Extract normalized features from a .docx file."""
+    docx_path = _ensure_styles_with_effects(docx_path)
+    docx_mod = _load_stock_python_docx(stock_site_packages)
+    # Open the file. We re-import via the isolated stock docx to avoid
+    # confusion with our SDK's `docx` namespace.
+    orig_sys_path = sys.path.copy()
+    orig_mods = {k: v for k, v in sys.modules.items() if k.startswith("docx")}
+    for key in list(sys.modules):
+        if key.startswith("docx"):
+            del sys.modules[key]
+    sys.path.insert(0, stock_site_packages)
+    try:
+        docx_mod = importlib.import_module("docx")
+        stock_Document = docx_mod.Document  # type: ignore[attr-defined]
+
+        doc = stock_Document(str(docx_path))
+
+        paragraphs: list[ParaFeature] = []
+        for p in doc.paragraphs:
+            runs: list[RunFeature] = []
+            for r in p.runs:
+                rf: RunFeature = {
+                    "text": r.text,
+                    "bold": bool(r.bold) if r.bold is not None else False,
+                    "italic": bool(r.italic) if r.italic is not None else False,
+                    "underline": bool(r.underline) if r.underline is not None else False,
+                    "color": None,
+                }
+                try:
+                    rgb = r.font.color.rgb
+                    if rgb is not None:
+                        rf["color"] = str(rgb)
+                except Exception:
+                    pass
+                runs.append(rf)
+            style_name = ""
+            try:
+                style_name = p.style.name if p.style is not None else ""
+            except Exception:
+                style_name = ""
+            paragraphs.append({
+                "text": p.text,
+                "style": style_name,
+                "runs": runs,
+            })
+
+        tables: list[TableFeature] = []
+        for t in doc.tables:
+            rows = len(t.rows)
+            cols = len(t.rows[0].cells) if rows > 0 else 0
+            cells: list[list[CellFeature]] = []
+            for row in t.rows:
+                cells.append([{"text": c.text} for c in row.cells])
+            tables.append({
+                "rows": rows,
+                "cols": cols,
+                "cells": cells,
+            })
+
+        return {
+            "paragraphs": paragraphs,
+            "tables": tables,
+            "paragraph_count": len(paragraphs),
+            "table_count": len(tables),
+        }
+    finally:
+        sys.path[:] = orig_sys_path
+        for key in list(sys.modules):
+            if key.startswith("docx"):
+                del sys.modules[key]
+        sys.modules.update(orig_mods)
+
+
+# ── comparator ─────────────────────────────────────────────────────────
+
+
+def compare(stock: DocxFeatures, athena: DocxFeatures) -> list[str]:
+    """Return a list of semantic differences. Empty list = pass."""
+    diffs: list[str] = []
+
+    # Paragraph count. Superdoc's Word-default template often inserts an
+    # extra empty paragraph at the end (no text, no runs). We tolerate
+    # a +/- 1 gap as long as the non-empty content matches.
+    stock_nonempty = [p for p in stock["paragraphs"] if p["text"] or p["runs"]]
+    athena_nonempty = [p for p in athena["paragraphs"] if p["text"] or p["runs"]]
+    if len(stock_nonempty) != len(athena_nonempty):
+        diffs.append(
+            f"non-empty paragraph count: stock={len(stock_nonempty)} "
+            f"athena={len(athena_nonempty)}",
+        )
+
+    for i, (s, a) in enumerate(zip(stock_nonempty, athena_nonempty)):
+        if s["text"] != a["text"]:
+            diffs.append(
+                f"paragraph[{i}] text: stock={s['text']!r} athena={a['text']!r}",
+            )
+        # Style name: stock uses "Heading 1" style names; Superdoc may
+        # use "Heading1" (no space) or the same. Normalize whitespace.
+        s_style = (s.get("style") or "").replace(" ", "").lower()
+        a_style = (a.get("style") or "").replace(" ", "").lower()
+        if s_style != a_style:
+            diffs.append(
+                f"paragraph[{i}] style: stock={s['style']!r} athena={a['style']!r}",
+            )
+
+    # Table structure
+    if stock["table_count"] != athena["table_count"]:
+        diffs.append(
+            f"table count: stock={stock['table_count']} athena={athena['table_count']}",
+        )
+    for i, (st, at) in enumerate(zip(stock["tables"], athena["tables"])):
+        if st["rows"] != at["rows"] or st["cols"] != at["cols"]:
+            diffs.append(
+                f"table[{i}] shape: stock={st['rows']}x{st['cols']} "
+                f"athena={at['rows']}x{at['cols']}",
+            )
+        # Cell content
+        for r, (s_row, a_row) in enumerate(zip(st["cells"], at["cells"])):
+            for c, (s_cell, a_cell) in enumerate(zip(s_row, a_row)):
+                if s_cell["text"] != a_cell["text"]:
+                    diffs.append(
+                        f"table[{i}].cell({r},{c}).text: "
+                        f"stock={s_cell['text']!r} athena={a_cell['text']!r}",
+                    )
+
+    return diffs

athena_python_docx-0.1.6/tests/fidelity/runner.py

@@ -0,0 +1,453 @@
+"""Fidelity runner — orchestrates stock python-docx + athena-python-docx
+across all cases in ``cases.CASES`` and prints a scorecard.
+
+Usage
+-----
+    cd agora && doppler run --project agora --config dev -- \
+        uv run python ../docx-studio/python-sdk/tests/fidelity/runner.py
+
+The runner needs:
+  - a Daytona API key (from agora settings) — used to spin up a sandbox
+    and execute the athena-python-docx side
+  - a Keryx private key + WS URL (from agora settings) — to sign a token
+    for the sandbox
+  - a reusable SuperDoc asset to write against (default asset on staging
+    is hard-coded; override with ATHENA_DOCX_FIDELITY_ASSET_ID)
+  - stock python-docx installed somewhere (we install it on-demand into
+    ``/tmp/docx-fidelity-stock-site-packages`` if missing)
+
+Output is a table listing each case with status (PASS / FAIL / STUB-HIT /
+ERROR), plus any diffs.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+# Allow running this file as a script: add the sdk root so "docx.*" and
+# the tests/fidelity package both resolve.
+_SDK_ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(_SDK_ROOT))
+
+from tests.fidelity.cases import CASES, Case  # noqa: E402
+from tests.fidelity.extract import compare, extract_features  # noqa: E402
+
+
+# ── configuration ─────────────────────────────────────────────────────
+
+DEFAULT_ASSET_ID = os.environ.get(
+    "ATHENA_DOCX_FIDELITY_ASSET_ID",
+    "asset_de593a96-3031-45ea-bc52-60cb4b2b16ad",  # "PR18540 SuperDoc Test"
+)
+STOCK_SITE_PACKAGES = Path("/tmp/docx-fidelity-stock-site-packages")
+ARTIFACT_DIR = Path("/tmp/docx-fidelity-artifacts")
+
+
+# ── stock python-docx setup ───────────────────────────────────────────
+
+
+def ensure_stock_python_docx() -> Path:
+    """Install stock python-docx into an isolated directory if not present."""
+    marker = STOCK_SITE_PACKAGES / "docx" / "__init__.py"
+    if marker.exists():
+        return STOCK_SITE_PACKAGES
+    print(f"[fidelity] installing stock python-docx -> {STOCK_SITE_PACKAGES}")
+    STOCK_SITE_PACKAGES.mkdir(parents=True, exist_ok=True)
+    # Use `uv pip install --target` — agora's .venv is uv-managed and
+    # doesn't include stdlib `pip`. Fall back to `python -m pip` only if
+    # `uv` isn't on PATH (shouldn't happen inside `doppler run -- uv run`).
+    uv = _which("uv")
+    if uv:
+        cmd = [uv, "pip", "install", "--quiet", "--target", str(STOCK_SITE_PACKAGES), "python-docx>=1.1"]
+    else:
+        cmd = [sys.executable, "-m", "pip", "install", "--quiet", "--target", str(STOCK_SITE_PACKAGES), "python-docx>=1.1"]
+    subprocess.run(cmd, check=True)
+    return STOCK_SITE_PACKAGES
+
+
+def _which(name: str) -> str | None:
+    import shutil
+    return shutil.which(name)
+
+
+def run_stock_script(script: str, out_path: Path) -> None:
+    """Run a script with stock python-docx, writing output to `out_path`.
+
+    The subprocess is invoked with PYTHONPATH pointed at our isolated
+    stock site-packages so we don't collide with our SDK's `docx` ns.
+    """
+    stock = ensure_stock_python_docx()
+    wrapped = (
+        "import sys\n"
+        f"sys.path.insert(0, {str(stock)!r})\n"
+        "from docx import Document\n"
+        "doc = Document()\n"
+        f"{script}\n"
+        f"doc.save({str(out_path)!r})\n"
+    )
+    result = subprocess.run(
+        [sys.executable, "-c", wrapped],
+        capture_output=True,
+        text=True,
+        timeout=30,
+    )
+    if result.returncode != 0:
+        raise RuntimeError(
+            f"stock python-docx subprocess failed:\n"
+            f"  stdout: {result.stdout}\n"
+            f"  stderr: {result.stderr}",
+        )
+
+
+# ── athena-python-docx via Daytona ────────────────────────────────────
+
+
+def build_sandbox_script(cases_batch: list[Case], asset_id: str) -> str:
+    """Return a Python program to run inside the Daytona sandbox.
+
+    It iterates cases, for each:
+      - clears the doc via superdoc-sdk.doc.clear_content()
+      - executes the case's script with `doc` pre-bound
+      - exports the resulting .docx to /tmp/fidelity_out/<case_name>.docx
+      - records success/exception into /tmp/fidelity_results.json
+    """
+    cases_json = json.dumps(
+        [{"name": c.name, "script": c.script, "expected_exc": c.expected_athena_exc} for c in cases_batch],
+    )
+    return f'''
+import asyncio
+import json
+import os
+import traceback
+from pathlib import Path
+
+OUT_DIR = Path("/tmp/fidelity_out")
+OUT_DIR.mkdir(parents=True, exist_ok=True)
+ASSET_ID = {asset_id!r}
+CASES = json.loads({cases_json!r})
+
+from superdoc import AsyncSuperDocClient
+from superdoc.generated.client import DocOpenParams, DocOpenParamsCollaborationVariant1
+
+async def with_session(callback):
+    token = os.environ["SUPERDOC_COLLAB_TOKEN"]
+    ws_url = os.environ["KERYX_WS_URL"]
+    ws_id = os.environ["ATHENA_WORKSPACE_ID"]
+    client = AsyncSuperDocClient(
+        user={{"name":"fidelity-runner","email":"fidelity@athenaintel.com"}},
+        env={{"SUPERDOC_COLLAB_TOKEN": token}},
+        request_timeout_ms=30_000,
+        watchdog_timeout_ms=60_000,
+    )
+    try:
+        collab: DocOpenParamsCollaborationVariant1 = {{
+            "url": f"{{ws_url}}/ws/{{ws_id}}",
+            "documentId": ASSET_ID,
+            "tokenEnv": "SUPERDOC_COLLAB_TOKEN",
+            "providerType": "y-websocket",
+        }}
+        h = await client.open({{"collaboration": collab}})
+        try:
+            return await callback(h)
+        finally:
+            await asyncio.sleep(0.5)
+            await h.close({{"discard": True}})
+    finally:
+        await client.dispose()
+
+
+async def clear_asset():
+    async def _clear(h):
+        try:
+            await h.clear_content({{}})
+        except Exception as e:
+            # If "already empty" fine
+            if "already empty" not in str(e).lower():
+                raise
+    await with_session(_clear)
+
+
+async def export_to(path):
+    async def _export(h):
+        await h.save({{"out": str(path)}})
+    await with_session(_export)
+
+
+def run_case_locally(script):
+    """Run script against athena-python-docx via sync facade."""
+    from docx import Document
+    with Document(ASSET_ID) as doc:
+        # RGBColor and Pt are intentionally not pre-bound — each script that
+        # needs them imports them explicitly (matching how agents write code).
+        from docx.shared import RGBColor, Pt
+        local_ns = dict(doc=doc, RGBColor=RGBColor, Pt=Pt)
+        exec(compile(script, "<case>", "exec"), local_ns, local_ns)
+        doc.save()
+
+
+results = []
+for case in CASES:
+    name = case["name"]
+    script = case["script"]
+    expected = case["expected_exc"]
+
+    # Step 1: clear the asset
+    try:
+        asyncio.run(clear_asset())
+    except Exception as e:
+        results.append({{
+            "name": name,
+            "status": "SETUP_ERROR",
+            "detail": f"clear_content failed: {{e!r}}",
+        }})
+        continue
+
+    # Step 2: run the case's script via athena-python-docx
+    script_error = None
+    try:
+        run_case_locally(script)
+    except Exception as e:
+        script_error = (type(e).__name__, str(e)[:400])
+
+    # Step 3: compare error vs expectation
+    if expected is not None:
+        if script_error is not None and script_error[0] == expected:
+            results.append({{"name": name, "status": "STUB_HIT", "detail": script_error[1][:200]}})
+            continue
+        if script_error is None:
+            results.append({{"name": name, "status": "UNEXPECTED_PASS", "detail": f"expected {{expected}} but succeeded"}})
+            continue
+        results.append({{
+            "name": name,
+            "status": "WRONG_EXC",
+            "detail": f"expected {{expected}}, got {{script_error[0]}}: {{script_error[1]}}",
+        }})
+        continue
+
+    if script_error is not None:
+        results.append({{
+            "name": name,
+            "status": "SCRIPT_ERROR",
+            "detail": f"{{script_error[0]}}: {{script_error[1]}}",
+        }})
+        continue
+
+    # Step 4: export and record artifact
+    out_path = OUT_DIR / f"{{name}}.docx"
+    try:
+        asyncio.run(export_to(out_path))
+    except Exception as e:
+        results.append({{"name": name, "status": "EXPORT_ERROR", "detail": f"{{type(e).__name__}}: {{e}}"}})
+        continue
+
+    size = out_path.stat().st_size if out_path.exists() else 0
+    results.append({{"name": name, "status": "OK", "detail": f"exported {{size}} bytes", "export_path": str(out_path)}})
+
+Path("/tmp/fidelity_results.json").write_text(json.dumps(results, indent=2))
+print(json.dumps(results, indent=2))
+'''
+
+
+async def run_daytona_batch(cases: list[Case], asset_id: str) -> tuple[list[dict], dict[str, bytes]]:
+    """Spin up a sandbox, run all cases, download every exported .docx."""
+    from agora.services.keryx.client import get_keryx_client
+    from agora.services.auth.docx_auth import build_docx_collab_bundle  # noqa: F401 ensures agora is loadable
+    from agora.settings import settings
+    from daytona_sdk import AsyncDaytona, CreateSandboxFromSnapshotParams, DaytonaConfig
+
+    routing_bundle = await _build_collab_bundle(asset_id)
+    kc = get_keryx_client()
+    # Fresh token for the sandbox.
+    token = kc._sign_token(user_id="fidelity-runner")  # noqa: SLF001
+
+    daytona = AsyncDaytona(
+        config=DaytonaConfig(
+            api_key=settings.daytona_full_api_key,
+            api_url=settings.daytona_api_url,
+            organization_id=settings.daytona_organization_id,
+            target=settings.daytona_target,
+        ),
+    )
+    snapshot = settings.daytona_document_exec_snapshot or "document-exec:v1"
+    print(f"[fidelity] daytona snapshot: {snapshot}")
+
+    sandbox = await daytona.create(
+        CreateSandboxFromSnapshotParams(
+            name=f"docx-fidelity-{int(time.time())}",
+            snapshot=snapshot,
+            env_vars={
+                "SUPERDOC_COLLAB_TOKEN": token,
+                "KERYX_WS_URL": settings.keryx_ws_url,
+                "ATHENA_WORKSPACE_ID": routing_bundle["ATHENA_WORKSPACE_ID"],
+            },
+            auto_stop_interval=30,
+            auto_archive_interval=60,
+        ),
+        timeout=120,
+    )
+    print(f"[fidelity] sandbox {sandbox.id} started")
+
+    try:
+        script = build_sandbox_script(cases, asset_id)
+        await sandbox.fs.upload_file(script.encode(), "/tmp/fidelity_runner.py")
+        exec_result = await sandbox.process.exec(
+            "python3 /tmp/fidelity_runner.py",
+            timeout=600,
+        )
+        print(f"[fidelity] sandbox exit_code={exec_result.exit_code}")
+        # Load the results JSON
+        raw = await sandbox.fs.download_file("/tmp/fidelity_results.json")
+        results: list[dict] = json.loads(raw.decode())
+
+        # Download every successfully-exported .docx artifact
+        artifacts: dict[str, bytes] = {}
+        for r in results:
+            if r["status"] == "OK" and "export_path" in r:
+                try:
+                    data = await sandbox.fs.download_file(r["export_path"])
+                    artifacts[r["name"]] = data
+                except Exception as e:
+                    r["status"] = "DOWNLOAD_ERROR"
+                    r["detail"] = f"fs.download_file: {e}"
+        return results, artifacts
+    finally:
+        try:
+            await sandbox.delete()
+        except Exception as e:
+            print(f"[fidelity] sandbox delete failed: {e}")
+
+
+async def _build_collab_bundle(asset_id: str) -> dict[str, str]:
+    from agora.services.auth.docx_auth import build_docx_collab_bundle
+    b = await build_docx_collab_bundle(asset_id=asset_id, user_id="fidelity-runner")
+    return dict(b)
+
+
+# ── scorecard ─────────────────────────────────────────────────────────
+
+
+def print_scorecard(
+    cases: list[Case],
+    daytona_results: list[dict],
+    diffs_by_name: dict[str, list[str]],
+) -> int:
+    """Print a scorecard. Return exit code (0 = all green)."""
+    print()
+    print("=" * 88)
+    print("FIDELITY SCORECARD — athena-python-docx 0.1.5 vs stock python-docx")
+    print("=" * 88)
+
+    by_name = {r["name"]: r for r in daytona_results}
+    status_icons = {
+        "OK": "✅",
+        "STUB_HIT": "🟡",
+        "UNEXPECTED_PASS": "🟠",
+        "WRONG_EXC": "🔴",
+        "SCRIPT_ERROR": "❌",
+        "SETUP_ERROR": "❌",
+        "EXPORT_ERROR": "❌",
+        "DOWNLOAD_ERROR": "❌",
+    }
+
+    fails = 0
+    for case in cases:
+        r = by_name.get(case.name, {"status": "MISSING", "detail": "not run"})
+        status = r["status"]
+        icon = status_icons.get(status, "❓")
+        diff_suffix = ""
+        if status == "OK" and not case.skip_content_diff:
+            diffs = diffs_by_name.get(case.name, [])
+            if diffs:
+                status = "DIFF"
+                icon = "⚠️"
+                diff_suffix = f"  ({len(diffs)} diff(s))"
+
+        line = f"  {icon} [{status:14}] {case.name:32} — {case.description}{diff_suffix}"
+        print(line)
+        if status not in {"OK", "STUB_HIT"}:
+            if status == "DIFF":
+                for d in diffs_by_name.get(case.name, [])[:3]:
+                    print(f"      ↳ {d}")
+                if len(diffs_by_name.get(case.name, [])) > 3:
+                    print(f"      ↳ …and {len(diffs_by_name[case.name])-3} more")
+            else:
+                detail = r.get("detail", "")
+                if detail:
+                    print(f"      ↳ {detail[:150]}")
+            if status not in {"STUB_HIT"}:
+                fails += 1
+
+    print()
+    n_ok = sum(1 for r in daytona_results if r["status"] == "OK")
+    n_stub = sum(1 for r in daytona_results if r["status"] == "STUB_HIT")
+    n_diff = sum(
+        1 for c in cases
+        if by_name.get(c.name, {}).get("status") == "OK"
+        and diffs_by_name.get(c.name)
+    )
+    total = len(cases)
+    print(f"  {n_ok}/{total} cases ran successfully in Athena SDK")
+    print(f"  {n_stub}/{total} cases hit expected NotImplementedError stubs (known Phase-2 gaps)")
+    print(f"  {n_diff}/{n_ok} successful-run cases have content-level diffs vs stock python-docx")
+    print(f"  exit code: {0 if fails == 0 else 1}")
+    return 0 if fails == 0 else 1
+
+
+# ── main ──────────────────────────────────────────────────────────────
+
+
+async def main() -> int:
+    print(f"[fidelity] case count: {len(CASES)}")
+    print(f"[fidelity] target asset: {DEFAULT_ASSET_ID}")
+    ensure_stock_python_docx()
+    ARTIFACT_DIR.mkdir(parents=True, exist_ok=True)
+
+    # 1) Generate all stock python-docx outputs locally
+    print("[fidelity] running stock python-docx side ...")
+    stock_paths: dict[str, Path] = {}
+    for case in CASES:
+        out = ARTIFACT_DIR / f"stock_{case.name}.docx"
+        try:
+            run_stock_script(case.script, out)
+            stock_paths[case.name] = out
+        except Exception as e:
+            print(f"  !! {case.name} stock subprocess error: {e}")
+
+    # 2) Run the athena side via Daytona in a single batch
+    print("[fidelity] running athena-python-docx via Daytona ...")
+    daytona_results, athena_artifacts = await run_daytona_batch(CASES, DEFAULT_ASSET_ID)
+
+    # 3) Save athena artifacts locally and extract/compare features
+    diffs_by_name: dict[str, list[str]] = {}
+    for case in CASES:
+        if case.name not in athena_artifacts:
+            continue
+        if case.skip_content_diff:
+            continue
+        athena_path = ARTIFACT_DIR / f"athena_{case.name}.docx"
+        athena_path.write_bytes(athena_artifacts[case.name])
+        stock_path = stock_paths.get(case.name)
+        if stock_path is None:
+            continue
+        try:
+            stock_feat = extract_features(stock_path, str(STOCK_SITE_PACKAGES))
+            ath_feat = extract_features(athena_path, str(STOCK_SITE_PACKAGES))
+            diffs_by_name[case.name] = compare(stock_feat, ath_feat)
+        except Exception as e:
+            diffs_by_name[case.name] = [f"extractor error: {e}"]
+
+    # 4) Scorecard
+    code = print_scorecard(CASES, daytona_results, diffs_by_name)
+    print(f"\n[fidelity] artifacts in {ARTIFACT_DIR}")
+    return code
+
+
+if __name__ == "__main__":
+    sys.exit(asyncio.run(main()))