quality-harness 0.5.0__py3-none-any.whl

e2e_harness/AGENTS.md

@@ -0,0 +1,57 @@
+# e2e_harness（确定性契约测试：判定即数据）
+
+## 项目定位与边界
+
+接口结果是确定的：合理输入 → 固定响应，错误输入 → 固定错误码。不关心"回答质量"——那是 `eval_harness` 的事；也不关心压力下的表现——那是 `perf_harness` 的事。
+
+**命脉（判定即数据）**：case 是 spec-case 持有的结构化数据——`input` + `judge.e2e.assert`（`{path,op,value}`）。`engine` 吃 canonical Case → 协议 runner 发请求 → 跑 assert → `verdict`。两个编写前端归一到同一 Case：手写 CaseSet，或 `@case/@spec` 的 NL marker 经 `casegen` 编译。
+
+## 代码地图与核心模块
+
+目录按 pipeline 阶段切，链路 `NL marker →(casegen) canonical Case →(engine) verdict` 在结构上可读：
+
+```
+e2e_harness/
+# ── ① NL authoring 前端：NL marker → canonical Case ──
+├── casegen/
+│   ├── contract.py # @case/@spec marker + CaseSpec/Case + case_hash（marker 形状）
+│   ├── discover.py # AST 扫 @case/@spec(+ *_cases.yaml) → DiscoveredCase（不 import 被扫源码）
+│   └── compiler.py # Compiler seam(Stub/Draft/未来 LLMCompiler) + compile/check/list + casegen CLI
+# ── ② 执行 + 判定即数据：canonical Case → verdict ──
+├── engine.py    # run_case/run_cases：Case → runner → assert 求值 → Verdict
+├── caserun.py   # prepare/execute/judge/cleanup 生命周期、独立阶段 budget 与结果证据
+├── matrix.py    # variant Cartesian product；variant 进入 arm_id + facets
+├── temporal.py  # phase deadline 感知的 poll/retry/consistently
+├── command.py   # 项目原生命令的 transport/process 作用域；保留退出码，不替代 Case Verdict
+├── assertion.py # judge.e2e.assert 求值器：{path,op,value} over response view（结构化、纯函数）
+├── cli.py       # `e2e run case.yaml --base-url … → verdict.json`（+ __main__；JSON/SSE）
+# ── ③ 复用层 ──
+├── runner/      # 协议适配器（sync+async × JSON/SSE）→ 统一 Outcome（engine 复用）
+├── judge/metric/# 软评分 BaseMetric[Outcome]
+└── core/        # common.Experiment 的 E2E 子类 + E2EConfig / load_config + profile/capability gating
+```
+
+> 输入 Case/CaseSet 归 spec-case；输出 verdict 与 report wire 归 `harness_common`。这些 harness-neutral seam 不收进 e2e 包。
+
+## 关键约定
+
+- **主链路**：`case.yaml → engine.run_cases → CasePlan(execute/judge) → BaseRunner → Outcome → CaseRun → verdict.json`；CaseRun 是 common Execution 的 E2E 子类并保留 OperationRun。复杂测试可直接使用完整 `prepare → execute → judge → cleanup`，cleanup 总执行且失败进入 Verdict error。
+- **Experiment**：公共身份是具名验证意图；E2E 子类增加 `service + caseset`。单一配置或单一 Arm 仍然是 Experiment，不以是否存在对照组为前提。
+- **Kernel 对齐**：Runner `Outcome` 是 Observation，一次 CaseRun（`case_id + variant`）是 Unit；一组可复判 CaseRun Unit 构成 Dataset，本次选择的 assertion / deterministic judge / soft metric 直接定义评估侧重点。立即执行并直接写 Verdict 可以融合这些阶段，但必须保留 Outcome、来源 identity 和实际组件配置，使判断能够离线重放；详见 [`../../../docs/kernel.md`](../../../docs/kernel.md#dataset-与反复评估)。
+- **case 两个编写前端**：手写 canonical CaseSet，或 `@case/@spec` NL marker 经 `casegen compile` 编译（`casegen check` 是无-LLM intent-hash 漂移闸门）。`binding.symbol_id + optional spec_id` 对齐 spec-case 的 plural `specs[]`；未填 assertion 的草稿是 error，不会假绿。
+- **Outcome 是 runner↔judge 契约**：`status_code / headers / body / duration_ms / metadata / raw`；SSE events 进 `metadata['events']`，engine 的 `response_view` 暴露 `events[]` / `event_count`。
+- **Config**（`config.yaml`，语言无关 schema）：`service.{name,component,environment,base_url,headers}`，Environment 支持 generic、host、kubernetes 及可选 Host；另有 `runtime.*_timeout` / `discover.{source_root,test_root}`。环境实测值与配置分离，见 [Environment](../../../docs/environment.md)。harness 不内置服务专用 header 名。
+- **verdict**（`spec/verdict-schema.yaml`）：`e2e run` 跑完落 `runs/<scope>/<run-id>/verdict.json`（scope 默认 CaseSet 名，即 Experiment 名）。
+
+- **当前范围**：服务 API 契约测试；Playbook / Script 与 Web、移动端、产品 API Target 的长期边界见领域设计。
+- **跨语言对齐**：Python 与 Go 共享 CaseRun 生命周期、阶段 budget、cleanup 和 Verdict 语义，由仓库根目录 `conformance/e2e/` fixture 约束。
+
+## References
+
+- [E2E、Playbook 与 Target 设计](../../../docs/e2e-harness.md)
+
+- 判定即数据架构（engine / 多编写前端 / 新老收敛）：[`../../../docs/case-unification.md`](../../../docs/case-unification.md)
+- casegen（NL marker → 结构化 case 的 build-time 编译器）：[`../../../docs/casegen.md`](../../../docs/casegen.md)
+- 跨语言约定（case/config schema）：[`../../../spec/conventions.md`](../../../spec/conventions.md)
+- 统一判定出口：[`../../../spec/verdict-schema.yaml`](../../../spec/verdict-schema.yaml) + conventions.md「Run 产物与 verdict 出口」
+- 接入示例：[`../../../examples/api-test/`](../../../examples/api-test/) / [`../../../examples/agent-test/`](../../../examples/agent-test/)

e2e_harness/__init__.py

@@ -0,0 +1,80 @@
+"""Top-level public API.
+
+The canonical path is **judgment-as-data**: a structured ``common.Case`` (``input`` +
+``judge.e2e.assert``) is run by ``engine`` over a protocol ``runner`` → ``verdict`` (CLI
+``e2e run``). Cases are authored as ``case.yaml`` or as ``@case``/``@spec`` markers compiled by
+``casegen``. Submodules:
+
+- ``engine`` / ``assertion``: run a case + evaluate ``judge.e2e.assert`` → verdict
+- ``cli`` / ``casegen``: ``e2e run`` (run) and ``casegen`` (marker → case.yaml)
+- ``runner``: protocol adapters (JSON / SSE) producing ``Outcome``
+- ``core``: config loading + profile gating
+- ``judge.metric``: soft outcome metrics
+- ``casegen``: NL authoring front-end — ``@case``/``@spec`` markers + AST discovery + the
+  compiler that turns them into ``common.Case`` (``contract`` / ``discover`` / ``compiler``)
+
+Dataset-driven evaluation lives in the sibling ``eval_harness`` package; see its README.
+"""
+
+from e2e_harness.casegen.contract import CASE_ID_PATTERN as CASE_ID_PATTERN
+from e2e_harness.casegen.contract import Case as Case
+from e2e_harness.casegen.contract import CaseSpec as CaseSpec
+from e2e_harness.casegen.contract import case as case
+from e2e_harness.casegen.contract import case_hash as case_hash
+from e2e_harness.casegen.contract import get_cases as get_cases
+from e2e_harness.casegen.contract import get_links as get_links
+from e2e_harness.casegen.contract import get_rules as get_rules
+from e2e_harness.casegen.contract import get_spec as get_spec
+from e2e_harness.casegen.contract import get_spec_id as get_spec_id
+from e2e_harness.casegen.contract import link as link
+from e2e_harness.casegen.contract import load_cases_file as load_cases_file
+from e2e_harness.casegen.contract import rule as rule
+from e2e_harness.casegen.contract import spec as spec
+from e2e_harness.casegen.discover import DiscoverConfig as DiscoverConfig
+from e2e_harness.casegen.discover import DiscoveredCase as DiscoveredCase
+from e2e_harness.casegen.discover import discover as discover
+from e2e_harness.assertion import Assertion as Assertion
+from e2e_harness.assertion import run_asserts as run_asserts
+from e2e_harness.core.config import E2EConfig as E2EConfig
+from e2e_harness.core.config import Experiment as Experiment
+from e2e_harness.core.config import load_config as load_config
+from e2e_harness.core.profile import require_capability as require_capability
+from e2e_harness.core.profile import require_profile as require_profile
+from e2e_harness.engine import run_case as run_case
+from e2e_harness.engine import run_cases as run_cases
+from e2e_harness.caserun import Budgets as Budgets
+from e2e_harness.caserun import CaseRef as CaseRef
+from e2e_harness.caserun import CasePlan as CasePlan
+from e2e_harness.caserun import CaseRun as CaseRun
+from e2e_harness.caserun import Fail as Fail
+from e2e_harness.caserun import PhaseContext as PhaseContext
+from e2e_harness.caserun import Skip as Skip
+from e2e_harness.caserun import run_lifecycle as run_lifecycle
+from e2e_harness.matrix import Variant as Variant
+from e2e_harness.matrix import expand_matrix as expand_matrix
+from e2e_harness.judge.metric import BaseMetric as BaseMetric
+from e2e_harness.judge.metric import EventCountMetric as EventCountMetric
+from e2e_harness.judge.metric import LatencyMetric as LatencyMetric
+from e2e_harness.judge.metric import MetricKind as MetricKind
+from e2e_harness.judge.metric import MetricRegistry as MetricRegistry
+from e2e_harness.judge.metric import MetricResult as MetricResult
+from e2e_harness.judge.metric import StatusMetric as StatusMetric
+from e2e_harness.judge.metric import score_outcome as score_outcome
+from e2e_harness.runner.async_base import AsyncBaseRunner as AsyncBaseRunner
+from e2e_harness.runner.async_json_runner import AsyncJSONRunner as AsyncJSONRunner
+from e2e_harness.runner.async_sse_runner import AsyncSSERunner as AsyncSSERunner
+from e2e_harness.engine import E2ERun as E2ERun
+from e2e_harness.engine import run_experiment as run_experiment
+from e2e_harness.runner.base import BaseRunner as BaseRunner
+from e2e_harness.runner.base import Outcome as Outcome
+from e2e_harness.runner.base import Request as Request
+from e2e_harness.reducer import E2EReducer as E2EReducer
+from e2e_harness.runner.events import collect_text as collect_text
+from e2e_harness.runner.events import count_events as count_events
+from e2e_harness.runner.events import events as events
+from e2e_harness.runner.events import find_event as find_event
+from e2e_harness.runner.events import find_event_data as find_event_data
+from e2e_harness.runner.json_runner import JSONRunner as JSONRunner
+from harness_toolbox.sse_parser import SSEEvent as SSEEvent
+from harness_toolbox.sse_parser import SSEParser as SSEParser
+from e2e_harness.runner.sse_runner import SSERunner as SSERunner

e2e_harness/__main__.py

@@ -0,0 +1,10 @@
+"""``python -m e2e_harness …`` → the e2e CLI (see ``e2e_harness.cli``)."""
+
+from __future__ import annotations
+
+import sys
+
+from e2e_harness.cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

e2e_harness/assertion.py

@@ -0,0 +1,144 @@
+"""``judge.e2e.assert`` — the structured, executable form of e2e judgment (判定即数据).
+
+An assertion is a ``{path, op, value}`` triple checked against the SUT response. This is the
+deterministic core of the e2e face: no LLM, no human-written test body — **the case data IS
+the check**. The op set mirrors ``spec/case-schema.yaml`` (judge.e2e.assert) and the legacy
+``judge.assert_judge.AssertJudge`` methods, but driven by *data* instead of method calls, so
+the same checks are now reviewable, hashable (intent-drift) and cross-language.
+
+The evaluator works over a plain **response-view dict** (the Outcome→view bridge lives in
+``e2e_harness.engine``), so it is pure and unit-testable without a live runner. Paths address
+that view uniformly: ``status``, ``body.id``, ``headers.content-type``, ``events[].type``
+(project a field over a list), ``data.items[0].id`` (index).
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from typing import Any, Literal
+
+# op vocabulary — kept in lockstep with spec/case-schema.yaml (judge.e2e.assert.op enum).
+Op = Literal["eq", "not_empty", "contains", "matches", "in", "gt", "gte", "within"]
+
+_MISSING = object()  # path resolved to nothing — kept distinct from a present ``None``
+
+
+@dataclass(frozen=True)
+class Assertion:
+    """One structured check: ``op`` applied to the value at ``path``, against ``value``.
+
+    ``value`` is the comparison operand (omitted for ``not_empty``; a 2-list ``[lo, hi]`` for
+    ``within``; a list for ``in``).
+    """
+
+    path: str
+    op: str
+    value: Any = None
+
+    @classmethod
+    def from_dict(cls, d: dict) -> "Assertion":
+        return cls(path=str(d["path"]), op=str(d["op"]), value=d.get("value"))
+
+
+@dataclass(frozen=True)
+class AssertOutcome:
+    assertion: Assertion
+    ok: bool
+    actual: Any
+    detail: str  # human-facing why (empty when ok) — surfaced in the verdict reason
+
+
+def _tokens(path: str) -> list[str]:
+    """``a.b[].c`` → ``["a", "b", "[]", "c"]``;  ``x[0]`` → ``["x", "[0]"]``."""
+    out: list[str] = []
+    for part in path.split("."):
+        m = re.match(r"^([^\[\]]*)(.*)$", part)
+        key, brackets = (m.group(1), m.group(2)) if m else (part, "")
+        if key:
+            out.append(key)
+        out.extend(re.findall(r"\[\d*\]", brackets))
+    return out
+
+
+def _resolve(data: Any, tokens: list[str]) -> Any:
+    if not tokens:
+        return data
+    tok, rest = tokens[0], tokens[1:]
+    if tok == "[]":  # project the remaining path over every list element
+        if not isinstance(data, list):
+            return _MISSING
+        return [_resolve(el, rest) for el in data]
+    if tok.startswith("[") and tok.endswith("]"):  # list index, e.g. "[0]"
+        try:
+            return _resolve(data[int(tok[1:-1])], rest)
+        except (ValueError, IndexError, TypeError, KeyError):
+            return _MISSING
+    if isinstance(data, dict) and tok in data:  # dict key
+        return _resolve(data[tok], rest)
+    return _MISSING
+
+
+def resolve_path(data: dict, path: str) -> Any:
+    """Value at ``path`` in the response view, or ``_MISSING`` when absent."""
+    return _resolve(data, _tokens(path))
+
+
+def _is_empty(v: Any) -> bool:
+    # ``_MISSING`` and empty containers/strings/None are empty; 0 / False are *values*.
+    return v is _MISSING or v is None or v == "" or v == [] or v == {} or v == ()
+
+
+def check(op: str, actual: Any, value: Any) -> tuple[bool, str]:
+    """Apply one op to a resolved value; return ``(ok, short detail)``. Empty detail iff ok."""
+    if op == "not_empty":
+        return (not _is_empty(actual)), ("" if not _is_empty(actual) else "is empty")
+    if actual is _MISSING:
+        return False, "path not found"
+    if op == "eq":
+        ok = actual == value
+        return ok, "" if ok else f"expected {value!r}, got {actual!r}"
+    if op == "contains":
+        try:
+            ok = value in actual
+        except TypeError:
+            ok = False
+        return ok, "" if ok else f"{actual!r} does not contain {value!r}"
+    if op == "matches":
+        ok = re.search(str(value), str(actual)) is not None
+        return ok, "" if ok else f"{actual!r} does not match /{value}/"
+    if op == "in":
+        ok = isinstance(value, (list, tuple, set)) and actual in value
+        return ok, "" if ok else f"{actual!r} not in {value!r}"
+    if op in ("gt", "gte"):
+        try:
+            ok = actual > value if op == "gt" else actual >= value
+        except TypeError:
+            return False, f"{actual!r} not comparable to {value!r}"
+        return ok, "" if ok else f"{actual!r} not {op} {value!r}"
+    if op == "within":
+        try:
+            lo, hi = value
+            ok = lo <= actual <= hi
+        except (TypeError, ValueError):
+            return False, f"within needs [lo, hi], got {value!r}"
+        return ok, "" if ok else f"{actual!r} not within [{lo}, {hi}]"
+    return False, f"unknown op {op!r}"
+
+
+def evaluate(assertion: Assertion, data: dict) -> AssertOutcome:
+    actual = resolve_path(data, assertion.path)
+    ok, why = check(assertion.op, actual, assertion.value)
+    detail = "" if ok else f"{assertion.path} {assertion.op} {assertion.value!r}: {why}"
+    return AssertOutcome(
+        assertion=assertion,
+        ok=ok,
+        actual=(None if actual is _MISSING else actual),
+        detail=detail,
+    )
+
+
+def run_asserts(asserts: list[Assertion], data: dict) -> list[AssertOutcome]:
+    """Evaluate every assertion against the response view — all, not short-circuit (so one
+    run surfaces every failing check, not just the first)."""
+    return [evaluate(a, data) for a in asserts]

e2e_harness/casegen/__init__.py

@@ -0,0 +1,41 @@
+"""casegen — the NL-case authoring front-end, one home for ``NL marker → common.Case``.
+
+The pushed way to write an e2e case is NOT a hand-written ``case.yaml`` but an ``@case``/``@spec``
+NL marker in the service's own source; casegen is the compiler that turns those into the
+structured ``common.Case`` that ``engine`` executes as data (see ``docs/casegen.md``). The three
+files here are one pipeline stage, kept together so the front-end has a structural home:
+
+- ``contract``: ``@case`` / ``@spec`` / ``@rule`` / ``@link`` decorators + ``CaseSpec`` / ``Case`` + ``case_hash`` (the marker shape)
+- ``discover``: AST scanner for ``@case`` / ``@spec`` (+ companion ``*_cases.yaml``) → ``DiscoveredCase``
+- ``compiler``: ``Compiler`` seam (``StubCompiler`` / ``DraftCompiler`` / future ``LLMCompiler``) +
+  ``compile``/``check``/``list`` orchestration + the ``casegen`` CLI
+
+The judgment then runs as data via ``engine`` — no generated pytest body.
+"""
+
+from e2e_harness.casegen.compiler import CheckReport as CheckReport
+from e2e_harness.casegen.compiler import Compiler as Compiler
+from e2e_harness.casegen.compiler import CompileReport as CompileReport
+from e2e_harness.casegen.compiler import DraftCompiler as DraftCompiler
+from e2e_harness.casegen.compiler import StubCompiler as StubCompiler
+from e2e_harness.casegen.compiler import check_cases as check_cases
+from e2e_harness.casegen.compiler import compile_cases as compile_cases
+from e2e_harness.casegen.compiler import list_cases as list_cases
+from e2e_harness.casegen.compiler import main as main
+from e2e_harness.casegen.contract import CASE_ID_PATTERN as CASE_ID_PATTERN
+from e2e_harness.casegen.contract import Case as Case
+from e2e_harness.casegen.contract import CaseSpec as CaseSpec
+from e2e_harness.casegen.contract import case as case
+from e2e_harness.casegen.contract import case_hash as case_hash
+from e2e_harness.casegen.contract import get_cases as get_cases
+from e2e_harness.casegen.contract import get_links as get_links
+from e2e_harness.casegen.contract import get_rules as get_rules
+from e2e_harness.casegen.contract import get_spec as get_spec
+from e2e_harness.casegen.contract import get_spec_id as get_spec_id
+from e2e_harness.casegen.contract import link as link
+from e2e_harness.casegen.contract import load_cases_file as load_cases_file
+from e2e_harness.casegen.contract import rule as rule
+from e2e_harness.casegen.contract import spec as spec
+from e2e_harness.casegen.discover import DiscoverConfig as DiscoverConfig
+from e2e_harness.casegen.discover import DiscoveredCase as DiscoveredCase
+from e2e_harness.casegen.discover import discover as discover

e2e_harness/casegen/compiler.py

@@ -0,0 +1,288 @@
+"""casegen — build-time compiler: NL markers (``@case``/``@spec``) → structured ``case.yaml``.
+
+The build-time half of judgment-as-data (see ``docs/casegen.md``): ``discover`` extracts the NL
+markers, a ``Compiler`` turns each into a structured ``common.Case`` (with ``judge.e2e.assert``),
+and casegen writes a committed ``case.yaml`` that ``e2e run`` executes. An ``intent hash`` (the
+marker's ``case_hash`` over NL + ``@spec``) is recorded per case in a top-level ``compiled_from``
+map, so ``check`` is a deterministic, no-LLM drift gate and ``compile`` only re-compiles markers
+whose intent actually drifted (preserving human/LLM-filled asserts on the rest).
+
+This module is step ① — the deterministic skeleton (Compiler protocol + ``StubCompiler`` +
+orchestration + ``check``). The real fill (``DraftCompiler`` → claude, then ``LLMCompiler`` →
+``harness_toolbox.llm``) is a drop-in for ``Compiler``; nothing else changes.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Protocol
+
+import yaml
+
+from spec_case.model import Binding, Case, case_to_raw, load_caseset, validate
+from e2e_harness.casegen.discover import DiscoverConfig, DiscoveredCase, discover
+
+
+class Compiler(Protocol):
+    """NL marker → structured ``common.Case``. The one seam the LLM lives behind."""
+
+    def compile(self, nl: DiscoveredCase) -> Case: ...
+
+
+class StubCompiler:
+    """Deterministic, no-LLM — for testing orchestration/drift and as the abstraction's anchor.
+
+    Projects only the identity (``id``) and leaves ``judge.e2e.assert`` empty; a real compiler
+    (``DraftCompiler`` / ``LLMCompiler``) fills the asserts from the NL intent.
+    """
+
+    def compile(self, nl: DiscoveredCase) -> Case:
+        return Case(
+            id=nl.case.id,
+            input={},
+            judge={"e2e": {"assert": []}},
+            binding=_binding(nl),
+        )
+
+
+class DraftCompiler:
+    """v1, no LLM — a self-contained, fillable draft (mirrors eval-api's "hand to claude" loop).
+
+    Projects the identity and **inlines the NL intent into ``desc`` as fill-guidance**, leaving
+    ``input`` and ``judge.e2e.assert`` empty for a human / claude to complete. The guidance lives
+    in ``desc`` (round-trips through yaml, excluded from ``case_hash``, human + LLM readable) so the
+    structured ``input`` / ``judge`` namespaces stay clean. An unfilled draft (e2e face declared, no
+    asserts) → ``error`` in ``e2e run`` — it can't be hidden green by a sibling pass; fill the
+    asserts to make it judgeable. (The future ``LLMCompiler`` fills ``input`` + ``assert`` directly.)
+
+    Why not auto-fill asserts by rule: ``input`` is NL (e.g. "req body"), can't be parsed to a real
+    request deterministically — and asserts without a real ``input`` would false-fail at run. So a
+    draft fills *neither*; both are completed together.
+    """
+
+    def compile(self, nl: DiscoveredCase) -> Case:
+        c = nl.case
+        guide = [c.desc] if c.desc else []
+        guide.append(
+            "—— casegen draft: fill input + judge.e2e.assert from the intent below ——"
+        )
+        for label, val in (
+            ("input(NL)", c.input),
+            ("expect", c.expect),
+            ("forbid", c.forbid),
+            ("spec", nl.spec_text),
+        ):
+            if val:
+                guide.append(f"{label}: {val}")
+        return Case(
+            id=c.id,
+            input={},
+            desc="\n".join(guide),
+            judge={"e2e": {"assert": []}},
+            binding=_binding(nl),
+        )
+
+
+# --------------------------------------------------------------------------- serialization
+
+
+def _case_to_dict(case: Case) -> dict:
+    """Delegate canonical Case serialization to spec-case."""
+    return case_to_raw(case)
+
+
+def _binding(nl: DiscoveredCase) -> Binding:
+    return Binding(
+        symbol_id=nl.symbol_id,
+        spec=nl.spec_text or "",
+        spec_id=nl.spec_id,
+    )
+
+
+def _write_caseset(
+    out: Path, caseset: str, cases: list[Case], compiled_from: dict[str, str]
+) -> None:
+    doc = {
+        "caseset": caseset,
+        # casegen-owned metadata: case_id → intent hash it was compiled from. `load_caseset`
+        # ignores this top-level key; `check` reads it to detect marker drift.
+        "compiled_from": dict(sorted(compiled_from.items())),
+        "cases": [_case_to_dict(c) for c in cases],
+    }
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(
+        yaml.safe_dump(doc, sort_keys=False, allow_unicode=True), encoding="utf-8"
+    )
+
+
+def _read_compiled(out: Path) -> tuple[dict[str, Case], dict[str, str]]:
+    """Existing compiled file → (cases by id, ``compiled_from`` map). Empty if absent."""
+    if not out.exists():
+        return {}, {}
+    cases = {c.id: c for c in load_caseset(out).cases}
+    raw = yaml.safe_load(out.read_text()) or {}
+    return cases, dict(raw.get("compiled_from") or {})
+
+
+# --------------------------------------------------------------------------- orchestration
+
+
+@dataclass
+class CompileReport:
+    compiled: list[str] = field(default_factory=list)  # new or drifted → (re)compiled
+    reused: list[str] = field(
+        default_factory=list
+    )  # intent unchanged → existing kept (edits preserved)
+    dropped: list[str] = field(
+        default_factory=list
+    )  # in old file but no longer a marker (orphaned)
+
+
+def compile_cases(
+    config: DiscoverConfig, compiler: Compiler, out: Path, *, caseset: str
+) -> CompileReport:
+    """(Re)compile drifted/new markers → write ``out``. Markers whose intent is unchanged keep
+    their existing compiled case verbatim — so a human/LLM's filled asserts survive re-runs."""
+    discovered = discover(config)
+    prev_cases, prev_hashes = _read_compiled(out)
+    rep = CompileReport()
+    cases: list[Case] = []
+    compiled_from: dict[str, str] = {}
+    for dc in discovered:
+        cid = dc.case.id
+        compiled_from[cid] = dc.case_hash
+        if cid in prev_cases and prev_hashes.get(cid) == dc.case_hash:
+            cases.append(
+                prev_cases[cid]
+            )  # intent unchanged → keep (don't clobber filled asserts)
+            rep.reused.append(cid)
+        else:
+            cases.append(compiler.compile(dc))  # new or drifted → recompile
+            rep.compiled.append(cid)
+    rep.dropped = sorted(set(prev_cases) - {dc.case.id for dc in discovered})
+    _write_caseset(out, caseset, cases, compiled_from)
+    return rep
+
+
+@dataclass
+class CheckReport:
+    drifted: list[str] = field(
+        default_factory=list
+    )  # marker intent ≠ compiled (or never compiled)
+    orphaned: list[str] = field(default_factory=list)  # compiled but marker gone
+
+    @property
+    def ok(self) -> bool:
+        return not self.drifted and not self.orphaned
+
+
+def check_cases(config: DiscoverConfig, out: Path) -> CheckReport:
+    """Deterministic, no-LLM drift gate: compiled ``case.yaml`` must be in sync with the markers."""
+    discovered = discover(config)
+    _, prev_hashes = _read_compiled(out)
+    marker_ids = {dc.case.id for dc in discovered}
+    rep = CheckReport()
+    rep.drifted = sorted(
+        dc.case.id for dc in discovered if prev_hashes.get(dc.case.id) != dc.case_hash
+    )
+    rep.orphaned = sorted(set(prev_hashes) - marker_ids)
+    return rep
+
+
+def list_cases(config: DiscoverConfig, out: Path) -> list[tuple[str, str]]:
+    """(case_id, status) where status ∈ in-sync / drifted / new / orphaned."""
+    discovered = discover(config)
+    prev_cases, prev_hashes = _read_compiled(out)
+    rows: list[tuple[str, str]] = []
+    for dc in discovered:
+        cid = dc.case.id
+        if cid not in prev_hashes:
+            rows.append((cid, "new"))
+        elif prev_hashes[cid] != dc.case_hash:
+            rows.append((cid, "drifted"))
+        else:
+            rows.append((cid, "in-sync"))
+    rows += [
+        (cid, "orphaned")
+        for cid in sorted(set(prev_cases) - {dc.case.id for dc in discovered})
+    ]
+    return rows
+
+
+# --------------------------------------------------------------------------- CLI
+
+
+def _config(args: argparse.Namespace) -> DiscoverConfig:
+    src = Path(args.source)
+    if not src.is_dir():
+        raise SystemExit(f"casegen: --source {src} is not a directory")
+    return DiscoverConfig(source_root=src, test_root=src)  # test_root unused by casegen
+
+
+def _out(args: argparse.Namespace) -> Path:
+    return Path(args.out)
+
+
+def cmd_list(args: argparse.Namespace) -> int:
+    for cid, status in list_cases(_config(args), _out(args)):
+        print(f"  {status:<9} {cid}")
+    return 0
+
+
+def cmd_compile(args: argparse.Namespace) -> int:
+    rep = compile_cases(
+        _config(args), DraftCompiler(), _out(args), caseset=args.caseset
+    )
+    print(
+        f"casegen: compiled {len(rep.compiled)}, reused {len(rep.reused)}, dropped {len(rep.dropped)} → {args.out}"
+    )
+    if rep.compiled:
+        validate(load_caseset(_out(args)))  # the written file is a valid case set
+    return 0
+
+
+def cmd_check(args: argparse.Namespace) -> int:
+    rep = check_cases(_config(args), _out(args))
+    if rep.ok:
+        print("casegen: in sync ✓")
+        return 0
+    for cid in rep.drifted:
+        print(f"  drifted   {cid}  (marker changed — re-run `casegen compile`)")
+    for cid in rep.orphaned:
+        print(f"  orphaned  {cid}  (compiled case has no marker)")
+    return 1
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="casegen", description="Compile NL @case markers → structured case.yaml."
+    )
+    sub = parser.add_subparsers(dest="cmd", required=True)
+    for name, fn, help_ in [
+        ("list", cmd_list, "list markers + drift status"),
+        ("compile", cmd_compile, "(re)compile drifted/new markers → case.yaml"),
+        ("check", cmd_check, "drift gate (no LLM): markers vs compiled case.yaml"),
+    ]:
+        p = sub.add_parser(name, help=help_)
+        p.add_argument(
+            "--source", required=True, help="dir to scan for @case/@spec markers"
+        )
+        p.add_argument("--out", required=True, help="compiled case.yaml path")
+        if name == "compile":
+            p.add_argument(
+                "--caseset", default="cases", help="case set name written into the file"
+            )
+        p.set_defaults(func=fn)
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())