trace2eval-cli 0.2.1__tar.gz → 0.3.0__tar.gz

{trace2eval_cli-0.2.1 → trace2eval_cli-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: trace2eval-cli
-Version: 0.2.1
+Version: 0.3.0
 Summary: Turn production LLM traces into a regression eval set. Zero dependencies.
 Author: rfioly
 License: MIT
@@ -37,7 +37,9 @@ Dynamic: license-file
 零依赖、不用 API key、不调模型。同样的日志进，同样的用例出。
 
 [![CI](https://github.com/rfioly/trace2eval/actions/workflows/ci.yml/badge.svg)](https://github.com/rfioly/trace2eval/actions/workflows/ci.yml)
-[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+[![PyPI](https://img.shields.io/pypi/v/trace2eval-cli)](https://pypi.org/project/trace2eval-cli/)
+[![Downloads](https://img.shields.io/pypi/dm/trace2eval-cli)](https://pypi.org/project/trace2eval-cli/)
+[![Python versions](https://img.shields.io/pypi/pyversions/trace2eval-cli)](https://pypi.org/project/trace2eval-cli/)
 [![Dependencies](https://img.shields.io/badge/dependencies-0-brightgreen)](#)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
 
@@ -46,7 +48,7 @@ Dynamic: license-file
 ## 快速开始
 
 ```bash
-pip install -e .
+pip install trace2eval-cli
 
 # 1. 把日志变成用例集
 trace2eval build traces.jsonl -o evalset/
@@ -60,6 +62,9 @@ trace2eval check --baseline runs/baseline.json --current runs/current.json
 
 需要 Python 3.10+。
 
+> 分发名带 `-cli` 后缀，因为 `trace2eval` 这个名字在 PyPI 上被一个空壳占着。
+> **命令没变**——装完还是敲 `trace2eval`。
+
 ---
 
 ## 三个命令
@@ -79,6 +84,7 @@ trace2eval check --baseline runs/baseline.json --current runs/current.json
 - **一个问题一个用例。** 被问了 200 次的问题只变成一个用例，并且老实记录它代表 200 次调用。
 - **形状从干净答案学，不从失败那条学。** 失败种子的长度下限来自同问题的干净答案中位数。
 - **每个用例自检。** 报告里会标明哪些用例的检查**抓不住**它当初为什么坏，不混进"已覆盖"里。
+- **抓不住的用例可以人工补。** 失败发生在"调用之外"（用户重问、差评）而不在答案里时，自动检查无能为力——报告会把它们单列出来，`build` 同时生成一份待填模板，你写好期望答案后它们就变成真正的门禁。标注按**问题**索引而不是按用例编号，所以重新生成用例集不会失效。
 - **相似度可换。** `--similarity module:function` 换成你自己的实现，包本身不引依赖。
 - **天生适合 CI。** `check` 检出回归返回非零退出码。
 
@@ -190,10 +196,35 @@ flowchart LR
 
 ---
 
+## 人工标注（可选）
+
+有些失败发生在"调用之外"——用户又问了一遍、点了差评、接口慢到超时。**这种答案本身往往是没问题的**，所以任何对输出文本的自动检查都抓不到它。工具能识别出这些用例，但没法凭空想出"正确答案长什么样"，那得人来写。
+
+`build` 会在用例集旁边生成一份待填模板，列出的正是这些用例：
+
+```jsonl
+{"input": "修改手机号", "expect": {}, "note": "TODO: fill in expect. (user_retried)"}
+```
+
+把 `expect` 填好，另存为 `expectations.jsonl`，重新 `build` 就生效了：
+
+```jsonl
+{"input": "修改手机号", "expect": {"min_chars": 20, "contains": ["手机号"]}, "note": "用户重问说明第一版没解决，正确答案要说清验证流程"}
+```
+
+`expect` 里能用的键和 trace 自己的 `expect` 块**完全一致**：`min_chars`、`max_chars`、`contains`、`not_contains`、`regex`、`json`、`not_fallback`。
+
+两条设计上要紧的地方：
+
+- **按问题索引，不按用例编号。** 用例编号是按分数排序临时分配的，加一条日志就可能全部重排。标注要是挂在 `case-011` 上，下次重建就会**悄悄脱离**——而一个脱了钩的标注比没有标注更糟，因为那个用例看起来还是有覆盖的。
+- **标注不会改变自检结论。** 它不会让用例"复现失败"（行为型失败本来就复现不了），它是换了个目标：从"让这个失败不可能再发生"变成"答案必须长这样"。报告里两个数分开写，不合并。
+
+---
+
 ## 已知边界
 
 - **相似度是字符 n-gram**，不含相同字符的改写会被当成两个问题。可用 `--similarity` 换成你自己的实现。
-- **行为型失败抓不住。** 样本日志 16 个用例里有 3 个，失败原因只是"用户又问了一遍"，输出本身没毛病——任何对输出文本的确定性检查都抓不到。报告里已单独标出。
+- **行为型失败自动抓不住，但可以人工补。** 样本日志 16 个用例里有 3 个，失败原因只是"用户又问了一遍"，输出本身没毛病——任何对输出文本的确定性检查都抓不到，这是方法的边界而非欠账。它们现在由 `evalset/expectations.jsonl` 里的人工标注兜住（仓库里那份是示例）。报告会**分开**说明"有几个抓不住""有几个已有标注覆盖"，不会含糊成一个数。
 - **比较成本是 O(用例数 × 每簇不同问法数)**，不是 O(日志大小)：同一问题问 5000 次只留 1 个待比对指纹。`MAX_DISTINCT_FINGERPRINTS = 512` 是拍的保护值，没压测过。
 - **用例仍是提案**，但会自检：`report.md` 里单列"需要人看一眼"的用例。
 
@@ -203,16 +234,17 @@ flowchart LR
 
 ```
 src/trace2eval/
-  schema.py    trace 加载、字段别名、容错解析
-  signals.py   一条 trace 为什么值得测
-  select.py    相似度、聚类、用例构建
-  checks.py    确定性输出检查
-  runner.py    评分与回归对比
-  report.py    markdown 渲染
-  matchers.py  可替换的相似度实现
-tests/         42 个测试
-examples/      一份 42 行样本日志 + 一次基线 / 一次回归运行
-evalset/       提交进仓库的构建产物
+  schema.py       trace 加载、字段别名、容错解析
+  signals.py      一条 trace 为什么值得测
+  select.py       相似度、聚类、用例构建
+  checks.py       确定性输出检查
+  runner.py       评分与回归对比
+  report.py       markdown 渲染
+  matchers.py     可替换的相似度实现
+  annotations.py  人工标注的加载与合并
+tests/            62 个测试
+examples/         一份 42 行样本日志 + 一次基线 / 一次回归运行
+evalset/          提交进仓库的构建产物 + 示例标注
 ```
 
 ## 开发

{trace2eval_cli-0.2.1 → trace2eval_cli-0.3.0}/README.md

@@ -9,7 +9,9 @@
 零依赖、不用 API key、不调模型。同样的日志进，同样的用例出。
 
 [![CI](https://github.com/rfioly/trace2eval/actions/workflows/ci.yml/badge.svg)](https://github.com/rfioly/trace2eval/actions/workflows/ci.yml)
-[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+[![PyPI](https://img.shields.io/pypi/v/trace2eval-cli)](https://pypi.org/project/trace2eval-cli/)
+[![Downloads](https://img.shields.io/pypi/dm/trace2eval-cli)](https://pypi.org/project/trace2eval-cli/)
+[![Python versions](https://img.shields.io/pypi/pyversions/trace2eval-cli)](https://pypi.org/project/trace2eval-cli/)
 [![Dependencies](https://img.shields.io/badge/dependencies-0-brightgreen)](#)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
 
@@ -18,7 +20,7 @@
 ## 快速开始
 
 ```bash
-pip install -e .
+pip install trace2eval-cli
 
 # 1. 把日志变成用例集
 trace2eval build traces.jsonl -o evalset/
@@ -32,6 +34,9 @@ trace2eval check --baseline runs/baseline.json --current runs/current.json
 
 需要 Python 3.10+。
 
+> 分发名带 `-cli` 后缀，因为 `trace2eval` 这个名字在 PyPI 上被一个空壳占着。
+> **命令没变**——装完还是敲 `trace2eval`。
+
 ---
 
 ## 三个命令
@@ -51,6 +56,7 @@ trace2eval check --baseline runs/baseline.json --current runs/current.json
 - **一个问题一个用例。** 被问了 200 次的问题只变成一个用例，并且老实记录它代表 200 次调用。
 - **形状从干净答案学，不从失败那条学。** 失败种子的长度下限来自同问题的干净答案中位数。
 - **每个用例自检。** 报告里会标明哪些用例的检查**抓不住**它当初为什么坏，不混进"已覆盖"里。
+- **抓不住的用例可以人工补。** 失败发生在"调用之外"（用户重问、差评）而不在答案里时，自动检查无能为力——报告会把它们单列出来，`build` 同时生成一份待填模板，你写好期望答案后它们就变成真正的门禁。标注按**问题**索引而不是按用例编号，所以重新生成用例集不会失效。
 - **相似度可换。** `--similarity module:function` 换成你自己的实现，包本身不引依赖。
 - **天生适合 CI。** `check` 检出回归返回非零退出码。
 
@@ -162,10 +168,35 @@ flowchart LR
 
 ---
 
+## 人工标注（可选）
+
+有些失败发生在"调用之外"——用户又问了一遍、点了差评、接口慢到超时。**这种答案本身往往是没问题的**，所以任何对输出文本的自动检查都抓不到它。工具能识别出这些用例，但没法凭空想出"正确答案长什么样"，那得人来写。
+
+`build` 会在用例集旁边生成一份待填模板，列出的正是这些用例：
+
+```jsonl
+{"input": "修改手机号", "expect": {}, "note": "TODO: fill in expect. (user_retried)"}
+```
+
+把 `expect` 填好，另存为 `expectations.jsonl`，重新 `build` 就生效了：
+
+```jsonl
+{"input": "修改手机号", "expect": {"min_chars": 20, "contains": ["手机号"]}, "note": "用户重问说明第一版没解决，正确答案要说清验证流程"}
+```
+
+`expect` 里能用的键和 trace 自己的 `expect` 块**完全一致**：`min_chars`、`max_chars`、`contains`、`not_contains`、`regex`、`json`、`not_fallback`。
+
+两条设计上要紧的地方：
+
+- **按问题索引，不按用例编号。** 用例编号是按分数排序临时分配的，加一条日志就可能全部重排。标注要是挂在 `case-011` 上，下次重建就会**悄悄脱离**——而一个脱了钩的标注比没有标注更糟，因为那个用例看起来还是有覆盖的。
+- **标注不会改变自检结论。** 它不会让用例"复现失败"（行为型失败本来就复现不了），它是换了个目标：从"让这个失败不可能再发生"变成"答案必须长这样"。报告里两个数分开写，不合并。
+
+---
+
 ## 已知边界
 
 - **相似度是字符 n-gram**，不含相同字符的改写会被当成两个问题。可用 `--similarity` 换成你自己的实现。
-- **行为型失败抓不住。** 样本日志 16 个用例里有 3 个，失败原因只是"用户又问了一遍"，输出本身没毛病——任何对输出文本的确定性检查都抓不到。报告里已单独标出。
+- **行为型失败自动抓不住，但可以人工补。** 样本日志 16 个用例里有 3 个，失败原因只是"用户又问了一遍"，输出本身没毛病——任何对输出文本的确定性检查都抓不到，这是方法的边界而非欠账。它们现在由 `evalset/expectations.jsonl` 里的人工标注兜住（仓库里那份是示例）。报告会**分开**说明"有几个抓不住""有几个已有标注覆盖"，不会含糊成一个数。
 - **比较成本是 O(用例数 × 每簇不同问法数)**，不是 O(日志大小)：同一问题问 5000 次只留 1 个待比对指纹。`MAX_DISTINCT_FINGERPRINTS = 512` 是拍的保护值，没压测过。
 - **用例仍是提案**，但会自检：`report.md` 里单列"需要人看一眼"的用例。
 
@@ -175,16 +206,17 @@ flowchart LR
 
 ```
 src/trace2eval/
-  schema.py    trace 加载、字段别名、容错解析
-  signals.py   一条 trace 为什么值得测
-  select.py    相似度、聚类、用例构建
-  checks.py    确定性输出检查
-  runner.py    评分与回归对比
-  report.py    markdown 渲染
-  matchers.py  可替换的相似度实现
-tests/         42 个测试
-examples/      一份 42 行样本日志 + 一次基线 / 一次回归运行
-evalset/       提交进仓库的构建产物
+  schema.py       trace 加载、字段别名、容错解析
+  signals.py      一条 trace 为什么值得测
+  select.py       相似度、聚类、用例构建
+  checks.py       确定性输出检查
+  runner.py       评分与回归对比
+  report.py       markdown 渲染
+  matchers.py     可替换的相似度实现
+  annotations.py  人工标注的加载与合并
+tests/            62 个测试
+examples/         一份 42 行样本日志 + 一次基线 / 一次回归运行
+evalset/          提交进仓库的构建产物 + 示例标注
 ```
 
 ## 开发

{trace2eval_cli-0.2.1 → trace2eval_cli-0.3.0}/pyproject.toml

@@ -8,7 +8,7 @@ build-backend = "setuptools.build_meta"
 # suffix on the distribution name. The console script is still "trace2eval",
 # so the command users type is unchanged.
 name = "trace2eval-cli"
-version = "0.2.1"
+version = "0.3.0"
 description = "Turn production LLM traces into a regression eval set. Zero dependencies."
 readme = "README.md"
 requires-python = ">=3.10"

{trace2eval_cli-0.2.1 → trace2eval_cli-0.3.0}/src/trace2eval/__init__.py

@@ -16,6 +16,13 @@ Everything here is deterministic and offline. No model is called, no API key is
 needed, and the same log always produces the same case set.
 """
 
+from .annotations import (
+    Annotation,
+    AnnotationSet,
+    load_annotations,
+    merge_checks,
+    render_template,
+)
 from .checks import CheckResult, detect_fallback, run_check, run_checks
 from .matchers import char_bigram_jaccard, word_jaccard
 from .runner import RunMetrics, compare_runs, load_outputs, run_cases
@@ -32,10 +39,12 @@ from .select import (
 )
 from .signals import DEFAULT_WEIGHTS, Signal, build_context, compute_signals, score
 
-__version__ = "0.2.1"
+__version__ = "0.3.0"
 
 __all__ = [
     "__version__",
+    "Annotation",
+    "AnnotationSet",
     "CheckResult",
     "DEFAULT_DEDUP_THRESHOLD",
     "DEFAULT_WEIGHTS",
@@ -50,11 +59,14 @@ __all__ = [
     "compare_runs",
     "compute_signals",
     "detect_fallback",
+    "load_annotations",
     "load_matcher",
     "load_outputs",
     "load_traces",
+    "merge_checks",
     "overlap_coefficient",
     "parse_trace",
+    "render_template",
     "run_check",
     "run_checks",
     "run_cases",

trace2eval_cli-0.3.0/src/trace2eval/annotations.py

@@ -0,0 +1,212 @@
+"""Human-supplied expected answers.
+
+The gap this closes: a case whose failure was *behavioural* -- the user asked
+again, the down-vote landed, the call was just slow -- has a perfectly acceptable
+answer attached to it. No deterministic check on that text can reproduce the
+failure, because the failure was never in the text.
+
+The tool can detect those cases and say so (that is what the
+``failure_not_reproduced`` verdict is for). It cannot invent the missing
+expectation. A person has to write down what the answer should have looked like.
+
+The design constraint that matters is **survival**. Generated case ids are not
+stable -- they are assigned in descending score order, so one new trace can
+renumber everything. An annotation keyed on ``case-011`` would silently detach
+itself the next time the case set is rebuilt, and a detached annotation is worse
+than no annotation: the case would quietly go back to being weak while still
+looking annotated.
+
+So annotations are keyed on the *question*, not on the case id:
+
+    {"input": "修改手机号", "expect": {"min_chars": 20, "contains": ["验证码"]}}
+
+``input`` is matched after the same folding used for clustering (case, whitespace
+and punctuation stripped), so the question can be rephrased in the file without
+breaking the link. ``build`` writes a template listing every case that needs one,
+so filling this in is closer to editing a form than writing a file from scratch.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+#: Check types a human annotation may set. Deliberately the same vocabulary a
+#: trace's own ``expect`` block uses, so there is one thing to learn rather than
+#: two -- and so ``_checks_from_expect`` can be reused rather than reimplemented.
+ANNOTATABLE_CHECKS = frozenset(
+    {"not_fallback", "min_chars", "max_chars", "regex", "contains", "not_contains", "json"}
+)
+
+
+@dataclass
+class Annotation:
+    """One human-written expectation."""
+
+    input: str
+    expect: dict[str, Any] = field(default_factory=dict)
+    note: str = ""
+    line_number: int = 0
+
+    def is_empty(self) -> bool:
+        """True when the entry is still the unfilled template placeholder."""
+        return not self.expect
+
+
+@dataclass
+class AnnotationSet:
+    """Every annotation found in one file, plus what was skipped loading it."""
+
+    annotations: list[Annotation] = field(default_factory=list)
+    skipped: list[str] = field(default_factory=list)
+    path: Path | None = None
+
+    def __bool__(self) -> bool:
+        return any(not a.is_empty() for a in self.annotations)
+
+    @property
+    def filled(self) -> list[Annotation]:
+        return [a for a in self.annotations if not a.is_empty()]
+
+    def lookup(self, key: str) -> Annotation | None:
+        """The annotation for ``key`` (an already-folded input string), if any."""
+        for annotation in self.filled:
+            if _fold(annotation.input) == key:
+                return annotation
+        return None
+
+
+def fold_input(text: str) -> str:
+    """Fold an input for matching purposes.
+
+    Kept local rather than imported from :mod:`trace2eval.select` so this module
+    stays free of the selection machinery -- annotations are loaded before
+    selection runs, and a cycle here would be easy to introduce and annoying to
+    find.
+    """
+    keep = []
+    for char in text.strip().lower():
+        if char.isspace() or char in "，。！？、；：,.!?;:\"'“”‘’()（）[]【】":
+            continue
+        keep.append(char)
+    return "".join(keep)
+
+
+#: Same function the lookup uses on annotation inputs. Exported so callers
+#: cannot accidentally match against a differently-folded key.
+_fold = fold_input
+
+
+def parse_annotation(payload: dict[str, Any], line_number: int) -> Annotation:
+    raw_expect = payload.get("expect")
+    expect = raw_expect if isinstance(raw_expect, dict) else {}
+    unknown = set(expect) - ANNOTATABLE_CHECKS
+    if unknown:
+        raise ValueError(
+            f"unknown check type(s) {sorted(unknown)}; "
+            f"allowed: {sorted(ANNOTATABLE_CHECKS)}"
+        )
+    return Annotation(
+        input=str(payload.get("input", "")).strip(),
+        expect=expect,
+        note=str(payload.get("note", "")).strip(),
+        line_number=line_number,
+    )
+
+
+def load_annotations(path: Path) -> AnnotationSet:
+    """Load an expectations file. Missing file is not an error -- it means none.
+
+    A malformed line is collected rather than raised. Losing every annotation to
+    one bad comma would be a bad trade: an annotation set is small, hand-edited,
+    and the person editing it is the person who most needs to know which line
+    they broke.
+    """
+    result = AnnotationSet(path=path)
+    if not path.exists():
+        return result
+
+    for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
+        stripped = line.strip()
+        if not stripped or stripped.startswith("#"):
+            continue
+        if stripped.startswith("//"):
+            continue
+        try:
+            payload = json.loads(stripped)
+        except json.JSONDecodeError as exc:
+            result.skipped.append(f"line {number}: not valid JSON ({exc.msg})")
+            continue
+        if not isinstance(payload, dict):
+            result.skipped.append(f"line {number}: expected a JSON object")
+            continue
+        try:
+            annotation = parse_annotation(payload, number)
+        except ValueError as exc:
+            result.skipped.append(f"line {number}: {exc}")
+            continue
+        if not annotation.input:
+            result.skipped.append(f"line {number}: no 'input' field")
+            continue
+        result.annotations.append(annotation)
+
+    return result
+
+
+def merge_checks(
+    inferred: list[dict[str, Any]], human: list[dict[str, Any]]
+) -> list[dict[str, Any]]:
+    """Overlay human checks on inferred ones, type by type.
+
+    Human wins wherever the two overlap -- it is evidence, the inferred value was
+    a guess. Where they do not overlap the inferred check is kept, because a
+    human writing ``contains`` has no opinion about truncation and silently
+    dropping the length check would make the annotation a downgrade.
+
+    ``not_fallback`` is always first, whatever either side said.
+    """
+    by_type: dict[str, dict[str, Any]] = {check["type"]: check for check in inferred}
+    for check in human:
+        by_type[check["type"]] = check
+    ordered = [by_type.pop("not_fallback")] if "not_fallback" in by_type else []
+    ordered.extend(by_type.values())
+    return ordered
+
+
+def render_template(cases: list[dict[str, Any]]) -> str:
+    """A fill-in-the-blanks file for every case whose checks cannot catch it.
+
+    Emitted only when no expectations file exists yet, so a real one is never
+    overwritten. The ``expect`` object is left empty on purpose: a pre-filled
+    guess would be exactly the kind of over-confidence this whole mechanism
+    exists to avoid.
+    """
+    lines = [
+        "// trace2eval expectations -- one JSON object per line, '//' comments allowed.",
+        "//",
+        "// Why these cases are listed: their failure happened around the call, not in",
+        "// the answer, so no check on the output text can reproduce it. Write down what",
+        "// a correct answer looks like and they become real gates.",
+        "//",
+        "// Available keys inside 'expect' (same vocabulary as a trace's own expect block):",
+        "//   min_chars, max_chars, contains, not_contains, regex, json, not_fallback",
+        "//",
+        "// When you are done, save this file as expectations.jsonl and rebuild.",
+        "// Matching is on 'input', folded for case/space/punctuation -- so edit the",
+        "// wording of 'note' freely, but leave 'input' alone.",
+        "",
+    ]
+    for case in cases:
+        payload = {
+            "input": case["input"],
+            "expect": {},
+            "note": (
+                "TODO: fill in expect. "
+                f"({', '.join(signal['name'] for signal in case['signals'])})"
+            ),
+        }
+        lines.append(json.dumps(payload, ensure_ascii=False))
+    lines.append("")
+    return "\n".join(lines)

{trace2eval_cli-0.2.1 → trace2eval_cli-0.3.0}/src/trace2eval/cli.py

@@ -29,6 +29,7 @@ from .runner import (
     run_cases,
 )
 from .schema import TraceFormatError, load_traces
+from .annotations import load_annotations, render_template
 from .select import (
     DEFAULT_DEDUP_THRESHOLD,
     MatcherSpecError,
@@ -80,6 +81,15 @@ def cmd_build(args: argparse.Namespace) -> int:
                 print(f"  line {line_no}: {reason}", file=sys.stderr)
         return EXIT_BAD_INPUT
 
+    out_dir = Path(args.out)
+
+    expectations_path = (
+        Path(args.expectations) if args.expectations else out_dir / "expectations.jsonl"
+    )
+    expectations = load_annotations(expectations_path)
+    for problem in expectations.skipped:
+        print(f"warning: {expectations_path.name}: {problem}", file=sys.stderr)
+
     matcher = None
     if args.similarity:
         try:
@@ -94,14 +104,21 @@ def cmd_build(args: argparse.Namespace) -> int:
         min_score=args.min_score,
         dedup_threshold=args.dedup_threshold,
         matcher=matcher,
+        expectations=expectations,
     )
 
-    out_dir = Path(args.out)
     _write_jsonl(out_dir / "cases.jsonl", result.cases)
     # as_posix() so the report is byte-identical on Windows and Linux, which is
     # what lets CI diff the committed case set against a fresh build.
     _write_text(out_dir / "report.md", build_report(result, Path(args.traces).as_posix()))
 
+    # Written only when there is no expectations file yet, so a real one is never
+    # clobbered by a rebuild.
+    template_path: Path | None = None
+    if not expectations_path.exists() and result.cases_needing_annotation:
+        template_path = expectations_path.with_name("expectations.template.jsonl")
+        _write_text(template_path, render_template(result.cases_needing_annotation))
+
     stats = result.stats()
     print(f"read {stats['total_traces']} traces from {args.traces}")
     if loaded.skipped:
@@ -120,10 +137,19 @@ def cmd_build(args: argparse.Namespace) -> int:
             f"  {stats['cases_whose_reference_fails']} trusted reference(s) fail their "
             f"own checks -- look at these before committing the set"
         )
+    if stats["cases_with_annotation"]:
+        print(f"  {stats['cases_with_annotation']} case(s) use a human-written expectation")
     if stats["cases_with_weak_checks"]:
+        covered = stats["cases_with_annotation"]
         print(
             f"  {stats['cases_with_weak_checks']} case(s) carry checks that cannot "
             f"detect the failure they came from"
+            + (f" ({covered} covered by an annotation)" if covered else "")
+        )
+    if stats["cases_needing_annotation"]:
+        print(
+            f"  {stats['cases_needing_annotation']} of those still need a human-written "
+            f"expectation -> fill in {template_path or expectations_path}"
         )
     print(f"report -> {out_dir / 'report.md'}")
     return EXIT_OK
@@ -247,6 +273,16 @@ def build_parser() -> argparse.ArgumentParser:
             "clustering gets slower -- see trace2eval.matchers."
         ),
     )
+    build.add_argument(
+        "--expectations",
+        default=None,
+        metavar="PATH",
+        help=(
+            "human-written expected answers, one JSON object per line. Defaults to "
+            "expectations.jsonl beside the case set. If the file does not exist, a "
+            "fill-in template is written for the cases that need one."
+        ),
+    )
     build.set_defaults(func=cmd_build)
 
     run = subparsers.add_parser(

{trace2eval_cli-0.2.1 → trace2eval_cli-0.3.0}/src/trace2eval/report.py

@@ -58,6 +58,13 @@ def build_report(result: SelectionResult, source_name: str) -> str:
         f"| Trusted references failing their own checks "
         f"| {stats['cases_whose_reference_fails']} |"
     )
+    lines.append(
+        f"| Cases using a human-written expectation "
+        f"| {stats['cases_with_annotation']} |"
+    )
+    lines.append(
+        f"| Cases still needing one | {stats['cases_needing_annotation']} |"
+    )
     lines.append("")
 
     lines.append("## Log-wide baselines")
@@ -102,7 +109,7 @@ def build_report(result: SelectionResult, source_name: str) -> str:
         )
     lines.append("")
 
-    flagged = result.cases_whose_reference_fails + result.cases_with_weak_checks
+    flagged = result.cases_whose_reference_fails + result.cases_needing_annotation
     if flagged:
         lines.append("## Cases that need a human eye")
         lines.append("")
@@ -117,15 +124,14 @@ def build_report(result: SelectionResult, source_name: str) -> str:
                 f"checks (`{', '.join(case['self_check']['failed_checks'])}`). Either "
                 f"the reference is wrong or the checks are."
             )
-        for case in result.cases_with_weak_checks:
+        for case in result.cases_needing_annotation:
             if case["failure_kind"] == "behaviour":
                 lines.append(
                     f"- {case['id']} — the failure here was behavioural "
                     f"(`{', '.join(signal['name'] for signal in case['signals'])}`): the "
                     f"call itself produced a perfectly acceptable answer, and what went "
                     f"wrong happened around it. No check on the output text can "
-                    f"reproduce that. Keep the case as a pinned input, but it needs a "
-                    f"labelled expected answer before it can gate anything."
+                    f"reproduce that."
                 )
             else:
                 lines.append(
@@ -135,6 +141,30 @@ def build_report(result: SelectionResult, source_name: str) -> str:
                     f"a limit of the approach -- worth investigating."
                 )
         lines.append("")
+        lines.append(
+            "The way out for the behavioural ones is an expectation file: one JSON "
+            "object per line, keyed on `input`, holding the checks a correct answer "
+            "should satisfy. `build` writes a fill-in template next to the case set "
+            "listing exactly these cases; save it as `expectations.jsonl` and rebuild. "
+            "It is keyed on the question rather than the case id, so it survives "
+            "regeneration."
+        )
+        lines.append("")
+
+    if result.cases_with_annotation:
+        lines.append("## Cases carrying a human-written expectation")
+        lines.append("")
+        lines.append(
+            "The expectation below does not make these cases reproduce their original "
+            "failure -- nothing can, the failure was never in the answer. What it does "
+            "is give them something worth asserting."
+        )
+        lines.append("")
+        for case in result.cases_with_annotation:
+            lines.append(
+                f"- `{case['id']}` ({case['input']}) — {case['annotation']['note'] or '(no note)'}"
+            )
+        lines.append("")
 
     if result.cases:
         lines.append("## Why the top case was chosen")

{trace2eval_cli-0.2.1 → trace2eval_cli-0.3.0}/src/trace2eval/select.py

@@ -30,6 +30,7 @@ import statistics
 from dataclasses import dataclass, field
 from typing import Any, Callable, Iterable
 
+from .annotations import AnnotationSet, fold_input, merge_checks
 from .checks import run_checks
 from .schema import Trace
 from .signals import (
@@ -220,6 +221,25 @@ class SelectionResult:
             if case["self_check"]["verdict"] == "reference_fails_own_checks"
         ]
 
+    @property
+    def cases_with_annotation(self) -> list[dict[str, Any]]:
+        return [case for case in self.cases if case["annotation"]["applied"]]
+
+    @property
+    def cases_needing_annotation(self) -> list[dict[str, Any]]:
+        """The remaining TODO list.
+
+        A weak case that already carries a human expectation is off this list:
+        the expectation does not make the case reproduce its original failure --
+        nothing can, the failure was not in the text -- but it does give the case
+        something worth asserting, which is the actual goal.
+        """
+        return [
+            case
+            for case in self.cases_with_weak_checks
+            if not case["annotation"]["applied"]
+        ]
+
     def stats(self) -> dict[str, Any]:
         return {
             "total_traces": self.total_traces,
@@ -231,6 +251,8 @@ class SelectionResult:
             "dropped_beyond_limit": self.dropped_beyond_limit,
             "cases_with_weak_checks": len(self.cases_with_weak_checks),
             "cases_whose_reference_fails": len(self.cases_whose_reference_fails),
+            "cases_with_annotation": len(self.cases_with_annotation),
+            "cases_needing_annotation": len(self.cases_needing_annotation),
         }
 
 
@@ -569,7 +591,12 @@ def _self_check(checks: list[dict[str, Any]], output: str) -> dict[str, Any]:
     return {"passed": not failed, "failed_checks": failed}
 
 
-def build_case(index: int, cluster: Cluster, context: TraceContext | None = None) -> dict[str, Any]:
+def build_case(
+    index: int,
+    cluster: Cluster,
+    context: TraceContext | None = None,
+    expectations: AnnotationSet | None = None,
+) -> dict[str, Any]:
     trace = cluster.representative
     names = {signal.name for signal in cluster.signals}
     is_failure_seed = bool(names & QUALITY_SIGNALS)
@@ -614,6 +641,23 @@ def build_case(index: int, cluster: Cluster, context: TraceContext | None = None
         else:
             shape_source = "none available -- only behaviour was asserted"
 
+    # A human-written expectation outranks anything inferred. It does not make
+    # the case reproduce its original failure -- for a behavioural failure
+    # nothing on the output text can, by definition -- but it does give the case
+    # a specification worth asserting, which is the actual point.
+    annotation = (
+        expectations.lookup(fold_input(trace.input)) if expectations is not None else None
+    )
+    if annotation is not None:
+        checks = merge_checks(checks, _checks_from_expect(annotation.expect))
+        if not any(check["type"] == "not_fallback" for check in checks):
+            checks.insert(0, {"type": "not_fallback"})
+        shape_source = "a human-written expectation"
+        notes.append(
+            "human expectation applied, overriding anything inferred — "
+            + (annotation.note or "(no note given)")
+        )
+
     duplicates = cluster.duplicate_ids
     if duplicates:
         notes.append(
@@ -647,6 +691,10 @@ def build_case(index: int, cluster: Cluster, context: TraceContext | None = None
         "reference_is_trusted": not is_failure_seed,
         "shape_source": shape_source,
         "failure_kind": failure_kind,
+        "annotation": {
+            "applied": annotation is not None,
+            "note": annotation.note if annotation is not None else "",
+        },
         "source_trace_id": trace.id,
         "score": cluster.score,
         "signals": [signal.to_dict() for signal in cluster.signals],
@@ -665,6 +713,7 @@ def select_cases(
     dedup_threshold: float = DEFAULT_DEDUP_THRESHOLD,
     weights: dict[str, float] | None = None,
     matcher: Matcher | None = None,
+    expectations: AnnotationSet | None = None,
 ) -> SelectionResult:
     trace_list = list(traces)
     result = SelectionResult(total_traces=len(trace_list))
@@ -703,7 +752,7 @@ def select_cases(
 
     result.dropped_as_duplicate = sum(cluster.duplicate_count for cluster in interesting)
     result.cases = [
-        build_case(index, cluster, context)
+        build_case(index, cluster, context, expectations)
         for index, cluster in enumerate(interesting, 1)
     ]
     return result

{trace2eval_cli-0.2.1 → trace2eval_cli-0.3.0}/src/trace2eval_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: trace2eval-cli
-Version: 0.2.1
+Version: 0.3.0
 Summary: Turn production LLM traces into a regression eval set. Zero dependencies.
 Author: rfioly
 License: MIT
@@ -37,7 +37,9 @@ Dynamic: license-file
 零依赖、不用 API key、不调模型。同样的日志进，同样的用例出。
 
 [![CI](https://github.com/rfioly/trace2eval/actions/workflows/ci.yml/badge.svg)](https://github.com/rfioly/trace2eval/actions/workflows/ci.yml)
-[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+[![PyPI](https://img.shields.io/pypi/v/trace2eval-cli)](https://pypi.org/project/trace2eval-cli/)
+[![Downloads](https://img.shields.io/pypi/dm/trace2eval-cli)](https://pypi.org/project/trace2eval-cli/)
+[![Python versions](https://img.shields.io/pypi/pyversions/trace2eval-cli)](https://pypi.org/project/trace2eval-cli/)
 [![Dependencies](https://img.shields.io/badge/dependencies-0-brightgreen)](#)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
 
@@ -46,7 +48,7 @@ Dynamic: license-file
 ## 快速开始
 
 ```bash
-pip install -e .
+pip install trace2eval-cli
 
 # 1. 把日志变成用例集
 trace2eval build traces.jsonl -o evalset/
@@ -60,6 +62,9 @@ trace2eval check --baseline runs/baseline.json --current runs/current.json
 
 需要 Python 3.10+。
 
+> 分发名带 `-cli` 后缀，因为 `trace2eval` 这个名字在 PyPI 上被一个空壳占着。
+> **命令没变**——装完还是敲 `trace2eval`。
+
 ---
 
 ## 三个命令
@@ -79,6 +84,7 @@ trace2eval check --baseline runs/baseline.json --current runs/current.json
 - **一个问题一个用例。** 被问了 200 次的问题只变成一个用例，并且老实记录它代表 200 次调用。
 - **形状从干净答案学，不从失败那条学。** 失败种子的长度下限来自同问题的干净答案中位数。
 - **每个用例自检。** 报告里会标明哪些用例的检查**抓不住**它当初为什么坏，不混进"已覆盖"里。
+- **抓不住的用例可以人工补。** 失败发生在"调用之外"（用户重问、差评）而不在答案里时，自动检查无能为力——报告会把它们单列出来，`build` 同时生成一份待填模板，你写好期望答案后它们就变成真正的门禁。标注按**问题**索引而不是按用例编号，所以重新生成用例集不会失效。
 - **相似度可换。** `--similarity module:function` 换成你自己的实现，包本身不引依赖。
 - **天生适合 CI。** `check` 检出回归返回非零退出码。
 
@@ -190,10 +196,35 @@ flowchart LR
 
 ---
 
+## 人工标注（可选）
+
+有些失败发生在"调用之外"——用户又问了一遍、点了差评、接口慢到超时。**这种答案本身往往是没问题的**，所以任何对输出文本的自动检查都抓不到它。工具能识别出这些用例，但没法凭空想出"正确答案长什么样"，那得人来写。
+
+`build` 会在用例集旁边生成一份待填模板，列出的正是这些用例：
+
+```jsonl
+{"input": "修改手机号", "expect": {}, "note": "TODO: fill in expect. (user_retried)"}
+```
+
+把 `expect` 填好，另存为 `expectations.jsonl`，重新 `build` 就生效了：
+
+```jsonl
+{"input": "修改手机号", "expect": {"min_chars": 20, "contains": ["手机号"]}, "note": "用户重问说明第一版没解决，正确答案要说清验证流程"}
+```
+
+`expect` 里能用的键和 trace 自己的 `expect` 块**完全一致**：`min_chars`、`max_chars`、`contains`、`not_contains`、`regex`、`json`、`not_fallback`。
+
+两条设计上要紧的地方：
+
+- **按问题索引，不按用例编号。** 用例编号是按分数排序临时分配的，加一条日志就可能全部重排。标注要是挂在 `case-011` 上，下次重建就会**悄悄脱离**——而一个脱了钩的标注比没有标注更糟，因为那个用例看起来还是有覆盖的。
+- **标注不会改变自检结论。** 它不会让用例"复现失败"（行为型失败本来就复现不了），它是换了个目标：从"让这个失败不可能再发生"变成"答案必须长这样"。报告里两个数分开写，不合并。
+
+---
+
 ## 已知边界
 
 - **相似度是字符 n-gram**，不含相同字符的改写会被当成两个问题。可用 `--similarity` 换成你自己的实现。
-- **行为型失败抓不住。** 样本日志 16 个用例里有 3 个，失败原因只是"用户又问了一遍"，输出本身没毛病——任何对输出文本的确定性检查都抓不到。报告里已单独标出。
+- **行为型失败自动抓不住，但可以人工补。** 样本日志 16 个用例里有 3 个，失败原因只是"用户又问了一遍"，输出本身没毛病——任何对输出文本的确定性检查都抓不到，这是方法的边界而非欠账。它们现在由 `evalset/expectations.jsonl` 里的人工标注兜住（仓库里那份是示例）。报告会**分开**说明"有几个抓不住""有几个已有标注覆盖"，不会含糊成一个数。
 - **比较成本是 O(用例数 × 每簇不同问法数)**，不是 O(日志大小)：同一问题问 5000 次只留 1 个待比对指纹。`MAX_DISTINCT_FINGERPRINTS = 512` 是拍的保护值，没压测过。
 - **用例仍是提案**，但会自检：`report.md` 里单列"需要人看一眼"的用例。
 
@@ -203,16 +234,17 @@ flowchart LR
 
 ```
 src/trace2eval/
-  schema.py    trace 加载、字段别名、容错解析
-  signals.py   一条 trace 为什么值得测
-  select.py    相似度、聚类、用例构建
-  checks.py    确定性输出检查
-  runner.py    评分与回归对比
-  report.py    markdown 渲染
-  matchers.py  可替换的相似度实现
-tests/         42 个测试
-examples/      一份 42 行样本日志 + 一次基线 / 一次回归运行
-evalset/       提交进仓库的构建产物
+  schema.py       trace 加载、字段别名、容错解析
+  signals.py      一条 trace 为什么值得测
+  select.py       相似度、聚类、用例构建
+  checks.py       确定性输出检查
+  runner.py       评分与回归对比
+  report.py       markdown 渲染
+  matchers.py     可替换的相似度实现
+  annotations.py  人工标注的加载与合并
+tests/            62 个测试
+examples/         一份 42 行样本日志 + 一次基线 / 一次回归运行
+evalset/          提交进仓库的构建产物 + 示例标注
 ```
 
 ## 开发

{trace2eval_cli-0.2.1 → trace2eval_cli-0.3.0}/src/trace2eval_cli.egg-info/SOURCES.txt

@@ -3,6 +3,7 @@ README.md
 pyproject.toml
 src/trace2eval/__init__.py
 src/trace2eval/__main__.py
+src/trace2eval/annotations.py
 src/trace2eval/checks.py
 src/trace2eval/cli.py
 src/trace2eval/matchers.py
@@ -17,6 +18,7 @@ src/trace2eval_cli.egg-info/dependency_links.txt
 src/trace2eval_cli.egg-info/entry_points.txt
 src/trace2eval_cli.egg-info/requires.txt
 src/trace2eval_cli.egg-info/top_level.txt
+tests/test_annotations.py
 tests/test_dedup.py
 tests/test_matchers.py
 tests/test_pipeline.py

trace2eval_cli-0.3.0/tests/test_annotations.py

@@ -0,0 +1,359 @@
+"""Tests for human-written expectations.
+
+The property that matters most is survival: generated case ids are assigned in
+descending score order, so adding one trace to a log can renumber everything. An
+annotation keyed on a case id would detach itself on the next rebuild and the
+case would quietly go back to being weak while still looking annotated. Several
+of these tests exist to make sure that cannot happen unnoticed.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from trace2eval.annotations import (
+    Annotation,
+    AnnotationSet,
+    fold_input,
+    load_annotations,
+    merge_checks,
+    render_template,
+)
+from trace2eval.cli import main
+from trace2eval.schema import Trace, load_traces
+from trace2eval.select import select_cases
+
+SAMPLE_LOG = Path(__file__).resolve().parents[1] / "examples" / "sample_traces.jsonl"
+SAMPLE_EXPECTATIONS = Path(__file__).resolve().parents[1] / "evalset" / "expectations.jsonl"
+
+
+def read_lines(path: Path) -> list[dict]:
+    return [
+        json.loads(line)
+        for line in path.read_text(encoding="utf-8").splitlines()
+        if line.strip()
+    ]
+
+
+# --------------------------------------------------------------------------- #
+# Loading
+# --------------------------------------------------------------------------- #
+
+
+def test_loads_objects_and_ignores_comments_and_blanks(tmp_path):
+    path = tmp_path / "expectations.jsonl"
+    path.write_text(
+        "// a comment\n"
+        "\n"
+        '{"input": "修改手机号", "expect": {"min_chars": 20}, "note": "why"}\n'
+        "// another comment\n",
+        encoding="utf-8",
+    )
+    loaded = load_annotations(path)
+
+    assert not loaded.skipped
+    assert len(loaded.filled) == 1
+    assert loaded.filled[0].input == "修改手机号"
+    assert loaded.filled[0].expect == {"min_chars": 20}
+
+
+def test_a_missing_file_is_not_an_error(tmp_path):
+    loaded = load_annotations(tmp_path / "nope.jsonl")
+    assert loaded.annotations == []
+    assert not loaded
+
+
+def test_one_broken_line_does_not_lose_the_others(tmp_path):
+    """An annotation set is small and hand-edited. Losing it to one bad comma is a bad trade."""
+    path = tmp_path / "expectations.jsonl"
+    path.write_text(
+        '{"input": "a", "expect": {"min_chars": 5}}\n'
+        '{"input": "b", "expect": {"min_chars": 5}\n'  # unclosed brace
+        '{"input": "c", "expect": {"min_chars": 5}}\n',
+        encoding="utf-8",
+    )
+    loaded = load_annotations(path)
+
+    assert len(loaded.filled) == 2
+    assert len(loaded.skipped) == 1
+    assert "line 2" in loaded.skipped[0]
+
+
+def test_an_unknown_check_type_is_rejected_with_a_useful_message(tmp_path):
+    path = tmp_path / "expectations.jsonl"
+    path.write_text('{"input": "a", "expect": {"is_nice": true}}\n', encoding="utf-8")
+    loaded = load_annotations(path)
+
+    assert loaded.filled == []
+    assert "is_nice" in loaded.skipped[0]
+
+
+def test_a_template_placeholder_is_not_applied(tmp_path):
+    """'expect: {}' means 'not filled in yet', not 'assert nothing'."""
+    path = tmp_path / "expectations.jsonl"
+    path.write_text('{"input": "a", "expect": {}, "note": "TODO"}\n', encoding="utf-8")
+    loaded = load_annotations(path)
+
+    assert len(loaded.annotations) == 1
+    assert loaded.filled == []
+    assert not loaded  # bool() is False when nothing is actually filled in
+
+
+# --------------------------------------------------------------------------- #
+# Matching
+# --------------------------------------------------------------------------- #
+
+
+def test_matching_folds_case_space_and_punctuation():
+    assert fold_input("  修改手机号。 ") == fold_input("修改手机号")
+    assert fold_input("API Key 轮换") == fold_input("apikey轮换")
+
+
+def test_lookup_matches_a_rephrased_input():
+    loaded = AnnotationSet(annotations=[Annotation(input="修改手机号", expect={"min_chars": 20})])
+    assert loaded.lookup(fold_input(" 修改手机号， ")) is not None
+    assert loaded.lookup(fold_input("退款政策")) is None
+
+
+# --------------------------------------------------------------------------- #
+# Merging
+# --------------------------------------------------------------------------- #
+
+
+def test_human_checks_override_inferred_ones_of_the_same_type():
+    inferred = [{"type": "not_fallback"}, {"type": "min_chars", "value": 8}]
+    human = [{"type": "min_chars", "value": 20}]
+
+    merged = merge_checks(inferred, human)
+    assert {"type": "min_chars", "value": 20} in merged
+    assert {"type": "min_chars", "value": 8} not in merged
+
+
+def test_inferred_checks_survive_when_the_human_did_not_mention_them():
+    """Writing 'contains' is not an opinion about truncation.
+
+    Dropping the inferred length check here would make annotating a case a
+    downgrade, which would be a strange thing for a feature to do.
+    """
+    inferred = [{"type": "not_fallback"}, {"type": "min_chars", "value": 14}]
+    human = [{"type": "contains", "value": ["客服"]}]
+
+    types = [check["type"] for check in merge_checks(inferred, human)]
+    assert types == ["not_fallback", "min_chars", "contains"]
+
+
+def test_not_fallback_always_comes_first():
+    merged = merge_checks([], [{"type": "contains", "value": ["x"]}, {"type": "not_fallback"}])
+    assert merged[0]["type"] == "not_fallback"
+
+
+# --------------------------------------------------------------------------- #
+# Survival across rebuilds -- the whole point
+# --------------------------------------------------------------------------- #
+
+
+def _trace(trace_id: str, text: str, output: str, **kwargs) -> Trace:
+    return Trace(id=trace_id, input=text, output=output, **kwargs)
+
+
+def test_an_annotation_follows_the_question_when_case_ids_shift():
+    """The design property. Case ids are positional; annotations must not be."""
+    weak = _trace("w1", "修改手机号", "在设置里改。", retried=True)
+    filler = _trace("f1", "退款政策是什么", "退款申请提交后会在 14 个工作日内原路退回。")
+
+    before = select_cases([weak, filler], expectations=_annotation_set("修改手机号"))
+    # Add a much higher-scoring trace. It takes case-001, pushing everything down.
+    louder = _trace(
+        "x1", "系统崩了", "抱歉，我无法回答这个问题。", feedback="negative", retried=True
+    )
+    after = select_cases([weak, filler, louder], expectations=_annotation_set("修改手机号"))
+
+    def find(cases, text):
+        return next(c for c in cases if c["input"] == text)
+
+    weak_before, weak_after = find(before.cases, "修改手机号"), find(after.cases, "修改手机号")
+    assert weak_before["id"] != weak_after["id"], "the fixture must actually renumber the case"
+    assert weak_before["annotation"]["applied"] and weak_after["annotation"]["applied"]
+    assert "contains" in [check["type"] for check in weak_after["checks"]]
+
+
+def _annotation_set(text: str, expect: dict | None = None) -> AnnotationSet:
+    return AnnotationSet(
+        annotations=[
+            Annotation(
+                input=text,
+                expect=expect if expect is not None else {"min_chars": 20, "contains": ["手机号"]},
+                note="test",
+            )
+        ]
+    )
+
+
+def test_build_case_records_that_an_annotation_was_applied():
+    weak = _trace("w1", "修改手机号", "在设置里改。", retried=True)
+    result = select_cases([weak], expectations=_annotation_set("修改手机号"))
+    case = result.cases[0]
+
+    assert case["annotation"]["applied"] is True
+    assert case["annotation"]["note"] == "test"
+    assert case["shape_source"] == "a human-written expectation"
+    assert any("human expectation applied" in note for note in case["notes"])
+
+
+def test_an_annotation_does_not_pretend_to_fix_the_verdict():
+    """Honesty check: the failure still is not reproducible, and the report says so.
+
+    An annotation gives the case something worth asserting. It does not make the
+    original failure detectable, and pretending otherwise would be the exact
+    over-claiming this field exists to avoid.
+
+    Note the reference here is a *good* answer, which is what a behavioural
+    failure looks like -- the call went fine, the user just asked again. That is
+    precisely why no check on this text can reproach the failure, and why the
+    verdict has to stay ``failure_not_reproduced`` even after annotating.
+
+    (A failure seed whose reference is genuinely bad behaves differently: the
+    human's checks fail on it and the verdict becomes ``ok``. Also correct, just
+    a different situation.)
+    """
+    weak = _trace(
+        "w1",
+        "修改手机号",
+        "修改绑定手机号需要先通过原手机号接收验证码，验证通过后在账户设置里填写新号码。",
+        retried=True,
+    )
+    result = select_cases([weak], expectations=_annotation_set("修改手机号"))
+
+    case = result.cases[0]
+    assert case["self_check"]["passed"] is True, "the reference is a fine answer"
+    assert case["self_check"]["verdict"] == "failure_not_reproduced"
+    assert result.stats()["cases_with_annotation"] == 1
+    assert result.stats()["cases_needing_annotation"] == 0
+
+
+def test_an_annotation_can_turn_a_leaky_seed_into_a_real_gate():
+    """The other branch: when the reference really is bad, the human's checks catch it.
+
+    This is the payoff. A failure seed whose reference is a truncated answer used
+    to assert almost nothing; with an expectation attached the checks fail on it,
+    so the verdict flips to ``ok`` -- meaning the case now does reproduce a
+    failure. Same mechanism, opposite outcome, and both are honest.
+    """
+    weak = _trace("w1", "修改手机号", "在设置里改。", retried=True)
+    result = select_cases([weak], expectations=_annotation_set("修改手机号"))
+
+    case = result.cases[0]
+    assert case["self_check"]["passed"] is False
+    assert case["self_check"]["verdict"] == "ok"
+    assert "min_chars" in case["self_check"]["failed_checks"]
+
+
+def test_a_weak_case_without_an_annotation_stays_on_the_todo_list():
+    weak = _trace("w1", "修改手机号", "在设置里改。", retried=True)
+    result = select_cases([weak])
+
+    assert result.stats()["cases_needing_annotation"] == 1
+    assert result.stats()["cases_with_annotation"] == 0
+
+
+# --------------------------------------------------------------------------- #
+# The committed sample
+# --------------------------------------------------------------------------- #
+
+
+def test_the_sample_annotation_file_covers_every_weak_case():
+    loaded = load_traces(SAMPLE_LOG)
+    result = select_cases(loaded.traces, expectations=load_annotations(SAMPLE_EXPECTATIONS))
+
+    assert not load_annotations(SAMPLE_EXPECTATIONS).skipped, "the shipped file must parse cleanly"
+    assert result.stats()["cases_with_annotation"] == 3
+    assert result.stats()["cases_needing_annotation"] == 0
+    # The 3 behavioural cases are still honestly reported as not reproducible.
+    assert result.stats()["cases_with_weak_checks"] == 3
+
+
+def test_the_sample_annotation_changes_the_case_set():
+    """Without it, an answer that misses the point of the question still passes."""
+    loaded = load_traces(SAMPLE_LOG)
+    bare = select_cases(loaded.traces)
+    annotated = select_cases(
+        loaded.traces, expectations=load_annotations(SAMPLE_EXPECTATIONS)
+    )
+
+    def checks_for(result, text):
+        case = next(c for c in result.cases if c["input"] == text)
+        return [check["type"] for check in case["checks"]]
+
+    assert checks_for(bare, "修改手机号") == ["not_fallback", "min_chars"]
+    assert checks_for(annotated, "修改手机号") == ["not_fallback", "min_chars", "contains"]
+
+
+# --------------------------------------------------------------------------- #
+# The template
+# --------------------------------------------------------------------------- #
+
+
+def test_the_template_is_parseable_and_lists_only_the_cases_handed_to_it():
+    loaded = load_traces(SAMPLE_LOG)
+    bare = select_cases(loaded.traces)
+    text = render_template(bare.cases_needing_annotation)
+
+    import tempfile
+
+    with tempfile.TemporaryDirectory() as tmp:
+        path = Path(tmp) / "template.jsonl"
+        path.write_text(text, encoding="utf-8")
+        parsed = load_annotations(path)
+
+    assert not parsed.skipped, "every emitted line must be valid"
+    assert len(parsed.annotations) == len(bare.cases_needing_annotation) == 3
+    # Placeholders are deliberately empty: a pre-filled guess is exactly the
+    # over-confidence this mechanism exists to avoid.
+    assert parsed.filled == []
+
+
+def test_build_writes_a_template_only_when_there_is_no_expectations_file(tmp_path):
+    log = tmp_path / "traces.jsonl"
+    log.write_text(
+        "\n".join(
+            json.dumps(
+                {"input": "修改手机号", "output": "在设置里改。", "retried": True},
+                ensure_ascii=False,
+            )
+            for _ in range(1)
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    out = tmp_path / "evalset"
+
+    assert main(["build", str(log), "-o", str(out)]) == 0
+    template = out / "expectations.template.jsonl"
+    assert template.exists()
+
+    # Once a real expectations file is in place, the template is not rewritten.
+    template.unlink()
+    (out / "expectations.jsonl").write_text(
+        '{"input": "修改手机号", "expect": {"min_chars": 20}}\n', encoding="utf-8"
+    )
+    assert main(["build", str(log), "-o", str(out)]) == 0
+    assert not template.exists()
+    assert read_lines(out / "cases.jsonl")[0]["annotation"]["applied"] is True
+
+
+def test_a_broken_annotation_line_is_reported_but_does_not_fail_the_build(tmp_path):
+    log = tmp_path / "traces.jsonl"
+    log.write_text(
+        json.dumps({"input": "修改手机号", "output": "在设置里改。", "retried": True}, ensure_ascii=False)
+        + "\n",
+        encoding="utf-8",
+    )
+    out = tmp_path / "evalset"
+    out.mkdir()
+    (out / "expectations.jsonl").write_text("{not json\n", encoding="utf-8")
+
+    assert main(["build", str(log), "-o", str(out)]) == 0
+    assert read_lines(out / "cases.jsonl")[0]["annotation"]["applied"] is False