agentcheckup 0.4.3__tar.gz → 0.4.4__tar.gz

{agentcheckup-0.4.3 → agentcheckup-0.4.4}/CHANGELOG.md

@@ -91,8 +91,9 @@ trusts, so the scoring and the report both changed.
 ### Notes
 
 - Conflict detection produced **zero false positives** across the 18 real tools
-  from the three official servers. That is the number this project lives or dies
-  on, and it held.
+  from the three official servers. *(Corrected in 0.4.4: that measurement was
+  vacuous — those 18 tools contained no conflicts, so there was nothing to get
+  wrong. A real benchmark replaced it.)*
 
 ## [0.3.0] - 2026-11-07
 

{agentcheckup-0.4.3 → agentcheckup-0.4.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentcheckup
-Version: 0.4.3
+Version: 0.4.4
 Summary: Diagnose your agent setup: tool conflicts, risky config, context budget.
 Project-URL: Homepage, https://github.com/jiaqinq/agentcheckup
 Project-URL: Issues, https://github.com/jiaqinq/agentcheckup/issues
@@ -149,7 +149,7 @@ The fastest way to keep a manifest from regressing, and the version you pin is
 the version you run — the action installs from its own checkout.
 
 ```yaml
-- uses: jiaqinq/agentcheckup@v0.4.3
+- uses: jiaqinq/agentcheckup@v0.4.4
   with:
     config: .mcp.json
     fail-on: warning
@@ -205,6 +205,30 @@ false-positive rate made the report untrustworthy, which is worse than no report
 
 It is a heuristic, and the docs say so. `AGT203` is a warning, not a verdict.
 
+## How well does conflict detection work?
+
+Measured, not asserted. `tests/fixtures/collision_benchmark.json` holds 44
+hand-labelled tool pairs — 20 that should be flagged, 24 that should not — and
+`scripts/benchmark.py` scores the detector against them:
+
+| | value |
+|---|---|
+| pairs | 44 (20 positive, 24 negative) |
+| precision | **0.952** |
+| recall | **1.000** |
+| F1 | 0.976 |
+
+The one disagreement is a pair like `lookup_contact` / `lookup_user` with
+near-identical descriptions: the detector flags it, the label says it should not.
+That is arguably the detector being right and the label being strict.
+
+Two honest caveats. The labels are the author's, not users' — the criterion was
+"would a model picking the wrong one get a materially different result". And this
+benchmark did not exist before 0.4.4; the earlier claim of "zero false positives"
+came from running on 18 tools that contained no conflicts at all, which measured
+nothing. `tests/test_benchmark.py` fails the build if precision or recall drops
+below 0.90.
+
 ## How the score works
 
 Most linters fail here: a rule that fires on 18 of 18 tools is a *preference*,

{agentcheckup-0.4.3 → agentcheckup-0.4.4}/README.md

@@ -121,7 +121,7 @@ The fastest way to keep a manifest from regressing, and the version you pin is
 the version you run — the action installs from its own checkout.
 
 ```yaml
-- uses: jiaqinq/agentcheckup@v0.4.3
+- uses: jiaqinq/agentcheckup@v0.4.4
   with:
     config: .mcp.json
     fail-on: warning
@@ -177,6 +177,30 @@ false-positive rate made the report untrustworthy, which is worse than no report
 
 It is a heuristic, and the docs say so. `AGT203` is a warning, not a verdict.
 
+## How well does conflict detection work?
+
+Measured, not asserted. `tests/fixtures/collision_benchmark.json` holds 44
+hand-labelled tool pairs — 20 that should be flagged, 24 that should not — and
+`scripts/benchmark.py` scores the detector against them:
+
+| | value |
+|---|---|
+| pairs | 44 (20 positive, 24 negative) |
+| precision | **0.952** |
+| recall | **1.000** |
+| F1 | 0.976 |
+
+The one disagreement is a pair like `lookup_contact` / `lookup_user` with
+near-identical descriptions: the detector flags it, the label says it should not.
+That is arguably the detector being right and the label being strict.
+
+Two honest caveats. The labels are the author's, not users' — the criterion was
+"would a model picking the wrong one get a materially different result". And this
+benchmark did not exist before 0.4.4; the earlier claim of "zero false positives"
+came from running on 18 tools that contained no conflicts at all, which measured
+nothing. `tests/test_benchmark.py` fails the build if precision or recall drops
+below 0.90.
+
 ## How the score works
 
 Most linters fail here: a rule that fires on 18 of 18 tools is a *preference*,

{agentcheckup-0.4.3 → agentcheckup-0.4.4}/README.zh-CN.md

@@ -128,6 +128,25 @@ agentcheckup rules                              # 列出全部规则
 
 这是启发式，文档里写明了。`AGT203` 是警告，不是判决。
 
+## 冲突检测到底准不准
+
+这是测出来的，不是声称的。`tests/fixtures/collision_benchmark.json` 里有 44 对人工标注的工具对
+（20 对该报、24 对不该报），`scripts/benchmark.py` 拿检测器去跑：
+
+| 指标 | 值 |
+|---|---|
+| 样本 | 44 对（20 正 / 24 负） |
+| 精确率 | **0.952** |
+| 召回率 | **1.000** |
+| F1 | 0.976 |
+
+唯一的分歧是一对类似 `lookup_contact` / `lookup_user`、描述几乎一样的工具——检测器报了，
+我的标注说不该报。**这更像是检测器对、标注太严。**
+
+两个必须说明的限制：标注是我自己做的，不是用户标注（判断标准是「模型选错会不会拿到实质不同的结果」）；
+这个基准是 0.4.4 才有的，**之前那句「零误报」是在 18 个根本没有冲突的工具上跑出来的，什么都没测到**。
+`tests/test_benchmark.py` 会在精确率或召回率跌破 0.90 时让 CI 失败。
+
 ## 分数是怎么算的
 
 大多数 linter 在这里翻车：一条在 18 个工具上全部触发的规则，是**偏好**不是缺陷，

{agentcheckup-0.4.3 → agentcheckup-0.4.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agentcheckup"
-version = "0.4.3"
+version = "0.4.4"
 description = "Diagnose your agent setup: tool conflicts, risky config, context budget."
 readme = "README.md"
 requires-python = ">=3.10"

agentcheckup-0.4.4/scripts/benchmark.py

@@ -0,0 +1,101 @@
+"""Measure conflict detection against a labelled set of tool pairs.
+
+The README used to claim "zero false positives" on the strength of three official
+servers that contained no conflicts at all. That is not a measurement. This is:
+every pair below was labelled by hand, so precision and recall can be computed
+and regressions become visible.
+
+    python scripts/benchmark.py
+
+Labels are the author's, not users'. The criterion applied to each pair:
+"would a maintainer reasonably consider these two tools confusable by a model
+choosing from one flat list?" A pair is positive when a user picking the wrong
+one would get a materially different result.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT / "src"))
+
+from agentcheckup.conflicts import ToolRef, detect_conflicts  # noqa: E402
+from agentcheckup.models import Tool  # noqa: E402
+
+FIXTURE = ROOT / "tests" / "fixtures" / "collision_benchmark.json"
+
+
+def to_ref(side: dict) -> ToolRef:
+    tool = Tool.from_raw(
+        {
+            "name": side["name"],
+            "description": side["description"],
+            "inputSchema": {
+                "type": "object",
+                "properties": {"path": {"type": "string", "pattern": "^.*$"}},
+                "additionalProperties": False,
+            },
+        }
+    )
+    return ToolRef(server=side["server"], tool=tool)
+
+
+def main() -> int:
+    cases = json.loads(FIXTURE.read_text(encoding="utf-8"))
+
+    tp = fp = tn = fn = 0
+    false_positives: list[str] = []
+    false_negatives: list[str] = []
+
+    for case in cases:
+        found = detect_conflicts([to_ref(case["a"]), to_ref(case["b"])])
+        predicted = bool(found)
+        actual = case["conflict"]
+
+        if predicted and actual:
+            tp += 1
+        elif predicted and not actual:
+            fp += 1
+            rules = ",".join(sorted({c.rule for c in found}))
+            false_positives.append(f"  {case['id']}  {rules}  {case['why']}")
+        elif not predicted and actual:
+            fn += 1
+            false_negatives.append(f"  {case['id']}  missed  {case['why']}")
+        else:
+            tn += 1
+
+    total = tp + fp + tn + fn
+    precision = tp / (tp + fp) if tp + fp else 0.0
+    recall = tp / (tp + fn) if tp + fn else 0.0
+    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
+    accuracy = (tp + tn) / total if total else 0.0
+
+    print(f"pairs       {total}")
+    print(f"positives   {tp + fn}")
+    print(f"negatives   {tn + fp}")
+    print()
+    print(f"true  pos   {tp}")
+    print(f"false pos   {fp}")
+    print(f"true  neg   {tn}")
+    print(f"false neg   {fn}")
+    print()
+    print(f"precision   {precision:.3f}")
+    print(f"recall      {recall:.3f}")
+    print(f"f1          {f1:.3f}")
+    print(f"accuracy    {accuracy:.3f}")
+
+    if false_positives:
+        print("\nfalse positives")
+        print("\n".join(false_positives))
+    if false_negatives:
+        print("\nfalse negatives")
+        print("\n".join(false_negatives))
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

{agentcheckup-0.4.3 → agentcheckup-0.4.4}/src/agentcheckup/report.py

@@ -10,7 +10,7 @@ from . import rules as _rules
 from .engine import Report
 from .models import Finding, Severity
 
-__version__ = "0.4.3"
+__version__ = "0.4.4"
 
 SARIF_LEVEL = {
     Severity.ERROR: "error",

agentcheckup-0.4.4/tests/fixtures/collision_benchmark.json

@@ -0,0 +1,177 @@
+[
+  {"id": "p01", "conflict": true, "why": "同名工具，跨 server",
+   "a": {"server": "files", "name": "read_file", "description": "Read a text file from the sandbox and return its contents."},
+   "b": {"server": "git", "name": "read_file", "description": "Read a file as it exists in a git revision."}},
+
+  {"id": "p02", "conflict": true, "why": "同名，仅大小写不同，检测器会先 lower()",
+   "a": {"server": "files", "name": "read_file", "description": "Read a file from disk."},
+   "b": {"server": "s3", "name": "Read_File", "description": "Read an object from a bucket."}},
+
+  {"id": "p03", "conflict": true, "why": "单复数差异，归一化后同名",
+   "a": {"server": "files", "name": "read_files", "description": "Read several files and return their contents."},
+   "b": {"server": "git", "name": "read_file", "description": "Read one file from a revision."}},
+
+  {"id": "p04", "conflict": true, "why": "描述几乎一致，名字共享 search",
+   "a": {"server": "github", "name": "search_public", "description": "Search public repositories by keyword and return matching repository names."},
+   "b": {"server": "docs", "name": "search_private", "description": "Search private repositories by keyword and return matching repository names."}},
+
+  {"id": "p05", "conflict": true, "why": "同义动词 + 同一宾语，名字共享 file",
+   "a": {"server": "files", "name": "get_file", "description": "Fetch a file by path and return its bytes."},
+   "b": {"server": "docs", "name": "get_files", "description": "Fetch files by path and return their bytes."}},
+
+  {"id": "p06", "conflict": true, "why": "中文近义，描述几乎相同",
+   "a": {"server": "文件服务", "name": "读取文件", "description": "读取指定路径的文件内容并返回文本"},
+   "b": {"server": "文档服务", "name": "读取文档", "description": "读取指定路径的文件内容并返回字符串"}},
+
+  {"id": "p07", "conflict": true, "why": "同名删除工具",
+   "a": {"server": "files", "name": "delete_file", "description": "Delete a file at the given path."},
+   "b": {"server": "storage", "name": "delete_file", "description": "Delete an object from the bucket."}},
+
+  {"id": "p08", "conflict": true, "why": "单复数 + 同一目录语义",
+   "a": {"server": "files", "name": "list_dir", "description": "List entries in one directory under the sandbox root."},
+   "b": {"server": "shell", "name": "list_dirs", "description": "List entries in one directory under the sandbox root."}},
+
+  {"id": "p09", "conflict": true, "why": "同一动作的 issue 创建，描述高度重叠",
+   "a": {"server": "github", "name": "create_issue", "description": "Create an issue with a title, body and assignee in a repository."},
+   "b": {"server": "jira", "name": "create_issues", "description": "Create an issue with a title, body and assignee in a project."}},
+
+  {"id": "p10", "conflict": true, "why": "描述相同，名字只差一个词且共享 run",
+   "a": {"server": "shell", "name": "run_command", "description": "Run a shell command and return stdout and stderr."},
+   "b": {"server": "docker", "name": "run_commands", "description": "Run a shell command and return stdout and stderr."}},
+
+  {"id": "p11", "conflict": true, "why": "同名，一个本地一个远端",
+   "a": {"server": "files", "name": "write_file", "description": "Write text to a file on disk."},
+   "b": {"server": "s3", "name": "write_file", "description": "Write text to an object in a bucket."}},
+
+  {"id": "p12", "conflict": true, "why": "描述几乎逐字相同",
+   "a": {"server": "db", "name": "query_table", "description": "Query rows from a table using a SQL where clause and return them as JSON."},
+   "b": {"server": "warehouse", "name": "query_tables", "description": "Query rows from a table using a SQL where clause and return them as JSON."}},
+
+  {"id": "p13", "conflict": true, "why": "同一份数据的两种读取入口，名字共享 read",
+   "a": {"server": "cache", "name": "read_cache", "description": "Read a value from the cache by key."},
+   "b": {"server": "redis", "name": "read_cached", "description": "Read a value from the cache by key."}},
+
+  {"id": "p14", "conflict": true, "why": "中文同名",
+   "a": {"server": "文件服务", "name": "搜索文件", "description": "按关键词搜索文件并返回匹配的路径"},
+   "b": {"server": "代码服务", "name": "搜索文件", "description": "按关键词搜索仓库中的文件"}},
+
+  {"id": "p15", "conflict": true, "why": "单复数 + 同一搜索动作",
+   "a": {"server": "docs", "name": "search_doc", "description": "Search documents by title across the workspace and return matching ids."},
+   "b": {"server": "wiki", "name": "search_docs", "description": "Search documents by title across the workspace and return matching ids."}},
+
+  {"id": "p16", "conflict": true, "why": "同名上传",
+   "a": {"server": "s3", "name": "upload", "description": "Upload a local file to remote storage."},
+   "b": {"server": "gcs", "name": "upload", "description": "Upload a local file to remote storage."}},
+
+  {"id": "p17", "conflict": true, "why": "描述一致，名字共享 get 与 issue",
+   "a": {"server": "github", "name": "get_issue", "description": "Get one issue by number and return its title, body, state and labels."},
+   "b": {"server": "jira", "name": "get_issues", "description": "Get one issue by number and return its title, body, state and labels."}},
+
+  {"id": "p18", "conflict": true, "why": "同一动作的中英文混写，描述相同",
+   "a": {"server": "files", "name": "read_file", "description": "读取文件内容并返回文本"},
+   "b": {"server": "docs", "name": "read_file", "description": "读取文件内容并返回文本"}},
+
+  {"id": "p19", "conflict": true, "why": "单复数 + 同一删除动作，描述相同",
+   "a": {"server": "db", "name": "drop_table", "description": "Drop a table and all of its rows."},
+   "b": {"server": "warehouse", "name": "drop_tables", "description": "Drop a table and all of its rows."}},
+
+  {"id": "p20", "conflict": true, "why": "同名创建",
+   "a": {"server": "k8s", "name": "create_namespace", "description": "Create a namespace in the cluster."},
+   "b": {"server": "cloud", "name": "create_namespace", "description": "Create a namespace in the project."}},
+
+  {"id": "n01", "conflict": false, "why": "完全不同的事",
+   "a": {"server": "files", "name": "read_file", "description": "Read a text file from the sandbox and return its contents."},
+   "b": {"server": "jira", "name": "create_ticket", "description": "Create a ticket with a title, body and assignee."}},
+
+  {"id": "n02", "conflict": false, "why": "共享 search 但宾语不同，这是最常见的误报来源",
+   "a": {"server": "github", "name": "search_repositories", "description": "Search repositories by name across the org."},
+   "b": {"server": "docs", "name": "search_documents", "description": "Search documents by title across the workspace."}},
+
+  {"id": "n03", "conflict": false, "why": "共享 get 但对象完全不同",
+   "a": {"server": "time", "name": "get_current_time", "description": "Return the current time in a given timezone."},
+   "b": {"server": "weather", "name": "get_weather", "description": "Return the forecast for a city and date."}},
+
+  {"id": "n04", "conflict": false, "why": "共享 list 但宾语不同",
+   "a": {"server": "files", "name": "list_files", "description": "List files under a directory."},
+   "b": {"server": "db", "name": "list_tables", "description": "List tables in a database."}},
+
+  {"id": "n05", "conflict": false, "why": "共享 upload 但媒体类型不同",
+   "a": {"server": "assets", "name": "upload_image", "description": "Upload an image and return its asset id."},
+   "b": {"server": "docs", "name": "upload_document", "description": "Upload a document and return its document id."}},
+
+  {"id": "n06", "conflict": false, "why": "中文共享文件但动作相反",
+   "a": {"server": "文件服务", "name": "读取文件", "description": "读取指定路径的文件内容并返回文本"},
+   "b": {"server": "文件服务", "name": "删除文件", "description": "删除指定路径的文件"}},
+
+  {"id": "n07", "conflict": false, "why": "共享 file 但读与写相反",
+   "a": {"server": "files", "name": "read_file", "description": "Read a file and return its contents."},
+   "b": {"server": "files", "name": "write_file", "description": "Write text to a file."}},
+
+  {"id": "n08", "conflict": false, "why": "共享 git 但动作不同",
+   "a": {"server": "git", "name": "git_commit", "description": "Record staged changes as a commit."},
+   "b": {"server": "git", "name": "git_status", "description": "Show the working tree status."}},
+
+  {"id": "n09", "conflict": false, "why": "名字完全不共享词",
+   "a": {"server": "fetch", "name": "fetch", "description": "Fetch a URL and convert it to markdown."},
+   "b": {"server": "time", "name": "convert_time", "description": "Convert a time between timezones."}},
+
+  {"id": "n10", "conflict": false, "why": "共享 create 但资源不同",
+   "a": {"server": "github", "name": "create_branch", "description": "Create a git branch in a repository."},
+   "b": {"server": "jira", "name": "create_project", "description": "Create a project with a key and name."}},
+
+  {"id": "n11", "conflict": false, "why": "名字共享 file，但一个是读内容一个是读元数据",
+   "a": {"server": "files", "name": "read_file", "description": "Read the full text content of a file from disk."},
+   "b": {"server": "meta", "name": "file_info", "description": "Return size, mtime and permissions for a file."}},
+
+  {"id": "n12", "conflict": false, "why": "共享 search 但一个是代码一个是日志",
+   "a": {"server": "code", "name": "search_code", "description": "Search source files for a pattern."},
+   "b": {"server": "logs", "name": "search_logs", "description": "Search application logs for a pattern."}},
+
+  {"id": "n13", "conflict": false, "why": "共享 send 但渠道不同",
+   "a": {"server": "slack", "name": "send_message", "description": "Send a message to a channel."},
+   "b": {"server": "mail", "name": "send_email", "description": "Send an email to a recipient."}},
+
+  {"id": "n14", "conflict": false, "why": "中文共享仓库但动作不同",
+   "a": {"server": "代码服务", "name": "克隆仓库", "description": "把远端仓库克隆到本地目录"},
+   "b": {"server": "代码服务", "name": "删除仓库", "description": "删除远端仓库及其全部分支"}},
+
+  {"id": "n15", "conflict": false, "why": "共享 run 但一个是测试一个是迁移",
+   "a": {"server": "ci", "name": "run_tests", "description": "Run the test suite and return failures."},
+   "b": {"server": "db", "name": "run_migrations", "description": "Apply pending database migrations."}},
+
+  {"id": "n16", "conflict": false, "why": "共享 get 但一个是配置一个是统计",
+   "a": {"server": "config", "name": "get_config", "description": "Return the current configuration as JSON."},
+   "b": {"server": "metrics", "name": "get_stats", "description": "Return request counts for the last hour."}},
+
+  {"id": "n17", "conflict": false, "why": "描述用词相似但对象不同（跨域同义）",
+   "a": {"server": "crm", "name": "lookup_contact", "description": "Look up a contact by email and return the profile."},
+   "b": {"server": "auth", "name": "lookup_user", "description": "Look up a user by email and return the profile."}},
+
+  {"id": "n18", "conflict": false, "why": "共享 list 但一个是文件一个是容器",
+   "a": {"server": "docker", "name": "list_containers", "description": "List running containers."},
+   "b": {"server": "files", "name": "list_directory", "description": "List entries in a directory."}},
+
+  {"id": "n19", "conflict": false, "why": "共享 delete 但资源不同",
+   "a": {"server": "k8s", "name": "delete_pod", "description": "Delete a pod by name."},
+   "b": {"server": "s3", "name": "delete_object", "description": "Delete an object by key."}},
+
+  {"id": "n20", "conflict": false, "why": "名字共享 repo，一个是读一个是搜",
+   "a": {"server": "github", "name": "get_repo", "description": "Return metadata for one repository."},
+   "b": {"server": "github", "name": "search_repos", "description": "Search repositories by keyword."}},
+
+  {"id": "n21", "conflict": false, "why": "共享 write 但一个是文件一个是日志",
+   "a": {"server": "files", "name": "write_file", "description": "Write text to a file on disk."},
+   "b": {"server": "logs", "name": "write_log", "description": "Append a line to the application log."}},
+
+  {"id": "n22", "conflict": false, "why": "共享 create 但一个是目录一个是分支",
+   "a": {"server": "files", "name": "create_dir", "description": "Create a directory under the sandbox root."},
+   "b": {"server": "git", "name": "create_branch", "description": "Create a branch at the current HEAD."}},
+
+  {"id": "n23", "conflict": false, "why": "中文共享搜索但对象不同",
+   "a": {"server": "代码服务", "name": "搜索代码", "description": "在源码文件中搜索指定模式"},
+   "b": {"server": "日志服务", "name": "搜索日志", "description": "在应用日志中搜索指定模式"}},
+
+  {"id": "n24", "conflict": false, "why": "共享 read 但一个是文件一个是环境变量",
+   "a": {"server": "files", "name": "read_file", "description": "Read a file and return its contents."},
+   "b": {"server": "env", "name": "read_env", "description": "Read an environment variable by name."}}
+]

agentcheckup-0.4.4/tests/test_benchmark.py

@@ -0,0 +1,82 @@
+"""Regression guard on the labelled collision benchmark.
+
+`scripts/benchmark.py` prints the numbers for humans; this keeps them from
+silently decaying. The floors are set a little below the measured values so a
+small, deliberate trade-off does not fail the build, but a real regression does.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT / "src"))
+
+from agentcheckup.conflicts import ToolRef, detect_conflicts  # noqa: E402
+from agentcheckup.models import Tool  # noqa: E402
+
+FIXTURE = Path(__file__).resolve().parent / "fixtures" / "collision_benchmark.json"
+
+MIN_PRECISION = 0.90
+MIN_RECALL = 0.90
+
+
+def to_ref(side: dict) -> ToolRef:
+    tool = Tool.from_raw(
+        {
+            "name": side["name"],
+            "description": side["description"],
+            "inputSchema": {
+                "type": "object",
+                "properties": {"path": {"type": "string", "pattern": "^.*$"}},
+                "additionalProperties": False,
+            },
+        }
+    )
+    return ToolRef(server=side["server"], tool=tool)
+
+
+def score() -> tuple[float, float, list[str], list[str]]:
+    cases = json.loads(FIXTURE.read_text(encoding="utf-8"))
+    tp = fp = fn = 0
+    false_positives: list[str] = []
+    false_negatives: list[str] = []
+
+    for case in cases:
+        found = detect_conflicts([to_ref(case["a"]), to_ref(case["b"])])
+        predicted = bool(found)
+        if predicted and case["conflict"]:
+            tp += 1
+        elif predicted and not case["conflict"]:
+            fp += 1
+            false_positives.append(case["id"])
+        elif not predicted and case["conflict"]:
+            fn += 1
+            false_negatives.append(case["id"])
+
+    precision = tp / (tp + fp) if tp + fp else 0.0
+    recall = tp / (tp + fn) if tp + fn else 0.0
+    return precision, recall, false_positives, false_negatives
+
+
+def test_benchmark_precision_and_recall_hold():
+    precision, recall, false_positives, false_negatives = score()
+    assert precision >= MIN_PRECISION, f"precision dropped to {precision:.3f}; FP: {false_positives}"
+    assert recall >= MIN_RECALL, f"recall dropped to {recall:.3f}; FN: {false_negatives}"
+
+
+def test_benchmark_covers_both_labels():
+    cases = json.loads(FIXTURE.read_text(encoding="utf-8"))
+    positives = sum(1 for c in cases if c["conflict"])
+    negatives = len(cases) - positives
+    # A benchmark that drifts to one class measures nothing.
+    assert positives >= 10, positives
+    assert negatives >= 10, negatives
+
+
+def test_benchmark_includes_chinese_cases():
+    cases = json.loads(FIXTURE.read_text(encoding="utf-8"))
+    chinese = [c for c in cases if any("\u4e00" <= ch <= "\u9fff" for ch in c["a"]["name"])]
+    assert len(chinese) >= 4, len(chinese)