agent-comparability 0.5.2__tar.gz → 0.5.3__tar.gz

{agent_comparability-0.5.2/src/agent_comparability.egg-info → agent_comparability-0.5.3}/PKG-INFO

@@ -1,8 +1,12 @@
 Metadata-Version: 2.4
 Name: agent-comparability
-Version: 0.5.2
+Version: 0.5.3
 Summary: Agent evaluation comparability semantics: spec + conformance suite + thin reference implementation
 License: Apache-2.0
+Project-URL: Homepage, https://github.com/Rethymus/agent-comparability
+Project-URL: Repository, https://github.com/Rethymus/agent-comparability
+Project-URL: Issues, https://github.com/Rethymus/agent-comparability/issues
+Project-URL: Changelog, https://github.com/Rethymus/agent-comparability/blob/main/CHANGELOG.md
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -25,6 +29,7 @@ Dynamic: license-file
 [![CI](https://github.com/Rethymus/agent-comparability/actions/workflows/ci.yml/badge.svg)](https://github.com/Rethymus/agent-comparability/actions/workflows/ci.yml)
 [![License: Apache-2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-3776AB?logo=python&logoColor=white)](pyproject.toml)
+[![PyPI](https://img.shields.io/pypi/v/agent-comparability)](https://pypi.org/project/agent-comparability/)
 
 **English** | [简体中文](README.zh-CN.md)
 
@@ -74,6 +79,12 @@ Agent Comparability is a framework-independent **specification** ([spec/SPEC.md]
 a frozen **conformance suite**, and a thin **reference implementation** that adjudicates
 *comparability* of agent evaluation records — never the scores themselves.
 
+**Does it actually refuse unsupported claims?** See
+[experiments/scenarios_001/RECORD.md](experiments/scenarios_001/RECORD.md): four real-pipeline
+scenarios (real `.eval` logs → converter → core) covering all four verdicts — including the
+confounded comparison and the missing-trajectory case, both correctly **refused** rather than
+packaged as supported conclusions.
+
 **Who is this for**: developers comparing agent/prompt/harness evaluation results,
 benchmark authors documenting what conclusions their data supports, and teams building
 reliability pipelines on top of existing eval frameworks.
@@ -97,6 +108,8 @@ Three failure modes silently corrupt agent-evaluation conclusions:
 | **Evidence gaps** | Missing traces quietly inflate success rates; rankings reverse across harnesses |
 
 These boundaries should be **machine-checkable**, not vibes. That is what this project ships.
+How this relates to benchmark-rigor and transcript-audit work (verified literature anchors):
+see [docs/positioning.md](docs/positioning.md).
 
 ## 3. What Agent Comparability Does
 
@@ -153,10 +166,30 @@ pip install -e .[suite]                 # + suite basics (pyyaml, jsonschema)
 pip install -e .[suite,inspect,pydantic]  # FULL conformance run — required by run_suite.py
 ```
 
-Requires Python ≥ 3.10. No Docker, no model keys.
+Requires Python ≥ 3.10. No Docker, no model keys. Note: the core package supports 3.10+;
+the optional `inspect`/`pydantic` extras pull dependencies that require Python ≥ 3.11 —
+on 3.10 pip resolves older versions of them.
 
 ## 7. Quickstart
 
+**Command line** (offline, zero extra dependencies — input contract below):
+
+```bash
+python -m agent_comparability examples/demo-case.json --claim score_comparison
+# → CDR JSON on stdout (exit 0). INADMISSIBLE / PARTIAL / INCONCLUSIVE are verdicts,
+#   not program failures. Invalid input (bad JSON / missing or mistyped fields) →
+#   exit 2 with a path-precise diagnostic — never silently converted into a verdict.
+#   -o cdr.json writes to a file instead. The demo case is SYNTHETIC/illustrative.
+```
+
+Input contract (`{"id": str, "records": [{"task_id", "harness", "model", "reward":
+number, "reward_status", ...optional}]}`); unknown record fields are ignored and
+recorded in `unknown_conditions` (SPEC §5). Add `--manifest out.manifest.json` to
+also emit an Evidence Manifest documenting artifact digests and evidence trust
+states (never auto-upgraded; see [docs/evidence-manifest.md](docs/evidence-manifest.md)).
+
+**Python API**:
+
 ```python
 from agent_comparability import adjudicate
 
@@ -281,6 +314,10 @@ Both wrap the *same* core (no semantic duplication). Inspect scorer uses
 The Pydantic evaluator checks the **task's real output** (`ctx.output`) against
 frozen expectations — it never re-adjudicates inside the evaluator — and the
 adapter's exit code derives from the full 22-pair `report` (cases + failures).
+For **real Inspect EvalLog files** (not frozen cases) use the converter:
+`python -m agent_comparability.inspect_log run.eval --claim … --status …`
+(official APIs, explicit mapping policy, no silent guessing — see
+[docs/inspect-log.md](docs/inspect-log.md)).
 
 ## 12. Specification & Versioning
 

{agent_comparability-0.5.2 → agent_comparability-0.5.3}/README.md

@@ -7,6 +7,7 @@
 [![CI](https://github.com/Rethymus/agent-comparability/actions/workflows/ci.yml/badge.svg)](https://github.com/Rethymus/agent-comparability/actions/workflows/ci.yml)
 [![License: Apache-2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-3776AB?logo=python&logoColor=white)](pyproject.toml)
+[![PyPI](https://img.shields.io/pypi/v/agent-comparability)](https://pypi.org/project/agent-comparability/)
 
 **English** | [简体中文](README.zh-CN.md)
 
@@ -56,6 +57,12 @@ Agent Comparability is a framework-independent **specification** ([spec/SPEC.md]
 a frozen **conformance suite**, and a thin **reference implementation** that adjudicates
 *comparability* of agent evaluation records — never the scores themselves.
 
+**Does it actually refuse unsupported claims?** See
+[experiments/scenarios_001/RECORD.md](experiments/scenarios_001/RECORD.md): four real-pipeline
+scenarios (real `.eval` logs → converter → core) covering all four verdicts — including the
+confounded comparison and the missing-trajectory case, both correctly **refused** rather than
+packaged as supported conclusions.
+
 **Who is this for**: developers comparing agent/prompt/harness evaluation results,
 benchmark authors documenting what conclusions their data supports, and teams building
 reliability pipelines on top of existing eval frameworks.
@@ -79,6 +86,8 @@ Three failure modes silently corrupt agent-evaluation conclusions:
 | **Evidence gaps** | Missing traces quietly inflate success rates; rankings reverse across harnesses |
 
 These boundaries should be **machine-checkable**, not vibes. That is what this project ships.
+How this relates to benchmark-rigor and transcript-audit work (verified literature anchors):
+see [docs/positioning.md](docs/positioning.md).
 
 ## 3. What Agent Comparability Does
 
@@ -135,10 +144,30 @@ pip install -e .[suite]                 # + suite basics (pyyaml, jsonschema)
 pip install -e .[suite,inspect,pydantic]  # FULL conformance run — required by run_suite.py
 ```
 
-Requires Python ≥ 3.10. No Docker, no model keys.
+Requires Python ≥ 3.10. No Docker, no model keys. Note: the core package supports 3.10+;
+the optional `inspect`/`pydantic` extras pull dependencies that require Python ≥ 3.11 —
+on 3.10 pip resolves older versions of them.
 
 ## 7. Quickstart
 
+**Command line** (offline, zero extra dependencies — input contract below):
+
+```bash
+python -m agent_comparability examples/demo-case.json --claim score_comparison
+# → CDR JSON on stdout (exit 0). INADMISSIBLE / PARTIAL / INCONCLUSIVE are verdicts,
+#   not program failures. Invalid input (bad JSON / missing or mistyped fields) →
+#   exit 2 with a path-precise diagnostic — never silently converted into a verdict.
+#   -o cdr.json writes to a file instead. The demo case is SYNTHETIC/illustrative.
+```
+
+Input contract (`{"id": str, "records": [{"task_id", "harness", "model", "reward":
+number, "reward_status", ...optional}]}`); unknown record fields are ignored and
+recorded in `unknown_conditions` (SPEC §5). Add `--manifest out.manifest.json` to
+also emit an Evidence Manifest documenting artifact digests and evidence trust
+states (never auto-upgraded; see [docs/evidence-manifest.md](docs/evidence-manifest.md)).
+
+**Python API**:
+
 ```python
 from agent_comparability import adjudicate
 
@@ -263,6 +292,10 @@ Both wrap the *same* core (no semantic duplication). Inspect scorer uses
 The Pydantic evaluator checks the **task's real output** (`ctx.output`) against
 frozen expectations — it never re-adjudicates inside the evaluator — and the
 adapter's exit code derives from the full 22-pair `report` (cases + failures).
+For **real Inspect EvalLog files** (not frozen cases) use the converter:
+`python -m agent_comparability.inspect_log run.eval --claim … --status …`
+(official APIs, explicit mapping policy, no silent guessing — see
+[docs/inspect-log.md](docs/inspect-log.md)).
 
 ## 12. Specification & Versioning
 

{agent_comparability-0.5.2 → agent_comparability-0.5.3}/pyproject.toml

@@ -4,13 +4,19 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "agent-comparability"
-version = "0.5.2"
+version = "0.5.3"
 description = "Agent evaluation comparability semantics: spec + conformance suite + thin reference implementation"
 readme = "README.md"
 requires-python = ">=3.10"
 license = {text = "Apache-2.0"}
 dependencies = []
 
+[project.urls]
+Homepage = "https://github.com/Rethymus/agent-comparability"
+Repository = "https://github.com/Rethymus/agent-comparability"
+Issues = "https://github.com/Rethymus/agent-comparability/issues"
+Changelog = "https://github.com/Rethymus/agent-comparability/blob/main/CHANGELOG.md"
+
 [project.optional-dependencies]
 suite = ["pyyaml", "jsonschema"]
 inspect = ["inspect-ai"]

{agent_comparability-0.5.2 → agent_comparability-0.5.3}/src/agent_comparability/__init__.py

@@ -1,3 +1,3 @@
 """Agent Comparability — cmp/0.3.1 参考实现。"""
 from .core import CDR, adjudicate, POLICY_VERSION
-__version__ = "0.5.2"
+__version__ = "0.5.3"

agent_comparability-0.5.3/src/agent_comparability/__main__.py

@@ -0,0 +1,131 @@
+"""离线 JSON → CDR 命令行闭环（#16，零额外运行时依赖）。
+
+用法：
+    python -m agent_comparability input.json --claim score_comparison [-o cdr.json]
+
+输入契约（= adjudicate 既有输入形状）：
+    {"id": str,
+     "records": [{"task_id": str, "harness": str, "model": str,
+                  "reward": number, "reward_status": str,
+                  ...可选 trajectory_available/score_scale/repetition_count}]}
+
+退出码契约：
+    0 = 产出 CDR（ADMISSIBLE/INADMISSIBLE/PARTIAL/INCONCLUSIVE 均为裁决成功，非程序失败）
+    2 = 输入错误（文件不可读/JSON 语法/契约字段缺失或类型错）——诊断含出错路径；
+        无效输入绝不允许被转换为貌似成功的裁决
+
+本模块只做契约存在性与类型检查；一切判定语义由 adjudicate() 负责（不重新实现判定表，
+不改规范语义）。未知字段按 SPEC §5 忽略并记入 CDR 的 unknown_conditions，不在此报错。
+"""
+import argparse
+import json
+import sys
+
+from . import adjudicate
+
+REQUIRED_STR = ("task_id", "harness", "model", "reward_status")
+
+
+class ContractError(Exception):
+    """输入违反契约：消息含出错路径，如 records[2].reward。"""
+
+
+def _is_number(v):
+    return isinstance(v, (int, float)) and not isinstance(v, bool)
+
+
+def _is_int(v):
+    return isinstance(v, int) and not isinstance(v, bool)
+
+
+def validate_case(case):
+    if not isinstance(case, dict):
+        raise ContractError("输入顶层必须是 JSON 对象，实际是 " + type(case).__name__)
+    if "id" not in case:
+        raise ContractError("缺少必填字段 id")
+    if not isinstance(case["id"], str):
+        raise ContractError("id 必须是字符串，实际是 " + type(case["id"]).__name__)
+    if "records" not in case:
+        raise ContractError("缺少必填字段 records")
+    if not isinstance(case["records"], list) or not case["records"]:
+        raise ContractError("records 必须是非空数组")
+    for i, r in enumerate(case["records"]):
+        where = f"records[{i}]"
+        if not isinstance(r, dict):
+            raise ContractError(f"{where} 必须是 JSON 对象，实际是 " + type(r).__name__)
+        for k in REQUIRED_STR:
+            if k not in r:
+                raise ContractError(f"{where}.{k} 缺失（必填字段）")
+            if not isinstance(r[k], str):
+                raise ContractError(f"{where}.{k} 必须是字符串，实际是 " + type(r[k]).__name__)
+        if "reward" not in r:
+            raise ContractError(f"{where}.reward 缺失（必填字段）")
+        if not _is_number(r["reward"]):
+            raise ContractError(f"{where}.reward 必须是数值，实际是 " + type(r["reward"]).__name__)
+        if "trajectory_available" in r and not isinstance(r["trajectory_available"], bool):
+            raise ContractError(f"{where}.trajectory_available 必须是布尔值")
+        if "score_scale" in r and not isinstance(r["score_scale"], dict):
+            raise ContractError(f"{where}.score_scale 必须是对象 {{range, unit, verifier}}")
+        if "repetition_count" in r and (not _is_int(r["repetition_count"]) or r["repetition_count"] < 1):
+            raise ContractError(f"{where}.repetition_count 必须是 >=1 的整数")
+
+
+def main(argv=None) -> int:
+    p = argparse.ArgumentParser(
+        prog="python -m agent_comparability",
+        description="离线评测记录 JSON → 结构化 CDR（Comparison Decision Record）")
+    p.add_argument("input", help="输入 JSON 文件路径（契约见模块文档）")
+    p.add_argument("--claim", required=True,
+                   help="请求结论类型：score_comparison / causal / descriptive_registration"
+                        "（未支持类型 → INCONCLUSIVE，不猜测）")
+    p.add_argument("-o", "--output", help="CDR 输出文件路径（缺省打印到 stdout）")
+    p.add_argument("--manifest", help="同时输出 Evidence Manifest（#18）：输入文件作为证据制品"
+                                      "实算 sha256，全部记录 artifact_observed，不含任何独立验证声明")
+    args = p.parse_args(argv)
+
+    try:
+        raw = open(args.input, encoding="utf-8").read()
+    except OSError as e:
+        print(f"error: 无法读取输入文件 {args.input}: {e.strerror or e}", file=sys.stderr)
+        return 2
+    try:
+        case = json.loads(raw)
+    except json.JSONDecodeError as e:
+        print(f"error: JSON 解析错误 {args.input}（第 {e.lineno} 行第 {e.colno} 列）: {e.msg}", file=sys.stderr)
+        return 2
+    try:
+        validate_case(case)
+    except ContractError as e:
+        print(f"error: 输入违反契约 {args.input}: {e}", file=sys.stderr)
+        return 2
+
+    cdr = adjudicate(case, args.claim).to_dict()
+    out = json.dumps(cdr, ensure_ascii=False, indent=2)
+    if args.output:
+        try:
+            with open(args.output, "w", encoding="utf-8") as f:
+                f.write(out + "\n")
+        except OSError as e:
+            print(f"error: 无法写入输出文件 {args.output}: {e.strerror or e}", file=sys.stderr)
+            return 2
+    else:
+        print(out)
+    if args.manifest:
+        from .evidence import build_manifest
+        # 离线 JSON 的诚实默认：输入文件即证据制品（读后实算摘要）；无来源端验证能力，
+        # 无独立验证声明——绝不把 source_claimed 冒充为已验证。
+        m = build_manifest(case, source_kind="offline_json",
+                           artifacts={i: args.input for i in range(len(case["records"]))},
+                           requested_claim=args.claim,
+                           unsupported_capabilities=["source_side_verification_unavailable"])
+        try:
+            with open(args.manifest, "w", encoding="utf-8") as f:
+                f.write(json.dumps(m, ensure_ascii=False, indent=2) + "\n")
+        except OSError as e:
+            print(f"error: 无法写入 manifest 文件 {args.manifest}: {e.strerror or e}", file=sys.stderr)
+            return 2
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

agent_comparability-0.5.3/src/agent_comparability/evidence.py

@@ -0,0 +1,109 @@
+"""Evidence Manifest（#18）——CDR 外部的来源与证据可信边界。
+
+三态（**证据制品级**，与 SPEC §1 的记录级评分信任四层是不同维度，绝不进入
+adjudicate、不改变任何 CDR 字段或判定结果）：
+
+    source_claimed < artifact_observed < independently_verified
+
+升级只能由显式证据触发：
+  - artifact_observed ⟺ 制品路径存在且被实际读取且 sha256 已实算记录；
+  - independently_verified ⟺ 在 artifact_observed 之上存在显式 verification 条目
+    （method/operator/at 三者齐备）；
+  - 无制品 → 地板 source_claimed + 警告；有 verification 无制品 → 不一致，不升级。
+不存在任何隐式跃迁（防"证据不足被包装成可信结论"）。
+"""
+import hashlib
+
+from . import POLICY_VERSION
+
+MANIFEST_VERSION = "evidence/0.1"
+STATES = ("source_claimed", "artifact_observed", "independently_verified")
+
+
+def _digest(path):
+    h = hashlib.sha256()
+    try:
+        with open(path, "rb") as f:
+            for chunk in iter(lambda: f.read(65536), b""):
+                h.update(chunk)
+    except OSError:
+        return None
+    return "sha256:" + h.hexdigest()
+
+
+def build_manifest(case, source_kind, source_version=None, artifacts=None,
+                   verifications=None, requested_claim=None,
+                   unsupported_capabilities=None):
+    """构建 manifest。artifacts: {record_index: 制品路径}；verifications: {record_index:
+    {method, operator, at}}。状态只由提供的证据决定——本函数绝不主动升级。"""
+    artifacts = artifacts or {}
+    verifications = verifications or {}
+    warnings = []
+    records = []
+    for i, r in enumerate(case["records"]):
+        state, digest, art_ref, reason = "source_claimed", None, None, None
+        if i in artifacts:
+            art_ref = artifacts[i]
+            digest = _digest(art_ref)
+            if digest is None:
+                warnings.append(f"records[{i}]:artifact_missing({art_ref} 不可读——保持 source_claimed)")
+                reason = f"artifact_unreadable:{art_ref}"
+            else:
+                state = "artifact_observed"
+        else:
+            warnings.append(f"records[{i}]:artifact_missing(未提供制品——保持 source_claimed)")
+            reason = "no_artifact_provided"
+        verification = None
+        if i in verifications:
+            v = verifications[i]
+            valid = isinstance(v, dict) and {"method", "operator", "at"} <= set(v)
+            if state != "artifact_observed":
+                warnings.append(f"records[{i}]:verification_without_artifact("
+                                f"无法独立验证从未观察过的制品——不升级)")
+            elif not valid:
+                warnings.append(f"records[{i}]:verification_entry_invalid(需 method/operator/at)")
+            else:
+                state = "independently_verified"
+                verification = {"method": v["method"], "operator": v["operator"], "at": v["at"]}
+        records.append({
+            "record_id": f"records[{i}]",
+            "task_identity": {"task_id": r["task_id"], "model": r["model"], "harness": r["harness"]},
+            "artifact_ref": art_ref,
+            "file_digest": digest,
+            "verification_state": state,
+            "verification": verification,
+            "unverified_reason": reason,
+            "provenance_note": (f"state={state}; artifact="
+                                f"{'read+sha256' if digest else 'none'}; source={source_kind}"),
+        })
+    return {
+        "manifest_version": MANIFEST_VERSION,
+        "policy_version": POLICY_VERSION,  # 引用锚定：标识判定所用规范版本，不参与判定
+        "source_kind": source_kind,
+        "source_version": source_version,  # None = 不可取得（诚实记录，不猜测）
+        "case_id": case["id"],
+        "requested_claim": requested_claim,
+        "records": records,
+        "mapping_warnings": warnings,
+        "unsupported_capabilities": unsupported_capabilities or [],
+    }
+
+
+def verify_manifest(manifest):
+    """重算校验（供审计与后续转换器复用）：重读 artifact 实算摘要比对记录值。
+    返回问题列表；空列表 = 干净。只验证已声明观察过的制品——不产生新状态。"""
+    problems = []
+    for e in manifest.get("records", []):
+        rid, ref, dig = e.get("record_id"), e.get("artifact_ref"), e.get("file_digest")
+        if e.get("verification_state") in ("artifact_observed", "independently_verified"):
+            if not ref or not dig:
+                problems.append(f"{rid}:structural(声明已观察但缺 artifact_ref/file_digest)")
+                continue
+            now = _digest(ref)
+            if now is None:
+                problems.append(f"{rid}:artifact_missing_on_reverify({ref})")
+            elif now != dig:
+                problems.append(f"{rid}:digest_mismatch({ref})")
+        if e.get("verification_state") == "independently_verified" and not e.get("verification"):
+            problems.append(f"{rid}:structural(verified 态缺 verification 条目)")
+    return problems