bioai-evidence-validator 0.5.0__py3-none-any.whl → 0.7.0__py3-none-any.whl

{bioai_evidence_validator-0.5.0.dist-info → bioai_evidence_validator-0.7.0.dist-info}/METADATA

@@ -1,9 +1,9 @@
 Metadata-Version: 2.5
 Name: bioai-evidence-validator
-Version: 0.5.0
+Version: 0.7.0
 Summary: Standards-aligned evidence policy validation for AI-assisted biological curation
 Project-URL: Homepage, https://github.com/NingyuSUN/bioai-evidence-validator
-Project-URL: Documentation, https://github.com/NingyuSUN/bioai-evidence-validator/tree/main/docs
+Project-URL: Documentation, https://ningyusun.github.io/bioai-evidence-validator/
 Project-URL: Changelog, https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/CHANGELOG.md
 Project-URL: Issues, https://github.com/NingyuSUN/bioai-evidence-validator/issues
 Author: Ningyu Sun
@@ -25,7 +25,13 @@ Requires-Dist: jsonschema<5,>=4.23
 Requires-Dist: linkml<2,>=1.8
 Requires-Dist: pyyaml<7,>=6.0
 Provides-Extra: dev
+Requires-Dist: mypy<3,>=2.3; extra == 'dev'
+Requires-Dist: openpyxl<4,>=3.1; extra == 'dev'
+Requires-Dist: pytest-cov<8,>=7; extra == 'dev'
 Requires-Dist: pytest<9,>=8; extra == 'dev'
+Requires-Dist: ruff<0.17,>=0.16; extra == 'dev'
+Requires-Dist: types-jsonschema; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
 Description-Content-Type: text/markdown
 
 # BioAI Evidence Validator
@@ -35,6 +41,7 @@ Description-Content-Type: text/markdown
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/pyproject.toml)
 [![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/LICENSE)
 [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NingyuSUN/bioai-evidence-validator/blob/main/examples/quickstart.ipynb)
+[![Docs](https://img.shields.io/badge/docs-site-blue)](https://ningyusun.github.io/bioai-evidence-validator/)
 
 **Stop AI-extracted biological claims from entering your knowledge base or
 training set before their evidence is good enough for that use.**
@@ -52,6 +59,7 @@ pip install bioai-evidence-validator
 
 Or try it in the browser, nothing to install:
 [quickstart notebook on Colab](https://colab.research.google.com/github/NingyuSUN/bioai-evidence-validator/blob/main/examples/quickstart.ipynb).
+Full documentation: **[ningyusun.github.io/bioai-evidence-validator](https://ningyusun.github.io/bioai-evidence-validator/)**.
 
 ## 30-second example
 
@@ -103,9 +111,12 @@ record is trustworthy enough for a particular purpose.
 | Provenance consistency (source hashes, resolved references, scope) | — | ✅ |
 | Human adjudications bound to a specific statement and use | — | ✅ |
 | Machine-readable audit report with hashes of input, schema and profile | — | ✅ |
+| Mapped to ECO, Biolink and GA4GH VA-Spec ([standards alignment](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/STANDARDS.md)) | — | ✅ |
 
-On the real-data benchmark below, schema-only checks admitted **160/160**
-injected faults; the full validator admitted **0/160**.
+On both real-data benchmarks below, schema-only checks admitted **160/160**
+injected faults; the full validator admitted **0/160**. On ClinVar, 1★ and 2★ variants
+the validator holds back were **about 4–5× more likely** to be reclassified or put in
+conflict three years later.
 
 ## Use it
 
@@ -178,7 +189,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: NingyuSUN/bioai-evidence-validator@v0.5.0
+      - uses: NingyuSUN/bioai-evidence-validator@v0.7.0
         with:
           files: records/**/*.yaml        # whitespace-separated globs
           format: draft                   # or: record (default)
@@ -250,7 +261,36 @@ Use the [annotation templates](https://github.com/NingyuSUN/bioai-evidence-valid
 source version and intended use. Document reviewer roles and whether labels are
 single-reviewed or independently reviewed by multiple people.
 
-## Real-data case
+## Real-data cases
+
+### ClinVar germline classifications, three years later
+
+The [ClinVar case](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/examples/clinvar_germline/README.md)
+turns every 2023-09 lab submission into evidence, validates 5,026 sampled variants with a
+ClinVar-style profile, and checks what happened to them by 2026-09.
+
+![ClinVar benchmark: share of 2023 pathogenic classifications reclassified or conflicting by 2026, with and without a dissenting submission, and false admissions under controlled faults and the trust boundary](https://raw.githubusercontent.com/NingyuSUN/bioai-evidence-validator/main/docs/assets/clinvar_germline_benchmark.svg)
+
+- **Policy reproduction:** the profile matches NCBI's own 2023-09 review status on 98–99% of
+  decisions; every disagreement is listed with its cause.
+- **Where the validator is stricter, classifications were less stable.** It sends any P/LP
+  variant with a dissenting submission to review, even when ClinVar's aggregate does not.
+  Across all 218,920 germline P/LP variants, the share later reclassified or put in conflict:
+
+| 2023-09 ClinVar review status | No dissenting submission | With one (validator: review) |
+|---|---:|---:|
+| 1★ single submitter | 2.63% (2.54–2.71) | **13.77%** (11.46–16.47) |
+| 2★ multiple submitters | 2.19% (2.05–2.33) | **8.32%** (6.51–10.59) |
+| 3★ expert panel | 0.07% (0.03–0.16) | **1.30%** (0.63–2.66) |
+
+Wilson 95% intervals. Stability is not correctness, and this is observational; see the
+case's interpretation limits. Not for clinical use.
+
+```bash
+uv run python examples/clinvar_germline/run.py --output artifacts/clinvar
+```
+
+### VBO canine name mapping
 
 [VBO canine name mapping](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/examples/vbo_canine/README.md) uses a frozen public ontology:
 72 real-name cases, 160 controlled errors, and 16 separately reported trust-boundary
@@ -261,7 +301,7 @@ per-required-evidence-type validation. Source-derived labels are not expert anno
 uv run python examples/vbo_canine/run.py --output artifacts/vbo-canine
 ```
 
-### Benchmark results (v0.4.1)
+#### Benchmark results (v0.4.1)
 
 ![VBO canine benchmark comparing false admissions across three validation methods](https://raw.githubusercontent.com/NingyuSUN/bioai-evidence-validator/main/docs/assets/vbo_canine_benchmark.svg)
 
@@ -271,20 +311,31 @@ On 72 real-source name mappings, the full validator admitted all 48 unambiguous
 
 ## Scope
 
-The VBO case uses attributed public data; other fixtures are synthetic.
+The VBO and ClinVar cases use attributed public data; other fixtures are synthetic.
 Admission means **the supplied record meets the selected
 profile**, not that a biological claim is true. The toolkit does not retrieve papers,
-verify reviewer identities, train models, or measure prediction accuracy. The generic core compares supplied hashes; the VBO importer also hashes its local source
-projection. External source truth and cohort independence require upstream verification.
+verify reviewer identities, train models, or measure prediction accuracy. The generic core compares supplied hashes; the VBO and ClinVar importers also hash their local source
+projections. External source truth and cohort independence require upstream verification.
+Neither benchmark has independent expert annotation yet; a blinded [expert-review kit](https://github.com/NingyuSUN/bioai-evidence-validator/tree/main/evaluation/clinvar_review) for the ClinVar case is ready for reviewers.
 
 ## Versions and branches
 
-`main` is the domain-neutral framework (0.5.0). The complete canine implementation
-and SQLite adapter from 0.3 live on the
-[`canine-breed` branch](https://github.com/NingyuSUN/bioai-evidence-validator/tree/canine-breed);
+`main` is the domain-neutral framework (0.7.0). The complete canine implementation
+and SQLite adapter from 0.3 are preserved at the
+[`canine-0.3` tag](https://github.com/NingyuSUN/bioai-evidence-validator/tree/canine-0.3)
+(also the `canine-breed` branch);
 see the [0.4 migration guide](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/MIGRATION-0.4.md) and
 [changelog](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/CHANGELOG.md).
 
+## Contributing
+
+Bug reports, domain profiles and new benchmarks are welcome. See
+[CONTRIBUTING.md](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/CONTRIBUTING.md),
+the [community profiles](https://github.com/NingyuSUN/bioai-evidence-validator/tree/main/community/profiles)
+and issues labelled [`good first issue`](https://github.com/NingyuSUN/bioai-evidence-validator/labels/good%20first%20issue).
+Report security problems privately as described in
+[SECURITY.md](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/SECURITY.md).
+
 ## Citing
 
 If you use this toolkit in research, please cite it using the metadata in
@@ -293,6 +344,7 @@ If you use this toolkit in research, please cite it using the metadata in
 
 [Create a profile](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/PROFILES.md) ·
 [Draft format](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/DRAFTS.md) ·
+[Standards alignment](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/STANDARDS.md) ·
 [Engineering contract](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/ENGINEERING.md) ·
 [Design case study](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/CASE_STUDY.md) ·
 [Architecture decision](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/ADR-002-domain-neutral-main.md) ·

bioai_evidence_validator-0.7.0.dist-info/RECORD

@@ -0,0 +1,15 @@
+bioevidence_validator/__init__.py,sha256=PsEzz-ifHjfLOJX7KkUqMFNbniIQMNPKRFxgNdoF3JI,312
+bioevidence_validator/cli.py,sha256=7TxCuCN0jcbFwUnim0QiIARJC3XPQcsYN5qBcT7EFF0,9965
+bioevidence_validator/config.py,sha256=Ahc9eRPNR_ArqnbnJkdZsPFq9yERTYied0oGwUkANYI,1110
+bioevidence_validator/draft.py,sha256=2ge9l1BA44e-_ukO_Nnz9UZAV_RS8yqj3Ujr86eFxtY,13631
+bioevidence_validator/engine.py,sha256=OSH9Zd-UqCVqYRU6eLkyoavRWAJGzz8Q6mcCNkhW48I,14989
+bioevidence_validator/review.py,sha256=yp9DI2TafTiZqZmNTejtUb9oHeeXPEjZHXx3fe2qCGI,20293
+bioevidence_validator/profiles/dataset-label.yaml,sha256=4utz9YcFFF2UnaLjIb0ATb-27hF8Smw7zmv0sBTaQdM,843
+bioevidence_validator/profiles/general.yaml,sha256=i9fJCWovJZ2e2d4rbn0Bx7mrhumNb2zjS9TOarkBWZw,778
+bioevidence_validator/profiles/literature-claim.yaml,sha256=7l82gX3oGGyYFq9B29-1qGXeL4LRcMhJi8Y9fieVE0c,612
+bioevidence_validator/schema/bioevidence_core.yaml,sha256=c2YIBOhuT6v7j4MDsMScImdyLZCMy-3X--ExCB_IYu8,7514
+bioai_evidence_validator-0.7.0.dist-info/METADATA,sha256=mt7dqba6hjGQjYBUoBRTS_09bvyD7GU3x1c_RmK4amg,18818
+bioai_evidence_validator-0.7.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+bioai_evidence_validator-0.7.0.dist-info/entry_points.txt,sha256=6oufzT2x1nIhTqM5B6y4aWubExKAoFb_wXXTdz3mBQQ,63
+bioai_evidence_validator-0.7.0.dist-info/licenses/LICENSE,sha256=z8d0m5b2O9McPEK1xHG_dWgUBT6EfBDz6wA0F7xSPTA,11358
+bioai_evidence_validator-0.7.0.dist-info/RECORD,,

bioevidence_validator/__init__.py

@@ -1,6 +1,6 @@
 """Evidence validation primitives and domain policy runners."""
 
-__version__ = "0.5.0"
+__version__ = "0.7.0"
 
 from .draft import build_record, draft_json_schema, load_draft  # noqa: E402
 from .engine import validate_record  # noqa: E402

bioevidence_validator/cli.py

@@ -9,8 +9,9 @@ from pathlib import Path
 
 import yaml
 
+from . import review as review_module
 from .draft import build_record, draft_json_schema, load_draft
-from .engine import default_schema_path, generate_json_schema, validate_record, profile_path, list_profiles
+from .engine import default_schema_path, generate_json_schema, list_profiles, profile_path, validate_record
 
 
 def parser() -> argparse.ArgumentParser:
@@ -31,6 +32,36 @@ def parser() -> argparse.ArgumentParser:
     generate.add_argument("--output", type=Path, required=True)
     generate.add_argument("--schema", type=Path, default=default_schema_path())
     commands.add_parser("profiles", help="List built-in profiles and their use contracts")
+
+    review = commands.add_parser("review", help="Independent human review: agreement, adjudication, scoring")
+    steps = review.add_subparsers(dest="step", required=True)
+    check = steps.add_parser("check", help="Check annotations (and adjudications) against the protocol format")
+    check.add_argument("annotations", type=Path)
+    check.add_argument("--adjudications", type=Path)
+    agree = steps.add_parser("agreement", help="Inter-reviewer agreement per use, before adjudication")
+    agree.add_argument("annotations", type=Path)
+    agree.add_argument("--output", type=Path)
+    sheet = steps.add_parser("adjudication-sheet", help="Blank adjudication rows for every disagreement")
+    sheet.add_argument("annotations", type=Path)
+    sheet.add_argument("--output", type=Path, required=True)
+    sheet.add_argument("--min-reviewers", type=int, default=2)
+    scoring = steps.add_parser("score", help="Compare predictions with the resolved reference labels")
+    scoring.add_argument("--annotations", type=Path, required=True)
+    scoring.add_argument("--adjudications", type=Path, required=True)
+    scoring.add_argument("--predictions", type=Path, required=True)
+    scoring.add_argument("--split", default="test", choices=sorted(review_module.SPLITS))
+    scoring.add_argument("--min-reviewers", type=int, default=2)
+    scoring.add_argument("--output", type=Path)
+    frozen = steps.add_parser("freeze", help="Write a frozen manifest for a completed reference set")
+    frozen.add_argument("--manifest", type=Path, required=True, help="Draft manifest to complete")
+    frozen.add_argument("--annotations", type=Path, required=True)
+    frozen.add_argument("--adjudications", type=Path, required=True)
+    frozen.add_argument("--file", type=Path, action="append", default=[], help="Extra file to hash (repeatable)")
+    frozen.add_argument("--dataset-id", required=True)
+    frozen.add_argument("--version", required=True)
+    frozen.add_argument("--frozen-at", required=True, help="ISO 8601 time of the freeze")
+    frozen.add_argument("--min-reviewers", type=int, default=2)
+    frozen.add_argument("--output", type=Path, required=True)
     return result
 
 
@@ -88,6 +119,9 @@ def _run(args) -> int:
         print(json.dumps(list_profiles(), indent=2))
         return 0
 
+    if args.command == "review":
+        return _review(args)
+
     if args.command in {"build", "draft-schema"}:
         if args.command == "build":
             if args.output and args.output.resolve() == args.draft.resolve():
@@ -123,6 +157,48 @@ def _run(args) -> int:
     return 0
 
 
+def _review(args) -> int:
+    rv = review_module
+    annotations = rv.load_annotations(args.annotations)
+    if args.step == "check":
+        adjudications = rv.load_adjudications(args.adjudications, annotations) if args.adjudications else []
+        units = {(a["case_id"], a["requested_use"]) for a in annotations}
+        print(f"OK: {len(annotations)} annotations, {len(units)} case-use units, "
+              f"{len({a['reviewer_id'] for a in annotations})} reviewers, {len(adjudications)} adjudications")
+        return 0
+    if args.step == "agreement":
+        report = rv.agreement(annotations)
+        if args.output:
+            _write_json(args.output, report)
+        print(rv.render_agreement(report), end="")
+        return 0
+    if args.step == "adjudication-sheet":
+        if args.output.exists():
+            raise ValueError(f"{args.output} exists; refusing to overwrite adjudication work")
+        rows, incomplete = rv.adjudication_sheet(annotations, min_reviewers=args.min_reviewers)
+        rv.write_csv(args.output, rows, rv.ADJUDICATION_COLUMNS)
+        print(f"{len(rows)} disagreement(s) to adjudicate; {len(incomplete)} unit(s) with too few reviews")
+        return 0
+    adjudications = rv.load_adjudications(args.adjudications, annotations)
+    final = rv.resolve(annotations, adjudications, min_reviewers=args.min_reviewers)
+    if args.step == "score":
+        report = rv.score(final, rv.load_predictions(args.predictions), split=args.split)
+        if args.output:
+            _write_json(args.output, report)
+        print(rv.render_score(report), end="")
+        return 0
+    manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
+    files = [args.annotations, args.adjudications, *args.file]
+    if args.output.resolve() in {f.resolve() for f in [args.manifest, *files]}:
+        raise ValueError("Output must not overwrite an input")
+    frozen = rv.freeze(manifest, final, annotations, files, dataset_id=args.dataset_id,
+                       version=args.version, frozen_at=args.frozen_at)
+    _write_json(args.output, frozen)
+    print(f"Frozen {frozen['reviewed_case_use_count']} case-use labels from "
+          f"{frozen['independent_human_reviewer_count']} reviewers ({frozen['reference_type']})")
+    return 0
+
+
 def main(argv: list[str] | None = None) -> int:
     args = parser().parse_args(argv)
     try:

bioevidence_validator/draft.py

@@ -80,8 +80,6 @@ def _entity(value: Any, path: str) -> dict:
 
 def _source(value: Any, path: str, base_dir: Path) -> tuple[str, dict]:
     source = _fields(value, SOURCE_KEYS, path)
-    if "sha256" not in source and "file" not in source:
-        raise ValueError(f"{path} needs sha256, file, or both")
     artifact = {"title": _text(source["title"], path + ".title"),
                 "source_type": _text(source["type"], path + ".type", SOURCE_TYPES)}
     if "uri" in source:
@@ -96,8 +94,12 @@ def _source(value: Any, path: str, base_dir: Path) -> tuple[str, dict]:
         observed = hashlib.sha256(file.read_bytes()).hexdigest()
     # A stated hash is the frozen reference; a file hash is what was observed now.
     # With both, validation reports a mismatch (BEV002) instead of trusting either.
-    artifact["sha256"] = _text(source["sha256"], path + ".sha256").lower() if "sha256" in source else observed
-    if "sha256" in source and observed is not None:
+    stated = _text(source["sha256"], path + ".sha256").lower() if "sha256" in source else None
+    frozen = stated or observed
+    if frozen is None:
+        raise ValueError(f"{path} needs sha256, file, or both")
+    artifact["sha256"] = frozen
+    if stated is not None and observed is not None:
         artifact["observed_sha256"] = observed
     return _text(source["id"], path + ".id"), artifact
 
@@ -129,14 +131,15 @@ def build_record(draft: dict, *, base_dir: Path | str = ".") -> dict[str, Any]:
         local_ids[name] = f"{record_id}/source/{name}"
         sources.append({"id": local_ids[name], **artifact})
 
-    items, lines = [], {}
+    items: list[dict[str, Any]] = []
+    lines: dict[str, list[str]] = {}
     for index, row in enumerate(_rows(draft["evidence"], "draft.evidence")):
         path = f"draft.evidence[{index}]"
         evidence = _fields(row, EVIDENCE_KEYS, path)
         source = _text(evidence["source"], path + ".source")
         if source not in local_ids:
             raise ValueError(f"{path}.source {source!r} is not a declared source id")
-        item = {"id": f"{record_id}/evidence/{index + 1}", "source_artifact_id": local_ids[source],
+        item: dict[str, Any] = {"id": f"{record_id}/evidence/{index + 1}", "source_artifact_id": local_ids[source],
                 "locator": _text(evidence["locator"], path + ".locator")}
         if "text" in evidence:
             item["extracted_text"] = _text(evidence["text"], path + ".text")

bioevidence_validator/engine.py

@@ -3,7 +3,7 @@ from __future__ import annotations
 import hashlib
 import json
 from dataclasses import asdict, dataclass
-from datetime import datetime, timezone
+from datetime import UTC, datetime
 from pathlib import Path
 from typing import Any
 
@@ -255,7 +255,7 @@ class RecordValidator:
             "profile_sha256": self.profile_sha256,
             "schema_version": self.schema_version, "schema_sha256": self.schema_sha256,
             "schema_sources": [dict(source) for source in self.schema_sources],
-            "input_sha256": sha256_bytes(canonical), "validated_at": datetime.now(timezone.utc).isoformat(),
+            "input_sha256": sha256_bytes(canonical), "validated_at": datetime.now(UTC).isoformat(),
             "schema_valid": not any(f.rule_id == "SCHEMA" for f in findings),
             "overall_status": overall, "findings": [asdict(f) for f in findings], "use_decisions": decisions,
         }

bioevidence_validator/review.py

@@ -0,0 +1,388 @@
+"""Independent human review: check annotations, measure agreement, adjudicate, score and freeze.
+
+Implements the workflow in docs/GOLD_STANDARD.md for the CSV formats in evaluation/gold_standard/.
+Reference labels live here, never in the engine: they evaluate decisions, they do not make them.
+"""
+from __future__ import annotations
+
+import csv
+import datetime as dt
+import hashlib
+import json
+import math
+import random
+import re
+from collections import Counter, defaultdict
+from collections.abc import Sequence
+from pathlib import Path
+from typing import Any
+
+ANNOTATION_COLUMNS = [
+    "annotation_id", "case_id", "record_sha256", "profile_id", "profile_sha256", "requested_use", "group_id",
+    "split", "reviewer_id", "reviewer_qualification", "annotated_at", "mapping_label", "mapping_rationale",
+    "admission_label", "admission_rationale", "evidence_refs_json",
+]
+OPTIONAL_ANNOTATION_COLUMNS = ["later_information_seen", "minutes_spent"]
+ADJUDICATION_COLUMNS = [
+    "adjudication_id", "case_id", "record_sha256", "profile_sha256", "requested_use", "annotation_ids_json",
+    "mapping_label", "admission_label", "adjudicator_id", "adjudicated_at", "mapping_rationale",
+    "admission_rationale", "evidence_refs_json",
+]
+PREDICTION_COLUMNS = ["case_id", "requested_use", "record_sha256", "profile_sha256", "method", "predicted_status"]
+MAPPING_LABELS = ["correct", "incorrect", "uncertain"]
+ADMISSION_LABELS = ["admitted", "review_required", "rejected"]
+PREDICTED = ADMISSION_LABELS + ["not_admitted"]  # binary baselines predict admitted / not_admitted
+SPLITS = {"development", "test"}
+HEX64 = re.compile(r"^[0-9a-f]{64}$")
+
+Unit = tuple[str, str]  # (case_id, requested_use)
+
+
+def _read_csv(path: Path, required: list[str], optional: Sequence[str] = ()) -> list[dict[str, str]]:
+    with Path(path).open(encoding="utf-8-sig", newline="") as handle:  # tolerate Excel's BOM
+        reader = csv.DictReader(handle)
+        header = reader.fieldnames or []
+        missing = [c for c in required if c not in header]
+        unknown = [c for c in header if c not in required and c not in optional]
+        if missing or unknown:
+            raise ValueError(f"{path}: missing columns {missing}, unknown columns {unknown}")
+        return [dict(row) for row in reader]
+
+
+def _iso(value: str, where: str) -> None:
+    try:
+        dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
+    except ValueError:
+        raise ValueError(f"{where}: not an ISO 8601 time: {value!r}") from None
+
+
+def _json_list(value: str, where: str) -> list:
+    try:
+        parsed = json.loads(value)
+    except json.JSONDecodeError:
+        raise ValueError(f"{where}: not valid JSON: {value!r}") from None
+    if not isinstance(parsed, list) or not all(isinstance(x, str) and x.strip() for x in parsed):
+        raise ValueError(f"{where}: must be a JSON array of nonblank strings")
+    return parsed
+
+
+def load_annotations(path: Path) -> list[dict[str, str]]:
+    """Read and strictly check an annotations CSV; raise ValueError listing the first problems."""
+    rows = _read_csv(path, ANNOTATION_COLUMNS, OPTIONAL_ANNOTATION_COLUMNS)
+    errors: list[str] = []
+    seen_ids: set[str] = set()
+    seen_reviews: set[tuple[str, str, str]] = set()
+    bindings: dict[Unit, tuple[str, ...]] = {}
+    for number, row in enumerate(rows, start=2):  # line 1 is the header
+        where = f"{path}:{number}"
+        blank = [c for c in ANNOTATION_COLUMNS
+                 if c not in ("mapping_rationale", "admission_rationale") and not (row[c] or "").strip()]
+        if blank:
+            errors.append(f"{where}: blank {blank}")
+            continue
+        if row["annotation_id"] in seen_ids:
+            errors.append(f"{where}: duplicate annotation_id {row['annotation_id']!r}")
+        seen_ids.add(row["annotation_id"])
+        review = (row["case_id"], row["requested_use"], row["reviewer_id"])
+        if review in seen_reviews:
+            errors.append(f"{where}: reviewer {row['reviewer_id']!r} labelled {review[:2]} twice")
+        seen_reviews.add(review)
+        if row["mapping_label"] not in MAPPING_LABELS:
+            errors.append(f"{where}: mapping_label must be one of {MAPPING_LABELS}")
+        if row["admission_label"] not in ADMISSION_LABELS:
+            errors.append(f"{where}: admission_label must be one of {ADMISSION_LABELS}")
+        if row["split"] not in SPLITS:
+            errors.append(f"{where}: split must be one of {sorted(SPLITS)}")
+        for column in ("record_sha256", "profile_sha256"):
+            if not HEX64.match(row[column]):
+                errors.append(f"{where}: {column} must be 64 lowercase hex characters")
+        try:
+            _iso(row["annotated_at"], where)
+            _json_list(row["evidence_refs_json"], where)
+        except ValueError as exc:
+            errors.append(str(exc))
+        binding = (row["record_sha256"], row["profile_sha256"], row["profile_id"], row["group_id"], row["split"])
+        if bindings.setdefault(review[:2], binding) != binding:
+            errors.append(f"{where}: reviewers of {review[:2]} saw a different record, profile, group or split")
+    if errors:
+        raise ValueError("Invalid annotations:\n" + "\n".join(errors[:20]))
+    return rows
+
+
+def load_adjudications(path: Path, annotations: list[dict[str, str]]) -> list[dict[str, str]]:
+    rows = _read_csv(path, ADJUDICATION_COLUMNS)
+    by_id = {a["annotation_id"]: a for a in annotations}
+    errors, units = [], set()
+    for number, row in enumerate(rows, start=2):
+        where = f"{path}:{number}"
+        blank = [c for c in ADJUDICATION_COLUMNS if not (row[c] or "").strip()]
+        if blank:
+            errors.append(f"{where}: blank {blank}")
+            continue
+        unit = (row["case_id"], row["requested_use"])
+        if unit in units:
+            errors.append(f"{where}: {unit} adjudicated twice")
+        units.add(unit)
+        if row["mapping_label"] not in MAPPING_LABELS or row["admission_label"] not in ADMISSION_LABELS:
+            errors.append(f"{where}: labels must be from {MAPPING_LABELS} and {ADMISSION_LABELS}")
+        try:
+            _iso(row["adjudicated_at"], where)
+            _json_list(row["evidence_refs_json"], where)
+            linked = _json_list(row["annotation_ids_json"], where)
+        except ValueError as exc:
+            errors.append(str(exc))
+            continue
+        for annotation_id in linked:
+            original = by_id.get(annotation_id)
+            if original is None or (original["case_id"], original["requested_use"]) != unit:
+                errors.append(f"{where}: {annotation_id!r} is not an annotation of {unit}")
+            elif (original["record_sha256"], original["profile_sha256"]) != (row["record_sha256"], row["profile_sha256"]):
+                errors.append(f"{where}: record or profile hash differs from annotation {annotation_id!r}")
+    if errors:
+        raise ValueError("Invalid adjudications:\n" + "\n".join(errors[:20]))
+    return rows
+
+
+def _units(annotations: list[dict[str, str]], field: str, use: str | None = None) -> dict[Unit, dict[str, str]]:
+    units: dict[Unit, dict[str, str]] = defaultdict(dict)
+    for row in annotations:
+        if use is None or row["requested_use"] == use:
+            units[(row["case_id"], row["requested_use"])][row["reviewer_id"]] = row[field]
+    return units
+
+
+def krippendorff_alpha_nominal(units: list[list[str]]) -> float | None:
+    """Krippendorff's alpha for nominal data; units with fewer than two values are not pairable.
+
+    Returns None when there is no variation to measure agreement against (expected disagreement is 0).
+    """
+    coincidence: dict[tuple[str, str], float] = defaultdict(float)
+    for values in units:
+        if len(values) < 2:
+            continue
+        for i, c in enumerate(values):
+            for j, k in enumerate(values):
+                if i != j:
+                    coincidence[(c, k)] += 1 / (len(values) - 1)
+    totals: dict[str, float] = defaultdict(float)
+    for (c, _), weight in coincidence.items():
+        totals[c] += weight
+    n = sum(totals.values())
+    observed = sum(w for (c, k), w in coincidence.items() if c != k)
+    expected = (n * n - sum(t * t for t in totals.values())) / (n - 1) if n > 1 else 0.0
+    return None if expected == 0 else 1 - observed / expected
+
+
+def cohen_kappa(pairs: list[tuple[str, str]]) -> float | None:
+    if not pairs:
+        return None
+    n = len(pairs)
+    observed = sum(a == b for a, b in pairs) / n
+    first, second = Counter(a for a, _ in pairs), Counter(b for _, b in pairs)
+    expected = sum(first[c] * second[c] for c in set(first) | set(second)) / (n * n)
+    return None if expected == 1 else (observed - expected) / (1 - expected)
+
+
+def wilson(events: int, n: int, z: float = 1.959963984540054) -> list[float] | None:
+    if not n:
+        return None
+    p = events / n
+    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
+    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
+    return [round(max(0.0, centre - half), 6), round(min(1.0, centre + half), 6)]
+
+
+def _bootstrap_alpha(units: list[list[str]], resamples: int, seed: int) -> list[float] | None:
+    rng = random.Random(seed)
+    values = []
+    for _ in range(resamples):
+        alpha = krippendorff_alpha_nominal([units[rng.randrange(len(units))] for _ in units])
+        if alpha is not None:
+            values.append(alpha)
+    if len(values) < resamples * 0.9:
+        return None  # too many resamples without variation for a meaningful interval
+    values.sort()
+    last = len(values) - 1
+    return [round(values[int(0.025 * last)], 4), round(values[math.ceil(0.975 * last)], 4)]
+
+
+def agreement(annotations: list[dict[str, str]], *, resamples: int = 2000, seed: int = 20260925) -> dict[str, Any]:
+    """Inter-reviewer agreement per requested use, before adjudication."""
+    reviewers = sorted({row["reviewer_id"] for row in annotations})
+    result: dict[str, Any] = {"reviewers": reviewers, "uses": {}}
+    for use in sorted({row["requested_use"] for row in annotations}):
+        per_field = {}
+        for field in ("admission_label", "mapping_label"):
+            units = _units(annotations, field, use)
+            multi = [list(v.values()) for v in units.values() if len(v) >= 2]
+            alpha = krippendorff_alpha_nominal(multi)
+            entry: dict[str, Any] = {
+                "units": len(units), "units_with_2plus_reviews": len(multi),
+                "percent_agreement": round(sum(len(set(v)) == 1 for v in multi) / len(multi), 4) if multi else None,
+                "krippendorff_alpha": None if alpha is None else round(alpha, 4),
+                "alpha_95_bootstrap": _bootstrap_alpha(multi, resamples, seed) if multi else None,
+            }
+            if len(reviewers) == 2:
+                a, b = reviewers
+                pairs = [(v[a], v[b]) for v in units.values() if a in v and b in v]
+                kappa = cohen_kappa(pairs)
+                entry["cohen_kappa"] = None if kappa is None else round(kappa, 4)
+                entry["confusion"] = {f"{x}|{y}": n for (x, y), n in sorted(Counter(pairs).items())}
+            per_field[field] = entry
+        result["uses"][use] = per_field
+    return result
+
+
+def adjudication_sheet(annotations: list[dict[str, str]], *, min_reviewers: int = 2) -> tuple[list[dict], list[Unit]]:
+    """Blank adjudication rows for every disagreement, plus units that still lack enough reviews."""
+    grouped: dict[Unit, list[dict[str, str]]] = defaultdict(list)
+    for row in annotations:
+        grouped[(row["case_id"], row["requested_use"])].append(row)
+    rows, incomplete = [], []
+    for unit, reviews in sorted(grouped.items()):
+        if len(reviews) < min_reviewers:
+            incomplete.append(unit)
+            continue
+        if len({r["admission_label"] for r in reviews}) > 1 or len({r["mapping_label"] for r in reviews}) > 1:
+            first = reviews[0]
+            rows.append({"adjudication_id": f"adj:{unit[0]}:{unit[1]}", "case_id": unit[0],
+                         "record_sha256": first["record_sha256"], "profile_sha256": first["profile_sha256"],
+                         "requested_use": unit[1],
+                         "annotation_ids_json": json.dumps(sorted(r["annotation_id"] for r in reviews)),
+                         **{c: "" for c in ADJUDICATION_COLUMNS[6:]}})
+    return rows, incomplete
+
+
+def resolve(annotations: list[dict[str, str]], adjudications: list[dict[str, str]], *,
+            min_reviewers: int = 2) -> dict[Unit, dict[str, str]]:
+    """Final reference labels: unanimous reviews, or the adjudicated label where reviewers disagreed."""
+    grouped: dict[Unit, list[dict[str, str]]] = defaultdict(list)
+    for row in annotations:
+        grouped[(row["case_id"], row["requested_use"])].append(row)
+    decided = {(a["case_id"], a["requested_use"]): a for a in adjudications}
+    final, problems = {}, []
+    for unit, reviews in sorted(grouped.items()):
+        first = reviews[0]
+        base = {"record_sha256": first["record_sha256"], "profile_sha256": first["profile_sha256"],
+                "split": first["split"], "group_id": first["group_id"], "reviews": str(len(reviews))}
+        if unit in decided:
+            final[unit] = {**base, "admission_label": decided[unit]["admission_label"],
+                           "mapping_label": decided[unit]["mapping_label"], "resolution": "adjudicated"}
+        elif len(reviews) < min_reviewers:
+            problems.append(f"{unit}: {len(reviews)} review(s), {min_reviewers} required")
+        elif len({r["admission_label"] for r in reviews}) == 1 and len({r["mapping_label"] for r in reviews}) == 1:
+            final[unit] = {**base, "admission_label": first["admission_label"],
+                           "mapping_label": first["mapping_label"], "resolution": "unanimous"}
+        else:
+            problems.append(f"{unit}: reviewers disagree and there is no adjudication")
+    unknown = sorted(set(decided) - set(grouped))
+    problems += [f"{unit}: adjudicated but never reviewed" for unit in unknown]
+    if problems:
+        raise ValueError("Unresolved reference labels:\n" + "\n".join(problems[:20]))
+    return final
+
+
+def load_predictions(path: Path) -> list[dict[str, str]]:
+    rows = _read_csv(path, PREDICTION_COLUMNS, ["subset"])
+    seen = set()
+    for number, row in enumerate(rows, start=2):
+        key = (row["case_id"], row["requested_use"], row["method"])
+        if key in seen:
+            raise ValueError(f"{path}:{number}: duplicate prediction for {key}")
+        seen.add(key)
+        if row["predicted_status"] not in PREDICTED:
+            raise ValueError(f"{path}:{number}: predicted_status must be one of {PREDICTED}")
+    return rows
+
+
+def score(final: dict[Unit, dict[str, str]], predictions: list[dict[str, str]], *, split: str = "test") -> dict[str, Any]:
+    """Compare predictions with reference labels on one split; hashes must match the reviewed records."""
+    selected = {u: v for u, v in final.items() if v["split"] == split}
+    if not selected:
+        raise ValueError(f"No reference labels in split {split!r}")
+    rows: dict[tuple[str, str], list[tuple[str, str]]] = defaultdict(list)  # (method, subset) -> (gold, predicted)
+    for p in predictions:
+        unit = (p["case_id"], p["requested_use"])
+        if unit not in final:
+            continue
+        gold = final[unit]
+        if (p["record_sha256"], p["profile_sha256"]) != (gold["record_sha256"], gold["profile_sha256"]):
+            raise ValueError(f"{unit}: prediction was made on a different record or profile than was reviewed")
+        if unit in selected:
+            for subset in {"all", p.get("subset") or "all"}:
+                rows[(p["method"], subset)].append((gold["admission_label"], p["predicted_status"]))
+    predicted = {(p["method"], p["case_id"], p["requested_use"]) for p in predictions}
+    missing = [(m, u) for m in sorted({p["method"] for p in predictions}) for u in sorted(selected)
+               if (m, *u) not in predicted]
+    if missing:
+        raise ValueError(f"Missing predictions for {len(missing)} reviewed unit(s), e.g. {missing[0]}")
+    report: dict[str, Any] = {"split": split, "reference_units": len(selected), "methods": {}}
+    for (method, subset), pairs in sorted(rows.items()):
+        positives = [p for g, p in pairs if g == "admitted"]
+        negatives = [p for g, p in pairs if g != "admitted"]
+        false_admissions = sum(p == "admitted" for p in negatives)
+        false_blocks = sum(p != "admitted" for p in positives)
+        report["methods"].setdefault(method, {})[subset] = {
+            "n": len(pairs), "reference_admitted": len(positives), "reference_not_admitted": len(negatives),
+            "false_admissions": false_admissions, "false_admission_rate_wilson_95": wilson(false_admissions, len(negatives)),
+            "false_blocks": false_blocks, "false_block_rate_wilson_95": wilson(false_blocks, len(positives)),
+            "admission_agreement": sum((g == "admitted") == (p == "admitted") for g, p in pairs),
+            "exact_matches": sum(g == p for g, p in pairs),
+            "confusion": {f"{g}|{p}": n for (g, p), n in sorted(Counter(pairs).items())},
+        }
+    return report
+
+
+def sha256_file(path: Path) -> str:
+    return hashlib.sha256(Path(path).read_bytes()).hexdigest()
+
+
+def freeze(manifest: dict[str, Any], final: dict[Unit, dict[str, str]], annotations: list[dict[str, str]],
+           files: list[Path], *, dataset_id: str, version: str, frozen_at: str) -> dict[str, Any]:
+    """Fill a manifest for a completed reference set. Hashes identify files; they do not prove review happened."""
+    _iso(frozen_at, "frozen_at")
+    reviewers = {row["reviewer_id"] for row in annotations}
+    single = all(v["reviews"] == "1" for v in final.values())
+    return {**manifest, "dataset_id": dataset_id, "version": version, "status": "frozen",
+            "reference_type": "single-reviewer reference set" if single else "independently reviewed reference set",
+            "reviewed_case_use_count": len(final), "independent_human_reviewer_count": len(reviewers),
+            "resolution_counts": dict(sorted(Counter(v["resolution"] for v in final.values()).items())),
+            "split_groups": sorted({v["group_id"] for v in final.values()}),
+            "files": {Path(f).name: sha256_file(f) for f in files}, "frozen_at": frozen_at,
+            "gold_standard_metrics_available": True}
+
+
+def _fmt(value: Any) -> str:
+    return "–" if value is None else str(value)
+
+
+def render_agreement(report: dict[str, Any]) -> str:
+    lines = [f"Reviewers: {', '.join(report['reviewers'])}", "",
+             "| Use | Label | Units (2+ reviews) | % agreement | Krippendorff α (95% bootstrap) | Cohen κ |",
+             "|---|---|---:|---:|---|---:|"]
+    for use, fields in report["uses"].items():
+        for field, e in fields.items():
+            ci = e["alpha_95_bootstrap"]
+            alpha = _fmt(e["krippendorff_alpha"]) + (f" ({ci[0]}–{ci[1]})" if ci else "")
+            lines.append(f"| {use} | {field} | {e['units_with_2plus_reviews']} | {_fmt(e['percent_agreement'])} | "
+                         f"{alpha} | {_fmt(e.get('cohen_kappa'))} |")
+    return "\n".join(lines) + "\n"
+
+
+def render_score(report: dict[str, Any]) -> str:
+    lines = [f"Split: {report['split']} · reference units: {report['reference_units']}", "",
+             "| Method | Subset | n | False admissions | False blocks | Admission agreement | Exact |",
+             "|---|---|---:|---:|---:|---:|---:|"]
+    for method, subsets in report["methods"].items():
+        for subset, m in subsets.items():
+            lines.append(f"| {method} | {subset} | {m['n']} | {m['false_admissions']}/{m['reference_not_admitted']} | "
+                         f"{m['false_blocks']}/{m['reference_admitted']} | {m['admission_agreement']}/{m['n']} | "
+                         f"{m['exact_matches']}/{m['n']} |")
+    return "\n".join(lines) + "\n"
+
+
+def write_csv(path: Path, rows: list[dict[str, str]], columns: list[str]) -> None:
+    with Path(path).open("w", encoding="utf-8", newline="") as handle:
+        writer = csv.DictWriter(handle, fieldnames=columns, lineterminator="\n")
+        writer.writeheader()
+        writer.writerows(rows)

bioevidence_validator/schema/bioevidence_core.yaml

@@ -10,6 +10,8 @@ prefixes:
   linkml: https://w3id.org/linkml/
   prov: http://www.w3.org/ns/prov#
   sepio: http://purl.obolibrary.org/obo/SEPIO_
+  ECO: http://purl.obolibrary.org/obo/ECO_
+  biolink: https://w3id.org/biolink/vocab/
 default_prefix: bioev
 default_range: string
 imports:
@@ -33,12 +35,24 @@ enums:
       supports: null
       contradicts: null
       neutral: null
+  # How an evidence item was produced. Meanings are the closest ECO terms (release
+  # 2026-07-10); see docs/STANDARDS.md for Biolink agent types and caveats. Kept as a
+  # comment so the compiled JSON Schema, and every report's schema_sha256, are unchanged.
   ExtractionMethod:
     permissible_values:
-      deterministic_parser: null
-      manual_curation: null
-      normalized_string_match: null
-      llm_extraction: null
+      deterministic_parser:
+        description: Rule-based import of an existing source record.
+        meaning: ECO:0000313
+      manual_curation:
+        description: A person read the source and recorded the evidence.
+        meaning: ECO:0000352
+      normalized_string_match:
+        description: Matching normalized strings, e.g. names to identifiers.
+        meaning: ECO:0008021
+      llm_extraction:
+        description: A language model extracted the evidence from text. ECO has no
+          LLM-specific term; the machine-learning term is the closest.
+        meaning: ECO:0008004
   StatementStatus:
     permissible_values:
       proposed: null

bioai_evidence_validator-0.5.0.dist-info/RECORD

@@ -1,14 +0,0 @@
-bioevidence_validator/__init__.py,sha256=O0lprvrrJN270yZxpMRi7jF6CTawvgc-f36QbAm0HgU,312
-bioevidence_validator/cli.py,sha256=50z5Aj6y6L3L9bwI_KZLMMYvtNHpWfwWY2O_XXGNVNI,5442
-bioevidence_validator/config.py,sha256=Ahc9eRPNR_ArqnbnJkdZsPFq9yERTYied0oGwUkANYI,1110
-bioevidence_validator/draft.py,sha256=8m5Yh68advpWrDb3m6ivkWa0Cq2SEM8CCbevBqXoRec,13552
-bioevidence_validator/engine.py,sha256=fm1gIOySsLiBbI3SklbEStiMD93RrhR3lA0vqdLJJXQ,15003
-bioevidence_validator/profiles/dataset-label.yaml,sha256=4utz9YcFFF2UnaLjIb0ATb-27hF8Smw7zmv0sBTaQdM,843
-bioevidence_validator/profiles/general.yaml,sha256=i9fJCWovJZ2e2d4rbn0Bx7mrhumNb2zjS9TOarkBWZw,778
-bioevidence_validator/profiles/literature-claim.yaml,sha256=7l82gX3oGGyYFq9B29-1qGXeL4LRcMhJi8Y9fieVE0c,612
-bioevidence_validator/schema/bioevidence_core.yaml,sha256=18MMngua0jlu4VnAbpb7SfI3numnCdjO986bZP6-Xn4,6700
-bioai_evidence_validator-0.5.0.dist-info/METADATA,sha256=1gG5J1j0CxRTbQJDYGBYz7vhMMzqqy7K_uqLE6cQ9ds,15360
-bioai_evidence_validator-0.5.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
-bioai_evidence_validator-0.5.0.dist-info/entry_points.txt,sha256=6oufzT2x1nIhTqM5B6y4aWubExKAoFb_wXXTdz3mBQQ,63
-bioai_evidence_validator-0.5.0.dist-info/licenses/LICENSE,sha256=z8d0m5b2O9McPEK1xHG_dWgUBT6EfBDz6wA0F7xSPTA,11358
-bioai_evidence_validator-0.5.0.dist-info/RECORD,,