bioai-evidence-validator 0.4.1__py3-none-any.whl → 0.7.0__py3-none-any.whl

{bioai_evidence_validator-0.4.1.dist-info → bioai_evidence_validator-0.7.0.dist-info}/METADATA

@@ -1,9 +1,9 @@
 Metadata-Version: 2.5
 Name: bioai-evidence-validator
-Version: 0.4.1
+Version: 0.7.0
 Summary: Standards-aligned evidence policy validation for AI-assisted biological curation
 Project-URL: Homepage, https://github.com/NingyuSUN/bioai-evidence-validator
-Project-URL: Documentation, https://github.com/NingyuSUN/bioai-evidence-validator/tree/main/docs
+Project-URL: Documentation, https://ningyusun.github.io/bioai-evidence-validator/
 Project-URL: Changelog, https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/CHANGELOG.md
 Project-URL: Issues, https://github.com/NingyuSUN/bioai-evidence-validator/issues
 Author: Ningyu Sun
@@ -25,7 +25,13 @@ Requires-Dist: jsonschema<5,>=4.23
 Requires-Dist: linkml<2,>=1.8
 Requires-Dist: pyyaml<7,>=6.0
 Provides-Extra: dev
+Requires-Dist: mypy<3,>=2.3; extra == 'dev'
+Requires-Dist: openpyxl<4,>=3.1; extra == 'dev'
+Requires-Dist: pytest-cov<8,>=7; extra == 'dev'
 Requires-Dist: pytest<9,>=8; extra == 'dev'
+Requires-Dist: ruff<0.17,>=0.16; extra == 'dev'
+Requires-Dist: types-jsonschema; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
 Description-Content-Type: text/markdown
 
 # BioAI Evidence Validator
@@ -34,6 +40,8 @@ Description-Content-Type: text/markdown
 [![PyPI](https://img.shields.io/pypi/v/bioai-evidence-validator)](https://pypi.org/project/bioai-evidence-validator/)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/pyproject.toml)
 [![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/LICENSE)
+[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NingyuSUN/bioai-evidence-validator/blob/main/examples/quickstart.ipynb)
+[![Docs](https://img.shields.io/badge/docs-site-blue)](https://ningyusun.github.io/bioai-evidence-validator/)
 
 **Stop AI-extracted biological claims from entering your knowledge base or
 training set before their evidence is good enough for that use.**
@@ -49,6 +57,10 @@ each requested use.
 pip install bioai-evidence-validator
 ```
 
+Or try it in the browser, nothing to install:
+[quickstart notebook on Colab](https://colab.research.google.com/github/NingyuSUN/bioai-evidence-validator/blob/main/examples/quickstart.ipynb).
+Full documentation: **[ningyusun.github.io/bioai-evidence-validator](https://ningyusun.github.io/bioai-evidence-validator/)**.
+
 ## 30-second example
 
 The two records below are identical except for one field: how the supporting
@@ -99,19 +111,54 @@ record is trustworthy enough for a particular purpose.
 | Provenance consistency (source hashes, resolved references, scope) | — | ✅ |
 | Human adjudications bound to a specific statement and use | — | ✅ |
 | Machine-readable audit report with hashes of input, schema and profile | — | ✅ |
+| Mapped to ECO, Biolink and GA4GH VA-Spec ([standards alignment](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/STANDARDS.md)) | — | ✅ |
 
-On the real-data benchmark below, schema-only checks admitted **160/160**
-injected faults; the full validator admitted **0/160**.
+On both real-data benchmarks below, schema-only checks admitted **160/160**
+injected faults; the full validator admitted **0/160**. On ClinVar, 1★ and 2★ variants
+the validator holds back were **about 4–5× more likely** to be reclassified or put in
+conflict three years later.
 
 ## Use it
 
+### Write a draft, not a full record
+
+A full record spells out identifiers, evidence lines and hashes. A draft states each fact
+once; `bioevidence build` derives the rest and hashes local source files:
+
+```yaml
+profile: literature-claim
+uses: [research_summary]
+statement:
+  subject: {id: "SYN:GENE_A", label: Synthetic gene A, type: gene}
+  predicate: associated_with
+  object: {id: "SYN:PHENOTYPE_A", label: Synthetic phenotype A, type: phenotype}
+  scope: ["taxon:synthetic"]
+sources:
+  - {id: paper, title: Synthetic paper, type: publication, version: v1,
+     retrieved_at: "2026-09-21T00:00:00Z", file: synthetic_paper.txt}
+evidence:
+  - {source: paper, locator: Table 2, type: publication_result,
+     method: llm_extraction, scope: ["taxon:synthetic"]}
+```
+
+```bash
+bioevidence build examples/drafts/llm_claim.yaml --output record.json
+bioevidence validate record.json --profile literature-claim    # exit 2: review required
+```
+
+Building never fills in scope, extraction method, retrieval time or review decisions for
+you. For LLM pipelines, `bioevidence draft-schema --profile literature-claim` prints a JSON
+Schema for structured output. See the [draft format](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/DRAFTS.md).
+
 ### Command line
 
 ```bash
 bioevidence profiles                                   # list built-in profiles and their use contracts
+bioevidence build draft.yaml --output record.json       # expand a compact draft
 bioevidence validate record.json --profile literature-claim
 bioevidence validate record.json --profile my_profile.yaml --output report.json
-bioevidence generate-schema --output record.schema.json  # JSON Schema for the input format
+bioevidence draft-schema --profile literature-claim     # JSON Schema for drafts (e.g. LLM output)
+bioevidence generate-schema --output record.schema.json  # JSON Schema for full records
 ```
 
 Exit codes: **0** admitted, **1** rejected, **2** review required, **3** input or configuration error.
@@ -119,12 +166,9 @@ Exit codes: **0** admitted, **1** rejected, **2** review required, **3** input o
 ### Python
 
 ```python
-import json
-from pathlib import Path
-
-from bioevidence_validator.engine import validate_record
+from bioevidence_validator import build_record, load_draft, validate_record
 
-record = json.loads(Path("record.json").read_text(encoding="utf-8"))
+record = build_record(load_draft("draft.yaml"), base_dir=".")   # or load a full record JSON
 report = validate_record(record, profile="literature-claim")
 
 for decision in report["use_decisions"]:
@@ -133,6 +177,30 @@ for decision in report["use_decisions"]:
 
 `profile` accepts a built-in name or a path to your own YAML profile.
 
+### Check records in CI
+
+Validate every record or draft in a pull request, with a summary table and inline annotations:
+
+```yaml
+# .github/workflows/evidence.yml
+on: pull_request
+jobs:
+  evidence:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: NingyuSUN/bioai-evidence-validator@v0.7.0
+        with:
+          files: records/**/*.yaml        # whitespace-separated globs
+          format: draft                   # or: record (default)
+          profile: literature-claim       # built-in name or path to your profile YAML
+          fail-on: review                 # or: rejected
+```
+
+With `fail-on: review` (default) the job fails unless every file is admitted; with
+`fail-on: rejected` it fails only on rejected files or files that cannot be read. The
+action's outputs `admitted`, `review_required`, `rejected` and `error` hold the counts.
+
 ## How it works
 
 ```mermaid
@@ -193,7 +261,36 @@ Use the [annotation templates](https://github.com/NingyuSUN/bioai-evidence-valid
 source version and intended use. Document reviewer roles and whether labels are
 single-reviewed or independently reviewed by multiple people.
 
-## Real-data case
+## Real-data cases
+
+### ClinVar germline classifications, three years later
+
+The [ClinVar case](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/examples/clinvar_germline/README.md)
+turns every 2023-09 lab submission into evidence, validates 5,026 sampled variants with a
+ClinVar-style profile, and checks what happened to them by 2026-09.
+
+![ClinVar benchmark: share of 2023 pathogenic classifications reclassified or conflicting by 2026, with and without a dissenting submission, and false admissions under controlled faults and the trust boundary](https://raw.githubusercontent.com/NingyuSUN/bioai-evidence-validator/main/docs/assets/clinvar_germline_benchmark.svg)
+
+- **Policy reproduction:** the profile matches NCBI's own 2023-09 review status on 98–99% of
+  decisions; every disagreement is listed with its cause.
+- **Where the validator is stricter, classifications were less stable.** It sends any P/LP
+  variant with a dissenting submission to review, even when ClinVar's aggregate does not.
+  Across all 218,920 germline P/LP variants, the share later reclassified or put in conflict:
+
+| 2023-09 ClinVar review status | No dissenting submission | With one (validator: review) |
+|---|---:|---:|
+| 1★ single submitter | 2.63% (2.54–2.71) | **13.77%** (11.46–16.47) |
+| 2★ multiple submitters | 2.19% (2.05–2.33) | **8.32%** (6.51–10.59) |
+| 3★ expert panel | 0.07% (0.03–0.16) | **1.30%** (0.63–2.66) |
+
+Wilson 95% intervals. Stability is not correctness, and this is observational; see the
+case's interpretation limits. Not for clinical use.
+
+```bash
+uv run python examples/clinvar_germline/run.py --output artifacts/clinvar
+```
+
+### VBO canine name mapping
 
 [VBO canine name mapping](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/examples/vbo_canine/README.md) uses a frozen public ontology:
 72 real-name cases, 160 controlled errors, and 16 separately reported trust-boundary
@@ -204,7 +301,7 @@ per-required-evidence-type validation. Source-derived labels are not expert anno
 uv run python examples/vbo_canine/run.py --output artifacts/vbo-canine
 ```
 
-### Benchmark results (v0.4.1)
+#### Benchmark results (v0.4.1)
 
 ![VBO canine benchmark comparing false admissions across three validation methods](https://raw.githubusercontent.com/NingyuSUN/bioai-evidence-validator/main/docs/assets/vbo_canine_benchmark.svg)
 
@@ -214,20 +311,31 @@ On 72 real-source name mappings, the full validator admitted all 48 unambiguous
 
 ## Scope
 
-The VBO case uses attributed public data; other fixtures are synthetic.
+The VBO and ClinVar cases use attributed public data; other fixtures are synthetic.
 Admission means **the supplied record meets the selected
 profile**, not that a biological claim is true. The toolkit does not retrieve papers,
-verify reviewer identities, train models, or measure prediction accuracy. The generic core compares supplied hashes; the VBO importer also hashes its local source
-projection. External source truth and cohort independence require upstream verification.
+verify reviewer identities, train models, or measure prediction accuracy. The generic core compares supplied hashes; the VBO and ClinVar importers also hash their local source
+projections. External source truth and cohort independence require upstream verification.
+Neither benchmark has independent expert annotation yet; a blinded [expert-review kit](https://github.com/NingyuSUN/bioai-evidence-validator/tree/main/evaluation/clinvar_review) for the ClinVar case is ready for reviewers.
 
 ## Versions and branches
 
-`main` is the domain-neutral framework (0.4.1). The complete canine implementation
-and SQLite adapter from 0.3 live on the
-[`canine-breed` branch](https://github.com/NingyuSUN/bioai-evidence-validator/tree/canine-breed);
+`main` is the domain-neutral framework (0.7.0). The complete canine implementation
+and SQLite adapter from 0.3 are preserved at the
+[`canine-0.3` tag](https://github.com/NingyuSUN/bioai-evidence-validator/tree/canine-0.3)
+(also the `canine-breed` branch);
 see the [0.4 migration guide](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/MIGRATION-0.4.md) and
 [changelog](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/CHANGELOG.md).
 
+## Contributing
+
+Bug reports, domain profiles and new benchmarks are welcome. See
+[CONTRIBUTING.md](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/CONTRIBUTING.md),
+the [community profiles](https://github.com/NingyuSUN/bioai-evidence-validator/tree/main/community/profiles)
+and issues labelled [`good first issue`](https://github.com/NingyuSUN/bioai-evidence-validator/labels/good%20first%20issue).
+Report security problems privately as described in
+[SECURITY.md](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/SECURITY.md).
+
 ## Citing
 
 If you use this toolkit in research, please cite it using the metadata in
@@ -235,6 +343,8 @@ If you use this toolkit in research, please cite it using the metadata in
 (GitHub's "Cite this repository" button generates APA and BibTeX).
 
 [Create a profile](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/PROFILES.md) ·
+[Draft format](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/DRAFTS.md) ·
+[Standards alignment](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/STANDARDS.md) ·
 [Engineering contract](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/ENGINEERING.md) ·
 [Design case study](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/CASE_STUDY.md) ·
 [Architecture decision](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/ADR-002-domain-neutral-main.md) ·

bioai_evidence_validator-0.7.0.dist-info/RECORD

@@ -0,0 +1,15 @@
+bioevidence_validator/__init__.py,sha256=PsEzz-ifHjfLOJX7KkUqMFNbniIQMNPKRFxgNdoF3JI,312
+bioevidence_validator/cli.py,sha256=7TxCuCN0jcbFwUnim0QiIARJC3XPQcsYN5qBcT7EFF0,9965
+bioevidence_validator/config.py,sha256=Ahc9eRPNR_ArqnbnJkdZsPFq9yERTYied0oGwUkANYI,1110
+bioevidence_validator/draft.py,sha256=2ge9l1BA44e-_ukO_Nnz9UZAV_RS8yqj3Ujr86eFxtY,13631
+bioevidence_validator/engine.py,sha256=OSH9Zd-UqCVqYRU6eLkyoavRWAJGzz8Q6mcCNkhW48I,14989
+bioevidence_validator/review.py,sha256=yp9DI2TafTiZqZmNTejtUb9oHeeXPEjZHXx3fe2qCGI,20293
+bioevidence_validator/profiles/dataset-label.yaml,sha256=4utz9YcFFF2UnaLjIb0ATb-27hF8Smw7zmv0sBTaQdM,843
+bioevidence_validator/profiles/general.yaml,sha256=i9fJCWovJZ2e2d4rbn0Bx7mrhumNb2zjS9TOarkBWZw,778
+bioevidence_validator/profiles/literature-claim.yaml,sha256=7l82gX3oGGyYFq9B29-1qGXeL4LRcMhJi8Y9fieVE0c,612
+bioevidence_validator/schema/bioevidence_core.yaml,sha256=c2YIBOhuT6v7j4MDsMScImdyLZCMy-3X--ExCB_IYu8,7514
+bioai_evidence_validator-0.7.0.dist-info/METADATA,sha256=mt7dqba6hjGQjYBUoBRTS_09bvyD7GU3x1c_RmK4amg,18818
+bioai_evidence_validator-0.7.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+bioai_evidence_validator-0.7.0.dist-info/entry_points.txt,sha256=6oufzT2x1nIhTqM5B6y4aWubExKAoFb_wXXTdz3mBQQ,63
+bioai_evidence_validator-0.7.0.dist-info/licenses/LICENSE,sha256=z8d0m5b2O9McPEK1xHG_dWgUBT6EfBDz6wA0F7xSPTA,11358
+bioai_evidence_validator-0.7.0.dist-info/RECORD,,

bioevidence_validator/__init__.py

@@ -1,3 +1,8 @@
 """Evidence validation primitives and domain policy runners."""
 
-__version__ = "0.4.1"
+__version__ = "0.7.0"
+
+from .draft import build_record, draft_json_schema, load_draft  # noqa: E402
+from .engine import validate_record  # noqa: E402
+
+__all__ = ["__version__", "build_record", "draft_json_schema", "load_draft", "validate_record"]

bioevidence_validator/cli.py

@@ -9,7 +9,9 @@ from pathlib import Path
 
 import yaml
 
-from .engine import default_schema_path, generate_json_schema, validate_record, profile_path, list_profiles
+from . import review as review_module
+from .draft import build_record, draft_json_schema, load_draft
+from .engine import default_schema_path, generate_json_schema, list_profiles, profile_path, validate_record
 
 
 def parser() -> argparse.ArgumentParser:
@@ -20,10 +22,46 @@ def parser() -> argparse.ArgumentParser:
     validate.add_argument("--output", type=Path)
     validate.add_argument("--schema", type=Path, default=default_schema_path())
     validate.add_argument("--profile", default="general", help="Built-in profile name or YAML file path")
+    build = commands.add_parser("build", help="Expand a compact YAML/JSON draft into a full record")
+    build.add_argument("draft", type=Path)
+    build.add_argument("--output", type=Path)
+    draft_schema = commands.add_parser("draft-schema", help="JSON Schema for drafts under one profile")
+    draft_schema.add_argument("--profile", default="general", help="Built-in profile name or YAML file path")
+    draft_schema.add_argument("--output", type=Path)
     generate = commands.add_parser("generate-schema", help="Generate JSON Schema from LinkML")
     generate.add_argument("--output", type=Path, required=True)
     generate.add_argument("--schema", type=Path, default=default_schema_path())
     commands.add_parser("profiles", help="List built-in profiles and their use contracts")
+
+    review = commands.add_parser("review", help="Independent human review: agreement, adjudication, scoring")
+    steps = review.add_subparsers(dest="step", required=True)
+    check = steps.add_parser("check", help="Check annotations (and adjudications) against the protocol format")
+    check.add_argument("annotations", type=Path)
+    check.add_argument("--adjudications", type=Path)
+    agree = steps.add_parser("agreement", help="Inter-reviewer agreement per use, before adjudication")
+    agree.add_argument("annotations", type=Path)
+    agree.add_argument("--output", type=Path)
+    sheet = steps.add_parser("adjudication-sheet", help="Blank adjudication rows for every disagreement")
+    sheet.add_argument("annotations", type=Path)
+    sheet.add_argument("--output", type=Path, required=True)
+    sheet.add_argument("--min-reviewers", type=int, default=2)
+    scoring = steps.add_parser("score", help="Compare predictions with the resolved reference labels")
+    scoring.add_argument("--annotations", type=Path, required=True)
+    scoring.add_argument("--adjudications", type=Path, required=True)
+    scoring.add_argument("--predictions", type=Path, required=True)
+    scoring.add_argument("--split", default="test", choices=sorted(review_module.SPLITS))
+    scoring.add_argument("--min-reviewers", type=int, default=2)
+    scoring.add_argument("--output", type=Path)
+    frozen = steps.add_parser("freeze", help="Write a frozen manifest for a completed reference set")
+    frozen.add_argument("--manifest", type=Path, required=True, help="Draft manifest to complete")
+    frozen.add_argument("--annotations", type=Path, required=True)
+    frozen.add_argument("--adjudications", type=Path, required=True)
+    frozen.add_argument("--file", type=Path, action="append", default=[], help="Extra file to hash (repeatable)")
+    frozen.add_argument("--dataset-id", required=True)
+    frozen.add_argument("--version", required=True)
+    frozen.add_argument("--frozen-at", required=True, help="ISO 8601 time of the freeze")
+    frozen.add_argument("--min-reviewers", type=int, default=2)
+    frozen.add_argument("--output", type=Path, required=True)
     return result
 
 
@@ -81,6 +119,24 @@ def _run(args) -> int:
         print(json.dumps(list_profiles(), indent=2))
         return 0
 
+    if args.command == "review":
+        return _review(args)
+
+    if args.command in {"build", "draft-schema"}:
+        if args.command == "build":
+            if args.output and args.output.resolve() == args.draft.resolve():
+                raise ValueError("Output must not overwrite the draft")
+            draft = load_draft(args.draft)
+            _check_depth(draft)
+            result = build_record(draft, base_dir=args.draft.parent)
+        else:
+            result = draft_json_schema(args.profile)
+        if args.output:
+            _write_json(args.output, result)
+        else:
+            print(json.dumps(result, indent=2))
+        return 0
+
     selected_profile = profile_path(args.profile)
     protected = [args.input, args.schema, selected_profile, default_schema_path()]
     if args.output and args.output.resolve() in {p.resolve() for p in protected}:
@@ -101,6 +157,48 @@ def _run(args) -> int:
     return 0
 
 
+def _review(args) -> int:
+    rv = review_module
+    annotations = rv.load_annotations(args.annotations)
+    if args.step == "check":
+        adjudications = rv.load_adjudications(args.adjudications, annotations) if args.adjudications else []
+        units = {(a["case_id"], a["requested_use"]) for a in annotations}
+        print(f"OK: {len(annotations)} annotations, {len(units)} case-use units, "
+              f"{len({a['reviewer_id'] for a in annotations})} reviewers, {len(adjudications)} adjudications")
+        return 0
+    if args.step == "agreement":
+        report = rv.agreement(annotations)
+        if args.output:
+            _write_json(args.output, report)
+        print(rv.render_agreement(report), end="")
+        return 0
+    if args.step == "adjudication-sheet":
+        if args.output.exists():
+            raise ValueError(f"{args.output} exists; refusing to overwrite adjudication work")
+        rows, incomplete = rv.adjudication_sheet(annotations, min_reviewers=args.min_reviewers)
+        rv.write_csv(args.output, rows, rv.ADJUDICATION_COLUMNS)
+        print(f"{len(rows)} disagreement(s) to adjudicate; {len(incomplete)} unit(s) with too few reviews")
+        return 0
+    adjudications = rv.load_adjudications(args.adjudications, annotations)
+    final = rv.resolve(annotations, adjudications, min_reviewers=args.min_reviewers)
+    if args.step == "score":
+        report = rv.score(final, rv.load_predictions(args.predictions), split=args.split)
+        if args.output:
+            _write_json(args.output, report)
+        print(rv.render_score(report), end="")
+        return 0
+    manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
+    files = [args.annotations, args.adjudications, *args.file]
+    if args.output.resolve() in {f.resolve() for f in [args.manifest, *files]}:
+        raise ValueError("Output must not overwrite an input")
+    frozen = rv.freeze(manifest, final, annotations, files, dataset_id=args.dataset_id,
+                       version=args.version, frozen_at=args.frozen_at)
+    _write_json(args.output, frozen)
+    print(f"Frozen {frozen['reviewed_case_use_count']} case-use labels from "
+          f"{frozen['independent_human_reviewer_count']} reviewers ({frozen['reference_type']})")
+    return 0
+
+
 def main(argv: list[str] | None = None) -> int:
     args = parser().parse_args(argv)
     try:

bioevidence_validator/draft.py

@@ -0,0 +1,253 @@
+"""Compact drafts: a short, strict authoring format that expands to a full evidence record.
+
+A draft names each fact once. Building it derives identifiers, groups evidence items into
+evidence lines by direction, and hashes local source files. It never fills in facts the
+caller did not state: scopes, extraction methods, retrieval times and reviewer decisions
+must be written explicitly, and the resulting record still goes through full validation.
+"""
+from __future__ import annotations
+
+import datetime as dt
+import hashlib
+import json
+from pathlib import Path
+from typing import Any
+
+from .config import load_mapping, nonblank
+from .engine import load_profile, profile_path
+
+DRAFT_KEYS = {"required": {"profile", "uses", "statement", "sources", "evidence"},
+              "optional": {"id", "status", "reviews"}}
+STATEMENT_KEYS = {"required": {"subject", "predicate", "object", "scope"}, "optional": set()}
+ENTITY_KEYS = {"required": {"id", "label", "type"}, "optional": set()}
+SOURCE_KEYS = {"required": {"id", "title", "type", "version", "retrieved_at"},
+               "optional": {"uri", "sha256", "file"}}
+EVIDENCE_KEYS = {"required": {"source", "locator", "type", "method", "scope"},
+                 "optional": {"text", "direction"}}
+REVIEW_KEYS = {"required": {"reviewer", "decision", "uses", "rationale", "decided_at"}, "optional": set()}
+REVIEWER_KEYS = {"required": {"id", "type"}, "optional": {"name"}}
+
+STATUSES = ["proposed", "accepted", "rejected", "superseded"]
+SOURCE_TYPES = ["ontology_snapshot", "registry_snapshot", "dataset_snapshot", "publication", "web_page", "local_file"]
+METHODS = ["deterministic_parser", "manual_curation", "normalized_string_match", "llm_extraction"]
+DIRECTIONS = ["supports", "contradicts", "neutral"]
+DECISIONS = ["accept", "reject", "defer"]
+AGENT_TYPES = ["human", "software", "organization"]
+
+
+def _fields(value: Any, keys: dict, path: str) -> dict:
+    if not isinstance(value, dict):
+        raise ValueError(f"{path} must be a mapping")
+    missing = keys["required"] - value.keys()
+    unknown = value.keys() - keys["required"] - keys["optional"]
+    if missing:
+        raise ValueError(f"{path} is missing: {', '.join(sorted(missing))}")
+    if unknown:
+        raise ValueError(f"{path} has unknown fields: {', '.join(sorted(map(str, unknown)))}")
+    return value
+
+
+def _text(value: Any, path: str, choices: list[str] | None = None) -> str:
+    if isinstance(value, (dt.date, dt.datetime)):
+        value = value.isoformat()  # YAML loads unquoted timestamps as datetime objects.
+    if not nonblank(value):
+        raise ValueError(f"{path} must be a nonblank string (quote numbers and versions in YAML)")
+    if choices is not None and value not in choices:
+        raise ValueError(f"{path} must be one of: {', '.join(choices)}")
+    return value
+
+
+def _texts(value: Any, path: str) -> list[str]:
+    if not isinstance(value, list) or not value:
+        raise ValueError(f"{path} must be a nonempty list")
+    items = [_text(item, f"{path}[{index}]") for index, item in enumerate(value)]
+    if len(set(items)) != len(items):
+        raise ValueError(f"{path} must not repeat values")
+    return items
+
+
+def _rows(value: Any, path: str, required: bool = True) -> list:
+    if not isinstance(value, list) or (required and not value):
+        raise ValueError(f"{path} must be a {'nonempty ' if required else ''}list")
+    return value
+
+
+def _entity(value: Any, path: str) -> dict:
+    entity = _fields(value, ENTITY_KEYS, path)
+    return {"id": _text(entity["id"], path + ".id"), "label": _text(entity["label"], path + ".label"),
+            "entity_type": _text(entity["type"], path + ".type")}
+
+
+def _source(value: Any, path: str, base_dir: Path) -> tuple[str, dict]:
+    source = _fields(value, SOURCE_KEYS, path)
+    artifact = {"title": _text(source["title"], path + ".title"),
+                "source_type": _text(source["type"], path + ".type", SOURCE_TYPES)}
+    if "uri" in source:
+        artifact["uri"] = _text(source["uri"], path + ".uri")
+    artifact.update({"version": _text(source["version"], path + ".version"),
+                     "retrieved_at": _text(source["retrieved_at"], path + ".retrieved_at")})
+    observed = None
+    if "file" in source:
+        file = base_dir / _text(source["file"], path + ".file")
+        if not file.is_file():
+            raise ValueError(f"{path}.file does not exist: {file}")
+        observed = hashlib.sha256(file.read_bytes()).hexdigest()
+    # A stated hash is the frozen reference; a file hash is what was observed now.
+    # With both, validation reports a mismatch (BEV002) instead of trusting either.
+    stated = _text(source["sha256"], path + ".sha256").lower() if "sha256" in source else None
+    frozen = stated or observed
+    if frozen is None:
+        raise ValueError(f"{path} needs sha256, file, or both")
+    artifact["sha256"] = frozen
+    if stated is not None and observed is not None:
+        artifact["observed_sha256"] = observed
+    return _text(source["id"], path + ".id"), artifact
+
+
+def load_draft(path: Path) -> dict:
+    """Read a YAML or JSON draft with duplicate-key rejection."""
+    return load_mapping(Path(path).read_bytes(), "Draft")
+
+
+def build_record(draft: dict, *, base_dir: Path | str = ".") -> dict[str, Any]:
+    """Expand a compact draft into a full record. Raises ValueError on any malformed field."""
+    draft = _fields(draft, DRAFT_KEYS, "draft")
+    base_dir = Path(base_dir)
+    profile = _text(draft["profile"], "draft.profile")
+    uses = _texts(draft["uses"], "draft.uses")
+    statement = _fields(draft["statement"], STATEMENT_KEYS, "draft.statement")
+    scope = _texts(statement["scope"], "draft.statement.scope")
+
+    record_id = _text(draft["id"], "draft.id") if "id" in draft else None
+    if record_id is None:
+        canonical = json.dumps(draft, sort_keys=True, separators=(",", ":"), default=str).encode()
+        record_id = "bioev:draft-" + hashlib.sha256(canonical).hexdigest()[:16]
+
+    sources, local_ids = [], {}
+    for index, row in enumerate(_rows(draft["sources"], "draft.sources")):
+        name, artifact = _source(row, f"draft.sources[{index}]", base_dir)
+        if name in local_ids:
+            raise ValueError(f"draft.sources[{index}].id repeats {name!r}")
+        local_ids[name] = f"{record_id}/source/{name}"
+        sources.append({"id": local_ids[name], **artifact})
+
+    items: list[dict[str, Any]] = []
+    lines: dict[str, list[str]] = {}
+    for index, row in enumerate(_rows(draft["evidence"], "draft.evidence")):
+        path = f"draft.evidence[{index}]"
+        evidence = _fields(row, EVIDENCE_KEYS, path)
+        source = _text(evidence["source"], path + ".source")
+        if source not in local_ids:
+            raise ValueError(f"{path}.source {source!r} is not a declared source id")
+        item: dict[str, Any] = {"id": f"{record_id}/evidence/{index + 1}", "source_artifact_id": local_ids[source],
+                "locator": _text(evidence["locator"], path + ".locator")}
+        if "text" in evidence:
+            item["extracted_text"] = _text(evidence["text"], path + ".text")
+        item.update({"evidence_type": _text(evidence["type"], path + ".type"),
+                     "extraction_method": _text(evidence["method"], path + ".method", METHODS),
+                     "scope": _texts(evidence["scope"], path + ".scope")})
+        items.append(item)
+        direction = _text(evidence.get("direction", "supports"), path + ".direction", DIRECTIONS)
+        lines.setdefault(direction, []).append(item["id"])
+
+    statement_id = f"{record_id}/statement"
+    adjudications = []
+    for index, row in enumerate(_rows(draft.get("reviews", []), "draft.reviews", required=False)):
+        path = f"draft.reviews[{index}]"
+        review = _fields(row, REVIEW_KEYS, path)
+        reviewer = _fields(review["reviewer"], REVIEWER_KEYS, path + ".reviewer")
+        agent = {"id": _text(reviewer["id"], path + ".reviewer.id")}
+        if "name" in reviewer:
+            agent["name"] = _text(reviewer["name"], path + ".reviewer.name")
+        agent["agent_type"] = _text(reviewer["type"], path + ".reviewer.type", AGENT_TYPES)
+        adjudications.append({"id": f"{record_id}/review/{index + 1}", "statement_id": statement_id,
+                              "applies_to_uses": _texts(review["uses"], path + ".uses"),
+                              "decision": _text(review["decision"], path + ".decision", DECISIONS),
+                              "reviewer": agent, "rationale": _text(review["rationale"], path + ".rationale"),
+                              "decided_at": _text(review["decided_at"], path + ".decided_at")})
+
+    return {
+        "record_id": record_id,
+        "profile_id": profile,
+        "statement": {
+            "id": statement_id,
+            "subject": _entity(statement["subject"], "draft.statement.subject"),
+            "predicate": _text(statement["predicate"], "draft.statement.predicate"),
+            "object": _entity(statement["object"], "draft.statement.object"),
+            "scope": scope,
+            "evidence_lines": [{"id": f"{record_id}/line/{direction}", "direction": direction,
+                                "evidence_item_ids": ids} for direction, ids in lines.items()],
+            "statement_status": _text(draft.get("status", "proposed"), "draft.status", STATUSES),
+        },
+        "source_artifacts": sources,
+        "evidence_items": items,
+        "adjudications": adjudications,
+        "requested_uses": uses,
+    }
+
+
+def _object(keys: dict, properties: dict, description: str | None = None) -> dict:
+    if properties.keys() != keys["required"] | keys["optional"]:
+        raise RuntimeError("Draft schema and parser fields diverged")
+    schema = {"type": "object", "additionalProperties": False,
+              "required": sorted(keys["required"]), "properties": properties}
+    return {"description": description, **schema} if description else schema
+
+
+def _choice(values: list[str], description: str | None = None) -> dict:
+    if not values:
+        return {"type": "string", "minLength": 1, **({"description": description} if description else {})}
+    return {"type": "string", "enum": list(values), **({"description": description} if description else {})}
+
+
+def draft_json_schema(profile: str | Path = "general") -> dict[str, Any]:
+    """JSON Schema for drafts under one profile, e.g. for LLM structured output.
+
+    Profile allowlists become enums. The schema guides authoring only; build_record and
+    validation remain authoritative.
+    """
+    contract = load_profile(profile_path(profile).read_bytes())
+    evidence_types = sorted({kind for use in contract["uses"].values() for kind in use["required_evidence_types"]})
+    text = {"type": "string", "minLength": 1}
+    tokens = {"type": "array", "minItems": 1, "uniqueItems": True, "items": text}
+
+    def entity(types):
+        return _object(ENTITY_KEYS, {"id": {**text, "description": "Stable identifier, e.g. a CURIE."},
+                                     "label": text, "type": _choice(types)})
+
+    return {
+        "$schema": "https://json-schema.org/draft/2020-12/schema",
+        "title": f"BioAI evidence draft ({contract['id']} profile {contract['version']})",
+        **_object(DRAFT_KEYS, {
+            "id": {**text, "description": "Optional record id; derived from the draft content when omitted."},
+            "profile": {"const": contract["id"]},
+            "uses": {"type": "array", "minItems": 1, "uniqueItems": True,
+                     "items": _choice(sorted(contract["uses"]))},
+            "status": _choice(STATUSES, "Defaults to proposed."),
+            "statement": _object(STATEMENT_KEYS, {
+                "subject": entity(contract["subject_types"]),
+                "predicate": _choice(contract["predicates"]),
+                "object": entity(contract["object_types"]),
+                "scope": {**tokens, "description": "Explicit scope tokens, e.g. taxon:9606."}}),
+            "sources": {"type": "array", "minItems": 1, "items": _object(SOURCE_KEYS, {
+                "id": {**text, "description": "Local name referenced by evidence[].source."},
+                "title": text, "type": _choice(SOURCE_TYPES), "uri": text, "version": text,
+                "retrieved_at": {"type": "string", "format": "date-time"},
+                "sha256": {"type": "string", "pattern": "^[0-9a-fA-F]{64}$",
+                           "description": "Frozen source hash."},
+                "file": {**text, "description": "Local file hashed at build time, relative to the draft."}},
+                "Provide sha256, file, or both.")},
+            "evidence": {"type": "array", "minItems": 1, "items": _object(EVIDENCE_KEYS, {
+                "source": text, "locator": {**text, "description": "Where in the source, e.g. 'Table 2'."},
+                "text": text, "type": _choice(evidence_types),
+                "method": _choice(METHODS, "Set by the pipeline that produced this item, not by a model."),
+                "scope": {**tokens, "description": "Scope the source actually supports."},
+                "direction": _choice(DIRECTIONS, "Defaults to supports.")})},
+            "reviews": {"type": "array", "items": _object(REVIEW_KEYS, {
+                "reviewer": _object(REVIEWER_KEYS, {"id": text, "name": text, "type": _choice(AGENT_TYPES)}),
+                "decision": _choice(DECISIONS),
+                "uses": {"type": "array", "minItems": 1, "uniqueItems": True,
+                         "items": _choice(sorted(contract["uses"]))},
+                "rationale": text, "decided_at": {"type": "string", "format": "date-time"}})},
+        }),
+    }

bioevidence_validator/engine.py

@@ -3,7 +3,7 @@ from __future__ import annotations
 import hashlib
 import json
 from dataclasses import asdict, dataclass
-from datetime import datetime, timezone
+from datetime import UTC, datetime
 from pathlib import Path
 from typing import Any
 
@@ -255,7 +255,7 @@ class RecordValidator:
             "profile_sha256": self.profile_sha256,
             "schema_version": self.schema_version, "schema_sha256": self.schema_sha256,
             "schema_sources": [dict(source) for source in self.schema_sources],
-            "input_sha256": sha256_bytes(canonical), "validated_at": datetime.now(timezone.utc).isoformat(),
+            "input_sha256": sha256_bytes(canonical), "validated_at": datetime.now(UTC).isoformat(),
             "schema_valid": not any(f.rule_id == "SCHEMA" for f in findings),
             "overall_status": overall, "findings": [asdict(f) for f in findings], "use_decisions": decisions,
         }

bioevidence_validator/review.py

@@ -0,0 +1,388 @@
+"""Independent human review: check annotations, measure agreement, adjudicate, score and freeze.
+
+Implements the workflow in docs/GOLD_STANDARD.md for the CSV formats in evaluation/gold_standard/.
+Reference labels live here, never in the engine: they evaluate decisions, they do not make them.
+"""
+from __future__ import annotations
+
+import csv
+import datetime as dt
+import hashlib
+import json
+import math
+import random
+import re
+from collections import Counter, defaultdict
+from collections.abc import Sequence
+from pathlib import Path
+from typing import Any
+
+ANNOTATION_COLUMNS = [
+    "annotation_id", "case_id", "record_sha256", "profile_id", "profile_sha256", "requested_use", "group_id",
+    "split", "reviewer_id", "reviewer_qualification", "annotated_at", "mapping_label", "mapping_rationale",
+    "admission_label", "admission_rationale", "evidence_refs_json",
+]
+OPTIONAL_ANNOTATION_COLUMNS = ["later_information_seen", "minutes_spent"]
+ADJUDICATION_COLUMNS = [
+    "adjudication_id", "case_id", "record_sha256", "profile_sha256", "requested_use", "annotation_ids_json",
+    "mapping_label", "admission_label", "adjudicator_id", "adjudicated_at", "mapping_rationale",
+    "admission_rationale", "evidence_refs_json",
+]
+PREDICTION_COLUMNS = ["case_id", "requested_use", "record_sha256", "profile_sha256", "method", "predicted_status"]
+MAPPING_LABELS = ["correct", "incorrect", "uncertain"]
+ADMISSION_LABELS = ["admitted", "review_required", "rejected"]
+PREDICTED = ADMISSION_LABELS + ["not_admitted"]  # binary baselines predict admitted / not_admitted
+SPLITS = {"development", "test"}
+HEX64 = re.compile(r"^[0-9a-f]{64}$")
+
+Unit = tuple[str, str]  # (case_id, requested_use)
+
+
+def _read_csv(path: Path, required: list[str], optional: Sequence[str] = ()) -> list[dict[str, str]]:
+    with Path(path).open(encoding="utf-8-sig", newline="") as handle:  # tolerate Excel's BOM
+        reader = csv.DictReader(handle)
+        header = reader.fieldnames or []
+        missing = [c for c in required if c not in header]
+        unknown = [c for c in header if c not in required and c not in optional]
+        if missing or unknown:
+            raise ValueError(f"{path}: missing columns {missing}, unknown columns {unknown}")
+        return [dict(row) for row in reader]
+
+
+def _iso(value: str, where: str) -> None:
+    try:
+        dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
+    except ValueError:
+        raise ValueError(f"{where}: not an ISO 8601 time: {value!r}") from None
+
+
+def _json_list(value: str, where: str) -> list:
+    try:
+        parsed = json.loads(value)
+    except json.JSONDecodeError:
+        raise ValueError(f"{where}: not valid JSON: {value!r}") from None
+    if not isinstance(parsed, list) or not all(isinstance(x, str) and x.strip() for x in parsed):
+        raise ValueError(f"{where}: must be a JSON array of nonblank strings")
+    return parsed
+
+
+def load_annotations(path: Path) -> list[dict[str, str]]:
+    """Read and strictly check an annotations CSV; raise ValueError listing the first problems."""
+    rows = _read_csv(path, ANNOTATION_COLUMNS, OPTIONAL_ANNOTATION_COLUMNS)
+    errors: list[str] = []
+    seen_ids: set[str] = set()
+    seen_reviews: set[tuple[str, str, str]] = set()
+    bindings: dict[Unit, tuple[str, ...]] = {}
+    for number, row in enumerate(rows, start=2):  # line 1 is the header
+        where = f"{path}:{number}"
+        blank = [c for c in ANNOTATION_COLUMNS
+                 if c not in ("mapping_rationale", "admission_rationale") and not (row[c] or "").strip()]
+        if blank:
+            errors.append(f"{where}: blank {blank}")
+            continue
+        if row["annotation_id"] in seen_ids:
+            errors.append(f"{where}: duplicate annotation_id {row['annotation_id']!r}")
+        seen_ids.add(row["annotation_id"])
+        review = (row["case_id"], row["requested_use"], row["reviewer_id"])
+        if review in seen_reviews:
+            errors.append(f"{where}: reviewer {row['reviewer_id']!r} labelled {review[:2]} twice")
+        seen_reviews.add(review)
+        if row["mapping_label"] not in MAPPING_LABELS:
+            errors.append(f"{where}: mapping_label must be one of {MAPPING_LABELS}")
+        if row["admission_label"] not in ADMISSION_LABELS:
+            errors.append(f"{where}: admission_label must be one of {ADMISSION_LABELS}")
+        if row["split"] not in SPLITS:
+            errors.append(f"{where}: split must be one of {sorted(SPLITS)}")
+        for column in ("record_sha256", "profile_sha256"):
+            if not HEX64.match(row[column]):
+                errors.append(f"{where}: {column} must be 64 lowercase hex characters")
+        try:
+            _iso(row["annotated_at"], where)
+            _json_list(row["evidence_refs_json"], where)
+        except ValueError as exc:
+            errors.append(str(exc))
+        binding = (row["record_sha256"], row["profile_sha256"], row["profile_id"], row["group_id"], row["split"])
+        if bindings.setdefault(review[:2], binding) != binding:
+            errors.append(f"{where}: reviewers of {review[:2]} saw a different record, profile, group or split")
+    if errors:
+        raise ValueError("Invalid annotations:\n" + "\n".join(errors[:20]))
+    return rows
+
+
+def load_adjudications(path: Path, annotations: list[dict[str, str]]) -> list[dict[str, str]]:
+    rows = _read_csv(path, ADJUDICATION_COLUMNS)
+    by_id = {a["annotation_id"]: a for a in annotations}
+    errors, units = [], set()
+    for number, row in enumerate(rows, start=2):
+        where = f"{path}:{number}"
+        blank = [c for c in ADJUDICATION_COLUMNS if not (row[c] or "").strip()]
+        if blank:
+            errors.append(f"{where}: blank {blank}")
+            continue
+        unit = (row["case_id"], row["requested_use"])
+        if unit in units:
+            errors.append(f"{where}: {unit} adjudicated twice")
+        units.add(unit)
+        if row["mapping_label"] not in MAPPING_LABELS or row["admission_label"] not in ADMISSION_LABELS:
+            errors.append(f"{where}: labels must be from {MAPPING_LABELS} and {ADMISSION_LABELS}")
+        try:
+            _iso(row["adjudicated_at"], where)
+            _json_list(row["evidence_refs_json"], where)
+            linked = _json_list(row["annotation_ids_json"], where)
+        except ValueError as exc:
+            errors.append(str(exc))
+            continue
+        for annotation_id in linked:
+            original = by_id.get(annotation_id)
+            if original is None or (original["case_id"], original["requested_use"]) != unit:
+                errors.append(f"{where}: {annotation_id!r} is not an annotation of {unit}")
+            elif (original["record_sha256"], original["profile_sha256"]) != (row["record_sha256"], row["profile_sha256"]):
+                errors.append(f"{where}: record or profile hash differs from annotation {annotation_id!r}")
+    if errors:
+        raise ValueError("Invalid adjudications:\n" + "\n".join(errors[:20]))
+    return rows
+
+
+def _units(annotations: list[dict[str, str]], field: str, use: str | None = None) -> dict[Unit, dict[str, str]]:
+    units: dict[Unit, dict[str, str]] = defaultdict(dict)
+    for row in annotations:
+        if use is None or row["requested_use"] == use:
+            units[(row["case_id"], row["requested_use"])][row["reviewer_id"]] = row[field]
+    return units
+
+
+def krippendorff_alpha_nominal(units: list[list[str]]) -> float | None:
+    """Krippendorff's alpha for nominal data; units with fewer than two values are not pairable.
+
+    Returns None when there is no variation to measure agreement against (expected disagreement is 0).
+    """
+    coincidence: dict[tuple[str, str], float] = defaultdict(float)
+    for values in units:
+        if len(values) < 2:
+            continue
+        for i, c in enumerate(values):
+            for j, k in enumerate(values):
+                if i != j:
+                    coincidence[(c, k)] += 1 / (len(values) - 1)
+    totals: dict[str, float] = defaultdict(float)
+    for (c, _), weight in coincidence.items():
+        totals[c] += weight
+    n = sum(totals.values())
+    observed = sum(w for (c, k), w in coincidence.items() if c != k)
+    expected = (n * n - sum(t * t for t in totals.values())) / (n - 1) if n > 1 else 0.0
+    return None if expected == 0 else 1 - observed / expected
+
+
+def cohen_kappa(pairs: list[tuple[str, str]]) -> float | None:
+    if not pairs:
+        return None
+    n = len(pairs)
+    observed = sum(a == b for a, b in pairs) / n
+    first, second = Counter(a for a, _ in pairs), Counter(b for _, b in pairs)
+    expected = sum(first[c] * second[c] for c in set(first) | set(second)) / (n * n)
+    return None if expected == 1 else (observed - expected) / (1 - expected)
+
+
+def wilson(events: int, n: int, z: float = 1.959963984540054) -> list[float] | None:
+    if not n:
+        return None
+    p = events / n
+    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
+    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
+    return [round(max(0.0, centre - half), 6), round(min(1.0, centre + half), 6)]
+
+
+def _bootstrap_alpha(units: list[list[str]], resamples: int, seed: int) -> list[float] | None:
+    rng = random.Random(seed)
+    values = []
+    for _ in range(resamples):
+        alpha = krippendorff_alpha_nominal([units[rng.randrange(len(units))] for _ in units])
+        if alpha is not None:
+            values.append(alpha)
+    if len(values) < resamples * 0.9:
+        return None  # too many resamples without variation for a meaningful interval
+    values.sort()
+    last = len(values) - 1
+    return [round(values[int(0.025 * last)], 4), round(values[math.ceil(0.975 * last)], 4)]
+
+
+def agreement(annotations: list[dict[str, str]], *, resamples: int = 2000, seed: int = 20260925) -> dict[str, Any]:
+    """Inter-reviewer agreement per requested use, before adjudication."""
+    reviewers = sorted({row["reviewer_id"] for row in annotations})
+    result: dict[str, Any] = {"reviewers": reviewers, "uses": {}}
+    for use in sorted({row["requested_use"] for row in annotations}):
+        per_field = {}
+        for field in ("admission_label", "mapping_label"):
+            units = _units(annotations, field, use)
+            multi = [list(v.values()) for v in units.values() if len(v) >= 2]
+            alpha = krippendorff_alpha_nominal(multi)
+            entry: dict[str, Any] = {
+                "units": len(units), "units_with_2plus_reviews": len(multi),
+                "percent_agreement": round(sum(len(set(v)) == 1 for v in multi) / len(multi), 4) if multi else None,
+                "krippendorff_alpha": None if alpha is None else round(alpha, 4),
+                "alpha_95_bootstrap": _bootstrap_alpha(multi, resamples, seed) if multi else None,
+            }
+            if len(reviewers) == 2:
+                a, b = reviewers
+                pairs = [(v[a], v[b]) for v in units.values() if a in v and b in v]
+                kappa = cohen_kappa(pairs)
+                entry["cohen_kappa"] = None if kappa is None else round(kappa, 4)
+                entry["confusion"] = {f"{x}|{y}": n for (x, y), n in sorted(Counter(pairs).items())}
+            per_field[field] = entry
+        result["uses"][use] = per_field
+    return result
+
+
+def adjudication_sheet(annotations: list[dict[str, str]], *, min_reviewers: int = 2) -> tuple[list[dict], list[Unit]]:
+    """Blank adjudication rows for every disagreement, plus units that still lack enough reviews."""
+    grouped: dict[Unit, list[dict[str, str]]] = defaultdict(list)
+    for row in annotations:
+        grouped[(row["case_id"], row["requested_use"])].append(row)
+    rows, incomplete = [], []
+    for unit, reviews in sorted(grouped.items()):
+        if len(reviews) < min_reviewers:
+            incomplete.append(unit)
+            continue
+        if len({r["admission_label"] for r in reviews}) > 1 or len({r["mapping_label"] for r in reviews}) > 1:
+            first = reviews[0]
+            rows.append({"adjudication_id": f"adj:{unit[0]}:{unit[1]}", "case_id": unit[0],
+                         "record_sha256": first["record_sha256"], "profile_sha256": first["profile_sha256"],
+                         "requested_use": unit[1],
+                         "annotation_ids_json": json.dumps(sorted(r["annotation_id"] for r in reviews)),
+                         **{c: "" for c in ADJUDICATION_COLUMNS[6:]}})
+    return rows, incomplete
+
+
+def resolve(annotations: list[dict[str, str]], adjudications: list[dict[str, str]], *,
+            min_reviewers: int = 2) -> dict[Unit, dict[str, str]]:
+    """Final reference labels: unanimous reviews, or the adjudicated label where reviewers disagreed."""
+    grouped: dict[Unit, list[dict[str, str]]] = defaultdict(list)
+    for row in annotations:
+        grouped[(row["case_id"], row["requested_use"])].append(row)
+    decided = {(a["case_id"], a["requested_use"]): a for a in adjudications}
+    final, problems = {}, []
+    for unit, reviews in sorted(grouped.items()):
+        first = reviews[0]
+        base = {"record_sha256": first["record_sha256"], "profile_sha256": first["profile_sha256"],
+                "split": first["split"], "group_id": first["group_id"], "reviews": str(len(reviews))}
+        if unit in decided:
+            final[unit] = {**base, "admission_label": decided[unit]["admission_label"],
+                           "mapping_label": decided[unit]["mapping_label"], "resolution": "adjudicated"}
+        elif len(reviews) < min_reviewers:
+            problems.append(f"{unit}: {len(reviews)} review(s), {min_reviewers} required")
+        elif len({r["admission_label"] for r in reviews}) == 1 and len({r["mapping_label"] for r in reviews}) == 1:
+            final[unit] = {**base, "admission_label": first["admission_label"],
+                           "mapping_label": first["mapping_label"], "resolution": "unanimous"}
+        else:
+            problems.append(f"{unit}: reviewers disagree and there is no adjudication")
+    unknown = sorted(set(decided) - set(grouped))
+    problems += [f"{unit}: adjudicated but never reviewed" for unit in unknown]
+    if problems:
+        raise ValueError("Unresolved reference labels:\n" + "\n".join(problems[:20]))
+    return final
+
+
+def load_predictions(path: Path) -> list[dict[str, str]]:
+    rows = _read_csv(path, PREDICTION_COLUMNS, ["subset"])
+    seen = set()
+    for number, row in enumerate(rows, start=2):
+        key = (row["case_id"], row["requested_use"], row["method"])
+        if key in seen:
+            raise ValueError(f"{path}:{number}: duplicate prediction for {key}")
+        seen.add(key)
+        if row["predicted_status"] not in PREDICTED:
+            raise ValueError(f"{path}:{number}: predicted_status must be one of {PREDICTED}")
+    return rows
+
+
+def score(final: dict[Unit, dict[str, str]], predictions: list[dict[str, str]], *, split: str = "test") -> dict[str, Any]:
+    """Compare predictions with reference labels on one split; hashes must match the reviewed records."""
+    selected = {u: v for u, v in final.items() if v["split"] == split}
+    if not selected:
+        raise ValueError(f"No reference labels in split {split!r}")
+    rows: dict[tuple[str, str], list[tuple[str, str]]] = defaultdict(list)  # (method, subset) -> (gold, predicted)
+    for p in predictions:
+        unit = (p["case_id"], p["requested_use"])
+        if unit not in final:
+            continue
+        gold = final[unit]
+        if (p["record_sha256"], p["profile_sha256"]) != (gold["record_sha256"], gold["profile_sha256"]):
+            raise ValueError(f"{unit}: prediction was made on a different record or profile than was reviewed")
+        if unit in selected:
+            for subset in {"all", p.get("subset") or "all"}:
+                rows[(p["method"], subset)].append((gold["admission_label"], p["predicted_status"]))
+    predicted = {(p["method"], p["case_id"], p["requested_use"]) for p in predictions}
+    missing = [(m, u) for m in sorted({p["method"] for p in predictions}) for u in sorted(selected)
+               if (m, *u) not in predicted]
+    if missing:
+        raise ValueError(f"Missing predictions for {len(missing)} reviewed unit(s), e.g. {missing[0]}")
+    report: dict[str, Any] = {"split": split, "reference_units": len(selected), "methods": {}}
+    for (method, subset), pairs in sorted(rows.items()):
+        positives = [p for g, p in pairs if g == "admitted"]
+        negatives = [p for g, p in pairs if g != "admitted"]
+        false_admissions = sum(p == "admitted" for p in negatives)
+        false_blocks = sum(p != "admitted" for p in positives)
+        report["methods"].setdefault(method, {})[subset] = {
+            "n": len(pairs), "reference_admitted": len(positives), "reference_not_admitted": len(negatives),
+            "false_admissions": false_admissions, "false_admission_rate_wilson_95": wilson(false_admissions, len(negatives)),
+            "false_blocks": false_blocks, "false_block_rate_wilson_95": wilson(false_blocks, len(positives)),
+            "admission_agreement": sum((g == "admitted") == (p == "admitted") for g, p in pairs),
+            "exact_matches": sum(g == p for g, p in pairs),
+            "confusion": {f"{g}|{p}": n for (g, p), n in sorted(Counter(pairs).items())},
+        }
+    return report
+
+
+def sha256_file(path: Path) -> str:
+    return hashlib.sha256(Path(path).read_bytes()).hexdigest()
+
+
+def freeze(manifest: dict[str, Any], final: dict[Unit, dict[str, str]], annotations: list[dict[str, str]],
+           files: list[Path], *, dataset_id: str, version: str, frozen_at: str) -> dict[str, Any]:
+    """Fill a manifest for a completed reference set. Hashes identify files; they do not prove review happened."""
+    _iso(frozen_at, "frozen_at")
+    reviewers = {row["reviewer_id"] for row in annotations}
+    single = all(v["reviews"] == "1" for v in final.values())
+    return {**manifest, "dataset_id": dataset_id, "version": version, "status": "frozen",
+            "reference_type": "single-reviewer reference set" if single else "independently reviewed reference set",
+            "reviewed_case_use_count": len(final), "independent_human_reviewer_count": len(reviewers),
+            "resolution_counts": dict(sorted(Counter(v["resolution"] for v in final.values()).items())),
+            "split_groups": sorted({v["group_id"] for v in final.values()}),
+            "files": {Path(f).name: sha256_file(f) for f in files}, "frozen_at": frozen_at,
+            "gold_standard_metrics_available": True}
+
+
+def _fmt(value: Any) -> str:
+    return "–" if value is None else str(value)
+
+
+def render_agreement(report: dict[str, Any]) -> str:
+    lines = [f"Reviewers: {', '.join(report['reviewers'])}", "",
+             "| Use | Label | Units (2+ reviews) | % agreement | Krippendorff α (95% bootstrap) | Cohen κ |",
+             "|---|---|---:|---:|---|---:|"]
+    for use, fields in report["uses"].items():
+        for field, e in fields.items():
+            ci = e["alpha_95_bootstrap"]
+            alpha = _fmt(e["krippendorff_alpha"]) + (f" ({ci[0]}–{ci[1]})" if ci else "")
+            lines.append(f"| {use} | {field} | {e['units_with_2plus_reviews']} | {_fmt(e['percent_agreement'])} | "
+                         f"{alpha} | {_fmt(e.get('cohen_kappa'))} |")
+    return "\n".join(lines) + "\n"
+
+
+def render_score(report: dict[str, Any]) -> str:
+    lines = [f"Split: {report['split']} · reference units: {report['reference_units']}", "",
+             "| Method | Subset | n | False admissions | False blocks | Admission agreement | Exact |",
+             "|---|---|---:|---:|---:|---:|---:|"]
+    for method, subsets in report["methods"].items():
+        for subset, m in subsets.items():
+            lines.append(f"| {method} | {subset} | {m['n']} | {m['false_admissions']}/{m['reference_not_admitted']} | "
+                         f"{m['false_blocks']}/{m['reference_admitted']} | {m['admission_agreement']}/{m['n']} | "
+                         f"{m['exact_matches']}/{m['n']} |")
+    return "\n".join(lines) + "\n"
+
+
+def write_csv(path: Path, rows: list[dict[str, str]], columns: list[str]) -> None:
+    with Path(path).open("w", encoding="utf-8", newline="") as handle:
+        writer = csv.DictWriter(handle, fieldnames=columns, lineterminator="\n")
+        writer.writeheader()
+        writer.writerows(rows)

bioevidence_validator/schema/bioevidence_core.yaml

@@ -10,6 +10,8 @@ prefixes:
   linkml: https://w3id.org/linkml/
   prov: http://www.w3.org/ns/prov#
   sepio: http://purl.obolibrary.org/obo/SEPIO_
+  ECO: http://purl.obolibrary.org/obo/ECO_
+  biolink: https://w3id.org/biolink/vocab/
 default_prefix: bioev
 default_range: string
 imports:
@@ -33,12 +35,24 @@ enums:
       supports: null
       contradicts: null
       neutral: null
+  # How an evidence item was produced. Meanings are the closest ECO terms (release
+  # 2026-07-10); see docs/STANDARDS.md for Biolink agent types and caveats. Kept as a
+  # comment so the compiled JSON Schema, and every report's schema_sha256, are unchanged.
   ExtractionMethod:
     permissible_values:
-      deterministic_parser: null
-      manual_curation: null
-      normalized_string_match: null
-      llm_extraction: null
+      deterministic_parser:
+        description: Rule-based import of an existing source record.
+        meaning: ECO:0000313
+      manual_curation:
+        description: A person read the source and recorded the evidence.
+        meaning: ECO:0000352
+      normalized_string_match:
+        description: Matching normalized strings, e.g. names to identifiers.
+        meaning: ECO:0008021
+      llm_extraction:
+        description: A language model extracted the evidence from text. ECO has no
+          LLM-specific term; the machine-learning term is the closest.
+        meaning: ECO:0008004
   StatementStatus:
     permissible_values:
       proposed: null

bioai_evidence_validator-0.4.1.dist-info/RECORD

@@ -1,13 +0,0 @@
-bioevidence_validator/__init__.py,sha256=w-OLV_MserELKh9U9FrLM8ZICJqNoWNOlqMwglL1lIk,87
-bioevidence_validator/cli.py,sha256=fknYldtjN50OvguGMcp_kmAky5PxZ0i5WRUF4HEQVkA,4326
-bioevidence_validator/config.py,sha256=Ahc9eRPNR_ArqnbnJkdZsPFq9yERTYied0oGwUkANYI,1110
-bioevidence_validator/engine.py,sha256=fm1gIOySsLiBbI3SklbEStiMD93RrhR3lA0vqdLJJXQ,15003
-bioevidence_validator/profiles/dataset-label.yaml,sha256=4utz9YcFFF2UnaLjIb0ATb-27hF8Smw7zmv0sBTaQdM,843
-bioevidence_validator/profiles/general.yaml,sha256=i9fJCWovJZ2e2d4rbn0Bx7mrhumNb2zjS9TOarkBWZw,778
-bioevidence_validator/profiles/literature-claim.yaml,sha256=7l82gX3oGGyYFq9B29-1qGXeL4LRcMhJi8Y9fieVE0c,612
-bioevidence_validator/schema/bioevidence_core.yaml,sha256=18MMngua0jlu4VnAbpb7SfI3numnCdjO986bZP6-Xn4,6700
-bioai_evidence_validator-0.4.1.dist-info/METADATA,sha256=JkN3mmJrD-cx86fBjdrjrTaxadrM1SdzbTrtHv81uO0,12563
-bioai_evidence_validator-0.4.1.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
-bioai_evidence_validator-0.4.1.dist-info/entry_points.txt,sha256=6oufzT2x1nIhTqM5B6y4aWubExKAoFb_wXXTdz3mBQQ,63
-bioai_evidence_validator-0.4.1.dist-info/licenses/LICENSE,sha256=z8d0m5b2O9McPEK1xHG_dWgUBT6EfBDz6wA0F7xSPTA,11358
-bioai_evidence_validator-0.4.1.dist-info/RECORD,,