bioai-evidence-validator 0.4.1__tar.gz → 0.7.0__tar.gz

{bioai_evidence_validator-0.4.1 → bioai_evidence_validator-0.7.0}/.gitattributes

@@ -1 +1,2 @@
 * text=auto eol=lf
+*.gz binary

bioai_evidence_validator-0.7.0/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,53 @@
+name: Bug report
+description: The validator, CLI or GitHub Action does something it should not.
+labels: [bug]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for reporting. For security problems (for example a record admitted that a
+        documented rule should reject), use [private reporting](https://github.com/NingyuSUN/bioai-evidence-validator/security/advisories/new) instead.
+        **Do not paste real patient data**; a synthetic record that shows the behavior is enough.
+  - type: input
+    id: version
+    attributes:
+      label: Version
+      description: Output of `pip show bioai-evidence-validator` or `bioevidence --help` header, or the commit.
+      placeholder: "0.7.0"
+    validations:
+      required: true
+  - type: textarea
+    id: command
+    attributes:
+      label: Command or code
+      description: The exact command or Python call, including `--profile`.
+      render: shell
+    validations:
+      required: true
+  - type: textarea
+    id: input
+    attributes:
+      label: Smallest record, draft or profile that reproduces it
+      render: yaml
+    validations:
+      required: true
+  - type: textarea
+    id: expected
+    attributes:
+      label: Expected result
+      description: Which status, reason code or exit code did you expect, and which rule says so?
+    validations:
+      required: true
+  - type: textarea
+    id: actual
+    attributes:
+      label: Actual result
+      description: The report's `overall_status`, `findings` and exit code, or the error message.
+      render: json
+    validations:
+      required: true
+  - type: input
+    id: environment
+    attributes:
+      label: Environment
+      placeholder: "Python 3.12, Ubuntu 24.04"

bioai_evidence_validator-0.7.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Security vulnerability
+    url: https://github.com/NingyuSUN/bioai-evidence-validator/security/advisories/new
+    about: Report privately; please do not open a public issue.
+  - name: Documentation
+    url: https://github.com/NingyuSUN/bioai-evidence-validator#readme
+    about: Usage, draft format, profiles and benchmarks.

bioai_evidence_validator-0.7.0/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,28 @@
+name: Feature request
+description: Propose a change to the engine, CLI, formats or integrations.
+labels: [enhancement]
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: Problem
+      description: What are you trying to do, and what gets in the way today?
+    validations:
+      required: true
+  - type: textarea
+    id: proposal
+    attributes:
+      label: Proposal
+      description: The behavior you would like. For new rules, describe a record that should pass and one that should not.
+    validations:
+      required: true
+  - type: textarea
+    id: failure
+    attributes:
+      label: Failure mode
+      description: If this goes wrong, does it admit too much or block too much? How would a user notice?
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives considered
+      description: For example, doing it in a profile or an importer instead of the engine.

bioai_evidence_validator-0.7.0/.github/ISSUE_TEMPLATE/profile_proposal.yml

@@ -0,0 +1,42 @@
+name: Domain profile proposal
+description: Propose an admission profile for a biological domain.
+labels: [profile]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Profiles decide which evidence each intended use requires; they do not change the engine.
+        See [docs/PROFILES.md](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/PROFILES.md)
+        and the [community profile template](https://github.com/NingyuSUN/bioai-evidence-validator/tree/main/community/profiles/_template).
+  - type: input
+    id: domain
+    attributes:
+      label: Domain and assertion
+      placeholder: "Protein–protein interactions: protein A interacts_with protein B"
+    validations:
+      required: true
+  - type: textarea
+    id: uses
+    attributes:
+      label: Intended uses and their evidence requirements
+      description: For each use (e.g. research summary, knowledge-base admission, training data), which evidence types are required, and is human acceptance required?
+    validations:
+      required: true
+  - type: textarea
+    id: basis
+    attributes:
+      label: Basis for the policy
+      description: Community guideline, database policy or published standard the requirements follow (with links), or state that it is your own proposal.
+    validations:
+      required: true
+  - type: textarea
+    id: cases
+    attributes:
+      label: Example cases
+      description: At least one claim that should be admitted and one that should not, with the reason.
+  - type: checkboxes
+    id: pr
+    attributes:
+      label: Contribution
+      options:
+        - label: I am willing to open a pull request with the profile and example cases.

bioai_evidence_validator-0.7.0/.github/pull_request_template.md

@@ -0,0 +1,18 @@
+## What and why
+
+<!-- One purpose per pull request. Link the issue: "Closes #123". -->
+
+## Admission boundary
+
+<!-- Does this change what gets admitted, sent to review or rejected? If yes, say which
+     records move and why, and point to the tests that pin the new boundary. -->
+
+- [ ] No change to admission decisions
+- [ ] Changes admission decisions (explained above; tests on both sides of the boundary)
+
+## Checklist
+
+- [ ] `uv run --frozen ruff check .`, `uv run --frozen mypy` and `uv run --frozen pytest --cov` pass
+- [ ] Committed benchmark results regenerated if report contents changed
+- [ ] `CHANGELOG.md` updated if users will notice
+- [ ] New third-party data has its license, attribution and exact version recorded

bioai_evidence_validator-0.7.0/.github/workflows/ci.yml

@@ -0,0 +1,90 @@
+name: Repository checks
+on:
+  push:
+    branches: [main]
+  pull_request:
+permissions:
+  contents: read
+jobs:
+  lint:
+    timeout-minutes: 10
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install locked tooling
+        run: python -m pip install uv==0.12.17
+      - name: Install locked project
+        run: uv sync --frozen --extra dev
+      - name: Lint
+        run: uv run --frozen ruff check --output-format github .
+      - name: Type-check
+        run: uv run --frozen mypy
+
+  verify:
+    timeout-minutes: 15
+    strategy:
+      matrix:
+        include:
+          - os: ubuntu-latest
+            python: '3.11'
+          - os: ubuntu-latest
+            python: '3.12'
+          - os: ubuntu-latest
+            python: '3.13'
+          - os: windows-latest
+            python: '3.13'
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Install locked tooling
+        run: python -m pip install uv==0.12.17
+      - name: Install locked project
+        run: uv sync --frozen --extra dev --python ${{ matrix.python }}
+      - name: Run tests with coverage (minimum in pyproject.toml)
+        run: uv run --frozen pytest --cov --cov-report=term
+      - name: Coverage summary
+        if: always()
+        shell: bash
+        run: uv run --frozen coverage report --format=markdown >> "$GITHUB_STEP_SUMMARY" || true
+      - name: Build distributable
+        run: uv build
+      - name: Verify installed wheel and CLI outside editable source
+        shell: bash
+        run: uv run --isolated --no-project --python ${{ matrix.python }} --with ./dist/*.whl python tools/check_distribution.py
+
+  action:
+    timeout-minutes: 10
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Admitted and review-required drafts pass with fail-on rejected
+        id: drafts
+        uses: ./
+        with:
+          files: examples/drafts/*.yaml
+          profile: literature-claim
+          format: draft
+          fail-on: rejected
+      - name: A rejected record fails the action
+        id: rejected
+        continue-on-error: true
+        uses: ./
+        with:
+          files: examples/dataset_label/*.json
+          profile: dataset-label
+      - name: Check action outcomes and outputs
+        shell: bash
+        run: |
+          test "${{ steps.drafts.outputs.admitted }}" = 1
+          test "${{ steps.drafts.outputs.review_required }}" = 1
+          test "${{ steps.rejected.outcome }}" = failure
+          test "${{ steps.rejected.outputs.rejected }}" = 1

bioai_evidence_validator-0.7.0/.github/workflows/docs.yml

@@ -0,0 +1,43 @@
+name: Documentation
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+permissions:
+  contents: read
+concurrency:
+  group: docs-${{ github.ref }}
+  cancel-in-progress: true
+jobs:
+  build:
+    timeout-minutes: 10
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      # Pinned: MkDocs 2.0 drops the plugin and theme system this site uses.
+      - name: Install MkDocs
+        run: python -m pip install mkdocs==1.6.1 mkdocs-material==9.7.7
+      - name: Build with strict link checking
+        run: mkdocs build --strict --site-dir _site
+      - if: github.event_name != 'pull_request'
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: _site
+
+  deploy:
+    if: github.event_name != 'pull_request'
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - id: deployment
+        uses: actions/deploy-pages@v4

{bioai_evidence_validator-0.4.1 → bioai_evidence_validator-0.7.0}/.gitignore

@@ -7,3 +7,7 @@ dist/
 build/
 artifacts/
 validation_report.json
+_site/
+site/
+.coverage
+htmlcov/

bioai_evidence_validator-0.7.0/CHANGELOG.md

@@ -0,0 +1,89 @@
+# Changelog
+
+## 0.7.0 — Expert review, quality checks, community and documentation site
+
+- Add `bioevidence review` (`check`, `agreement`, `adjudication-sheet`, `score`, `freeze`)
+  implementing the gold-standard protocol: strict annotation checks, Krippendorff's α with
+  bootstrap interval and Cohen's κ (both cross-checked against reference implementations),
+  adjudication of disagreements, scoring on the test split with hash-bound joins, and frozen
+  manifests. It never produces labels.
+- Add a blinded ClinVar expert-review kit (`evaluation/clinvar_review/`): all 95 variants where
+  the validator and NCBI disagree plus 95 stratum-matched controls, an Excel workbook with
+  dropdowns, a reviewer rubric, a private key, and an importer into the protocol format.
+- CI runs ruff and mypy, and enforces a 95% test-coverage minimum (currently 98%).
+- Add CONTRIBUTING, CODE_OF_CONDUCT (Contributor Covenant 2.1), SECURITY, issue forms and a
+  pull request template.
+- Add `community/profiles/`: contributed domain profiles whose example cases are built,
+  validated and checked against expected outcomes in CI; `_template/` to copy.
+- Add a documentation site (MkDocs, strict link checking) published to GitHub Pages.
+- The canine 0.3 implementation is also preserved at the `canine-0.3` tag.
+- No change to validation decisions or report format; both benchmarks reproduce 0.6.0 exactly
+  apart from `validator_version`.
+
+## 0.6.0 — ClinVar case and standards alignment
+
+- Add a second real-data case: ClinVar germline classifications. 5,026 sampled variants
+  from the 2023-09 release are built from per-submission evidence and validated with a
+  ClinVar-style profile; decisions are compared with NCBI's own 2023-09 review status and
+  with each classification's 2026-09 outcome, plus controlled faults and a trust-boundary
+  cohort. Frozen, hash-pinned sample; rebuild script verifies the three upstream files.
+- Add `docs/STANDARDS.md`, mapping the record model to ECO, Biolink 4.4.4, GA4GH VA-Spec
+  1.0.1 and PROV-O, with mapping strength and caveats.
+- Annotate `ExtractionMethod` values with ECO meanings in the LinkML schema. The compiled
+  JSON Schema, and therefore every report's `schema_sha256`, is unchanged.
+- The VBO benchmark summary changes only `validator_version`.
+
+## 0.5.0 — Drafts, LLM draft schema and GitHub Action
+
+- Add compact YAML/JSON drafts: `bioevidence build` and `build_record()` derive identifiers,
+  group evidence lines and hash named local files, without supplying scope, method, times
+  or review decisions. Strict parsing rejects unknown, missing and out-of-choice fields.
+- Add `bioevidence draft-schema` and `draft_json_schema()`: a per-profile JSON Schema for
+  drafts, e.g. for LLM structured output.
+- Add a composite GitHub Action that validates records or drafts in pull requests, with a
+  job summary, file annotations and count outputs.
+- Add a Colab quickstart notebook and draft examples.
+- Export `build_record`, `load_draft`, `draft_json_schema` and `validate_record` from the package root.
+
+## 0.4.1 — Required-evidence quality and real-source evaluation
+
+- Apply extraction-method quality gates to each required evidence type independently.
+- Prevent unrelated manual/parser evidence from admitting an LLM-only required result.
+- Add an attributed, frozen VBO canine name-mapping case and source rebuild script.
+- Evaluate 72 real-source names, 160 controlled faults, and 16 explicit trust-boundary cases separately.
+- Preserve generic main and the legacy canine-breed branch.
+- Publish to PyPI from version tags; add package metadata and citation file.
+- Reject CLI input nested deeper than 100 levels (exit 3) on every platform and Python version.
+
+## 0.4.0 — Domain-neutral main
+
+- Preserve canine 0.3 functionality on the `canine-breed` branch.
+- Replace canine defaults with a generic entity/relation/evidence schema and strict YAML profiles.
+- Add general, literature-claim, dataset-label, and custom assay examples.
+- Bind human adjudications to statements and uses; enforce evidence scope and reference integrity.
+- Replace `--policy` with `--profile`; add profile discovery and generic audit fields.
+- Remove the canine SQLite command from main. See `docs/MIGRATION-0.4.md` for breaking changes.
+
+## 0.3.0 — Evidence and snapshot contract hardening
+
+- Require resolved supporting evidence, selected-candidate consistency and
+  compatible claim predicates; block relationship claims from sample/frequency label use.
+- Enforce scope exclusions from concept scope, use only supporting scope evidence,
+  reject blank provenance and unresolved references in unused evidence.
+- Reject incomplete/duplicate/unknown policy configuration; custom schemas cannot
+  relax the packaged structural baseline. Snapshot validation context per batch.
+- Reject duplicate database IDs, WAL/SHM/journal sidecars, malformed manifests
+  and unrecognized boolean evidence. Include resolution-row provenance,
+  limits/accounting and batch hashes; publish the completion summary last.
+- Preserve earlier policy files, synthetic examples and operational CLI exit codes.
+
+## 0.2.0 — Evidence admission contracts and reproducible CLI
+
+- Version schema/policy 0.2; preserve policy 0.1 and record generated-schema hashes.
+- Prevent withdrawn statements or conflicting human judgments from silently being admitted.
+- Reject missing, empty, malformed, duplicated or unknown requested uses.
+- Make structural and identity errors block the entire record, including unknown uses.
+- Require evidence collections and reject duplicate IDs before dictionary lookup.
+- Return structured operational CLI errors; parse UTF-8 strictly and write reports atomically.
+- Add regression tests, a Linux/Windows CI workflow and an installed-wheel smoke check.
+- Include the Apache-2.0 license already declared in project metadata.

{bioai_evidence_validator-0.4.1 → bioai_evidence_validator-0.7.0}/CITATION.cff

@@ -9,7 +9,7 @@ abstract: >-
 authors:
   - family-names: Sun
     given-names: Ningyu
-version: 0.4.1
+version: 0.7.0
 license: Apache-2.0
 repository-code: "https://github.com/NingyuSUN/bioai-evidence-validator"
 keywords:

bioai_evidence_validator-0.7.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,16 @@
+# Code of conduct
+
+This project adopts the
+[Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/)
+as its code of conduct. It applies to all project spaces (issues, pull requests,
+discussions and reviews) and to anyone representing the project elsewhere.
+
+In short: be respectful and constructive, assume good faith, critique work rather than
+people, and remember that contributors bring expertise from many fields, from biocuration
+to software engineering.
+
+## Reporting
+
+Report unacceptable behavior to the maintainer, Ningyu Sun, at <woshiwosunny@gmail.com>.
+Reports are handled confidentially, and the enforcement guidelines of the Contributor
+Covenant 2.1 apply.

bioai_evidence_validator-0.7.0/CONTRIBUTING.md

@@ -0,0 +1,76 @@
+# Contributing
+
+Thanks for helping make AI-assisted biocuration safer. Contributions of every size are
+welcome: a typo fix, a bug report with a failing record, a new domain profile, or a new
+real-data benchmark.
+
+By participating you agree to follow the [code of conduct](CODE_OF_CONDUCT.md).
+
+## Ways to contribute
+
+| You have… | Start here |
+|---|---|
+| A record the validator judges wrongly | [Bug report](https://github.com/NingyuSUN/bioai-evidence-validator/issues/new?template=bug_report.yml); attach the smallest record or draft that reproduces it |
+| An admission policy for your domain | [Profile proposal](https://github.com/NingyuSUN/bioai-evidence-validator/issues/new?template=profile_proposal.yml), then a pull request to [`community/profiles/`](community/profiles/README.md) |
+| An idea for the engine, CLI or formats | [Feature request](https://github.com/NingyuSUN/bioai-evidence-validator/issues/new?template=feature_request.yml) first, so we can agree on the contract before code |
+| A security problem | Do **not** open an issue; see [SECURITY.md](SECURITY.md) |
+
+Issues labelled [`good first issue`](https://github.com/NingyuSUN/bioai-evidence-validator/labels/good%20first%20issue)
+are scoped to be finished in an afternoon.
+
+## Development setup
+
+Python 3.11+ and [uv](https://docs.astral.sh/uv/):
+
+```bash
+git clone https://github.com/NingyuSUN/bioai-evidence-validator.git
+cd bioai-evidence-validator
+uv sync --frozen --extra dev
+```
+
+Before opening a pull request, run what CI runs:
+
+```bash
+uv run --frozen ruff check .
+uv run --frozen mypy
+uv run --frozen pytest --cov
+```
+
+CI also runs the tests on Linux (Python 3.11–3.13) and Windows, builds the wheel and
+smoke-tests it outside the source tree, and runs the GitHub Action. Coverage must stay at
+or above the minimum in `pyproject.toml`.
+
+## Ground rules for changes
+
+This project's value is that it **fails closed** and **says exactly what it checked**.
+Changes are reviewed against that:
+
+- **No silent weakening.** A change that admits something previously rejected needs an
+  explicit reason in the pull request and a test that shows the new boundary.
+- **Every rule has a test on both sides**: a record that passes and a minimal one that fails.
+- **Reports stay reproducible.** If a change alters report contents, the committed benchmark
+  results must be regenerated in the same pull request, and the diff explained.
+- **Domain logic stays out of the engine.** New domains are profiles and importers, not
+  special cases in `engine.py`.
+- **Honest limits.** Benchmarks state what their labels are (source-derived, authored, or
+  independently reviewed) and what they do not measure.
+- Match the surrounding style; ruff enforces correctness rules, not formatting.
+
+## Contributing a domain profile
+
+Community profiles live in [`community/profiles/`](community/profiles/README.md). Each one is
+a folder with a profile, a short README and example cases whose expected outcomes are
+checked by the test suite. Copy `community/profiles/_template/` to get started.
+
+## Pull requests
+
+- Keep each pull request to one purpose; link the issue it resolves.
+- Update `CHANGELOG.md` under a new heading if users will notice the change.
+- New third-party data needs its license, attribution and exact source version recorded,
+  as in `examples/*/sources/README.md`.
+
+## Releases
+
+Maintainers release by bumping the version in `pyproject.toml` and
+`src/bioevidence_validator/__init__.py`, adding a `CHANGELOG.md` section, and pushing a
+`vX.Y.Z` tag. The release workflow tests, publishes to PyPI and creates the GitHub release.