bioclaim 0.7.0__tar.gz

bioclaim-0.7.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sohil Ananth
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

bioclaim-0.7.0/PKG-INFO

@@ -0,0 +1,144 @@
+Metadata-Version: 2.4
+Name: bioclaim
+Version: 0.7.0
+Summary: A grounding firewall for biomedical LLMs: verifies biological identifiers and claims against authoritative databases, and flags fabrications.
+Author-email: Sohil Ananth <sohil.r@icloud.com>
+License: MIT
+Project-URL: Homepage, https://github.com/SxR24/bioclaim
+Project-URL: Issues, https://github.com/SxR24/bioclaim/issues
+Keywords: bioinformatics,llm,hallucination,grounding,genomics,ontology
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# bioclaim
+
+**A grounding firewall for biomedical LLMs.** Wrap it around any model's output and it
+verifies the biological identifiers against authoritative databases — UniProtKB, Ensembl,
+and EBI's Ontology Lookup Service — flagging the ones that are fabricated.
+
+Large language models constantly emit identifiers that *look* real but don't exist
+(`GO:9999999`, `HP:9999999`), alongside false gene–function and gene–disease claims. In
+research and clinical settings that is a documented safety risk. `bioclaim` is a
+deterministic, low-latency layer that catches it.
+
+> **Real-model finding:** across three models, **48–68% of specialist biology
+> answers contained a wrong biomedical identifier** (Llama-3.1-8B 68%, gpt-oss-120B
+> 60%, Llama-3.3-70B 48%). The dominant error is a real, valid accession pointing at
+> the **wrong gene** — invisible to existence checks, independently verified.
+> Full study: [RESULTS.md](RESULTS.md).
+
+## Benchmark result
+
+On a labeled set of **500 LLM-style answers** (394 injected fabrications):
+
+| Metric | Score |
+| --- | --- |
+| Precision (flagged that were truly fake) | **100.0%** |
+| Recall (fabrications caught) | **95.7%** (377 / 394) |
+| F1 | **0.978** |
+| False accusations | **0** |
+
+Every uncaught fabrication was a network-throttled lookup that safely returned
+`UNVERIFIED` — not a single fake was checked and missed. Reproduce with:
+
+```bash
+python scripts/generate_benchmark.py 500
+python scripts/benchmark.py data/benchmark_large.jsonl
+```
+
+## How it works
+
+Two layers, both designed to **never falsely accuse** (if a claim can't be verified it is
+marked `UNVERIFIED`, never `NOT_FOUND`):
+
+1. **FORMAT** — offline, deterministic. Rejects malformed identifiers.
+2. **EXISTS** — live existence check against the source database. Well-formed but absent
+   identifiers are flagged `NOT_FOUND`. Rate-limit/network errors are retried with backoff,
+   then degrade to `UNVERIFIED`.
+3. **CLAIM (v0.5)** — label consistency. A *real* identifier with a *wrong* description
+   (`GO:0006281 (photosynthesis)`, actually "DNA repair") is flagged
+   `SUPPORTED_LABEL_MISMATCH`. Synonym-aware, so paraphrases aren't falsely flagged.
+4. **ENTITY (v0.6)** — correspondence. A *real* UniProt/Ensembl ID attached to the
+   *wrong* gene ("the UniProt for TP53 is P38398", actually BRCA1) is flagged
+   `SUPPORTED_ENTITY_MISMATCH` — the hardest class, invisible to existence checks.
+
+Supported identifier types: GO, HP, MONDO, DOID, CHEBI, Ensembl gene (ENSG), UniProtKB.
+
+Every lookup is **cached to disk** ($BIOCLAIM_CACHE, else `~/.cache/bioclaim/`), so
+after a warm-up bioclaim runs fast, offline-capable, and immune to rate limits.
+
+## Install & use
+
+```bash
+pip install -e .           # from a clone; PyPI release: pip install bioclaim
+```
+
+**One-call API:**
+
+```python
+from bioclaim import check
+
+result = check("TP53 (P04637) is annotated with GO:9999999 (apoptosis).")
+print(result.ok)           # False
+for p in result.problems:
+    print(p)               # GO:9999999: fabricated (does not exist)
+```
+
+**Guard any model call** (raises if a fabrication slips through):
+
+```python
+from bioclaim import Firewall
+guarded = Firewall(raise_on_flag=True).guard(call_my_llm)
+answer = guarded(prompt)   # BioclaimFlag raised if the answer cites a fake ID
+```
+
+**Command line:**
+
+```bash
+bioclaim "TP53 is P04637 and the fake GO:9999999"
+echo "some model output" | bioclaim --entity BRCA1
+```
+
+## Project layout
+
+```
+bioclaim/              core package (patterns, live sources, validator)
+scripts/               runnable CLIs: demo, batch, benchmark, generate_benchmark
+data/                  sample_answers.jsonl (labeled demo set)
+tests/                 offline unit tests
+```
+
+Run the pieces (from the repo root):
+
+```bash
+python scripts/demo.py                     # catch fakes in one example answer
+python scripts/benchmark.py                # precision/recall on the sample set
+python scripts/batch.py your_answers.jsonl # scan your own AI answers
+python -m pytest                           # run tests
+```
+
+## Roadmap (land-and-expand)
+
+- **v0.1–0.3** Identifier validation across ontologies, genes, and proteins *(done)*
+- **v0.4** Local database snapshots — microsecond lookups, fully offline, true 100% recall
+- **v0.5** Claim verification — relationships, not just IDs (gene–disease, gene–function)
+- **v0.6** Calibrated confidence per claim
+- **v1.0** Public leaderboard + LLM-framework integrations
+
+## Why this design wins
+
+Generality (all identifier types, one layer) · Deployability (pure standard library,
+3-line integration) · Speed (deterministic lookups, not agent loops) · Trust (never a
+false accusation).
+
+## License
+
+MIT — see [LICENSE](LICENSE).

bioclaim-0.7.0/README.md

@@ -0,0 +1,124 @@
+# bioclaim
+
+**A grounding firewall for biomedical LLMs.** Wrap it around any model's output and it
+verifies the biological identifiers against authoritative databases — UniProtKB, Ensembl,
+and EBI's Ontology Lookup Service — flagging the ones that are fabricated.
+
+Large language models constantly emit identifiers that *look* real but don't exist
+(`GO:9999999`, `HP:9999999`), alongside false gene–function and gene–disease claims. In
+research and clinical settings that is a documented safety risk. `bioclaim` is a
+deterministic, low-latency layer that catches it.
+
+> **Real-model finding:** across three models, **48–68% of specialist biology
+> answers contained a wrong biomedical identifier** (Llama-3.1-8B 68%, gpt-oss-120B
+> 60%, Llama-3.3-70B 48%). The dominant error is a real, valid accession pointing at
+> the **wrong gene** — invisible to existence checks, independently verified.
+> Full study: [RESULTS.md](RESULTS.md).
+
+## Benchmark result
+
+On a labeled set of **500 LLM-style answers** (394 injected fabrications):
+
+| Metric | Score |
+| --- | --- |
+| Precision (flagged that were truly fake) | **100.0%** |
+| Recall (fabrications caught) | **95.7%** (377 / 394) |
+| F1 | **0.978** |
+| False accusations | **0** |
+
+Every uncaught fabrication was a network-throttled lookup that safely returned
+`UNVERIFIED` — not a single fake was checked and missed. Reproduce with:
+
+```bash
+python scripts/generate_benchmark.py 500
+python scripts/benchmark.py data/benchmark_large.jsonl
+```
+
+## How it works
+
+Two layers, both designed to **never falsely accuse** (if a claim can't be verified it is
+marked `UNVERIFIED`, never `NOT_FOUND`):
+
+1. **FORMAT** — offline, deterministic. Rejects malformed identifiers.
+2. **EXISTS** — live existence check against the source database. Well-formed but absent
+   identifiers are flagged `NOT_FOUND`. Rate-limit/network errors are retried with backoff,
+   then degrade to `UNVERIFIED`.
+3. **CLAIM (v0.5)** — label consistency. A *real* identifier with a *wrong* description
+   (`GO:0006281 (photosynthesis)`, actually "DNA repair") is flagged
+   `SUPPORTED_LABEL_MISMATCH`. Synonym-aware, so paraphrases aren't falsely flagged.
+4. **ENTITY (v0.6)** — correspondence. A *real* UniProt/Ensembl ID attached to the
+   *wrong* gene ("the UniProt for TP53 is P38398", actually BRCA1) is flagged
+   `SUPPORTED_ENTITY_MISMATCH` — the hardest class, invisible to existence checks.
+
+Supported identifier types: GO, HP, MONDO, DOID, CHEBI, Ensembl gene (ENSG), UniProtKB.
+
+Every lookup is **cached to disk** ($BIOCLAIM_CACHE, else `~/.cache/bioclaim/`), so
+after a warm-up bioclaim runs fast, offline-capable, and immune to rate limits.
+
+## Install & use
+
+```bash
+pip install -e .           # from a clone; PyPI release: pip install bioclaim
+```
+
+**One-call API:**
+
+```python
+from bioclaim import check
+
+result = check("TP53 (P04637) is annotated with GO:9999999 (apoptosis).")
+print(result.ok)           # False
+for p in result.problems:
+    print(p)               # GO:9999999: fabricated (does not exist)
+```
+
+**Guard any model call** (raises if a fabrication slips through):
+
+```python
+from bioclaim import Firewall
+guarded = Firewall(raise_on_flag=True).guard(call_my_llm)
+answer = guarded(prompt)   # BioclaimFlag raised if the answer cites a fake ID
+```
+
+**Command line:**
+
+```bash
+bioclaim "TP53 is P04637 and the fake GO:9999999"
+echo "some model output" | bioclaim --entity BRCA1
+```
+
+## Project layout
+
+```
+bioclaim/              core package (patterns, live sources, validator)
+scripts/               runnable CLIs: demo, batch, benchmark, generate_benchmark
+data/                  sample_answers.jsonl (labeled demo set)
+tests/                 offline unit tests
+```
+
+Run the pieces (from the repo root):
+
+```bash
+python scripts/demo.py                     # catch fakes in one example answer
+python scripts/benchmark.py                # precision/recall on the sample set
+python scripts/batch.py your_answers.jsonl # scan your own AI answers
+python -m pytest                           # run tests
+```
+
+## Roadmap (land-and-expand)
+
+- **v0.1–0.3** Identifier validation across ontologies, genes, and proteins *(done)*
+- **v0.4** Local database snapshots — microsecond lookups, fully offline, true 100% recall
+- **v0.5** Claim verification — relationships, not just IDs (gene–disease, gene–function)
+- **v0.6** Calibrated confidence per claim
+- **v1.0** Public leaderboard + LLM-framework integrations
+
+## Why this design wins
+
+Generality (all identifier types, one layer) · Deployability (pure standard library,
+3-line integration) · Speed (deterministic lookups, not agent loops) · Trust (never a
+false accusation).
+
+## License
+
+MIT — see [LICENSE](LICENSE).

bioclaim-0.7.0/bioclaim/__init__.py

@@ -0,0 +1,10 @@
+from .validator import scan, report, Verdict
+from .claims import check_claims, report_claims, ClaimVerdict, extract_target_entity
+from .api import check, Result, Problem, Firewall, BioclaimFlag
+
+__all__ = [
+    "check", "Result", "Problem", "Firewall", "BioclaimFlag",   # primary API
+    "scan", "report", "Verdict",
+    "check_claims", "report_claims", "ClaimVerdict", "extract_target_entity",
+]
+__version__ = "0.7.0"

bioclaim-0.7.0/bioclaim/__main__.py

@@ -0,0 +1,33 @@
+"""Command-line interface:  bioclaim "text with GO:9999999 ..."
+
+Reads text from the arguments or stdin, verifies every biomedical identifier,
+and prints the problems. Exit code is non-zero if any identifier is flagged
+(handy in CI / guardrail scripts).
+"""
+import sys
+import argparse
+from . import check, __version__
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="bioclaim",
+                                 description="Verify biomedical identifiers in text.")
+    ap.add_argument("text", nargs="*", help="text to check (or pipe via stdin)")
+    ap.add_argument("--entity", default=None,
+                    help="the gene/protein the text is about (enables wrong-gene check)")
+    ap.add_argument("--offline", action="store_true",
+                    help="format check only; skip database lookups")
+    ap.add_argument("--version", action="version", version=f"bioclaim {__version__}")
+    args = ap.parse_args(argv)
+
+    text = " ".join(args.text) if args.text else sys.stdin.read()
+    if not text.strip():
+        ap.error("no text provided (pass as arguments or via stdin)")
+
+    result = check(text, entity_hint=args.entity, online=not args.offline)
+    print(result)
+    return 1 if not result.ok else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

bioclaim-0.7.0/bioclaim/api.py

@@ -0,0 +1,110 @@
+"""The one-call public API for bioclaim.
+
+    from bioclaim import check
+    result = check("TP53 (P04637) is annotated with GO:9999999.")
+    print(result.ok, result.problems)
+
+Also a Firewall wrapper to guard any function that returns LLM text.
+"""
+from dataclasses import dataclass
+from typing import List, Optional
+from .claims import check_claims, FLAGGED_STATUSES
+
+_LABELS = {
+    "NOT_FOUND": "fabricated (does not exist)",
+    "INVALID_FORMAT": "malformed identifier",
+    "SUPPORTED_LABEL_MISMATCH": "wrong description",
+    "SUPPORTED_ENTITY_MISMATCH": "wrong gene/entity",
+    "SUPPORTED_OBSOLETE": "obsolete / deprecated",
+}
+
+
+@dataclass
+class Problem:
+    curie: str
+    kind: str          # human-readable problem type
+    status: str
+    claimed: Optional[str]
+    actual: Optional[str]
+
+    def __str__(self):
+        extra = ""
+        if self.claimed and self.actual:
+            extra = f'  (said "{self.claimed}", actually "{self.actual}")'
+        elif self.actual:
+            extra = f'  (actually "{self.actual}")'
+        return f"{self.curie}: {self.kind}{extra}"
+
+
+@dataclass
+class Result:
+    text: str
+    n_ids: int
+    problems: List[Problem]
+
+    @property
+    def ok(self) -> bool:
+        """True if no problems were found."""
+        return not self.problems
+
+    def __bool__(self):
+        return self.ok
+
+    def __str__(self):
+        if self.ok:
+            return f"OK - {self.n_ids} identifier(s) checked, none flagged"
+        lines = [f"{len(self.problems)} problem(s) in {self.n_ids} identifier(s):"]
+        lines += [f"  - {p}" for p in self.problems]
+        return "\n".join(lines)
+
+
+def check(text, entity_hint=None, online=True) -> Result:
+    """Verify every biomedical identifier in `text`. Returns a Result.
+
+    entity_hint: the gene/protein the text is about, if known (enables the
+    wrong-gene check on free text).
+    """
+    verdicts = check_claims(text, online=online, entity_hint=entity_hint)
+    problems = [
+        Problem(v.curie, _LABELS.get(v.status, v.status), v.status,
+                v.claimed_label, v.canonical_label)
+        for v in verdicts if v.status in FLAGGED_STATUSES
+    ]
+    return Result(text=text, n_ids=len(verdicts), problems=problems)
+
+
+class Firewall:
+    """Guard any function that returns model text.
+
+        fw = Firewall()
+        answer = fw.guard(call_my_llm)(prompt)   # raises if a fabrication slips through
+
+    or non-raising:
+
+        result = fw(answer_text)
+    """
+
+    def __init__(self, entity_hint=None, online=True, raise_on_flag=False):
+        self.entity_hint = entity_hint
+        self.online = online
+        self.raise_on_flag = raise_on_flag
+
+    def __call__(self, text) -> Result:
+        r = check(text, entity_hint=self.entity_hint, online=self.online)
+        if self.raise_on_flag and not r.ok:
+            raise BioclaimFlag(r)
+        return r
+
+    def guard(self, fn):
+        """Decorator: run fn, then verify its returned text."""
+        def wrapped(*args, **kwargs):
+            text = fn(*args, **kwargs)
+            self(text)          # may raise if raise_on_flag
+            return text
+        return wrapped
+
+
+class BioclaimFlag(Exception):
+    def __init__(self, result: Result):
+        self.result = result
+        super().__init__(str(result))

bioclaim-0.7.0/bioclaim/cache.py

@@ -0,0 +1,77 @@
+"""Transparent, persistent on-disk cache for database lookups.
+
+Every existence / entity lookup is cached to disk, so:
+  - repeat lookups are instant,
+  - a warmed cache works fully offline,
+  - rate limits stop mattering (each id is fetched at most once, ever).
+
+Only *definitive* results are cached (a real answer from the database). Transient
+failures (network/None) are never cached, so a hiccup can't poison the cache.
+
+Cache location: $BIOCLAIM_CACHE, else ~/.cache/bioclaim/. Disable with
+BIOCLAIM_CACHE=off.
+"""
+import os
+import json
+import atexit
+import pathlib
+import threading
+
+
+def _cache_dir():
+    override = os.environ.get("BIOCLAIM_CACHE")
+    if override and override.lower() == "off":
+        return None
+    base = pathlib.Path(override) if override else pathlib.Path.home() / ".cache" / "bioclaim"
+    try:
+        base.mkdir(parents=True, exist_ok=True)
+        return base
+    except Exception:
+        return None
+
+
+class DiskCache:
+    """A tiny JSON-backed dict with atomic, batched persistence."""
+
+    def __init__(self, name):
+        self._dir = _cache_dir()
+        self._path = (self._dir / f"{name}.json") if self._dir else None
+        self._lock = threading.Lock()
+        self._data = {}
+        self._writes = 0
+        if self._path and self._path.exists():
+            try:
+                self._data = json.loads(self._path.read_text())
+            except Exception:
+                self._data = {}
+        atexit.register(self.flush)
+
+    _MISS = object()
+
+    def get(self, key):
+        return self._data.get(key, DiskCache._MISS)
+
+    def set(self, key, value):
+        with self._lock:
+            self._data[key] = value
+            self._writes += 1
+            due = self._writes % 25 == 0
+        if due:
+            self.flush()
+
+    def flush(self):
+        if not self._path:
+            return
+        with self._lock:
+            try:
+                tmp = self._path.with_suffix(".tmp")
+                tmp.write_text(json.dumps(self._data))
+                tmp.replace(self._path)
+            except Exception:
+                pass
+
+    def __len__(self):
+        return len(self._data)
+
+
+MISS = DiskCache._MISS