hponorm 1.0.0__tar.gz

hponorm-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Tim Hearn 2026
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hponorm-1.0.0/PKG-INFO

@@ -0,0 +1,10 @@
+Metadata-Version: 2.4
+Name: hponorm
+Version: 1.0.0
+Summary: Interactively validate and normalise GA4GH phenopacket phenotypic-feature terms against the HPO ontology
+Requires-Python: >=3.9
+License-File: LICENSE
+Provides-Extra: recommended
+Requires-Dist: rapidfuzz>=3.0; extra == "recommended"
+Requires-Dist: rich>=13.0; extra == "recommended"
+Dynamic: license-file

hponorm-1.0.0/README.md

@@ -0,0 +1,156 @@
+# hponorm
+
+Interactively **validate and normalise the phenotypic-feature terms** in GA4GH
+phenopacket / family JSON files against the **HPO** (Human Phenotype Ontology).
+
+It's the sibling of `mondonorm` (which does disease terms → MONDO) and shares the
+same look and feel. This one targets `phenotypicFeatures[].type` and HPO.
+
+Your generator currently emits phenotype terms with an empty `id` and a
+free-text label:
+
+```json
+"phenotypicFeatures": [ { "type": { "id": "", "label": "long qt interval" } } ]
+```
+
+`hponorm` walks the file, finds every phenotypic feature, suggests the most
+likely HPO terms for each label, lets you pick one (or type an HP id yourself),
+and writes a normalised copy:
+
+```json
+"phenotypicFeatures": [ { "type": { "id": "HP:0001657", "label": "Prolonged QT interval" } } ]
+```
+
+## How suggestions are sourced
+
+Two interchangeable backends; pick with `--backend`:
+
+| Backend | What it uses | Network |
+|---------|--------------|---------|
+| `ols`   | EBI Ontology Lookup Service v4 (live HPO) | required |
+| `local` | A local `hp.obo` / `hp.json`, downloaded + cached on first use | only for the one-time download |
+| `auto`  | OLS if reachable, otherwise local (the default) | preferred |
+
+The official ontology file can be fetched automatically (from
+`purl.obolibrary.org/obo/hp.obo`) or supplied with `--hpo-file`.
+A tiny **illustrative** offline sample (`hponorm/data/hpo-sample.obo`, 8 terms)
+ships with the package so you can try the tool with no network — it is **not** a
+substitute for the real ontology.
+
+## Install
+
+```bash
+pip install -e .                 # installs the `hponorm` command
+pip install -e ".[recommended]"  # + rapidfuzz (better matching) and rich (nicer UI)
+```
+
+Both extras are optional: without `rapidfuzz` it falls back to stdlib `difflib`;
+without `rich` it prints plain text.
+
+## Quick start
+
+```bash
+# Online (auto-detects OLS):
+hponorm test3_phenopackets.json
+
+# Fully offline against a downloaded ontology:
+hponorm myfile.json --backend local --hpo-file /path/to/hp.obo
+
+# Try it offline with the bundled sample (diabetes / long QT / a few others):
+hponorm myfile.json --backend local --hpo-file hponorm/data/hpo-sample.obo
+
+# Equivalent without installing:
+python -m hponorm myfile.json
+```
+
+## The interactive review
+
+Each **distinct** label is reviewed once and the decision is applied to every
+occurrence (so "diabetes" appearing 8 times is one question, not eight). For
+each label you see a ranked table and a prompt:
+
+```
+Select [#, s words, h HP:id, r, k, x, ?, q]:
+  <number>     select that suggestion
+  s <words>    search again with different words
+  h HP:id      enter an HPO id manually (validated, canonical label fetched)
+  r            reuse a decision remembered from earlier in this session
+  k            keep the current term unchanged
+  x            skip this label (leave it unmapped)
+  ?            help
+  q            finish now: apply decisions made so far and save
+```
+
+If a feature already has an `id`, it is validated: the tool reports whether the
+id resolves in HPO and whether its canonical label matches the existing label.
+A feature's `excluded` flag (a negated phenotype) is preserved untouched — only
+`type.id` / `type.label` are changed.
+
+## Output
+
+For `input.json` (unless you pass `--out` or `--in-place`):
+
+* `input.normalized.json` — the normalised phenopacket (only `type.id` /
+  `type.label` are changed; everything else, including the pedigree and any
+  `diseases` terms, is left exactly as-is).
+* `input.normalized.json.mapping.json` — a decision log (label → HP id, count).
+
+Because `hponorm` only touches `phenotypicFeatures`, you can run it alongside
+`mondonorm` (which only touches `diseases`) on the same file without conflict.
+
+## Useful options
+
+```
+--out PATH            output path (single input file only)
+--in-place            overwrite the input file(s)
+--backend {auto,ols,local}
+--hpo-file PATH       local hp.obo or hp.json
+--cache-dir DIR       where the downloaded HPO + parsed index are cached
+--update              re-download / re-parse HPO
+--no-download         never download (local backend must find a file)
+--no-online           never use OLS, even in auto mode
+--limit N             suggestions shown per label (default 8)
+--mapping FILE        load/save remembered label->term decisions across runs
+--auto-remembered     auto-apply remembered labels without prompting
+--no-color            plain text output
+```
+
+Process several files in one go; remembered decisions carry across them:
+
+```bash
+hponorm *.json --mapping team-hpo-map.json
+# next time, reuse without re-typing:
+hponorm new/*.json --mapping team-hpo-map.json --auto-remembered
+```
+
+## Use as a library
+
+```python
+from hponorm import HpoIndex, OlsClient, Suggester
+from hponorm import phenopackets as pp
+
+sug = Suggester(OlsClient(), name="ols")          # or Suggester(HpoIndex.load(...))
+for c in sug.suggest("long qt interval", limit=5):
+    print(c.id, c.label, c.score)
+
+data = pp.load("myfile.json")
+for ref in pp.find_phenotypic_features(data):
+    print(ref.path, ref.label, ref.id, "excluded" if ref.excluded else "")
+```
+
+## Getting the full HPO ontology
+
+* Browser / search: https://hpo.jax.org
+* Direct files: `http://purl.obolibrary.org/obo/hp.obo` or `.../hp.json`
+* Releases: https://github.com/obophenotype/human-phenotype-ontology/releases
+
+Phenopackets reference: https://phenopacket-schema.readthedocs.io
+(phenotypic features use HPO `OntologyClass` values in `PhenotypicFeature.type`).
+
+## A note on the bundled sample's ids
+
+The HP ids in `hpo-sample.obo` were checked against HPO browsers (e.g. Diabetes
+mellitus HP:0000819, Prolonged QT interval HP:0001657, Seizure HP:0001250,
+Microcephaly HP:0000252, Short stature HP:0004322, Global developmental delay
+HP:0001263). It is still only a tiny demo fixture — verify against the live HPO
+for real curation work.

hponorm-1.0.0/hponorm/cli.py

@@ -0,0 +1,390 @@
+"""Interactive command-line tool to validate and normalise the phenotypic
+feature terms in GA4GH phenopacket / family JSON files against the HPO ontology.
+
+Run:  python -m hponorm FILE.json [FILE2.json ...]
+"""
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from . import phenopackets as pp
+from .hpo import HpoIndex, normalise_curie
+from .ols import OlsClient
+from .suggest import Candidate, Suggester
+
+# --------------------------------------------------------------------- pretty
+try:
+    from rich.console import Console
+    from rich.table import Table
+
+    _console = Console()
+    _HAVE_RICH = True
+except Exception:  # pragma: no cover
+    _console = None
+    _HAVE_RICH = False
+
+
+class UI:
+    def __init__(self, color: bool = True):
+        self.rich = _HAVE_RICH and color
+
+    def rule(self, text: str = "") -> None:
+        if self.rich:
+            _console.rule(f"[bold]{text}")
+        else:
+            print("\n" + "=" * 70)
+            if text:
+                print(text)
+                print("-" * 70)
+
+    def say(self, text: str = "") -> None:
+        (_console.print if self.rich else print)(text)
+
+    def warn(self, text: str) -> None:
+        self.say(f"[yellow]! {text}[/yellow]" if self.rich else f"! {text}")
+
+    def ok(self, text: str) -> None:
+        self.say(f"[green]\u2713 {text}[/green]" if self.rich else f"\u2713 {text}")
+
+    def err(self, text: str) -> None:
+        self.say(f"[red]\u2717 {text}[/red]" if self.rich else f"\u2717 {text}")
+
+    def candidates(self, cands: list[Candidate]) -> None:
+        if not cands:
+            self.warn("No suggestions found. Try 's <better search words>' or 'h <HP:id>'.")
+            return
+        if self.rich:
+            t = Table(show_header=True, header_style="bold cyan", box=None, pad_edge=False)
+            t.add_column("#", justify="right")
+            t.add_column("HPO id", style="magenta", no_wrap=True)
+            t.add_column("Label", style="white")
+            t.add_column("Score", justify="right")
+            t.add_column("Matched on", style="dim")
+            t.add_column("Definition", style="dim")
+            for i, c in enumerate(cands, 1):
+                t.add_row(str(i), c.id, c.label, f"{c.score:g}", c.matched_on, c.short_def(70))
+            _console.print(t)
+        else:
+            for i, c in enumerate(cands, 1):
+                print(f"  {i:>2}. {c.id}  {c.label}  (score {c.score:g}, {c.matched_on})")
+                if c.short_def(80):
+                    print(f"      {c.short_def(80)}")
+
+    def prompt(self, text: str) -> str:
+        try:
+            return input(text)
+        except EOFError:
+            return "q"
+
+
+HELP = """\
+Choices:
+  <number>      select that suggestion
+  s <words>     search again with different words
+  h <HP:id>     enter an HPO id manually (e.g. h HP:0001657) - it is validated
+  k             keep the current term unchanged
+  x             skip this label (leave it unmapped)
+  ?             show this help
+  q             finish now: apply decisions made so far and save
+"""
+
+
+def build_suggester(args, ui: UI) -> Suggester:
+    backend = args.backend
+    if backend == "ols":
+        ui.say("Backend: EBI OLS (online).")
+        return Suggester(OlsClient(), name="ols")
+    if backend == "local":
+        idx = HpoIndex.load(
+            args.hpo_file,
+            allow_download=not args.no_download,
+            update=args.update,
+            cache_dir=args.cache_dir,
+            log=ui.say,
+        )
+        ui.say(f"Backend: local HPO ({len(idx)} terms from {idx.source}).")
+        return Suggester(idx, name="hpo-local")
+    # auto
+    ols = OlsClient()
+    if not args.no_online and ols.available():
+        ui.say("Backend: EBI OLS (online).")
+        # Local is used as a silent fallback only if it is already cheap to load.
+        fb = None
+        try:
+            if args.hpo_file or (Path(args.cache_dir or _default_cache()) / "hp.obo").exists():
+                fb = HpoIndex.load(
+                    args.hpo_file, allow_download=False,
+                    cache_dir=args.cache_dir, log=lambda *a, **k: None,
+                )
+        except Exception:
+            fb = None
+        return Suggester(ols, fallback=fb, name="ols")
+    ui.warn("OLS not reachable; using local HPO.")
+    idx = HpoIndex.load(
+        args.hpo_file,
+        allow_download=not args.no_download,
+        update=args.update,
+        cache_dir=args.cache_dir,
+        log=ui.say,
+    )
+    ui.say(f"Backend: local HPO ({len(idx)} terms from {idx.source}).")
+    return Suggester(idx, name="hpo-local")
+
+
+def _default_cache():
+    from .hpo import default_cache_dir
+
+    return default_cache_dir()
+
+
+def review_group(
+    ui: UI,
+    suggester: Suggester,
+    label: str,
+    existing_id: str,
+    count: int,
+    index: int,
+    total: int,
+    remembered: dict[str, dict],
+    limit: int,
+) -> dict | None:
+    """Drive the interactive review of one label group.
+
+    Returns a decision dict {id,label,action} or None to abort/quit.
+    The special action 'quit' tells the caller to stop and save.
+    """
+    ui.rule(f"[{index}/{total}] phenotype label: {label!r}   ({count} occurrence(s))")
+
+    if existing_id:
+        existing = suggester.resolve(existing_id)
+        if existing and existing.label.lower() == label.lower():
+            ui.ok(f"Already a valid HPO term: {existing.id}  {existing.label}")
+        elif existing:
+            ui.warn(
+                f"Existing id {existing_id} resolves to {existing.id} '{existing.label}', "
+                f"which differs from the label '{label}'."
+            )
+        else:
+            ui.warn(f"Existing id {existing_id!r} could not be validated against HPO.")
+
+    query = label
+    cands = suggester.suggest(query, limit=limit)
+
+    # Float a remembered decision for this label to the very top.
+    mem = remembered.get(label.lower())
+    if mem:
+        ui.ok(f"Remembered from earlier: {mem['id']}  {mem['label']}  (enter 'r' to reuse)")
+
+    while True:
+        ui.candidates(cands)
+        raw = ui.prompt("Select [#, s words, h HP:id, r, k, x, ?, q]: ").strip()
+
+        if raw == "":
+            continue
+        if raw in ("?", "help"):
+            ui.say(HELP)
+            continue
+        if raw in ("q", "quit"):
+            return {"action": "quit"}
+        if raw in ("k", "keep"):
+            return {"action": "keep"}
+        if raw in ("x", "skip"):
+            return {"action": "skip"}
+        if raw in ("r", "reuse") and mem:
+            return {"action": "map", "id": mem["id"], "label": mem["label"]}
+
+        # search again
+        if raw.startswith("s ") or raw == "s":
+            query = raw[2:].strip() or ui.prompt("Search words: ").strip()
+            if query:
+                cands = suggester.suggest(query, limit=limit)
+            continue
+
+        # explicit manual id  (h HP:0001657)  or a bare curie
+        manual = None
+        if raw.startswith("h "):
+            manual = raw[2:].strip()
+        elif normalise_curie(raw):
+            # Only treat as a curie if it is NOT a small selection index.
+            if not (raw.isdigit() and 1 <= int(raw) <= len(cands)):
+                manual = raw
+        if manual is not None:
+            c = normalise_curie(manual)
+            if not c:
+                ui.err(f"{manual!r} is not a valid HPO id (expected HP:0000000).")
+                continue
+            resolved = suggester.resolve(c)
+            if resolved:
+                ui.ok(f"{resolved.id}  {resolved.label}")
+                if _confirm(ui, "Use this term?"):
+                    return {"action": "map", "id": resolved.id, "label": resolved.label}
+                continue
+            ui.warn(f"Could not validate {c} against HPO.")
+            if _confirm(ui, f"Use {c} anyway with label {label!r}?"):
+                return {"action": "map", "id": c, "label": label}
+            continue
+
+        # numeric selection
+        if raw.isdigit():
+            i = int(raw)
+            if 1 <= i <= len(cands):
+                chosen = cands[i - 1]
+                return {"action": "map", "id": chosen.id, "label": chosen.label}
+            ui.err(f"Pick a number between 1 and {len(cands)}.")
+            continue
+
+        ui.err("Unrecognised input. Type ? for help.")
+
+
+def _confirm(ui: UI, text: str) -> bool:
+    ans = ui.prompt(f"{text} [Y/n]: ").strip().lower()
+    return ans in ("", "y", "yes")
+
+
+def process_file(path: Path, args, ui: UI, suggester: Suggester, remembered: dict) -> dict:
+    ui.rule(f"FILE: {path.name}")
+    data = pp.load(path)
+    refs = pp.find_phenotypic_features(data)
+    if not refs:
+        ui.warn("No phenotypic features found in this file.")
+        return {"file": str(path), "mapped": 0, "kept": 0, "skipped": 0, "total": 0}
+
+    groups = pp.group_by_label(refs)
+    ui.say(f"Found {len(refs)} phenotypic feature(s) across {len(groups)} distinct label(s).")
+
+    decisions: list[dict] = []
+    stop = False
+    items = list(groups.items())
+    for index, ((label, existing_id), group_refs) in enumerate(items, 1):
+        if stop:
+            break
+        if args.auto_remembered and label.lower() in remembered:
+            mem = remembered[label.lower()]
+            for r in group_refs:
+                r.apply(mem["id"], mem["label"])
+            ui.ok(f"[auto] {label!r} -> {mem['id']} {mem['label']}")
+            decisions.append({"label": label, **mem, "action": "map", "count": len(group_refs)})
+            continue
+
+        decision = review_group(
+            ui, suggester, label, existing_id, len(group_refs),
+            index, len(items), remembered, args.limit,
+        )
+        action = decision.get("action")
+        if action == "quit":
+            stop = True
+            ui.warn("Finishing early - remaining labels left unchanged.")
+            break
+        if action == "keep":
+            ui.say(f"Kept {label!r} unchanged.")
+            decisions.append({"label": label, "action": "keep", "count": len(group_refs)})
+            continue
+        if action == "skip":
+            ui.say(f"Skipped {label!r}.")
+            decisions.append({"label": label, "action": "skip", "count": len(group_refs)})
+            continue
+        # map
+        cid, clabel = decision["id"], decision["label"]
+        for r in group_refs:
+            r.apply(cid, clabel)
+        remembered[label.lower()] = {"id": cid, "label": clabel}
+        ui.ok(f"Mapped {label!r} -> {cid} {clabel} ({len(group_refs)} occurrence(s)).")
+        decisions.append({"label": label, "id": cid, "label_hpo": clabel,
+                          "action": "map", "count": len(group_refs)})
+
+    # ---- write output
+    if args.out and len(args._files) == 1:
+        out_path = Path(args.out)
+    elif args.in_place:
+        out_path = path
+    else:
+        out_path = path.with_name(path.stem + ".normalized.json")
+    pp.save(data, out_path)
+    ui.ok(f"Wrote normalised file: {out_path}")
+
+    # ---- write a mapping sidecar
+    sidecar = out_path.with_suffix(out_path.suffix + ".mapping.json")
+    pp.save({"file": str(path), "decisions": decisions}, sidecar)
+    ui.say(f"Wrote decision log: {sidecar}")
+
+    mapped = sum(d["count"] for d in decisions if d["action"] == "map")
+    kept = sum(d["count"] for d in decisions if d["action"] == "keep")
+    skipped = sum(d["count"] for d in decisions if d["action"] == "skip")
+    return {"file": str(path), "mapped": mapped, "kept": kept,
+            "skipped": skipped, "total": len(refs)}
+
+
+def main(argv: list[str] | None = None) -> int:
+    p = argparse.ArgumentParser(
+        prog="hponorm",
+        description="Interactively map free-text phenopacket phenotype labels to HPO terms.",
+    )
+    p.add_argument("files", nargs="+", help="Phenopacket / family JSON file(s).")
+    p.add_argument("-o", "--out", help="Output path (only valid with a single input file).")
+    p.add_argument("--in-place", action="store_true", help="Overwrite the input file(s).")
+    p.add_argument("--backend", choices=["auto", "ols", "local"], default="auto",
+                   help="Suggestion source (default: auto -> OLS online, else local).")
+    p.add_argument("--hpo-file", help="Path to a local hp.obo or hp.json.")
+    p.add_argument("--cache-dir", help="Where to cache the downloaded HPO file.")
+    p.add_argument("--update", action="store_true", help="Re-download / re-parse HPO.")
+    p.add_argument("--no-download", action="store_true",
+                   help="Never download HPO (local backend must find a file).")
+    p.add_argument("--no-online", action="store_true",
+                   help="Do not use OLS even in auto mode.")
+    p.add_argument("--limit", type=int, default=8, help="Suggestions shown per label.")
+    p.add_argument("--mapping", help="JSON file of remembered label->term decisions to "
+                                     "load and update across runs.")
+    p.add_argument("--auto-remembered", action="store_true",
+                   help="Auto-apply remembered labels without prompting.")
+    p.add_argument("--no-color", action="store_true", help="Disable rich/colour output.")
+    args = p.parse_args(argv)
+    args._files = args.files
+
+    ui = UI(color=not args.no_color)
+
+    if args.out and len(args.files) > 1:
+        ui.err("--out can only be used with a single input file.")
+        return 2
+
+    # Load remembered decisions, if any.
+    remembered: dict[str, dict] = {}
+    if args.mapping and Path(args.mapping).exists():
+        try:
+            remembered = json.loads(Path(args.mapping).read_text())
+            ui.say(f"Loaded {len(remembered)} remembered mapping(s) from {args.mapping}.")
+        except Exception:
+            ui.warn(f"Could not read mapping file {args.mapping}; starting fresh.")
+
+    try:
+        suggester = build_suggester(args, ui)
+    except Exception as exc:
+        ui.err(f"Could not initialise an HPO backend: {exc}")
+        return 1
+
+    summaries = []
+    for f in args.files:
+        path = Path(f)
+        if not path.exists():
+            ui.err(f"File not found: {path}")
+            continue
+        summaries.append(process_file(path, args, ui, suggester, remembered))
+
+    # Persist remembered decisions for next time.
+    if args.mapping:
+        try:
+            Path(args.mapping).write_text(json.dumps(remembered, indent=2, ensure_ascii=False))
+            ui.say(f"Saved {len(remembered)} remembered mapping(s) to {args.mapping}.")
+        except Exception as exc:
+            ui.warn(f"Could not write mapping file: {exc}")
+
+    ui.rule("Summary")
+    for s in summaries:
+        ui.say(f"{Path(s['file']).name}: {s['mapped']} mapped, {s['kept']} kept, "
+               f"{s['skipped']} skipped (of {s['total']} feature(s)).")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

hponorm-1.0.0/hponorm/hpo.py

@@ -0,0 +1,317 @@
+"""Local HPO (Human Phenotype Ontology) index.
+
+Loads HPO from a local ``.obo`` or obographs ``.json`` file (downloading and
+caching the official release on first use if permitted), builds an in-memory
+search index over term labels + synonyms, and answers two questions:
+
+* ``search(query)``  -> ranked candidate terms for a free-text phenotype label
+* ``get(curie)``     -> the canonical term for a specific ``HP:xxxxxxx`` id
+
+Fuzzy matching uses ``rapidfuzz`` when available and falls back to the stdlib
+``difflib`` otherwise, so the tool runs even in a minimal environment.
+"""
+from __future__ import annotations
+
+import json
+import os
+import pickle
+import re
+import urllib.request
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from .suggest import Candidate
+
+# Official, version-pinned-to-"latest" PURLs maintained by the Monarch Initiative.
+HPO_OBO_URL = "http://purl.obolibrary.org/obo/hp.obo"
+HPO_JSON_URL = "http://purl.obolibrary.org/obo/hp.json"
+
+_CURIE_RE = re.compile(r"^HP:\d{7}$")
+_IRI_RE = re.compile(r"HP_(\d{7})")
+
+try:  # Optional, much faster + better quality.
+    from rapidfuzz import fuzz, process
+
+    _HAVE_RAPIDFUZZ = True
+except Exception:  # pragma: no cover - exercised only when dep is absent
+    import difflib
+
+    _HAVE_RAPIDFUZZ = False
+
+
+@dataclass
+class HpoTerm:
+    id: str
+    label: str
+    synonyms: list[str] = field(default_factory=list)
+    definition: str | None = None
+    obsolete: bool = False
+
+
+def default_cache_dir() -> Path:
+    base = os.environ.get("XDG_CACHE_HOME") or str(Path.home() / ".cache")
+    return Path(base) / "hponorm"
+
+
+def normalise_curie(raw: str) -> str | None:
+    """Coerce user input / IRIs to a canonical ``HP:xxxxxxx`` curie, or None."""
+    s = raw.strip()
+    if not s:
+        return None
+    m = _IRI_RE.search(s)
+    if m:
+        return f"HP:{m.group(1)}"
+    s = s.replace("_", ":")
+    if not s.upper().startswith("HP"):
+        # Allow bare 7-digit ids like "0001657".
+        digits = re.sub(r"\D", "", s)
+        if len(digits) == 7:
+            return f"HP:{digits}"
+        return None
+    digits = re.sub(r"\D", "", s)
+    if len(digits) != 7:
+        return None
+    return f"HP:{digits}"
+
+
+class HpoIndex:
+    """An in-memory, searchable index of HPO phenotype terms."""
+
+    name = "hpo-local"
+
+    def __init__(self, terms: dict[str, HpoTerm], source: str):
+        self.source = source
+        self.terms: dict[str, HpoTerm] = terms
+        # Parallel arrays for fast fuzzy scanning: searchable string -> term id.
+        self._names: list[str] = []
+        self._owner: list[str] = []  # term id for _names[i]
+        self._kind: list[str] = []  # "label" or "synonym"
+        self._exact: dict[str, list[tuple[str, str]]] = {}  # lower text -> [(id, kind)]
+        for t in terms.values():
+            if t.obsolete:
+                continue
+            self._add_name(t.id, t.label, "label")
+            for syn in t.synonyms:
+                self._add_name(t.id, syn, "synonym")
+
+    def _add_name(self, term_id: str, text: str, kind: str) -> None:
+        text = (text or "").strip()
+        if not text:
+            return
+        self._names.append(text)
+        self._owner.append(term_id)
+        self._kind.append(kind)
+        self._exact.setdefault(text.lower(), []).append((term_id, kind))
+
+    # ------------------------------------------------------------------ lookup
+    def get(self, curie: str) -> HpoTerm | None:
+        c = normalise_curie(curie)
+        return self.terms.get(c) if c else None
+
+    def __len__(self) -> int:
+        return len(self.terms)
+
+    # ------------------------------------------------------------------ search
+    def search(self, query: str, limit: int = 8) -> list[Candidate]:
+        query = (query or "").strip()
+        if not query:
+            return []
+        scored: dict[str, Candidate] = {}
+
+        def consider(term_id: str, score: float, matched_on: str, kind: str) -> None:
+            t = self.terms.get(term_id)
+            if t is None or t.obsolete:
+                return
+            tag = "label" if kind == "label" else f"synonym: {matched_on}"
+            prev = scored.get(term_id)
+            if prev is None or score > prev.score:
+                scored[term_id] = Candidate(
+                    id=t.id,
+                    label=t.label,
+                    score=round(float(score), 1),
+                    matched_on=tag,
+                    definition=t.definition,
+                    source="hpo-local",
+                )
+
+        # 1) Exact (case-insensitive) label / synonym hits get a strong score.
+        for term_id, kind in self._exact.get(query.lower(), []):
+            consider(term_id, 100.0 if kind == "label" else 98.0, query, kind)
+
+        # 2) Fuzzy scan across all label + synonym strings.
+        if _HAVE_RAPIDFUZZ:
+            matches = process.extract(
+                query, self._names, scorer=fuzz.WRatio, limit=limit * 6
+            )
+            for matched_text, score, idx in matches:
+                consider(self._owner[idx], score, matched_text, self._kind[idx])
+        else:  # difflib fallback
+            ratios = (
+                (difflib.SequenceMatcher(None, query.lower(), n.lower()).ratio(), i)
+                for i, n in enumerate(self._names)
+            )
+            top = sorted(ratios, reverse=True)[: limit * 6]
+            for ratio, idx in top:
+                consider(
+                    self._owner[idx], ratio * 100.0, self._names[idx], self._kind[idx]
+                )
+
+        ranked = sorted(scored.values(), key=lambda c: c.score, reverse=True)
+        return ranked[:limit]
+
+    # ----------------------------------------------------------- construction
+    @classmethod
+    def load(
+        cls,
+        path: str | os.PathLike | None = None,
+        *,
+        allow_download: bool = True,
+        update: bool = False,
+        cache_dir: str | os.PathLike | None = None,
+        log=lambda *a, **k: None,
+    ) -> "HpoIndex":
+        """Load HPO from *path*, the cache, or by downloading the release."""
+        cache = Path(cache_dir) if cache_dir else default_cache_dir()
+        cache.mkdir(parents=True, exist_ok=True)
+
+        src_path: Path
+        if path is not None:
+            src_path = Path(path)
+            if not src_path.exists():
+                raise FileNotFoundError(f"HPO file not found: {src_path}")
+        else:
+            src_path = cache / "hp.obo"
+            if update or not src_path.exists():
+                if not allow_download:
+                    raise FileNotFoundError(
+                        "No local HPO file and downloading is disabled. "
+                        "Pass --hpo-file or allow downloading."
+                    )
+                log(f"Downloading HPO from {HPO_OBO_URL} (one-time) ...")
+                _download(HPO_OBO_URL, src_path)
+                log(f"Saved HPO to {src_path}")
+
+        # Use a parsed-index pickle cache keyed on source size+mtime.
+        stat = src_path.stat()
+        key = f"{src_path.name}-{stat.st_size}-{int(stat.st_mtime)}.idx.pkl"
+        idx_cache = cache / key
+        if idx_cache.exists() and not update:
+            try:
+                with idx_cache.open("rb") as fh:
+                    terms = pickle.load(fh)
+                log(f"Loaded {len(terms)} HPO terms from cache.")
+                return cls(terms, source=str(src_path))
+            except Exception:
+                pass  # fall through and re-parse
+
+        log(f"Parsing HPO from {src_path} ...")
+        terms = _parse_file(src_path)
+        log(f"Indexed {len(terms)} HPO terms.")
+        try:
+            with idx_cache.open("wb") as fh:
+                pickle.dump(terms, fh, protocol=pickle.HIGHEST_PROTOCOL)
+        except Exception:
+            pass
+        return cls(terms, source=str(src_path))
+
+
+# --------------------------------------------------------------------- parsing
+def _download(url: str, dest: Path, timeout: int = 120) -> None:
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    tmp = dest.with_suffix(dest.suffix + ".part")
+    req = urllib.request.Request(url, headers={"User-Agent": "hponorm/1.0"})
+    with urllib.request.urlopen(req, timeout=timeout) as resp, tmp.open("wb") as out:
+        while True:
+            chunk = resp.read(1 << 16)
+            if not chunk:
+                break
+            out.write(chunk)
+    tmp.replace(dest)
+
+
+def _parse_file(path: Path) -> dict[str, HpoTerm]:
+    head = path.open("rb").read(256).lstrip()
+    if head[:1] == b"{":
+        return _parse_obographs(path)
+    return _parse_obo(path)
+
+
+def _parse_obo(path: Path) -> dict[str, HpoTerm]:
+    terms: dict[str, HpoTerm] = {}
+    cur: dict | None = None
+    in_term = False
+
+    def flush(block: dict | None) -> None:
+        if not block:
+            return
+        tid = block.get("id")
+        if not tid or not _CURIE_RE.match(tid):
+            return
+        terms[tid] = HpoTerm(
+            id=tid,
+            label=block.get("name", tid),
+            synonyms=block.get("synonyms", []),
+            definition=block.get("def"),
+            obsolete=block.get("obsolete", False),
+        )
+
+    syn_re = re.compile(r'synonym:\s*"((?:[^"\\]|\\.)*)"')
+    def_re = re.compile(r'def:\s*"((?:[^"\\]|\\.)*)"')
+    with path.open("r", encoding="utf-8", errors="replace") as fh:
+        for line in fh:
+            line = line.rstrip("\n")
+            if line.startswith("[Term]"):
+                flush(cur)
+                cur = {"synonyms": []}
+                in_term = True
+                continue
+            if line.startswith("[") and line.endswith("]"):
+                flush(cur)
+                cur = None
+                in_term = False
+                continue
+            if not in_term or cur is None:
+                continue
+            if line.startswith("id:"):
+                cur["id"] = line[3:].strip()
+            elif line.startswith("name:"):
+                cur["name"] = line[5:].strip()
+            elif line.startswith("def:"):
+                m = def_re.match(line)
+                if m:
+                    cur["def"] = _unescape(m.group(1))
+            elif line.startswith("synonym:"):
+                m = syn_re.match(line)
+                if m:
+                    cur["synonyms"].append(_unescape(m.group(1)))
+            elif line.startswith("is_obsolete:") and line.split(":", 1)[1].strip() == "true":
+                cur["obsolete"] = True
+    flush(cur)
+    return terms
+
+
+def _parse_obographs(path: Path) -> dict[str, HpoTerm]:
+    with path.open("r", encoding="utf-8", errors="replace") as fh:
+        data = json.load(fh)
+    terms: dict[str, HpoTerm] = {}
+    for graph in data.get("graphs", []):
+        for node in graph.get("nodes", []):
+            m = _IRI_RE.search(node.get("id", ""))
+            if not m:
+                continue
+            tid = f"HP:{m.group(1)}"
+            meta = node.get("meta", {}) or {}
+            synonyms = [s.get("val", "") for s in meta.get("synonyms", []) if s.get("val")]
+            definition = (meta.get("definition") or {}).get("val")
+            terms[tid] = HpoTerm(
+                id=tid,
+                label=node.get("lbl") or tid,
+                synonyms=synonyms,
+                definition=definition,
+                obsolete=bool(meta.get("deprecated", False)),
+            )
+    return terms
+
+
+def _unescape(s: str) -> str:
+    return s.replace('\\"', '"').replace("\\n", " ").replace("\\\\", "\\").strip()

hponorm-1.0.0/hponorm/ols.py

@@ -0,0 +1,112 @@
+"""EBI Ontology Lookup Service (OLS4) backend for HPO.
+
+Provides high-quality ranked search over the live HPO ontology without any
+local download. Used when the machine is online; otherwise the CLI falls back
+to the local :class:`~hponorm.hpo.HpoIndex`.
+
+OLS4 API docs: https://www.ebi.ac.uk/ols4/help
+"""
+from __future__ import annotations
+
+import json
+import urllib.parse
+import urllib.request
+
+from .suggest import Candidate
+
+OLS_BASE = "https://www.ebi.ac.uk/ols4/api"
+
+
+class OlsClient:
+    name = "ols"
+
+    def __init__(self, base: str = OLS_BASE, timeout: int = 15):
+        self.base = base.rstrip("/")
+        self.timeout = timeout
+
+    # ------------------------------------------------------------------ utils
+    def _get_json(self, url: str) -> dict:
+        req = urllib.request.Request(url, headers={"User-Agent": "hponorm/1.0"})
+        with urllib.request.urlopen(req, timeout=self.timeout) as resp:
+            return json.loads(resp.read().decode("utf-8"))
+
+    def available(self) -> bool:
+        try:
+            self._get_json(f"{self.base}/ontologies/hp")
+            return True
+        except Exception:
+            return False
+
+    # ----------------------------------------------------------------- search
+    def search(self, query: str, limit: int = 8) -> list[Candidate]:
+        query = (query or "").strip()
+        if not query:
+            return []
+        params = urllib.parse.urlencode(
+            {
+                "q": query,
+                "ontology": "hp",
+                "type": "class",
+                "rows": max(limit, 10),
+                "fieldList": "obo_id,label,description,synonym,is_obsolete",
+            }
+        )
+        data = self._get_json(f"{self.base}/search?{params}")
+        docs = (data.get("response") or {}).get("docs", [])
+        out: list[Candidate] = []
+        seen: set[str] = set()
+        # OLS returns docs in relevance order; map that to a descending score.
+        for rank, doc in enumerate(docs):
+            obo_id = doc.get("obo_id")
+            if not obo_id or not obo_id.startswith("HP:"):
+                continue
+            if doc.get("is_obsolete"):
+                continue
+            if obo_id in seen:
+                continue
+            seen.add(obo_id)
+            desc = doc.get("description")
+            if isinstance(desc, list):
+                desc = desc[0] if desc else None
+            out.append(
+                Candidate(
+                    id=obo_id,
+                    label=doc.get("label") or obo_id,
+                    score=round(max(1.0, 100.0 - rank * 4), 1),
+                    matched_on="OLS relevance",
+                    definition=desc,
+                    source="ols",
+                )
+            )
+            if len(out) >= limit:
+                break
+        return out
+
+    # ------------------------------------------------------------------- term
+    def get(self, curie: str) -> Candidate | None:
+        from .hpo import normalise_curie
+
+        c = normalise_curie(curie)
+        if not c:
+            return None
+        iri = f"http://purl.obolibrary.org/obo/{c.replace(':', '_')}"
+        params = urllib.parse.urlencode({"iri": iri})
+        try:
+            data = self._get_json(f"{self.base}/ontologies/hp/terms?{params}")
+        except Exception:
+            return None
+        terms = (data.get("_embedded") or {}).get("terms", [])
+        if not terms:
+            return None
+        t = terms[0]
+        desc = t.get("description")
+        if isinstance(desc, list):
+            desc = desc[0] if desc else None
+        return Candidate(
+            id=c,
+            label=t.get("label") or c,
+            score=100.0,
+            matched_on="exact id",
+            definition=desc,
+            source="ols",
+        )

hponorm-1.0.0/hponorm/phenopackets.py

@@ -0,0 +1,86 @@
+"""Read a phenopacket / family JSON, locate every phenotypic-feature ``type``,
+apply chosen HPO mappings in place, and write the normalised file back out.
+
+The finder is deliberately schema-tolerant: it recursively walks the document
+and collects the ``type`` object of every entry under any ``phenotypicFeatures``
+array. That covers the GA4GH ``Phenopacket.phenotypicFeatures`` and
+``Family.proband`` / ``Family.relatives`` layouts (and any nesting of them)
+without hard-coding paths. An entry's ``excluded`` flag is left untouched -- it
+qualifies the feature, not the term, so the ``type`` is mapped either way.
+"""
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from pathlib import Path
+
+
+@dataclass
+class FeatureRef:
+    """A live, mutable reference to one phenotypic-feature ``type`` object."""
+
+    path: str
+    type: dict  # the actual {"id": ..., "label": ...} dict inside the loaded JSON
+    excluded: bool = False  # GA4GH PhenotypicFeature.excluded (informational)
+
+    @property
+    def label(self) -> str:
+        return (self.type.get("label") or "").strip()
+
+    @property
+    def id(self) -> str:
+        return (self.type.get("id") or "").strip()
+
+    def apply(self, curie: str, label: str) -> None:
+        self.type["id"] = curie
+        self.type["label"] = label
+
+
+def load(path: str | Path) -> dict:
+    with Path(path).open("r", encoding="utf-8") as fh:
+        return json.load(fh)
+
+
+def find_phenotypic_features(obj, path: str = "$") -> list[FeatureRef]:
+    """Recursively collect every phenotypic-feature ``type`` dict in *obj*."""
+    found: list[FeatureRef] = []
+
+    def walk(node, p):
+        if isinstance(node, dict):
+            for key, value in node.items():
+                if key == "phenotypicFeatures" and isinstance(value, list):
+                    for i, entry in enumerate(value):
+                        if isinstance(entry, dict) and isinstance(
+                            entry.get("type"), dict
+                        ):
+                            found.append(
+                                FeatureRef(
+                                    f"{p}.phenotypicFeatures[{i}].type",
+                                    entry["type"],
+                                    bool(entry.get("excluded", False)),
+                                )
+                            )
+                        walk(entry, f"{p}.phenotypicFeatures[{i}]")
+                else:
+                    walk(value, f"{p}.{key}")
+        elif isinstance(node, list):
+            for i, item in enumerate(node):
+                walk(item, f"{p}[{i}]")
+
+    walk(obj, path)
+    return found
+
+
+def group_by_label(refs: list[FeatureRef]) -> "dict[tuple[str, str], list[FeatureRef]]":
+    """Group feature references by their current (label, id) so each distinct
+    label is reviewed once and the decision applied to every occurrence."""
+    groups: dict[tuple[str, str], list[FeatureRef]] = {}
+    for ref in refs:
+        groups.setdefault((ref.label, ref.id), []).append(ref)
+    return groups
+
+
+def save(data: dict, path: str | Path) -> None:
+    with Path(path).open("w", encoding="utf-8") as fh:
+        json.dump(data, fh, indent=2, ensure_ascii=False)
+        fh.write("\n")

hponorm-1.0.0/hponorm/suggest.py

@@ -0,0 +1,75 @@
+"""Unified suggestion layer.
+
+A ``Suggester`` wraps one of two interchangeable backends:
+
+* ``hpo``  - a local :class:`~hponorm.hpo.HpoIndex` (offline capable)
+* ``ols``  - the EBI Ontology Lookup Service v4 (online, best ranking)
+
+and exposes a backend-agnostic API used by the CLI:
+
+* ``suggest(label)``    -> ranked list of :class:`Candidate`
+* ``resolve(curie)``    -> validate a specific HPO id and return its canonical term
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass
+class Candidate:
+    """A single suggested ontology term."""
+
+    id: str  # e.g. "HP:0001657"
+    label: str  # canonical HPO label
+    score: float  # 0-100 match score (higher is better)
+    matched_on: str  # "label" or 'synonym: "..."'
+    definition: str | None = None
+    source: str = ""  # which backend produced it
+
+    def short_def(self, n: int = 90) -> str:
+        if not self.definition:
+            return ""
+        d = " ".join(self.definition.split())
+        return d if len(d) <= n else d[: n - 1] + "\u2026"
+
+
+class Suggester:
+    def __init__(self, backend, *, fallback=None, name: str = ""):
+        self.backend = backend
+        self.fallback = fallback
+        self.name = name or getattr(backend, "name", "backend")
+
+    def suggest(self, label: str, limit: int = 8) -> list[Candidate]:
+        try:
+            results = self.backend.search(label, limit=limit)
+        except Exception:
+            results = []
+        if not results and self.fallback is not None:
+            try:
+                results = self.fallback.search(label, limit=limit)
+            except Exception:
+                results = []
+        return results
+
+    def resolve(self, curie: str) -> Candidate | None:
+        """Look up a specific HPO id to validate it and fetch its canonical label."""
+        for be in (self.backend, self.fallback):
+            if be is None:
+                continue
+            try:
+                term = be.get(curie)
+            except Exception:
+                term = None
+            if term is not None:
+                # ``HpoIndex.get`` returns an HpoTerm; OLS returns a Candidate.
+                if isinstance(term, Candidate):
+                    return term
+                return Candidate(
+                    id=term.id,
+                    label=term.label,
+                    score=100.0,
+                    matched_on="exact id",
+                    definition=getattr(term, "definition", None),
+                    source=getattr(be, "name", self.name),
+                )
+        return None

hponorm-1.0.0/hponorm.egg-info/PKG-INFO

@@ -0,0 +1,10 @@
+Metadata-Version: 2.4
+Name: hponorm
+Version: 1.0.0
+Summary: Interactively validate and normalise GA4GH phenopacket phenotypic-feature terms against the HPO ontology
+Requires-Python: >=3.9
+License-File: LICENSE
+Provides-Extra: recommended
+Requires-Dist: rapidfuzz>=3.0; extra == "recommended"
+Requires-Dist: rich>=13.0; extra == "recommended"
+Dynamic: license-file

hponorm-1.0.0/hponorm.egg-info/SOURCES.txt

@@ -0,0 +1,14 @@
+LICENSE
+README.md
+pyproject.toml
+hponorm/cli.py
+hponorm/hpo.py
+hponorm/ols.py
+hponorm/phenopackets.py
+hponorm/suggest.py
+hponorm.egg-info/PKG-INFO
+hponorm.egg-info/SOURCES.txt
+hponorm.egg-info/dependency_links.txt
+hponorm.egg-info/entry_points.txt
+hponorm.egg-info/requires.txt
+hponorm.egg-info/top_level.txt

hponorm-1.0.0/hponorm.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

hponorm-1.0.0/hponorm.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+hponorm = hponorm.cli:main

hponorm-1.0.0/hponorm.egg-info/requires.txt

@@ -0,0 +1,4 @@
+
+[recommended]
+rapidfuzz>=3.0
+rich>=13.0

hponorm-1.0.0/hponorm.egg-info/top_level.txt

@@ -0,0 +1 @@
+hponorm

hponorm-1.0.0/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hponorm"
+version = "1.0.0"
+description = "Interactively validate and normalise GA4GH phenopacket phenotypic-feature terms against the HPO ontology"
+requires-python = ">=3.9"
+dependencies = []  # stdlib-only by default
+
+[project.optional-dependencies]
+recommended = ["rapidfuzz>=3.0", "rich>=13.0"]
+
+[project.scripts]
+hponorm = "hponorm.cli:main"
+
+[tool.setuptools]
+packages = ["hponorm"]
+
+[tool.setuptools.package-data]
+hponorm = ["data/*.obo"]

hponorm-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+