allelix 2.1.2__tar.gz → 2.2.1__tar.gz

{allelix-2.1.2 → allelix-2.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: allelix
-Version: 2.1.2
+Version: 2.2.1
 Summary: Open-source genotype analysis toolkit. Format-agnostic ingestion, database-agnostic annotation, offline-first.
 Author: Allelix
 Maintainer-email: dial481 <dial481@users.noreply.github.com>
@@ -95,6 +95,7 @@ See [Development](#development) for source installs and running tests, [Managing
 | MyHeritage DNA | ✓ | CSV, same structure as FTDNA. Detected by "MyHeritage" in comment header. Handles double-double-quoted field variant. |
 | Living DNA | ✓ | Tab-delimited despite `.csv` extension. Handles AX-, AFFX-prefixed and CHR:POS positional SNP IDs. |
 | FTDNA Illumina raw | ✓ | Tab-delimited variant of the FTDNA export (distinct from the CSV format above). `RSID/CHROMOSOME/POSITION/RESULT` columns. Build 37 default. |
+| FTDNA FamFinder | ✓ | Third FTDNA file shape: tab-delimited with **separate `ALLELE1` / `ALLELE2` columns** instead of a concatenated `RESULT`. Detected by the `famfinder` substring in the file header plus the 5-column canonical header. Build 37 default. |
 | VCF / gVCF | ✓ | REF/ALT encoding, `0/1` genotype notation. Plain VCF: absence at a position means reference. gVCF: explicit reference blocks (lines with `<NON_REF>` ALT and `END=` INFO) are skipped — they match nothing in any annotation database. Multi-sample files require `--sample <ID>`. Streams via stdlib; `.vcf.gz` handled transparently. Optional `pip install allelix[vcf-index]` enables pysam-backed tabix random access for fast `extract --snps` on huge VCFs. |
 
 Adding a new format means adding one file to `allelix/parsers/` and registering an instance in the `PARSERS` list in `allelix/parsers/__init__.py`.
@@ -253,28 +254,44 @@ Allelix stores reference database caches at `~/.local/share/allelix/` (Linux/mac
 **The caches are disposable and re-downloadable.** Everything here was fetched by `allelix db update` and can be re-fetched at any time. There's no per-user state in the cache — your genotype files never enter this directory. Safe to delete to reclaim space (or to force a full refresh if you suspect a cache went stale):
 
 ```bash
-# Reclaim all ~16-22 GB
-rm -rf ~/.local/share/allelix/
+# Reclaim all ~16-22 GB, with size report and confirmation prompt
+allelix db clean
 
-# Reclaim everything except CADD (avoids re-downloading the slowest one)
-find ~/.local/share/allelix/ -mindepth 1 ! -name 'cadd*.sqlite*' -delete
+# Preserve the CADD cache (avoids re-downloading the slowest one, ~5.8 GB)
+allelix db clean --keep-cadd
+
+# Show what would be deleted without acting
+allelix db clean --dry-run
+
+# Scripted use — skip the confirmation prompt
+allelix db clean --yes
 
 # Re-populate from scratch next time you analyze a file
 allelix db update
 ```
 
-**The one thing worth backing up: `config.toml`.** It lives separately (XDG config dir, typically `~/.config/allelix/config.toml`) and stores your license assertions (commercial-mode toggle, CADD opt-in confirmation) and per-source enable/disable settings. It's the only non-reproducible state in the install. The caches will rebuild on the next `db update`; `config.toml` won't.
+`db clean` refuses to act on a directory that doesn't contain any recognized allelix cache files — a typo'd `--data-dir ~/Documents` won't result in `rm -rf` against an unrelated tree, even with `--yes`. Pass `--force` if you've staged caches under a non-standard name and want to delete them anyway.
+
+For scripting and backup integration, `allelix db path` prints the resolved data directory:
+
+```bash
+ALLELIX_DATA=$(allelix db path)
+du -sh "$ALLELIX_DATA"
+
+# --check additionally verifies the path is writable; exit non-zero if not
+allelix db path --check
+```
+
+**The one thing worth backing up: `config.toml`.** It lives separately (XDG config dir, typically `~/.config/allelix/config.toml`) and stores your license assertions (commercial-mode toggle, CADD opt-in confirmation) and per-source enable/disable settings. It's the only non-reproducible state in the install. The caches will rebuild on the next `db update`; `config.toml` won't. `allelix db clean` preserves `config.toml` even when it lives inside the data directory (older installs).
 
 To fully uninstall allelix and reclaim everything:
 
 ```bash
 pip uninstall allelix
-rm -rf ~/.local/share/allelix/    # reference database caches
+allelix db clean --yes            # reference database caches
 rm -rf ~/.config/allelix/         # license + source-toggle config
 ```
 
-A `allelix db clean` subcommand to automate the cache-clear step (with confirmation prompt and size report) is planned for v2.2 alongside `allelix db path` (prints the resolved data directory for scripting / backup integration).
-
 ## Data Sources & Licensing
 
 Allelix source code is licensed under the **GNU Affero General Public License v3.0 or later** (AGPL-3.0-or-later). Allelix ships with **zero third-party data**. All reference databases are downloaded by the user at runtime via `allelix db update`. Each database retains its original license on the user's machine:
@@ -383,6 +400,8 @@ pytest
 
 The pre-commit hook enforces `ruff check` + `ruff format --check`. If a commit is blocked, fix the underlying problem rather than skipping the hook.
 
+For the full real-data release-validation battery — every parser format, VCF/gVCF, PLINK round-trip, edge cases, the GIAB/1000G corpus — see [`test_data/FULL_TEST_PROTOCOL.md`](test_data/FULL_TEST_PROTOCOL.md). That document is the project's real-data testing story; running `pytest` covers the fast suite that gates every commit.
+
 ## License
 
 GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later). See `LICENSE`.

{allelix-2.1.2 → allelix-2.2.1}/README.md

@@ -56,6 +56,7 @@ See [Development](#development) for source installs and running tests, [Managing
 | MyHeritage DNA | ✓ | CSV, same structure as FTDNA. Detected by "MyHeritage" in comment header. Handles double-double-quoted field variant. |
 | Living DNA | ✓ | Tab-delimited despite `.csv` extension. Handles AX-, AFFX-prefixed and CHR:POS positional SNP IDs. |
 | FTDNA Illumina raw | ✓ | Tab-delimited variant of the FTDNA export (distinct from the CSV format above). `RSID/CHROMOSOME/POSITION/RESULT` columns. Build 37 default. |
+| FTDNA FamFinder | ✓ | Third FTDNA file shape: tab-delimited with **separate `ALLELE1` / `ALLELE2` columns** instead of a concatenated `RESULT`. Detected by the `famfinder` substring in the file header plus the 5-column canonical header. Build 37 default. |
 | VCF / gVCF | ✓ | REF/ALT encoding, `0/1` genotype notation. Plain VCF: absence at a position means reference. gVCF: explicit reference blocks (lines with `<NON_REF>` ALT and `END=` INFO) are skipped — they match nothing in any annotation database. Multi-sample files require `--sample <ID>`. Streams via stdlib; `.vcf.gz` handled transparently. Optional `pip install allelix[vcf-index]` enables pysam-backed tabix random access for fast `extract --snps` on huge VCFs. |
 
 Adding a new format means adding one file to `allelix/parsers/` and registering an instance in the `PARSERS` list in `allelix/parsers/__init__.py`.
@@ -214,28 +215,44 @@ Allelix stores reference database caches at `~/.local/share/allelix/` (Linux/mac
 **The caches are disposable and re-downloadable.** Everything here was fetched by `allelix db update` and can be re-fetched at any time. There's no per-user state in the cache — your genotype files never enter this directory. Safe to delete to reclaim space (or to force a full refresh if you suspect a cache went stale):
 
 ```bash
-# Reclaim all ~16-22 GB
-rm -rf ~/.local/share/allelix/
+# Reclaim all ~16-22 GB, with size report and confirmation prompt
+allelix db clean
 
-# Reclaim everything except CADD (avoids re-downloading the slowest one)
-find ~/.local/share/allelix/ -mindepth 1 ! -name 'cadd*.sqlite*' -delete
+# Preserve the CADD cache (avoids re-downloading the slowest one, ~5.8 GB)
+allelix db clean --keep-cadd
+
+# Show what would be deleted without acting
+allelix db clean --dry-run
+
+# Scripted use — skip the confirmation prompt
+allelix db clean --yes
 
 # Re-populate from scratch next time you analyze a file
 allelix db update
 ```
 
-**The one thing worth backing up: `config.toml`.** It lives separately (XDG config dir, typically `~/.config/allelix/config.toml`) and stores your license assertions (commercial-mode toggle, CADD opt-in confirmation) and per-source enable/disable settings. It's the only non-reproducible state in the install. The caches will rebuild on the next `db update`; `config.toml` won't.
+`db clean` refuses to act on a directory that doesn't contain any recognized allelix cache files — a typo'd `--data-dir ~/Documents` won't result in `rm -rf` against an unrelated tree, even with `--yes`. Pass `--force` if you've staged caches under a non-standard name and want to delete them anyway.
+
+For scripting and backup integration, `allelix db path` prints the resolved data directory:
+
+```bash
+ALLELIX_DATA=$(allelix db path)
+du -sh "$ALLELIX_DATA"
+
+# --check additionally verifies the path is writable; exit non-zero if not
+allelix db path --check
+```
+
+**The one thing worth backing up: `config.toml`.** It lives separately (XDG config dir, typically `~/.config/allelix/config.toml`) and stores your license assertions (commercial-mode toggle, CADD opt-in confirmation) and per-source enable/disable settings. It's the only non-reproducible state in the install. The caches will rebuild on the next `db update`; `config.toml` won't. `allelix db clean` preserves `config.toml` even when it lives inside the data directory (older installs).
 
 To fully uninstall allelix and reclaim everything:
 
 ```bash
 pip uninstall allelix
-rm -rf ~/.local/share/allelix/    # reference database caches
+allelix db clean --yes            # reference database caches
 rm -rf ~/.config/allelix/         # license + source-toggle config
 ```
 
-A `allelix db clean` subcommand to automate the cache-clear step (with confirmation prompt and size report) is planned for v2.2 alongside `allelix db path` (prints the resolved data directory for scripting / backup integration).
-
 ## Data Sources & Licensing
 
 Allelix source code is licensed under the **GNU Affero General Public License v3.0 or later** (AGPL-3.0-or-later). Allelix ships with **zero third-party data**. All reference databases are downloaded by the user at runtime via `allelix db update`. Each database retains its original license on the user's machine:
@@ -344,6 +361,8 @@ pytest
 
 The pre-commit hook enforces `ruff check` + `ruff format --check`. If a commit is blocked, fix the underlying problem rather than skipping the hook.
 
+For the full real-data release-validation battery — every parser format, VCF/gVCF, PLINK round-trip, edge cases, the GIAB/1000G corpus — see [`test_data/FULL_TEST_PROTOCOL.md`](test_data/FULL_TEST_PROTOCOL.md). That document is the project's real-data testing story; running `pytest` covers the fast suite that gates every commit.
+
 ## License
 
 GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later). See `LICENSE`.

{allelix-2.1.2 → allelix-2.2.1}/allelix/annotators/clinvar.py

@@ -25,7 +25,7 @@ from allelix.databases.manager import (
     download,
     fetch_remote_text,
     get_database_info,
-    load_clinvar_vcf,
+    load_clinvar_tsv,
     stamp_existing_clinvar_cache,
     verify_file_hash,
 )
@@ -92,9 +92,99 @@ def _magnitude(clnsig: str) -> float:
     return _CLNSIG_MAGNITUDE.get(_normalize_clnsig(clnsig), 5.0)
 
 
-def _vcf_filename_for_url(url: str) -> str:
-    """Pick the right local filename suffix based on the URL."""
-    return "clinvar.vcf.gz" if url.endswith(".gz") else "clinvar.vcf"
+# GH #109: ClinVar review-status ranking — lower index = stronger
+# evidence per ClinVar's star-rating documentation. Used to pick the
+# winning row when N SCVs with identical (significance, condition,
+# gene) tuples collapse during per-SCV dedup; the surviving Annotation
+# carries the highest-rated review_status across the group instead of
+# silently dropping provenance.
+#
+# Real cache rows use the comma-and-underscore form
+# (`criteria_provided,_multiple_submitters,_no_conflicts`). The rank
+# helper normalizes via _normalize_clnsig so it's robust to spacing
+# variation. Unknown values fall through to "weakest known."
+_REVIEW_STATUS_RANK: tuple[str, ...] = (
+    "practice_guideline",
+    "reviewed_by_expert_panel",
+    "criteria_provided,_multiple_submitters,_no_conflicts",
+    "criteria_provided,_conflicting_classifications",
+    "criteria_provided,_conflicting_interpretations",
+    "criteria_provided,_single_submitter",
+    "no_assertion_criteria_provided",
+    "no_assertion_provided",
+    "no_classification_provided",
+)
+
+
+def _review_status_rank(status: str) -> int:
+    """Return the rank index for a review_status string.
+
+    Lower index = stronger evidence. Unknown / unrecognized values get
+    the max rank (weakest), so a known SCV always beats an unknown one
+    when collapsing.
+    """
+    if not status:
+        return len(_REVIEW_STATUS_RANK)
+    normalized = _normalize_clnsig(status)
+    for i, candidate in enumerate(_REVIEW_STATUS_RANK):
+        if candidate == normalized:
+            return i
+    return len(_REVIEW_STATUS_RANK)
+
+
+def _collapse_identical_scvs(annotations: list[Annotation]) -> list[Annotation]:
+    """Collapse identical per-SCV rows from the #42 TSV loader (GH #109).
+
+    Two Annotations are identical for dedup purposes if they share
+    (rsid, significance, condition, gene, alt). The surviving row carries:
+
+      - the strongest ``review_status`` across the collapsed group
+        (per ``_REVIEW_STATUS_RANK``)
+      - the union of ``references`` across the collapsed group
+        (preserving first-seen order — never silently drops provenance)
+
+    Distinct (significance, condition) pairs on the same variant are
+    preserved as separate rows — that's exactly what #42 was built to
+    surface, and collapsing them would be the Frankenstein-pair
+    regression #42 retired.
+
+    GH #109 cross-PR review (Finding 1): ``alt`` is in the key for the
+    #18 wrong-allele safety invariant. Two SCVs at a multi-allelic site
+    can share (significance, condition, gene) but differ in alt — e.g.
+    rs1065852 (G→A drug_response/Codeine_response + G→C drug_response/
+    Codeine_response would be a hypothetical). Collapsing across alts
+    would (a) drop a distinct-allele annotation and (b) mis-attach the
+    surviving row's ``alt`` field to the merged result, breaking the
+    downstream exact-(rsid, alt) lookup that gnomAD / AlphaMissense /
+    CADD use for allele-specific enrichment. The pre-#109 v2.2.0 code
+    emitted one Annotation per cache row, so this case was always
+    handled correctly; the post-#109 dedup must preserve that property.
+
+    Input order is preserved for the kept rows: each group is replaced
+    by its winner at the position of the group's first occurrence.
+    """
+    from dataclasses import replace
+
+    groups: dict[tuple[str, str, str, str, str], list[Annotation]] = {}
+    for a in annotations:
+        key = (a.rsid, a.significance, a.condition, a.gene, a.alt)
+        groups.setdefault(key, []).append(a)
+
+    out: list[Annotation] = []
+    for group in groups.values():
+        if len(group) == 1:
+            out.append(group[0])
+            continue
+        winner = min(group, key=lambda a: _review_status_rank(a.review_status))
+        seen_refs: set[str] = set()
+        merged_refs: list[str] = []
+        for a in group:
+            for ref in a.references:
+                if ref not in seen_refs:
+                    seen_refs.add(ref)
+                    merged_refs.append(ref)
+        out.append(replace(winner, references=merged_refs))
+    return out
 
 
 class ClinVarAnnotator(Annotator):
@@ -186,39 +276,88 @@ class ClinVarAnnotator(Annotator):
         return self._conns[build]
 
     def setup(self) -> None:
-        """Download each managed build's ClinVar VCF and ingest atomically."""
-        for build in self._builds:
-            self._setup_one(build)
-
-    def _setup_one(self, build: str) -> None:
-        url = _manager_module.CLINVAR_URL_BY_BUILD[build]
-        signal = self._fetch_remote_signal_for(build)
+        """Download the per-SCV TSV pair once and ingest both builds from it.
+
+        v2.2 #42 stage B: the VCF-per-build path is replaced by the
+        variant_summary + submission_summary TSV pair. variant_summary
+        carries per-(VariationID, Assembly) rows so the same downloaded
+        file populates both GRCh37 and GRCh38 caches with no second
+        download. submission_summary is build-agnostic; one fetch
+        applies across builds.
+
+        Freshness signal is derived from submission_summary's md5
+        endpoint — that file drives the meaningful change cadence
+        (per-SCV submissions land daily) and variant_summary turns over
+        weekly on a slower clock.
+        """
+        signal = self._fetch_remote_signal_for_tsv()
         if signal is None:
             msg = (
-                f"clinvar ({build}): cannot verify remote freshness signal. "
+                "clinvar: cannot verify remote freshness signal. "
                 "Refresh aborted to avoid persisting an incomplete cache stamp. "
                 "Retry, or pass --force if you accept that next `db update` "
                 "will re-download to re-establish the signal."
             )
             raise RuntimeError(msg)
-        vcf_path = self.data_dir / _vcf_filename_for_url(url)
-        download(url, vcf_path)
+
+        vs_url = _manager_module.CLINVAR_VARIANT_SUMMARY_URL
+        ss_url = _manager_module.CLINVAR_SUBMISSION_SUMMARY_URL
+        vs_path = self.data_dir / "variant_summary.txt.gz"
+        ss_path = self.data_dir / "submission_summary.txt.gz"
+
+        download(vs_url, vs_path)
+        download(ss_url, ss_path)
+        # We don't md5-verify variant_summary; signal is calibrated against
+        # submission_summary (the more frequently-updated file) and that
+        # endpoint's md5 is what we hashed below. variant_summary integrity
+        # is asserted structurally by the loader (rows that fail to parse
+        # are skipped, mismatched assemblies are dropped).
         try:
-            verify_file_hash(vcf_path, "md5", signal.removeprefix("md5:"))
-            load_clinvar_vcf(
-                vcf_path,
-                self._db_paths[build],
-                source_url=url,
-                remote_signal=signal,
-                record_name=clinvar_record_name(build),
-            )
+            verify_file_hash(ss_path, "md5", signal.removeprefix("md5:"))
+            for build in self._builds:
+                self._setup_one_from_tsv(vs_path, ss_path, build, signal, vs_url)
         finally:
-            try:
-                vcf_path.unlink()
-            except FileNotFoundError:
-                pass
-            except OSError:
-                logger.warning("Could not remove staged VCF at %s", vcf_path)
+            for tmp in (vs_path, ss_path):
+                try:
+                    tmp.unlink()
+                except FileNotFoundError:
+                    pass
+                except OSError:
+                    logger.warning("Could not remove staged TSV at %s", tmp)
+
+    def _setup_one_from_tsv(
+        self,
+        vs_path: Path,
+        ss_path: Path,
+        build: str,
+        signal: str,
+        source_url: str,
+    ) -> None:
+        """Run the per-build cache build against an already-downloaded TSV pair."""
+        load_clinvar_tsv(
+            vs_path,
+            ss_path,
+            self._db_paths[build],
+            build,
+            source_url=source_url,
+            remote_signal=signal,
+        )
+        # load_clinvar_tsv stamps the record name as "clinvar"; for
+        # backward compatibility with the per-build record-name scheme
+        # (clinvar.GRCh37 / clinvar.GRCh38), rename the row in place.
+        # This keeps is_ready() / version() per-build dispatch working
+        # against the existing database_versions table layout.
+        import contextlib
+        import sqlite3
+
+        per_build_name = clinvar_record_name(build)
+        if per_build_name != "clinvar":
+            with contextlib.closing(sqlite3.connect(self._db_paths[build])) as conn:
+                conn.execute(
+                    "UPDATE database_versions SET name = ? WHERE name = 'clinvar'",
+                    (per_build_name,),
+                )
+                conn.commit()
 
     def is_ready(self) -> bool:
         """True iff EVERY managed build has a populated, version-stamped cache.
@@ -300,11 +439,15 @@ class ClinVarAnnotator(Annotator):
         # can't suppress a single-base array readout. The per-build cache
         # may have BOTH SNV and indel rows for the same rsid; the WHERE
         # filters those out so we keep only the SNV REF.
-        rows = conn.execute(
+        #
+        # GH #28 item 1: iterate the cursor directly instead of fetchall()
+        # so peak memory is one row + the dict, not the entire result set
+        # twice.
+        cursor = conn.execute(
             "SELECT DISTINCT rsid, ref FROM clinvar_variants WHERE length(ref) = 1"
-        ).fetchall()
+        )
         out: dict[str, str] = {}
-        for rsid, ref in rows:
+        for rsid, ref in cursor:
             # If a rsid has multiple single-base REFs (shouldn't happen at
             # one position but defending against future data shapes), keep
             # the first.
@@ -313,23 +456,21 @@ class ClinVarAnnotator(Annotator):
         return out
 
     def fetch_remote_signal(self) -> str | None:
-        r"""Composite freshness signal across managed builds.
-
-        Format: `"GRCh37:md5:<hex>|GRCh38:md5:<hex>"`. Returns None if
-        ANY managed build's signal probe fails — the CLI then prints
-        "can't verify" and skips refresh per ADR-0012's policy.
+        r"""Freshness signal derived from the per-SCV submission_summary md5.
+
+        v2.2 #42 stage B: pre-stage-B this returned a composite VCF-md5
+        per build. After stage B, the cache is built from a build-agnostic
+        TSV pair, and submission_summary's md5 endpoint is the change
+        cadence we want to track (per-SCV submissions land daily; the
+        slower-moving variant_summary only adds new VariationIDs).
+        Returns the signal as ``"md5:<hex>"`` to keep ``setup()``'s
+        ``signal.removeprefix("md5:")`` contract unchanged.
         """
-        parts: list[str] = []
-        for build in self._builds:
-            sig = self._fetch_remote_signal_for(build)
-            if sig is None:
-                return None
-            parts.append(f"{build}:{sig}")
-        return "|".join(parts) if parts else None
+        return self._fetch_remote_signal_for_tsv()
 
     @staticmethod
-    def _fetch_remote_signal_for(build: str) -> str | None:
-        body = fetch_remote_text(_manager_module.CLINVAR_URL_BY_BUILD[build] + ".md5")
+    def _fetch_remote_signal_for_tsv() -> str | None:
+        body = fetch_remote_text(_manager_module.CLINVAR_SUBMISSION_SUMMARY_URL + ".md5")
         if not body:
             return None
         first_token = body.strip().split(None, 1)[0] if body.strip() else ""
@@ -338,19 +479,29 @@ class ClinVarAnnotator(Annotator):
             # as a transient signal failure rather than poisoning the
             # cache: callers handle `None` as "freshness unknown, skip"
             # in `db update`, and `setup()` raises rather than passing
-            # garbage to `verify_file_hash` (which would delete the VCF).
+            # garbage to `verify_file_hash` (which would delete the TSV).
             logger.warning(
-                "clinvar(%s): .md5 endpoint returned a body whose first token "
-                "is not a 32-char hex digest (got %r); treating as no signal",
-                build,
+                "clinvar: submission_summary.md5 endpoint returned a body whose "
+                "first token is not a 32-char hex digest (got %r); treating as "
+                "no signal",
                 first_token[:32],
             )
             return None
         return f"md5:{first_token}"
 
     def cached_remote_signal(self) -> str | None:
-        """Composite cached signal across managed builds. None if any missing."""
-        parts: list[str] = []
+        """Return the signal stamped at the last successful setup, or None.
+
+        None if any managed build's cache is absent / unstamped or if the
+        builds' stamps diverge (an interrupted partial reingest).
+
+        Stage B: both builds now share the same source TSVs and therefore
+        the same remote_signal. We still cross-check that EVERY managed
+        build's cache carries the stamp; a missing build means the
+        last-setup didn't complete, treat as "no signal" and let
+        ``db update`` decide.
+        """
+        signals: set[str] = set()
         for build in self._builds:
             info = get_database_info(self._db_paths[build], clinvar_record_name(build))
             if info is None or info["remote_signal"] is None:
@@ -358,8 +509,16 @@ class ClinVarAnnotator(Annotator):
             sig = info["remote_signal"]
             if not sig:
                 return None
-            parts.append(f"{build}:{sig}")
-        return "|".join(parts) if parts else None
+            signals.add(sig)
+        if not signals:
+            return None
+        # All builds should carry the same signal (single download, single
+        # ingest run). If they diverge — e.g. someone reingested one build
+        # against a newer TSV — treat as "no signal" so the next
+        # `db update` re-syncs both.
+        if len(signals) > 1:
+            return None
+        return signals.pop()
 
     def annotate(self, variant: Variant) -> list[Annotation]:
         """Return ClinVar annotations whose REF/ALT matches the user's genotype.
@@ -438,7 +597,11 @@ class ClinVarAnnotator(Annotator):
                     alt=alt,
                 )
             )
-        return annotations
+        # GH #109: collapse identical per-SCV rows from the #42 TSV loader.
+        # Without this, a variant with N identical Pathogenic submissions
+        # emits N identical Annotation objects (visible duplicates on every
+        # surface). Distinct (significance, condition) pairs survive.
+        return _collapse_identical_scvs(annotations)
 
     def bulk_resolve_rsids(self, variants: list[Variant]) -> dict[tuple[str, int, str, str], str]:
         """Resolve rsIDs by (chromosome, position) for variants with no ID.
@@ -572,6 +735,12 @@ class ClinVarAnnotator(Annotator):
                 continue
             user_is_multibase = len(variant.allele1) > 1 or len(variant.allele2) > 1
             user_diploid = _user_diploid(variant)
+            # GH #109: collect per-variant Annotations and dedup before
+            # yielding, mirroring the annotate() collapse. Without this,
+            # bulk callers (the analyze pipeline goes through
+            # batch_annotate) leak the same per-SCV duplicates the
+            # single-variant path used to.
+            variant_annotations: list[Annotation] = []
             for row in rows:
                 (
                     _chrom,
@@ -601,21 +770,24 @@ class ClinVarAnnotator(Annotator):
                     f"{clnsig.replace('_', ' ') if clnsig else 'unknown significance'}"
                 )
                 references = [f"clinvar:allele/{allele_id}"] if allele_id else []
-                yield Annotation(
-                    source=self.name,
-                    rsid=variant.rsid,
-                    significance=f"clinvar_{sig_label}",
-                    category="clinical",
-                    magnitude=_magnitude(clnsig),
-                    description=description,
-                    attribution=self.attribution,
-                    genotype_match=user_diploid,
-                    references=references,
-                    condition="" if not condition or condition == "." else condition,
-                    gene=gene or "",
-                    review_status=review_status or "",
-                    alt=alt,
+                variant_annotations.append(
+                    Annotation(
+                        source=self.name,
+                        rsid=variant.rsid,
+                        significance=f"clinvar_{sig_label}",
+                        category="clinical",
+                        magnitude=_magnitude(clnsig),
+                        description=description,
+                        attribution=self.attribution,
+                        genotype_match=user_diploid,
+                        references=references,
+                        condition="" if not condition or condition == "." else condition,
+                        gene=gene or "",
+                        review_status=review_status or "",
+                        alt=alt,
+                    )
                 )
+            yield from _collapse_identical_scvs(variant_annotations)
 
 
 def _user_diploid(variant: Variant) -> str:

{allelix-2.1.2 → allelix-2.2.1}/allelix/cli/_helpers.py

@@ -417,6 +417,63 @@ def _emit_build_diagnostics(result: object) -> None:
         )
 
 
+def _emit_runtime_nudges(result: object) -> None:
+    """Surface runtime-degradation notes the analyze run can detect.
+
+    Both are once-per-run, post-pipeline. Neither blocks the report; the
+    intent is transparency for cases where the silent default behavior
+    used to leave the user uncertain about WHY their count looked low.
+
+    - **GH #90: strand-aware matching inactive.** When carrier/genotype
+      matching ran with ``Variant.ref`` unavailable across the input —
+      typically array data with ``--no-gnomad`` or no gnomAD cache —
+      the strand-flip path in ``utils/allele.py`` falls back to the
+      v2.0.x direct-match-only behavior. Behavior is correct (forward-
+      normalized array data matches the direct path regardless), just
+      degraded relative to v2.1+'s contract. VCF inputs always carry
+      REF from the file, so this never fires for them.
+    - **GH #91: GRCh38 rsID-less undercount.** When the effective
+      build is GRCh38 AND most variants entered the pipeline without
+      rsIDs (the variant-caller ``ID=.`` case), annotation coverage
+      is materially lower than the GRCh37 equivalent because rsID
+      resolution falls back to ClinVar position lookup — a smaller
+      surface than dbSNP. Real fix is #62; this is the interim
+      honesty signal until then.
+    """
+    no_ref = getattr(result, "no_ref_variant_count", 0)
+    rsidless = getattr(result, "rsidless_variant_count", 0)
+    total = getattr(result, "total_variants", 0)
+    parser_name = getattr(result, "parser_name", "")
+    diag = getattr(result, "build_diagnostics", None)
+    effective_build = diag.effective_build if diag is not None else None
+
+    # GH #90: array inputs (everything except the VCF parser) running
+    # without a usable gnomAD cache leave every variant ref=None. Gate
+    # on "the whole input lacked ref" rather than "any variant lacked
+    # ref" so a hybrid file (e.g. a few ref-less rows in an otherwise
+    # populated VCF) doesn't flap the message on.
+    if parser_name != "vcf" and total > 0 and no_ref == total:
+        console.print(
+            "[dim]strand-aware matching inactive — no reference context "
+            "(run `allelix db update` for gnomAD-backed ref resolution, "
+            "or this is expected with `--no-gnomad`)[/dim]"
+        )
+
+    # GH #91: GRCh38 + a high rsID-less fraction. The threshold (>50%)
+    # distinguishes a variant-caller VCF (mostly ID=. — fires) from a
+    # 23andMe-style array (rsID-bearing — no warning) or a mostly-
+    # rsID-bearing VCF where a handful of novel calls came in with
+    # ID=. (no warning, ID=. for novel sites is expected).
+    if effective_build == "GRCh38" and total > 0 and rsidless / total > 0.5:
+        console.print(
+            "[yellow]GRCh38 input without rsIDs: annotation coverage is "
+            "currently lower than GRCh37 because rsID resolution falls "
+            "back to ClinVar positions. Full dbSNP resolution is planned "
+            "(#62). If you have a GRCh37 build available, it currently "
+            "surfaces more annotations.[/yellow]"
+        )
+
+
 def _run_analysis_command(
     file_path: Path,
     fmt: str | None,
@@ -496,11 +553,13 @@ def _run_analysis_command(
             alphamissense=cast("AlphaMissenseAnnotator | None", am_annotator),
             cadd=cast("CaddAnnotator | None", cadd_annotator),
             high_value_rsids=hv_rsids,
+            panel_rsids=rsids,
         )
     finally:
         _unwire_parser_logging(counter, stderr_handler, snapshot)
 
     _emit_build_diagnostics(result)
+    _emit_runtime_nudges(result)
 
     hv_warnings = scan_no_calls(result.hv_variants, high_value)
     if hv_warnings:

{allelix-2.1.2 → allelix-2.2.1}/allelix/cli/_options.py

@@ -33,6 +33,25 @@ _MIN_MAG_OPT = click.option(
     show_default=True,
     help="Filter annotations below this magnitude. Use 0 for the full unfiltered set.",
 )
+# The focused subcommands (`allelix methylation`, `allelix pharmacogenomics`)
+# default to a lower floor than the general `analyze` report. The intent of
+# a focused report is to surface focused-level signal — ClinPGx LoE 3 hits
+# on COMT / MTHFR / MTR / CBS for `methylation`, similar for the pharma
+# subcommand — which the magnitude-5 floor that protects `analyze` from
+# noise would filter out as if they were noise. Floor 3.0 keeps LoE 3
+# in, keeps ClinVar Benign (mag 1.0) out.
+_FOCUSED_MIN_MAG_OPT = click.option(
+    "--min-magnitude",
+    type=float,
+    default=3.0,
+    show_default=True,
+    help=(
+        "Filter annotations below this magnitude. Focused subcommands default "
+        "to 3.0 (vs. analyze's 5.0) so ClinPGx LoE 3 hits on panel genes — "
+        "the intended signal of a focused report — aren't filtered out. "
+        "Use 0 for the full unfiltered set."
+    ),
+)
 _OUTPUT_OPT = click.option(
     "--output",
     type=click.Path(dir_okay=False, path_type=Path),