allelix 2.2.2__tar.gz → 2.3.0__tar.gz

{allelix-2.2.2 → allelix-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: allelix
-Version: 2.2.2
+Version: 2.3.0
 Summary: Open-source genotype analysis toolkit. Format-agnostic ingestion, database-agnostic annotation, offline-first.
 Author: Allelix
 Maintainer-email: dial481 <dial481@users.noreply.github.com>
@@ -43,7 +43,7 @@ Dynamic: license-file
 
 **Open-source genotype analysis. Format-agnostic. Offline-first.**
 
-[Website](https://allelix.io) · [Quickstart](#quickstart) · [Changelog](https://github.com/allelix/allelix/blob/main/CHANGELOG.md)
+[Website](https://allelix.io) · [Demo](https://analyze.allelix.io) · [Quickstart](#quickstart) · [Changelog](https://github.com/allelix/allelix/blob/main/CHANGELOG.md)
 
 [![python](https://img.shields.io/pypi/pyversions/allelix.svg)](https://www.python.org/downloads/)
 [![pypi](https://img.shields.io/pypi/v/allelix.svg)](https://pypi.org/project/allelix/)
@@ -55,7 +55,7 @@ Dynamic: license-file
 
 # Allelix
 
-> **Status:** Production. Eight parser formats (including VCF + gVCF), four annotators (ClinVar + ClinPGx + GWAS Catalog + SNPedia), three enrichment sources (gnomAD + AlphaMissense + CADD), licensable-source gating for commercial users, dual-build ClinVar caches (GRCh37 + GRCh38), HTML/JSON/terminal reports, methylation + pharmacogenomics focused commands, report diffing, persistent config with commercial-mode safety switch. Build auto-detection from position data (ADR-0021). No regex on prose anywhere in production. The [Changelog](https://github.com/allelix/allelix/blob/main/CHANGELOG.md) tracks every release.
+> **Status:** Production. Eight parser formats (including VCF + gVCF), four annotators (ClinVar + ClinPGx + GWAS Catalog + SNPedia), three enrichment sources (gnomAD + AlphaMissense + CADD), licensable-source gating for commercial users, dual-build ClinVar caches (GRCh37 + GRCh38), HTML/JSON/terminal reports and annotated VCF output for pipeline integration (v2.3.0+), methylation + pharmacogenomics focused commands, report diffing, persistent config with commercial-mode safety switch. Build auto-detection from position data (ADR-0021). No regex on prose anywhere in production. The [Changelog](https://github.com/allelix/allelix/blob/main/CHANGELOG.md) tracks every release.
 
 ## Quickstart
 
@@ -80,6 +80,11 @@ allelix analyze trio.vcf.gz --sample HG002 --output report.html
 
 # Filter to a custom panel (rsIDs + gene names, one per line; '#' comments and blank lines ignored)
 allelix analyze your_genotype_file.txt --filter-file my_panel.txt --output report.html
+
+# Annotated VCF output — VCF-in / VCF-out for pipeline integration
+# Writes a valid VCF with annotations as INFO fields + full provenance in the header.
+# Refuses to overwrite; requires VCF or gVCF input.
+allelix analyze your_wgs.vcf.gz --vcf-out annotated.vcf
 ```
 
 See [Development](#development) for source installs and running tests, [Managing your data](#managing-your-data) for cache locations and cleanup, and [Troubleshooting](#troubleshooting) for common failure states.
@@ -190,8 +195,10 @@ disease findings. Wherever you write them via `--output <path>`,
 that's where they sit until you delete them. Allelix doesn't
 auto-clean and won't warn you when you write to `/tmp/` or any
 other shared location. Treat the files as personal data: read them,
-move them somewhere you control, or delete when you're done. A
-data-lifecycle subcommand is planned for v2.1.
+move them somewhere you control, or delete when you're done.
+`allelix db clean` (shipped in v2.2.0) handles the cache side of
+the lifecycle; output reports are still the user's responsibility
+to manage.
 
 ## Configuration
 

{allelix-2.2.2 → allelix-2.3.0}/README.md

@@ -4,7 +4,7 @@
 
 **Open-source genotype analysis. Format-agnostic. Offline-first.**
 
-[Website](https://allelix.io) · [Quickstart](#quickstart) · [Changelog](https://github.com/allelix/allelix/blob/main/CHANGELOG.md)
+[Website](https://allelix.io) · [Demo](https://analyze.allelix.io) · [Quickstart](#quickstart) · [Changelog](https://github.com/allelix/allelix/blob/main/CHANGELOG.md)
 
 [![python](https://img.shields.io/pypi/pyversions/allelix.svg)](https://www.python.org/downloads/)
 [![pypi](https://img.shields.io/pypi/v/allelix.svg)](https://pypi.org/project/allelix/)
@@ -16,7 +16,7 @@
 
 # Allelix
 
-> **Status:** Production. Eight parser formats (including VCF + gVCF), four annotators (ClinVar + ClinPGx + GWAS Catalog + SNPedia), three enrichment sources (gnomAD + AlphaMissense + CADD), licensable-source gating for commercial users, dual-build ClinVar caches (GRCh37 + GRCh38), HTML/JSON/terminal reports, methylation + pharmacogenomics focused commands, report diffing, persistent config with commercial-mode safety switch. Build auto-detection from position data (ADR-0021). No regex on prose anywhere in production. The [Changelog](https://github.com/allelix/allelix/blob/main/CHANGELOG.md) tracks every release.
+> **Status:** Production. Eight parser formats (including VCF + gVCF), four annotators (ClinVar + ClinPGx + GWAS Catalog + SNPedia), three enrichment sources (gnomAD + AlphaMissense + CADD), licensable-source gating for commercial users, dual-build ClinVar caches (GRCh37 + GRCh38), HTML/JSON/terminal reports and annotated VCF output for pipeline integration (v2.3.0+), methylation + pharmacogenomics focused commands, report diffing, persistent config with commercial-mode safety switch. Build auto-detection from position data (ADR-0021). No regex on prose anywhere in production. The [Changelog](https://github.com/allelix/allelix/blob/main/CHANGELOG.md) tracks every release.
 
 ## Quickstart
 
@@ -41,6 +41,11 @@ allelix analyze trio.vcf.gz --sample HG002 --output report.html
 
 # Filter to a custom panel (rsIDs + gene names, one per line; '#' comments and blank lines ignored)
 allelix analyze your_genotype_file.txt --filter-file my_panel.txt --output report.html
+
+# Annotated VCF output — VCF-in / VCF-out for pipeline integration
+# Writes a valid VCF with annotations as INFO fields + full provenance in the header.
+# Refuses to overwrite; requires VCF or gVCF input.
+allelix analyze your_wgs.vcf.gz --vcf-out annotated.vcf
 ```
 
 See [Development](#development) for source installs and running tests, [Managing your data](#managing-your-data) for cache locations and cleanup, and [Troubleshooting](#troubleshooting) for common failure states.
@@ -151,8 +156,10 @@ disease findings. Wherever you write them via `--output <path>`,
 that's where they sit until you delete them. Allelix doesn't
 auto-clean and won't warn you when you write to `/tmp/` or any
 other shared location. Treat the files as personal data: read them,
-move them somewhere you control, or delete when you're done. A
-data-lifecycle subcommand is planned for v2.1.
+move them somewhere you control, or delete when you're done.
+`allelix db clean` (shipped in v2.2.0) handles the cache side of
+the lifecycle; output reports are still the user's responsibility
+to manage.
 
 ## Configuration
 

{allelix-2.2.2 → allelix-2.3.0}/allelix/annotators/gnomad.py

@@ -247,34 +247,29 @@ class GnomadAnnotator(Annotator):
             return {}
         conn = self._connection()
         result: dict[tuple[str, int], list[tuple[str, str, str]]] = {}
-        # GH #134: issue a single combined SQL statement per chunk
-        # using SQLite's row-value ``(chrom, pos) IN ((?, ?), ...)``
-        # form instead of grouping by chromosome and issuing one
-        # statement per (chrom, chunk) pair. On WGS input each batch
-        # touches all ~24 chromosomes; the per-chrom shape was a 24x
-        # query multiplier the new ``idx_gnomad_position`` index could
-        # not amortize away. The row-value IN form keeps the same
-        # carrier-rule contract — both ``chrom`` and ``pos`` must
-        # match the input pair simultaneously — and the
-        # ``idx_gnomad_position`` covering index on (chrom, pos)
-        # serves the lookup directly. ``_BULK_BATCH_SIZE`` here
-        # refers to (chrom, pos) PAIRS not single ints, so each
-        # chunk binds 2 * batch_size parameters.
+        # Issue a single combined SQL statement per chunk using SQLite's
+        # row-value ``(chrom, pos) IN ((?, ?), ...)`` form instead of
+        # grouping by chromosome and issuing one statement per
+        # (chrom, chunk) pair. On WGS input each batch touches all ~24
+        # chromosomes; the per-chrom shape was a 24x query multiplier
+        # the ``idx_gnomad_position`` index could not amortize away. The
+        # row-value IN form keeps the same carrier-rule contract — both
+        # ``chrom`` and ``pos`` must match the input pair simultaneously
+        # — and the ``idx_gnomad_position`` covering index on
+        # ``(chrom, pos)`` serves the lookup directly.
         #
-        # SQLite parameter-limit awareness: 2 * _BULK_BATCH_SIZE =
-        # 1800 bind parameters per chunk. The historical SQLite
-        # ``SQLITE_MAX_VARIABLE_NUMBER`` default was 999, raised to
-        # 32766 in SQLite 3.32 (2020-05-22). Every Python version
-        # Allelix supports (>= 3.11) ships with SQLite >= 3.34, so
-        # the 1800-param chunk is well within the modern limit. If a
-        # future change bumps ``_BULK_BATCH_SIZE`` toward 16000 or
-        # the project drops Python's bundled SQLite for a system
-        # ``libsqlite3`` of unknown vintage, re-check this against
-        # the runtime ``sqlite3.sqlite_version`` before assuming
-        # capacity.
+        # Each input pair binds 2 SQLite parameters (chrom + pos), so
+        # the chunk size is ``_BULK_BATCH_SIZE // 2`` to stay inside the
+        # 999-param convention every other annotator and gnomAD's own
+        # multi-param queries respect (``bulk_lookup_by_alt`` does the
+        # same 2-param shape with the same ``// 2``;
+        # ``bulk_lookup_by_position`` uses ``// 4`` for its 4-param
+        # shape, same convention scaled). Net effective batch size:
+        # 450 pairs x 2 = 900 params per chunk.
+        chunk_pairs = _BULK_BATCH_SIZE // 2
         position_list = list(positions)
-        for i in range(0, len(position_list), _BULK_BATCH_SIZE):
-            batch = position_list[i : i + _BULK_BATCH_SIZE]
+        for i in range(0, len(position_list), chunk_pairs):
+            batch = position_list[i : i + chunk_pairs]
             placeholders = ",".join("(?, ?)" for _ in batch)
             params: list[str | int] = []
             for chrom, pos in batch:

{allelix-2.2.2 → allelix-2.3.0}/allelix/cli/_helpers.py

@@ -496,12 +496,29 @@ def _run_analysis_command(
     no_alphamissense: bool = False,
     no_cadd: bool = False,
     sample: str | None = None,
+    vcf_out: Path | None = None,
 ) -> None:
     resolved = resolve_data_dir(data_dir)
     if not no_update:
         _maybe_refresh_databases(resolved)
     parser = _resolve_parser(file_path, fmt, sample=sample)
 
+    # #150: --vcf-out preflight. Bail before any expensive setup so a
+    # user who mistyped the flag or aimed it at a 23andMe file finds
+    # out immediately, not after the analyze pipeline runs.
+    if vcf_out is not None:
+        if parser.name != "vcf":
+            raise click.ClickException(
+                f"--vcf-out requires VCF or gVCF input; got {parser.display_name} "
+                f"({parser.name!r}). Convert to VCF first, or drop --vcf-out to "
+                "produce the default report."
+            )
+        if vcf_out.exists():
+            raise click.ClickException(
+                f"--vcf-out refuses to overwrite an existing file: {vcf_out}. "
+                "Move or delete the existing file, or choose a different path."
+            )
+
     from allelix.config import load_config
 
     cfg = load_config(resolved)
@@ -554,6 +571,7 @@ def _run_analysis_command(
             cadd=cast("CaddAnnotator | None", cadd_annotator),
             high_value_rsids=hv_rsids,
             panel_rsids=rsids,
+            collect_vcf_write_inputs=vcf_out is not None,
         )
     finally:
         _unwire_parser_logging(counter, stderr_handler, snapshot)
@@ -650,6 +668,26 @@ def _run_analysis_command(
             )
         console.print(f"[green]Wrote {rendered:,} annotation(s) to {output}[/green]")
 
+    if vcf_out is not None:
+        from allelix.reports.vcf import render_vcf
+
+        # Pipeline populates result.vcf_write_inputs whenever the
+        # ``collect_vcf_write_inputs`` flag is set — which the preflight
+        # above guarantees for this branch — so the None case here is
+        # a defensive assert, not a runtime path.
+        assert result.vcf_write_inputs is not None
+        stamped = render_vcf(
+            input_path=file_path,
+            output_path=vcf_out,
+            build=result.build,
+            annotators_used=result.annotators_used,
+            annotations_by_key=result.vcf_write_inputs.annotations_by_key,
+            recovered_rsids=result.vcf_write_inputs.recovered_rsids,
+        )
+        console.print(
+            f"[green]Wrote annotated VCF with {stamped:,} stamped row(s) to {vcf_out}[/green]"
+        )
+
     console.print(
         f"[dim]{len(result.annotations):,} total annotation(s) from {len(ready)} "
         f"database(s) across {result.total_variants:,} variant(s).[/dim]"

{allelix-2.2.2 → allelix-2.3.0}/allelix/cli/_options.py

@@ -169,3 +169,17 @@ _SAMPLE_OPT = click.option(
         "or any plain VCF tool to list available samples."
     ),
 )
+# #150: annotated VCF output. Requires VCF/gVCF input (any other format
+# errors out early). Existing output paths are refused (no clobber, no
+# --force in v2.3.0). See ADR-0036 for the output contract.
+_VCF_OUT_OPT = click.option(
+    "--vcf-out",
+    "vcf_out",
+    type=click.Path(dir_okay=False, path_type=Path),
+    default=None,
+    help=(
+        "Write an annotated VCF at this path (v2.3.0+; experimental output "
+        "contract, ADR-0036). Requires VCF or gVCF input. Refuses to "
+        "overwrite an existing file."
+    ),
+)

{allelix-2.2.2 → allelix-2.3.0}/allelix/cli/analyze.py

@@ -29,6 +29,7 @@ from allelix.cli._options import (
     _REPORT_FORMAT_OPT,
     _SAMPLE_OPT,
     _SNPEDIA_MIN_MAG_OPT,
+    _VCF_OUT_OPT,
 )
 
 if TYPE_CHECKING:
@@ -61,6 +62,7 @@ if TYPE_CHECKING:
 @_NO_ALPHAMISSENSE_OPT
 @_NO_CADD_OPT
 @_SAMPLE_OPT
+@_VCF_OUT_OPT
 def analyze(
     file_path: Path,
     fmt: str | None,
@@ -82,6 +84,7 @@ def analyze(
     no_alphamissense: bool,
     no_cadd: bool,
     sample: str | None,
+    vcf_out: Path | None,
 ) -> None:
     """Annotate a genotype file against all ready reference databases."""
     filter_genes: frozenset[str] | None = None
@@ -113,4 +116,5 @@ def analyze(
         no_alphamissense=no_alphamissense,
         no_cadd=no_cadd,
         sample=sample,
+        vcf_out=vcf_out,
     )

{allelix-2.2.2 → allelix-2.3.0}/allelix/parsers/twentythreeandme.py

@@ -42,7 +42,26 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
-_CANONICAL_PREFIX = "# This data file generated by 23andMe"
+# Known canonical first-line banner variants 23andMe wraps raw exports in.
+# The original v2/v3/v4-era exports used the verbatim ``generated by`` form;
+# the June 2026 v5 export prose was edited to ``is generated by`` (with a
+# trailing period in some samples). Match either; future vendor banner
+# edits get a new entry here.
+#
+# Why a tuple of exact prefixes instead of a loose ``"23andMe" in line``
+# substring match: a bare substring check false-positives on transcoded
+# fixtures and third-party tool output. Files that strip the canonical
+# banner entirely (WGSExtract output, Genetic Lifehacks converter output,
+# privacy-conscious users who delete the comment block) are out of scope
+# for this parser by design — they are handled by ``GenericTsvParser`` at
+# the end of the dispatch chain (planned, see GH #146 / GH #127). That
+# two-layer design keeps banner-bearing 23andMe files attributed to
+# "23andMe" and banner-less 23andMe-shape files attributed to "generic-tsv"
+# without guessing a vendor.
+_CANONICAL_PREFIXES: tuple[str, ...] = (
+    "# This data file generated by 23andMe",
+    "# This data file is generated by 23andMe",
+)
 SNIFF_LINE_LIMIT = 50
 EXPECTED_COLUMNS = 4
 
@@ -58,10 +77,13 @@ class TwentyThreeAndMeParser(GenotypeParser):
     def can_parse(self, file_path: Path) -> bool:
         """Recognize a real 23andMe export by its canonical first-line header.
 
-        Real exports start with ``# This data file generated by 23andMe at: ...``.
+        Real exports start with one of the variants in ``_CANONICAL_PREFIXES``:
+        the historical ``# This data file generated by 23andMe at: ...`` form
+        or the June 2026 v5 ``# This data file is generated by 23andMe.`` form.
         Some files have user-prepended comments before the canonical line; the
-        fallback loop scans up to ``SNIFF_LINE_LIMIT`` comment lines. A bare
-        "23andMe" mention without the canonical prefix is rejected.
+        scan loop reads up to ``SNIFF_LINE_LIMIT`` comment lines. A bare
+        "23andMe" substring mention without one of the canonical prefixes is
+        rejected (see ``_CANONICAL_PREFIXES`` rationale).
         """
         try:
             with file_path.open("r", encoding="utf-8") as fh:
@@ -71,7 +93,7 @@ class TwentyThreeAndMeParser(GenotypeParser):
                         return False
                     if not line.startswith("#"):
                         return False
-                    if line.startswith(_CANONICAL_PREFIX):
+                    if any(line.startswith(p) for p in _CANONICAL_PREFIXES):
                         return True
         except (OSError, UnicodeDecodeError):
             return False

{allelix-2.2.2 → allelix-2.3.0}/allelix/reports/_pipeline.py

@@ -39,6 +39,7 @@ if TYPE_CHECKING:
     from allelix.annotators.gnomad import GnomadAnnotator
     from allelix.models import Annotation, Variant
     from allelix.parsers.base import GenotypeParser
+    from allelix.reports.vcf import VcfWriteInputs
 
 
 # How many input variants to buffer while waiting for detection to
@@ -155,6 +156,12 @@ class AnalysisResult:
     # Used by the CLI to surface the GRCh38 rsID-less undercount
     # warning until #62's dbSNP resolution lands.
     rsidless_variant_count: int = 0
+    # #150: pipeline-produced inputs for the annotated VCF writer.
+    # Populated only when ``run_analysis`` is called with
+    # ``collect_vcf_write_inputs=True`` — the ``--vcf-out`` code path.
+    # None on every other analyze run (zero memory / CPU overhead on the
+    # default path). Consumed by ``allelix.reports.vcf.render_vcf``.
+    vcf_write_inputs: VcfWriteInputs | None = None
 
     def panel_coverage(
         self, filtered_annotations: list[Annotation] | None = None
@@ -439,6 +446,7 @@ def run_analysis(
     cadd: CaddAnnotator | None = None,
     high_value_rsids: set[str] | None = None,
     panel_rsids: frozenset[str] | None = None,
+    collect_vcf_write_inputs: bool = False,
 ) -> AnalysisResult:
     """Stream the file once; batch-annotate; return a fully-populated result.
 
@@ -515,6 +523,13 @@ def run_analysis(
     # back to position-keyed gnomAD / AlphaMissense lookups for resolved
     # rsIDs that don't appear in those caches' rsid index.
     resolved_coords: dict[str, tuple[str, int, str, str]] = {}
+    # #150: annotation dict keyed by (chrom, pos, ref, alt) for the
+    # annotated-VCF writer. Populated per batch inside ``_flush`` when
+    # ``collect_vcf_write_inputs`` is True. Values are references to the
+    # Annotation objects in ``annotations`` — enrichment mutations
+    # (gnomAD AF, AlphaMissense, CADD) that happen after streaming are
+    # visible through these refs without a per-annotation copy.
+    vcf_annotations_by_key: dict[tuple[str, int, str, str], list[Annotation]] = {}
 
     with contextlib.ExitStack() as stack:
         bound = [stack.enter_context(a) for a in annotators]
@@ -617,8 +632,29 @@ def run_analysis(
             nonlocal no_ref_total, rsidless_total
             no_ref_total += sum(1 for v in batch_buf if v.ref is None)
             rsidless_total += sum(1 for v in batch_buf if not v.rsid.startswith("rs"))
+            batch_new: list[Annotation] = []
             for annotator in bound:
-                annotations.extend(annotator.batch_annotate(batch_buf))
+                batch_new.extend(annotator.batch_annotate(batch_buf))
+            annotations.extend(batch_new)
+            # #150: collect writer inputs for this batch. Only touches
+            # variants whose ``ref`` is populated (VCF inputs; array
+            # data with ref=None can't produce a well-formed VCF key).
+            # Skips annotations with empty ``alt`` — GWAS rows keyed by
+            # rsID alone can't be VCF-stamped without allele context.
+            if collect_vcf_write_inputs:
+                rsid_to_coords: dict[str, tuple[str, int, str]] = {
+                    v.rsid: (v.chromosome, v.position, v.ref)
+                    for v in batch_buf
+                    if v.ref is not None
+                }
+                for a in batch_new:
+                    if not a.alt:
+                        continue
+                    coords = rsid_to_coords.get(a.rsid)
+                    if coords is None:
+                        continue
+                    chrom, pos, ref = coords
+                    vcf_annotations_by_key.setdefault((chrom, pos, ref, a.alt), []).append(a)
             batch_buf.clear()
 
         def _accept(v: Variant) -> None:
@@ -733,6 +769,21 @@ def run_analysis(
         result_annotated = None
         result_genotypes = None
 
+    vcf_write_inputs: VcfWriteInputs | None = None
+    if collect_vcf_write_inputs:
+        # Import here to avoid a top-level cycle with allelix.reports.vcf
+        # (which reaches into the annotator registry for attribution).
+        from allelix.reports.vcf import VcfWriteInputs as _VcfWriteInputs
+
+        vcf_recovered_rsids = {
+            (chrom, pos, ref, alt): rsid
+            for rsid, (chrom, pos, ref, alt) in resolved_coords.items()
+        }
+        vcf_write_inputs = _VcfWriteInputs(
+            annotations_by_key=vcf_annotations_by_key,
+            recovered_rsids=vcf_recovered_rsids,
+        )
+
     return AnalysisResult(
         file_path=file_path,
         parser_name=parser.name,
@@ -751,6 +802,7 @@ def run_analysis(
         panel_genotypes=result_genotypes,
         no_ref_variant_count=no_ref_total,
         rsidless_variant_count=rsidless_total,
+        vcf_write_inputs=vcf_write_inputs,
     )