afquery 0.1.2__tar.gz → 0.1.2.2__tar.gz

afquery-0.1.2.2/.github/workflows/docs.yml

@@ -0,0 +1,38 @@
+name: Deploy Documentation
+
+on:
+  push:
+    branches: [master]
+    paths:
+      - 'docs/**'
+      - 'mkdocs.yml'
+      - 'src/**'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: pages
+  cancel-in-progress: false
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Install docs dependencies
+        run: pip install -e ".[docs]"
+
+      - name: Build and deploy docs
+        run: mkdocs gh-deploy --force

afquery-0.1.2.2/PKG-INFO

@@ -0,0 +1,395 @@
+Metadata-Version: 2.4
+Name: afquery
+Version: 0.1.2.2
+Summary: Genomic allele frequency query engine with bitmap-encoded genotypes
+Requires-Python: >=3.10
+Requires-Dist: click>=8.1
+Requires-Dist: cyvcf2>=0.30
+Requires-Dist: duckdb>=0.10
+Requires-Dist: pyarrow>=14.0
+Requires-Dist: pyranges<0.2,>=0.1.2
+Requires-Dist: pyroaring>=0.4.8
+Requires-Dist: tqdm>=4.60
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=0.25; extra == 'docs'
+Requires-Dist: pymdown-extensions>=10.7; extra == 'docs'
+Description-Content-Type: text/markdown
+
+# AFQuery
+
+Fast, file-based genomic allele frequency queries for large cohorts (10K–50K samples).
+No server, no cloud — just files. Sub-100ms point queries with flexible filtering
+by sex, phenotype (ICD codes), and sequencing technology.
+
+## Quick Example
+
+```bash
+pip install afquery
+
+# Build database from your VCFs
+afquery create-db --manifest samples.tsv --output-dir ./db/ --genome-build GRCh38
+
+# Query allele frequency
+afquery query --db ./db/ --chrom chr1 --pos 123456 --phenotype E11.9 --sex female
+
+# Annotate a VCF
+afquery annotate --db ./db/ --input variants.vcf --output annotated.vcf
+```
+
+**[Full documentation →](https://davidlaewen.github.io/afquery/)**
+
+## Features
+
+- Sub-100ms point queries on 50K-sample cohorts
+- Filter by sex, phenotype (ICD codes), and sequencing technology
+- Bitmap-compressed storage (Roaring Bitmaps + Parquet)
+- Incremental updates (add/remove samples without full rebuild)
+- VCF annotation with custom sample subsets
+- Ploidy-aware AN for sex chromosomes (X/Y/MT)
+- Zero infrastructure — purely file-based
+
+## Installation
+
+```bash
+pip install afquery
+# or
+conda install -c bioconda -c conda-forge afquery
+```
+
+Requires Python 3.10+.
+
+## Quick Start
+
+### 1. Create a Database
+
+First, prepare a manifest TSV with sample metadata:
+
+```tsv
+sample_name	sex	tech_name	vcf_path	phenotype_codes
+sample_1	male	wgs	vcfs/sample_1.vcf	E11.9,I10
+sample_2	female	wes_kit_a	vcfs/sample_2.vcf	E11.9
+sample_3	male	wgs	vcfs/sample_3.vcf	I10
+```
+
+**Key points about the manifest:**
+- `tech_name`: Either `wgs` (case-insensitive) for whole genome, or a custom technology name
+- `vcf_path`: Path to the single-sample VCF file (relative to manifest directory, or absolute)
+- For WES/exome technologies, capture regions are loaded from `--bed-dir/{tech_name}.bed`
+  - Example: `tech_name=wes_kit_a` → loads `beds/wes_kit_a.bed`
+
+Organize your files:
+```
+project/
+├── manifest.tsv
+├── vcfs/
+│   ├── sample_1.vcf
+│   ├── sample_2.vcf
+│   └── sample_3.vcf
+└── beds/              # Required for non-WGS technologies
+    └── wes_kit_a.bed  # BED file for WES kit A
+```
+
+Then preprocess your VCF files:
+
+```bash
+afquery preprocess \
+  --manifest manifest.tsv \
+  --bed-dir ./beds/ \
+  --output-dir ./my_db/ \
+  --genome-build GRCh38
+```
+
+This creates:
+- `my_db/manifest.json` — database metadata
+- `my_db/metadata.sqlite` — samples, technologies, phenotype codes, precomputed bitmaps
+- `my_db/variants/{chrom}.parquet` — variant data with encoded genotypes
+- `my_db/capture/` — capture regions for each technology
+
+### 2. Query Allele Frequencies
+
+```bash
+# Point query
+afquery query --db my_db --chrom chr1 --pos 1000 --alt G
+
+# Batch query (100 positions)
+afquery query-batch --db my_db --positions positions.tsv --phenotype E11.9
+
+# Region query
+afquery query --db my_db --chrom chr1 --start 1000 --end 10000 --phenotype E11.9 --sex M
+```
+
+### 3. Annotate VCF Files
+
+```bash
+afquery annotate \
+  --db my_db \
+  --vcf input.vcf \
+  --output annotated.vcf \
+  --phenotype E11.9 \
+  --tech WGS
+```
+
+Adds `AFQUERY_AC`, `AFQUERY_AN`, `AFQUERY_AF` and genotype fields to your VCF.
+
+## Python API
+
+```python
+from afquery import Database
+
+db = Database("/path/to/db")
+
+# Single position query
+# Automatically filters samples by: sex + phenotype codes + capture coverage
+result = db.query(
+    chrom="chr1",
+    pos=1000,
+    alt="G",
+    phenotype_codes=["E11.9"],
+    sex="both"
+)
+print(f"AC={result.ac}, AN={result.an}, AF={result.af}")
+
+# Batch query (multi-variant)
+results = db.query_batch(
+    "chr1",
+    variants=[(1500, "A", "T"), (3500, "G", "C")],
+    phenotype=["E11.9"],
+)
+
+# Region query (genomic range)
+results = db.query_region(
+    chrom="chr1",
+    start=1000,
+    end=10000,
+    phenotype_codes=["E11.9", "I10"]
+)
+
+# Annotate VCF with allele frequencies
+# Note: tech_name filters annotation to samples of that technology
+db.annotate_vcf(
+    vcf_path="input.vcf",
+    output_path="annotated.vcf",
+    phenotype_codes=["E11.9"],
+    tech_name="wgs"  # Only annotate using WGS samples
+)
+```
+
+**How samples are filtered in queries**:
+- Sex filter: `male`, `female`, or `both`
+- phenotype filter: All codes must match
+- Capture filter: Automatic—only samples whose tech's BED covers the position
+
+## Database Structure
+
+```
+my_db/
+├── manifest.json          # Metadata: genome_build, sample_count, schema_version
+├── metadata.sqlite        # SQLite: samples, technologies, phenotype codes, bitmaps
+├── variants/
+│   ├── chr1.parquet
+│   ├── chr2.parquet
+│   └── ...
+└── capture/
+    ├── tech_0.pickle      # WGS capture region (always covered)
+    └── tech_1.pickle      # WES kit capture region
+```
+
+Each variant row contains:
+- `pos` — 1-based genomic position
+- `ref` — reference allele
+- `alt` — alternate allele
+- `het_bitmap` — Roaring Bitmap of heterozygous samples
+- `hom_bitmap` — Roaring Bitmap of homozygous samples
+
+## How Capture BED Files Are Associated with Samples
+
+Samples are linked to capture regions through their **technology**:
+
+1. **Manifest specifies technology**: Each sample lists `tech_name` (e.g., `wgs`, `wes_kit_a`)
+2. **Technology maps to BED file**:
+   - **WGS**: No BED file needed (always fully covered)
+   - **WES/Custom**: BED file loaded from `{bed_dir}/{tech_name}.bed`
+3. **Storage in database**:
+   - `metadata.sqlite::technologies` stores tech_id, tech_name, and bed_path
+   - `metadata.sqlite::samples` stores sample_id, sample_name, and tech_id (foreign key)
+4. **Query-time filtering**: When querying, samples are filtered by:
+   - Sex (male/female/both)
+   - phenotype diagnosis codes
+   - Capture region coverage (via tech's BED file)
+
+**Example**: If you have samples on two exome kits:
+```tsv
+sample_name	sex	tech_name	vcf_path	phenotype_codes
+S001	male	exome_v1	vcfs/S001.vcf	E11.9
+S002	female	exome_v1	vcfs/S002.vcf	E11.9
+S003	male	exome_v2	vcfs/S003.vcf	I10
+```
+
+Then provide:
+```
+beds/
+├── exome_v1.bed    # Coverage for samples S001, S002
+└── exome_v2.bed    # Coverage for sample S003
+```
+
+At query time, each sample's eligible regions are determined by its tech's BED file.
+
+## Advanced Features
+
+### Incremental Updates (add_samples)
+
+```bash
+afquery add-samples \
+  --db my_db \
+  --manifest new_samples.tsv \
+  --vcf-dir ./new_vcfs/
+```
+
+Adds new samples without rebuilding the entire database.
+
+### Compact Database
+
+Remove samples and reclaim disk space:
+
+```bash
+afquery compact --db my_db --samples-to-remove sample_1,sample_2
+```
+
+### Run Benchmarks
+
+```bash
+afquery benchmark --db my_db --n-queries 1000 --query-type point
+```
+
+### Generate Synthetic Data
+
+```bash
+afquery synth --output synthetic_db/ --n-samples 5000 --n-variants 100000
+```
+
+## Ploidy Rules
+
+AF computation respects chromosome-specific ploidy:
+
+| Chromosome | Formula |
+|---|---|
+| Autosomes | `AN = 2 × eligible_samples` |
+| chrM | `AN = 1 × eligible_samples` |
+| chrY | `AN = 1 × eligible_males` |
+| chrX (PAR) | `AN = 2 × eligible_samples` |
+| chrX (non-PAR) | `AN = 2 × eligible_females + 1 × eligible_males` |
+
+Where `eligible` = samples matching sex, phenotype, and technology capture filters.
+
+## Performance Targets
+
+- **Point query (cold)**: <100 ms
+- **Point query (warm)**: ~10 ms
+- **Batch 100 positions**: ~200 ms
+- **VCF annotation (5K variants)**: ~30 s
+- **VCF annotation (5M variants)**: ~30 min
+
+## Command Reference
+
+```
+afquery query                Query single position
+afquery query-batch          Batch query multiple positions
+afquery annotate             Annotate VCF file
+afquery info                 Show database info
+afquery preprocess           Build database from VCFs
+afquery add-samples          Add new samples to database
+afquery compact              Remove samples and reclaim space
+afquery synth                Generate synthetic test database
+afquery benchmark            Run performance benchmarks
+```
+
+Run `afquery --help` for full options.
+
+## Development
+
+### Running Tests
+
+```bash
+# All 190 tests
+python3 -m pytest --tb=short -q
+
+# Specific test module
+python3 -m pytest tests/test_query.py -v
+```
+
+### Key Modules
+
+- `afquery.query` — QueryEngine, point/batch/region queries
+- `afquery.annotate` — VCF annotation pipeline
+- `afquery.database` — Database wrapper (public API)
+- `afquery.preprocess` — Manifest parsing, VCF ingestion, Parquet building
+- `afquery.bitmaps` — Roaring Bitmap encoding/decoding
+- `afquery.ploidy` — Chromosome-specific ploidy rules
+- `afquery.models` — Data classes (QueryResult, ParsedSample, etc.)
+
+### Architecture
+
+See `brain/architecture.md` for detailed system design, data flow, and query algorithm.
+
+## Genome Builds
+
+- **GRCh37** (hg19) — PAR regions: chrX:1-2649520
+- **GRCh38** (hg38) — PAR regions: chrX:1-3099677
+
+## Technologies Supported
+
+- **WGS** — Whole genome sequencing (always fully covered, no BED file needed)
+  - Manifest: `tech_name = wgs` (case-insensitive)
+  - Query-time: All positions in genome considered covered
+
+- **WES** — Whole exome sequencing (coverage defined by BED file)
+  - Manifest: `tech_name = wes_kit_a` (or any custom name)
+  - Preprocessing: Loads `{bed_dir}/wes_kit_a.bed` (0-based half-open BED format)
+  - Query-time: Only positions within BED intervals considered covered
+
+- **Custom** — Any technology with a BED file (e.g., gene panels, targeted sequencing)
+  - Manifest: Use any `tech_name`
+  - Preprocessing: Loads `{bed_dir}/{tech_name}.bed`
+  - Query-time: Respects BED file coverage
+
+## Troubleshooting
+
+### ImportError: `cyvcf2`
+
+`cyvcf2` import happens inside worker processes during preprocessing. Do not import at module level.
+
+### DuckDB Temp Files
+
+Uses Parquet format (not Arrow IPC) for compatibility. Set `DUCKDB_TEMP_DIRECTORY` if needed.
+
+### Sample IDs
+
+Sample IDs are 0-indexed and monotonically increasing. Never reuse removed IDs—use `compact` to reclaim space.
+
+## Contributing
+
+1. Read `brain/project_state.json` for current phase and test count
+2. Read `brain/architecture.md` for system design
+3. Follow code conventions in `CLAUDE.md`
+4. Update `brain/` docs after architectural changes
+5. Run tests before submitting
+
+## License
+
+(Add your license here)
+
+## Citation
+
+If you use afquery in research, please cite:
+
+```
+(Citation format to be determined)
+```
+
+---
+
+**Status**: Phase 5 complete (190 tests passing). Active development.

{afquery-0.1.2 → afquery-0.1.2.2}/README.md

@@ -1,23 +1,42 @@
-# afquery
+# AFQuery
 
-Genomic allele frequency query engine with bitmap-encoded genotypes. Fast, file-based queries over 10K-50K samples at sub-100ms latency.
+Fast, file-based genomic allele frequency queries for large cohorts (10K–50K samples).
+No server, no cloud — just files. Sub-100ms point queries with flexible filtering
+by sex, phenotype (ICD codes), and sequencing technology.
+
+## Quick Example
+
+```bash
+pip install afquery
+
+# Build database from your VCFs
+afquery create-db --manifest samples.tsv --output-dir ./db/ --genome-build GRCh38
+
+# Query allele frequency
+afquery query --db ./db/ --chrom chr1 --pos 123456 --phenotype E11.9 --sex female
+
+# Annotate a VCF
+afquery annotate --db ./db/ --input variants.vcf --output annotated.vcf
+```
+
+**[Full documentation →](https://davidlaewen.github.io/afquery/)**
 
 ## Features
 
-- **Fast point queries**: <100ms cold start, ~10ms warm queries on single positions
-- **Batch queries**: Multi-position queries via SQL IN clauses or temporary tables
-- **Region queries**: Genomic range queries with automatic partitioning
-- **VCF annotation**: Annotate VCF files with computed allele frequencies and sample genotypes
-- **Ploidy-aware**: Correct AN/AC computation for autosomes, chrX, chrY, and chrM
-- **Bitmap compression**: Roaring Bitmaps for efficient genotype storage
-- **In-process**: No server process—queries run locally with DuckDB
-- **Incremental updates**: Add new samples to existing databases
-- **Multiple genome builds**: Support for GRCh37 and GRCh38
+- Sub-100ms point queries on 50K-sample cohorts
+- Filter by sex, phenotype (ICD codes), and sequencing technology
+- Bitmap-compressed storage (Roaring Bitmaps + Parquet)
+- Incremental updates (add/remove samples without full rebuild)
+- VCF annotation with custom sample subsets
+- Ploidy-aware AN for sex chromosomes (X/Y/MT)
+- Zero infrastructure — purely file-based
 
 ## Installation
 
 ```bash
-pip install -e /home/dan/projects/afquery/ --break-system-packages
+pip install afquery
+# or
+conda install -c bioconda -c conda-forge afquery
 ```
 
 Requires Python 3.10+.

afquery-0.1.2.2/docs/advanced/benchmarking.md

@@ -0,0 +1,108 @@
+# Benchmarking
+
+`afquery benchmark` measures query performance on synthetic or real data and produces a JSON report.
+
+---
+
+## Quick Benchmark
+
+Run against synthetic data (no real database required):
+
+```bash
+afquery benchmark
+```
+
+This generates 1,000 synthetic samples and 10,000 variants per chromosome, builds an in-memory database, runs a suite of query types, and writes results to `benchmark_report.json`.
+
+---
+
+## Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--n-samples` | `1000` | Number of synthetic samples |
+| `--n-variants` | `10000` | Number of variants per chromosome |
+| `--output` | `benchmark_report.json` | Output path for JSON report |
+| `--db-dir` | None | Run against an existing database instead of synthetic data |
+
+---
+
+## Benchmark Against a Real Database
+
+```bash
+afquery benchmark --db-dir ./my_db/ --output my_db_benchmark.json
+```
+
+This uses your actual Parquet files and sample metadata, giving realistic performance numbers for your specific cohort size and variant density.
+
+---
+
+## Report Format
+
+The JSON report contains timing results for each query type:
+
+```json
+{
+  "n_samples": 1000,
+  "n_variants": 10000,
+  "genome_build": "GRCh38",
+  "queries": {
+    "point_query_cold_ms": 87.3,
+    "point_query_warm_ms": 9.1,
+    "region_query_1mbp_ms": 312.4,
+    "batch_query_100_ms": 198.7,
+    "annotation_100_variants_ms": 523.1
+  },
+  "timestamp": "2026-03-16T10:00:00Z"
+}
+```
+
+---
+
+## Interpreting Results
+
+| Metric | Target | Notes |
+|--------|--------|-------|
+| `point_query_cold_ms` | < 100 ms | First query after cold start; includes Parquet I/O |
+| `point_query_warm_ms` | < 20 ms | Subsequent queries; OS page cache active |
+| `region_query_1mbp_ms` | < 500 ms | Depends on variant density in region |
+| `batch_query_100_ms` | < 300 ms | 100 pre-specified variants on same chromosome |
+| `annotation_100_variants_ms` | < 2000 ms | VCF annotation, single-threaded |
+
+If `point_query_cold_ms` exceeds 500 ms, check disk I/O performance. If `point_query_warm_ms` is slow, check available RAM for OS page cache.
+
+---
+
+## Synthetic Data Generation
+
+The benchmark generates:
+- `N` samples with random sex (50/50) and random technology assignment
+- `M` variants per chromosome with uniformly random positions
+- Random genotypes with configurable carrier rates
+
+Synthetic data is written to a temporary directory and cleaned up after the benchmark completes.
+
+---
+
+## Tracking Performance Over Time
+
+Run the benchmark after major updates to detect regressions:
+
+```bash
+# Before update
+afquery benchmark --db-dir ./db/ --output before.json
+
+# After adding 500 samples
+afquery update-db --db ./db/ --add-samples new_batch.tsv
+afquery benchmark --db-dir ./db/ --output after.json
+
+# Compare
+python3 -c "
+import json
+before = json.load(open('before.json'))['queries']
+after  = json.load(open('after.json'))['queries']
+for k in before:
+    delta = after[k] - before[k]
+    print(f'{k}: {before[k]:.1f} → {after[k]:.1f} ms  ({delta:+.1f})')
+"
+```

afquery-0.1.2.2/docs/advanced/filter-pass-tracking.md

@@ -0,0 +1,111 @@
+# FILTER=PASS Tracking
+
+Schema version 2 introduces tracking of variants that are called but fail quality filters (`FILTER≠PASS`). This allows distinguishing between "variant not present" and "variant present but low quality" in your cohort.
+
+---
+
+## Background: VCF FILTER Field
+
+In VCF format, the FILTER column indicates whether a variant call passed quality filters:
+- `PASS` or `.` (missing) — the variant passed all filters
+- Any other value (e.g., `LowQual`, `VQSRTrancheSNP99.90to100.00`) — the variant failed one or more filters
+
+AFQuery default behavior:
+- **PASS-only** (default): only `FILTER=PASS` variants are counted in AC/AN
+- **All filters** (`--include-all-filters`): all variant calls are counted regardless of FILTER status
+
+---
+
+## Schema v2: fail_bitmap
+
+In schema v2, AFQuery stores a third bitmap per variant alongside `het_bitmap` and `hom_bitmap`:
+
+- **`fail_bitmap`** — bit set for each sample that has a non-ref genotype (AC>0) AND `FILTER≠PASS`
+
+This means:
+- A sample in `fail_bitmap` was genotyped with the alt allele but the call failed QC
+- Such samples are **not** counted in AC/AN (they don't affect AF)
+- Their count is exposed as `FAIL_SAMPLES` / `N_FAIL`
+
+---
+
+## Enabling Schema v2
+
+Schema v2 is the default for new databases. The `fail_bitmap` is always written; its content depends on the `--include-all-filters` flag:
+
+```bash
+# Default: PASS-only ingestion (fail_bitmap tracks failed calls)
+afquery create-db --manifest manifest.tsv --output-dir ./db/ --genome-build GRCh38
+
+# Include all filters: AC/AN counts all calls; fail_bitmap is all-zeros
+afquery create-db --manifest manifest.tsv --output-dir ./db/ --genome-build GRCh38 \
+  --include-all-filters
+```
+
+---
+
+## Querying FAIL_SAMPLES
+
+### CLI
+
+The `FAIL_SAMPLES` count is shown automatically in query output for v2 databases:
+
+```bash
+afquery query --db ./db/ --chrom chr1 --pos 925952
+```
+
+```
+chr1:925952
+  REF=G  ALT=A  AC=142  AN=2742  AF=0.0518  N_HET=138  N_HOM_ALT=2  FAIL=7
+```
+
+`FAIL=7` means 7 eligible samples had the alt allele called but with FILTER≠PASS.
+
+### Python API
+
+```python
+results = db.query("chr1", pos=925952)
+for r in results:
+    print(f"AC={r.AC}  AN={r.AN}  FAIL={r.N_FAIL}")
+    if r.N_FAIL is not None and r.N_FAIL > 0:
+        print(f"  Warning: {r.N_FAIL} samples have low-quality calls at this site")
+```
+
+`N_FAIL` is `0` for v1 databases (no fail tracking), not `None`. Use `QueryResult.N_FAIL` directly.
+
+---
+
+## VCF Annotation
+
+Schema v2 databases add an additional INFO field to annotated VCFs:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `AFQUERY_N_FAIL` | Integer | Eligible samples with FILTER≠PASS at this variant |
+
+```bash
+afquery annotate --db ./db/ --input variants.vcf --output annotated.vcf
+```
+
+v1 databases do not write `AFQUERY_N_FAIL`.
+
+---
+
+## Migration from v1
+
+v1 databases (no `schema_version` in `manifest.json`, or `schema_version < 2.0`) are fully compatible with AFQuery v2 code. They return `N_FAIL=0` in query results and do not write `AFQUERY_N_FAIL` in annotation.
+
+There is no automatic migration from v1 to v2. To gain fail tracking, rebuild the database with `create-db`.
+
+---
+
+## When to Use --include-all-filters
+
+By default (PASS-only), AF reflects the quality-filtered allele frequency — the frequency of the alt allele among high-quality calls. This is the recommended setting for most clinical and research use cases.
+
+Use `--include-all-filters` when:
+- You want to count all called genotypes regardless of quality
+- You are studying filter performance or variant calling artifacts
+- Your upstream pipeline handles filtering separately
+
+With `--include-all-filters`, `fail_bitmap` stores zeros for all variants (since all calls are counted in AC/AN).