ag2kit 0.1.0__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (28) hide show
  1. {ag2kit-0.1.0 → ag2kit-0.2.0}/PKG-INFO +40 -10
  2. {ag2kit-0.1.0 → ag2kit-0.2.0}/README.md +37 -6
  3. ag2kit-0.2.0/benchmarks/bench.py +252 -0
  4. ag2kit-0.2.0/benchmarks/coverage_sim.py +104 -0
  5. {ag2kit-0.1.0 → ag2kit-0.2.0}/docs/api.md +33 -4
  6. ag2kit-0.2.0/docs/methods.md +125 -0
  7. {ag2kit-0.1.0 → ag2kit-0.2.0}/pyproject.toml +16 -6
  8. {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/__init__.py +14 -2
  9. {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/client.py +233 -123
  10. ag2kit-0.2.0/src/ag2kit/evaluate.py +568 -0
  11. {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/gtex.py +15 -6
  12. {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/io.py +4 -8
  13. {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/stats.py +41 -9
  14. {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/variants.py +146 -17
  15. {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_ag2kit.py +128 -0
  16. {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_client.py +186 -0
  17. {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_evaluate.py +161 -0
  18. ag2kit-0.1.0/docs/methods.md +0 -97
  19. ag2kit-0.1.0/src/ag2kit/evaluate.py +0 -351
  20. {ag2kit-0.1.0 → ag2kit-0.2.0}/.gitignore +0 -0
  21. {ag2kit-0.1.0 → ag2kit-0.2.0}/LICENSE +0 -0
  22. {ag2kit-0.1.0 → ag2kit-0.2.0}/examples/gtex_eqtl_eval.py +0 -0
  23. {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/provenance.py +0 -0
  24. {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/py.typed +0 -0
  25. {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/conftest.py +0 -0
  26. {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/data/gtex_v10_mx1_whole_blood.json +0 -0
  27. {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_io.py +0 -0
  28. {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_release.py +0 -0
@@ -1,7 +1,7 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: ag2kit
3
- Version: 0.1.0
4
- Summary: Reproducible AlphaGenome 2 variant-effect analysis: VCF/Parquet input, resumable cached scoring, GTEx eQTL evaluation
3
+ Version: 0.2.0
4
+ Summary: Reproducible AlphaGenome 2 variant-effect analysis
5
5
  Project-URL: Homepage, https://github.com/zeyuyaoy/alphagenome-research
6
6
  Project-URL: Issues, https://github.com/zeyuyaoy/alphagenome-research/issues
7
7
  Author: Peter Yao
@@ -20,9 +20,8 @@ Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
20
20
  Classifier: Typing :: Typed
21
21
  Requires-Python: >=3.11
22
22
  Requires-Dist: numpy>=1.26
23
- Requires-Dist: pandas-stubs~=2.3.3
24
23
  Requires-Dist: pandas>=2.1
25
- Requires-Dist: pyarrow>=14
24
+ Requires-Dist: pyarrow>=15
26
25
  Requires-Dist: requests>=2.31
27
26
  Requires-Dist: scipy>=1.11
28
27
  Provides-Extra: model
@@ -31,7 +30,7 @@ Provides-Extra: vcf
31
30
  Requires-Dist: pysam>=0.22; extra == 'vcf'
32
31
  Description-Content-Type: text/markdown
33
32
 
34
- ### ag2kit: reproducible variant-effect analysis with the AlphaGenome SDK
33
+ ### ag2kit: Reproducible AlphaGenome 2 variant-effect analysis
35
34
 
36
35
  AlphaGenome predicts how non-coding DNA variants affect gene regulation [1]. AlphaGenome 2 also supports predictions conditioned on a reference cell type, tissue, or single-cell dataset [4]. Its SDK handles individual predictions and batch scoring. Evaluating those predictions against observed effects requires additional steps, including input validation, allele matching, error handling, and uncertainty estimation that accounts for linkage disequilibrium.
37
36
 
@@ -42,9 +41,9 @@ The package covers four tasks:
42
41
  | Capability | Description |
43
42
  |------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
44
43
  | Variant input | `read_variants` loads VCF/BCF (via `pysam`), delimited text, Parquet, and DataFrames into a common GRCh38 schema. It splits multiallelic records, assigns an `input_status` to every row, and optionally checks REF alleles against an indexed FASTA. |
45
- | Scoring | `AG2Client.score_variants` caches complete SDK results for each unique request. Scoring one variant against multiple genes uses one API call. Runs can resume after interruption, and failed requests receive error records. Output rows retain their input order. |
44
+ | Scoring | `AG2Client.score_variants` caches complete SDK results for each unique request. Scoring one variant against multiple genes uses one API call. Memory stays bounded for large batches, runs resume after interruption, and failed requests receive error records. Output rows retain their input order. |
46
45
  | Observed effects | `ag2kit.gtex` retrieves significant single-tissue eQTLs from GTEx Portal API v2 (GTEx v8, v10, and v11) or reads downloaded significant-pairs files [2]. |
47
- | Evaluation | `evaluate` orients observed effects to the predicted ALT allele and records a `pair_status` for every pair. It reports sign AUROC, signed Spearman correlation, and sign accuracy, with gene-cluster bootstrap intervals and optional tissue weighting. |
46
+ | Evaluation | `evaluate` orients observed effects to the predicted ALT allele and records a `pair_status` for every pair. It reports sign AUROC, signed Spearman correlation, and sign accuracy, with gene-cluster bootstrap intervals, stratified results, and paired comparisons between prediction sets. |
48
47
 
49
48
  > [!NOTE]
50
49
  > Use the official SDK for single predictions, track plots, in-silico mutagenesis, and gene annotation (`alphagenome.data.gene_annotation`). Use separate tools for liftover and summary-statistic quality control. For indel normalisation, use `bcftools norm` [3].
@@ -155,7 +154,37 @@ print(result.metrics[["metric", "estimate", "ci95_low", "ci95_high", "n_clusters
155
154
  print(result.pairs[["pos", "beta", "beta_alt", "allele_flipped", "pair_status"]])
156
155
  ```
157
156
 
158
- Confidence intervals are `NaN` when fewer than five gene clusters are available.
157
+ Confidence intervals are `NaN` with fewer than 20 clusters (`min_clusters`), because percentile intervals from few clusters are too narrow. Here there are only two genes.
158
+
159
+ #### Stratified and paired evaluation
160
+
161
+ `evaluate` is `join` followed by `evaluate_pairs`. Calling them separately lets you add columns to the pair ledger and stratify by them. `compare` estimates the difference between two prediction sets on the same observations, using the same cluster resamples for both:
162
+
163
+ ```python
164
+ import numpy as np
165
+ import pandas as pd
166
+
167
+ import ag2kit
168
+
169
+ rng = np.random.default_rng(0)
170
+ genes = np.repeat([f"G{i}" for i in range(40)], 5)
171
+ beta = np.repeat(rng.normal(size=40), 5) + rng.normal(scale=0.5, size=200)
172
+ keys = pd.DataFrame(
173
+ {"chrom": "chr1", "pos": np.arange(1, 201), "ref": "A", "alt": "G", "gene_id": genes}
174
+ )
175
+ obs = keys.assign(beta=beta, effect_allele="G")
176
+ good = keys.assign(raw_effect=beta + rng.normal(size=200), status="completed")
177
+ noisy = keys.assign(raw_effect=beta + rng.normal(scale=3, size=200), status="completed")
178
+
179
+ pairs = ag2kit.join(good, obs)
180
+ pairs["magnitude"] = pd.qcut(pairs["predicted"].abs(), 2, labels=["small", "large"])
181
+ by_size = ag2kit.evaluate_pairs(pairs, group="magnitude", n_boot=200)
182
+ print(by_size.per_group[["metric", "group", "n", "value", "ci95_low", "ci95_high"]])
183
+
184
+ print(ag2kit.compare(pairs, ag2kit.join(noisy, obs), n_boot=200))
185
+ ```
186
+
187
+ Comparing two scorers, track filters, or conditions this way accounts for the dependence between their predictions, which the McNemar test in `ag2kit.stats` does not. To check performance on regions held out from training, score with `AG2Client(model_version="FOLD_1")`, add `ag2kit.annotate_folds(pairs)` labels, and evaluate with `group="fold_subset"`. See [Methods](docs/methods.md#held-out-regions) for the caveats.
159
188
 
160
189
  #### Conditioning
161
190
 
@@ -164,12 +193,13 @@ Each AlphaGenome 2 request uses one conditioning input. You can choose one of tw
164
193
  1. **Reference condition:** Set `condition` to an ontology CURIE such as `UBERON:0013756`, or a composite such as `UBERON:0000178_CL:0001054`. Use a scorer such as `RNA_SEQ`.
165
194
  2. **Single-cell data:** Set `conditioning` to an `AnnData` object containing raw UMI counts. Use Ensembl IDs for `var_names` and include up to 256 cells. Set `condition_label` and use `SC_GENE_EXPRESSION_DENOISED`.
166
195
 
167
- To find valid reference conditions, call `supported_conditions()`. For finer control, build `RequestSpec` objects and use `score` or `score_many`. You can then inspect cached SDK tracks through `AG2Client.raw_scores`.
196
+ To find valid reference conditions, call `client.connect()` and then the SDK method `client.model.supported_conditions()`. For finer control, build `RequestSpec` objects and use `score` or `score_many`. You can then inspect cached SDK tracks through `AG2Client.raw_scores`.
168
197
 
169
198
  #### Interpretation and limitations
170
199
 
171
200
  - `RNA_SEQ` predicts log fold changes, while GTEx reports slopes for normalised expression. Accordingly, `ag2kit` compares the direction and rank of effects.
172
- - GTEx significant-pairs files contain selected eQTL associations, and many variants linked to the same gene are in linkage disequilibrium. As a result, sign metrics may be higher than those measured across all tested pairs. By default, bootstrap intervals resample whole genes to account for within-gene dependence.
201
+ - GTEx significant-pairs files contain selected eQTL associations, and many variants linked to the same gene are in linkage disequilibrium. As a result, sign metrics may be higher than those measured across all tested pairs. By default, bootstrap intervals resample whole genes to account for within-gene dependence; use locus clusters (`locus_clusters`) when variants are shared across genes.
202
+ - Bootstrap intervals describe sampling uncertainty over the evaluated loci. They do not include model-output variability, such as differences between checkpoints, track sets, or conditioning draws.
173
203
  - Sign agreement describes performance at the tested loci. It does not establish causality or resolve individual variant effects. In addition, the p-values from `concordance`, `spearman`, and `mcnemar` assume independent pairs, so `evaluate` leaves them out.
174
204
  - All coordinates are GRCh38 and 1-based. The package performs no liftover. The API does not expose a model revision, so reproducible reruns depend on cached results ([Reproducibility](docs/api.md#reproducibility)). Tests cover the implementation, but the package has not reproduced a published AlphaGenome benchmark. See [Implementation validation](docs/methods.md#implementation-validation) and [Scope and limitations](docs/methods.md#scope-and-limitations).
175
205
  - Finally, AlphaGenome outputs are subject to the SDK's usage restrictions: non-commercial research only, with no use for training other machine-learning models or making clinical decisions. The AlphaGenome terms also govern cached and shared outputs. When using GTEx data, acknowledge the GTEx Project [2].
@@ -1,4 +1,4 @@
1
- ### ag2kit: reproducible variant-effect analysis with the AlphaGenome SDK
1
+ ### ag2kit: Reproducible AlphaGenome 2 variant-effect analysis
2
2
 
3
3
  AlphaGenome predicts how non-coding DNA variants affect gene regulation [1]. AlphaGenome 2 also supports predictions conditioned on a reference cell type, tissue, or single-cell dataset [4]. Its SDK handles individual predictions and batch scoring. Evaluating those predictions against observed effects requires additional steps, including input validation, allele matching, error handling, and uncertainty estimation that accounts for linkage disequilibrium.
4
4
 
@@ -9,9 +9,9 @@ The package covers four tasks:
9
9
  | Capability | Description |
10
10
  |------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
11
11
  | Variant input | `read_variants` loads VCF/BCF (via `pysam`), delimited text, Parquet, and DataFrames into a common GRCh38 schema. It splits multiallelic records, assigns an `input_status` to every row, and optionally checks REF alleles against an indexed FASTA. |
12
- | Scoring | `AG2Client.score_variants` caches complete SDK results for each unique request. Scoring one variant against multiple genes uses one API call. Runs can resume after interruption, and failed requests receive error records. Output rows retain their input order. |
12
+ | Scoring | `AG2Client.score_variants` caches complete SDK results for each unique request. Scoring one variant against multiple genes uses one API call. Memory stays bounded for large batches, runs resume after interruption, and failed requests receive error records. Output rows retain their input order. |
13
13
  | Observed effects | `ag2kit.gtex` retrieves significant single-tissue eQTLs from GTEx Portal API v2 (GTEx v8, v10, and v11) or reads downloaded significant-pairs files [2]. |
14
- | Evaluation | `evaluate` orients observed effects to the predicted ALT allele and records a `pair_status` for every pair. It reports sign AUROC, signed Spearman correlation, and sign accuracy, with gene-cluster bootstrap intervals and optional tissue weighting. |
14
+ | Evaluation | `evaluate` orients observed effects to the predicted ALT allele and records a `pair_status` for every pair. It reports sign AUROC, signed Spearman correlation, and sign accuracy, with gene-cluster bootstrap intervals, stratified results, and paired comparisons between prediction sets. |
15
15
 
16
16
  > [!NOTE]
17
17
  > Use the official SDK for single predictions, track plots, in-silico mutagenesis, and gene annotation (`alphagenome.data.gene_annotation`). Use separate tools for liftover and summary-statistic quality control. For indel normalisation, use `bcftools norm` [3].
@@ -122,7 +122,37 @@ print(result.metrics[["metric", "estimate", "ci95_low", "ci95_high", "n_clusters
122
122
  print(result.pairs[["pos", "beta", "beta_alt", "allele_flipped", "pair_status"]])
123
123
  ```
124
124
 
125
- Confidence intervals are `NaN` when fewer than five gene clusters are available.
125
+ Confidence intervals are `NaN` with fewer than 20 clusters (`min_clusters`), because percentile intervals from few clusters are too narrow. Here there are only two genes.
126
+
127
+ #### Stratified and paired evaluation
128
+
129
+ `evaluate` is `join` followed by `evaluate_pairs`. Calling them separately lets you add columns to the pair ledger and stratify by them. `compare` estimates the difference between two prediction sets on the same observations, using the same cluster resamples for both:
130
+
131
+ ```python
132
+ import numpy as np
133
+ import pandas as pd
134
+
135
+ import ag2kit
136
+
137
+ rng = np.random.default_rng(0)
138
+ genes = np.repeat([f"G{i}" for i in range(40)], 5)
139
+ beta = np.repeat(rng.normal(size=40), 5) + rng.normal(scale=0.5, size=200)
140
+ keys = pd.DataFrame(
141
+ {"chrom": "chr1", "pos": np.arange(1, 201), "ref": "A", "alt": "G", "gene_id": genes}
142
+ )
143
+ obs = keys.assign(beta=beta, effect_allele="G")
144
+ good = keys.assign(raw_effect=beta + rng.normal(size=200), status="completed")
145
+ noisy = keys.assign(raw_effect=beta + rng.normal(scale=3, size=200), status="completed")
146
+
147
+ pairs = ag2kit.join(good, obs)
148
+ pairs["magnitude"] = pd.qcut(pairs["predicted"].abs(), 2, labels=["small", "large"])
149
+ by_size = ag2kit.evaluate_pairs(pairs, group="magnitude", n_boot=200)
150
+ print(by_size.per_group[["metric", "group", "n", "value", "ci95_low", "ci95_high"]])
151
+
152
+ print(ag2kit.compare(pairs, ag2kit.join(noisy, obs), n_boot=200))
153
+ ```
154
+
155
+ Comparing two scorers, track filters, or conditions this way accounts for the dependence between their predictions, which the McNemar test in `ag2kit.stats` does not. To check performance on regions held out from training, score with `AG2Client(model_version="FOLD_1")`, add `ag2kit.annotate_folds(pairs)` labels, and evaluate with `group="fold_subset"`. See [Methods](docs/methods.md#held-out-regions) for the caveats.
126
156
 
127
157
  #### Conditioning
128
158
 
@@ -131,12 +161,13 @@ Each AlphaGenome 2 request uses one conditioning input. You can choose one of tw
131
161
  1. **Reference condition:** Set `condition` to an ontology CURIE such as `UBERON:0013756`, or a composite such as `UBERON:0000178_CL:0001054`. Use a scorer such as `RNA_SEQ`.
132
162
  2. **Single-cell data:** Set `conditioning` to an `AnnData` object containing raw UMI counts. Use Ensembl IDs for `var_names` and include up to 256 cells. Set `condition_label` and use `SC_GENE_EXPRESSION_DENOISED`.
133
163
 
134
- To find valid reference conditions, call `supported_conditions()`. For finer control, build `RequestSpec` objects and use `score` or `score_many`. You can then inspect cached SDK tracks through `AG2Client.raw_scores`.
164
+ To find valid reference conditions, call `client.connect()` and then the SDK method `client.model.supported_conditions()`. For finer control, build `RequestSpec` objects and use `score` or `score_many`. You can then inspect cached SDK tracks through `AG2Client.raw_scores`.
135
165
 
136
166
  #### Interpretation and limitations
137
167
 
138
168
  - `RNA_SEQ` predicts log fold changes, while GTEx reports slopes for normalised expression. Accordingly, `ag2kit` compares the direction and rank of effects.
139
- - GTEx significant-pairs files contain selected eQTL associations, and many variants linked to the same gene are in linkage disequilibrium. As a result, sign metrics may be higher than those measured across all tested pairs. By default, bootstrap intervals resample whole genes to account for within-gene dependence.
169
+ - GTEx significant-pairs files contain selected eQTL associations, and many variants linked to the same gene are in linkage disequilibrium. As a result, sign metrics may be higher than those measured across all tested pairs. By default, bootstrap intervals resample whole genes to account for within-gene dependence; use locus clusters (`locus_clusters`) when variants are shared across genes.
170
+ - Bootstrap intervals describe sampling uncertainty over the evaluated loci. They do not include model-output variability, such as differences between checkpoints, track sets, or conditioning draws.
140
171
  - Sign agreement describes performance at the tested loci. It does not establish causality or resolve individual variant effects. In addition, the p-values from `concordance`, `spearman`, and `mcnemar` assume independent pairs, so `evaluate` leaves them out.
141
172
  - All coordinates are GRCh38 and 1-based. The package performs no liftover. The API does not expose a model revision, so reproducible reruns depend on cached results ([Reproducibility](docs/api.md#reproducibility)). Tests cover the implementation, but the package has not reproduced a published AlphaGenome benchmark. See [Implementation validation](docs/methods.md#implementation-validation) and [Scope and limitations](docs/methods.md#scope-and-limitations).
142
173
  - Finally, AlphaGenome outputs are subject to the SDK's usage restrictions: non-commercial research only, with no use for training other machine-learning models or making clinical decisions. The AlphaGenome terms also govern cached and shared outputs. When using GTEx data, acknowledge the GTEx Project [2].
@@ -0,0 +1,252 @@
1
+ """Wall-time and peak-memory benchmark for ag2kit's public API.
2
+
3
+ Uses only public names so it runs unchanged against v0.1 and v0.2. Never calls
4
+ the AlphaGenome API: scoring goes through an in-process fake model.
5
+
6
+ uv run python benchmarks/bench.py # full run
7
+ uv run python benchmarks/bench.py --quick # CI smoke run (<~20 s)
8
+ PYTHONPATH=<checkout>/src uv run python benchmarks/bench.py # other version
9
+ """
10
+
11
+ from __future__ import annotations
12
+
13
+ import argparse
14
+ import os
15
+ import shutil
16
+ import subprocess
17
+ import sys
18
+ import tempfile
19
+ import threading
20
+ import time
21
+ import tracemalloc
22
+ import zlib
23
+ from pathlib import Path
24
+
25
+ import numpy as np
26
+ import pandas as pd
27
+
28
+ # Keep any user cache overrides from redirecting benchmark I/O.
29
+ for _var in ("AG2KIT_AG2_CACHE", "AG2KIT_RESULTS_ROOT"):
30
+ os.environ.pop(_var, None)
31
+
32
+ import ag2kit # noqa: E402
33
+
34
+ N_GENES, N_TRACKS = 30, 50
35
+
36
+
37
+ def scorer_output(x, obs, var, scorer, variant):
38
+ """SDK-shaped AnnData accepted by variant_scorers.tidy_anndata (copied from tests)."""
39
+ import anndata as ad
40
+ from alphagenome.data import genome
41
+ from alphagenome.models.v2 import variant_scorers
42
+
43
+ obs, var = pd.DataFrame(obs), pd.DataFrame(var)
44
+ obs.index, var.index = obs.index.astype(str), var.index.astype(str)
45
+ adata = ad.AnnData(X=np.asarray(x, dtype=float), obs=obs, var=var)
46
+ adata.uns["interval"] = genome.Interval("chr21", 40897866, 41946442)
47
+ adata.uns["variant"] = genome.Variant.from_str(str(variant))
48
+ adata.uns["variant_scorer"] = variant_scorers.RECOMMENDED_VARIANT_SCORERS[scorer]
49
+ return adata
50
+
51
+
52
+ class FakeModel:
53
+ """SDK stand-in returning N_GENES x N_TRACKS scores seeded by the variant."""
54
+
55
+ def __init__(self):
56
+ self.calls = 0
57
+ self.lock = threading.Lock()
58
+ strand = np.array(["+", "-", "."])[np.arange(N_TRACKS) % 3]
59
+ self.var = {
60
+ "strand": strand,
61
+ "name": [f"track{j}" for j in range(N_TRACKS)],
62
+ "imputation_mode": ["none"] * N_TRACKS,
63
+ }
64
+
65
+ def score_variant(self, **kw):
66
+ with self.lock:
67
+ self.calls += 1
68
+ v = kw["variant"]
69
+ rng = np.random.default_rng(zlib.crc32(str(v).encode()))
70
+ locus = v.position // 100_000
71
+ genes = [f"ENSG{locus * 100 + k:011d}" for k in range(N_GENES)]
72
+ obs = {
73
+ "gene_id": genes,
74
+ "strand": np.array(["+", "-"])[np.arange(N_GENES) % 2],
75
+ "gene_name": [f"G{locus}_{k}" for k in range(N_GENES)],
76
+ }
77
+ x = rng.normal(size=(N_GENES, N_TRACKS))
78
+ scorer = "SC_GENE_EXPRESSION_DENOISED" if "cell_conditioning" in kw else "RNA_SEQ"
79
+ return [scorer_output(x, obs, self.var, scorer, str(v))]
80
+
81
+
82
+ def make_variants(n, seed=0):
83
+ rng = np.random.default_rng(seed)
84
+ pos = rng.choice(np.arange(1_000_000, 40_000_000), size=n, replace=False)
85
+ ref = rng.choice(list("ACGT"), size=n)
86
+ alt = (pd.Series(ref).map({"A": "C", "C": "G", "G": "T", "T": "A"})).to_numpy()
87
+ return pd.DataFrame(
88
+ {
89
+ "variant_id": [f"v{i}" for i in range(n)],
90
+ "chrom": "chr1",
91
+ "pos": pos,
92
+ "ref": ref,
93
+ "alt": alt,
94
+ }
95
+ )
96
+
97
+
98
+ def make_cells(n_cells=256, n_genes=20_000, seed=0):
99
+ import anndata as ad
100
+ import scipy.sparse as sp
101
+
102
+ rng = np.random.default_rng(seed)
103
+ x = sp.random(n_cells, n_genes, density=0.1, format="csr", dtype=np.float32, random_state=rng)
104
+ x.data = rng.poisson(3.0, size=x.data.size).astype(np.float32) + 1
105
+ adata = ad.AnnData(X=x)
106
+ adata.var_names = [f"ENSG{i:011d}" for i in range(n_genes)]
107
+ adata.obs_names = [f"cell{i}" for i in range(n_cells)]
108
+ return adata
109
+
110
+
111
+ def make_eval(n_pairs, n_genes, seed=0):
112
+ rng = np.random.default_rng(seed)
113
+ gene_idx = rng.integers(0, n_genes, n_pairs)
114
+ g_eff = rng.normal(size=n_genes)
115
+ beta = g_eff[gene_idx] + rng.normal(size=n_pairs)
116
+ pred = 0.5 * g_eff[gene_idx] + rng.normal(size=n_pairs)
117
+ ref = rng.choice(list("ACGT"), size=n_pairs)
118
+ alt = pd.Series(ref).map({"A": "C", "C": "G", "G": "T", "T": "A"}).to_numpy()
119
+ keys = pd.DataFrame(
120
+ {
121
+ "chrom": "chr1",
122
+ "pos": np.arange(n_pairs) * 50 + 1_000_000,
123
+ "ref": ref,
124
+ "alt": alt,
125
+ "gene_id": [f"ENSG{g:011d}" for g in gene_idx],
126
+ }
127
+ )
128
+ predictions = keys.assign(raw_effect=pred, status="completed")
129
+ tissues = np.array([f"tissue{t}" for t in range(5)])
130
+ observed = keys.assign(
131
+ beta=beta, effect_allele=alt, tissue=tissues[rng.integers(0, 5, n_pairs)]
132
+ )
133
+ return predictions, observed
134
+
135
+
136
+ def make_loci(n, seed=0):
137
+ rng = np.random.default_rng(seed)
138
+ chrom = rng.integers(1, 23, n)
139
+ return pd.DataFrame(
140
+ {
141
+ "variant_id": [f"rs{i}" for i in range(n)],
142
+ "chrom": [f"chr{c}" for c in chrom],
143
+ "pos": rng.integers(1, 150_000_000, n),
144
+ "gene": [f"ENSG{g:011d}" for g in rng.integers(0, n // 10 + 1, n)],
145
+ "pval": rng.uniform(size=n),
146
+ }
147
+ )
148
+
149
+
150
+ def measure(fn, setup=lambda: None, repeats=1):
151
+ """Best-of-`repeats` untraced wall time plus one traced run for peak memory.
152
+
153
+ repeats=0 reports the traced run's own time (slow cases; tracemalloc adds overhead).
154
+ """
155
+ times = []
156
+ for _ in range(repeats):
157
+ arg = setup()
158
+ t0 = time.perf_counter()
159
+ fn(arg)
160
+ times.append(time.perf_counter() - t0)
161
+ arg = setup()
162
+ tracemalloc.start()
163
+ t0 = time.perf_counter()
164
+ fn(arg)
165
+ times.append(time.perf_counter() - t0) if not repeats else None
166
+ peak = tracemalloc.get_traced_memory()[1]
167
+ tracemalloc.stop()
168
+ return min(times), peak / 2**20
169
+
170
+
171
+ def main(argv=None):
172
+ ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
173
+ ap.add_argument("--n", type=float, default=1.0, help="scale factor for problem sizes")
174
+ ap.add_argument("--quick", action="store_true", help="small sizes for a CI smoke run")
175
+ ap.add_argument("--only", default="", help="comma-separated case prefixes (a,b,c,d)")
176
+ args = ap.parse_args(argv)
177
+ s = args.n * (0.01 if args.quick else 1.0)
178
+ sz = lambda n: max(1, int(round(n * s))) # noqa: E731
179
+ only = set(filter(None, args.only.split(",")))
180
+ want = lambda c: not only or c in only # noqa: E731
181
+
182
+ root = Path(tempfile.mkdtemp(prefix="ag2kit-bench-"))
183
+ os.environ["AG2KIT_CACHE"] = str(root / "ag2kit-cache")
184
+ fresh = lambda: Path(tempfile.mkdtemp(dir=root)) # noqa: E731
185
+ src = Path(ag2kit.__file__).resolve().parent
186
+ sha = (
187
+ subprocess.run(
188
+ ["git", "-C", str(src), "rev-parse", "--short", "HEAD"], capture_output=True, text=True
189
+ ).stdout.strip()
190
+ or "unknown"
191
+ )
192
+ print(f"ag2kit {ag2kit.__version__} @ {sha} from {ag2kit.__file__}")
193
+ print(f"python {sys.version.split()[0]}, scale={s:g}\n")
194
+ rows = []
195
+
196
+ def client(cache):
197
+ return ag2kit.AG2Client(dry_run=False, model=FakeModel(), cache_dir=cache, backoff=0)
198
+
199
+ def score(cache, variants, **kw):
200
+ out = client(cache).score_variants(variants, **kw)
201
+ assert (out["status"] == "completed").all(), out["status"].value_counts()
202
+ return out
203
+
204
+ if want("a"):
205
+ kw = dict(condition="UBERON:0013756", scorer="RNA_SEQ")
206
+ for n in (sz(500), sz(2000)):
207
+ v = make_variants(n)
208
+ reps = 3 if n <= 10 or n == 500 else 1
209
+ t, m = measure(lambda c, v=v: score(c, v, **kw), fresh, repeats=reps)
210
+ rows.append(("a score_variants RNA_SEQ cold", n, t, m))
211
+ warm = fresh()
212
+ score(warm, v, **kw)
213
+ t, m = measure(lambda c, v=v: score(c, v, **kw), lambda w=warm: w, repeats=reps)
214
+ rows.append(("a score_variants RNA_SEQ warm", n, t, m))
215
+
216
+ if want("b"):
217
+ # ponytail: one traced run each (cold, then warm on the same cache); this case is slow.
218
+ n = 1 if args.quick else sz(200)
219
+ v, cells = make_variants(n, seed=1), make_cells()
220
+ kw = dict(
221
+ scorer="SC_GENE_EXPRESSION_DENOISED", conditioning=cells, condition_label="control"
222
+ )
223
+ cache = fresh()
224
+ t, m = measure(lambda c: score(c, v, **kw), lambda: cache, repeats=0)
225
+ rows.append(("b score_variants cells 256x20000 cold (traced time)", n, t, m))
226
+ t, m = measure(lambda c: score(c, v, **kw), lambda: cache, repeats=0)
227
+ rows.append(("b score_variants cells 256x20000 warm (traced time)", n, t, m))
228
+
229
+ if want("c"):
230
+ n = sz(20_000)
231
+ pred, obs = make_eval(n, max(5, n // 10))
232
+ nb = 1000 if not args.quick else 50
233
+ for group in (None, "tissue"):
234
+ fn = lambda _, g=group: ag2kit.evaluate(pred, obs, n_boot=nb, group=g) # noqa: E731
235
+ t, m = measure(fn, repeats=1)
236
+ rows.append((f"c evaluate n_boot={nb} group={group}", n, t, m))
237
+
238
+ if want("d"):
239
+ n = sz(50_000)
240
+ loci = make_loci(n)
241
+ t, m = measure(lambda _: ag2kit.cluster_loci(loci), repeats=1 if n > 10_000 else 3)
242
+ rows.append(("d cluster_loci", n, t, m))
243
+
244
+ print("| case | N | seconds | peak MiB |")
245
+ print("|---|---:|---:|---:|")
246
+ for case, n, t, m in rows:
247
+ print(f"| {case} | {n} | {t:.3f} | {m:.1f} |")
248
+ shutil.rmtree(root, ignore_errors=True)
249
+
250
+
251
+ if __name__ == "__main__":
252
+ main()
@@ -0,0 +1,104 @@
1
+ """Empirical 95% CI coverage of ag2kit.evaluate's sign-AUROC bootstrap.
2
+
3
+ Compares the cluster bootstrap (cluster="gene_id") with the pair bootstrap
4
+ (cluster=None) on data where pairs within a gene share a gene-level random
5
+ effect in both predicted and observed effects.
6
+
7
+ uv run python benchmarks/coverage_sim.py # 200 replicates, n_boot=500
8
+ uv run python benchmarks/coverage_sim.py --quick # smoke run
9
+ """
10
+
11
+ from __future__ import annotations
12
+
13
+ import argparse
14
+ import time
15
+
16
+ import numpy as np
17
+ import pandas as pd
18
+ from scipy.stats import mannwhitneyu
19
+
20
+ import ag2kit
21
+
22
+ # Generative model: gene effect u_g drives beta; prediction sees u_g plus its
23
+ # own gene-level error v_g, so both sides are correlated within a gene.
24
+ SD_BETA_NOISE, PRED_SLOPE, SD_PRED_GENE, SD_PRED_NOISE = 0.5, 0.6, 0.8, 0.5
25
+
26
+
27
+ def draw(n_genes, per_gene, rng):
28
+ g = np.repeat(np.arange(n_genes), per_gene)
29
+ u, v = rng.normal(size=n_genes), rng.normal(scale=SD_PRED_GENE, size=n_genes)
30
+ beta = u[g] + rng.normal(scale=SD_BETA_NOISE, size=g.size)
31
+ pred = PRED_SLOPE * u[g] + v[g] + rng.normal(scale=SD_PRED_NOISE, size=g.size)
32
+ return g, pred, beta
33
+
34
+
35
+ def auroc(pred, beta):
36
+ pos, neg = pred[beta > 0], pred[beta < 0]
37
+ return mannwhitneyu(pos, neg).statistic / (pos.size * neg.size)
38
+
39
+
40
+ def tables(g, pred, beta):
41
+ n = g.size
42
+ keys = pd.DataFrame(
43
+ {
44
+ "chrom": "chr1",
45
+ "pos": np.arange(n) * 100 + 1_000_000,
46
+ "ref": "A",
47
+ "alt": "G",
48
+ "gene_id": [f"ENSG{x:011d}" for x in g],
49
+ }
50
+ )
51
+ return keys.assign(raw_effect=pred, status="completed"), keys.assign(
52
+ beta=beta, effect_allele="G"
53
+ )
54
+
55
+
56
+ def main(argv=None):
57
+ ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
58
+ ap.add_argument("--reps", type=int, default=200)
59
+ ap.add_argument("--n-boot", type=int, default=500)
60
+ ap.add_argument("--genes", type=int, default=60)
61
+ ap.add_argument("--per-gene", type=int, default=6)
62
+ ap.add_argument("--quick", action="store_true", help="20 replicates x n_boot=200")
63
+ args = ap.parse_args(argv)
64
+ if args.quick:
65
+ args.reps, args.n_boot = 20, 200
66
+ print(f"ag2kit {ag2kit.__version__} from {ag2kit.__file__}")
67
+
68
+ # Population AUROC from one very large draw of the same model.
69
+ truth = auroc(*draw(500_000, args.per_gene, np.random.default_rng(12345))[1:])
70
+ print(
71
+ f"genes={args.genes} per_gene={args.per_gene} reps={args.reps} "
72
+ f"n_boot={args.n_boot}; true sign AUROC (3M-pair MC) = {truth:.4f}\n"
73
+ )
74
+
75
+ hits = {"gene_id": [], None: []}
76
+ widths = {"gene_id": [], None: []}
77
+ estimates = []
78
+ t0 = time.perf_counter()
79
+ for r in range(args.reps):
80
+ g, pred, beta = draw(args.genes, args.per_gene, np.random.default_rng(r))
81
+ predictions, observed = tables(g, pred, beta)
82
+ for cluster in hits:
83
+ m = ag2kit.evaluate(predictions, observed, cluster=cluster, n_boot=args.n_boot, seed=r)
84
+ row = m.metrics.set_index("metric").loc["sign_auroc"]
85
+ hits[cluster].append(row["ci95_low"] <= truth <= row["ci95_high"])
86
+ widths[cluster].append(row["ci95_high"] - row["ci95_low"])
87
+ estimates.append(row["estimate"])
88
+ secs = time.perf_counter() - t0
89
+
90
+ print("| bootstrap | coverage | binomial SE | mean CI width |")
91
+ print("|---|---:|---:|---:|")
92
+ for cluster, h in hits.items():
93
+ p = float(np.mean(h))
94
+ se = np.sqrt(p * (1 - p) / len(h))
95
+ name = "cluster (gene_id)" if cluster else "pair (cluster=None)"
96
+ print(f"| {name} | {p:.3f} | {se:.3f} | {np.mean(widths[cluster]):.4f} |")
97
+ print(
98
+ f"\nmean estimate {np.mean(estimates):.4f} (truth {truth:.4f}), "
99
+ f"empirical SD of estimate {np.std(estimates, ddof=1):.4f}; {secs:.0f} s"
100
+ )
101
+
102
+
103
+ if __name__ == "__main__":
104
+ main()
@@ -37,7 +37,7 @@ client = ag2kit.AG2Client(
37
37
  dry_run=True,
38
38
  cache_dir=...,
39
39
  model_version="ALL_FOLDS",
40
- max_retries=2,
40
+ max_retries=4,
41
41
  backoff=30.0,
42
42
  cache_namespace="",
43
43
  model=None,
@@ -46,6 +46,8 @@ client = ag2kit.AG2Client(
46
46
 
47
47
  `AG2Client` manages scoring requests and their cache. By default, `dry_run=True` plans requests without calling the API.
48
48
 
49
+ Batch methods fetch each distinct request once and reduce it to gene records as soon as it completes, so memory use depends on `max_workers`, not on the number of variants. An interrupt (Ctrl-C) cancels requests that have not started; completed requests stay cached, so rerunning resumes the batch. Retries apply to `quota`, `transient`, and `network` errors: attempt *k* waits a random delay between half and all of `backoff * 2**(k-1)` seconds, capped at 600 seconds. Progress is logged at `INFO` level on the `ag2kit` logger.
50
+
49
51
  | Method | Description |
50
52
  |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
51
53
  | `score_variants(variants, *, condition="", scorer="RNA_SEQ", gene_column=None, conditioning=None, condition_label="", sequence_length=2**20, max_workers=4, tracks=None)` | Score a table of variants. Returns one row per input row and gene, or one row per input row when `gene_column` identifies a gene. Input columns use the `input_` prefix. |
@@ -121,10 +123,21 @@ Supported datasets are `gtex_v8` (GENCODE v26), `gtex_v10` (v39), and `gtex_v11`
121
123
  | `tissues` | List tissue IDs, UBERON IDs, and eGene counts. |
122
124
  | `gencode_id` | Resolve a gene to its release-specific GENCODE identifier. |
123
125
 
126
+ Portal failures raise `ag2kit.gtex.SourceError`, a subclass of both `AG2KitError` and `RuntimeError`.
127
+
124
128
  `eqtls` resolves gene symbols and unversioned Ensembl IDs to the selected GENCODE release. Versioned Ensembl IDs (containing `.`) must match that release. `tissues` and `gencode_id` also accept `refresh`. GTEx responses are cached at `$AG2KIT_CACHE/gtex`.
125
129
 
126
130
  #### Evaluation
127
131
 
132
+ Evaluation has two steps, which `evaluate` combines:
133
+
134
+ ```python
135
+ pairs = ag2kit.join(predictions, observed, *, match=None, effect="raw_effect", forward_strand=True)
136
+ result = ag2kit.evaluate_pairs(pairs, *, group=None, cluster="gene_id", n_boot=1000, seed=0, min_group=2, min_clusters=20)
137
+ ```
138
+
139
+ `join` returns the pair ledger. Add columns to it before calling `evaluate_pairs` to stratify by anything you can compute, such as effect-size bins, distance to the TSS, or `annotate_folds` labels.
140
+
128
141
  ```python
129
142
  ag2kit.evaluate(
130
143
  predictions,
@@ -138,7 +151,7 @@ ag2kit.evaluate(
138
151
  n_boot=1000,
139
152
  seed=0,
140
153
  min_group=2,
141
- min_clusters=5,
154
+ min_clusters=20,
142
155
  ) -> Evaluation
143
156
  ```
144
157
 
@@ -157,15 +170,27 @@ Use `match` to require agreement on additional columns. For example, `match={"on
157
170
  |---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
158
171
  | `pairs` | One row per observed record, including `predicted`, `beta_alt`, `allele_flipped`, `pair_status`, and matched prediction fields prefixed with `pred_`. |
159
172
  | `metrics` | `sign_auroc`, `spearman_rho`, and `sign_accuracy`, with estimates and uncertainty fields. |
160
- | `per_group` | Group-level results: `group`, `n`, `value`, `metric`. |
173
+ | `per_group` | Group-level results: `group`, `n`, `value`, `metric`, `ci95_low`, `ci95_high`, `n_clusters`, `n_boot_finite`. |
161
174
  | `settings` | Evaluation configuration. |
162
175
  | `summary()` | Print metrics and counts by `pair_status`. |
163
176
  | `save(directory)` | Save `pairs.parquet`, `metrics.tsv`, `per_group.tsv`, and `evaluation.json`. |
164
177
 
165
178
  The `metrics` table includes `estimate`, `ci95_low`, `ci95_high`, `n_pairs`, `n_pairs_used`, `n_clusters`, `n_boot_finite`, and `weighting`. `evaluation.json` includes the settings, software versions, and write timestamp.
166
179
 
180
+ All metrics and groups use the same bootstrap resamples, so intervals do not depend on metric order. Intervals need at least `min_clusters` clusters overall, and per-group intervals need that many clusters within the group. Only groups that contribute in the original sample can contribute to a weighted bootstrap estimate. In each resample, groups below `min_group` or with undefined metrics are omitted and the remaining group weights are renormalized; the replicate is discarded if no eligible groups remain.
181
+
182
+ The ledger also carries the prediction's `pred_model_version`, `pred_track_filter`, and `pred_n_reduced_values` when present, so stratified and fold analyses can check what was scored.
183
+
167
184
  `pair_status` can be `ok`, `no_prediction`, `prediction_failed`, `ambiguous_prediction`, `ref_mismatch`, `missing_statistics`, `ref_equals_alt`, `not_snv`, `palindromic`, or `effect_allele_not_in_ref_alt`. The `not_snv` and `palindromic` statuses apply only when `forward_strand=False`.
168
185
 
186
+ ##### Comparing prediction sets
187
+
188
+ ```python
189
+ ag2kit.compare(pairs_a, pairs_b, *, group=None, cluster="gene_id", n_boot=1000, seed=0, min_group=2, min_clusters=20) -> pandas.DataFrame
190
+ ```
191
+
192
+ `compare` estimates `a − b` for each metric, using pairs that are usable in both ledgers and a paired cluster bootstrap. Both ledgers must come from `join` with the same observed table; misaligned ledgers raise `ValueError`. Each ledger is evaluated against its own ALT orientation. The result has `estimate_a`, `estimate_b`, `difference`, `ci95_low`, `ci95_high`, `n_pairs`, `n_usable_a_only`, `n_usable_b_only`, `n_clusters`, `n_boot_finite`, and `weighting`. Large `n_usable_*_only` counts mean the two prediction sets fail on different variants, so also report each set's own `evaluate_pairs` result. Use it to compare scorers, track filters, conditions, or model versions.
193
+
169
194
  #### Statistics and utilities
170
195
 
171
196
  | Function or type | Description |
@@ -177,7 +202,9 @@ The `metrics` table includes `estimate`, `ci95_low`, `ci95_high`, `n_pairs`, `n_
177
202
  | `ag2kit.stats.mcnemar` | Exact McNemar test for discordant pairs. |
178
203
  | `ag2kit.stats.monte_carlo_summary` | Summary across conditioning draws. |
179
204
  | `harmonize_effect(estimate, effect_allele, ref, alt, forward_strand=False)` | Orient an observed effect to ALT. The default is `False`; `evaluate` defaults to `True`. |
180
- | `cluster_loci(df, window=1_000_000)` | Greedy clustering by distance and gene, with one lead row per locus. This is not LD-based clustering. |
205
+ | `cluster_loci(df, window=1_000_000, *, gene="gene", pval="pval")` | Greedy clustering by distance and gene, with one lead row per locus. This is not LD-based clustering. |
206
+ | `locus_clusters(df, window=1_000_000, *, gene="gene_id")` | Connected components of rows linked by gene or proximity; use as the `cluster` column for genome-wide sets. |
207
+ | `annotate_folds(df, model_version="FOLD_1", regions_path=None)` | Add `fold_subset` (`train`, `valid`, `test`, or `none`) from the checkpoint's training regions. |
181
208
  | `AG2KitError`, `CredentialsError`, `ScoringError` | Exceptions. `ScoringError` provides `error_type` and `record`. |
182
209
  | `Concordance` | Return type of `concordance`. |
183
210
  | `ReferenceBase` | Cached GRCh38 reference-base lookup through Ensembl REST. |
@@ -205,6 +232,8 @@ Requests with the same identity reuse cached predictions. Each cache entry store
205
232
 
206
233
  Cell-conditioned requests use separate, ordered lists of cell and gene names in the cache fingerprint. Entries created with the earlier fingerprint require rescoring.
207
234
 
235
+ Cache files written since v0.2.0 carry Parquet page checksums, which are verified on read. A corrupted or mismatched entry is logged and treated as a miss. Variants on `MT` or `chrMT` are scored as `chrM` since v0.2.0, so their cache keys differ from v0.1.0.
236
+
208
237
  The AlphaGenome API does not expose a model revision or guarantee identical results across calls. Results may change after cache deletion or a server-side model update. For reproducible analyses:
209
238
 
210
239
  1. Archive the supporting `predictions.parquet`, `evaluation.json`, and cache entries.