ag2kit 0.1.0__tar.gz → 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {ag2kit-0.1.0 → ag2kit-0.2.0}/PKG-INFO +40 -10
- {ag2kit-0.1.0 → ag2kit-0.2.0}/README.md +37 -6
- ag2kit-0.2.0/benchmarks/bench.py +252 -0
- ag2kit-0.2.0/benchmarks/coverage_sim.py +104 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/docs/api.md +33 -4
- ag2kit-0.2.0/docs/methods.md +125 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/pyproject.toml +16 -6
- {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/__init__.py +14 -2
- {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/client.py +233 -123
- ag2kit-0.2.0/src/ag2kit/evaluate.py +568 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/gtex.py +15 -6
- {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/io.py +4 -8
- {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/stats.py +41 -9
- {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/variants.py +146 -17
- {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_ag2kit.py +128 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_client.py +186 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_evaluate.py +161 -0
- ag2kit-0.1.0/docs/methods.md +0 -97
- ag2kit-0.1.0/src/ag2kit/evaluate.py +0 -351
- {ag2kit-0.1.0 → ag2kit-0.2.0}/.gitignore +0 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/LICENSE +0 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/examples/gtex_eqtl_eval.py +0 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/provenance.py +0 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/src/ag2kit/py.typed +0 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/conftest.py +0 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/data/gtex_v10_mx1_whole_blood.json +0 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_io.py +0 -0
- {ag2kit-0.1.0 → ag2kit-0.2.0}/tests/test_release.py +0 -0
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: ag2kit
|
|
3
|
-
Version: 0.
|
|
4
|
-
Summary: Reproducible AlphaGenome 2 variant-effect analysis
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Reproducible AlphaGenome 2 variant-effect analysis
|
|
5
5
|
Project-URL: Homepage, https://github.com/zeyuyaoy/alphagenome-research
|
|
6
6
|
Project-URL: Issues, https://github.com/zeyuyaoy/alphagenome-research/issues
|
|
7
7
|
Author: Peter Yao
|
|
@@ -20,9 +20,8 @@ Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
|
|
|
20
20
|
Classifier: Typing :: Typed
|
|
21
21
|
Requires-Python: >=3.11
|
|
22
22
|
Requires-Dist: numpy>=1.26
|
|
23
|
-
Requires-Dist: pandas-stubs~=2.3.3
|
|
24
23
|
Requires-Dist: pandas>=2.1
|
|
25
|
-
Requires-Dist: pyarrow>=
|
|
24
|
+
Requires-Dist: pyarrow>=15
|
|
26
25
|
Requires-Dist: requests>=2.31
|
|
27
26
|
Requires-Dist: scipy>=1.11
|
|
28
27
|
Provides-Extra: model
|
|
@@ -31,7 +30,7 @@ Provides-Extra: vcf
|
|
|
31
30
|
Requires-Dist: pysam>=0.22; extra == 'vcf'
|
|
32
31
|
Description-Content-Type: text/markdown
|
|
33
32
|
|
|
34
|
-
### ag2kit:
|
|
33
|
+
### ag2kit: Reproducible AlphaGenome 2 variant-effect analysis
|
|
35
34
|
|
|
36
35
|
AlphaGenome predicts how non-coding DNA variants affect gene regulation [1]. AlphaGenome 2 also supports predictions conditioned on a reference cell type, tissue, or single-cell dataset [4]. Its SDK handles individual predictions and batch scoring. Evaluating those predictions against observed effects requires additional steps, including input validation, allele matching, error handling, and uncertainty estimation that accounts for linkage disequilibrium.
|
|
37
36
|
|
|
@@ -42,9 +41,9 @@ The package covers four tasks:
|
|
|
42
41
|
| Capability | Description |
|
|
43
42
|
|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
44
43
|
| Variant input | `read_variants` loads VCF/BCF (via `pysam`), delimited text, Parquet, and DataFrames into a common GRCh38 schema. It splits multiallelic records, assigns an `input_status` to every row, and optionally checks REF alleles against an indexed FASTA. |
|
|
45
|
-
| Scoring | `AG2Client.score_variants` caches complete SDK results for each unique request. Scoring one variant against multiple genes uses one API call.
|
|
44
|
+
| Scoring | `AG2Client.score_variants` caches complete SDK results for each unique request. Scoring one variant against multiple genes uses one API call. Memory stays bounded for large batches, runs resume after interruption, and failed requests receive error records. Output rows retain their input order. |
|
|
46
45
|
| Observed effects | `ag2kit.gtex` retrieves significant single-tissue eQTLs from GTEx Portal API v2 (GTEx v8, v10, and v11) or reads downloaded significant-pairs files [2]. |
|
|
47
|
-
| Evaluation | `evaluate` orients observed effects to the predicted ALT allele and records a `pair_status` for every pair. It reports sign AUROC, signed Spearman correlation, and sign accuracy, with gene-cluster bootstrap intervals and
|
|
46
|
+
| Evaluation | `evaluate` orients observed effects to the predicted ALT allele and records a `pair_status` for every pair. It reports sign AUROC, signed Spearman correlation, and sign accuracy, with gene-cluster bootstrap intervals, stratified results, and paired comparisons between prediction sets. |
|
|
48
47
|
|
|
49
48
|
> [!NOTE]
|
|
50
49
|
> Use the official SDK for single predictions, track plots, in-silico mutagenesis, and gene annotation (`alphagenome.data.gene_annotation`). Use separate tools for liftover and summary-statistic quality control. For indel normalisation, use `bcftools norm` [3].
|
|
@@ -155,7 +154,37 @@ print(result.metrics[["metric", "estimate", "ci95_low", "ci95_high", "n_clusters
|
|
|
155
154
|
print(result.pairs[["pos", "beta", "beta_alt", "allele_flipped", "pair_status"]])
|
|
156
155
|
```
|
|
157
156
|
|
|
158
|
-
Confidence intervals are `NaN`
|
|
157
|
+
Confidence intervals are `NaN` with fewer than 20 clusters (`min_clusters`), because percentile intervals from few clusters are too narrow. Here there are only two genes.
|
|
158
|
+
|
|
159
|
+
#### Stratified and paired evaluation
|
|
160
|
+
|
|
161
|
+
`evaluate` is `join` followed by `evaluate_pairs`. Calling them separately lets you add columns to the pair ledger and stratify by them. `compare` estimates the difference between two prediction sets on the same observations, using the same cluster resamples for both:
|
|
162
|
+
|
|
163
|
+
```python
|
|
164
|
+
import numpy as np
|
|
165
|
+
import pandas as pd
|
|
166
|
+
|
|
167
|
+
import ag2kit
|
|
168
|
+
|
|
169
|
+
rng = np.random.default_rng(0)
|
|
170
|
+
genes = np.repeat([f"G{i}" for i in range(40)], 5)
|
|
171
|
+
beta = np.repeat(rng.normal(size=40), 5) + rng.normal(scale=0.5, size=200)
|
|
172
|
+
keys = pd.DataFrame(
|
|
173
|
+
{"chrom": "chr1", "pos": np.arange(1, 201), "ref": "A", "alt": "G", "gene_id": genes}
|
|
174
|
+
)
|
|
175
|
+
obs = keys.assign(beta=beta, effect_allele="G")
|
|
176
|
+
good = keys.assign(raw_effect=beta + rng.normal(size=200), status="completed")
|
|
177
|
+
noisy = keys.assign(raw_effect=beta + rng.normal(scale=3, size=200), status="completed")
|
|
178
|
+
|
|
179
|
+
pairs = ag2kit.join(good, obs)
|
|
180
|
+
pairs["magnitude"] = pd.qcut(pairs["predicted"].abs(), 2, labels=["small", "large"])
|
|
181
|
+
by_size = ag2kit.evaluate_pairs(pairs, group="magnitude", n_boot=200)
|
|
182
|
+
print(by_size.per_group[["metric", "group", "n", "value", "ci95_low", "ci95_high"]])
|
|
183
|
+
|
|
184
|
+
print(ag2kit.compare(pairs, ag2kit.join(noisy, obs), n_boot=200))
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
Comparing two scorers, track filters, or conditions this way accounts for the dependence between their predictions, which the McNemar test in `ag2kit.stats` does not. To check performance on regions held out from training, score with `AG2Client(model_version="FOLD_1")`, add `ag2kit.annotate_folds(pairs)` labels, and evaluate with `group="fold_subset"`. See [Methods](docs/methods.md#held-out-regions) for the caveats.
|
|
159
188
|
|
|
160
189
|
#### Conditioning
|
|
161
190
|
|
|
@@ -164,12 +193,13 @@ Each AlphaGenome 2 request uses one conditioning input. You can choose one of tw
|
|
|
164
193
|
1. **Reference condition:** Set `condition` to an ontology CURIE such as `UBERON:0013756`, or a composite such as `UBERON:0000178_CL:0001054`. Use a scorer such as `RNA_SEQ`.
|
|
165
194
|
2. **Single-cell data:** Set `conditioning` to an `AnnData` object containing raw UMI counts. Use Ensembl IDs for `var_names` and include up to 256 cells. Set `condition_label` and use `SC_GENE_EXPRESSION_DENOISED`.
|
|
166
195
|
|
|
167
|
-
To find valid reference conditions, call `supported_conditions()`. For finer control, build `RequestSpec` objects and use `score` or `score_many`. You can then inspect cached SDK tracks through `AG2Client.raw_scores`.
|
|
196
|
+
To find valid reference conditions, call `client.connect()` and then the SDK method `client.model.supported_conditions()`. For finer control, build `RequestSpec` objects and use `score` or `score_many`. You can then inspect cached SDK tracks through `AG2Client.raw_scores`.
|
|
168
197
|
|
|
169
198
|
#### Interpretation and limitations
|
|
170
199
|
|
|
171
200
|
- `RNA_SEQ` predicts log fold changes, while GTEx reports slopes for normalised expression. Accordingly, `ag2kit` compares the direction and rank of effects.
|
|
172
|
-
- GTEx significant-pairs files contain selected eQTL associations, and many variants linked to the same gene are in linkage disequilibrium. As a result, sign metrics may be higher than those measured across all tested pairs. By default, bootstrap intervals resample whole genes to account for within-gene dependence.
|
|
201
|
+
- GTEx significant-pairs files contain selected eQTL associations, and many variants linked to the same gene are in linkage disequilibrium. As a result, sign metrics may be higher than those measured across all tested pairs. By default, bootstrap intervals resample whole genes to account for within-gene dependence; use locus clusters (`locus_clusters`) when variants are shared across genes.
|
|
202
|
+
- Bootstrap intervals describe sampling uncertainty over the evaluated loci. They do not include model-output variability, such as differences between checkpoints, track sets, or conditioning draws.
|
|
173
203
|
- Sign agreement describes performance at the tested loci. It does not establish causality or resolve individual variant effects. In addition, the p-values from `concordance`, `spearman`, and `mcnemar` assume independent pairs, so `evaluate` leaves them out.
|
|
174
204
|
- All coordinates are GRCh38 and 1-based. The package performs no liftover. The API does not expose a model revision, so reproducible reruns depend on cached results ([Reproducibility](docs/api.md#reproducibility)). Tests cover the implementation, but the package has not reproduced a published AlphaGenome benchmark. See [Implementation validation](docs/methods.md#implementation-validation) and [Scope and limitations](docs/methods.md#scope-and-limitations).
|
|
175
205
|
- Finally, AlphaGenome outputs are subject to the SDK's usage restrictions: non-commercial research only, with no use for training other machine-learning models or making clinical decisions. The AlphaGenome terms also govern cached and shared outputs. When using GTEx data, acknowledge the GTEx Project [2].
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
### ag2kit:
|
|
1
|
+
### ag2kit: Reproducible AlphaGenome 2 variant-effect analysis
|
|
2
2
|
|
|
3
3
|
AlphaGenome predicts how non-coding DNA variants affect gene regulation [1]. AlphaGenome 2 also supports predictions conditioned on a reference cell type, tissue, or single-cell dataset [4]. Its SDK handles individual predictions and batch scoring. Evaluating those predictions against observed effects requires additional steps, including input validation, allele matching, error handling, and uncertainty estimation that accounts for linkage disequilibrium.
|
|
4
4
|
|
|
@@ -9,9 +9,9 @@ The package covers four tasks:
|
|
|
9
9
|
| Capability | Description |
|
|
10
10
|
|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
11
11
|
| Variant input | `read_variants` loads VCF/BCF (via `pysam`), delimited text, Parquet, and DataFrames into a common GRCh38 schema. It splits multiallelic records, assigns an `input_status` to every row, and optionally checks REF alleles against an indexed FASTA. |
|
|
12
|
-
| Scoring | `AG2Client.score_variants` caches complete SDK results for each unique request. Scoring one variant against multiple genes uses one API call.
|
|
12
|
+
| Scoring | `AG2Client.score_variants` caches complete SDK results for each unique request. Scoring one variant against multiple genes uses one API call. Memory stays bounded for large batches, runs resume after interruption, and failed requests receive error records. Output rows retain their input order. |
|
|
13
13
|
| Observed effects | `ag2kit.gtex` retrieves significant single-tissue eQTLs from GTEx Portal API v2 (GTEx v8, v10, and v11) or reads downloaded significant-pairs files [2]. |
|
|
14
|
-
| Evaluation | `evaluate` orients observed effects to the predicted ALT allele and records a `pair_status` for every pair. It reports sign AUROC, signed Spearman correlation, and sign accuracy, with gene-cluster bootstrap intervals and
|
|
14
|
+
| Evaluation | `evaluate` orients observed effects to the predicted ALT allele and records a `pair_status` for every pair. It reports sign AUROC, signed Spearman correlation, and sign accuracy, with gene-cluster bootstrap intervals, stratified results, and paired comparisons between prediction sets. |
|
|
15
15
|
|
|
16
16
|
> [!NOTE]
|
|
17
17
|
> Use the official SDK for single predictions, track plots, in-silico mutagenesis, and gene annotation (`alphagenome.data.gene_annotation`). Use separate tools for liftover and summary-statistic quality control. For indel normalisation, use `bcftools norm` [3].
|
|
@@ -122,7 +122,37 @@ print(result.metrics[["metric", "estimate", "ci95_low", "ci95_high", "n_clusters
|
|
|
122
122
|
print(result.pairs[["pos", "beta", "beta_alt", "allele_flipped", "pair_status"]])
|
|
123
123
|
```
|
|
124
124
|
|
|
125
|
-
Confidence intervals are `NaN`
|
|
125
|
+
Confidence intervals are `NaN` with fewer than 20 clusters (`min_clusters`), because percentile intervals from few clusters are too narrow. Here there are only two genes.
|
|
126
|
+
|
|
127
|
+
#### Stratified and paired evaluation
|
|
128
|
+
|
|
129
|
+
`evaluate` is `join` followed by `evaluate_pairs`. Calling them separately lets you add columns to the pair ledger and stratify by them. `compare` estimates the difference between two prediction sets on the same observations, using the same cluster resamples for both:
|
|
130
|
+
|
|
131
|
+
```python
|
|
132
|
+
import numpy as np
|
|
133
|
+
import pandas as pd
|
|
134
|
+
|
|
135
|
+
import ag2kit
|
|
136
|
+
|
|
137
|
+
rng = np.random.default_rng(0)
|
|
138
|
+
genes = np.repeat([f"G{i}" for i in range(40)], 5)
|
|
139
|
+
beta = np.repeat(rng.normal(size=40), 5) + rng.normal(scale=0.5, size=200)
|
|
140
|
+
keys = pd.DataFrame(
|
|
141
|
+
{"chrom": "chr1", "pos": np.arange(1, 201), "ref": "A", "alt": "G", "gene_id": genes}
|
|
142
|
+
)
|
|
143
|
+
obs = keys.assign(beta=beta, effect_allele="G")
|
|
144
|
+
good = keys.assign(raw_effect=beta + rng.normal(size=200), status="completed")
|
|
145
|
+
noisy = keys.assign(raw_effect=beta + rng.normal(scale=3, size=200), status="completed")
|
|
146
|
+
|
|
147
|
+
pairs = ag2kit.join(good, obs)
|
|
148
|
+
pairs["magnitude"] = pd.qcut(pairs["predicted"].abs(), 2, labels=["small", "large"])
|
|
149
|
+
by_size = ag2kit.evaluate_pairs(pairs, group="magnitude", n_boot=200)
|
|
150
|
+
print(by_size.per_group[["metric", "group", "n", "value", "ci95_low", "ci95_high"]])
|
|
151
|
+
|
|
152
|
+
print(ag2kit.compare(pairs, ag2kit.join(noisy, obs), n_boot=200))
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Comparing two scorers, track filters, or conditions this way accounts for the dependence between their predictions, which the McNemar test in `ag2kit.stats` does not. To check performance on regions held out from training, score with `AG2Client(model_version="FOLD_1")`, add `ag2kit.annotate_folds(pairs)` labels, and evaluate with `group="fold_subset"`. See [Methods](docs/methods.md#held-out-regions) for the caveats.
|
|
126
156
|
|
|
127
157
|
#### Conditioning
|
|
128
158
|
|
|
@@ -131,12 +161,13 @@ Each AlphaGenome 2 request uses one conditioning input. You can choose one of tw
|
|
|
131
161
|
1. **Reference condition:** Set `condition` to an ontology CURIE such as `UBERON:0013756`, or a composite such as `UBERON:0000178_CL:0001054`. Use a scorer such as `RNA_SEQ`.
|
|
132
162
|
2. **Single-cell data:** Set `conditioning` to an `AnnData` object containing raw UMI counts. Use Ensembl IDs for `var_names` and include up to 256 cells. Set `condition_label` and use `SC_GENE_EXPRESSION_DENOISED`.
|
|
133
163
|
|
|
134
|
-
To find valid reference conditions, call `supported_conditions()`. For finer control, build `RequestSpec` objects and use `score` or `score_many`. You can then inspect cached SDK tracks through `AG2Client.raw_scores`.
|
|
164
|
+
To find valid reference conditions, call `client.connect()` and then the SDK method `client.model.supported_conditions()`. For finer control, build `RequestSpec` objects and use `score` or `score_many`. You can then inspect cached SDK tracks through `AG2Client.raw_scores`.
|
|
135
165
|
|
|
136
166
|
#### Interpretation and limitations
|
|
137
167
|
|
|
138
168
|
- `RNA_SEQ` predicts log fold changes, while GTEx reports slopes for normalised expression. Accordingly, `ag2kit` compares the direction and rank of effects.
|
|
139
|
-
- GTEx significant-pairs files contain selected eQTL associations, and many variants linked to the same gene are in linkage disequilibrium. As a result, sign metrics may be higher than those measured across all tested pairs. By default, bootstrap intervals resample whole genes to account for within-gene dependence.
|
|
169
|
+
- GTEx significant-pairs files contain selected eQTL associations, and many variants linked to the same gene are in linkage disequilibrium. As a result, sign metrics may be higher than those measured across all tested pairs. By default, bootstrap intervals resample whole genes to account for within-gene dependence; use locus clusters (`locus_clusters`) when variants are shared across genes.
|
|
170
|
+
- Bootstrap intervals describe sampling uncertainty over the evaluated loci. They do not include model-output variability, such as differences between checkpoints, track sets, or conditioning draws.
|
|
140
171
|
- Sign agreement describes performance at the tested loci. It does not establish causality or resolve individual variant effects. In addition, the p-values from `concordance`, `spearman`, and `mcnemar` assume independent pairs, so `evaluate` leaves them out.
|
|
141
172
|
- All coordinates are GRCh38 and 1-based. The package performs no liftover. The API does not expose a model revision, so reproducible reruns depend on cached results ([Reproducibility](docs/api.md#reproducibility)). Tests cover the implementation, but the package has not reproduced a published AlphaGenome benchmark. See [Implementation validation](docs/methods.md#implementation-validation) and [Scope and limitations](docs/methods.md#scope-and-limitations).
|
|
142
173
|
- Finally, AlphaGenome outputs are subject to the SDK's usage restrictions: non-commercial research only, with no use for training other machine-learning models or making clinical decisions. The AlphaGenome terms also govern cached and shared outputs. When using GTEx data, acknowledge the GTEx Project [2].
|
|
@@ -0,0 +1,252 @@
|
|
|
1
|
+
"""Wall-time and peak-memory benchmark for ag2kit's public API.
|
|
2
|
+
|
|
3
|
+
Uses only public names so it runs unchanged against v0.1 and v0.2. Never calls
|
|
4
|
+
the AlphaGenome API: scoring goes through an in-process fake model.
|
|
5
|
+
|
|
6
|
+
uv run python benchmarks/bench.py # full run
|
|
7
|
+
uv run python benchmarks/bench.py --quick # CI smoke run (<~20 s)
|
|
8
|
+
PYTHONPATH=<checkout>/src uv run python benchmarks/bench.py # other version
|
|
9
|
+
"""
|
|
10
|
+
|
|
11
|
+
from __future__ import annotations
|
|
12
|
+
|
|
13
|
+
import argparse
|
|
14
|
+
import os
|
|
15
|
+
import shutil
|
|
16
|
+
import subprocess
|
|
17
|
+
import sys
|
|
18
|
+
import tempfile
|
|
19
|
+
import threading
|
|
20
|
+
import time
|
|
21
|
+
import tracemalloc
|
|
22
|
+
import zlib
|
|
23
|
+
from pathlib import Path
|
|
24
|
+
|
|
25
|
+
import numpy as np
|
|
26
|
+
import pandas as pd
|
|
27
|
+
|
|
28
|
+
# Keep any user cache overrides from redirecting benchmark I/O.
|
|
29
|
+
for _var in ("AG2KIT_AG2_CACHE", "AG2KIT_RESULTS_ROOT"):
|
|
30
|
+
os.environ.pop(_var, None)
|
|
31
|
+
|
|
32
|
+
import ag2kit # noqa: E402
|
|
33
|
+
|
|
34
|
+
N_GENES, N_TRACKS = 30, 50
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
def scorer_output(x, obs, var, scorer, variant):
|
|
38
|
+
"""SDK-shaped AnnData accepted by variant_scorers.tidy_anndata (copied from tests)."""
|
|
39
|
+
import anndata as ad
|
|
40
|
+
from alphagenome.data import genome
|
|
41
|
+
from alphagenome.models.v2 import variant_scorers
|
|
42
|
+
|
|
43
|
+
obs, var = pd.DataFrame(obs), pd.DataFrame(var)
|
|
44
|
+
obs.index, var.index = obs.index.astype(str), var.index.astype(str)
|
|
45
|
+
adata = ad.AnnData(X=np.asarray(x, dtype=float), obs=obs, var=var)
|
|
46
|
+
adata.uns["interval"] = genome.Interval("chr21", 40897866, 41946442)
|
|
47
|
+
adata.uns["variant"] = genome.Variant.from_str(str(variant))
|
|
48
|
+
adata.uns["variant_scorer"] = variant_scorers.RECOMMENDED_VARIANT_SCORERS[scorer]
|
|
49
|
+
return adata
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
class FakeModel:
|
|
53
|
+
"""SDK stand-in returning N_GENES x N_TRACKS scores seeded by the variant."""
|
|
54
|
+
|
|
55
|
+
def __init__(self):
|
|
56
|
+
self.calls = 0
|
|
57
|
+
self.lock = threading.Lock()
|
|
58
|
+
strand = np.array(["+", "-", "."])[np.arange(N_TRACKS) % 3]
|
|
59
|
+
self.var = {
|
|
60
|
+
"strand": strand,
|
|
61
|
+
"name": [f"track{j}" for j in range(N_TRACKS)],
|
|
62
|
+
"imputation_mode": ["none"] * N_TRACKS,
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
def score_variant(self, **kw):
|
|
66
|
+
with self.lock:
|
|
67
|
+
self.calls += 1
|
|
68
|
+
v = kw["variant"]
|
|
69
|
+
rng = np.random.default_rng(zlib.crc32(str(v).encode()))
|
|
70
|
+
locus = v.position // 100_000
|
|
71
|
+
genes = [f"ENSG{locus * 100 + k:011d}" for k in range(N_GENES)]
|
|
72
|
+
obs = {
|
|
73
|
+
"gene_id": genes,
|
|
74
|
+
"strand": np.array(["+", "-"])[np.arange(N_GENES) % 2],
|
|
75
|
+
"gene_name": [f"G{locus}_{k}" for k in range(N_GENES)],
|
|
76
|
+
}
|
|
77
|
+
x = rng.normal(size=(N_GENES, N_TRACKS))
|
|
78
|
+
scorer = "SC_GENE_EXPRESSION_DENOISED" if "cell_conditioning" in kw else "RNA_SEQ"
|
|
79
|
+
return [scorer_output(x, obs, self.var, scorer, str(v))]
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
def make_variants(n, seed=0):
|
|
83
|
+
rng = np.random.default_rng(seed)
|
|
84
|
+
pos = rng.choice(np.arange(1_000_000, 40_000_000), size=n, replace=False)
|
|
85
|
+
ref = rng.choice(list("ACGT"), size=n)
|
|
86
|
+
alt = (pd.Series(ref).map({"A": "C", "C": "G", "G": "T", "T": "A"})).to_numpy()
|
|
87
|
+
return pd.DataFrame(
|
|
88
|
+
{
|
|
89
|
+
"variant_id": [f"v{i}" for i in range(n)],
|
|
90
|
+
"chrom": "chr1",
|
|
91
|
+
"pos": pos,
|
|
92
|
+
"ref": ref,
|
|
93
|
+
"alt": alt,
|
|
94
|
+
}
|
|
95
|
+
)
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
def make_cells(n_cells=256, n_genes=20_000, seed=0):
|
|
99
|
+
import anndata as ad
|
|
100
|
+
import scipy.sparse as sp
|
|
101
|
+
|
|
102
|
+
rng = np.random.default_rng(seed)
|
|
103
|
+
x = sp.random(n_cells, n_genes, density=0.1, format="csr", dtype=np.float32, random_state=rng)
|
|
104
|
+
x.data = rng.poisson(3.0, size=x.data.size).astype(np.float32) + 1
|
|
105
|
+
adata = ad.AnnData(X=x)
|
|
106
|
+
adata.var_names = [f"ENSG{i:011d}" for i in range(n_genes)]
|
|
107
|
+
adata.obs_names = [f"cell{i}" for i in range(n_cells)]
|
|
108
|
+
return adata
|
|
109
|
+
|
|
110
|
+
|
|
111
|
+
def make_eval(n_pairs, n_genes, seed=0):
|
|
112
|
+
rng = np.random.default_rng(seed)
|
|
113
|
+
gene_idx = rng.integers(0, n_genes, n_pairs)
|
|
114
|
+
g_eff = rng.normal(size=n_genes)
|
|
115
|
+
beta = g_eff[gene_idx] + rng.normal(size=n_pairs)
|
|
116
|
+
pred = 0.5 * g_eff[gene_idx] + rng.normal(size=n_pairs)
|
|
117
|
+
ref = rng.choice(list("ACGT"), size=n_pairs)
|
|
118
|
+
alt = pd.Series(ref).map({"A": "C", "C": "G", "G": "T", "T": "A"}).to_numpy()
|
|
119
|
+
keys = pd.DataFrame(
|
|
120
|
+
{
|
|
121
|
+
"chrom": "chr1",
|
|
122
|
+
"pos": np.arange(n_pairs) * 50 + 1_000_000,
|
|
123
|
+
"ref": ref,
|
|
124
|
+
"alt": alt,
|
|
125
|
+
"gene_id": [f"ENSG{g:011d}" for g in gene_idx],
|
|
126
|
+
}
|
|
127
|
+
)
|
|
128
|
+
predictions = keys.assign(raw_effect=pred, status="completed")
|
|
129
|
+
tissues = np.array([f"tissue{t}" for t in range(5)])
|
|
130
|
+
observed = keys.assign(
|
|
131
|
+
beta=beta, effect_allele=alt, tissue=tissues[rng.integers(0, 5, n_pairs)]
|
|
132
|
+
)
|
|
133
|
+
return predictions, observed
|
|
134
|
+
|
|
135
|
+
|
|
136
|
+
def make_loci(n, seed=0):
|
|
137
|
+
rng = np.random.default_rng(seed)
|
|
138
|
+
chrom = rng.integers(1, 23, n)
|
|
139
|
+
return pd.DataFrame(
|
|
140
|
+
{
|
|
141
|
+
"variant_id": [f"rs{i}" for i in range(n)],
|
|
142
|
+
"chrom": [f"chr{c}" for c in chrom],
|
|
143
|
+
"pos": rng.integers(1, 150_000_000, n),
|
|
144
|
+
"gene": [f"ENSG{g:011d}" for g in rng.integers(0, n // 10 + 1, n)],
|
|
145
|
+
"pval": rng.uniform(size=n),
|
|
146
|
+
}
|
|
147
|
+
)
|
|
148
|
+
|
|
149
|
+
|
|
150
|
+
def measure(fn, setup=lambda: None, repeats=1):
|
|
151
|
+
"""Best-of-`repeats` untraced wall time plus one traced run for peak memory.
|
|
152
|
+
|
|
153
|
+
repeats=0 reports the traced run's own time (slow cases; tracemalloc adds overhead).
|
|
154
|
+
"""
|
|
155
|
+
times = []
|
|
156
|
+
for _ in range(repeats):
|
|
157
|
+
arg = setup()
|
|
158
|
+
t0 = time.perf_counter()
|
|
159
|
+
fn(arg)
|
|
160
|
+
times.append(time.perf_counter() - t0)
|
|
161
|
+
arg = setup()
|
|
162
|
+
tracemalloc.start()
|
|
163
|
+
t0 = time.perf_counter()
|
|
164
|
+
fn(arg)
|
|
165
|
+
times.append(time.perf_counter() - t0) if not repeats else None
|
|
166
|
+
peak = tracemalloc.get_traced_memory()[1]
|
|
167
|
+
tracemalloc.stop()
|
|
168
|
+
return min(times), peak / 2**20
|
|
169
|
+
|
|
170
|
+
|
|
171
|
+
def main(argv=None):
|
|
172
|
+
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
|
173
|
+
ap.add_argument("--n", type=float, default=1.0, help="scale factor for problem sizes")
|
|
174
|
+
ap.add_argument("--quick", action="store_true", help="small sizes for a CI smoke run")
|
|
175
|
+
ap.add_argument("--only", default="", help="comma-separated case prefixes (a,b,c,d)")
|
|
176
|
+
args = ap.parse_args(argv)
|
|
177
|
+
s = args.n * (0.01 if args.quick else 1.0)
|
|
178
|
+
sz = lambda n: max(1, int(round(n * s))) # noqa: E731
|
|
179
|
+
only = set(filter(None, args.only.split(",")))
|
|
180
|
+
want = lambda c: not only or c in only # noqa: E731
|
|
181
|
+
|
|
182
|
+
root = Path(tempfile.mkdtemp(prefix="ag2kit-bench-"))
|
|
183
|
+
os.environ["AG2KIT_CACHE"] = str(root / "ag2kit-cache")
|
|
184
|
+
fresh = lambda: Path(tempfile.mkdtemp(dir=root)) # noqa: E731
|
|
185
|
+
src = Path(ag2kit.__file__).resolve().parent
|
|
186
|
+
sha = (
|
|
187
|
+
subprocess.run(
|
|
188
|
+
["git", "-C", str(src), "rev-parse", "--short", "HEAD"], capture_output=True, text=True
|
|
189
|
+
).stdout.strip()
|
|
190
|
+
or "unknown"
|
|
191
|
+
)
|
|
192
|
+
print(f"ag2kit {ag2kit.__version__} @ {sha} from {ag2kit.__file__}")
|
|
193
|
+
print(f"python {sys.version.split()[0]}, scale={s:g}\n")
|
|
194
|
+
rows = []
|
|
195
|
+
|
|
196
|
+
def client(cache):
|
|
197
|
+
return ag2kit.AG2Client(dry_run=False, model=FakeModel(), cache_dir=cache, backoff=0)
|
|
198
|
+
|
|
199
|
+
def score(cache, variants, **kw):
|
|
200
|
+
out = client(cache).score_variants(variants, **kw)
|
|
201
|
+
assert (out["status"] == "completed").all(), out["status"].value_counts()
|
|
202
|
+
return out
|
|
203
|
+
|
|
204
|
+
if want("a"):
|
|
205
|
+
kw = dict(condition="UBERON:0013756", scorer="RNA_SEQ")
|
|
206
|
+
for n in (sz(500), sz(2000)):
|
|
207
|
+
v = make_variants(n)
|
|
208
|
+
reps = 3 if n <= 10 or n == 500 else 1
|
|
209
|
+
t, m = measure(lambda c, v=v: score(c, v, **kw), fresh, repeats=reps)
|
|
210
|
+
rows.append(("a score_variants RNA_SEQ cold", n, t, m))
|
|
211
|
+
warm = fresh()
|
|
212
|
+
score(warm, v, **kw)
|
|
213
|
+
t, m = measure(lambda c, v=v: score(c, v, **kw), lambda w=warm: w, repeats=reps)
|
|
214
|
+
rows.append(("a score_variants RNA_SEQ warm", n, t, m))
|
|
215
|
+
|
|
216
|
+
if want("b"):
|
|
217
|
+
# ponytail: one traced run each (cold, then warm on the same cache); this case is slow.
|
|
218
|
+
n = 1 if args.quick else sz(200)
|
|
219
|
+
v, cells = make_variants(n, seed=1), make_cells()
|
|
220
|
+
kw = dict(
|
|
221
|
+
scorer="SC_GENE_EXPRESSION_DENOISED", conditioning=cells, condition_label="control"
|
|
222
|
+
)
|
|
223
|
+
cache = fresh()
|
|
224
|
+
t, m = measure(lambda c: score(c, v, **kw), lambda: cache, repeats=0)
|
|
225
|
+
rows.append(("b score_variants cells 256x20000 cold (traced time)", n, t, m))
|
|
226
|
+
t, m = measure(lambda c: score(c, v, **kw), lambda: cache, repeats=0)
|
|
227
|
+
rows.append(("b score_variants cells 256x20000 warm (traced time)", n, t, m))
|
|
228
|
+
|
|
229
|
+
if want("c"):
|
|
230
|
+
n = sz(20_000)
|
|
231
|
+
pred, obs = make_eval(n, max(5, n // 10))
|
|
232
|
+
nb = 1000 if not args.quick else 50
|
|
233
|
+
for group in (None, "tissue"):
|
|
234
|
+
fn = lambda _, g=group: ag2kit.evaluate(pred, obs, n_boot=nb, group=g) # noqa: E731
|
|
235
|
+
t, m = measure(fn, repeats=1)
|
|
236
|
+
rows.append((f"c evaluate n_boot={nb} group={group}", n, t, m))
|
|
237
|
+
|
|
238
|
+
if want("d"):
|
|
239
|
+
n = sz(50_000)
|
|
240
|
+
loci = make_loci(n)
|
|
241
|
+
t, m = measure(lambda _: ag2kit.cluster_loci(loci), repeats=1 if n > 10_000 else 3)
|
|
242
|
+
rows.append(("d cluster_loci", n, t, m))
|
|
243
|
+
|
|
244
|
+
print("| case | N | seconds | peak MiB |")
|
|
245
|
+
print("|---|---:|---:|---:|")
|
|
246
|
+
for case, n, t, m in rows:
|
|
247
|
+
print(f"| {case} | {n} | {t:.3f} | {m:.1f} |")
|
|
248
|
+
shutil.rmtree(root, ignore_errors=True)
|
|
249
|
+
|
|
250
|
+
|
|
251
|
+
if __name__ == "__main__":
|
|
252
|
+
main()
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
"""Empirical 95% CI coverage of ag2kit.evaluate's sign-AUROC bootstrap.
|
|
2
|
+
|
|
3
|
+
Compares the cluster bootstrap (cluster="gene_id") with the pair bootstrap
|
|
4
|
+
(cluster=None) on data where pairs within a gene share a gene-level random
|
|
5
|
+
effect in both predicted and observed effects.
|
|
6
|
+
|
|
7
|
+
uv run python benchmarks/coverage_sim.py # 200 replicates, n_boot=500
|
|
8
|
+
uv run python benchmarks/coverage_sim.py --quick # smoke run
|
|
9
|
+
"""
|
|
10
|
+
|
|
11
|
+
from __future__ import annotations
|
|
12
|
+
|
|
13
|
+
import argparse
|
|
14
|
+
import time
|
|
15
|
+
|
|
16
|
+
import numpy as np
|
|
17
|
+
import pandas as pd
|
|
18
|
+
from scipy.stats import mannwhitneyu
|
|
19
|
+
|
|
20
|
+
import ag2kit
|
|
21
|
+
|
|
22
|
+
# Generative model: gene effect u_g drives beta; prediction sees u_g plus its
|
|
23
|
+
# own gene-level error v_g, so both sides are correlated within a gene.
|
|
24
|
+
SD_BETA_NOISE, PRED_SLOPE, SD_PRED_GENE, SD_PRED_NOISE = 0.5, 0.6, 0.8, 0.5
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
def draw(n_genes, per_gene, rng):
|
|
28
|
+
g = np.repeat(np.arange(n_genes), per_gene)
|
|
29
|
+
u, v = rng.normal(size=n_genes), rng.normal(scale=SD_PRED_GENE, size=n_genes)
|
|
30
|
+
beta = u[g] + rng.normal(scale=SD_BETA_NOISE, size=g.size)
|
|
31
|
+
pred = PRED_SLOPE * u[g] + v[g] + rng.normal(scale=SD_PRED_NOISE, size=g.size)
|
|
32
|
+
return g, pred, beta
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
def auroc(pred, beta):
|
|
36
|
+
pos, neg = pred[beta > 0], pred[beta < 0]
|
|
37
|
+
return mannwhitneyu(pos, neg).statistic / (pos.size * neg.size)
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def tables(g, pred, beta):
|
|
41
|
+
n = g.size
|
|
42
|
+
keys = pd.DataFrame(
|
|
43
|
+
{
|
|
44
|
+
"chrom": "chr1",
|
|
45
|
+
"pos": np.arange(n) * 100 + 1_000_000,
|
|
46
|
+
"ref": "A",
|
|
47
|
+
"alt": "G",
|
|
48
|
+
"gene_id": [f"ENSG{x:011d}" for x in g],
|
|
49
|
+
}
|
|
50
|
+
)
|
|
51
|
+
return keys.assign(raw_effect=pred, status="completed"), keys.assign(
|
|
52
|
+
beta=beta, effect_allele="G"
|
|
53
|
+
)
|
|
54
|
+
|
|
55
|
+
|
|
56
|
+
def main(argv=None):
|
|
57
|
+
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
|
58
|
+
ap.add_argument("--reps", type=int, default=200)
|
|
59
|
+
ap.add_argument("--n-boot", type=int, default=500)
|
|
60
|
+
ap.add_argument("--genes", type=int, default=60)
|
|
61
|
+
ap.add_argument("--per-gene", type=int, default=6)
|
|
62
|
+
ap.add_argument("--quick", action="store_true", help="20 replicates x n_boot=200")
|
|
63
|
+
args = ap.parse_args(argv)
|
|
64
|
+
if args.quick:
|
|
65
|
+
args.reps, args.n_boot = 20, 200
|
|
66
|
+
print(f"ag2kit {ag2kit.__version__} from {ag2kit.__file__}")
|
|
67
|
+
|
|
68
|
+
# Population AUROC from one very large draw of the same model.
|
|
69
|
+
truth = auroc(*draw(500_000, args.per_gene, np.random.default_rng(12345))[1:])
|
|
70
|
+
print(
|
|
71
|
+
f"genes={args.genes} per_gene={args.per_gene} reps={args.reps} "
|
|
72
|
+
f"n_boot={args.n_boot}; true sign AUROC (3M-pair MC) = {truth:.4f}\n"
|
|
73
|
+
)
|
|
74
|
+
|
|
75
|
+
hits = {"gene_id": [], None: []}
|
|
76
|
+
widths = {"gene_id": [], None: []}
|
|
77
|
+
estimates = []
|
|
78
|
+
t0 = time.perf_counter()
|
|
79
|
+
for r in range(args.reps):
|
|
80
|
+
g, pred, beta = draw(args.genes, args.per_gene, np.random.default_rng(r))
|
|
81
|
+
predictions, observed = tables(g, pred, beta)
|
|
82
|
+
for cluster in hits:
|
|
83
|
+
m = ag2kit.evaluate(predictions, observed, cluster=cluster, n_boot=args.n_boot, seed=r)
|
|
84
|
+
row = m.metrics.set_index("metric").loc["sign_auroc"]
|
|
85
|
+
hits[cluster].append(row["ci95_low"] <= truth <= row["ci95_high"])
|
|
86
|
+
widths[cluster].append(row["ci95_high"] - row["ci95_low"])
|
|
87
|
+
estimates.append(row["estimate"])
|
|
88
|
+
secs = time.perf_counter() - t0
|
|
89
|
+
|
|
90
|
+
print("| bootstrap | coverage | binomial SE | mean CI width |")
|
|
91
|
+
print("|---|---:|---:|---:|")
|
|
92
|
+
for cluster, h in hits.items():
|
|
93
|
+
p = float(np.mean(h))
|
|
94
|
+
se = np.sqrt(p * (1 - p) / len(h))
|
|
95
|
+
name = "cluster (gene_id)" if cluster else "pair (cluster=None)"
|
|
96
|
+
print(f"| {name} | {p:.3f} | {se:.3f} | {np.mean(widths[cluster]):.4f} |")
|
|
97
|
+
print(
|
|
98
|
+
f"\nmean estimate {np.mean(estimates):.4f} (truth {truth:.4f}), "
|
|
99
|
+
f"empirical SD of estimate {np.std(estimates, ddof=1):.4f}; {secs:.0f} s"
|
|
100
|
+
)
|
|
101
|
+
|
|
102
|
+
|
|
103
|
+
if __name__ == "__main__":
|
|
104
|
+
main()
|
|
@@ -37,7 +37,7 @@ client = ag2kit.AG2Client(
|
|
|
37
37
|
dry_run=True,
|
|
38
38
|
cache_dir=...,
|
|
39
39
|
model_version="ALL_FOLDS",
|
|
40
|
-
max_retries=
|
|
40
|
+
max_retries=4,
|
|
41
41
|
backoff=30.0,
|
|
42
42
|
cache_namespace="",
|
|
43
43
|
model=None,
|
|
@@ -46,6 +46,8 @@ client = ag2kit.AG2Client(
|
|
|
46
46
|
|
|
47
47
|
`AG2Client` manages scoring requests and their cache. By default, `dry_run=True` plans requests without calling the API.
|
|
48
48
|
|
|
49
|
+
Batch methods fetch each distinct request once and reduce it to gene records as soon as it completes, so memory use depends on `max_workers`, not on the number of variants. An interrupt (Ctrl-C) cancels requests that have not started; completed requests stay cached, so rerunning resumes the batch. Retries apply to `quota`, `transient`, and `network` errors: attempt *k* waits a random delay between half and all of `backoff * 2**(k-1)` seconds, capped at 600 seconds. Progress is logged at `INFO` level on the `ag2kit` logger.
|
|
50
|
+
|
|
49
51
|
| Method | Description |
|
|
50
52
|
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
51
53
|
| `score_variants(variants, *, condition="", scorer="RNA_SEQ", gene_column=None, conditioning=None, condition_label="", sequence_length=2**20, max_workers=4, tracks=None)` | Score a table of variants. Returns one row per input row and gene, or one row per input row when `gene_column` identifies a gene. Input columns use the `input_` prefix. |
|
|
@@ -121,10 +123,21 @@ Supported datasets are `gtex_v8` (GENCODE v26), `gtex_v10` (v39), and `gtex_v11`
|
|
|
121
123
|
| `tissues` | List tissue IDs, UBERON IDs, and eGene counts. |
|
|
122
124
|
| `gencode_id` | Resolve a gene to its release-specific GENCODE identifier. |
|
|
123
125
|
|
|
126
|
+
Portal failures raise `ag2kit.gtex.SourceError`, a subclass of both `AG2KitError` and `RuntimeError`.
|
|
127
|
+
|
|
124
128
|
`eqtls` resolves gene symbols and unversioned Ensembl IDs to the selected GENCODE release. Versioned Ensembl IDs (containing `.`) must match that release. `tissues` and `gencode_id` also accept `refresh`. GTEx responses are cached at `$AG2KIT_CACHE/gtex`.
|
|
125
129
|
|
|
126
130
|
#### Evaluation
|
|
127
131
|
|
|
132
|
+
Evaluation has two steps, which `evaluate` combines:
|
|
133
|
+
|
|
134
|
+
```python
|
|
135
|
+
pairs = ag2kit.join(predictions, observed, *, match=None, effect="raw_effect", forward_strand=True)
|
|
136
|
+
result = ag2kit.evaluate_pairs(pairs, *, group=None, cluster="gene_id", n_boot=1000, seed=0, min_group=2, min_clusters=20)
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
`join` returns the pair ledger. Add columns to it before calling `evaluate_pairs` to stratify by anything you can compute, such as effect-size bins, distance to the TSS, or `annotate_folds` labels.
|
|
140
|
+
|
|
128
141
|
```python
|
|
129
142
|
ag2kit.evaluate(
|
|
130
143
|
predictions,
|
|
@@ -138,7 +151,7 @@ ag2kit.evaluate(
|
|
|
138
151
|
n_boot=1000,
|
|
139
152
|
seed=0,
|
|
140
153
|
min_group=2,
|
|
141
|
-
min_clusters=
|
|
154
|
+
min_clusters=20,
|
|
142
155
|
) -> Evaluation
|
|
143
156
|
```
|
|
144
157
|
|
|
@@ -157,15 +170,27 @@ Use `match` to require agreement on additional columns. For example, `match={"on
|
|
|
157
170
|
|---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
158
171
|
| `pairs` | One row per observed record, including `predicted`, `beta_alt`, `allele_flipped`, `pair_status`, and matched prediction fields prefixed with `pred_`. |
|
|
159
172
|
| `metrics` | `sign_auroc`, `spearman_rho`, and `sign_accuracy`, with estimates and uncertainty fields. |
|
|
160
|
-
| `per_group` | Group-level results: `group`, `n`, `value`, `metric`.
|
|
173
|
+
| `per_group` | Group-level results: `group`, `n`, `value`, `metric`, `ci95_low`, `ci95_high`, `n_clusters`, `n_boot_finite`. |
|
|
161
174
|
| `settings` | Evaluation configuration. |
|
|
162
175
|
| `summary()` | Print metrics and counts by `pair_status`. |
|
|
163
176
|
| `save(directory)` | Save `pairs.parquet`, `metrics.tsv`, `per_group.tsv`, and `evaluation.json`. |
|
|
164
177
|
|
|
165
178
|
The `metrics` table includes `estimate`, `ci95_low`, `ci95_high`, `n_pairs`, `n_pairs_used`, `n_clusters`, `n_boot_finite`, and `weighting`. `evaluation.json` includes the settings, software versions, and write timestamp.
|
|
166
179
|
|
|
180
|
+
All metrics and groups use the same bootstrap resamples, so intervals do not depend on metric order. Intervals need at least `min_clusters` clusters overall, and per-group intervals need that many clusters within the group. Only groups that contribute in the original sample can contribute to a weighted bootstrap estimate. In each resample, groups below `min_group` or with undefined metrics are omitted and the remaining group weights are renormalized; the replicate is discarded if no eligible groups remain.
|
|
181
|
+
|
|
182
|
+
The ledger also carries the prediction's `pred_model_version`, `pred_track_filter`, and `pred_n_reduced_values` when present, so stratified and fold analyses can check what was scored.
|
|
183
|
+
|
|
167
184
|
`pair_status` can be `ok`, `no_prediction`, `prediction_failed`, `ambiguous_prediction`, `ref_mismatch`, `missing_statistics`, `ref_equals_alt`, `not_snv`, `palindromic`, or `effect_allele_not_in_ref_alt`. The `not_snv` and `palindromic` statuses apply only when `forward_strand=False`.
|
|
168
185
|
|
|
186
|
+
##### Comparing prediction sets
|
|
187
|
+
|
|
188
|
+
```python
|
|
189
|
+
ag2kit.compare(pairs_a, pairs_b, *, group=None, cluster="gene_id", n_boot=1000, seed=0, min_group=2, min_clusters=20) -> pandas.DataFrame
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
`compare` estimates `a − b` for each metric, using pairs that are usable in both ledgers and a paired cluster bootstrap. Both ledgers must come from `join` with the same observed table; misaligned ledgers raise `ValueError`. Each ledger is evaluated against its own ALT orientation. The result has `estimate_a`, `estimate_b`, `difference`, `ci95_low`, `ci95_high`, `n_pairs`, `n_usable_a_only`, `n_usable_b_only`, `n_clusters`, `n_boot_finite`, and `weighting`. Large `n_usable_*_only` counts mean the two prediction sets fail on different variants, so also report each set's own `evaluate_pairs` result. Use it to compare scorers, track filters, conditions, or model versions.
|
|
193
|
+
|
|
169
194
|
#### Statistics and utilities
|
|
170
195
|
|
|
171
196
|
| Function or type | Description |
|
|
@@ -177,7 +202,9 @@ The `metrics` table includes `estimate`, `ci95_low`, `ci95_high`, `n_pairs`, `n_
|
|
|
177
202
|
| `ag2kit.stats.mcnemar` | Exact McNemar test for discordant pairs. |
|
|
178
203
|
| `ag2kit.stats.monte_carlo_summary` | Summary across conditioning draws. |
|
|
179
204
|
| `harmonize_effect(estimate, effect_allele, ref, alt, forward_strand=False)` | Orient an observed effect to ALT. The default is `False`; `evaluate` defaults to `True`. |
|
|
180
|
-
| `cluster_loci(df, window=1_000_000)`
|
|
205
|
+
| `cluster_loci(df, window=1_000_000, *, gene="gene", pval="pval")` | Greedy clustering by distance and gene, with one lead row per locus. This is not LD-based clustering. |
|
|
206
|
+
| `locus_clusters(df, window=1_000_000, *, gene="gene_id")` | Connected components of rows linked by gene or proximity; use as the `cluster` column for genome-wide sets. |
|
|
207
|
+
| `annotate_folds(df, model_version="FOLD_1", regions_path=None)` | Add `fold_subset` (`train`, `valid`, `test`, or `none`) from the checkpoint's training regions. |
|
|
181
208
|
| `AG2KitError`, `CredentialsError`, `ScoringError` | Exceptions. `ScoringError` provides `error_type` and `record`. |
|
|
182
209
|
| `Concordance` | Return type of `concordance`. |
|
|
183
210
|
| `ReferenceBase` | Cached GRCh38 reference-base lookup through Ensembl REST. |
|
|
@@ -205,6 +232,8 @@ Requests with the same identity reuse cached predictions. Each cache entry store
|
|
|
205
232
|
|
|
206
233
|
Cell-conditioned requests use separate, ordered lists of cell and gene names in the cache fingerprint. Entries created with the earlier fingerprint require rescoring.
|
|
207
234
|
|
|
235
|
+
Cache files written since v0.2.0 carry Parquet page checksums, which are verified on read. A corrupted or mismatched entry is logged and treated as a miss. Variants on `MT` or `chrMT` are scored as `chrM` since v0.2.0, so their cache keys differ from v0.1.0.
|
|
236
|
+
|
|
208
237
|
The AlphaGenome API does not expose a model revision or guarantee identical results across calls. Results may change after cache deletion or a server-side model update. For reproducible analyses:
|
|
209
238
|
|
|
210
239
|
1. Archive the supporting `predictions.parquet`, `evaluation.json`, and cache entries.
|