phensim 1.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. phensim-1.0.0/LICENSE +19 -0
  2. phensim-1.0.0/PKG-INFO +331 -0
  3. phensim-1.0.0/README.md +301 -0
  4. phensim-1.0.0/phensim/__init__.py +109 -0
  5. phensim-1.0.0/phensim/_coalescent.py +649 -0
  6. phensim-1.0.0/phensim/_common.py +76 -0
  7. phensim-1.0.0/phensim/_numba.py +19 -0
  8. phensim-1.0.0/phensim/ancestry.py +596 -0
  9. phensim-1.0.0/phensim/genotypes.py +551 -0
  10. phensim-1.0.0/phensim/hapnest.py +197 -0
  11. phensim-1.0.0/phensim/io.py +198 -0
  12. phensim-1.0.0/phensim/kinship.py +222 -0
  13. phensim-1.0.0/phensim/pedigree.py +444 -0
  14. phensim-1.0.0/phensim/phenotypes.py +629 -0
  15. phensim-1.0.0/phensim/sumstats.py +749 -0
  16. phensim-1.0.0/phensim.egg-info/PKG-INFO +331 -0
  17. phensim-1.0.0/phensim.egg-info/SOURCES.txt +37 -0
  18. phensim-1.0.0/phensim.egg-info/dependency_links.txt +1 -0
  19. phensim-1.0.0/phensim.egg-info/requires.txt +13 -0
  20. phensim-1.0.0/phensim.egg-info/top_level.txt +1 -0
  21. phensim-1.0.0/pyproject.toml +55 -0
  22. phensim-1.0.0/setup.cfg +4 -0
  23. phensim-1.0.0/tests/test_ancestry.py +383 -0
  24. phensim-1.0.0/tests/test_coalescent_invariants.py +289 -0
  25. phensim-1.0.0/tests/test_family_parity.py +235 -0
  26. phensim-1.0.0/tests/test_gaussian_individuals.py +75 -0
  27. phensim-1.0.0/tests/test_hapnest.py +222 -0
  28. phensim-1.0.0/tests/test_kinship.py +92 -0
  29. phensim-1.0.0/tests/test_memory_lean.py +92 -0
  30. phensim-1.0.0/tests/test_normal_tails.py +64 -0
  31. phensim-1.0.0/tests/test_pedigree.py +167 -0
  32. phensim-1.0.0/tests/test_phenotype_edges.py +109 -0
  33. phensim-1.0.0/tests/test_phensim.py +300 -0
  34. phensim-1.0.0/tests/test_plink_interop.py +69 -0
  35. phensim-1.0.0/tests/test_regressions.py +336 -0
  36. phensim-1.0.0/tests/test_review_fixes.py +651 -0
  37. phensim-1.0.0/tests/test_review_optimizations.py +403 -0
  38. phensim-1.0.0/tests/test_structured_ld.py +84 -0
  39. phensim-1.0.0/tests/test_sumstats.py +188 -0
phensim-1.0.0/LICENSE ADDED
@@ -0,0 +1,19 @@
1
+ Copyright (c) 2026 Bjarni J Vilhjalmsson
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining a copy
4
+ of this software and associated documentation files (the "Software"), to deal
5
+ in the Software without restriction, including without limitation the rights
6
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7
+ copies of the Software, and to permit persons to whom the Software is
8
+ furnished to do so, subject to the following conditions:
9
+
10
+ The above copyright notice and this permission notice shall be included in all
11
+ copies or substantial portions of the Software.
12
+
13
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
19
+ SOFTWARE.
phensim-1.0.0/PKG-INFO ADDED
@@ -0,0 +1,331 @@
1
+ Metadata-Version: 2.4
2
+ Name: phensim
3
+ Version: 1.0.0
4
+ Summary: Genotype and phenotype simulators for genetic association studies
5
+ License-Expression: MIT
6
+ Project-URL: Homepage, https://github.com/bvilhjal/phensim
7
+ Project-URL: Repository, https://github.com/bvilhjal/phensim
8
+ Project-URL: Issues, https://github.com/bvilhjal/phensim/issues
9
+ Classifier: Intended Audience :: Science/Research
10
+ Classifier: Programming Language :: Python :: 3
11
+ Classifier: Programming Language :: Python :: 3.9
12
+ Classifier: Programming Language :: Python :: 3.10
13
+ Classifier: Programming Language :: Python :: 3.11
14
+ Classifier: Programming Language :: Python :: 3.12
15
+ Classifier: Programming Language :: Python :: 3.13
16
+ Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
17
+ Requires-Python: >=3.9
18
+ Description-Content-Type: text/markdown
19
+ License-File: LICENSE
20
+ Requires-Dist: numpy>=1.20
21
+ Provides-Extra: fast
22
+ Requires-Dist: numba>=0.59; extra == "fast"
23
+ Provides-Extra: msprime
24
+ Requires-Dist: msprime>=1.3; extra == "msprime"
25
+ Provides-Extra: test
26
+ Requires-Dist: pytest>=7; extra == "test"
27
+ Provides-Extra: lint
28
+ Requires-Dist: ruff>=0.5; extra == "lint"
29
+ Dynamic: license-file
30
+
31
+ # phensim
32
+
33
+ Genotype and phenotype simulators for genetic association studies.
34
+
35
+ phensim 1.0 is a complete revision of the 2019 snippet repo (preserved
36
+ under the `v0.1-legacy` git tag). It provides, in plain NumPy:
37
+
38
+ **Genotype simulators** (`phensim.genotypes`)
39
+
40
+ Table 1. Available genotype structures and their computational character.
41
+
42
+ | Function | Structure | Cost |
43
+ |---|---|---|
44
+ | `simulate_independent` | none; fixed or SFS-shaped frequencies (beta / uniform / rare / common) | instant |
45
+ | `simulate_population_structure` | normal or Balding–Nichols drift in K populations; optional AR(1) LD blocks; `fst=0` control | O(nm) |
46
+ | `simulate_haplotype_blocks` | founder-haplotype copying: genuine haplotypic LD within blocks | one pass |
47
+ | `simulate_ar1_blocks` | latent-Gaussian haplotypes with AR(1) decay `rho`, thresholded at the MAF quantile; right-skewed geometry via `realistic_block_sizes`; `method="scan"` opts into an O(nk) forward recursion instead of the per-block Cholesky | one pass |
48
+ | `simulate_coalescent` | coalescent with recombination: haplotype + recombination LD; target SNP count, contiguous LD blocks | seconds |
49
+ | `simulate_by_mutation_rate` | same, but fixed segment and mutation-rate density lever on a seed-fixed genealogy | seconds |
50
+ | `simulate_hapnest` / `iter_hapnest` | HAPNEST age-dependent copying from phased references; discrete populations; optional Numba, batched or disk-backed output | O(nm), bounded working memory |
51
+
52
+ The coalescent family has two backends: the **msprime** C library
53
+ (the `msprime` extra) and a **built-in Numba
54
+ coalescent-with-recombination** engine (Hudson ancestry with Fenwick-tree
55
+ lineage sampling, infinite-sites mutations, JIT end-to-end; correct
56
+ pure-Python fallback) extracted from the
57
+ [ldpred3](https://github.com/bvilhjal/ldpred3) benchmark suite. All
58
+ simulators return sample-major `int8` dosages in {0, 1, 2}, columns in
59
+ physical order.
60
+ Coalescent dosages count the derived allele without random flips;
61
+ `simulate_independent` randomly orients the counted allele at each site.
62
+
63
+ **Several populations and admixture** (`phensim.ancestry`)
64
+
65
+ Table 2. Multi-population simulators and the ancestry truth they return.
66
+
67
+ | Function | Structure | Truth returned |
68
+ |---|---|---|
69
+ | `drift_frequencies` | per-population frequencies drifted (Balding–Nichols or normal) from one ancestral spectrum; one F_ST per population | `(K, m)` frequencies |
70
+ | `simulate_populations` | discrete populations with exact sizes and their own AR(1) LD (`rho`, block lengths per population) | population labels |
71
+ | `simulate_admixed` | pulse-admixture mosaics: tracts along a genetic map, each following its ancestry's AR(1) model | local ancestry per haplotype and variant |
72
+ | `simulate_split_coalescent` | msprime split of K populations at a target F_ST, optionally with an admixture pulse | labels; local ancestry of admixed samples |
73
+ | `simulate_genome` | independently seeded coalescent chromosomes; cross-chromosome LD exactly zero; bit-parity with bipred's block genome | chromosome labels; one LD block per chromosome |
74
+ | `simulate_meta_sumstats` | per-population GWAS under each ancestry's own LD plus the fixed-effect meta-analysis | per-population `bhat`, inverse-variance meta and weights |
75
+
76
+ **Phenotype simulators** (`phensim.phenotypes`) — all draw the
77
+ infinitesimal component as u ~ N(0, sigma2 K) on the empirical GRM, so
78
+ the data-generating covariance matches what a mixed model will fit.
79
+ `simulate_trait` (and the `simulate_binary_trait`/`simulate_gxe_trait`
80
+ wrappers delegating to it) and `simulate_correlated_traits` draw
81
+ matrix-free when no kinship is supplied: `m + 1` innovations through an
82
+ exact factor `F` of the scaled GRM (`F F' = K`), never an `n x n`
83
+ matrix or eigendecomposition. A supplied `K` keeps the classic
84
+ eigendecomposition draw. `simulate_confounded_trait` also draws matrix-free
85
+ when given an explicit `environment`; its default leading kinship axis
86
+ requires one eigendecomposition shared with the background:
87
+
88
+ - `simulate_trait`: quantitative traits with `mixed` / `infinitesimal`
89
+ / `qtl` architectures, h2 targeting, normal or equal effect sizes;
90
+ - `simulate_binary_trait`: liability-threshold case/control with a
91
+ target prevalence;
92
+ - `simulate_confounded_trait`: an explicit environmental exposure or the
93
+ leading kinship eigenvector, plus a chosen base genetic architecture;
94
+ - `simulate_gxe_trait`: genotype-environment interaction variance;
95
+ - `simulate_correlated_traits`: bivariate traits with a target genetic
96
+ correlation rg.
97
+
98
+ For a matched structure/LD experiment, pass `model="balding-nichols"`,
99
+ `block_sizes=[50] * (m // 50)` and `rho=0.8` to
100
+ `simulate_population_structure`; use `fst=0` for the control. Block lengths
101
+ must sum to `m`. Pass the returned population labels, or another chosen
102
+ exposure, as `environment=` to `simulate_confounded_trait`. Its
103
+ `confounding_strength` sets a component target, not the realized fraction
104
+ of phenotype variance: genetic and environmental components can covary.
105
+
106
+ **Ascertainment and scale conversions** (`phensim.phenotypes`):
107
+ `ascertain_case_control` samples exact case/control counts from a
108
+ simulated trait (the register / balanced-cohort schemes); the trait dict
109
+ must carry an exact binary `case_control` vector and a matching finite
110
+ `liability`, and counts must be nonnegative integers.
111
+ `n_eff_case_control` and `h2_liability` (Lee et al. 2011) convert between
112
+ observed and liability scales.
113
+
114
+ **Summary statistics** (`phensim.sumstats`) — the RSS layer on top of
115
+ block-diagonal population LD:
116
+
117
+ - `simulate_effects`: effect draws (sparse / polygenic / MAF-exponent /
118
+ equal) pinned so `beta' R beta = h2`; the causal set is `n_causal`
119
+ variants or Bernoulli(`p`), with `p` scalar or per variant;
120
+ - `simulate_effects_pair`: two traits with correlated shared effects --
121
+ a shared Bernoulli(`p`) causal set, or exact per-trait/shared counts
122
+ (the MiXeR four-state truth) -- each pinned to its h2;
123
+ `genetic_correlation` gives the realized rg under the LD;
124
+ - `simulate_sumstats`: the oracle `bhat = R beta + N(0, R/n)`, scalar or
125
+ per-variant N;
126
+ - `simulate_sumstats_pair`: two GWAS with an explicit sampling-noise
127
+ correlation and per-trait N (`n_b`);
128
+ - `simulate_meta_sumstats` (`phensim.ancestry`): K populations' GWAS under
129
+ their own LD with independent noise, plus the fixed-effect
130
+ meta-analysis (inverse-variance = sample-size weights on this scale);
131
+ - `gwas_scan`: marginal GWAS (beta / se / z / p) from individual-level
132
+ genotypes and a phenotype;
133
+ - `shake_ld`: finite reference-panel LD noise (Wishart panels), optionally
134
+ shrunk toward I (`shrink`), with opt-in `chunk_size` row-chunked
135
+ accumulation for large `n_ref`;
136
+ - `prepare_blocks`: one-time LD-block validation and factorization into
137
+ a read-only snapshot every block consumer accepts.
138
+ - `simulate_gaussian_individuals`: continuous Gaussian individuals from raw
139
+ or prepared LD blocks, with population covariance R and no sample rescaling.
140
+ `shake_ld(factors=...)` also accepts rectangular or eigenvalue-clipped roots;
141
+ its finite panel samples from the supplied factor's covariance.
142
+
143
+ **Pedigrees** (`phensim.pedigree`): `simulate_pedigree` (multi-
144
+ generation trio columns with remarriage and half-sibs),
145
+ `pedigree_birth_times` (generation-coherent calendar years),
146
+ `kinship_from_pedigree` (dense additive relationship matrix A with
147
+ inbreeding, Henderson tabular recursion), and `mendelian_draw`
148
+ (genetic values with covariance A in O(n) storage, exact inbreeding
149
+ variances). Parent references that are not listed ids warn once per
150
+ call and are treated as unknown founders.
151
+ Birth-time placement additionally requires discrete generations: some valid
152
+ acyclic pedigrees, including uncle–niece matings, cannot satisfy this constraint.
153
+
154
+ Plus `phensim.kinship` — `grm` (Yang-2010 called-only
155
+ standardization), `ibs_kinship`, exact leave-one-chromosome-out
156
+ `loco_kinships` and its lazy `iter_loco_kinships` generator, and
157
+ `windowed_kinships` local/global pairs — and
158
+ `phensim.io.write_plink` (PLINK 1 binary output for external tools).
159
+
160
+ ## Documentation
161
+
162
+ - [docs/guide.md](docs/guide.md): what phensim simulates, how to pick a
163
+ model, and recipes.
164
+ - [docs/technical.md](docs/technical.md): the models, the algorithms and
165
+ the numerical contracts.
166
+ - [docs/hapnest.md](docs/hapnest.md): large reference-based cohorts, mutation
167
+ ages, model boundaries, and speed/memory controls.
168
+ - [report/phensim_report.pdf](report/phensim_report.pdf): the technical
169
+ report, with current API contracts and historical dev1 measurements:
170
+ targets versus realized, built-in coalescent versus msprime, null
171
+ calibration and indicative costs. From the repo root, compile it with
172
+ `tectonic report/phensim_report.tex`. To refresh the measurements first,
173
+ run `python report/make_evidence.py`; this replaces the saved evidence.
174
+
175
+ ## Install
176
+
177
+ ```sh
178
+ pip install -e "./phensim[fast,msprime,test,lint]" # from a checkout
179
+ pip install "phensim @ git+https://github.com/bvilhjal/phensim.git" # or from GitHub
180
+ ```
181
+
182
+ phensim is not on PyPI. Core dependency: NumPy only.
183
+
184
+ ## Simulation truth and input contracts
185
+
186
+ Quantitative `y` is standardized; `liability`, `u`, `q` and `e` retain their
187
+ raw scale, with `liability = u + q + e`. Returned `effects` multiply centred,
188
+ unit-SD causal genotypes to recover `q`. Divide the effects by
189
+ `liability.std()` to express them on the standardized-y scale. Confounded
190
+ traits add `structure`; GxE traits add `interaction`, with matching
191
+ `interaction_effects`. GxE component targets are `h2-interaction_h2`,
192
+ `interaction_h2` and `1-h2`. Finite-sample covariance between components can
193
+ change their realized variance fractions. Trait inputs must be complete.
194
+ Automatic causal selection uses only columns that vary across samples,
195
+ capped at the number available; explicit constant causal columns are rejected.
196
+ A supplied `K` is used without rescaling: for positive-semidefinite `K`,
197
+ mean diagonal one gives the intended average marginal background variance.
198
+
199
+ Bivariate `rg` targets the **total** genetic correlation, including QTLs;
200
+ `g_a` and `g_b` expose those genetic values. Finite-sample correlations
201
+ fluctuate, while `rg=+/-1` gives proportional genetic values. Binary traits
202
+ use a standard-normal liability threshold: the requested prevalence is
203
+ approximate when a sparse or structured liability is not normal.
204
+
205
+ The bivariate backgrounds `u_a` and `u_b` retain covariance K without
206
+ normalizing each draw's sample SD. Correcting that former normalization changes
207
+ seeded bivariate outputs; archived measurements retain their producing revision.
208
+
209
+ Summary-statistic blocks are dense `(R, indices)` pairs covering every index
210
+ in `0..m-1` exactly once, in any block order. R must be finite, symmetric,
211
+ unit-diagonal and positive semidefinite. Singular LD is supported without
212
+ adding diagonal noise; only roundoff-sized negative eigenvalues are clipped
213
+ in its sampling factor, with roundoff judged at the input's precision
214
+ (float32 LD at float32). Validation uses bounded row tiles, as in LDpred3.
215
+ The `maf` effect architecture uses ldpred3's `alpha`: per-allele variance
216
+ proportional to `[2f(1-f)]^alpha`, flat on the standardized scale at `alpha=-1`.
217
+ Convert encoded/low-rank LDpred3 representations with `ldpred3.dense_ld`
218
+ before passing them; phensim itself does not depend on LDpred3.
219
+
220
+ The noise factor defaults to the exact PSD factor of each block; the
221
+ draws above also take `jitter=` (`chol(R + jitter I)`) or caller-made
222
+ `factors=` (e.g. a clipped root of thresholded LD, which need not be
223
+ PSD), while the signal always uses `R`. With these options the
224
+ siblings' benchmark draws -- ldpred3 `_metrics.sumstats` and
225
+ `panel_genome`, bipred `sim_effects`, `sumstats_pair`, `ref_panel` and
226
+ `_sim_mixture` -- are reproduced bit for bit (`tests/test_family_parity.py`).
227
+
228
+ `simulate_sumstats_pair(noise_correlation=rho)` controls the correlation
229
+ of GWAS sampling errors, not the fraction of shared participants. For the
230
+ equal-size conditional RSS model it is overlap fraction times standardized
231
+ residual correlation. The old `overlap` keyword preserves its numerical
232
+ meaning with a warning. `gwas_scan` fits each variant on its called samples;
233
+ its historical `z` field is the OLS t statistic and `p` is a large-sample
234
+ normal approximation. Untestable variants return NaN statistics.
235
+
236
+ When the same LD blocks feed several draws, `prepare_blocks(blocks)`
237
+ validates and factors them once; pass the result anywhere the raw
238
+ `(R, ix)` list is accepted — identical draws, no repeated work:
239
+
240
+ ```python
241
+ ld = phensim.prepare_blocks(blocks)
242
+ bhat = phensim.simulate_sumstats(beta, ld, n, seed=1)
243
+ panel = phensim.shake_ld(ld, n_ref=2000, seed=2, chunk_size=500)
244
+ ```
245
+
246
+ PLINK output counts BIM allele 2 (G); negative values and NaN denote missing
247
+ calls. In PLINK 1.9, name G in `--score` input; use `--recode-allele` with
248
+ `--recode A` to request G dosages explicitly. Fractional dosages cannot be
249
+ written as binary hard calls.
250
+ `write_plink` validates the whole genotype matrix and all metadata before
251
+ opening any file: sample ids must be unique nonempty whitespace-free
252
+ strings, chromosomes are nonnegative integer codes or the X/Y/XY/MT
253
+ labels, and positions are nonnegative integral base-pair values (0 means
254
+ unknown). Haplotype copying returns exactly the requested SNP count,
255
+ including partial blocks.
256
+
257
+ ## Quickstart
258
+
259
+ ```python
260
+ import numpy as np
261
+ import phensim
262
+
263
+ # LD-structured genotypes: 500 diploids, 10k common SNPs, 200-SNP blocks
264
+ G, blocks = phensim.simulate_coalescent(500, 10_000, 200, seed=1)
265
+
266
+ # a 60%-heritable trait, half infinitesimal, half 15 QTLs
267
+ tr = phensim.simulate_trait(G, h2=0.6, n_causal=15, seed=2)
268
+ y, causal = tr["y"], tr["causal"]
269
+
270
+ # case/control with 5% prevalence
271
+ bin = phensim.simulate_binary_trait(G, prevalence=0.05, h2=0.5, seed=3)
272
+ ```
273
+
274
+ ## Validation
275
+
276
+ Run `pytest -q -m "not slow"` for the core and regression tests. These include
277
+ independent BED decoding, direct OLS with missingness, total genetic
278
+ correlation endpoints, variance-component reconstruction, window-complement
279
+ GRMs and the coalescent's linked-segment/recombination-weight invariants.
280
+ Core tests run without msprime or Numba. `pytest -q` additionally compares
281
+ selected site-count, diversity and LD summaries with msprime over multiple
282
+ seeds when msprime is installed; these checks do not establish general
283
+ backend equivalence. Two scripts in `benchmarks/` measure more:
284
+ `coalescent_backend.py` compares speed, peak memory and the SFS across sizes,
285
+ and `ld_decay_validation.py` compares the full LD-decay curve with msprime
286
+ and with the Sved and Hill–Weir expectations.
287
+
288
+ The coalescent repairs of the 1.0 rewrite change seeded recombining draws
289
+ from the legacy 2019 snippets (the `v0.1-legacy` tag). Record package
290
+ version and source revision with benchmark results; legacy results are not
291
+ evidence for the corrected simulator.
292
+
293
+ ## Citation
294
+
295
+ If phensim contributes to your research, cite the technical report
296
+ ([report/phensim_report.pdf](report/phensim_report.pdf)) and the version
297
+ you used:
298
+
299
+ ```bibtex
300
+ @misc{phensim,
301
+ title = {phensim: genotype, phenotype and summary-statistic
302
+ simulation for statistical genetics},
303
+ author = {{The phensim authors}},
304
+ year = {2026},
305
+ howpublished = {Version 1.0.0 technical report and source code},
306
+ url = {https://github.com/bvilhjal/phensim}
307
+ }
308
+ ```
309
+
310
+ ## References
311
+
312
+ Primary sources for the models implemented; algorithmic sources are
313
+ listed in [docs/technical.md](docs/technical.md).
314
+
315
+ - Balding & Nichols (1995), *Genetica* 96:3–12 — F_ST frequency drift.
316
+ - Hudson (1983), *Theor. Popul. Biol.* 23:183–201 — the coalescent with recombination.
317
+ - Kelleher et al. (2016), *PLoS Comput. Biol.* 12:e1004842, and
318
+ Baumdicker et al. (2022), *Genetics* 220:iyab229 — msprime.
319
+ - Wharrie et al. (2023), *Bioinformatics* 39:btad535 — HAPNEST reference-based copying.
320
+ - Yang et al. (2010), *Nat. Genet.* 42:565–569 — the called-standardized GRM.
321
+ - Kang et al. (2010), *Nat. Genet.* 42:348–354 — the EMMAX kinship scaling.
322
+ - Lee et al. (2011), *Am. J. Hum. Genet.* 88:294–305 — liability-scale h² conversion.
323
+ - Zhu & Stephens (2017), *Ann. Appl. Stat.* 11:1561–1592 — the RSS oracle.
324
+ - Frei et al. (2019), *Nat. Commun.* 10:2417 — the MiXeR bivariate overlap truth.
325
+ - Zeng et al. (2018), *Nat. Genet.* 50:746–753 — the MAF-exponent effect architecture (SBayesS `S`).
326
+ - Speed et al. (2012), *Am. J. Hum. Genet.* 91:1011–1021 — LDAK-style variant weighting.
327
+ - Henderson (1976), *Biometrics* 32:69–83 — the tabular relationship matrix.
328
+ - Sved (1971), *Theor. Popul. Biol.* 2:125–141, and Hill & Weir (1988),
329
+ *Theor. Popul. Biol.* 33:54–78 — LD-decay expectations used by the benchmarks.
330
+ - Purcell et al. (2007), *Am. J. Hum. Genet.* 81:559–575, and Chang et al.
331
+ (2015), *GigaScience* 4:7 — PLINK.
@@ -0,0 +1,301 @@
1
+ # phensim
2
+
3
+ Genotype and phenotype simulators for genetic association studies.
4
+
5
+ phensim 1.0 is a complete revision of the 2019 snippet repo (preserved
6
+ under the `v0.1-legacy` git tag). It provides, in plain NumPy:
7
+
8
+ **Genotype simulators** (`phensim.genotypes`)
9
+
10
+ Table 1. Available genotype structures and their computational character.
11
+
12
+ | Function | Structure | Cost |
13
+ |---|---|---|
14
+ | `simulate_independent` | none; fixed or SFS-shaped frequencies (beta / uniform / rare / common) | instant |
15
+ | `simulate_population_structure` | normal or Balding–Nichols drift in K populations; optional AR(1) LD blocks; `fst=0` control | O(nm) |
16
+ | `simulate_haplotype_blocks` | founder-haplotype copying: genuine haplotypic LD within blocks | one pass |
17
+ | `simulate_ar1_blocks` | latent-Gaussian haplotypes with AR(1) decay `rho`, thresholded at the MAF quantile; right-skewed geometry via `realistic_block_sizes`; `method="scan"` opts into an O(nk) forward recursion instead of the per-block Cholesky | one pass |
18
+ | `simulate_coalescent` | coalescent with recombination: haplotype + recombination LD; target SNP count, contiguous LD blocks | seconds |
19
+ | `simulate_by_mutation_rate` | same, but fixed segment and mutation-rate density lever on a seed-fixed genealogy | seconds |
20
+ | `simulate_hapnest` / `iter_hapnest` | HAPNEST age-dependent copying from phased references; discrete populations; optional Numba, batched or disk-backed output | O(nm), bounded working memory |
21
+
22
+ The coalescent family has two backends: the **msprime** C library
23
+ (the `msprime` extra) and a **built-in Numba
24
+ coalescent-with-recombination** engine (Hudson ancestry with Fenwick-tree
25
+ lineage sampling, infinite-sites mutations, JIT end-to-end; correct
26
+ pure-Python fallback) extracted from the
27
+ [ldpred3](https://github.com/bvilhjal/ldpred3) benchmark suite. All
28
+ simulators return sample-major `int8` dosages in {0, 1, 2}, columns in
29
+ physical order.
30
+ Coalescent dosages count the derived allele without random flips;
31
+ `simulate_independent` randomly orients the counted allele at each site.
32
+
33
+ **Several populations and admixture** (`phensim.ancestry`)
34
+
35
+ Table 2. Multi-population simulators and the ancestry truth they return.
36
+
37
+ | Function | Structure | Truth returned |
38
+ |---|---|---|
39
+ | `drift_frequencies` | per-population frequencies drifted (Balding–Nichols or normal) from one ancestral spectrum; one F_ST per population | `(K, m)` frequencies |
40
+ | `simulate_populations` | discrete populations with exact sizes and their own AR(1) LD (`rho`, block lengths per population) | population labels |
41
+ | `simulate_admixed` | pulse-admixture mosaics: tracts along a genetic map, each following its ancestry's AR(1) model | local ancestry per haplotype and variant |
42
+ | `simulate_split_coalescent` | msprime split of K populations at a target F_ST, optionally with an admixture pulse | labels; local ancestry of admixed samples |
43
+ | `simulate_genome` | independently seeded coalescent chromosomes; cross-chromosome LD exactly zero; bit-parity with bipred's block genome | chromosome labels; one LD block per chromosome |
44
+ | `simulate_meta_sumstats` | per-population GWAS under each ancestry's own LD plus the fixed-effect meta-analysis | per-population `bhat`, inverse-variance meta and weights |
45
+
46
+ **Phenotype simulators** (`phensim.phenotypes`) — all draw the
47
+ infinitesimal component as u ~ N(0, sigma2 K) on the empirical GRM, so
48
+ the data-generating covariance matches what a mixed model will fit.
49
+ `simulate_trait` (and the `simulate_binary_trait`/`simulate_gxe_trait`
50
+ wrappers delegating to it) and `simulate_correlated_traits` draw
51
+ matrix-free when no kinship is supplied: `m + 1` innovations through an
52
+ exact factor `F` of the scaled GRM (`F F' = K`), never an `n x n`
53
+ matrix or eigendecomposition. A supplied `K` keeps the classic
54
+ eigendecomposition draw. `simulate_confounded_trait` also draws matrix-free
55
+ when given an explicit `environment`; its default leading kinship axis
56
+ requires one eigendecomposition shared with the background:
57
+
58
+ - `simulate_trait`: quantitative traits with `mixed` / `infinitesimal`
59
+ / `qtl` architectures, h2 targeting, normal or equal effect sizes;
60
+ - `simulate_binary_trait`: liability-threshold case/control with a
61
+ target prevalence;
62
+ - `simulate_confounded_trait`: an explicit environmental exposure or the
63
+ leading kinship eigenvector, plus a chosen base genetic architecture;
64
+ - `simulate_gxe_trait`: genotype-environment interaction variance;
65
+ - `simulate_correlated_traits`: bivariate traits with a target genetic
66
+ correlation rg.
67
+
68
+ For a matched structure/LD experiment, pass `model="balding-nichols"`,
69
+ `block_sizes=[50] * (m // 50)` and `rho=0.8` to
70
+ `simulate_population_structure`; use `fst=0` for the control. Block lengths
71
+ must sum to `m`. Pass the returned population labels, or another chosen
72
+ exposure, as `environment=` to `simulate_confounded_trait`. Its
73
+ `confounding_strength` sets a component target, not the realized fraction
74
+ of phenotype variance: genetic and environmental components can covary.
75
+
76
+ **Ascertainment and scale conversions** (`phensim.phenotypes`):
77
+ `ascertain_case_control` samples exact case/control counts from a
78
+ simulated trait (the register / balanced-cohort schemes); the trait dict
79
+ must carry an exact binary `case_control` vector and a matching finite
80
+ `liability`, and counts must be nonnegative integers.
81
+ `n_eff_case_control` and `h2_liability` (Lee et al. 2011) convert between
82
+ observed and liability scales.
83
+
84
+ **Summary statistics** (`phensim.sumstats`) — the RSS layer on top of
85
+ block-diagonal population LD:
86
+
87
+ - `simulate_effects`: effect draws (sparse / polygenic / MAF-exponent /
88
+ equal) pinned so `beta' R beta = h2`; the causal set is `n_causal`
89
+ variants or Bernoulli(`p`), with `p` scalar or per variant;
90
+ - `simulate_effects_pair`: two traits with correlated shared effects --
91
+ a shared Bernoulli(`p`) causal set, or exact per-trait/shared counts
92
+ (the MiXeR four-state truth) -- each pinned to its h2;
93
+ `genetic_correlation` gives the realized rg under the LD;
94
+ - `simulate_sumstats`: the oracle `bhat = R beta + N(0, R/n)`, scalar or
95
+ per-variant N;
96
+ - `simulate_sumstats_pair`: two GWAS with an explicit sampling-noise
97
+ correlation and per-trait N (`n_b`);
98
+ - `simulate_meta_sumstats` (`phensim.ancestry`): K populations' GWAS under
99
+ their own LD with independent noise, plus the fixed-effect
100
+ meta-analysis (inverse-variance = sample-size weights on this scale);
101
+ - `gwas_scan`: marginal GWAS (beta / se / z / p) from individual-level
102
+ genotypes and a phenotype;
103
+ - `shake_ld`: finite reference-panel LD noise (Wishart panels), optionally
104
+ shrunk toward I (`shrink`), with opt-in `chunk_size` row-chunked
105
+ accumulation for large `n_ref`;
106
+ - `prepare_blocks`: one-time LD-block validation and factorization into
107
+ a read-only snapshot every block consumer accepts.
108
+ - `simulate_gaussian_individuals`: continuous Gaussian individuals from raw
109
+ or prepared LD blocks, with population covariance R and no sample rescaling.
110
+ `shake_ld(factors=...)` also accepts rectangular or eigenvalue-clipped roots;
111
+ its finite panel samples from the supplied factor's covariance.
112
+
113
+ **Pedigrees** (`phensim.pedigree`): `simulate_pedigree` (multi-
114
+ generation trio columns with remarriage and half-sibs),
115
+ `pedigree_birth_times` (generation-coherent calendar years),
116
+ `kinship_from_pedigree` (dense additive relationship matrix A with
117
+ inbreeding, Henderson tabular recursion), and `mendelian_draw`
118
+ (genetic values with covariance A in O(n) storage, exact inbreeding
119
+ variances). Parent references that are not listed ids warn once per
120
+ call and are treated as unknown founders.
121
+ Birth-time placement additionally requires discrete generations: some valid
122
+ acyclic pedigrees, including uncle–niece matings, cannot satisfy this constraint.
123
+
124
+ Plus `phensim.kinship` — `grm` (Yang-2010 called-only
125
+ standardization), `ibs_kinship`, exact leave-one-chromosome-out
126
+ `loco_kinships` and its lazy `iter_loco_kinships` generator, and
127
+ `windowed_kinships` local/global pairs — and
128
+ `phensim.io.write_plink` (PLINK 1 binary output for external tools).
129
+
130
+ ## Documentation
131
+
132
+ - [docs/guide.md](docs/guide.md): what phensim simulates, how to pick a
133
+ model, and recipes.
134
+ - [docs/technical.md](docs/technical.md): the models, the algorithms and
135
+ the numerical contracts.
136
+ - [docs/hapnest.md](docs/hapnest.md): large reference-based cohorts, mutation
137
+ ages, model boundaries, and speed/memory controls.
138
+ - [report/phensim_report.pdf](report/phensim_report.pdf): the technical
139
+ report, with current API contracts and historical dev1 measurements:
140
+ targets versus realized, built-in coalescent versus msprime, null
141
+ calibration and indicative costs. From the repo root, compile it with
142
+ `tectonic report/phensim_report.tex`. To refresh the measurements first,
143
+ run `python report/make_evidence.py`; this replaces the saved evidence.
144
+
145
+ ## Install
146
+
147
+ ```sh
148
+ pip install -e "./phensim[fast,msprime,test,lint]" # from a checkout
149
+ pip install "phensim @ git+https://github.com/bvilhjal/phensim.git" # or from GitHub
150
+ ```
151
+
152
+ phensim is not on PyPI. Core dependency: NumPy only.
153
+
154
+ ## Simulation truth and input contracts
155
+
156
+ Quantitative `y` is standardized; `liability`, `u`, `q` and `e` retain their
157
+ raw scale, with `liability = u + q + e`. Returned `effects` multiply centred,
158
+ unit-SD causal genotypes to recover `q`. Divide the effects by
159
+ `liability.std()` to express them on the standardized-y scale. Confounded
160
+ traits add `structure`; GxE traits add `interaction`, with matching
161
+ `interaction_effects`. GxE component targets are `h2-interaction_h2`,
162
+ `interaction_h2` and `1-h2`. Finite-sample covariance between components can
163
+ change their realized variance fractions. Trait inputs must be complete.
164
+ Automatic causal selection uses only columns that vary across samples,
165
+ capped at the number available; explicit constant causal columns are rejected.
166
+ A supplied `K` is used without rescaling: for positive-semidefinite `K`,
167
+ mean diagonal one gives the intended average marginal background variance.
168
+
169
+ Bivariate `rg` targets the **total** genetic correlation, including QTLs;
170
+ `g_a` and `g_b` expose those genetic values. Finite-sample correlations
171
+ fluctuate, while `rg=+/-1` gives proportional genetic values. Binary traits
172
+ use a standard-normal liability threshold: the requested prevalence is
173
+ approximate when a sparse or structured liability is not normal.
174
+
175
+ The bivariate backgrounds `u_a` and `u_b` retain covariance K without
176
+ normalizing each draw's sample SD. Correcting that former normalization changes
177
+ seeded bivariate outputs; archived measurements retain their producing revision.
178
+
179
+ Summary-statistic blocks are dense `(R, indices)` pairs covering every index
180
+ in `0..m-1` exactly once, in any block order. R must be finite, symmetric,
181
+ unit-diagonal and positive semidefinite. Singular LD is supported without
182
+ adding diagonal noise; only roundoff-sized negative eigenvalues are clipped
183
+ in its sampling factor, with roundoff judged at the input's precision
184
+ (float32 LD at float32). Validation uses bounded row tiles, as in LDpred3.
185
+ The `maf` effect architecture uses ldpred3's `alpha`: per-allele variance
186
+ proportional to `[2f(1-f)]^alpha`, flat on the standardized scale at `alpha=-1`.
187
+ Convert encoded/low-rank LDpred3 representations with `ldpred3.dense_ld`
188
+ before passing them; phensim itself does not depend on LDpred3.
189
+
190
+ The noise factor defaults to the exact PSD factor of each block; the
191
+ draws above also take `jitter=` (`chol(R + jitter I)`) or caller-made
192
+ `factors=` (e.g. a clipped root of thresholded LD, which need not be
193
+ PSD), while the signal always uses `R`. With these options the
194
+ siblings' benchmark draws -- ldpred3 `_metrics.sumstats` and
195
+ `panel_genome`, bipred `sim_effects`, `sumstats_pair`, `ref_panel` and
196
+ `_sim_mixture` -- are reproduced bit for bit (`tests/test_family_parity.py`).
197
+
198
+ `simulate_sumstats_pair(noise_correlation=rho)` controls the correlation
199
+ of GWAS sampling errors, not the fraction of shared participants. For the
200
+ equal-size conditional RSS model it is overlap fraction times standardized
201
+ residual correlation. The old `overlap` keyword preserves its numerical
202
+ meaning with a warning. `gwas_scan` fits each variant on its called samples;
203
+ its historical `z` field is the OLS t statistic and `p` is a large-sample
204
+ normal approximation. Untestable variants return NaN statistics.
205
+
206
+ When the same LD blocks feed several draws, `prepare_blocks(blocks)`
207
+ validates and factors them once; pass the result anywhere the raw
208
+ `(R, ix)` list is accepted — identical draws, no repeated work:
209
+
210
+ ```python
211
+ ld = phensim.prepare_blocks(blocks)
212
+ bhat = phensim.simulate_sumstats(beta, ld, n, seed=1)
213
+ panel = phensim.shake_ld(ld, n_ref=2000, seed=2, chunk_size=500)
214
+ ```
215
+
216
+ PLINK output counts BIM allele 2 (G); negative values and NaN denote missing
217
+ calls. In PLINK 1.9, name G in `--score` input; use `--recode-allele` with
218
+ `--recode A` to request G dosages explicitly. Fractional dosages cannot be
219
+ written as binary hard calls.
220
+ `write_plink` validates the whole genotype matrix and all metadata before
221
+ opening any file: sample ids must be unique nonempty whitespace-free
222
+ strings, chromosomes are nonnegative integer codes or the X/Y/XY/MT
223
+ labels, and positions are nonnegative integral base-pair values (0 means
224
+ unknown). Haplotype copying returns exactly the requested SNP count,
225
+ including partial blocks.
226
+
227
+ ## Quickstart
228
+
229
+ ```python
230
+ import numpy as np
231
+ import phensim
232
+
233
+ # LD-structured genotypes: 500 diploids, 10k common SNPs, 200-SNP blocks
234
+ G, blocks = phensim.simulate_coalescent(500, 10_000, 200, seed=1)
235
+
236
+ # a 60%-heritable trait, half infinitesimal, half 15 QTLs
237
+ tr = phensim.simulate_trait(G, h2=0.6, n_causal=15, seed=2)
238
+ y, causal = tr["y"], tr["causal"]
239
+
240
+ # case/control with 5% prevalence
241
+ bin = phensim.simulate_binary_trait(G, prevalence=0.05, h2=0.5, seed=3)
242
+ ```
243
+
244
+ ## Validation
245
+
246
+ Run `pytest -q -m "not slow"` for the core and regression tests. These include
247
+ independent BED decoding, direct OLS with missingness, total genetic
248
+ correlation endpoints, variance-component reconstruction, window-complement
249
+ GRMs and the coalescent's linked-segment/recombination-weight invariants.
250
+ Core tests run without msprime or Numba. `pytest -q` additionally compares
251
+ selected site-count, diversity and LD summaries with msprime over multiple
252
+ seeds when msprime is installed; these checks do not establish general
253
+ backend equivalence. Two scripts in `benchmarks/` measure more:
254
+ `coalescent_backend.py` compares speed, peak memory and the SFS across sizes,
255
+ and `ld_decay_validation.py` compares the full LD-decay curve with msprime
256
+ and with the Sved and Hill–Weir expectations.
257
+
258
+ The coalescent repairs of the 1.0 rewrite change seeded recombining draws
259
+ from the legacy 2019 snippets (the `v0.1-legacy` tag). Record package
260
+ version and source revision with benchmark results; legacy results are not
261
+ evidence for the corrected simulator.
262
+
263
+ ## Citation
264
+
265
+ If phensim contributes to your research, cite the technical report
266
+ ([report/phensim_report.pdf](report/phensim_report.pdf)) and the version
267
+ you used:
268
+
269
+ ```bibtex
270
+ @misc{phensim,
271
+ title = {phensim: genotype, phenotype and summary-statistic
272
+ simulation for statistical genetics},
273
+ author = {{The phensim authors}},
274
+ year = {2026},
275
+ howpublished = {Version 1.0.0 technical report and source code},
276
+ url = {https://github.com/bvilhjal/phensim}
277
+ }
278
+ ```
279
+
280
+ ## References
281
+
282
+ Primary sources for the models implemented; algorithmic sources are
283
+ listed in [docs/technical.md](docs/technical.md).
284
+
285
+ - Balding & Nichols (1995), *Genetica* 96:3–12 — F_ST frequency drift.
286
+ - Hudson (1983), *Theor. Popul. Biol.* 23:183–201 — the coalescent with recombination.
287
+ - Kelleher et al. (2016), *PLoS Comput. Biol.* 12:e1004842, and
288
+ Baumdicker et al. (2022), *Genetics* 220:iyab229 — msprime.
289
+ - Wharrie et al. (2023), *Bioinformatics* 39:btad535 — HAPNEST reference-based copying.
290
+ - Yang et al. (2010), *Nat. Genet.* 42:565–569 — the called-standardized GRM.
291
+ - Kang et al. (2010), *Nat. Genet.* 42:348–354 — the EMMAX kinship scaling.
292
+ - Lee et al. (2011), *Am. J. Hum. Genet.* 88:294–305 — liability-scale h² conversion.
293
+ - Zhu & Stephens (2017), *Ann. Appl. Stat.* 11:1561–1592 — the RSS oracle.
294
+ - Frei et al. (2019), *Nat. Commun.* 10:2417 — the MiXeR bivariate overlap truth.
295
+ - Zeng et al. (2018), *Nat. Genet.* 50:746–753 — the MAF-exponent effect architecture (SBayesS `S`).
296
+ - Speed et al. (2012), *Am. J. Hum. Genet.* 91:1011–1021 — LDAK-style variant weighting.
297
+ - Henderson (1976), *Biometrics* 32:69–83 — the tabular relationship matrix.
298
+ - Sved (1971), *Theor. Popul. Biol.* 2:125–141, and Hill & Weir (1988),
299
+ *Theor. Popul. Biol.* 33:54–78 — LD-decay expectations used by the benchmarks.
300
+ - Purcell et al. (2007), *Am. J. Hum. Genet.* 81:559–575, and Chang et al.
301
+ (2015), *GigaScience* 4:7 — PLINK.