structboost 0.3.0__tar.gz → 0.5.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. {structboost-0.3.0 → structboost-0.5.0}/CHANGELOG.md +180 -0
  2. {structboost-0.3.0 → structboost-0.5.0}/CITATION.cff +2 -2
  3. {structboost-0.3.0 → structboost-0.5.0}/PKG-INFO +59 -11
  4. {structboost-0.3.0 → structboost-0.5.0}/README.md +58 -10
  5. {structboost-0.3.0 → structboost-0.5.0}/pyproject.toml +1 -1
  6. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_model.py +167 -49
  7. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_persistence.py +10 -2
  8. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_types.py +115 -28
  9. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_utils.py +67 -30
  10. {structboost-0.3.0 → structboost-0.5.0}/tests/test_bae.py +219 -59
  11. {structboost-0.3.0 → structboost-0.5.0}/tests/test_bae_init.py +10 -2
  12. {structboost-0.3.0 → structboost-0.5.0}/tests/test_bae_persistence.py +24 -0
  13. {structboost-0.3.0 → structboost-0.5.0}/tests/test_bae_transfer.py +35 -2
  14. {structboost-0.3.0 → structboost-0.5.0}/tests/test_stability.py +156 -6
  15. {structboost-0.3.0 → structboost-0.5.0}/.gitignore +0 -0
  16. {structboost-0.3.0 → structboost-0.5.0}/CODE_OF_CONDUCT.md +0 -0
  17. {structboost-0.3.0 → structboost-0.5.0}/CONTRIBUTING.md +0 -0
  18. {structboost-0.3.0 → structboost-0.5.0}/LICENSE +0 -0
  19. {structboost-0.3.0 → structboost-0.5.0}/conftest.py +0 -0
  20. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/__init__.py +0 -0
  21. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_annotation.py +0 -0
  22. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_boosting.py +0 -0
  23. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_decoder.py +0 -0
  24. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_encoder.py +0 -0
  25. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_explorer.py +0 -0
  26. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_io.py +0 -0
  27. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_plotting.py +0 -0
  28. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_simulation.py +0 -0
  29. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/_stability.py +0 -0
  30. {structboost-0.3.0 → structboost-0.5.0}/src/structboost/py.typed +0 -0
  31. {structboost-0.3.0 → structboost-0.5.0}/tests/test_allboost.py +0 -0
  32. {structboost-0.3.0 → structboost-0.5.0}/tests/test_annotation.py +0 -0
  33. {structboost-0.3.0 → structboost-0.5.0}/tests/test_bae_diagnostics.py +0 -0
  34. {structboost-0.3.0 → structboost-0.5.0}/tests/test_bae_layer.py +0 -0
  35. {structboost-0.3.0 → structboost-0.5.0}/tests/test_batch_integration.py +0 -0
  36. {structboost-0.3.0 → structboost-0.5.0}/tests/test_covcache_precompute.py +0 -0
  37. {structboost-0.3.0 → structboost-0.5.0}/tests/test_explorer.py +0 -0
  38. {structboost-0.3.0 → structboost-0.5.0}/tests/test_linear_ceiling.py +0 -0
  39. {structboost-0.3.0 → structboost-0.5.0}/tests/test_obs_encoding.py +0 -0
  40. {structboost-0.3.0 → structboost-0.5.0}/tests/test_public_api.py +0 -0
  41. {structboost-0.3.0 → structboost-0.5.0}/tests/test_resolve_mandatory.py +0 -0
  42. {structboost-0.3.0 → structboost-0.5.0}/tests/test_rng_isolation.py +0 -0
  43. {structboost-0.3.0 → structboost-0.5.0}/tests/test_simulation.py +0 -0
@@ -3,6 +3,186 @@
3
3
  Releases follow [semantic versioning](https://semver.org). While the project is
4
4
  pre-1.0, a minor bump may break API.
5
5
 
6
+ ### [0.5.0] - 2026-08-21
7
+
8
+ **Breaking**, in two parts: the decoder update became one pass over the cells,
9
+ and the disentanglement default changed.
10
+
11
+ **`BAEConfig.decoder_updates_per_iteration` is gone.** Each training
12
+ iteration now gives the decoder **one shuffled pass over the cells** —
13
+ `ceil(n_cells / batch_size)` AdamW steps, with every cell contributing to exactly
14
+ one of them — so the step count is derived from the data rather than set.
15
+
16
+ The reason is an asymmetry that grew with dataset size. The boosting half of the
17
+ alternation is full-batch at every size: `z*` is computed on all cells and the
18
+ encoder is re-solved from zero against all cells, every iteration. A fixed
19
+ decoder step count made a *cell's* participation depend on how many other cells
20
+ existed — at the old `10 × 512` the decoder saw every cell below 5,120 cells,
21
+ 31% at 16,000 and 5% at 100,000, while the decoder that defines the target for
22
+ all of them had been trained on that shrinking slice. Tying the budget to the
23
+ data makes both halves consume the same cells per iteration, and it is the same
24
+ per-cell invariance that `target_optim_lr` already provides on the target side.
25
+
26
+ **No quality claim is attached to this change, because none could be
27
+ established.** On human pancreas (16,382 cells, 2,000 batch-aware HVGs, nine
28
+ protocols, `nu=0.3`, 1000 iterations, one seed) the epoch rule raised in-sample
29
+ variance explained from 0.3100 to 0.3217 at `stepno=50` and 0.3414 to 0.3556 at
30
+ `stepno=100` — but in-sample reconstruction rewards the arm taking 3.2x the
31
+ steps, so that is not evidence. Cell-type silhouette moved the other way,
32
+ 0.298 → 0.172 and 0.327 → 0.169; on an 80/20 split of the same data it moved the
33
+ *opposite* way again (0.265 → 0.333, 0.218 → 0.268). Two conditions, opposite
34
+ orderings, one seed each: within run-to-run variance for a method whose encoder
35
+ support random-walks. The case for the change is the invariance above, not a
36
+ measured improvement.
37
+
38
+ What is established is the cost: **roughly 2x wall clock** at this dataset size
39
+ (99s → 201s at `stepno=50`, 131s → 238s at `stepno=100`), and it grows with cell
40
+ count because `max_iterations` does not decay to compensate. scVI, which is
41
+ epoch-based in the same way, pairs epochs with a `max_epochs` cap that decays as
42
+ 1/n; no such heuristic is added here, so raising `batch_size` is the lever.
43
+
44
+ Keeping a dial (`decoder_epochs_per_iteration = 1.0`, same default, still
45
+ tunable) was considered and rejected: the goal was to remove a setting, and a
46
+ float multiplier on a pass is a second way to say what `batch_size` already says.
47
+ The consequence is that decoder effort per boosting iteration is no longer
48
+ adjustable except through `batch_size`.
49
+
50
+ **Small datasets need a smaller `batch_size` than the default suggests.** Below
51
+ about 1,000 cells the default 512 yields one or two steps per iteration, and
52
+ effects that require the decoder to move *within* an iteration weaken or reverse.
53
+ Measured on 300 cells, a PCA warm start raised the first iteration's loss at one
54
+ step per iteration (1.083 against 1.023 for a zero start), tied at ten, and only
55
+ paid off at thirty-eight (0.920 against 1.000). The guide documents this.
56
+
57
+ **Migration:** drop the argument. `BAEConfig(decoder_updates_per_iteration=10)`
58
+ now raises `TypeError`. Setting `batch_size = ceil(n_cells / 10)` reproduces the
59
+ old *step count*, though not the old batch size, so results will differ either
60
+ way.
61
+
62
+ **`disentanglement="leave_one_out"` is renamed to `"orthogonal"`, and is now the
63
+ default.** It replaces the boosting targets with the nearest mutually orthogonal
64
+ set of the same column norms — symmetric Löwdin orthogonalization, computed from
65
+ the thin SVD as `U @ Vt`, since `T (T'T)^{-1/2} = U V'`. The old name is refused
66
+ with a message naming the new one, rather than falling through to the generic
67
+ "must be one of" error.
68
+
69
+ The previous implementation was not an orthogonalization. It regressed each
70
+ target column on all the others and kept the residual, which yields
71
+ `corr(r_j, r_k) = -rho_{jk|rest}` exactly: the marginal correlation structure
72
+ replaced by the *negated partial* correlation structure rather than removed. In
73
+ two dimensions that reduces to `corr -> -corr`, an exact sign flip achieving
74
+ nothing; in higher dimensions it could amplify a correlation, or manufacture one
75
+ between columns that were near-independent. Measured on human pancreas it reached
76
+ mean `|corr|` 0.048 against 0.033 for a correct orthogonalization, while selecting
77
+ fewer genes (276 against 294) and reconstructing slightly worse.
78
+
79
+ Symmetric rather than sequential (Gram-Schmidt), which also orthogonalizes but
80
+ depends on the order the columns are visited — it returns the last dimension
81
+ untouched and strips the first hardest. Latent indices permute freely, which is
82
+ why `stability_selection` matches them before counting, so an order-dependent
83
+ transform would impose an arbitrary hierarchy. The two measured within seed noise
84
+ of each other on both real datasets. Löwdin is also cheaper: `O(n * d**2)` against
85
+ `O(n * d**3)` for anything running one least-squares solve per column, measured
86
+ 6-16x faster and widening with latent width, and it needs no inverse so a
87
+ rank-deficient target matrix orthogonalizes instead of raising.
88
+
89
+ **New `BAEConfig.disentanglement_alpha`, default `1.0`.** Targets become
90
+ `(1 - alpha) * targets + alpha * orthogonalized`, so decorrelation can be softened
91
+ by choosing an alpha strictly between 0 and 1; `0.0` is equivalent to
92
+ `disentanglement="none"`. Interpolation is meaningful because Löwdin returns the
93
+ *closest* orthogonal matrix, so the endpoints are already sign-aligned.
94
+
95
+ **`disentanglement_lambda` default raised from `1e-4` to `1e-2`.** The old default
96
+ was inert: on simulated data, Tasic mouse cortex and human pancreas it moved the
97
+ mean absolute latent correlation by less than its own seed-to-seed noise, and so
98
+ did `1e-3`. On pancreas, mean `|corr|` ran 0.103 (off), 0.100 (`1e-4`), 0.112
99
+ (`1e-3`), 0.080 (`1e-2`), 0.061 (`1e-1`). The documented sweep grid, which topped
100
+ out at `1e-3`, ended below where the penalty begins to act; the guide now sweeps
101
+ upward from `1e-2`.
102
+
103
+ **Expect the default to cost some biological structure.** Decorrelation is an
104
+ extra constraint and real gene programs are not orthogonal, so it trades fidelity
105
+ to that structure, and the interpretability resting on it, for a less redundant
106
+ representation. Measured against ground truth on simulated data, marker-recovery
107
+ F1 moved from 0.98 to 0.88; on Tasic and pancreas the share of latent variance
108
+ explained by the annotated cell type fell. `disentanglement="none"` remains a
109
+ reasonable choice, and `disentanglement_alpha` exists so the trade can be made
110
+ partially.
111
+
112
+ **Migration:** `disentanglement="leave_one_out"` becomes `"orthogonal"`, or
113
+ `"none"` to restore the 0.4.0 default of no constraint. Note that `"orthogonal"`
114
+ is not a renamed version of the old behaviour — the method itself changed, so a
115
+ 0.4.0 fit is reproduced by `"none"`, not by the new name.
116
+
117
+ **New `BAE.stability_selection(continue_optimizer=...)`, default `False`.** Carries
118
+ the decoder's AdamW moment estimates over from `fit` instead of restarting them at
119
+ zero. A fresh AdamW restarts its step counter, so bias correction begins again and
120
+ `exp_avg_sq` needs on the order of `1/(1 - beta2) = 1000` steps to become a usable
121
+ variance estimate; at `ceil(n_cells / batch_size)` steps per iteration that
122
+ transient spans roughly `1000 * batch_size / n_cells` of the counted iterations —
123
+ brief on a large dataset, most of the window on a small one, and it falls at the
124
+ end where the support is furthest from equilibrium.
125
+
126
+ The state is the one from the iteration `fit` *restored*, not from its last, so the
127
+ moments belong to the decoder actually returned. It is held in memory only: `save`
128
+ excludes optimizer state, so a model read back from a checkpoint warns and falls
129
+ back rather than silently measuring something different. A caller who lowered
130
+ `config.decoder_lr` for the counting phase keeps that change — only the moments are
131
+ carried across. Default `False` preserves the behaviour every earlier result was
132
+ measured under.
133
+
134
+ **Checkpoints written by 0.4.0 no longer load.** `restore_payload` splats the
135
+ stored config into `BAEConfig`, so a format-5 file's `decoder_updates_per_iteration`
136
+ is an unexpected keyword argument. The format is bumped to 6 and the loader
137
+ refuses 5 by name. No migration is written: the setting no longer exists, so
138
+ there is nothing to migrate it to, and a checkpoint is cheap to regenerate.
139
+
140
+ ### [0.4.0] - 2026-08-12
141
+
142
+ Three defaults change. No configuration fields are added or removed, and anyone
143
+ who sets these explicitly is unaffected.
144
+
145
+ **`enable_early_stopping` now defaults to `False`.** The criterion is a
146
+ convergence check being used as a quality check. There is no validation split, so
147
+ the training loss cannot see a model that is starting to memorize, and patience
148
+ fires long before the model is done: measured against ground truth it stopped at
149
+ iteration 90 on a dataset peaking at 259, and at 196 on a simulated scenario
150
+ peaking at 560, returning marker-recovery F1 0.502 against 0.787. Across three
151
+ real datasets (mouse cortex, human pancreas, human immune) the best iteration
152
+ ranged from 154 to 1975 — always past where patience fires.
153
+
154
+ **No stopping rule replaced it, deliberately.** Latent stability, encoder-support
155
+ overlap and held-out reconstruction were each measured as candidates and each
156
+ rejected. The representation settles long before gene selection does — on one
157
+ dataset consecutive latent codes were rank-identical while the gene set still
158
+ turned over 65% cumulatively — and no observable signal tracks the quality peak.
159
+ A latent-stability rule was built and tuned; it fired at iteration ~109 on all
160
+ three real datasets regardless of where quality peaked, and on one of them it was
161
+ worse than not stopping at all. It is not shipped, not even off by default: an
162
+ option that should never be enabled is pure carrying cost, which is the same
163
+ argument that removed four fields in 0.3.0. `max_iterations=1000` is a defensible
164
+ middle of the measured range, not an optimum.
165
+
166
+ **`BAE.stability_selection(threshold=...)` now defaults to `0.5`, from `0.7`.**
167
+ 0.7 is too aggressive whenever the latent representation is still moving: across
168
+ three real datasets it removed 21-52% of recovered marker genes relative to the
169
+ fitted encoder, and 0-7% even when the representation had settled. At 0.5 the
170
+ worst loss over the same six runs was 6%. The standalone
171
+ `structboost.stability_selection` keeps 0.7, because its Meinshausen-Buhlmann
172
+ bound is undefined at or below 0.5.
173
+
174
+ **The `dim_match_quality` warning now fires below 0.85, from 0.5.** It is the gate
175
+ on whether iteration frequencies mean anything: they describe gene-set drift only
176
+ if the counted iterations describe one representation. Runs sitting at 0.70-0.79 —
177
+ comfortably above the old warning — already lost a quarter to a half of their
178
+ recovered markers at the default threshold, while runs at 0.93 and above lost
179
+ none. The guide now documents it as a lookup: at least 0.93, either threshold is
180
+ safe; 0.70-0.79, use 0.3-0.5 and prefer the flat union.
181
+
182
+ Also fixed: the gene-selection guide still showed
183
+ `fit(adata, stability_selection="iteration")`, which stopped being valid in 0.3.0
184
+ when that argument became a bool.
185
+
6
186
  ### [0.3.0] - 2026-08-11
7
187
 
8
188
  **Breaking.** Five settings are gone and `BAE.stability_selection` has one mode
@@ -2,8 +2,8 @@ cff-version: 1.2.0
2
2
  message: "If you use structboost in your research, please cite it."
3
3
  title: "structboost"
4
4
  type: software
5
- version: "0.3.0"
6
- date-released: "2026-08-11"
5
+ version: "0.5.0"
6
+ date-released: "2026-08-21"
7
7
  license: MIT
8
8
  authors:
9
9
  - family-names: Brunn
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: structboost
3
- Version: 0.3.0
3
+ Version: 0.5.0
4
4
  Summary: Boosting Autoencoders (BAE) and componentwise L2 boosting utilities for scverse-style workflows.
5
5
  Project-URL: Homepage, https://github.com/NiklasBrunn/structboost
6
6
  Project-URL: Documentation, https://niklasbrunn.github.io/structboost
@@ -85,9 +85,12 @@ The **Boosting Autoencoder (BAE)** pairs a linear encoder fitted by componentwis
85
85
  L2 boosting with an MLP decoder trained by gradient descent. Each training
86
86
  iteration takes a gradient step on the latent code itself and hands the result to
87
87
  the boosting fit as a regression target, so the encoder is fitted against the
88
- negative gradient of the reconstruction loss rather than by backpropagation.
89
- Componentwise boosting adds one gene at a time and shrinks each step, which keeps
90
- the encoder weights sparse by construction rather than by a post-hoc threshold.
88
+ gradient-updated latent code rather than by backpropagation. The target is that
89
+ updated code rather than the gradient alone because the encoder is rebuilt from
90
+ zero every iteration: it has to reproduce where the code should be, not the
91
+ correction to where it already is. Componentwise boosting adds one gene at a time
92
+ and shrinks each step, which keeps the encoder weights sparse by construction
93
+ rather than by a post-hoc threshold.
91
94
 
92
95
  Each latent dimension is therefore a short, signed gene list, and `X_bae` is
93
96
  exactly `X @ varm["BAE_encoder_weights"]`.
@@ -147,16 +150,26 @@ adata.obsm["X_bae"] # (n_cells, 10) latent space
147
150
  adata.varm["BAE_encoder_weights"] # (n_genes, 10), sparse
148
151
  ```
149
152
 
150
- A single fit gives one gene list, and that list is **not reproducible**. In a
151
- high-dimensional feature space with strongly correlated genes the encoder support
152
- is not identifiable: many different sparse gene sets reconstruct the data about
153
- equally well, and a fit returns one of them.
153
+ To integrate over a batch or any other unwanted covariate, name the obs column:
154
154
 
155
155
  ```python
156
- res = model.stability_selection(adata)
157
- genes = [adata.var_names[res.stable_support[:, j]] for j in range(res.frequency.shape[1])]
156
+ model.fit(adata, batch_key="batch") # or ["batch", "donor"]
157
+
158
+ adata.uns["bae"]["latent_obs_r2_per_dim"] # near zero means it worked
158
159
  ```
159
160
 
161
+ By default this both conditions the decoder on the covariate and adds it to the
162
+ boosting design as a mandatory regressor, so a batch-correlated gene is not
163
+ selected *because of* the batch. The covariate is never an encoder input, so
164
+ `transform` stays gene-only and needs no batch labels on new data.
165
+
166
+ A single fit gives one gene list, and that list is **not reproducible**: in a
167
+ high-dimensional feature space with strongly correlated genes the encoder support
168
+ is not identifiable, so many different sparse gene sets reconstruct the data about
169
+ equally well and a fit returns one of them. See
170
+ [gene selection](https://niklasbrunn.github.io/structboost/guide/tasks/gene-selection.html)
171
+ before trusting a single list.
172
+
160
173
  Componentwise L2 boosting on its own, no autoencoder involved:
161
174
 
162
175
  ```python
@@ -173,11 +186,25 @@ Each of these has a guide page.
173
186
  | | |
174
187
  | --- | --- |
175
188
  | [Batch integration](https://niklasbrunn.github.io/structboost/guide/tasks/batch-integration.html) | `batch_key` names the covariate and `batch_integration_mode` chooses whether it conditions the decoder, protects gene selection, or both. `transform` stays gene-only and needs no batch labels. |
176
- | [Transfer](https://niklasbrunn.github.io/structboost/guide/tasks/transfer.html) | Carry a trained encoder matrix onto a new dataset with `from_reference`, aligned by gene name, with the prior programs frozen. |
189
+ | [Mandatory features](https://niklasbrunn.github.io/structboost/guide/tasks/gene-selection.html) | `mandatory_genes` puts known markers in boosting's unpenalized adjustment block, so they are never subject to competitive selection. Flat or per latent dimension. It forces them into the *specification*, not into the fitted support. |
177
190
  | [Persistence](https://niklasbrunn.github.io/structboost/guide/tasks/persistence.html) | `save` and `load` a fitted model as one checkpoint, readable with `weights_only=True`. |
178
191
  | [Interpretation](https://niklasbrunn.github.io/structboost/guide/tasks/interpreting.html) | Ranked gene lists per dimension, stored functional annotations, and a self-contained interactive HTML explorer. |
179
192
  | [Simulation](https://niklasbrunn.github.io/structboost/guide/tasks/simulating.html) | Negative-binomial counts with planted gene programs and a cell-type hierarchy, so marker recovery can be scored against ground truth. |
180
193
 
194
+ ## Exploratory features
195
+
196
+ The whole package is pre-1.0, but these four are **under active development** and
197
+ less settled than the rest. They work, and each is documented with what is known
198
+ about it — but their behaviour, defaults and APIs are more likely to change, and
199
+ results from them warrant more scepticism than the core fit does.
200
+
201
+ | Feature | Why it is still exploratory |
202
+ | --- | --- |
203
+ | **Stability selection** (`BAE.stability_selection`) | Provides **no formal error control** — `expected_false_positives` is deliberately `NaN`, because training iterations are neither independent nor exchangeable. Per-dimension frequencies are only meaningful when dimensions keep their identity, which `dim_match_quality` reports and does not guarantee. |
204
+ | **Disentanglement** (`disentanglement=`) | On by default since 0.5.0, and both methods are provisional. Decorrelation is an extra constraint that real gene programs do not satisfy, so it costs biological structure — measured at marker-recovery F1 0.98 to 0.88 on simulated data. `disentanglement_alpha` softens it; `"none"` turns it off. |
205
+ | **Starting from an existing representation** (`init_pca`, `init_obsm`) | The warm start is applied once, on the first iteration, and silently overrides `latent_dim` if the supplied representation is a different width. It also depends on the decoder having enough steps in that first iteration to follow it — on few cells at the default `batch_size` the effect reverses. |
206
+ | **Starting from a prior encoder matrix** (`BAE.from_reference`) | Transfer works, but the two latent blocks land on **incomparable scales** — measured at a 232× gap in per-dimension standard deviation — so anything Euclidean must be handed `obsm["X_bae_scaled"]` rather than `X_bae`. `novel_variance_share` is not evidence of novel biology on the fitting data. |
207
+
181
208
  ## Citation
182
209
 
183
210
  If you use the **BAE**:
@@ -204,6 +231,27 @@ Everything committed was reviewed, and the behavioural claims in the docstrings
204
231
  and the user guide are backed by the test suite or by the measurements cited
205
232
  alongside them.
206
233
 
234
+ ## Open points
235
+
236
+ Known gaps and planned work.
237
+
238
+ - [ ] **Revise the early-stopping criterion.** Early stopping is off by default
239
+ because the training-loss rule is a convergence check being used as a quality
240
+ check, and it stops well before gene selection has settled. No replacement has
241
+ been found yet.
242
+ - [ ] **Stability selection.** Iteration mode carries no formal error control, and
243
+ its per-dimension frequencies are only interpretable when `dim_match_quality` is
244
+ high. A scheme with a defensible bound under a model fitted on the same cells is
245
+ still open.
246
+ - [ ] **Multimodal architecture** — reconstruction-based, with a shared latent
247
+ space across modalities, for paired single-cell data.
248
+ - [ ] **Contrastive objective**, as an alternative or addition to the
249
+ reconstruction target the boosting step is currently fitted against.
250
+ - [ ] **Stochastic gradient boosting** ([Friedman
251
+ 2002](https://doi.org/10.1016/S0167-9473(01)00065-2)): subsample the cells at
252
+ each boosting step, which is both a regularizer and a route to cheaper
253
+ iterations on large datasets.
254
+
207
255
  ## Contributing
208
256
 
209
257
  See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, the versioning
@@ -10,9 +10,12 @@ The **Boosting Autoencoder (BAE)** pairs a linear encoder fitted by componentwis
10
10
  L2 boosting with an MLP decoder trained by gradient descent. Each training
11
11
  iteration takes a gradient step on the latent code itself and hands the result to
12
12
  the boosting fit as a regression target, so the encoder is fitted against the
13
- negative gradient of the reconstruction loss rather than by backpropagation.
14
- Componentwise boosting adds one gene at a time and shrinks each step, which keeps
15
- the encoder weights sparse by construction rather than by a post-hoc threshold.
13
+ gradient-updated latent code rather than by backpropagation. The target is that
14
+ updated code rather than the gradient alone because the encoder is rebuilt from
15
+ zero every iteration: it has to reproduce where the code should be, not the
16
+ correction to where it already is. Componentwise boosting adds one gene at a time
17
+ and shrinks each step, which keeps the encoder weights sparse by construction
18
+ rather than by a post-hoc threshold.
16
19
 
17
20
  Each latent dimension is therefore a short, signed gene list, and `X_bae` is
18
21
  exactly `X @ varm["BAE_encoder_weights"]`.
@@ -72,16 +75,26 @@ adata.obsm["X_bae"] # (n_cells, 10) latent space
72
75
  adata.varm["BAE_encoder_weights"] # (n_genes, 10), sparse
73
76
  ```
74
77
 
75
- A single fit gives one gene list, and that list is **not reproducible**. In a
76
- high-dimensional feature space with strongly correlated genes the encoder support
77
- is not identifiable: many different sparse gene sets reconstruct the data about
78
- equally well, and a fit returns one of them.
78
+ To integrate over a batch or any other unwanted covariate, name the obs column:
79
79
 
80
80
  ```python
81
- res = model.stability_selection(adata)
82
- genes = [adata.var_names[res.stable_support[:, j]] for j in range(res.frequency.shape[1])]
81
+ model.fit(adata, batch_key="batch") # or ["batch", "donor"]
82
+
83
+ adata.uns["bae"]["latent_obs_r2_per_dim"] # near zero means it worked
83
84
  ```
84
85
 
86
+ By default this both conditions the decoder on the covariate and adds it to the
87
+ boosting design as a mandatory regressor, so a batch-correlated gene is not
88
+ selected *because of* the batch. The covariate is never an encoder input, so
89
+ `transform` stays gene-only and needs no batch labels on new data.
90
+
91
+ A single fit gives one gene list, and that list is **not reproducible**: in a
92
+ high-dimensional feature space with strongly correlated genes the encoder support
93
+ is not identifiable, so many different sparse gene sets reconstruct the data about
94
+ equally well and a fit returns one of them. See
95
+ [gene selection](https://niklasbrunn.github.io/structboost/guide/tasks/gene-selection.html)
96
+ before trusting a single list.
97
+
85
98
  Componentwise L2 boosting on its own, no autoencoder involved:
86
99
 
87
100
  ```python
@@ -98,11 +111,25 @@ Each of these has a guide page.
98
111
  | | |
99
112
  | --- | --- |
100
113
  | [Batch integration](https://niklasbrunn.github.io/structboost/guide/tasks/batch-integration.html) | `batch_key` names the covariate and `batch_integration_mode` chooses whether it conditions the decoder, protects gene selection, or both. `transform` stays gene-only and needs no batch labels. |
101
- | [Transfer](https://niklasbrunn.github.io/structboost/guide/tasks/transfer.html) | Carry a trained encoder matrix onto a new dataset with `from_reference`, aligned by gene name, with the prior programs frozen. |
114
+ | [Mandatory features](https://niklasbrunn.github.io/structboost/guide/tasks/gene-selection.html) | `mandatory_genes` puts known markers in boosting's unpenalized adjustment block, so they are never subject to competitive selection. Flat or per latent dimension. It forces them into the *specification*, not into the fitted support. |
102
115
  | [Persistence](https://niklasbrunn.github.io/structboost/guide/tasks/persistence.html) | `save` and `load` a fitted model as one checkpoint, readable with `weights_only=True`. |
103
116
  | [Interpretation](https://niklasbrunn.github.io/structboost/guide/tasks/interpreting.html) | Ranked gene lists per dimension, stored functional annotations, and a self-contained interactive HTML explorer. |
104
117
  | [Simulation](https://niklasbrunn.github.io/structboost/guide/tasks/simulating.html) | Negative-binomial counts with planted gene programs and a cell-type hierarchy, so marker recovery can be scored against ground truth. |
105
118
 
119
+ ## Exploratory features
120
+
121
+ The whole package is pre-1.0, but these four are **under active development** and
122
+ less settled than the rest. They work, and each is documented with what is known
123
+ about it — but their behaviour, defaults and APIs are more likely to change, and
124
+ results from them warrant more scepticism than the core fit does.
125
+
126
+ | Feature | Why it is still exploratory |
127
+ | --- | --- |
128
+ | **Stability selection** (`BAE.stability_selection`) | Provides **no formal error control** — `expected_false_positives` is deliberately `NaN`, because training iterations are neither independent nor exchangeable. Per-dimension frequencies are only meaningful when dimensions keep their identity, which `dim_match_quality` reports and does not guarantee. |
129
+ | **Disentanglement** (`disentanglement=`) | On by default since 0.5.0, and both methods are provisional. Decorrelation is an extra constraint that real gene programs do not satisfy, so it costs biological structure — measured at marker-recovery F1 0.98 to 0.88 on simulated data. `disentanglement_alpha` softens it; `"none"` turns it off. |
130
+ | **Starting from an existing representation** (`init_pca`, `init_obsm`) | The warm start is applied once, on the first iteration, and silently overrides `latent_dim` if the supplied representation is a different width. It also depends on the decoder having enough steps in that first iteration to follow it — on few cells at the default `batch_size` the effect reverses. |
131
+ | **Starting from a prior encoder matrix** (`BAE.from_reference`) | Transfer works, but the two latent blocks land on **incomparable scales** — measured at a 232× gap in per-dimension standard deviation — so anything Euclidean must be handed `obsm["X_bae_scaled"]` rather than `X_bae`. `novel_variance_share` is not evidence of novel biology on the fitting data. |
132
+
106
133
  ## Citation
107
134
 
108
135
  If you use the **BAE**:
@@ -129,6 +156,27 @@ Everything committed was reviewed, and the behavioural claims in the docstrings
129
156
  and the user guide are backed by the test suite or by the measurements cited
130
157
  alongside them.
131
158
 
159
+ ## Open points
160
+
161
+ Known gaps and planned work.
162
+
163
+ - [ ] **Revise the early-stopping criterion.** Early stopping is off by default
164
+ because the training-loss rule is a convergence check being used as a quality
165
+ check, and it stops well before gene selection has settled. No replacement has
166
+ been found yet.
167
+ - [ ] **Stability selection.** Iteration mode carries no formal error control, and
168
+ its per-dimension frequencies are only interpretable when `dim_match_quality` is
169
+ high. A scheme with a defensible bound under a model fitted on the same cells is
170
+ still open.
171
+ - [ ] **Multimodal architecture** — reconstruction-based, with a shared latent
172
+ space across modalities, for paired single-cell data.
173
+ - [ ] **Contrastive objective**, as an alternative or addition to the
174
+ reconstruction target the boosting step is currently fitted against.
175
+ - [ ] **Stochastic gradient boosting** ([Friedman
176
+ 2002](https://doi.org/10.1016/S0167-9473(01)00065-2)): subsample the cells at
177
+ each boosting step, which is both a regularizer and a route to cheaper
178
+ iterations on large datasets.
179
+
132
180
  ## Contributing
133
181
 
134
182
  See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, the versioning
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
4
4
 
5
5
  [project]
6
6
  name = "structboost"
7
- version = "0.3.0"
7
+ version = "0.5.0"
8
8
  description = "Boosting Autoencoders (BAE) and componentwise L2 boosting utilities for scverse-style workflows."
9
9
  readme = "README.md"
10
10
  license = { file = "LICENSE" }