structboost 0.4.0__tar.gz → 0.6.0__tar.gz

{structboost-0.4.0 → structboost-0.6.0}/CHANGELOG.md

@@ -3,6 +3,234 @@
 Releases follow [semantic versioning](https://semver.org). While the project is
 pre-1.0, a minor bump may break API.
 
+### [0.6.0] - 2026-09-09
+
+**Not breaking.** Nothing existing changes behaviour; this adds a way to read a
+fitted model.
+
+**New `plot_latent_dimensions`: one row per latent dimension, up to three
+panels.** The sorted score curve, the per-gene contribution distributions, and
+the scores split by a grouping you name. It exists because a fitted BAE has been
+readable in principle — the encoder *is* the gene list — and awkward in practice:
+`extract_gene_rankings` gives ranked lists, `export_interactive_html` gives an
+explorer, and neither says whether a dimension describes a subgroup or a
+gradient, nor which cells sit at which end.
+
+**New `plot_dimension_gene_umaps`: a UMAP grid, dimensions by top genes**, drawn
+through `scanpy.pl.umap` so the panels match a scanpy figure. It answers the one
+question the other panels structurally cannot — whether a dimension's genes light
+up the *same* cells — because both the score curve and the contribution violins
+have already summed over cells. On real data it separates a coherent programme
+from a dimension summing unrelated signals.
+
+**A `"shares"` panel and a new `plot_dimension_correlation`.** The contribution
+panel shows a dimension's top handful of genes; the shares panel plots *every*
+selected gene's share, sorted, which is the only view that says whether a
+dimension rests on three genes or spreads evenly over forty — both legitimate,
+and read differently. Ordering it by `rank_by="weight"` while the axis stays on
+the share draws the disagreement between the two rankings directly.
+
+The correlation heatmap answers a question no per-dimension panel can: whether
+two dimensions are near-duplicates, and therefore whether they can be read as two
+findings or one. Pearson by default, Spearman on tie-averaged ranks as an option —
+ties are not incidental here, since a sparse encoder leaves many cells at exactly
+one value on a subgroup dimension. Signed values get the diverging map centred on
+zero; `absolute=True` plots `|r|` with a sequential one, since a magnitude has no
+midpoint.
+
+**Genes are ranked by variance share, not `|weight|`,** exposed as the new
+`gene_variance_shares`. From `Var(s) = Cov(s, s)` with `s = X @ w` it follows
+that `w_g Cov(X_g, s) / Var(s)` decomposes a dimension's variance exactly and
+sums to 1, needing no orthogonality assumption and splitting credit correctly
+between correlated genes. The weight alone ignores how much a gene actually
+varies: measured on 59k PBMCs the two rankings agree on 8.6 of 10 genes per
+dimension, and where they differ `|w|` promotes genes that barely move — SCT
+ranked #4 by weight and #35 of 38 by share, being near-absent in that tissue.
+The share is a **magnitude**: because a negative-weight gene is anti-correlated
+with the score, the product is positive either way and only ~1% of genes come out
+negative. Direction is therefore the weight's sign, carried by the colour of the
+gene name, and the share prints without one.
+
+**The score panel paints in a seeded random order.** Markers are wider than the
+spacing between adjacent ranks — about 112 cells overlap at a given x on 59k —
+so whatever is drawn last wins every overlap. Painting in score order hands each
+contest to the higher-scoring cell, and since score correlates with group, one
+group systematically covers its neighbours. Every point keeps its own
+`(rank, score)`, so the curve is unchanged and only the paint order is permuted.
+This is the *opposite* choice from a UMAP panel, where rare groups are drawn last
+on purpose: there position is data and overlap is unavoidable, here every cell
+has its own x and any deterministic order is a bias.
+
+**Seven hues, then grey, and the palette is measured rather than chosen** — see
+the new `palette_audit`. The package's existing Okabe-Ito set separates colours
+from each other under colour-blindness but does not hold contrast against white
+at small mark sizes: its `#F0E442` measures 1.32 WCAG contrast against a floor of
+3.0, so a cell type holding 5.9% of cells was invisible as 8pt dots while its
+violin read perfectly well. **Contrast requirements scale with mark size, so a
+palette validated on one mark is not validated on another.** At twelve hues
+nothing separates anyway — 1.5 ΔE under simulated deuteranopia against a target
+of 8; Kelly's "20 colours of maximum contrast" reaches 0.4, scanpy's `default_20`
+0.5. Twenty reliably distinguishable hues do not exist. The default therefore
+follows the group count (`tab10`'s first five, then `tab10`, then scanpy's
+`default_20`), groups past the palette are grey, and they lose nothing because
+the violin panel labels every row — the score panel, which has no labels, is
+where the cap binds.
+
+**Signed quantities get a diverging map centred on zero; sequential is for
+magnitudes.** The score always, and the gene panels as soon as `scale="zscore"`,
+where a sequential map would put its neutral colour at the data mean rather than
+at zero. The encoder is bias-free, so zero is a real level — no selected gene
+departs from its mean in that cell — and the point at which a dimension's sign
+flips.
+
+`scanpy` becomes a new optional extra, needed only by `plot_dimension_gene_umaps`
+and imported inside it. Absent, that one function raises naming
+`plot_latent_dimensions` as the matplotlib-only alternative; everything else is
+unaffected.
+
+Two things are deliberately *not* offered. `c_ig / score_ik`, the intuitive
+"fraction of this cell's score", is unusable: the flat plateau of a subgroup
+dimension is exactly the cells whose score is ≈0, so the ratio diverges (measured
+−8,921 to +1,191 for one gene), flips sign with the score, and inflates without
+bound wherever contributions cancel — `normalize="cell"` divides by total
+*absolute* contribution instead, which is bounded. And nothing here is
+inferential: every number is an exact algebraic decomposition or a descriptive
+statistic.
+
+### [0.5.0] - 2026-08-21
+
+**Breaking**, in two parts: the decoder update became one pass over the cells,
+and the disentanglement default changed.
+
+**`BAEConfig.decoder_updates_per_iteration` is gone.** Each training
+iteration now gives the decoder **one shuffled pass over the cells** —
+`ceil(n_cells / batch_size)` AdamW steps, with every cell contributing to exactly
+one of them — so the step count is derived from the data rather than set.
+
+The reason is an asymmetry that grew with dataset size. The boosting half of the
+alternation is full-batch at every size: `z*` is computed on all cells and the
+encoder is re-solved from zero against all cells, every iteration. A fixed
+decoder step count made a *cell's* participation depend on how many other cells
+existed — at the old `10 × 512` the decoder saw every cell below 5,120 cells,
+31% at 16,000 and 5% at 100,000, while the decoder that defines the target for
+all of them had been trained on that shrinking slice. Tying the budget to the
+data makes both halves consume the same cells per iteration, and it is the same
+per-cell invariance that `target_optim_lr` already provides on the target side.
+
+**No quality claim is attached to this change, because none could be
+established.** On human pancreas (16,382 cells, 2,000 batch-aware HVGs, nine
+protocols, `nu=0.3`, 1000 iterations, one seed) the epoch rule raised in-sample
+variance explained from 0.3100 to 0.3217 at `stepno=50` and 0.3414 to 0.3556 at
+`stepno=100` — but in-sample reconstruction rewards the arm taking 3.2x the
+steps, so that is not evidence. Cell-type silhouette moved the other way,
+0.298 → 0.172 and 0.327 → 0.169; on an 80/20 split of the same data it moved the
+*opposite* way again (0.265 → 0.333, 0.218 → 0.268). Two conditions, opposite
+orderings, one seed each: within run-to-run variance for a method whose encoder
+support random-walks. The case for the change is the invariance above, not a
+measured improvement.
+
+What is established is the cost: **roughly 2x wall clock** at this dataset size
+(99s → 201s at `stepno=50`, 131s → 238s at `stepno=100`), and it grows with cell
+count because `max_iterations` does not decay to compensate. scVI, which is
+epoch-based in the same way, pairs epochs with a `max_epochs` cap that decays as
+1/n; no such heuristic is added here, so raising `batch_size` is the lever.
+
+Keeping a dial (`decoder_epochs_per_iteration = 1.0`, same default, still
+tunable) was considered and rejected: the goal was to remove a setting, and a
+float multiplier on a pass is a second way to say what `batch_size` already says.
+The consequence is that decoder effort per boosting iteration is no longer
+adjustable except through `batch_size`.
+
+**Small datasets need a smaller `batch_size` than the default suggests.** Below
+about 1,000 cells the default 512 yields one or two steps per iteration, and
+effects that require the decoder to move *within* an iteration weaken or reverse.
+Measured on 300 cells, a PCA warm start raised the first iteration's loss at one
+step per iteration (1.083 against 1.023 for a zero start), tied at ten, and only
+paid off at thirty-eight (0.920 against 1.000). The guide documents this.
+
+**Migration:** drop the argument. `BAEConfig(decoder_updates_per_iteration=10)`
+now raises `TypeError`. Setting `batch_size = ceil(n_cells / 10)` reproduces the
+old *step count*, though not the old batch size, so results will differ either
+way.
+
+**`disentanglement="leave_one_out"` is renamed to `"orthogonal"`, and is now the
+default.** It replaces the boosting targets with the nearest mutually orthogonal
+set of the same column norms — symmetric Löwdin orthogonalization, computed from
+the thin SVD as `U @ Vt`, since `T (T'T)^{-1/2} = U V'`. The old name is refused
+with a message naming the new one, rather than falling through to the generic
+"must be one of" error.
+
+The previous implementation was not an orthogonalization. It regressed each
+target column on all the others and kept the residual, which yields
+`corr(r_j, r_k) = -rho_{jk|rest}` exactly: the marginal correlation structure
+replaced by the *negated partial* correlation structure rather than removed. In
+two dimensions that reduces to `corr -> -corr`, an exact sign flip achieving
+nothing; in higher dimensions it could amplify a correlation, or manufacture one
+between columns that were near-independent. Measured on human pancreas it reached
+mean `|corr|` 0.048 against 0.033 for a correct orthogonalization, while selecting
+fewer genes (276 against 294) and reconstructing slightly worse.
+
+Symmetric rather than sequential (Gram-Schmidt), which also orthogonalizes but
+depends on the order the columns are visited — it returns the last dimension
+untouched and strips the first hardest. Latent indices permute freely, which is
+why `stability_selection` matches them before counting, so an order-dependent
+transform would impose an arbitrary hierarchy. The two measured within seed noise
+of each other on both real datasets. Löwdin is also cheaper: `O(n * d**2)` against
+`O(n * d**3)` for anything running one least-squares solve per column, measured
+6-16x faster and widening with latent width, and it needs no inverse so a
+rank-deficient target matrix orthogonalizes instead of raising.
+
+**New `BAEConfig.disentanglement_alpha`, default `1.0`.** Targets become
+`(1 - alpha) * targets + alpha * orthogonalized`, so decorrelation can be softened
+by choosing an alpha strictly between 0 and 1; `0.0` is equivalent to
+`disentanglement="none"`. Interpolation is meaningful because Löwdin returns the
+*closest* orthogonal matrix, so the endpoints are already sign-aligned.
+
+**`disentanglement_lambda` default raised from `1e-4` to `1e-2`.** The old default
+was inert: on simulated data, Tasic mouse cortex and human pancreas it moved the
+mean absolute latent correlation by less than its own seed-to-seed noise, and so
+did `1e-3`. On pancreas, mean `|corr|` ran 0.103 (off), 0.100 (`1e-4`), 0.112
+(`1e-3`), 0.080 (`1e-2`), 0.061 (`1e-1`). The documented sweep grid, which topped
+out at `1e-3`, ended below where the penalty begins to act; the guide now sweeps
+upward from `1e-2`.
+
+**Expect the default to cost some biological structure.** Decorrelation is an
+extra constraint and real gene programs are not orthogonal, so it trades fidelity
+to that structure, and the interpretability resting on it, for a less redundant
+representation. Measured against ground truth on simulated data, marker-recovery
+F1 moved from 0.98 to 0.88; on Tasic and pancreas the share of latent variance
+explained by the annotated cell type fell. `disentanglement="none"` remains a
+reasonable choice, and `disentanglement_alpha` exists so the trade can be made
+partially.
+
+**Migration:** `disentanglement="leave_one_out"` becomes `"orthogonal"`, or
+`"none"` to restore the 0.4.0 default of no constraint. Note that `"orthogonal"`
+is not a renamed version of the old behaviour — the method itself changed, so a
+0.4.0 fit is reproduced by `"none"`, not by the new name.
+
+**New `BAE.stability_selection(continue_optimizer=...)`, default `False`.** Carries
+the decoder's AdamW moment estimates over from `fit` instead of restarting them at
+zero. A fresh AdamW restarts its step counter, so bias correction begins again and
+`exp_avg_sq` needs on the order of `1/(1 - beta2) = 1000` steps to become a usable
+variance estimate; at `ceil(n_cells / batch_size)` steps per iteration that
+transient spans roughly `1000 * batch_size / n_cells` of the counted iterations —
+brief on a large dataset, most of the window on a small one, and it falls at the
+end where the support is furthest from equilibrium.
+
+The state is the one from the iteration `fit` *restored*, not from its last, so the
+moments belong to the decoder actually returned. It is held in memory only: `save`
+excludes optimizer state, so a model read back from a checkpoint warns and falls
+back rather than silently measuring something different. A caller who lowered
+`config.decoder_lr` for the counting phase keeps that change — only the moments are
+carried across. Default `False` preserves the behaviour every earlier result was
+measured under.
+
+**Checkpoints written by 0.4.0 no longer load.** `restore_payload` splats the
+stored config into `BAEConfig`, so a format-5 file's `decoder_updates_per_iteration`
+is an unexpected keyword argument. The format is bumped to 6 and the loader
+refuses 5 by name. No migration is written: the setting no longer exists, so
+there is nothing to migrate it to, and a checkpoint is cheap to regenerate.
+
 ### [0.4.0] - 2026-08-12
 
 Three defaults change. No configuration fields are added or removed, and anyone

{structboost-0.4.0 → structboost-0.6.0}/CITATION.cff

@@ -2,8 +2,8 @@ cff-version: 1.2.0
 message: "If you use structboost in your research, please cite it."
 title: "structboost"
 type: software
-version: "0.4.0"
-date-released: "2026-08-11"
+version: "0.5.0"
+date-released: "2026-08-21"
 license: MIT
 authors:
   - family-names: Brunn

{structboost-0.4.0 → structboost-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: structboost
-Version: 0.4.0
+Version: 0.6.0
 Summary: Boosting Autoencoders (BAE) and componentwise L2 boosting utilities for scverse-style workflows.
 Project-URL: Homepage, https://github.com/NiklasBrunn/structboost
 Project-URL: Documentation, https://niklasbrunn.github.io/structboost
@@ -64,6 +64,9 @@ Requires-Dist: pandas>=1.5; extra == 'io'
 Requires-Dist: pyarrow>=12; extra == 'io'
 Provides-Extra: plot
 Requires-Dist: matplotlib>=3.7; extra == 'plot'
+Provides-Extra: scanpy
+Requires-Dist: matplotlib>=3.7; extra == 'scanpy'
+Requires-Dist: scanpy>=1.9; extra == 'scanpy'
 Provides-Extra: test
 Requires-Dist: anndata>=0.10; extra == 'test'
 Requires-Dist: pandas>=1.5; extra == 'test'
@@ -85,9 +88,12 @@ The **Boosting Autoencoder (BAE)** pairs a linear encoder fitted by componentwis
 L2 boosting with an MLP decoder trained by gradient descent. Each training
 iteration takes a gradient step on the latent code itself and hands the result to
 the boosting fit as a regression target, so the encoder is fitted against the
-negative gradient of the reconstruction loss rather than by backpropagation.
-Componentwise boosting adds one gene at a time and shrinks each step, which keeps
-the encoder weights sparse by construction rather than by a post-hoc threshold.
+gradient-updated latent code rather than by backpropagation. The target is that
+updated code rather than the gradient alone because the encoder is rebuilt from
+zero every iteration: it has to reproduce where the code should be, not the
+correction to where it already is. Componentwise boosting adds one gene at a time
+and shrinks each step, which keeps the encoder weights sparse by construction
+rather than by a post-hoc threshold.
 
 Each latent dimension is therefore a short, signed gene list, and `X_bae` is
 exactly `X @ varm["BAE_encoder_weights"]`.
@@ -147,16 +153,26 @@ adata.obsm["X_bae"]                 # (n_cells, 10) latent space
 adata.varm["BAE_encoder_weights"]   # (n_genes, 10), sparse
 ```
 
-A single fit gives one gene list, and that list is **not reproducible**. In a
-high-dimensional feature space with strongly correlated genes the encoder support
-is not identifiable: many different sparse gene sets reconstruct the data about
-equally well, and a fit returns one of them.
+To integrate over a batch or any other unwanted covariate, name the obs column:
 
 ```python
-res = model.stability_selection(adata)
-genes = [adata.var_names[res.stable_support[:, j]] for j in range(res.frequency.shape[1])]
+model.fit(adata, batch_key="batch")          # or ["batch", "donor"]
+
+adata.uns["bae"]["latent_obs_r2_per_dim"]    # near zero means it worked
 ```
 
+By default this both conditions the decoder on the covariate and adds it to the
+boosting design as a mandatory regressor, so a batch-correlated gene is not
+selected *because of* the batch. The covariate is never an encoder input, so
+`transform` stays gene-only and needs no batch labels on new data.
+
+A single fit gives one gene list, and that list is **not reproducible**: in a
+high-dimensional feature space with strongly correlated genes the encoder support
+is not identifiable, so many different sparse gene sets reconstruct the data about
+equally well and a fit returns one of them. See
+[gene selection](https://niklasbrunn.github.io/structboost/guide/tasks/gene-selection.html)
+before trusting a single list.
+
 Componentwise L2 boosting on its own, no autoencoder involved:
 
 ```python
@@ -173,11 +189,26 @@ Each of these has a guide page.
 | | |
 | --- | --- |
 | [Batch integration](https://niklasbrunn.github.io/structboost/guide/tasks/batch-integration.html) | `batch_key` names the covariate and `batch_integration_mode` chooses whether it conditions the decoder, protects gene selection, or both. `transform` stays gene-only and needs no batch labels. |
-| [Transfer](https://niklasbrunn.github.io/structboost/guide/tasks/transfer.html) | Carry a trained encoder matrix onto a new dataset with `from_reference`, aligned by gene name, with the prior programs frozen. |
+| [Mandatory features](https://niklasbrunn.github.io/structboost/guide/tasks/gene-selection.html) | `mandatory_genes` puts known markers in boosting's unpenalized adjustment block, so they are never subject to competitive selection. Flat or per latent dimension. It forces them into the *specification*, not into the fitted support. |
+| [Reading a fit](https://niklasbrunn.github.io/structboost/guide/tasks/reading-dimensions.html) | `plot_latent_dimensions` draws one row per latent dimension — the sorted score curve, the per-gene contributions, and the scores split by any grouping — plus a UMAP grid and a dimension-correlation heatmap. Genes are ranked by their exact share of a dimension's variance, not by coefficient size. |
 | [Persistence](https://niklasbrunn.github.io/structboost/guide/tasks/persistence.html) | `save` and `load` a fitted model as one checkpoint, readable with `weights_only=True`. |
 | [Interpretation](https://niklasbrunn.github.io/structboost/guide/tasks/interpreting.html) | Ranked gene lists per dimension, stored functional annotations, and a self-contained interactive HTML explorer. |
 | [Simulation](https://niklasbrunn.github.io/structboost/guide/tasks/simulating.html) | Negative-binomial counts with planted gene programs and a cell-type hierarchy, so marker recovery can be scored against ground truth. |
 
+## Exploratory features
+
+The whole package is pre-1.0, but these four are **under active development** and
+less settled than the rest. They work, and each is documented with what is known
+about it — but their behaviour, defaults and APIs are more likely to change, and
+results from them warrant more scepticism than the core fit does.
+
+| Feature | Why it is still exploratory |
+| --- | --- |
+| **Stability selection** (`BAE.stability_selection`) | Provides **no formal error control** — `expected_false_positives` is deliberately `NaN`, because training iterations are neither independent nor exchangeable. Per-dimension frequencies are only meaningful when dimensions keep their identity, which `dim_match_quality` reports and does not guarantee. |
+| **Disentanglement** (`disentanglement=`) | On by default since 0.5.0, and both methods are provisional. Decorrelation is an extra constraint that real gene programs do not satisfy, so it costs biological structure — measured at marker-recovery F1 0.98 to 0.88 on simulated data. `disentanglement_alpha` softens it; `"none"` turns it off. |
+| **Starting from an existing representation** (`init_pca`, `init_obsm`) | The warm start is applied once, on the first iteration, and silently overrides `latent_dim` if the supplied representation is a different width. It also depends on the decoder having enough steps in that first iteration to follow it — on few cells at the default `batch_size` the effect reverses. |
+| **Starting from a prior encoder matrix** (`BAE.from_reference`) | Transfer works, but the two latent blocks land on **incomparable scales** — measured at a 232× gap in per-dimension standard deviation — so anything Euclidean must be handed `obsm["X_bae_scaled"]` rather than `X_bae`. `novel_variance_share` is not evidence of novel biology on the fitting data. |
+
 ## Citation
 
 If you use the **BAE**:
@@ -204,6 +235,27 @@ Everything committed was reviewed, and the behavioural claims in the docstrings
 and the user guide are backed by the test suite or by the measurements cited
 alongside them.
 
+## Open points
+
+Known gaps and planned work.
+
+- [ ] **Revise the early-stopping criterion.** Early stopping is off by default
+  because the training-loss rule is a convergence check being used as a quality
+  check, and it stops well before gene selection has settled. No replacement has
+  been found yet.
+- [ ] **Stability selection.** Iteration mode carries no formal error control, and
+  its per-dimension frequencies are only interpretable when `dim_match_quality` is
+  high. A scheme with a defensible bound under a model fitted on the same cells is
+  still open.
+- [ ] **Multimodal architecture** — reconstruction-based, with a shared latent
+  space across modalities, for paired single-cell data.
+- [ ] **Contrastive objective**, as an alternative or addition to the
+  reconstruction target the boosting step is currently fitted against.
+- [ ] **Stochastic gradient boosting** ([Friedman
+  2002](https://doi.org/10.1016/S0167-9473(01)00065-2)): subsample the cells at
+  each boosting step, which is both a regularizer and a route to cheaper
+  iterations on large datasets.
+
 ## Contributing
 
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, the versioning

{structboost-0.4.0 → structboost-0.6.0}/README.md

@@ -10,9 +10,12 @@ The **Boosting Autoencoder (BAE)** pairs a linear encoder fitted by componentwis
 L2 boosting with an MLP decoder trained by gradient descent. Each training
 iteration takes a gradient step on the latent code itself and hands the result to
 the boosting fit as a regression target, so the encoder is fitted against the
-negative gradient of the reconstruction loss rather than by backpropagation.
-Componentwise boosting adds one gene at a time and shrinks each step, which keeps
-the encoder weights sparse by construction rather than by a post-hoc threshold.
+gradient-updated latent code rather than by backpropagation. The target is that
+updated code rather than the gradient alone because the encoder is rebuilt from
+zero every iteration: it has to reproduce where the code should be, not the
+correction to where it already is. Componentwise boosting adds one gene at a time
+and shrinks each step, which keeps the encoder weights sparse by construction
+rather than by a post-hoc threshold.
 
 Each latent dimension is therefore a short, signed gene list, and `X_bae` is
 exactly `X @ varm["BAE_encoder_weights"]`.
@@ -72,16 +75,26 @@ adata.obsm["X_bae"]                 # (n_cells, 10) latent space
 adata.varm["BAE_encoder_weights"]   # (n_genes, 10), sparse
 ```
 
-A single fit gives one gene list, and that list is **not reproducible**. In a
-high-dimensional feature space with strongly correlated genes the encoder support
-is not identifiable: many different sparse gene sets reconstruct the data about
-equally well, and a fit returns one of them.
+To integrate over a batch or any other unwanted covariate, name the obs column:
 
 ```python
-res = model.stability_selection(adata)
-genes = [adata.var_names[res.stable_support[:, j]] for j in range(res.frequency.shape[1])]
+model.fit(adata, batch_key="batch")          # or ["batch", "donor"]
+
+adata.uns["bae"]["latent_obs_r2_per_dim"]    # near zero means it worked
 ```
 
+By default this both conditions the decoder on the covariate and adds it to the
+boosting design as a mandatory regressor, so a batch-correlated gene is not
+selected *because of* the batch. The covariate is never an encoder input, so
+`transform` stays gene-only and needs no batch labels on new data.
+
+A single fit gives one gene list, and that list is **not reproducible**: in a
+high-dimensional feature space with strongly correlated genes the encoder support
+is not identifiable, so many different sparse gene sets reconstruct the data about
+equally well and a fit returns one of them. See
+[gene selection](https://niklasbrunn.github.io/structboost/guide/tasks/gene-selection.html)
+before trusting a single list.
+
 Componentwise L2 boosting on its own, no autoencoder involved:
 
 ```python
@@ -98,11 +111,26 @@ Each of these has a guide page.
 | | |
 | --- | --- |
 | [Batch integration](https://niklasbrunn.github.io/structboost/guide/tasks/batch-integration.html) | `batch_key` names the covariate and `batch_integration_mode` chooses whether it conditions the decoder, protects gene selection, or both. `transform` stays gene-only and needs no batch labels. |
-| [Transfer](https://niklasbrunn.github.io/structboost/guide/tasks/transfer.html) | Carry a trained encoder matrix onto a new dataset with `from_reference`, aligned by gene name, with the prior programs frozen. |
+| [Mandatory features](https://niklasbrunn.github.io/structboost/guide/tasks/gene-selection.html) | `mandatory_genes` puts known markers in boosting's unpenalized adjustment block, so they are never subject to competitive selection. Flat or per latent dimension. It forces them into the *specification*, not into the fitted support. |
+| [Reading a fit](https://niklasbrunn.github.io/structboost/guide/tasks/reading-dimensions.html) | `plot_latent_dimensions` draws one row per latent dimension — the sorted score curve, the per-gene contributions, and the scores split by any grouping — plus a UMAP grid and a dimension-correlation heatmap. Genes are ranked by their exact share of a dimension's variance, not by coefficient size. |
 | [Persistence](https://niklasbrunn.github.io/structboost/guide/tasks/persistence.html) | `save` and `load` a fitted model as one checkpoint, readable with `weights_only=True`. |
 | [Interpretation](https://niklasbrunn.github.io/structboost/guide/tasks/interpreting.html) | Ranked gene lists per dimension, stored functional annotations, and a self-contained interactive HTML explorer. |
 | [Simulation](https://niklasbrunn.github.io/structboost/guide/tasks/simulating.html) | Negative-binomial counts with planted gene programs and a cell-type hierarchy, so marker recovery can be scored against ground truth. |
 
+## Exploratory features
+
+The whole package is pre-1.0, but these four are **under active development** and
+less settled than the rest. They work, and each is documented with what is known
+about it — but their behaviour, defaults and APIs are more likely to change, and
+results from them warrant more scepticism than the core fit does.
+
+| Feature | Why it is still exploratory |
+| --- | --- |
+| **Stability selection** (`BAE.stability_selection`) | Provides **no formal error control** — `expected_false_positives` is deliberately `NaN`, because training iterations are neither independent nor exchangeable. Per-dimension frequencies are only meaningful when dimensions keep their identity, which `dim_match_quality` reports and does not guarantee. |
+| **Disentanglement** (`disentanglement=`) | On by default since 0.5.0, and both methods are provisional. Decorrelation is an extra constraint that real gene programs do not satisfy, so it costs biological structure — measured at marker-recovery F1 0.98 to 0.88 on simulated data. `disentanglement_alpha` softens it; `"none"` turns it off. |
+| **Starting from an existing representation** (`init_pca`, `init_obsm`) | The warm start is applied once, on the first iteration, and silently overrides `latent_dim` if the supplied representation is a different width. It also depends on the decoder having enough steps in that first iteration to follow it — on few cells at the default `batch_size` the effect reverses. |
+| **Starting from a prior encoder matrix** (`BAE.from_reference`) | Transfer works, but the two latent blocks land on **incomparable scales** — measured at a 232× gap in per-dimension standard deviation — so anything Euclidean must be handed `obsm["X_bae_scaled"]` rather than `X_bae`. `novel_variance_share` is not evidence of novel biology on the fitting data. |
+
 ## Citation
 
 If you use the **BAE**:
@@ -129,6 +157,27 @@ Everything committed was reviewed, and the behavioural claims in the docstrings
 and the user guide are backed by the test suite or by the measurements cited
 alongside them.
 
+## Open points
+
+Known gaps and planned work.
+
+- [ ] **Revise the early-stopping criterion.** Early stopping is off by default
+  because the training-loss rule is a convergence check being used as a quality
+  check, and it stops well before gene selection has settled. No replacement has
+  been found yet.
+- [ ] **Stability selection.** Iteration mode carries no formal error control, and
+  its per-dimension frequencies are only interpretable when `dim_match_quality` is
+  high. A scheme with a defensible bound under a model fitted on the same cells is
+  still open.
+- [ ] **Multimodal architecture** — reconstruction-based, with a shared latent
+  space across modalities, for paired single-cell data.
+- [ ] **Contrastive objective**, as an alternative or addition to the
+  reconstruction target the boosting step is currently fitted against.
+- [ ] **Stochastic gradient boosting** ([Friedman
+  2002](https://doi.org/10.1016/S0167-9473(01)00065-2)): subsample the cells at
+  each boosting step, which is both a regularizer and a route to cheaper
+  iterations on large datasets.
+
 ## Contributing
 
 See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, the versioning

{structboost-0.4.0 → structboost-0.6.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "structboost"
-version = "0.4.0"
+version = "0.6.0"
 description = "Boosting Autoencoders (BAE) and componentwise L2 boosting utilities for scverse-style workflows."
 readme = "README.md"
 license = { file = "LICENSE" }
@@ -41,6 +41,16 @@ bae = [
 plot = [
   "matplotlib>=3.7",
 ]
+# scanpy, for `plot_dimension_gene_umaps` alone. It draws through `sc.pl.umap` so
+# its panels match a scanpy figure rather than approximating one, and it is
+# imported inside that function: without it the function raises naming
+# `plot_latent_dimensions` as the matplotlib-only alternative, and nothing else in
+# the package notices. Kept out of `plot` deliberately -- scanpy pulls a far larger
+# tree than matplotlib, and the other plot functions do not need it.
+scanpy = [
+  "scanpy>=1.9",
+  "structboost[plot]",
+]
 # Parquet encoder-weight files (imported lazily; CSV needs only pandas)
 io = [
   "pandas>=1.5",

{structboost-0.4.0 → structboost-0.6.0}/src/structboost/__init__.py

@@ -56,9 +56,14 @@ __all__ = [
     "encode_obs_covariates",
     "export_interactive_html",
     "extract_gene_rankings",
+    "gene_variance_shares",
     "linear_ceiling",
     "looks_like_ensembl",
+    "palette_audit",
     "plot_boosting_coefficient_paths",
+    "plot_dimension_correlation",
+    "plot_dimension_gene_umaps",
+    "plot_latent_dimensions",
     "plot_top_boosting_coefficients",
     "plot_training_diagnostics",
     "read_encoder_weights",
@@ -70,9 +75,17 @@ __all__ = [
     "write_encoder_weights",
 ]
 
+#: Resolved lazily through `__getattr__`. `gene_variance_shares` and
+#: `palette_audit` live in `_plotting` beside their callers but need no
+#: matplotlib -- that module imports it inside the drawing functions only.
 _PLOT_FUNCTIONS = frozenset(
     {
+        "gene_variance_shares",
+        "palette_audit",
         "plot_boosting_coefficient_paths",
+        "plot_dimension_correlation",
+        "plot_dimension_gene_umaps",
+        "plot_latent_dimensions",
         "plot_top_boosting_coefficients",
         "plot_training_diagnostics",
     }