combatlearn 2.3.0__tar.gz → 2.4.0__tar.gz

{combatlearn-2.3.0 → combatlearn-2.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: combatlearn
-Version: 2.3.0
+Version: 2.4.0
 Summary: Batch-effect harmonization for machine learning frameworks.
 Author-email: Ettore Rocchi <ettoreroc@gmail.com>
 License-Expression: MIT
@@ -18,6 +18,7 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: pandas>=1.3
 Requires-Dist: numpy>=1.21
+Requires-Dist: scipy>=1.8
 Requires-Dist: scikit-learn>=1.2
 Requires-Dist: matplotlib>=3.4
 Requires-Dist: seaborn>=0.12
@@ -33,6 +34,7 @@ Requires-Dist: pre-commit>=3.0; extra == "dev"
 Requires-Dist: mypy>=1.0; extra == "dev"
 Requires-Dist: pandas-stubs; extra == "dev"
 Requires-Dist: statsmodels>=0.14; extra == "dev"
+Requires-Dist: inmoose>=0.7; extra == "dev"
 Provides-Extra: docs
 Requires-Dist: sphinx>=7.0.0; extra == "docs"
 Requires-Dist: pydata-sphinx-theme>=0.15.0; extra == "docs"
@@ -45,7 +47,7 @@ Dynamic: license-file
 
 [![Python versions](https://img.shields.io/badge/python-%3E%3D3.10-blue?logo=python)](https://www.python.org/)
 [![Test](https://github.com/EttoreRocchi/combatlearn/actions/workflows/test.yaml/badge.svg)](https://github.com/EttoreRocchi/combatlearn/actions/workflows/test.yaml)
-[![Documentation](https://readthedocs.org/projects/combatlearn/badge/?version=latest)](https://combatlearn.readthedocs.io)
+[![Documentation](https://app.readthedocs.org/projects/combatlearn/badge/?version=latest)](https://combatlearn.readthedocs.io)
 [![PyPI Version](https://img.shields.io/pypi/v/combatlearn?cacheSeconds=300)](https://pypi.org/project/combatlearn/)
 [![License](https://img.shields.io/github/license/EttoreRocchi/combatlearn)](https://github.com/EttoreRocchi/combatlearn/blob/main/LICENSE)
 
@@ -62,9 +64,15 @@ Dynamic: license-file
 - `method="gam"` - ComBat-GAM, nonlinear (spline) covariate effects (Pomponio _et al._, 2020)
 - `method="covbat_gam"` - CovBat with the same nonlinear covariate modeling
 
-**Multiple batch variables** - `NestedComBat` harmonizes over several batch variables at once (e.g. site, scanner, protocol) with optional order optimization and Gaussian-mixture grouping (Nested / OPNested / GMM ComBat, Horng _et al._, 2022).
+**Inductive estimators with their own signature** (same fit/transform contract, separate classes):
+- `NestedComBat` - harmonizes over several batch variables at once (e.g. site, scanner, protocol) with optional order optimization and Gaussian-mixture grouping (Nested / OPNested / GMM ComBat, Horng _et al._, 2022)
+- `LongitudinalComBat` - repeated-measures / longitudinal designs with a per-subject random intercept (Beer _et al._, 2020). Inductive under a stricter precondition than `ComBat`: the batch **and** the subject must have been seen at fit; unseen subjects degrade to the population correction, or raise with `on_unseen_subject="error"`
+- `GLMComBat` - non-Gaussian data via a parametric GLM plus a strictly per-observation quantile map:
+  - `method="seq"` - ComBat-seq for raw RNA-seq counts, negative-binomial with a library-size offset (Zhang _et al._, 2020). Integer in, integer out
+  - `method="met"` - ComBat-met for DNA-methylation beta-values, beta regression with a logit link (Wang, 2025). Output stays strictly inside `(0, 1)`
+  - Both are **cross-validation-safe**, which no published implementation of either method offers: upstream they are whole-cohort functions
 
-**Whole-cohort (transductive)** - `combatlearn.transductive.TransductiveComBat(method="longitudinal")` for repeated-measures / longitudinal designs (Beer _et al._, 2020), used as a one-shot `fit_transform`. (`ComBat(method="longitudinal")` still works but is deprecated in favor of this and will be removed in v3.0.0.)
+**Whole-cohort (transductive)** - `combatlearn.transductive.TransductiveComBat` holds configurations that cannot be frozen at fit, namely those protecting a condition-of-interest / outcome term in the design, used as a one-shot `fit_transform`. Its principal occupants are `method="seq"` and `method="met"`, the condition-protected ComBat-seq / ComBat-met variants for differential expression and differential methylation. `method="longitudinal"` is deprecated in favor of `LongitudinalComBat`, as is `ComBat(method="longitudinal")`; both are removed in v3.0.0.
 
 ## Installation
 
@@ -151,6 +159,52 @@ X_corrected = nested.fit_transform(X)
 print("Harmonization order:", nested.order_)
 ```
 
+## Counts and proportions (`GLMComBat`)
+
+RNA-seq counts and methylation beta-values are not Gaussian, and standardizing them destroys the structure downstream tools rely on. `GLMComBat` fits a negative-binomial GLM (`method="seq"`) or a beta regression (`method="met"`) and then maps each observation through the CDF of its own batch's fitted distribution into the corresponding quantile of a batch-free one.
+
+That quantile map is **strictly per observation** - it uses only the sample's own measurements, batch label, covariates and (for `"seq"`) library size - so the model freezes at `fit` and `transform` applies to held-out data. The published ComBat-seq and ComBat-met implementations are whole-cohort functions with no such split.
+
+```python
+from combatlearn import GLMComBat
+
+harmonizer = GLMComBat(
+    batch=batch,
+    method="seq",                 # raw integer counts; "met" for 0-1 beta-values
+    discrete_covariates=tissue,   # ordinary covariates, preserved
+    reference_batch="site_A",     # recommended under cross-validation
+)
+
+counts_corrected = harmonizer.fit_transform(counts)   # reproduces the reference exactly
+
+harmonizer.fit(counts_train)                          # ... or freeze and reuse
+counts_test_corrected = harmonizer.transform(counts_test)
+```
+
+`fit_transform` on a whole cohort is verified bit-exact against InMoose `pycombat_seq`, and to within 1e-6 against committed fixtures from the ComBat-met R package. The **inductive** path has no external reference to compare with - it is validated by its defining property, that a held-out sample is corrected identically alone or in company.
+
+If the term you want to protect *is* the supervised target of a differential analysis on the same cohort, use `TransductiveComBat(method="seq"/"met", condition=...)` instead; if it is known at intake, it is an ordinary covariate and belongs here.
+
+## Repeated measures (`LongitudinalComBat`)
+
+When the same subjects are measured more than once, `LongitudinalComBat` adds a per-subject random intercept (REML) to the Fortin model, so within-subject correlation is modeled rather than treated as noise. It keeps the `fit`/`transform` contract, so a frozen correction can be applied to later visits of subjects already in the reference cohort.
+
+```python
+from combatlearn import LongitudinalComBat
+
+harmonizer = LongitudinalComBat(
+    batch=batch,
+    subject_id=subject,        # required: one label per sample
+    time_covariate=visit,      # optional: continuous time variable
+    continuous_covariates=age,
+)
+
+X_corrected = harmonizer.fit_transform(X)          # whole cohort in one pass
+X_new = harmonizer.fit(X_ref).transform(X_visits)  # or freeze and apply
+```
+
+Its precondition is one step stricter than `ComBat`'s: `ComBat` needs the *batch* to have been seen at fit, `LongitudinalComBat` needs the *batch and the subject*. A subject unseen at fit degrades to the population correction (zero random intercept), or raises with `on_unseen_subject="error"`. The transform is leakage-free in every split - each subject's intercept is estimated from that subject's training visits only - but for cross-validated prediction on *genuinely new* subjects there is no advantage over `method="fortin"`; the payoff is incremental harmonization of returning subjects.
+
 ## `ComBat` parameters
 
 The following section provides a detailed explanation of all parameters available in the scikit-learn-compatible `ComBat` class. For complete API documentation, see the [API Reference](https://combatlearn.readthedocs.io/en/latest/api/).
@@ -169,11 +223,11 @@ The following section provides a detailed explanation of all parameters availabl
 
 | Parameter | Type | Default | Description |
 | --- | --- | --- | --- |
-| `method` | str | `"johnson"` | ComBat method to use: <ul><li>`"johnson"` - Classical ComBat (_Johnson et al. 2007_)</li><li>`"fortin"` - ComBat with covariates (_Fortin et al. 2018_)</li><li>`"chen"` - CovBat, PCA-based correction (_Chen et al. 2022_)</li><li>`"longitudinal"` - Longitudinal ComBat with a per-subject random intercept (_Beer et al. 2020_); **deprecated**, use `combatlearn.transductive.TransductiveComBat`</li><li>`"gam"` - ComBat-GAM, nonlinear (spline) covariates (_Pomponio et al. 2020_)</li><li>`"covbat_gam"` - CovBat with the same nonlinear covariate modeling</li></ul> Case- and separator-insensitive literature aliases are also accepted (e.g. `"covbat"`, `"neurocombat"`, `"combat_gam"`). |
-| `parametric` | bool | `True` | Whether to use the **parametric empirical Bayes** formulation. If `False`, a non-parametric iterative scheme is used. |
-| `mean_only` | bool | `False` | If `True`, only the **mean** is corrected, while variances are left unchanged. Useful for preserving variance structure in the data. |
+| `method` | str | `"johnson"` | ComBat method to use: <ul><li>`"johnson"` - Classical ComBat (_Johnson et al. 2007_)</li><li>`"fortin"` - ComBat with covariates (_Fortin et al. 2018_)</li><li>`"chen"` - CovBat, PCA-based correction (_Chen et al. 2022_)</li><li>`"longitudinal"` - Longitudinal ComBat with a per-subject random intercept (_Beer et al. 2020_); **deprecated**, use `combatlearn.LongitudinalComBat`</li><li>`"gam"` - ComBat-GAM, nonlinear (spline) covariates (_Pomponio et al. 2020_)</li><li>`"covbat_gam"` - CovBat with the same nonlinear covariate modeling</li></ul> Case- and separator-insensitive literature aliases are also accepted (e.g. `"covbat"`, `"neurocombat"`, `"combat_gam"`). |
+| `parametric` | bool | `True` | Whether to use the **parametric empirical Bayes** formulation. If `False`, the non-parametric estimate of `sva::ComBat` (`int.eprior`) is used, whose cost grows with the square of the number of features. |
+| `mean_only` | bool | `False` | If `True`, only the **mean** is corrected, while variances are left unchanged. Useful for preserving variance structure in the data. For `"chen"` and `"covbat_gam"` it applies to the ComBat step only; the covariance step still harmonizes the PC-score variances. |
 | `reference_batch` | str or `None` | `None` | If specified, acts as a reference batch - other batches will be corrected to match this one. |
-| `covbat_cov_thresh` | float, int | `0.9` | For `"chen"` method only: Cumulative variance threshold in `(0, 1]` to retain PCs in PCA space (e.g., 0.9 = retain 90% explained variance). If an integer is provided, it represents the number of principal components to use. |
+| `covbat_cov_thresh` | float, int | `0.9` | For the `"chen"` and `"covbat_gam"` methods (also inside `NestedComBat`): Cumulative variance threshold in `(0, 1]` to retain PCs in PCA space (e.g., 0.9 = retain 90% explained variance). If an integer is provided, it represents the number of principal components to use. |
 | `smooth_terms` | list of str or int, or `None` | `None` | For `"gam"`/`"covbat_gam"` only: continuous covariates to model nonlinearly with B-splines, by column name or integer position. `None` smooths all continuous covariates. |
 | `spline_df` | int | `10` | For the GAM methods: B-spline degrees of freedom (basis functions) per smooth term. |
 | `spline_degree` | int | `3` | For the GAM methods: B-spline degree (3 = cubic). |
@@ -235,3 +289,7 @@ Please consider citing the original papers:
 - [**Longitudinal ComBat**](https://github.com/jcbeer/longCombat) - Beer JC et al. _Neuroimage_. 2020. doi: [10.1016/j.neuroimage.2020.117129](https://doi.org/10.1016/j.neuroimage.2020.117129)
 
 - [**ComBat-GAM**](https://github.com/rpomponio/neuroHarmonize) - Pomponio R et al. _Neuroimage_. 2020. doi: [10.1016/j.neuroimage.2019.116450](https://doi.org/10.1016/j.neuroimage.2019.116450)
+
+- [**ComBat-seq**](https://github.com/zhangyuqing/ComBat-seq) - Zhang Y, Parmigiani G, Johnson WE. _NAR Genom Bioinform_. 2020. doi: [10.1093/nargab/lqaa078](https://doi.org/10.1093/nargab/lqaa078)
+
+- [**ComBat-met**](https://github.com/JmWangBio/ComBatMet) - Wang J. _NAR Genom Bioinform_. 2025. doi: [10.1093/nargab/lqaf062](https://doi.org/10.1093/nargab/lqaf062)

{combatlearn-2.3.0 → combatlearn-2.4.0}/README.md

@@ -2,7 +2,7 @@
 
 [![Python versions](https://img.shields.io/badge/python-%3E%3D3.10-blue?logo=python)](https://www.python.org/)
 [![Test](https://github.com/EttoreRocchi/combatlearn/actions/workflows/test.yaml/badge.svg)](https://github.com/EttoreRocchi/combatlearn/actions/workflows/test.yaml)
-[![Documentation](https://readthedocs.org/projects/combatlearn/badge/?version=latest)](https://combatlearn.readthedocs.io)
+[![Documentation](https://app.readthedocs.org/projects/combatlearn/badge/?version=latest)](https://combatlearn.readthedocs.io)
 [![PyPI Version](https://img.shields.io/pypi/v/combatlearn?cacheSeconds=300)](https://pypi.org/project/combatlearn/)
 [![License](https://img.shields.io/github/license/EttoreRocchi/combatlearn)](https://github.com/EttoreRocchi/combatlearn/blob/main/LICENSE)
 
@@ -19,9 +19,15 @@
 - `method="gam"` - ComBat-GAM, nonlinear (spline) covariate effects (Pomponio _et al._, 2020)
 - `method="covbat_gam"` - CovBat with the same nonlinear covariate modeling
 
-**Multiple batch variables** - `NestedComBat` harmonizes over several batch variables at once (e.g. site, scanner, protocol) with optional order optimization and Gaussian-mixture grouping (Nested / OPNested / GMM ComBat, Horng _et al._, 2022).
+**Inductive estimators with their own signature** (same fit/transform contract, separate classes):
+- `NestedComBat` - harmonizes over several batch variables at once (e.g. site, scanner, protocol) with optional order optimization and Gaussian-mixture grouping (Nested / OPNested / GMM ComBat, Horng _et al._, 2022)
+- `LongitudinalComBat` - repeated-measures / longitudinal designs with a per-subject random intercept (Beer _et al._, 2020). Inductive under a stricter precondition than `ComBat`: the batch **and** the subject must have been seen at fit; unseen subjects degrade to the population correction, or raise with `on_unseen_subject="error"`
+- `GLMComBat` - non-Gaussian data via a parametric GLM plus a strictly per-observation quantile map:
+  - `method="seq"` - ComBat-seq for raw RNA-seq counts, negative-binomial with a library-size offset (Zhang _et al._, 2020). Integer in, integer out
+  - `method="met"` - ComBat-met for DNA-methylation beta-values, beta regression with a logit link (Wang, 2025). Output stays strictly inside `(0, 1)`
+  - Both are **cross-validation-safe**, which no published implementation of either method offers: upstream they are whole-cohort functions
 
-**Whole-cohort (transductive)** - `combatlearn.transductive.TransductiveComBat(method="longitudinal")` for repeated-measures / longitudinal designs (Beer _et al._, 2020), used as a one-shot `fit_transform`. (`ComBat(method="longitudinal")` still works but is deprecated in favor of this and will be removed in v3.0.0.)
+**Whole-cohort (transductive)** - `combatlearn.transductive.TransductiveComBat` holds configurations that cannot be frozen at fit, namely those protecting a condition-of-interest / outcome term in the design, used as a one-shot `fit_transform`. Its principal occupants are `method="seq"` and `method="met"`, the condition-protected ComBat-seq / ComBat-met variants for differential expression and differential methylation. `method="longitudinal"` is deprecated in favor of `LongitudinalComBat`, as is `ComBat(method="longitudinal")`; both are removed in v3.0.0.
 
 ## Installation
 
@@ -108,6 +114,52 @@ X_corrected = nested.fit_transform(X)
 print("Harmonization order:", nested.order_)
 ```
 
+## Counts and proportions (`GLMComBat`)
+
+RNA-seq counts and methylation beta-values are not Gaussian, and standardizing them destroys the structure downstream tools rely on. `GLMComBat` fits a negative-binomial GLM (`method="seq"`) or a beta regression (`method="met"`) and then maps each observation through the CDF of its own batch's fitted distribution into the corresponding quantile of a batch-free one.
+
+That quantile map is **strictly per observation** - it uses only the sample's own measurements, batch label, covariates and (for `"seq"`) library size - so the model freezes at `fit` and `transform` applies to held-out data. The published ComBat-seq and ComBat-met implementations are whole-cohort functions with no such split.
+
+```python
+from combatlearn import GLMComBat
+
+harmonizer = GLMComBat(
+    batch=batch,
+    method="seq",                 # raw integer counts; "met" for 0-1 beta-values
+    discrete_covariates=tissue,   # ordinary covariates, preserved
+    reference_batch="site_A",     # recommended under cross-validation
+)
+
+counts_corrected = harmonizer.fit_transform(counts)   # reproduces the reference exactly
+
+harmonizer.fit(counts_train)                          # ... or freeze and reuse
+counts_test_corrected = harmonizer.transform(counts_test)
+```
+
+`fit_transform` on a whole cohort is verified bit-exact against InMoose `pycombat_seq`, and to within 1e-6 against committed fixtures from the ComBat-met R package. The **inductive** path has no external reference to compare with - it is validated by its defining property, that a held-out sample is corrected identically alone or in company.
+
+If the term you want to protect *is* the supervised target of a differential analysis on the same cohort, use `TransductiveComBat(method="seq"/"met", condition=...)` instead; if it is known at intake, it is an ordinary covariate and belongs here.
+
+## Repeated measures (`LongitudinalComBat`)
+
+When the same subjects are measured more than once, `LongitudinalComBat` adds a per-subject random intercept (REML) to the Fortin model, so within-subject correlation is modeled rather than treated as noise. It keeps the `fit`/`transform` contract, so a frozen correction can be applied to later visits of subjects already in the reference cohort.
+
+```python
+from combatlearn import LongitudinalComBat
+
+harmonizer = LongitudinalComBat(
+    batch=batch,
+    subject_id=subject,        # required: one label per sample
+    time_covariate=visit,      # optional: continuous time variable
+    continuous_covariates=age,
+)
+
+X_corrected = harmonizer.fit_transform(X)          # whole cohort in one pass
+X_new = harmonizer.fit(X_ref).transform(X_visits)  # or freeze and apply
+```
+
+Its precondition is one step stricter than `ComBat`'s: `ComBat` needs the *batch* to have been seen at fit, `LongitudinalComBat` needs the *batch and the subject*. A subject unseen at fit degrades to the population correction (zero random intercept), or raises with `on_unseen_subject="error"`. The transform is leakage-free in every split - each subject's intercept is estimated from that subject's training visits only - but for cross-validated prediction on *genuinely new* subjects there is no advantage over `method="fortin"`; the payoff is incremental harmonization of returning subjects.
+
 ## `ComBat` parameters
 
 The following section provides a detailed explanation of all parameters available in the scikit-learn-compatible `ComBat` class. For complete API documentation, see the [API Reference](https://combatlearn.readthedocs.io/en/latest/api/).
@@ -126,11 +178,11 @@ The following section provides a detailed explanation of all parameters availabl
 
 | Parameter | Type | Default | Description |
 | --- | --- | --- | --- |
-| `method` | str | `"johnson"` | ComBat method to use: <ul><li>`"johnson"` - Classical ComBat (_Johnson et al. 2007_)</li><li>`"fortin"` - ComBat with covariates (_Fortin et al. 2018_)</li><li>`"chen"` - CovBat, PCA-based correction (_Chen et al. 2022_)</li><li>`"longitudinal"` - Longitudinal ComBat with a per-subject random intercept (_Beer et al. 2020_); **deprecated**, use `combatlearn.transductive.TransductiveComBat`</li><li>`"gam"` - ComBat-GAM, nonlinear (spline) covariates (_Pomponio et al. 2020_)</li><li>`"covbat_gam"` - CovBat with the same nonlinear covariate modeling</li></ul> Case- and separator-insensitive literature aliases are also accepted (e.g. `"covbat"`, `"neurocombat"`, `"combat_gam"`). |
-| `parametric` | bool | `True` | Whether to use the **parametric empirical Bayes** formulation. If `False`, a non-parametric iterative scheme is used. |
-| `mean_only` | bool | `False` | If `True`, only the **mean** is corrected, while variances are left unchanged. Useful for preserving variance structure in the data. |
+| `method` | str | `"johnson"` | ComBat method to use: <ul><li>`"johnson"` - Classical ComBat (_Johnson et al. 2007_)</li><li>`"fortin"` - ComBat with covariates (_Fortin et al. 2018_)</li><li>`"chen"` - CovBat, PCA-based correction (_Chen et al. 2022_)</li><li>`"longitudinal"` - Longitudinal ComBat with a per-subject random intercept (_Beer et al. 2020_); **deprecated**, use `combatlearn.LongitudinalComBat`</li><li>`"gam"` - ComBat-GAM, nonlinear (spline) covariates (_Pomponio et al. 2020_)</li><li>`"covbat_gam"` - CovBat with the same nonlinear covariate modeling</li></ul> Case- and separator-insensitive literature aliases are also accepted (e.g. `"covbat"`, `"neurocombat"`, `"combat_gam"`). |
+| `parametric` | bool | `True` | Whether to use the **parametric empirical Bayes** formulation. If `False`, the non-parametric estimate of `sva::ComBat` (`int.eprior`) is used, whose cost grows with the square of the number of features. |
+| `mean_only` | bool | `False` | If `True`, only the **mean** is corrected, while variances are left unchanged. Useful for preserving variance structure in the data. For `"chen"` and `"covbat_gam"` it applies to the ComBat step only; the covariance step still harmonizes the PC-score variances. |
 | `reference_batch` | str or `None` | `None` | If specified, acts as a reference batch - other batches will be corrected to match this one. |
-| `covbat_cov_thresh` | float, int | `0.9` | For `"chen"` method only: Cumulative variance threshold in `(0, 1]` to retain PCs in PCA space (e.g., 0.9 = retain 90% explained variance). If an integer is provided, it represents the number of principal components to use. |
+| `covbat_cov_thresh` | float, int | `0.9` | For the `"chen"` and `"covbat_gam"` methods (also inside `NestedComBat`): Cumulative variance threshold in `(0, 1]` to retain PCs in PCA space (e.g., 0.9 = retain 90% explained variance). If an integer is provided, it represents the number of principal components to use. |
 | `smooth_terms` | list of str or int, or `None` | `None` | For `"gam"`/`"covbat_gam"` only: continuous covariates to model nonlinearly with B-splines, by column name or integer position. `None` smooths all continuous covariates. |
 | `spline_df` | int | `10` | For the GAM methods: B-spline degrees of freedom (basis functions) per smooth term. |
 | `spline_degree` | int | `3` | For the GAM methods: B-spline degree (3 = cubic). |
@@ -191,4 +243,8 @@ Please consider citing the original papers:
 
 - [**Longitudinal ComBat**](https://github.com/jcbeer/longCombat) - Beer JC et al. _Neuroimage_. 2020. doi: [10.1016/j.neuroimage.2020.117129](https://doi.org/10.1016/j.neuroimage.2020.117129)
 
-- [**ComBat-GAM**](https://github.com/rpomponio/neuroHarmonize) - Pomponio R et al. _Neuroimage_. 2020. doi: [10.1016/j.neuroimage.2019.116450](https://doi.org/10.1016/j.neuroimage.2019.116450)
+- [**ComBat-GAM**](https://github.com/rpomponio/neuroHarmonize) - Pomponio R et al. _Neuroimage_. 2020. doi: [10.1016/j.neuroimage.2019.116450](https://doi.org/10.1016/j.neuroimage.2019.116450)
+
+- [**ComBat-seq**](https://github.com/zhangyuqing/ComBat-seq) - Zhang Y, Parmigiani G, Johnson WE. _NAR Genom Bioinform_. 2020. doi: [10.1093/nargab/lqaa078](https://doi.org/10.1093/nargab/lqaa078)
+
+- [**ComBat-met**](https://github.com/JmWangBio/ComBatMet) - Wang J. _NAR Genom Bioinform_. 2025. doi: [10.1093/nargab/lqaf062](https://doi.org/10.1093/nargab/lqaf062)

combatlearn-2.4.0/combatlearn/__init__.py

@@ -0,0 +1,6 @@
+from .glm import GLMComBat
+from .sklearn_api import ComBat, LongitudinalComBat, NestedComBat
+
+__all__ = ["ComBat", "GLMComBat", "LongitudinalComBat", "NestedComBat"]
+__version__ = "2.4.0"
+__author__ = "Ettore Rocchi"

{combatlearn-2.3.0 → combatlearn-2.4.0}/combatlearn/_mixed.py

@@ -1,23 +1,8 @@
-"""Random-intercept linear mixed model fitting for Longitudinal ComBat.
+"""Random-intercept linear mixed model, fit by REML, for Longitudinal ComBat.
 
-Fits, for every feature (column of ``Y``) simultaneously, the model
-
-    y = X @ beta + Z @ u + eps,    u ~ N(0, tau^2 I),   eps ~ N(0, sigma^2 I)
-
-where ``Z`` is the subject (random-intercept) indicator matrix. Estimation is
-by REML, profiling out ``beta`` and ``sigma^2`` so that only the scalar
-variance ratio ``lambda = tau^2 / sigma^2`` is optimized per feature.
-
-The single random-intercept grouping makes the marginal covariance
-``V = sigma^2 (I + lambda Z Z^T)`` block diagonal (one block per subject), so
-``V^{-1}`` is available in closed form via Sherman-Morrison and every operation
-is O(n) per feature. The variance ratio is found by a vectorized grid search
-over ``lambda`` followed by a parabolic refinement, after which the fixed
-effects, residual variance, and random-intercept BLUPs are computed in a single
-per-feature pass.
-
-This keeps Longitudinal ComBat dependency-free (numpy only); correctness is
-cross-checked against ``statsmodels.MixedLM`` in the test suite.
+Fits ``y = X beta + Z u + eps`` per feature, profiling out ``beta`` and ``sigma^2``
+so only ``lambda = tau^2 / sigma^2`` is searched; the block-diagonal covariance is
+inverted in O(n) via Sherman-Morrison.
 """
 
 from __future__ import annotations
@@ -84,6 +69,8 @@ def fit_random_intercept(
 ) -> tuple[FloatArray, FloatArray, FloatArray, FloatArray]:
     """Fit a random-intercept LME per feature by REML.
 
+    ``lambda`` is found by a log-spaced grid search refined by a parabolic step.
+
     Parameters
     ----------
     X : ndarray of shape (n, p)
@@ -120,7 +107,6 @@ def fit_random_intercept(
     n_features = Y.shape[1]
     ridge = eps * float(np.trace(X.T @ X)) / max(p, 1)
 
-    # Vectorized grid search for the variance ratio (per feature).
     log_grid = np.linspace(-4.0, 4.0, n_grid)
     lam_grid = np.concatenate([[0.0], 10.0**log_grid])
     fvals = np.empty((lam_grid.size, n_features), dtype=np.float64)
@@ -129,7 +115,6 @@ def fit_random_intercept(
     best = np.argmin(fvals, axis=0)
     lam_opt = lam_grid[best].astype(np.float64)
 
-    # Vectorized parabolic refinement (log10 space) for interior positive optima.
     interior = (best >= 2) & (best <= n_grid - 1)
     if np.any(interior):
         bi = best[interior]
@@ -142,7 +127,6 @@ def fit_random_intercept(
         step = log_grid[1] - log_grid[0]
         lam_opt[interior] = 10.0 ** (log_grid[bi - 1] + delta * step)
 
-    # Final per-feature fixed effects, residual variance, and random-intercept BLUPs.
     beta = np.empty((p, n_features), dtype=np.float64)
     sigma2 = np.empty(n_features, dtype=np.float64)
     blup = np.zeros((n_groups, n_features), dtype=np.float64)

{combatlearn-2.3.0 → combatlearn-2.4.0}/combatlearn/_nested.py

@@ -1,21 +1,7 @@
 """Orchestration helpers for NestedComBat (*Horng et al.* 2022).
 
-Nested ComBat harmonizes several batch variables (e.g. site, scanner, protocol)
-by applying single-batch ComBat to each one in sequence, every step delegating
-to a :class:`~combatlearn.core.ComBatModel`. This module holds the pieces that
-surround that sequence and add no new empirical-Bayes math:
-
-* :func:`fit_nested_sequence` - fit one ``ComBatModel`` per batch variable in a
-  given order, chaining the transforms.
-* :func:`anderson_darling_batch_count` - the residual-batch-effect objective the
-  OPNested order search minimizes (Anderson-Darling k-sample feature count).
-* :func:`search_order` - exhaustive order search over all ``k!`` orderings, with
-  a factorial guardrail that falls back to greedy forward selection.
-* :class:`GMMGrouping` - the per-feature Gaussian-mixture grouping fed back into
-  the sequence as an extra batch variable (``+GMM``) or covariate (``-GMM``).
-
-Everything here is learned on the training data and frozen for transform, so the
-resulting estimator stays inductive / cross-validation-safe.
+The sequential single-batch ComBat fits, the Anderson-Darling order-search
+objective, and the GMM grouping; all are learned at fit and frozen for transform.
 """
 
 from __future__ import annotations
@@ -61,12 +47,9 @@ def anderson_darling_batch_count(
 ) -> int:
     """Number of features with a significant residual batch effect.
 
-    For every batch variable and every feature the Anderson-Darling k-sample test
-    (:func:`scipy.stats.anderson_ksamp`) compares the feature distribution across
-    that variable's levels; a feature is counted once per batch variable whose
-    test is significant at ``alpha``. The counts are summed over all batch
-    variables, so a lower value means less residual batch structure. This is the
-    objective minimized by the OPNested order search.
+    Counts (feature, batch variable) pairs whose Anderson-Darling k-sample test
+    (:func:`scipy.stats.anderson_ksamp`) is significant at ``alpha``; degenerate
+    features do not count. This is the objective the OPNested order search minimizes.
 
     Parameters
     ----------
@@ -86,8 +69,7 @@ def anderson_darling_batch_count(
     n_features = col_values.shape[1]
     total = 0
     with warnings.catch_warnings():
-        # anderson_ksamp warns when the p-value is capped to [0.001, 0.25]; that
-        # capping never changes the alpha=0.05 decision, so the warning is considered noise.
+        # Warns only when p is capped to [0.001, 0.25]; harmless at alpha=0.05.
         warnings.simplefilter("ignore")
         for batch in batch_vars.values():
             labels = np.asarray(batch)
@@ -101,8 +83,6 @@ def anderson_darling_batch_count(
                 try:
                     result = anderson_ksamp(samples)
                 except ValueError:
-                    # Degenerate feature (e.g. constant within the pooled sample):
-                    # no detectable batch effect, so it does not count.
                     continue
                 if result.significance_level < alpha:
                     total += 1
@@ -119,11 +99,8 @@ def fit_nested_sequence(
 ) -> tuple[list[ModelStep], pd.DataFrame]:
     """Fit one ``ComBatModel`` per batch variable in ``order``, chaining transforms.
 
-    The covariates ``disc`` / ``cont`` are preserved across every step, so nesting
-    composes with the covariate-aware, CovBat, and GAM engines.
-
-    Returns the list of ``(batch_variable, fitted_model)`` steps and the fully
-    harmonized training data.
+    Covariates are protected at every step. Returns the ``(batch_variable, model)``
+    steps and the fully harmonized training data.
     """
     models: list[ModelStep] = []
     X_current = X
@@ -187,8 +164,7 @@ def _search_greedy(
     make_model: Callable[[str], ComBatModel],
     alpha: float,
 ) -> tuple[list[str], list[ModelStep]]:
-    """Greedy forward selection: at each position pick the step that most reduces
-    the AD feature count, harmonize with it, then continue with the rest."""
+    """Greedy forward selection: repeatedly apply the step that most lowers the AD count."""
     remaining = list(names)
     order: list[str] = []
     models: list[ModelStep] = []
@@ -232,13 +208,9 @@ def search_order(
 ) -> tuple[list[str], list[ModelStep], bool]:
     """Select a harmonization order that minimizes the residual batch effect.
 
-    All ``k!`` orderings are tried exhaustively when ``k <= max_exhaustive_vars``;
-    above the cap the factorial cost is prohibitive, so a warning is issued and
-    the search falls back to greedy forward selection. A large exhaustive search
-    (``k >= 4``) also warns up front with the number of ComBat fits it will run.
-
-    Returns the chosen order, the fitted sequence for that order, and whether the
-    greedy fallback was used.
+    Exhaustive over all ``k!`` orderings when ``k <= max_exhaustive_vars`` (warning
+    from ``k >= 4``), else greedy with a warning. Returns the order, its fitted
+    sequence, and whether the greedy fallback was used.
     """
     k = len(names)
     if k <= max_exhaustive_vars:
@@ -270,16 +242,9 @@ def search_order(
 class GMMGrouping:
     """A per-feature Gaussian-mixture grouping (*Horng et al.* 2022, GMM ComBat).
 
-    A 2-component Gaussian mixture is fit to each feature; features whose split
-    leaves either cluster below ``min_cluster_frac`` of the samples are discarded
-    (an unbalanced split rarely reflects a real latent group), and the balanced
-    feature with the lowest AIC is retained. Its two-component assignment is the
-    latent grouping fed back into NestedComBat as an extra batch variable
-    (``+GMM``) or a protected covariate (``-GMM``).
-
-    The mixture is fit on the training data and stored, so held-out samples are
-    assigned by :meth:`assign` (``predict`` on the same fitted mixture), keeping
-    the grouping leakage-free.
+    A 2-component mixture is fit to each feature; among those whose smaller cluster
+    exceeds ``min_cluster_frac`` of the samples, the lowest-AIC one gives the grouping,
+    used as a batch variable (``+GMM``) or covariate (``-GMM``).
     """
 
     def __init__(self, feature: object, gmm: GaussianMixture) -> None:

combatlearn-2.4.0/combatlearn/_utils.py

@@ -0,0 +1,122 @@
+"""Internal utilities shared across combatlearn modules."""
+
+from __future__ import annotations
+
+import warnings
+from collections.abc import Iterable
+from typing import TYPE_CHECKING
+
+import numpy as np
+import pandas as pd
+
+if TYPE_CHECKING:
+    from .core import ArrayLike
+
+
+def _fit_levels(values: pd.Series) -> pd.Index:
+    """Levels a discrete covariate takes in the fitted rows (unused categories dropped)."""
+    levels: pd.Index = pd.Index(pd.Categorical(values).remove_unused_categories().categories)
+    return levels
+
+
+def _code_levels(values: pd.Series, levels: pd.Index, label: str) -> pd.Categorical:
+    """Code values against the levels frozen at fit; a level not seen at fit raises.
+
+    Dummy coding would otherwise turn it into an all-zero row, silently the reference level.
+    """
+    codes = levels.get_indexer(pd.Index(values))
+    unseen = pd.unique(values[(codes == -1) & values.notna().to_numpy()])
+    if len(unseen):
+        raise ValueError(
+            f"{label} has level(s) {sorted(map(str, unseen))} that were "
+            f"not seen at fit. A level with no training samples has no estimated covariate "
+            f"effect, so it cannot be corrected; coding it as the reference level would "
+            f"silently apply the wrong one. Seen at fit: {sorted(map(str, levels))}."
+        )
+    return pd.Categorical.from_codes(codes.tolist(), categories=levels)
+
+
+def _check_positional_alignment(
+    X: ArrayLike, auxiliaries: Iterable[tuple[str, ArrayLike | None]]
+) -> None:
+    """Require length-matched auxiliaries when ``X`` carries no index labels.
+
+    A plain-array ``X`` can only align positionally, which breaks inside CV when the
+    splitter subsets ``X`` but construction-time ``batch``/covariates stay full-length.
+    """
+    if isinstance(X, pd.DataFrame):
+        return
+    n = len(X)
+    for name, obj in auxiliaries:
+        if obj is None:
+            continue
+        if len(obj) != n:
+            raise ValueError(
+                f"`{name}` has length {len(obj)} but X has {n} rows. When X is not a "
+                f"pandas DataFrame it has no index labels, so combatlearn can only align "
+                f"`{name}` to X positionally, which requires equal lengths. This typically "
+                f"happens inside cross-validation, where the splitter subsets X but the "
+                f"batch/covariates passed at construction stay full-length. Pass X and "
+                f"`{name}` as pandas objects sharing the same index so folds stay "
+                f"label-aligned, or pass a same-length `{name}`."
+            )
+
+
+def _subset(obj: ArrayLike | None, idx: pd.Index) -> pd.DataFrame | pd.Series | None:
+    """Subset array-like object by index."""
+    if obj is None:
+        return None
+    if isinstance(obj, pd.Series | pd.DataFrame):
+        return obj.loc[idx]
+    else:
+        if isinstance(obj, np.ndarray) and obj.ndim == 1:
+            return pd.Series(obj, index=idx)
+        else:
+            return pd.DataFrame(obj, index=idx)
+
+
+def _warn_if_covariate_determines_target(
+    y: ArrayLike | None,
+    discrete_covariates: pd.DataFrame | pd.Series | None,
+    *,
+    estimator: str,
+    stacklevel: int = 4,
+) -> None:
+    """Warn when a discrete covariate perfectly determines ``y`` (likely the target).
+
+    Protecting the target inflates cross-validated scores. ``discrete_covariates``
+    must already be subset to the fitted rows; covariates averaging fewer than two
+    samples per level are skipped to avoid false alarms.
+    """
+    if y is None or discrete_covariates is None:
+        return
+
+    target = pd.Series(np.asarray(y).ravel())
+    if len(target) != len(discrete_covariates) or target.nunique(dropna=False) < 2:
+        return
+
+    frame = (
+        discrete_covariates.to_frame()
+        if isinstance(discrete_covariates, pd.Series)
+        else discrete_covariates
+    )
+    n_samples = len(frame)
+    for column in frame.columns:
+        levels = pd.Series(np.asarray(frame[column]))
+        n_levels = levels.nunique(dropna=False)
+        if n_levels < 2 or n_levels * 2 > n_samples:
+            continue
+        if target.groupby(list(levels), dropna=False).nunique(dropna=False).max() > 1:
+            continue
+        warnings.warn(
+            f"{estimator}: discrete covariate {column!r} determines `y` on the training "
+            f"data, so this correction preserves the very signal you are about to predict, "
+            f"and `transform` applies it using each held-out sample's own label - which "
+            f"would not be available at prediction time. Any cross-validated score will be "
+            f"optimistic. If you are harmonizing a whole cohort for differential expression "
+            f"or methylation, use combatlearn.transductive.TransductiveComBat(condition=...), "
+            f"which protects the term and refuses `transform`. If {column!r} is genuinely "
+            f"known at intake and is not your target, this warning does not apply.",
+            UserWarning,
+            stacklevel=stacklevel,
+        )