pystatistics 2.1.0__tar.gz → 2.3.0__tar.gz

{pystatistics-2.1.0 → pystatistics-2.3.0}/CHANGELOG.md

@@ -1,5 +1,145 @@
 # Changelog
 
+## 2.3.0
+
+- **New subpackage `pystatistics.nonparametric_mcar`** for distribution-
+  free MCAR tests, motivated by the Lacuna ablation finding that the
+  MVN-based Little's MCAR feature (MLE or MoM plug-in) does not help
+  mechanism classification on heavy-tailed / categorical tabular data.
+
+  - Added `propensity_mcar_test(data, *, model='rf'|'gbm', cv_folds=5,
+    n_permutations=199, seed=0, alpha=0.05)`. Fits a sklearn
+    `RandomForestClassifier` or `GradientBoostingClassifier` to predict
+    each column's missingness indicator from the other columns
+    (mean-imputed + per-column missing-indicator features), computes
+    out-of-fold AUC, and calibrates against a permutation null. Returns
+    a `NonparametricMCARResult` with `statistic = mean_auc - 0.5` and
+    permutation-smoothed `p_value`. scikit-learn is an optional extra:
+    `pip install pystatistics[nonparametric_mcar]`.
+
+  - Added `NonparametricMCARResult` dataclass (statistic, p_value,
+    rejected, alpha, method, n_observations, n_variables,
+    n_missing_cells, extra). Intentionally narrower than the MVN-based
+    `MCARTestResult` — no df / ml_mean / ml_cov / patterns, because
+    nonparametric tests don't produce those.
+
+  - Tests: 11 in `tests/nonparametric_mcar/test_propensity.py` covering
+    normal cases (MCAR non-rejection, MAR rejection, reproducibility
+    under fixed seed, GBM option), edge cases (fully-observed column
+    ignored), and failure cases (1D input, too few rows/columns, no
+    missingness, invalid hyperparameters).
+
+  - Added `hsic_mcar_test(data, *, alpha=0.05, n_permutations=199,
+    seed=0)`. Hilbert-Schmidt Independence Criterion (Gretton et al.
+    2005/2008) between stochastically-imputed observed values and the
+    missingness-indicator matrix, with Gaussian RBF kernel and
+    median-heuristic bandwidth. Biased HSIC estimator, permutation null
+    for calibration. Uses **stochastic** (column-mean + column-std
+    noise) imputation rather than plain mean imputation — pure
+    mean-imputation pulls heavy-missing rows toward the column
+    centroid, which creates a systematic X-R coupling and rejects MCAR
+    spuriously on MCAR-generated data. Pure numpy; no sklearn dep.
+
+  - Added `missmech_mcar_test(data, *, alpha=0.05, n_permutations=199,
+    n_neighbors=5, min_pattern_size=6, seed=0)`. Jamshidian-Jalal-style
+    test of homogeneity of means across missingness-pattern groups,
+    after k-NN imputation (via `sklearn.impute.KNNImputer`). Statistic
+    is the between-pattern weighted sum of squared mean differences
+    (Σ_p n_p ||μ_p − μ||²), calibrated against a pattern-label
+    permutation null — equivalent in hypothesis to Jamshidian & Jalal
+    (2010)'s bootstrap, but faster for the cached-scalar use case.
+    Requires the same `nonparametric_mcar` extra for sklearn.
+
+  - Tests: 9 in `test_hsic.py`, 10 in `test_missmech.py` — both
+    covering MCAR non-rejection, MAR rejection, reproducibility under
+    seed, and the same failure-case matrix as propensity.
+
+  - Suite: 156/156 across `tests/nonparametric_mcar/` and `tests/mvnmle/`
+    pass; no mvnmle regressions from the new subpackage.
+
+- **Split `pystatistics/mvnmle/backends/_em_batched.py`** (501 SLOC →
+  over the Rule 4 hard limit of 500) into three focused files plus a
+  compatibility shim:
+    - `_em_batched_patterns.py` (63 SLOC) — `_BatchedPatternIndex`
+      dataclass, `_pattern_n`, `build_pattern_index`.
+    - `_em_batched_np.py` (203 SLOC) — NumPy CPU backend
+      (`compute_conditional_parameters_np`, `e_step_full_batched_np`,
+      `compute_loglik_batched_np`, `chi_square_mcar_batched_np`).
+    - `_em_batched_torch.py` (239 SLOC) — Torch GPU backend
+      (`_e_step_full_torch`, `_loglik_full_torch`,
+      `chi_square_mcar_batched_torch`, `compute_conditional_parameters_torch`).
+    - `_em_batched.py` (30 SLOC) — thin shim re-exporting every
+      symbol so existing importers in `em.py`, `solvers.py`, and
+      `mcar_test.py` need no changes.
+  The 157-test suite across `tests/test_code_quality.py`,
+  `tests/mvnmle/`, and `tests/nonparametric_mcar/` passes after the
+  split; `test_no_file_exceeds_500_code_lines` now passes.
+
+
+## 2.2.0
+
+- Fixed a `torch._C._LinAlgError` crash in `chi_square_mcar_batched_torch`
+  (`pystatistics/mvnmle/backends/_em_batched.py`) on GPU FP32. The batched
+  MoM fast path selected `cholesky_solve` when the SVD-based condition
+  number was below threshold, but Cholesky requires positive-definiteness,
+  which is strictly stronger than good conditioning. On GPU FP32, real
+  tabular data (e.g. `lacuna_tabular_110` applied to UCI/OpenML datasets)
+  can produce `sigma_oo` with good cond number but tiny negative eigenvalues
+  from roundoff, making Cholesky fail. Fix: wrap the fast path in
+  `try/except torch._C._LinAlgError`; on failure, fall back to
+  `torch.linalg.pinv` for the batch (honouring the `regularize` flag —
+  `regularize=False` still raises). Surfaced by dogfooding via Project
+  Lacuna on 3,080 (dataset × generator) pairs across the 110-generator
+  tabular registry; previously `mom_mcar_test` crashed on the first batch
+  containing breast_cancer / wine / credit_card_default.
+
+- Fixed an exception-type leak in `little_mcar_test`
+  (`pystatistics/mvnmle/mcar_test.py:~250`). The ML-estimation try/except
+  wrapped *every* exception — including `PyStatisticsError` subclasses
+  like `NumericalError` — as a bare `RuntimeError`, breaking the
+  documented `except PyStatisticsError:` pattern downstream and losing
+  the original exception chain. Fix: explicitly re-raise
+  `PyStatisticsError`, and use `raise ... from e` for anything else so
+  the chain is preserved. Surfaced by Project Lacuna's cache builder,
+  which catches `PyStatisticsError` to fall back to a sentinel entry
+  when Little's test is numerically unfit for a particular
+  (dataset, generator) pair; MLE failures were leaking past the catch
+  and killing the whole build.
+
+- Added ridge-fallback to the batched Cholesky sites inside the EM
+  E-step / log-likelihood
+  (`pystatistics/mvnmle/backends/_em_batched.py`):
+  `e_step_full_batched_np` (line ~361), `_e_step_full_torch` (~680), and
+  `_loglik_full_batched_torch` (~797). These compute per-pattern
+  Cholesky of sigma_oo sub-blocks; real tabular data can produce
+  individual sub-blocks that are numerically indefinite even when the
+  global sigma is PD (integer-encoded categoricals with heavy
+  collinearity in the intersection of a given missingness pattern's
+  observed variables). Fix: wrap each site in `try/except LinAlgError`
+  with a `ridge·I` retry (ridge = 1e-10 at pattern scale; statistically
+  invisible). Also removed a dead Cholesky call in `e_step_batched_np`
+  whose result was never used — it was only a crash liability.
+  `np.linalg.solve` at that same site now has a pinv fallback for
+  singular sub-blocks.
+
+- Added `regularize: bool = True` to `mlest`, `_solve_em`, and
+  `EMBackend.solve`, mirroring the existing convention on
+  `mom_mcar_test` / `little_mcar_test`. When True (new default),
+  `EMBackend._ensure_pd` applies a small diagonal ridge
+  (`max(0, 1e-10 - min_eig) + 1e-12`) to the M-step sigma whenever its
+  smallest eigenvalue falls below the PD threshold, rather than raising
+  `NumericalError` outright. The ridge is vanishingly small relative to
+  any real data scale — the typical case the old path rejected had
+  min_eig ≈ 1e-13 from pure FP64 roundoff — and a UserWarning makes the
+  event visible in logs. Call sites that need strict bit-for-bit
+  behaviour pass `regularize=False`. Motivated by Project Lacuna
+  dogfooding: applying real missingness generators to real UCI datasets
+  (credit_card_default × MNAR-NonLinSocial produced min_eig ≈ -0.66) was
+  hard-raising and killing the full cache build; the ridge fallback
+  keeps the test numerically well-defined with negligible statistical
+  impact, and the build proceeds.
+
+
 ## 2.1.0
 
 - **`mom_mcar_test`: new method-of-moments MCAR test**

{pystatistics-2.1.0 → pystatistics-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pystatistics
-Version: 2.1.0
+Version: 2.3.0
 Summary: GPU-accelerated statistical computing for Python
 Project-URL: Homepage, https://sgcx.org/technology/pystatistics/
 Project-URL: Documentation, https://sgcx.org/docs/pystatistics/
@@ -43,6 +43,8 @@ Requires-Dist: furo; extra == 'docs'
 Requires-Dist: sphinx>=6.0; extra == 'docs'
 Provides-Extra: gpu
 Requires-Dist: torch>=2.0; extra == 'gpu'
+Provides-Extra: nonparametric-mcar
+Requires-Dist: scikit-learn>=1.3; extra == 'nonparametric-mcar'
 Description-Content-Type: text/markdown
 
 # PyStatistics
@@ -51,6 +53,72 @@ GPU-accelerated statistical computing for Python.
 
 ## What's New
 
+### 2.3.0 — Nonparametric MCAR tests for heavy-tailed tabular data
+
+Little's MCAR test (1988) and the method-of-moments variant both assume
+multivariate normality of the observed values. On realistic tabular
+data — heavy-tailed, skewed, integer-encoded categoricals — MVN is
+badly violated, and Project Lacuna's 2026-04 ablation study found that
+cached Little's features add nothing to mechanism classification.
+This release ships three distribution-free alternatives in a new
+`pystatistics.nonparametric_mcar` subpackage so downstream projects
+can test which family (if any) provides useful MCAR-detection signal
+when MVN fails.
+
+**`propensity_mcar_test`** fits a random forest (or gradient boosting)
+classifier to predict each column's missingness indicator from the
+other columns (mean-imputed + per-column missing-indicator features),
+computes out-of-fold AUC, and calibrates against a permutation null.
+Native mixed-type handling, nonlinear; deterministic under a seed.
+Exposes `statistic = mean_observed_auc − 0.5` (clipped to [0, 0.5])
+and a permutation-smoothed p-value. Optional scikit-learn dependency:
+`pip install pystatistics[nonparametric_mcar]`.
+
+**`hsic_mcar_test`** is a Hilbert-Schmidt Independence Criterion test
+(Gretton et al. 2005/2008) between stochastically-imputed observed
+values and the missingness-indicator matrix, with a Gaussian RBF
+kernel and median-heuristic bandwidth. Pure numpy, no sklearn. Uses
+**stochastic** imputation rather than plain mean imputation — the
+latter pulls heavy-missing rows toward the column centroid, which
+creates a systematic X-R coupling that rejects MCAR spuriously on
+MCAR-generated data.
+
+**`missmech_mcar_test`** is a Jamshidian-Jalal-style (2010)
+nonparametric test of homogeneity of means across missingness
+patterns. Groups rows by pattern, k-NN imputes to full columns, and
+calibrates a between-pattern weighted sum-of-squared-mean-differences
+statistic against a permutation null over the pattern labels.
+Equivalent in hypothesis to Jamshidian & Jalal's bootstrap but faster
+for the cached-scalar use case.
+
+All three return a shared `NonparametricMCARResult` dataclass
+(statistic, p_value, rejected, alpha, method, n_observations,
+n_variables, n_missing_cells, extra) — intentionally narrower than
+the MVN-based `MCARTestResult` from `mvnmle`, since concepts like
+`df` / `ml_mean` / `ml_cov` / `patterns` don't apply to
+distribution-free tests. 30 new tests across propensity, HSIC, and
+MissMech; no mvnmle regressions.
+
+**Internal refactor.** Split
+`pystatistics/mvnmle/backends/_em_batched.py` (501 SLOC, over the
+Rule 4 hard limit) into three focused modules — patterns,
+NumPy backend, Torch backend — plus a compatibility shim that
+re-exports every symbol so no importer needs to change. Code-quality
+suite now reports zero files over 500 SLOC.
+
+### 2.2.0 — Real-data robustness from Project Lacuna dogfooding
+
+Four classes of numerical failure on realistic tabular data — Cholesky
+fast-path crash on GPU FP32 roundoff, bare-`RuntimeError` wrapping
+breaking `PyStatisticsError` catch patterns, M-step sigma PD-check
+false negatives from FP64 roundoff, and per-pattern Cholesky on
+indefinite sub-blocks — all fixed in this release with a unified
+`regularize=True` opt-out-to-strict convention across `mlest`,
+`mom_mcar_test`, `little_mcar_test`, and the batched E-step. The
+Project Lacuna cache build on 3,080 (dataset × generator) pairs went
+from crashing on the first batch to completing in a single pass at
+0.9% MoM / 16.4% MLE sentinel rates. No API breaks.
+
 ### 2.1.0 — Real-data EM speedup + monotone closed-form MLE
 
 Dogfooding via Project Lacuna surfaced that ``little_mcar_test`` on

{pystatistics-2.1.0 → pystatistics-2.3.0}/README.md

@@ -4,6 +4,72 @@ GPU-accelerated statistical computing for Python.
 
 ## What's New
 
+### 2.3.0 — Nonparametric MCAR tests for heavy-tailed tabular data
+
+Little's MCAR test (1988) and the method-of-moments variant both assume
+multivariate normality of the observed values. On realistic tabular
+data — heavy-tailed, skewed, integer-encoded categoricals — MVN is
+badly violated, and Project Lacuna's 2026-04 ablation study found that
+cached Little's features add nothing to mechanism classification.
+This release ships three distribution-free alternatives in a new
+`pystatistics.nonparametric_mcar` subpackage so downstream projects
+can test which family (if any) provides useful MCAR-detection signal
+when MVN fails.
+
+**`propensity_mcar_test`** fits a random forest (or gradient boosting)
+classifier to predict each column's missingness indicator from the
+other columns (mean-imputed + per-column missing-indicator features),
+computes out-of-fold AUC, and calibrates against a permutation null.
+Native mixed-type handling, nonlinear; deterministic under a seed.
+Exposes `statistic = mean_observed_auc − 0.5` (clipped to [0, 0.5])
+and a permutation-smoothed p-value. Optional scikit-learn dependency:
+`pip install pystatistics[nonparametric_mcar]`.
+
+**`hsic_mcar_test`** is a Hilbert-Schmidt Independence Criterion test
+(Gretton et al. 2005/2008) between stochastically-imputed observed
+values and the missingness-indicator matrix, with a Gaussian RBF
+kernel and median-heuristic bandwidth. Pure numpy, no sklearn. Uses
+**stochastic** imputation rather than plain mean imputation — the
+latter pulls heavy-missing rows toward the column centroid, which
+creates a systematic X-R coupling that rejects MCAR spuriously on
+MCAR-generated data.
+
+**`missmech_mcar_test`** is a Jamshidian-Jalal-style (2010)
+nonparametric test of homogeneity of means across missingness
+patterns. Groups rows by pattern, k-NN imputes to full columns, and
+calibrates a between-pattern weighted sum-of-squared-mean-differences
+statistic against a permutation null over the pattern labels.
+Equivalent in hypothesis to Jamshidian & Jalal's bootstrap but faster
+for the cached-scalar use case.
+
+All three return a shared `NonparametricMCARResult` dataclass
+(statistic, p_value, rejected, alpha, method, n_observations,
+n_variables, n_missing_cells, extra) — intentionally narrower than
+the MVN-based `MCARTestResult` from `mvnmle`, since concepts like
+`df` / `ml_mean` / `ml_cov` / `patterns` don't apply to
+distribution-free tests. 30 new tests across propensity, HSIC, and
+MissMech; no mvnmle regressions.
+
+**Internal refactor.** Split
+`pystatistics/mvnmle/backends/_em_batched.py` (501 SLOC, over the
+Rule 4 hard limit) into three focused modules — patterns,
+NumPy backend, Torch backend — plus a compatibility shim that
+re-exports every symbol so no importer needs to change. Code-quality
+suite now reports zero files over 500 SLOC.
+
+### 2.2.0 — Real-data robustness from Project Lacuna dogfooding
+
+Four classes of numerical failure on realistic tabular data — Cholesky
+fast-path crash on GPU FP32 roundoff, bare-`RuntimeError` wrapping
+breaking `PyStatisticsError` catch patterns, M-step sigma PD-check
+false negatives from FP64 roundoff, and per-pattern Cholesky on
+indefinite sub-blocks — all fixed in this release with a unified
+`regularize=True` opt-out-to-strict convention across `mlest`,
+`mom_mcar_test`, `little_mcar_test`, and the batched E-step. The
+Project Lacuna cache build on 3,080 (dataset × generator) pairs went
+from crashing on the first batch to completing in a single pass at
+0.9% MoM / 16.4% MLE sentinel rates. No API breaks.
+
 ### 2.1.0 — Real-data EM speedup + monotone closed-form MLE
 
 Dogfooding via Project Lacuna surfaced that ``little_mcar_test`` on

{pystatistics-2.1.0 → pystatistics-2.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pystatistics"
-version = "2.1.0"
+version = "2.3.0"
 description = "GPU-accelerated statistical computing for Python"
 readme = "README.md"
 license = "MIT"
@@ -48,6 +48,12 @@ dependencies = [
 gpu = [
     "torch>=2.0",
 ]
+nonparametric_mcar = [
+    # scikit-learn for propensity_mcar_test (RandomForestClassifier /
+    # GradientBoostingClassifier). HSIC and MissMech in the same subpackage
+    # are pure numpy/scipy and do not need sklearn.
+    "scikit-learn>=1.3",
+]
 dev = [
     "pytest>=7.0",
     "pytest-cov>=4.0",

{pystatistics-2.1.0 → pystatistics-2.3.0}/pystatistics/__init__.py

@@ -16,7 +16,7 @@ Usage:
     result = fit(design)
 """
 
-__version__ = "2.1.0"
+__version__ = "2.3.0"
 __author__ = "Hai-Shuo"
 __email__ = "contact@sgcx.org"
 

pystatistics-2.3.0/pystatistics/mvnmle/backends/_em_batched.py

@@ -0,0 +1,44 @@
+"""Compatibility shim for the batched EM/MCAR building blocks.
+
+The implementation was split into three modules on 2026-04-20 to stay
+under the 500-SLOC file limit (Coding Bible rule 4):
+
+  * ``_em_batched_patterns`` — pattern-index dataclass + ``build_pattern_index``
+  * ``_em_batched_np``       — NumPy backend (CPU)
+  * ``_em_batched_torch``    — Torch backend (GPU + CPU torch)
+
+This module continues to expose every name that callers imported
+pre-split, so no call-site changes were required. New code may import
+from the specific backend module directly.
+"""
+from pystatistics.mvnmle.backends._em_batched_patterns import (
+    _BatchedPatternIndex,
+    _pattern_n,
+    build_pattern_index,
+)
+from pystatistics.mvnmle.backends._em_batched_np import (
+    chi_square_mcar_batched_np,
+    compute_conditional_parameters_np,
+    compute_loglik_batched_np,
+    e_step_full_batched_np,
+)
+from pystatistics.mvnmle.backends._em_batched_torch import (
+    _e_step_full_torch,
+    _loglik_full_torch,
+    chi_square_mcar_batched_torch,
+    compute_conditional_parameters_torch,
+)
+
+__all__ = [
+    "_BatchedPatternIndex",
+    "_pattern_n",
+    "build_pattern_index",
+    "chi_square_mcar_batched_np",
+    "compute_conditional_parameters_np",
+    "compute_loglik_batched_np",
+    "e_step_full_batched_np",
+    "_e_step_full_torch",
+    "_loglik_full_torch",
+    "chi_square_mcar_batched_torch",
+    "compute_conditional_parameters_torch",
+]