pystatistics 2.2.0__tar.gz → 2.3.0__tar.gz

{pystatistics-2.2.0 → pystatistics-2.3.0}/CHANGELOG.md

@@ -1,5 +1,81 @@
 # Changelog
 
+## 2.3.0
+
+- **New subpackage `pystatistics.nonparametric_mcar`** for distribution-
+  free MCAR tests, motivated by the Lacuna ablation finding that the
+  MVN-based Little's MCAR feature (MLE or MoM plug-in) does not help
+  mechanism classification on heavy-tailed / categorical tabular data.
+
+  - Added `propensity_mcar_test(data, *, model='rf'|'gbm', cv_folds=5,
+    n_permutations=199, seed=0, alpha=0.05)`. Fits a sklearn
+    `RandomForestClassifier` or `GradientBoostingClassifier` to predict
+    each column's missingness indicator from the other columns
+    (mean-imputed + per-column missing-indicator features), computes
+    out-of-fold AUC, and calibrates against a permutation null. Returns
+    a `NonparametricMCARResult` with `statistic = mean_auc - 0.5` and
+    permutation-smoothed `p_value`. scikit-learn is an optional extra:
+    `pip install pystatistics[nonparametric_mcar]`.
+
+  - Added `NonparametricMCARResult` dataclass (statistic, p_value,
+    rejected, alpha, method, n_observations, n_variables,
+    n_missing_cells, extra). Intentionally narrower than the MVN-based
+    `MCARTestResult` — no df / ml_mean / ml_cov / patterns, because
+    nonparametric tests don't produce those.
+
+  - Tests: 11 in `tests/nonparametric_mcar/test_propensity.py` covering
+    normal cases (MCAR non-rejection, MAR rejection, reproducibility
+    under fixed seed, GBM option), edge cases (fully-observed column
+    ignored), and failure cases (1D input, too few rows/columns, no
+    missingness, invalid hyperparameters).
+
+  - Added `hsic_mcar_test(data, *, alpha=0.05, n_permutations=199,
+    seed=0)`. Hilbert-Schmidt Independence Criterion (Gretton et al.
+    2005/2008) between stochastically-imputed observed values and the
+    missingness-indicator matrix, with Gaussian RBF kernel and
+    median-heuristic bandwidth. Biased HSIC estimator, permutation null
+    for calibration. Uses **stochastic** (column-mean + column-std
+    noise) imputation rather than plain mean imputation — pure
+    mean-imputation pulls heavy-missing rows toward the column
+    centroid, which creates a systematic X-R coupling and rejects MCAR
+    spuriously on MCAR-generated data. Pure numpy; no sklearn dep.
+
+  - Added `missmech_mcar_test(data, *, alpha=0.05, n_permutations=199,
+    n_neighbors=5, min_pattern_size=6, seed=0)`. Jamshidian-Jalal-style
+    test of homogeneity of means across missingness-pattern groups,
+    after k-NN imputation (via `sklearn.impute.KNNImputer`). Statistic
+    is the between-pattern weighted sum of squared mean differences
+    (Σ_p n_p ||μ_p − μ||²), calibrated against a pattern-label
+    permutation null — equivalent in hypothesis to Jamshidian & Jalal
+    (2010)'s bootstrap, but faster for the cached-scalar use case.
+    Requires the same `nonparametric_mcar` extra for sklearn.
+
+  - Tests: 9 in `test_hsic.py`, 10 in `test_missmech.py` — both
+    covering MCAR non-rejection, MAR rejection, reproducibility under
+    seed, and the same failure-case matrix as propensity.
+
+  - Suite: 156/156 across `tests/nonparametric_mcar/` and `tests/mvnmle/`
+    pass; no mvnmle regressions from the new subpackage.
+
+- **Split `pystatistics/mvnmle/backends/_em_batched.py`** (501 SLOC →
+  over the Rule 4 hard limit of 500) into three focused files plus a
+  compatibility shim:
+    - `_em_batched_patterns.py` (63 SLOC) — `_BatchedPatternIndex`
+      dataclass, `_pattern_n`, `build_pattern_index`.
+    - `_em_batched_np.py` (203 SLOC) — NumPy CPU backend
+      (`compute_conditional_parameters_np`, `e_step_full_batched_np`,
+      `compute_loglik_batched_np`, `chi_square_mcar_batched_np`).
+    - `_em_batched_torch.py` (239 SLOC) — Torch GPU backend
+      (`_e_step_full_torch`, `_loglik_full_torch`,
+      `chi_square_mcar_batched_torch`, `compute_conditional_parameters_torch`).
+    - `_em_batched.py` (30 SLOC) — thin shim re-exporting every
+      symbol so existing importers in `em.py`, `solvers.py`, and
+      `mcar_test.py` need no changes.
+  The 157-test suite across `tests/test_code_quality.py`,
+  `tests/mvnmle/`, and `tests/nonparametric_mcar/` passes after the
+  split; `test_no_file_exceeds_500_code_lines` now passes.
+
+
 ## 2.2.0
 
 - Fixed a `torch._C._LinAlgError` crash in `chi_square_mcar_batched_torch`

{pystatistics-2.2.0 → pystatistics-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pystatistics
-Version: 2.2.0
+Version: 2.3.0
 Summary: GPU-accelerated statistical computing for Python
 Project-URL: Homepage, https://sgcx.org/technology/pystatistics/
 Project-URL: Documentation, https://sgcx.org/docs/pystatistics/
@@ -43,6 +43,8 @@ Requires-Dist: furo; extra == 'docs'
 Requires-Dist: sphinx>=6.0; extra == 'docs'
 Provides-Extra: gpu
 Requires-Dist: torch>=2.0; extra == 'gpu'
+Provides-Extra: nonparametric-mcar
+Requires-Dist: scikit-learn>=1.3; extra == 'nonparametric-mcar'
 Description-Content-Type: text/markdown
 
 # PyStatistics
@@ -51,73 +53,71 @@ GPU-accelerated statistical computing for Python.
 
 ## What's New
 
+### 2.3.0 — Nonparametric MCAR tests for heavy-tailed tabular data
+
+Little's MCAR test (1988) and the method-of-moments variant both assume
+multivariate normality of the observed values. On realistic tabular
+data — heavy-tailed, skewed, integer-encoded categoricals — MVN is
+badly violated, and Project Lacuna's 2026-04 ablation study found that
+cached Little's features add nothing to mechanism classification.
+This release ships three distribution-free alternatives in a new
+`pystatistics.nonparametric_mcar` subpackage so downstream projects
+can test which family (if any) provides useful MCAR-detection signal
+when MVN fails.
+
+**`propensity_mcar_test`** fits a random forest (or gradient boosting)
+classifier to predict each column's missingness indicator from the
+other columns (mean-imputed + per-column missing-indicator features),
+computes out-of-fold AUC, and calibrates against a permutation null.
+Native mixed-type handling, nonlinear; deterministic under a seed.
+Exposes `statistic = mean_observed_auc − 0.5` (clipped to [0, 0.5])
+and a permutation-smoothed p-value. Optional scikit-learn dependency:
+`pip install pystatistics[nonparametric_mcar]`.
+
+**`hsic_mcar_test`** is a Hilbert-Schmidt Independence Criterion test
+(Gretton et al. 2005/2008) between stochastically-imputed observed
+values and the missingness-indicator matrix, with a Gaussian RBF
+kernel and median-heuristic bandwidth. Pure numpy, no sklearn. Uses
+**stochastic** imputation rather than plain mean imputation — the
+latter pulls heavy-missing rows toward the column centroid, which
+creates a systematic X-R coupling that rejects MCAR spuriously on
+MCAR-generated data.
+
+**`missmech_mcar_test`** is a Jamshidian-Jalal-style (2010)
+nonparametric test of homogeneity of means across missingness
+patterns. Groups rows by pattern, k-NN imputes to full columns, and
+calibrates a between-pattern weighted sum-of-squared-mean-differences
+statistic against a permutation null over the pattern labels.
+Equivalent in hypothesis to Jamshidian & Jalal's bootstrap but faster
+for the cached-scalar use case.
+
+All three return a shared `NonparametricMCARResult` dataclass
+(statistic, p_value, rejected, alpha, method, n_observations,
+n_variables, n_missing_cells, extra) — intentionally narrower than
+the MVN-based `MCARTestResult` from `mvnmle`, since concepts like
+`df` / `ml_mean` / `ml_cov` / `patterns` don't apply to
+distribution-free tests. 30 new tests across propensity, HSIC, and
+MissMech; no mvnmle regressions.
+
+**Internal refactor.** Split
+`pystatistics/mvnmle/backends/_em_batched.py` (501 SLOC, over the
+Rule 4 hard limit) into three focused modules — patterns,
+NumPy backend, Torch backend — plus a compatibility shim that
+re-exports every symbol so no importer needs to change. Code-quality
+suite now reports zero files over 500 SLOC.
+
 ### 2.2.0 — Real-data robustness from Project Lacuna dogfooding
 
-Continuation of the 2.1.0 dogfooding track. Running `little_mcar_test`
-and `mom_mcar_test` on 3,080 (dataset × generator) pairs drawn from 28
-real UCI / OpenML / sklearn tabular datasets under
-`lacuna_tabular_110` missingness generators surfaced four classes of
-numerical failure that synthetic unit tests did not exhibit. All fixed
-in this release; no API breaks.
-
-**Batched MoM GPU Cholesky crash.** `chi_square_mcar_batched_torch`'s
-fast path selected `cholesky_solve` whenever the SVD-based condition
-number check passed, on the assumption that good conditioning implies
-positive-definiteness. On GPU FP32, roundoff can produce covariances
-that pass the cond-number check but have tiny negative eigenvalues —
-Cholesky fails, the call raises `torch._C._LinAlgError`. The fast path
-is now wrapped with `try/except` and falls back to pseudo-inverse when
-the ``regularize`` flag allows. Surfaced on `credit_card_default` ×
-`MNAR-NonLinSocial` during the Lacuna cache build.
-
-**Exception-type preservation in `little_mcar_test`.** The
-ML-estimation `try/except` at the top of `little_mcar_test` wrapped
-*every* exception as a bare `RuntimeError` — including
-`PyStatisticsError` subclasses. This broke the documented
-`except PyStatisticsError:` catch pattern downstream: users falling
-back to a sentinel on MLE failure saw their handler bypassed and the
-full build crash. Fix: explicitly re-raise `PyStatisticsError`, and
-use `raise ... from e` for anything else so the chain is preserved.
-
-**`regularize=True` default on the EM path (opt-out).** `mlest`,
-`_solve_em`, and `EMBackend.solve` gain `regularize: bool = True`,
-mirroring the existing convention on `mom_mcar_test` and
-`little_mcar_test`. When True, `EMBackend._ensure_pd` applies a small
-diagonal ridge — `max(0, 1e-10 − min_eig) + 1e-12` — to the M-step
-sigma whenever its smallest eigenvalue falls below the PD threshold,
-rather than raising `NumericalError`. The ridge is well below any
-statistical precision on real data — the typical case the old path
-rejected had `min_eig ≈ 1e-13` from pure FP64 roundoff on a matrix
-that's theoretically PSD. Dogfooding surfaced cases where `min_eig`
-hit `−0.66` on realistic MNAR mechanisms; the ridge fallback keeps EM
-progressing. Callers needing strict bit-for-bit behaviour pass
-`regularize=False` to restore the old raise.
-
-**Three additional Cholesky ridge-fallbacks** in `_em_batched.py`:
-`e_step_full_batched_np`, `_e_step_full_torch`, and
-`_loglik_full_batched_torch` all compute per-pattern Cholesky of
-`sigma_oo` sub-blocks. Real tabular data can produce individual
-sub-blocks that are numerically indefinite even when the global sigma
-is PD (integer-encoded categoricals with heavy collinearity in the
-intersection of a given missingness pattern's observed variables).
-Each site now wraps Cholesky in `try/except LinAlgError` with a
-`ridge·I` retry (ridge = 1e-10 at pattern scale; statistically
-invisible). Also removed a dead Cholesky call in `e_step_batched_np`
-whose factor was never used downstream — pure crash liability — and
-added a `pinv` fallback to the `np.linalg.solve` at the same site for
-singular sub-blocks.
-
-**Impact.** The Project Lacuna cache build on 3,080 (dataset ×
-generator) pairs went from crashing on the first batch containing
-`breast_cancer` or `credit_card_default` (pre-2.2.0) to completing in
-a single pass at 0.9% MoM sentinel rate and 16.4% MLE sentinel rate
-(the MLE sentinels are legitimate EM non-convergence on 1000-pattern
-datasets — not crashes). Synthetic unit tests: 125/125 mvnmle pass.
-
-**No API breaks.** New defaults (`regularize=True`) are strictly more
-permissive than the old raises — any caller that was crashing before
-will now proceed with a small `UserWarning`. Callers needing strict
-behaviour pass `regularize=False`.
+Four classes of numerical failure on realistic tabular data — Cholesky
+fast-path crash on GPU FP32 roundoff, bare-`RuntimeError` wrapping
+breaking `PyStatisticsError` catch patterns, M-step sigma PD-check
+false negatives from FP64 roundoff, and per-pattern Cholesky on
+indefinite sub-blocks — all fixed in this release with a unified
+`regularize=True` opt-out-to-strict convention across `mlest`,
+`mom_mcar_test`, `little_mcar_test`, and the batched E-step. The
+Project Lacuna cache build on 3,080 (dataset × generator) pairs went
+from crashing on the first batch to completing in a single pass at
+0.9% MoM / 16.4% MLE sentinel rates. No API breaks.
 
 ### 2.1.0 — Real-data EM speedup + monotone closed-form MLE
 

{pystatistics-2.2.0 → pystatistics-2.3.0}/README.md

@@ -4,73 +4,71 @@ GPU-accelerated statistical computing for Python.
 
 ## What's New
 
+### 2.3.0 — Nonparametric MCAR tests for heavy-tailed tabular data
+
+Little's MCAR test (1988) and the method-of-moments variant both assume
+multivariate normality of the observed values. On realistic tabular
+data — heavy-tailed, skewed, integer-encoded categoricals — MVN is
+badly violated, and Project Lacuna's 2026-04 ablation study found that
+cached Little's features add nothing to mechanism classification.
+This release ships three distribution-free alternatives in a new
+`pystatistics.nonparametric_mcar` subpackage so downstream projects
+can test which family (if any) provides useful MCAR-detection signal
+when MVN fails.
+
+**`propensity_mcar_test`** fits a random forest (or gradient boosting)
+classifier to predict each column's missingness indicator from the
+other columns (mean-imputed + per-column missing-indicator features),
+computes out-of-fold AUC, and calibrates against a permutation null.
+Native mixed-type handling, nonlinear; deterministic under a seed.
+Exposes `statistic = mean_observed_auc − 0.5` (clipped to [0, 0.5])
+and a permutation-smoothed p-value. Optional scikit-learn dependency:
+`pip install pystatistics[nonparametric_mcar]`.
+
+**`hsic_mcar_test`** is a Hilbert-Schmidt Independence Criterion test
+(Gretton et al. 2005/2008) between stochastically-imputed observed
+values and the missingness-indicator matrix, with a Gaussian RBF
+kernel and median-heuristic bandwidth. Pure numpy, no sklearn. Uses
+**stochastic** imputation rather than plain mean imputation — the
+latter pulls heavy-missing rows toward the column centroid, which
+creates a systematic X-R coupling that rejects MCAR spuriously on
+MCAR-generated data.
+
+**`missmech_mcar_test`** is a Jamshidian-Jalal-style (2010)
+nonparametric test of homogeneity of means across missingness
+patterns. Groups rows by pattern, k-NN imputes to full columns, and
+calibrates a between-pattern weighted sum-of-squared-mean-differences
+statistic against a permutation null over the pattern labels.
+Equivalent in hypothesis to Jamshidian & Jalal's bootstrap but faster
+for the cached-scalar use case.
+
+All three return a shared `NonparametricMCARResult` dataclass
+(statistic, p_value, rejected, alpha, method, n_observations,
+n_variables, n_missing_cells, extra) — intentionally narrower than
+the MVN-based `MCARTestResult` from `mvnmle`, since concepts like
+`df` / `ml_mean` / `ml_cov` / `patterns` don't apply to
+distribution-free tests. 30 new tests across propensity, HSIC, and
+MissMech; no mvnmle regressions.
+
+**Internal refactor.** Split
+`pystatistics/mvnmle/backends/_em_batched.py` (501 SLOC, over the
+Rule 4 hard limit) into three focused modules — patterns,
+NumPy backend, Torch backend — plus a compatibility shim that
+re-exports every symbol so no importer needs to change. Code-quality
+suite now reports zero files over 500 SLOC.
+
 ### 2.2.0 — Real-data robustness from Project Lacuna dogfooding
 
-Continuation of the 2.1.0 dogfooding track. Running `little_mcar_test`
-and `mom_mcar_test` on 3,080 (dataset × generator) pairs drawn from 28
-real UCI / OpenML / sklearn tabular datasets under
-`lacuna_tabular_110` missingness generators surfaced four classes of
-numerical failure that synthetic unit tests did not exhibit. All fixed
-in this release; no API breaks.
-
-**Batched MoM GPU Cholesky crash.** `chi_square_mcar_batched_torch`'s
-fast path selected `cholesky_solve` whenever the SVD-based condition
-number check passed, on the assumption that good conditioning implies
-positive-definiteness. On GPU FP32, roundoff can produce covariances
-that pass the cond-number check but have tiny negative eigenvalues —
-Cholesky fails, the call raises `torch._C._LinAlgError`. The fast path
-is now wrapped with `try/except` and falls back to pseudo-inverse when
-the ``regularize`` flag allows. Surfaced on `credit_card_default` ×
-`MNAR-NonLinSocial` during the Lacuna cache build.
-
-**Exception-type preservation in `little_mcar_test`.** The
-ML-estimation `try/except` at the top of `little_mcar_test` wrapped
-*every* exception as a bare `RuntimeError` — including
-`PyStatisticsError` subclasses. This broke the documented
-`except PyStatisticsError:` catch pattern downstream: users falling
-back to a sentinel on MLE failure saw their handler bypassed and the
-full build crash. Fix: explicitly re-raise `PyStatisticsError`, and
-use `raise ... from e` for anything else so the chain is preserved.
-
-**`regularize=True` default on the EM path (opt-out).** `mlest`,
-`_solve_em`, and `EMBackend.solve` gain `regularize: bool = True`,
-mirroring the existing convention on `mom_mcar_test` and
-`little_mcar_test`. When True, `EMBackend._ensure_pd` applies a small
-diagonal ridge — `max(0, 1e-10 − min_eig) + 1e-12` — to the M-step
-sigma whenever its smallest eigenvalue falls below the PD threshold,
-rather than raising `NumericalError`. The ridge is well below any
-statistical precision on real data — the typical case the old path
-rejected had `min_eig ≈ 1e-13` from pure FP64 roundoff on a matrix
-that's theoretically PSD. Dogfooding surfaced cases where `min_eig`
-hit `−0.66` on realistic MNAR mechanisms; the ridge fallback keeps EM
-progressing. Callers needing strict bit-for-bit behaviour pass
-`regularize=False` to restore the old raise.
-
-**Three additional Cholesky ridge-fallbacks** in `_em_batched.py`:
-`e_step_full_batched_np`, `_e_step_full_torch`, and
-`_loglik_full_batched_torch` all compute per-pattern Cholesky of
-`sigma_oo` sub-blocks. Real tabular data can produce individual
-sub-blocks that are numerically indefinite even when the global sigma
-is PD (integer-encoded categoricals with heavy collinearity in the
-intersection of a given missingness pattern's observed variables).
-Each site now wraps Cholesky in `try/except LinAlgError` with a
-`ridge·I` retry (ridge = 1e-10 at pattern scale; statistically
-invisible). Also removed a dead Cholesky call in `e_step_batched_np`
-whose factor was never used downstream — pure crash liability — and
-added a `pinv` fallback to the `np.linalg.solve` at the same site for
-singular sub-blocks.
-
-**Impact.** The Project Lacuna cache build on 3,080 (dataset ×
-generator) pairs went from crashing on the first batch containing
-`breast_cancer` or `credit_card_default` (pre-2.2.0) to completing in
-a single pass at 0.9% MoM sentinel rate and 16.4% MLE sentinel rate
-(the MLE sentinels are legitimate EM non-convergence on 1000-pattern
-datasets — not crashes). Synthetic unit tests: 125/125 mvnmle pass.
-
-**No API breaks.** New defaults (`regularize=True`) are strictly more
-permissive than the old raises — any caller that was crashing before
-will now proceed with a small `UserWarning`. Callers needing strict
-behaviour pass `regularize=False`.
+Four classes of numerical failure on realistic tabular data — Cholesky
+fast-path crash on GPU FP32 roundoff, bare-`RuntimeError` wrapping
+breaking `PyStatisticsError` catch patterns, M-step sigma PD-check
+false negatives from FP64 roundoff, and per-pattern Cholesky on
+indefinite sub-blocks — all fixed in this release with a unified
+`regularize=True` opt-out-to-strict convention across `mlest`,
+`mom_mcar_test`, `little_mcar_test`, and the batched E-step. The
+Project Lacuna cache build on 3,080 (dataset × generator) pairs went
+from crashing on the first batch to completing in a single pass at
+0.9% MoM / 16.4% MLE sentinel rates. No API breaks.
 
 ### 2.1.0 — Real-data EM speedup + monotone closed-form MLE
 

{pystatistics-2.2.0 → pystatistics-2.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pystatistics"
-version = "2.2.0"
+version = "2.3.0"
 description = "GPU-accelerated statistical computing for Python"
 readme = "README.md"
 license = "MIT"
@@ -48,6 +48,12 @@ dependencies = [
 gpu = [
     "torch>=2.0",
 ]
+nonparametric_mcar = [
+    # scikit-learn for propensity_mcar_test (RandomForestClassifier /
+    # GradientBoostingClassifier). HSIC and MissMech in the same subpackage
+    # are pure numpy/scipy and do not need sklearn.
+    "scikit-learn>=1.3",
+]
 dev = [
     "pytest>=7.0",
     "pytest-cov>=4.0",

{pystatistics-2.2.0 → pystatistics-2.3.0}/pystatistics/__init__.py

@@ -16,7 +16,7 @@ Usage:
     result = fit(design)
 """
 
-__version__ = "2.2.0"
+__version__ = "2.3.0"
 __author__ = "Hai-Shuo"
 __email__ = "contact@sgcx.org"
 

pystatistics-2.3.0/pystatistics/mvnmle/backends/_em_batched.py

@@ -0,0 +1,44 @@
+"""Compatibility shim for the batched EM/MCAR building blocks.
+
+The implementation was split into three modules on 2026-04-20 to stay
+under the 500-SLOC file limit (Coding Bible rule 4):
+
+  * ``_em_batched_patterns`` — pattern-index dataclass + ``build_pattern_index``
+  * ``_em_batched_np``       — NumPy backend (CPU)
+  * ``_em_batched_torch``    — Torch backend (GPU + CPU torch)
+
+This module continues to expose every name that callers imported
+pre-split, so no call-site changes were required. New code may import
+from the specific backend module directly.
+"""
+from pystatistics.mvnmle.backends._em_batched_patterns import (
+    _BatchedPatternIndex,
+    _pattern_n,
+    build_pattern_index,
+)
+from pystatistics.mvnmle.backends._em_batched_np import (
+    chi_square_mcar_batched_np,
+    compute_conditional_parameters_np,
+    compute_loglik_batched_np,
+    e_step_full_batched_np,
+)
+from pystatistics.mvnmle.backends._em_batched_torch import (
+    _e_step_full_torch,
+    _loglik_full_torch,
+    chi_square_mcar_batched_torch,
+    compute_conditional_parameters_torch,
+)
+
+__all__ = [
+    "_BatchedPatternIndex",
+    "_pattern_n",
+    "build_pattern_index",
+    "chi_square_mcar_batched_np",
+    "compute_conditional_parameters_np",
+    "compute_loglik_batched_np",
+    "e_step_full_batched_np",
+    "_e_step_full_torch",
+    "_loglik_full_torch",
+    "chi_square_mcar_batched_torch",
+    "compute_conditional_parameters_torch",
+]