nestkit 0.1.0__tar.gz

nestkit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ettore Rocchi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

nestkit-0.1.0/PKG-INFO

@@ -0,0 +1,207 @@
+Metadata-Version: 2.4
+Name: nestkit
+Version: 0.1.0
+Summary: Nested cross-validation toolkit with optional probability calibration, threshold optimization, and comprehensive diagnostics
+Author: Ettore Rocchi
+License: MIT
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: scikit-learn>=1.2
+Requires-Dist: numpy>=1.22
+Requires-Dist: pandas>=1.4
+Requires-Dist: joblib>=1.2
+Requires-Dist: scipy>=1.9
+Provides-Extra: plotting
+Requires-Dist: matplotlib>=3.5; extra == "plotting"
+Requires-Dist: seaborn>=0.12; extra == "plotting"
+Provides-Extra: full
+Requires-Dist: matplotlib>=3.5; extra == "full"
+Requires-Dist: seaborn>=0.12; extra == "full"
+Requires-Dist: shap>=0.42; extra == "full"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: hypothesis>=6.0; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=7.0; extra == "docs"
+Requires-Dist: pydata-sphinx-theme>=0.14; extra == "docs"
+Requires-Dist: sphinx-copybutton; extra == "docs"
+Requires-Dist: myst-parser>=2.0; extra == "docs"
+Requires-Dist: numpydoc; extra == "docs"
+Requires-Dist: nbsphinx; extra == "docs"
+Requires-Dist: ipykernel; extra == "docs"
+Requires-Dist: sphinx-design; extra == "docs"
+Dynamic: license-file
+
+<p align="center">
+  <img src="docs/_static/nestkit_logo.png" alt="nestkit" width="300">
+</p>
+
+<p align="center">
+  <em>A nested cross-validation toolkit for scikit-learn</em>
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/nestkit/"><img src="https://img.shields.io/pypi/v/nestkit" alt="PyPI"></a>
+  <a href="https://pypi.org/project/nestkit/"><img src="https://img.shields.io/pypi/pyversions/nestkit" alt="Python"></a>
+  <a href="https://github.com/ettorerocchi/nestkit/blob/main/LICENSE"><img src="https://img.shields.io/github/license/ettorerocchi/nestkit" alt="License"></a>
+</p>
+
+---
+
+## Motivation
+
+Standard cross-validation conflates model selection with performance estimation, producing optimistically biased scores. When the same data that guides hyperparameter tuning is also used to report performance, the resulting estimates no longer reflect true generalization ability  -  a subtle but pervasive form of data leakage.
+
+**Nested cross-validation** (nested CV) addresses this by separating the two concerns into an inner loop for hyperparameter search and an outer loop for unbiased evaluation. While conceptually simple, implementing nested CV correctly  -  with proper calibration, threshold optimization, and statistical comparison  -  requires careful engineering that is tedious and error-prone to write from scratch.
+
+**nestkit** provides a scikit-learn-compatible implementation that handles the full pipeline: nested CV with integrated probability calibration, decision-threshold optimization, statistical model comparison, hyperparameter stability diagnostics, feature importance aggregation, and visualizations  -  all through a familiar `fit`/`predict` API.
+
+## Key Features
+
+- **Nested cross-validation** for classification and regression with full scikit-learn API compatibility
+- **Post-hoc probability calibration**  -  Platt scaling, isotonic regression, beta calibration, and Venn-ABERS prediction
+- **Threshold optimization**  -  Youden's J, F-beta, cost-sensitive, balanced accuracy, and precision-at-recall criteria with pooled or fold-specific strategies
+- **Statistical model comparison**  -  Nadeau-Bengio corrected t-test, Bayesian correlated t-test with ROPE, and Holm-Bonferroni multi-model correction
+- **Hyperparameter stability diagnostics**  -  selection frequency analysis, pairwise Jaccard similarity
+- **Feature importance aggregation**  -  cross-fold importance with Nogueira stability index and consensus feature selection
+- **Callback system**  -  progress tracking, logging, checkpointing, and custom hooks
+- **25+ plotting functions**  -  ROC curves, confusion matrices, calibration diagrams, threshold sensitivity, critical difference diagrams, and more
+- **Residual-based prediction intervals** for regression tasks
+
+## Installation
+
+```bash
+pip install nestkit
+```
+
+Optional dependency groups:
+
+```bash
+pip install nestkit[plotting]   # matplotlib + seaborn
+pip install nestkit[full]       # plotting + SHAP
+pip install nestkit[dev]        # testing + linting
+pip install nestkit[docs]       # Sphinx documentation
+```
+
+## Quick Start
+
+### Classification
+
+```python
+from sklearn.datasets import load_breast_cancer
+from sklearn.ensemble import RandomForestClassifier
+from nestkit import NestedCVClassifier
+
+X, y = load_breast_cancer(return_X_y=True)
+
+ncv = NestedCVClassifier(
+    estimator=RandomForestClassifier(random_state=42),
+    param_grid={"n_estimators": [50, 100], "max_depth": [3, 5, 10]},
+    outer_cv=5,
+    inner_cv=3,
+    scoring="accuracy",
+    random_state=42,
+)
+ncv.fit(X, y)
+
+results = ncv.results_
+print(results.summary_default_)
+print(results.best_params_per_fold_)
+```
+
+With calibration and threshold optimization:
+
+```python
+ncv = NestedCVClassifier(
+    estimator=RandomForestClassifier(random_state=42),
+    param_grid={"n_estimators": [50, 100], "max_depth": [3, 5]},
+    outer_cv=5,
+    inner_cv=3,
+    calibration_method="isotonic",
+    threshold_strategy="pooled",
+    threshold_criterion="youden",
+    random_state=42,
+)
+ncv.fit(X, y)
+print(ncv.results_.threshold_comparison())
+```
+
+### Regression
+
+```python
+from sklearn.datasets import load_diabetes
+from sklearn.linear_model import Ridge
+from nestkit import NestedCVRegressor
+
+X, y = load_diabetes(return_X_y=True)
+
+ncv = NestedCVRegressor(
+    estimator=Ridge(),
+    param_grid={"alpha": [0.01, 0.1, 1.0, 10.0]},
+    outer_cv=5,
+    inner_cv=3,
+    prediction_intervals=True,
+    random_state=42,
+)
+ncv.fit(X, y)
+
+results = ncv.results_
+print(results.summary_default_)
+print(f"PI coverage: {results.prediction_interval_coverage_['mean']:.3f}")
+```
+
+## Architecture
+
+nestkit's nested CV procedure executes four phases per outer fold:
+
+1. **Inner CV search**  -  hyperparameter tuning via `GridSearchCV` or `RandomizedSearchCV` on the outer training set
+2. **Post-inner processing**  -  probability calibration (classification) or residual collection for prediction intervals (regression)
+3. **Refit**  -  the best hyperparameters are used to refit the estimator on the full outer training set
+4. **Outer evaluation**  -  the refitted model is scored on the held-out outer test fold
+
+**Class hierarchy:**
+
+| Class | Purpose |
+|-------|---------|
+| `NestedCVClassifier` | Classification with calibration + thresholding |
+| `NestedCVRegressor` | Regression with prediction intervals |
+| `ClassifierResults` / `RegressorResults` | Rich result containers |
+| `NestedCVComparator` | Statistical model comparison |
+| `FeatureImportanceAggregator` | Cross-fold importance analysis |
+| `HyperparameterStability` | Selection consistency diagnostics |
+| `PostHocCalibrator` | Standalone probability calibration |
+| `InnerCVReport` | Inner CV analysis and reporting |
+
+## Documentation
+
+Full documentation is available at the [nestkit documentation site](https://nestkit.readthedocs.io/).
+
+## Citation
+
+If you use nestkit in your research, please cite:
+
+```bibtex
+@software{rocchi2026nestkit,
+  author       = {Rocchi, Ettore},
+  title        = {nestkit: A Nested Cross-Validation Toolkit for Scikit-Learn},
+  year         = {2026},
+  url          = {https://github.com/ettorerocchi/nestkit},
+}
+```
+
+## License
+
+MIT  -  see [LICENSE](LICENSE) for details.

nestkit-0.1.0/README.md

@@ -0,0 +1,160 @@
+<p align="center">
+  <img src="docs/_static/nestkit_logo.png" alt="nestkit" width="300">
+</p>
+
+<p align="center">
+  <em>A nested cross-validation toolkit for scikit-learn</em>
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/nestkit/"><img src="https://img.shields.io/pypi/v/nestkit" alt="PyPI"></a>
+  <a href="https://pypi.org/project/nestkit/"><img src="https://img.shields.io/pypi/pyversions/nestkit" alt="Python"></a>
+  <a href="https://github.com/ettorerocchi/nestkit/blob/main/LICENSE"><img src="https://img.shields.io/github/license/ettorerocchi/nestkit" alt="License"></a>
+</p>
+
+---
+
+## Motivation
+
+Standard cross-validation conflates model selection with performance estimation, producing optimistically biased scores. When the same data that guides hyperparameter tuning is also used to report performance, the resulting estimates no longer reflect true generalization ability  -  a subtle but pervasive form of data leakage.
+
+**Nested cross-validation** (nested CV) addresses this by separating the two concerns into an inner loop for hyperparameter search and an outer loop for unbiased evaluation. While conceptually simple, implementing nested CV correctly  -  with proper calibration, threshold optimization, and statistical comparison  -  requires careful engineering that is tedious and error-prone to write from scratch.
+
+**nestkit** provides a scikit-learn-compatible implementation that handles the full pipeline: nested CV with integrated probability calibration, decision-threshold optimization, statistical model comparison, hyperparameter stability diagnostics, feature importance aggregation, and visualizations  -  all through a familiar `fit`/`predict` API.
+
+## Key Features
+
+- **Nested cross-validation** for classification and regression with full scikit-learn API compatibility
+- **Post-hoc probability calibration**  -  Platt scaling, isotonic regression, beta calibration, and Venn-ABERS prediction
+- **Threshold optimization**  -  Youden's J, F-beta, cost-sensitive, balanced accuracy, and precision-at-recall criteria with pooled or fold-specific strategies
+- **Statistical model comparison**  -  Nadeau-Bengio corrected t-test, Bayesian correlated t-test with ROPE, and Holm-Bonferroni multi-model correction
+- **Hyperparameter stability diagnostics**  -  selection frequency analysis, pairwise Jaccard similarity
+- **Feature importance aggregation**  -  cross-fold importance with Nogueira stability index and consensus feature selection
+- **Callback system**  -  progress tracking, logging, checkpointing, and custom hooks
+- **25+ plotting functions**  -  ROC curves, confusion matrices, calibration diagrams, threshold sensitivity, critical difference diagrams, and more
+- **Residual-based prediction intervals** for regression tasks
+
+## Installation
+
+```bash
+pip install nestkit
+```
+
+Optional dependency groups:
+
+```bash
+pip install nestkit[plotting]   # matplotlib + seaborn
+pip install nestkit[full]       # plotting + SHAP
+pip install nestkit[dev]        # testing + linting
+pip install nestkit[docs]       # Sphinx documentation
+```
+
+## Quick Start
+
+### Classification
+
+```python
+from sklearn.datasets import load_breast_cancer
+from sklearn.ensemble import RandomForestClassifier
+from nestkit import NestedCVClassifier
+
+X, y = load_breast_cancer(return_X_y=True)
+
+ncv = NestedCVClassifier(
+    estimator=RandomForestClassifier(random_state=42),
+    param_grid={"n_estimators": [50, 100], "max_depth": [3, 5, 10]},
+    outer_cv=5,
+    inner_cv=3,
+    scoring="accuracy",
+    random_state=42,
+)
+ncv.fit(X, y)
+
+results = ncv.results_
+print(results.summary_default_)
+print(results.best_params_per_fold_)
+```
+
+With calibration and threshold optimization:
+
+```python
+ncv = NestedCVClassifier(
+    estimator=RandomForestClassifier(random_state=42),
+    param_grid={"n_estimators": [50, 100], "max_depth": [3, 5]},
+    outer_cv=5,
+    inner_cv=3,
+    calibration_method="isotonic",
+    threshold_strategy="pooled",
+    threshold_criterion="youden",
+    random_state=42,
+)
+ncv.fit(X, y)
+print(ncv.results_.threshold_comparison())
+```
+
+### Regression
+
+```python
+from sklearn.datasets import load_diabetes
+from sklearn.linear_model import Ridge
+from nestkit import NestedCVRegressor
+
+X, y = load_diabetes(return_X_y=True)
+
+ncv = NestedCVRegressor(
+    estimator=Ridge(),
+    param_grid={"alpha": [0.01, 0.1, 1.0, 10.0]},
+    outer_cv=5,
+    inner_cv=3,
+    prediction_intervals=True,
+    random_state=42,
+)
+ncv.fit(X, y)
+
+results = ncv.results_
+print(results.summary_default_)
+print(f"PI coverage: {results.prediction_interval_coverage_['mean']:.3f}")
+```
+
+## Architecture
+
+nestkit's nested CV procedure executes four phases per outer fold:
+
+1. **Inner CV search**  -  hyperparameter tuning via `GridSearchCV` or `RandomizedSearchCV` on the outer training set
+2. **Post-inner processing**  -  probability calibration (classification) or residual collection for prediction intervals (regression)
+3. **Refit**  -  the best hyperparameters are used to refit the estimator on the full outer training set
+4. **Outer evaluation**  -  the refitted model is scored on the held-out outer test fold
+
+**Class hierarchy:**
+
+| Class | Purpose |
+|-------|---------|
+| `NestedCVClassifier` | Classification with calibration + thresholding |
+| `NestedCVRegressor` | Regression with prediction intervals |
+| `ClassifierResults` / `RegressorResults` | Rich result containers |
+| `NestedCVComparator` | Statistical model comparison |
+| `FeatureImportanceAggregator` | Cross-fold importance analysis |
+| `HyperparameterStability` | Selection consistency diagnostics |
+| `PostHocCalibrator` | Standalone probability calibration |
+| `InnerCVReport` | Inner CV analysis and reporting |
+
+## Documentation
+
+Full documentation is available at the [nestkit documentation site](https://nestkit.readthedocs.io/).
+
+## Citation
+
+If you use nestkit in your research, please cite:
+
+```bibtex
+@software{rocchi2026nestkit,
+  author       = {Rocchi, Ettore},
+  title        = {nestkit: A Nested Cross-Validation Toolkit for Scikit-Learn},
+  year         = {2026},
+  url          = {https://github.com/ettorerocchi/nestkit},
+}
+```
+
+## License
+
+MIT  -  see [LICENSE](LICENSE) for details.

nestkit-0.1.0/pyproject.toml

@@ -0,0 +1,80 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "nestkit"
+description = "Nested cross-validation toolkit with optional probability calibration, threshold optimization, and comprehensive diagnostics"
+readme = "README.md"
+license = {text = "MIT"}
+authors = [{name = "Ettore Rocchi"}]
+requires-python = ">=3.10"
+dynamic = ["version", "dependencies"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+
+[project.optional-dependencies]
+plotting = ["matplotlib>=3.5", "seaborn>=0.12"]
+full = ["matplotlib>=3.5", "seaborn>=0.12", "shap>=0.42"]
+dev = [
+    "pytest>=7",
+    "pytest-cov",
+    "hypothesis>=6.0",
+    "ruff",
+    "pre-commit",
+]
+docs = [
+    "sphinx>=7.0",
+    "pydata-sphinx-theme>=0.14",
+    "sphinx-copybutton",
+    "myst-parser>=2.0",
+    "numpydoc",
+    "nbsphinx",
+    "ipykernel",
+    "sphinx-design",
+]
+
+[tool.setuptools.dynamic]
+version = {attr = "nestkit.__version__"}
+dependencies = {file = "requirements.txt"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+target-version = "py310"
+line-length = 99
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = [
+    "E",    # pycodestyle errors
+    "W",    # pycodestyle warnings
+    "F",    # pyflakes
+    "I",    # isort
+    "UP",   # pyupgrade
+    "B",    # flake8-bugbear
+    "SIM",  # flake8-simplify
+    "RUF",  # ruff-specific
+]
+ignore = [
+    "E501",   # line too long (handled by formatter)
+    "B905",   # zip strict
+    "SIM108", # ternary operator
+]
+
+[tool.ruff.lint.isort]
+known-first-party = ["nestkit"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-ra -q"

nestkit-0.1.0/requirements.txt

@@ -0,0 +1,5 @@
+scikit-learn>=1.2
+numpy>=1.22
+pandas>=1.4
+joblib>=1.2
+scipy>=1.9

nestkit-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

nestkit-0.1.0/src/nestkit/__init__.py

@@ -0,0 +1,49 @@
+"""nestkit  -  A rigorous nested cross-validation toolkit for scikit-learn.
+
+Core estimators
+---------------
+- :class:`NestedCVClassifier`  -  Classification with optional calibration
+  and threshold optimization.
+- :class:`NestedCVRegressor`  -  Regression with optional residual-based
+  prediction intervals.
+
+Results
+-------
+- :class:`ClassifierResults` / :class:`RegressorResults`  -  Rich result
+  containers with summary statistics, export methods, and plots.
+
+Submodules
+----------
+- :mod:`nestkit.calibration`  -  Post-hoc probability calibration and
+  calibration diagnostics.
+- :mod:`nestkit.callbacks`  -  Fold-level monitoring (progress, logging,
+  checkpointing).
+- :mod:`nestkit.comparison`  -  Statistical model comparison.
+- :mod:`nestkit.diagnostics`  -  Hyperparameter stability analysis.
+- :mod:`nestkit.importance`  -  Cross-fold feature importance aggregation.
+- :mod:`nestkit.inner`  -  Inner CV tuning reports.
+- :mod:`nestkit.plotting`  -  25+ plotting functions for nested CV results.
+- :mod:`nestkit.thresholding`  -  Decision-threshold optimization criteria.
+"""
+
+from __future__ import annotations
+
+import logging
+
+from nestkit.classifier import NestedCVClassifier
+from nestkit.regressor import NestedCVRegressor
+from nestkit.results.classifier_results import ClassifierResults
+from nestkit.results.regressor_results import RegressorResults
+
+__version__ = "0.1.0"
+
+logger = logging.getLogger("nestkit")
+logger.addHandler(logging.NullHandler())
+
+__all__ = [
+    "ClassifierResults",
+    "NestedCVClassifier",
+    "NestedCVRegressor",
+    "RegressorResults",
+    "__version__",
+]