trustcv 1.0.0__tar.gz

trustcv-1.0.0/AUTHORS.md

@@ -0,0 +1,27 @@
+# Authors and Contributors
+
+## Core Development Team
+
+### Lead Developers
+- **Farhad Abtahi**
+  - Email: farhad.abtahi@ki.se
+  
+- **Abdelamir Karbalaie**
+  - Email: abdolamir.karbalaie@ki.se
+
+## Acknowledgments
+
+We thank all contributors who have helped improve trustcv through:
+- Bug reports and feature requests
+- Documentation improvements
+- Code contributions
+- Testing and validation
+- Community support
+
+## Contributing
+
+We welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for details on how to get involved.
+
+---
+
+*Last updated: August 2024*

trustcv-1.0.0/CONTRIBUTING.md

@@ -0,0 +1,246 @@
+# Contributing to trustcv
+
+Thank you for your interest in contributing to trustcv! This toolkit aims to improve medical machine learning validation practices, and we welcome contributions from the community.
+
+## 🤝 Code of Conduct
+
+By participating in this project, you agree to abide by our [Code of Conduct](CODE_OF_CONDUCT.md). Please read it before contributing.
+
+## 🚀 How to Contribute
+
+### 1. Reporting Bugs
+
+Before creating bug reports, please check existing issues to avoid duplicates.
+
+**To report a bug:**
+1. Use the [Bug Report](.github/ISSUE_TEMPLATE/bug_report.md) template
+2. Include:
+   - Clear description of the bug
+   - Steps to reproduce
+   - Expected vs actual behavior
+   - System information (OS, Python version)
+   - Minimal code example
+
+### 2. Suggesting Features
+
+We welcome feature suggestions that align with trustcv's mission.
+
+**To suggest a feature:**
+1. Use the [Feature Request](.github/ISSUE_TEMPLATE/feature_request.md) template
+2. Include:
+   - Clear use case
+   - Medical/clinical context
+   - Proposed implementation (if applicable)
+   - Alternative solutions considered
+
+### 3. Contributing Code
+
+#### Setup Development Environment
+
+```bash
+# Fork and clone the repository
+git clone https://github.com/your-username/trustcv.git
+cd trustcv
+
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+
+# Install in development mode
+pip install -e .[dev]
+
+# Install pre-commit hooks
+pre-commit install
+```
+
+#### Development Workflow
+
+1. **Create a branch**
+   ```bash
+   git checkout -b feature/your-feature-name
+   # or
+   git checkout -b fix/issue-number
+   ```
+
+2. **Make your changes**
+   - Follow the coding standards (see below)
+   - Add/update tests
+   - Update documentation
+
+3. **Test your changes**
+   ```bash
+   # Run tests
+   pytest tests/
+   
+   # Check code style
+   flake8 trustcv/
+   black --check trustcv/
+   
+   # Run type checking
+   mypy trustcv/
+   ```
+
+4. **Commit your changes**
+   ```bash
+   git add .
+   git commit -m "feat: add new cross-validation method for spatial data"
+   ```
+   
+   Follow [Conventional Commits](https://www.conventionalcommits.org/):
+   - `feat:` New feature
+   - `fix:` Bug fix
+   - `docs:` Documentation changes
+   - `test:` Test additions/changes
+   - `refactor:` Code refactoring
+   - `style:` Code style changes
+   - `perf:` Performance improvements
+
+5. **Push and create Pull Request**
+   ```bash
+   git push origin feature/your-feature-name
+   ```
+   Then create a PR on GitHub.
+
+## 📝 Coding Standards
+
+### Python Style Guide
+
+We follow PEP 8 with these additions:
+- Line length: 88 characters (Black default)
+- Use type hints for function parameters and returns
+- Document all public functions with docstrings (NumPy style)
+
+### Example Code Style
+
+```python
+from typing import Optional, Tuple, Union
+import numpy as np
+import pandas as pd
+
+
+def calculate_clinical_metric(
+    y_true: np.ndarray,
+    y_pred: np.ndarray,
+    metric: str = "sensitivity",
+    confidence_level: float = 0.95
+) -> Tuple[float, Tuple[float, float]]:
+    """
+    Calculate clinical metric with confidence interval.
+    
+    Parameters
+    ----------
+    y_true : np.ndarray
+        True labels
+    y_pred : np.ndarray
+        Predicted labels
+    metric : str, default='sensitivity'
+        Metric to calculate
+    confidence_level : float, default=0.95
+        Confidence level for interval
+        
+    Returns
+    -------
+    metric_value : float
+        Calculated metric value
+    confidence_interval : Tuple[float, float]
+        Lower and upper bounds of CI
+        
+    Examples
+    --------
+    >>> sensitivity, (ci_lower, ci_upper) = calculate_clinical_metric(
+    ...     y_true, y_pred, metric="sensitivity"
+    ... )
+    """
+    # Implementation
+    pass
+```
+
+### Testing Requirements
+
+- All new features must have tests
+- Maintain test coverage above 80%
+- Use pytest for testing
+- Include both unit and integration tests
+
+### Documentation
+
+- Update docstrings for all changes
+- Update relevant .md files in `/docs`
+- Add examples to docstrings
+- Update notebooks if applicable
+
+## 🏥 Medical/Clinical Considerations
+
+When contributing medical-specific features:
+
+1. **Cite sources**: Include references to medical literature
+2. **Clinical validation**: Explain clinical relevance
+3. **Safety checks**: Consider patient safety implications
+4. **Regulatory aspects**: Note any FDA/CE considerations
+5. **Privacy**: Ensure no PHI in examples
+
+## 📚 Areas We Need Help
+
+### High Priority
+- [ ] Additional CV methods from literature
+- [ ] Integration with medical imaging libraries
+- [ ] Real-world clinical case studies
+- [ ] Performance optimization
+- [ ] More comprehensive tests
+
+### Documentation
+- [ ] Video tutorials
+- [ ] Clinical use case examples
+- [ ] Translation to other languages
+- [ ] API documentation improvements
+
+### Research
+- [ ] Implementing new CV methods from recent papers
+- [ ] Benchmarking against other libraries
+- [ ] Clinical validation studies
+
+## 🔄 Pull Request Process
+
+1. **PR Title**: Use conventional commit format
+2. **Description**: Use the PR template
+3. **Tests**: Ensure all tests pass
+4. **Documentation**: Update if needed
+5. **Review**: Address reviewer feedback
+6. **Squash**: We squash commits on merge
+
+### PR Checklist
+- [ ] Tests pass locally
+- [ ] Code follows style guidelines
+- [ ] Documentation updated
+- [ ] Changelog entry added (if applicable)
+- [ ] No sensitive medical data included
+
+## 📦 Release Process
+
+We use semantic versioning (MAJOR.MINOR.PATCH):
+- MAJOR: Breaking changes
+- MINOR: New features (backward compatible)
+- PATCH: Bug fixes
+
+Releases are automated via GitHub Actions when tags are pushed.
+
+## 💬 Getting Help
+
+- **Discord**: [Join our community](https://discord.gg/medicalcv)
+- **Discussions**: Use GitHub Discussions for questions
+- **Email**: medicalcv@example.com
+
+## 🙏 Recognition
+
+Contributors are recognized in:
+- [AUTHORS.md](AUTHORS.md) file
+- Release notes
+- Project documentation
+
+## 📄 License
+
+By contributing, you agree that your contributions will be licensed under the MIT License.
+
+---
+
+Thank you for helping make medical machine learning more reliable! 🏥🤖

trustcv-1.0.0/LICENSE

@@ -0,0 +1,26 @@
+MIT License
+
+Copyright (c) 2025 trustcv Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+MEDICAL DISCLAIMER:
+This software is for research and educational purposes only. It is not intended
+for clinical use or medical diagnosis. Always consult with qualified healthcare
+professionals and validate any models thoroughly before clinical deployment.

trustcv-1.0.0/MANIFEST.in

@@ -0,0 +1,30 @@
+# Include documentation
+include README.md
+include LICENSE
+include CONTRIBUTING.md
+include AUTHORS.md
+include requirements.txt
+include requirements-dev.txt
+include requirements-frameworks.txt
+
+# Include all package data
+recursive-include trustcv *.py
+
+# Include example scripts
+recursive-include examples *.py
+
+# Include test files
+recursive-include tests *.py
+
+# Exclude unnecessary files
+global-exclude *.pyc
+global-exclude __pycache__
+global-exclude .DS_Store
+global-exclude *.swp
+global-exclude *~
+
+# Exclude development files
+exclude .gitignore
+exclude .git
+exclude pyproject.toml
+exclude PYPI_DEPLOYMENT_GUIDE.md

trustcv-1.0.0/PKG-INFO

@@ -0,0 +1,340 @@
+Metadata-Version: 2.4
+Name: trustcv
+Version: 1.0.0
+Summary: Trustworthy Cross-Validation: Framework-agnostic CV with data leakage detection
+Home-page: https://github.com/ki-smile/trustcv
+Author: SMAILE Team, Karolinska Institutet
+Author-email: "SMAILE Team, Karolinska Institutet" <contact@smile.ki.se>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/ki-smile/trustcv
+Project-URL: Documentation, https://ki-smile.github.io/trustcv
+Project-URL: Repository, https://github.com/ki-smile/trustcv
+Project-URL: Bug Tracker, https://github.com/ki-smile/trustcv/issues
+Keywords: cross-validation,medical,machine-learning,healthcare,validation
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Healthcare Industry
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: AUTHORS.md
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: pandas>=1.3.0
+Requires-Dist: scikit-learn>=1.0.0
+Requires-Dist: scipy>=1.7.0
+Requires-Dist: matplotlib>=3.4.0
+Requires-Dist: seaborn>=0.11.0
+Requires-Dist: plotly>=5.0.0
+Requires-Dist: joblib>=1.0.0
+Requires-Dist: tqdm>=4.60.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
+Requires-Dist: flake8>=4.0.0; extra == "dev"
+Requires-Dist: mypy>=0.950; extra == "dev"
+Requires-Dist: black>=22.0.0; extra == "dev"
+Requires-Dist: isort>=5.10.0; extra == "dev"
+Requires-Dist: bandit>=1.7.0; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=4.5.0; extra == "docs"
+Requires-Dist: sphinx-rtd-theme>=1.0.0; extra == "docs"
+Requires-Dist: sphinx-autodoc-typehints>=1.12.0; extra == "docs"
+Requires-Dist: myst-parser>=0.18.0; extra == "docs"
+Requires-Dist: nbsphinx>=0.9.0; extra == "docs"
+Provides-Extra: viz
+Requires-Dist: bokeh>=2.4.0; extra == "viz"
+Provides-Extra: pytorch
+Requires-Dist: torch; extra == "pytorch"
+Requires-Dist: torchvision; extra == "pytorch"
+Provides-Extra: tensorflow
+Requires-Dist: tensorflow>=2.16; extra == "tensorflow"
+Provides-Extra: monai
+Requires-Dist: monai>=1.0.0; extra == "monai"
+Requires-Dist: nibabel>=3.2.0; extra == "monai"
+Provides-Extra: all
+Requires-Dist: torch; extra == "all"
+Requires-Dist: torchvision; extra == "all"
+Requires-Dist: tensorflow>=2.16; extra == "all"
+Requires-Dist: monai>=1.0.0; extra == "all"
+Requires-Dist: nibabel>=3.2.0; extra == "all"
+Requires-Dist: pytest>=7.0.0; extra == "all"
+Requires-Dist: flake8>=4.0.0; extra == "all"
+Requires-Dist: mypy>=0.950; extra == "all"
+Requires-Dist: black>=22.0.0; extra == "all"
+Requires-Dist: sphinx>=4.5.0; extra == "all"
+Requires-Dist: nbsphinx>=0.9.0; extra == "all"
+Requires-Dist: pytest-cov>=3.0.0; extra == "all"
+Dynamic: author
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-python
+
+# trustcv — Trustworthy Cross-Validation Toolkit
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Documentation](https://img.shields.io/badge/docs-available-brightgreen.svg)](https://github.com/ki-smile/trustcv)
+
+
+
+**TrustCV** is a framework-agnostic toolkit for **reliable cross-validation** in safety-critical and regulated settings.
+It builds on familiar scikit-learn idioms, but adds:
+
+- Carefully designed cross-validation splitters (starting with IID in v0.1).
+- Automatic **data leakage** and **class balance** checks.
+- **Clinical/industrial metrics** with confidence intervals.
+- Simple, **regulatory-ready reporting**.
+
+> **Status:** v1.0.0 – Production release with 29 CV methods across IID, Grouped, Temporal, and Spatial categories. Full data leakage detection and regulatory compliance features.
+
+---
+
+## Why TrustCV?
+
+Standard cross-validation is easy to misuse:
+
+- Train/test splits can accidentally **leak information** (e.g., shared patients, timestamps, engineered features).
+- Imbalanced datasets can give **overly optimistic** metrics if not stratified or monitored.
+- For clinical and industrial applications, we often need **meaningful metrics** and reproducible reports, not just accuracy.
+
+TrustCV addresses these issues by:
+
+- Providing **well-tested IID splitters** with clear semantics.
+- Running **leakage and balance checks** alongside your CV.
+- Exposing **clinical metrics** and simple **reporting utilities** for audits and regulatory files.
+
+---
+
+## What's in v1.0.0
+
+This release includes **29 cross-validation methods** across four categories:
+
+- **IID splitters** (9 methods):
+  `HoldOut`, `KFold`, `StratifiedKFold`, `RepeatedKFold`, `LeaveOneOut`, `LeavePOut`, `BootstrapValidation`, `MonteCarloCV`, `NestedCV`
+
+- **Grouped splitters** (6 methods):
+  `GroupKFold`, `StratifiedGroupKFold`, `LeaveOneGroupOut`, `RepeatedGroupKFold`, `NestedGroupedCV`, `HierarchicalGroupKFold`
+
+- **Temporal splitters** (8 methods):
+  `TimeSeriesSplit`, `BlockedTimeSeriesSplit`, `RollingWindowCV`, `ExpandingWindowCV`, `PurgedKFold`, `CombinatorialPurgedKFold`, `PurgedGroupTimeSeriesSplit`, `NestedTemporalCV`
+
+- **Spatial splitters** (4 methods):
+  `SpatialBlockCV`, `BufferedSpatialCV`, `SpatiotemporalBlockCV`, `EnvironmentalHealthCV`
+
+- **Framework-agnostic runner**:
+  `UniversalCVRunner` + `CVResults` for consistent, reusable CV loops across scikit-learn, PyTorch, TensorFlow, MONAI, and JAX.
+
+- **High-level validator**:
+  `TrustCVValidator` with automatic method selection and leakage detection.
+
+- **Data integrity checks**:
+  `DataLeakageChecker` (6 leakage types), `BalanceChecker`, and `LeakageReport`.
+
+- **Clinical/medical metrics**:
+  `ClinicalMetrics` with confidence intervals (sensitivity, specificity, PPV/NPV, etc.).
+
+- **Regulatory reporting**:
+  `RegulatoryReport` for FDA/CE MDR compliance documentation.
+
+---
+
+
+## Quick Start
+
+### Installation
+
+```bash
+# Install from source (recommended for latest features)
+git clone https://github.com/ki-smile/trustcv.git
+cd trustcv
+pip install -e .
+
+# Or install from PyPI (when released)
+pip install trustcv
+```
+
+
+## Quickstart – IID CV with TrustCV
+
+Here is a minimal example using `StratifiedKFold` and `UniversalCVRunner`:
+
+
+```python
+from trustcv import TrustCVValidator
+from sklearn.datasets import load_breast_cancer
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.pipeline import make_pipeline
+from sklearn.preprocessing import StandardScaler
+
+X, y = load_breast_cancer(return_X_y=True)
+model = make_pipeline(StandardScaler(), RandomForestClassifier(random_state=42))
+
+# Validates with leakage checks and computes clinical CIs
+validator = TrustCVValidator(
+    method="StratifiedKfold",  # IID, stratified
+    n_splits=5,
+    random_state=42,
+    check_leakage=True,
+    check_balance=True,
+)
+results = validator.validate(model=model, X=X, y=y)
+
+print(results.summary()) 
+
+# Output:
+#=== Trustworthy Cross-Validation Results ===
+
+#Performance Metrics (mean +/- std) (method: bootstrap):
+#  accuracy: 0.956 +/- 0.014 [95% CI (bootstrap): 0.946-0.967]
+#  roc_auc: 0.989 +/- 0.009 [95% CI (bootstrap): 0.981-0.995]
+#  sensitivity: 0.966 +/- 0.030 [95% CI (bootstrap): 0.939-0.986]
+#  specificity: 0.939 +/- 0.056 [95% CI (bootstrap): 0.893-0.981]
+#  precision: 0.965 +/- 0.031 [95% CI (bootstrap): 0.937-0.989]
+#  recall: 0.966 +/- 0.030 [95% CI (bootstrap): 0.939-0.986]
+#  f1: 0.965 +/- 0.011 [95% CI (bootstrap): 0.957-0.973]
+
+
+#Data Integrity Checks:
+#  Leakage Check: PASSED
+#  Class Balance: PASSED
+```
+
+For a higher-level workflow with leakage and balance checks, see the
+ [Quickstart: IID CV with TrustCV](https://github.com/ki-smile/trustcv/blob/main/docs/Quickstart%3A%20IID%20Cross-Validation%20with%20TrustCV.md) and  [IID Splitters](https://github.com/ki-smile/trustcv/blob/main/docs/iid_splitters.md) tutorial.
+
+ ### Run the example notebooks
+
+Prefer to learn by running code? From the repo root, open the notebooks in `notebooks/`:
+
+- `notebooks/01_IID_Methods_Showcase.ipynb` – quick tour of the IID splitters and metrics.
+- `notebooks/02_Advanced_Workflow_UniversalRunner.ipynb` – end-to-end UniversalCVRunner workflow.
+- `notebooks/03_TrustCVValidator_Showcase.ipynb` – TrustCVValidator examples with leakage/balance checks.
+- `notebooks/04_TrustCVValidator_IID_Comparison.ipynb` – side-by-side IID method comparison.
+- `notebooks/05_CrossValidation_Comparison.ipynb` – comprehensive CV methods comparison.
+- Reports generated by the notebooks are saved in `notebooks/reports/` (HTML/PDF).
+
+
+
+------
+
+## How TrustCV relates to scikit-learn
+
+**Similarities:**
+
+- Uses familiar scikit-learn idioms: estimators with `fit`/`predict`, splitter objects with `split(X, y)`.
+- Works seamlessly with scikit-learn models, pipelines, and metrics.
+- IID splitters follow scikit-learn semantics (e.g., `KFold`, `StratifiedKFold`).
+
+**Added value:**
+
+- **Leakage and balance checks**: `DataLeakageChecker` and `BalanceChecker` run alongside your CV.
+- **Clinical metrics**: `ClinicalMetrics` computes sensitivity, specificity, PPV/NPV, ROC/PR metrics, and CIs.
+- **Structured results**: `CVResults` and `ValidationResult` standardize fold-level outputs.
+- **Reporting**: `UniversalRegulatoryReport` turns your evaluation into a reproducible HTML/JSON report.
+
+------
+
+## Roadmap
+
+TrustCV v1.0.0 includes all 29 CV methods. Future versions will focus on:
+
+- **v1.1 – Enhanced reporting**
+   Extended regulatory report templates and audit trail features.
+- **v1.2 – Additional frameworks**
+   Expanded support for JAX/Flax, XGBoost, LightGBM, and CatBoost.
+- **v1.3 – AutoML integration**
+   Integration with Optuna for hyperparameter tuning within nested CV.
+
+------
+
+## Contributors
+
+See [AUTHORS.md](https://github.com/ki-smile/trustcv/blob/main/AUTHORS.md) for a full list of contributors and acknowledgments.
+
+### Lead Contributors
+- **[Farhad Abtahi](https://github.com/farhad-abtahi)**
+- **[Abdelamir Karbalaie](https://github.com/abdkar)**
+
+
+
+
+
+### Contributing
+We welcome contributions!
+- Code contributions
+- Medical use case examples
+- Documentation improvements
+- Bug reports and feature requests
+
+Please see:
+
+- [`CONTRIBUTING.md`](https://github.com/ki-smile/trustcv/blob/main/CONTRIBUTING.md)
+- [`CODE_OF_CONDUCT.md`](https://github.com/ki-smile/trustcv/blob/main/CODE_OF_CONDUCT.md)
+
+------
+
+
+
+## 3. Quickstart: IID CV with TrustCV – outline
+
+**See file:** `docs/quickstart_iid.md`  
+(and a matching notebook: `notebooks/Quickstart_IID_TrustCV.ipynb`)
+
+
+## Repository Structure
+
+```
+trustcv/       # Python package (splitters, validators, metrics, core)
+docs/          # Documentation & guides
+notebooks/     # Jupyter tutorials
+examples/      # Real-world examples
+tests/         # Unit & integration tests
+website/       # Static site and visualizations
+```
+
+## Development
+
+```bash
+pip install -e .[dev]
+pytest tests/
+cd docs && make html
+```
+
+## Citation
+
+If you use trustcv in your research, please cite:
+
+```bibtex
+@software{trustcv2025,
+  title = {trustcv: Trustworthy Cross-Validation Toolkit},
+  author = {Abtahi, Farhad and Karbalaie, Abdolamir},
+  year = {2025},
+  url = {https://github.com/ki-smile/trustcv}
+}
+```
+
+## License
+
+MIT License — see [LICENSE](LICENSE).
+
+## Contact & Support
+
+- GitHub Issues: https://github.com/ki-smile/trustcv/issues
+
+---
+
+## ⚠️ Disclaimer
+
+This toolkit is for research and educational purposes. Always validate results with domain experts before clinical deployment.
+
+---
+
+Advancing Medical AI Through Rigorous Validation