ssbc 0.1.0__tar.gz

ssbc-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,119 @@
+# Contributing
+
+Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
+
+You can contribute in many ways:
+
+## Types of Contributions
+
+### Report Bugs
+
+Report bugs at https://github.com/phzwart/ssbc/issues.
+
+If you are reporting a bug, please include:
+
+- Your operating system name and version.
+- Any details about your local setup that might be helpful in troubleshooting.
+- Detailed steps to reproduce the bug.
+
+### Fix Bugs
+
+Look through the GitHub issues for bugs. Anything tagged with "bug" and "help wanted" is open to whoever wants to implement it.
+
+### Implement Features
+
+Look through the GitHub issues for features. Anything tagged with "enhancement" and "help wanted" is open to whoever wants to implement it.
+
+### Write Documentation
+
+ssbc could always use more documentation, whether as part of the official docs, in docstrings, or even on the web in blog posts, articles, and such.
+
+### Submit Feedback
+
+The best way to send feedback is to file an issue at https://github.com/phzwart/ssbc/issues.
+
+If you are proposing a feature:
+
+- Explain in detail how it would work.
+- Keep the scope as narrow as possible, to make it easier to implement.
+- Remember that this is a volunteer-driven project, and that contributions are welcome :)
+
+## Get Started!
+
+Ready to contribute? Here's how to set up `ssbc` for local development.
+
+1. Fork the `ssbc` repo on GitHub.
+2. Clone your fork locally:
+
+   ```sh
+   git clone git@github.com:your_name_here/ssbc.git
+   ```
+
+3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:
+
+   ```sh
+   mkvirtualenv ssbc
+   cd ssbc/
+   python setup.py develop
+   ```
+
+4. Create a branch for local development:
+
+   ```sh
+   git checkout -b name-of-your-bugfix-or-feature
+   ```
+
+   Now you can make your changes locally.
+
+5. When you're done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:
+
+   ```sh
+   make lint
+   make test
+   # Or
+   make test-all
+   ```
+
+   To get flake8 and tox, just pip install them into your virtualenv.
+
+6. Commit your changes and push your branch to GitHub:
+
+   ```sh
+   git add .
+   git commit -m "Your detailed description of your changes."
+   git push origin name-of-your-bugfix-or-feature
+   ```
+
+7. Submit a pull request through the GitHub website.
+
+## Pull Request Guidelines
+
+Before you submit a pull request, check that it meets these guidelines:
+
+1. The pull request should include tests.
+2. If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.md.
+3. The pull request should work for Python 3.12 and 3.13. Tests run in GitHub Actions on every pull request to the main branch, make sure that the tests pass for all supported Python versions.
+
+## Tips
+
+To run a subset of tests:
+
+```sh
+pytest tests.test_ssbc
+```
+
+## Deploying
+
+A reminder for the maintainers on how to deploy. Make sure all your changes are committed (including an entry in HISTORY.md). Then run:
+
+```sh
+bump2version patch # possible: major / minor / patch
+git push
+git push --tags
+```
+
+You can set up a [GitHub Actions workflow](https://docs.github.com/en/actions/use-cases-and-examples/building-and-testing/building-and-testing-python#publishing-to-pypi) to automatically deploy your package to PyPI when you push a new tag.
+
+## Code of Conduct
+
+Please note that this project is released with a [Contributor Code of Conduct](CODE_OF_CONDUCT.md). By participating in this project you agree to abide by its terms.

ssbc-0.1.0/HISTORY.md

@@ -0,0 +1,5 @@
+# History
+
+## 0.1.0 (2025-10-10)
+
+* First release on PyPI.

ssbc-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025, Petrus H Zwart
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ssbc-0.1.0/MANIFEST.in

@@ -0,0 +1,10 @@
+include CONTRIBUTING.md
+include HISTORY.md
+include LICENSE
+include README.md
+
+recursive-include tests *
+recursive-exclude * __pycache__
+recursive-exclude * *.py[co]
+
+recursive-include docs *.md Makefile *.jpg *.png *.gif

ssbc-0.1.0/PKG-INFO

@@ -0,0 +1,266 @@
+Metadata-Version: 2.4
+Name: ssbc
+Version: 0.1.0
+Summary: Small Sample Beta Correction - PAC guarantees with small datasets
+Author-email: Petrus H Zwart <phzwart@lbl.gov>
+Maintainer-email: Petrus H Zwart <phzwart@lbl.gov>
+License-Expression: MIT
+Project-URL: bugs, https://github.com/phzwart/ssbc/issues
+Project-URL: changelog, https://github.com/phzwart/ssbc/blob/master/changelog.md
+Project-URL: homepage, https://github.com/phzwart/ssbc
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Mathematics
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: matplotlib
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: plotly
+Requires-Dist: rich
+Requires-Dist: scipy
+Requires-Dist: typer
+Provides-Extra: test
+Requires-Dist: coverage; extra == "test"
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+Requires-Dist: ruff; extra == "test"
+Requires-Dist: ty; extra == "test"
+Requires-Dist: ipdb; extra == "test"
+Provides-Extra: dev
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: bandit[toml]; extra == "dev"
+Dynamic: license-file
+
+# SSBC: Small-Sample Beta Correction
+
+![PyPI version](https://img.shields.io/pypi/v/ssbc.svg)
+[![Documentation Status](https://readthedocs.org/projects/ssbc/badge/?version=latest)](https://ssbc.readthedocs.io/en/latest/?version=latest)
+
+**Small-Sample Beta Correction** provides PAC (Probably Approximately Correct) guarantees for conformal prediction with small calibration sets.
+
+* PyPI package: https://pypi.org/project/ssbc/
+* Free software: MIT License
+* Documentation: https://ssbc.readthedocs.io.
+
+## Overview
+
+SSBC addresses the challenge of constructing valid prediction sets when you have limited calibration data. Traditional conformal prediction assumes large calibration sets, but in practice, data is often scarce. SSBC provides finite-sample correction with PAC guarantees.
+
+### Key Features
+
+- ✅ **Small-Sample Correction**: PAC-valid conformal prediction for small calibration sets
+- ✅ **Mondrian Conformal Prediction**: Per-class calibration for handling class imbalance
+- ✅ **Comprehensive Statistics**: Detailed reporting with Clopper-Pearson confidence intervals
+- ✅ **Hyperparameter Tuning**: Interactive parallel coordinates visualization for parameter optimization
+- ✅ **Simulation Tools**: Built-in data generators for testing and validation
+
+## Installation
+
+```bash
+pip install ssbc
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/yourusername/ssbc.git
+cd ssbc
+pip install -e .
+```
+
+## Quick Start
+
+```python
+import numpy as np
+from ssbc import (
+    ssbc_correct,
+    BinaryClassifierSimulator,
+    split_by_class,
+    mondrian_conformal_calibrate,
+    report_prediction_stats,
+)
+
+# 1. Generate simulated data
+sim = BinaryClassifierSimulator(
+    p_class1=0.1,
+    beta_params_class0=(2, 8),
+    beta_params_class1=(8, 2),
+    seed=42
+)
+labels, probs = sim.generate(n_samples=100)
+
+# 2. Split by class for Mondrian CP
+class_data = split_by_class(labels, probs)
+
+# 3. Calibrate with SSBC correction
+cal_result, pred_stats = mondrian_conformal_calibrate(
+    class_data=class_data,
+    alpha_target=0.10,  # 10% miscoverage
+    delta=0.10,         # 90% PAC guarantee
+    mode="beta"
+)
+
+# 4. Generate comprehensive report
+summary = report_prediction_stats(pred_stats, cal_result, verbose=True)
+```
+
+## Core Algorithm: SSBC
+
+The SSBC algorithm finds the optimal corrected miscoverage rate α' that satisfies:
+
+**P(Coverage(α') ≥ 1 - α_target) ≥ 1 - δ**
+
+```python
+from ssbc import ssbc_correct
+
+result = ssbc_correct(
+    alpha_target=0.10,  # Target 10% miscoverage
+    n=50,               # Calibration set size
+    delta=0.10,         # PAC parameter (90% confidence)
+    mode="beta"         # Infinite test window
+)
+
+print(f"Corrected α: {result.alpha_corrected:.4f}")
+print(f"u*: {result.u_star}")
+```
+
+### Parameters
+
+- `alpha_target`: Target miscoverage rate (e.g., 0.10 for 90% coverage)
+- `n`: Calibration set size
+- `delta`: PAC risk tolerance (probability of violating guarantee)
+- `mode`: "beta" (infinite test) or "beta-binomial" (finite test)
+
+## Module Structure
+
+The library is organized into focused modules:
+
+### Core Modules
+
+- **`ssbc.core`**: Core SSBC algorithm (`ssbc_correct`, `SSBCResult`)
+- **`ssbc.conformal`**: Mondrian conformal prediction (`mondrian_conformal_calibrate`, `split_by_class`)
+- **`ssbc.statistics`**: Statistical utilities (`clopper_pearson_intervals`, `cp_interval`)
+
+### Analysis & Visualization
+
+- **`ssbc.visualization`**: Reporting and plotting (`report_prediction_stats`, `plot_parallel_coordinates_plotly`)
+- **`ssbc.hyperparameter`**: Parameter tuning (`sweep_hyperparams_and_collect`, `sweep_and_plot_parallel_plotly`)
+
+### Testing & Simulation
+
+- **`ssbc.simulation`**: Data generators (`BinaryClassifierSimulator`)
+
+## Examples
+
+The `examples/` directory contains comprehensive demonstrations:
+
+### 1. Core SSBC Algorithm
+```bash
+python examples/ssbc_core_example.py
+```
+Demonstrates the SSBC algorithm for different calibration set sizes.
+
+### 2. Mondrian Conformal Prediction
+```bash
+python examples/mondrian_conformal_example.py
+```
+Complete workflow: simulation → calibration → reporting.
+
+### 3. Hyperparameter Sweep
+```bash
+python examples/hyperparameter_sweep_example.py
+```
+Interactive parameter tuning with parallel coordinates visualization.
+
+## Hyperparameter Tuning
+
+Sweep over α and δ values to find optimal configurations:
+
+```python
+from ssbc import sweep_and_plot_parallel_plotly
+import numpy as np
+
+# Define grid
+alpha_grid = np.arange(0.05, 0.20, 0.05)
+delta_grid = np.arange(0.05, 0.20, 0.05)
+
+# Run sweep and visualize
+df, fig = sweep_and_plot_parallel_plotly(
+    class_data=class_data,
+    alpha_0=alpha_grid, delta_0=delta_grid,
+    alpha_1=alpha_grid, delta_1=delta_grid,
+    color='err_all'  # Color by error rate
+)
+
+# Save interactive plot
+fig.write_html("sweep_results.html")
+
+# Analyze results
+print(df[['a0', 'd0', 'cov', 'sing_rate', 'err_all']].head())
+```
+
+The interactive plot allows you to:
+- Brush (select) ranges on any axis to filter configurations
+- Explore trade-offs between coverage, automation, and error rates
+- Identify Pareto-optimal hyperparameter settings
+
+## Understanding the Output
+
+### Per-Class Statistics (Conditioned on True Label)
+
+For each class, the report shows:
+- **Abstentions**: Empty prediction sets
+- **Singletons**: Confident predictions (automated decisions)
+  - Correct: True label in singleton set
+  - Incorrect: True label not in singleton set
+- **Doublets**: Both labels included (escalated to human review)
+
+### Marginal Statistics (Deployment View)
+
+Overall performance metrics ignoring true labels:
+- **Coverage**: Fraction of predictions containing the true label
+- **Singleton rate**: Fraction of confident predictions (automation level)
+- **Escalation rate**: Fraction requiring human review
+- **Error rates**: By predicted class and overall
+
+### PAC Bounds
+
+The report includes theoretical and observed singleton error rates:
+- **α'_bound**: Theoretical upper bound from PAC analysis
+- **α'_observed**: Observed error rate on calibration data
+- ✓ if observed ≤ bound (PAC guarantee satisfied)
+
+## Citation
+
+If you use SSBC in your research, please cite:
+
+```bibtex
+@software{ssbc2024,
+  author = {Zwart, Petrus H},
+  title = {SSBC: Small-Sample Beta Correction},
+  year = {2024},
+  url = {https://github.com/yourusername/ssbc}
+}
+```
+
+## Contributing
+
+Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## Credits
+
+This package was created with [Cookiecutter](https://github.com/audreyfeldroy/cookiecutter) and the [audreyfeldroy/cookiecutter-pypackage](https://github.com/audreyfeldroy/cookiecutter-pypackage) project template.

ssbc-0.1.0/README.md

@@ -0,0 +1,223 @@
+# SSBC: Small-Sample Beta Correction
+
+![PyPI version](https://img.shields.io/pypi/v/ssbc.svg)
+[![Documentation Status](https://readthedocs.org/projects/ssbc/badge/?version=latest)](https://ssbc.readthedocs.io/en/latest/?version=latest)
+
+**Small-Sample Beta Correction** provides PAC (Probably Approximately Correct) guarantees for conformal prediction with small calibration sets.
+
+* PyPI package: https://pypi.org/project/ssbc/
+* Free software: MIT License
+* Documentation: https://ssbc.readthedocs.io.
+
+## Overview
+
+SSBC addresses the challenge of constructing valid prediction sets when you have limited calibration data. Traditional conformal prediction assumes large calibration sets, but in practice, data is often scarce. SSBC provides finite-sample correction with PAC guarantees.
+
+### Key Features
+
+- ✅ **Small-Sample Correction**: PAC-valid conformal prediction for small calibration sets
+- ✅ **Mondrian Conformal Prediction**: Per-class calibration for handling class imbalance
+- ✅ **Comprehensive Statistics**: Detailed reporting with Clopper-Pearson confidence intervals
+- ✅ **Hyperparameter Tuning**: Interactive parallel coordinates visualization for parameter optimization
+- ✅ **Simulation Tools**: Built-in data generators for testing and validation
+
+## Installation
+
+```bash
+pip install ssbc
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/yourusername/ssbc.git
+cd ssbc
+pip install -e .
+```
+
+## Quick Start
+
+```python
+import numpy as np
+from ssbc import (
+    ssbc_correct,
+    BinaryClassifierSimulator,
+    split_by_class,
+    mondrian_conformal_calibrate,
+    report_prediction_stats,
+)
+
+# 1. Generate simulated data
+sim = BinaryClassifierSimulator(
+    p_class1=0.1,
+    beta_params_class0=(2, 8),
+    beta_params_class1=(8, 2),
+    seed=42
+)
+labels, probs = sim.generate(n_samples=100)
+
+# 2. Split by class for Mondrian CP
+class_data = split_by_class(labels, probs)
+
+# 3. Calibrate with SSBC correction
+cal_result, pred_stats = mondrian_conformal_calibrate(
+    class_data=class_data,
+    alpha_target=0.10,  # 10% miscoverage
+    delta=0.10,         # 90% PAC guarantee
+    mode="beta"
+)
+
+# 4. Generate comprehensive report
+summary = report_prediction_stats(pred_stats, cal_result, verbose=True)
+```
+
+## Core Algorithm: SSBC
+
+The SSBC algorithm finds the optimal corrected miscoverage rate α' that satisfies:
+
+**P(Coverage(α') ≥ 1 - α_target) ≥ 1 - δ**
+
+```python
+from ssbc import ssbc_correct
+
+result = ssbc_correct(
+    alpha_target=0.10,  # Target 10% miscoverage
+    n=50,               # Calibration set size
+    delta=0.10,         # PAC parameter (90% confidence)
+    mode="beta"         # Infinite test window
+)
+
+print(f"Corrected α: {result.alpha_corrected:.4f}")
+print(f"u*: {result.u_star}")
+```
+
+### Parameters
+
+- `alpha_target`: Target miscoverage rate (e.g., 0.10 for 90% coverage)
+- `n`: Calibration set size
+- `delta`: PAC risk tolerance (probability of violating guarantee)
+- `mode`: "beta" (infinite test) or "beta-binomial" (finite test)
+
+## Module Structure
+
+The library is organized into focused modules:
+
+### Core Modules
+
+- **`ssbc.core`**: Core SSBC algorithm (`ssbc_correct`, `SSBCResult`)
+- **`ssbc.conformal`**: Mondrian conformal prediction (`mondrian_conformal_calibrate`, `split_by_class`)
+- **`ssbc.statistics`**: Statistical utilities (`clopper_pearson_intervals`, `cp_interval`)
+
+### Analysis & Visualization
+
+- **`ssbc.visualization`**: Reporting and plotting (`report_prediction_stats`, `plot_parallel_coordinates_plotly`)
+- **`ssbc.hyperparameter`**: Parameter tuning (`sweep_hyperparams_and_collect`, `sweep_and_plot_parallel_plotly`)
+
+### Testing & Simulation
+
+- **`ssbc.simulation`**: Data generators (`BinaryClassifierSimulator`)
+
+## Examples
+
+The `examples/` directory contains comprehensive demonstrations:
+
+### 1. Core SSBC Algorithm
+```bash
+python examples/ssbc_core_example.py
+```
+Demonstrates the SSBC algorithm for different calibration set sizes.
+
+### 2. Mondrian Conformal Prediction
+```bash
+python examples/mondrian_conformal_example.py
+```
+Complete workflow: simulation → calibration → reporting.
+
+### 3. Hyperparameter Sweep
+```bash
+python examples/hyperparameter_sweep_example.py
+```
+Interactive parameter tuning with parallel coordinates visualization.
+
+## Hyperparameter Tuning
+
+Sweep over α and δ values to find optimal configurations:
+
+```python
+from ssbc import sweep_and_plot_parallel_plotly
+import numpy as np
+
+# Define grid
+alpha_grid = np.arange(0.05, 0.20, 0.05)
+delta_grid = np.arange(0.05, 0.20, 0.05)
+
+# Run sweep and visualize
+df, fig = sweep_and_plot_parallel_plotly(
+    class_data=class_data,
+    alpha_0=alpha_grid, delta_0=delta_grid,
+    alpha_1=alpha_grid, delta_1=delta_grid,
+    color='err_all'  # Color by error rate
+)
+
+# Save interactive plot
+fig.write_html("sweep_results.html")
+
+# Analyze results
+print(df[['a0', 'd0', 'cov', 'sing_rate', 'err_all']].head())
+```
+
+The interactive plot allows you to:
+- Brush (select) ranges on any axis to filter configurations
+- Explore trade-offs between coverage, automation, and error rates
+- Identify Pareto-optimal hyperparameter settings
+
+## Understanding the Output
+
+### Per-Class Statistics (Conditioned on True Label)
+
+For each class, the report shows:
+- **Abstentions**: Empty prediction sets
+- **Singletons**: Confident predictions (automated decisions)
+  - Correct: True label in singleton set
+  - Incorrect: True label not in singleton set
+- **Doublets**: Both labels included (escalated to human review)
+
+### Marginal Statistics (Deployment View)
+
+Overall performance metrics ignoring true labels:
+- **Coverage**: Fraction of predictions containing the true label
+- **Singleton rate**: Fraction of confident predictions (automation level)
+- **Escalation rate**: Fraction requiring human review
+- **Error rates**: By predicted class and overall
+
+### PAC Bounds
+
+The report includes theoretical and observed singleton error rates:
+- **α'_bound**: Theoretical upper bound from PAC analysis
+- **α'_observed**: Observed error rate on calibration data
+- ✓ if observed ≤ bound (PAC guarantee satisfied)
+
+## Citation
+
+If you use SSBC in your research, please cite:
+
+```bibtex
+@software{ssbc2024,
+  author = {Zwart, Petrus H},
+  title = {SSBC: Small-Sample Beta Correction},
+  year = {2024},
+  url = {https://github.com/yourusername/ssbc}
+}
+```
+
+## Contributing
+
+Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## Credits
+
+This package was created with [Cookiecutter](https://github.com/audreyfeldroy/cookiecutter) and the [audreyfeldroy/cookiecutter-pypackage](https://github.com/audreyfeldroy/cookiecutter-pypackage) project template.

ssbc-0.1.0/docs/index.md

@@ -0,0 +1,16 @@
+# Welcome to ssbc's documentation!
+
+## Contents
+
+- [Readme](readme.md)
+- [Installation](installation.md)
+- [Usage](usage.md)
+- [Modules](modules.md)
+- [Contributing](contributing.md)
+- [History](history.md)
+
+## Indices and tables
+
+- [Index](genindex)
+- [Module Index](modindex)
+- [Search](search)

ssbc-0.1.0/docs/installation.md

@@ -0,0 +1,38 @@
+# Installation
+
+## Stable release
+
+To install ssbc, run this command in your terminal:
+
+```sh
+uv add ssbc
+```
+
+Or if you prefer to use `pip`:
+
+```sh
+pip install ssbc
+```
+
+## From source
+
+The source files for ssbc can be downloaded from the [Github repo](https://github.com/phzwart/ssbc).
+
+You can either clone the public repository:
+
+```sh
+git clone git://github.com/phzwart/ssbc
+```
+
+Or download the [tarball](https://github.com/phzwart/ssbc/tarball/master):
+
+```sh
+curl -OJL https://github.com/phzwart/ssbc/tarball/master
+```
+
+Once you have a copy of the source, you can install it with:
+
+```sh
+cd ssbc
+uv pip install .
+```