tinyconformal 0.1.2__tar.gz → 0.2.0__tar.gz

{tinyconformal-0.1.2 → tinyconformal-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tinyconformal
-Version: 0.1.2
+Version: 0.2.0
 Summary: A small toolbox for conformal prediction
 Author-email: Lucas Leão <heylucasleao@gmail.com>
 License-Expression: MIT
@@ -14,18 +14,21 @@ Requires-Dist: venn-abers>=1.4.6
 Requires-Dist: scikit-learn>=1.3.0
 Requires-Dist: quantile-forest>=1.4.2
 Provides-Extra: plot
-Requires-Dist: plotly>=5.22.0; extra == "plot"
-Requires-Dist: kaleido<=0.2.1; extra == "plot"
+Requires-Dist: tinyshift[plot]; extra == "plot"
 Provides-Extra: notebook
 Requires-Dist: nbformat>=5.10.4; extra == "notebook"
-Requires-Dist: ipykernel>=6.7.0; extra == "notebook"
-Provides-Extra: all
-Requires-Dist: plot; extra == "all"
-Requires-Dist: notebook; extra == "all"
+Requires-Dist: ipykernel>6.7.0; extra == "notebook"
+Provides-Extra: dev
+Requires-Dist: tinyconformal[notebook,plot]; extra == "dev"
+Requires-Dist: pytest>=7.0.0; extra == "dev"
 Dynamic: license-file
 
 # TinyConformal
-TinyConformal is an experimental Python library for conformal predictions, providing tools to generate valid prediction sets with a specified significance level (alpha). This project aims to facilitate the implementation of personal and future projects on the topic.
+
+Related project: [tinyshift](https://github.com/HeyLucasLeao/tinyshift)
+
+TinyConformal is a Python library for conformal prediction in classification and regression.
+It provides tools to build valid prediction sets and prediction intervals with a target significance level (`alpha`).
 
 For more information on a previous project related to Out-of-Bag (OOB) solutions, visit [this link](https://github.com/HeyLucasLeao/cp-study).
 
@@ -40,36 +43,68 @@ Currently, TinyConformal supports Out-of-Bag (OOB) solutions for `RandomForestCl
 
 ## Installation
 
-Install TinyConformal using pip:
+### Using pip
 
 ```bash
 pip install tinyconformal
 ```
 
-> **Note:** If you want to enable plotting capabilities, you need to install the extras using Poetry:
+Optional extras:
+
+```bash
+pip install "tinyconformal[plot]"
+pip install "tinyconformal[notebook]"
+pip install "tinyconformal[dev]"
+```
+
+### Using uv
+
+Install in the current environment:
 
 ```bash
-poetry install --E plot
+uv pip install tinyconformal
 ```
 
-## Usage
+Add as a dependency in a project:
 
-### Importing Classifiers
+```bash
+uv add tinyconformal
+```
+
+Optional extras with uv:
+
+```bash
+uv pip install "tinyconformal[plot]"
+uv pip install "tinyconformal[notebook]"
+uv pip install "tinyconformal[dev]"
+```
 
-Import the conformal classifiers from the `tinyconformal.classifier` module:
+## Submodules and usage
+
+TinyConformal is organized into two main submodules:
+
+- `tinyconformal.classifier`: conformal classifiers for binary classification.
+- `tinyconformal.regressor`: conformal regressors and exactness-bound utilities.
+
+### Classifier submodule
+
+Import from `tinyconformal.classifier`:
 
 ```python
-from tinyconformal.classifier import BinaryClassConditionalConformalClassifier
 from tinyconformal.classifier import BinaryMarginalConformalClassifier
+from tinyconformal.classifier import BinaryClassConditionalConformalClassifier
 ```
-### Importing Regressors
 
-Import the conformal regressors from the `tinyconformal.regressor` module:
+### Regressor submodule
+
+Import from `tinyconformal.regressor`:
 
 ```python
 from tinyconformal.regressor import ConformalizedRegressor
 from tinyconformal.regressor import ConformalizedQuantileRegressor
+from tinyconformal.regressor import ExactnessBound
 ```
+
 ### Example
 
 Example usage of `BinaryClassConditionalConformalClassifier`:
@@ -85,7 +120,7 @@ learner.fit(X_train, y_train)
 
 # Create and fit the conformal classifier
 conformal_classifier = BinaryClassConditionalConformalClassifier(learner)
-conformal_classifier.fit(X=X_train, y=y_train, oob=True)
+conformal_classifier.fit(y=y_train, oob=True)
 
 # Make predictions
 X_test = ...  # your test data
@@ -113,13 +148,18 @@ For regressors, you can combine an exactness bound estimate with unlabeled calib
 
 ```python
 from sklearn.ensemble import RandomForestRegressor
-from tinyconformal import ConformalizedRegressor, ExactnessBound
+from tinyconformal.regressor import ConformalizedRegressor, ExactnessBound
 
 learner = RandomForestRegressor(random_state=42)
-tilde_beta = ExactnessBound.estimate_icp_bound(learner, X_train, y_train, p=0.95, cv=5)
+tilde_beta, beta = ExactnessBound.estimate_icp_bound(
+    learner, X_train, y_train, p=0.95, cv=5
+)
+
+# Fit learner before using conformal regressor
+learner.fit(X_train, y_train)
 
 regressor = ConformalizedRegressor(learner, alpha=0.05)
-regressor.unlabeled_fit(X_unlabeled, tilde_beta=tilde_beta)
+regressor.unlabeled_fit(X_unlabeled, tilde_beta=tilde_beta, beta=beta)
 
 intervals = regressor.predict_interval(X_test)
 ```
@@ -140,16 +180,16 @@ print(results)
 `BinaryMarginalConformalClassifier` is a marginal-coverage conformal classifier that uses a classifier as the underlying learner.
 
 - Training via labeled calibration: `fit(X, y)`
-- Training via OOB calibration: `fit(X, y, oob=True)`
-- Training via unlabeled calibration: `unlabeled_fit(X)`
+- Training via OOB calibration: `fit(y=y_train, oob=True)`
+- Training via unlabeled calibration: `unlabeled_fit(X, beta=...)`
 
 ### BinaryClassConditionalConformalClassifier
 
 `BinaryClassConditionalConformalClassifier` is a class-conditional conformal classifier that uses a classifier as the underlying learner.
 
 - Training via labeled calibration: `fit(X, y)`
-- Training via OOB calibration: `fit(X, y, oob=True)`
-- Training via unlabeled calibration: `unlabeled_fit(X)` using pseudo-labels derived from the model probabilities
+- Training via OOB calibration: `fit(y=y_train, oob=True)`
+- Training via unlabeled calibration: `unlabeled_fit(X, beta=...)` using pseudo-labels derived from the model probabilities
 
 ### ConformalizedRegressor
 
@@ -157,7 +197,7 @@ print(results)
 
 - Training via labeled calibration: `fit(X, y)`
 - Training via OOB calibration: `fit(X, y, oob=True)`
-- Training via unlabeled calibration: `unlabeled_fit(X, tilde_beta=...)` using an exactness bound
+- Training via unlabeled calibration: `unlabeled_fit(X, tilde_beta=..., beta=...)` using an exactness bound
 
 ### ConformalizedQuantileRegressor
 
@@ -165,7 +205,7 @@ print(results)
 
 - Training via labeled calibration: `fit(X, y)`
 - Training via OOB calibration: `fit(X, y, oob=True)`
-- Training via unlabeled calibration: `unlabeled_fit(X, tilde_beta=...)` using an exactness bound
+- Training via unlabeled calibration: `unlabeled_fit(X, tilde_beta=..., beta=...)` using an exactness bound
 
 ### ExactnessBound
 

{tinyconformal-0.1.2 → tinyconformal-0.2.0}/README.md

@@ -1,5 +1,9 @@
 # TinyConformal
-TinyConformal is an experimental Python library for conformal predictions, providing tools to generate valid prediction sets with a specified significance level (alpha). This project aims to facilitate the implementation of personal and future projects on the topic.
+
+Related project: [tinyshift](https://github.com/HeyLucasLeao/tinyshift)
+
+TinyConformal is a Python library for conformal prediction in classification and regression.
+It provides tools to build valid prediction sets and prediction intervals with a target significance level (`alpha`).
 
 For more information on a previous project related to Out-of-Bag (OOB) solutions, visit [this link](https://github.com/HeyLucasLeao/cp-study).
 
@@ -14,36 +18,68 @@ Currently, TinyConformal supports Out-of-Bag (OOB) solutions for `RandomForestCl
 
 ## Installation
 
-Install TinyConformal using pip:
+### Using pip
 
 ```bash
 pip install tinyconformal
 ```
 
-> **Note:** If you want to enable plotting capabilities, you need to install the extras using Poetry:
+Optional extras:
+
+```bash
+pip install "tinyconformal[plot]"
+pip install "tinyconformal[notebook]"
+pip install "tinyconformal[dev]"
+```
+
+### Using uv
+
+Install in the current environment:
 
 ```bash
-poetry install --E plot
+uv pip install tinyconformal
 ```
 
-## Usage
+Add as a dependency in a project:
 
-### Importing Classifiers
+```bash
+uv add tinyconformal
+```
+
+Optional extras with uv:
+
+```bash
+uv pip install "tinyconformal[plot]"
+uv pip install "tinyconformal[notebook]"
+uv pip install "tinyconformal[dev]"
+```
 
-Import the conformal classifiers from the `tinyconformal.classifier` module:
+## Submodules and usage
+
+TinyConformal is organized into two main submodules:
+
+- `tinyconformal.classifier`: conformal classifiers for binary classification.
+- `tinyconformal.regressor`: conformal regressors and exactness-bound utilities.
+
+### Classifier submodule
+
+Import from `tinyconformal.classifier`:
 
 ```python
-from tinyconformal.classifier import BinaryClassConditionalConformalClassifier
 from tinyconformal.classifier import BinaryMarginalConformalClassifier
+from tinyconformal.classifier import BinaryClassConditionalConformalClassifier
 ```
-### Importing Regressors
 
-Import the conformal regressors from the `tinyconformal.regressor` module:
+### Regressor submodule
+
+Import from `tinyconformal.regressor`:
 
 ```python
 from tinyconformal.regressor import ConformalizedRegressor
 from tinyconformal.regressor import ConformalizedQuantileRegressor
+from tinyconformal.regressor import ExactnessBound
 ```
+
 ### Example
 
 Example usage of `BinaryClassConditionalConformalClassifier`:
@@ -59,7 +95,7 @@ learner.fit(X_train, y_train)
 
 # Create and fit the conformal classifier
 conformal_classifier = BinaryClassConditionalConformalClassifier(learner)
-conformal_classifier.fit(X=X_train, y=y_train, oob=True)
+conformal_classifier.fit(y=y_train, oob=True)
 
 # Make predictions
 X_test = ...  # your test data
@@ -87,13 +123,18 @@ For regressors, you can combine an exactness bound estimate with unlabeled calib
 
 ```python
 from sklearn.ensemble import RandomForestRegressor
-from tinyconformal import ConformalizedRegressor, ExactnessBound
+from tinyconformal.regressor import ConformalizedRegressor, ExactnessBound
 
 learner = RandomForestRegressor(random_state=42)
-tilde_beta = ExactnessBound.estimate_icp_bound(learner, X_train, y_train, p=0.95, cv=5)
+tilde_beta, beta = ExactnessBound.estimate_icp_bound(
+    learner, X_train, y_train, p=0.95, cv=5
+)
+
+# Fit learner before using conformal regressor
+learner.fit(X_train, y_train)
 
 regressor = ConformalizedRegressor(learner, alpha=0.05)
-regressor.unlabeled_fit(X_unlabeled, tilde_beta=tilde_beta)
+regressor.unlabeled_fit(X_unlabeled, tilde_beta=tilde_beta, beta=beta)
 
 intervals = regressor.predict_interval(X_test)
 ```
@@ -114,16 +155,16 @@ print(results)
 `BinaryMarginalConformalClassifier` is a marginal-coverage conformal classifier that uses a classifier as the underlying learner.
 
 - Training via labeled calibration: `fit(X, y)`
-- Training via OOB calibration: `fit(X, y, oob=True)`
-- Training via unlabeled calibration: `unlabeled_fit(X)`
+- Training via OOB calibration: `fit(y=y_train, oob=True)`
+- Training via unlabeled calibration: `unlabeled_fit(X, beta=...)`
 
 ### BinaryClassConditionalConformalClassifier
 
 `BinaryClassConditionalConformalClassifier` is a class-conditional conformal classifier that uses a classifier as the underlying learner.
 
 - Training via labeled calibration: `fit(X, y)`
-- Training via OOB calibration: `fit(X, y, oob=True)`
-- Training via unlabeled calibration: `unlabeled_fit(X)` using pseudo-labels derived from the model probabilities
+- Training via OOB calibration: `fit(y=y_train, oob=True)`
+- Training via unlabeled calibration: `unlabeled_fit(X, beta=...)` using pseudo-labels derived from the model probabilities
 
 ### ConformalizedRegressor
 
@@ -131,7 +172,7 @@ print(results)
 
 - Training via labeled calibration: `fit(X, y)`
 - Training via OOB calibration: `fit(X, y, oob=True)`
-- Training via unlabeled calibration: `unlabeled_fit(X, tilde_beta=...)` using an exactness bound
+- Training via unlabeled calibration: `unlabeled_fit(X, tilde_beta=..., beta=...)` using an exactness bound
 
 ### ConformalizedQuantileRegressor
 
@@ -139,7 +180,7 @@ print(results)
 
 - Training via labeled calibration: `fit(X, y)`
 - Training via OOB calibration: `fit(X, y, oob=True)`
-- Training via unlabeled calibration: `unlabeled_fit(X, tilde_beta=...)` using an exactness bound
+- Training via unlabeled calibration: `unlabeled_fit(X, tilde_beta=..., beta=...)` using an exactness bound
 
 ### ExactnessBound
 

{tinyconformal-0.1.2 → tinyconformal-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tinyconformal"
-version = "0.1.2"
+version = "0.2.0"
 description = "A small toolbox for conformal prediction"
 license = "MIT"
 authors = [{name = "Lucas Leão", email = "heylucasleao@gmail.com"}]
@@ -18,6 +18,16 @@ dependencies = [
 requires-python = ">=3.10"
 
 [project.optional-dependencies]
-plot = ["plotly (>=5.22.0)", "kaleido (<=0.2.1)"]
-notebook = ["nbformat (>=5.10.4)", "ipykernel (>=6.7.0)"]
-all = ["plot", "notebook"]
+plot = [
+    "tinyshift[plot]",
+]
+
+notebook = [
+    "nbformat>=5.10.4",
+    "ipykernel>6.7.0",
+]
+
+dev = [
+    "tinyconformal[plot,notebook]",
+    "pytest>=7.0.0",
+]

tinyconformal-0.2.0/tests/test_classifiers.py

@@ -0,0 +1,200 @@
+# Copyright (c) 2024-2026 Lucas Leão
+# TinyConformal - A small toolbox for conformal prediction
+# Licensed under the MIT License
+
+
+import pytest
+import numpy as np
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import make_classification
+from tinyconformal.classifier.marginal import BinaryMarginalConformalClassifier
+from tinyconformal.classifier.class_conditional import (
+    BinaryClassConditionalConformalClassifier,
+)
+
+
+@pytest.fixture
+def dataset():
+    weights = [0.4, 0.6]
+    seed = 42
+
+    X, y = make_classification(
+        n_samples=1500,
+        n_features=20,
+        n_informative=2,
+        weights=weights,
+        random_state=seed,
+        n_redundant=2,
+    )
+
+    X_train, X_test, y_train, y_test = train_test_split(
+        X, y, test_size=0.2, random_state=seed, stratify=y
+    )
+    X_train, X_calib, y_train, y_calib = train_test_split(
+        X_train, y_train, test_size=0.25, random_state=seed, stratify=y_train
+    )
+
+    return {
+        "X_train": X_train,
+        "y_train": y_train,
+        "X_calib": X_calib,
+        "y_calib": y_calib,
+        "X_test": X_test,
+        "y_test": y_test,
+    }
+
+
+@pytest.fixture
+def learner(dataset):
+    model = RandomForestClassifier(oob_score=True, n_estimators=10)
+    model.fit(dataset["X_train"], dataset["y_train"])
+    return model
+
+
+def _assert_classifier_outputs(classifier, dataset):
+    classifier.calibrate(dataset["X_calib"], dataset["y_calib"])
+    assert 0 < classifier.alpha <= 0.2
+
+    y_proba = classifier.predict_proba(dataset["X_test"])
+    assert y_proba.shape == (dataset["X_test"].shape[0], 2)
+
+    prediction_set = classifier.predict_set(dataset["X_test"])
+    assert prediction_set.shape == (dataset["X_test"].shape[0], 2)
+
+    p_values = classifier.predict_p(dataset["X_test"])
+    assert p_values.shape == (dataset["X_test"].shape[0], 2)
+
+    y_pred = classifier.predict(dataset["X_test"])
+    assert y_pred.shape == (dataset["X_test"].shape[0],)
+
+    eval_dict = classifier.evaluate(dataset["X_test"], dataset["y_test"])
+    assert isinstance(eval_dict, dict)
+    expected_keys = {
+        "total",
+        "alpha",
+        "beta",
+        "coverage_rate",
+        "one_c",
+        "avg_c",
+        "empty",
+        "error",
+        "log_loss",
+        "ece",
+        "bm",
+        "mcc",
+        "f1",
+        "fpr",
+    }
+    assert set(eval_dict.keys()) == expected_keys
+
+
+def test_marginal_classifier(dataset, learner):
+    classifier = BinaryMarginalConformalClassifier(learner)
+    classifier.fit(dataset["X_calib"], dataset["y_calib"], oob=False)
+    _assert_classifier_outputs(classifier, dataset)
+
+
+def test_class_cond_classifier(dataset, learner):
+    classifier = BinaryClassConditionalConformalClassifier(learner)
+    classifier.fit(dataset["X_calib"], dataset["y_calib"], oob=False)
+    _assert_classifier_outputs(classifier, dataset)
+
+
+def test_oob_marginal_classifier(dataset, learner):
+    classifier = BinaryMarginalConformalClassifier(learner)
+    classifier.fit(y=dataset["y_train"], oob=True)
+    _assert_classifier_outputs(classifier, dataset)
+
+
+def test_oob_class_conditional_classifier(dataset, learner):
+    classifier = BinaryClassConditionalConformalClassifier(learner)
+    classifier.fit(y=dataset["y_train"], oob=True)
+    _assert_classifier_outputs(classifier, dataset)
+
+
+@pytest.mark.parametrize(
+    "classifier_cls",
+    [BinaryMarginalConformalClassifier, BinaryClassConditionalConformalClassifier],
+)
+def test_unlabeled_fit_requires_X(classifier_cls, learner):
+    classifier = classifier_cls(learner)
+
+    with pytest.raises(ValueError, match="Unlabeled calibration data"):
+        classifier.unlabeled_fit(X=None)
+
+
+@pytest.mark.parametrize(
+    "classifier_cls",
+    [BinaryMarginalConformalClassifier, BinaryClassConditionalConformalClassifier],
+)
+def test_unlabeled_fit_warns_when_beta_missing(classifier_cls, learner, dataset):
+    classifier = classifier_cls(learner)
+
+    with pytest.warns(UserWarning, match="beta"):
+        classifier.unlabeled_fit(dataset["X_calib"])
+
+
+@pytest.mark.parametrize(
+    "classifier_cls",
+    [BinaryMarginalConformalClassifier, BinaryClassConditionalConformalClassifier],
+)
+def test_unlabeled_fit_disables_calibration(classifier_cls, learner, dataset):
+    classifier = classifier_cls(learner)
+    classifier.unlabeled_fit(dataset["X_calib"], beta=0.1)
+
+    with pytest.raises(ValueError, match="Calibration is not applicable"):
+        classifier.calibrate(dataset["X_test"], dataset["y_test"])
+
+
+def test_unlabeled_fit_marginal_predict_p_properties(dataset, learner):
+    classifier = BinaryMarginalConformalClassifier(learner)
+    classifier.unlabeled_fit(dataset["X_calib"], beta=0.1)
+
+    assert classifier.is_unlabeled is True
+    assert classifier.beta == 0.1
+    assert classifier.hinge.shape == (dataset["X_calib"].shape[0],)
+    assert classifier.n == dataset["X_calib"].shape[0]
+
+    p_values = classifier.predict_p(dataset["X_test"])
+    assert p_values.shape == (dataset["X_test"].shape[0], 2)
+    assert np.all(p_values >= 0)
+    assert np.all(p_values <= 1)
+
+
+def test_unlabeled_fit_class_conditional_predict_p_properties(dataset, learner):
+    classifier = BinaryClassConditionalConformalClassifier(learner)
+    classifier.unlabeled_fit(dataset["X_calib"], beta=0.1)
+
+    assert classifier.is_unlabeled is True
+    assert classifier.beta == 0.1
+    assert len(classifier.hinge) == 2
+    assert len(classifier.n) == 2
+    assert sum(classifier.n) == dataset["X_calib"].shape[0]
+
+    p_values = classifier.predict_p(dataset["X_test"])
+    assert p_values.shape == (dataset["X_test"].shape[0], 2)
+    assert np.all(p_values >= 0)
+    assert np.all(p_values <= 1)
+
+
+@pytest.mark.parametrize(
+    "classifier_cls",
+    [BinaryMarginalConformalClassifier, BinaryClassConditionalConformalClassifier],
+)
+def test_fit_requires_y(classifier_cls, learner, dataset):
+    classifier = classifier_cls(learner)
+
+    with pytest.raises(ValueError, match="true labels"):
+        classifier.fit(dataset["X_calib"], y=None, oob=False)
+
+
+@pytest.mark.parametrize(
+    "classifier_cls",
+    [BinaryMarginalConformalClassifier, BinaryClassConditionalConformalClassifier],
+)
+def test_fit_oob_rejects_X(classifier_cls, learner, dataset):
+    classifier = classifier_cls(learner)
+
+    with pytest.raises(ValueError, match="should not be provided"):
+        classifier.fit(dataset["X_calib"], y=dataset["y_train"], oob=True)