imbeval 0.1.0__tar.gz

imbeval-0.1.0/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to this project are documented here.
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), versioning follows [SemVer](https://semver.org/).
+
+## [0.1.0] - 2026-06-26
+
+### Added
+- `evaluation_report`: combined production-readiness report with plain-English verdict.
+- `minority_class_report`: precision/recall/F1/confusion matrix focused on the minority class.
+- `per_class_confidence`: mean predicted confidence per true class.
+- `calibration_score` / `reliability_curve`: Expected Calibration Error and reliability diagram data.
+- `optimal_threshold`: F1-maximizing decision threshold sweep.
+- `cost_sensitive_threshold`: business-cost-minimizing decision threshold sweep.
+- Initial test suite (8 tests, all passing).
+- Full docs: README, usage guide, API reference, publishing guide.
+
+### Roadmap (not yet implemented)
+- Multi-class cost-sensitive thresholding (currently binary only).
+- Built-in matplotlib plotting helper for `reliability_curve`.
+- Bootstrap confidence intervals on all reported metrics.

imbeval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Srikanth Sridhar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

imbeval-0.1.0/PKG-INFO

@@ -0,0 +1,118 @@
+Metadata-Version: 2.4
+Name: imbeval
+Version: 0.1.0
+Summary: Honest, production-readiness evaluation for imbalanced classification models.
+Project-URL: Homepage, https://github.com/sricodings
+Project-URL: Repository, https://github.com/sricodings/imbeval
+Project-URL: Issues, https://github.com/sricodings/imbeval/issues
+Project-URL: Documentation, https://github.com/sricodings/imbeval#readme
+Author-email: Srikanth Sridhar <srisrikanthtvs@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: calibration,imbalanced-classification,machine-learning,model-evaluation,threshold-tuning
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.21
+Requires-Dist: scikit-learn>=1.0
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# imbeval
+
+**Honest production-readiness evaluation for imbalanced classification models.**
+
+Standard metric libraries hand you precision/recall/F1 and stop there. On imbalanced
+data (fraud, churn, medical diagnosis, anomaly detection, rare-event prediction) that's
+not enough to know if a model is actually safe to ship. `imbeval` answers the real
+question: **is this model usable in production, and at what threshold?**
+
+It combines three things most teams check manually and inconsistently:
+
+1. **Minority-class performance** — not buried inside macro-averages.
+2. **Calibration quality** — is the model's confidence trustworthy, or just confidently wrong?
+3. **Threshold tuning** — the default 0.5 threshold is almost always wrong on imbalanced data; `imbeval` finds a better one, optionally weighted by real business cost (cost of a false positive vs a false negative).
+
+## Install
+
+```bash
+pip install imbeval
+```
+
+(Once published — see the [publishing guide](docs/publishing.md) if you're building this from source.)
+
+## Quickstart
+
+```python
+from imbeval import evaluation_report
+
+# y_true: ground truth labels (0/1)
+# y_pred_proba: predicted probability of the positive class, from model.predict_proba(X)[:, 1]
+report = evaluation_report(
+    y_true,
+    y_pred_proba,
+    cost_fp=1,     # cost of a false alarm
+    cost_fn=25,    # cost of missing a true positive (e.g. missed fraud)
+)
+
+print(report["verdict"])
+print(report["minority_class"])
+print(report["optimal_f1_threshold"])
+print(report["cost_sensitive_threshold"])
+```
+
+Example output:
+
+```
+Not yet production-ready: minority-class recall is below 50% at the default 0.5 threshold;
+default 0.5 threshold is far from optimal; consider using optimal_f1_threshold.
+```
+
+## What's inside
+
+| Function | What it does |
+|---|---|
+| `evaluation_report(y_true, y_pred_proba, ...)` | One combined report + plain-English verdict |
+| `minority_class_report(y_true, y_pred)` | Precision/recall/F1 focused on the minority class |
+| `per_class_confidence(y_true, y_pred_proba)` | Mean model confidence per true class |
+| `calibration_score(y_true, y_pred_proba)` | Expected Calibration Error (ECE) |
+| `reliability_curve(y_true, y_pred_proba)` | Data for plotting a reliability diagram |
+| `optimal_threshold(y_true, y_pred_proba)` | Best decision threshold by F1 |
+| `cost_sensitive_threshold(y_true, y_pred_proba, cost_fp, cost_fn)` | Best threshold by real business cost |
+
+Full API reference: [docs/api.md](docs/api.md)
+Usage guide and recipes: [docs/usage.md](docs/usage.md)
+Publishing this package yourself: [docs/publishing.md](docs/publishing.md)
+
+## Why this exists
+
+Most "imbalanced learning" tools (e.g. `imbalanced-learn`) focus on *fixing* the data
+(SMOTE and friends). `imbeval` focuses on the other end of the pipeline: telling you
+honestly whether the *model* you already trained is good enough, and at what threshold,
+once class imbalance is in play. It's meant to sit right before you ship.
+
+## Status
+
+Early (v0.1.0). The core API (`evaluation_report`, threshold tools, calibration tools)
+is stable for binary classification. Multi-class support is on the roadmap — see
+[CHANGELOG.md](CHANGELOG.md).
+
+## Contributing
+
+Issues and PRs welcome once the repo is public. See [docs/usage.md](docs/usage.md) for
+how the modules fit together if you want to extend it.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

imbeval-0.1.0/README.md

@@ -0,0 +1,87 @@
+# imbeval
+
+**Honest production-readiness evaluation for imbalanced classification models.**
+
+Standard metric libraries hand you precision/recall/F1 and stop there. On imbalanced
+data (fraud, churn, medical diagnosis, anomaly detection, rare-event prediction) that's
+not enough to know if a model is actually safe to ship. `imbeval` answers the real
+question: **is this model usable in production, and at what threshold?**
+
+It combines three things most teams check manually and inconsistently:
+
+1. **Minority-class performance** — not buried inside macro-averages.
+2. **Calibration quality** — is the model's confidence trustworthy, or just confidently wrong?
+3. **Threshold tuning** — the default 0.5 threshold is almost always wrong on imbalanced data; `imbeval` finds a better one, optionally weighted by real business cost (cost of a false positive vs a false negative).
+
+## Install
+
+```bash
+pip install imbeval
+```
+
+(Once published — see the [publishing guide](docs/publishing.md) if you're building this from source.)
+
+## Quickstart
+
+```python
+from imbeval import evaluation_report
+
+# y_true: ground truth labels (0/1)
+# y_pred_proba: predicted probability of the positive class, from model.predict_proba(X)[:, 1]
+report = evaluation_report(
+    y_true,
+    y_pred_proba,
+    cost_fp=1,     # cost of a false alarm
+    cost_fn=25,    # cost of missing a true positive (e.g. missed fraud)
+)
+
+print(report["verdict"])
+print(report["minority_class"])
+print(report["optimal_f1_threshold"])
+print(report["cost_sensitive_threshold"])
+```
+
+Example output:
+
+```
+Not yet production-ready: minority-class recall is below 50% at the default 0.5 threshold;
+default 0.5 threshold is far from optimal; consider using optimal_f1_threshold.
+```
+
+## What's inside
+
+| Function | What it does |
+|---|---|
+| `evaluation_report(y_true, y_pred_proba, ...)` | One combined report + plain-English verdict |
+| `minority_class_report(y_true, y_pred)` | Precision/recall/F1 focused on the minority class |
+| `per_class_confidence(y_true, y_pred_proba)` | Mean model confidence per true class |
+| `calibration_score(y_true, y_pred_proba)` | Expected Calibration Error (ECE) |
+| `reliability_curve(y_true, y_pred_proba)` | Data for plotting a reliability diagram |
+| `optimal_threshold(y_true, y_pred_proba)` | Best decision threshold by F1 |
+| `cost_sensitive_threshold(y_true, y_pred_proba, cost_fp, cost_fn)` | Best threshold by real business cost |
+
+Full API reference: [docs/api.md](docs/api.md)
+Usage guide and recipes: [docs/usage.md](docs/usage.md)
+Publishing this package yourself: [docs/publishing.md](docs/publishing.md)
+
+## Why this exists
+
+Most "imbalanced learning" tools (e.g. `imbalanced-learn`) focus on *fixing* the data
+(SMOTE and friends). `imbeval` focuses on the other end of the pipeline: telling you
+honestly whether the *model* you already trained is good enough, and at what threshold,
+once class imbalance is in play. It's meant to sit right before you ship.
+
+## Status
+
+Early (v0.1.0). The core API (`evaluation_report`, threshold tools, calibration tools)
+is stable for binary classification. Multi-class support is on the roadmap — see
+[CHANGELOG.md](CHANGELOG.md).
+
+## Contributing
+
+Issues and PRs welcome once the repo is public. See [docs/usage.md](docs/usage.md) for
+how the modules fit together if you want to extend it.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

imbeval-0.1.0/docs/api.md

@@ -0,0 +1,92 @@
+# API Reference
+
+All public functions are importable directly from `imbeval`.
+
+---
+
+### `evaluation_report(y_true, y_pred_proba, cost_fp=None, cost_fn=None, n_bins=10)`
+
+Combined production-readiness report.
+
+**Parameters**
+- `y_true` (array-like): binary ground truth labels (0/1).
+- `y_pred_proba` (array-like): predicted probability of the positive class.
+- `cost_fp` (float, optional): cost of one false positive. Required together with `cost_fn` to get a cost-sensitive threshold.
+- `cost_fn` (float, optional): cost of one false negative.
+- `n_bins` (int, default 10): bins used for calibration scoring.
+
+**Returns** `dict`:
+```python
+{
+  "minority_class": {...},          # see minority_class_report
+  "calibration_error": 0.04,        # float, ECE
+  "optimal_f1_threshold": {...},    # see optimal_threshold
+  "cost_sensitive_threshold": {...} or None,
+  "verdict": "..."                  # plain-English summary
+}
+```
+
+---
+
+### `minority_class_report(y_true, y_pred, classes=None, minority_label=None)`
+
+Precision/recall/F1/support/confusion matrix focused on the minority class.
+
+**Parameters**
+- `y_true`, `y_pred` (array-like): true and predicted integer labels.
+- `minority_label`: explicit label to treat as minority. If omitted, auto-detected as the class with lowest support.
+
+**Returns** `dict` with keys: `minority_label`, `support`, `precision`, `recall`, `f1`, `confusion_matrix`.
+
+---
+
+### `per_class_confidence(y_true, y_pred_proba, classes=None)`
+
+Mean predicted probability the model assigns to the *true* class, per class.
+
+**Parameters**
+- `y_true` (array-like): integer class labels.
+- `y_pred_proba` (array-like, shape `(n_samples, n_classes)`): predicted probabilities.
+- `classes` (list, optional): display labels for each class index.
+
+**Returns** `dict` mapping class label → mean confidence (or `None` if no samples for that class).
+
+---
+
+### `calibration_score(y_true, y_pred_proba, n_bins=10)`
+
+Expected Calibration Error (ECE). 0 = perfectly calibrated.
+
+**Returns** `float`.
+
+---
+
+### `reliability_curve(y_true, y_pred_proba, n_bins=10)`
+
+Binned data for plotting a reliability diagram.
+
+**Returns** `dict` with keys `bin_confidence`, `bin_accuracy`, `bin_count` (each a list of length `n_bins`).
+
+---
+
+### `optimal_threshold(y_true, y_pred_proba, metric="f1", n_steps=200)`
+
+Sweeps thresholds and returns the one maximizing the chosen metric.
+
+**Returns** `dict`: `{"threshold": float, "score": float}`.
+
+---
+
+### `cost_sensitive_threshold(y_true, y_pred_proba, cost_fp, cost_fn, n_steps=200)`
+
+Finds the threshold minimizing `fp * cost_fp + fn * cost_fn`.
+
+**Returns** `dict`: `{"threshold": float, "total_cost": float, "false_positives": int, "false_negatives": int}`.
+
+---
+
+## Type conventions
+
+- All `y_true` / `y_pred` for binary functions use `0`/`1` integer labels.
+- All `y_pred_proba` for binary functions is the probability of the **positive (1)** class — i.e. `model.predict_proba(X)[:, 1]`.
+- `per_class_confidence` is the only function expecting a full `(n_samples, n_classes)` probability matrix, for multi-class use.

imbeval-0.1.0/docs/publishing.md

@@ -0,0 +1,120 @@
+# Publishing Guide (for first-time package authors)
+
+This walks you through every step to get `imbeval` (or any Python package you build
+this way) onto GitHub and PyPI, properly licensed, with zero prior packaging experience.
+
+## 0. Before you publish anything
+
+- Double-check the name isn't taken: https://pypi.org/project/imbeval/ — if it's taken, rename it everywhere (`pyproject.toml`'s `name`, the folder under `src/`, imports in tests/docs).
+- Make sure tests pass locally (we already verified 8/8 pass for this package).
+- Fill in real values in `pyproject.toml`: your name, email, and GitHub URL (currently placeholders).
+
+## 1. Choose and apply a license
+
+You already have an MIT `LICENSE` file in this package — MIT is the most permissive,
+common choice for libraries because it lets anyone use, modify, and redistribute your
+code (even commercially) as long as they keep your copyright notice. This maximizes
+adoption, which is what you want for a library you hope many people use.
+
+If you'd rather require derivative works to stay open-source, use Apache 2.0 (also
+patent-safe, popular for ML tooling) or GPLv3 (strongest copyleft, but it discourages
+commercial adoption — usually a bad choice for "I want everyone to use this").
+
+You do **not** need to register or pay for anything to apply a license — putting the
+`LICENSE` file in your repo root and referencing it in `pyproject.toml` (`license = "MIT"`)
+is legally sufficient. GitHub will also detect and display it automatically.
+
+## 2. Put the code on GitHub
+
+```bash
+cd imbeval
+git init
+git add .
+git commit -m "Initial commit: imbeval v0.1.0"
+gh repo create imbeval --public --source=. --remote=origin
+git push -u origin main
+```
+
+(No `gh` CLI? Create the repo manually on github.com, then `git remote add origin <url>` and `git push -u origin main`.)
+
+## 3. Create a PyPI account and API token
+
+1. Register at https://pypi.org/account/register/ (and verify your email).
+2. Enable 2FA (PyPI requires it for publishing as of recent policy).
+3. Go to Account Settings → API tokens → "Add API token". Scope it to "Entire account" for your first upload (you can scope it to just this project after the first release).
+4. Save the token somewhere safe — it's shown once.
+
+It's good practice to also register on **TestPyPI** (https://test.pypi.org) first, to do a dry run without polluting the real index.
+
+## 4. Build the distribution files
+
+From the project root:
+
+```bash
+pip install --upgrade build twine
+python -m build
+```
+
+This creates a `dist/` folder with a `.tar.gz` (source distribution) and a `.whl`
+(wheel) — these are the files that actually get uploaded to PyPI.
+
+## 5. (Recommended) Upload to TestPyPI first
+
+```bash
+python -m twine upload --repository testpypi dist/*
+```
+
+It will prompt for a username (`__token__`) and password (your TestPyPI token).
+Then verify it installs cleanly:
+
+```bash
+pip install --index-url https://test.pypi.org/simple/ imbeval
+```
+
+## 6. Publish to the real PyPI
+
+```bash
+python -m twine upload dist/*
+```
+
+Username: `__token__`. Password: your PyPI API token (starts with `pypi-`).
+
+Once this succeeds, anyone in the world can run:
+
+```bash
+pip install imbeval
+```
+
+## 7. Tag the release on GitHub
+
+```bash
+git tag v0.1.0
+git push origin v0.1.0
+```
+
+Then create a "Release" on GitHub from that tag with short release notes — this is
+what shows up when people check "is this actively maintained."
+
+## 8. For every future update
+
+1. Make your changes, add/update tests, run `pytest` — all green before proceeding.
+2. Bump the version in `pyproject.toml` (and `__version__` in `__init__.py`) — follow [semantic versioning](https://semver.org/): patch (0.1.1) for fixes, minor (0.2.0) for new backward-compatible features, major (1.0.0) for breaking changes.
+3. Update `CHANGELOG.md`.
+4. `rm -rf dist/ && python -m build && python -m twine upload dist/*`
+5. Tag and release on GitHub as in step 7.
+
+## 9. Getting people to actually use it
+
+A license and a PyPI listing alone won't get adoption — that comes from people
+finding it useful and finding out it exists:
+
+- Write one clear, narrow blog post: "Why your imbalanced classifier's 99% accuracy is lying to you" — demonstrating the exact problem this library solves, ending with a 3-line code example.
+- Post it where the target audience already is: relevant subreddits (r/MachineLearning, r/datascience), a Show HN on Hacker News, relevant Discord/Slack ML communities.
+- Add a badge-rich README (build status, PyPI version, license) — people skim-judge trust from this in seconds. Shields.io provides free badges once you have CI set up.
+- Respond to every GitHub issue quickly for the first few months — early responsiveness is the single biggest driver of whether a library gets a second look.
+
+## Common first-publish mistakes to avoid
+
+- Forgetting to bump the version before re-uploading — PyPI rejects re-uploading the same version number, even for a typo fix.
+- Uploading secrets/API tokens accidentally committed in the repo — double check `git log` and use a `.gitignore` (see below) before your first push.
+- Publishing without a `README.md` that has a working code example — this is the #1 reason people bounce off a new package's PyPI page.

imbeval-0.1.0/docs/usage.md

@@ -0,0 +1,107 @@
+# Usage Guide
+
+## 1. The core workflow
+
+```python
+from sklearn.linear_model import LogisticRegression
+from imbeval import evaluation_report
+
+model = LogisticRegression().fit(X_train, y_train)
+y_pred_proba = model.predict_proba(X_test)[:, 1]
+
+report = evaluation_report(y_test, y_pred_proba, cost_fp=1, cost_fn=20)
+print(report["verdict"])
+```
+
+`evaluation_report` is the one function most people need. It is intentionally
+opinionated: it tells you in plain English whether your model has a problem,
+not just a pile of numbers to interpret yourself.
+
+## 2. Reading the minority-class report
+
+```python
+from imbeval import minority_class_report
+
+preds = (y_pred_proba >= 0.5).astype(int)
+report = minority_class_report(y_test, preds)
+print(report)
+# {'minority_label': 1, 'support': 48, 'precision': 0.62, 'recall': 0.41, 'f1': 0.5, ...}
+```
+
+If you already know which label is the minority/rare class, pass it explicitly:
+
+```python
+minority_class_report(y_test, preds, minority_label=1)
+```
+
+## 3. Checking calibration
+
+A model can have great accuracy and still be miscalibrated — e.g. it says
+"90% confident" but is only right 60% of the time at that confidence level.
+This matters a lot if you use the probability output for downstream decisions
+(e.g. ranking leads, setting alert priority).
+
+```python
+from imbeval import calibration_score, reliability_curve
+
+ece = calibration_score(y_test, y_pred_proba)
+print(f"Expected Calibration Error: {ece:.3f}")  # 0 = perfect, >0.1 = concerning
+
+curve = reliability_curve(y_test, y_pred_proba, n_bins=10)
+# Plot curve["bin_confidence"] vs curve["bin_accuracy"] for a reliability diagram
+```
+
+## 4. Picking a better threshold than 0.5
+
+The 0.5 default threshold assumes balanced classes and equal costs. Neither
+is usually true for imbalanced problems.
+
+```python
+from imbeval import optimal_threshold
+
+result = optimal_threshold(y_test, y_pred_proba)
+print(result)  # {'threshold': 0.27, 'score': 0.71}
+```
+
+## 5. Cost-sensitive thresholding (the most useful function for real decisions)
+
+If you know roughly what a false positive and a false negative cost your
+business, use this instead — it directly minimizes cost rather than an
+abstract metric.
+
+```python
+from imbeval import cost_sensitive_threshold
+
+# Example: fraud detection. A false alarm costs $1 in review time.
+# A missed fraud case costs $200 on average.
+result = cost_sensitive_threshold(y_test, y_pred_proba, cost_fp=1, cost_fn=200)
+print(result)
+# {'threshold': 0.08, 'total_cost': 1340.0, 'false_positives': 210, 'false_negatives': 6}
+```
+
+This typically pushes the threshold much lower than 0.5 when false negatives
+are expensive — which is the common case in fraud, medical screening, and
+safety-critical anomaly detection.
+
+## 6. Multi-class confidence breakdown
+
+```python
+from imbeval import per_class_confidence
+
+confidences = per_class_confidence(y_test, y_pred_proba_matrix, classes=["normal", "fraud", "abuse"])
+print(confidences)
+# {'normal': 0.91, 'fraud': 0.58, 'abuse': 0.44}
+```
+
+A low number for a specific class tells you the model is unsure specifically
+about that class, even if overall accuracy looks fine.
+
+## Common pitfalls this library is built to catch
+
+- **"99% accuracy" on a 1%-positive-rate dataset** — `minority_class_report`
+  exposes that this can mean the model just predicts the majority class always.
+- **High AUC, useless probabilities** — `calibration_score` catches this even
+  when ranking metrics look great.
+- **Using 0.5 by default out of habit** — `optimal_threshold` and
+  `cost_sensitive_threshold` replace that habit with a number backed by your
+  actual data and actual costs.

imbeval-0.1.0/examples/quickstart.py

@@ -0,0 +1,43 @@
+"""
+Quickstart example for imbeval.
+
+Run with: python examples/quickstart.py
+"""
+
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import make_classification
+
+from imbeval import evaluation_report
+
+# Simulate a realistic imbalanced dataset (5% positive class — e.g. fraud)
+X, y = make_classification(
+    n_samples=5000,
+    n_features=20,
+    weights=[0.95, 0.05],
+    flip_y=0.02,
+    random_state=42,
+)
+
+X_train, X_test, y_train, y_test = train_test_split(
+    X, y, test_size=0.3, stratify=y, random_state=42
+)
+
+model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
+y_pred_proba = model.predict_proba(X_test)[:, 1]
+
+report = evaluation_report(
+    y_test,
+    y_pred_proba,
+    cost_fp=1,      # cost of a false alarm
+    cost_fn=25,     # cost of a missed positive case
+)
+
+print("=" * 60)
+print("VERDICT:", report["verdict"])
+print("=" * 60)
+print("Minority class report:", report["minority_class"])
+print("Calibration error (ECE):", round(report["calibration_error"], 4))
+print("Optimal F1 threshold:", report["optimal_f1_threshold"])
+print("Cost-sensitive threshold:", report["cost_sensitive_threshold"])

imbeval-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "imbeval"
+version = "0.1.0"
+description = "Honest, production-readiness evaluation for imbalanced classification models."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.9"
+authors = [
+    { name = "Srikanth Sridhar", email = "srisrikanthtvs@gmail.com" }
+]
+keywords = ["machine-learning", "imbalanced-classification", "model-evaluation", "calibration", "threshold-tuning"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "numpy>=1.21",
+    "scikit-learn>=1.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/sricodings"
+Repository = "https://github.com/sricodings/imbeval"
+Issues = "https://github.com/sricodings/imbeval/issues"
+Documentation = "https://github.com/sricodings/imbeval#readme"
+
+[project.optional-dependencies]
+dev = ["pytest>=7.0", "build", "twine"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/imbeval"]

imbeval-0.1.0/src/imbeval/__init__.py

@@ -0,0 +1,27 @@
+"""
+imbeval — Honest evaluation for imbalanced classification models.
+
+Most metric libraries hand you numbers; they don't tell you whether your
+model is actually safe to ship on imbalanced data (fraud, medical, anomaly
+detection, churn, etc). imbeval combines per-class confidence, calibration
+quality, and cost-sensitive thresholding into one report so you can answer
+the real question: "is this model usable in production?"
+"""
+
+from .report import evaluation_report
+from .calibration import calibration_score, reliability_curve
+from .threshold import optimal_threshold, cost_sensitive_threshold
+from .metrics import per_class_confidence, minority_class_report
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "evaluation_report",
+    "calibration_score",
+    "reliability_curve",
+    "optimal_threshold",
+    "cost_sensitive_threshold",
+    "per_class_confidence",
+    "minority_class_report",
+    "__version__",
+]

imbeval-0.1.0/src/imbeval/calibration.py

@@ -0,0 +1,66 @@
+"""Calibration quality checks — is the model's confidence trustworthy?"""
+
+from __future__ import annotations
+
+import numpy as np
+
+
+def reliability_curve(y_true, y_pred_proba, n_bins: int = 10):
+    """
+    Bin predictions by confidence and compare to observed accuracy in
+    each bin. Returns arrays suitable for plotting a reliability diagram.
+
+    Parameters
+    ----------
+    y_true : array-like, binary (0/1) ground truth.
+    y_pred_proba : array-like, predicted probability of the positive class.
+    n_bins : int
+
+    Returns
+    -------
+    dict with keys: bin_confidence, bin_accuracy, bin_count
+    """
+    y_true = np.asarray(y_true)
+    y_pred_proba = np.asarray(y_pred_proba)
+
+    bin_edges = np.linspace(0.0, 1.0, n_bins + 1)
+    bin_confidence, bin_accuracy, bin_count = [], [], []
+
+    for i in range(n_bins):
+        lo, hi = bin_edges[i], bin_edges[i + 1]
+        mask = (y_pred_proba >= lo) & (y_pred_proba < hi if i < n_bins - 1 else y_pred_proba <= hi)
+        count = int(mask.sum())
+        bin_count.append(count)
+        if count == 0:
+            bin_confidence.append(None)
+            bin_accuracy.append(None)
+        else:
+            bin_confidence.append(float(np.mean(y_pred_proba[mask])))
+            bin_accuracy.append(float(np.mean(y_true[mask])))
+
+    return {
+        "bin_confidence": bin_confidence,
+        "bin_accuracy": bin_accuracy,
+        "bin_count": bin_count,
+    }
+
+
+def calibration_score(y_true, y_pred_proba, n_bins: int = 10) -> float:
+    """
+    Expected Calibration Error (ECE): the weighted average gap between
+    predicted confidence and observed accuracy across bins. Lower is
+    better; 0 is perfect calibration.
+    """
+    curve = reliability_curve(y_true, y_pred_proba, n_bins=n_bins)
+    total = sum(c for c in curve["bin_count"] if c)
+    if total == 0:
+        return 0.0
+
+    ece = 0.0
+    for conf, acc, count in zip(
+        curve["bin_confidence"], curve["bin_accuracy"], curve["bin_count"]
+    ):
+        if count == 0:
+            continue
+        ece += (count / total) * abs(conf - acc)
+    return float(ece)

imbeval-0.1.0/src/imbeval/metrics.py

@@ -0,0 +1,87 @@
+"""Per-class confidence and minority-class focused metrics."""
+
+from __future__ import annotations
+
+import numpy as np
+from sklearn.metrics import (
+    precision_score,
+    recall_score,
+    f1_score,
+    confusion_matrix,
+)
+
+
+def per_class_confidence(y_true, y_pred_proba, classes=None):
+    """
+    Compute the mean predicted-probability "confidence" the model assigns
+    to the correct class, broken down per class.
+
+    Parameters
+    ----------
+    y_true : array-like of shape (n_samples,)
+        True integer class labels (0..n_classes-1).
+    y_pred_proba : array-like of shape (n_samples, n_classes)
+        Predicted probabilities from `model.predict_proba`.
+    classes : list, optional
+        Labels for each class index, for display purposes.
+
+    Returns
+    -------
+    dict
+        Mapping of class label -> mean confidence on correctly-attributed
+        probability mass for samples truly belonging to that class.
+    """
+    y_true = np.asarray(y_true)
+    y_pred_proba = np.asarray(y_pred_proba)
+    n_classes = y_pred_proba.shape[1]
+    if classes is None:
+        classes = list(range(n_classes))
+
+    result = {}
+    for idx, label in enumerate(classes):
+        mask = y_true == idx
+        if mask.sum() == 0:
+            result[label] = None
+            continue
+        result[label] = float(np.mean(y_pred_proba[mask, idx]))
+    return result
+
+
+def minority_class_report(y_true, y_pred, classes=None, minority_label=None):
+    """
+    Precision/recall/F1 with explicit emphasis on the minority class.
+
+    If `minority_label` is not given, the class with the lowest support
+    in `y_true` is auto-detected.
+
+    Returns
+    -------
+    dict with keys: minority_label, support, precision, recall, f1,
+    confusion_matrix
+    """
+    y_true = np.asarray(y_true)
+    y_pred = np.asarray(y_pred)
+
+    labels, counts = np.unique(y_true, return_counts=True)
+    if minority_label is None:
+        minority_label = labels[np.argmin(counts)]
+
+    precision = precision_score(
+        y_true, y_pred, labels=[minority_label], average="macro", zero_division=0
+    )
+    recall = recall_score(
+        y_true, y_pred, labels=[minority_label], average="macro", zero_division=0
+    )
+    f1 = f1_score(
+        y_true, y_pred, labels=[minority_label], average="macro", zero_division=0
+    )
+    support = int(np.sum(y_true == minority_label))
+
+    return {
+        "minority_label": minority_label,
+        "support": support,
+        "precision": float(precision),
+        "recall": float(recall),
+        "f1": float(f1),
+        "confusion_matrix": confusion_matrix(y_true, y_pred).tolist(),
+    }

imbeval-0.1.0/src/imbeval/report.py

@@ -0,0 +1,80 @@
+"""The single entry point: one honest report on production-readiness."""
+
+from __future__ import annotations
+
+import numpy as np
+
+from .calibration import calibration_score
+from .threshold import optimal_threshold, cost_sensitive_threshold
+from .metrics import minority_class_report
+
+
+def evaluation_report(
+    y_true,
+    y_pred_proba,
+    cost_fp: float = None,
+    cost_fn: float = None,
+    n_bins: int = 10,
+):
+    """
+    Produce one combined evaluation report for a binary classifier on
+    imbalanced data: minority-class performance, calibration quality,
+    a tuned decision threshold, and (optionally) a cost-aware threshold.
+
+    Parameters
+    ----------
+    y_true : array-like, binary ground truth labels (0/1).
+    y_pred_proba : array-like, predicted probability of the positive (1) class.
+    cost_fp : float, optional. Business cost of one false positive.
+    cost_fn : float, optional. Business cost of one false negative.
+        If both cost_fp and cost_fn are given, a cost-sensitive threshold
+        is included in the report.
+    n_bins : int, bins used for calibration scoring.
+
+    Returns
+    -------
+    dict
+        {
+          "minority_class": {...},
+          "calibration_error": float,
+          "default_threshold_0.5": {...},
+          "optimal_f1_threshold": {...},
+          "cost_sensitive_threshold": {...} or None,
+          "verdict": str
+        }
+    """
+    y_true = np.asarray(y_true)
+    y_pred_proba = np.asarray(y_pred_proba)
+    preds_at_half = (y_pred_proba >= 0.5).astype(int)
+
+    minority = minority_class_report(y_true, preds_at_half)
+    ece = calibration_score(y_true, y_pred_proba, n_bins=n_bins)
+    f1_opt = optimal_threshold(y_true, y_pred_proba)
+
+    cost_result = None
+    if cost_fp is not None and cost_fn is not None:
+        cost_result = cost_sensitive_threshold(y_true, y_pred_proba, cost_fp, cost_fn)
+
+    verdict = _build_verdict(minority, ece, f1_opt)
+
+    return {
+        "minority_class": minority,
+        "calibration_error": ece,
+        "optimal_f1_threshold": f1_opt,
+        "cost_sensitive_threshold": cost_result,
+        "verdict": verdict,
+    }
+
+
+def _build_verdict(minority: dict, ece: float, f1_opt: dict) -> str:
+    flags = []
+    if minority["recall"] < 0.5:
+        flags.append("minority-class recall is below 50% at the default 0.5 threshold")
+    if ece > 0.1:
+        flags.append(f"calibration error is high (ECE={ece:.3f}); confidence scores are not trustworthy")
+    if f1_opt["score"] - minority["f1"] > 0.15:
+        flags.append("default 0.5 threshold is far from optimal; consider using optimal_f1_threshold")
+
+    if not flags:
+        return "Looks production-ready on the dimensions checked. Validate further on a held-out set."
+    return "Not yet production-ready: " + "; ".join(flags) + "."

imbeval-0.1.0/src/imbeval/threshold.py

@@ -0,0 +1,83 @@
+"""Decision threshold tuning for imbalanced problems."""
+
+from __future__ import annotations
+
+import numpy as np
+from sklearn.metrics import f1_score
+
+
+def optimal_threshold(y_true, y_pred_proba, metric: str = "f1", n_steps: int = 200):
+    """
+    Sweep decision thresholds and return the one that maximizes the
+    chosen metric. Default metric is 0.5-agnostic F1, which is usually
+    a far better default than 0.5 on imbalanced data.
+
+    Parameters
+    ----------
+    y_true : array-like, binary ground truth.
+    y_pred_proba : array-like, predicted probability of positive class.
+    metric : {"f1"} currently supported.
+    n_steps : int, number of thresholds to test between 0 and 1.
+
+    Returns
+    -------
+    dict with keys: threshold, score
+    """
+    y_true = np.asarray(y_true)
+    y_pred_proba = np.asarray(y_pred_proba)
+
+    thresholds = np.linspace(0.01, 0.99, n_steps)
+    best_threshold, best_score = 0.5, -1.0
+
+    for t in thresholds:
+        preds = (y_pred_proba >= t).astype(int)
+        if metric == "f1":
+            score = f1_score(y_true, preds, zero_division=0)
+        else:
+            raise ValueError(f"Unsupported metric: {metric}")
+        if score > best_score:
+            best_score, best_threshold = score, t
+
+    return {"threshold": float(best_threshold), "score": float(best_score)}
+
+
+def cost_sensitive_threshold(y_true, y_pred_proba, cost_fp: float, cost_fn: float, n_steps: int = 200):
+    """
+    Find the decision threshold that minimizes total business cost,
+    given the real-world cost of a false positive vs a false negative.
+
+    This is usually what people actually want on imbalanced data
+    (e.g. fraud: missing fraud is far costlier than a false alarm).
+
+    Parameters
+    ----------
+    y_true : array-like, binary ground truth.
+    y_pred_proba : array-like, predicted probability of positive class.
+    cost_fp : float, cost incurred per false positive.
+    cost_fn : float, cost incurred per false negative.
+    n_steps : int
+
+    Returns
+    -------
+    dict with keys: threshold, total_cost, false_positives, false_negatives
+    """
+    y_true = np.asarray(y_true)
+    y_pred_proba = np.asarray(y_pred_proba)
+
+    thresholds = np.linspace(0.01, 0.99, n_steps)
+    best = {"threshold": 0.5, "total_cost": float("inf"), "false_positives": 0, "false_negatives": 0}
+
+    for t in thresholds:
+        preds = (y_pred_proba >= t).astype(int)
+        fp = int(np.sum((preds == 1) & (y_true == 0)))
+        fn = int(np.sum((preds == 0) & (y_true == 1)))
+        total_cost = fp * cost_fp + fn * cost_fn
+        if total_cost < best["total_cost"]:
+            best = {
+                "threshold": float(t),
+                "total_cost": float(total_cost),
+                "false_positives": fp,
+                "false_negatives": fn,
+            }
+
+    return best

imbeval-0.1.0/tests/test_imbeval.py

@@ -0,0 +1,94 @@
+import numpy as np
+import pytest
+
+from imbeval import (
+    evaluation_report,
+    calibration_score,
+    reliability_curve,
+    optimal_threshold,
+    cost_sensitive_threshold,
+    per_class_confidence,
+    minority_class_report,
+)
+
+
+@pytest.fixture
+def imbalanced_binary_data():
+    rng = np.random.default_rng(42)
+    n = 1000
+    y_true = np.zeros(n, dtype=int)
+    y_true[:50] = 1  # 5% minority class
+    rng.shuffle(y_true)
+
+    # Simulate a decently-calibrated model: higher proba for true positives
+    base = rng.uniform(0, 0.3, size=n)
+    y_pred_proba = np.where(y_true == 1, base + rng.uniform(0.3, 0.6, size=n), base)
+    y_pred_proba = np.clip(y_pred_proba, 0, 1)
+    return y_true, y_pred_proba
+
+
+def test_calibration_score_perfect_calibration():
+    rng = np.random.default_rng(0)
+    proba = rng.uniform(0, 1, 5000)
+    y_true = (rng.uniform(0, 1, 5000) < proba).astype(int)
+    ece = calibration_score(y_true, proba, n_bins=10)
+    assert ece < 0.08  # roughly well calibrated by construction
+
+
+def test_reliability_curve_shape():
+    y_true = [0, 1, 0, 1, 1, 0]
+    proba = [0.1, 0.9, 0.2, 0.8, 0.7, 0.3]
+    curve = reliability_curve(y_true, proba, n_bins=5)
+    assert len(curve["bin_confidence"]) == 5
+    assert len(curve["bin_accuracy"]) == 5
+    assert len(curve["bin_count"]) == 5
+    assert sum(c for c in curve["bin_count"]) == len(y_true)
+
+
+def test_optimal_threshold_beats_or_matches_default(imbalanced_binary_data):
+    y_true, y_pred_proba = imbalanced_binary_data
+    result = optimal_threshold(y_true, y_pred_proba)
+    assert 0.0 < result["threshold"] < 1.0
+    assert 0.0 <= result["score"] <= 1.0
+
+
+def test_cost_sensitive_threshold_favors_recall_when_fn_costly(imbalanced_binary_data):
+    y_true, y_pred_proba = imbalanced_binary_data
+    cheap_fp = cost_sensitive_threshold(y_true, y_pred_proba, cost_fp=1, cost_fn=1)
+    costly_fn = cost_sensitive_threshold(y_true, y_pred_proba, cost_fp=1, cost_fn=50)
+    # Penalizing false negatives heavily should push the threshold down
+    assert costly_fn["threshold"] <= cheap_fp["threshold"]
+
+
+def test_per_class_confidence_basic():
+    y_true = [0, 0, 1, 1]
+    proba = np.array([[0.9, 0.1], [0.6, 0.4], [0.3, 0.7], [0.2, 0.8]])
+    result = per_class_confidence(y_true, proba, classes=["neg", "pos"])
+    assert result["neg"] == pytest.approx(0.75)
+    assert result["pos"] == pytest.approx(0.75)
+
+
+def test_minority_class_report_autodetects_minority(imbalanced_binary_data):
+    y_true, y_pred_proba = imbalanced_binary_data
+    preds = (y_pred_proba >= 0.5).astype(int)
+    report = minority_class_report(y_true, preds)
+    assert report["minority_label"] == 1
+    assert report["support"] == int(np.sum(y_true == 1))
+    assert 0.0 <= report["precision"] <= 1.0
+    assert 0.0 <= report["recall"] <= 1.0
+
+
+def test_evaluation_report_end_to_end(imbalanced_binary_data):
+    y_true, y_pred_proba = imbalanced_binary_data
+    report = evaluation_report(y_true, y_pred_proba, cost_fp=1, cost_fn=20)
+    assert "minority_class" in report
+    assert "calibration_error" in report
+    assert "optimal_f1_threshold" in report
+    assert report["cost_sensitive_threshold"] is not None
+    assert isinstance(report["verdict"], str)
+
+
+def test_evaluation_report_without_costs(imbalanced_binary_data):
+    y_true, y_pred_proba = imbalanced_binary_data
+    report = evaluation_report(y_true, y_pred_proba)
+    assert report["cost_sensitive_threshold"] is None