polarbearings 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- polarbearings-0.2.2/LICENSE +21 -0
- polarbearings-0.2.2/PKG-INFO +314 -0
- polarbearings-0.2.2/README.md +288 -0
- polarbearings-0.2.2/pyproject.toml +250 -0
- polarbearings-0.2.2/src/polarbearings/__init__.py +136 -0
- polarbearings-0.2.2/src/polarbearings/_common.py +242 -0
- polarbearings-0.2.2/src/polarbearings/average_precision.py +114 -0
- polarbearings-0.2.2/src/polarbearings/bootstrap.py +602 -0
- polarbearings-0.2.2/src/polarbearings/brier_score.py +65 -0
- polarbearings-0.2.2/src/polarbearings/calibration.py +457 -0
- polarbearings-0.2.2/src/polarbearings/class_weight.py +84 -0
- polarbearings-0.2.2/src/polarbearings/classification.py +530 -0
- polarbearings-0.2.2/src/polarbearings/confusion_curve.py +326 -0
- polarbearings-0.2.2/src/polarbearings/curves.py +302 -0
- polarbearings-0.2.2/src/polarbearings/gini.py +155 -0
- polarbearings-0.2.2/src/polarbearings/log_loss.py +72 -0
- polarbearings-0.2.2/src/polarbearings/py.typed +0 -0
- polarbearings-0.2.2/src/polarbearings/ranking.py +139 -0
- polarbearings-0.2.2/src/polarbearings/regression.py +743 -0
- polarbearings-0.2.2/src/polarbearings/roc_auc.py +137 -0
- polarbearings-0.2.2/src/polarbearings/split.py +277 -0
- polarbearings-0.2.2/src/polarbearings/thresholds.py +146 -0
- polarbearings-0.2.2/tests/conftest.py +39 -0
- polarbearings-0.2.2/tests/test_average_precision.py +216 -0
- polarbearings-0.2.2/tests/test_bootstrap.py +554 -0
- polarbearings-0.2.2/tests/test_brier_score.py +115 -0
- polarbearings-0.2.2/tests/test_calibration.py +340 -0
- polarbearings-0.2.2/tests/test_class_weight.py +140 -0
- polarbearings-0.2.2/tests/test_classification.py +874 -0
- polarbearings-0.2.2/tests/test_confusion_curve.py +296 -0
- polarbearings-0.2.2/tests/test_confusion_matrix.py +246 -0
- polarbearings-0.2.2/tests/test_curves.py +356 -0
- polarbearings-0.2.2/tests/test_d2_scores.py +256 -0
- polarbearings-0.2.2/tests/test_degenerate_inputs.py +169 -0
- polarbearings-0.2.2/tests/test_gini.py +268 -0
- polarbearings-0.2.2/tests/test_jaccard.py +171 -0
- polarbearings-0.2.2/tests/test_log_loss.py +180 -0
- polarbearings-0.2.2/tests/test_missing_values.py +120 -0
- polarbearings-0.2.2/tests/test_ndcg.py +175 -0
- polarbearings-0.2.2/tests/test_null_policy.py +172 -0
- polarbearings-0.2.2/tests/test_pos_label.py +129 -0
- polarbearings-0.2.2/tests/test_regression.py +1189 -0
- polarbearings-0.2.2/tests/test_roc_auc.py +265 -0
- polarbearings-0.2.2/tests/test_split.py +399 -0
- polarbearings-0.2.2/tests/test_thresholds.py +151 -0
- polarbearings-0.2.2/tests/test_tweedie_deviance.py +266 -0
- polarbearings-0.2.2/tests/test_weight_expression.py +73 -0
- polarbearings-0.2.2/tests/test_weights.py +366 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 Polarbearings Contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,314 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: polarbearings
|
|
3
|
+
Version: 0.2.2
|
|
4
|
+
Summary: High-performance machine learning metrics for Polars DataFrames
|
|
5
|
+
Keywords: polars,metrics,machine-learning,data-science,roc-auc,classification,regression
|
|
6
|
+
Author: Jeremy Heffner
|
|
7
|
+
Author-email: Jeremy Heffner <3886031+jerheff@users.noreply.github.com>
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Intended Audience :: Science/Research
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
18
|
+
Classifier: Topic :: Scientific/Engineering
|
|
19
|
+
Classifier: Typing :: Typed
|
|
20
|
+
Requires-Dist: polars>=1.0.0
|
|
21
|
+
Requires-Python: >=3.11
|
|
22
|
+
Project-URL: Homepage, https://github.com/jerheff/polarbearings
|
|
23
|
+
Project-URL: Repository, https://github.com/jerheff/polarbearings
|
|
24
|
+
Project-URL: Issues, https://github.com/jerheff/polarbearings/issues
|
|
25
|
+
Description-Content-Type: text/markdown
|
|
26
|
+
|
|
27
|
+
# Polarbearings ๐ปโโ๏ธ๐งญ
|
|
28
|
+
|
|
29
|
+
High-performance machine learning metrics implemented as native Polars expressions.
|
|
30
|
+
|
|
31
|
+
## Features
|
|
32
|
+
|
|
33
|
+
- **Fast where it counts**: Native Polars expressions โ large wins on grouped and probabilistic metrics (see [Performance](#performance))
|
|
34
|
+
- **Weighted**: Optional sample weights on nearly every metric
|
|
35
|
+
- **Flexible labels**: Any positive class โ `1`, `100`, `"cancer"`, `True` โ via `pos_label`
|
|
36
|
+
- **Correct**: scikit-learn-faithful, verified with property-based testing
|
|
37
|
+
- **Composable**: Plain Polars expressions โ drop into `group_by`, `over`, and lazy pipelines
|
|
38
|
+
- **Type-safe**: Full type hints
|
|
39
|
+
|
|
40
|
+
## Installation
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
uv add polarbearings
|
|
44
|
+
# or: pip install polarbearings
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
The only runtime dependency is Polars.
|
|
48
|
+
|
|
49
|
+
## Quick Start
|
|
50
|
+
|
|
51
|
+
```python
|
|
52
|
+
import polars as pl
|
|
53
|
+
from polarbearings import roc_auc
|
|
54
|
+
|
|
55
|
+
df = pl.DataFrame({
|
|
56
|
+
"label": [0, 0, 1, 1, 1],
|
|
57
|
+
"score": [0.1, 0.4, 0.35, 0.8, 0.9]
|
|
58
|
+
})
|
|
59
|
+
|
|
60
|
+
# Every metric is a Polars expression โ use it anywhere an expression is allowed.
|
|
61
|
+
df.select(roc_auc("label", "score"))
|
|
62
|
+
# shape: (1, 1)
|
|
63
|
+
# โโโโโโโโโโโโโโโโโโโโโโโ
|
|
64
|
+
# โ roc_auc_label_score โ
|
|
65
|
+
# โ f64 โ
|
|
66
|
+
# โโโโโโโโโโโโโโโโโโโโโโโก
|
|
67
|
+
# โ 0.833333 โ
|
|
68
|
+
# โโโโโโโโโโโโโโโโโโโโโโโ
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Why Polarbearings
|
|
72
|
+
|
|
73
|
+
Every metric is a *pure Polars expression*. That single design choice is where the
|
|
74
|
+
strengths come from โ things a scikit-learn wrapper or a compiled plugin can't
|
|
75
|
+
easily match:
|
|
76
|
+
|
|
77
|
+
- **Group-wise metrics at scale** โ metrics drop straight into `group_by().agg()`
|
|
78
|
+
and Polars parallelizes across groups (15โ65x faster than a Python loop calling
|
|
79
|
+
scikit-learn per segment).
|
|
80
|
+
- **A whole metric suite in one pass** โ bundle every metric into a single
|
|
81
|
+
`df.select([...])`; Polars shares the column scans and parallelizes across the
|
|
82
|
+
independent outputs. 13 metrics on 10M rows run in **~1.05 s vs scikit-learn's
|
|
83
|
+
~6.1 s (5.8x)** โ see [Performance](#performance).
|
|
84
|
+
- **Sample weights almost everywhere** โ nearly every metric accepts an optional
|
|
85
|
+
`weight` column, including ROC AUC, log loss, MCC, and Cohen's kappa (a few
|
|
86
|
+
exceptions are noted per metric).
|
|
87
|
+
- **Any positive class** โ `pos_label` accepts `1`, `100`, `"cancer"`, `True`, โฆ
|
|
88
|
+
no remapping your labels to 0/1.
|
|
89
|
+
- **scikit-learn-faithful** โ names and edge-case semantics mirror scikit-learn
|
|
90
|
+
(e.g. `null` for undefined cases), so migration is ~1:1.
|
|
91
|
+
- **Composable & lazy-friendly** โ pure expressions fold into lazy query plans,
|
|
92
|
+
`over()` windows, and the rest of your Polars pipeline.
|
|
93
|
+
- **Zero build, one dependency** โ pure Python emitting Polars expressions; no
|
|
94
|
+
compiled extension, installs anywhere Polars runs, supports Polars 1.0.0+.
|
|
95
|
+
|
|
96
|
+
### A whole evaluation report in one `select`
|
|
97
|
+
|
|
98
|
+
Because every metric is just an expression, a full report is one `df.select(...)`
|
|
99
|
+
โ Polars reads each column once and fans the work across the independent outputs:
|
|
100
|
+
|
|
101
|
+
<!--- invisible-code-block: python
|
|
102
|
+
df = pl.DataFrame({
|
|
103
|
+
"label": [0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0],
|
|
104
|
+
"prob": [0.2, 0.45, 0.6, 0.55, 0.75, 0.35, 0.3, 0.8, 0.65, 0.5, 0.7, 0.4],
|
|
105
|
+
})
|
|
106
|
+
--->
|
|
107
|
+
|
|
108
|
+
```python
|
|
109
|
+
from polarbearings import (
|
|
110
|
+
precision, recall, f1_score, roc_auc, average_precision,
|
|
111
|
+
log_loss, brier_score, confusion_matrix,
|
|
112
|
+
)
|
|
113
|
+
|
|
114
|
+
df.select(
|
|
115
|
+
precision("label", "prob"),
|
|
116
|
+
recall("label", "prob"),
|
|
117
|
+
f1_score("label", "prob"),
|
|
118
|
+
roc_auc("label", "prob"),
|
|
119
|
+
average_precision("label", "prob"),
|
|
120
|
+
log_loss("label", "prob"),
|
|
121
|
+
brier_score("label", "prob"),
|
|
122
|
+
confusion_matrix("label", "prob"), # struct {threshold, tp, fp, fn, tn}
|
|
123
|
+
)
|
|
124
|
+
# One tidy row, one column per metric (8 columns; abbreviated):
|
|
125
|
+
# shape: (1, 8)
|
|
126
|
+
# โโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
127
|
+
# โ precision_label_prob_0.5 โ recall_label_prob_0.5 โ โฆ โ brier_score_label_prob โ
|
|
128
|
+
# โ f64 โ f64 โ โ f64 โ
|
|
129
|
+
# โโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโโโก
|
|
130
|
+
# โ 0.714286 โ 0.833333 โ โฆ โ 0.161944 โ
|
|
131
|
+
# โโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
`benchmarks/bench_multi_metric.py` reproduces this timing across sizes from 100 to
|
|
135
|
+
1M rows.
|
|
136
|
+
|
|
137
|
+
## Metrics
|
|
138
|
+
|
|
139
|
+
~40 metrics across ranking, probabilistic, classification, calibration, and
|
|
140
|
+
regression families, plus curve generators, bootstrap CIs, deterministic splitting,
|
|
141
|
+
and threshold utilities. The **[full metrics reference โ
|
|
142
|
+
docs/guides/METRICS.md](https://github.com/jerheff/polarbearings/blob/main/docs/guides/METRICS.md)** documents every metric's
|
|
143
|
+
semantics, edge cases, and scikit-learn correspondence.
|
|
144
|
+
|
|
145
|
+
| Family | What's in it |
|
|
146
|
+
| --- | --- |
|
|
147
|
+
| **Ranking** | `roc_auc`, `average_precision`, `gini_coefficient`, `dcg_score`, `ndcg_score` |
|
|
148
|
+
| **Probabilistic** | `log_loss`, `brier_score` |
|
|
149
|
+
| **Classification** | `precision`, `recall`, `f1_score`, `fbeta_score`, `specificity`, `accuracy`, `balanced_accuracy`, `matthews_corrcoef`, `cohens_kappa`, `jaccard_score`, `confusion_matrix` |
|
|
150
|
+
| **Thresholds** | `threshold_sweep`, `quantiles` / `equal_width` / `linspace` specs |
|
|
151
|
+
| **Curves** | `roc_curve`, `pr_curve`, `det_curve`, `expected_cost`, `confusion_curve` |
|
|
152
|
+
| **Regression** | `mae`, `mse`, `rmse`, `r2_score`, `mape`, `smape`, MSLE/RMSLE, `huber_loss`, `log_cosh_loss`, pinball, Tweedie/Poisson/gamma deviance, Dยฒ scores, `max_error`, `median_absolute_error` |
|
|
153
|
+
| **Calibration** | `calibration_curve`, `expected_calibration_error` (ECE), `maximum_calibration_error` (MCE) |
|
|
154
|
+
| **Weights & CIs** | `balanced_sample_weight`, `balanced_class_weights`, `bootstrap_ci`, `bootstrap_weight` |
|
|
155
|
+
| **Splitting** | `hash_split`, `hash_splits`, `hash_fold`, `hash_uniform` |
|
|
156
|
+
|
|
157
|
+
<!--- invisible-code-block: python
|
|
158
|
+
reg = pl.DataFrame({
|
|
159
|
+
"y": [1.0, 2.0, 3.0, 4.0, 5.0],
|
|
160
|
+
"pred": [1.1, 1.9, 3.2, 3.8, 5.5],
|
|
161
|
+
"w": [1.0, 1.0, 1.0, 2.0, 1.0],
|
|
162
|
+
})
|
|
163
|
+
--->
|
|
164
|
+
|
|
165
|
+
```python
|
|
166
|
+
from polarbearings import roc_auc, f1_score, mae, calibration_curve
|
|
167
|
+
|
|
168
|
+
# Classification + probabilistic metrics are plain expressions:
|
|
169
|
+
df.select(roc_auc("label", "prob"), f1_score("label", "prob", threshold=0.7))
|
|
170
|
+
|
|
171
|
+
# Regression metrics too:
|
|
172
|
+
reg.select(mae("y", "pred", weight="w"))
|
|
173
|
+
|
|
174
|
+
# Curve helpers take a (Lazy)Frame and return a plot-ready LazyFrame:
|
|
175
|
+
calibration_curve(df, "label", "prob", n_bins=10, strategy="quantile").collect()
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
### Cross-cutting behaviour
|
|
179
|
+
|
|
180
|
+
Four behaviours are shared by the metrics rather than specific to one โ full
|
|
181
|
+
details in the [metrics reference](https://github.com/jerheff/polarbearings/blob/main/docs/guides/METRICS.md#cross-cutting-behaviour):
|
|
182
|
+
|
|
183
|
+
- **Output names** โ each expression is pre-aliased `<metric>_<target>_<score>`
|
|
184
|
+
(plus suffixes for a weight, non-default `pos_label`, or threshold), so
|
|
185
|
+
`roc_auc("label", "score")` yields a `roc_auc_label_score` column. Chain
|
|
186
|
+
`.alias("auc")` to rename it.
|
|
187
|
+
- **Sample weights** โ pass `weight="col"` to nearly any metric. Six omit it
|
|
188
|
+
(`dcg_score`, `ndcg_score`, `max_error`, `median_absolute_error`,
|
|
189
|
+
`d2_absolute_error_score`, `d2_pinball_score`) where a weighted form is undefined
|
|
190
|
+
or not cleanly expressible.
|
|
191
|
+
- **Custom positive class** โ `pos_label` accepts ints, strings, or booleans; no
|
|
192
|
+
remapping to 0/1.
|
|
193
|
+
- **Missing values** โ any `null`/`NaN` in `target`, score, or `weight` makes a
|
|
194
|
+
metric return `null` (loud, not silent), scoped to the evaluation context. Drop
|
|
195
|
+
rows yourself for complete-case behaviour. Curve helpers are the exception โ they
|
|
196
|
+
drop incomplete rows.
|
|
197
|
+
|
|
198
|
+
### Diagnostic plots
|
|
199
|
+
|
|
200
|
+
`notebooks/diagnostics.ipynb` gives examples of ROC/PR/DET/calibration curves and bootstrap
|
|
201
|
+
confidence bands with these helpers and visualizing them with Plotly.
|
|
202
|
+
|
|
203
|
+
## Use Cases
|
|
204
|
+
|
|
205
|
+
Polarbearings is a good fit for:
|
|
206
|
+
|
|
207
|
+
1. **Large-scale model evaluation** โ millions of predictions, efficiently
|
|
208
|
+
2. **Group-wise metrics** โ per-segment analysis via Polars' `group_by`
|
|
209
|
+
3. **Streaming / out-of-core** โ lazy expressions over `LazyFrame`s
|
|
210
|
+
4. **Composed evaluation reports** โ a whole metric suite in one pass
|
|
211
|
+
|
|
212
|
+
<!--- invisible-code-block: python
|
|
213
|
+
df = pl.DataFrame({
|
|
214
|
+
"segment": ["A", "A", "B", "B"],
|
|
215
|
+
"label": [0, 1, 0, 1],
|
|
216
|
+
"score": [0.1, 0.9, 0.5, 0.5],
|
|
217
|
+
})
|
|
218
|
+
--->
|
|
219
|
+
|
|
220
|
+
```python
|
|
221
|
+
from polarbearings import roc_auc
|
|
222
|
+
|
|
223
|
+
# ROC AUC per customer segment, in one parallelized pass:
|
|
224
|
+
df.group_by("segment").agg(roc_auc("label", "score"))
|
|
225
|
+
# shape: (2, 2)
|
|
226
|
+
# โโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโ
|
|
227
|
+
# โ segment โ roc_auc_label_score โ
|
|
228
|
+
# โโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโก
|
|
229
|
+
# โ A โ 1.0 โ
|
|
230
|
+
# โ B โ 0.5 โ
|
|
231
|
+
# โโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโ
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
## Performance
|
|
235
|
+
|
|
236
|
+
Polarbearings runs every metric as a native Polars expression. The advantage is
|
|
237
|
+
**large and real where there's work to parallelize, and honest where there
|
|
238
|
+
isn't.** Numbers below are speedup vs scikit-learn (scikit-learn time รท
|
|
239
|
+
polarbearings time; higher = faster), median of clean benchmark runs.
|
|
240
|
+
|
|
241
|
+
**Where polarbearings wins big** โ grouped, probabilistic, and ranking metrics.
|
|
242
|
+
Ranges span polars 1.0.0 and 1.41.2; see
|
|
243
|
+
[PERFORMANCE.md](https://github.com/jerheff/polarbearings/blob/main/docs/technical/PERFORMANCE.md) for the per-version, per-size matrix.
|
|
244
|
+
|
|
245
|
+
| Metric | 100k rows | 10M rows |
|
|
246
|
+
|--------|:---:|:---:|
|
|
247
|
+
| Grouped metrics (per segment) | **~15โ60x** | โ |
|
|
248
|
+
| Precision / F1 | ~5โ22x | ~14โ23x |
|
|
249
|
+
| Brier Score | ~9โ10x | ~7โ8x |
|
|
250
|
+
| Log Loss | ~5โ6x | ~4โ5x |
|
|
251
|
+
| ROC AUC | ~4.5โ6x | ~5x |
|
|
252
|
+
| 13 metrics in one `select` | โ | **5.8x** |
|
|
253
|
+
|
|
254
|
+
**Where it's at parity or slower** โ trivial reductions (MAE, MSE, MAPE, Rยฒ) are
|
|
255
|
+
roughly even with scikit-learn at small-to-mid sizes and *slower* on a single
|
|
256
|
+
very large array, where NumPy's tight single-threaded loop beats one Polars
|
|
257
|
+
expression. Reach for polarbearings on grouped/composed pipelines and the
|
|
258
|
+
probabilistic metrics; for a one-off MAE over a giant array, NumPy is fine.
|
|
259
|
+
|
|
260
|
+
See [docs/technical/PERFORMANCE.md](https://github.com/jerheff/polarbearings/blob/main/docs/technical/PERFORMANCE.md) for the full
|
|
261
|
+
per-metric breakdown, the size-scaling curve, and the ceteris-paribus Polars
|
|
262
|
+
version comparison.
|
|
263
|
+
|
|
264
|
+
## Development
|
|
265
|
+
|
|
266
|
+
This project uses [uv](https://github.com/astral-sh/uv) for dependency management
|
|
267
|
+
and [just](https://github.com/casey/just) for task running. Run `just` to list all
|
|
268
|
+
recipes.
|
|
269
|
+
|
|
270
|
+
```bash
|
|
271
|
+
uv sync --all-groups # install dependencies
|
|
272
|
+
just test # run tests (or: uv run pytest)
|
|
273
|
+
just quality # lint + type-check
|
|
274
|
+
just check # fast local check: lint + type-check + tests (not full CI)
|
|
275
|
+
just test-compat # test against min / mid / latest Polars
|
|
276
|
+
just bench # benchmarks vs scikit-learn
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
Testing combines unit tests, property-based tests (Hypothesis), and
|
|
280
|
+
scikit-learn compatibility tests across multiple Polars versions โ see
|
|
281
|
+
[docs/guides/TESTING.md](https://github.com/jerheff/polarbearings/blob/main/docs/guides/TESTING.md).
|
|
282
|
+
|
|
283
|
+
## Requirements
|
|
284
|
+
|
|
285
|
+
- **Python**: 3.11+
|
|
286
|
+
- **Polars**: 1.0.0+
|
|
287
|
+
|
|
288
|
+
## Roadmap
|
|
289
|
+
|
|
290
|
+
- [x] Publish to PyPI (first tagged release)
|
|
291
|
+
- [ ] KS statistic & lift/gain curves (churn / credit workflows)
|
|
292
|
+
- [ ] Ranking metrics: precision@k, recall@k, MRR, MAP
|
|
293
|
+
- [ ] Weighted (linear / quadratic) Cohen's kappa for ordinal targets
|
|
294
|
+
- [ ] `.metrics` Polars expression namespace (`pl.col(...).metrics.roc_auc(...)`)
|
|
295
|
+
|
|
296
|
+
More candidate metrics and their implementation notes live in
|
|
297
|
+
[docs/FUTURE_IDEAS.md](https://github.com/jerheff/polarbearings/blob/main/docs/FUTURE_IDEAS.md).
|
|
298
|
+
|
|
299
|
+
## Contributing
|
|
300
|
+
|
|
301
|
+
Contributions are welcome! Please:
|
|
302
|
+
|
|
303
|
+
1. Run `just check` to verify your changes locally (lint + type-check + tests)
|
|
304
|
+
2. Submit a Pull Request with a clear description
|
|
305
|
+
|
|
306
|
+
## License
|
|
307
|
+
|
|
308
|
+
MIT License - see LICENSE file for details.
|
|
309
|
+
|
|
310
|
+
## Acknowledgments
|
|
311
|
+
|
|
312
|
+
- Built with [Polars](https://www.pola.rs/)
|
|
313
|
+
- Tested against [scikit-learn](https://scikit-learn.org/)
|
|
314
|
+
- Property-based testing with [Hypothesis](https://hypothesis.readthedocs.io/)
|
|
@@ -0,0 +1,288 @@
|
|
|
1
|
+
# Polarbearings ๐ปโโ๏ธ๐งญ
|
|
2
|
+
|
|
3
|
+
High-performance machine learning metrics implemented as native Polars expressions.
|
|
4
|
+
|
|
5
|
+
## Features
|
|
6
|
+
|
|
7
|
+
- **Fast where it counts**: Native Polars expressions โ large wins on grouped and probabilistic metrics (see [Performance](#performance))
|
|
8
|
+
- **Weighted**: Optional sample weights on nearly every metric
|
|
9
|
+
- **Flexible labels**: Any positive class โ `1`, `100`, `"cancer"`, `True` โ via `pos_label`
|
|
10
|
+
- **Correct**: scikit-learn-faithful, verified with property-based testing
|
|
11
|
+
- **Composable**: Plain Polars expressions โ drop into `group_by`, `over`, and lazy pipelines
|
|
12
|
+
- **Type-safe**: Full type hints
|
|
13
|
+
|
|
14
|
+
## Installation
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
uv add polarbearings
|
|
18
|
+
# or: pip install polarbearings
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
The only runtime dependency is Polars.
|
|
22
|
+
|
|
23
|
+
## Quick Start
|
|
24
|
+
|
|
25
|
+
```python
|
|
26
|
+
import polars as pl
|
|
27
|
+
from polarbearings import roc_auc
|
|
28
|
+
|
|
29
|
+
df = pl.DataFrame({
|
|
30
|
+
"label": [0, 0, 1, 1, 1],
|
|
31
|
+
"score": [0.1, 0.4, 0.35, 0.8, 0.9]
|
|
32
|
+
})
|
|
33
|
+
|
|
34
|
+
# Every metric is a Polars expression โ use it anywhere an expression is allowed.
|
|
35
|
+
df.select(roc_auc("label", "score"))
|
|
36
|
+
# shape: (1, 1)
|
|
37
|
+
# โโโโโโโโโโโโโโโโโโโโโโโ
|
|
38
|
+
# โ roc_auc_label_score โ
|
|
39
|
+
# โ f64 โ
|
|
40
|
+
# โโโโโโโโโโโโโโโโโโโโโโโก
|
|
41
|
+
# โ 0.833333 โ
|
|
42
|
+
# โโโโโโโโโโโโโโโโโโโโโโโ
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Why Polarbearings
|
|
46
|
+
|
|
47
|
+
Every metric is a *pure Polars expression*. That single design choice is where the
|
|
48
|
+
strengths come from โ things a scikit-learn wrapper or a compiled plugin can't
|
|
49
|
+
easily match:
|
|
50
|
+
|
|
51
|
+
- **Group-wise metrics at scale** โ metrics drop straight into `group_by().agg()`
|
|
52
|
+
and Polars parallelizes across groups (15โ65x faster than a Python loop calling
|
|
53
|
+
scikit-learn per segment).
|
|
54
|
+
- **A whole metric suite in one pass** โ bundle every metric into a single
|
|
55
|
+
`df.select([...])`; Polars shares the column scans and parallelizes across the
|
|
56
|
+
independent outputs. 13 metrics on 10M rows run in **~1.05 s vs scikit-learn's
|
|
57
|
+
~6.1 s (5.8x)** โ see [Performance](#performance).
|
|
58
|
+
- **Sample weights almost everywhere** โ nearly every metric accepts an optional
|
|
59
|
+
`weight` column, including ROC AUC, log loss, MCC, and Cohen's kappa (a few
|
|
60
|
+
exceptions are noted per metric).
|
|
61
|
+
- **Any positive class** โ `pos_label` accepts `1`, `100`, `"cancer"`, `True`, โฆ
|
|
62
|
+
no remapping your labels to 0/1.
|
|
63
|
+
- **scikit-learn-faithful** โ names and edge-case semantics mirror scikit-learn
|
|
64
|
+
(e.g. `null` for undefined cases), so migration is ~1:1.
|
|
65
|
+
- **Composable & lazy-friendly** โ pure expressions fold into lazy query plans,
|
|
66
|
+
`over()` windows, and the rest of your Polars pipeline.
|
|
67
|
+
- **Zero build, one dependency** โ pure Python emitting Polars expressions; no
|
|
68
|
+
compiled extension, installs anywhere Polars runs, supports Polars 1.0.0+.
|
|
69
|
+
|
|
70
|
+
### A whole evaluation report in one `select`
|
|
71
|
+
|
|
72
|
+
Because every metric is just an expression, a full report is one `df.select(...)`
|
|
73
|
+
โ Polars reads each column once and fans the work across the independent outputs:
|
|
74
|
+
|
|
75
|
+
<!--- invisible-code-block: python
|
|
76
|
+
df = pl.DataFrame({
|
|
77
|
+
"label": [0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0],
|
|
78
|
+
"prob": [0.2, 0.45, 0.6, 0.55, 0.75, 0.35, 0.3, 0.8, 0.65, 0.5, 0.7, 0.4],
|
|
79
|
+
})
|
|
80
|
+
--->
|
|
81
|
+
|
|
82
|
+
```python
|
|
83
|
+
from polarbearings import (
|
|
84
|
+
precision, recall, f1_score, roc_auc, average_precision,
|
|
85
|
+
log_loss, brier_score, confusion_matrix,
|
|
86
|
+
)
|
|
87
|
+
|
|
88
|
+
df.select(
|
|
89
|
+
precision("label", "prob"),
|
|
90
|
+
recall("label", "prob"),
|
|
91
|
+
f1_score("label", "prob"),
|
|
92
|
+
roc_auc("label", "prob"),
|
|
93
|
+
average_precision("label", "prob"),
|
|
94
|
+
log_loss("label", "prob"),
|
|
95
|
+
brier_score("label", "prob"),
|
|
96
|
+
confusion_matrix("label", "prob"), # struct {threshold, tp, fp, fn, tn}
|
|
97
|
+
)
|
|
98
|
+
# One tidy row, one column per metric (8 columns; abbreviated):
|
|
99
|
+
# shape: (1, 8)
|
|
100
|
+
# โโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
101
|
+
# โ precision_label_prob_0.5 โ recall_label_prob_0.5 โ โฆ โ brier_score_label_prob โ
|
|
102
|
+
# โ f64 โ f64 โ โ f64 โ
|
|
103
|
+
# โโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโโโก
|
|
104
|
+
# โ 0.714286 โ 0.833333 โ โฆ โ 0.161944 โ
|
|
105
|
+
# โโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
`benchmarks/bench_multi_metric.py` reproduces this timing across sizes from 100 to
|
|
109
|
+
1M rows.
|
|
110
|
+
|
|
111
|
+
## Metrics
|
|
112
|
+
|
|
113
|
+
~40 metrics across ranking, probabilistic, classification, calibration, and
|
|
114
|
+
regression families, plus curve generators, bootstrap CIs, deterministic splitting,
|
|
115
|
+
and threshold utilities. The **[full metrics reference โ
|
|
116
|
+
docs/guides/METRICS.md](https://github.com/jerheff/polarbearings/blob/main/docs/guides/METRICS.md)** documents every metric's
|
|
117
|
+
semantics, edge cases, and scikit-learn correspondence.
|
|
118
|
+
|
|
119
|
+
| Family | What's in it |
|
|
120
|
+
| --- | --- |
|
|
121
|
+
| **Ranking** | `roc_auc`, `average_precision`, `gini_coefficient`, `dcg_score`, `ndcg_score` |
|
|
122
|
+
| **Probabilistic** | `log_loss`, `brier_score` |
|
|
123
|
+
| **Classification** | `precision`, `recall`, `f1_score`, `fbeta_score`, `specificity`, `accuracy`, `balanced_accuracy`, `matthews_corrcoef`, `cohens_kappa`, `jaccard_score`, `confusion_matrix` |
|
|
124
|
+
| **Thresholds** | `threshold_sweep`, `quantiles` / `equal_width` / `linspace` specs |
|
|
125
|
+
| **Curves** | `roc_curve`, `pr_curve`, `det_curve`, `expected_cost`, `confusion_curve` |
|
|
126
|
+
| **Regression** | `mae`, `mse`, `rmse`, `r2_score`, `mape`, `smape`, MSLE/RMSLE, `huber_loss`, `log_cosh_loss`, pinball, Tweedie/Poisson/gamma deviance, Dยฒ scores, `max_error`, `median_absolute_error` |
|
|
127
|
+
| **Calibration** | `calibration_curve`, `expected_calibration_error` (ECE), `maximum_calibration_error` (MCE) |
|
|
128
|
+
| **Weights & CIs** | `balanced_sample_weight`, `balanced_class_weights`, `bootstrap_ci`, `bootstrap_weight` |
|
|
129
|
+
| **Splitting** | `hash_split`, `hash_splits`, `hash_fold`, `hash_uniform` |
|
|
130
|
+
|
|
131
|
+
<!--- invisible-code-block: python
|
|
132
|
+
reg = pl.DataFrame({
|
|
133
|
+
"y": [1.0, 2.0, 3.0, 4.0, 5.0],
|
|
134
|
+
"pred": [1.1, 1.9, 3.2, 3.8, 5.5],
|
|
135
|
+
"w": [1.0, 1.0, 1.0, 2.0, 1.0],
|
|
136
|
+
})
|
|
137
|
+
--->
|
|
138
|
+
|
|
139
|
+
```python
|
|
140
|
+
from polarbearings import roc_auc, f1_score, mae, calibration_curve
|
|
141
|
+
|
|
142
|
+
# Classification + probabilistic metrics are plain expressions:
|
|
143
|
+
df.select(roc_auc("label", "prob"), f1_score("label", "prob", threshold=0.7))
|
|
144
|
+
|
|
145
|
+
# Regression metrics too:
|
|
146
|
+
reg.select(mae("y", "pred", weight="w"))
|
|
147
|
+
|
|
148
|
+
# Curve helpers take a (Lazy)Frame and return a plot-ready LazyFrame:
|
|
149
|
+
calibration_curve(df, "label", "prob", n_bins=10, strategy="quantile").collect()
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
### Cross-cutting behaviour
|
|
153
|
+
|
|
154
|
+
Four behaviours are shared by the metrics rather than specific to one โ full
|
|
155
|
+
details in the [metrics reference](https://github.com/jerheff/polarbearings/blob/main/docs/guides/METRICS.md#cross-cutting-behaviour):
|
|
156
|
+
|
|
157
|
+
- **Output names** โ each expression is pre-aliased `<metric>_<target>_<score>`
|
|
158
|
+
(plus suffixes for a weight, non-default `pos_label`, or threshold), so
|
|
159
|
+
`roc_auc("label", "score")` yields a `roc_auc_label_score` column. Chain
|
|
160
|
+
`.alias("auc")` to rename it.
|
|
161
|
+
- **Sample weights** โ pass `weight="col"` to nearly any metric. Six omit it
|
|
162
|
+
(`dcg_score`, `ndcg_score`, `max_error`, `median_absolute_error`,
|
|
163
|
+
`d2_absolute_error_score`, `d2_pinball_score`) where a weighted form is undefined
|
|
164
|
+
or not cleanly expressible.
|
|
165
|
+
- **Custom positive class** โ `pos_label` accepts ints, strings, or booleans; no
|
|
166
|
+
remapping to 0/1.
|
|
167
|
+
- **Missing values** โ any `null`/`NaN` in `target`, score, or `weight` makes a
|
|
168
|
+
metric return `null` (loud, not silent), scoped to the evaluation context. Drop
|
|
169
|
+
rows yourself for complete-case behaviour. Curve helpers are the exception โ they
|
|
170
|
+
drop incomplete rows.
|
|
171
|
+
|
|
172
|
+
### Diagnostic plots
|
|
173
|
+
|
|
174
|
+
`notebooks/diagnostics.ipynb` gives examples of ROC/PR/DET/calibration curves and bootstrap
|
|
175
|
+
confidence bands with these helpers and visualizing them with Plotly.
|
|
176
|
+
|
|
177
|
+
## Use Cases
|
|
178
|
+
|
|
179
|
+
Polarbearings is a good fit for:
|
|
180
|
+
|
|
181
|
+
1. **Large-scale model evaluation** โ millions of predictions, efficiently
|
|
182
|
+
2. **Group-wise metrics** โ per-segment analysis via Polars' `group_by`
|
|
183
|
+
3. **Streaming / out-of-core** โ lazy expressions over `LazyFrame`s
|
|
184
|
+
4. **Composed evaluation reports** โ a whole metric suite in one pass
|
|
185
|
+
|
|
186
|
+
<!--- invisible-code-block: python
|
|
187
|
+
df = pl.DataFrame({
|
|
188
|
+
"segment": ["A", "A", "B", "B"],
|
|
189
|
+
"label": [0, 1, 0, 1],
|
|
190
|
+
"score": [0.1, 0.9, 0.5, 0.5],
|
|
191
|
+
})
|
|
192
|
+
--->
|
|
193
|
+
|
|
194
|
+
```python
|
|
195
|
+
from polarbearings import roc_auc
|
|
196
|
+
|
|
197
|
+
# ROC AUC per customer segment, in one parallelized pass:
|
|
198
|
+
df.group_by("segment").agg(roc_auc("label", "score"))
|
|
199
|
+
# shape: (2, 2)
|
|
200
|
+
# โโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโ
|
|
201
|
+
# โ segment โ roc_auc_label_score โ
|
|
202
|
+
# โโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโก
|
|
203
|
+
# โ A โ 1.0 โ
|
|
204
|
+
# โ B โ 0.5 โ
|
|
205
|
+
# โโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโ
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
## Performance
|
|
209
|
+
|
|
210
|
+
Polarbearings runs every metric as a native Polars expression. The advantage is
|
|
211
|
+
**large and real where there's work to parallelize, and honest where there
|
|
212
|
+
isn't.** Numbers below are speedup vs scikit-learn (scikit-learn time รท
|
|
213
|
+
polarbearings time; higher = faster), median of clean benchmark runs.
|
|
214
|
+
|
|
215
|
+
**Where polarbearings wins big** โ grouped, probabilistic, and ranking metrics.
|
|
216
|
+
Ranges span polars 1.0.0 and 1.41.2; see
|
|
217
|
+
[PERFORMANCE.md](https://github.com/jerheff/polarbearings/blob/main/docs/technical/PERFORMANCE.md) for the per-version, per-size matrix.
|
|
218
|
+
|
|
219
|
+
| Metric | 100k rows | 10M rows |
|
|
220
|
+
|--------|:---:|:---:|
|
|
221
|
+
| Grouped metrics (per segment) | **~15โ60x** | โ |
|
|
222
|
+
| Precision / F1 | ~5โ22x | ~14โ23x |
|
|
223
|
+
| Brier Score | ~9โ10x | ~7โ8x |
|
|
224
|
+
| Log Loss | ~5โ6x | ~4โ5x |
|
|
225
|
+
| ROC AUC | ~4.5โ6x | ~5x |
|
|
226
|
+
| 13 metrics in one `select` | โ | **5.8x** |
|
|
227
|
+
|
|
228
|
+
**Where it's at parity or slower** โ trivial reductions (MAE, MSE, MAPE, Rยฒ) are
|
|
229
|
+
roughly even with scikit-learn at small-to-mid sizes and *slower* on a single
|
|
230
|
+
very large array, where NumPy's tight single-threaded loop beats one Polars
|
|
231
|
+
expression. Reach for polarbearings on grouped/composed pipelines and the
|
|
232
|
+
probabilistic metrics; for a one-off MAE over a giant array, NumPy is fine.
|
|
233
|
+
|
|
234
|
+
See [docs/technical/PERFORMANCE.md](https://github.com/jerheff/polarbearings/blob/main/docs/technical/PERFORMANCE.md) for the full
|
|
235
|
+
per-metric breakdown, the size-scaling curve, and the ceteris-paribus Polars
|
|
236
|
+
version comparison.
|
|
237
|
+
|
|
238
|
+
## Development
|
|
239
|
+
|
|
240
|
+
This project uses [uv](https://github.com/astral-sh/uv) for dependency management
|
|
241
|
+
and [just](https://github.com/casey/just) for task running. Run `just` to list all
|
|
242
|
+
recipes.
|
|
243
|
+
|
|
244
|
+
```bash
|
|
245
|
+
uv sync --all-groups # install dependencies
|
|
246
|
+
just test # run tests (or: uv run pytest)
|
|
247
|
+
just quality # lint + type-check
|
|
248
|
+
just check # fast local check: lint + type-check + tests (not full CI)
|
|
249
|
+
just test-compat # test against min / mid / latest Polars
|
|
250
|
+
just bench # benchmarks vs scikit-learn
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
Testing combines unit tests, property-based tests (Hypothesis), and
|
|
254
|
+
scikit-learn compatibility tests across multiple Polars versions โ see
|
|
255
|
+
[docs/guides/TESTING.md](https://github.com/jerheff/polarbearings/blob/main/docs/guides/TESTING.md).
|
|
256
|
+
|
|
257
|
+
## Requirements
|
|
258
|
+
|
|
259
|
+
- **Python**: 3.11+
|
|
260
|
+
- **Polars**: 1.0.0+
|
|
261
|
+
|
|
262
|
+
## Roadmap
|
|
263
|
+
|
|
264
|
+
- [x] Publish to PyPI (first tagged release)
|
|
265
|
+
- [ ] KS statistic & lift/gain curves (churn / credit workflows)
|
|
266
|
+
- [ ] Ranking metrics: precision@k, recall@k, MRR, MAP
|
|
267
|
+
- [ ] Weighted (linear / quadratic) Cohen's kappa for ordinal targets
|
|
268
|
+
- [ ] `.metrics` Polars expression namespace (`pl.col(...).metrics.roc_auc(...)`)
|
|
269
|
+
|
|
270
|
+
More candidate metrics and their implementation notes live in
|
|
271
|
+
[docs/FUTURE_IDEAS.md](https://github.com/jerheff/polarbearings/blob/main/docs/FUTURE_IDEAS.md).
|
|
272
|
+
|
|
273
|
+
## Contributing
|
|
274
|
+
|
|
275
|
+
Contributions are welcome! Please:
|
|
276
|
+
|
|
277
|
+
1. Run `just check` to verify your changes locally (lint + type-check + tests)
|
|
278
|
+
2. Submit a Pull Request with a clear description
|
|
279
|
+
|
|
280
|
+
## License
|
|
281
|
+
|
|
282
|
+
MIT License - see LICENSE file for details.
|
|
283
|
+
|
|
284
|
+
## Acknowledgments
|
|
285
|
+
|
|
286
|
+
- Built with [Polars](https://www.pola.rs/)
|
|
287
|
+
- Tested against [scikit-learn](https://scikit-learn.org/)
|
|
288
|
+
- Property-based testing with [Hypothesis](https://hypothesis.readthedocs.io/)
|