aggmetrics 0.1.0.dev0__py3-none-any.whl

aggmetrics-0.1.0.dev0.dist-info/METADATA

@@ -0,0 +1,203 @@
+Metadata-Version: 2.4
+Name: aggmetrics
+Version: 0.1.0.dev0
+Summary: Aggregating classification performance measures for machine learning: indicator matrices and aggregation operators
+Author-email: "Ronaldo C. Prati" <ronaldo.prati@ufabc.edu.br>
+License: BSD-3-Clause
+Project-URL: Homepage, https://github.com/rcprati/aggmetrics
+Project-URL: Repository, https://github.com/rcprati/aggmetrics
+Project-URL: Documentation, https://aggmetrics.readthedocs.io
+Project-URL: Issues, https://github.com/rcprati/aggmetrics/issues
+Keywords: machine learning,classification,evaluation,performance measures,metrics,multilabel,multiclass,class imbalance,confusion matrix
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.20
+Provides-Extra: sklearn
+Requires-Dist: scikit-learn>=1.0; extra == "sklearn"
+Provides-Extra: frame
+Requires-Dist: pandas>=1.3; extra == "frame"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Requires-Dist: scikit-learn>=1.0; extra == "test"
+Requires-Dist: pandas>=1.3; extra == "test"
+Provides-Extra: docs
+Requires-Dist: sphinx>=5.0; extra == "docs"
+Requires-Dist: furo; extra == "docs"
+Provides-Extra: all
+Requires-Dist: scikit-learn>=1.0; extra == "all"
+Requires-Dist: pandas>=1.3; extra == "all"
+Dynamic: license-file
+
+# aggmetrics
+
+**Aggregating classification performance measures for machine learning: indicator matrices and aggregation operators.**
+
+Any binary performance measure expressible in terms of the four confusion-matrix
+counts is extended to multiclass, multilabel, ordinal, cost-sensitive and
+soft-label settings by substituting one of a small set of aggregation operators.
+Micro-, macro- and exemplar-averaging (and pooling across cross-validation folds)
+become instances of a single construction rather than separate conventions.
+
+The package also reports when a measure is being used outside the range where it
+behaves as commonly assumed — which is the part that is hard to get from a
+formula library.
+
+```python
+import numpy as np
+from aggmetrics import IndicatorMatrix, diagnose
+
+y_true = np.array([0, 1, 2, 2, 1, 0])
+y_pred = np.array([0, 2, 2, 1, 1, 1])
+im = IndicatorMatrix(y_true, y_pred, task='multiclass')
+
+im.compute('f1', 'macro')          # 0.5222
+im.compute('f1', 'micro')          # 0.5   -- equals accuracy on single-label data
+im.compute('mcc', 'macro')         # macro-MCC: no standard implementation exists
+im.compute('informedness', 'micro')
+im.accuracy()                      # 0.5
+
+im.report()                        # DataFrame: measures x operators
+```
+
+## Installation
+
+```bash
+pip install aggmetrics                # core, requires only numpy
+pip install "aggmetrics[all]"         # adds scikit-learn and pandas support
+```
+
+## Why not just use `sklearn.metrics`?
+
+`sklearn` exposes four `average` values for a fixed list of measures. The
+correspondence is exact — `'micro'`, `'macro'`, `'weighted'` and `'samples'` are
+Σ₁, Σₘ with uniform weights, Σₘ with prevalence weights and Σₙ respectively — but
+the combinations available are a subset of those the construction generates, and
+nothing in the library tells you when a combination is misleading.
+
+Three things this package adds:
+
+**Any measure under any operator.** Macro-MCC, exemplar-Jaccard, micro-informedness,
+weighted G-mean. 21 binary measures × 4 operators, without special-casing.
+
+**Structural properties as first-class data.** Each measure records whether it is
+*aggregation-decomposable*, whether it is skew-invariant, and whether its value at
+`tp = 0` depends on `tn`. These determine which identities apply.
+
+```python
+from aggmetrics import get_measure, list_measures
+
+list_measures(decomposable=True)       # micro == denominator-weighted macro
+get_measure('mcc').decomposable        # False
+get_measure('recall').bayes_threshold()  # 0.0 -- maximised by predicting all positive
+get_measure('mcc').attainable_floor(10)  # -0.111, not -1
+```
+
+**The generative claim, as an operation.** Supply a binary formula; the properties
+that decide which identities apply are *derived*, not declared:
+
+```python
+from aggmetrics import register_measure, infer_properties
+
+def prevalence_threshold(tp, fp, fn, tn):
+    r, s = tp / (tp + fn), tn / (tn + fp)
+    return (np.sqrt(r * (1 - s)) + s - 1) / (r + s - 1)
+
+m = register_measure('prevalence_threshold', prevalence_threshold)
+m.decomposable          # False  -- inferred by fitting <a,v>/<b,v> and validating
+m.skew_invariant        # True   -- a function of recall and specificity
+m.bayes_threshold()     # None   -- no closed form outside the decomposable class
+
+im.compute('prevalence_threshold', 'exemplar')   # usable immediately
+```
+
+Inference is checked against the 21 measures whose properties are known
+analytically — `verify_inference()` — and all 21 agree, with recovered coefficients
+accurate to 4.2e-15. The check paid for itself: it found two errors in our own hand
+annotations (error rate and FPR both retain a `tn` dependence at `tp = 0`, so both
+*flatter* a wholly wrong classifier, reaching 2/m and 1/(m-1) respectively).
+
+**Diagnostics.** `diagnose()` flags the traps:
+
+```python
+y = np.arange(500) % 10
+yhat = (y + 1) % 10                    # every single prediction is wrong
+im = IndicatorMatrix(y, yhat, task='multiclass')
+
+im.compute('label_accuracy', 'micro')  # 0.800  (!)
+im.compute('mcc', 'micro')             # -0.111 (!)
+im.accuracy()                          # 0.000
+
+for f in diagnose(im):
+    print(f)
+# [warning] inflated_floor: 'label_accuracy' cannot fall below 0.8000 on this
+#           10-class one-hot problem even if every prediction is wrong ...
+# [note]    micro_collapse: micro-precision, micro-recall and micro-F1 are all
+#           equal to accuracy ...
+# [warning] degenerate_optimum: 'recall' is maximised at threshold 0.000 ...
+```
+
+Checks currently implemented: pooled value outside the per-class range;
+`tn`-dependent measures on one-hot multiclass data; micro-collapse on single-label
+data; undefined per-class entries and the size of the convention effect; measures
+whose optimal decision rule is not the Bayes rule for 0/1 loss.
+
+## Features
+
+- **21 binary measures**, resolvable by name or alias (`iou` → `jaccard`, `dice` → `f1`),
+  plus `register_measure` for your own, with properties inferred.
+- **Four operators**: `micro`, `macro`, `weighted`, `exemplar`.
+- **Explicit undefined-value conventions**: `exclude` (default), `zero`
+  (`sklearn`'s behaviour), `one`, `nan`. The choice is a parameter, not a default
+  buried in the implementation.
+- **Soft ground truth** via t-norms, with `marginal_defect()` showing that only the
+  product t-norm preserves the marginals.
+- **Cost matrices**, recovering MAE and MSE as special cases.
+- **Observation masks** for partial labels, semi-supervised evaluation and abstention.
+- **scikit-learn scorers** via `make_scorer`, usable in `cross_val_score` and
+  `GridSearchCV`.
+- **AI Agent Skills included** (`.agents/skills/aggmetrics-guide/`), providing machine-readable
+  runbooks, axiomatic invariants, and diagnostic protocols for AI coding assistants.
+
+## The test suite verifies the theory
+
+Each result of the companion paper is an executable test, so a change that breaks
+a theorem breaks a test:
+
+```bash
+python -m pytest tests/test_theory.py -v
+```
+
+| Test | Result verified |
+|---|---|
+| `test_theorem1_holds_for_decomposable` | micro = denominator-weighted macro, 15 measures × 3 samples |
+| `test_theorem1_fails_for_non_decomposable` | the identity has no extension beyond that class |
+| `test_theorem2_product_is_uniquely_marginally_consistent` | only the product t-norm preserves marginals |
+| `test_theorem5_micro_collapse` | micro-P = micro-R = micro-F₁ = accuracy, single-label |
+| `test_proposition5_thresholds` | Bayes threshold `(β+δ)/(α+β+γ+δ)` |
+| `test_proposition7_floors` | attainable floors under one-hot encoding |
+| `test_proposition2_recall_prior_free_specificity_not` | one-vs-rest skew behaviour |
+| `test_corollary5_gap_is_normalised_covariance` | the micro–macro gap identity |
+| `test_theorem3_mae_and_mse_as_cost_matrices` | ordinal losses as cost matrices |
+| `test_inference_reproduces_every_annotation` | inference recovers all 21 analytic annotations |
+
+138 tests, no skips.
+
+## Examples
+
+```bash
+python examples/quickstart.py
+```
+
+## Citing
+
+Please cite both the software and the companion paper; see `CITATION.cff`.
+
+## License
+
+BSD 3-Clause.

aggmetrics-0.1.0.dev0.dist-info/RECORD

@@ -0,0 +1,13 @@
+aggmetrics/__init__.py,sha256=sNcuut_3lyhaGmPdfg8dBRCvpQFjqEWqz5cndq_ecoc,2768
+aggmetrics/aggregation.py,sha256=lzDROGNlOgbl8ZEu0PgN8CyEM7-K_clxPUzcEhzwjCc,8620
+aggmetrics/core.py,sha256=OO4p7CDfoP20w9JbFZxSyu51ONqrlSjvbmPcPdgBO5I,15691
+aggmetrics/diagnostics.py,sha256=s65hgdXbPTRDJNi01JJHmI-fVIqVl5Mcjvp2mmZm6pc,13440
+aggmetrics/inference.py,sha256=Q7pPTX-wOpZPnMEiiTGwMfD0pBkUH--r4_uAwLpzi8o,13615
+aggmetrics/measures.py,sha256=s3WcPZ85aiq3w4wzzf94_mRmErDTNTjj5m8XKZ8eeT0,13586
+aggmetrics/sklearn_compat.py,sha256=dHIoChd7OFHXR_J2GFQrMeW-sg8zN7UiygnNafhy7-k,3396
+aggmetrics/tnorms.py,sha256=jN_ar6XfG55Thi9igypKsQDPB_hk3IcUCi3bIk4yR9c,3674
+aggmetrics-0.1.0.dev0.dist-info/licenses/LICENSE,sha256=ZrA8zubLlbGMZUl3UeJbDFPD1u4mgs9MoMXRMQNIcg0,1524
+aggmetrics-0.1.0.dev0.dist-info/METADATA,sha256=qX5Mrn8GTmTwdlYqIMnAa2nJXYroUub7JR9r-Wg-Qm8,8728
+aggmetrics-0.1.0.dev0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+aggmetrics-0.1.0.dev0.dist-info/top_level.txt,sha256=CHCjg30FAscOgGgqmN7mu1MckrPVNphD6pG6BY_Di0s,11
+aggmetrics-0.1.0.dev0.dist-info/RECORD,,

aggmetrics-0.1.0.dev0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

aggmetrics-0.1.0.dev0.dist-info/licenses/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2026, Ronaldo C. Prati
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors
+   may be used to endorse or promote products derived from this software without
+   specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

aggmetrics-0.1.0.dev0.dist-info/top_level.txt

@@ -0,0 +1 @@
+aggmetrics