eb-evaluation 0.1.1__tar.gz

eb_evaluation-0.1.1/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2025, Kyle Corrie
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

eb_evaluation-0.1.1/PKG-INFO

@@ -0,0 +1,169 @@
+Metadata-Version: 2.4
+Name: eb-evaluation
+Version: 0.1.1
+Summary: Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics.
+Author-email: "Kyle Corrie (Economistician)" <kcorrie@economistician.com>
+License: BSD-3-Clause
+Project-URL: Homepage, https://github.com/Economistician/eb-evaluation
+Project-URL: Repository, https://github.com/Economistician/eb-evaluation
+Project-URL: Issues, https://github.com/Economistician/eb-evaluation/issues
+Project-URL: Documentation, https://github.com/Economistician/eb-docs
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: eb-metrics>=0.1.1
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
+Dynamic: license-file
+
+# Electric Barometer Evaluation (`eb-evaluation`)
+
+![License: BSD-3-Clause](https://img.shields.io/badge/License-BSD_3--Clause-blue.svg)
+![Python Versions](https://img.shields.io/badge/Python-3.10%2B-blue)
+[![Docs](https://img.shields.io/badge/docs-electric--barometer-blue)](https://economistician.github.io/eb-docs/)
+![Project Status](https://img.shields.io/badge/Status-Alpha-yellow)
+
+This repository contains the **evaluation and orchestration layer** of the
+*Electric Barometer* ecosystem.
+
+`eb-evaluation` sits above core metric implementations (`eb-metrics`) and
+provides structured tools for applying Electric Barometer concepts to
+real-world forecasting workflows, including readiness adjustment, model
+comparison, sensitivity analysis, and dataframe-based evaluation.
+
+Conceptual definitions and theoretical framing for the evaluation logic are
+maintained in the companion research repository:
+**`eb-papers`**.
+
+---
+
+## Naming convention
+
+Electric Barometer packages follow standard Python packaging conventions:
+
+- **Distribution names** (used with `pip install`) use hyphens  
+  e.g. `pip install eb-evaluation`
+- **Python import paths** use underscores  
+  e.g. `import eb_evaluation`
+
+This distinction is intentional and consistent across the Electric Barometer
+ecosystem.
+
+---
+
+## Role Within Electric Barometer
+
+Within the Electric Barometer ecosystem:
+
+- **`eb-papers`** defines *concepts, frameworks, and meaning*
+- **`eb-metrics`** implements *individual metrics*
+- **`eb-evaluation`** orchestrates *how metrics are applied, combined, and interpreted*
+
+This repository focuses on *evaluation logic*, not raw metric computation.
+
+---
+
+## What This Library Provides
+
+- **Readiness adjustment logic** for modifying evaluation outputs based on
+  operational readiness signals
+- **Model selection and comparison utilities** grounded in asymmetric loss and
+  readiness-aware metrics
+- **Sensitivity and tolerance analysis** for cost ratios and service thresholds
+- **DataFrame-oriented evaluation tools** for entity-level and time-based analysis
+- **Feature engineering utilities** to support evaluation pipelines
+
+---
+
+## Scope
+
+This repository focuses on **evaluation workflows and orchestration**, not
+low-level metric definitions.
+
+**In scope:**
+- Applying EB metrics to datasets and model outputs
+- Combining metrics into readiness-aware evaluation artifacts
+- Model comparison and selection logic
+- Sensitivity analysis and tolerance handling
+
+**Out of scope:**
+- Metric definitions and loss formulations (see `eb-metrics`)
+- Conceptual frameworks and theory (see `eb-papers`)
+- Model training or forecasting algorithms
+
+---
+
+## Installation
+
+Once published, the package will be installable via PyPI:
+
+```bash
+pip install eb-evaluation
+```
+
+For development or local use:
+
+```bash
+pip install -e .
+```
+
+---
+
+## Package Structure
+
+The repository follows a modern Python package layout:
+
+```text
+eb-evaluation/
+├── src/eb_evaluation/
+│   ├── adjustment/        # Readiness and evaluation adjustments
+│   ├── dataframe/         # DataFrame-based evaluation utilities
+│   ├── features/          # Feature engineering helpers
+│   ├── model_selection/   # Model comparison and selection logic
+│   └── utils/              # Shared validation and helpers
+│
+├── tests/                  # Unit tests mirroring package structure
+├── pyproject.toml          # Build and dependency configuration
+├── README.md               # Project documentation
+└── LICENSE                 # BSD-3-Clause license
+```
+
+---
+
+## Relationship to Other EB Repositories
+
+- `eb-papers`  
+  Source of truth for conceptual definitions and evaluation philosophy.
+
+- `eb-metrics`  
+  Provides the metric implementations used during evaluation.
+
+- `eb-evaluation`  
+  Orchestrates evaluation workflows using adapted models.
+
+- `eb-adapters`  
+  Ensures heterogeneous models can be evaluated consistently.
+
+When discrepancies arise, conceptual intent in `eb-papers` should be treated as authoritative.
+
+---
+
+## Development and Testing
+
+Tests are located under the `tests/` directory and mirror the package structure.
+
+To run the test suite:
+
+```bash
+pytest
+```
+
+---
+
+## Status
+
+This package is under active development.
+Public APIs may evolve prior to the first stable release.

eb_evaluation-0.1.1/README.md

@@ -0,0 +1,148 @@
+# Electric Barometer Evaluation (`eb-evaluation`)
+
+![License: BSD-3-Clause](https://img.shields.io/badge/License-BSD_3--Clause-blue.svg)
+![Python Versions](https://img.shields.io/badge/Python-3.10%2B-blue)
+[![Docs](https://img.shields.io/badge/docs-electric--barometer-blue)](https://economistician.github.io/eb-docs/)
+![Project Status](https://img.shields.io/badge/Status-Alpha-yellow)
+
+This repository contains the **evaluation and orchestration layer** of the
+*Electric Barometer* ecosystem.
+
+`eb-evaluation` sits above core metric implementations (`eb-metrics`) and
+provides structured tools for applying Electric Barometer concepts to
+real-world forecasting workflows, including readiness adjustment, model
+comparison, sensitivity analysis, and dataframe-based evaluation.
+
+Conceptual definitions and theoretical framing for the evaluation logic are
+maintained in the companion research repository:
+**`eb-papers`**.
+
+---
+
+## Naming convention
+
+Electric Barometer packages follow standard Python packaging conventions:
+
+- **Distribution names** (used with `pip install`) use hyphens  
+  e.g. `pip install eb-evaluation`
+- **Python import paths** use underscores  
+  e.g. `import eb_evaluation`
+
+This distinction is intentional and consistent across the Electric Barometer
+ecosystem.
+
+---
+
+## Role Within Electric Barometer
+
+Within the Electric Barometer ecosystem:
+
+- **`eb-papers`** defines *concepts, frameworks, and meaning*
+- **`eb-metrics`** implements *individual metrics*
+- **`eb-evaluation`** orchestrates *how metrics are applied, combined, and interpreted*
+
+This repository focuses on *evaluation logic*, not raw metric computation.
+
+---
+
+## What This Library Provides
+
+- **Readiness adjustment logic** for modifying evaluation outputs based on
+  operational readiness signals
+- **Model selection and comparison utilities** grounded in asymmetric loss and
+  readiness-aware metrics
+- **Sensitivity and tolerance analysis** for cost ratios and service thresholds
+- **DataFrame-oriented evaluation tools** for entity-level and time-based analysis
+- **Feature engineering utilities** to support evaluation pipelines
+
+---
+
+## Scope
+
+This repository focuses on **evaluation workflows and orchestration**, not
+low-level metric definitions.
+
+**In scope:**
+- Applying EB metrics to datasets and model outputs
+- Combining metrics into readiness-aware evaluation artifacts
+- Model comparison and selection logic
+- Sensitivity analysis and tolerance handling
+
+**Out of scope:**
+- Metric definitions and loss formulations (see `eb-metrics`)
+- Conceptual frameworks and theory (see `eb-papers`)
+- Model training or forecasting algorithms
+
+---
+
+## Installation
+
+Once published, the package will be installable via PyPI:
+
+```bash
+pip install eb-evaluation
+```
+
+For development or local use:
+
+```bash
+pip install -e .
+```
+
+---
+
+## Package Structure
+
+The repository follows a modern Python package layout:
+
+```text
+eb-evaluation/
+├── src/eb_evaluation/
+│   ├── adjustment/        # Readiness and evaluation adjustments
+│   ├── dataframe/         # DataFrame-based evaluation utilities
+│   ├── features/          # Feature engineering helpers
+│   ├── model_selection/   # Model comparison and selection logic
+│   └── utils/              # Shared validation and helpers
+│
+├── tests/                  # Unit tests mirroring package structure
+├── pyproject.toml          # Build and dependency configuration
+├── README.md               # Project documentation
+└── LICENSE                 # BSD-3-Clause license
+```
+
+---
+
+## Relationship to Other EB Repositories
+
+- `eb-papers`  
+  Source of truth for conceptual definitions and evaluation philosophy.
+
+- `eb-metrics`  
+  Provides the metric implementations used during evaluation.
+
+- `eb-evaluation`  
+  Orchestrates evaluation workflows using adapted models.
+
+- `eb-adapters`  
+  Ensures heterogeneous models can be evaluated consistently.
+
+When discrepancies arise, conceptual intent in `eb-papers` should be treated as authoritative.
+
+---
+
+## Development and Testing
+
+Tests are located under the `tests/` directory and mirror the package structure.
+
+To run the test suite:
+
+```bash
+pytest
+```
+
+---
+
+## Status
+
+This package is under active development.
+Public APIs may evolve prior to the first stable release.

eb_evaluation-0.1.1/pyproject.toml

@@ -0,0 +1,40 @@
+[project]
+name = "eb-evaluation"
+version = "0.1.1"
+description = "Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "BSD-3-Clause" }
+
+authors = [
+  { name = "Kyle Corrie (Economistician)", email = "kcorrie@economistician.com" }
+]
+
+dependencies = [
+  "numpy>=1.24",
+  "pandas>=2.0",
+  "eb-metrics>=0.1.1",
+]
+
+[project.urls]
+Homepage = "https://github.com/Economistician/eb-evaluation"
+Repository = "https://github.com/Economistician/eb-evaluation"
+Issues = "https://github.com/Economistician/eb-evaluation/issues"
+Documentation = "https://github.com/Economistician/eb-docs"
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=8.0",
+  "pytest-cov>=5.0",
+]
+
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+addopts = "-ra"

eb_evaluation-0.1.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

eb_evaluation-0.1.1/src/eb_evaluation/__init__.py

@@ -0,0 +1,82 @@
+"""
+Electric Barometer Evaluation Toolkit (eb-evaluation).
+
+This package provides **DataFrame-oriented evaluation, diagnostics, and
+model-selection utilities** built around Cost-Weighted Service Loss (CWSL)
+and related readiness metrics.
+
+Scope
+-----
+The eb-evaluation package sits *above* ``eb_metrics`` in the Electric Barometer
+architecture:
+
+- ``eb_metrics`` defines **metric math** (CWSL, NSL, UD, HR@τ, FRS, etc.)
+- ``eb-evaluation`` provides **tabular orchestration**, grouping logic,
+  sensitivity analysis, tolerance calibration, and model selection
+
+Primary capabilities include:
+
+DataFrame evaluation
+    - Single-slice and grouped CWSL computation
+    - Hierarchical evaluation across multiple aggregation levels
+    - Long-form (tidy) panel outputs for plotting and reporting
+    - Entity-aware evaluation with entity-specific cost ratios
+    - Cost-ratio (R = cu / co) sensitivity analysis
+
+Tolerance (τ) calibration
+    - Data-driven τ estimation from historical residuals
+    - Global and entity-level τ estimation with governance guards
+    - HR@τ computation with automatically selected tolerances
+
+Model selection
+    - Cost-aware comparison of forecast models
+    - Holdout and cross-validated selection by minimum CWSL
+    - sklearn-style wrappers for downstream pipelines
+
+Design principles
+-----------------
+- **Separation of concerns**: metric definitions live in ``eb_metrics``,
+  orchestration and evaluation live here.
+- **Operational alignment**: selection and diagnostics are driven by cost
+  and readiness, not symmetric error alone.
+- **Deterministic & explicit**: no hidden heuristics; all behavior is
+  controlled via parameters and documented outputs.
+
+This package is intended to be used alongside ``eb_metrics`` and
+``eb-adapters`` as part of the broader Electric Barometer ecosystem.
+"""
+
+from .dataframe import (
+    compute_cwsl_df,
+    evaluate_groups_df,
+    evaluate_hierarchy_df,
+    evaluate_panel_df,
+    evaluate_panel_with_entity_R,
+    compute_cwsl_sensitivity_df,
+    cwsl_sensitivity_df,
+    estimate_entity_R_from_balance,
+)
+
+from .dataframe.tolerance import (
+    hr_at_tau,
+    estimate_tau,
+    estimate_entity_tau,
+    hr_auto_tau,
+    TauEstimate,
+)
+
+__all__ = [
+    "compute_cwsl_df",
+    "evaluate_groups_df",
+    "evaluate_hierarchy_df",
+    "evaluate_panel_df",
+    "evaluate_panel_with_entity_R",
+    "compute_cwsl_sensitivity_df",
+    "cwsl_sensitivity_df",
+    "estimate_entity_R_from_balance",
+    "hr_at_tau",
+    "estimate_tau",
+    "estimate_entity_tau",
+    "hr_auto_tau",
+    "TauEstimate",
+]

eb_evaluation-0.1.1/src/eb_evaluation/adjustment/__init__.py

@@ -0,0 +1,25 @@
+"""
+Adjustment utilities for Electric Barometer evaluation.
+
+The `eb_evaluation.adjustment` package contains the **Readiness Adjustment Layer (RAL)**,
+a lightweight post-processing component that converts a baseline statistical forecast into an
+operationally conservative *readiness forecast* via a learned multiplicative uplift.
+
+Key ideas
+---------
+- **Metrics live in** `eb_metrics.metrics` (definitions only).
+- **Adjustments live here** (evaluation / selection utilities that *consume* metrics).
+- RAL learns an uplift by grid-searching multipliers and selecting the value that minimizes
+  **Cost-Weighted Service Loss (CWSL)** on historical data.
+- Uplifts can be learned globally or per-segment (with a global fallback for unseen segments).
+
+Public API
+----------
+- `ReadinessAdjustmentLayer`
+"""
+
+from .readiness_adjustment import ReadinessAdjustmentLayer
+
+__all__ = [
+    "ReadinessAdjustmentLayer",
+]

eb_evaluation-0.1.1/src/eb_evaluation/adjustment/_utils.py

@@ -0,0 +1,168 @@
+from __future__ import annotations
+
+"""
+Internal utilities for the Readiness Adjustment Layer.
+
+This module contains small, focused helpers used by the Readiness Adjustment Layer (RAL)
+implementation. These utilities are **not** part of the public API and may change without
+notice.
+
+The intent is to keep the main algorithm (and its public surface area) in
+`eb_evaluation.adjustment.readiness_adjustment` clean and readable.
+"""
+
+from collections.abc import Callable, Sequence
+from typing import Any, Union, overload
+
+import numpy as np
+import pandas as pd
+
+
+ArrayLike = Union[np.ndarray, Sequence[float], pd.Series, pd.DataFrame]
+
+
+# ---------------------------------------------------------------------------
+# Array validation utilities
+# ---------------------------------------------------------------------------
+def validate_numeric_array(arr: ArrayLike, name: str = "array") -> np.ndarray:
+    """Validate and coerce an input to a finite float NumPy array.
+
+    This helper is intentionally strict: it ensures the array is not scalar and that
+    all values are finite. This is appropriate for evaluation utilities where silent
+    propagation of NaNs can lead to misleading metric comparisons.
+
+    Parameters
+    ----------
+    arr
+        Input array-like object. Common inputs include lists, NumPy arrays, pandas Series,
+        or a single-column DataFrame.
+    name
+        Name used in error messages to make debugging easier.
+
+    Returns
+    -------
+    numpy.ndarray
+        A NumPy array of dtype ``float64``. The returned array may be 1D or 2D depending on
+        the input.
+
+    Raises
+    ------
+    ValueError
+        If ``arr`` is scalar (0-dimensional) or contains NaN/infinite values.
+
+    Notes
+    -----
+    - If you pass a DataFrame, its underlying NumPy representation is used (i.e., you are
+      responsible for selecting appropriate columns before calling this helper).
+    - This helper does *not* drop missing values. If you need filtering behavior, perform
+      it upstream and call this only once the array should be clean.
+    """
+    if isinstance(arr, (pd.Series, pd.DataFrame)):
+        arr = arr.to_numpy()
+
+    out = np.asarray(arr, dtype=float)
+
+    if out.ndim == 0:
+        raise ValueError(f"{name} must be an array-like (not a scalar).")
+
+    if not np.isfinite(out).all():
+        raise ValueError(f"{name} contains NaN or infinite values.")
+
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Safe statistical helpers
+# ---------------------------------------------------------------------------
+def safe_mean(values: np.ndarray) -> float:
+    """Compute a mean with a defined result for empty inputs.
+
+    Parameters
+    ----------
+    values
+        Numeric array. Typically a 1D array of values after filtering.
+
+    Returns
+    -------
+    float
+        The arithmetic mean of ``values``. If ``values`` is empty (``values.size == 0``),
+        returns ``0.0``.
+
+    Notes
+    -----
+    This is mainly used in group-level computations where a group may end up empty after
+    filtering invalid rows. Returning ``0.0`` is a pragmatic default for diagnostics; it
+    should not be used as a substitute for input validation in the primary metric pathway.
+    """
+    values = np.asarray(values, dtype=float)
+    if values.size == 0:
+        return 0.0
+    return float(np.mean(values))
+
+
+# ---------------------------------------------------------------------------
+# Groupby helpers
+# ---------------------------------------------------------------------------
+def groupby_apply_values(
+    df: pd.DataFrame,
+    group_cols: Union[str, Sequence[str]],
+    value_col: str,
+    func: Callable[[np.ndarray], float],
+) -> pd.DataFrame:
+    """Apply a numeric reducer to a column, grouped by one or more keys.
+
+    This helper groups ``df`` by ``group_cols`` and applies ``func`` to the values of
+    ``value_col`` for each group. The function is called with a **validated**, finite
+    float array.
+
+    Parameters
+    ----------
+    df
+        Input DataFrame containing grouping keys and the numeric value column.
+    group_cols
+        Column name or sequence of column names to group by.
+    value_col
+        Name of the column whose values are passed to ``func``.
+    func
+        Reducer function taking a 1D NumPy array and returning a scalar (float).
+
+    Returns
+    -------
+    pandas.DataFrame
+        A tidy DataFrame with columns:
+
+        - ``group_cols`` (one column per grouping key)
+        - ``f"{value_col}_agg"`` (the aggregated scalar result)
+
+    Raises
+    ------
+    KeyError
+        If ``value_col`` or any ``group_cols`` are missing from ``df``.
+    ValueError
+        If group values contain NaN/infinite values (via :func:`validate_numeric_array`).
+
+    Examples
+    --------
+    >>> out = groupby_apply_values(df, ["cluster", "daypart"], "uplift", np.mean)
+    >>> out.columns
+    Index(['cluster', 'daypart', 'uplift_agg'], dtype='object')
+    """
+    if isinstance(group_cols, str):
+        group_cols_seq: list[str] = [group_cols]
+    else:
+        group_cols_seq = list(group_cols)
+
+    missing = [c for c in [*group_cols_seq, value_col] if c not in df.columns]
+    if missing:
+        raise KeyError(f"Missing required columns: {missing}")
+
+    agg_col = f"{value_col}_agg"
+
+    grouped = (
+        df.groupby(group_cols_seq, dropna=False)[value_col]
+        .apply(lambda s: func(validate_numeric_array(s.to_numpy(), name=value_col)))
+        .reset_index()
+        .rename(columns={value_col: agg_col})
+    )
+
+    return grouped