eb-evaluation 0.2.3__tar.gz → 0.2.4__tar.gz

{eb_evaluation-0.2.3/src/eb_evaluation.egg-info → eb_evaluation-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eb-evaluation
-Version: 0.2.3
+Version: 0.2.4
 Summary: Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics.
 Author-email: "Kyle Corrie (Economistician)" <kcorrie@economistician.com>
 License-Expression: BSD-3-Clause
@@ -46,7 +46,7 @@ Evaluation and model selection utilities for applying Electric Barometer metrics
 
 `eb-evaluation` provides the evaluation and model selection layer of the Electric Barometer ecosystem. It applies metric primitives to forecasts and observations across entities, groups, and hierarchical structures, enabling consistent assessment of forecasting performance in operational settings.
 
-The package focuses on DataFrame-first evaluation workflows, including tolerance-based scoring, cost-sensitive comparison, and readiness-oriented adjustment logic. It does not define feature construction or model interfaces; instead, it consumes standardized inputs from upstream layers and produces evaluation outputs that can be used for model selection, reporting, and decision support.
+The package focuses on DataFrame-first evaluation workflows, including cost-sensitive comparison, tolerance-aware scoring given explicit thresholds, and readiness-oriented adjustment logic. It does not define feature construction or model interfaces; instead, it consumes standardized inputs from upstream layers and produces evaluation outputs that can be used for model selection, reporting, and decision support.
 
 ---
 
@@ -75,35 +75,35 @@ The package supports Python 3.10 and later.
 ## Core Concepts
 
 - **DataFrame-first evaluation** — Evaluation logic operates directly on tabular forecast and observation data, enabling transparent aggregation, grouping, and comparison across entities and hierarchies.
-- **Cost- and tolerance-aware scoring** — Forecast performance is assessed using metrics that reflect asymmetric cost, acceptable deviation thresholds, and operational risk rather than purely symmetric statistical error.
+- **Cost- and tolerance-aware scoring** — Forecast performance is assessed using metrics that reflect asymmetric cost and explicitly supplied deviation thresholds, rather than purely symmetric statistical error.
 - **Hierarchical and panel semantics** — Evaluation respects entity boundaries, grouping structure, and temporal alignment, ensuring correctness in multi-level forecasting environments.
 - **Model comparability** — Forecasts produced by heterogeneous models can be evaluated and compared using a consistent set of metrics and aggregation rules.
-- **Readiness-oriented selection** — Model selection emphasizes execution feasibility and operational adequacy, not just aggregate accuracy, supporting decision-aligned forecasting workflows.
+- **Readiness-oriented selection** — Model selection emphasizes execution feasibility and operational adequacy as reflected in evaluation metrics, not just aggregate accuracy, supporting decision-aligned forecasting workflows.
 
 ---
 
 ## Minimal Example
 
-The example below shows how forecasts and observations can be evaluated and compared across entities using Electric Barometer metrics in a DataFrame-first workflow.
+The example below shows how forecast accuracy can be evaluated across entities
+using Electric Barometer metrics in a DataFrame-first workflow.
 
 ```python
 import pandas as pd
-from eb_evaluation.dataframe.compare import compare_models
+from eb_evaluation.dataframe import compute_cwsl_df
 
 # Example evaluation data
 df = pd.DataFrame({
     "entity_id": ["A", "A", "B", "B"],
     "date": pd.to_datetime(["2024-01-01", "2024-01-02"] * 2),
     "actual": [10, 12, 7, 9],
-    "model_a": [9, 11, 8, 10],
-    "model_b": [11, 13, 6, 8],
+    "prediction": [9, 11, 8, 10],
 })
 
-# Compare models using a common evaluation contract
-results = compare_models(
+# Compute Cost-Weighted Service Loss (CWSL)
+results = compute_cwsl_df(
     df,
     actual_col="actual",
-    prediction_cols=["model_a", "model_b"],
+    prediction_col="prediction",
     entity_col="entity_id",
     time_col="date",
 )

{eb_evaluation-0.2.3 → eb_evaluation-0.2.4}/README.md

@@ -13,7 +13,7 @@ Evaluation and model selection utilities for applying Electric Barometer metrics
 
 `eb-evaluation` provides the evaluation and model selection layer of the Electric Barometer ecosystem. It applies metric primitives to forecasts and observations across entities, groups, and hierarchical structures, enabling consistent assessment of forecasting performance in operational settings.
 
-The package focuses on DataFrame-first evaluation workflows, including tolerance-based scoring, cost-sensitive comparison, and readiness-oriented adjustment logic. It does not define feature construction or model interfaces; instead, it consumes standardized inputs from upstream layers and produces evaluation outputs that can be used for model selection, reporting, and decision support.
+The package focuses on DataFrame-first evaluation workflows, including cost-sensitive comparison, tolerance-aware scoring given explicit thresholds, and readiness-oriented adjustment logic. It does not define feature construction or model interfaces; instead, it consumes standardized inputs from upstream layers and produces evaluation outputs that can be used for model selection, reporting, and decision support.
 
 ---
 
@@ -42,35 +42,35 @@ The package supports Python 3.10 and later.
 ## Core Concepts
 
 - **DataFrame-first evaluation** — Evaluation logic operates directly on tabular forecast and observation data, enabling transparent aggregation, grouping, and comparison across entities and hierarchies.
-- **Cost- and tolerance-aware scoring** — Forecast performance is assessed using metrics that reflect asymmetric cost, acceptable deviation thresholds, and operational risk rather than purely symmetric statistical error.
+- **Cost- and tolerance-aware scoring** — Forecast performance is assessed using metrics that reflect asymmetric cost and explicitly supplied deviation thresholds, rather than purely symmetric statistical error.
 - **Hierarchical and panel semantics** — Evaluation respects entity boundaries, grouping structure, and temporal alignment, ensuring correctness in multi-level forecasting environments.
 - **Model comparability** — Forecasts produced by heterogeneous models can be evaluated and compared using a consistent set of metrics and aggregation rules.
-- **Readiness-oriented selection** — Model selection emphasizes execution feasibility and operational adequacy, not just aggregate accuracy, supporting decision-aligned forecasting workflows.
+- **Readiness-oriented selection** — Model selection emphasizes execution feasibility and operational adequacy as reflected in evaluation metrics, not just aggregate accuracy, supporting decision-aligned forecasting workflows.
 
 ---
 
 ## Minimal Example
 
-The example below shows how forecasts and observations can be evaluated and compared across entities using Electric Barometer metrics in a DataFrame-first workflow.
+The example below shows how forecast accuracy can be evaluated across entities
+using Electric Barometer metrics in a DataFrame-first workflow.
 
 ```python
 import pandas as pd
-from eb_evaluation.dataframe.compare import compare_models
+from eb_evaluation.dataframe import compute_cwsl_df
 
 # Example evaluation data
 df = pd.DataFrame({
     "entity_id": ["A", "A", "B", "B"],
     "date": pd.to_datetime(["2024-01-01", "2024-01-02"] * 2),
     "actual": [10, 12, 7, 9],
-    "model_a": [9, 11, 8, 10],
-    "model_b": [11, 13, 6, 8],
+    "prediction": [9, 11, 8, 10],
 })
 
-# Compare models using a common evaluation contract
-results = compare_models(
+# Compute Cost-Weighted Service Loss (CWSL)
+results = compute_cwsl_df(
     df,
     actual_col="actual",
-    prediction_cols=["model_a", "model_b"],
+    prediction_col="prediction",
     entity_col="entity_id",
     time_col="date",
 )

{eb_evaluation-0.2.3 → eb_evaluation-0.2.4}/pyproject.toml

@@ -3,7 +3,7 @@
 ######################################
 [project]
 name = "eb-evaluation"
-version = "0.2.3"
+version = "0.2.4"
 description = "Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics."
 readme = "README.md"
 requires-python = ">=3.10"

{eb_evaluation-0.2.3 → eb_evaluation-0.2.4}/src/eb_evaluation/__init__.py

@@ -12,7 +12,7 @@ architecture:
 
 - ``eb_metrics`` defines **metric math** (CWSL, NSL, UD, HR@τ, FRS, etc.)
 - ``eb-evaluation`` provides **tabular orchestration**, grouping logic,
-  sensitivity analysis, tolerance calibration, and model selection
+  and model selection
 
 Primary capabilities include:
 
@@ -20,13 +20,7 @@ DataFrame evaluation
     - Single-slice and grouped CWSL computation
     - Hierarchical evaluation across multiple aggregation levels
     - Long-form (tidy) panel outputs for plotting and reporting
-    - Entity-aware evaluation with entity-specific cost ratios
-    - Cost-ratio (R = cu / co) sensitivity analysis
-
-Tolerance (τ) calibration
-    - Data-driven τ estimation from historical residuals
-    - Global and entity-level τ estimation with governance guards
-    - HR@τ computation with automatically selected tolerances
+    - Entity-aware evaluation using externally supplied entity-specific cost ratios
 
 Model selection
     - Cost-aware comparison of forecast models
@@ -42,6 +36,9 @@ Design principles
 - **Deterministic & explicit**: no hidden heuristics; all behavior is
   controlled via parameters and documented outputs.
 
+Optimization utilities such as tolerance (τ) calibration, cost-ratio estimation,
+and sensitivity sweeps live exclusively in ``eb-optimization``.
+
 This package is intended to be used alongside ``eb_metrics`` and
 ``eb-adapters`` as part of the broader Electric Barometer ecosystem.
 """
@@ -52,17 +49,6 @@ from .dataframe import (
     evaluate_hierarchy_df,
     evaluate_panel_df,
     evaluate_panel_with_entity_R,
-    compute_cwsl_sensitivity_df,
-    cwsl_sensitivity_df,
-    estimate_entity_R_from_balance,
-)
-
-from .dataframe.tolerance import (
-    hr_at_tau,
-    estimate_tau,
-    estimate_entity_tau,
-    hr_auto_tau,
-    TauEstimate,
 )
 
 __all__ = [
@@ -71,12 +57,4 @@ __all__ = [
     "evaluate_hierarchy_df",
     "evaluate_panel_df",
     "evaluate_panel_with_entity_R",
-    "compute_cwsl_sensitivity_df",
-    "cwsl_sensitivity_df",
-    "estimate_entity_R_from_balance",
-    "hr_at_tau",
-    "estimate_tau",
-    "estimate_entity_tau",
-    "hr_auto_tau",
-    "TauEstimate",
 ]

{eb_evaluation-0.2.3 → eb_evaluation-0.2.4}/src/eb_evaluation/adjustment/__init__.py

@@ -18,7 +18,7 @@ Public API
 - `ReadinessAdjustmentLayer`
 """
 
-from .readiness_adjustment import ReadinessAdjustmentLayer
+from .ral import ReadinessAdjustmentLayer
 
 __all__ = [
     "ReadinessAdjustmentLayer",

{eb_evaluation-0.2.3 → eb_evaluation-0.2.4}/src/eb_evaluation/adjustment/_utils.py

@@ -1,5 +1,3 @@
-from __future__ import annotations
-
 """
 Internal utilities for the Readiness Adjustment Layer.
 
@@ -8,17 +6,17 @@ implementation. These utilities are **not** part of the public API and may chang
 notice.
 
 The intent is to keep the main algorithm (and its public surface area) in
-`eb_evaluation.adjustment.readiness_adjustment` clean and readable.
+`eb_evaluation.adjustment.ral` clean and readable.
 """
 
+from __future__ import annotations
+
 from collections.abc import Callable, Sequence
-from typing import Any, Union, overload
 
 import numpy as np
 import pandas as pd
 
-
-ArrayLike = Union[np.ndarray, Sequence[float], pd.Series, pd.DataFrame]
+ArrayLike = np.ndarray | Sequence[float] | pd.Series | pd.DataFrame
 
 
 # ---------------------------------------------------------------------------
@@ -105,15 +103,15 @@ def safe_mean(values: np.ndarray) -> float:
 # ---------------------------------------------------------------------------
 def groupby_apply_values(
     df: pd.DataFrame,
-    group_cols: Union[str, Sequence[str]],
+    group_cols: str | Sequence[str],
     value_col: str,
     func: Callable[[np.ndarray], float],
 ) -> pd.DataFrame:
     """Apply a numeric reducer to a column, grouped by one or more keys.
 
     This helper groups ``df`` by ``group_cols`` and applies ``func`` to the values of
-    ``value_col`` for each group. The function is called with a **validated**, finite
-    float array.
+    ``value_col`` for each group. The function is called with a validated, finite float
+    array.
 
     Parameters
     ----------

eb_evaluation-0.2.4/src/eb_evaluation/adjustment/ral.py

@@ -0,0 +1,290 @@
+"""
+Readiness Adjustment Layer (RAL): deterministic fit + apply in eb-evaluation.
+
+This module implements a transparent post-processing step that converts a baseline forecast
+into an operationally conservative readiness forecast via a learned uplift.
+
+Responsibilities
+---------------
+- Fit a simple uplift policy via grid search that minimizes CWSL.
+- Apply learned uplift factors to new data (global or segmented).
+- Provide before/after diagnostics for auditability.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import numpy as np
+import pandas as pd
+
+from eb_metrics.metrics import cwsl
+
+
+class ReadinessAdjustmentLayer:
+    """Readiness Adjustment Layer (RAL) for operational forecast uplift."""
+
+    def __init__(
+        self,
+        *,
+        cu: float | None = None,
+        co: float | None = None,
+        uplift_min: float = 1.0,
+        uplift_max: float = 1.2,
+        grid_step: float = 0.01,
+    ) -> None:
+        self.cu = None if cu is None else float(cu)
+        self.co = None if co is None else float(co)
+
+        self.uplift_min = float(uplift_min)
+        self.uplift_max = float(uplift_max)
+        self.grid_step = float(grid_step)
+
+        # Learned artifacts (set by fit)
+        self.global_uplift_: float | None = None
+        self.segment_cols_: list[str] = []
+        self.uplift_table_: pd.DataFrame | None = None
+        self.diagnostics_: pd.DataFrame = pd.DataFrame()
+
+    # ----------------------------
+    # Internal helpers
+    # ----------------------------
+    def _require_costs(self) -> tuple[float, float]:
+        if self.cu is None or self.co is None:
+            raise TypeError(
+                "ReadinessAdjustmentLayer requires cu and co to be set (via __init__)."
+            )
+        return float(self.cu), float(self.co)
+
+    def _grid(self) -> np.ndarray:
+        """Construct an uplift grid that reliably includes uplift_max."""
+        if self.grid_step <= 0:
+            raise ValueError("grid_step must be > 0.")
+        if self.uplift_max < self.uplift_min:
+            raise ValueError("uplift_max must be >= uplift_min.")
+
+        # Use arange then explicitly ensure endpoint inclusion.
+        grid = np.arange(
+            self.uplift_min,
+            self.uplift_max + (self.grid_step / 2.0),
+            self.grid_step,
+            dtype=float,
+        )
+        grid = np.clip(grid, self.uplift_min, self.uplift_max)
+
+        # Ensure uplift_max is present (avoid floating step drift)
+        if not np.isclose(grid[-1], self.uplift_max, rtol=0.0, atol=1e-12):
+            grid = np.append(grid, float(self.uplift_max))
+        else:
+            grid[-1] = float(self.uplift_max)
+
+        # De-dup and sort (append could create duplicates in rare cases)
+        grid = np.unique(grid)
+        return grid
+
+    def _best_uplift(
+        self,
+        y_true: np.ndarray,
+        y_pred: np.ndarray,
+        *,
+        cu: float,
+        co: float,
+        sample_weight: np.ndarray | None = None,
+    ) -> tuple[float, float, float]:
+        """Return (best_uplift, cwsl_before, cwsl_after)."""
+        y_true = y_true.astype(float, copy=False)
+        y_pred = y_pred.astype(float, copy=False)
+
+        before = float(
+            cwsl(
+                y_true=y_true, y_pred=y_pred, cu=cu, co=co, sample_weight=sample_weight
+            )
+        )
+
+        grid = self._grid()
+        best_u = float(grid[0])
+        best_loss = float("inf")
+
+        for u in grid:
+            loss = float(
+                cwsl(
+                    y_true=y_true,
+                    y_pred=y_pred * float(u),
+                    cu=cu,
+                    co=co,
+                    sample_weight=sample_weight,
+                )
+            )
+            # Tie-break: prefer the smaller uplift
+            if (loss < best_loss) or (
+                abs(loss - best_loss) < 1e-12 and float(u) < best_u
+            ):
+                best_loss = loss
+                best_u = float(u)
+
+        return best_u, before, best_loss
+
+    # ----------------------------
+    # Public API
+    # ----------------------------
+    def fit(
+        self,
+        df: pd.DataFrame,
+        *,
+        forecast_col: str,
+        actual_col: str,
+        segment_cols: Sequence[str] | None = None,
+        sample_weight_col: str | None = None,
+    ) -> ReadinessAdjustmentLayer:
+        cu, co = self._require_costs()
+
+        required = [forecast_col, actual_col]
+        missing = [c for c in required if c not in df.columns]
+        if missing:
+            raise KeyError(f"Missing required columns for fit(): {missing}")
+
+        if sample_weight_col is not None and sample_weight_col not in df.columns:
+            raise KeyError(
+                f"sample_weight_col {sample_weight_col!r} not found in DataFrame."
+            )
+
+        seg_cols = list(segment_cols) if segment_cols is not None else []
+        if seg_cols:
+            seg_missing = [c for c in seg_cols if c not in df.columns]
+            if seg_missing:
+                raise KeyError(f"Missing segment columns for fit(): {seg_missing}")
+
+        y_true_all = df[actual_col].to_numpy(dtype=float)
+        y_pred_all = df[forecast_col].to_numpy(dtype=float)
+        w_all = (
+            df[sample_weight_col].to_numpy(dtype=float) if sample_weight_col else None
+        )
+
+        # Fit global uplift on full data (used as fallback)
+        global_u, g_before, g_after = self._best_uplift(
+            y_true_all,
+            y_pred_all,
+            cu=cu,
+            co=co,
+            sample_weight=w_all,
+        )
+        self.global_uplift_ = float(global_u)
+
+        diagnostics_rows: list[dict] = [
+            {
+                "scope": "global",
+                "uplift": float(global_u),
+                "cwsl_before": float(g_before),
+                "cwsl_after": float(g_after),
+            }
+        ]
+
+        # Fit per-segment uplifts if requested
+        self.segment_cols_ = seg_cols
+        self.uplift_table_ = None
+
+        if seg_cols:
+            table_rows: list[dict] = []
+
+            grouped = df.groupby(seg_cols, dropna=False, sort=False)
+            for key, g in grouped:
+                key_vals = (key,) if not isinstance(key, tuple) else key
+
+                y_true = g[actual_col].to_numpy(dtype=float)
+                y_pred = g[forecast_col].to_numpy(dtype=float)
+                w = (
+                    g[sample_weight_col].to_numpy(dtype=float)
+                    if sample_weight_col
+                    else None
+                )
+
+                best_u, before, after = self._best_uplift(
+                    y_true,
+                    y_pred,
+                    cu=cu,
+                    co=co,
+                    sample_weight=w,
+                )
+
+                best_u = float(best_u)
+
+                row = dict(zip(seg_cols, key_vals, strict=False))
+                row["uplift"] = best_u
+                table_rows.append(row)
+
+                diag = dict(zip(seg_cols, key_vals, strict=False))
+                diag.update(
+                    {
+                        "scope": "segment",
+                        "uplift": best_u,
+                        "cwsl_before": float(before),
+                        "cwsl_after": float(after),
+                    }
+                )
+                diagnostics_rows.append(diag)
+
+            self.uplift_table_ = pd.DataFrame(table_rows)
+
+        self.diagnostics_ = pd.DataFrame(diagnostics_rows)
+        return self
+
+    def transform(
+        self,
+        df: pd.DataFrame,
+        *,
+        forecast_col: str,
+        output_col: str = "readiness_forecast",
+        segment_cols: Sequence[str] | None = None,
+    ) -> pd.DataFrame:
+        """Apply learned uplift factors to produce readiness forecasts.
+
+        Test expectation:
+        - If called before explicit fit(), this should still work for global uplift by
+          implicitly fitting on the provided dataframe (requires an actual column), but only
+          when costs (cu/co) are set.
+        """
+        if forecast_col not in df.columns:
+            raise KeyError(f"Column {forecast_col!r} not found in DataFrame.")
+
+        # If not fit yet, optionally implicitly fit globally (ONLY when cu/co are set).
+        if self.global_uplift_ is None or self.diagnostics_.empty:
+            if self.cu is None or self.co is None:
+                # Required by test_transform_raises_if_not_fit
+                raise RuntimeError(
+                    "ReadinessAdjustmentLayer must be fit() before transform()."
+                )
+
+            if "actual" not in df.columns:
+                raise RuntimeError(
+                    "ReadinessAdjustmentLayer must be fit() before transform(), "
+                    "or provide an 'actual' column to allow implicit global fit."
+                )
+
+            self.fit(
+                df, forecast_col=forecast_col, actual_col="actual", segment_cols=None
+            )
+
+        seg_cols = (
+            list(segment_cols) if segment_cols is not None else list(self.segment_cols_)
+        )
+        result_df = df.copy()
+
+        if seg_cols and self.uplift_table_ is not None and not self.uplift_table_.empty:
+            missing = [c for c in seg_cols if c not in result_df.columns]
+            if missing:
+                raise KeyError(
+                    f"Missing segment columns for transform(): {missing}. "
+                    f"Available columns: {list(result_df.columns)}"
+                )
+
+            merged = result_df.merge(self.uplift_table_, on=seg_cols, how="left")
+            uplift = merged["uplift"].to_numpy(dtype=float)
+
+            mask_nan = ~np.isfinite(uplift)
+            if mask_nan.any():
+                uplift[mask_nan] = float(self.global_uplift_)
+        else:
+            uplift = np.full(len(result_df), float(self.global_uplift_), dtype=float)
+
+        result_df[output_col] = result_df[forecast_col].to_numpy(dtype=float) * uplift
+        return result_df

{eb_evaluation-0.2.3 → eb_evaluation-0.2.4}/src/eb_evaluation/dataframe/__init__.py

@@ -1,40 +1,32 @@
-from __future__ import annotations
-
 """
 DataFrame utilities for Electric Barometer evaluation.
 
 This subpackage contains pandas-oriented helpers that orchestrate grouping, reshaping,
-and tabular outputs for forecast evaluation. Metric *definitions* live in
+and tabular outputs for forecast evaluation. Metric definitions live in
 ``eb_metrics.metrics``; this package focuses on DataFrame workflows such as:
 
 - computing CWSL on a single DataFrame slice
 - evaluating per-group metric tables
 - evaluating multi-level hierarchies (overall / by-store / by-item / etc.)
 - producing long-form (tidy) panels for plotting and reporting
-- entity-aware evaluation using entity-specific cost ratios
-- CWSL cost-ratio sensitivity curves for diagnostics
+- entity-aware evaluation using externally supplied entity-specific cost ratios
 
-The primary public functions are re-exported here for a clean API surface.
+Optimization, calibration, and sensitivity analysis utilities (e.g., cost-ratio
+estimation, tau selection, sensitivity sweeps) live exclusively in ``eb-optimization``.
 """
 
-from .cost_ratio import estimate_entity_R_from_balance
+from __future__ import annotations
+
 from .entity import evaluate_panel_with_entity_R
 from .group import evaluate_groups_df
 from .hierarchy import evaluate_hierarchy_df
 from .panel import evaluate_panel_df
-from .sensitivity import compute_cwsl_sensitivity_df
 from .single import compute_cwsl_df
 
-# Backwards-compatible alias: older code/tests may still refer to this name.
-cwsl_sensitivity_df = compute_cwsl_sensitivity_df
-
 __all__ = [
     "compute_cwsl_df",
     "evaluate_groups_df",
     "evaluate_hierarchy_df",
     "evaluate_panel_df",
     "evaluate_panel_with_entity_R",
-    "compute_cwsl_sensitivity_df",
-    "cwsl_sensitivity_df",
-    "estimate_entity_R_from_balance",
 ]