forelint 0.1.0__tar.gz

forelint-0.1.0/.gitignore

@@ -0,0 +1,7 @@
+.venv/
+__pycache__/
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+*.zip

forelint-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 David Mashiah
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

forelint-0.1.0/PKG-INFO

@@ -0,0 +1,125 @@
+Metadata-Version: 2.5
+Name: forelint
+Version: 0.1.0
+Summary: Lint your forecast backtests: leakage between related series, the run-to-run noise floor, model comparison tests, and per-item bias before you trust a score.
+Project-URL: Homepage, https://github.com/davidmashiah/forelint
+Project-URL: Source, https://github.com/davidmashiah/forelint
+Project-URL: Author, https://davidmashiah.com/
+Author: David Mashiah
+License-Expression: MIT
+License-File: LICENSE
+Keywords: backtesting,data leakage,demand forecasting,diebold-mariano,forecast bias,forecast evaluation,forecast value added,forecasting,group k-fold,model confidence set,time series cross-validation,tracking signal
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Scientific/Engineering :: Mathematics
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.22
+Requires-Dist: pandas>=1.5
+Requires-Dist: scikit-learn>=1.2
+Requires-Dist: scipy>=1.9
+Provides-Extra: test
+Requires-Dist: arch>=6; extra == 'test'
+Requires-Dist: pytest>=7; extra == 'test'
+Description-Content-Type: text/markdown
+
+# forelint
+
+**Lint your forecast backtests before you trust a score.**
+
+forelint checks the things a backtest score hides: leakage between related
+series, the run-to-run noise floor, whether model A is really better than
+model B, and which items carry the bias and the error. It works on the
+long-format tables that statsforecast, mlforecast and most forecasting code
+already produce, and on any scikit-learn estimator.
+
+```bash
+pip install forelint
+```
+
+## What it checks
+
+| Question | Function |
+|---|---|
+| Does my validation leak through related stores, products or regions? | `split_optimism`, `GroupTimeSplit` |
+| Is this improvement bigger than seed-to-seed noise? | `noise_floor` |
+| Is model A really better than B across many series? | `dm_test`, `diebold_mariano`, `model_confidence_set` |
+| Which items are always over or under forecast? Does the model beat a naive forecast? Where does the error live? | `audit` |
+
+## Quickstart
+
+```python
+import forelint as fl
+
+# df: one row per item and date, the actual in "y", one column per model,
+# e.g. the output of statsforecast's cross_validation()
+report = fl.audit(df, models=["LightGBM", "ETS"], baseline="SeasonalNaive")
+print(report)
+
+# Is LightGBM really better than the seasonal naive, or is it noise?
+print(fl.dm_test(df, "LightGBM", "SeasonalNaive"))
+```
+
+```python
+# How much does random K-fold flatter the model, compared with holding out
+# the future of entities the model never saw?
+fl.split_optimism(model, X, y, groups=store_ids, time=dates)
+
+# What difference between two runs is just noise?
+nf = fl.noise_floor(lambda seed: train_and_score(seed), seeds=range(10))
+nf.is_meaningful(new_mae - old_mae)
+```
+
+## On real data: one Walmart store in M5
+
+`benchmarks/m5_benchmark.py` runs every check on the public M5 data: store
+CA_1, department FOODS_3, 823 items, the last 365 days, a direct 28-day-ahead
+gradient-boosting model from scikit-learn, scored on the final 28 days.
+Measured, not assumed:
+
+- **The model added nothing over a 28-day moving average.** Forecast value
+  added was **−0.1%** on MAE, the model was worse than the moving average on
+  **49% of items**, and Diebold-Mariano gives **p = 0.31**. The model
+  confidence set keeps both. An average-MAE table alone would not show this.
+- **It under-forecasts by 9.3%** of actual sales.
+- **10% of items hold 73% of the squared error.**
+- **The noise floor is small here:** over 10 seeds the MAE sd is 0.0014, so
+  any change under 0.0045 is noise.
+- **Splitting barely mattered for this data:** random K-fold was 1% optimistic
+  against a time split. Leakage depends on the features; forelint measures it
+  rather than assuming it.
+
+The model is deliberately plain (six features, no tuning). The point is the
+audit, not the model.
+
+## Why
+
+In an international machine-learning competition on well-geology prediction,
+validation ranked five builds against the hidden leaderboard with a
+correlation of −0.745: the build it liked most scored worst, and identical code scored 6.241, 6.259 and
+6.368 on three runs. Both failures were measurable before the deadline with
+the right checks. forelint is those checks, packaged.
+
+Background: [My validation ranked five models backwards](https://davidmashiah.com/notes/validation-ranked-my-models-backwards).
+
+## Conventions
+
+- Error is **forecast − actual**. Positive bias means over-forecasting.
+- Tracking signal is the sum of errors divided by the mean absolute error;
+  the default alarm level is ±4.
+- `dm_test` averages the loss difference across series at each time stamp,
+  then applies Diebold-Mariano with the Harvey-Leybourne-Newbold correction.
+- `model_confidence_set` follows Hansen, Lunde and Nason (2011) with the
+  T_max statistic and a moving-block bootstrap. Its kept set matches
+  `arch.bootstrap.MCS` in the test suite.
+
+## Status
+
+Alpha. The API may change before 1.0. Issues and pull requests are welcome.
+
+## Author
+
+[David Mashiah](https://davidmashiah.com). MIT licensed.

forelint-0.1.0/README.md

@@ -0,0 +1,97 @@
+# forelint
+
+**Lint your forecast backtests before you trust a score.**
+
+forelint checks the things a backtest score hides: leakage between related
+series, the run-to-run noise floor, whether model A is really better than
+model B, and which items carry the bias and the error. It works on the
+long-format tables that statsforecast, mlforecast and most forecasting code
+already produce, and on any scikit-learn estimator.
+
+```bash
+pip install forelint
+```
+
+## What it checks
+
+| Question | Function |
+|---|---|
+| Does my validation leak through related stores, products or regions? | `split_optimism`, `GroupTimeSplit` |
+| Is this improvement bigger than seed-to-seed noise? | `noise_floor` |
+| Is model A really better than B across many series? | `dm_test`, `diebold_mariano`, `model_confidence_set` |
+| Which items are always over or under forecast? Does the model beat a naive forecast? Where does the error live? | `audit` |
+
+## Quickstart
+
+```python
+import forelint as fl
+
+# df: one row per item and date, the actual in "y", one column per model,
+# e.g. the output of statsforecast's cross_validation()
+report = fl.audit(df, models=["LightGBM", "ETS"], baseline="SeasonalNaive")
+print(report)
+
+# Is LightGBM really better than the seasonal naive, or is it noise?
+print(fl.dm_test(df, "LightGBM", "SeasonalNaive"))
+```
+
+```python
+# How much does random K-fold flatter the model, compared with holding out
+# the future of entities the model never saw?
+fl.split_optimism(model, X, y, groups=store_ids, time=dates)
+
+# What difference between two runs is just noise?
+nf = fl.noise_floor(lambda seed: train_and_score(seed), seeds=range(10))
+nf.is_meaningful(new_mae - old_mae)
+```
+
+## On real data: one Walmart store in M5
+
+`benchmarks/m5_benchmark.py` runs every check on the public M5 data: store
+CA_1, department FOODS_3, 823 items, the last 365 days, a direct 28-day-ahead
+gradient-boosting model from scikit-learn, scored on the final 28 days.
+Measured, not assumed:
+
+- **The model added nothing over a 28-day moving average.** Forecast value
+  added was **−0.1%** on MAE, the model was worse than the moving average on
+  **49% of items**, and Diebold-Mariano gives **p = 0.31**. The model
+  confidence set keeps both. An average-MAE table alone would not show this.
+- **It under-forecasts by 9.3%** of actual sales.
+- **10% of items hold 73% of the squared error.**
+- **The noise floor is small here:** over 10 seeds the MAE sd is 0.0014, so
+  any change under 0.0045 is noise.
+- **Splitting barely mattered for this data:** random K-fold was 1% optimistic
+  against a time split. Leakage depends on the features; forelint measures it
+  rather than assuming it.
+
+The model is deliberately plain (six features, no tuning). The point is the
+audit, not the model.
+
+## Why
+
+In an international machine-learning competition on well-geology prediction,
+validation ranked five builds against the hidden leaderboard with a
+correlation of −0.745: the build it liked most scored worst, and identical code scored 6.241, 6.259 and
+6.368 on three runs. Both failures were measurable before the deadline with
+the right checks. forelint is those checks, packaged.
+
+Background: [My validation ranked five models backwards](https://davidmashiah.com/notes/validation-ranked-my-models-backwards).
+
+## Conventions
+
+- Error is **forecast − actual**. Positive bias means over-forecasting.
+- Tracking signal is the sum of errors divided by the mean absolute error;
+  the default alarm level is ±4.
+- `dm_test` averages the loss difference across series at each time stamp,
+  then applies Diebold-Mariano with the Harvey-Leybourne-Newbold correction.
+- `model_confidence_set` follows Hansen, Lunde and Nason (2011) with the
+  T_max statistic and a moving-block bootstrap. Its kept set matches
+  `arch.bootstrap.MCS` in the test suite.
+
+## Status
+
+Alpha. The API may change before 1.0. Issues and pull requests are welcome.
+
+## Author
+
+[David Mashiah](https://davidmashiah.com). MIT licensed.

forelint-0.1.0/benchmarks/m5_benchmark.py

@@ -0,0 +1,96 @@
+"""forelint on real retail data: the M5 competition, one Walmart store.
+
+Usage:  python benchmarks/m5_benchmark.py path/to/m5.zip [STORE] [DEPT]
+
+Reads the public M5 sales file (from the Kaggle competition or Nixtla's
+mirror), keeps one store and department, and builds a direct 28-day-ahead
+global model with scikit-learn. Every number this prints is measured on that
+data; nothing is assumed.
+"""
+from __future__ import annotations
+
+import sys
+import time
+import zipfile
+
+import numpy as np
+import pandas as pd
+from sklearn.ensemble import HistGradientBoostingRegressor
+
+import forelint as fl
+
+H = 28          # forecast horizon, as in M5
+HISTORY = 420   # days of history kept
+
+
+def load(path, store, dept):
+    with zipfile.ZipFile(path) as z, z.open("sales_train_evaluation.csv") as f:
+        wide = pd.read_csv(f)
+    wide = wide[(wide.store_id == store) & (wide.dept_id == dept)]
+    days = [c for c in wide.columns if c.startswith("d_")][-HISTORY:]
+    long = wide.melt(id_vars=["item_id"], value_vars=days, var_name="d", value_name="y")
+    long["ds"] = long["d"].str[2:].astype(int)
+    return long[["item_id", "ds", "y"]].sort_values(["item_id", "ds"]).reset_index(drop=True)
+
+
+def features(df):
+    g = df.groupby("item_id")["y"]
+    out = df.copy()
+    out["lag28"] = g.shift(H)
+    out["lag35"] = g.shift(H + 7)
+    out["rmean7"] = g.transform(lambda s: s.shift(H).rolling(7).mean())
+    out["rmean28"] = g.transform(lambda s: s.shift(H).rolling(28).mean())
+    out["dow"] = out["ds"] % 7
+    out["item"] = out["item_id"].astype("category").cat.codes
+    return out.dropna().reset_index(drop=True)
+
+
+FEATS = ["item", "dow", "lag28", "lag35", "rmean7", "rmean28"]
+
+
+def main(path, store="CA_1", dept="FOODS_3"):
+    t0 = time.time()
+    df = features(load(path, store, dept))
+    print(f"M5 {store} {dept}: {df.item_id.nunique()} items, {df.ds.nunique()} days, {len(df):,} rows\n")
+    X, y = df[FEATS].to_numpy(float), df["y"].to_numpy(float)
+    model = HistGradientBoostingRegressor(max_iter=200, random_state=0)
+
+    # 1. How much each split strategy flatters the model.
+    opt = fl.split_optimism(model, X, y, groups=df["item"].to_numpy(), time=df["ds"].to_numpy(),
+                            reference="time", n_splits=5, gap=0, random_state=0)
+    print("1. split optimism (MAE; reference = time split, the way the model is used)")
+    print(opt.round(4).to_string(), "\n")
+
+    # Final holdout: the last 28 days.
+    cut = df.ds.max() - H
+    tr, te = df[df.ds <= cut], df[df.ds > cut].copy()
+
+    # 2. Run-to-run noise floor: same data, same settings, different seeds.
+    def run(seed):
+        m = HistGradientBoostingRegressor(max_iter=200, random_state=seed, early_stopping=True)
+        m.fit(tr[FEATS], tr["y"])
+        return float(np.abs(m.predict(te[FEATS]) - te["y"]).mean())
+    nf = fl.noise_floor(run, seeds=range(10))
+    print("2.", nf)
+    print(f"   run MAEs: {np.round(nf.scores, 4).tolist()}\n")
+
+    # 3. Audit against two naive baselines.
+    m = HistGradientBoostingRegressor(max_iter=200, random_state=0).fit(tr[FEATS], tr["y"])
+    te["GBM"] = m.predict(te[FEATS])
+    te["Naive28"] = te["lag28"]
+    last = tr.groupby("item_id")["y"].apply(lambda s: s.iloc[-28:].mean())
+    te["MA28"] = te["item_id"].map(last)
+    rep = fl.audit(te, ["GBM", "Naive28"], baseline="MA28", y="y", id_col="item_id", time_col="ds")
+    print("3.", rep, "\n")
+
+    # 4. Is the GBM really better than the moving average?
+    print("4. Diebold-Mariano, GBM vs MA28 (squared loss, h = 1 on the per-day mean):")
+    print("  ", fl.dm_test(te, "GBM", "MA28", id_col="item_id"))
+    losses = pd.DataFrame({c: ((te[c] - te["y"]) ** 2).groupby(te["ds"]).mean() for c in ["GBM", "Naive28", "MA28"]})
+    print("   model confidence set (10%):")
+    print(fl.model_confidence_set(losses, size=0.10, random_state=0).round(4).to_string())
+    print(f"\nran in {time.time() - t0:.0f} s")
+
+
+if __name__ == "__main__":
+    main(*sys.argv[1:])

forelint-0.1.0/pyproject.toml

@@ -0,0 +1,47 @@
+[build-system]
+requires = ["hatchling>=1.21"]
+build-backend = "hatchling.build"
+
+[project]
+name = "forelint"
+version = "0.1.0"
+description = "Lint your forecast backtests: leakage between related series, the run-to-run noise floor, model comparison tests, and per-item bias before you trust a score."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "David Mashiah" }]
+requires-python = ">=3.9"
+dependencies = [
+  "numpy>=1.22",
+  "pandas>=1.5",
+  "scipy>=1.9",
+  "scikit-learn>=1.2",
+]
+keywords = [
+  "forecasting", "forecast evaluation", "time series cross-validation", "backtesting",
+  "data leakage", "group k-fold", "diebold-mariano", "model confidence set",
+  "tracking signal", "forecast bias", "forecast value added", "demand forecasting",
+]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Science/Research",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+  "Topic :: Scientific/Engineering :: Information Analysis",
+  "Topic :: Scientific/Engineering :: Mathematics",
+]
+
+[project.optional-dependencies]
+test = ["pytest>=7", "arch>=6"]
+
+[project.urls]
+Homepage = "https://github.com/davidmashiah/forelint"
+Source = "https://github.com/davidmashiah/forelint"
+Author = "https://davidmashiah.com/"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/forelint"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

forelint-0.1.0/src/forelint/__init__.py

@@ -0,0 +1,23 @@
+"""forelint: lint your forecast backtests before you trust a score.
+
+- `GroupTimeSplit`: hold out the future of entities the model never saw.
+- `split_optimism`: how much each split strategy flatters the model.
+- `noise_floor`: the run-to-run spread, below which a difference is noise.
+- `diebold_mariano`, `dm_test`, `model_confidence_set`: is A really better than B.
+- `audit`: per-item bias, tracking signal, error concentration, forecast value added.
+"""
+from .audit import AuditReport, audit
+from .compare import DMResult, diebold_mariano, dm_test, model_confidence_set
+from .noise import NoiseFloor, noise_floor
+from .optimism import split_optimism
+from .splits import GroupTimeSplit
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "AuditReport", "audit",
+    "DMResult", "diebold_mariano", "dm_test", "model_confidence_set",
+    "NoiseFloor", "noise_floor",
+    "split_optimism",
+    "GroupTimeSplit",
+]

forelint-0.1.0/src/forelint/audit.py

@@ -0,0 +1,122 @@
+"""One report on a backtest: bias, tracking signal, error concentration and
+forecast value added, per item and overall.
+
+Average error hides the two things that cost money: items whose forecast is
+always too high or always too low, and the few items that carry most of the
+error. The audit surfaces both, and compares every model with a baseline the
+way forecast value added does, because a model that loses to a naive forecast
+on most items should not ship whatever its average says.
+
+Sign convention: error = forecast − actual, so positive bias is over-forecasting.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import numpy as np
+import pandas as pd
+
+__all__ = ["AuditReport", "audit"]
+
+
+@dataclass
+class AuditReport:
+    per_item: pd.DataFrame
+    summary: pd.DataFrame
+    baseline: str | None
+    top_frac: float
+    ts_limit: float
+
+    def __str__(self):
+        lines = ["forelint audit", ""]
+        for model, r in self.summary.iterrows():
+            lines.append(f"{model}")
+            lines.append(f"  MAE {r['mae']:.4g}   RMSE {r['rmse']:.4g}   bias {r['bias']:+.4g} "
+                         f"({r['pct_bias']:+.1%} of actuals)")
+            lines.append(f"  items with |tracking signal| > {self.ts_limit:g}: "
+                         f"{int(r['n_biased_items'])} of {int(r['n_items'])} ({r['share_biased_items']:.0%})")
+            lines.append(f"  top {self.top_frac:.0%} of items hold {r['error_concentration']:.0%} of squared error")
+            if self.baseline is not None and model != self.baseline:
+                lines.append(f"  forecast value added vs {self.baseline}: {r['fva']:+.1%} on MAE; "
+                             f"worse than {self.baseline} on {r['share_items_worse']:.0%} of items")
+            lines.append("")
+        return "\n".join(lines).rstrip()
+
+
+def _tracking_signal(e):
+    mad = np.abs(e).mean()
+    return np.nan if mad == 0 else e.sum() / mad
+
+
+def audit(df, models, *, baseline=None, y="y", id_col="unique_id", time_col="ds",
+          top_frac=0.10, ts_limit=4.0) -> AuditReport:
+    """Audit a long-format backtest.
+
+    Parameters
+    ----------
+    df : DataFrame with one row per item and time stamp, the actual in ``y``
+        and one column per model. statsforecast / mlforecast
+        ``cross_validation`` output works as is.
+    models : list of model column names.
+    baseline : optional model column to measure forecast value added against,
+        typically a naive or seasonal-naive forecast.
+    top_frac : share of items used for the error-concentration figure.
+    ts_limit : tracking-signal alarm level; 4 is a common choice.
+
+    Returns
+    -------
+    AuditReport with ``per_item`` (one row per item and model) and ``summary``
+    (one row per model). ``print(report)`` gives a readable summary.
+    """
+    models = list(models)
+    cols = models + ([baseline] if baseline and baseline not in models else [])
+    missing = [c for c in [y, id_col, time_col, *cols] if c not in df.columns]
+    if missing:
+        raise KeyError(f"columns not found: {missing}")
+    d = df.sort_values([id_col, time_col])
+    items = []
+    for model in cols:
+        e = d[model] - d[y]
+        g = pd.DataFrame({id_col: d[id_col], "e": e, "ae": e.abs(), "se": e ** 2, "y": d[y]}).groupby(id_col)
+        part = pd.DataFrame({
+            "n": g["e"].size(),
+            "mae": g["ae"].mean(),
+            "rmse": np.sqrt(g["se"].mean()),
+            "bias": g["e"].mean(),
+            "sse": g["se"].sum(),
+            "sum_actual": g["y"].sum(),
+            "tracking_signal": g["e"].apply(lambda s: _tracking_signal(s.to_numpy())),
+        })
+        part["model"] = model
+        items.append(part.reset_index())
+    per_item = pd.concat(items, ignore_index=True)
+    if baseline:
+        base_mae = per_item[per_item["model"] == baseline].set_index(id_col)["mae"]
+        per_item["fva"] = 1 - per_item["mae"] / per_item[id_col].map(base_mae)
+
+    rows = {}
+    for model in cols:
+        p = per_item[per_item["model"] == model]
+        e = d[model] - d[y]
+        sse_sorted = np.sort(p["sse"].to_numpy())[::-1]
+        k = max(1, int(round(top_frac * len(sse_sorted))))
+        biased = (p["tracking_signal"].abs() > ts_limit).sum()
+        r = {
+            "n_items": len(p),
+            "mae": e.abs().mean(),
+            "rmse": float(np.sqrt((e ** 2).mean())),
+            "bias": e.mean(),
+            "pct_bias": e.sum() / d[y].sum() if d[y].sum() != 0 else np.nan,
+            "n_biased_items": int(biased),
+            "share_biased_items": biased / len(p),
+            "error_concentration": sse_sorted[:k].sum() / sse_sorted.sum() if sse_sorted.sum() else np.nan,
+        }
+        if baseline:
+            base_e = d[baseline] - d[y]
+            r["fva"] = 1 - e.abs().mean() / base_e.abs().mean()
+            r["share_items_worse"] = float((p["fva"] < 0).mean())
+        rows[model] = r
+    summary = pd.DataFrame.from_dict(rows, orient="index")
+    summary.index.name = "model"
+    return AuditReport(per_item=per_item, summary=summary, baseline=baseline,
+                       top_frac=top_frac, ts_limit=ts_limit)

forelint-0.1.0/src/forelint/compare.py

@@ -0,0 +1,151 @@
+"""Is model A really better than model B, or is it noise?
+
+`diebold_mariano` tests one pair on one loss series, with the Harvey,
+Leybourne and Newbold (1997) small-sample correction. `dm_test` does the same
+for a long-format forecast table with many series, by averaging the loss
+difference across series at each time stamp. `model_confidence_set` keeps the
+set of models that cannot be told apart from the best (Hansen, Lunde and
+Nason 2011), using a moving-block bootstrap and the T_max statistic.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+__all__ = ["DMResult", "diebold_mariano", "dm_test", "model_confidence_set"]
+
+_LOSSES = {
+    "squared": lambda e: e ** 2,
+    "absolute": lambda e: np.abs(e),
+}
+
+
+@dataclass
+class DMResult:
+    statistic: float
+    p_value: float
+    mean_loss_diff: float
+    n: int
+
+    def __str__(self):
+        side = "A worse" if self.mean_loss_diff > 0 else "A better"
+        return (f"DM = {self.statistic:.3f}, p = {self.p_value:.4f} over {self.n} points "
+                f"(mean loss difference {self.mean_loss_diff:.4g}, {side})")
+
+
+def _dm_from_diff(d, h):
+    d = np.asarray(d, dtype=float)
+    d = d[~np.isnan(d)]
+    n = d.size
+    if n < 3:
+        raise ValueError("need at least 3 loss differences")
+    mean = d.mean()
+    dc = d - mean
+    var = dc @ dc / n
+    for k in range(1, h):
+        var += 2.0 * (dc[k:] @ dc[:-k]) / n
+    if var <= 0:
+        return DMResult(np.nan, np.nan, float(mean), n)
+    stat = mean / np.sqrt(var / n)
+    stat *= np.sqrt((n + 1 - 2 * h + h * (h - 1) / n) / n)  # HLN correction
+    p = 2.0 * stats.t.sf(abs(stat), df=n - 1)
+    return DMResult(float(stat), float(p), float(mean), n)
+
+
+def diebold_mariano(errors_a, errors_b, h=1, loss="squared") -> DMResult:
+    """Diebold-Mariano test with the HLN correction.
+
+    A positive statistic means model A has the larger loss. ``h`` is the
+    forecast horizon; losses up to lag ``h - 1`` are allowed to be correlated.
+    """
+    if loss not in _LOSSES:
+        raise ValueError(f"loss must be one of {list(_LOSSES)}")
+    ea, eb = np.asarray(errors_a, dtype=float), np.asarray(errors_b, dtype=float)
+    if ea.shape != eb.shape:
+        raise ValueError("error arrays must have the same shape")
+    return _dm_from_diff(_LOSSES[loss](ea) - _LOSSES[loss](eb), h)
+
+
+def dm_test(df, model_a, model_b, *, y="y", id_col="unique_id", time_col="ds", h=1,
+            loss="squared") -> DMResult:
+    """Diebold-Mariano on a long-format table of many series.
+
+    ``df`` has one row per series and time stamp, with the actual in ``y`` and
+    one column per model (the layout of statsforecast/mlforecast
+    ``cross_validation`` output). The loss difference is averaged across
+    series at each time stamp, then tested as one series.
+    """
+    if loss not in _LOSSES:
+        raise ValueError(f"loss must be one of {list(_LOSSES)}")
+    f = _LOSSES[loss]
+    d = f(df[model_a] - df[y]) - f(df[model_b] - df[y])
+    per_time = d.groupby(df[time_col]).mean().sort_index()
+    return _dm_from_diff(per_time.to_numpy(), h)
+
+
+def _block_indices(n, block, reps, rng):
+    n_blocks = int(np.ceil(n / block))
+    starts = rng.integers(0, n, size=(reps, n_blocks))
+    idx = (starts[:, :, None] + np.arange(block)[None, None, :]) % n
+    return idx.reshape(reps, -1)[:, :n]
+
+
+def model_confidence_set(losses, size=0.10, reps=2000, block_size=None, random_state=None):
+    """Hansen's model confidence set with the T_max statistic.
+
+    Parameters
+    ----------
+    losses : DataFrame of shape (n_times, n_models), one column per model,
+        each cell that model's loss at that time. For many series, average the
+        loss across series per time stamp first.
+    size : the test size; models whose MCS p-value is at least ``size`` stay
+        in the set.
+    block_size : moving-block length for the bootstrap. Defaults to
+        ``ceil(n_times ** (1/3))``.
+
+    Returns
+    -------
+    DataFrame indexed by model with ``pvalue`` and ``in_set``, ordered from
+    first eliminated to last.
+    """
+    L = pd.DataFrame(losses).astype(float)
+    if L.isna().any().any():
+        raise ValueError("losses contain missing values")
+    n, m = L.shape
+    if m < 2:
+        raise ValueError("need at least two models")
+    block = block_size or int(np.ceil(n ** (1 / 3)))
+    rng = np.random.default_rng(random_state)
+    idx = _block_indices(n, block, reps, rng)
+    vals = L.to_numpy()
+    lbar = vals.mean(axis=0)                       # (m,)
+    lboot = vals[idx].mean(axis=1)                 # (reps, m)
+
+    included = list(range(m))
+    order, pvals = [], []
+    running = 0.0
+    while len(included) > 1:
+        s = np.array(included)
+        dbar = lbar[s] - lbar[s].mean()
+        dboot = lboot[:, s] - lboot[:, s].mean(axis=1, keepdims=True)
+        dev = dboot - dbar
+        var = (dev ** 2).mean(axis=0)
+        var[var <= 0] = np.finfo(float).tiny
+        t = dbar / np.sqrt(var)
+        tmax = t.max()
+        tboot = (dev / np.sqrt(var)).max(axis=1)
+        p = float((tboot >= tmax).mean())
+        running = max(running, p)
+        worst = int(s[np.argmax(t)])
+        order.append(worst)
+        pvals.append(running)
+        included.remove(worst)
+    order.append(included[0])
+    pvals.append(1.0)
+    out = pd.DataFrame({"pvalue": pvals}, index=L.columns[order])
+    out["in_set"] = out["pvalue"] >= size
+    out.index.name = "model"
+    return out

forelint-0.1.0/src/forelint/noise.py

@@ -0,0 +1,58 @@
+"""The run-to-run noise floor.
+
+Seeds, bootstrap draws, GPU nondeterminism and data loading order all move a
+score. Two runs cannot estimate that spread: in one recorded case the first
+two runs of identical code differed by 0.018 and the third moved the spread to
+a standard deviation of 0.069. Measure the floor first; any difference smaller
+than it is not a result.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+import numpy as np
+from scipy import stats
+
+__all__ = ["NoiseFloor", "noise_floor"]
+
+
+@dataclass
+class NoiseFloor:
+    """Scores from repeated runs of one unchanged setup."""
+
+    scores: np.ndarray
+    confidence: float = 0.95
+    mean: float = field(init=False)
+    sd: float = field(init=False)
+    mde: float = field(init=False)
+
+    def __post_init__(self):
+        self.scores = np.asarray(self.scores, dtype=float)
+        if self.scores.size < 3:
+            raise ValueError("a noise floor needs at least 3 runs; two cannot estimate a spread")
+        self.mean = float(self.scores.mean())
+        self.sd = float(self.scores.std(ddof=1))
+        # Smallest difference between two single runs that exceeds noise at
+        # this confidence, using the t quantile for the runs we actually have.
+        q = stats.t.ppf(0.5 + self.confidence / 2, df=self.scores.size - 1)
+        self.mde = float(q * np.sqrt(2.0) * self.sd)
+
+    def is_meaningful(self, difference: float) -> bool:
+        """True if a single-run difference is larger than the noise floor."""
+        return abs(difference) > self.mde
+
+    def __str__(self):
+        return (f"noise floor over {self.scores.size} runs: mean {self.mean:.4g}, "
+                f"sd {self.sd:.4g}; differences under {self.mde:.4g} are noise "
+                f"at {self.confidence:.0%}")
+
+
+def noise_floor(run, seeds=(0, 1, 2, 3, 4), confidence=0.95) -> NoiseFloor:
+    """Call ``run(seed) -> float`` once per seed and summarise the spread.
+
+    Example
+    -------
+    >>> nf = noise_floor(lambda s: train_and_score(seed=s), seeds=range(5))
+    >>> nf.is_meaningful(new_score - old_score)
+    """
+    return NoiseFloor(np.array([float(run(s)) for s in seeds]), confidence=confidence)

forelint-0.1.0/src/forelint/optimism.py

@@ -0,0 +1,73 @@
+"""How much does the way you split flatter the model?
+
+Run one estimator under several validation schemes and report each scheme's
+error next to the strictest one. If random K-fold reports half the error of a
+split that holds out the future of unseen entities, the random score is not a
+measurement of the deployed model; it is a measurement of the leak.
+"""
+from __future__ import annotations
+
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import GroupKFold, KFold, cross_validate
+
+from .splits import GroupTimeSplit
+
+__all__ = ["split_optimism"]
+
+_STRATEGIES = ("random", "group", "time", "group_time")
+
+
+def _splitter(name, *, time, n_splits, test_group_frac, gap, random_state):
+    if name == "random":
+        return KFold(n_splits=n_splits, shuffle=True, random_state=random_state)
+    if name == "group":
+        return GroupKFold(n_splits=n_splits)
+    if name == "time":
+        return GroupTimeSplit(time, n_splits=n_splits, gap=gap, holdout="none")
+    if name == "group_time":
+        return GroupTimeSplit(time, n_splits=n_splits, gap=gap, holdout="groups",
+                              test_group_frac=test_group_frac, random_state=random_state)
+    raise ValueError(f"unknown strategy {name!r}; choose from {_STRATEGIES}")
+
+
+def split_optimism(estimator, X, y, *, groups, time, scoring="neg_mean_absolute_error",
+                   strategies=_STRATEGIES, reference="group_time", n_splits=5,
+                   test_group_frac=0.2, gap=0, random_state=0):
+    """Score one estimator under several split strategies.
+
+    Parameters
+    ----------
+    estimator : scikit-learn estimator
+    X, y : training data
+    groups : array-like, the entity of each row (store, SKU, well, region)
+    time : array-like, the time stamp of each row
+    scoring : str, a scikit-learn scorer. ``neg_*`` scorers are turned back
+        into positive errors.
+    strategies : subset of ``("random", "group", "time", "group_time")``
+    reference : the strategy to compare against, normally the one that
+        matches how the model will be used.
+
+    Returns
+    -------
+    pandas.DataFrame indexed by strategy with ``error`` (mean over folds),
+    ``error_sd``, ``n_folds`` and ``vs_reference``: the strategy's error as a
+    share of the reference error. ``0.6`` means that strategy reports 40%
+    less error than the reference, i.e. it is 40% optimistic.
+    """
+    if reference not in strategies:
+        raise ValueError("reference must be one of the strategies")
+    sign = -1.0 if scoring.startswith("neg_") else 1.0
+    rows = {}
+    for name in strategies:
+        cv = _splitter(name, time=time, n_splits=n_splits, test_group_frac=test_group_frac,
+                       gap=gap, random_state=random_state)
+        res = cross_validate(estimator, X, y, groups=None if name == "random" else groups,
+                             cv=cv, scoring=scoring)
+        err = sign * np.asarray(res["test_score"], dtype=float)
+        rows[name] = {"error": err.mean(), "error_sd": err.std(ddof=1) if len(err) > 1 else np.nan,
+                      "n_folds": len(err)}
+    out = pd.DataFrame.from_dict(rows, orient="index")
+    out["vs_reference"] = out["error"] / out.loc[reference, "error"]
+    out.index.name = "strategy"
+    return out

forelint-0.1.0/src/forelint/splits.py

@@ -0,0 +1,108 @@
+"""Splitters that hold out the future *and* unseen entities.
+
+scikit-learn ships `GroupKFold` (unseen entities, any time) and
+`TimeSeriesSplit` (future rows, every entity seen). Forecasting new stores,
+new products or new wells needs both at once: train on the past of some
+entities, test on the future of entities the model never saw. Without that,
+related series leak through validation and the score flatters the model.
+"""
+from __future__ import annotations
+
+import numpy as np
+from sklearn.model_selection import BaseCrossValidator
+
+__all__ = ["GroupTimeSplit"]
+
+
+class GroupTimeSplit(BaseCrossValidator):
+    """Expanding-window splits that can also hold out whole groups.
+
+    Parameters
+    ----------
+    time : array-like of shape (n_samples,)
+        The time stamp of every row, aligned with ``X``. Any sortable type.
+    n_splits : int, default 5
+        Number of test windows, taken from the end of the time range.
+    test_size : int, optional
+        Unique time points per test window. Defaults to
+        ``n_unique_times // (n_splits + 1)``.
+    gap : int, default 0
+        Unique time points dropped between the end of training and the
+        start of each test window, to mimic a forecast lead time.
+    holdout : {"groups", "none"}, default "groups"
+        ``"groups"`` tests only on groups that are absent from training
+        (new entities). ``"none"`` keeps every group on both sides, which is
+        an ordinary time-series split.
+    test_group_frac : float, default 0.2
+        Share of groups held out in each fold when ``holdout="groups"``.
+    random_state : int, optional
+        Seed for which groups are held out.
+
+    Notes
+    -----
+    ``split`` needs ``groups`` when ``holdout="groups"``. Works with
+    ``sklearn.model_selection.cross_validate(..., groups=groups)``.
+    """
+
+    def __init__(self, time, n_splits=5, test_size=None, gap=0, holdout="groups",
+                 test_group_frac=0.2, random_state=None):
+        if holdout not in ("groups", "none"):
+            raise ValueError("holdout must be 'groups' or 'none'")
+        if not 0 < test_group_frac < 1:
+            raise ValueError("test_group_frac must be between 0 and 1")
+        if n_splits < 1:
+            raise ValueError("n_splits must be at least 1")
+        self.time = time
+        self.n_splits = n_splits
+        self.test_size = test_size
+        self.gap = gap
+        self.holdout = holdout
+        self.test_group_frac = test_group_frac
+        self.random_state = random_state
+
+    def get_n_splits(self, X=None, y=None, groups=None):
+        return self.n_splits
+
+    def _windows(self, time):
+        uniq = np.unique(time)
+        size = self.test_size or len(uniq) // (self.n_splits + 1)
+        if size < 1:
+            raise ValueError("not enough unique time points for this many splits")
+        first_test = len(uniq) - self.n_splits * size
+        if first_test - self.gap < 1:
+            raise ValueError("no training time left before the first test window")
+        for k in range(self.n_splits):
+            start = first_test + k * size
+            yield uniq[: start - self.gap], uniq[start: start + size]
+
+    def split(self, X, y=None, groups=None):
+        time = np.asarray(self.time)
+        n = len(time) if X is None else len(X)
+        if len(time) != n:
+            raise ValueError(f"time has {len(time)} rows but X has {n}")
+        idx = np.arange(n)
+        if self.holdout == "groups":
+            if groups is None:
+                raise ValueError("holdout='groups' needs the groups argument")
+            groups = np.asarray(groups)
+            uniq_groups = np.unique(groups)
+            n_test = max(1, int(round(self.test_group_frac * len(uniq_groups))))
+            if n_test >= len(uniq_groups):
+                raise ValueError("test_group_frac leaves no groups for training")
+            rng = np.random.default_rng(self.random_state)
+        for train_times, test_times in self._windows(time):
+            in_train_time = np.isin(time, train_times)
+            in_test_time = np.isin(time, test_times)
+            if self.holdout == "groups":
+                held = rng.choice(uniq_groups, size=n_test, replace=False)
+                is_held = np.isin(groups, held)
+                train, test = idx[in_train_time & ~is_held], idx[in_test_time & is_held]
+            else:
+                train, test = idx[in_train_time], idx[in_test_time]
+            if len(train) == 0 or len(test) == 0:
+                raise ValueError("a fold came out empty; use fewer splits or a larger test_group_frac")
+            yield train, test
+
+    def _iter_test_indices(self, X=None, y=None, groups=None):  # pragma: no cover
+        for _, test in self.split(X, y, groups):
+            yield test

forelint-0.1.0/tests/test_forelint.py

@@ -0,0 +1,176 @@
+import numpy as np
+import pandas as pd
+import pytest
+from scipy import stats
+from sklearn.ensemble import RandomForestRegressor
+
+import forelint as fl
+
+
+# ── audit: hand-checked against the eight-day forecast-bias example ─────────
+def _eight_days():
+    demand = [12, 9, 15, 10, 14, 11, 13, 10]
+    eh = [3, 1, 2, 3, 1, 2, 3, 1]
+    en = [3, -2, -3, 2, 1, -1, -3, 3]
+    return pd.DataFrame({
+        "unique_id": "sku1", "ds": range(8), "y": demand,
+        "H": np.add(demand, eh), "N": np.add(demand, en),
+    })
+
+
+def test_audit_bias_tracking_signal_and_fva():
+    rep = fl.audit(_eight_days(), ["H"], baseline="N")
+    s = rep.summary
+    assert s.loc["H", "bias"] == pytest.approx(2.0)
+    assert s.loc["H", "mae"] == pytest.approx(2.0)
+    assert s.loc["H", "rmse"] == pytest.approx(np.sqrt(38 / 8))
+    assert s.loc["H", "pct_bias"] == pytest.approx(16 / 94)
+    assert s.loc["N", "bias"] == pytest.approx(0.0)
+    assert s.loc["N", "mae"] == pytest.approx(2.25)
+    # tracking signal = sum of errors / MAD = 16 / 2 = 8 for H, 0 for N
+    pi = rep.per_item.set_index("model")
+    assert pi.loc["H", "tracking_signal"] == pytest.approx(8.0)
+    assert pi.loc["N", "tracking_signal"] == pytest.approx(0.0)
+    assert s.loc["H", "n_biased_items"] == 1
+    # forecast value added of H over N: 1 - 2.00 / 2.25
+    assert s.loc["H", "fva"] == pytest.approx(1 - 2.0 / 2.25)
+    assert "forecast value added" in str(rep)
+
+
+def test_audit_error_concentration():
+    rows = []
+    for i in range(10):
+        err = 10.0 if i == 0 else 1.0
+        rows += [{"unique_id": i, "ds": t, "y": 0.0, "m": err} for t in range(4)]
+    rep = fl.audit(pd.DataFrame(rows), ["m"], top_frac=0.1)
+    # item 0: SSE 400; others: 4 each -> 400 / (400 + 36)
+    assert rep.summary.loc["m", "error_concentration"] == pytest.approx(400 / 436)
+
+
+def test_audit_missing_column():
+    with pytest.raises(KeyError):
+        fl.audit(_eight_days(), ["nope"])
+
+
+# ── noise floor ─────────────────────────────────────────────────────────────
+def test_noise_floor_values():
+    nf = fl.noise_floor(lambda s: [1.0, 2.0, 3.0][s], seeds=[0, 1, 2])
+    assert nf.mean == pytest.approx(2.0)
+    assert nf.sd == pytest.approx(1.0)
+    assert nf.mde == pytest.approx(stats.t.ppf(0.975, 2) * np.sqrt(2))
+    assert not nf.is_meaningful(1.0)
+    assert nf.is_meaningful(7.0)
+
+
+def test_noise_floor_needs_three_runs():
+    with pytest.raises(ValueError, match="at least 3"):
+        fl.noise_floor(lambda s: s, seeds=[0, 1])
+
+
+# ── Diebold-Mariano ─────────────────────────────────────────────────────────
+def test_dm_hand_computed():
+    # loss differences d = 1..5 with squared loss: e_a^2 - e_b^2
+    ea = np.sqrt(np.array([1, 2, 3, 4, 5], dtype=float))
+    eb = np.zeros(5)
+    r = fl.diebold_mariano(ea, eb, h=1, loss="squared")
+    raw = 3 / np.sqrt(2 / 5)
+    expected = raw * np.sqrt((5 + 1 - 2) / 5)
+    assert r.statistic == pytest.approx(expected)
+    assert r.p_value == pytest.approx(2 * stats.t.sf(expected, df=4))
+    assert r.mean_loss_diff == pytest.approx(3.0)
+
+
+def test_dm_is_antisymmetric_and_detects_a_better_model():
+    rng = np.random.default_rng(1)
+    good = rng.normal(0, 1, 300)
+    bad = rng.normal(0, 2, 300)
+    ab = fl.diebold_mariano(bad, good)
+    ba = fl.diebold_mariano(good, bad)
+    assert ab.statistic == pytest.approx(-ba.statistic)
+    assert ab.statistic > 0 and ab.p_value < 1e-6
+
+
+def test_dm_panel_long_format():
+    rng = np.random.default_rng(2)
+    rows = []
+    for i in range(20):
+        for t in range(60):
+            y = rng.normal(10, 1)
+            rows.append({"unique_id": i, "ds": t, "y": y,
+                         "good": y + rng.normal(0, 0.5), "bad": y + rng.normal(0, 1.5)})
+    df = pd.DataFrame(rows)
+    r = fl.dm_test(df, "bad", "good")
+    assert r.n == 60 and r.statistic > 0 and r.p_value < 0.01
+
+
+# ── model confidence set ────────────────────────────────────────────────────
+def _losses(seed=3, n=400):
+    rng = np.random.default_rng(seed)
+    base = rng.normal(0, 1, n)
+    return pd.DataFrame({
+        "A": 1.00 + base + rng.normal(0, 0.3, n),
+        "B": 1.02 + base + rng.normal(0, 0.3, n),
+        "C": 1.60 + base + rng.normal(0, 0.3, n),
+    })
+
+
+def test_mcs_drops_the_clearly_worse_model():
+    out = fl.model_confidence_set(_losses(), size=0.10, reps=2000, random_state=0)
+    assert not out.loc["C", "in_set"]
+    assert out.loc["A", "in_set"] and out.loc["B", "in_set"]
+    assert out["pvalue"].is_monotonic_increasing
+
+
+def test_mcs_agrees_with_arch():
+    arch = pytest.importorskip("arch.bootstrap")
+    L = _losses()
+    ours = fl.model_confidence_set(L, size=0.10, reps=2000, block_size=8, random_state=0)
+    ref = arch.MCS(L, size=0.10, reps=2000, block_size=8, method="max", bootstrap="mbb", seed=0)
+    ref.compute()
+    assert set(ours.index[ours["in_set"]]) == set(ref.included)
+
+
+# ── splitter ────────────────────────────────────────────────────────────────
+def _panel(n_groups=20, n_times=30):
+    g = np.repeat(np.arange(n_groups), n_times)
+    t = np.tile(np.arange(n_times), n_groups)
+    return g, t
+
+
+def test_group_time_split_holds_out_future_and_new_groups():
+    g, t = _panel()
+    cv = fl.GroupTimeSplit(t, n_splits=3, gap=2, test_group_frac=0.25, random_state=0)
+    folds = list(cv.split(np.zeros(len(g)), groups=g))
+    assert len(folds) == 3
+    for train, test in folds:
+        assert not set(g[train]) & set(g[test])
+        assert t[train].max() + 2 < t[test].min()
+
+
+def test_time_only_split_keeps_all_groups():
+    g, t = _panel()
+    cv = fl.GroupTimeSplit(t, n_splits=3, holdout="none")
+    for train, test in cv.split(np.zeros(len(g))):
+        assert set(g[test]) <= set(g[train])
+        assert t[train].max() < t[test].min()
+
+
+def test_group_time_split_validates_inputs():
+    g, t = _panel()
+    with pytest.raises(ValueError):
+        fl.GroupTimeSplit(t, holdout="bogus")
+    with pytest.raises(ValueError, match="groups"):
+        next(fl.GroupTimeSplit(t).split(np.zeros(len(g))))
+
+
+# ── split optimism: a leak through the entity shows up ──────────────────────
+def test_split_optimism_exposes_entity_leak():
+    rng = np.random.default_rng(4)
+    g, t = _panel(n_groups=40, n_times=24)
+    level = rng.normal(0, 5, 40)
+    y = level[g] + 0.05 * t + rng.normal(0, 0.5, len(g))
+    X = np.column_stack([g, t])  # the group code lets a model memorise each entity
+    model = RandomForestRegressor(n_estimators=50, random_state=0)
+    out = fl.split_optimism(model, X, y, groups=g, time=t, n_splits=3)
+    assert out.loc["random", "vs_reference"] < 0.5   # random CV hides most of the error
+    assert out.loc["group", "vs_reference"] > 0.7