idaniloju 0.1.0__tar.gz

idaniloju-0.1.0/PKG-INFO

@@ -0,0 +1,168 @@
+Metadata-Version: 2.4
+Name: idaniloju
+Version: 0.1.0
+Summary: Conformal prediction from scratch: the exact finite-sample guarantee, what drift does to it, and the online repair
+Author-email: Kenny Obidele <obidelek19@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/Kenny0bi/idaniloju
+Project-URL: Issues, https://github.com/Kenny0bi/idaniloju/issues
+Keywords: conformal-prediction,uncertainty-quantification,prediction-intervals,distribution-free,drift
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.24
+
+# idaniloju
+
+Conformal prediction from scratch: the exact finite-sample guarantee,
+proof by experiment, the deployment that breaks it, and the online
+repair. The tenth and last project of this series, and it closes the
+loop: the drift that kills the guarantee here is detected by
+[iyipada](https://github.com/Kenny0bi/iyipada), my drift library,
+installed from PyPI like any other user would.
+
+```
+pip install idaniloju
+```
+
+*Idaniloju* is Yoruba for assurance. Conformal prediction is the cleanest
+promise in machine learning: wrap ANY point predictor, and its intervals
+will contain the truth 90% of the time, guaranteed at finite sample size
+with no distributional assumptions at all. The entire method is one
+counting argument, and this repository walks it from the theorem to the
+hurricane that broke it.
+
+## The guarantee is a law, not a hope
+
+![The law](assets/law.svg)
+
+600 fresh train/calibrate/test splits of the same data. Conditional
+coverage is not just ">= 90% on average": it has a known exact
+distribution, Beta(n+1-l, l) over the calibration draw (Vovk 2012). The
+600-run empirical mean lands on the law's mean to the fourth digit
+(0.9001 vs 0.9001), and the histogram sits inside the density like it
+was poured there. Also visible: 47% of individual runs are below 0.90,
+which is exactly what the law says should happen, and exactly the fine
+print people miss.
+
+![The animation](assets/ranks.gif)
+
+The whole proof: an exchangeable point's score has uniform rank among
+the n+1 scores, so it falls at or below the ceil((n+1)(1-alpha))-th
+calibration score with probability at least 1-alpha. Counting, nothing
+else. (Source: [assets/manim_ranks.py](assets/manim_ranks.py), video in
+[assets/ranks.mp4](assets/ranks.mp4).)
+
+## Then you deploy it, and the fine print detonates
+
+Train on early 2011 of the Washington DC bike-sharing panel, calibrate
+on late 2011, then serve every hour of 2012 in order:
+
+![The break](assets/break.svg)
+
+2012 ridership grew about 65% over 2011. Exchangeability with the
+calibration set is simply false, and the guarantee follows it down:
+78% delivered against 90% promised for the year, and when Hurricane
+Sandy closes the city on October 29 (22 rides all day against a normal
+7,500), the rolling window bottoms out at 42%.
+
+Three things about that figure that I care about:
+
+- **iyipada alarms in the first week of 2012** (PSI 0.15 on the model's
+  conformity scores, p = 0.005), eleven months before Sandy. The drift
+  was structural, not meteorological.
+- **The shuffled control behaves**: run the same machinery on a
+  shuffled (exchangeable) version of the stream and coverage holds at
+  90.3% with zero alarm weeks out of 51. The alarm is not trigger-happy;
+  the guarantee did not break by accident.
+- **Adaptive conformal repairs it online.** ACI (Gibbs & Candes 2021)
+  moves the miscoverage dial with the observed errors,
+  alpha_t+1 = alpha_t + gamma(alpha - err_t), and ends the year at
+  89.97% against the 90% target, hurricane included, without ever
+  refitting the model.
+
+## What the repair costs
+
+![The price](assets/price.svg)
+
+Nothing is free: the adaptive intervals are about 5% wider in the median
+hour, and for 36 hours around Sandy the dial demands more assurance than
+4,395 calibration points can certify, so the honest interval is
+unbounded. I consider that a feature: a system that says "this week I
+can promise nothing" is telling the truth, and the fixed system that
+kept quoting tidy intervals through a hurricane was not.
+
+![The anatomy](assets/anatomy.svg)
+
+## Using it
+
+```python
+from idaniloju import Ridge, SplitConformal, AdaptiveConformal, coverage
+
+sc = SplitConformal(Ridge(lam=5.0), alpha=0.1)   # any model with fit/predict
+sc.fit(X_train, y_train, X_cal, y_cal)
+lo, hi = sc.predict(X_new)
+
+aci = AdaptiveConformal(sc, gamma=0.01)           # for streams
+lo, hi, err = aci.step(x_t, y_t)
+```
+
+`SplitConformal` takes any object with fit/predict (a scale model makes
+the intervals difficulty-aware without touching the guarantee), and
+`conformal_quantile` is the exact order statistic the theorem is proved
+for, not an interpolated quantile: at small n and small alpha it
+returns an infinite interval instead of a fake finite one.
+
+## Honest limits
+
+- The guarantee is **marginal**: 90% over everything, not 90% for every
+  hour of the day or every kind of input. Group-conditional coverage
+  needs group-wise calibration this library does not implement.
+- ACI trades a broken assumption for a feedback loop: it needs y_t
+  observed reasonably soon after predicting. With delayed labels the
+  dial lags exactly that delay.
+- The base model here is a ridge regression on purpose (any model
+  works). Sharper models give narrower intervals; nothing about the
+  coverage story changes.
+- The 65% growth and the hurricane make this dataset a vivid drift
+  story, which is why I chose it; datasets with gentler drift break more
+  slowly and less photogenically.
+
+## Reproduce it
+
+```bash
+python -m venv .venv && .venv/bin/pip install numpy pytest iyipada
+bash data/get_data.sh                        # or let the loader fetch it
+
+.venv/bin/python -m pytest tests/ -q         # 6 contracts
+.venv/bin/python benchmarks/bench_guarantee.py   # the Beta law, 600 refits
+.venv/bin/python benchmarks/bench_stream.py      # the 2012 deployment
+.venv/bin/python assets/make_visuals.py          # the four figures
+```
+
+## Layout
+
+- [idaniloju/conformal.py](idaniloju/conformal.py) the method: exact
+  quantile, split conformal, difficulty scaling, ACI
+- [idaniloju/models.py](idaniloju/models.py) a closed-form ridge, the
+  deliberately boring predictor
+- [idaniloju/bike.py](idaniloju/bike.py) the panel, with the year column
+  deliberately withheld from the features (deployment does not get told
+  the world changed)
+- [benchmarks/](benchmarks/) the law and the deployment, everything into
+  JSON the figures and tests read
+
+## Papers
+
+- Vovk, Gammerman, Shafer (2005), *Algorithmic Learning in a Random
+  World*. The theory.
+- Lei et al. (2018), *Distribution-Free Predictive Inference for
+  Regression*, JASA. The split conformal recipe used here.
+- Vovk (2012), *Conditional validity of inductive conformal predictors*.
+  The Beta law the first figure draws.
+- Gibbs & Candes (2021), *Adaptive Conformal Inference Under
+  Distribution Shift*. The repair.

idaniloju-0.1.0/README.md

@@ -0,0 +1,150 @@
+# idaniloju
+
+Conformal prediction from scratch: the exact finite-sample guarantee,
+proof by experiment, the deployment that breaks it, and the online
+repair. The tenth and last project of this series, and it closes the
+loop: the drift that kills the guarantee here is detected by
+[iyipada](https://github.com/Kenny0bi/iyipada), my drift library,
+installed from PyPI like any other user would.
+
+```
+pip install idaniloju
+```
+
+*Idaniloju* is Yoruba for assurance. Conformal prediction is the cleanest
+promise in machine learning: wrap ANY point predictor, and its intervals
+will contain the truth 90% of the time, guaranteed at finite sample size
+with no distributional assumptions at all. The entire method is one
+counting argument, and this repository walks it from the theorem to the
+hurricane that broke it.
+
+## The guarantee is a law, not a hope
+
+![The law](assets/law.svg)
+
+600 fresh train/calibrate/test splits of the same data. Conditional
+coverage is not just ">= 90% on average": it has a known exact
+distribution, Beta(n+1-l, l) over the calibration draw (Vovk 2012). The
+600-run empirical mean lands on the law's mean to the fourth digit
+(0.9001 vs 0.9001), and the histogram sits inside the density like it
+was poured there. Also visible: 47% of individual runs are below 0.90,
+which is exactly what the law says should happen, and exactly the fine
+print people miss.
+
+![The animation](assets/ranks.gif)
+
+The whole proof: an exchangeable point's score has uniform rank among
+the n+1 scores, so it falls at or below the ceil((n+1)(1-alpha))-th
+calibration score with probability at least 1-alpha. Counting, nothing
+else. (Source: [assets/manim_ranks.py](assets/manim_ranks.py), video in
+[assets/ranks.mp4](assets/ranks.mp4).)
+
+## Then you deploy it, and the fine print detonates
+
+Train on early 2011 of the Washington DC bike-sharing panel, calibrate
+on late 2011, then serve every hour of 2012 in order:
+
+![The break](assets/break.svg)
+
+2012 ridership grew about 65% over 2011. Exchangeability with the
+calibration set is simply false, and the guarantee follows it down:
+78% delivered against 90% promised for the year, and when Hurricane
+Sandy closes the city on October 29 (22 rides all day against a normal
+7,500), the rolling window bottoms out at 42%.
+
+Three things about that figure that I care about:
+
+- **iyipada alarms in the first week of 2012** (PSI 0.15 on the model's
+  conformity scores, p = 0.005), eleven months before Sandy. The drift
+  was structural, not meteorological.
+- **The shuffled control behaves**: run the same machinery on a
+  shuffled (exchangeable) version of the stream and coverage holds at
+  90.3% with zero alarm weeks out of 51. The alarm is not trigger-happy;
+  the guarantee did not break by accident.
+- **Adaptive conformal repairs it online.** ACI (Gibbs & Candes 2021)
+  moves the miscoverage dial with the observed errors,
+  alpha_t+1 = alpha_t + gamma(alpha - err_t), and ends the year at
+  89.97% against the 90% target, hurricane included, without ever
+  refitting the model.
+
+## What the repair costs
+
+![The price](assets/price.svg)
+
+Nothing is free: the adaptive intervals are about 5% wider in the median
+hour, and for 36 hours around Sandy the dial demands more assurance than
+4,395 calibration points can certify, so the honest interval is
+unbounded. I consider that a feature: a system that says "this week I
+can promise nothing" is telling the truth, and the fixed system that
+kept quoting tidy intervals through a hurricane was not.
+
+![The anatomy](assets/anatomy.svg)
+
+## Using it
+
+```python
+from idaniloju import Ridge, SplitConformal, AdaptiveConformal, coverage
+
+sc = SplitConformal(Ridge(lam=5.0), alpha=0.1)   # any model with fit/predict
+sc.fit(X_train, y_train, X_cal, y_cal)
+lo, hi = sc.predict(X_new)
+
+aci = AdaptiveConformal(sc, gamma=0.01)           # for streams
+lo, hi, err = aci.step(x_t, y_t)
+```
+
+`SplitConformal` takes any object with fit/predict (a scale model makes
+the intervals difficulty-aware without touching the guarantee), and
+`conformal_quantile` is the exact order statistic the theorem is proved
+for, not an interpolated quantile: at small n and small alpha it
+returns an infinite interval instead of a fake finite one.
+
+## Honest limits
+
+- The guarantee is **marginal**: 90% over everything, not 90% for every
+  hour of the day or every kind of input. Group-conditional coverage
+  needs group-wise calibration this library does not implement.
+- ACI trades a broken assumption for a feedback loop: it needs y_t
+  observed reasonably soon after predicting. With delayed labels the
+  dial lags exactly that delay.
+- The base model here is a ridge regression on purpose (any model
+  works). Sharper models give narrower intervals; nothing about the
+  coverage story changes.
+- The 65% growth and the hurricane make this dataset a vivid drift
+  story, which is why I chose it; datasets with gentler drift break more
+  slowly and less photogenically.
+
+## Reproduce it
+
+```bash
+python -m venv .venv && .venv/bin/pip install numpy pytest iyipada
+bash data/get_data.sh                        # or let the loader fetch it
+
+.venv/bin/python -m pytest tests/ -q         # 6 contracts
+.venv/bin/python benchmarks/bench_guarantee.py   # the Beta law, 600 refits
+.venv/bin/python benchmarks/bench_stream.py      # the 2012 deployment
+.venv/bin/python assets/make_visuals.py          # the four figures
+```
+
+## Layout
+
+- [idaniloju/conformal.py](idaniloju/conformal.py) the method: exact
+  quantile, split conformal, difficulty scaling, ACI
+- [idaniloju/models.py](idaniloju/models.py) a closed-form ridge, the
+  deliberately boring predictor
+- [idaniloju/bike.py](idaniloju/bike.py) the panel, with the year column
+  deliberately withheld from the features (deployment does not get told
+  the world changed)
+- [benchmarks/](benchmarks/) the law and the deployment, everything into
+  JSON the figures and tests read
+
+## Papers
+
+- Vovk, Gammerman, Shafer (2005), *Algorithmic Learning in a Random
+  World*. The theory.
+- Lei et al. (2018), *Distribution-Free Predictive Inference for
+  Regression*, JASA. The split conformal recipe used here.
+- Vovk (2012), *Conditional validity of inductive conformal predictors*.
+  The Beta law the first figure draws.
+- Gibbs & Candes (2021), *Adaptive Conformal Inference Under
+  Distribution Shift*. The repair.

idaniloju-0.1.0/idaniloju/__init__.py

@@ -0,0 +1,16 @@
+"""idaniloju: conformal prediction from scratch, and what drift does to it.
+
+*Idaniloju* is Yoruba for assurance. Conformal prediction turns any point
+predictor into interval predictions with a finite-sample coverage
+guarantee that assumes nothing about the data's distribution, only that
+calibration and future points are exchangeable. That one assumption is
+also the method's weak joint: break it with drift and the guarantee
+quietly dies. This library ships the estimator, the proof-by-experiment,
+the failure, and the online repair.
+"""
+
+from .conformal import SplitConformal, AdaptiveConformal, coverage
+from .models import Ridge
+
+__version__ = "0.1.0"
+__all__ = ["SplitConformal", "AdaptiveConformal", "Ridge", "coverage"]

idaniloju-0.1.0/idaniloju/bike.py

@@ -0,0 +1,69 @@
+"""The Washington DC bike-sharing panel (UCI 275): 17,379 hours over
+2011-2012, with weather. Chosen because its drift is real, not
+simulated: 2012 ridership grew about 65% over 2011, and no model trained
+on 2011 has seen that world.
+
+The year indicator is deliberately excluded from the features: a
+deployed model does not get told "the world has changed" as a column.
+"""
+
+import csv
+import io
+import urllib.request
+import zipfile
+from pathlib import Path
+
+import numpy as np
+
+DATA = Path(__file__).resolve().parents[1] / "data" / "hour.csv"
+URL = "https://archive.ics.uci.edu/static/public/275/bike+sharing+dataset.zip"
+
+
+def _fetch():
+    """Installed from PyPI there is no data/ folder; fetch to a cache."""
+    cache = Path.home() / ".cache" / "idaniloju" / "hour.csv"
+    if not cache.exists():
+        cache.parent.mkdir(parents=True, exist_ok=True)
+        with urllib.request.urlopen(URL, timeout=120) as r:
+            z = zipfile.ZipFile(io.BytesIO(r.read()))
+        cache.write_bytes(z.read("hour.csv"))
+    return cache
+
+
+def load(path=None):
+    p = Path(path) if path else (DATA if DATA.exists() else _fetch())
+    rows = list(csv.DictReader(open(p, encoding="utf-8")))
+    n = len(rows)
+
+    def f(name):
+        return np.array([float(r[name]) for r in rows])
+
+    hr, wd = f("hr"), f("weekday")
+    season = f("season").astype(int)
+    weather = f("weathersit").astype(int)
+    feats = [
+        np.sin(2 * np.pi * wd / 7), np.cos(2 * np.pi * wd / 7),
+        f("temp"), f("atemp"), f("hum"), f("windspeed"),
+        f("workingday"), f("holiday"),
+    ]
+    names = ["wd_sin", "wd_cos", "temp", "atemp", "hum", "windspeed",
+             "workingday", "holiday"]
+    hh = hr.astype(int)
+    for h in range(1, 24):        # hour-of-day dummies: the ridership
+        feats.append((hh == h).astype(float))   # pattern is not sinusoidal
+        names.append(f"hr_{h}")
+    # working-day x hour interactions for the commute spikes
+    work = f("workingday")
+    for h in (7, 8, 9, 17, 18, 19):
+        feats.append(((hh == h) * work).astype(float))
+        names.append(f"work_hr_{h}")
+    for s in (2, 3, 4):
+        feats.append((season == s).astype(float))
+        names.append(f"season_{s}")
+    for w in (2, 3, 4):
+        feats.append((weather == w).astype(float))
+        names.append(f"weather_{w}")
+    X = np.column_stack(feats)
+    y = np.log1p(f("cnt"))
+    dates = np.array([r["dteday"] for r in rows])
+    return X, y, dates, names

idaniloju-0.1.0/idaniloju/conformal.py

@@ -0,0 +1,104 @@
+"""Split conformal prediction, and the online repair for when drift
+breaks it.
+
+The whole method is a counting argument. Hold out n calibration points,
+score each by how wrong the model was (here |y - yhat|, optionally
+scaled by a difficulty estimate), and let q be the
+ceil((n+1)(1-alpha))-th smallest score. If a new point is exchangeable
+with the calibration points, its score's rank among all n+1 is uniform,
+so
+
+    P(score_new <= q) >= (1 - alpha)
+
+exactly, for any model, any distribution, any finite n. No asymptotics,
+no Gaussians. The price is the assumption of exchangeability, which
+drift destroys; AdaptiveConformal (Gibbs & Candes 2021) repairs coverage
+online by moving alpha with the observed errors.
+"""
+
+import numpy as np
+
+
+def conformal_quantile(scores, alpha):
+    """The ceil((n+1)(1-alpha))/n empirical quantile, the exact one the
+    guarantee is proved for (not numpy's interpolated quantiles)."""
+    s = np.sort(np.asarray(scores, float))
+    n = len(s)
+    k = int(np.ceil((n + 1) * (1 - alpha)))
+    if k > n:
+        return np.inf  # not enough calibration data for this alpha
+    return float(s[k - 1])
+
+
+def coverage(y, lo, hi):
+    y = np.asarray(y, float)
+    return float(np.mean((y >= np.asarray(lo)) & (y <= np.asarray(hi))))
+
+
+class SplitConformal:
+    """Intervals yhat +/- q, with q from a held-out calibration set.
+
+    scale_model, if given, is fit to |residual| on the calibration set's
+    training half and divides the scores, so hard inputs get wide
+    intervals and easy ones narrow (locally weighted conformal). The
+    guarantee is unchanged: it never depends on either model being good.
+    """
+
+    def __init__(self, model, alpha=0.1, scale_model=None):
+        self.model = model
+        self.alpha = alpha
+        self.scale_model = scale_model
+
+    def _sigma(self, X):
+        if self.scale_model is None:
+            return np.ones(len(X))
+        return np.maximum(self.scale_model.predict(X), 1e-6)
+
+    def fit(self, X_train, y_train, X_cal, y_cal):
+        self.model.fit(X_train, y_train)
+        if self.scale_model is not None:
+            resid = np.abs(y_train - self.model.predict(X_train))
+            self.scale_model.fit(X_train, resid)
+        scores = (np.abs(y_cal - self.model.predict(X_cal))
+                  / self._sigma(X_cal))
+        self.scores_ = np.sort(scores)
+        self.q_ = conformal_quantile(scores, self.alpha)
+        return self
+
+    def predict(self, X, alpha=None):
+        q = (self.q_ if alpha is None
+             else conformal_quantile(self.scores_, alpha))
+        yhat = self.model.predict(X)
+        w = q * self._sigma(X)
+        return yhat - w, yhat + w
+
+
+class AdaptiveConformal:
+    """Adaptive conformal inference on a stream (Gibbs & Candes 2021).
+
+    Keeps the fitted SplitConformal but retunes its miscoverage level
+    online:
+
+        alpha_{t+1} = alpha_t + gamma * (alpha_target - err_t)
+
+    with err_t = 1 when y_t misses the interval. Coverage errors above
+    target push alpha down (wider intervals), quiet stretches push it
+    back up. The long-run error rate converges to alpha_target for ANY
+    sequence, even adversarial: the guarantee moves from exchangeability
+    to online feedback.
+    """
+
+    def __init__(self, conformal, gamma=0.005):
+        self.c = conformal
+        self.gamma = gamma
+        self.alpha_t = conformal.alpha
+        self.history = []
+
+    def step(self, x, y):
+        a = float(np.clip(self.alpha_t, 1e-4, 0.999))
+        lo, hi = self.c.predict(x[None, :], alpha=a)
+        err = float(not (lo[0] <= y <= hi[0]))
+        self.alpha_t += self.gamma * (self.c.alpha - err)
+        self.history.append({"alpha": a, "lo": float(lo[0]),
+                             "hi": float(hi[0]), "err": err})
+        return lo[0], hi[0], err

idaniloju-0.1.0/idaniloju/models.py

@@ -0,0 +1,30 @@
+"""A plain ridge regression, closed form, numpy only.
+
+Conformal prediction wraps any point predictor, so the predictor here is
+deliberately boring: standardized features, an intercept, and one line of
+linear algebra. The interesting guarantees live in conformal.py.
+"""
+
+import numpy as np
+
+
+class Ridge:
+    def __init__(self, lam=1.0):
+        self.lam = lam
+
+    def fit(self, X, y):
+        X = np.asarray(X, float)
+        y = np.asarray(y, float)
+        self.mu_ = X.mean(0)
+        self.sd_ = X.std(0)
+        self.sd_[self.sd_ == 0] = 1.0
+        Xs = (X - self.mu_) / self.sd_
+        n, d = Xs.shape
+        A = Xs.T @ Xs + self.lam * np.eye(d)
+        self.coef_ = np.linalg.solve(A, Xs.T @ (y - y.mean()))
+        self.intercept_ = float(y.mean())
+        return self
+
+    def predict(self, X):
+        Xs = (np.asarray(X, float) - self.mu_) / self.sd_
+        return Xs @ self.coef_ + self.intercept_

idaniloju-0.1.0/idaniloju.egg-info/PKG-INFO

@@ -0,0 +1,168 @@
+Metadata-Version: 2.4
+Name: idaniloju
+Version: 0.1.0
+Summary: Conformal prediction from scratch: the exact finite-sample guarantee, what drift does to it, and the online repair
+Author-email: Kenny Obidele <obidelek19@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/Kenny0bi/idaniloju
+Project-URL: Issues, https://github.com/Kenny0bi/idaniloju/issues
+Keywords: conformal-prediction,uncertainty-quantification,prediction-intervals,distribution-free,drift
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.24
+
+# idaniloju
+
+Conformal prediction from scratch: the exact finite-sample guarantee,
+proof by experiment, the deployment that breaks it, and the online
+repair. The tenth and last project of this series, and it closes the
+loop: the drift that kills the guarantee here is detected by
+[iyipada](https://github.com/Kenny0bi/iyipada), my drift library,
+installed from PyPI like any other user would.
+
+```
+pip install idaniloju
+```
+
+*Idaniloju* is Yoruba for assurance. Conformal prediction is the cleanest
+promise in machine learning: wrap ANY point predictor, and its intervals
+will contain the truth 90% of the time, guaranteed at finite sample size
+with no distributional assumptions at all. The entire method is one
+counting argument, and this repository walks it from the theorem to the
+hurricane that broke it.
+
+## The guarantee is a law, not a hope
+
+![The law](assets/law.svg)
+
+600 fresh train/calibrate/test splits of the same data. Conditional
+coverage is not just ">= 90% on average": it has a known exact
+distribution, Beta(n+1-l, l) over the calibration draw (Vovk 2012). The
+600-run empirical mean lands on the law's mean to the fourth digit
+(0.9001 vs 0.9001), and the histogram sits inside the density like it
+was poured there. Also visible: 47% of individual runs are below 0.90,
+which is exactly what the law says should happen, and exactly the fine
+print people miss.
+
+![The animation](assets/ranks.gif)
+
+The whole proof: an exchangeable point's score has uniform rank among
+the n+1 scores, so it falls at or below the ceil((n+1)(1-alpha))-th
+calibration score with probability at least 1-alpha. Counting, nothing
+else. (Source: [assets/manim_ranks.py](assets/manim_ranks.py), video in
+[assets/ranks.mp4](assets/ranks.mp4).)
+
+## Then you deploy it, and the fine print detonates
+
+Train on early 2011 of the Washington DC bike-sharing panel, calibrate
+on late 2011, then serve every hour of 2012 in order:
+
+![The break](assets/break.svg)
+
+2012 ridership grew about 65% over 2011. Exchangeability with the
+calibration set is simply false, and the guarantee follows it down:
+78% delivered against 90% promised for the year, and when Hurricane
+Sandy closes the city on October 29 (22 rides all day against a normal
+7,500), the rolling window bottoms out at 42%.
+
+Three things about that figure that I care about:
+
+- **iyipada alarms in the first week of 2012** (PSI 0.15 on the model's
+  conformity scores, p = 0.005), eleven months before Sandy. The drift
+  was structural, not meteorological.
+- **The shuffled control behaves**: run the same machinery on a
+  shuffled (exchangeable) version of the stream and coverage holds at
+  90.3% with zero alarm weeks out of 51. The alarm is not trigger-happy;
+  the guarantee did not break by accident.
+- **Adaptive conformal repairs it online.** ACI (Gibbs & Candes 2021)
+  moves the miscoverage dial with the observed errors,
+  alpha_t+1 = alpha_t + gamma(alpha - err_t), and ends the year at
+  89.97% against the 90% target, hurricane included, without ever
+  refitting the model.
+
+## What the repair costs
+
+![The price](assets/price.svg)
+
+Nothing is free: the adaptive intervals are about 5% wider in the median
+hour, and for 36 hours around Sandy the dial demands more assurance than
+4,395 calibration points can certify, so the honest interval is
+unbounded. I consider that a feature: a system that says "this week I
+can promise nothing" is telling the truth, and the fixed system that
+kept quoting tidy intervals through a hurricane was not.
+
+![The anatomy](assets/anatomy.svg)
+
+## Using it
+
+```python
+from idaniloju import Ridge, SplitConformal, AdaptiveConformal, coverage
+
+sc = SplitConformal(Ridge(lam=5.0), alpha=0.1)   # any model with fit/predict
+sc.fit(X_train, y_train, X_cal, y_cal)
+lo, hi = sc.predict(X_new)
+
+aci = AdaptiveConformal(sc, gamma=0.01)           # for streams
+lo, hi, err = aci.step(x_t, y_t)
+```
+
+`SplitConformal` takes any object with fit/predict (a scale model makes
+the intervals difficulty-aware without touching the guarantee), and
+`conformal_quantile` is the exact order statistic the theorem is proved
+for, not an interpolated quantile: at small n and small alpha it
+returns an infinite interval instead of a fake finite one.
+
+## Honest limits
+
+- The guarantee is **marginal**: 90% over everything, not 90% for every
+  hour of the day or every kind of input. Group-conditional coverage
+  needs group-wise calibration this library does not implement.
+- ACI trades a broken assumption for a feedback loop: it needs y_t
+  observed reasonably soon after predicting. With delayed labels the
+  dial lags exactly that delay.
+- The base model here is a ridge regression on purpose (any model
+  works). Sharper models give narrower intervals; nothing about the
+  coverage story changes.
+- The 65% growth and the hurricane make this dataset a vivid drift
+  story, which is why I chose it; datasets with gentler drift break more
+  slowly and less photogenically.
+
+## Reproduce it
+
+```bash
+python -m venv .venv && .venv/bin/pip install numpy pytest iyipada
+bash data/get_data.sh                        # or let the loader fetch it
+
+.venv/bin/python -m pytest tests/ -q         # 6 contracts
+.venv/bin/python benchmarks/bench_guarantee.py   # the Beta law, 600 refits
+.venv/bin/python benchmarks/bench_stream.py      # the 2012 deployment
+.venv/bin/python assets/make_visuals.py          # the four figures
+```
+
+## Layout
+
+- [idaniloju/conformal.py](idaniloju/conformal.py) the method: exact
+  quantile, split conformal, difficulty scaling, ACI
+- [idaniloju/models.py](idaniloju/models.py) a closed-form ridge, the
+  deliberately boring predictor
+- [idaniloju/bike.py](idaniloju/bike.py) the panel, with the year column
+  deliberately withheld from the features (deployment does not get told
+  the world changed)
+- [benchmarks/](benchmarks/) the law and the deployment, everything into
+  JSON the figures and tests read
+
+## Papers
+
+- Vovk, Gammerman, Shafer (2005), *Algorithmic Learning in a Random
+  World*. The theory.
+- Lei et al. (2018), *Distribution-Free Predictive Inference for
+  Regression*, JASA. The split conformal recipe used here.
+- Vovk (2012), *Conditional validity of inductive conformal predictors*.
+  The Beta law the first figure draws.
+- Gibbs & Candes (2021), *Adaptive Conformal Inference Under
+  Distribution Shift*. The repair.

idaniloju-0.1.0/idaniloju.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+README.md
+pyproject.toml
+idaniloju/__init__.py
+idaniloju/bike.py
+idaniloju/conformal.py
+idaniloju/models.py
+idaniloju.egg-info/PKG-INFO
+idaniloju.egg-info/SOURCES.txt
+idaniloju.egg-info/dependency_links.txt
+idaniloju.egg-info/requires.txt
+idaniloju.egg-info/top_level.txt
+tests/test_idaniloju.py

idaniloju-0.1.0/idaniloju.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

idaniloju-0.1.0/idaniloju.egg-info/requires.txt

@@ -0,0 +1 @@
+numpy>=1.24

idaniloju-0.1.0/idaniloju.egg-info/top_level.txt

@@ -0,0 +1 @@
+idaniloju

idaniloju-0.1.0/pyproject.toml

@@ -0,0 +1,29 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "idaniloju"
+version = "0.1.0"
+description = "Conformal prediction from scratch: the exact finite-sample guarantee, what drift does to it, and the online repair"
+readme = "README.md"
+requires-python = ">=3.9"
+license = {text = "MIT"}
+authors = [{name = "Kenny Obidele", email = "obidelek19@gmail.com"}]
+keywords = ["conformal-prediction", "uncertainty-quantification",
+            "prediction-intervals", "distribution-free", "drift"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+dependencies = ["numpy>=1.24"]
+
+[project.urls]
+Homepage = "https://github.com/Kenny0bi/idaniloju"
+Issues = "https://github.com/Kenny0bi/idaniloju/issues"
+
+[tool.setuptools]
+packages = ["idaniloju"]

idaniloju-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

idaniloju-0.1.0/tests/test_idaniloju.py

@@ -0,0 +1,97 @@
+"""Contracts: the quantile the guarantee is proved for, the guarantee
+itself on synthetic data, ACI's error convergence, and the benchmark
+numbers once they exist."""
+
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from idaniloju import (AdaptiveConformal, Ridge, SplitConformal,  # noqa
+                       coverage)
+from idaniloju.conformal import conformal_quantile                # noqa
+
+STREAM = ROOT / "benchmarks" / "stream.json"
+GUAR = ROOT / "benchmarks" / "guarantee.json"
+
+
+def test_conformal_quantile_is_the_exact_order_statistic():
+    scores = np.arange(1, 11, dtype=float)   # 1..10, n = 10
+    # ceil(11 * 0.9) = 10 -> the 10th smallest = 10
+    assert conformal_quantile(scores, 0.1) == 10.0
+    # ceil(11 * 0.5) = 6 -> 6
+    assert conformal_quantile(scores, 0.5) == 6.0
+    # alpha so small that k > n: must refuse with an infinite interval
+    assert conformal_quantile(scores, 0.05) == np.inf
+
+
+def test_guarantee_holds_on_synthetic_exchangeable_data():
+    rng = np.random.default_rng(11)
+    covs = []
+    for _ in range(200):
+        X = rng.normal(0, 1, (700, 3))
+        y = X @ np.array([2.0, -1.0, 0.5]) + rng.standard_t(3, 700)
+        sc = SplitConformal(Ridge(lam=1.0), alpha=0.2)
+        sc.fit(X[:300], y[:300], X[300:500], y[300:500])
+        lo, hi = sc.predict(X[500:])
+        covs.append(coverage(y[500:], lo, hi))
+    # marginal coverage >= 1 - alpha in expectation; with 200 trials the
+    # mean should sit within a point of the target band
+    assert np.mean(covs) >= 0.79
+
+
+def test_aci_error_rate_converges_even_under_shift():
+    rng = np.random.default_rng(5)
+    X = rng.normal(0, 1, (3000, 2))
+    y = X @ np.array([1.0, 1.0]) + rng.normal(0, 1, 3000)
+    sc = SplitConformal(Ridge(lam=1.0), alpha=0.1)
+    sc.fit(X[:1000], y[:1000], X[1000:1500], y[1000:1500])
+    # the stream drifts hard: outputs shift by +3 halfway through
+    Xs, ys = X[1500:], y[1500:].copy()
+    ys[750:] += 3.0
+    aci = AdaptiveConformal(sc, gamma=0.02)
+    errs = [aci.step(Xs[i], ys[i])[2] for i in range(len(ys))]
+    # long-run error close to alpha despite the jump
+    assert abs(np.mean(errs) - 0.1) < 0.03
+
+
+def test_scale_model_widens_hard_inputs():
+    rng = np.random.default_rng(2)
+    X = rng.uniform(-1, 1, (2000, 1))
+    y = (X[:, 0] * 2 + rng.normal(0, 0.2 + 1.3 * (X[:, 0] > 0), 2000))
+    sc = SplitConformal(Ridge(0.1), alpha=0.1, scale_model=Ridge(0.1))
+    sc.fit(X[:800], y[:800], X[800:1400], y[800:1400])
+    lo, hi = sc.predict(X[1400:])
+    w = hi - lo
+    right = X[1400:, 0] > 0
+    assert w[right].mean() > 1.5 * w[~right].mean()
+    assert coverage(y[1400:], lo, hi) > 0.85
+
+
+@pytest.mark.skipif(not STREAM.exists(), reason="run bench_stream first")
+def test_stream_results_hold():
+    s = json.loads(STREAM.read_text())
+    assert s["static_coverage"] < 0.80          # drift broke the guarantee
+    assert abs(s["aci_coverage"] - 0.9) < 0.01  # ACI repaired it
+    assert s["first_alarm"]["start"] == "2012-01-01"  # iyipada saw it
+    ex = s["exchangeable_check"]
+    assert ex["weeks_alarmed"] == 0             # and does not cry wolf
+    assert ex["coverage"] > 0.89
+
+
+@pytest.mark.skipif(not GUAR.exists(), reason="run bench_guarantee first")
+def test_coverage_distribution_matches_beta_law():
+    g = json.loads(GUAR.read_text())
+    covs = np.array(g["coverages"])
+    a, b = g["beta_a"], g["beta_b"]
+    beta_mean = a / (a + b)
+    beta_sd = np.sqrt(a * b / ((a + b) ** 2 * (a + b + 1)))
+    assert abs(covs.mean() - beta_mean) < 0.004
+    # spread of the same order as the law predicts (test-set noise adds a
+    # little on top of the Beta's own width)
+    assert beta_sd * 0.8 < covs.std() < beta_sd * 2.2