iyipada 0.1.0__tar.gz

iyipada-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kenny Obidele
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

iyipada-0.1.0/PKG-INFO

@@ -0,0 +1,242 @@
+Metadata-Version: 2.4
+Name: iyipada
+Version: 0.1.0
+Summary: Drift detection with calibrated thresholds instead of folklore ones
+Author: Kenny Obidele
+License: MIT
+Project-URL: Homepage, https://github.com/Kenny0bi/iyipada
+Keywords: drift,monitoring,mlops,statistics,psi,data-drift
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: scipy>=1.10
+Provides-Extra: benchmarks
+Requires-Dist: pandas>=2.0; extra == "benchmarks"
+Dynamic: license-file
+
+# iyipada
+
+Drift detection where the threshold is calculated instead of quoted.
+
+Production drift monitoring runs on numbers that came from a blog post. The
+most common rule in the industry is "PSI above 0.10 means watch it, above 0.25
+means act". Those numbers are stated as though they were properties of the
+statistic. They are not. On data where **nothing has changed at all**, that
+first rule fires 85% of the time at 100 samples per batch, and it is
+mathematically incapable of firing at 1,000.
+
+This library computes the threshold from the null distribution, so a 1%
+false-alarm rate is a 1% false-alarm rate.
+
+![The threshold everyone uses is not a constant](assets/ruler.svg)
+
+*iyipada* is Yoruba for change.
+
+## The three ways a drift monitor lies to you
+
+Each of these is measured in this repository, not asserted.
+
+**One. A fixed threshold means different things at different sample sizes.**
+PSI under the null is distributed as `(1/n + 1/m) * chi2_{B-1}`, so it depends
+on both sample sizes and the bin count. The industry rule depends on neither.
+False-alarm rate of the "watch" rule on identically distributed data:
+
+| samples per batch | 5 bins | 10 bins | 20 bins |
+|---|---|---|---|
+| 50 | 64.3% | 98.2% | **100.0%** |
+| 100 | 29.5% | 85.1% | 100.0% |
+| 200 | 4.1% | 37.0% | 96.0% |
+| 500 | 0.0% | 0.3% | 17.6% |
+| 1,000 and up | 0.0% | 0.0% | 0.0% |
+
+The calibrated 1% threshold across that same grid runs from 1.4476 down to
+0.0004, a span of about 5,400. One number cannot serve all of it.
+
+**Two. Twenty features is twenty chances to be wrong.** Twenty clean features
+tested independently at alpha = 0.05 produce at least one alarm on 73% of clean
+batches. With Benjamini-Hochberg, 5%. Monitoring is a multiple-testing problem
+before it is a drift problem.
+
+**Three. Fifty-two weekly checks is not one check.** A test calibrated to 5%,
+run weekly, is wrong at least once during a year with probability 93%. Getting
+the single check right and then scheduling it re-creates the problem it solved.
+
+![A monitor that runs every week is not one test](assets/year.svg)
+
+## Using it
+
+```python
+from iyipada import Detector, Monitor
+
+d = Detector(reference_values, statistic="psi", alpha=0.01)
+r = d.check(this_weeks_values)
+
+r.drifted     # bool, decided against a calibrated threshold
+r.pvalue      # what you actually want to look at
+r.threshold   # computed from your sample sizes, not quoted from anywhere
+r.warning     # set when the batch is too small for the test to mean much
+```
+
+Nothing in the API accepts a raw divergence threshold, because a raw divergence
+threshold cannot be interpreted without knowing the sample sizes and the bin
+count.
+
+For a table of features, with the multiple testing handled:
+
+```python
+m = Monitor(reference_dataframe_as_dict, alpha=0.05, correction="fdr")
+result = m.check(live_dataframe_as_dict)
+result.drifted        # the features that moved, after FDR control
+```
+
+For a monitor that runs on a schedule, stop re-testing and accumulate:
+
+```python
+from iyipada import sequential as sq
+
+h, arl = sq.calibrate_run_length("cusum", target_arl=260)   # weekly checks
+# h = 3.75, one false alarm every 4.8 years of weekly monitoring
+c = sq.CUSUM(h=h)
+c.update(this_weeks_statistic)   # True when the evidence has accumulated
+```
+
+## Which detector should you actually use
+
+Wasserstein, in most cases. Every statistic below was first calibrated to the
+same 1% false-alarm rate, so this compares power and nothing else. Detection
+rate at 1,000 samples a side:
+
+| | mean shift | variance | skew | discretisation | mixture | tail only |
+|---|---|---|---|---|---|---|
+| **Wasserstein** | 100% | 100% | 100% | 100% | **99.5%** | **15.7%** |
+| PSI | 99.5% | 100% | 100% | 100% | 86.2% | 3.5% |
+| Jensen-Shannon | 99.5% | 100% | 100% | 100% | 85.8% | 3.5% |
+| chi-square | 99.5% | 100% | 100% | 100% | 85.7% | 3.5% |
+| Kolmogorov-Smirnov | 100% | 99.8% | 88.7% | 100% | 83.0% | 2.3% |
+| total variation | 98.3% | 100% | 99.8% | 100% | 59.7% | 2.8% |
+
+![Which detector actually catches which change](assets/power.svg)
+
+Wasserstein wins or ties everywhere, and it reports in the units of the
+variable, so a result reads as "the distribution moved by 0.3 mg/dL" rather
+than as an abstract divergence. PSI, the industry default, is beaten on the two
+hardest cases.
+
+### The blind spot
+
+![The change only one detector sees](assets/blindspot.svg)
+
+A small fraction of values jumping to the far tail is close to invisible. At 5%
+contamination Wasserstein catches it every time, PSI 29% of the time and
+Kolmogorov-Smirnov 18%. At 1.5% nothing is reliable.
+
+The binned statistics fail because quantile binning puts the reference's entire
+top decile into one bin, so moving mass from +2 to +6 does not change the
+histogram at all. KS fails because a few per cent in the far tail cannot move
+the largest vertical gap between two cumulative curves. Wasserstein survives
+because it measures horizontal distance.
+
+If rare extreme values are the failure you fear, monitor the tail directly with
+a quantile or an exceedance count. A distributional distance is the wrong
+instrument, however well calibrated.
+
+## What is verified, and how
+
+![The theory, checked against simulation](assets/nullcheck.svg)
+
+The asymptotic null is not taken on faith. Simulated 95th percentile divided by
+theory, across ten sample sizes and three bin counts, sits within 2% of 1.0 from
+200 samples upward. It is 6 to 9% optimistic around n = 100 and it breaks down
+at n = 50 with 20 bins, where each bin holds two or three points and the
+chi-square approximation has nothing to stand on.
+
+So the library uses the closed form above 300 samples and simulates the null
+below it, rather than trusting one answer everywhere. With that switch, the
+calibrated threshold holds between 0.0% and 2.2% actual false-alarm rate across
+the entire grid, against 0% to 100% for the fixed rule.
+
+![What those rules deliver](assets/alarms.svg)
+
+The ringed cell is 112 samples at 10 bins. That is a real case from
+[loom](https://github.com/Kenny0bi/loom), an earlier project of mine, where a
+fixed 0.10 rule fired a spurious retrain on identically distributed data. It
+fires there 78.6% of the time. That bug is why this library exists.
+
+## The score, animated
+
+![Why a fixed threshold cannot work](assets/null.gif)
+
+The null distribution of PSI collapsing as the sample grows, while the fixed
+rule stays exactly where it is. Every number in the animation comes from
+`benchmarks/false_alarms.py`. (Source:
+[assets/manim_null.py](assets/manim_null.py), video in
+[assets/null.mp4](assets/null.mp4).)
+
+## What is in here
+
+- [iyipada/divergence.py](iyipada/divergence.py) the statistics: PSI,
+  Jensen-Shannon, total variation, Kolmogorov-Smirnov, Wasserstein,
+  chi-square. None of them decides anything.
+- [iyipada/null.py](iyipada/null.py) the null distributions: analytic for PSI,
+  chi-square and KS, permutation for everything else, plus
+  `false_alarm_rate()` for measuring what a threshold you already use is
+  actually doing.
+- [iyipada/detect.py](iyipada/detect.py) `Detector` and `Monitor`, the latter
+  with FDR or Bonferroni control.
+- [iyipada/sequential.py](iyipada/sequential.py) CUSUM and Page-Hinkley
+  calibrated to an average run length, alpha spending across a known horizon,
+  and `detection_delay()` so a monitor that never false-alarms can be checked
+  for being merely deaf.
+- [benchmarks/](benchmarks/) the three studies behind every number above.
+
+## Honest limits
+
+- **Univariate only.** Correlation between features can shift while every
+  marginal stays put, and nothing here would see it. Multivariate drift needs a
+  different tool.
+- **The permutation path is slow.** Wasserstein and total variation have no
+  closed-form null here, so their thresholds are simulated. That is a few
+  seconds per feature, cached per sample size.
+- **Independence is assumed across checks.** The Sidak correction and the run
+  length calibration both assume successive batches are independent. Weekly
+  batches of the same seasonal process are not, and the true false-alarm rate
+  will be somewhat higher than advertised.
+- **Detecting drift is not detecting harm.** A feature can move a long way
+  without the model caring, and can stay still while the relationship between
+  it and the target rots. This library measures the first thing. Watching the
+  second needs labels.
+- **Below about 100 samples, nothing works.** The thresholds are honest there,
+  which mostly means they are honestly enormous. A detector that cannot fire is
+  correctly calibrated and useless. `Detector` says so in `r.warning`.
+
+## Running it
+
+```bash
+pip install numpy scipy pandas
+
+python tests/test_null.py         # 9 contracts on the null distributions
+python tests/test_detect.py       # 10 on the detector and monitor APIs
+python tests/test_sequential.py   # 8 on run lengths and detection delay
+
+python benchmarks/false_alarms.py # the headline table, about 7 minutes
+python benchmarks/power.py        # detector against drift type, about 4 minutes
+python assets/make_visuals.py     # the six figures
+```
+
+27 tests. The interesting ones assert the findings themselves, so if the
+industry thresholds ever stop being miscalibrated, the suite will say so.
+
+## Papers
+
+- B. Yurdakul (2018), *Statistical properties of population stability index*,
+  Western Michigan University dissertation. Derives the asymptotic
+  distribution, and states plainly that the 0.10 and 0.25 traffic-light values
+  have no support in the literature.
+- Bracher et al. and the wider proper-scoring literature for why a decision
+  rule needs a stated error rate rather than a threshold.
+- Page (1954), *Continuous inspection schemes*, Biometrika. The original CUSUM,
+  and the reason average run length is the right quantity for a monitor that
+  runs forever.
+- Benjamini and Hochberg (1995), *Controlling the false discovery rate*, JRSS-B.
+  What `Monitor` uses when you hand it fifty features.

iyipada-0.1.0/README.md

@@ -0,0 +1,225 @@
+# iyipada
+
+Drift detection where the threshold is calculated instead of quoted.
+
+Production drift monitoring runs on numbers that came from a blog post. The
+most common rule in the industry is "PSI above 0.10 means watch it, above 0.25
+means act". Those numbers are stated as though they were properties of the
+statistic. They are not. On data where **nothing has changed at all**, that
+first rule fires 85% of the time at 100 samples per batch, and it is
+mathematically incapable of firing at 1,000.
+
+This library computes the threshold from the null distribution, so a 1%
+false-alarm rate is a 1% false-alarm rate.
+
+![The threshold everyone uses is not a constant](assets/ruler.svg)
+
+*iyipada* is Yoruba for change.
+
+## The three ways a drift monitor lies to you
+
+Each of these is measured in this repository, not asserted.
+
+**One. A fixed threshold means different things at different sample sizes.**
+PSI under the null is distributed as `(1/n + 1/m) * chi2_{B-1}`, so it depends
+on both sample sizes and the bin count. The industry rule depends on neither.
+False-alarm rate of the "watch" rule on identically distributed data:
+
+| samples per batch | 5 bins | 10 bins | 20 bins |
+|---|---|---|---|
+| 50 | 64.3% | 98.2% | **100.0%** |
+| 100 | 29.5% | 85.1% | 100.0% |
+| 200 | 4.1% | 37.0% | 96.0% |
+| 500 | 0.0% | 0.3% | 17.6% |
+| 1,000 and up | 0.0% | 0.0% | 0.0% |
+
+The calibrated 1% threshold across that same grid runs from 1.4476 down to
+0.0004, a span of about 5,400. One number cannot serve all of it.
+
+**Two. Twenty features is twenty chances to be wrong.** Twenty clean features
+tested independently at alpha = 0.05 produce at least one alarm on 73% of clean
+batches. With Benjamini-Hochberg, 5%. Monitoring is a multiple-testing problem
+before it is a drift problem.
+
+**Three. Fifty-two weekly checks is not one check.** A test calibrated to 5%,
+run weekly, is wrong at least once during a year with probability 93%. Getting
+the single check right and then scheduling it re-creates the problem it solved.
+
+![A monitor that runs every week is not one test](assets/year.svg)
+
+## Using it
+
+```python
+from iyipada import Detector, Monitor
+
+d = Detector(reference_values, statistic="psi", alpha=0.01)
+r = d.check(this_weeks_values)
+
+r.drifted     # bool, decided against a calibrated threshold
+r.pvalue      # what you actually want to look at
+r.threshold   # computed from your sample sizes, not quoted from anywhere
+r.warning     # set when the batch is too small for the test to mean much
+```
+
+Nothing in the API accepts a raw divergence threshold, because a raw divergence
+threshold cannot be interpreted without knowing the sample sizes and the bin
+count.
+
+For a table of features, with the multiple testing handled:
+
+```python
+m = Monitor(reference_dataframe_as_dict, alpha=0.05, correction="fdr")
+result = m.check(live_dataframe_as_dict)
+result.drifted        # the features that moved, after FDR control
+```
+
+For a monitor that runs on a schedule, stop re-testing and accumulate:
+
+```python
+from iyipada import sequential as sq
+
+h, arl = sq.calibrate_run_length("cusum", target_arl=260)   # weekly checks
+# h = 3.75, one false alarm every 4.8 years of weekly monitoring
+c = sq.CUSUM(h=h)
+c.update(this_weeks_statistic)   # True when the evidence has accumulated
+```
+
+## Which detector should you actually use
+
+Wasserstein, in most cases. Every statistic below was first calibrated to the
+same 1% false-alarm rate, so this compares power and nothing else. Detection
+rate at 1,000 samples a side:
+
+| | mean shift | variance | skew | discretisation | mixture | tail only |
+|---|---|---|---|---|---|---|
+| **Wasserstein** | 100% | 100% | 100% | 100% | **99.5%** | **15.7%** |
+| PSI | 99.5% | 100% | 100% | 100% | 86.2% | 3.5% |
+| Jensen-Shannon | 99.5% | 100% | 100% | 100% | 85.8% | 3.5% |
+| chi-square | 99.5% | 100% | 100% | 100% | 85.7% | 3.5% |
+| Kolmogorov-Smirnov | 100% | 99.8% | 88.7% | 100% | 83.0% | 2.3% |
+| total variation | 98.3% | 100% | 99.8% | 100% | 59.7% | 2.8% |
+
+![Which detector actually catches which change](assets/power.svg)
+
+Wasserstein wins or ties everywhere, and it reports in the units of the
+variable, so a result reads as "the distribution moved by 0.3 mg/dL" rather
+than as an abstract divergence. PSI, the industry default, is beaten on the two
+hardest cases.
+
+### The blind spot
+
+![The change only one detector sees](assets/blindspot.svg)
+
+A small fraction of values jumping to the far tail is close to invisible. At 5%
+contamination Wasserstein catches it every time, PSI 29% of the time and
+Kolmogorov-Smirnov 18%. At 1.5% nothing is reliable.
+
+The binned statistics fail because quantile binning puts the reference's entire
+top decile into one bin, so moving mass from +2 to +6 does not change the
+histogram at all. KS fails because a few per cent in the far tail cannot move
+the largest vertical gap between two cumulative curves. Wasserstein survives
+because it measures horizontal distance.
+
+If rare extreme values are the failure you fear, monitor the tail directly with
+a quantile or an exceedance count. A distributional distance is the wrong
+instrument, however well calibrated.
+
+## What is verified, and how
+
+![The theory, checked against simulation](assets/nullcheck.svg)
+
+The asymptotic null is not taken on faith. Simulated 95th percentile divided by
+theory, across ten sample sizes and three bin counts, sits within 2% of 1.0 from
+200 samples upward. It is 6 to 9% optimistic around n = 100 and it breaks down
+at n = 50 with 20 bins, where each bin holds two or three points and the
+chi-square approximation has nothing to stand on.
+
+So the library uses the closed form above 300 samples and simulates the null
+below it, rather than trusting one answer everywhere. With that switch, the
+calibrated threshold holds between 0.0% and 2.2% actual false-alarm rate across
+the entire grid, against 0% to 100% for the fixed rule.
+
+![What those rules deliver](assets/alarms.svg)
+
+The ringed cell is 112 samples at 10 bins. That is a real case from
+[loom](https://github.com/Kenny0bi/loom), an earlier project of mine, where a
+fixed 0.10 rule fired a spurious retrain on identically distributed data. It
+fires there 78.6% of the time. That bug is why this library exists.
+
+## The score, animated
+
+![Why a fixed threshold cannot work](assets/null.gif)
+
+The null distribution of PSI collapsing as the sample grows, while the fixed
+rule stays exactly where it is. Every number in the animation comes from
+`benchmarks/false_alarms.py`. (Source:
+[assets/manim_null.py](assets/manim_null.py), video in
+[assets/null.mp4](assets/null.mp4).)
+
+## What is in here
+
+- [iyipada/divergence.py](iyipada/divergence.py) the statistics: PSI,
+  Jensen-Shannon, total variation, Kolmogorov-Smirnov, Wasserstein,
+  chi-square. None of them decides anything.
+- [iyipada/null.py](iyipada/null.py) the null distributions: analytic for PSI,
+  chi-square and KS, permutation for everything else, plus
+  `false_alarm_rate()` for measuring what a threshold you already use is
+  actually doing.
+- [iyipada/detect.py](iyipada/detect.py) `Detector` and `Monitor`, the latter
+  with FDR or Bonferroni control.
+- [iyipada/sequential.py](iyipada/sequential.py) CUSUM and Page-Hinkley
+  calibrated to an average run length, alpha spending across a known horizon,
+  and `detection_delay()` so a monitor that never false-alarms can be checked
+  for being merely deaf.
+- [benchmarks/](benchmarks/) the three studies behind every number above.
+
+## Honest limits
+
+- **Univariate only.** Correlation between features can shift while every
+  marginal stays put, and nothing here would see it. Multivariate drift needs a
+  different tool.
+- **The permutation path is slow.** Wasserstein and total variation have no
+  closed-form null here, so their thresholds are simulated. That is a few
+  seconds per feature, cached per sample size.
+- **Independence is assumed across checks.** The Sidak correction and the run
+  length calibration both assume successive batches are independent. Weekly
+  batches of the same seasonal process are not, and the true false-alarm rate
+  will be somewhat higher than advertised.
+- **Detecting drift is not detecting harm.** A feature can move a long way
+  without the model caring, and can stay still while the relationship between
+  it and the target rots. This library measures the first thing. Watching the
+  second needs labels.
+- **Below about 100 samples, nothing works.** The thresholds are honest there,
+  which mostly means they are honestly enormous. A detector that cannot fire is
+  correctly calibrated and useless. `Detector` says so in `r.warning`.
+
+## Running it
+
+```bash
+pip install numpy scipy pandas
+
+python tests/test_null.py         # 9 contracts on the null distributions
+python tests/test_detect.py       # 10 on the detector and monitor APIs
+python tests/test_sequential.py   # 8 on run lengths and detection delay
+
+python benchmarks/false_alarms.py # the headline table, about 7 minutes
+python benchmarks/power.py        # detector against drift type, about 4 minutes
+python assets/make_visuals.py     # the six figures
+```
+
+27 tests. The interesting ones assert the findings themselves, so if the
+industry thresholds ever stop being miscalibrated, the suite will say so.
+
+## Papers
+
+- B. Yurdakul (2018), *Statistical properties of population stability index*,
+  Western Michigan University dissertation. Derives the asymptotic
+  distribution, and states plainly that the 0.10 and 0.25 traffic-light values
+  have no support in the literature.
+- Bracher et al. and the wider proper-scoring literature for why a decision
+  rule needs a stated error rate rather than a threshold.
+- Page (1954), *Continuous inspection schemes*, Biometrika. The original CUSUM,
+  and the reason average run length is the right quantity for a monitor that
+  runs forever.
+- Benjamini and Hochberg (1995), *Controlling the false discovery rate*, JRSS-B.
+  What `Monitor` uses when you hand it fifty features.

iyipada-0.1.0/iyipada/__init__.py

@@ -0,0 +1,32 @@
+"""Drift detection where the threshold is calculated instead of quoted.
+
+    from iyipada import Detector
+    d = Detector(reference, statistic="wasserstein", alpha=0.01)
+    r = d.check(live)
+    r.drifted, r.pvalue, r.threshold
+
+The point of the library, in one sentence: a drift threshold is not a property
+of the statistic, it is a property of how much data you have, and every
+published rule of thumb ignores that.
+"""
+
+from .detect import Detector, Monitor, Result, TableResult
+from .divergence import (chi_square, jensen_shannon, ks_statistic, psi,
+                         quantile_edges, total_variation, wasserstein)
+from .null import (analytic_threshold, false_alarm_rate, permutation_null,
+                   psi_null, simulated_threshold, threshold)
+from .sequential import (CUSUM, PageHinkley, any_alarm_probability,
+                         calibrate_run_length, detection_delay,
+                         family_wise_alpha, run_length)
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "Detector", "Monitor", "Result", "TableResult",
+    "psi", "jensen_shannon", "total_variation", "ks_statistic",
+    "wasserstein", "chi_square", "quantile_edges",
+    "threshold", "analytic_threshold", "simulated_threshold",
+    "permutation_null", "psi_null", "false_alarm_rate",
+    "CUSUM", "PageHinkley", "calibrate_run_length", "run_length",
+    "detection_delay", "family_wise_alpha", "any_alarm_probability",
+]