abkit 0.1.0__tar.gz

abkit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hadi Mohammadi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

abkit-0.1.0/PKG-INFO

@@ -0,0 +1,262 @@
+Metadata-Version: 2.4
+Name: abkit
+Version: 0.1.0
+Summary: Audit an A/B-test readout before you ship the decision
+Author: Hadi Mohammadi
+License: MIT
+Project-URL: Homepage, https://github.com/mohammadi-hadi/abkit
+Project-URL: Issues, https://github.com/mohammadi-hadi/abkit/issues
+Keywords: ab-testing,experimentation,srm,statistics,audit
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.26
+Requires-Dist: pydantic>=2.5
+Requires-Dist: matplotlib>=3.8
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Requires-Dist: scipy>=1.12; extra == "dev"
+Requires-Dist: statsmodels>=0.14; extra == "dev"
+Dynamic: license-file
+
+# abkit
+
+Audit an A/B-test readout before you ship the decision.
+
+[![ci](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml/badge.svg)](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
+
+An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
+split was broken, three whale customers carried the whole effect, the analyst
+stopped at the first significant refresh, the winning metric was one of forty
+tried, or the experiment never had the power to estimate what it now claims.
+None of that shows up in the p-value. abkit reads the per-unit data and the
+intended design, measures each of these failure modes, and flags whatever
+leaves the range a healthy experiment could occupy.
+
+## The audit in one table
+
+Eight synthetic experiments, seven implanted defects. Bold values are flags;
+each lands on the row where its defect was implanted, and the clean row
+carries none.
+
+<!-- abkit:demo -->
+| experiment | implanted defect | SRM p | contam. | fragility | look FPR | novelty gap | uncorrected | exagg. | flags |
+|---|---|---|---|---|---|---|---|---|---|
+| clean | nothing | 0.89 | 0 | 20 | - | -0.02 | - | 1.0 | none |
+| traffic-leak | 3% of traffic diverted to treatment | **4.1e-14** | 0 | 20 | - | -0.02 | - | 1.0 | sample ratio mismatch |
+| double-dipper | 25 units assigned to both arms | 0.67 | **25** | 20 | - | -0.03 | - | 1.0 | assignment contamination |
+| whale-driven | 3 extreme units carry the significance | 0.11 | 0 | **1** | - | 0.26 | - | 1.2 | outlier fragility |
+| peeker | null effect, stopped at the first significant look | 0.55 | 0 | 8 | **0.19** | -0.17 | - | 1.2 | peeking |
+| fading-novelty | early-half effect 0.30, late-half effect 0.00 | 0.041 | 0 | 20 | - | **0.31** | - | 1.0 | novelty |
+| metric-fisher | null effect, 40 secondary metrics tested | 0.054 | 0 | 0 | - | -0.11 | **2** | 3.4 | uncorrected winners |
+| underpowered-winner | true effect 0.1, powered for 0.14 of that chance | 0.91 | 0 | 8 | - | -0.38 | - | **2.7** | winner's curse |
+<!-- /abkit:demo -->
+
+The row to sit with is `peeker`: a true null effect, analyzed after every
+batch of 250 units and stopped the moment z crossed 1.96. Its final readout
+is indistinguishable from a real win — p < 0.05, decent sample size — but the
+schedule it followed has a 19% false-positive rate, not 5%. The audit flags
+the process, which is the only place that defect lives.
+
+![peeking](results/figures/peeking.png)
+
+![exaggeration](results/figures/exaggeration.png)
+
+`make demo` regenerates the table, the [full report](results/report.md) and
+the figures from fixed seeds; CI rebuilds them from pinned dependencies and
+fails if a committed number differs from what the code produces.
+
+## 4,873 real experiments, audited
+
+[`examples/upworthy/`](examples/upworthy/) runs abkit's aggregate checks over
+the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038/s41597-021-00934-7)
+— 4,873 real headline A/B tests from 2013–2015. Committed results:
+[examples/upworthy/results/upworthy.md](examples/upworthy/results/upworthy.md).
+
+- **16% of real tests fail the sample-ratio check** at p < 0.001 against the
+  platform's equal-allocation design — and not because packages were added
+  mid-test; the rate is, if anything, higher in tests whose variants all
+  launched together. Those readouts carried unknown exposure bias.
+- **10.9% of tests have a "significant winner"** (best headline beats the
+  runner-up at raw p < 0.05), but Benjamini–Hochberg across all 4,873
+  winner comparisons keeps only **2.7%** of them.
+- The median significant winning lift (0.85 CTR points, power 0.62) is
+  expected to **overstate its true effect by 1.27x** even when the winner is
+  real — the winner's curse, measured on real experiments.
+
+## Install
+
+```
+pip install git+https://github.com/mohammadi-hadi/abkit
+```
+
+Python 3.11+. Runtime dependencies: numpy, pydantic, matplotlib.
+
+## Quickstart
+
+Log one JSON object per randomized unit, and the design the experiment was
+supposed to follow:
+
+```json
+{"unit_id": "u1", "arm": "treatment", "metrics": {"revenue": 12.4, "conversion": 1.0}, "pre": {"revenue": 9.1}, "t": 1718040000}
+```
+
+```json
+{
+  "split": {"control": 0.5, "treatment": 0.5},
+  "control": "control",
+  "treatment": "treatment",
+  "primary_metric": "revenue",
+  "alpha": 0.05,
+  "mde": 0.5,
+  "looks": [{"n": 20000, "z": 1.31}, {"n": 40000, "z": 2.11}]
+}
+```
+
+```python
+from abkit import load_units, load_design, run_audit
+
+audit = run_audit(load_units("units.jsonl"), load_design("design.json"))
+for flag in audit.flags:
+    print(flag.name, "--", flag.detail)
+```
+
+Or from the shell, with an exit code CI can gate on:
+
+```
+abkit report units.jsonl --design design.json --out audit --fail-on-flags
+```
+
+Alongside `report.md` the audit writes `report.json` for pipelines that gate
+on specific numbers. Any check that could not run is listed with the exact
+fields to log to enable it — a skip describes the log file, not the
+experiment.
+
+## What it checks
+
+| check | question it answers | needs |
+|---|---|---|
+| sample ratio mismatch | did the split the readout assumes actually happen? | arms + intended split |
+| assignment contamination | is any unit in more than one arm? | unit ids |
+| covariate balance | were the groups equivalent before the treatment? | `pre` covariates |
+| outlier fragility | how many extreme units does significance rest on? | per-unit primary metric |
+| uncorrected winners | do the claimed wins survive multiple-testing control? | 2+ metrics |
+| peeking | what false-positive rate did the stopping rule really have? | `design.looks` |
+| novelty | is the single reported effect averaging over a changing one? | `t` per unit |
+| winner's curse | how exaggerated is a significant estimate at this power? | `design.mde` (or observed) |
+| variance ratio | does the treatment move the distribution, not just the mean? | per-unit primary metric |
+| CUPED headroom | how much tighter could the CI have been? | `pre` covariates |
+
+## How the numbers are defended
+
+- **Cross-checked implementations.** The chi-square survival function matches
+  scipy to 1e-10 across df 1–10, Welch's statistic matches
+  `scipy.stats.ttest_ind`, the two-proportion z matches statsmodels
+  `proportions_ztest`, and Benjamini–Hochberg reproduces statsmodels
+  `multipletests` decisions exactly. The library itself depends on none of
+  them.
+- **Known closed-form results.** The Gelman–Carlin power / Type-S /
+  exaggeration triple is checked against numerical integration to 1e-6, and
+  the peeking simulator reproduces the classical repeated-significance rates
+  (0.083 at 2 looks, 0.142 at 5, 0.193 at 10) within Monte-Carlo tolerance.
+- **Validation by implantation.** The synthetic experiments have defect
+  dials, and the tests require each probe to recover its dial — an implanted
+  0.3 covariate shift must read as SMD 0.3 — to stay monotone in it, and to
+  stay silent on the clean experiment.
+- **Drift-checked results.** The demo table, report and README numbers are
+  regenerated by CI from pinned dependencies and diffed against the
+  committed copy.
+
+## Design notes
+
+- **Innocent bands, not point nulls.** With enough data an SMD of 0.02
+  excludes zero while meaning nothing. Band probes declare the interval a
+  healthy experiment could plausibly occupy — SMD and novelty gap within
+  ±0.10 — and flag only when the whole 95% bootstrap CI leaves it. Exact
+  procedures use the field's conventional thresholds: SRM at p < 0.001,
+  contamination at zero tolerance. All constants live in one place
+  ([`probes.py`](src/abkit/probes.py)) and are easy to disagree with.
+- **The stopping rule is audited, not the final p.** A peeked experiment's
+  final test statistic looks ordinary; what is broken is the procedure that
+  produced it. The peeking probe Monte-Carlos the analyst's actual look
+  schedule under the null and reports the false-positive rate that schedule
+  had.
+- **Fragility is counted in units, not sigmas.** "Removing two customers
+  overturns the decision" is a statement a stakeholder can act on; a
+  leave-one-out influence statistic is not.
+- **Exaggeration needs a basis effect.** With `design.mde` the winner's-curse
+  probe reports power and exaggeration at the effect the experiment was
+  designed for; without it, the observed effect is used and labeled an
+  optimistic bound, since conditioning on significance inflates it.
+- **Stratified bootstrap.** Units are resampled within their arm, never
+  across, so group sizes — which determine every statistic's sampling
+  distribution — are preserved.
+- **Deterministic to the digit.** Seeded PCG64 everywhere, no BLAS in any
+  statistic's path, and a pinned drift environment: the same data gives the
+  same report on any machine.
+
+## Limitations
+
+- An audit is bounded by what was logged. No pre-period covariates, no
+  balance or CUPED checks; no look history, no peeking check; no per-unit
+  data at all, and only the aggregate checks (`abkit.stats`) apply.
+- The innocent bands are judgment calls, not derivations.
+- The peeking probe assumes the recorded looks are all the looks there were.
+- Fragility uses a greedy removal path; it upper-bounds, and in adversarial
+  cases may not find, the true minimal flip set.
+- Implantation shows the probes detect the mechanisms simulated; real
+  experiments can fail in ways not simulated here.
+- The Upworthy analysis treats the archive's documented equal allocation as
+  the intended split; if some tests intentionally used unequal splits, their
+  SRM flags are misattributed.
+
+## Related work
+
+The failure modes measured here are the standard ones from the online
+experimentation literature: sample-ratio mismatch and its taxonomy
+([Fabijan et al., 2019](https://doi.org/10.1145/3292500.3330722)), trustworthy
+experimentation practice ([Kohavi, Tang & Xu, 2020](https://experimentguide.com/)),
+peeking and always-valid inference ([Johari et al., 2017](https://doi.org/10.1145/3097983.3097992)),
+variance reduction with pre-experiment data — CUPED
+([Deng et al., 2013](https://doi.org/10.1145/2433396.2433413)),
+Type-M/Type-S errors ([Gelman & Carlin, 2014](https://doi.org/10.1177/1745691614551642)),
+the winner's curse in A/B testing ([Lee & Shen, 2018](https://doi.org/10.1145/3219819.3219905)),
+and experimentation culture at scale
+([Kaufman, Pitchforth & Vermeer, 2017](https://arxiv.org/abs/1710.08217)).
+The real-data case study uses the Upworthy Research Archive
+([Matias, Munger & Watts, 2021](https://doi.org/10.1038/s41597-021-00934-7)).
+abkit's contribution is packaging the checks as one auditable tool with
+uncertainty on every number and validation by implantation.
+
+Companion projects: [judgekit](https://github.com/mohammadi-hadi/judgekit)
+applies the same audit-before-you-trust pattern to LLM judges;
+[trajectory-judge](https://github.com/mohammadi-hadi/trajectory-judge)
+measures what outcome-only judges miss on agent trajectories.
+
+## Citation
+
+If abkit is useful in your work, please cite it (see
+[CITATION.cff](CITATION.cff)):
+
+```bibtex
+@software{mohammadi_abkit,
+  author  = {Mohammadi, Hadi},
+  title   = {abkit: audit an A/B-test readout before you ship the decision},
+  url     = {https://github.com/mohammadi-hadi/abkit},
+  version = {0.1.0},
+  year    = {2026}
+}
+```
+
+## License
+
+MIT

abkit-0.1.0/README.md

@@ -0,0 +1,232 @@
+# abkit
+
+Audit an A/B-test readout before you ship the decision.
+
+[![ci](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml/badge.svg)](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
+
+An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
+split was broken, three whale customers carried the whole effect, the analyst
+stopped at the first significant refresh, the winning metric was one of forty
+tried, or the experiment never had the power to estimate what it now claims.
+None of that shows up in the p-value. abkit reads the per-unit data and the
+intended design, measures each of these failure modes, and flags whatever
+leaves the range a healthy experiment could occupy.
+
+## The audit in one table
+
+Eight synthetic experiments, seven implanted defects. Bold values are flags;
+each lands on the row where its defect was implanted, and the clean row
+carries none.
+
+<!-- abkit:demo -->
+| experiment | implanted defect | SRM p | contam. | fragility | look FPR | novelty gap | uncorrected | exagg. | flags |
+|---|---|---|---|---|---|---|---|---|---|
+| clean | nothing | 0.89 | 0 | 20 | - | -0.02 | - | 1.0 | none |
+| traffic-leak | 3% of traffic diverted to treatment | **4.1e-14** | 0 | 20 | - | -0.02 | - | 1.0 | sample ratio mismatch |
+| double-dipper | 25 units assigned to both arms | 0.67 | **25** | 20 | - | -0.03 | - | 1.0 | assignment contamination |
+| whale-driven | 3 extreme units carry the significance | 0.11 | 0 | **1** | - | 0.26 | - | 1.2 | outlier fragility |
+| peeker | null effect, stopped at the first significant look | 0.55 | 0 | 8 | **0.19** | -0.17 | - | 1.2 | peeking |
+| fading-novelty | early-half effect 0.30, late-half effect 0.00 | 0.041 | 0 | 20 | - | **0.31** | - | 1.0 | novelty |
+| metric-fisher | null effect, 40 secondary metrics tested | 0.054 | 0 | 0 | - | -0.11 | **2** | 3.4 | uncorrected winners |
+| underpowered-winner | true effect 0.1, powered for 0.14 of that chance | 0.91 | 0 | 8 | - | -0.38 | - | **2.7** | winner's curse |
+<!-- /abkit:demo -->
+
+The row to sit with is `peeker`: a true null effect, analyzed after every
+batch of 250 units and stopped the moment z crossed 1.96. Its final readout
+is indistinguishable from a real win — p < 0.05, decent sample size — but the
+schedule it followed has a 19% false-positive rate, not 5%. The audit flags
+the process, which is the only place that defect lives.
+
+![peeking](results/figures/peeking.png)
+
+![exaggeration](results/figures/exaggeration.png)
+
+`make demo` regenerates the table, the [full report](results/report.md) and
+the figures from fixed seeds; CI rebuilds them from pinned dependencies and
+fails if a committed number differs from what the code produces.
+
+## 4,873 real experiments, audited
+
+[`examples/upworthy/`](examples/upworthy/) runs abkit's aggregate checks over
+the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038/s41597-021-00934-7)
+— 4,873 real headline A/B tests from 2013–2015. Committed results:
+[examples/upworthy/results/upworthy.md](examples/upworthy/results/upworthy.md).
+
+- **16% of real tests fail the sample-ratio check** at p < 0.001 against the
+  platform's equal-allocation design — and not because packages were added
+  mid-test; the rate is, if anything, higher in tests whose variants all
+  launched together. Those readouts carried unknown exposure bias.
+- **10.9% of tests have a "significant winner"** (best headline beats the
+  runner-up at raw p < 0.05), but Benjamini–Hochberg across all 4,873
+  winner comparisons keeps only **2.7%** of them.
+- The median significant winning lift (0.85 CTR points, power 0.62) is
+  expected to **overstate its true effect by 1.27x** even when the winner is
+  real — the winner's curse, measured on real experiments.
+
+## Install
+
+```
+pip install git+https://github.com/mohammadi-hadi/abkit
+```
+
+Python 3.11+. Runtime dependencies: numpy, pydantic, matplotlib.
+
+## Quickstart
+
+Log one JSON object per randomized unit, and the design the experiment was
+supposed to follow:
+
+```json
+{"unit_id": "u1", "arm": "treatment", "metrics": {"revenue": 12.4, "conversion": 1.0}, "pre": {"revenue": 9.1}, "t": 1718040000}
+```
+
+```json
+{
+  "split": {"control": 0.5, "treatment": 0.5},
+  "control": "control",
+  "treatment": "treatment",
+  "primary_metric": "revenue",
+  "alpha": 0.05,
+  "mde": 0.5,
+  "looks": [{"n": 20000, "z": 1.31}, {"n": 40000, "z": 2.11}]
+}
+```
+
+```python
+from abkit import load_units, load_design, run_audit
+
+audit = run_audit(load_units("units.jsonl"), load_design("design.json"))
+for flag in audit.flags:
+    print(flag.name, "--", flag.detail)
+```
+
+Or from the shell, with an exit code CI can gate on:
+
+```
+abkit report units.jsonl --design design.json --out audit --fail-on-flags
+```
+
+Alongside `report.md` the audit writes `report.json` for pipelines that gate
+on specific numbers. Any check that could not run is listed with the exact
+fields to log to enable it — a skip describes the log file, not the
+experiment.
+
+## What it checks
+
+| check | question it answers | needs |
+|---|---|---|
+| sample ratio mismatch | did the split the readout assumes actually happen? | arms + intended split |
+| assignment contamination | is any unit in more than one arm? | unit ids |
+| covariate balance | were the groups equivalent before the treatment? | `pre` covariates |
+| outlier fragility | how many extreme units does significance rest on? | per-unit primary metric |
+| uncorrected winners | do the claimed wins survive multiple-testing control? | 2+ metrics |
+| peeking | what false-positive rate did the stopping rule really have? | `design.looks` |
+| novelty | is the single reported effect averaging over a changing one? | `t` per unit |
+| winner's curse | how exaggerated is a significant estimate at this power? | `design.mde` (or observed) |
+| variance ratio | does the treatment move the distribution, not just the mean? | per-unit primary metric |
+| CUPED headroom | how much tighter could the CI have been? | `pre` covariates |
+
+## How the numbers are defended
+
+- **Cross-checked implementations.** The chi-square survival function matches
+  scipy to 1e-10 across df 1–10, Welch's statistic matches
+  `scipy.stats.ttest_ind`, the two-proportion z matches statsmodels
+  `proportions_ztest`, and Benjamini–Hochberg reproduces statsmodels
+  `multipletests` decisions exactly. The library itself depends on none of
+  them.
+- **Known closed-form results.** The Gelman–Carlin power / Type-S /
+  exaggeration triple is checked against numerical integration to 1e-6, and
+  the peeking simulator reproduces the classical repeated-significance rates
+  (0.083 at 2 looks, 0.142 at 5, 0.193 at 10) within Monte-Carlo tolerance.
+- **Validation by implantation.** The synthetic experiments have defect
+  dials, and the tests require each probe to recover its dial — an implanted
+  0.3 covariate shift must read as SMD 0.3 — to stay monotone in it, and to
+  stay silent on the clean experiment.
+- **Drift-checked results.** The demo table, report and README numbers are
+  regenerated by CI from pinned dependencies and diffed against the
+  committed copy.
+
+## Design notes
+
+- **Innocent bands, not point nulls.** With enough data an SMD of 0.02
+  excludes zero while meaning nothing. Band probes declare the interval a
+  healthy experiment could plausibly occupy — SMD and novelty gap within
+  ±0.10 — and flag only when the whole 95% bootstrap CI leaves it. Exact
+  procedures use the field's conventional thresholds: SRM at p < 0.001,
+  contamination at zero tolerance. All constants live in one place
+  ([`probes.py`](src/abkit/probes.py)) and are easy to disagree with.
+- **The stopping rule is audited, not the final p.** A peeked experiment's
+  final test statistic looks ordinary; what is broken is the procedure that
+  produced it. The peeking probe Monte-Carlos the analyst's actual look
+  schedule under the null and reports the false-positive rate that schedule
+  had.
+- **Fragility is counted in units, not sigmas.** "Removing two customers
+  overturns the decision" is a statement a stakeholder can act on; a
+  leave-one-out influence statistic is not.
+- **Exaggeration needs a basis effect.** With `design.mde` the winner's-curse
+  probe reports power and exaggeration at the effect the experiment was
+  designed for; without it, the observed effect is used and labeled an
+  optimistic bound, since conditioning on significance inflates it.
+- **Stratified bootstrap.** Units are resampled within their arm, never
+  across, so group sizes — which determine every statistic's sampling
+  distribution — are preserved.
+- **Deterministic to the digit.** Seeded PCG64 everywhere, no BLAS in any
+  statistic's path, and a pinned drift environment: the same data gives the
+  same report on any machine.
+
+## Limitations
+
+- An audit is bounded by what was logged. No pre-period covariates, no
+  balance or CUPED checks; no look history, no peeking check; no per-unit
+  data at all, and only the aggregate checks (`abkit.stats`) apply.
+- The innocent bands are judgment calls, not derivations.
+- The peeking probe assumes the recorded looks are all the looks there were.
+- Fragility uses a greedy removal path; it upper-bounds, and in adversarial
+  cases may not find, the true minimal flip set.
+- Implantation shows the probes detect the mechanisms simulated; real
+  experiments can fail in ways not simulated here.
+- The Upworthy analysis treats the archive's documented equal allocation as
+  the intended split; if some tests intentionally used unequal splits, their
+  SRM flags are misattributed.
+
+## Related work
+
+The failure modes measured here are the standard ones from the online
+experimentation literature: sample-ratio mismatch and its taxonomy
+([Fabijan et al., 2019](https://doi.org/10.1145/3292500.3330722)), trustworthy
+experimentation practice ([Kohavi, Tang & Xu, 2020](https://experimentguide.com/)),
+peeking and always-valid inference ([Johari et al., 2017](https://doi.org/10.1145/3097983.3097992)),
+variance reduction with pre-experiment data — CUPED
+([Deng et al., 2013](https://doi.org/10.1145/2433396.2433413)),
+Type-M/Type-S errors ([Gelman & Carlin, 2014](https://doi.org/10.1177/1745691614551642)),
+the winner's curse in A/B testing ([Lee & Shen, 2018](https://doi.org/10.1145/3219819.3219905)),
+and experimentation culture at scale
+([Kaufman, Pitchforth & Vermeer, 2017](https://arxiv.org/abs/1710.08217)).
+The real-data case study uses the Upworthy Research Archive
+([Matias, Munger & Watts, 2021](https://doi.org/10.1038/s41597-021-00934-7)).
+abkit's contribution is packaging the checks as one auditable tool with
+uncertainty on every number and validation by implantation.
+
+Companion projects: [judgekit](https://github.com/mohammadi-hadi/judgekit)
+applies the same audit-before-you-trust pattern to LLM judges;
+[trajectory-judge](https://github.com/mohammadi-hadi/trajectory-judge)
+measures what outcome-only judges miss on agent trajectories.
+
+## Citation
+
+If abkit is useful in your work, please cite it (see
+[CITATION.cff](CITATION.cff)):
+
+```bibtex
+@software{mohammadi_abkit,
+  author  = {Mohammadi, Hadi},
+  title   = {abkit: audit an A/B-test readout before you ship the decision},
+  url     = {https://github.com/mohammadi-hadi/abkit},
+  version = {0.1.0},
+  year    = {2026}
+}
+```
+
+## License
+
+MIT

abkit-0.1.0/pyproject.toml

@@ -0,0 +1,60 @@
+[build-system]
+requires = ["setuptools>=69"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "abkit"
+version = "0.1.0"
+description = "Audit an A/B-test readout before you ship the decision"
+readme = "README.md"
+license = {text = "MIT"}
+authors = [{name = "Hadi Mohammadi"}]
+requires-python = ">=3.11"
+dependencies = [
+    "numpy>=1.26",
+    "pydantic>=2.5",
+    "matplotlib>=3.8",
+]
+keywords = ["ab-testing", "experimentation", "srm", "statistics", "audit"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+
+[project.urls]
+Homepage = "https://github.com/mohammadi-hadi/abkit"
+Issues = "https://github.com/mohammadi-hadi/abkit/issues"
+
+[project.scripts]
+abkit = "abkit.cli:main"
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "ruff>=0.4", "mypy>=1.10", "scipy>=1.12", "statsmodels>=0.14"]
+
+[tool.setuptools.package-data]
+abkit = ["py.typed"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+
+[tool.mypy]
+disallow_untyped_defs = true
+warn_return_any = true
+warn_unused_ignores = true
+no_implicit_optional = true
+
+[[tool.mypy.overrides]]
+module = "matplotlib.*"
+ignore_missing_imports = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

abkit-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

abkit-0.1.0/src/abkit/__init__.py

@@ -0,0 +1,23 @@
+"""abkit: audit an A/B-test readout before you ship the decision."""
+
+from .audit import run_audit
+from .io import dump_design, dump_units, load_design, load_units
+from .result import AuditResult, ProbeResult, SkippedProbe
+from .schema import Design, Look, Unit
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "AuditResult",
+    "Design",
+    "Look",
+    "ProbeResult",
+    "SkippedProbe",
+    "Unit",
+    "__version__",
+    "dump_design",
+    "dump_units",
+    "load_design",
+    "load_units",
+    "run_audit",
+]

abkit-0.1.0/src/abkit/audit.py

@@ -0,0 +1,58 @@
+"""Run every applicable probe against one experiment readout."""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+from . import probes
+from .result import AuditResult, ProbeResult, SkippedProbe
+from .schema import Design, Unit
+from .stats import welch
+
+
+def run_audit(units: Sequence[Unit], design: Design, name: str = "experiment") -> AuditResult:
+    result = AuditResult(experiment=name)
+
+    treat = [
+        u.metrics[design.primary_metric]
+        for u in units
+        if u.arm == design.treatment and design.primary_metric in u.metrics
+    ]
+    ctrl = [
+        u.metrics[design.primary_metric]
+        for u in units
+        if u.arm == design.control and design.primary_metric in u.metrics
+    ]
+    if len(treat) >= 2 and len(ctrl) >= 2:
+        effect, se, z, p = welch(treat, ctrl)
+        result.summary = {
+            "primary_metric": design.primary_metric,
+            "effect": effect,
+            "se": se,
+            "z": z,
+            "p": p,
+            "n_treatment": len(treat),
+            "n_control": len(ctrl),
+            "alpha": design.alpha,
+        }
+
+    outcomes: list[ProbeResult | SkippedProbe | list[ProbeResult]] = [
+        probes.sample_ratio(units, design),
+        probes.contamination(units),
+        probes.covariate_balance(units, design),
+        probes.outlier_fragility(units, design),
+        probes.uncorrected_winners(units, design),
+        probes.peeking(design),
+        probes.novelty(units, design),
+        probes.winners_curse(units, design),
+        probes.variance_ratio(units, design),
+        probes.cuped_headroom(units, design),
+    ]
+    for outcome in outcomes:
+        if isinstance(outcome, SkippedProbe):
+            result.skipped.append(outcome)
+        elif isinstance(outcome, list):
+            result.probes.extend(outcome)
+        else:
+            result.probes.append(outcome)
+    return result