abkit 0.1.1__tar.gz → 0.1.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (31) hide show
  1. {abkit-0.1.1/src/abkit.egg-info → abkit-0.1.2}/PKG-INFO +23 -20
  2. {abkit-0.1.1 → abkit-0.1.2}/README.md +22 -19
  3. {abkit-0.1.1 → abkit-0.1.2}/pyproject.toml +1 -1
  4. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/__init__.py +1 -1
  5. {abkit-0.1.1 → abkit-0.1.2/src/abkit.egg-info}/PKG-INFO +23 -20
  6. {abkit-0.1.1 → abkit-0.1.2}/LICENSE +0 -0
  7. {abkit-0.1.1 → abkit-0.1.2}/setup.cfg +0 -0
  8. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/audit.py +0 -0
  9. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/bootstrap.py +0 -0
  10. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/cli.py +0 -0
  11. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/demo.py +0 -0
  12. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/io.py +0 -0
  13. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/probes.py +0 -0
  14. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/py.typed +0 -0
  15. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/report.py +0 -0
  16. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/result.py +0 -0
  17. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/schema.py +0 -0
  18. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/stats.py +0 -0
  19. {abkit-0.1.1 → abkit-0.1.2}/src/abkit/synthetic.py +0 -0
  20. {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/SOURCES.txt +0 -0
  21. {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/dependency_links.txt +0 -0
  22. {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/entry_points.txt +0 -0
  23. {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/requires.txt +0 -0
  24. {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/top_level.txt +0 -0
  25. {abkit-0.1.1 → abkit-0.1.2}/tests/test_bootstrap.py +0 -0
  26. {abkit-0.1.1 → abkit-0.1.2}/tests/test_cli.py +0 -0
  27. {abkit-0.1.1 → abkit-0.1.2}/tests/test_probes.py +0 -0
  28. {abkit-0.1.1 → abkit-0.1.2}/tests/test_result.py +0 -0
  29. {abkit-0.1.1 → abkit-0.1.2}/tests/test_schema.py +0 -0
  30. {abkit-0.1.1 → abkit-0.1.2}/tests/test_stats.py +0 -0
  31. {abkit-0.1.1 → abkit-0.1.2}/tests/test_synthetic.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: abkit
3
- Version: 0.1.1
3
+ Version: 0.1.2
4
4
  Summary: Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts
5
5
  Author: Hadi Mohammadi
6
6
  License: MIT
@@ -28,14 +28,19 @@ Requires-Dist: scipy>=1.12; extra == "dev"
28
28
  Requires-Dist: statsmodels>=0.14; extra == "dev"
29
29
  Dynamic: license-file
30
30
 
31
- # abkit
31
+ <div align="center">
32
32
 
33
- Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.
33
+ # abkit
34
34
 
35
35
  [![ci](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml/badge.svg)](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
36
36
  [![PyPI](https://img.shields.io/pypi/v/abkit)](https://pypi.org/project/abkit/)
37
37
  [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21809478.svg)](https://doi.org/10.5281/zenodo.21809478)
38
- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
38
+ [![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
39
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/mohammadi-hadi/abkit/blob/main/LICENSE)
40
+
41
+ *Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.*
42
+
43
+ </div>
39
44
 
40
45
  An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
41
46
  split was broken, three whale customers carried the whole effect, the analyst
@@ -70,22 +75,22 @@ is indistinguishable from a real win — p < 0.05, decent sample size — but th
70
75
  schedule it followed has a 19% false-positive rate, not 5%. The audit flags
71
76
  the process, which is the only place that defect lives.
72
77
 
73
- ![peeking](results/figures/peeking.png)
78
+ ![peeking](https://raw.githubusercontent.com/mohammadi-hadi/abkit/main/results/figures/peeking.png)
74
79
 
75
- ![exaggeration](results/figures/exaggeration.png)
80
+ ![exaggeration](https://raw.githubusercontent.com/mohammadi-hadi/abkit/main/results/figures/exaggeration.png)
76
81
 
77
- `make demo` regenerates the table, the [full report](results/report.md) and
82
+ `make demo` regenerates the table, the [full report](https://github.com/mohammadi-hadi/abkit/blob/main/results/report.md) and
78
83
  the figures from fixed seeds; CI rebuilds them from pinned dependencies and
79
84
  fails if a committed number differs from what the code produces.
80
85
 
81
86
  ## 4,873 real experiments, audited
82
87
 
83
- [`examples/upworthy/`](examples/upworthy/) runs abkit's aggregate checks over
88
+ [`examples/upworthy/`](https://github.com/mohammadi-hadi/abkit/tree/main/examples/upworthy/) runs abkit's aggregate checks over
84
89
  the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038/s41597-021-00934-7)
85
90
  — 4,873 real headline A/B tests from 2013–2015. Committed results:
86
- [examples/upworthy/results/upworthy.md](examples/upworthy/results/upworthy.md).
91
+ [examples/upworthy/results/upworthy.md](https://github.com/mohammadi-hadi/abkit/blob/main/examples/upworthy/results/upworthy.md).
87
92
 
88
- - **16% of real tests fail the sample-ratio check** at p < 0.001 against the
93
+ - **15% of real tests fail the sample-ratio check** at p < 0.001 against the
89
94
  platform's equal-allocation design — and not because packages were added
90
95
  mid-test; the rate is, if anything, higher in tests whose variants all
91
96
  launched together. Those readouts carried unknown exposure bias.
@@ -187,7 +192,7 @@ experiment.
187
192
  ±0.10 — and flag only when the whole 95% bootstrap CI leaves it. Exact
188
193
  procedures use the field's conventional thresholds: SRM at p < 0.001,
189
194
  contamination at zero tolerance. All constants live in one place
190
- ([`probes.py`](src/abkit/probes.py)) and are easy to disagree with.
195
+ ([`probes.py`](https://github.com/mohammadi-hadi/abkit/blob/main/src/abkit/probes.py)) and are easy to disagree with.
191
196
  - **The stopping rule is audited, not the final p.** A peeked experiment's
192
197
  final test statistic looks ordinary; what is broken is the procedure that
193
198
  produced it. The peeking probe Monte-Carlos the analyst's actual look
@@ -240,20 +245,18 @@ The real-data case study uses the Upworthy Research Archive
240
245
  abkit's contribution is packaging the checks as one auditable tool with
241
246
  uncertainty on every number and validation by implantation.
242
247
 
243
- Companion projects: [judgekit](https://github.com/mohammadi-hadi/judgekit)
244
- applies the same audit-before-you-trust pattern to LLM judges;
245
- [raterkit](https://github.com/mohammadi-hadi/raterkit) applies it to the human
246
- labels underneath both; [trajectory-judge](https://github.com/mohammadi-hadi/trajectory-judge)
247
- measures what outcome-only judges miss on agent trajectories;
248
- [arenakit](https://github.com/mohammadi-hadi/arenakit) applies it to pairwise
249
- model leaderboards.
248
+ ## Part of evalstack
249
+
250
+ [evalstack](https://github.com/mohammadi-hadi/evalstack) is the map of these
251
+ eleven evaluation tools: what each one measures, what it found on real data,
252
+ and the two chains that run end to end.
250
253
 
251
254
  ## Citation
252
255
 
253
256
  Releases are archived on Zenodo. Cite the concept DOI
254
257
  [10.5281/zenodo.21809478](https://doi.org/10.5281/zenodo.21809478), which always
255
258
  resolves to the latest version; structured metadata is in
256
- [CITATION.cff](CITATION.cff).
259
+ [CITATION.cff](https://github.com/mohammadi-hadi/abkit/blob/main/CITATION.cff).
257
260
 
258
261
  ```bibtex
259
262
  @software{mohammadi_abkit,
@@ -261,7 +264,7 @@ resolves to the latest version; structured metadata is in
261
264
  title = {abkit: audit an A/B-test readout before you ship the decision},
262
265
  url = {https://github.com/mohammadi-hadi/abkit},
263
266
  doi = {10.5281/zenodo.21809478},
264
- version = {0.1.0},
267
+ version = {0.1.2},
265
268
  year = {2026}
266
269
  }
267
270
  ```
@@ -1,11 +1,16 @@
1
- # abkit
1
+ <div align="center">
2
2
 
3
- Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.
3
+ # abkit
4
4
 
5
5
  [![ci](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml/badge.svg)](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
6
6
  [![PyPI](https://img.shields.io/pypi/v/abkit)](https://pypi.org/project/abkit/)
7
7
  [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21809478.svg)](https://doi.org/10.5281/zenodo.21809478)
8
- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
8
+ [![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
9
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/mohammadi-hadi/abkit/blob/main/LICENSE)
10
+
11
+ *Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.*
12
+
13
+ </div>
9
14
 
10
15
  An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
11
16
  split was broken, three whale customers carried the whole effect, the analyst
@@ -40,22 +45,22 @@ is indistinguishable from a real win — p < 0.05, decent sample size — but th
40
45
  schedule it followed has a 19% false-positive rate, not 5%. The audit flags
41
46
  the process, which is the only place that defect lives.
42
47
 
43
- ![peeking](results/figures/peeking.png)
48
+ ![peeking](https://raw.githubusercontent.com/mohammadi-hadi/abkit/main/results/figures/peeking.png)
44
49
 
45
- ![exaggeration](results/figures/exaggeration.png)
50
+ ![exaggeration](https://raw.githubusercontent.com/mohammadi-hadi/abkit/main/results/figures/exaggeration.png)
46
51
 
47
- `make demo` regenerates the table, the [full report](results/report.md) and
52
+ `make demo` regenerates the table, the [full report](https://github.com/mohammadi-hadi/abkit/blob/main/results/report.md) and
48
53
  the figures from fixed seeds; CI rebuilds them from pinned dependencies and
49
54
  fails if a committed number differs from what the code produces.
50
55
 
51
56
  ## 4,873 real experiments, audited
52
57
 
53
- [`examples/upworthy/`](examples/upworthy/) runs abkit's aggregate checks over
58
+ [`examples/upworthy/`](https://github.com/mohammadi-hadi/abkit/tree/main/examples/upworthy/) runs abkit's aggregate checks over
54
59
  the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038/s41597-021-00934-7)
55
60
  — 4,873 real headline A/B tests from 2013–2015. Committed results:
56
- [examples/upworthy/results/upworthy.md](examples/upworthy/results/upworthy.md).
61
+ [examples/upworthy/results/upworthy.md](https://github.com/mohammadi-hadi/abkit/blob/main/examples/upworthy/results/upworthy.md).
57
62
 
58
- - **16% of real tests fail the sample-ratio check** at p < 0.001 against the
63
+ - **15% of real tests fail the sample-ratio check** at p < 0.001 against the
59
64
  platform's equal-allocation design — and not because packages were added
60
65
  mid-test; the rate is, if anything, higher in tests whose variants all
61
66
  launched together. Those readouts carried unknown exposure bias.
@@ -157,7 +162,7 @@ experiment.
157
162
  ±0.10 — and flag only when the whole 95% bootstrap CI leaves it. Exact
158
163
  procedures use the field's conventional thresholds: SRM at p < 0.001,
159
164
  contamination at zero tolerance. All constants live in one place
160
- ([`probes.py`](src/abkit/probes.py)) and are easy to disagree with.
165
+ ([`probes.py`](https://github.com/mohammadi-hadi/abkit/blob/main/src/abkit/probes.py)) and are easy to disagree with.
161
166
  - **The stopping rule is audited, not the final p.** A peeked experiment's
162
167
  final test statistic looks ordinary; what is broken is the procedure that
163
168
  produced it. The peeking probe Monte-Carlos the analyst's actual look
@@ -210,20 +215,18 @@ The real-data case study uses the Upworthy Research Archive
210
215
  abkit's contribution is packaging the checks as one auditable tool with
211
216
  uncertainty on every number and validation by implantation.
212
217
 
213
- Companion projects: [judgekit](https://github.com/mohammadi-hadi/judgekit)
214
- applies the same audit-before-you-trust pattern to LLM judges;
215
- [raterkit](https://github.com/mohammadi-hadi/raterkit) applies it to the human
216
- labels underneath both; [trajectory-judge](https://github.com/mohammadi-hadi/trajectory-judge)
217
- measures what outcome-only judges miss on agent trajectories;
218
- [arenakit](https://github.com/mohammadi-hadi/arenakit) applies it to pairwise
219
- model leaderboards.
218
+ ## Part of evalstack
219
+
220
+ [evalstack](https://github.com/mohammadi-hadi/evalstack) is the map of these
221
+ eleven evaluation tools: what each one measures, what it found on real data,
222
+ and the two chains that run end to end.
220
223
 
221
224
  ## Citation
222
225
 
223
226
  Releases are archived on Zenodo. Cite the concept DOI
224
227
  [10.5281/zenodo.21809478](https://doi.org/10.5281/zenodo.21809478), which always
225
228
  resolves to the latest version; structured metadata is in
226
- [CITATION.cff](CITATION.cff).
229
+ [CITATION.cff](https://github.com/mohammadi-hadi/abkit/blob/main/CITATION.cff).
227
230
 
228
231
  ```bibtex
229
232
  @software{mohammadi_abkit,
@@ -231,7 +234,7 @@ resolves to the latest version; structured metadata is in
231
234
  title = {abkit: audit an A/B-test readout before you ship the decision},
232
235
  url = {https://github.com/mohammadi-hadi/abkit},
233
236
  doi = {10.5281/zenodo.21809478},
234
- version = {0.1.0},
237
+ version = {0.1.2},
235
238
  year = {2026}
236
239
  }
237
240
  ```
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "abkit"
7
- version = "0.1.1"
7
+ version = "0.1.2"
8
8
  description = "Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts"
9
9
  readme = "README.md"
10
10
  license = {text = "MIT"}
@@ -5,7 +5,7 @@ from .io import dump_design, dump_units, load_design, load_units
5
5
  from .result import AuditResult, ProbeResult, SkippedProbe
6
6
  from .schema import Design, Look, Unit
7
7
 
8
- __version__ = "0.1.1"
8
+ __version__ = "0.1.2"
9
9
 
10
10
  __all__ = [
11
11
  "AuditResult",
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: abkit
3
- Version: 0.1.1
3
+ Version: 0.1.2
4
4
  Summary: Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts
5
5
  Author: Hadi Mohammadi
6
6
  License: MIT
@@ -28,14 +28,19 @@ Requires-Dist: scipy>=1.12; extra == "dev"
28
28
  Requires-Dist: statsmodels>=0.14; extra == "dev"
29
29
  Dynamic: license-file
30
30
 
31
- # abkit
31
+ <div align="center">
32
32
 
33
- Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.
33
+ # abkit
34
34
 
35
35
  [![ci](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml/badge.svg)](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
36
36
  [![PyPI](https://img.shields.io/pypi/v/abkit)](https://pypi.org/project/abkit/)
37
37
  [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21809478.svg)](https://doi.org/10.5281/zenodo.21809478)
38
- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
38
+ [![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
39
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/mohammadi-hadi/abkit/blob/main/LICENSE)
40
+
41
+ *Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.*
42
+
43
+ </div>
39
44
 
40
45
  An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
41
46
  split was broken, three whale customers carried the whole effect, the analyst
@@ -70,22 +75,22 @@ is indistinguishable from a real win — p < 0.05, decent sample size — but th
70
75
  schedule it followed has a 19% false-positive rate, not 5%. The audit flags
71
76
  the process, which is the only place that defect lives.
72
77
 
73
- ![peeking](results/figures/peeking.png)
78
+ ![peeking](https://raw.githubusercontent.com/mohammadi-hadi/abkit/main/results/figures/peeking.png)
74
79
 
75
- ![exaggeration](results/figures/exaggeration.png)
80
+ ![exaggeration](https://raw.githubusercontent.com/mohammadi-hadi/abkit/main/results/figures/exaggeration.png)
76
81
 
77
- `make demo` regenerates the table, the [full report](results/report.md) and
82
+ `make demo` regenerates the table, the [full report](https://github.com/mohammadi-hadi/abkit/blob/main/results/report.md) and
78
83
  the figures from fixed seeds; CI rebuilds them from pinned dependencies and
79
84
  fails if a committed number differs from what the code produces.
80
85
 
81
86
  ## 4,873 real experiments, audited
82
87
 
83
- [`examples/upworthy/`](examples/upworthy/) runs abkit's aggregate checks over
88
+ [`examples/upworthy/`](https://github.com/mohammadi-hadi/abkit/tree/main/examples/upworthy/) runs abkit's aggregate checks over
84
89
  the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038/s41597-021-00934-7)
85
90
  — 4,873 real headline A/B tests from 2013–2015. Committed results:
86
- [examples/upworthy/results/upworthy.md](examples/upworthy/results/upworthy.md).
91
+ [examples/upworthy/results/upworthy.md](https://github.com/mohammadi-hadi/abkit/blob/main/examples/upworthy/results/upworthy.md).
87
92
 
88
- - **16% of real tests fail the sample-ratio check** at p < 0.001 against the
93
+ - **15% of real tests fail the sample-ratio check** at p < 0.001 against the
89
94
  platform's equal-allocation design — and not because packages were added
90
95
  mid-test; the rate is, if anything, higher in tests whose variants all
91
96
  launched together. Those readouts carried unknown exposure bias.
@@ -187,7 +192,7 @@ experiment.
187
192
  ±0.10 — and flag only when the whole 95% bootstrap CI leaves it. Exact
188
193
  procedures use the field's conventional thresholds: SRM at p < 0.001,
189
194
  contamination at zero tolerance. All constants live in one place
190
- ([`probes.py`](src/abkit/probes.py)) and are easy to disagree with.
195
+ ([`probes.py`](https://github.com/mohammadi-hadi/abkit/blob/main/src/abkit/probes.py)) and are easy to disagree with.
191
196
  - **The stopping rule is audited, not the final p.** A peeked experiment's
192
197
  final test statistic looks ordinary; what is broken is the procedure that
193
198
  produced it. The peeking probe Monte-Carlos the analyst's actual look
@@ -240,20 +245,18 @@ The real-data case study uses the Upworthy Research Archive
240
245
  abkit's contribution is packaging the checks as one auditable tool with
241
246
  uncertainty on every number and validation by implantation.
242
247
 
243
- Companion projects: [judgekit](https://github.com/mohammadi-hadi/judgekit)
244
- applies the same audit-before-you-trust pattern to LLM judges;
245
- [raterkit](https://github.com/mohammadi-hadi/raterkit) applies it to the human
246
- labels underneath both; [trajectory-judge](https://github.com/mohammadi-hadi/trajectory-judge)
247
- measures what outcome-only judges miss on agent trajectories;
248
- [arenakit](https://github.com/mohammadi-hadi/arenakit) applies it to pairwise
249
- model leaderboards.
248
+ ## Part of evalstack
249
+
250
+ [evalstack](https://github.com/mohammadi-hadi/evalstack) is the map of these
251
+ eleven evaluation tools: what each one measures, what it found on real data,
252
+ and the two chains that run end to end.
250
253
 
251
254
  ## Citation
252
255
 
253
256
  Releases are archived on Zenodo. Cite the concept DOI
254
257
  [10.5281/zenodo.21809478](https://doi.org/10.5281/zenodo.21809478), which always
255
258
  resolves to the latest version; structured metadata is in
256
- [CITATION.cff](CITATION.cff).
259
+ [CITATION.cff](https://github.com/mohammadi-hadi/abkit/blob/main/CITATION.cff).
257
260
 
258
261
  ```bibtex
259
262
  @software{mohammadi_abkit,
@@ -261,7 +264,7 @@ resolves to the latest version; structured metadata is in
261
264
  title = {abkit: audit an A/B-test readout before you ship the decision},
262
265
  url = {https://github.com/mohammadi-hadi/abkit},
263
266
  doi = {10.5281/zenodo.21809478},
264
- version = {0.1.0},
267
+ version = {0.1.2},
265
268
  year = {2026}
266
269
  }
267
270
  ```
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes