abkit 0.1.1__tar.gz → 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {abkit-0.1.1/src/abkit.egg-info → abkit-0.1.2}/PKG-INFO +23 -20
- {abkit-0.1.1 → abkit-0.1.2}/README.md +22 -19
- {abkit-0.1.1 → abkit-0.1.2}/pyproject.toml +1 -1
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/__init__.py +1 -1
- {abkit-0.1.1 → abkit-0.1.2/src/abkit.egg-info}/PKG-INFO +23 -20
- {abkit-0.1.1 → abkit-0.1.2}/LICENSE +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/setup.cfg +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/audit.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/bootstrap.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/cli.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/demo.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/io.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/probes.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/py.typed +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/report.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/result.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/schema.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/stats.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit/synthetic.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/SOURCES.txt +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/dependency_links.txt +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/entry_points.txt +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/requires.txt +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/src/abkit.egg-info/top_level.txt +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/tests/test_bootstrap.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/tests/test_cli.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/tests/test_probes.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/tests/test_result.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/tests/test_schema.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/tests/test_stats.py +0 -0
- {abkit-0.1.1 → abkit-0.1.2}/tests/test_synthetic.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: abkit
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts
|
|
5
5
|
Author: Hadi Mohammadi
|
|
6
6
|
License: MIT
|
|
@@ -28,14 +28,19 @@ Requires-Dist: scipy>=1.12; extra == "dev"
|
|
|
28
28
|
Requires-Dist: statsmodels>=0.14; extra == "dev"
|
|
29
29
|
Dynamic: license-file
|
|
30
30
|
|
|
31
|
-
|
|
31
|
+
<div align="center">
|
|
32
32
|
|
|
33
|
-
|
|
33
|
+
# abkit
|
|
34
34
|
|
|
35
35
|
[](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
|
|
36
36
|
[](https://pypi.org/project/abkit/)
|
|
37
37
|
[](https://doi.org/10.5281/zenodo.21809478)
|
|
38
|
-
[](https://www.python.org/downloads/)
|
|
39
|
+
[](https://github.com/mohammadi-hadi/abkit/blob/main/LICENSE)
|
|
40
|
+
|
|
41
|
+
*Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.*
|
|
42
|
+
|
|
43
|
+
</div>
|
|
39
44
|
|
|
40
45
|
An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
|
|
41
46
|
split was broken, three whale customers carried the whole effect, the analyst
|
|
@@ -70,22 +75,22 @@ is indistinguishable from a real win — p < 0.05, decent sample size — but th
|
|
|
70
75
|
schedule it followed has a 19% false-positive rate, not 5%. The audit flags
|
|
71
76
|
the process, which is the only place that defect lives.
|
|
72
77
|
|
|
73
|
-

|
|
78
|
+

|
|
74
79
|
|
|
75
|
-

|
|
80
|
+

|
|
76
81
|
|
|
77
|
-
`make demo` regenerates the table, the [full report](results/report.md) and
|
|
82
|
+
`make demo` regenerates the table, the [full report](https://github.com/mohammadi-hadi/abkit/blob/main/results/report.md) and
|
|
78
83
|
the figures from fixed seeds; CI rebuilds them from pinned dependencies and
|
|
79
84
|
fails if a committed number differs from what the code produces.
|
|
80
85
|
|
|
81
86
|
## 4,873 real experiments, audited
|
|
82
87
|
|
|
83
|
-
[`examples/upworthy/`](examples/upworthy/) runs abkit's aggregate checks over
|
|
88
|
+
[`examples/upworthy/`](https://github.com/mohammadi-hadi/abkit/tree/main/examples/upworthy/) runs abkit's aggregate checks over
|
|
84
89
|
the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038/s41597-021-00934-7)
|
|
85
90
|
— 4,873 real headline A/B tests from 2013–2015. Committed results:
|
|
86
|
-
[examples/upworthy/results/upworthy.md](examples/upworthy/results/upworthy.md).
|
|
91
|
+
[examples/upworthy/results/upworthy.md](https://github.com/mohammadi-hadi/abkit/blob/main/examples/upworthy/results/upworthy.md).
|
|
87
92
|
|
|
88
|
-
- **
|
|
93
|
+
- **15% of real tests fail the sample-ratio check** at p < 0.001 against the
|
|
89
94
|
platform's equal-allocation design — and not because packages were added
|
|
90
95
|
mid-test; the rate is, if anything, higher in tests whose variants all
|
|
91
96
|
launched together. Those readouts carried unknown exposure bias.
|
|
@@ -187,7 +192,7 @@ experiment.
|
|
|
187
192
|
±0.10 — and flag only when the whole 95% bootstrap CI leaves it. Exact
|
|
188
193
|
procedures use the field's conventional thresholds: SRM at p < 0.001,
|
|
189
194
|
contamination at zero tolerance. All constants live in one place
|
|
190
|
-
([`probes.py`](src/abkit/probes.py)) and are easy to disagree with.
|
|
195
|
+
([`probes.py`](https://github.com/mohammadi-hadi/abkit/blob/main/src/abkit/probes.py)) and are easy to disagree with.
|
|
191
196
|
- **The stopping rule is audited, not the final p.** A peeked experiment's
|
|
192
197
|
final test statistic looks ordinary; what is broken is the procedure that
|
|
193
198
|
produced it. The peeking probe Monte-Carlos the analyst's actual look
|
|
@@ -240,20 +245,18 @@ The real-data case study uses the Upworthy Research Archive
|
|
|
240
245
|
abkit's contribution is packaging the checks as one auditable tool with
|
|
241
246
|
uncertainty on every number and validation by implantation.
|
|
242
247
|
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
[
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
[arenakit](https://github.com/mohammadi-hadi/arenakit) applies it to pairwise
|
|
249
|
-
model leaderboards.
|
|
248
|
+
## Part of evalstack
|
|
249
|
+
|
|
250
|
+
[evalstack](https://github.com/mohammadi-hadi/evalstack) is the map of these
|
|
251
|
+
eleven evaluation tools: what each one measures, what it found on real data,
|
|
252
|
+
and the two chains that run end to end.
|
|
250
253
|
|
|
251
254
|
## Citation
|
|
252
255
|
|
|
253
256
|
Releases are archived on Zenodo. Cite the concept DOI
|
|
254
257
|
[10.5281/zenodo.21809478](https://doi.org/10.5281/zenodo.21809478), which always
|
|
255
258
|
resolves to the latest version; structured metadata is in
|
|
256
|
-
[CITATION.cff](CITATION.cff).
|
|
259
|
+
[CITATION.cff](https://github.com/mohammadi-hadi/abkit/blob/main/CITATION.cff).
|
|
257
260
|
|
|
258
261
|
```bibtex
|
|
259
262
|
@software{mohammadi_abkit,
|
|
@@ -261,7 +264,7 @@ resolves to the latest version; structured metadata is in
|
|
|
261
264
|
title = {abkit: audit an A/B-test readout before you ship the decision},
|
|
262
265
|
url = {https://github.com/mohammadi-hadi/abkit},
|
|
263
266
|
doi = {10.5281/zenodo.21809478},
|
|
264
|
-
version = {0.1.
|
|
267
|
+
version = {0.1.2},
|
|
265
268
|
year = {2026}
|
|
266
269
|
}
|
|
267
270
|
```
|
|
@@ -1,11 +1,16 @@
|
|
|
1
|
-
|
|
1
|
+
<div align="center">
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
# abkit
|
|
4
4
|
|
|
5
5
|
[](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
|
|
6
6
|
[](https://pypi.org/project/abkit/)
|
|
7
7
|
[](https://doi.org/10.5281/zenodo.21809478)
|
|
8
|
-
[](https://www.python.org/downloads/)
|
|
9
|
+
[](https://github.com/mohammadi-hadi/abkit/blob/main/LICENSE)
|
|
10
|
+
|
|
11
|
+
*Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.*
|
|
12
|
+
|
|
13
|
+
</div>
|
|
9
14
|
|
|
10
15
|
An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
|
|
11
16
|
split was broken, three whale customers carried the whole effect, the analyst
|
|
@@ -40,22 +45,22 @@ is indistinguishable from a real win — p < 0.05, decent sample size — but th
|
|
|
40
45
|
schedule it followed has a 19% false-positive rate, not 5%. The audit flags
|
|
41
46
|
the process, which is the only place that defect lives.
|
|
42
47
|
|
|
43
|
-

|
|
48
|
+

|
|
44
49
|
|
|
45
|
-

|
|
50
|
+

|
|
46
51
|
|
|
47
|
-
`make demo` regenerates the table, the [full report](results/report.md) and
|
|
52
|
+
`make demo` regenerates the table, the [full report](https://github.com/mohammadi-hadi/abkit/blob/main/results/report.md) and
|
|
48
53
|
the figures from fixed seeds; CI rebuilds them from pinned dependencies and
|
|
49
54
|
fails if a committed number differs from what the code produces.
|
|
50
55
|
|
|
51
56
|
## 4,873 real experiments, audited
|
|
52
57
|
|
|
53
|
-
[`examples/upworthy/`](examples/upworthy/) runs abkit's aggregate checks over
|
|
58
|
+
[`examples/upworthy/`](https://github.com/mohammadi-hadi/abkit/tree/main/examples/upworthy/) runs abkit's aggregate checks over
|
|
54
59
|
the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038/s41597-021-00934-7)
|
|
55
60
|
— 4,873 real headline A/B tests from 2013–2015. Committed results:
|
|
56
|
-
[examples/upworthy/results/upworthy.md](examples/upworthy/results/upworthy.md).
|
|
61
|
+
[examples/upworthy/results/upworthy.md](https://github.com/mohammadi-hadi/abkit/blob/main/examples/upworthy/results/upworthy.md).
|
|
57
62
|
|
|
58
|
-
- **
|
|
63
|
+
- **15% of real tests fail the sample-ratio check** at p < 0.001 against the
|
|
59
64
|
platform's equal-allocation design — and not because packages were added
|
|
60
65
|
mid-test; the rate is, if anything, higher in tests whose variants all
|
|
61
66
|
launched together. Those readouts carried unknown exposure bias.
|
|
@@ -157,7 +162,7 @@ experiment.
|
|
|
157
162
|
±0.10 — and flag only when the whole 95% bootstrap CI leaves it. Exact
|
|
158
163
|
procedures use the field's conventional thresholds: SRM at p < 0.001,
|
|
159
164
|
contamination at zero tolerance. All constants live in one place
|
|
160
|
-
([`probes.py`](src/abkit/probes.py)) and are easy to disagree with.
|
|
165
|
+
([`probes.py`](https://github.com/mohammadi-hadi/abkit/blob/main/src/abkit/probes.py)) and are easy to disagree with.
|
|
161
166
|
- **The stopping rule is audited, not the final p.** A peeked experiment's
|
|
162
167
|
final test statistic looks ordinary; what is broken is the procedure that
|
|
163
168
|
produced it. The peeking probe Monte-Carlos the analyst's actual look
|
|
@@ -210,20 +215,18 @@ The real-data case study uses the Upworthy Research Archive
|
|
|
210
215
|
abkit's contribution is packaging the checks as one auditable tool with
|
|
211
216
|
uncertainty on every number and validation by implantation.
|
|
212
217
|
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
[
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
[arenakit](https://github.com/mohammadi-hadi/arenakit) applies it to pairwise
|
|
219
|
-
model leaderboards.
|
|
218
|
+
## Part of evalstack
|
|
219
|
+
|
|
220
|
+
[evalstack](https://github.com/mohammadi-hadi/evalstack) is the map of these
|
|
221
|
+
eleven evaluation tools: what each one measures, what it found on real data,
|
|
222
|
+
and the two chains that run end to end.
|
|
220
223
|
|
|
221
224
|
## Citation
|
|
222
225
|
|
|
223
226
|
Releases are archived on Zenodo. Cite the concept DOI
|
|
224
227
|
[10.5281/zenodo.21809478](https://doi.org/10.5281/zenodo.21809478), which always
|
|
225
228
|
resolves to the latest version; structured metadata is in
|
|
226
|
-
[CITATION.cff](CITATION.cff).
|
|
229
|
+
[CITATION.cff](https://github.com/mohammadi-hadi/abkit/blob/main/CITATION.cff).
|
|
227
230
|
|
|
228
231
|
```bibtex
|
|
229
232
|
@software{mohammadi_abkit,
|
|
@@ -231,7 +234,7 @@ resolves to the latest version; structured metadata is in
|
|
|
231
234
|
title = {abkit: audit an A/B-test readout before you ship the decision},
|
|
232
235
|
url = {https://github.com/mohammadi-hadi/abkit},
|
|
233
236
|
doi = {10.5281/zenodo.21809478},
|
|
234
|
-
version = {0.1.
|
|
237
|
+
version = {0.1.2},
|
|
235
238
|
year = {2026}
|
|
236
239
|
}
|
|
237
240
|
```
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "abkit"
|
|
7
|
-
version = "0.1.
|
|
7
|
+
version = "0.1.2"
|
|
8
8
|
description = "Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
license = {text = "MIT"}
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: abkit
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts
|
|
5
5
|
Author: Hadi Mohammadi
|
|
6
6
|
License: MIT
|
|
@@ -28,14 +28,19 @@ Requires-Dist: scipy>=1.12; extra == "dev"
|
|
|
28
28
|
Requires-Dist: statsmodels>=0.14; extra == "dev"
|
|
29
29
|
Dynamic: license-file
|
|
30
30
|
|
|
31
|
-
|
|
31
|
+
<div align="center">
|
|
32
32
|
|
|
33
|
-
|
|
33
|
+
# abkit
|
|
34
34
|
|
|
35
35
|
[](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
|
|
36
36
|
[](https://pypi.org/project/abkit/)
|
|
37
37
|
[](https://doi.org/10.5281/zenodo.21809478)
|
|
38
|
-
[](https://www.python.org/downloads/)
|
|
39
|
+
[](https://github.com/mohammadi-hadi/abkit/blob/main/LICENSE)
|
|
40
|
+
|
|
41
|
+
*Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.*
|
|
42
|
+
|
|
43
|
+
</div>
|
|
39
44
|
|
|
40
45
|
An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
|
|
41
46
|
split was broken, three whale customers carried the whole effect, the analyst
|
|
@@ -70,22 +75,22 @@ is indistinguishable from a real win — p < 0.05, decent sample size — but th
|
|
|
70
75
|
schedule it followed has a 19% false-positive rate, not 5%. The audit flags
|
|
71
76
|
the process, which is the only place that defect lives.
|
|
72
77
|
|
|
73
|
-

|
|
78
|
+

|
|
74
79
|
|
|
75
|
-

|
|
80
|
+

|
|
76
81
|
|
|
77
|
-
`make demo` regenerates the table, the [full report](results/report.md) and
|
|
82
|
+
`make demo` regenerates the table, the [full report](https://github.com/mohammadi-hadi/abkit/blob/main/results/report.md) and
|
|
78
83
|
the figures from fixed seeds; CI rebuilds them from pinned dependencies and
|
|
79
84
|
fails if a committed number differs from what the code produces.
|
|
80
85
|
|
|
81
86
|
## 4,873 real experiments, audited
|
|
82
87
|
|
|
83
|
-
[`examples/upworthy/`](examples/upworthy/) runs abkit's aggregate checks over
|
|
88
|
+
[`examples/upworthy/`](https://github.com/mohammadi-hadi/abkit/tree/main/examples/upworthy/) runs abkit's aggregate checks over
|
|
84
89
|
the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038/s41597-021-00934-7)
|
|
85
90
|
— 4,873 real headline A/B tests from 2013–2015. Committed results:
|
|
86
|
-
[examples/upworthy/results/upworthy.md](examples/upworthy/results/upworthy.md).
|
|
91
|
+
[examples/upworthy/results/upworthy.md](https://github.com/mohammadi-hadi/abkit/blob/main/examples/upworthy/results/upworthy.md).
|
|
87
92
|
|
|
88
|
-
- **
|
|
93
|
+
- **15% of real tests fail the sample-ratio check** at p < 0.001 against the
|
|
89
94
|
platform's equal-allocation design — and not because packages were added
|
|
90
95
|
mid-test; the rate is, if anything, higher in tests whose variants all
|
|
91
96
|
launched together. Those readouts carried unknown exposure bias.
|
|
@@ -187,7 +192,7 @@ experiment.
|
|
|
187
192
|
±0.10 — and flag only when the whole 95% bootstrap CI leaves it. Exact
|
|
188
193
|
procedures use the field's conventional thresholds: SRM at p < 0.001,
|
|
189
194
|
contamination at zero tolerance. All constants live in one place
|
|
190
|
-
([`probes.py`](src/abkit/probes.py)) and are easy to disagree with.
|
|
195
|
+
([`probes.py`](https://github.com/mohammadi-hadi/abkit/blob/main/src/abkit/probes.py)) and are easy to disagree with.
|
|
191
196
|
- **The stopping rule is audited, not the final p.** A peeked experiment's
|
|
192
197
|
final test statistic looks ordinary; what is broken is the procedure that
|
|
193
198
|
produced it. The peeking probe Monte-Carlos the analyst's actual look
|
|
@@ -240,20 +245,18 @@ The real-data case study uses the Upworthy Research Archive
|
|
|
240
245
|
abkit's contribution is packaging the checks as one auditable tool with
|
|
241
246
|
uncertainty on every number and validation by implantation.
|
|
242
247
|
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
[
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
[arenakit](https://github.com/mohammadi-hadi/arenakit) applies it to pairwise
|
|
249
|
-
model leaderboards.
|
|
248
|
+
## Part of evalstack
|
|
249
|
+
|
|
250
|
+
[evalstack](https://github.com/mohammadi-hadi/evalstack) is the map of these
|
|
251
|
+
eleven evaluation tools: what each one measures, what it found on real data,
|
|
252
|
+
and the two chains that run end to end.
|
|
250
253
|
|
|
251
254
|
## Citation
|
|
252
255
|
|
|
253
256
|
Releases are archived on Zenodo. Cite the concept DOI
|
|
254
257
|
[10.5281/zenodo.21809478](https://doi.org/10.5281/zenodo.21809478), which always
|
|
255
258
|
resolves to the latest version; structured metadata is in
|
|
256
|
-
[CITATION.cff](CITATION.cff).
|
|
259
|
+
[CITATION.cff](https://github.com/mohammadi-hadi/abkit/blob/main/CITATION.cff).
|
|
257
260
|
|
|
258
261
|
```bibtex
|
|
259
262
|
@software{mohammadi_abkit,
|
|
@@ -261,7 +264,7 @@ resolves to the latest version; structured metadata is in
|
|
|
261
264
|
title = {abkit: audit an A/B-test readout before you ship the decision},
|
|
262
265
|
url = {https://github.com/mohammadi-hadi/abkit},
|
|
263
266
|
doi = {10.5281/zenodo.21809478},
|
|
264
|
-
version = {0.1.
|
|
267
|
+
version = {0.1.2},
|
|
265
268
|
year = {2026}
|
|
266
269
|
}
|
|
267
270
|
```
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|