abkit 0.1.0__tar.gz → 0.1.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {abkit-0.1.0/src/abkit.egg-info → abkit-0.1.1}/PKG-INFO +17 -8
- {abkit-0.1.0 → abkit-0.1.1}/README.md +15 -6
- {abkit-0.1.0 → abkit-0.1.1}/pyproject.toml +2 -2
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/__init__.py +2 -2
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/cli.py +3 -1
- {abkit-0.1.0 → abkit-0.1.1/src/abkit.egg-info}/PKG-INFO +17 -8
- {abkit-0.1.0 → abkit-0.1.1}/LICENSE +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/setup.cfg +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/audit.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/bootstrap.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/demo.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/io.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/probes.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/py.typed +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/report.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/result.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/schema.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/stats.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit/synthetic.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit.egg-info/SOURCES.txt +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit.egg-info/dependency_links.txt +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit.egg-info/entry_points.txt +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit.egg-info/requires.txt +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/src/abkit.egg-info/top_level.txt +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/tests/test_bootstrap.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/tests/test_cli.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/tests/test_probes.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/tests/test_result.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/tests/test_schema.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/tests/test_stats.py +0 -0
- {abkit-0.1.0 → abkit-0.1.1}/tests/test_synthetic.py +0 -0
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: abkit
|
|
3
|
-
Version: 0.1.
|
|
4
|
-
Summary:
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts
|
|
5
5
|
Author: Hadi Mohammadi
|
|
6
6
|
License: MIT
|
|
7
7
|
Project-URL: Homepage, https://github.com/mohammadi-hadi/abkit
|
|
@@ -30,9 +30,12 @@ Dynamic: license-file
|
|
|
30
30
|
|
|
31
31
|
# abkit
|
|
32
32
|
|
|
33
|
-
|
|
33
|
+
Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.
|
|
34
34
|
|
|
35
35
|
[](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
|
|
36
|
+
[](https://pypi.org/project/abkit/)
|
|
37
|
+
[](https://doi.org/10.5281/zenodo.21809478)
|
|
38
|
+
[](LICENSE)
|
|
36
39
|
|
|
37
40
|
An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
|
|
38
41
|
split was broken, three whale customers carried the whole effect, the analyst
|
|
@@ -96,7 +99,7 @@ the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038
|
|
|
96
99
|
## Install
|
|
97
100
|
|
|
98
101
|
```
|
|
99
|
-
pip install
|
|
102
|
+
pip install abkit
|
|
100
103
|
```
|
|
101
104
|
|
|
102
105
|
Python 3.11+. Runtime dependencies: numpy, pydantic, matplotlib.
|
|
@@ -239,19 +242,25 @@ uncertainty on every number and validation by implantation.
|
|
|
239
242
|
|
|
240
243
|
Companion projects: [judgekit](https://github.com/mohammadi-hadi/judgekit)
|
|
241
244
|
applies the same audit-before-you-trust pattern to LLM judges;
|
|
242
|
-
[
|
|
243
|
-
|
|
245
|
+
[raterkit](https://github.com/mohammadi-hadi/raterkit) applies it to the human
|
|
246
|
+
labels underneath both; [trajectory-judge](https://github.com/mohammadi-hadi/trajectory-judge)
|
|
247
|
+
measures what outcome-only judges miss on agent trajectories;
|
|
248
|
+
[arenakit](https://github.com/mohammadi-hadi/arenakit) applies it to pairwise
|
|
249
|
+
model leaderboards.
|
|
244
250
|
|
|
245
251
|
## Citation
|
|
246
252
|
|
|
247
|
-
|
|
248
|
-
[
|
|
253
|
+
Releases are archived on Zenodo. Cite the concept DOI
|
|
254
|
+
[10.5281/zenodo.21809478](https://doi.org/10.5281/zenodo.21809478), which always
|
|
255
|
+
resolves to the latest version; structured metadata is in
|
|
256
|
+
[CITATION.cff](CITATION.cff).
|
|
249
257
|
|
|
250
258
|
```bibtex
|
|
251
259
|
@software{mohammadi_abkit,
|
|
252
260
|
author = {Mohammadi, Hadi},
|
|
253
261
|
title = {abkit: audit an A/B-test readout before you ship the decision},
|
|
254
262
|
url = {https://github.com/mohammadi-hadi/abkit},
|
|
263
|
+
doi = {10.5281/zenodo.21809478},
|
|
255
264
|
version = {0.1.0},
|
|
256
265
|
year = {2026}
|
|
257
266
|
}
|
|
@@ -1,8 +1,11 @@
|
|
|
1
1
|
# abkit
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.
|
|
4
4
|
|
|
5
5
|
[](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
|
|
6
|
+
[](https://pypi.org/project/abkit/)
|
|
7
|
+
[](https://doi.org/10.5281/zenodo.21809478)
|
|
8
|
+
[](LICENSE)
|
|
6
9
|
|
|
7
10
|
An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
|
|
8
11
|
split was broken, three whale customers carried the whole effect, the analyst
|
|
@@ -66,7 +69,7 @@ the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038
|
|
|
66
69
|
## Install
|
|
67
70
|
|
|
68
71
|
```
|
|
69
|
-
pip install
|
|
72
|
+
pip install abkit
|
|
70
73
|
```
|
|
71
74
|
|
|
72
75
|
Python 3.11+. Runtime dependencies: numpy, pydantic, matplotlib.
|
|
@@ -209,19 +212,25 @@ uncertainty on every number and validation by implantation.
|
|
|
209
212
|
|
|
210
213
|
Companion projects: [judgekit](https://github.com/mohammadi-hadi/judgekit)
|
|
211
214
|
applies the same audit-before-you-trust pattern to LLM judges;
|
|
212
|
-
[
|
|
213
|
-
|
|
215
|
+
[raterkit](https://github.com/mohammadi-hadi/raterkit) applies it to the human
|
|
216
|
+
labels underneath both; [trajectory-judge](https://github.com/mohammadi-hadi/trajectory-judge)
|
|
217
|
+
measures what outcome-only judges miss on agent trajectories;
|
|
218
|
+
[arenakit](https://github.com/mohammadi-hadi/arenakit) applies it to pairwise
|
|
219
|
+
model leaderboards.
|
|
214
220
|
|
|
215
221
|
## Citation
|
|
216
222
|
|
|
217
|
-
|
|
218
|
-
[
|
|
223
|
+
Releases are archived on Zenodo. Cite the concept DOI
|
|
224
|
+
[10.5281/zenodo.21809478](https://doi.org/10.5281/zenodo.21809478), which always
|
|
225
|
+
resolves to the latest version; structured metadata is in
|
|
226
|
+
[CITATION.cff](CITATION.cff).
|
|
219
227
|
|
|
220
228
|
```bibtex
|
|
221
229
|
@software{mohammadi_abkit,
|
|
222
230
|
author = {Mohammadi, Hadi},
|
|
223
231
|
title = {abkit: audit an A/B-test readout before you ship the decision},
|
|
224
232
|
url = {https://github.com/mohammadi-hadi/abkit},
|
|
233
|
+
doi = {10.5281/zenodo.21809478},
|
|
225
234
|
version = {0.1.0},
|
|
226
235
|
year = {2026}
|
|
227
236
|
}
|
|
@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "abkit"
|
|
7
|
-
version = "0.1.
|
|
8
|
-
description = "
|
|
7
|
+
version = "0.1.1"
|
|
8
|
+
description = "Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
license = {text = "MIT"}
|
|
11
11
|
authors = [{name = "Hadi Mohammadi"}]
|
|
@@ -1,11 +1,11 @@
|
|
|
1
|
-
"""abkit:
|
|
1
|
+
"""abkit: sample-ratio, peeking, multiple-testing and winner's-curse checks."""
|
|
2
2
|
|
|
3
3
|
from .audit import run_audit
|
|
4
4
|
from .io import dump_design, dump_units, load_design, load_units
|
|
5
5
|
from .result import AuditResult, ProbeResult, SkippedProbe
|
|
6
6
|
from .schema import Design, Look, Unit
|
|
7
7
|
|
|
8
|
-
__version__ = "0.1.
|
|
8
|
+
__version__ = "0.1.1"
|
|
9
9
|
|
|
10
10
|
__all__ = [
|
|
11
11
|
"AuditResult",
|
|
@@ -14,7 +14,9 @@ from .report import inject_readme, write_report
|
|
|
14
14
|
|
|
15
15
|
def main(argv: list[str] | None = None) -> int:
|
|
16
16
|
parser = argparse.ArgumentParser(
|
|
17
|
-
prog="abkit",
|
|
17
|
+
prog="abkit",
|
|
18
|
+
description="Sample-ratio, peeking, multiple-testing and winner's-curse checks "
|
|
19
|
+
"for experiment readouts.",
|
|
18
20
|
)
|
|
19
21
|
parser.add_argument("--version", action="version", version=f"abkit {__version__}")
|
|
20
22
|
sub = parser.add_subparsers(dest="command", required=True)
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: abkit
|
|
3
|
-
Version: 0.1.
|
|
4
|
-
Summary:
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts
|
|
5
5
|
Author: Hadi Mohammadi
|
|
6
6
|
License: MIT
|
|
7
7
|
Project-URL: Homepage, https://github.com/mohammadi-hadi/abkit
|
|
@@ -30,9 +30,12 @@ Dynamic: license-file
|
|
|
30
30
|
|
|
31
31
|
# abkit
|
|
32
32
|
|
|
33
|
-
|
|
33
|
+
Sample-ratio, peeking, multiple-testing and winner's-curse checks for experiment readouts.
|
|
34
34
|
|
|
35
35
|
[](https://github.com/mohammadi-hadi/abkit/actions/workflows/ci.yml)
|
|
36
|
+
[](https://pypi.org/project/abkit/)
|
|
37
|
+
[](https://doi.org/10.5281/zenodo.21809478)
|
|
38
|
+
[](LICENSE)
|
|
36
39
|
|
|
37
40
|
An experiment readout can say "B wins, p = 0.03, ship it" while the traffic
|
|
38
41
|
split was broken, three whale customers carried the whole effect, the analyst
|
|
@@ -96,7 +99,7 @@ the exploratory split of the [Upworthy Research Archive](https://doi.org/10.1038
|
|
|
96
99
|
## Install
|
|
97
100
|
|
|
98
101
|
```
|
|
99
|
-
pip install
|
|
102
|
+
pip install abkit
|
|
100
103
|
```
|
|
101
104
|
|
|
102
105
|
Python 3.11+. Runtime dependencies: numpy, pydantic, matplotlib.
|
|
@@ -239,19 +242,25 @@ uncertainty on every number and validation by implantation.
|
|
|
239
242
|
|
|
240
243
|
Companion projects: [judgekit](https://github.com/mohammadi-hadi/judgekit)
|
|
241
244
|
applies the same audit-before-you-trust pattern to LLM judges;
|
|
242
|
-
[
|
|
243
|
-
|
|
245
|
+
[raterkit](https://github.com/mohammadi-hadi/raterkit) applies it to the human
|
|
246
|
+
labels underneath both; [trajectory-judge](https://github.com/mohammadi-hadi/trajectory-judge)
|
|
247
|
+
measures what outcome-only judges miss on agent trajectories;
|
|
248
|
+
[arenakit](https://github.com/mohammadi-hadi/arenakit) applies it to pairwise
|
|
249
|
+
model leaderboards.
|
|
244
250
|
|
|
245
251
|
## Citation
|
|
246
252
|
|
|
247
|
-
|
|
248
|
-
[
|
|
253
|
+
Releases are archived on Zenodo. Cite the concept DOI
|
|
254
|
+
[10.5281/zenodo.21809478](https://doi.org/10.5281/zenodo.21809478), which always
|
|
255
|
+
resolves to the latest version; structured metadata is in
|
|
256
|
+
[CITATION.cff](CITATION.cff).
|
|
249
257
|
|
|
250
258
|
```bibtex
|
|
251
259
|
@software{mohammadi_abkit,
|
|
252
260
|
author = {Mohammadi, Hadi},
|
|
253
261
|
title = {abkit: audit an A/B-test readout before you ship the decision},
|
|
254
262
|
url = {https://github.com/mohammadi-hadi/abkit},
|
|
263
|
+
doi = {10.5281/zenodo.21809478},
|
|
255
264
|
version = {0.1.0},
|
|
256
265
|
year = {2026}
|
|
257
266
|
}
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|