bayes-ab 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +254 -0
- package/dist/cli.js +1447 -0
- package/dist/core/analyze.d.ts +106 -0
- package/dist/core/math.d.ts +20 -0
- package/dist/core/plan.d.ts +41 -0
- package/dist/core/project.d.ts +58 -0
- package/dist/core/share.d.ts +10 -0
- package/dist/index.d.ts +8 -0
- package/dist/index.js +580 -0
- package/package.json +42 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Joe Brundage
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,254 @@
|
|
|
1
|
+
# bayes-ab
|
|
2
|
+
|
|
3
|
+
Bayesian A/B test analysis on the command line, for humans and for agents.
|
|
4
|
+
|
|
5
|
+
It is the [Bayesian A/B Test Calculator](https://www.joebrundage.com/tools/bayesian-ab-test) as a command. Same math, same defaults, plus a forward projection that answers the question the web page cannot: **how long will this test take to decide, and which way?**
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
$ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714
|
|
9
|
+
|
|
10
|
+
Bayesian A/B test Beta(1,1) prior · decision threshold 95% · meaningful effect ±10% relative
|
|
11
|
+
|
|
12
|
+
Decision Continue test
|
|
13
|
+
Evidence Low (46% of the way to a stable estimate)
|
|
14
|
+
Bayes action now Ship B (expected regret 0.0050 pp, about 0.50 missed conversions per 10,000 visitors)
|
|
15
|
+
|
|
16
|
+
Variant A Variant B
|
|
17
|
+
Visitors 12,000 11,850
|
|
18
|
+
Conversions 660 714
|
|
19
|
+
Observed rate 5.50% 6.03%
|
|
20
|
+
Posterior mean 5.507% 6.033%
|
|
21
|
+
Posterior Beta(661, 11,341) Beta(715, 11,137)
|
|
22
|
+
|
|
23
|
+
P(B beats A) 95.91%
|
|
24
|
+
P(relative lift > +10%) 46.42%
|
|
25
|
+
P(relative lift < -10%) 0.02%
|
|
26
|
+
P(B is harmful at all) 4.06%
|
|
27
|
+
|
|
28
|
+
Relative lift +9.54% 95% CI -1.17% to +21.30%
|
|
29
|
+
Absolute difference +0.53 pp 95% CI -0.07 pp to +1.11 pp
|
|
30
|
+
Observed difference +0.53 pp relative +9.55%
|
|
31
|
+
|
|
32
|
+
Expected loss if you ship B 0.0050 pp
|
|
33
|
+
Expected loss if you ship A 0.5280 pp
|
|
34
|
+
Two-sided p-value 0.082 frequentist reference only
|
|
35
|
+
...
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Install
|
|
39
|
+
|
|
40
|
+
Requires Node.js 18.11 or newer. Check with `node --version`.
|
|
41
|
+
|
|
42
|
+
**Global install**, so `bayes-ab` is on your PATH:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
npm install -g bayes-ab
|
|
46
|
+
bayes-ab --help
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
**No install**, run it straight from the registry:
|
|
50
|
+
|
|
51
|
+
```sh
|
|
52
|
+
npx bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
**From source**, if you are hacking on it:
|
|
56
|
+
|
|
57
|
+
```sh
|
|
58
|
+
git clone https://github.com/brundagejoe/bayes-ab.git
|
|
59
|
+
cd bayes-ab
|
|
60
|
+
npm install
|
|
61
|
+
npm run build
|
|
62
|
+
npm link # puts a bayes-ab symlink on your PATH pointing at this checkout
|
|
63
|
+
bayes-ab --version
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
`npm link` is reversible with `npm unlink -g bayes-ab`. To upgrade a global install later, run `npm install -g bayes-ab@latest`.
|
|
67
|
+
|
|
68
|
+
If `npm install -g` fails with a permissions error on macOS or Linux, the clean fix is to point npm at a directory you own rather than using `sudo`:
|
|
69
|
+
|
|
70
|
+
```sh
|
|
71
|
+
mkdir -p ~/.npm-global
|
|
72
|
+
npm config set prefix ~/.npm-global
|
|
73
|
+
echo 'export PATH="$HOME/.npm-global/bin:$PATH"' >> ~/.zshrc # or ~/.bashrc
|
|
74
|
+
source ~/.zshrc
|
|
75
|
+
npm install -g bayes-ab
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## Commands
|
|
79
|
+
|
|
80
|
+
| Command | Question it answers |
|
|
81
|
+
|---|---|
|
|
82
|
+
| `analyze` | Given what I have observed, what should I do? The web tool's full readout. |
|
|
83
|
+
| `project` | If traffic continues at this pace, how long until this test decides, and which way? |
|
|
84
|
+
| `plan` | Before any data: how many visitors per variant do I need to detect a given effect? |
|
|
85
|
+
| `schema` | What fields will the JSON output contain? |
|
|
86
|
+
|
|
87
|
+
### analyze
|
|
88
|
+
|
|
89
|
+
```sh
|
|
90
|
+
bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714
|
|
91
|
+
bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 --threshold 90 --min-effect 5
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
Ask a what-if by giving a rate instead of a count. The tool rounds to the nearest whole conversion and tells you what it used:
|
|
95
|
+
|
|
96
|
+
```sh
|
|
97
|
+
bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 10000 --rate-b 6.1% --brief
|
|
98
|
+
# Continue test · P(B beats A) 97.13% · P(lift > +10%) 56.76% · lift +10.92% (CI -0.30% to +23.39%) · loss if ship B 0.0034 pp B = 610/10000 from --rate-b 6.10%
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### project
|
|
102
|
+
|
|
103
|
+
Takes the current posterior, draws a plausible pair of true conversion rates, simulates each future day of traffic, and re-runs the decision rule after every day. Repeats that thousands of times.
|
|
104
|
+
|
|
105
|
+
```sh
|
|
106
|
+
bayes-ab project --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 --daily-visitors 1500
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
```
|
|
110
|
+
Forward projection 1,500 visitors/day per variant · 28-day horizon · 2,000 simulated futures
|
|
111
|
+
Stopping rule P(lift > +10%) ≥ 95% or P(lift < -10%) ≥ 95%
|
|
112
|
+
|
|
113
|
+
Chance of a decision within 28 days 20%
|
|
114
|
+
Ship B 20%
|
|
115
|
+
Keep A 0%
|
|
116
|
+
Still running at day 28 80%
|
|
117
|
+
|
|
118
|
+
If it decides, when?
|
|
119
|
+
25% of decisions by day 10
|
|
120
|
+
50% 14
|
|
121
|
+
75% 21
|
|
122
|
+
90% 25
|
|
123
|
+
|
|
124
|
+
Cumulative chance of a decision
|
|
125
|
+
by day 7 3%
|
|
126
|
+
by day 14 10%
|
|
127
|
+
by day 21 16%
|
|
128
|
+
by day 28 20%
|
|
129
|
+
|
|
130
|
+
Why so slow The estimated lift sits on the ±10% meaningful-effect line.
|
|
131
|
+
More data sharpens the estimate around that line instead of pushing it across.
|
|
132
|
+
Try a smaller --min-effect, or --stop-rule superiority to ask only whether B beats A.
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Three stopping rules:
|
|
136
|
+
|
|
137
|
+
- `lift` (default) matches the web tool's decision status: stop when the probability of a meaningful lift, or a meaningful loss, reaches the threshold.
|
|
138
|
+
- `superiority` stops when P(B beats A) or P(A beats B) reaches the threshold. Ignores `--min-effect`. Decides sooner.
|
|
139
|
+
- `loss` stops when the expected loss of the better variant drops below `--max-loss` (in percentage points, default 0.01).
|
|
140
|
+
|
|
141
|
+
If the rule is already met by today's data, `project` says so and skips the simulation.
|
|
142
|
+
|
|
143
|
+
### plan
|
|
144
|
+
|
|
145
|
+
```sh
|
|
146
|
+
bayes-ab plan --baseline 5.5% --min-effect 10 --daily-visitors 1500
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
```
|
|
150
|
+
Sample size plan baseline 5.50% · detect a ±10% relative change (to 6.05% or 4.95%) · 80% power, 5% alpha
|
|
151
|
+
|
|
152
|
+
Visitors per variant 26,972
|
|
153
|
+
Total visitors 53,944
|
|
154
|
+
Days at 1,500/day/variant 18
|
|
155
|
+
|
|
156
|
+
Sensitivity
|
|
157
|
+
min effect 5% 107,887 per variant 72 days
|
|
158
|
+
min effect 10% 26,972 per variant 18 days ← you are here
|
|
159
|
+
min effect 15% 11,988 per variant 8 days
|
|
160
|
+
min effect 20% 6,743 per variant 5 days
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
This is the frequentist sample-size formula, the same yardstick the web tool uses for its maturity bar. It is a planning number, not the Bayesian decision rule.
|
|
164
|
+
|
|
165
|
+
## Output for agents and scripts
|
|
166
|
+
|
|
167
|
+
When stdout is not a terminal, output is one JSON object on one line. `--json` forces that; `--pretty` indents it.
|
|
168
|
+
|
|
169
|
+
```sh
|
|
170
|
+
bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 --json --pretty
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
```json
|
|
174
|
+
{
|
|
175
|
+
"tool": "bayes-ab",
|
|
176
|
+
"version": "0.1.0",
|
|
177
|
+
"command": "analyze",
|
|
178
|
+
"inputs": { "visitorsA": 12000, "conversionsA": 660, "visitorsB": 11850, "conversionsB": 714, "threshold": 0.95, "minEffect": 0.1, "prior": [1, 1], "samples": 12000, "seed": 112396 },
|
|
179
|
+
"decision": { "status": "continue", "label": "Continue test", "evidence": "low", "maturity": "building", "maturityProgress": 0.462241, "bayesActionNow": "B", "expectedRegretNow": 0.00005 },
|
|
180
|
+
"probability": { "bBeatsA": 0.959114, "aBeatsB": 0.040886, "meaningfulLift": 0.464167, "meaningfulHarm": 0.000167, "harmful": 0.040583, "liftAbove1pct": 0.937833, "liftAbove5pct": 0.788417, "harmBeyond5pct": 0.00325, "harmBeyond10pct": 0.000167 },
|
|
181
|
+
"effect": { "relativeLift": 0.095384, "relativeLiftCI95": [-0.011657, 0.213021], "absoluteDiff": 0.005253, "absoluteDiffCI95": [-0.000675, 0.011123], "observedRelativeLift": 0.095512, "observedAbsoluteDiff": 0.005253 },
|
|
182
|
+
"expectedLoss": { "shipA": 0.00528, "shipB": 0.00005 },
|
|
183
|
+
"variants": { "A": { "...": "..." }, "B": { "...": "..." } },
|
|
184
|
+
"sampleSize": { "requiredPerVariant": 25636, "additionalPerVariant": 13786, "detectableEffectNow": 0.147083 },
|
|
185
|
+
"waiting": [ { "extraPerVariant": 1000, "expectedRegret": 0.00006, "valueOfWaiting": 0 }, "..." ],
|
|
186
|
+
"frequentist": { "pValueTwoSided": 0.081709 },
|
|
187
|
+
"shareUrl": "https://www.joebrundage.com/tools/bayesian-ab-test?visitorsA=12000&conversionsA=660&visitorsB=11850&conversionsB=714&thresholdPercent=95&meaningfulLiftPercent=10"
|
|
188
|
+
}
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Rules that hold everywhere:
|
|
192
|
+
|
|
193
|
+
- Rates, probabilities and losses are fractions in JSON (`0.0603`), never percent strings. Numbers are rounded to six decimals.
|
|
194
|
+
- `decision.status` is one of `ship_b`, `keep_a`, `continue`, `inconclusive`.
|
|
195
|
+
- Identical inputs give identical output. Seeds derive from the inputs; `--seed N` overrides.
|
|
196
|
+
- Errors are JSON on stdout when output is JSON, with exit code 2: `{"error":{"code":"invalid_input","message":"...","field":"conversions-a"}}`.
|
|
197
|
+
- Every analysis includes `shareUrl`, the web tool with these inputs filled in.
|
|
198
|
+
- `bayes-ab schema analyze` prints the JSON Schema for a command's output.
|
|
199
|
+
|
|
200
|
+
Because each call is stateless and prints one line, sweeps parallelize trivially:
|
|
201
|
+
|
|
202
|
+
```sh
|
|
203
|
+
for d in 750 1500 3000; do
|
|
204
|
+
bayes-ab project --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 \
|
|
205
|
+
--daily-visitors $d --json &
|
|
206
|
+
done; wait
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
### Giving an agent the tool
|
|
210
|
+
|
|
211
|
+
The `--help` text is written to be pasted into an agent's context. It covers every flag, how to read each output field, and worked examples. For Claude Code, Codex or similar, either of these works:
|
|
212
|
+
|
|
213
|
+
```sh
|
|
214
|
+
bayes-ab --help > .claude/bayes-ab-help.txt # then reference it from CLAUDE.md
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
or add a line like this to your agent instructions:
|
|
218
|
+
|
|
219
|
+
> `bayes-ab` is installed. Run `bayes-ab --help` before using it. Use `analyze` to read a test, `project` to estimate how long it needs, and `plan` to size a new one. Prefer `--json`. Run several calls in parallel when sweeping a question.
|
|
220
|
+
|
|
221
|
+
## Using it as a library
|
|
222
|
+
|
|
223
|
+
The math is exported without the CLI:
|
|
224
|
+
|
|
225
|
+
```ts
|
|
226
|
+
import { analyze, project, plan } from "bayes-ab"
|
|
227
|
+
|
|
228
|
+
const result = analyze({ visitorsA: 12000, conversionsA: 660, visitorsB: 11850, conversionsB: 714 })
|
|
229
|
+
result.probability.bBeatsA // 0.9591138673472563
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
## Method
|
|
233
|
+
|
|
234
|
+
- Each conversion rate gets a Beta(1,1) prior. Posteriors are Beta(1 + conversions, 1 + non-conversions).
|
|
235
|
+
- P(B beats A) is the exact closed form from [Evan Miller](https://www.evanmiller.org/bayesian-ab-testing.html).
|
|
236
|
+
- Lift intervals, threshold probabilities and expected loss come from 12,000 seeded Monte Carlo draws.
|
|
237
|
+
- The meaningful effect is relative and symmetric: `--min-effect 10` means B must be 10% better than A to count as a win or 10% worse to count as a loss.
|
|
238
|
+
- `project` samples true rates from the exact posterior, then uses a normal approximation to the posterior when re-checking the stopping rule on each simulated day. Day zero is checked exactly.
|
|
239
|
+
|
|
240
|
+
## Development
|
|
241
|
+
|
|
242
|
+
```sh
|
|
243
|
+
npm install
|
|
244
|
+
npm test # node's built-in runner against the TypeScript source
|
|
245
|
+
npm run typecheck
|
|
246
|
+
npm run build # bundles dist/cli.js and dist/index.js with esbuild
|
|
247
|
+
node dist/cli.js --help
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
Tests pin the outputs to the web tool's default example so the two never drift.
|
|
251
|
+
|
|
252
|
+
## License
|
|
253
|
+
|
|
254
|
+
MIT
|