bayes-ab 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Joe Brundage
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,254 @@
1
+ # bayes-ab
2
+
3
+ Bayesian A/B test analysis on the command line, for humans and for agents.
4
+
5
+ It is the [Bayesian A/B Test Calculator](https://www.joebrundage.com/tools/bayesian-ab-test) as a command. Same math, same defaults, plus a forward projection that answers the question the web page cannot: **how long will this test take to decide, and which way?**
6
+
7
+ ```
8
+ $ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714
9
+
10
+ Bayesian A/B test Beta(1,1) prior · decision threshold 95% · meaningful effect ±10% relative
11
+
12
+ Decision Continue test
13
+ Evidence Low (46% of the way to a stable estimate)
14
+ Bayes action now Ship B (expected regret 0.0050 pp, about 0.50 missed conversions per 10,000 visitors)
15
+
16
+ Variant A Variant B
17
+ Visitors 12,000 11,850
18
+ Conversions 660 714
19
+ Observed rate 5.50% 6.03%
20
+ Posterior mean 5.507% 6.033%
21
+ Posterior Beta(661, 11,341) Beta(715, 11,137)
22
+
23
+ P(B beats A) 95.91%
24
+ P(relative lift > +10%) 46.42%
25
+ P(relative lift < -10%) 0.02%
26
+ P(B is harmful at all) 4.06%
27
+
28
+ Relative lift +9.54% 95% CI -1.17% to +21.30%
29
+ Absolute difference +0.53 pp 95% CI -0.07 pp to +1.11 pp
30
+ Observed difference +0.53 pp relative +9.55%
31
+
32
+ Expected loss if you ship B 0.0050 pp
33
+ Expected loss if you ship A 0.5280 pp
34
+ Two-sided p-value 0.082 frequentist reference only
35
+ ...
36
+ ```
37
+
38
+ ## Install
39
+
40
+ Requires Node.js 18.11 or newer. Check with `node --version`.
41
+
42
+ **Global install**, so `bayes-ab` is on your PATH:
43
+
44
+ ```sh
45
+ npm install -g bayes-ab
46
+ bayes-ab --help
47
+ ```
48
+
49
+ **No install**, run it straight from the registry:
50
+
51
+ ```sh
52
+ npx bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714
53
+ ```
54
+
55
+ **From source**, if you are hacking on it:
56
+
57
+ ```sh
58
+ git clone https://github.com/brundagejoe/bayes-ab.git
59
+ cd bayes-ab
60
+ npm install
61
+ npm run build
62
+ npm link # puts a bayes-ab symlink on your PATH pointing at this checkout
63
+ bayes-ab --version
64
+ ```
65
+
66
+ `npm link` is reversible with `npm unlink -g bayes-ab`. To upgrade a global install later, run `npm install -g bayes-ab@latest`.
67
+
68
+ If `npm install -g` fails with a permissions error on macOS or Linux, the clean fix is to point npm at a directory you own rather than using `sudo`:
69
+
70
+ ```sh
71
+ mkdir -p ~/.npm-global
72
+ npm config set prefix ~/.npm-global
73
+ echo 'export PATH="$HOME/.npm-global/bin:$PATH"' >> ~/.zshrc # or ~/.bashrc
74
+ source ~/.zshrc
75
+ npm install -g bayes-ab
76
+ ```
77
+
78
+ ## Commands
79
+
80
+ | Command | Question it answers |
81
+ |---|---|
82
+ | `analyze` | Given what I have observed, what should I do? The web tool's full readout. |
83
+ | `project` | If traffic continues at this pace, how long until this test decides, and which way? |
84
+ | `plan` | Before any data: how many visitors per variant do I need to detect a given effect? |
85
+ | `schema` | What fields will the JSON output contain? |
86
+
87
+ ### analyze
88
+
89
+ ```sh
90
+ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714
91
+ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 --threshold 90 --min-effect 5
92
+ ```
93
+
94
+ Ask a what-if by giving a rate instead of a count. The tool rounds to the nearest whole conversion and tells you what it used:
95
+
96
+ ```sh
97
+ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 10000 --rate-b 6.1% --brief
98
+ # Continue test · P(B beats A) 97.13% · P(lift > +10%) 56.76% · lift +10.92% (CI -0.30% to +23.39%) · loss if ship B 0.0034 pp B = 610/10000 from --rate-b 6.10%
99
+ ```
100
+
101
+ ### project
102
+
103
+ Takes the current posterior, draws a plausible pair of true conversion rates, simulates each future day of traffic, and re-runs the decision rule after every day. Repeats that thousands of times.
104
+
105
+ ```sh
106
+ bayes-ab project --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 --daily-visitors 1500
107
+ ```
108
+
109
+ ```
110
+ Forward projection 1,500 visitors/day per variant · 28-day horizon · 2,000 simulated futures
111
+ Stopping rule P(lift > +10%) ≥ 95% or P(lift < -10%) ≥ 95%
112
+
113
+ Chance of a decision within 28 days 20%
114
+ Ship B 20%
115
+ Keep A 0%
116
+ Still running at day 28 80%
117
+
118
+ If it decides, when?
119
+ 25% of decisions by day 10
120
+ 50% 14
121
+ 75% 21
122
+ 90% 25
123
+
124
+ Cumulative chance of a decision
125
+ by day 7 3%
126
+ by day 14 10%
127
+ by day 21 16%
128
+ by day 28 20%
129
+
130
+ Why so slow The estimated lift sits on the ±10% meaningful-effect line.
131
+ More data sharpens the estimate around that line instead of pushing it across.
132
+ Try a smaller --min-effect, or --stop-rule superiority to ask only whether B beats A.
133
+ ```
134
+
135
+ Three stopping rules:
136
+
137
+ - `lift` (default) matches the web tool's decision status: stop when the probability of a meaningful lift, or a meaningful loss, reaches the threshold.
138
+ - `superiority` stops when P(B beats A) or P(A beats B) reaches the threshold. Ignores `--min-effect`. Decides sooner.
139
+ - `loss` stops when the expected loss of the better variant drops below `--max-loss` (in percentage points, default 0.01).
140
+
141
+ If the rule is already met by today's data, `project` says so and skips the simulation.
142
+
143
+ ### plan
144
+
145
+ ```sh
146
+ bayes-ab plan --baseline 5.5% --min-effect 10 --daily-visitors 1500
147
+ ```
148
+
149
+ ```
150
+ Sample size plan baseline 5.50% · detect a ±10% relative change (to 6.05% or 4.95%) · 80% power, 5% alpha
151
+
152
+ Visitors per variant 26,972
153
+ Total visitors 53,944
154
+ Days at 1,500/day/variant 18
155
+
156
+ Sensitivity
157
+ min effect 5% 107,887 per variant 72 days
158
+ min effect 10% 26,972 per variant 18 days ← you are here
159
+ min effect 15% 11,988 per variant 8 days
160
+ min effect 20% 6,743 per variant 5 days
161
+ ```
162
+
163
+ This is the frequentist sample-size formula, the same yardstick the web tool uses for its maturity bar. It is a planning number, not the Bayesian decision rule.
164
+
165
+ ## Output for agents and scripts
166
+
167
+ When stdout is not a terminal, output is one JSON object on one line. `--json` forces that; `--pretty` indents it.
168
+
169
+ ```sh
170
+ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 --json --pretty
171
+ ```
172
+
173
+ ```json
174
+ {
175
+ "tool": "bayes-ab",
176
+ "version": "0.1.0",
177
+ "command": "analyze",
178
+ "inputs": { "visitorsA": 12000, "conversionsA": 660, "visitorsB": 11850, "conversionsB": 714, "threshold": 0.95, "minEffect": 0.1, "prior": [1, 1], "samples": 12000, "seed": 112396 },
179
+ "decision": { "status": "continue", "label": "Continue test", "evidence": "low", "maturity": "building", "maturityProgress": 0.462241, "bayesActionNow": "B", "expectedRegretNow": 0.00005 },
180
+ "probability": { "bBeatsA": 0.959114, "aBeatsB": 0.040886, "meaningfulLift": 0.464167, "meaningfulHarm": 0.000167, "harmful": 0.040583, "liftAbove1pct": 0.937833, "liftAbove5pct": 0.788417, "harmBeyond5pct": 0.00325, "harmBeyond10pct": 0.000167 },
181
+ "effect": { "relativeLift": 0.095384, "relativeLiftCI95": [-0.011657, 0.213021], "absoluteDiff": 0.005253, "absoluteDiffCI95": [-0.000675, 0.011123], "observedRelativeLift": 0.095512, "observedAbsoluteDiff": 0.005253 },
182
+ "expectedLoss": { "shipA": 0.00528, "shipB": 0.00005 },
183
+ "variants": { "A": { "...": "..." }, "B": { "...": "..." } },
184
+ "sampleSize": { "requiredPerVariant": 25636, "additionalPerVariant": 13786, "detectableEffectNow": 0.147083 },
185
+ "waiting": [ { "extraPerVariant": 1000, "expectedRegret": 0.00006, "valueOfWaiting": 0 }, "..." ],
186
+ "frequentist": { "pValueTwoSided": 0.081709 },
187
+ "shareUrl": "https://www.joebrundage.com/tools/bayesian-ab-test?visitorsA=12000&conversionsA=660&visitorsB=11850&conversionsB=714&thresholdPercent=95&meaningfulLiftPercent=10"
188
+ }
189
+ ```
190
+
191
+ Rules that hold everywhere:
192
+
193
+ - Rates, probabilities and losses are fractions in JSON (`0.0603`), never percent strings. Numbers are rounded to six decimals.
194
+ - `decision.status` is one of `ship_b`, `keep_a`, `continue`, `inconclusive`.
195
+ - Identical inputs give identical output. Seeds derive from the inputs; `--seed N` overrides.
196
+ - Errors are JSON on stdout when output is JSON, with exit code 2: `{"error":{"code":"invalid_input","message":"...","field":"conversions-a"}}`.
197
+ - Every analysis includes `shareUrl`, the web tool with these inputs filled in.
198
+ - `bayes-ab schema analyze` prints the JSON Schema for a command's output.
199
+
200
+ Because each call is stateless and prints one line, sweeps parallelize trivially:
201
+
202
+ ```sh
203
+ for d in 750 1500 3000; do
204
+ bayes-ab project --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 \
205
+ --daily-visitors $d --json &
206
+ done; wait
207
+ ```
208
+
209
+ ### Giving an agent the tool
210
+
211
+ The `--help` text is written to be pasted into an agent's context. It covers every flag, how to read each output field, and worked examples. For Claude Code, Codex or similar, either of these works:
212
+
213
+ ```sh
214
+ bayes-ab --help > .claude/bayes-ab-help.txt # then reference it from CLAUDE.md
215
+ ```
216
+
217
+ or add a line like this to your agent instructions:
218
+
219
+ > `bayes-ab` is installed. Run `bayes-ab --help` before using it. Use `analyze` to read a test, `project` to estimate how long it needs, and `plan` to size a new one. Prefer `--json`. Run several calls in parallel when sweeping a question.
220
+
221
+ ## Using it as a library
222
+
223
+ The math is exported without the CLI:
224
+
225
+ ```ts
226
+ import { analyze, project, plan } from "bayes-ab"
227
+
228
+ const result = analyze({ visitorsA: 12000, conversionsA: 660, visitorsB: 11850, conversionsB: 714 })
229
+ result.probability.bBeatsA // 0.9591138673472563
230
+ ```
231
+
232
+ ## Method
233
+
234
+ - Each conversion rate gets a Beta(1,1) prior. Posteriors are Beta(1 + conversions, 1 + non-conversions).
235
+ - P(B beats A) is the exact closed form from [Evan Miller](https://www.evanmiller.org/bayesian-ab-testing.html).
236
+ - Lift intervals, threshold probabilities and expected loss come from 12,000 seeded Monte Carlo draws.
237
+ - The meaningful effect is relative and symmetric: `--min-effect 10` means B must be 10% better than A to count as a win or 10% worse to count as a loss.
238
+ - `project` samples true rates from the exact posterior, then uses a normal approximation to the posterior when re-checking the stopping rule on each simulated day. Day zero is checked exactly.
239
+
240
+ ## Development
241
+
242
+ ```sh
243
+ npm install
244
+ npm test # node's built-in runner against the TypeScript source
245
+ npm run typecheck
246
+ npm run build # bundles dist/cli.js and dist/index.js with esbuild
247
+ node dist/cli.js --help
248
+ ```
249
+
250
+ Tests pin the outputs to the web tool's default example so the two never drift.
251
+
252
+ ## License
253
+
254
+ MIT