bayes-ab 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -98,6 +98,14 @@ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 10000 --rat
98
98
  # Continue test · P(B beats A) 97.13% · P(lift > +10%) 56.76% · lift +10.92% (CI -0.30% to +23.39%) · loss if ship B 0.0034 pp B = 610/10000 from --rate-b 6.10%
99
99
  ```
100
100
 
101
+ Choose a prior. Every analysis defaults to a flat Beta(1,1) on each conversion rate; `--prior-alpha` and `--prior-beta` replace it, the same prior for both variants. Beta(α, β) reads as α conversions in α + β visitors of evidence you already had, so a prior worth 1,000 visitors centred on 5.5% is Beta(55, 945). Fractional values work, e.g. the Jeffreys prior Beta(0.5, 0.5):
102
+
103
+ ```sh
104
+ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 --prior-alpha 55 --prior-beta 945
105
+ ```
106
+
107
+ A shared prior pulls both arms toward the same rate, so a strong one shrinks the estimated lift and makes decisions more conservative.
108
+
101
109
  ### project
102
110
 
103
111
  Takes the current posterior, draws a plausible pair of true conversion rates, simulates each future day of traffic, and re-runs the decision rule after every day. Repeats that thousands of times.
@@ -173,7 +181,7 @@ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --con
173
181
  ```json
174
182
  {
175
183
  "tool": "bayes-ab",
176
- "version": "0.2.0",
184
+ "version": "0.3.0",
177
185
  "command": "analyze",
178
186
  "inputs": { "visitorsA": 12000, "conversionsA": 660, "visitorsB": 11850, "conversionsB": 714, "threshold": 0.95, "minEffect": 0.1, "prior": [1, 1], "samples": 12000, "seed": 116218 },
179
187
  "decision": { "status": "continue", "label": "Continue test", "evidence": "low", "maturity": "building", "maturityProgress": 0.462241, "bayesActionNow": "B", "expectedRegretNow": 0.00005 },
@@ -227,8 +235,12 @@ import { analyze, project, plan } from "bayes-ab"
227
235
 
228
236
  const result = analyze({ visitorsA: 12000, conversionsA: 660, visitorsB: 11850, conversionsB: 714 })
229
237
  result.probability.bBeatsA // 0.9591138673472563
238
+
239
+ analyze({ visitorsA: 12000, conversionsA: 660, visitorsB: 11850, conversionsB: 714, prior: [55, 945] })
230
240
  ```
231
241
 
242
+ `analyze` and `project` take an optional `prior: [alpha, beta]`, default `[1, 1]`. Non-positive values throw a `RangeError`; `validatePrior` and `isDefaultPrior` are exported for forms that want to check first.
243
+
232
244
  `analyze({ ..., includeChart: true })` adds `chart` (both posterior densities) and `liftChart` (the exact density of the relative lift, peak scaled to 1), which is what the web tool plots.
233
245
 
234
246
  The primitives behind it are exported too, for anything that needs the same math without a full analysis:
@@ -236,7 +248,8 @@ The primitives behind it are exported too, for anything that needs the same math
236
248
  | Export | What it does |
237
249
  |---|---|
238
250
  | `betaPdf(x, alpha, beta)`, `logBeta`, `logGamma` | Beta density and its log-gamma building blocks |
239
- | `probabilityBBeatsA(alphaA, betaA, alphaB, betaB)` | Exact P(B > A) for two Beta posteriors |
251
+ | `probabilityBBeatsA(alphaA, betaA, alphaB, betaB)` | P(B > A) for two Beta posteriors: exact closed form for whole-number alphas, numerical integration otherwise |
252
+ | `regularizedIncompleteBeta(x, a, b)` | The Beta(a, b) CDF at x |
240
253
  | `integratedExpectedLoss(alphaA, betaA, alphaB, betaB)` | Expected loss of shipping each variant by numerical integration, no sampling |
241
254
  | `liftDensity(alphaA, betaA, alphaB, betaB, sortedLifts)` | Exact relative-lift density over the range the sorted lifts span |
242
255
 
@@ -244,8 +257,8 @@ This package is the source of truth for the math on joebrundage.com: the [calcul
244
257
 
245
258
  ## Method
246
259
 
247
- - Each conversion rate gets a Beta(1,1) prior. Posteriors are Beta(1 + conversions, 1 + non-conversions).
248
- - P(B beats A) is the exact closed form from [Evan Miller](https://www.evanmiller.org/bayesian-ab-testing.html). P(B is harmful) is its exact complement.
260
+ - Each conversion rate gets a Beta(α, β) prior, Beta(1,1) unless you choose another. Posteriors are Beta(α + conversions, β + non-conversions).
261
+ - P(B beats A) is the exact closed form from [Evan Miller](https://www.evanmiller.org/bayesian-ab-testing.html) when α is a whole number. For a fractional α it is integrated numerically (tanh-sinh quadrature over the incomplete beta function), accurate to about 1e-10. P(B is harmful) is its exact complement.
249
262
  - Lift intervals, threshold probabilities and expected loss come from 12,000 seeded Monte Carlo draws.
250
263
  - The meaningful effect is relative and symmetric: `--min-effect 10` means B must be 10% better than A to count as a win or 10% worse to count as a loss.
251
264
  - `project` samples true rates from the exact posterior, then uses a normal approximation to the posterior when re-checking the stopping rule on each simulated day. Day zero is checked exactly.
package/dist/cli.js CHANGED
@@ -119,11 +119,80 @@ function probabilityRightBeatsLeftDirect(alphaLeft, betaLeft, alphaRight, betaRi
119
119
  return Math.exp(maxLog) * scaledSum;
120
120
  }
121
121
  function probabilityBBeatsA(alphaA, betaA, alphaB, betaB) {
122
+ if (!Number.isInteger(alphaA) || !Number.isInteger(alphaB)) {
123
+ return probabilityBBeatsANumeric(alphaA, betaA, alphaB, betaB);
124
+ }
122
125
  if (alphaB <= alphaA) {
123
126
  return probabilityRightBeatsLeftDirect(alphaA, betaA, alphaB, betaB);
124
127
  }
125
128
  return 1 - probabilityRightBeatsLeftDirect(alphaB, betaB, alphaA, betaA);
126
129
  }
130
+ function betaContinuedFraction(x, a, b) {
131
+ const TINY = 1e-300;
132
+ const EPSILON = 1e-15;
133
+ const MAX_ITERATIONS = 1e5;
134
+ const qab = a + b;
135
+ const qap = a + 1;
136
+ const qam = a - 1;
137
+ let c = 1;
138
+ let d = 1 - qab * x / qap;
139
+ if (Math.abs(d) < TINY) d = TINY;
140
+ d = 1 / d;
141
+ let h = d;
142
+ for (let m = 1; m <= MAX_ITERATIONS; m += 1) {
143
+ const m2 = 2 * m;
144
+ let aa = m * (b - m) * x / ((qam + m2) * (a + m2));
145
+ d = 1 + aa * d;
146
+ if (Math.abs(d) < TINY) d = TINY;
147
+ c = 1 + aa / c;
148
+ if (Math.abs(c) < TINY) c = TINY;
149
+ d = 1 / d;
150
+ h *= d * c;
151
+ aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2));
152
+ d = 1 + aa * d;
153
+ if (Math.abs(d) < TINY) d = TINY;
154
+ c = 1 + aa / c;
155
+ if (Math.abs(c) < TINY) c = TINY;
156
+ d = 1 / d;
157
+ const delta = d * c;
158
+ h *= delta;
159
+ if (Math.abs(delta - 1) < EPSILON) break;
160
+ }
161
+ return h;
162
+ }
163
+ function regularizedIncompleteBeta(x, a, b) {
164
+ if (x <= 0) return 0;
165
+ if (x >= 1) return 1;
166
+ if (x > (a + 1) / (a + b + 2)) return 1 - regularizedIncompleteBeta(1 - x, b, a);
167
+ const logFront = a * Math.log(x) + b * Math.log1p(-x) - logBeta(a, b);
168
+ return Math.exp(logFront) * betaContinuedFraction(x, a, b) / a;
169
+ }
170
+ function probabilityBBeatsANumeric(alphaA, betaA, alphaB, betaB) {
171
+ const total = alphaA + betaA;
172
+ const meanA = alphaA / total;
173
+ const sdA = Math.sqrt(alphaA * betaA / (total * total * (total + 1)));
174
+ const low = Math.max(0, meanA - 40 * sdA);
175
+ const high = Math.min(1, meanA + 40 * sdA);
176
+ const center = (low + high) / 2;
177
+ const halfWidth = (high - low) / 2;
178
+ const logNormalizer = logBeta(alphaA, betaA);
179
+ const STEP = 1 / 128;
180
+ const T_MAX = 3.5;
181
+ let mass = 0;
182
+ let weighted = 0;
183
+ for (let t = -T_MAX; t <= T_MAX + 1e-12; t += STEP) {
184
+ const inner = Math.PI / 2 * Math.sinh(t);
185
+ const x = center + halfWidth * Math.tanh(inner);
186
+ if (x <= 0 || x >= 1) continue;
187
+ const coshInner = Math.cosh(inner);
188
+ const weight = Math.PI / 2 * Math.cosh(t) / (coshInner * coshInner);
189
+ const density = Math.exp((alphaA - 1) * Math.log(x) + (betaA - 1) * Math.log1p(-x) - logNormalizer);
190
+ if (!Number.isFinite(density)) continue;
191
+ mass += weight * density;
192
+ weighted += weight * density * (1 - regularizedIncompleteBeta(x, alphaB, betaB));
193
+ }
194
+ return mass > 0 ? weighted / mass : 0.5;
195
+ }
127
196
  function inverseNormalCdf(p) {
128
197
  const a = [-39.69683028665376, 220.9460984245205, -275.9285104469687, 138.357751867269, -30.66479806614716, 2.506628277459239];
129
198
  const b = [-54.47609879822406, 161.5858368580409, -155.6989798598866, 66.80131188771972, -13.28068155288572];
@@ -248,6 +317,10 @@ function shareUrl(inputs) {
248
317
  thresholdPercent: formatPercentParam(inputs.threshold),
249
318
  meaningfulLiftPercent: formatPercentParam(inputs.minEffect)
250
319
  });
320
+ if (inputs.prior && !(inputs.prior[0] === 1 && inputs.prior[1] === 1)) {
321
+ params.set("priorAlpha", String(inputs.prior[0]));
322
+ params.set("priorBeta", String(inputs.prior[1]));
323
+ }
251
324
  return `${WEB_TOOL_URL}?${params.toString()}`;
252
325
  }
253
326
 
@@ -266,6 +339,13 @@ var DECISION_LABELS = {
266
339
  continue: "Continue test",
267
340
  inconclusive: "Inconclusive"
268
341
  };
342
+ function validatePrior(prior) {
343
+ const [alpha, beta] = prior;
344
+ if (!(Number.isFinite(alpha) && alpha > 0 && Number.isFinite(beta) && beta > 0)) {
345
+ throw new RangeError(`Prior parameters must be finite and greater than 0, got Beta(${alpha}, ${beta}).`);
346
+ }
347
+ return [alpha, beta];
348
+ }
269
349
  function deriveSeed(...parts) {
270
350
  let hash = 2166136261;
271
351
  for (const part of parts) {
@@ -348,12 +428,14 @@ function analyze({
348
428
  minEffect = DEFAULT_MIN_EFFECT,
349
429
  samples = DEFAULT_SAMPLES,
350
430
  seed = null,
351
- includeChart = false
431
+ includeChart = false,
432
+ prior = [PRIOR_ALPHA, PRIOR_BETA]
352
433
  }) {
353
- const alphaA = PRIOR_ALPHA + conversionsA;
354
- const betaA = PRIOR_BETA + (visitorsA - conversionsA);
355
- const alphaB = PRIOR_ALPHA + conversionsB;
356
- const betaB = PRIOR_BETA + (visitorsB - conversionsB);
434
+ const [priorAlpha, priorBeta] = validatePrior(prior);
435
+ const alphaA = priorAlpha + conversionsA;
436
+ const betaA = priorBeta + (visitorsA - conversionsA);
437
+ const alphaB = priorAlpha + conversionsB;
438
+ const betaB = priorBeta + (visitorsB - conversionsB);
357
439
  const rateA = conversionsA / visitorsA;
358
440
  const rateB = conversionsB / visitorsB;
359
441
  const posteriorMeanA = alphaA / (alphaA + betaA);
@@ -388,7 +470,7 @@ function analyze({
388
470
  conversionsB,
389
471
  threshold,
390
472
  minEffect,
391
- prior: [PRIOR_ALPHA, PRIOR_BETA],
473
+ prior: [priorAlpha, priorBeta],
392
474
  samples,
393
475
  seed: resolvedSeed
394
476
  },
@@ -436,7 +518,7 @@ function analyze({
436
518
  },
437
519
  waiting: waitingScenarios(alphaA, betaA, alphaB, betaB, expectedRegretNow),
438
520
  frequentist: { pValueTwoSided: twoSidedPValue(visitorsA, conversionsA, visitorsB, conversionsB) },
439
- shareUrl: shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect })
521
+ shareUrl: shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] })
440
522
  };
441
523
  if (includeChart) {
442
524
  const sortedA = [...samplesA].sort((left, right) => left - right);
@@ -524,9 +606,11 @@ function project({
524
606
  stopRule = "lift",
525
607
  maxLoss = DEFAULT_MAX_LOSS,
526
608
  simulations = DEFAULT_SIMULATIONS,
527
- seed = null
609
+ seed = null,
610
+ prior = [PRIOR_ALPHA, PRIOR_BETA]
528
611
  }) {
529
- const current = analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect });
612
+ const [priorAlpha, priorBeta] = validatePrior(prior);
613
+ const current = analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] });
530
614
  const resolvedSeed = seed ?? deriveSeed(visitorsA, conversionsA, visitorsB, conversionsB, dailyVisitors, horizonDays, ["lift", "superiority", "loss"].indexOf(stopRule), simulations);
531
615
  const inputs = {
532
616
  visitorsA,
@@ -535,7 +619,7 @@ function project({
535
619
  conversionsB,
536
620
  threshold,
537
621
  minEffect,
538
- prior: [PRIOR_ALPHA, PRIOR_BETA],
622
+ prior: [priorAlpha, priorBeta],
539
623
  dailyVisitors,
540
624
  horizonDays,
541
625
  stopRule,
@@ -543,7 +627,7 @@ function project({
543
627
  simulations,
544
628
  seed: resolvedSeed
545
629
  };
546
- const link = shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect });
630
+ const link = shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] });
547
631
  const diagnostic = stopRule === "lift" && liftNearThreshold(current, minEffect) ? "lift_near_threshold" : null;
548
632
  const decisionNow = decisionNowFrom(current, stopRule, threshold, maxLoss);
549
633
  if (decisionNow) {
@@ -560,10 +644,10 @@ function project({
560
644
  shareUrl: link
561
645
  };
562
646
  }
563
- const alphaA0 = PRIOR_ALPHA + conversionsA;
564
- const betaA0 = PRIOR_BETA + (visitorsA - conversionsA);
565
- const alphaB0 = PRIOR_ALPHA + conversionsB;
566
- const betaB0 = PRIOR_BETA + (visitorsB - conversionsB);
647
+ const alphaA0 = priorAlpha + conversionsA;
648
+ const betaA0 = priorBeta + (visitorsA - conversionsA);
649
+ const alphaB0 = priorAlpha + conversionsB;
650
+ const betaB0 = priorBeta + (visitorsB - conversionsB);
567
651
  const random = makeDeterministicRandom(resolvedSeed);
568
652
  const decidedByDay = new Array(horizonDays + 1).fill(0);
569
653
  const decisionDays = [];
@@ -643,6 +727,8 @@ var OPTIONS = {
643
727
  "rate-b": { type: "string" },
644
728
  threshold: { type: "string" },
645
729
  "min-effect": { type: "string" },
730
+ "prior-alpha": { type: "string" },
731
+ "prior-beta": { type: "string" },
646
732
  "daily-visitors": { type: "string" },
647
733
  horizon: { type: "string" },
648
734
  "stop-rule": { type: "string" },
@@ -733,10 +819,19 @@ function parseMinEffect(values) {
733
819
  if (minEffect < 0 || minEffect >= 1) throw new CliError("--min-effect must be a percent between 0 and 100.", { field: "min-effect" });
734
820
  return minEffect;
735
821
  }
822
+ function parsePriorParameter(flag, value, fallback) {
823
+ if (value === void 0) return fallback;
824
+ const parsed = parseNumber(flag, value);
825
+ if (parsed <= 0) throw new CliError(`--${flag} must be greater than 0, got "${value}".`, { field: flag });
826
+ return parsed;
827
+ }
828
+ function parsePrior(values) {
829
+ return [parsePriorParameter("prior-alpha", values["prior-alpha"], PRIOR_ALPHA), parsePriorParameter("prior-beta", values["prior-beta"], PRIOR_BETA)];
830
+ }
736
831
  function parseAnalyzeArgs(values) {
737
832
  const samples = values.samples === void 0 ? DEFAULT_SAMPLES : parseCount("samples", values.samples, { min: 100 });
738
833
  const seed = values.seed === void 0 ? null : parseCount("seed", values.seed);
739
- return { ...parseVariants(values), threshold: parseThreshold(values), minEffect: parseMinEffect(values), samples, seed };
834
+ return { ...parseVariants(values), threshold: parseThreshold(values), minEffect: parseMinEffect(values), prior: parsePrior(values), samples, seed };
740
835
  }
741
836
  function parseStopRule(value) {
742
837
  if (value === void 0) return "lift";
@@ -812,7 +907,8 @@ function helpText(version) {
812
907
  return `bayes-ab ${version} Bayesian A/B test analysis for humans and agents
813
908
 
814
909
  Compares two variants of a binary outcome (signup, click, purchase) using
815
- Beta(1,1) posteriors on each conversion rate. Reports the probability that B
910
+ Beta posteriors on each conversion rate, from a flat Beta(1,1) prior unless
911
+ you choose another. Reports the probability that B
816
912
  beats A, the probability the lift clears a meaningful threshold, expected
817
913
  loss from shipping either variant, and a forward projection of how long the
818
914
  test needs to run. Same math as ${WEB_TOOL_URL}
@@ -839,6 +935,12 @@ INPUTS (analyze, project)
839
935
  --min-effect PCT Meaningful relative effect, symmetric: with 10, B must be
840
936
  10% better than A to count as a win or 10% worse to count
841
937
  as a loss. Relative, not percentage points. default 10
938
+ --prior-alpha N, --prior-beta N Beta(alpha, beta) prior on each variant's conversion rate,
939
+ the same for both. Any positive number. Reads as alpha
940
+ conversions in alpha + beta visitors of prior evidence,
941
+ with prior mean alpha / (alpha + beta).
942
+ Beta(1,1) is flat; Beta(0.5,0.5) is Jeffreys; Beta(55,945)
943
+ says "about 5.5%, worth 1,000 visitors". default 1, 1
842
944
  Whole numbers may include commas or underscores: 12,000 and 12_000 both work.
843
945
 
844
946
  PROJECTION (project)
@@ -934,6 +1036,10 @@ function unsignedPoints(fraction2, digits = 4) {
934
1036
  function count(value) {
935
1037
  return Math.round(value).toLocaleString("en-US");
936
1038
  }
1039
+ function betaParameter(value) {
1040
+ if (Number.isInteger(value)) return count(value);
1041
+ return value.toLocaleString("en-US", { maximumFractionDigits: 3 });
1042
+ }
937
1043
  function missedConversions(regretRate, visitors = 1e4) {
938
1044
  const missed = regretRate * visitors;
939
1045
  const digits = missed < 1 ? 2 : missed < 10 ? 1 : 0;
@@ -987,6 +1093,8 @@ function renderAnalyzeBrief(result, derivedFrom2) {
987
1093
  `lift ${signedPercent(result.effect.relativeLift)} (CI ${signedPercent(lower)} to ${signedPercent(upper)})`,
988
1094
  `loss if ship B ${unsignedPoints(result.expectedLoss.shipB)}`
989
1095
  ];
1096
+ const [priorAlpha, priorBeta] = result.inputs.prior;
1097
+ if (priorAlpha !== 1 || priorBeta !== 1) parts.push(`Beta(${betaParameter(priorAlpha)},${betaParameter(priorBeta)}) prior`);
990
1098
  const note = derivedNote(result, derivedFrom2);
991
1099
  return parts.join(" \xB7 ") + (note ? ` ${note}` : "");
992
1100
  }
@@ -998,7 +1106,7 @@ function renderAnalyze(result, derivedFrom2) {
998
1106
  const lines = [];
999
1107
  const note = derivedNote(result, derivedFrom2);
1000
1108
  lines.push(
1001
- `Bayesian A/B test Beta(${inputs.prior[0]},${inputs.prior[1]}) prior \xB7 decision threshold ${percent(inputs.threshold, 0)} \xB7 meaningful effect \xB1${minEffect} relative`
1109
+ `Bayesian A/B test Beta(${betaParameter(inputs.prior[0])},${betaParameter(inputs.prior[1])}) prior \xB7 decision threshold ${percent(inputs.threshold, 0)} \xB7 meaningful effect \xB1${minEffect} relative`
1002
1110
  );
1003
1111
  if (note) lines.push(` ${note}`);
1004
1112
  lines.push("");
@@ -1024,7 +1132,7 @@ function renderAnalyze(result, derivedFrom2) {
1024
1132
  ["Conversions", count(variants.A.conversions), count(variants.B.conversions)],
1025
1133
  ["Observed rate", percent(variants.A.rate), percent(variants.B.rate)],
1026
1134
  ["Posterior mean", percent(variants.A.posteriorMean, 3), percent(variants.B.posteriorMean, 3)],
1027
- ["Posterior", `Beta(${count(variants.A.posterior[0])}, ${count(variants.A.posterior[1])})`, `Beta(${count(variants.B.posterior[0])}, ${count(variants.B.posterior[1])})`]
1135
+ ["Posterior", `Beta(${betaParameter(variants.A.posterior[0])}, ${betaParameter(variants.A.posterior[1])})`, `Beta(${betaParameter(variants.B.posterior[0])}, ${betaParameter(variants.B.posterior[1])})`]
1028
1136
  ])
1029
1137
  );
1030
1138
  lines.push("");
@@ -1162,6 +1270,8 @@ function renderProjectBrief(result) {
1162
1270
  result.daysToDecision ? `median day ${result.daysToDecision.p50} if it decides (p90 day ${result.daysToDecision.p90})` : "no simulated future decided",
1163
1271
  `${count(dailyVisitors)}/day per variant`
1164
1272
  ];
1273
+ const [priorAlpha, priorBeta] = result.inputs.prior;
1274
+ if (priorAlpha !== 1 || priorBeta !== 1) parts.push(`Beta(${betaParameter(priorAlpha)},${betaParameter(priorBeta)}) prior`);
1165
1275
  return parts.join(" \xB7 ");
1166
1276
  }
1167
1277
  function renderProject(result) {
@@ -1171,6 +1281,8 @@ function renderProject(result) {
1171
1281
  `Forward projection ${count(inputs.dailyVisitors)} visitors/day per variant \xB7 ${inputs.horizonDays}-day horizon \xB7 ${count(inputs.simulations)} simulated futures`
1172
1282
  );
1173
1283
  lines.push(`Stopping rule ${describeRule(result)}`);
1284
+ const [priorAlpha, priorBeta] = inputs.prior;
1285
+ if (priorAlpha !== 1 || priorBeta !== 1) lines.push(`Prior Beta(${betaParameter(priorAlpha)},${betaParameter(priorBeta)}) on each variant`);
1174
1286
  lines.push("");
1175
1287
  if (result.alreadyDecided && result.decisionNow) {
1176
1288
  lines.push(`Already decided ${decisionLabel(result.decisionNow)}`);
@@ -1236,7 +1348,7 @@ var variantInputs = {
1236
1348
  conversionsB: integer,
1237
1349
  threshold: fraction,
1238
1350
  minEffect: fraction,
1239
- prior: pair(number)
1351
+ prior: { ...pair({ type: "number", exclusiveMinimum: 0 }), description: "The Beta(alpha, beta) prior applied to each variant. [1, 1] unless --prior-alpha or --prior-beta was given." }
1240
1352
  };
1241
1353
  var derivedFrom = {
1242
1354
  ...object({ rateA: fraction, rateB: fraction }, []),
@@ -1400,7 +1512,7 @@ var SCHEMAS = {
1400
1512
  };
1401
1513
 
1402
1514
  // src/cli/main.ts
1403
- var VERSION = true ? "0.2.0" : "dev";
1515
+ var VERSION = true ? "0.3.0" : "dev";
1404
1516
  var JSON_DECIMALS = 6;
1405
1517
  function roundDeep(value) {
1406
1518
  if (typeof value === "number") return round(value, JSON_DECIMALS);
@@ -19,6 +19,8 @@ export type AnalyzeInputs = {
19
19
  samples?: number;
20
20
  seed?: number | null;
21
21
  includeChart?: boolean;
22
+ /** Beta(alpha, beta) prior applied to both variants. Defaults to the flat Beta(1, 1). */
23
+ prior?: [number, number];
22
24
  };
23
25
  export type VariantSummary = {
24
26
  visitors: number;
@@ -100,6 +102,9 @@ export type AnalyzeResult = {
100
102
  liftChart?: LiftDensityPoint[];
101
103
  shareUrl: string;
102
104
  };
105
+ /** Throws a RangeError unless both prior parameters are finite and positive. */
106
+ export declare function validatePrior(prior: [number, number]): [number, number];
107
+ export declare function isDefaultPrior([alpha, beta]: [number, number]): boolean;
103
108
  export declare function deriveSeed(...parts: number[]): number;
104
109
  export declare function decisionStatus({ probabilityMeaningfulLift, probabilityMeaningfulHarm, threshold, evidence, }: {
105
110
  probabilityMeaningfulLift: number;
@@ -107,4 +112,4 @@ export declare function decisionStatus({ probabilityMeaningfulLift, probabilityM
107
112
  threshold: number;
108
113
  evidence: EvidenceStrength;
109
114
  }): DecisionStatus;
110
- export declare function analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, samples, seed, includeChart, }: AnalyzeInputs): AnalyzeResult;
115
+ export declare function analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, samples, seed, includeChart, prior, }: AnalyzeInputs): AnalyzeResult;
@@ -10,8 +10,16 @@ export declare function sampleBeta(alpha: number, beta: number, random: Random):
10
10
  export declare function sampleBinomialApprox(trials: number, probability: number, random: Random): number;
11
11
  export declare function quantile(sortedValues: number[], probability: number): number;
12
12
  export declare function mean(values: number[]): number;
13
- /** Exact P(B > A) for independent Beta posteriors, via Evan Miller's closed form. */
13
+ /**
14
+ * P(B > A) for independent Beta posteriors. Exact via Evan Miller's closed form
15
+ * when the alphas are whole numbers, which they are under any whole-number
16
+ * prior. The closed form sums over 0..alpha-1, so a fractional alpha (a
17
+ * Jeffreys Beta(0.5, 0.5) prior, say) falls back to numerical integration,
18
+ * accurate to roughly 1e-10.
19
+ */
14
20
  export declare function probabilityBBeatsA(alphaA: number, betaA: number, alphaB: number, betaB: number): number;
21
+ /** Regularized incomplete beta function I_x(a, b): the Beta(a, b) CDF at x. */
22
+ export declare function regularizedIncompleteBeta(x: number, a: number, b: number): number;
15
23
  export declare function inverseNormalCdf(p: number): number;
16
24
  export declare function erf(x: number): number;
17
25
  export declare function normalCdf(x: number): number;
@@ -16,6 +16,8 @@ export type ProjectInputs = {
16
16
  maxLoss?: number;
17
17
  simulations?: number;
18
18
  seed?: number | null;
19
+ /** Beta(alpha, beta) prior applied to both variants. Defaults to the flat Beta(1, 1). */
20
+ prior?: [number, number];
19
21
  };
20
22
  export type ProjectResult = {
21
23
  inputs: {
@@ -55,4 +57,4 @@ export type ProjectResult = {
55
57
  approximation: "normal";
56
58
  shareUrl: string;
57
59
  };
58
- export declare function project({ visitorsA, conversionsA, visitorsB, conversionsB, dailyVisitors, horizonDays, threshold, minEffect, stopRule, maxLoss, simulations, seed, }: ProjectInputs): ProjectResult;
60
+ export declare function project({ visitorsA, conversionsA, visitorsB, conversionsB, dailyVisitors, horizonDays, threshold, minEffect, stopRule, maxLoss, simulations, seed, prior, }: ProjectInputs): ProjectResult;
@@ -6,5 +6,6 @@ export type ShareInputs = {
6
6
  conversionsB: number;
7
7
  threshold: number;
8
8
  minEffect: number;
9
+ prior?: [number, number];
9
10
  };
10
11
  export declare function shareUrl(inputs: ShareInputs): string;
package/dist/index.d.ts CHANGED
@@ -1,11 +1,11 @@
1
- export { analyze, DECISION_LABELS, DEFAULT_MIN_EFFECT, DEFAULT_SAMPLES, DEFAULT_THRESHOLD, PRIOR_ALPHA, PRIOR_BETA } from "./core/analyze.ts";
1
+ export { analyze, DECISION_LABELS, DEFAULT_MIN_EFFECT, DEFAULT_SAMPLES, DEFAULT_THRESHOLD, PRIOR_ALPHA, PRIOR_BETA, isDefaultPrior, validatePrior } from "./core/analyze.ts";
2
2
  export type { AnalyzeInputs, AnalyzeResult, ChartPoint, DecisionStatus, EvidenceStrength, Maturity, Variant, VariantSummary, WaitingScenario, } from "./core/analyze.ts";
3
3
  export { project, DEFAULT_HORIZON_DAYS, DEFAULT_MAX_LOSS, DEFAULT_SIMULATIONS } from "./core/project.ts";
4
4
  export type { ProjectInputs, ProjectResult, ProjectedDecision, StopRule } from "./core/project.ts";
5
5
  export { plan, requiredSamplePerVariant } from "./core/plan.ts";
6
6
  export type { PlanInputs, PlanResult, PlanRow } from "./core/plan.ts";
7
7
  export { shareUrl, WEB_TOOL_URL } from "./core/share.ts";
8
- export { betaPdf, logBeta, logGamma, probabilityBBeatsA } from "./core/math.ts";
8
+ export { betaPdf, logBeta, logGamma, probabilityBBeatsA, regularizedIncompleteBeta } from "./core/math.ts";
9
9
  export { liftDensity } from "./core/lift.ts";
10
10
  export type { LiftDensityPoint } from "./core/lift.ts";
11
11
  export { integratedExpectedLoss } from "./core/loss.ts";
package/dist/index.js CHANGED
@@ -117,11 +117,80 @@ function probabilityRightBeatsLeftDirect(alphaLeft, betaLeft, alphaRight, betaRi
117
117
  return Math.exp(maxLog) * scaledSum;
118
118
  }
119
119
  function probabilityBBeatsA(alphaA, betaA, alphaB, betaB) {
120
+ if (!Number.isInteger(alphaA) || !Number.isInteger(alphaB)) {
121
+ return probabilityBBeatsANumeric(alphaA, betaA, alphaB, betaB);
122
+ }
120
123
  if (alphaB <= alphaA) {
121
124
  return probabilityRightBeatsLeftDirect(alphaA, betaA, alphaB, betaB);
122
125
  }
123
126
  return 1 - probabilityRightBeatsLeftDirect(alphaB, betaB, alphaA, betaA);
124
127
  }
128
+ function betaContinuedFraction(x, a, b) {
129
+ const TINY = 1e-300;
130
+ const EPSILON = 1e-15;
131
+ const MAX_ITERATIONS = 1e5;
132
+ const qab = a + b;
133
+ const qap = a + 1;
134
+ const qam = a - 1;
135
+ let c = 1;
136
+ let d = 1 - qab * x / qap;
137
+ if (Math.abs(d) < TINY) d = TINY;
138
+ d = 1 / d;
139
+ let h = d;
140
+ for (let m = 1; m <= MAX_ITERATIONS; m += 1) {
141
+ const m2 = 2 * m;
142
+ let aa = m * (b - m) * x / ((qam + m2) * (a + m2));
143
+ d = 1 + aa * d;
144
+ if (Math.abs(d) < TINY) d = TINY;
145
+ c = 1 + aa / c;
146
+ if (Math.abs(c) < TINY) c = TINY;
147
+ d = 1 / d;
148
+ h *= d * c;
149
+ aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2));
150
+ d = 1 + aa * d;
151
+ if (Math.abs(d) < TINY) d = TINY;
152
+ c = 1 + aa / c;
153
+ if (Math.abs(c) < TINY) c = TINY;
154
+ d = 1 / d;
155
+ const delta = d * c;
156
+ h *= delta;
157
+ if (Math.abs(delta - 1) < EPSILON) break;
158
+ }
159
+ return h;
160
+ }
161
+ function regularizedIncompleteBeta(x, a, b) {
162
+ if (x <= 0) return 0;
163
+ if (x >= 1) return 1;
164
+ if (x > (a + 1) / (a + b + 2)) return 1 - regularizedIncompleteBeta(1 - x, b, a);
165
+ const logFront = a * Math.log(x) + b * Math.log1p(-x) - logBeta(a, b);
166
+ return Math.exp(logFront) * betaContinuedFraction(x, a, b) / a;
167
+ }
168
+ function probabilityBBeatsANumeric(alphaA, betaA, alphaB, betaB) {
169
+ const total = alphaA + betaA;
170
+ const meanA = alphaA / total;
171
+ const sdA = Math.sqrt(alphaA * betaA / (total * total * (total + 1)));
172
+ const low = Math.max(0, meanA - 40 * sdA);
173
+ const high = Math.min(1, meanA + 40 * sdA);
174
+ const center = (low + high) / 2;
175
+ const halfWidth = (high - low) / 2;
176
+ const logNormalizer = logBeta(alphaA, betaA);
177
+ const STEP = 1 / 128;
178
+ const T_MAX = 3.5;
179
+ let mass = 0;
180
+ let weighted = 0;
181
+ for (let t = -T_MAX; t <= T_MAX + 1e-12; t += STEP) {
182
+ const inner = Math.PI / 2 * Math.sinh(t);
183
+ const x = center + halfWidth * Math.tanh(inner);
184
+ if (x <= 0 || x >= 1) continue;
185
+ const coshInner = Math.cosh(inner);
186
+ const weight = Math.PI / 2 * Math.cosh(t) / (coshInner * coshInner);
187
+ const density = Math.exp((alphaA - 1) * Math.log(x) + (betaA - 1) * Math.log1p(-x) - logNormalizer);
188
+ if (!Number.isFinite(density)) continue;
189
+ mass += weight * density;
190
+ weighted += weight * density * (1 - regularizedIncompleteBeta(x, alphaB, betaB));
191
+ }
192
+ return mass > 0 ? weighted / mass : 0.5;
193
+ }
125
194
  function inverseNormalCdf(p) {
126
195
  const a = [-39.69683028665376, 220.9460984245205, -275.9285104469687, 138.357751867269, -30.66479806614716, 2.506628277459239];
127
196
  const b = [-54.47609879822406, 161.5858368580409, -155.6989798598866, 66.80131188771972, -13.28068155288572];
@@ -241,6 +310,10 @@ function shareUrl(inputs) {
241
310
  thresholdPercent: formatPercentParam(inputs.threshold),
242
311
  meaningfulLiftPercent: formatPercentParam(inputs.minEffect)
243
312
  });
313
+ if (inputs.prior && !(inputs.prior[0] === 1 && inputs.prior[1] === 1)) {
314
+ params.set("priorAlpha", String(inputs.prior[0]));
315
+ params.set("priorBeta", String(inputs.prior[1]));
316
+ }
244
317
  return `${WEB_TOOL_URL}?${params.toString()}`;
245
318
  }
246
319
 
@@ -259,6 +332,16 @@ var DECISION_LABELS = {
259
332
  continue: "Continue test",
260
333
  inconclusive: "Inconclusive"
261
334
  };
335
+ function validatePrior(prior) {
336
+ const [alpha, beta] = prior;
337
+ if (!(Number.isFinite(alpha) && alpha > 0 && Number.isFinite(beta) && beta > 0)) {
338
+ throw new RangeError(`Prior parameters must be finite and greater than 0, got Beta(${alpha}, ${beta}).`);
339
+ }
340
+ return [alpha, beta];
341
+ }
342
+ function isDefaultPrior([alpha, beta]) {
343
+ return alpha === PRIOR_ALPHA && beta === PRIOR_BETA;
344
+ }
262
345
  function deriveSeed(...parts) {
263
346
  let hash = 2166136261;
264
347
  for (const part of parts) {
@@ -341,12 +424,14 @@ function analyze({
341
424
  minEffect = DEFAULT_MIN_EFFECT,
342
425
  samples = DEFAULT_SAMPLES,
343
426
  seed = null,
344
- includeChart = false
427
+ includeChart = false,
428
+ prior = [PRIOR_ALPHA, PRIOR_BETA]
345
429
  }) {
346
- const alphaA = PRIOR_ALPHA + conversionsA;
347
- const betaA = PRIOR_BETA + (visitorsA - conversionsA);
348
- const alphaB = PRIOR_ALPHA + conversionsB;
349
- const betaB = PRIOR_BETA + (visitorsB - conversionsB);
430
+ const [priorAlpha, priorBeta] = validatePrior(prior);
431
+ const alphaA = priorAlpha + conversionsA;
432
+ const betaA = priorBeta + (visitorsA - conversionsA);
433
+ const alphaB = priorAlpha + conversionsB;
434
+ const betaB = priorBeta + (visitorsB - conversionsB);
350
435
  const rateA = conversionsA / visitorsA;
351
436
  const rateB = conversionsB / visitorsB;
352
437
  const posteriorMeanA = alphaA / (alphaA + betaA);
@@ -381,7 +466,7 @@ function analyze({
381
466
  conversionsB,
382
467
  threshold,
383
468
  minEffect,
384
- prior: [PRIOR_ALPHA, PRIOR_BETA],
469
+ prior: [priorAlpha, priorBeta],
385
470
  samples,
386
471
  seed: resolvedSeed
387
472
  },
@@ -429,7 +514,7 @@ function analyze({
429
514
  },
430
515
  waiting: waitingScenarios(alphaA, betaA, alphaB, betaB, expectedRegretNow),
431
516
  frequentist: { pValueTwoSided: twoSidedPValue(visitorsA, conversionsA, visitorsB, conversionsB) },
432
- shareUrl: shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect })
517
+ shareUrl: shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] })
433
518
  };
434
519
  if (includeChart) {
435
520
  const sortedA = [...samplesA].sort((left, right) => left - right);
@@ -517,9 +602,11 @@ function project({
517
602
  stopRule = "lift",
518
603
  maxLoss = DEFAULT_MAX_LOSS,
519
604
  simulations = DEFAULT_SIMULATIONS,
520
- seed = null
605
+ seed = null,
606
+ prior = [PRIOR_ALPHA, PRIOR_BETA]
521
607
  }) {
522
- const current = analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect });
608
+ const [priorAlpha, priorBeta] = validatePrior(prior);
609
+ const current = analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] });
523
610
  const resolvedSeed = seed ?? deriveSeed(visitorsA, conversionsA, visitorsB, conversionsB, dailyVisitors, horizonDays, ["lift", "superiority", "loss"].indexOf(stopRule), simulations);
524
611
  const inputs = {
525
612
  visitorsA,
@@ -528,7 +615,7 @@ function project({
528
615
  conversionsB,
529
616
  threshold,
530
617
  minEffect,
531
- prior: [PRIOR_ALPHA, PRIOR_BETA],
618
+ prior: [priorAlpha, priorBeta],
532
619
  dailyVisitors,
533
620
  horizonDays,
534
621
  stopRule,
@@ -536,7 +623,7 @@ function project({
536
623
  simulations,
537
624
  seed: resolvedSeed
538
625
  };
539
- const link = shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect });
626
+ const link = shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] });
540
627
  const diagnostic = stopRule === "lift" && liftNearThreshold(current, minEffect) ? "lift_near_threshold" : null;
541
628
  const decisionNow = decisionNowFrom(current, stopRule, threshold, maxLoss);
542
629
  if (decisionNow) {
@@ -553,10 +640,10 @@ function project({
553
640
  shareUrl: link
554
641
  };
555
642
  }
556
- const alphaA0 = PRIOR_ALPHA + conversionsA;
557
- const betaA0 = PRIOR_BETA + (visitorsA - conversionsA);
558
- const alphaB0 = PRIOR_ALPHA + conversionsB;
559
- const betaB0 = PRIOR_BETA + (visitorsB - conversionsB);
643
+ const alphaA0 = priorAlpha + conversionsA;
644
+ const betaA0 = priorBeta + (visitorsA - conversionsA);
645
+ const alphaB0 = priorAlpha + conversionsB;
646
+ const betaB0 = priorBeta + (visitorsB - conversionsB);
560
647
  const random = makeDeterministicRandom(resolvedSeed);
561
648
  const decidedByDay = new Array(horizonDays + 1).fill(0);
562
649
  const decisionDays = [];
@@ -650,12 +737,15 @@ export {
650
737
  analyze,
651
738
  betaPdf,
652
739
  integratedExpectedLoss,
740
+ isDefaultPrior,
653
741
  liftDensity,
654
742
  logBeta,
655
743
  logGamma,
656
744
  plan,
657
745
  probabilityBBeatsA,
658
746
  project,
747
+ regularizedIncompleteBeta,
659
748
  requiredSamplePerVariant,
660
- shareUrl
749
+ shareUrl,
750
+ validatePrior
661
751
  };
package/package.json CHANGED
@@ -1,8 +1,15 @@
1
1
  {
2
2
  "name": "bayes-ab",
3
- "version": "0.2.0",
3
+ "version": "0.3.0",
4
4
  "description": "Bayesian A/B test analysis on the command line, for humans and agents",
5
- "keywords": ["ab-testing", "bayesian", "statistics", "cli", "experimentation", "conversion-rate"],
5
+ "keywords": [
6
+ "ab-testing",
7
+ "bayesian",
8
+ "statistics",
9
+ "cli",
10
+ "experimentation",
11
+ "conversion-rate"
12
+ ],
6
13
  "homepage": "https://www.joebrundage.com/tools/bayesian-ab-test",
7
14
  "repository": {
8
15
  "type": "git",
@@ -23,7 +30,11 @@
23
30
  "import": "./dist/index.js"
24
31
  }
25
32
  },
26
- "files": ["dist", "README.md", "LICENSE"],
33
+ "files": [
34
+ "dist",
35
+ "README.md",
36
+ "LICENSE"
37
+ ],
27
38
  "engines": {
28
39
  "node": ">=18.11"
29
40
  },