bayes-ab 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +17 -4
- package/dist/cli.js +133 -21
- package/dist/core/analyze.d.ts +6 -1
- package/dist/core/math.d.ts +9 -1
- package/dist/core/project.d.ts +3 -1
- package/dist/core/share.d.ts +1 -0
- package/dist/index.d.ts +2 -2
- package/dist/index.js +106 -16
- package/package.json +14 -3
package/README.md
CHANGED
|
@@ -98,6 +98,14 @@ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 10000 --rat
|
|
|
98
98
|
# Continue test · P(B beats A) 97.13% · P(lift > +10%) 56.76% · lift +10.92% (CI -0.30% to +23.39%) · loss if ship B 0.0034 pp B = 610/10000 from --rate-b 6.10%
|
|
99
99
|
```
|
|
100
100
|
|
|
101
|
+
Choose a prior. Every analysis defaults to a flat Beta(1,1) on each conversion rate; `--prior-alpha` and `--prior-beta` replace it, the same prior for both variants. Beta(α, β) reads as α conversions in α + β visitors of evidence you already had, so a prior worth 1,000 visitors centred on 5.5% is Beta(55, 945). Fractional values work, e.g. the Jeffreys prior Beta(0.5, 0.5):
|
|
102
|
+
|
|
103
|
+
```sh
|
|
104
|
+
bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --conversions-b 714 --prior-alpha 55 --prior-beta 945
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
A shared prior pulls both arms toward the same rate, so a strong one shrinks the estimated lift and makes decisions more conservative.
|
|
108
|
+
|
|
101
109
|
### project
|
|
102
110
|
|
|
103
111
|
Takes the current posterior, draws a plausible pair of true conversion rates, simulates each future day of traffic, and re-runs the decision rule after every day. Repeats that thousands of times.
|
|
@@ -173,7 +181,7 @@ bayes-ab analyze --visitors-a 12000 --conversions-a 660 --visitors-b 11850 --con
|
|
|
173
181
|
```json
|
|
174
182
|
{
|
|
175
183
|
"tool": "bayes-ab",
|
|
176
|
-
"version": "0.
|
|
184
|
+
"version": "0.3.0",
|
|
177
185
|
"command": "analyze",
|
|
178
186
|
"inputs": { "visitorsA": 12000, "conversionsA": 660, "visitorsB": 11850, "conversionsB": 714, "threshold": 0.95, "minEffect": 0.1, "prior": [1, 1], "samples": 12000, "seed": 116218 },
|
|
179
187
|
"decision": { "status": "continue", "label": "Continue test", "evidence": "low", "maturity": "building", "maturityProgress": 0.462241, "bayesActionNow": "B", "expectedRegretNow": 0.00005 },
|
|
@@ -227,8 +235,12 @@ import { analyze, project, plan } from "bayes-ab"
|
|
|
227
235
|
|
|
228
236
|
const result = analyze({ visitorsA: 12000, conversionsA: 660, visitorsB: 11850, conversionsB: 714 })
|
|
229
237
|
result.probability.bBeatsA // 0.9591138673472563
|
|
238
|
+
|
|
239
|
+
analyze({ visitorsA: 12000, conversionsA: 660, visitorsB: 11850, conversionsB: 714, prior: [55, 945] })
|
|
230
240
|
```
|
|
231
241
|
|
|
242
|
+
`analyze` and `project` take an optional `prior: [alpha, beta]`, default `[1, 1]`. Non-positive values throw a `RangeError`; `validatePrior` and `isDefaultPrior` are exported for forms that want to check first.
|
|
243
|
+
|
|
232
244
|
`analyze({ ..., includeChart: true })` adds `chart` (both posterior densities) and `liftChart` (the exact density of the relative lift, peak scaled to 1), which is what the web tool plots.
|
|
233
245
|
|
|
234
246
|
The primitives behind it are exported too, for anything that needs the same math without a full analysis:
|
|
@@ -236,7 +248,8 @@ The primitives behind it are exported too, for anything that needs the same math
|
|
|
236
248
|
| Export | What it does |
|
|
237
249
|
|---|---|
|
|
238
250
|
| `betaPdf(x, alpha, beta)`, `logBeta`, `logGamma` | Beta density and its log-gamma building blocks |
|
|
239
|
-
| `probabilityBBeatsA(alphaA, betaA, alphaB, betaB)` |
|
|
251
|
+
| `probabilityBBeatsA(alphaA, betaA, alphaB, betaB)` | P(B > A) for two Beta posteriors: exact closed form for whole-number alphas, numerical integration otherwise |
|
|
252
|
+
| `regularizedIncompleteBeta(x, a, b)` | The Beta(a, b) CDF at x |
|
|
240
253
|
| `integratedExpectedLoss(alphaA, betaA, alphaB, betaB)` | Expected loss of shipping each variant by numerical integration, no sampling |
|
|
241
254
|
| `liftDensity(alphaA, betaA, alphaB, betaB, sortedLifts)` | Exact relative-lift density over the range the sorted lifts span |
|
|
242
255
|
|
|
@@ -244,8 +257,8 @@ This package is the source of truth for the math on joebrundage.com: the [calcul
|
|
|
244
257
|
|
|
245
258
|
## Method
|
|
246
259
|
|
|
247
|
-
- Each conversion rate gets a Beta(1,1)
|
|
248
|
-
- P(B beats A) is the exact closed form from [Evan Miller](https://www.evanmiller.org/bayesian-ab-testing.html). P(B is harmful) is its exact complement.
|
|
260
|
+
- Each conversion rate gets a Beta(α, β) prior, Beta(1,1) unless you choose another. Posteriors are Beta(α + conversions, β + non-conversions).
|
|
261
|
+
- P(B beats A) is the exact closed form from [Evan Miller](https://www.evanmiller.org/bayesian-ab-testing.html) when α is a whole number. For a fractional α it is integrated numerically (tanh-sinh quadrature over the incomplete beta function), accurate to about 1e-10. P(B is harmful) is its exact complement.
|
|
249
262
|
- Lift intervals, threshold probabilities and expected loss come from 12,000 seeded Monte Carlo draws.
|
|
250
263
|
- The meaningful effect is relative and symmetric: `--min-effect 10` means B must be 10% better than A to count as a win or 10% worse to count as a loss.
|
|
251
264
|
- `project` samples true rates from the exact posterior, then uses a normal approximation to the posterior when re-checking the stopping rule on each simulated day. Day zero is checked exactly.
|
package/dist/cli.js
CHANGED
|
@@ -119,11 +119,80 @@ function probabilityRightBeatsLeftDirect(alphaLeft, betaLeft, alphaRight, betaRi
|
|
|
119
119
|
return Math.exp(maxLog) * scaledSum;
|
|
120
120
|
}
|
|
121
121
|
function probabilityBBeatsA(alphaA, betaA, alphaB, betaB) {
|
|
122
|
+
if (!Number.isInteger(alphaA) || !Number.isInteger(alphaB)) {
|
|
123
|
+
return probabilityBBeatsANumeric(alphaA, betaA, alphaB, betaB);
|
|
124
|
+
}
|
|
122
125
|
if (alphaB <= alphaA) {
|
|
123
126
|
return probabilityRightBeatsLeftDirect(alphaA, betaA, alphaB, betaB);
|
|
124
127
|
}
|
|
125
128
|
return 1 - probabilityRightBeatsLeftDirect(alphaB, betaB, alphaA, betaA);
|
|
126
129
|
}
|
|
130
|
+
function betaContinuedFraction(x, a, b) {
|
|
131
|
+
const TINY = 1e-300;
|
|
132
|
+
const EPSILON = 1e-15;
|
|
133
|
+
const MAX_ITERATIONS = 1e5;
|
|
134
|
+
const qab = a + b;
|
|
135
|
+
const qap = a + 1;
|
|
136
|
+
const qam = a - 1;
|
|
137
|
+
let c = 1;
|
|
138
|
+
let d = 1 - qab * x / qap;
|
|
139
|
+
if (Math.abs(d) < TINY) d = TINY;
|
|
140
|
+
d = 1 / d;
|
|
141
|
+
let h = d;
|
|
142
|
+
for (let m = 1; m <= MAX_ITERATIONS; m += 1) {
|
|
143
|
+
const m2 = 2 * m;
|
|
144
|
+
let aa = m * (b - m) * x / ((qam + m2) * (a + m2));
|
|
145
|
+
d = 1 + aa * d;
|
|
146
|
+
if (Math.abs(d) < TINY) d = TINY;
|
|
147
|
+
c = 1 + aa / c;
|
|
148
|
+
if (Math.abs(c) < TINY) c = TINY;
|
|
149
|
+
d = 1 / d;
|
|
150
|
+
h *= d * c;
|
|
151
|
+
aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2));
|
|
152
|
+
d = 1 + aa * d;
|
|
153
|
+
if (Math.abs(d) < TINY) d = TINY;
|
|
154
|
+
c = 1 + aa / c;
|
|
155
|
+
if (Math.abs(c) < TINY) c = TINY;
|
|
156
|
+
d = 1 / d;
|
|
157
|
+
const delta = d * c;
|
|
158
|
+
h *= delta;
|
|
159
|
+
if (Math.abs(delta - 1) < EPSILON) break;
|
|
160
|
+
}
|
|
161
|
+
return h;
|
|
162
|
+
}
|
|
163
|
+
function regularizedIncompleteBeta(x, a, b) {
|
|
164
|
+
if (x <= 0) return 0;
|
|
165
|
+
if (x >= 1) return 1;
|
|
166
|
+
if (x > (a + 1) / (a + b + 2)) return 1 - regularizedIncompleteBeta(1 - x, b, a);
|
|
167
|
+
const logFront = a * Math.log(x) + b * Math.log1p(-x) - logBeta(a, b);
|
|
168
|
+
return Math.exp(logFront) * betaContinuedFraction(x, a, b) / a;
|
|
169
|
+
}
|
|
170
|
+
function probabilityBBeatsANumeric(alphaA, betaA, alphaB, betaB) {
|
|
171
|
+
const total = alphaA + betaA;
|
|
172
|
+
const meanA = alphaA / total;
|
|
173
|
+
const sdA = Math.sqrt(alphaA * betaA / (total * total * (total + 1)));
|
|
174
|
+
const low = Math.max(0, meanA - 40 * sdA);
|
|
175
|
+
const high = Math.min(1, meanA + 40 * sdA);
|
|
176
|
+
const center = (low + high) / 2;
|
|
177
|
+
const halfWidth = (high - low) / 2;
|
|
178
|
+
const logNormalizer = logBeta(alphaA, betaA);
|
|
179
|
+
const STEP = 1 / 128;
|
|
180
|
+
const T_MAX = 3.5;
|
|
181
|
+
let mass = 0;
|
|
182
|
+
let weighted = 0;
|
|
183
|
+
for (let t = -T_MAX; t <= T_MAX + 1e-12; t += STEP) {
|
|
184
|
+
const inner = Math.PI / 2 * Math.sinh(t);
|
|
185
|
+
const x = center + halfWidth * Math.tanh(inner);
|
|
186
|
+
if (x <= 0 || x >= 1) continue;
|
|
187
|
+
const coshInner = Math.cosh(inner);
|
|
188
|
+
const weight = Math.PI / 2 * Math.cosh(t) / (coshInner * coshInner);
|
|
189
|
+
const density = Math.exp((alphaA - 1) * Math.log(x) + (betaA - 1) * Math.log1p(-x) - logNormalizer);
|
|
190
|
+
if (!Number.isFinite(density)) continue;
|
|
191
|
+
mass += weight * density;
|
|
192
|
+
weighted += weight * density * (1 - regularizedIncompleteBeta(x, alphaB, betaB));
|
|
193
|
+
}
|
|
194
|
+
return mass > 0 ? weighted / mass : 0.5;
|
|
195
|
+
}
|
|
127
196
|
function inverseNormalCdf(p) {
|
|
128
197
|
const a = [-39.69683028665376, 220.9460984245205, -275.9285104469687, 138.357751867269, -30.66479806614716, 2.506628277459239];
|
|
129
198
|
const b = [-54.47609879822406, 161.5858368580409, -155.6989798598866, 66.80131188771972, -13.28068155288572];
|
|
@@ -248,6 +317,10 @@ function shareUrl(inputs) {
|
|
|
248
317
|
thresholdPercent: formatPercentParam(inputs.threshold),
|
|
249
318
|
meaningfulLiftPercent: formatPercentParam(inputs.minEffect)
|
|
250
319
|
});
|
|
320
|
+
if (inputs.prior && !(inputs.prior[0] === 1 && inputs.prior[1] === 1)) {
|
|
321
|
+
params.set("priorAlpha", String(inputs.prior[0]));
|
|
322
|
+
params.set("priorBeta", String(inputs.prior[1]));
|
|
323
|
+
}
|
|
251
324
|
return `${WEB_TOOL_URL}?${params.toString()}`;
|
|
252
325
|
}
|
|
253
326
|
|
|
@@ -266,6 +339,13 @@ var DECISION_LABELS = {
|
|
|
266
339
|
continue: "Continue test",
|
|
267
340
|
inconclusive: "Inconclusive"
|
|
268
341
|
};
|
|
342
|
+
function validatePrior(prior) {
|
|
343
|
+
const [alpha, beta] = prior;
|
|
344
|
+
if (!(Number.isFinite(alpha) && alpha > 0 && Number.isFinite(beta) && beta > 0)) {
|
|
345
|
+
throw new RangeError(`Prior parameters must be finite and greater than 0, got Beta(${alpha}, ${beta}).`);
|
|
346
|
+
}
|
|
347
|
+
return [alpha, beta];
|
|
348
|
+
}
|
|
269
349
|
function deriveSeed(...parts) {
|
|
270
350
|
let hash = 2166136261;
|
|
271
351
|
for (const part of parts) {
|
|
@@ -348,12 +428,14 @@ function analyze({
|
|
|
348
428
|
minEffect = DEFAULT_MIN_EFFECT,
|
|
349
429
|
samples = DEFAULT_SAMPLES,
|
|
350
430
|
seed = null,
|
|
351
|
-
includeChart = false
|
|
431
|
+
includeChart = false,
|
|
432
|
+
prior = [PRIOR_ALPHA, PRIOR_BETA]
|
|
352
433
|
}) {
|
|
353
|
-
const
|
|
354
|
-
const
|
|
355
|
-
const
|
|
356
|
-
const
|
|
434
|
+
const [priorAlpha, priorBeta] = validatePrior(prior);
|
|
435
|
+
const alphaA = priorAlpha + conversionsA;
|
|
436
|
+
const betaA = priorBeta + (visitorsA - conversionsA);
|
|
437
|
+
const alphaB = priorAlpha + conversionsB;
|
|
438
|
+
const betaB = priorBeta + (visitorsB - conversionsB);
|
|
357
439
|
const rateA = conversionsA / visitorsA;
|
|
358
440
|
const rateB = conversionsB / visitorsB;
|
|
359
441
|
const posteriorMeanA = alphaA / (alphaA + betaA);
|
|
@@ -388,7 +470,7 @@ function analyze({
|
|
|
388
470
|
conversionsB,
|
|
389
471
|
threshold,
|
|
390
472
|
minEffect,
|
|
391
|
-
prior: [
|
|
473
|
+
prior: [priorAlpha, priorBeta],
|
|
392
474
|
samples,
|
|
393
475
|
seed: resolvedSeed
|
|
394
476
|
},
|
|
@@ -436,7 +518,7 @@ function analyze({
|
|
|
436
518
|
},
|
|
437
519
|
waiting: waitingScenarios(alphaA, betaA, alphaB, betaB, expectedRegretNow),
|
|
438
520
|
frequentist: { pValueTwoSided: twoSidedPValue(visitorsA, conversionsA, visitorsB, conversionsB) },
|
|
439
|
-
shareUrl: shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect })
|
|
521
|
+
shareUrl: shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] })
|
|
440
522
|
};
|
|
441
523
|
if (includeChart) {
|
|
442
524
|
const sortedA = [...samplesA].sort((left, right) => left - right);
|
|
@@ -524,9 +606,11 @@ function project({
|
|
|
524
606
|
stopRule = "lift",
|
|
525
607
|
maxLoss = DEFAULT_MAX_LOSS,
|
|
526
608
|
simulations = DEFAULT_SIMULATIONS,
|
|
527
|
-
seed = null
|
|
609
|
+
seed = null,
|
|
610
|
+
prior = [PRIOR_ALPHA, PRIOR_BETA]
|
|
528
611
|
}) {
|
|
529
|
-
const
|
|
612
|
+
const [priorAlpha, priorBeta] = validatePrior(prior);
|
|
613
|
+
const current = analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] });
|
|
530
614
|
const resolvedSeed = seed ?? deriveSeed(visitorsA, conversionsA, visitorsB, conversionsB, dailyVisitors, horizonDays, ["lift", "superiority", "loss"].indexOf(stopRule), simulations);
|
|
531
615
|
const inputs = {
|
|
532
616
|
visitorsA,
|
|
@@ -535,7 +619,7 @@ function project({
|
|
|
535
619
|
conversionsB,
|
|
536
620
|
threshold,
|
|
537
621
|
minEffect,
|
|
538
|
-
prior: [
|
|
622
|
+
prior: [priorAlpha, priorBeta],
|
|
539
623
|
dailyVisitors,
|
|
540
624
|
horizonDays,
|
|
541
625
|
stopRule,
|
|
@@ -543,7 +627,7 @@ function project({
|
|
|
543
627
|
simulations,
|
|
544
628
|
seed: resolvedSeed
|
|
545
629
|
};
|
|
546
|
-
const link = shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect });
|
|
630
|
+
const link = shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] });
|
|
547
631
|
const diagnostic = stopRule === "lift" && liftNearThreshold(current, minEffect) ? "lift_near_threshold" : null;
|
|
548
632
|
const decisionNow = decisionNowFrom(current, stopRule, threshold, maxLoss);
|
|
549
633
|
if (decisionNow) {
|
|
@@ -560,10 +644,10 @@ function project({
|
|
|
560
644
|
shareUrl: link
|
|
561
645
|
};
|
|
562
646
|
}
|
|
563
|
-
const alphaA0 =
|
|
564
|
-
const betaA0 =
|
|
565
|
-
const alphaB0 =
|
|
566
|
-
const betaB0 =
|
|
647
|
+
const alphaA0 = priorAlpha + conversionsA;
|
|
648
|
+
const betaA0 = priorBeta + (visitorsA - conversionsA);
|
|
649
|
+
const alphaB0 = priorAlpha + conversionsB;
|
|
650
|
+
const betaB0 = priorBeta + (visitorsB - conversionsB);
|
|
567
651
|
const random = makeDeterministicRandom(resolvedSeed);
|
|
568
652
|
const decidedByDay = new Array(horizonDays + 1).fill(0);
|
|
569
653
|
const decisionDays = [];
|
|
@@ -643,6 +727,8 @@ var OPTIONS = {
|
|
|
643
727
|
"rate-b": { type: "string" },
|
|
644
728
|
threshold: { type: "string" },
|
|
645
729
|
"min-effect": { type: "string" },
|
|
730
|
+
"prior-alpha": { type: "string" },
|
|
731
|
+
"prior-beta": { type: "string" },
|
|
646
732
|
"daily-visitors": { type: "string" },
|
|
647
733
|
horizon: { type: "string" },
|
|
648
734
|
"stop-rule": { type: "string" },
|
|
@@ -733,10 +819,19 @@ function parseMinEffect(values) {
|
|
|
733
819
|
if (minEffect < 0 || minEffect >= 1) throw new CliError("--min-effect must be a percent between 0 and 100.", { field: "min-effect" });
|
|
734
820
|
return minEffect;
|
|
735
821
|
}
|
|
822
|
+
function parsePriorParameter(flag, value, fallback) {
|
|
823
|
+
if (value === void 0) return fallback;
|
|
824
|
+
const parsed = parseNumber(flag, value);
|
|
825
|
+
if (parsed <= 0) throw new CliError(`--${flag} must be greater than 0, got "${value}".`, { field: flag });
|
|
826
|
+
return parsed;
|
|
827
|
+
}
|
|
828
|
+
function parsePrior(values) {
|
|
829
|
+
return [parsePriorParameter("prior-alpha", values["prior-alpha"], PRIOR_ALPHA), parsePriorParameter("prior-beta", values["prior-beta"], PRIOR_BETA)];
|
|
830
|
+
}
|
|
736
831
|
function parseAnalyzeArgs(values) {
|
|
737
832
|
const samples = values.samples === void 0 ? DEFAULT_SAMPLES : parseCount("samples", values.samples, { min: 100 });
|
|
738
833
|
const seed = values.seed === void 0 ? null : parseCount("seed", values.seed);
|
|
739
|
-
return { ...parseVariants(values), threshold: parseThreshold(values), minEffect: parseMinEffect(values), samples, seed };
|
|
834
|
+
return { ...parseVariants(values), threshold: parseThreshold(values), minEffect: parseMinEffect(values), prior: parsePrior(values), samples, seed };
|
|
740
835
|
}
|
|
741
836
|
function parseStopRule(value) {
|
|
742
837
|
if (value === void 0) return "lift";
|
|
@@ -812,7 +907,8 @@ function helpText(version) {
|
|
|
812
907
|
return `bayes-ab ${version} Bayesian A/B test analysis for humans and agents
|
|
813
908
|
|
|
814
909
|
Compares two variants of a binary outcome (signup, click, purchase) using
|
|
815
|
-
Beta
|
|
910
|
+
Beta posteriors on each conversion rate, from a flat Beta(1,1) prior unless
|
|
911
|
+
you choose another. Reports the probability that B
|
|
816
912
|
beats A, the probability the lift clears a meaningful threshold, expected
|
|
817
913
|
loss from shipping either variant, and a forward projection of how long the
|
|
818
914
|
test needs to run. Same math as ${WEB_TOOL_URL}
|
|
@@ -839,6 +935,12 @@ INPUTS (analyze, project)
|
|
|
839
935
|
--min-effect PCT Meaningful relative effect, symmetric: with 10, B must be
|
|
840
936
|
10% better than A to count as a win or 10% worse to count
|
|
841
937
|
as a loss. Relative, not percentage points. default 10
|
|
938
|
+
--prior-alpha N, --prior-beta N Beta(alpha, beta) prior on each variant's conversion rate,
|
|
939
|
+
the same for both. Any positive number. Reads as alpha
|
|
940
|
+
conversions in alpha + beta visitors of prior evidence,
|
|
941
|
+
with prior mean alpha / (alpha + beta).
|
|
942
|
+
Beta(1,1) is flat; Beta(0.5,0.5) is Jeffreys; Beta(55,945)
|
|
943
|
+
says "about 5.5%, worth 1,000 visitors". default 1, 1
|
|
842
944
|
Whole numbers may include commas or underscores: 12,000 and 12_000 both work.
|
|
843
945
|
|
|
844
946
|
PROJECTION (project)
|
|
@@ -934,6 +1036,10 @@ function unsignedPoints(fraction2, digits = 4) {
|
|
|
934
1036
|
function count(value) {
|
|
935
1037
|
return Math.round(value).toLocaleString("en-US");
|
|
936
1038
|
}
|
|
1039
|
+
function betaParameter(value) {
|
|
1040
|
+
if (Number.isInteger(value)) return count(value);
|
|
1041
|
+
return value.toLocaleString("en-US", { maximumFractionDigits: 3 });
|
|
1042
|
+
}
|
|
937
1043
|
function missedConversions(regretRate, visitors = 1e4) {
|
|
938
1044
|
const missed = regretRate * visitors;
|
|
939
1045
|
const digits = missed < 1 ? 2 : missed < 10 ? 1 : 0;
|
|
@@ -987,6 +1093,8 @@ function renderAnalyzeBrief(result, derivedFrom2) {
|
|
|
987
1093
|
`lift ${signedPercent(result.effect.relativeLift)} (CI ${signedPercent(lower)} to ${signedPercent(upper)})`,
|
|
988
1094
|
`loss if ship B ${unsignedPoints(result.expectedLoss.shipB)}`
|
|
989
1095
|
];
|
|
1096
|
+
const [priorAlpha, priorBeta] = result.inputs.prior;
|
|
1097
|
+
if (priorAlpha !== 1 || priorBeta !== 1) parts.push(`Beta(${betaParameter(priorAlpha)},${betaParameter(priorBeta)}) prior`);
|
|
990
1098
|
const note = derivedNote(result, derivedFrom2);
|
|
991
1099
|
return parts.join(" \xB7 ") + (note ? ` ${note}` : "");
|
|
992
1100
|
}
|
|
@@ -998,7 +1106,7 @@ function renderAnalyze(result, derivedFrom2) {
|
|
|
998
1106
|
const lines = [];
|
|
999
1107
|
const note = derivedNote(result, derivedFrom2);
|
|
1000
1108
|
lines.push(
|
|
1001
|
-
`Bayesian A/B test Beta(${inputs.prior[0]},${inputs.prior[1]}) prior \xB7 decision threshold ${percent(inputs.threshold, 0)} \xB7 meaningful effect \xB1${minEffect} relative`
|
|
1109
|
+
`Bayesian A/B test Beta(${betaParameter(inputs.prior[0])},${betaParameter(inputs.prior[1])}) prior \xB7 decision threshold ${percent(inputs.threshold, 0)} \xB7 meaningful effect \xB1${minEffect} relative`
|
|
1002
1110
|
);
|
|
1003
1111
|
if (note) lines.push(` ${note}`);
|
|
1004
1112
|
lines.push("");
|
|
@@ -1024,7 +1132,7 @@ function renderAnalyze(result, derivedFrom2) {
|
|
|
1024
1132
|
["Conversions", count(variants.A.conversions), count(variants.B.conversions)],
|
|
1025
1133
|
["Observed rate", percent(variants.A.rate), percent(variants.B.rate)],
|
|
1026
1134
|
["Posterior mean", percent(variants.A.posteriorMean, 3), percent(variants.B.posteriorMean, 3)],
|
|
1027
|
-
["Posterior", `Beta(${
|
|
1135
|
+
["Posterior", `Beta(${betaParameter(variants.A.posterior[0])}, ${betaParameter(variants.A.posterior[1])})`, `Beta(${betaParameter(variants.B.posterior[0])}, ${betaParameter(variants.B.posterior[1])})`]
|
|
1028
1136
|
])
|
|
1029
1137
|
);
|
|
1030
1138
|
lines.push("");
|
|
@@ -1162,6 +1270,8 @@ function renderProjectBrief(result) {
|
|
|
1162
1270
|
result.daysToDecision ? `median day ${result.daysToDecision.p50} if it decides (p90 day ${result.daysToDecision.p90})` : "no simulated future decided",
|
|
1163
1271
|
`${count(dailyVisitors)}/day per variant`
|
|
1164
1272
|
];
|
|
1273
|
+
const [priorAlpha, priorBeta] = result.inputs.prior;
|
|
1274
|
+
if (priorAlpha !== 1 || priorBeta !== 1) parts.push(`Beta(${betaParameter(priorAlpha)},${betaParameter(priorBeta)}) prior`);
|
|
1165
1275
|
return parts.join(" \xB7 ");
|
|
1166
1276
|
}
|
|
1167
1277
|
function renderProject(result) {
|
|
@@ -1171,6 +1281,8 @@ function renderProject(result) {
|
|
|
1171
1281
|
`Forward projection ${count(inputs.dailyVisitors)} visitors/day per variant \xB7 ${inputs.horizonDays}-day horizon \xB7 ${count(inputs.simulations)} simulated futures`
|
|
1172
1282
|
);
|
|
1173
1283
|
lines.push(`Stopping rule ${describeRule(result)}`);
|
|
1284
|
+
const [priorAlpha, priorBeta] = inputs.prior;
|
|
1285
|
+
if (priorAlpha !== 1 || priorBeta !== 1) lines.push(`Prior Beta(${betaParameter(priorAlpha)},${betaParameter(priorBeta)}) on each variant`);
|
|
1174
1286
|
lines.push("");
|
|
1175
1287
|
if (result.alreadyDecided && result.decisionNow) {
|
|
1176
1288
|
lines.push(`Already decided ${decisionLabel(result.decisionNow)}`);
|
|
@@ -1236,7 +1348,7 @@ var variantInputs = {
|
|
|
1236
1348
|
conversionsB: integer,
|
|
1237
1349
|
threshold: fraction,
|
|
1238
1350
|
minEffect: fraction,
|
|
1239
|
-
prior: pair(number)
|
|
1351
|
+
prior: { ...pair({ type: "number", exclusiveMinimum: 0 }), description: "The Beta(alpha, beta) prior applied to each variant. [1, 1] unless --prior-alpha or --prior-beta was given." }
|
|
1240
1352
|
};
|
|
1241
1353
|
var derivedFrom = {
|
|
1242
1354
|
...object({ rateA: fraction, rateB: fraction }, []),
|
|
@@ -1400,7 +1512,7 @@ var SCHEMAS = {
|
|
|
1400
1512
|
};
|
|
1401
1513
|
|
|
1402
1514
|
// src/cli/main.ts
|
|
1403
|
-
var VERSION = true ? "0.
|
|
1515
|
+
var VERSION = true ? "0.3.0" : "dev";
|
|
1404
1516
|
var JSON_DECIMALS = 6;
|
|
1405
1517
|
function roundDeep(value) {
|
|
1406
1518
|
if (typeof value === "number") return round(value, JSON_DECIMALS);
|
package/dist/core/analyze.d.ts
CHANGED
|
@@ -19,6 +19,8 @@ export type AnalyzeInputs = {
|
|
|
19
19
|
samples?: number;
|
|
20
20
|
seed?: number | null;
|
|
21
21
|
includeChart?: boolean;
|
|
22
|
+
/** Beta(alpha, beta) prior applied to both variants. Defaults to the flat Beta(1, 1). */
|
|
23
|
+
prior?: [number, number];
|
|
22
24
|
};
|
|
23
25
|
export type VariantSummary = {
|
|
24
26
|
visitors: number;
|
|
@@ -100,6 +102,9 @@ export type AnalyzeResult = {
|
|
|
100
102
|
liftChart?: LiftDensityPoint[];
|
|
101
103
|
shareUrl: string;
|
|
102
104
|
};
|
|
105
|
+
/** Throws a RangeError unless both prior parameters are finite and positive. */
|
|
106
|
+
export declare function validatePrior(prior: [number, number]): [number, number];
|
|
107
|
+
export declare function isDefaultPrior([alpha, beta]: [number, number]): boolean;
|
|
103
108
|
export declare function deriveSeed(...parts: number[]): number;
|
|
104
109
|
export declare function decisionStatus({ probabilityMeaningfulLift, probabilityMeaningfulHarm, threshold, evidence, }: {
|
|
105
110
|
probabilityMeaningfulLift: number;
|
|
@@ -107,4 +112,4 @@ export declare function decisionStatus({ probabilityMeaningfulLift, probabilityM
|
|
|
107
112
|
threshold: number;
|
|
108
113
|
evidence: EvidenceStrength;
|
|
109
114
|
}): DecisionStatus;
|
|
110
|
-
export declare function analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, samples, seed, includeChart, }: AnalyzeInputs): AnalyzeResult;
|
|
115
|
+
export declare function analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, samples, seed, includeChart, prior, }: AnalyzeInputs): AnalyzeResult;
|
package/dist/core/math.d.ts
CHANGED
|
@@ -10,8 +10,16 @@ export declare function sampleBeta(alpha: number, beta: number, random: Random):
|
|
|
10
10
|
export declare function sampleBinomialApprox(trials: number, probability: number, random: Random): number;
|
|
11
11
|
export declare function quantile(sortedValues: number[], probability: number): number;
|
|
12
12
|
export declare function mean(values: number[]): number;
|
|
13
|
-
/**
|
|
13
|
+
/**
|
|
14
|
+
* P(B > A) for independent Beta posteriors. Exact via Evan Miller's closed form
|
|
15
|
+
* when the alphas are whole numbers, which they are under any whole-number
|
|
16
|
+
* prior. The closed form sums over 0..alpha-1, so a fractional alpha (a
|
|
17
|
+
* Jeffreys Beta(0.5, 0.5) prior, say) falls back to numerical integration,
|
|
18
|
+
* accurate to roughly 1e-10.
|
|
19
|
+
*/
|
|
14
20
|
export declare function probabilityBBeatsA(alphaA: number, betaA: number, alphaB: number, betaB: number): number;
|
|
21
|
+
/** Regularized incomplete beta function I_x(a, b): the Beta(a, b) CDF at x. */
|
|
22
|
+
export declare function regularizedIncompleteBeta(x: number, a: number, b: number): number;
|
|
15
23
|
export declare function inverseNormalCdf(p: number): number;
|
|
16
24
|
export declare function erf(x: number): number;
|
|
17
25
|
export declare function normalCdf(x: number): number;
|
package/dist/core/project.d.ts
CHANGED
|
@@ -16,6 +16,8 @@ export type ProjectInputs = {
|
|
|
16
16
|
maxLoss?: number;
|
|
17
17
|
simulations?: number;
|
|
18
18
|
seed?: number | null;
|
|
19
|
+
/** Beta(alpha, beta) prior applied to both variants. Defaults to the flat Beta(1, 1). */
|
|
20
|
+
prior?: [number, number];
|
|
19
21
|
};
|
|
20
22
|
export type ProjectResult = {
|
|
21
23
|
inputs: {
|
|
@@ -55,4 +57,4 @@ export type ProjectResult = {
|
|
|
55
57
|
approximation: "normal";
|
|
56
58
|
shareUrl: string;
|
|
57
59
|
};
|
|
58
|
-
export declare function project({ visitorsA, conversionsA, visitorsB, conversionsB, dailyVisitors, horizonDays, threshold, minEffect, stopRule, maxLoss, simulations, seed, }: ProjectInputs): ProjectResult;
|
|
60
|
+
export declare function project({ visitorsA, conversionsA, visitorsB, conversionsB, dailyVisitors, horizonDays, threshold, minEffect, stopRule, maxLoss, simulations, seed, prior, }: ProjectInputs): ProjectResult;
|
package/dist/core/share.d.ts
CHANGED
package/dist/index.d.ts
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
|
-
export { analyze, DECISION_LABELS, DEFAULT_MIN_EFFECT, DEFAULT_SAMPLES, DEFAULT_THRESHOLD, PRIOR_ALPHA, PRIOR_BETA } from "./core/analyze.ts";
|
|
1
|
+
export { analyze, DECISION_LABELS, DEFAULT_MIN_EFFECT, DEFAULT_SAMPLES, DEFAULT_THRESHOLD, PRIOR_ALPHA, PRIOR_BETA, isDefaultPrior, validatePrior } from "./core/analyze.ts";
|
|
2
2
|
export type { AnalyzeInputs, AnalyzeResult, ChartPoint, DecisionStatus, EvidenceStrength, Maturity, Variant, VariantSummary, WaitingScenario, } from "./core/analyze.ts";
|
|
3
3
|
export { project, DEFAULT_HORIZON_DAYS, DEFAULT_MAX_LOSS, DEFAULT_SIMULATIONS } from "./core/project.ts";
|
|
4
4
|
export type { ProjectInputs, ProjectResult, ProjectedDecision, StopRule } from "./core/project.ts";
|
|
5
5
|
export { plan, requiredSamplePerVariant } from "./core/plan.ts";
|
|
6
6
|
export type { PlanInputs, PlanResult, PlanRow } from "./core/plan.ts";
|
|
7
7
|
export { shareUrl, WEB_TOOL_URL } from "./core/share.ts";
|
|
8
|
-
export { betaPdf, logBeta, logGamma, probabilityBBeatsA } from "./core/math.ts";
|
|
8
|
+
export { betaPdf, logBeta, logGamma, probabilityBBeatsA, regularizedIncompleteBeta } from "./core/math.ts";
|
|
9
9
|
export { liftDensity } from "./core/lift.ts";
|
|
10
10
|
export type { LiftDensityPoint } from "./core/lift.ts";
|
|
11
11
|
export { integratedExpectedLoss } from "./core/loss.ts";
|
package/dist/index.js
CHANGED
|
@@ -117,11 +117,80 @@ function probabilityRightBeatsLeftDirect(alphaLeft, betaLeft, alphaRight, betaRi
|
|
|
117
117
|
return Math.exp(maxLog) * scaledSum;
|
|
118
118
|
}
|
|
119
119
|
function probabilityBBeatsA(alphaA, betaA, alphaB, betaB) {
|
|
120
|
+
if (!Number.isInteger(alphaA) || !Number.isInteger(alphaB)) {
|
|
121
|
+
return probabilityBBeatsANumeric(alphaA, betaA, alphaB, betaB);
|
|
122
|
+
}
|
|
120
123
|
if (alphaB <= alphaA) {
|
|
121
124
|
return probabilityRightBeatsLeftDirect(alphaA, betaA, alphaB, betaB);
|
|
122
125
|
}
|
|
123
126
|
return 1 - probabilityRightBeatsLeftDirect(alphaB, betaB, alphaA, betaA);
|
|
124
127
|
}
|
|
128
|
+
function betaContinuedFraction(x, a, b) {
|
|
129
|
+
const TINY = 1e-300;
|
|
130
|
+
const EPSILON = 1e-15;
|
|
131
|
+
const MAX_ITERATIONS = 1e5;
|
|
132
|
+
const qab = a + b;
|
|
133
|
+
const qap = a + 1;
|
|
134
|
+
const qam = a - 1;
|
|
135
|
+
let c = 1;
|
|
136
|
+
let d = 1 - qab * x / qap;
|
|
137
|
+
if (Math.abs(d) < TINY) d = TINY;
|
|
138
|
+
d = 1 / d;
|
|
139
|
+
let h = d;
|
|
140
|
+
for (let m = 1; m <= MAX_ITERATIONS; m += 1) {
|
|
141
|
+
const m2 = 2 * m;
|
|
142
|
+
let aa = m * (b - m) * x / ((qam + m2) * (a + m2));
|
|
143
|
+
d = 1 + aa * d;
|
|
144
|
+
if (Math.abs(d) < TINY) d = TINY;
|
|
145
|
+
c = 1 + aa / c;
|
|
146
|
+
if (Math.abs(c) < TINY) c = TINY;
|
|
147
|
+
d = 1 / d;
|
|
148
|
+
h *= d * c;
|
|
149
|
+
aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2));
|
|
150
|
+
d = 1 + aa * d;
|
|
151
|
+
if (Math.abs(d) < TINY) d = TINY;
|
|
152
|
+
c = 1 + aa / c;
|
|
153
|
+
if (Math.abs(c) < TINY) c = TINY;
|
|
154
|
+
d = 1 / d;
|
|
155
|
+
const delta = d * c;
|
|
156
|
+
h *= delta;
|
|
157
|
+
if (Math.abs(delta - 1) < EPSILON) break;
|
|
158
|
+
}
|
|
159
|
+
return h;
|
|
160
|
+
}
|
|
161
|
+
function regularizedIncompleteBeta(x, a, b) {
|
|
162
|
+
if (x <= 0) return 0;
|
|
163
|
+
if (x >= 1) return 1;
|
|
164
|
+
if (x > (a + 1) / (a + b + 2)) return 1 - regularizedIncompleteBeta(1 - x, b, a);
|
|
165
|
+
const logFront = a * Math.log(x) + b * Math.log1p(-x) - logBeta(a, b);
|
|
166
|
+
return Math.exp(logFront) * betaContinuedFraction(x, a, b) / a;
|
|
167
|
+
}
|
|
168
|
+
function probabilityBBeatsANumeric(alphaA, betaA, alphaB, betaB) {
|
|
169
|
+
const total = alphaA + betaA;
|
|
170
|
+
const meanA = alphaA / total;
|
|
171
|
+
const sdA = Math.sqrt(alphaA * betaA / (total * total * (total + 1)));
|
|
172
|
+
const low = Math.max(0, meanA - 40 * sdA);
|
|
173
|
+
const high = Math.min(1, meanA + 40 * sdA);
|
|
174
|
+
const center = (low + high) / 2;
|
|
175
|
+
const halfWidth = (high - low) / 2;
|
|
176
|
+
const logNormalizer = logBeta(alphaA, betaA);
|
|
177
|
+
const STEP = 1 / 128;
|
|
178
|
+
const T_MAX = 3.5;
|
|
179
|
+
let mass = 0;
|
|
180
|
+
let weighted = 0;
|
|
181
|
+
for (let t = -T_MAX; t <= T_MAX + 1e-12; t += STEP) {
|
|
182
|
+
const inner = Math.PI / 2 * Math.sinh(t);
|
|
183
|
+
const x = center + halfWidth * Math.tanh(inner);
|
|
184
|
+
if (x <= 0 || x >= 1) continue;
|
|
185
|
+
const coshInner = Math.cosh(inner);
|
|
186
|
+
const weight = Math.PI / 2 * Math.cosh(t) / (coshInner * coshInner);
|
|
187
|
+
const density = Math.exp((alphaA - 1) * Math.log(x) + (betaA - 1) * Math.log1p(-x) - logNormalizer);
|
|
188
|
+
if (!Number.isFinite(density)) continue;
|
|
189
|
+
mass += weight * density;
|
|
190
|
+
weighted += weight * density * (1 - regularizedIncompleteBeta(x, alphaB, betaB));
|
|
191
|
+
}
|
|
192
|
+
return mass > 0 ? weighted / mass : 0.5;
|
|
193
|
+
}
|
|
125
194
|
function inverseNormalCdf(p) {
|
|
126
195
|
const a = [-39.69683028665376, 220.9460984245205, -275.9285104469687, 138.357751867269, -30.66479806614716, 2.506628277459239];
|
|
127
196
|
const b = [-54.47609879822406, 161.5858368580409, -155.6989798598866, 66.80131188771972, -13.28068155288572];
|
|
@@ -241,6 +310,10 @@ function shareUrl(inputs) {
|
|
|
241
310
|
thresholdPercent: formatPercentParam(inputs.threshold),
|
|
242
311
|
meaningfulLiftPercent: formatPercentParam(inputs.minEffect)
|
|
243
312
|
});
|
|
313
|
+
if (inputs.prior && !(inputs.prior[0] === 1 && inputs.prior[1] === 1)) {
|
|
314
|
+
params.set("priorAlpha", String(inputs.prior[0]));
|
|
315
|
+
params.set("priorBeta", String(inputs.prior[1]));
|
|
316
|
+
}
|
|
244
317
|
return `${WEB_TOOL_URL}?${params.toString()}`;
|
|
245
318
|
}
|
|
246
319
|
|
|
@@ -259,6 +332,16 @@ var DECISION_LABELS = {
|
|
|
259
332
|
continue: "Continue test",
|
|
260
333
|
inconclusive: "Inconclusive"
|
|
261
334
|
};
|
|
335
|
+
function validatePrior(prior) {
|
|
336
|
+
const [alpha, beta] = prior;
|
|
337
|
+
if (!(Number.isFinite(alpha) && alpha > 0 && Number.isFinite(beta) && beta > 0)) {
|
|
338
|
+
throw new RangeError(`Prior parameters must be finite and greater than 0, got Beta(${alpha}, ${beta}).`);
|
|
339
|
+
}
|
|
340
|
+
return [alpha, beta];
|
|
341
|
+
}
|
|
342
|
+
function isDefaultPrior([alpha, beta]) {
|
|
343
|
+
return alpha === PRIOR_ALPHA && beta === PRIOR_BETA;
|
|
344
|
+
}
|
|
262
345
|
function deriveSeed(...parts) {
|
|
263
346
|
let hash = 2166136261;
|
|
264
347
|
for (const part of parts) {
|
|
@@ -341,12 +424,14 @@ function analyze({
|
|
|
341
424
|
minEffect = DEFAULT_MIN_EFFECT,
|
|
342
425
|
samples = DEFAULT_SAMPLES,
|
|
343
426
|
seed = null,
|
|
344
|
-
includeChart = false
|
|
427
|
+
includeChart = false,
|
|
428
|
+
prior = [PRIOR_ALPHA, PRIOR_BETA]
|
|
345
429
|
}) {
|
|
346
|
-
const
|
|
347
|
-
const
|
|
348
|
-
const
|
|
349
|
-
const
|
|
430
|
+
const [priorAlpha, priorBeta] = validatePrior(prior);
|
|
431
|
+
const alphaA = priorAlpha + conversionsA;
|
|
432
|
+
const betaA = priorBeta + (visitorsA - conversionsA);
|
|
433
|
+
const alphaB = priorAlpha + conversionsB;
|
|
434
|
+
const betaB = priorBeta + (visitorsB - conversionsB);
|
|
350
435
|
const rateA = conversionsA / visitorsA;
|
|
351
436
|
const rateB = conversionsB / visitorsB;
|
|
352
437
|
const posteriorMeanA = alphaA / (alphaA + betaA);
|
|
@@ -381,7 +466,7 @@ function analyze({
|
|
|
381
466
|
conversionsB,
|
|
382
467
|
threshold,
|
|
383
468
|
minEffect,
|
|
384
|
-
prior: [
|
|
469
|
+
prior: [priorAlpha, priorBeta],
|
|
385
470
|
samples,
|
|
386
471
|
seed: resolvedSeed
|
|
387
472
|
},
|
|
@@ -429,7 +514,7 @@ function analyze({
|
|
|
429
514
|
},
|
|
430
515
|
waiting: waitingScenarios(alphaA, betaA, alphaB, betaB, expectedRegretNow),
|
|
431
516
|
frequentist: { pValueTwoSided: twoSidedPValue(visitorsA, conversionsA, visitorsB, conversionsB) },
|
|
432
|
-
shareUrl: shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect })
|
|
517
|
+
shareUrl: shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] })
|
|
433
518
|
};
|
|
434
519
|
if (includeChart) {
|
|
435
520
|
const sortedA = [...samplesA].sort((left, right) => left - right);
|
|
@@ -517,9 +602,11 @@ function project({
|
|
|
517
602
|
stopRule = "lift",
|
|
518
603
|
maxLoss = DEFAULT_MAX_LOSS,
|
|
519
604
|
simulations = DEFAULT_SIMULATIONS,
|
|
520
|
-
seed = null
|
|
605
|
+
seed = null,
|
|
606
|
+
prior = [PRIOR_ALPHA, PRIOR_BETA]
|
|
521
607
|
}) {
|
|
522
|
-
const
|
|
608
|
+
const [priorAlpha, priorBeta] = validatePrior(prior);
|
|
609
|
+
const current = analyze({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] });
|
|
523
610
|
const resolvedSeed = seed ?? deriveSeed(visitorsA, conversionsA, visitorsB, conversionsB, dailyVisitors, horizonDays, ["lift", "superiority", "loss"].indexOf(stopRule), simulations);
|
|
524
611
|
const inputs = {
|
|
525
612
|
visitorsA,
|
|
@@ -528,7 +615,7 @@ function project({
|
|
|
528
615
|
conversionsB,
|
|
529
616
|
threshold,
|
|
530
617
|
minEffect,
|
|
531
|
-
prior: [
|
|
618
|
+
prior: [priorAlpha, priorBeta],
|
|
532
619
|
dailyVisitors,
|
|
533
620
|
horizonDays,
|
|
534
621
|
stopRule,
|
|
@@ -536,7 +623,7 @@ function project({
|
|
|
536
623
|
simulations,
|
|
537
624
|
seed: resolvedSeed
|
|
538
625
|
};
|
|
539
|
-
const link = shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect });
|
|
626
|
+
const link = shareUrl({ visitorsA, conversionsA, visitorsB, conversionsB, threshold, minEffect, prior: [priorAlpha, priorBeta] });
|
|
540
627
|
const diagnostic = stopRule === "lift" && liftNearThreshold(current, minEffect) ? "lift_near_threshold" : null;
|
|
541
628
|
const decisionNow = decisionNowFrom(current, stopRule, threshold, maxLoss);
|
|
542
629
|
if (decisionNow) {
|
|
@@ -553,10 +640,10 @@ function project({
|
|
|
553
640
|
shareUrl: link
|
|
554
641
|
};
|
|
555
642
|
}
|
|
556
|
-
const alphaA0 =
|
|
557
|
-
const betaA0 =
|
|
558
|
-
const alphaB0 =
|
|
559
|
-
const betaB0 =
|
|
643
|
+
const alphaA0 = priorAlpha + conversionsA;
|
|
644
|
+
const betaA0 = priorBeta + (visitorsA - conversionsA);
|
|
645
|
+
const alphaB0 = priorAlpha + conversionsB;
|
|
646
|
+
const betaB0 = priorBeta + (visitorsB - conversionsB);
|
|
560
647
|
const random = makeDeterministicRandom(resolvedSeed);
|
|
561
648
|
const decidedByDay = new Array(horizonDays + 1).fill(0);
|
|
562
649
|
const decisionDays = [];
|
|
@@ -650,12 +737,15 @@ export {
|
|
|
650
737
|
analyze,
|
|
651
738
|
betaPdf,
|
|
652
739
|
integratedExpectedLoss,
|
|
740
|
+
isDefaultPrior,
|
|
653
741
|
liftDensity,
|
|
654
742
|
logBeta,
|
|
655
743
|
logGamma,
|
|
656
744
|
plan,
|
|
657
745
|
probabilityBBeatsA,
|
|
658
746
|
project,
|
|
747
|
+
regularizedIncompleteBeta,
|
|
659
748
|
requiredSamplePerVariant,
|
|
660
|
-
shareUrl
|
|
749
|
+
shareUrl,
|
|
750
|
+
validatePrior
|
|
661
751
|
};
|
package/package.json
CHANGED
|
@@ -1,8 +1,15 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "bayes-ab",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "Bayesian A/B test analysis on the command line, for humans and agents",
|
|
5
|
-
"keywords": [
|
|
5
|
+
"keywords": [
|
|
6
|
+
"ab-testing",
|
|
7
|
+
"bayesian",
|
|
8
|
+
"statistics",
|
|
9
|
+
"cli",
|
|
10
|
+
"experimentation",
|
|
11
|
+
"conversion-rate"
|
|
12
|
+
],
|
|
6
13
|
"homepage": "https://www.joebrundage.com/tools/bayesian-ab-test",
|
|
7
14
|
"repository": {
|
|
8
15
|
"type": "git",
|
|
@@ -23,7 +30,11 @@
|
|
|
23
30
|
"import": "./dist/index.js"
|
|
24
31
|
}
|
|
25
32
|
},
|
|
26
|
-
"files": [
|
|
33
|
+
"files": [
|
|
34
|
+
"dist",
|
|
35
|
+
"README.md",
|
|
36
|
+
"LICENSE"
|
|
37
|
+
],
|
|
27
38
|
"engines": {
|
|
28
39
|
"node": ">=18.11"
|
|
29
40
|
},
|