abfactor 0.5.1__tar.gz

abfactor-0.5.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Thomas Kealy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

abfactor-0.5.1/PKG-INFO

@@ -0,0 +1,250 @@
+Metadata-Version: 2.4
+Name: abfactor
+Version: 0.5.1
+Summary: Bayesian A/B testing models with Bayes factors: informed priors, default (Rouder) priors, BREASE, pooling across experiments, sequential and pre-period designs.
+Author: Thomas Kealy
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/TomKealy/abfactor
+Project-URL: Changelog, https://github.com/TomKealy/abfactor/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/TomKealy/abfactor/issues
+Keywords: a/b testing,bayes factor,experimentation,bayesian,pymc,sequential testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Mathematics
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=2.1
+Requires-Dist: scipy>=1.14
+Requires-Dist: pandas>=2.2
+Requires-Dist: xarray>=2025.1
+Requires-Dist: pymc>=6.0
+Requires-Dist: arviz>=1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: pdoc>=15; extra == "docs"
+Dynamic: license-file
+
+# abfactor
+
+Bayesian A/B testing with Bayes factors, built on the aggregated data experimentation
+platforms already produce, and checked by simulation.
+
+- **Bayes factors, not just P(B > A).** Evidence for *and against* "no effect", directional
+  tests, and exact pairwise comparisons in multi-arm tests.
+- **Default priors that suit online experiments.** The fixed-effects construction of
+  Rouder et al. (2012), with the prior scale on the model scale (log odds, log rate) so tiny
+  real effects aren't swamped by Lindley's paradox.
+- **Experiments that learn from each other.** A shared effect scale and a spike-and-slab
+  across a batch, with each experiment's Bayes factor computed given the rest.
+- **Aggregated data done properly.** Per-user squares and cross products give dispersion
+  for counts, real variances for per-user means and delta-method SEs for ratios.
+- **Time and pre-periods.** One-off conversions as a discrete-time hazard on the risk set,
+  A/A periods that catch broken buckets, CUPED and difference-in-differences.
+- **Sequential monitoring** that keeps its false-positive guarantee.
+
+## Install
+
+```bash
+pip install -e ".[dev]"
+pytest              # fast checks, ~25 s
+pytest -m slow      # also checks the quadrature engine against NUTS and SMC
+```
+
+## How it fits together
+
+An analysis is four choices (see [`docs/design.md`](docs/design.md)):
+
+| Piece | Options |
+|---|---|
+| **Metric** | `Proportion`, `TimeToEvent(link="logit" \| "cloglog")`, `Count(dispersion=...)`, `Mean(adjust=PrePeriod(...))`, `Ratio(num, den)` |
+| **Prior** | `DefaultFixedEffects(r)`, `NormalEffects(sd)`, `LearnedScale` (from completed experiments), `Brease(...)` |
+| **Pooling** | `Independent()`, `SharedScale()`, `SpikeAndSlab()` |
+| **Engine** | `"quadrature"` (default, fast, exact on the Gaussian reduction), `"nuts"`, `"smc"` (exact likelihoods) |
+
+Data come in as `Stats`: sufficient statistics with dimensions experiment × arm × period
+(`n`, `x`, `x__sq`, `x__x__y`: plain column names any warehouse accepts), built from
+user rows, event times, a long frame of query output, or arrays.
+
+## Quick start
+
+```python
+from abfactor import Analysis, Stats, Proportion, DefaultFixedEffects
+
+stats = Stats.from_arrays(arms=["control", "b", "c"], n=[[20000, 20000, 20000]],
+                          converted=[[1061, 1102, 1131]])
+result = Analysis(Proportion("converted"), baseline=0.05, prior=DefaultFixedEffects(r=0.1)).fit(stats)
+
+result.omnibus      # log BF10: does any arm differ?
+result.pairwise     # each variant vs control: BF10, BF+0, P(better), contrast
+result.summary()    # plus the relative lift with a 90% interval
+```
+
+From user-level data, with a ratio metric:
+
+```python
+from abfactor import Ratio, NormalEffects
+
+stats = Stats.from_users(users, metrics={"revenue": "revenue", "orders": "boxes"}, control="control")
+aov = Analysis(Ratio("revenue", "orders"), baseline=50, prior=NormalEffects(5)).fit(stats)
+```
+
+A batch of experiments that learn from each other:
+
+```python
+from abfactor import SpikeAndSlab
+
+result = Analysis(Proportion("converted"), 0.05, prior=DefaultFixedEffects(0.5),
+                  pooling=SpikeAndSlab()).fit(batch)
+result.omnibus["P_effect"]       # each experiment's probability of having any effect
+result.pooling["w_mean"]         # share of experiments with effects
+```
+
+One-off conversions with an A/A period before the treatment, guarding against broken buckets:
+
+```python
+from abfactor import TimeToEvent
+
+stats = Stats.from_events(users, event_period="converted_period", n_periods=2, control="control")
+result = Analysis(TimeToEvent("conversion"), 0.05, treatment_start=1, offsets="free").fit(stats)
+```
+
+Straight from a warehouse query, one row per experiment and arm:
+
+```sql
+SELECT experiment_id, variation,
+       COUNT(*)               AS n,
+       SUM(revenue)           AS revenue,  SUM(revenue * revenue) AS revenue__sq,
+       SUM(orders)            AS orders,   SUM(orders * orders)   AS orders__sq,
+       SUM(revenue * orders)  AS revenue__x__orders
+FROM user_level GROUP BY 1, 2
+```
+
+```python
+stats = Stats.from_frame(query_result, control="control", experiment="experiment_id", arm="variation")
+```
+
+Sequential monitoring, pooling across *completed* experiments only:
+
+```python
+from abfactor import Monitor
+
+prior = Analysis(Proportion("converted"), 0.05, prior=DefaultFixedEffects(0.5)).learn_prior(completed)
+monitor = Monitor(Analysis(Proportion("converted"), 0.05, prior=prior), threshold=20)
+monitor.update(stats_today)      # cumulative counts; stops the first time the evidence reaches 20
+```
+
+For conversion rates the monitor uses an exact e-process (`method="exact"`): the
+false-positive rate stays below 1/threshold however often you look and whatever the
+baseline. Other metrics use the Bayes factor (`method="bayes"`), checked by simulation.
+See [`docs/sequential.md`](docs/sequential.md).
+
+## Noise between identical arms
+
+In real experiments identical arms often differ by more than sampling noise allows. In
+the Upworthy headline archive the excess was about 13% of the click-through rate, and
+ignoring it made one A/A test in 16–40 show strong evidence of an effect
+([`docs/real_data.md`](docs/real_data.md)). Measure it from your own A/A tests:
+
+```python
+from abfactor.aa import estimate_arm_sd, aa_check
+
+est = estimate_arm_sd(Proportion("converted"), aa_stats)     # SD on the model scale, with a 95% interval
+analysis = Analysis(Proportion("converted"), 0.05, arm_sd=est["arm_sd"])
+aa_check(analysis, aa_stats)                                 # share of A/A tests with BF10 > 3, > 20, < 1/3
+```
+
+## Saving and reproducing results
+
+```python
+result.to_netcdf("checkout_test.nc")           # draws, Bayes factors, settings, package versions
+result = Result.from_netcdf("checkout_test.nc")
+result.to_frame()                               # flat table: one row per experiment and variant
+
+Analysis.from_settings(result.settings).fit(stats)   # recomputes the same draws exactly
+```
+
+## Small samples
+
+The default engine approximates each metric's likelihood by a normal on its model scale.
+Measured against exact integration ([`docs/accuracy.md`](docs/accuracy.md)), it never
+overstated the evidence for an effect, but:
+
+- with fewer than 5 conversions (or non-conversions) in an arm, intervals can be far too
+  narrow: `fit` raises a `SmallSampleWarning`, and `engine="smc"` or `"nuts"` should be used;
+- with 5 to 49, Bayes factors can understate the evidence by up to a factor of about 2.3
+  near the decision thresholds: the experiment is listed in
+  `result.diagnostics["small_sample"]["conservative_bf"]`, so borderline calls can be
+  confirmed with `engine="smc"`.
+
+## Choosing the prior scale
+
+Bayes factors depend on how big the alternative expects effects to be. A 10% lift on a
+5% conversion rate is a log odds ratio of about 0.1. With `r = 0.5` (Rouder et al.'s
+standardised default) the alternative bets on effects around 0.7 and the null wins almost
+regardless of the data. Use `r` around 0.1 for conversion rates, or pool across experiments
+and let the batch learn the scale.
+
+## What the simulations show
+
+From `python harness/harness.py --quick` (60 experiments per scenario; full results in
+`harness/results/`):
+
+- **Sequential monitoring:** stopping at BF10 ≥ 20 gave 0.8% false positives over 30 looks
+  (the guarantee is 5%), against 33% for peeking at a z-test.
+- **Overdispersed counts:** a Poisson model gave false positives of 22% in one tail; the
+  quasi-Poisson correction and the per-user mean brought it back to 3–12%.
+- **AOV:** treating orders as independent gave intervals that held the true lift 62–68% of
+  the time; the delta method 85–92%.
+- **Broken buckets:** with an A/A period, modelling arm offsets cut false positives from
+  62% to 2–5%.
+- **CUPED:** at a pre/post correlation of 0.62 it cut the SD of the lift by 20% (theory:
+  21%), with level intervals now calibrated too (89–91% coverage at 200 experiments).
+
+## Known limitations
+
+- The exact sequential guarantee covers conversion rates with binomial arms. For other
+  metrics, and with `arm_sd`, the Bayes-factor monitor's error rate is checked by
+  simulation, not proven.
+- The quadrature engine's normal approximation is miscalibrated for the control level when
+  arms have few events (about 10–50); contrasts hold up better
+  ([`docs/calibration.md`](docs/calibration.md)).
+- Learned arm offsets (`offsets="shrunk"`) need `engine="nuts"`; spike-and-slab under NUTS
+  gives P(effect) but not pairwise Bayes factors.
+- `Ratio` has no pre-period adjustment yet.
+- Small-sample thresholds were measured for binary outcomes only (see `docs/accuracy.md`).
+
+## Also included
+
+- `abfactor.brease`: the BREASE framework (Irons & Cinelli 2023) in full; also usable as
+  `prior=Brease(...)` for a two-arm binary experiment.
+- `abfactor.anova`: Rouder et al.'s standardised-effects ANOVA Bayes factors, for reference.
+- `abfactor.baselines`: independent-beta and abtest-style logit-transform Bayes factors.
+- `examples/`: a tour of the API (`quickstart.py`) and reproductions of the BREASE and abtest papers.
+- `harness/sbc.py`: simulation-based calibration of every engine and metric
+  ([`docs/calibration.md`](docs/calibration.md)); `harness/upworthy.py`: the real-data checks.
+- `docs/posts/`: three posts explaining the methods with plain PyMC, each ending with the
+  same analysis in abfactor.
+
+## Licence, stability and roadmap
+
+MIT licensed (`LICENSE`). What counts as public API, and what 1.0 will promise about it, is in
+[`docs/api.md`](docs/api.md); the API reference is built from the docstrings (`pip install -e ".[docs]"`,
+then `pdoc abfactor`). The remaining work before 1.0 is in [`docs/roadmap.md`](docs/roadmap.md).
+
+## References
+
+- Rouder, Morey, Speckman & Province (2012). Default Bayes factors for ANOVA designs. *Journal of Mathematical Psychology* 56(5).
+- Rouder, Speckman, Sun, Morey & Iverson (2009). Bayesian t tests for accepting and rejecting the null hypothesis. *Psychonomic Bulletin & Review* 16(2).
+- Irons & Cinelli (2023). Causally sound priors for binary experiments. arXiv.
+- Gronau, Raj K. N. & Wagenmakers (2021). Informed Bayesian inference for the A/B test. *Journal of Statistical Software* 100(17).
+- Deng, Knoblich & Lu (2018). Applying the delta method in metric analytics. *KDD*.
+- Wedderburn (1974). Quasi-likelihood functions. *Biometrika* 61(3).
+- Dickey & Lientz (1970); Wagenmakers et al. (2010): the Savage–Dickey density ratio.
+
+Full bibliography: `docs/references.bib`.

abfactor-0.5.1/README.md

@@ -0,0 +1,218 @@
+# abfactor
+
+Bayesian A/B testing with Bayes factors, built on the aggregated data experimentation
+platforms already produce, and checked by simulation.
+
+- **Bayes factors, not just P(B > A).** Evidence for *and against* "no effect", directional
+  tests, and exact pairwise comparisons in multi-arm tests.
+- **Default priors that suit online experiments.** The fixed-effects construction of
+  Rouder et al. (2012), with the prior scale on the model scale (log odds, log rate) so tiny
+  real effects aren't swamped by Lindley's paradox.
+- **Experiments that learn from each other.** A shared effect scale and a spike-and-slab
+  across a batch, with each experiment's Bayes factor computed given the rest.
+- **Aggregated data done properly.** Per-user squares and cross products give dispersion
+  for counts, real variances for per-user means and delta-method SEs for ratios.
+- **Time and pre-periods.** One-off conversions as a discrete-time hazard on the risk set,
+  A/A periods that catch broken buckets, CUPED and difference-in-differences.
+- **Sequential monitoring** that keeps its false-positive guarantee.
+
+## Install
+
+```bash
+pip install -e ".[dev]"
+pytest              # fast checks, ~25 s
+pytest -m slow      # also checks the quadrature engine against NUTS and SMC
+```
+
+## How it fits together
+
+An analysis is four choices (see [`docs/design.md`](docs/design.md)):
+
+| Piece | Options |
+|---|---|
+| **Metric** | `Proportion`, `TimeToEvent(link="logit" \| "cloglog")`, `Count(dispersion=...)`, `Mean(adjust=PrePeriod(...))`, `Ratio(num, den)` |
+| **Prior** | `DefaultFixedEffects(r)`, `NormalEffects(sd)`, `LearnedScale` (from completed experiments), `Brease(...)` |
+| **Pooling** | `Independent()`, `SharedScale()`, `SpikeAndSlab()` |
+| **Engine** | `"quadrature"` (default, fast, exact on the Gaussian reduction), `"nuts"`, `"smc"` (exact likelihoods) |
+
+Data come in as `Stats`: sufficient statistics with dimensions experiment × arm × period
+(`n`, `x`, `x__sq`, `x__x__y`: plain column names any warehouse accepts), built from
+user rows, event times, a long frame of query output, or arrays.
+
+## Quick start
+
+```python
+from abfactor import Analysis, Stats, Proportion, DefaultFixedEffects
+
+stats = Stats.from_arrays(arms=["control", "b", "c"], n=[[20000, 20000, 20000]],
+                          converted=[[1061, 1102, 1131]])
+result = Analysis(Proportion("converted"), baseline=0.05, prior=DefaultFixedEffects(r=0.1)).fit(stats)
+
+result.omnibus      # log BF10: does any arm differ?
+result.pairwise     # each variant vs control: BF10, BF+0, P(better), contrast
+result.summary()    # plus the relative lift with a 90% interval
+```
+
+From user-level data, with a ratio metric:
+
+```python
+from abfactor import Ratio, NormalEffects
+
+stats = Stats.from_users(users, metrics={"revenue": "revenue", "orders": "boxes"}, control="control")
+aov = Analysis(Ratio("revenue", "orders"), baseline=50, prior=NormalEffects(5)).fit(stats)
+```
+
+A batch of experiments that learn from each other:
+
+```python
+from abfactor import SpikeAndSlab
+
+result = Analysis(Proportion("converted"), 0.05, prior=DefaultFixedEffects(0.5),
+                  pooling=SpikeAndSlab()).fit(batch)
+result.omnibus["P_effect"]       # each experiment's probability of having any effect
+result.pooling["w_mean"]         # share of experiments with effects
+```
+
+One-off conversions with an A/A period before the treatment, guarding against broken buckets:
+
+```python
+from abfactor import TimeToEvent
+
+stats = Stats.from_events(users, event_period="converted_period", n_periods=2, control="control")
+result = Analysis(TimeToEvent("conversion"), 0.05, treatment_start=1, offsets="free").fit(stats)
+```
+
+Straight from a warehouse query, one row per experiment and arm:
+
+```sql
+SELECT experiment_id, variation,
+       COUNT(*)               AS n,
+       SUM(revenue)           AS revenue,  SUM(revenue * revenue) AS revenue__sq,
+       SUM(orders)            AS orders,   SUM(orders * orders)   AS orders__sq,
+       SUM(revenue * orders)  AS revenue__x__orders
+FROM user_level GROUP BY 1, 2
+```
+
+```python
+stats = Stats.from_frame(query_result, control="control", experiment="experiment_id", arm="variation")
+```
+
+Sequential monitoring, pooling across *completed* experiments only:
+
+```python
+from abfactor import Monitor
+
+prior = Analysis(Proportion("converted"), 0.05, prior=DefaultFixedEffects(0.5)).learn_prior(completed)
+monitor = Monitor(Analysis(Proportion("converted"), 0.05, prior=prior), threshold=20)
+monitor.update(stats_today)      # cumulative counts; stops the first time the evidence reaches 20
+```
+
+For conversion rates the monitor uses an exact e-process (`method="exact"`): the
+false-positive rate stays below 1/threshold however often you look and whatever the
+baseline. Other metrics use the Bayes factor (`method="bayes"`), checked by simulation.
+See [`docs/sequential.md`](docs/sequential.md).
+
+## Noise between identical arms
+
+In real experiments identical arms often differ by more than sampling noise allows. In
+the Upworthy headline archive the excess was about 13% of the click-through rate, and
+ignoring it made one A/A test in 16–40 show strong evidence of an effect
+([`docs/real_data.md`](docs/real_data.md)). Measure it from your own A/A tests:
+
+```python
+from abfactor.aa import estimate_arm_sd, aa_check
+
+est = estimate_arm_sd(Proportion("converted"), aa_stats)     # SD on the model scale, with a 95% interval
+analysis = Analysis(Proportion("converted"), 0.05, arm_sd=est["arm_sd"])
+aa_check(analysis, aa_stats)                                 # share of A/A tests with BF10 > 3, > 20, < 1/3
+```
+
+## Saving and reproducing results
+
+```python
+result.to_netcdf("checkout_test.nc")           # draws, Bayes factors, settings, package versions
+result = Result.from_netcdf("checkout_test.nc")
+result.to_frame()                               # flat table: one row per experiment and variant
+
+Analysis.from_settings(result.settings).fit(stats)   # recomputes the same draws exactly
+```
+
+## Small samples
+
+The default engine approximates each metric's likelihood by a normal on its model scale.
+Measured against exact integration ([`docs/accuracy.md`](docs/accuracy.md)), it never
+overstated the evidence for an effect, but:
+
+- with fewer than 5 conversions (or non-conversions) in an arm, intervals can be far too
+  narrow: `fit` raises a `SmallSampleWarning`, and `engine="smc"` or `"nuts"` should be used;
+- with 5 to 49, Bayes factors can understate the evidence by up to a factor of about 2.3
+  near the decision thresholds: the experiment is listed in
+  `result.diagnostics["small_sample"]["conservative_bf"]`, so borderline calls can be
+  confirmed with `engine="smc"`.
+
+## Choosing the prior scale
+
+Bayes factors depend on how big the alternative expects effects to be. A 10% lift on a
+5% conversion rate is a log odds ratio of about 0.1. With `r = 0.5` (Rouder et al.'s
+standardised default) the alternative bets on effects around 0.7 and the null wins almost
+regardless of the data. Use `r` around 0.1 for conversion rates, or pool across experiments
+and let the batch learn the scale.
+
+## What the simulations show
+
+From `python harness/harness.py --quick` (60 experiments per scenario; full results in
+`harness/results/`):
+
+- **Sequential monitoring:** stopping at BF10 ≥ 20 gave 0.8% false positives over 30 looks
+  (the guarantee is 5%), against 33% for peeking at a z-test.
+- **Overdispersed counts:** a Poisson model gave false positives of 22% in one tail; the
+  quasi-Poisson correction and the per-user mean brought it back to 3–12%.
+- **AOV:** treating orders as independent gave intervals that held the true lift 62–68% of
+  the time; the delta method 85–92%.
+- **Broken buckets:** with an A/A period, modelling arm offsets cut false positives from
+  62% to 2–5%.
+- **CUPED:** at a pre/post correlation of 0.62 it cut the SD of the lift by 20% (theory:
+  21%), with level intervals now calibrated too (89–91% coverage at 200 experiments).
+
+## Known limitations
+
+- The exact sequential guarantee covers conversion rates with binomial arms. For other
+  metrics, and with `arm_sd`, the Bayes-factor monitor's error rate is checked by
+  simulation, not proven.
+- The quadrature engine's normal approximation is miscalibrated for the control level when
+  arms have few events (about 10–50); contrasts hold up better
+  ([`docs/calibration.md`](docs/calibration.md)).
+- Learned arm offsets (`offsets="shrunk"`) need `engine="nuts"`; spike-and-slab under NUTS
+  gives P(effect) but not pairwise Bayes factors.
+- `Ratio` has no pre-period adjustment yet.
+- Small-sample thresholds were measured for binary outcomes only (see `docs/accuracy.md`).
+
+## Also included
+
+- `abfactor.brease`: the BREASE framework (Irons & Cinelli 2023) in full; also usable as
+  `prior=Brease(...)` for a two-arm binary experiment.
+- `abfactor.anova`: Rouder et al.'s standardised-effects ANOVA Bayes factors, for reference.
+- `abfactor.baselines`: independent-beta and abtest-style logit-transform Bayes factors.
+- `examples/`: a tour of the API (`quickstart.py`) and reproductions of the BREASE and abtest papers.
+- `harness/sbc.py`: simulation-based calibration of every engine and metric
+  ([`docs/calibration.md`](docs/calibration.md)); `harness/upworthy.py`: the real-data checks.
+- `docs/posts/`: three posts explaining the methods with plain PyMC, each ending with the
+  same analysis in abfactor.
+
+## Licence, stability and roadmap
+
+MIT licensed (`LICENSE`). What counts as public API, and what 1.0 will promise about it, is in
+[`docs/api.md`](docs/api.md); the API reference is built from the docstrings (`pip install -e ".[docs]"`,
+then `pdoc abfactor`). The remaining work before 1.0 is in [`docs/roadmap.md`](docs/roadmap.md).
+
+## References
+
+- Rouder, Morey, Speckman & Province (2012). Default Bayes factors for ANOVA designs. *Journal of Mathematical Psychology* 56(5).
+- Rouder, Speckman, Sun, Morey & Iverson (2009). Bayesian t tests for accepting and rejecting the null hypothesis. *Psychonomic Bulletin & Review* 16(2).
+- Irons & Cinelli (2023). Causally sound priors for binary experiments. arXiv.
+- Gronau, Raj K. N. & Wagenmakers (2021). Informed Bayesian inference for the A/B test. *Journal of Statistical Software* 100(17).
+- Deng, Knoblich & Lu (2018). Applying the delta method in metric analytics. *KDD*.
+- Wedderburn (1974). Quasi-likelihood functions. *Biometrika* 61(3).
+- Dickey & Lientz (1970); Wagenmakers et al. (2010): the Savage–Dickey density ratio.
+
+Full bibliography: `docs/references.bib`.

abfactor-0.5.1/pyproject.toml

@@ -0,0 +1,48 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "abfactor"
+version = "0.5.1"
+description = "Bayesian A/B testing models with Bayes factors: informed priors, default (Rouder) priors, BREASE, pooling across experiments, sequential and pre-period designs."
+readme = "README.md"
+requires-python = ">=3.12"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Thomas Kealy" }]
+keywords = ["a/b testing", "bayes factor", "experimentation", "bayesian", "pymc", "sequential testing"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Mathematics",
+]
+dependencies = [
+    "numpy>=2.1",
+    "scipy>=1.14",
+    "pandas>=2.2",
+    "xarray>=2025.1",
+    "pymc>=6.0",
+    "arviz>=1.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/TomKealy/abfactor"
+Changelog = "https://github.com/TomKealy/abfactor/blob/main/CHANGELOG.md"
+Issues = "https://github.com/TomKealy/abfactor/issues"
+
+[project.optional-dependencies]
+dev = ["pytest>=8"]
+docs = ["pdoc>=15"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-m 'not slow'"
+markers = ["slow: runs MCMC or SMC (deselected by default; run with -m slow)"]

abfactor-0.5.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

abfactor-0.5.1/src/abfactor/__init__.py

@@ -0,0 +1,27 @@
+"""Bayesian A/B testing with Bayes factors.
+
+    from abfactor import Analysis, Stats, Proportion, DefaultFixedEffects, SpikeAndSlab
+
+    stats = Stats.from_users(users, metrics={"converted": "converted"}, control="control")
+    result = Analysis(Proportion("converted"), baseline=0.05,
+                      prior=DefaultFixedEffects(r=0.1)).fit(stats)
+    result.summary()
+
+Pieces: Stats (sufficient statistics) x Metric (Proportion, TimeToEvent, Count, Mean,
+Ratio) x prior (DefaultFixedEffects, NormalEffects, LearnedScale, Brease) x pooling
+(Independent, SharedScale, SpikeAndSlab) x engine ("quadrature", "nuts", "smc").
+Monitor runs sequential tests. See docs/design.md.
+"""
+from .analysis import Analysis, SmallSampleWarning
+from .metrics import Count, Mean, PrePeriod, Proportion, Ratio, TimeToEvent
+from .pooling import Independent, SharedScale, SpikeAndSlab
+from .priors import Brease, DefaultFixedEffects, LearnedScale, NormalEffects
+from .results import Result
+from .sequential import Monitor
+from .stats import Stats, cross, square
+
+__version__ = "0.5.1"
+
+__all__ = ["Analysis", "SmallSampleWarning", "Stats", "Result", "Monitor", "Proportion", "TimeToEvent", "Count", "Mean",
+           "Ratio", "PrePeriod", "DefaultFixedEffects", "NormalEffects", "LearnedScale", "Brease",
+           "Independent", "SharedScale", "SpikeAndSlab", "square", "cross"]

abfactor-0.5.1/src/abfactor/aa.py

@@ -0,0 +1,75 @@
+"""Learning from A/A tests: how much identical arms really differ.
+
+In real experiments, arms that show users exactly the same thing still differ by more
+than sampling noise allows: traffic shifts during a test, repeat visitors, bots and
+logging all add variation between arms that more data does not average away. In the
+Upworthy Research Archive's exploratory set, identical arms differed by about 12% in
+click-through rate beyond binomial noise (docs/real_data.md). A model that ignores it
+reports effects that are not there.
+
+estimate_arm_sd measures that extra noise from your own A/A tests, as an SD on the
+metric's model scale; pass it to Analysis(arm_sd=...). aa_check reports how often an
+analysis finds evidence of an effect in A/A tests, which should be rare.
+"""
+from __future__ import annotations
+
+import numpy as np
+import pandas as pd
+from scipy.optimize import minimize_scalar
+
+from .analysis import Analysis
+from .design import sum_to_zero_basis
+from .metrics import Metric
+from .stats import Stats
+
+__all__ = ["estimate_arm_sd", "aa_check", "arm_noise_loglik"]
+
+
+def _contrasts(metric: Metric, stats: Stats):
+    d, v = metric.estimates(stats)
+    E, A, P = d.shape
+    Q = sum_to_zero_basis(A)
+    c = np.einsum("eap,ak->ekp", d, Q)                       # within-period contrasts, (E, A-1, P)
+    return c, v, Q
+
+
+def arm_noise_loglik(tau: float, metric: Metric, stats: Stats) -> float:
+    """Restricted log-likelihood of an extra per-arm SD tau: the contrasts Q'd within each
+    experiment and period are N(0, Q'(diag(v) + tau^2 I)Q) = N(0, Q' diag(v) Q + tau^2 I)."""
+    c, v, Q = _contrasts(metric, stats)
+    total = 0.0
+    E, K, P = c.shape
+    for p in range(P):
+        S = np.einsum("ak,ea,al->ekl", Q, v[..., p], Q) + tau ** 2 * np.eye(K)[None]
+        sign, logdet = np.linalg.slogdet(S)
+        quad = np.einsum("ek,ekl,el->e", c[..., p], np.linalg.inv(S), c[..., p])
+        total += float(-0.5 * (logdet + quad + K * np.log(2 * np.pi)).sum())
+    return total
+
+
+def estimate_arm_sd(metric: Metric, aa_stats: Stats, upper: float = 2.0) -> dict:
+    """Estimate the extra per-arm SD from A/A tests (all arms identical), by restricted
+    maximum likelihood on the contrasts, so each test's unknown common level drops out.
+
+    aa_stats may contain experiments with any number of identical arms as separate Stats;
+    pass a list to combine them. Returns the estimate, an approximate 95% interval from
+    the likelihood ratio, and the number of tests and arms used."""
+    parts = aa_stats if isinstance(aa_stats, (list, tuple)) else [aa_stats]
+    f = lambda t: -sum(arm_noise_loglik(t, metric, s) for s in parts)
+    best = minimize_scalar(f, bounds=(0.0, upper), method="bounded", options={"xatol": 1e-4})
+    tau, lmax = float(best.x), -float(best.fun)
+    grid = np.linspace(0, upper, 801)
+    ll = np.array([-f(t) for t in grid])
+    inside = grid[ll >= lmax - 1.92]                               # chi2(1) 95% / 2
+    return {"arm_sd": tau, "interval_95": (float(inside.min()), float(inside.max())),
+            "loglik_gain_vs_zero": lmax - (-f(0.0)),
+            "tests": int(sum(len(s) for s in parts)), "arms": int(sum(len(s) * s.shape[1] for s in parts))}
+
+
+def aa_check(analysis: Analysis, aa_stats: Stats) -> dict:
+    """How often an analysis finds evidence of a difference in A/A tests. For a calibrated
+    analysis, Bayes factors above 20 should be very rare, and evidence for no difference
+    (BF10 below 1/3) common."""
+    lbf = analysis.log_bf10(aa_stats)
+    return {"tests": len(lbf), "BF10>3": float(np.mean(lbf > np.log(3))),
+            "BF10>20": float(np.mean(lbf > np.log(20))), "BF10<1/3": float(np.mean(lbf < -np.log(3)))}