@variantlab/core 0.1.4 → 0.1.6

package/docs/features/killer-features.md

@@ -0,0 +1,308 @@
+# Killer features
+
+The 10 features that make variantlab different from every free and paid A/B testing tool we surveyed. Each links to its own detailed spec.
+
+## Table of contents
+
+1. [On-device debug overlay](#1-on-device-debug-overlay)
+2. [Route-scoped experiments](#2-route-scoped-experiments)
+3. [Type-safe codegen from JSON](#3-type-safe-codegen-from-json)
+4. [Value AND render experiments](#4-value-and-render-experiments)
+5. [Deep-link + QR state sharing](#5-deep-link--qr-state-sharing)
+6. [Crash-triggered automatic rollback](#6-crash-triggered-automatic-rollback)
+7. [Time-travel inspector](#7-time-travel-inspector)
+8. [Offline-first by default](#8-offline-first-by-default)
+9. [HMAC-signed remote configs](#9-hmac-signed-remote-configs)
+10. [Multivariate + feature flags in one tool](#10-multivariate--feature-flags-in-one-tool)
+
+---
+
+## 1. On-device debug overlay
+
+**What it is**: A floating button you embed in dev builds. Tap it to open a bottom sheet listing every experiment active on the current route, with a tappable picker to switch variants live.
+
+**Why it matters**: Existing tools either make you edit a config file + restart (slow), or open a web dashboard in a browser (outside the app context). Neither lets a designer or PM stand next to the phone and *feel* the difference in real time.
+
+**Where we got the idea**: Directly from the Drishtikon small-phone card problem. See [`origin-story.md`](../research/origin-story.md).
+
+**Who has it**:
+
+- variantlab: **Yes**, first-class, on device
+- Firebase Remote Config: **No**
+- GrowthBook: **No** (web dashboard only)
+- Statsig: **No** (web dashboard only)
+- LaunchDarkly: **No** (web dashboard only)
+- Amplitude: **No**
+- react-native-ab: **No**
+
+Spec: [`debug-overlay.md`](./debug-overlay.md)
+
+---
+
+## 2. Route-scoped experiments
+
+**What it is**: Every experiment can declare a `routes` field (glob patterns). The debug overlay automatically filters to show only experiments relevant to the current screen.
+
+**Why it matters**: Real apps have dozens of experiments. Showing all of them on every screen drowns the signal. Route scoping turns the debug overlay from a giant list into a focused "what's on this page" view.
+
+**Extra benefit**: Experiments evaluate faster because non-matching ones are skipped entirely.
+
+**Who has it**:
+
+- variantlab: **Yes**
+- GrowthBook: Partial (URL targeting in dashboard, no per-route client filtering)
+- Everyone else: **No**
+
+Spec: [`targeting.md`](./targeting.md#routes)
+
+---
+
+## 3. Type-safe codegen from JSON
+
+**What it is**: The `variantlab generate` CLI reads `experiments.json` and emits a TypeScript `.d.ts` with literal-union types for every experiment ID and variant ID. The hooks are overloaded on these types.
+
+```ts
+const variant = useVariant("news-card-layout");
+// variant is typed as:
+// "responsive" | "scale-to-fit" | "pip-thumbnail"
+
+const broken = useVariant("news-card-lay0ut");
+// ❌ Type error: argument of type '"news-card-lay0ut"' is not assignable
+```
+
+**Why it matters**: Stringly-typed experiments are a source of production bugs. One typo and you're silently stuck on the default variant. Codegen turns every typo into a compile error.
+
+**Who has it**:
+
+- variantlab: **Yes** (first-class, via CLI)
+- Statsig: Partial (via Statsig Console, requires network)
+- LaunchDarkly: Partial (enterprise feature only)
+- Everyone else: **No**
+
+Spec: [`codegen.md`](./codegen.md)
+
+---
+
+## 4. Value AND render experiments
+
+**What it is**: A single config format supports two experiment types:
+
+- `type: "render"` — component swaps via `<Variant experimentId="...">`
+- `type: "value"` — returned values via `useVariantValue<T>(id)`
+
+Same JSON, same targeting, same debug overlay.
+
+```json
+{ "id": "cta-copy", "type": "value", "variants": [
+    { "id": "buy", "value": "Buy now" },
+    { "id": "get", "value": "Get started" }
+]}
+```
+
+```ts
+const copy = useVariantValue("cta-copy"); // "Buy now" | "Get started"
+```
+
+**Why it matters**: Other libraries force you to pick: either feature flags (values) or component-swap tools (render). variantlab is both. You learn one API, use it everywhere.
+
+**Who has it**:
+
+- variantlab: **Yes**, unified
+- Firebase Remote Config: Values only
+- LaunchDarkly: Values + custom JSON (no component-swap helper)
+- react-native-ab: Render only
+- Others: Usually one or the other
+
+Spec: [`value-experiments.md`](./value-experiments.md)
+
+---
+
+## 5. Deep-link + QR state sharing
+
+**What it is**: Any variant override can be encoded as a URL or QR code. A QA engineer can say "try this" by sharing a link. Scanning sets the exact same variants on another device.
+
+```
+drishtikon://variantlab?override=bmV3cy1jYXJkLWxheW91dDoyNQ
+```
+
+The payload is:
+
+- Base64url-encoded
+- Length-limited
+- Signed with HMAC (if enabled)
+- Rejected if the experiment has `overridable: false`
+
+**Why it matters**: Remote debugging becomes trivial. No screen recordings, no "press button X then Y". Just send a link or a QR.
+
+**Who has it**:
+
+- variantlab: **Yes**
+- Everyone else: **No** (you can build it yourself in every tool; no one ships it)
+
+Spec: [`qr-sharing.md`](./qr-sharing.md)
+
+---
+
+## 6. Crash-triggered automatic rollback
+
+**What it is**: Wrap a variant in `<VariantErrorBoundary>`. If that variant crashes `threshold` times within `window` ms, the engine clears the assignment and forces the default. A warning is emitted and (optionally) persisted so subsequent sessions stay on default.
+
+```json
+"rollback": { "threshold": 3, "window": 60000, "persistent": true }
+```
+
+**Why it matters**: A bad variant can crash 100% of the users targeted by it. Without rollback, your only fix is to ship a new config or new app version. With rollback, the user self-heals on the second crash.
+
+**Where we got the idea**: From building a card layout that crashed on certain article data. We needed a safety net.
+
+**Who has it**:
+
+- variantlab: **Yes**, built in
+- Sentry: Can detect crashes but can't rollback experiments
+- LaunchDarkly: Manual kill-switch only
+- Everyone else: **No**
+
+Spec: [`crash-rollback.md`](./crash-rollback.md)
+
+---
+
+## 7. Time-travel inspector
+
+**What it is**: The engine records every assignment, override, config load, and rollback event with a timestamp. The debug overlay exposes a "history" tab that lets you scrub backwards to see what was active at any point.
+
+**Why it matters**:
+
+- "Why did user X see variant Y?" becomes easy to answer
+- QA can reproduce issues without guessing
+- Rollback events become visible
+
+**Who has it**:
+
+- variantlab: **Yes** (dev-only)
+- Redux DevTools: time-travel for state, not experiments
+- Everyone else: **No**
+
+Spec: [`time-travel.md`](./time-travel.md)
+
+---
+
+## 8. Offline-first by default
+
+**What it is**: The engine loads its config from local storage on boot. Network fetches are background refreshes. If the network fails, nothing breaks. If the device is offline for a week, experiments still resolve.
+
+**Why it matters**: Mobile apps spend 30%+ of their runtime offline or on flaky networks. A tool that requires the network to resolve experiments is a tool that degrades user experience.
+
+**How we do it**:
+
+- The initial config ships bundled with the app
+- Remote configs are cached after the first fetch
+- The engine resolves from cache synchronously
+- Background refreshes update the cache without blocking
+
+**Who has it**:
+
+- variantlab: **Yes**, by design
+- Firebase Remote Config: Yes, but blocks app startup in many configurations
+- GrowthBook: Partial
+- LaunchDarkly: Yes (caching SDK)
+- Unleash: Partial
+
+Nothing unique here, but it's table stakes and we refuse to ship without it.
+
+---
+
+## 9. HMAC-signed remote configs
+
+**What it is**: Remote configs are optionally signed with HMAC-SHA256 using a shared secret. The engine verifies the signature before applying the config. Tampered or unauthorized configs are rejected.
+
+```json
+{
+  "signature": "dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk",
+  "version": 1,
+  "experiments": [...]
+}
+```
+
+**Why it matters**: Without signing, anyone who MITM's your CDN can push a malicious config that changes which variant your users see — including weaponized variants that collect data.
+
+**How we do it**:
+
+- Web Crypto API (universal across runtimes)
+- Constant-time verification
+- Secret is embedded at build time, not transmitted
+- Signing is a CLI command: `variantlab sign experiments.json --key $SECRET`
+
+**Who has it**:
+
+- variantlab: **Yes** (optional)
+- LaunchDarkly: Yes (enterprise)
+- Statsig: Partial (TLS only)
+- Everyone else: **No**
+
+Spec: [`hmac-signing.md`](./hmac-signing.md)
+
+---
+
+## 10. Multivariate + feature flags in one tool
+
+**What it is**: variantlab isn't just A/B testing. The same engine handles:
+
+- **2-variant A/B**: `[control, treatment]`
+- **Multivariate**: `[A, B, C, D, E]` with arbitrary splits
+- **Feature flags**: `[off, on]` with targeting and rollback
+- **Kill switches**: global `enabled: false` or per-experiment
+- **Time-boxed experiments**: `startDate` / `endDate`
+- **Weighted rollouts**: 10% → 50% → 100% via split updates
+
+**Why it matters**: Most teams use 2-3 different tools: one for A/B, one for feature flags, one for rollouts. variantlab does all of them with one config file, one API, one mental model.
+
+**Who has it**:
+
+- variantlab: **Yes**
+- LaunchDarkly: Yes (but $$$)
+- Statsig: Yes
+- GrowthBook: Yes
+- Firebase Remote Config: Partial (weak A/B support)
+- Unleash: Flags only
+- ConfigCat: Flags only
+
+variantlab is unique in being **free + open-source + lightweight + all-in-one**.
+
+Spec: [`multivariate.md`](./multivariate.md)
+
+---
+
+## What's NOT in the killer-feature list (and why)
+
+We deliberately don't market these as killer features, even though we support them:
+
+- **Sticky hashing** — every tool has it
+- **User targeting** — every tool has it
+- **Percentage rollouts** — every tool has it
+- **Session persistence** — table stakes
+- **Telemetry hooks** — we expose the interface but don't ship telemetry
+
+Our differentiators are the 10 above. Everything else is baseline.
+
+---
+
+## The "one product" positioning
+
+Looking at all 10 features together, variantlab's positioning is:
+
+> "The developer experience of Storybook + the power of LaunchDarkly + the footprint of a tweet, for every frontend framework."
+
+- **Storybook-like DX**: on-device picker, route-scoped, time-travel
+- **LaunchDarkly-like power**: targeting, rollback, signing, multivariate
+- **Tweet-sized footprint**: < 3 KB gzipped core
+- **Framework-agnostic**: 10+ adapters from the same core
+
+No competitor hits all four quadrants. That's the gap we fill.
+
+---
+
+## See also
+
+- [`origin-story.md`](../research/origin-story.md) — why we built it
+- [`competitors.md`](../research/competitors.md) — how we compare
+- [`ROADMAP.md`](../../ROADMAP.md) — when each feature ships

package/docs/features/multivariate.md

@@ -0,0 +1,339 @@
+# Multivariate experiments
+
+Running experiments with 3 or more variants, including weighted splits and mutually exclusive groups.
+
+## Table of contents
+
+- [What is a multivariate experiment](#what-is-a-multivariate-experiment)
+- [When to use one](#when-to-use-one)
+- [Defining 3+ variants](#defining-3-variants)
+- [Weighted splits](#weighted-splits)
+- [Mutual exclusion](#mutual-exclusion)
+- [Statistical considerations](#statistical-considerations)
+- [Real-world examples](#real-world-examples)
+
+---
+
+## What is a multivariate experiment
+
+An experiment with more than 2 variants. In the classic A/B sense, it's an "A/B/C/D/..." test. variantlab treats all experiments uniformly — whether you have 2 variants or 30, the config and API are the same.
+
+**Upper limit**: 100 variants per experiment. More than that and the picker UI becomes unusable anyway.
+
+---
+
+## When to use one
+
+### Good fits
+
+- **Copy testing with many candidates** — "Buy now", "Get started", "Try free", "Start trial", "Shop now"
+- **Layout exploration** — the Drishtikon 30 card modes
+- **Color palette experiments** — 5 palettes, pick the best
+- **Pricing sweeps** — $4.99, $6.99, $9.99, $14.99
+- **Feature sets** — "minimal", "standard", "all features"
+
+### Bad fits
+
+- **Anything with fewer than 100 users per variant** — statistical noise will dominate. Stick to 2 variants.
+- **When variants are very different** — measuring differences becomes meaningless if variants are apples and oranges. Split into multiple experiments.
+- **When you don't have a success metric** — multivariate tests burn traffic. Without a clear metric, you're just randomly showing different UIs.
+
+---
+
+## Defining 3+ variants
+
+Just add more variants to the array:
+
+```json
+{
+  "id": "cta-copy",
+  "type": "value",
+  "default": "buy-now",
+  "variants": [
+    { "id": "buy-now", "value": "Buy now" },
+    { "id": "get-started", "value": "Get started" },
+    { "id": "try-free", "value": "Try it free" },
+    { "id": "start-trial", "value": "Start your trial" },
+    { "id": "shop-now", "value": "Shop now" }
+  ]
+}
+```
+
+No special config. The assignment strategy handles the rest.
+
+### With the render-prop component
+
+```tsx
+<Variant experimentId="cta-copy">
+  {{
+    "buy-now": <Button>Buy now</Button>,
+    "get-started": <Button>Get started</Button>,
+    "try-free": <Button>Try it free</Button>,
+    "start-trial": <Button>Start your trial</Button>,
+    "shop-now": <Button>Shop now</Button>,
+  }}
+</Variant>
+```
+
+TypeScript enforces exhaustiveness — missing any key is a compile error with codegen active.
+
+---
+
+## Weighted splits
+
+By default, multivariate experiments split traffic evenly. To customize, use `assignment: "weighted"` with a `split`:
+
+```json
+{
+  "id": "cta-copy",
+  "assignment": "weighted",
+  "split": {
+    "buy-now": 40,
+    "get-started": 30,
+    "try-free": 20,
+    "start-trial": 5,
+    "shop-now": 5
+  },
+  "default": "buy-now",
+  "variants": [...]
+}
+```
+
+Rules:
+
+- All variant IDs must appear in `split`
+- Values are integer percentages
+- Must sum to exactly 100
+- Assignment is deterministic via sticky-hash of `(userId, experimentId)`
+
+### Staged rollout pattern
+
+Start with a heavy default:
+
+```json
+"split": { "buy-now": 90, "get-started": 5, "try-free": 5 }
+```
+
+Gradually shift traffic:
+
+```json
+"split": { "buy-now": 50, "get-started": 25, "try-free": 25 }
+```
+
+Until the winner is clear:
+
+```json
+"split": { "buy-now": 0, "get-started": 100, "try-free": 0 }
+```
+
+Then archive the experiment and bake the winner into the code.
+
+### Zero-percent variants
+
+A variant with `0` in the split still appears in config, still has types generated, but never gets assigned. Useful for:
+
+- Temporary kill switches on one arm
+- Preparing a variant before ramping it up
+
+---
+
+## Mutual exclusion
+
+When two experiments should never run on the same user:
+
+```json
+{ "id": "card-layout-a", "mutex": "card-layout", ... },
+{ "id": "card-layout-b", "mutex": "card-layout", ... },
+{ "id": "card-layout-c", "mutex": "card-layout", ... }
+```
+
+All three experiments with `mutex: "card-layout"` form a group. A single user is enrolled in **at most one** experiment from the group, chosen by stable hash.
+
+### Why?
+
+Competing experiments can interact:
+
+- Two card-layout experiments both redesign the card; which one won?
+- Two onboarding experiments both change flow; which did the user complete?
+
+Mutex groups prevent interactions by ensuring one wins per user.
+
+### Rules
+
+- All experiments in a mutex group must have the same targeting (or else the "winner" is unpredictable)
+- A user who matches multiple experiments gets exactly one, deterministically
+- The choice is stable across sessions (sticky-hash)
+- Debug overlay shows mutex status on each experiment card
+
+---
+
+## Statistical considerations
+
+variantlab does not ship analytics. But we have opinions about how to run multivariate tests.
+
+### Traffic cost
+
+With 5 variants and an even split, each variant gets 20% of your traffic. To reach statistical significance, you need 5x the users a 2-variant test would need.
+
+Rule of thumb: **fewer variants is better**. Only use multivariate if you're explicitly exploring a design space.
+
+### Winner selection
+
+Don't pick a winner until:
+
+- Each variant has at least 1000 exposures (per your metric)
+- The p-value is below 0.05 (your telemetry tool should compute this)
+- The effect is practically significant, not just statistically
+
+### Holdout arm
+
+Consider keeping a "control" variant at a fixed percentage (e.g., 10%) even after picking a winner. This lets you measure long-term effects vs. the original baseline.
+
+```json
+"split": {
+  "control": 10,
+  "new-winner": 90
+}
+```
+
+### Novelty effects
+
+Users react to any change. Measure at least 2 weeks post-launch, not just the first day.
+
+---
+
+## Real-world examples
+
+### Drishtikon card layout (30 variants)
+
+The origin story. See [`origin-story.md`](../research/origin-story.md).
+
+```json
+{
+  "id": "news-card-detailed-layout",
+  "type": "render",
+  "default": "responsive-image",
+  "assignment": "default",
+  "variants": [
+    { "id": "responsive-image", "label": "Responsive image" },
+    { "id": "no-scroll", "label": "No scroll" },
+    { "id": "tap-collapse", "label": "Tap to collapse" },
+    { "id": "drag-handle", "label": "Drag handle" },
+    { "id": "scale-to-fit", "label": "Scale to fit" },
+    { "id": "pip-thumbnail", "label": "Picture-in-picture" },
+    // ... 24 more
+  ]
+}
+```
+
+During development, `assignment: "default"` means everyone sees the default and the developer uses the debug overlay to switch manually. Once the best variant is identified, the experiment is archived and the winner is baked in.
+
+### Pricing experiment (4 variants)
+
+```json
+{
+  "id": "pro-monthly-price",
+  "type": "value",
+  "assignment": "weighted",
+  "split": {
+    "low": 25,
+    "mid-low": 25,
+    "mid-high": 25,
+    "high": 25
+  },
+  "default": "mid-low",
+  "variants": [
+    { "id": "low", "value": 4.99 },
+    { "id": "mid-low", "value": 7.99 },
+    { "id": "mid-high", "value": 9.99 },
+    { "id": "high", "value": 12.99 }
+  ],
+  "targeting": { "attributes": { "isNewUser": true } }
+}
+```
+
+Shows different prices to new users to find the optimum.
+
+### Onboarding flow (3 variants)
+
+```json
+{
+  "id": "onboarding-flow",
+  "type": "render",
+  "assignment": "weighted",
+  "split": {
+    "current": 50,
+    "3-step": 25,
+    "1-page": 25
+  },
+  "default": "current",
+  "mutex": "onboarding",
+  "variants": [
+    { "id": "current", "label": "Current flow" },
+    { "id": "3-step", "label": "3 steps" },
+    { "id": "1-page", "label": "Single page" }
+  ]
+}
+```
+
+Mutex ensures the onboarding test doesn't collide with a future onboarding experiment.
+
+---
+
+## Debugging multivariate experiments
+
+### Debug overlay
+
+Shows:
+
+- All variants of the experiment
+- Which one is currently active
+- Why (targeting, assignment, override)
+- A picker to switch manually
+
+### Eval CLI
+
+```bash
+variantlab eval experiments.json \
+  --experiment cta-copy \
+  --context '{"userId":"alice"}'
+```
+
+Output:
+
+```
+Experiment: cta-copy
+  Targeting: ✅ pass (no targeting)
+  Assignment: weighted
+  Variant: get-started (40% bucket)
+  Value: "Get started"
+```
+
+### Variant distribution check
+
+```bash
+variantlab distribution experiments.json \
+  --experiment cta-copy \
+  --users 10000
+```
+
+Output:
+
+```
+cta-copy distribution over 10000 simulated users:
+  buy-now:     4028  (40.3%)  expected: 40%  ✅
+  get-started: 3012  (30.1%)  expected: 30%  ✅
+  try-free:    1987  (19.9%)  expected: 20%  ✅
+  start-trial:  491  (4.9%)   expected: 5%   ✅
+  shop-now:     482  (4.8%)   expected: 5%   ✅
+```
+
+Useful for verifying the assignment logic is working correctly.
+
+---
+
+## See also
+
+- [`config-format.md`](../design/config-format.md) — the `split` and `mutex` fields
+- [`value-experiments.md`](./value-experiments.md) — when variants are values
+- [`targeting.md`](./targeting.md) — scoping multivariate tests