@variantlab/core 0.1.4 → 0.1.6

package/docs/phases/phase-4-advanced.md

@@ -0,0 +1,306 @@
+# Phase 4 — Advanced (v0.4)
+
+**Status**: Not started
+**Goal**: Ship the features that differentiate variantlab from every free tool and most paid ones.
+**Target version**: `0.4.0`
+
+## Table of contents
+
+- [Exit criteria](#exit-criteria)
+- [HMAC-signed configs](#hmac-signed-configs)
+- [Crash rollback persistence](#crash-rollback-persistence)
+- [Time-travel replay](#time-travel-replay)
+- [Statistical tools](#statistical-tools)
+- [Remote config improvements](#remote-config-improvements)
+- [Security hardening](#security-hardening)
+- [Milestones](#milestones)
+
+---
+
+## Exit criteria
+
+Phase 4 is done when:
+
+1. `variantlab sign` CLI command ships with tests
+2. Engine verifies signatures via Web Crypto API on all runtimes
+3. Crash rollback is persistent across sessions when configured
+4. Time travel "replay" mode works (read-only variant history replay)
+5. `variantlab distribution` grows into a proper stats tool
+6. SSE / WebSocket config push is GA (if phase 2 was experimental)
+7. A third-party security audit is initiated
+8. The devtools extension goes from beta to stable
+9. All phase 4 features are documented with examples
+
+---
+
+## HMAC-signed configs
+
+See [`hmac-signing.md`](../features/hmac-signing.md) for the full spec. Phase 4 ships:
+
+### Core
+
+- [ ] `verifySignature(config, key)` using Web Crypto API
+- [ ] Canonical JSON form per RFC 8785
+- [ ] Key rotation support (multiple keys, key ID prefix)
+- [ ] `onSignatureFailure: "reject" | "warn"` modes
+- [ ] Constant-time verification (from Web Crypto, not hand-rolled)
+- [ ] Tests covering: valid sig, invalid sig, missing sig, tampered body, wrong key
+
+### CLI
+
+- [ ] `variantlab sign <file> [--out <file>] [--key <env>]`
+- [ ] `variantlab verify <file> [--key <env>]`
+- [ ] `variantlab rotate-key` helper
+- [ ] Exit codes per docs
+
+### Integration
+
+- [ ] Example GitHub Actions workflow that signs on merge
+- [ ] Example S3 + CloudFront deployment guide
+- [ ] Example Cloudflare Workers deployment with KV key storage
+- [ ] Documentation of the threat model
+
+---
+
+## Crash rollback persistence
+
+See [`crash-rollback.md`](../features/crash-rollback.md). Phase 4 finalizes:
+
+### Core
+
+- [ ] Rollback state persisted to Storage
+- [ ] Rollback state cleared on config version bump
+- [ ] Rollback state cleared on manual reset
+- [ ] `persistent: "forever"` option (never auto-clear)
+- [ ] Engine emits `rollback` events with full context
+- [ ] `engine.getRollbacks()` returns all active rollbacks
+- [ ] `engine.clearRollback(id)` clears one
+
+### Adapters
+
+- [ ] `<VariantErrorBoundary>` refined with custom fallback
+- [ ] Automatic crash reporting forward to Telemetry
+- [ ] Native iOS crash hook (via NSException → JS)
+- [ ] Native Android crash hook (via UncaughtExceptionHandler → JS)
+- [ ] Integration with `sentry-expo` (user-supplied adapter)
+
+### Debug overlay
+
+- [ ] History tab shows rollback events prominently
+- [ ] "Clear rollback" button per experiment
+- [ ] Simulate rollback button (dev only) for testing
+
+---
+
+## Time-travel replay
+
+See [`time-travel.md`](../features/time-travel.md). Phase 4 ships the replay mode:
+
+### Core
+
+- [ ] `replaySession(history, newConfig)` pure function
+- [ ] Diff output: which experiments change assignment under new config
+- [ ] Export session as JSON
+- [ ] Import session from JSON
+- [ ] Persistent history (optional)
+
+### Debug overlay
+
+- [ ] Replay tab in the overlay
+- [ ] "Import session" file picker
+- [ ] Scrubber for time-based navigation
+- [ ] "Apply this state" button
+
+### Use cases
+
+- Regression testing configs before shipping
+- Reproducing user bug reports
+- Migration validation
+
+---
+
+## Statistical tools
+
+### `variantlab distribution` improvements
+
+- [ ] Bootstrap simulation (10,000 users by default)
+- [ ] Bucket deviation analysis
+- [ ] Chi-square test for assignment uniformity
+- [ ] JSON output for CI integration
+
+### `variantlab sample-size` — NEW
+
+Compute the sample size needed to detect a given effect size:
+
+```bash
+variantlab sample-size --baseline 0.10 --effect 0.02 --power 0.8
+```
+
+Output: minimum users per arm.
+
+### `variantlab power` — NEW
+
+Compute statistical power given observed sample size:
+
+```bash
+variantlab power --baseline 0.10 --variant 0.12 --users 1000
+```
+
+### Not shipping
+
+- [ ] ❌ No t-tests / z-tests on actual user data (we don't collect data)
+- [ ] ❌ No dashboard for "how is my experiment doing" (we have no data to show)
+- [ ] ❌ No auto-decision logic (that's a telemetry tool's job)
+
+We ship **pre-launch planning tools**, not runtime stats.
+
+---
+
+## Remote config improvements
+
+### SSE GA
+
+- [ ] Production-ready `createEventSourceFetcher`
+- [ ] Automatic reconnection with exponential backoff
+- [ ] Connection state in debug overlay
+- [ ] Tests against a test SSE server
+
+### Partial config updates
+
+- [ ] Support delta updates: server sends only changed experiments
+- [ ] Merge logic in the engine
+- [ ] Reduces bandwidth for frequent pushes
+
+### Config versioning
+
+- [ ] Engine tracks the config version hash
+- [ ] `If-None-Match` + ETag support in `createHttpFetcher`
+- [ ] 304 responses don't reload
+
+### Config caching
+
+- [ ] Pluggable cache backends (LRU in-memory, LocalStorage, AsyncStorage)
+- [ ] Cache TTL per config
+- [ ] Manual cache invalidation
+
+---
+
+## Security hardening
+
+### Third-party audit
+
+- [ ] Engage a security firm for an audit of the core engine
+- [ ] Focus areas: prototype pollution, HMAC verification, config parsing, DoS vectors
+- [ ] Publish the audit report
+
+### Fuzz testing
+
+- [ ] Integrate with OSS-Fuzz or a custom fuzzer
+- [ ] Fuzz the config parser, targeting evaluator, and HMAC verifier
+- [ ] Run fuzz tests in CI nightly
+
+### Supply chain
+
+- [ ] All releases signed with Sigstore
+- [ ] SBOMs published per release
+- [ ] Provenance attestation via GitHub Actions OIDC
+- [ ] SLSA level 3 compliance
+
+### CSP strict mode
+
+- [ ] Verify variantlab works in CSP-strict environments
+- [ ] Document CSP headers users should set
+- [ ] Tests that exercise CSP violations
+
+---
+
+## Milestones
+
+### M1: HMAC signing end-to-end
+
+- Signer
+- Verifier
+- CLI commands
+- Canonicalization
+- Tests
+- Deployment guide
+
+Gate: a signed config fetched from a CDN is verified on mobile and rejected if tampered.
+
+### M2: Crash rollback persistence
+
+- Storage integration
+- Error boundary refinement
+- Native crash hooks
+- Debug overlay integration
+- Tests
+
+Gate: a crashing variant auto-rolls back within 3 crashes; rollback survives app restart; a config update clears it.
+
+### M3: Time travel replay
+
+- Replay pure function
+- Diff output
+- Debug overlay UI
+- Tests
+
+Gate: a recorded session can be replayed against a different config and the diffs are accurate.
+
+### M4: Statistical tools
+
+- Distribution bootstrap
+- Sample size calculator
+- Power analysis
+- Tests
+
+Gate: sample-size output matches published statistical tables for common inputs.
+
+### M5: Security audit + hardening
+
+- Engage auditor
+- Address findings
+- Publish report
+- Fuzz integration
+
+Gate: audit finds no high/critical issues (or they're fixed before release).
+
+### M6: v0.4.0 release
+
+- All features shipped
+- Docs updated
+- Migration guide
+- Blog post
+
+---
+
+## Risks
+
+### Risk: HMAC canonicalization bugs
+
+Mitigation: use RFC 8785 and test against the published test vectors. Don't hand-roll canonicalization rules.
+
+### Risk: Persistent rollback never clears
+
+Mitigation: hard cap on persistent rollback duration (e.g., 7 days) unless explicitly opted in. Clear on config version bump.
+
+### Risk: Security audit surfaces architectural issues
+
+Mitigation: the earlier we audit, the cheaper it is to fix. Phase 4 is the right time because the API is stable but not yet v1.
+
+### Risk: Time travel replay doesn't match real behavior
+
+Mitigation: replay is read-only and runs the same evaluation code as the engine. If it diverges, that's a bug we fix.
+
+---
+
+## Transition to phase 5
+
+Phase 4 exits when:
+
+- All advanced features GA
+- Security audit complete
+- Devtools extension stable
+- v0.4.0 published
+- Community is using advanced features in production
+
+Phase 5 is the v1.0 stable release: API freeze, long-term support commitment, and polish. See [`phase-5-v1-stable.md`](./phase-5-v1-stable.md).

package/docs/phases/phase-5-v1-stable.md

@@ -0,0 +1,350 @@
+# Phase 5 — v1.0 Stable
+
+**Status**: Not started
+**Goal**: Freeze the public API, commit to long-term support, and polish for a stable release.
+**Target version**: `1.0.0`
+
+## Table of contents
+
+- [Exit criteria](#exit-criteria)
+- [API freeze](#api-freeze)
+- [LTS commitment](#lts-commitment)
+- [Polish](#polish)
+- [Marketing / launch](#marketing--launch)
+- [Governance](#governance)
+- [Milestones](#milestones)
+- [Post-1.0 roadmap](#post-10-roadmap)
+
+---
+
+## Exit criteria
+
+Phase 5 is done when:
+
+1. Every public API is documented with examples
+2. Every public API is type-safe and has test coverage > 95%
+3. Every adapter has a working example app in `examples/`
+4. All known bugs with labels `bug` and severity `high` are fixed
+5. The npm package metadata is complete (keywords, repository, homepage, author, license)
+6. Public docs site is live (optional — README is enough)
+7. Security audit findings are all resolved
+8. v1.0.0 tag is published
+9. A public announcement is made (blog post, HN, Reddit, Twitter)
+10. At least one case study from a non-Drishtikon user is documented
+
+---
+
+## API freeze
+
+The public API becomes stable at v1.0. Breaking changes require a major version bump and a formal RFC.
+
+### What "stable" means
+
+- **Types**: adding new fields to interfaces is fine. Removing or changing existing fields is breaking.
+- **Hooks**: the signature cannot change. New overloads can be added.
+- **Components**: the prop names cannot change. New props can be added.
+- **Config**: `version: 1` remains supported forever.
+- **Core engine**: the `VariantEngine` interface cannot change.
+
+### Deprecation policy
+
+- Deprecated APIs emit a console warning in dev builds
+- Deprecated APIs remain functional for at least one major version
+- Deprecations are documented in the changelog
+- Replacement APIs are documented in the deprecation notice
+
+### What's NOT stable
+
+- Internal APIs (modules under `@variantlab/core/internal`)
+- Debug overlay UI (can change look/feel in minor releases)
+- CLI output format (can improve in minor releases)
+- Error message text (but not error types)
+
+---
+
+## LTS commitment
+
+v1.0 comes with a Long-Term Support commitment:
+
+- **Security patches**: for the latest 2 major versions
+- **Bug fixes**: for the latest major version
+- **Performance improvements**: for the latest major version
+- **New features**: only in the latest major version
+
+### v1.x line
+
+- Minor versions (`1.1`, `1.2`, ...) — new features, backward-compatible
+- Patch versions (`1.0.1`, `1.0.2`, ...) — bug fixes only
+- Security patches for v1.x continue until v3.0 ships
+
+### Migration assurance
+
+Users on v1.x will never be forced to upgrade to v2.x. When v2 ships, we'll:
+
+1. Document the breaking changes
+2. Ship a `variantlab migrate` command
+3. Support v1.x for at least 12 months after v2 ships
+
+---
+
+## Polish
+
+### Documentation
+
+- [ ] Every exported function has a JSDoc comment with an example
+- [ ] Every type has a comment explaining its purpose
+- [ ] The `docs/` tree is navigable with a table of contents
+- [ ] `CHANGELOG.md` is up to date and formatted cleanly
+- [ ] `README.md` screenshots/GIFs are updated
+- [ ] Typos are fixed (we'll run a pass with a spell-checker)
+
+### Error messages
+
+- [ ] Every error message includes a docs link
+- [ ] Every error has a unique code (e.g., `E_CONFIG_INVALID`)
+- [ ] Error codes are documented in `docs/errors.md`
+- [ ] Errors suggest fixes when possible
+
+### Performance
+
+- [ ] Run benchmarks across all hot paths
+- [ ] Publish benchmark results in `docs/performance.md`
+- [ ] Compare against LaunchDarkly, Firebase, GrowthBook (where possible)
+- [ ] Target: core evaluation < 10 µs per experiment
+
+### Bundle size
+
+- [ ] Final size audit
+- [ ] Publish bundle size analysis for every package
+- [ ] Compare against competitors
+- [ ] Target: core < 3 KB gzipped
+
+### DX
+
+- [ ] IDE autocomplete works smoothly with codegen
+- [ ] TypeScript error messages are helpful, not cryptic
+- [ ] `variantlab init` produces a working project
+- [ ] Example apps are deployable with one command
+
+---
+
+## Marketing / launch
+
+### Pre-launch
+
+- [ ] Beta release (1.0.0-beta.1) shared with a small group
+- [ ] Collect feedback for 2-4 weeks
+- [ ] Fix issues
+- [ ] Release candidate (1.0.0-rc.1)
+
+### Launch
+
+- [ ] Blog post: "Introducing variantlab"
+- [ ] Hacker News post
+- [ ] Reddit: r/reactnative, r/webdev, r/javascript, r/typescript
+- [ ] Twitter/X thread
+- [ ] Product Hunt launch
+- [ ] Dev.to cross-post
+- [ ] Mailing list announcement (if we have one)
+
+### Materials
+
+- [ ] Logo (simple, recognizable)
+- [ ] Social card images
+- [ ] Demo video (< 2 minutes, shows the debug overlay in action)
+- [ ] Example gallery
+
+### Press targets (stretch)
+
+- [ ] React Newsletter
+- [ ] JavaScript Weekly
+- [ ] Bytes.dev
+- [ ] This Week in React
+
+### Anti-marketing
+
+Things we **won't** do:
+
+- ❌ Launch a mailing list to nag users
+- ❌ Buy ads
+- ❌ Sponsor influencers
+- ❌ Gate features behind "contact sales"
+- ❌ Claim unverified benchmarks
+
+---
+
+## Governance
+
+### Maintainers
+
+At v1.0, we formalize the maintainer list:
+
+- [ ] At least 2 maintainers with publish access (avoid bus factor of 1)
+- [ ] Maintainer responsibilities documented in `GOVERNANCE.md`
+- [ ] Code review policy: 1 approval for minor changes, 2 for breaking
+
+### Decision-making
+
+- Minor changes: maintainer discretion
+- Major changes: RFC process
+- Breaking changes: RFC + 2 maintainer approvals + community discussion
+
+### RFC process
+
+- RFCs filed as GitHub issues with the `rfc` label
+- Minimum 1 week open for comment
+- Final decision recorded in the issue
+- Implementation PR links back to the RFC
+
+### Code of conduct
+
+- [ ] Adopt the Contributor Covenant v2.1
+- [ ] Publish enforcement guidelines
+- [ ] Designate a CoC response team
+
+### Security policy
+
+- [ ] `SECURITY.md` specifies the reporting flow
+- [ ] Dedicated security contact (email or form)
+- [ ] 90-day disclosure window
+- [ ] CVE assignment for critical issues
+
+---
+
+## Milestones
+
+### M1: API freeze audit
+
+- Review every exported API
+- Remove internal APIs from the public surface
+- Document every public API
+- Tag release candidate
+
+### M2: Polish pass
+
+- Fix every `bug`-labeled issue
+- Improve error messages
+- Update docs
+
+### M3: Beta → RC
+
+- Ship beta to small group
+- Collect feedback
+- Fix issues
+- Ship RC
+
+### M4: Security audit wrap-up
+
+- Resolve any remaining audit findings
+- Publish audit report
+
+### M5: Launch
+
+- Publish v1.0.0
+- Blog post
+- Social posts
+- HN + Reddit
+- PH launch
+
+### M6: Post-launch
+
+- Monitor issues
+- Address launch-day bugs
+- Write retrospective
+
+---
+
+## Post-1.0 roadmap
+
+Ideas for v1.x and beyond, not committed:
+
+### v1.1: Quality of life
+
+- Better error messages
+- More example apps
+- More documentation
+- Community-contributed adapters
+
+### v1.2: Advanced targeting
+
+- Time-based interpolation (rollout curves)
+- Geolocation via adapter
+- Device capability targeting (GPU tier, RAM)
+
+### v1.3: Analytics integration
+
+- First-class adapters for PostHog, Mixpanel, Amplitude, Segment
+- Exposure deduplication
+- Funnel-aware targeting
+
+### v1.4: Collaborative sessions
+
+- Real-time multi-device sync (self-hosted)
+- Session sharing for design review
+- Multi-user debug overlay
+
+### v1.5: Cohort targeting
+
+- User cohorts defined as saved queries
+- Cross-experiment cohort reuse
+
+### v2.0 (speculative)
+
+- Schema v2 with breaking improvements informed by v1 usage
+- Formal migration tooling
+- Possible rename of some types if v1 names turned out to be wrong
+- Released only when v1 has clear limitations, not on a calendar
+
+### Things we'll never add
+
+- ❌ A hosted SaaS dashboard
+- ❌ Default telemetry
+- ❌ A full expression language in JSON
+- ❌ Proprietary features
+- ❌ Paid tiers
+
+---
+
+## The v1.0 contract
+
+At v1.0, we commit to our users:
+
+1. **The public API is stable.** Breaking changes require a major version.
+2. **Security patches continue for the latest 2 majors.**
+3. **No telemetry will ever be added without explicit opt-in.**
+4. **No paid tier will ever gate existing functionality.**
+5. **The license stays MIT.** Always.
+6. **The repo stays open source.** Always.
+7. **The code stays hand-reviewed.** No auto-merge bots touching source.
+8. **The size budgets stay.** If a feature can't fit, it doesn't ship.
+9. **The framework list keeps growing.** We accept reasonable framework adapter contributions.
+10. **Docs remain first-class.** Every feature ships with its spec.
+
+---
+
+## Definition of done
+
+Phase 5 is done when:
+
+- v1.0.0 is published on npm
+- The README, docs, and examples match reality
+- A user can go from `npm install @variantlab/react` to a working experiment in under 5 minutes
+- The project has survived its first launch-day traffic without breaking
+- The maintainer team is comfortable with ongoing stewardship
+
+When all 5 are true, variantlab is a mature, stable open source project. We shift to maintenance + incremental improvement mode.
+
+---
+
+## Anti-goals for phase 5
+
+Things we deliberately avoid at v1.0:
+
+- ❌ Shipping "just one more feature" that isn't in the phase plan
+- ❌ Rushing the release to hit an arbitrary date
+- ❌ Skipping the security audit
+- ❌ Over-promising on post-1.0 features
+- ❌ Hiring a community manager
+- ❌ Setting up a Discord (we use GitHub discussions)
+
+Quality > velocity at v1.0.