arkaos 4.26.0 → 4.27.0

package/departments/marketing/skills/ab-test/references/test-templates.md

@@ -0,0 +1,277 @@
+# A/B Test Templates Reference
+
+Templates for planning, documenting, and analyzing experiments.
+
+## Contents
+- Test Plan Template
+- Results Documentation Template
+- Test Repository Entry Template
+- Quick Test Brief Template
+- Stakeholder Update Template
+- Experiment Prioritization Scorecard
+- Hypothesis Bank Template
+
+## Test Plan Template
+
+```markdown
+# A/B Test: [Name]
+
+## Overview
+- **Owner**: [Name]
+- **Test ID**: [ID in testing tool]
+- **Page/Feature**: [What's being tested]
+- **Planned dates**: [Start] - [End]
+
+## Hypothesis
+
+Because [observation/data],
+we believe [change]
+will cause [expected outcome]
+for [audience].
+We'll know this is true when [metrics].
+
+## Test Design
+
+| Element | Details |
+|---------|---------|
+| Test type | A/B / A/B/n / MVT |
+| Duration | X weeks |
+| Sample size | X per variant |
+| Traffic allocation | 50/50 |
+| Tool | [Tool name] |
+| Implementation | Client-side / Server-side |
+
+## Variants
+
+### Control (A)
+[Screenshot]
+- Current experience
+- [Key details about current state]
+
+### Variant (B)
+[Screenshot or mockup]
+- [Specific change #1]
+- [Specific change #2]
+- Rationale: [Why we think this will win]
+
+## Metrics
+
+### Primary
+- **Metric**: [metric name]
+- **Definition**: [how it's calculated]
+- **Current baseline**: [X%]
+- **Minimum detectable effect**: [X%]
+
+### Secondary
+- [Metric 1]: [what it tells us]
+- [Metric 2]: [what it tells us]
+- [Metric 3]: [what it tells us]
+
+### Guardrails
+- [Metric that shouldn't get worse]
+- [Another safety metric]
+
+## Segment Analysis Plan
+- Mobile vs. desktop
+- New vs. returning visitors
+- Traffic source
+- [Other relevant segments]
+
+## Success Criteria
+- Winner: [Primary metric improves by X% with 95% confidence]
+- Loser: [Primary metric decreases significantly]
+- Inconclusive: [What we'll do if no significant result]
+
+## Pre-Launch Checklist
+- [ ] Hypothesis documented and reviewed
+- [ ] Primary metric defined and trackable
+- [ ] Sample size calculated
+- [ ] Test duration estimated
+- [ ] Variants implemented correctly
+- [ ] Tracking verified in all variants
+- [ ] QA completed on all variants
+- [ ] Stakeholders informed
+- [ ] Calendar hold for analysis date
+```
+
+---
+
+## Results Documentation Template
+
+```markdown
+# A/B Test Results: [Name]
+
+## Summary
+| Element | Value |
+|---------|-------|
+| Test ID | [ID] |
+| Dates | [Start] - [End] |
+| Duration | X days |
+| Result | Winner / Loser / Inconclusive |
+| Decision | [What we're doing] |
+
+## Hypothesis (Reminder)
+[Copy from test plan]
+
+## Results
+
+### Sample Size
+| Variant | Target | Actual | % of target |
+|---------|--------|--------|-------------|
+| Control | X | Y | Z% |
+| Variant | X | Y | Z% |
+
+### Primary Metric: [Metric Name]
+| Variant | Value | 95% CI | vs. Control |
+|---------|-------|--------|-------------|
+| Control | X% | [X%, Y%] | — |
+| Variant | X% | [X%, Y%] | +X% |
+
+**Statistical significance**: p = X.XX (95% = sig / not sig)
+**Practical significance**: [Is this lift meaningful for the business?]
+
+### Secondary Metrics
+
+| Metric | Control | Variant | Change | Significant? |
+|--------|---------|---------|--------|--------------|
+| [Metric 1] | X | Y | +Z% | Yes/No |
+| [Metric 2] | X | Y | +Z% | Yes/No |
+
+### Guardrail Metrics
+
+| Metric | Control | Variant | Change | Concern? |
+|--------|---------|---------|--------|----------|
+| [Metric 1] | X | Y | +Z% | Yes/No |
+
+### Segment Analysis
+
+**Mobile vs. Desktop**
+| Segment | Control | Variant | Lift |
+|---------|---------|---------|------|
+| Mobile | X% | Y% | +Z% |
+| Desktop | X% | Y% | +Z% |
+
+**New vs. Returning**
+| Segment | Control | Variant | Lift |
+|---------|---------|---------|------|
+| New | X% | Y% | +Z% |
+| Returning | X% | Y% | +Z% |
+
+## Interpretation
+
+### What happened?
+[Explanation of results in plain language]
+
+### Why do we think this happened?
+[Analysis and reasoning]
+
+### Caveats
+[Any limitations, external factors, or concerns]
+
+## Decision
+
+**Winner**: [Control / Variant]
+
+**Action**: [Implement variant / Keep control / Re-test]
+
+**Timeline**: [When changes will be implemented]
+
+## Learnings
+
+### What we learned
+- [Key insight 1]
+- [Key insight 2]
+
+### What to test next
+- [Follow-up test idea 1]
+- [Follow-up test idea 2]
+
+### Impact
+- **Projected lift**: [X% improvement in Y metric]
+- **Business impact**: [Revenue, conversions, etc.]
+```
+
+---
+
+## Test Repository Entry Template
+
+For tracking all tests in a central location:
+
+```markdown
+| Test ID | Name | Page | Dates | Primary Metric | Result | Lift | Link |
+|---------|------|------|-------|----------------|--------|------|------|
+| 001 | Hero headline test | Homepage | 1/1-1/15 | CTR | Winner | +12% | [Link] |
+| 002 | Pricing table layout | Pricing | 1/10-1/31 | Plan selection | Loser | -5% | [Link] |
+| 003 | Signup form fields | Signup | 2/1-2/14 | Completion | Inconclusive | +2% | [Link] |
+```
+
+---
+
+## Quick Test Brief Template
+
+For simple tests that don't need full documentation:
+
+```markdown
+## [Test Name]
+
+**What**: [One sentence description]
+**Why**: [One sentence hypothesis]
+**Metric**: [Primary metric]
+**Duration**: [X weeks]
+**Result**: [TBD / Winner / Loser / Inconclusive]
+**Learnings**: [Key takeaway]
+```
+
+---
+
+## Stakeholder Update Template
+
+```markdown
+## A/B Test Update: [Name]
+
+**Status**: Running / Complete
+**Days remaining**: X (or complete)
+**Current sample**: X% of target
+
+### Preliminary observations
+[What we're seeing - without making decisions yet]
+
+### Next steps
+[What happens next]
+
+### Timeline
+- [Date]: Analysis complete
+- [Date]: Decision and recommendation
+- [Date]: Implementation (if winner)
+```
+
+---
+
+## Experiment Prioritization Scorecard
+
+For deciding which tests to run:
+
+| Factor | Weight | Test A | Test B | Test C |
+|--------|--------|--------|--------|--------|
+| Potential impact | 30% | | | |
+| Confidence in hypothesis | 25% | | | |
+| Ease of implementation | 20% | | | |
+| Risk if wrong | 15% | | | |
+| Strategic alignment | 10% | | | |
+| **Total** | | | | |
+
+Scoring: 1-5 (5 = best)
+
+---
+
+## Hypothesis Bank Template
+
+For collecting test ideas:
+
+```markdown
+| ID | Page/Area | Observation | Hypothesis | Potential Impact | Status |
+|----|-----------|-------------|------------|------------------|--------|
+| H1 | Homepage | Low scroll depth | Shorter hero will increase scroll | High | Testing |
+| H2 | Pricing | Users compare plans | Comparison table will help | Medium | Backlog |
+| H3 | Signup | Drop-off at email | Social login will increase completion | Medium | Backlog |
+```