@ak--47/dungeon-master 1.5.3 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/analyze-soup/SKILL.md +21 -12
- package/.claude/skills/create-dungeon/SKILL.md +65 -5
- package/.claude/skills/create-project/SKILL.md +96 -0
- package/.claude/skills/create-project/provision.mjs +336 -0
- package/.claude/skills/verify-dungeon/SKILL.md +56 -25
- package/.claude/skills/verify-dungeon/references/counting-semantics.md +26 -6
- package/.claude/skills/verify-dungeon/references/report-format.md +14 -8
- package/.claude/skills/write-hooks/SKILL.md +156 -18
- package/CHANGELOG.md +211 -0
- package/HOOKS.md +566 -112
- package/README.md +28 -9
- package/docs/guides/1.3.0-upgrade-guide.md +262 -0
- package/docs/guides/1.3.1-upgrade-guide.md +328 -0
- package/docs/guides/1.4.0-upgrade-guide.md +130 -0
- package/docs/guides/1.4.1-upgrade-guide.md +125 -0
- package/docs/guides/1.4.3-upgrade-guide.md +200 -0
- package/docs/guides/1.4.4-upgrade-guide.md +95 -0
- package/docs/guides/1.5.0-upgrade-guide.md +426 -0
- package/docs/guides/1.5.1-upgrade-guide.md +178 -0
- package/docs/guides/1.5.3-upgrade-guide.md +117 -0
- package/docs/guides/1.6.0-upgrade-guide.md +204 -0
- package/dungeons/technical/stories-verify.js +152 -0
- package/dungeons/vertical/README.md +154 -0
- package/dungeons/vertical/ai-platform/ai-platform.js +1424 -0
- package/dungeons/vertical/ai-platform/ai-platform.sql +184 -0
- package/dungeons/vertical/ai-platform/ai-platform.verify.mjs +71 -0
- package/dungeons/vertical/community/community.js +1589 -0
- package/dungeons/vertical/community/community.sql +216 -0
- package/dungeons/vertical/community/community.verify.mjs +72 -0
- package/dungeons/vertical/crypto/crypto.js +1483 -0
- package/dungeons/vertical/crypto/crypto.sql +194 -0
- package/dungeons/vertical/crypto/crypto.verify.mjs +78 -0
- package/dungeons/vertical/dating/dating.js +1540 -0
- package/dungeons/vertical/dating/dating.sql +288 -0
- package/dungeons/vertical/dating/dating.verify.mjs +72 -0
- package/dungeons/vertical/devtools/devtools.js +1524 -0
- package/dungeons/vertical/devtools/devtools.sql +161 -0
- package/dungeons/vertical/devtools/devtools.verify.mjs +73 -0
- package/dungeons/vertical/ecommerce/ecommerce.js +1480 -0
- package/dungeons/vertical/ecommerce/ecommerce.sql +218 -0
- package/dungeons/vertical/ecommerce/ecommerce.verify.mjs +70 -0
- package/dungeons/vertical/education/education.js +1763 -0
- package/dungeons/vertical/education/education.sql +243 -0
- package/dungeons/vertical/education/education.verify.mjs +75 -0
- package/dungeons/vertical/fintech/fintech.js +1410 -0
- package/dungeons/vertical/fintech/fintech.sql +175 -0
- package/dungeons/vertical/fintech/fintech.verify.mjs +70 -0
- package/dungeons/vertical/fitness/fitness.js +1495 -0
- package/dungeons/vertical/fitness/fitness.sql +206 -0
- package/dungeons/vertical/fitness/fitness.verify.mjs +71 -0
- package/dungeons/vertical/food-delivery/food-delivery.js +1641 -0
- package/dungeons/vertical/food-delivery/food-delivery.sql +183 -0
- package/dungeons/vertical/food-delivery/food-delivery.verify.mjs +75 -0
- package/dungeons/vertical/gaming/gaming.js +1741 -0
- package/dungeons/vertical/gaming/gaming.sql +249 -0
- package/dungeons/vertical/gaming/gaming.verify.mjs +70 -0
- package/dungeons/vertical/healthcare/healthcare.js +1488 -0
- package/dungeons/vertical/healthcare/healthcare.sql +193 -0
- package/dungeons/vertical/healthcare/healthcare.verify.mjs +72 -0
- package/dungeons/vertical/insurance-application/insurance-application.js +1852 -0
- package/dungeons/vertical/insurance-application/insurance-application.sql +174 -0
- package/dungeons/vertical/insurance-application/insurance-application.verify.mjs +74 -0
- package/dungeons/vertical/logistics/logistics.js +1541 -0
- package/dungeons/vertical/logistics/logistics.sql +205 -0
- package/dungeons/vertical/logistics/logistics.verify.mjs +73 -0
- package/dungeons/vertical/marketplace/marketplace.js +1438 -0
- package/dungeons/vertical/marketplace/marketplace.sql +148 -0
- package/dungeons/vertical/marketplace/marketplace.verify.mjs +74 -0
- package/dungeons/vertical/{media.js → media/media.js} +557 -58
- package/dungeons/vertical/media/media.sql +173 -0
- package/dungeons/vertical/media/media.verify.mjs +74 -0
- package/dungeons/vertical/real-estate/real-estate.js +1629 -0
- package/dungeons/vertical/real-estate/real-estate.sql +167 -0
- package/dungeons/vertical/real-estate/real-estate.verify.mjs +75 -0
- package/dungeons/vertical/sass/sass.js +1733 -0
- package/dungeons/vertical/sass/sass.sql +173 -0
- package/dungeons/vertical/sass/sass.verify.mjs +74 -0
- package/dungeons/vertical/social/social.js +1456 -0
- package/dungeons/vertical/social/social.sql +160 -0
- package/dungeons/vertical/social/social.verify.mjs +76 -0
- package/dungeons/vertical/streaming/streaming.js +677 -0
- package/dungeons/vertical/streaming/streaming.sql +101 -0
- package/dungeons/vertical/streaming/streaming.verify.mjs +75 -0
- package/dungeons/vertical/support-desk/support-desk.js +741 -0
- package/dungeons/vertical/support-desk/support-desk.sql +137 -0
- package/dungeons/vertical/support-desk/support-desk.verify.mjs +75 -0
- package/dungeons/vertical/travel/travel.js +1603 -0
- package/dungeons/vertical/travel/travel.sql +236 -0
- package/dungeons/vertical/travel/travel.verify.mjs +72 -0
- package/index.js +2 -2
- package/lib/core/config-validator.js +37 -2
- package/lib/core/context.js +4 -2
- package/lib/core/dungeon-loader.js +20 -15
- package/lib/core/dungeon-to-json.js +22 -4
- package/lib/core/storage.js +3 -3
- package/lib/generators/funnels.js +36 -9
- package/lib/hook-helpers/cohort.js +53 -0
- package/lib/hook-helpers/index.js +8 -0
- package/lib/hook-helpers/mutate.js +1 -1
- package/lib/hook-helpers/shape.js +319 -0
- package/lib/hook-patterns/aggregate-per-user-by-bin.js +13 -5
- package/lib/hook-patterns/attributed-by-source.js +60 -58
- package/lib/hook-patterns/frequency-by-frequency.js +14 -4
- package/lib/hook-patterns/funnel-frequency-breakdown.js +24 -12
- package/lib/hook-patterns/index.js +1 -1
- package/lib/hook-patterns/time-to-convert-by-segment.js +75 -13
- package/lib/orchestrators/user-loop.js +41 -0
- package/lib/templates/story-spec.schema.json +175 -0
- package/lib/utils/json-evaluator.js +1 -1
- package/lib/utils/utils.js +5 -4
- package/lib/verify/coerce.js +186 -0
- package/lib/verify/counting.js +270 -32
- package/lib/verify/emulate-breakdown.js +1285 -203
- package/lib/verify/first-time.js +89 -0
- package/lib/verify/flows.js +679 -0
- package/lib/verify/formula.js +259 -0
- package/lib/verify/funnel-engine.js +697 -113
- package/lib/verify/index.js +32 -4
- package/lib/verify/sessionize.js +275 -0
- package/lib/verify/story-runner.js +483 -0
- package/lib/verify/verify-dungeon.js +59 -37
- package/package.json +3 -2
- package/scripts/extract-dungeon-schema.mjs +2 -2
- package/scripts/run-many.mjs +2 -2
- package/scripts/verify-runner.mjs +2 -2
- package/scripts/verify-stories.mjs +244 -0
- package/types.d.ts +296 -30
- package/dungeons/vertical/ai-platform.js +0 -818
- package/dungeons/vertical/community.js +0 -966
- package/dungeons/vertical/crypto.js +0 -895
- package/dungeons/vertical/dating.js +0 -830
- package/dungeons/vertical/devtools.js +0 -1050
- package/dungeons/vertical/ecommerce.js +0 -811
- package/dungeons/vertical/education.js +0 -964
- package/dungeons/vertical/fintech.js +0 -946
- package/dungeons/vertical/fitness.js +0 -935
- package/dungeons/vertical/food-delivery.js +0 -934
- package/dungeons/vertical/gaming.js +0 -1120
- package/dungeons/vertical/healthcare.js +0 -945
- package/dungeons/vertical/insurance-application.js +0 -1039
- package/dungeons/vertical/logistics.js +0 -986
- package/dungeons/vertical/marketplace.js +0 -1019
- package/dungeons/vertical/real-estate.js +0 -872
- package/dungeons/vertical/sass.js +0 -995
- package/dungeons/vertical/social.js +0 -904
- package/dungeons/vertical/travel.js +0 -930
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: verify-dungeon
|
|
3
|
-
description: Use when a dungeon's hooks need verification before pushing data to Mixpanel — runs the dungeon at full scale, validates schema integrity (catches flag-stamping),
|
|
4
|
-
argument-hint: [dungeon path(s), e.g. dungeons/gaming.js or dungeons/fintech.js]
|
|
3
|
+
description: Use when a dungeon's hooks need verification before pushing data to Mixpanel — runs the dungeon at full scale, evaluates the dungeon's `stories` export with the mechanical story runner (verify-stories.mjs), validates schema integrity (catches flag-stamping), and reports per-hook NAILED/STRONG/WEAK/NONE/INVERSE verdicts. LLM investigation is reserved for failures and duckdb-type assertions.
|
|
4
|
+
argument-hint: [dungeon path(s), e.g. dungeons/vertical/gaming/gaming.js or dungeons/vertical/fintech/fintech.js]
|
|
5
5
|
model: claude-opus-4-6
|
|
6
6
|
effort: max
|
|
7
7
|
---
|
|
8
8
|
|
|
9
9
|
# Verify Dungeon
|
|
10
10
|
|
|
11
|
-
Verify a dungeon at full scale: validate schema integrity,
|
|
11
|
+
Verify a dungeon at full scale: run the story runner (`scripts/verify-stories.mjs`) as the primary mechanical check, validate schema integrity, investigate only what the runner can't settle (failures + `duckdb`-type assertions), write a single consolidated `hook-results.md`.
|
|
12
12
|
|
|
13
|
-
**Dungeon file(s):** `$ARGUMENTS` — single path, multiple space-separated paths, or glob pattern. In batch mode, process each dungeon sequentially through Steps 1–
|
|
13
|
+
**Dungeon file(s):** `$ARGUMENTS` — single path, multiple space-separated paths, or glob pattern. In batch mode, process each dungeon sequentially through Steps 1–5, then write one consolidated report in Step 7. Use a unique `name` prefix per dungeon (e.g., `verify-fintech`, `verify-gaming`) so output files don't collide.
|
|
14
14
|
|
|
15
15
|
## Reference files
|
|
16
16
|
|
|
@@ -28,9 +28,10 @@ Also: `HOOKS.md` (recipe encyclopedia) and `types.d.ts` (source of truth for hoo
|
|
|
28
28
|
|
|
29
29
|
Read the dungeon at `$ARGUMENTS`. If it's a bare filename (no `/`), check `dungeons/`. Find:
|
|
30
30
|
|
|
31
|
-
1. **The `
|
|
32
|
-
2. **The
|
|
33
|
-
3. **
|
|
31
|
+
1. **The `stories` named export** — the machine-checkable contract (see `lib/templates/story-spec.schema.json` and the `DungeonStory` typedef in `types.d.ts`). If present, the story runner in Step 3 does the heavy lifting. If absent (legacy dungeon), the full per-hook flow in Step 5 applies to every hook.
|
|
32
|
+
2. **The `hook:` function** — read the full body
|
|
33
|
+
3. **The documentation comment block** describing engineered patterns
|
|
34
|
+
4. **Module-level closure state** (Map / Set / tracking variables defined outside the hook function but used inside)
|
|
34
35
|
|
|
35
36
|
For each hook/pattern, catalog:
|
|
36
37
|
- Hook number and name (e.g., "Hook #1: Compass users have 3x quest completion")
|
|
@@ -52,7 +53,7 @@ node scripts/verify-runner.mjs <dungeon-path> <run-name>
|
|
|
52
53
|
|
|
53
54
|
Example:
|
|
54
55
|
```bash
|
|
55
|
-
node scripts/verify-runner.mjs dungeons/vertical/gaming.js verify-gaming
|
|
56
|
+
node scripts/verify-runner.mjs dungeons/vertical/gaming/gaming.js verify-gaming
|
|
56
57
|
```
|
|
57
58
|
|
|
58
59
|
Full-fidelity runs can take minutes (50K+ user dungeons). Plan accordingly — kick off the run, do other reading, return when the file lands. If a run takes longer than your budget, report that as a finding ("dungeon too large to verify in current session") rather than falling back to `--small`.
|
|
@@ -65,7 +66,27 @@ Full-fidelity runs can take minutes (50K+ user dungeons). Plan accordingly — k
|
|
|
65
66
|
|
|
66
67
|
For batched output (>2M events), the runner writes `<run-name>-EVENTS-part-*.json`. See [sql-recipes.md "Multi-part EVENTS files"](references/sql-recipes.md#multi-part-events-files-batch-mode).
|
|
67
68
|
|
|
68
|
-
### Step 3:
|
|
69
|
+
### Step 3: Run the story runner (primary mechanical check)
|
|
70
|
+
|
|
71
|
+
For any dungeon with a `stories` export, the runner — not the LLM — computes the verdicts:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
node scripts/verify-stories.mjs <dungeon-path> --data-prefix <run-name>
|
|
75
|
+
node scripts/verify-stories.mjs <dungeon-path> --data-prefix <run-name> --json # machine-readable, for hook-results.md
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
The runner streams the shards from Step 2, evaluates every assertion against its declared `target` / `floor` / `minCohort`, substitutes `{{PREFIX}}` into `duckdb`-type assertions and shells them out, enforces hook coverage (every numbered hook in the HOOK STORIES comment block must be targeted by at least one story), and prints a five-tier verdict table. Exit code is non-zero when any story lands WEAK / NONE / INVERSE or coverage is incomplete.
|
|
79
|
+
|
|
80
|
+
**Verdicts are computed, not judged.** They include the population floor: a cohort smaller than the assertion's `minCohort` caps at WEAK — a 12-user cohort can no longer score NAILED regardless of how clean its ratio looks. See [report-format.md "Verdict criteria"](references/report-format.md#verdict-criteria-5-tier) for the mechanical definitions.
|
|
81
|
+
|
|
82
|
+
**What the LLM investigates after this step — and nothing else:**
|
|
83
|
+
1. **Stories below STRONG** (WEAK / NONE / INVERSE) — root-cause via Step 5's decision table. A miss means fixing the hook or the assertion's derivation, never relaxing the number to match output.
|
|
84
|
+
2. **`duckdb`-type assertions** — the runner executes them but can't interpret bespoke shapes; sanity-check their output against the story narrative.
|
|
85
|
+
3. **Dungeons without a `stories` export** — legacy fallback: full per-hook flow (Step 5) for every documented hook.
|
|
86
|
+
|
|
87
|
+
Do NOT re-derive verdicts the runner already computed as passing. `hook-results.md` (Step 7) renders the runner's JSON.
|
|
88
|
+
|
|
89
|
+
### Step 4: Validate schema (BEFORE per-hook checks)
|
|
69
90
|
|
|
70
91
|
Catches hooks that introduce undeclared columns (flag stamping). For each unique event type, compare actual columns against config-declared properties. See [sql-recipes.md "Schema validation queries"](references/sql-recipes.md#schema-validation-queries) for the SQL and the expected-schema source table.
|
|
71
92
|
|
|
@@ -75,7 +96,9 @@ Catches hooks that introduce undeclared columns (flag stamping). For each unique
|
|
|
75
96
|
|
|
76
97
|
If any event type has SCHEMA-FAIL, flag it prominently in the report header with specific remediation: which hook line adds the property and how to remove it while preserving the intended pattern.
|
|
77
98
|
|
|
78
|
-
### Step
|
|
99
|
+
### Step 5: Investigate failures (and legacy no-stories dungeons)
|
|
100
|
+
|
|
101
|
+
Applies only to the investigation targets from Step 3 — failing stories, `duckdb`-type assertions, and dungeons with no `stories` export.
|
|
79
102
|
|
|
80
103
|
**Decision: emulator vs DuckDB**
|
|
81
104
|
|
|
@@ -84,14 +107,22 @@ If any event type has SCHEMA-FAIL, flag it prominently in the report header with
|
|
|
84
107
|
| Funnel completion / step conversion | `emulateBreakdown({type: 'funnelFrequency'})` |
|
|
85
108
|
| Insights frequency-distribution | `emulateBreakdown({type: 'frequencyByFrequency'})` |
|
|
86
109
|
| Avg(prop) by per-user count(B) | `emulateBreakdown({type: 'aggregatePerUser'})` |
|
|
87
|
-
| Funnel TTC by segment | `emulateBreakdown({type: 'timeToConvert'})` |
|
|
110
|
+
| Funnel TTC by segment (steps-based or funnel-based) | `emulateBreakdown({type: 'timeToConvert'})` |
|
|
88
111
|
| First/last touch attribution | `emulateBreakdown({type: 'attributedBy'})` |
|
|
89
|
-
| Retention curves | `emulateBreakdown({type: 'retention'})` |
|
|
112
|
+
| Retention curves — birth or compounded | `emulateBreakdown({type: 'retention'})` (`compounded: true` for "DAU coming back") |
|
|
90
113
|
| Per-session metrics | `emulateBreakdown({type: 'sessionMetrics'})` |
|
|
114
|
+
| Lifecycle (new / retained / resurrected / dormant) | `emulateBreakdown({type: 'lifecycle'})` |
|
|
115
|
+
| Flows / top paths (Sankey) | `emulateBreakdown({type: 'topPaths'})` |
|
|
116
|
+
| Event totals segmented by property | `emulateBreakdown({type: 'eventBreakdown'})` (`countType: 'general' \| 'sessions'`) |
|
|
117
|
+
| Uniques per segment | `emulateBreakdown({type: 'uniques'})` |
|
|
118
|
+
| COUNT DISTINCT of a property + top values | `emulateBreakdown({type: 'distinctCount'})` |
|
|
119
|
+
| Ratio / composite metrics (conversion %, ARPU, blends) | `evaluateFormula` (`lib/verify/formula.js`) over emulator rows |
|
|
91
120
|
| Schema integrity / column coverage | DuckDB |
|
|
92
121
|
| Identity-model invariants | DuckDB |
|
|
93
122
|
| Experiment invariants | DuckDB |
|
|
94
|
-
|
|
|
123
|
+
| True bespokes (no emulator analysis fits) | DuckDB |
|
|
124
|
+
|
|
125
|
+
The emulator now covers lifecycle, flows, sessions, event breakdowns, formulas, and compounded retention — DuckDB's remit is schema / identity / experiment invariants plus true bespoke shapes. If a "bespoke" check is really a funnel, frequency, path, or breakdown in disguise, it belongs in the emulator.
|
|
95
126
|
|
|
96
127
|
**Hand-written DuckDB funnel SQL diverges from Mixpanel — never hand-roll.** If you find yourself writing `WITH step1 AS ..., step2 AS ...` for a funnel, STOP — use `emulateBreakdown` with `funnelFrequency` instead.
|
|
97
128
|
|
|
@@ -107,18 +138,18 @@ For emulator details, identity-model dungeons (must pass `profiles`), and time-s
|
|
|
107
138
|
**Everything about a dungeon lives in its folder.** When the dungeon being
|
|
108
139
|
verified is a user dungeon at `dungeons/user/<name>/<name>.js`, write ALL
|
|
109
140
|
generated artifacts into `dungeons/user/<name>/`:
|
|
110
|
-
- `hook-results.md` (Step
|
|
111
|
-
- `hook-query-log.txt` (Step
|
|
112
|
-
- `<name>-verifications.sql` (Step
|
|
141
|
+
- `hook-results.md` (Step 7)
|
|
142
|
+
- `hook-query-log.txt` (Step 6)
|
|
143
|
+
- `<name>-verifications.sql` (Step 7b)
|
|
113
144
|
|
|
114
145
|
The ONLY exception is the throwaway verification data the run writes to
|
|
115
146
|
`./data/` (`verify-*` event/user files) — that stays in `./data/` and is
|
|
116
|
-
deleted in Step
|
|
147
|
+
deleted in Step 8.
|
|
117
148
|
|
|
118
149
|
For non-user dungeons (technical/vertical) or batch runs across many dungeons,
|
|
119
150
|
fall back to `./research/` for `hook-results.md` / `hook-query-log.txt`.
|
|
120
151
|
|
|
121
|
-
### Step
|
|
152
|
+
### Step 6: Stash query log
|
|
122
153
|
|
|
123
154
|
Write every DuckDB query execution to `hook-query-log.txt`:
|
|
124
155
|
- **User dungeon:** always write to `dungeons/user/<name>/hook-query-log.txt`.
|
|
@@ -126,21 +157,21 @@ Write every DuckDB query execution to `hook-query-log.txt`:
|
|
|
126
157
|
|
|
127
158
|
Format and conventions: see [report-format.md "Query log format"](references/report-format.md#query-log-format).
|
|
128
159
|
|
|
129
|
-
### Step
|
|
160
|
+
### Step 7: Write `hook-results.md`
|
|
130
161
|
|
|
131
|
-
Write to `dungeons/user/<name>/hook-results.md` for a user dungeon, else `./research/hook-results.md`. Use the templates in [report-format.md](references/report-format.md):
|
|
162
|
+
Write to `dungeons/user/<name>/hook-results.md` for a user dungeon, else `./research/hook-results.md`. For story-backed dungeons, **the report renders the runner's JSON** (`verify-stories.mjs --json`): the hook summary table comes straight from the runner's per-story verdicts (story id, hook, archetype, observed vs target, verdict), and detailed blocks exist only for the Step-3 investigation targets. Use the templates in [report-format.md](references/report-format.md):
|
|
132
163
|
- Single-dungeon report structure
|
|
133
164
|
- Multi-dungeon report structure (when batch mode)
|
|
134
165
|
- Per-hook detail block
|
|
135
|
-
- Verdict criteria (5-tier)
|
|
166
|
+
- Verdict criteria (5-tier, mechanical)
|
|
136
167
|
|
|
137
168
|
**Order failures first** within each dungeon section: INVERSE → NONE → WEAK → STRONG → NAILED. Sort the summary table the same way. Actionable issues at the top.
|
|
138
169
|
|
|
139
|
-
### Step
|
|
170
|
+
### Step 7b: Write verification SQL (mandatory for user dungeons)
|
|
140
171
|
|
|
141
|
-
When verifying a dungeon in `dungeons/user/`, also write a standalone DuckDB SQL file alongside the dungeon in its folder at `dungeons/user/<name>/<name>-verifications.sql`. Vertical dungeons already have their SQL
|
|
172
|
+
When verifying a dungeon in `dungeons/user/`, also write a standalone DuckDB SQL file alongside the dungeon in its folder at `dungeons/user/<name>/<name>-verifications.sql`. Vertical dungeons already have their SQL co-located at `dungeons/vertical/<name>/<name>.sql`. Format: see [report-format.md "Verification SQL file"](references/report-format.md#verification-sql-file-mandatory-for-user-dungeons).
|
|
142
173
|
|
|
143
|
-
### Step
|
|
174
|
+
### Step 8: Cleanup
|
|
144
175
|
|
|
145
176
|
```bash
|
|
146
177
|
rm -f ./data/verify-* ./verify-*
|
|
@@ -171,7 +202,7 @@ Tell the user:
|
|
|
171
202
|
1. Report path: `dungeons/user/<name>/hook-results.md` (user dungeon) or `./research/hook-results.md`
|
|
172
203
|
2. Verification SQL path (for user dungeons): `dungeons/user/<name>/<name>-verifications.sql`
|
|
173
204
|
3. Query log path (if written): alongside the report (`dungeons/user/<name>/hook-query-log.txt`, else `./research/hook-query-log.txt`)
|
|
174
|
-
4.
|
|
205
|
+
4. Verdict counts from the story runner (per dungeon if batch mode), plus which stories needed LLM investigation
|
|
175
206
|
5. One-line summary of the most interesting finding
|
|
176
207
|
|
|
177
208
|
If hooks failed, note that `hook-results.md` can be used as context for fixing them: "read hook-results.md and fix the failing hooks in <dungeon-file>".
|
|
@@ -14,13 +14,10 @@ Mixpanel does NOT count the way naive SQL does. The verifier (and any DuckDB que
|
|
|
14
14
|
| Sessions | 3-trigger split: timeout `>`, max duration `>`, day-idx change (`session_query.cpp:906-911`) | Trust pre-stamped `session_id`; group by `(user, session_id)` |
|
|
15
15
|
| Retention | Birth-anchored, ms-strict gate (default `birth_can_retain=false` → `<`), bucketed by `floor((ret−birth)/unit)` (`retention_query.cpp:1097-1109,1228-1231`) | Use `emulateBreakdown` with `retention` |
|
|
16
16
|
|
|
17
|
-
**Known divergences from Mixpanel C++** (1.
|
|
17
|
+
**Known divergences from Mixpanel C++** (1.6.0):
|
|
18
18
|
- `countDistinctPeriods` default = `algorithm: 'calendar'` (UTC bucket).
|
|
19
19
|
Mixpanel C++ (`addiction_query.cpp:359`) uses ROLLING window. Pass
|
|
20
20
|
`algorithm: 'rolling'` for exact Frequency-Distribution parity.
|
|
21
|
-
- COMPOUNDED retention is NOT implemented — verifier silently ignores
|
|
22
|
-
`compounded: true`. Use DuckDB or query Mixpanel directly for "DAU
|
|
23
|
-
coming back" reports.
|
|
24
21
|
- Touchpoint sampling: generator stamps uniform-random across user
|
|
25
22
|
lifetime; verifier reads last-N before conversion (matches C++).
|
|
26
23
|
For users with ≤10 attribution events lifetime, no divergence.
|
|
@@ -31,7 +28,7 @@ Full rules: see [HOOKS.md Section 2](../../../../HOOKS.md#2-how-mixpanel-counts-
|
|
|
31
28
|
|
|
32
29
|
## When to use the emulator vs DuckDB
|
|
33
30
|
|
|
34
|
-
The emulator (`emulateBreakdown` from `@ak--47/dungeon-master/verify`) implements Mixpanel's rules natively. **ALWAYS use the emulator for funnel, frequency, aggregate, TTC, and
|
|
31
|
+
The emulator (`emulateBreakdown` from `@ak--47/dungeon-master/verify`) implements Mixpanel's rules natively. **ALWAYS use the emulator for funnel, frequency, aggregate, TTC, attribution, retention, lifecycle, flows, sessions, and event-breakdown patterns.** Hand-written DuckDB queries for these pattern types diverge from what Mixpanel shows in reports — even when they look correct.
|
|
35
32
|
|
|
36
33
|
Use DuckDB ONLY for:
|
|
37
34
|
- Schema integrity checks (column coverage, flag detection)
|
|
@@ -51,10 +48,33 @@ If you find yourself writing `WITH step1 AS ..., step2 AS ...` for a funnel, STO
|
|
|
51
48
|
| Funnel TTC by user property | `timeToConvert` | "Trial users take 4× longer than enterprise" |
|
|
52
49
|
| First/last touch attribution | `attributedBy` | "Conversions by Source" |
|
|
53
50
|
| Birth retention curves | `retention` | "Sign Up → Login on day N" — requires `cohortEvent`, `returnEvent`, `dayBuckets` |
|
|
51
|
+
| Compounded retention ("DAU coming back") | `retention` + `compounded: true` | Sets `returnEvent := cohortEvent` internally; throws on a conflicting `returnEvent` |
|
|
54
52
|
| Per-session metrics | `sessionMetrics` | Count / duration / events distributions per session |
|
|
53
|
+
| Lifecycle states (new / retained / resurrected / dormant) | `lifecycle` | "Resurrection wave in week N" — per-period user-state classification |
|
|
54
|
+
| Flows / top paths | `topPaths` | "What do users do after X" — Sankey top-path shares |
|
|
55
|
+
| Event totals segmented by property | `eventBreakdown` | "Purchases by plan tier" — `countType: 'general' \| 'sessions'`; list props fan out per item |
|
|
56
|
+
| Uniques per segment | `uniques` | "Unique users by plan tier" |
|
|
57
|
+
| Distinct property values | `distinctCount` | "How many distinct SKUs" — see below |
|
|
58
|
+
| Ratio / composite metrics | `evaluateFormula` (`lib/verify/formula.js`) | Conversion %, ARPU, any PEMDAS formula over emulator series |
|
|
55
59
|
|
|
56
60
|
Cross-cutting on EVERY type: `timeBucket: 'day' | 'week' | 'month'` partitions events into UTC buckets and emits one row per period.
|
|
57
61
|
|
|
62
|
+
### `distinctCount`
|
|
63
|
+
|
|
64
|
+
Counts distinct values of a flat event property, Mixpanel COUNT_DISTINCT-style, with a top-values breakdown:
|
|
65
|
+
|
|
66
|
+
```js
|
|
67
|
+
emulateBreakdown(events, {
|
|
68
|
+
type: 'distinctCount',
|
|
69
|
+
property: 'sku', // required — flat property name (dot-paths not supported)
|
|
70
|
+
event: 'Purchase', // optional — restrict to one event type; omit = all events
|
|
71
|
+
topN: 25, // optional — top values by count (default 25)
|
|
72
|
+
});
|
|
73
|
+
// → [{ distinct_count: 143, top_values: [{ value: 'SKU-001', count: 812 }, ...] }]
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
Single-row result. Use it for catalog-breadth stories ("power sellers list 5× more distinct SKUs") where `uniques` / `eventBreakdown` answer a different question (users or totals per bucket, not distinct values of the property itself).
|
|
77
|
+
|
|
58
78
|
Quick emulator script:
|
|
59
79
|
|
|
60
80
|
```js
|
|
@@ -98,7 +118,7 @@ For CI-style assertions, use `verifyDungeon` with a checks array; see `tests/e2e
|
|
|
98
118
|
| `readFileSync` ENOMEM on shards >500MB | Node string cap at ~512MB | Stream-load with `readline.createInterface` over `data/PREFIX-EVENTS*.json` glob |
|
|
99
119
|
| Hook reads `e.event === 'login'` but cohort empty | `login` is a funnel-step event auto-promoted to `isStrictEvent: true` | Add `isStrictEvent: false` to the event config to keep standalone occurrences |
|
|
100
120
|
|
|
101
|
-
When writing per-dungeon verify scripts, follow the template in HOOKS.md §9.9. Reference proofs for all
|
|
121
|
+
When writing per-dungeon verify scripts, follow the template in HOOKS.md §9.9. Reference proofs for all 22 vertical dungeons live at `dungeons/vertical/<name>/` — consult them as exemplars before authoring a new one.
|
|
102
122
|
|
|
103
123
|
**Coverage discipline:** count documented hooks in the dungeon's top-level comment block; count `check()` calls in your verify script; the two MUST match. A "NAILED 7/7" claim against an 11-hook dungeon is misleading — either add the missing checks or document the limitation explicitly in the verify script and status file.
|
|
104
124
|
|
|
@@ -4,13 +4,17 @@ Templates and conventions for writing `hook-results.md` and per-dungeon verifica
|
|
|
4
4
|
|
|
5
5
|
## Verdict criteria (5-tier)
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
- **STRONG** — Within 25% of expected. Direction correct, clearly visible. An analyst would find this pattern immediately.
|
|
9
|
-
- **WEAK** — Within 50% of expected. Directionally correct but magnitude is off, OR sample size is too small to be conclusive.
|
|
10
|
-
- **NONE** — No statistically meaningful difference between cohorts. The hook has no observable effect.
|
|
11
|
-
- **INVERSE** — Effect goes the opposite direction from intended. The story is backwards.
|
|
7
|
+
Verdicts are **mechanical** — computed by `scripts/verify-stories.mjs` from each story assertion's declared `target` / `floor` / `minCohort` (see the `StoryVerdict` typedef in `types.d.ts`), not eyeballed percentages:
|
|
12
8
|
|
|
13
|
-
NAILED
|
|
9
|
+
- **NAILED** — observed within ±10% of `target`.
|
|
10
|
+
- **STRONG** — passes `floor` (or `target` when no floor is declared).
|
|
11
|
+
- **WEAK** — fails `floor` but effect direction is correct, **or** the selected cohort is smaller than `minCohort`. The population floor is a hard cap: a 12-user cohort can never score NAILED, no matter how clean its ratio.
|
|
12
|
+
- **NONE** — no measurable effect, or the selection matched no rows.
|
|
13
|
+
- **INVERSE** — effect direction is opposite the assertion.
|
|
14
|
+
|
|
15
|
+
Story verdict = worst assertion verdict. NAILED and STRONG are passing; WEAK, NONE, and INVERSE fail and require investigation.
|
|
16
|
+
|
|
17
|
+
Hand-assigned verdicts appear only in the legacy no-stories fallback and MUST follow the same definitions: derive a target from the hook's knob constants, compute the band the observed value lands in, and state the derivation in the detail block — never assign a tier by feel.
|
|
14
18
|
|
|
15
19
|
## Ordering: failures first
|
|
16
20
|
|
|
@@ -25,6 +29,8 @@ The summary table should also be sorted this way (INVERSE → NONE → WEAK →
|
|
|
25
29
|
|
|
26
30
|
## Single-dungeon report structure
|
|
27
31
|
|
|
32
|
+
For story-backed dungeons, `hook-results.md` **renders the runner's JSON**: run `verify-stories.mjs --json` and build the Hook Summary table directly from its per-story records (story id, hook number, archetype, observed vs target per assertion, computed verdict). Do not recompute verdicts the runner already settled. Detailed Results blocks exist only for stories below STRONG, `duckdb`-type assertions, and legacy no-stories hooks.
|
|
33
|
+
|
|
28
34
|
```markdown
|
|
29
35
|
# Dungeon Verification Report
|
|
30
36
|
|
|
@@ -191,7 +197,7 @@ grep "^DUNGEON:" research/hook-query-log.txt # list of dungeons querie
|
|
|
191
197
|
|
|
192
198
|
When verifying a dungeon in `dungeons/user/`, write a standalone DuckDB SQL file alongside the dungeon in its folder at `dungeons/user/<name>/<name>-verifications.sql`. This file is the reproducible verification artifact — anyone can re-run it against fresh data.
|
|
193
199
|
|
|
194
|
-
Follow the format in `
|
|
200
|
+
Follow the format in `dungeons/vertical/<name>/`:
|
|
195
201
|
|
|
196
202
|
```sql
|
|
197
203
|
-- ============================================================================
|
|
@@ -211,4 +217,4 @@ Follow the format in `verification/verticals/`:
|
|
|
211
217
|
|
|
212
218
|
Each query block includes the pattern description, observed result, and verdict as SQL comments. This makes the file self-documenting and grep-friendly.
|
|
213
219
|
|
|
214
|
-
**This step is mandatory for user dungeons.** Vertical dungeons already have their SQL
|
|
220
|
+
**This step is mandatory for user dungeons.** Vertical dungeons already have their SQL co-located at `dungeons/vertical/<name>/<name>.sql`. User dungeons keep theirs co-located with the dungeon file.
|
|
@@ -20,7 +20,11 @@ already complete (produced by `create-dungeon`). After writing, hand off to
|
|
|
20
20
|
In scope:
|
|
21
21
|
- The `hook: function(record, type, meta) { ... }` body
|
|
22
22
|
- Documentation comments above the hook explaining each engineered pattern,
|
|
23
|
-
including a reference Mixpanel report block per pattern
|
|
23
|
+
including a reference Mixpanel report block per pattern and the mandatory
|
|
24
|
+
EXPECTED METRICS SUMMARY table
|
|
25
|
+
- The `stories` named export — one machine-checkable story per engineered
|
|
26
|
+
pattern (see "Stories export" below). A hook without stories is
|
|
27
|
+
unverifiable; this skill is not done until the stories exist.
|
|
24
28
|
|
|
25
29
|
Out of scope:
|
|
26
30
|
- Schema changes (events, properties, funnels, superProps, userProps).
|
|
@@ -29,6 +33,18 @@ Out of scope:
|
|
|
29
33
|
- New top-level config knobs.
|
|
30
34
|
- Removing the `hook: function...` body to start over with a new schema.
|
|
31
35
|
|
|
36
|
+
**Before writing a hook, ask: is this trend structural?** Between-path
|
|
37
|
+
comparisons (path X converts worse / slower than path Y, detour-takers drop
|
|
38
|
+
off, mix shift drags the blended rate) are better architected as initial
|
|
39
|
+
conditions — duplicate funnels with swapped steps/props/`conversionRate`/
|
|
40
|
+
`timeToConvert`/`weight` (see the "Structural trend engineering" section in
|
|
41
|
+
`create-dungeon`). If a story reduces to structure, recommend the funnel
|
|
42
|
+
change back to the schema instead of writing a hook to fight the engine —
|
|
43
|
+
the knob IS the expected value, which makes the story band knob-derivable
|
|
44
|
+
(NAILED-capable) instead of confounded (STRONG-capped). Hooks are for
|
|
45
|
+
within-cohort behavior: segments doing more/less of something over time,
|
|
46
|
+
property values that differ by cohort, injected bursts, lifecycle waves.
|
|
47
|
+
|
|
32
48
|
## Reference reading
|
|
33
49
|
|
|
34
50
|
- `lib/hook-helpers/index.js` — atoms (cohort, mutate, timing, inject,
|
|
@@ -36,8 +52,11 @@ Out of scope:
|
|
|
36
52
|
- `lib/hook-patterns/index.js` — high-level recipes (one per Mixpanel
|
|
37
53
|
analysis type).
|
|
38
54
|
- `lib/verify/emulate-breakdown.js` — what `verify-dungeon` will check.
|
|
39
|
-
- `
|
|
40
|
-
|
|
55
|
+
- `lib/templates/story-spec.schema.json` + `DungeonStory` in `types.d.ts` —
|
|
56
|
+
the story-spec grammar the `stories` export must follow.
|
|
57
|
+
- `dungeons/vertical/ecommerce/ecommerce.js` — reference dungeon using a mix
|
|
58
|
+
of atoms and hand-rolled logic, with the mandatory EXPECTED METRICS SUMMARY
|
|
59
|
+
table and a full `stories` export.
|
|
41
60
|
- `dungeons/technical/pattern-*.js` — five minimal pattern fixtures, one per
|
|
42
61
|
recipe.
|
|
43
62
|
- `HOOKS.md` — encyclopedia of hook recipes organized by story pattern. Contains
|
|
@@ -134,6 +153,10 @@ if (type === 'funnel-post' && meta.experiment) {
|
|
|
134
153
|
| inject | `injectOnNewDays(events, eventName, targetDistinctDays)` | **Cohort-only.** Spreads injections across previously-empty days. Use for cohort-conditional active-day boosts; for global active-day shape, use `Dungeon.avgActiveDaysPerUser` config knob. |
|
|
135
154
|
| identity | `isPreAuthEvent(event, authTime)` | Standalone variant of meta.isPreAuth |
|
|
136
155
|
| identity | `splitByAuth(events, authTime)` | { preAuth, postAuth, stitch } partition |
|
|
156
|
+
| cohort | `hashCohort(id, pct)` | Deterministic pct% cohort (0–100 scale). **Use this first for hidden cohorts** — replaces ad-hoc `charCodeAt % N`. When one dungeon needs several NON-overlapping cohorts, gate on disjoint `hashFloat(uid)` bands instead (e.g. `[0, 0.45)`, `[0.45, 0.70)`) |
|
|
157
|
+
| shape | `applyLifecycleWave(events, uid, opts)` | Dormancy window + resurrection burst for Lifecycle reports. When-to-use: the story is "users go quiet, then come back". Gap discipline: ONE stray value moment inside the window destroys the Resurrected read — size `dormantDays` to cover ≥2 whole lifecycle periods, keep the window inside the user's lifespan |
|
|
158
|
+
| shape | `applyPathBias(events, uid, opts)` | Inject a Flows path after the user's FIRST anchor occurrence. When-to-use: the story is "X% of users take this route". `share` is a 0–1 FRACTION (not `hashCohort`'s pct scale); needs ~≥0.20–0.25 to survive Sankey top-3-per-level pruning; per-step gaps clamped ≥1s so ordering survives |
|
|
159
|
+
| shape | `applySessionShape(events, uid, opts)` | Retime the whole stream into `sessionsPerWeek` clusters of `sessionMinutes`. When-to-use: session-duration/cadence stories (sessionMetrics reads). Retiming ONLY — no adds/drops; intra-session gaps stay <28min, inter-session >30min, no cluster crosses UTC midnight. Combine with `hashCohort` for per-role shapes; call BEFORE `applyPathBias` so injected paths keep their own tight gaps |
|
|
137
160
|
|
|
138
161
|
### Hook anti-patterns
|
|
139
162
|
|
|
@@ -214,17 +237,51 @@ on undeclared entries.
|
|
|
214
237
|
Higher-level recipes. Each maps to ONE Mixpanel analysis the verify-dungeon
|
|
215
238
|
emulator can re-derive.
|
|
216
239
|
|
|
217
|
-
| Pattern | Mixpanel analysis | Hook type |
|
|
218
|
-
|
|
219
|
-
| `applyFrequencyByFrequency` | Insights — count(A) by per-user count(B) | everything |
|
|
220
|
-
| `applyFunnelFrequencyBreakdown` | Funnels — completion by per-user count(X) | funnel-post |
|
|
221
|
-
| `applyAggregateByBin` | Insights — avg(prop X) by per-user count(B) | everything |
|
|
222
|
-
| `
|
|
223
|
-
| `applyAttributedBySource` | Conversions by Source (first/last touch) | everything |
|
|
240
|
+
| Pattern | Mixpanel analysis | Hook type | Caveat (HOOKS.md) |
|
|
241
|
+
|---------|-------------------|-----------|-------------------|
|
|
242
|
+
| `applyFrequencyByFrequency` | Insights — count(A) by per-user count(B) | everything | `binBy` defaults to `'distinctDays'` (v1.6) — bins match Mixpanel's per-user distinct-day counting, not raw event totals |
|
|
243
|
+
| `applyFunnelFrequencyBreakdown` | Funnels — completion by per-user count(X) | funnel-post | When funnels share a step prefix, restrict scaling to the target funnel — scaling every instance lets first-occurrence funnel evaluation assemble chains across unscaled instances and the ratio never reaches the report |
|
|
244
|
+
| `applyAggregateByBin` | Insights — avg(prop X) by per-user count(B) | everything | Same `binBy: 'distinctDays'` default as above |
|
|
245
|
+
| `applyTTCBySegmentV2` | Funnel TTC — broken down by user-property segment | everything | v1 (`applyTTCBySegment`, funnel-post) is **deprecated**: Mixpanel TTC reads each step's FIRST occurrence per user, so per-run gap scaling only reaches the report on `isFirstFunnel` runs. V2 finds the greedy first sequence (`findFirstSequence`) and scales that |
|
|
246
|
+
| `applyAttributedBySource` | Conversions by Source (first/last touch) | everything | OVERWRITES the engine-stamped touch the chosen model reads; never stamps UTMs onto unstamped events (would blow the `maxTouchpointsPerUser` cap and land outside the last-10 lookback) |
|
|
224
247
|
|
|
225
248
|
Use a pattern when the trend matches its analysis 1:1. Drop down to atoms when
|
|
226
249
|
the trend is bespoke or composite.
|
|
227
250
|
|
|
251
|
+
### New story archetypes (v1.6) — design rules per report family
|
|
252
|
+
|
|
253
|
+
Four archetypes joined the story-spec enum in v1.6. Each has a
|
|
254
|
+
non-negotiable design rule learned the hard way:
|
|
255
|
+
|
|
256
|
+
- **`lifecycle-wave`** (`applyLifecycleWave`) — GAP DISCIPLINE. Mixpanel's
|
|
257
|
+
"dormant" state is an `EqualTo 0` filter over the whole period: one stray
|
|
258
|
+
value-moment event inside the dormancy window (including events OTHER hooks
|
|
259
|
+
injected earlier in the same `everything` pass) flips the user out of
|
|
260
|
+
Resurrected. Run the wave AFTER every injecting hook, size `dormantDays` to
|
|
261
|
+
≥2 lifecycle periods, and keep `dormantFromDay + dormantDays` inside the
|
|
262
|
+
user's lifespan (the future-time guard eats bursts past dataset end).
|
|
263
|
+
- **`path-share`** (`applyPathBias`) — FIRST-FLOW ANCHORING. Flows' unique
|
|
264
|
+
counting reads only each user's FIRST flow past the anchor, so the injected
|
|
265
|
+
path must follow the FIRST anchor occurrence (the atom does this; don't
|
|
266
|
+
hand-roll a later anchor). Sankey prunes to the top ~3 branches per level:
|
|
267
|
+
an engineered branch below ~20–25% share silently disappears from the
|
|
268
|
+
visualization even though the data is there. Label-only path reads can
|
|
269
|
+
INVERT when a busier cohort glues extra visible events between path steps —
|
|
270
|
+
assert the share on the cohort you engineered, not globally.
|
|
271
|
+
- **`session-shape`** (`applySessionShape`) — 30-MIN STRADDLING + MIDNIGHT
|
|
272
|
+
RULE. Mixpanel derives sessions with a 30-min idle timeout and splits at
|
|
273
|
+
UTC midnight. Engineered cadences must keep intra-session gaps clearly
|
|
274
|
+
UNDER 30min and inter-session gaps clearly OVER it — a gap that straddles
|
|
275
|
+
the timeout makes session counts jitter across runs. Never let an
|
|
276
|
+
engineered session cross UTC midnight (the day split cuts it in two). The
|
|
277
|
+
atom guarantees all three; hand-rolled retiming must too.
|
|
278
|
+
- **`composition-drift`** — the breakdown's SHARE of a segment moves over
|
|
279
|
+
time while totals stay flat (e.g. plan-mix shifts toward premium). Engineer
|
|
280
|
+
by flipping an existing property value on a date-gated cohort, never by
|
|
281
|
+
changing volumes — volume changes read as `temporal-inflection` instead.
|
|
282
|
+
Assert with a `timeBucket` breakdown comparing first-window vs last-window
|
|
283
|
+
share.
|
|
284
|
+
|
|
228
285
|
## Anti-flag-stamping rule (HARD WALL)
|
|
229
286
|
|
|
230
287
|
Hooks MUST NOT add new properties to records. The schema (config) defines what
|
|
@@ -318,6 +375,71 @@ checks against and what consumers read to understand the dataset.
|
|
|
318
375
|
* Expected ratio: bin>=15 / bin<5 ≈ 3x (within ±15%)
|
|
319
376
|
```
|
|
320
377
|
|
|
378
|
+
### EXPECTED METRICS SUMMARY table (MANDATORY)
|
|
379
|
+
|
|
380
|
+
The doc block MUST end with an EXPECTED METRICS SUMMARY table — one row per
|
|
381
|
+
verifiable read, with the DERIVATION of each expected number from the hook's
|
|
382
|
+
knob constants (never a bare number someone has to trust). Follow the style
|
|
383
|
+
in `dungeons/vertical/ecommerce/ecommerce.js`:
|
|
384
|
+
|
|
385
|
+
```
|
|
386
|
+
* EXPECTED METRICS SUMMARY
|
|
387
|
+
* ============================================================================
|
|
388
|
+
*
|
|
389
|
+
* Hook | Metric | Derivation | Expected | Measured (full)
|
|
390
|
+
* -----|---------------------------------|---------------------|----------|----------------
|
|
391
|
+
* H2 | avg watchTimeSec post/pre | 1.52/0.48 | 3.17x | 3.19x
|
|
392
|
+
* H6 | sweet/over avg cart item amount | SWEET_CART_BOOST | 1.25x | 1.233x
|
|
393
|
+
* H9 | dark/light checkouts per user | 20/13 diluted | ~1.48x | 1.412x
|
|
394
|
+
* ============================================================================
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
Fill the Measured column from the verification run (a reduced-scale iteration
|
|
398
|
+
number is fine if labeled). "Diluted" derivations must say WHAT dilutes
|
|
399
|
+
(organic events, cohort mixing) — the Derivation column is the anchor
|
|
400
|
+
`/verify-dungeon` uses when a read misses.
|
|
401
|
+
|
|
402
|
+
## Stories export (MANDATORY — the machine contract)
|
|
403
|
+
|
|
404
|
+
Every engineered pattern ships with one story in a `stories` named export —
|
|
405
|
+
the machine-checkable form of the doc block. `scripts/verify-stories.mjs`
|
|
406
|
+
evaluates them; `/verify-dungeon` runs it as step 1. Grammar:
|
|
407
|
+
`lib/templates/story-spec.schema.json` and `DungeonStory` in `types.d.ts`.
|
|
408
|
+
|
|
409
|
+
```js
|
|
410
|
+
export const stories = [
|
|
411
|
+
{
|
|
412
|
+
id: "H1-power-buyers",
|
|
413
|
+
hook: "H1",
|
|
414
|
+
archetype: "frequency-sweet-spot",
|
|
415
|
+
narrative: "Users with 15+ browse days buy 3x as often as light browsers.",
|
|
416
|
+
mixpanelReport: { type: "Insights", event: "Purchase", breakdown: "per-user count of Browse" },
|
|
417
|
+
assertions: [{
|
|
418
|
+
breakdown: { type: "frequency", event: "Purchase", cohortEvent: "Browse", bins: [5, 15] },
|
|
419
|
+
select: { hi: { where: { bin: ">=15" } }, lo: { where: { bin: "<5" } } },
|
|
420
|
+
expect: { metric: "hi.avg / lo.avg", op: ">=", target: POWER_BUYER_MULT, floor: POWER_BUYER_MULT * 0.8 },
|
|
421
|
+
minCohort: 200,
|
|
422
|
+
}],
|
|
423
|
+
},
|
|
424
|
+
];
|
|
425
|
+
```
|
|
426
|
+
|
|
427
|
+
Rules:
|
|
428
|
+
|
|
429
|
+
- **Thresholds derive from the knob you just wrote.** Export the hook's knob
|
|
430
|
+
constants (`const POWER_BUYER_MULT = 3`) and compute `target` from them —
|
|
431
|
+
never paste the number twice. If the read is diluted (organic mixing),
|
|
432
|
+
derive the dilution too and say so in a comment.
|
|
433
|
+
- `floor` must itself be derived (e.g. `target * 0.8`) — never hand-tuned to
|
|
434
|
+
a run. A missed assertion means fixing the hook or the derivation, never
|
|
435
|
+
relaxing the number to match output.
|
|
436
|
+
- Set `minCohort` from the cohort math (share × numUsers × ~0.5 safety) so
|
|
437
|
+
reduced-scale runs cap at WEAK instead of passing on noise.
|
|
438
|
+
- One story per pattern; story `hook` matches the doc-block numbering (`H3`).
|
|
439
|
+
- The `assert` function escape hatch is discouraged — each use needs a
|
|
440
|
+
comment saying why the declarative `expect` grammar can't express it.
|
|
441
|
+
- Stories are JS-dungeon-only (`dungeon-to-json` drops them).
|
|
442
|
+
|
|
321
443
|
## Hook Ordering Within `everything`
|
|
322
444
|
|
|
323
445
|
The order of operations inside the everything hook matters when hooks interact:
|
|
@@ -466,22 +588,37 @@ cohort.
|
|
|
466
588
|
2. Translate the user's story description into 3–5 engineered patterns.
|
|
467
589
|
Consult `HOOKS.md` for recipe ideas that match the user's story. Each recipe
|
|
468
590
|
includes the hook type, code snippet, and Mixpanel report format.
|
|
469
|
-
3.
|
|
591
|
+
3. **Calibrate thresholds against the real distribution.** Before choosing
|
|
592
|
+
any "N+ events" gate or cohort cutoff, generate a small run and query the
|
|
593
|
+
actual per-user distribution (see "Threshold Calibration" above for the
|
|
594
|
+
query). Set gates at ~the 80th percentile of what the data shows — a
|
|
595
|
+
threshold picked from intuition is the most common cause of empty cohorts.
|
|
596
|
+
```bash
|
|
597
|
+
node scripts/verify-runner.mjs <dungeon> calib --small
|
|
598
|
+
```
|
|
599
|
+
4. For each pattern:
|
|
470
600
|
- Pick a pattern from `lib/hook-patterns/` if it fits the analysis 1:1.
|
|
471
601
|
- Otherwise compose atoms from `lib/hook-helpers/`.
|
|
472
602
|
- Document the pattern in a comment block (Mixpanel report instructions).
|
|
473
|
-
|
|
474
|
-
|
|
603
|
+
5. Write the `hook` function, importing atoms/patterns at the top of the
|
|
604
|
+
file. Finish the doc block with the EXPECTED METRICS SUMMARY table.
|
|
605
|
+
6. Write the `stories` export — one story per pattern, `target`/`floor`
|
|
606
|
+
derived from the exported knob constants (see "Stories export" above).
|
|
607
|
+
7. Smoke-test generation, then evaluate the stories:
|
|
475
608
|
```bash
|
|
476
609
|
node scripts/verify-runner.mjs <dungeon> verify-dungeon --small
|
|
610
|
+
node scripts/verify-stories.mjs <dungeon> --data-prefix verify-dungeon
|
|
477
611
|
```
|
|
478
|
-
|
|
479
|
-
|
|
612
|
+
Reduced-scale runs legitimately cap at WEAK on `minCohort` guards; what
|
|
613
|
+
you're checking here is no NONE/INVERSE and no assertion errors.
|
|
614
|
+
8. Hand off:
|
|
480
615
|
```
|
|
481
616
|
/verify-dungeon <dungeon>
|
|
482
617
|
```
|
|
483
|
-
If verify-dungeon returns WEAK, NONE, or INVERSE on any pattern
|
|
484
|
-
step 4 and refine
|
|
618
|
+
If verify-dungeon returns WEAK, NONE, or INVERSE on any pattern at full
|
|
619
|
+
fidelity, return to step 4 and refine — fix the hook or the derivation,
|
|
620
|
+
never relax a threshold to match output. Iterate until all patterns score
|
|
621
|
+
STRONG or NAILED.
|
|
485
622
|
|
|
486
623
|
## Stopping condition
|
|
487
624
|
|
|
@@ -492,5 +629,6 @@ still off in the dungeon's overview comment and report the gap to the user.
|
|
|
492
629
|
## Output
|
|
493
630
|
|
|
494
631
|
Modify the dungeon file in place. Add the `hook` function. Add the imports.
|
|
495
|
-
Add the documentation block
|
|
632
|
+
Add the documentation block (with EXPECTED METRICS SUMMARY) above the config.
|
|
633
|
+
Add the `stories` export after the config. Do NOT modify any other file.
|
|
496
634
|
Tell the user to run `/verify-dungeon <dungeon>` next.
|