@ak--47/dungeon-master 1.5.4 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/analyze-soup/SKILL.md +21 -12
- package/.claude/skills/create-dungeon/SKILL.md +65 -5
- package/.claude/skills/create-project/SKILL.md +1 -1
- package/.claude/skills/create-project/provision.mjs +57 -10
- package/.claude/skills/verify-dungeon/SKILL.md +56 -25
- package/.claude/skills/verify-dungeon/references/counting-semantics.md +26 -6
- package/.claude/skills/verify-dungeon/references/report-format.md +14 -8
- package/.claude/skills/write-hooks/SKILL.md +156 -18
- package/CHANGELOG.md +183 -0
- package/HOOKS.md +566 -112
- package/README.md +28 -9
- package/docs/guides/1.3.0-upgrade-guide.md +262 -0
- package/docs/guides/1.3.1-upgrade-guide.md +328 -0
- package/docs/guides/1.4.0-upgrade-guide.md +130 -0
- package/docs/guides/1.4.1-upgrade-guide.md +125 -0
- package/docs/guides/1.4.3-upgrade-guide.md +200 -0
- package/docs/guides/1.4.4-upgrade-guide.md +95 -0
- package/docs/guides/1.5.0-upgrade-guide.md +426 -0
- package/docs/guides/1.5.1-upgrade-guide.md +178 -0
- package/docs/guides/1.5.3-upgrade-guide.md +117 -0
- package/docs/guides/1.6.0-upgrade-guide.md +204 -0
- package/dungeons/technical/stories-verify.js +152 -0
- package/dungeons/vertical/README.md +154 -0
- package/dungeons/vertical/ai-platform/ai-platform.js +1424 -0
- package/dungeons/vertical/ai-platform/ai-platform.sql +184 -0
- package/dungeons/vertical/ai-platform/ai-platform.verify.mjs +71 -0
- package/dungeons/vertical/community/community.js +1589 -0
- package/dungeons/vertical/community/community.sql +216 -0
- package/dungeons/vertical/community/community.verify.mjs +72 -0
- package/dungeons/vertical/crypto/crypto.js +1483 -0
- package/dungeons/vertical/crypto/crypto.sql +194 -0
- package/dungeons/vertical/crypto/crypto.verify.mjs +78 -0
- package/dungeons/vertical/dating/dating.js +1540 -0
- package/dungeons/vertical/dating/dating.sql +288 -0
- package/dungeons/vertical/dating/dating.verify.mjs +72 -0
- package/dungeons/vertical/devtools/devtools.js +1524 -0
- package/dungeons/vertical/devtools/devtools.sql +161 -0
- package/dungeons/vertical/devtools/devtools.verify.mjs +73 -0
- package/dungeons/vertical/ecommerce/ecommerce.js +1480 -0
- package/dungeons/vertical/ecommerce/ecommerce.sql +218 -0
- package/dungeons/vertical/ecommerce/ecommerce.verify.mjs +70 -0
- package/dungeons/vertical/education/education.js +1763 -0
- package/dungeons/vertical/education/education.sql +243 -0
- package/dungeons/vertical/education/education.verify.mjs +75 -0
- package/dungeons/vertical/fintech/fintech.js +1410 -0
- package/dungeons/vertical/fintech/fintech.sql +175 -0
- package/dungeons/vertical/fintech/fintech.verify.mjs +70 -0
- package/dungeons/vertical/fitness/fitness.js +1495 -0
- package/dungeons/vertical/fitness/fitness.sql +206 -0
- package/dungeons/vertical/fitness/fitness.verify.mjs +71 -0
- package/dungeons/vertical/food-delivery/food-delivery.js +1641 -0
- package/dungeons/vertical/food-delivery/food-delivery.sql +183 -0
- package/dungeons/vertical/food-delivery/food-delivery.verify.mjs +75 -0
- package/dungeons/vertical/gaming/gaming.js +1741 -0
- package/dungeons/vertical/gaming/gaming.sql +249 -0
- package/dungeons/vertical/gaming/gaming.verify.mjs +70 -0
- package/dungeons/vertical/healthcare/healthcare.js +1488 -0
- package/dungeons/vertical/healthcare/healthcare.sql +193 -0
- package/dungeons/vertical/healthcare/healthcare.verify.mjs +72 -0
- package/dungeons/vertical/insurance-application/insurance-application.js +1852 -0
- package/dungeons/vertical/insurance-application/insurance-application.sql +174 -0
- package/dungeons/vertical/insurance-application/insurance-application.verify.mjs +74 -0
- package/dungeons/vertical/logistics/logistics.js +1541 -0
- package/dungeons/vertical/logistics/logistics.sql +205 -0
- package/dungeons/vertical/logistics/logistics.verify.mjs +73 -0
- package/dungeons/vertical/marketplace/marketplace.js +1438 -0
- package/dungeons/vertical/marketplace/marketplace.sql +148 -0
- package/dungeons/vertical/marketplace/marketplace.verify.mjs +74 -0
- package/dungeons/vertical/{media.js → media/media.js} +557 -58
- package/dungeons/vertical/media/media.sql +173 -0
- package/dungeons/vertical/media/media.verify.mjs +74 -0
- package/dungeons/vertical/real-estate/real-estate.js +1629 -0
- package/dungeons/vertical/real-estate/real-estate.sql +167 -0
- package/dungeons/vertical/real-estate/real-estate.verify.mjs +75 -0
- package/dungeons/vertical/sass/sass.js +1733 -0
- package/dungeons/vertical/sass/sass.sql +173 -0
- package/dungeons/vertical/sass/sass.verify.mjs +74 -0
- package/dungeons/vertical/social/social.js +1456 -0
- package/dungeons/vertical/social/social.sql +160 -0
- package/dungeons/vertical/social/social.verify.mjs +76 -0
- package/dungeons/vertical/streaming/streaming.js +677 -0
- package/dungeons/vertical/streaming/streaming.sql +101 -0
- package/dungeons/vertical/streaming/streaming.verify.mjs +75 -0
- package/dungeons/vertical/support-desk/support-desk.js +741 -0
- package/dungeons/vertical/support-desk/support-desk.sql +137 -0
- package/dungeons/vertical/support-desk/support-desk.verify.mjs +75 -0
- package/dungeons/vertical/travel/travel.js +1603 -0
- package/dungeons/vertical/travel/travel.sql +236 -0
- package/dungeons/vertical/travel/travel.verify.mjs +72 -0
- package/index.js +2 -2
- package/lib/core/config-validator.js +37 -2
- package/lib/core/context.js +4 -2
- package/lib/core/dungeon-loader.js +20 -15
- package/lib/core/dungeon-to-json.js +22 -4
- package/lib/core/storage.js +3 -3
- package/lib/generators/funnels.js +36 -9
- package/lib/hook-helpers/cohort.js +53 -0
- package/lib/hook-helpers/index.js +8 -0
- package/lib/hook-helpers/mutate.js +1 -1
- package/lib/hook-helpers/shape.js +319 -0
- package/lib/hook-patterns/aggregate-per-user-by-bin.js +13 -5
- package/lib/hook-patterns/attributed-by-source.js +60 -58
- package/lib/hook-patterns/frequency-by-frequency.js +14 -4
- package/lib/hook-patterns/funnel-frequency-breakdown.js +24 -12
- package/lib/hook-patterns/index.js +1 -1
- package/lib/hook-patterns/time-to-convert-by-segment.js +75 -13
- package/lib/orchestrators/user-loop.js +41 -0
- package/lib/templates/story-spec.schema.json +175 -0
- package/lib/utils/json-evaluator.js +1 -1
- package/lib/utils/utils.js +5 -4
- package/lib/verify/coerce.js +186 -0
- package/lib/verify/counting.js +270 -32
- package/lib/verify/emulate-breakdown.js +1285 -203
- package/lib/verify/first-time.js +89 -0
- package/lib/verify/flows.js +679 -0
- package/lib/verify/formula.js +259 -0
- package/lib/verify/funnel-engine.js +697 -113
- package/lib/verify/index.js +32 -4
- package/lib/verify/sessionize.js +275 -0
- package/lib/verify/story-runner.js +483 -0
- package/lib/verify/verify-dungeon.js +59 -37
- package/package.json +2 -1
- package/scripts/extract-dungeon-schema.mjs +2 -2
- package/scripts/run-many.mjs +2 -2
- package/scripts/verify-runner.mjs +2 -2
- package/scripts/verify-stories.mjs +244 -0
- package/types.d.ts +296 -30
- package/dungeons/vertical/ai-platform.js +0 -818
- package/dungeons/vertical/community.js +0 -966
- package/dungeons/vertical/crypto.js +0 -895
- package/dungeons/vertical/dating.js +0 -830
- package/dungeons/vertical/devtools.js +0 -1050
- package/dungeons/vertical/ecommerce.js +0 -811
- package/dungeons/vertical/education.js +0 -964
- package/dungeons/vertical/fintech.js +0 -946
- package/dungeons/vertical/fitness.js +0 -935
- package/dungeons/vertical/food-delivery.js +0 -934
- package/dungeons/vertical/gaming.js +0 -1120
- package/dungeons/vertical/healthcare.js +0 -945
- package/dungeons/vertical/insurance-application.js +0 -1039
- package/dungeons/vertical/logistics.js +0 -986
- package/dungeons/vertical/marketplace.js +0 -1019
- package/dungeons/vertical/real-estate.js +0 -872
- package/dungeons/vertical/sass.js +0 -995
- package/dungeons/vertical/social.js +0 -904
- package/dungeons/vertical/travel.js +0 -930
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# 1.5.3 Upgrade Guide
|
|
2
|
+
|
|
3
|
+
**TL;DR: No breaking changes — two new exports.** `dungeonToJSON` serializes a dungeon to the JSON/UI wrapper format (the inverse of `parseJSONDungeon`), and `extractComments` pulls the `OVERVIEW` / `HOOK STORIES` doc blocks out of a dungeon's source. Nothing else changed. Existing dungeons and code run unmodified.
|
|
4
|
+
|
|
5
|
+
## What Changed
|
|
6
|
+
|
|
7
|
+
### 1. `dungeonToJSON(input, options?)` — dungeon → JSON
|
|
8
|
+
|
|
9
|
+
**What**: A new named export that converts a dungeon into the `{ schema, hooks, timestamp, version }` JSON/UI wrapper format — the inverse of `parseJSONDungeon`. It accepts the same input flavors as the default export.
|
|
10
|
+
|
|
11
|
+
```js
|
|
12
|
+
import DUNGEON_MASTER, { dungeonToJSON, parseJSONDungeon } from "@ak--47/dungeon-master";
|
|
13
|
+
|
|
14
|
+
// From a config object
|
|
15
|
+
const json = await dungeonToJSON(config);
|
|
16
|
+
|
|
17
|
+
// From a file path (.js / .mjs / .json)
|
|
18
|
+
const json = await dungeonToJSON("./dungeons/vertical/ecommerce.js");
|
|
19
|
+
|
|
20
|
+
// From raw source text
|
|
21
|
+
const json = await dungeonToJSON("export default { numUsers: 100, events: [...] }");
|
|
22
|
+
|
|
23
|
+
// From an array of paths → array of results
|
|
24
|
+
const all = await dungeonToJSON(["./a.js", "./b.js"]);
|
|
25
|
+
|
|
26
|
+
// Round-trips back into a runnable config
|
|
27
|
+
const config = parseJSONDungeon(await dungeonToJSON("./dungeons/vertical/ecommerce.js"));
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
The output shape:
|
|
31
|
+
|
|
32
|
+
```js
|
|
33
|
+
{
|
|
34
|
+
schema, // config with functions serialized to { functionName, body | args }, hook excluded
|
|
35
|
+
hooks, // the hook function stringified, or null
|
|
36
|
+
timestamp, // ISO string
|
|
37
|
+
version: "4.0" // UI schema format version
|
|
38
|
+
}
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
**Why**: The package could already ingest JSON dungeons (`parseJSONDungeon`, `loadFromFile`, `loadFromText`) but had no exported way to go the other direction. That logic lived only inside `scripts/dungeon-to-json.mjs` as an un-importable CLI internal.
|
|
42
|
+
|
|
43
|
+
**Action needed**: None — it's a new export. Two things to know if you use it:
|
|
44
|
+
|
|
45
|
+
- **Credentials are stripped by default.** `token`, `serviceAccount`, `serviceSecret`, `projectId`, and `secret` are removed from the output so tokens never leak into JSON. Pass `{ includeCredentials: true }` to keep them.
|
|
46
|
+
- **It's best effort, but type-preserving.** Arrow functions and bound `chance.*` methods round-trip cleanly. Detected utility calls (`weighArray`, `weighNumRange`, `pickAWinner`, …) are serialized by name **without their arguments** and revive to `null` — the config validator handles that gracefully, but the revived dungeon won't reproduce those exact property generators. To keep the field's **type** visible even when the generator is lost, every function is sampled at serialization time and its output type is recorded as `dataType`:
|
|
47
|
+
|
|
48
|
+
```js
|
|
49
|
+
// a property like `amount: weighNumRange(5, 500)` serializes to:
|
|
50
|
+
{ functionName: "weighNumRange", args: [], dataType: "number" }
|
|
51
|
+
// an arrow `tag: () => chance.pickone([...])` keeps its body AND records:
|
|
52
|
+
{ functionName: "arrow", body: "() => chance.pickone([...])", dataType: "string" }
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
`dataType` is one of `"number" | "string" | "boolean" | "date" | "object" | "<elementType>[]" | "array"`, omitted only when sampling the function threw. For lossless persistence, keep the `.js` source as the source of truth.
|
|
56
|
+
|
|
57
|
+
The JSON-representation shapes are formally typed — `DungeonJSON`, `DungeonComments`, and `SerializedFunction` are exported from `types.d.ts`.
|
|
58
|
+
|
|
59
|
+
`dungeonToJSON` is always async (file/text/array input must be awaited) — `await` it even for object input.
|
|
60
|
+
|
|
61
|
+
### 2. `extractComments(input)` — read a dungeon's doc blocks
|
|
62
|
+
|
|
63
|
+
**What**: A new named export that pulls the human-readable comment blocks out of a dungeon's **source**. Dungeons authored by the `create-dungeon` / `write-hooks` skills use a canonical section convention:
|
|
64
|
+
|
|
65
|
+
```js
|
|
66
|
+
// ── OVERVIEW ──
|
|
67
|
+
/*
|
|
68
|
+
* NAME: Acme
|
|
69
|
+
* APP: ...
|
|
70
|
+
*/
|
|
71
|
+
|
|
72
|
+
// ── HOOK STORIES ──
|
|
73
|
+
/*
|
|
74
|
+
* 1. POWER USERS BUY 3X MORE (everything)
|
|
75
|
+
* ...
|
|
76
|
+
*/
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
`extractComments` returns those blocks as cleaned prose:
|
|
80
|
+
|
|
81
|
+
```js
|
|
82
|
+
import { extractComments } from "@ak--47/dungeon-master";
|
|
83
|
+
|
|
84
|
+
const { overview, hookStories, sections } = extractComments("./dungeons/vertical/ecommerce.js");
|
|
85
|
+
// overview → cleaned text of the OVERVIEW block (or null)
|
|
86
|
+
// hookStories → cleaned text of the HOOK STORIES block (or null)
|
|
87
|
+
// sections → { OVERVIEW: "...", "HOOK STORIES": "...", ... } — every `// ── LABEL ──`
|
|
88
|
+
// header immediately followed by a block comment, keyed by exact label
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
It accepts a file path, a raw source string, or an array of paths (→ array of results). The comment scaffolding (`// ──`, `/* */`, leading ` * `) is stripped to readable text.
|
|
92
|
+
|
|
93
|
+
**Why**: Tools (UIs, LLM pipelines, doc generators) need to read a dungeon's intent without parsing it by hand. There was no programmatic accessor.
|
|
94
|
+
|
|
95
|
+
**Action needed**: None — it's a new export. Two things to know:
|
|
96
|
+
|
|
97
|
+
- **It operates on source, never the imported module.** Passing a config object throws — importing a dungeon discards its comments, so there's nothing to extract from a live config. Pass a file path or the raw source string.
|
|
98
|
+
- **It's best effort.** It relies on the canonical `// ── LABEL ──` header + block-comment convention. Hand-written dungeons that deviate (different separator characters, no block comment after the header) may return `null` for `overview`/`hookStories` or omit sections.
|
|
99
|
+
|
|
100
|
+
### 3. `scripts/dungeon-to-json.mjs` is now a thin wrapper
|
|
101
|
+
|
|
102
|
+
**What**: The CLI script's inline `convertToJSON` / `convertFunctionToObject` logic moved into `lib/core/dungeon-to-json.js`. The script now calls the exported `dungeonToJSON` (with `includeCredentials: true` to preserve its legacy full-config output for UI round-trips).
|
|
103
|
+
|
|
104
|
+
**Action needed**: None. The CLI behaves identically:
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
node scripts/dungeon-to-json.mjs ./dungeons/vertical/ecommerce.js
|
|
108
|
+
node scripts/json-to-dungeon.mjs ./dungeons/vertical/ecommerce.json
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
## Migration Checklist
|
|
112
|
+
|
|
113
|
+
Nothing required. This release is additive.
|
|
114
|
+
|
|
115
|
+
- [ ] (Optional) Replace any hand-rolled dungeon→JSON code with `dungeonToJSON`.
|
|
116
|
+
- [ ] (Optional) Use `extractComments` to surface OVERVIEW / HOOK STORIES docs in tooling.
|
|
117
|
+
- [ ] If you serialize dungeons that carry credentials, decide whether you need `{ includeCredentials: true }` (default strips them).
|
|
@@ -0,0 +1,204 @@
|
|
|
1
|
+
# 1.6.0 Upgrade Guide
|
|
2
|
+
|
|
3
|
+
**TL;DR: No breaking config API changes.** Every 1.5.x dungeon config runs
|
|
4
|
+
unmodified — the only new config field is `ExperimentConfig.sticky`, whose
|
|
5
|
+
default preserves existing behavior byte-for-byte. 1.6.0 is primarily a
|
|
6
|
+
**verifier-alignment release**: the `/verify` emulator gained five new
|
|
7
|
+
analysis types and was re-derived against Mixpanel's ARB C++ line-by-line,
|
|
8
|
+
so verify-side **numbers can shift** (they are now more correct). Three
|
|
9
|
+
generation-side fixes change output at the same seed, each gated on a
|
|
10
|
+
feature you'd know you're using (experiments with shuffle orders,
|
|
11
|
+
time-mutating hooks, `isChurnEvent`). Shipped vertical dungeons moved to a
|
|
12
|
+
one-folder-per-vertical layout.
|
|
13
|
+
|
|
14
|
+
## What Changed
|
|
15
|
+
|
|
16
|
+
### 1. Story layer (new, additive)
|
|
17
|
+
|
|
18
|
+
**What**: Dungeons can now export `stories` alongside the default config —
|
|
19
|
+
one machine-checkable story per hook (`DungeonStory` in types.d.ts, JSON
|
|
20
|
+
schema at `lib/templates/story-spec.schema.json`). The new
|
|
21
|
+
`scripts/verify-stories.mjs` runner evaluates them mechanically into
|
|
22
|
+
five-tier verdicts (NAILED / STRONG / WEAK / NONE / INVERSE) with
|
|
23
|
+
hook-coverage discipline, population floors, disk + in-memory modes, and
|
|
24
|
+
`--json` output.
|
|
25
|
+
|
|
26
|
+
**Why**: "Did the engineered pattern actually appear in the data?" used to
|
|
27
|
+
be an LLM judgment call. Stories make it a mechanical check.
|
|
28
|
+
|
|
29
|
+
**Action needed**: None. Dungeons without a `stories` export behave exactly
|
|
30
|
+
as before. All 22 shipped verticals now carry stories — use them as
|
|
31
|
+
reference implementations.
|
|
32
|
+
|
|
33
|
+
### 2. Emulator expansion (new, additive)
|
|
34
|
+
|
|
35
|
+
**What**: `emulateBreakdown` gains five types — `eventBreakdown` (Insights
|
|
36
|
+
totals broken down by property, with Mixpanel's exact segment coercion),
|
|
37
|
+
`uniques` (per-interval dedup, rolling XAU, cumulative), `lifecycle`
|
|
38
|
+
(new / retained / resurrected / dormant), `topPaths` (Flows), and
|
|
39
|
+
`distinctCount`. Retention is completed: `compounded`, `birthCanRetain`,
|
|
40
|
+
`carryForward` / `carryBack` / `consecutiveForward`, `calendarStart`,
|
|
41
|
+
`cohortWindow`, `segmentOn: 'return'`, week/month buckets. New standalone
|
|
42
|
+
primitives: `sessionize()`, `filterFirstTimeEver()`, `evaluateFormula()`,
|
|
43
|
+
`extractFlows` / `aggregateFlows`, `frequencyHistogram`, null-aware
|
|
44
|
+
avg/sum `{ flatten: true }`.
|
|
45
|
+
|
|
46
|
+
**Action needed**: None — all additive. If you previously verified these
|
|
47
|
+
report types in DuckDB or a live project, the emulator now covers them.
|
|
48
|
+
|
|
49
|
+
### 3. Verify-side behavior changes (numbers can shift)
|
|
50
|
+
|
|
51
|
+
These make the emulator agree with Mixpanel's ARB implementation. If you
|
|
52
|
+
pinned expected values against 1.5.x emulator output, re-baseline.
|
|
53
|
+
|
|
54
|
+
- **Retention option keys are strict.** Unknown keys in a `retention`
|
|
55
|
+
config now **throw** instead of being silently ignored (a typo'd option
|
|
56
|
+
used to run with defaults and produce plausible-but-wrong numbers).
|
|
57
|
+
`carry_forward: true` survives as a deprecated alias for
|
|
58
|
+
`unbounded: 'carryForward'`.
|
|
59
|
+
- **Funnel exclusions no longer fire before step 0.** An exclusion event
|
|
60
|
+
before the first step no longer condemns the attempt (ARB semantics);
|
|
61
|
+
it only matters inside the 2-second grace rule at step 0.
|
|
62
|
+
- **Non-sequential funnel orders verify with full ARB semantics.**
|
|
63
|
+
`first-fixed` / `last-fixed` / `first-and-last-fixed` / `outside-in` /
|
|
64
|
+
`random` used to verify via loose set-membership; they now enforce
|
|
65
|
+
conversion windows, the 2-second rule, exclusions, and anchor ordering.
|
|
66
|
+
Users that passed the loose check but violate a window no longer
|
|
67
|
+
convert — funnel counts can drop.
|
|
68
|
+
- **`sessionMetrics` defaults to query-time derived sessions.** New
|
|
69
|
+
`source: 'derived' | 'stamped'` option, default `'derived'`: sessions
|
|
70
|
+
are re-derived from raw timestamps via `sessionize()` — what Mixpanel
|
|
71
|
+
actually computes — instead of reading the generator's pre-stamped
|
|
72
|
+
`session_id`. Pass `source: 'stamped'` to restore the 1.5 reading; the
|
|
73
|
+
per-row `stampedDivergence` count audits the gap.
|
|
74
|
+
|
|
75
|
+
### 4. Generation output changes at the same seed (feature-gated)
|
|
76
|
+
|
|
77
|
+
- **`$experiment_started` is pinned to funnel-pass start.** Pre-existing
|
|
78
|
+
bug since 1.4.0: for experiment funnels with a shuffle `order`
|
|
79
|
+
(`last-fixed`, `random`, ...), the exposure event was shuffled into the
|
|
80
|
+
funnel body, compressing exposure→conversion TTC and undercounting
|
|
81
|
+
variant lift. It now stays at execution index 0. Output (event order +
|
|
82
|
+
RNG stream) changes for experiment funnels with shuffle orders;
|
|
83
|
+
`sequential` experiment funnels are unaffected.
|
|
84
|
+
- **Session IDs are re-derived after the `everything` hook.** Hooks that
|
|
85
|
+
mutate event times (TTC scaling, injected bursts) used to leave stale
|
|
86
|
+
`session_id` values that disagreed with query-time derivation. A second
|
|
87
|
+
relabel pass now runs on the final event set. Only affects dungeons
|
|
88
|
+
whose hooks move timestamps; identity fields are never touched.
|
|
89
|
+
- **Churn is a hard activity boundary.** Users who hit an `isChurnEvent`
|
|
90
|
+
(and don't roll a return) no longer carry events dated after the churn
|
|
91
|
+
event — their stream truncates at it. Only affects `isChurnEvent`
|
|
92
|
+
dungeons.
|
|
93
|
+
|
|
94
|
+
### 5. Hook helpers and patterns
|
|
95
|
+
|
|
96
|
+
- **Bin-based patterns bin by distinct days by default.**
|
|
97
|
+
`applyFrequencyByFrequency`, `applyFunnelFrequencyBreakdown`, and
|
|
98
|
+
`applyAggregateByBin` gain `binBy: 'events' | 'distinctDays'` (default
|
|
99
|
+
`'distinctDays'`) — matching how Mixpanel frequency reports bucket
|
|
100
|
+
users. Pass `binBy: 'events'` to restore the pre-1.6 axis.
|
|
101
|
+
- **`applyAttributedBySource` rewritten** to overwrite engine-stamped
|
|
102
|
+
touches (`{ weights, property, model: 'firstTouch'|'lastTouch'|'both' }`)
|
|
103
|
+
instead of stamping fresh values that land outside the touchpoint cap's
|
|
104
|
+
lookback and never move the report.
|
|
105
|
+
- **`applyTTCBySegment` deprecated** (still works, warns once). Use
|
|
106
|
+
`applyTTCBySegmentV2` (`everything` hook) — the funnel-post variant only
|
|
107
|
+
reaches the Mixpanel report for `isFirstFunnel` runs, because TTC
|
|
108
|
+
measures the first occurrence of each step per user.
|
|
109
|
+
- **New atoms**: `hashCohort`, `applyLifecycleWave`, `applyPathBias`,
|
|
110
|
+
`applySessionShape`, `binByDistinctPeriods`, `findFirstSequence`,
|
|
111
|
+
`scaleFunnelTTC`.
|
|
112
|
+
|
|
113
|
+
### 6. Experiments: `sticky` knob
|
|
114
|
+
|
|
115
|
+
**What**: `ExperimentConfig.sticky`, default `true`. Sticky bucketing —
|
|
116
|
+
the deterministic per-user hash that has always been the variant
|
|
117
|
+
assignment — is now explicit and opt-out-able. `sticky: false` re-rolls
|
|
118
|
+
the variant on every funnel pass via the seeded RNG.
|
|
119
|
+
|
|
120
|
+
```js
|
|
121
|
+
experiment: {
|
|
122
|
+
name: 'Checkout Redesign',
|
|
123
|
+
sticky: false, // multi-pass users can land in different variants
|
|
124
|
+
variants: [{ name: 'Control' }, { name: 'Treatment', conversionMultiplier: 1.25 }],
|
|
125
|
+
}
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
**Action needed**: None. The default is the pre-1.6 behavior.
|
|
129
|
+
|
|
130
|
+
### 7. Vertical dungeon layout
|
|
131
|
+
|
|
132
|
+
**What**: `dungeons/vertical/` restructured from flat files to one folder
|
|
133
|
+
per vertical: `dungeons/vertical/<name>/<name>.js` plus siblings
|
|
134
|
+
`<name>.verify.mjs` (verification script) and `<name>.sql` (DuckDB
|
|
135
|
+
inspection queries). Two new showcase dungeons: `streaming` (lifecycle)
|
|
136
|
+
and `support-desk` (flows + sessions). All verticals import engine
|
|
137
|
+
helpers via package subpaths (`@ak--47/dungeon-master/utils`,
|
|
138
|
+
`/hook-helpers`) — no relative `../../lib/` imports remain.
|
|
139
|
+
|
|
140
|
+
**Action needed**: Anything globbing `dungeons/vertical/*.js` inside
|
|
141
|
+
`node_modules` must switch to `dungeons/vertical/*/*.js` (the nested glob
|
|
142
|
+
matches only the dungeon — verify scripts are `.mjs`, SQL is `.sql`).
|
|
143
|
+
|
|
144
|
+
### 8. Deprecations
|
|
145
|
+
|
|
146
|
+
- `applyTTCBySegment` → `applyTTCBySegmentV2` (see §5).
|
|
147
|
+
- `Persona.churnRate`, `Persona.activeWindow`, `Persona.soupOverride` —
|
|
148
|
+
declared but never implemented; the validator now warns once per
|
|
149
|
+
process if a dungeon sets them.
|
|
150
|
+
- `carry_forward: true` retention option → `unbounded: 'carryForward'`.
|
|
151
|
+
|
|
152
|
+
## For DM4 specifically
|
|
153
|
+
|
|
154
|
+
Checklist derived from dm4's current usage surface (as of dm4 @ dm 1.5.4):
|
|
155
|
+
|
|
156
|
+
1. **`server/services/template-loader.js` — REQUIRED.** The
|
|
157
|
+
`dungeons/vertical/*.js` quickload glob finds nothing in 1.6.0; switch
|
|
158
|
+
to `dungeons/vertical/*/*.js`. The relative-import rewriting
|
|
159
|
+
(`../../lib/utils/utils.js` → package paths) is now a no-op — shipped
|
|
160
|
+
verticals already import `@ak--47/dungeon-master/utils` and
|
|
161
|
+
`/hook-helpers` directly. Vertical count is 22 (was 20): `streaming`
|
|
162
|
+
and `support-desk` are new.
|
|
163
|
+
2. **`verifier-canonical.js` sessionMetrics — numbers shift.** The
|
|
164
|
+
`{ type: 'sessionMetrics' }` call now derives sessions from timestamps
|
|
165
|
+
(§3). Recommended: keep the new default (it's what Mixpanel computes)
|
|
166
|
+
and re-baseline expected avg/median/p90; `source: 'stamped'` restores
|
|
167
|
+
1.5 readings if needed.
|
|
168
|
+
3. **`verifier-canonical.js` evaluateFunnel — counts can drop** for
|
|
169
|
+
non-`sequential` `funnelOrder` values and for configs passing
|
|
170
|
+
`exclusionSteps` (§3). Sequential funnels with no exclusions are
|
|
171
|
+
unchanged.
|
|
172
|
+
4. **`ALLOWED_DUNGEON_OPTION_KEYS` — no changes required.** 1.6.0 adds no
|
|
173
|
+
new top-level `Dungeon` config keys. The only new config field is
|
|
174
|
+
funnel-level `experiment.sticky` — add it wherever dm4 validates
|
|
175
|
+
`ExperimentConfig` shapes.
|
|
176
|
+
5. **Error-string regexes — re-run `error-messages` tests.** The SCD
|
|
177
|
+
credentials message is unchanged verbatim. New throw surfaces exist in
|
|
178
|
+
the verify layer (strict retention keys, funnel option conflicts) —
|
|
179
|
+
only relevant if dm4 starts calling those APIs with unknown options.
|
|
180
|
+
6. **Unchanged, no action**: `onProgress` payload contract (1.5.4 shape),
|
|
181
|
+
`dungeonToJSON` / `extractComments` (the `stories` named export is
|
|
182
|
+
deliberately NOT carried into JSON — the `.js` file stays the story
|
|
183
|
+
source of record), `validateSchema` report shape,
|
|
184
|
+
`buildIdentityMap` / `resolveUserId`, `_drop: true` profile semantics,
|
|
185
|
+
HOOKS.md §4 recipe numbering (4.29–4.31 appended, existing numbers
|
|
186
|
+
stable), `types.d.ts` / `HOOKS.md` package-file paths, validator
|
|
187
|
+
clamps (safe-range table identical to 1.5.1), two-arg
|
|
188
|
+
`DUNGEON_MASTER(config, overrides)` overload.
|
|
189
|
+
7. **Optional adoption**: the story layer (§1) is a better fit for dm4's
|
|
190
|
+
hook-plan/verify pipeline than free-form checks — generated dungeons
|
|
191
|
+
can emit a `stories` export and dm4 can shell out to
|
|
192
|
+
`scripts/verify-stories.mjs --json` for mechanical verdicts instead of
|
|
193
|
+
LLM-graded verification.
|
|
194
|
+
|
|
195
|
+
## Verifying the upgrade
|
|
196
|
+
|
|
197
|
+
```bash
|
|
198
|
+
npm install @ak--47/dungeon-master@1.6.0
|
|
199
|
+
npx vitest run # your suite; re-baseline any pinned emulator numbers per §3
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
For dungeons using experiments with shuffle orders, time-mutating hooks,
|
|
203
|
+
or `isChurnEvent`: regenerate any golden fixtures — same seed now produces
|
|
204
|
+
corrected (different) output per §4.
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
// ── IMPORTS ──
|
|
2
|
+
import { hashCohort, cloneEvent, dropEventsWhere } from '../../lib/hook-helpers/index.js';
|
|
3
|
+
/** @typedef {import("../../types").Dungeon} Config */
|
|
4
|
+
|
|
5
|
+
// ── OVERVIEW ──
|
|
6
|
+
/*
|
|
7
|
+
* NAME: stories-verify
|
|
8
|
+
* PURPOSE: P3.3 fixture — exercises the story runner (scripts/verify-stories.mjs)
|
|
9
|
+
* in both disk and in-memory modes. Small scale, fixed seed, and
|
|
10
|
+
* DETERMINISTIC hooks (index parity, hash cohorts — no RNG) so the
|
|
11
|
+
* story verdicts are stable run-to-run.
|
|
12
|
+
* SCALE: 200 users, ~12K events, 14 days
|
|
13
|
+
* EVENTS (2): browse, purchase
|
|
14
|
+
* FUNNELS (0): none
|
|
15
|
+
*
|
|
16
|
+
* Verified by: node scripts/verify-stories.mjs dungeons/technical/stories-verify.js
|
|
17
|
+
*/
|
|
18
|
+
|
|
19
|
+
// ── HOOK STORIES ──
|
|
20
|
+
/*
|
|
21
|
+
* One `everything` hook, plan pinned per user via hashCohort (the engine stamps
|
|
22
|
+
* the `plan` superProp randomly per event; the hook overwrites it per-user):
|
|
23
|
+
*
|
|
24
|
+
* H1: pro-plan browse amplification — every pro browse event is cloned
|
|
25
|
+
* BROWSE_CLONES more times, so pro browse volume is exactly
|
|
26
|
+
* (BROWSE_CLONES + 1)x its base. Aggregate pro/free count ratio also
|
|
27
|
+
* absorbs the hash-split cohort-size noise (Binomial(200, 0.5): the
|
|
28
|
+
* pro/free user ratio sits within ~[0.8, 1.25] at 3σ), so the a-priori
|
|
29
|
+
* floor is (BROWSE_CLONES + 1) * 0.8 rounded down to 2.0 — derived from
|
|
30
|
+
* the knobs, not tuned to observations.
|
|
31
|
+
* H2: free-plan purchase suppression — every 2nd free purchase is dropped
|
|
32
|
+
* (index parity, deterministic). Free purchase volume halves (keep =
|
|
33
|
+
* ceil(n/2) per user, so slightly over half survives at small n), putting
|
|
34
|
+
* the pro/free purchase ratio a bit under 2.0 — floor 1.5.
|
|
35
|
+
*/
|
|
36
|
+
|
|
37
|
+
// ── KNOBS (stories compute their thresholds from these) ──
|
|
38
|
+
const PRO_PCT = 50; // hashCohort split: ~half the users are plan=pro
|
|
39
|
+
const BROWSE_CLONES = 2; // pro browse volume = (BROWSE_CLONES + 1) x base
|
|
40
|
+
|
|
41
|
+
// ── CONFIG ──
|
|
42
|
+
/** @type {Config} */
|
|
43
|
+
const config = {
|
|
44
|
+
name: 'stories-verify',
|
|
45
|
+
seed: 'p33-stories-verify',
|
|
46
|
+
numUsers: 200,
|
|
47
|
+
numDays: 14,
|
|
48
|
+
avgEventsPerUserPerDay: 4,
|
|
49
|
+
percentUsersBornInDataset: 0,
|
|
50
|
+
format: 'json',
|
|
51
|
+
concurrency: 1,
|
|
52
|
+
writeToDisk: false,
|
|
53
|
+
verbose: false,
|
|
54
|
+
superProps: { plan: ['free', 'pro'] },
|
|
55
|
+
events: [
|
|
56
|
+
{ event: 'browse', weight: 8, properties: { surface: ['home', 'category', 'search'] } },
|
|
57
|
+
{ event: 'purchase', weight: 2, properties: { amount: [10, 20, 30, 40, 50] } },
|
|
58
|
+
],
|
|
59
|
+
hook: function (record, type, _meta) {
|
|
60
|
+
if (type !== 'everything' || !Array.isArray(record) || !record.length) return record;
|
|
61
|
+
const uid = record[0].user_id || record[0].distinct_id;
|
|
62
|
+
if (!uid) return record;
|
|
63
|
+
const isPro = hashCohort(uid, PRO_PCT);
|
|
64
|
+
const plan = isPro ? 'pro' : 'free';
|
|
65
|
+
for (const ev of record) ev.plan = plan;
|
|
66
|
+
if (isPro) {
|
|
67
|
+
// H1: clone each pro browse BROWSE_CLONES times (same timestamp —
|
|
68
|
+
// auto-sort after `everything` keeps the stream ordered).
|
|
69
|
+
const clones = [];
|
|
70
|
+
for (const ev of record) {
|
|
71
|
+
if (ev.event !== 'browse') continue;
|
|
72
|
+
for (let i = 0; i < BROWSE_CLONES; i++) clones.push(cloneEvent(ev));
|
|
73
|
+
}
|
|
74
|
+
record.push(...clones);
|
|
75
|
+
} else {
|
|
76
|
+
// H2: drop every 2nd free purchase (parity — deterministic, no RNG).
|
|
77
|
+
let n = 0;
|
|
78
|
+
dropEventsWhere(record, (e) => e.event === 'purchase' && (n++ % 2 === 1));
|
|
79
|
+
}
|
|
80
|
+
return record;
|
|
81
|
+
},
|
|
82
|
+
};
|
|
83
|
+
|
|
84
|
+
// ── STORIES ──
|
|
85
|
+
/** @type {import("../../types").DungeonStory[]} */
|
|
86
|
+
export const stories = [
|
|
87
|
+
{
|
|
88
|
+
id: 'H1-pro-browse-3x',
|
|
89
|
+
hook: 'H1',
|
|
90
|
+
archetype: 'cohort-count-scale',
|
|
91
|
+
narrative: `pro-plan users generate ${BROWSE_CLONES + 1}x browse volume (each pro browse cloned ${BROWSE_CLONES} more times)`,
|
|
92
|
+
assertions: [
|
|
93
|
+
{
|
|
94
|
+
breakdown: { type: 'eventBreakdown', event: 'browse', breakdownProperty: 'plan' },
|
|
95
|
+
select: {
|
|
96
|
+
pro: { where: { value: 'pro' } },
|
|
97
|
+
free: { where: { value: 'free' } },
|
|
98
|
+
},
|
|
99
|
+
// target from the knob; floor = target * 0.8 hash-split bound (see HOOK STORIES)
|
|
100
|
+
expect: { metric: 'pro.count / free.count', op: '>=', target: BROWSE_CLONES + 1, floor: 2.0 },
|
|
101
|
+
},
|
|
102
|
+
{
|
|
103
|
+
// single-ref sanity: the pro cohort is roughly half of 200 users
|
|
104
|
+
breakdown: { type: 'eventBreakdown', event: 'browse', breakdownProperty: 'plan' },
|
|
105
|
+
select: { pro: { where: { value: 'pro' } } },
|
|
106
|
+
expect: { metric: 'pro.total_users', op: 'between', target: [60, 140] },
|
|
107
|
+
},
|
|
108
|
+
],
|
|
109
|
+
},
|
|
110
|
+
{
|
|
111
|
+
id: 'H2-free-purchase-drop',
|
|
112
|
+
hook: 'H2',
|
|
113
|
+
archetype: 'composition-drift',
|
|
114
|
+
narrative: 'every 2nd free purchase dropped — pro/free purchase count ratio approaches 2x (ceil-keep parity leaves it slightly under)',
|
|
115
|
+
assertions: [
|
|
116
|
+
{
|
|
117
|
+
breakdown: { type: 'eventBreakdown', event: 'purchase', breakdownProperty: 'plan' },
|
|
118
|
+
select: {
|
|
119
|
+
pro: { where: { value: 'pro' } },
|
|
120
|
+
free: { where: { value: 'free' } },
|
|
121
|
+
},
|
|
122
|
+
expect: { metric: 'pro.count / free.count', op: '>=', target: 2.0, floor: 1.5 },
|
|
123
|
+
},
|
|
124
|
+
],
|
|
125
|
+
},
|
|
126
|
+
{
|
|
127
|
+
id: 'H2-duckdb-crosscheck',
|
|
128
|
+
hook: 'H2',
|
|
129
|
+
archetype: 'bespoke',
|
|
130
|
+
narrative: 'raw-shard SQL cross-check of the purchase suppression (duckdb escape hatch; disk mode only — skipped in-memory)',
|
|
131
|
+
assertions: [
|
|
132
|
+
{
|
|
133
|
+
breakdown: {
|
|
134
|
+
type: 'duckdb',
|
|
135
|
+
sql: "SELECT plan, count(*) AS n FROM read_json_auto('{{PREFIX}}-EVENTS*.json') WHERE event = 'purchase' GROUP BY plan",
|
|
136
|
+
},
|
|
137
|
+
assert: (rows) => {
|
|
138
|
+
const by = Object.fromEntries((rows || []).map(r => [r.plan, Number(r.n)]));
|
|
139
|
+
const ratio = by.pro / by.free;
|
|
140
|
+
const pass = Number.isFinite(ratio) && ratio >= 1.5;
|
|
141
|
+
return {
|
|
142
|
+
pass,
|
|
143
|
+
verdict: pass ? (ratio >= 1.8 ? 'NAILED' : 'STRONG') : 'NONE',
|
|
144
|
+
detail: `pro=${by.pro} free=${by.free} ratio=${Number.isFinite(ratio) ? ratio.toFixed(2) : 'n/a'}`,
|
|
145
|
+
};
|
|
146
|
+
},
|
|
147
|
+
},
|
|
148
|
+
],
|
|
149
|
+
},
|
|
150
|
+
];
|
|
151
|
+
|
|
152
|
+
export default config;
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
# Vertical Dungeons — Proof of Story-in-Data
|
|
2
|
+
|
|
3
|
+
One folder per vertical dungeon (v1.6 layout). Each
|
|
4
|
+
`dungeons/vertical/<name>/` holds three sibling files:
|
|
5
|
+
|
|
6
|
+
- **`<name>.js`** — the dungeon, exporting `stories` alongside the default
|
|
7
|
+
config. Stories are the machine-checkable contract for every numbered
|
|
8
|
+
hook; evaluate them with
|
|
9
|
+
`node scripts/verify-stories.mjs dungeons/vertical/<name>/<name>.js`.
|
|
10
|
+
- **`<name>.verify.mjs`** — Node script using `@ak--47/dungeon-master/verify`
|
|
11
|
+
primitives (`emulateBreakdown`, `evaluateFunnel`, `buildIdentityMap`,
|
|
12
|
+
`resolveUserId`). Emulator-backed where Mixpanel-equivalent semantics
|
|
13
|
+
matter; per-user JS aggregation where bespoke. Each `check()` call asserts
|
|
14
|
+
one engineered hook produces measurable signal in the generated data.
|
|
15
|
+
- **`<name>.sql`** — DuckDB queries for human-eyeball inspection of the
|
|
16
|
+
same hooks. Run after generation to inspect raw outputs the way an
|
|
17
|
+
analyst would in Mixpanel.
|
|
18
|
+
|
|
19
|
+
These files are the **proof** that the engineered story patterns documented
|
|
20
|
+
in each dungeon's top-level comment block actually appear in the generated
|
|
21
|
+
data — at full fidelity, not at smoke-test scale.
|
|
22
|
+
|
|
23
|
+
## Coverage
|
|
24
|
+
|
|
25
|
+
All 22 vertical dungeons. Every documented hook has a story-backed
|
|
26
|
+
verification check — no documented hook is unverified. (Aggregate hook
|
|
27
|
+
counts are rebuilt at each release; see the per-dungeon stories exports
|
|
28
|
+
for the authoritative contract.)
|
|
29
|
+
|
|
30
|
+
Score column regenerated 2026-07-04 from the fix-round sweep: every
|
|
31
|
+
dungeon regenerated at full fidelity (its shipped `numUsers`) under the
|
|
32
|
+
post-review verifier and scored by `scripts/verify-stories.mjs` — the
|
|
33
|
+
score is the worst story verdict the runner printed, not an editorial
|
|
34
|
+
judgment. 15 NAILED / 7 STRONG, 22/22 at STRONG or better. This is
|
|
35
|
+
lower than the 2026-07-03 headline (20/2) **by design**: the fix-round
|
|
36
|
+
band policy below re-derived NAILED bands from hook knobs, and verdicts
|
|
37
|
+
that previously leaned on measurement-anchored bands now grade STRONG
|
|
38
|
+
honestly. No hook regressed — the data is unchanged; the grading got
|
|
39
|
+
stricter.
|
|
40
|
+
|
|
41
|
+
| # | Dungeon | Score | Hooks | Iter |
|
|
42
|
+
|---|---------|-------|-------|------|
|
|
43
|
+
| 1 | fitness | NAILED | 10/10 | 2 |
|
|
44
|
+
| 2 | dating | NAILED | 10/10 | 2 |
|
|
45
|
+
| 3 | community | NAILED | 10/10 | 3 |
|
|
46
|
+
| 4 | travel | NAILED | 10/10 | 2 |
|
|
47
|
+
| 5 | logistics | NAILED | 10/10 | 3 |
|
|
48
|
+
| 6 | education | STRONG | 10/10 | 3 |
|
|
49
|
+
| 7 | real-estate | NAILED | 10/10 | 2 |
|
|
50
|
+
| 8 | insurance-application | STRONG | 10/10 | 3 |
|
|
51
|
+
| 9 | food-delivery | STRONG | 10/10 | 2 |
|
|
52
|
+
| 10 | devtools | NAILED | 10/10 | 1 |
|
|
53
|
+
| 11 | healthcare | NAILED | 10/10 | 2 |
|
|
54
|
+
| 12 | fintech | NAILED | 10/10 | 2 |
|
|
55
|
+
| 13 | ai-platform | NAILED | 10/10 | 3 |
|
|
56
|
+
| 14 | marketplace | NAILED | 10/10 | 2 |
|
|
57
|
+
| 15 | media | NAILED | 10/10 | 2 |
|
|
58
|
+
| 16 | ecommerce | NAILED | 10/10 | 3 |
|
|
59
|
+
| 17 | sass | STRONG | 11/11 | 2 |
|
|
60
|
+
| 18 | gaming | STRONG | 13/13 | 2 |
|
|
61
|
+
| 19 | social | STRONG | 10/10 | 2 |
|
|
62
|
+
| 20 | crypto | NAILED | 11/11 | 2 |
|
|
63
|
+
| 21 | streaming | NAILED | 4/4 | 2 |
|
|
64
|
+
| 22 | support-desk | STRONG | 3/3 | 1 |
|
|
65
|
+
|
|
66
|
+
**Score legend.** Grading is mechanical (the runner reports where the
|
|
67
|
+
measured value landed), but the bands themselves are author-declared —
|
|
68
|
+
band *selection* is an editorial act, and the score is only as honest
|
|
69
|
+
as the band derivation. The fix-round band policy (2026-07-04):
|
|
70
|
+
- A NAILED band must be derived from the hook's knob (knob ±10%),
|
|
71
|
+
never from a measurement of the dungeon's own output — a band
|
|
72
|
+
centered on a measurement passes by construction.
|
|
73
|
+
- Where the realized magnitude is confounded (selection effects,
|
|
74
|
+
mixtures, budget attenuation), the assertion uses a knob-derived
|
|
75
|
+
floor or ceiling instead, which grades STRONG by design. STRONG is
|
|
76
|
+
not a blemish: it is the honest verdict for a real effect whose
|
|
77
|
+
exact magnitude is not knob-derivable.
|
|
78
|
+
- **NAILED** — every assertion in every story landed inside its
|
|
79
|
+
knob-derived NAILED band at full fidelity.
|
|
80
|
+
- **STRONG** — every assertion passed, but at least one landed in the
|
|
81
|
+
STRONG band outside the NAILED band (or is a floor/ceiling check).
|
|
82
|
+
Each such case documents why in its story's derivation notes.
|
|
83
|
+
|
|
84
|
+
## Running
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
# 1. Generate fresh data (full fidelity — uses dungeon's shipped numUsers)
|
|
88
|
+
node scripts/verify-runner.mjs dungeons/vertical/${NAME}/${NAME}.js verify-${NAME}
|
|
89
|
+
|
|
90
|
+
# 2. Evaluate the dungeon's stories (five-tier verdict table)
|
|
91
|
+
node scripts/verify-stories.mjs dungeons/vertical/${NAME}/${NAME}.js --data-prefix verify-${NAME}
|
|
92
|
+
|
|
93
|
+
# 3. Run the .mjs verifier (CI gate)
|
|
94
|
+
node --max-old-space-size=4096 dungeons/vertical/${NAME}/${NAME}.verify.mjs
|
|
95
|
+
|
|
96
|
+
# 4. (Optional) Run the SQL for human inspection
|
|
97
|
+
duckdb -c ".read dungeons/vertical/${NAME}/${NAME}.sql"
|
|
98
|
+
|
|
99
|
+
# 5. Cleanup
|
|
100
|
+
rm -f data/verify-${NAME}-*
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## Verify-script anatomy
|
|
104
|
+
|
|
105
|
+
Every `<name>.verify.mjs` follows the same template (see
|
|
106
|
+
[HOOKS.md §9.9](../../HOOKS.md#99-per-dungeon-verify-script-template)):
|
|
107
|
+
|
|
108
|
+
1. Stream-load shards (`readline.createInterface`) — handles dungeons up to
|
|
109
|
+
~1M events without `readFileSync`'s 512MB cap.
|
|
110
|
+
2. Build identity map (`buildIdentityMap(profiles)`) and per-user event
|
|
111
|
+
bucket (`resolveUserId`).
|
|
112
|
+
3. One `check(name, pass, detail)` per documented hook.
|
|
113
|
+
4. Exit non-zero if any check fails.
|
|
114
|
+
|
|
115
|
+
## Reading the SQL
|
|
116
|
+
|
|
117
|
+
SQL files are human-readable Mixpanel inspection queries. They name each
|
|
118
|
+
hook the same way as the `.verify.mjs` file and the dungeon's docstring.
|
|
119
|
+
Numbering is consistent across all three artifacts:
|
|
120
|
+
|
|
121
|
+
- Dungeon comment: `* 1. WHALE WALLETS (everything)`
|
|
122
|
+
- `<name>.verify.mjs`: `check('H1 whale 5x+ trade amount', ...)`
|
|
123
|
+
- `<name>.sql`: `-- Hook 1: WHALE WALLETS — top 2% drive most volume`
|
|
124
|
+
|
|
125
|
+
## Known limitations (historical — list now empty)
|
|
126
|
+
|
|
127
|
+
Several engineered hooks intentionally use `funnel-post` to compress
|
|
128
|
+
time-to-convert within a single funnel-instance. The verifier's
|
|
129
|
+
`evaluateFunnel` is a greedy single-pass over the user's full event
|
|
130
|
+
history — it picks the first matching event for each step regardless of
|
|
131
|
+
which funnel-instance the hook touched. Through v1.5 this forced some
|
|
132
|
+
dungeons to check TTC hooks for population presence only. As of v1.6
|
|
133
|
+
every affected dungeon (ai-platform, dating, community, travel,
|
|
134
|
+
logistics, education, real-estate, devtools, marketplace, crypto) has
|
|
135
|
+
graduated: their TTC stories assert the TTC delta itself through
|
|
136
|
+
`emulateBreakdown`'s `timeToConvert` at conversion windows covering the
|
|
137
|
+
stretched support (see each story narrative for the censoring
|
|
138
|
+
analysis). Three graduation lessons generalize: pick the conversion
|
|
139
|
+
window where each cohort's TTC distribution is unimodal (media — at
|
|
140
|
+
multi-day windows the median sits on a bimodal mode boundary and flips
|
|
141
|
+
on sampling noise); restrict funnel-post scaling to the target
|
|
142
|
+
funnel when another funnel shares a step prefix (marketplace — scaling
|
|
143
|
+
everything let the greedy evaluator assemble chains across unscaled
|
|
144
|
+
instances, collapsing the read); and anchor the scaled funnel on a
|
|
145
|
+
unique first step (crypto — `wallet connected` is `isFirstEvent` +
|
|
146
|
+
`isAuthEvent`, so it occurs exactly once per user and the greedy
|
|
147
|
+
evaluator has no earlier instance to latch onto, making the read
|
|
148
|
+
stable across 1h–24h windows).
|
|
149
|
+
|
|
150
|
+
See
|
|
151
|
+
[`research/1.5.0-vertical-eval.md`](../../research/1.5.0-vertical-eval.md)
|
|
152
|
+
for the aggregate evaluation methodology and pattern catalog, and
|
|
153
|
+
[HOOKS.md §9](../../HOOKS.md#9-verification-patterns-from-the-v150-vertical-eval)
|
|
154
|
+
for the verification recipe encyclopedia.
|