@ak--47/dungeon-master 1.7.0 → 1.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/.claude/skills/analyze-soup/SKILL.md +30 -11
  2. package/.claude/skills/create-dungeon/SKILL.md +84 -44
  3. package/.claude/skills/create-project/SKILL.md +28 -3
  4. package/.claude/skills/create-project/context.mjs +89 -0
  5. package/.claude/skills/create-project/provision.mjs +1 -60
  6. package/.claude/skills/headless-build/SKILL.md +39 -12
  7. package/.claude/skills/powertools/SKILL.md +26 -3
  8. package/.claude/skills/release-check/SKILL.md +124 -0
  9. package/.claude/skills/verify-dungeon/SKILL.md +103 -29
  10. package/.claude/skills/verify-dungeon/references/alignment-contract.md +84 -0
  11. package/.claude/skills/verify-dungeon/references/counting-semantics.md +41 -16
  12. package/.claude/skills/verify-dungeon/references/report-format.md +41 -10
  13. package/.claude/skills/verify-dungeon/references/sql-recipes.md +171 -226
  14. package/.claude/skills/warehouse-metrics/GAPS-template.md +34 -0
  15. package/.claude/skills/warehouse-metrics/SKILL.md +111 -0
  16. package/.claude/skills/warehouse-metrics/deploy.mjs +651 -0
  17. package/.claude/skills/write-hooks/SKILL.md +94 -51
  18. package/CHANGELOG.md +183 -0
  19. package/HOOKS.md +165 -18
  20. package/README.md +265 -1
  21. package/docs/guides/1.8.0-upgrade-guide.md +151 -0
  22. package/docs/guides/1.8.1-upgrade-guide.md +153 -0
  23. package/dungeons/technical/warehouse.js +187 -0
  24. package/index.js +116 -2
  25. package/lib/core/config-validator.js +21 -0
  26. package/lib/core/dungeon-loader.js +1 -1
  27. package/lib/core/storage.js +51 -3
  28. package/lib/generators/events.js +6 -0
  29. package/lib/generators/funnels.js +15 -0
  30. package/lib/generators/standalone.js +248 -0
  31. package/lib/generators/warehouse.js +828 -0
  32. package/lib/hook-helpers/shape.js +73 -17
  33. package/lib/orchestrators/mixpanel-sender.js +27 -2
  34. package/lib/orchestrators/user-loop.js +83 -15
  35. package/lib/templates/story-spec.schema.json +41 -16
  36. package/lib/utils/utils.js +37 -12
  37. package/lib/verify/funnel-engine.js +66 -26
  38. package/lib/verify/index.js +1 -0
  39. package/lib/verify/story-runner.js +71 -8
  40. package/lib/verify/warehouse.js +683 -0
  41. package/package.json +4 -2
  42. package/scripts/verify-stories.mjs +150 -44
  43. package/types.d.ts +312 -9
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: headless-build
3
3
  description: Use after a dungeon's data is loaded into a real Mixpanel project — builds the full demoable environment with mixpanel-headless (dashboards, charts, Lexicon, cohorts, custom properties, behaviors/metrics/formulas, annotations) targeted at that dungeon's engineered stories, then verifies the stories still read live. Final step after create-dungeon / write-hooks / verify-dungeon / create-project.
4
- argument-hint: [dungeon path, e.g. dungeons/user/nyc-dcp/nyc-dcp.js]
4
+ argument-hint: '[dungeon path, e.g. dungeons/user/nyc-dcp/nyc-dcp.js]'
5
5
  model: claude-opus-4-6
6
6
  effort: max
7
7
  ---
@@ -14,16 +14,37 @@ Last step of the pipeline:
14
14
  /create-dungeon → /write-hooks → /verify-dungeon → /create-project → /headless-build
15
15
  ```
16
16
 
17
+ For `warehouseMetrics`, insert `/warehouse-metrics` after verification and project
18
+ provisioning, before this build. Use the verified local disk artifact prefix.
19
+ Ordinary event import loads `standaloneEvents` but does not deploy warehouse tables.
20
+
21
+ Keep cadence and warehouse metrics separate from people analyses. Synthetic
22
+ standalone `distinct_id` values identify series; never count them as users or
23
+ include them in funnels, retention, or identity stitching. Build aggregate reports
24
+ from their declared numeric properties and dimensions.
25
+
26
+ For warehouse-backed charts, read the manifest and saved metric ids from the
27
+ warehouse handoff. `history` may extend the metric window before the user-event
28
+ window; preserve that range and label synthetic backfill. Check saved definitions
29
+ and `previewWarehouseMetric` results before narrating values. A successful
30
+ `refreshWarehouseMetric` only invalidates cache; it does not execute the query.
31
+ Use the existing `/macro/setup-bq-warehouse` flow through `/warehouse-metrics`
32
+ for source setup. Do not reimplement source creation or IAM grants in build code.
33
+
17
34
  The project already has data. This skill builds everything a human sees: themed
18
35
  dashboards whose narrative is computed from the live data, an annotated Lexicon,
19
36
  saved cohorts and custom properties, saved behaviors/metrics/formulas, and
20
37
  annotations that explain the engineered moments. Then it re-measures the hook
21
38
  stories **against the live project** and fails if they no longer read.
22
39
 
23
- That last step is the point. Local story verification passes on the generated
24
- array in memory; it never sees what survived ingest. A build can render five
40
+ Local story verification uses generated records in memory or retained disk
41
+ artifacts. It does not establish what survived ingest. A build can render five
25
42
  perfect dashboards on top of a story that silently collapsed on the way in.
26
43
 
44
+ Apply the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md)
45
+ when translating local stories into live reports. This build requires authorized
46
+ online work; it is never an automatic step of offline verification.
47
+
27
48
  ## Scope
28
49
 
29
50
  In scope: dashboards + charts, Lexicon enrichment, cohorts, custom properties,
@@ -165,13 +186,14 @@ size, require each bucket to be observed END TO END (`(b+1)*unit - 1` days of hi
165
186
  not `b*unit`), and cap the cohort date. Put both numbers on the board — the diluted
166
187
  one is what the chart shows, and naming why is a better demo than hiding it.
167
188
 
168
- **Time-to-convert is a MEAN, and means are tail-dominated.** Funnel frames carry
189
+ **These funnel response fields report means.** Funnel frames carry
169
190
  `avg_time` and `avg_time_from_start` (seconds; not monotonic across steps — each is
170
191
  over that step's own survivors). A dungeon knob expressed as a median ratio will not
171
192
  reproduce: Square's designed 13x median gap measured 1.95x as a mean over a 30-day
172
- window. Use the **speed curve** instead run the same funnel at 1/3/7/14-day
193
+ window. A **speed curve** can supplement the report: run the same funnel at 1/3/7/14-day
173
194
  conversion windows and read what share of each segment's eventual conversions had
174
- landed by then. Same effect, expressed in a statistic Mixpanel actually computes.
195
+ landed by then. Preserve the requested real report and its statistic. A speed
196
+ curve cannot silently replace its acceptance check.
175
197
 
176
198
  ### 3. Auto-detect the data window
177
199
 
@@ -243,10 +265,14 @@ filters have to move into `show[i].behavior.filters`.
243
265
  - **structure** — every registered entity still exists.
244
266
  - **stories** — re-measure each hook effect live and compare to the knob.
245
267
 
246
- Report MATCH / DIRECTIONAL / MISS per story and exit non-zero on any MISS.
247
- Use a wider tolerance than the dungeon's own ±10% bar (~25%): Mixpanel's cohort
248
- membership is computed over the whole window, not the dungeon's internal binning,
249
- so depth-band style cohorts will not line up exactly.
268
+ Report MATCH / DIRECTIONAL / MISS / INSUFFICIENT_EVIDENCE per story. Exit non-zero
269
+ on a MISS or unresolved evidence gap; insufficient eligible users or converters
270
+ are neither a pass nor a measured miss. Use each story's declared tolerance.
271
+ Document report-semantic differences separately rather than widening tolerances.
272
+ Preserve explicit counting options: local totals default to `reentry: false`.
273
+ Derive local sessions from the full user stream before HPC partitioning; local
274
+ defaults are UTC, 30-minute idle timeout, and 24-hour maximum. Project exclusions
275
+ and timezone variants remain outside the source-derived proof.
250
276
 
251
277
  ### 7. Report
252
278
 
@@ -296,8 +322,9 @@ every build. Prefer one report with N metrics wherever the chart allows it.
296
322
  **`ws._api_client` is lazy** — None immediately after construction. Force it via
297
323
  `ws._get_api_client()` before patching request headers.
298
324
 
299
- **Idempotency.** Dashboards: delete-by-title then recreate. Everything else:
300
- look up by name and reuse. Re-running must never duplicate.
325
+ **Idempotency.** Dashboards: create the replacement first, then retire only the
326
+ recorded old ids after confirmation. Everything else: look up by name and reuse.
327
+ Re-running must never duplicate.
301
328
 
302
329
  ## Commands
303
330
 
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: powertools
3
3
  description: Use when any task needs the Mixpanel Power Tools API ("use powertools") — schema export (get-schema), event volumes, project CRUD, query methods, macros, or snapshotting a prod project's schema to copy it into a dungeon. Companion to create-project (which handles provisioning specifically).
4
- argument-hint: [what to do, e.g. "get schema for project 12345" or "copy project 12345 into a dungeon"]
4
+ argument-hint: '[what to do, e.g. "get schema for project 12345" or "copy project 12345 into a dungeon"]'
5
5
  ---
6
6
 
7
7
  # Power Tools API
@@ -10,7 +10,9 @@ Base URL: `https://mixpanel-power-tools-api-lmozz6xkha-uc.a.run.app`
10
10
 
11
11
  ## Auth model — read this first
12
12
 
13
- - **GET any endpoint path = documentation, no auth.** Always `curl -s GET <base><path>` before first use of an unfamiliar endpoint — docs include exact body params and response shapes.
13
+ - **GET any endpoint path = documentation, no auth.** For authorized online work,
14
+ GET an unfamiliar endpoint's docs before use. During offline verification, use
15
+ local references and report unavailable documentation; make no network calls.
14
16
  - **POST = execute.** `Authorization: Bearer <oauth-token>` (employee OAuth from repo `.env` `BEARER_TOKEN`, or a customer's OAuth token) or `Basic base64(service_acct:secret)`.
15
17
  - **Customer OAuth tokens are accepted** (verified 2026-07-06): `/auth` and `/macro/get-schema` work with a customer token on projects that token can access. `ai_endpoints_allowed: false` for non-employees — the `ai-*` family stays employee-only.
16
18
  - **Every POST body** should include `client_id: "dungeon-master"` and `region` (`US` default).
@@ -43,7 +45,28 @@ node .claude/skills/powertools/snapshot-project.mjs <project_id> --bearer <token
43
45
 
44
46
  ## Endpoint catalog (the useful subset)
45
47
 
46
- GET the path for full docs. Full list: GET `/` and GET `/macro`.
48
+ ### Warehouse handoff (v1.8.0)
49
+
50
+ For a dungeon with `warehouseMetrics`, use `/warehouse-metrics` after local
51
+ verification produces disk tables and a manifest, and `/create-project` writes
52
+ the project id. Pass the explicit verified `--data-prefix`; preserve those files.
53
+ That skill owns load, source setup, SQL preview, and metric create-or-skip.
54
+ Reuse `/macro/setup-bq-warehouse` through `pt.mjs`; do not reimplement its
55
+ source creation, GCP IAM grant, or dataset ACL handling here.
56
+
57
+ Warehouse metric endpoints include `getWarehouseMetrics`, `previewWarehouseMetric`,
58
+ `createWarehouseMetric`, and `refreshWarehouseMetric` in the `/crud` family.
59
+ Read endpoint docs before live use. Preview executes the read query; create does
60
+ not validate SQL, and refresh only invalidates cache. A docs-route 404 uses the
61
+ warehouse skill's manual-setup fallback; other errors must surface.
62
+
63
+ `standaloneEvents` imports as a separate event stream through the ordinary sender.
64
+ Its synthetic ids are series keys, so schema snapshots and dashboard counts must
65
+ distinguish cadence telemetry from user `events[]`. It is not a warehouse source.
66
+
67
+ During authorized online work, GET the path for full docs. Full list: GET `/`
68
+ and GET `/macro`. Preserve explicit report options when comparing local and live
69
+ queries; see the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md).
47
70
 
48
71
  **crud** — 187 endpoints. GET `/crud` for the full list.
49
72
 
@@ -0,0 +1,124 @@
1
+ ---
2
+ name: release-check
3
+ description: "Use before a dungeon-master release to audit versioned docs, skills, tests, determinism, package contents, and Git state; report release blockers and prepare an explicitly authorized PR and merge handoff."
4
+ argument-hint: "[version, e.g. 1.8.0] [optional base branch]"
5
+ ---
6
+
7
+ # Release check
8
+
9
+ Check the current checkout before release. Read `AGENTS.md`, `package.json`,
10
+ the target changelog entry, and its upgrade guide. Default to validation only.
11
+ Never publish to npm unless the user gives explicit authorization for npm publishing.
12
+ An instruction to merge a PR is not permission to publish a package.
13
+
14
+ ## Establish the release scope
15
+
16
+ 1. Inspect `git status --short`, the current branch, remotes, and recent history.
17
+ Preserve unrelated changes. Never reset or clean the worktree to make checks pass.
18
+ 2. Confirm the requested version matches `package.json`, the lockfile, changelog,
19
+ and guide. Do not invent a release date or bump a version without authorization.
20
+ 3. Compare the release diff against the base branch. Check exports, dependencies,
21
+ config defaults, compatibility notes, examples, and migration instructions.
22
+ 4. Inspect test setup before running it. This repo's suite prunes `data/` and `tmp/`.
23
+ If pending deployment artifacts exist, preserve them outside those directories
24
+ or obtain consent before running the suite. Do not silently delete verified inputs.
25
+
26
+ ## Run executable gates
27
+
28
+ Use `set -o pipefail` for every piped command. Keep all Vitest output behind
29
+ `2>&1 | tail -50`; a successful tail command alone is not passing evidence.
30
+
31
+ ```bash
32
+ set -o pipefail
33
+ npm test 2>&1 | tail -50
34
+ npm run typecheck
35
+ git diff --check
36
+ npx vitest run tests/unit/engine-shape-canary.test.js 2>&1 | tail -50
37
+ RUN_FULL_SWEEP=1 npx vitest run tests/e2e/engine-shape-full-sweep.test.js 2>&1 | tail -50
38
+ node tests/engine/smoke-test-all.mjs
39
+ npm pack --dry-run --json
40
+ ```
41
+
42
+ Run commands separately or chain with `&&` so a failure cannot disappear behind
43
+ a later success. Record exit codes, pass/skip counts, and the tested commit/diff.
44
+ The smoke script discovers vertical dungeons only. Verify changed technical or
45
+ customer fixtures separately with the existing verification runners.
46
+
47
+ `npm test` excludes alignment. Run its independent gates separately:
48
+
49
+ ```bash
50
+ node tests/alignment/run.mjs
51
+ node tests/alignment/run.mjs --sweep --timeout-ms=600000
52
+ ```
53
+
54
+ Run the sweep when required by release scope. Both commands enforce macOS network
55
+ denial and a ten-minute maximum. If editor tasks are unavailable, use these
56
+ repository commands. If the runner or platform is unavailable, report NOT RUN;
57
+ do not substitute unsandboxed execution. Offline checks must use installed
58
+ dependencies, OS network denial, and empty global/setup-file lists. The default
59
+ suite above is not the offline audit route: do not run pruning setup, customer
60
+ evaluations, imports, or industry-generation scripts during an offline docs audit.
61
+ For a skills-only audit, run only `tests/e2e/skills-contract.test.js` with
62
+ `configFile: false`, that exact include, `globals: true`, `globalSetup: []`,
63
+ `setupFiles: []`, `fileParallelism: false`, and `sequence.concurrent: false`
64
+ through programmatic Vitest under `sandbox-exec` network denial.
65
+
66
+ For generation changes, require the event-stream determinism test. Compare runs
67
+ with the same seed, pinned window, and `concurrency: 1`; strip only `insert_id`.
68
+ Use sequential test cases because the RNG is shared. Include warehouse noise in
69
+ warehouse determinism checks. Never accept an unrun or failing determinism gate.
70
+
71
+ ## Audit docs and skills
72
+
73
+ - Run `tests/e2e/skills-contract.test.js`. Parse YAML frontmatter rather than
74
+ guessing from appearance: names match folders, descriptions are strings, and
75
+ `argument-hint` is a quoted string.
76
+ - The canonical skills live in `.claude/skills`. Verify `.agents/skills` and
77
+ `.github/skills` resolve to that directory. Do not duplicate skill content.
78
+ - Check README, HOOKS, type comments, changelog, and guide against actual code.
79
+ Do not carry stale test counts, old skill paths, or historical operational
80
+ failures forward as current release claims.
81
+ - Audit the documented order of the complete handoff: author, optional hooks, verify, provision,
82
+ generate/import, optional `/warehouse-metrics`, then headless build.
83
+ Offline release verification does not execute provisioning, imports, warehouse
84
+ deployment, or headless builds.
85
+ - Distinguish user events, identity-less `standaloneEvents`, and warehouse rows.
86
+ Synthetic IDs are not people. Standalone hooks return retained records;
87
+ warehouse hooks mutate rows and ignore returns. Preserve deployment artifacts.
88
+ - If API integrations changed, read local endpoint docs and handback evidence
89
+ during offline checks. Fetch endpoint docs only during authorized online work.
90
+ Distinguish docs discovery, mocked execution, dry-run, and live verification.
91
+ Live writes require explicit authorization. Never claim an endpoint GET proves
92
+ authenticated write permission or end-to-end deployment.
93
+
94
+ ## Inspect the package and internal files
95
+
96
+ Read the file list from `npm pack --dry-run --json`. Confirm new runtime modules,
97
+ dependencies, scripts, and canonical skill files ship. Reject credentials,
98
+ customer data, `plans/`, `research/`, `.superpowers/`, or obsolete skill paths.
99
+ Check symlink handling in the package separately from workspace discovery.
100
+
101
+ Verify `git ls-files .superpowers` is empty and `.gitignore` covers the directory.
102
+ Ignoring an already tracked file does not untrack it. Obtain authorization before
103
+ removing index entries; preserve local reports. Archive completed plan folders
104
+ whole under `plans/archived/`; leave active or ambiguous work in place.
105
+
106
+ ## Report and optional shipping handoff
107
+
108
+ Report blockers first, then evidence, skipped checks, compatibility changes,
109
+ and remaining operational limits. Do not fix unrelated failures or weaken tests.
110
+ Apply the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md):
111
+ report source-derived scope and remaining gaps, separate insufficient evidence
112
+ from passes and measured failures, and preserve actual deployment reports.
113
+
114
+ Only when explicitly requested: inspect all staged files for secrets, commit the
115
+ authorized changes, push the feature branch, create or reuse its PR, and inspect
116
+ CI and mergeability. Require successful completed checks for the current PR head
117
+ before squash merge. If no CI checks are configured, report that explicitly and
118
+ use recorded local release gates; do not describe absent CI as passing. A failed
119
+ release gate requires investigation and an explicit operator decision before shipping.
120
+ Never bypass protections or use admin merge
121
+ to hide a failing check. Confirm the PR is merged before switching local branches.
122
+ Fetch, switch to `main`, and pull with `--ff-only`; stop rather than discard local
123
+ changes or divergent commits. Report the PR URL, merge SHA, current branch, and
124
+ working-tree status. Leave npm publishing to the operator unless separately authorized.
@@ -1,14 +1,14 @@
1
1
  ---
2
2
  name: verify-dungeon
3
- description: Use when a dungeon's hooks need verification before pushing data to Mixpanel runs the dungeon at full scale, evaluates the dungeon's `stories` export with the mechanical story runner (verify-stories.mjs), validates schema integrity (catches flag-stamping), and reports per-hook NAILED/STRONG/WEAK/NONE/INVERSE verdicts. LLM investigation is reserved for failures and duckdb-type assertions.
4
- argument-hint: [dungeon path(s), e.g. dungeons/vertical/gaming/gaming.js or dungeons/vertical/fintech/fintech.js]
3
+ description: Use when a dungeon needs verification before import or warehouse deployment, including standaloneEvents cadence streams and warehouseMetrics tables. Runs full-scale story checks, schema validation, and automatic warehouse audits even without stories. Reports per-hook NAILED/STRONG/WEAK/NONE/INVERSE verdicts; investigates failures and duckdb assertions.
4
+ argument-hint: '[dungeon path(s), e.g. dungeons/vertical/gaming/gaming.js or dungeons/vertical/fintech/fintech.js]'
5
5
  model: claude-opus-4-6
6
6
  effort: max
7
7
  ---
8
8
 
9
9
  # Verify Dungeon
10
10
 
11
- Verify a dungeon at full scale: run the story runner (`scripts/verify-stories.mjs`) as the primary mechanical check, validate schema integrity, investigate only what the runner can't settle (failures + `duckdb`-type assertions), write a single consolidated `hook-results.md`.
11
+ Verify a dungeon at its configured scale: run `scripts/verify-stories.mjs`, validate schema, and review report semantics and evidence sufficiency for every story. Investigate misses, bespoke SQL, and semantic gaps, then write one consolidated `hook-results.md`.
12
12
 
13
13
  **Dungeon file(s):** `$ARGUMENTS` — single path, multiple space-separated paths, or glob pattern. In batch mode, process each dungeon sequentially through Steps 1–5, then write one consolidated report in Step 7. Use a unique `name` prefix per dungeon (e.g., `verify-fintech`, `verify-gaming`) so output files don't collide.
14
14
 
@@ -16,6 +16,7 @@ Verify a dungeon at full scale: run the story runner (`scripts/verify-stories.mj
16
16
 
17
17
  Load these on demand:
18
18
 
19
+ - [references/alignment-contract.md](references/alignment-contract.md) - required 1.8.1 proof contract: independent report specification, neutral controls, sufficient populations, and source-derived limits.
19
20
  - [references/counting-semantics.md](references/counting-semantics.md) — Mixpanel counting rules, when to use the emulator vs DuckDB, emulator analysis types, identity-model verification, time-series, common gotchas.
20
21
  - [references/sql-recipes.md](references/sql-recipes.md) — every DuckDB query template (schema validation, identity / experiment invariants, hook archetype queries, pitfalls, TTC verification, dataset-window computation).
21
22
  - [references/report-format.md](references/report-format.md) — single + multi-dungeon report templates, per-hook detail block, verdict criteria, query log format, mandatory verification SQL file for user dungeons.
@@ -35,17 +36,24 @@ Read the dungeon at `$ARGUMENTS`. If it's a bare filename (no `/`), check `dunge
35
36
 
36
37
  For each hook/pattern, catalog:
37
38
  - Hook number and name (e.g., "Hook #1: Compass users have 3x quest completion")
38
- - Hook type (`event`, `everything`, `funnel-pre`, `funnel-post`, `user`, `scd-pre`)
39
+ - Hook type (`event`, `everything`, `funnel-pre`, `funnel-post`, `user`, `scd-pre`, `standalone`, `warehouse`)
39
40
  - Mechanism — what the code actually does
40
41
  - Expected signal — specific, measurable outcome (e.g., "compass_user=true events should have ~1.5x reward_gold")
41
- - Which output file the signal lives in (events, users, groups)
42
+ - Which output file the signal lives in (user events, users, groups, standalone cadence shards, warehouse tables)
42
43
  - Mixpanel report instructions — flag missing/vague ones for the report
43
44
 
45
+ Before running, write the independent report specification from the intended
46
+ analysis, then compare the assertion's actual options and defaults with it.
47
+ Plan paired baselines and neutral controls using the shared contract.
48
+
44
49
  ### Step 2: Run the dungeon
45
50
 
46
51
  The runner already exists at `scripts/verify-runner.mjs`. Use it — do NOT recreate.
47
52
 
48
- **ALWAYS run at full fidelity. Never use `--small` for verification.** `--small` runs (1K users, 100K events) compress per-cohort populations and shift ratios within ±25%, hiding real bugs and flagging fake ones. They exist in the runner only as a developer-troubleshooting escape hatch.
53
+ **Run at full fidelity for acceptance.** `--small` is for troubleshooting only.
54
+ Configured scale does not guarantee sufficient eligible users or converters.
55
+ Report `INSUFFICIENT_EVIDENCE` when actual independent populations or observation
56
+ horizons cannot support the claim; do not classify sparse data as a measured miss.
49
57
 
50
58
  ```bash
51
59
  node scripts/verify-runner.mjs <dungeon-path> <run-name>
@@ -59,16 +67,25 @@ node scripts/verify-runner.mjs dungeons/vertical/gaming/gaming.js verify-gaming
59
67
  Full-fidelity runs can take minutes (50K+ user dungeons). Plan accordingly — kick off the run, do other reading, return when the file lands. If a run takes longer than your budget, report that as a finding ("dungeon too large to verify in current session") rather than falling back to `--small`.
60
68
 
61
69
  **Expected output files** (in `./data/`, prefixed by `<run-name>`):
62
- - `<run-name>-EVENTS.json` — all events (JSONL)
70
+ - `<run-name>-EVENTS.json` — user events only (JSONL)
63
71
  - `<run-name>-USERS.json` — user profiles
64
72
  - `<run-name>-*-GROUPS.json` — group profiles (if dungeon has groups)
65
73
  - `<run-name>-*-SCD.json` — SCD data (if dungeon has SCDs)
74
+ - `<run-name>-STANDALONE*.json` — `standaloneEvents` cadence records, separate from users
75
+ - `<run-name>-WAREHOUSE-<metric>.*` — `warehouseMetrics` tables; use manifest paths and formats
76
+ - `<run-name>-WAREHOUSE-MANIFEST.json` — table columns, grain, paths, SQL, and deployment metadata
77
+
78
+ Use local disk output, `format: 'json'`, `gzip: false`, and sending disabled
79
+ (`token: ''`) for this verification path. The existing runner supplies these
80
+ overrides. Record the exact prefix, for example `data/verify-gaming`, and pass
81
+ it explicitly to every downstream command. Preserve table files and their
82
+ matching manifest for `/warehouse-metrics`; do not regenerate silently.
66
83
 
67
84
  For batched output (>2M events), the runner writes `<run-name>-EVENTS-part-*.json`. See [sql-recipes.md "Multi-part EVENTS files"](references/sql-recipes.md#multi-part-events-files-batch-mode).
68
85
 
69
86
  ### Step 3: Run the story runner (primary mechanical check)
70
87
 
71
- For any dungeon with a `stories` export, the runner — not the LLM — computes the verdicts:
88
+ Run this for any dungeon with a `stories` export or `warehouseMetrics`:
72
89
 
73
90
  ```bash
74
91
  node scripts/verify-stories.mjs <dungeon-path> --data-prefix <run-name>
@@ -77,28 +94,66 @@ node scripts/verify-stories.mjs <dungeon-path> --data-prefix <run-name> --json
77
94
 
78
95
  The runner streams the shards from Step 2, evaluates every assertion against its declared `target` / `floor` / `minCohort`, substitutes `{{PREFIX}}` into `duckdb`-type assertions and shells them out, enforces hook coverage (every numbered hook in the HOOK STORIES comment block must be targeted by at least one story), and prints a five-tier verdict table. Exit code is non-zero when any story lands WEAK / NONE / INVERSE or coverage is incomplete.
79
96
 
80
- **Verdicts are computed, not judged.** They include the population floor: a cohort smaller than the assertion's `minCohort` caps at WEAK a 12-user cohort can no longer score NAILED regardless of how clean its ratio looks. See [report-format.md "Verdict criteria"](references/report-format.md#verdict-criteria-5-tier) for the mechanical definitions.
97
+ When the dungeon declares `warehouseMetrics`, the runner also performs an automatic warehouse shape audit even if the dungeon exports no `stories`: it checks declared-column integrity, dense-gap absence, monotonic time ordering, empty numeric cells, sparse repeated-value suppression, and row-count sanity against the dataset window. Audit failures are reported alongside story verdicts and fail the CLI.
98
+
99
+ Warehouse stories support `warehouse` row assertions and `warehouse-stats`
100
+ summary assertions in the story schema. Automatic audits do not replace engineered story
101
+ assertions, and an audit-only dungeon must not be reported as having passing stories.
102
+
103
+ If a request calls these checks `rawStats`, translate that intent into supported
104
+ `warehouse-stats` fields or disk DuckDB. There is no `rawStats` breakdown type
105
+ in the local story schema or dispatcher. Inspect `lib/verify/warehouse.js` and
106
+ `lib/verify/story-runner.js` for the available summary fields; never invent one.
107
+
108
+ For `standaloneEvents`, use disk-backed `duckdb` assertions with
109
+ `read_json_auto('{{PREFIX}}-STANDALONE*.json', union_by_name=true, sample_size=-1)`.
110
+ The CLI's emulator input contains user EVENTS shards only. `--in-memory` skips
111
+ disk-only DuckDB assertions; it cannot prove standalone stories. The disk CLI
112
+ also requires nonempty user EVENTS shards. For a standalone-only artifact set
113
+ without those shards, run the same SQL directly and report that CLI limitation.
114
+ A standalone-only config without stories does not receive an automatic cadence
115
+ audit; perform the explicit schema, tick, dimension, and identity checks below.
116
+
117
+ **Verdicts are computed, not judged.** `minCohort` caps verdicts when supported
118
+ selected-row population fields fall below the floor. It does not validate every
119
+ denominator or establish full proof. Check actual eligible users/converters and
120
+ mature observation windows separately. See [report-format.md "Verdict criteria"](references/report-format.md#verdict-criteria-5-tier).
81
121
 
82
- **What the LLM investigates after this step — and nothing else:**
83
- 1. **Stories below STRONG** (WEAK / NONE / INVERSE) root-cause via Step 5's decision table. A miss means fixing the hook or the assertion's derivation, never relaxing the number to match output.
84
- 2. **`duckdb`-type assertions** the runner executes them but can't interpret bespoke shapes; sanity-check their output against the story narrative.
85
- 3. **Dungeons without a `stories` export** legacy fallback: full per-hook flow (Step 5) for every documented hook.
122
+ **What to review after this step:**
123
+ 1. **Every story, including passing targets** - check its options against the independent report specification and inspect neutral controls and population sufficiency. An inherited default can verify the wrong report.
124
+ 2. **Stories below STRONG** (WEAK / NONE / INVERSE) - root-cause via Step 5. Distinguish insufficient evidence from an effect failure; never relax a target to match output.
125
+ 3. **`duckdb`-type assertions** - the runner executes them but cannot validate the bespoke query's report semantics.
126
+ 4. **Dungeons without a `stories` export** - legacy fallback: Step 5 for every documented hook.
86
127
 
87
- Do NOT re-derive verdicts the runner already computed as passing. `hook-results.md` (Step 7) renders the runner's JSON.
128
+ Preserve computed verdicts in `hook-results.md`. Add separate semantic and evidence
129
+ statuses; a passing runner verdict alone does not establish acceptance.
88
130
 
89
131
  ### Step 4: Validate schema (BEFORE per-hook checks)
90
132
 
91
133
  Catches hooks that introduce undeclared columns (flag stamping). For each unique event type, compare actual columns against config-declared properties. See [sql-recipes.md "Schema validation queries"](references/sql-recipes.md#schema-validation-queries) for the SQL and the expected-schema source table.
92
134
 
135
+ Keep schemas separate. User-event checks use `events[]`, superProps, and enabled
136
+ SDK fields. Standalone checks use only core `event`, `time`, `insert_id`,
137
+ `distinct_id`, plus that spec's dimensions and properties. Require no `user_id`
138
+ or `device_id`; synthetic ids never count as people. Check one row per cadence
139
+ tick and dimension tuple unless a documented hook intentionally changes it.
140
+ Warehouse checks use the manifest's fixed columns and the metric's time column,
141
+ group keys, value column, and declared extras. Never apply user superProp or
142
+ identity requirements to cadence records or warehouse rows.
143
+
93
144
  **Schema verdicts:**
94
- - **SCHEMA-PASS** added column appears on 100% of events of this type (uniform enrichment, acceptable)
95
- - **SCHEMA-FAIL** added column appears on <100% (flag stamping; conditional property creates inconsistent schema)
145
+ - **SCHEMA-PASS** - every output property is declared or a recognized enabled engine/SDK field for this stream.
146
+ - **SCHEMA-FAIL** - any undeclared property, even at 100% coverage. Uniform flag stamping still violates schema-first authorship.
147
+
148
+ If the runtime validator permits uniform enrichment, record that result separately
149
+ and enforce this stricter authorship gate explicitly.
96
150
 
97
151
  If any event type has SCHEMA-FAIL, flag it prominently in the report header with specific remediation: which hook line adds the property and how to remove it while preserving the intended pattern.
98
152
 
99
153
  ### Step 5: Investigate failures (and legacy no-stories dungeons)
100
154
 
101
- Applies only to the investigation targets from Step 3 failing stories, `duckdb`-type assertions, and dungeons with no `stories` export.
155
+ Applies to Step 3's targets: failed stories, semantic or evidence gaps (including
156
+ passing targets), bespoke SQL, and dungeons with no `stories` export.
102
157
 
103
158
  **Decision: emulator vs DuckDB**
104
159
 
@@ -121,12 +176,17 @@ Applies only to the investigation targets from Step 3 — failing stories, `duck
121
176
  | Identity-model invariants | DuckDB |
122
177
  | Experiment invariants | DuckDB |
123
178
  | True bespokes (no emulator analysis fits) | DuckDB |
179
+ | Standalone cadence values, dimensions, tick counts | Disk DuckDB on `{{PREFIX}}-STANDALONE*.json` |
180
+ | Warehouse value stories | `warehouse` or `warehouse-stats` assertion |
181
+ | Warehouse schema, gaps, ordering, numeric cells | Automatic warehouse audit, with or without stories |
124
182
 
125
183
  The emulator now covers lifecycle, flows, sessions, event breakdowns, formulas, and compounded retention — DuckDB's remit is schema / identity / experiment invariants plus true bespoke shapes. If a "bespoke" check is really a funnel, frequency, path, or breakdown in disguise, it belongs in the emulator.
126
184
 
127
185
  **Hand-written DuckDB funnel SQL diverges from Mixpanel — never hand-roll.** If you find yourself writing `WITH step1 AS ..., step2 AS ...` for a funnel, STOP — use `emulateBreakdown` with `funnelFrequency` instead.
128
186
 
129
- For emulator details, identity-model dungeons (must pass `profiles`), and time-series breakdown via `timeBucket`, see [counting-semantics.md](references/counting-semantics.md). For DuckDB query templates by hook archetype, pitfalls, and standard checks, see [sql-recipes.md](references/sql-recipes.md).
187
+ For emulator options, emitted-evidence identity maps, profile segments, and
188
+ time-series breakdown via `timeBucket`, see [counting-semantics.md](references/counting-semantics.md).
189
+ For DuckDB diagnostics and standard checks, see [sql-recipes.md](references/sql-recipes.md).
130
190
 
131
191
  **Always run for every dungeon** (before per-hook checks):
132
192
  - Standard identity-model invariants (stitch counts, pre-existing user stamping) when the dungeon uses the identity model
@@ -142,9 +202,9 @@ generated artifacts into `dungeons/user/<name>/`:
142
202
  - `hook-query-log.txt` (Step 6)
143
203
  - `<name>-verifications.sql` (Step 7b)
144
204
 
145
- The ONLY exception is the throwaway verification data the run writes to
146
- `./data/` (`verify-*` event/user files) that stays in `./data/` and is
147
- deleted in Step 8.
205
+ The exception is run data under `./data/`, including cadence shards, warehouse
206
+ tables, and the manifest. Keep it under its explicit prefix until verification
207
+ and deployment finish. Step 8 requires consent before any scoped cleanup.
148
208
 
149
209
  For non-user dungeons (technical/vertical) or batch runs across many dungeons,
150
210
  fall back to `./research/` for `hook-results.md` / `hook-query-log.txt`.
@@ -159,7 +219,7 @@ Format and conventions: see [report-format.md "Query log format"](references/rep
159
219
 
160
220
  ### Step 7: Write `hook-results.md`
161
221
 
162
- Write to `dungeons/user/<name>/hook-results.md` for a user dungeon, else `./research/hook-results.md`. For story-backed dungeons, **the report renders the runner's JSON** (`verify-stories.mjs --json`): the hook summary table comes straight from the runner's per-story verdicts (story id, hook, archetype, observed vs target, verdict), and detailed blocks exist only for the Step-3 investigation targets. Use the templates in [report-format.md](references/report-format.md):
222
+ Write to `dungeons/user/<name>/hook-results.md` for a user dungeon, else `./research/hook-results.md`. Preserve the runner's JSON verdicts alongside independent semantic and evidence statuses, eligible populations, neutral controls, and source-derived scope. Add details for Step 3's investigation targets. Use [report-format.md](references/report-format.md):
163
223
  - Single-dungeon report structure
164
224
  - Multi-dungeon report structure (when batch mode)
165
225
  - Per-hook detail block
@@ -171,22 +231,25 @@ Write to `dungeons/user/<name>/hook-results.md` for a user dungeon, else `./rese
171
231
 
172
232
  When verifying a dungeon in `dungeons/user/`, also write a standalone DuckDB SQL file alongside the dungeon in its folder at `dungeons/user/<name>/<name>-verifications.sql`. Vertical dungeons already have their SQL co-located at `dungeons/vertical/<name>/<name>.sql`. Format: see [report-format.md "Verification SQL file"](references/report-format.md#verification-sql-file-mandatory-for-user-dungeons).
173
233
 
174
- ### Step 8: Cleanup
234
+ ### Step 8: Preserve artifacts and hand off
175
235
 
176
- ```bash
177
- rm -f ./data/verify-* ./verify-*
178
- ```
236
+ Record the exact verified prefix and retained files in the report. If warehouse
237
+ tables are present, provision with `/create-project`, then hand off to
238
+ `/warehouse-metrics` using `--data-prefix <verified-prefix>`. Provisioning and
239
+ ordinary event import do not deploy these tables.
179
240
 
180
- Remove ALL files matching `verify-*` in `./data/` and project root. Also remove any temporary runner scripts.
241
+ Do not run blanket prune or delete `verify-*` globs. Only after deployment is
242
+ complete, list files for this exact run and obtain user consent for scoped
243
+ cleanup. Keep reports and reproducible SQL. Do not delete files during an audit.
181
244
 
182
245
  ## Hook execution model
183
246
 
184
247
  Per user, hooks fire in this order:
185
248
 
186
249
  1. `"user"` — profile created (mutate in-place; return ignored)
187
- 2. `"scd-pre"` — SCD entries created (mutate in-place OR return new array)
250
+ 2. `"scd-pre"` — SCD entries created (mutate in-place; return ignored)
188
251
  3. For each funnel: `"funnel-pre"` → `"event"` (per step) → `"funnel-post"`
189
- 4. `"event"` — for non-funnel standalone events (return value REPLACES the event)
252
+ 4. `"event"` — for non-funnel user events from `events[]` (return value REPLACES the event)
190
253
  5. `"everything"` — array of ALL the user's events (return array to replace)
191
254
  6. **Storage phase** — data written to disk. Hooks for `event`, `user`, `scd` do NOT re-fire (already applied above). Hooks for `mirror`, `ad-spend`, `group`, `lookup` fire only in storage.
192
255
 
@@ -195,6 +258,15 @@ Return-value behavior:
195
258
  - `everything` hook: return value IS used if it's an array (replaces event list)
196
259
  - `user`, `scd-pre`, `funnel-post`: return value IGNORED — only in-place mutations work
197
260
  - `funnel-pre`: return value IGNORED — mutate the `record` object in-place (e.g., `record.conversionRate = 0.9`)
261
+ - `standalone`: runs before the user loop. Return an object or array; `undefined`
262
+ drops the record. `meta.spec` and `meta.config` describe the cadence stream.
263
+ - `warehouse`: runs after the user loop. Mutate the row; return value is ignored.
264
+ Meta includes `spec`, `config`, `metricName`, bucket fields, `grain`,
265
+ `seriesKey`, `isBackfill`, and `raw` source aggregates.
266
+
267
+ Neither `standalone` nor `warehouse` receives person metadata or passes through
268
+ `everything`. Their generation phases differ even though both hooks fire on
269
+ storage push. Other storage-only types return objects or arrays to retain rows.
198
270
 
199
271
  ## Final output
200
272
 
@@ -204,5 +276,7 @@ Tell the user:
204
276
  3. Query log path (if written): alongside the report (`dungeons/user/<name>/hook-query-log.txt`, else `./research/hook-query-log.txt`)
205
277
  4. Verdict counts from the story runner (per dungeon if batch mode), plus which stories needed LLM investigation
206
278
  5. One-line summary of the most interesting finding
279
+ 6. Separate standalone checks and `warehouseAudits`, including audit-only runs,
280
+ skipped assertions, retained artifact prefix, and warehouse deployment handoff
207
281
 
208
282
  If hooks failed, note that `hook-results.md` can be used as context for fixing them: "read hook-results.md and fix the failing hooks in <dungeon-file>".
@@ -0,0 +1,84 @@
1
+ # 1.8.1 verification contract
2
+
3
+ Use this contract when authoring, verifying, provisioning, or presenting a dungeon.
4
+ The [1.8.1 guide](../../../../docs/guides/1.8.1-upgrade-guide.md) describes the
5
+ release. The local repository's `tests/alignment/REPORT.md` and
6
+ `tests/alignment/API-COMPATIBILITY.md` retain the audit evidence; those test
7
+ artifacts are not shipped in npm.
8
+
9
+ ## Define the report before measuring
10
+
11
+ Write an independent report specification before choosing assertions. Record
12
+ events, filters, identity rules, cohort definition, denominator, statistic,
13
+ window, timezone, ordering, reentry, session settings, and property selection.
14
+ Compare every assertion with that specification, including passing assertions.
15
+ An inherited default can pass the wrong report. Do not replace funnel TTC with
16
+ a numeric property, a mean with a median, or conversion with event volume to
17
+ rescue a failed story. Supplementary diagnostics must keep their own labels.
18
+
19
+ Local checks prove selected source-derived contracts, not universal Mixpanel
20
+ parity. The audit read analytics source without compiling or executing it. It
21
+ did not run live differential queries. Macro/soup interactions, table surfaces,
22
+ arbitrary hooks, parallel execution, and several counting variants remain gaps.
23
+
24
+ ## Separate a measured effect from enough evidence
25
+
26
+ Retain the story runner's computed verdict. Report semantic correctness and
27
+ evidence sufficiency separately. Full-fidelity scale does not guarantee enough
28
+ independent eligible users, mature retention cohorts, or converters. Inspect the
29
+ actual denominator on each side; event count is not independent user count.
30
+ `minCohort` checks supported selected-row population fields, not every bespoke
31
+ denominator. Meeting it alone is not full proof. Label unresolved acceptance
32
+ `INSUFFICIENT_EVIDENCE`; do not count it as a pass or a measured effect failure.
33
+
34
+ For engineered effects, measure a paired baseline with the same seed, window,
35
+ and report options. Add a neutral control (factor one, zero injection, or hook
36
+ disabled) and an unaffected control cohort where applicable. A factor-one
37
+ control must be measured, not inferred from two differently modified groups.
38
+ Report baseline, treatment, eligible users/converters, and paired lift. Keep
39
+ targets and tolerances fixed before the run; disclose any revised criterion
40
+ and preserve its original failed evidence.
41
+
42
+ ## Preserve the local counting boundaries
43
+
44
+ - `Funnel.reentry` is verifier-only. Generated repetitions depend on usage
45
+ selection and event budget. Local `reentry` defaults to `false`, including
46
+ `countMode: 'totals'`; repeated histories require explicit `reentry: true`.
47
+ - Funnel histories include ordered restart/shared-edge behavior and inclusive
48
+ 2000ms completion grace. TTC uses completed histories under the requested
49
+ options, not independent first occurrences of each event. First/last segment
50
+ selection merges reached-step properties with defined fallback values;
51
+ explicit-step selection is separate.
52
+ - Derive sessions from the full resolved user stream before HPC partitioning.
53
+ Local defaults are UTC, a 30-minute idle timeout, and a 24-hour maximum.
54
+ `session_id` is a diagnostic or explicit legacy mode, not the default truth.
55
+ Non-UTC projects, list-valued HPC, and project-specific hidden/excluded session
56
+ events remain outside this proof.
57
+ - Profile device pools alone establish no identity mapping evidence. The public
58
+ `buildIdentityMap(profiles)` helper retains profile-based compatibility behavior.
59
+ For emitted-identity proof, derive links from valid ordinary both-ID events,
60
+ including later Login events, and pass that explicit `identityMap` to the
61
+ verifier. Earlier device-only rows can resolve retrospectively. Report ID
62
+ conflicts and validation limits; generator auth policy is not ingestion parity.
63
+ - `applyPathBias` is append-only. `share` selects eligible injection recipients;
64
+ it does not promise exact visible branch share. Measure appended payloads,
65
+ competing events, observed path share, and paired lift separately.
66
+ - `applySessionShape` preserves records. Pass known `datasetStart`/`datasetEnd`
67
+ bounds when clipping matters. Explicit bounds reject insufficient capacity
68
+ before mutation; unbounded placement can cross an unknown dataset end.
69
+ - `retentionCurve` weights active-day plans. Budget, funnel spill, and observation
70
+ horizon can prevent literal requested retention percentages. Usage anchors do
71
+ not accumulate previous funnel TTC; retries belong to born-user first funnels.
72
+
73
+ ## Keep schema and operational gates independent
74
+
75
+ Undeclared output properties fail schema review even at 100% coverage. Accept
76
+ only config-declared properties or recognized engine/SDK fields enabled for that
77
+ stream. A permissive runtime schema summary cannot override this authorship rule.
78
+ Keep user events, standalone cadence, and warehouse schemas separate.
79
+
80
+ Offline verification stops at local evidence and a deployment handoff. Do not
81
+ automatically provision, import, probe endpoints, deploy warehouse tables, or run
82
+ headless builds. Retain exact artifact prefixes and actual deployment reports.
83
+ Dry-run output is a plan, not evidence of a live outcome; preserve existing reports
84
+ before a dry run writes local files. Live work requires separate authorization.