@ak--47/dungeon-master 1.6.5 → 1.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/analyze-soup/SKILL.md +21 -11
- package/.claude/skills/create-dungeon/SKILL.md +49 -10
- package/.claude/skills/create-project/SKILL.md +22 -3
- package/.claude/skills/create-project/context.mjs +89 -0
- package/.claude/skills/create-project/provision.mjs +1 -60
- package/.claude/skills/headless-build/SKILL.md +18 -1
- package/.claude/skills/powertools/SKILL.md +20 -1
- package/.claude/skills/release-check/SKILL.md +99 -0
- package/.claude/skills/verify-dungeon/SKILL.md +71 -16
- package/.claude/skills/verify-dungeon/references/counting-semantics.md +14 -0
- package/.claude/skills/verify-dungeon/references/report-format.md +18 -1
- package/.claude/skills/verify-dungeon/references/sql-recipes.md +36 -1
- package/.claude/skills/warehouse-metrics/GAPS-template.md +34 -0
- package/.claude/skills/warehouse-metrics/SKILL.md +105 -0
- package/.claude/skills/warehouse-metrics/deploy.mjs +651 -0
- package/.claude/skills/write-hooks/SKILL.md +33 -3
- package/CHANGELOG.md +331 -0
- package/HOOKS.md +154 -5
- package/README.md +357 -8
- package/docs/guides/1.7.0-upgrade-guide.md +154 -0
- package/docs/guides/1.8.0-upgrade-guide.md +151 -0
- package/dungeons/technical/warehouse.js +187 -0
- package/index.js +131 -2
- package/lib/core/config-validator.js +264 -13
- package/lib/core/context.js +39 -0
- package/lib/core/dungeon-loader.js +5 -2
- package/lib/core/storage.js +51 -3
- package/lib/generators/events.js +53 -7
- package/lib/generators/funnels.js +85 -11
- package/lib/generators/profiles.js +9 -4
- package/lib/generators/standalone.js +248 -0
- package/lib/generators/warehouse.js +828 -0
- package/lib/orchestrators/mixpanel-sender.js +39 -3
- package/lib/orchestrators/user-loop.js +240 -9
- package/lib/templates/story-spec.schema.json +41 -16
- package/lib/utils/conditions.js +62 -0
- package/lib/utils/json-evaluator.js +12 -2
- package/lib/utils/utils.js +115 -19
- package/lib/verify/index.js +1 -0
- package/lib/verify/schema-validator.js +8 -0
- package/lib/verify/story-runner.js +71 -8
- package/lib/verify/warehouse.js +683 -0
- package/package.json +5 -11
- package/scripts/verify-stories.mjs +150 -44
- package/types.d.ts +606 -38
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: verify-dungeon
|
|
3
|
-
description: Use when a dungeon
|
|
4
|
-
argument-hint: [dungeon path(s), e.g. dungeons/vertical/gaming/gaming.js or dungeons/vertical/fintech/fintech.js]
|
|
3
|
+
description: Use when a dungeon needs verification before import or warehouse deployment, including standaloneEvents cadence streams and warehouseMetrics tables. Runs full-scale story checks, schema validation, and automatic warehouse audits even without stories. Reports per-hook NAILED/STRONG/WEAK/NONE/INVERSE verdicts; investigates failures and duckdb assertions.
|
|
4
|
+
argument-hint: '[dungeon path(s), e.g. dungeons/vertical/gaming/gaming.js or dungeons/vertical/fintech/fintech.js]'
|
|
5
5
|
model: claude-opus-4-6
|
|
6
6
|
effort: max
|
|
7
7
|
---
|
|
@@ -35,10 +35,10 @@ Read the dungeon at `$ARGUMENTS`. If it's a bare filename (no `/`), check `dunge
|
|
|
35
35
|
|
|
36
36
|
For each hook/pattern, catalog:
|
|
37
37
|
- Hook number and name (e.g., "Hook #1: Compass users have 3x quest completion")
|
|
38
|
-
- Hook type (`event`, `everything`, `funnel-pre`, `funnel-post`, `user`, `scd-pre`)
|
|
38
|
+
- Hook type (`event`, `everything`, `funnel-pre`, `funnel-post`, `user`, `scd-pre`, `standalone`, `warehouse`)
|
|
39
39
|
- Mechanism — what the code actually does
|
|
40
40
|
- Expected signal — specific, measurable outcome (e.g., "compass_user=true events should have ~1.5x reward_gold")
|
|
41
|
-
- Which output file the signal lives in (events, users, groups)
|
|
41
|
+
- Which output file the signal lives in (user events, users, groups, standalone cadence shards, warehouse tables)
|
|
42
42
|
- Mixpanel report instructions — flag missing/vague ones for the report
|
|
43
43
|
|
|
44
44
|
### Step 2: Run the dungeon
|
|
@@ -59,16 +59,25 @@ node scripts/verify-runner.mjs dungeons/vertical/gaming/gaming.js verify-gaming
|
|
|
59
59
|
Full-fidelity runs can take minutes (50K+ user dungeons). Plan accordingly — kick off the run, do other reading, return when the file lands. If a run takes longer than your budget, report that as a finding ("dungeon too large to verify in current session") rather than falling back to `--small`.
|
|
60
60
|
|
|
61
61
|
**Expected output files** (in `./data/`, prefixed by `<run-name>`):
|
|
62
|
-
- `<run-name>-EVENTS.json` —
|
|
62
|
+
- `<run-name>-EVENTS.json` — user events only (JSONL)
|
|
63
63
|
- `<run-name>-USERS.json` — user profiles
|
|
64
64
|
- `<run-name>-*-GROUPS.json` — group profiles (if dungeon has groups)
|
|
65
65
|
- `<run-name>-*-SCD.json` — SCD data (if dungeon has SCDs)
|
|
66
|
+
- `<run-name>-STANDALONE*.json` — `standaloneEvents` cadence records, separate from users
|
|
67
|
+
- `<run-name>-WAREHOUSE-<metric>.*` — `warehouseMetrics` tables; use manifest paths and formats
|
|
68
|
+
- `<run-name>-WAREHOUSE-MANIFEST.json` — table columns, grain, paths, SQL, and deployment metadata
|
|
69
|
+
|
|
70
|
+
Use local disk output, `format: 'json'`, `gzip: false`, and sending disabled
|
|
71
|
+
(`token: ''`) for this verification path. The existing runner supplies these
|
|
72
|
+
overrides. Record the exact prefix, for example `data/verify-gaming`, and pass
|
|
73
|
+
it explicitly to every downstream command. Preserve table files and their
|
|
74
|
+
matching manifest for `/warehouse-metrics`; do not regenerate silently.
|
|
66
75
|
|
|
67
76
|
For batched output (>2M events), the runner writes `<run-name>-EVENTS-part-*.json`. See [sql-recipes.md "Multi-part EVENTS files"](references/sql-recipes.md#multi-part-events-files-batch-mode).
|
|
68
77
|
|
|
69
78
|
### Step 3: Run the story runner (primary mechanical check)
|
|
70
79
|
|
|
71
|
-
|
|
80
|
+
Run this for any dungeon with a `stories` export or `warehouseMetrics`:
|
|
72
81
|
|
|
73
82
|
```bash
|
|
74
83
|
node scripts/verify-stories.mjs <dungeon-path> --data-prefix <run-name>
|
|
@@ -77,6 +86,26 @@ node scripts/verify-stories.mjs <dungeon-path> --data-prefix <run-name> --json
|
|
|
77
86
|
|
|
78
87
|
The runner streams the shards from Step 2, evaluates every assertion against its declared `target` / `floor` / `minCohort`, substitutes `{{PREFIX}}` into `duckdb`-type assertions and shells them out, enforces hook coverage (every numbered hook in the HOOK STORIES comment block must be targeted by at least one story), and prints a five-tier verdict table. Exit code is non-zero when any story lands WEAK / NONE / INVERSE or coverage is incomplete.
|
|
79
88
|
|
|
89
|
+
When the dungeon declares `warehouseMetrics`, the runner also performs an automatic warehouse shape audit even if the dungeon exports no `stories`: it checks declared-column integrity, dense-gap absence, monotonic time ordering, empty numeric cells, sparse repeated-value suppression, and row-count sanity against the dataset window. Audit failures are reported alongside story verdicts and fail the CLI.
|
|
90
|
+
|
|
91
|
+
Warehouse stories support `warehouse` row assertions and `warehouse-stats`
|
|
92
|
+
summary assertions in the story schema. Automatic audits do not replace engineered story
|
|
93
|
+
assertions, and an audit-only dungeon must not be reported as having passing stories.
|
|
94
|
+
|
|
95
|
+
If a request calls these checks `rawStats`, translate that intent into supported
|
|
96
|
+
`warehouse-stats` fields or disk DuckDB. There is no `rawStats` breakdown type
|
|
97
|
+
in the local story schema or dispatcher. Inspect `lib/verify/warehouse.js` and
|
|
98
|
+
`lib/verify/story-runner.js` for the available summary fields; never invent one.
|
|
99
|
+
|
|
100
|
+
For `standaloneEvents`, use disk-backed `duckdb` assertions with
|
|
101
|
+
`read_json_auto('{{PREFIX}}-STANDALONE*.json', union_by_name=true, sample_size=-1)`.
|
|
102
|
+
The CLI's emulator input contains user EVENTS shards only. `--in-memory` skips
|
|
103
|
+
disk-only DuckDB assertions; it cannot prove standalone stories. The disk CLI
|
|
104
|
+
also requires nonempty user EVENTS shards. For a standalone-only artifact set
|
|
105
|
+
without those shards, run the same SQL directly and report that CLI limitation.
|
|
106
|
+
A standalone-only config without stories does not receive an automatic cadence
|
|
107
|
+
audit; perform the explicit schema, tick, dimension, and identity checks below.
|
|
108
|
+
|
|
80
109
|
**Verdicts are computed, not judged.** They include the population floor: a cohort smaller than the assertion's `minCohort` caps at WEAK — a 12-user cohort can no longer score NAILED regardless of how clean its ratio looks. See [report-format.md "Verdict criteria"](references/report-format.md#verdict-criteria-5-tier) for the mechanical definitions.
|
|
81
110
|
|
|
82
111
|
**What the LLM investigates after this step — and nothing else:**
|
|
@@ -90,6 +119,15 @@ Do NOT re-derive verdicts the runner already computed as passing. `hook-results.
|
|
|
90
119
|
|
|
91
120
|
Catches hooks that introduce undeclared columns (flag stamping). For each unique event type, compare actual columns against config-declared properties. See [sql-recipes.md "Schema validation queries"](references/sql-recipes.md#schema-validation-queries) for the SQL and the expected-schema source table.
|
|
92
121
|
|
|
122
|
+
Keep schemas separate. User-event checks use `events[]`, superProps, and enabled
|
|
123
|
+
SDK fields. Standalone checks use only core `event`, `time`, `insert_id`,
|
|
124
|
+
`distinct_id`, plus that spec's dimensions and properties. Require no `user_id`
|
|
125
|
+
or `device_id`; synthetic ids never count as people. Check one row per cadence
|
|
126
|
+
tick and dimension tuple unless a documented hook intentionally changes it.
|
|
127
|
+
Warehouse checks use the manifest's fixed columns and the metric's time column,
|
|
128
|
+
group keys, value column, and declared extras. Never apply user superProp or
|
|
129
|
+
identity requirements to cadence records or warehouse rows.
|
|
130
|
+
|
|
93
131
|
**Schema verdicts:**
|
|
94
132
|
- **SCHEMA-PASS** — added column appears on 100% of events of this type (uniform enrichment, acceptable)
|
|
95
133
|
- **SCHEMA-FAIL** — added column appears on <100% (flag stamping; conditional property creates inconsistent schema)
|
|
@@ -121,6 +159,9 @@ Applies only to the investigation targets from Step 3 — failing stories, `duck
|
|
|
121
159
|
| Identity-model invariants | DuckDB |
|
|
122
160
|
| Experiment invariants | DuckDB |
|
|
123
161
|
| True bespokes (no emulator analysis fits) | DuckDB |
|
|
162
|
+
| Standalone cadence values, dimensions, tick counts | Disk DuckDB on `{{PREFIX}}-STANDALONE*.json` |
|
|
163
|
+
| Warehouse value stories | `warehouse` or `warehouse-stats` assertion |
|
|
164
|
+
| Warehouse schema, gaps, ordering, numeric cells | Automatic warehouse audit, with or without stories |
|
|
124
165
|
|
|
125
166
|
The emulator now covers lifecycle, flows, sessions, event breakdowns, formulas, and compounded retention — DuckDB's remit is schema / identity / experiment invariants plus true bespoke shapes. If a "bespoke" check is really a funnel, frequency, path, or breakdown in disguise, it belongs in the emulator.
|
|
126
167
|
|
|
@@ -142,9 +183,9 @@ generated artifacts into `dungeons/user/<name>/`:
|
|
|
142
183
|
- `hook-query-log.txt` (Step 6)
|
|
143
184
|
- `<name>-verifications.sql` (Step 7b)
|
|
144
185
|
|
|
145
|
-
The
|
|
146
|
-
|
|
147
|
-
|
|
186
|
+
The exception is run data under `./data/`, including cadence shards, warehouse
|
|
187
|
+
tables, and the manifest. Keep it under its explicit prefix until verification
|
|
188
|
+
and deployment finish. Step 8 requires consent before any scoped cleanup.
|
|
148
189
|
|
|
149
190
|
For non-user dungeons (technical/vertical) or batch runs across many dungeons,
|
|
150
191
|
fall back to `./research/` for `hook-results.md` / `hook-query-log.txt`.
|
|
@@ -171,22 +212,25 @@ Write to `dungeons/user/<name>/hook-results.md` for a user dungeon, else `./rese
|
|
|
171
212
|
|
|
172
213
|
When verifying a dungeon in `dungeons/user/`, also write a standalone DuckDB SQL file alongside the dungeon in its folder at `dungeons/user/<name>/<name>-verifications.sql`. Vertical dungeons already have their SQL co-located at `dungeons/vertical/<name>/<name>.sql`. Format: see [report-format.md "Verification SQL file"](references/report-format.md#verification-sql-file-mandatory-for-user-dungeons).
|
|
173
214
|
|
|
174
|
-
### Step 8:
|
|
215
|
+
### Step 8: Preserve artifacts and hand off
|
|
175
216
|
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
217
|
+
Record the exact verified prefix and retained files in the report. If warehouse
|
|
218
|
+
tables are present, provision with `/create-project`, then hand off to
|
|
219
|
+
`/warehouse-metrics` using `--data-prefix <verified-prefix>`. Provisioning and
|
|
220
|
+
ordinary event import do not deploy these tables.
|
|
179
221
|
|
|
180
|
-
|
|
222
|
+
Do not run blanket prune or delete `verify-*` globs. Only after deployment is
|
|
223
|
+
complete, list files for this exact run and obtain user consent for scoped
|
|
224
|
+
cleanup. Keep reports and reproducible SQL. Do not delete files during an audit.
|
|
181
225
|
|
|
182
226
|
## Hook execution model
|
|
183
227
|
|
|
184
228
|
Per user, hooks fire in this order:
|
|
185
229
|
|
|
186
230
|
1. `"user"` — profile created (mutate in-place; return ignored)
|
|
187
|
-
2. `"scd-pre"` — SCD entries created (mutate in-place
|
|
231
|
+
2. `"scd-pre"` — SCD entries created (mutate in-place; return ignored)
|
|
188
232
|
3. For each funnel: `"funnel-pre"` → `"event"` (per step) → `"funnel-post"`
|
|
189
|
-
4. `"event"` — for non-funnel
|
|
233
|
+
4. `"event"` — for non-funnel user events from `events[]` (return value REPLACES the event)
|
|
190
234
|
5. `"everything"` — array of ALL the user's events (return array to replace)
|
|
191
235
|
6. **Storage phase** — data written to disk. Hooks for `event`, `user`, `scd` do NOT re-fire (already applied above). Hooks for `mirror`, `ad-spend`, `group`, `lookup` fire only in storage.
|
|
192
236
|
|
|
@@ -195,6 +239,15 @@ Return-value behavior:
|
|
|
195
239
|
- `everything` hook: return value IS used if it's an array (replaces event list)
|
|
196
240
|
- `user`, `scd-pre`, `funnel-post`: return value IGNORED — only in-place mutations work
|
|
197
241
|
- `funnel-pre`: return value IGNORED — mutate the `record` object in-place (e.g., `record.conversionRate = 0.9`)
|
|
242
|
+
- `standalone`: runs before the user loop. Return an object or array; `undefined`
|
|
243
|
+
drops the record. `meta.spec` and `meta.config` describe the cadence stream.
|
|
244
|
+
- `warehouse`: runs after the user loop. Mutate the row; return value is ignored.
|
|
245
|
+
Meta includes `spec`, `config`, `metricName`, bucket fields, `grain`,
|
|
246
|
+
`seriesKey`, `isBackfill`, and `raw` source aggregates.
|
|
247
|
+
|
|
248
|
+
Neither `standalone` nor `warehouse` receives person metadata or passes through
|
|
249
|
+
`everything`. Their generation phases differ even though both hooks fire on
|
|
250
|
+
storage push. Other storage-only types return objects or arrays to retain rows.
|
|
198
251
|
|
|
199
252
|
## Final output
|
|
200
253
|
|
|
@@ -204,5 +257,7 @@ Tell the user:
|
|
|
204
257
|
3. Query log path (if written): alongside the report (`dungeons/user/<name>/hook-query-log.txt`, else `./research/hook-query-log.txt`)
|
|
205
258
|
4. Verdict counts from the story runner (per dungeon if batch mode), plus which stories needed LLM investigation
|
|
206
259
|
5. One-line summary of the most interesting finding
|
|
260
|
+
6. Separate standalone checks and `warehouseAudits`, including audit-only runs,
|
|
261
|
+
skipped assertions, retained artifact prefix, and warehouse deployment handoff
|
|
207
262
|
|
|
208
263
|
If hooks failed, note that `hook-results.md` can be used as context for fixing them: "read hook-results.md and fix the failing hooks in <dungeon-file>".
|
|
@@ -1,3 +1,17 @@
|
|
|
1
|
+
# v1.8.0 metric streams
|
|
2
|
+
|
|
3
|
+
The user-event emulator receives EVENTS and profiles. `standaloneEvents` writes
|
|
4
|
+
separate STANDALONE shards; verify these with disk `duckdb` assertions on
|
|
5
|
+
`{{PREFIX}}-STANDALONE*.json`. The CLI's `--in-memory` mode skips disk-only SQL.
|
|
6
|
+
Standalone synthetic `distinct_id` values identify series, never people, so
|
|
7
|
+
exclude them from funnels, retention, lifecycle, identity stitching, and user counts.
|
|
8
|
+
|
|
9
|
+
`warehouseMetrics` reads only user `events[]` sources. It materializes after
|
|
10
|
+
the user loop and supports `warehouse` and `warehouse-stats` assertions.
|
|
11
|
+
Automatic warehouse audits run even without stories. Its history buckets can
|
|
12
|
+
precede the user-event window; sparse point-in-time tables intentionally omit
|
|
13
|
+
unchanged rows. Do not apply cadence or user-population counting rules to them.
|
|
14
|
+
|
|
1
15
|
# Counting Semantics — Mixpanel-Accurate Verification
|
|
2
16
|
|
|
3
17
|
Mixpanel does NOT count the way naive SQL does. The verifier (and any DuckDB query you write) must match Mixpanel's rules.
|
|
@@ -4,6 +4,23 @@ Templates and conventions for writing `hook-results.md` and per-dungeon verifica
|
|
|
4
4
|
|
|
5
5
|
## Verdict criteria (5-tier)
|
|
6
6
|
|
|
7
|
+
Every report records the exact data prefix and retained artifact paths. Separate
|
|
8
|
+
user EVENTS/USERS, standalone cadence shards, and warehouse tables/manifest.
|
|
9
|
+
Keep deployment inputs until `/warehouse-metrics` completes; cleanup needs
|
|
10
|
+
consent and must name this run's files only.
|
|
11
|
+
|
|
12
|
+
Report standalone schema, cadence counts, dimension coverage, numeric values,
|
|
13
|
+
and absence of person ids separately from user schema and identity checks.
|
|
14
|
+
Name the disk DuckDB assertions and flag any skipped in-memory checks. Synthetic
|
|
15
|
+
standalone ids never contribute to users, funnels, or retention.
|
|
16
|
+
|
|
17
|
+
Render `warehouseAudits` from the runner JSON in a separate pass/fail table,
|
|
18
|
+
including declared columns, gaps, ordering, numeric cells, sparse repeats, and
|
|
19
|
+
row-count checks. Report these even without stories. Label an audit-only run
|
|
20
|
+
"no stories declared"; do not invent NAILED/STRONG story verdicts. Distinguish
|
|
21
|
+
warehouse history backfill from the user-event date window and record the
|
|
22
|
+
project-provisioning and `/warehouse-metrics` handoff status.
|
|
23
|
+
|
|
7
24
|
Verdicts are **mechanical** — computed by `scripts/verify-stories.mjs` from each story assertion's declared `target` / `floor` / `minCohort` (see the `StoryVerdict` typedef in `types.d.ts`), not eyeballed percentages:
|
|
8
25
|
|
|
9
26
|
- **NAILED** — observed within ±10% of `target`.
|
|
@@ -206,7 +223,7 @@ Follow the format in `dungeons/vertical/<name>/`:
|
|
|
206
223
|
-- USAGE:
|
|
207
224
|
-- 1. node scripts/verify-runner.mjs dungeons/user/<name>/<name>.js verify-<name>
|
|
208
225
|
-- 2. duckdb < dungeons/user/<name>/<name>-verifications.sql
|
|
209
|
-
-- 3.
|
|
226
|
+
-- 3. Preserve this prefix through deployment; cleanup requires explicit consent.
|
|
210
227
|
-- ============================================================================
|
|
211
228
|
|
|
212
229
|
-- HOOK N: NAME (TYPE)
|
|
@@ -60,6 +60,41 @@ If any event type has SCHEMA-FAIL, flag it prominently and include specific reme
|
|
|
60
60
|
|
|
61
61
|
## Standard identity-model invariants
|
|
62
62
|
|
|
63
|
+
### Metric artifacts use separate schemas
|
|
64
|
+
|
|
65
|
+
The expected-schema table above applies only to user EVENTS. For `standaloneEvents`,
|
|
66
|
+
expect `event`, `time`, `insert_id`, `distinct_id`, plus the matching spec's
|
|
67
|
+
dimension and property keys. User superProps, session ids, and SDK flags do not
|
|
68
|
+
apply. Check undeclared columns explicitly; the story CLI's user schema pass
|
|
69
|
+
does not cover standalone shards.
|
|
70
|
+
|
|
71
|
+
Use a `duckdb` story assertion with this source, filtering the declared event:
|
|
72
|
+
|
|
73
|
+
```sql
|
|
74
|
+
SELECT event, count(*) AS records, min(time::TIMESTAMP) AS first_tick,
|
|
75
|
+
max(time::TIMESTAMP) AS last_tick
|
|
76
|
+
FROM read_json_auto('{{PREFIX}}-STANDALONE*.json',
|
|
77
|
+
union_by_name=true, sample_size=-1)
|
|
78
|
+
GROUP BY event;
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Compare the count with cadence ticks times the dimension cross-product size.
|
|
82
|
+
Check duplicate `(event, time, <dimension keys>)` tuples and undeclared keys,
|
|
83
|
+
and assert that `user_id` and `device_id` are absent. Synthetic `distinct_id`
|
|
84
|
+
values identify series, never people. Do not union these shards into EVENTS
|
|
85
|
+
for funnels, retention, stitching, or user counts. Substitute `{{PREFIX}}`
|
|
86
|
+
with the exact artifact prefix when running SQL outside the story CLI.
|
|
87
|
+
|
|
88
|
+
For `warehouseMetrics`, use the matching `-WAREHOUSE-MANIFEST.json` to resolve
|
|
89
|
+
each table file and schema. Read JSONL with `read_json_auto` or CSV with
|
|
90
|
+
`read_csv_auto` according to that manifest. Check the declared `timeColumn`,
|
|
91
|
+
group keys, `valueColumn`, and extra `columns`; no identity fields are required.
|
|
92
|
+
Use `warehouse` or `warehouse-stats` assertions for stories. The
|
|
93
|
+
automatic warehouse audit runs even without stories. Account for `history`
|
|
94
|
+
backfill and `sparse` point-in-time rows before judging counts or time coverage.
|
|
95
|
+
|
|
96
|
+
### User-event identity checks
|
|
97
|
+
|
|
63
98
|
Run these for every dungeon that uses the identity model (`isAuthEvent` + `attempts` + `identity.avgDevicePerUser`), BEFORE per-pattern checks:
|
|
64
99
|
|
|
65
100
|
```sql
|
|
@@ -775,7 +810,7 @@ Choice depends on whether the JSDoc's stated ranges are load-bearing for the dun
|
|
|
775
810
|
If you edit a hook then query the existing data files, you'll get STALE results. The verifier must re-run the dungeon AND wait for full completion before re-querying:
|
|
776
811
|
|
|
777
812
|
```bash
|
|
778
|
-
|
|
813
|
+
# Keep this run's files for verification and deployment; cleanup needs explicit consent.
|
|
779
814
|
node scripts/verify-runner.mjs dungeons/vertical/<NAME>.js verify-<NAME>
|
|
780
815
|
# Wait for the {"mode":"full","eventCount":...} JSON to print before querying
|
|
781
816
|
```
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Warehouse Metric CRUD Gap Report
|
|
2
|
+
|
|
3
|
+
Dungeon: {{DUNGEON_PATH}}
|
|
4
|
+
Warehouse dir: {{WAREHOUSE_DIR}}
|
|
5
|
+
Dataset: {{DATASET}}
|
|
6
|
+
Source id: {{SOURCE_ID}}
|
|
7
|
+
|
|
8
|
+
Note: {{NOTE}}
|
|
9
|
+
|
|
10
|
+
## Desired Powertools Contract
|
|
11
|
+
|
|
12
|
+
All under `/crud`, POST to execute, GET for docs, standard `client_id` / `region` body convention.
|
|
13
|
+
|
|
14
|
+
| Endpoint | Required | Notes |
|
|
15
|
+
|---|---|---|
|
|
16
|
+
| `createWarehouseMetric` | `project_id`, `source_id`, `name`, `sql`, `value_column` | Optional: `metric_type`, `time_column`, `aggregation`, `refresh`, `description` |
|
|
17
|
+
| `getWarehouseMetrics` | `project_id` | |
|
|
18
|
+
| `getWarehouseMetric` | `project_id`, `metric_id` | |
|
|
19
|
+
| `updateWarehouseMetric` | `project_id`, `metric_id`, `payload` | `source_id` is immutable; delete and recreate to rebind |
|
|
20
|
+
| `deleteWarehouseMetric` | `project_id`, `metric_id` | |
|
|
21
|
+
| `refreshWarehouseMetric` | `project_id`, `metric_id` | Cache invalidation only; does not execute the query |
|
|
22
|
+
| `previewWarehouseMetric` | `project_id`, `source_id`, `sql` | Returns rows; use before save because create does not validate SQL |
|
|
23
|
+
|
|
24
|
+
## Manual Notes
|
|
25
|
+
|
|
26
|
+
- `value_column` is required for every warehouse metric, including numeric ones.
|
|
27
|
+
- `source_id` must come from the actual `/macro/setup-bq-warehouse` response.
|
|
28
|
+
- `previewWarehouseMetric` rejects raw SQL containing `DROP`, `DELETE`, `TRUNCATE`, `ALTER`, `CREATE`, `INSERT`, or `UPDATE` as substrings.
|
|
29
|
+
- `created_at` and `updated_at` therefore fail preview unless the blocked text is removed from the query entirely.
|
|
30
|
+
- `refreshWarehouseMetric` invalidates cache only; it does not execute the query.
|
|
31
|
+
|
|
32
|
+
## Per-table Checklist
|
|
33
|
+
|
|
34
|
+
{{TABLE_CHECKLIST}}
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: warehouse-metrics
|
|
3
|
+
description: 'Use when a completed warehouse dungeon needs its warehouse tables loaded into BigQuery and saved as Mixpanel warehouse metrics. Triggers: "deploy warehouse metrics", "load warehouse tables", "connect warehouse metric source", "save warehouse metrics", after a dungeon has already run.'
|
|
4
|
+
argument-hint: '[dungeon path, e.g. dungeons/user/acme/acme.js]'
|
|
5
|
+
model: claude-opus-4-6
|
|
6
|
+
effort: max
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Deploy Warehouse Tables and Metrics
|
|
10
|
+
|
|
11
|
+
Load the generated warehouse tables for a dungeon into BigQuery, connect the dataset to Mixpanel with the existing powertools macro, preview each metric SQL, and save the warehouse metrics when the CRUD endpoints are available.
|
|
12
|
+
|
|
13
|
+
**Dungeon file:** `$ARGUMENTS`
|
|
14
|
+
|
|
15
|
+
## What it does
|
|
16
|
+
|
|
17
|
+
The orchestrator is `.claude/skills/warehouse-metrics/deploy.mjs`. It works from the warehouse manifest emitted by a completed dungeon run.
|
|
18
|
+
|
|
19
|
+
1. Loads the dungeon with the package loader and requires `warehouseMetrics`.
|
|
20
|
+
2. Finds the latest warehouse manifest and table files, or uses `--data-prefix` when supplied.
|
|
21
|
+
3. Normalizes the BigQuery dataset name to `dm_<dungeon_name>`, unless `--dataset` overrides it.
|
|
22
|
+
4. Writes per-table SQL files under the dungeon's sibling `warehouse/` directory with `{{DATASET}}` already substituted for the dataset id. The metric SQL itself is fully qualified to `mixpanel-gtm-training.<dataset>.<table>`.
|
|
23
|
+
5. Runs a non-destructive `bq ls` preflight first so missing CLI or ADC fails before any write.
|
|
24
|
+
6. Probes `GET /crud/getWarehouseMetrics` first and, when available, lists metrics before any BigQuery writes so malformed list payloads or upstream 500s fail early.
|
|
25
|
+
7. Loads the tables into BigQuery with explicit schemas from the manifest, then reuses `.claude/skills/powertools/pt.mjs` to call `/macro/setup-bq-warehouse`. Do not reimplement that flow here. Only the macro performs the GCP-side IAM grant.
|
|
26
|
+
8. If the docs route 404s, the script still completes the BigQuery load and source setup, then writes `warehouse/GAPS.md` from the template for manual metric setup.
|
|
27
|
+
9. When the endpoint exists, it dedupes by `name`, previews SQL with `previewWarehouseMetric`, then saves new metrics with `createWarehouseMetric`.
|
|
28
|
+
|
|
29
|
+
## Flags
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
node .claude/skills/warehouse-metrics/deploy.mjs <dungeon-path> [--dataset dm_name] [--data-prefix path/prefix] [--dry-run]
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
- `--dataset`: override the normalized `dm_<name>` dataset.
|
|
36
|
+
- `--data-prefix`: explicit run artifact prefix, for example `/tmp/run/warehouse-demo` for `/tmp/run/warehouse-demo-WAREHOUSE-MANIFEST.json`.
|
|
37
|
+
- `--dry-run`: prints the full `bq ls`, docs probe, metric list, load, source, preview, and create plan without executing commands or requiring credentials. It still writes the SQL files and renders `warehouse/GAPS.md` for review.
|
|
38
|
+
|
|
39
|
+
## Preflight
|
|
40
|
+
|
|
41
|
+
- The dungeon must have passed `/verify-dungeon` and produced local warehouse
|
|
42
|
+
files with `writeToDisk: true` and `gzip: false`. Use the exact verified
|
|
43
|
+
`--data-prefix` and its matching `-WAREHOUSE-MANIFEST.json`; preserve all table
|
|
44
|
+
files referenced by the manifest. Do not use blanket prune before deployment.
|
|
45
|
+
- Live mode requires `.env` `BEARER_TOKEN` for powertools.
|
|
46
|
+
- Live mode requires working `bq` / gcloud ADC.
|
|
47
|
+
- The dungeon must already have `credentials.projectId` from `/create-project`.
|
|
48
|
+
- The Power Tools runtime principal needs `roles/resourcemanager.projectIamAdmin`
|
|
49
|
+
to grant project-level `roles/bigquery.jobUser`, and `roles/bigquery.admin`
|
|
50
|
+
(or equivalent permissions) for dataset creation and ACL changes. The macro
|
|
51
|
+
grants the Mixpanel principal `roles/bigquery.dataViewer` on the source dataset.
|
|
52
|
+
Keep these prerequisites; do not describe IAM as currently blocked. The operator
|
|
53
|
+
confirms the grant is fixed. Local Power Tools revision `7ae78aa` records the
|
|
54
|
+
successful one-shot live path and legacy `READER`/`WRITER`/`OWNER` ACL fix.
|
|
55
|
+
This audit used local evidence only and made no live calls.
|
|
56
|
+
|
|
57
|
+
## Warnings
|
|
58
|
+
|
|
59
|
+
- `bq load --replace` overwrites the destination table contents. Treat live execution as destructive for existing warehouse tables and get explicit user confirmation before running it.
|
|
60
|
+
- `createWarehouseMetric` does not validate SQL. This skill previews every query first so the summary means something.
|
|
61
|
+
- `source_id` is immutable on update. This flow is create-or-skip by metric name; it does not try to update a metric onto a new source.
|
|
62
|
+
- `refreshWarehouseMetric` only invalidates cache. It does not execute the query.
|
|
63
|
+
|
|
64
|
+
## Preview gotcha
|
|
65
|
+
|
|
66
|
+
`previewWarehouseMetric` rejects raw SQL containing `DROP`, `DELETE`, `TRUNCATE`, `ALTER`, `CREATE`, `INSERT`, or `UPDATE` as plain substrings. That means `created_at` trips `CREATE` and `updated_at` trips `UPDATE`.
|
|
67
|
+
|
|
68
|
+
This skill fails clearly in that case. It does not pretend preview succeeded. It also does not auto-rewrite the SQL. Aliasing only helps if the blocked text disappears from the query entirely. If the generated table uses one of those names, rename the column or finish the metric manually.
|
|
69
|
+
|
|
70
|
+
## Typical flow
|
|
71
|
+
|
|
72
|
+
### 1. Show the plan
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
node .claude/skills/warehouse-metrics/deploy.mjs <dungeon-path> --data-prefix <verified-prefix> --dry-run
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Review the printed commands, the emitted SQL files, and the rendered `warehouse/GAPS.md`.
|
|
79
|
+
|
|
80
|
+
### 2. Confirm live execution
|
|
81
|
+
|
|
82
|
+
Live mode writes or replaces BigQuery tables and saves metrics into a real Mixpanel project. Confirm with the user before running it.
|
|
83
|
+
|
|
84
|
+
### 3. Run live
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
node .claude/skills/warehouse-metrics/deploy.mjs <dungeon-path> --data-prefix <verified-prefix>
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
### 4. Report
|
|
91
|
+
|
|
92
|
+
Relay:
|
|
93
|
+
|
|
94
|
+
- dataset loaded
|
|
95
|
+
- source id returned by `/macro/setup-bq-warehouse`
|
|
96
|
+
- metrics saved vs skipped by name
|
|
97
|
+
- SQL file paths
|
|
98
|
+
- whether `warehouse/GAPS.md` was written or an older one was intentionally left in place
|
|
99
|
+
|
|
100
|
+
## Error handling
|
|
101
|
+
|
|
102
|
+
- A missing manifest or missing warehouse files is a hard stop. Tell the user to run `node scripts/run-dungeon.mjs <dungeon>` first, or pass `--data-prefix`.
|
|
103
|
+
- A `GET /crud/getWarehouseMetrics` 404 is the only fallback. The script still loads tables and connects the source, then writes `warehouse/GAPS.md` instead of pretending metrics were saved.
|
|
104
|
+
- Any other docs probe error, `bq` failure, preview failure, or create failure surfaces immediately. No catch-and-continue.
|
|
105
|
+
- Existing `warehouse/GAPS.md` is never deleted silently. If a new one is written, the script says so. If CRUD is now available, the old file is left in place and reported as stale.
|