@ak--47/dungeon-master 1.7.0 → 1.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/analyze-soup/SKILL.md +21 -11
- package/.claude/skills/create-dungeon/SKILL.md +49 -10
- package/.claude/skills/create-project/SKILL.md +22 -3
- package/.claude/skills/create-project/context.mjs +89 -0
- package/.claude/skills/create-project/provision.mjs +1 -60
- package/.claude/skills/headless-build/SKILL.md +18 -1
- package/.claude/skills/powertools/SKILL.md +20 -1
- package/.claude/skills/release-check/SKILL.md +99 -0
- package/.claude/skills/verify-dungeon/SKILL.md +71 -16
- package/.claude/skills/verify-dungeon/references/counting-semantics.md +14 -0
- package/.claude/skills/verify-dungeon/references/report-format.md +18 -1
- package/.claude/skills/verify-dungeon/references/sql-recipes.md +36 -1
- package/.claude/skills/warehouse-metrics/GAPS-template.md +34 -0
- package/.claude/skills/warehouse-metrics/SKILL.md +105 -0
- package/.claude/skills/warehouse-metrics/deploy.mjs +651 -0
- package/.claude/skills/write-hooks/SKILL.md +33 -3
- package/CHANGELOG.md +142 -0
- package/HOOKS.md +105 -5
- package/README.md +228 -0
- package/docs/guides/1.8.0-upgrade-guide.md +151 -0
- package/dungeons/technical/warehouse.js +187 -0
- package/index.js +116 -2
- package/lib/core/config-validator.js +21 -0
- package/lib/core/dungeon-loader.js +1 -1
- package/lib/core/storage.js +51 -3
- package/lib/generators/standalone.js +248 -0
- package/lib/generators/warehouse.js +828 -0
- package/lib/orchestrators/mixpanel-sender.js +27 -2
- package/lib/orchestrators/user-loop.js +1 -0
- package/lib/templates/story-spec.schema.json +41 -16
- package/lib/utils/utils.js +37 -12
- package/lib/verify/index.js +1 -0
- package/lib/verify/story-runner.js +71 -8
- package/lib/verify/warehouse.js +683 -0
- package/package.json +4 -2
- package/scripts/verify-stories.mjs +150 -44
- package/types.d.ts +303 -4
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: write-hooks
|
|
3
3
|
description: Use when an existing dungeon needs engineered story trends or "magic number" patterns — writes the `hook` function using atom helpers and high-level patterns. Adds no new event flags; never mutates the schema.
|
|
4
|
-
argument-hint: [path/to/dungeon.js] [free-text story / trend description]
|
|
4
|
+
argument-hint: '[path/to/dungeon.js] [free-text story / trend description]'
|
|
5
5
|
model: claude-opus-4-6
|
|
6
6
|
effort: max
|
|
7
7
|
---
|
|
@@ -68,7 +68,7 @@ property values that differ by cohort, injected bursts, lifecycle waves.
|
|
|
68
68
|
Hooks fire in this order per user (see `CLAUDE.md` for the canonical reference):
|
|
69
69
|
|
|
70
70
|
1. `"user"` — profile created. Mutate in place; return ignored.
|
|
71
|
-
2. `"scd-pre"` — SCD entries created. Mutate in place
|
|
71
|
+
2. `"scd-pre"` — SCD entries created. Mutate in place; return ignored.
|
|
72
72
|
3. For each funnel: `"funnel-pre"` → `"event"` (per step) → `"funnel-post"`.
|
|
73
73
|
|
|
74
74
|
**`funnel-pre` is now reliable for temporal patterns.** Usage funnels advance a
|
|
@@ -76,7 +76,7 @@ cursor after each run, so successive `meta.firstEventTime` values spread across
|
|
|
76
76
|
the user's active window. Persona and world-event modifiers apply BEFORE the
|
|
77
77
|
hook — the hook has final authority on `conversionRate`, `timeToConvert`, and
|
|
78
78
|
`props`.
|
|
79
|
-
4. `"event"` — for non-funnel
|
|
79
|
+
4. `"event"` — for non-funnel user events from `events[]`. Return value REPLACES the event.
|
|
80
80
|
5. `"everything"` — array of ALL the user's events. Return array to replace.
|
|
81
81
|
|
|
82
82
|
**Most engineered trends belong in `everything`.** It sees the full user stream,
|
|
@@ -86,6 +86,36 @@ and you can mutate freely.
|
|
|
86
86
|
Storage-only hooks (`ad-spend`, `group`, `mirror`, `lookup`) fire later in the
|
|
87
87
|
pipeline and don't see the same `meta` shape.
|
|
88
88
|
|
|
89
|
+
### Cadence streams and warehouse rows (v1.8.0)
|
|
90
|
+
|
|
91
|
+
These hooks sit outside the per-user sequence and never enter `everything`.
|
|
92
|
+
Neither has person metadata (`meta.profile`, auth state, sessions, or SCDs).
|
|
93
|
+
|
|
94
|
+
- `standaloneEvents`: `type === 'standalone'` fires on storage push before the
|
|
95
|
+
user loop. Read `meta.spec` and `meta.config` to identify the stream. Return
|
|
96
|
+
the record object or an array of records. Returning `undefined` drops it.
|
|
97
|
+
Mutating without returning is therefore insufficient. Preserve required keys
|
|
98
|
+
and use fresh `insert_id` values for clones. Its synthetic `distinct_id` is a
|
|
99
|
+
series identifier, never a person or a retention cohort.
|
|
100
|
+
- `warehouseMetrics`: `type === 'warehouse'` fires after user generation, once
|
|
101
|
+
per materialized row. Mutate the row in place; its return value is ignored.
|
|
102
|
+
Keep the time column and group keys stable. Modify only the value column and
|
|
103
|
+
declared extra columns. Meta includes `spec`, `config`, `metricName`,
|
|
104
|
+
`bucketIndex`, `bucketCount`, `grain`, `seriesKey`, `isBackfill`, and `raw`.
|
|
105
|
+
`raw` describes plus/minus source aggregates before scaling, noise, and carry.
|
|
106
|
+
|
|
107
|
+
Do not add either schema here. Send missing `standaloneEvents` properties or
|
|
108
|
+
warehouse `columns` back to `/create-dungeon`. Warehouse sources consume user
|
|
109
|
+
`events[]` only, including both plus and minus legs; they cannot consume cadence
|
|
110
|
+
streams. Standalone value functions use tick context, warehouse column functions
|
|
111
|
+
use bucket context; neither supplies a user profile.
|
|
112
|
+
|
|
113
|
+
Verify standalone stories with disk-backed `duckdb` assertions against
|
|
114
|
+
`{{PREFIX}}-STANDALONE*.json`. The user-event emulator and `--in-memory` CLI mode
|
|
115
|
+
do not evaluate this stream. Warehouse stories can use `warehouse` assertions
|
|
116
|
+
or `warehouse-stats` assertions; automatic warehouse audits also run without
|
|
117
|
+
stories. Hand off to `/verify-dungeon` with an explicit artifact prefix.
|
|
118
|
+
|
|
89
119
|
## Hook meta — identity context
|
|
90
120
|
|
|
91
121
|
Inside `funnel-pre` and `funnel-post`:
|
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,148 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to `@ak--47/dungeon-master`.
|
|
4
4
|
|
|
5
|
+
## 1.8.0 — 2026-09-10
|
|
6
|
+
|
|
7
|
+
### Added — `standaloneEvents`: identity-less metric snapshots
|
|
8
|
+
|
|
9
|
+
A new top-level config key that generates records describing a **system, not a
|
|
10
|
+
person**. They carry no `user_id` and no `device_id`. Before 1.8.0 the only
|
|
11
|
+
identity-less stream the engine could produce was `$ad_spend` via `hasAdSpend`,
|
|
12
|
+
which is hard-coded to one shape, one cadence, and a Mixpanel reserved event
|
|
13
|
+
name. `standaloneEvents` is the general form.
|
|
14
|
+
|
|
15
|
+
```js
|
|
16
|
+
standaloneEvents: [{
|
|
17
|
+
event: 'cdn_egress',
|
|
18
|
+
cadence: 'day', // 'hour' | 'day' | 'week', default 'day'
|
|
19
|
+
dimensions: { region: ['us-east', 'us-west', 'eu'] }, // cross-producted
|
|
20
|
+
distinctIdFrom: 'region', // synthetic id, never a person
|
|
21
|
+
properties: {
|
|
22
|
+
gb_out: (ctx) => 400 + ctx.tickIndex * 3,
|
|
23
|
+
cost_usd: (ctx) => (400 + ctx.tickIndex * 3) * 0.085,
|
|
24
|
+
p95_ms: [120, 140, 160],
|
|
25
|
+
},
|
|
26
|
+
}]
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
- One record per cadence tick per dimension cross-product row.
|
|
30
|
+
- Ticks start at the dataset start and step by the cadence. The last tick is the
|
|
31
|
+
final one at or before the dataset end, so nothing lands in the future.
|
|
32
|
+
- Each record carries `event`, `time`, `insert_id`, `distinct_id`, every
|
|
33
|
+
dimension as a flat property, and every resolved entry in `properties`.
|
|
34
|
+
- `distinct_id` is the value of the dimension named by `distinctIdFrom`, else the
|
|
35
|
+
event name. It exists so Mixpanel accepts the record; it never maps to a person.
|
|
36
|
+
- Property value functions receive a `StandaloneValueContext`:
|
|
37
|
+
`{ time, config, dimensions, tickIndex, tickCount, cadence, event }`.
|
|
38
|
+
`tickIndex / (tickCount - 1)` is window progress — use it to shape a trend.
|
|
39
|
+
- New hook type `standalone` (storage-only). Return the record or an array of
|
|
40
|
+
records; returning nothing drops the record. `meta.spec` carries the resolved
|
|
41
|
+
stream config. The `warehouse` hook instead mutates its row and ignores returns.
|
|
42
|
+
- Lands in `result.standaloneEventData`, writes to a `-STANDALONE` file shard,
|
|
43
|
+
and imports to Mixpanel as its own event stream.
|
|
44
|
+
- Validation **throws** on a malformed entry rather than skipping it. A silent
|
|
45
|
+
skip would drop a whole data stream without the author noticing.
|
|
46
|
+
|
|
47
|
+
New types: `StandaloneEventConfig`, `ResolvedStandaloneEventConfig`,
|
|
48
|
+
`StandaloneValueContext`, `HookMetaStandalone`. `WritePaths` gains
|
|
49
|
+
`standaloneFiles`; `Result` gains `standaloneEventData`; `hookTypes` gains
|
|
50
|
+
`"standalone"`.
|
|
51
|
+
|
|
52
|
+
**Output compatibility.** Additive only. A config without `standaloneEvents` is
|
|
53
|
+
byte-identical to 1.7.0 — the generation pass is gated on the key being present,
|
|
54
|
+
so the seeded RNG stream is untouched. `config.standaloneEvents` normalizes to
|
|
55
|
+
`[]` when absent. Event determinism comparisons exclude the fresh `insert_id`.
|
|
56
|
+
|
|
57
|
+
New tests: `tests/unit/standalone-events.test.js` (25),
|
|
58
|
+
`tests/integration/standalone-events.test.js` (15).
|
|
59
|
+
|
|
60
|
+
### Added — `warehouseMetrics`: manifest-driven warehouse source tables
|
|
61
|
+
|
|
62
|
+
A new top-level config key that materializes warehouse-ready tables from the
|
|
63
|
+
run's own events after generation completes. This is the local source-table side
|
|
64
|
+
of a warehouse metric demo: bookings rollups, active subscription levels, ARR
|
|
65
|
+
snapshots, and other time-series tables that should read like a real warehouse.
|
|
66
|
+
|
|
67
|
+
```js
|
|
68
|
+
warehouseMetrics: [{
|
|
69
|
+
name: 'daily_new_bookings',
|
|
70
|
+
source: { event: 'new_booking', measure: 'sum', property: 'booking_value' },
|
|
71
|
+
valueColumn: 'bookings',
|
|
72
|
+
}]
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
- Supports additive and point-in-time metrics.
|
|
76
|
+
- Grain: `day`, `week`, `month`.
|
|
77
|
+
- Supports subtractive `minus` legs, `groupBy` on up to two declared keys,
|
|
78
|
+
optional `history` backfill, sparse point-in-time emission, seeded `noise`,
|
|
79
|
+
`scale`, and derived `columns`.
|
|
80
|
+
- Lands in `result.warehouseMetricData` keyed by metric name and emits
|
|
81
|
+
`result.warehouseManifest` with table schemas, SQL, and recommended
|
|
82
|
+
aggregation.
|
|
83
|
+
- Writes `<name>-WAREHOUSE-<table>.csv|json` plus
|
|
84
|
+
`<name>-WAREHOUSE-MANIFEST.json` when `writeToDisk` is enabled.
|
|
85
|
+
- Never imports through `token`. Warehouse deploy is a separate flow.
|
|
86
|
+
- New hook type `warehouse` fires once per materialized row with
|
|
87
|
+
`metricName`, `bucketIndex`, `bucketCount`, `grain`, `seriesKey`,
|
|
88
|
+
`isBackfill`, and `raw` bucket stats.
|
|
89
|
+
- Warehouse verification adds `warehouse` / `warehouse-stats` story breakdowns
|
|
90
|
+
plus automatic audits over declared columns, gaps, monotonic time, empty numeric
|
|
91
|
+
cells, and sparse first-bucket coverage. Source correlation is available to story
|
|
92
|
+
assertions; it is not an automatic pass/fail gate.
|
|
93
|
+
- Sparse point-in-time comparison carries emitted levels across missing buckets,
|
|
94
|
+
excludes history from correlation, and compares the first live delta to the scaled
|
|
95
|
+
baseline. Warehouse generation preserves the user event stream, including with noise.
|
|
96
|
+
- Disk verification parses quoted multiline CSV records with `csv-parse` and keeps
|
|
97
|
+
disk and in-memory audit results consistent.
|
|
98
|
+
|
|
99
|
+
### Added — `/warehouse-metrics`: BigQuery load + warehouse metric save flow
|
|
100
|
+
|
|
101
|
+
The shipped skill at `.claude/skills/warehouse-metrics/` loads the generated
|
|
102
|
+
warehouse tables into BigQuery, connects that dataset to Mixpanel with the
|
|
103
|
+
existing powertools macro, previews each metric SQL, and saves new metrics when
|
|
104
|
+
the CRUD endpoints are available.
|
|
105
|
+
|
|
106
|
+
- Uses the emitted warehouse manifest as the contract.
|
|
107
|
+
- Maps manifest `recommendedAggregation: 'last value'` to the API's
|
|
108
|
+
`aggregation: 'last_value'`.
|
|
109
|
+
- If `GET /crud/getWarehouseMetrics` returns 404, the script still completes the
|
|
110
|
+
BigQuery load and source setup, then writes `warehouse/GAPS.md` for manual
|
|
111
|
+
metric creation.
|
|
112
|
+
- Preview fails fast on the raw substring block (`CREATE`, `UPDATE`, etc.), so
|
|
113
|
+
identifiers like `created_at` and `updated_at` are a real deploy-time trap.
|
|
114
|
+
- The canonical skill name is `/warehouse-metrics`. Bundled commands and handoffs
|
|
115
|
+
use `.claude/skills/warehouse-metrics/`.
|
|
116
|
+
- Uses the shipped Powertools warehouse CRUD and IAM setup macro. Runtime IAM is
|
|
117
|
+
configured; permission failures still stop deployment with the original error.
|
|
118
|
+
- Dry-run works before project provisioning, using placeholders without credentials.
|
|
119
|
+
Live table replacement requires explicit operator consent; the script does not prompt.
|
|
120
|
+
|
|
121
|
+
### Changed — skill workflow and provisioning context
|
|
122
|
+
|
|
123
|
+
- All nine bundled skills have parsed string argument hints and matching folder names.
|
|
124
|
+
Release tests check frontmatter and warehouse handoffs.
|
|
125
|
+
- Authoring, hooks, and verification distinguish person events, identity-less cadence
|
|
126
|
+
events, and identity-free warehouse rows. Synthetic IDs never count as people.
|
|
127
|
+
- Verification preserves the explicit run prefix and local uncompressed warehouse
|
|
128
|
+
artifacts for deployment. Soup analysis remains scoped to user-event timestamps.
|
|
129
|
+
- Project business context includes separate cadence and warehouse summaries without
|
|
130
|
+
evaluating property functions or including credentials.
|
|
131
|
+
- Headless builds preserve warehouse history and use query preview for fresh results;
|
|
132
|
+
warehouse refresh only invalidates the saved metric cache.
|
|
133
|
+
|
|
134
|
+
### Added - `/release-check`
|
|
135
|
+
|
|
136
|
+
The new `/release-check` skill audits tests, determinism, documentation, package
|
|
137
|
+
contents, and release handoffs. `.agents/skills` and `.github/skills` link to the
|
|
138
|
+
canonical `.claude/skills` directory for shared agent discovery. Publishing remains
|
|
139
|
+
an explicitly authorized operator action.
|
|
140
|
+
|
|
141
|
+
### Changed — `streamCSV` preserves falsy cells
|
|
142
|
+
|
|
143
|
+
CSV serialization now writes `0` and `false` as literal cell values instead of
|
|
144
|
+
empty strings. If downstream warehouse SQL or fixtures were treating blank cells
|
|
145
|
+
as zero or false, update them to read the actual value.
|
|
146
|
+
|
|
5
147
|
## 1.7.0 — 2026-09-03
|
|
6
148
|
|
|
7
149
|
The engine round for DM4 v5. Executes the 1.6.4 "Deferred to 1.7.0" table plus
|
package/HOOKS.md
CHANGED
|
@@ -21,10 +21,12 @@ hook: function (record, type, meta) { ... return record; }
|
|
|
21
21
|
| `event` | `events.js:176` | Single event (flat props) | **Used** (replaces event) | `user: { distinct_id }`, `config`, `datasetStart`, `datasetEnd` |
|
|
22
22
|
| `funnel-post` | `funnels.js:153` | Array of funnel events | Ignored (mutate in-place) | `user`, `profile`, `scd`, `funnel`, `config`, `experiment` |
|
|
23
23
|
| `everything` | `user-loop.js:280` | Array of ALL user events | **Used** if array returned | `profile`, `scd`, `config`, `datasetStart`, `datasetEnd`, `userIsBornInDataset`, `authTime`, `isPreAuth`, `persona` |
|
|
24
|
-
| `ad-spend` | `storage.js` | Ad spend event |
|
|
25
|
-
| `group` | `storage.js` | Group profile |
|
|
26
|
-
| `mirror` | `storage.js` | Mirror data point |
|
|
27
|
-
| `lookup` | `storage.js` | Lookup table entry |
|
|
24
|
+
| `ad-spend` | `storage.js` | Ad spend event | Object or array used | -- |
|
|
25
|
+
| `group` | `storage.js` | Group profile | Object or array used | -- |
|
|
26
|
+
| `mirror` | `storage.js` | Mirror data point | Object or array used | -- |
|
|
27
|
+
| `lookup` | `storage.js` | Lookup table entry | Object or array used | -- |
|
|
28
|
+
| `standalone` | `storage.js`, before user loop | Identity-less cadence event | Object or array used; `undefined` drops | `spec`, `config`, `datasetStart`, `datasetEnd` |
|
|
29
|
+
| `warehouse` | `storage.js` | One materialized warehouse row | Ignored | `spec`, `config`, `metricName`, `bucketIndex`, `bucketCount`, `grain`, `seriesKey`, `isBackfill`, `raw`, `datasetStart`, `datasetEnd` |
|
|
28
30
|
|
|
29
31
|
**Per-user execution order:** `user` -> `scd-pre` -> `funnel-pre` -> `event` -> `funnel-post` -> `everything`
|
|
30
32
|
|
|
@@ -36,7 +38,105 @@ double-fire mutations.
|
|
|
36
38
|
**Return rules:**
|
|
37
39
|
- `event`: return the (possibly replaced) event object.
|
|
38
40
|
- `everything`: return the (possibly modified) array. Filtered array removes events.
|
|
39
|
-
-
|
|
41
|
+
- `ad-spend`, `group`, `mirror`, `lookup`, `standalone`: return the record or an
|
|
42
|
+
array of records. Returning `undefined` drops the record.
|
|
43
|
+
- `user`, `scd-pre`, `funnel-pre`, `funnel-post`, `warehouse`: mutate `record`
|
|
44
|
+
in place. Return value is ignored.
|
|
45
|
+
|
|
46
|
+
`standaloneEvents` runs before the user loop; `warehouseMetrics` materializes
|
|
47
|
+
after it. Neither hook receives person metadata or enters `everything`.
|
|
48
|
+
Standalone synthetic `distinct_id` values identify series, never people. Use
|
|
49
|
+
disk-backed `duckdb` assertions on `{{PREFIX}}-STANDALONE*.json` for cadence
|
|
50
|
+
stories. Warehouse stories support `warehouse` and `warehouse-stats`
|
|
51
|
+
assertions, with automatic table audits even when no stories are exported.
|
|
52
|
+
|
|
53
|
+
### 1.1 Warehouse rows (`type === 'warehouse'`)
|
|
54
|
+
|
|
55
|
+
`warehouse` fires once per materialized row, after the user loop and before the
|
|
56
|
+
rows are written to disk. the row already matches the manifest contract:
|
|
57
|
+
|
|
58
|
+
- `timeColumn`
|
|
59
|
+
- every `source.groupBy` key, in order
|
|
60
|
+
- `valueColumn`
|
|
61
|
+
- every declared key in `columns`
|
|
62
|
+
|
|
63
|
+
declare every key up front. warehouse containers are created with a fixed column
|
|
64
|
+
list, and the manifest is built from that same list. an undeclared key is not
|
|
65
|
+
part of the output contract even if it exists briefly in memory.
|
|
66
|
+
|
|
67
|
+
`meta.seriesKey` is the joined group tuple in `source.groupBy` order, separated
|
|
68
|
+
by `|`. examples:
|
|
69
|
+
|
|
70
|
+
- no `groupBy` → `''`
|
|
71
|
+
- `groupBy: ['region']` and `row.region === 'us'` → `'us'`
|
|
72
|
+
- `groupBy: ['region', 'plan_tier']` and `row.region === 'us'`, `row.plan_tier === 'enterprise'` → `'us|enterprise'`
|
|
73
|
+
|
|
74
|
+
`meta.bucketIndex` and `meta.bucketCount` are chronological and include history
|
|
75
|
+
buckets even when `sparse: true` skips repeated rows. `meta.isBackfill` is true
|
|
76
|
+
for the synthetic buckets created by `history`. `meta.raw` is the bucketed
|
|
77
|
+
source truth before `scale`, `noise`, and point-in-time carry-forward.
|
|
78
|
+
|
|
79
|
+
Treat the time axis as immutable. `row[spec.timeColumn]` drives ordering,
|
|
80
|
+
manifest SQL, and warehouse verification. mutate the value column or declared
|
|
81
|
+
extra columns instead.
|
|
82
|
+
|
|
83
|
+
Recipe: scale a point-in-time level for an in-window story slice
|
|
84
|
+
|
|
85
|
+
```js
|
|
86
|
+
warehouseMetrics: [{
|
|
87
|
+
name: 'daily_active_subscriptions',
|
|
88
|
+
type: 'point-in-time',
|
|
89
|
+
source: { event: 'subscription_started', minus: 'subscription_cancelled', groupBy: 'region' },
|
|
90
|
+
baseline: 40,
|
|
91
|
+
timeColumn: 'date',
|
|
92
|
+
valueColumn: 'active_subscriptions',
|
|
93
|
+
columns: { lifted: false },
|
|
94
|
+
}],
|
|
95
|
+
|
|
96
|
+
hook: (row, type, meta) => {
|
|
97
|
+
if (type !== 'warehouse') return row;
|
|
98
|
+
if (meta.metricName !== 'daily_active_subscriptions') return row;
|
|
99
|
+
if (meta.isBackfill) return row;
|
|
100
|
+
|
|
101
|
+
const liveIndex = meta.bucketIndex - meta.spec.history;
|
|
102
|
+
if (meta.seriesKey === 'us' && liveIndex >= 7 && liveIndex < 14) {
|
|
103
|
+
row.active_subscriptions = Math.round(row.active_subscriptions * 1.2);
|
|
104
|
+
row.lifted = true;
|
|
105
|
+
}
|
|
106
|
+
return row;
|
|
107
|
+
}
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Recipe: apply an incident dip to one series only
|
|
111
|
+
|
|
112
|
+
```js
|
|
113
|
+
warehouseMetrics: [{
|
|
114
|
+
name: 'daily_new_bookings',
|
|
115
|
+
source: {
|
|
116
|
+
event: 'new_booking',
|
|
117
|
+
measure: 'sum',
|
|
118
|
+
property: 'booking_value',
|
|
119
|
+
groupBy: ['region', 'plan_tier'],
|
|
120
|
+
},
|
|
121
|
+
timeColumn: 'date',
|
|
122
|
+
valueColumn: 'bookings',
|
|
123
|
+
columns: { incident: false },
|
|
124
|
+
}],
|
|
125
|
+
|
|
126
|
+
hook: (row, type, meta) => {
|
|
127
|
+
if (type !== 'warehouse') return row;
|
|
128
|
+
if (meta.metricName !== 'daily_new_bookings') return row;
|
|
129
|
+
if (meta.isBackfill) return row;
|
|
130
|
+
if (meta.seriesKey !== 'us|enterprise') return row;
|
|
131
|
+
|
|
132
|
+
const liveIndex = meta.bucketIndex - meta.spec.history;
|
|
133
|
+
if (liveIndex >= 14 && liveIndex <= 16) {
|
|
134
|
+
row.bookings = Math.round(row.bookings * 0.35);
|
|
135
|
+
row.incident = true;
|
|
136
|
+
}
|
|
137
|
+
return row;
|
|
138
|
+
}
|
|
139
|
+
```
|
|
40
140
|
|
|
41
141
|
**What 1.7.0 changed for hooks.** No hook signature, `meta` field, or firing
|
|
42
142
|
order changed, and the hook-helper atoms and patterns are untouched. What a hook
|
package/README.md
CHANGED
|
@@ -91,6 +91,11 @@ const result = await DUNGEON_MASTER({
|
|
|
91
91
|
console.log(result.importResults);
|
|
92
92
|
```
|
|
93
93
|
|
|
94
|
+
`token` imports event-shaped streams only: events, users, groups, ad spend, and
|
|
95
|
+
`standaloneEvents`. `warehouseMetrics` does **not** import through this path.
|
|
96
|
+
warehouse tables are materialized locally and need a separate warehouse deploy
|
|
97
|
+
step after the run.
|
|
98
|
+
|
|
94
99
|
## dungeons
|
|
95
100
|
|
|
96
101
|
a dungeon is a javascript file that exports a configuration object. it defines your entire data model: events, funnels, user properties, group analytics, SCDs, and a hook function that engineers discoverable patterns into the data.
|
|
@@ -537,6 +542,8 @@ dungeon-master generates multiple data types that mirror a real analytics implem
|
|
|
537
542
|
| SCDs | `scdProps` | slowly changing dimensions (subscription tier over time) |
|
|
538
543
|
| lookup tables | `lookupTables` | dimension tables (product catalog, region mapping) |
|
|
539
544
|
| ad spend | `hasAdSpend` | daily ad spend with impressions, clicks, cost metrics |
|
|
545
|
+
| standalone events | `standaloneEvents` | identity-less metric snapshots on a cadence (infrastructure, finance, ops) |
|
|
546
|
+
| warehouse metrics | `warehouseMetrics` | warehouse source tables derived from generated events, with a manifest for downstream deploy |
|
|
540
547
|
| mirror datasets | `mirrorProps` | transformed copies of event data (A/B versions) |
|
|
541
548
|
| organic text | `createTextGenerator` | reviews, support tickets, search queries, chat messages |
|
|
542
549
|
|
|
@@ -606,6 +613,222 @@ variant assignment is **sticky by default**: a deterministic hash of `user_id` +
|
|
|
606
613
|
|
|
607
614
|
**the variant lands on the user profile (1.7.0).** every exposed user carries `"Experiment: <name>": "<variant>"` (e.g. `"Experiment: Checkout Redesign": "New Checkout"`), so the funnel breaks down by variant in Mixpanel with a user-property breakdown — no cohort built from the exposure event. stamped when the user is first exposed (respects `startDaysBeforeEnd`); never-exposed users carry nothing; the `user` hook fires before exposure and does not see it, the `everything` hook does. `stampProfile: false` turns it off; `sticky: false` implies off. measured: 0 mismatches between the profile value and the `Variant name` on 13,005 exposure events.
|
|
608
615
|
|
|
616
|
+
## standalone events (identity-less metric snapshots)
|
|
617
|
+
|
|
618
|
+
`standaloneEvents` generates records that describe a **system, not a person**. they carry
|
|
619
|
+
no `user_id` and no `device_id`. use them for infrastructure, finance, and ops telemetry:
|
|
620
|
+
daily CDN egress per region, weekly billing rollups per plan tier, hourly queue depth per
|
|
621
|
+
cluster. `hasAdSpend` is the same idea hard-coded to `$ad_spend`; this is the general form
|
|
622
|
+
and it does not use a Mixpanel reserved event name.
|
|
623
|
+
|
|
624
|
+
```javascript
|
|
625
|
+
standaloneEvents: [
|
|
626
|
+
{
|
|
627
|
+
event: 'cdn_egress',
|
|
628
|
+
cadence: 'day', // 'hour' | 'day' | 'week' (default 'day')
|
|
629
|
+
dimensions: { region: ['us-east', 'us-west', 'eu'] }, // cross-producted
|
|
630
|
+
distinctIdFrom: 'region', // synthetic id, never a person
|
|
631
|
+
properties: {
|
|
632
|
+
gb_out: (ctx) => 400 + ctx.tickIndex * 3, // shape a trend across the window
|
|
633
|
+
cost_usd: (ctx) => (400 + ctx.tickIndex * 3) * 0.085,
|
|
634
|
+
p95_ms: [120, 140, 160], // same ValueValid forms as event props
|
|
635
|
+
},
|
|
636
|
+
},
|
|
637
|
+
{
|
|
638
|
+
event: 'billing_rollup',
|
|
639
|
+
cadence: 'week',
|
|
640
|
+
dimensions: { tier: ['free', 'pro', 'max'] },
|
|
641
|
+
properties: { mrr_usd: (ctx) => ..., churn_usd: (ctx) => ... },
|
|
642
|
+
},
|
|
643
|
+
]
|
|
644
|
+
```
|
|
645
|
+
|
|
646
|
+
the engine emits **one record per cadence tick per dimension cross-product row**. the
|
|
647
|
+
example above produces 3 records per day (`cdn_egress`) plus 3 records per week
|
|
648
|
+
(`billing_rollup`). ticks start at the dataset start and step by the cadence; the last tick
|
|
649
|
+
is the final one at or before the dataset end, so nothing lands in the future.
|
|
650
|
+
|
|
651
|
+
each record carries `event`, `time`, `insert_id`, `distinct_id`, every dimension as a flat
|
|
652
|
+
property, and every resolved entry in `properties`.
|
|
653
|
+
|
|
654
|
+
| field | behavior |
|
|
655
|
+
|---|---|
|
|
656
|
+
| `event` | required, unique across `standaloneEvents` |
|
|
657
|
+
| `cadence` | `'hour'`, `'day'`, or `'week'`. default `'day'` |
|
|
658
|
+
| `dimensions` | object of non-empty arrays, cross-producted. omit for one record per tick |
|
|
659
|
+
| `distinctIdFrom` | must name a declared dimension. omitted → `distinct_id` is the event name |
|
|
660
|
+
| `properties` | keys may not collide with a dimension or with `event`/`time`/`insert_id`/`distinct_id`/`user_id`/`device_id` |
|
|
661
|
+
|
|
662
|
+
property value functions receive a `StandaloneValueContext`: `{ time, config, dimensions,
|
|
663
|
+
tickIndex, tickCount, cadence, event }`. `tickIndex / (tickCount - 1)` is window progress —
|
|
664
|
+
use it to shape growth, a dip, or a spike, guarding `tickCount <= 1` before division.
|
|
665
|
+
|
|
666
|
+
the stream lands in `result.standaloneEventData`, writes to its own `-STANDALONE` file
|
|
667
|
+
shard, and imports to Mixpanel as its own event stream. hooks fire with type
|
|
668
|
+
`"standalone"`; `meta.spec` carries the resolved config so a hook can tell streams apart.
|
|
669
|
+
the hook runs before the user loop. return the record object or an array of records;
|
|
670
|
+
returning `undefined` drops the record. it has no person metadata and never enters
|
|
671
|
+
`everything`. warehouse hooks have a different contract: mutate the row in place;
|
|
672
|
+
their return values are ignored.
|
|
673
|
+
|
|
674
|
+
```javascript
|
|
675
|
+
hook: (record, type, meta) => {
|
|
676
|
+
if (type === 'standalone' && meta.spec.event === 'cdn_egress' && record.region === 'us-east') {
|
|
677
|
+
record.p95_ms *= 40;
|
|
678
|
+
}
|
|
679
|
+
return record;
|
|
680
|
+
}
|
|
681
|
+
```
|
|
682
|
+
|
|
683
|
+
validation throws rather than skipping. a malformed entry would silently drop a whole data
|
|
684
|
+
stream, and you would not notice until the charts were wrong.
|
|
685
|
+
|
|
686
|
+
## warehouse metrics (local source tables)
|
|
687
|
+
|
|
688
|
+
`warehouseMetrics` materializes warehouse-ready tables from the run's own event
|
|
689
|
+
stream after user generation completes. use it when you need a bookings table, a
|
|
690
|
+
subscription level snapshot, or an ARR table that reads like a real warehouse
|
|
691
|
+
source. these rows land in `result.warehouseMetricData`, write to
|
|
692
|
+
`<name>-WAREHOUSE-<table>.csv|json`, and emit one manifest at
|
|
693
|
+
`<name>-WAREHOUSE-MANIFEST.json`.
|
|
694
|
+
|
|
695
|
+
they are **not** imported by `token`. that is deliberate. the live path is:
|
|
696
|
+
|
|
697
|
+
1. run the dungeon
|
|
698
|
+
2. review `/warehouse-metrics` in dry-run mode
|
|
699
|
+
3. obtain explicit operator consent for live execution
|
|
700
|
+
4. load the tables to bigquery and save the metrics there
|
|
701
|
+
|
|
702
|
+
live deploy uses `bq load --replace`, so it overwrites the destination warehouse
|
|
703
|
+
tables. the shipped script does not prompt on its own, so the operator or agent
|
|
704
|
+
must obtain explicit consent before running it in live mode. if the
|
|
705
|
+
warehouse CRUD docs route returns 404, the deploy still loads tables and connects
|
|
706
|
+
the source, then writes `warehouse/GAPS.md` for manual metric creation.
|
|
707
|
+
|
|
708
|
+
the manifest carries `recommendedAggregation: 'sum' | 'last value'`. the
|
|
709
|
+
Mixpanel warehouse metric API spells that second value as `last_value`; the
|
|
710
|
+
deploy flow maps it for you.
|
|
711
|
+
|
|
712
|
+
there is one real preview trap: `previewWarehouseMetric` rejects raw SQL
|
|
713
|
+
containing `DROP`, `DELETE`, `TRUNCATE`, `ALTER`, `CREATE`, `INSERT`, or
|
|
714
|
+
`UPDATE` as plain substrings. `created_at` trips `CREATE`; `updated_at` trips
|
|
715
|
+
`UPDATE`. aliasing only helps if the blocked text disappears from the query
|
|
716
|
+
entirely.
|
|
717
|
+
|
|
718
|
+
### canonical shapes
|
|
719
|
+
|
|
720
|
+
additive daily bookings:
|
|
721
|
+
|
|
722
|
+
```javascript
|
|
723
|
+
warehouseMetrics: [{
|
|
724
|
+
name: 'daily_new_bookings',
|
|
725
|
+
source: {
|
|
726
|
+
event: 'new_booking',
|
|
727
|
+
measure: 'sum',
|
|
728
|
+
property: 'booking_value',
|
|
729
|
+
},
|
|
730
|
+
timeColumn: 'date',
|
|
731
|
+
valueColumn: 'bookings',
|
|
732
|
+
}]
|
|
733
|
+
```
|
|
734
|
+
|
|
735
|
+
point-in-time daily active subscriptions:
|
|
736
|
+
|
|
737
|
+
```javascript
|
|
738
|
+
warehouseMetrics: [{
|
|
739
|
+
name: 'daily_active_subscriptions',
|
|
740
|
+
type: 'point-in-time',
|
|
741
|
+
source: {
|
|
742
|
+
event: 'subscription_started',
|
|
743
|
+
minus: 'subscription_cancelled',
|
|
744
|
+
measure: 'count',
|
|
745
|
+
},
|
|
746
|
+
baseline: 40,
|
|
747
|
+
timeColumn: 'date',
|
|
748
|
+
valueColumn: 'active_subscriptions',
|
|
749
|
+
}]
|
|
750
|
+
```
|
|
751
|
+
|
|
752
|
+
sparse monthly ARR with backfill:
|
|
753
|
+
|
|
754
|
+
```javascript
|
|
755
|
+
warehouseMetrics: [{
|
|
756
|
+
name: 'monthly_arr_snapshot',
|
|
757
|
+
type: 'point-in-time',
|
|
758
|
+
grain: 'month',
|
|
759
|
+
sparse: true,
|
|
760
|
+
history: 18,
|
|
761
|
+
source: {
|
|
762
|
+
event: 'subscription_started',
|
|
763
|
+
minus: 'subscription_cancelled',
|
|
764
|
+
measure: 'sum',
|
|
765
|
+
property: 'monthly_value',
|
|
766
|
+
},
|
|
767
|
+
baseline: 24000,
|
|
768
|
+
scale: 12,
|
|
769
|
+
timeColumn: 'month',
|
|
770
|
+
valueColumn: 'arr_usd',
|
|
771
|
+
}]
|
|
772
|
+
```
|
|
773
|
+
|
|
774
|
+
the shipped technical fixture uses a 60-day live window plus 18 monthly backfill
|
|
775
|
+
buckets. sample row counts are illustrative only. `grain`, `history`, `sparse`,
|
|
776
|
+
and `groupBy` all change how many rows a table emits.
|
|
777
|
+
|
|
778
|
+
### config surface
|
|
779
|
+
|
|
780
|
+
| key | default | range / contract |
|
|
781
|
+
|---|---|---|
|
|
782
|
+
| `name` | required | unique table / metric name, `/^[a-z][a-z0-9_]{0,63}$/` |
|
|
783
|
+
| `type` | `'additive'` | `'additive'` or `'point-in-time'` |
|
|
784
|
+
| `grain` | `'day'` | `'day'`, `'week'`, `'month'` |
|
|
785
|
+
| `sparse` | `false` | boolean, valid only with `type: 'point-in-time'` |
|
|
786
|
+
| `source.event` | required | string or string[] of declared source events |
|
|
787
|
+
| `source.minus` | `[]` | string or string[] of declared subtractive events |
|
|
788
|
+
| `source.measure` | `'count'` | `'count'`, `'sum'`, `'avg'`, `'dau'`, `'users'`; point-in-time forbids `'avg'` and `'dau'` |
|
|
789
|
+
| `source.property` | `null` | required for `'sum'` and `'avg'`; must be declared on every source event or in `superProps` |
|
|
790
|
+
| `source.where` | `null` | optional function over flat event rows |
|
|
791
|
+
| `source.groupBy` | `[]` | up to 2 keys, each declared on every source event or in `superProps`; observed cardinality above 50 warns |
|
|
792
|
+
| `timeColumn` | `'date'` | valid JS identifier; becomes the ordered time axis in rows and manifest |
|
|
793
|
+
| `valueColumn` | `'value'` | valid JS identifier |
|
|
794
|
+
| `baseline` | `0` | number `>= 0`; used only for point-in-time metrics, ignored on additive |
|
|
795
|
+
| `scale` | `1` | finite number `> 0`, applied after bucket aggregation |
|
|
796
|
+
| `noise` | `0` | finite number, clamped to `[0, 0.5]` with a warning |
|
|
797
|
+
| `history` | `0` | integer `>= 0`; warns above roughly 3 years at each grain (`1095` day, `156` week, `36` month) |
|
|
798
|
+
| `columns` | `{}` | extra declared output columns; keys must be valid identifiers and cannot collide with time/value/groupBy columns |
|
|
799
|
+
| `format` | dungeon `format`, else `'csv'` | `'csv'` or `'json'` |
|
|
800
|
+
|
|
801
|
+
materialized tables are deterministic at the same seed and do not perturb the
|
|
802
|
+
event stream. the warehouse pass runs after the user loop, so seeded noise and
|
|
803
|
+
derived columns never change generated events.
|
|
804
|
+
|
|
805
|
+
### result and manifest
|
|
806
|
+
|
|
807
|
+
```javascript
|
|
808
|
+
const result = await DUNGEON_MASTER(config);
|
|
809
|
+
|
|
810
|
+
result.warehouseMetricData.daily_new_bookings
|
|
811
|
+
result.warehouseManifest.tables
|
|
812
|
+
result.files
|
|
813
|
+
```
|
|
814
|
+
|
|
815
|
+
each manifest table includes:
|
|
816
|
+
|
|
817
|
+
| field | meaning |
|
|
818
|
+
|---|---|
|
|
819
|
+
| `table` | warehouse table name |
|
|
820
|
+
| `file` | file prefix without extension |
|
|
821
|
+
| `format` | `'csv'` or `'json'` |
|
|
822
|
+
| `grain` | bucket grain |
|
|
823
|
+
| `type` | additive vs point-in-time |
|
|
824
|
+
| `timeColumn` | date axis column |
|
|
825
|
+
| `valueColumn` | numeric value column |
|
|
826
|
+
| `dimensionColumns` | copied `groupBy` keys |
|
|
827
|
+
| `columns` | ordered BigQuery schema (`DATE`, `FLOAT64`, `BOOL`, `STRING`) |
|
|
828
|
+
| `recommendedAggregation` | `'sum'` or `'last value'` |
|
|
829
|
+
| `sql` | `SELECT * FROM \`{{DATASET}}.<table>\` ORDER BY <timeColumn>` |
|
|
830
|
+
| `refreshHint` | currently `'hourly'` |
|
|
831
|
+
|
|
609
832
|
## user generation
|
|
610
833
|
|
|
611
834
|
users are generated with configurable birth distributions, normally controlled via the `macro` preset (see "time shape" above). these three knobs can also be set directly on the dungeon config — they override the preset's values.
|
|
@@ -697,8 +920,11 @@ result.userProfilesData // user profiles
|
|
|
697
920
|
result.scdTableData // SCD mutations
|
|
698
921
|
result.groupProfilesData // group profiles
|
|
699
922
|
result.adSpendData // ad spend data
|
|
923
|
+
result.standaloneEventData // identity-less event snapshots
|
|
700
924
|
result.lookupTableData // lookup table entries
|
|
701
925
|
result.mirrorEventData // mirror dataset
|
|
926
|
+
result.warehouseMetricData // warehouse tables keyed by metric name
|
|
927
|
+
result.warehouseManifest // warehouse table manifest
|
|
702
928
|
|
|
703
929
|
result.eventCount // total event count
|
|
704
930
|
result.userCount // total user count
|
|
@@ -899,6 +1125,8 @@ see [types.d.ts](types.d.ts) for the complete `Dungeon` interface. here are the
|
|
|
899
1125
|
| `hasLocation` | boolean | false | include geo properties |
|
|
900
1126
|
| `hasCampaigns` | boolean | false | include UTM properties |
|
|
901
1127
|
| `hasAdSpend` | boolean | false | generate ad spend data |
|
|
1128
|
+
| `standaloneEvents` | array | `[]` | identity-less cadence streams that import as events |
|
|
1129
|
+
| `warehouseMetrics` | array | `[]` | local warehouse source tables + manifest, derived from generated events |
|
|
902
1130
|
| `hasAnonIds` | boolean | false | generate anonymous IDs |
|
|
903
1131
|
| `hasSessionIds` | boolean | false | generate session IDs |
|
|
904
1132
|
| `alsoInferFunnels` | boolean | false | auto-generate funnels from events |
|