@ak--47/dungeon-master 1.6.3 → 1.6.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/headless-build/SKILL.md +71 -1
- package/CHANGELOG.md +127 -0
- package/HOOKS.md +68 -5
- package/README.md +59 -2
- package/index.js +7 -2
- package/lib/core/config-validator.js +63 -2
- package/lib/generators/funnels.js +4 -1
- package/lib/hook-helpers/mutate.js +40 -6
- package/lib/orchestrators/mixpanel-sender.js +1 -0
- package/lib/templates/abbreviated.d.ts +7 -25
- package/lib/templates/schema.d.ts +10 -31
- package/lib/utils/utils.js +0 -1
- package/package.json +1 -1
- package/types.d.ts +74 -25
|
@@ -81,6 +81,7 @@ dungeons/user/<name>/build/
|
|
|
81
81
|
│ ├── 06..09_dash_<story>.py # one board per engineered story
|
|
82
82
|
│ ├── 10_annotations.py
|
|
83
83
|
│ ├── 11_behaviors_metrics.py # via power-tools; headless lacks these
|
|
84
|
+
│ ├── 13_render_check.py # saved params must be RENDERABLE, not just queryable
|
|
84
85
|
│ └── 99_verify.py
|
|
85
86
|
└── results/ # entities.json registry, window.json, verification_*.json
|
|
86
87
|
```
|
|
@@ -103,11 +104,75 @@ actually works. Verified on both NYC DCP and Peloton:
|
|
|
103
104
|
- Insights `group_by` / `where` / saved-cohort filters: **work**.
|
|
104
105
|
- Funnels `group_by`: **silently does not segment** — returns rows identical to
|
|
105
106
|
ungrouped. Use one funnel per segment with a `where` filter instead.
|
|
106
|
-
- Day-granularity queries reject ranges **over 366 days**.
|
|
107
|
+
- Day-granularity queries reject ranges **over 366 days**. `unit="hour"` works
|
|
108
|
+
fine, including with `group_by` — that is how you read hour-of-day, since the
|
|
109
|
+
App API rejects `hour(A)` in a custom-property formula as an unknown function.
|
|
107
110
|
- Valid `displayOptions.chartType`: `bar`, `column`, `frequency-curve`,
|
|
108
111
|
`funnel-steps`, `funnel-top-paths`, `insights-metric`, `line`, `pie`,
|
|
109
112
|
`retention-curve`, `table`. There is no `stacked-area` — the API rejects it.
|
|
110
113
|
|
|
114
|
+
**Filters and breakdowns fail SILENTLY, returning 0 rather than erroring.** Probe
|
|
115
|
+
every one before a board depends on it. Verified on Square:
|
|
116
|
+
|
|
117
|
+
| Intent | Wrong (returns 0 / "undefined") | Right |
|
|
118
|
+
|---|---|---|
|
|
119
|
+
| Profile property | `Filter.equals("plan", "Pro")` | `Filter.equals("plan", "Pro", resource_type="people")` |
|
|
120
|
+
| Profile property breakdown | `group_by="plan"` | not supported — one query per segment with a user-scoped `where` |
|
|
121
|
+
| Numeric event property | `Filter.equals("depth", 3)` | `Filter.equals("depth", "3")` — as a **string** |
|
|
122
|
+
| Boolean event property | `Filter.equals("flag", True)` | `Filter.is_true("flag")` / `Filter.is_false(...)` |
|
|
123
|
+
| Custom-property breakdown | `group_by="Ticket Band"` | `GroupBy(CustomPropertyRef(<id>))` |
|
|
124
|
+
| Sum of a property | `math="sum"` (raises) | `math="total"` **with** `math_property` |
|
|
125
|
+
| Flows, split by property | `query_flow(where=[Filter.equals(...)])` (raises `Invalid filter type: resourceType`) | cohort filters only — `query_flow(where=[Filter.in_cohort(...)])` |
|
|
126
|
+
|
|
127
|
+
**Querying successfully is NOT evidence that a report renders.** The single most
|
|
128
|
+
expensive bug in the Square build: `Filter.equals(..., resource_type="user")`
|
|
129
|
+
instead of `"people"`. The SDK's type is `Literal["events", "people"]`, but the
|
|
130
|
+
value is not validated, the query endpoint accepts it, and it returns byte-identical
|
|
131
|
+
numbers — so the build printed correct figures, `99_verify.py` re-measured the story
|
|
132
|
+
live and called it a MATCH, `query_saved_report` executed the saved bookmark without
|
|
133
|
+
complaint, and `get_dashboard_erf` reported `is_valid_for_erf: true`. Six dashboard
|
|
134
|
+
cards nonetheless rendered **"The client has issued a malformed request."** The
|
|
135
|
+
defect only existed in what was *persisted*: `sections.filter[0].resourceType`.
|
|
136
|
+
|
|
137
|
+
Two rules follow:
|
|
138
|
+
|
|
139
|
+
- **Never invent an enum value.** If the SDK declares a `Literal`, use one of its
|
|
140
|
+
members even when another string is accepted and works.
|
|
141
|
+
- **Ship a render check.** `scripts/13_render_check.py` in the Square build walks
|
|
142
|
+
every saved bookmark on every registered board and fails on any `resourceType`
|
|
143
|
+
outside `{events, people, user_profiles, cohort}`. It runs as the first step of
|
|
144
|
+
the `verify` phase, before `99_verify`. Inspect persisted params, not query
|
|
145
|
+
results — every query-based check passed while the boards were visibly broken.
|
|
146
|
+
|
|
147
|
+
**Build the new board BEFORE deleting the old one.** The obvious `replace_dashboard`
|
|
148
|
+
(delete by title, then create) means any interruption — and a rate-limit wall
|
|
149
|
+
partway through a board's queries is routine — leaves the project with no board at
|
|
150
|
+
all. Capture the old ids first, create the replacement, then retire the old ones
|
|
151
|
+
(`existing_dashboard_ids` → `publish(..., supersedes=...)` in the Square `_common.py`).
|
|
152
|
+
|
|
153
|
+
**Budget the per-hour query cap across the whole session, not per run.** Probing,
|
|
154
|
+
building, and verifying all draw on the same hourly allowance. Interactive probes
|
|
155
|
+
early in a session can exhaust it and strand a rebuild hours later. Boards with
|
|
156
|
+
many segment-per-query funnels are the expensive ones — the Square activation board
|
|
157
|
+
alone fires ~30.
|
|
158
|
+
|
|
159
|
+
**Retention: cohort maturity will eat your effect.** `query_retention` pools every
|
|
160
|
+
cohort in the range, including ones a fortnight old that cannot yet have failed to
|
|
161
|
+
return in week 4, and there is no server-side way to restrict the cohort window.
|
|
162
|
+
On Square the same attach-retention gap read **1.71x pooled and 2.98x** over cohorts
|
|
163
|
+
with a fully observed horizon. Aggregate the frame yourself: weight buckets by cohort
|
|
164
|
+
size, require each bucket to be observed END TO END (`(b+1)*unit - 1` days of history,
|
|
165
|
+
not `b*unit`), and cap the cohort date. Put both numbers on the board — the diluted
|
|
166
|
+
one is what the chart shows, and naming why is a better demo than hiding it.
|
|
167
|
+
|
|
168
|
+
**Time-to-convert is a MEAN, and means are tail-dominated.** Funnel frames carry
|
|
169
|
+
`avg_time` and `avg_time_from_start` (seconds; not monotonic across steps — each is
|
|
170
|
+
over that step's own survivors). A dungeon knob expressed as a median ratio will not
|
|
171
|
+
reproduce: Square's designed 13x median gap measured 1.95x as a mean over a 30-day
|
|
172
|
+
window. Use the **speed curve** instead — run the same funnel at 1/3/7/14-day
|
|
173
|
+
conversion windows and read what share of each segment's eventual conversions had
|
|
174
|
+
landed by then. Same effect, expressed in a statistic Mixpanel actually computes.
|
|
175
|
+
|
|
111
176
|
### 3. Auto-detect the data window
|
|
112
177
|
|
|
113
178
|
Never hardcode dates. A regenerated dungeon lands on new ones and every chart
|
|
@@ -196,6 +261,11 @@ is accepted and applied, but every definition collapses to the same membership
|
|
|
196
261
|
three different cohorts return byte-identical numbers instead of erroring. Create
|
|
197
262
|
cohorts first, then filter by id: `Filter.in_cohort(<saved_id>, "<name>")`.
|
|
198
263
|
|
|
264
|
+
**`getCohorts` can report `count: 0` for every cohort** even when they hold thousands
|
|
265
|
+
of members and filter queries correctly. Do not use the listing's count as the
|
|
266
|
+
membership check in `99_verify.py` — run one cheap query per cohort instead. This is
|
|
267
|
+
the same call that warms them, so it costs nothing extra.
|
|
268
|
+
|
|
199
269
|
**`create_cohort` via headless 500s** on some projects; `CreateCohortParams.definition`
|
|
200
270
|
also wants `.to_dict()`, not the builder object. Use `/crud/createCohort` — see the
|
|
201
271
|
`powertools` skill for the payload shape and its limits (behavioral counts yes;
|
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,133 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to `@ak--47/dungeon-master`.
|
|
4
4
|
|
|
5
|
+
## 1.6.5 — 2026-09-02
|
|
6
|
+
|
|
7
|
+
### Changed
|
|
8
|
+
|
|
9
|
+
- **`matchMixpanelDefaults: true` on every live send to Mixpanel.** Set in
|
|
10
|
+
`commonOpts` in `lib/orchestrators/mixpanel-sender.js`, passed straight through
|
|
11
|
+
to `mixpanel-import`. Renames warehouse-style property keys to Mixpanel's
|
|
12
|
+
reserved names (e.g. `current_url` → `$current_url`, `_browser` → `$browser`)
|
|
13
|
+
as records stream out. Only affects the wire payload sent to Mixpanel's
|
|
14
|
+
ingestion API — generated events, profiles, and files on disk are unchanged.
|
|
15
|
+
Same seed, same config still produces byte-identical generated output.
|
|
16
|
+
|
|
17
|
+
## 1.6.4 — 2026-09-01
|
|
18
|
+
|
|
19
|
+
Answers the doc/type half of the DM4 v5 engine request
|
|
20
|
+
(`dungeon-master-library-changes-requested.md`, 2026-09-01). Every claim in that
|
|
21
|
+
document was checked against source; all were accurate except the three noted
|
|
22
|
+
under "Already correct" below.
|
|
23
|
+
|
|
24
|
+
This release is deliberately scoped to changes that cannot alter generated data:
|
|
25
|
+
docs, types, one additive helper option, one additive config form, and one
|
|
26
|
+
warning. The feature requests are tracked for 1.7.0 — see "Deferred to 1.7.0".
|
|
27
|
+
|
|
28
|
+
No generated output changes. Verified: same seed, same config, `concurrency: 1`,
|
|
29
|
+
pinned dataset window — 1.6.4 and 1.6.3 produce byte-identical events, user
|
|
30
|
+
profiles, and group profiles.
|
|
31
|
+
|
|
32
|
+
### Added
|
|
33
|
+
|
|
34
|
+
- **`scaleEventCount(events, name, factor, { spreadDays })`.** Scatters clones
|
|
35
|
+
uniformly across the next N days instead of stepping 1 second at a time. The
|
|
36
|
+
default 1-second spread cannot create a new distinct active day and cannot open
|
|
37
|
+
a new session under Mixpanel's 30-minute gap rule, so a default call moves event
|
|
38
|
+
volume and nothing else — not active days, DAU, stickiness, sessions, frequency
|
|
39
|
+
bins, or retention. 33 dungeons in the DM4 corpus used the default and then
|
|
40
|
+
documented an active-day, session, frequency, or stickiness claim. All 33 were
|
|
41
|
+
wrong. The limitation is now stated in the helper's JSDoc, in HOOKS.md §2.1, and
|
|
42
|
+
in HOOKS.md gotcha #22. Clones that land past `FIXED_NOW` are still dropped by
|
|
43
|
+
the future-time guard. Default behavior is unchanged.
|
|
44
|
+
- **Named-object form for `groupKeys`.** `{ key, cardinality, events? }` alongside
|
|
45
|
+
the positional tuple `[key, cardinality]` / `[key, cardinality, events]`. Both
|
|
46
|
+
forms may be mixed in one array. The validator normalizes to tuples, so hooks,
|
|
47
|
+
the generators, and the verifier still see exactly one shape. Added so a
|
|
48
|
+
form-driven config generator never has to emit an untyped positional tuple.
|
|
49
|
+
Bad input throws with the offending index.
|
|
50
|
+
- **HOOKS.md §2.1.1 — "Cloned events MUST carry a fresh `insert_id`."** The
|
|
51
|
+
consequence was only in a source comment before. Four DM4 corpus files
|
|
52
|
+
hand-rolled `JSON.parse(JSON.stringify(e))` and had their entire engineered
|
|
53
|
+
surge deduped away by Mixpanel at ingest, with local verification still
|
|
54
|
+
reporting the surge as present. Includes the note that data generated this way
|
|
55
|
+
on 1.6.2 or earlier is wrong in-project and must be regenerated.
|
|
56
|
+
- **README "one config surface, not two."** States that the `credentials` /
|
|
57
|
+
`switches` / `identity` sub-objects are the canonical form, that the flat
|
|
58
|
+
top-level keys are a back-compat alias, and that top-level wins when both are
|
|
59
|
+
set. Mirrored in CLAUDE.md and in the three sub-object JSDoc blocks.
|
|
60
|
+
|
|
61
|
+
### Changed
|
|
62
|
+
|
|
63
|
+
- **The `avgActiveDaysPerUser` + `engagementDecay` warning is no longer gated
|
|
64
|
+
behind `verbose`.** The combination silently returns an active-day count at or
|
|
65
|
+
below the configured value, never above. A config UI that shows the requested
|
|
66
|
+
number was lying about it. Documented in HOOKS.md §2.5 and in the
|
|
67
|
+
`avgActiveDaysPerUser` JSDoc.
|
|
68
|
+
- **`retentionCurve` precedence is now documented.** When both `retentionCurve`
|
|
69
|
+
and `avgActiveDaysPerUser` are set, the curve wins and `avgActiveDaysPerUser` is
|
|
70
|
+
ignored entirely. Behavior is unchanged — only the docs were missing. Added to
|
|
71
|
+
the CLAUDE.md safe-range table, the README config table, HOOKS.md §2.5, and the
|
|
72
|
+
`avgActiveDaysPerUser` JSDoc.
|
|
73
|
+
|
|
74
|
+
### Fixed (types and docs)
|
|
75
|
+
|
|
76
|
+
- **`writeToDisk` doc was wrong.** It read "If true (default), writes output files
|
|
77
|
+
to ./data/". The runtime default is `false` (`config-validator.js`).
|
|
78
|
+
- **`region` types disagreed.** Top-level `region` was `"US" | "EU"` while
|
|
79
|
+
`DungeonCredentials.region` allowed `"IN"`. Both now use a shared `Region` type
|
|
80
|
+
of `'US' | 'EU' | 'IN'`, which matches what `mixpanel-import` accepts.
|
|
81
|
+
- **`hasAttributionFlags` was presented as a settable switch.** The validator
|
|
82
|
+
unconditionally overwrites it with `events.some(e => e.isAttributionEvent)`. It
|
|
83
|
+
is removed from `DungeonSwitches` and from the `switches` hoisting allowlist,
|
|
84
|
+
and marked `@internal` on `Dungeon`. Read it off `result.validatedConfig`;
|
|
85
|
+
setting it never did anything.
|
|
86
|
+
- **`GroupEventConfig` / `config.groupEvents` removed from the public types.** A
|
|
87
|
+
declared-only stub — nothing in `lib/` ever read it, its own doc comment said
|
|
88
|
+
"not yet implemented", and no shipped dungeon set it. Removed from `types.d.ts`,
|
|
89
|
+
`lib/templates/abbreviated.d.ts`, `lib/templates/schema.d.ts`, and the `wrapFunc`
|
|
90
|
+
whitelist. To scope an event to a group, list it in that group key's `events`
|
|
91
|
+
array.
|
|
92
|
+
|
|
93
|
+
- **The determinism claim was overstated.** README and CLAUDE.md both said "same
|
|
94
|
+
seed + same config + `concurrency: 1` = byte-identical output". That has been
|
|
95
|
+
false since 1.4.0: `insert_id` is a `randomUUID()`, so it differs on every run
|
|
96
|
+
by design — which is what keeps Mixpanel from deduping a re-import of the same
|
|
97
|
+
dataset. Found while verifying that this release changes no output. Everything
|
|
98
|
+
else is byte-identical; strip `insert_id` before diffing two runs. Both docs now
|
|
99
|
+
say so.
|
|
100
|
+
|
|
101
|
+
### Already correct (no change needed)
|
|
102
|
+
|
|
103
|
+
- `funnels[].reentry` and `funnels[].stepFilters` were reported as reading like
|
|
104
|
+
generation features. Their JSDoc already says "Verifier-only hint … Generator
|
|
105
|
+
behavior unchanged."
|
|
106
|
+
- The dead persona fields (`churnRate`, `activeWindow`, `soupOverride`) were
|
|
107
|
+
reported as `verbose`-gated. Their warning already fires unconditionally, once
|
|
108
|
+
per process. They stay accepted and warned; removing them from the `Persona`
|
|
109
|
+
type is a 1.7.0 change because it is a type-level break.
|
|
110
|
+
|
|
111
|
+
### Deferred to 1.7.0
|
|
112
|
+
|
|
113
|
+
Everything below is new public surface or a behavior change, so none of it belongs
|
|
114
|
+
in a patch. Per-item design lives in the maintainer's local `plans/1.7.0/SPEC.md`
|
|
115
|
+
(the `plans/` tree is not published).
|
|
116
|
+
|
|
117
|
+
| Request | Why not in 1.6.4 |
|
|
118
|
+
|---|---|
|
|
119
|
+
| `funnels[].conditions` operators (`in`, `gte`, `neq`, …) | New public surface. Also throws on function/array condition values, which is a break. |
|
|
120
|
+
| Experiment variant stamped on the user profile | New profile property; needs a change to when variants resolve. |
|
|
121
|
+
| `(ctx) => value` for property value functions | New signature. Requires an arity guard on the `choose` source-string cache first, or context-aware functions get frozen at their first evaluation. |
|
|
122
|
+
| `stickyEventProps` | New surface. Also needs `lib/verify/schema-validator.js` taught about it, or `/verify-dungeon` reports every sticky prop as flag stamping. |
|
|
123
|
+
| Stable per-user `location` under `hasLocation` | A real fix (`featureCtx.userLocation` is computed and never read), but it changes generated event geo. |
|
|
124
|
+
| `personas[].ttcModifier` | New surface. |
|
|
125
|
+
| Removing the three dead persona fields from the `Persona` type | Type-level break. |
|
|
126
|
+
| `campaignPerUser` | New surface. |
|
|
127
|
+
| `autoPowerLaw: false` and `{ __weights }` | New surface. |
|
|
128
|
+
| `result.warnings[]` for clamps | New result surface. |
|
|
129
|
+
| Ad spend derived from users acquired per campaign | Needs a new cross-user aggregate pass. Deferred past 1.7.0. |
|
|
130
|
+
| Engine-side `conversionRate` saturation reporting | Depends on `result.warnings[]`. Note: the engine can report its own `Math.min(100, …)` clamps, but not a hook's own `Math.min(95, rate * 3)` — that cap belongs to the hook and the engine never sees the intended value. |
|
|
131
|
+
|
|
5
132
|
## 1.6.3 — 2026-08-17
|
|
6
133
|
|
|
7
134
|
### Fixed
|
package/HOOKS.md
CHANGED
|
@@ -88,6 +88,58 @@ users between frequency bins. Both `injectOnNewDays` and the default
|
|
|
88
88
|
`countDistinctPeriods` algorithm use calendar-bucket math, so they agree
|
|
89
89
|
at boundaries.
|
|
90
90
|
|
|
91
|
+
One second is also below Mixpanel's 30-minute session gap, so the default
|
|
92
|
+
spread cannot create a new session either. The full list of metrics a default
|
|
93
|
+
`scaleEventCount` call **cannot** move: active days per user, DAU, stickiness
|
|
94
|
+
(DAU÷MAU), sessions per user, frequency bins, and retention. It moves event
|
|
95
|
+
volume, and nothing else.
|
|
96
|
+
|
|
97
|
+
As of v1.6.4 you can pass `{ spreadDays: N }` to scatter the clones across the
|
|
98
|
+
next N days instead:
|
|
99
|
+
|
|
100
|
+
```js
|
|
101
|
+
// volume only — same day, same session
|
|
102
|
+
scaleEventCount(record, "commit pushed", 3);
|
|
103
|
+
|
|
104
|
+
// volume AND active days AND sessions
|
|
105
|
+
scaleEventCount(record, "commit pushed", 3, { spreadDays: 7 });
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
`injectOnNewDays` remains the better tool when you want a specific target
|
|
109
|
+
day count rather than a multiplier.
|
|
110
|
+
|
|
111
|
+
### 2.1.1 Cloned events MUST carry a fresh `insert_id`
|
|
112
|
+
|
|
113
|
+
Mixpanel deduplicates on `insert_id` at ingest. A clone that keeps its source's
|
|
114
|
+
`insert_id` is silently discarded — the surge you engineered never appears in the
|
|
115
|
+
project, no error is raised, and local verification does not catch it because
|
|
116
|
+
`emulateBreakdown` never inspects `insert_id`.
|
|
117
|
+
|
|
118
|
+
Leaving `insert_id` blank is not safe either. The importer content-hashes events
|
|
119
|
+
that lack one, so identical clones hash to the same value and collide the same way.
|
|
120
|
+
|
|
121
|
+
```js
|
|
122
|
+
// PREFERRED — cloneEvent stamps a fresh insert_id
|
|
123
|
+
record.push(cloneEvent(sourceEvent, { time: newTime }));
|
|
124
|
+
|
|
125
|
+
// ALSO FINE — the engine re-stamps the duplicate id
|
|
126
|
+
record.push({ ...sourceEvent, time: newTime });
|
|
127
|
+
|
|
128
|
+
// HISTORICALLY BROKEN — pre-1.6.3 this deduped the whole surge away
|
|
129
|
+
const clone = JSON.parse(JSON.stringify(sourceEvent));
|
|
130
|
+
clone.time = newTime;
|
|
131
|
+
record.push(clone);
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Since v1.6.3 the engine re-stamps any duplicate or missing `insert_id` across each
|
|
135
|
+
user's final stream ([user-loop.js:753-776](lib/orchestrators/user-loop.js#L753-L776)),
|
|
136
|
+
so all three shapes now survive ingest. Prefer
|
|
137
|
+
[`cloneEvent`](lib/hook-helpers/mutate.js) anyway: the engine pass is a per-user
|
|
138
|
+
last resort, and it cannot help a clone that a hook moves onto a different user.
|
|
139
|
+
|
|
140
|
+
If you generated data with a hand-rolled deep-copy clone on **1.6.2 or earlier**,
|
|
141
|
+
that data is wrong in the project. Regenerate and re-import.
|
|
142
|
+
|
|
91
143
|
### 2.2 Funnels are GREEDY single-pass with a 2-second grace
|
|
92
144
|
|
|
93
145
|
Mixpanel processes events in chronological order, single pass. Each event is
|
|
@@ -217,6 +269,15 @@ count BELOW the configured target. Pick one. If you need both effects, set
|
|
|
217
269
|
`avgActiveDaysPerUser` and write decay logic in an `everything` hook scoped
|
|
218
270
|
to specific cohorts (gives explicit control over the interaction).
|
|
219
271
|
|
|
272
|
+
As of v1.6.4 the validator warns **unconditionally** when both are set — the
|
|
273
|
+
warning is not gated behind `verbose`, because the combination silently returns
|
|
274
|
+
an active-day count the author did not ask for.
|
|
275
|
+
|
|
276
|
+
**Precedence with `retentionCurve`:** when `retentionCurve` is set, it wins.
|
|
277
|
+
The active-day planner runs from the curve and `avgActiveDaysPerUser` is ignored
|
|
278
|
+
([user-loop.js:326-337](lib/orchestrators/user-loop.js#L326-L337)). Set one or
|
|
279
|
+
the other, not both.
|
|
280
|
+
|
|
220
281
|
### 2.6 Sessions are query-time computed (30-min gap, 24h max, day-boundary split)
|
|
221
282
|
|
|
222
283
|
Reference: `backend/arb/reader/queries/session_query.cpp:828-830, 905-928`.
|
|
@@ -666,10 +727,12 @@ Reference: `flows_query.cpp:988-994` (next-anchor-only), `flows.cpp:680-717`
|
|
|
666
727
|
still correct.
|
|
667
728
|
|
|
668
729
|
22. **`scaleEventCount` does not move users between frequency bins.** Cloning
|
|
669
|
-
Buy events at sub-second offsets places them on the same calendar day
|
|
670
|
-
the
|
|
730
|
+
Buy events at sub-second offsets places them on the same calendar day and
|
|
731
|
+
inside the same 30-minute session window, so distinct days, sessions, DAU,
|
|
732
|
+
stickiness, and retention are all unchanged. It moves event volume only. To
|
|
733
|
+
shift any of the others, pass `{ spreadDays: N }` (v1.6.4) or use
|
|
671
734
|
[`injectOnNewDays`](lib/hook-helpers/inject.js), which spreads injections
|
|
672
|
-
across previously empty days within the user's active window.
|
|
735
|
+
across previously empty days within the user's active window. See §2.1.
|
|
673
736
|
|
|
674
737
|
23. **Out-of-order injected events get consumed by the funnel engine.** Adding
|
|
675
738
|
a "step C" event before "step B" in the stream causes Mixpanel's greedy
|
|
@@ -1705,9 +1768,9 @@ Import from `@ak--47/dungeon-master/hook-helpers`:
|
|
|
1705
1768
|
| `userInProfileSegment` | cohort | `(profile, key, values) -> boolean` | Profile property match |
|
|
1706
1769
|
| **`hashFloat`** | cohort | `(id) -> number` | FNV-1a over the FULL id string → [0,1). Deterministic bucketing primitive (v1.6) — replaces `charCodeAt(0) % N` idioms, which bias cohort rates on hex-ish id alphabets |
|
|
1707
1770
|
| **`hashCohort`** | cohort | `(id, pct) -> boolean` | True for ~`pct`% of ids (pct on a 0–100 scale). Membership nests: `pct=5` ⊂ `pct=20` |
|
|
1708
|
-
| `cloneEvent` | mutate | `(template, overrides?) -> event` | Shallow clone with overrides |
|
|
1771
|
+
| `cloneEvent` | mutate | `(template, overrides?) -> event` | Shallow clone with overrides **and a fresh `insert_id`** — never hand-roll this (see Section 2.1) |
|
|
1709
1772
|
| `dropEventsWhere` | mutate | `(events, predicate) -> number` | Remove matching events in-place |
|
|
1710
|
-
| `scaleEventCount` | mutate | `(events, eventName, factor) -> number` | Scale total count via clones
|
|
1773
|
+
| `scaleEventCount` | mutate | `(events, eventName, factor, options?) -> number` | Scale total count via clones. Default 1s offsets do NOT move frequency, session, active-day, or retention metrics — see Section 2.1. Pass `{ spreadDays: N }` (v1.6.4) or use `injectOnNewDays` |
|
|
1711
1774
|
| `scalePropertyValue` | mutate | `(events, predicate, prop, factor) -> number` | Multiply numeric property; null-aware safe |
|
|
1712
1775
|
| `shiftEventTime` | mutate | `(event, deltaMs) -> event` | Shift one timestamp |
|
|
1713
1776
|
| `scaleTimingBetween` | timing | `(events, eventA, eventB, factor) -> boolean` | Scale gap between first A and first B |
|
package/README.md
CHANGED
|
@@ -584,6 +584,15 @@ all randomness is seeded. same seed + same config + concurrency=1 = identical ou
|
|
|
584
584
|
}
|
|
585
585
|
```
|
|
586
586
|
|
|
587
|
+
pin `datasetStart` and `datasetEnd` too, or the dataset window moves with the
|
|
588
|
+
calendar and every timestamp shifts.
|
|
589
|
+
|
|
590
|
+
**one exception: `insert_id`.** since 1.4.0 it is a `randomUUID()`, so it differs
|
|
591
|
+
on every run by design — that is what keeps Mixpanel from deduping re-imports of
|
|
592
|
+
the same dataset. strip `insert_id` before diffing two runs. everything else
|
|
593
|
+
(event count, order, timestamps, every property, profiles, groups) is
|
|
594
|
+
byte-identical.
|
|
595
|
+
|
|
587
596
|
## what gets generated
|
|
588
597
|
|
|
589
598
|
the result object contains everything:
|
|
@@ -689,6 +698,53 @@ engine tests are NOT shipped in the npm package and NOT run as part of `npm test
|
|
|
689
698
|
|
|
690
699
|
## config reference
|
|
691
700
|
|
|
701
|
+
### one config surface, not two
|
|
702
|
+
|
|
703
|
+
three groups of keys accept both a nested sub-object and a flat top-level form:
|
|
704
|
+
|
|
705
|
+
| sub-object | keys it groups |
|
|
706
|
+
|---|---|
|
|
707
|
+
| `credentials` | `token`, `region`, `serviceAccount`, `serviceSecret`, `projectId` |
|
|
708
|
+
| `switches` | `hasLocation`, `hasCampaigns`, `hasAdSpend`, `hasSessionIds`, `hasAvatar`, `hasIOSDevices`, `hasAndroidDevices`, `hasDesktopDevices`, `hasBrowser`, `isAnonymous`, `alsoInferFunnels` |
|
|
709
|
+
| `identity` | `avgDevicePerUser`, `sessionTimeout` |
|
|
710
|
+
|
|
711
|
+
**the sub-object form is canonical.** the flat top-level keys are a back-compat
|
|
712
|
+
alias and stay supported. emit one form or the other — never both. when a key is
|
|
713
|
+
set in both places the **top-level value wins**, with a `verbose`-gated warning
|
|
714
|
+
you will not see unless `verbose: true`.
|
|
715
|
+
|
|
716
|
+
```javascript
|
|
717
|
+
// canonical
|
|
718
|
+
{ credentials: { token: process.env.MIXPANEL_TOKEN, region: 'US' },
|
|
719
|
+
switches: { hasCampaigns: true, hasAdSpend: true },
|
|
720
|
+
identity: { avgDevicePerUser: 2 } }
|
|
721
|
+
|
|
722
|
+
// back-compat alias — still works
|
|
723
|
+
{ token: process.env.MIXPANEL_TOKEN, region: 'US',
|
|
724
|
+
hasCampaigns: true, hasAdSpend: true, avgDevicePerUser: 2 }
|
|
725
|
+
```
|
|
726
|
+
|
|
727
|
+
`hasAttributionFlags` is **not** a switch. the validator derives it from
|
|
728
|
+
`events[].isAttributionEvent`; setting it has no effect.
|
|
729
|
+
|
|
730
|
+
### group keys
|
|
731
|
+
|
|
732
|
+
`groupKeys` accepts a positional tuple or a named object. both normalize to the
|
|
733
|
+
tuple internally, so hooks and the verifier see one shape:
|
|
734
|
+
|
|
735
|
+
```javascript
|
|
736
|
+
groupKeys: [
|
|
737
|
+
['company_id', 50], // tuple
|
|
738
|
+
['team_id', 200, ['Deploy', 'Merge PR']], // tuple + scoped events
|
|
739
|
+
{ key: 'org_id', cardinality: 25 }, // named (v1.6.4)
|
|
740
|
+
{ key: 'workspace_id', cardinality: 80, events: ['Save'] },
|
|
741
|
+
]
|
|
742
|
+
```
|
|
743
|
+
|
|
744
|
+
an omitted or empty `events` list means every event carries that group key.
|
|
745
|
+
|
|
746
|
+
### commonly used properties
|
|
747
|
+
|
|
692
748
|
see [types.d.ts](types.d.ts) for the complete `Dungeon` interface. here are the most commonly used properties:
|
|
693
749
|
|
|
694
750
|
| property | type | default | description |
|
|
@@ -702,7 +758,7 @@ see [types.d.ts](types.d.ts) for the complete `Dungeon` interface. here are the
|
|
|
702
758
|
| `seed` | string | random | RNG seed for reproducibility |
|
|
703
759
|
| `format` | string | `'csv'` | output format (csv, json, parquet) |
|
|
704
760
|
| `token` | string | null | mixpanel project token (triggers import) |
|
|
705
|
-
| `region` | string | `'US'` | mixpanel data residency |
|
|
761
|
+
| `region` | string | `'US'` | mixpanel data residency (`US` / `EU` / `IN`) |
|
|
706
762
|
| `writeToDisk` | boolean/string | false | write files to ./data/ or a gs:// path |
|
|
707
763
|
| `gzip` | boolean | false | compress output files |
|
|
708
764
|
| `verbose` | boolean | false | print progress |
|
|
@@ -714,7 +770,8 @@ see [types.d.ts](types.d.ts) for the complete `Dungeon` interface. here are the
|
|
|
714
770
|
| `bornRecentBias` | number | 0 (from macro `flat`) | user birth date skew (safe range [-0.5, 0.5]; user-explicit values outside the band are clamped) |
|
|
715
771
|
| `percentUsersBornInDataset` | number | 12 (from macro `flat`) | % of users born in window (clamped per-macro when both `macro` and this field are explicit) |
|
|
716
772
|
| `preExistingSpread` | string | `'uniform'` (from macro `flat`) | placement of pre-existing users' first event |
|
|
717
|
-
| `avgActiveDaysPerUser` | number | undefined | concentrate events onto N distinct UTC days per user (preserves total event count) |
|
|
773
|
+
| `avgActiveDaysPerUser` | number | undefined | concentrate events onto N distinct UTC days per user (preserves total event count). ignored when `retentionCurve` is set; warns when combined with `engagementDecay` |
|
|
774
|
+
| `retentionCurve` | object | undefined | per-day return probabilities. **wins over `avgActiveDaysPerUser`** when both are set |
|
|
718
775
|
| `maxTouchpointsPerUser` | number | 10 | UTM stamping cap per user (Mixpanel `TOUCHPOINTS_LIMIT` parity) |
|
|
719
776
|
| `autoSortAfterEverything` | boolean | true | sort events by time after `everything` hook (defends greedy funnel engine) |
|
|
720
777
|
| `hook` | function/string | passthrough | data transformation function |
|
package/index.js
CHANGED
|
@@ -347,7 +347,10 @@ async function generateAdSpendData(context) {
|
|
|
347
347
|
*/
|
|
348
348
|
async function generateGroupProfiles(context) {
|
|
349
349
|
const { config, storage } = context;
|
|
350
|
-
|
|
350
|
+
// `config` here is the VALIDATED config, so `normalizeGroupKeys` has already
|
|
351
|
+
// converted any `{ key, cardinality }` entries to positional tuples.
|
|
352
|
+
const groupKeys = /** @type {import('./types').GroupKeyTuple[]} */ (config.groupKeys);
|
|
353
|
+
const { groupProps = {} } = config;
|
|
351
354
|
|
|
352
355
|
if (config.verbose) {
|
|
353
356
|
logger.info('Generating group profiles...');
|
|
@@ -430,7 +433,9 @@ async function generateLookupTables(context) {
|
|
|
430
433
|
*/
|
|
431
434
|
async function generateGroupSCDs(context) {
|
|
432
435
|
const { config, storage } = context;
|
|
433
|
-
const { scdProps
|
|
436
|
+
const { scdProps } = config;
|
|
437
|
+
// Validated config: `normalizeGroupKeys` has already flattened named entries.
|
|
438
|
+
const groupKeys = /** @type {import('./types').GroupKeyTuple[]} */ (config.groupKeys);
|
|
434
439
|
|
|
435
440
|
if (config.verbose) {
|
|
436
441
|
logger.info('Generating group SCDs...');
|
|
@@ -181,7 +181,11 @@ const KILLED_CONFIG_KEYS = ['subscription', 'attribution', 'geo', 'features', 'a
|
|
|
181
181
|
// v1.5.1 — migration is gradual.
|
|
182
182
|
const CONFIG_SUBOBJECTS = {
|
|
183
183
|
credentials: ['token', 'region', 'serviceAccount', 'serviceSecret', 'projectId'],
|
|
184
|
-
|
|
184
|
+
// v1.6.4: `hasAttributionFlags` removed from this list. It is derived, not
|
|
185
|
+
// settable — line ~942 unconditionally overwrites it with
|
|
186
|
+
// `validatedEvents.some(e => e.isAttributionEvent)`. Listing it here presented
|
|
187
|
+
// a knob that never did anything.
|
|
188
|
+
switches: ['hasLocation', 'hasCampaigns', 'hasAdSpend', 'hasSessionIds', 'hasAvatar', 'hasIOSDevices', 'hasAndroidDevices', 'hasDesktopDevices', 'hasBrowser', 'isAnonymous', 'alsoInferFunnels'],
|
|
185
189
|
identity: ['avgDevicePerUser', 'sessionTimeout'],
|
|
186
190
|
};
|
|
187
191
|
|
|
@@ -744,6 +748,23 @@ export function validateDungeonConfig(config) {
|
|
|
744
748
|
if (verbose && numDays < 14) {
|
|
745
749
|
console.warn(`⚠️ numDays=${numDays} < 14. Strict-bar engine-validation metrics use 14-day windows; results are noisy below 14 days. Consider increasing numDays.`);
|
|
746
750
|
}
|
|
751
|
+
|
|
752
|
+
// v1.6.4 (P3-9): warn when `avgActiveDaysPerUser` and `engagementDecay` are both
|
|
753
|
+
// set. The active-day planner picks a fixed number of distinct days; decay then
|
|
754
|
+
// filters events off the late ones. v1.5 Fix #1 protects the last surviving event
|
|
755
|
+
// on each picked day, so the loss is bounded — but the realized distinct-day count
|
|
756
|
+
// still lands at or below the configured value, never above. This warning is NOT
|
|
757
|
+
// verbose-gated: the combination quietly returns a number the author did not ask for.
|
|
758
|
+
if (
|
|
759
|
+
avgActiveDaysPerUser !== undefined && avgActiveDaysPerUser !== null &&
|
|
760
|
+
Number.isFinite(avgActiveDaysPerUser) && config.engagementDecay
|
|
761
|
+
) {
|
|
762
|
+
console.warn(
|
|
763
|
+
`⚠️ [dungeon-master] avgActiveDaysPerUser=${avgActiveDaysPerUser} is set together with engagementDecay. ` +
|
|
764
|
+
`Decay erodes the effective active-day count, so the realized value will be at or below ` +
|
|
765
|
+
`${avgActiveDaysPerUser}, never above. Prefer one knob or the other. See HOOKS.md §2.5.`
|
|
766
|
+
);
|
|
767
|
+
}
|
|
747
768
|
// ──────────────────────────────────────────────────────────────────────
|
|
748
769
|
|
|
749
770
|
// Use provided name if non-empty string, otherwise generate one
|
|
@@ -909,6 +930,9 @@ export function validateDungeonConfig(config) {
|
|
|
909
930
|
dataQuality = validateDataQuality(dataQuality);
|
|
910
931
|
}
|
|
911
932
|
|
|
933
|
+
// v1.6.4: accept the named-object groupKeys form alongside the legacy tuple.
|
|
934
|
+
const normalizedGroupKeys = normalizeGroupKeys(groupKeys);
|
|
935
|
+
|
|
912
936
|
// Phase 1: validate Funnel.attempts on every funnel (additive — most have none).
|
|
913
937
|
validateAttempts(funnels);
|
|
914
938
|
|
|
@@ -971,7 +995,7 @@ export function validateDungeonConfig(config) {
|
|
|
971
995
|
userProps,
|
|
972
996
|
scdProps,
|
|
973
997
|
mirrorProps,
|
|
974
|
-
groupKeys,
|
|
998
|
+
groupKeys: normalizedGroupKeys,
|
|
975
999
|
groupProps,
|
|
976
1000
|
lookupTables,
|
|
977
1001
|
hasAnonIds: hasAnonIdsResolved,
|
|
@@ -1104,6 +1128,43 @@ function transformSCDPropsWithoutCredentials(config) {
|
|
|
1104
1128
|
if (config.verbose === true) console.log('\u2713 SCD properties converted to static properties\n');
|
|
1105
1129
|
}
|
|
1106
1130
|
|
|
1131
|
+
/**
|
|
1132
|
+
* v1.6.4: normalize `groupKeys` to the positional tuple form the generators,
|
|
1133
|
+
* hooks, and verifier all expect.
|
|
1134
|
+
*
|
|
1135
|
+
* Accepts either form, mixed freely in one array:
|
|
1136
|
+
* - `["company_id", 50]` → unchanged
|
|
1137
|
+
* - `["company_id", 50, ["Purchase"]]` → unchanged
|
|
1138
|
+
* - `{ key: "company_id", cardinality: 50 }` → `["company_id", 50]`
|
|
1139
|
+
* - `{ key, cardinality, events: [...] }` → `[key, cardinality, events]`
|
|
1140
|
+
*
|
|
1141
|
+
* The named form exists so a form-driven generator never has to emit an untyped
|
|
1142
|
+
* positional tuple. Nothing downstream needs to know which form the author used.
|
|
1143
|
+
*
|
|
1144
|
+
* @param {Array} groupKeys
|
|
1145
|
+
* @returns {Array} tuple-form group keys
|
|
1146
|
+
*/
|
|
1147
|
+
function normalizeGroupKeys(groupKeys) {
|
|
1148
|
+
if (!Array.isArray(groupKeys)) return [];
|
|
1149
|
+
return groupKeys.map((entry, i) => {
|
|
1150
|
+
if (Array.isArray(entry)) return entry;
|
|
1151
|
+
if (entry && typeof entry === 'object') {
|
|
1152
|
+
const { key, cardinality, events } = entry;
|
|
1153
|
+
if (typeof key !== 'string' || !key) {
|
|
1154
|
+
throw new Error(`groupKeys[${i}]: named form requires a non-empty string "key" (got ${JSON.stringify(key)})`);
|
|
1155
|
+
}
|
|
1156
|
+
if (!Number.isFinite(cardinality) || cardinality < 1) {
|
|
1157
|
+
throw new Error(`groupKeys[${i}] ("${key}"): named form requires a "cardinality" >= 1 (got ${JSON.stringify(cardinality)})`);
|
|
1158
|
+
}
|
|
1159
|
+
if (events !== undefined && !Array.isArray(events)) {
|
|
1160
|
+
throw new Error(`groupKeys[${i}] ("${key}"): "events" must be an array of event names when present (got ${typeof events})`);
|
|
1161
|
+
}
|
|
1162
|
+
return events && events.length ? [key, cardinality, events] : [key, cardinality];
|
|
1163
|
+
}
|
|
1164
|
+
throw new Error(`groupKeys[${i}]: expected a [key, cardinality] tuple or a { key, cardinality } object (got ${typeof entry})`);
|
|
1165
|
+
});
|
|
1166
|
+
}
|
|
1167
|
+
|
|
1107
1168
|
// ── Advanced Feature Validation Functions ──
|
|
1108
1169
|
|
|
1109
1170
|
// P2.5 (v1.6): churnRate / activeWindow / soupOverride are declared Persona
|
|
@@ -376,7 +376,10 @@ export async function makeFunnel(context, funnel, user, firstEventTime, profile
|
|
|
376
376
|
if (Number.isFinite(lastTimeMs)) {
|
|
377
377
|
const IDENTITY_KEYS = ['user_id', 'device_id', 'distinct_id', 'session_id', 'insert_id'];
|
|
378
378
|
const superPropKeys = Object.keys(superProps || {});
|
|
379
|
-
|
|
379
|
+
// Validated config: group keys are always positional tuples here.
|
|
380
|
+
const groupKeyNames = /** @type {string[]} */ (
|
|
381
|
+
(groupKeys || []).map(gk => Array.isArray(gk) ? gk[0] : gk).filter(gk => typeof gk === 'string')
|
|
382
|
+
);
|
|
380
383
|
const numToInject = chance.integer({ min: 1, max: 2 });
|
|
381
384
|
for (let i = 0; i < numToInject; i++) {
|
|
382
385
|
const excName = chance.pickone(funnel.exclusionEvents);
|
|
@@ -53,12 +53,27 @@ export function dropEventsWhere(events, predicate) {
|
|
|
53
53
|
* Scale the count of events with name `eventName` in `events` by `factor`. Mutates
|
|
54
54
|
* `events` in place.
|
|
55
55
|
*
|
|
56
|
-
* - factor > 1: clones existing matches
|
|
57
|
-
* so duplicates land just after their source. Returns positive integer = clones added.
|
|
56
|
+
* - factor > 1: clones existing matches. Returns positive integer = clones added.
|
|
58
57
|
* - factor < 1: drops matches at random using the seeded RNG. Returns negative
|
|
59
58
|
* integer = -dropped.
|
|
60
59
|
* - factor === 1 or no matches: no-op, returns 0.
|
|
61
60
|
*
|
|
61
|
+
* ## WARNING — the default spread cannot move frequency or session metrics
|
|
62
|
+
*
|
|
63
|
+
* By default clones land 1 second after their source (monotonic 1s steps). One
|
|
64
|
+
* second is not enough to create a new distinct active day, and not enough to
|
|
65
|
+
* open a new session under Mixpanel's 30-minute inactivity gap. So a default
|
|
66
|
+
* `scaleEventCount` call raises **event volume only**. It does NOT raise:
|
|
67
|
+
*
|
|
68
|
+
* - active days per user / DAU / stickiness (DAU÷MAU)
|
|
69
|
+
* - sessions per user or session count
|
|
70
|
+
* - "days active" or frequency-of-use reports
|
|
71
|
+
* - retention, which is computed on distinct days
|
|
72
|
+
*
|
|
73
|
+
* If your story claims any of those, pass `{ spreadDays: N }` to scatter clones
|
|
74
|
+
* across the following N days, or use `injectOnNewDays` — the helper built for
|
|
75
|
+
* exactly this.
|
|
76
|
+
*
|
|
62
77
|
* Note: cloned events get a FRESH `insert_id` — they must not inherit the
|
|
63
78
|
* source's, and leaving it blank is not safe either (the importer content-hashes
|
|
64
79
|
* missing ids, so identical clones collide and Mixpanel dedupes them away).
|
|
@@ -66,21 +81,40 @@ export function dropEventsWhere(events, predicate) {
|
|
|
66
81
|
* @param {Array<{event:string,time:string|number,insert_id?:string}>} events
|
|
67
82
|
* @param {string} eventName
|
|
68
83
|
* @param {number} factor
|
|
84
|
+
* @param {{spreadDays?: number}} [options] - `spreadDays`: scatter clones uniformly
|
|
85
|
+
* over `[source_time, source_time + spreadDays]` instead of stepping 1s at a time.
|
|
86
|
+
* Use this when the clones must land on new calendar days or in new sessions.
|
|
87
|
+
* Clones that land past `FIXED_NOW` are dropped by the engine's future-time guard,
|
|
88
|
+
* so a large `spreadDays` on late-window events yields fewer surviving clones than
|
|
89
|
+
* the return value reports.
|
|
69
90
|
* @returns {number}
|
|
70
91
|
*/
|
|
71
|
-
export function scaleEventCount(events, eventName, factor) {
|
|
92
|
+
export function scaleEventCount(events, eventName, factor, options = {}) {
|
|
72
93
|
if (!events || !eventName || typeof factor !== 'number' || factor === 1) return 0;
|
|
94
|
+
const { spreadDays } = options || {};
|
|
95
|
+
const spreadMs = (Number.isFinite(spreadDays) && spreadDays > 0)
|
|
96
|
+
? spreadDays * 86_400_000
|
|
97
|
+
: 0;
|
|
73
98
|
if (factor > 1) {
|
|
74
99
|
const matches = events.filter(e => e && e.event === eventName);
|
|
75
100
|
if (!matches.length) return 0;
|
|
76
101
|
const additionalNeeded = Math.round(matches.length * (factor - 1));
|
|
102
|
+
const chance = spreadMs ? getChance() : null;
|
|
77
103
|
let added = 0;
|
|
78
104
|
for (let i = 0; i < additionalNeeded; i++) {
|
|
79
105
|
const src = matches[i % matches.length];
|
|
80
106
|
const baseMs = toMs(src.time);
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
107
|
+
let newTime;
|
|
108
|
+
if (!Number.isFinite(baseMs)) {
|
|
109
|
+
newTime = src.time;
|
|
110
|
+
} else if (spreadMs) {
|
|
111
|
+
// Seeded uniform offset across the spread window. Floor at 1s so a
|
|
112
|
+
// clone never collides exactly with its source's timestamp.
|
|
113
|
+
const offset = Math.max(1000, Math.round(chance.floating({ min: 0, max: spreadMs })));
|
|
114
|
+
newTime = new Date(baseMs + offset).toISOString();
|
|
115
|
+
} else {
|
|
116
|
+
newTime = new Date(baseMs + (i + 1) * 1000).toISOString();
|
|
117
|
+
}
|
|
84
118
|
const clone = stampFreshInsertId({ ...src, time: newTime });
|
|
85
119
|
events.push(clone);
|
|
86
120
|
added++;
|
|
@@ -37,15 +37,16 @@ export interface Dungeon {
|
|
|
37
37
|
/** Properties that change for users or groups over time (Slowly Changing Dimensions). */
|
|
38
38
|
scdProps?: Record<string, SCDProp>;
|
|
39
39
|
|
|
40
|
-
/**
|
|
41
|
-
|
|
42
|
-
|
|
40
|
+
/**
|
|
41
|
+
* Defines group entities, like companies or teams, and how many of each to create.
|
|
42
|
+
* Tuple form: `[key, numGroups]` or `[key, numGroups, [events]]`.
|
|
43
|
+
* Named form: `{ key, cardinality, events? }`. Both are accepted.
|
|
44
|
+
*/
|
|
45
|
+
groupKeys?: ([string, number, string[]?] | { key: string; cardinality: number; events?: string[] })[];
|
|
46
|
+
|
|
43
47
|
/** Properties that define the characteristics of the groups defined in groupKeys. */
|
|
44
48
|
groupProps?: Record<string, Record<string, ValueValid>>;
|
|
45
49
|
|
|
46
|
-
/** Events that are attributed to a group entity rather than an individual user. */
|
|
47
|
-
groupEvents?: GroupEventConfig[];
|
|
48
|
-
|
|
49
50
|
/** Static data tables (e.g., product catalogs) that can be referenced in events. */
|
|
50
51
|
lookupTables?: LookupTableSchema[];
|
|
51
52
|
|
|
@@ -134,25 +135,6 @@ interface SCDProp {
|
|
|
134
135
|
}
|
|
135
136
|
|
|
136
137
|
|
|
137
|
-
/**
|
|
138
|
-
* Defines an event that is attributed to a group and occurs on a regular schedule.
|
|
139
|
-
* (e.g., a monthly subscription charge for a company).
|
|
140
|
-
*/
|
|
141
|
-
interface GroupEventConfig extends EventConfig {
|
|
142
|
-
/** How often the event occurs (in days). */
|
|
143
|
-
frequency: number;
|
|
144
|
-
|
|
145
|
-
/** The group key this event is associated with (e.g., "company_id"). */
|
|
146
|
-
group_key: string;
|
|
147
|
-
|
|
148
|
-
/** If true, a random user within the group is also associated with the event. */
|
|
149
|
-
attribute_to_user: boolean;
|
|
150
|
-
|
|
151
|
-
/** The number of entities in this group. */
|
|
152
|
-
group_size: number;
|
|
153
|
-
}
|
|
154
|
-
|
|
155
|
-
|
|
156
138
|
/**
|
|
157
139
|
* Defines the schema for a static lookup table, which can be used to enrich event data.
|
|
158
140
|
* For example, a "products" table could hold details about product IDs.
|
|
@@ -17,7 +17,7 @@ type ValueValid = Primitives | Primitives[] | FunctionCall;
|
|
|
17
17
|
* This is the high-level object you will be constructing.
|
|
18
18
|
*
|
|
19
19
|
* REQUIRED fields: events, funnels, superProps, userProps
|
|
20
|
-
* OPTIONAL fields: scdProps, groupKeys, groupProps
|
|
20
|
+
* OPTIONAL fields: scdProps, groupKeys, groupProps
|
|
21
21
|
*/
|
|
22
22
|
export interface Dungeon {
|
|
23
23
|
/** REQUIRED: A list of all possible events that can occur in the simulation. */
|
|
@@ -35,14 +35,15 @@ export interface Dungeon {
|
|
|
35
35
|
/** OPTIONAL: Properties that change for users or groups over time (Slowly Changing Dimensions). Only include when properties explicitly change over time. */
|
|
36
36
|
scdProps?: Record<string, SCDProp>;
|
|
37
37
|
|
|
38
|
-
/**
|
|
39
|
-
|
|
38
|
+
/**
|
|
39
|
+
* OPTIONAL: Defines group entities (companies, teams). ONLY for B2B/SaaS scenarios.
|
|
40
|
+
* Tuple form: `[["group_key", count], ...]`.
|
|
41
|
+
* Named form: `[{ key: "group_key", cardinality: count, events: [...] }, ...]`.
|
|
42
|
+
*/
|
|
43
|
+
groupKeys?: ([string, number] | { key: string; cardinality: number; events?: string[] })[];
|
|
40
44
|
|
|
41
45
|
/** OPTIONAL: Properties for groups defined in groupKeys. ONLY include if groupKeys is defined. */
|
|
42
46
|
groupProps?: Record<string, Record<string, ValueValid>>;
|
|
43
|
-
|
|
44
|
-
/** OPTIONAL: Events attributed to groups on a schedule (e.g., monthly billing). Rarely needed. */
|
|
45
|
-
groupEvents?: GroupEventConfig[];
|
|
46
47
|
}
|
|
47
48
|
|
|
48
49
|
|
|
@@ -146,28 +147,6 @@ interface SCDProp {
|
|
|
146
147
|
}
|
|
147
148
|
|
|
148
149
|
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
*
|
|
153
|
-
* This is rarely needed - only use for B2B scenarios with recurring group-level events.
|
|
154
|
-
*/
|
|
155
|
-
interface GroupEventConfig {
|
|
156
|
-
/** REQUIRED: The name of the event. */
|
|
157
|
-
event: string;
|
|
158
|
-
|
|
159
|
-
/** REQUIRED: How often the event occurs (in days). e.g., 30 for monthly. */
|
|
160
|
-
frequency: number;
|
|
161
|
-
|
|
162
|
-
/** REQUIRED: The group key this event belongs to (e.g., "company_id"). */
|
|
163
|
-
group_key: string;
|
|
164
|
-
|
|
165
|
-
/** OPTIONAL: If true, a random user in the group is also attributed to the event. */
|
|
166
|
-
attribute_to_user?: boolean;
|
|
167
|
-
|
|
168
|
-
/** OPTIONAL: Properties for this event. */
|
|
169
|
-
properties?: Record<string, ValueValid>;
|
|
170
|
-
|
|
171
|
-
/** OPTIONAL: Relative frequency of this event. */
|
|
172
|
-
weight?: number;
|
|
173
|
-
}
|
|
150
|
+
// v1.6.4 — `GroupEventConfig` removed. It was a declared-only stub that nothing in
|
|
151
|
+
// `lib/` ever read. To scope an event to a group, list its name in that group key's
|
|
152
|
+
// `events` array instead.
|
package/lib/utils/utils.js
CHANGED
package/package.json
CHANGED
package/types.d.ts
CHANGED
|
@@ -13,22 +13,56 @@ type Primitives = string | number | boolean | Date | Record<string, any>;
|
|
|
13
13
|
export type ValueValid = Primitives | ValueValid[] | (() => ValueValid);
|
|
14
14
|
|
|
15
15
|
/**
|
|
16
|
-
*
|
|
17
|
-
|
|
18
|
-
|
|
16
|
+
* Mixpanel data residency region. Matches the set `mixpanel-import` accepts.
|
|
17
|
+
*/
|
|
18
|
+
export type Region = 'US' | 'EU' | 'IN';
|
|
19
|
+
|
|
20
|
+
/**
|
|
21
|
+
* v1.6.4 — named-object form of a group key. Equivalent to the positional tuple
|
|
22
|
+
* `[key, cardinality]` / `[key, cardinality, events]`, which stays supported.
|
|
23
|
+
* The validator normalizes this to the tuple form.
|
|
24
|
+
*/
|
|
25
|
+
export interface GroupKeyObject {
|
|
26
|
+
/** The group key property name (e.g. `"company_id"`). */
|
|
27
|
+
key: string;
|
|
28
|
+
/** How many distinct group entities to generate. */
|
|
29
|
+
cardinality: number;
|
|
30
|
+
/** Event names that carry this group key. Omit or leave empty for all events. */
|
|
31
|
+
events?: string[];
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
/**
|
|
35
|
+
* The normalized positional form of a group key. Everything downstream of
|
|
36
|
+
* `validateDungeonConfig` sees this shape — the validator converts
|
|
37
|
+
* `GroupKeyObject` entries for you.
|
|
38
|
+
*/
|
|
39
|
+
export type GroupKeyTuple = [string, number] | [string, number, string[]];
|
|
40
|
+
|
|
41
|
+
/**
|
|
42
|
+
* A group analytics key as an author may write it: either the legacy tuple form
|
|
43
|
+
* or the named-object form. `result.validatedConfig.groupKeys` is always
|
|
44
|
+
* `GroupKeyTuple[]`.
|
|
45
|
+
*/
|
|
46
|
+
export type GroupKey = GroupKeyTuple | GroupKeyObject;
|
|
47
|
+
|
|
48
|
+
/**
|
|
49
|
+
* v1.5.1 — credentials sub-object. **This is the canonical form.** Groups Mixpanel
|
|
50
|
+
* project credentials. Top-level `token` / `region` / etc. remain functional as a
|
|
51
|
+
* back-compat alias; when both are set, the top-level value wins with a verbose
|
|
52
|
+
* warning. Emit one form or the other, never both.
|
|
19
53
|
*/
|
|
20
54
|
export interface DungeonCredentials {
|
|
21
55
|
token?: string;
|
|
22
|
-
region?:
|
|
56
|
+
region?: Region;
|
|
23
57
|
serviceAccount?: string;
|
|
24
58
|
serviceSecret?: string;
|
|
25
59
|
projectId?: string;
|
|
26
60
|
}
|
|
27
61
|
|
|
28
62
|
/**
|
|
29
|
-
* v1.5.1 — switches sub-object. Groups data-shape
|
|
30
|
-
* remain functional as a back-compat alias; same
|
|
31
|
-
* `DungeonCredentials`.
|
|
63
|
+
* v1.5.1 — switches sub-object. **This is the canonical form.** Groups data-shape
|
|
64
|
+
* booleans. Top-level keys remain functional as a back-compat alias; same
|
|
65
|
+
* precedence rules as `DungeonCredentials`. Emit one form or the other, never both.
|
|
32
66
|
*/
|
|
33
67
|
export interface DungeonSwitches {
|
|
34
68
|
hasLocation?: boolean;
|
|
@@ -42,13 +76,13 @@ export interface DungeonSwitches {
|
|
|
42
76
|
hasBrowser?: boolean;
|
|
43
77
|
isAnonymous?: boolean;
|
|
44
78
|
alsoInferFunnels?: boolean;
|
|
45
|
-
hasAttributionFlags?: boolean;
|
|
46
79
|
}
|
|
47
80
|
|
|
48
81
|
/**
|
|
49
|
-
* v1.5.1 — identity sub-object.
|
|
50
|
-
* `avgDevicePerUser` / `sessionTimeout` remain
|
|
51
|
-
* alias
|
|
82
|
+
* v1.5.1 — identity sub-object. **This is the canonical form.** Groups
|
|
83
|
+
* identity-model knobs. Top-level `avgDevicePerUser` / `sessionTimeout` remain
|
|
84
|
+
* functional as a back-compat alias; when both are set, the top-level value wins
|
|
85
|
+
* with a verbose warning. Emit one form or the other, never both.
|
|
52
86
|
*
|
|
53
87
|
* `hasAnonIds` is DEPRECATED — when present here, it maps to
|
|
54
88
|
* `avgDevicePerUser: 1` with a verbose warning. Use `avgDevicePerUser` instead.
|
|
@@ -143,8 +177,8 @@ export interface Dungeon {
|
|
|
143
177
|
avgEventsPerUserPerDay?: number;
|
|
144
178
|
/** Output format for files written to disk. */
|
|
145
179
|
format?: "csv" | "json" | "parquet" | string;
|
|
146
|
-
/** Mixpanel data residency region. */
|
|
147
|
-
region?:
|
|
180
|
+
/** Mixpanel data residency region. Back-compat alias for `credentials.region`. */
|
|
181
|
+
region?: Region;
|
|
148
182
|
/** User generation concurrency. Default: 1. Values > 1 break seed reproducibility and provide no performance benefit (CPU-bound). */
|
|
149
183
|
concurrency?: number;
|
|
150
184
|
/**
|
|
@@ -186,6 +220,12 @@ export interface Dungeon {
|
|
|
186
220
|
* @see EventConfig.isAttributionEvent
|
|
187
221
|
*/
|
|
188
222
|
hasCampaigns?: boolean;
|
|
223
|
+
/**
|
|
224
|
+
* @internal Derived, not settable. The validator unconditionally sets this to
|
|
225
|
+
* `events.some(e => e.isAttributionEvent)`. Read it off `result.validatedConfig`;
|
|
226
|
+
* setting it on an input config has no effect.
|
|
227
|
+
*/
|
|
228
|
+
hasAttributionFlags?: boolean;
|
|
189
229
|
/** If true, generates ad spend data (impressions, clicks, cost). */
|
|
190
230
|
hasAdSpend?: boolean;
|
|
191
231
|
/** If true, device pool includes iOS devices. */
|
|
@@ -196,7 +236,7 @@ export interface Dungeon {
|
|
|
196
236
|
hasDesktopDevices?: boolean;
|
|
197
237
|
/** If true, events include browser properties. */
|
|
198
238
|
hasBrowser?: boolean;
|
|
199
|
-
/** If true
|
|
239
|
+
/** If true, writes output files to ./data/. Can also be a directory path string or gs:// URI. Default: `false` — data is returned in memory only. */
|
|
200
240
|
writeToDisk?: boolean | string;
|
|
201
241
|
/** If true, deletes all written files (local and GCS) at end of run regardless of import success/failure. Default: false. */
|
|
202
242
|
cleanup?: boolean;
|
|
@@ -289,12 +329,19 @@ export interface Dungeon {
|
|
|
289
329
|
scdProps?: Record<string, SCDProp>;
|
|
290
330
|
/** Mirror dataset definitions: create transformed copies of event data. */
|
|
291
331
|
mirrorProps?: Record<string, MirrorProps>;
|
|
292
|
-
/**
|
|
293
|
-
|
|
332
|
+
/**
|
|
333
|
+
* Group analytics keys. Two interchangeable forms:
|
|
334
|
+
*
|
|
335
|
+
* - Tuple (legacy): `[key, numGroups]` or `[key, numGroups, [associatedEvents]]`
|
|
336
|
+
* - Named object (v1.6.4, preferred): `{ key, cardinality, events? }`
|
|
337
|
+
*
|
|
338
|
+
* The validator normalizes the named form to the tuple form, so hooks and the
|
|
339
|
+
* verifier always see tuples. Mixing both forms in one array is allowed.
|
|
340
|
+
* An empty or omitted `events` list means every event carries the group key.
|
|
341
|
+
*/
|
|
342
|
+
groupKeys?: GroupKey[];
|
|
294
343
|
/** Properties for each group key's entities. */
|
|
295
344
|
groupProps?: Record<string, Record<string, ValueValid>>;
|
|
296
|
-
/** Group-level events (stub — not yet implemented). */
|
|
297
|
-
groupEvents?: GroupEventConfig[];
|
|
298
345
|
/** Lookup table definitions for dimension tables. */
|
|
299
346
|
lookupTables?: LookupTableSchema[];
|
|
300
347
|
/** TimeSoup configuration: shapes intra-week and intra-day rhythm (peaks, deviation, DOW/HOD weights). Pair with `macro` for big-picture trend control. */
|
|
@@ -370,7 +417,11 @@ export interface Dungeon {
|
|
|
370
417
|
* **Incompatibility with `engagementDecay`:** decay drops events from late picked days,
|
|
371
418
|
* eroding the effective active-day count below the configured target. Use one or the
|
|
372
419
|
* other; if you need both, write decay logic in an `everything` hook scoped to specific
|
|
373
|
-
* cohorts. See HOOKS.md §2.5.
|
|
420
|
+
* cohorts. See HOOKS.md §2.5. v1.6.4: the validator warns unconditionally (not gated
|
|
421
|
+
* behind `verbose`) when both are set.
|
|
422
|
+
*
|
|
423
|
+
* **Precedence with `retentionCurve`:** the curve WINS. When `retentionCurve` is set,
|
|
424
|
+
* the active-day plan is built from the curve and this value is ignored entirely.
|
|
374
425
|
*
|
|
375
426
|
* Safe range: `[1, numDays * 0.5]`. Above 50% of `numDays` defeats the concentrator
|
|
376
427
|
* purpose; the v1.5 validator strict-clamps to `floor(numDays * 0.5)` with a warning.
|
|
@@ -928,12 +979,10 @@ export interface EventConfig {
|
|
|
928
979
|
isAttributionEvent?: boolean;
|
|
929
980
|
}
|
|
930
981
|
|
|
931
|
-
|
|
932
|
-
|
|
933
|
-
|
|
934
|
-
|
|
935
|
-
group_size: number; //the number of users in the group
|
|
936
|
-
}
|
|
982
|
+
// v1.6.4 — `GroupEventConfig` and `Dungeon.groupEvents` were removed from the
|
|
983
|
+
// public types. They were a declared-only stub: nothing in `lib/` ever read them,
|
|
984
|
+
// and no shipped dungeon set them. Group-scoped events are modeled today by
|
|
985
|
+
// listing the event name in a `groupKeys` entry's `events` array.
|
|
937
986
|
|
|
938
987
|
/**
|
|
939
988
|
* the generated event data
|