@ak--47/dungeon-master 1.8.0 → 1.8.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/analyze-soup/SKILL.md +9 -0
- package/.claude/skills/create-dungeon/SKILL.md +35 -34
- package/.claude/skills/create-project/SKILL.md +6 -0
- package/.claude/skills/headless-build/SKILL.md +21 -11
- package/.claude/skills/powertools/SKILL.md +6 -2
- package/.claude/skills/release-check/SKILL.md +27 -2
- package/.claude/skills/verify-dungeon/SKILL.md +32 -13
- package/.claude/skills/verify-dungeon/references/alignment-contract.md +110 -0
- package/.claude/skills/verify-dungeon/references/counting-semantics.md +29 -16
- package/.claude/skills/verify-dungeon/references/report-format.md +23 -9
- package/.claude/skills/verify-dungeon/references/sql-recipes.md +135 -225
- package/.claude/skills/warehouse-metrics/SKILL.md +6 -0
- package/.claude/skills/write-hooks/SKILL.md +61 -48
- package/CHANGELOG.md +82 -0
- package/HOOKS.md +105 -47
- package/README.md +41 -1
- package/docs/guides/1.8.1-upgrade-guide.md +153 -0
- package/docs/guides/1.8.2-upgrade-guide.md +110 -0
- package/lib/generators/events.js +6 -0
- package/lib/generators/funnels.js +16 -0
- package/lib/hook-helpers/shape.js +73 -17
- package/lib/hook-patterns/attributed-by-source.js +4 -3
- package/lib/hook-patterns/funnel-frequency-breakdown.js +4 -7
- package/lib/orchestrators/user-loop.js +82 -15
- package/lib/verify/counting.js +7 -10
- package/lib/verify/emulate-breakdown.js +48 -29
- package/lib/verify/funnel-engine.js +93 -40
- package/lib/verify/identity.js +32 -9
- package/lib/verify/story-runner.js +93 -30
- package/lib/verify/verify-dungeon.js +4 -1
- package/package.json +1 -1
- package/scripts/verify-stories.mjs +3 -3
- package/types.d.ts +9 -5
|
@@ -169,6 +169,15 @@ its folder. Otherwise write it to the project root. (The generated
|
|
|
169
169
|
6. **Assessment**: Overall quality judgment and recommendations
|
|
170
170
|
|
|
171
171
|
### Quality Criteria
|
|
172
|
+
|
|
173
|
+
These thresholds are diagnostic heuristics, not alignment acceptance criteria.
|
|
174
|
+
Interpret them against the configured macro, soup, resolved window, and intentional
|
|
175
|
+
hook effects. Read the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md)
|
|
176
|
+
for proof scope. Report insufficient observations separately from shape failures.
|
|
177
|
+
Include empty buckets and identify partial buckets before computing distribution
|
|
178
|
+
statistics; the SQL above reports observed buckets only. Sparse or partial buckets
|
|
179
|
+
alone do not establish a distribution defect.
|
|
180
|
+
|
|
172
181
|
- **Daily CV**: 0.2-0.6 is ideal (some variation, not flat or spiky)
|
|
173
182
|
- **Max-to-avg ratio**: < 2.0 at daily level, < 3.0 at hourly level
|
|
174
183
|
- **Last day spike**: < 1.5x average = PASS, < 2x = WARN, > 2x = FAIL
|
|
@@ -35,11 +35,12 @@ In scope here:
|
|
|
35
35
|
- `credentials: { token, region, serviceAccount, serviceSecret, projectId }`
|
|
36
36
|
- `switches: { hasLocation, hasCampaigns, hasSessionIds, hasAvatar,
|
|
37
37
|
hasIOSDevices, hasAndroidDevices, hasDesktopDevices, hasBrowser,
|
|
38
|
-
isAnonymous, alsoInferFunnels, hasAdSpend
|
|
38
|
+
isAnonymous, alsoInferFunnels, hasAdSpend }`
|
|
39
39
|
- `identity: { avgDevicePerUser, sessionTimeout }`
|
|
40
40
|
|
|
41
41
|
Old top-level keys keep working (verbose warn nudges migration), but new
|
|
42
42
|
dungeons should ship the sub-object shape.
|
|
43
|
+
`hasAttributionFlags` is derived from `events[].isAttributionEvent`; do not set it.
|
|
43
44
|
- Surviving advanced entities: `personas`, `worldEvents`, `engagementDecay`,
|
|
44
45
|
`dataQuality` — use sparingly
|
|
45
46
|
|
|
@@ -64,6 +65,7 @@ see `HOOKS.md` at the project root.
|
|
|
64
65
|
|
|
65
66
|
Before writing any code, scan:
|
|
66
67
|
|
|
68
|
+
- [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md) - report definitions, neutral controls, and proof limits.
|
|
67
69
|
- `types.d.ts` — the complete API reference. Every Dungeon field, EventConfig
|
|
68
70
|
flag, Funnel option, AttemptsConfig, and Hook meta interface is documented
|
|
69
71
|
with full JSDoc. **Treat this as the source of truth.**
|
|
@@ -234,8 +236,8 @@ When using experiments, include `$experiment_started` in the events array with
|
|
|
234
236
|
|
|
235
237
|
- First funnel: includes `isFirstEvent` AND has `isAuthEvent: true` on the
|
|
236
238
|
identity-transition step.
|
|
237
|
-
- Usage funnels: ordinary sequences without `isFirstFunnel`.
|
|
238
|
-
`attempts`
|
|
239
|
+
- Usage funnels: ordinary sequences without `isFirstFunnel`. Volume and funnel
|
|
240
|
+
selection control repetitions; `attempts` applies only to born-user first funnels.
|
|
239
241
|
- Pick `conversionRate` between 30 and 80; `timeToConvert` in hours.
|
|
240
242
|
|
|
241
243
|
Funnel `props` stamp constant properties on all events in that funnel run.
|
|
@@ -267,9 +269,9 @@ consumed standalone instances as funnel matches. Set `isStrictEvent: false`
|
|
|
267
269
|
explicitly only when you intend mixed funnel/standalone semantics for that
|
|
268
270
|
event.
|
|
269
271
|
|
|
270
|
-
**
|
|
272
|
+
**Use `isStrictEvent: false` when a cohort needs standalone occurrences.** Auto-promotion removes standalone traffic but leaves funnel-generated occurrences readable. Declare the opt-out at schema time only when that extra traffic is part of the intended behavior; a hook reading an event name alone does not require it.
|
|
271
273
|
|
|
272
|
-
|
|
274
|
+
**`reentry` is verifier-only.** Usage volume, funnel selection, and the event budget control generated repetitions. Set `reentry: true` only when the intended report counts repeated histories. It does not generate loops; local totals also default to `reentry: false`.
|
|
273
275
|
|
|
274
276
|
**Structural trend engineering — duplicate funnels instead of reaching for
|
|
275
277
|
hooks.** Not every trend needs a hook: initial conditions can raise or lower
|
|
@@ -294,20 +296,19 @@ to architect a comparison directly into the schema:
|
|
|
294
296
|
Prefer structure when the story is a *between-path comparison* (this path
|
|
295
297
|
converts worse / slower than that one). Reach for hooks when the story is a
|
|
296
298
|
*within-cohort behavior* (these users do more of X over time, this segment's
|
|
297
|
-
values differ). Structural trends
|
|
298
|
-
|
|
299
|
+
values differ). Structural trends still need report-level proof: competing
|
|
300
|
+
histories, saturation, and finite budgets can change the observed effect.
|
|
299
301
|
|
|
300
302
|
### 3. SuperProps (2–3)
|
|
301
303
|
|
|
302
304
|
Properties present on EVERY event. Common picks: `Plan`, `Region`, `Platform`,
|
|
303
|
-
`App Version`.
|
|
304
|
-
|
|
305
|
+
`App Version`. Matching `userProps` enumerations only shares a value domain.
|
|
306
|
+
Use `stickyEventProps` for values that must match each user's profile.
|
|
305
307
|
|
|
306
308
|
### 4. UserProps (4–8)
|
|
307
309
|
|
|
308
|
-
User profile properties. Set once per user.
|
|
309
|
-
|
|
310
|
-
Region).
|
|
310
|
+
User profile properties. Set once per user. Declare overlapping event keys in
|
|
311
|
+
`superProps` and use `stickyEventProps` when profile/event equality is required.
|
|
311
312
|
|
|
312
313
|
### 5. Groups (0–2)
|
|
313
314
|
|
|
@@ -417,9 +418,9 @@ Typical ranges:
|
|
|
417
418
|
|
|
418
419
|
- Direct-acquisition first funnels: `{ min: 0, max: 0 }` (single attempt) or omit
|
|
419
420
|
- Shared-link / friction-heavy onboarding: `{ min: 0, max: 2 }` (some retry)
|
|
420
|
-
-
|
|
421
|
+
- Usage funnels do not consume this retry plan; model their repetitions through usage selection and volume.
|
|
421
422
|
|
|
422
|
-
|
|
423
|
+
For a born user's first funnel, `attempts.conversionRate` (optional) overrides `funnel.conversionRate`
|
|
423
424
|
on the FINAL attempt. Failed prior attempts truncate before the first
|
|
424
425
|
`isAuthEvent` step (no stitch fires for those attempts).
|
|
425
426
|
|
|
@@ -504,12 +505,12 @@ intra-day rhythm). Only override when you have a specific reason:
|
|
|
504
505
|
|
|
505
506
|
### Macro × born% / bias compatibility (strict clamps)
|
|
506
507
|
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
viral=55, decline=5.
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
508
|
+
The per-preset born% cap applies only to a named preset (`macro: 'growth'` or
|
|
509
|
+
`macro: { preset: 'growth', ... }`) with explicit `percentUsersBornInDataset`:
|
|
510
|
+
flat=12, steady=12, growth=30, viral=55, decline=5. A custom macro object without
|
|
511
|
+
`preset` has no per-preset cap; own and verify its shape. The global born% clamp
|
|
512
|
+
to [0,100] still applies. Explicit `bornRecentBias` is separately clamped to
|
|
513
|
+
[-0.5,0.5], with compound protection for high born% and bias. Preset bias is exempt.
|
|
513
514
|
|
|
514
515
|
The clamp warning explains why and points to safe alternatives — read it.
|
|
515
516
|
|
|
@@ -574,18 +575,19 @@ Top-level optional knob. Shape retention via log-linear interpolation
|
|
|
574
575
|
between waypoints. Independent of `engagementDecay`.
|
|
575
576
|
|
|
576
577
|
```js
|
|
577
|
-
retentionCurve:
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
578
|
+
retentionCurve: {
|
|
579
|
+
type: 'logarithmic',
|
|
580
|
+
day1: 0.80,
|
|
581
|
+
day7: 0.50,
|
|
582
|
+
day30: 0.20,
|
|
583
|
+
}
|
|
583
584
|
```
|
|
584
585
|
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
586
|
+
Use `type: 'logarithmic'` (default) or `'linear'`, with `dayN` active-day
|
|
587
|
+
weights; day 0 is implicitly 1. The curve weights the active-day plan and takes
|
|
588
|
+
precedence over `avgActiveDaysPerUser`. It does not directly filter each event
|
|
589
|
+
or guarantee report D1/D7/D30 percentages. Verify mature cohorts and account for
|
|
590
|
+
funnel spill and finite event budgets.
|
|
589
591
|
|
|
590
592
|
### `userSeed` (separate distinct_id RNG seed, v1.5+)
|
|
591
593
|
|
|
@@ -601,16 +603,15 @@ userSeed: "users-v1", // user-pool RNG (stable across versions)
|
|
|
601
603
|
|
|
602
604
|
## SuperProp consistency rule
|
|
603
605
|
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
will fire events with `superProps.Plan = 'free'` — Mixpanel will see broken
|
|
607
|
-
breakdowns.
|
|
606
|
+
Matching enumerations do not guarantee profile/event equality: independent draws
|
|
607
|
+
can differ. Declare both keys and project the profile values with `stickyEventProps`.
|
|
608
608
|
|
|
609
609
|
```js
|
|
610
610
|
const PLANS = ["Free", "Free", "Free", "Pro", "Pro", "Enterprise"];
|
|
611
611
|
// ...
|
|
612
612
|
superProps: { Plan: PLANS, Region: REGIONS },
|
|
613
613
|
userProps: { Plan: PLANS, Region: REGIONS, Role: ROLES, ... },
|
|
614
|
+
stickyEventProps: ['Plan', 'Region'],
|
|
614
615
|
```
|
|
615
616
|
|
|
616
617
|
## Verification
|
|
@@ -16,6 +16,12 @@ This is the step after a dungeon is authored, reviewed, and tweaked. It turns a
|
|
|
16
16
|
local dungeon into one you can actually send to Mixpanel by creating the project
|
|
17
17
|
and stamping `credentials` back into the file.
|
|
18
18
|
|
|
19
|
+
Follow the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md)
|
|
20
|
+
when carrying verification claims into business context. Preserve the actual report
|
|
21
|
+
definition and explicit counting options. Local evidence is selected source-derived
|
|
22
|
+
verification, not live Mixpanel parity. Provisioning is a separately authorized
|
|
23
|
+
handoff; never run it automatically during offline verification.
|
|
24
|
+
|
|
19
25
|
## What it does
|
|
20
26
|
|
|
21
27
|
All work runs through the orchestrator `provision.mjs` (this skill's directory),
|
|
@@ -37,10 +37,14 @@ saved cohorts and custom properties, saved behaviors/metrics/formulas, and
|
|
|
37
37
|
annotations that explain the engineered moments. Then it re-measures the hook
|
|
38
38
|
stories **against the live project** and fails if they no longer read.
|
|
39
39
|
|
|
40
|
-
|
|
41
|
-
|
|
40
|
+
Local story verification uses generated records in memory or retained disk
|
|
41
|
+
artifacts. It does not establish what survived ingest. A build can render five
|
|
42
42
|
perfect dashboards on top of a story that silently collapsed on the way in.
|
|
43
43
|
|
|
44
|
+
Apply the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md)
|
|
45
|
+
when translating local stories into live reports. This build requires authorized
|
|
46
|
+
online work; it is never an automatic step of offline verification.
|
|
47
|
+
|
|
44
48
|
## Scope
|
|
45
49
|
|
|
46
50
|
In scope: dashboards + charts, Lexicon enrichment, cohorts, custom properties,
|
|
@@ -182,13 +186,14 @@ size, require each bucket to be observed END TO END (`(b+1)*unit - 1` days of hi
|
|
|
182
186
|
not `b*unit`), and cap the cohort date. Put both numbers on the board — the diluted
|
|
183
187
|
one is what the chart shows, and naming why is a better demo than hiding it.
|
|
184
188
|
|
|
185
|
-
**
|
|
189
|
+
**These funnel response fields report means.** Funnel frames carry
|
|
186
190
|
`avg_time` and `avg_time_from_start` (seconds; not monotonic across steps — each is
|
|
187
191
|
over that step's own survivors). A dungeon knob expressed as a median ratio will not
|
|
188
192
|
reproduce: Square's designed 13x median gap measured 1.95x as a mean over a 30-day
|
|
189
|
-
window.
|
|
193
|
+
window. A **speed curve** can supplement the report: run the same funnel at 1/3/7/14-day
|
|
190
194
|
conversion windows and read what share of each segment's eventual conversions had
|
|
191
|
-
landed by then.
|
|
195
|
+
landed by then. Preserve the requested real report and its statistic. A speed
|
|
196
|
+
curve cannot silently replace its acceptance check.
|
|
192
197
|
|
|
193
198
|
### 3. Auto-detect the data window
|
|
194
199
|
|
|
@@ -260,10 +265,14 @@ filters have to move into `show[i].behavior.filters`.
|
|
|
260
265
|
- **structure** — every registered entity still exists.
|
|
261
266
|
- **stories** — re-measure each hook effect live and compare to the knob.
|
|
262
267
|
|
|
263
|
-
Report MATCH / DIRECTIONAL / MISS per story
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
268
|
+
Report MATCH / DIRECTIONAL / MISS / INSUFFICIENT_EVIDENCE per story. Exit non-zero
|
|
269
|
+
on a MISS or unresolved evidence gap; insufficient eligible users or converters
|
|
270
|
+
are neither a pass nor a measured miss. Use each story's declared tolerance.
|
|
271
|
+
Document report-semantic differences separately rather than widening tolerances.
|
|
272
|
+
Preserve explicit counting options: local totals default to `reentry: false`.
|
|
273
|
+
Derive local sessions from the full user stream before HPC partitioning; local
|
|
274
|
+
defaults are UTC, 30-minute idle timeout, and 24-hour maximum. Project exclusions
|
|
275
|
+
and timezone variants remain outside the source-derived proof.
|
|
267
276
|
|
|
268
277
|
### 7. Report
|
|
269
278
|
|
|
@@ -313,8 +322,9 @@ every build. Prefer one report with N metrics wherever the chart allows it.
|
|
|
313
322
|
**`ws._api_client` is lazy** — None immediately after construction. Force it via
|
|
314
323
|
`ws._get_api_client()` before patching request headers.
|
|
315
324
|
|
|
316
|
-
**Idempotency.** Dashboards:
|
|
317
|
-
look up by name and reuse.
|
|
325
|
+
**Idempotency.** Dashboards: create the replacement first, then retire only the
|
|
326
|
+
recorded old ids after confirmation. Everything else: look up by name and reuse.
|
|
327
|
+
Re-running must never duplicate.
|
|
318
328
|
|
|
319
329
|
## Commands
|
|
320
330
|
|
|
@@ -10,7 +10,9 @@ Base URL: `https://mixpanel-power-tools-api-lmozz6xkha-uc.a.run.app`
|
|
|
10
10
|
|
|
11
11
|
## Auth model — read this first
|
|
12
12
|
|
|
13
|
-
- **GET any endpoint path = documentation, no auth.**
|
|
13
|
+
- **GET any endpoint path = documentation, no auth.** For authorized online work,
|
|
14
|
+
GET an unfamiliar endpoint's docs before use. During offline verification, use
|
|
15
|
+
local references and report unavailable documentation; make no network calls.
|
|
14
16
|
- **POST = execute.** `Authorization: Bearer <oauth-token>` (employee OAuth from repo `.env` `BEARER_TOKEN`, or a customer's OAuth token) or `Basic base64(service_acct:secret)`.
|
|
15
17
|
- **Customer OAuth tokens are accepted** (verified 2026-07-06): `/auth` and `/macro/get-schema` work with a customer token on projects that token can access. `ai_endpoints_allowed: false` for non-employees — the `ai-*` family stays employee-only.
|
|
16
18
|
- **Every POST body** should include `client_id: "dungeon-master"` and `region` (`US` default).
|
|
@@ -62,7 +64,9 @@ warehouse skill's manual-setup fallback; other errors must surface.
|
|
|
62
64
|
Its synthetic ids are series keys, so schema snapshots and dashboard counts must
|
|
63
65
|
distinguish cadence telemetry from user `events[]`. It is not a warehouse source.
|
|
64
66
|
|
|
65
|
-
GET the path for full docs. Full list: GET `/`
|
|
67
|
+
During authorized online work, GET the path for full docs. Full list: GET `/`
|
|
68
|
+
and GET `/macro`. Preserve explicit report options when comparing local and live
|
|
69
|
+
queries; see the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md).
|
|
66
70
|
|
|
67
71
|
**crud** — 187 endpoints. GET `/crud` for the full list.
|
|
68
72
|
|
|
@@ -44,6 +44,25 @@ a later success. Record exit codes, pass/skip counts, and the tested commit/diff
|
|
|
44
44
|
The smoke script discovers vertical dungeons only. Verify changed technical or
|
|
45
45
|
customer fixtures separately with the existing verification runners.
|
|
46
46
|
|
|
47
|
+
`npm test` excludes alignment. Run its independent gates separately:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
node tests/alignment/run.mjs
|
|
51
|
+
node tests/alignment/run.mjs --sweep --timeout-ms=600000
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Run the sweep when required by release scope. Both commands enforce macOS network
|
|
55
|
+
denial and a ten-minute maximum. If editor tasks are unavailable, use these
|
|
56
|
+
repository commands. If the runner or platform is unavailable, report NOT RUN;
|
|
57
|
+
do not substitute unsandboxed execution. Offline checks must use installed
|
|
58
|
+
dependencies, OS network denial, and empty global/setup-file lists. The default
|
|
59
|
+
suite above is not the offline audit route: do not run pruning setup, customer
|
|
60
|
+
evaluations, imports, or industry-generation scripts during an offline docs audit.
|
|
61
|
+
For a skills-only audit, run only `tests/e2e/skills-contract.test.js` with
|
|
62
|
+
`configFile: false`, that exact include, `globals: true`, `globalSetup: []`,
|
|
63
|
+
`setupFiles: []`, `fileParallelism: false`, and `sequence.concurrent: false`
|
|
64
|
+
through programmatic Vitest under `sandbox-exec` network denial.
|
|
65
|
+
|
|
47
66
|
For generation changes, require the event-stream determinism test. Compare runs
|
|
48
67
|
with the same seed, pinned window, and `concurrency: 1`; strip only `insert_id`.
|
|
49
68
|
Use sequential test cases because the RNG is shared. Include warehouse noise in
|
|
@@ -59,12 +78,15 @@ warehouse determinism checks. Never accept an unrun or failing determinism gate.
|
|
|
59
78
|
- Check README, HOOKS, type comments, changelog, and guide against actual code.
|
|
60
79
|
Do not carry stale test counts, old skill paths, or historical operational
|
|
61
80
|
failures forward as current release claims.
|
|
62
|
-
-
|
|
81
|
+
- Audit the documented order of the complete handoff: author, optional hooks, verify, provision,
|
|
63
82
|
generate/import, optional `/warehouse-metrics`, then headless build.
|
|
83
|
+
Offline release verification does not execute provisioning, imports, warehouse
|
|
84
|
+
deployment, or headless builds.
|
|
64
85
|
- Distinguish user events, identity-less `standaloneEvents`, and warehouse rows.
|
|
65
86
|
Synthetic IDs are not people. Standalone hooks return retained records;
|
|
66
87
|
warehouse hooks mutate rows and ignore returns. Preserve deployment artifacts.
|
|
67
|
-
- If API integrations changed, read
|
|
88
|
+
- If API integrations changed, read local endpoint docs and handback evidence
|
|
89
|
+
during offline checks. Fetch endpoint docs only during authorized online work.
|
|
68
90
|
Distinguish docs discovery, mocked execution, dry-run, and live verification.
|
|
69
91
|
Live writes require explicit authorization. Never claim an endpoint GET proves
|
|
70
92
|
authenticated write permission or end-to-end deployment.
|
|
@@ -85,6 +107,9 @@ whole under `plans/archived/`; leave active or ambiguous work in place.
|
|
|
85
107
|
|
|
86
108
|
Report blockers first, then evidence, skipped checks, compatibility changes,
|
|
87
109
|
and remaining operational limits. Do not fix unrelated failures or weaken tests.
|
|
110
|
+
Apply the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md):
|
|
111
|
+
report source-derived scope and remaining gaps, separate insufficient evidence
|
|
112
|
+
from passes and measured failures, and preserve actual deployment reports.
|
|
88
113
|
|
|
89
114
|
Only when explicitly requested: inspect all staged files for secrets, commit the
|
|
90
115
|
authorized changes, push the feature branch, create or reuse its PR, and inspect
|
|
@@ -8,7 +8,7 @@ effort: max
|
|
|
8
8
|
|
|
9
9
|
# Verify Dungeon
|
|
10
10
|
|
|
11
|
-
Verify a dungeon at
|
|
11
|
+
Verify a dungeon at its configured scale: run `scripts/verify-stories.mjs`, validate schema, and review report semantics and evidence sufficiency for every story. Investigate misses, bespoke SQL, and semantic gaps, then write one consolidated `hook-results.md`.
|
|
12
12
|
|
|
13
13
|
**Dungeon file(s):** `$ARGUMENTS` — single path, multiple space-separated paths, or glob pattern. In batch mode, process each dungeon sequentially through Steps 1–5, then write one consolidated report in Step 7. Use a unique `name` prefix per dungeon (e.g., `verify-fintech`, `verify-gaming`) so output files don't collide.
|
|
14
14
|
|
|
@@ -16,6 +16,7 @@ Verify a dungeon at full scale: run the story runner (`scripts/verify-stories.mj
|
|
|
16
16
|
|
|
17
17
|
Load these on demand:
|
|
18
18
|
|
|
19
|
+
- [references/alignment-contract.md](references/alignment-contract.md) - required 1.8.1 proof contract: independent report specification, neutral controls, sufficient populations, and source-derived limits.
|
|
19
20
|
- [references/counting-semantics.md](references/counting-semantics.md) — Mixpanel counting rules, when to use the emulator vs DuckDB, emulator analysis types, identity-model verification, time-series, common gotchas.
|
|
20
21
|
- [references/sql-recipes.md](references/sql-recipes.md) — every DuckDB query template (schema validation, identity / experiment invariants, hook archetype queries, pitfalls, TTC verification, dataset-window computation).
|
|
21
22
|
- [references/report-format.md](references/report-format.md) — single + multi-dungeon report templates, per-hook detail block, verdict criteria, query log format, mandatory verification SQL file for user dungeons.
|
|
@@ -41,11 +42,18 @@ For each hook/pattern, catalog:
|
|
|
41
42
|
- Which output file the signal lives in (user events, users, groups, standalone cadence shards, warehouse tables)
|
|
42
43
|
- Mixpanel report instructions — flag missing/vague ones for the report
|
|
43
44
|
|
|
45
|
+
Before running, write the independent report specification from the intended
|
|
46
|
+
analysis, then compare the assertion's actual options and defaults with it.
|
|
47
|
+
Plan paired baselines and neutral controls using the shared contract.
|
|
48
|
+
|
|
44
49
|
### Step 2: Run the dungeon
|
|
45
50
|
|
|
46
51
|
The runner already exists at `scripts/verify-runner.mjs`. Use it — do NOT recreate.
|
|
47
52
|
|
|
48
|
-
**
|
|
53
|
+
**Run at full fidelity for acceptance.** `--small` is for troubleshooting only.
|
|
54
|
+
Configured scale does not guarantee sufficient eligible users or converters.
|
|
55
|
+
Report `INSUFFICIENT_EVIDENCE` when actual independent populations or observation
|
|
56
|
+
horizons cannot support the claim; do not classify sparse data as a measured miss.
|
|
49
57
|
|
|
50
58
|
```bash
|
|
51
59
|
node scripts/verify-runner.mjs <dungeon-path> <run-name>
|
|
@@ -106,14 +114,19 @@ without those shards, run the same SQL directly and report that CLI limitation.
|
|
|
106
114
|
A standalone-only config without stories does not receive an automatic cadence
|
|
107
115
|
audit; perform the explicit schema, tick, dimension, and identity checks below.
|
|
108
116
|
|
|
109
|
-
**Verdicts are computed, not judged.**
|
|
117
|
+
**Verdicts are computed, not judged.** `minCohort` caps verdicts when supported
|
|
118
|
+
selected-row population fields fall below the floor. It does not validate every
|
|
119
|
+
denominator or establish full proof. Check actual eligible users/converters and
|
|
120
|
+
mature observation windows separately. See [report-format.md "Verdict criteria"](references/report-format.md#verdict-criteria-5-tier).
|
|
110
121
|
|
|
111
|
-
**What
|
|
112
|
-
1. **
|
|
113
|
-
2.
|
|
114
|
-
3. **
|
|
122
|
+
**What to review after this step:**
|
|
123
|
+
1. **Every story, including passing targets** - check its options against the independent report specification and inspect neutral controls and population sufficiency. An inherited default can verify the wrong report.
|
|
124
|
+
2. **Stories below STRONG** (WEAK / NONE / INVERSE) - root-cause via Step 5. Distinguish insufficient evidence from an effect failure; never relax a target to match output.
|
|
125
|
+
3. **`duckdb`-type assertions** - the runner executes them but cannot validate the bespoke query's report semantics.
|
|
126
|
+
4. **Dungeons without a `stories` export** - legacy fallback: Step 5 for every documented hook.
|
|
115
127
|
|
|
116
|
-
|
|
128
|
+
Preserve computed verdicts in `hook-results.md`. Add separate semantic and evidence
|
|
129
|
+
statuses; a passing runner verdict alone does not establish acceptance.
|
|
117
130
|
|
|
118
131
|
### Step 4: Validate schema (BEFORE per-hook checks)
|
|
119
132
|
|
|
@@ -129,14 +142,18 @@ group keys, value column, and declared extras. Never apply user superProp or
|
|
|
129
142
|
identity requirements to cadence records or warehouse rows.
|
|
130
143
|
|
|
131
144
|
**Schema verdicts:**
|
|
132
|
-
- **SCHEMA-PASS**
|
|
133
|
-
- **SCHEMA-FAIL**
|
|
145
|
+
- **SCHEMA-PASS** - every output property is declared or a recognized enabled engine/SDK field for this stream.
|
|
146
|
+
- **SCHEMA-FAIL** - any undeclared property, even at 100% coverage. Uniform flag stamping still violates schema-first authorship.
|
|
147
|
+
|
|
148
|
+
If the runtime validator permits uniform enrichment, record that result separately
|
|
149
|
+
and enforce this stricter authorship gate explicitly.
|
|
134
150
|
|
|
135
151
|
If any event type has SCHEMA-FAIL, flag it prominently in the report header with specific remediation: which hook line adds the property and how to remove it while preserving the intended pattern.
|
|
136
152
|
|
|
137
153
|
### Step 5: Investigate failures (and legacy no-stories dungeons)
|
|
138
154
|
|
|
139
|
-
Applies
|
|
155
|
+
Applies to Step 3's targets: failed stories, semantic or evidence gaps (including
|
|
156
|
+
passing targets), bespoke SQL, and dungeons with no `stories` export.
|
|
140
157
|
|
|
141
158
|
**Decision: emulator vs DuckDB**
|
|
142
159
|
|
|
@@ -167,7 +184,9 @@ The emulator now covers lifecycle, flows, sessions, event breakdowns, formulas,
|
|
|
167
184
|
|
|
168
185
|
**Hand-written DuckDB funnel SQL diverges from Mixpanel — never hand-roll.** If you find yourself writing `WITH step1 AS ..., step2 AS ...` for a funnel, STOP — use `emulateBreakdown` with `funnelFrequency` instead.
|
|
169
186
|
|
|
170
|
-
For emulator
|
|
187
|
+
For emulator options, emitted-evidence identity maps, profile segments, and
|
|
188
|
+
time-series breakdown via `timeBucket`, see [counting-semantics.md](references/counting-semantics.md).
|
|
189
|
+
For DuckDB diagnostics and standard checks, see [sql-recipes.md](references/sql-recipes.md).
|
|
171
190
|
|
|
172
191
|
**Always run for every dungeon** (before per-hook checks):
|
|
173
192
|
- Standard identity-model invariants (stitch counts, pre-existing user stamping) when the dungeon uses the identity model
|
|
@@ -200,7 +219,7 @@ Format and conventions: see [report-format.md "Query log format"](references/rep
|
|
|
200
219
|
|
|
201
220
|
### Step 7: Write `hook-results.md`
|
|
202
221
|
|
|
203
|
-
Write to `dungeons/user/<name>/hook-results.md` for a user dungeon, else `./research/hook-results.md`.
|
|
222
|
+
Write to `dungeons/user/<name>/hook-results.md` for a user dungeon, else `./research/hook-results.md`. Preserve the runner's JSON verdicts alongside independent semantic and evidence statuses, eligible populations, neutral controls, and source-derived scope. Add details for Step 3's investigation targets. Use [report-format.md](references/report-format.md):
|
|
204
223
|
- Single-dungeon report structure
|
|
205
224
|
- Multi-dungeon report structure (when batch mode)
|
|
206
225
|
- Per-hook detail block
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# 1.8.1 verification contract
|
|
2
|
+
|
|
3
|
+
## 1.8.2 live verification update
|
|
4
|
+
|
|
5
|
+
The [1.8.2 guide](../../../../docs/guides/1.8.2-upgrade-guide.md) and repository
|
|
6
|
+
`tests/alignment/live/REPORT.md` extend this contract with retained live comparisons.
|
|
7
|
+
The 1.8.1 audit below remains historical. It used selected source-derived contracts;
|
|
8
|
+
1.8.2 also imports isolated datasets and compares native Mixpanel query results.
|
|
9
|
+
Public options and defaults remain unchanged.
|
|
10
|
+
|
|
11
|
+
Automatic verifier identity now derives from emitted both-ID events. Explicit
|
|
12
|
+
`buildIdentityMap(profiles)` overrides retain their compatibility behavior.
|
|
13
|
+
List-valued held-property expansion and supported HPC/session report combinations
|
|
14
|
+
have source and live fixtures. Non-UTC/DST and custom session-exclusion variants
|
|
15
|
+
still need separate verification.
|
|
16
|
+
|
|
17
|
+
Import success is not query readiness. Reconcile totals and identity before
|
|
18
|
+
accepting comparisons. Historical anonymous events can join later without another
|
|
19
|
+
import. Every live test must use a run property and disjoint run identities.
|
|
20
|
+
Retain incomplete responses and do other work before a bounded retry.
|
|
21
|
+
|
|
22
|
+
Distinguish raw per-user count histograms, UTC calendar activity, and rolling
|
|
23
|
+
Frequency/Addiction. The default local calendar algorithm is not a universal
|
|
24
|
+
Mixpanel frequency rule. Lifetime attribution endpoints require conversion/lookback
|
|
25
|
+
eligibility; conflicting values with tied timestamps are not stable across ingest.
|
|
26
|
+
`minCohort` now enforces conservative independent-user lower bounds, including
|
|
27
|
+
custom callbacks. Unknown evidence caps a passing verdict instead of counting rows.
|
|
28
|
+
|
|
29
|
+
Use this contract when authoring, verifying, provisioning, or presenting a dungeon.
|
|
30
|
+
The [1.8.1 guide](../../../../docs/guides/1.8.1-upgrade-guide.md) describes the
|
|
31
|
+
release. The local repository's `tests/alignment/REPORT.md` and
|
|
32
|
+
`tests/alignment/API-COMPATIBILITY.md` retain the audit evidence; those test
|
|
33
|
+
artifacts are not shipped in npm.
|
|
34
|
+
|
|
35
|
+
## Define the report before measuring
|
|
36
|
+
|
|
37
|
+
Write an independent report specification before choosing assertions. Record
|
|
38
|
+
events, filters, identity rules, cohort definition, denominator, statistic,
|
|
39
|
+
window, timezone, ordering, reentry, session settings, and property selection.
|
|
40
|
+
Compare every assertion with that specification, including passing assertions.
|
|
41
|
+
An inherited default can pass the wrong report. Do not replace funnel TTC with
|
|
42
|
+
a numeric property, a mean with a median, or conversion with event volume to
|
|
43
|
+
rescue a failed story. Supplementary diagnostics must keep their own labels.
|
|
44
|
+
|
|
45
|
+
Local checks prove selected source-derived contracts, not universal Mixpanel
|
|
46
|
+
parity. The 1.8.1 audit read analytics source without compiling or executing it.
|
|
47
|
+
That audit did not run live differential queries. Macro/soup interactions, table surfaces,
|
|
48
|
+
arbitrary hooks, parallel execution, and several counting variants remain gaps.
|
|
49
|
+
|
|
50
|
+
## Separate a measured effect from enough evidence
|
|
51
|
+
|
|
52
|
+
Retain the story runner's computed verdict. Report semantic correctness and
|
|
53
|
+
evidence sufficiency separately. Full-fidelity scale does not guarantee enough
|
|
54
|
+
independent eligible users, mature retention cohorts, or converters. Inspect the
|
|
55
|
+
actual denominator on each side; event count is not independent user count.
|
|
56
|
+
`minCohort` checks supported selected-row population fields, not every bespoke
|
|
57
|
+
denominator. Meeting it alone is not full proof. Label unresolved acceptance
|
|
58
|
+
`INSUFFICIENT_EVIDENCE`; do not count it as a pass or a measured effect failure.
|
|
59
|
+
|
|
60
|
+
For engineered effects, measure a paired baseline with the same seed, window,
|
|
61
|
+
and report options. Add a neutral control (factor one, zero injection, or hook
|
|
62
|
+
disabled) and an unaffected control cohort where applicable. A factor-one
|
|
63
|
+
control must be measured, not inferred from two differently modified groups.
|
|
64
|
+
Report baseline, treatment, eligible users/converters, and paired lift. Keep
|
|
65
|
+
targets and tolerances fixed before the run; disclose any revised criterion
|
|
66
|
+
and preserve its original failed evidence.
|
|
67
|
+
|
|
68
|
+
## Preserve the local counting boundaries
|
|
69
|
+
|
|
70
|
+
- `Funnel.reentry` is verifier-only. Generated repetitions depend on usage
|
|
71
|
+
selection and event budget. Local `reentry` defaults to `false`, including
|
|
72
|
+
`countMode: 'totals'`; repeated histories require explicit `reentry: true`.
|
|
73
|
+
- Funnel histories include ordered restart/shared-edge behavior and inclusive
|
|
74
|
+
2000ms completion grace. TTC uses completed histories under the requested
|
|
75
|
+
options, not independent first occurrences of each event. First/last segment
|
|
76
|
+
selection merges reached-step properties with defined fallback values;
|
|
77
|
+
explicit-step selection is separate.
|
|
78
|
+
- Derive sessions from the full resolved user stream before HPC partitioning.
|
|
79
|
+
Local defaults are UTC, a 30-minute idle timeout, and a 24-hour maximum.
|
|
80
|
+
`session_id` is a diagnostic or explicit legacy mode, not the default truth.
|
|
81
|
+
Non-UTC projects and project-specific hidden/excluded session events remain
|
|
82
|
+
outside the live proof. List-valued HPC has 1.8.2 source and live fixtures.
|
|
83
|
+
- Profile device pools alone establish no identity mapping evidence. The public
|
|
84
|
+
`buildIdentityMap(profiles)` helper retains profile-based compatibility behavior.
|
|
85
|
+
For emitted-identity proof, derive links from valid ordinary both-ID events,
|
|
86
|
+
including later Login events, and pass that explicit `identityMap` to the
|
|
87
|
+
verifier. Earlier device-only rows can resolve retrospectively. Report ID
|
|
88
|
+
conflicts and validation limits; generator auth policy is not ingestion parity.
|
|
89
|
+
- `applyPathBias` is append-only. `share` selects eligible injection recipients;
|
|
90
|
+
it does not promise exact visible branch share. Measure appended payloads,
|
|
91
|
+
competing events, observed path share, and paired lift separately.
|
|
92
|
+
- `applySessionShape` preserves records. Pass known `datasetStart`/`datasetEnd`
|
|
93
|
+
bounds when clipping matters. Explicit bounds reject insufficient capacity
|
|
94
|
+
before mutation; unbounded placement can cross an unknown dataset end.
|
|
95
|
+
- `retentionCurve` weights active-day plans. Budget, funnel spill, and observation
|
|
96
|
+
horizon can prevent literal requested retention percentages. Usage anchors do
|
|
97
|
+
not accumulate previous funnel TTC; retries belong to born-user first funnels.
|
|
98
|
+
|
|
99
|
+
## Keep schema and operational gates independent
|
|
100
|
+
|
|
101
|
+
Undeclared output properties fail schema review even at 100% coverage. Accept
|
|
102
|
+
only config-declared properties or recognized engine/SDK fields enabled for that
|
|
103
|
+
stream. A permissive runtime schema summary cannot override this authorship rule.
|
|
104
|
+
Keep user events, standalone cadence, and warehouse schemas separate.
|
|
105
|
+
|
|
106
|
+
Offline verification stops at local evidence and a deployment handoff. Do not
|
|
107
|
+
automatically provision, import, probe endpoints, deploy warehouse tables, or run
|
|
108
|
+
headless builds. Retain exact artifact prefixes and actual deployment reports.
|
|
109
|
+
Dry-run output is a plan, not evidence of a live outcome; preserve existing reports
|
|
110
|
+
before a dry run writes local files. Live work requires separate authorization.
|