@ak--47/dungeon-master 1.8.0 → 1.8.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (33) hide show
  1. package/.claude/skills/analyze-soup/SKILL.md +9 -0
  2. package/.claude/skills/create-dungeon/SKILL.md +35 -34
  3. package/.claude/skills/create-project/SKILL.md +6 -0
  4. package/.claude/skills/headless-build/SKILL.md +21 -11
  5. package/.claude/skills/powertools/SKILL.md +6 -2
  6. package/.claude/skills/release-check/SKILL.md +27 -2
  7. package/.claude/skills/verify-dungeon/SKILL.md +32 -13
  8. package/.claude/skills/verify-dungeon/references/alignment-contract.md +110 -0
  9. package/.claude/skills/verify-dungeon/references/counting-semantics.md +29 -16
  10. package/.claude/skills/verify-dungeon/references/report-format.md +23 -9
  11. package/.claude/skills/verify-dungeon/references/sql-recipes.md +135 -225
  12. package/.claude/skills/warehouse-metrics/SKILL.md +6 -0
  13. package/.claude/skills/write-hooks/SKILL.md +61 -48
  14. package/CHANGELOG.md +82 -0
  15. package/HOOKS.md +105 -47
  16. package/README.md +41 -1
  17. package/docs/guides/1.8.1-upgrade-guide.md +153 -0
  18. package/docs/guides/1.8.2-upgrade-guide.md +110 -0
  19. package/lib/generators/events.js +6 -0
  20. package/lib/generators/funnels.js +16 -0
  21. package/lib/hook-helpers/shape.js +73 -17
  22. package/lib/hook-patterns/attributed-by-source.js +4 -3
  23. package/lib/hook-patterns/funnel-frequency-breakdown.js +4 -7
  24. package/lib/orchestrators/user-loop.js +82 -15
  25. package/lib/verify/counting.js +7 -10
  26. package/lib/verify/emulate-breakdown.js +48 -29
  27. package/lib/verify/funnel-engine.js +93 -40
  28. package/lib/verify/identity.js +32 -9
  29. package/lib/verify/story-runner.js +93 -30
  30. package/lib/verify/verify-dungeon.js +4 -1
  31. package/package.json +1 -1
  32. package/scripts/verify-stories.mjs +3 -3
  33. package/types.d.ts +9 -5
@@ -2,26 +2,18 @@
2
2
 
3
3
  Use DuckDB only for schema integrity, identity-model invariants, experiment invariants, and bespoke patterns the emulator can't express. For funnel / frequency / aggregate / TTC / attribution patterns, use `emulateBreakdown` instead — see [counting-semantics.md](counting-semantics.md).
4
4
 
5
- ## Schema validation queries
6
-
7
- For each unique event type in the output, compare actual columns against the config-declared properties:
5
+ Apply the [1.8.1 verification contract](alignment-contract.md) to every query.
6
+ The SQL examples below are diagnostics for their named measures. They cannot
7
+ replace a different report's acceptance check. Use explicit report options,
8
+ paired baselines, neutral controls, and actual eligible population counts.
8
9
 
9
- ```sql
10
- WITH event_data AS (
11
- SELECT * FROM read_json_auto('./data/<run-name>-EVENTS.json', sample_size=-1)
12
- WHERE event = '<EVENT_TYPE>'
13
- )
14
- SELECT
15
- unnest(map_keys(columns(*))) as col_name,
16
- COUNT(*) as total_events,
17
- COUNT(col_name) FILTER (WHERE col_name IS NOT NULL) as non_null_count,
18
- ROUND(COUNT(col_name) FILTER (WHERE col_name IS NOT NULL) * 100.0 / COUNT(*), 1) as coverage_pct
19
- FROM event_data
20
- GROUP BY col_name
21
- ORDER BY coverage_pct DESC;
22
- ```
10
+ ## Schema validation queries
23
11
 
24
- Or use the programmatic API (`lib/verify/schema-validator.js`):
12
+ For each event type, compare raw record keys against config-declared properties
13
+ and enabled engine/SDK fields. SQL unioned columns lose per-record key presence;
14
+ null coverage alone cannot distinguish an absent key from a declared null.
15
+ Use the programmatic API (`lib/verify/schema-validator.js`) to derive expected
16
+ keys, then explicitly fail any undeclared key in the raw records:
25
17
 
26
18
  ```javascript
27
19
  import { deriveExpectedSchema, validateSchema } from './lib/verify/index.js';
@@ -53,8 +45,11 @@ The expected set of columns per event type is derived from config:
53
45
 
54
46
  For each event type, classify any column present in output but NOT in expected schema:
55
47
 
56
- - **SCHEMA-PASS** Column appears on 100% of events of this type. Uniform enrichment is acceptable.
57
- - **SCHEMA-FAIL** Column appears on <100% of events of this type. This is flag stamping hook conditionally adds a property, creating an inconsistent schema.
48
+ - **SCHEMA-PASS** - every observed key is declared or a recognized enabled engine/SDK field.
49
+ - **SCHEMA-FAIL** - any undeclared key, even at 100% coverage. Uniform enrichment does not bypass schema-first authorship.
50
+
51
+ The runtime summary may permit uniform enrichment. Retain its output, but apply
52
+ the stricter authorship gate separately. Declared nullable fields remain valid.
58
53
 
59
54
  If any event type has SCHEMA-FAIL, flag it prominently and include specific remediation: which hook line adds the property and how to remove it while preserving the intended pattern.
60
55
 
@@ -95,10 +90,12 @@ backfill and `sparse` point-in-time rows before judging counts or time coverage.
95
90
 
96
91
  ### User-event identity checks
97
92
 
98
- Run these for every dungeon that uses the identity model (`isAuthEvent` + `attempts` + `identity.avgDevicePerUser`), BEFORE per-pattern checks:
93
+ Run identity checks whenever the dungeon uses device identity, before per-pattern
94
+ checks. Count configured auth rows as a diagnostic, not as a universal one-stitch
95
+ invariant; ordinary both-ID events can also establish a link.
99
96
 
100
97
  ```sql
101
- -- Stitch event count must match converted-born count, exactly one per user.
98
+ -- Diagnostic counts for a configured auth event, not a mapping proof.
102
99
  WITH e AS (SELECT * FROM read_json_auto('./data/<file>-EVENTS.json')),
103
100
  auth_event AS (SELECT 'Sign Up' AS name) -- name of your isAuthEvent
104
101
  SELECT
@@ -106,17 +103,14 @@ SELECT
106
103
  SUM(CASE WHEN user_id IS NOT NULL AND device_id IS NOT NULL THEN 1 ELSE 0 END) AS stitches,
107
104
  COUNT(DISTINCT CASE WHEN user_id IS NOT NULL THEN user_id END) AS converted_users
108
105
  FROM e WHERE event = (SELECT name FROM auth_event);
109
-
110
- -- Pre-existing users must have user_id on every event (no anon-only records).
111
- WITH e AS (SELECT * FROM read_json_auto('./data/<file>-EVENTS.json')),
112
- u AS (SELECT * FROM read_json_auto('./data/<file>-USERS.json'))
113
- SELECT COUNT(*) AS preexisting_anon_only_records
114
- FROM e JOIN u ON u.distinct_id::VARCHAR = e.user_id::VARCHAR
115
- WHERE u.created < (SELECT MIN(time::TIMESTAMP) FROM e)
116
- AND e.user_id IS NULL;
117
106
  ```
118
107
 
119
- Failures usually indicate incomplete identity-model migration. Flag in report.
108
+ Build the proof map from valid emitted both-ID events, including later ordinary
109
+ Login events, and resolve earlier device-only rows retrospectively. Inspect
110
+ conflicting links. Profile pools are not mapping evidence. For pre-existing
111
+ stamping, use generator ownership evidence and resolved dataset bounds; joining
112
+ on `e.user_id` and then testing it for NULL can never detect missing IDs. If row
113
+ ownership is unavailable in retained artifacts, report that check as unproved.
120
114
 
121
115
  ## Experiment invariants
122
116
 
@@ -267,7 +261,10 @@ FROM events
267
261
  GROUP BY period;
268
262
  ```
269
263
 
270
- ### Retention / Churn (e.g., "early guild joiners retain better")
264
+ ### Activity-span diagnostic (not a retention report)
265
+
266
+ This measures first-to-last activity span. Use `retention` with the report's
267
+ cohort, return event, buckets, and mature horizon for a retention claim.
271
268
  ```sql
272
269
  WITH user_first_event AS (
273
270
  SELECT user_id, MIN(time::TIMESTAMP) as first_seen
@@ -319,29 +316,13 @@ JOIN read_json_auto('./data/verify-dungeon-EVENTS.json') e ON b.user_id = e.user
319
316
  GROUP BY b.is_target_buyer;
320
317
  ```
321
318
 
322
- ### Funnel Conversion by Segment (when emulator can't do it)
323
- ```sql
324
- WITH step1 AS (
325
- SELECT DISTINCT user_id, segment_prop
326
- FROM read_json_auto('./data/verify-dungeon-EVENTS.json')
327
- WHERE event = 'funnel_step_1'
328
- ),
329
- step2 AS (
330
- SELECT DISTINCT user_id
331
- FROM read_json_auto('./data/verify-dungeon-EVENTS.json')
332
- WHERE event = 'funnel_step_2'
333
- )
334
- SELECT
335
- s1.segment_prop,
336
- COUNT(DISTINCT s1.user_id) as started,
337
- COUNT(DISTINCT s2.user_id) as completed,
338
- ROUND(COUNT(DISTINCT s2.user_id) * 100.0 / COUNT(DISTINCT s1.user_id), 2) as conversion_pct
339
- FROM step1 s1
340
- LEFT JOIN step2 s2 ON s1.user_id = s2.user_id
341
- GROUP BY s1.segment_prop;
342
- ```
319
+ ### Funnel conversion by segment
343
320
 
344
- For Mixpanel-accurate funnel verification, prefer `emulateBreakdown({type: 'funnelFrequency'})` see [counting-semantics.md](counting-semantics.md).
321
+ Use `emulateBreakdown({type: 'funnelFrequency'})` with explicit report options.
322
+ A join between users who did A and users who did B does not check ordered
323
+ completion, restart, grace, exclusions, or the conversion window. If the emulator
324
+ cannot express the requested report, record the semantic gap instead of substituting
325
+ an unordered SQL intersection. See [counting-semantics.md](counting-semantics.md).
345
326
 
346
327
  ### Property Distribution Shift
347
328
  ```sql
@@ -366,34 +347,13 @@ WHERE event = 'find treasure' AND treasure_type = 'Shadowmourne Legendary'
366
347
  GROUP BY period;
367
348
  ```
368
349
 
369
- ### Value Magnitude by Behavioral Segment (sessionize derived cohorts)
370
- ```sql
371
- WITH ordered AS (
372
- SELECT *, time::TIMESTAMP as ts,
373
- LAG(time::TIMESTAMP) OVER (PARTITION BY user_id ORDER BY time) as prev_ts
374
- FROM read_json_auto('./data/verify-dungeon-EVENTS.json')
375
- ),
376
- sessions AS (
377
- SELECT user_id,
378
- SUM(CASE WHEN prev_ts IS NULL OR ts - prev_ts > INTERVAL '30 minutes' THEN 1 ELSE 0 END) as session_count
379
- FROM ordered
380
- GROUP BY user_id
381
- ),
382
- segments AS (
383
- SELECT user_id,
384
- CASE WHEN session_count > 20 THEN 'power_user' ELSE 'regular' END as segment
385
- FROM sessions
386
- )
387
- SELECT
388
- seg.segment,
389
- COUNT(*) as purchase_count,
390
- ROUND(AVG(TRY_CAST(e.amount AS DOUBLE)), 2) as avg_amount,
391
- COUNT(DISTINCT seg.user_id) as users
392
- FROM segments seg
393
- JOIN read_json_auto('./data/verify-dungeon-EVENTS.json') e ON seg.user_id = e.user_id
394
- WHERE e.event = 'purchase'
395
- GROUP BY seg.segment;
396
- ```
350
+ ### Value magnitude by session-derived cohort
351
+
352
+ Derive sessions with the verifier from the full resolved user stream before
353
+ filtering events or partitioning by hold-property value. Use all three split
354
+ rules (idle timeout, maximum duration, UTC day change). A timeout-only SQL `LAG`
355
+ query and generator-stamped `session_id` do not establish this contract. Export
356
+ the resulting user/cohort mapping for a SQL value diagnostic if needed.
397
357
 
398
358
  ### Temporal Value Scaling (e.g., 3x amounts on 1st/15th)
399
359
  ```sql
@@ -470,7 +430,9 @@ SELECT 'bots' as metric, count(*) FROM read_json_auto('./data/verify-dungeon-USE
470
430
  UNION ALL SELECT 'null_props', count(*) FROM read_json_auto('./data/verify-dungeon-EVENTS.json') WHERE category IS NULL;
471
431
  ```
472
432
 
473
- Advanced feature patterns should ALWAYS be present (deterministic from config), unlike hooks which may have statistical variance.
433
+ Check whether the relevant population and time window are present before asserting
434
+ an advanced-feature effect. Seeded determinism does not guarantee that a finite
435
+ sample contains every configured segment or outcome.
474
436
 
475
437
  **Deprecated config blocks (silently stripped by validator since 1.4):**
476
438
  `subscription`, `attribution`, `geo`, `features`, `anomalies`. If a
@@ -500,7 +462,8 @@ For pre-existing-only dungeons (`percentUsersBornInDataset: 0`) expect
500
462
  `dropped = 0`.
501
463
 
502
464
  ### 1. SuperProp Consistency
503
- Verify each user has exactly 1 value per superProp:
465
+ For properties declared in `stickyEventProps` or explicitly promised as stable,
466
+ check per-user consistency. Other `superProps` may legitimately vary by event:
504
467
 
505
468
  ```sql
506
469
  SELECT
@@ -515,10 +478,13 @@ FROM (
515
478
  GROUP BY user_id
516
479
  );
517
480
  ```
518
- Verdict: **STRONG** ≥99% consistent, **WEAK** 90-99%, **FAIL** <90%.
481
+ For a strict profile projection contract, investigate every mismatch. Do not
482
+ replace the declared contract with a generic percentage tolerance.
519
483
 
520
484
  ### 2. SuperProp-UserProp Mirror Check
521
- Every superProp key should also appear on user profiles. Compare the dungeon's `superProps` keys against columns in the USERS file. Any superProp not mirrored in `userProps` means the stamping fix is incomplete.
485
+ Only keys promised as profile projections must mirror `userProps`. Matching
486
+ enumerations alone do not guarantee equality; use `stickyEventProps`. Event-only
487
+ context such as an app version does not require a profile mirror.
522
488
 
523
489
  ### 3. Mixpanel Default Property Casing Check
524
490
  The system generates device properties with Mixpanel's standard casing
@@ -536,9 +502,9 @@ intentional opt-in — not a casing conflict.
536
502
 
537
503
  ### 4. funnel-pre Dilution Check
538
504
  For any dungeon with `funnel-pre` conversionRate modifications, verify the actual visible effect:
539
- - A `conversionRate *= 1.5` in funnel-pre typically shows as ~1.02-1.08x in the data (diluted by organic events)
540
- - If observed ratio is <1.1x for a funnel-pre conversionRate hook, verdict is **FAIL** with note: "funnel-pre conversionRate diluted by organic events — migrate to `everything` hook event filtering"
541
- - When the dungeon uses `everything` hook filtering instead, expect the full intended ratio (1.3-1.5x)
505
+ - Compare the declared funnel report on paired baseline and treatment streams.
506
+ - Inspect organic competitors, repeated opportunities, saturation, and eligible populations.
507
+ - Keep the target and report fixed; neither funnel-pre scaling nor everything filtering guarantees a universal ratio.
542
508
 
543
509
  ## Population threshold validation
544
510
 
@@ -555,14 +521,15 @@ GROUP BY segment_column
555
521
  ORDER BY users DESC;
556
522
  ```
557
523
 
558
- **Thresholds (at 1K users):**
559
- - Segment <20 users (<2%): hook signal will be WEAK or invisible — flag as "insufficient population"
560
- - Segment 20-50 users: may show signal but with high variance — note in report
561
- - Segment >50 users: should show clear signal if hook effect ≥1.3x
524
+ Set population floors before measuring, based on the intended effect and report.
525
+ Report actual independent eligible users and converters on both sides. No fixed
526
+ user count guarantees a clear signal for every effect or distribution.
562
527
 
563
528
  ## Statistical caveats
564
529
 
565
- This skill always runs at full fidelity (the dungeon's own scale). At full fidelity, cohorts of all sizes should produce clear signal because the absolute population is large. WEAK or FAIL results at full fidelity indicate a real problem — investigate, do not retry at smaller scale.
530
+ This skill uses the dungeon's configured scale for acceptance. Full fidelity does
531
+ not guarantee enough eligible users, converters, or mature cohorts. Distinguish
532
+ `INSUFFICIENT_EVIDENCE` from measured failure and investigate each accordingly.
566
533
 
567
534
  `--small` mode is a developer-troubleshooting escape hatch on the runner script; verdicts from `--small` runs are unreliable and not permitted in this skill's output.
568
535
 
@@ -614,35 +581,15 @@ GROUP BY b.bucket;
614
581
 
615
582
  When the dungeon doesn't have a natural "per-X" denominator, compute one from the cohort-binning event: `target_events / cohort_event_count`.
616
583
 
617
- **Time-to-convert (funnel-post) verification** compute median A→B time per profile segment:
584
+ **Time-to-convert verification:** use the steps-based `timeToConvert` emulator
585
+ with explicit report options and completed histories. Independent MIN(A)/MIN(B)
586
+ timestamps can mix attempts. Preserve the requested statistic and derive its
587
+ target from the hook, with a measured factor-one or hook-disabled control. Two
588
+ differently modified segments do not constitute a neutral baseline.
618
589
 
619
- ```sql
620
- WITH funnel AS (
621
- SELECT user_id,
622
- MIN(time::TIMESTAMP) FILTER (WHERE event = '<STEP_A>') AS a_time,
623
- MIN(time::TIMESTAMP) FILTER (WHERE event = '<STEP_B>') AS b_time
624
- FROM read_json_auto('./data/<run>-EVENTS.json')
625
- GROUP BY user_id
626
- )
627
- SELECT u.<SEGMENT_KEY>,
628
- COUNT(*) AS users,
629
- ROUND(MEDIAN(EXTRACT(EPOCH FROM (b_time - a_time)) / 60), 2) AS median_min_a_to_b
630
- FROM funnel f
631
- JOIN read_json_auto('./data/<run>-USERS.json') u ON f.user_id = u.distinct_id
632
- WHERE a_time IS NOT NULL AND b_time IS NOT NULL
633
- GROUP BY u.<SEGMENT_KEY>
634
- ORDER BY median_min_a_to_b;
635
- ```
636
-
637
- **Verdict for T2C**:
638
- - Fast segment ≤0.85x baseline → STRONG
639
- - Slow segment ≥1.2x baseline → STRONG
640
- - Both directions visible → STRONG
641
- - One/both missing → check that funnel exists in `funnels:` config and segment property is on `meta.profile`
642
-
643
- **Two-tier T2C interpretation**: When a dungeon has only 2 tiers (e.g. Free vs Paid) the funnel-post hook factor `1.0` branch never fires — both tiers fall into either fast or slow. Pick the slower of the two as the implicit baseline, then verify the faster shows ≤0.85x of it.
644
-
645
- **No-flag verification rule**: NEVER attempt to verify a hook by querying for a flag like `WHERE sweet_spot = true`. If a dungeon has such flags, treat them as a doc bug — the hook should be reworked to hide the cohort behaviorally. The validator's job is to derive cohorts behaviorally.
590
+ **No-flag verification rule:** derive hidden cohorts behaviorally or with the
591
+ declared hash. A flag with a schema-declared default is allowed; an undeclared
592
+ flag fails even when present on every record.
646
593
 
647
594
  ## Drop-event funnel dilution diagnosis
648
595
 
@@ -650,22 +597,27 @@ Many dungeons have hooks of pattern `record.filter(e => e.event === 'X' && chanc
650
597
 
651
598
  **Why:** the hook drops EVENTS not users. A user with 5 step-3 events still appears in the funnel after losing 1-2 events. Funnel completion = `users with ≥1 step-3 event` — only zero-step-3 users disappear from the conversion count, which is rare.
652
599
 
653
- **Correct verification metric:** per-user volume of step-3 events by tier:
600
+ **Supplementary diagnostic:** per-user volume of step-3 events by tier. This
601
+ does not replace the declared funnel completion check:
654
602
 
655
603
  ```sql
656
604
  SELECT u.subscription_tier,
657
605
  COUNT(DISTINCT user_id) AS users,
658
606
  COUNT(*) AS total_step3,
659
607
  ROUND(COUNT(*) * 1.0 / COUNT(DISTINCT user_id), 2) AS per_user
660
- FROM read_json_auto('./data/<run>-EVENTS.json')
661
- WHERE event = '<STEP_3_EVENT>'
608
+ FROM read_json_auto('./data/<run>-EVENTS.json') e
609
+ JOIN read_json_auto('./data/<run>-USERS.json') u
610
+ ON e.user_id::VARCHAR = u.distinct_id::VARCHAR
611
+ WHERE e.event = '<STEP_3_EVENT>'
662
612
  GROUP BY u.subscription_tier
663
613
  ORDER BY per_user DESC;
664
614
  ```
665
615
 
666
616
  Expected: paid tier ~1.5x non-paid per_user (matches 30% drop on non-paid → paid keeps 100%, non-paid keeps 70%, ratio 1/0.7 = 1.43x).
667
617
 
668
- If funnel completion gap <5pt but per_user gap ≥30%, the hook IS firing the doc just points to the wrong metric. Mark STRONG, recommend doc redirect to per-user query.
618
+ A volume gap can show the mutation fired while the declared conversion story
619
+ still fails. Preserve that failure. Change the report only through an explicit
620
+ story revision, then verify the revised claim with its own controls.
669
621
 
670
622
  ## Subscription tier cohort sizing check
671
623
 
@@ -676,12 +628,10 @@ SELECT subscription_plan, COUNT(*) FROM read_json_auto('./data/<run>-USERS.json'
676
628
  GROUP BY subscription_plan;
677
629
  ```
678
630
 
679
- The default subscription lifecycle (`trialToPayRate=0.30`, `upgradeRate=0.06-0.08`) produces ~85% NULL/Free, ~10-15% Monthly, <2% Annual, ~0% Family at 5K users. Cohorts <50 users will not produce statistically clean signal at any effect size.
680
-
681
- **If annual cohort <50 users:**
682
- - Don't trust per-tier ratios note "cohort too small" in results.md
683
- - Bump `numUsers` up to 5x to enlarge cohorts
684
- - Or recommend dungeon author tighten subscription lifecycle config
631
+ There is no active subscription lifecycle config block. Read the declared
632
+ `userProps` distribution and hook logic, then measure eligible populations.
633
+ If evidence is insufficient, request a larger run or an authorized schema change;
634
+ do not invent lifecycle defaults or silently relax the acceptance threshold.
685
635
 
686
636
  ## Per-day normalization for time-window hooks
687
637
 
@@ -701,41 +651,40 @@ For any spike/burst hook with a tight day window, ALWAYS normalize by window len
701
651
 
702
652
  ## Determinism check (optional confidence test)
703
653
 
704
- The pinned `datasetStart`/`datasetEnd` window plus seeded RNG produces near-bit-exact output across runs. To confirm no NEW non-determinism crept in (e.g. wall-clock leak in a hook):
654
+ For seeded generation, pin `datasetStart`/`datasetEnd` and `concurrency: 1`.
655
+ Use isolated sequential runs and strip only `insert_id` before comparing events:
705
656
 
706
657
  1. Run a previously-passing dungeon a second time.
707
- 2. Compare `eventCount` in the runner's JSON output should match within ~0.5%.
708
- 3. Re-run the hook's headline query and verify ratios match to 2 decimals.
658
+ 2. Require identical event counts, timestamps, ordering, and seeded property values.
659
+ 3. Require identical report output under the same explicit options.
709
660
 
710
- **Tolerance note**: most vertical dungeons produce bit-exact event counts across runs, but a few show <0.5% variance from RNG-state interactions. Variance at this scale does NOT affect hook signal direction or magnitude — all signals remain stable across runs. Treat <1% event-count drift as acceptable; investigate only if drift exceeds 1% OR a hook ratio swings meaningfully (>10% relative change between runs).
711
-
712
- If event count differs by >1% OR a hook ratio swings sharply, the hook has a fresh non-determinism source (typically `dayjs()`, `Date.now()`, `Math.random()`, or stale module-level state). Fix before continuing.
661
+ Investigate differences, including wall-clock calls, unseeded RNG, and stale
662
+ module state. Document known unseeded property functions separately; do not
663
+ accept a generic percentage drift as determinism proof.
713
664
 
714
665
  ## Critical time-window verification pattern
715
666
 
716
- Many dungeons use relative time windows (e.g., "spike on days 75-85"). The post-shift dataset start is exposed to hooks as `meta.datasetStart` (unix seconds). For DuckDB verification, use the same anchor:
667
+ Use the resolved dataset bounds, also exposed as `meta.datasetStart` and
668
+ `meta.datasetEnd` in unix seconds. Generation occurs inside this window without
669
+ a post-generation shift. Observed event extrema do not reconstruct configured
670
+ bounds, especially for sparse or partially observed windows:
717
671
 
718
672
  ```sql
719
- -- WRONG: uses MIN(time) which is up to 30 days BEFORE dataset start (pre-existing user spread)
720
- SELECT *, EXTRACT(EPOCH FROM (time::TIMESTAMP - (SELECT MIN(time::TIMESTAMP) FROM events))) / 86400 as day_in
721
- FROM events;
722
-
723
- -- RIGHT: anchor to MAX(time) - num_days, which is the post-shift dataset start
724
673
  WITH bounds AS (
725
- SELECT MAX(time::TIMESTAMP) - INTERVAL 'NUM_DAYS' day as datasetStart
726
- FROM events
674
+ SELECT TIMESTAMP '<RESOLVED_DATASET_START_UTC>' as datasetStart
727
675
  )
728
676
  SELECT *, EXTRACT(EPOCH FROM (e.time::TIMESTAMP - b.datasetStart)) / 86400 as day_in
729
677
  FROM events e, bounds b;
730
678
  ```
731
679
 
732
- Pre-existing users have events for up to 30 days BEFORE the dataset start (`preExistingSpread: 'uniform'` default in macro). MIN(time) reflects those pre-existing events, not the dataset window. Always anchor to MAX(time) - num_days for "day in dataset" calculations.
680
+ Pre-existing profile creation can precede the window; generated user events still
681
+ stay inside the resolved bounds. Never substitute `MAX(time) - numDays` for them.
733
682
 
734
683
  ## TTC hook verification — two approaches
735
684
 
736
685
  TTC hooks come in two forms. Use the matching verification approach:
737
686
 
738
- ### Approach 1: Property-Scaling TTC (preferred — produces NAILED verdicts)
687
+ ### Approach 1: Numeric timing-property report
739
688
 
740
689
  The hook scales a timing PROPERTY (e.g., `response_time_mins *= 0.67`) by segment. Verification is trivial:
741
690
 
@@ -748,49 +697,33 @@ WHERE event IN ('alert acknowledged', 'alert resolved')
748
697
  GROUP BY segment ORDER BY avg_response;
749
698
  ```
750
699
 
751
- This consistently produces exact matches to the hook factors (e.g., 0.67x target 0.665x measured).
700
+ This proves a property aggregate only. Measure paired baseline/treatment and a
701
+ factor-one control; raw segment ratios can reflect different starting distributions.
752
702
 
753
- ### Approach 2: Timestamp-Shifting TTC (use when no timing property exists)
703
+ ### Approach 2: Funnel timestamp TTC report
754
704
 
755
- The hook shifts event timestamps in the everything hook using `scaleFunnelTTC()` or manual gap scaling. Verification requires a **bound-sequence query** — never use the lazy MIN→MIN proxy:
756
-
757
- ```sql
758
- -- WRONG: lazy MIN→MIN proxy (mixes events from different funnel passes)
759
- SELECT user_id, MIN(a.time) AS start, MIN(b.time) AS end ...
760
-
761
- -- RIGHT: bound-sequence (first A, then first B AFTER that A)
762
- WITH steps AS (
763
- SELECT user_id, event, time::TIMESTAMP AS t
764
- FROM events WHERE event IN ('step_a', 'step_b', 'step_c')
765
- ),
766
- funnel AS (
767
- SELECT DISTINCT ON (a.user_id) a.user_id, a.t AS start_t,
768
- (SELECT MIN(t) FROM steps c
769
- WHERE c.user_id = a.user_id AND c.event = 'step_c' AND c.t > a.t) AS end_t
770
- FROM steps a WHERE a.event = 'step_a'
771
- ORDER BY a.user_id, a.t
772
- )
773
- SELECT segment,
774
- COUNT(*) AS users,
775
- ROUND(MEDIAN(EXTRACT(EPOCH FROM (end_t - start_t)) / 60), 1) AS median_min
776
- FROM funnel JOIN users USING (user_id)
777
- WHERE end_t IS NOT NULL
778
- GROUP BY segment ORDER BY median_min;
779
- ```
780
-
781
- The bound-sequence pattern finds the first A per user, then the first C strictly after that A. This matches how the everything hook operates and typically produces STRONG verdicts. The lazy MIN→MIN proxy produces flat or inverted results because it grabs unrelated events from different funnel passes.
705
+ Use steps-based `timeToConvert` on completed histories with the report's window,
706
+ order, filters, identity, and reentry options. A first-A/next-C query can skip
707
+ required B, miss restarts, or mishandle grace. Compare the same report on baseline,
708
+ treatment, and neutral-control streams. Keep mean and median claims separate.
782
709
 
783
710
  ### Which approach to recommend when writing hooks
784
711
 
785
- Property scaling is strictly better for verification. When creating new TTC hooks, always prefer scaling timing properties (see HOOKS.md principle #15). Reserve timestamp shifting for cases where no numeric timing property exists on the relevant events.
712
+ Choose the hook based on the intended report. Numeric property scaling answers
713
+ a property report; timestamp changes target funnel elapsed time. Ease of
714
+ verification does not authorize replacing one with the other.
786
715
 
787
716
  ### Legacy funnel-post TTC hooks
788
717
 
789
- If a dungeon still uses `funnel-post` for TTC (not yet migrated to `everything`), the effect is only visible in Mixpanel's funnel median TTC report, not in any SQL query. Mark as STRONG by code inspection and recommend migration to property scaling or everything-hook timestamp shifting.
718
+ Measure legacy `funnel-post` effects with the same report contract and controls.
719
+ Never assign STRONG by code inspection. If instance-level mutations fail to move
720
+ completed report histories, retain the miss and investigate competing instances.
790
721
 
791
722
  ## Magic-number cohort sizing — inspect distribution first
792
723
 
793
- Before checking inverted-U signal magnitude, confirm the cohort sizes are statistically meaningful (≥200 in sweet bucket). If cohort is too small, signal magnitude is irrelevant:
724
+ Before checking inverted-U signal magnitude, count independent eligible users in
725
+ each bucket against a predeclared population floor. A floor such as 200 is a
726
+ design choice, not universal statistical proof:
794
727
 
795
728
  ```sql
796
729
  SELECT pn, COUNT(*) FROM (
@@ -799,11 +732,9 @@ SELECT pn, COUNT(*) FROM (
799
732
  ) GROUP BY pn ORDER BY pn LIMIT 20;
800
733
  ```
801
734
 
802
- If 90%+ of users have 0-1 events of X, the dungeon's `sweet=4-7 / over=8+` ranges produce <50 users in sweet → no signal possible. Two fixes:
803
- 1. Bump `numUsers` 5x (cohort grows linearly with users; preserves story)
804
- 2. Recommend the dungeon author redefine ranges to match actual distribution (e.g. `sweet=2-5 / over=6+`)
805
-
806
- Choice depends on whether the JSDoc's stated ranges are load-bearing for the dungeon's narrative ("you need 8+ photos to seem fake" — preserve range, scale up users) or arbitrary ("sweet 4-7" can shift to "sweet 2-5" without losing the story).
735
+ If too few users reach the declared buckets, report insufficient evidence.
736
+ Request a larger run while preserving the story. Changing bucket ranges changes
737
+ the report specification and requires an explicit story revision and new proof.
807
738
 
808
739
  ## Re-run required after hook edits
809
740
 
@@ -811,7 +742,7 @@ If you edit a hook then query the existing data files, you'll get STALE results.
811
742
 
812
743
  ```bash
813
744
  # Keep this run's files for verification and deployment; cleanup needs explicit consent.
814
- node scripts/verify-runner.mjs dungeons/vertical/<NAME>.js verify-<NAME>
745
+ node scripts/verify-runner.mjs dungeons/vertical/<NAME>/<NAME>.js verify-<NAME>-r2
815
746
  # Wait for the {"mode":"full","eventCount":...} JSON to print before querying
816
747
  ```
817
748
 
@@ -839,49 +770,26 @@ The `event` hook receives `meta.datasetStart` as a unix timestamp, but temporal
839
770
 
840
771
  ## Property baseline dilution
841
772
 
842
- When a hook overrides a property value (e.g., `event_type = "plan_upgraded"`), the effect is invisible if the baseline distribution already has a high rate of that value. Example: if `plan_upgraded` is 1 of 5 values (20% baseline), a 40% hook override produces ~28% observed — nearly invisible.
843
-
844
- **Fix:** skew the baseline distribution AWAY from the hook's target value. Make `plan_upgraded` 1 of 8+ values (12.5% baseline), then the 40% hook produces ~48% in the window a clear 4x spike.
845
-
846
- Similarly, if a hook forces `scale_direction = "down"` but the baseline is already 86% "down" (6:1 ratio in config), the hook is invisible. Change the baseline to favor "up" (e.g., 3:1 up:down) so the hook's forced "down" creates a measurable shift.
773
+ If an independent hook forces a value on fraction `q` of eligible events with
774
+ baseline prevalence `p`, expected prevalence is `q + (1 - q) * p`. For `p=0.20`
775
+ and `q=0.40`, that is 0.52. Other targeting and time-window rules need their own
776
+ derivation. Measure the neutral baseline; request an authorized schema change if
777
+ the baseline distribution must change. Do not tune it silently after a miss.
847
778
 
848
779
  ## Computing the dataset window
849
780
 
850
- Dungeons declare their time window in one of three ways — the verifier must derive the actual start/end before writing DuckDB queries:
851
-
852
- | Config shape | How to derive window |
853
- |---|---|
854
- | `datasetStart` + `datasetEnd` | Use directly |
855
- | `numDays` only (no explicit start/end) | `datasetEnd = NOW`, `datasetStart = NOW - numDays` |
856
- | `datasetStart` + `numDays` | `datasetEnd = datasetStart + numDays` |
857
-
858
- The engine always resolves to a `[datasetStart, datasetEnd]` pair internally (see `config-validator.js`). To find the actual window from the OUTPUT data:
859
-
860
- ```sql
861
- SELECT
862
- MAX(time::TIMESTAMP) as datasetEnd,
863
- MAX(time::TIMESTAMP) - INTERVAL '<numDays>' DAY as datasetStart
864
- FROM read_json_auto('./data/verify-X-EVENTS*.json', sample_size=-1);
865
- ```
866
-
867
- Use `datasetStart` (derived above) as the DuckDB anchor for day-in-dataset:
868
-
869
- ```sql
870
- WITH bounds AS (
871
- SELECT MAX(time::TIMESTAMP) - INTERVAL '<numDays>' DAY as ds_start
872
- FROM read_json_auto('./data/verify-X-EVENTS*.json', sample_size=-1)
873
- )
874
- SELECT EXTRACT(EPOCH FROM (e.time::TIMESTAMP - b.ds_start)) / 86400 as day_in
875
- FROM events e, bounds b;
876
- ```
877
-
878
- Do NOT use `MIN(time)` as the anchor — pre-existing users have events up to 30 days before `datasetStart` (from `preExistingSpread: 'uniform'`).
879
-
880
- When the dungeon has explicit `datasetStart` (e.g., `"2026-01-01T00:00:00Z"`), use it directly: `TIMESTAMP '2026-01-01'`. When `numDays` is used without explicit start, derive from MAX(time) as shown above.
781
+ Record the engine's resolved `datasetStart`/`datasetEnd` from the run, including
782
+ derived windows. Use those values in the query shown under "Critical time-window
783
+ verification pattern". If the artifacts do not retain bounds, report the missing
784
+ metadata or regenerate a separately named pinned run. Neither observed MIN/MAX
785
+ nor the current wall clock can recover the original resolved window reliably.
881
786
 
882
787
  ## No flag stamping audit
883
788
 
884
- Hooks must NEVER add cohort flags like `is_whale`, `power_user`, `sweet_spot`, `is_churned`, etc. All cohorts must be derived behaviorally from raw event data. When auditing a dungeon, check the hook for any property assignments that create boolean/categorical flags not defined in the original schema. If found, remove them and rewrite the hook to achieve the same effect through property value mutations, event filtering, or event injection.
789
+ Hooks must never add undeclared flags. A schema-declared flag with a default is
790
+ valid; hidden cohorts can use behavioral or hash definitions. Flag undeclared
791
+ assignments as SCHEMA-FAIL and request an authorized schema declaration or a hook
792
+ rewrite using existing fields. Uniform coverage does not make them acceptable.
885
793
 
886
794
  ## Clone dilution of temporal effects
887
795
 
@@ -900,7 +808,7 @@ If Hook A classifies users by event presence (`events.some(e => e.event === X)`)
900
808
  **Fixes:**
901
809
  1. Require 3+ marker events instead of 1+ (surviving events still identify)
902
810
  2. Accept the verification limitation and note it in the report
903
- 3. Use a metric that doesn't depend on cohort reconstruction (e.g., overall distribution shift instead of cohort comparison)
811
+ 3. Propose a separate distribution diagnostic while retaining the original cohort report's unresolved status; changing the report requires explicit revision.
904
812
 
905
813
  ## Deprecated feature property gaps
906
814
 
@@ -908,4 +816,6 @@ Dungeons using deprecated config blocks (`subscription`, `attribution`, `feature
908
816
 
909
817
  **Diagnosis:** Hook logic references a property that's always NULL/undefined in the output. Check if the property was produced by a deprecated feature.
910
818
 
911
- **Fix:** The dungeon author must add equivalent property generation in the hook itself (via `user` or `everything` hook) or add the property to `superProps`/`userProps` with appropriate values. This is a schema-level fix, not a verification fix — flag it in the report as "NONE: deprecated feature property missing" with the recommended fix.
819
+ **Fix:** Request a schema declaration with a default through `/create-dungeon`
820
+ before hooks assign values. Report the missing field and preserve the runner's
821
+ verdict; adding undeclared property generation inside a hook is not a valid fix.
@@ -36,8 +36,14 @@ node .claude/skills/warehouse-metrics/deploy.mjs <dungeon-path> [--dataset dm_na
36
36
  - `--data-prefix`: explicit run artifact prefix, for example `/tmp/run/warehouse-demo` for `/tmp/run/warehouse-demo-WAREHOUSE-MANIFEST.json`.
37
37
  - `--dry-run`: prints the full `bq ls`, docs probe, metric list, load, source, preview, and create plan without executing commands or requiring credentials. It still writes the SQL files and renders `warehouse/GAPS.md` for review.
38
38
 
39
+ Dry-run is not read-only. Preserve any existing actual deployment report before
40
+ running it. Keep proposed setup gaps separate from recorded live outcomes.
41
+
39
42
  ## Preflight
40
43
 
44
+ - Apply the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md).
45
+ Offline verification stops at local evidence and the deployment handoff. Do not
46
+ automatically run deployment, endpoint probes, previews, or cloud commands.
41
47
  - The dungeon must have passed `/verify-dungeon` and produced local warehouse
42
48
  files with `writeToDisk: true` and `gzip: false`. Use the exact verified
43
49
  `--data-prefix` and its matching `-WAREHOUSE-MANIFEST.json`; preserve all table