@ak--47/dungeon-master 1.8.0 → 1.8.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/analyze-soup/SKILL.md +9 -0
- package/.claude/skills/create-dungeon/SKILL.md +35 -34
- package/.claude/skills/create-project/SKILL.md +6 -0
- package/.claude/skills/headless-build/SKILL.md +21 -11
- package/.claude/skills/powertools/SKILL.md +6 -2
- package/.claude/skills/release-check/SKILL.md +27 -2
- package/.claude/skills/verify-dungeon/SKILL.md +32 -13
- package/.claude/skills/verify-dungeon/references/alignment-contract.md +110 -0
- package/.claude/skills/verify-dungeon/references/counting-semantics.md +29 -16
- package/.claude/skills/verify-dungeon/references/report-format.md +23 -9
- package/.claude/skills/verify-dungeon/references/sql-recipes.md +135 -225
- package/.claude/skills/warehouse-metrics/SKILL.md +6 -0
- package/.claude/skills/write-hooks/SKILL.md +61 -48
- package/CHANGELOG.md +82 -0
- package/HOOKS.md +105 -47
- package/README.md +41 -1
- package/docs/guides/1.8.1-upgrade-guide.md +153 -0
- package/docs/guides/1.8.2-upgrade-guide.md +110 -0
- package/lib/generators/events.js +6 -0
- package/lib/generators/funnels.js +16 -0
- package/lib/hook-helpers/shape.js +73 -17
- package/lib/hook-patterns/attributed-by-source.js +4 -3
- package/lib/hook-patterns/funnel-frequency-breakdown.js +4 -7
- package/lib/orchestrators/user-loop.js +82 -15
- package/lib/verify/counting.js +7 -10
- package/lib/verify/emulate-breakdown.js +48 -29
- package/lib/verify/funnel-engine.js +93 -40
- package/lib/verify/identity.js +32 -9
- package/lib/verify/story-runner.js +93 -30
- package/lib/verify/verify-dungeon.js +4 -1
- package/package.json +1 -1
- package/scripts/verify-stories.mjs +3 -3
- package/types.d.ts +9 -5
|
@@ -2,26 +2,18 @@
|
|
|
2
2
|
|
|
3
3
|
Use DuckDB only for schema integrity, identity-model invariants, experiment invariants, and bespoke patterns the emulator can't express. For funnel / frequency / aggregate / TTC / attribution patterns, use `emulateBreakdown` instead — see [counting-semantics.md](counting-semantics.md).
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
5
|
+
Apply the [1.8.1 verification contract](alignment-contract.md) to every query.
|
|
6
|
+
The SQL examples below are diagnostics for their named measures. They cannot
|
|
7
|
+
replace a different report's acceptance check. Use explicit report options,
|
|
8
|
+
paired baselines, neutral controls, and actual eligible population counts.
|
|
8
9
|
|
|
9
|
-
|
|
10
|
-
WITH event_data AS (
|
|
11
|
-
SELECT * FROM read_json_auto('./data/<run-name>-EVENTS.json', sample_size=-1)
|
|
12
|
-
WHERE event = '<EVENT_TYPE>'
|
|
13
|
-
)
|
|
14
|
-
SELECT
|
|
15
|
-
unnest(map_keys(columns(*))) as col_name,
|
|
16
|
-
COUNT(*) as total_events,
|
|
17
|
-
COUNT(col_name) FILTER (WHERE col_name IS NOT NULL) as non_null_count,
|
|
18
|
-
ROUND(COUNT(col_name) FILTER (WHERE col_name IS NOT NULL) * 100.0 / COUNT(*), 1) as coverage_pct
|
|
19
|
-
FROM event_data
|
|
20
|
-
GROUP BY col_name
|
|
21
|
-
ORDER BY coverage_pct DESC;
|
|
22
|
-
```
|
|
10
|
+
## Schema validation queries
|
|
23
11
|
|
|
24
|
-
|
|
12
|
+
For each event type, compare raw record keys against config-declared properties
|
|
13
|
+
and enabled engine/SDK fields. SQL unioned columns lose per-record key presence;
|
|
14
|
+
null coverage alone cannot distinguish an absent key from a declared null.
|
|
15
|
+
Use the programmatic API (`lib/verify/schema-validator.js`) to derive expected
|
|
16
|
+
keys, then explicitly fail any undeclared key in the raw records:
|
|
25
17
|
|
|
26
18
|
```javascript
|
|
27
19
|
import { deriveExpectedSchema, validateSchema } from './lib/verify/index.js';
|
|
@@ -53,8 +45,11 @@ The expected set of columns per event type is derived from config:
|
|
|
53
45
|
|
|
54
46
|
For each event type, classify any column present in output but NOT in expected schema:
|
|
55
47
|
|
|
56
|
-
- **SCHEMA-PASS**
|
|
57
|
-
- **SCHEMA-FAIL**
|
|
48
|
+
- **SCHEMA-PASS** - every observed key is declared or a recognized enabled engine/SDK field.
|
|
49
|
+
- **SCHEMA-FAIL** - any undeclared key, even at 100% coverage. Uniform enrichment does not bypass schema-first authorship.
|
|
50
|
+
|
|
51
|
+
The runtime summary may permit uniform enrichment. Retain its output, but apply
|
|
52
|
+
the stricter authorship gate separately. Declared nullable fields remain valid.
|
|
58
53
|
|
|
59
54
|
If any event type has SCHEMA-FAIL, flag it prominently and include specific remediation: which hook line adds the property and how to remove it while preserving the intended pattern.
|
|
60
55
|
|
|
@@ -95,10 +90,12 @@ backfill and `sparse` point-in-time rows before judging counts or time coverage.
|
|
|
95
90
|
|
|
96
91
|
### User-event identity checks
|
|
97
92
|
|
|
98
|
-
Run
|
|
93
|
+
Run identity checks whenever the dungeon uses device identity, before per-pattern
|
|
94
|
+
checks. Count configured auth rows as a diagnostic, not as a universal one-stitch
|
|
95
|
+
invariant; ordinary both-ID events can also establish a link.
|
|
99
96
|
|
|
100
97
|
```sql
|
|
101
|
-
--
|
|
98
|
+
-- Diagnostic counts for a configured auth event, not a mapping proof.
|
|
102
99
|
WITH e AS (SELECT * FROM read_json_auto('./data/<file>-EVENTS.json')),
|
|
103
100
|
auth_event AS (SELECT 'Sign Up' AS name) -- name of your isAuthEvent
|
|
104
101
|
SELECT
|
|
@@ -106,17 +103,14 @@ SELECT
|
|
|
106
103
|
SUM(CASE WHEN user_id IS NOT NULL AND device_id IS NOT NULL THEN 1 ELSE 0 END) AS stitches,
|
|
107
104
|
COUNT(DISTINCT CASE WHEN user_id IS NOT NULL THEN user_id END) AS converted_users
|
|
108
105
|
FROM e WHERE event = (SELECT name FROM auth_event);
|
|
109
|
-
|
|
110
|
-
-- Pre-existing users must have user_id on every event (no anon-only records).
|
|
111
|
-
WITH e AS (SELECT * FROM read_json_auto('./data/<file>-EVENTS.json')),
|
|
112
|
-
u AS (SELECT * FROM read_json_auto('./data/<file>-USERS.json'))
|
|
113
|
-
SELECT COUNT(*) AS preexisting_anon_only_records
|
|
114
|
-
FROM e JOIN u ON u.distinct_id::VARCHAR = e.user_id::VARCHAR
|
|
115
|
-
WHERE u.created < (SELECT MIN(time::TIMESTAMP) FROM e)
|
|
116
|
-
AND e.user_id IS NULL;
|
|
117
106
|
```
|
|
118
107
|
|
|
119
|
-
|
|
108
|
+
Build the proof map from valid emitted both-ID events, including later ordinary
|
|
109
|
+
Login events, and resolve earlier device-only rows retrospectively. Inspect
|
|
110
|
+
conflicting links. Profile pools are not mapping evidence. For pre-existing
|
|
111
|
+
stamping, use generator ownership evidence and resolved dataset bounds; joining
|
|
112
|
+
on `e.user_id` and then testing it for NULL can never detect missing IDs. If row
|
|
113
|
+
ownership is unavailable in retained artifacts, report that check as unproved.
|
|
120
114
|
|
|
121
115
|
## Experiment invariants
|
|
122
116
|
|
|
@@ -267,7 +261,10 @@ FROM events
|
|
|
267
261
|
GROUP BY period;
|
|
268
262
|
```
|
|
269
263
|
|
|
270
|
-
###
|
|
264
|
+
### Activity-span diagnostic (not a retention report)
|
|
265
|
+
|
|
266
|
+
This measures first-to-last activity span. Use `retention` with the report's
|
|
267
|
+
cohort, return event, buckets, and mature horizon for a retention claim.
|
|
271
268
|
```sql
|
|
272
269
|
WITH user_first_event AS (
|
|
273
270
|
SELECT user_id, MIN(time::TIMESTAMP) as first_seen
|
|
@@ -319,29 +316,13 @@ JOIN read_json_auto('./data/verify-dungeon-EVENTS.json') e ON b.user_id = e.user
|
|
|
319
316
|
GROUP BY b.is_target_buyer;
|
|
320
317
|
```
|
|
321
318
|
|
|
322
|
-
### Funnel
|
|
323
|
-
```sql
|
|
324
|
-
WITH step1 AS (
|
|
325
|
-
SELECT DISTINCT user_id, segment_prop
|
|
326
|
-
FROM read_json_auto('./data/verify-dungeon-EVENTS.json')
|
|
327
|
-
WHERE event = 'funnel_step_1'
|
|
328
|
-
),
|
|
329
|
-
step2 AS (
|
|
330
|
-
SELECT DISTINCT user_id
|
|
331
|
-
FROM read_json_auto('./data/verify-dungeon-EVENTS.json')
|
|
332
|
-
WHERE event = 'funnel_step_2'
|
|
333
|
-
)
|
|
334
|
-
SELECT
|
|
335
|
-
s1.segment_prop,
|
|
336
|
-
COUNT(DISTINCT s1.user_id) as started,
|
|
337
|
-
COUNT(DISTINCT s2.user_id) as completed,
|
|
338
|
-
ROUND(COUNT(DISTINCT s2.user_id) * 100.0 / COUNT(DISTINCT s1.user_id), 2) as conversion_pct
|
|
339
|
-
FROM step1 s1
|
|
340
|
-
LEFT JOIN step2 s2 ON s1.user_id = s2.user_id
|
|
341
|
-
GROUP BY s1.segment_prop;
|
|
342
|
-
```
|
|
319
|
+
### Funnel conversion by segment
|
|
343
320
|
|
|
344
|
-
|
|
321
|
+
Use `emulateBreakdown({type: 'funnelFrequency'})` with explicit report options.
|
|
322
|
+
A join between users who did A and users who did B does not check ordered
|
|
323
|
+
completion, restart, grace, exclusions, or the conversion window. If the emulator
|
|
324
|
+
cannot express the requested report, record the semantic gap instead of substituting
|
|
325
|
+
an unordered SQL intersection. See [counting-semantics.md](counting-semantics.md).
|
|
345
326
|
|
|
346
327
|
### Property Distribution Shift
|
|
347
328
|
```sql
|
|
@@ -366,34 +347,13 @@ WHERE event = 'find treasure' AND treasure_type = 'Shadowmourne Legendary'
|
|
|
366
347
|
GROUP BY period;
|
|
367
348
|
```
|
|
368
349
|
|
|
369
|
-
### Value
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
sessions AS (
|
|
377
|
-
SELECT user_id,
|
|
378
|
-
SUM(CASE WHEN prev_ts IS NULL OR ts - prev_ts > INTERVAL '30 minutes' THEN 1 ELSE 0 END) as session_count
|
|
379
|
-
FROM ordered
|
|
380
|
-
GROUP BY user_id
|
|
381
|
-
),
|
|
382
|
-
segments AS (
|
|
383
|
-
SELECT user_id,
|
|
384
|
-
CASE WHEN session_count > 20 THEN 'power_user' ELSE 'regular' END as segment
|
|
385
|
-
FROM sessions
|
|
386
|
-
)
|
|
387
|
-
SELECT
|
|
388
|
-
seg.segment,
|
|
389
|
-
COUNT(*) as purchase_count,
|
|
390
|
-
ROUND(AVG(TRY_CAST(e.amount AS DOUBLE)), 2) as avg_amount,
|
|
391
|
-
COUNT(DISTINCT seg.user_id) as users
|
|
392
|
-
FROM segments seg
|
|
393
|
-
JOIN read_json_auto('./data/verify-dungeon-EVENTS.json') e ON seg.user_id = e.user_id
|
|
394
|
-
WHERE e.event = 'purchase'
|
|
395
|
-
GROUP BY seg.segment;
|
|
396
|
-
```
|
|
350
|
+
### Value magnitude by session-derived cohort
|
|
351
|
+
|
|
352
|
+
Derive sessions with the verifier from the full resolved user stream before
|
|
353
|
+
filtering events or partitioning by hold-property value. Use all three split
|
|
354
|
+
rules (idle timeout, maximum duration, UTC day change). A timeout-only SQL `LAG`
|
|
355
|
+
query and generator-stamped `session_id` do not establish this contract. Export
|
|
356
|
+
the resulting user/cohort mapping for a SQL value diagnostic if needed.
|
|
397
357
|
|
|
398
358
|
### Temporal Value Scaling (e.g., 3x amounts on 1st/15th)
|
|
399
359
|
```sql
|
|
@@ -470,7 +430,9 @@ SELECT 'bots' as metric, count(*) FROM read_json_auto('./data/verify-dungeon-USE
|
|
|
470
430
|
UNION ALL SELECT 'null_props', count(*) FROM read_json_auto('./data/verify-dungeon-EVENTS.json') WHERE category IS NULL;
|
|
471
431
|
```
|
|
472
432
|
|
|
473
|
-
|
|
433
|
+
Check whether the relevant population and time window are present before asserting
|
|
434
|
+
an advanced-feature effect. Seeded determinism does not guarantee that a finite
|
|
435
|
+
sample contains every configured segment or outcome.
|
|
474
436
|
|
|
475
437
|
**Deprecated config blocks (silently stripped by validator since 1.4):**
|
|
476
438
|
`subscription`, `attribution`, `geo`, `features`, `anomalies`. If a
|
|
@@ -500,7 +462,8 @@ For pre-existing-only dungeons (`percentUsersBornInDataset: 0`) expect
|
|
|
500
462
|
`dropped = 0`.
|
|
501
463
|
|
|
502
464
|
### 1. SuperProp Consistency
|
|
503
|
-
|
|
465
|
+
For properties declared in `stickyEventProps` or explicitly promised as stable,
|
|
466
|
+
check per-user consistency. Other `superProps` may legitimately vary by event:
|
|
504
467
|
|
|
505
468
|
```sql
|
|
506
469
|
SELECT
|
|
@@ -515,10 +478,13 @@ FROM (
|
|
|
515
478
|
GROUP BY user_id
|
|
516
479
|
);
|
|
517
480
|
```
|
|
518
|
-
|
|
481
|
+
For a strict profile projection contract, investigate every mismatch. Do not
|
|
482
|
+
replace the declared contract with a generic percentage tolerance.
|
|
519
483
|
|
|
520
484
|
### 2. SuperProp-UserProp Mirror Check
|
|
521
|
-
|
|
485
|
+
Only keys promised as profile projections must mirror `userProps`. Matching
|
|
486
|
+
enumerations alone do not guarantee equality; use `stickyEventProps`. Event-only
|
|
487
|
+
context such as an app version does not require a profile mirror.
|
|
522
488
|
|
|
523
489
|
### 3. Mixpanel Default Property Casing Check
|
|
524
490
|
The system generates device properties with Mixpanel's standard casing
|
|
@@ -536,9 +502,9 @@ intentional opt-in — not a casing conflict.
|
|
|
536
502
|
|
|
537
503
|
### 4. funnel-pre Dilution Check
|
|
538
504
|
For any dungeon with `funnel-pre` conversionRate modifications, verify the actual visible effect:
|
|
539
|
-
-
|
|
540
|
-
-
|
|
541
|
-
-
|
|
505
|
+
- Compare the declared funnel report on paired baseline and treatment streams.
|
|
506
|
+
- Inspect organic competitors, repeated opportunities, saturation, and eligible populations.
|
|
507
|
+
- Keep the target and report fixed; neither funnel-pre scaling nor everything filtering guarantees a universal ratio.
|
|
542
508
|
|
|
543
509
|
## Population threshold validation
|
|
544
510
|
|
|
@@ -555,14 +521,15 @@ GROUP BY segment_column
|
|
|
555
521
|
ORDER BY users DESC;
|
|
556
522
|
```
|
|
557
523
|
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
- Segment >50 users: should show clear signal if hook effect ≥1.3x
|
|
524
|
+
Set population floors before measuring, based on the intended effect and report.
|
|
525
|
+
Report actual independent eligible users and converters on both sides. No fixed
|
|
526
|
+
user count guarantees a clear signal for every effect or distribution.
|
|
562
527
|
|
|
563
528
|
## Statistical caveats
|
|
564
529
|
|
|
565
|
-
This skill
|
|
530
|
+
This skill uses the dungeon's configured scale for acceptance. Full fidelity does
|
|
531
|
+
not guarantee enough eligible users, converters, or mature cohorts. Distinguish
|
|
532
|
+
`INSUFFICIENT_EVIDENCE` from measured failure and investigate each accordingly.
|
|
566
533
|
|
|
567
534
|
`--small` mode is a developer-troubleshooting escape hatch on the runner script; verdicts from `--small` runs are unreliable and not permitted in this skill's output.
|
|
568
535
|
|
|
@@ -614,35 +581,15 @@ GROUP BY b.bucket;
|
|
|
614
581
|
|
|
615
582
|
When the dungeon doesn't have a natural "per-X" denominator, compute one from the cohort-binning event: `target_events / cohort_event_count`.
|
|
616
583
|
|
|
617
|
-
**Time-to-convert
|
|
584
|
+
**Time-to-convert verification:** use the steps-based `timeToConvert` emulator
|
|
585
|
+
with explicit report options and completed histories. Independent MIN(A)/MIN(B)
|
|
586
|
+
timestamps can mix attempts. Preserve the requested statistic and derive its
|
|
587
|
+
target from the hook, with a measured factor-one or hook-disabled control. Two
|
|
588
|
+
differently modified segments do not constitute a neutral baseline.
|
|
618
589
|
|
|
619
|
-
|
|
620
|
-
|
|
621
|
-
|
|
622
|
-
MIN(time::TIMESTAMP) FILTER (WHERE event = '<STEP_A>') AS a_time,
|
|
623
|
-
MIN(time::TIMESTAMP) FILTER (WHERE event = '<STEP_B>') AS b_time
|
|
624
|
-
FROM read_json_auto('./data/<run>-EVENTS.json')
|
|
625
|
-
GROUP BY user_id
|
|
626
|
-
)
|
|
627
|
-
SELECT u.<SEGMENT_KEY>,
|
|
628
|
-
COUNT(*) AS users,
|
|
629
|
-
ROUND(MEDIAN(EXTRACT(EPOCH FROM (b_time - a_time)) / 60), 2) AS median_min_a_to_b
|
|
630
|
-
FROM funnel f
|
|
631
|
-
JOIN read_json_auto('./data/<run>-USERS.json') u ON f.user_id = u.distinct_id
|
|
632
|
-
WHERE a_time IS NOT NULL AND b_time IS NOT NULL
|
|
633
|
-
GROUP BY u.<SEGMENT_KEY>
|
|
634
|
-
ORDER BY median_min_a_to_b;
|
|
635
|
-
```
|
|
636
|
-
|
|
637
|
-
**Verdict for T2C**:
|
|
638
|
-
- Fast segment ≤0.85x baseline → STRONG
|
|
639
|
-
- Slow segment ≥1.2x baseline → STRONG
|
|
640
|
-
- Both directions visible → STRONG
|
|
641
|
-
- One/both missing → check that funnel exists in `funnels:` config and segment property is on `meta.profile`
|
|
642
|
-
|
|
643
|
-
**Two-tier T2C interpretation**: When a dungeon has only 2 tiers (e.g. Free vs Paid) the funnel-post hook factor `1.0` branch never fires — both tiers fall into either fast or slow. Pick the slower of the two as the implicit baseline, then verify the faster shows ≤0.85x of it.
|
|
644
|
-
|
|
645
|
-
**No-flag verification rule**: NEVER attempt to verify a hook by querying for a flag like `WHERE sweet_spot = true`. If a dungeon has such flags, treat them as a doc bug — the hook should be reworked to hide the cohort behaviorally. The validator's job is to derive cohorts behaviorally.
|
|
590
|
+
**No-flag verification rule:** derive hidden cohorts behaviorally or with the
|
|
591
|
+
declared hash. A flag with a schema-declared default is allowed; an undeclared
|
|
592
|
+
flag fails even when present on every record.
|
|
646
593
|
|
|
647
594
|
## Drop-event funnel dilution diagnosis
|
|
648
595
|
|
|
@@ -650,22 +597,27 @@ Many dungeons have hooks of pattern `record.filter(e => e.event === 'X' && chanc
|
|
|
650
597
|
|
|
651
598
|
**Why:** the hook drops EVENTS not users. A user with 5 step-3 events still appears in the funnel after losing 1-2 events. Funnel completion = `users with ≥1 step-3 event` — only zero-step-3 users disappear from the conversion count, which is rare.
|
|
652
599
|
|
|
653
|
-
**
|
|
600
|
+
**Supplementary diagnostic:** per-user volume of step-3 events by tier. This
|
|
601
|
+
does not replace the declared funnel completion check:
|
|
654
602
|
|
|
655
603
|
```sql
|
|
656
604
|
SELECT u.subscription_tier,
|
|
657
605
|
COUNT(DISTINCT user_id) AS users,
|
|
658
606
|
COUNT(*) AS total_step3,
|
|
659
607
|
ROUND(COUNT(*) * 1.0 / COUNT(DISTINCT user_id), 2) AS per_user
|
|
660
|
-
FROM read_json_auto('./data/<run>-EVENTS.json')
|
|
661
|
-
|
|
608
|
+
FROM read_json_auto('./data/<run>-EVENTS.json') e
|
|
609
|
+
JOIN read_json_auto('./data/<run>-USERS.json') u
|
|
610
|
+
ON e.user_id::VARCHAR = u.distinct_id::VARCHAR
|
|
611
|
+
WHERE e.event = '<STEP_3_EVENT>'
|
|
662
612
|
GROUP BY u.subscription_tier
|
|
663
613
|
ORDER BY per_user DESC;
|
|
664
614
|
```
|
|
665
615
|
|
|
666
616
|
Expected: paid tier ~1.5x non-paid per_user (matches 30% drop on non-paid → paid keeps 100%, non-paid keeps 70%, ratio 1/0.7 = 1.43x).
|
|
667
617
|
|
|
668
|
-
|
|
618
|
+
A volume gap can show the mutation fired while the declared conversion story
|
|
619
|
+
still fails. Preserve that failure. Change the report only through an explicit
|
|
620
|
+
story revision, then verify the revised claim with its own controls.
|
|
669
621
|
|
|
670
622
|
## Subscription tier cohort sizing check
|
|
671
623
|
|
|
@@ -676,12 +628,10 @@ SELECT subscription_plan, COUNT(*) FROM read_json_auto('./data/<run>-USERS.json'
|
|
|
676
628
|
GROUP BY subscription_plan;
|
|
677
629
|
```
|
|
678
630
|
|
|
679
|
-
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
|
|
683
|
-
- Bump `numUsers` up to 5x to enlarge cohorts
|
|
684
|
-
- Or recommend dungeon author tighten subscription lifecycle config
|
|
631
|
+
There is no active subscription lifecycle config block. Read the declared
|
|
632
|
+
`userProps` distribution and hook logic, then measure eligible populations.
|
|
633
|
+
If evidence is insufficient, request a larger run or an authorized schema change;
|
|
634
|
+
do not invent lifecycle defaults or silently relax the acceptance threshold.
|
|
685
635
|
|
|
686
636
|
## Per-day normalization for time-window hooks
|
|
687
637
|
|
|
@@ -701,41 +651,40 @@ For any spike/burst hook with a tight day window, ALWAYS normalize by window len
|
|
|
701
651
|
|
|
702
652
|
## Determinism check (optional confidence test)
|
|
703
653
|
|
|
704
|
-
|
|
654
|
+
For seeded generation, pin `datasetStart`/`datasetEnd` and `concurrency: 1`.
|
|
655
|
+
Use isolated sequential runs and strip only `insert_id` before comparing events:
|
|
705
656
|
|
|
706
657
|
1. Run a previously-passing dungeon a second time.
|
|
707
|
-
2.
|
|
708
|
-
3.
|
|
658
|
+
2. Require identical event counts, timestamps, ordering, and seeded property values.
|
|
659
|
+
3. Require identical report output under the same explicit options.
|
|
709
660
|
|
|
710
|
-
|
|
711
|
-
|
|
712
|
-
|
|
661
|
+
Investigate differences, including wall-clock calls, unseeded RNG, and stale
|
|
662
|
+
module state. Document known unseeded property functions separately; do not
|
|
663
|
+
accept a generic percentage drift as determinism proof.
|
|
713
664
|
|
|
714
665
|
## Critical time-window verification pattern
|
|
715
666
|
|
|
716
|
-
|
|
667
|
+
Use the resolved dataset bounds, also exposed as `meta.datasetStart` and
|
|
668
|
+
`meta.datasetEnd` in unix seconds. Generation occurs inside this window without
|
|
669
|
+
a post-generation shift. Observed event extrema do not reconstruct configured
|
|
670
|
+
bounds, especially for sparse or partially observed windows:
|
|
717
671
|
|
|
718
672
|
```sql
|
|
719
|
-
-- WRONG: uses MIN(time) which is up to 30 days BEFORE dataset start (pre-existing user spread)
|
|
720
|
-
SELECT *, EXTRACT(EPOCH FROM (time::TIMESTAMP - (SELECT MIN(time::TIMESTAMP) FROM events))) / 86400 as day_in
|
|
721
|
-
FROM events;
|
|
722
|
-
|
|
723
|
-
-- RIGHT: anchor to MAX(time) - num_days, which is the post-shift dataset start
|
|
724
673
|
WITH bounds AS (
|
|
725
|
-
SELECT
|
|
726
|
-
FROM events
|
|
674
|
+
SELECT TIMESTAMP '<RESOLVED_DATASET_START_UTC>' as datasetStart
|
|
727
675
|
)
|
|
728
676
|
SELECT *, EXTRACT(EPOCH FROM (e.time::TIMESTAMP - b.datasetStart)) / 86400 as day_in
|
|
729
677
|
FROM events e, bounds b;
|
|
730
678
|
```
|
|
731
679
|
|
|
732
|
-
Pre-existing
|
|
680
|
+
Pre-existing profile creation can precede the window; generated user events still
|
|
681
|
+
stay inside the resolved bounds. Never substitute `MAX(time) - numDays` for them.
|
|
733
682
|
|
|
734
683
|
## TTC hook verification — two approaches
|
|
735
684
|
|
|
736
685
|
TTC hooks come in two forms. Use the matching verification approach:
|
|
737
686
|
|
|
738
|
-
### Approach 1:
|
|
687
|
+
### Approach 1: Numeric timing-property report
|
|
739
688
|
|
|
740
689
|
The hook scales a timing PROPERTY (e.g., `response_time_mins *= 0.67`) by segment. Verification is trivial:
|
|
741
690
|
|
|
@@ -748,49 +697,33 @@ WHERE event IN ('alert acknowledged', 'alert resolved')
|
|
|
748
697
|
GROUP BY segment ORDER BY avg_response;
|
|
749
698
|
```
|
|
750
699
|
|
|
751
|
-
This
|
|
700
|
+
This proves a property aggregate only. Measure paired baseline/treatment and a
|
|
701
|
+
factor-one control; raw segment ratios can reflect different starting distributions.
|
|
752
702
|
|
|
753
|
-
### Approach 2:
|
|
703
|
+
### Approach 2: Funnel timestamp TTC report
|
|
754
704
|
|
|
755
|
-
|
|
756
|
-
|
|
757
|
-
|
|
758
|
-
|
|
759
|
-
SELECT user_id, MIN(a.time) AS start, MIN(b.time) AS end ...
|
|
760
|
-
|
|
761
|
-
-- RIGHT: bound-sequence (first A, then first B AFTER that A)
|
|
762
|
-
WITH steps AS (
|
|
763
|
-
SELECT user_id, event, time::TIMESTAMP AS t
|
|
764
|
-
FROM events WHERE event IN ('step_a', 'step_b', 'step_c')
|
|
765
|
-
),
|
|
766
|
-
funnel AS (
|
|
767
|
-
SELECT DISTINCT ON (a.user_id) a.user_id, a.t AS start_t,
|
|
768
|
-
(SELECT MIN(t) FROM steps c
|
|
769
|
-
WHERE c.user_id = a.user_id AND c.event = 'step_c' AND c.t > a.t) AS end_t
|
|
770
|
-
FROM steps a WHERE a.event = 'step_a'
|
|
771
|
-
ORDER BY a.user_id, a.t
|
|
772
|
-
)
|
|
773
|
-
SELECT segment,
|
|
774
|
-
COUNT(*) AS users,
|
|
775
|
-
ROUND(MEDIAN(EXTRACT(EPOCH FROM (end_t - start_t)) / 60), 1) AS median_min
|
|
776
|
-
FROM funnel JOIN users USING (user_id)
|
|
777
|
-
WHERE end_t IS NOT NULL
|
|
778
|
-
GROUP BY segment ORDER BY median_min;
|
|
779
|
-
```
|
|
780
|
-
|
|
781
|
-
The bound-sequence pattern finds the first A per user, then the first C strictly after that A. This matches how the everything hook operates and typically produces STRONG verdicts. The lazy MIN→MIN proxy produces flat or inverted results because it grabs unrelated events from different funnel passes.
|
|
705
|
+
Use steps-based `timeToConvert` on completed histories with the report's window,
|
|
706
|
+
order, filters, identity, and reentry options. A first-A/next-C query can skip
|
|
707
|
+
required B, miss restarts, or mishandle grace. Compare the same report on baseline,
|
|
708
|
+
treatment, and neutral-control streams. Keep mean and median claims separate.
|
|
782
709
|
|
|
783
710
|
### Which approach to recommend when writing hooks
|
|
784
711
|
|
|
785
|
-
|
|
712
|
+
Choose the hook based on the intended report. Numeric property scaling answers
|
|
713
|
+
a property report; timestamp changes target funnel elapsed time. Ease of
|
|
714
|
+
verification does not authorize replacing one with the other.
|
|
786
715
|
|
|
787
716
|
### Legacy funnel-post TTC hooks
|
|
788
717
|
|
|
789
|
-
|
|
718
|
+
Measure legacy `funnel-post` effects with the same report contract and controls.
|
|
719
|
+
Never assign STRONG by code inspection. If instance-level mutations fail to move
|
|
720
|
+
completed report histories, retain the miss and investigate competing instances.
|
|
790
721
|
|
|
791
722
|
## Magic-number cohort sizing — inspect distribution first
|
|
792
723
|
|
|
793
|
-
Before checking inverted-U signal magnitude,
|
|
724
|
+
Before checking inverted-U signal magnitude, count independent eligible users in
|
|
725
|
+
each bucket against a predeclared population floor. A floor such as 200 is a
|
|
726
|
+
design choice, not universal statistical proof:
|
|
794
727
|
|
|
795
728
|
```sql
|
|
796
729
|
SELECT pn, COUNT(*) FROM (
|
|
@@ -799,11 +732,9 @@ SELECT pn, COUNT(*) FROM (
|
|
|
799
732
|
) GROUP BY pn ORDER BY pn LIMIT 20;
|
|
800
733
|
```
|
|
801
734
|
|
|
802
|
-
If
|
|
803
|
-
|
|
804
|
-
|
|
805
|
-
|
|
806
|
-
Choice depends on whether the JSDoc's stated ranges are load-bearing for the dungeon's narrative ("you need 8+ photos to seem fake" — preserve range, scale up users) or arbitrary ("sweet 4-7" can shift to "sweet 2-5" without losing the story).
|
|
735
|
+
If too few users reach the declared buckets, report insufficient evidence.
|
|
736
|
+
Request a larger run while preserving the story. Changing bucket ranges changes
|
|
737
|
+
the report specification and requires an explicit story revision and new proof.
|
|
807
738
|
|
|
808
739
|
## Re-run required after hook edits
|
|
809
740
|
|
|
@@ -811,7 +742,7 @@ If you edit a hook then query the existing data files, you'll get STALE results.
|
|
|
811
742
|
|
|
812
743
|
```bash
|
|
813
744
|
# Keep this run's files for verification and deployment; cleanup needs explicit consent.
|
|
814
|
-
node scripts/verify-runner.mjs dungeons/vertical/<NAME>.js verify-<NAME
|
|
745
|
+
node scripts/verify-runner.mjs dungeons/vertical/<NAME>/<NAME>.js verify-<NAME>-r2
|
|
815
746
|
# Wait for the {"mode":"full","eventCount":...} JSON to print before querying
|
|
816
747
|
```
|
|
817
748
|
|
|
@@ -839,49 +770,26 @@ The `event` hook receives `meta.datasetStart` as a unix timestamp, but temporal
|
|
|
839
770
|
|
|
840
771
|
## Property baseline dilution
|
|
841
772
|
|
|
842
|
-
|
|
843
|
-
|
|
844
|
-
|
|
845
|
-
|
|
846
|
-
|
|
773
|
+
If an independent hook forces a value on fraction `q` of eligible events with
|
|
774
|
+
baseline prevalence `p`, expected prevalence is `q + (1 - q) * p`. For `p=0.20`
|
|
775
|
+
and `q=0.40`, that is 0.52. Other targeting and time-window rules need their own
|
|
776
|
+
derivation. Measure the neutral baseline; request an authorized schema change if
|
|
777
|
+
the baseline distribution must change. Do not tune it silently after a miss.
|
|
847
778
|
|
|
848
779
|
## Computing the dataset window
|
|
849
780
|
|
|
850
|
-
|
|
851
|
-
|
|
852
|
-
|
|
853
|
-
|
|
854
|
-
|
|
855
|
-
| `numDays` only (no explicit start/end) | `datasetEnd = NOW`, `datasetStart = NOW - numDays` |
|
|
856
|
-
| `datasetStart` + `numDays` | `datasetEnd = datasetStart + numDays` |
|
|
857
|
-
|
|
858
|
-
The engine always resolves to a `[datasetStart, datasetEnd]` pair internally (see `config-validator.js`). To find the actual window from the OUTPUT data:
|
|
859
|
-
|
|
860
|
-
```sql
|
|
861
|
-
SELECT
|
|
862
|
-
MAX(time::TIMESTAMP) as datasetEnd,
|
|
863
|
-
MAX(time::TIMESTAMP) - INTERVAL '<numDays>' DAY as datasetStart
|
|
864
|
-
FROM read_json_auto('./data/verify-X-EVENTS*.json', sample_size=-1);
|
|
865
|
-
```
|
|
866
|
-
|
|
867
|
-
Use `datasetStart` (derived above) as the DuckDB anchor for day-in-dataset:
|
|
868
|
-
|
|
869
|
-
```sql
|
|
870
|
-
WITH bounds AS (
|
|
871
|
-
SELECT MAX(time::TIMESTAMP) - INTERVAL '<numDays>' DAY as ds_start
|
|
872
|
-
FROM read_json_auto('./data/verify-X-EVENTS*.json', sample_size=-1)
|
|
873
|
-
)
|
|
874
|
-
SELECT EXTRACT(EPOCH FROM (e.time::TIMESTAMP - b.ds_start)) / 86400 as day_in
|
|
875
|
-
FROM events e, bounds b;
|
|
876
|
-
```
|
|
877
|
-
|
|
878
|
-
Do NOT use `MIN(time)` as the anchor — pre-existing users have events up to 30 days before `datasetStart` (from `preExistingSpread: 'uniform'`).
|
|
879
|
-
|
|
880
|
-
When the dungeon has explicit `datasetStart` (e.g., `"2026-01-01T00:00:00Z"`), use it directly: `TIMESTAMP '2026-01-01'`. When `numDays` is used without explicit start, derive from MAX(time) as shown above.
|
|
781
|
+
Record the engine's resolved `datasetStart`/`datasetEnd` from the run, including
|
|
782
|
+
derived windows. Use those values in the query shown under "Critical time-window
|
|
783
|
+
verification pattern". If the artifacts do not retain bounds, report the missing
|
|
784
|
+
metadata or regenerate a separately named pinned run. Neither observed MIN/MAX
|
|
785
|
+
nor the current wall clock can recover the original resolved window reliably.
|
|
881
786
|
|
|
882
787
|
## No flag stamping audit
|
|
883
788
|
|
|
884
|
-
Hooks must
|
|
789
|
+
Hooks must never add undeclared flags. A schema-declared flag with a default is
|
|
790
|
+
valid; hidden cohorts can use behavioral or hash definitions. Flag undeclared
|
|
791
|
+
assignments as SCHEMA-FAIL and request an authorized schema declaration or a hook
|
|
792
|
+
rewrite using existing fields. Uniform coverage does not make them acceptable.
|
|
885
793
|
|
|
886
794
|
## Clone dilution of temporal effects
|
|
887
795
|
|
|
@@ -900,7 +808,7 @@ If Hook A classifies users by event presence (`events.some(e => e.event === X)`)
|
|
|
900
808
|
**Fixes:**
|
|
901
809
|
1. Require 3+ marker events instead of 1+ (surviving events still identify)
|
|
902
810
|
2. Accept the verification limitation and note it in the report
|
|
903
|
-
3.
|
|
811
|
+
3. Propose a separate distribution diagnostic while retaining the original cohort report's unresolved status; changing the report requires explicit revision.
|
|
904
812
|
|
|
905
813
|
## Deprecated feature property gaps
|
|
906
814
|
|
|
@@ -908,4 +816,6 @@ Dungeons using deprecated config blocks (`subscription`, `attribution`, `feature
|
|
|
908
816
|
|
|
909
817
|
**Diagnosis:** Hook logic references a property that's always NULL/undefined in the output. Check if the property was produced by a deprecated feature.
|
|
910
818
|
|
|
911
|
-
**Fix:**
|
|
819
|
+
**Fix:** Request a schema declaration with a default through `/create-dungeon`
|
|
820
|
+
before hooks assign values. Report the missing field and preserve the runner's
|
|
821
|
+
verdict; adding undeclared property generation inside a hook is not a valid fix.
|
|
@@ -36,8 +36,14 @@ node .claude/skills/warehouse-metrics/deploy.mjs <dungeon-path> [--dataset dm_na
|
|
|
36
36
|
- `--data-prefix`: explicit run artifact prefix, for example `/tmp/run/warehouse-demo` for `/tmp/run/warehouse-demo-WAREHOUSE-MANIFEST.json`.
|
|
37
37
|
- `--dry-run`: prints the full `bq ls`, docs probe, metric list, load, source, preview, and create plan without executing commands or requiring credentials. It still writes the SQL files and renders `warehouse/GAPS.md` for review.
|
|
38
38
|
|
|
39
|
+
Dry-run is not read-only. Preserve any existing actual deployment report before
|
|
40
|
+
running it. Keep proposed setup gaps separate from recorded live outcomes.
|
|
41
|
+
|
|
39
42
|
## Preflight
|
|
40
43
|
|
|
44
|
+
- Apply the [1.8.1 verification contract](../verify-dungeon/references/alignment-contract.md).
|
|
45
|
+
Offline verification stops at local evidence and the deployment handoff. Do not
|
|
46
|
+
automatically run deployment, endpoint probes, previews, or cloud commands.
|
|
41
47
|
- The dungeon must have passed `/verify-dungeon` and produced local warehouse
|
|
42
48
|
files with `writeToDisk: true` and `gzip: false`. Use the exact verified
|
|
43
49
|
`--data-prefix` and its matching `-WAREHOUSE-MANIFEST.json`; preserve all table
|