@ak--47/dungeon-master 1.5.3 → 1.6.0

package/docs/guides/1.4.3-upgrade-guide.md

@@ -0,0 +1,200 @@
+# 1.4.3 Upgrade Guide
+
+**TL;DR: `percentUsersBornInDataset` defaults raised across all macro presets. No API changes. Existing dungeons that set `macro` or `percentUsersBornInDataset` explicitly are unaffected. Dungeons relying on the implicit "flat" default will now have 50% born-in-dataset users instead of 15%.**
+
+## What Changed
+
+### Breaking: `percentUsersBornInDataset` defaults raised
+
+The "flat" macro preset (the default when no `macro` is specified) changed from 15% to 50%. All other presets were raised proportionally. No preset goes below 25%.
+
+| Preset | Before (1.4.1) | After (1.4.3) | Notes |
+|--------|---------------|---------------|-------|
+| `flat` | 15 | **50** | Default when no macro specified |
+| `steady` | 10 | **35** | |
+| `growth` | 25 | **60** | |
+| `viral` | 50 | **95** | Nearly all users born in window |
+| `decline` | 5 | **25** | Floor — nothing below 25% |
+
+#### Why
+
+The old 15% default meant only ~750 out of 5,000 users were "born" during the dataset window. The other 85% were pre-existing users with full-window event histories. This caused two problems:
+
+1. **Retention and onboarding hooks had tiny cohorts.** A hook that splits born-in-dataset users into "activated" vs "not activated" cohorts was working with ~750 users total. After splitting, cohorts of 50-100 users produced noisy, unreliable signal in verification and Mixpanel reports.
+
+2. **The "end-bunchiness" fix overcorrected.** The original bunchiness problem (events piling up at the right edge) was solved in 1.3.0 by `avgEventsPerUserPerDay` — born-late users now get `rate × remaining_days` events instead of compressing `numEvents/numUsers` into a tiny window. With that fix in place, there's no reason to keep born-in-dataset users artificially low. More born users = richer onboarding/retention stories without any right-edge blow-up.
+
+#### Migration
+
+**Dungeons that explicitly set `macro` or `percentUsersBornInDataset`:** No change needed. Explicit values override the preset.
+
+**Dungeons relying on the implicit default:** Will now generate more born-in-dataset users. This means:
+- More "account created" / first-event traffic spread across the window
+- Retention hooks will have larger cohorts (good)
+- Total event count may decrease slightly because born-late users have shorter active windows
+- If you need the old behavior, add `macro: { preset: "flat", percentUsersBornInDataset: 15 }` to your config
+
+**Tests:** If you have tests with hardcoded event count thresholds (e.g., `expect(events.length).toBeGreaterThan(N)`) on small configs without explicit `percentUsersBornInDataset`, you may need to lower the thresholds. More born-in-dataset users = some users have shorter windows = fewer total events.
+
+### New: HOOKS.md Recipe 3.22 — Retention Magic Number
+
+A new hook recipe for the "N actions in first X days predicts retention" pattern, drawn from the Twitter dungeon. This is the most requested hook pattern for consumer products.
+
+#### The Pattern
+
+Users who perform a threshold action (e.g., follow 5 accounts) within an activation window (e.g., first 14 days) are retained. Users who don't hit the threshold go silent after a cutoff (e.g., 36 days after their first event).
+
+```js
+if (meta.userIsBornInDataset) {
+  const userStart = dayjs(record[0]?.time);
+  const windowEnd = userStart.add(14, "days").toISOString();
+  const bin = binUsersByEventInRange(
+    record, "user followed",
+    record[0]?.time, windowEnd,
+    { retained: [5, Infinity], not_retained: [0, 5] }
+  );
+  if (bin === "not_retained") {
+    const cutoff = userStart.add(36, "days");
+    dropEventsWhere(record, e => dayjs(e.time).isAfter(cutoff));
+  }
+}
+```
+
+#### Key Design Lessons (from the Twitter dungeon iteration)
+
+These lessons came from three rounds of verify-hooks iteration. Each one produced a WEAK or INVERSE verdict before the fix was found.
+
+**1. User-relative cutoff, not dataset-relative.**
+
+```js
+// WRONG — misses users born after day 36
+const cutoff = datasetStart.add(36, "days");
+
+// RIGHT — each user's cutoff is relative to their own first event
+const cutoff = userStart.add(36, "days");
+```
+
+A dataset-anchored cutoff (e.g., `datasetStart + 36 days` = Feb 6) means users born on day 60 already have their first event past the cutoff — nothing gets dropped. The hook appears to fire but produces no visible retention difference. This was the most common failure mode.
+
+**2. Threshold calibration against actual event distribution.**
+
+Before picking a threshold, compute what's achievable:
+
+```
+events/user/day × (event_weight / total_weight) × window_days = expected_count
+```
+
+For the Twitter dungeon: `5 events/day × (5/84) × 14 days ≈ 4.2 follows in 14 days`. A threshold of 5 means ~15-20% of born users qualify — large enough for signal, small enough to be non-trivial.
+
+The original threshold (7 follows in 10 days) produced a retained cohort of only 37 users out of 3,075 born-in-dataset. Too small. Lowering to 5 follows in 14 days expanded it to 900 users and the retention signal jumped from 1.56x to 2.19x.
+
+**3. `percentUsersBornInDataset` is load-bearing for retention hooks.**
+
+At the old 15% default, only ~750 users are born-in-dataset. After splitting into retained/not-retained, cohorts can be <100 users. The new 50% default ensures ~2,500 born users at 5K total — enough for both cohorts to produce clean signal.
+
+**4. Verify with pct_past_dN, not avg_active_days.**
+
+`avg_active_days` is confounded by birth time — users born late in the dataset naturally have fewer active days regardless of the hook. The correct metric is `pct_past_d36`: what fraction of each cohort has events beyond 36 days after their first event. This isolates the hook's effect from the birth-time distribution.
+
+```sql
+-- RIGHT metric: what % of each cohort survives past day 36
+SELECT cohort,
+  COUNT(*) FILTER (WHERE active_days > 36) * 100.0 / COUNT(*) AS pct_past_d36
+FROM ...
+```
+
+### New: Twitter/X Dungeon
+
+`dungeons/user/twitter.js` — consumer social media platform with 4 verified hooks:
+
+| Hook | Pattern | Verdict |
+|------|---------|---------|
+| 5 follows in 14 days → retention | Retention magic number | STRONG (2.19x) |
+| Web ad bidding bug (25-day window) | Degradation and recovery | NAILED (6.0x latency, 0.56x conversion) |
+| 5-8 tweets/week → verified purchase | Inverted-U sweet spot | NAILED (8333x signal) |
+| Top business advertisers → premium placement | Profile-segment boost | STRONG (100% timeline, 1.62x quality) |
+
+The dungeon also includes:
+- Server-side ad auction pipeline (ad bid → ad impression → ad click) with heavy bid properties (auction ID, bid/winning amounts, keyword targeting, quality scores, bid latency)
+- Group analytics on `advertiser_id` (business account_type users only)
+- SCDs for membership_status, follower_count, following_count
+- Multi-device identity model (iOS + Android + Desktop)
+
+### Lesson: `advertiser_id` (group key) is NOT available during `everything` hook
+
+The `everything` hook fires at `user-loop.js:280`, but group key stamping happens during storage push (after the everything hook completes). Any hook that needs to condition on `advertiser_id` (or any `groupKeys` value) cannot read it from events during the everything hook — it will be `undefined`.
+
+**Workaround:** Use profile-based conditions (`meta.profile.account_type`) combined with deterministic hash splitting to simulate group-level cohort effects within the everything hook. The group key will be stamped correctly on the output events by the storage phase, but the hook logic must use a proxy.
+
+```js
+// WRONG — advertiser_id not yet stamped during everything hook
+const advId = userEvents.find(e => e.advertiser_id)?.advertiser_id;
+
+// RIGHT — use profile + hash as proxy for group membership
+if (profile.account_type === "business") {
+  const hash = String(uid).split("").reduce((a, c) => a + c.charCodeAt(0), 0);
+  const isTopAdvertiser = (hash % 5) < 2;
+  // ... modify events for top advertisers
+}
+```
+
+### New: Schema Validation (`lib/verify/schema-validator.js`)
+
+Catches hooks that introduce undeclared columns (flag stamping). Integrated into `verifyDungeon()` — runs automatically.
+
+```javascript
+import { deriveExpectedSchema, validateSchema } from '@ak--47/dungeon-master/verify';
+
+const schema = deriveExpectedSchema(config);  // Map<eventName, Set<propKey>>
+const report = validateSchema(events, config); // { pass, eventTypes, summary, flagStamping }
+```
+
+**Rule:** Hook-introduced columns with 100% coverage within their event type = PASS. Partial coverage = FAIL (flag stamping).
+
+`verifyDungeon()` now returns `schemaReport` alongside `results`:
+
+```javascript
+const { pass, results, schemaReport } = await verifyDungeon(config, checks);
+```
+
+### New: Property Type Helpers (complete Mixpanel coverage)
+
+Three new helpers for the three missing Mixpanel property data types. All exported from `@ak--47/dungeon-master/utils`.
+
+| Helper | Mixpanel Type | Example | Output |
+|--------|--------------|---------|--------|
+| `dateRange(start?, end?)` | Date | `dateRange()` | `"2024-03-15T14:22:33"` |
+| `listOf(pool, {min, max})` | List | `listOf(["A","B","C"], {min:1, max:3})` | `["A","C"]` |
+| `objectList(template, {min, max})` | List of Objects | `objectList({id: weighNumRange(1,100), cat: ["A","B"]}, {min:1, max:3})` | `[{id:42, cat:"A"}]` |
+
+**Implementation detail:** `ListValue extends Array` marker class prevents `choose()` from picking a single element from the returned array. Serializes identically to plain arrays in JSON.
+
+**`dateRange()` defaults to the dataset window** (`datasetStart` → `datasetEnd`). Accepts ISO strings, unix seconds, or dayjs objects for custom bounds.
+
+### Skill Rename: `verify-hooks` → `verify-dungeon`
+
+Reflects broader scope (schema + hooks + identity + experiments). All active code references updated; historical docs left as-is.
+
+- Slash command: `/verify-dungeon <dungeon-path>`
+- Default verify-runner.mjs run name: `verify-dungeon`
+
+## Full Changelog
+
+- `lib/templates/macro-presets.js`: Raised `percentUsersBornInDataset` across all 5 presets (floor 25%, flat default 50%)
+- `HOOKS.md`: Added recipe 3.22 (Retention Magic Number), renumbered 3.22→3.23, 3.23→3.24
+- `CLAUDE.md`: Updated default references from 15 to 50, added Property Type Helpers section, updated public API imports, updated skill references
+- `types.d.ts`: Updated JSDoc on `percentUsersBornInDataset`, added `SchemaReport` interface, added `@ak--47/dungeon-master/utils` module declaration, updated `verifyDungeon` return type
+- `.claude/skills/create-dungeon/SKILL.md`: Updated macro section, added Property Type Reference
+- `.claude/skills/verify-dungeon/SKILL.md`: **Renamed** from verify-hooks, added Step 2b Schema Validation
+- `.claude/skills/write-hooks/SKILL.md`: Updated verify-hooks → verify-dungeon references
+- `dungeons/user/twitter.js`: New Twitter/X dungeon with 4 verified hooks
+- `lib/verify/schema-validator.js`: **New** — `deriveExpectedSchema()`, `validateSchema()`
+- `lib/verify/verify-dungeon.js`: Integrated schema validation, updated return type
+- `lib/verify/index.js`: Added schema validator exports
+- `lib/utils/utils.js`: Added `dateRange()`, `listOf()`, `objectList()`, `ListValue`, `DATASET_BEGIN` + `setDatasetBegin()`, `choose()` ListValue passthrough
+- `index.js`: Import + call `setDatasetBegin()`
+- `scripts/verify-runner.mjs`: Default run name → `verify-dungeon`
+- `tests/schema-validator.test.js`: **New** — 17 tests
+- `tests/property-helpers.test.js`: **New** — 21 tests
+- `tests/macro-and-rate.test.js`: Added pinned-value snapshot test for all presets, added floor guard (≥25%)
+- `tests/int.test.js`, `tests/sanity.test.js`: Adjusted event count thresholds for new default

package/docs/guides/1.4.4-upgrade-guide.md

@@ -0,0 +1,95 @@
+# 1.4.4 Upgrade Guide
+
+**TL;DR: Fixes GCS-sourced imports hanging indefinitely. GCS writes no longer use `Content-Encoding: gzip`. Default format for GCS is now JSONL (was CSV). Group profile batch imports fixed. No API changes.**
+
+## What Changed
+
+### Fixed: GCS import hang (critical)
+
+GCS-sourced imports (`writeToDisk: 'gs://...'` + `token`) hung indefinitely — the `DUNGEON_MASTER()` promise never resolved, zero events arrived in Mixpanel, and cleanup never ran. Three independent production runs on Cloud Run (Node 24) reproduced the issue.
+
+**Root cause:** The three stream-write functions (`streamJSON`, `streamCSV`, `streamParquet`) used `createWriteStream({ gzip: true })` for all GCS paths. This set `Content-Encoding: gzip` on the GCS object — HTTP-level transparent compression. When `mixpanel-import` read the files back:
+
+1. The HTTP transport layer automatically decompressed the data (transparent `Content-Encoding` handling)
+2. But `mixpanel-import` checked the file extension (`.csv`, `.json`) and found no `.gz` → no gunzip applied
+3. This happened to work when the HTTP layer decompressed, but **broke silently on environments where it didn't** (Cloud Run internal networking)
+4. On failure: raw gzip bytes reached the parser, stream errors didn't propagate through `.pipe()` chains, downstream Transform streams never received `'end'`, the pipeline promise never resolved
+
+**Fix:** GCS writes no longer use `createWriteStream({ gzip: true })`. Gzip is now handled at the application level (same as local writes):
+
+- **No `gzip` config (default):** plain JSONL files written to GCS, no compression metadata
+- **`gzip: true`:** data piped through `zlib.createGzip()` before writing to GCS. File gets `.gz` extension. No `Content-Encoding` header. `mixpanel-import` detects `.gz` extension, applies gunzip, parses correctly
+
+Both modes verified end-to-end: write → GCS → `mixpanel-import` stream → parse → records.
+
+### Changed: GCS default format is now JSONL
+
+When `writeToDisk` is a `gs://` path and no `format` is specified, the default is now `"json"` (JSONL) instead of `"csv"`. This ensures `mixpanel-import` can stream-parse all data types (events, users, groups, SCDs, ad spend) without format-specific edge cases.
+
+If you explicitly set `format: "csv"` or `format: "json"` in your config, nothing changes.
+
+### Fixed: Group profiles skipped in batch mode
+
+After `flushStorageToDisk()`, all HookedArray containers have `length === 0` (data is on disk). The group profile import loop had an early-exit guard:
+
+```js
+if (!groupEntity || groupEntity.length === 0) continue;
+```
+
+This skipped group profiles even when batch files existed on disk. Events, users, and ad spend all had `|| isBATCH_MODE` fallbacks — groups didn't. Fixed:
+
+```js
+if (!groupEntity) continue;
+if (groupEntity.length === 0 && !isBATCH_MODE) continue;
+```
+
+### Fixed: Group events batch mode fallback (latent)
+
+The group events import block (`groupEventData?.length > 0`) had no `isBATCH_MODE` fallback. Currently latent (group events aren't initialized in storage), but fixed defensively for when that changes.
+
+### Fixed: GCS gzip finish-event timing
+
+When writing gzipped data to GCS, the promise resolved on the gzip transform's `finish` event (compression done) rather than the GCS stream's `finish` event (upload complete). This could produce truncated files on GCS. Fixed — the promise now resolves when the GCS upload stream finishes.
+
+### Added: GCS round-trip tests
+
+Three new tests in `tests/file-tracking.test.js`:
+
+1. **JSONL round-trip** — write to GCS with default config, verify `.json` extension, stream back through `mixpanel-import`, verify records parse correctly
+2. **Gzipped JSONL round-trip** — write with `gzip: true`, verify `.json.gz` extension, verify no `Content-Encoding: gzip` on GCS object, stream back and verify parsing
+3. **Full dungeon round-trip** — events + users + groups + SCDs + ad spend, all written to GCS, all verified readable
+
+### Added: `data/.gitkeep`
+
+Ensures the `./data` directory exists for tests that write to the default local output path.
+
+### Included: HOOKS.md in npm package
+
+`HOOKS.md` (the hook encyclopedia — 24 recipes, 20 principles, atom/pattern reference) is now shipped with the npm package.
+
+## Migration
+
+### If you use `writeToDisk: 'gs://...'`
+
+**Your imports will now work.** Previously they hung. No config changes needed.
+
+If you were passing `format: 'csv'` explicitly, that still works — but JSONL is now the recommended format for GCS because `mixpanel-import` handles it more reliably across all data types.
+
+### If you relied on GCS gzip compression
+
+Previously, all GCS writes were silently gzipped via `Content-Encoding: gzip` regardless of `config.gzip`. Now:
+
+- **Default (no `gzip`):** files are written uncompressed. Slightly larger on GCS, but these are temporary files that get cleaned up after import (`cleanup: true`).
+- **`gzip: true`:** files are properly gzipped at the application level with `.gz` extension. Works correctly with `mixpanel-import`'s extension-based format detection.
+
+### If you use group analytics with GCS
+
+Group profiles were silently not imported in batch mode. They will now be imported correctly. You may see group profiles appear in Mixpanel that were previously missing.
+
+## Files Changed
+
+- `lib/utils/utils.js` — removed `{ gzip: true }` from all GCS `createWriteStream()` calls; added application-level gzip for GCS (pipe through `zlib.createGzip()`); fixed finish-event to wait on GCS stream
+- `lib/core/config-validator.js` — GCS paths default to `format: 'json'`
+- `lib/orchestrators/mixpanel-sender.js` — group profile batch guard; group events batch guard
+- `tests/file-tracking.test.js` — 3 new GCS round-trip tests, `mixpanel-import` stream verification
+- `tests/sanity.test.js` — ensure `./data` directory exists in test setup