backend-manager 5.1.4 → 5.2.1

package/.claude/settings.local.json

@@ -0,0 +1,12 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(grep -B2 -A20 \"Generate a newsletter preview\" /tmp/bem-test-run-3.log | head -50)",
+      "Bash(sed 's/\\\\x1b\\\\[[0-9;]*m//g')",
+      "Read(//tmp/**)",
+      "Bash(TEST_EXTENDED_MODE=1 npx mgr test 2>&1 | sed 's/\\\\x1b\\\\[[0-9;]*m//g' > /tmp/bem-run-6.log; grep -E \"passing|failing|skipped\" /tmp/bem-run-6.log | tail -5)",
+      "Bash(awk -F'`' '{print $2}')",
+      "Bash(TEST_EXTENDED_MODE=1 npx mgr test 2>&1 | sed 's/\\\\x1b\\\\[[0-9;]*m//g' > /tmp/bem-cleanup-run2.log; grep -E \"\\(passing|failing|skipped\\)\" /tmp/bem-cleanup-run2.log | tail -5)"
+    ]
+  }
+}

package/CHANGELOG.md

@@ -14,6 +14,35 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+# [5.2.1] - 2026-05-21
+
+### Added
+
+- **`BACKEND_MANAGER_WEBHOOK_KEY` in `templates/_.env`.** The `.env` scaffold the setup CLI copies into consumer projects now declares the webhook key alongside `BACKEND_MANAGER_KEY`. Required for the new `/marketing/webhook` + `/marketing/webhook/forward` routes shipped in 5.2.0. Existing consumers should add it to their own `.env` manually.
+
+# [5.2.0] - 2026-05-21
+
+### Added
+
+- **Marketing consent capture (Phase A-B).** Canonical `consent.{legal,marketing}` sub-tree on every user doc (`status` + `grantedAt` / `revokedAt` with timestamp/source/ip/text). Signup route (`src/manager/routes/user/signup/post.js`) builds the record from the client payload using server-side time + IP, defending against client-clock spoofing. `mailer.sync(uid)` is gated on `consent.marketing.status === 'granted'` — opted-out signups never enter SendGrid/Beehiiv marketing lists.
+- **Email-preferences route (Phase D).** `POST /marketing/email-preferences` now supports authenticated opt-in/opt-out (writes user doc + hits both providers via `email.sync()`/`email.remove()`) in addition to the existing HMAC anonymous unsub flow. Anonymous unsub also writes the consent revoke on the user doc with the right `source`.
+- **Cross-provider unsubscribe webhooks (Phase E).** New `POST /marketing/webhook?provider={sendgrid|beehiiv}&key=...` dispatcher with per-provider processor modules. SendGrid events (`unsubscribe`, `group_unsubscribe`, `spamreport`, `bounce`, `dropped`) and Beehiiv events (`subscription.unsubscribed`, `.deleted`, `.paused`) flip `consent.marketing.status` to `revoked`, attribute via `source`, and propagate to the OTHER provider. Idempotent via `marketing-webhooks/{eventId}` docs.
+- **Parent BEM forwarder.** `POST /marketing/webhook/forward` lets a parent BEM (one with `config.parent === 'self'`) fan webhook events out to sibling brands sharing a SendGrid account or Beehiiv publication. New `Manager.getParentUrl()`, `Manager.getParentApiUrl()`, `Manager.isParent()` helpers — children store the parent's `brand.url` with NO `api.` subdomain and the helper inserts `api.` at call time.
+- **Self-contained TEST_EXTENDED_MODE.** `src/test/runner.js` + `src/test/test-accounts.js` now do pre + post-run cleanup of SendGrid/Beehiiv contacts (the only third-party state we can't wipe at start). Pure Firestore/Auth state is still wiped only at start, per existing convention.
+- **New docs.** `docs/consent.md` (consent system + webhook flows + migration template). `docs/testing.md` updated with the post-run-cleanup exception.
+
+### Changed
+
+- `src/test/runner.js`, `src/test/utils/http-client.js`, `src/cli/commands/test.js`, plus emulator/serve/watch CLIs: renamed `hostingUrl` → `apiUrl` to match the rest of the codebase. Touches most test files for the matching context-API rename.
+- `src/manager/libraries/email/generators/newsletter.js`: uses `Manager.getParentApiUrl()` instead of reading `Manager.config.parent` directly so the `'self'` sentinel and the missing `api.` subdomain are both handled in one place.
+- `src/manager/libraries/email/providers/beehiiv.js`: exposes `getPublicationId` so generators and tests can read it without reaching into the module.
+- Disposable-domain blacklist refreshed (8 new domains) via `prepare-package`'s pre-hook.
+
+### Fixed
+
+- `mailer.sync()` and `mailer.add()` short-circuit when the target user's `consent.marketing.status === 'revoked'` so we never re-add an opted-out user.
+- Payment processor + cancel route touchups picked up by the runner-API rename pass — no behavior change, just keeping signatures aligned.
+
 # [5.1.4] - 2026-05-18
 
 ### Fixed

package/CLAUDE.md

@@ -122,6 +122,7 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 - [docs/admin-post-route.md](docs/admin-post-route.md) — `POST/PUT /admin/post` blog creation via GitHub (image extraction + `@post/` rewriting)
 - [docs/payment-system.md](docs/payment-system.md) — full payment pipeline: Intent → Webhook → On-Write → Transition; subscription model, statuses, `resolveSubscription()`, transition handlers, processor interface, product config, test processor
 - [docs/marketing-campaigns.md](docs/marketing-campaigns.md) — campaign CRUD routes, recurring campaigns, generator pipeline (newsletter), template-owned schemas, asset hosting, seed campaigns
+- [docs/consent.md](docs/consent.md) — marketing consent capture: canonical `consent.{legal,marketing}` user-doc shape, signup-form capture, account-page toggle, HMAC unsub link, SendGrid+Beehiiv webhook receivers, parent forwarder (`/marketing/webhook/forward`), migration script template
 - [docs/mcp.md](docs/mcp.md) — Model Context Protocol server: 19 tools, stdio + HTTP transports, OAuth, Claude Chat/Code configuration
 
 ### Subsystems & Libraries
@@ -133,6 +134,6 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 
 ### Testing & CLI
 
-- [docs/testing.md](docs/testing.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels
+- [docs/testing.md](docs/testing.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels. **For Firestore/Auth/local state, cleanup runs at the START of every run, never at the end** — if you add a test that writes Firestore data, register the collection/namespace in the runner's pre-test wipe list, don't add a trailing cleanup step. **Exception:** third-party providers we can't wipe at start (e.g. SendGrid/Beehiiv contact lists) get a symmetric pre + post cleanup hook in the runner — see `cleanupMarketingProviders` and [docs/consent.md](docs/consent.md). Don't pattern-match this exception for new Firestore code.
 - [docs/cli-firestore-auth.md](docs/cli-firestore-auth.md) — `npx mgr firestore:*` and `auth:*` commands, shared flags, examples
 - [docs/cli-logs.md](docs/cli-logs.md) — `npx mgr logs:read` / `logs:tail` with full flag reference and built-in Cloud Function names

package/README.md

@@ -419,6 +419,21 @@ Built-in marketing system with multi-provider support (SendGrid + Beehiiv + FCM
 
 Configure via `marketing` section in `backend-manager-config.json`. See CLAUDE.md for full documentation.
 
+## Marketing Consent
+
+GDPR/CASL-compliant consent capture and cross-provider unsubscribe sync.
+
+- **Two-checkbox signup form** — separate legal (required) and marketing (optional) consent
+- **Canonical user-doc shape** — `consent.{legal,marketing}.{status, grantedAt, revokedAt}` with full audit metadata (timestamp, source, IP, exact label text)
+- **Server-authoritative timestamps** — client timestamps ignored, defending against clock manipulation
+- **Account-page toggle** — `/account` notifications section lets logged-in users opt in/out, hits both SendGrid + Beehiiv
+- **HMAC unsubscribe links** — email-footer one-click flow continues to work
+- **Provider webhook receivers** — `POST /marketing/webhook?provider=sendgrid|beehiiv&key=X` catches unsubscribe / spam / bounce events from SendGrid and Beehiiv, writes the user doc + syncs to the OTHER provider
+- **Parent forwarder** — single public webhook endpoint (`/marketing/webhook/forward`) on the parent BEM fans out to every brand's child BEM so each one updates its own Firestore
+- **Guard against marketing sync without consent** — signup route and `email.add()` short-circuit when `consent.marketing.status !== 'granted'`
+
+See [docs/consent.md](docs/consent.md) for the full architecture, source enum reference, migration script template, and provider configuration steps.
+
 ## Helper Classes
 
 ### Assistant

package/docs/common-mistakes.md

@@ -9,3 +9,4 @@
 7. **Use short-circuit returns** — Return early from error conditions
 8. **Increment usage before update** — Call `usage.increment()` then `usage.update()`
 9. **Add Firestore composite indexes for new compound queries** — Any new Firestore query using multiple `.where()` clauses or `.where()` + `.orderBy()` requires a composite index. Add it to `src/cli/commands/setup-tests/helpers/required-indexes.js` (the SSOT). Consumer projects pick these up via `npx mgr setup`, which syncs them into `firestore.indexes.json`. Without the index, the query will crash with `FAILED_PRECONDITION` in production.
+10. **Don't put test data cleanup at the END of a test** — End-of-test cleanup doesn't fire when the previous run was killed mid-execution, so the next run inherits stale state. ALL test-data cleanup belongs in the runner's pre-test phase ([../src/test/test-accounts.js](../src/test/test-accounts.js) `deleteTestUsers()` + [../src/test/runner.js](../src/test/runner.js) `setupAccounts()`). When adding a test that writes data: register the Firestore collection in `testDataCollections` (mixed prod+test data) or `testOnlyCollections` (test-only), or use the `_test-` doc-id prefix so the id-keyed pass catches it. The only acceptable trailing cleanup is within-run state isolation (one test removes a doc so the NEXT test in the same run sees a clean slate) — that's not preparing the next run, it's intra-run housekeeping. See [testing.md](testing.md) "Test Data Cleanup".

package/docs/consent.md

@@ -0,0 +1,333 @@
+# Marketing Consent System
+
+Captures, stores, and synchronizes user consent for legal terms (ToS + Privacy) and marketing communications across SendGrid + Beehiiv. Designed for GDPR / CASL / CAN-SPAM compliance with full audit metadata.
+
+This doc covers the **server-side** (BEM) part of the system. The matching frontend pieces live in [ultimate-jekyll-manager](https://github.com/itw-creative-works/ultimate-jekyll-manager) and [web-manager](https://github.com/itw-creative-works/web-manager).
+
+## Why this exists
+
+1. **Capture explicit, affirmative consent** at signup with separate checkboxes for legal terms (required) and marketing communications (optional). Store the exact label text the user agreed to.
+2. **Let users withdraw consent at any time** via the account-page toggle or the email-footer unsubscribe link.
+3. **Stay in sync with provider-side actions** — when a user clicks unsubscribe in a SendGrid or Beehiiv email, the user doc updates AND the OTHER provider is also notified.
+4. **Never re-add an unsubscribed user** — `email.add()` and `email.sync()` short-circuit when `consent.marketing.status === 'revoked'`.
+
+## Canonical user-doc shape
+
+Every user doc has a `consent` object with two sub-trees:
+
+```js
+consent: {
+  legal: {
+    status: 'granted' | 'revoked',
+    grantedAt: { timestamp, timestampUNIX, source, ip, text },
+  },
+  marketing: {
+    status: 'granted' | 'revoked',
+    grantedAt: { timestamp, timestampUNIX, source, ip, text },
+    revokedAt: { timestamp, timestampUNIX, source, ip, text },
+  },
+}
+```
+
+**Field semantics:**
+
+- `status` — single-source-of-truth boolean state, expressed as a string enum so future states (`'pending'`, `'expired'`) don't break the schema.
+- `grantedAt` / `revokedAt` — full audit metadata for the **most recent** transition of each kind. Both ALWAYS present on the doc; nulls live at the leaves (e.g. `grantedAt: { timestamp: null, ... }`), never at the object boundary. Matches BEM's `subscription.expires` / `payment.startDate` conventions.
+- `legal` only has `grantedAt` (no `revokedAt`) because revoking legal consent = deleting the account.
+- `text` records the **exact wording** the user agreed to. Critical for audit defense if a marketing label is challenged later.
+
+**Object always present, nulls at leaves.** No special-casing required when reading — `account.consent.marketing.grantedAt.timestamp` is either an ISO string or `null`, never `undefined`.
+
+### Source enum
+
+| Source | Where it fires | Side |
+|---|---|---|
+| `'signup'` | Signup form checkbox toggled at account creation | both |
+| `'account'` | `/account` notifications page toggle | both |
+| `'admin'` | Manual admin override | both |
+| `'imported'` | Legacy user migration backfill | both |
+| `'sendgrid'` | SendGrid webhook event (`group_unsubscribe`, etc.) | revoke only |
+| `'beehiiv'` | Beehiiv webhook event (`subscription.unsubscribed`, etc.) | revoke only |
+
+## Capture points
+
+There are four places where consent gets recorded or updated. All four converge on the same canonical shape.
+
+### 1. Signup form (Phase B)
+
+[src/manager/routes/user/signup/post.js](../src/manager/routes/user/signup/post.js) — the existing `/user/signup` route now accepts a `consent` settings field:
+
+```js
+// Client sends (lightweight transit shape):
+{
+  consent: {
+    legal: { granted: true, text: 'I agree to ...' },
+    marketing: { granted: false, text: 'I agree to receive ...' },
+  },
+}
+```
+
+`buildConsentRecord(assistant, settings.consent)` translates this into the canonical user-doc shape:
+
+- `legal.granted: true` → `legal.status = 'granted'`, `grantedAt` populated with `source: 'signup'` + **server timestamp** + server-detected IP + exact label text.
+- `marketing.granted: false` → `marketing.status = 'revoked'`, `grantedAt` all-null, `revokedAt` populated with `source: 'signup'`. (Records the explicit decline.)
+- Missing `consent` block (legacy client) → both default to `'revoked'`.
+
+**Server time is authoritative.** Client-supplied timestamps are ignored — defends against clock manipulation by malicious clients.
+
+**Strict boolean check.** Only `granted === true` counts as granted. `'true'`, `1`, or other truthy values are rejected.
+
+**Marketing sync gating.** After writing the user doc, the route checks `userRecord.consent.marketing.status === 'granted'` before calling `mailer.sync(uid)`. Declining the marketing checkbox means the user is created normally, gets transactional emails, but is NEVER added to SendGrid / Beehiiv marketing lists.
+
+### 2. Account-page toggle (Phase D)
+
+[src/manager/routes/marketing/email-preferences/post.js](../src/manager/routes/marketing/email-preferences/post.js) — authenticated mode.
+
+```
+POST /backend-manager/marketing/email-preferences
+Body: { action: 'subscribe' | 'unsubscribe' }
+```
+
+- Requires authentication (uses the calling user's auth UID and email).
+- Rate-limited per-user via `Manager.Usage().init(assistant)` (5/day).
+- Writes `consent.marketing.{status, grantedAt|revokedAt}` to the user doc with `source: 'account'` + server time + server IP.
+- Calls `mailer.sync(uid)` on subscribe, `mailer.remove(email)` on unsubscribe — hits both SendGrid + Beehiiv via the email library.
+
+Note: `grantedAt.text` is `null` for account-page subscribes because the marketing label text is not currently passed from the frontend toggle (TODO if needed).
+
+### 3. HMAC unsubscribe link (legacy, Phase D)
+
+Same route, anonymous mode. The existing email-footer unsubscribe link flow:
+
+```
+POST /backend-manager/marketing/email-preferences
+Body: { email, asmId, sig, action: 'subscribe' | 'unsubscribe' }
+```
+
+- `sig = HMAC-SHA256(email, UNSUBSCRIBE_HMAC_KEY)` — proves we generated the link.
+- IP-rate-limited (5/day per IP).
+- **Also writes the user doc** if the email maps to a user — `consent.marketing.{status, revokedAt}` with `source: 'sendgrid'` (since HMAC links only appear in SendGrid email footers).
+- Backward-compatible — old in-flight email links continue to work.
+
+### 4. Provider webhooks (Phase E)
+
+[src/manager/routes/marketing/webhook/post.js](../src/manager/routes/marketing/webhook/post.js) — receives unsub / spam / bounce events from SendGrid and Beehiiv.
+
+```
+POST /backend-manager/marketing/webhook?provider=sendgrid&key=<BACKEND_MANAGER_WEBHOOK_KEY>
+POST /backend-manager/marketing/webhook?provider=beehiiv&key=<BACKEND_MANAGER_WEBHOOK_KEY>
+```
+
+The dispatcher loads `processors/{provider}.js`, parses the event(s), and for each event:
+
+1. Checks `isSupported(eventType)` — filters out non-revoke events like `delivered` / `open`.
+2. Idempotency lookup in `marketing-webhooks/{eventId}` — skips if already processed.
+3. Calls `handleEvent({ Manager, assistant, parsed })` on the processor.
+4. Marks the idempotency doc `status: 'completed'` (or `'failed'` with error details).
+
+Each processor's `handleEvent` does the same shape of work:
+
+1. Look up the user by `auth.email` in THIS brand's Firestore. Silent skip if not found (the email may belong to a sibling brand — see "Parent forwarder" below).
+2. Write `consent.marketing.status = 'revoked'` with the appropriate `source` ('sendgrid' or 'beehiiv'), preserving `grantedAt` as informational audit history.
+3. Call `mailer.remove(email)` to sync the unsubscribe to the OTHER provider (best-effort, idempotent on 404).
+
+**Supported event types:**
+
+| Provider | Event types treated as revoke |
+|---|---|
+| SendGrid | `unsubscribe`, `group_unsubscribe`, `spamreport`, `bounce`, `dropped` |
+| Beehiiv | `subscription.unsubscribed`, `subscription.deleted`, `subscription.paused` |
+
+**Beehiiv publication filter.** Each Beehiiv event includes a `publication_id`. The processor compares this against the result of `beehiivProvider.getPublicationId()` (which reads `Manager.config.marketing.beehiiv.publicationId` or fuzzy-matches against the Beehiiv API by brand name). Mismatch → silent skip. This is how shared-publication events (e.g. devbeans shared by 6 brands) get routed correctly — each brand processes only events matching its own publication.
+
+## Parent forwarder (Phase E)
+
+[src/manager/routes/marketing/webhook/forward/post.js](../src/manager/routes/marketing/webhook/forward/post.js)
+
+SendGrid and Beehiiv only let you configure a small number of webhook URLs (often one per account). With many brands sharing the same SendGrid account, we can't point the webhook at every brand's BEM directly. Instead:
+
+```
+SendGrid → POST https://api.itwcreativeworks.com/backend-manager/marketing/webhook/forward?provider=sendgrid&key=X
+Beehiiv  → POST https://api.itwcreativeworks.com/backend-manager/marketing/webhook/forward?provider=beehiiv&key=X
+```
+
+The **parent BEM** (the one whose `backend-manager-config.json` has `parent: 'self'`) exposes the forwarder route. Every other BEM has the route but it returns 404 (gated on `Manager.config.parent === 'self'`).
+
+The parent forwarder:
+
+1. Validates `?provider=X&key=Y` (same `BACKEND_MANAGER_WEBHOOK_KEY` env var — shared across all brands).
+2. Reads the `brands` collection from the parent's own Firestore.
+3. For each brand: derives the child API URL by inserting `api.` into the brand's URL (`https://somiibo.com` → `https://api.somiibo.com/backend-manager/marketing/webhook?provider=X&key=Y`).
+4. POSTs the raw provider body to every child in parallel via `Promise.allSettled`.
+5. Returns 200 even if some children fail — child idempotency makes provider retries safe.
+
+### Why fan-out instead of central processing
+
+Each brand has its own Firebase project, so its `users` collection is separate. The parent can't write to a child's Firestore. By having each child process the event against its own users, we get:
+
+- **Correct per-brand updates** — only brands where the user actually has an account update their user docs.
+- **Failure isolation** — one child being down doesn't block updates on the others.
+- **Local idempotency** — each child tracks `marketing-webhooks/{eventId}` in its own Firestore.
+- **No new schema** — no need for the parent to maintain a brand → publication map; each child filters on its own.
+
+### Why self IS in the fan-out
+
+The parent BEM has its own brand (e.g. `itw-creative-works`) with its own users. By fanning out via HTTP to itself like any other child, the parent's brand processes its users the same way as siblings — no special-case inline path.
+
+### Shared-publication scenario (Beehiiv devbeans)
+
+1. User on shared "devbeans" publication clicks unsubscribe.
+2. Beehiiv posts event with `publication_id: pub_69c961a7...` to parent's `/marketing/webhook/forward`.
+3. Parent fans out the raw event to all N brands.
+4. The 6 brands sharing the devbeans publication: `getPublicationId()` matches, they each look up the user, only the brand(s) with the user write the doc and call `mailer.remove`.
+5. The brands with dedicated publications: `getPublicationId()` mismatch, silent skip.
+
+## Email library short-circuit
+
+[src/manager/libraries/email/marketing/index.js](../src/manager/libraries/email/marketing/index.js)
+
+`email.add()` and `email.sync()` check the user's `consent.marketing.status` before contacting providers. A user marked `'revoked'` is never re-added. This is the safety net against accidental re-subscription via batch syncs or campaign sends.
+
+## Configuration
+
+### Env vars (per brand)
+
+```bash
+# All brands
+BACKEND_MANAGER_WEBHOOK_KEY="<shared-across-all-brands>"
+
+# Existing (unchanged)
+UNSUBSCRIBE_HMAC_KEY="<existing-value>"
+SENDGRID_API_KEY="<account-wide>"
+```
+
+The webhook key is shared because it has to be the same value the parent forwards to each child. Rotate by updating every brand's env in lockstep.
+
+### Provider dashboard setup
+
+**SendGrid Event Webhook** (Settings → Mail Settings → Event Webhook):
+```
+URL: https://api.itwcreativeworks.com/backend-manager/marketing/webhook/forward?provider=sendgrid&key=<BACKEND_MANAGER_WEBHOOK_KEY>
+Events: Group Unsubscribe, Unsubscribe, Spam Report, Bounce, Dropped
+```
+
+**Beehiiv Webhooks** (per-publication setup):
+- Dedicated publications: point at the single brand's parent URL.
+- Shared "devbeans" publication: point at the parent URL — fan-out handles the routing.
+
+```
+URL: https://api.itwcreativeworks.com/backend-manager/marketing/webhook/forward?provider=beehiiv&key=<BACKEND_MANAGER_WEBHOOK_KEY>
+Events: subscription.unsubscribed, subscription.deleted, subscription.paused
+```
+
+### Parent vs child config
+
+Parent's `backend-manager-config.json`:
+```js
+{
+  parent: 'self',
+  brand: { id: 'itw-creative-works', url: 'https://itwcreativeworks.com', ... },
+  ...
+}
+```
+
+Every other brand:
+```js
+{
+  parent: 'https://itwcreativeworks.com',  // NO `api.` subdomain — inserted at call time
+  brand: { id: 'somiibo', url: 'https://somiibo.com', ... },
+  ...
+}
+```
+
+**Convention:** `parent` stores the parent's brand URL (matching the format of `brand.url`), NOT the API URL. The `api.` subdomain is inserted at call time by `Manager.getParentApiUrl()`. This keeps stored config in one consistent format and lets the deployment convention (`api.` subdomain) live in one place.
+
+**Three helpers** on the Manager instance for working with this:
+
+- `Manager.getParentUrl()` — returns the parent's brand URL. Resolves `'self'` to `Manager.config.brand.url`.
+- `Manager.getParentApiUrl()` — returns the parent's API URL (`https://api.{host}`). **Always live** — does NOT redirect to localhost in dev mode, because you can't run two Firebase emulators simultaneously. The parent's API is always the production URL regardless of which environment THIS brand is in.
+- `Manager.isParent()` — boolean, true when `config.parent === 'self'`.
+
+Only the BEM where `Manager.isParent()` returns true exposes `/marketing/webhook/forward`. Everywhere else, the route is invisible (404).
+
+## Legacy user migration
+
+Existing users created BEFORE the consent system has no `consent` field. They need a one-time backfill. The shape per the agreed strategy:
+
+```js
+// For every legacy user doc
+{
+  consent: {
+    legal: {
+      status: 'granted', // implicit from active account
+      grantedAt: {
+        timestamp: userDoc.metadata?.created?.timestamp || null,
+        timestampUNIX: userDoc.metadata?.created?.timestampUNIX || null,
+        source: 'imported',
+        ip: userDoc.activity?.geolocation?.ip || null,
+        text: null, // don't fabricate label text
+      },
+    },
+    marketing: {
+      status: 'revoked', // no opt-in on record
+      grantedAt: { timestamp: null, timestampUNIX: null, source: null, ip: null, text: null },
+      revokedAt: {
+        timestamp: userDoc.metadata?.created?.timestamp || null,
+        timestampUNIX: userDoc.metadata?.created?.timestampUNIX || null,
+        source: 'imported',
+        ip: userDoc.activity?.geolocation?.ip || null,
+        text: null,
+      },
+    },
+  },
+}
+```
+
+**Idempotency guard:** skip docs where `consent.legal.grantedAt.source` already has a non-null value (those went through the new signup flow or a prior migration run).
+
+Run the migration BEFORE enabling the frontend's page-load consent guard (see UJM `ENFORCE_CONSENT_GUARD` flag in `src/assets/js/core/auth.js`). Otherwise legacy users without `consent.legal.status === 'granted'` get signed out on every page load.
+
+After the migration: optionally run a re-opt-in drip campaign to legally recover marketing consent for the users you bulk-revoked.
+
+## Test coverage
+
+**BEM tests:**
+
+- [test/helpers/user.js](../test/helpers/user.js) — 31 tests covering the canonical schema, defaults, granted/revoked states, round-tripping
+- [test/routes/user/signup.js](../test/routes/user/signup.js) — 3 tests for signup-time consent capture (granted both, marketing declined, missing payload)
+- [test/routes/marketing/email-preferences.js](../test/routes/marketing/email-preferences.js) — 14 tests for the email-preferences route (anonymous HMAC + authenticated)
+- [test/routes/marketing/webhook.js](../test/routes/marketing/webhook.js) — 15+ tests covering SendGrid + Beehiiv processors against the emulator
+- [test/routes/marketing/webhook-forward.js](../test/routes/marketing/webhook-forward.js) — verifies the forwarder route returns 404 on non-parent BEMs
+- [test/helpers/webhook-forward.js](../test/helpers/webhook-forward.js) — 12 unit-style tests with mocked admin + fetch, covering fan-out, URL derivation, failure isolation, self-inclusion, edge cases
+
+**Total: 75+ tests across the consent system.**
+
+Run with `npx mgr test` (full suite) or `npx mgr test routes/marketing/webhook` (just the webhook tests).
+
+### Provider-side cleanup (extended mode only)
+
+Normally BEM test runs are self-contained because all data lives in the local emulator. Marketing tests are the exception — when `TEST_EXTENDED_MODE=true` is set, the routes make real API calls to SendGrid + Beehiiv and leave real contacts behind.
+
+To keep this self-contained, the runner calls [`cleanupMarketingProviders`](../src/test/test-accounts.js) **twice** per run:
+
+1. **Pre-run** — scrubs leftovers from any previous run that crashed mid-suite.
+2. **Post-run** — scrubs everything this run created. Best-effort (wrapped in try/catch — cleanup failures don't change the test result).
+
+Both fire ONLY when `TEST_EXTENDED_MODE=true`. In normal mode (the default), no provider calls happen and no cleanup is needed.
+
+This is the one exception to the "Firestore cleanup runs at start only" rule documented in [docs/testing.md](testing.md) — we can't wipe SendGrid/Beehiiv state in our own Firestore wipe, so we have to ask the providers to do it for us, twice for defense in depth.
+
+## Frontend pieces (cross-references)
+
+- **UJM signup form** — [signup.html](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/defaults/dist/_layouts/themes/classy/frontend/pages/auth/signup.html) (two consent checkboxes, inline error UX)
+- **UJM auth library** — [libs/auth.js](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/assets/js/libs/auth.js) (`captureSignupConsent`, `validateConsent`, `reverseAccidentalSignup` for the Google-on-signin quirk)
+- **UJM core auth listener** — [core/auth.js](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/assets/js/core/auth.js) (`ENFORCE_CONSENT_GUARD` flag, page-load silent-signout for orphan accounts)
+- **UJM account page** — [account/index.html](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/defaults/dist/_layouts/themes/classy/frontend/pages/account/index.html) + [sections/notifications.js](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/assets/js/pages/account/sections/notifications.js)
+- **Web Manager DEFAULT_ACCOUNT** — [modules/auth.js](https://github.com/itw-creative-works/web-manager/blob/main/src/modules/auth.js) (consent fields with `'revoked'` defaults so legacy reads don't crash)
+
+## Future work
+
+- **Country-aware default checkbox state** — pre-check both checkboxes in jurisdictions where it's legally permitted (e.g. US under CAN-SPAM). Out of scope for the initial rollout; TODO comment in signup.html.
+- **Re-consent flow for material label changes** — if the marketing label text changes meaningfully, prompt existing users to re-consent (versioning via the stored `text` field).
+- **Audit-log sub-collection** — currently only the most-recent transition is kept. If legal needs full history, add `users/{uid}/consent-history/{transition-id}`.
+- **ECDSA / HMAC signature verification** on webhooks — SendGrid supports it, Beehiiv requires HMAC. Currently bearer-token only (`?key=`). Future hardening.

package/docs/testing.md

@@ -11,6 +11,42 @@ npx mgr test      # Terminal 2 - runs tests
 npx mgr test
 ```
 
+## Test Data Cleanup — at the START of every run
+
+**Hard rule: all LOCAL cleanup happens BEFORE the suite runs, never after.** If a previous run was killed mid-execution (Ctrl-C, OOM, emulator crash), end-of-run cleanup would never fire and the next run would inherit polluted state — broken trial-eligibility checks, leftover dispute alerts, stale webhook docs, polluted marketing-provider lists. Pre-test cleanup makes every run idempotent regardless of how the last one died.
+
+What the runner wipes pre-test (in [src/test/test-accounts.js](../src/test/test-accounts.js) `deleteTestUsers()` and [src/test/runner.js](../src/test/runner.js) `setupAccounts()`):
+
+1. **`meta/stats`** doc ensured (required for on-create batch writes).
+2. **`users/_test-*`** Firebase Auth users + Firestore docs (delete).
+3. **Marketing providers** (SendGrid + Beehiiv) — leftover test contacts removed. Runs only under `TEST_EXTENDED_MODE`. Runs **before** test users are recreated so a killed run can't leave a contact pinned to an about-to-be-recreated uid.
+4. **Mixed Firestore collections** — `payments-orders`, `payments-webhooks`, `payments-intents`, `payments-disputes`, `marketing-webhooks`. Two-pass cleanup per collection:
+   - Pass 1 — owner-keyed: `where('owner', 'in', [...testUids])` (batched at 30 uids per `in` query).
+   - Pass 2 — id-keyed: any doc whose ID starts with `_test-` (catches ownerless test docs like dispute alerts and raw test webhooks).
+5. **Test-only Firestore collections** — `_test`, `_test_query` — wiped in full.
+6. **Realtime Database** — the `_test` namespace removed in full (`admin.database().ref('_test').remove()`).
+
+### The one exception: third-party provider cleanup runs at start AND end
+
+The `cleanupMarketingProviders` call also fires **after** the suite completes (extended mode only). Reason: pre-run cleanup catches what a crashed previous run left behind, but during a normal-completion run we'd still leak real SendGrid/Beehiiv contacts until the NEXT run cleaned them up. By running cleanup post-suite too, each extended-mode run leaves the provider lists in the same state it found them.
+
+This is **specifically** for third-party state that lives outside the local emulator. **Do not use trailing cleanup for Firestore or Auth data** — those follow the start-only rule because we control the local state and pre-run cleanup is enough.
+
+### When adding a new test that writes data
+
+| If your test writes to... | Then... |
+|---|---|
+| A new Firestore collection (mixed test + prod data) | Add the collection name to `testDataCollections` in `src/test/test-accounts.js`. |
+| A new Firestore collection (test-only data) | Add it to `testOnlyCollections` in `src/test/test-accounts.js`. |
+| Realtime Database | Use a path under `_test/...` — already wiped pre-test. |
+| Anywhere else | Use the `_test-` doc-id prefix so id-keyed pass-2 catches it. |
+
+### Within-run state isolation is different
+
+Per-test cleanup is still appropriate when a test sets up DB state that would pollute a **later test in the same run** — e.g. trial-eligibility's `try/finally` that removes the fake `payments-orders/_test-trial-eligibility-*` doc so the next sibling test sees a clean slate. Those stay in the test. They are *intra-run* state management, not *next-run* cleanup, and the distinction matters.
+
+The rule: **never put cleanup at the END of a test file or suite for the purpose of preparing the next run** — for LOCAL state. If a test cleans up local Firestore/Auth data only to "leave no trace," that cleanup belongs in the runner's pre-test phase instead. Add the collection/namespace to the runner's wipe list and remove the trailing cleanup step. The third-party provider exception (see above) lives in the runner's post-suite hook, not in individual tests.
+
 ## Extended Mode (`TEST_EXTENDED_MODE`)
 
 Several routes/handlers skip external API calls (SendGrid, Beehiiv, Stripe webhooks, dispute handlers, marketing libraries) when `process.env.TEST_EXTENDED_MODE` is unset, so unit tests don't fire real emails or webhook side effects. Set the flag to opt **in** to those side effects for a full end-to-end run.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backend-manager",
-  "version": "5.1.4",
+  "version": "5.2.1",
   "description": "Quick tools for developing Firebase functions",
   "main": "src/manager/index.js",
   "bin": {

package/src/cli/commands/emulator.js

@@ -82,15 +82,47 @@ class EmulatorCommand extends BaseCommand {
       : 'BEM_TESTING=true';
     const emulatorCommand = `${envPrefix} firebase emulators:exec --only functions,firestore,auth,database,hosting,pubsub --ui "${command}"`;
 
-    // Set up log file in the project directory
+    // Set up log file in the project directory.
+    // We use a mutable `currentStream` so the test command can request a fresh log
+    // by touching emulator.log.reset — the watcher below detects it, closes the
+    // current stream, reopens with flags: 'w' (truncating cleanly from our process'
+    // perspective, no sparse-file issue), and deletes the sentinel.
     const logPath = path.join(projectDir, 'functions', 'emulator.log');
-    const logStream = fs.createWriteStream(logPath, { flags: 'w' });
+    const resetSentinelPath = `${logPath}.reset`;
     const stripAnsi = (str) => str.replace(/\x1B\[[0-9;]*[a-zA-Z]/g, '');
 
+    let currentStream = fs.createWriteStream(logPath, { flags: 'w' });
+
+    function writeToLog(data) {
+      if (currentStream && !currentStream.destroyed) {
+        currentStream.write(stripAnsi(data.toString()));
+      }
+    }
+
+    // Clean up any stale sentinel from a prior crashed emulator run
+    try { fs.unlinkSync(resetSentinelPath); } catch (e) { /* not present, ok */ }
+
+    // Watch for the test command's request to roll the log.
+    // Poll every 500ms — cheap, no fs.watch quirks across platforms.
+    const resetWatcher = setInterval(() => {
+      if (!fs.existsSync(resetSentinelPath)) {
+        return;
+      }
+
+      try {
+        const oldStream = currentStream;
+        currentStream = fs.createWriteStream(logPath, { flags: 'w' });
+        oldStream.end();
+        fs.unlinkSync(resetSentinelPath);
+      } catch (e) {
+        // Best-effort. If reset fails the test still runs, the log just won't be fresh.
+      }
+    }, 500);
+
     // Write pre-emulator info to log file
     if (process.env.TEST_EXTENDED_MODE) {
-      EXTENDED_MODE_WARNING.forEach((line) => logStream.write(`${line}\n`));
-      logStream.write('\n');
+      EXTENDED_MODE_WARNING.forEach((line) => writeToLog(`${line}\n`));
+      writeToLog('\n');
     }
 
     this.log(chalk.gray(`  Logs saving to: ${logPath}\n`));
@@ -106,18 +138,22 @@ class EmulatorCommand extends BaseCommand {
       // Tee stdout to both console and log file (strip ANSI codes for clean log)
       child.stdout.on('data', (data) => {
         process.stdout.write(data);
-        logStream.write(stripAnsi(data.toString()));
+        writeToLog(data);
       });
 
       // Tee stderr to both console and log file (strip ANSI codes for clean log)
       child.stderr.on('data', (data) => {
         process.stderr.write(data);
-        logStream.write(stripAnsi(data.toString()));
+        writeToLog(data);
       });
 
-      // Clean up log stream when child exits
+      // Clean up log stream + watcher when child exits
       child.on('close', () => {
-        logStream.end();
+        clearInterval(resetWatcher);
+        if (currentStream && !currentStream.destroyed) {
+          currentStream.end();
+        }
+        try { fs.unlinkSync(resetSentinelPath); } catch (e) { /* ok */ }
       });
     });
   }