backend-manager 5.1.2 → 5.2.0

package/.claude/settings.local.json

@@ -0,0 +1,12 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(grep -B2 -A20 \"Generate a newsletter preview\" /tmp/bem-test-run-3.log | head -50)",
+      "Bash(sed 's/\\\\x1b\\\\[[0-9;]*m//g')",
+      "Read(//tmp/**)",
+      "Bash(TEST_EXTENDED_MODE=1 npx mgr test 2>&1 | sed 's/\\\\x1b\\\\[[0-9;]*m//g' > /tmp/bem-run-6.log; grep -E \"passing|failing|skipped\" /tmp/bem-run-6.log | tail -5)",
+      "Bash(awk -F'`' '{print $2}')",
+      "Bash(TEST_EXTENDED_MODE=1 npx mgr test 2>&1 | sed 's/\\\\x1b\\\\[[0-9;]*m//g' > /tmp/bem-cleanup-run2.log; grep -E \"\\(passing|failing|skipped\\)\" /tmp/bem-cleanup-run2.log | tail -5)"
+    ]
+  }
+}

package/CHANGELOG.md

@@ -14,6 +14,58 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+# [5.2.0] - 2026-05-21
+
+### Added
+
+- **Marketing consent capture (Phase A-B).** Canonical `consent.{legal,marketing}` sub-tree on every user doc (`status` + `grantedAt` / `revokedAt` with timestamp/source/ip/text). Signup route (`src/manager/routes/user/signup/post.js`) builds the record from the client payload using server-side time + IP, defending against client-clock spoofing. `mailer.sync(uid)` is gated on `consent.marketing.status === 'granted'` — opted-out signups never enter SendGrid/Beehiiv marketing lists.
+- **Email-preferences route (Phase D).** `POST /marketing/email-preferences` now supports authenticated opt-in/opt-out (writes user doc + hits both providers via `email.sync()`/`email.remove()`) in addition to the existing HMAC anonymous unsub flow. Anonymous unsub also writes the consent revoke on the user doc with the right `source`.
+- **Cross-provider unsubscribe webhooks (Phase E).** New `POST /marketing/webhook?provider={sendgrid|beehiiv}&key=...` dispatcher with per-provider processor modules. SendGrid events (`unsubscribe`, `group_unsubscribe`, `spamreport`, `bounce`, `dropped`) and Beehiiv events (`subscription.unsubscribed`, `.deleted`, `.paused`) flip `consent.marketing.status` to `revoked`, attribute via `source`, and propagate to the OTHER provider. Idempotent via `marketing-webhooks/{eventId}` docs.
+- **Parent BEM forwarder.** `POST /marketing/webhook/forward` lets a parent BEM (one with `config.parent === 'self'`) fan webhook events out to sibling brands sharing a SendGrid account or Beehiiv publication. New `Manager.getParentUrl()`, `Manager.getParentApiUrl()`, `Manager.isParent()` helpers — children store the parent's `brand.url` with NO `api.` subdomain and the helper inserts `api.` at call time.
+- **Self-contained TEST_EXTENDED_MODE.** `src/test/runner.js` + `src/test/test-accounts.js` now do pre + post-run cleanup of SendGrid/Beehiiv contacts (the only third-party state we can't wipe at start). Pure Firestore/Auth state is still wiped only at start, per existing convention.
+- **New docs.** `docs/consent.md` (consent system + webhook flows + migration template). `docs/testing.md` updated with the post-run-cleanup exception.
+
+### Changed
+
+- `src/test/runner.js`, `src/test/utils/http-client.js`, `src/cli/commands/test.js`, plus emulator/serve/watch CLIs: renamed `hostingUrl` → `apiUrl` to match the rest of the codebase. Touches most test files for the matching context-API rename.
+- `src/manager/libraries/email/generators/newsletter.js`: uses `Manager.getParentApiUrl()` instead of reading `Manager.config.parent` directly so the `'self'` sentinel and the missing `api.` subdomain are both handled in one place.
+- `src/manager/libraries/email/providers/beehiiv.js`: exposes `getPublicationId` so generators and tests can read it without reaching into the module.
+- Disposable-domain blacklist refreshed (8 new domains) via `prepare-package`'s pre-hook.
+
+### Fixed
+
+- `mailer.sync()` and `mailer.add()` short-circuit when the target user's `consent.marketing.status === 'revoked'` so we never re-add an opted-out user.
+- Payment processor + cancel route touchups picked up by the runner-API rename pass — no behavior change, just keeping signatures aligned.
+
+# [5.1.4] - 2026-05-18
+
+### Fixed
+
+- **`POST /admin/post` response `path`** now returns the full `.md` file path (e.g. `src/_posts/2026/guest/2026-05-14-my-post.md`) instead of the parent directory. Consumers (e.g. the sponsorship system) were treating it as a file path — which it was named to be — and downstream deletes failed with `"sha" wasn't supplied` because GitHub got a directory listing. The parent directory is now exposed separately as `directory` for consumers that still want it. Updated `test/routes/admin/create-post.js` accordingly.
+
+# [5.1.3] - 2026-05-18
+
+### Added
+
+- **Live `TEST_EXTENDED_MODE` sync between `npx mgr test` and the running emulator.** Test command writes an allowlisted env subset to `<projectRoot>/.temp/test-mode.json` pre-flight; emulator's function workers watch the file via `fs.watch` and mutate their own `process.env` in place — flag flips take effect within ~50ms with no env coordination across terminals. No more "restart the emulator with `TEST_EXTENDED_MODE=true`" dance. Health endpoint re-reads the file as a freshness guard. New helper `src/test/utils/test-mode-file.js` is the SSOT for the file format and allowlist.
+- **Real BEM Manager in test contexts.** `run-tests.js` sets `BEM_TEST_RUNNER=1` before loading any BEM code; `Manager.init()` auto-detects this and skips Functions/server/Sentry wiring + `admin.initializeApp()` (which can't run outside a real Functions runtime). Result: tests receive `{ Manager, assistant }` in their context and can call `Manager.AI()`, `Manager.Email()`, `Manager.User()`, etc. exactly like production — no hand-rolled stubs.
+- **Newsletter markdown + summary outputs.** `lib/markdown-renderer.js` walks the same `structure` JSON the HTML is rendered from (no AI cost) and emits `newsletter.md` — each section/dispatch is a standalone `## heading` block ready to paste into Beehiiv's editor one block at a time with ad blocks inserted between dispatches. A separate `summary.md` (2-3 sentence editorial recap) is written alongside.
+- **`summary` and `tags` fields on the structure schema.** `summary` is ≤600 chars, used for the `summary.md` body and as a share snippet (distinct from preheader, which is an inbox hook). `tags` is 0-5 lowercase kebab-case topical tags, passed to Beehiiv's `content_tags` on draft creation.
+- **GitHub asset host writes MD + summary alongside HTML.** `lib/image-host.js` accepts `markdown` + `summary` parameters and uploads `newsletter.md` / `summary.md` to `{brandId}/{campaignId}/` in the same atomic two-commit upload as PNGs + HTML. URLs surface on `assets.markdownUrl` / `assets.summaryUrl`.
+- **Beehiiv fallback alert email.** When Beehiiv draft creation fails (e.g. free-plan `SEND_API_NOT_ENTERPRISE_PLAN`), the generator sends an internal alert via `sender: 'internal'` (alerts@{brandDomain}) to `brand.contact.email` containing the failure reason + subject/preheader/tags + direct links to HTML/MD/summary/folder. The newsletter is never stuck. Best-effort; failure to send is logged but never blocks the Firestore campaign-doc write.
+- **Universal AI system-prompt injections.** `Manager.AI()` `normalizeOptions()` now prepends two rules (em-dash ban, confidentiality) to every system prompt — every caller picks them up automatically.
+- **GPT-5 Codex family pricing** (`gpt-5.3-codex`, `gpt-5.2-codex`, `gpt-5.1-codex-max`, `gpt-5.1-codex`, `gpt-5.1-codex-mini`, `gpt-5-codex`, `codex-mini-latest`) added to the OpenAI provider's MODEL_TABLE.
+- **`docs/common-mistakes.md`** and **`docs/key-files.md`** — extracted from CLAUDE.md to keep the architectural overview under 250 lines.
+
+### Changed
+
+- **SVG illustrator default flipped to `gpt-5.3-codex`.** Codex is the markup/code-specialized GPT-5 variant; SVG is structured markup, so it's the right fit. Anthropic remains supported as a fallback provider.
+- **`structure` schema now requires `summary` and `tags`.** Existing generators pick these up automatically — the AI prompt was updated to instruct on both.
+- **CTAs removed from generated section bodies.** The AI cannot author URLs reliably (no browse access, no real source URLs), so any link it produced was invented. Newsletters are self-contained reads; outbound links come exclusively from the template shell's sponsorship blocks (`marketing.beehiiv.content.sponsorships[]`). Test fixtures updated.
+- **CLAUDE.md restructured** into the standard skeleton (Identity → Recommended skills → Quick Start → Architecture → CLI → File Conventions → Doc-update parity → Documentation index). Per-subsystem details extracted to `docs/`.
+- **Consumer default CLAUDE.md** updated for the new `logs:read` / `logs:tail` / `firestore:*` / `auth:*` CLI commands.
+- **Runner mode reporting.** The old `TEST_EXTENDED_MODE mismatch` warning is gone (made impossible by the live sync) — replaced with a "Mode: EXTENDED/normal" line sourced from the emulator's health-endpoint confirmation.
+
 # [5.1.2] - 2026-05-14
 
 ### Changed

package/CLAUDE.md

@@ -122,6 +122,7 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 - [docs/admin-post-route.md](docs/admin-post-route.md) — `POST/PUT /admin/post` blog creation via GitHub (image extraction + `@post/` rewriting)
 - [docs/payment-system.md](docs/payment-system.md) — full payment pipeline: Intent → Webhook → On-Write → Transition; subscription model, statuses, `resolveSubscription()`, transition handlers, processor interface, product config, test processor
 - [docs/marketing-campaigns.md](docs/marketing-campaigns.md) — campaign CRUD routes, recurring campaigns, generator pipeline (newsletter), template-owned schemas, asset hosting, seed campaigns
+- [docs/consent.md](docs/consent.md) — marketing consent capture: canonical `consent.{legal,marketing}` user-doc shape, signup-form capture, account-page toggle, HMAC unsub link, SendGrid+Beehiiv webhook receivers, parent forwarder (`/marketing/webhook/forward`), migration script template
 - [docs/mcp.md](docs/mcp.md) — Model Context Protocol server: 19 tools, stdio + HTTP transports, OAuth, Claude Chat/Code configuration
 
 ### Subsystems & Libraries
@@ -133,6 +134,6 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 
 ### Testing & CLI
 
-- [docs/testing.md](docs/testing.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels
+- [docs/testing.md](docs/testing.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels. **For Firestore/Auth/local state, cleanup runs at the START of every run, never at the end** — if you add a test that writes Firestore data, register the collection/namespace in the runner's pre-test wipe list, don't add a trailing cleanup step. **Exception:** third-party providers we can't wipe at start (e.g. SendGrid/Beehiiv contact lists) get a symmetric pre + post cleanup hook in the runner — see `cleanupMarketingProviders` and [docs/consent.md](docs/consent.md). Don't pattern-match this exception for new Firestore code.
 - [docs/cli-firestore-auth.md](docs/cli-firestore-auth.md) — `npx mgr firestore:*` and `auth:*` commands, shared flags, examples
 - [docs/cli-logs.md](docs/cli-logs.md) — `npx mgr logs:read` / `logs:tail` with full flag reference and built-in Cloud Function names

package/README.md

@@ -419,6 +419,21 @@ Built-in marketing system with multi-provider support (SendGrid + Beehiiv + FCM
 
 Configure via `marketing` section in `backend-manager-config.json`. See CLAUDE.md for full documentation.
 
+## Marketing Consent
+
+GDPR/CASL-compliant consent capture and cross-provider unsubscribe sync.
+
+- **Two-checkbox signup form** — separate legal (required) and marketing (optional) consent
+- **Canonical user-doc shape** — `consent.{legal,marketing}.{status, grantedAt, revokedAt}` with full audit metadata (timestamp, source, IP, exact label text)
+- **Server-authoritative timestamps** — client timestamps ignored, defending against clock manipulation
+- **Account-page toggle** — `/account` notifications section lets logged-in users opt in/out, hits both SendGrid + Beehiiv
+- **HMAC unsubscribe links** — email-footer one-click flow continues to work
+- **Provider webhook receivers** — `POST /marketing/webhook?provider=sendgrid|beehiiv&key=X` catches unsubscribe / spam / bounce events from SendGrid and Beehiiv, writes the user doc + syncs to the OTHER provider
+- **Parent forwarder** — single public webhook endpoint (`/marketing/webhook/forward`) on the parent BEM fans out to every brand's child BEM so each one updates its own Firestore
+- **Guard against marketing sync without consent** — signup route and `email.add()` short-circuit when `consent.marketing.status !== 'granted'`
+
+See [docs/consent.md](docs/consent.md) for the full architecture, source enum reference, migration script template, and provider configuration steps.
+
 ## Helper Classes
 
 ### Assistant
@@ -809,6 +824,21 @@ npx mgr test      # Terminal 2 - runs tests
 npx mgr test
 ```
 
+### Extended Mode (real APIs)
+
+Set `TEST_EXTENDED_MODE=true` on the **test command** to opt into real external API calls (SendGrid, Beehiiv, Stripe webhook handlers, marketing libraries). The flag flows automatically to the running emulator via `<projectRoot>/.temp/test-mode.json` — no need to set it on the emulator too:
+
+```bash
+# Terminal 1 — start once, no flag needed
+npx mgr emulator
+
+# Terminal 2 — toggle freely between runs
+TEST_EXTENDED_MODE=true npx mgr test ...   # extended mode
+npx mgr test ...                            # normal mode (next run flips back)
+```
+
+See [docs/testing.md](docs/testing.md#extended-mode-test_extended_mode) for the full mechanism.
+
 ### Filtering Tests
 
 ```bash

package/docs/common-mistakes.md

@@ -9,3 +9,4 @@
 7. **Use short-circuit returns** — Return early from error conditions
 8. **Increment usage before update** — Call `usage.increment()` then `usage.update()`
 9. **Add Firestore composite indexes for new compound queries** — Any new Firestore query using multiple `.where()` clauses or `.where()` + `.orderBy()` requires a composite index. Add it to `src/cli/commands/setup-tests/helpers/required-indexes.js` (the SSOT). Consumer projects pick these up via `npx mgr setup`, which syncs them into `firestore.indexes.json`. Without the index, the query will crash with `FAILED_PRECONDITION` in production.
+10. **Don't put test data cleanup at the END of a test** — End-of-test cleanup doesn't fire when the previous run was killed mid-execution, so the next run inherits stale state. ALL test-data cleanup belongs in the runner's pre-test phase ([../src/test/test-accounts.js](../src/test/test-accounts.js) `deleteTestUsers()` + [../src/test/runner.js](../src/test/runner.js) `setupAccounts()`). When adding a test that writes data: register the Firestore collection in `testDataCollections` (mixed prod+test data) or `testOnlyCollections` (test-only), or use the `_test-` doc-id prefix so the id-keyed pass catches it. The only acceptable trailing cleanup is within-run state isolation (one test removes a doc so the NEXT test in the same run sees a clean slate) — that's not preparing the next run, it's intra-run housekeeping. See [testing.md](testing.md) "Test Data Cleanup".

package/docs/consent.md

@@ -0,0 +1,333 @@
+# Marketing Consent System
+
+Captures, stores, and synchronizes user consent for legal terms (ToS + Privacy) and marketing communications across SendGrid + Beehiiv. Designed for GDPR / CASL / CAN-SPAM compliance with full audit metadata.
+
+This doc covers the **server-side** (BEM) part of the system. The matching frontend pieces live in [ultimate-jekyll-manager](https://github.com/itw-creative-works/ultimate-jekyll-manager) and [web-manager](https://github.com/itw-creative-works/web-manager).
+
+## Why this exists
+
+1. **Capture explicit, affirmative consent** at signup with separate checkboxes for legal terms (required) and marketing communications (optional). Store the exact label text the user agreed to.
+2. **Let users withdraw consent at any time** via the account-page toggle or the email-footer unsubscribe link.
+3. **Stay in sync with provider-side actions** — when a user clicks unsubscribe in a SendGrid or Beehiiv email, the user doc updates AND the OTHER provider is also notified.
+4. **Never re-add an unsubscribed user** — `email.add()` and `email.sync()` short-circuit when `consent.marketing.status === 'revoked'`.
+
+## Canonical user-doc shape
+
+Every user doc has a `consent` object with two sub-trees:
+
+```js
+consent: {
+  legal: {
+    status: 'granted' | 'revoked',
+    grantedAt: { timestamp, timestampUNIX, source, ip, text },
+  },
+  marketing: {
+    status: 'granted' | 'revoked',
+    grantedAt: { timestamp, timestampUNIX, source, ip, text },
+    revokedAt: { timestamp, timestampUNIX, source, ip, text },
+  },
+}
+```
+
+**Field semantics:**
+
+- `status` — single-source-of-truth boolean state, expressed as a string enum so future states (`'pending'`, `'expired'`) don't break the schema.
+- `grantedAt` / `revokedAt` — full audit metadata for the **most recent** transition of each kind. Both ALWAYS present on the doc; nulls live at the leaves (e.g. `grantedAt: { timestamp: null, ... }`), never at the object boundary. Matches BEM's `subscription.expires` / `payment.startDate` conventions.
+- `legal` only has `grantedAt` (no `revokedAt`) because revoking legal consent = deleting the account.
+- `text` records the **exact wording** the user agreed to. Critical for audit defense if a marketing label is challenged later.
+
+**Object always present, nulls at leaves.** No special-casing required when reading — `account.consent.marketing.grantedAt.timestamp` is either an ISO string or `null`, never `undefined`.
+
+### Source enum
+
+| Source | Where it fires | Side |
+|---|---|---|
+| `'signup'` | Signup form checkbox toggled at account creation | both |
+| `'account'` | `/account` notifications page toggle | both |
+| `'admin'` | Manual admin override | both |
+| `'imported'` | Legacy user migration backfill | both |
+| `'sendgrid'` | SendGrid webhook event (`group_unsubscribe`, etc.) | revoke only |
+| `'beehiiv'` | Beehiiv webhook event (`subscription.unsubscribed`, etc.) | revoke only |
+
+## Capture points
+
+There are four places where consent gets recorded or updated. All four converge on the same canonical shape.
+
+### 1. Signup form (Phase B)
+
+[src/manager/routes/user/signup/post.js](../src/manager/routes/user/signup/post.js) — the existing `/user/signup` route now accepts a `consent` settings field:
+
+```js
+// Client sends (lightweight transit shape):
+{
+  consent: {
+    legal: { granted: true, text: 'I agree to ...' },
+    marketing: { granted: false, text: 'I agree to receive ...' },
+  },
+}
+```
+
+`buildConsentRecord(assistant, settings.consent)` translates this into the canonical user-doc shape:
+
+- `legal.granted: true` → `legal.status = 'granted'`, `grantedAt` populated with `source: 'signup'` + **server timestamp** + server-detected IP + exact label text.
+- `marketing.granted: false` → `marketing.status = 'revoked'`, `grantedAt` all-null, `revokedAt` populated with `source: 'signup'`. (Records the explicit decline.)
+- Missing `consent` block (legacy client) → both default to `'revoked'`.
+
+**Server time is authoritative.** Client-supplied timestamps are ignored — defends against clock manipulation by malicious clients.
+
+**Strict boolean check.** Only `granted === true` counts as granted. `'true'`, `1`, or other truthy values are rejected.
+
+**Marketing sync gating.** After writing the user doc, the route checks `userRecord.consent.marketing.status === 'granted'` before calling `mailer.sync(uid)`. Declining the marketing checkbox means the user is created normally, gets transactional emails, but is NEVER added to SendGrid / Beehiiv marketing lists.
+
+### 2. Account-page toggle (Phase D)
+
+[src/manager/routes/marketing/email-preferences/post.js](../src/manager/routes/marketing/email-preferences/post.js) — authenticated mode.
+
+```
+POST /backend-manager/marketing/email-preferences
+Body: { action: 'subscribe' | 'unsubscribe' }
+```
+
+- Requires authentication (uses the calling user's auth UID and email).
+- Rate-limited per-user via `Manager.Usage().init(assistant)` (5/day).
+- Writes `consent.marketing.{status, grantedAt|revokedAt}` to the user doc with `source: 'account'` + server time + server IP.
+- Calls `mailer.sync(uid)` on subscribe, `mailer.remove(email)` on unsubscribe — hits both SendGrid + Beehiiv via the email library.
+
+Note: `grantedAt.text` is `null` for account-page subscribes because the marketing label text is not currently passed from the frontend toggle (TODO if needed).
+
+### 3. HMAC unsubscribe link (legacy, Phase D)
+
+Same route, anonymous mode. The existing email-footer unsubscribe link flow:
+
+```
+POST /backend-manager/marketing/email-preferences
+Body: { email, asmId, sig, action: 'subscribe' | 'unsubscribe' }
+```
+
+- `sig = HMAC-SHA256(email, UNSUBSCRIBE_HMAC_KEY)` — proves we generated the link.
+- IP-rate-limited (5/day per IP).
+- **Also writes the user doc** if the email maps to a user — `consent.marketing.{status, revokedAt}` with `source: 'sendgrid'` (since HMAC links only appear in SendGrid email footers).
+- Backward-compatible — old in-flight email links continue to work.
+
+### 4. Provider webhooks (Phase E)
+
+[src/manager/routes/marketing/webhook/post.js](../src/manager/routes/marketing/webhook/post.js) — receives unsub / spam / bounce events from SendGrid and Beehiiv.
+
+```
+POST /backend-manager/marketing/webhook?provider=sendgrid&key=<BACKEND_MANAGER_WEBHOOK_KEY>
+POST /backend-manager/marketing/webhook?provider=beehiiv&key=<BACKEND_MANAGER_WEBHOOK_KEY>
+```
+
+The dispatcher loads `processors/{provider}.js`, parses the event(s), and for each event:
+
+1. Checks `isSupported(eventType)` — filters out non-revoke events like `delivered` / `open`.
+2. Idempotency lookup in `marketing-webhooks/{eventId}` — skips if already processed.
+3. Calls `handleEvent({ Manager, assistant, parsed })` on the processor.
+4. Marks the idempotency doc `status: 'completed'` (or `'failed'` with error details).
+
+Each processor's `handleEvent` does the same shape of work:
+
+1. Look up the user by `auth.email` in THIS brand's Firestore. Silent skip if not found (the email may belong to a sibling brand — see "Parent forwarder" below).
+2. Write `consent.marketing.status = 'revoked'` with the appropriate `source` ('sendgrid' or 'beehiiv'), preserving `grantedAt` as informational audit history.
+3. Call `mailer.remove(email)` to sync the unsubscribe to the OTHER provider (best-effort, idempotent on 404).
+
+**Supported event types:**
+
+| Provider | Event types treated as revoke |
+|---|---|
+| SendGrid | `unsubscribe`, `group_unsubscribe`, `spamreport`, `bounce`, `dropped` |
+| Beehiiv | `subscription.unsubscribed`, `subscription.deleted`, `subscription.paused` |
+
+**Beehiiv publication filter.** Each Beehiiv event includes a `publication_id`. The processor compares this against the result of `beehiivProvider.getPublicationId()` (which reads `Manager.config.marketing.beehiiv.publicationId` or fuzzy-matches against the Beehiiv API by brand name). Mismatch → silent skip. This is how shared-publication events (e.g. devbeans shared by 6 brands) get routed correctly — each brand processes only events matching its own publication.
+
+## Parent forwarder (Phase E)
+
+[src/manager/routes/marketing/webhook/forward/post.js](../src/manager/routes/marketing/webhook/forward/post.js)
+
+SendGrid and Beehiiv only let you configure a small number of webhook URLs (often one per account). With many brands sharing the same SendGrid account, we can't point the webhook at every brand's BEM directly. Instead:
+
+```
+SendGrid → POST https://api.itwcreativeworks.com/backend-manager/marketing/webhook/forward?provider=sendgrid&key=X
+Beehiiv  → POST https://api.itwcreativeworks.com/backend-manager/marketing/webhook/forward?provider=beehiiv&key=X
+```
+
+The **parent BEM** (the one whose `backend-manager-config.json` has `parent: 'self'`) exposes the forwarder route. Every other BEM has the route but it returns 404 (gated on `Manager.config.parent === 'self'`).
+
+The parent forwarder:
+
+1. Validates `?provider=X&key=Y` (same `BACKEND_MANAGER_WEBHOOK_KEY` env var — shared across all brands).
+2. Reads the `brands` collection from the parent's own Firestore.
+3. For each brand: derives the child API URL by inserting `api.` into the brand's URL (`https://somiibo.com` → `https://api.somiibo.com/backend-manager/marketing/webhook?provider=X&key=Y`).
+4. POSTs the raw provider body to every child in parallel via `Promise.allSettled`.
+5. Returns 200 even if some children fail — child idempotency makes provider retries safe.
+
+### Why fan-out instead of central processing
+
+Each brand has its own Firebase project, so its `users` collection is separate. The parent can't write to a child's Firestore. By having each child process the event against its own users, we get:
+
+- **Correct per-brand updates** — only brands where the user actually has an account update their user docs.
+- **Failure isolation** — one child being down doesn't block updates on the others.
+- **Local idempotency** — each child tracks `marketing-webhooks/{eventId}` in its own Firestore.
+- **No new schema** — no need for the parent to maintain a brand → publication map; each child filters on its own.
+
+### Why self IS in the fan-out
+
+The parent BEM has its own brand (e.g. `itw-creative-works`) with its own users. By fanning out via HTTP to itself like any other child, the parent's brand processes its users the same way as siblings — no special-case inline path.
+
+### Shared-publication scenario (Beehiiv devbeans)
+
+1. User on shared "devbeans" publication clicks unsubscribe.
+2. Beehiiv posts event with `publication_id: pub_69c961a7...` to parent's `/marketing/webhook/forward`.
+3. Parent fans out the raw event to all N brands.
+4. The 6 brands sharing the devbeans publication: `getPublicationId()` matches, they each look up the user, only the brand(s) with the user write the doc and call `mailer.remove`.
+5. The brands with dedicated publications: `getPublicationId()` mismatch, silent skip.
+
+## Email library short-circuit
+
+[src/manager/libraries/email/marketing/index.js](../src/manager/libraries/email/marketing/index.js)
+
+`email.add()` and `email.sync()` check the user's `consent.marketing.status` before contacting providers. A user marked `'revoked'` is never re-added. This is the safety net against accidental re-subscription via batch syncs or campaign sends.
+
+## Configuration
+
+### Env vars (per brand)
+
+```bash
+# All brands
+BACKEND_MANAGER_WEBHOOK_KEY="<shared-across-all-brands>"
+
+# Existing (unchanged)
+UNSUBSCRIBE_HMAC_KEY="<existing-value>"
+SENDGRID_API_KEY="<account-wide>"
+```
+
+The webhook key is shared because it has to be the same value the parent forwards to each child. Rotate by updating every brand's env in lockstep.
+
+### Provider dashboard setup
+
+**SendGrid Event Webhook** (Settings → Mail Settings → Event Webhook):
+```
+URL: https://api.itwcreativeworks.com/backend-manager/marketing/webhook/forward?provider=sendgrid&key=<BACKEND_MANAGER_WEBHOOK_KEY>
+Events: Group Unsubscribe, Unsubscribe, Spam Report, Bounce, Dropped
+```
+
+**Beehiiv Webhooks** (per-publication setup):
+- Dedicated publications: point at the single brand's parent URL.
+- Shared "devbeans" publication: point at the parent URL — fan-out handles the routing.
+
+```
+URL: https://api.itwcreativeworks.com/backend-manager/marketing/webhook/forward?provider=beehiiv&key=<BACKEND_MANAGER_WEBHOOK_KEY>
+Events: subscription.unsubscribed, subscription.deleted, subscription.paused
+```
+
+### Parent vs child config
+
+Parent's `backend-manager-config.json`:
+```js
+{
+  parent: 'self',
+  brand: { id: 'itw-creative-works', url: 'https://itwcreativeworks.com', ... },
+  ...
+}
+```
+
+Every other brand:
+```js
+{
+  parent: 'https://itwcreativeworks.com',  // NO `api.` subdomain — inserted at call time
+  brand: { id: 'somiibo', url: 'https://somiibo.com', ... },
+  ...
+}
+```
+
+**Convention:** `parent` stores the parent's brand URL (matching the format of `brand.url`), NOT the API URL. The `api.` subdomain is inserted at call time by `Manager.getParentApiUrl()`. This keeps stored config in one consistent format and lets the deployment convention (`api.` subdomain) live in one place.
+
+**Three helpers** on the Manager instance for working with this:
+
+- `Manager.getParentUrl()` — returns the parent's brand URL. Resolves `'self'` to `Manager.config.brand.url`.
+- `Manager.getParentApiUrl()` — returns the parent's API URL (`https://api.{host}`). **Always live** — does NOT redirect to localhost in dev mode, because you can't run two Firebase emulators simultaneously. The parent's API is always the production URL regardless of which environment THIS brand is in.
+- `Manager.isParent()` — boolean, true when `config.parent === 'self'`.
+
+Only the BEM where `Manager.isParent()` returns true exposes `/marketing/webhook/forward`. Everywhere else, the route is invisible (404).
+
+## Legacy user migration
+
+Existing users created BEFORE the consent system has no `consent` field. They need a one-time backfill. The shape per the agreed strategy:
+
+```js
+// For every legacy user doc
+{
+  consent: {
+    legal: {
+      status: 'granted', // implicit from active account
+      grantedAt: {
+        timestamp: userDoc.metadata?.created?.timestamp || null,
+        timestampUNIX: userDoc.metadata?.created?.timestampUNIX || null,
+        source: 'imported',
+        ip: userDoc.activity?.geolocation?.ip || null,
+        text: null, // don't fabricate label text
+      },
+    },
+    marketing: {
+      status: 'revoked', // no opt-in on record
+      grantedAt: { timestamp: null, timestampUNIX: null, source: null, ip: null, text: null },
+      revokedAt: {
+        timestamp: userDoc.metadata?.created?.timestamp || null,
+        timestampUNIX: userDoc.metadata?.created?.timestampUNIX || null,
+        source: 'imported',
+        ip: userDoc.activity?.geolocation?.ip || null,
+        text: null,
+      },
+    },
+  },
+}
+```
+
+**Idempotency guard:** skip docs where `consent.legal.grantedAt.source` already has a non-null value (those went through the new signup flow or a prior migration run).
+
+Run the migration BEFORE enabling the frontend's page-load consent guard (see UJM `ENFORCE_CONSENT_GUARD` flag in `src/assets/js/core/auth.js`). Otherwise legacy users without `consent.legal.status === 'granted'` get signed out on every page load.
+
+After the migration: optionally run a re-opt-in drip campaign to legally recover marketing consent for the users you bulk-revoked.
+
+## Test coverage
+
+**BEM tests:**
+
+- [test/helpers/user.js](../test/helpers/user.js) — 31 tests covering the canonical schema, defaults, granted/revoked states, round-tripping
+- [test/routes/user/signup.js](../test/routes/user/signup.js) — 3 tests for signup-time consent capture (granted both, marketing declined, missing payload)
+- [test/routes/marketing/email-preferences.js](../test/routes/marketing/email-preferences.js) — 14 tests for the email-preferences route (anonymous HMAC + authenticated)
+- [test/routes/marketing/webhook.js](../test/routes/marketing/webhook.js) — 15+ tests covering SendGrid + Beehiiv processors against the emulator
+- [test/routes/marketing/webhook-forward.js](../test/routes/marketing/webhook-forward.js) — verifies the forwarder route returns 404 on non-parent BEMs
+- [test/helpers/webhook-forward.js](../test/helpers/webhook-forward.js) — 12 unit-style tests with mocked admin + fetch, covering fan-out, URL derivation, failure isolation, self-inclusion, edge cases
+
+**Total: 75+ tests across the consent system.**
+
+Run with `npx mgr test` (full suite) or `npx mgr test routes/marketing/webhook` (just the webhook tests).
+
+### Provider-side cleanup (extended mode only)
+
+Normally BEM test runs are self-contained because all data lives in the local emulator. Marketing tests are the exception — when `TEST_EXTENDED_MODE=true` is set, the routes make real API calls to SendGrid + Beehiiv and leave real contacts behind.
+
+To keep this self-contained, the runner calls [`cleanupMarketingProviders`](../src/test/test-accounts.js) **twice** per run:
+
+1. **Pre-run** — scrubs leftovers from any previous run that crashed mid-suite.
+2. **Post-run** — scrubs everything this run created. Best-effort (wrapped in try/catch — cleanup failures don't change the test result).
+
+Both fire ONLY when `TEST_EXTENDED_MODE=true`. In normal mode (the default), no provider calls happen and no cleanup is needed.
+
+This is the one exception to the "Firestore cleanup runs at start only" rule documented in [docs/testing.md](testing.md) — we can't wipe SendGrid/Beehiiv state in our own Firestore wipe, so we have to ask the providers to do it for us, twice for defense in depth.
+
+## Frontend pieces (cross-references)
+
+- **UJM signup form** — [signup.html](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/defaults/dist/_layouts/themes/classy/frontend/pages/auth/signup.html) (two consent checkboxes, inline error UX)
+- **UJM auth library** — [libs/auth.js](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/assets/js/libs/auth.js) (`captureSignupConsent`, `validateConsent`, `reverseAccidentalSignup` for the Google-on-signin quirk)
+- **UJM core auth listener** — [core/auth.js](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/assets/js/core/auth.js) (`ENFORCE_CONSENT_GUARD` flag, page-load silent-signout for orphan accounts)
+- **UJM account page** — [account/index.html](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/defaults/dist/_layouts/themes/classy/frontend/pages/account/index.html) + [sections/notifications.js](https://github.com/itw-creative-works/ultimate-jekyll-manager/blob/main/src/assets/js/pages/account/sections/notifications.js)
+- **Web Manager DEFAULT_ACCOUNT** — [modules/auth.js](https://github.com/itw-creative-works/web-manager/blob/main/src/modules/auth.js) (consent fields with `'revoked'` defaults so legacy reads don't crash)
+
+## Future work
+
+- **Country-aware default checkbox state** — pre-check both checkboxes in jurisdictions where it's legally permitted (e.g. US under CAN-SPAM). Out of scope for the initial rollout; TODO comment in signup.html.
+- **Re-consent flow for material label changes** — if the marketing label text changes meaningfully, prompt existing users to re-consent (versioning via the stored `text` field).
+- **Audit-log sub-collection** — currently only the most-recent transition is kept. If legal needs full history, add `users/{uid}/consent-history/{transition-id}`.
+- **ECDSA / HMAC signature verification** on webhooks — SendGrid supports it, Beehiiv requires HMAC. Currently bearer-token only (`?key=`). Future hardening.

package/docs/marketing-campaigns.md

@@ -152,7 +152,17 @@ AI provider defaults live in code (openai for structure, anthropic for SVG — e
 
 ## Asset hosting (production cron flow)
 
-The daily cron uploads PNGs + the rendered `newsletter.html` to the public `itw-creative-works/newsletter-assets` repo as two atomic Git Trees commits per issue. Folder layout: `{brandId}/{campaignId}/section-N.png` + `{brandId}/{campaignId}/newsletter.html`.
+The daily cron uploads per-section PNGs + the rendered `newsletter.html` + `newsletter.md` + `summary.md` to the public `itw-creative-works/newsletter-assets` repo as two atomic Git Trees commits per issue (PNGs first so URLs exist for embedding, then HTML/MD/summary in a second commit). Folder layout:
+
+```
+{brandId}/{campaignId}/
+  section-N.png       — per-section illustration (embedded in HTML)
+  newsletter.html     — final rendered email-safe HTML
+  newsletter.md       — programmatic markdown view (per-section ## blocks, ready for Beehiiv paste)
+  summary.md          — short editorial recap (2-3 sentences)
+```
+
+`newsletter.md` is built programmatically from the same `structure` JSON the HTML is rendered from (no AI cost) by `lib/markdown-renderer.js`. Each section/dispatch becomes a standalone `## heading` block — drop it into the Beehiiv editor one block at a time and insert ad blocks between dispatches.
 
 The `campaignId` is the same Firestore doc ID the cron uses for the generated `marketing-campaigns/{newId}` doc, reserved up front so the GitHub URLs and the Firestore doc always match.
 
@@ -164,14 +174,40 @@ marketing-campaigns/{newId}: {
   assets: {
     campaignId,           // same as the doc id
     folderUrl,            // https://github.com/itw-creative-works/newsletter-assets/tree/main/{brandId}/{campaignId}
-    htmlUrl,              // https://raw.githubusercontent.com/.../newsletter.html  — paste this into Beehiiv
-    imageUrls: [...]      // raw.githubusercontent.com URLs already embedded in contentHtml
+    htmlUrl,              // https://raw.githubusercontent.com/.../newsletter.html  — paste this into Beehiiv as one block
+    markdownUrl,          // https://raw.githubusercontent.com/.../newsletter.md    — per-section blocks (ads between)
+    summaryUrl,           // https://raw.githubusercontent.com/.../summary.md       — share-snippet recap
+    imageUrls: [...],     // raw.githubusercontent.com URLs already embedded in contentHtml
+    beehiivPostId,        // ID of the draft post created on Beehiiv (null if disabled/failed)
+    tags: [...]           // AI-generated content tags (also passed to Beehiiv `content_tags`)
   },
   meta:        { tokens, cost, durations, source scores },
   ...
 }
 ```
 
+**`structure` schema (universals)** — every newsletter the generator produces satisfies this regardless of template:
+
+| Field | Purpose |
+|---|---|
+| `subject` | Email subject (≤80 chars) |
+| `preheader` | Inbox preview text (≤120 chars) |
+| `summary` | 2-3 sentence editorial recap (≤600 chars) — written to `summary.md` and used as share snippet. Distinct from preheader (which is an inbox hook). |
+| `tags` | 0-5 topical tags (lowercase kebab-case) — passed to Beehiiv `content_tags` |
+| `signoff` | Two-line closing |
+| `citations` | 0-10 `{note, source}` pairs rendered as footnotes |
+
+Templates add their own fields on top (e.g. classic adds `intro` + `sections`; field-report adds `tldr` + `dateline` + `dispatches`).
+
+**No CTAs in generated content.** The schema intentionally does NOT include section-level CTAs / outbound links. The AI cannot author URLs reliably — it has no browse access to your site and no real source URLs to reference, so any URL it produces is invented. Newsletters are self-contained reads; outbound links come from sponsorship blocks rendered by the template shell (driven by `marketing.beehiiv.content.sponsorships[]`), not from generated section bodies.
+
+**Beehiiv failure → fallback alert email.** When Beehiiv draft creation fails (e.g. `SEND_API_NOT_ENTERPRISE_PLAN` on the free plan), the generator sends an internal alert email via `sender: 'internal'` (resolves to `alerts@{brandDomain}`) to `brand.contact.email` with:
+- The failure reason
+- Subject, preheader, tags
+- Direct links to the rendered HTML, per-section markdown, summary, and the full GitHub folder
+
+This means the newsletter is never "stuck" — even with Beehiiv disabled or failing, you get an actionable email pointing to ready-to-paste assets. The alert is best-effort; failure to send is logged but does not block the Firestore campaign-doc write.
+
 Requires `GITHUB_TOKEN` env var (org-scoped, write access to `newsletter-assets`). Without it, the cron's HTML/image upload calls throw and the run aborts.
 
 ## Iteration test asset story
@@ -232,7 +268,8 @@ marketing: {
 | Newsletter copy (AI) | `src/manager/libraries/email/generators/lib/structure.js` |
 | Newsletter SVG (AI) | `src/manager/libraries/email/generators/lib/svg-illustrator.js` |
 | Newsletter MJML → HTML | `src/manager/libraries/email/generators/lib/mjml-template.js` |
-| Newsletter asset host (GitHub upload — PNGs + newsletter.html) | `src/manager/libraries/email/generators/lib/image-host.js` |
+| Newsletter asset host (GitHub upload — PNGs + newsletter.html + newsletter.md + summary.md) | `src/manager/libraries/email/generators/lib/image-host.js` |
+| Newsletter markdown renderer (programmatic, no AI) | `src/manager/libraries/email/generators/lib/markdown-renderer.js` |
 | Unified AI library | `src/manager/libraries/ai/index.js` (OpenAI + Anthropic via `Manager.AI(assistant).request({ provider, ... })`) |
 | Notification library | `src/manager/libraries/notification.js` |
 | SendGrid provider | `src/manager/libraries/email/providers/sendgrid.js` |