backend-manager 5.2.3 → 5.2.6

package/CHANGELOG.md

@@ -14,6 +14,58 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+# [5.2.6] - 2026-05-24
+
+### Added
+
+- **`scripts/test-helper-providers.js`** — small CLI for live-test verification. Run from any consumer's `functions/` dir: `node <bem>/scripts/test-helper-providers.js find <email>` (check SendGrid + Beehiiv state without dashboards) or `purge <email>` (remove from both providers). Used during end-to-end consent-pipeline testing.
+- **Email-template namespacing.** `general/email` route now translates `:` in `settings.id` to a folder separator, so `general:download-app-link` resolves to `templates/general/download-app-link.js`. Existing `templates/download-app-link.js` moved into `templates/general/` to match.
+
+### Changed
+
+- **Payment webhook + dispute-alert routes now accept either `BACKEND_MANAGER_WEBHOOK_KEY` (preferred) or `BACKEND_MANAGER_KEY` (legacy).** Phase 1 of the webhook-key migration plan in `TODO-WEBHOOK-KEY-UPGRADE.md` — non-breaking dual-acceptance so consumers can roll over at their own pace. `src/manager/routes/payments/webhook/post.js:28` + `src/manager/routes/payments/dispute-alert/post.js:20` validate against either env var. Test-processor self-fire in `src/manager/routes/payments/intent/processors/test.js:158` now uses `BACKEND_MANAGER_WEBHOOK_KEY` directly. All 13 payment test files (`test/routes/payments/*.js`, `test/events/payments/journey-*.js`) updated to use `BACKEND_MANAGER_WEBHOOK_KEY` for webhook URLs. `src/test/utils/http-client.js` + `src/test/runner.js` thread the new env var through the test context. `docs/stripe-webhook-forwarding.md` reflects the dual-key acceptance. Phase 2 (drop the legacy fallback) is tracked in `TODO-WEBHOOK-KEY-LEGACY-REMOVAL.md`.
+- **`npx mgr emulator` no longer wraps the emulator in a keep-alive subprocess.** Previously spawned a sleep-86400 wrapper via `runWithEmulator`; now spawns the emulator child directly and uses a clean SIGINT handler. Cleaner shutdown, no orphaned subprocesses, no `node-powertools` dep on the boot path. `npx mgr test`'s auto-start path uses the same helper.
+
+### Fixed
+
+- **Finished the v5.2.3 SendGrid timeout bump that missed 5 sites.** v5.2.3's CHANGELOG claimed "All 9 fetch sites updated" but actually 5 sites in `sendgrid.js` (including `upsertContacts:118` — the hot path used by `Marketing.sync()`) were still using `timeout: 15000`. Confirmed live on Somiibo: a signup with marketing consent granted silently dropped both SendGrid and Beehiiv with `Request timed out` after ~18s. Now every API call in `sendgrid.js` goes through the `SENDGRID_TIMEOUT_MS` (60s) constant. The only remaining literal is the S3 CSV download in `getSegmentContacts:578` (30s — not a SendGrid API call).
+- **`beehiiv.js` timeouts unified under a new `BEEHIIV_TIMEOUT_MS` (60s) constant.** Two hot-path sites (`addSubscriber:71` and the unsub endpoint at `:455`) were still on `timeout: 15000`. Same silent-drop failure mode as SendGrid above. Every Beehiiv API call now uses the constant.
+- **Audited entire BEM codebase for short timeouts. Zero prod-path timeouts under 60s remain.**
+  - **Bumped 6 sites from 10-15s → 60s:**
+    - `src/manager/libraries/email/validation.js:136` — ZeroBounce mailbox check
+    - `src/manager/libraries/email/generators/newsletter.js:520, 549` — parent-server `/newsletter-sources` GET + PUT
+    - `src/manager/routes/payments/intent/processors/test.js:163` — test-processor self-fire webhook
+    - `src/manager/routes/marketing/email-preferences/post.js:169, 179` — SendGrid ASM suppression POST + DELETE
+  - **Bumped 9 sites from 30s → 60s:**
+    - `src/manager/libraries/infer-contact.js:46` — PeopleDataLabs AI enrichment
+    - `src/manager/libraries/email/providers/sendgrid.js:578` — S3 CSV download for segment exports
+    - `src/manager/functions/core/actions/api/user/delete.js:34` + `src/manager/routes/user/delete.js:51` — internal sign-out fan-out
+    - `src/manager/events/firestore/payments-webhooks/analytics.js:228, 301` — Facebook + Reddit Conversions API
+    - `src/manager/routes/user/oauth2/providers/discord.js:25` + `…/google.js:30` — OAuth token-revoke
+    - `src/manager/helpers/analytics.js:331` — itwcw-package-analytics event fire
+  - **Bumped 2 sites from 30s → 120s:**
+    - `src/manager/events/cron/daily/ghostii-auto-publisher.js:106` — gpt-image-1 hero image generation (regularly 30-60s)
+    - `src/manager/events/cron/daily/ghostii-auto-publisher.js:177` — gpt-5+web-search blog post generation (regularly 60-90s)
+- **Audit rule going forward:** any external-API call in BEM prod paths uses **60s minimum** (120s+ for LLM/image generation). Short timeouts are silent-failure machines in serverless code where Cloud Function timeouts already give us a generous outer bound. `_legacy/` files, `src/test/` infra, and `src/cli/` commands keep their existing timeouts — they're not on the request hot path.
+
+# [5.2.5] - 2026-05-22
+
+### Added
+
+- **`utilities.trim(input)`** in `src/manager/helpers/utilities.js`. Walks objects/arrays recursively and trims whitespace on every string. Does NOT strip HTML. Middleware uses it to clean up incoming request data without mangling URLs, Markdown, or any payload with `<`/`>`/`&`.
+- **`findContact(email)`** on the SendGrid + Beehiiv providers, and **`findSubscriber(email, publicationId)`** on Beehiiv. Both extracted from the existing `removeContact`/`removeSubscriber` search-by-email logic so tests (and other call sites) can verify provider state without forcing a delete.
+- **`test/marketing/consent-lifecycle.js`** — 5-phase live integration test (gated on `TEST_EXTENDED_MODE=true`) that walks two long-lived sentinel accounts through pre-check → sync → declined-stays-out → unsubscribe → validation-gate, asserting actual SendGrid + Beehiiv state at every step. Uses `pollProvider()` to handle SendGrid's async background jobs (upsert/delete can take 10-60s+ to surface).
+
+### Changed
+
+- **Middleware no longer strips HTML by default.** The auto-`sanitize-html` pass on incoming request data was mangling legitimate input — URL query strings (`?a=1&b=2` → `?a=1&amp;b=2`), Markdown, and any payload containing `<`/`>`/`&`. The middleware now only trims whitespace by default. HTML sanitization is opt-in per route via `Manager.Middleware(req, res).run('route', { sanitize: true })`. Sanitize at the HTML-insertion site (`utilities.sanitize()` in your template/email render) rather than at the request boundary. The existing schema-level `sanitize: false` opt-out still works when route-level sanitize is on.
+- **Test sentinel accounts `consent-granted` and `consent-declined` now use the `_test.allow_*` prefix** (`_test.allow_consent-granted@...`, `_test.allow_consent-declined@...`). The `_test.*` validation block from v5.2.3 also blocks the previous `_test.consent-*` names from reaching providers — `_test.allow_*` is the carved-out exception specifically for live-provider integration tests.
+- **Beehiiv API timeouts bumped to 60s** on the two remaining stragglers (`findSubscriber`, `resolveSegmentIds`) to match the SendGrid+Beehiiv 60s convention from v5.2.3.
+
+### Removed
+
+- **`cleanupMarketingProviders()` function and its pre-run/post-run hooks** in `src/test/test-accounts.js` + `src/test/runner.js`. No longer needed — the validation gate added in v5.2.3 blocks `_test.*` emails from reaching SendGrid + Beehiiv upstream, so there's nothing to clean up. The new `consent-lifecycle.js` test (which intentionally round-trips real contacts via `_test.allow_*`) manages its own pre-check force-clean inline.
+
 # [5.2.3] - 2026-05-22
 
 ### Added

package/CLAUDE.md

@@ -76,7 +76,7 @@ See [docs/cli-firestore-auth.md](docs/cli-firestore-auth.md) and [docs/cli-logs.
 - **Firestore shorthand**: `admin.firestore().doc('users/abc123')` (path string) rather than `.collection('users').doc('abc123')`.
 - **Template strings for requires**: `` require(`${functionsDir}/node_modules/backend-manager`) `` rather than string concat.
 - **No backwards compatibility** unless explicitly requested.
-- **Routes receive sanitized data by default** — see [docs/sanitization.md](docs/sanitization.md) for opt-out rules.
+- **Routes receive whitespace-trimmed data; HTML is preserved.** Sanitize at the HTML-insertion site via `utilities.sanitize()`. Opt into middleware-level HTML strip per-route with `{ sanitize: true }`. See [docs/sanitization.md](docs/sanitization.md).
 - **Match schema names to route names** — if route is `myEndpoint`, schema is `myEndpoint`.
 - **Always use `assistant.respond()` for responses** — do NOT use `res.send()` directly.
 - **Add Firestore composite indexes** for any compound query (`where` + `orderBy`, or multiple `where`s) to `src/cli/commands/setup-tests/helpers/required-indexes.js` (the SSOT). Without the index, queries crash with `FAILED_PRECONDITION` in production.
@@ -113,7 +113,7 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 
 - [docs/routes.md](docs/routes.md) — recipes for new API commands, routes, event handlers, cron jobs (with code templates)
 - [docs/schemas.md](docs/schemas.md) — schema definition format, defaults vs premium overrides
-- [docs/sanitization.md](docs/sanitization.md) — automatic XSS sanitization, schema opt-out (`sanitize: false`), route-level opt-out, manual `utilities.sanitize()`
+- [docs/sanitization.md](docs/sanitization.md) — middleware trim-only default; opt-in HTML strip (`{ sanitize: true }`) with per-field opt-out (`sanitize: false`); manual `utilities.sanitize()` for HTML-insertion sites
 - [docs/auth-hooks.md](docs/auth-hooks.md) — consumer hooks for `before-create`/`before-signin`/`on-create`/`on-delete` (blocking + non-blocking examples)
 - [docs/common-operations.md](docs/common-operations.md) — inside-the-handler patterns: authenticate, read/write Firestore, error handling, send response, `bm_api` hook
 
@@ -134,6 +134,6 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 
 ### Testing & CLI
 
-- [docs/testing.md](docs/testing.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels. **For Firestore/Auth/local state, cleanup runs at the START of every run, never at the end** — if you add a test that writes Firestore data, register the collection/namespace in the runner's pre-test wipe list, don't add a trailing cleanup step. **Exception:** third-party providers we can't wipe at start (e.g. SendGrid/Beehiiv contact lists) get a symmetric pre + post cleanup hook in the runner — see `cleanupMarketingProviders` and [docs/consent.md](docs/consent.md). Don't pattern-match this exception for new Firestore code.
+- [docs/testing.md](docs/testing.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels. **All cleanup runs at the START of every run, never at the end.** If you add a test that writes Firestore data, register the collection/namespace in the runner's pre-test wipe list, don't add a trailing cleanup step. Marketing providers (SendGrid/Beehiiv) don't need a special exception — `_test.*` emails are blocked at the validation layer so test signups never reach providers. The `_test.allow_*` carve-out exists only for the live-provider lifecycle test (`test/marketing/consent-lifecycle.js`), which manages its own teardown.
 - [docs/cli-firestore-auth.md](docs/cli-firestore-auth.md) — `npx mgr firestore:*` and `auth:*` commands, shared flags, examples
 - [docs/cli-logs.md](docs/cli-logs.md) — `npx mgr logs:read` / `logs:tail` with full flag reference and built-in Cloud Function names

package/TODO-CANCEL-EMAIL-MISSING-ORDER-ID.md

@@ -0,0 +1,159 @@
+# TODO: Cancel email missing Order # on non-trial cancels
+
+## Symptom
+
+Observed 2026-05-23 in `_test.journey-payments-cancel@somiibo.com`'s inbox:
+
+```
+Subject: Your subscription has been cancelled #
+Body:    Order #
+         Your subscription has been cancelled.
+```
+
+The `#` after both `Subject` and `Order #` is bare — `order.id` rendered as empty string.
+
+The trial-cancel sibling (`_test.journey-payments-trial-cancel@somiibo.com`) on the SAME run got the order ID correctly:
+
+```
+Subject: Your subscription has been cancelled #3794-5306-7041
+Body:    Order #3794-5306-7041
+```
+
+So this is path-specific, not a template bug.
+
+## Root cause (narrowed but not yet proven)
+
+Two cancel paths feed the same `subscription-cancelled` transition handler ([src/manager/events/firestore/payments-webhooks/transitions/subscription/subscription-cancelled.js:17](src/manager/events/firestore/payments-webhooks/transitions/subscription/subscription-cancelled.js#L17)):
+
+```js
+subject: `Your subscription has been cancelled #${order?.id || ''}`,
+```
+
+Subject + template both render `order.id`. When `order.id` is falsy it falls through to `''`.
+
+`order.id` is built in [src/manager/events/firestore/payments-webhooks/on-write.js:241-242](src/manager/events/firestore/payments-webhooks/on-write.js#L241-L242):
+
+```js
+const order = {
+  id: orderId,
+  ...
+};
+```
+
+Where `orderId` (line 118) is:
+
+```js
+orderId = library.getOrderId(resource) || passThruOrderId;
+```
+
+`library.getOrderId(resource)` reads the order ID off the processor resource's `meta_data` field. `passThruOrderId` is a PayPal-only fallback from the hosted-page `pass_thru_content`.
+
+**Working path** (`journey-payments-trial-cancel`): the trial cancel goes through the `/payments/cancel` endpoint, which already has the `orderId` from `users/{uid}.subscription.payment.orderId`. By the time Stripe/PayPal fires the cancellation webhook back, `meta_data.orderId` on the subscription resource is set, so `library.getOrderId(resource)` returns the order ID. Email shows the ID.
+
+**Broken path** (`journey-payments-cancel`): this is the webhook-driven cancel (no `/payments/cancel` endpoint call — `orderDoc.requests.cancellation` is null in the test assertions). The webhook arrives with a subscription resource whose `meta_data.orderId` is empty (or `library.getOrderId(resource)` returns null/empty string). `passThruOrderId` is null because this is Stripe, not PayPal. `orderId` ends up null. `order.id = null`. Email renders `Order #`.
+
+## What's still unknown
+
+1. **Why is `meta_data.orderId` missing on the Stripe subscription resource at cancel time?** Possibilities:
+   - The journey test never set `meta_data` on the subscription (only on the customer or the initial intent).
+   - `library.setMetaData` was called during the new-subscription transition but Stripe's subscription resource didn't persist it (Stripe quirk where `metadata` on subscription vs. customer vs. invoice diverge).
+   - The `customer.subscription.deleted` event payload doesn't include `metadata` even when it was set — this would be a Stripe-side gotcha.
+2. **Does this affect production?** Possibly — depends on whether real users' Stripe subscription resources have `meta_data.orderId` set. Worth checking one real cancelled subscription in the Stripe dashboard.
+
+## Investigation steps
+
+1. **Re-run the cancel journey with `TEST_EXTENDED_MODE=true`** and watch the BEM logs:
+   ```bash
+   cd /Users/ian/Developer/Repositories/Somiibo/somiibo-backend/functions
+   TEST_EXTENDED_MODE=true npx mgr test events/payments/journey-payments-cancel
+   ```
+   Search the logs around the cancellation webhook for the resolved `orderId` value:
+   ```bash
+   npx mgr logs:read --filter "journey-payments-cancel" --limit 200 | grep -i "orderid\|order id"
+   ```
+
+2. **Inspect the raw webhook payload** stored in `payments-webhooks/{eventId}` for that test run:
+   ```bash
+   npx mgr firestore:query "payments-webhooks" --limit 20  # find the deleted-subscription event
+   npx mgr firestore:get "payments-webhooks/<eventId>"     # inspect raw.data.object.metadata
+   ```
+   If `raw.data.object.metadata.orderId` is empty → confirms Stripe didn't include it in the event.
+
+3. **Check `library.getOrderId(resource)` implementation** for the Stripe library:
+   ```bash
+   grep -n "getOrderId" /Users/ian/Developer/Repositories/ITW-Creative-Works/backend-manager/src/manager/libraries/payment/stripe.js
+   ```
+   Confirm what field path it reads. If it reads only `resource.metadata.orderId`, that's the bug — should also check `resource.metadata.bm_orderId` (some processors have different conventions) or fall through to a Firestore lookup.
+
+## Fix options (in order of preference)
+
+### Option A — Fall back to userDoc on the transition side (safest)
+
+In [subscription-cancelled.js:17](src/manager/events/firestore/payments-webhooks/transitions/subscription/subscription-cancelled.js#L17), if `order.id` is missing, fall back to `userDoc.subscription.payment.orderId`:
+
+```js
+const orderId = order?.id || userDoc?.subscription?.payment?.orderId || '';
+
+sendOrderEmail({
+  template: 'core/order/cancelled',
+  subject: `Your subscription has been cancelled${orderId ? ` #${orderId}` : ''}`,
+  ...
+  data: {
+    order: {
+      ...order,
+      id: orderId,
+      ...
+    },
+  },
+});
+```
+
+This is the **safest** fix because it works regardless of whether the bug is in `library.getOrderId`, in Stripe's event payload, or in how `meta_data` was set originally. `userDoc.subscription.payment.orderId` is written when the subscription first activates and reliably has the ID by cancel time.
+
+The subject also degrades gracefully — `Your subscription has been cancelled` (no bare `#`) when the ID truly can't be found.
+
+### Option B — Fix at the source (`on-write.js`)
+
+In [on-write.js:118](src/manager/events/firestore/payments-webhooks/on-write.js#L118), add another fallback after `library.getOrderId`:
+
+```js
+orderId = library.getOrderId(resource)
+  || passThruOrderId
+  || userDoc?.subscription?.payment?.orderId
+  || null;
+```
+
+This fixes it for ALL handlers (not just cancel), and at the cost of a Firestore read of the user doc earlier in `on-write.js` than today. May not need a new read if userDoc is already fetched by this point — confirm by reading the function top-to-bottom.
+
+### Option C — Fix `library.getOrderId(resource)` (Stripe library)
+
+If investigation step 3 shows the Stripe library is reading the wrong field path, fix it there. Smallest blast radius if the bug is genuinely in one processor's resolver and not a Stripe-event-payload gotcha.
+
+## Recommendation
+
+Do **A + B together** — A makes the email correct today regardless of root cause; B prevents the same gap from biting any other handler (refund, suspend, etc.). C only if investigation step 3 reveals an actual mis-read.
+
+## Regression test
+
+Add an assertion to `test/events/payments/journey-payments-cancel.js` that the post-cancellation userDoc still has `subscription.payment.orderId` set AND that the cancel email's `order.id` would render non-empty. Without a regression test, the same bug will silently come back the next time someone refactors `library.getOrderId`.
+
+Pseudo-assertion:
+
+```js
+assert.ok(userDoc.subscription?.payment?.orderId, 'userDoc.subscription.payment.orderId should be set after cancellation');
+// And if we capture the email payload (via a test-mode hook in sendOrderEmail), assert order.id !== '' in the rendered data.
+```
+
+If `sendOrderEmail` doesn't already have a test-mode capture, that's a small enhancement worth doing once — it would let every email-emitting transition assert its rendered payload, not just cancel.
+
+## Affected versions
+
+Confirmed broken: BEM 5.2.5 (current live on Somiibo as of 2026-05-23). Likely broken in earlier versions too — the `order?.id || ''` template defensive fallback was added explicitly to handle missing IDs, suggesting this gap has been latent for a while. The trial-cancel path masked it because trial cancels go through the endpoint and have `orderId` set in a different code path.
+
+## Related files
+
+- [src/manager/events/firestore/payments-webhooks/transitions/subscription/subscription-cancelled.js](src/manager/events/firestore/payments-webhooks/transitions/subscription/subscription-cancelled.js) — handler with `order?.id || ''` fallback
+- [src/manager/events/firestore/payments-webhooks/on-write.js:241](src/manager/events/firestore/payments-webhooks/on-write.js#L241) — `order` object construction
+- [src/manager/events/firestore/payments-webhooks/on-write.js:118](src/manager/events/firestore/payments-webhooks/on-write.js#L118) — `orderId` resolution from `library.getOrderId` + pass-through fallback
+- [test/events/payments/journey-payments-cancel.js](test/events/payments/journey-payments-cancel.js) — broken-path test (currently passes because it doesn't assert the email payload)
+- [test/events/payments/journey-payments-trial-cancel.js](test/events/payments/journey-payments-trial-cancel.js) — working-path test

package/TODO-WEBHOOK-KEY-LEGACY-REMOVAL.md

@@ -0,0 +1,15 @@
+# TODO: Remove legacy BACKEND_MANAGER_KEY acceptance from webhook routes
+
+When v5.2.x shipped, the two payment-webhook routes were switched to prefer `BACKEND_MANAGER_WEBHOOK_KEY` but still accept `BACKEND_MANAGER_KEY` as a transitional fallback so existing provider URLs (registered with the old key) keep working until OMEGA re-registers them with the new key.
+
+Once every brand's provider URLs (Stripe / PayPal / Chargebee / Chargeblast / Coinbase / etc.) have been re-registered via OMEGA to use `BACKEND_MANAGER_WEBHOOK_KEY`, drop the legacy fallback:
+
+## Files to update
+
+- `src/manager/routes/payments/webhook/post.js` — line ~28: remove the `|| key === process.env.BACKEND_MANAGER_KEY` half of the validation check.
+- `src/manager/routes/payments/dispute-alert/post.js` — line ~20: same removal. Also update the docstring at line ~11 to drop the "BACKEND_MANAGER_KEY accepted as legacy fallback" note.
+- `docs/stripe-webhook-forwarding.md` — drop the "BACKEND_MANAGER_KEY also accepted as a legacy fallback" parenthetical.
+
+## How to verify it's safe
+
+Grep production logs (`npx mgr logs:read --fn bm_api --grep "payments/webhook" --limit 1000`) and confirm zero `401 Invalid key` hits over a meaningful window. If there are any, those are providers still on the old URL — re-register them via OMEGA before dropping the fallback.

package/TODO-WEBHOOK-KEY-UPGRADE.md

@@ -0,0 +1,138 @@
+# TODO: Payment Webhook Key Upgrade — `BACKEND_MANAGER_KEY` → `BACKEND_MANAGER_WEBHOOK_KEY`
+
+## Context
+
+In BEM v5.2.0 we introduced `BACKEND_MANAGER_WEBHOOK_KEY` as a dedicated env var for third-party webhook authentication, scoped narrowly so it can be rotated independently of the admin key. The marketing routes (`/marketing/webhook`, `/marketing/webhook/forward`) use it correctly.
+
+The **payment webhook** (`/payments/webhook`) and **dispute-alert webhook** (`/payments/dispute-alert`) still validate against `BACKEND_MANAGER_KEY` — the general-purpose admin key. This is a security misalignment: a leaked admin key could forge payment webhooks, and rotating the admin key forces every Stripe/PayPal/Chargebee webhook URL to be re-registered.
+
+Plan: gradual rollout. Phase 1 (this BEM release) makes the payment routes accept **either** key so existing brand deploys (with only `BACKEND_MANAGER_KEY` set) keep working AND new brands can start using `BACKEND_MANAGER_WEBHOOK_KEY` immediately. Phase 2 (a later BEM release) drops the legacy fallback after every brand has been migrated.
+
+## Files needing the dual-key check
+
+| File | Line | Current |
+|---|---|---|
+| [src/manager/routes/payments/webhook/post.js](src/manager/routes/payments/webhook/post.js) | 28 | `if (!key \|\| key !== process.env.BACKEND_MANAGER_KEY)` |
+| [src/manager/routes/payments/dispute-alert/post.js](src/manager/routes/payments/dispute-alert/post.js) | 20 | `if (!key \|\| key !== process.env.BACKEND_MANAGER_KEY)` |
+| [src/manager/routes/payments/intent/processors/test.js](src/manager/routes/payments/intent/processors/test.js) | 158 | `key=${process.env.BACKEND_MANAGER_KEY}` |
+
+## Phase 1 — Dual-key acceptance (this BEM release)
+
+### Validation change (webhook + dispute-alert)
+
+Replace the single-key check with a dual-key check. Accept the request if `key` matches EITHER `BACKEND_MANAGER_WEBHOOK_KEY` OR the legacy `BACKEND_MANAGER_KEY`.
+
+```js
+// Before
+if (!key || key !== process.env.BACKEND_MANAGER_KEY) {
+  return assistant.respond('Invalid key', { code: 401 });
+}
+
+// After (Phase 1 — dual-key fallback)
+const validKeys = [
+  process.env.BACKEND_MANAGER_WEBHOOK_KEY,
+  process.env.BACKEND_MANAGER_KEY, // legacy — remove in Phase 2
+].filter(Boolean);
+
+if (!key || !validKeys.includes(key)) {
+  return assistant.respond('Invalid key', { code: 401 });
+}
+```
+
+Notes:
+- `.filter(Boolean)` matters — if a brand hasn't set `BACKEND_MANAGER_WEBHOOK_KEY` yet, `validKeys` becomes `[BACKEND_MANAGER_KEY]` (single-element). If a brand HAS set it, both are accepted.
+- If BOTH env vars are unset (misconfigured brand), `validKeys` is `[]` and every request 401s. That's the right behavior — explicit failure beats silent acceptance.
+- Update the route header comments (lines 11 and 27 of dispute-alert, 27 of webhook) to mention both keys.
+
+### Test processor self-fire URL
+
+Switch [src/manager/routes/payments/intent/processors/test.js:158](src/manager/routes/payments/intent/processors/test.js#L158) to PREFER `BACKEND_MANAGER_WEBHOOK_KEY` and fall back to `BACKEND_MANAGER_KEY`:
+
+```js
+const webhookKey = process.env.BACKEND_MANAGER_WEBHOOK_KEY || process.env.BACKEND_MANAGER_KEY;
+const webhookUrl = `${assistant.Manager.project.apiUrl}/backend-manager/payments/webhook?processor=test&key=${webhookKey}`;
+```
+
+This way the test processor exercises the new key when it's set, and falls back to the legacy key on brands that haven't migrated yet.
+
+### Test infrastructure updates
+
+**`src/test/utils/http-client.js`** — add `backendManagerWebhookKey` field alongside the existing `backendManagerKey`.
+
+**`src/test/runner.js`** — thread `backendManagerWebhookKey` through the runner context the same way `backendManagerKey` is threaded today (3 sites).
+
+**`src/cli/commands/test.js`** — read `BACKEND_MANAGER_WEBHOOK_KEY` from env (parallel to the existing `BACKEND_MANAGER_KEY` read), pass it through to the runner, fall back to the admin key if unset.
+
+**Route + journey tests** — update test fixtures to prefer `BACKEND_MANAGER_WEBHOOK_KEY` when set:
+- `test/routes/payments/webhook.js` — all `${config.backendManagerKey}` interpolations in `payments/webhook?...&key=` URLs
+- `test/routes/payments/dispute-alert.js` — same (13 sites)
+- `test/events/payments/journey-*.js` — only the `payments/webhook?...&key=` URLs (NOT the admin-auth uses elsewhere in journey tests)
+
+Use `config.backendManagerWebhookKey || config.backendManagerKey` everywhere so both keys are exercised.
+
+### Docs
+
+- **`docs/stripe-webhook-forwarding.md`** — line 7 + 18 — note the dual-key acceptance, recommend `BACKEND_MANAGER_WEBHOOK_KEY` for new setups, note legacy `BACKEND_MANAGER_KEY` is still accepted for backwards compatibility.
+- **`CHANGELOG.md`** — entry under the next version. Categorize under `Changed` (NOT `BREAKING` — that's reserved for Phase 2). Spell out: "Payment webhook + dispute-alert routes now accept either `BACKEND_MANAGER_WEBHOOK_KEY` (preferred) or `BACKEND_MANAGER_KEY` (legacy). Set `BACKEND_MANAGER_WEBHOOK_KEY` in every consumer brand's `.env` and run OMEGA payment ensure before Phase 2 ships."
+- **`templates/_.env`** — already declares both keys, no change.
+
+### Verification (manual)
+
+1. **Local test (no `BACKEND_MANAGER_WEBHOOK_KEY` set)** — confirm legacy fallback still passes:
+   ```bash
+   npx mgr test routes/payments/webhook routes/payments/dispute-alert events/payments
+   ```
+   Expect green.
+
+2. **Local test (`BACKEND_MANAGER_WEBHOOK_KEY` set to a different value)** — confirm the new key is accepted AND the legacy key still works:
+   ```bash
+   BACKEND_MANAGER_WEBHOOK_KEY=test-webhook-key npx mgr test routes/payments/webhook routes/payments/dispute-alert events/payments
+   ```
+   Expect green.
+
+3. **Live spot-check on one brand (Somiibo)** — deploy this BEM release to Somiibo (whose `.env` currently only has `BACKEND_MANAGER_KEY` set OR has both), trigger a Stripe test event, confirm the webhook doc lands in Firestore.
+
+## Phase 2 — Drop legacy fallback (future BEM release)
+
+**Do NOT do this until every consumer brand has been migrated and verified.** Tracking checklist for migration readiness:
+
+- [ ] Every consumer `.env` has `BACKEND_MANAGER_WEBHOOK_KEY` set (spot-check via OMEGA across all brands)
+- [ ] OMEGA `payment/ensure/{stripe,paypal,chargebee}-webhook.js` updated to use `BACKEND_MANAGER_WEBHOOK_KEY` when constructing webhook URLs (see "OMEGA changes" below)
+- [ ] OMEGA `--service payment` run across every brand to re-register webhook URLs at Stripe/PayPal/Chargebee with the new key
+- [ ] Stale webhook URLs (the old ones with `key=BACKEND_MANAGER_KEY`) deleted from each provider — these will start 401-ing the moment Phase 2 ships
+- [ ] At least one live billing cycle (Stripe charge, refund, dispute) observed working end-to-end on the new key for at least 3 brands
+
+When all boxes are checked, Phase 2 ships as a **breaking change**:
+
+```js
+// Phase 2 — single-key (legacy removed)
+if (!key || key !== process.env.BACKEND_MANAGER_WEBHOOK_KEY) {
+  return assistant.respond('Invalid key', { code: 401 });
+}
+```
+
+Phase 2 CHANGELOG entry goes under `BREAKING`.
+
+## OMEGA changes (separate repo, needed for Phase 2 prep)
+
+Touched files (do NOT do these in Phase 1 — they're for Phase 2 prep):
+
+| File | Change |
+|---|---|
+| `/Users/ian/Developer/Repositories/ITW-Creative-Works/omega-manager/src/services/payment/ensure/stripe-webhook.js` | Line 38: swap `BACKEND_MANAGER_KEY` → `BACKEND_MANAGER_WEBHOOK_KEY`. Line 76 check + line 77 message + line 64 doc comment: update env var name. |
+| `/Users/ian/Developer/Repositories/ITW-Creative-Works/omega-manager/src/services/payment/ensure/paypal-webhook.js` | Same change at lines 40, 66, 77, 78, 79. |
+| `/Users/ian/Developer/Repositories/ITW-Creative-Works/omega-manager/src/services/payment/ensure/chargebee-webhook.js` | Same change at lines 34, 45, 57, 58, 59. |
+
+**Idempotent re-registration + stale cleanup:** Each ensure file looks up existing webhooks **by URL**. Since the URL will change (the `key=` query param differs), the existing webhook will NOT match and a new one will be created. The OLD webhook (with the admin key) needs to be cleaned up — otherwise the third party will deliver to both and the old one will start failing 401 once Phase 2 ships.
+
+Cleanup approach in each ensure file: after creating the new webhook, scan for any other webhook whose URL points at `/payments/webhook?processor={name}` (regardless of key) and delete the stale ones. Match on path prefix, not full URL. Stripe + PayPal + Chargebee all support webhook deletion via the same API methods these files already use for listing.
+
+Helper: extract a `pathMatchesOurProcessor(url, processor)` in each ensure file — strip query string, match the path-and-processor segment.
+
+## Known constraints / gotchas
+
+1. **Phase 1 is non-breaking.** Any consumer can deploy this BEM release without touching their `.env` — the legacy key still works.
+2. **`BACKEND_MANAGER_KEY` continues to grant admin via `assistant.authenticate()`.** That hasn't changed; it's still the admin secret for internal API calls (newsletter, send-email, user/delete, ghostii cron, electron-client, oauth2 state encryption). This TODO is *only* about scoping webhook endpoints to a separate key.
+3. **`UNSUBSCRIBE_HMAC_KEY` is unrelated** and stays separate.
+4. **Admin grant is NOT triggered by the webhook routes.** Middleware auto-runs `assistant.authenticate()` via `Usage.init()`, which reads `data.backendManagerKey` from the request body. The three webhook routes get their key via `?key=` query string (NOT body), and their request bodies are third-party payloads (Stripe events, PayPal events, Chargeblast alerts) that can't contain the real admin secret. So switching the routes' validation does NOT remove any admin-grant the handlers depend on — they write to Firestore via `libraries.admin` (Admin SDK, always full access) anyway.
+5. **Test processor self-fire** must use the same key the BEM endpoint accepts, which is why the test processor URL also gets updated in Phase 1.

package/docs/consent.md

@@ -138,7 +138,7 @@ Each processor's `handleEvent` does the same shape of work:
 | SendGrid | `unsubscribe`, `group_unsubscribe`, `spamreport`, `bounce`, `dropped` |
 | Beehiiv | `subscription.unsubscribed`, `subscription.deleted`, `subscription.paused` |
 
-**Beehiiv publication filter.** Each Beehiiv event includes a `publication_id`. The processor compares this against the result of `beehiivProvider.getPublicationId()` (which reads `Manager.config.marketing.beehiiv.publicationId` or fuzzy-matches against the Beehiiv API by brand name). Mismatch → silent skip. This is how shared-publication events (e.g. devbeans shared by 6 brands) get routed correctly — each brand processes only events matching its own publication.
+**Beehiiv publication filter.** Each Beehiiv event includes a `publication_id`. The processor compares this against `beehiivProvider.getPublicationId()`, which reads `Manager.config.marketing.beehiiv.publicationId` (populated at brand-onboarding time by OMEGA's `beehiiv/ensure/publication.js`). Mismatch → silent skip. This is how shared-publication events (e.g. devbeans shared by 6 brands) get routed correctly — each brand processes only events matching its own publication. Brands without `publicationId` in config silently skip all Beehiiv webhook events. The same convention applies to SendGrid: `marketing.sendgrid.listId` is populated by OMEGA's `sendgrid/ensure/list.js`.
 
 ## Parent forwarder (Phase E)
 
@@ -304,18 +304,13 @@ After the migration: optionally run a re-opt-in drip campaign to legally recover
 
 Run with `npx mgr test` (full suite) or `npx mgr test routes/marketing/webhook` (just the webhook tests).
 
-### Provider-side cleanup (extended mode only)
+### Live-provider tests (extended mode only)
 
-Normally BEM test runs are self-contained because all data lives in the local emulator. Marketing tests are the exception — when `TEST_EXTENDED_MODE=true` is set, the routes make real API calls to SendGrid + Beehiiv and leave real contacts behind.
+Most BEM tests are self-contained against the local emulator. The marketing-consent system has one test that's an exception — [test/marketing/consent-lifecycle.js](../test/marketing/consent-lifecycle.js) — which makes real API calls to SendGrid + Beehiiv to verify the full round-trip works end-to-end.
 
-To keep this self-contained, the runner calls [`cleanupMarketingProviders`](../src/test/test-accounts.js) **twice** per run:
+The validation pipeline (`src/manager/libraries/email/validation.js`) blocks all `_test.*` emails from reaching providers via the `/^_test\.(?!allow_)/` pattern in `blocked-local-patterns.js`. The two `_test.allow_*` sentinels (`_test.allow_consent-granted` and `_test.allow_consent-declined`) used by the lifecycle test bypass that gate intentionally, and the test cleans up after itself (phase-3 removes the granted contact via `Manager.Email().remove()`).
 
-1. **Pre-run** — scrubs leftovers from any previous run that crashed mid-suite.
-2. **Post-run** — scrubs everything this run created. Best-effort (wrapped in try/catch — cleanup failures don't change the test result).
-
-Both fire ONLY when `TEST_EXTENDED_MODE=true`. In normal mode (the default), no provider calls happen and no cleanup is needed.
-
-This is the one exception to the "Firestore cleanup runs at start only" rule documented in [docs/testing.md](testing.md) — we can't wipe SendGrid/Beehiiv state in our own Firestore wipe, so we have to ask the providers to do it for us, twice for defense in depth.
+The "all cleanup runs at start, never at the end" rule documented in [docs/testing.md](testing.md) applies to all test data, including third-party providers.
 
 ## Frontend pieces (cross-references)
 

package/docs/sanitization.md

@@ -1,16 +1,25 @@
 # Sanitization (XSS Prevention)
 
-BEM automatically sanitizes all incoming request data — stripping HTML tags and trimming whitespace from every string field. This happens in the middleware pipeline before route handlers execute, so **routes receive clean data by default**.
+BEM middleware always **trims** whitespace on incoming string fields (via `utilities.trim()`). HTML sanitization is **opt-in** — it's not run by default because it mangles legitimate input like URL query strings (`&` → `&`) and Markdown.
+
+The expectation is that you sanitize at the **HTML-insertion site** (in the template, in the email body, etc.) — not at the request boundary.
 
 ## How It Works
 
-1. **Schema fields**: Sanitized per-field during the middleware pipeline. Fields can opt out with `sanitize: false` in the schema.
-2. **Non-schema fields** (when `setupSettings: false` or `includeNonSchemaSettings: true`): All strings are sanitized with no opt-out.
-3. The middleware uses `Manager.Utilities().sanitize()` under the hood.
+1. **Trimming**: Every string in `settings` is whitespace-trimmed by the middleware (objects + arrays walked recursively). Always on.
+2. **HTML sanitization**: Off by default. Opt in per-route with `{ sanitize: true }` on `Manager.Middleware(...).run(...)`.
+3. **Schemas** can mark individual fields with `sanitize: false` to skip the HTML strip for that field when route-level sanitize is enabled (used for fields that legitimately need raw HTML — rich-text editors, email templates).
+
+## Route-Level Opt-In
+
+```javascript
+// In functions/index.js — enable HTML strip for a specific route
+Manager.Middleware(req, res).run('my-route', { sanitize: true });
+```
 
-## Schema Opt-Out
+When enabled, every string in `settings` is run through `sanitize-html` (strip all tags) unless the schema marks the field with `sanitize: false`.
 
-For fields that legitimately need HTML (rich text, email templates, etc.), set `sanitize: false` in the schema:
+## Schema Field Opt-Out (when route-level sanitize is on)
 
 ```javascript
 // This field will NOT be sanitized — raw HTML is preserved
@@ -19,43 +28,42 @@ htmlContent: {
   default: '',
   sanitize: false,
 },
-// This field IS sanitized (default behavior, no flag needed)
+// This field IS sanitized when route-level sanitize is enabled
 name: {
   types: ['string'],
   default: '',
 },
 ```
 
-## Route-Level Opt-Out
+## Manual Sanitization (Recommended)
 
-Disable sanitization entirely for a route (rare — only for routes that handle raw HTML everywhere):
-
-```javascript
-// In functions/index.js
-Manager.Middleware(req, res).run('my-route', { sanitize: false });
-```
-
-## Manual Sanitization (Outside Middleware)
-
-For cron jobs, event handlers, or anywhere outside the request pipeline, use `utilities.sanitize()` directly:
+For most use cases — particularly anywhere you're inserting user-supplied content into HTML — call `utilities.sanitize()` directly at the insertion site:
 
 ```javascript
 // Available in route context
-const clean = utilities.sanitize(untrustedData);
+const safeHtml = utilities.sanitize(untrustedData);
 
 // Or via Manager
-const clean = Manager.Utilities().sanitize(untrustedData);
+const safeHtml = Manager.Utilities().sanitize(untrustedData);
 ```
 
 Accepts any data type — strings, objects, arrays, primitives. Walks objects/arrays recursively, strips HTML from strings, passes everything else through unchanged.
 
-## Route Handler Context
+## Why HTML Sanitization Is Not the Middleware Default
+
+Stripping HTML from every incoming string at the request boundary is too aggressive — it corrupts legitimate input:
 
-The middleware injects these into every route handler:
+- URL query strings: `https://example.com/?a=1&b=2` becomes `https://example.com/?a=1&b=2`
+- Markdown / code snippets with `<`, `>`, `&` characters get mangled
+- API payloads round-tripped through the system get silently rewritten
+
+Stored XSS comes from rendering, not from receiving. Sanitize at the render site (where you know the output context — HTML body, attribute, URL, JSON) instead of at the front door.
+
+## Route Handler Context
 
 ```javascript
 module.exports = async ({ Manager, assistant, analytics, usage, user, settings, libraries, utilities }) => {
-  // settings    — already sanitized by middleware
-  // utilities   — Manager.Utilities() instance for manual sanitization
+  // settings    — whitespace-trimmed by middleware; HTML preserved unless route opts in via { sanitize: true }
+  // utilities   — Manager.Utilities() instance for manual sanitize()/trim()
 };
 ```

package/docs/schemas.md

@@ -36,4 +36,4 @@ module.exports = function (assistant, settings, options) {
 
 ## Field Sanitization
 
-By default, every string field in a schema is sanitized (HTML tags stripped, whitespace trimmed) during the middleware pipeline. To preserve raw HTML (rich text, email templates), set `sanitize: false` on the field. See [docs/sanitization.md](sanitization.md).
+Middleware always trims whitespace on string fields. HTML sanitization is **opt-in per route** — `Manager.Middleware(req, res).run('my-route', { sanitize: true })`. When opted in, fields can individually opt back out with `sanitize: false`. See [docs/sanitization.md](sanitization.md).

package/docs/stripe-webhook-forwarding.md

@@ -4,7 +4,7 @@ BEM auto-starts Stripe CLI webhook forwarding when running `npx mgr serve` or `n
 
 **Requirements:**
 - `STRIPE_SECRET_KEY` set in `functions/.env`
-- `BACKEND_MANAGER_KEY` set in `functions/.env`
+- `BACKEND_MANAGER_WEBHOOK_KEY` set in `functions/.env` (`BACKEND_MANAGER_KEY` also accepted as a legacy fallback)
 - [Stripe CLI](https://stripe.com/docs/stripe-cli) installed
 
 **Standalone usage:**
@@ -15,4 +15,4 @@ npx mgr stripe
 
 If any prerequisite is missing, webhook forwarding is silently skipped with an info message.
 
-The forwarding URL is: `http://localhost:{hostingPort}/backend-manager/payments/webhook?processor=stripe&key={BACKEND_MANAGER_KEY}`
+The forwarding URL is: `http://localhost:{hostingPort}/backend-manager/payments/webhook?processor=stripe&key={BACKEND_MANAGER_WEBHOOK_KEY}`

package/docs/testing.md

@@ -19,18 +19,19 @@ What the runner wipes pre-test (in [src/test/test-accounts.js](../src/test/test-
 
 1. **`meta/stats`** doc ensured (required for on-create batch writes).
 2. **`users/_test-*`** Firebase Auth users + Firestore docs (delete).
-3. **Marketing providers** (SendGrid + Beehiiv) — leftover test contacts removed. Runs only under `TEST_EXTENDED_MODE`. Runs **before** test users are recreated so a killed run can't leave a contact pinned to an about-to-be-recreated uid.
-4. **Mixed Firestore collections** — `payments-orders`, `payments-webhooks`, `payments-intents`, `payments-disputes`, `marketing-webhooks`. Two-pass cleanup per collection:
+3. **Mixed Firestore collections** — `payments-orders`, `payments-webhooks`, `payments-intents`, `payments-disputes`, `marketing-webhooks`. Two-pass cleanup per collection:
    - Pass 1 — owner-keyed: `where('owner', 'in', [...testUids])` (batched at 30 uids per `in` query).
    - Pass 2 — id-keyed: any doc whose ID starts with `_test-` (catches ownerless test docs like dispute alerts and raw test webhooks).
-5. **Test-only Firestore collections** — `_test`, `_test_query` — wiped in full.
-6. **Realtime Database** — the `_test` namespace removed in full (`admin.database().ref('_test').remove()`).
+4. **Test-only Firestore collections** — `_test`, `_test_query` — wiped in full.
+5. **Realtime Database** — the `_test` namespace removed in full (`admin.database().ref('_test').remove()`).
 
-### The one exception: third-party provider cleanup runs at start AND end
+### Marketing-provider cleanup
 
-The `cleanupMarketingProviders` call also fires **after** the suite completes (extended mode only). Reason: pre-run cleanup catches what a crashed previous run left behind, but during a normal-completion run we'd still leak real SendGrid/Beehiiv contacts until the NEXT run cleaned them up. By running cleanup post-suite too, each extended-mode run leaves the provider lists in the same state it found them.
+Test signups never reach SendGrid + Beehiiv. The validation pipeline (`src/manager/libraries/email/validation.js`) blocks all `_test.*` emails at the marketing-library layer via the `/^_test\.(?!allow_)/` pattern in `blocked-local-patterns.js`.
 
-This is **specifically** for third-party state that lives outside the local emulator. **Do not use trailing cleanup for Firestore or Auth data** — those follow the start-only rule because we control the local state and pre-run cleanup is enough.
+The single exception is the `_test.allow_*` prefix. Two long-lived test accounts (`_test.allow_consent-granted@...` and `_test.allow_consent-declined@...`) intentionally round-trip through SendGrid + Beehiiv as the live-provider integration sentinels. They are exercised by `test/marketing/consent-lifecycle.js`, which manages its own setup, assertions, and teardown.
+
+All cleanup follows the start-only rule. No trailing-cleanup exception.
 
 ### When adding a new test that writes data
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backend-manager",
-  "version": "5.2.3",
+  "version": "5.2.6",
   "description": "Quick tools for developing Firebase functions",
   "main": "src/manager/index.js",
   "bin": {