backend-manager 5.2.3 → 5.2.5

package/CHANGELOG.md

@@ -14,6 +14,24 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+# [5.2.5] - 2026-05-22
+
+### Added
+
+- **`utilities.trim(input)`** in `src/manager/helpers/utilities.js`. Walks objects/arrays recursively and trims whitespace on every string. Does NOT strip HTML. Middleware uses it to clean up incoming request data without mangling URLs, Markdown, or any payload with `<`/`>`/`&`.
+- **`findContact(email)`** on the SendGrid + Beehiiv providers, and **`findSubscriber(email, publicationId)`** on Beehiiv. Both extracted from the existing `removeContact`/`removeSubscriber` search-by-email logic so tests (and other call sites) can verify provider state without forcing a delete.
+- **`test/marketing/consent-lifecycle.js`** — 5-phase live integration test (gated on `TEST_EXTENDED_MODE=true`) that walks two long-lived sentinel accounts through pre-check → sync → declined-stays-out → unsubscribe → validation-gate, asserting actual SendGrid + Beehiiv state at every step. Uses `pollProvider()` to handle SendGrid's async background jobs (upsert/delete can take 10-60s+ to surface).
+
+### Changed
+
+- **Middleware no longer strips HTML by default.** The auto-`sanitize-html` pass on incoming request data was mangling legitimate input — URL query strings (`?a=1&b=2` → `?a=1&amp;b=2`), Markdown, and any payload containing `<`/`>`/`&`. The middleware now only trims whitespace by default. HTML sanitization is opt-in per route via `Manager.Middleware(req, res).run('route', { sanitize: true })`. Sanitize at the HTML-insertion site (`utilities.sanitize()` in your template/email render) rather than at the request boundary. The existing schema-level `sanitize: false` opt-out still works when route-level sanitize is on.
+- **Test sentinel accounts `consent-granted` and `consent-declined` now use the `_test.allow_*` prefix** (`_test.allow_consent-granted@...`, `_test.allow_consent-declined@...`). The `_test.*` validation block from v5.2.3 also blocks the previous `_test.consent-*` names from reaching providers — `_test.allow_*` is the carved-out exception specifically for live-provider integration tests.
+- **Beehiiv API timeouts bumped to 60s** on the two remaining stragglers (`findSubscriber`, `resolveSegmentIds`) to match the SendGrid+Beehiiv 60s convention from v5.2.3.
+
+### Removed
+
+- **`cleanupMarketingProviders()` function and its pre-run/post-run hooks** in `src/test/test-accounts.js` + `src/test/runner.js`. No longer needed — the validation gate added in v5.2.3 blocks `_test.*` emails from reaching SendGrid + Beehiiv upstream, so there's nothing to clean up. The new `consent-lifecycle.js` test (which intentionally round-trips real contacts via `_test.allow_*`) manages its own pre-check force-clean inline.
+
 # [5.2.3] - 2026-05-22
 
 ### Added

package/CLAUDE.md

@@ -76,7 +76,7 @@ See [docs/cli-firestore-auth.md](docs/cli-firestore-auth.md) and [docs/cli-logs.
 - **Firestore shorthand**: `admin.firestore().doc('users/abc123')` (path string) rather than `.collection('users').doc('abc123')`.
 - **Template strings for requires**: `` require(`${functionsDir}/node_modules/backend-manager`) `` rather than string concat.
 - **No backwards compatibility** unless explicitly requested.
-- **Routes receive sanitized data by default** — see [docs/sanitization.md](docs/sanitization.md) for opt-out rules.
+- **Routes receive whitespace-trimmed data; HTML is preserved.** Sanitize at the HTML-insertion site via `utilities.sanitize()`. Opt into middleware-level HTML strip per-route with `{ sanitize: true }`. See [docs/sanitization.md](docs/sanitization.md).
 - **Match schema names to route names** — if route is `myEndpoint`, schema is `myEndpoint`.
 - **Always use `assistant.respond()` for responses** — do NOT use `res.send()` directly.
 - **Add Firestore composite indexes** for any compound query (`where` + `orderBy`, or multiple `where`s) to `src/cli/commands/setup-tests/helpers/required-indexes.js` (the SSOT). Without the index, queries crash with `FAILED_PRECONDITION` in production.
@@ -113,7 +113,7 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 
 - [docs/routes.md](docs/routes.md) — recipes for new API commands, routes, event handlers, cron jobs (with code templates)
 - [docs/schemas.md](docs/schemas.md) — schema definition format, defaults vs premium overrides
-- [docs/sanitization.md](docs/sanitization.md) — automatic XSS sanitization, schema opt-out (`sanitize: false`), route-level opt-out, manual `utilities.sanitize()`
+- [docs/sanitization.md](docs/sanitization.md) — middleware trim-only default; opt-in HTML strip (`{ sanitize: true }`) with per-field opt-out (`sanitize: false`); manual `utilities.sanitize()` for HTML-insertion sites
 - [docs/auth-hooks.md](docs/auth-hooks.md) — consumer hooks for `before-create`/`before-signin`/`on-create`/`on-delete` (blocking + non-blocking examples)
 - [docs/common-operations.md](docs/common-operations.md) — inside-the-handler patterns: authenticate, read/write Firestore, error handling, send response, `bm_api` hook
 
@@ -134,6 +134,6 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 
 ### Testing & CLI
 
-- [docs/testing.md](docs/testing.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels. **For Firestore/Auth/local state, cleanup runs at the START of every run, never at the end** — if you add a test that writes Firestore data, register the collection/namespace in the runner's pre-test wipe list, don't add a trailing cleanup step. **Exception:** third-party providers we can't wipe at start (e.g. SendGrid/Beehiiv contact lists) get a symmetric pre + post cleanup hook in the runner — see `cleanupMarketingProviders` and [docs/consent.md](docs/consent.md). Don't pattern-match this exception for new Firestore code.
+- [docs/testing.md](docs/testing.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels. **All cleanup runs at the START of every run, never at the end.** If you add a test that writes Firestore data, register the collection/namespace in the runner's pre-test wipe list, don't add a trailing cleanup step. Marketing providers (SendGrid/Beehiiv) don't need a special exception — `_test.*` emails are blocked at the validation layer so test signups never reach providers. The `_test.allow_*` carve-out exists only for the live-provider lifecycle test (`test/marketing/consent-lifecycle.js`), which manages its own teardown.
 - [docs/cli-firestore-auth.md](docs/cli-firestore-auth.md) — `npx mgr firestore:*` and `auth:*` commands, shared flags, examples
 - [docs/cli-logs.md](docs/cli-logs.md) — `npx mgr logs:read` / `logs:tail` with full flag reference and built-in Cloud Function names

package/docs/consent.md

@@ -138,7 +138,7 @@ Each processor's `handleEvent` does the same shape of work:
 | SendGrid | `unsubscribe`, `group_unsubscribe`, `spamreport`, `bounce`, `dropped` |
 | Beehiiv | `subscription.unsubscribed`, `subscription.deleted`, `subscription.paused` |
 
-**Beehiiv publication filter.** Each Beehiiv event includes a `publication_id`. The processor compares this against the result of `beehiivProvider.getPublicationId()` (which reads `Manager.config.marketing.beehiiv.publicationId` or fuzzy-matches against the Beehiiv API by brand name). Mismatch → silent skip. This is how shared-publication events (e.g. devbeans shared by 6 brands) get routed correctly — each brand processes only events matching its own publication.
+**Beehiiv publication filter.** Each Beehiiv event includes a `publication_id`. The processor compares this against `beehiivProvider.getPublicationId()`, which reads `Manager.config.marketing.beehiiv.publicationId` (populated at brand-onboarding time by OMEGA's `beehiiv/ensure/publication.js`). Mismatch → silent skip. This is how shared-publication events (e.g. devbeans shared by 6 brands) get routed correctly — each brand processes only events matching its own publication. Brands without `publicationId` in config silently skip all Beehiiv webhook events. The same convention applies to SendGrid: `marketing.sendgrid.listId` is populated by OMEGA's `sendgrid/ensure/list.js`.
 
 ## Parent forwarder (Phase E)
 
@@ -304,18 +304,13 @@ After the migration: optionally run a re-opt-in drip campaign to legally recover
 
 Run with `npx mgr test` (full suite) or `npx mgr test routes/marketing/webhook` (just the webhook tests).
 
-### Provider-side cleanup (extended mode only)
+### Live-provider tests (extended mode only)
 
-Normally BEM test runs are self-contained because all data lives in the local emulator. Marketing tests are the exception — when `TEST_EXTENDED_MODE=true` is set, the routes make real API calls to SendGrid + Beehiiv and leave real contacts behind.
+Most BEM tests are self-contained against the local emulator. The marketing-consent system has one test that's an exception — [test/marketing/consent-lifecycle.js](../test/marketing/consent-lifecycle.js) — which makes real API calls to SendGrid + Beehiiv to verify the full round-trip works end-to-end.
 
-To keep this self-contained, the runner calls [`cleanupMarketingProviders`](../src/test/test-accounts.js) **twice** per run:
+The validation pipeline (`src/manager/libraries/email/validation.js`) blocks all `_test.*` emails from reaching providers via the `/^_test\.(?!allow_)/` pattern in `blocked-local-patterns.js`. The two `_test.allow_*` sentinels (`_test.allow_consent-granted` and `_test.allow_consent-declined`) used by the lifecycle test bypass that gate intentionally, and the test cleans up after itself (phase-3 removes the granted contact via `Manager.Email().remove()`).
 
-1. **Pre-run** — scrubs leftovers from any previous run that crashed mid-suite.
-2. **Post-run** — scrubs everything this run created. Best-effort (wrapped in try/catch — cleanup failures don't change the test result).
-
-Both fire ONLY when `TEST_EXTENDED_MODE=true`. In normal mode (the default), no provider calls happen and no cleanup is needed.
-
-This is the one exception to the "Firestore cleanup runs at start only" rule documented in [docs/testing.md](testing.md) — we can't wipe SendGrid/Beehiiv state in our own Firestore wipe, so we have to ask the providers to do it for us, twice for defense in depth.
+The "all cleanup runs at start, never at the end" rule documented in [docs/testing.md](testing.md) applies to all test data, including third-party providers.
 
 ## Frontend pieces (cross-references)
 

package/docs/sanitization.md

@@ -1,16 +1,25 @@
 # Sanitization (XSS Prevention)
 
-BEM automatically sanitizes all incoming request data — stripping HTML tags and trimming whitespace from every string field. This happens in the middleware pipeline before route handlers execute, so **routes receive clean data by default**.
+BEM middleware always **trims** whitespace on incoming string fields (via `utilities.trim()`). HTML sanitization is **opt-in** — it's not run by default because it mangles legitimate input like URL query strings (`&` → `&`) and Markdown.
+
+The expectation is that you sanitize at the **HTML-insertion site** (in the template, in the email body, etc.) — not at the request boundary.
 
 ## How It Works
 
-1. **Schema fields**: Sanitized per-field during the middleware pipeline. Fields can opt out with `sanitize: false` in the schema.
-2. **Non-schema fields** (when `setupSettings: false` or `includeNonSchemaSettings: true`): All strings are sanitized with no opt-out.
-3. The middleware uses `Manager.Utilities().sanitize()` under the hood.
+1. **Trimming**: Every string in `settings` is whitespace-trimmed by the middleware (objects + arrays walked recursively). Always on.
+2. **HTML sanitization**: Off by default. Opt in per-route with `{ sanitize: true }` on `Manager.Middleware(...).run(...)`.
+3. **Schemas** can mark individual fields with `sanitize: false` to skip the HTML strip for that field when route-level sanitize is enabled (used for fields that legitimately need raw HTML — rich-text editors, email templates).
+
+## Route-Level Opt-In
+
+```javascript
+// In functions/index.js — enable HTML strip for a specific route
+Manager.Middleware(req, res).run('my-route', { sanitize: true });
+```
 
-## Schema Opt-Out
+When enabled, every string in `settings` is run through `sanitize-html` (strip all tags) unless the schema marks the field with `sanitize: false`.
 
-For fields that legitimately need HTML (rich text, email templates, etc.), set `sanitize: false` in the schema:
+## Schema Field Opt-Out (when route-level sanitize is on)
 
 ```javascript
 // This field will NOT be sanitized — raw HTML is preserved
@@ -19,43 +28,42 @@ htmlContent: {
   default: '',
   sanitize: false,
 },
-// This field IS sanitized (default behavior, no flag needed)
+// This field IS sanitized when route-level sanitize is enabled
 name: {
   types: ['string'],
   default: '',
 },
 ```
 
-## Route-Level Opt-Out
+## Manual Sanitization (Recommended)
 
-Disable sanitization entirely for a route (rare — only for routes that handle raw HTML everywhere):
-
-```javascript
-// In functions/index.js
-Manager.Middleware(req, res).run('my-route', { sanitize: false });
-```
-
-## Manual Sanitization (Outside Middleware)
-
-For cron jobs, event handlers, or anywhere outside the request pipeline, use `utilities.sanitize()` directly:
+For most use cases — particularly anywhere you're inserting user-supplied content into HTML — call `utilities.sanitize()` directly at the insertion site:
 
 ```javascript
 // Available in route context
-const clean = utilities.sanitize(untrustedData);
+const safeHtml = utilities.sanitize(untrustedData);
 
 // Or via Manager
-const clean = Manager.Utilities().sanitize(untrustedData);
+const safeHtml = Manager.Utilities().sanitize(untrustedData);
 ```
 
 Accepts any data type — strings, objects, arrays, primitives. Walks objects/arrays recursively, strips HTML from strings, passes everything else through unchanged.
 
-## Route Handler Context
+## Why HTML Sanitization Is Not the Middleware Default
+
+Stripping HTML from every incoming string at the request boundary is too aggressive — it corrupts legitimate input:
 
-The middleware injects these into every route handler:
+- URL query strings: `https://example.com/?a=1&b=2` becomes `https://example.com/?a=1&b=2`
+- Markdown / code snippets with `<`, `>`, `&` characters get mangled
+- API payloads round-tripped through the system get silently rewritten
+
+Stored XSS comes from rendering, not from receiving. Sanitize at the render site (where you know the output context — HTML body, attribute, URL, JSON) instead of at the front door.
+
+## Route Handler Context
 
 ```javascript
 module.exports = async ({ Manager, assistant, analytics, usage, user, settings, libraries, utilities }) => {
-  // settings    — already sanitized by middleware
-  // utilities   — Manager.Utilities() instance for manual sanitization
+  // settings    — whitespace-trimmed by middleware; HTML preserved unless route opts in via { sanitize: true }
+  // utilities   — Manager.Utilities() instance for manual sanitize()/trim()
 };
 ```

package/docs/schemas.md

@@ -36,4 +36,4 @@ module.exports = function (assistant, settings, options) {
 
 ## Field Sanitization
 
-By default, every string field in a schema is sanitized (HTML tags stripped, whitespace trimmed) during the middleware pipeline. To preserve raw HTML (rich text, email templates), set `sanitize: false` on the field. See [docs/sanitization.md](sanitization.md).
+Middleware always trims whitespace on string fields. HTML sanitization is **opt-in per route** — `Manager.Middleware(req, res).run('my-route', { sanitize: true })`. When opted in, fields can individually opt back out with `sanitize: false`. See [docs/sanitization.md](sanitization.md).

package/docs/testing.md

@@ -19,18 +19,19 @@ What the runner wipes pre-test (in [src/test/test-accounts.js](../src/test/test-
 
 1. **`meta/stats`** doc ensured (required for on-create batch writes).
 2. **`users/_test-*`** Firebase Auth users + Firestore docs (delete).
-3. **Marketing providers** (SendGrid + Beehiiv) — leftover test contacts removed. Runs only under `TEST_EXTENDED_MODE`. Runs **before** test users are recreated so a killed run can't leave a contact pinned to an about-to-be-recreated uid.
-4. **Mixed Firestore collections** — `payments-orders`, `payments-webhooks`, `payments-intents`, `payments-disputes`, `marketing-webhooks`. Two-pass cleanup per collection:
+3. **Mixed Firestore collections** — `payments-orders`, `payments-webhooks`, `payments-intents`, `payments-disputes`, `marketing-webhooks`. Two-pass cleanup per collection:
    - Pass 1 — owner-keyed: `where('owner', 'in', [...testUids])` (batched at 30 uids per `in` query).
    - Pass 2 — id-keyed: any doc whose ID starts with `_test-` (catches ownerless test docs like dispute alerts and raw test webhooks).
-5. **Test-only Firestore collections** — `_test`, `_test_query` — wiped in full.
-6. **Realtime Database** — the `_test` namespace removed in full (`admin.database().ref('_test').remove()`).
+4. **Test-only Firestore collections** — `_test`, `_test_query` — wiped in full.
+5. **Realtime Database** — the `_test` namespace removed in full (`admin.database().ref('_test').remove()`).
 
-### The one exception: third-party provider cleanup runs at start AND end
+### Marketing-provider cleanup
 
-The `cleanupMarketingProviders` call also fires **after** the suite completes (extended mode only). Reason: pre-run cleanup catches what a crashed previous run left behind, but during a normal-completion run we'd still leak real SendGrid/Beehiiv contacts until the NEXT run cleaned them up. By running cleanup post-suite too, each extended-mode run leaves the provider lists in the same state it found them.
+Test signups never reach SendGrid + Beehiiv. The validation pipeline (`src/manager/libraries/email/validation.js`) blocks all `_test.*` emails at the marketing-library layer via the `/^_test\.(?!allow_)/` pattern in `blocked-local-patterns.js`.
 
-This is **specifically** for third-party state that lives outside the local emulator. **Do not use trailing cleanup for Firestore or Auth data** — those follow the start-only rule because we control the local state and pre-run cleanup is enough.
+The single exception is the `_test.allow_*` prefix. Two long-lived test accounts (`_test.allow_consent-granted@...` and `_test.allow_consent-declined@...`) intentionally round-trip through SendGrid + Beehiiv as the live-provider integration sentinels. They are exercised by `test/marketing/consent-lifecycle.js`, which manages its own setup, assertions, and teardown.
+
+All cleanup follows the start-only rule. No trailing-cleanup exception.
 
 ### When adding a new test that writes data
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backend-manager",
-  "version": "5.2.3",
+  "version": "5.2.5",
   "description": "Quick tools for developing Firebase functions",
   "main": "src/manager/index.js",
   "bin": {

package/src/manager/helpers/middleware.js

@@ -35,7 +35,7 @@ Middleware.prototype.run = function (libPath, options) {
     options.setupAnalytics = typeof options.setupAnalytics === 'boolean' ? options.setupAnalytics : true;
     options.setupUsage = typeof options.setupUsage === 'boolean' ? options.setupUsage : true;
     options.setupSettings = typeof options.setupSettings === 'undefined' ? true : options.setupSettings;
-    options.sanitize = typeof options.sanitize === 'undefined' ? true : options.sanitize;
+    options.sanitize = typeof options.sanitize === 'undefined' ? false : options.sanitize;
     options.includeNonSchemaSettings = typeof options.includeNonSchemaSettings === 'undefined' ? false : options.includeNonSchemaSettings;
     options.schema = typeof options.schema === 'undefined' ? libPath : options.schema;
     options.parseMultipartFormData = typeof options.parseMultipartFormData === 'undefined' ? true : options.parseMultipartFormData;
@@ -177,13 +177,16 @@ Middleware.prototype.run = function (libPath, options) {
       assistant.settings = data;
     }
 
-    // Sanitize settings — trim whitespace and strip HTML from all strings
-    // Respects sanitize: false on individual schema fields
+    // Trim whitespace on all string settings (always on — harmless and useful).
+    assistant.settings = Manager.Utilities().trim(assistant.settings);
+
+    // Optional HTML strip (off by default — opt in with `{ sanitize: true }`).
+    // Sanitize at the HTML-insertion site instead unless you need a belt-and-suspenders pass here.
+    // Respects sanitize: false on individual schema fields.
     if (options.sanitize) {
       const schema = options.setupSettings ? Manager.Settings().schema : null;
       const utilities = Manager.Utilities();
 
-      // Walk settings, skipping fields the schema marks as sanitize: false
       assistant.settings = sanitizeWithSchema(utilities, assistant.settings, schema);
     }
 

package/src/manager/helpers/utilities.js

@@ -528,4 +528,35 @@ Utilities.prototype.sanitize = function (input) {
   return input;
 };
 
+/**
+ * Trim whitespace from all strings in input. Walks objects/arrays recursively.
+ * Does NOT strip HTML — use sanitize() for that.
+ *
+ * @param {*} input - The data to trim (string, object, array, or primitive)
+ * @returns {*} Trimmed copy (objects/arrays) or trimmed value (strings)
+ */
+Utilities.prototype.trim = function (input) {
+  if (input == null) {
+    return input;
+  }
+
+  if (typeof input === 'string') {
+    return input.trim();
+  }
+
+  if (Array.isArray(input)) {
+    return input.map(item => this.trim(item));
+  }
+
+  if (typeof input === 'object') {
+    const result = {};
+    for (const [key, value] of Object.entries(input)) {
+      result[key] = this.trim(value);
+    }
+    return result;
+  }
+
+  return input;
+};
+
 module.exports = Utilities;

package/src/manager/libraries/email/providers/beehiiv.js

@@ -79,39 +79,55 @@ async function addSubscriber({ email, firstName, lastName, source, publicationId
 }
 
 /**
- * Remove a subscriber from a Beehiiv publication by email.
+ * Look up a Beehiiv subscriber by email. Returns the subscription object
+ * (id, email, status, custom_fields, ...) or null if not found.
+ *
+ * Useful for tests that need to verify whether a subscriber landed in the
+ * publication after a marketing sync.
  *
  * @param {string} email
  * @param {string} publicationId
- * @returns {{ success: boolean, deleted?: boolean, skipped?: boolean, error?: string }}
+ * @returns {Promise<object|null>}
  */
-async function removeSubscriber(email, publicationId) {
+async function findSubscriber(email, publicationId) {
   try {
     const encodedEmail = encodeURIComponent(email);
-
-    // Step 1: Get subscription by email
-    let searchData;
-    try {
-      searchData = await fetch(
-        `${BASE_URL}/publications/${publicationId}/subscriptions/by_email/${encodedEmail}`,
-        {
-          response: 'json',
-          headers: headers(),
-          timeout: 10000,
-        }
-      );
-    } catch (e) {
-      if (e.status === 404) {
-        return { success: true, skipped: true, reason: 'Subscriber not found' };
+    const searchData = await fetch(
+      `${BASE_URL}/publications/${publicationId}/subscriptions/by_email/${encodedEmail}`,
+      {
+        response: 'json',
+        headers: headers(),
+        timeout: 60000,
       }
-      throw e;
+    );
+
+    return searchData.data || null;
+  } catch (e) {
+    if (e.status === 404) {
+      return null;
     }
+    console.error('Beehiiv findSubscriber error:', e);
+    return null;
+  }
+}
 
-    if (!searchData.data?.id) {
-      return { success: true, skipped: true, reason: 'Subscription not found' };
+/**
+ * Remove a subscriber from a Beehiiv publication by email.
+ *
+ * @param {string} email
+ * @param {string} publicationId
+ * @returns {{ success: boolean, deleted?: boolean, skipped?: boolean, error?: string }}
+ */
+async function removeSubscriber(email, publicationId) {
+  try {
+    // Step 1: Look up the subscription
+    const subscription = await findSubscriber(email, publicationId);
+
+    if (!subscription?.id) {
+      return { success: true, skipped: true, reason: 'Subscriber not found' };
     }
 
-    const subscriptionId = searchData.data.id;
+    const subscriptionId = subscription.id;
 
     // Step 2: Permanently delete the subscription
     await fetch(
@@ -119,7 +135,7 @@ async function removeSubscriber(email, publicationId) {
       {
         method: 'delete',
         headers: headers(),
-        timeout: 10000,
+        timeout: 60000,
       }
     );
 
@@ -186,7 +202,7 @@ function getPublicationId() {
 //       const data = await fetch(`${BASE_URL}/publications?limit=${limit}&page=${page}`, {
 //         response: 'json',
 //         headers: headers(),
-//         timeout: 10000,
+//         timeout: 60000,
 //       });
 //
 //       if (!data.data || data.data.length === 0) {
@@ -270,6 +286,25 @@ async function removeContact(email) {
   return removeSubscriber(email, publicationId);
 }
 
+/**
+ * Look up a contact in this brand's Beehiiv publication. Resolves the
+ * publicationId from config and calls findSubscriber. Mirrors SendGrid's
+ * findContact() surface so tests can use the same pattern across both
+ * providers.
+ *
+ * @param {string} email
+ * @returns {Promise<object|null>}
+ */
+async function findContact(email) {
+  const publicationId = getPublicationId();
+
+  if (!publicationId) {
+    return null;
+  }
+
+  return findSubscriber(email, publicationId);
+}
+
 /**
  * Build Beehiiv custom_fields array from a user doc.
  * Resolves all field values, then maps to display names for Beehiiv.
@@ -316,7 +351,7 @@ async function resolveSegmentIds() {
     const data = await fetch(`${BASE_URL}/publications/${publicationId}/segments?limit=100`, {
       response: 'json',
       headers: headers(),
-      timeout: 10000,
+      timeout: 60000,
     });
 
     _segmentIdCache = {};
@@ -435,6 +470,8 @@ module.exports = {
 
   // Contacts
   addContact,
+  findContact,
+  findSubscriber,
   removeContact,
   buildFields,
 

package/src/manager/libraries/email/providers/sendgrid.js

@@ -131,14 +131,17 @@ async function upsertContacts({ contacts, listIds }) {
 }
 
 /**
- * Remove a contact from SendGrid by email address.
+ * Look up a SendGrid contact by email. Returns the contact object (id, email,
+ * list_ids, custom_fields, ...) or null if not found.
+ *
+ * Useful for tests that need to verify whether a contact landed in the list
+ * after a marketing sync.
  *
  * @param {string} email
- * @returns {{ success: boolean, jobId?: string, skipped?: boolean, error?: string }}
+ * @returns {Promise<object|null>}
  */
-async function removeContact(email) {
+async function findContact(email) {
   try {
-    // Step 1: Get contact ID by email
     const searchData = await fetch(`${BASE_URL}/marketing/contacts/search/emails`, {
       method: 'post',
       response: 'json',
@@ -147,11 +150,33 @@ async function removeContact(email) {
       body: { emails: [email] },
     });
 
-    if (!searchData.result?.[email]?.contact?.id) {
+    return searchData.result?.[email]?.contact || null;
+  } catch (e) {
+    // 404 is the normal "not in contacts" response — return null silently.
+    if (e.status === 404) {
+      return null;
+    }
+    console.error('SendGrid findContact error:', e);
+    return null;
+  }
+}
+
+/**
+ * Remove a contact from SendGrid by email address.
+ *
+ * @param {string} email
+ * @returns {{ success: boolean, jobId?: string, skipped?: boolean, error?: string }}
+ */
+async function removeContact(email) {
+  try {
+    // Step 1: Get contact ID by email
+    const contact = await findContact(email);
+
+    if (!contact?.id) {
       return { success: true, skipped: true, reason: 'Contact not found' };
     }
 
-    const contactId = searchData.result[email].contact.id;
+    const contactId = contact.id;
 
     // Step 2: Delete contact by ID
     const deleteData = await fetch(`${BASE_URL}/marketing/contacts?ids=${contactId}`, {
@@ -635,6 +660,7 @@ module.exports = {
 
   // Contacts
   addContact,
+  findContact,
   removeContact,
   getSegmentContacts,
   bulkDeleteContacts,

package/src/test/runner.js

@@ -119,25 +119,6 @@ class TestRunner {
       await this.runTestsInDir(projectTestsDir, 'project');
     }
 
-    // Post-run cleanup: scrub test accounts from third-party marketing providers
-    // (SendGrid/Beehiiv) so each test run leaves the contact list in the same
-    // state it found it. Pairs with the pre-run cleanup as defense in depth —
-    // pre-run handles crashed previous runs, post-run handles the current run.
-    // Only fires in extended mode (normal mode never touches real providers).
-    if (process.env.TEST_EXTENDED_MODE) {
-      process.stdout.write(chalk.gray('\n  Cleaning up test accounts from marketing providers... '));
-      try {
-        const cleanupResult = await testAccounts.cleanupMarketingProviders(this.options.domain, {
-          apiUrl: this.options.apiUrl,
-          backendManagerKey: this.options.backendManagerKey,
-        });
-        console.log(chalk.green(`✓ (${cleanupResult.cleaned} cleaned)`));
-      } catch (e) {
-        // Post-run cleanup is best-effort — failures shouldn't change the test result
-        console.log(chalk.yellow(`⚠ cleanup error: ${e.message}`));
-      }
-    }
-
     // Cleanup rules context
     if (this.rulesContext) {
       await this.rulesContext.cleanup();
@@ -229,18 +210,6 @@ class TestRunner {
     const deleteResult = await testAccounts.deleteTestUsers(this.options.admin);
     console.log(chalk.green(`✓ (${deleteResult.deleted} deleted, ${deleteResult.skipped} skipped)`));
 
-    // Clean any leftover test accounts from third-party marketing providers
-    // (SendGrid/Beehiiv). Runs BEFORE we create fresh users so a previously
-    // killed run doesn't leave the contact list polluted.
-    if (process.env.TEST_EXTENDED_MODE) {
-      process.stdout.write(chalk.gray('  Cleaning test accounts from marketing providers... '));
-      const cleanupResult = await testAccounts.cleanupMarketingProviders(this.options.domain, {
-        apiUrl: this.options.apiUrl,
-        backendManagerKey: this.options.backendManagerKey,
-      });
-      console.log(chalk.green(`✓ (${cleanupResult.cleaned} cleaned)`));
-    }
-
     process.stdout.write(chalk.gray('  Creating test accounts... '));
 
     // Create fresh test accounts

package/src/test/test-accounts.js

@@ -162,10 +162,14 @@ const STATIC_ACCOUNTS = {
       subscription: { product: { id: 'basic' }, status: 'active' },
     },
   },
+  // The two `consent-*` accounts use the `_test.allow_*` prefix so they bypass
+  // the `_test.*` marketing-block in blocked-local-patterns.js. They're the
+  // live-provider integration sentinels — they intentionally round-trip through
+  // SendGrid + Beehiiv to verify the consent gate works end-to-end.
   'consent-granted': {
     id: 'consent-granted',
-    uid: '_test-consent-granted',
-    email: '_test.consent-granted@{domain}',
+    uid: '_test-allow-consent-granted',
+    email: '_test.allow_consent-granted@{domain}',
     properties: {
       roles: {},
       subscription: { product: { id: 'basic' }, status: 'active' },
@@ -173,8 +177,8 @@ const STATIC_ACCOUNTS = {
   },
   'consent-declined': {
     id: 'consent-declined',
-    uid: '_test-consent-declined',
-    email: '_test.consent-declined@{domain}',
+    uid: '_test-allow-consent-declined',
+    email: '_test.allow_consent-declined@{domain}',
     properties: {
       roles: {},
       subscription: { product: { id: 'basic' }, status: 'active' },
@@ -805,64 +809,6 @@ const TEST_DATA = {
   defaultProjectId: 'demo-test',
 };
 
-/**
- * Clean up test accounts from marketing providers (SendGrid + Beehiiv)
- * Called after account setup when TEST_EXTENDED_MODE is set to remove
- * contacts added by auth:on-create
- * @param {string} domain - Domain for email addresses
- * @param {object} options - Options with apiUrl and backendManagerKey
- * @returns {Promise<object>} Result with cleaned count
- */
-async function cleanupMarketingProviders(domain, options = {}) {
-  const fetch = require('wonderful-fetch');
-  const results = { cleaned: 0, errors: [] };
-
-  const { apiUrl, backendManagerKey } = options;
-  if (!apiUrl || !backendManagerKey) {
-    console.error('cleanupMarketingProviders: Missing apiUrl or backendManagerKey');
-    return results;
-  }
-
-  // Get all test account emails (test contacts like rachel.greene+bem cleaned up by their own tests)
-  const definitions = getAccountDefinitions(domain);
-  const emails = Object.values(definitions).map(acc => acc.email);
-
-  // Clean up each email via the API endpoint (uses hosting port 5002)
-  await Promise.all(
-    emails.map(async (email) => {
-      try {
-        const response = await fetch(`${apiUrl}/backend-manager/marketing/contact`, {
-          method: 'DELETE',
-          response: 'json',
-          timeout: 30000,
-          body: {
-            backendManagerKey,
-            email,
-          },
-        });
-
-        // Log the result for debugging
-        if (response.providers?.beehiiv?.deleted) {
-          results.cleaned++;
-        } else if (response.providers?.beehiiv?.skipped) {
-          // Skipped means not found - that's fine
-          results.cleaned++;
-        } else if (response.providers?.beehiiv?.error) {
-          console.error(`Failed to delete ${email} from Beehiiv:`, response.providers.beehiiv.error);
-          results.errors.push({ email, error: response.providers.beehiiv.error });
-        } else {
-          results.cleaned++;
-        }
-      } catch (error) {
-        console.error(`Failed to cleanup ${email}:`, error.message);
-        results.errors.push({ email, error: error.message });
-      }
-    })
-  );
-
-  return results;
-}
-
 module.exports = {
   STATIC_ACCOUNTS,
   JOURNEY_ACCOUNTS,
@@ -873,5 +819,4 @@ module.exports = {
   fetchPrivateKeys,
   deleteTestUsers,
   createTestAccounts,
-  cleanupMarketingProviders,
 };

package/test/marketing/consent-lifecycle.js

@@ -0,0 +1,255 @@
+/**
+ * Test: Marketing Provider Lifecycle (end-to-end against live SendGrid + Beehiiv)
+ *
+ * Walks two long-lived test accounts through their full lifecycle and verifies
+ * provider state at every transition:
+ *
+ *   1. Pre-check    — both accounts should be absent from SendGrid + Beehiiv
+ *   2. Sync         — flip consent.marketing.status and call Marketing.sync()
+ *   3. Verify       — granted account present in both, declined account absent from both
+ *   4. Unsubscribe  — hit the email-preferences endpoint for the granted account
+ *   5. Verify       — granted account now absent from both
+ *
+ * The two accounts use the `_test.allow_*` prefix so they bypass the
+ * blocked-local-patterns gate (which blocks plain `_test.*` from reaching
+ * providers). They are the only accounts intentionally allowed to round-trip
+ * through SendGrid + Beehiiv.
+ *
+ * Run with TEST_EXTENDED_MODE=true (no-op otherwise). Requires SENDGRID_API_KEY
+ * and BEEHIIV_API_KEY in env. Total runtime is ~60-90s — most of it spent waiting
+ * for SendGrid's async upsert/delete background jobs to surface.
+ */
+const sendgridProvider = require('../../src/manager/libraries/email/providers/sendgrid.js');
+const beehiivProvider = require('../../src/manager/libraries/email/providers/beehiiv.js');
+
+const SETTLE_MS = 5000; // Beehiiv settles in 1-2s; SendGrid's background-job upsert can take 10-20s+
+const POLL_INTERVAL_MS = 2000;
+const POLL_MAX_MS = 90000; // SendGrid's delete-contact job can take 30-60s+ to surface
+
+function sleep(ms) {
+  return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
+/**
+ * Poll a provider lookup until it matches the expected state (present/absent).
+ * Returns the resolved value (contact object or null) once condition is met, or
+ * the last value seen after timing out.
+ *
+ * @param {Function} fetchFn - async function that returns contact or null
+ * @param {boolean} expectPresent - true = wait until present, false = wait until absent
+ */
+async function pollProvider(fetchFn, expectPresent) {
+  const start = Date.now();
+  let lastValue = null;
+
+  while (Date.now() - start < POLL_MAX_MS) {
+    lastValue = await fetchFn();
+    const present = !!lastValue;
+    if (present === expectPresent) {
+      return lastValue;
+    }
+    await sleep(POLL_INTERVAL_MS);
+  }
+
+  return lastValue;
+}
+
+module.exports = {
+  description: 'Marketing provider lifecycle (live SendGrid + Beehiiv round-trip)',
+  type: 'group',
+  skip: !process.env.TEST_EXTENDED_MODE
+    ? 'TEST_EXTENDED_MODE not set (this test hits live SendGrid + Beehiiv APIs)'
+    : false,
+  tests: [
+    // ─────────────────────────────────────────────────────────────────────
+    // Phase 1 — Pre-check: both accounts should be absent from providers
+    // ─────────────────────────────────────────────────────────────────────
+    {
+      name: 'phase-1-pre-check-both-accounts-absent',
+      auth: 'admin',
+      timeout: 180000,
+
+      async run({ accounts, assert }) {
+        const granted = accounts['consent-granted'];
+        const declined = accounts['consent-declined'];
+
+        // Force-clean any leftovers from a prior run (e.g. the previous phase-2a
+        // left a contact in SendGrid that THIS run's phase-1 needs to start
+        // without). SendGrid's delete is an async job, so wait for absence.
+        await sendgridProvider.removeContact(granted.email);
+        await sendgridProvider.removeContact(declined.email);
+        await beehiivProvider.removeContact(granted.email);
+        await beehiivProvider.removeContact(declined.email);
+
+        const grantedSg = await pollProvider(() => sendgridProvider.findContact(granted.email), false);
+        const declinedSg = await pollProvider(() => sendgridProvider.findContact(declined.email), false);
+        const grantedBh = await pollProvider(() => beehiivProvider.findContact(granted.email), false);
+        const declinedBh = await pollProvider(() => beehiivProvider.findContact(declined.email), false);
+
+        assert.equal(grantedSg, null, 'granted account should be absent from SendGrid');
+        assert.equal(declinedSg, null, 'declined account should be absent from SendGrid');
+        assert.equal(grantedBh, null, 'granted account should be absent from Beehiiv');
+        assert.equal(declinedBh, null, 'declined account should be absent from Beehiiv');
+      },
+    },
+
+    // ─────────────────────────────────────────────────────────────────────
+    // Phase 2 — Sync granted account → both providers
+    // ─────────────────────────────────────────────────────────────────────
+    {
+      name: 'phase-2a-granted-account-syncs-to-both-providers',
+      auth: 'admin',
+      timeout: 180000,
+
+      async run({ accounts, assert, Manager, assistant }) {
+        const granted = accounts['consent-granted'];
+        const admin = Manager.libraries.admin;
+
+        // Set consent.marketing.status = 'granted' on the user doc
+        await admin.firestore().doc(`users/${granted.uid}`).set({
+          consent: {
+            marketing: {
+              status: 'granted',
+              grantedAt: {
+                timestamp: new Date().toISOString(),
+                timestampUNIX: Math.floor(Date.now() / 1000),
+                source: 'test',
+                ip: null,
+                text: 'consent-lifecycle test',
+              },
+            },
+            legal: {
+              status: 'granted',
+              grantedAt: {
+                timestamp: new Date().toISOString(),
+                timestampUNIX: Math.floor(Date.now() / 1000),
+                source: 'test',
+                ip: null,
+                text: 'consent-lifecycle test',
+              },
+            },
+          },
+        }, { merge: true });
+
+        // Trigger marketing sync via the Email() surface
+        const result = await Manager.Email(assistant).sync(granted.uid);
+        assert.ok(result, 'sync should return a result');
+        assert.notEqual(result.blocked, 'validation', 'sync should not be blocked by validation (uses _test.allow_*)');
+
+        // Poll providers until they reflect the upsert. SendGrid's upsert is
+        // an async background job (returns a job_id, not the inserted contact)
+        // so a single check 5s later isn't enough — it can take 10-20s to
+        // surface. Beehiiv is usually instant but uses the same poll for symmetry.
+        const sgContact = await pollProvider(() => sendgridProvider.findContact(granted.email), true);
+        const bhContact = await pollProvider(() => beehiivProvider.findContact(granted.email), true);
+
+        assert.ok(sgContact, 'granted account should now exist in SendGrid');
+        assert.ok(bhContact, 'granted account should now exist in Beehiiv');
+      },
+    },
+
+    // ─────────────────────────────────────────────────────────────────────
+    // Phase 2b — Declined account: verify it stays out of providers when we
+    // DON'T call sync (which is what the signup route does for declined
+    // marketing consent).
+    // ─────────────────────────────────────────────────────────────────────
+    {
+      name: 'phase-2b-declined-account-stays-out-of-providers',
+      auth: 'admin',
+      timeout: 180000,
+
+      async run({ accounts, assert, Manager, assistant }) {
+        const declined = accounts['consent-declined'];
+        const admin = Manager.libraries.admin;
+
+        // Set consent.marketing.status = 'revoked' on the user doc.
+        // We deliberately do NOT call sync — the production signup route gates
+        // on consent.marketing.status === 'granted' before calling sync, so a
+        // declined user simply never gets a sync call. We verify that contract
+        // by NOT calling sync and asserting the user stays absent.
+        await admin.firestore().doc(`users/${declined.uid}`).set({
+          consent: {
+            marketing: {
+              status: 'revoked',
+              grantedAt: { timestamp: null, timestampUNIX: null, source: null, ip: null, text: null },
+              revokedAt: {
+                timestamp: new Date().toISOString(),
+                timestampUNIX: Math.floor(Date.now() / 1000),
+                source: 'test',
+                ip: null,
+                text: null,
+              },
+            },
+            legal: {
+              status: 'granted',
+              grantedAt: {
+                timestamp: new Date().toISOString(),
+                timestampUNIX: Math.floor(Date.now() / 1000),
+                source: 'test',
+                ip: null,
+                text: 'consent-lifecycle test',
+              },
+            },
+          },
+        }, { merge: true });
+
+        await sleep(SETTLE_MS);
+
+        const sgContact = await sendgridProvider.findContact(declined.email);
+        const bhContact = await beehiivProvider.findContact(declined.email);
+
+        assert.equal(sgContact, null, 'declined account should remain absent from SendGrid');
+        assert.equal(bhContact, null, 'declined account should remain absent from Beehiiv');
+      },
+    },
+
+    // ─────────────────────────────────────────────────────────────────────
+    // Phase 3 — Unsubscribe: granted account is removed via Manager.Email().remove
+    // ─────────────────────────────────────────────────────────────────────
+    {
+      name: 'phase-3-granted-account-unsubscribe-removes-from-both',
+      auth: 'admin',
+      timeout: 180000,
+
+      async run({ accounts, assert, Manager, assistant }) {
+        const granted = accounts['consent-granted'];
+
+        // Trigger removal — simulates the email-preferences opt-out flow
+        const result = await Manager.Email(assistant).remove(granted.email);
+        assert.ok(result, 'remove should return a result');
+
+        // Poll for absence — SendGrid's contact delete is also an async job.
+        const sgContact = await pollProvider(() => sendgridProvider.findContact(granted.email), false);
+        const bhContact = await pollProvider(() => beehiivProvider.findContact(granted.email), false);
+
+        assert.equal(sgContact, null, 'granted account should now be absent from SendGrid');
+        assert.equal(bhContact, null, 'granted account should now be absent from Beehiiv');
+      },
+    },
+
+    // ─────────────────────────────────────────────────────────────────────
+    // Phase 4 — Validation gate: _test.* (non-allow) is blocked by validate()
+    // ─────────────────────────────────────────────────────────────────────
+    {
+      name: 'phase-4-non-allow-test-email-blocked-by-validation',
+      auth: 'admin',
+      timeout: 30000,
+
+      async run({ assert, Manager, assistant }) {
+        // _test.never-reaches-providers@... is NOT _test.allow_* → should be blocked
+        const blockedEmail = '_test.never-reaches-providers@somiibo.com';
+
+        const result = await Manager.Email(assistant).add({ email: blockedEmail });
+
+        assert.equal(result.blocked, 'validation', 'non-allow _test.* email should be blocked by validation');
+
+        // And the provider lookup should show nothing
+        const sgContact = await sendgridProvider.findContact(blockedEmail);
+        const bhContact = await beehiivProvider.findContact(blockedEmail);
+
+        assert.equal(sgContact, null, 'blocked email should never appear in SendGrid');
+        assert.equal(bhContact, null, 'blocked email should never appear in Beehiiv');
+      },
+    },
+  ],
+};