backend-manager 5.11.1 → 5.11.4

package/CLAUDE.md

@@ -183,7 +183,7 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 
 ### Testing & CLI
 
-- [docs/test-framework.md](docs/test-framework.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels. **NEVER mock — test against the real emulator.** No `mockManager`/`mockAdmin`/fake `firestore`/stubbed `assistant`; every `run()` gets the real `Manager`/`assistant`/`firestore`/`http`/`accounts` — use them. Pure functions (zero I/O) are the only thing you call directly; anything touching Firestore or an external API runs for real. Real external APIs (OpenAI/PayPal/GitHub/SendGrid/Stripe) are gated behind `TEST_EXTENDED_MODE` in-source (not mocked) — opt in with `--extended` or `TEST_EXTENDED_MODE=true` (shared, unprefixed across BEM/BXM/UJM/EM; propagates to BOTH runner + emulator) — and anything an extended test creates externally must be cleaned up by the test. **Each test file `module.exports` a `{ description, type, tests }` object — NOT raw Mocha (`describe`/`it`/`beforeEach`); those globals are not injected and the file fails to load. Split tests one-file-per-concern under `test/<area>/`, never one giant `test/test.js`.** **All cleanup runs at the START of every run, never at the end** — the runner flushes the ENTIRE emulator Firestore before every run, so there's nothing to register; seed any needed fixtures in `test/_init.js`'s `setup()`, and never add a trailing cleanup step. Marketing providers (SendGrid/Beehiiv) don't need a special exception — `_test.*` emails are blocked at the validation layer so test signups never reach providers. The `_test.allow_*` carve-out exists only for the live-provider lifecycle test (`test/marketing/consent-lifecycle.js`), which manages its own teardown.
+- [docs/test-framework.md](docs/test-framework.md) — running, filtering, log files, test types (standalone/suite/group), context object, assertions, auth levels. **NEVER mock — test against the real emulator.** No `mockManager`/`mockAdmin`/fake `firestore`/stubbed `assistant`; every `run()` gets the real `Manager`/`assistant`/`firestore`/`http`/`accounts` — use them. Pure functions (zero I/O) are the only thing you call directly; anything touching Firestore or an external API runs for real. Real external APIs (OpenAI/PayPal/GitHub/SendGrid/Stripe) are gated behind `TEST_EXTENDED_MODE` in-source (not mocked) — opt in with `--extended` or `TEST_EXTENDED_MODE=true` (shared, unprefixed across BEM/BXM/UJM/EM; propagates to BOTH runner + emulator) — and anything an extended test creates externally must be cleaned up by the test. **Each test file `module.exports` a `{ description, type, tests }` object — NOT raw Mocha (`describe`/`it`/`beforeEach`); those globals are not injected and the file fails to load. The only lifecycle hook is `cleanup` — exported `before`/`after` properties are silently IGNORED (do setup inside the tests via an idempotent helper, or in `test/_init.js`). Split tests one-file-per-concern under `test/<area>/`, never one giant `test/test.js`.** **All cleanup runs at the START of every run, never at the end** — the runner flushes the ENTIRE emulator Firestore before every run, so there's nothing to register; seed any needed fixtures in `test/_init.js`'s `setup()`, and never add a trailing cleanup step. Marketing providers (SendGrid/Beehiiv) don't need a special exception — `_test.*` emails are blocked at the validation layer so test signups never reach providers. The `_test.allow_*` carve-out exists only for the live-provider lifecycle test (`test/marketing/consent-lifecycle.js`), which manages its own teardown.
 - [docs/test-boot-layer.md](docs/test-boot-layer.md) — the `boot/` smoke layer: framework self-test from the repo via the bundled fixture project + `BEM_TEST_BOOT_PROJECT` (BEM's analog of BXM/UJM `*_TEST_BOOT_PROJECT`)
 - [docs/cli-firestore-auth.md](docs/cli-firestore-auth.md) — `npx mgr firestore:*` and `auth:*` commands, shared flags, examples
 - [docs/cli-logs.md](docs/cli-logs.md) — `npx mgr logs:read` / `logs:tail` with full flag reference and built-in Cloud Function names

package/docs/marketing-campaigns.md

@@ -15,13 +15,16 @@
 {
   settings: { name, subject, preheader, content, template, sender, segments, excludeSegments, ... },
   sendAt: 1743465600,        // Unix timestamp (any format accepted, normalized on create)
-  status: 'pending',         // pending | sent | failed
+  status: 'pending',         // pending | processing | sent | failed
   type: 'email',             // email | push
   recurrence: { pattern, day, hour, minute?, nth? },  // Optional — makes it recurring
   generator: 'newsletter',   // Optional — runs content generator before sending
   recurringId: '_recurring-sale',      // Present on history docs (links to parent template)
   generatedFrom: '_recurring-newsletter', // Present on generated docs
   results: { campaigns: {...}, newsletter: {...} },
+  processingStartedAt: 1743465610,  // Set while claimed by a cron run (lease)
+  generatorAttempts: 2,      // Consecutive empty generator runs (retry-cap bookkeeping)
+  error: 'Unknown generator "..."',  // Set when marked failed without sending
   metadata: { created: {...}, updated: {...} },
 }
 ```
@@ -32,29 +35,44 @@
 - **Push**: dispatches to FCM via `notification.send()` (shared library)
 - Content is **markdown** — converted to HTML at send time. Template variables resolved before conversion.
 
+## Claim/Lease Lifecycle
+
+Every due campaign is **claimed** before any work happens: the cron transactionally flips `status: 'pending'` → `'processing'` (stamping `processingStartedAt`). A campaign another (overlapping) run already claimed is skipped — double-sends are impossible by construction.
+
+- Every terminal path writes a final status: `sent`/`failed` for one-offs, back to `pending` (with an advanced `sendAt`) for recurring
+- If a run dies mid-flight (crash, function timeout), the doc stays `processing` until the **stale-lease reclaim** flips it back to `pending` after `PROCESSING_LEASE_SECONDS` (30 min) — a natural retry backoff
+- Unknown campaign `type`s and unknown `generator`s are marked `failed` with an `error` field (a config typo must not retry every 10 minutes forever)
+- `PUT /marketing/campaign` only edits `pending` campaigns, so a mid-flight campaign can't be edited out from under the cron
+
 ## Recurring Campaigns
 
 Campaigns with a `recurrence` field repeat automatically:
 - Cron fires → creates a **history doc** (same collection, `recurringId` set) → advances `sendAt` to next occurrence
-- Status stays `pending` on the recurring template, history docs are `sent`/`failed`
+- Status returns to `pending` on the recurring template after each run, history docs are `sent`/`failed`
 - `_` prefix on IDs groups them at top of Firestore console
 
 Recurrence patterns: `daily`, `weekly`, `monthly`, `monthly-weekday`, `quarterly`, `yearly`
 
 The `monthly-weekday` pattern targets the Nth weekday of each month (e.g., 2nd Wednesday). Requires `nth` (1-4) and `day` (0=Sun–6=Sat) in the recurrence object. All other patterns use simple interval addition from the current `sendAt`.
 
-All scheduling helpers live in `constants.js` (SSOT): `nextWeekday()`, `nextNthWeekday()`, `nextMonthDay()`, `getNextOccurrence()`. The cron job imports from there — no duplicated logic.
+All scheduling helpers live in `constants.js` (SSOT): `nextWeekday()`, `nextNthWeekday()`, `nextMonthDay()`, `getNextOccurrence()`, `getNextFutureOccurrence()`. The cron job imports from there — no duplicated logic.
+
+**Catch-up safety:** the cron advances recurring campaigns with `getNextFutureOccurrence()`, which skips ALL missed occurrences and always lands in the future. A weekly campaign stalled for 3 weeks (cron outage, dried-up sources) sends ONE issue and resumes its schedule — never a catch-up burst of back-to-back sends.
 
 ## Generator Campaigns
 
 Campaigns with a `generator` field (e.g. `generator: 'newsletter'`) are handled by the frequent cron inline — generate content and send in one shot when `sendAt` is due:
-1. Frequent cron finds the generator campaign past its `sendAt`
+1. Frequent cron claims the generator campaign past its `sendAt`
 2. Runs the generator module (e.g., `generators/newsletter.js`)
 3. Sends the generated content immediately
-4. Stores a history record with generated content + send results
-5. Advances the recurring template's `sendAt` to the next occurrence
+4. Stores a history record with generated content + send results (bulky byproducts — `article`, `structure`, `mjml`, image buffers — are stripped from the stored settings)
+5. Recurring: advances the template's `sendAt` to the next FUTURE occurrence and returns it to `pending`. One-off: finalized to `sent`/`failed` so it is never picked up again
+
+**Empty-run retry cap:** when the generator yields nothing (no sources, filter dropped everything), the campaign returns to `pending` with `generatorAttempts` incremented and retries next run. After `GENERATOR_MAX_ATTEMPTS` (36 ≈ 6h at the 10-minute cadence) a recurring campaign skips to its next occurrence (counter reset); a one-off is marked `failed`.
+
+**Test gating:** `generate()` short-circuits to `null` under `assistant.isTesting()` without `TEST_EXTENDED_MODE` — same convention as the marketing providers and the blog cron. A normal consumer test run never spends AI tokens or touches GitHub/Beehiiv; extended runs are production-equivalent.
 
-In production, Beehiiv posts are published (`status: 'confirmed'`). In testing, they're forced to draft (`status: 'draft'`). If Beehiiv upload fails, a fallback alert email is sent to `alerts@{brandDomain}` with all asset links for manual upload.
+In production, Beehiiv posts are published (`status: 'confirmed'`). In testing, they're forced to draft (`status: 'draft'`). The report email always goes to `alerts@{brandDomain}` with all asset links; its subject adapts to whether Beehiiv upload succeeded.
 
 ## Email Rendering
 

package/docs/test-framework.md

@@ -236,6 +236,8 @@ Use `bem:` or `project:` prefix to filter by source. **Mirror the source path so
 ## Test Types
 
 > **The runner reads each file's `module.exports` object — it does NOT inject Mocha/Jest globals.** A test file that calls `describe`/`it`/`before`/`beforeEach`/`after` at top level throws `ReferenceError: beforeEach is not defined` and shows as `Failed to load`. There is no global `assert` either — use the `assert` passed into `run({ assert })`. Every test file MUST export one of the shapes below.
+>
+> **The only lifecycle hook is `cleanup`** (per-test and module-level). Exporting `before`/`after`/`beforeEach` properties on the module does NOT error — the runner silently ignores them, so setup they were supposed to do never happens (this bit `helpers/ai-schema-resolve`, which failed for weeks because its fixtures were written in a `before()` that never ran). Seed disk/DB fixtures inside the tests themselves via an idempotent helper, or in `test/_init.js`'s `setup()`.
 
 | Type | Use When | Behavior |
 |------|----------|----------|

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backend-manager",
-  "version": "5.11.1",
+  "version": "5.11.4",
   "description": "Quick tools for developing Firebase functions",
   "main": "src/manager/index.js",
   "files": [

package/src/manager/events/cron/daily/blog-auto-publisher.js

@@ -33,6 +33,15 @@ const {
 let postId;
 
 module.exports = async ({ Manager, assistant, context, libraries }) => {
+  // External boundary (live feed fetches + AI generation + publishing) — the
+  // cron entry gates itself like every other external call. Ungated, a normal
+  // test run harvests real feeds from the consumer's blog config, delaying the
+  // serial bm_cronDaily sequence past the usage suite's reset-usage deadline.
+  if (assistant.isTesting() && !process.env.TEST_EXTENDED_MODE) {
+    assistant.log('Blog auto-publisher skipped (test mode without TEST_EXTENDED_MODE)');
+    return;
+  }
+
   if (!Manager.config.blog?.enabled) {
     assistant.log('Blog auto-publisher disabled in config');
     return;

package/src/manager/events/cron/frequent/marketing-campaigns.js

@@ -6,31 +6,79 @@
  *   - email: fires through mailer.sendCampaign()
  *   - push: fires through notification.send()
  *
+ * Claim/lease lifecycle (prevents double-sends across overlapping runs):
+ *   - Each due campaign is CLAIMED by transactionally flipping status
+ *     'pending' → 'processing' before any work happens. A campaign another
+ *     run already claimed is skipped.
+ *   - Every terminal path writes a final status: 'sent'/'failed' for
+ *     one-offs, back to 'pending' (with an advanced sendAt) for recurring.
+ *   - If a run dies mid-flight (crash, function timeout), the doc stays
+ *     'processing' until the stale-lease reclaim flips it back to 'pending'
+ *     after PROCESSING_LEASE_SECONDS — a natural retry backoff.
+ *
  * Generator campaigns (has `generator` field, e.g. 'newsletter'):
  *   - Runs the content generation pipeline (AI content, images, uploads)
  *   - Sends the generated content immediately
  *   - Stores a history record with the generated content + send results
- *   - Advances the recurring template's sendAt to the next occurrence
+ *   - When generation yields nothing, retries next run — up to
+ *     GENERATOR_MAX_ATTEMPTS total, then the occurrence is skipped
+ *     (recurring) or the campaign is marked failed (one-off)
  *
  * Recurring campaigns (has `recurrence` field):
  *   - Creates a history doc in the same collection with results
- *   - Advances the recurring doc's sendAt to the next occurrence
- *   - Status stays 'pending' on the recurring doc
+ *   - Advances the recurring doc's sendAt to the next FUTURE occurrence
+ *     (missed occurrences are skipped — no catch-up bursts)
+ *   - Status returns to 'pending' on the recurring doc
+ *
+ * Unknown campaign types / generators are marked 'failed' (they can never
+ * succeed — usually a config typo — so retrying forever just burns runs).
  *
  * Runs on bm_cronFrequent (every 10 minutes).
  */
 const moment = require('moment');
 const pushid = require('pushid');
 const notification = require('../../../libraries/notification.js');
-const { getNextOccurrence } = require('../../../libraries/email/constants.js');
+const { getNextFutureOccurrence } = require('../../../libraries/email/constants.js');
+
+// How long a 'processing' lease is honored before the campaign of a crashed or
+// timed-out run is reclaimed for retry. Must comfortably exceed the function
+// timeout so an in-flight run is never reclaimed out from under itself.
+const PROCESSING_LEASE_SECONDS = 30 * 60;
+
+// How many times a generator campaign may yield nothing before its occurrence
+// is skipped (recurring) or the campaign is failed (one-off). At the 10-minute
+// cron cadence this is ~6 hours of retries.
+const GENERATOR_MAX_ATTEMPTS = 36;
 
 module.exports = async ({ Manager, assistant, libraries }) => {
   const { admin } = libraries;
   const now = Math.round(Date.now() / 1000);
+  const collection = admin.firestore().collection('marketing-campaigns');
+
+  // --- Reclaim stale processing leases (crashed or timed-out runs) ---
+  // Single-equality query + in-memory cutoff — no composite index required.
+  const processingSnapshot = await collection
+    .where('status', '==', 'processing')
+    .limit(50)
+    .get();
+
+  for (const doc of processingSnapshot.docs) {
+    const startedAt = doc.data().processingStartedAt || 0;
+
+    if (startedAt > now - PROCESSING_LEASE_SECONDS) {
+      continue;
+    }
+
+    await doc.ref.set({
+      status: 'pending',
+      metadata: { updated: stamp() },
+    }, { merge: true });
+
+    assistant.log(`Reclaimed stale processing lease on ${doc.id} (started ${moment.unix(startedAt).toISOString()})`);
+  }
 
-  // Query campaigns that are ready to send
-  const snapshot = await admin.firestore()
-    .collection('marketing-campaigns')
+  // --- Query campaigns that are ready to send ---
+  const snapshot = await collection
     .where('status', '==', 'pending')
     .where('sendAt', '<=', now)
     .limit(20)
@@ -47,9 +95,18 @@ module.exports = async ({ Manager, assistant, libraries }) => {
 
   const results = await Promise.allSettled(snapshot.docs.map(async (doc) => {
     const data = doc.data();
-    let { settings, type, recurrence, generator } = data;
+    const { settings, type, recurrence, generator } = data;
     const campaignId = doc.id;
 
+    // Claim the campaign before doing ANY work. If another (overlapping) run
+    // already claimed it, skip — this is what makes double-sends impossible.
+    const claimed = await claimCampaign(admin, doc, now);
+
+    if (!claimed) {
+      assistant.log(`Campaign ${campaignId} already claimed by another run, skipping`);
+      return;
+    }
+
     assistant.log(`Processing campaign ${campaignId} (${type}): ${settings.name}`);
 
     // --- Generator campaigns: generate content + send in one shot ---
@@ -59,7 +116,13 @@ module.exports = async ({ Manager, assistant, libraries }) => {
       };
 
       if (!generators[generator]) {
-        assistant.log(`Unknown generator "${generator}" on ${campaignId}, skipping`);
+        await doc.ref.set({
+          status: 'failed',
+          error: `Unknown generator "${generator}"`,
+          metadata: { updated: stamp() },
+        }, { merge: true });
+
+        assistant.log(`Unknown generator "${generator}" on ${campaignId} — marked failed`);
         return;
       }
 
@@ -72,16 +135,55 @@ module.exports = async ({ Manager, assistant, libraries }) => {
         publishArticle: Manager.isProduction(),
       });
 
+      // Nothing generated (no sources, filter dropped everything, disabled in
+      // test mode, ...). Retry next run — up to the attempts cap.
       if (!generated) {
-        assistant.log(`Generator "${generator}" returned no content for ${campaignId}, will retry next run`);
+        const attempts = (data.generatorAttempts || 0) + 1;
+
+        if (attempts >= GENERATOR_MAX_ATTEMPTS) {
+          if (recurrence) {
+            const nextSendAt = getNextFutureOccurrence(data.sendAt, recurrence, now);
+
+            await doc.ref.set({
+              status: 'pending',
+              sendAt: nextSendAt,
+              generatorAttempts: admin.firestore.FieldValue.delete(),
+              metadata: { updated: stamp() },
+            }, { merge: true });
+
+            assistant.log(`Generator "${generator}" yielded nothing ${attempts}x on ${campaignId} — skipping to next occurrence: ${moment.unix(nextSendAt).toISOString()}`);
+          } else {
+            await doc.ref.set({
+              status: 'failed',
+              error: `Generator "${generator}" yielded no content after ${attempts} attempts`,
+              metadata: { updated: stamp() },
+            }, { merge: true });
+
+            assistant.log(`Generator "${generator}" yielded nothing ${attempts}x on one-off ${campaignId} — marked failed`);
+          }
+
+          return;
+        }
+
+        await doc.ref.set({
+          status: 'pending',
+          generatorAttempts: attempts,
+          metadata: { updated: stamp() },
+        }, { merge: true });
+
+        assistant.log(`Generator "${generator}" returned no content for ${campaignId}, will retry next run (attempt ${attempts}/${GENERATOR_MAX_ATTEMPTS})`);
         return;
       }
 
+      // Strip generation byproducts: bulky debug/asset fields stay OUT of the
+      // send payload and the history doc's settings (article carries the full
+      // post body; assets/meta are stored as their own fields below).
       const {
         images: _images,
         mjml: _mjml,
         structure: _structure,
         contentMarkdown: _contentMarkdown,
+        article: _article,
         assets,
         meta,
         ...generatedSettings
@@ -93,9 +195,6 @@ module.exports = async ({ Manager, assistant, libraries }) => {
       const campaignResults = await email.sendCampaign({ ...generatedSettings, sendAt: 'now' });
       const success = Object.values(campaignResults).some(r => r.success || r.sent > 0);
 
-      const nowISO = new Date().toISOString();
-      const nowUNIX = Math.round(Date.now() / 1000);
-
       // Store history record
       const historyId = pushid();
       await admin.firestore().doc(`marketing-campaigns/${historyId}`).set({
@@ -107,23 +206,30 @@ module.exports = async ({ Manager, assistant, libraries }) => {
         status: success ? 'sent' : 'failed',
         results: campaignResults,
         generatedFrom: campaignId,
-        metadata: {
-          created: { timestamp: nowISO, timestampUNIX: nowUNIX },
-          updated: { timestamp: nowISO, timestampUNIX: nowUNIX },
-        },
+        metadata: { created: stamp(), updated: stamp() },
       });
 
-      // Advance sendAt to next occurrence
       if (recurrence) {
-        const nextSendAt = getNextOccurrence(data.sendAt, recurrence);
+        // Advance sendAt to the next FUTURE occurrence
+        const nextSendAt = getNextFutureOccurrence(data.sendAt, recurrence, now);
+
         await doc.ref.set({
+          status: 'pending',
           sendAt: nextSendAt,
-          metadata: {
-            updated: { timestamp: nowISO, timestampUNIX: nowUNIX },
-          },
+          generatorAttempts: admin.firestore.FieldValue.delete(),
+          metadata: { updated: stamp() },
         }, { merge: true });
+
         assistant.log(`${success ? 'Sent' : 'Failed'} generator campaign ${campaignId}, next: ${moment.unix(nextSendAt).toISOString()}`);
       } else {
+        // One-off: finalize so it is never picked up again
+        await doc.ref.set({
+          status: success ? 'sent' : 'failed',
+          results: campaignResults,
+          generatorAttempts: admin.firestore.FieldValue.delete(),
+          metadata: { updated: stamp() },
+        }, { merge: true });
+
         assistant.log(`${success ? 'Sent' : 'Failed'} generator campaign ${campaignId} (one-off)`);
       }
 
@@ -150,13 +256,17 @@ module.exports = async ({ Manager, assistant, libraries }) => {
         }),
       };
     } else {
-      assistant.log(`Unknown campaign type "${type}", skipping ${campaignId}`);
+      await doc.ref.set({
+        status: 'failed',
+        error: `Unknown campaign type "${type}"`,
+        metadata: { updated: stamp() },
+      }, { merge: true });
+
+      assistant.log(`Unknown campaign type "${type}" on ${campaignId} — marked failed`);
       return;
     }
 
     const success = Object.values(campaignResults).some(r => r.success || r.sent > 0);
-    const nowISO = new Date().toISOString();
-    const nowUNIX = Math.round(Date.now() / 1000);
 
     // --- Handle recurring vs one-off ---
     if (recurrence) {
@@ -170,20 +280,16 @@ module.exports = async ({ Manager, assistant, libraries }) => {
         status: success ? 'sent' : 'failed',
         results: campaignResults,
         recurringId: campaignId,
-        metadata: {
-          created: { timestamp: nowISO, timestampUNIX: nowUNIX },
-          updated: { timestamp: nowISO, timestampUNIX: nowUNIX },
-        },
+        metadata: { created: stamp(), updated: stamp() },
       });
 
-      // Advance sendAt to next occurrence
-      const nextSendAt = getNextOccurrence(data.sendAt, recurrence);
+      // Advance sendAt to the next FUTURE occurrence
+      const nextSendAt = getNextFutureOccurrence(data.sendAt, recurrence, now);
 
       await doc.ref.set({
+        status: 'pending',
         sendAt: nextSendAt,
-        metadata: {
-          updated: { timestamp: nowISO, timestampUNIX: nowUNIX },
-        },
+        metadata: { updated: stamp() },
       }, { merge: true });
 
       assistant.log(`Recurring campaign ${campaignId} ${success ? 'sent' : 'failed'}, next: ${moment.unix(nextSendAt).toISOString()}`);
@@ -192,9 +298,7 @@ module.exports = async ({ Manager, assistant, libraries }) => {
       await doc.ref.set({
         status: success ? 'sent' : 'failed',
         results: campaignResults,
-        metadata: {
-          updated: { timestamp: nowISO, timestampUNIX: nowUNIX },
-        },
+        metadata: { updated: stamp() },
       }, { merge: true });
 
       assistant.log(`Campaign ${campaignId} ${success ? 'sent' : 'failed'}`);
@@ -206,9 +310,45 @@ module.exports = async ({ Manager, assistant, libraries }) => {
 
   for (const r of results) {
     if (r.status === 'rejected') {
+      // The campaign doc stays 'processing' — the stale-lease reclaim retries
+      // it after PROCESSING_LEASE_SECONDS instead of every 10 minutes.
       assistant.error(`Failed to process campaign: ${r.reason?.message}`, r.reason);
     }
   }
 
   assistant.log(`Completed! (${sent} processed, ${failed} failed)`);
 };
+
+/**
+ * Transactionally claim a pending campaign by flipping status to 'processing'.
+ * Returns false when another run (or an earlier claim) got there first.
+ */
+function claimCampaign(admin, doc, now) {
+  return admin.firestore().runTransaction(async (tx) => {
+    const fresh = await tx.get(doc.ref);
+
+    if (!fresh.exists || fresh.data().status !== 'pending') {
+      return false;
+    }
+
+    tx.set(doc.ref, {
+      status: 'processing',
+      processingStartedAt: now,
+      metadata: { updated: stamp() },
+    }, { merge: true });
+
+    return true;
+  });
+}
+
+function stamp() {
+  return {
+    timestamp: new Date().toISOString(),
+    timestampUNIX: Math.round(Date.now() / 1000),
+  };
+}
+
+// Exposed for tests (test/email/campaign-cron-pipeline.js) — single source of
+// truth for the lease + retry-cap tuning.
+module.exports.PROCESSING_LEASE_SECONDS = PROCESSING_LEASE_SECONDS;
+module.exports.GENERATOR_MAX_ATTEMPTS = GENERATOR_MAX_ATTEMPTS;

package/src/manager/index.js

@@ -1126,8 +1126,12 @@ Manager.prototype.setupFunctions = function (exporter, options) {
   .pubsub.schedule('0 0 * * *')
   .onRun((context) => self.EventMiddleware({ context }).run(`${cron}/daily.js`));
 
+  // Frequent cron runs the inline newsletter generator (AI structure + section
+  // images + article + uploads) — needs the v1 max timeout and headroom for
+  // image buffers. If a run still times out, the campaign lease reclaim in
+  // marketing-campaigns.js retries it safely.
   exporter.bm_cronFrequent =
-  fn({memory: '256MB', timeoutSeconds: 60 * 5})
+  fn({memory: '512MB', timeoutSeconds: 540})
   .pubsub.schedule('*/10 * * * *')
   .onRun((context) => self.EventMiddleware({ context }).run(`${cron}/frequent.js`));
 };

package/src/manager/libraries/content/source-resolver.js

@@ -526,7 +526,14 @@ function getURLContent(url) {
   });
 }
 
+// http(s) only — a bare `new URL()` check misclassifies colon-prefixed text
+// seeds ("AI: the future of work" parses with scheme "ai:") as URLs, which
+// then fail to fetch and silently lose the seed.
 function isURL(url) {
+  if (typeof url !== 'string' || !/^https?:\/\//i.test(url)) {
+    return false;
+  }
+
   try {
     return !!new URL(url);
   } catch (e) {

package/src/manager/libraries/email/constants.js

@@ -193,6 +193,37 @@ function getNextOccurrence(currentSendAt, recurrence) {
   }
 }
 
+/**
+ * Like getNextOccurrence, but guarantees the returned time is in the FUTURE.
+ *
+ * A recurring campaign that stalled for several periods (cron outage, sources
+ * dried up, repeated failures) advances past ALL missed occurrences instead of
+ * firing a catch-up burst — one send per cron tick until sendAt catches up.
+ *
+ * @param {number} currentSendAt - Current fire time (unix)
+ * @param {object} recurrence - { pattern, hour, day, nth?, minute? }
+ * @param {number} [now] - Unix time to advance past (defaults to Date.now())
+ * @returns {number} Next fire time strictly after `now`
+ */
+function getNextFutureOccurrence(currentSendAt, recurrence, now) {
+  now = now || Math.round(Date.now() / 1000);
+
+  let next = getNextOccurrence(currentSendAt, recurrence);
+
+  while (next <= now) {
+    const after = getNextOccurrence(next, recurrence);
+
+    // Safety: a recurrence must strictly advance, otherwise bail out
+    if (after <= next) {
+      break;
+    }
+
+    next = after;
+  }
+
+  return next;
+}
+
 /**
  * Convert SVG image URLs to PNG equivalents — email clients don't render SVGs.
  * CDN naming convention: `-x.svg` -> `-1024.png`
@@ -401,4 +432,5 @@ module.exports = {
   nextNthWeekday,
   nextMonthDay,
   getNextOccurrence,
+  getNextFutureOccurrence,
 };

package/src/manager/libraries/email/generators/lib/image-host.js

@@ -238,10 +238,9 @@ async function uploadAssets({ images, html, markdown, summary, brandId, campaign
     });
   }
 
-  // 4. Commit + push with retry. The newsletter pipeline calls uploadAssets
-  //    twice sequentially (images then HTML). GitHub's API can return a stale
-  //    ref on the second getRef if the first updateRef hasn't fully propagated.
-  //    Retry with a fresh ref read + jitter handles this.
+  // 4. Commit + push with retry. Per-brand branches make ref races effectively
+  //    impossible (one writer per branch), but keep a defensive retry in case
+  //    the same brand ever uploads concurrently or GitHub returns a stale ref.
   const defaultSubject = subject ? subject.trim() : `${files.length} newsletter asset${files.length === 1 ? '' : 's'}`;
   const message = commitMessage || `[${brandId}] ${campaignId} — ${defaultSubject}`;
   const MAX_RETRIES = 3;

package/src/manager/libraries/email/generators/newsletter.js

@@ -69,6 +69,15 @@ const { trackContentSource, contentSourceHash, resolveSources } = require('../..
  * @returns {object|null} Updated settings with content filled in, or null if unavailable
  */
 async function generate(Manager, assistant, settings, opts = {}) {
+  // Same convention as the marketing providers (email/marketing/index.js):
+  // default test runs never hit the real pipeline (AI calls + GitHub commit +
+  // Beehiiv post + report email). TEST_EXTENDED_MODE opts into
+  // production-equivalent side effects.
+  if (assistant.isTesting() && !process.env.TEST_EXTENDED_MODE) {
+    assistant.log('Newsletter generator: skipped in test mode (set TEST_EXTENDED_MODE to run the real pipeline)');
+    return null;
+  }
+
   const newsletterRoleConfig = Manager.config?.marketing?.newsletter;
   const rawContent = newsletterRoleConfig?.content;
 
@@ -270,6 +279,7 @@ async function generate(Manager, assistant, settings, opts = {}) {
       brand,
       config,
       structure,
+      sources: filteredSources,
       publish: !!opts.publishArticle,
     }).catch((e) => {
       assistant.error(`Newsletter generator: linked article failed — ${e.message}`);
@@ -523,10 +533,12 @@ async function generate(Manager, assistant, settings, opts = {}) {
  * @param {object} args.brand - { id, name, url, ... }
  * @param {object} args.config - marketing.newsletter.content (tone, instructions, article.author)
  * @param {object} args.structure - newsletter structure (sections[0] is the lead)
+ * @param {object[]} [args.sources] - The brand-fit sources that fed the issue. Their URLs are
+ *                                    passed to admin/post as the article's `source` attribution.
  * @param {boolean} [args.publish] - Commit the post to GitHub via admin/post. Default false.
  * @returns {Promise<{url, slug, path, published}|null>}
  */
-async function buildLinkedArticle({ Manager, assistant, brand, config, structure, publish }) {
+async function buildLinkedArticle({ Manager, assistant, brand, config, structure, sources, publish }) {
   const lead = structure.sections[0] || {};
   const publicConfig = buildPublicConfig(Manager.config);
 
@@ -566,7 +578,7 @@ async function buildLinkedArticle({ Manager, assistant, brand, config, structure
   }
 
   try {
-    const sourceUrls = sources.map(s => s.url || s.id).filter(Boolean);
+    const sourceUrls = (sources || []).map(s => s.url || s.id).filter(Boolean);
     const result = await publishArticle(assistant, {
       brand: publicConfig,
       article,

package/src/test/fixtures/firebase-project/.temp/test-mode.json

@@ -2,5 +2,5 @@
   "env": {
     "TEST_EXTENDED_MODE": ""
   },
-  "updatedAt": "2026-07-02T07:46:48.765Z"
+  "updatedAt": "2026-07-02T08:47:59.846Z"
 }

package/src/test/fixtures/firebase-project/database-debug.log

@@ -2,11 +2,11 @@ WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
 WARNING: sun.misc.Unsafe::objectFieldOffset has been called by akka.util.Unsafe (file:/Users/ian/.cache/firebase/emulators/firebase-database-emulator-v4.11.2.jar)
 WARNING: Please consider reporting this to the maintainers of class akka.util.Unsafe
 WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
-00:46:55.447 [NamespaceSystem-akka.actor.default-dispatcher-5] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
-00:46:55.566 [main] INFO com.firebase.server.forge.App$ - Listening at 127.0.0.1:9000
-00:47:05.782 [NamespaceSystem-akka.actor.default-dispatcher-5] INFO com.firebase.core.namespace.NamespaceActor - demo-backend-manager-default-rtdb successfully activated FBKV (SurveyIdle(0)) wait: 82ms, init: 0ms
-00:47:05.814 [NamespaceSystem-blocking-namespace-operation-dispatcher-7] INFO com.firebase.core.namespace.StateManager - Namespace demo-backend-manager-default-rtdb status Active to Active
-00:47:19.548 [Thread-0] INFO com.firebase.server.forge.App$ - Attempting graceful shutdown.
-00:47:19.556 [NamespaceSystem-akka.actor.default-dispatcher-5] INFO com.firebase.core.namespace.Terminator$Terminator - 1 actors left to terminate: demo-backend-manager-default-rtdb
-00:47:19.561 [NamespaceSystem-akka.actor.default-dispatcher-6] INFO com.firebase.core.namespace.NamespaceActor - stopped namespace actor for demo-backend-manager-default-rtdb
-00:47:19.565 [Thread-0] INFO com.firebase.server.forge.App$ - Graceful shutdown complete.
+01:48:07.812 [NamespaceSystem-akka.actor.default-dispatcher-4] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
+01:48:07.913 [main] INFO com.firebase.server.forge.App$ - Listening at 127.0.0.1:9000
+01:48:18.289 [NamespaceSystem-akka.actor.default-dispatcher-4] INFO com.firebase.core.namespace.NamespaceActor - demo-backend-manager-default-rtdb successfully activated FBKV (SurveyIdle(0)) wait: 73ms, init: 0ms
+01:48:18.324 [NamespaceSystem-blocking-namespace-operation-dispatcher-6] INFO com.firebase.core.namespace.StateManager - Namespace demo-backend-manager-default-rtdb status Active to Active
+01:48:29.496 [Thread-0] INFO com.firebase.server.forge.App$ - Attempting graceful shutdown.
+01:48:29.562 [NamespaceSystem-akka.actor.default-dispatcher-5] INFO com.firebase.core.namespace.Terminator$Terminator - 1 actors left to terminate: demo-backend-manager-default-rtdb
+01:48:29.581 [NamespaceSystem-akka.actor.default-dispatcher-7] INFO com.firebase.core.namespace.NamespaceActor - stopped namespace actor for demo-backend-manager-default-rtdb
+01:48:29.604 [Thread-0] INFO com.firebase.server.forge.App$ - Graceful shutdown complete.