backend-manager 5.10.2 → 5.10.3

package/CLAUDE.md

@@ -98,7 +98,7 @@ See [docs/cli-firestore-auth.md](docs/cli-firestore-auth.md) and [docs/cli-logs.
 - **🚫 NEVER run `npx mgr serve` / `npx mgr emulator`** (consumer projects) — they're the user's long-running dev processes. Assume they're already running; if they aren't, **instruct the user to run them** rather than running them yourself (running them again kills theirs). To see output, **read the `functions/*.log` files** (`dev.log`, `emulator.log`, `test.log`) — never tail/attach to the process. Running `npx mgr test` is fine (it auto-starts its own emulator if needed).
 - **Where the output logs live:** BEM CLI commands tee output to `<projectDir>/functions/` (not `logs/` — BEM's deliberate exception, co-located with firebase-tools' `*-debug.log`): `dev.log` (`npx mgr serve`), `deploy.log` (`npx mgr deploy`), `emulator.log` (`npx mgr emulator` / test with own emulator), `test.log` (`npx mgr test`), `production.log` (`npx mgr logs`). The `dev`/`test` names match EM/BXM/UJM; see [docs/logging.md](docs/logging.md).
 - **If the user reports an error**, check the emulator/test output for the root cause before guessing.
-- **Live-test UI changes via CDP.** When working on admin dashboards or browser-facing endpoints, use the `chrome-devtools` MCP tools (screenshots, click, evaluate JS, console logs) to verify the change works in the running browser. See `~/.claude/mcp-server/servers/chrome-devtools/CLAUDE.md`.
+- **Live-test UI changes via CDP.** When working on admin dashboards or browser-facing endpoints, use the `chrome-devtools` MCP tools (screenshots, click, evaluate JS, console logs) to verify the change works in the running browser — your session auto-launches its own private Chrome on the first tool call (no setup, no ports). See [docs/cdp-debugging.md](docs/cdp-debugging.md) + `~/.claude/mcp-server/servers/chrome-devtools/CLAUDE.md`.
 
 ## Supply-Chain Security
 
@@ -133,6 +133,8 @@ Whenever you make a behavioral change (new command, new flag, new pattern, remov
 
 Don't ship behavioral changes with stale docs. Validate first, then document — write docs that describe shipped reality, not intentions.
 
+**The OMEGA docs are structurally MIRRORED.** This file's section skeleton, the consumer template (`src/defaults/CLAUDE.md`), shared-concept `docs/*.md` filenames, and the `omega:*` skills are identical in structure and order across the sister frameworks (UJM / BEM / BXM / EM / MAM — WM mirrors the library subset). Never add, rename, or reorder a section here without making the SAME change in every sister repo in the same pass. The canonical skeletons + omission rules live in the `omega:main` skill's `mirror-spec.md` resource.
+
 ## Documentation
 
 Deep references live in `docs/`. **Whenever you make a behavioral change, update both this overview AND the relevant `docs/*.md` deep reference.**

package/docs/admin-post-route.md

@@ -11,10 +11,13 @@ The `POST /admin/post` route creates blog posts via GitHub's API. It handles ima
 1. Receives markdown body with external image URLs (e.g., `![alt](https://images.unsplash.com/...)`)
 2. Extracts all `![alt](url)` patterns from the body using regex
 3. Downloads each image to a tmp dir
-4. **Resizes** each image in place if its long edge exceeds `IMAGE_MAX_DIMENSION` (see below)
-5. Commits all images to `src/assets/images/blog/post-{id}/` on GitHub (single commit via Git Trees API)
-6. **Rewrites the body** to replace external URLs with `@post/{filename}` format
-7. The `@post/` prefix is resolved at Jekyll build time by `jekyll-uj-powertools` to the full path
+4. **Converts** png/webp sources to progressive JPEG in place (`convertToJpeg` — alpha flattened onto white); other non-JPG formats are rejected, naming the offending URL
+5. **Resizes** each image in place if its long edge exceeds `IMAGE_MAX_DIMENSION` (see below)
+6. Commits all images to `src/assets/images/blog/post-{id}/` on GitHub (single commit via Git Trees API)
+7. **Rewrites the body** to replace external URLs with `@post/{filename}` format
+8. The `@post/` prefix is resolved at Jekyll build time by `jekyll-uj-powertools` to the full path
+
+**Download failures:** the header image is fatal (the whole request 400s); body images are skipped with a warning (the body keeps the original external URL — the post still publishes). Failure reasons go through `formatImageDownloadError` — `Could not download image (<url>): <reason>` with HTML stripped from the reason (CDN 404 pages return raw HTML bodies) and long reasons truncated, so callers (e.g. sponsorship failure emails) surface something readable. Unsupported formats (anything that isn't jpg/png/webp) are rejected, naming the offending URL. Tests: `test/routes/admin/post-download-error.js` + `test/routes/admin/post-convert-image.js`.
 
 ## Image resize
 
@@ -22,7 +25,7 @@ Sources from guest-post submissions can be enormous (16384×10576 has been seen
 
 Two defenses:
 
-1. **CDN pre-scale** — `applyImageCDNParams(src)` adds server-side resize params to supported CDN URLs *before* downloading. Currently supports Unsplash (`images.unsplash.com`), which uses Imgix-style `?w=&q=` params. The CDN delivers a pre-scaled image (e.g. ~314KB instead of 3.8MB for a 2048px cap), so sharp never sees the massive original. Params are only added if not already present on the URL.
+1. **CDN pre-scale** — `applyImageCDNParams(src)` adds server-side resize params to supported CDN URLs *before* downloading. Currently supports Unsplash (`images.unsplash.com`, Imgix-style `?w=&q=` params) and Pexels (`images.pexels.com`, `?w=&auto=compress`). The CDN delivers a pre-scaled image (e.g. ~314KB instead of 3.8MB for a 2048px cap), so sharp never sees the massive original. Params are only added if not already present on the URL.
 
 2. **Local sharp resize** — after download, `resizeImage()` checks the long edge against `IMAGE_MAX_DIMENSION` and re-encodes as progressive JPEG at `IMAGE_JPEG_QUALITY` if it exceeds the limit. Images already within the limit pass through untouched. `sharp.cache(false)` is set so decoded pixel buffers are freed immediately between images — without this, processing several images serially can OOM even at 256MB.
 

package/docs/cdp-debugging.md

@@ -1,44 +1,29 @@
 # CDP Debugging (driving a live browser)
 
-How to launch a browser you can CONTROL — see a site live, screenshot it, click, type, read console logs, inspect network requests — for agents (Claude via MCP/CDP) and humans. BEM has no UI of its own; reach for this when **verifying the frontend that consumes your backend** (the UJM site, a deployed app): drive the auth flow, watch the network panel for calls to your routes, read the actual request/response payloads.
+How to drive a browser you can CONTROL — see the frontend live, screenshot it, click, type, read console logs, inspect network requests against your routes — for agents (Claude via MCP/CDP) and humans.
 
-> Mirrored across the four sister frameworks (UJM / BEM / BXM / EM) — same core section, framework-flavored. Edit all four together.
+> Mirrored across the five sister frameworks (UJM / BEM / BXM / EM / WM) — same core section, framework-flavored. Edit all five together.
 
-## Launching a controllable Chrome (the canonical command)
+## The browser: your Claude session owns one
 
-```bash
-open -gna "Google Chrome" --args \
-  --remote-debugging-port=9223 \
-  --user-data-dir="$HOME/Library/Application Support/chrome-profiles/agent" \
-  --no-first-run --no-default-browser-check \
-  --disable-background-timer-throttling \
-  --disable-backgrounding-occluded-windows \
-  --disable-renderer-backgrounding \
-  https://localhost:4000          # ← the frontend talking to your backend (or the IP from .temp/_config_browsersync.yml if localhost doesn't connect)
-```
+Browser work runs through the **`chrome-devtools` MCP** (via mcp-router). There is NO launch procedure anymore — no ports, no profile dirs, no curl checks:
 
-Verify it's up: `curl -s http://127.0.0.1:9223/json/version`
+- **Just call the tools** — `new_page`, `navigate_page`, `take_screenshot`, `click`, `fill`, `evaluate_script`, `list_console_messages`, `list_network_requests`. The browser auto-launches on the first call.
+- **Each Claude session gets its OWN private Chrome** (`--isolated`): temp profile, CDP over an internal pipe. Parallel sessions cannot see or touch each other's pages — open and close pages freely, the whole browser is yours.
+- **It dies with the session.** No orphans, no cleanup, nothing to kill.
+- **Ephemeral profile** — cookies/logins do NOT persist between sessions. If a flow needs auth, log in during the task.
+- **Self-signed HTTPS is pre-accepted** (`--acceptInsecureCerts` in the upstream) — dev servers load without certificate interstitials.
+- **NEVER quit/kill Chrome by app name** (`killall "Google Chrome"`, osascript) — that's the user's personal browser, not yours.
 
-The rules that make this work (each one learned the hard way):
+Humans: the agent's Chrome window is visible — you can watch it drive. Full reference: `~/.claude/mcp-server/servers/chrome-devtools/CLAUDE.md`.
 
-- **`open -gna` launches WITHOUT stealing focus.** `-g` = don't bring to foreground, `-n` = new instance (required — without it `open` just activates the already-running daily Chrome and the `--args` are ignored). Launching the Chrome binary directly ALWAYS activates the app and steals focus. Do NOT use `-j`/`--hide` — animations need a visible window; instead the three `--disable-*` flags keep timers/rAF/rendering at FULL speed while the window sits behind your work (verified: rAF at the display's native 120fps while backgrounded, focus never moved).
-- **`--user-data-dir` is REQUIRED, not optional.** Chrome 136+ **silently ignores** `--remote-debugging-port` on the default profile — no error, no port, nothing (verified on Chrome 149). This is the #1 "why isn't CDP up" trap.
-- **The profile dir IS the persistent login state.** Cookies + localStorage survive relaunches (verified by round-trip). **Log into sites once in the agent profile and every agent reuses the authenticated state** — auth'd flows against your routes work without re-login. Ecosystem convention: ONE shared profile at `~/Library/Application Support/chrome-profiles/agent` across all four frameworks, so logins are a one-time setup.
-- **One Chrome instance per profile dir — but MANY agents per instance.** CDP is multi-client (verified: two concurrent clients driving different tabs of one instance): agents and sessions attach to the SAME port, each drives its own tab, and all share the profile's logins. One agent per tab is the only rule. A second launch with the same dir just opens a window in the existing instance and **ignores the new debug port** — attach to the running one instead. Reach for a second profile + port (`…/b` on 9224) only for a different IDENTITY (a different account = a different cookie jar) or hard isolation.
-- It runs **side-by-side with the daily Chrome** — a different `--user-data-dir` is a fully separate instance.
-- **Quit by profile match, never by app name**: `pkill -f "chrome-profiles/agent"`. (`osascript 'tell app "Google Chrome" to quit'` hits the daily browser too — same app name.)
+## Electron apps are the exception (attach, don't launch)
 
-## Driving it
-
-| Client | Good for | Port handoff |
-|---|---|---|
-| `chrome-devtools` MCP | rich interaction — click, fill, type, screenshots, network requests, console messages, performance traces | `CHROME_CDP_PORT` env var, **expanded ONCE when the Claude session spawns its MCP — set it BEFORE launching `claude`** (mid-session changes do nothing) |
-| Any CDP client — including EM's `npx mgr cdp` run from any EM project | quick JS eval, per-renderer screenshots | per invocation: `EM_CDP_PORT=9223 npx mgr cdp eval ":4000" 'document.title'` |
-
-Port conventions: **9222** = Electron apps (EM), **9223+** = Chrome instances.
+An Electron dev app is a running singleton — you ATTACH to it instead of launching a browser: the `chrome-devtools-electron` MCP upstream (reads `EM_CDP_PORT`, default 9222, expanded once at session start) or EM's per-invocation `npx mgr cdp`. See EM's `docs/cdp-debugging.md`.
 
 ## BEM specifics
 
-- **The UJM dev site is HTTPS.** BrowserSync serves over HTTPS (self-signed cert). Prefer `https://localhost:4000`; fall back to the machine's local network IP (e.g. `https://192.168.x.x:4000`) if localhost doesn't connect. Port 4000 by default, increments to 4001+ when multiple sites run. The exact URL is in `.temp/_config_browsersync.yml` at the root of the WEBSITE project (the UJM consumer — e.g. `<brand>-website/.temp/_config_browsersync.yml`, NOT this backend repo) — read that file first, every time, before navigating.
+- **The UJM dev site URL is `https://localhost:4000` — NEVER the LAN IP** (`https://192.168.x.x:...`). Port 4000 by default, increments (4001, …) when multiple sites run; the exact port is in `.temp/_config_browsersync.yml` at the root of the WEBSITE project (the UJM consumer — e.g. `<brand>-website/.temp/_config_browsersync.yml`, NOT this backend repo).
 - The network tab is the payoff: `list_network_requests` shows every call the frontend makes to your routes — method, status, and payloads — while you click through the real UI.
+- **Auth'd flows**: the profile is ephemeral, so log in through the real UI at the start of the session (test creds) — then exercise the authenticated routes.
 - Backend-side observation stays where it always was: `npx mgr logs` (gcloud logs) and the emulator suite; this doc only covers the browser half of the loop.

package/docs/ghostii.md

@@ -14,13 +14,13 @@ config.blog.content[] entry
   |
 blog-auto-publisher cron (daily)
   | (per entry, per article)
-resolveSource() -- detect source type, fetch/parse if needed
+resolveSources({ count: 1 }) -- unified resolver: random pick + type fallback + Firestore dedup
   |
 provider.writeArticle() -- call provider API (e.g. Ghostii: api.ghostii.ai/write/article)
   |
-provider.publishArticle() -- POST to admin/post -> GitHub commit
+provider.publishArticle() -- POST to admin/post -> GitHub commit (passes source URL)
   |
-trackContentSource() -- Firestore (for $feed:, $parent sources)
+trackContentSource() -- Firestore, AFTER successful publish
 ```
 
 ### Key files
@@ -90,21 +90,28 @@ content-sources/{sha256(origin + '::' + url).slice(0,20)}: {
 
 ### Topic deduplication
 
-The blog auto-publisher prevents near-duplicate articles at three levels:
+The blog auto-publisher prevents near-duplicate articles at two levels:
 
-1. **Per-feed dedup**: `processFeedSource()` queries `content-sources` by `feedUrl` to skip items already used from that specific feed.
+1. **Per-item dedup** (structural): the resolver queries `content-sources` before picking — feed items by `feedUrl`, parent sources by `origin == '$parent'`. Used items are never re-picked. Within one `resolveSources()` call, a session-used set also prevents the same item from resolving twice.
 
-2. **Cross-feed dedup** (structural): Before picking a feed item, `processFeedSource()` compares each candidate's title against ALL recent titles (from any source) using `isTitleTooSimilar()` — a word-overlap check with basic stemming. This catches the same news event reported by different publications (e.g. Guardian and NYT both covering Apple's price hike). Items with >= 50% significant-word overlap are skipped.
+2. **Topic dedup** (prompt-based): Before generating each article, `harvest()` appends all recent titles (Firestore history + same-run) to the Ghostii prompt as a strict avoidance list. The prompt forbids writing about the same topic, theme, or keyword combination — not just the same story.
 
-3. **Topic dedup** (prompt-based): Before generating each article, `harvest()` appends all recent titles (Firestore history + same-run) to the Ghostii prompt as a strict avoidance list. The prompt forbids writing about the same topic, theme, or keyword combination — not just the same story.
+Within a single cron run, both generated post titles AND source item titles are tracked in `runTitles` and fed into the prompt for subsequent articles.
 
-Within a single cron run, both generated post titles AND source item titles are tracked in `runTitles` and fed into levels 2 and 3 for subsequent articles.
+Sources are marked used in `content-sources` ONLY after the article actually publishes ($brand posts get a synthetic tracking entry so recent-title dedup still sees them).
 
-All source types (`$feed`, `$parent`, `$brand`, URL, text) are tracked in `content-sources` after publishing.
+### Source picking + fallback hierarchy
 
-### Source fallback behavior
+Both blog and newsletter resolve through the same `resolveSources()` (source-resolver.js). Each needed source starts as a RANDOM pick from the entry's `sources` array — nothing outside the array is ever used. On failure, a strict type hierarchy applies:
 
-When a source fails or is exhausted, `resolveSource()` only falls back to `$brand` if `$brand` is explicitly listed in the entry's `sources` array. Otherwise it returns null, and `harvest()` tries the next source in a shuffled copy of the pool. If all sources are exhausted, that article slot is skipped.
+| Picked type | Fallback chain |
+|---|---|
+| `$feed:` | other items in the same feed → other `$feed` sources (random order) → `$parent` (only if listed) → give up |
+| `$parent` | other unused parent-pool items → give up (never falls to feeds) |
+| `$brand` | none — resolves directly; **nothing ever falls back TO `$brand`** |
+| URL / text | none — pick-only |
+
+If a pick's chain is exhausted, that slot is unfilled: the blog skips that article, the newsletter proceeds with fewer sources (or skips entirely at zero).
 
 ## Configuration
 

package/docs/marketing-campaigns.md

@@ -156,11 +156,11 @@ Per-brand subusers provide full contact/segment/field isolation under one billin
 `generators/newsletter.js` orchestrates a multi-step pipeline that produces a fully rendered, email-safe newsletter. Output is HTML (not markdown) — the marketing library detects `settings.contentHtml` and uses it directly, skipping the markdown pipeline.
 
 Pipeline:
-1. Fetch sources: `GET {parentUrl}/newsletter/sources?category=X&claimFor=brandId` (atomic claim)
+1. Resolve sources: `resolveSources({ sources, count: sourceCount || 6, categories })` — the unified blog/newsletter resolver (source-resolver.js). Each pick is a RANDOM source from the entry's `sources` array; failures follow the type hierarchy ($feed → other feeds → $parent; $parent → other parent items; nothing falls to $brand). Firestore-checked so used items never repeat, session-checked so one issue never gets duplicates. See [docs/ghostii.md](ghostii.md#source-picking--fallback-hierarchy) for the full hierarchy.
 2. **structure.js** — Generic dispatcher. Resolves the active template, merges `BASE_SCHEMA` (universal fields: subject, preheader, signoff, citations) with the template's own `schema` fragment, calls the template's `buildPrompt({brand, newsletterConfig, sources})` to get the AI brief, runs the AI call, and normalizes the result via the template's optional `normalize()`. Default provider: `openai` (override per-run only via `NEWSLETTER_PROVIDER_STRUCTURE` env).
 3. **image-illustrator.js** (default) — One flat-vector PNG per section in parallel (`Promise.all`), generated directly via `Manager.AI(assistant).image()` → `gpt-image-2`. Iterates `structure.sections` — templates whose content shape isn't section-based (e.g. field-report uses `dispatches`) populate `sections` in their `normalize()` step so this loop keeps working unchanged. The prompt enforces a clean flat 2D vector style (Stripe / Linear / undraw.co aesthetic) built from the brand palette (`content.theme.{primary,secondary,accent}Color`), on a white background, no text. **Legacy method:** set `marketing.newsletter.content.method.image = 'svg'` to use the older `svg-illustrator.js` (AI authors an `<svg>`, rasterized via `@resvg/resvg-js`). Both methods return the same `{ png: Buffer, fallback, meta }` contract.
 4. **mjml-template.js** — Resolves the template by name from `templates/index.js`, calls `template.build({structure, imagePaths, theme, ...})` for the MJML, compiles to email-safe HTML via the `mjml` package. Brand-domain links get UTM-tagged via the existing `tagLinks()` utility.
-5. Mark used: `PUT {parentUrl}/newsletter/sources` per source
+5. Mark used: `trackContentSource()` per source into the local `content-sources` collection — ONLY after generation succeeds. No PUT to the parent; the child tracks its own usage.
 
 > **Step 3 runs concurrently with the linked-article build** (see below) when `article.enabled` is on — both are slow AI calls, so they share one `Promise.all`. The article URL is stamped onto the lead section before steps 4/5 render.
 
@@ -283,12 +283,16 @@ Templates add their own fields on top (e.g. classic adds `intro` + `sections`; f
 
 **No AI-authored CTAs in generated content.** The schema intentionally does NOT include section-level CTAs / outbound links. The AI cannot author URLs reliably — it has no browse access to your site and no real source URLs to reference, so any URL it produces is invented. Newsletters are self-contained reads; outbound links come from sponsorship blocks rendered by the template shell (driven by `marketing.newsletter.content.sponsorships[]`), not from generated section bodies. The **one exception** is the linked-article CTA: when `article.enabled` is on, `section.cta = { label, url }` is injected onto the lead section by code *after* the article is published (a real, verified URL) — never authored by the AI. Both `sectionCard` (MJML) and the markdown renderer render `section.cta` when present.
 
-**Beehiiv failure → fallback alert email.** When Beehiiv draft creation fails (e.g. `SEND_API_NOT_ENTERPRISE_PLAN` on the free plan), the generator sends an internal alert email via `sender: 'internal'` (resolves to `alerts@{brandDomain}`) to `brand.contact.email` with:
-- The failure reason
+**Newsletter report email — always sent.** After every generation, the generator sends an internal report email via `sender: 'internal'` (resolves to `alerts@{brandDomain}`) with:
+- Beehiiv status (uploaded, or the failure reason when it failed — subject line adapts)
+- A prominent "Preview Newsletter" button (GitHub Pages viewer)
 - Subject, preheader, tags
-- Direct links to the rendered HTML, per-section markdown, summary, and the full GitHub folder
+- Direct links to the rendered HTML, per-section markdown, and summary
+- Linked articles (published ones only — unpublished articles are never referenced)
+- All sources used (linked when they have real URLs, feed hostname in parentheses)
+- The full GitHub folder
 
-This means the newsletter is never "stuck" — even with Beehiiv disabled or failing, you get an actionable email pointing to ready-to-paste assets. The alert is best-effort; failure to send is logged but does not block the Firestore campaign-doc write.
+This means the newsletter is never "stuck" — even with Beehiiv disabled or failing, you get an actionable email pointing to ready-to-paste assets. The report is best-effort; failure to send is logged but does not block the Firestore campaign-doc write.
 
 Requires `GH_TOKEN` env var (org-scoped, write access to `newsletter-assets`). Without it, the cron's HTML/image upload calls throw and the run aborts.
 
@@ -307,7 +311,7 @@ Requires `GH_TOKEN` env var (org-scoped, write access to `newsletter-assets`). W
 | `lib/templates/shared-campaign.js` | Shared utilities for all templates (`escape()`, `resolveTheme()`, `formatAddress()`) |
 | `lib/templates/editorial/helpers.js` | Editorial-only helpers (pullquote, issue number, eyebrow) |
 | `lib/templates/field-report/helpers.js` | Field-report-only helpers (kicker, dispatch dateline, terminal block, terminator) |
-| `newsletter.js` | Orchestrator — calls lib modules, fetches sources, claims sources |
+| `newsletter.js` | Orchestrator — resolves sources via the unified source-resolver, calls lib modules, tracks usage locally |
 | `test/marketing/fixtures/{name}.json` | Hand-crafted structure per template (loaded by iteration test in fixture mode) |
 
 ## Seed Campaigns

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backend-manager",
-  "version": "5.10.2",
+  "version": "5.10.3",
   "description": "Quick tools for developing Firebase functions",
   "main": "src/manager/index.js",
   "files": [

package/src/cli/index.js

@@ -1,6 +1,12 @@
 const os = require('os');
 const path = require('path');
-const argv = require('yargs')(process.argv.slice(2)).argv;
+// Universal boolean flags must be declared — otherwise yargs treats the next
+// positional as the flag's VALUE (`mgr test --extended project:foo` became
+// extended='project:foo' with NO targets, silently running EVERYTHING in
+// extended mode against real external APIs).
+const argv = require('yargs')(process.argv.slice(2))
+  .boolean(['extended', 'legacy', 'force', 'raw', 'emulator'])
+  .argv;
 const _ = require('lodash');
 
 // Abort if running from ~/node_modules (accidental home directory install)

package/src/defaults/CLAUDE.md

@@ -1,6 +1,8 @@
 # ========== Default Values ==========
 # Backend Manager (BEM) — consumer project
 
+<!-- MAINTAINERS (framework repo): this consumer template is MIRRORED across UJM/BEM/BXM/EM/MAM — same sections, same order (framework-specific extras may be inserted; canonical sections are never reordered/renamed). Edit all five together. Canonical skeleton: omega:main skill → resources/mirror-spec.md -->
+
 ## Framework
 
 This project consumes **Backend Manager** (BEM) — a comprehensive framework for building modern Firebase Cloud Functions backends. BEM provides a single `Manager.init(exports, {...})` bootstrap that wires built-in functions (`bm_api`, auth events, cron jobs), helper classes (Assistant, User, Analytics, Usage, Middleware, Settings, Utilities, Metadata), payment processor integrations (Stripe / PayPal), Firestore-trigger pipelines, and a deploy/emulator/watch tooling pipeline.

package/src/manager/events/cron/daily/blog-auto-publisher.js

@@ -13,13 +13,11 @@
  *   'https://...'       — fetch URL content as prompt seed
  *   '<text>'            — use directly as prompt seed
  *
- * All feed/parent sources are tracked in Firestore (`content-sources`) so the
- * same article is never processed twice. When a feed is unreachable or
- * exhausted, harvest() tries the next source in the pool. Only falls
- * back to $brand if '$brand' is explicitly listed in the entry's sources.
- *
- * Source resolution, tracking, and prompt constants live in the shared
- * source-resolver library (src/manager/libraries/content/source-resolver.js).
+ * Source picking, fallback hierarchy, Firestore dedup, and tracking live in
+ * the shared source-resolver library — the SAME resolution the newsletter
+ * generator uses. Per article, ONE random source is picked from the pool;
+ * failures follow the type hierarchy ($feed → other feeds → $parent; $parent
+ * → other parent items; NOTHING falls back to $brand).
  */
 const powertools = require('node-powertools');
 const moment = require('moment');
@@ -27,15 +25,9 @@ const moment = require('moment');
 const {
   PROMPT_SOURCE,
   PROMPT,
-  FEED_PREFIX,
-  processFeedSource,
-  processParentSource,
+  resolveSources,
   trackContentSource,
-  contentSourceHash,
-  getProcessedItemIds,
   getRecentTitles,
-  getURLContent,
-  isURL,
 } = require('../../../libraries/content/source-resolver.js');
 
 let postId;
@@ -62,7 +54,7 @@ module.exports = async ({ Manager, assistant, context, libraries }) => {
 
   for (const entry of contentArray) {
     entry.quantity = entry.quantity || 0;
-    entry.sources = randomize(entry.sources || []);
+    entry.sources = entry.sources || [];
     entry.links = randomize(entry.links || []);
     entry.instructions = entry.instructions || '';
     entry.tone = entry.tone || 'professional';
@@ -135,25 +127,28 @@ async function harvest(assistant, entry, admin, provider, Manager) {
   }
 
   for (let index = 0; index < entry.quantity; index++) {
-    const allKnownTitles = [...recentTitles, ...runTitles];
+    assistant.log(`harvest(): Processing ${index + 1}/${entry.quantity}`);
 
-    let resolved = null;
-    const shuffled = randomize([...entry.sources]);
-    for (const source of shuffled) {
-      assistant.log(`harvest(): Processing ${index + 1}/${entry.quantity}`, source);
+    const allKnownTitles = [...recentTitles, ...runTitles];
 
-      resolved = await resolveSource(assistant, source, entry, admin, Manager).catch((e) => {
-        assistant.error('harvest(): Error resolving source', e);
-        return null;
-      });
-      if (resolved) { break; }
-    }
+    // ONE random source per article — fallback hierarchy handled by the resolver.
+    // Tracking between iterations means the next resolve sees this run's usage.
+    const [source] = await resolveSources({
+      sources: entry.sources,
+      count: 1,
+      categories: entry.categories,
+      admin,
+      Manager,
+      assistant,
+    });
 
-    if (!resolved) {
-      assistant.log('harvest(): All sources exhausted for this article, skipping');
+    if (!source) {
+      assistant.log('harvest(): No source could be resolved for this article, skipping');
       continue;
     }
 
+    const resolved = buildPromptFromSource(source, entry);
+
     if (allKnownTitles.length) {
       resolved.description += '\n\nTOPIC DEDUPLICATION (STRICT):\n'
         + 'These articles were RECENTLY published on our blog. You MUST NOT write about the same topic, theme, or subject area as ANY of them.\n'
@@ -165,7 +160,7 @@ async function harvest(assistant, entry, admin, provider, Manager) {
         + allKnownTitles.slice(0, 25).map((t) => `- ${t}`).join('\n');
     }
 
-    assistant.log('harvest(): Resolved source', resolved);
+    assistant.log('harvest(): Resolved source', { type: source.type, title: source.title, url: source.url });
 
     const overrides = { ...entry.overrides };
     if (!overrides.keywords && entry.keywords.length) {
@@ -195,7 +190,7 @@ async function harvest(assistant, entry, admin, provider, Manager) {
       id: postId++,
       author: entry.author,
       postPath: entry.postPath,
-      source: resolved.trackingData?.url || null,
+      source: source.trackingData?.url || null,
     };
     assistant.log('harvest(): publishArgs', publishArgs);
 
@@ -210,21 +205,22 @@ async function harvest(assistant, entry, admin, provider, Manager) {
     if (generatedTitle) {
       runTitles.push(generatedTitle);
     }
-    if (resolved.trackingData?.itemTitle) {
-      runTitles.push(resolved.trackingData.itemTitle);
+    if (source.trackingData?.itemTitle) {
+      runTitles.push(source.trackingData.itemTitle);
     }
 
-    if (!resolved.trackingData) {
-      resolved.trackingData = {
-        url: uploadedPost.slug || `brand-${postId}`,
-        origin: '$brand',
-        usedBy: 'blog',
-      };
-    }
+    // Mark the source used ONLY now that the article actually published.
+    // $brand/url/text picks have no trackingData — synthesize one for $brand
+    // so recent-title dedup still sees the post.
+    const trackingData = source.trackingData || {
+      url: uploadedPost.slug || `brand-${postId}`,
+      origin: '$brand',
+    };
 
     if (admin) {
       await trackContentSource(admin, {
-        ...resolved.trackingData,
+        ...trackingData,
+        usedBy: 'blog',
         brandId: entry.brand.brand.id,
         postUrl: uploadedPost.url,
         postSlug: uploadedPost.slug,
@@ -236,133 +232,48 @@ async function harvest(assistant, entry, admin, provider, Manager) {
   }
 }
 
-async function resolveSource(assistant, source, entry, admin, Manager) {
-  const date = moment().format('MMMM YYYY');
-
+/**
+ * Map a resolved source to the provider prompt shape.
+ * feed/parent sources rewrite an existing article (PROMPT_SOURCE);
+ * brand/url/text sources seed a fresh topic (PROMPT).
+ *
+ * @param {object} source - resolved source from resolveSources()
+ * @param {object} entry - the blog content entry
+ * @returns {{ description: string, sourceContent: string }}
+ */
+function buildPromptFromSource(source, entry) {
   const templateVars = {
     ...entry,
     instructions: entry.instructions,
-    date: date,
+    date: moment().format('MMMM YYYY'),
     tone: entry.tone || '',
     categories: (entry.categories || []).join(', '),
     keywords: (entry.keywords || []).join(', '),
   };
 
-  // --- $feed: source ---
-  if (typeof source === 'string' && source.startsWith(FEED_PREFIX)) {
-    const feedUrl = source.slice(FEED_PREFIX.length);
-    assistant.log(`resolveSource(): Processing feed: ${feedUrl}`);
-
-    const feedResult = await processFeedSource(assistant, feedUrl, entry.brand.brand.id, admin).catch((e) => e);
-
-    if (feedResult instanceof Error || !feedResult) {
-      assistant.log('resolveSource(): Feed failed or exhausted');
-      if (entry.sources.includes('$brand')) {
-        return resolveSource(assistant, '$brand', entry, admin, Manager);
-      }
-      return null;
-    }
-
-    const description = powertools.template(PROMPT_SOURCE, {
-      ...templateVars,
-      sourceTitle: feedResult.item.title,
-    });
-
-    return {
-      description,
-      sourceContent: feedResult.content || feedResult.item.summary || '',
-      trackingData: {
-        url: feedResult.item.url || feedResult.item.id,
-        origin: source,
-        feedUrl: feedUrl,
-        itemId: feedResult.item.id,
-        itemTitle: feedResult.item.title,
-        usedBy: 'blog',
-      },
-    };
-  }
-
-  // --- $parent source ---
-  if (source === '$parent') {
-    assistant.log('resolveSource(): Processing $parent source');
-
-    const parentResults = await processParentSource(assistant, entry, admin, Manager).catch((e) => e);
-
-    if (parentResults instanceof Error || !parentResults?.length) {
-      assistant.log('resolveSource(): Parent source failed or exhausted');
-      if (entry.sources.includes('$brand')) {
-        return resolveSource(assistant, '$brand', entry, admin, Manager);
-      }
-      return null;
-    }
-
-    const parentResult = parentResults[0];
-    const description = powertools.template(PROMPT_SOURCE, {
-      ...templateVars,
-      sourceTitle: parentResult.title,
-    });
-
+  if (source.type === 'feed' || source.type === 'parent') {
     return {
-      description,
-      sourceContent: parentResult.content || parentResult.summary || '',
-      trackingData: {
-        url: parentResult.url || parentResult.id,
-        origin: '$parent',
-        itemId: parentResult.id,
-        itemTitle: parentResult.title,
-        usedBy: 'blog',
-      },
+      description: powertools.template(PROMPT_SOURCE, {
+        ...templateVars,
+        sourceTitle: source.title,
+      }),
+      sourceContent: source.content || '',
     };
   }
 
-  // --- $brand source ---
-  if (source === '$brand') {
-    const suggestion = 'Write an article about any topic that would be relevant to our website and business (it does not have to be about our company, but it can be)';
-    const description = powertools.template(PROMPT, {
-      ...templateVars,
-      suggestion: suggestion,
-    });
+  const suggestion = source.type === 'brand'
+    ? 'Write an article about any topic that would be relevant to our website and business (it does not have to be about our company, but it can be)'
+    : source.content;
 
-    return { description, sourceContent: '' };
-  }
-
-  // --- URL source ---
-  if (isURL(source)) {
-    const suggestion = await getURLContent(source).catch((e) => e);
-
-    if (suggestion instanceof Error) {
-      assistant.error(`resolveSource(): Error fetching URL ${source}`, suggestion);
-      if (entry.sources.includes('$brand')) {
-        return resolveSource(assistant, '$brand', entry, admin, Manager);
-      }
-      return null;
-    }
-
-    const description = powertools.template(PROMPT, {
+  return {
+    description: powertools.template(PROMPT, {
       ...templateVars,
       suggestion: suggestion,
-    });
-
-    return { description, sourceContent: '' };
-  }
-
-  // --- Text source ---
-  const description = powertools.template(PROMPT, {
-    ...templateVars,
-    suggestion: source,
-  });
-
-  return { description, sourceContent: '' };
+    }),
+    sourceContent: '',
+  };
 }
 
 function randomize(array) {
   return array.sort(() => Math.random() - 0.5);
 }
-
-// Re-export shared functions for backwards compatibility (newsletter + tests import from here)
-module.exports.resolveSource = resolveSource;
-module.exports.processFeedSource = processFeedSource;
-module.exports.contentSourceHash = contentSourceHash;
-module.exports.getProcessedItemIds = getProcessedItemIds;
-module.exports.trackContentSource = trackContentSource;
-module.exports.isURL = isURL;