backend-manager 5.7.6 → 5.8.1

package/CHANGELOG.md

@@ -14,6 +14,27 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+# [5.8.1] - 2026-06-19
+
+### Fixed
+- **Ghostii 502 timeout fix.** Switched `writeArticle()` from the Firebase Hosting URL (`api.ghostii.ai`) to the raw Cloud Functions URL. Firebase Hosting rewrites have a hard 60-second proxy timeout that caused 502 errors when the AI article pipeline exceeded that limit. The raw URL uses the function's own 5-minute timeout.
+
+### Added
+- **Ghostii call duration logging.** `writeArticle()` now logs the round-trip time of each Ghostii API call (`[ghostii] writeArticle() completed in Xms`).
+
+# [5.8.0] - 2026-06-19
+
+### Added
+- **RSS/Atom feed sources for Ghostii.** New `$feed:<url>` source type in `config.ghostii[]` entries. Parses RSS 2.0, Atom 1.0, and JSON Feed formats, selects unprocessed articles, extracts content, and passes it to the Ghostii API as `sourceContent` for AI-assisted original article generation. Feed items tracked in Firestore (`ghostii-feed-items`) to prevent re-processing. Falls back to `$app` when feeds are unreachable or exhausted.
+- **Per-entry Ghostii API overrides.** `config.ghostii[].overrides` object lets each entry customize keywords, length, research, insertImages, headerImageUrl, maxLinks, sectionQuantity, and feedUrl — previously hardcoded in `writeArticle()`.
+- **Configurable postPath.** `config.ghostii[].postPath` controls the `_posts/{year}/` sub-folder for published articles (default: `'ghostii'`).
+- **Feed parser library.** New `src/manager/libraries/content/feed-parser.js` — `parseFeed()` and `extractArticleContent()` via Cheerio for parsing syndication feeds and extracting readable article text from URLs.
+- **New dependencies:** `fast-xml-parser` for RSS/Atom parsing, `cheerio` for article content extraction.
+
+### Changed
+- **`writeArticle()` accepts overrides and sourceContent.** Signature changed from `({ brand, description, links })` to `({ brand, description, links, sourceContent, overrides })`. Existing callers (newsletter generator) pass no new args and get identical behavior.
+- **Newsletter fact paraphrasing.** Newsletter writer prompt now instructs the AI to paraphrase all facts, figures, and statistics from sources — never copy exact numbers or phrasing.
+
 # [5.7.6] - 2026-06-18
 
 ### Fixed

package/CLAUDE.md

@@ -162,6 +162,7 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 
 ### Subsystems & Libraries
 
+- [docs/ghostii.md](docs/ghostii.md) — Ghostii article publisher: source types (`$app` / `$feed:` / URL / text), per-entry API overrides, RSS/Atom feed parser, Firestore feed-item tracking, `sourceContent` pass-through to Ghostii API
 - [docs/email-system.md](docs/email-system.md) — unified MJML email rendering pipeline: shared preparation (`prepare.js`), composable template system (`base.js` blocks), 4 email templates (card, plain, order, feedback), no SendGrid dynamic templates — everything rendered server-side
 - [docs/usage-rate-limiting.md](docs/usage-rate-limiting.md) — usage tracking, monthly/daily caps, `setUser()` + mirrors for proxy usage, reset schedule
 - [docs/ai-library.md](docs/ai-library.md) — `Manager.AI()` unified entry for OpenAI + Anthropic (text via `.request()`, images via `.image()` → `gpt-image-2`)

package/PROGRESS.md

@@ -2,13 +2,37 @@
 > Agents and maintainers should update this file regularly to reflect the current state of the project.
 
 ## 🎯 Current Focus
-* **Goal:** Root package.json proxy for running scripts from project root
-* **Current Phase:** Implementation complete, untested in consumer
-* **Priority:** Medium
-* **Last Updated:** 2026-06-17 7:40 PM PDT
-* **Notes:** v5.7.2 shipped (npm + GitHub release). New setup test `root-package-json` generates a root `package.json` with proxy scripts so `npm test`/`npm start`/etc. work from the Firebase project root (not just `functions/`). Includes `preinstall` guard to block accidental `npm install` at root.
+* **Goal:** RSS/Atom feed-based article generation for the Ghostii system
+* **Current Phase:** Shipped
+* **Priority:** Complete
+* **Last Updated:** 2026-06-18 7:30 PM PDT
+* **Notes:** BEM v5.8.0 published to npm. Ghostii-backend v1.0.5 deployed to Firebase. All code, tests, and docs shipped.
 
 ## 📌 Active Task List
+* [ ] Phase 5: Ghostii feed-based article system
+  * [x] Task 5.1: Create `feed-parser.js` (RSS 2.0, Atom 1.0, JSON Feed parser + article extractor)
+  * [x] Task 5.2: Add `fast-xml-parser` dependency to BEM
+  * [x] Task 5.3: Add `sourceContent` field to Ghostii backend schema (16KB)
+  * [x] Task 5.4: Update Ghostii outline prompt (Step 2 only) with sourceContent for efficient spinning
+  * [x] Task 5.5: Update `writeArticle()` to accept `sourceContent` and per-entry `overrides`
+  * [x] Task 5.6: Upgrade `ghostii-auto-publisher.js` — `$feed:` source type, feed processing, Firestore tracking, fallback
+  * [x] Task 5.7: Update config template with new source types and overrides
+  * [x] Task 5.8: Write extensive test suite (4 test files: unit, integration, extended E2E)
+  * [x] Task 5.9: Verify all tests pass (84 standard + 8 extended against real RSS feeds)
+  * [x] Task 5.10: Create `docs/ghostii.md` deep reference, update CLAUDE.md + CHANGELOG.md
+  * [x] Task 5.11: Verify all recommended feed URLs work (The Verge removed — CDN blocks programmatic access)
+  * [x] Task 5.12: Write ghostii-backend tests for sourceContent (4 schema + 1 extended generation)
+  * [x] Task 5.13: Ghostii sourceContent accepts URL (auto-fetches + extracts article text)
+  * [x] Task 5.14: Add fact paraphrasing to outline prompt + newsletter writer
+  * [x] Task 5.15: Fix Step 3 prompts — assertive link insertion + blockquote in every body section
+  * [x] Task 5.16: Fix humanizer stripping links — `injectBurstiness` was breaking `[text](url)` syntax; added protect-and-restore pattern + tests
+  * [x] Task 5.17: Fix 403 links treated as "working" in link verification step
+  * [x] Task 5.18: Add detective-level `[LINKS]` diagnostic logging to article pipeline
+  * [x] Task 5.19: Ship BEM v5.8.0 to npm
+  * [x] Task 5.20: Ship Ghostii-backend v1.0.5 + deploy to Firebase
+  * [ ] Task 5.17: Publish BEM with feed support
+  * [ ] Task 5.12: Publish BEM with feed support
+  * [ ] Task 5.13: Configure consumer project(s) with `$feed:` sources
 * [ ] Phase 3: Post-audit bug fixes
   * [x] Newsletter ReferenceError: `beehiivConfig` → `newsletterRoleConfig` (committed v5.7.1)
   * [x] HTTPS proxy: `serve.js` returns boolean, caller uses `httpsReady` not `httpsEnabled`

package/docs/ghostii.md

@@ -0,0 +1,127 @@
+# Ghostii Article Publisher
+
+The Ghostii system generates and publishes blog posts via the daily `bm_cronDaily` cron job. Two independent paths:
+
+1. **Standalone publisher** — `config.ghostii[]` entries: daily cron picks sources, calls the Ghostii AI API, publishes via `admin/post`.
+2. **Newsletter-linked articles** — `marketing.newsletter.content.article.enabled`: the newsletter generator expands its lead section into a full blog post via Ghostii.
+
+This doc covers the standalone publisher. For newsletter integration, see [marketing-campaigns.md](marketing-campaigns.md).
+
+## Architecture
+
+```
+config.ghostii[] entry
+  ↓
+ghostii-auto-publisher cron (daily)
+  ↓ (per entry, per article)
+resolveSource() — detect source type, fetch/parse if needed
+  ↓
+writeArticle() — call Ghostii AI API (api.ghostii.ai/write/article)
+  ↓
+publishArticle() — POST to admin/post → GitHub commit
+  ↓
+trackFeedItem() — Firestore (for $feed: sources only)
+```
+
+### Key files
+
+| File | Purpose |
+|---|---|
+| `src/manager/libraries/content/ghostii.js` | `writeArticle()`, `publishArticle()`, `blocksToPost()` — API client + post transform |
+| `src/manager/libraries/content/feed-parser.js` | `parseFeed()`, `extractArticleContent()` — RSS/Atom/JSON parser + article extractor |
+| `src/manager/events/cron/daily/ghostii-auto-publisher.js` | Daily cron: source resolution, feed processing, Firestore tracking |
+| `src/manager/routes/admin/post/post.js` | Publishing endpoint: image download + resize, GitHub commit |
+
+## Source Types
+
+Each `ghostii[]` entry has a `sources` array. The cron picks one at random per article.
+
+| Source | Format | Behavior |
+|---|---|---|
+| Generic topic | `'$app'` | Prompts Ghostii to write about any topic relevant to the brand |
+| RSS/Atom feed | `'$feed:https://example.com/feed/'` | Parses feed, picks one unprocessed article, extracts content, passes as `sourceContent` to Ghostii |
+| URL | `'https://example.com/page'` | Fetches page HTML, extracts body text, uses as prompt seed |
+| Text | `'Write about blockchain'` | Uses directly as prompt seed |
+
+### Feed source flow
+
+1. Fetch and parse the RSS/Atom/JSON feed via `feed-parser.parseFeed()`
+2. Query `ghostii-feed-items` in Firestore for already-processed items
+3. Filter to unprocessed items, pick the newest
+4. Extract full article content from the item URL via `feed-parser.extractArticleContent()`
+5. Pass extracted text as `sourceContent` to the Ghostii API (separate from the `description` prompt)
+6. After publish, write a tracking doc to `ghostii-feed-items/{hash}`
+7. On failure (feed unreachable, unparseable, exhausted): fall back to `$app` behavior
+
+### Feed item tracking
+
+Collection: `ghostii-feed-items` (consumer project Firestore)
+
+```js
+ghostii-feed-items/{sha256(feedUrl + '::' + itemId).slice(0,20)}: {
+  feedUrl: 'https://...',
+  itemId: 'guid-or-url',
+  itemUrl: 'https://...',
+  itemTitle: '...',
+  processedAt: Timestamp,
+  brandId: '...',
+  postUrl: '...',
+  postSlug: '...',
+  metadata: { created: Timestamp },
+}
+```
+
+## Configuration
+
+```js
+// backend-manager-config.json
+ghostii: [
+  {
+    articles: 1,                    // articles per cron run (0 = disabled)
+    sources: ['$app'],              // source pool (mixed types OK)
+    links: [],                      // URLs to inject into article body
+    prompt: '',                     // custom instructions for the AI
+    chance: 1.0,                    // probability of running (0-1)
+    author: null,                   // author slug (null = admin/post default)
+    postPath: 'ghostii',            // sub-folder under _posts/{year}/
+    overrides: {                    // per-entry Ghostii API param overrides
+      // keywords: ['AI'],
+      // length: 'long',            // short | medium | long | comprehensive
+      // research: true,            // web search for real links
+      // insertImages: true,
+      // insertLinks: true,
+      // headerImageUrl: 'unsplash', // disabled | unsplash | generate
+      // maxLinks: 6,
+      // sectionQuantity: 5,
+      // feedUrl: '',               // brand blog feed (title dedup)
+    },
+    // brand: 'other-brand-id',     // cross-brand publishing
+    // brandUrl: 'https://api.other.com',
+  }
+]
+```
+
+### Per-entry overrides
+
+All Ghostii API params were previously hardcoded in `writeArticle()`. They now accept per-entry overrides via the `overrides` object. Unspecified values fall back to framework defaults (length: 'long', research: true, etc.).
+
+### `sourceContent` — Ghostii API field
+
+When a `$feed:` source provides extracted article text, it's passed to the Ghostii API as `sourceContent` (max 16KB). This is separate from `description` (the editorial brief, max 2KB). Ghostii's title generation, outline, and section writing prompts reference the source content as context for writing original articles — not rewrites.
+
+## Feed Parser
+
+`src/manager/libraries/content/feed-parser.js` exports:
+
+- **`parseFeed(text)`** — Parse RSS 2.0, Atom 1.0, or JSON Feed into `{ items: [{ id, title, url, summary, content, publishedAt }] }`. Handles CDATA, namespaced elements (`content:encoded`), BOM, attribute-based links. Returns `{ items: [] }` on invalid input.
+
+- **`extractArticleContent(url)`** — Fetch URL, extract readable text from `<article>` → `<main>` → `<body>`, strip scripts/styles/nav/footer/header/aside, normalize whitespace, truncate to 14KB.
+
+## Tests
+
+```bash
+npx mgr test bem:helpers/content/feed-parser         # 32 tests: RSS/Atom/JSON parsing, edge cases, HTML extraction
+npx mgr test bem:helpers/content/ghostii-write-article # 15 tests: override pass-through, sourceContent, backwards compat
+npx mgr test bem:helpers/content/ghostii-auto-publisher # 20 tests: source detection, feed processing, Firestore tracking
+npx mgr test bem:helpers/content/ghostii-blocks        # 8 tests: blocksToPost() (unchanged)
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backend-manager",
-  "version": "5.7.6",
+  "version": "5.8.1",
   "description": "Quick tools for developing Firebase functions",
   "main": "src/manager/index.js",
   "bin": {
@@ -72,9 +72,11 @@
     "body-parser": "^2.2.2",
     "busboy": "^1.6.0",
     "chalk": "^5.6.2",
+    "cheerio": "^1.2.0",
     "cors": "^2.8.6",
     "dotenv": "^17.4.2",
     "express": "^5.2.1",
+    "fast-xml-parser": "^5.9.2",
     "fs-jetpack": "^5.1.0",
     "glob": "^13.0.6",
     "hcaptcha": "^0.2.0",

package/src/manager/events/cron/daily/ghostii-auto-publisher.js

@@ -1,9 +1,29 @@
+/**
+ * Ghostii Auto Publisher — daily cron job.
+ *
+ * Generates and publishes blog posts using the Ghostii AI article engine.
+ * Iterates `config.ghostii[]` entries, each defining a source pool (generic
+ * topics, URLs, text prompts, or RSS/Atom feeds) and per-entry overrides for
+ * the Ghostii API params.
+ *
+ * Source types:
+ *   '$app'              — generic brand-topic generation
+ *   '$feed:<url>'       — RSS/Atom/JSON feed: pick one unprocessed article per run
+ *   'https://...'       — fetch URL content as prompt seed
+ *   '<text>'            — use directly as prompt seed
+ *
+ * Feed items are tracked in Firestore (`ghostii-feed-items`) so the same
+ * article is never processed twice. When a feed is unreachable or exhausted,
+ * the entry falls back to $app behavior.
+ */
+const crypto = require('crypto');
 const fetch = require('wonderful-fetch');
 const powertools = require('node-powertools');
 const moment = require('moment');
 const JSON5 = require('json5');
 
 const { writeArticle, publishArticle } = require('../../../libraries/content/ghostii.js');
+const { parseFeed, extractArticleContent } = require('../../../libraries/content/feed-parser.js');
 
 const PROMPT = `
   Company: {brand.brand.name}: {brand.brand.description}
@@ -14,7 +34,18 @@ const PROMPT = `
   {suggestion}
 `;
 
+const PROMPT_SOURCE = `
+  Company: {brand.brand.name}: {brand.brand.description}
+  Date: {date}
+  Instructions: {prompt}
+
+  Write an original article inspired by and referencing this source material.
+  Do NOT copy the source — use it as context and inspiration for a unique take.
+  Source title: {sourceTitle}
+`;
+
 const USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36';
+const FEED_PREFIX = '$feed:';
 
 // State
 let postId;
@@ -40,6 +71,9 @@ module.exports = async ({ Manager, assistant, context, libraries }) => {
   // Get settings
   const settingsArray = powertools.arrayify(Manager.config.ghostii);
 
+  // Get admin for Firestore tracking
+  const { admin } = libraries;
+
   // Loop through each item
   for (const settings of settingsArray) {
     // Fix settings
@@ -49,6 +83,8 @@ module.exports = async ({ Manager, assistant, context, libraries }) => {
     settings.prompt = settings.prompt || '';
     settings.chance = settings.chance || 1.0;
     settings.author = settings.author || undefined;
+    settings.postPath = settings.postPath || 'ghostii';
+    settings.overrides = settings.overrides || {};
 
     // Resolve brand data for this ghostii item
     if (settings.brand && settings.brandUrl) {
@@ -81,7 +117,7 @@ module.exports = async ({ Manager, assistant, context, libraries }) => {
     }
 
     // Harvest articles
-    const result = await harvest(assistant, settings).catch((e) => e);
+    const result = await harvest(assistant, settings, admin).catch((e) => e);
     if (result instanceof Error) {
       throw result;
     }
@@ -111,7 +147,7 @@ function fetchRemoteBrand(brandUrl) {
   });
 }
 
-async function harvest(assistant, settings) {
+async function harvest(assistant, settings, admin) {
   const date = moment().format('MMMM YYYY');
 
   // Log
@@ -120,43 +156,27 @@ async function harvest(assistant, settings) {
   // Process the number of sources in the settings
   for (let index = 0; index < settings.articles; index++) {
     const source = powertools.random(settings.sources);
-    const sourceIsURL = isURL(source);
-    let suggestion = null;
 
     // Log
-    assistant.log(`harvest(): Processing ${index + 1}/${settings.articles} sources isURL=${sourceIsURL}`, source);
-
-    // Get suggestion
-    if (source === '$app') {
-      suggestion = 'Write an article about any topic that would be relevant to our website and business (it does not have to be about our company, but it can be)';
-    } else if (sourceIsURL) {
-      suggestion = await getURLContent(source).catch((e) => e);
-    } else {
-      suggestion = source;
-    }
+    assistant.log(`harvest(): Processing ${index + 1}/${settings.articles}`, source);
 
-    // Check for errors
-    if (suggestion instanceof Error) {
-      assistant.error(`harvest(): Error fetching ${source} suggestion`, suggestion);
+    // Resolve the source into a prompt description + optional sourceContent
+    const resolved = await resolveSource(assistant, source, settings, admin).catch((e) => e);
+    if (resolved instanceof Error) {
+      assistant.error('harvest(): Error resolving source', resolved);
       break;
     }
 
-    // Set suggestion
-    const final = powertools.template(PROMPT, {
-      ...settings,
-      prompt: settings.prompt,
-      date: date,
-      suggestion: suggestion,
-    });
-
     // Log
-    assistant.log('harvest(): Get final content', final);
+    assistant.log('harvest(): Resolved source', resolved);
 
     // Request to Ghostii
     const article = await writeArticle({
       brand: settings.brand,
-      description: final,
+      description: resolved.description,
       links: settings.links,
+      sourceContent: resolved.sourceContent || '',
+      overrides: settings.overrides,
     }).catch((e) => e);
     if (article instanceof Error) {
       assistant.error('harvest(): Error requesting Ghostii', article);
@@ -172,7 +192,7 @@ async function harvest(assistant, settings) {
       article,
       id: postId++,
       author: settings.author,
-      postPath: 'ghostii',
+      postPath: settings.postPath,
     }).catch((e) => e);
     if (uploadedPost instanceof Error) {
       assistant.error('harvest(): Error uploading post to blog', uploadedPost);
@@ -181,7 +201,239 @@ async function harvest(assistant, settings) {
 
     // Log
     assistant.log('harvest(): Uploaded post', uploadedPost);
+
+    // Track feed item in Firestore (if this was a feed source)
+    if (resolved.feedItem && admin) {
+      await trackFeedItem(admin, {
+        feedUrl: resolved.feedUrl,
+        item: resolved.feedItem,
+        brandId: settings.brand.brand.id,
+        postUrl: uploadedPost.url,
+        postSlug: uploadedPost.slug,
+      }).catch((e) => {
+        assistant.error('harvest(): Error tracking feed item (non-fatal)', e);
+      });
+    }
+  }
+}
+
+/**
+ * Resolve a source entry into { description, sourceContent, feedItem?, feedUrl? }.
+ *
+ * Source types:
+ *   '$app'         — generic topic prompt, no sourceContent
+ *   '$feed:<url>'  — RSS/Atom feed: pick unprocessed article, extract content
+ *   URL            — fetch page content as prompt seed
+ *   text           — use directly as prompt seed
+ *
+ * Feed failures fall back to $app behavior.
+ */
+async function resolveSource(assistant, source, settings, admin) {
+  const date = moment().format('MMMM YYYY');
+
+  // --- $feed: source ---
+  if (typeof source === 'string' && source.startsWith(FEED_PREFIX)) {
+    const feedUrl = source.slice(FEED_PREFIX.length);
+    assistant.log(`resolveSource(): Processing feed: ${feedUrl}`);
+
+    const feedResult = await processFeedSource(assistant, feedUrl, settings.brand.brand.id, admin).catch((e) => e);
+
+    if (feedResult instanceof Error || !feedResult) {
+      assistant.log('resolveSource(): Feed failed or exhausted, falling back to $app');
+      return resolveSource(assistant, '$app', settings, admin);
+    }
+
+    const description = powertools.template(PROMPT_SOURCE, {
+      ...settings,
+      prompt: settings.prompt,
+      date: date,
+      sourceTitle: feedResult.item.title,
+    });
+
+    return {
+      description,
+      sourceContent: feedResult.content || feedResult.item.summary || '',
+      feedItem: feedResult.item,
+      feedUrl: feedUrl,
+    };
+  }
+
+  // --- $app source ---
+  if (source === '$app') {
+    const suggestion = 'Write an article about any topic that would be relevant to our website and business (it does not have to be about our company, but it can be)';
+    const description = powertools.template(PROMPT, {
+      ...settings,
+      prompt: settings.prompt,
+      date: date,
+      suggestion: suggestion,
+    });
+
+    return { description, sourceContent: '' };
+  }
+
+  // --- URL source ---
+  if (isURL(source)) {
+    const suggestion = await getURLContent(source).catch((e) => e);
+
+    if (suggestion instanceof Error) {
+      assistant.error(`resolveSource(): Error fetching URL ${source}`, suggestion);
+      return resolveSource(assistant, '$app', settings, admin);
+    }
+
+    const description = powertools.template(PROMPT, {
+      ...settings,
+      prompt: settings.prompt,
+      date: date,
+      suggestion: suggestion,
+    });
+
+    return { description, sourceContent: '' };
+  }
+
+  // --- Text source ---
+  const description = powertools.template(PROMPT, {
+    ...settings,
+    prompt: settings.prompt,
+    date: date,
+    suggestion: source,
+  });
+
+  return { description, sourceContent: '' };
+}
+
+/**
+ * Process a $feed: source — fetch, parse, select unprocessed item, extract content.
+ *
+ * @param {object} assistant - BEM assistant
+ * @param {string} feedUrl - RSS/Atom/JSON feed URL
+ * @param {string} brandId - Brand ID for tracking
+ * @param {object} admin - Firebase Admin SDK
+ * @returns {Promise<{ item: FeedItem, content: string }|null>} Selected item + extracted content, or null
+ */
+async function processFeedSource(assistant, feedUrl, brandId, admin) {
+  // Fetch the feed
+  const feedText = await fetch(feedUrl, {
+    timeout: 30000,
+    tries: 2,
+    response: 'text',
+    headers: { 'User-Agent': USER_AGENT },
+  }).catch((e) => e);
+
+  if (feedText instanceof Error) {
+    assistant.error(`processFeedSource(): Failed to fetch feed: ${feedUrl}`, feedText);
+    return null;
+  }
+
+  // Parse the feed
+  const { items } = parseFeed(feedText);
+  if (!items.length) {
+    assistant.log(`processFeedSource(): No items in feed: ${feedUrl}`);
+    return null;
+  }
+
+  assistant.log(`processFeedSource(): Parsed ${items.length} items from feed`);
+
+  // Get already-processed item IDs from Firestore
+  const processedIds = await getProcessedItemIds(admin, feedUrl).catch((e) => {
+    assistant.error('processFeedSource(): Error querying tracked items (continuing)', e);
+    return new Set();
+  });
+
+  // Filter to unprocessed items
+  const unprocessed = items.filter((item) => !processedIds.has(item.id) && !processedIds.has(item.url));
+  if (!unprocessed.length) {
+    assistant.log(`processFeedSource(): All ${items.length} items already processed for: ${feedUrl}`);
+    return null;
+  }
+
+  assistant.log(`processFeedSource(): ${unprocessed.length} unprocessed items available`);
+
+  // Pick the first (newest) unprocessed item
+  const item = unprocessed[0];
+
+  // Extract full article content from the item URL
+  let content = '';
+  if (item.url) {
+    content = await extractArticleContent(item.url).catch((e) => {
+      assistant.error(`processFeedSource(): Failed to extract content from ${item.url} (using summary)`, e);
+      return '';
+    });
+  }
+
+  // Fall back to inline feed content or summary
+  if (!content || content.length < 100) {
+    content = item.content || item.summary || '';
+  }
+
+  return { item, content };
+}
+
+/**
+ * Query Firestore for already-processed feed item IDs.
+ *
+ * @param {object} admin - Firebase Admin SDK
+ * @param {string} feedUrl - The feed URL to query
+ * @returns {Promise<Set<string>>} Set of processed item IDs
+ */
+async function getProcessedItemIds(admin, feedUrl) {
+  if (!admin) {
+    return new Set();
   }
+
+  const snapshot = await admin.firestore()
+    .collection('ghostii-feed-items')
+    .where('feedUrl', '==', feedUrl)
+    .select('itemId', 'itemUrl')
+    .get();
+
+  const ids = new Set();
+  snapshot.docs.forEach((doc) => {
+    const data = doc.data();
+    if (data.itemId) {
+      ids.add(data.itemId);
+    }
+    if (data.itemUrl) {
+      ids.add(data.itemUrl);
+    }
+  });
+
+  return ids;
+}
+
+/**
+ * Write a tracking doc for a processed feed item.
+ *
+ * @param {object} admin - Firebase Admin SDK
+ * @param {object} args
+ * @param {string} args.feedUrl - The source feed URL
+ * @param {object} args.item - The processed feed item
+ * @param {string} args.brandId - Brand that processed it
+ * @param {string} args.postUrl - Published blog post URL
+ * @param {string} args.postSlug - Published blog post slug
+ */
+async function trackFeedItem(admin, { feedUrl, item, brandId, postUrl, postSlug }) {
+  const docId = feedItemHash(feedUrl, item.id || item.url);
+
+  await admin.firestore().doc(`ghostii-feed-items/${docId}`).set({
+    feedUrl: feedUrl,
+    itemId: item.id || item.url,
+    itemUrl: item.url,
+    itemTitle: item.title,
+    processedAt: admin.firestore.FieldValue.serverTimestamp(),
+    brandId: brandId,
+    postUrl: postUrl || null,
+    postSlug: postSlug || null,
+    metadata: {
+      created: admin.firestore.FieldValue.serverTimestamp(),
+    },
+  });
+}
+
+/**
+ * Deterministic hash for a feed item — used as the Firestore doc ID.
+ */
+function feedItemHash(feedUrl, itemId) {
+  return crypto.createHash('sha256').update(`${feedUrl}::${itemId}`).digest('hex').slice(0, 20);
 }
 
 function getURLContent(url) {
@@ -250,3 +502,10 @@ function tryParse(json) {
 function randomize(array) {
   return array.sort(() => Math.random() - 0.5);
 }
+
+module.exports.resolveSource = resolveSource;
+module.exports.processFeedSource = processFeedSource;
+module.exports.feedItemHash = feedItemHash;
+module.exports.getProcessedItemIds = getProcessedItemIds;
+module.exports.trackFeedItem = trackFeedItem;
+module.exports.isURL = isURL;