npm - backend-manager - Versions diffs - 5.9.17 → 5.9.18 - Mend

backend-manager 5.9.17 → 5.9.18

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (6) hide show

package/docs/ghostii.md +63 -5
package/package.json +1 -1
package/src/manager/events/cron/daily/blog-auto-publisher.js +22 -398
package/src/manager/libraries/content/source-resolver.js +397 -0
package/src/manager/libraries/email/generators/newsletter.js +18 -46
package/test/email/newsletter-generate.js +65 -50

package/docs/ghostii.md CHANGED Viewed

@@ -27,9 +27,10 @@ trackContentSource() -- Firestore (for $feed:, $parent sources)
 | File | Purpose |
 |---|---|
+| `src/manager/libraries/content/source-resolver.js` | **Shared SSOT**: prompt templates, anti-traceability rules, feed/parent resolution, Firestore tracking, fallback chain. Used by both blog + newsletter. |
 | `src/manager/libraries/content/ghostii.js` | `writeArticle()`, `publishArticle()`, `blocksToPost()` -- Ghostii API client + post transform |
 | `src/manager/libraries/content/feed-parser.js` | `parseFeed()`, `extractArticleContent()` -- RSS/Atom/JSON parser + article extractor |
-| `src/manager/events/cron/daily/blog-auto-publisher.js` | Daily cron: source resolution, feed processing, Firestore tracking, provider dispatch |
+| `src/manager/events/cron/daily/blog-auto-publisher.js` | Daily cron: imports from source-resolver, manages harvest loop + provider dispatch |
 | `src/manager/routes/admin/post/post.js` | Publishing endpoint: image download + resize, GitHub commit |
 ## Source Types
@@ -145,11 +146,68 @@ When a `$feed:` or `$parent` source provides extracted article text, it's passed
 - **`extractArticleContent(url)`** -- Fetch URL, extract readable text from `<article>` -> `<main>` -> `<body>`, strip scripts/styles/nav/footer/header/aside, normalize whitespace, truncate to 14KB.
+## Prompt Templates (source-resolver.js)
+Both blog and newsletter use shared prompt constants from `source-resolver.js`:
+- **`PROMPT_SOURCE`** — used when a `$feed:` or `$parent` source is resolved. Directs the AI to cover the SAME topic with a different angle, title, and structure. Includes anti-traceability rules.
+- **`PROMPT`** — used for `$brand`, URL, and text sources. Open-ended topic generation.
+- **`ANTI_TRACEABILITY`** — shared rules: never name the source publication, paraphrase all data. Embedded in `PROMPT_SOURCE`, also used by the newsletter's system prompt.
+The brand's `instructions` field is injected into both templates and can override the default behavior (e.g. a brand that WANTS genericized output can say so in instructions).
 ## Tests
+### Unit tests (fast, free)
+```bash
+npx mgr test mgr:helpers/content/feed-parser          # 32 tests: RSS/Atom/JSON parsing, edge cases
+npx mgr test mgr:helpers/content/ghostii-write-article # 15 tests: override pass-through, sourceContent
+npx mgr test mgr:helpers/content/blog-auto-publisher   # 25+ tests: source detection, feed processing, tracking
+npx mgr test mgr:helpers/content/ghostii-blocks        # 8 tests: blocksToPost()
+```
+### Blog generation (full AI pipeline)
+```bash
+# Config check only (fast, free)
+npx mgr test mgr:content/blog-generate
+# Full AI pipeline — generates article, does NOT publish
+BLOG_NO_PUBLISH=1 TEST_EXTENDED_MODE=1 npx mgr test mgr:content/blog-generate
+# Override source type
+BLOG_SOURCE='$feed:https://feeds.arstechnica.com/arstechnica/index' \
+  BLOG_NO_PUBLISH=1 TEST_EXTENDED_MODE=1 npx mgr test mgr:content/blog-generate
+```
+| Env var | Description |
+|---|---|
+| `TEST_EXTENDED_MODE=1` | Enable full AI pipeline (costs ~$0.10-0.50 per run) |
+| `BLOG_NO_PUBLISH=1` | Generate but do NOT publish to website repo |
+| `BLOG_SOURCE=<type>` | Override source: `$brand`, `$parent`, `$feed:<url>`, URL, or text |
+| `BLOG_OPEN=1` | Auto-open generated article (macOS only) |
+### Newsletter generation (full AI pipeline)
 ```bash
-npx mgr test bem:helpers/content/feed-parser          # 32 tests: RSS/Atom/JSON parsing, edge cases, HTML extraction
-npx mgr test bem:helpers/content/ghostii-write-article # 15 tests: override pass-through, sourceContent, backwards compat
-npx mgr test bem:helpers/content/blog-auto-publisher   # 25+ tests: source detection, feed processing, Firestore tracking
-npx mgr test bem:helpers/content/ghostii-blocks        # 8 tests: blocksToPost() (unchanged)
+# Fixture render only (fast, free)
+npx mgr test mgr:email/newsletter-generate
+# Full AI pipeline with feed source — generates newsletter, does NOT publish article
+NEWSLETTER_SOURCE='$feed:https://feeds.arstechnica.com/arstechnica/index' \
+  NEWSLETTER_NO_IMAGES=1 TEST_EXTENDED_MODE=1 npx mgr test mgr:email/newsletter-generate
 ```
+| Env var | Description |
+|---|---|
+| `TEST_EXTENDED_MODE=1` | Enable full AI pipeline |
+| `NEWSLETTER_SOURCE=<type>` | Override source: `$feed:<url>`, `$parent`. Bypasses parent-server fetch, lets the generator's resolver run. |
+| `NEWSLETTER_NO_IMAGES=1` | Skip image generation (fast iteration on copy) |
+| `NEWSLETTER_CREATE_ARTICLE=1` | Publish linked blog article (OFF by default) |
+| `NEWSLETTER_FIXTURE=<name>` | Load specific fixture for render test |
+| `NEWSLETTER_TEMPLATE=<name>` | Override layout template |
+| `NEWSLETTER_OPEN=1` | Auto-open preview (macOS only) |
+| `NEWSLETTER_PEEK=1` | List ready parent sources, don't generate |
+| `NEWSLETTER_SOURCE_ID=<id>` | Test specific parent source |
+| `NEWSLETTER_CAMPAIGN_ID=<id>` | Override campaign folder name |

package/package.json CHANGED Viewed

@@ -1,6 +1,6 @@
 {
   "name": "backend-manager",
-  "version": "5.9.17",
+  "version": "5.9.18",
   "description": "Quick tools for developing Firebase functions",
   "main": "src/manager/index.js",
   "files": [

package/src/manager/events/cron/daily/blog-auto-publisher.js CHANGED Viewed

@@ -16,87 +16,49 @@
  * All feed/parent sources are tracked in Firestore (`content-sources`) so the
  * same article is never processed twice. When a feed is unreachable or
  * exhausted, the entry falls back to $brand behavior.
+ *
+ * Source resolution, tracking, and prompt constants live in the shared
+ * source-resolver library (src/manager/libraries/content/source-resolver.js).
  */
-const crypto = require('crypto');
-const fetch = require('wonderful-fetch');
 const powertools = require('node-powertools');
 const moment = require('moment');
-const JSON5 = require('json5');
-const { parseFeed, extractArticleContent } = require('../../../libraries/content/feed-parser.js');
-const PROMPT = `
-  Company: {brand.brand.name}: {brand.brand.description}
-  Date: {date}
-  Instructions: {instructions}
-  Tone: {tone}
-  Categories: {categories}
-  Keywords: {keywords}
-  Use the following information to find a topic for our company blog (it can be about our company OR any topic that would be relevant to our website and business BUT not about a competitor):
-  {suggestion}
-`;
-const PROMPT_SOURCE = `
-  Company: {brand.brand.name}: {brand.brand.description}
-  Date: {date}
-  Instructions: {instructions}
-  Tone: {tone}
-  Categories: {categories}
-  Keywords: {keywords}
-  Write an original article inspired by and referencing this source material.
-  Do NOT copy the source — use it as context and inspiration for a unique take.
-  Source title: {sourceTitle}
-`;
-const USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36';
-const FEED_PREFIX = '$feed:';
-const CONTENT_SOURCES_COLLECTION = 'content-sources';
-// State
+const {
+  PROMPT_SOURCE,
+  PROMPT,
+  FEED_PREFIX,
+  processFeedSource,
+  processParentSource,
+  trackContentSource,
+  contentSourceHash,
+  getProcessedItemIds,
+  getURLContent,
+  isURL,
+} = require('../../../libraries/content/source-resolver.js');
 let postId;
-/**
- * Blog Auto Publisher cron job
- *
- * Automatically generates and publishes blog posts using the configured platform provider.
- */
 module.exports = async ({ Manager, assistant, context, libraries }) => {
-  // Gate on blog.enabled
   if (!Manager.config.blog?.enabled) {
     assistant.log('Blog auto-publisher disabled in config');
     return;
   }
-  // Log
   assistant.log('Starting...');
-  // Set post ID
   postId = moment().unix();
-  // Build brand config from local config
   const brandConfig = buildBrandConfig(Manager.config);
-  // Log
   assistant.log('Brand config', brandConfig);
-  // Get blog config
   const blog = Manager.config.blog;
-  // Get content entries (array or single object)
   const contentArray = powertools.arrayify(blog.content);
-  // Get admin for Firestore tracking
   const { admin } = libraries;
-  // Load the provider based on blog.platform
   const platform = blog.platform || 'ghostii';
   const provider = require(`../../../libraries/content/${platform}.js`);
-  // Loop through each content entry
   for (const entry of contentArray) {
-    // Normalize entry fields
     entry.quantity = entry.quantity || 0;
     entry.sources = randomize(entry.sources || []);
     entry.links = randomize(entry.links || []);
@@ -109,60 +71,45 @@ module.exports = async ({ Manager, assistant, context, libraries }) => {
     entry.postPath = entry.postPath || platform;
     entry.overrides = entry.overrides || {};
-    // Resolve brand data for this entry
     if (entry.brand && entry.brandUrl) {
-      // Cross-brand: fetch from the other project's /brand endpoint
       entry.brand = await fetchRemoteBrand(entry.brandUrl).catch((e) => e);
       if (entry.brand instanceof Error) {
         assistant.error('Error fetching remote brand data', entry.brand);
         continue;
       }
     } else {
-      // Same-brand: use local config
       entry.brand = brandConfig;
     }
-    // Log
     assistant.log(`Entry (brand=${entry.brand.brand.id})`, entry);
-    // Quit if quantity is zero or no sources
     if (!entry.quantity || !entry.sources.length) {
       assistant.log('Quitting because quantity is 0 or no sources');
       continue;
     }
-    // Quit if the chance is not met
     const chance = Math.random();
     if (chance > entry.chance) {
       assistant.log(`Quitting because the chance is not met (${chance} <= ${entry.chance})`);
       continue;
     }
-    // Harvest articles
     const result = await harvest(assistant, entry, admin, provider, Manager).catch((e) => e);
     if (result instanceof Error) {
       throw result;
     }
-    // Log
     assistant.log('Finished!', result);
   }
 };
-/**
- * Build brand config from Manager.config (same shape as /brand endpoint response)
- */
 function buildBrandConfig(config) {
   const { buildPublicConfig } = require(require('path').join(__dirname, '..', '..', '..', 'routes', 'brand', 'get.js'));
   return buildPublicConfig(config);
 }
-/**
- * Fetch brand data from a remote BEM project's /brand endpoint
- */
 function fetchRemoteBrand(brandUrl) {
+  const fetch = require('wonderful-fetch');
   return fetch(`${brandUrl}/backend-manager/brand`, {
     timeout: 120000,
     tries: 3,
@@ -173,33 +120,26 @@ function fetchRemoteBrand(brandUrl) {
 async function harvest(assistant, entry, admin, provider, Manager) {
   const date = moment().format('MMMM YYYY');
-  // Log
   assistant.log(`harvest(): Starting ${entry.brand.brand.id}...`);
-  // Process the number of sources in the entry
   for (let index = 0; index < entry.quantity; index++) {
     const source = powertools.random(entry.sources);
-    // Log
     assistant.log(`harvest(): Processing ${index + 1}/${entry.quantity}`, source);
-    // Resolve the source into a prompt description + optional sourceContent
     const resolved = await resolveSource(assistant, source, entry, admin, Manager).catch((e) => e);
     if (resolved instanceof Error) {
       assistant.error('harvest(): Error resolving source', resolved);
       break;
     }
-    // Log
     assistant.log('harvest(): Resolved source', resolved);
-    // Use keywords from entry if overrides.keywords is not set
     const overrides = { ...entry.overrides };
     if (!overrides.keywords && entry.keywords.length) {
       overrides.keywords = entry.keywords;
     }
-    // Request article from provider
     const article = await provider.writeArticle({
       brand: entry.brand,
       description: resolved.description,
@@ -212,10 +152,8 @@ async function harvest(assistant, entry, admin, provider, Manager) {
       break;
     }
-    // Log
     assistant.log('harvest(): Article', article);
-    // Upload post to blog
     const uploadedPost = await provider.publishArticle(assistant, {
       brand: entry.brand,
       article,
@@ -229,10 +167,8 @@ async function harvest(assistant, entry, admin, provider, Manager) {
       break;
     }
-    // Log
     assistant.log('harvest(): Uploaded post', uploadedPost);
-    // Track content source in Firestore
     if (resolved.trackingData && admin) {
       await trackContentSource(admin, {
         ...resolved.trackingData,
@@ -246,22 +182,9 @@ async function harvest(assistant, entry, admin, provider, Manager) {
   }
 }
-/**
- * Resolve a source entry into { description, sourceContent, trackingData? }.
- *
- * Source types:
- *   '$brand'       — generic topic prompt, no sourceContent
- *   '$feed:<url>'  — RSS/Atom feed: pick unprocessed article, extract content
- *   '$parent'      — fetch sources from parent server without claiming
- *   URL            — fetch page content as prompt seed
- *   text           — use directly as prompt seed
- *
- * Feed/parent failures fall back to $brand behavior.
- */
 async function resolveSource(assistant, source, entry, admin, Manager) {
   const date = moment().format('MMMM YYYY');
-  // Build template vars for prompts
   const templateVars = {
     ...entry,
     instructions: entry.instructions,
@@ -306,13 +229,14 @@ async function resolveSource(assistant, source, entry, admin, Manager) {
   if (source === '$parent') {
     assistant.log('resolveSource(): Processing $parent source');
-    const parentResult = await processParentSource(assistant, entry, admin, Manager).catch((e) => e);
+    const parentResults = await processParentSource(assistant, entry, admin, Manager).catch((e) => e);
-    if (parentResult instanceof Error || !parentResult) {
+    if (parentResults instanceof Error || !parentResults?.length) {
       assistant.log('resolveSource(): Parent source failed or exhausted, falling back to $brand');
       return resolveSource(assistant, '$brand', entry, admin, Manager);
     }
+    const parentResult = parentResults[0];
     const description = powertools.template(PROMPT_SOURCE, {
       ...templateVars,
       sourceTitle: parentResult.title,
@@ -368,311 +292,11 @@ async function resolveSource(assistant, source, entry, admin, Manager) {
   return { description, sourceContent: '' };
 }
-/**
- * Process a $parent source — fetch sources from the parent server without claiming.
- *
- * Fetches available sources via GET /newsletter-sources (no claimFor parameter),
- * filters out any already tracked locally in `content-sources`, and returns
- * the first available source.
- *
- * @param {object} assistant - BEM assistant
- * @param {object} entry - Content entry config
- * @param {object} admin - Firebase Admin SDK
- * @param {object} Manager - BEM Manager instance
- * @returns {Promise<{ id, title, url, content, summary }|null>} Selected source or null
- */
-async function processParentSource(assistant, entry, admin, Manager) {
-  const parentUrl = Manager?.getParentApiUrl?.();
-  if (!parentUrl) {
-    assistant.log('processParentSource(): No parent URL configured');
-    return null;
-  }
-  const categories = entry.categories || [];
-  const allSources = [];
-  // Fetch sources from each category (or all if no categories)
-  const categoriesToFetch = categories.length ? categories : [''];
-  for (const category of categoriesToFetch) {
-    const query = {
-      limit: 3,
-      backendManagerKey: process.env.BACKEND_MANAGER_KEY,
-    };
-    if (category) {
-      query.category = category;
-    }
-    const data = await fetch(`${parentUrl}/newsletter-sources`, {
-      method: 'get',
-      response: 'json',
-      timeout: 60000,
-      query: query,
-    }).catch((e) => {
-      assistant.error(`processParentSource(): Failed to fetch sources for category=${category}: ${e.message}`);
-      return null;
-    });
-    if (data?.sources?.length) {
-      allSources.push(...data.sources);
-    }
-  }
-  if (!allSources.length) {
-    assistant.log('processParentSource(): No sources available from parent');
-    return null;
-  }
-  // Filter out already-used sources by checking content-sources collection locally
-  if (admin) {
-    const usedUrls = new Set();
-    const snapshot = await admin.firestore()
-      .collection(CONTENT_SOURCES_COLLECTION)
-      .where('origin', '==', '$parent')
-      .select('url')
-      .get()
-      .catch((e) => {
-        assistant.error('processParentSource(): Error querying tracked sources (continuing)', e);
-        return { docs: [] };
-      });
-    snapshot.docs.forEach((doc) => {
-      const data = doc.data();
-      if (data.url) {
-        usedUrls.add(data.url);
-      }
-    });
-    const available = allSources.filter((s) => !usedUrls.has(s.url || s.id));
-    if (!available.length) {
-      assistant.log('processParentSource(): All parent sources already used');
-      return null;
-    }
-    return available[0];
-  }
-  // No admin — just return the first source
-  return allSources[0];
-}
-/**
- * Process a $feed: source — fetch, parse, select unprocessed item, extract content.
- *
- * @param {object} assistant - BEM assistant
- * @param {string} feedUrl - RSS/Atom/JSON feed URL
- * @param {string} brandId - Brand ID for tracking
- * @param {object} admin - Firebase Admin SDK
- * @returns {Promise<{ item: FeedItem, content: string }|null>} Selected item + extracted content, or null
- */
-async function processFeedSource(assistant, feedUrl, brandId, admin) {
-  // Fetch the feed
-  const feedText = await fetch(feedUrl, {
-    timeout: 30000,
-    tries: 2,
-    response: 'text',
-    headers: { 'User-Agent': USER_AGENT },
-  }).catch((e) => e);
-  if (feedText instanceof Error) {
-    assistant.error(`processFeedSource(): Failed to fetch feed: ${feedUrl}`, feedText);
-    return null;
-  }
-  // Parse the feed
-  const { items } = parseFeed(feedText);
-  if (!items.length) {
-    assistant.log(`processFeedSource(): No items in feed: ${feedUrl}`);
-    return null;
-  }
-  assistant.log(`processFeedSource(): Parsed ${items.length} items from feed`);
-  // Get already-processed item IDs from Firestore
-  const processedIds = await getProcessedItemIds(admin, feedUrl).catch((e) => {
-    assistant.error('processFeedSource(): Error querying tracked items (continuing)', e);
-    return new Set();
-  });
-  // Filter to unprocessed items
-  const unprocessed = items.filter((item) => !processedIds.has(item.id) && !processedIds.has(item.url));
-  if (!unprocessed.length) {
-    assistant.log(`processFeedSource(): All ${items.length} items already processed for: ${feedUrl}`);
-    return null;
-  }
-  assistant.log(`processFeedSource(): ${unprocessed.length} unprocessed items available`);
-  // Pick the first (newest) unprocessed item
-  const item = unprocessed[0];
-  // Extract full article content from the item URL
-  let content = '';
-  if (item.url) {
-    content = await extractArticleContent(item.url).catch((e) => {
-      assistant.error(`processFeedSource(): Failed to extract content from ${item.url} (using summary)`, e);
-      return '';
-    });
-  }
-  // Fall back to inline feed content or summary
-  if (!content || content.length < 100) {
-    content = item.content || item.summary || '';
-  }
-  return { item, content };
-}
-/**
- * Query Firestore for already-processed feed item IDs.
- *
- * @param {object} admin - Firebase Admin SDK
- * @param {string} feedUrl - The feed URL to query
- * @returns {Promise<Set<string>>} Set of processed item IDs
- */
-async function getProcessedItemIds(admin, feedUrl) {
-  if (!admin) {
-    return new Set();
-  }
-  const snapshot = await admin.firestore()
-    .collection(CONTENT_SOURCES_COLLECTION)
-    .where('feedUrl', '==', feedUrl)
-    .select('itemId', 'url')
-    .get();
-  const ids = new Set();
-  snapshot.docs.forEach((doc) => {
-    const data = doc.data();
-    if (data.itemId) {
-      ids.add(data.itemId);
-    }
-    if (data.url) {
-      ids.add(data.url);
-    }
-  });
-  return ids;
-}
-/**
- * Write a tracking doc for a processed content source.
- *
- * Unified tracker for all source types (feed, parent, brand).
- * Used by both blog-auto-publisher and newsletter generator.
- *
- * @param {object} admin - Firebase Admin SDK
- * @param {object} args
- * @param {string} args.url - Unique identifier (article URL or source ID)
- * @param {string} args.origin - Source type ('$feed:https://...', '$parent', '$brand')
- * @param {string} [args.feedUrl] - The source feed URL (for feed sources)
- * @param {string} [args.itemId] - Item ID within the feed
- * @param {string} [args.itemTitle] - Item title
- * @param {string} args.usedBy - Which system used it ('blog' or 'newsletter')
- * @param {string} args.brandId - Brand that processed it
- * @param {string} [args.postUrl] - Published blog post URL
- * @param {string} [args.postSlug] - Published blog post slug
- */
-async function trackContentSource(admin, { url, origin, feedUrl, itemId, itemTitle, usedBy, brandId, postUrl, postSlug }) {
-  const docId = contentSourceHash(origin || '', url || '');
-  const nowISO = new Date().toISOString();
-  const nowUNIX = Math.round(Date.now() / 1000);
-  await admin.firestore().doc(`${CONTENT_SOURCES_COLLECTION}/${docId}`).set({
-    url: url || null,
-    origin: origin || null,
-    feedUrl: feedUrl || null,
-    itemId: itemId || null,
-    itemTitle: itemTitle || null,
-    usedBy: usedBy || null,
-    brandId: brandId || null,
-    postUrl: postUrl || null,
-    postSlug: postSlug || null,
-    metadata: {
-      created: { timestamp: nowISO, timestampUNIX: nowUNIX },
-      updated: { timestamp: nowISO, timestampUNIX: nowUNIX },
-    },
-  });
-}
-/**
- * Deterministic hash for a content source — used as the Firestore doc ID.
- */
-function contentSourceHash(origin, url) {
-  return crypto.createHash('sha256').update(`${origin}::${url}`).digest('hex').slice(0, 20);
-}
-function getURLContent(url) {
-  return fetch(url, {
-    timeout: 120000,
-    tries: 3,
-    response: 'raw',
-    headers: {
-      'User-Agent': USER_AGENT,
-    },
-  })
-  .then(async (res) => {
-    const contentType = res.headers.get('content-type');
-    const text = await res.text();
-    return extractBodyContent(text, contentType, url);
-  });
-}
-function isURL(url) {
-  try {
-    return !!new URL(url);
-  } catch (e) {
-    return false;
-  }
-}
-function extractBodyContent(text, contentType, url) {
-  const parsed = tryParse(text);
-  // Try JSON
-  if (parsed) {
-    // If it's from rss.app, extract the content
-    if (parsed.items) {
-      return parsed.items.map((i) => `${i.title}: ${i.content_text}`).join('\n');
-    }
-    // If we can't recognize the JSON, return the original text
-    return text;
-  }
-  // Extract the content within the body tag
-  const bodyMatch = text.match(/<body[^>]*>([\s\S]*?)<\/body>/i);
-  if (!bodyMatch) {
-    return '';
-  }
-  let bodyContent = bodyMatch[1];
-  // Remove script and meta tags
-  bodyContent = bodyContent.replace(/<script[^>]*>[\s\S]*?<\/script>/gi, '');
-  bodyContent = bodyContent.replace(/<meta[^>]*>/gi, '');
-  // Remove remaining HTML tags
-  return bodyContent.replace(/<[^>]*>/g, ' ').replace(/\s+/g, ' ').trim();
-}
-function tryParse(json) {
-  try {
-    return JSON5.parse(json);
-  } catch (e) {
-    return null;
-  }
-}
 function randomize(array) {
   return array.sort(() => Math.random() - 0.5);
 }
-// Exported for testing and for newsletter to import
+// Re-export shared functions for backwards compatibility (newsletter + tests import from here)
 module.exports.resolveSource = resolveSource;
 module.exports.processFeedSource = processFeedSource;
 module.exports.contentSourceHash = contentSourceHash;

package/src/manager/libraries/content/source-resolver.js ADDED Viewed

@@ -0,0 +1,397 @@
+/**
+ * Shared content-source resolution for blog + newsletter pipelines.
+ *
+ * Both blog-auto-publisher and newsletter-generator need to:
+ *   - Resolve $feed:, $parent, $brand, URL, and text sources
+ *   - Track used sources in Firestore (content-sources collection)
+ *   - Deduplicate across runs
+ *   - Follow anti-traceability rules
+ *
+ * This module is the single source of truth for all of that.
+ */
+const crypto = require('crypto');
+const fetch = require('wonderful-fetch');
+const { parseFeed, extractArticleContent } = require('./feed-parser.js');
+const USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36';
+const FEED_PREFIX = '$feed:';
+const CONTENT_SOURCES_COLLECTION = 'content-sources';
+// ── Prompt constants ─────────────────────────────────────────────────
+const ANTI_TRACEABILITY = [
+  'ATTRIBUTION RULES:',
+  '- NEVER name the source publication, newsletter, blog, or author.',
+  '- NEVER use phrases like "according to sources", "a recent article said", "as reported by", or similar.',
+  '- Write AS IF the source did not exist — content should read as original, first-party work.',
+  '- If the source mentions a third-party platform, product, or company by name, THAT is fine — those are subjects, not sources.',
+  '- Paraphrase all facts, figures, and data. Never copy exact numbers verbatim.',
+].join('\n');
+const PROMPT_SOURCE = `
+  Company: {brand.brand.name}: {brand.brand.description}
+  Date: {date}
+  Instructions: {instructions}
+  Tone: {tone}
+  Categories: {categories}
+  Keywords: {keywords}
+  Write an article covering the SAME topic as this source material.
+  Keep the specific subject matter — names, companies, events, dates, numbers.
+  Do NOT copy the source text — rewrite it in your own voice with a different angle, title, and structure.
+  The article should read as if written by a completely different author covering the same subject.
+  ${ANTI_TRACEABILITY}
+  Source title: {sourceTitle}
+`;
+const PROMPT = `
+  Company: {brand.brand.name}: {brand.brand.description}
+  Date: {date}
+  Instructions: {instructions}
+  Tone: {tone}
+  Categories: {categories}
+  Keywords: {keywords}
+  Use the following information to find a topic for our company blog (it can be about our company OR any topic that would be relevant to our website and business BUT not about a competitor):
+  {suggestion}
+`;
+// ── Feed source processing ───────────────────────────────────────────
+async function processFeedSource(assistant, feedUrl, brandId, admin) {
+  const feedText = await fetch(feedUrl, {
+    timeout: 30000,
+    tries: 2,
+    response: 'text',
+    headers: { 'User-Agent': USER_AGENT },
+  }).catch((e) => e);
+  if (feedText instanceof Error) {
+    assistant.error(`processFeedSource(): Failed to fetch feed: ${feedUrl}`, feedText);
+    return null;
+  }
+  const { items } = parseFeed(feedText);
+  if (!items.length) {
+    assistant.log(`processFeedSource(): No items in feed: ${feedUrl}`);
+    return null;
+  }
+  assistant.log(`processFeedSource(): Parsed ${items.length} items from feed`);
+  const processedIds = await getProcessedItemIds(admin, feedUrl).catch((e) => {
+    assistant.error('processFeedSource(): Error querying tracked items (continuing)', e);
+    return new Set();
+  });
+  const unprocessed = items.filter((item) => !processedIds.has(item.id) && !processedIds.has(item.url));
+  if (!unprocessed.length) {
+    assistant.log(`processFeedSource(): All ${items.length} items already processed for: ${feedUrl}`);
+    return null;
+  }
+  assistant.log(`processFeedSource(): ${unprocessed.length} unprocessed items available`);
+  const item = unprocessed[0];
+  let content = '';
+  if (item.url) {
+    content = await extractArticleContent(item.url).catch((e) => {
+      assistant.error(`processFeedSource(): Failed to extract content from ${item.url} (using summary)`, e);
+      return '';
+    });
+  }
+  if (!content || content.length < 100) {
+    content = item.content || item.summary || '';
+  }
+  return { item, content };
+}
+// ── Parent source processing ─────────────────────────────────────────
+async function processParentSource(assistant, entry, admin, Manager) {
+  const parentUrl = Manager?.getParentApiUrl?.();
+  if (!parentUrl) {
+    assistant.log('processParentSource(): No parent URL configured');
+    return null;
+  }
+  const categories = entry.categories || [];
+  const allSources = [];
+  const categoriesToFetch = categories.length ? categories : [''];
+  for (const category of categoriesToFetch) {
+    const query = {
+      limit: 3,
+      backendManagerKey: process.env.BACKEND_MANAGER_KEY,
+    };
+    if (category) {
+      query.category = category;
+    }
+    const data = await fetch(`${parentUrl}/newsletter-sources`, {
+      method: 'get',
+      response: 'json',
+      timeout: 60000,
+      query: query,
+    }).catch((e) => {
+      assistant.error(`processParentSource(): Failed to fetch sources for category=${category}: ${e.message}`);
+      return null;
+    });
+    if (data?.sources?.length) {
+      allSources.push(...data.sources);
+    }
+  }
+  if (!allSources.length) {
+    assistant.log('processParentSource(): No sources available from parent');
+    return null;
+  }
+  if (admin) {
+    const usedUrls = new Set();
+    const snapshot = await admin.firestore()
+      .collection(CONTENT_SOURCES_COLLECTION)
+      .where('origin', '==', '$parent')
+      .select('url')
+      .get()
+      .catch((e) => {
+        assistant.error('processParentSource(): Error querying tracked sources (continuing)', e);
+        return { docs: [] };
+      });
+    snapshot.docs.forEach((doc) => {
+      const data = doc.data();
+      if (data.url) {
+        usedUrls.add(data.url);
+      }
+    });
+    const available = allSources.filter((s) => !usedUrls.has(s.url || s.id));
+    if (!available.length) {
+      assistant.log('processParentSource(): All parent sources already used');
+      return null;
+    }
+    return available;
+  }
+  return allSources;
+}
+// ── Parent source fetching for newsletter ────────────────────────────
+async function fetchParentSources(parentUrl, categories, assistant) {
+  const allSources = [];
+  for (const category of categories) {
+    try {
+      const data = await fetch(`${parentUrl}/newsletter-sources`, {
+        method: 'get',
+        response: 'json',
+        timeout: 60000,
+        query: {
+          category,
+          limit: 3,
+          backendManagerKey: process.env.BACKEND_MANAGER_KEY,
+        },
+      });
+      if (data.sources?.length) {
+        allSources.push(...data.sources);
+      }
+    } catch (e) {
+      assistant.error(`fetchParentSources(): Failed to fetch ${category} sources: ${e.message}`);
+    }
+  }
+  return allSources;
+}
+// ── Feed-to-newsletter normalization ─────────────────────────────────
+function normalizeFeedItemForNewsletter(feedResult, feedUrl, categories) {
+  const item = feedResult.item;
+  let hostname = '';
+  try { hostname = new URL(feedUrl).hostname; } catch (e) { /* ignore */ }
+  return {
+    id: item.id || item.url || '',
+    title: item.title || '',
+    subject: item.title || '',
+    category: (categories && categories[0]) || '',
+    categories: categories || [],
+    url: item.url || '',
+    source: {
+      from: hostname,
+      subject: item.title || '',
+      content: feedResult.content || item.content || item.summary || '',
+    },
+    ai: null,
+  };
+}
+// ── Resolve newsletter sources (feeds + parent + fallback) ───────────
+async function resolveNewsletterSources({ sources, categories, admin, Manager, assistant }) {
+  const resolved = [];
+  const feedUrls = (sources || []).filter((s) => typeof s === 'string' && s.startsWith(FEED_PREFIX));
+  const hasParent = (sources || []).includes('$parent');
+  const brandId = Manager?.config?.brand?.id || '';
+  for (const source of feedUrls) {
+    const feedUrl = source.slice(FEED_PREFIX.length);
+    const feedResult = await processFeedSource(assistant, feedUrl, brandId, admin);
+    if (feedResult) {
+      resolved.push(normalizeFeedItemForNewsletter(feedResult, feedUrl, categories));
+    }
+  }
+  if (hasParent) {
+    const parentUrl = Manager?.getParentApiUrl?.();
+    if (parentUrl) {
+      const parentSources = await fetchParentSources(parentUrl, categories || [], assistant);
+      resolved.push(...parentSources);
+    }
+  }
+  return resolved;
+}
+// ── Firestore tracking ──────────────────────────────────────────────
+async function trackContentSource(admin, { url, origin, feedUrl, itemId, itemTitle, usedBy, brandId, postUrl, postSlug }) {
+  const docId = contentSourceHash(origin || '', url || '');
+  const nowISO = new Date().toISOString();
+  const nowUNIX = Math.round(Date.now() / 1000);
+  await admin.firestore().doc(`${CONTENT_SOURCES_COLLECTION}/${docId}`).set({
+    url: url || null,
+    origin: origin || null,
+    feedUrl: feedUrl || null,
+    itemId: itemId || null,
+    itemTitle: itemTitle || null,
+    usedBy: usedBy || null,
+    brandId: brandId || null,
+    postUrl: postUrl || null,
+    postSlug: postSlug || null,
+    metadata: {
+      created: { timestamp: nowISO, timestampUNIX: nowUNIX },
+      updated: { timestamp: nowISO, timestampUNIX: nowUNIX },
+    },
+  });
+}
+function contentSourceHash(origin, url) {
+  return crypto.createHash('sha256').update(`${origin}::${url}`).digest('hex').slice(0, 20);
+}
+async function getProcessedItemIds(admin, feedUrl) {
+  if (!admin) {
+    return new Set();
+  }
+  const snapshot = await admin.firestore()
+    .collection(CONTENT_SOURCES_COLLECTION)
+    .where('feedUrl', '==', feedUrl)
+    .select('itemId', 'url')
+    .get();
+  const ids = new Set();
+  snapshot.docs.forEach((doc) => {
+    const data = doc.data();
+    if (data.itemId) {
+      ids.add(data.itemId);
+    }
+    if (data.url) {
+      ids.add(data.url);
+    }
+  });
+  return ids;
+}
+// ── URL content extraction ──────────────────────────────────────────
+function getURLContent(url) {
+  return fetch(url, {
+    timeout: 120000,
+    tries: 3,
+    response: 'raw',
+    headers: { 'User-Agent': USER_AGENT },
+  })
+  .then(async (res) => {
+    const contentType = res.headers.get('content-type');
+    const text = await res.text();
+    return extractBodyContent(text, contentType, url);
+  });
+}
+function isURL(url) {
+  try {
+    return !!new URL(url);
+  } catch (e) {
+    return false;
+  }
+}
+function extractBodyContent(text) {
+  const parsed = tryParse(text);
+  if (parsed) {
+    if (parsed.items) {
+      return parsed.items.map((i) => `${i.title}: ${i.content_text}`).join('\n');
+    }
+    return text;
+  }
+  const bodyMatch = text.match(/<body[^>]*>([\s\S]*?)<\/body>/i);
+  if (!bodyMatch) {
+    return '';
+  }
+  let bodyContent = bodyMatch[1];
+  bodyContent = bodyContent.replace(/<script[^>]*>[\s\S]*?<\/script>/gi, '');
+  bodyContent = bodyContent.replace(/<meta[^>]*>/gi, '');
+  return bodyContent.replace(/<[^>]*>/g, ' ').replace(/\s+/g, ' ').trim();
+}
+function tryParse(json) {
+  try {
+    return require('json5').parse(json);
+  } catch (e) {
+    return null;
+  }
+}
+// ── Exports ─────────────────────────────────────────────────────────
+module.exports = {
+  PROMPT_SOURCE,
+  PROMPT,
+  ANTI_TRACEABILITY,
+  FEED_PREFIX,
+  CONTENT_SOURCES_COLLECTION,
+  USER_AGENT,
+  processFeedSource,
+  processParentSource,
+  fetchParentSources,
+  resolveNewsletterSources,
+  normalizeFeedItemForNewsletter,
+  trackContentSource,
+  contentSourceHash,
+  getProcessedItemIds,
+  getURLContent,
+  isURL,
+  extractBodyContent,
+};

package/src/manager/libraries/email/generators/newsletter.js CHANGED Viewed

@@ -42,7 +42,7 @@ const { renderMarkdown } = require('./lib/markdown-renderer.js');
 const { uploadAssets, RAW_BASE } = require('./lib/image-host.js');
 const { buildPublicConfig } = require('../../../routes/brand/get.js');
 const { writeArticle, publishArticle } = require('../../../libraries/content/ghostii.js');
-const { trackContentSource, contentSourceHash } = require('../../../events/cron/daily/blog-auto-publisher.js');
+const { trackContentSource, contentSourceHash, resolveNewsletterSources } = require('../../../libraries/content/source-resolver.js');
 /**
  * Generate newsletter content from parent server sources.
@@ -99,24 +99,22 @@ async function generate(Manager, assistant, settings, opts = {}) {
   let sources = opts.sources;
   if (!sources) {
-    // For now, newsletter resolves $parent sources via the parent server
-    if (contentSources.includes('$parent')) {
-      const categories = config.categories || [];
+    const categories = config.categories || [];
-      if (!categories.length) {
-        assistant.log('Newsletter generator: no categories configured');
-        return null;
-      }
-      const parentUrl = Manager.getParentApiUrl();
+    if (!categories.length) {
+      assistant.log('Newsletter generator: no categories configured');
+      return null;
+    }
-      if (!parentUrl) {
-        assistant.log('Newsletter generator: no parent URL configured');
-        return null;
-      }
+    const admin = Manager.libraries?.admin;
-      sources = await fetchSources(parentUrl, categories, assistant);
-    }
+    sources = await resolveNewsletterSources({
+      sources: contentSources,
+      categories,
+      admin,
+      Manager,
+      assistant,
+    });
   }
   if (!sources?.length) {
@@ -402,9 +400,11 @@ async function generate(Manager, assistant, settings, opts = {}) {
   const admin = Manager.libraries?.admin;
   if (admin) {
     for (const source of sources) {
+      const origin = source.source?.from?.startsWith?.('http') ? `$feed:${source.source.from}` : '$parent';
       await trackContentSource(admin, {
         url: source.url || source.id,
-        origin: '$parent',
+        origin,
+        feedUrl: origin.startsWith('$feed:') ? origin.slice(6) : undefined,
         itemId: source.id,
         itemTitle: source.title || '',
         usedBy: 'newsletter',
@@ -696,35 +696,7 @@ function aggregateTotals(filterMeta, structureMeta, images) {
   };
 }
-/**
- * Fetch ready newsletter sources from the parent server.
- */
-async function fetchSources(parentUrl, categories, assistant) {
-  const allSources = [];
-  for (const category of categories) {
-    try {
-      const data = await fetch(`${parentUrl}/newsletter-sources`, {
-        method: 'get',
-        response: 'json',
-        timeout: 60000,
-        query: {
-          category,
-          limit: 3,
-          backendManagerKey: process.env.BACKEND_MANAGER_KEY,
-        },
-      });
-      if (data.sources?.length) {
-        allSources.push(...data.sources);
-      }
-    } catch (e) {
-      assistant.error(`Newsletter generator: Failed to fetch ${category} sources: ${e.message}`);
-    }
-  }
-  return allSources;
-}
+// fetchSources() removed — replaced by resolveNewsletterSources() from source-resolver.js
 /**

package/test/email/newsletter-generate.js CHANGED Viewed

@@ -41,6 +41,11 @@
  *   NEWSLETTER_OPEN=1                  Auto-open the rendered preview in the default browser (macOS only).
  *
  *   --- AI-mode-only env vars (require TEST_EXTENDED_MODE=1) ---
+ *   NEWSLETTER_SOURCE=<type>           Override the source type for this run. Bypasses parent-server
+ *                                       fetch and lets the generator's built-in resolver run.
+ *                                       Values: '$feed:<url>', '$parent'. When set, sources[] in the
+ *                                       config is replaced with this single source.
+ *                                       Example: NEWSLETTER_SOURCE='$feed:https://feeds.arstechnica.com/arstechnica/index'
  *   NEWSLETTER_PEEK=1                  Fetch + list ready sources, do not claim, exit.
  *   NEWSLETTER_SOURCE_ID=<id>          Generate from one specific source WITHOUT claiming it.
  *   NEWSLETTER_LIMIT=10                Sources per category for PEEK (default 10).
@@ -325,60 +330,57 @@ module.exports = {
     // `api.` subdomain (e.g. 'https://itwcreativeworks.com'); the helper
     // inserts `api.` at call time. PARENT_API_URL env override is honored
     // verbatim for one-off testing against a different parent.
-    const parentUrl = env.PARENT_API_URL || Manager.getParentApiUrl();
-    assert.ok(parentUrl, 'PARENT_API_URL (env) or parent (config) must be set for the AI pipeline. Set TEST_EXTENDED_MODE=1 to run it, or omit TEST_EXTENDED_MODE for the fast fixture preview.');
+    // When NEWSLETTER_SOURCE is set, skip parent URL checks and source fetching —
+    // the generator's built-in resolver (resolveNewsletterSources) handles everything.
+    let sources = [];
+    if (!env.NEWSLETTER_SOURCE) {
+      const parentUrl = env.PARENT_API_URL || Manager.getParentApiUrl();
+      assert.ok(parentUrl, 'PARENT_API_URL (env) or parent (config) must be set for the AI pipeline. Set TEST_EXTENDED_MODE=1 to run it, or omit TEST_EXTENDED_MODE for the fast fixture preview.');
+      // --- Peek mode (early return) ---
+      if (env.NEWSLETTER_PEEK) {
+        const peeked = await peekSources({
+          parentUrl,
+          categories: newsletterConfig.categories || [],
+          limit: parseInt(env.NEWSLETTER_LIMIT, 10) || 10,
+          key: env.BACKEND_MANAGER_KEY,
+        });
+        console.log(`\nPeek mode — ${peeked.length} ready source(s):\n`);
+        peeked.forEach((s, i) => {
+          const raw = s.source || {};
+          const cats = (s.categories || []).join(', ') || s.category || '(none)';
+          console.log(`[${i + 1}] ${s.id}`);
+          console.log(`    Categories: ${cats}`);
+          console.log(`    From:       ${raw.from || s.from || '(unknown)'}`);
+          console.log(`    Subject:    ${raw.subject || s.subject || '(none)'}`);
+          console.log(`    Headline:   ${s.ai?.headline || '(none — not yet AI-processed)'}`);
+          console.log('');
+        });
+        assert.ok(true, `Peeked ${peeked.length} sources`);
+        return;
+      }
-    // --- Peek mode (early return) ---
-    if (env.NEWSLETTER_PEEK) {
-      const sources = await peekSources({
+      // --- Fetch sources ---
+      const claim = !!env.NEWSLETTER_CLAIM;
+      sources = await fetchSourcesForRun({
         parentUrl,
-        categories: newsletterConfig.categories || [],
-        limit: parseInt(env.NEWSLETTER_LIMIT, 10) || 10,
+        newsletterConfig,
+        brandId: config.brand?.id,
+        sourceId: env.NEWSLETTER_SOURCE_ID,
         key: env.BACKEND_MANAGER_KEY,
+        claim,
       });
-      console.log(`\nPeek mode — ${sources.length} ready source(s):\n`);
-      sources.forEach((s, i) => {
-        const raw = s.source || {};
-        const cats = (s.categories || []).join(', ') || s.category || '(none)';
-        console.log(`[${i + 1}] ${s.id}`);
-        console.log(`    Categories: ${cats}`);
-        console.log(`    From:       ${raw.from || s.from || '(unknown)'}`);
-        console.log(`    Subject:    ${raw.subject || s.subject || '(none)'}`);
-        console.log(`    Headline:   ${s.ai?.headline || '(none — not yet AI-processed)'}`);
-        console.log('');
-      });
-      assert.ok(true, `Peeked ${sources.length} sources`);
-      return;
-    }
-    // --- Fetch sources ---
-    // By DEFAULT the test does NOT claim sources — it fetches them without
-    // claimFor, so they stay 'ready' and available for the real newsletter.
-    // This keeps test runs repeatable and non-destructive (a test should never
-    // silently consume production resources). Set NEWSLETTER_CLAIM=1 to exercise
-    // the real claim/consume path (then use NEWSLETTER_RELEASE=1 to put them back).
-    const claim = !!env.NEWSLETTER_CLAIM;
-    const sources = await fetchSourcesForRun({
-      parentUrl,
-      newsletterConfig,
-      brandId: config.brand?.id,
-      sourceId: env.NEWSLETTER_SOURCE_ID,
-      key: env.BACKEND_MANAGER_KEY,
-      claim,
-    });
-    // Environmental precondition: the parent server must have ready sources in
-    // at least one configured category. Skip cleanly when the pool is empty
-    // (transient state — no point hard-failing CI on an external queue).
-    if (sources.length === 0) {
-      return skip('No ready newsletter sources available on parent server (environmental)');
-    }
+      if (sources.length === 0) {
+        return skip('No ready newsletter sources available on parent server (environmental)');
+      }
-    // Track claimed IDs for later --release-all (only when we actually claimed)
-    if (claim && !env.NEWSLETTER_SOURCE_ID) {
-      appendClaimed(claimedFile, sources.map((s) => s.id));
+      if (claim && !env.NEWSLETTER_SOURCE_ID) {
+        appendClaimed(claimedFile, sources.map((s) => s.id));
+      }
     }
     // Force `newsletter.enabled: true` and inject the per-run newsletter config
@@ -432,12 +434,25 @@ module.exports = {
     console.log(`[extended] uploading to itw-creative-works/newsletter-assets/${config.brand?.id}/${campaignId || '<auto-id>'}/ + Beehiiv draft`);
+    // NEWSLETTER_SOURCE overrides the configured sources[] and lets the
+    // generator's built-in resolver run (resolveNewsletterSources). Without
+    // this, the test always passes pre-fetched parent sources via opts.sources,
+    // bypassing feed resolution entirely.
+    //   NEWSLETTER_SOURCE='$feed:https://feeds.arstechnica.com/arstechnica/index'
+    //   NEWSLETTER_SOURCE='$parent'   (default behavior, explicit)
+    const useResolverSources = !!env.NEWSLETTER_SOURCE;
+    if (useResolverSources) {
+      newsletterConfig.sources = [env.NEWSLETTER_SOURCE];
+      Manager.config.marketing.newsletter.content = newsletterConfig;
+    }
     const result = await generator.generate(
       Manager,
       assistant,
-      { name: `Somiibo Newsletter — Iteration ${stamp}` },
+      { name: `${config.brand?.name || 'Brand'} Newsletter — Iteration ${stamp}` },
       {
-        sources,
+        sources: useResolverSources ? undefined : sources,
         skipClaim: true, // We manage the claim/release lifecycle ourselves
         skipImages: !!env.NEWSLETTER_NO_IMAGES,
         // The article is GENERATED whenever the brand's config.article.enabled is on