backend-manager 5.8.3 → 5.9.1

package/CHANGELOG.md

@@ -14,6 +14,37 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+# [5.9.1] - 2026-06-20
+
+### Fixed
+- Missing `require('node-powertools')` in newsletter generator — broke `powertools.arrayify()` for array content support added in v5.9.0
+
+# [5.9.0] - 2026-06-19
+
+### BREAKING
+- **Config key `ghostii` renamed to `blog`.** Consumer projects must migrate their `backend-manager-config.json`: `ghostii[]` → `blog.content[]` with `blog.enabled` and `blog.platform: 'ghostii'`.
+- **Field renames in blog content entries.** `articles` → `quantity`, `prompt` → `instructions`, `$app` → `$brand`.
+- **Newsletter `claimSources()` removed.** Parent server now serves sources read-only (no claiming). Children track usage locally in `content-sources`.
+
+### Added
+- **Provider-based blog architecture.** `blog.platform` selects the article-generation provider (dispatched via `require(../content/${platform}.js)`). Only `ghostii` for now.
+- **`$parent` source type.** Blog entries can fetch sources from the parent server's pool, filtered by categories, with local dedup tracking.
+- **`$brand` source type.** Renamed from `$app` — generic brand-topic generation.
+- **New content entry fields.** `tone`, `categories`, `keywords` added to both blog and newsletter content entries. Injected into AI prompts for better content targeting.
+- **`links` and `keywords` support in newsletter.** Newsletter AI prompt now weaves in brand links and SEO keywords from content config.
+- **`sources` field on newsletter content.** Newsletter entries now explicitly declare source types (default: `['$parent']`), matching blog config shape.
+- **Newsletter `content` supports array format.** Matches blog config shape for structural parity.
+- **Unified `content-sources` Firestore collection.** Both blog and newsletter track used sources in the same collection with `usedBy` field (`'blog'` or `'newsletter'`).
+
+### Changed
+- **Cron renamed.** `ghostii-auto-publisher.js` → `blog-auto-publisher.js`.
+- **`postPath` defaults to platform name.** Instead of hardcoded `'blog'`, defaults to the platform value (e.g. `'ghostii'`).
+- **Consistent field ordering.** `sources → categories → links → instructions → tone → keywords` across blog and newsletter config templates.
+
+### Removed
+- **`ghostii-auto-publisher.js`** — replaced by `blog-auto-publisher.js`.
+- **`claimSources()` from newsletter generator** — parent no longer tracks claim state.
+
 # [5.8.3] - 2026-06-19
 
 ### Fixed

package/CLAUDE.md

@@ -162,7 +162,7 @@ Deep references live in `docs/`. **Whenever you make a behavioral change, update
 
 ### Subsystems & Libraries
 
-- [docs/ghostii.md](docs/ghostii.md) — Ghostii article publisher: source types (`$app` / `$feed:` / URL / text), per-entry API overrides, RSS/Atom feed parser, Firestore feed-item tracking, `sourceContent` pass-through to Ghostii API
+- [docs/ghostii.md](docs/ghostii.md) — Blog auto-publisher (Ghostii provider): source types (`$brand` / `$feed:` / `$parent` / URL / text), provider-based architecture, per-entry API overrides, RSS/Atom feed parser, unified `content-sources` Firestore tracking, `sourceContent` pass-through to Ghostii API
 - [docs/email-system.md](docs/email-system.md) — unified MJML email rendering pipeline: shared preparation (`prepare.js`), composable template system (`base.js` blocks), 4 email templates (card, plain, order, feedback), no SendGrid dynamic templates — everything rendered server-side
 - [docs/usage-rate-limiting.md](docs/usage-rate-limiting.md) — usage tracking, monthly/daily caps, `setUser()` + mirrors for proxy usage, reset schedule
 - [docs/ai-library.md](docs/ai-library.md) — `Manager.AI()` unified entry for OpenAI + Anthropic (text via `.request()`, images via `.image()` → `gpt-image-2`)

package/docs/admin-post-route.md

@@ -2,7 +2,7 @@
 
 The `POST /admin/post` route creates blog posts via GitHub's API. It handles image extraction, resize, upload, and body rewriting.
 
-**Consumers.** Besides direct admin/blogger HTTP calls, this route is the publish target for the **Ghostii article engine** (`src/manager/libraries/content/ghostii.js` → `publishArticle()`), which is used by two paths: the standalone daily `ghostii-auto-publisher.js` cron (off by default), and the newsletter generator's linked-article flow (`marketing.newsletter.content.article.enabled`). Both POST `title`/`url`/`description`/`headerImageURL`/`body`/`author`/`categories`/`tags`/`postPath`/`githubUser`/`githubRepo` with a `backendManagerKey`. Note: `headerImageURL` MUST resolve to a `.jpg` (the downloader rejects non-`.jpg` headers); Ghostii's `unsplash` hero satisfies this. See [docs/marketing-campaigns.md](marketing-campaigns.md).
+**Consumers.** Besides direct admin/blogger HTTP calls, this route is the publish target for the **Ghostii article engine** (`src/manager/libraries/content/ghostii.js` → `publishArticle()`), which is used by two paths: the standalone daily `blog-auto-publisher.js` cron (off by default), and the newsletter generator's linked-article flow (`marketing.newsletter.content.article.enabled`). Both POST `title`/`url`/`description`/`headerImageURL`/`body`/`author`/`categories`/`tags`/`postPath`/`githubUser`/`githubRepo` with a `backendManagerKey`. Note: `headerImageURL` MUST resolve to a `.jpg` (the downloader rejects non-`.jpg` headers); Ghostii's `unsplash` hero satisfies this. See [docs/marketing-campaigns.md](marketing-campaigns.md).
 
 **Ghostii is unopinionated about BEM.** Its `/write/article` response is a generic article — a `json` block array (`[{ name, content }]` where name ∈ `heading-1..6`/`image`/`paragraph`/`blockquote`/`list`) plus top-level `title`/`description`/`headerImageUrl`/`images`/`categories`/`keywords`. BEM owns the transform into this route's shape: `blocksToPost(article.json)` extracts the `heading-1` as the title, the first `image` block as `headerImageURL`, and joins every remaining block as the `body` (content only — NO title, NO header image embedded, since this route adds those itself; section images stay in the body and are extracted normally). Older Ghostii responses without `json` fall back to the flat `article.{title,body,headerImageUrl}` fields.
 

package/docs/ghostii.md

@@ -1,8 +1,8 @@
-# Ghostii Article Publisher
+# Blog Auto-Publisher (Ghostii Provider)
 
-The Ghostii system generates and publishes blog posts via the daily `bm_cronDaily` cron job. Two independent paths:
+The blog system generates and publishes blog posts via the daily `bm_cronDaily` cron job. Two independent paths:
 
-1. **Standalone publisher** — `config.ghostii[]` entries: daily cron picks sources, calls the Ghostii AI API, publishes via `admin/post`.
+1. **Standalone publisher** — `config.blog` with `enabled: true`: daily cron picks sources, calls the configured platform provider (default: Ghostii AI API), publishes via `admin/post`.
 2. **Newsletter-linked articles** — `marketing.newsletter.content.article.enabled`: the newsletter generator expands its lead section into a full blog post via Ghostii.
 
 This doc covers the standalone publisher. For newsletter integration, see [marketing-campaigns.md](marketing-campaigns.md).
@@ -10,59 +10,71 @@ This doc covers the standalone publisher. For newsletter integration, see [marke
 ## Architecture
 
 ```
-config.ghostii[] entry
-  ↓
-ghostii-auto-publisher cron (daily)
-  ↓ (per entry, per article)
-resolveSource() — detect source type, fetch/parse if needed
-  ↓
-writeArticle() — call Ghostii AI API (api.ghostii.ai/write/article)
-  ↓
-publishArticle() — POST to admin/post → GitHub commit
-  ↓
-trackFeedItem() — Firestore (for $feed: sources only)
+config.blog.content[] entry
+  |
+blog-auto-publisher cron (daily)
+  | (per entry, per article)
+resolveSource() -- detect source type, fetch/parse if needed
+  |
+provider.writeArticle() -- call provider API (e.g. Ghostii: api.ghostii.ai/write/article)
+  |
+provider.publishArticle() -- POST to admin/post -> GitHub commit
+  |
+trackContentSource() -- Firestore (for $feed:, $parent sources)
 ```
 
 ### Key files
 
 | File | Purpose |
 |---|---|
-| `src/manager/libraries/content/ghostii.js` | `writeArticle()`, `publishArticle()`, `blocksToPost()` — API client + post transform |
-| `src/manager/libraries/content/feed-parser.js` | `parseFeed()`, `extractArticleContent()` — RSS/Atom/JSON parser + article extractor |
-| `src/manager/events/cron/daily/ghostii-auto-publisher.js` | Daily cron: source resolution, feed processing, Firestore tracking |
+| `src/manager/libraries/content/ghostii.js` | `writeArticle()`, `publishArticle()`, `blocksToPost()` -- Ghostii API client + post transform |
+| `src/manager/libraries/content/feed-parser.js` | `parseFeed()`, `extractArticleContent()` -- RSS/Atom/JSON parser + article extractor |
+| `src/manager/events/cron/daily/blog-auto-publisher.js` | Daily cron: source resolution, feed processing, Firestore tracking, provider dispatch |
 | `src/manager/routes/admin/post/post.js` | Publishing endpoint: image download + resize, GitHub commit |
 
 ## Source Types
 
-Each `ghostii[]` entry has a `sources` array. The cron picks one at random per article.
+Each `blog.content[]` entry has a `sources` array. The cron picks one at random per article.
 
 | Source | Format | Behavior |
 |---|---|---|
-| Generic topic | `'$app'` | Prompts Ghostii to write about any topic relevant to the brand |
-| RSS/Atom feed | `'$feed:https://example.com/feed/'` | Parses feed, picks one unprocessed article, extracts content, passes as `sourceContent` to Ghostii |
+| Generic topic | `'$brand'` | Prompts the provider to write about any topic relevant to the brand |
+| RSS/Atom feed | `'$feed:https://example.com/feed/'` | Parses feed, picks one unprocessed article, extracts content, passes as `sourceContent` to the provider |
+| Parent server | `'$parent'` | Fetches sources from parent server's source pool (without claiming), filters out already-used sources locally |
 | URL | `'https://example.com/page'` | Fetches page HTML, extracts body text, uses as prompt seed |
 | Text | `'Write about blockchain'` | Uses directly as prompt seed |
 
 ### Feed source flow
 
 1. Fetch and parse the RSS/Atom/JSON feed via `feed-parser.parseFeed()`
-2. Query `ghostii-sources` in Firestore for already-processed items
+2. Query `content-sources` in Firestore for already-processed items
 3. Filter to unprocessed items, pick the newest
 4. Extract full article content from the item URL via `feed-parser.extractArticleContent()`
-5. Pass extracted text as `sourceContent` to the Ghostii API (separate from the `description` prompt)
-6. After publish, write a tracking doc to `ghostii-sources/{hash}`
-7. On failure (feed unreachable, unparseable, exhausted): fall back to `$app` behavior
+5. Pass extracted text as `sourceContent` to the provider API (separate from the `description` prompt)
+6. After publish, write a tracking doc to `content-sources/{hash}`
+7. On failure (feed unreachable, unparseable, exhausted): fall back to `$brand` behavior
 
-### Feed item tracking
+### Parent source flow
 
-Collection: `ghostii-sources` (consumer project Firestore)
+1. Fetch available sources from parent server via `GET {parentUrl}/newsletter-sources` (no `claimFor` -- read-only)
+2. Query `content-sources` in Firestore for already-used sources with `origin: '$parent'`
+3. Filter out already-used sources, pick the first available
+4. On failure (no parent URL, no sources, all exhausted): fall back to `$brand` behavior
+
+### Content source tracking
+
+Collection: `content-sources` (consumer project Firestore)
+
+Unified tracker for all source types -- shared between blog-auto-publisher and newsletter generator.
 
 ```js
-ghostii-sources/{sha256(feedUrl + '::' + itemId).slice(0,20)}: {
-  feedUrl: 'https://...',
+content-sources/{sha256(origin + '::' + url).slice(0,20)}: {
+  url: 'https://...',            // unique identifier (article URL or source ID)
+  origin: '$feed:https://...',   // or '$parent', '$brand'
+  feedUrl: 'https://...',        // for feed sources
   itemId: 'guid-or-url',
-  itemUrl: 'https://...',
   itemTitle: '...',
+  usedBy: 'blog',               // or 'newsletter'
   brandId: '...',
   postUrl: '...',
   postSlug: '...',
@@ -70,7 +82,6 @@ ghostii-sources/{sha256(feedUrl + '::' + itemId).slice(0,20)}: {
     created: { timestamp: '...ISO...', timestampUNIX: 1234567890 },
     updated: { timestamp: '...ISO...', timestampUNIX: 1234567890 },
   },
-  metadata: { created: Timestamp },
 }
 ```
 
@@ -78,53 +89,67 @@ ghostii-sources/{sha256(feedUrl + '::' + itemId).slice(0,20)}: {
 
 ```js
 // backend-manager-config.json
-ghostii: [
-  {
-    articles: 1,                    // articles per cron run (0 = disabled)
-    sources: ['$app'],              // source pool (mixed types OK)
-    links: [],                      // URLs to inject into article body
-    prompt: '',                     // custom instructions for the AI
-    chance: 1.0,                    // probability of running (0-1)
-    author: null,                   // author slug (null = admin/post default)
-    postPath: 'ghostii',            // sub-folder under _posts/{year}/
-    overrides: {                    // per-entry Ghostii API param overrides
-      // keywords: ['AI'],
-      // length: 'long',            // short | medium | long | comprehensive
-      // research: true,            // web search for real links
-      // insertImages: true,
-      // insertLinks: true,
-      // headerImageUrl: 'unsplash', // disabled | unsplash | generate
-      // maxLinks: 6,
-      // sectionQuantity: 5,
-      // feedUrl: '',               // brand blog feed (title dedup)
-    },
-    // brand: 'other-brand-id',     // cross-brand publishing
-    // brandUrl: 'https://api.other.com',
-  }
-]
+blog: {
+  enabled: false,
+  platform: 'ghostii',           // provider: only 'ghostii' for now
+  content: [
+    {
+      quantity: 1,                // articles per cron run (0 = disabled)
+      sources: ['$brand'],        // source pool (mixed types OK)
+      links: [],                  // URLs to inject into article body
+      instructions: '',           // custom instructions for the AI
+      tone: 'professional',       // 'professional', 'casual', 'actionable', etc.
+      categories: [],             // topic constraints + output tags
+      keywords: [],               // SEO keywords to weave in
+      chance: 1.0,                // probability of running (0-1)
+      author: null,               // author slug (null = admin/post default)
+      postPath: 'blog',           // sub-folder under _posts/{year}/
+      overrides: {                // per-entry Ghostii API param overrides
+        // keywords: ['AI'],
+        // length: 'long',        // short | medium | long | comprehensive
+        // research: true,        // web search for real links
+        // insertImages: true,
+        // insertLinks: true,
+        // headerImageUrl: 'unsplash', // disabled | unsplash | generate
+        // maxLinks: 6,
+        // sectionQuantity: 5,
+        // feedUrl: '',           // brand blog feed (title dedup)
+      },
+      // brand: 'other-brand-id',     // cross-brand publishing
+      // brandUrl: 'https://api.other.com',
+    }
+  ],
+}
 ```
 
 ### Per-entry overrides
 
 All Ghostii API params were previously hardcoded in `writeArticle()`. They now accept per-entry overrides via the `overrides` object. Unspecified values fall back to framework defaults (length: 'long', research: true, etc.).
 
-### `sourceContent` — Ghostii API field
+### Keywords resolution
+
+Keywords are resolved in this order:
+1. `overrides.keywords` -- per-entry Ghostii API override (highest priority)
+2. `entry.keywords` -- top-level entry keywords (used if overrides.keywords is not set)
+3. Empty array -- framework default
+
+### `sourceContent` -- Ghostii API field
 
-When a `$feed:` source provides extracted article text, it's passed to the Ghostii API as `sourceContent` (max 16KB). This is separate from `description` (the editorial brief, max 2KB). Ghostii's title generation, outline, and section writing prompts reference the source content as context for writing original articles — not rewrites.
+When a `$feed:` or `$parent` source provides extracted article text, it's passed to the Ghostii API as `sourceContent` (max 16KB). This is separate from `description` (the editorial brief, max 2KB). Ghostii's title generation, outline, and section writing prompts reference the source content as context for writing original articles -- not rewrites.
 
 ## Feed Parser
 
 `src/manager/libraries/content/feed-parser.js` exports:
 
-- **`parseFeed(text)`** — Parse RSS 2.0, Atom 1.0, or JSON Feed into `{ items: [{ id, title, url, summary, content, publishedAt }] }`. Handles CDATA, namespaced elements (`content:encoded`), BOM, attribute-based links. Returns `{ items: [] }` on invalid input.
+- **`parseFeed(text)`** -- Parse RSS 2.0, Atom 1.0, or JSON Feed into `{ items: [{ id, title, url, summary, content, publishedAt }] }`. Handles CDATA, namespaced elements (`content:encoded`), BOM, attribute-based links. Returns `{ items: [] }` on invalid input.
 
-- **`extractArticleContent(url)`** — Fetch URL, extract readable text from `<article>` → `<main>` → `<body>`, strip scripts/styles/nav/footer/header/aside, normalize whitespace, truncate to 14KB.
+- **`extractArticleContent(url)`** -- Fetch URL, extract readable text from `<article>` -> `<main>` -> `<body>`, strip scripts/styles/nav/footer/header/aside, normalize whitespace, truncate to 14KB.
 
 ## Tests
 
 ```bash
-npx mgr test bem:helpers/content/feed-parser         # 32 tests: RSS/Atom/JSON parsing, edge cases, HTML extraction
+npx mgr test bem:helpers/content/feed-parser          # 32 tests: RSS/Atom/JSON parsing, edge cases, HTML extraction
 npx mgr test bem:helpers/content/ghostii-write-article # 15 tests: override pass-through, sourceContent, backwards compat
-npx mgr test bem:helpers/content/ghostii-auto-publisher # 20 tests: source detection, feed processing, Firestore tracking
+npx mgr test bem:helpers/content/blog-auto-publisher   # 25+ tests: source detection, feed processing, Firestore tracking
 npx mgr test bem:helpers/content/ghostii-blocks        # 8 tests: blocksToPost() (unchanged)
 ```

package/docs/marketing-campaigns.md

@@ -174,7 +174,7 @@ Flow (runs in parallel with SVG generation, between structure and render):
 
 **Failure is isolated** — if the article build throws, it resolves to `null`, no CTA is injected, and the newsletter ships normally. The newsletter never depends on the article succeeding.
 
-The published URL is surfaced on the return as `assets.articleUrl` and `meta.article`. Config block: `marketing.newsletter.content.article = { enabled, author }` (`enabled` default `false`; `author` is the post author slug). Standalone article publishing (independent of the newsletter) still lives in the daily `ghostii-auto-publisher.js` cron, which now shares the same `libraries/content/ghostii.js` engine — see [docs/admin-post-route.md](admin-post-route.md). Standalone Ghostii is **disabled by default** (`ghostii[0].articles: 0`).
+The published URL is surfaced on the return as `assets.articleUrl` and `meta.article`. Config block: `marketing.newsletter.content.article = { enabled, author }` (`enabled` default `false`; `author` is the post author slug). Standalone article publishing (independent of the newsletter) lives in the daily `blog-auto-publisher.js` cron, which uses the same `libraries/content/ghostii.js` engine — see [docs/admin-post-route.md](admin-post-route.md). Standalone blog publishing is **disabled by default** (`blog.enabled: false`).
 
 ## Template-owned schemas
 
@@ -360,7 +360,7 @@ marketing: {
 | Newsletter asset host (GitHub upload — PNGs + newsletter.html + newsletter.md + summary.md) | `src/manager/libraries/email/generators/lib/image-host.js` |
 | Newsletter markdown renderer (programmatic, no AI) | `src/manager/libraries/email/generators/lib/markdown-renderer.js` |
 | Ghostii article engine (writeArticle + publishArticle) | `src/manager/libraries/content/ghostii.js` |
-| Standalone Ghostii article cron (off by default) | `src/manager/events/cron/daily/ghostii-auto-publisher.js` |
+| Standalone blog article cron (off by default) | `src/manager/events/cron/daily/blog-auto-publisher.js` |
 | Unified AI library | `src/manager/libraries/ai/index.js` (OpenAI + Anthropic via `Manager.AI(assistant).request({ provider, ... })`) |
 | Notification library | `src/manager/libraries/notification.js` |
 | SendGrid provider | `src/manager/libraries/email/providers/sendgrid.js` |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backend-manager",
-  "version": "5.8.3",
+  "version": "5.9.1",
   "description": "Quick tools for developing Firebase functions",
   "main": "src/manager/index.js",
   "bin": {