backend-manager 5.8.0 → 5.8.2

package/CHANGELOG.md

@@ -14,6 +14,27 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+# [5.8.2] - 2026-06-19
+
+### Fixed
+- **Setup crash on fresh projects.** `npx mgr setup` crashed with `Cannot read properties of undefined (reading 'default')` when `.firebaserc`, `firebase.json`, or `backend-manager-config.json` didn't exist. Setup now scaffolds all three from templates before reading from them.
+- **`engines.node` hard crash.** Missing `engines.node` in `package.json` threw an unrecoverable error. Now auto-fixes it using the current Node.js major version.
+- **`dependencies['backend-manager']` crash.** The local-BEM detection crashed when `backend-manager` was listed in `devDependencies` instead of `dependencies`.
+
+### Changed
+- **Unified scaffold pass.** Consolidated config scaffolding, `engines.node` fix, and doc defaults into one `[DEFAULTS]` section with a single file reload afterwards. Eliminates scattered scaffolding steps and conditional re-loads.
+
+### Added
+- **`templates/firebase.json`** — standard Firebase config template (hosting, functions, firestore, database, storage, emulators) scaffolded into new projects by `npx mgr setup`.
+
+# [5.8.1] - 2026-06-19
+
+### Fixed
+- **Ghostii 502 timeout fix.** Switched `writeArticle()` from the Firebase Hosting URL (`api.ghostii.ai`) to the raw Cloud Functions URL. Firebase Hosting rewrites have a hard 60-second proxy timeout that caused 502 errors when the AI article pipeline exceeded that limit. The raw URL uses the function's own 5-minute timeout.
+
+### Added
+- **Ghostii call duration logging.** `writeArticle()` now logs the round-trip time of each Ghostii API call (`[ghostii] writeArticle() completed in Xms`).
+
 # [5.8.0] - 2026-06-19
 
 ### Added

package/CLAUDE.md

@@ -18,7 +18,7 @@ Backend Manager (BEM) is a comprehensive framework for building modern Firebase
 ### For Consuming Projects
 
 1. `npm install backend-manager --save-dev` (inside `functions/`)
-2. `npx mgr setup` — validates config, scaffolds defaults (CLAUDE.md, CHANGELOG.md, docs/, test/), provisions Firestore indexes
+2. `npx mgr setup` — bootstraps a new project (scaffolds `.firebaserc`, `firebase.json`, `backend-manager-config.json`, `engines.node`, CLAUDE.md, CHANGELOG.md, docs/, test/), validates config, provisions Firestore indexes
 3. `npx mgr emulator` — start Firebase emulators (auth/firestore/functions/database/storage)
 4. `npx mgr serve` — local serve with Stripe webhook forwarding (if `STRIPE_SECRET_KEY` is set)
 5. `npx mgr test` — runs framework + project test suites against an emulator. Positional target(s) select which test FILES run, by source + path (multiple space-separated targets compose):
@@ -64,7 +64,7 @@ Every feature ships with tests at EVERY surface it exposes — logic (`test/rout
 
 | Command | Description |
 |---|---|
-| `setup` | Validate config, scaffold defaults (CLAUDE.md, CHANGELOG.md, docs/, test/), provision Firestore indexes |
+| `setup` | Bootstrap new projects (scaffolds config files + doc defaults), validate config, provision Firestore indexes |
 | `emulator` | Start Firebase emulators (auth/firestore/functions/database/storage) |
 | `serve` | Local Firebase serve (with auto Stripe webhook forwarding if keys set) |
 | `watch` | Auto-reload functions on file change |

package/PROGRESS.md

@@ -2,13 +2,23 @@
 > Agents and maintainers should update this file regularly to reflect the current state of the project.
 
 ## 🎯 Current Focus
-* **Goal:** RSS/Atom feed-based article generation for the Ghostii system
-* **Current Phase:** Implementation complete, pending deploy
+* **Goal:** Fix setup crash on fresh projects missing config files
+* **Current Phase:** Phase 6 — in progress (code done, needs docs + publish)
 * **Priority:** High
-* **Last Updated:** 2026-06-18 6:45 PM PDT
-* **Notes:** All code + tests complete. Found and fixed humanizer bug — `injectBurstiness` was destroying markdown links by splitting on commas inside `[text](url)` syntax. Fixed with protect-and-restore pattern. Also fixed 403 links treated as "working" in verification step. Ghostii backend needs deploy.
+* **Last Updated:** 2026-06-18 11:30 PM PDT
+* **Notes:** Setup now fully bootstraps a bare project (just package.json + .nvmrc). Scaffolds configs, engines.node, everything — stops cleanly at service-account with a Firebase console link. Still needs docs + publish.
 
 ## 📌 Active Task List
+* [ ] Phase 6: Setup scaffolds essential configs for fresh projects
+  * [x] Task 6.1: Add `templates/firebase.json` standard template
+  * [x] Task 6.2: Add `scaffoldConfigs()` + `resolveProjectId()` to setup.js (runs before config resolution)
+  * [x] Task 6.3: Auto-fix `engines.node` instead of throwing
+  * [x] Task 6.4: Fix `dependencies['backend-manager']` crash when BEM is in devDependencies
+  * [x] Task 6.5: Verify fix on truly bare project (dailyembers-backend — 13/14 pass, only service-account expected)
+  * [x] Task 6.6: Optimize — consolidate scattered scaffolding into one `[DEFAULTS]` pass with `loadFiles()` DRY extraction
+  * [x] Task 6.7: Regression test on existing project (ultimate-jekyll-backend — all pass)
+  * [x] Task 6.8: Update docs (CLAUDE.md, CHANGELOG.md)
+  * [ ] Task 6.9: Publish new BEM version
 * [ ] Phase 5: Ghostii feed-based article system
   * [x] Task 5.1: Create `feed-parser.js` (RSS 2.0, Atom 1.0, JSON Feed parser + article extractor)
   * [x] Task 5.2: Add `fast-xml-parser` dependency to BEM
@@ -28,7 +38,8 @@
   * [x] Task 5.16: Fix humanizer stripping links — `injectBurstiness` was breaking `[text](url)` syntax; added protect-and-restore pattern + tests
   * [x] Task 5.17: Fix 403 links treated as "working" in link verification step
   * [x] Task 5.18: Add detective-level `[LINKS]` diagnostic logging to article pipeline
-  * [ ] Task 5.19: Deploy Ghostii backend
+  * [x] Task 5.19: Ship BEM v5.8.0 to npm
+  * [x] Task 5.20: Ship Ghostii-backend v1.0.5 + deploy to Firebase
   * [ ] Task 5.17: Publish BEM with feed support
   * [ ] Task 5.12: Publish BEM with feed support
   * [ ] Task 5.13: Configure consumer project(s) with `$feed:` sources

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backend-manager",
-  "version": "5.8.0",
+  "version": "5.8.2",
   "description": "Quick tools for developing Firebase functions",
   "main": "src/manager/index.js",
   "bin": {

package/src/cli/commands/setup.js

@@ -70,14 +70,8 @@ class SetupCommand extends BaseCommand {
     // prints it on a hard failure; we print it here on success).
     self.setupSummary = new ui.Summary().start();
 
-    // Load files
-    self.package = loadJSON(`${self.firebaseProjectPath}/functions/package.json`);
-    self.firebaseJSON = loadJSON(`${self.firebaseProjectPath}/firebase.json`);
-    self.firebaseRC = loadJSON(`${self.firebaseProjectPath}/.firebaserc`);
-    self.remoteconfigJSON = loadJSON(`${self.firebaseProjectPath}/functions/remoteconfig.template.json`);
-    self.projectPackage = loadJSON(`${self.firebaseProjectPath}/package.json`);
-    self.bemConfigJSON = loadJSON(`${self.firebaseProjectPath}/functions/backend-manager-config.json`);
-    self.gitignore = jetpack.read(`${self.firebaseProjectPath}/.gitignore`) || '';
+    // Initial load — returns {} for missing files so scaffold checks can run.
+    this.loadFiles();
 
     // Check if package exists
     if (!hasContent(self.package)) {
@@ -93,12 +87,23 @@ class SetupCommand extends BaseCommand {
       process.exit(1);
     }
 
-    // Load the rules files
-    this.getRulesFile();
+    // One unified scaffold pass: config files, package.json fixes, doc defaults.
+    // Everything that creates/fixes files goes here, BEFORE any code reads from
+    // them. One reload afterwards picks up the final state.
+    ui.section('Defaults');
+    this.scaffoldConfigs();
+    this.scaffoldPackageJson();
+    this.copyDefaults();
+    this.loadFiles();
+
+    // Clean up leftover trigger files + stale log files from older BEM versions
+    this.cleanupGeneratedArtifacts();
 
+    // Load the rules files (reads from BEM's own templates/, not consumer files)
+    this.getRulesFile();
     self.default.rulesVersionRegex = new RegExp(`///---version=${self.default.version}---///`);
 
-    // Set project info
+    // Resolve project info — safe now, scaffoldConfigs guarantees these exist.
     self.projectId = self.firebaseRC.projects.default;
     self.projectUrl = `https://console.firebase.google.com/project/${self.projectId}`;
     self.apiUrl = `https://api.${(self.bemConfigJSON.brand?.url || '').replace(/^https?:\/\//, '')}`;
@@ -110,24 +115,15 @@ class SetupCommand extends BaseCommand {
     ui.field('Project', self.projectId, { pad: 9 });
     ui.field('API', self.apiUrl, { pad: 9, valueColor: chalk.cyan });
 
-    if (!self.package || !self.package.engines || !self.package.engines.node) {
-      throw new Error('Missing <engines.node> in package.json');
-    }
-
-    // Clean up leftover trigger files + stale log files from older BEM versions
-    this.cleanupGeneratedArtifacts();
-
-    // Copy / merge defaults into consumer project root (matches EM/BXM/UJM pattern).
-    // Runs BEFORE tests so any test that inspects scaffolded files sees the merged state.
-    ui.section('Defaults');
-    this.copyDefaults();
-
     // Run all tests
     ui.section('Checks');
     await this.runTests();
 
     // Warn if using local backend-manager
-    if (self.package.dependencies['backend-manager'].includes('file:')) {
+    const bemDep = self.package.dependencies?.['backend-manager']
+      || self.package.devDependencies?.['backend-manager']
+      || '';
+    if (bemDep.includes('file:')) {
       ui.section('Notices');
       ui.status('warn', `Using the local ${chalk.bold('backend-manager')} source (file: dependency)`, { level: 2 });
     }
@@ -245,6 +241,84 @@ class SetupCommand extends BaseCommand {
     }
   }
 
+  loadFiles() {
+    const self = this.main;
+    self.package = loadJSON(`${self.firebaseProjectPath}/functions/package.json`);
+    self.firebaseJSON = loadJSON(`${self.firebaseProjectPath}/firebase.json`);
+    self.firebaseRC = loadJSON(`${self.firebaseProjectPath}/.firebaserc`);
+    self.remoteconfigJSON = loadJSON(`${self.firebaseProjectPath}/functions/remoteconfig.template.json`);
+    self.projectPackage = loadJSON(`${self.firebaseProjectPath}/package.json`);
+    self.bemConfigJSON = loadJSON(`${self.firebaseProjectPath}/functions/backend-manager-config.json`);
+    self.gitignore = jetpack.read(`${self.firebaseProjectPath}/.gitignore`) || '';
+  }
+
+  scaffoldPackageJson() {
+    const self = this.main;
+    const ui = this.ui;
+
+    if (!self.package.engines || !self.package.engines.node) {
+      const nodeVer = String(parseInt(process.versions.node, 10));
+      self.package.engines = self.package.engines || {};
+      self.package.engines.node = nodeVer;
+      jetpack.write(`${self.firebaseProjectPath}/functions/package.json`, JSON.stringify(self.package, null, 2));
+      ui.status('add', `Added ${chalk.cyan('engines.node')} = ${chalk.bold(nodeVer)} to package.json`, { level: 2 });
+    }
+  }
+
+  scaffoldConfigs() {
+    const self = this.main;
+    const ui = this.ui;
+    const templatesDir = path.resolve(`${__dirname}/../../../templates`);
+    let touched = 0;
+
+    // .firebaserc — resolve project ID from service account or env
+    const firebasercPath = `${self.firebaseProjectPath}/.firebaserc`;
+    if (!hasContent(self.firebaseRC)) {
+      const projectId = this.resolveProjectId();
+      jetpack.write(firebasercPath, JSON.stringify({ projects: { default: projectId } }, null, 2) + '\n');
+      ui.status('add', `Created ${chalk.cyan('.firebaserc')} (project: ${chalk.bold(projectId)})`, { level: 2 });
+      touched++;
+    }
+
+    // firebase.json
+    const firebaseJsonPath = `${self.firebaseProjectPath}/firebase.json`;
+    if (!hasContent(self.firebaseJSON)) {
+      const templatePath = path.join(templatesDir, 'firebase.json');
+      jetpack.copy(templatePath, firebaseJsonPath);
+      ui.status('add', `Created ${chalk.cyan('firebase.json')}`, { level: 2 });
+      touched++;
+    }
+
+    // backend-manager-config.json
+    const bemConfigPath = `${self.firebaseProjectPath}/functions/backend-manager-config.json`;
+    if (!hasContent(self.bemConfigJSON)) {
+      const templatePath = path.join(templatesDir, 'backend-manager-config.json');
+      jetpack.copy(templatePath, bemConfigPath);
+      ui.status('add', `Created ${chalk.cyan('functions/backend-manager-config.json')}`, { level: 2 });
+      touched++;
+    }
+
+    return touched;
+  }
+
+  resolveProjectId() {
+    const self = this.main;
+    const saPath = `${self.firebaseProjectPath}/functions/service-account.json`;
+
+    if (jetpack.exists(saPath)) {
+      try {
+        const sa = JSON.parse(jetpack.read(saPath));
+        if (sa.project_id) {
+          return sa.project_id;
+        }
+      } catch (e) {
+        // Fall through
+      }
+    }
+
+    return process.env.GCLOUD_PROJECT || 'demo-project';
+  }
+
   cleanupGeneratedArtifacts() {
     const self = this.main;
 

package/src/manager/libraries/content/ghostii.js

@@ -37,7 +37,7 @@ const powertools = require('node-powertools');
  *   where `json` is the structured block array ([{ name, content }]) that
  *   blocksToPost() consumes to build BEM's post shape.
  */
-function writeArticle({ brand, description, links, sourceContent, overrides }) {
+async function writeArticle({ brand, description, links, sourceContent, overrides }) {
   const o = overrides || {};
 
   const body = {
@@ -60,13 +60,24 @@ function writeArticle({ brand, description, links, sourceContent, overrides }) {
     body.sourceContent = sourceContent;
   }
 
-  return fetch('https://api.ghostii.ai/write/article', {
+  const start = Date.now();
+
+  // Original URL: https://api.ghostii.ai/write/article
+  // Firebase Hosting rewrites have a hard 60s proxy timeout that causes 502s
+  // on slow AI pipelines. The raw Cloud Functions URL uses the function's own
+  // 5-minute timeout instead.
+  const result = await fetch('https://us-central1-ghostii.cloudfunctions.net/writeGlobal/write/article', {
     method: 'post',
     timeout: 180000,
     tries: 1,
     response: 'json',
     body: body,
   });
+
+  const duration = Date.now() - start;
+  console.log(`[ghostii] writeArticle() completed in ${duration}ms (${(duration / 1000).toFixed(1)}s)`);
+
+  return result;
 }
 
 /**

package/src/manager/libraries/email/data/disposable-domains.json

@@ -237,6 +237,7 @@
   "4warding.org",
   "50set.ru",
   "51baidu.tech",
+  "521883.xyz",
   "52mails.com",
   "55hosting.net",
   "5ghgfhfghfgh.tk",
@@ -252,6 +253,7 @@
   "5y.loseyourip.com",
   "5ymail.com",
   "60minutemail.com",
+  "628868.xyz",
   "66-77.cc",
   "668fmail.com",
   "672643.net",
@@ -508,6 +510,7 @@
   "alienware13.com",
   "aligamel.com",
   "alightmotion.cloud",
+  "alightmotion.id",
   "aligroup.uk",
   "alilot.com",
   "alin.cc.cd",
@@ -568,6 +571,7 @@
   "amail.com",
   "amail1.com",
   "amail4.me",
+  "amazinggift.life",
   "amazingly.online",
   "amazon-aws.org",
   "amberwe.us",
@@ -1399,6 +1403,7 @@
   "code-gmail.com",
   "codeandscotch.com",
   "codeguard.net",
+  "codei.site",
   "coderdir.com",
   "codestar.site",
   "codesuc.top",
@@ -1483,6 +1488,7 @@
   "crossmailjet.com",
   "crossroadsmail.com",
   "crunchcompass.com",
+  "cruncoau.asia",
   "crusthost.com",
   "cryptoavalonsolhub.cloud",
   "cryptogmail.com",
@@ -3441,6 +3447,7 @@
   "jav8.de",
   "javadmin.com",
   "javaemail.com",
+  "javaka.live",
   "javindra.cfd",
   "jbsze.com",
   "jbsze.ne",
@@ -3539,6 +3546,7 @@
   "kagi.be",
   "kakadua.net",
   "kakao-mail.com",
+  "kakaomail.kr",
   "kakator.com",
   "kakaveo3.io.vn",
   "kakslsie.store",
@@ -4042,6 +4050,7 @@
   "maildrop.ml",
   "maildu.de",
   "maildx.com",
+  "maildy.site",
   "maileater.com",
   "mailed.in",
   "mailed.ro",
@@ -5031,6 +5040,8 @@
   "otpku.com",
   "ouchcloud.com",
   "ouk.me",
+  "oupul8b1k.bond",
+  "oupul8b1k.cyou",
   "ourklips.com",
   "ourl.me",
   "ourpreviewdomain.com",
@@ -5045,6 +5056,8 @@
   "owleyes.ch",
   "owlpic.com",
   "ownsyou.de",
+  "owo-mailteam.bond",
+  "owo-mailteam.cyou",
   "owube.com",
   "oxfo.edu.pl",
   "oxmail.homes",
@@ -5638,12 +5651,14 @@
   "satisfyme.club",
   "satukosong.com",
   "saturrshop.uk",
+  "saungadaid.pro",
   "sausen.com",
   "savests.com",
   "say0.com",
   "saynotospams.com",
   "sayverfox.com",
   "scatmail.com",
+  "scatterteam.com",
   "scay.net",
   "schachrol.com",
   "schafmail.de",
@@ -5764,6 +5779,7 @@
   "shuelder.com",
   "shut.name",
   "shut.ws",
+  "shxwonew.xyz",
   "siap-sepuh.com",
   "siberpay.com",
   "sicmg.com",
@@ -7310,8 +7326,12 @@
   "yopmail.fr",
   "yopmail.gq",
   "yopmail.kro.kr",
+  "yopmail.loseyourip.com",
   "yopmail.net",
+  "yopmail.ooguy.com",
   "yopmail.pp.ua",
+  "yopmail.theworkpc.com",
+  "yopmail.webredirect.org",
   "yoptruc.fr.nf",
   "yordanmail.cf",
   "yorushika.one",
@@ -7345,6 +7365,7 @@
   "ytpayy.com",
   "yugasandrika.com",
   "yui.it",
+  "yujiehanjiao.cc",
   "yuki.ren",
   "yun.pics",
   "yuoia.com",
@@ -7468,6 +7489,7 @@
   "zzzzzzzzzzzzzz-2092.dynv6.net",
   "zzzzzzzzzzzzzz-277.dynv6.net",
   "zzzzzzzzzzzzzz-3294.dynv6.net",
+  "zzzzzzzzzzzzzz-3330.dynv6.net",
   "zzzzzzzzzzzzzz-3535.dynv6.net",
   "zzzzzzzzzzzzzz-3928.dynv6.net",
   "zzzzzzzzzzzzzz-3940.dynv6.net",

package/templates/firebase.json

@@ -0,0 +1,66 @@
+{
+  "hosting": {
+    "public": "public",
+    "ignore": [
+      "firebase.json",
+      "**/.*",
+      "**/node_modules/**"
+    ],
+    "rewrites": [
+      {
+        "source": "{/backend-manager,/backend-manager/**,/mcp,/mcp/**,/.well-known/oauth-protected-resource,/.well-known/oauth-authorization-server,/authorize,/token,/register}",
+        "function": "bm_api"
+      }
+    ]
+  },
+  "functions": [
+    {
+      "source": "functions",
+      "codebase": "default",
+      "ignore": [
+        "node_modules",
+        ".git",
+        "firebase-debug.log",
+        "firebase-debug.*.log"
+      ]
+    }
+  ],
+  "firestore": {
+    "rules": "firestore.rules",
+    "indexes": "firestore.indexes.json"
+  },
+  "database": {
+    "rules": "database.rules.json"
+  },
+  "storage": {
+    "rules": "storage.rules"
+  },
+  "emulators": {
+    "auth": {
+      "port": 9099
+    },
+    "functions": {
+      "port": 5001
+    },
+    "firestore": {
+      "port": 8080
+    },
+    "database": {
+      "port": 9000
+    },
+    "hosting": {
+      "port": 5002
+    },
+    "storage": {
+      "port": 9199
+    },
+    "ui": {
+      "enabled": true,
+      "port": 4050
+    },
+    "pubsub": {
+      "port": 8085
+    },
+    "singleProjectMode": true
+  }
+}