backend-manager 5.5.4 → 5.6.0

package/docs/migration.md

@@ -0,0 +1,107 @@
+# Migration
+
+Procedures for migrating old BEM consumer projects to the current format: environment variables (Part 1), legacy code patterns (Part 2), and routes/schemas (Part 3).
+
+## Part 1: Environment Variable Migration
+
+Convert old config formats (runtime config / nested JSON) into individual top-level environment variables in `functions/.env`.
+
+### Key Mapping
+
+| Old Path | New ENV Key |
+|----------|-------------|
+| `backend_manager.key` or `backendmanager.key` | `BACKEND_MANAGER_KEY` |
+| `backend_manager.namespace` or `backendmanager.namespace` | `BACKEND_MANAGER_NAMESPACE` |
+| `github.key` or `github.token` | `GITHUB_TOKEN` |
+| `openai.key` or `openai.api_key` | `OPENAI_API_KEY` |
+| `paypal.client_id` | `PAYPAL_CLIENT_ID` |
+| `paypal.client_secret` | `PAYPAL_CLIENT_SECRET` |
+| `stripe.secret_key` or `stripe.key` | `STRIPE_SECRET_KEY` |
+| `chargebee.site` | `CHARGEBEE_SITE` |
+| `chargebee.api_key` or `chargebee.key` | `CHARGEBEE_API_KEY` |
+| `coinbase.api_key` or `coinbase.key` | `COINBASE_API_KEY` |
+| `cloudflare.token` or `cloudflare.key` | `CLOUDFLARE_TOKEN` |
+| `recaptcha.secret_key` or `recaptcha.key` | `RECAPTCHA_SECRET_KEY` |
+| `sendgrid.api_key` or `sendgrid.key` | `SENDGRID_API_KEY` |
+| `beehiiv.api_key` or `beehiiv.key` | `BEEHIIV_API_KEY` |
+| `zerobounce.api_key` or `zerobounce.key` | `ZEROBOUNCE_API_KEY` |
+
+### Steps
+
+1. **Check for config sources**: first `functions/.runtimeconfig.json` (parse JSON); else a `RUNTIME_CONFIG` variable inside `functions/.env` (parse the object inside).
+2. **Extract key-value pairs** using the mapping table.
+3. **Backup existing `.env`** as `functions/.env.backup` if it exists.
+4. **Check existing `.env` for conflicts**: skip existing keys and warn.
+5. **Write/update `functions/.env`**: each mapped key as a top-level variable.
+6. **Delete source files**: remove `functions/.runtimeconfig.json` if it existed.
+7. **Update `functions/backend-manager-config.json`**: remove the deprecated `mailchimp` key entirely; update `brand` to the nested structure `{ name, url, contact: { email }, images: { brandmark, wordmark, combomark } }`; set `github.user` to `"itw-creative-works"`.
+8. Update `functions/.nvmrc` to `v22/*` and `functions/package.json` `engines.node` to `"22"`.
+9. Clean up `functions/.gitignore` duplicates.
+
+## Part 2: Legacy Code Migration
+
+Search all `.js` files under `functions/` for legacy config reads and convert to `process.env`:
+
+- `Manager.config.*` → `process.env.KEY_NAME`
+- `RUNTIME_CONFIG` → individual `process.env` vars
+- `functions.config()` → `process.env` vars
+
+| Old Pattern | New Pattern |
+|-------------|-------------|
+| `Manager.config.github.key` | `process.env.GITHUB_TOKEN` |
+| `Manager.config.sendgrid.key` | `process.env.SENDGRID_API_KEY` |
+| `Manager.config.stripe.secret_key` | `process.env.STRIPE_SECRET_KEY` |
+| `Manager.config.openai.key` | `process.env.OPENAI_API_KEY` |
+| `Manager.config.paypal.client_id` | `process.env.PAYPAL_CLIENT_ID` |
+| `Manager.config.paypal.client_secret` | `process.env.PAYPAL_CLIENT_SECRET` |
+| `Manager.config.chargebee.site` | `process.env.CHARGEBEE_SITE` |
+| `Manager.config.chargebee.api_key` | `process.env.CHARGEBEE_API_KEY` |
+| `Manager.config.coinbase.api_key` | `process.env.COINBASE_API_KEY` |
+| `Manager.config.cloudflare.token` | `process.env.CLOUDFLARE_TOKEN` |
+| `Manager.config.recaptcha.secret_key` | `process.env.RECAPTCHA_SECRET_KEY` |
+| `Manager.config.beehiiv.api_key` | `process.env.BEEHIIV_API_KEY` |
+| `Manager.config.zerobounce.api_key` | `process.env.ZEROBOUNCE_API_KEY` |
+| `Manager.config.backend_manager.key` | `process.env.BACKEND_MANAGER_KEY` |
+| `Manager.config.backend_manager.namespace` | `process.env.BACKEND_MANAGER_NAMESPACE` |
+
+## Part 3: Route/Schema Migration
+
+**IMPORTANT:** Only migrate routes that already use the middleware system in `functions/index.js`:
+
+```javascript
+// MIGRATE these (uses Manager.Middleware)
+.https.onRequest((req, res) => Manager.Middleware(req, res).run('example'));
+
+// DO NOT migrate these (old manual route loading)
+.https.onRequest(async (req, res) => {
+  return new (require(`${__dirname}/routes/example/index.js`))().main(Manager, req, res);
+});
+```
+
+### Old → New Format
+
+**Route:** constructor pattern → context-object export ([routes.md](routes.md)):
+
+- `routes/example/index.js` → `routes/example/post.js` (or the appropriate method file)
+- Remove the constructor; use `module.exports = async ({ Manager, assistant, analytics, usage, user, settings, libraries, utilities }) => {}`
+
+**Schema:** wrapped tiers → flat ([schemas.md](schemas.md)):
+
+- `schemas/example/index.js` → `schemas/example/post.js`
+- Remove the `['defaults']:` wrapper; flatten the structure (plan adjustments move INSIDE the function, branching on `user`)
+- Change the signature to the context object: `({ assistant, user, data, method, headers, geolocation, client })`
+- Remove `value: undefined` noise
+
+| Aspect | Old Format | New Format |
+|--------|-----------|------------|
+| Route export | `module.exports = Route` (constructor) | `module.exports = async ({ ... }) => {}` |
+| Self reference | `const self = this;` | Not needed |
+| File naming | `index.js` | `get.js`, `post.js`, `put.js`, `delete.js` |
+| Schema wrapper | `['defaults']: { ... }` | Flat structure (no wrapper) |
+| Schema params | `(assistant)` | `({ assistant, user, data, ... })` context object |
+
+## See also
+
+- [routes.md](routes.md) — the current route format being migrated TO
+- [schemas.md](schemas.md) — the current schema contract
+- [environment-detection.md](environment-detection.md) — env var conventions

package/docs/routes.md

@@ -36,35 +36,131 @@ module.exports = Module;
 
 ## New Route (Consumer Project)
 
-Create `routes/{name}/index.js`:
+Routes live at `functions/routes/{path}/{method}.js` — BEM routes requests to the matching method file (`get.js` / `post.js` / `put.js` / `delete.js`), falling back to `index.js` if no method-specific file exists.
 
-```javascript
-function Route() {}
-
-Route.prototype.main = async function (assistant) {
-  const Manager = assistant.Manager;
-  const usage = assistant.usage;
-  const user = assistant.usage.user;
-  const analytics = assistant.analytics;
-  const settings = assistant.settings;
+A route exports an **async function receiving a context object** (built by the middleware — see `src/manager/helpers/middleware.js`):
 
-  // Check authentication if needed
+```javascript
+/**
+ * POST /items - Create a new item
+ */
+module.exports = async ({ Manager, assistant, analytics, usage, user, settings, libraries, utilities }) => {
   if (!user.authenticated) {
     return assistant.respond('Authentication required', { code: 401 });
   }
 
+  const { admin } = libraries;
+  const firestore = admin.firestore();
+
   // Track usage
   await usage.validate('requests');
   usage.increment('requests');
   await usage.update();
 
-  // Send response
-  assistant.respond({ success: true, data: settings });
+  // settings strings are whitespace-trimmed by middleware; HTML is preserved.
+  // Call utilities.sanitize() at the HTML-insertion site, or opt the whole
+  // route in to middleware HTML strip via { sanitize: true } on .run().
+  const id = settings.id; // auto-generated by the schema (see schemas.md)
+
+  await firestore.doc(`items/${id}`).set({ id, owner: user.auth.uid, ...settings });
+
+  return assistant.respond({ id });
 };
+```
+
+Context object fields: `Manager`, `assistant`, `user` (from `assistant.getUser()`), `usage`, `settings` (schema-resolved), `analytics`, `libraries`, `utilities`.
+
+### CRUD method files
 
-module.exports = Route;
+Every resource endpoint follows proper CRUD with **method-specific files** (plural-noun route names — `items`, not `item`):
+
+| Method | File | Purpose | Path |
+|--------|------|---------|------|
+| GET | `get.js` | List all or fetch one | `/items` or `/items/{id}` |
+| POST | `post.js` | Create new | `/items` |
+| PUT | `put.js` | Update existing | `/items/{id}` |
+| DELETE | `delete.js` | Delete existing | `/items/{id}` |
+
+**GET routes exist for external API consumers.** The dashboard/frontend reads data directly from Firestore (faster, cheaper, no cold starts). POST/PUT/DELETE still go through Cloud Functions for server-side validation, usage tracking, and analytics.
+
+**PUT and DELETE must verify ownership** before mutating:
+
+```javascript
+const existing = docSnap.data();
+
+if (existing.owner !== user.auth.uid) {
+  return assistant.respond('Not authorized', { code: 403 });
+}
 ```
 
+Immutable fields (`id`, `owner`, `stats`, `metadata.created`) should NOT be editable via PUT. ID generation (POST) and ID-from-path extraction (GET/PUT/DELETE) happen in the **schema**, not the route — see [schemas.md](schemas.md).
+
+### Key route patterns
+
+- Short-circuit returns for auth/validation checks
+- `settings` contains the parsed + validated request data (from schemas)
+- `assistant.respond()` for ALL responses (success and error)
+- `admin.firestore().doc('collection/id')` shorthand for Firestore access ([firestore.md](firestore.md))
+- Timestamps under `metadata.{created,updated}` ([firestore.md](firestore.md#document-metadata))
+
+### Functions entry point (`functions/index.js`)
+
+```javascript
+const Manager = (new (require('backend-manager'))).init(exports, {
+  setupFunctionsIdentity: false,
+});
+const { functions } = Manager.libraries;
+
+/**
+ * @route /items
+ *
+ * @method GET    /items            - List all items
+ * @method GET    /items/:itemId    - Get single item
+ * @method POST   /items            - Create a new item
+ * @method PUT    /items/:itemId    - Update an item
+ * @method DELETE /items/:itemId    - Delete an item
+ */
+exports.items = functions
+.runWith({ memory: '256MB', timeoutSeconds: 120 })
+.https.onRequest((req, res) => Manager.Middleware(req, res).run('items'));
+```
+
+The schema defaults to the route name (`.run('items')` loads `functions/schemas/items/`); pass `{ schema: 'custom' }` only when the schema path differs.
+
+For operations that don't fit CRUD (e.g. `/items/:itemId/export`), add an **action sub-path** handled within the same Cloud Function (parsed from the request path) or as a separate function if resource needs differ significantly.
+
+### firebase.json routing
+
+Make routes public with rewrites. Use the bracket syntax so sub-paths like `/items/{id}` route correctly:
+
+```json
+{
+  "hosting": {
+    "rewrites": [
+      { "source": "{/items,/items/**}", "function": "items" }
+    ]
+  }
+}
+```
+
+**CRITICAL**: Without the `/**` wildcard, requests to `/items/{id}` won't reach the function.
+
+**CRITICAL: Rewrite order matters — first match wins.** When multiple functions share a path prefix, the most specific routes MUST come first and the catch-all last, otherwise it swallows all sub-routes:
+
+```json
+{
+  "hosting": {
+    "rewrites": [
+      { "source": "{/agents/*/chat,/agents/*/chat/**}", "function": "agentsChat" },
+      { "source": "/agents/*/conversations/**", "function": "agentsConversations" },
+      { "source": "{/agents,/agents/**}", "function": "agents" }
+    ]
+  }
+}
+```
+
+Order: most specific → least specific → catch-all.
+
 ## New Event Handler
 
 Create `src/manager/functions/core/events/{type}/{event}.js`:

package/docs/schemas.md

@@ -1,39 +1,135 @@
 # Schemas
 
-Schemas define and validate the payload your routes accept. Schema names should match the route name (e.g. route `myEndpoint` ↔ schema `myEndpoint`).
+Schemas define and validate the payload your routes accept. They live at `functions/schemas/{name}/{method}.js` — the resolver loads the method-specific file first (`get.js` / `post.js` / …), falling back to `{name}/index.js` (see `src/manager/helpers/settings.js`). The schema name defaults to the route name (`.run('items')` ↔ `functions/schemas/items/`).
 
-## New Schema (Consumer Project)
+## Schema function contract
 
-Create `schemas/{name}/index.js`:
+A schema exports a function that receives a **context object** and returns a **flat schema object**. Plan-based adjustments happen INSIDE the function (there is no tier system):
 
 ```javascript
-module.exports = function (assistant, settings, options) {
-  const user = options.user;
-
-  return {
-    defaults: {
-      fieldName: {
-        types: ['string'],
-        default: 'default value',
-        required: false,
-      },
-      numericField: {
-        types: ['number'],
-        default: 10,
-        min: 1,
-        max: 100,
-      },
+module.exports = ({ assistant, user, data, method, headers, geolocation, client }) => {
+  const planId = user?.subscription?.product?.id || 'basic';
+  const isPremium = planId !== 'basic';
+
+  const schema = {
+    name: {
+      types: ['string'],
+      default: '',
+      required: true,
     },
-    // Override for premium users
-    premium: {
-      numericField: {
-        max: 1000,
-      },
+    limit: {
+      types: ['number'],
+      default: 10,
+      min: 1,
+      max: isPremium ? 1000 : 100,   // plan-based limit
     },
   };
+
+  // Premium-only field (not present for basic users)
+  if (isPremium) {
+    schema.premiumOnlyField = {
+      types: ['string'],
+      default: 'premium-feature',
+    };
+  }
+
+  return schema;
 };
 ```
 
+Context fields: `assistant`, `user` (resolved user), `data` (raw request data), `method`, `headers`, `geolocation`, `client`.
+
+## Field Properties
+
+- `types` — array of allowed types: `['string']`, `['number']`, `['boolean']`, `['object']`, `['array']`, `['any']`, or multiple (`['string', 'number']`)
+- `default` — default value if not provided; may be a function (`default: () => ...`)
+- `value` — force-set value (ignores user input — e.g. auto-generated IDs)
+- `required` — `true`/`false` or a function `(assistant, settings, options) => bool`. **NEVER combine with `default`** — see the footgun below
+- `min` / `max` — validation bounds (string length, number range, array length); numbers clamp
+- `clean` — a RegExp (matched chars removed) or function `(value) => cleaned`
+- `sanitize` — per-field opt-out (`false`) for HTML sanitization; only meaningful when the route opts in via `Manager.Middleware(req, res).run('route', { sanitize: true })`. See [sanitization.md](sanitization.md)
+
+### ⚠️ `required` + `default` footgun
+
+BEM checks `required` against the ORIGINAL request value, before defaults apply — so `required: true` on a field with a `default` throws `Required key {field} is missing in settings` before the default is ever used. For fields that must be non-empty but have a derived default (like path-extracted IDs), use `min: 1` instead.
+
+## ID Generation (POST — Create)
+
+IDs are auto-generated in the **schema**, NOT in the route. Use `value` to force-generate via BEM's built-in `randomId()` (14-char nanoid, 62-char alphabet, no `-` or `_`):
+
+```javascript
+id: {
+  types: ['string'],
+  value: () => assistant.Manager.Utilities().randomId(),
+},
+```
+
+The route just reads `settings.id` — no ID generation logic needed.
+
+## ID Extraction from Path (GET, PUT, DELETE)
+
+For single-item operations, extract the ID from the URL path in the **schema**:
+
+```javascript
+// /items/{id} → split('/') = ['', 'items', '{id}'] → index 2
+id: {
+  types: ['string'],
+  default: (assistant.request.path || '').split('/')[2] || '',
+  min: 1,     // enforce non-empty (NOT required: true — see footgun above)
+  max: 128,
+},
+```
+
+For GET list endpoints, omit `min` so an empty ID means "list":
+
+```javascript
+id: {
+  types: ['string'],
+  default: (assistant.request.path || '').split('/')[2] || '',
+},
+limit: { types: ['number'], default: 20, min: 1, max: 100 },
+startAfter: { types: ['string'], default: '' },
+```
+
+## Dynamic Schemas
+
+Schemas can branch on request data (e.g. different fields per type):
+
+```javascript
+module.exports = ({ assistant, data }) => {
+  const type = data?.type || '';
+
+  const schema = {
+    name: { types: ['string'], default: undefined, required: true },
+    type: { types: ['string'], default: undefined, required: true },
+    options: {},
+  };
+
+  switch (type) {
+    case 'url':
+      schema.options = { url: { types: ['string'], default: undefined, required: true } };
+      break;
+    case 'text':
+      schema.options = { text: { types: ['string'], default: undefined, required: true } };
+      break;
+    default:
+      schema.options = {};
+  }
+
+  return schema;
+};
+```
+
+## Reference Implementation
+
+The comprehensive test schema exercising every field option (types, function defaults, forced `value`, conditional `required`, min/max clamping, `clean` regex + function, nested objects, plan-based fields): [`src/manager/schemas/test/schema/post.js`](../src/manager/schemas/test/schema/post.js).
+
 ## Field Sanitization
 
-Middleware always trims whitespace on string fields. HTML sanitization is **opt-in per route** — `Manager.Middleware(req, res).run('my-route', { sanitize: true })`. When opted in, fields can individually opt back out with `sanitize: false`. See [docs/sanitization.md](sanitization.md).
+Middleware always trims whitespace on string fields. HTML sanitization is **opt-in per route** — `Manager.Middleware(req, res).run('my-route', { sanitize: true })`. When opted in, fields can individually opt back out with `sanitize: false`. See [sanitization.md](sanitization.md).
+
+## See also
+
+- [routes.md](routes.md) — the routes consuming `settings`
+- [sanitization.md](sanitization.md) — trim vs HTML-strip behavior
+- [test-framework.md](test-framework.md) — schema tests (`test/routes/test/schema.js`)

package/docs/test-boot-layer.md

@@ -0,0 +1,65 @@
+# Test Framework — Boot Layer
+
+The `boot` layer is BEM's framework self-test. When `npx mgr test` runs from the backend-manager repo itself (no `firebase.json` in cwd), the runner boots a **bundled fixture Firebase project**, brings up the emulator against it, and runs the `test/boot/` smoke suite. It replaces the old "you can't run tests from the framework repo" gap with a deterministic pass/fail. BEM's equivalent of BXM's `BXM_TEST_BOOT_PROJECT` boot layer and UJM's `UJ_TEST_BOOT_PROJECT` site-boot layer.
+
+## What boot tests verify
+
+Things that ONLY break when the whole self-test path assembles correctly:
+- The Firebase emulator boots against the fixture (functions, firestore, auth, hosting, …)
+- The fixture's `functions/index.js` runs `Manager.init()` inside the emulator's functions runtime — i.e. the **local** `backend-manager` (symlinked in) loads and wires the built-in `bm_api` function
+- The hosting rewrite routes `/backend-manager/**` → `bm_api`
+- A health request returns `200` with the fixture's `projectId` (`demo-backend-manager`) and the live `bemVersion`
+
+If the boot smoke passes, the framework at minimum *boots a consumer backend end-to-end* — catching a class of integration breaks (broken `Manager.init`, mis-wired `bm_api`, bad hosting rewrites) that no single handler test covers.
+
+## Test file shape
+
+Same `{ description, type, tests }` contract as every other BEM suite — the boot suite is just scoped to the `test/boot/` directory and gated to self-test runs:
+
+```js
+module.exports = {
+  description: 'Boot smoke — fixture emulator + bm_api reachable',
+  type: 'group',
+  timeout: 30000,
+  tests: [
+    {
+      name: 'bm_api-health-responds-over-hosting-rewrite',
+      async run({ http, assert }) {
+        const response = await http.get('backend-manager/test/health');
+        assert.isSuccess(response, 'bm_api /test/health should respond through the emulator');
+      },
+    },
+  ],
+};
+```
+
+## The bundled fixture project
+
+`src/test/fixtures/firebase-project/` — a minimal, committed BEM consumer backend:
+
+- `firebase.json` + `.firebaserc` — a **`demo-` project** (`demo-backend-manager`) so the emulator NEVER touches real Firebase; emulator ports from `DEFAULT_EMULATOR_PORTS`; hosting rewrite to `bm_api`.
+- `functions/index.js` — the one-line `Manager.init()` bootstrap (mirrors a real consumer).
+- `functions/package.json` + `backend-manager-config.json` — fake brand/config (no real secrets).
+- `firestore.rules` / `storage.rules` / `database.rules.json` / `firestore.indexes.json` — minimal locked rules (`bm_api` uses the Admin SDK, which bypasses rules).
+
+**Runtime-only, gitignored** (never committed): before boot, the test command symlinks the local `backend-manager` (+ `firebase-admin`/`firebase-functions` from BEM's own `node_modules`) into the fixture's `functions/node_modules`, injects the fixture admin keys into the env, and generates a **throwaway RSA `service-account.json`** (emulator-only — a `demo-` project never authenticates against Google). All of this lives in `setupSelfTest()` / `linkFixtureDeps()` / `ensureFixtureServiceAccount()` in [src/cli/commands/test.js](../src/cli/commands/test.js).
+
+## `BEM_TEST_BOOT_PROJECT`
+
+| Env | Purpose |
+|---|---|
+| `BEM_TEST_BOOT_PROJECT` | Root of a Firebase project to boot instead of the bundled fixture. Auto-set to `src/test/fixtures/firebase-project` when BEM tests itself; set it explicitly to self-test against a **real consumer** (e.g. `ultimate-jekyll-backend`) without `cd`-ing into it. |
+
+## What happens in a consumer run
+
+The `boot/` smoke is **excluded from real-consumer runs** (`runner.js` `discoverTests` skips `boot/` unless `isFrameworkSelfTest`) — it targets the bundled fixture, so it would be redundant noise in a consumer's suite. Consumers run the full `routes`/`events`/`rules` suites against their own emulator as normal.
+
+## Why this exists
+
+BEM has no pure-logic test layer — every `routes`/`events`/`rules` suite needs a live emulator + a real project, so they run against a real consumer. The boot layer fills the remaining gap: a fast, self-contained smoke proving the framework still boots a consumer backend from the repo itself. It's the BEM analog of "does the extension load?" (BXM) / "does the site boot?" (UJM).
+
+## See also
+
+- [test-framework.md](test-framework.md) — overall harness, running/filtering, context object, assertions, auth levels
+- [logging.md](logging.md) — `functions/*.log` files (the emulator/test logs the boot run writes)
+- [environment-detection.md](environment-detection.md) — `Manager.isTesting()` and the environment signals