@loynazkovacs/theitemapp-backend-sdk 0.3.0 → 0.4.0

package/CHANGELOG.md

@@ -0,0 +1,56 @@
+# Changelog
+
+All notable changes to `@loynazkovacs/theitemapp-backend-sdk`. Pre-1.0, minor
+versions may add features; `0.x` caret ranges (`^0.3.0`) do **not** auto-bump
+across minors, so consumers upgrade deliberately.
+
+## 0.4.0
+
+Reliability & safety layers, so the most demanding clients (chat, coding-agent)
+can adopt the SDK as thin adapters with no behaviour loss. All opt-in, default off.
+
+### Added
+- **`retry`** config — automatic retry on configured statuses (default `[429]`),
+  honouring `Retry-After` (header or "retry in N seconds" body), with jitter and
+  a per-wait cap. Protects fire-and-forget writes from rate-limit drops.
+- **`validateXrefs`** config — load a collection's `items` schema (cached,
+  single-flight) and validate x-ref fields hold valid ObjectIds before each
+  write. `onInvalidXref: 'warn' | 'throw'`.
+- **`logger`** config — route retry/validation/decrypt warnings (default `console`).
+- **`CoreApiError.collection` / `.method` / `.target`** — structured fields
+  alongside `.status` / `.body`.
+- **`upsert(..., { skipIfUnchanged })`** — skip the write when the existing row
+  already deep-equals the body (no spurious change-stream events).
+- Exported utilities: **`extractXrefFields`**, **`checkXrefFields`**, **`deepEqual`**.
+- `asUser(...)` now propagates the parent's `retry` / `validateXrefs` / `logger`.
+
+### Docs
+- Comprehensive README (quick start, all three pillars, full API reference,
+  reliability/safety layers, adoption recipe) + this changelog.
+
+## 0.3.0
+
+Closed the gaps surfaced migrating 14 backends.
+
+### Added
+- `count()`, `bulkCreate()` (`/bulk`), `hardDeleteByFilter()`.
+- `upsertOn(matchOn[])` — composite-key atomic upsert → `{ ok, created, doc }`.
+- `get(id, { populate })`, `downloadFile()`, `decrypt(..., { throwOnError })`.
+- Per-call `WriteOptions { skipWebhooks, headers }` on create/update/upsertOn/bulkCreate.
+- **`asUser(creds, { keepApiKey })`** — scoped client that forwards the end
+  user's `Authorization`/`Cookie`.
+- **`verifyJwtLocal(token, secret)`** — offline HS256 verification (zero-dep).
+
+## 0.2.0
+
+### Fixed
+- `verifyUser` reads `id` from `/api/auth/me` (the JWT `sub`), not `_id` — it
+  previously returned `null` for every authenticated caller. `AuthUser` gained
+  `username`.
+
+## 0.1.0
+
+Initial release: `CoreApiClient` (typed `/api/dynamic` CRUD + file upload +
+decrypt + upsert/bulk-upsert), `startAppRegistration` (register/retry/heartbeat/
+deregister + provisioned-key capture), `verifyUser` + Express/Fastify auth
+adapters. Zero runtime dependencies.

package/README.md

@@ -5,18 +5,42 @@
 Server-side SDK for **TheItemApp app backends** — the counterpart to the
 frontend [`@loynazkovacs/theitemapp-platform-sdk`](../platform-sdk).
 
-Every app with its own backend (chat, coding-agent, vanda, music, image-generator,
-…) previously hand-rolled the same three things and let them drift:
+Every app with its own backend used to hand-roll the same three things and let
+them drift:
 
-1. a `coreApiClient.ts` wrapping `/api/dynamic` CRUD,
-2. a registration / heartbeat / deregister loop in `index.ts`,
-3. an auth middleware proxying `/api/auth/me`.
+1. a **core API client** wrapping `/api/dynamic` CRUD,
+2. a **registration / heartbeat / deregister** loop in `index.ts`,
+3. an **auth middleware** proxying `/api/auth/me`.
 
-This package is the single canonical implementation of all three.
+This package is the single, canonical, typed implementation of all three.
 
-> **Scope:** this SDK is for apps that run **their own backend**. Apps with no
-> backend ship via the [`seed-server`](../../images/seed-server) base image
-> instead and do not need this.
+> **Scope.** This SDK is for apps that run their **own backend**. Apps with no
+> backend ship via the [`seed-server`](../../images/seed-server) base image and
+> do not need it.
+
+- **Zero runtime dependencies.** Node ≥ 18 (uses the global `fetch`/`FormData`/`crypto`).
+- **Everything opt-in & back-compat.** New reliability/safety layers default off.
+
+---
+
+## Table of contents
+
+- [Install](#install)
+- [Quick start](#quick-start)
+- [Pillar 1 — Registration lifecycle](#pillar-1--registration-lifecycle)
+- [Pillar 2 — CoreApiClient](#pillar-2--coreapiclient)
+  - [Reads](#reads) · [Writes](#writes) · [Upserts](#upserts) · [Files](#files) · [Decrypt](#decrypt)
+  - [Acting as an end user (`asUser`)](#acting-as-an-end-user-asuser)
+  - [Reliability: automatic retry](#reliability-automatic-retry)
+  - [Safety: x-ref validation](#safety-x-ref-validation)
+  - [Error handling](#error-handling)
+  - [Full config reference](#full-config-reference)
+- [Pillar 3 — Auth](#pillar-3--auth)
+- [Adoption recipe (existing backends)](#adoption-recipe-existing-backends)
+- [Versioning & publishing](#versioning--publishing)
+- [API index](#api-index)
+
+---
 
 ## Install
 
@@ -24,95 +48,400 @@ This package is the single canonical implementation of all three.
 npm install @loynazkovacs/theitemapp-backend-sdk
 ```
 
-Requires Node ≥ 18 (uses the global `fetch`/`FormData`). Zero runtime dependencies.
+Node ≥ 18, ESM (`"type": "module"`).
+
+---
 
-## Usage
+## Quick start
+
+A typical Fastify app backend wires all three pillars at startup:
 
 ```ts
+import Fastify from 'fastify';
 import {
   CoreApiClient,
   startAppRegistration,
-  createFastifyAuthPreHandler, // or createExpressAuthMiddleware
+  createFastifyAuthPreHandler,
 } from '@loynazkovacs/theitemapp-backend-sdk';
 import manifest from './dbseed/manifest.json' assert { type: 'json' };
 
+const app = Fastify();
 const coreUrl = process.env.CORE_API_URL ?? 'http://backend:3001';
 
-// 1. Core API client — key gets filled in by registration below.
-const coreApi = new CoreApiClient({ baseUrl: coreUrl, apiKey: null });
+// 1) Core API client — key gets filled in by registration below.
+const coreApi = new CoreApiClient({
+  baseUrl: coreUrl,
+  apiKey: null,
+  retry: { maxAttempts: 5 },   // optional: survive 429 bursts
+});
 
-// 2. Registration lifecycle: retry-until-up, heartbeat, deregister-on-exit.
-const registration = startAppRegistration({
+// 2) Registration: retry-until-up, capture the rotated key, heartbeat, deregister.
+let registration;
+app.post('/app/re-register', async () => { registration?.reRegister(); return { ok: true }; });
+
+await app.listen({ host: '0.0.0.0', port: 3000 });
+
+registration = startAppRegistration({
   coreUrl,
   manifest,
   selfUrl: process.env.SELF_URL ?? 'http://myapp:80',
   registrationKey: process.env.APP_REGISTRATION_KEY,
+  installSignalHandlers: false,                 // we run our own shutdown below
   onApiKey: (key) => coreApi.updateApiKey(key), // core rotates the key per register
 });
 
-// Wire core's reboot recovery kick into your own route:
-app.post('/app/re-register', async () => {
-  registration.reRegister();
-  return { ok: true };
+process.on('SIGTERM', async () => {
+  registration?.stop();
+  await registration?.deregister();
+  await app.close();
+  process.exit(0);
+});
+
+// 3) Auth on protected routes — verifies the caller against core.
+app.get('/api/things', { preHandler: createFastifyAuthPreHandler(coreUrl) }, async () => {
+  return coreApi.list('things');
+});
+```
+
+---
+
+## Pillar 1 — Registration lifecycle
+
+`startAppRegistration(opts)` owns the full handshake every backend needs:
+register-with-retry (core can take minutes to boot), capture the
+**auto-provisioned & rotated** functional API key, a 5-minute heartbeat,
+re-register on core's reboot, and deregister on shutdown.
+
+```ts
+const registration = startAppRegistration({
+  coreUrl,                       // core base URL
+  manifest,                      // your dbseed manifest (must contain appKey)
+  selfUrl,                       // how core reaches this container, e.g. http://myapp:80
+  registrationKey,               // optional shared secret (X-Registration-Key)
+  onApiKey: (key) => coreApi.updateApiKey(key),
+  heartbeatMs: 5 * 60_000,       // optional (default 5 min; 0 disables)
+  maxRetries: 30,                // optional (default 30)
+  retryIntervalMs: 5_000,        // optional (default 5s)
+  installSignalHandlers: true,   // default true; set false to run your own shutdown
+  logger: app.log,               // optional { info, warn, error }
+});
+```
+
+| Returns | |
+|---|---|
+| `register()` | run one registration attempt now → `Promise<boolean>` |
+| `reRegister()` | fire-and-forget; wire into your `POST /app/re-register` route |
+| `deregister()` | best-effort `DELETE` from core's catalog |
+| `stop()` | stop the heartbeat timer |
+
+**Custom shutdown.** If your app must clean up on exit (stop a collector, close
+a DB, end sessions), pass `installSignalHandlers: false` and call
+`registration.stop()` + `await registration.deregister()` inside your own
+`SIGTERM`/`SIGINT` handler.
+
+**Provisioned key.** Core returns a fresh functional `apiKey` on each register
+(rotated, prefix-stable) **only if the app seeds a functional user**. `onApiKey`
+fires whenever a key is issued — use it to update your `CoreApiClient`. Apps
+that seed no functional user simply never receive one (that's fine).
+
+---
+
+## Pillar 2 — CoreApiClient
+
+A typed wrapper over `/api/dynamic/<collection>` plus the file and decrypt
+endpoints. Construct once; update the key when registration provides it.
+
+```ts
+const coreApi = new CoreApiClient({ baseUrl, apiKey: null });
+coreApi.updateApiKey(key);   // after registration
+coreApi.isReady();           // true once a non-empty key is set — gate writes on this
+```
+
+### Reads
+
+```ts
+await coreApi.list('apps');                                  // _l=500 by default
+await coreApi.list('apps', { _l: '50', _s: '-createdAt' });  // override params
+await coreApi.get('apps', id);                               // null on 404, throws otherwise
+await coreApi.get('apps', id, { populate: false });          // keep x-refs as id strings (?populate=0)
+await coreApi.findBy('apps', 'key', 'system');               // first match by indexed field, or null
+await coreApi.count('apps', { active: 'true' });             // GET .../count → number
+```
+
+> **`populate`.** Core returns single x-ref fields as **full objects** by
+> default. If your code compares x-ref ids as strings (e.g. diffing rows),
+> read with `{ populate: false }` so they stay as 24-hex strings.
+
+### Writes
+
+```ts
+await coreApi.create('apps', { name: 'X' });
+await coreApi.update('apps', id, { name: 'Y' });             // PUT (partial merge)
+await coreApi.delete('apps', id);                            // soft-delete → true (also on 404)
+await coreApi.bulkCreate('events', docs);                    // ≤500 → { ok, created, failed, results }
+await coreApi.hardDeleteByFilter('events', { stale: true }); // api-key only → { ok, deletedCount }
+```
+
+Per-call `WriteOptions` on `create`/`update`/`upsertOn`/`bulkCreate`:
+
+```ts
+await coreApi.create('audit', row, { skipWebhooks: false }); // let this write fan out
+await coreApi.update('x', id, body, { headers: { 'x-foo': '1' } });
+```
+
+### Upserts
+
+```ts
+// single field → returns the doc (or null); falls back to find+update/create on old cores
+await coreApi.upsert('apps', 'key', 'system', body);
+
+// skip the write if nothing changed (no spurious change-stream events)
+await coreApi.upsert('host', 'name', 'localhost', body, { skipIfUnchanged: true });
+
+// composite key → full result so you can branch on created vs updated
+const { created, doc } = await coreApi.upsertOn('links', ['srcId', 'dstId'], body);
+
+// bulk upsert ≤500 by composite key (usually ['_id'])
+await coreApi.bulkUpsert('rows', ['_id'], docs);             // → { upsertedCount, modifiedCount, errors }
+```
+
+### Files
+
+```ts
+const { _id } = await coreApi.uploadFile(bytes, {            // Uint8Array (a Buffer works)
+  filename: 'cover.png', mimeType: 'image/png', kind: 'image',
+});
+const file = await coreApi.downloadFile(_id);               // { data, contentType, filename } | null
+```
+
+### Decrypt
+
+```ts
+await coreApi.decrypt('connections', id);                    // null on failure (logs a warn)
+await coreApi.decrypt('connections', id, { throwOnError: true });
+```
+
+### Acting as an end user (`asUser`)
+
+By default the client authenticates with the **functional `x-api-key`** (the
+app/system actor). To make core attribute a call to the **human user** — and
+apply *their* RBAC — derive a scoped client that forwards their session:
+
+```ts
+// attribute the write to the user (drops the functional key):
+await coreApi.asUser({ authorization: req.headers.authorization }).create('notes', body);
+
+// forward the cookie session, keep the functional key too (app identity + user):
+await coreApi.asUser({ cookie: req.headers.cookie }, { keepApiKey: true })
+  .uploadFile(bytes, { filename, mimeType });
+
+// works for every method — reads, writes, upload, download, decrypt:
+await coreApi.asUser({ jwt }).list('my_private_things');
+```
+
+The scoped client **inherits** the parent's `retry` / `validateXrefs` / `logger`
+config.
+
+### Reliability: automatic retry
+
+Off by default. When enabled, requests retry on configured statuses (default
+`[429]`), honouring `Retry-After` (header **or** a "retry in N seconds" body),
+with jitter and a per-wait cap. Essential for fire-and-forget writes
+(transcript/audit/usage) that core may rate-limit.
+
+```ts
+new CoreApiClient({
+  baseUrl, apiKey,
+  retry: {
+    maxAttempts: 5,        // default 5 when `retry` is set
+    retryOn: [429],        // default [429]; add 503 etc. if you want
+    honorRetryAfter: true, // default true
+    maxDelayMs: 20_000,    // default 20s cap per wait
+  },
+});
+```
+
+### Safety: x-ref validation
+
+Off by default. When enabled, the client loads each collection's `items`
+schema (cached, single-flight), and before every write checks that x-ref
+fields hold valid 24-hex ObjectIds — catching a class of bugs where a name or
+slug is passed where an id belongs.
+
+```ts
+new CoreApiClient({
+  baseUrl, apiKey,
+  validateXrefs: true,
+  onInvalidXref: 'warn',   // 'warn' (default, logs) or 'throw' (refuses the write)
 });
+```
+
+The building blocks are also exported for custom use:
+`extractXrefFields(schema)`, `checkXrefFields(body, map)`.
+
+### Error handling
 
-// 3. Auth: verify users against core on protected routes.
-app.addHook('preHandler', createFastifyAuthPreHandler(coreUrl)); // sets request.user
+Reads return `null` for genuine "not found" (`get`/`findBy` on 404). Everything
+else **throws `CoreApiError`** on a non-2xx response:
+
+```ts
+import { CoreApiError } from '@loynazkovacs/theitemapp-backend-sdk';
+
+try {
+  await coreApi.update('apps', id, body);
+} catch (err) {
+  if (err instanceof CoreApiError) {
+    err.status;     // 409
+    err.method;     // 'UPDATE'
+    err.collection; // 'apps'
+    err.target;     // 'apps/<id>'
+    err.body;       // raw response body
+  }
+}
 ```
 
-### Framework-agnostic auth
+Prefer a null-returns contract? Wrap the call:
+`const doc = await coreApi.create(c, b).catch(() => null);`
 
-If you're not on Express/Fastify, call the core directly:
+### Full config reference
 
 ```ts
-import { verifyUser, userInAnyGroup } from '@loynazkovacs/theitemapp-backend-sdk';
+new CoreApiClient({
+  baseUrl: 'http://backend:3001', // required
+  apiKey: null,                   // functional key (set later via updateApiKey)
+  skipWebhooks: true,             // default true (x-theitemapp-skip-webhooks on writes)
+  retry: { /* RetryConfig */ },   // default off
+  validateXrefs: false,           // default off
+  onInvalidXref: 'warn',          // 'warn' | 'throw'
+  logger: console,                // { info?, warn?, error? } — default console
+});
+```
+
+---
+
+## Pillar 3 — Auth
+
+App backends don't verify JWTs themselves by default — they forward the
+caller's `Authorization`/`Cookie` to core's `GET /api/auth/me`, the single
+source of truth for identity and group membership.
 
-const user = await verifyUser(coreUrl, { cookie, authorization });
-if (!user) return reply401();
+```ts
+import {
+  verifyUser, userInAnyGroup, normalizeId,
+  createExpressAuthMiddleware, createFastifyAuthPreHandler,
+  verifyJwtLocal,
+} from '@loynazkovacs/theitemapp-backend-sdk';
 ```
 
-## Exports
+### Verify against core (default)
 
-| Export | Purpose |
-| --- | --- |
-| `CoreApiClient`, `CoreApiError` | Typed CRUD over `/api/dynamic`: `list`/`get`/`findBy`/`count`/`create`/`bulkCreate`/`update`/`updateAsUser`/`delete`/`hardDeleteByFilter`/`upsert`/`upsertOn`/`bulkUpsert`/`uploadFile`/`downloadFile`/`decrypt`. |
-| `client.asUser(creds, opts?)` | Scoped client that acts AS an end user — forwards their `Authorization`/`Cookie` (optionally alongside the functional key). Full method surface. |
-| `startAppRegistration` | Registration retry loop, API-key rotation capture, heartbeat, deregister, signal handlers. |
-| `verifyUser`, `normalizeId`, `userInAnyGroup`, `verifyJwtLocal` | Framework-agnostic auth primitives (+ offline HS256 verification). |
-| `createExpressAuthMiddleware`, `createFastifyAuthPreHandler` | Drop-in middleware/preHandler. |
+```ts
+const user = await verifyUser(coreUrl, {
+  cookie: req.headers.cookie,
+  authorization: req.headers.authorization,
+});
+// → { _id, username, email?, groupIds, raw } | null
+//   null = no credentials or core rejected them; throws only if core is unreachable.
+```
 
-### Read/write controls
+> Core's `/api/auth/me` returns the user id as **`id`** (the JWT `sub`), not
+> `_id` — `verifyUser` handles this; `user._id` is always populated.
 
-- `get(c, id, { populate: false })` — keep x-ref fields as id strings (`?populate=0`).
-- `create`/`update`/`upsertOn`/`bulkCreate` take an optional `WriteOptions` (`{ skipWebhooks?, headers? }`) for per-call control.
-- `upsertOn(c, matchOn[], doc)` — composite-key atomic upsert returning `{ ok, created, doc }`.
-- `decrypt(c, id, { throwOnError: true })` — throw instead of returning `null`.
+Group gating:
 
-### Acting as an end user
+```ts
+if (!userInAnyGroup(user, adminGroupIds)) return reply.code(403).send();
+```
+
+### Drop-in middleware
 
 ```ts
-// forward the caller's session so core attributes the write to them:
-await coreApi.asUser({ authorization: req.headers.authorization, cookie: req.headers.cookie })
-  .create('notes', { text: 'hi' });
+// Fastify — assigns request.user
+app.addHook('preHandler', createFastifyAuthPreHandler(coreUrl));
+
+// Express — assigns res.locals.user
+app.use(createExpressAuthMiddleware(coreUrl));
+```
+
+Both respond `401` (no/invalid creds) or `503` (core unreachable) themselves.
 
-// keep the functional key too (app identity + user session):
-await coreApi.asUser({ jwt }, { keepApiKey: true }).uploadFile(bytes, { filename, mimeType });
+### Offline verification
+
+For backends that hold core's `JWT_SECRET` and want to skip the network round
+trip (e.g. high-frequency idempotency checks):
+
+```ts
+const payload = verifyJwtLocal(token, process.env.JWT_SECRET!);
+// → decoded HS256 payload | null (null = malformed / bad signature / expired / not HS256)
 ```
 
-## Not yet covered (roadmap)
+---
 
-- `aggregate()` — core exposes pivot aggregation through the dynamic list route
-  (`_agg`), which isn't a clean stable HTTP contract yet. Use `list()` with
-  query params for now.
-- Seed/function HTTP serving — that belongs to apps without a backend, handled
-  by the `seed-server` image.
+## Adoption recipe (existing backends)
 
-## Publishing
+Migrate a hand-rolled backend onto the SDK with **zero behaviour change** by
+making each existing module a thin shell over the SDK:
 
-Same flow as the frontend SDK:
+1. Add the dependency (`^0.4.0`). If the Dockerfile uses `npm ci`, regenerate
+   `package-lock.json`.
+2. **Registration** → replace the register/retry/heartbeat/deregister block
+   with `startAppRegistration` (`installSignalHandlers: false` if you have a
+   custom shutdown).
+3. **coreApiClient.ts** → keep your exported class/signatures, but back it with
+   an internal `CoreApiClient` and delegate. Preserve your conventions via
+   config: `populate=false` reads, `retry` for a 429 layer, `validateXrefs` for
+   id checks, `asUser` for user-attributed writes. Re-export your `DynRow`/error
+   types so call-sites don't change.
+4. **Auth** → swap your `/api/auth/me` proxy internals for `verifyUser` (or
+   `verifyJwtLocal` if you verify offline). Keep your middleware's exported shape.
+5. Typecheck, deploy, verify (registration log, auth 401/200, a real read/write),
+   then commit + push.
+
+The `system` backend is the reference implementation; see the
+`backend-sdk-adoption` agent skill for the full per-app checklist.
+
+---
+
+## Versioning & publishing
+
+Published from `main` by the `Build Images` workflow (`publish-backend-sdk`
+job) on any change under `libs/backend-sdk/**`. The job is idempotent (skips if
+the version already exists) and **fails loudly** if a publish genuinely fails.
 
 ```bash
 cd libs/backend-sdk
 npm run build      # tsc → dist/
 npm publish        # uses .npmrc registry + publishConfig (public)
 ```
+
+See [CHANGELOG.md](./CHANGELOG.md) for version history. Caret ranges on `0.x`
+do **not** auto-bump across minors (`^0.3.0` ≠ `0.4.0`), so pinned apps upgrade
+deliberately.
+
+---
+
+## API index
+
+**Core client:** `CoreApiClient` · `CoreApiError` ·
+`list` `get` `findBy` `count` `create` `bulkCreate` `update` `updateAsUser`
+`delete` `hardDeleteByFilter` `upsert` `upsertOn` `bulkUpsert` `uploadFile`
+`downloadFile` `decrypt` · `asUser` · `updateApiKey` `getApiKey` `isReady`
+`loadXrefFields`
+
+**Utilities:** `extractXrefFields` · `checkXrefFields` · `deepEqual`
+
+**Registration:** `startAppRegistration` (`RegistrationHandle`, `AppManifest`)
+
+**Auth:** `verifyUser` · `verifyJwtLocal` · `userInAnyGroup` · `normalizeId` ·
+`createExpressAuthMiddleware` · `createFastifyAuthPreHandler`
+
+**Types:** `CoreApiConfig` `RetryConfig` `SdkLogger` `WriteOptions` `GetOptions`
+`UserCredentials` `AuthUser` `JwtPayload` `BulkCreateResult` `UpsertResult`
+`BulkUpsertResult` `UploadFileOptions` `DownloadedFile` `XrefFieldMap`
+
+## Not covered (by design)
+
+- `aggregate()` — core's pivot aggregation rides the dynamic list route
+  (`_agg`), not a stable HTTP contract yet. Use `list()` with params.
+- Seed/function HTTP serving — that's the `seed-server` image's job (apps
+  without a backend).