api-observe 1.1.2 → 1.1.3

package/README.md

@@ -8,9 +8,12 @@ Works with **Fastify**, **Express**, **NestJS**, **Koa**, or **plain Node.js HTT
 
 - Captures upstream API failures (via Axios interceptors)
 - Captures controller/route-level errors (via framework hooks)
+- **Failure records are keyed to the inbound request** — when `/parties` calls `/api/aggregate` and it fails, the log entry shows `/parties` at the top level with the full `/api/aggregate` failure nested inside
 - Built-in HTML dashboard at `/observe`
 - REST API for querying failures programmatically
 - In-memory ring buffer with configurable size and TTL
+- **`onCapture` hook** — persist failures to your own database alongside the in-memory store
+- **Custom store** — replace the in-memory store entirely with a database-backed implementation
 - Automatic redaction of sensitive headers (authorization, tokens, passwords)
 - Zero external dependencies
 
@@ -55,6 +58,15 @@ The Fastify plugin automatically captures:
 - **Controller errors** - any error thrown during the request lifecycle (via `onError` hook)
 - **404s** - requests to non-existent routes (via `onResponse` hook)
 
+#### How upstream failures are recorded
+
+When an inbound request (e.g. `POST /parties`) triggers a downstream call that fails (e.g. `GET /api/aggregate` returns 500), the failure record is keyed to the **inbound** route — not the downstream URL. Opening the record in the dashboard shows:
+
+- **Top level**: `POST /parties` — the service, method, URL, correlation ID, and inbound request headers/body
+- **Upstream Call** section: the downstream service name, `GET /api/aggregate`, status code, duration, error message, outgoing request headers/body, and the downstream response
+
+This lets you immediately answer "which of my endpoints is broken?" rather than having to trace a downstream URL back to the calling route.
+
 ---
 
 ### Express
@@ -183,9 +195,12 @@ bootstrap();
 Visit `/observe` in your browser to access the built-in dashboard.
 
 The dashboard provides:
-- Real-time failure table with filtering by type, service, method, status code, and URL
+- Real-time failure table with filtering by type, service, method, status code, URL, and correlation ID
 - Summary cards showing total failures, upstream vs controller errors, and affected services
-- Detail modal with full request/response headers, bodies, and stack traces
+- Detail modal with:
+  - **Inbound request** headers, query params, and body (the route your app received)
+  - **Upstream Call** section — downstream service name, URL, status, duration, outgoing request, and downstream response headers/body (only shown for upstream failures)
+  - Stack trace (for controller errors)
 - Auto-refresh every 30 seconds
 - Pagination for large result sets
 
@@ -210,6 +225,7 @@ All endpoints are served under `/observe/api/`:
 | `method` | string | Filter by HTTP method (`GET`, `POST`, etc.) |
 | `statusCode` | number | Filter by response status code |
 | `url` | string | Filter by URL (partial match) |
+| `correlationId` | string | Filter by correlation ID (partial match) |
 | `from` | string | Start date (ISO 8601) |
 | `to` | string | End date (ISO 8601) |
 | `limit` | number | Results per page (default: 50) |
@@ -221,6 +237,9 @@ All endpoints are served under `/observe/api/`:
 # Get all 5xx failures from the auth service
 curl "http://localhost:3000/observe/api/failures?service=auth&statusCode=500"
 
+# Get failures by correlation ID
+curl "http://localhost:3000/observe/api/failures?correlationId=abc-123"
+
 # Get summary stats
 curl "http://localhost:3000/observe/api/summary"
 ```
@@ -234,6 +253,8 @@ All adapters accept the same options:
 | `maxEntries` | number | `500` | Maximum failure records to keep in memory |
 | `ttlMs` | number | `86400000` (24h) | Time-to-live in ms before entries are evicted |
 | `sanitize` | function | built-in | Custom sanitizer for request/response bodies |
+| `onCapture` | function | — | Called with each captured record; use to persist to a database |
+| `store` | object | — | Replace the in-memory store with a custom implementation (see [Persisting Failures](#persisting-failures-to-a-database)) |
 
 ### Default Sanitization
 
@@ -369,6 +390,116 @@ observe.attachToAll(services);
 - Direct `BaseHttpClient` instances (has `_axios` + `serviceName`)
 - Service wrappers with a `.client` property that is a `BaseHttpClient`
 
+## Persisting Failures to a Database
+
+By default `api-observe` keeps failures in an in-memory ring buffer. Two options let you persist them externally.
+
+---
+
+### Option A — `onCapture` callback (recommended for most apps)
+
+Pass an `onCapture` function. It is called with every captured record immediately after it is stored in memory. Use it to write to your own database while still benefiting from the built-in dashboard and in-memory query API.
+
+```js
+// Fastify
+await fastify.register(apiObserve, {
+  onCapture: async (record) => {
+    await db.collection('api_failures').insertOne(record);
+  },
+});
+
+// Express
+const observe = expressMiddleware({
+  onCapture: async (record) => {
+    await db.query(
+      'INSERT INTO api_failures(data) VALUES($1)',
+      [JSON.stringify(record)],
+    );
+  },
+});
+```
+
+`record` is the fully-formed failure object (same shape as returned by `GET /observe/api/failures/:id`). Errors thrown inside `onCapture` are silently swallowed so a failing hook never disrupts your application.
+
+---
+
+### Option B — Custom store (full replacement)
+
+Pass your own `store` object to replace the in-memory store entirely. The dashboard and REST API will query it, so failures are served from your database.
+
+Your store must implement this interface:
+
+```ts
+interface FailureStore {
+  add(failure: object): object;          // store a record, return it with id + timestamp
+  query(filters: object): { data: object[], total: number, limit: number, offset: number };
+  getById(id: number | string): object | null;
+  summary(): object;
+  clear(): void;
+}
+```
+
+#### Example — MongoDB-backed store
+
+```js
+class MongoFailureStore {
+  constructor(collection) {
+    this._col = collection;
+    this._seq = 0;
+  }
+
+  async add(failure) {
+    const record = { id: ++this._seq, timestamp: new Date().toISOString(), ...failure };
+    await this._col.insertOne(record);
+    return record;
+  }
+
+  async query({ service, statusCode, method, url, correlationId, type, limit = 50, offset = 0 } = {}) {
+    const filter = {};
+    if (type)          filter.type = type;
+    if (service)       filter.service = new RegExp(service, 'i');
+    if (method)        filter.method = method.toUpperCase();
+    if (statusCode)    filter.statusCode = Number(statusCode);
+    if (url)           filter.url = new RegExp(url, 'i');
+    if (correlationId) filter.correlationId = new RegExp(correlationId, 'i');
+
+    const total = await this._col.countDocuments(filter);
+    const data  = await this._col.find(filter).sort({ timestamp: -1 }).skip(offset).limit(limit).toArray();
+    return { data, total, limit, offset };
+  }
+
+  async getById(id) {
+    return this._col.findOne({ id: Number(id) }) ?? null;
+  }
+
+  async summary() {
+    const all = await this._col.find({}).toArray();
+    const byService = {}, byStatusCode = {}, byType = {};
+    for (const e of all) {
+      byService[e.service || 'unknown'] = (byService[e.service || 'unknown'] || 0) + 1;
+      const code = e.statusCode || 'unknown';
+      byStatusCode[code] = (byStatusCode[code] || 0) + 1;
+      const type = e.type || 'upstream';
+      byType[type] = (byType[type] || 0) + 1;
+    }
+    return { totalFailures: all.length, byService, byStatusCode, byType };
+  }
+
+  async clear() {
+    await this._col.deleteMany({});
+  }
+}
+
+// Pass it to the adapter
+const observe = expressMiddleware({
+  store: new MongoFailureStore(db.collection('api_failures')),
+});
+```
+
+> **Note:** When using a custom async store, your `query` / `getById` / `summary` methods may return Promises — the built-in router will await them automatically.
+
+---
+
 ## Exports
 
 ```js

package/lib/adapters/express.js

@@ -27,7 +27,7 @@
  */
 
 const { parse: parseUrl } = require('url');
-const { FailureStore } = require('../failure-store');
+const { FailureStore, resolveStore } = require('../failure-store');
 const { attachInterceptor, attachToAll, defaultSanitize } = require('../interceptor');
 const { createRouter } = require('../router');
 const { requestContext } = require('../request-context');
@@ -39,10 +39,7 @@ const { requestContext } = require('../request-context');
  * @returns {function} Express middleware with .attach(), .attachToAll(), .store, .errorHandler()
  */
 function expressMiddleware(opts = {}) {
-  const store = new FailureStore({
-    maxEntries: opts.maxEntries,
-    ttlMs: opts.ttlMs,
-  });
+  const store = resolveStore(opts);
 
   const sanitize = opts.sanitize ?? defaultSanitize;
   const sanitizeOpts = opts.sanitize ? { sanitize: opts.sanitize } : {};
@@ -59,19 +56,18 @@ function expressMiddleware(opts = {}) {
       return;
     }
 
-    const result = router(req.method, pathname, parsed.query);
-    if (!result) {
-      next();
-      return;
-    }
-
-    res.statusCode = result.status;
-    for (const [key, value] of Object.entries(result.headers)) {
-      res.setHeader(key, value);
-    }
-
-    const body = typeof result.body === 'string' ? result.body : JSON.stringify(result.body);
-    res.end(body);
+    router(req.method, pathname, parsed.query).then(result => {
+      if (!result) {
+        next();
+        return;
+      }
+      res.statusCode = result.status;
+      for (const [key, value] of Object.entries(result.headers)) {
+        res.setHeader(key, value);
+      }
+      const body = typeof result.body === 'string' ? result.body : JSON.stringify(result.body);
+      res.end(body);
+    }).catch(next);
   }
 
   // ── Error-capturing middleware (use AFTER your routes) ───────────────────

package/lib/adapters/http.js

@@ -34,7 +34,7 @@
  */
 
 const { parse: parseUrl } = require('url');
-const { FailureStore } = require('../failure-store');
+const { FailureStore, resolveStore } = require('../failure-store');
 const { attachInterceptor, attachToAll, defaultSanitize } = require('../interceptor');
 const { createRouter } = require('../router');
 
@@ -45,10 +45,7 @@ const { createRouter } = require('../router');
  * @returns {object} Handler with .handle(), .attach(), .attachToAll(), .captureError(), .store
  */
 function createHttpHandler(opts = {}) {
-  const store = new FailureStore({
-    maxEntries: opts.maxEntries,
-    ttlMs: opts.ttlMs,
-  });
+  const store = resolveStore(opts);
 
   const sanitize = opts.sanitize ?? defaultSanitize;
   const sanitizeOpts = opts.sanitize ? { sanitize: opts.sanitize } : {};
@@ -70,18 +67,24 @@ function createHttpHandler(opts = {}) {
       return false;
     }
 
-    const result = router(req.method, pathname, parsed.query);
-    if (!result) {
-      return false;
-    }
-
-    res.statusCode = result.status;
-    for (const [key, value] of Object.entries(result.headers)) {
-      res.setHeader(key, value);
-    }
-
-    const body = typeof result.body === 'string' ? result.body : JSON.stringify(result.body);
-    res.end(body);
+    // routeHandler is async (supports custom async stores); signal handled
+    // synchronously by returning true, then resolve asynchronously.
+    router(req.method, pathname, parsed.query).then(result => {
+      if (!result) {
+        res.statusCode = 404;
+        res.end(JSON.stringify({ error: 'Not found' }));
+        return;
+      }
+      res.statusCode = result.status;
+      for (const [key, value] of Object.entries(result.headers)) {
+        res.setHeader(key, value);
+      }
+      const body = typeof result.body === 'string' ? result.body : JSON.stringify(result.body);
+      res.end(body);
+    }).catch(err => {
+      res.statusCode = 500;
+      res.end(JSON.stringify({ error: 'Internal error' }));
+    });
     return true;
   }
 

package/lib/failure-store.js

@@ -13,13 +13,14 @@
 
 class FailureStore {
   /**
-   * @param {{ maxEntries?: number, ttlMs?: number }} [opts]
+   * @param {{ maxEntries?: number, ttlMs?: number, onCapture?: (record: object) => void }} [opts]
    */
   constructor(opts = {}) {
     this._maxEntries = opts.maxEntries ?? 500;
     this._ttlMs = opts.ttlMs ?? 24 * 60 * 60 * 1000; // 24 hours
     this._entries = [];
     this._idCounter = 0;
+    this._onCapture = typeof opts.onCapture === 'function' ? opts.onCapture : null;
   }
 
   /**
@@ -41,6 +42,12 @@ class FailureStore {
       this._entries = this._entries.slice(-this._maxEntries);
     }
 
+    // Fire the capture hook (e.g. persist to an external database).
+    // Errors are swallowed so a failing hook never disrupts the app.
+    if (this._onCapture) {
+      try { this._onCapture(record); } catch (_) {}
+    }
+
     return record;
   }
 
@@ -161,3 +168,33 @@ class FailureStore {
 }
 
 module.exports = { FailureStore };
+
+/**
+ * Resolve a store from adapter options.
+ *
+ * - If `opts.store` is provided it is used as-is (custom/DB-backed store).
+ *   It must implement: add(failure), query(filters), getById(id), summary(), clear().
+ * - Otherwise a new FailureStore is created, optionally with an onCapture hook.
+ *
+ * @param {{ store?: object, maxEntries?: number, ttlMs?: number, onCapture?: function }} opts
+ * @returns {FailureStore|object}
+ */
+function resolveStore(opts = {}) {
+  if (opts.store) {
+    const required = ['add', 'query', 'getById', 'summary', 'clear'];
+    for (const method of required) {
+      if (typeof opts.store[method] !== 'function') {
+        throw new Error(`api-observe: custom store is missing required method "${method}"`);
+      }
+    }
+    return opts.store;
+  }
+
+  return new FailureStore({
+    maxEntries: opts.maxEntries,
+    ttlMs: opts.ttlMs,
+    onCapture: opts.onCapture,
+  });
+}
+
+module.exports = { FailureStore, resolveStore };

package/lib/plugin.js

@@ -50,7 +50,7 @@
  *  recording a duplicate when the same request has already been captured.
  */
 
-const { FailureStore } = require('./failure-store');
+const { FailureStore, resolveStore } = require('./failure-store');
 const { attachInterceptor, attachToAll, defaultSanitize } = require('./interceptor');
 const { registerRoutes } = require('./routes');
 const { requestContext } = require('./request-context');
@@ -58,10 +58,7 @@ const { requestContext } = require('./request-context');
 const CTX_KEY = Symbol.for('ps-observe.ctx');
 
 async function observerPlugin(fastify, opts) {
-  const store = new FailureStore({
-    maxEntries: opts.maxEntries,
-    ttlMs: opts.ttlMs,
-  });
+  const store = resolveStore(opts);
 
   const sanitizeOpts = opts.sanitize ? { sanitize: opts.sanitize } : {};
   const sanitize = opts.sanitize ?? defaultSanitize;

package/lib/router.js

@@ -25,9 +25,9 @@ function createRouter(store) {
    * @param {string} pathname - URL path, e.g. "/observe/api/failures/3"
    * @param {object} query   - Parsed query parameters
    * @returns {{ status: number, headers: object, body: any } | null}
-   *          null means "no matching route"
+   *          null means "no matching route" — may be a Promise for async stores
    */
-  return function routeHandler(method, pathname, query) {
+  return async function routeHandler(method, pathname, query) {
     // Normalise: strip trailing slash (except root "/observe")
     const path = pathname.replace(/\/$/, '') || '/';
 
@@ -43,14 +43,14 @@ function createRouter(store) {
     // ── List failures ──────────────────────────────────────────────────────
     if (method === 'GET' && path === '/observe/api/failures') {
       const { service, statusCode, method: m, url, from, to, limit, offset, type, correlationId } = query || {};
-      const result = store.query({ service, statusCode, method: m, url, from, to, limit, offset, type, correlationId });
+      const result = await store.query({ service, statusCode, method: m, url, from, to, limit, offset, type, correlationId });
       return { status: 200, headers: { 'content-type': 'application/json' }, body: result };
     }
 
     // ── Single failure detail ──────────────────────────────────────────────
     const detailMatch = path.match(/^\/observe\/api\/failures\/(\d+)$/);
     if (method === 'GET' && detailMatch) {
-      const record = store.getById(detailMatch[1]);
+      const record = await store.getById(detailMatch[1]);
       if (!record) {
         return { status: 404, headers: { 'content-type': 'application/json' }, body: { error: 'Failure record not found' } };
       }
@@ -59,12 +59,12 @@ function createRouter(store) {
 
     // ── Summary stats ──────────────────────────────────────────────────────
     if (method === 'GET' && path === '/observe/api/summary') {
-      return { status: 200, headers: { 'content-type': 'application/json' }, body: store.summary() };
+      return { status: 200, headers: { 'content-type': 'application/json' }, body: await store.summary() };
     }
 
     // ── Clear all ──────────────────────────────────────────────────────────
     if (method === 'DELETE' && path === '/observe/api/failures') {
-      store.clear();
+      await store.clear();
       return { status: 200, headers: { 'content-type': 'application/json' }, body: { cleared: true } };
     }
 

package/lib/routes.js

@@ -22,9 +22,9 @@ function registerRoutes(fastify, store) {
   const router = createRouter(store);
 
   // Catch-all handler for /observe routes
-  const handler = (req, reply) => {
+  const handler = async (req, reply) => {
     const url = new URL(req.url, 'http://localhost');
-    const result = router(req.method, url.pathname, req.query);
+    const result = await router(req.method, url.pathname, req.query);
     if (!result) {
       reply.code(404).send({ error: 'Not found' });
       return;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "api-observe",
-  "version": "1.1.2",
+  "version": "1.1.3",
   "description": "Framework-agnostic plugin that captures and displays API failure details — works with Fastify, Express, NestJS, Koa, or plain Node.js HTTP",
   "main": "lib/index.js",
   "files": [