midline-agent 0.2.0 → 0.4.0

package/README.md

@@ -1,6 +1,7 @@
 # midline-agent
 
 **The Node.js SDK for Midline** — request and error monitoring with security and threat detection.
+It also ships **`midline-agent/browser`** for web apps: see [Browser apps](#browser-apps).
 
 Midline itself is not tied to Node. Every event lands on the same stream through a plain JSON
 endpoint, so a Django, Rails, Laravel, Spring, Go or .NET service reports exactly what an Express
@@ -102,7 +103,8 @@ bootstrap();
 ```
 
 Nest handles exceptions in its own filters, so they rarely reach Express error middleware. The
-request middleware still records every 4xx/5xx response.
+request middleware still records every 4xx/5xx response, and Midline groups the 5xx responses into
+Issues by method, route and status.
 
 ### Plain Node `http`
 
@@ -246,6 +248,7 @@ MidlineAgent.init({
   },
   redactFields?: string[],        // added to the built-in list (maskFields still works)
   redactHeaders?: string[],
+  captureConsole?: boolean,       // MIDLINE_CAPTURE_CONSOLE — also send what the process prints (off by default)
 
   // Delivery
   flushIntervalMs?: number,       // default 1500
@@ -266,6 +269,7 @@ MidlineAgent.init({
 ```env
 MIDLINE_API_KEY=...
 MIDLINE_ENDPOINT=https://api.usemidline.com
+# MIDLINE_CAPTURE_CONSOLE=true
 
 # proxy mode
 TARGET_API_URL=http://localhost:4000
@@ -278,6 +282,33 @@ TARGET_API_URL=http://localhost:4000
 
 ---
 
+## Console output
+
+Set `captureConsole: true` (or `MIDLINE_CAPTURE_CONSOLE=true`) and whatever the process prints
+is sent as `console` events, one per line, and shows up on the dashboard's **Terminal** page. That
+covers `console.log`, Nest's logger, pino, winston: anything written to stdout or stderr. The Logs page
+stays request traffic.
+
+```ts
+MidlineAgent.init({ apiKey: process.env.MIDLINE_API_KEY, captureConsole: true });
+const app = await NestFactory.create(AppModule); // startup lines are captured from here on
+```
+
+- Initialise the agent before creating the app. Anything printed before `init()` isn't captured.
+- Your terminal output doesn't change. The agent reads each write after it has happened.
+- Colour codes are stripped, each line is capped at 4096 characters, and lines are redacted like any
+  other text.
+- Severity comes from the line: `ERROR` or `FATAL` is high, `WARN` is medium, other stderr output is
+  medium, and everything else is low.
+- Up to 100 lines a second are sent, with room for a 1,000-line burst such as Nest mapping its routes
+  at startup. Lines past that still print but aren't sent.
+- Printed lines don't count towards request totals, error rates, latency or Issues.
+- It needs a Midline server that knows the `console` event type. An older server refuses the first
+  line; the agent then turns console capture off, says so once, and keeps sending requests and errors.
+- The agent's own diagnostics are never captured.
+
+---
+
 ## Sensitive data
 
 Redaction happens **in your process, before an event is queued**. What's masked never reaches a
@@ -293,7 +324,7 @@ defence.
   `privatekey`, `authorization`, `cookie`, `session`, `credential`, `csrf`, `xsrf`, `signature`,
   `creditcard`, `cardnumber`, `cvv`, `cvc`, `ssn`, `socialsecurity`; and the exact keys `auth`,
   `pwd`, `pin`, `otp`, `sid`, `jwt`, `bearer`.
-- **Values inside any text** — error messages, stack traces, routes: bearer/basic credentials,
+- **Values inside any text** — error messages, stack traces, routes, console lines: bearer/basic credentials,
   JWTs, Midline keys (`ak_…`), Stripe and AWS key formats, PEM private keys, `user:pass@` in URLs,
   and `key=value` / `"key": value` pairs whose key is sensitive.
 - Query strings are never part of `route`.
@@ -344,6 +375,88 @@ app.get("/x", (req, res) => { logger.info({ requestId: getRequestContext(req)?.r
 
 ---
 
+## Browser apps
+
+`midline-agent/browser` monitors a web app from the page: uncaught errors and unhandled rejections,
+failed `fetch` and `XMLHttpRequest` calls, Web Vitals (LCP, INP, CLS, FCP, TTFB) and, if you ask,
+console output. It has no dependencies, no Node code, and works under React, Vue, Angular, Svelte,
+Next.js or no framework at all, because it instruments the page rather than a framework.
+
+```ts
+import * as Midline from "midline-agent/browser";
+
+Midline.init({
+  apiKey: import.meta.env.VITE_MIDLINE_BROWSER_KEY, // pk_… browser key
+  service: "checkout-web",
+  environment: import.meta.env.MODE,
+  release: import.meta.env.VITE_RELEASE,
+});
+```
+
+**Use a browser key, never a server key.** Anything in a bundle can be read by whoever loads the page,
+so browser keys (`pk_…`) are public by design and the Midline server accepts them only from the
+origins listed on the key (create one under **Project → API Keys → Browser**). Server keys (`ak_…`)
+are refused whenever a browser sends them, and the SDK won't start with one.
+
+Errors a framework catches itself never reach the window. Forward them:
+
+```tsx
+// React error boundary
+componentDidCatch(error: Error, info: ErrorInfo) {
+  Midline.captureException(error, { extra: { componentStack: info.componentStack } });
+}
+
+// Vue
+app.config.errorHandler = (error, _instance, info) => Midline.captureException(error, { extra: { info } });
+```
+
+**Linking a page to your backend.** Same-origin requests get a W3C `traceparent` header, which the
+Node agent on the backend turns into `traceId`/`spanId` (see above), so the page's failed call and
+the server request behind it share a trace id. For an API on another origin, list it in
+`tracePropagationTargets` and allow the `traceparent` header in that API's CORS configuration, or
+the browser will block the call's preflight.
+
+| Option | Default | |
+| --- | --- | --- |
+| `apiKey` | — | Browser key (`pk_…`). |
+| `endpoint` | `https://api.usemidline.com` | `http://` only for localhost. |
+| `service`, `environment`, `release` | — | Stamped on every event. |
+| `captureErrors` | `true` | Uncaught errors and unhandled rejections. |
+| `captureRequests` | `"failed"` | `"failed"` (4xx, 5xx, network errors), `"all"`, or `false`. Every call is a breadcrumb either way. |
+| `captureWebVitals` | `true` | Reported once, when the page is first hidden. |
+| `captureConsole` | `false` | `true` for `error` and `warn`, or a list of levels. Lines appear on the Terminal page. Opt-in because wrapped console calls show the SDK as their source in devtools. |
+| `tracePropagationTargets` | same origin | Strings match as URL prefixes (or path prefixes starting with `/`); RegExps match the full URL. |
+| `ignoreErrors`, `ignoreUrls` | `[]` | Strings match as substrings. |
+| `sampleRate` | `1` | Fraction of events sent. |
+| `maxEventsPerMinute` | `120` | So an error in a render loop can't flood the project. |
+| `beforeSend` | — | Return `null` to drop an event; edit `payload` and `metadata` freely. |
+| `redactFields` | — | Extra field names to redact. |
+| `enabled`, `debug` | `true`, `false` | |
+
+Also: `captureMessage(message, severity)`, `setUser({ id })`, `setTag(key, value)`,
+`addBreadcrumb(message)`, `flush()` and `close()`, which restores everything the SDK wrapped.
+
+What it will not do: it never sends cookies, query strings of the page URL, or request and response
+bodies; it redacts the same credential patterns as the Node agent before an event is queued; it
+never throws into your code; and if Midline is down it keeps a bounded queue and backs off. Pending
+events leave with `keepalive` when the tab is hidden. During server-side rendering, `init` does
+nothing.
+
+No bundler? Load the ES module build directly:
+
+```html
+<script type="module">
+  import * as Midline from "https://cdn.jsdelivr.net/npm/midline-agent@0.4.0/dist/esm/browser/index.js";
+  Midline.init({ apiKey: "pk_…", service: "web" });
+</script>
+```
+
+The Midline server must be recent enough to accept browser keys (it answers the CORS preflight on
+the ingest routes). Against an older server the SDK logs nothing unless `debug` is on and keeps
+retrying with backoff.
+
+---
+
 ## Any other backend
 
 The agent does two things: it turns a request or an error into an event, and it POSTs that event to

package/browser/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "midline-agent/browser",
+  "private": true,
+  "main": "../dist/browser/index.js",
+  "module": "../dist/esm/browser/index.js",
+  "types": "../dist/browser/index.d.ts",
+  "sideEffects": false
+}

package/dist/agent.d.ts

@@ -34,7 +34,7 @@ export interface ExchangeMessage {
     truncated?: boolean;
 }
 /**
- * Ships request/error events to the Midline server.
+ * Ships request, error and console events to the Midline server.
  *
  * The contract with the host application is that the agent is never allowed to
  * affect it: an unreachable, untrusted, slow or misconfigured Midline server costs a
@@ -65,6 +65,11 @@ export declare class MidlineAgent {
     private dropped;
     private maxRequestBytes;
     private batchLimit;
+    private consoleCapture;
+    private consoleTokens;
+    private consoleRefilledAt;
+    /** The server refused a console event: it predates them, so no more are sent. */
+    private consoleUnsupported;
     constructor(config?: MidlineConfig);
     /** False when the agent is off: no key, bad config, disabled, rejected key, or closed. */
     get active(): boolean;
@@ -73,6 +78,8 @@ export declare class MidlineAgent {
     addEvent(event: MidlineEvent): void;
     /** Records one HTTP exchange. Captured parts are redacted here, before queueing. */
     recordHttp(exchange: HttpExchange): void;
+    private recordConsole;
+    private takeConsoleToken;
     /** Sends whatever is buffered now, ignoring backoff, and keeps the process alive until done. */
     flush(): Promise<void>;
     /** Flushes with a deadline, then closes. For graceful shutdown. */
@@ -98,6 +105,12 @@ export declare class MidlineAgent {
     /** Returns false if delivery should stop for this drain. */
     private sendIndividually;
     private send;
+    /**
+     * True when a rejected event is a console line the server has no event type for,
+     * i.e. the server predates console capture. Capture switches itself off rather
+     * than paying a refused request for every line printed from then on.
+     */
+    private refusedConsole;
     /** One line per distinct fault, not one per failed event. */
     private report;
     private describe;

package/dist/agent.js

@@ -2,6 +2,7 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.MidlineAgent = exports.SDK_VERSION = void 0;
 const config_1 = require("./config");
+const console_1 = require("./console");
 const redact_1 = require("./redact");
 const transport_1 = require("./transport");
 exports.SDK_VERSION = (() => {
@@ -15,6 +16,10 @@ exports.SDK_VERSION = (() => {
 const MAX_REQUEST_BYTES = 512 * 1024;
 const MIN_REQUEST_BYTES = 16 * 1024;
 const MAX_QUEUE_BYTES = 16 * 1024 * 1024;
+/** Console lines: a startup burst (Nest mapping every route) fits, a log storm can't crowd out requests. */
+const CONSOLE_BURST_LINES = 1000;
+const CONSOLE_LINES_PER_SECOND = 100;
+const CONSOLE_SEVERITY = { info: "low", warn: "medium", error: "high" };
 /** Verification failures. Retrying is still right: they clear once the server is fixed. */
 const TLS_ERROR_REASONS = {
     DEPTH_ZERO_SELF_SIGNED_CERT: "presented a self-signed certificate",
@@ -31,7 +36,7 @@ const TLS_ERROR_REASONS = {
     HOSTNAME_MISMATCH: "presented a certificate issued for a different hostname",
 };
 /**
- * Ships request/error events to the Midline server.
+ * Ships request, error and console events to the Midline server.
  *
  * The contract with the host application is that the agent is never allowed to
  * affect it: an unreachable, untrusted, slow or misconfigured Midline server costs a
@@ -71,6 +76,11 @@ class MidlineAgent {
         this.dropped = 0;
         this.maxRequestBytes = MAX_REQUEST_BYTES;
         this.batchLimit = Number.MAX_SAFE_INTEGER;
+        this.consoleCapture = null;
+        this.consoleTokens = CONSOLE_BURST_LINES;
+        this.consoleRefilledAt = Date.now();
+        /** The server refused a console event: it predates them, so no more are sent. */
+        this.consoleUnsupported = false;
         this.onErrorHook = config.onError;
         this.debugLogs = config.debug ?? (0, config_1.envFlag)("MIDLINE_DEBUG") ?? false;
         this.redactor = new redact_1.Redactor([...(config.redactFields ?? []), ...(config.maskFields ?? [])], config.redactHeaders);
@@ -101,10 +111,15 @@ class MidlineAgent {
             userAgent: `midline-agent/${exports.SDK_VERSION} node/${process.version}`,
         });
         this.timer = setInterval(() => {
+            this.consoleCapture?.flushPending();
             void this.drain(false);
         }, resolved.flushIntervalMs);
         // Telemetry must never be the reason a process refuses to exit.
         this.timer.unref?.();
+        if (resolved.captureConsole) {
+            this.consoleCapture = new console_1.ConsoleCapture((line) => this.recordConsole(line));
+            this.consoleCapture.install();
+        }
     }
     /** False when the agent is off: no key, bad config, disabled, rejected key, or closed. */
     get active() {
@@ -165,10 +180,43 @@ class MidlineAgent {
             this.report("event-build", `midline: could not record a request (${err?.message}); skipped it.`);
         }
     }
+    recordConsole(line) {
+        if (!this.active || this.consoleUnsupported)
+            return;
+        if (!this.takeConsoleToken()) {
+            this.dropped += 1;
+            this.report("console-rate", `midline: console output is arriving faster than ${CONSOLE_LINES_PER_SECOND} lines/s; the extra lines still print but are not sent.`);
+            return;
+        }
+        try {
+            this.enqueue(this.toWire({
+                type: "console",
+                path: line.stream,
+                message: line.text,
+                timestamp: line.timestamp,
+                severity: CONSOLE_SEVERITY[line.level],
+                category: "application",
+                integration: "console",
+            }, false));
+        }
+        catch (err) {
+            this.report("event-build", `midline: could not record a console line (${err?.message}); skipped it.`);
+        }
+    }
+    takeConsoleToken() {
+        const now = Date.now();
+        this.consoleTokens = Math.min(CONSOLE_BURST_LINES, this.consoleTokens + ((now - this.consoleRefilledAt) / 1000) * CONSOLE_LINES_PER_SECOND);
+        this.consoleRefilledAt = now;
+        if (this.consoleTokens < 1)
+            return false;
+        this.consoleTokens -= 1;
+        return true;
+    }
     /** Sends whatever is buffered now, ignoring backoff, and keeps the process alive until done. */
     async flush() {
         if (!this.active)
             return;
+        this.consoleCapture?.flushPending(true);
         this.retryAfter = 0;
         if (this.drainPromise) {
             await this.drainPromise;
@@ -197,6 +245,8 @@ class MidlineAgent {
             clearInterval(this.timer);
             this.timer = null;
         }
+        this.consoleCapture?.uninstall();
+        this.consoleCapture = null;
         this.queue = [];
         this.queueBytes = 0;
         this.transport?.destroy();
@@ -241,13 +291,17 @@ class MidlineAgent {
     toWire(event, preRedacted) {
         const config = this.config;
         const type = event.type;
+        const isConsole = type === "console";
         const statusCode = Number.isInteger(event.statusCode) && event.statusCode >= 100 && event.statusCode <= 599
             ? event.statusCode
             : type === "error" ? 500 : undefined;
         const { severity, category } = classify(event, statusCode);
         const route = this.redactor.string(stripQuery(event.path) || "/", 2048);
         const payload = {};
-        if (type === "error" || event.message) {
+        if (isConsole) {
+            payload.message = this.redactor.string(event.message ?? "", console_1.MAX_LINE_CHARS);
+        }
+        else if (type === "error" || event.message) {
             payload.error = event.message ? this.redactor.string(event.message, 1024) : type === "error" ? "Unknown error" : undefined;
         }
         if (event.stack)
@@ -290,9 +344,10 @@ class MidlineAgent {
             apiKey: config.apiKey,
             eventType: type,
             route,
-            method: (event.method || "GET").toUpperCase().slice(0, 16),
+            // A printed line has no method or timing; defaults here would make it look like a request.
+            method: isConsole ? undefined : (event.method || "GET").toUpperCase().slice(0, 16),
             statusCode,
-            responseTime: Math.min(Math.max(0, Math.round(Number(event.duration) || 0)), 86400000),
+            responseTime: isConsole ? undefined : Math.min(Math.max(0, Math.round(Number(event.duration) || 0)), 86400000),
             timestamp: validTimestamp(event.timestamp),
             // Clamped to the server's validation limits: one over-long field would
             // otherwise get every event rejected.
@@ -405,6 +460,8 @@ class MidlineAgent {
             }
             if (outcome.kind === "rejected") {
                 if (batch.length === 1) {
+                    if (this.refusedConsole(batch[0], outcome.detail))
+                        continue;
                     this.dropped += 1;
                     this.report(`rejected:${outcome.detail}`, `midline: the Midline server rejected an event (${outcome.detail}); dropped it.`);
                     continue;
@@ -424,13 +481,17 @@ class MidlineAgent {
     /** Returns false if delivery should stop for this drain. */
     async sendIndividually(batch, keepProcessAlive) {
         for (let index = 0; index < batch.length; index++) {
+            if (this.consoleUnsupported && batch[index].wire.eventType === "console")
+                continue;
             const outcome = await this.send([batch[index]], keepProcessAlive);
             if (outcome.kind === "ok") {
                 this.onSuccess();
             }
             else if (outcome.kind === "rejected" || outcome.kind === "tooLarge") {
-                this.dropped += 1;
                 const detail = outcome.kind === "rejected" ? outcome.detail : "HTTP 413";
+                if (outcome.kind === "rejected" && this.refusedConsole(batch[index], detail))
+                    continue;
+                this.dropped += 1;
                 this.report(`rejected:${detail}`, `midline: the Midline server rejected an event (${detail}); dropped it.`);
             }
             else if (outcome.kind === "stop") {
@@ -493,6 +554,24 @@ class MidlineAgent {
         }
         return { kind: "rejected", detail: `HTTP ${status}${serverMessage(result.body)}` };
     }
+    /**
+     * True when a rejected event is a console line the server has no event type for,
+     * i.e. the server predates console capture. Capture switches itself off rather
+     * than paying a refused request for every line printed from then on.
+     */
+    refusedConsole(item, detail) {
+        if (item.wire.eventType !== "console" || !detail.includes("eventType"))
+            return false;
+        if (!this.consoleUnsupported) {
+            this.consoleUnsupported = true;
+            this.consoleCapture?.uninstall();
+            this.consoleCapture = null;
+            this.queue = this.queue.filter((queued) => queued.wire.eventType !== "console");
+            this.queueBytes = this.queue.reduce((sum, queued) => sum + queued.bytes, 0);
+            this.log("warn", "midline: this Midline server does not accept console events yet (it needs updating), so console capture is off. Request and error monitoring carry on.");
+        }
+        return true;
+    }
     /** One line per distinct fault, not one per failed event. */
     report(signature, message) {
         if (this.lastNotice === signature && !this.debugLogs) {
@@ -558,22 +637,26 @@ class MidlineAgent {
         this.close();
     }
     log(level, message) {
-        if (this.onErrorHook) {
-            try {
-                this.onErrorHook(message);
-            }
-            catch {
-                // A broken logging hook must not become the host application's problem.
+        // The agent's own diagnostics are never the application's console output,
+        // including when a hook hands them to a logger that prints them.
+        (0, console_1.withoutConsoleCapture)(() => {
+            if (this.onErrorHook) {
+                try {
+                    this.onErrorHook(message);
+                }
+                catch {
+                    // A broken logging hook must not become the host application's problem.
+                }
+                if (!this.debugLogs)
+                    return;
             }
-            if (!this.debugLogs)
-                return;
-        }
-        if (level === "error")
-            console.error(message);
-        else if (level === "warn")
-            console.warn(message);
-        else
-            console.info(message);
+            if (level === "error")
+                console.error(message);
+            else if (level === "warn")
+                console.warn(message);
+            else
+                console.info(message);
+        });
     }
 }
 exports.MidlineAgent = MidlineAgent;
@@ -595,6 +678,9 @@ function classify(event, statusCode) {
     else if (event.type === "custom") {
         category = "business";
     }
+    else if (event.type === "console") {
+        category = "application";
+    }
     else if (statusCode && statusCode >= 500) {
         severity = "high";
         category = "application";

package/dist/browser/client.d.ts

@@ -0,0 +1,59 @@
+import type { EventSeverity } from "../types.js";
+import type { CaptureContext, MidlineBrowserConfig, MidlineUser } from "./types.js";
+export declare class BrowserConfigError extends Error {
+}
+/** Accepts a base URL or the ingest URL, and returns the batch URL. */
+export declare function resolveBatchUrl(endpoint?: string): string;
+/**
+ * One installed SDK instance. Everything it does to the page — wrapped fetch,
+ * XHR, console and history, window listeners — is undone by `close()`.
+ */
+export declare class BrowserClient {
+    private readonly win;
+    private readonly config;
+    private readonly redactor;
+    private readonly transport;
+    private readonly teardowns;
+    private readonly breadcrumbs;
+    private readonly recentErrors;
+    private readonly sessionId;
+    private readonly originalConsole;
+    private traceId;
+    private user;
+    private tags;
+    private tokens;
+    private lastRefill;
+    private active;
+    private inConsoleHook;
+    private rateLimitNoted;
+    private constructor();
+    /** Returns undefined, after one console warning, when the SDK can't run. It never throws into the app. */
+    static create(config: MidlineBrowserConfig): BrowserClient | undefined;
+    private install;
+    captureException(error: unknown, context?: CaptureContext): void;
+    captureMessage(message: string, severity?: EventSeverity, context?: CaptureContext): void;
+    setUser(user: MidlineUser | null): void;
+    setTag(key: string, value: string): void;
+    addBreadcrumb(message: string, type?: string): void;
+    flush(): Promise<void>;
+    close(): Promise<void>;
+    private handleError;
+    private handleRequest;
+    private handleConsole;
+    private handleNavigation;
+    private handleVitals;
+    private propagation;
+    private base;
+    private emit;
+    private takeToken;
+    private breadcrumb;
+    private contextPayload;
+    private applyTags;
+    /** The page path, plus a `#/…` hash route when the app routes by hash. Never the query string. */
+    private currentRoute;
+    private pageUrl;
+    private guard;
+    private debug;
+}
+/** W3C ids: lowercase hex, never all zeros. */
+export declare function randomHex(bytes: number): string;