@forge-ops/tracker 0.11.0 → 0.13.0

package/README.md

@@ -375,6 +375,56 @@ flush and would otherwise grow it for as long as the process lives. A NaN or inf
 dropped at capture: `JSON.stringify` turns it into `null` and the server would reject the whole
 batch behind it. Requires a ForgeOps plan that includes custom metrics / infrastructure monitoring.
 
+## What changed
+
+ForgeOps can show what changed in your system next to the errors and slowdowns that followed it.
+Two ways in:
+
+**Record a change yourself** when something changes that no deploy captures, like a feature flag
+flipped, a config value edited, or a migration run by hand:
+
+```js
+import * as forgeOpsTracker from "@forge-ops/tracker";
+
+forgeOpsTracker.recordChange({
+  kind: "feature_flag", // feature_flag, config, migration, dependency, infrastructure, or other
+  title: "Enabled new_checkout for 10% of users",
+  details: { flag: "new_checkout", rolloutPercent: 10 },
+  actor: "ops@example.com",
+  url: "https://flags.example.com/new_checkout",
+});
+```
+
+`kind` and `title` are required; `details`, `environment` (defaults to the configured one),
+`service`, `actor`, `url`, `id` (an idempotency key, so sending the same change twice records it
+once), and `occurredAt` (a `Date` or ISO 8601 string, defaulting to now) are optional. An unknown
+`kind` is sent as `other`. It's queued and delivered on the same async loop as error events, so it
+never slows down the caller, never throws, and is a no-op when the client isn't enabled for the
+environment.
+
+**Changes between deploys are detected for you.** Once per process, `init()` schedules a snapshot
+of what the process is running: the Node version, and each of the `dependencies` in your app's
+`package.json` (read from `appRoot`, the working directory by default) resolved to the version
+actually installed in `node_modules`. ForgeOps compares it with the previous boot's and records
+whatever changed, such as a package upgrade. It's sent on a later event-loop turn, so startup never
+waits on it.
+
+```js
+forgeOpsTracker.init({
+  dsn: "https://<api_key>@getforgeops.net/api/v1/events",
+  detectChanges: true, // default; false sends no startup snapshot
+  trackEnvVarNames: false, // default; true also sends environment variable names
+});
+```
+
+With `trackEnvVarNames` on, the snapshot lists the names of your environment variables (never their
+values), so an added or removed variable shows up as a change. Names that differ from host to host,
+like `HOSTNAME`, `PATH`, `PORT`, `LC_*`, and Kubernetes service variables, are left out, as are the
+SDK's own `FORGE_OPS_*` settings.
+
+Requires a ForgeOps plan that includes change tracking; on a plan that doesn't, both are rejected
+server-side and dropped, exactly like any other delivery failure.
+
 ## Distributed tracing
 
 For one slow or errored request, `forgeOpsTrackerTracingExpressMiddleware` (Express) and
@@ -489,10 +539,39 @@ app.use(forgeOpsTrackerExpressMiddleware);
 app.listen(3000);
 ```
 
-**Known gaps:** no database span capture (this SDK has no existing query/ORM instrumentation hook
-of any kind yet to extend) and no Redis span capture (no existing Redis hook or dependency exists
-here either); a database call or Redis call inside a traced request just won't show up as its own
-span for now.
+### Database spans with their SQL
+
+Database queries aren't captured automatically (this SDK has no query or ORM instrumentation hook
+to extend), so wrap a query by hand with `kind: "database"` to have it show up as its own span. Pass
+the SQL as `statement` and ForgeOps shows which query a slow request spent its time in:
+
+```js
+import pg from "pg";
+import * as forgeOpsTracker from "@forge-ops/tracker";
+
+const pool = new pg.Pool();
+
+app.get("/orders", async (req, res) => {
+  const sql = "SELECT id, total FROM orders WHERE customer_id = $1 AND status = 'open'";
+  const { rows } = await forgeOpsTracker.span("Load open orders", () => pool.query(sql, [req.user.id]), {
+    kind: "database",
+    statement: sql,
+    dbSystem: "postgresql",
+  });
+  res.json(rows);
+});
+```
+
+The statement is masked before it's stored on the span: every string and number becomes `?`, so the
+span above carries `SELECT id, total FROM orders WHERE customer_id = $1 AND status = ?`, and
+ForgeOps masks it again on arrival. It's cut to 4000 characters. Bind values (the `[req.user.id]`
+array) are never read or sent. `dbSystem` is optional (`"postgresql"`, `"mysql"`, `"sqlite"`,
+`"mssql"`, `"oracle"`, or any other lowercase name) and both options are ignored on spans of any
+other kind. They go out in the span's data as `db.statement` and `db.system`.
+
+**Known gaps:** no automatic database span capture (see above) and no Redis span capture (no
+existing Redis hook or dependency exists here either); a Redis call inside a traced request just
+won't show up as its own span for now.
 
 ## Database errors
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@forge-ops/tracker",
-  "version": "0.11.0",
+  "version": "0.13.0",
   "description": "ForgeOps error tracking client: captures unhandled exceptions (Express/Fastify integration, plus explicit capture anywhere else) and delivers them to ForgeOps over HTTP.",
   "type": "module",
   "main": "src/index.js",

package/src/changes.js

@@ -0,0 +1,147 @@
+import { readFile } from "node:fs/promises";
+import path from "node:path";
+
+/**
+ * Payloads for "what changed": one explicit recordChange() call (POST /api/v1/changes), and the
+ * startup snapshot (POST /api/v1/change_snapshots) that ForgeOps diffs against the previous boot's
+ * to record what changed between deploys. Mirrors gems/forge_ops_tracker's Change and
+ * ChangeSnapshot.
+ *
+ * The snapshot leaves out any key it can't determine reliably rather than guessing, since the
+ * server reads a missing key as "unknown", never as "everything was removed". Environment variable
+ * names are only ever names, never values, and only when trackEnvVarNames is on.
+ */
+
+/** The server rejects any other kind outright (422), so an unknown one is sent as "other" rather
+ * than dropped: the change still gets recorded, just less specifically categorized. */
+export const CHANGE_KINDS = Object.freeze(["feature_flag", "config", "migration", "dependency", "infrastructure", "other"]);
+const MAX_TITLE_LENGTH = 200;
+
+const MAX_DEPENDENCIES = 3000;
+const MAX_NAME_LENGTH = 200;
+const MAX_VERSION_LENGTH = 100;
+const MAX_ENV_VAR_NAMES = 3000;
+
+/** Host-specific variables that differ from one box to the next (or one boot to the next) without
+ * anything about the deploy having changed, so a fleet doesn't look like it's changing on every
+ * restart. The SDK's own FORGE_OPS_* settings are dropped too. */
+const ENV_VAR_DENYLIST = new Set([
+  "HOSTNAME", "HOST", "HOME", "PATH", "PWD", "OLDPWD", "SHLVL", "_", "TERM", "USER", "LOGNAME", "SHELL",
+  "LANG", "TMPDIR", "TZ", "PORT", "DYNO", "INVOCATION_ID", "JOURNAL_STREAM",
+]);
+const ENV_VAR_DENYLIST_PATTERNS = [
+  /^LC_/, /^SYSTEMD_/, /^MEMORY_PRESSURE_/, /^KUBERNETES_/, /^FORGE_OPS_/,
+  /_SERVICE_HOST$/, /_SERVICE_PORT/, /_PORT_.*_TCP/,
+];
+
+/** @param {unknown} kind */
+export function normalizeKind(kind) {
+  return CHANGE_KINDS.includes(kind) ? kind : "other";
+}
+
+/**
+ * The body for one recordChange() call, or null when there's nothing sendable (a blank title: the
+ * server requires one, so posting it anyway would only ever come back 422).
+ * @param {import("./configuration.js").Configuration} configuration
+ * @param {{ kind: string, title: string, details?: Record<string, unknown>, environment?: string,
+ *   service?: string, actor?: string, url?: string, id?: string, occurredAt?: Date | string }} change
+ */
+export function buildChange(configuration, { kind, title, details, environment, service, actor, url, id, occurredAt } = {}) {
+  const trimmed = title == null ? "" : String(title).trim();
+  if (trimmed === "") {
+    return null;
+  }
+
+  const payload = {
+    kind: normalizeKind(kind),
+    title: trimmed.slice(0, MAX_TITLE_LENGTH),
+    details: details !== null && typeof details === "object" && !Array.isArray(details) ? details : {},
+    environment: String(environment ?? configuration.environment),
+  };
+  for (const [key, value] of Object.entries({ service, actor, url, id })) {
+    if (value != null) {
+      payload[key] = String(value);
+    }
+  }
+  payload.occurred_at = iso8601(occurredAt);
+  return payload;
+}
+
+/** @param {Date | string | undefined} value */
+function iso8601(value) {
+  if (typeof value === "string") {
+    return value;
+  }
+  if (value instanceof Date && !Number.isNaN(value.getTime())) {
+    return value.toISOString();
+  }
+  return new Date().toISOString();
+}
+
+/** @param {string} name */
+export function isDeniedEnvVar(name) {
+  return ENV_VAR_DENYLIST.has(name) || ENV_VAR_DENYLIST_PATTERNS.some((pattern) => pattern.test(name));
+}
+
+/**
+ * Names only, never values, sorted so the same set always serializes the same way.
+ * @param {Record<string, string | undefined>} [env]
+ */
+export function envVarNames(env = process.env) {
+  return Object.keys(env)
+    .filter((name) => !isDeniedEnvVar(name))
+    .sort()
+    .slice(0, MAX_ENV_VAR_NAMES);
+}
+
+export function runtime() {
+  return `node ${process.versions.node}`;
+}
+
+/**
+ * The app's own direct dependencies (the "dependencies" in `<appRoot>/package.json`), each
+ * resolved to the version actually installed under `<appRoot>/node_modules/<name>/package.json`,
+ * never the semver range the manifest asks for. A dependency that isn't installed there is left
+ * out rather than guessed at, and so is the whole key (null) when there's no package.json to read.
+ * @param {string} appRoot
+ */
+export async function dependencies(appRoot) {
+  let manifest;
+  try {
+    manifest = JSON.parse(await readFile(path.join(appRoot, "package.json"), "utf8"));
+  } catch {
+    return null;
+  }
+
+  const names = Object.keys(manifest?.dependencies ?? {}).sort().slice(0, MAX_DEPENDENCIES);
+  const resolved = await Promise.all(names.map((name) => installedVersion(appRoot, name)));
+  const found = {};
+  names.forEach((name, index) => {
+    if (resolved[index]) {
+      found[name.slice(0, MAX_NAME_LENGTH)] = resolved[index].slice(0, MAX_VERSION_LENGTH);
+    }
+  });
+  return Object.keys(found).length > 0 ? found : null;
+}
+
+async function installedVersion(appRoot, name) {
+  try {
+    const installed = JSON.parse(await readFile(path.join(appRoot, "node_modules", name, "package.json"), "utf8"));
+    return typeof installed?.version === "string" ? installed.version : null;
+  } catch {
+    return null;
+  }
+}
+
+/** @param {import("./configuration.js").Configuration} configuration */
+export async function buildSnapshot(configuration) {
+  const state = { runtime: runtime() };
+  const deps = await dependencies(configuration.appRoot);
+  if (deps) {
+    state.dependencies = deps;
+  }
+  if (configuration.trackEnvVarNames) {
+    state.env_var_names = envVarNames();
+  }
+  return { environment: configuration.environment, state };
+}

package/src/client.js

@@ -68,6 +68,23 @@ export class Client {
     return this.#post(this.#configuration.infrastructureMetricsUri(), { metrics });
   }
 
+  /**
+   * One recordChange() call; see Api::V1::ChangesController. A plan without change tracking
+   * answers 403, which is just a false here like any other non-2xx.
+   * @param {Record<string, unknown>} payload
+   */
+  async deliverChange(payload) {
+    return this.#post(this.#configuration.changesUri(), payload);
+  }
+
+  /**
+   * The startup snapshot; see Api::V1::ChangeSnapshotsController.
+   * @param {Record<string, unknown>} payload
+   */
+  async deliverChangeSnapshot(payload) {
+    return this.#post(this.#configuration.changeSnapshotsUri(), payload);
+  }
+
   /**
    * @param {string | null} uri
    * @param {Record<string, unknown>} payload

package/src/configuration.js

@@ -136,6 +136,21 @@ export class Configuration {
    */
   tracePropagationTargets = null;
 
+  /**
+   * Sends one startup snapshot per process (the Node version and the installed versions of your
+   * package.json dependencies; see changes.js) which ForgeOps diffs against the previous boot's to
+   * record what changed between deploys. On by default, same posture as every other automatic
+   * behavior here.
+   */
+  detectChanges = true;
+  /**
+   * Whether that snapshot also lists the names of this process's environment variables, so an
+   * added or removed variable shows up as a change. Off by default: names only, never values, but
+   * even names can say more about an app than some teams want to share. Host-specific names
+   * (HOSTNAME, PATH, PORT, and so on; see changes.js) are always left out.
+   */
+  trackEnvVarNames = false;
+
   /** @returns {string | null} */
   apiKey() {
     const parsed = this.#parsedDsn();
@@ -223,6 +238,31 @@ export class Configuration {
     return uri.replace(/\/events$/, "/spans");
   }
 
+  /**
+   * Same derivation again, swapping the trailing /events for /changes (recordChange()).
+   * @returns {string | null}
+   */
+  changesUri() {
+    const uri = this.ingestionUri();
+    if (!uri) {
+      return null;
+    }
+    return uri.replace(/\/events$/, "/changes");
+  }
+
+  /**
+   * Same derivation again, swapping the trailing /events for /change_snapshots (the startup
+   * snapshot).
+   * @returns {string | null}
+   */
+  changeSnapshotsUri() {
+    const uri = this.ingestionUri();
+    if (!uri) {
+      return null;
+    }
+    return uri.replace(/\/events$/, "/change_snapshots");
+  }
+
   /**
    * Whether an outbound request to `host` should carry a traceparent header; see
    * propagateTraces/tracePropagationTargets above. Case-insensitive, since hostnames are.

package/src/index.js

@@ -1,5 +1,6 @@
 import { AsyncLocalStorage } from "node:async_hooks";
 import { BreadcrumbBuffer } from "./breadcrumbBuffer.js";
+import { buildChange, buildSnapshot } from "./changes.js";
 import { Client } from "./client.js";
 import { Configuration } from "./configuration.js";
 import { DeliveryQueue } from "./deliveryQueue.js";
@@ -11,6 +12,7 @@ import { Reporter } from "./reporter.js";
 import { requestStateFor } from "./requestState.js";
 import { SessionFlusher } from "./sessionFlusher.js";
 import { randomSpanId, SpanBuffer } from "./spanBuffer.js";
+import { mask as maskSql } from "./sqlStatement.js";
 import { buildTraceparent } from "./traceParent.js";
 
 export { Configuration };
@@ -21,6 +23,8 @@ let reporter = null;
 let sessionFlusher = null;
 let performanceFlusher = null;
 let spanQueue = null;
+let changeQueue = null;
+let changeSnapshotStarted = false;
 let metricBuffer = null;
 let infrastructureMetricBuffer = null;
 let processHandlersInstalled = false;
@@ -183,6 +187,70 @@ export async function flushMetrics() {
   await Promise.all([metricBuffer.flush(), infrastructureMetricBuffer.flush()]);
 }
 
+function getChangeQueue() {
+  if (changeQueue === null) {
+    const config = getConfiguration();
+    changeQueue = new DeliveryQueue(config, new Client(config), { deliverMethod: "deliverChange", label: "change" });
+  }
+  return changeQueue;
+}
+
+/**
+ * Records one thing that changed in a running system, so ForgeOps can show it next to the errors
+ * and slowdowns that followed:
+ *
+ *     recordChange({ kind: "feature_flag", title: "Enabled new_checkout for 10%", details: { rollout: 10 } });
+ *
+ * `kind` is one of feature_flag, config, migration, dependency, infrastructure, or other (anything
+ * else is sent as "other"). `environment` defaults to the configured one; `occurredAt` (a Date or an
+ * ISO 8601 string) defaults to now; `id` is an optional idempotency key. Queued and delivered on the
+ * same async loop as error events, so it never blocks the caller. A no-op when the client isn't
+ * enabled. Never throws: returns false when nothing was queued.
+ *
+ * @param {{ kind: string, title: string, details?: Record<string, unknown>, environment?: string,
+ *   service?: string, actor?: string, url?: string, id?: string, occurredAt?: Date | string }} change
+ * @returns {boolean}
+ */
+export function recordChange(change) {
+  try {
+    const config = getConfiguration();
+    if (!config.isEnabled()) {
+      return false;
+    }
+    const payload = buildChange(config, change ?? {});
+    if (payload === null) {
+      return false;
+    }
+    return getChangeQueue().push(payload);
+  } catch (e) {
+    getConfiguration().log(`[forge-ops-tracker] recordChange failed: ${e.name}: ${e.message}`);
+    return false;
+  }
+}
+
+/**
+ * Sends the one startup snapshot this process sends (see changes.js), on a later event-loop turn so
+ * init() returns immediately and startup never waits on it. A second call is a no-op. Does nothing,
+ * and doesn't use up the once, when the client isn't enabled or detectChanges is off. The timer is
+ * unref'd, so a short script that's otherwise done doesn't stay alive just to send it. Never throws.
+ */
+function startChangeSnapshot(config) {
+  if (changeSnapshotStarted || !config.isEnabled() || !config.detectChanges) {
+    return false;
+  }
+  changeSnapshotStarted = true;
+
+  const timer = setTimeout(async () => {
+    try {
+      await new Client(config).deliverChangeSnapshot(await buildSnapshot(config));
+    } catch (e) {
+      config.log(`[forge-ops-tracker] change snapshot failed: ${e.name}: ${e.message}`);
+    }
+  }, 0);
+  timer.unref?.();
+  return true;
+}
+
 /**
  * A DeliveryQueue reused for spans (see that class's own comment for why it takes a
  * deliverMethod/label rather than needing a whole second, near-identical class the way
@@ -442,6 +510,8 @@ export function init(options = {}) {
   // _recordSpan above), never at install time.
   installHttpTracing(_recordSpan, _traceparentFor);
 
+  startChangeSnapshot(config);
+
   return config;
 }
 
@@ -541,17 +611,30 @@ export function runWithUser(user, callback) {
  * request-level trace around it, and no middleware left running to ever flush it, would just
  * accumulate in memory forever with nothing to ever send it, worse than not recording it at all.
  *
+ * A "database" span can also carry the SQL it ran, so ForgeOps can show which query was slow:
+ *
+ *     await forgeOpsTracker.span("Load orders", () => pool.query(sql, [userId]), {
+ *       kind: "database",
+ *       statement: sql,
+ *       dbSystem: "postgresql",
+ *     });
+ *
+ * The statement is masked (every string and number replaced by "?") before it's ever stored on
+ * the span, and sent as data["db.statement"], with dbSystem lowercased as data["db.system"]. Bind
+ * values are never taken, and both options are ignored on any other kind of span.
+ *
  * @template T
  * @param {string} name
  * @param {() => T} callback
- * @param {{ kind?: string, data?: Record<string, unknown> }} [options]
+ * @param {{ kind?: string, data?: Record<string, unknown>, statement?: string, dbSystem?: string }} [options]
  * @returns {T}
  */
-export function span(name, callback, { kind = "service", data = {} } = {}) {
+export function span(name, callback, { kind = "service", data = {}, statement, dbSystem } = {}) {
   const buffer = spanStorage.getStore();
   if (!buffer) {
     return callback();
   }
+  data = databaseSpanData(kind, data, statement, dbSystem);
 
   const spanId = randomSpanId();
   const parentSpanId = spanParentStorage.getStore() ?? buffer.rootSpanId;
@@ -583,6 +666,33 @@ export function span(name, callback, { kind = "service", data = {} } = {}) {
   }
 }
 
+/**
+ * The data a span is recorded with: unchanged unless it's a database span, which gets the masked
+ * statement as "db.statement" and the lowercased database name as "db.system". A "db.statement"
+ * passed in `data` directly is masked as well, so the raw SQL can't reach the payload either way.
+ * @param {string} kind
+ * @param {Record<string, unknown>} data
+ * @param {string | undefined} statement
+ * @param {string | undefined} dbSystem
+ * @returns {Record<string, unknown>}
+ */
+function databaseSpanData(kind, data, statement, dbSystem) {
+  if (kind !== "database") {
+    return data;
+  }
+  const result = { ...data };
+  const masked = maskSql(typeof statement === "string" ? statement : result["db.statement"]);
+  if (masked === null) {
+    delete result["db.statement"];
+  } else {
+    result["db.statement"] = masked;
+  }
+  if (typeof dbSystem === "string" && dbSystem.trim() !== "") {
+    result["db.system"] = dbSystem.trim().toLowerCase();
+  }
+  return result;
+}
+
 /**
  * Reports anything that would otherwise crash the process outright (a
  * plain script, an unhandled promise rejection) with no further wiring:
@@ -621,8 +731,13 @@ function installProcessLevelHandlers() {
   process.on("unhandledRejection", unhandledRejectionListener);
 }
 
-/** @internal not part of the public API: resets module state between test cases */
-export function _resetForTesting() {
+/**
+ * @internal not part of the public API: resets module state between test cases. The startup change
+ * snapshot is left marked as already sent, so the many tests that init() an enabled client and
+ * count fetch calls don't see an extra one; pass { changeSnapshot: true } to let the next init()
+ * send it (see test/changes.test.js).
+ */
+export function _resetForTesting({ changeSnapshot = false } = {}) {
   if (uncaughtExceptionListener) {
     process.off("uncaughtExceptionMonitor", uncaughtExceptionListener);
   }
@@ -634,6 +749,8 @@ export function _resetForTesting() {
   sessionFlusher = null;
   performanceFlusher = null;
   spanQueue = null;
+  changeQueue = null;
+  changeSnapshotStarted = !changeSnapshot;
   metricBuffer?.discard();
   infrastructureMetricBuffer?.discard();
   metricBuffer = null;