@forge-ops/tracker 0.14.0 → 0.15.0

package/README.md

@@ -174,8 +174,9 @@ to override what was auto-detected.
 A trail of what happened right before an error. With `forgeOpsTrackerBreadcrumbContextExpressMiddleware`/
 `registerForgeOpsTrackerBreadcrumbContext` installed (see the Express/Fastify snippets above), every
 request gets its own trail, and the Express performance middleware records a `"controller"`
-breadcrumb into it automatically, no further setup needed. Shows up alongside the error on an
-issue's own detail page.
+breadcrumb into it automatically, no further setup needed. Every `pg` query adds a `"query"`
+breadcrumb and every Redis command a `"redis"` one too (see "Database spans" and "Redis spans"
+below), inside a request or not. Shows up alongside the error on an issue's own detail page.
 
 ```js
 forgeOpsTracker.init({
@@ -212,9 +213,9 @@ the end of that same request. Unlike the affected user above, a breadcrumb's `me
 free text (a bind parameter showing up in a message, a URL with a token in it) the scrubber exists
 to catch, not a deliberately-structured field the way `user` is.
 
-There's currently no query-level automatic breadcrumb source: this client has no ORM/DB driver
-integration to hang one off of yet, the same reason there's no query-level performance
-instrumentation either. Fastify also gets no automatic `"controller"` breadcrumb today, since
+The `"query"` and `"redis"` breadcrumbs need no middleware: a query or command run outside any
+request (a script, a worker) lands in the same standalone trail `addBreadcrumb()` would start
+there. Fastify gets no automatic `"controller"` breadcrumb today, since
 there's no Fastify performance/timing integration in this client for one to ride alongside (see
 "Performance monitoring" below); `registerForgeOpsTrackerBreadcrumbContext` still gives Fastify
 requests their own trail, so `addBreadcrumb()` calls from inside a Fastify route handler work
@@ -570,6 +571,11 @@ on arrival. It's cut to 4000 characters. Bind values (the `[req.user.id]` array)
 sent. `db.system` is `"postgresql"`. A failed query is recorded too, and its error reaches your
 code exactly as before.
 
+Each query also adds a `"query"` breadcrumb (see "Breadcrumbs" above), named the same way as the
+span (`"SELECT orders"`), with only `{ duration_ms }` as its data: never the statement, masked or
+not. It's recorded whether or not the request is being traced, and outside a request too, unless
+`trackBreadcrumbs` is off.
+
 For any other database library, wrap a query by hand with `kind: "database"` and pass the SQL as
 `statement`; it's masked the same way:
 
@@ -592,8 +598,14 @@ app.get("/orders", async (req, res) => {
 
 `dbSystem` is optional (`"postgresql"`, `"mysql"`, `"sqlite"`, `"mssql"`, `"oracle"`, or any other
 lowercase name) and both options are ignored on spans of any other kind. They go out in the span's
-data as `db.statement` and `db.system`. A hand-made span around a `pg` query still works: the
-automatic query span just nests under it.
+data as `db.statement` and `db.system`. With `dbSystem` set to `"mysql"` or `"mariadb"`, text in
+"double quotes" is masked too, since it's a string there rather than a name. A hand-made span
+around a `pg` query still works: the automatic query span just nests under it.
+
+Masking follows the same rules ForgeOps applies on arrival: `''` and backslash escapes inside a
+string, a string's type prefix (`E'...'`, `X'DEADBEEF'`, `N'...'`, `B'...'`, `U&'...'`), `$tag$`
+dollar quotes, and numbers of every shape (`42`, `1.5`, `.5`, `3e10`, `1.5E-3`, `0x1F`, `0b101`),
+never the digits inside a name like `orders2` or a `$1` placeholder.
 
 ### Redis spans
 
@@ -615,14 +627,24 @@ app.get("/cart", async (req, res) => {
 
 A span is named after the command alone (`"Redis GET"`, `"Redis SET"`, `"Redis LPUSH"`) and
 carries no data: never the key or any other argument, since a key can easily hold a user's id or
-another sensitive value. A failed command is recorded too. `ioredis` is covered from version 4 on;
-`redis` from 5.12 on (with Node 18.19 or later), through the diagnostics channel it publishes every
-command on, so nothing in it is patched.
+another sensitive value. A failed command is recorded too.
+
+Each command also adds a `"redis"` breadcrumb (see "Breadcrumbs" above) with the same name as its
+message, level `"error"` when the command failed (`"info"` otherwise), and only `{ duration_ms }`
+as its data. It's recorded whether or not the request is being traced, and outside a request too,
+unless `trackBreadcrumbs` is off.
+
+`ioredis` is covered from version 4 on. `redis` is covered from 4.0 on: from 5.12 (with Node 18.19
+or later) through the diagnostics channel it publishes every command on, so nothing in it is
+patched; before that (all of 4.x, and 5.0 to 5.11) by wrapping its client's command queue, which
+every command goes through. A version that publishes on the channel is never wrapped, so no
+command is counted twice. On those older versions every command in a pipeline or a `MULTI` gets
+its own span (`MULTI` and `EXEC` included), like `ioredis` pipelines.
 
 ### When `init()` can't find `pg` or `ioredis`
 
-`init()` looks for `pg` and `ioredis` from the directory your app was started in, and from where
-this package is installed. Neither is a dependency of this package: one your app doesn't have is
+`init()` looks for `pg`, `ioredis` and `redis` from the directory your app was started in, and
+from where this package is installed. Neither is a dependency of this package: one your app doesn't have is
 simply skipped. If your app is bundled, or started from a directory its `node_modules` isn't
 under, hand the module over yourself, once, at startup:
 
@@ -637,13 +659,19 @@ forgeOpsTracker.instrumentIoredis(Redis);
 ```
 
 Both are safe to call more than once, and return `false` when given something that isn't `pg` or
-an `ioredis` class. `redis` never needs this.
+an `ioredis` class. `redis` from 5.12 on never needs this.
 
 **Known gaps:**
 
-- `redis` (node-redis) before 5.12 (all of 4.x, and 5.0 to 5.11) publishes nothing to listen to,
-  so its commands don't show up as spans. Upgrade, or wrap the calls you care about with
-  `span(name, fn, { kind: "redis" })`.
+- `redis` (node-redis) before 5.12 can't be handed over by hand: in a bundled app, or one whose
+  `node_modules` `init()` can't find, its commands aren't captured. Upgrade to 5.12 or later, or
+  wrap the calls you care about with `span(name, fn, { kind: "redis" })`.
+- `redis` from 5.12 on doesn't publish the commands inside a `MULTI` transaction on its
+  diagnostics channel, so they don't show up (a pipeline's commands do).
+- Pub/sub commands sent by `redis` before 5.12 (`SUBSCRIBE`, `UNSUBSCRIBE` and the like) aren't
+  captured. The commands it sends on its own when it connects (`SELECT`, `CLIENT SETINFO`, `AUTH`,
+  by name only) are, like any other command, and so are the `PING`s its `pingInterval` option
+  sends.
 - `pg-native` (`pg.native`), and cursors and streams passed to `query()` (`pg-cursor`,
   `pg-query-stream`), aren't captured.
 - Other database libraries (`mysql2`, `better-sqlite3`, Prisma, and so on) aren't captured

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@forge-ops/tracker",
-  "version": "0.14.0",
+  "version": "0.15.0",
   "description": "ForgeOps error tracking client: captures unhandled exceptions (Express/Fastify integration, plus explicit capture anywhere else) and delivers them to ForgeOps over HTTP.",
   "type": "module",
   "main": "src/index.js",
@@ -49,6 +49,10 @@
     "fastify": "^5.0.0",
     "ioredis": "^6.0.0",
     "pg": "^8.23.0",
-    "redis": "^6.2.1"
+    "redis": "^6.2.1",
+    "redis4": "npm:redis@^4.7.1",
+    "redis4-0": "npm:redis@4.0.6",
+    "redis5-0": "npm:redis@5.0.1",
+    "redis5-11": "npm:redis@5.11.0"
   }
 }

package/src/configuration.js

@@ -89,8 +89,8 @@ export class Configuration {
 
   /**
    * Whether the Express/Fastify integrations give every request its own breadcrumb trail, and
-   * whatever automatic sources this client already times (currently just the Express
-   * controller/route lifecycle; see integrations/performance.js) also record a breadcrumb
+   * whatever automatic sources this client already times (the Express controller/route lifecycle,
+   * see integrations/performance.js; every pg query; every Redis command) also record a breadcrumb
    * alongside the timing they already do. On by default, the same posture trackSessions/
    * trackPerformance above already have. addBreadcrumb() itself is never gated by this: only the
    * automatic sources are, matching gems/forge_ops_tracker's own track_breadcrumbs.

package/src/databaseTracing.js

@@ -9,11 +9,12 @@ let installed = false;
 const patches = [];
 
 /**
- * Records one "database" span per query run through node-postgres (pg), for any app that has it
- * installed; installed from index.js's init() the same way installHttpTracing is, exactly once
- * for the life of the process (see the installed guard), and checking whether a trace is being
- * captured fresh on every query rather than at install time. An app without pg gets nothing
- * patched and nothing loaded (see requireOptional).
+ * Records one "database" span and one "query" breadcrumb per query run through node-postgres (pg),
+ * for any app that has it installed; installed from index.js's init() the same way
+ * installHttpTracing is, exactly once for the life of the process (see the installed guard), and
+ * checking whether a trace is being captured, and whether breadcrumbs are on, fresh on every query
+ * rather than at install time. An app without pg gets nothing patched and nothing loaded (see
+ * requireOptional).
  *
  * Wraps pg.Client.prototype.query, which pg.Pool's own query() also goes through, so a pooled
  * query is covered with no separate hook. Every calling style is handled: a callback (as the last
@@ -28,16 +29,25 @@ const patches = [];
  * same masking span({ statement }) and gems/forge_ops_tracker's database spans apply) and
  * "postgresql" as data["db.system"]. Bind values are never read.
  *
+ * The breadcrumb is the one gems/forge_ops_tracker and the Python SDK's SQLAlchemy integration add
+ * per query: the span's own name as its message (never the statement), category "query", and only
+ * { duration_ms } as data. A span is only recorded inside a trace; a breadcrumb whenever automatic
+ * breadcrumbs are on, into whatever trail is current (a request's own, or the standalone one
+ * addBreadcrumb would start outside a request).
+ *
  * @param {(name: string, kind: string, startedAt: Date, durationMs: number, data?: Record<string, unknown>) => void} recordSpan
  * @param {() => boolean} isTracing whether the current async context has a trace to record into
+ * @param {(message: string, category: string, level?: string, data?: Record<string, unknown>) => void} [recordBreadcrumb]
+ * @param {() => boolean} [ensureBreadcrumbTrail] whether a breadcrumb recorded from the current
+ *   async context would be kept (see index.js's _ensureBreadcrumbTrail)
  */
-export function installDatabaseTracing(recordSpan, isTracing) {
+export function installDatabaseTracing(recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail) {
   if (installed) {
     return;
   }
   installed = true;
   for (const pg of requireOptional("pg")) {
-    instrumentPg(pg, recordSpan, isTracing);
+    instrumentPg(pg, recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail);
   }
 }
 
@@ -46,25 +56,28 @@ export function installDatabaseTracing(recordSpan, isTracing) {
  * same module: a method this already wrapped is left alone.
  *
  * Also binds a callback passed to pg.Pool#connect to the async context it was called from, while a
- * trace is being captured. When every pooled client is busy, pg-pool queues the request and later
- * hands it a client from inside whichever other request released one, so without this, a query
- * run on that client would be recorded into the wrong request's trace.
+ * trace is being captured or breadcrumbs are being kept. When every pooled client is busy, pg-pool
+ * queues the request and later hands it a client from inside whichever other request released
+ * one, so without this, a query run on that client would be recorded into the wrong request's
+ * trace and breadcrumb trail.
  *
  * @param {any} pg
  * @param {Parameters<typeof installDatabaseTracing>[0]} recordSpan
  * @param {() => boolean} isTracing
+ * @param {Parameters<typeof installDatabaseTracing>[2]} [recordBreadcrumb]
+ * @param {() => boolean} [ensureBreadcrumbTrail]
  * @returns {boolean} whether there was a pg Client to patch
  */
-export function instrumentPg(pg, recordSpan, isTracing) {
+export function instrumentPg(pg, recordSpan, isTracing, recordBreadcrumb = () => {}, ensureBreadcrumbTrail = () => false) {
   const clientPrototype = pg?.Client?.prototype;
   if (typeof clientPrototype?.query !== "function") {
     return false;
   }
-  patch(clientPrototype, "query", (original) => wrapQuery(original, recordSpan, isTracing));
+  patch(clientPrototype, "query", (original) => wrapQuery(original, recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail));
 
   const poolPrototype = pg.Pool?.prototype;
   if (typeof poolPrototype?.connect === "function") {
-    patch(poolPrototype, "connect", (original) => wrapConnect(original, isTracing));
+    patch(poolPrototype, "connect", (original) => wrapConnect(original, isTracing, ensureBreadcrumbTrail));
   }
   return true;
 }
@@ -93,30 +106,43 @@ function patch(target, key, wrapper) {
   target[key] = wrapped;
 }
 
-function wrapQuery(originalQuery, recordSpan, isTracing) {
+function wrapQuery(originalQuery, recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail) {
   return function patchedQuery(config, values, callback) {
-    if (config == null || typeof config.submit === "function" || !isTracing()) {
+    if (config == null || typeof config.submit === "function") {
+      return originalQuery.apply(this, arguments);
+    }
+    const span = isTracing();
+    const breadcrumb = ensureBreadcrumbTrail();
+    if (!span && !breadcrumb) {
       return originalQuery.apply(this, arguments);
     }
 
     const statement = typeof config === "string" ? config : config.text;
-    const masked = typeof statement === "string" ? mask(statement) : null;
+    const masked = typeof statement === "string" ? mask(statement, "postgresql") : null;
     const startedAt = new Date();
     const start = performance.now();
 
     // Bound to the caller's async context now: pg calls back (and settles its promise) from its
-    // socket's own event handlers, where the trace this query belongs to isn't the current one.
+    // socket's own event handlers, where the trace and the breadcrumb trail this query belongs to
+    // aren't the current ones.
     let finished = false;
     const finish = AsyncResource.bind(() => {
       if (finished) {
         return;
       }
       finished = true;
-      const data = { "db.system": "postgresql" };
-      if (masked !== null) {
-        data["db.statement"] = masked;
+      const name = queryName(masked);
+      const durationMs = performance.now() - start;
+      if (breadcrumb) {
+        recordBreadcrumb(name, "query", "info", { duration_ms: Math.round(durationMs * 10) / 10 });
+      }
+      if (span) {
+        const data = { "db.system": "postgresql" };
+        if (masked !== null) {
+          data["db.statement"] = masked;
+        }
+        recordSpan(name, "database", startedAt, durationMs, data);
       }
-      recordSpan(queryName(masked), "database", startedAt, performance.now() - start, data);
     });
 
     const args = [...arguments];
@@ -160,9 +186,9 @@ function withFinish(callback, finish) {
   };
 }
 
-function wrapConnect(originalConnect, isTracing) {
+function wrapConnect(originalConnect, isTracing, ensureBreadcrumbTrail) {
   return function patchedConnect(callback, ...rest) {
-    if (typeof callback === "function" && isTracing()) {
+    if (typeof callback === "function" && (isTracing() || ensureBreadcrumbTrail())) {
       return originalConnect.call(this, AsyncResource.bind(callback), ...rest);
     }
     return originalConnect.apply(this, arguments);

package/src/index.js

@@ -474,9 +474,9 @@ export function _isTracing() {
 }
 
 /**
- * Records a "database" span for every query run through this pg module (`import pg from "pg"`),
- * for an app where init() couldn't find pg by itself: a bundled app, or one whose node_modules
- * isn't under the directory it was started from. init() already does this for the pg it can
+ * Records a "database" span and a "query" breadcrumb for every query run through this pg module
+ * (`import pg from "pg"`), for an app where init() couldn't find pg by itself: a bundled app, or
+ * one whose node_modules isn't under the directory it was started from. init() already does this for the pg it can
  * find, so most apps never need to call it. Safe to call more than once.
  *
  *     import pg from "pg";
@@ -486,13 +486,13 @@ export function _isTracing() {
  * @returns {boolean} whether it was a pg module this could patch
  */
 export function instrumentPg(pg) {
-  return instrumentPgModule(pg, _recordSpan, _isTracing);
+  return instrumentPgModule(pg, _recordSpan, _isTracing, _recordBreadcrumb, _ensureBreadcrumbTrail);
 }
 
 /**
- * Records a "redis" span for every command sent through this ioredis Redis class, for an app
- * where init() couldn't find ioredis by itself (see instrumentPg above). Safe to call more than
- * once.
+ * Records a "redis" span and a "redis" breadcrumb for every command sent through this ioredis
+ * Redis class, for an app where init() couldn't find ioredis by itself (see instrumentPg above).
+ * Safe to call more than once.
  *
  *     import Redis from "ioredis";
  *     forgeOpsTracker.instrumentIoredis(Redis);
@@ -501,13 +501,13 @@ export function instrumentPg(pg) {
  * @returns {boolean} whether it was an ioredis class this could patch
  */
 export function instrumentIoredis(Redis) {
-  return instrumentIoredisClass(Redis, _recordSpan, _isTracing);
+  return instrumentIoredisClass(Redis, _recordSpan, _isTracing, _recordBreadcrumb, _ensureBreadcrumbTrail);
 }
 
 /**
- * Internal; called by whatever this client already automatically times (currently just the
- * Express controller/route lifecycle; see integrations/performance.js), never by host app code
- * directly, same reasoning as _recordPerformance above. Independent of trackPerformance: an app
+ * Internal; called by whatever this client already automatically times (the Express
+ * controller/route lifecycle, see integrations/performance.js, and the pg and Redis wrappers in
+ * databaseTracing.js and redisTracing.js), never by host app code directly, same reasoning as _recordPerformance above. Independent of trackPerformance: an app
  * could want the trail without the timing data, or vice versa, and each call site already has
  * to check its own flag regardless, so there's no real cost to keeping the two independent
  * rather than tying breadcrumbs to whether performance monitoring happens to be on (mirrors
@@ -527,6 +527,28 @@ export function _recordBreadcrumb(message, category, level = "info", data = {})
   addBreadcrumb(message, { category, level, data });
 }
 
+/**
+ * Internal; whether an automatic breadcrumb recorded from the current async context would be kept
+ * (trackBreadcrumbs is on and the client is enabled), for the pg and Redis wrappers to check before
+ * timing a query or command, the same way they check _isTracing above. When it would be and there's
+ * no trail yet (a query or command outside any request), starts the same standalone trail
+ * addBreadcrumb() below would, right now, from the app's own call: the breadcrumb itself is only
+ * recorded once the reply arrives, from a callback bound to this context, and a trail started
+ * there would never be seen again.
+ *
+ * @returns {boolean}
+ */
+export function _ensureBreadcrumbTrail() {
+  const config = getConfiguration();
+  if (!config.trackBreadcrumbs || !config.isEnabled()) {
+    return false;
+  }
+  if (breadcrumbStorage.getStore() === undefined) {
+    breadcrumbStorage.enterWith(new BreadcrumbBuffer(config));
+  }
+  return true;
+}
+
 /**
  * Configure the client. Call once at startup.
  *
@@ -556,8 +578,8 @@ export function init(options = {}) {
   installHttpTracing(_recordSpan, _traceparentFor);
   // The same for database queries (pg) and Redis commands (ioredis, node-redis), for whichever of
   // those the app has installed; nothing is loaded for one it doesn't (see optionalModules.js).
-  installDatabaseTracing(_recordSpan, _isTracing);
-  installRedisTracing(_recordSpan, _isTracing);
+  installDatabaseTracing(_recordSpan, _isTracing, _recordBreadcrumb, _ensureBreadcrumbTrail);
+  installRedisTracing(_recordSpan, _isTracing, _recordBreadcrumb, _ensureBreadcrumbTrail);
 
   startChangeSnapshot(config);
 
@@ -720,7 +742,8 @@ export function span(name, callback, { kind = "service", data = {}, statement, d
 /**
  * The data a span is recorded with: unchanged unless it's a database span, which gets the masked
  * statement as "db.statement" and the lowercased database name as "db.system". A "db.statement"
- * passed in `data` directly is masked as well, so the raw SQL can't reach the payload either way.
+ * passed in `data` directly is masked as well, so the raw SQL can't reach the payload either way,
+ * and masked for whichever system the span names (dbSystem, or a "db.system" in `data`).
  * @param {string} kind
  * @param {Record<string, unknown>} data
  * @param {string | undefined} statement
@@ -732,15 +755,17 @@ function databaseSpanData(kind, data, statement, dbSystem) {
     return data;
   }
   const result = { ...data };
-  const masked = maskSql(typeof statement === "string" ? statement : result["db.statement"]);
+  if (typeof dbSystem === "string" && dbSystem.trim() !== "") {
+    result["db.system"] = dbSystem.trim().toLowerCase();
+  }
+  // The system matters to the masking: MySQL and MariaDB "double quoted" text is a value.
+  const system = typeof result["db.system"] === "string" ? result["db.system"].trim() : null;
+  const masked = maskSql(typeof statement === "string" ? statement : result["db.statement"], system);
   if (masked === null) {
     delete result["db.statement"];
   } else {
     result["db.statement"] = masked;
   }
-  if (typeof dbSystem === "string" && dbSystem.trim() !== "") {
-    result["db.system"] = dbSystem.trim().toLowerCase();
-  }
   return result;
 }
 

package/src/integrations/performance.js

@@ -27,14 +27,11 @@ import { _recordBreadcrumb, _recordPerformance } from "../index.js";
  * mirroring gems/forge_ops_tracker/lib/forge_ops_tracker/railtie.rb's own
  * process_action.action_controller subscription, which records both a performance sample and a
  * breadcrumb from the one event): an app could want the trail without the timing data, or vice
- * versa. This is currently the only place this client automatically records a breadcrumb at all:
- * unlike Rails' framework-wide ActiveSupport::Notifications, this client has no automatic
- * instrumentation source that isn't also this same opt-in middleware, so an app that wants
- * automatic breadcrumbs (not just manual ones from addBreadcrumb()) needs this middleware
- * installed regardless of whether it also wants the performance data. There's no query-level
- * automatic breadcrumb source for the same reason there's no query-level performance
- * instrumentation in this client at all today (no ORM/DB driver integration exists yet to hang
- * one off of); this doesn't invent one.
+ * versa. The pg query and Redis command wrappers (databaseTracing.js, redisTracing.js) are the
+ * only other automatic breadcrumb sources, and they need no middleware: a query or command outside
+ * any request lands in the standalone trail addBreadcrumb() would start. The controller breadcrumb
+ * itself still needs this middleware installed, whether or not the app also wants the performance
+ * data.
  *
  * Deliberately recorded with the request's final status already known (from the same
  * res.on("finish") firing this middleware already needed for timing), the same "recorded once

package/src/optionalModules.js

@@ -19,21 +19,47 @@ import path from "node:path";
  * one Client/Redis class to patch either way.
  *
  * @param {string} name
+ * @param {{ via?: string }} [options] see resolveOptional
  * @returns {unknown[]}
  */
-export function requireOptional(name) {
+export function requireOptional(name, options) {
+  const require = createRequire(import.meta.url);
+  return resolveOptional(name, options).map((resolved) => require(resolved));
+}
+
+/**
+ * Where requireOptional would load `name` from: the absolute path of every distinct copy found,
+ * without loading any of them.
+ *
+ * `via` names the package the app actually installs when `name` is one of its own dependencies
+ * (node-redis's `redis` package depends on `@redis/client`): `name` is then also resolved from
+ * wherever each copy of `via` is, since an isolating package manager only lets `via` itself see
+ * it.
+ *
+ * @param {string} name
+ * @param {{ via?: string }} [options]
+ * @returns {string[]}
+ */
+export function resolveOptional(name, { via } = {}) {
   const bases = [path.join(process.cwd(), "package.json"), import.meta.url];
-  const loaded = new Map();
+  if (via !== undefined) {
+    bases.push(...bases.flatMap((base) => resolveFrom(base, via) ?? []));
+  }
+  const resolved = new Set();
   for (const base of bases) {
-    try {
-      const require = createRequire(base);
-      const resolved = require.resolve(name);
-      if (!loaded.has(resolved)) {
-        loaded.set(resolved, require(name));
-      }
-    } catch {
-      // Not installed where this base looks: nothing to patch from here.
+    const found = resolveFrom(base, name);
+    if (found !== null) {
+      resolved.add(found);
     }
   }
-  return [...loaded.values()];
+  return [...resolved];
+}
+
+function resolveFrom(base, name) {
+  try {
+    return createRequire(base).resolve(name);
+  } catch {
+    // Not installed where this base looks: nothing to patch from here.
+    return null;
+  }
 }

package/src/redisTracing.js

@@ -1,11 +1,15 @@
 import { AsyncResource } from "node:async_hooks";
 import diagnosticsChannel from "node:diagnostics_channel";
-import { requireOptional } from "./optionalModules.js";
+import { createRequire } from "node:module";
+import { requireOptional, resolveOptional } from "./optionalModules.js";
 
 const WRAPPED = Symbol("forgeOpsTrackerWrapped");
 const SEEN = Symbol("forgeOpsTrackerSeen");
 const SPAN = Symbol("forgeOpsTrackerSpan");
 const NODE_REDIS_COMMAND_CHANNEL = "node-redis:command";
+// The package node-redis keeps its client in: @redis/client from node-redis 4.1 on, and
+// @node-redis/client for 4.0.x.
+const NODE_REDIS_CLIENT_PACKAGES = ["@redis/client", "@node-redis/client"];
 
 let installed = false;
 // Every patch made, so _uninstallRedisTracing can put each method back exactly as found.
@@ -13,36 +17,51 @@ const patches = [];
 let nodeRedisSubscription = null;
 
 /**
- * Records one "redis" span per Redis command, for an app using ioredis or node-redis (the `redis`
- * package); installed from index.js's init() the same way installHttpTracing is, exactly once for
- * the life of the process, and checking whether a trace is being captured fresh on every command.
- * Neither library is loaded or required unless the app has it installed.
+ * Records one "redis" span and one "redis" breadcrumb per Redis command, for an app using ioredis
+ * or node-redis (the `redis` package); installed from index.js's init() the same way
+ * installHttpTracing is, exactly once for the life of the process, and checking whether a trace is
+ * being captured, and whether breadcrumbs are on, fresh on every command. Neither library is
+ * loaded or required unless the app has it installed.
  *
  * Every span is named "Redis <COMMAND>" (the command alone, upcased: GET, SET, LPUSH...) with no
  * data at all: never a key or any other argument, since a key can easily carry a customer's own
  * id or other sensitive value. The same name and kind gems/forge_ops_tracker's
- * Integrations::RedisClientTiming records.
+ * Integrations::RedisClientTiming records. The breadcrumb is the one that gem and the Python SDK's
+ * integrations/redis.py add: that same name as its message, category "redis", level "error" when
+ * the command failed, and only { duration_ms } as data. A span is only recorded inside a trace; a
+ * breadcrumb whenever automatic breadcrumbs are on, into whatever trail is current (a request's
+ * own, or the standalone one addBreadcrumb would start outside a request).
  *
  * ioredis (every version from 4.x on) is covered by wrapping Redis.prototype.sendCommand, which
  * every command goes through, pipelined and MULTI ones included (see instrumentIoredis).
  *
- * node-redis is covered through the "node-redis:command" diagnostics channel it publishes every
- * command on from version 5.12 on (@redis/client), so nothing in it is patched and it's never
- * loaded from here: subscribing by name works whether or not the app ever loads it. Earlier
- * node-redis versions (4.x, and 5.x before 5.12) publish nothing there and aren't covered.
+ * node-redis from 5.12 on is covered through the "node-redis:command" diagnostics channel it
+ * publishes every command on (@redis/client), so nothing in it is patched and it's never loaded
+ * from here: subscribing by name works whether or not the app ever loads it. Earlier versions
+ * (4.x, and 5.x before 5.12) publish nothing there, so their command queue is patched instead
+ * (see instrumentNodeRedisClient); a copy new enough to publish is never patched, so no command
+ * is recorded twice.
  *
  * @param {(name: string, kind: string, startedAt: Date, durationMs: number, data?: Record<string, unknown>) => void} recordSpan
  * @param {() => boolean} isTracing whether the current async context has a trace to record into
+ * @param {(message: string, category: string, level?: string, data?: Record<string, unknown>) => void} [recordBreadcrumb]
+ * @param {() => boolean} [ensureBreadcrumbTrail] whether a breadcrumb recorded from the current
+ *   async context would be kept (see index.js's _ensureBreadcrumbTrail)
  */
-export function installRedisTracing(recordSpan, isTracing) {
+export function installRedisTracing(recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail) {
   if (installed) {
     return;
   }
   installed = true;
   for (const Redis of requireOptional("ioredis")) {
-    instrumentIoredis(Redis, recordSpan, isTracing);
+    instrumentIoredis(Redis, recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail);
   }
-  subscribeNodeRedis(recordSpan, isTracing);
+  for (const name of NODE_REDIS_CLIENT_PACKAGES) {
+    for (const packageJsonPath of resolveOptional(`${name}/package.json`, { via: "redis" })) {
+      instrumentNodeRedisClient(packageJsonPath, recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail);
+    }
+  }
+  subscribeNodeRedis(hooksFor(recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail));
 }
 
 /**
@@ -59,19 +78,25 @@ export function installRedisTracing(recordSpan, isTracing) {
  * @param {any} Redis
  * @param {Parameters<typeof installRedisTracing>[0]} recordSpan
  * @param {() => boolean} isTracing
+ * @param {Parameters<typeof installRedisTracing>[2]} [recordBreadcrumb]
+ * @param {() => boolean} [ensureBreadcrumbTrail]
  * @returns {boolean} whether there was a sendCommand to patch
  */
-export function instrumentIoredis(Redis, recordSpan, isTracing) {
+export function instrumentIoredis(Redis, recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail) {
   const prototype = Redis?.prototype;
   if (typeof prototype?.sendCommand !== "function" || prototype.sendCommand[WRAPPED]) {
     return typeof prototype?.sendCommand === "function";
   }
+  const hooks = hooksFor(recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail);
   const originalSendCommand = prototype.sendCommand;
 
   const patchedSendCommand = function patchedSendCommand(command, ...rest) {
-    if (command && typeof command === "object" && !command[SEEN] && isTracing()) {
+    if (command && typeof command === "object" && !command[SEEN]) {
       command[SEEN] = true;
-      observeCommand(command, recordSpan);
+      const finish = startCommand(command.name, hooks);
+      if (finish !== null) {
+        observeCommand(command, finish);
+      }
     }
     return originalSendCommand.call(this, command, ...rest);
   };
@@ -82,6 +107,86 @@ export function instrumentIoredis(Redis, recordSpan, isTracing) {
   return true;
 }
 
+/**
+ * Patches one copy of node-redis's client package (@redis/client or @node-redis/client), given
+ * the path of its package.json, if it's a version that doesn't publish on the
+ * "node-redis:command" channel: @redis/client 1.x and @node-redis/client 1.x (node-redis 4.x),
+ * and @redis/client 5.x before 5.12. Anything newer is left alone for subscribeNodeRedis to cover.
+ *
+ * Wraps the client's command queue (RedisCommandsQueue.prototype.addCommand), the one method every
+ * command goes through in all of those versions: a plain command, sendCommand(), every command in
+ * a pipeline or a MULTI (MULTI and EXEC included), each one its own span, the same as ioredis
+ * pipelines get. The queue settles its promise from the socket's own handlers, so the patch
+ * returns a promise of its own in the queue's place, settling exactly like the queue's: a
+ * rejection nothing ever handles is still reported as unhandled.
+ *
+ * @param {string} packageJsonPath
+ * @param {Parameters<typeof installRedisTracing>[0]} recordSpan
+ * @param {() => boolean} isTracing
+ * @param {Parameters<typeof installRedisTracing>[2]} [recordBreadcrumb]
+ * @param {() => boolean} [ensureBreadcrumbTrail]
+ * @returns {boolean} whether its command queue is patched now
+ */
+export function instrumentNodeRedisClient(packageJsonPath, recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail) {
+  let prototype;
+  try {
+    const require = createRequire(packageJsonPath);
+    if (!publishesNothing(require(packageJsonPath).version)) {
+      return false;
+    }
+    prototype = require("./dist/lib/client/commands-queue.js").default?.prototype;
+  } catch {
+    // Not a layout this knows: leave it alone rather than guess.
+    return false;
+  }
+  if (typeof prototype?.addCommand !== "function") {
+    return false;
+  }
+  if (prototype.addCommand[WRAPPED]) {
+    return true;
+  }
+  const hooks = hooksFor(recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail);
+  const originalAddCommand = prototype.addCommand;
+
+  const patchedAddCommand = function patchedAddCommand(args) {
+    const finish = Array.isArray(args) ? startCommand(commandName(args[0]), hooks) : null;
+    const result = originalAddCommand.apply(this, arguments);
+    if (finish === null || typeof result?.then !== "function") {
+      return result;
+    }
+    return result.then(
+      (value) => {
+        finish(false);
+        return value;
+      },
+      (error) => {
+        finish(true);
+        throw error;
+      },
+    );
+  };
+  patchedAddCommand[WRAPPED] = true;
+
+  patches.push({ target: prototype, key: "addCommand", original: originalAddCommand, own: Object.prototype.hasOwnProperty.call(prototype, "addCommand") });
+  prototype.addCommand = patchedAddCommand;
+  return true;
+}
+
+// node-redis 4.x ships its client as 1.x; 5.12 is the first version with the diagnostics channel.
+function publishesNothing(version) {
+  const match = /^(\d+)\.(\d+)/.exec(String(version));
+  if (match === null) {
+    return false;
+  }
+  const [major, minor] = [Number(match[1]), Number(match[2])];
+  return major === 1 || (major === 5 && minor < 12);
+}
+
+// node-redis passes each argument as a string or a Buffer; only the first (the command) is read.
+function commandName(first) {
+  return String(first ?? "").split(" ", 1)[0];
+}
+
 /** @internal test-only: restores every method installRedisTracing replaced, and unsubscribes. */
 export function _uninstallRedisTracing() {
   while (patches.length > 0) {
@@ -99,59 +204,75 @@ export function _uninstallRedisTracing() {
   installed = false;
 }
 
-function observeCommand(command, recordSpan) {
+function hooksFor(recordSpan, isTracing, recordBreadcrumb = () => {}, ensureBreadcrumbTrail = () => false) {
+  return { recordSpan, isTracing, recordBreadcrumb, ensureBreadcrumbTrail };
+}
+
+function observeCommand(command, finish) {
   const { resolve, reject } = command;
   if (typeof resolve !== "function" || typeof reject !== "function") {
     return;
   }
-  const finish = startSpan(command.name, recordSpan);
   command.resolve = function resolveWithSpan(...args) {
-    finish();
+    finish(false);
     return resolve.apply(this, args);
   };
   command.reject = function rejectWithSpan(...args) {
-    finish();
+    finish(true);
     return reject.apply(this, args);
   };
 }
 
 /**
- * Starts timing one command now and returns the function that records it, bound to the caller's
- * async context: the reply arrives from the connection's own socket handlers, where the trace the
- * command belongs to isn't the current one. Only ever records once.
+ * Starts timing one command now, from the app's own call, and returns the function that records
+ * it (given whether the command failed), or null when there's nothing to record it into: no trace
+ * being captured and no breadcrumb being kept. The function is bound to the caller's async
+ * context: the reply arrives from the connection's own socket handlers, where the trace and the
+ * breadcrumb trail the command belongs to aren't the current ones. Only ever records once.
  */
-function startSpan(commandName, recordSpan) {
-  const name = `Redis ${String(commandName).toUpperCase()}`;
+function startCommand(command, hooks) {
+  const span = hooks.isTracing();
+  const breadcrumb = hooks.ensureBreadcrumbTrail();
+  if (!span && !breadcrumb) {
+    return null;
+  }
+  const name = `Redis ${String(command).toUpperCase()}`;
   const startedAt = new Date();
   const start = performance.now();
   let finished = false;
-  return AsyncResource.bind(() => {
+  return AsyncResource.bind((failed) => {
     if (finished) {
       return;
     }
     finished = true;
-    recordSpan(name, "redis", startedAt, performance.now() - start);
+    const durationMs = performance.now() - start;
+    if (breadcrumb) {
+      hooks.recordBreadcrumb(name, "redis", failed ? "error" : "info", { duration_ms: Math.round(durationMs * 10) / 10 });
+    }
+    if (span) {
+      hooks.recordSpan(name, "redis", startedAt, durationMs);
+    }
   });
 }
 
 /**
  * node-redis runs each command inside TracingChannel#tracePromise: "start" is published
  * synchronously from the app's own call (so isTracing() sees the right trace), and "asyncEnd" once
- * the reply has settled the command either way. Only `context.command` (already upcased by
- * node-redis) is read, never `context.args`.
+ * the reply has settled the command either way, with `context.error` set if it failed. Only
+ * `context.command` (already upcased by node-redis) is read, never `context.args`.
  */
-function subscribeNodeRedis(recordSpan, isTracing) {
+function subscribeNodeRedis(hooks) {
   if (typeof diagnosticsChannel.tracingChannel !== "function") {
     return;
   }
   nodeRedisSubscription = {
     start(context) {
-      if (context && typeof context === "object" && isTracing()) {
-        context[SPAN] = startSpan(context.command, recordSpan);
+      if (context && typeof context === "object") {
+        context[SPAN] = startCommand(context.command, hooks);
       }
     },
     asyncEnd(context) {
-      context?.[SPAN]?.();
+      context?.[SPAN]?.("error" in context);
     },
   };
   diagnosticsChannel.tracingChannel(NODE_REDIS_COMMAND_CHANNEL).subscribe(nodeRedisSubscription);

package/src/sqlStatement.js

@@ -8,8 +8,8 @@
 //
 // Deliberately a single pass over a few patterns, not a SQL parser. No regex lookbehind either:
 // the same source is shared with the browser and React Native SDKs, where an older engine would
-// fail to even parse one, so the "not part of an identifier" check on numbers captures the
-// preceding character and puts it back instead.
+// fail to even parse one, so mask() checks the character before each match itself, and moves on
+// one character when it doesn't count, exactly as the server's lookbehind would.
 
 const MASK = "?";
 const MAX_LENGTH = 4000;
@@ -17,9 +17,24 @@ const MAX_NAMES = 10;
 const MAX_NAME_LENGTH = 200;
 const MAX_CAUSE_DEPTH = 5;
 
-// 1: string literal (or one cut off by truncation)  2: dollar-quote tag  3: char before a number
-// 4: the number, not part of an identifier or a $1 placeholder
-const LITERAL = /'(?:[^']|'')*(?:'|$)|(\$[A-Za-z_]*\$)[\s\S]*?(?:\1|$)|(^|[^\w$.])(\d+(?:\.\d+)?)(?!\w)/g;
+// db.system values where "double quotes" are a string, not a name.
+const DOUBLE_QUOTED_STRING_SYSTEMS = new Set(["mysql", "mariadb"]);
+
+// A 'single quoted' string, '' and backslash escapes included, with its type prefix (E'', X'',
+// N'', B'', U&''; group 1), or one cut off by truncation (masked to the end), even right after a
+// backslash ('secret\ is still a string).
+const STRING = String.raw`((?:[EeXxNnBb]|[Uu]&)?)'(?:[^'\\]|\\(?:[\s\S]|$)|'')*(?:'|$)`;
+const DOUBLE_QUOTED_STRING = String.raw`"(?:[^"\\]|\\(?:[\s\S]|$)|"")*(?:"|$)`;
+// A $tag$ dollar-quoted string (group 2 is the tag).
+const DOLLAR_QUOTED = String.raw`(\$[A-Za-z_]*\$)[\s\S]*?(?:\2|$)`;
+// Hex, binary, integer, decimal, leading-dot decimal, each with an optional exponent (group 3).
+const NUMBER = String.raw`(0[xX][0-9A-Fa-f]+|0[bB][01]+|(?:\d+(?:\.\d+)?|\.\d+)(?:[eE][+-]?\d+)?)(?!\w)`;
+const LITERAL = new RegExp(`${STRING}|${DOLLAR_QUOTED}|${NUMBER}`, "g");
+const LITERAL_WITH_DOUBLE_QUOTES = new RegExp(`${STRING}|${DOUBLE_QUOTED_STRING}|${DOLLAR_QUOTED}|${NUMBER}`, "g");
+// What may not come right before a string's type prefix, or before a number: the end of a word,
+// a $1 placeholder, or (for a number) a decimal point.
+const WORD_END = /[\w$]/;
+const NUMBER_PREFIX = /[\w$.]/;
 
 const PART = '(?:[\\w$#@]+|"[^"]+"|\\[[^\\]]+\\]|`[^`]+`)';
 const NAME = `${PART}(?:\\.${PART})*`;
@@ -72,14 +87,35 @@ export function findIn(error) {
 }
 
 /**
+ * Every string and number replaced by "?". `system` is the db.system the statement ran against,
+ * when known: for MySQL and MariaDB, "double quoted" text is a string there and masked too.
+ *
  * @param {string | null | undefined} statement
+ * @param {string | null} [system]
  * @returns {string | null}
  */
-export function mask(statement) {
+export function mask(statement, system = null) {
   if (typeof statement !== "string" || statement.trim() === "") {
     return null;
   }
-  const masked = statement.replace(LITERAL, (whole, _tag, before, number) => (number === undefined ? MASK : `${before}${MASK}`));
+  const pattern = new RegExp(DOUBLE_QUOTED_STRING_SYSTEMS.has(String(system ?? "").toLowerCase()) ? LITERAL_WITH_DOUBLE_QUOTES : LITERAL);
+  let masked = "";
+  let emitted = 0;
+  let match;
+  while ((match = pattern.exec(statement)) !== null) {
+    const before = statement[match.index - 1] ?? "";
+    const prefixed = match[1] !== undefined && match[1] !== "" && WORD_END.test(before);
+    if (prefixed || (match[3] !== undefined && NUMBER_PREFIX.test(before))) {
+      // Not a match where it starts: try again from the next character, the way a lookbehind
+      // failing there would. A prefix that's the end of a word stays, and the quote after it
+      // still starts a string (LIKE'%x%' masks to LIKE?).
+      pattern.lastIndex = match.index + 1;
+      continue;
+    }
+    masked += statement.slice(emitted, match.index) + MASK;
+    emitted = pattern.lastIndex;
+  }
+  masked += statement.slice(emitted);
   return masked.length > MAX_LENGTH ? `${masked.slice(0, MAX_LENGTH)}...` : masked;
 }