opentel-mcp 0.7.0 → 0.9.0

package/CHANGELOG.md

@@ -1,5 +1,305 @@
 # Changelog
 
+## 0.9.0
+
+**⚠️ Fixed, with a fingerprint behavior change — read this before the
+feature below.** The `auth` failure classifier
+(`src/fingerprint/classify/auth.js`) missed "permission denied" and
+"access denied" — the standard phrasing from Unix, git, AWS IAM, and GCP
+for a permission/authorization failure. It only recognized
+HTTP-status-derived wording (`unauthorized`, `forbidden`,
+`authenticat(e|ion)`) plus 401/403 status codes and a handful of known
+auth-library error names. Messages using the OS/CLI phrasing above fell
+through every classifier and landed in the `internal` catch-all instead.
+Found by running realistic error text through this project's own UI demo
+(`packages/ui/demo/populate.js`) and checking what `DEFAULT_CLASSIFIERS`
+actually returned for it — not assumed. Now also matches "not authorized",
+"permission(s) denied", "access denied", "insufficient permission(s)", and
+Node's own `EACCES`/`EPERM` error codes; still does not match bare
+"authorized" or "permission" alone (see the classifier's own docblock for
+the false-positive cases this deliberately excludes, e.g. "user denied the
+permission request").
+
+**This changes fingerprints for affected messages.** `category` is one of
+the hashed inputs `computeFingerprint()` combines into
+`mcp.failure.fingerprint` (see ADR 006). A permission-denied failure that
+previously classified as `internal` now classifies as `auth` — the fields
+feeding the hash change, so the fingerprint itself changes for anyone whose
+tool emits this wording. This does **not** amend the closed 8-category
+taxonomy ADR 006 established (`validation | timeout | network | auth |
+dependency | serialization | internal | unknown`) — `auth` already existed;
+this is a pattern-coverage fix to when the existing category fires, not a
+new category. If you alert or dashboard on a specific `mcp.failure.fingerprint`
+value for a permission error, expect a new value after upgrading.
+
+### Added — `instanceKey`: sharing tracker state across `instrumentMcpServer()` calls
+
+`instanceKey` (a string option on `instrumentMcpServer()`, or the
+`OTEL_MCP_INSTANCE_KEY` env var — lower precedence than the option) lets
+repeated `instrumentMcpServer()` calls that pass the same key share Agent
+Thrash Detection, budget tracking, schema drift detection, and the
+`ToolOutcome` counter's state, instead of each call constructing all four
+fresh and discarding them. Fixes the gap documented in the README's
+"In-memory tracker state is scoped to one `instrumentMcpServer()` call"
+section and `docs/known-gaps.md` entry 6, under a "stateless" Streamable
+HTTP deployment shape (a fresh `Server`/`McpServer` re-instrumented on
+every incoming request). Full design: ADR 012
+(`docs/adr/012-tracker-lifecycle-and-shared-state.md`).
+
+Backed by an internal, bounded, TTL-evicting registry (1000 distinct keys
+per process, 24h TTL renewed on every use — both ADR 012's proposed
+defaults) — fully internal, no new public type. Omit `instanceKey` (the
+default) for behavior byte-identical to every prior version: trackers are
+constructed fresh on every call, and the registry is never touched.
+
+**⚠️ Composition requirement: `instanceKey` alone does not fix Agent
+Thrash Detection.** `ThrashDetector` looks episodes up by `(sessionId,
+toolName, fingerprint)` — `instanceKey` shares the tracker object, but
+without a real, transport-provided `extra.sessionId` on every call, each
+`instrumentMcpServer()` call still generates its own random per-connection
+fallback session id, fresh, regardless of `instanceKey`. Sharing the
+tracker doesn't help if the lookup key inside it differs every call —
+each request lands as its own one-off episode instead of contributing to
+one shared loop. Real Streamable HTTP transports provide a real session id
+automatically, so the common case works with `instanceKey` alone — but a
+custom `Transport`, `assumeSingleSession: true`, or anything else on the
+generated-fallback path will set `instanceKey`, see nothing happen, and
+have every reason to think the fix is broken. Same silent-inertness shape
+as the original gap, one layer deeper. See the README's new "instanceKey"
+section for the full explanation and what to do about it — this is not a
+footnote there either.
+
+**⚠️ Does not help across process boundaries.** `instanceKey`'s registry
+is one process's in-memory state. On Lambda, Cloud Run, or any
+horizontally-scaled deployment, concurrent/recycled instances each hold
+their own independent registry — passing the identical `instanceKey`
+string everywhere does not change that. Counters remain instance-local and
+best-effort by design; this is a structural limitation, not a
+configuration gap, and this library deliberately does not add an external
+store (Redis/DynamoDB) to close it — see ADR 012's Update section for the
+full reasoning.
+
+**Documentation:** the "Metrics" section now covers wiring the Prometheus
+exporter specifically (`@opentelemetry/exporter-prometheus`), not just the
+OTLP example that was already there. Its pull-based text exposition format
+does not attach resource attributes (including `service.name`) to
+individual metric points by default — only to a separate `target_info`
+series — which is invisible with one service but means every series looks
+identical the moment you're scraping more than one instrumented server
+into the same Prometheus. Documents the fix
+(`withResourceConstantLabels: /^service\.name$/`) with a worked example.
+No code change; this behavior was always there, just undocumented. Found
+building `dashboards/grafana-mcp-health.json`'s verification harness.
+
+## 0.8.0
+
+Three features. **Tool schema drift detection**: a server that silently
+changes a tool's `inputSchema` between deployments — a parameter renamed, a
+type tightened, a `required` field added — currently breaks agents with no
+signal pointing at the actual cause. Full investigation and design: ADR 010
+(`docs/adr/010-schema-drift.md`). **Two-axis observation contract**:
+prompted by external review (Massimiliano Brighindi), who first raised that
+`instrumentMcpServer()` with no `TracerProvider`/`MeterProvider` registered
+silently no-ops — a failed tool call in that state produces zero telemetry,
+indistinguishable from one that never failed — and then supplied the reframe
+that shaped what shipped: not "detect a broken pipeline," but "stop implying
+health by omission." Full investigation and design: ADR 008
+(`docs/adr/008-observation-liveness.md`, "Update (2026-08-05)" section).
+**Cost-aware trace sampling**: investigated whether traces that were
+expensive or thrashed could be kept regardless of the head sampler's
+decision. Found that this cannot be an in-process library feature — a
+`Sampler` decides at span start, cost/thrash are only known at span end, and
+this package doesn't own the `Sampler`/`SpanProcessor` chain in its default
+configuration anyway. Ships a marker attribute plus a documented Collector
+recipe instead of a sampler. Full investigation and design: ADR 011
+(`docs/adr/011-cost-aware-sampling.md`).
+
+**⚠️ Read "Changed — behavior change on upgrade" immediately below before
+updating.** One of the three features above changes when
+`instrumentMcpServer()` throws, for a subset of low-level `Server` users,
+purely from the new default-on config — no code change of your own
+required to hit it.
+
+### Changed — behavior change on upgrade, read before updating
+
+- **The instrument-first ordering requirement now also covers `tools/list`,
+  for low-level `Server` users specifically, because `schemaDrift.enabled`
+  defaults to `true`.** `instrumentMcpServer()` has always required being
+  called before any `tools/call` handler is registered; as of this release,
+  with schema drift enabled (the default), it also requires being called
+  before any `tools/list` handler is registered. If your low-level `Server`
+  code registers `server.setRequestHandler(ListToolsRequestSchema, ...)`
+  before calling `instrumentMcpServer()` — never previously an error, since
+  this library was blind to `tools/list` entirely before this release —
+  upgrading will make that throw `INSTRUMENT_FIRST_ERROR` where it didn't
+  before, with no other code changes on your part.
+  - **`McpServer` users are unaffected.** `McpServer` registers `tools/list`
+    and `tools/call` together, atomically, the first time `.tool()` or
+    `.registerTool()` is called — so anyone already following the
+    documented instrument-before-registration rule for `tools/call`
+    automatically satisfies it for `tools/list` too.
+  - **Migration**: either reorder your `tools/list` registration to after
+    `instrumentMcpServer()`, or pass `schemaDrift: { enabled: false }` to
+    opt out and keep your existing registration order — both fully
+    restore v0.7.0 behavior. There is no change if you don't use a
+    low-level `Server` with an independently-registered `tools/list`
+    handler.
+  - Only `schemaDrift` (schema drift detection, below) causes this — the
+    two-axis observation contract and cost-aware sampling features in this
+    same release are purely additive, with no effect on when
+    `instrumentMcpServer()` throws.
+
+### Added
+
+- **Tool schema drift detection.** Every `tools/list` response is captured,
+  canonicalized, and hashed per tool (`inputSchema` only — `description` is
+  a deliberately separate, not-yet-built dimension; see ADR 010). When a
+  previously-observed tool's schema hash changes, this emits:
+  - An `mcp.tool.schema_drift.detected` counter, labeled `gen_ai.tool.name`
+    and `mcp.tool.schema_drift.type` (both bounded — see
+    `METRIC_SAFE_ATTRIBUTES`, `src/schema-drift/attributes.js`).
+  - An `mcp.tool.schema_drift.detected` span event on the new `tools/list`
+    span (see Changed, below), carrying the drift `type`, the previous/
+    current schema hash, and — only when non-empty, never set to `[]` —
+    which field names were added/removed/changed. Field names are
+    span-only, never a metric label (unbounded across tools/deployments,
+    same reasoning as `mcp.failure.validation_paths`, ADR 009).
+  - `type` is one of `field_added` \| `field_removed` \| `type_changed` \|
+    `required_changed` \| `multiple` (more than one at once, never guessed
+    down to a single answer) \| `unknown` (a change this differ can't
+    confidently characterize — e.g. a top-level `oneOf`/`anyOf`/`allOf`
+    composition, or a change hidden behind a `$ref`/`$defs` indirection
+    that doesn't touch the referencing property's own value — still
+    reported as drift, just not attributable to a specific field).
+  - The **first** observation of any given tool is never reported as
+    drift (nothing to compare against yet — cold start). A tool that
+    stops appearing in `tools/list` responses for a while and later
+    reappears is compared against its last-seen schema, not treated as a
+    fresh cold start — see the README's "Tool schema drift detection"
+    section for why this is the correct, and possibly counter-intuitive,
+    behavior.
+  - State is scoped **per instrumented server instance, not per session**
+    (ADR 010, Q4) — every client session sees the same tool registry, so
+    session-keyed state would produce false cold-starts per new session
+    and could silently swallow drift that happened between sessions.
+  - New `schemaDrift` option on `instrumentMcpServer()`, following the
+    exact `thrashDetection`/`costTracking` partial-overrides-individual-
+    defaults pattern: `enabled` (default `true`) and `maxTrackedTools`
+    (default `1000`, an LRU cap — defense-in-depth, not a response to an
+    expected failure mode; a server's own tool count is normally small).
+    Each independently overridable via `OTEL_MCP_SCHEMA_DRIFT_ENABLED` /
+    `OTEL_MCP_SCHEMA_DRIFT_MAX_TRACKED_TOOLS`, following the existing
+    `OTEL_MCP_THRASH_*` env-var convention.
+    **Unlike** `thrashDetection`/`costTracking`, `schemaDrift.enabled: false`
+    is a true no-op: `tools/list` is not wrapped at all (no span, no
+    capture, no detector/emitter construction) — the `tools/list` span
+    this feature introduces exists purely for schema drift, unlike the
+    `tools/call` span, which already serves other purposes regardless of
+    sub-feature flags.
+  - `SchemaDriftConfig`, `SchemaDriftKind`, `SchemaDriftEvent` types,
+    exported from the package root.
+
+### Added — Two-axis observation contract
+
+- **`getObservationState()`**, a new accessor attached to the object
+  `instrumentMcpServer()` returns — unconditional (not gated behind
+  `setupNodeSdk`), same additive pattern as `shutdown()`/`getThrashSummary()`,
+  omitted entirely when `options.enabled` is `false`. Returns:
+  ```
+  {
+    toolOutcome: { success, failure, unknown },
+    observationIntegrity: 'DEGRADED' | 'UNKNOWN',
+  }
+  ```
+  No OTel emission — purely in-process, nothing sent anywhere, safe to
+  call from application code (a health-check endpoint, a periodic
+  `console.log`, a debugger). See the README's "Two-axis observation
+  contract" section for the full design rationale.
+  - **`toolOutcome`**: cumulative tool-call outcome counts since
+    instrumentation, from a **new counter that increments on every tool
+    call unconditionally** — independent of `fingerprinting`,
+    `thrashDetection`, and `enableMetrics`. Deliberately NOT read off
+    `ThrashDetector`/`getThrashSummary()`: that bookkeeping only runs
+    when a fingerprint was computed, so with `fingerprinting: false` (a
+    fully supported configuration) it would silently report zero
+    failures regardless of how many actually occurred — the exact
+    silent-success failure mode this feature exists to close. A
+    malformed, unrecognizable tool result (not a real `CallToolResult`
+    shape) increments `unknown` rather than silently defaulting to
+    `success`.
+  - **`observationIntegrity`**: `'DEGRADED' | 'UNKNOWN'` — note there is
+    no `'HEALTHY'` value, and this is not an oversight. Investigated and
+    found structurally unreachable in every configuration: the one lead
+    (OTel SDK self-observability metrics) is a write-only `Counter` with
+    no synchronous read-back API in `@opentelemetry/api`, so this
+    library's own code can never positively confirm telemetry is
+    flowing, no matter how it's wired up. `HEALTHY` is therefore absent
+    from the type entirely, not merely never returned — enforced by
+    TypeScript, not just documentation (see the new type-level tests in
+    `test/index.exports.test-d.ts`).
+    - `DEGRADED` is detected via a fragile `ProxyTracerProvider`
+      reference-equality check, and is reachable **only** under
+      `setupNodeSdk: false` (the default) — when no `TracerProvider` has
+      been registered globally at all.
+    - Under `setupNodeSdk: true`, this library registers the provider
+      itself, so absence can never be confirmed — `observationIntegrity`
+      is **always** `'UNKNOWN'` in that configuration, without even
+      attempting the check.
+    - Recomputed fresh on **every call** to `getObservationState()`,
+      never cached from instrument time — a host may register a
+      `TracerProvider` asynchronously after `instrumentMcpServer()`
+      already ran, and a value cached at startup would go stale the
+      moment that happens.
+  - `ToolOutcome`, `ToolOutcomeCounts`, `ObservationIntegrity`,
+    `ObservationState` types, exported from the package root.
+
+### Added — Cost-aware trace sampling (marker attribute + Collector recipe)
+
+- **`mcp.tool.thrash_detected`, a new boolean span attribute**, set
+  alongside (never instead of) the existing `mcp.loop.detected` span
+  event, in the same `thrash/emitter.js` call site — set only when
+  `thrashDetection` is enabled and a loop was actually detected on this
+  call, same reachability as the existing event, no new failure mode.
+  Exists specifically so an OpenTelemetry Collector's
+  `tailsamplingprocessor` has an unambiguous, attribute-level signal to
+  key on: whether a `boolean_attribute` policy can also match span-*event*
+  data was investigated and left genuinely unverified (the processor is
+  Go source in a separate repository, not installed here), so this
+  attribute removes that uncertainty entirely rather than leaving tail
+  sampling dependent on an unconfirmed answer. **Named deliberately
+  differently** from the pre-existing `mcp.tool.loop.detected` **metric**
+  counter, not reusing its string as ADR 011 originally specified — see
+  that ADR's "Update" note. A metric name and a span attribute key are
+  unrelated OTel namespaces with no technical conflict, but reusing the
+  name left the one reader who most needs it to be unambiguous — someone
+  writing a Collector tail-sampling policy — unable to tell, from the
+  name alone, which of the two same-named signals they were keying on.
+- **No new cost-threshold attribute or config.** `mcp.tool.cost.usd` and
+  `mcp.tool.cost.budget_exceeded` (both already shipped, v0.5.0) already
+  fully suffice for a Collector `numeric_attribute` / `boolean_attribute`
+  policy — the numeric threshold itself lives entirely in the
+  Collector's own policy config (the YAML), not in this package's
+  `InstrumentOptions`. No new env var, no new `instrumentMcpServer()`
+  option.
+- **A documented, pasteable OpenTelemetry Collector `tailsamplingprocessor`
+  config** (README's "Cost-aware trace sampling" section) keeping any
+  trace with an expensive call, a budget-exceeded call, or a detected
+  thrash loop, alongside an ordinary probabilistic sample for everything
+  else.
+- **No in-process sampler or buffering `SpanProcessor` was built, and none
+  is planned** — investigated and rejected on two independent grounds
+  (ADR 011): this package doesn't own the `Sampler`/`SpanProcessor` chain
+  in its default configuration (no public API to inject either into a
+  host-owned `TracerProvider`), and even where a custom processor could
+  theoretically be installed, an in-process decision can only ever rescue
+  the one span this package itself creates — never an already-finished
+  child span from other instrumentation, never an upstream span in a
+  different process. "Keep the trace" is not achievable in-process; at
+  best, "keep this one span" is, which is a materially smaller guarantee
+  than the stated goal. Real cross-span, cross-process trace buffering is
+  what the Collector's `tailsamplingprocessor` already does correctly —
+  not something to partially re-implement inside this package.
+
 ## 0.7.0
 
 Origin-aware failure classification for Agent Thrash Detection. Prompted by