opentel-mcp 0.6.1 → 0.8.0

package/CHANGELOG.md

@@ -1,5 +1,351 @@
 # Changelog
 
+## 0.8.0
+
+Three features. **Tool schema drift detection**: a server that silently
+changes a tool's `inputSchema` between deployments — a parameter renamed, a
+type tightened, a `required` field added — currently breaks agents with no
+signal pointing at the actual cause. Full investigation and design: ADR 010
+(`docs/adr/010-schema-drift.md`). **Two-axis observation contract**:
+prompted by external review (Massimiliano Brighindi), who first raised that
+`instrumentMcpServer()` with no `TracerProvider`/`MeterProvider` registered
+silently no-ops — a failed tool call in that state produces zero telemetry,
+indistinguishable from one that never failed — and then supplied the reframe
+that shaped what shipped: not "detect a broken pipeline," but "stop implying
+health by omission." Full investigation and design: ADR 008
+(`docs/adr/008-observation-liveness.md`, "Update (2026-08-05)" section).
+**Cost-aware trace sampling**: investigated whether traces that were
+expensive or thrashed could be kept regardless of the head sampler's
+decision. Found that this cannot be an in-process library feature — a
+`Sampler` decides at span start, cost/thrash are only known at span end, and
+this package doesn't own the `Sampler`/`SpanProcessor` chain in its default
+configuration anyway. Ships a marker attribute plus a documented Collector
+recipe instead of a sampler. Full investigation and design: ADR 011
+(`docs/adr/011-cost-aware-sampling.md`).
+
+**⚠️ Read "Changed — behavior change on upgrade" immediately below before
+updating.** One of the three features above changes when
+`instrumentMcpServer()` throws, for a subset of low-level `Server` users,
+purely from the new default-on config — no code change of your own
+required to hit it.
+
+### Changed — behavior change on upgrade, read before updating
+
+- **The instrument-first ordering requirement now also covers `tools/list`,
+  for low-level `Server` users specifically, because `schemaDrift.enabled`
+  defaults to `true`.** `instrumentMcpServer()` has always required being
+  called before any `tools/call` handler is registered; as of this release,
+  with schema drift enabled (the default), it also requires being called
+  before any `tools/list` handler is registered. If your low-level `Server`
+  code registers `server.setRequestHandler(ListToolsRequestSchema, ...)`
+  before calling `instrumentMcpServer()` — never previously an error, since
+  this library was blind to `tools/list` entirely before this release —
+  upgrading will make that throw `INSTRUMENT_FIRST_ERROR` where it didn't
+  before, with no other code changes on your part.
+  - **`McpServer` users are unaffected.** `McpServer` registers `tools/list`
+    and `tools/call` together, atomically, the first time `.tool()` or
+    `.registerTool()` is called — so anyone already following the
+    documented instrument-before-registration rule for `tools/call`
+    automatically satisfies it for `tools/list` too.
+  - **Migration**: either reorder your `tools/list` registration to after
+    `instrumentMcpServer()`, or pass `schemaDrift: { enabled: false }` to
+    opt out and keep your existing registration order — both fully
+    restore v0.7.0 behavior. There is no change if you don't use a
+    low-level `Server` with an independently-registered `tools/list`
+    handler.
+  - Only `schemaDrift` (schema drift detection, below) causes this — the
+    two-axis observation contract and cost-aware sampling features in this
+    same release are purely additive, with no effect on when
+    `instrumentMcpServer()` throws.
+
+### Added
+
+- **Tool schema drift detection.** Every `tools/list` response is captured,
+  canonicalized, and hashed per tool (`inputSchema` only — `description` is
+  a deliberately separate, not-yet-built dimension; see ADR 010). When a
+  previously-observed tool's schema hash changes, this emits:
+  - An `mcp.tool.schema_drift.detected` counter, labeled `gen_ai.tool.name`
+    and `mcp.tool.schema_drift.type` (both bounded — see
+    `METRIC_SAFE_ATTRIBUTES`, `src/schema-drift/attributes.js`).
+  - An `mcp.tool.schema_drift.detected` span event on the new `tools/list`
+    span (see Changed, below), carrying the drift `type`, the previous/
+    current schema hash, and — only when non-empty, never set to `[]` —
+    which field names were added/removed/changed. Field names are
+    span-only, never a metric label (unbounded across tools/deployments,
+    same reasoning as `mcp.failure.validation_paths`, ADR 009).
+  - `type` is one of `field_added` \| `field_removed` \| `type_changed` \|
+    `required_changed` \| `multiple` (more than one at once, never guessed
+    down to a single answer) \| `unknown` (a change this differ can't
+    confidently characterize — e.g. a top-level `oneOf`/`anyOf`/`allOf`
+    composition, or a change hidden behind a `$ref`/`$defs` indirection
+    that doesn't touch the referencing property's own value — still
+    reported as drift, just not attributable to a specific field).
+  - The **first** observation of any given tool is never reported as
+    drift (nothing to compare against yet — cold start). A tool that
+    stops appearing in `tools/list` responses for a while and later
+    reappears is compared against its last-seen schema, not treated as a
+    fresh cold start — see the README's "Tool schema drift detection"
+    section for why this is the correct, and possibly counter-intuitive,
+    behavior.
+  - State is scoped **per instrumented server instance, not per session**
+    (ADR 010, Q4) — every client session sees the same tool registry, so
+    session-keyed state would produce false cold-starts per new session
+    and could silently swallow drift that happened between sessions.
+  - New `schemaDrift` option on `instrumentMcpServer()`, following the
+    exact `thrashDetection`/`costTracking` partial-overrides-individual-
+    defaults pattern: `enabled` (default `true`) and `maxTrackedTools`
+    (default `1000`, an LRU cap — defense-in-depth, not a response to an
+    expected failure mode; a server's own tool count is normally small).
+    Each independently overridable via `OTEL_MCP_SCHEMA_DRIFT_ENABLED` /
+    `OTEL_MCP_SCHEMA_DRIFT_MAX_TRACKED_TOOLS`, following the existing
+    `OTEL_MCP_THRASH_*` env-var convention.
+    **Unlike** `thrashDetection`/`costTracking`, `schemaDrift.enabled: false`
+    is a true no-op: `tools/list` is not wrapped at all (no span, no
+    capture, no detector/emitter construction) — the `tools/list` span
+    this feature introduces exists purely for schema drift, unlike the
+    `tools/call` span, which already serves other purposes regardless of
+    sub-feature flags.
+  - `SchemaDriftConfig`, `SchemaDriftKind`, `SchemaDriftEvent` types,
+    exported from the package root.
+
+### Added — Two-axis observation contract
+
+- **`getObservationState()`**, a new accessor attached to the object
+  `instrumentMcpServer()` returns — unconditional (not gated behind
+  `setupNodeSdk`), same additive pattern as `shutdown()`/`getThrashSummary()`,
+  omitted entirely when `options.enabled` is `false`. Returns:
+  ```
+  {
+    toolOutcome: { success, failure, unknown },
+    observationIntegrity: 'DEGRADED' | 'UNKNOWN',
+  }
+  ```
+  No OTel emission — purely in-process, nothing sent anywhere, safe to
+  call from application code (a health-check endpoint, a periodic
+  `console.log`, a debugger). See the README's "Two-axis observation
+  contract" section for the full design rationale.
+  - **`toolOutcome`**: cumulative tool-call outcome counts since
+    instrumentation, from a **new counter that increments on every tool
+    call unconditionally** — independent of `fingerprinting`,
+    `thrashDetection`, and `enableMetrics`. Deliberately NOT read off
+    `ThrashDetector`/`getThrashSummary()`: that bookkeeping only runs
+    when a fingerprint was computed, so with `fingerprinting: false` (a
+    fully supported configuration) it would silently report zero
+    failures regardless of how many actually occurred — the exact
+    silent-success failure mode this feature exists to close. A
+    malformed, unrecognizable tool result (not a real `CallToolResult`
+    shape) increments `unknown` rather than silently defaulting to
+    `success`.
+  - **`observationIntegrity`**: `'DEGRADED' | 'UNKNOWN'` — note there is
+    no `'HEALTHY'` value, and this is not an oversight. Investigated and
+    found structurally unreachable in every configuration: the one lead
+    (OTel SDK self-observability metrics) is a write-only `Counter` with
+    no synchronous read-back API in `@opentelemetry/api`, so this
+    library's own code can never positively confirm telemetry is
+    flowing, no matter how it's wired up. `HEALTHY` is therefore absent
+    from the type entirely, not merely never returned — enforced by
+    TypeScript, not just documentation (see the new type-level tests in
+    `test/index.exports.test-d.ts`).
+    - `DEGRADED` is detected via a fragile `ProxyTracerProvider`
+      reference-equality check, and is reachable **only** under
+      `setupNodeSdk: false` (the default) — when no `TracerProvider` has
+      been registered globally at all.
+    - Under `setupNodeSdk: true`, this library registers the provider
+      itself, so absence can never be confirmed — `observationIntegrity`
+      is **always** `'UNKNOWN'` in that configuration, without even
+      attempting the check.
+    - Recomputed fresh on **every call** to `getObservationState()`,
+      never cached from instrument time — a host may register a
+      `TracerProvider` asynchronously after `instrumentMcpServer()`
+      already ran, and a value cached at startup would go stale the
+      moment that happens.
+  - `ToolOutcome`, `ToolOutcomeCounts`, `ObservationIntegrity`,
+    `ObservationState` types, exported from the package root.
+
+### Added — Cost-aware trace sampling (marker attribute + Collector recipe)
+
+- **`mcp.tool.thrash_detected`, a new boolean span attribute**, set
+  alongside (never instead of) the existing `mcp.loop.detected` span
+  event, in the same `thrash/emitter.js` call site — set only when
+  `thrashDetection` is enabled and a loop was actually detected on this
+  call, same reachability as the existing event, no new failure mode.
+  Exists specifically so an OpenTelemetry Collector's
+  `tailsamplingprocessor` has an unambiguous, attribute-level signal to
+  key on: whether a `boolean_attribute` policy can also match span-*event*
+  data was investigated and left genuinely unverified (the processor is
+  Go source in a separate repository, not installed here), so this
+  attribute removes that uncertainty entirely rather than leaving tail
+  sampling dependent on an unconfirmed answer. **Named deliberately
+  differently** from the pre-existing `mcp.tool.loop.detected` **metric**
+  counter, not reusing its string as ADR 011 originally specified — see
+  that ADR's "Update" note. A metric name and a span attribute key are
+  unrelated OTel namespaces with no technical conflict, but reusing the
+  name left the one reader who most needs it to be unambiguous — someone
+  writing a Collector tail-sampling policy — unable to tell, from the
+  name alone, which of the two same-named signals they were keying on.
+- **No new cost-threshold attribute or config.** `mcp.tool.cost.usd` and
+  `mcp.tool.cost.budget_exceeded` (both already shipped, v0.5.0) already
+  fully suffice for a Collector `numeric_attribute` / `boolean_attribute`
+  policy — the numeric threshold itself lives entirely in the
+  Collector's own policy config (the YAML), not in this package's
+  `InstrumentOptions`. No new env var, no new `instrumentMcpServer()`
+  option.
+- **A documented, pasteable OpenTelemetry Collector `tailsamplingprocessor`
+  config** (README's "Cost-aware trace sampling" section) keeping any
+  trace with an expensive call, a budget-exceeded call, or a detected
+  thrash loop, alongside an ordinary probabilistic sample for everything
+  else.
+- **No in-process sampler or buffering `SpanProcessor` was built, and none
+  is planned** — investigated and rejected on two independent grounds
+  (ADR 011): this package doesn't own the `Sampler`/`SpanProcessor` chain
+  in its default configuration (no public API to inject either into a
+  host-owned `TracerProvider`), and even where a custom processor could
+  theoretically be installed, an in-process decision can only ever rescue
+  the one span this package itself creates — never an already-finished
+  child span from other instrumentation, never an upstream span in a
+  different process. "Keep the trace" is not achievable in-process; at
+  best, "keep this one span" is, which is a materially smaller guarantee
+  than the stated goal. Real cross-span, cross-process trace buffering is
+  what the Collector's `tailsamplingprocessor` already does correctly —
+  not something to partially re-implement inside this package.
+
+## 0.7.0
+
+Origin-aware failure classification for Agent Thrash Detection. Prompted by
+external review (Reddit) pointing out that failure detection only read the
+`isError` channel, missing JSON-RPC protocol-level failures. Verifying that
+report surfaced a separate, more consequential finding: a real false
+positive already live in every published version — output-validation
+failures (a server-side bug) being counted as agent thrash. Full
+investigation and design: ADR 007 (`docs/adr/007-protocol-error-channel.md`).
+
+Also investigates a second external report (u/Pleasant-Ad192): whether a
+different schema field failing validation each attempt (an agent
+converging) can be told apart from the same field failing repeatedly (an
+ambiguous tool schema). Finding: it mostly already can be, as a side
+effect of how failures are fingerprinted — see ADR 009
+(`docs/adr/009-field-level-convergence.md`) — now pinned down by
+regression tests and a diagnostic span attribute, with one real gap
+(partial convergence) still open pending a design decision.
+
+Known gaps and open questions this release didn't close: `docs/known-gaps.md`.
+
+### Fixed
+
+- **Output-validation failures were miscategorized and incorrectly counted
+  toward Agent Thrash Detection.** When a tool's own handler returned output
+  that didn't match its declared output schema, the resulting failure
+  (surfaced as `isError: true`, whether thrown directly or converted by the
+  high-level `McpServer`) landed in fingerprint category `validation` or
+  `internal` depending on the exact wording, and — since Agent Thrash
+  Detection shipped in v0.6.0 — was tracked exactly like a normal
+  business-logic failure. An agent retrying such a tool would eventually
+  cross the default threshold and get flagged as "thrashing," even though
+  the failure is entirely the tool author's bug: no argument the agent
+  supplies can ever fix a server that never returns valid structured
+  content. **Affected range: the miscategorization itself has been present
+  since v0.4.0 (deep-failure fingerprinting); the false-positive thrash
+  count has been present since v0.6.0 (Agent Thrash Detection), through the
+  last published release, v0.6.1.** Fixed by classifying which *channel* a
+  failure arrived on (see Added, below) and excluding the `protocol.output`
+  channel from thrash detection entirely — not merely relabeling it.
+
+### Added
+
+- **`mcp.failure.channel` span attribute** — one of `execution` |
+  `protocol.not_found` | `protocol.input` | `protocol.output` |
+  `protocol.other` | `unknown`, classifying which channel a tools/call
+  failure arrived on (`classifyFailureChannel()`,
+  `src/fingerprint/classify/channel.js`). Additive: never part of
+  `computeFingerprint()`'s hash input (see Unchanged, below) — deliberately
+  a separate attribute from the pre-existing `mcp.failure.origin`, which
+  means something different (`tool_error` \| `thrown` \| `transport`) and
+  has been hashed since v0.4.0.
+- **Per-channel Agent Thrash Detection thresholds**: `inputThreshold`
+  (default `5`, higher than the base `threshold`) for the `protocol.input`
+  channel — an agent retrying with different arguments after an
+  input-validation failure may be genuinely converging, not thrashing —
+  and `notFoundThreshold` (default `1`, an immediate flag) for
+  `protocol.not_found` — retrying a tool name that doesn't exist is never
+  convergence. Each independently overridable via its own env var
+  (`OTEL_MCP_THRASH_INPUT_THRESHOLD` / `OTEL_MCP_THRASH_NOT_FOUND_THRESHOLD`),
+  following the exact existing `OTEL_MCP_THRASH_*` pattern.
+- `FailureChannel` type, exported from the package root alongside the
+  existing `FailureCategory` / `FailureOrigin` types.
+- For high-level `McpServer` users specifically: since `McpServer` converts
+  most protocol-shaped failures (tool not found, disabled, input/output
+  validation) to `isError: true` before this library ever sees a thrown
+  error, `classifyFailureChannel()` also recovers the real channel from
+  that disguised form by reading the `MCP error {code}: ` wrapper
+  `McpError`'s constructor always applies, which `McpServer` preserves
+  verbatim. Without this, `protocol.output`'s exclusion (the fix above)
+  would only have applied to hand-rolled low-level `Server` apps, not to
+  `McpServer` — see the README's "Agent Thrash Detection" section and ADR
+  007's addendum for the full reachability picture and its limits.
+- **`mcp.failure.validation_paths` span attribute** — which schema
+  field(s) a Zod validation failure named, one dot-joined path per
+  failing issue (e.g. `["email", "user.profile.age"]`), best-effort
+  extracted from the same message text `classifyFailureChannel()` already
+  reads (`extractValidationPaths()`,
+  `src/fingerprint/classify/validation-paths.js`). Omitted entirely —
+  never set to an empty array — when nothing confidently parseable was
+  found. Span-only, permanently excluded from
+  `METRIC_SAFE_ATTRIBUTES`: field/path names are bounded per tool but
+  unbounded across every tool anyone registers, the same reasoning that
+  already keeps `mcp.failure.fingerprint`/`signature`/`error_class` off
+  metric labels. Full investigation and design: ADR 009
+  (`docs/adr/009-field-level-convergence.md`).
+
+### Unchanged
+
+- **Fingerprints (`mcp.failure.fingerprint` and every other
+  `FingerprintInputs` field) are byte-identical to v0.6.1 for the same
+  inputs.** The new `channel` dimension is deliberately kept out of
+  `computeFingerprint()`'s hash input (ADR 007) specifically so this
+  release cannot change any consumer's existing `mcp.failure.fingerprint`
+  values — a change there would silently break any alert or dashboard
+  built on fingerprint identity. Verified, not just asserted: by extracting
+  the actual, published `v0.6.1` git tag's `src/fingerprint/` tree via `git
+  archive` into an isolated directory and running its `computeFingerprint()`
+  directly, independent of this working tree, against six fixture inputs —
+  see `test/fingerprint/compose.fixtures.test.js`. If you have alerts or
+  dashboards keyed on `mcp.failure.fingerprint`, they keep working exactly
+  as they did on v0.6.1, with no changes required on your end.
+- **Field-level discrimination in Agent Thrash Detection is not a new
+  capability — it already worked, as a side effect of fingerprinting the
+  full Zod issues JSON, and was simply incidental until now.** A
+  validation failure repeating on the *same* schema field across attempts
+  already hashed to the *same* fingerprint (accumulating correctly toward
+  `inputThreshold`), and a *different* field failing each attempt already
+  hashed to a *different* fingerprint each time (never accumulating,
+  matching a converging agent). Investigated and confirmed in ADR 009; now
+  pinned down by regression tests
+  (`test/fingerprint/field-level-convergence.test.js`) so a future Zod or
+  SDK change that silently breaks it gets caught, rather than discovered
+  as a production regression. One real gap remains open and is *not*
+  fixed by this: partial convergence (fixing one of several failing
+  fields changes the issues array's shape and breaks fingerprint
+  continuity) — tracked in `docs/known-gaps.md`, pending a design ADR 009
+  did not settle on.
+
+### Documentation
+
+- `docs/known-gaps.md` (new): five tracked gaps this release didn't close,
+  each written as a ready-to-paste GitHub issue — field-level convergence
+  tracking, partial convergence in field-level validation, the
+  observation-liveness contract, the pre-handler parse-failure gap, and
+  how client-side retry caps interact with
+  detection thresholds.
+- README's "Agent Thrash Detection" section now covers channel-aware
+  thresholds, the `McpServer`-vs-low-level-`Server` reachability
+  difference (with ADR 007's full table), the pre-handler parse-failure
+  gap (deferred, not solved — closing it means revisiting ADR 001), and
+  the forwarded-error collision risk as a named known limitation.
+- README's "Failure Fingerprinting" section now documents
+  `mcp.failure.validation_paths` and states plainly that field-level
+  discrimination is a property of the fingerprint, not a separate
+  detector — see ADR 009.
+
 ## 0.6.1
 
 ### Fixed