opentel-mcp 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +153 -0
- package/README.md +175 -3
- package/package.json +6 -3
- package/src/cost/calculator.d.ts +13 -0
- package/src/cost/extractor.d.ts +8 -0
- package/src/cost/pricing.d.ts +8 -0
- package/src/fingerprint/attributes.d.ts +20 -0
- package/src/fingerprint/attributes.js +38 -1
- package/src/fingerprint/classify/channel.d.ts +9 -0
- package/src/fingerprint/classify/channel.js +223 -0
- package/src/fingerprint/classify/index.d.ts +18 -0
- package/src/fingerprint/classify/validation-paths.js +166 -0
- package/src/fingerprint/compose.d.ts +11 -0
- package/src/fingerprint/types.d.ts +40 -0
- package/src/index.d.ts +1 -0
- package/src/instrument.js +106 -8
- package/src/thrash/config.js +17 -0
- package/src/thrash/detector.js +79 -8
- package/src/thrash/types.d.ts +35 -3
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,158 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.7.0
|
|
4
|
+
|
|
5
|
+
Origin-aware failure classification for Agent Thrash Detection. Prompted by
|
|
6
|
+
external review (Reddit) pointing out that failure detection only read the
|
|
7
|
+
`isError` channel, missing JSON-RPC protocol-level failures. Verifying that
|
|
8
|
+
report surfaced a separate, more consequential finding: a real false
|
|
9
|
+
positive already live in every published version — output-validation
|
|
10
|
+
failures (a server-side bug) being counted as agent thrash. Full
|
|
11
|
+
investigation and design: ADR 007 (`docs/adr/007-protocol-error-channel.md`).
|
|
12
|
+
|
|
13
|
+
Also investigates a second external report (u/Pleasant-Ad192): whether a
|
|
14
|
+
different schema field failing validation each attempt (an agent
|
|
15
|
+
converging) can be told apart from the same field failing repeatedly (an
|
|
16
|
+
ambiguous tool schema). Finding: it mostly already can be, as a side
|
|
17
|
+
effect of how failures are fingerprinted — see ADR 009
|
|
18
|
+
(`docs/adr/009-field-level-convergence.md`) — now pinned down by
|
|
19
|
+
regression tests and a diagnostic span attribute, with one real gap
|
|
20
|
+
(partial convergence) still open pending a design decision.
|
|
21
|
+
|
|
22
|
+
Known gaps and open questions this release didn't close: `docs/known-gaps.md`.
|
|
23
|
+
|
|
24
|
+
### Fixed
|
|
25
|
+
|
|
26
|
+
- **Output-validation failures were miscategorized and incorrectly counted
|
|
27
|
+
toward Agent Thrash Detection.** When a tool's own handler returned output
|
|
28
|
+
that didn't match its declared output schema, the resulting failure
|
|
29
|
+
(surfaced as `isError: true`, whether thrown directly or converted by the
|
|
30
|
+
high-level `McpServer`) landed in fingerprint category `validation` or
|
|
31
|
+
`internal` depending on the exact wording, and — since Agent Thrash
|
|
32
|
+
Detection shipped in v0.6.0 — was tracked exactly like a normal
|
|
33
|
+
business-logic failure. An agent retrying such a tool would eventually
|
|
34
|
+
cross the default threshold and get flagged as "thrashing," even though
|
|
35
|
+
the failure is entirely the tool author's bug: no argument the agent
|
|
36
|
+
supplies can ever fix a server that never returns valid structured
|
|
37
|
+
content. **Affected range: the miscategorization itself has been present
|
|
38
|
+
since v0.4.0 (deep-failure fingerprinting); the false-positive thrash
|
|
39
|
+
count has been present since v0.6.0 (Agent Thrash Detection), through the
|
|
40
|
+
last published release, v0.6.1.** Fixed by classifying which *channel* a
|
|
41
|
+
failure arrived on (see Added, below) and excluding the `protocol.output`
|
|
42
|
+
channel from thrash detection entirely — not merely relabeling it.
|
|
43
|
+
|
|
44
|
+
### Added
|
|
45
|
+
|
|
46
|
+
- **`mcp.failure.channel` span attribute** — one of `execution` |
|
|
47
|
+
`protocol.not_found` | `protocol.input` | `protocol.output` |
|
|
48
|
+
`protocol.other` | `unknown`, classifying which channel a tools/call
|
|
49
|
+
failure arrived on (`classifyFailureChannel()`,
|
|
50
|
+
`src/fingerprint/classify/channel.js`). Additive: never part of
|
|
51
|
+
`computeFingerprint()`'s hash input (see Unchanged, below) — deliberately
|
|
52
|
+
a separate attribute from the pre-existing `mcp.failure.origin`, which
|
|
53
|
+
means something different (`tool_error` \| `thrown` \| `transport`) and
|
|
54
|
+
has been hashed since v0.4.0.
|
|
55
|
+
- **Per-channel Agent Thrash Detection thresholds**: `inputThreshold`
|
|
56
|
+
(default `5`, higher than the base `threshold`) for the `protocol.input`
|
|
57
|
+
channel — an agent retrying with different arguments after an
|
|
58
|
+
input-validation failure may be genuinely converging, not thrashing —
|
|
59
|
+
and `notFoundThreshold` (default `1`, an immediate flag) for
|
|
60
|
+
`protocol.not_found` — retrying a tool name that doesn't exist is never
|
|
61
|
+
convergence. Each independently overridable via its own env var
|
|
62
|
+
(`OTEL_MCP_THRASH_INPUT_THRESHOLD` / `OTEL_MCP_THRASH_NOT_FOUND_THRESHOLD`),
|
|
63
|
+
following the exact existing `OTEL_MCP_THRASH_*` pattern.
|
|
64
|
+
- `FailureChannel` type, exported from the package root alongside the
|
|
65
|
+
existing `FailureCategory` / `FailureOrigin` types.
|
|
66
|
+
- For high-level `McpServer` users specifically: since `McpServer` converts
|
|
67
|
+
most protocol-shaped failures (tool not found, disabled, input/output
|
|
68
|
+
validation) to `isError: true` before this library ever sees a thrown
|
|
69
|
+
error, `classifyFailureChannel()` also recovers the real channel from
|
|
70
|
+
that disguised form by reading the `MCP error {code}: ` wrapper
|
|
71
|
+
`McpError`'s constructor always applies, which `McpServer` preserves
|
|
72
|
+
verbatim. Without this, `protocol.output`'s exclusion (the fix above)
|
|
73
|
+
would only have applied to hand-rolled low-level `Server` apps, not to
|
|
74
|
+
`McpServer` — see the README's "Agent Thrash Detection" section and ADR
|
|
75
|
+
007's addendum for the full reachability picture and its limits.
|
|
76
|
+
- **`mcp.failure.validation_paths` span attribute** — which schema
|
|
77
|
+
field(s) a Zod validation failure named, one dot-joined path per
|
|
78
|
+
failing issue (e.g. `["email", "user.profile.age"]`), best-effort
|
|
79
|
+
extracted from the same message text `classifyFailureChannel()` already
|
|
80
|
+
reads (`extractValidationPaths()`,
|
|
81
|
+
`src/fingerprint/classify/validation-paths.js`). Omitted entirely —
|
|
82
|
+
never set to an empty array — when nothing confidently parseable was
|
|
83
|
+
found. Span-only, permanently excluded from
|
|
84
|
+
`METRIC_SAFE_ATTRIBUTES`: field/path names are bounded per tool but
|
|
85
|
+
unbounded across every tool anyone registers, the same reasoning that
|
|
86
|
+
already keeps `mcp.failure.fingerprint`/`signature`/`error_class` off
|
|
87
|
+
metric labels. Full investigation and design: ADR 009
|
|
88
|
+
(`docs/adr/009-field-level-convergence.md`).
|
|
89
|
+
|
|
90
|
+
### Unchanged
|
|
91
|
+
|
|
92
|
+
- **Fingerprints (`mcp.failure.fingerprint` and every other
|
|
93
|
+
`FingerprintInputs` field) are byte-identical to v0.6.1 for the same
|
|
94
|
+
inputs.** The new `channel` dimension is deliberately kept out of
|
|
95
|
+
`computeFingerprint()`'s hash input (ADR 007) specifically so this
|
|
96
|
+
release cannot change any consumer's existing `mcp.failure.fingerprint`
|
|
97
|
+
values — a change there would silently break any alert or dashboard
|
|
98
|
+
built on fingerprint identity. Verified, not just asserted: by extracting
|
|
99
|
+
the actual, published `v0.6.1` git tag's `src/fingerprint/` tree via `git
|
|
100
|
+
archive` into an isolated directory and running its `computeFingerprint()`
|
|
101
|
+
directly, independent of this working tree, against six fixture inputs —
|
|
102
|
+
see `test/fingerprint/compose.fixtures.test.js`. If you have alerts or
|
|
103
|
+
dashboards keyed on `mcp.failure.fingerprint`, they keep working exactly
|
|
104
|
+
as they did on v0.6.1, with no changes required on your end.
|
|
105
|
+
- **Field-level discrimination in Agent Thrash Detection is not a new
|
|
106
|
+
capability — it already worked, as a side effect of fingerprinting the
|
|
107
|
+
full Zod issues JSON, and was simply incidental until now.** A
|
|
108
|
+
validation failure repeating on the *same* schema field across attempts
|
|
109
|
+
already hashed to the *same* fingerprint (accumulating correctly toward
|
|
110
|
+
`inputThreshold`), and a *different* field failing each attempt already
|
|
111
|
+
hashed to a *different* fingerprint each time (never accumulating,
|
|
112
|
+
matching a converging agent). Investigated and confirmed in ADR 009; now
|
|
113
|
+
pinned down by regression tests
|
|
114
|
+
(`test/fingerprint/field-level-convergence.test.js`) so a future Zod or
|
|
115
|
+
SDK change that silently breaks it gets caught, rather than discovered
|
|
116
|
+
as a production regression. One real gap remains open and is *not*
|
|
117
|
+
fixed by this: partial convergence (fixing one of several failing
|
|
118
|
+
fields changes the issues array's shape and breaks fingerprint
|
|
119
|
+
continuity) — tracked in `docs/known-gaps.md`, pending a design ADR 009
|
|
120
|
+
did not settle on.
|
|
121
|
+
|
|
122
|
+
### Documentation
|
|
123
|
+
|
|
124
|
+
- `docs/known-gaps.md` (new): five tracked gaps this release didn't close,
|
|
125
|
+
each written as a ready-to-paste GitHub issue — field-level convergence
|
|
126
|
+
tracking, partial convergence in field-level validation, the
|
|
127
|
+
observation-liveness contract, the pre-handler parse-failure gap, and
|
|
128
|
+
how client-side retry caps interact with
|
|
129
|
+
detection thresholds.
|
|
130
|
+
- README's "Agent Thrash Detection" section now covers channel-aware
|
|
131
|
+
thresholds, the `McpServer`-vs-low-level-`Server` reachability
|
|
132
|
+
difference (with ADR 007's full table), the pre-handler parse-failure
|
|
133
|
+
gap (deferred, not solved — closing it means revisiting ADR 001), and
|
|
134
|
+
the forwarded-error collision risk as a named known limitation.
|
|
135
|
+
- README's "Failure Fingerprinting" section now documents
|
|
136
|
+
`mcp.failure.validation_paths` and states plainly that field-level
|
|
137
|
+
discrimination is a property of the fingerprint, not a separate
|
|
138
|
+
detector — see ADR 009.
|
|
139
|
+
|
|
140
|
+
## 0.6.1
|
|
141
|
+
|
|
142
|
+
### Fixed
|
|
143
|
+
|
|
144
|
+
- `src/index.d.ts` re-exported values (`computeFingerprint`,
|
|
145
|
+
`toSpanAttributes`, `ATTRIBUTE_KEYS`, `METRIC_SAFE_ATTRIBUTES`,
|
|
146
|
+
`DEFAULT_CLASSIFIERS`, `DEFAULT_PRICING`, `defaultExtractor`,
|
|
147
|
+
`calculateCost`) from six `.js` modules that had no corresponding `.d.ts`
|
|
148
|
+
file, so any consumer with `strict`/`noImplicitAny` got a TS7016 error
|
|
149
|
+
just from importing the package. Added `src/fingerprint/compose.d.ts`,
|
|
150
|
+
`src/fingerprint/attributes.d.ts`, `src/fingerprint/classify/index.d.ts`,
|
|
151
|
+
`src/cost/pricing.d.ts`, `src/cost/extractor.d.ts`, and
|
|
152
|
+
`src/cost/calculator.d.ts`. Pre-existing since v0.4.0 (fingerprinting) and
|
|
153
|
+
v0.5.0 (cost tracking) — first caught verifying the v0.6.0 published
|
|
154
|
+
tarball.
|
|
155
|
+
|
|
3
156
|
## 0.6.0
|
|
4
157
|
|
|
5
158
|
### Added — Agent Thrash Detection
|
package/README.md
CHANGED
|
@@ -254,6 +254,8 @@ in `docs/adr/`.
|
|
|
254
254
|
| mcp.failure.category | One of 8 categories (below) | "timeout" |
|
|
255
255
|
| mcp.failure.origin | `tool_error` \| `thrown` \| `transport` | "thrown" |
|
|
256
256
|
| mcp.failure.error_class | Error class / constructor name | "TypeError" |
|
|
257
|
+
| mcp.failure.channel | `execution` \| `protocol.not_found` \| `protocol.input` \| `protocol.output` \| `protocol.other` \| `unknown` — see "Agent Thrash Detection" below | "protocol.input" |
|
|
258
|
+
| mcp.failure.validation_paths | Which schema field(s) a Zod validation failure named, one dot-joined path per failing issue — omitted entirely when nothing parseable was found (ADR 009) | `["email", "user.profile.age"]` |
|
|
257
259
|
|
|
258
260
|
Source of truth: `src/fingerprint/attributes.js`. Every category:
|
|
259
261
|
|
|
@@ -278,7 +280,26 @@ reach a fingerprint-derived value through
|
|
|
278
280
|
`origin` (24 combinations max). There is no code path today that could
|
|
279
281
|
accidentally attach `fingerprint`, `signature`, or `error_class` to a
|
|
280
282
|
counter or histogram label. See `src/fingerprint/attributes.js` and ADR
|
|
281
|
-
006's "Consequences" section.
|
|
283
|
+
006's "Consequences" section. `mcp.failure.validation_paths` above is
|
|
284
|
+
held to the exact same rule and for the exact same reason — field/path
|
|
285
|
+
names are bounded per tool but unbounded across every tool anyone
|
|
286
|
+
registers, so it is permanently excluded from `METRIC_SAFE_ATTRIBUTES`,
|
|
287
|
+
span-only, no exceptions (ADR 009).
|
|
288
|
+
|
|
289
|
+
**Field-level discrimination is a property of the fingerprint, not a
|
|
290
|
+
separate detector.** Two validation failures on the *same* schema field
|
|
291
|
+
normalize to the same message and hash to the *same* fingerprint, even
|
|
292
|
+
across different invalid values tried; a failure on a *different* field
|
|
293
|
+
normalizes differently and hashes to a *different* fingerprint. This
|
|
294
|
+
already falls out of hashing the full Zod issues JSON (which embeds each
|
|
295
|
+
failing field's path) — it is not a dedicated field-convergence detector,
|
|
296
|
+
and `mcp.failure.validation_paths` doesn't change this behavior, it just
|
|
297
|
+
makes it queryable instead of implicit in an opaque hash. See ADR 009
|
|
298
|
+
(`docs/adr/009-field-level-convergence.md`) for the full investigation,
|
|
299
|
+
including the one gap this doesn't cover: fixing one of several failing
|
|
300
|
+
fields changes the issues array's shape, which changes the fingerprint
|
|
301
|
+
even though another field is still failing underneath — tracked in
|
|
302
|
+
`docs/known-gaps.md`, not solved here.
|
|
282
303
|
|
|
283
304
|
**Extending it:** `computeFingerprint(err, ctx, opts)`
|
|
284
305
|
(`src/fingerprint/compose.js`) accepts `opts.classifiers` to prepend your
|
|
@@ -420,7 +441,7 @@ convenience default, not a maintained price list.
|
|
|
420
441
|
session-scoped limits are skipped gracefully (not enforced against a
|
|
421
442
|
fallback key) for transports with no session id, like stdio.
|
|
422
443
|
|
|
423
|
-
## Agent Thrash Detection (v0.6.0)
|
|
444
|
+
## Agent Thrash Detection (v0.6.0+)
|
|
424
445
|
|
|
425
446
|
Watches for the same tool failing with the same v0.4 failure fingerprint
|
|
426
447
|
several times in a row inside one session — the pattern an LLM agent
|
|
@@ -448,6 +469,104 @@ fingerprint inside 60 seconds gets flagged automatically. **Requires
|
|
|
448
469
|
disabled, thrash detection silently never fires, regardless of
|
|
449
470
|
`thrashDetection`'s own settings.
|
|
450
471
|
|
|
472
|
+
### Channel-aware thresholds (v0.7.0)
|
|
473
|
+
|
|
474
|
+
Not every repeated failure means the same thing. `mcp.failure.channel`
|
|
475
|
+
(see "Span attributes" above) classifies *where* a tools/call failure
|
|
476
|
+
actually came from, and thrash detection uses that classification to pick
|
|
477
|
+
a different threshold per channel instead of treating every repeat
|
|
478
|
+
identically:
|
|
479
|
+
|
|
480
|
+
| `mcp.failure.channel` | What it means | Threshold |
|
|
481
|
+
|---|---|---|
|
|
482
|
+
| `execution` | The call reached the tool and the tool itself reports a business-logic failure (`isError: true`). Retrying an unchanged upstream failure identically **is** thrash. | `threshold` (default 3) — unchanged from v0.6.0 |
|
|
483
|
+
| `protocol.input` | A JSON-RPC `InvalidParams` (-32602) whose message indicates the *agent* supplied bad arguments. An agent retrying with adjusted arguments may be genuinely converging on a correct call, not thrashing. | `inputThreshold` (default 5) — higher |
|
|
484
|
+
| `protocol.not_found` | `MethodNotFound` (-32601), or `InvalidParams` indicating an unknown/disabled tool. Retrying a tool name that doesn't exist is never convergence — there's no "getting closer" to a tool that isn't there. | `notFoundThreshold` (default 1) — an immediate flag |
|
|
485
|
+
| `protocol.output` | An `InvalidParams` (-32602) whose message indicates the **tool's own output** failed its declared output schema. This is the server author's bug — no argument the agent supplies can ever fix it. | **Excluded from thrash detection entirely.** Never counted, no matter how many times it repeats. |
|
|
486
|
+
| `protocol.other` | Any other JSON-RPC error code, or an unrecognized `-32602` message shape. | `threshold` (same as `execution`) |
|
|
487
|
+
| `unknown` | The failure doesn't confidently resemble either the execution or protocol shape. | `threshold` (same as `execution`) |
|
|
488
|
+
|
|
489
|
+
**This closes a real false positive present in every published version
|
|
490
|
+
through v0.6.1**: an output-validation bug — entirely the tool author's
|
|
491
|
+
fault — that an agent naively retried was being counted as agent thrash
|
|
492
|
+
before this release, because nothing distinguished it from an ordinary
|
|
493
|
+
repeated business-logic failure. It no longer is. Full investigation and
|
|
494
|
+
design rationale: ADR 007 (`docs/adr/007-protocol-error-channel.md`).
|
|
495
|
+
|
|
496
|
+
Configure the two new thresholds the same way as every other
|
|
497
|
+
`thrashDetection` field — see "Configuration" below.
|
|
498
|
+
|
|
499
|
+
### Reachability: high-level `McpServer` vs. low-level `Server`
|
|
500
|
+
|
|
501
|
+
**Read this before assuming `mcp.failure.channel` gives you full protocol
|
|
502
|
+
visibility on every server.** How much of the table above you actually
|
|
503
|
+
see depends on which server API you instrument, and the honest picture is
|
|
504
|
+
more limited than "protocol errors are now detected everywhere":
|
|
505
|
+
|
|
506
|
+
The high-level `McpServer` (`.tool()`/`.registerTool()` — the ergonomic,
|
|
507
|
+
documented API most real MCP servers use) already catches nearly every
|
|
508
|
+
protocol-shaped failure itself and converts it to `isError: true` *before*
|
|
509
|
+
this library ever sees a thrown error — tool not found, tool disabled,
|
|
510
|
+
input validation, output validation, or any other bug in a handler, all
|
|
511
|
+
land as `isError: true`, with the sole exception of one narrow
|
|
512
|
+
elicitation-flow error type. That means most of what `mcp.failure.channel`
|
|
513
|
+
reveals for `McpServer` users was **already visible via `isError`** before
|
|
514
|
+
this release — this feature isn't adding protocol-error detection where
|
|
515
|
+
none existed; it's adding *sub-classification* on top of detection that,
|
|
516
|
+
for the most part, already existed.
|
|
517
|
+
|
|
518
|
+
| `mcp.failure.channel` value | High-level `McpServer` | Low-level `Server` (hand-rolled dispatcher) |
|
|
519
|
+
|---|---|---|
|
|
520
|
+
| `execution` | Reachable — and the *only* value most failures produced before this release (see below) | Reachable when the handler returns `{isError:true}` itself |
|
|
521
|
+
| `protocol.not_found` | Not reachable via a raw thrown error (`McpServer` swallows it) — reachable only via the recovery mechanism below | Reachable directly |
|
|
522
|
+
| `protocol.input` | Same — recovery-only | Reachable directly |
|
|
523
|
+
| `protocol.output` | Same — recovery-only | Reachable directly |
|
|
524
|
+
| `protocol.other` | Reachable via a raw thrown error only for one narrow elicitation-flow error code; not a general catch-all for `McpServer` | Reachable for any other code, or an unrecognized `-32602` message |
|
|
525
|
+
| `unknown` | Effectively not reachable via a raw thrown error — `McpServer`'s catch swallows *any* error, not just protocol-shaped ones | Reachable for a thrown value with no error code at all |
|
|
526
|
+
|
|
527
|
+
For the low-level `Server`, all six values are directly reachable, since
|
|
528
|
+
nothing intercepts a thrown error before this library's own span/thrash
|
|
529
|
+
wrapping runs — this attribute's richest, most direct value is for
|
|
530
|
+
low-level `Server` users.
|
|
531
|
+
|
|
532
|
+
For `McpServer` users, closing the false positive above (the point of
|
|
533
|
+
this release) required a **recovery step**: `McpServer` preserves the
|
|
534
|
+
original error's message verbatim when it converts a thrown error to
|
|
535
|
+
`isError: true`, including the exact `MCP error {code}: ` wrapper its
|
|
536
|
+
error class's constructor always adds. `classifyFailureChannel()` reads
|
|
537
|
+
that wrapper back out of the disguised `isError: true` result and
|
|
538
|
+
recovers the real channel from it, falling back to `execution` only when
|
|
539
|
+
the message doesn't match that shape (i.e. it's a genuine, tool-authored
|
|
540
|
+
business message, not a disguised protocol failure). This is what makes
|
|
541
|
+
`protocol.output`'s exclusion actually work for `McpServer` users too —
|
|
542
|
+
without it, the false positive this release fixes would only have been
|
|
543
|
+
fixed for hand-rolled low-level `Server` apps.
|
|
544
|
+
|
|
545
|
+
**This recovery step is inherently fragile**, coupled to matching the
|
|
546
|
+
exact prose the installed SDK version happens to use — both the `MCP
|
|
547
|
+
error {code}: ` wrapper and the `-32602` sub-case message markers
|
|
548
|
+
(`"Input validation error:"`, `"Output validation error:"`, and the
|
|
549
|
+
looser `"not found"`/`"disabled"` substring matches). If the SDK changes
|
|
550
|
+
either format, the classifier degrades safely to `execution` (never a
|
|
551
|
+
thrown error, never a wrong specific answer) rather than breaking — but a
|
|
552
|
+
future SDK version could silently reopen part of the gap this release
|
|
553
|
+
closes. See ADR 007's addendum for the full verification, including how
|
|
554
|
+
this was confirmed against a real `McpServer` and a real Zod output
|
|
555
|
+
schema before being fixed.
|
|
556
|
+
|
|
557
|
+
**Known limitation — a forwarded/proxied error can collide with this
|
|
558
|
+
recovery.** A tool that forwards another MCP call's error text verbatim
|
|
559
|
+
(an orchestrator or proxy tool surfacing a downstream failure) could
|
|
560
|
+
plausibly produce a message starting with the same `MCP error {code}: `
|
|
561
|
+
wrapper, purely by coincidence of forwarding real McpError text. For most
|
|
562
|
+
codes this is cosmetic (`protocol.other` shares `execution`'s threshold).
|
|
563
|
+
The sharp case: a forwarded output-validation-shaped message would be
|
|
564
|
+
**excluded from thrash detection entirely**, even though it may be a
|
|
565
|
+
genuine, repeatable failure from the forwarding tool's own perspective —
|
|
566
|
+
a false negative, not a false positive. Assessed as an acceptable,
|
|
567
|
+
narrow risk for this release (see ADR 007's addendum for the full
|
|
568
|
+
reasoning); revisit if this pattern turns out to be common in practice.
|
|
569
|
+
|
|
451
570
|
### Metrics
|
|
452
571
|
|
|
453
572
|
Same API-only pattern as every other metric in this README — nothing
|
|
@@ -504,6 +623,22 @@ silently, never throws. Source of truth: `src/thrash/config.js`.
|
|
|
504
623
|
| `entryTtlMs` | `OTEL_MCP_THRASH_ENTRY_TTL_MS` | number | `900000` | How long an idle tracked key survives before expiry |
|
|
505
624
|
| `reEmitAfter` | `OTEL_MCP_THRASH_RE_EMIT_AFTER` | number | `3` | Re-emit every N further failures past `threshold` (e.g. 3, 6, 9, ...) instead of once |
|
|
506
625
|
| `assumeSingleSession` | `OTEL_MCP_THRASH_ASSUME_SINGLE_SESSION` | boolean | `false` | Force-permits the fallback session id even when the transport can't be determined — see below |
|
|
626
|
+
| `inputThreshold` | `OTEL_MCP_THRASH_INPUT_THRESHOLD` | number | `5` | Per-origin threshold (ADR 007) for `mcp.failure.channel: protocol.input` — higher than `threshold`, since an agent retrying with adjusted arguments may be converging |
|
|
627
|
+
| `notFoundThreshold` | `OTEL_MCP_THRASH_NOT_FOUND_THRESHOLD` | number | `1` | Per-origin threshold (ADR 007) for `mcp.failure.channel: protocol.not_found` — lower than `threshold`; retrying a nonexistent tool is never convergence |
|
|
628
|
+
|
|
629
|
+
**A note on defaults and client-side retry caps.** Every threshold above
|
|
630
|
+
assumes an effectively uncapped agent — one that keeps retrying an
|
|
631
|
+
identically-failing call at least as many times as the threshold. Some
|
|
632
|
+
agent frameworks impose their own client-side cap on same-arguments
|
|
633
|
+
retries (e.g. giving up and reporting failure after 2 identical
|
|
634
|
+
attempts). If an agent's own cap is lower than the relevant threshold
|
|
635
|
+
(the default `threshold` is 3), that agent's thrashing never crosses the
|
|
636
|
+
threshold and `mcp.tool.loop.detected` never fires for it — arguably
|
|
637
|
+
correct in isolation (2 identical failures is a weaker signal than 3),
|
|
638
|
+
but worth knowing before assuming detection is silently catching
|
|
639
|
+
everything. If you know your agent framework caps retries at N, consider
|
|
640
|
+
setting the relevant threshold to N. Tracked as an open question, not
|
|
641
|
+
solved here: `docs/known-gaps.md`.
|
|
507
642
|
|
|
508
643
|
### Session id resolution — read this before setting `assumeSingleSession`
|
|
509
644
|
|
|
@@ -592,6 +727,32 @@ has this process wasted since it started" (or since the last call to an
|
|
|
592
727
|
internal `reset()`), not "what's currently active." Don't read
|
|
593
728
|
`topOffenders` as an audit log; read the cumulative totals for that.
|
|
594
729
|
|
|
730
|
+
### Known limitations
|
|
731
|
+
|
|
732
|
+
**A malformed `tools/call` request produces zero telemetry — no span, no
|
|
733
|
+
fingerprint, nothing.** If a request fails `CallToolRequestSchema`
|
|
734
|
+
validation itself (e.g. a missing or wrongly-typed `name`/`arguments`
|
|
735
|
+
field), the SDK's own request-parsing step throws *before*
|
|
736
|
+
`instrumentMcpServer()`'s wrapped handler is ever invoked — there's no
|
|
737
|
+
span to attach a status to and no error object reaches
|
|
738
|
+
`computeFingerprint()`. This is invisible by construction: closing it
|
|
739
|
+
means wrapping a layer above where this library currently patches
|
|
740
|
+
(`setRequestHandler`'s `handler` argument), which is exactly the larger,
|
|
741
|
+
less stable dependency surface ADR 001 chose not to depend on. **Deferred,
|
|
742
|
+
not solved** — tracked in `docs/known-gaps.md`, and would need its own
|
|
743
|
+
design pass (effectively revisiting ADR 001) rather than a patch-level
|
|
744
|
+
fix.
|
|
745
|
+
|
|
746
|
+
See `docs/known-gaps.md` for this gap in full, plus four more not fully
|
|
747
|
+
covered by this release: field-level convergence tracking for
|
|
748
|
+
`protocol.input` (distinguishing "ambiguous tool schema" from "agent is
|
|
749
|
+
converging" — mostly already works as a side effect of fingerprinting,
|
|
750
|
+
see "Failure Fingerprinting" above and ADR 009, but partial convergence
|
|
751
|
+
within that is its own separate, still-open entry), an
|
|
752
|
+
observation-liveness contract for `getThrashSummary()` (so "nothing
|
|
753
|
+
failed" and "nothing is being observed at all" stop looking identical),
|
|
754
|
+
and how client-side agent retry caps interact with the thresholds above.
|
|
755
|
+
|
|
595
756
|
### Benchmarks
|
|
596
757
|
|
|
597
758
|
Two kinds, both under `bench/` and `test/thrash/`:
|
|
@@ -712,7 +873,7 @@ pragmatic choice rather than a spec-pure one.
|
|
|
712
873
|
- Supports both low-level `Server` and high-level `McpServer` APIs
|
|
713
874
|
- @modelcontextprotocol/sdk ^1.0.0
|
|
714
875
|
- @opentelemetry/api ^1.9.0
|
|
715
|
-
-
|
|
876
|
+
- 498 tests (`npm test`) — see `test/`
|
|
716
877
|
- `npm run typecheck` (`tsc --noEmit`) type-checks the public `.d.ts`
|
|
717
878
|
surface (`src/index.d.ts` and friends) — see CONTRIBUTING.md
|
|
718
879
|
|
|
@@ -722,6 +883,17 @@ pragmatic choice rather than a spec-pure one.
|
|
|
722
883
|
and ADR 006.
|
|
723
884
|
- v0.5: Cost & Token Attribution ✓ — see "Cost & Token Attribution" above.
|
|
724
885
|
- v0.6: Agent Thrash Detection ✓ — see "Agent Thrash Detection" above.
|
|
886
|
+
- v0.7: Channel-aware thrash detection ✓ — fixes a real false positive
|
|
887
|
+
(output-validation failures counted as thrash) present since v0.6.0; adds
|
|
888
|
+
`mcp.failure.channel` and per-channel thresholds. See "Agent Thrash
|
|
889
|
+
Detection" above and ADR 007. Also confirms (ADR 009) that field-level
|
|
890
|
+
discrimination for `protocol.input` failures already worked as a side
|
|
891
|
+
effect of fingerprinting — now regression-tested and surfaced via
|
|
892
|
+
`mcp.failure.validation_paths` (see "Failure Fingerprinting" above). Five
|
|
893
|
+
gaps this release didn't fully close — field-level convergence tracking,
|
|
894
|
+
partial convergence within it, an observation-liveness contract, the
|
|
895
|
+
pre-handler parse-failure gap, and client-side retry caps — are tracked
|
|
896
|
+
in `docs/known-gaps.md`, not silently dropped.
|
|
725
897
|
- Future: failure clustering + regression detection; recovery hints;
|
|
726
898
|
root-cause chaining across parent spans; alignment with the OTel GenAI
|
|
727
899
|
SIG's MCP semantic conventions when published
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "opentel-mcp",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.0",
|
|
4
4
|
"description": "One-line OpenTelemetry instrumentation for Model Context Protocol (MCP) servers",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
|
@@ -25,9 +25,11 @@
|
|
|
25
25
|
"test:watch": "vitest",
|
|
26
26
|
"test:coverage": "vitest run --coverage",
|
|
27
27
|
"typecheck": "tsc --noEmit",
|
|
28
|
+
"verify:tarball": "node scripts/verify-tarball.js",
|
|
28
29
|
"bench": "vitest bench --run",
|
|
29
30
|
"prepack": "node scripts/strip-workspaces.js",
|
|
30
|
-
"postpack": "node scripts/restore-workspaces.js"
|
|
31
|
+
"postpack": "node scripts/restore-workspaces.js",
|
|
32
|
+
"prepublishOnly": "npm run verify:tarball"
|
|
31
33
|
},
|
|
32
34
|
"keywords": [
|
|
33
35
|
"mcp",
|
|
@@ -64,6 +66,7 @@
|
|
|
64
66
|
"@opentelemetry/sdk-trace-base": "^2.9.0",
|
|
65
67
|
"@vitest/coverage-v8": "^2.1.9",
|
|
66
68
|
"typescript": "^7.0.2",
|
|
67
|
-
"vitest": "^2.1.8"
|
|
69
|
+
"vitest": "^2.1.8",
|
|
70
|
+
"zod": "^4.4.3"
|
|
68
71
|
}
|
|
69
72
|
}
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
import type { PricingTable } from './types.d.ts';
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Calculates the USD cost of a request from its input/output token counts, a
|
|
5
|
+
* model name, and a pricing table. Never throws: an unrecognized model or an
|
|
6
|
+
* invalid token count resolves to `null` rather than an exception.
|
|
7
|
+
*/
|
|
8
|
+
export function calculateCost(
|
|
9
|
+
inputTokens: number,
|
|
10
|
+
outputTokens: number,
|
|
11
|
+
model: string,
|
|
12
|
+
pricingTable: PricingTable,
|
|
13
|
+
): number | null;
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
import type { UsageExtractor } from './types.d.ts';
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Default {@link UsageExtractor}. Recognizes Anthropic/OpenAI/Bedrock usage
|
|
5
|
+
* field conventions, JSON-in-text content, and the MCP `_meta.usage`
|
|
6
|
+
* extension point. Never throws — returns `null` for any unrecognized shape.
|
|
7
|
+
*/
|
|
8
|
+
export const defaultExtractor: UsageExtractor;
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
import type { PricingTable } from './types.d.ts';
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Static per-model token pricing used to estimate MCP tool-call cost, in USD
|
|
5
|
+
* per 1M tokens. A best-effort snapshot — callers who need accurate cost
|
|
6
|
+
* attribution should supply their own {@link PricingTable}.
|
|
7
|
+
*/
|
|
8
|
+
export const DEFAULT_PRICING: PricingTable;
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
import type { Attributes } from '@opentelemetry/api';
|
|
2
|
+
import type { FingerprintResult } from './types.d.ts';
|
|
3
|
+
|
|
4
|
+
/** The `mcp.failure.*` OpenTelemetry span attribute keys. */
|
|
5
|
+
export const ATTRIBUTE_KEYS: Readonly<
|
|
6
|
+
Record<'FINGERPRINT' | 'SIGNATURE' | 'CATEGORY' | 'ORIGIN' | 'ERROR_CLASS' | 'CHANNEL' | 'VALIDATION_PATHS', string>
|
|
7
|
+
>;
|
|
8
|
+
|
|
9
|
+
/**
|
|
10
|
+
* Attribute keys safe to attach to metric labels — `category` and `origin`
|
|
11
|
+
* only; everything else in {@link ATTRIBUTE_KEYS} is unbounded or
|
|
12
|
+
* medium-cardinality and must stay span-only.
|
|
13
|
+
*/
|
|
14
|
+
export const METRIC_SAFE_ATTRIBUTES: readonly string[];
|
|
15
|
+
|
|
16
|
+
/**
|
|
17
|
+
* Builds the span attributes for a fingerprinted failure. Never throws —
|
|
18
|
+
* returns `{}` if `result` is malformed in any way.
|
|
19
|
+
*/
|
|
20
|
+
export function toSpanAttributes(result: FingerprintResult): Attributes;
|
|
@@ -15,13 +15,35 @@
|
|
|
15
15
|
/** @typedef {import('@opentelemetry/api').Attributes} Attributes */
|
|
16
16
|
/** @typedef {import('./types.d.ts').FingerprintResult} FingerprintResult */
|
|
17
17
|
|
|
18
|
-
/** @type {Readonly<Record<'FINGERPRINT' | 'SIGNATURE' | 'CATEGORY' | 'ORIGIN' | 'ERROR_CLASS', string>>} */
|
|
18
|
+
/** @type {Readonly<Record<'FINGERPRINT' | 'SIGNATURE' | 'CATEGORY' | 'ORIGIN' | 'ERROR_CLASS' | 'CHANNEL' | 'VALIDATION_PATHS', string>>} */
|
|
19
19
|
export const ATTRIBUTE_KEYS = Object.freeze({
|
|
20
20
|
FINGERPRINT: 'mcp.failure.fingerprint',
|
|
21
21
|
SIGNATURE: 'mcp.failure.signature',
|
|
22
22
|
CATEGORY: 'mcp.failure.category',
|
|
23
23
|
ORIGIN: 'mcp.failure.origin',
|
|
24
24
|
ERROR_CLASS: 'mcp.failure.error_class',
|
|
25
|
+
/**
|
|
26
|
+
* ADR 007's channel dimension (`classifyFailureChannel()`,
|
|
27
|
+
* `fingerprint/classify/channel.js`): 'execution' | 'protocol.not_found'
|
|
28
|
+
* | 'protocol.input' | 'protocol.output' | 'protocol.other' | 'unknown'.
|
|
29
|
+
* Deliberately a DIFFERENT attribute from ORIGIN above — ORIGIN already
|
|
30
|
+
* carries FingerprintInputs' `origin` ('tool_error' | 'thrown' |
|
|
31
|
+
* 'transport'), hashed into the fingerprint since v0.4.0. CHANNEL is
|
|
32
|
+
* additive, set independently of computeFingerprint(), and never part
|
|
33
|
+
* of the hash — see ADR 007's "Where the new dimension lives".
|
|
34
|
+
*/
|
|
35
|
+
CHANNEL: 'mcp.failure.channel',
|
|
36
|
+
/**
|
|
37
|
+
* ADR 009's diagnostic attribute (`extractValidationPaths()`,
|
|
38
|
+
* `fingerprint/classify/validation-paths.js`): one dot-joined path per
|
|
39
|
+
* failing Zod issue found in the message (e.g. `["email"]` or
|
|
40
|
+
* `["user.profile.age", "status"]`), best-effort and omitted entirely
|
|
41
|
+
* when nothing parseable was found — never an empty array. Like
|
|
42
|
+
* CHANNEL, additive and never part of the fingerprint hash: the path
|
|
43
|
+
* text is already implicit in the hashed normalized message (see ADR
|
|
44
|
+
* 009), so hashing it again would be redundant, not more correct.
|
|
45
|
+
*/
|
|
46
|
+
VALIDATION_PATHS: 'mcp.failure.validation_paths',
|
|
25
47
|
});
|
|
26
48
|
|
|
27
49
|
/**
|
|
@@ -29,6 +51,21 @@ export const ATTRIBUTE_KEYS = Object.freeze({
|
|
|
29
51
|
* {@link ATTRIBUTE_KEYS} is unbounded or medium-cardinality and must stay
|
|
30
52
|
* span-only.
|
|
31
53
|
*
|
|
54
|
+
* CHANNEL is deliberately NOT included here yet: ADR 007 doesn't decide
|
|
55
|
+
* metric-label safety for it one way or the other (it's silent on this
|
|
56
|
+
* file entirely), and per the Phase 2 instructions this omission should
|
|
57
|
+
* default to span-only rather than inventing a decision the ADR didn't
|
|
58
|
+
* make. It's a small closed set (6 values) and would likely qualify on
|
|
59
|
+
* cardinality grounds alone — revisit explicitly, in the ADR, before
|
|
60
|
+
* adding it here.
|
|
61
|
+
*
|
|
62
|
+
* VALIDATION_PATHS is explicitly, permanently excluded — this one IS
|
|
63
|
+
* decided, not merely deferred (ADR 009): field/path names are bounded
|
|
64
|
+
* per tool but unbounded across every tool anyone ever registers, and
|
|
65
|
+
* unbounded again across every deployment a shared metrics backend might
|
|
66
|
+
* aggregate — the exact same reasoning that already keeps FINGERPRINT/
|
|
67
|
+
* SIGNATURE/ERROR_CLASS off this list.
|
|
68
|
+
*
|
|
32
69
|
* @type {readonly string[]}
|
|
33
70
|
*/
|
|
34
71
|
export const METRIC_SAFE_ATTRIBUTES = Object.freeze([ATTRIBUTE_KEYS.CATEGORY, ATTRIBUTE_KEYS.ORIGIN]);
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
import type { FailureChannel } from '../types.d.ts';
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Determines which channel an MCP tools/call failure arrived on. Never
|
|
5
|
+
* throws — see `src/fingerprint/classify/channel.js`'s docblock for the
|
|
6
|
+
* full behavior, including recovery of a protocol failure McpServer has
|
|
7
|
+
* disguised as `isError: true`.
|
|
8
|
+
*/
|
|
9
|
+
export function classifyFailureChannel(failure: unknown): FailureChannel;
|