@agent-watch-ai/otel 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +201 -2
- package/dist/call.d.ts +51 -0
- package/dist/call.js +289 -0
- package/dist/fingerprint.d.ts +79 -0
- package/dist/fingerprint.js +188 -0
- package/dist/gate.d.ts +109 -0
- package/dist/gate.js +498 -0
- package/dist/index.d.ts +5 -0
- package/dist/index.js +10 -0
- package/dist/post.d.ts +54 -0
- package/dist/post.js +149 -0
- package/dist/processor.d.ts +103 -0
- package/dist/processor.js +220 -0
- package/dist/refusal.d.ts +26 -0
- package/dist/refusal.js +39 -0
- package/package.json +40 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 AgentWatch contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,202 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @agent-watch-ai/otel
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**Your production LLM spend in Agent Watch, hashed in your own process, and over-budget calls refused before they leave it.**
|
|
4
|
+
|
|
5
|
+
One line beside your OpenTelemetry setup. Prompt text, tool definitions and arguments, and model
|
|
6
|
+
responses never leave the process — only digests, token counts and ids do. Your own processors and
|
|
7
|
+
exporters keep receiving everything, unchanged. The Node twin of the Python `agent-watch-otel`: the
|
|
8
|
+
same span produces the same payload in both.
|
|
9
|
+
|
|
10
|
+
```sh
|
|
11
|
+
npm install @agent-watch-ai/otel
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
```ts
|
|
15
|
+
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
|
|
16
|
+
import { AgentWatchSpanProcessor } from '@agent-watch-ai/otel';
|
|
17
|
+
|
|
18
|
+
const provider = new NodeTracerProvider({
|
|
19
|
+
spanProcessors: [yourProcessor, new AgentWatchSpanProcessor()] // beside yours
|
|
20
|
+
});
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
The token is an Agent Watch SDK token scoped `ingest:production`. `token` and `endpoint` come from
|
|
24
|
+
the options or from `AGENT_WATCH_TOKEN` and `AGENT_WATCH_ENDPOINT`. `OTEL_*` is never read: it
|
|
25
|
+
belongs to your exporter.
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## What it sends
|
|
30
|
+
|
|
31
|
+
Finished spans whose `openinference.span.kind` is `LLM`, and nothing else. Per call:
|
|
32
|
+
|
|
33
|
+
| Sent | Not sent |
|
|
34
|
+
|---|---|
|
|
35
|
+
| sha256 of the system/developer prefix, and its 8-token window digests | the prompt |
|
|
36
|
+
| sha256 of the tool definitions | the tool definitions, and every tool argument |
|
|
37
|
+
| model, provider, token counts (input, output, cache read/write, reasoning) | the response |
|
|
38
|
+
| span id, trace id, span name, end time | anything the span carries that is not in this table |
|
|
39
|
+
| `service.name` and `service.version` from the OTel resource | |
|
|
40
|
+
|
|
41
|
+
Before a model call, the gate sends a decision request (see "Refusing over-budget calls"). It
|
|
42
|
+
carries the span name, `service.name`, the model and the deployment environment, plus counts of
|
|
43
|
+
local refusals and fail-opens. Never prompt text, tool definitions or responses.
|
|
44
|
+
|
|
45
|
+
Digests are computed in `onEnd`, inside your `span.end()`, before anything is queued. The queue
|
|
46
|
+
holds hashes.
|
|
47
|
+
|
|
48
|
+
## What it does to your application
|
|
49
|
+
|
|
50
|
+
- **`onEnd` never touches the network.** It hashes and enqueues. Requests start from an unref'd
|
|
51
|
+
timer, one batch of at most 100 calls at a time (under 2 MB of JSON per request).
|
|
52
|
+
- **Its own request is invisible to your instrumentation.** The POST is sent with OpenTelemetry
|
|
53
|
+
tracing suppressed, so HTTP instrumentation neither traces it nor injects trace headers.
|
|
54
|
+
- **`onEnd` never throws, and nothing it starts rejects unhandled.** A bug here costs you a cost
|
|
55
|
+
record, never a request, and never an `unhandledRejection` crash.
|
|
56
|
+
- **The queue is bounded.** When Agent Watch is unreachable or slow, calls are dropped and counted
|
|
57
|
+
rather than buffered without limit inside your process. A queued call is at most about 18 KB
|
|
58
|
+
(512 shingle digests), so the default `maxQueue` of 512 holds at most about 9 MB of payload.
|
|
59
|
+
- **It never holds your process open.** Its timer and its request's socket are unref'd: a process
|
|
60
|
+
whose own work is done exits without waiting on a slow ingest. `shutdown()` is bounded by
|
|
61
|
+
`shutdownMillis` however many times it is called; a request still in flight when the budget
|
|
62
|
+
runs out is aborted. While you await `forceFlush()` or `shutdown()` the request is held open
|
|
63
|
+
(each bounded by its timeout), so a short job that ends with `await provider.forceFlush()`
|
|
64
|
+
delivers its last batch before it exits.
|
|
65
|
+
- **The token never crosses a network in clear.** The endpoint must be `https`; plain `http` is
|
|
66
|
+
accepted only on loopback (`localhost`, `127.0.0.0/8`, `::1`) and anything else throws at
|
|
67
|
+
construction.
|
|
68
|
+
|
|
69
|
+
```ts
|
|
70
|
+
processor.stats();
|
|
71
|
+
// { sent: 412, droppedQueueFull: 0, droppedSendFailed: 3, droppedNoModel: 0, droppedError: 0 }
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
A call with no `llm.model_name` and no `llm.model` is counted under `droppedNoModel` rather than
|
|
75
|
+
sent under a guessed model. Failed batches are dropped, not retried. Warnings go to the
|
|
76
|
+
OpenTelemetry `diag` logger, so they appear only if you configured one.
|
|
77
|
+
|
|
78
|
+
## Refusing over-budget calls
|
|
79
|
+
|
|
80
|
+
When a feature's production budget is spent, the agent's next model call is refused before it
|
|
81
|
+
leaves your process. Registering the processor wraps `globalThis.fetch`. The first request made
|
|
82
|
+
inside an LLM span (`openinference.span.kind = LLM`, the spans this processor reports) asks Agent
|
|
83
|
+
Watch about the agent the span names. Tool, agent and embedding spans, and spans that only set
|
|
84
|
+
`gen_ai.operation.name`, are never gated. A refused call fails with:
|
|
85
|
+
|
|
86
|
+
```
|
|
87
|
+
AgentWatchRefusal: An Agent Watch budget refused this call.
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
It names no cap, amount, feature or policy, and `retryable` is `false`: a retry gets the same
|
|
91
|
+
answer from the held decision, with no request to Agent Watch or to the provider. Provider SDKs
|
|
92
|
+
that wrap transport errors (OpenAI, Anthropic) raise their own connection error with the refusal
|
|
93
|
+
as its `cause`, and may retry with their own backoff first. Test for it with `isAgentWatchRefusal`:
|
|
94
|
+
|
|
95
|
+
```ts
|
|
96
|
+
import { isAgentWatchRefusal } from '@agent-watch-ai/otel';
|
|
97
|
+
|
|
98
|
+
try {
|
|
99
|
+
await client.chat.completions.create(request);
|
|
100
|
+
} catch (err) {
|
|
101
|
+
if (isAgentWatchRefusal(err)) return budgetSpentAnswer();
|
|
102
|
+
throw err;
|
|
103
|
+
}
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
**The cost.** Answers are held per agent and deployment environment for as long as Agent Watch
|
|
107
|
+
says (at most 60 s); a held answer adds no request. A fresh one waits at most 300 ms, with one
|
|
108
|
+
request per agent in flight at a time: concurrent calls for the same agent wait on that one
|
|
109
|
+
request. While it waits, the decision request keeps the process alive, so a script whose only
|
|
110
|
+
work is that model call still makes it. A request still unanswered at 300 ms is not cancelled:
|
|
111
|
+
the call goes ahead as a fail-open, the request runs on in the background (at most 5 s, never
|
|
112
|
+
holding the process open), and its answer is held for the agent's next calls. So a gateway far
|
|
113
|
+
away, slower than 300 ms on a cold connection, still gets to refuse. An abort of the call's
|
|
114
|
+
`signal` (in `init`, or on a `Request` passed in) while it waits rejects at once with the abort
|
|
115
|
+
reason.
|
|
116
|
+
|
|
117
|
+
**Register the processor before you create provider clients.** The gate sees only requests made
|
|
118
|
+
through `globalThis.fetch` as it is after the processor is constructed. A client that keeps its
|
|
119
|
+
own reference to `fetch`, taken when the client was created, bypasses the gate if it was created
|
|
120
|
+
first: construct `AgentWatchSpanProcessor` in your tracing setup, before any module creates an
|
|
121
|
+
OpenAI or Anthropic client. SDK versions that do not use the global `fetch` at all (OpenAI's v4
|
|
122
|
+
SDK on Node defaults to `node-fetch`) are never gated that way: pass `fetch: globalThis.fetch` in
|
|
123
|
+
the client options, or await `guard(agentName)` before each call. Such calls are not counted as
|
|
124
|
+
fail-opens either, since the gate never saw them.
|
|
125
|
+
|
|
126
|
+
**Which agent.** The agent is the span name. OpenInference's OpenAI instrumentation names every
|
|
127
|
+
span `ChatCompletion`, so all such calls are one agent unless you name the span yourself (wrap the
|
|
128
|
+
call in a span named for the agent, or rename it).
|
|
129
|
+
|
|
130
|
+
**Fail-open, always.** If Agent Watch is slow, unreachable, answers anything but 2xx, or answers
|
|
131
|
+
something it cannot read, the call goes ahead, and the fail-open is reported on the next decision
|
|
132
|
+
request for that agent. What is owed travels as one batch with a `report_id`, resent unchanged
|
|
133
|
+
until a request carrying it is answered, so Agent Watch records it once. After a fail-open the agent is allowed without asking for 30 s, or for
|
|
134
|
+
5 minutes after a 401, 403 or 404 (the route is missing or the token lacks the scope); each call
|
|
135
|
+
let through meanwhile is counted as a fail-open too. There is no fail-closed mode and no switch
|
|
136
|
+
for one. Nothing inside the
|
|
137
|
+
package reaches your application as anything but the refusal.
|
|
138
|
+
|
|
139
|
+
**What passes ungated.** Requests outside an LLM span; requests to `metadata.google.internal`,
|
|
140
|
+
`oauth2.googleapis.com` and `sts.googleapis.com`, so a workload-identity token fetch is never
|
|
141
|
+
refused; and the package's own requests. A refused call is still reported, with `refused: true`,
|
|
142
|
+
and counts no spend.
|
|
143
|
+
|
|
144
|
+
**Known limit: sampled-out spans are never gated.** A span your sampler drops carries no name or
|
|
145
|
+
attributes, so a call inside it goes ahead unjudged and is not counted anywhere. Keep LLM spans
|
|
146
|
+
sampled (a parent-based sampler that always records LLM spans, or sample everything) wherever you
|
|
147
|
+
rely on the gate.
|
|
148
|
+
|
|
149
|
+
**Other transports.** `node:http`, gRPC and other HTTP clients are not intercepted. Await
|
|
150
|
+
`guard(agentName)` before such a call; it throws `AgentWatchRefusal` when refused, and resolves at
|
|
151
|
+
once when no processor is registered. It states the agent and the deployment environment from
|
|
152
|
+
`OTEL_RESOURCE_ATTRIBUTES`; with neither environment key set there, it is judged as production.
|
|
153
|
+
|
|
154
|
+
**Dev and CI.** Set the resource attribute `deployment.environment.name` (or the older
|
|
155
|
+
`deployment.environment`, read when the first is unset) to anything but `production` or `prod`
|
|
156
|
+
and calls are allowed without judging; unset, they are judged as production.
|
|
157
|
+
|
|
158
|
+
## Options
|
|
159
|
+
|
|
160
|
+
| Option | Default | What it is |
|
|
161
|
+
|---|---|---|
|
|
162
|
+
| `token` | `$AGENT_WATCH_TOKEN` | SDK token scoped `ingest:production` |
|
|
163
|
+
| `endpoint` | `$AGENT_WATCH_ENDPOINT` | Ingest base URL; the route `v1/runtime/calls` is joined onto it |
|
|
164
|
+
| `instanceId` | the resource's `service.name` | This deployment's name, as you know it |
|
|
165
|
+
| `maxQueue` | 512 | Calls held while the sender is behind; a positive integer |
|
|
166
|
+
| `flushMillis` | 5000 | How long a partial batch waits, in (0, 2^31-1] |
|
|
167
|
+
| `timeoutMillis` | 10000 | Network timeout for one POST |
|
|
168
|
+
| `shutdownMillis` | 3000 | The whole budget `shutdown()` may spend, across every call |
|
|
169
|
+
|
|
170
|
+
A missing token or endpoint throws at construction — on your startup path, where a
|
|
171
|
+
misconfiguration is visible — never later from `onEnd`. Any other invalid number falls back to
|
|
172
|
+
its default rather than throwing.
|
|
173
|
+
|
|
174
|
+
## Requirements
|
|
175
|
+
|
|
176
|
+
Node 20.19+ or 22.12+, with `@opentelemetry/api` 1.9+ and `@opentelemetry/sdk-trace-base` 1.x or
|
|
177
|
+
2.x (types only) as peer dependencies. The package is ESM only; those Node versions load it from
|
|
178
|
+
CommonJS too, through `require()` of ES modules, so there is no second CommonJS build to keep in
|
|
179
|
+
step. No other dependency, by policy: this code runs inside your application.
|
|
180
|
+
|
|
181
|
+
## Development
|
|
182
|
+
|
|
183
|
+
```sh
|
|
184
|
+
npm ci # at the repository root: TypeScript, vitest and ESLint come from there
|
|
185
|
+
cd node/agent-watch-otel && npm ci
|
|
186
|
+
npm run typecheck && npm run lint && npm test
|
|
187
|
+
npx vitest run tests/fingerprint-vectors.test.ts tests/call-vectors.test.ts tests/gate-conformance.test.ts # parity with Python
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
`conformance/runtime/gate-cases.json` is the gate's behaviour (gateway answers in, outcomes out),
|
|
191
|
+
replayed by `tests/gate-conformance.test.ts` and by the Python package.
|
|
192
|
+
|
|
193
|
+
Both parity files live in `conformance/runtime/` at the repository root and are read in place by
|
|
194
|
+
this package and the Python one. `fingerprint-vectors.json` is a byte copy of core's golden
|
|
195
|
+
fingerprint vectors (scheme 1); `call-vectors.json` is generated from the Python `build_call` by
|
|
196
|
+
`generate_call_vectors.py`. Neither is edited to make a test pass.
|
|
197
|
+
|
|
198
|
+
`scripts/system-check.mjs` posts one real span to a running ingestion gateway and prints its verdict.
|
|
199
|
+
|
|
200
|
+
## Licence
|
|
201
|
+
|
|
202
|
+
MIT.
|
package/dist/call.d.ts
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
import type { Attributes, AttributeValue } from '@opentelemetry/api';
|
|
2
|
+
import type { ReadableSpan } from '@opentelemetry/sdk-trace-base';
|
|
3
|
+
/**
|
|
4
|
+
* One finished LLM span in, one `runtime.call` out: digests, token counts, ids and timestamps.
|
|
5
|
+
*
|
|
6
|
+
* The confidentiality boundary of the package. Prompt text, tool definitions and responses are
|
|
7
|
+
* read here, hashed here and left here; nothing returned holds a message body. A port of
|
|
8
|
+
* `python/agent-watch-otel/agent_watch_otel/call.py`, held to it by `conformance/runtime/call-vectors.json`.
|
|
9
|
+
*/
|
|
10
|
+
/** The platform enum member for a customer-run span processor (AWT-70). */
|
|
11
|
+
export declare const PLATFORM = "otel_processor";
|
|
12
|
+
/** What `buildCall` reads from a span: a subset of the SDK's `ReadableSpan`. */
|
|
13
|
+
export type CallSpan = Pick<ReadableSpan, 'name' | 'attributes' | 'endTime' | 'spanContext'> & {
|
|
14
|
+
readonly resource?: {
|
|
15
|
+
readonly attributes: Attributes;
|
|
16
|
+
};
|
|
17
|
+
};
|
|
18
|
+
/** The `runtime.call` payload. Strict on the gateway: no key outside this shape is ever sent. */
|
|
19
|
+
export type RuntimeCall = Readonly<Record<string, string | number | boolean | readonly string[]>>;
|
|
20
|
+
/** Set to `true` by the runtime gate on a span whose call it refused. */
|
|
21
|
+
export declare const REFUSED_ATTRIBUTE = "agent_watch.refused";
|
|
22
|
+
/**
|
|
23
|
+
* Whether the processor reports this span.
|
|
24
|
+
* @param span Any finished span.
|
|
25
|
+
* @returns `true` when its OpenInference kind is `LLM`.
|
|
26
|
+
*/
|
|
27
|
+
export declare function isLlmSpan(span: Pick<ReadableSpan, 'attributes'>): boolean;
|
|
28
|
+
/**
|
|
29
|
+
* Turn a finished LLM span into the `runtime.call` the ingest route accepts.
|
|
30
|
+
* @param span A finished span whose OpenInference kind is `LLM`.
|
|
31
|
+
* @param instanceId The tenant's name for this deployment; empty falls back to `service.name`.
|
|
32
|
+
* @returns The payload, or `null` when the span names no model: unpriceable spend is counted as
|
|
33
|
+
* dropped rather than sent under a guess. A refused call has no spend, so it is sent as model `unknown`.
|
|
34
|
+
* @throws Only on a bug; the processor counts it as `dropped_error`.
|
|
35
|
+
*/
|
|
36
|
+
export declare function buildCall(span: CallSpan, instanceId: string): RuntimeCall | null;
|
|
37
|
+
/**
|
|
38
|
+
* The resource's `service.name`, unless it is the SDK's `unknown_service[:<process>]` placeholder,
|
|
39
|
+
* which would put a fictional service on the connection page.
|
|
40
|
+
* @param value The resource attribute.
|
|
41
|
+
* @returns The storable name, or `null`.
|
|
42
|
+
*/
|
|
43
|
+
export declare function reportedService(value: AttributeValue | undefined): string | null;
|
|
44
|
+
/**
|
|
45
|
+
* Trimmed, length-capped, and free of anything the gateway would refuse. One bad code point in
|
|
46
|
+
* one span name is a 400 for the whole batch, which would cost every other call in it.
|
|
47
|
+
* @param value Any attribute value; anything but a string reads as absent.
|
|
48
|
+
* @param limit The contract's maximum length for this field, in UTF-16 units.
|
|
49
|
+
* @returns The storable text, or `null` when nothing is left of it.
|
|
50
|
+
*/
|
|
51
|
+
export declare function storable(value: unknown, limit: number): string | null;
|
package/dist/call.js
ADDED
|
@@ -0,0 +1,289 @@
|
|
|
1
|
+
import { EMPTY_INSTRUCTIONS, NO_INSTRUCTIONS, ToolDepthExceeded, prefixText, prefixTextDigest, shingleDigests, toolsetDigest } from './fingerprint.js';
|
|
2
|
+
/**
|
|
3
|
+
* One finished LLM span in, one `runtime.call` out: digests, token counts, ids and timestamps.
|
|
4
|
+
*
|
|
5
|
+
* The confidentiality boundary of the package. Prompt text, tool definitions and responses are
|
|
6
|
+
* read here, hashed here and left here; nothing returned holds a message body. A port of
|
|
7
|
+
* `python/agent-watch-otel/agent_watch_otel/call.py`, held to it by `conformance/runtime/call-vectors.json`.
|
|
8
|
+
*/
|
|
9
|
+
/** The platform enum member for a customer-run span processor (AWT-70). */
|
|
10
|
+
export const PLATFORM = 'otel_processor';
|
|
11
|
+
/** Set to `true` by the runtime gate on a span whose call it refused. */
|
|
12
|
+
export const REFUSED_ATTRIBUTE = 'agent_watch.refused';
|
|
13
|
+
const SPAN_KIND = 'openinference.span.kind';
|
|
14
|
+
const LLM_KIND = 'LLM';
|
|
15
|
+
// The model a refused call is reported under when its span names none: the contract requires one.
|
|
16
|
+
const UNKNOWN_MODEL = 'unknown';
|
|
17
|
+
const AGENT_NAME_MAX = 200;
|
|
18
|
+
const ID_MAX = 200;
|
|
19
|
+
const TOKEN_MAX = 1_000_000_000;
|
|
20
|
+
const INPUT_HIDDEN = 'input_hidden';
|
|
21
|
+
const TOOLS_UNPARSED = 'tools_unparsed';
|
|
22
|
+
const GEN_AI_TOOLS = 'gen_ai.tool.definitions';
|
|
23
|
+
const MESSAGE_ROLE = /^llm\.input_messages\.(\d+)\.message\.role$/;
|
|
24
|
+
const MESSAGE_PART = /^llm\.input_messages\.(\d+)\.message\.contents\.(\d+)\.message_content\.type$/;
|
|
25
|
+
const INSTRUCTION_ROLES = new Set(['system', 'developer']);
|
|
26
|
+
const TOOL_SCHEMA = /^llm\.tools\.(\d+)\.tool\.json_schema$/;
|
|
27
|
+
// Control, unassigned and surrogate code points: Postgres cannot store them or the contract refuses them.
|
|
28
|
+
const UNSTORABLE = /[\p{Cc}\p{Cn}\p{Cs}]/gu;
|
|
29
|
+
// Python's str.strip(): Unicode White_Space plus U+001C..U+001F. Not String.prototype.trim, which
|
|
30
|
+
// also strips U+FEFF and would send a different agent name than the Python package for one span.
|
|
31
|
+
// Tested one UTF-16 unit at a time from each end, never as an anchored `[...]+$` regex: that
|
|
32
|
+
// one retries the run from every position and goes quadratic on a long run of spaces, inside
|
|
33
|
+
// the customer's span.end(). Every such character is in the BMP, so a unit scan is exact.
|
|
34
|
+
// eslint-disable-next-line no-control-regex -- U+001C..U+001F are exactly what Python's strip() adds.
|
|
35
|
+
const EDGE_SPACE = /^[\p{White_Space}\x1c-\x1f]$/u;
|
|
36
|
+
/**
|
|
37
|
+
* Whether the processor reports this span.
|
|
38
|
+
* @param span Any finished span.
|
|
39
|
+
* @returns `true` when its OpenInference kind is `LLM`.
|
|
40
|
+
*/
|
|
41
|
+
export function isLlmSpan(span) {
|
|
42
|
+
return span.attributes[SPAN_KIND] === LLM_KIND;
|
|
43
|
+
}
|
|
44
|
+
/**
|
|
45
|
+
* Turn a finished LLM span into the `runtime.call` the ingest route accepts.
|
|
46
|
+
* @param span A finished span whose OpenInference kind is `LLM`.
|
|
47
|
+
* @param instanceId The tenant's name for this deployment; empty falls back to `service.name`.
|
|
48
|
+
* @returns The payload, or `null` when the span names no model: unpriceable spend is counted as
|
|
49
|
+
* dropped rather than sent under a guess. A refused call has no spend, so it is sent as model `unknown`.
|
|
50
|
+
* @throws Only on a bug; the processor counts it as `dropped_error`.
|
|
51
|
+
*/
|
|
52
|
+
export function buildCall(span, instanceId) {
|
|
53
|
+
const attributes = span.attributes;
|
|
54
|
+
const refused = attributes[REFUSED_ATTRIBUTE] === true;
|
|
55
|
+
const model = storable(firstString(attributes, 'llm.model_name', 'llm.model', 'gen_ai.request.model'), ID_MAX) ?? (refused ? UNKNOWN_MODEL : null);
|
|
56
|
+
if (model === null) {
|
|
57
|
+
return null;
|
|
58
|
+
}
|
|
59
|
+
const resource = span.resource?.attributes ?? {};
|
|
60
|
+
const serviceName = reportedService(resource['service.name']);
|
|
61
|
+
const context = span.spanContext();
|
|
62
|
+
return {
|
|
63
|
+
type: 'runtime.call',
|
|
64
|
+
platform: PLATFORM,
|
|
65
|
+
instance_id: storable(instanceId, ID_MAX) ?? serviceName ?? PLATFORM,
|
|
66
|
+
external_id: context.spanId,
|
|
67
|
+
occurred_at: isoInstant(span.endTime),
|
|
68
|
+
agent_name: storable(span.name, AGENT_NAME_MAX) ?? 'unnamed',
|
|
69
|
+
trace_id: context.traceId,
|
|
70
|
+
provider: storable(firstString(attributes, 'llm.provider', 'llm.system'), ID_MAX) ?? 'unknown',
|
|
71
|
+
model,
|
|
72
|
+
input_tokens: tokenCount(attributes['llm.token_count.prompt']) ?? 0,
|
|
73
|
+
output_tokens: tokenCount(attributes['llm.token_count.completion']) ?? 0,
|
|
74
|
+
...present('cache_read_tokens', tokenCount(attributes['llm.token_count.prompt_details.cache_read'])),
|
|
75
|
+
...present('cache_write_tokens', tokenCount(attributes['llm.token_count.prompt_details.cache_write'])),
|
|
76
|
+
...present('reasoning_tokens', tokenCount(attributes['llm.token_count.completion_details.reasoning'])),
|
|
77
|
+
...present('service_name', serviceName),
|
|
78
|
+
...present('service_version', storable(resource['service.version'], ID_MAX)),
|
|
79
|
+
...fingerprintFields(attributes),
|
|
80
|
+
// The gate refused this call before it reached a provider: a call, but no spend (AWT-336).
|
|
81
|
+
// Last, so every call the gate let through keeps the shared vectors' key order.
|
|
82
|
+
...(refused ? { refused: true } : {})
|
|
83
|
+
};
|
|
84
|
+
}
|
|
85
|
+
// The contract allows a prefix digest beside tools_unparsed, never a digest beside its own
|
|
86
|
+
// absence cause, so the prefix's cause wins when there is one.
|
|
87
|
+
function fingerprintFields(attributes) {
|
|
88
|
+
const messages = leadingInstructions(attributes);
|
|
89
|
+
const { tools, parsed } = toolDefinitions(attributes);
|
|
90
|
+
const toolset = parsed ? boundedToolset(tools) : null;
|
|
91
|
+
const toolsetField = present('toolset_sha256', toolset);
|
|
92
|
+
// Masked input does not mask the tools: half a fingerprint beats none.
|
|
93
|
+
if (messages === null) {
|
|
94
|
+
return { ...toolsetField, uncovered_cause: INPUT_HIDDEN };
|
|
95
|
+
}
|
|
96
|
+
const text = prefixText(messages);
|
|
97
|
+
const prefix = prefixTextDigest(text);
|
|
98
|
+
const shingles = prefix.digest === null || text === null ? [] : shingleDigests(text);
|
|
99
|
+
const prefixCause = prefix.cause === NO_INSTRUCTIONS || prefix.cause === EMPTY_INSTRUCTIONS ? prefix.cause : null;
|
|
100
|
+
const toolsCause = !parsed || (tools.length > 0 && toolset === null) ? TOOLS_UNPARSED : null;
|
|
101
|
+
return {
|
|
102
|
+
...toolsetField,
|
|
103
|
+
...present('prefix_sha256', prefix.digest),
|
|
104
|
+
// Omitted rather than sent empty: a prompt under eight tokens has no windows.
|
|
105
|
+
...(shingles.length > 0 ? { prefix_shingles_sha256: shingles } : {}),
|
|
106
|
+
...present('uncovered_cause', prefixCause ?? toolsCause)
|
|
107
|
+
};
|
|
108
|
+
}
|
|
109
|
+
// An over-deep definition is reported as unparsed rather than hashed, exactly as in Python.
|
|
110
|
+
function boundedToolset(tools) {
|
|
111
|
+
try {
|
|
112
|
+
return toolsetDigest(tools);
|
|
113
|
+
}
|
|
114
|
+
catch (error) {
|
|
115
|
+
if (error instanceof ToolDepthExceeded) {
|
|
116
|
+
return null;
|
|
117
|
+
}
|
|
118
|
+
throw error;
|
|
119
|
+
}
|
|
120
|
+
}
|
|
121
|
+
// One pass over the attributes, bucketed by message index. `null` when there are no messages
|
|
122
|
+
// at all; otherwise the messages in order up to and including the first that is not an
|
|
123
|
+
// instruction. Content is built only for the ones `prefixText` reads: the leading run.
|
|
124
|
+
function leadingInstructions(attributes) {
|
|
125
|
+
const byIndex = new Map();
|
|
126
|
+
const bucket = (index) => {
|
|
127
|
+
let message = byIndex.get(index);
|
|
128
|
+
if (message === undefined) {
|
|
129
|
+
message = { role: undefined, parts: [] };
|
|
130
|
+
byIndex.set(index, message);
|
|
131
|
+
}
|
|
132
|
+
return message;
|
|
133
|
+
};
|
|
134
|
+
const roles = new Set();
|
|
135
|
+
for (const [key, value] of Object.entries(attributes)) {
|
|
136
|
+
const role = MESSAGE_ROLE.exec(key);
|
|
137
|
+
if (role !== null) {
|
|
138
|
+
const index = Number(role[1]);
|
|
139
|
+
bucket(index).role = value;
|
|
140
|
+
roles.add(index);
|
|
141
|
+
continue;
|
|
142
|
+
}
|
|
143
|
+
const part = MESSAGE_PART.exec(key);
|
|
144
|
+
if (part !== null && value === 'text') {
|
|
145
|
+
bucket(Number(part[1])).parts.push([Number(part[2]), key]);
|
|
146
|
+
}
|
|
147
|
+
}
|
|
148
|
+
if (roles.size === 0) {
|
|
149
|
+
return null;
|
|
150
|
+
}
|
|
151
|
+
const leading = [];
|
|
152
|
+
for (const index of [...roles].sort((a, b) => a - b)) {
|
|
153
|
+
const message = byIndex.get(index);
|
|
154
|
+
const role = message?.role;
|
|
155
|
+
if (!INSTRUCTION_ROLES.has(role)) {
|
|
156
|
+
// Its content is never read: `prefixText` stops at the first message that is not an instruction.
|
|
157
|
+
leading.push({ role, content: undefined });
|
|
158
|
+
break;
|
|
159
|
+
}
|
|
160
|
+
leading.push({ role, content: messageContent(attributes, index, message?.parts ?? []) });
|
|
161
|
+
}
|
|
162
|
+
return leading;
|
|
163
|
+
}
|
|
164
|
+
// The flat string, else the message's `contents.N` text parts in order.
|
|
165
|
+
function messageContent(attributes, index, parts) {
|
|
166
|
+
const flat = attributes[`llm.input_messages.${String(index)}.message.content`];
|
|
167
|
+
if (typeof flat === 'string') {
|
|
168
|
+
return flat;
|
|
169
|
+
}
|
|
170
|
+
const texts = [];
|
|
171
|
+
for (const [part, typeKey] of parts) {
|
|
172
|
+
const text = attributes[`${typeKey.slice(0, -'type'.length)}text`];
|
|
173
|
+
if (typeof text === 'string') {
|
|
174
|
+
texts.push([part, { type: 'text', text }]);
|
|
175
|
+
}
|
|
176
|
+
}
|
|
177
|
+
return texts.sort((a, b) => a[0] - b[0]).map(([, part]) => part);
|
|
178
|
+
}
|
|
179
|
+
// `llm.tools.N.tool.json_schema` first; Phoenix leaves `gen_ai.tool.definitions` unconverted, so
|
|
180
|
+
// that is read too. `parsed: false` means tools were there in a shape nothing parsed, which is
|
|
181
|
+
// not the same as no tools.
|
|
182
|
+
function toolDefinitions(attributes) {
|
|
183
|
+
const indexed = [];
|
|
184
|
+
for (const [key, value] of Object.entries(attributes)) {
|
|
185
|
+
const match = TOOL_SCHEMA.exec(key);
|
|
186
|
+
if (match !== null && typeof value === 'string') {
|
|
187
|
+
indexed.push([Number(match[1]), value]);
|
|
188
|
+
}
|
|
189
|
+
}
|
|
190
|
+
if (indexed.length > 0) {
|
|
191
|
+
const loaded = indexed.sort((a, b) => a[0] - b[0]).map(([, schema]) => parseJson(schema));
|
|
192
|
+
return { tools: loaded.filter((tool) => tool !== null), parsed: loaded.every((tool) => tool !== null) };
|
|
193
|
+
}
|
|
194
|
+
const raw = attributes[GEN_AI_TOOLS];
|
|
195
|
+
if (typeof raw !== 'string') {
|
|
196
|
+
return { tools: [], parsed: true };
|
|
197
|
+
}
|
|
198
|
+
const parsed = parseJson(raw);
|
|
199
|
+
if (parsed === null) {
|
|
200
|
+
return { tools: [], parsed: false };
|
|
201
|
+
}
|
|
202
|
+
return { tools: Array.isArray(parsed) ? parsed : [parsed], parsed: true };
|
|
203
|
+
}
|
|
204
|
+
// `null` for anything that is not JSON, and for the JSON literal `null`, as Python's `None` is both.
|
|
205
|
+
function parseJson(text) {
|
|
206
|
+
try {
|
|
207
|
+
return JSON.parse(text);
|
|
208
|
+
}
|
|
209
|
+
catch {
|
|
210
|
+
return null;
|
|
211
|
+
}
|
|
212
|
+
}
|
|
213
|
+
function firstString(attributes, ...keys) {
|
|
214
|
+
for (const key of keys) {
|
|
215
|
+
const value = attributes[key];
|
|
216
|
+
if (typeof value === 'string' && trimEdges(value) !== '') {
|
|
217
|
+
return value;
|
|
218
|
+
}
|
|
219
|
+
}
|
|
220
|
+
return null;
|
|
221
|
+
}
|
|
222
|
+
// A count clamped into the contract's range. Non-finite reads as absent: Python's int() raises
|
|
223
|
+
// there and the call is dropped, and sending zero is the better of the two answers.
|
|
224
|
+
function tokenCount(value) {
|
|
225
|
+
if (typeof value !== 'number' || !Number.isFinite(value)) {
|
|
226
|
+
return null;
|
|
227
|
+
}
|
|
228
|
+
return Math.max(0, Math.min(TOKEN_MAX, Math.trunc(value)));
|
|
229
|
+
}
|
|
230
|
+
/**
|
|
231
|
+
* The resource's `service.name`, unless it is the SDK's `unknown_service[:<process>]` placeholder,
|
|
232
|
+
* which would put a fictional service on the connection page.
|
|
233
|
+
* @param value The resource attribute.
|
|
234
|
+
* @returns The storable name, or `null`.
|
|
235
|
+
*/
|
|
236
|
+
export function reportedService(value) {
|
|
237
|
+
const name = storable(value, ID_MAX);
|
|
238
|
+
return name === null || name.split(':')[0] === 'unknown_service' ? null : name;
|
|
239
|
+
}
|
|
240
|
+
/**
|
|
241
|
+
* Trimmed, length-capped, and free of anything the gateway would refuse. One bad code point in
|
|
242
|
+
* one span name is a 400 for the whole batch, which would cost every other call in it.
|
|
243
|
+
* @param value Any attribute value; anything but a string reads as absent.
|
|
244
|
+
* @param limit The contract's maximum length for this field, in UTF-16 units.
|
|
245
|
+
* @returns The storable text, or `null` when nothing is left of it.
|
|
246
|
+
*/
|
|
247
|
+
export function storable(value, limit) {
|
|
248
|
+
if (typeof value !== 'string') {
|
|
249
|
+
return null;
|
|
250
|
+
}
|
|
251
|
+
const kept = trimEdges(capUtf16(trimEdges(value.replace(UNSTORABLE, '')), limit));
|
|
252
|
+
return kept === '' ? null : kept;
|
|
253
|
+
}
|
|
254
|
+
// Python's str.strip(), linear in the length however long the runs of space are.
|
|
255
|
+
function trimEdges(text) {
|
|
256
|
+
let start = 0;
|
|
257
|
+
let end = text.length;
|
|
258
|
+
while (start < end && EDGE_SPACE.test(text.charAt(start))) {
|
|
259
|
+
start++;
|
|
260
|
+
}
|
|
261
|
+
while (end > start && EDGE_SPACE.test(text.charAt(end - 1))) {
|
|
262
|
+
end--;
|
|
263
|
+
}
|
|
264
|
+
return start === 0 && end === text.length ? text : text.slice(start, end);
|
|
265
|
+
}
|
|
266
|
+
// String.length is already UTF-16 units, which is what zod's .max() counts. Whole code points are
|
|
267
|
+
// copied, so a surrogate pair is never split.
|
|
268
|
+
function capUtf16(text, limit) {
|
|
269
|
+
if (text.length <= limit) {
|
|
270
|
+
return text;
|
|
271
|
+
}
|
|
272
|
+
let kept = '';
|
|
273
|
+
for (const char of text) {
|
|
274
|
+
if (kept.length + char.length > limit) {
|
|
275
|
+
break;
|
|
276
|
+
}
|
|
277
|
+
kept += char;
|
|
278
|
+
}
|
|
279
|
+
return kept;
|
|
280
|
+
}
|
|
281
|
+
// An ISO-8601 instant with an explicit offset, millisecond precision, truncated not rounded: the
|
|
282
|
+
// same text Python's isoformat(timespec="milliseconds") writes.
|
|
283
|
+
function isoInstant([seconds, nanos]) {
|
|
284
|
+
return `${new Date(seconds * 1000 + Math.floor(nanos / 1_000_000)).toISOString().slice(0, 23)}+00:00`;
|
|
285
|
+
}
|
|
286
|
+
// The schema refuses null, so an optional field is either set or absent.
|
|
287
|
+
function present(key, value) {
|
|
288
|
+
return value === null ? {} : { [key]: value };
|
|
289
|
+
}
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* A Node port of `@agent-watch/fingerprint` scheme 1, digest for digest with the Python port.
|
|
3
|
+
*
|
|
4
|
+
* Copied from agent-watch-core `packages/fingerprint/src/fingerprint.ts` at 207ba472, the commit
|
|
5
|
+
* `conformance/runtime/fingerprint-vectors.json` was cut from, with one addition: the
|
|
6
|
+
* `MAX_TOOL_DEPTH` bound the Python port states, so an over-deep tool definition is refused at
|
|
7
|
+
* the same depth in both ports instead of at whatever depth the stack runs out.
|
|
8
|
+
*/
|
|
9
|
+
/** Names the rules in the shared vectors. Digests from different schemes never compare. */
|
|
10
|
+
export declare const FINGERPRINT_SCHEME = "agent-watch-fingerprint/1";
|
|
11
|
+
/** How deep a tool definition may nest before canonical JSON refuses it. Python holds the same number. */
|
|
12
|
+
export declare const MAX_TOOL_DEPTH = 100;
|
|
13
|
+
/** The first message is not `system` or `developer`, or there are no messages. */
|
|
14
|
+
export declare const NO_INSTRUCTIONS = "no_instructions";
|
|
15
|
+
/** The leading instruction messages normalise to the empty string. */
|
|
16
|
+
export declare const EMPTY_INSTRUCTIONS = "empty_instructions";
|
|
17
|
+
export type PrefixCause = typeof NO_INSTRUCTIONS | typeof EMPTY_INSTRUCTIONS;
|
|
18
|
+
export type PrefixDigest = {
|
|
19
|
+
readonly digest: string;
|
|
20
|
+
readonly cause: null;
|
|
21
|
+
} | {
|
|
22
|
+
readonly digest: null;
|
|
23
|
+
readonly cause: PrefixCause;
|
|
24
|
+
};
|
|
25
|
+
/** A tool definition nested deeper than `MAX_TOOL_DEPTH`. */
|
|
26
|
+
export declare class ToolDepthExceeded extends RangeError {
|
|
27
|
+
}
|
|
28
|
+
/**
|
|
29
|
+
* NFC, CRLF to LF, trailing whitespace off every line and off the end. Never lowercases.
|
|
30
|
+
* @param text Prompt or source text as the platform stored it.
|
|
31
|
+
* @returns The text every digest here is taken over.
|
|
32
|
+
*/
|
|
33
|
+
export declare function normalise(text: string): string;
|
|
34
|
+
/**
|
|
35
|
+
* The normalised leading `system`/`developer` run, joined by a blank line. Producers shingle
|
|
36
|
+
* exactly this text; any other join changes the shingles.
|
|
37
|
+
* @param messages Messages in order, in whatever shape the platform stored them.
|
|
38
|
+
* @returns The prefix text, or `null` when the first message is not an instruction.
|
|
39
|
+
*/
|
|
40
|
+
export declare function prefixText(messages: readonly unknown[]): string | null;
|
|
41
|
+
/**
|
|
42
|
+
* sha256 hex of `prefixText`, or the reason there is none.
|
|
43
|
+
* @param messages Messages in order.
|
|
44
|
+
* @returns The digest with a `null` cause, or a `null` digest with its cause.
|
|
45
|
+
*/
|
|
46
|
+
export declare function prefixDigest(messages: readonly unknown[]): PrefixDigest;
|
|
47
|
+
/**
|
|
48
|
+
* `prefixDigest` from text `prefixText` already returned, so a caller that also shingles it
|
|
49
|
+
* builds the text once.
|
|
50
|
+
* @param text `prefixText`'s output.
|
|
51
|
+
* @returns The digest with a `null` cause, or a `null` digest with its cause.
|
|
52
|
+
*/
|
|
53
|
+
export declare function prefixTextDigest(text: string | null): PrefixDigest;
|
|
54
|
+
/**
|
|
55
|
+
* sha256 hex of the tool definitions as canonical JSON, sorted by (name, JSON) and newline-joined.
|
|
56
|
+
* @param tools Tool definitions as parsed JSON, OpenAI (`function.name`) or Anthropic (`name`) shape.
|
|
57
|
+
* @returns 64 hex characters, or `null` when there are no tools.
|
|
58
|
+
* @throws {ToolDepthExceeded} A definition nests past `MAX_TOOL_DEPTH`.
|
|
59
|
+
*/
|
|
60
|
+
export declare function toolsetDigest(tools: readonly unknown[]): string | null;
|
|
61
|
+
/**
|
|
62
|
+
* 16-byte hex digests of every 8-token window, deduplicated in order, at most 512.
|
|
63
|
+
* @param text Prompt text, normally `prefixText`'s output.
|
|
64
|
+
* @returns Up to 512 digests of 32 hex characters each.
|
|
65
|
+
*/
|
|
66
|
+
export declare function shingleDigests(text: string): string[];
|
|
67
|
+
/**
|
|
68
|
+
* The words a window is taken over. Exported for the source-file runner in the tests.
|
|
69
|
+
* @param text Prompt or source text.
|
|
70
|
+
* @returns Its tokens in order.
|
|
71
|
+
*/
|
|
72
|
+
export declare function tokens(text: string): string[];
|
|
73
|
+
/**
|
|
74
|
+
* 16-byte hex digests of every 8-token window, deduplicated in order.
|
|
75
|
+
* @param words `tokens`' output.
|
|
76
|
+
* @param cap At most this many digests.
|
|
77
|
+
* @returns The digests.
|
|
78
|
+
*/
|
|
79
|
+
export declare function windowDigests(words: readonly string[], cap: number): string[];
|