toad-eye 2.4.0 → 2.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +123 -30
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -18,6 +18,8 @@ You don't know what your LLMs cost, how slow they are, or why they fail. toad-ey
|
|
|
18
18
|
|
|
19
19
|
## Quick start
|
|
20
20
|
|
|
21
|
+
**Prerequisites:** [Docker Desktop](https://docs.docker.com/get-started/get-docker/) (or Docker Engine + Compose plugin)
|
|
22
|
+
|
|
21
23
|
```bash
|
|
22
24
|
npm install toad-eye
|
|
23
25
|
npx toad-eye init # scaffold observability stack
|
|
@@ -25,6 +27,8 @@ npx toad-eye up # start Grafana + Prometheus + Jaeger
|
|
|
25
27
|
npx toad-eye demo # send mock LLM traffic, see data immediately
|
|
26
28
|
```
|
|
27
29
|
|
|
30
|
+
> Dashboards will be empty until data arrives. Run `npx toad-eye demo` to send mock traffic and verify the stack works.
|
|
31
|
+
|
|
28
32
|
```typescript
|
|
29
33
|
import { initObservability } from "toad-eye";
|
|
30
34
|
|
|
@@ -36,6 +40,8 @@ initObservability({
|
|
|
36
40
|
// That's it. Every SDK call is auto-traced — including streaming.
|
|
37
41
|
```
|
|
38
42
|
|
|
43
|
+
> ⚠️ By default, toad-eye records prompt and completion text in spans. Set `recordContent: false` for PII-sensitive workloads (healthcare, finance, legal).
|
|
44
|
+
|
|
39
45
|
Open [localhost:3100](http://localhost:3100) (Grafana, admin/admin) to see your dashboards.
|
|
40
46
|
|
|
41
47
|
## What you get
|
|
@@ -46,16 +52,24 @@ Open [localhost:3100](http://localhost:3100) (Grafana, admin/admin) to see your
|
|
|
46
52
|
| **8 Grafana dashboards** | Overview, Cost, Latency, Errors, Model Comparison, FinOps, Provider Health, Agent Workflow |
|
|
47
53
|
| **Cost tracking** | Per-request USD cost, daily totals, projected monthly spend |
|
|
48
54
|
| **Budget guards** | Daily/per-user/per-model spend limits — warn, block, or auto-downgrade |
|
|
49
|
-
| **Multi-agent tracing** | ReAct steps, handoffs, loop detection, maxSteps guard |
|
|
50
|
-
| **Shadow guardrails** | Record what _would_ be blocked without blocking — tune thresholds on live traffic |
|
|
51
|
-
| **Quality proxy** | Empty response rate, latency per token — zero-dependency quality metrics |
|
|
52
|
-
| **Semantic drift** | Detect silent quality degradation via embedding comparison |
|
|
53
55
|
| **Privacy controls** | Built-in PII redaction (email, SSN, CC, phone), hashing, content masking |
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
|
56
|
+
|
|
57
|
+
<details>
|
|
58
|
+
<summary>Advanced features</summary>
|
|
59
|
+
|
|
60
|
+
| Feature | Description |
|
|
61
|
+
| ----------------------- | --------------------------------------------------------------------------------- |
|
|
62
|
+
| **Multi-agent tracing** | ReAct steps, handoffs, loop detection, maxSteps guard |
|
|
63
|
+
| **Shadow guardrails** | Record what _would_ be blocked without blocking — tune thresholds on live traffic |
|
|
64
|
+
| **Quality proxy** | Empty response rate, latency per token — zero-dependency quality metrics |
|
|
65
|
+
| **Semantic drift** | Detect silent quality degradation via embedding comparison |
|
|
66
|
+
| **Tail sampling** | Keep 100% errors + slow traces, sample healthy traffic via OTel Collector |
|
|
67
|
+
| **Alerting** | Cost spikes, latency anomalies, error rate — via Telegram, Slack, email, webhook |
|
|
68
|
+
| **FinOps attribution** | Break down costs by team, user, feature, environment |
|
|
69
|
+
| **TTFT metric** | Time To First Token for streaming — separate from total duration |
|
|
70
|
+
| **Trace export** | Convert production Jaeger traces into regression test cases |
|
|
71
|
+
|
|
72
|
+
</details>
|
|
59
73
|
|
|
60
74
|
## Auto-instrumentation 🤖
|
|
61
75
|
|
|
@@ -77,17 +91,35 @@ Vercel AI SDK uses a SpanProcessor (not monkey-patching) — add `instrument: ['
|
|
|
77
91
|
<details>
|
|
78
92
|
<summary>Manual instrumentation (custom providers)</summary>
|
|
79
93
|
|
|
94
|
+
Use `traceLLMCall` when you need to instrument a provider not in the auto-instrument list, or want explicit control over span data:
|
|
95
|
+
|
|
80
96
|
```typescript
|
|
97
|
+
import OpenAI from "openai";
|
|
81
98
|
import { initObservability, traceLLMCall } from "toad-eye";
|
|
82
99
|
|
|
83
100
|
initObservability({ serviceName: "my-app" });
|
|
84
101
|
|
|
102
|
+
const openai = new OpenAI();
|
|
103
|
+
|
|
85
104
|
const result = await traceLLMCall(
|
|
86
|
-
{ provider: "
|
|
87
|
-
() =>
|
|
105
|
+
{ provider: "openai", model: "gpt-4o", prompt: "Hello" },
|
|
106
|
+
async () => {
|
|
107
|
+
const res = await openai.chat.completions.create({
|
|
108
|
+
model: "gpt-4o",
|
|
109
|
+
messages: [{ role: "user", content: "Hello" }],
|
|
110
|
+
});
|
|
111
|
+
return {
|
|
112
|
+
completion: res.choices[0]?.message.content ?? "",
|
|
113
|
+
inputTokens: res.usage?.prompt_tokens ?? 0,
|
|
114
|
+
outputTokens: res.usage?.completion_tokens ?? 0,
|
|
115
|
+
// cost is auto-calculated from model + tokens if omitted
|
|
116
|
+
};
|
|
117
|
+
},
|
|
88
118
|
);
|
|
89
119
|
```
|
|
90
120
|
|
|
121
|
+
> Or skip this entirely — use `instrument: ['openai']` for zero-code instrumentation.
|
|
122
|
+
|
|
91
123
|
</details>
|
|
92
124
|
|
|
93
125
|
## Budget guards
|
|
@@ -114,11 +146,36 @@ initObservability({
|
|
|
114
146
|
|
|
115
147
|
## Agent observability
|
|
116
148
|
|
|
117
|
-
Structured tracing for ReAct agents with multi-agent support, handoffs, and loop detection.
|
|
149
|
+
Structured tracing for ReAct agents with multi-agent support, handoffs, and loop detection. Span names follow [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/).
|
|
118
150
|
|
|
119
151
|
```typescript
|
|
120
152
|
import { traceAgentQuery } from "toad-eye";
|
|
121
153
|
|
|
154
|
+
const result = await traceAgentQuery(
|
|
155
|
+
{
|
|
156
|
+
query: "What's the weather like?",
|
|
157
|
+
agentName: "weather-bot", // → gen_ai.agent.name (OTel semconv)
|
|
158
|
+
},
|
|
159
|
+
async (step) => {
|
|
160
|
+
step({ type: "think", stepNumber: 1, content: "Need weather data" });
|
|
161
|
+
const data = await getWeather();
|
|
162
|
+
step({
|
|
163
|
+
type: "act",
|
|
164
|
+
stepNumber: 2,
|
|
165
|
+
toolName: "get_weather",
|
|
166
|
+
toolType: "function",
|
|
167
|
+
});
|
|
168
|
+
step({ type: "answer", stepNumber: 3, content: data.summary });
|
|
169
|
+
return { answer: data.summary };
|
|
170
|
+
},
|
|
171
|
+
);
|
|
172
|
+
// Produces spans: invoke_agent weather-bot → execute_tool get_weather
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
<details>
|
|
176
|
+
<summary>Full ReAct pattern (think → act → observe → answer)</summary>
|
|
177
|
+
|
|
178
|
+
```typescript
|
|
122
179
|
const result = await traceAgentQuery(
|
|
123
180
|
"Is anything dangerous near Earth?",
|
|
124
181
|
async (step) => {
|
|
@@ -144,6 +201,8 @@ const result = await traceAgentQuery(
|
|
|
144
201
|
);
|
|
145
202
|
```
|
|
146
203
|
|
|
204
|
+
</details>
|
|
205
|
+
|
|
147
206
|
## FinOps attribution
|
|
148
207
|
|
|
149
208
|
Track costs by team, user, feature, or any business dimension:
|
|
@@ -201,6 +260,14 @@ monitor.checkInBackground(response, "openai", "gpt-4o");
|
|
|
201
260
|
|
|
202
261
|
### Privacy controls
|
|
203
262
|
|
|
263
|
+
| Goal | Config |
|
|
264
|
+
| --------------------------- | -------------------------------- |
|
|
265
|
+
| Don't store text at all | `recordContent: false` |
|
|
266
|
+
| Remove common PII | `redactDefaults: true` |
|
|
267
|
+
| Remove custom patterns | `redactPatterns: [/regex/g]` |
|
|
268
|
+
| Store hashes only (no text) | `hashContent: true, salt: "..."` |
|
|
269
|
+
| Audit what was masked | `auditMasking: true` |
|
|
270
|
+
|
|
204
271
|
```typescript
|
|
205
272
|
initObservability({
|
|
206
273
|
serviceName: "my-app",
|
|
@@ -289,11 +356,25 @@ Self-hosted mode remains the default. Cloud mode activates automatically when `a
|
|
|
289
356
|
| **Provider Health** | Provider status (healthy/degraded/down), uptime, error breakdown |
|
|
290
357
|
| **Agent Workflow** | Steps per query, tool usage frequency, step type breakdown |
|
|
291
358
|
|
|
359
|
+
## Subpath imports
|
|
360
|
+
|
|
361
|
+
Advanced modules are available as separate entry points to keep your bundle lean:
|
|
362
|
+
|
|
363
|
+
```typescript
|
|
364
|
+
import { initObservability, traceLLMCall } from "toad-eye"; // core
|
|
365
|
+
import { AlertManager } from "toad-eye/alerts"; // alerting
|
|
366
|
+
import { createDriftMonitor } from "toad-eye/drift"; // semantic drift
|
|
367
|
+
import { exportTrace } from "toad-eye/export"; // trace → YAML
|
|
368
|
+
import { ToadEyeAISpanProcessor, withToadEye } from "toad-eye/vercel"; // Vercel AI SDK
|
|
369
|
+
```
|
|
370
|
+
|
|
371
|
+
> Everything is also re-exported from the main `"toad-eye"` entry point for convenience.
|
|
372
|
+
|
|
292
373
|
## CLI
|
|
293
374
|
|
|
294
375
|
| Command | Description |
|
|
295
376
|
| -------------------------------- | ----------------------------------------------- |
|
|
296
|
-
| `npx toad-eye init`
|
|
377
|
+
| `npx toad-eye init [--force]` | Scaffold Docker Compose + observability configs |
|
|
297
378
|
| `npx toad-eye up` | Start the stack |
|
|
298
379
|
| `npx toad-eye down` | Stop the stack |
|
|
299
380
|
| `npx toad-eye status` | Show running services and URLs |
|
|
@@ -330,22 +411,26 @@ Self-hosted mode remains the default. Cloud mode activates automatically when `a
|
|
|
330
411
|
| `gen_ai.toad_eye.response.empty` | Counter | Empty/whitespace responses |
|
|
331
412
|
| `gen_ai.toad_eye.response.latency_per_token` | Histogram | Generation speed (ms/token) |
|
|
332
413
|
|
|
333
|
-
### Span attributes
|
|
334
|
-
|
|
335
|
-
| Attribute
|
|
336
|
-
|
|
|
337
|
-
| `gen_ai.
|
|
338
|
-
| `gen_ai.
|
|
339
|
-
| `gen_ai.
|
|
340
|
-
| `gen_ai.usage.
|
|
341
|
-
| `gen_ai.
|
|
342
|
-
| `gen_ai.
|
|
343
|
-
| `gen_ai.agent.
|
|
344
|
-
| `gen_ai.agent.
|
|
345
|
-
| `gen_ai.
|
|
346
|
-
| `gen_ai.
|
|
347
|
-
| `gen_ai.toad_eye.
|
|
348
|
-
| `
|
|
414
|
+
### Span attributes (OTel GenAI semconv)
|
|
415
|
+
|
|
416
|
+
| Attribute | Description |
|
|
417
|
+
| ---------------------------------- | ----------------------------------------------- |
|
|
418
|
+
| `gen_ai.operation.name` | `chat`, `invoke_agent`, `execute_tool` |
|
|
419
|
+
| `gen_ai.provider.name` | `anthropic`, `gemini`, `openai` |
|
|
420
|
+
| `gen_ai.request.model` | Model identifier |
|
|
421
|
+
| `gen_ai.usage.input_tokens` | Tokens in the prompt |
|
|
422
|
+
| `gen_ai.usage.output_tokens` | Tokens in the completion |
|
|
423
|
+
| `gen_ai.request.temperature` | Temperature parameter |
|
|
424
|
+
| `gen_ai.agent.name` | Agent name (for agent spans) |
|
|
425
|
+
| `gen_ai.agent.id` | Agent identifier |
|
|
426
|
+
| `gen_ai.tool.name` | Tool name for execute_tool spans |
|
|
427
|
+
| `gen_ai.tool.type` | `function`, `extension`, `retrieval`, `builtin` |
|
|
428
|
+
| `gen_ai.toad_eye.cost` | Cost in USD |
|
|
429
|
+
| `gen_ai.toad_eye.agent.step.type` | ReAct step: think/act/observe/answer/handoff |
|
|
430
|
+
| `gen_ai.toad_eye.agent.handoff.to` | Target agent for handoff steps |
|
|
431
|
+
| `gen_ai.toad_eye.agent.loop_count` | Agent loop iterations (observe→think) |
|
|
432
|
+
| `gen_ai.toad_eye.guard.mode` | Guard mode: shadow/enforce |
|
|
433
|
+
| `session.id` | Session identifier (if configured) |
|
|
349
434
|
|
|
350
435
|
</details>
|
|
351
436
|
|
|
@@ -358,9 +443,17 @@ Self-hosted mode remains the default. Cloud mode activates automatically when `a
|
|
|
358
443
|
| Prometheus | [localhost:9090](http://localhost:9090) |
|
|
359
444
|
| OTel Collector | [localhost:4318](http://localhost:4318) |
|
|
360
445
|
|
|
446
|
+
## OTel GenAI semconv
|
|
447
|
+
|
|
448
|
+
toad-eye follows [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) for span names, attributes, and metrics. Traces are natively compatible with Jaeger, Arize Phoenix, SigNoz, Datadog, Langfuse, and other OTel backends.
|
|
449
|
+
|
|
450
|
+
See [COMPATIBILITY.md](https://github.com/vola-trebla/toad-eye/blob/main/packages/instrumentation/COMPATIBILITY.md) for a full backend support matrix.
|
|
451
|
+
|
|
452
|
+
Set `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` to disable deprecated attribute aliases and emit only the latest conventions.
|
|
453
|
+
|
|
361
454
|
## Tech stack
|
|
362
455
|
|
|
363
456
|
- TypeScript, OpenTelemetry SDK 2.x, OTel GenAI semantic conventions
|
|
364
457
|
- Hono (demo server + cloud ingestion server)
|
|
365
458
|
- Docker Compose (Prometheus, Jaeger, Grafana, OTel Collector)
|
|
366
|
-
- Vitest (
|
|
459
|
+
- Vitest (252+ tests)
|