toad-eye 2.6.1 → 2.8.0

package/README.md

@@ -1,165 +1,98 @@
-# toad-eye 🐸👁️‍🗨️🤖
+# toad-eye 🐸👁️
 
+**Observability for MCP servers and LLM applications.**
+
+One line of code. Full traces, metrics, and Grafana dashboards.
+Self-hosted. Privacy-first. No vendor lock-in.
+
+[![npm](https://img.shields.io/npm/v/toad-eye?color=CB3837&logo=npm&logoColor=white)](https://www.npmjs.com/package/toad-eye)
 ![TypeScript](https://img.shields.io/badge/TypeScript-3178C6?logo=typescript&logoColor=white)
 ![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-000000?logo=opentelemetry&logoColor=white)
-![Hono](https://img.shields.io/badge/Hono-E36002?logo=hono&logoColor=white)
-![Prometheus](https://img.shields.io/badge/Prometheus-E6522C?logo=prometheus&logoColor=white)
-![Grafana](https://img.shields.io/badge/Grafana-F46800?logo=grafana&logoColor=white)
-![Docker](https://img.shields.io/badge/Docker-2496ED?logo=docker&logoColor=white)
-[![npm](https://img.shields.io/npm/v/toad-eye?color=CB3837&logo=npm&logoColor=white)](https://www.npmjs.com/package/toad-eye)
 ![CI](https://github.com/vola-trebla/toad-eye/actions/workflows/ci.yml/badge.svg)
 ![License](https://img.shields.io/badge/License-ISC-blue)
 
-**LLM observability in 3 commands. Auto-instrument, trace, alert, save money.**
-
-You don't know what your LLMs cost, how slow they are, or why they fail. toad-eye gives you complete visibility with zero manual instrumentation.
-
 ![toad-eye demo](https://raw.githubusercontent.com/vola-trebla/toad-eye/main/demo/toad-eye-demo-gifski.gif)
 
-## Quick start
-
-**Prerequisites:** [Docker Desktop](https://docs.docker.com/get-started/get-docker/) (or Docker Engine + Compose plugin)
+---
 
-```bash
-npm install toad-eye
-npx toad-eye init       # scaffold observability stack
-npx toad-eye up         # start Grafana + Prometheus + Jaeger
-npx toad-eye demo       # send mock LLM traffic, see data immediately
-```
+## Quick Start — MCP Server
 
-> Dashboards will be empty until data arrives. Run `npx toad-eye demo` to send mock traffic and verify the stack works.
+Add observability to any MCP server in 2 lines:
 
 ```typescript
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 import { initObservability } from "toad-eye";
+import { toadEyeMiddleware } from "toad-eye/mcp";
 
-initObservability({
-  serviceName: "my-app",
-  instrument: ["openai", "anthropic"],
-});
-
-// That's it. Every SDK call is auto-traced — including streaming.
-```
-
-> ⚠️ By default, toad-eye records prompt and completion text in spans. Set `recordContent: false` for PII-sensitive workloads (healthcare, finance, legal).
-
-Open [localhost:3100](http://localhost:3100) (Grafana, admin/admin) to see your dashboards.
+initObservability({ serviceName: "my-mcp-server" });
 
-## What you get
+const server = new McpServer({ name: "my-server", version: "1.0.0" });
+toadEyeMiddleware(server);
 
-| Feature                  | Description                                                                                |
-| ------------------------ | ------------------------------------------------------------------------------------------ |
-| **Auto-instrumentation** | OpenAI, Anthropic, Gemini, Vercel AI SDK — regular + streaming                             |
-| **8 Grafana dashboards** | Overview, Cost, Latency, Errors, Model Comparison, FinOps, Provider Health, Agent Workflow |
-| **Cost tracking**        | Per-request USD cost, daily totals, projected monthly spend                                |
-| **Budget guards**        | Daily/per-user/per-model spend limits — warn, block, or auto-downgrade                     |
-| **Privacy controls**     | Built-in PII redaction (email, SSN, CC, phone), hashing, content masking                   |
+// Every tool call, resource read, and prompt is now traced.
+// Spans appear in Jaeger. Metrics flow to Prometheus. Dashboards ready in Grafana.
+```
 
-<details>
-<summary>Advanced features</summary>
+Privacy by default — tool arguments and results are NOT recorded unless you opt in:
 
-| Feature                 | Description                                                                       |
-| ----------------------- | --------------------------------------------------------------------------------- |
-| **Multi-agent tracing** | ReAct steps, handoffs, loop detection, maxSteps guard                             |
-| **Shadow guardrails**   | Record what _would_ be blocked without blocking — tune thresholds on live traffic |
-| **Quality proxy**       | Empty response rate, latency per token — zero-dependency quality metrics          |
-| **Semantic drift**      | Detect silent quality degradation via embedding comparison                        |
-| **Tail sampling**       | Keep 100% errors + slow traces, sample healthy traffic via OTel Collector         |
-| **Alerting**            | Cost spikes, latency anomalies, error rate — via Telegram, Slack, email, webhook  |
-| **FinOps attribution**  | Break down costs by team, user, feature, environment                              |
-| **TTFT metric**         | Time To First Token for streaming — separate from total duration                  |
-| **Trace export**        | Convert production Jaeger traces into regression test cases                       |
+```typescript
+toadEyeMiddleware(server, {
+  recordInputs: true,
+  redactKeys: ["apiKey", "token"],
+});
+```
 
-</details>
+Safe for stdio transport — OTel diagnostics are redirected to stderr.
 
-## Auto-instrumentation 🤖
+## Quick Start — LLM Calls
 
-Zero-code instrumentation for popular LLM SDKs. No wrappers needed.
+Auto-instrument OpenAI, Anthropic, Gemini, and Vercel AI SDK — zero wrappers:
 
 ```typescript
+import { initObservability } from "toad-eye";
+
 initObservability({
   serviceName: "my-app",
   instrument: ["openai", "anthropic"],
 });
-```
-
-**Supported SDKs:** OpenAI (`openai`), Anthropic (`@anthropic-ai/sdk`), Google GenAI (`@google/generative-ai`), Vercel AI SDK (`ai`)
-
-Both regular and **streaming** calls are fully instrumented — spans, metrics, and cost tracking work transparently for `stream: true`.
-
-Vercel AI SDK uses a SpanProcessor (not monkey-patching) — add `instrument: ['ai']` and `experimental_telemetry: withToadEye()` to your calls.
-
-<details>
-<summary>Manual instrumentation (custom providers)</summary>
-
-Use `traceLLMCall` when you need to instrument a provider not in the auto-instrument list, or want explicit control over span data:
 
-```typescript
-import OpenAI from "openai";
-import { initObservability, traceLLMCall } from "toad-eye";
-
-initObservability({ serviceName: "my-app" });
-
-const openai = new OpenAI();
-
-const result = await traceLLMCall(
-  { provider: "openai", model: "gpt-4o", prompt: "Hello" },
-  async () => {
-    const res = await openai.chat.completions.create({
-      model: "gpt-4o",
-      messages: [{ role: "user", content: "Hello" }],
-    });
-    return {
-      completion: res.choices[0]?.message.content ?? "",
-      inputTokens: res.usage?.prompt_tokens ?? 0,
-      outputTokens: res.usage?.completion_tokens ?? 0,
-      // cost is auto-calculated from model + tokens if omitted
-    };
-  },
-);
+// Every SDK call is auto-traced — including streaming.
 ```
 
-> Or skip this entirely — use `instrument: ['openai']` for zero-code instrumentation.
+## Set Up the Stack
 
-> **Note:** In short-lived scripts, call `await shutdown()` before exit to flush buffered spans. Long-running servers (Express, Hono, Next.js) don't need this — spans flush automatically.
+```bash
+npm install toad-eye
+npx toad-eye init       # scaffold observability configs
+npx toad-eye up         # start Grafana + Prometheus + Jaeger + OTel Collector
+npx toad-eye demo       # send mock traffic — see data in Grafana immediately
+```
 
-</details>
+Open [localhost:3100](http://localhost:3100) (Grafana, admin/admin). 11 dashboards are pre-built and ready.
 
-<details>
-<summary>MCP Server observability</summary>
+> **Requires:** [Docker Desktop](https://docs.docker.com/get-started/get-docker/) (or Docker Engine + Compose plugin)
 
-Drop-in middleware for MCP servers — one line for full tracing of every tool call, resource read, and prompt request.
+<!-- TODO: add Grafana dashboard screenshot here -->
 
-```typescript
-import { McpServer } from "@modelcontextprotocol/sdk/server/index.js";
-import { toadEyeMiddleware } from "toad-eye/mcp";
+## What You Get
 
-const server = new McpServer({ name: "my-server", version: "1.0.0" });
-toadEyeMiddleware(server);
+**Instrumentation:** auto-instrument OpenAI, Anthropic, Gemini, Vercel AI SDK, and MCP servers — regular and streaming calls, with full cost tracking.
 
-// All handlers are now instrumented — spans appear in Jaeger
-server.tool("calculate", { expression: z.string() }, async ({ expression }) => {
-  return { content: [{ type: "text", text: String(eval(expression)) }] };
-});
-```
+**11 Grafana dashboards:** Overview, Cost Breakdown, Latency Analysis, Errors, Model Comparison, FinOps Attribution, Provider Health, Agent Workflow, MCP Server, MCP End-to-End, MCP Tool Analytics.
 
-Spans follow OTel GenAI semconv: `execute_tool calculate`, `retrieval file:///docs`, `prompt greeting`.
+**Budget guards:** daily, per-user, per-model spend limits. Three modes — warn, block, or auto-downgrade to a cheaper model.
 
-Privacy by default — tool arguments and results are NOT recorded unless you opt in:
+**Agent tracing:** structured ReAct tracing with multi-agent support, handoffs, and loop detection. Follows OTel GenAI semantic conventions.
 
-```typescript
-toadEyeMiddleware(server, {
-  recordInputs: true,
-  redactKeys: ["apiKey", "token"],
-});
-```
+**Privacy controls:** `recordContent: false` to disable prompt/completion recording, built-in PII redaction (email, SSN, CC, phone), SHA-256 hashing, key redaction.
 
-> Safe for stdio transport — OTel diagnostics are redirected to stderr.
+**Alerting:** cost spikes, latency anomalies, error rate alerts via Telegram, Slack, email, or webhook.
 
-</details>
+**Semantic drift detection:** catch silent LLM quality degradation by comparing responses to a saved baseline via embeddings.
 
-## Budget guards
+**Trace export:** convert production Jaeger traces into regression test cases for [toad-eval](https://github.com/vola-trebla/toad-eval).
 
-Prevent cost overruns at runtime. Three modes: warn, block, or auto-downgrade to a cheaper model.
+## Budget Guards
 
 ```typescript
 initObservability({
@@ -169,326 +102,71 @@ initObservability({
     perUser: 5, // $5/day per user
     perModel: { "gpt-4o": 30 }, // $30/day on GPT-4o
   },
-  onBudgetExceeded: "block", // throws ToadBudgetExceededError
+  onBudgetExceeded: "block", // or "warn" or "downgrade"
 });
 ```
 
-| Mode        | Behavior                                            |
-| ----------- | --------------------------------------------------- |
-| `warn`      | Log warning, continue normally                      |
-| `block`     | Throw `ToadBudgetExceededError` before LLM call     |
-| `downgrade` | Call `downgradeCallback` to switch to cheaper model |
-
-## Agent observability
-
-Structured tracing for ReAct agents with multi-agent support, handoffs, and loop detection. Span names follow [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/).
+## Agent Observability
 
 ```typescript
 import { traceAgentQuery } from "toad-eye";
 
 const result = await traceAgentQuery(
-  {
-    query: "What's the weather like?",
-    agentName: "weather-bot", // → gen_ai.agent.name (OTel semconv)
-  },
+  { query: "What's the weather?", agentName: "weather-bot" },
   async (step) => {
     step({ type: "think", stepNumber: 1, content: "Need weather data" });
     const data = await getWeather();
-    step({
-      type: "act",
-      stepNumber: 2,
-      toolName: "get_weather",
-      toolType: "function",
-    });
+    step({ type: "act", stepNumber: 2, toolName: "get_weather" });
     step({ type: "answer", stepNumber: 3, content: data.summary });
     return { answer: data.summary };
   },
 );
-// Produces spans: invoke_agent weather-bot → execute_tool get_weather
-```
-
-<details>
-<summary>Full ReAct pattern (think → act → observe → answer)</summary>
-
-```typescript
-const result = await traceAgentQuery(
-  "Is anything dangerous near Earth?",
-  async (step) => {
-    step({
-      type: "think",
-      stepNumber: 1,
-      content: "I need to check asteroids",
-    });
-    const data = await callTool("near-earth-asteroids");
-    step({ type: "act", stepNumber: 2, toolName: "near-earth-asteroids" });
-    step({
-      type: "observe",
-      stepNumber: 3,
-      content: `${data.length} asteroids found`,
-    });
-    step({
-      type: "answer",
-      stepNumber: 4,
-      content: "7 asteroids passing safely",
-    });
-    return { answer: "7 asteroids passing safely" };
-  },
-);
-```
-
-</details>
-
-## FinOps attribution
-
-Track costs by team, user, feature, or any business dimension:
-
-```typescript
-initObservability({
-  serviceName: "my-app",
-  attributes: { team: "checkout", environment: "production" },
-});
-
-await traceLLMCall(
-  {
-    provider: "openai",
-    model: "gpt-4o",
-    prompt: "Summarize order",
-    attributes: { userId: "user-123", feature: "order-summary" },
-  },
-  () => callLLM(),
-);
-```
-
-<details>
-<summary>More features</summary>
-
-### Shadow guardrails
-
-Record what _would_ have been blocked without blocking. Tune thresholds on live traffic.
-
-```typescript
-import { recordGuardResult } from "toad-eye";
-
-recordGuardResult({
-  mode: "shadow",
-  passed: false,
-  ruleName: "pii_filter",
-  failureReason: "SSN detected in response",
-});
-```
-
-### Semantic drift monitoring
-
-Detect silent LLM quality degradation by comparing responses to a saved baseline via embeddings.
-
-```typescript
-import { createDriftMonitor } from "toad-eye";
-
-const monitor = createDriftMonitor({
-  embedding: { provider: "openai", apiKey: process.env.OPENAI_API_KEY! },
-  baselinePath: "./baseline.json",
-  sampleRate: 10,
-});
-
-monitor.checkInBackground(response, "openai", "gpt-4o");
-```
-
-### Privacy controls
-
-| Goal                        | Config                           |
-| --------------------------- | -------------------------------- |
-| Don't store text at all     | `recordContent: false`           |
-| Remove common PII           | `redactDefaults: true`           |
-| Remove custom patterns      | `redactPatterns: [/regex/g]`     |
-| Store hashes only (no text) | `hashContent: true, salt: "..."` |
-| Audit what was masked       | `auditMasking: true`             |
-
-```typescript
-initObservability({
-  serviceName: "my-app",
-  redactDefaults: true, // built-in PII patterns (email, SSN, CC, phone)
-  recordContent: false, // disable prompt/completion recording
-  hashContent: true, // SHA-256 hash instead of plain text
-  salt: "my-secret", // prevent rainbow table attacks
-  auditMasking: true, // log what was masked (debug only)
-  redactPatterns: [/custom-pattern/g], // additional regex redaction
-});
-```
-
-### Session tracking
-
-```typescript
-initObservability({
-  serviceName: "my-app",
-  sessionId: "static-session-id",
-  sessionExtractor: () => getCurrentSessionId(), // or dynamic
-});
-```
-
-### Built-in cost tracking
-
-```typescript
-import { setCustomPricing } from "toad-eye";
-
-setCustomPricing({
-  "my-fine-tuned-model": { inputPer1M: 5, outputPer1M: 15 },
-});
+// Produces: invoke_agent weather-bot → execute_tool get_weather
 ```
 
-### Alerting
-
-```yaml
-alerts:
-  - name: cost_spike
-    metric: gen_ai.client.request.cost
-    condition: sum_1h > 10
-    channels: [slack]
+## CLI
 
-  - name: error_rate
-    metric: gen_ai.client.errors
-    condition: ratio_15m > 0.05
-    channels: [telegram]
 ```
-
-Channels: Telegram, Slack webhook, HTTP webhook, email (SMTP).
-
-### Trace-to-dataset export
-
-Convert production Jaeger traces into test cases:
-
-```bash
-npx toad-eye export-trace <trace_id> --output ./evals/
+npx toad-eye init [--force]     Scaffold Docker Compose + observability configs
+npx toad-eye up                 Start the stack
+npx toad-eye down               Stop the stack
+npx toad-eye status             Show running services and URLs
+npx toad-eye demo               Send mock LLM traffic to Grafana
+npx toad-eye export-trace <id>  Export a Jaeger trace to toad-eval YAML
 ```
 
-### Cloud mode (coming soon)
-
-No Docker needed. Send telemetry to toad-eye cloud with one line:
-
-```typescript
-initObservability({
-  serviceName: "my-app",
-  apiKey: "toad_xxxxxxxx",
-  instrument: ["openai"],
-});
-```
-
-Self-hosted mode remains the default. Cloud mode activates automatically when `apiKey` is set.
-
-</details>
-
-## Grafana dashboards
-
-8 pre-built dashboards auto-provisioned on `npx toad-eye init`:
-
-| Dashboard              | What it shows                                                    |
-| ---------------------- | ---------------------------------------------------------------- |
-| **Overview**           | Request rate, error rate, latency p50/p95, cost, totals          |
-| **Cost Breakdown**     | Spend by provider/model, daily trend, projected monthly          |
-| **Latency Analysis**   | p50/p95/p99 by model, distribution histogram, TTFT               |
-| **Error Drill-down**   | Error rate by provider/model, error vs success                   |
-| **Model Comparison**   | Latency vs cost vs error rate vs throughput per model            |
-| **FinOps Attribution** | Cost by team/user/feature, projected spend, what-if table        |
-| **Provider Health**    | Provider status (healthy/degraded/down), uptime, error breakdown |
-| **Agent Workflow**     | Steps per query, tool usage frequency, step type breakdown       |
+## Architecture
 
-## Subpath imports
+![toad-eye architecture](https://raw.githubusercontent.com/vola-trebla/toad-eye/main/demo/toad_eye_architecture_v2.svg)
 
-Advanced modules are available as separate entry points to keep your bundle lean:
+## Imports
 
 ```typescript
 import { initObservability, traceLLMCall } from "toad-eye"; // core
-import { AlertManager } from "toad-eye/alerts"; // alerting
+import { toadEyeMiddleware } from "toad-eye/mcp"; // MCP server middleware
+import { AlertManager } from "toad-eye/alerts"; // alerting engine
 import { createDriftMonitor } from "toad-eye/drift"; // semantic drift
 import { exportTrace } from "toad-eye/export"; // trace → YAML
 import { ToadEyeAISpanProcessor, withToadEye } from "toad-eye/vercel"; // Vercel AI SDK
 ```
 
-> Everything is also re-exported from the main `"toad-eye"` entry point for convenience.
-
-## CLI
-
-| Command                          | Description                                     |
-| -------------------------------- | ----------------------------------------------- |
-| `npx toad-eye init [--force]`    | Scaffold Docker Compose + observability configs |
-| `npx toad-eye up`                | Start the stack                                 |
-| `npx toad-eye down`              | Stop the stack                                  |
-| `npx toad-eye status`            | Show running services and URLs                  |
-| `npx toad-eye demo`              | Send mock LLM traffic to see data in Grafana    |
-| `npx toad-eye export-trace <id>` | Export a Jaeger trace to toad-eval YAML         |
-
-## Architecture
-
-![toad-eye architecture](https://raw.githubusercontent.com/vola-trebla/toad-eye/main/demo/toad_eye_architecture_v2.svg)
-
-## What it tracks
-
-<details>
-<summary>Metrics & span attributes</summary>
-
-### Metrics (OTel GenAI semconv)
-
-| Metric                                       | Type      | Description                 |
-| -------------------------------------------- | --------- | --------------------------- |
-| `gen_ai.client.operation.duration`           | Histogram | Request latency (ms)        |
-| `gen_ai.client.time_to_first_token`          | Histogram | TTFT for streaming (ms)     |
-| `gen_ai.client.request.cost`                 | Histogram | Cost per request (USD)      |
-| `gen_ai.client.token.usage`                  | Counter   | Total tokens consumed       |
-| `gen_ai.client.requests`                     | Counter   | Total requests              |
-| `gen_ai.client.errors`                       | Counter   | Total failed requests       |
-| `gen_ai.agent.steps_per_query`               | Histogram | Agent steps per query       |
-| `gen_ai.agent.tool_usage`                    | Counter   | Agent tool invocations      |
-| `gen_ai.toad_eye.guard.evaluations`          | Counter   | Guard evaluations per rule  |
-| `gen_ai.toad_eye.guard.would_block`          | Counter   | Would-have-blocked per rule |
-| `gen_ai.toad_eye.semantic_drift`             | Histogram | Drift from baseline (0..1)  |
-| `gen_ai.toad_eye.budget.exceeded`            | Counter   | Budget exceeded events      |
-| `gen_ai.toad_eye.budget.blocked`             | Counter   | LLM calls blocked by budget |
-| `gen_ai.toad_eye.budget.downgraded`          | Counter   | LLM calls downgraded        |
-| `gen_ai.toad_eye.response.empty`             | Counter   | Empty/whitespace responses  |
-| `gen_ai.toad_eye.response.latency_per_token` | Histogram | Generation speed (ms/token) |
-
-### Span attributes (OTel GenAI semconv)
-
-| Attribute                          | Description                                     |
-| ---------------------------------- | ----------------------------------------------- |
-| `gen_ai.operation.name`            | `chat`, `invoke_agent`, `execute_tool`          |
-| `gen_ai.provider.name`             | `anthropic`, `gemini`, `openai`                 |
-| `gen_ai.request.model`             | Model identifier                                |
-| `gen_ai.usage.input_tokens`        | Tokens in the prompt                            |
-| `gen_ai.usage.output_tokens`       | Tokens in the completion                        |
-| `gen_ai.request.temperature`       | Temperature parameter                           |
-| `gen_ai.agent.name`                | Agent name (for agent spans)                    |
-| `gen_ai.agent.id`                  | Agent identifier                                |
-| `gen_ai.tool.name`                 | Tool name for execute_tool spans                |
-| `gen_ai.tool.type`                 | `function`, `extension`, `retrieval`, `builtin` |
-| `gen_ai.toad_eye.cost`             | Cost in USD                                     |
-| `gen_ai.toad_eye.agent.step.type`  | ReAct step: think/act/observe/answer/handoff    |
-| `gen_ai.toad_eye.agent.handoff.to` | Target agent for handoff steps                  |
-| `gen_ai.toad_eye.agent.loop_count` | Agent loop iterations (observe→think)           |
-| `gen_ai.toad_eye.guard.mode`       | Guard mode: shadow/enforce                      |
-| `session.id`                       | Session identifier (if configured)              |
-
-</details>
-
 ## Services
 
-| Service        | URL                                                     |
-| -------------- | ------------------------------------------------------- |
-| Grafana        | [localhost:3100](http://localhost:3100) (admin / admin) |
-| Jaeger UI      | [localhost:16686](http://localhost:16686)               |
-| Prometheus     | [localhost:9090](http://localhost:9090)                 |
-| OTel Collector | [localhost:4318](http://localhost:4318)                 |
+After `npx toad-eye up`:
 
-## OTel GenAI semconv
+```
+Grafana         http://localhost:3100   (admin / admin)
+Jaeger UI       http://localhost:16686
+Prometheus      http://localhost:9090
+OTel Collector  http://localhost:4318
+```
 
-toad-eye follows [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) for span names, attributes, and metrics. Traces are natively compatible with Jaeger, Arize Phoenix, SigNoz, Datadog, Langfuse, and other OTel backends.
+## OTel Compatibility
 
-See [COMPATIBILITY.md](https://github.com/vola-trebla/toad-eye/blob/main/packages/instrumentation/COMPATIBILITY.md) for a full backend support matrix.
+toad-eye follows [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). Traces work natively with Jaeger, Datadog, Grafana Tempo, Honeycomb, SigNoz, Arize Phoenix, and any OTel-compatible backend.
 
-Set `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` to disable deprecated attribute aliases and emit only the latest conventions.
+Full metrics and span attribute reference: [COMPATIBILITY.md](https://github.com/vola-trebla/toad-eye/blob/main/packages/instrumentation/COMPATIBILITY.md)
 
-## Tech stack
+## Tech Stack
 
-- TypeScript, OpenTelemetry SDK 2.x, OTel GenAI semantic conventions
-- Hono (demo server + cloud ingestion server)
-- Docker Compose (Prometheus, Jaeger, Grafana, OTel Collector)
-- Vitest (252+ tests)
+TypeScript · OpenTelemetry SDK 2.x · Hono · Docker Compose (Prometheus, Jaeger, Grafana, OTel Collector) · Vitest (273+ tests)

package/dist/alerts/conditions.js

@@ -5,6 +5,12 @@ const METRIC_MAP = {
     "gen_ai.client.requests": "gen_ai_client_requests_total",
     "gen_ai.client.errors": "gen_ai_client_errors_total",
     "gen_ai.client.token.usage": "gen_ai_client_token_usage_total",
+    // MCP metrics
+    "gen_ai.mcp.tool.calls": "gen_ai_mcp_tool_calls_total",
+    "gen_ai.mcp.tool.errors": "gen_ai_mcp_tool_errors_total",
+    "gen_ai.mcp.tool.duration": "gen_ai_mcp_tool_duration",
+    "gen_ai.mcp.resource.reads": "gen_ai_mcp_resource_reads_total",
+    "gen_ai.mcp.tool.callers": "gen_ai_mcp_tool_callers_total",
     // Legacy names (backward compat)
     "llm.request.cost": "gen_ai_client_request_cost_USD_sum",
     "llm.request.duration": "gen_ai_client_operation_duration_milliseconds",
@@ -15,6 +21,9 @@ const METRIC_MAP = {
 function toPromMetric(metric) {
     return METRIC_MAP[metric] ?? metric.replace(/\./g, "_");
 }
+function isMcpMetric(metric) {
+    return metric.startsWith("gen_ai.mcp.");
+}
 /**
  * Parse condition string into structured form.
  *
@@ -65,7 +74,10 @@ function buildPromQL(metric, operator, window) {
 }
 function buildTopModelsQuery(metric, window) {
     const m = toPromMetric(metric);
-    return `topk(5, sum by (gen_ai_request_model) (increase(${m}[${window}])))`;
+    const groupBy = isMcpMetric(metric)
+        ? "gen_ai_tool_name"
+        : "gen_ai_request_model";
+    return `topk(5, sum by (${groupBy}) (increase(${m}[${window}])))`;
 }
 function matches(value, comparator, threshold) {
     switch (comparator) {
@@ -94,10 +106,13 @@ async function queryTopModels(prometheusUrl, metric, window) {
     const res = await fetch(url);
     if (!res.ok)
         return [];
+    const labelKey = isMcpMetric(metric)
+        ? "gen_ai_tool_name"
+        : "gen_ai_request_model";
     const data = (await res.json());
     return data.data.result
         .map((r) => ({
-        model: r.metric.gen_ai_request_model ?? "unknown",
+        model: r.metric[labelKey] ?? "unknown",
         value: parseFloat(r.value[1]),
     }))
         .sort((a, b) => b.value - a.value);
@@ -113,9 +128,19 @@ async function evalP95Baseline(prometheusUrl, metric, window, baselineWindow) {
     return ((currentP95 - baselineP95) / baselineP95) * 100;
 }
 /** Evaluate error ratio: errors / requests over window */
-async function evalErrorRatio(prometheusUrl, window) {
-    const errors = await queryScalar(prometheusUrl, `sum(increase(gen_ai_client_errors_total[${window}]))`);
-    const requests = await queryScalar(prometheusUrl, `sum(increase(gen_ai_client_requests_total[${window}]))`);
+async function evalErrorRatio(prometheusUrl, metric, window) {
+    let errorMetric;
+    let requestMetric;
+    if (isMcpMetric(metric)) {
+        errorMetric = "gen_ai_mcp_tool_errors_total";
+        requestMetric = "gen_ai_mcp_tool_calls_total";
+    }
+    else {
+        errorMetric = "gen_ai_client_errors_total";
+        requestMetric = "gen_ai_client_requests_total";
+    }
+    const errors = await queryScalar(prometheusUrl, `sum(increase(${errorMetric}[${window}]))`);
+    const requests = await queryScalar(prometheusUrl, `sum(increase(${requestMetric}[${window}]))`);
     if (requests === 0)
         return 0;
     return errors / requests;
@@ -128,7 +153,7 @@ export async function evaluateCondition(prometheusUrl, rule) {
             value = await evalP95Baseline(prometheusUrl, rule.metric, parsed.window, parsed.baselineWindow);
         }
         else if (parsed.operator === "ratio") {
-            value = await evalErrorRatio(prometheusUrl, parsed.window);
+            value = await evalErrorRatio(prometheusUrl, rule.metric, parsed.window);
         }
         else {
             const promql = buildPromQL(rule.metric, parsed.operator, parsed.window);

package/dist/alerts/grafana.js

@@ -2,8 +2,13 @@ export async function postGrafanaAnnotation(grafanaUrl, alert, apiKey) {
     const headers = {
         "Content-Type": "application/json",
     };
-    if (apiKey)
+    if (apiKey) {
         headers["Authorization"] = `Bearer ${apiKey}`;
+    }
+    else {
+        // Fall back to Basic Auth with default Grafana credentials
+        headers["Authorization"] = `Basic ${btoa("admin:admin")}`;
+    }
     const res = await fetch(`${grafanaUrl}/api/annotations`, {
         method: "POST",
         headers,