toad-eye 2.2.0 → 2.4.0

package/README.md

@@ -1,4 +1,4 @@
-# toad-eye
+# toad-eye 🐸👁️‍🗨️🤖
 
 ![TypeScript](https://img.shields.io/badge/TypeScript-3178C6?logo=typescript&logoColor=white)
 ![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-000000?logo=opentelemetry&logoColor=white)
@@ -10,9 +10,9 @@
 ![CI](https://github.com/vola-trebla/toad-eye/actions/workflows/ci.yml/badge.svg)
 ![License](https://img.shields.io/badge/License-ISC-blue)
 
-OpenTelemetry-based observability toolkit for LLM systems.
+**LLM observability in 3 commands. Auto-instrument, trace, alert, save money.**
 
-Auto-instrument any LLM SDK (including streaming), get traces, metrics, cost tracking, budget guards, agent observability, guardrail monitoring, and 6 Grafana dashboards out of the box. Self-hosted or cloud mode.
+You don't know what your LLMs cost, how slow they are, or why they fail. toad-eye gives you complete visibility with zero manual instrumentation.
 
 ![toad-eye demo](https://raw.githubusercontent.com/vola-trebla/toad-eye/main/demo/toad-eye-demo-gifski.gif)
 
@@ -25,48 +25,57 @@ npx toad-eye up         # start Grafana + Prometheus + Jaeger
 npx toad-eye demo       # send mock LLM traffic, see data immediately
 ```
 
-Open [localhost:3100](http://localhost:3100) (Grafana, admin/admin) to see your dashboards.
-
-### Cloud mode
-
-No Docker needed. Send telemetry to toad-eye cloud with one line:
-
 ```typescript
+import { initObservability } from "toad-eye";
+
 initObservability({
   serviceName: "my-app",
-  apiKey: "toad_xxxxxxxx",
-  instrument: ["openai"],
+  instrument: ["openai", "anthropic"],
 });
+
+// That's it. Every SDK call is auto-traced — including streaming.
 ```
 
-Self-hosted mode remains the default. Cloud mode activates automatically when `apiKey` is set.
+Open [localhost:3100](http://localhost:3100) (Grafana, admin/admin) to see your dashboards.
 
-## Auto-instrumentation
+## What you get
+
+| Feature                  | Description                                                                                |
+| ------------------------ | ------------------------------------------------------------------------------------------ |
+| **Auto-instrumentation** | OpenAI, Anthropic, Gemini, Vercel AI SDK — regular + streaming                             |
+| **8 Grafana dashboards** | Overview, Cost, Latency, Errors, Model Comparison, FinOps, Provider Health, Agent Workflow |
+| **Cost tracking**        | Per-request USD cost, daily totals, projected monthly spend                                |
+| **Budget guards**        | Daily/per-user/per-model spend limits — warn, block, or auto-downgrade                     |
+| **Multi-agent tracing**  | ReAct steps, handoffs, loop detection, maxSteps guard                                      |
+| **Shadow guardrails**    | Record what _would_ be blocked without blocking — tune thresholds on live traffic          |
+| **Quality proxy**        | Empty response rate, latency per token — zero-dependency quality metrics                   |
+| **Semantic drift**       | Detect silent quality degradation via embedding comparison                                 |
+| **Privacy controls**     | Built-in PII redaction (email, SSN, CC, phone), hashing, content masking                   |
+| **Tail sampling**        | Keep 100% errors + slow traces, sample healthy traffic via OTel Collector                  |
+| **Alerting**             | Cost spikes, latency anomalies, error rate — via Telegram, Slack, email, webhook           |
+| **FinOps attribution**   | Break down costs by team, user, feature, environment                                       |
+| **TTFT metric**          | Time To First Token for streaming — separate from total duration                           |
+| **Trace export**         | Convert production Jaeger traces into regression test cases                                |
+
+## Auto-instrumentation 🤖
 
 Zero-code instrumentation for popular LLM SDKs. No wrappers needed.
 
 ```typescript
-import { initObservability } from "toad-eye";
-
 initObservability({
   serviceName: "my-app",
   instrument: ["openai", "anthropic"],
 });
-
-// That's it. Every SDK call is auto-traced.
-const result = await openai.chat.completions.create({
-  model: "gpt-4o",
-  messages: [{ role: "user", content: "Hello" }],
-});
 ```
 
-**Supported SDKs:** OpenAI (`openai`), Anthropic (`@anthropic-ai/sdk`), Google GenAI (`@google/generative-ai`)
+**Supported SDKs:** OpenAI (`openai`), Anthropic (`@anthropic-ai/sdk`), Google GenAI (`@google/generative-ai`), Vercel AI SDK (`ai`)
 
 Both regular and **streaming** calls are fully instrumented — spans, metrics, and cost tracking work transparently for `stream: true`.
 
-### Manual instrumentation
+Vercel AI SDK uses a SpanProcessor (not monkey-patching) — add `instrument: ['ai']` and `experimental_telemetry: withToadEye()` to your calls.
 
-For custom providers or fine-grained control:
+<details>
+<summary>Manual instrumentation (custom providers)</summary>
 
 ```typescript
 import { initObservability, traceLLMCall } from "toad-eye";
@@ -79,9 +88,33 @@ const result = await traceLLMCall(
 );
 ```
 
+</details>
+
+## Budget guards
+
+Prevent cost overruns at runtime. Three modes: warn, block, or auto-downgrade to a cheaper model.
+
+```typescript
+initObservability({
+  serviceName: "my-app",
+  budgets: {
+    daily: 50, // $50/day max
+    perUser: 5, // $5/day per user
+    perModel: { "gpt-4o": 30 }, // $30/day on GPT-4o
+  },
+  onBudgetExceeded: "block", // throws ToadBudgetExceededError
+});
+```
+
+| Mode        | Behavior                                            |
+| ----------- | --------------------------------------------------- |
+| `warn`      | Log warning, continue normally                      |
+| `block`     | Throw `ToadBudgetExceededError` before LLM call     |
+| `downgrade` | Call `downgradeCallback` to switch to cheaper model |
+
 ## Agent observability
 
-Structured tracing for ReAct agents (think / act / observe / answer) as nested OpenTelemetry spans.
+Structured tracing for ReAct agents with multi-agent support, handoffs, and loop detection.
 
 ```typescript
 import { traceAgentQuery } from "toad-eye";
@@ -111,17 +144,37 @@ const result = await traceAgentQuery(
 );
 ```
 
-Metrics: steps per query (histogram), tool usage frequency (counter per tool name).
+## FinOps attribution
+
+Track costs by team, user, feature, or any business dimension:
+
+```typescript
+initObservability({
+  serviceName: "my-app",
+  attributes: { team: "checkout", environment: "production" },
+});
+
+await traceLLMCall(
+  {
+    provider: "openai",
+    model: "gpt-4o",
+    prompt: "Summarize order",
+    attributes: { userId: "user-123", feature: "order-summary" },
+  },
+  () => callLLM(),
+);
+```
+
+<details>
+<summary>More features</summary>
 
-## Shadow guardrails
+### Shadow guardrails
 
-Validate LLM responses without blocking them. Record what _would_ have been blocked, then decide when to enforce.
+Record what _would_ have been blocked without blocking. Tune thresholds on live traffic.
 
 ```typescript
 import { recordGuardResult } from "toad-eye";
 
-// toad-guard validates, toad-eye records
-const result = guard.validate(response);
 recordGuardResult({
   mode: "shadow",
   passed: false,
@@ -130,49 +183,20 @@ recordGuardResult({
 });
 ```
 
-Metrics: `guard.evaluations` (total checks), `guard.would_block` (would-have-blocked count) per rule name.
+### Semantic drift monitoring
 
-## Semantic drift monitoring
-
-Detect silent LLM quality degradation by comparing current responses to a saved baseline via embeddings.
+Detect silent LLM quality degradation by comparing responses to a saved baseline via embeddings.
 
 ```typescript
-import { createDriftMonitor, saveBaseline } from "toad-eye";
+import { createDriftMonitor } from "toad-eye";
 
 const monitor = createDriftMonitor({
   embedding: { provider: "openai", apiKey: process.env.OPENAI_API_KEY! },
   baselinePath: "./baseline.json",
-  sampleRate: 10, // check every 10th response
+  sampleRate: 10,
 });
 
-// In your LLM call handler:
-const drift = await monitor.check(response, "openai", "gpt-4o");
-// drift = 0 → identical to baseline
-// drift > 0.3 → significant deviation
-```
-
-## Trace-to-dataset export
-
-Convert production Jaeger traces into test cases. Production failure becomes a regression test.
-
-```bash
-npx toad-eye export-trace <trace_id> --output ./evals/
-```
-
-Generates YAML in toad-eval format with auto-generated assertions (max length, refusal detection, JSON validation).
-
-## Features
-
-### Built-in cost tracking
-
-Automatic cost calculation for major models (GPT-4o, Claude, Gemini) based on token usage. Custom pricing for enterprise contracts:
-
-```typescript
-import { setCustomPricing } from "toad-eye";
-
-setCustomPricing({
-  "my-fine-tuned-model": { inputPer1M: 5, outputPer1M: 15 },
-});
+monitor.checkInBackground(response, "openai", "gpt-4o");
 ```
 
 ### Privacy controls
@@ -180,28 +204,36 @@ setCustomPricing({
 ```typescript
 initObservability({
   serviceName: "my-app",
+  redactDefaults: true, // built-in PII patterns (email, SSN, CC, phone)
   recordContent: false, // disable prompt/completion recording
   hashContent: true, // SHA-256 hash instead of plain text
-  redactPatterns: [/\b\S+@\S+\.\S+\b/g], // regex redaction
+  salt: "my-secret", // prevent rainbow table attacks
+  auditMasking: true, // log what was masked (debug only)
+  redactPatterns: [/custom-pattern/g], // additional regex redaction
 });
 ```
 
 ### Session tracking
 
-Group traces by conversation session:
-
 ```typescript
 initObservability({
   serviceName: "my-app",
   sessionId: "static-session-id",
-  // or dynamic:
-  sessionExtractor: () => getCurrentSessionId(),
+  sessionExtractor: () => getCurrentSessionId(), // or dynamic
 });
 ```
 
-### Alerting
+### Built-in cost tracking
+
+```typescript
+import { setCustomPricing } from "toad-eye";
 
-Cost spikes, latency anomalies, error rate monitoring via YAML config:
+setCustomPricing({
+  "my-fine-tuned-model": { inputPer1M: 5, outputPer1M: 15 },
+});
+```
+
+### Alerting
 
 ```yaml
 alerts:
@@ -210,78 +242,52 @@ alerts:
     condition: sum_1h > 10
     channels: [slack]
 
-  - name: latency_anomaly
-    metric: gen_ai.client.operation.duration
-    condition: p95_pct_5m_7d > 50
-    channels: [slack]
-
   - name: error_rate
     metric: gen_ai.client.errors
     condition: ratio_15m > 0.05
-    channels: [slack]
+    channels: [telegram]
 ```
 
-Delivery channels: Telegram, Slack webhook, generic HTTP webhook, email (SMTP).
+Channels: Telegram, Slack webhook, HTTP webhook, email (SMTP).
 
-### Budget guards
+### Trace-to-dataset export
 
-Prevent cost overruns at runtime. Three modes: warn, block, or auto-downgrade to a cheaper model.
+Convert production Jaeger traces into test cases:
 
-```typescript
-initObservability({
-  serviceName: "my-app",
-  budgets: {
-    daily: 50, // $50/day max
-    perUser: 5, // $5/day per user
-    perModel: { "gpt-4o": 30 }, // $30/day on GPT-4o
-  },
-  onBudgetExceeded: "block", // throws ToadBudgetExceededError
-});
+```bash
+npx toad-eye export-trace <trace_id> --output ./evals/
 ```
 
-| Mode        | Behavior                                            |
-| ----------- | --------------------------------------------------- |
-| `warn`      | Log warning, continue normally                      |
-| `block`     | Throw `ToadBudgetExceededError` before LLM call     |
-| `downgrade` | Call `downgradeCallback` to switch to cheaper model |
-
-## FinOps attribution
+### Cloud mode (coming soon)
 
-Track costs by team, user, feature, or any business dimension:
+No Docker needed. Send telemetry to toad-eye cloud with one line:
 
 ```typescript
-// Global attributes — applied to all spans/metrics
 initObservability({
   serviceName: "my-app",
-  attributes: { team: "checkout", environment: "production" },
+  apiKey: "toad_xxxxxxxx",
+  instrument: ["openai"],
 });
-
-// Per-request attributes — override or extend global
-await traceLLMCall(
-  {
-    provider: "openai",
-    model: "gpt-4o",
-    prompt: "Summarize order",
-    attributes: { userId: "user-123", feature: "order-summary" },
-  },
-  () => callLLM(),
-);
 ```
 
-## Grafana dashboards
+Self-hosted mode remains the default. Cloud mode activates automatically when `apiKey` is set.
+
+</details>
 
-6 pre-built dashboards auto-provisioned on `npx toad-eye init`:
+## Grafana dashboards
 
-| Dashboard              | What it shows                                             |
-| ---------------------- | --------------------------------------------------------- |
-| **Overview**           | Request rate, error rate, latency p50/p95, cost, totals   |
-| **Cost Breakdown**     | Spend by provider/model, daily trend, projected monthly   |
-| **Latency Analysis**   | p50/p95/p99 by model, distribution histogram              |
-| **Error Drill-down**   | Error rate by provider/model, error vs success            |
-| **Model Comparison**   | Latency vs cost vs error rate vs throughput per model     |
-| **FinOps Attribution** | Cost by team/user/feature, projected spend, what-if table |
+8 pre-built dashboards auto-provisioned on `npx toad-eye init`:
 
-All dashboards have template variables for filtering (`$provider`, `$model`, `$team`, `$feature`).
+| Dashboard              | What it shows                                                    |
+| ---------------------- | ---------------------------------------------------------------- |
+| **Overview**           | Request rate, error rate, latency p50/p95, cost, totals          |
+| **Cost Breakdown**     | Spend by provider/model, daily trend, projected monthly          |
+| **Latency Analysis**   | p50/p95/p99 by model, distribution histogram, TTFT               |
+| **Error Drill-down**   | Error rate by provider/model, error vs success                   |
+| **Model Comparison**   | Latency vs cost vs error rate vs throughput per model            |
+| **FinOps Attribution** | Cost by team/user/feature, projected spend, what-if table        |
+| **Provider Health**    | Provider status (healthy/degraded/down), uptime, error breakdown |
+| **Agent Workflow**     | Steps per query, tool usage frequency, step type breakdown       |
 
 ## CLI
 
@@ -300,40 +306,48 @@ All dashboards have template variables for filtering (`$provider`, `$model`, `$t
 
 ## What it tracks
 
+<details>
+<summary>Metrics & span attributes</summary>
+
 ### Metrics (OTel GenAI semconv)
 
-| Metric                              | Type      | Description                         |
-| ----------------------------------- | --------- | ----------------------------------- |
-| `gen_ai.client.operation.duration`  | Histogram | Request latency (ms)                |
-| `gen_ai.client.request.cost`        | Histogram | Cost per request (USD)              |
-| `gen_ai.client.token.usage`         | Counter   | Total tokens consumed               |
-| `gen_ai.client.requests`            | Counter   | Total requests                      |
-| `gen_ai.client.errors`              | Counter   | Total failed requests               |
-| `gen_ai.agent.steps_per_query`      | Histogram | Agent steps per query               |
-| `gen_ai.agent.tool_usage`           | Counter   | Agent tool invocations by tool name |
-| `gen_ai.toad_eye.guard.evaluations` | Counter   | Guard evaluations per rule          |
-| `gen_ai.toad_eye.guard.would_block` | Counter   | Would-have-blocked per rule         |
-| `gen_ai.toad_eye.semantic_drift`    | Histogram | Semantic drift from baseline (0..1) |
-| `gen_ai.toad_eye.budget.exceeded`   | Counter   | Budget limit exceeded events        |
-| `gen_ai.toad_eye.budget.blocked`    | Counter   | LLM calls blocked by budget         |
-| `gen_ai.toad_eye.budget.downgraded` | Counter   | LLM calls downgraded by budget      |
-
-All metrics labeled with `gen_ai.provider.name` and `gen_ai.request.model`.
+| Metric                                       | Type      | Description                 |
+| -------------------------------------------- | --------- | --------------------------- |
+| `gen_ai.client.operation.duration`           | Histogram | Request latency (ms)        |
+| `gen_ai.client.time_to_first_token`          | Histogram | TTFT for streaming (ms)     |
+| `gen_ai.client.request.cost`                 | Histogram | Cost per request (USD)      |
+| `gen_ai.client.token.usage`                  | Counter   | Total tokens consumed       |
+| `gen_ai.client.requests`                     | Counter   | Total requests              |
+| `gen_ai.client.errors`                       | Counter   | Total failed requests       |
+| `gen_ai.agent.steps_per_query`               | Histogram | Agent steps per query       |
+| `gen_ai.agent.tool_usage`                    | Counter   | Agent tool invocations      |
+| `gen_ai.toad_eye.guard.evaluations`          | Counter   | Guard evaluations per rule  |
+| `gen_ai.toad_eye.guard.would_block`          | Counter   | Would-have-blocked per rule |
+| `gen_ai.toad_eye.semantic_drift`             | Histogram | Drift from baseline (0..1)  |
+| `gen_ai.toad_eye.budget.exceeded`            | Counter   | Budget exceeded events      |
+| `gen_ai.toad_eye.budget.blocked`             | Counter   | LLM calls blocked by budget |
+| `gen_ai.toad_eye.budget.downgraded`          | Counter   | LLM calls downgraded        |
+| `gen_ai.toad_eye.response.empty`             | Counter   | Empty/whitespace responses  |
+| `gen_ai.toad_eye.response.latency_per_token` | Histogram | Generation speed (ms/token) |
 
 ### Span attributes
 
-| Attribute                    | Description                          |
-| ---------------------------- | ------------------------------------ |
-| `gen_ai.provider.name`       | `anthropic`, `gemini`, `openai`      |
-| `gen_ai.request.model`       | Model identifier                     |
-| `gen_ai.usage.input_tokens`  | Tokens in the prompt                 |
-| `gen_ai.usage.output_tokens` | Tokens in the completion             |
-| `gen_ai.request.temperature` | Temperature parameter                |
-| `gen_ai.toad_eye.cost`       | Cost in USD                          |
-| `gen_ai.agent.step.type`     | Agent step: think/act/observe/answer |
-| `gen_ai.agent.tool.name`     | Tool name for agent act steps        |
-| `gen_ai.toad_eye.guard.mode` | Guard mode: shadow/enforce           |
-| `session.id`                 | Session identifier (if configured)   |
+| Attribute                    | Description                                  |
+| ---------------------------- | -------------------------------------------- |
+| `gen_ai.provider.name`       | `anthropic`, `gemini`, `openai`              |
+| `gen_ai.request.model`       | Model identifier                             |
+| `gen_ai.usage.input_tokens`  | Tokens in the prompt                         |
+| `gen_ai.usage.output_tokens` | Tokens in the completion                     |
+| `gen_ai.request.temperature` | Temperature parameter                        |
+| `gen_ai.toad_eye.cost`       | Cost in USD                                  |
+| `gen_ai.agent.step.type`     | Agent step: think/act/observe/answer/handoff |
+| `gen_ai.agent.tool.name`     | Tool name for agent act steps                |
+| `gen_ai.agent.handoff.to`    | Target agent for handoff steps               |
+| `gen_ai.agent.loop_count`    | Agent loop iterations (observe→think)        |
+| `gen_ai.toad_eye.guard.mode` | Guard mode: shadow/enforce                   |
+| `session.id`                 | Session identifier (if configured)           |
+
+</details>
 
 ## Services
 
@@ -349,4 +363,4 @@ All metrics labeled with `gen_ai.provider.name` and `gen_ai.request.model`.
 - TypeScript, OpenTelemetry SDK 2.x, OTel GenAI semantic conventions
 - Hono (demo server + cloud ingestion server)
 - Docker Compose (Prometheus, Jaeger, Grafana, OTel Collector)
-- Vitest (154 tests)
+- Vitest (143+ tests)

package/dist/agent.d.ts

@@ -1,41 +1,44 @@
-import type { AgentStepInput } from "./types/index.js";
+import type { AgentStepInput, AgentQueryOptions, AgentQueryInput } from "./types/index.js";
 /**
- * Record a single agent step as a child span (ReAct pattern: think → act → observe → answer).
- *
- * Creates a short-lived span under the current active context. Call this inside
- * {@link traceAgentQuery} (preferred) or any other active span (e.g. traceLLMCall).
- *
- * - Respects `recordContent` config — content is omitted when recording is disabled.
- * - For `act` steps with a `toolName`, automatically increments the `agent.tool_usage` metric.
+ * Record a single agent step as a child span (ReAct pattern).
+ * Supports: think, act, observe, answer, handoff.
  */
 declare function traceAgentStep(input: AgentStepInput): void;
 /**
- * Wrap an entire agent query in a parent span, producing structured ReAct traces.
+ * Wrap an entire agent query in a parent span with ReAct tracing.
  *
- * The callback receives a `step` function to record individual steps.
- * When the callback completes, the wrapper automatically records the
- * `agent.steps_per_query` histogram metric with the total step count.
+ * Accepts either a string query (backward-compatible) or an object with
+ * agent metadata per OTel GenAI spec (agentName, agentId).
  *
- * Span hierarchy in Jaeger:
- * ```
- * [parent] agent.query "Is anything dangerous near Earth?"
- *   ├── agent.step.think   "I need to check asteroids"
- *   ├── agent.step.act     tool=near-earth-asteroids
- *   ├── agent.step.observe "7 asteroids found"
- *   └── agent.step.answer  "7 asteroids passing safely..."
- * ```
+ * Supports multi-agent via nesting — child traceAgentQuery calls
+ * automatically become child spans in Jaeger.
+ *
+ * Loop detection: counts observe→think transitions. Records loop count
+ * as span attribute. Emits warning when maxSteps exceeded.
  *
  * @example
  * ```ts
- * const result = await traceAgentQuery("Is anything dangerous near Earth?", async (step) => {
- *   step({ type: "think", stepNumber: 1, content: "I need to check asteroids" });
- *   const data = await callTool("near-earth-asteroids");
- *   step({ type: "act", stepNumber: 2, toolName: "near-earth-asteroids" });
- *   step({ type: "observe", stepNumber: 3, content: `${data.length} asteroids found` });
- *   step({ type: "answer", stepNumber: 4, content: "7 asteroids passing safely..." });
- *   return { answer: "7 asteroids passing safely..." };
+ * // Simple form (backward-compatible)
+ * await traceAgentQuery('Is anything dangerous?', async (step) => { ... });
+ *
+ * // Object form with agent metadata (OTel GenAI spec)
+ * await traceAgentQuery(
+ *   { query: 'Is anything dangerous?', agentName: 'space-monitor', agentId: 'agent-001' },
+ *   async (step) => { ... }
+ * );
+ *
+ * // Multi-agent: orchestrator delegates to specialist
+ * await traceAgentQuery({ query: 'Analyze data', agentName: 'orchestrator' }, async (step) => {
+ *   step({ type: 'think', stepNumber: 1, content: 'Need domain expert' });
+ *   step({ type: 'handoff', stepNumber: 2, toAgent: 'specialist', handoffReason: 'domain expertise' });
+ *
+ *   const result = await traceAgentQuery({ query: 'Analyze', agentName: 'specialist' }, async (step) => {
+ *     step({ type: 'act', stepNumber: 1, toolName: 'analyze', toolType: 'function' });
+ *     step({ type: 'answer', stepNumber: 2, content: 'analysis complete' });
+ *     return { answer: 'done' };
+ *   });
  * });
  * ```
  */
-export declare function traceAgentQuery<T>(query: string, fn: (step: (input: AgentStepInput) => void) => Promise<T>): Promise<T>;
+export declare function traceAgentQuery<T>(queryOrInput: string | AgentQueryInput, fn: (step: (input: AgentStepInput) => void) => Promise<T>, options?: AgentQueryOptions): Promise<T>;
 export { traceAgentStep };