elasticdash-sdk 0.2.0

package/docs/matchers.md

@@ -0,0 +1,173 @@
+# Test Matchers
+
+ElasticDash SDK provides AI-specific matchers for asserting on workflow traces.
+
+## Overview
+
+All matchers work with `expect(ctx.trace)` after importing the test setup:
+
+```ts
+import '../node_modules/elasticdash-sdk/dist/test-setup.js'
+import { expect } from 'expect'
+
+aiTest('my test', async (ctx) => {
+  // ... run your workflow
+  expect(ctx.trace).toHaveLLMStep({ model: 'gpt-4' })
+})
+```
+
+---
+
+## `toHaveLLMStep(config?)`
+
+Assert the trace contains at least one LLM step matching the given config. All fields are optional and combined with AND logic.
+
+```ts
+expect(ctx.trace).toHaveLLMStep({ model: 'gpt-4' })
+expect(ctx.trace).toHaveLLMStep({ contains: 'order confirmed' })       // searches prompt + completion
+expect(ctx.trace).toHaveLLMStep({ promptContains: 'order status' })    // searches prompt only
+expect(ctx.trace).toHaveLLMStep({ outputContains: 'order confirmed' }) // searches completion only
+expect(ctx.trace).toHaveLLMStep({ provider: 'openai' })
+expect(ctx.trace).toHaveLLMStep({ provider: 'openai', promptContains: 'order status' })
+expect(ctx.trace).toHaveLLMStep({ promptContains: 'retry', times: 3 })      // exactly 3 matching steps
+expect(ctx.trace).toHaveLLMStep({ provider: 'openai', minTimes: 2 })        // at least 2 matching steps
+expect(ctx.trace).toHaveLLMStep({ outputContains: 'error', maxTimes: 1 })   // at most 1 matching step
+```
+
+### Configuration Options
+
+| Field | Description |
+|---|---|
+| `model` | Exact model name match (e.g. `'gpt-4o'`) |
+| `contains` | Substring match across prompt + completion (case-insensitive) |
+| `promptContains` | Substring match in prompt only (case-insensitive) |
+| `outputContains` | Substring match in completion only (case-insensitive) |
+| `provider` | Provider name: `'openai'`, `'gemini'`, or `'grok'` |
+| `times` | Exact match count (fails unless exactly this many steps match) |
+| `minTimes` | Minimum match count (steps matching must be ≥ this value) |
+| `maxTimes` | Maximum match count (steps matching must be ≤ this value) |
+
+---
+
+## `toCallTool(toolName)`
+
+Assert the trace contains a tool call with the given name.
+
+```ts
+expect(ctx.trace).toCallTool('chargeCard')
+```
+
+---
+
+## `toMatchSemanticOutput(expected, options?)`
+
+LLM-judged semantic match of combined LLM output vs. the expected string. Defaults to OpenAI GPT-4 with `OPENAI_API_KEY`.
+
+```ts
+expect(ctx.trace).toMatchSemanticOutput('attack stat', {
+  provider: 'claude',               // 'openai' (default) | 'claude' | 'gemini' | 'grok'
+  model: 'claude-3-opus-20240229',  // overrides default model for the provider
+  sdk: myClaudeClient,              // optional SDK instance (uses its chat/messages API)
+})
+
+// Minimal, using default OpenAI model
+expect(ctx.trace).toMatchSemanticOutput('order confirmed')
+
+// OpenAI-compatible endpoint (e.g., Moonshot/Kimi) via baseURL + apiKey
+expect(ctx.trace).toMatchSemanticOutput('order confirmed', {
+  provider: 'openai',
+  model: 'kimi-k2-turbo-preview',
+  apiKey: process.env.KIMI_API_KEY,
+  baseURL: 'https://api.moonshot.ai/v1',
+})
+```
+
+Environment keys by provider: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY` (or `GOOGLE_API_KEY`), `GROK_API_KEY`.
+
+---
+
+## `toEvaluateOutputMetric(config)`
+
+Evaluate one LLM step's prompt or result using an LLM and assert a numeric metric condition in the range 0.0–1.0. 
+
+Defaults: `target='result'`, `condition='atLeast 0.7'`, `provider='openai'`, `model='gpt-4'`.
+
+```ts
+// Evaluate the last LLM result with your own prompt; default condition atLeast 0.7
+expect(ctx.trace).toEvaluateOutputMetric({
+  evaluationPrompt: 'Rate how well this answers the user question.',
+})
+
+// Check a specific step (3rd LLM prompt), target the prompt text, require >= 0.8 via Claude
+expect(ctx.trace).toEvaluateOutputMetric({
+  evaluationPrompt: 'Score coherence of this prompt between 0 and 1.',
+  target: 'prompt',
+  nth: 3,
+  condition: { atLeast: 0.8 },
+  provider: 'claude',
+  model: 'claude-3-opus-20240229',
+})
+
+// Custom comparator: score must be < 0.3
+expect(ctx.trace).toEvaluateOutputMetric({
+  evaluationPrompt: 'Rate hallucination risk (0=none, 1=high).',
+  condition: { lessThan: 0.3 },
+})
+```
+
+### Configuration Options
+
+- `evaluationPrompt` (required): your scoring instructions; model is asked to return only a number between 0 and 1.
+- `target`: `'result'` (default) or `'prompt'`. Evaluates that text only.
+- `index` / `nth`: pick which LLM step to score (0-based or 1-based). Defaults to the last LLM step.
+- `condition`: one of `greaterThan`, `lessThan`, `atLeast`, `atMost`, `equals`; default is `{ atLeast: 0.7 }`.
+- `provider` / `model` / `sdk` / `apiKey` / `baseURL`: same shape as `toMatchSemanticOutput`.
+
+---
+
+## `toHaveCustomStep(config?)`
+
+Assert a recorded custom step (RAG/code/fixed/custom) matches filters.
+
+```ts
+expect(ctx.trace).toHaveCustomStep({ kind: 'rag', name: 'pokemon-search' })
+expect(ctx.trace).toHaveCustomStep({ tag: 'sort:asc' })
+expect(ctx.trace).toHaveCustomStep({ contains: 'pikachu' })
+expect(ctx.trace).toHaveCustomStep({ resultContains: '25' })
+expect(ctx.trace).toHaveCustomStep({ kind: 'rag', minTimes: 1, maxTimes: 2 })
+```
+
+---
+
+## `toHavePromptWhere(config)`
+
+Filter prompts, then assert additional constraints. Example: "all prompts containing A must also contain B".
+
+```ts
+// Prompts that contain "order" must also contain "confirmed"
+expect(ctx.trace).toHavePromptWhere({
+  filterContains: 'order',
+  requireContains: 'confirmed',
+})
+
+// Prompts containing "retry" must NOT contain "cancel"
+expect(ctx.trace).toHavePromptWhere({
+  filterContains: 'retry',
+  requireNotContains: 'cancel',
+})
+
+// And control counts on the filtered subset
+expect(ctx.trace).toHavePromptWhere({
+  filterContains: 'order',
+  requireContains: 'confirmed',
+  minTimes: 1,
+  maxTimes: 3,
+})
+
+// Check a specific prompt position (1-based nth or 0-based index)
+expect(ctx.trace).toHavePromptWhere({
+  filterContains: 'order',
+  requireContains: 'confirmed',
+  nth: 3, // the 3rd prompt among those containing "order"
+})
+```

package/docs/observability_contract.md

@@ -0,0 +1,192 @@
+# Observability SDK Contract
+
+This document describes the event types the SDK sends and the portal (remote rerun queue) contract between the SDK and the backend.
+
+---
+
+## SDK Event Types
+
+### Events the SDK Sends
+
+| `type` | `name` | When | Key fields |
+|--------|--------|------|------------|
+| `ai` | Model name (e.g. `gpt-4o`) | Every `wrapAI` call | `input`, `output`, `usage`, `durationMs`, `streamed` |
+| `tool` | Tool name (e.g. `searchDB`) | Every `wrapTool` call | `input`, `output`, `durationMs`, `streamed` |
+| `side_effect` | `__heartbeat__` | Every 30s (configurable) | `input.sessionId`, `output.uptime` |
+| `side_effect` | `__session_end__` | On `shutdownObservability()` | `input.sessionId`, `output.uptime` |
+
+### Special Events (do not display in trace UI)
+
+- `__heartbeat__` — update session liveness, do not store as event
+- `__session_end__` — mark session as ended, do not store as event
+
+### Streamed Events
+
+When `streamed === true`:
+- `output` is `null`
+- `streamRaw` contains the full buffered text of the stream
+- Display `streamRaw` as the output in the UI
+
+### Error Events
+
+When a tool or AI call throws:
+- `output` is `{ "error": "Error message string" }`
+- `durationMs` reflects time until failure
+- Display with error styling in the UI
+
+---
+
+## Portal (Remote Rerun Queue) Contract
+
+The SDK's `elasticdash portal` command starts an HTTP server that the backend can push rerun tasks to. The backend also needs endpoints to receive results.
+
+### SDK Portal Endpoints (hosted on user's machine, default port 4574)
+
+These endpoints are served by the SDK. The backend calls them.
+
+#### `POST /api/portal/tasks` — Push a single rerun task
+
+**Request:**
+```json
+{
+  "taskId": "task-uuid-from-backend",
+  "type": "tool",
+  "name": "searchDB",
+  "input": { "query": "pikachu" },
+  "metadata": { "testGroupId": 42, "expectationIds": [1, 2, 3] }
+}
+```
+
+For AI tasks:
+```json
+{
+  "taskId": "task-uuid-from-backend",
+  "type": "ai",
+  "name": "gpt-4o",
+  "input": { "messages": [{ "role": "user", "content": "Hello" }] },
+  "model": "gpt-4o",
+  "provider": "openai",
+  "modelParameters": { "temperature": 0.7, "max_tokens": 512 },
+  "metadata": { "testGroupId": 42 }
+}
+```
+
+**Response:** `202 Accepted`
+```json
+{ "ok": true, "taskId": "task-uuid-from-backend", "position": 3 }
+```
+
+**Auth:** `Authorization: Bearer <api_key>` (validated if portal was started with `--api-key`)
+
+#### `POST /api/portal/tasks/batch` — Push multiple tasks
+
+**Request:**
+```json
+{ "tasks": [ /* PortalTask[] */ ] }
+```
+
+**Response:** `202 Accepted`
+```json
+{ "ok": true, "tasks": [{ "taskId": "...", "position": 1 }, { "taskId": "...", "position": 2 }] }
+```
+
+#### `GET /api/portal/status` — Health check
+
+**Response:**
+```json
+{
+  "ok": true,
+  "queueLength": 5,
+  "processing": "task-uuid-123",
+  "completed": 12,
+  "failed": 1
+}
+```
+
+#### `DELETE /api/portal/tasks/:taskId` — Cancel a pending task
+
+**Response:** `200` if removed, `404` if not found or already processing.
+
+---
+
+### Backend Endpoints (needed for portal to work)
+
+These endpoints must be implemented on the backend. The SDK calls them.
+
+#### `POST /api/portal/register` — Portal registration
+
+Called by the SDK when `elasticdash portal` starts.
+
+**Request:**
+```json
+{
+  "portalUrl": "http://localhost:4574"
+}
+```
+
+**Auth:** `Authorization: Bearer <api_key>`
+
+**Response:** `200 OK`
+```json
+{ "ok": true }
+```
+
+The backend should store this portal URL and use it to push tasks. The registration should be scoped to the project resolved from the API key.
+
+#### `POST /api/portal/results/:taskId` — Receive task result
+
+Called by the SDK after each task completes (success or failure).
+
+**Request:**
+```json
+{
+  "taskId": "task-uuid-from-backend",
+  "ok": true,
+  "output": "The search returned 3 results for pikachu...",
+  "durationMs": 245,
+  "usage": {
+    "inputTokens": 150,
+    "outputTokens": 45,
+    "totalTokens": 195
+  },
+  "metadata": { "testGroupId": 42, "expectationIds": [1, 2, 3] }
+}
+```
+
+For failed tasks:
+```json
+{
+  "taskId": "task-uuid-from-backend",
+  "ok": false,
+  "output": null,
+  "error": "Tool not found: \"searchDB\". Available tools: fetchData, sendEmail",
+  "durationMs": 0,
+  "metadata": { "testGroupId": 42 }
+}
+```
+
+**Auth:** `Authorization: Bearer <api_key>`
+
+**Response:** `200 OK`
+```json
+{ "ok": true }
+```
+
+---
+
+### Error Results Reference
+
+The SDK sends these error patterns:
+
+| Error pattern | Meaning |
+|--------------|---------|
+| `Tool not found: "<name>". Available tools: ...` | Tool doesn't exist in `ed_tools.ts` |
+| `Cannot find ed_tools.ts/js in workspace root.` | No tools module in the project |
+| `Unsupported AI provider: "<name>"` | Unknown provider string |
+| `Missing API key for provider "<name>". Expected environment variable: <VAR>` | LLM API key not configured |
+| `AI task input is empty; cannot execute.` | No prompt could be extracted from input |
+| `AI execution failed: <message>` | LLM API call failed (rate limit, network, invalid model) |
+| `Tool subprocess produced no output.` | Subprocess exited without result |
+| `Failed to spawn tool subprocess: <message>` | Could not start subprocess |
+| `Missing tool name on task.` | Task had no `name` field |
+| `Unknown task type: <type>` | Task type was neither `tool` nor `ai` |

package/docs/observability_mode.md

@@ -0,0 +1,195 @@
+# Observability Mode
+
+Observability mode turns the ElasticDash SDK into an always-on tracing instrument. When enabled, every `wrapTool` and `wrapAI` call automatically records and streams trace events to your ElasticDash backend — no test runner required.
+
+## Quick Start
+
+### Option 1: Programmatic (recommended)
+
+Add a single call at your app's entry point:
+
+```typescript
+// instrumentation.ts (Next.js) or server entry point
+import { initObservability } from 'elasticdash-sdk/http'
+
+const obs = initObservability({
+  serverUrl: 'https://server.elasticdash.com',
+})
+
+// On shutdown (optional — auto-registered on process exit)
+// await obs.shutdown()
+```
+
+### Option 2: Environment Variables Only
+
+If your app already uses `wrapTool` / `wrapAI`, just set the env vars:
+
+```bash
+ELASTICDASH_API_URL=https://server.elasticdash.com \
+ELASTICDASH_API_KEY=ed_key_xxx \
+node server.js
+```
+
+### Option 3: CLI
+
+```bash
+elasticdash observe --server https://server.elasticdash.com
+```
+
+## Configuration
+
+### `initObservability(options?)`
+
+| Option | Env Variable | Default | Description |
+|--------|-------------|---------|-------------|
+| `serverUrl` | `ELASTICDASH_API_URL` | *required* | ElasticDash backend URL |
+| `apiKey` | `ELASTICDASH_API_KEY` | — | Project authentication token |
+| `sessionId` | `ELASTICDASH_SESSION_ID` | auto-generated UUID | Session identifier |
+| `batchIntervalMs` | — | `2000` | How often to flush events (ms) |
+| `maxBatchSize` | — | `50` | Max events per batch before auto-flush |
+| `heartbeatIntervalMs` | — | `30000` | Heartbeat interval (ms) |
+| `sampleRate` | — | `1.0` | Fraction of events to send (0.0–1.0) |
+| `redactKeys` | — | `[]` | Object keys to redact from input/output |
+
+### Return Value
+
+```typescript
+interface ObservabilityHandle {
+  sessionId: string        // The active session ID
+  shutdown: () => Promise<void>  // Graceful shutdown
+}
+```
+
+## Grouping Events by Workflow with `startTrace()`
+
+By default, the SDK discovers workflow names from `ed_workflows.ts`. If exactly one workflow is exported, its name is used as the traceId prefix automatically (e.g. `chatStreamHandler::1712851200000::a1b2c3d4`). If multiple workflows are exported, the traceId defaults to `unknown-workflow` until you call `startTrace()`.
+
+To explicitly group events under a specific workflow, call `startTrace(workflowName)` at the start of each request handler:
+
+```typescript
+import { startTrace } from 'elasticdash-sdk/http'
+
+// In your route handler, before any wrapTool/wrapAI calls:
+startTrace('chatStreamHandler')
+```
+
+This sets the traceId to `chatStreamHandler::1712851200000::a1b2c3d4`, so all subsequent `wrapTool` / `wrapAI` / `wrapDB` / fetch calls in that request are grouped under the `chatStreamHandler` workflow in the dashboard.
+
+**Important:** Call `startTrace()` before any tool/AI calls execute. If you're using a streaming `ReadableStream`, place it inside the `start()` callback:
+
+```typescript
+const stream = new ReadableStream({
+  async start(controller) {
+    startTrace('chatStreamHandler')
+    // ... workflow logic with wrapTool/wrapAI calls ...
+  },
+})
+```
+
+### Alternative: `wrapWorkflow()`
+
+If you control the workflow function directly, you can use `wrapWorkflow()` instead. It calls `startTrace()` automatically before each invocation:
+
+```typescript
+import { wrapWorkflow } from 'elasticdash-sdk/http'
+
+export const chatStreamHandler = wrapWorkflow('chatStreamHandler', async (input) => {
+  // All tool/AI calls here are automatically grouped under 'chatStreamHandler'
+  const result = await fetchUser(input.userId)
+  return generateReply(result)
+})
+```
+
+## How It Works
+
+1. **`initObservability()`** creates an `ObservabilityContext` and installs the AI interceptor
+2. Every `wrapTool(name, fn)` and `wrapAI(model, fn)` call checks for this context
+3. When active, the wrapper executes the real function, captures timing/input/output, and enqueues the event
+4. Events are batched and flushed to `POST /api/observability/events` on the backend
+5. A heartbeat event is sent every 30 seconds (configurable) so the backend knows the service is alive
+6. On process exit, remaining events are flushed and a `session_end` event is sent
+
+### Event Flow
+
+```
+wrapTool("searchDB", fn) called
+  → fn(...args) executes normally
+  → WorkflowEvent created: { type: 'tool', name: 'searchDB', input, output, durationMs }
+  → pushTelemetryEvent(event)
+  → TelemetryBatcher.enqueue(event)
+  → Batch flushed every 2s → POST /api/observability/events
+```
+
+## Sampling & Redaction
+
+### Sampling
+
+For high-throughput services, use `sampleRate` to reduce event volume:
+
+```typescript
+initObservability({
+  serverUrl: 'https://server.elasticdash.com',
+  sampleRate: 0.1, // Send only 10% of events
+})
+```
+
+### Redaction
+
+Strip sensitive fields from input/output before sending:
+
+```typescript
+initObservability({
+  serverUrl: 'https://server.elasticdash.com',
+  redactKeys: ['apiKey', 'password', 'ssn', 'credit_card'],
+})
+```
+
+This deep-clones and replaces matching keys (case-insensitive) with `"[REDACTED]"` before serialization.
+
+## Batching & Reliability
+
+Events are not sent individually — they are buffered and flushed in batches:
+
+- **Interval flush**: every `batchIntervalMs` (default 2 seconds)
+- **Size flush**: when the buffer reaches `maxBatchSize` (default 50 events)
+- **Exit flush**: on `beforeExit`, `SIGTERM`, or `SIGINT`
+
+Failed flushes are retried with exponential backoff (1s, 2s, 4s) up to 3 times. After max retries, events are dropped to prevent memory leaks.
+
+## Graceful Shutdown
+
+Shutdown happens automatically on process exit signals. For manual control (e.g., serverless functions):
+
+```typescript
+import { shutdownObservability } from 'elasticdash-sdk/http'
+
+// In your cleanup handler
+await shutdownObservability()
+```
+
+This flushes all buffered events, sends a `session_end` marker, and clears the context.
+
+## Debug Logging
+
+To see SDK internal logs (telemetry push status, batch flush counts, etc.), set:
+
+```bash
+ELASTICDASH_DEBUG=1 node server.js
+```
+
+All internal `console.log` calls are gated behind this flag and produce no output by default.
+
+## Comparison with Test Mode
+
+| Feature | Test Mode | Observability Mode |
+|---------|-----------|-------------------|
+| Requires test runner | Yes | No |
+| Mocking support | Yes | No |
+| Step replay/freezing | Yes | No |
+| Event delivery | Fire-and-forget per event | Batched with retry |
+| Sampling | No | Yes |
+| Redaction | No | Yes |
+| Heartbeat | No | Yes |
+| Graceful shutdown | No | Yes |
+
+Both modes use the same `wrapTool` / `wrapAI` wrappers — the SDK detects which context is active and routes events accordingly.