elasticdash-sdk 0.2.0

package/docs/test-writing-guidelines.md

@@ -0,0 +1,444 @@
+# ElasticDash SDK Test Writing Guidelines
+
+This guide covers both testing approaches available in `elasticdash-sdk`:
+
+1. **`defineTest` + benchmarks** — VCR-style CI testing with recorded fixtures (recommended for CI/CD)
+2. **`aiTest` + matchers** — live workflow testing with the existing runner
+
+## Prerequisites
+
+- Node.js >= 20
+- `elasticdash-sdk` installed as a dev dependency
+- For `defineTest`: a TypeScript toolchain (e.g. `tsx`) if writing `.ts` test files
+- Optional: `ELASTICDASH_API_KEY` for uploading results to the dashboard
+
+---
+
+## Part 1: CI/CD Testing with `defineTest` (Record & Replay)
+
+### Overview
+
+ElasticDash CI testing follows a VCR-style record-and-replay pattern:
+
+1. **Record** — run your workflow locally with `ELASTICDASH_CAPTURE_TRACE=1` to capture a trace fixture
+2. **Write** — create an `ed_tests.ts` file with `defineTest()` calls that reference the fixture
+3. **Run** — execute `npx ed ed-test` to replay the fixture, measure performance, and compare against benchmarks
+4. **Upload** — results are automatically sent to the ElasticDash backend for historical analysis
+
+### Quick Start
+
+```bash
+# 1. Record a trace
+ELASTICDASH_CAPTURE_TRACE=1 node your-workflow.js
+
+# 2. Write a test (see below)
+
+# 3. Run tests locally
+npx ed ed-test --no-upload
+
+# 4. Run in CI with upload
+ELASTICDASH_API_KEY=your-key ELASTICDASH_API_URL=https://server.elasticdash.com npx ed ed-test
+```
+
+### Recording Traces
+
+Set the `ELASTICDASH_CAPTURE_TRACE=1` environment variable before running any workflow that uses the SDK's `wrapTool`/`wrapAI` interceptors or `runWorkflow()`.
+
+```bash
+ELASTICDASH_CAPTURE_TRACE=1 tsx your-workflow.ts
+```
+
+This writes a JSON trace file to `.ed_traces/` in the current directory:
+
+```
+.ed_traces/2026-04-19T14-23-07_a7f3.json
+```
+
+The trace contains every tool call and AI call the workflow made, including inputs, outputs, timing, and token usage. Sensitive fields (`authorization`, `api_key`, `password`, `secret`, `token`) are automatically scrubbed to `[REDACTED]`.
+
+Add `.ed_traces/` to your `.gitignore` if you don't want fixtures committed, or commit them if you want deterministic CI replay.
+
+#### Trace file structure
+
+A trace file captures the full workflow execution. Here's what one looks like (simplified from a real trace):
+
+```json
+{
+  "trace_id": "2026-04-20T00-57-44_800f",
+  "created_at": "2026-04-20T00:57:44.856Z",
+  "sdk_version": "unknown",
+  "workflow": {
+    "name": "chatStreamHandler",
+    "input": null,
+    "output": null
+  },
+  "steps": [
+    {
+      "step_id": "ai_call_0",
+      "type": "ai_call",
+      "name": "gpt-4o",
+      "input": {
+        "messages": [
+          { "role": "system", "content": "You are an expert that refines user queries..." },
+          { "role": "user", "content": "What's the attack of Groudon?" }
+        ],
+        "temperature": 0.5,
+        "max_tokens": 4096
+      },
+      "output": "Refined Query: \"What is the attack stat of Groudon?\"\nLanguage: \"en\"\nConcepts: [\"Groudon\", \"attack stat\"]\n...",
+      "started_at": "2026-04-20T00:57:26.425Z",
+      "ended_at": "2026-04-20T00:57:28.626Z",
+      "duration_ms": 2201,
+      "tokens": { "input": 1726, "output": 57, "total": 1783 }
+    },
+    {
+      "step_id": "tool_call_0",
+      "type": "tool_call",
+      "name": "queryRefinement",
+      "input": {
+        "userInput": "What's the attack of Groudon?",
+        "userToken": ""
+      },
+      "output": {
+        "refinedQuery": "What is the attack stat of Groudon?",
+        "entities": ["groudon details", "pokemon stats"],
+        "intentType": "FETCH"
+      },
+      "started_at": "2026-04-20T00:57:26.424Z",
+      "ended_at": "2026-04-20T00:57:28.627Z",
+      "duration_ms": 2203,
+      "tokens": null
+    }
+  ]
+}
+```
+
+Key fields to note:
+- **`step_id`** — Used in `defineTest` to target a specific step (e.g. `"ai_call_0"`, `"tool_call_0"`)
+- **`type`** — Either `"ai_call"` or `"tool_call"`, must match the `target.type` in your test
+- **`input`** — The recorded input/prompt. This is what gets overridden when you use the custom `input` field in `defineTest`
+- **`output`** — The recorded response, used for benchmark comparisons (`output_contains`, `llm_judge`, etc.)
+- **`duration_ms`** / **`tokens`** — The measurements compared against `max_duration_ms` and `max_tokens_total` benchmarks
+
+### Writing ed_tests
+
+Create a file named `ed_tests.ts` (or `ed_tests.js`) anywhere in your project. Multiple test files in different directories are all discovered automatically.
+
+```ts
+import { defineTest } from "elasticdash-sdk";
+import { runCheckoutWorkflow } from "./workflows";
+
+defineTest({
+  name: "checkout_happy_path_latency",
+  trace: "../.ed_traces/2026-04-19T14-23-07_a7f3.json",
+  target: { type: "tool_call", step_id: "tool_call_2" },
+  benchmarks: { max_duration_ms: 2000 },
+  run: async () => {
+    await runCheckoutWorkflow({ userId: "premium_123" });
+  },
+});
+
+defineTest({
+  name: "checkout_token_budget",
+  trace: "../.ed_traces/2026-04-19T14-23-07_a7f3.json",
+  target: { type: "ai_call", step_id: "ai_call_0" },
+  benchmarks: { max_tokens_total: 5000 },
+  run: async () => {
+    await runCheckoutWorkflow({ userId: "premium_123" });
+  },
+});
+```
+
+#### Field reference
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | string | yes | Unique test ID, used for historical comparison |
+| `trace` | string | yes | Path to trace file, relative to the test file's directory |
+| `target` | object | yes | `{ type: "tool_call" \| "ai_call", step_id: "tool_call_0" }` |
+| `benchmarks` | object | yes | At least one of `max_duration_ms` or `max_tokens_total` |
+| `input` | unknown or function | no | Custom input that overrides the trace's recorded input. Can be a static value or an async function for dynamic resolution |
+| `run` | function | yes* | Async function that invokes the workflow. Receives the resolved input (custom or from trace) as its argument. *Required for execution |
+| `timeout_ms` | number | no | Per-test timeout (default: 60000ms) |
+
+The `step_id` values (`tool_call_0`, `ai_call_1`, etc.) are found in the trace file's `steps` array. Open the trace JSON to find the step you want to target.
+
+#### Custom input
+
+By default, the test's input is read from the trace fixture. You can override it with a static value or an async function to source the prompt from anywhere:
+
+```ts
+// Static override — input is passed to run()
+defineTest({
+  name: "checkout_custom_prompt",
+  trace: "../.ed_traces/trace.json",
+  target: { type: "ai_call", step_id: "ai_call_0" },
+  benchmarks: { max_tokens_total: 5000 },
+  input: { prompt: "Custom prompt text" },
+  run: async (input) => { await runCheckoutWorkflow(input); },
+});
+
+// Dynamic resolution (database, API, environment, etc.)
+defineTest({
+  name: "checkout_dynamic_prompt",
+  trace: "../.ed_traces/trace.json",
+  target: { type: "ai_call", step_id: "ai_call_0" },
+  benchmarks: { max_tokens_total: 5000 },
+  input: async () => {
+    const row = await db.query("SELECT prompt FROM prompts WHERE id = 42");
+    return { prompt: row.prompt };
+  },
+  run: async (input) => { await runCheckoutWorkflow(input); },
+});
+```
+
+The custom input only affects the reported input in test results and uploads — replay matching still uses the trace fixture's recorded data.
+
+### Running Tests Locally
+
+```bash
+# Run all tests
+npx ed ed-test
+
+# Skip upload (local iteration)
+npx ed ed-test --no-upload
+
+# Filter by test name pattern
+npx ed ed-test --filter "checkout_*"
+
+# Stop on first failure
+npx ed ed-test --fail-fast
+
+# JSON output (for scripting)
+npx ed ed-test --reporter json
+
+# JUnit XML (for CI systems)
+npx ed ed-test --reporter junit
+```
+
+#### Exit codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | All tests passed |
+| 1 | One or more tests failed |
+| 3 | Configuration error (no tests found, bad files, missing fixtures) |
+| 4 | Upload failed, but tests completed successfully |
+
+### CI Integration
+
+#### GitHub Actions
+
+```yaml
+- name: Run ElasticDash tests
+  env:
+    ELASTICDASH_API_KEY: ${{ secrets.ELASTICDASH_API_KEY }}
+    ELASTICDASH_API_URL: https://server.elasticdash.com
+  run: npx ed ed-test
+```
+
+#### GitLab CI
+
+```yaml
+ed-test:
+  script:
+    - npx ed ed-test
+  variables:
+    ELASTICDASH_API_KEY: $ELASTICDASH_API_KEY
+    ELASTICDASH_API_URL: https://server.elasticdash.com
+```
+
+#### CircleCI
+
+```yaml
+- run:
+    name: Run ElasticDash tests
+    command: npx ed ed-test
+    environment:
+      ELASTICDASH_API_KEY: ${ELASTICDASH_API_KEY}
+      ELASTICDASH_API_URL: https://server.elasticdash.com
+```
+
+Git metadata (commit SHA, branch, PR number) is auto-detected from CI environment variables for GitHub Actions, GitLab CI, CircleCI, and Buildkite. For local runs, `git rev-parse` is used as a fallback.
+
+### Environment Variables
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `ELASTICDASH_API_KEY` | Project API key for authentication | — |
+| `ELASTICDASH_API_URL` | Backend server URL | — |
+| `ELASTICDASH_CAPTURE_TRACE` | Set to `1` to record a trace fixture | — |
+| `ELASTICDASH_ACCEPT_RERUNS` | Set to `false`, `0`, or `no` to reject rerun/trigger requests from the server. Useful when the SDK is installed across multiple projects and only one should process reruns. | Enabled (accepts reruns) |
+| `ELASTICDASH_PORTAL_PORT` | Port for the portal server | `4574` |
+| `ELASTICDASH_ALLOWED_ORIGINS` | Comma-separated allowed CORS origins for the portal | — |
+
+### How Replay Works
+
+During test execution, every `wrapTool` and `wrapAI` call is intercepted. Instead of calling the real service, the SDK looks up the matching step in the trace fixture and returns the recorded output.
+
+**Match key:** Each call is matched by `<type>::<name>::<hash(input)>` where the input is canonicalized (sorted keys, normalized whitespace, rounded floats). Calls are matched in order of occurrence.
+
+**Replay miss:** If the workflow makes a call that doesn't match any step in the fixture, the test fails immediately with a clear diagnostic:
+
+```
+FAIL  checkout_happy_path_latency
+      → replay miss: tool_call::stripe.customers.retrieve
+```
+
+This means the fixture is stale or the workflow has changed. Re-record the trace with `ELASTICDASH_CAPTURE_TRACE=1`.
+
+### Benchmarks
+
+Phase 3 supports absolute threshold benchmarks:
+
+- `max_duration_ms` — the step's recorded duration must not exceed this value
+- `max_tokens_total` — the step's total token usage must not exceed this value (ai_call only)
+
+Measurements come from the fixture's recorded values, not live timing. This makes tests deterministic — they always produce the same result for the same fixture.
+
+### Upload & Dashboard
+
+When `ELASTICDASH_API_KEY` and `ELASTICDASH_API_URL` are set, results are uploaded to `POST /api/v1/test-runs/create` after all tests complete.
+
+If the upload fails after retries, the payload is saved to `.ed_traces/failed_uploads/<run_id>.json` for manual inspection or retry. Upload failures don't change the test exit code unless all tests passed (exit code 4).
+
+### Troubleshooting
+
+**Replay miss errors:**
+The fixture doesn't contain a step matching what the workflow called. Common causes:
+- Workflow code changed since the trace was recorded
+- Non-deterministic inputs (timestamps, UUIDs) in tool call arguments
+
+Fix: re-record the trace with `ELASTICDASH_CAPTURE_TRACE=1`.
+
+**"test has no run function":**
+The `run` field is required for test execution. Add an async function that invokes your workflow.
+
+**Git detection shows "unknown":**
+No `.git` directory or `git` binary found. In CI, ensure the repo is checked out. Locally, run from within a git repository.
+
+**Upload auth errors:**
+Check that `ELASTICDASH_API_KEY` is set and valid. The API key must be scoped to the correct project.
+
+---
+
+## Part 2: Live Workflow Testing with `aiTest` (Existing Approach)
+
+The `aiTest` + matchers approach runs workflows live and asserts on the trace using custom matchers. This is useful for development-time testing but not for deterministic CI replay.
+
+### Test anatomy
+
+- Import test setup once per file: `import 'elasticdash-sdk/dist/test-setup.js'`
+- Each test receives `ctx` with `ctx.trace` to inspect recorded LLM/tool/custom steps.
+- Matchers live on `expect` (already registered by `test-setup`).
+- Files end with `.ai.test.ts` and use the global `aiTest(name, fn)`.
+
+### Hooks
+
+- `beforeAll(fn)` / `afterAll(fn)`: run once per file before/after all tests.
+- `beforeEach(fn)` / `afterEach(fn)`: run before/after every test in the file. `afterEach` still runs when the test fails.
+
+### Useful matchers (quick reference)
+
+- `toHaveLLMStep(config)`: Assert LLM calls match model/provider/prompt/output filters.
+- `toCallTool(name)`: Assert a tool call occurred.
+- `toHaveCustomStep(config)`: Assert custom (RAG/code/fixed/custom) steps.
+- `toHavePromptWhere(config)`: Filter prompts by substring, then require/include/exclude content (with optional nth/index positional checks).
+- `toMatchSemanticOutput(expected, options?)`: LLM-judged semantic match over combined LLM outputs.
+- `toEvaluateOutputMetric(config)`: LLM-scored numeric metric (0-1) on a specific LLM step's prompt or result, with threshold comparisons.
+
+### Patterns and examples
+
+#### Validate the order of steps in a workflow
+
+```ts
+aiTest('prompts occur in order', async (ctx) => {
+  await runWorkflow()
+
+  expect(ctx.trace).toHavePromptWhere({
+    filterContains: 'Goal Completion Validator',
+    nth: 1,
+  })
+
+  expect(ctx.trace).toHavePromptWhere({
+    filterContains: "User's Ultimate Goal:",
+    nth: 2,
+  })
+})
+```
+
+#### Validate fetched data from RAG/APIs
+
+```ts
+aiTest('RAG includes required source', async (ctx) => {
+  await runWorkflow()
+
+  expect(ctx.trace).toHaveCustomStep({
+    kind: 'rag',
+    contains: 'pokemon_stats',
+    resultContains: 'base_stat',
+  })
+})
+```
+
+#### Check LLM output
+
+```ts
+aiTest('planner output returns attack stat', async (ctx) => {
+  await runWorkflow()
+
+  expect(ctx.trace).toHaveLLMStep({
+    outputContains: 'attack stat of Pikachu',
+  })
+})
+```
+
+#### LLM-as-judge metric scoring
+
+```ts
+aiTest('plan is actionable', async (ctx) => {
+  await runWorkflow()
+
+  await expect(ctx.trace).toEvaluateOutputMetric({
+    evaluationPrompt:
+      'Score 0-1: is this execution plan concrete and directly executable? '
+      + '1.0 = concrete SQL with specific tables/columns; 0.0 = vague/placeholder.',
+    target: 'result',
+    nth: 2,
+    condition: { atLeast: 0.7 },
+    provider: 'claude',
+    model: 'claude-3-opus-20240229',
+  })
+})
+```
+
+### Tips
+
+- Always `await` async matchers (`toMatchSemanticOutput`, `toEvaluateOutputMetric`).
+- Use `nth`/`index` in `toHavePromptWhere` to avoid false positives when multiple prompts contain similar text.
+- Keep prompts/results concise in tests; log the trace when debugging: `console.log(ctx.trace.getLLMSteps())`.
+
+### Running aiTest tests
+
+```bash
+# Run all *.ai.test.ts files
+elasticdash test
+
+# Run in a subdir
+elasticdash test examples/
+
+# Single file
+elasticdash run path/to/file.ai.test.ts
+```
+
+### Minimal scaffold
+
+```ts
+import 'elasticdash-sdk/dist/test-setup.js'
+import { expect } from 'expect'
+
+aiTest('example', async (ctx) => {
+  await runWorkflow()
+  expect(ctx.trace).toHaveLLMStep({ provider: 'openai' })
+})
+```

package/docs/tools.md

@@ -0,0 +1,165 @@
+# Tool Recording and Replay
+
+ElasticDash automatically records and traces tool calls during workflow execution, providing replay and debugging capabilities.
+
+For HTTP response streaming capture (SSE/NDJSON fetch flows), see `README.md` and `docs/quickstart.md#capture-streaming-flows`. That behavior is handled by the HTTP interceptor and is separate from manual tool instrumentation in this document.
+
+## Manual Tool Recording
+
+For tools outside the normal import flow, or if you need explicit success/error logging control, use a resilient `recordToolCall` pattern where tracing is isolated from the main service path:
+
+```ts
+import { runSelectQuery } from 'path/to/tool/calls'
+
+export const dataService = async (input: any) => {
+  const { query } = input as { query: string }
+
+  return await runSelectQuery(query)
+    .then(async (result: any) => {
+      try {
+        const { recordToolCall } = await import('elasticdash-sdk')
+        recordToolCall('dataService', input, result)
+      } catch {
+        // trace logging errors must not break the main service
+      }
+      return result
+    })
+    .catch(async (error: any) => {
+      try {
+        const { recordToolCall } = await import('elasticdash-sdk')
+        recordToolCall('dataService', input, error)
+      } catch {
+        // trace logging errors must not break the main service
+      }
+      throw error
+    })
+}
+```
+
+Why this pattern is recommended for manual instrumentation:
+
+- Build-time or mainstream runtime contexts may not have tracing APIs available.
+- Dynamic import plus nested `try/catch` keeps `recordToolCall` best-effort.
+- Service success/failure behavior is preserved even when trace logging fails.
+
+If your runtime may execute outside ElasticDash worker context, dynamic import keeps behavior safe:
+
+```ts
+try {
+  const { recordToolCall } = await import('elasticdash-sdk')
+  recordToolCall('dataService', input, result)
+} catch {
+  // no-op outside elasticdash runtime
+}
+```
+
+**Note:** Manual recording is best-effort trace logging. Keep the same resilient pattern (dynamic import + nested `try/catch`) across all tools so trace logging never interrupts core service execution.
+
+## Calling Tools from Workflows
+
+**Always call tool functions from `ed_tools.ts` (or `ed_tools.js`), not from their source code locations.**
+
+In your workflows, import and use tools through the instrumented export:
+
+```ts
+// ✅ Correct - calls the traced version from ed_tools.ts
+import { dataService } from './ed_tools'
+
+export const checkoutWorkflow = async (orderId: string) => {
+  const orderData = await dataService({ query: `SELECT * FROM orders WHERE id = ${orderId}` })
+  // ... rest of workflow
+}
+```
+
+Not directly from the source file:
+```ts
+// ❌ Wrong - bypasses tracing instrumentation
+import { runSelectQuery } from './services/dataService'
+
+export const checkoutWorkflow = async (orderId: string) => {
+  const orderData = await runSelectQuery(`SELECT * FROM orders WHERE id = ${orderId}`)
+  // ... rest of workflow
+}
+```
+
+**Why this matters:**
+- Tool calls through `ed_tools.ts` are automatically traced and recorded
+- Direct imports bypass the `recordToolCall` instrumentation
+- Dashboard trace replay requires tools to be called through `ed_tools.ts`
+- LLM agents calling tools will record the call with the `name` from `ed_tools.ts`, so using the same import ensures name matching
+
+## Tool Function Compatibility (`ed_tools.ts/js`)
+
+Exports in `ed_tools.ts/js` should be plain callable functions that take serializable input and return serializable output.
+
+- Export directly callable functions
+- Use JSON-serializable args/results (object, array, string, number, boolean, or `null`)
+- Avoid exporting framework request/response handlers directly (for example Next.js `NextRequest`/`NextResponse` route handlers)
+
+Compatible export example:
+
+```ts
+export async function chargeCard(input: { amount: number; token: string }) {
+  return { success: true, transactionId: 'txn-123' }
+}
+```
+
+Not directly compatible as a tool export:
+
+```ts
+// Next.js route handler style
+export async function POST(req: NextRequest): Promise<NextResponse> {
+  return NextResponse.json({ ok: true })
+}
+```
+
+If your app uses framework handlers, keep `ed_tools.ts/js` as a plain callable boundary and invoke your framework-specific code behind that boundary.
+
+## Recording Without Passing `ctx.trace`
+
+Use Node's `AsyncLocalStorage` to record steps without threading `ctx.trace` through every function:
+
+```ts
+// In your test
+import { setCurrentTrace } from 'elasticdash-sdk'
+
+aiTest('flow test', async (ctx) => {
+  setCurrentTrace(ctx.trace)          // bind the trace to the current async context
+  await runFlowWithoutTraceArg()      // your existing code
+  expect(ctx.trace).toHaveCustomStep({ kind: 'rag', name: 'pokemon-search' })
+})
+
+// In your app/flow code (called during the test)
+import { getCurrentTrace } from 'elasticdash-sdk'
+
+function runFlowWithoutTraceArg() {
+  const trace = getCurrentTrace()
+  trace?.recordCustomStep({
+    kind: 'rag',
+    name: 'pokemon-search',
+    payload: { query: 'pikachu attack' },
+    result: { ids: [25] },
+    tags: ['source:db', 'sort:asc'],
+  })
+}
+```
+
+**Notes:**
+- Works per async context; if you spawn detached work (child processes/independent workers), pass `trace` explicitly there.
+- Still compatible with manual DI: you can continue passing `ctx.trace` explicitly if you prefer.
+
+## Optional LLM Capture Proxy (for Supabase Edge / Deno)
+
+For environments where Node fetch interception doesn't work (like Supabase Edge Functions or Deno Deploy):
+
+1. Set `ELASTICDASH_LLM_PROXY=1` (optional: `ELASTICDASH_LLM_PROXY_PORT`, default `8787`)
+2. The runner starts a local proxy and generates a per-test `ELASTICDASH_TRACE_ID`
+3. Point your LLM client at the proxy via base URL envs:
+   ```bash
+   OPENAI_BASE_URL=http://localhost:8787/v1
+   ANTHROPIC_API_URL=http://localhost:8787
+   ```
+4. Forward the trace ID to your Edge/Deno code (e.g., add `x-trace-id: process.env.ELASTICDASH_TRACE_ID` header)
+5. The proxy records model/prompt/completion and folds captured steps back into `ctx.trace`
+
+When `ELASTICDASH_LLM_PROXY` is unset, the existing Node fetch interceptor remains the default.