elasticdash-test 0.1.11-alpha-5 → 0.1.12-hotfix

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (38) hide show
  1. package/README.md +149 -581
  2. package/dist/capture/event.d.ts +4 -0
  3. package/dist/capture/event.d.ts.map +1 -1
  4. package/dist/capture/recorder.d.ts +5 -0
  5. package/dist/capture/recorder.d.ts.map +1 -1
  6. package/dist/capture/recorder.js +10 -0
  7. package/dist/capture/recorder.js.map +1 -1
  8. package/dist/cli.js +17 -0
  9. package/dist/cli.js.map +1 -1
  10. package/dist/dashboard-server.d.ts +12 -0
  11. package/dist/dashboard-server.d.ts.map +1 -1
  12. package/dist/dashboard-server.js +283 -51
  13. package/dist/dashboard-server.js.map +1 -1
  14. package/dist/index.cjs +2526 -0
  15. package/dist/index.d.ts +3 -1
  16. package/dist/index.d.ts.map +1 -1
  17. package/dist/index.js +2 -1
  18. package/dist/index.js.map +1 -1
  19. package/dist/interceptors/ai-interceptor.d.ts.map +1 -1
  20. package/dist/interceptors/ai-interceptor.js +101 -7
  21. package/dist/interceptors/ai-interceptor.js.map +1 -1
  22. package/dist/interceptors/http.d.ts +20 -0
  23. package/dist/interceptors/http.d.ts.map +1 -1
  24. package/dist/interceptors/http.js +184 -17
  25. package/dist/interceptors/http.js.map +1 -1
  26. package/dist/interceptors/tool.d.ts.map +1 -1
  27. package/dist/interceptors/tool.js +91 -0
  28. package/dist/interceptors/tool.js.map +1 -1
  29. package/dist/internals/mock-resolver.d.ts +25 -0
  30. package/dist/internals/mock-resolver.d.ts.map +1 -0
  31. package/dist/internals/mock-resolver.js +82 -0
  32. package/dist/internals/mock-resolver.js.map +1 -0
  33. package/dist/workflow-runner-worker.js +50 -3
  34. package/dist/workflow-runner-worker.js.map +1 -1
  35. package/dist/workflow-runner.d.ts.map +1 -1
  36. package/dist/workflow-runner.js +1 -0
  37. package/dist/workflow-runner.js.map +1 -1
  38. package/package.json +1 -1
package/README.md CHANGED
@@ -2,11 +2,31 @@
2
2
 
3
3
  An AI-native test runner for ElasticDash workflow testing. Built for async AI pipelines — not a general-purpose test runner.
4
4
 
5
- - Trace-first: every test receives a `trace` context to record and assert on LLM calls and tool invocations
6
- - Automatic fetch interception for OpenAI, Gemini, and Grok — no manual instrumentation required
7
- - AI-specific matchers: `toHaveLLMStep`, `toCallTool`, `toMatchSemanticOutput`, `toHaveCustomStep`, `toHavePromptWhere`, `toEvaluateOutputMetric`
8
- - Sequential execution, no parallelism overhead
9
- - No Jest dependency
5
+ ## Quick Links
6
+
7
+ ### Jump to Key Sections
8
+ - [Quick Start](#quick-start)
9
+ - [Documentation](#documentation)
10
+ - [Tool Recording](#tool-recording)
11
+ - [Configuration](#configuration)
12
+
13
+ ### Open Detailed Docs
14
+ - **[Quick Start Guide](docs/quickstart.md)** ← Start here to set up your first workflow
15
+ - [Test Writing Guidelines](docs/test-writing-guidelines.md)
16
+ - [Test Matchers](docs/matchers.md)
17
+ - [Tool Recording and Replay](docs/tools.md)
18
+ - [Workflows Dashboard](docs/dashboard.md)
19
+ - [Agent Mid-Trace Replay](docs/agents.md)
20
+ - [Deno Support](docs/deno.md)
21
+
22
+ ## Features
23
+
24
+ - 🎯 **Trace-first testing** — every test gets a `trace` context to record and assert on LLM calls and tool invocations
25
+ - 🔍 **Automatic AI interception** — captures OpenAI, Gemini, and Grok calls without code changes
26
+ - 🧪 **AI-specific matchers** — semantic output matching, LLM-judged evaluations, prompt assertions
27
+ - 🛠️ **Tool recording & replay** — automatically trace tool calls with checkpoint-based replay
28
+ - 📊 **Interactive dashboard** — browse workflows, debug traces, validate fixes visually
29
+ - 🤖 **Agent mid-trace replay** — resume long-running agents from any task without re-execution
10
30
 
11
31
  ---
12
32
 
@@ -16,7 +36,28 @@ An AI-native test runner for ElasticDash workflow testing. Built for async AI pi
16
36
  npm install elasticdash-test
17
37
  ```
18
38
 
19
- Requires Node 20+. For Deno projects, see [Using elasticdash-test in Deno](docs/deno.md).
39
+ **Requirements:** Node 20+. For Deno projects, see [Using elasticdash-test in Deno](docs/deno.md).
40
+
41
+ **Git ignore:** ElasticDash writes temporary runtime artifacts under `.temp/`. Add this to your `.gitignore`:
42
+
43
+ ```gitignore
44
+ .temp/
45
+ ```
46
+
47
+ **Running CLI commands:** Use `npx` to run commands with your locally installed version (recommended to avoid version drift):
48
+
49
+ ```bash
50
+ npx elasticdash test
51
+ npx elasticdash dashboard
52
+ ```
53
+
54
+ Alternatively, install globally if you prefer shorter commands:
55
+
56
+ ```bash
57
+ npm install -g elasticdash-test
58
+ elasticdash test
59
+ elasticdash dashboard
60
+ ```
20
61
 
21
62
  ---
22
63
 
@@ -39,9 +80,9 @@ aiTest('checkout flow', async (ctx) => {
39
80
  **2. Run it:**
40
81
 
41
82
  ```bash
42
- elasticdash test # discover all *.ai.test.ts files
43
- elasticdash test ./ai-tests # discover in a specific directory
44
- elasticdash run my-flow.ai.test.ts # run a single file
83
+ npx elasticdash test # discover all * *.ai.test.ts files
84
+ npx elasticdash test ./ai-tests # discover in a specific directory
85
+ npx elasticdash run my-flow.ai.test.ts # run a single file
45
86
  ```
46
87
 
47
88
  **3. Read the output:**
@@ -57,39 +98,45 @@ Total: 3
57
98
  Duration: 3.4s
58
99
  ```
59
100
 
101
+ **Workflow export requirements:**
102
+
103
+ - Export plain callable functions from `ed_workflows.ts/js`.
104
+ - Use JSON-serializable inputs/outputs (object or array) so dashboard replay can pass args and read results.
105
+ - Do not export framework-bound handlers directly (for example Next.js `NextRequest`/`NextResponse` route handlers).
106
+
60
107
  ---
61
108
 
62
- ## Writing Tests
109
+ ## Documentation
63
110
 
64
- See the full guide in [docs/test-writing-guidelines.md](docs/test-writing-guidelines.md).
111
+ ### Core Concepts
112
+ - **[Test Writing Guidelines](docs/test-writing-guidelines.md)** — comprehensive guide to writing AI workflow tests
113
+ - **[Test Matchers](docs/matchers.md)** — all available matchers with examples
114
+ - **[Tool Recording & Replay](docs/tools.md)** — automatic tool tracing and checkpoint-based replay
65
115
 
66
- ### Globals
116
+ ### Advanced Features
117
+ - **[Workflows Dashboard](docs/dashboard.md)** — interactive workflow browser, debugger, and fetching traces from Langfuse
118
+ - **[Agent Mid-Trace Replay](docs/agents.md)** — resume long-running agents from any task
119
+ - **[Deno Support](docs/deno.md)** — using ElasticDash Test in Deno projects
67
120
 
68
- After importing `test-setup`, these are available globally — no imports needed:
121
+ ---
122
+
123
+ ## Quick Reference
124
+
125
+ ### Test Globals
69
126
 
70
127
  | Global | Description |
71
128
  |---|---|
72
129
  | `aiTest(name, fn)` | Register a test |
73
130
  | `beforeAll(fn)` | Run once before all tests in the file |
74
131
  | `beforeEach(fn)` | Run before every test in the file |
75
- | `afterEach(fn)` | Run after every test in the file (runs even if the test fails) |
132
+ | `afterEach(fn)` | Run after every test in the file (runs even if test fails) |
76
133
  | `afterAll(fn)` | Run once after all tests in the file |
77
134
 
78
- ### Test context
79
-
80
- Each test function receives a `ctx: AITestContext` argument:
81
-
82
- ```ts
83
- aiTest('my test', async (ctx) => {
84
- // ctx.trace — record and inspect LLM steps and tool calls
85
- })
86
- ```
135
+ ### Recording Trace Data
87
136
 
88
- ### Recording trace data
137
+ **Automatic (recommended):** Workflow code making real API calls to OpenAI, Gemini, or Grok is automatically intercepted and recorded.
89
138
 
90
- **Automatic interception (recommended):** When your workflow code makes real API calls to OpenAI, Gemini, or Grok, the runner intercepts them automatically and records the LLM step — no changes to your workflow code needed. See [Automatic AI Interception](#automatic-ai-interception) below.
91
-
92
- **Manual recording:** Use this for providers not covered by the interceptor, when testing against stubs/mocks, or to capture RAG / code / fixed steps:
139
+ **Manual (for custom providers or mocks):**
93
140
 
94
141
  ```ts
95
142
  ctx.trace.recordLLMStep({
@@ -103,326 +150,125 @@ ctx.trace.recordToolCall({
103
150
  args: { amount: 99.99 },
104
151
  })
105
152
 
106
- // Record custom workflow steps (RAG fetches, code/fixed steps, etc.)
107
153
  ctx.trace.recordCustomStep({
108
- kind: 'rag', // 'rag' | 'code' | 'fixed' | 'custom'
154
+ kind: 'rag',
109
155
  name: 'pokemon-search',
110
- tags: ['sort:asc', 'source:db'],
111
- payload: { query: 'pikachu attack' },
156
+ payload: { query: 'pikachu' },
112
157
  result: { ids: [25] },
113
- metadata: { latencyMs: 120 },
114
158
  })
115
159
  ```
116
160
 
117
- ### Matchers
118
-
119
- #### `toHaveLLMStep(config?)`
120
-
121
- Assert the trace contains at least one LLM step matching the given config. All fields are optional and combined with AND logic.
161
+ ### Common Matchers
122
162
 
123
163
  ```ts
164
+ // Assert LLM calls
124
165
  expect(ctx.trace).toHaveLLMStep({ model: 'gpt-4' })
125
- expect(ctx.trace).toHaveLLMStep({ contains: 'order confirmed' }) // searches prompt + completion
126
- expect(ctx.trace).toHaveLLMStep({ promptContains: 'order status' }) // searches prompt only
127
- expect(ctx.trace).toHaveLLMStep({ outputContains: 'order confirmed' }) // searches completion only
128
- expect(ctx.trace).toHaveLLMStep({ provider: 'openai' })
129
- expect(ctx.trace).toHaveLLMStep({ provider: 'openai', promptContains: 'order status' })
130
- expect(ctx.trace).toHaveLLMStep({ promptContains: 'retry', times: 3 }) // exactly 3 matching steps
131
- expect(ctx.trace).toHaveLLMStep({ provider: 'openai', minTimes: 2 }) // at least 2 matching steps
132
- expect(ctx.trace).toHaveLLMStep({ outputContains: 'error', maxTimes: 1 }) // at most 1 matching step
133
- ```
134
-
135
- | Field | Description |
136
- |---|---|
137
- | `model` | Exact model name match (e.g. `'gpt-4o'`) |
138
- | `contains` | Substring match across prompt + completion (case-insensitive) |
139
- | `promptContains` | Substring match in prompt only (case-insensitive) |
140
- | `outputContains` | Substring match in completion only (case-insensitive) |
141
- | `provider` | Provider name: `'openai'`, `'gemini'`, or `'grok'` |
142
- | `times` | Exact match count (fails unless exactly this many steps match) |
143
- | `minTimes` | Minimum match count (steps matching must be ≥ this value) |
144
- | `maxTimes` | Maximum match count (steps matching must be ≤ this value) |
166
+ expect(ctx.trace).toHaveLLMStep({ promptContains: 'order status' })
145
167
 
146
- #### `toCallTool(toolName)`
147
-
148
- Assert the trace contains a tool call with the given name.
149
-
150
- ```ts
168
+ // Assert tool calls
151
169
  expect(ctx.trace).toCallTool('chargeCard')
152
- ```
153
170
 
154
- #### `toMatchSemanticOutput(expected, options?)`
155
-
156
- LLM-judged semantic match of combined LLM output vs. the expected string. Defaults to OpenAI GPT-4.1 with `OPENAI_API_KEY`. Optional options:
157
-
158
- ```ts
159
- expect(ctx.trace).toMatchSemanticOutput('attack stat', {
160
- provider: 'claude', // 'openai' (default) | 'claude' | 'gemini' | 'grok'
161
- model: 'claude-3-opus-20240229', // overrides default model for the provider
162
- sdk: myClaudeClient, // optional SDK instance (uses its chat/messages API)
163
- })
164
-
165
- // Minimal, using default OpenAI model
171
+ // Semantic output matching (LLM-judged)
166
172
  expect(ctx.trace).toMatchSemanticOutput('order confirmed')
167
173
 
168
- // OpenAI-compatible endpoint (e.g., Moonshot/Kimi) via baseURL + apiKey
169
- expect(ctx.trace).toMatchSemanticOutput('order confirmed', {
170
- provider: 'openai',
171
- model: 'kimi-k2-turbo-preview',
172
- apiKey: process.env.KIMI_API_KEY,
173
- baseURL: 'https://api.moonshot.ai/v1',
174
- })
175
- ```
176
-
177
- Environment keys by provider: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY` (or `GOOGLE_API_KEY`), `GROK_API_KEY`.
178
- For OpenAI-compatible endpoints, pass `apiKey`/`baseURL` in options or set an appropriate env var used by your SDK.
179
-
180
- #### `toEvaluateOutputMetric(config)`
181
-
182
- Evaluate one LLM step’s prompt or result using an LLM and assert a numeric metric condition in the range 0.0–1.0. Defaults: target=`result`, condition=`atLeast 0.7`, provider=`openai`, model=`gpt-4.1`.
183
-
184
- ```ts
185
- // Evaluate the last LLM result with your own prompt; default condition atLeast 0.7
186
- expect(ctx.trace).toEvaluateOutputMetric({
187
- evaluationPrompt: 'Rate how well this answers the user question.',
188
- })
189
-
190
- // Check a specific step (3rd LLM prompt), target the prompt text, require >= 0.8 via Claude
191
- expect(ctx.trace).toEvaluateOutputMetric({
192
- evaluationPrompt: 'Score coherence of this prompt between 0 and 1.',
193
- target: 'prompt',
194
- nth: 3,
195
- condition: { atLeast: 0.8 },
196
- provider: 'claude',
197
- model: 'claude-3-opus-20240229',
198
- })
199
-
200
- // Custom comparator: score must be < 0.3
201
- expect(ctx.trace).toEvaluateOutputMetric({
202
- evaluationPrompt: 'Rate hallucination risk (0=none, 1=high).',
203
- condition: { lessThan: 0.3 },
204
- })
205
- ```
206
-
207
- Options:
208
- - `evaluationPrompt` (required): your scoring instructions; model is asked to return only a number between 0 and 1.
209
- - `target`: `'result'` (default) or `'prompt'`. Mutually exclusive; evaluates that text only.
210
- - `index` / `nth`: pick which LLM step to score (0-based or 1-based). Defaults to the last LLM step.
211
- - `condition`: one of `greaterThan`, `lessThan`, `atLeast`, `atMost`, `equals`; default is `{ atLeast: 0.7 }`. Fails if the score is outside 0.0–1.0 or cannot be parsed.
212
- - `provider` / `model` / `sdk` / `apiKey` / `baseURL`: same shape as `toMatchSemanticOutput` (supports OpenAI, Claude, Gemini, Grok, and OpenAI-compatible via `baseURL`). Requires corresponding API key if no SDK is supplied.
213
-
214
- #### `toHaveCustomStep(config?)`
215
-
216
- Assert a recorded custom step (RAG/code/fixed/custom) matches filters.
217
-
218
- ```ts
174
+ // Custom steps (RAG, code, fixed)
219
175
  expect(ctx.trace).toHaveCustomStep({ kind: 'rag', name: 'pokemon-search' })
220
- expect(ctx.trace).toHaveCustomStep({ tag: 'sort:asc' })
221
- expect(ctx.trace).toHaveCustomStep({ contains: 'pikachu' })
222
- expect(ctx.trace).toHaveCustomStep({ resultContains: '25' })
223
- expect(ctx.trace).toHaveCustomStep({ kind: 'rag', minTimes: 1, maxTimes: 2 })
224
176
  ```
225
177
 
226
- #### `toHavePromptWhere(config)`
227
-
228
- Filter prompts, then assert additional constraints. Example: “all prompts containing A must also contain B”.
229
-
230
- ```ts
231
- // Prompts that contain "order" must also contain "confirmed"
232
- expect(ctx.trace).toHavePromptWhere({
233
- filterContains: 'order',
234
- requireContains: 'confirmed',
235
- })
236
-
237
- // Prompts containing "retry" must NOT contain "cancel"
238
- expect(ctx.trace).toHavePromptWhere({
239
- filterContains: 'retry',
240
- requireNotContains: 'cancel',
241
- })
242
-
243
- // And control counts on the filtered subset
244
- expect(ctx.trace).toHavePromptWhere({
245
- filterContains: 'order',
246
- requireContains: 'confirmed',
247
- minTimes: 1,
248
- maxTimes: 3,
249
- })
250
-
251
- // Check a specific prompt position (1-based nth or 0-based index)
252
- expect(ctx.trace).toHavePromptWhere({
253
- filterContains: 'order',
254
- requireContains: 'confirmed',
255
- nth: 3, // the 3rd prompt among those containing "order"
256
- })
257
- ```
178
+ **→ See [Test Matchers](docs/matchers.md) for complete documentation**
258
179
 
259
180
  ---
260
181
 
261
- ## Automatic AI Interception
182
+ ## Automatic AI & Tool Tracing
262
183
 
263
- The runner patches `globalThis.fetch` before tests run and automatically records LLM steps for calls to the following endpoints:
184
+ ### AI Interception
264
185
 
265
- | Provider | Endpoints intercepted |
266
- |---|---|
267
- | **OpenAI** | `api.openai.com/v1/chat/completions`, `/v1/completions` |
268
- | **Gemini** | `generativelanguage.googleapis.com/.../models/...:generateContent` |
269
- | **Grok** (xAI) | `api.x.ai/v1/chat/completions` |
186
+ The runner automatically intercepts and records calls to:
187
+ - OpenAI (`api.openai.com`)
188
+ - Gemini (`generativelanguage.googleapis.com`)
189
+ - Grok/xAI (`api.x.ai`)
270
190
 
271
- Each intercepted call records `model`, `provider`, `prompt`, and `completion` into `ctx.trace` automatically. Your workflow code needs no changes.
191
+ No code changes needed just run your workflow and assertions work automatically.
272
192
 
273
- ```ts
274
- aiTest('user lookup flow', async (ctx) => {
275
- // This makes a real OpenAI call — intercepted automatically
276
- await myWorkflow.run('Find all active users')
193
+ ### Tool Recording
277
194
 
278
- // Works without any ctx.trace.recordLLMStep() in your workflow
279
- expect(ctx.trace).toHaveLLMStep({ promptContains: 'Find all active users' })
280
- expect(ctx.trace).toHaveLLMStep({ provider: 'openai' })
281
- })
282
- ```
283
-
284
- **Streaming:** When `stream: true` is set on a request, the completion is recorded as `"(streamed)"` — the prompt and model are still captured.
285
-
286
- **Libraries using `https.request` directly** (older versions of some SDKs) are not covered by fetch interception. Use manual `ctx.trace.recordLLMStep()` for those.
287
-
288
- ## Automatic Tool Recording
289
-
290
- Tool functions are automatically wrapped and recorded when imported and called during workflow execution. This gives you tracing and replay support without any manual instrumentation.
291
-
292
- ### How to use tools
293
-
294
- Define your tools in `ed_tools.ts` and export them:
195
+ Manual instrumentation pattern: isolate tracing in the service `.then/.catch` path so tracing failures never block business logic:
295
196
 
296
197
  ```ts
297
- // ed_tools.ts
298
- import { wrapTool } from 'elasticdash-test'
299
-
300
- export const chargeCard = wrapTool('chargeCard', async (input: { amount: number; cardToken: string }) => {
301
- // your actual implementation
302
- return { success: true, transactionId: 'txn-123' }
303
- })
304
-
305
- export const fetchOrderStatus = wrapTool('fetchOrderStatus', async (input: { orderId: string }) => {
306
- // query your database or API
307
- return { status: 'shipped', tracking: 'track-456' }
308
- })
309
- ```
310
-
311
- In your workflows or agents, simply import and call them — the runner automatically wraps them and records each call:
312
-
313
- ```ts
314
- import { chargeCard, fetchOrderStatus } from './ed_tools'
315
-
316
- export async function checkoutFlow(orderId: string, cardToken: string) {
317
- const order = await fetchOrderStatus({ orderId })
318
- const charge = await chargeCard({ amount: order.total, cardToken })
319
- return { orderId, chargeId: charge.transactionId }
198
+ import { runSelectQuery } from './services/dataService'
199
+
200
+ export const dataService = async (input: any) => {
201
+ const { query } = input as { query: string }
202
+ return await runSelectQuery(query)
203
+ .then(async (res: any) => {
204
+ try {
205
+ const { recordToolCall } = await import('elasticdash-test')
206
+ recordToolCall('dataService', input, res)
207
+ } catch {
208
+ // tracing must never block the main service path
209
+ }
210
+ return res
211
+ })
212
+ .catch(async (err: any) => {
213
+ try {
214
+ const { recordToolCall } = await import('elasticdash-test')
215
+ recordToolCall('dataService', input, err)
216
+ } catch {
217
+ // tracing must never block the main service path
218
+ }
219
+ throw err
220
+ })
320
221
  }
321
222
  ```
322
223
 
323
- Each call to a tool is automatically recorded to `ctx.trace` with:
324
- - Tool name
325
- - Input arguments
326
- - Output result
327
- - Timestamp and duration
224
+ In manual mode, always isolate tracing in a separate `try/catch` so trace logging errors cannot interrupt core service execution.
328
225
 
329
- No `ctx.trace.recordToolCall()` needed it happens behind the scenes.
226
+ **→ See [Tool Recording & Replay](docs/tools.md) for checkpoint-based replay and freezing**
330
227
 
331
- Runtime behavior by mode:
332
- - Local/normal execution: wrapped tools run normally (no replay context).
333
- - Trace capture mode: wrapped tools run normally and are recorded.
334
- - Replay mode (`Run from here`): pre-checkpoint wrapped calls return historical results and skip tool body execution.
228
+ ### HTTP Streaming Capture and Replay
335
229
 
336
- ### Tool replay and freezing in "Run from here"
230
+ ElasticDash also captures non-AI `fetch` responses that stream over HTTP (for example SSE and NDJSON endpoints) in the HTTP interceptor.
337
231
 
338
- When you use the dashboard's "Run from here" feature to replay a workflow from a checkpoint:
232
+ Currently detected as streaming when response `content-type` includes:
233
+ - `text/event-stream`
234
+ - `application/x-ndjson`
235
+ - `application/stream+json`
236
+ - `application/jsonl`
339
237
 
340
- 1. **Tools called before the checkpoint** are **frozen** — their recorded results are returned instantly without re-executing the actual function.
341
- 2. **Tools called after the checkpoint** execute normally with live dependencies.
342
-
343
- This means:
344
- - If a tool made 3 API calls before the checkpoint, those calls won't happen again on replay.
345
- - If the tool is slow (waiting for external services), frozen calls are instant.
346
- - The tool's frozen status is visible in the dashboard with a frozen tag/styling, matching AI step behavior.
347
- - Side-effect replay is type-checked (`Date.now` vs `Math.random`) and malformed observation timestamps are auto-sanitized.
348
-
349
- Example:
350
- ```
351
- Task 1: fetchOrderStatus() ↓ frozen (checkpoint before this)
352
- │─ API call to orders service ↓ frozen
353
- └─ Result: { status: shipped } ↓ frozen
238
+ How it behaves today:
239
+ - During live execution, ElasticDash tees the response stream and returns a real stream to your app code.
240
+ - In parallel, ElasticDash buffers the recorder side of the stream as raw text for trace replay.
241
+ - During replay, ElasticDash reconstructs a stream from that captured raw payload and restores status, status text, and response headers.
354
242
 
355
- You modify input for Task 2 and re-run
356
- The fetchOrderStatus API call is skipped; the original result is reused instantly.
357
- ```
358
-
359
- ### Recording tool calls manually
360
-
361
- If you need to record tools outside the normal import flow or for providers not in `ed_tools.ts`, use:
362
-
363
- ```ts
364
- ctx.trace.recordToolCall({
365
- name: 'customTool',
366
- args: { param: 'value' },
367
- result: { success: true }
368
- })
369
- ```
243
+ Replay fidelity note:
244
+ - Replay preserves stream payload content, but not original chunk boundaries or timing cadence.
370
245
 
371
- Manual recording is best-effort trace logging. For full replay/freeze semantics in "Run from here" (including frozen tool tags and deterministic replay), run tools through the replay-aware wrapper path (auto-wrapped worker tools or `wrapTool(...)`).
246
+ Minimal stream consumption example:
372
247
 
373
248
  ```ts
374
- import { wrapTool } from 'elasticdash-test'
249
+ const res = await fetch('https://example.com/events')
250
+ if (!res.body) throw new Error('Expected a streaming response body')
375
251
 
376
- export const customTool = wrapTool('customTool', async (input: { param: string }) => {
377
- return await doWork(input)
378
- })
379
- ```
252
+ const reader = res.body.getReader()
253
+ const decoder = new TextDecoder()
254
+ let buffer = ''
380
255
 
381
- ### Recording flow steps without passing `ctx.trace` (AsyncLocalStorage)
382
-
383
- The runner now sets a per-test `currentTrace` using Node’s `AsyncLocalStorage`, so your app code can record steps without threading `ctx.trace` through every function. This remains safe under parallel execution.
384
-
385
- ```ts
386
- // In your test
387
- import { setCurrentTrace } from 'elasticdash-test'
388
-
389
- aiTest('flow test', async (ctx) => {
390
- setCurrentTrace(ctx.trace) // bind the trace to the current async context
391
- await runFlowWithoutTraceArg() // your existing code
392
- // assertions
393
- expect(ctx.trace).toHaveCustomStep({ kind: 'rag', name: 'pokemon-search' })
394
- })
395
-
396
- // In your app/flow code (called during the test)
397
- import { getCurrentTrace } from 'elasticdash-test'
398
-
399
- function runFlowWithoutTraceArg() {
400
- const trace = getCurrentTrace()
401
- trace?.recordCustomStep({
402
- kind: 'rag',
403
- name: 'pokemon-search',
404
- payload: { query: 'pikachu attack' },
405
- result: { ids: [25] },
406
- tags: ['source:db', 'sort:asc'],
407
- })
256
+ for (;;) {
257
+ const { done, value } = await reader.read()
258
+ if (done) break
259
+ buffer += decoder.decode(value, { stream: true })
408
260
  }
409
- ```
410
261
 
411
- Notes:
412
- - Works per async context; if you spawn detached work (child processes/independent workers), pass `trace` explicitly there.
413
- - Still compatible with manual DI: you can continue passing `ctx.trace` explicitly if you prefer.
262
+ buffer += decoder.decode()
263
+ ```
414
264
 
415
- ### Optional local LLM capture proxy (for Supabase Edge / Deno)
416
- - Opt in by setting `ELASTICDASH_LLM_PROXY=1` (optional: `ELASTICDASH_LLM_PROXY_PORT`, default `8787`). The runner will start a local proxy and generate a per-test `ELASTICDASH_TRACE_ID`.
417
- - Point your LLM client at the proxy via base URL envs (e.g., `OPENAI_BASE_URL=http://localhost:8787/v1`, `ANTHROPIC_API_URL=http://localhost:8787`). No code changes in your workflow; only env overrides when running tests.
418
- - Forward the trace ID to your Edge/Deno code (e.g., add `x-trace-id: process.env.ELASTICDASH_TRACE_ID` on the request to your Supabase Edge function). The proxy records model/prompt/completion (or `(streamed)`) and the runner folds captured steps back into `ctx.trace` after each test.
419
- - When `ELASTICDASH_LLM_PROXY` is unset, behavior is unchanged and the existing Node fetch interceptor remains the default.
265
+ **→ See [Quick Start Guide](docs/quickstart.md#capture-streaming-flows) for end-to-end setup guidance**
420
266
 
421
267
  ---
422
268
 
423
269
  ## Configuration
424
270
 
425
- Create an optional `elasticdash.config.ts` at the project root:
271
+ Optional `elasticdash.config.ts` at project root:
426
272
 
427
273
  ```ts
428
274
  export default {
@@ -431,311 +277,33 @@ export default {
431
277
  }
432
278
  ```
433
279
 
434
- | Option | Default | Description |
435
- |---|---|---|
436
- | `testMatch` | `['**/*.ai.test.ts']` | Glob patterns for test discovery |
437
- | `traceMode` | `'local'` | `'local'` (stub) or `'remote'` (future ElasticDash backend) |
438
-
439
- ---
280
+ Optional project file: `ed_workers.ts` can be used by your app architecture (for example, exporting worker handlers), but it is not required or discovered by the ElasticDash CLI/dashboard.
440
281
 
441
282
  ## TypeScript Setup
442
283
 
443
- Add an `examples/tsconfig.json` (or your test directory's `tsconfig.json`) that extends the root config and includes the `src` types:
284
+ For typed globals and matchers, extend your test directory's `tsconfig.json`:
444
285
 
445
286
  ```json
446
287
  {
447
288
  "extends": "../tsconfig.json",
448
- "compilerOptions": {
449
- "rootDir": "..",
450
- "noEmit": true
451
- },
452
- "include": [
453
- "../src/**/*",
454
- "./**/*"
455
- ]
289
+ "include": ["../src/**/*", "./**/*"]
456
290
  }
457
291
  ```
458
292
 
459
- This gives you typed `aiTest`, `beforeAll`, `afterAll` globals and typed custom matchers in your test files.
460
-
461
- ---
462
-
463
- ## Workflows Dashboard
464
-
465
- Browse and search all available workflow functions in your project:
466
-
467
- ```bash
468
- elasticdash dashboard # open dashboard at http://localhost:4573
469
- elasticdash dashboard --port 4572 # use custom port
470
- elasticdash dashboard --no-open # skip auto-opening browser
471
- ```
472
-
473
- The dashboard scans your workflow/tool files and displays:
474
- - If both `.ts` and `.js` versions of a file exist (e.g., `ed_workflows.ts` and `ed_workflows.js`), the dashboard will always use the `.ts` file.
475
- - If only `.ts` exists, it will be automatically transpiled to `.js` before scanning/importing—no manual build step required.
476
- - If only `.js` exists, it will be used directly.
477
-
478
- This means you can write your workflows and tools in TypeScript, and the dashboard will handle transpilation automatically. You do not need to run `tsc` or build manually for dashboard usage.
479
-
480
- **Example file selection logic:**
481
- | Scenario | File Used |
482
- |-------------------------|------------------|
483
- | Only `ed_workflows.ts` | Transpiled `.ts` |
484
- | Only `ed_workflows.js` | `.js` |
485
- | Both exist | `.ts` (preferred)|
486
-
487
- The dashboard displays:
488
- - **Function names** — all exported functions in the module
489
- - **Signatures** — function parameters and return types
490
- - **Async indicator** — marks async vs sync functions
491
- - **Source module** — where the function is imported from (if re-exported)
492
- - **File path** — location of the workflow file
493
-
494
- Use the search field to filter workflows by:
495
- - **Name** — find workflow by function name (e.g., `checkoutFlow`)
496
- - **Source module** — find all workflows from a specific module (e.g., `app.workflows`)
497
- - **File path** — filter by location in your codebase
498
-
499
- This is useful for discovering available workflows, understanding their signatures, and identifying where functions are defined before calling them in tests.
500
-
501
- ### `ed_workflows.ts`, `ed_tools.ts`, `ed_agents.ts`
502
-
503
- These optional files bundle and re-export existing functions from your codebase for use in tests.
504
-
505
- #### `ed_workflows.ts`
506
-
507
- Re-export workflow functions from your application:
508
-
509
- ```ts
510
- // ed_workflows.ts
511
- export { orderWorkflow, refundWorkflow } from './src/workflows'
512
- export { userLookupFlow } from './src/user-flows'
513
- ```
514
-
515
- Access in tests:
516
-
517
- ```ts
518
- import { orderWorkflow } from './ed_workflows'
519
-
520
- aiTest('full order workflow', async (ctx) => {
521
- const result = await orderWorkflow('order-123', 'cust-456')
522
- expect(ctx.trace).toCallTool('chargeCard')
523
- })
524
- ```
525
-
526
- #### `ed_tools.ts`
527
-
528
- Re-export tool functions that agents or workflows can invoke:
529
-
530
- ```ts
531
- // ed_tools.ts
532
- import { wrapTool } from 'elasticdash-test'
533
- import { chargeCard as chargeCardRaw, fetchOrderStatus as fetchOrderStatusRaw, sendNotification as sendNotificationRaw } from './src/tools'
534
-
535
- export const chargeCard = wrapTool('chargeCard', chargeCardRaw)
536
- export const fetchOrderStatus = wrapTool('fetchOrderStatus', fetchOrderStatusRaw)
537
- export const sendNotification = wrapTool('sendNotification', sendNotificationRaw)
538
- ```
539
-
540
- #### `ed_agents.ts`
541
-
542
- Re-export agent functions or create a config object:
543
-
544
- ```ts
545
- // ed_agents.ts
546
- export { checkoutAgent, paymentAgent } from './src/agents'
547
-
548
- // Or as a config object:
549
- export const agents = {
550
- checkout: checkoutAgent,
551
- payment: paymentAgent,
552
- }
553
- ```
554
-
555
- The dashboard command will scan these files and display all exported functions with their signatures, making it easy to explore your workflow API.
556
-
557
- ---
558
-
559
- ## Agent Mid-Trace Replay
560
-
561
- ElasticDash supports resuming a long-running agent from any task in its plan — without re-executing already-completed steps. This is useful for:
562
-
563
- - **Resuming after failures**: If task 3 of 5 fails, fix the issue and re-run from task 3 only.
564
- - **Pausing for approval**: Capture state after task 2, get human sign-off, then continue.
565
- - **Debugging in isolation**: Re-run a single task with modified input to diagnose a problem.
566
-
567
- ### How it works
568
-
569
- Agents are structured as an **AgentPlan** — an ordered list of **AgentTask** objects, each with a tool name, input, and output. When serialized, the plan plus all captured trace events form an **AgentState** that can be saved to disk/database and replayed later.
570
-
571
- ### Quick start
572
-
573
- ```ts
574
- import { plannerAgent, executorAgent, resumeAgentFromTrace } from './ed_agents'
575
- import { serializeAgentState, deserializeAgentState } from 'elasticdash-test'
576
- import fs from 'node:fs'
577
-
578
- // 1. Generate a plan
579
- const plan = await plannerAgent('Show me sales for Q1', { userToken: 'tok-abc' })
580
-
581
- // 2. Execute the plan (runs all tasks sequentially)
582
- const completedPlan = await executorAgent(plan)
583
-
584
- // 3. Serialize and save state (e.g., after partial execution)
585
- const state = serializeAgentState(completedPlan, [] /* pass recorder.events in worker context */)
586
- fs.writeFileSync('agent-state.json', JSON.stringify(state, null, 2))
587
-
588
- // 4. Later: load saved state and resume from task 2 (0-based index 1)
589
- const savedState = JSON.parse(fs.readFileSync('agent-state.json', 'utf8'))
590
- const stateToResume = deserializeAgentState({ ...savedState, resumeFromTaskIndex: 1 })
591
- const resumedPlan = await resumeAgentFromTrace(stateToResume)
592
-
593
- console.log('Resumed plan status:', resumedPlan.status)
594
- console.log('Task outputs:', resumedPlan.tasks.map((t) => ({ id: t.id, status: t.status })))
595
- ```
596
-
597
- ### AgentState structure
598
-
599
- ```ts
600
- interface AgentState {
601
- plan: AgentPlan // Full plan with all tasks (completed and pending)
602
- trace: WorkflowEvent[] // Captured trace events from previous execution
603
- resumeFromTaskIndex: number // Zero-based index — tasks before this are loaded from cache
604
- }
605
-
606
- interface AgentPlan {
607
- id: string
608
- tasks: AgentTask[]
609
- status: 'planning' | 'executing' | 'completed' | 'failed' | 'paused'
610
- currentTaskIndex: number
611
- context: Record<string, unknown>
612
- metadata: Record<string, unknown>
613
- }
614
-
615
- interface AgentTask {
616
- id: string
617
- status: 'pending' | 'in-progress' | 'completed' | 'failed'
618
- description: string
619
- tool: string // Name of the tool function to invoke
620
- input: unknown // May contain { $ref: "task-N.output.fieldName" } placeholders
621
- output?: unknown // Populated after execution
622
- error?: string
623
- startedAt?: number
624
- completedAt?: number
625
- }
626
- ```
627
-
628
- ### Task input placeholders
629
-
630
- Task inputs can reference previous task outputs using `{ $ref: "taskId.output.fieldPath" }`:
631
-
632
- ```ts
633
- // task-2 uses the embedding produced by task-1
634
- {
635
- id: 'task-2',
636
- tool: 'taskSelectorService',
637
- input: {
638
- queryEmbedding: { $ref: 'task-1.output.embedding' },
639
- topK: 3,
640
- }
641
- }
642
- ```
643
-
644
- Placeholders are resolved at execution time by `resolveTaskInput()`.
645
-
646
- ### Dashboard integration
647
-
648
- When running an agent workflow through the dashboard:
649
-
650
- 1. Agent task observations are **visually highlighted** with a purple background and left border, making them easy to identify.
651
- 2. Each agent observation shows a **T1 / T2 / T3** badge indicating which task it belongs to.
652
- 3. In the observation detail panel, a **"Resume from Task N"** button appears (agent steps only — non-agent steps have no button).
653
- 4. Clicking it calls `/api/resume-agent-from-task` with the serialized `AgentState` and the chosen `taskIndex`, then adds the resumed run as a new trace in the comparison table.
654
-
655
- ### Best practices for resumable agents
656
-
657
- - **Keep tasks idempotent** where possible — if a task must be re-run, ensure it produces the same result.
658
- - **Store minimal outputs** — only record what downstream tasks need, not full API responses.
659
- - **Version your state schema** — if tool interfaces change, old states may need migration.
660
- - **Use sequential tasks** — the current implementation runs tasks one-by-one; parallel task support is a planned future enhancement.
661
-
662
- ---
663
-
664
- ## Project Structure
665
-
666
- ```
667
- src/
668
- cli.ts CLI entry point (commander + fast-glob)
669
- runner.ts Sequential test runner engine
670
- reporter.ts Color-coded terminal output
671
- test-setup.ts Import in test files for globals + matcher types
672
- index.ts Programmatic API
673
- core/
674
- registry.ts aiTest / beforeAll / afterAll registry
675
- trace-adapter/
676
- context.ts TraceHandle, AITestContext, RunnerHooks scaffold
677
- matchers/
678
- index.ts Custom expect matchers
679
- interceptors/
680
- ai-interceptor.ts Automatic fetch interceptor for OpenAI / Gemini / Grok
681
- ```
682
-
683
293
  ---
684
294
 
685
295
  ## Programmatic API
686
296
 
687
297
  ```ts
688
- import { runFiles, reportResults, registerMatchers, installAIInterceptor, uninstallAIInterceptor } from 'elasticdash-test'
298
+ import { runFiles, reportResults, registerMatchers, installAIInterceptor } from 'elasticdash-test'
689
299
 
690
300
  registerMatchers()
691
- installAIInterceptor() // patch globalThis.fetch for automatic LLM tracing
301
+ installAIInterceptor()
692
302
 
693
303
  const results = await runFiles(['./tests/flow.ai.test.ts'])
694
304
  reportResults(results)
695
-
696
- uninstallAIInterceptor() // restore original fetch when done
697
- ```
698
-
699
- ---
700
-
701
- ## Recording Tool Calls Explicitly
702
-
703
- If you want to ensure tool calls are always recorded in the workflow trace—regardless of how your tools are imported or used—you can use the `recordToolCall` utility provided by this SDK.
704
-
705
- ### How to Use
706
-
707
-
708
- 1. Import the function in your tool implementation:
709
-
710
- ```ts
711
- import { recordToolCall } from 'elasticdash-test'
712
-
713
- export async function myTool(input: string) {
714
- // ...tool logic...
715
- const result = `Hello, ${input}!`
716
- recordToolCall('myTool', { input }, result)
717
- return result
718
- }
719
305
  ```
720
306
 
721
- 2. When running under ElasticDash, all calls to `recordToolCall` will be captured in the workflow trace. When running locally or outside the runner, this function is a no-op and will not affect your code.
722
-
723
- **This approach is robust and works with both imported and global tools.**
724
-
725
- **Summary:**
726
- - Use tools as globals (no import) for full traceability and automatic tool call capture.
727
- - This approach keeps your workflow code clean and enables powerful debugging and analytics in ElasticDash.
728
-
729
- ## Non-Goals
730
-
731
- This runner intentionally does not support:
732
-
733
- - Parallel execution
734
- - Watch mode
735
- - Snapshot testing
736
- - Coverage reporting
737
- - Jest compatibility
738
-
739
307
  ---
740
308
 
741
309
  ## License