elasticdash-test 0.1.11-alpha-5 → 0.1.12-hotfix
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +149 -581
- package/dist/capture/event.d.ts +4 -0
- package/dist/capture/event.d.ts.map +1 -1
- package/dist/capture/recorder.d.ts +5 -0
- package/dist/capture/recorder.d.ts.map +1 -1
- package/dist/capture/recorder.js +10 -0
- package/dist/capture/recorder.js.map +1 -1
- package/dist/cli.js +17 -0
- package/dist/cli.js.map +1 -1
- package/dist/dashboard-server.d.ts +12 -0
- package/dist/dashboard-server.d.ts.map +1 -1
- package/dist/dashboard-server.js +283 -51
- package/dist/dashboard-server.js.map +1 -1
- package/dist/index.cjs +2526 -0
- package/dist/index.d.ts +3 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +2 -1
- package/dist/index.js.map +1 -1
- package/dist/interceptors/ai-interceptor.d.ts.map +1 -1
- package/dist/interceptors/ai-interceptor.js +101 -7
- package/dist/interceptors/ai-interceptor.js.map +1 -1
- package/dist/interceptors/http.d.ts +20 -0
- package/dist/interceptors/http.d.ts.map +1 -1
- package/dist/interceptors/http.js +184 -17
- package/dist/interceptors/http.js.map +1 -1
- package/dist/interceptors/tool.d.ts.map +1 -1
- package/dist/interceptors/tool.js +91 -0
- package/dist/interceptors/tool.js.map +1 -1
- package/dist/internals/mock-resolver.d.ts +25 -0
- package/dist/internals/mock-resolver.d.ts.map +1 -0
- package/dist/internals/mock-resolver.js +82 -0
- package/dist/internals/mock-resolver.js.map +1 -0
- package/dist/workflow-runner-worker.js +50 -3
- package/dist/workflow-runner-worker.js.map +1 -1
- package/dist/workflow-runner.d.ts.map +1 -1
- package/dist/workflow-runner.js +1 -0
- package/dist/workflow-runner.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -2,11 +2,31 @@
|
|
|
2
2
|
|
|
3
3
|
An AI-native test runner for ElasticDash workflow testing. Built for async AI pipelines — not a general-purpose test runner.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
-
|
|
9
|
-
-
|
|
5
|
+
## Quick Links
|
|
6
|
+
|
|
7
|
+
### Jump to Key Sections
|
|
8
|
+
- [Quick Start](#quick-start)
|
|
9
|
+
- [Documentation](#documentation)
|
|
10
|
+
- [Tool Recording](#tool-recording)
|
|
11
|
+
- [Configuration](#configuration)
|
|
12
|
+
|
|
13
|
+
### Open Detailed Docs
|
|
14
|
+
- **[Quick Start Guide](docs/quickstart.md)** ← Start here to set up your first workflow
|
|
15
|
+
- [Test Writing Guidelines](docs/test-writing-guidelines.md)
|
|
16
|
+
- [Test Matchers](docs/matchers.md)
|
|
17
|
+
- [Tool Recording and Replay](docs/tools.md)
|
|
18
|
+
- [Workflows Dashboard](docs/dashboard.md)
|
|
19
|
+
- [Agent Mid-Trace Replay](docs/agents.md)
|
|
20
|
+
- [Deno Support](docs/deno.md)
|
|
21
|
+
|
|
22
|
+
## Features
|
|
23
|
+
|
|
24
|
+
- 🎯 **Trace-first testing** — every test gets a `trace` context to record and assert on LLM calls and tool invocations
|
|
25
|
+
- 🔍 **Automatic AI interception** — captures OpenAI, Gemini, and Grok calls without code changes
|
|
26
|
+
- 🧪 **AI-specific matchers** — semantic output matching, LLM-judged evaluations, prompt assertions
|
|
27
|
+
- 🛠️ **Tool recording & replay** — automatically trace tool calls with checkpoint-based replay
|
|
28
|
+
- 📊 **Interactive dashboard** — browse workflows, debug traces, validate fixes visually
|
|
29
|
+
- 🤖 **Agent mid-trace replay** — resume long-running agents from any task without re-execution
|
|
10
30
|
|
|
11
31
|
---
|
|
12
32
|
|
|
@@ -16,7 +36,28 @@ An AI-native test runner for ElasticDash workflow testing. Built for async AI pi
|
|
|
16
36
|
npm install elasticdash-test
|
|
17
37
|
```
|
|
18
38
|
|
|
19
|
-
|
|
39
|
+
**Requirements:** Node 20+. For Deno projects, see [Using elasticdash-test in Deno](docs/deno.md).
|
|
40
|
+
|
|
41
|
+
**Git ignore:** ElasticDash writes temporary runtime artifacts under `.temp/`. Add this to your `.gitignore`:
|
|
42
|
+
|
|
43
|
+
```gitignore
|
|
44
|
+
.temp/
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
**Running CLI commands:** Use `npx` to run commands with your locally installed version (recommended to avoid version drift):
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
npx elasticdash test
|
|
51
|
+
npx elasticdash dashboard
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Alternatively, install globally if you prefer shorter commands:
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
npm install -g elasticdash-test
|
|
58
|
+
elasticdash test
|
|
59
|
+
elasticdash dashboard
|
|
60
|
+
```
|
|
20
61
|
|
|
21
62
|
---
|
|
22
63
|
|
|
@@ -39,9 +80,9 @@ aiTest('checkout flow', async (ctx) => {
|
|
|
39
80
|
**2. Run it:**
|
|
40
81
|
|
|
41
82
|
```bash
|
|
42
|
-
elasticdash test # discover all *.ai.test.ts files
|
|
43
|
-
elasticdash test ./ai-tests # discover in a specific directory
|
|
44
|
-
elasticdash run my-flow.ai.test.ts # run a single file
|
|
83
|
+
npx elasticdash test # discover all * *.ai.test.ts files
|
|
84
|
+
npx elasticdash test ./ai-tests # discover in a specific directory
|
|
85
|
+
npx elasticdash run my-flow.ai.test.ts # run a single file
|
|
45
86
|
```
|
|
46
87
|
|
|
47
88
|
**3. Read the output:**
|
|
@@ -57,39 +98,45 @@ Total: 3
|
|
|
57
98
|
Duration: 3.4s
|
|
58
99
|
```
|
|
59
100
|
|
|
101
|
+
**Workflow export requirements:**
|
|
102
|
+
|
|
103
|
+
- Export plain callable functions from `ed_workflows.ts/js`.
|
|
104
|
+
- Use JSON-serializable inputs/outputs (object or array) so dashboard replay can pass args and read results.
|
|
105
|
+
- Do not export framework-bound handlers directly (for example Next.js `NextRequest`/`NextResponse` route handlers).
|
|
106
|
+
|
|
60
107
|
---
|
|
61
108
|
|
|
62
|
-
##
|
|
109
|
+
## Documentation
|
|
63
110
|
|
|
64
|
-
|
|
111
|
+
### Core Concepts
|
|
112
|
+
- **[Test Writing Guidelines](docs/test-writing-guidelines.md)** — comprehensive guide to writing AI workflow tests
|
|
113
|
+
- **[Test Matchers](docs/matchers.md)** — all available matchers with examples
|
|
114
|
+
- **[Tool Recording & Replay](docs/tools.md)** — automatic tool tracing and checkpoint-based replay
|
|
65
115
|
|
|
66
|
-
###
|
|
116
|
+
### Advanced Features
|
|
117
|
+
- **[Workflows Dashboard](docs/dashboard.md)** — interactive workflow browser, debugger, and fetching traces from Langfuse
|
|
118
|
+
- **[Agent Mid-Trace Replay](docs/agents.md)** — resume long-running agents from any task
|
|
119
|
+
- **[Deno Support](docs/deno.md)** — using ElasticDash Test in Deno projects
|
|
67
120
|
|
|
68
|
-
|
|
121
|
+
---
|
|
122
|
+
|
|
123
|
+
## Quick Reference
|
|
124
|
+
|
|
125
|
+
### Test Globals
|
|
69
126
|
|
|
70
127
|
| Global | Description |
|
|
71
128
|
|---|---|
|
|
72
129
|
| `aiTest(name, fn)` | Register a test |
|
|
73
130
|
| `beforeAll(fn)` | Run once before all tests in the file |
|
|
74
131
|
| `beforeEach(fn)` | Run before every test in the file |
|
|
75
|
-
| `afterEach(fn)` | Run after every test in the file (runs even if
|
|
132
|
+
| `afterEach(fn)` | Run after every test in the file (runs even if test fails) |
|
|
76
133
|
| `afterAll(fn)` | Run once after all tests in the file |
|
|
77
134
|
|
|
78
|
-
###
|
|
79
|
-
|
|
80
|
-
Each test function receives a `ctx: AITestContext` argument:
|
|
81
|
-
|
|
82
|
-
```ts
|
|
83
|
-
aiTest('my test', async (ctx) => {
|
|
84
|
-
// ctx.trace — record and inspect LLM steps and tool calls
|
|
85
|
-
})
|
|
86
|
-
```
|
|
135
|
+
### Recording Trace Data
|
|
87
136
|
|
|
88
|
-
|
|
137
|
+
**Automatic (recommended):** Workflow code making real API calls to OpenAI, Gemini, or Grok is automatically intercepted and recorded.
|
|
89
138
|
|
|
90
|
-
**
|
|
91
|
-
|
|
92
|
-
**Manual recording:** Use this for providers not covered by the interceptor, when testing against stubs/mocks, or to capture RAG / code / fixed steps:
|
|
139
|
+
**Manual (for custom providers or mocks):**
|
|
93
140
|
|
|
94
141
|
```ts
|
|
95
142
|
ctx.trace.recordLLMStep({
|
|
@@ -103,326 +150,125 @@ ctx.trace.recordToolCall({
|
|
|
103
150
|
args: { amount: 99.99 },
|
|
104
151
|
})
|
|
105
152
|
|
|
106
|
-
// Record custom workflow steps (RAG fetches, code/fixed steps, etc.)
|
|
107
153
|
ctx.trace.recordCustomStep({
|
|
108
|
-
kind: 'rag',
|
|
154
|
+
kind: 'rag',
|
|
109
155
|
name: 'pokemon-search',
|
|
110
|
-
|
|
111
|
-
payload: { query: 'pikachu attack' },
|
|
156
|
+
payload: { query: 'pikachu' },
|
|
112
157
|
result: { ids: [25] },
|
|
113
|
-
metadata: { latencyMs: 120 },
|
|
114
158
|
})
|
|
115
159
|
```
|
|
116
160
|
|
|
117
|
-
### Matchers
|
|
118
|
-
|
|
119
|
-
#### `toHaveLLMStep(config?)`
|
|
120
|
-
|
|
121
|
-
Assert the trace contains at least one LLM step matching the given config. All fields are optional and combined with AND logic.
|
|
161
|
+
### Common Matchers
|
|
122
162
|
|
|
123
163
|
```ts
|
|
164
|
+
// Assert LLM calls
|
|
124
165
|
expect(ctx.trace).toHaveLLMStep({ model: 'gpt-4' })
|
|
125
|
-
expect(ctx.trace).toHaveLLMStep({
|
|
126
|
-
expect(ctx.trace).toHaveLLMStep({ promptContains: 'order status' }) // searches prompt only
|
|
127
|
-
expect(ctx.trace).toHaveLLMStep({ outputContains: 'order confirmed' }) // searches completion only
|
|
128
|
-
expect(ctx.trace).toHaveLLMStep({ provider: 'openai' })
|
|
129
|
-
expect(ctx.trace).toHaveLLMStep({ provider: 'openai', promptContains: 'order status' })
|
|
130
|
-
expect(ctx.trace).toHaveLLMStep({ promptContains: 'retry', times: 3 }) // exactly 3 matching steps
|
|
131
|
-
expect(ctx.trace).toHaveLLMStep({ provider: 'openai', minTimes: 2 }) // at least 2 matching steps
|
|
132
|
-
expect(ctx.trace).toHaveLLMStep({ outputContains: 'error', maxTimes: 1 }) // at most 1 matching step
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
| Field | Description |
|
|
136
|
-
|---|---|
|
|
137
|
-
| `model` | Exact model name match (e.g. `'gpt-4o'`) |
|
|
138
|
-
| `contains` | Substring match across prompt + completion (case-insensitive) |
|
|
139
|
-
| `promptContains` | Substring match in prompt only (case-insensitive) |
|
|
140
|
-
| `outputContains` | Substring match in completion only (case-insensitive) |
|
|
141
|
-
| `provider` | Provider name: `'openai'`, `'gemini'`, or `'grok'` |
|
|
142
|
-
| `times` | Exact match count (fails unless exactly this many steps match) |
|
|
143
|
-
| `minTimes` | Minimum match count (steps matching must be ≥ this value) |
|
|
144
|
-
| `maxTimes` | Maximum match count (steps matching must be ≤ this value) |
|
|
166
|
+
expect(ctx.trace).toHaveLLMStep({ promptContains: 'order status' })
|
|
145
167
|
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
Assert the trace contains a tool call with the given name.
|
|
149
|
-
|
|
150
|
-
```ts
|
|
168
|
+
// Assert tool calls
|
|
151
169
|
expect(ctx.trace).toCallTool('chargeCard')
|
|
152
|
-
```
|
|
153
170
|
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
LLM-judged semantic match of combined LLM output vs. the expected string. Defaults to OpenAI GPT-4.1 with `OPENAI_API_KEY`. Optional options:
|
|
157
|
-
|
|
158
|
-
```ts
|
|
159
|
-
expect(ctx.trace).toMatchSemanticOutput('attack stat', {
|
|
160
|
-
provider: 'claude', // 'openai' (default) | 'claude' | 'gemini' | 'grok'
|
|
161
|
-
model: 'claude-3-opus-20240229', // overrides default model for the provider
|
|
162
|
-
sdk: myClaudeClient, // optional SDK instance (uses its chat/messages API)
|
|
163
|
-
})
|
|
164
|
-
|
|
165
|
-
// Minimal, using default OpenAI model
|
|
171
|
+
// Semantic output matching (LLM-judged)
|
|
166
172
|
expect(ctx.trace).toMatchSemanticOutput('order confirmed')
|
|
167
173
|
|
|
168
|
-
//
|
|
169
|
-
expect(ctx.trace).toMatchSemanticOutput('order confirmed', {
|
|
170
|
-
provider: 'openai',
|
|
171
|
-
model: 'kimi-k2-turbo-preview',
|
|
172
|
-
apiKey: process.env.KIMI_API_KEY,
|
|
173
|
-
baseURL: 'https://api.moonshot.ai/v1',
|
|
174
|
-
})
|
|
175
|
-
```
|
|
176
|
-
|
|
177
|
-
Environment keys by provider: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY` (or `GOOGLE_API_KEY`), `GROK_API_KEY`.
|
|
178
|
-
For OpenAI-compatible endpoints, pass `apiKey`/`baseURL` in options or set an appropriate env var used by your SDK.
|
|
179
|
-
|
|
180
|
-
#### `toEvaluateOutputMetric(config)`
|
|
181
|
-
|
|
182
|
-
Evaluate one LLM step’s prompt or result using an LLM and assert a numeric metric condition in the range 0.0–1.0. Defaults: target=`result`, condition=`atLeast 0.7`, provider=`openai`, model=`gpt-4.1`.
|
|
183
|
-
|
|
184
|
-
```ts
|
|
185
|
-
// Evaluate the last LLM result with your own prompt; default condition atLeast 0.7
|
|
186
|
-
expect(ctx.trace).toEvaluateOutputMetric({
|
|
187
|
-
evaluationPrompt: 'Rate how well this answers the user question.',
|
|
188
|
-
})
|
|
189
|
-
|
|
190
|
-
// Check a specific step (3rd LLM prompt), target the prompt text, require >= 0.8 via Claude
|
|
191
|
-
expect(ctx.trace).toEvaluateOutputMetric({
|
|
192
|
-
evaluationPrompt: 'Score coherence of this prompt between 0 and 1.',
|
|
193
|
-
target: 'prompt',
|
|
194
|
-
nth: 3,
|
|
195
|
-
condition: { atLeast: 0.8 },
|
|
196
|
-
provider: 'claude',
|
|
197
|
-
model: 'claude-3-opus-20240229',
|
|
198
|
-
})
|
|
199
|
-
|
|
200
|
-
// Custom comparator: score must be < 0.3
|
|
201
|
-
expect(ctx.trace).toEvaluateOutputMetric({
|
|
202
|
-
evaluationPrompt: 'Rate hallucination risk (0=none, 1=high).',
|
|
203
|
-
condition: { lessThan: 0.3 },
|
|
204
|
-
})
|
|
205
|
-
```
|
|
206
|
-
|
|
207
|
-
Options:
|
|
208
|
-
- `evaluationPrompt` (required): your scoring instructions; model is asked to return only a number between 0 and 1.
|
|
209
|
-
- `target`: `'result'` (default) or `'prompt'`. Mutually exclusive; evaluates that text only.
|
|
210
|
-
- `index` / `nth`: pick which LLM step to score (0-based or 1-based). Defaults to the last LLM step.
|
|
211
|
-
- `condition`: one of `greaterThan`, `lessThan`, `atLeast`, `atMost`, `equals`; default is `{ atLeast: 0.7 }`. Fails if the score is outside 0.0–1.0 or cannot be parsed.
|
|
212
|
-
- `provider` / `model` / `sdk` / `apiKey` / `baseURL`: same shape as `toMatchSemanticOutput` (supports OpenAI, Claude, Gemini, Grok, and OpenAI-compatible via `baseURL`). Requires corresponding API key if no SDK is supplied.
|
|
213
|
-
|
|
214
|
-
#### `toHaveCustomStep(config?)`
|
|
215
|
-
|
|
216
|
-
Assert a recorded custom step (RAG/code/fixed/custom) matches filters.
|
|
217
|
-
|
|
218
|
-
```ts
|
|
174
|
+
// Custom steps (RAG, code, fixed)
|
|
219
175
|
expect(ctx.trace).toHaveCustomStep({ kind: 'rag', name: 'pokemon-search' })
|
|
220
|
-
expect(ctx.trace).toHaveCustomStep({ tag: 'sort:asc' })
|
|
221
|
-
expect(ctx.trace).toHaveCustomStep({ contains: 'pikachu' })
|
|
222
|
-
expect(ctx.trace).toHaveCustomStep({ resultContains: '25' })
|
|
223
|
-
expect(ctx.trace).toHaveCustomStep({ kind: 'rag', minTimes: 1, maxTimes: 2 })
|
|
224
176
|
```
|
|
225
177
|
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
Filter prompts, then assert additional constraints. Example: “all prompts containing A must also contain B”.
|
|
229
|
-
|
|
230
|
-
```ts
|
|
231
|
-
// Prompts that contain "order" must also contain "confirmed"
|
|
232
|
-
expect(ctx.trace).toHavePromptWhere({
|
|
233
|
-
filterContains: 'order',
|
|
234
|
-
requireContains: 'confirmed',
|
|
235
|
-
})
|
|
236
|
-
|
|
237
|
-
// Prompts containing "retry" must NOT contain "cancel"
|
|
238
|
-
expect(ctx.trace).toHavePromptWhere({
|
|
239
|
-
filterContains: 'retry',
|
|
240
|
-
requireNotContains: 'cancel',
|
|
241
|
-
})
|
|
242
|
-
|
|
243
|
-
// And control counts on the filtered subset
|
|
244
|
-
expect(ctx.trace).toHavePromptWhere({
|
|
245
|
-
filterContains: 'order',
|
|
246
|
-
requireContains: 'confirmed',
|
|
247
|
-
minTimes: 1,
|
|
248
|
-
maxTimes: 3,
|
|
249
|
-
})
|
|
250
|
-
|
|
251
|
-
// Check a specific prompt position (1-based nth or 0-based index)
|
|
252
|
-
expect(ctx.trace).toHavePromptWhere({
|
|
253
|
-
filterContains: 'order',
|
|
254
|
-
requireContains: 'confirmed',
|
|
255
|
-
nth: 3, // the 3rd prompt among those containing "order"
|
|
256
|
-
})
|
|
257
|
-
```
|
|
178
|
+
**→ See [Test Matchers](docs/matchers.md) for complete documentation**
|
|
258
179
|
|
|
259
180
|
---
|
|
260
181
|
|
|
261
|
-
## Automatic AI
|
|
182
|
+
## Automatic AI & Tool Tracing
|
|
262
183
|
|
|
263
|
-
|
|
184
|
+
### AI Interception
|
|
264
185
|
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
| **Grok** (xAI) | `api.x.ai/v1/chat/completions` |
|
|
186
|
+
The runner automatically intercepts and records calls to:
|
|
187
|
+
- OpenAI (`api.openai.com`)
|
|
188
|
+
- Gemini (`generativelanguage.googleapis.com`)
|
|
189
|
+
- Grok/xAI (`api.x.ai`)
|
|
270
190
|
|
|
271
|
-
|
|
191
|
+
No code changes needed — just run your workflow and assertions work automatically.
|
|
272
192
|
|
|
273
|
-
|
|
274
|
-
aiTest('user lookup flow', async (ctx) => {
|
|
275
|
-
// This makes a real OpenAI call — intercepted automatically
|
|
276
|
-
await myWorkflow.run('Find all active users')
|
|
193
|
+
### Tool Recording
|
|
277
194
|
|
|
278
|
-
|
|
279
|
-
expect(ctx.trace).toHaveLLMStep({ promptContains: 'Find all active users' })
|
|
280
|
-
expect(ctx.trace).toHaveLLMStep({ provider: 'openai' })
|
|
281
|
-
})
|
|
282
|
-
```
|
|
283
|
-
|
|
284
|
-
**Streaming:** When `stream: true` is set on a request, the completion is recorded as `"(streamed)"` — the prompt and model are still captured.
|
|
285
|
-
|
|
286
|
-
**Libraries using `https.request` directly** (older versions of some SDKs) are not covered by fetch interception. Use manual `ctx.trace.recordLLMStep()` for those.
|
|
287
|
-
|
|
288
|
-
## Automatic Tool Recording
|
|
289
|
-
|
|
290
|
-
Tool functions are automatically wrapped and recorded when imported and called during workflow execution. This gives you tracing and replay support without any manual instrumentation.
|
|
291
|
-
|
|
292
|
-
### How to use tools
|
|
293
|
-
|
|
294
|
-
Define your tools in `ed_tools.ts` and export them:
|
|
195
|
+
Manual instrumentation pattern: isolate tracing in the service `.then/.catch` path so tracing failures never block business logic:
|
|
295
196
|
|
|
296
197
|
```ts
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
}
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
198
|
+
import { runSelectQuery } from './services/dataService'
|
|
199
|
+
|
|
200
|
+
export const dataService = async (input: any) => {
|
|
201
|
+
const { query } = input as { query: string }
|
|
202
|
+
return await runSelectQuery(query)
|
|
203
|
+
.then(async (res: any) => {
|
|
204
|
+
try {
|
|
205
|
+
const { recordToolCall } = await import('elasticdash-test')
|
|
206
|
+
recordToolCall('dataService', input, res)
|
|
207
|
+
} catch {
|
|
208
|
+
// tracing must never block the main service path
|
|
209
|
+
}
|
|
210
|
+
return res
|
|
211
|
+
})
|
|
212
|
+
.catch(async (err: any) => {
|
|
213
|
+
try {
|
|
214
|
+
const { recordToolCall } = await import('elasticdash-test')
|
|
215
|
+
recordToolCall('dataService', input, err)
|
|
216
|
+
} catch {
|
|
217
|
+
// tracing must never block the main service path
|
|
218
|
+
}
|
|
219
|
+
throw err
|
|
220
|
+
})
|
|
320
221
|
}
|
|
321
222
|
```
|
|
322
223
|
|
|
323
|
-
|
|
324
|
-
- Tool name
|
|
325
|
-
- Input arguments
|
|
326
|
-
- Output result
|
|
327
|
-
- Timestamp and duration
|
|
224
|
+
In manual mode, always isolate tracing in a separate `try/catch` so trace logging errors cannot interrupt core service execution.
|
|
328
225
|
|
|
329
|
-
|
|
226
|
+
**→ See [Tool Recording & Replay](docs/tools.md) for checkpoint-based replay and freezing**
|
|
330
227
|
|
|
331
|
-
|
|
332
|
-
- Local/normal execution: wrapped tools run normally (no replay context).
|
|
333
|
-
- Trace capture mode: wrapped tools run normally and are recorded.
|
|
334
|
-
- Replay mode (`Run from here`): pre-checkpoint wrapped calls return historical results and skip tool body execution.
|
|
228
|
+
### HTTP Streaming Capture and Replay
|
|
335
229
|
|
|
336
|
-
|
|
230
|
+
ElasticDash also captures non-AI `fetch` responses that stream over HTTP (for example SSE and NDJSON endpoints) in the HTTP interceptor.
|
|
337
231
|
|
|
338
|
-
|
|
232
|
+
Currently detected as streaming when response `content-type` includes:
|
|
233
|
+
- `text/event-stream`
|
|
234
|
+
- `application/x-ndjson`
|
|
235
|
+
- `application/stream+json`
|
|
236
|
+
- `application/jsonl`
|
|
339
237
|
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
- If a tool made 3 API calls before the checkpoint, those calls won't happen again on replay.
|
|
345
|
-
- If the tool is slow (waiting for external services), frozen calls are instant.
|
|
346
|
-
- The tool's frozen status is visible in the dashboard with a frozen tag/styling, matching AI step behavior.
|
|
347
|
-
- Side-effect replay is type-checked (`Date.now` vs `Math.random`) and malformed observation timestamps are auto-sanitized.
|
|
348
|
-
|
|
349
|
-
Example:
|
|
350
|
-
```
|
|
351
|
-
Task 1: fetchOrderStatus() ↓ frozen (checkpoint before this)
|
|
352
|
-
│─ API call to orders service ↓ frozen
|
|
353
|
-
└─ Result: { status: shipped } ↓ frozen
|
|
238
|
+
How it behaves today:
|
|
239
|
+
- During live execution, ElasticDash tees the response stream and returns a real stream to your app code.
|
|
240
|
+
- In parallel, ElasticDash buffers the recorder side of the stream as raw text for trace replay.
|
|
241
|
+
- During replay, ElasticDash reconstructs a stream from that captured raw payload and restores status, status text, and response headers.
|
|
354
242
|
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
```
|
|
358
|
-
|
|
359
|
-
### Recording tool calls manually
|
|
360
|
-
|
|
361
|
-
If you need to record tools outside the normal import flow or for providers not in `ed_tools.ts`, use:
|
|
362
|
-
|
|
363
|
-
```ts
|
|
364
|
-
ctx.trace.recordToolCall({
|
|
365
|
-
name: 'customTool',
|
|
366
|
-
args: { param: 'value' },
|
|
367
|
-
result: { success: true }
|
|
368
|
-
})
|
|
369
|
-
```
|
|
243
|
+
Replay fidelity note:
|
|
244
|
+
- Replay preserves stream payload content, but not original chunk boundaries or timing cadence.
|
|
370
245
|
|
|
371
|
-
|
|
246
|
+
Minimal stream consumption example:
|
|
372
247
|
|
|
373
248
|
```ts
|
|
374
|
-
|
|
249
|
+
const res = await fetch('https://example.com/events')
|
|
250
|
+
if (!res.body) throw new Error('Expected a streaming response body')
|
|
375
251
|
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
```
|
|
252
|
+
const reader = res.body.getReader()
|
|
253
|
+
const decoder = new TextDecoder()
|
|
254
|
+
let buffer = ''
|
|
380
255
|
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
```ts
|
|
386
|
-
// In your test
|
|
387
|
-
import { setCurrentTrace } from 'elasticdash-test'
|
|
388
|
-
|
|
389
|
-
aiTest('flow test', async (ctx) => {
|
|
390
|
-
setCurrentTrace(ctx.trace) // bind the trace to the current async context
|
|
391
|
-
await runFlowWithoutTraceArg() // your existing code
|
|
392
|
-
// assertions
|
|
393
|
-
expect(ctx.trace).toHaveCustomStep({ kind: 'rag', name: 'pokemon-search' })
|
|
394
|
-
})
|
|
395
|
-
|
|
396
|
-
// In your app/flow code (called during the test)
|
|
397
|
-
import { getCurrentTrace } from 'elasticdash-test'
|
|
398
|
-
|
|
399
|
-
function runFlowWithoutTraceArg() {
|
|
400
|
-
const trace = getCurrentTrace()
|
|
401
|
-
trace?.recordCustomStep({
|
|
402
|
-
kind: 'rag',
|
|
403
|
-
name: 'pokemon-search',
|
|
404
|
-
payload: { query: 'pikachu attack' },
|
|
405
|
-
result: { ids: [25] },
|
|
406
|
-
tags: ['source:db', 'sort:asc'],
|
|
407
|
-
})
|
|
256
|
+
for (;;) {
|
|
257
|
+
const { done, value } = await reader.read()
|
|
258
|
+
if (done) break
|
|
259
|
+
buffer += decoder.decode(value, { stream: true })
|
|
408
260
|
}
|
|
409
|
-
```
|
|
410
261
|
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
- Still compatible with manual DI: you can continue passing `ctx.trace` explicitly if you prefer.
|
|
262
|
+
buffer += decoder.decode()
|
|
263
|
+
```
|
|
414
264
|
|
|
415
|
-
|
|
416
|
-
- Opt in by setting `ELASTICDASH_LLM_PROXY=1` (optional: `ELASTICDASH_LLM_PROXY_PORT`, default `8787`). The runner will start a local proxy and generate a per-test `ELASTICDASH_TRACE_ID`.
|
|
417
|
-
- Point your LLM client at the proxy via base URL envs (e.g., `OPENAI_BASE_URL=http://localhost:8787/v1`, `ANTHROPIC_API_URL=http://localhost:8787`). No code changes in your workflow; only env overrides when running tests.
|
|
418
|
-
- Forward the trace ID to your Edge/Deno code (e.g., add `x-trace-id: process.env.ELASTICDASH_TRACE_ID` on the request to your Supabase Edge function). The proxy records model/prompt/completion (or `(streamed)`) and the runner folds captured steps back into `ctx.trace` after each test.
|
|
419
|
-
- When `ELASTICDASH_LLM_PROXY` is unset, behavior is unchanged and the existing Node fetch interceptor remains the default.
|
|
265
|
+
**→ See [Quick Start Guide](docs/quickstart.md#capture-streaming-flows) for end-to-end setup guidance**
|
|
420
266
|
|
|
421
267
|
---
|
|
422
268
|
|
|
423
269
|
## Configuration
|
|
424
270
|
|
|
425
|
-
|
|
271
|
+
Optional `elasticdash.config.ts` at project root:
|
|
426
272
|
|
|
427
273
|
```ts
|
|
428
274
|
export default {
|
|
@@ -431,311 +277,33 @@ export default {
|
|
|
431
277
|
}
|
|
432
278
|
```
|
|
433
279
|
|
|
434
|
-
|
|
435
|
-
|---|---|---|
|
|
436
|
-
| `testMatch` | `['**/*.ai.test.ts']` | Glob patterns for test discovery |
|
|
437
|
-
| `traceMode` | `'local'` | `'local'` (stub) or `'remote'` (future ElasticDash backend) |
|
|
438
|
-
|
|
439
|
-
---
|
|
280
|
+
Optional project file: `ed_workers.ts` can be used by your app architecture (for example, exporting worker handlers), but it is not required or discovered by the ElasticDash CLI/dashboard.
|
|
440
281
|
|
|
441
282
|
## TypeScript Setup
|
|
442
283
|
|
|
443
|
-
|
|
284
|
+
For typed globals and matchers, extend your test directory's `tsconfig.json`:
|
|
444
285
|
|
|
445
286
|
```json
|
|
446
287
|
{
|
|
447
288
|
"extends": "../tsconfig.json",
|
|
448
|
-
"
|
|
449
|
-
"rootDir": "..",
|
|
450
|
-
"noEmit": true
|
|
451
|
-
},
|
|
452
|
-
"include": [
|
|
453
|
-
"../src/**/*",
|
|
454
|
-
"./**/*"
|
|
455
|
-
]
|
|
289
|
+
"include": ["../src/**/*", "./**/*"]
|
|
456
290
|
}
|
|
457
291
|
```
|
|
458
292
|
|
|
459
|
-
This gives you typed `aiTest`, `beforeAll`, `afterAll` globals and typed custom matchers in your test files.
|
|
460
|
-
|
|
461
|
-
---
|
|
462
|
-
|
|
463
|
-
## Workflows Dashboard
|
|
464
|
-
|
|
465
|
-
Browse and search all available workflow functions in your project:
|
|
466
|
-
|
|
467
|
-
```bash
|
|
468
|
-
elasticdash dashboard # open dashboard at http://localhost:4573
|
|
469
|
-
elasticdash dashboard --port 4572 # use custom port
|
|
470
|
-
elasticdash dashboard --no-open # skip auto-opening browser
|
|
471
|
-
```
|
|
472
|
-
|
|
473
|
-
The dashboard scans your workflow/tool files and displays:
|
|
474
|
-
- If both `.ts` and `.js` versions of a file exist (e.g., `ed_workflows.ts` and `ed_workflows.js`), the dashboard will always use the `.ts` file.
|
|
475
|
-
- If only `.ts` exists, it will be automatically transpiled to `.js` before scanning/importing—no manual build step required.
|
|
476
|
-
- If only `.js` exists, it will be used directly.
|
|
477
|
-
|
|
478
|
-
This means you can write your workflows and tools in TypeScript, and the dashboard will handle transpilation automatically. You do not need to run `tsc` or build manually for dashboard usage.
|
|
479
|
-
|
|
480
|
-
**Example file selection logic:**
|
|
481
|
-
| Scenario | File Used |
|
|
482
|
-
|-------------------------|------------------|
|
|
483
|
-
| Only `ed_workflows.ts` | Transpiled `.ts` |
|
|
484
|
-
| Only `ed_workflows.js` | `.js` |
|
|
485
|
-
| Both exist | `.ts` (preferred)|
|
|
486
|
-
|
|
487
|
-
The dashboard displays:
|
|
488
|
-
- **Function names** — all exported functions in the module
|
|
489
|
-
- **Signatures** — function parameters and return types
|
|
490
|
-
- **Async indicator** — marks async vs sync functions
|
|
491
|
-
- **Source module** — where the function is imported from (if re-exported)
|
|
492
|
-
- **File path** — location of the workflow file
|
|
493
|
-
|
|
494
|
-
Use the search field to filter workflows by:
|
|
495
|
-
- **Name** — find workflow by function name (e.g., `checkoutFlow`)
|
|
496
|
-
- **Source module** — find all workflows from a specific module (e.g., `app.workflows`)
|
|
497
|
-
- **File path** — filter by location in your codebase
|
|
498
|
-
|
|
499
|
-
This is useful for discovering available workflows, understanding their signatures, and identifying where functions are defined before calling them in tests.
|
|
500
|
-
|
|
501
|
-
### `ed_workflows.ts`, `ed_tools.ts`, `ed_agents.ts`
|
|
502
|
-
|
|
503
|
-
These optional files bundle and re-export existing functions from your codebase for use in tests.
|
|
504
|
-
|
|
505
|
-
#### `ed_workflows.ts`
|
|
506
|
-
|
|
507
|
-
Re-export workflow functions from your application:
|
|
508
|
-
|
|
509
|
-
```ts
|
|
510
|
-
// ed_workflows.ts
|
|
511
|
-
export { orderWorkflow, refundWorkflow } from './src/workflows'
|
|
512
|
-
export { userLookupFlow } from './src/user-flows'
|
|
513
|
-
```
|
|
514
|
-
|
|
515
|
-
Access in tests:
|
|
516
|
-
|
|
517
|
-
```ts
|
|
518
|
-
import { orderWorkflow } from './ed_workflows'
|
|
519
|
-
|
|
520
|
-
aiTest('full order workflow', async (ctx) => {
|
|
521
|
-
const result = await orderWorkflow('order-123', 'cust-456')
|
|
522
|
-
expect(ctx.trace).toCallTool('chargeCard')
|
|
523
|
-
})
|
|
524
|
-
```
|
|
525
|
-
|
|
526
|
-
#### `ed_tools.ts`
|
|
527
|
-
|
|
528
|
-
Re-export tool functions that agents or workflows can invoke:
|
|
529
|
-
|
|
530
|
-
```ts
|
|
531
|
-
// ed_tools.ts
|
|
532
|
-
import { wrapTool } from 'elasticdash-test'
|
|
533
|
-
import { chargeCard as chargeCardRaw, fetchOrderStatus as fetchOrderStatusRaw, sendNotification as sendNotificationRaw } from './src/tools'
|
|
534
|
-
|
|
535
|
-
export const chargeCard = wrapTool('chargeCard', chargeCardRaw)
|
|
536
|
-
export const fetchOrderStatus = wrapTool('fetchOrderStatus', fetchOrderStatusRaw)
|
|
537
|
-
export const sendNotification = wrapTool('sendNotification', sendNotificationRaw)
|
|
538
|
-
```
|
|
539
|
-
|
|
540
|
-
#### `ed_agents.ts`
|
|
541
|
-
|
|
542
|
-
Re-export agent functions or create a config object:
|
|
543
|
-
|
|
544
|
-
```ts
|
|
545
|
-
// ed_agents.ts
|
|
546
|
-
export { checkoutAgent, paymentAgent } from './src/agents'
|
|
547
|
-
|
|
548
|
-
// Or as a config object:
|
|
549
|
-
export const agents = {
|
|
550
|
-
checkout: checkoutAgent,
|
|
551
|
-
payment: paymentAgent,
|
|
552
|
-
}
|
|
553
|
-
```
|
|
554
|
-
|
|
555
|
-
The dashboard command will scan these files and display all exported functions with their signatures, making it easy to explore your workflow API.
|
|
556
|
-
|
|
557
|
-
---
|
|
558
|
-
|
|
559
|
-
## Agent Mid-Trace Replay
|
|
560
|
-
|
|
561
|
-
ElasticDash supports resuming a long-running agent from any task in its plan — without re-executing already-completed steps. This is useful for:
|
|
562
|
-
|
|
563
|
-
- **Resuming after failures**: If task 3 of 5 fails, fix the issue and re-run from task 3 only.
|
|
564
|
-
- **Pausing for approval**: Capture state after task 2, get human sign-off, then continue.
|
|
565
|
-
- **Debugging in isolation**: Re-run a single task with modified input to diagnose a problem.
|
|
566
|
-
|
|
567
|
-
### How it works
|
|
568
|
-
|
|
569
|
-
Agents are structured as an **AgentPlan** — an ordered list of **AgentTask** objects, each with a tool name, input, and output. When serialized, the plan plus all captured trace events form an **AgentState** that can be saved to disk/database and replayed later.
|
|
570
|
-
|
|
571
|
-
### Quick start
|
|
572
|
-
|
|
573
|
-
```ts
|
|
574
|
-
import { plannerAgent, executorAgent, resumeAgentFromTrace } from './ed_agents'
|
|
575
|
-
import { serializeAgentState, deserializeAgentState } from 'elasticdash-test'
|
|
576
|
-
import fs from 'node:fs'
|
|
577
|
-
|
|
578
|
-
// 1. Generate a plan
|
|
579
|
-
const plan = await plannerAgent('Show me sales for Q1', { userToken: 'tok-abc' })
|
|
580
|
-
|
|
581
|
-
// 2. Execute the plan (runs all tasks sequentially)
|
|
582
|
-
const completedPlan = await executorAgent(plan)
|
|
583
|
-
|
|
584
|
-
// 3. Serialize and save state (e.g., after partial execution)
|
|
585
|
-
const state = serializeAgentState(completedPlan, [] /* pass recorder.events in worker context */)
|
|
586
|
-
fs.writeFileSync('agent-state.json', JSON.stringify(state, null, 2))
|
|
587
|
-
|
|
588
|
-
// 4. Later: load saved state and resume from task 2 (0-based index 1)
|
|
589
|
-
const savedState = JSON.parse(fs.readFileSync('agent-state.json', 'utf8'))
|
|
590
|
-
const stateToResume = deserializeAgentState({ ...savedState, resumeFromTaskIndex: 1 })
|
|
591
|
-
const resumedPlan = await resumeAgentFromTrace(stateToResume)
|
|
592
|
-
|
|
593
|
-
console.log('Resumed plan status:', resumedPlan.status)
|
|
594
|
-
console.log('Task outputs:', resumedPlan.tasks.map((t) => ({ id: t.id, status: t.status })))
|
|
595
|
-
```
|
|
596
|
-
|
|
597
|
-
### AgentState structure
|
|
598
|
-
|
|
599
|
-
```ts
|
|
600
|
-
interface AgentState {
|
|
601
|
-
plan: AgentPlan // Full plan with all tasks (completed and pending)
|
|
602
|
-
trace: WorkflowEvent[] // Captured trace events from previous execution
|
|
603
|
-
resumeFromTaskIndex: number // Zero-based index — tasks before this are loaded from cache
|
|
604
|
-
}
|
|
605
|
-
|
|
606
|
-
interface AgentPlan {
|
|
607
|
-
id: string
|
|
608
|
-
tasks: AgentTask[]
|
|
609
|
-
status: 'planning' | 'executing' | 'completed' | 'failed' | 'paused'
|
|
610
|
-
currentTaskIndex: number
|
|
611
|
-
context: Record<string, unknown>
|
|
612
|
-
metadata: Record<string, unknown>
|
|
613
|
-
}
|
|
614
|
-
|
|
615
|
-
interface AgentTask {
|
|
616
|
-
id: string
|
|
617
|
-
status: 'pending' | 'in-progress' | 'completed' | 'failed'
|
|
618
|
-
description: string
|
|
619
|
-
tool: string // Name of the tool function to invoke
|
|
620
|
-
input: unknown // May contain { $ref: "task-N.output.fieldName" } placeholders
|
|
621
|
-
output?: unknown // Populated after execution
|
|
622
|
-
error?: string
|
|
623
|
-
startedAt?: number
|
|
624
|
-
completedAt?: number
|
|
625
|
-
}
|
|
626
|
-
```
|
|
627
|
-
|
|
628
|
-
### Task input placeholders
|
|
629
|
-
|
|
630
|
-
Task inputs can reference previous task outputs using `{ $ref: "taskId.output.fieldPath" }`:
|
|
631
|
-
|
|
632
|
-
```ts
|
|
633
|
-
// task-2 uses the embedding produced by task-1
|
|
634
|
-
{
|
|
635
|
-
id: 'task-2',
|
|
636
|
-
tool: 'taskSelectorService',
|
|
637
|
-
input: {
|
|
638
|
-
queryEmbedding: { $ref: 'task-1.output.embedding' },
|
|
639
|
-
topK: 3,
|
|
640
|
-
}
|
|
641
|
-
}
|
|
642
|
-
```
|
|
643
|
-
|
|
644
|
-
Placeholders are resolved at execution time by `resolveTaskInput()`.
|
|
645
|
-
|
|
646
|
-
### Dashboard integration
|
|
647
|
-
|
|
648
|
-
When running an agent workflow through the dashboard:
|
|
649
|
-
|
|
650
|
-
1. Agent task observations are **visually highlighted** with a purple background and left border, making them easy to identify.
|
|
651
|
-
2. Each agent observation shows a **T1 / T2 / T3** badge indicating which task it belongs to.
|
|
652
|
-
3. In the observation detail panel, a **"Resume from Task N"** button appears (agent steps only — non-agent steps have no button).
|
|
653
|
-
4. Clicking it calls `/api/resume-agent-from-task` with the serialized `AgentState` and the chosen `taskIndex`, then adds the resumed run as a new trace in the comparison table.
|
|
654
|
-
|
|
655
|
-
### Best practices for resumable agents
|
|
656
|
-
|
|
657
|
-
- **Keep tasks idempotent** where possible — if a task must be re-run, ensure it produces the same result.
|
|
658
|
-
- **Store minimal outputs** — only record what downstream tasks need, not full API responses.
|
|
659
|
-
- **Version your state schema** — if tool interfaces change, old states may need migration.
|
|
660
|
-
- **Use sequential tasks** — the current implementation runs tasks one-by-one; parallel task support is a planned future enhancement.
|
|
661
|
-
|
|
662
|
-
---
|
|
663
|
-
|
|
664
|
-
## Project Structure
|
|
665
|
-
|
|
666
|
-
```
|
|
667
|
-
src/
|
|
668
|
-
cli.ts CLI entry point (commander + fast-glob)
|
|
669
|
-
runner.ts Sequential test runner engine
|
|
670
|
-
reporter.ts Color-coded terminal output
|
|
671
|
-
test-setup.ts Import in test files for globals + matcher types
|
|
672
|
-
index.ts Programmatic API
|
|
673
|
-
core/
|
|
674
|
-
registry.ts aiTest / beforeAll / afterAll registry
|
|
675
|
-
trace-adapter/
|
|
676
|
-
context.ts TraceHandle, AITestContext, RunnerHooks scaffold
|
|
677
|
-
matchers/
|
|
678
|
-
index.ts Custom expect matchers
|
|
679
|
-
interceptors/
|
|
680
|
-
ai-interceptor.ts Automatic fetch interceptor for OpenAI / Gemini / Grok
|
|
681
|
-
```
|
|
682
|
-
|
|
683
293
|
---
|
|
684
294
|
|
|
685
295
|
## Programmatic API
|
|
686
296
|
|
|
687
297
|
```ts
|
|
688
|
-
import { runFiles, reportResults, registerMatchers, installAIInterceptor
|
|
298
|
+
import { runFiles, reportResults, registerMatchers, installAIInterceptor } from 'elasticdash-test'
|
|
689
299
|
|
|
690
300
|
registerMatchers()
|
|
691
|
-
installAIInterceptor()
|
|
301
|
+
installAIInterceptor()
|
|
692
302
|
|
|
693
303
|
const results = await runFiles(['./tests/flow.ai.test.ts'])
|
|
694
304
|
reportResults(results)
|
|
695
|
-
|
|
696
|
-
uninstallAIInterceptor() // restore original fetch when done
|
|
697
|
-
```
|
|
698
|
-
|
|
699
|
-
---
|
|
700
|
-
|
|
701
|
-
## Recording Tool Calls Explicitly
|
|
702
|
-
|
|
703
|
-
If you want to ensure tool calls are always recorded in the workflow trace—regardless of how your tools are imported or used—you can use the `recordToolCall` utility provided by this SDK.
|
|
704
|
-
|
|
705
|
-
### How to Use
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
1. Import the function in your tool implementation:
|
|
709
|
-
|
|
710
|
-
```ts
|
|
711
|
-
import { recordToolCall } from 'elasticdash-test'
|
|
712
|
-
|
|
713
|
-
export async function myTool(input: string) {
|
|
714
|
-
// ...tool logic...
|
|
715
|
-
const result = `Hello, ${input}!`
|
|
716
|
-
recordToolCall('myTool', { input }, result)
|
|
717
|
-
return result
|
|
718
|
-
}
|
|
719
305
|
```
|
|
720
306
|
|
|
721
|
-
2. When running under ElasticDash, all calls to `recordToolCall` will be captured in the workflow trace. When running locally or outside the runner, this function is a no-op and will not affect your code.
|
|
722
|
-
|
|
723
|
-
**This approach is robust and works with both imported and global tools.**
|
|
724
|
-
|
|
725
|
-
**Summary:**
|
|
726
|
-
- Use tools as globals (no import) for full traceability and automatic tool call capture.
|
|
727
|
-
- This approach keeps your workflow code clean and enables powerful debugging and analytics in ElasticDash.
|
|
728
|
-
|
|
729
|
-
## Non-Goals
|
|
730
|
-
|
|
731
|
-
This runner intentionally does not support:
|
|
732
|
-
|
|
733
|
-
- Parallel execution
|
|
734
|
-
- Watch mode
|
|
735
|
-
- Snapshot testing
|
|
736
|
-
- Coverage reporting
|
|
737
|
-
- Jest compatibility
|
|
738
|
-
|
|
739
307
|
---
|
|
740
308
|
|
|
741
309
|
## License
|