elasticdash-test 0.1.14 → 0.1.16

package/README.md

@@ -28,7 +28,7 @@ An AI-native test runner for ElasticDash workflow testing. Built for async AI pi
 - 🎯 **Trace-first testing** — every test gets a `trace` context to record and assert on LLM calls and tool invocations
 - 🔍 **Automatic AI interception** — captures OpenAI, Gemini, and Grok calls without code changes
 - 🧪 **AI-specific matchers** — semantic output matching, LLM-judged evaluations, prompt assertions
-- 🛠️ **Tool recording & replay** — automatically trace tool calls with checkpoint-based replay
+- 🛠️ **Tool & LLM recording & replay** — automatically trace tool and AI calls with checkpoint-based replay and mock support
 - 📊 **Interactive dashboard** — browse workflows, debug traces, validate fixes visually
 - 🤖 **Agent mid-trace replay** — resume long-running agents from any task without re-execution
 - 🌐 **HTTP workflow mode** — run workflows against your live dev server for framework-heavy apps (Next.js, Remix, etc.) with full AI and tool call observability
@@ -266,6 +266,8 @@ export const callClaude = wrapAI('claude-sonnet-4-5', async (messages: Anthropic
 
 Use `wrapAI` when you have a custom AI wrapper or a provider not covered by automatic interception. For direct OpenAI/Anthropic/Gemini SDK calls inside a subprocess workflow, automatic interception via `installAIInterceptor` already handles recording without any code changes.
 
+**AI mocking (subprocess / test runner mode):** `wrapAI` also checks `resolveAIMock` at call time, so the dashboard can mock LLM responses the same way it mocks tool calls — without modifying your server code. Configure an `AIMockConfig` in the dashboard UI or pass it programmatically via the `aiMockConfig` option when running a workflow.
+
 ### HTTP Streaming Capture and Replay
 
 ElasticDash also captures non-AI `fetch` responses that stream over HTTP (for example SSE and NDJSON endpoints) in the HTTP interceptor.
@@ -354,6 +356,25 @@ The dashboard injects `x-elasticdash-run-id` and `x-elasticdash-server` headers
 
 > **Note:** Use `setHttpRunContext` (synchronous) if you only need observability and do not need step freezing. `initHttpRunContext` is required for the dashboard's breakpoint/replay functionality to work.
 
+### Dashboard Auto-Detection (env var mode)
+
+As an alternative to calling `initHttpRunContext` in your request handler, you can set two environment variables before starting your server or script. Every `wrapTool` and `wrapAI` call will then connect to the dashboard automatically — no code changes needed:
+
+```bash
+# Required: URL of the running ElasticDash dashboard
+ELASTICDASH_SERVER=http://localhost:4573
+
+# Optional: pre-registered run ID to fetch frozen steps for
+ELASTICDASH_RUN_ID=<run-id-from-dashboard>
+```
+
+- If only `ELASTICDASH_SERVER` is set, a fresh run ID is generated and all calls push live telemetry to the dashboard (observability only, no step freezing).
+- If both variables are set, frozen steps are fetched from the dashboard at startup and replayed as configured.
+- If the dashboard is unreachable the SDK falls through to live execution silently.
+- The initialization runs **once per process** — subsequent `wrapTool`/`wrapAI` calls reuse the cached context.
+
+This mode is intended for local development and testing scenarios. For production HTTP servers with concurrent requests, continue using `initHttpRunContext` inside your request handler.
+
 **Subprocess vs HTTP mode comparison:**
 
 | | Subprocess (default) | HTTP mode |
@@ -361,9 +382,10 @@ The dashboard injects `x-elasticdash-run-id` and `x-elasticdash-server` headers
 | Works with simple apps | Yes | Yes |
 | Works with Next.js / Remix | No | Yes |
 | Requires dev server running | No | Yes |
-| App code changes needed | Extract handler to `ed_workflows.ts` | Add `initHttpRunContext` to request handler |
+| App code changes needed | Extract handler to `ed_workflows.ts` | Add `initHttpRunContext` to request handler (or use env vars for auto-detect) |
 | AI / tool call observability | Automatic via interceptors | Via `wrapAI` / `wrapTool` push |
-| Step freezing / breakpoints | Yes | Yes (requires `initHttpRunContext`) |
+| Step freezing / breakpoints | Yes | Yes (`initHttpRunContext`, or `ELASTICDASH_SERVER` + `ELASTICDASH_RUN_ID` env vars) |
+| LLM response mocking | Yes (via `aiMockConfig`) | Yes (via frozen AI events) |
 
 ---
 
@@ -420,16 +442,25 @@ reportResults(results)
 **HTTP mode context (call inside your request handler):**
 
 ```ts
-import { initHttpRunContext } from 'elasticdash-test'
+import { initHttpRunContext, setHttpRunContext } from 'elasticdash-test'
 
 // Async — fetches frozen steps from dashboard to enable step freezing/breakpoints
 await initHttpRunContext(runId, dashboardUrl)
 
 // Synchronous alternative — observability only, no step freezing
-import { setHttpRunContext } from 'elasticdash-test'
 setHttpRunContext(runId, dashboardUrl)
 ```
 
+**Dashboard auto-detection (env var mode — no code changes needed):**
+
+```bash
+# Set before starting your server or script
+ELASTICDASH_SERVER=http://localhost:4573       # required
+ELASTICDASH_RUN_ID=<run-id-from-dashboard>    # optional, enables step freezing
+```
+
+`wrapTool` and `wrapAI` will auto-connect on their first call. See [Dashboard Auto-Detection](#dashboard-auto-detection-env-var-mode) for details.
+
 ---
 
 ## License

package/dist/dashboard-server.d.ts

@@ -40,6 +40,15 @@ export interface ToolMockEntry {
 export interface ToolMockConfig {
     [toolName: string]: ToolMockEntry;
 }
+/** Per-model AI mock configuration sent from the dashboard UI */
+export interface AIMockEntry {
+    mode: 'live' | 'mock-all' | 'mock-specific';
+    callIndices?: number[];
+    mockData?: Record<number, unknown>;
+}
+export interface AIMockConfig {
+    [modelName: string]: AIMockEntry;
+}
 export interface HttpWorkflowConfig {
     mode: 'http';
     url: string;

package/dist/dashboard-server.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"dashboard-server.d.ts","sourceRoot":"","sources":["../src/dashboard-server.ts"],"names":[],"mappings":"AAeA,MAAM,WAAW,YAAY;IAC3B,IAAI,EAAE,MAAM,CAAA;IACZ,OAAO,EAAE,OAAO,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,QAAQ,EAAE,MAAM,CAAA;IAChB,UAAU,CAAC,EAAE,MAAM,CAAA;IACnB,UAAU,CAAC,EAAE,MAAM,CAAA;IACnB,YAAY,CAAC,EAAE,MAAM,CAAA;IACrB,UAAU,CAAC,EAAE,MAAM,CAAA;CACpB;AAED,MAAM,WAAW,QAAQ;IACvB,IAAI,EAAE,MAAM,CAAA;IACZ,OAAO,EAAE,OAAO,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,QAAQ,EAAE,MAAM,CAAA;IAChB,UAAU,CAAC,EAAE,MAAM,CAAA;IACnB,UAAU,CAAC,EAAE,MAAM,CAAA;CACpB;AAED,MAAM,WAAW,SAAS;IACxB,SAAS,EAAE,YAAY,EAAE,CAAA;IACzB,KAAK,EAAE,QAAQ,EAAE,CAAA;CAClB;AAED,MAAM,WAAW,sBAAsB;IACrC,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,QAAQ,CAAC,EAAE,OAAO,CAAA;CACnB;AAED,MAAM,WAAW,eAAe;IAC9B,GAAG,EAAE,MAAM,CAAA;IACX,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAA;CACvB;AA2CD,6DAA6D;AAC7D,MAAM,WAAW,aAAa;IAC5B,oHAAoH;IACpH,IAAI,EAAE,MAAM,GAAG,UAAU,GAAG,eAAe,CAAA;IAC3C,uEAAuE;IACvE,WAAW,CAAC,EAAE,MAAM,EAAE,CAAA;IACtB,wEAAwE;IACxE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;CACnC;AAED,MAAM,WAAW,cAAc;IAC7B,CAAC,QAAQ,EAAE,MAAM,GAAG,aAAa,CAAA;CAClC;AA+1FD,MAAM,WAAW,kBAAkB;IACjC,IAAI,EAAE,MAAM,CAAA;IACZ,GAAG,EAAE,MAAM,CAAA;IACX,MAAM,CAAC,EAAE,MAAM,CAAA;IACf,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAA;IAChC,YAAY,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;IACtC,cAAc,CAAC,EAAE,kBAAkB,GAAG,MAAM,CAAA;CAC7C;AA0ID;;GAEG;AACH,wBAAsB,oBAAoB,CACxC,GAAG,EAAE,MAAM,EACX,OAAO,GAAE,sBAA2B,GACnC,OAAO,CAAC,eAAe,CAAC,CAoZ1B;AAiFD,eAAO,MAAM,aAAa,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAa,CAAC"}
+{"version":3,"file":"dashboard-server.d.ts","sourceRoot":"","sources":["../src/dashboard-server.ts"],"names":[],"mappings":"AAeA,MAAM,WAAW,YAAY;IAC3B,IAAI,EAAE,MAAM,CAAA;IACZ,OAAO,EAAE,OAAO,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,QAAQ,EAAE,MAAM,CAAA;IAChB,UAAU,CAAC,EAAE,MAAM,CAAA;IACnB,UAAU,CAAC,EAAE,MAAM,CAAA;IACnB,YAAY,CAAC,EAAE,MAAM,CAAA;IACrB,UAAU,CAAC,EAAE,MAAM,CAAA;CACpB;AAED,MAAM,WAAW,QAAQ;IACvB,IAAI,EAAE,MAAM,CAAA;IACZ,OAAO,EAAE,OAAO,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,QAAQ,EAAE,MAAM,CAAA;IAChB,UAAU,CAAC,EAAE,MAAM,CAAA;IACnB,UAAU,CAAC,EAAE,MAAM,CAAA;CACpB;AAED,MAAM,WAAW,SAAS;IACxB,SAAS,EAAE,YAAY,EAAE,CAAA;IACzB,KAAK,EAAE,QAAQ,EAAE,CAAA;CAClB;AAED,MAAM,WAAW,sBAAsB;IACrC,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,QAAQ,CAAC,EAAE,OAAO,CAAA;CACnB;AAED,MAAM,WAAW,eAAe;IAC9B,GAAG,EAAE,MAAM,CAAA;IACX,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAA;CACvB;AA2CD,6DAA6D;AAC7D,MAAM,WAAW,aAAa;IAC5B,oHAAoH;IACpH,IAAI,EAAE,MAAM,GAAG,UAAU,GAAG,eAAe,CAAA;IAC3C,uEAAuE;IACvE,WAAW,CAAC,EAAE,MAAM,EAAE,CAAA;IACtB,wEAAwE;IACxE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;CACnC;AAED,MAAM,WAAW,cAAc;IAC7B,CAAC,QAAQ,EAAE,MAAM,GAAG,aAAa,CAAA;CAClC;AAED,iEAAiE;AACjE,MAAM,WAAW,WAAW;IAC1B,IAAI,EAAE,MAAM,GAAG,UAAU,GAAG,eAAe,CAAA;IAC3C,WAAW,CAAC,EAAE,MAAM,EAAE,CAAA;IACtB,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;CACnC;AAED,MAAM,WAAW,YAAY;IAC3B,CAAC,SAAS,EAAE,MAAM,GAAG,WAAW,CAAA;CACjC;AAqgGD,MAAM,WAAW,kBAAkB;IACjC,IAAI,EAAE,MAAM,CAAA;IACZ,GAAG,EAAE,MAAM,CAAA;IACX,MAAM,CAAC,EAAE,MAAM,CAAA;IACf,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAA;IAChC,YAAY,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;IACtC,cAAc,CAAC,EAAE,kBAAkB,GAAG,MAAM,CAAA;CAC7C;AA4ID;;GAEG;AACH,wBAAsB,oBAAoB,CACxC,GAAG,EAAE,MAAM,EACX,OAAO,GAAE,sBAA2B,GACnC,OAAO,CAAC,eAAe,CAAC,CA4b1B;AAiFD,eAAO,MAAM,aAAa,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAa,CAAC"}

package/dist/dashboard-server.js

@@ -311,6 +311,8 @@ function runWorkflowInSubprocess(workflowsModulePath, toolsModulePath, workflowN
             ...(options?.history !== undefined ? { history: options.history } : {}),
             ...(options?.agentState !== undefined ? { agentState: options.agentState } : {}),
             ...(options?.toolMockConfig !== undefined ? { toolMockConfig: options.toolMockConfig } : {}),
+            ...(options?.aiMockConfig !== undefined ? { aiMockConfig: options.aiMockConfig } : {}),
+            ...(options?.promptMockConfig !== undefined ? { promptMockConfig: options.promptMockConfig } : {}),
         });
         child.stdin.write(payload);
         child.stdin.end(); // Always close stdin to avoid subprocess hang
@@ -633,6 +635,14 @@ async function validateWorkflowRuns(cwd, body) {
     const toolMockConfig = body.toolMockConfig && typeof body.toolMockConfig === 'object' && !Array.isArray(body.toolMockConfig)
         ? body.toolMockConfig
         : undefined;
+    // Parse AI mock config if provided
+    const aiMockConfig = body.aiMockConfig && typeof body.aiMockConfig === 'object' && !Array.isArray(body.aiMockConfig)
+        ? body.aiMockConfig
+        : undefined;
+    // Parse prompt mock config if provided
+    const promptMockConfig = body.promptMockConfig && typeof body.promptMockConfig === 'object' && !Array.isArray(body.promptMockConfig)
+        ? body.promptMockConfig
+        : undefined;
     const workflowsModulePath = resolveWorkflowModule(cwd);
     if (!workflowsModulePath) {
         return {
@@ -648,7 +658,7 @@ async function validateWorkflowRuns(cwd, body) {
     console.log(`[elasticdash] Running workflow "${workflowName}" ${runCount} time(s) in ${mode} mode via subprocess`);
     async function runOne(runNumber) {
         console.log(`[elasticdash] === Run ${runNumber}: Starting workflow "${workflowName}" ===`);
-        const result = await runWorkflowInSubprocess(workflowsModulePath, toolsModulePath, workflowName, workflowArgs, workflowInput, toolMockConfig ? { toolMockConfig } : undefined)
+        const result = await runWorkflowInSubprocess(workflowsModulePath, toolsModulePath, workflowName, workflowArgs, workflowInput, (toolMockConfig || aiMockConfig || promptMockConfig) ? { ...(toolMockConfig ? { toolMockConfig } : {}), ...(aiMockConfig ? { aiMockConfig } : {}), ...(promptMockConfig ? { promptMockConfig } : {}) } : undefined)
             .catch(err => {
             throw { ok: false, error: `Workflow subprocess failed: ${formatError(err)}` };
         });
@@ -1267,6 +1277,10 @@ function getDashboardHtml() {
             // ---- Tool Mock Config State ----
             window._toolMockConfig = {}; // { toolName: { mode: 'live'|'mock-all'|'mock-specific', callIndices: [], mockData: {} } }
 
+            // ---- Prompt Mock Config State ----
+            // { [originalSystemPrompt]: newSystemPrompt } — only keys where user enabled the override
+            window._promptMockConfig = {};
+
             function getToolsFromTrace() {
                 // Extract unique tool names and their call details from the uploaded trace observations
                 const toolCalls = {};
@@ -1422,6 +1436,136 @@ function getDashboardHtml() {
                 return html;
             }
 
+            // ---- Prompt Mock Helpers ----
+
+            /** Extract the system prompt string from an LLM call input object or JSON string. */
+            function extractSystemPromptFromInput(input) {
+                // Input may arrive as a JSON-encoded string (e.g. from Langfuse traces)
+                if (typeof input === 'string') {
+                    try { input = JSON.parse(input); } catch(e) { return null; }
+                }
+                if (!input || typeof input !== 'object') return null;
+                // Anthropic style: { system: "...", messages: [...] }
+                if (typeof input.system === 'string') return input.system;
+                // Custom wrapAI callers: { systemPrompt: "...", messages: [...] }
+                if (typeof input.systemPrompt === 'string' && input.systemPrompt.length > 0) return input.systemPrompt;
+                // OpenAI / plain array: messages with role === "system"
+                var msgs = Array.isArray(input.messages) ? input.messages : (Array.isArray(input) ? input : null);
+                if (msgs) {
+                    for (var i = 0; i < msgs.length; i++) {
+                        var m = msgs[i];
+                        if (m && typeof m === 'object' && m.role === 'system' && typeof m.content === 'string') {
+                            return m.content;
+                        }
+                    }
+                }
+                return null;
+            }
+
+            /**
+             * Returns an array of unique system prompts observed across all GENERATION observations.
+             * Each entry: { systemPrompt, modelName, count, rowIndex }
+             */
+            function getSystemPromptsFromTrace() {
+                var seen = []; // [{ systemPrompt, modelName, count }]
+                var seenMap = {}; // systemPrompt -> index in seen
+                currentObservations.forEach(function(obs) {
+                    if (obs.type !== 'GENERATION') return;
+                    var sp = extractSystemPromptFromInput(obs.input);
+                    if (!sp) return;
+                    if (seenMap[sp] === undefined) {
+                        seenMap[sp] = seen.length;
+                        seen.push({ systemPrompt: sp, modelName: obs.model || obs.name || '(unknown)', count: 0 });
+                    }
+                    seen[seenMap[sp]].count++;
+                });
+                return seen.map(function(e, i) { return Object.assign({}, e, { rowIndex: i }); });
+            }
+
+            function renderPromptMockSection() {
+                const prompts = getSystemPromptsFromTrace();
+                if (prompts.length === 0) {
+                    return '<div style="color:#999;font-size:13px;padding:6px 0;">No system prompts detected in trace. Only AI calls with a system prompt can be mocked here.</div>';
+                }
+
+                let html = '<div style="max-height:360px;overflow-y:auto;border:1px solid #e0e0e0;border-radius:6px;">';
+                html += '<table style="width:100%;border-collapse:collapse;font-size:13px;">';
+                html += '<thead><tr style="background:#f5f5f5;">';
+                html += '<th style="padding:6px 10px;text-align:left;border-bottom:1px solid #e0e0e0;width:24px;"></th>';
+                html += '<th style="padding:6px 10px;text-align:left;border-bottom:1px solid #e0e0e0;">Model</th>';
+                html += '<th style="padding:6px 10px;text-align:left;border-bottom:1px solid #e0e0e0;">Uses</th>';
+                html += '<th style="padding:6px 10px;text-align:left;border-bottom:1px solid #e0e0e0;">System Prompt (override applies to all calls using this prompt)</th>';
+                html += '</tr></thead><tbody>';
+
+                prompts.forEach(function(row) {
+                    const key = row.systemPrompt;
+                    const isEnabled = window._promptMockConfig[key] !== undefined;
+                    const currentVal = isEnabled ? window._promptMockConfig[key] : row.systemPrompt;
+                    const preview = key.length > 80 ? key.slice(0, 80) + '…' : key;
+
+                    html += '<tr class="prompt-mock-row" data-row-index="' + row.rowIndex + '" style="border-bottom:1px solid #f0f0f0;vertical-align:top;">';
+
+                    // Checkbox column
+                    html += '<td style="padding:8px 10px;white-space:nowrap;">';
+                    html += '<input type="checkbox" class="prompt-mock-enable" title="Override this system prompt"' + (isEnabled ? ' checked' : '') + ' onchange="window.onPromptMockToggle(' + row.rowIndex + ', this.checked)">';
+                    html += '</td>';
+
+                    // Model column
+                    html += '<td style="padding:8px 10px;font-family:Monaco,monospace;font-size:12px;white-space:nowrap;">' + esc(row.modelName) + '</td>';
+
+                    // Uses count column
+                    html += '<td style="padding:8px 10px;color:#555;white-space:nowrap;">' + row.count + 'x</td>';
+
+                    // System prompt column
+                    html += '<td style="padding:8px 10px;width:100%;">';
+                    if (!isEnabled) {
+                        html += '<div style="font-size:11px;color:#888;font-style:italic;font-family:Monaco,monospace;max-width:380px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;" title="' + esc(key) + '">' + esc(preview) + '</div>';
+                    } else {
+                        html += '<textarea class="prompt-mock-input" data-row-index="' + row.rowIndex + '" style="width:100%;box-sizing:border-box;font-size:11px;font-family:Monaco,monospace;padding:4px;border:1px solid #ddd;border-radius:4px;min-height:72px;resize:vertical;" oninput="window.onPromptMockInput(' + row.rowIndex + ', this.value)">' + esc(currentVal) + '</textarea>';
+                    }
+                    html += '</td>';
+
+                    html += '</tr>';
+                });
+
+                html += '</tbody></table></div>';
+                return html;
+            }
+
+            function buildPromptMockConfigFromUI() {
+                // Return _promptMockConfig as-is (string key → string value), filtering out blank values
+                const config = {};
+                Object.keys(window._promptMockConfig).forEach(function(key) {
+                    const val = window._promptMockConfig[key];
+                    if (typeof val === 'string' && val.trim()) config[key] = val;
+                });
+                return config;
+            }
+
+            window.onPromptMockToggle = function(rowIndex, enabled) {
+                const prompts = getSystemPromptsFromTrace();
+                const row = prompts[rowIndex];
+                if (!row) return;
+                const key = row.systemPrompt;
+                if (!enabled) {
+                    delete window._promptMockConfig[key];
+                } else {
+                    // Pre-fill with the original system prompt so user can edit from there
+                    if (window._promptMockConfig[key] === undefined) {
+                        window._promptMockConfig[key] = row.systemPrompt;
+                    }
+                }
+                const container = document.getElementById('promptMockContainer');
+                if (container) container.innerHTML = renderPromptMockSection();
+            };
+
+            window.onPromptMockInput = function(rowIndex, value) {
+                const prompts = getSystemPromptsFromTrace();
+                const row = prompts[rowIndex];
+                if (!row) return;
+                window._promptMockConfig[row.systemPrompt] = value;
+            };
+
             window.onToolMockModeChange = function(toolName, mode) {
                 if (!window._toolMockConfig[toolName]) window._toolMockConfig[toolName] = { mode: 'live' };
                 // Save current mock data before switching
@@ -1452,7 +1596,8 @@ function getDashboardHtml() {
 
             window.openLiveValidationDialog = function() {
                 if (window.liveValidationDialog) return;
-                window._toolMockConfig = {}; // Reset mock config each time dialog opens
+                window._toolMockConfig = {};    // Reset mock configs each time dialog opens
+                window._promptMockConfig = {};
 
                 const hasTraceTools = currentObservations.some(function(o) { return o.type === 'TOOL'; });
                 const hasRegisteredTools = codeIndex.tools && codeIndex.tools.length > 0;
@@ -1478,8 +1623,15 @@ function getDashboardHtml() {
                                 Show all registered tools
                             </label>
                         </div>
-                        <div id="toolMockContainer"></div>
+                        <div id="toolMockContainer" style="max-height:160px;overflow-y:auto;"></div>
                     </div>\` : ''}
+                    <div style="border-top:1px solid #eee;padding-top:16px;margin-bottom:16px;">
+                        <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:6px;">
+                            <div style="font-size:15px;font-weight:600;">Prompt Mocking</div>
+                            <span style="font-size:12px;color:#888;">Check a row to replace the system prompt for all calls using it</span>
+                        </div>
+                        <div id="promptMockContainer" style="max-height:160px;overflow-y:auto;"></div>
+                    </div>
                     <div style="display:flex;gap:12px;justify-content:space-between;align-items:center;">
                         <span id="liveValidationProgress" style="font-size:14px;color:#555;"></span>
                         <div style="display:flex;gap:12px;">
@@ -1490,11 +1642,15 @@ function getDashboardHtml() {
                     </div>
                 \`;
                 document.body.appendChild(window.liveValidationDialog);
-                // Render the tool mock section after DOM insertion
+                // Render mock sections after DOM insertion
                 const toolMockContainer = document.getElementById('toolMockContainer');
                 if (toolMockContainer) {
                     toolMockContainer.innerHTML = renderToolMockSection(false);
                 }
+                const promptMockContainer = document.getElementById('promptMockContainer');
+                if (promptMockContainer) {
+                    promptMockContainer.innerHTML = renderPromptMockSection();
+                }
                 document.getElementById('cancelLiveValidation').onclick = function() {
                     window.liveValidationDialog.remove();
                     window.liveValidationDialog = null;
@@ -1503,9 +1659,11 @@ function getDashboardHtml() {
                     const count = parseInt(document.getElementById('liveValidationCount').value, 10);
                     const sequential = document.getElementById('liveValidationSequential').checked;
                     if (count >= 1) {
-                        // Build the tool mock config from UI state and persist for "Run from here"
+                        // Build mock configs from UI state and persist for "Run from here"
                         const toolMockConfig = buildToolMockConfigFromUI();
                         window._toolMockConfig = toolMockConfig;
+                        const promptMockConfig = buildPromptMockConfigFromUI();
+                        window._promptMockConfig = promptMockConfig;
                         const submitBtn = document.getElementById('submitLiveValidation');
                         submitBtn.disabled = true;
                         submitBtn.textContent = 'Validating...';
@@ -1542,7 +1700,7 @@ function getDashboardHtml() {
                             const collectedTraces = [];
                             let fatalError = null;
                             for (let i = 0; i < count; i++) {
-                                const singlePayload = { workflowName: selectedWorkflow?.name, runCount: 1, sequential: false, observations: currentObservations, toolMockConfig };
+                                const singlePayload = { workflowName: selectedWorkflow?.name, runCount: 1, sequential: false, observations: currentObservations, toolMockConfig, promptMockConfig };
                                 try {
                                     const response = await fetch('/api/validate-workflow', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(singlePayload) });
                                     const data = await response.json();
@@ -1561,7 +1719,7 @@ function getDashboardHtml() {
                         } else {
                             // Parallel mode: single bulk request
                             if (progressEl) progressEl.textContent = \`Running \${count} workflow run\${count !== 1 ? 's' : ''} in parallel…\`;
-                            const payload = { workflowName: selectedWorkflow?.name, runCount: count, sequential: false, observations: currentObservations, toolMockConfig };
+                            const payload = { workflowName: selectedWorkflow?.name, runCount: count, sequential: false, observations: currentObservations, toolMockConfig, promptMockConfig };
                             try {
                                 const response = await fetch('/api/validate-workflow', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(payload) });
                                 const data = await response.json();
@@ -2096,7 +2254,7 @@ function getDashboardHtml() {
                     traceLayoutEl.classList.remove("step-4");
                     let headerHtml = '';
                     if (currentStep === 3) {
-                        headerHtml = '<tr><th style="width: 40px;">Check</th><th>Name</th><th>Type</th></tr>';
+                        headerHtml = '<tr><th style="width: 40px;">Check</th><th>Name</th><th>Type</th><th>Duration</th></tr>';
                     }
                     traceLayoutEl.innerHTML = \`
                         <div class="trace-left">
@@ -2265,6 +2423,7 @@ function getDashboardHtml() {
                         snapshotId: liveTrace.snapshotId,
                         observations: currentObservations,
                         toolMockConfig: window._toolMockConfig || {},
+                        promptMockConfig: window._promptMockConfig || {},
                     };
                     const response = await fetch('/api/run-from-breakpoint', {
                         method: 'POST',
@@ -2296,7 +2455,7 @@ function getDashboardHtml() {
                 step5RerunInFlight = true;
                 renderObservationTable();
                 try {
-                    const payload = { workflowName: selectedWorkflow?.name, runCount: 1, sequential: false, observations: currentObservations };
+                    const payload = { workflowName: selectedWorkflow?.name, runCount: 1, sequential: false, observations: currentObservations, toolMockConfig: window._toolMockConfig || {}, promptMockConfig: window._promptMockConfig || {} };
                     const response = await fetch('/api/validate-workflow', {
                         method: 'POST',
                         headers: { 'Content-Type': 'application/json' },
@@ -2347,6 +2506,7 @@ function getDashboardHtml() {
                         },
                         snapshotId: liveTrace.snapshotId,
                         toolMockConfig: window._toolMockConfig || {},
+                        promptMockConfig: window._promptMockConfig || {},
                     };
                     const response = await fetch('/api/resume-agent-from-task', {
                         method: 'POST',
@@ -2867,11 +3027,11 @@ function resolveTemplateValue(value, input) {
     return value;
 }
 async function runHttpWorkflow(opts) {
-    const { workflowName, workflowInput, frozenEvents = [], pushedEvents, runConfigs, config, dashboardPort } = opts;
+    const { workflowName, workflowInput, frozenEvents = [], promptMocks = {}, pushedEvents, runConfigs, config, dashboardPort } = opts;
     const runId = randomUUID();
-    // Register run config so the user's server can fetch frozen events
+    // Register run config so the user's server can fetch frozen events and prompt mocks
     pushedEvents.set(runId, []);
-    runConfigs.set(runId, { frozenEvents });
+    runConfigs.set(runId, { frozenEvents, promptMocks });
     try {
         const parsedInput = parseObservationInput(workflowInput);
         const inputObj = parsedInput && typeof parsedInput === 'object' && !Array.isArray(parsedInput) ? parsedInput : {};
@@ -2928,6 +3088,7 @@ async function runHttpWorkflow(opts) {
         const drainMs = parseInt(process.env.ELASTICDASH_HTTP_DRAIN_MS ?? '300', 10);
         await new Promise(resolve => setTimeout(resolve, drainMs));
         const events = (pushedEvents.get(runId) ?? []).sort((a, b) => a.timestamp - b.timestamp);
+        console.log(`[elasticdash] runHttpWorkflow drain complete: ${events.length} events collected for runId=${runId}`);
         const workflowTrace = { traceId: runId, events };
         return { ok: true, currentOutput, workflowTrace, steps: [], llmSteps: [], toolCalls: [], customSteps: [] };
     }
@@ -2948,8 +3109,8 @@ export async function startDashboardServer(cwd, options = {}) {
     // In-memory store for telemetry events pushed from HTTP workflow mode runs.
     // Maps runId -> accumulated WorkflowEvent[]
     const pushedEvents = new Map();
-    // Per-run config for HTTP workflow mode (frozen events for replay).
-    // Maps runId -> { frozenEvents }
+    // Per-run config for HTTP workflow mode (frozen events + prompt mocks for replay).
+    // Maps runId -> { frozenEvents, promptMocks }
     const runConfigs = new Map();
     // Scan workflows, tools, and config once at startup
     const workflows = scanWorkflows(cwd);
@@ -3015,10 +3176,14 @@ export async function startDashboardServer(cwd, options = {}) {
                         const resolvedInput = resolveWorkflowArgsFromObservations(body, workflowName);
                         const workflowInput = resolvedInput.input ?? null;
                         const traces = [];
+                        const promptMocks = body.promptMockConfig && typeof body.promptMockConfig === 'object' && !Array.isArray(body.promptMockConfig)
+                            ? body.promptMockConfig
+                            : {};
                         const runOne = async (runNumber) => {
                             const result = await runHttpWorkflow({
                                 workflowName, workflowInput, pushedEvents, runConfigs,
                                 config: httpConfig, dashboardPort: port,
+                                promptMocks,
                             });
                             const traceStub = { getSteps: () => [], getLLMSteps: () => [], getToolCalls: () => [], getCustomSteps: () => [], recordLLMStep: () => { }, recordToolCall: () => { }, recordCustomStep: () => { } };
                             return {
@@ -3085,12 +3250,15 @@ export async function startDashboardServer(cwd, options = {}) {
                     const frozenEventIds = new Set(frozenEvents.map((e) => e.id));
                     const httpConfig = elasticdashConfig.workflows?.[workflowName];
                     if (httpConfig?.mode === 'http') {
-                        // HTTP workflow mode — call user's dev server with frozen events for step replay
+                        // HTTP workflow mode — call user's dev server with frozen events + prompt mocks for step replay
+                        const bpPromptMocks = body.promptMockConfig && typeof body.promptMockConfig === 'object' && !Array.isArray(body.promptMockConfig)
+                            ? body.promptMockConfig
+                            : {};
                         console.log(`[elasticdash] Run from breakpoint (HTTP mode): workflow="${workflowName}" checkpoint=${checkpoint} frozen=${frozenEvents.length}`);
                         const result = await runHttpWorkflow({
                             workflowName, workflowInput, pushedEvents, runConfigs,
                             config: httpConfig, dashboardPort: port,
-                            frozenEvents,
+                            frozenEvents, promptMocks: bpPromptMocks,
                         });
                         const traceStub = { getSteps: () => [], getLLMSteps: () => [], getToolCalls: () => [], getCustomSteps: () => [], recordLLMStep: () => { }, recordToolCall: () => { }, recordCustomStep: () => { } };
                         const snapshotId = result.workflowTrace ? saveSnapshot(cwd, result.workflowTrace) : undefined;
@@ -3118,8 +3286,14 @@ export async function startDashboardServer(cwd, options = {}) {
                     const toolMockConfig = body.toolMockConfig && typeof body.toolMockConfig === 'object' && !Array.isArray(body.toolMockConfig)
                         ? body.toolMockConfig
                         : undefined;
+                    const aiMockConfig = body.aiMockConfig && typeof body.aiMockConfig === 'object' && !Array.isArray(body.aiMockConfig)
+                        ? body.aiMockConfig
+                        : undefined;
+                    const promptMockConfig = body.promptMockConfig && typeof body.promptMockConfig === 'object' && !Array.isArray(body.promptMockConfig)
+                        ? body.promptMockConfig
+                        : undefined;
                     console.log(`[elasticdash] Run from breakpoint: workflow="${workflowName}" checkpoint=${checkpoint} historyLen=${history.length}`);
-                    const result = await runWorkflowInSubprocess(workflowsModulePath, toolsModulePath, workflowName, workflowArgs, workflowInput, { replayMode: true, checkpoint, history, ...(toolMockConfig ? { toolMockConfig } : {}) });
+                    const result = await runWorkflowInSubprocess(workflowsModulePath, toolsModulePath, workflowName, workflowArgs, workflowInput, { replayMode: true, checkpoint, history, ...(toolMockConfig ? { toolMockConfig } : {}), ...(aiMockConfig ? { aiMockConfig } : {}), ...(promptMockConfig ? { promptMockConfig } : {}) });
                     const traceStub = {
                         getSteps: () => (result.steps ?? []),
                         getLLMSteps: () => (result.llmSteps ?? []),
@@ -3190,8 +3364,14 @@ export async function startDashboardServer(cwd, options = {}) {
                     const toolMockConfig = body.toolMockConfig && typeof body.toolMockConfig === 'object' && !Array.isArray(body.toolMockConfig)
                         ? body.toolMockConfig
                         : undefined;
+                    const aiMockConfig = body.aiMockConfig && typeof body.aiMockConfig === 'object' && !Array.isArray(body.aiMockConfig)
+                        ? body.aiMockConfig
+                        : undefined;
+                    const promptMockConfig = body.promptMockConfig && typeof body.promptMockConfig === 'object' && !Array.isArray(body.promptMockConfig)
+                        ? body.promptMockConfig
+                        : undefined;
                     console.log(`[elasticdash] Resume agent from task: workflow="${workflowName}" taskIndex=${taskIndex}`);
-                    const result = await runWorkflowInSubprocess(workflowsModulePath, toolsModulePath, workflowName, [], null, { replayMode: history.length > 0, checkpoint: 0, history, agentState, ...(toolMockConfig ? { toolMockConfig } : {}) });
+                    const result = await runWorkflowInSubprocess(workflowsModulePath, toolsModulePath, workflowName, [], null, { replayMode: history.length > 0, checkpoint: 0, history, agentState, ...(toolMockConfig ? { toolMockConfig } : {}), ...(aiMockConfig ? { aiMockConfig } : {}), ...(promptMockConfig ? { promptMockConfig } : {}) });
                     const traceStub = {
                         getSteps: () => (result.steps ?? []),
                         getLLMSteps: () => (result.llmSteps ?? []),
@@ -3242,7 +3422,7 @@ export async function startDashboardServer(cwd, options = {}) {
             const runId = url.pathname.slice('/api/run-configs/'.length);
             const cfg = runConfigs.get(runId);
             res.writeHead(cfg ? 200 : 404, { 'Content-Type': 'application/json' });
-            res.end(JSON.stringify({ frozenEvents: cfg?.frozenEvents ?? [] }));
+            res.end(JSON.stringify({ frozenEvents: cfg?.frozenEvents ?? [], promptMocks: cfg?.promptMocks ?? {} }));
         }
         else if (url.pathname === '/api/trace-events' && req.method === 'POST') {
             // Receive telemetry events pushed from wrapAI / wrapTool in HTTP workflow mode
@@ -3255,9 +3435,16 @@ export async function startDashboardServer(cwd, options = {}) {
                         res.end(JSON.stringify({ ok: false, error: 'runId (string) and event (object) are required.' }));
                         return;
                     }
-                    const existing = pushedEvents.get(body.runId) ?? [];
-                    existing.push(body.event);
-                    pushedEvents.set(body.runId, existing);
+                    const existing = pushedEvents.get(body.runId);
+                    if (!existing) {
+                        console.log(`[elasticdash] /api/trace-events: unknown runId=${body.runId}, known runIds=[${[...pushedEvents.keys()].join(',')}]`);
+                        res.writeHead(404, { 'Content-Type': 'application/json' });
+                        res.end(JSON.stringify({ ok: false, error: 'unknown runId' }));
+                        return;
+                    }
+                    const evt = body.event;
+                    existing.push(evt);
+                    console.log(`[elasticdash] /api/trace-events: stored event type=${evt.type} name=${('name' in evt ? evt.name : '?')} runId=${body.runId} total=${existing.length}`);
                     res.writeHead(200, { 'Content-Type': 'application/json' });
                     res.end(JSON.stringify({ ok: true }));
                 }