elasticdash-test 0.1.25-alpha-2 → 0.1.26-alpha-1

package/docs/agent-integration-guide.md

@@ -5,6 +5,10 @@
 > **When to use**: Follow this guide step-by-step when a user asks you to "integrate ElasticDash", "set up ElasticDash testing", "implement elasticdash-test", or similar. Each step produces a specific file — work through them in order.
 >
 > **How to use**: Read the full guide first to understand the project's structure, then execute each step. Use the decision trees section to choose the right pattern for the user's project (framework type, module system, tool architecture).
+>
+> **Important**: Do NOT leave `YOUR_TOOL_NAME` / `YOUR_WORKFLOW_NAME` placeholders in generated files. Before creating `ed_tools.ts` and `ed_workflows.ts`, scan the user's codebase to identify their actual tool functions (functions that call external APIs, databases, or services) and workflow functions (async functions that orchestrate LLM calls and tool calls). Create the files with the real function names, imports, and types found in the codebase. If no tools or workflows exist yet, ask the user which functions to instrument rather than generating placeholder files.
+>
+> **Automation level**: Automatically implement all steps unless you are not confident about a specific change (e.g., unclear which function is a tool vs. a utility, or ambiguous workflow boundaries). In those cases, ask the user before proceeding. For straightforward tool/workflow functions, proceed without asking.
 
 ---
 
@@ -12,9 +16,18 @@
 
 - Node.js >= 20
 - npm, yarn, or pnpm
-- At least one AI workflow function (a callable async function that makes LLM/tool calls)
 - LLM provider API keys for providers used in workflows (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
 
+### Identify tools and workflows in the codebase
+
+Before starting, scan the project to find:
+
+- **Tool functions**: Exported async functions that call external services (APIs, databases, search, etc.). Look for functions that use `fetch`, database clients, SDK calls, or any external I/O. These go in `ed_tools.ts`.
+- **Workflow functions**: Exported async functions that orchestrate an AI pipeline — typically calling an LLM and one or more tools. Look for functions that call OpenAI, Anthropic, Gemini, or similar SDKs. These go in `ed_workflows.ts`.
+
+If the project has no clear tool or workflow functions yet, ask the user:
+> "Which functions in your codebase make external API/service calls (tools), and which functions orchestrate your AI workflow? I need to know so I can set up ElasticDash instrumentation."
+
 ### Detect project type before starting
 
 Check these before choosing patterns:
@@ -28,6 +41,26 @@ Check these before choosing patterns:
 
 ---
 
+## Step 0: Check environment variables (do this FIRST)
+
+Before making any code changes, check the project's `.env` file for the required ElasticDash variables.
+
+### 1. Check for `ELASTICDASH_API_URL`
+
+- If present: use the existing value.
+- If missing: add `ELASTICDASH_API_URL=https://server.elasticdash.com` to `.env`.
+
+### 2. Check for `ELASTICDASH_API_KEY`
+
+- If present: proceed to Step 1.
+- If missing: **STOP and ask the user**:
+
+> "Your project needs an ElasticDash API key. Please complete the onboarding at https://app.elasticdash.com to get your project API key, then add `ELASTICDASH_API_KEY=ed_your_key_here` to your `.env` file. Let me know when it's ready."
+
+**Do not continue until `ELASTICDASH_API_KEY` is set.** The SDK cannot connect to the backend without it.
+
+---
+
 ## Step 1: Install
 
 ```bash
@@ -45,161 +78,63 @@ Add to `.gitignore`:
 
 ## Step 2: Create `ed_tools.ts`
 
-Create `ed_tools.ts` in the project root. This file wraps each tool function with tracing so ElasticDash can record and replay tool calls.
+Create `ed_tools.ts` in the project root. This file wraps each tool function with `wrapTool()` for automatic tracing, mocking, and telemetry.
 
-### Choose your pattern
+**Before writing this file**: Search the codebase for functions that call external services (APIs, databases, SDKs). Each one becomes a wrapped export in `ed_tools.ts`. Replace all `YOUR_TOOL_*` and `YOUR_SOURCE_PATH_*` placeholders below with the actual function names and import paths found in the project.
 
-- **Pattern B (recommended for most projects)**: inline async with mock support. Use when tools are imported individually, no central dispatcher exists, or you want dashboard mock support.
-- **Pattern A**: `withTrace` HOF with central dispatcher. Use when tools are pure functions called through a single `dispatchTool(name, args)` dispatcher.
-
-### Pattern B template (recommended)
+### Template
 
 ```ts
 // ed_tools.ts
-// Replace imports with your actual tool functions and source paths
-import { YOUR_TOOL_1 as YOUR_TOOL_1_impl } from './YOUR_SOURCE_PATH_1'
-import { YOUR_TOOL_2 as YOUR_TOOL_2_impl } from './YOUR_SOURCE_PATH_2'
+import { setElasticDashModule } from './ed_workflows'
+
+// Import original tool implementations from the actual source files
+import { originalTool1 } from './services/YOUR_SOURCE_1'
+import { originalTool2 } from './utils/YOUR_SOURCE_2'
 
 // ---------------------------------------------------------------------------
-// Helpers — copy as-is, no customization needed
+// Load elasticdash-test and share the module with ed_workflows.ts
 // ---------------------------------------------------------------------------
 
-function resolveMock(toolName: string): { mocked: true; result: unknown } | { mocked: false } {
-  const g = globalThis as any
-  const mocks = g.__ELASTICDASH_TOOL_MOCKS__
-  if (!mocks) return { mocked: false }
-
-  const entry = mocks[toolName]
-  if (!entry || entry.mode === 'live') return { mocked: false }
-
-  if (!g.__ELASTICDASH_TOOL_CALL_COUNTERS__) g.__ELASTICDASH_TOOL_CALL_COUNTERS__ = {}
-  const counters = g.__ELASTICDASH_TOOL_CALL_COUNTERS__
-  counters[toolName] = (counters[toolName] ?? 0) + 1
-  const callNumber = counters[toolName]
-
-  if (entry.mode === 'mock-all') {
-    const data = entry.mockData ?? {}
-    const result = data[callNumber] !== undefined ? data[callNumber] : data[0]
-    return { mocked: true, result }
-  }
-
-  if (entry.mode === 'mock-specific') {
-    const indices = entry.callIndices ?? []
-    if (indices.includes(callNumber)) {
-      return { mocked: true, result: (entry.mockData ?? {})[callNumber] }
-    }
-    return { mocked: false }
-  }
-
-  return { mocked: false }
-}
-
-async function safeRecordToolCall(tool: string, input: any, result: any) {
-  if (!(globalThis as any).__ELASTICDASH_WORKER__) return
-  try {
-    const { recordToolCall } = await import('elasticdash-test')
-    recordToolCall(tool, input, result)
-  } catch { /* tracing must never block business logic */ }
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+type WrapToolFn = <T extends (...args: any[]) => any>(name: string, fn: T) => T
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+let wrapTool: WrapToolFn = (_name: string, fn: any) => fn
+
+try {
+  // eslint-disable-next-line @typescript-eslint/no-explicit-any
+  const _edModule = (eval('require') as (id: string) => any)('elasticdash-test')
+  wrapTool = _edModule.wrapTool ?? wrapTool
+  // Share the module instance with ed_workflows.ts so trace hooks use the same context
+  setElasticDashModule(_edModule)
+} catch {
+  // elasticdash-test not available — all wrappers pass through to original functions
 }
 
 // ---------------------------------------------------------------------------
-// Tools — one export per tool, following this pattern
+// Wrapped tools — one export per tool
 // ---------------------------------------------------------------------------
 
-// TEMPLATE: Copy this block for each tool, replacing YOUR_TOOL_NAME and YOUR_TOOL_IMPL
-export const YOUR_TOOL_1 = async (input: any) => {
-  const mock = resolveMock('YOUR_TOOL_1')
-  if (mock.mocked) {
-    await safeRecordToolCall('YOUR_TOOL_1', input, mock.result)
-    return mock.result
-  }
-
-  return await YOUR_TOOL_1_impl(input)
-    .then(async (res: any) => {
-      await safeRecordToolCall('YOUR_TOOL_1', input, res)
-      return res
-    })
-    .catch(async (err: any) => {
-      await safeRecordToolCall('YOUR_TOOL_1', input, err)
-      throw err
-    })
-}
-
-export const YOUR_TOOL_2 = async (input: any) => {
-  const mock = resolveMock('YOUR_TOOL_2')
-  if (mock.mocked) {
-    await safeRecordToolCall('YOUR_TOOL_2', input, mock.result)
-    return mock.result
-  }
-
-  return await YOUR_TOOL_2_impl(input)
-    .then(async (res: any) => {
-      await safeRecordToolCall('YOUR_TOOL_2', input, res)
-      return res
-    })
-    .catch(async (err: any) => {
-      await safeRecordToolCall('YOUR_TOOL_2', input, err)
-      throw err
-    })
-}
-```
-
-### Pattern A template (dispatcher-based)
-
-```ts
-// ed_tools.ts
-// Replace imports with your actual tool functions and source paths
-import {
-  YOUR_TOOL_1 as _YOUR_TOOL_1,
-  YOUR_TOOL_2 as _YOUR_TOOL_2,
-  dispatchTool as _dispatchTool,
-} from './YOUR_TOOLS_SOURCE'
-
-async function withTrace<I, O>(
-  toolName: string,
-  input: I,
-  fn: (input: I) => Promise<O>,
-): Promise<O> {
-  const result = await fn(input)
-  try {
-    const { recordToolCall } = await import('elasticdash-test')
-    recordToolCall(toolName, input, result)
-  } catch { /* tracing must never block business logic */ }
-  return result
-}
-
-export function YOUR_TOOL_1(input: Parameters<typeof _YOUR_TOOL_1>[0]) {
-  return withTrace('YOUR_TOOL_1', input, _YOUR_TOOL_1)
-}
-
-export function YOUR_TOOL_2(input: Parameters<typeof _YOUR_TOOL_2>[0]) {
-  return withTrace('YOUR_TOOL_2', input, _YOUR_TOOL_2)
-}
+export const myTool1 = wrapTool('myTool1', async (input: any) => {
+  return await originalTool1(input)
+})
 
-export async function dispatchTool(name: string, args: Record<string, unknown>) {
-  switch (name) {
-    case 'YOUR_TOOL_1': return YOUR_TOOL_1(args as Parameters<typeof _YOUR_TOOL_1>[0])
-    case 'YOUR_TOOL_2': return YOUR_TOOL_2(args as Parameters<typeof _YOUR_TOOL_2>[0])
-    default: return _dispatchTool(name, args)
-  }
-}
+export const myTool2 = wrapTool('myTool2', async (input: any) => {
+  const { someField } = input as { someField: string }
+  return await originalTool2(someField)
+})
 ```
 
-### Alternative: `wrapTool` shorthand
-
-For simpler setups where you don't need mock support or dashboard replays:
-
-```ts
-// ed_tools.ts
-import { wrapTool } from 'elasticdash-test'
-import { YOUR_TOOL_1 as YOUR_TOOL_1_impl } from './YOUR_SOURCE_PATH'
+### Key patterns
 
-export const YOUR_TOOL_1 = wrapTool('YOUR_TOOL_1', YOUR_TOOL_1_impl)
-```
+- **`wrapTool(name, fn)`** wraps the function with automatic tracing, mocking, and telemetry. Falls back to a passthrough if `elasticdash-test` is not installed.
+- **`eval('require')`** is used instead of `import()` to share the same module instance across `ed_tools.ts` and `ed_workflows.ts`. This avoids ESM/CJS dual-instance issues.
+- **`setElasticDashModule`** shares the loaded module with `ed_workflows.ts` so `edStartTrace`/`edEndTrace` use the same tracing context as `wrapTool`.
+- The exported name (e.g., `myTool1`) can differ from the original function name (e.g., `originalTool1`). The call sites in existing source files will be updated to use the new name in Step 4.
 
 ### Important rules
 
-- The string name passed to `resolveMock()`, `safeRecordToolCall()`, or `wrapTool()` **must match** the exported function name exactly.
+- The string name passed to `wrapTool()` **must match** the exported function name exactly.
 - Each tool function must accept a single input object and return a plain value (JSON-serializable).
 - Tool functions must not close over HTTP context, framework state, or database clients — extract pure logic first.
 
@@ -219,92 +154,173 @@ export default nextConfig
 
 ## Step 3: Create `ed_workflows.ts`
 
-Create `ed_workflows.ts` in the project root. This file exports workflow functions for the ElasticDash runner.
+Create `ed_workflows.ts` in the project root. This file serves two purposes:
+
+1. **Trace lifecycle hooks** (`edStartTrace`, `edEndTrace`) — called from route handlers to mark workflow boundaries
+2. **Workflow exports** — callable functions for the ElasticDash dashboard and test runner
 
-### Simple case — direct re-export
+**Before writing this file**: Search the codebase for the main AI workflow functions — async functions that orchestrate LLM calls and tool calls (e.g., a chat handler, an agent loop, a pipeline function). Replace all `YOUR_WORKFLOW` and `YOUR_SOURCE_PATH` placeholders below with the actual function names and import paths.
+
+### Trace lifecycle hooks
+
+Every `ed_workflows.ts` should export `edStartTrace` and `edEndTrace`. These are called from route handlers (Step 4) to mark when a workflow starts and ends:
 
 ```ts
-// ed_workflows.ts
-// Replace with your actual workflow function and source path
-export { YOUR_WORKFLOW } from './YOUR_SOURCE_PATH'
+// ed_workflows.ts — trace hooks (copy as-is)
+
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+let _ed: any = null
+
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+export function setElasticDashModule(mod: any): void {
+  _ed = mod
+}
+
+export const edStartTrace = async (workflowName: string): Promise<void> => {
+  if (!_ed) return
+  try {
+    await _ed.tryAutoInitHttpContext()
+    _ed.startTrace(workflowName)
+  } catch (err) {
+    console.error('[ed_workflows] edStartTrace error:', err)
+  }
+}
+
+export const edEndTrace = (): void => {
+  if (!_ed) return
+  try {
+    _ed.endTrace()
+  } catch (err) {
+    console.error('[ed_workflows] edEndTrace error:', err)
+  }
+}
 ```
 
-### Framework adapter case (Next.js / Remix)
+### Workflow exports — simple case
 
-When the workflow lives inside a route handler, create a plain-value wrapper:
+For non-framework projects where the workflow can be imported directly:
 
 ```ts
 // ed_workflows.ts
-import { YOUR_HANDLER as _YOUR_HANDLER } from './app/api/YOUR_ROUTE/route'
-
-export async function YOUR_WORKFLOW(input: { message: string; sessionId: string }) {
-  return _YOUR_HANDLER(input)
-}
+export { YOUR_WORKFLOW } from './YOUR_SOURCE_PATH'
 ```
 
-### Streaming workflow case (Vercel AI SDK)
+### Workflow exports — HTTP mode (Next.js / Remix / framework projects)
 
-Create a separate handler file that is only imported by `ed_workflows.ts`:
+For framework projects, the workflow calls the running dev server instead of importing the route handler:
 
 ```ts
-// app/api/chat-stream/chatStreamHandler.ts
-import { NextRequest } from 'next/server'
-import { readVercelAIStream, recordToolCall } from 'elasticdash-test'
-import type { VercelAIStreamResult } from 'elasticdash-test'
-import { POST } from './route'
+// ed_workflows.ts
+
+const APP_URL = process.env.APP_URL ?? 'http://localhost:3000'
 
-export async function chatStreamHandler(args: {
+export const YOUR_WORKFLOW = async (input: {
   messages: Array<{ role: string; content: string }>
   sessionId?: string
-}): Promise<VercelAIStreamResult> {
-  const req = new NextRequest('http://localhost/api/chat-stream', {
+}): Promise<unknown> => {
+  const response = await fetch(`${APP_URL}/api/YOUR_ENDPOINT`, {
     method: 'POST',
     headers: { 'Content-Type': 'application/json' },
-    body: JSON.stringify(args),
+    body: JSON.stringify(input),
   })
 
-  const response = await POST(req)
-
-  if (response.headers.get('x-vercel-ai-data-stream') !== 'v1') {
-    const errorMessage = await response.text().catch(() => `HTTP ${response.status}`)
-    return { message: errorMessage, type: 'error', error: errorMessage }
+  if (!response.ok) {
+    const text = await response.text()
+    throw new Error(`HTTP ${response.status}: ${text}`)
   }
 
-  const result = await readVercelAIStream(response)
-  recordToolCall('chatStream', args, result)
-  return result
+  return response.json()
 }
 ```
 
-Then re-export from `ed_workflows.ts`:
-
-```ts
-// ed_workflows.ts
-export { chatStreamHandler } from './app/api/chat-stream/chatStreamHandler'
-```
-
 ### Requirements for all workflow exports
 
 - Accept only JSON-serializable inputs (strings, numbers, arrays, plain objects)
 - Return only JSON-serializable outputs
 - Must not depend on framework runtime APIs, HTTP request context, or live service clients
-- If a dependency is non-serializable (e.g., database client), instantiate it inside `ed_workflows.ts`, not passed as a parameter
 
 ---
 
-## Step 4: Update workflow imports
+## Step 4: Update existing source files to use `ed_tools`
+
+> **This step modifies the user's existing source files.** The files that call tool functions must import from `ed_tools` instead of the original source, so that all tool calls go through the ElasticDash tracing layer. Similarly, workflow entry points (route handlers, etc.) must call `edStartTrace` / `edEndTrace` from `ed_workflows`.
+
+### Architecture after integration
 
-Change your workflow code to import tools from `ed_tools.ts` instead of the original source files:
+```
+ed_tools.ts
+  ├── imports original functions from services/utils
+  ├── wraps each with wrapTool() for tracing
+  └── exports wrapped versions with the SAME or similar names
+
+ed_workflows.ts
+  ├── exports edStartTrace / edEndTrace for workflow-level tracing
+  └── exports workflow functions (for dashboard/test runner)
+
+Existing source files (MODIFIED):
+  app/api/chat/route.ts
+    ├── BEFORE: import { myTool } from '@/services/myService'
+    ├── AFTER:  import { myTool } from '@/ed_tools'
+    ├── ADDED:  import { edStartTrace, edEndTrace } from '@/ed_workflows'
+    └── ADDED:  await edStartTrace('workflowName') at handler entry
+                edEndTrace() at handler exit
+```
+
+### What to do
+
+**1. Find every file that calls a tool function and update its imports:**
+
+For each tool exported from `ed_tools.ts`, search the codebase for files that import the original function. Update the import to come from `ed_tools` instead.
 
 ```ts
-// BEFORE
-import { YOUR_TOOL_1 } from './services/YOUR_SOURCE'
+// BEFORE — app/api/chat/route.ts
+import { clarifyAndRefineUserInput } from '@/utils/queryRefinement'
+import { dynamicApiRequest } from '@/services/apiService'
 
-// AFTER
-import { YOUR_TOOL_1 } from './ed_tools'
+// AFTER — app/api/chat/route.ts
+import { queryRefinement } from '@/ed_tools'
+import { apiService } from '@/ed_tools'
 ```
 
-This single import change makes all tool calls observable by ElasticDash.
+```ts
+// BEFORE — app/api/chat/executor.ts
+import { dynamicApiRequest } from '@/services/apiService'
+
+// AFTER — app/api/chat/executor.ts
+import { apiService } from '@/ed_tools'
+```
+
+**2. Add trace hooks to route handlers / workflow entry points:**
+
+At the top of each route handler or workflow entry function, add `edStartTrace` and `edEndTrace`:
+
+```ts
+// app/api/chat/route.ts
+import { edStartTrace, edEndTrace } from '@/ed_workflows'
+
+export async function POST(req: Request) {
+  await edStartTrace('chatHandler')
+  try {
+    // ... existing handler logic, now using tools from ed_tools
+  } finally {
+    edEndTrace()
+  }
+}
+```
+
+**3. Update call sites if the wrapped function has a different name:**
+
+If `ed_tools.ts` exports a tool under a different name than the original (e.g., `queryRefinement` wrapping `clarifyAndRefineUserInput`), update the call sites to use the new name.
+
+### What NOT to modify
+
+- **Do not modify `ed_tools.ts` imports** — it must still import from the original source files.
+- **Do not modify the original tool implementation files** (e.g., `services/apiService.ts`, `utils/queryRefinement.ts`) — they stay as-is.
+- **Do not modify files that don't call tool functions** — only update files that directly import and call the wrapped tools.
+
+### When to ask the user
+
+If you are not confident about a change — e.g., a function might be a utility rather than a tool, or it's unclear whether a file should use the wrapped version — **ask the user** before modifying that file. For clear tool calls (API services, database queries, LLM calls), proceed automatically.
 
 ---
 
@@ -425,7 +441,30 @@ npx ed ed-test --no-upload
 
 ---
 
-## Step 8: Run and verify
+## Step 8: Validate the integration
+
+After all files are created, validate the setup by establishing a socket.io connection to the ElasticDash backend. Run the `observe` command — it connects to the server, authenticates with the API key, and registers the project's tools and workflows:
+
+```bash
+npx elasticdash observe
+```
+
+**Expected output on success:**
+
+```
+[elasticdash] Observability active
+  Session ID : <uuid>
+  Server     : https://server.elasticdash.com
+```
+
+This confirms:
+- `ELASTICDASH_API_URL` and `ELASTICDASH_API_KEY` are valid
+- The socket.io connection to the backend is established
+- The SDK can discover `ed_tools.ts` and `ed_workflows.ts`
+
+**If it fails:** Check that `.env` has valid `ELASTICDASH_API_URL` and `ELASTICDASH_API_KEY` values. If the API key is rejected, the user needs to get a new one from https://app.elasticdash.com.
+
+After validation, stop the observe process (Ctrl+C) and inform the user that ElasticDash is integrated. Provide these commands for ongoing use:
 
 ```bash
 # Run aiTest tests
@@ -437,6 +476,9 @@ npx ed ed-test --no-upload
 # Open the dashboard
 npx elasticdash dashboard
 
+# Start observability (connect to backend for live tracing)
+npx elasticdash observe
+
 # Record a trace fixture
 ELASTICDASH_CAPTURE_TRACE=1 tsx your-workflow.ts
 ```
@@ -542,16 +584,12 @@ After integration, verify these files exist:
 
 ```
 your-project/
-  ed_tools.ts           # Instrumented tool wrappers
-  ed_workflows.ts       # Workflow exports
+  .env                  # ELASTICDASH_API_URL and ELASTICDASH_API_KEY set
+  ed_tools.ts           # Instrumented tool wrappers (real function names, not placeholders)
+  ed_workflows.ts       # Workflow exports using tools from ed_tools.ts
   elasticdash.config.ts # Test runner config
   package.json          # dashboard:ai and test:ai scripts added
   .gitignore            # .temp/ and .ed_traces/ added
 ```
 
-Verify with:
-
-```bash
-npx elasticdash test        # Should discover and run *.ai.test.ts files
-npx elasticdash dashboard   # Should open the dashboard UI
-```
+Verify by running `npx elasticdash observe` — a successful socket.io connection confirms the integration is complete.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "elasticdash-test",
-  "version": "0.1.25-alpha-2",
+  "version": "0.1.26-alpha-1",
   "description": "AI-native test runner for ElasticDash workflow testing",
   "type": "module",
   "bin": {