elasticdash-test 0.1.25 → 0.1.26-alpha-1

package/docs/agent-integration-guide.md

@@ -7,6 +7,8 @@
 > **How to use**: Read the full guide first to understand the project's structure, then execute each step. Use the decision trees section to choose the right pattern for the user's project (framework type, module system, tool architecture).
 >
 > **Important**: Do NOT leave `YOUR_TOOL_NAME` / `YOUR_WORKFLOW_NAME` placeholders in generated files. Before creating `ed_tools.ts` and `ed_workflows.ts`, scan the user's codebase to identify their actual tool functions (functions that call external APIs, databases, or services) and workflow functions (async functions that orchestrate LLM calls and tool calls). Create the files with the real function names, imports, and types found in the codebase. If no tools or workflows exist yet, ask the user which functions to instrument rather than generating placeholder files.
+>
+> **Automation level**: Automatically implement all steps unless you are not confident about a specific change (e.g., unclear which function is a tool vs. a utility, or ambiguous workflow boundaries). In those cases, ask the user before proceeding. For straightforward tool/workflow functions, proceed without asking.
 
 ---
 
@@ -39,6 +41,26 @@ Check these before choosing patterns:
 
 ---
 
+## Step 0: Check environment variables (do this FIRST)
+
+Before making any code changes, check the project's `.env` file for the required ElasticDash variables.
+
+### 1. Check for `ELASTICDASH_API_URL`
+
+- If present: use the existing value.
+- If missing: add `ELASTICDASH_API_URL=https://server.elasticdash.com` to `.env`.
+
+### 2. Check for `ELASTICDASH_API_KEY`
+
+- If present: proceed to Step 1.
+- If missing: **STOP and ask the user**:
+
+> "Your project needs an ElasticDash API key. Please complete the onboarding at https://app.elasticdash.com to get your project API key, then add `ELASTICDASH_API_KEY=ed_your_key_here` to your `.env` file. Let me know when it's ready."
+
+**Do not continue until `ELASTICDASH_API_KEY` is set.** The SDK cannot connect to the backend without it.
+
+---
+
 ## Step 1: Install
 
 ```bash
@@ -56,163 +78,63 @@ Add to `.gitignore`:
 
 ## Step 2: Create `ed_tools.ts`
 
-Create `ed_tools.ts` in the project root. This file wraps each tool function with tracing so ElasticDash can record and replay tool calls.
+Create `ed_tools.ts` in the project root. This file wraps each tool function with `wrapTool()` for automatic tracing, mocking, and telemetry.
 
 **Before writing this file**: Search the codebase for functions that call external services (APIs, databases, SDKs). Each one becomes a wrapped export in `ed_tools.ts`. Replace all `YOUR_TOOL_*` and `YOUR_SOURCE_PATH_*` placeholders below with the actual function names and import paths found in the project.
 
-### Choose your pattern
-
-- **Pattern B (recommended for most projects)**: inline async with mock support. Use when tools are imported individually, no central dispatcher exists, or you want dashboard mock support.
-- **Pattern A**: `withTrace` HOF with central dispatcher. Use when tools are pure functions called through a single `dispatchTool(name, args)` dispatcher.
-
-### Pattern B template (recommended)
+### Template
 
 ```ts
 // ed_tools.ts
-// Replace imports with your actual tool functions and source paths
-import { YOUR_TOOL_1 as YOUR_TOOL_1_impl } from './YOUR_SOURCE_PATH_1'
-import { YOUR_TOOL_2 as YOUR_TOOL_2_impl } from './YOUR_SOURCE_PATH_2'
+import { setElasticDashModule } from './ed_workflows'
+
+// Import original tool implementations from the actual source files
+import { originalTool1 } from './services/YOUR_SOURCE_1'
+import { originalTool2 } from './utils/YOUR_SOURCE_2'
 
 // ---------------------------------------------------------------------------
-// Helpers — copy as-is, no customization needed
+// Load elasticdash-test and share the module with ed_workflows.ts
 // ---------------------------------------------------------------------------
 
-function resolveMock(toolName: string): { mocked: true; result: unknown } | { mocked: false } {
-  const g = globalThis as any
-  const mocks = g.__ELASTICDASH_TOOL_MOCKS__
-  if (!mocks) return { mocked: false }
-
-  const entry = mocks[toolName]
-  if (!entry || entry.mode === 'live') return { mocked: false }
-
-  if (!g.__ELASTICDASH_TOOL_CALL_COUNTERS__) g.__ELASTICDASH_TOOL_CALL_COUNTERS__ = {}
-  const counters = g.__ELASTICDASH_TOOL_CALL_COUNTERS__
-  counters[toolName] = (counters[toolName] ?? 0) + 1
-  const callNumber = counters[toolName]
-
-  if (entry.mode === 'mock-all') {
-    const data = entry.mockData ?? {}
-    const result = data[callNumber] !== undefined ? data[callNumber] : data[0]
-    return { mocked: true, result }
-  }
-
-  if (entry.mode === 'mock-specific') {
-    const indices = entry.callIndices ?? []
-    if (indices.includes(callNumber)) {
-      return { mocked: true, result: (entry.mockData ?? {})[callNumber] }
-    }
-    return { mocked: false }
-  }
-
-  return { mocked: false }
-}
-
-async function safeRecordToolCall(tool: string, input: any, result: any) {
-  if (!(globalThis as any).__ELASTICDASH_WORKER__) return
-  try {
-    const { recordToolCall } = await import('elasticdash-test')
-    recordToolCall(tool, input, result)
-  } catch { /* tracing must never block business logic */ }
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+type WrapToolFn = <T extends (...args: any[]) => any>(name: string, fn: T) => T
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+let wrapTool: WrapToolFn = (_name: string, fn: any) => fn
+
+try {
+  // eslint-disable-next-line @typescript-eslint/no-explicit-any
+  const _edModule = (eval('require') as (id: string) => any)('elasticdash-test')
+  wrapTool = _edModule.wrapTool ?? wrapTool
+  // Share the module instance with ed_workflows.ts so trace hooks use the same context
+  setElasticDashModule(_edModule)
+} catch {
+  // elasticdash-test not available — all wrappers pass through to original functions
 }
 
 // ---------------------------------------------------------------------------
-// Tools — one export per tool, following this pattern
+// Wrapped tools — one export per tool
 // ---------------------------------------------------------------------------
 
-// TEMPLATE: Copy this block for each tool, replacing YOUR_TOOL_NAME and YOUR_TOOL_IMPL
-export const YOUR_TOOL_1 = async (input: any) => {
-  const mock = resolveMock('YOUR_TOOL_1')
-  if (mock.mocked) {
-    await safeRecordToolCall('YOUR_TOOL_1', input, mock.result)
-    return mock.result
-  }
-
-  return await YOUR_TOOL_1_impl(input)
-    .then(async (res: any) => {
-      await safeRecordToolCall('YOUR_TOOL_1', input, res)
-      return res
-    })
-    .catch(async (err: any) => {
-      await safeRecordToolCall('YOUR_TOOL_1', input, err)
-      throw err
-    })
-}
-
-export const YOUR_TOOL_2 = async (input: any) => {
-  const mock = resolveMock('YOUR_TOOL_2')
-  if (mock.mocked) {
-    await safeRecordToolCall('YOUR_TOOL_2', input, mock.result)
-    return mock.result
-  }
-
-  return await YOUR_TOOL_2_impl(input)
-    .then(async (res: any) => {
-      await safeRecordToolCall('YOUR_TOOL_2', input, res)
-      return res
-    })
-    .catch(async (err: any) => {
-      await safeRecordToolCall('YOUR_TOOL_2', input, err)
-      throw err
-    })
-}
-```
-
-### Pattern A template (dispatcher-based)
-
-```ts
-// ed_tools.ts
-// Replace imports with your actual tool functions and source paths
-import {
-  YOUR_TOOL_1 as _YOUR_TOOL_1,
-  YOUR_TOOL_2 as _YOUR_TOOL_2,
-  dispatchTool as _dispatchTool,
-} from './YOUR_TOOLS_SOURCE'
-
-async function withTrace<I, O>(
-  toolName: string,
-  input: I,
-  fn: (input: I) => Promise<O>,
-): Promise<O> {
-  const result = await fn(input)
-  try {
-    const { recordToolCall } = await import('elasticdash-test')
-    recordToolCall(toolName, input, result)
-  } catch { /* tracing must never block business logic */ }
-  return result
-}
-
-export function YOUR_TOOL_1(input: Parameters<typeof _YOUR_TOOL_1>[0]) {
-  return withTrace('YOUR_TOOL_1', input, _YOUR_TOOL_1)
-}
-
-export function YOUR_TOOL_2(input: Parameters<typeof _YOUR_TOOL_2>[0]) {
-  return withTrace('YOUR_TOOL_2', input, _YOUR_TOOL_2)
-}
+export const myTool1 = wrapTool('myTool1', async (input: any) => {
+  return await originalTool1(input)
+})
 
-export async function dispatchTool(name: string, args: Record<string, unknown>) {
-  switch (name) {
-    case 'YOUR_TOOL_1': return YOUR_TOOL_1(args as Parameters<typeof _YOUR_TOOL_1>[0])
-    case 'YOUR_TOOL_2': return YOUR_TOOL_2(args as Parameters<typeof _YOUR_TOOL_2>[0])
-    default: return _dispatchTool(name, args)
-  }
-}
+export const myTool2 = wrapTool('myTool2', async (input: any) => {
+  const { someField } = input as { someField: string }
+  return await originalTool2(someField)
+})
 ```
 
-### Alternative: `wrapTool` shorthand
-
-For simpler setups where you don't need mock support or dashboard replays:
+### Key patterns
 
-```ts
-// ed_tools.ts
-import { wrapTool } from 'elasticdash-test'
-import { YOUR_TOOL_1 as YOUR_TOOL_1_impl } from './YOUR_SOURCE_PATH'
-
-export const YOUR_TOOL_1 = wrapTool('YOUR_TOOL_1', YOUR_TOOL_1_impl)
-```
+- **`wrapTool(name, fn)`** wraps the function with automatic tracing, mocking, and telemetry. Falls back to a passthrough if `elasticdash-test` is not installed.
+- **`eval('require')`** is used instead of `import()` to share the same module instance across `ed_tools.ts` and `ed_workflows.ts`. This avoids ESM/CJS dual-instance issues.
+- **`setElasticDashModule`** shares the loaded module with `ed_workflows.ts` so `edStartTrace`/`edEndTrace` use the same tracing context as `wrapTool`.
+- The exported name (e.g., `myTool1`) can differ from the original function name (e.g., `originalTool1`). The call sites in existing source files will be updated to use the new name in Step 4.
 
 ### Important rules
 
-- The string name passed to `resolveMock()`, `safeRecordToolCall()`, or `wrapTool()` **must match** the exported function name exactly.
+- The string name passed to `wrapTool()` **must match** the exported function name exactly.
 - Each tool function must accept a single input object and return a plain value (JSON-serializable).
 - Tool functions must not close over HTTP context, framework state, or database clients — extract pure logic first.
 
@@ -232,128 +154,173 @@ export default nextConfig
 
 ## Step 3: Create `ed_workflows.ts`
 
-Create `ed_workflows.ts` in the project root. This file exports workflow functions for the ElasticDash runner.
+Create `ed_workflows.ts` in the project root. This file serves two purposes:
+
+1. **Trace lifecycle hooks** (`edStartTrace`, `edEndTrace`) — called from route handlers to mark workflow boundaries
+2. **Workflow exports** — callable functions for the ElasticDash dashboard and test runner
 
 **Before writing this file**: Search the codebase for the main AI workflow functions — async functions that orchestrate LLM calls and tool calls (e.g., a chat handler, an agent loop, a pipeline function). Replace all `YOUR_WORKFLOW` and `YOUR_SOURCE_PATH` placeholders below with the actual function names and import paths.
 
-### Simple case — direct re-export
+### Trace lifecycle hooks
+
+Every `ed_workflows.ts` should export `edStartTrace` and `edEndTrace`. These are called from route handlers (Step 4) to mark when a workflow starts and ends:
 
 ```ts
-// ed_workflows.ts
-// Replace with your actual workflow function and source path
-export { YOUR_WORKFLOW } from './YOUR_SOURCE_PATH'
+// ed_workflows.ts — trace hooks (copy as-is)
+
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+let _ed: any = null
+
+// eslint-disable-next-line @typescript-eslint/no-explicit-any
+export function setElasticDashModule(mod: any): void {
+  _ed = mod
+}
+
+export const edStartTrace = async (workflowName: string): Promise<void> => {
+  if (!_ed) return
+  try {
+    await _ed.tryAutoInitHttpContext()
+    _ed.startTrace(workflowName)
+  } catch (err) {
+    console.error('[ed_workflows] edStartTrace error:', err)
+  }
+}
+
+export const edEndTrace = (): void => {
+  if (!_ed) return
+  try {
+    _ed.endTrace()
+  } catch (err) {
+    console.error('[ed_workflows] edEndTrace error:', err)
+  }
+}
 ```
 
-### Framework adapter case (Next.js / Remix)
+### Workflow exports — simple case
 
-When the workflow lives inside a route handler, create a plain-value wrapper:
+For non-framework projects where the workflow can be imported directly:
 
 ```ts
 // ed_workflows.ts
-import { YOUR_HANDLER as _YOUR_HANDLER } from './app/api/YOUR_ROUTE/route'
-
-export async function YOUR_WORKFLOW(input: { message: string; sessionId: string }) {
-  return _YOUR_HANDLER(input)
-}
+export { YOUR_WORKFLOW } from './YOUR_SOURCE_PATH'
 ```
 
-### Streaming workflow case (Vercel AI SDK)
+### Workflow exports — HTTP mode (Next.js / Remix / framework projects)
 
-Create a separate handler file that is only imported by `ed_workflows.ts`:
+For framework projects, the workflow calls the running dev server instead of importing the route handler:
 
 ```ts
-// app/api/chat-stream/chatStreamHandler.ts
-import { NextRequest } from 'next/server'
-import { readVercelAIStream, recordToolCall } from 'elasticdash-test'
-import type { VercelAIStreamResult } from 'elasticdash-test'
-import { POST } from './route'
+// ed_workflows.ts
+
+const APP_URL = process.env.APP_URL ?? 'http://localhost:3000'
 
-export async function chatStreamHandler(args: {
+export const YOUR_WORKFLOW = async (input: {
   messages: Array<{ role: string; content: string }>
   sessionId?: string
-}): Promise<VercelAIStreamResult> {
-  const req = new NextRequest('http://localhost/api/chat-stream', {
+}): Promise<unknown> => {
+  const response = await fetch(`${APP_URL}/api/YOUR_ENDPOINT`, {
     method: 'POST',
     headers: { 'Content-Type': 'application/json' },
-    body: JSON.stringify(args),
+    body: JSON.stringify(input),
   })
 
-  const response = await POST(req)
-
-  if (response.headers.get('x-vercel-ai-data-stream') !== 'v1') {
-    const errorMessage = await response.text().catch(() => `HTTP ${response.status}`)
-    return { message: errorMessage, type: 'error', error: errorMessage }
+  if (!response.ok) {
+    const text = await response.text()
+    throw new Error(`HTTP ${response.status}: ${text}`)
   }
 
-  const result = await readVercelAIStream(response)
-  recordToolCall('chatStream', args, result)
-  return result
+  return response.json()
 }
 ```
 
-Then re-export from `ed_workflows.ts`:
-
-```ts
-// ed_workflows.ts
-export { chatStreamHandler } from './app/api/chat-stream/chatStreamHandler'
-```
-
 ### Requirements for all workflow exports
 
 - Accept only JSON-serializable inputs (strings, numbers, arrays, plain objects)
 - Return only JSON-serializable outputs
 - Must not depend on framework runtime APIs, HTTP request context, or live service clients
-- If a dependency is non-serializable (e.g., database client), instantiate it inside `ed_workflows.ts`, not passed as a parameter
 
 ---
 
-## Step 4: Wire `ed_workflows.ts` to use `ed_tools.ts`
+## Step 4: Update existing source files to use `ed_tools`
 
-> **Do not modify the user's existing production source files.** Only `ed_workflows.ts` needs to import from `ed_tools.ts`. The production codebase stays untouched.
+> **This step modifies the user's existing source files.** The files that call tool functions must import from `ed_tools` instead of the original source, so that all tool calls go through the ElasticDash tracing layer. Similarly, workflow entry points (route handlers, etc.) must call `edStartTrace` / `edEndTrace` from `ed_workflows`.
 
-The ElasticDash layer sits alongside the production code, not inside it:
+### Architecture after integration
 
 ```
-Production code (unchanged):
-  src/workflows/checkout.ts → imports from src/services/payments.ts directly
+ed_tools.ts
+  ├── imports original functions from services/utils
+  ├── wraps each with wrapTool() for tracing
+  └── exports wrapped versions with the SAME or similar names
+
+ed_workflows.ts
+  ├── exports edStartTrace / edEndTrace for workflow-level tracing
+  └── exports workflow functions (for dashboard/test runner)
 
-ElasticDash layer (new files only):
-  ed_workflows.ts → imports tools from ed_tools.ts → calls the real service functions
+Existing source files (MODIFIED):
+  app/api/chat/route.ts
+    ├── BEFORE: import { myTool } from '@/services/myService'
+    ├── AFTER:  import { myTool } from '@/ed_tools'
+    ├── ADDED:  import { edStartTrace, edEndTrace } from '@/ed_workflows'
+    └── ADDED:  await edStartTrace('workflowName') at handler entry
+                edEndTrace() at handler exit
 ```
 
 ### What to do
 
-In `ed_workflows.ts`, write wrapper functions that call tools through `ed_tools.ts` instead of importing from the original source. The ElasticDash test runner and dashboard call `ed_workflows.ts` — not the production code directly.
+**1. Find every file that calls a tool function and update its imports:**
 
-### Example
+For each tool exported from `ed_tools.ts`, search the codebase for files that import the original function. Update the import to come from `ed_tools` instead.
 
 ```ts
-// ed_workflows.ts
-// Import tools from ed_tools.ts (instrumented), NOT from original source
-import { chargeCard, getOrderDetails } from './ed_tools'
-
-// Re-create the workflow logic using instrumented tools
-export async function checkoutFlow(input: { orderId: string }) {
-  const order = await getOrderDetails({ orderId: input.orderId })
-  const payment = await chargeCard({ amount: order.total })
-  return { orderId: order.orderId, paymentId: payment.id }
-}
+// BEFORE — app/api/chat/route.ts
+import { clarifyAndRefineUserInput } from '@/utils/queryRefinement'
+import { dynamicApiRequest } from '@/services/apiService'
+
+// AFTER — app/api/chat/route.ts
+import { queryRefinement } from '@/ed_tools'
+import { apiService } from '@/ed_tools'
 ```
 
 ```ts
-// ed_tools.ts
-// Import from the ORIGINAL source
-import { chargeCard as chargeCardImpl } from './src/services/payments'
-import { getOrderDetails as getOrderDetailsImpl } from './src/services/orders'
+// BEFORE — app/api/chat/executor.ts
+import { dynamicApiRequest } from '@/services/apiService'
 
-// ... wrap with resolveMock + safeRecordToolCall pattern
+// AFTER — app/api/chat/executor.ts
+import { apiService } from '@/ed_tools'
 ```
 
-### Why this approach
+**2. Add trace hooks to route handlers / workflow entry points:**
 
-- **No production risk**: Existing source files are never modified. Production imports stay as-is.
-- **Clean separation**: The `ed_` files are a parallel test/observability layer, not a refactor of the codebase.
-- **Easy removal**: Delete `ed_tools.ts` and `ed_workflows.ts` to fully remove ElasticDash — no rollback needed.
+At the top of each route handler or workflow entry function, add `edStartTrace` and `edEndTrace`:
+
+```ts
+// app/api/chat/route.ts
+import { edStartTrace, edEndTrace } from '@/ed_workflows'
+
+export async function POST(req: Request) {
+  await edStartTrace('chatHandler')
+  try {
+    // ... existing handler logic, now using tools from ed_tools
+  } finally {
+    edEndTrace()
+  }
+}
+```
+
+**3. Update call sites if the wrapped function has a different name:**
+
+If `ed_tools.ts` exports a tool under a different name than the original (e.g., `queryRefinement` wrapping `clarifyAndRefineUserInput`), update the call sites to use the new name.
+
+### What NOT to modify
+
+- **Do not modify `ed_tools.ts` imports** — it must still import from the original source files.
+- **Do not modify the original tool implementation files** (e.g., `services/apiService.ts`, `utils/queryRefinement.ts`) — they stay as-is.
+- **Do not modify files that don't call tool functions** — only update files that directly import and call the wrapped tools.
+
+### When to ask the user
+
+If you are not confident about a change — e.g., a function might be a utility rather than a tool, or it's unclear whether a file should use the wrapped version — **ask the user** before modifying that file. For clear tool calls (API services, database queries, LLM calls), proceed automatically.
 
 ---
 
@@ -474,7 +441,30 @@ npx ed ed-test --no-upload
 
 ---
 
-## Step 8: Run and verify
+## Step 8: Validate the integration
+
+After all files are created, validate the setup by establishing a socket.io connection to the ElasticDash backend. Run the `observe` command — it connects to the server, authenticates with the API key, and registers the project's tools and workflows:
+
+```bash
+npx elasticdash observe
+```
+
+**Expected output on success:**
+
+```
+[elasticdash] Observability active
+  Session ID : <uuid>
+  Server     : https://server.elasticdash.com
+```
+
+This confirms:
+- `ELASTICDASH_API_URL` and `ELASTICDASH_API_KEY` are valid
+- The socket.io connection to the backend is established
+- The SDK can discover `ed_tools.ts` and `ed_workflows.ts`
+
+**If it fails:** Check that `.env` has valid `ELASTICDASH_API_URL` and `ELASTICDASH_API_KEY` values. If the API key is rejected, the user needs to get a new one from https://app.elasticdash.com.
+
+After validation, stop the observe process (Ctrl+C) and inform the user that ElasticDash is integrated. Provide these commands for ongoing use:
 
 ```bash
 # Run aiTest tests
@@ -486,6 +476,9 @@ npx ed ed-test --no-upload
 # Open the dashboard
 npx elasticdash dashboard
 
+# Start observability (connect to backend for live tracing)
+npx elasticdash observe
+
 # Record a trace fixture
 ELASTICDASH_CAPTURE_TRACE=1 tsx your-workflow.ts
 ```
@@ -591,16 +584,12 @@ After integration, verify these files exist:
 
 ```
 your-project/
-  ed_tools.ts           # Instrumented tool wrappers
-  ed_workflows.ts       # Workflow exports
+  .env                  # ELASTICDASH_API_URL and ELASTICDASH_API_KEY set
+  ed_tools.ts           # Instrumented tool wrappers (real function names, not placeholders)
+  ed_workflows.ts       # Workflow exports using tools from ed_tools.ts
   elasticdash.config.ts # Test runner config
   package.json          # dashboard:ai and test:ai scripts added
   .gitignore            # .temp/ and .ed_traces/ added
 ```
 
-Verify with:
-
-```bash
-npx elasticdash test        # Should discover and run *.ai.test.ts files
-npx elasticdash dashboard   # Should open the dashboard UI
-```
+Verify by running `npx elasticdash observe` — a successful socket.io connection confirms the integration is complete.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "elasticdash-test",
-  "version": "0.1.25",
+  "version": "0.1.26-alpha-1",
   "description": "AI-native test runner for ElasticDash workflow testing",
   "type": "module",
   "bin": {