elasticdash-sdk 0.2.0

package/docs/instrumentation.md

@@ -0,0 +1,424 @@
+# Instrumentation Guide — `ed_tools`, `ed_workflows`, and `ed_agents`
+
+This document describes the three instrumentation files that connect your production business logic to the ElasticDash test SDK.
+
+> **Note — examples are from demo projects.**
+> Code samples use two reference projects: a Deno-based Pokémon API agent and a Node.js/Next.js chat application. Function names such as `getPokemon`, `generateAIResponse`, `dataService`, and `chatHandler` are **placeholders** — replace them with your own tool functions, workflow functions, and source file names. The patterns and rules apply to any project; runtime-specific differences are called out explicitly in the `ed_tools.ts` section.
+
+---
+
+## Prerequisites
+
+Before creating any of the three `ed_` files, the following conditions must hold in your production codebase.
+
+### 1. Tool logic must be isolated in callable functions
+
+Each tool must be a standalone exported async function that takes a single typed input object and returns a plain value. It must not close over HTTP context, framework state, or database clients.
+
+```ts
+// your-tools.ts — correct shape
+// Replace `getPokemon` / `GetPokemonInput` with your own tool name and input type
+export async function getPokemon(input: { name_or_id: string | number }) {
+  const res = await fetch(`https://pokeapi.co/api/v2/pokemon/${input.name_or_id}`)
+  return condense(await res.json())
+}
+```
+
+A tool that reads from `ctx.request`, accepts a framework `Context`, or calls a database client injected at construction time **cannot** be wrapped directly. Extract the pure logic first.
+
+### 2. Tool imports must be compatible with your runtime
+
+- **Deno projects:** tools may use `https://esm.sh/` URLs and Deno-native globals
+- **Node.js projects:** tools must use npm packages — no `Deno.*`, `jsr:`, or `https://deno.land/` URLs
+
+The global `fetch` API works in both runtimes.
+
+### 3. A `dispatchTool(name, args)` function must exist *(Pattern A only)*
+
+If you use the `withTrace` HOF pattern (Pattern A below), the workflow calls tools through a dispatcher keyed by string name. This is the seam `ed_tools.ts` replaces:
+
+```ts
+// your-tools.ts — required dispatcher shape (Pattern A only)
+// Replace case values with your own tool names
+export async function dispatchTool(name: string, args: Record<string, unknown>) {
+  switch (name) {
+    case 'getPokemon': return getPokemon(args as GetPokemonInput)
+    default: throw new Error(`Unknown tool: ${name}`)
+  }
+}
+```
+
+The string case values (`'getPokemon'`) must match the `function.name` fields in the LLM tool definitions array. If the LLM sees `"name": "get_pokemon"` but `dispatchTool` switches on `'getPokemon'`, tool calls will silently fail.
+
+Node.js projects using the inline async pattern (Pattern B) do not need a central dispatcher.
+
+### 4. Workflow logic must be a plain callable function
+
+The agentic loop must be an exported async function whose parameters and return value are plain JSON-serialisable values — no HTTP request objects, no framework context, no live database clients.
+
+```ts
+// your-ai.ts — correct shape
+// Replace `generateAIResponse` and its parameters with your own workflow function
+export async function generateAIResponse(
+  chatId: string,
+  userId: string,
+  history: ChatMessage[],
+): Promise<string> { ... }
+```
+
+If the workflow currently accepts a framework `Context` or a database client, those dependencies must be extracted before `ed_workflows.ts` can re-export the function. Move them to the call site (the route handler) and pass only plain values into the workflow.
+
+### 5. The workflow must call tools through the dispatcher *(Pattern A only)*
+
+For Pattern A, your workflow file must call `dispatchTool()` — not individual tool functions directly — so that swapping the import from your tools file to `ed_tools.ts` instruments all tool calls at once:
+
+```ts
+// your-ai.ts — required pattern (Pattern A only)
+const toolResult = await dispatchTool(tc.function.name, args)
+```
+
+For Pattern B (Node.js inline), the workflow imports individual tool functions from `ed_tools.ts` directly.
+
+### 6. Project files for the test runner must exist
+
+Three files are needed alongside your source:
+
+**`package.json`**
+
+```json
+{
+  "type": "module",
+  "scripts": {
+    "test": "elasticdash",
+    "dashboard": "elasticdash dashboard"
+  },
+  "dependencies": {
+    "elasticdash-sdk": "<version or local path>",
+    "openai": "^6.x"
+  }
+}
+```
+
+**`tsconfig.json`** — scoped to only the files the test runner needs:
+
+```json
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "ESNext",
+    "moduleResolution": "bundler",
+    "esModuleInterop": true,
+    "strict": true,
+    "skipLibCheck": true,
+    "types": ["node"]
+  },
+  "include": [
+    "your-tools.ts",
+    "ed_tools.ts",
+    "ed_workflows.ts",
+    "elasticdash.config.ts",
+    "**/*.ai.test.ts"
+  ]
+}
+```
+
+**`elasticdash.config.ts`**
+
+```ts
+export default {
+  testMatch: ['**/*.ai.test.ts'],
+  traceMode: 'local' as const,
+}
+```
+
+---
+
+## `ed_tools.ts` — Instrumented tool wrappers
+
+`ed_tools.ts` wraps every tool function from your tools file with `recordToolCall()` so each invocation is captured in the active ElasticDash trace.
+
+There are two patterns. Choose the one that fits your project structure.
+
+> **Import style note:**
+> Deno projects use `.ts` extensions in relative imports (`from './tools.ts'`).
+> Node.js projects with `tsx` omit extensions (`from './services/dataService'`).
+
+---
+
+### Pattern A — `withTrace` HOF *(Deno and simple projects)*
+
+Use this pattern when your tools are pure functions and the workflow calls them through a central `dispatchTool` dispatcher.
+
+#### The `withTrace` helper
+
+Copy this as-is. All tool functions go through this single private helper:
+
+```ts
+async function withTrace<I, O>(
+  toolName: string,
+  input: I,
+  fn: (input: I) => Promise<O>,
+): Promise<O> {
+  const result = await fn(input)
+  try {
+    const { recordToolCall } = await import('elasticdash-sdk')
+    recordToolCall(toolName, input, result)
+  } catch { /* tracing must never block business logic */ }
+  return result
+}
+```
+
+#### Exported tool functions
+
+Each tool gets a one-liner delegate:
+
+```ts
+// Replace `getPokemon` / `_getPokemon` with your tool name
+export function getPokemon(input: Parameters<typeof _getPokemon>[0]) {
+  return withTrace('getPokemon', input, _getPokemon)
+}
+```
+
+`Parameters<typeof _fn>[0]` keeps input types in sync with your tools file — no need to duplicate type definitions.
+
+#### The instrumented dispatcher
+
+`ed_tools.ts` re-exports a `dispatchTool` that routes to the traced functions, not the raw ones:
+
+```ts
+// Replace case strings with your own tool names
+export async function dispatchTool(name: string, args: Record<string, unknown>) {
+  switch (name) {
+    case 'getPokemon':  return getPokemon(args as Parameters<typeof _getPokemon>[0])
+    // ... one case per tool
+    default: return _dispatchTool(name, args)  // fallback to un-traced for unknowns
+  }
+}
+```
+
+Your workflow file imports `dispatchTool` from `ed_tools.ts` instead of from your raw tools file. This single import change makes the entire production workflow observable by ElasticDash.
+
+#### When you add a new tool
+
+1. Add a one-liner export in `ed_tools.ts` using `withTrace`
+2. Add a `case` in `ed_tools.ts`'s `dispatchTool` switch
+3. Add a `case` in your tools file's `dispatchTool` switch (the fallback)
+4. Add a matching entry in the LLM `TOOLS` array with the same name string
+
+All four must use the same name string or tool calls will be dispatched incorrectly.
+
+---
+
+### Pattern B — Inline async with mock support *(Node.js projects)*
+
+Use this pattern when tools call database clients or framework services that are imported at module level (not pure `fetch`), when there is no central dispatcher, or when you want mock support in the dashboard's "Validate Updated Flow with Live Data" panel.
+
+#### The `resolveMock` and `safeRecordToolCall` helpers
+
+Copy these as-is — no customisation needed.
+
+`resolveMock` enables the dashboard's **Tool Mocking** panel: it checks whether a tool should return recorded data instead of calling the real service. It is a zero-cost no-op outside the ElasticDash worker subprocess. Pattern A does not support tool mocking.
+
+`safeRecordToolCall` records the tool call in the trace, but only when running inside the worker subprocess. It is silent in production.
+
+```ts
+function resolveMock(toolName: string): { mocked: true; result: unknown } | { mocked: false } {
+  const g = globalThis as any
+  const mocks = g.__ELASTICDASH_TOOL_MOCKS__
+  if (!mocks) return { mocked: false }
+
+  const entry = mocks[toolName]
+  if (!entry || entry.mode === 'live') return { mocked: false }
+
+  if (!g.__ELASTICDASH_TOOL_CALL_COUNTERS__) g.__ELASTICDASH_TOOL_CALL_COUNTERS__ = {}
+  const counters = g.__ELASTICDASH_TOOL_CALL_COUNTERS__
+  counters[toolName] = (counters[toolName] ?? 0) + 1
+  const callNumber = counters[toolName]
+
+  if (entry.mode === 'mock-all') {
+    const data = entry.mockData ?? {}
+    const result = data[callNumber] !== undefined ? data[callNumber] : data[0]
+    return { mocked: true, result }
+  }
+
+  if (entry.mode === 'mock-specific') {
+    const indices = entry.callIndices ?? []
+    if (indices.includes(callNumber)) {
+      return { mocked: true, result: (entry.mockData ?? {})[callNumber] }
+    }
+    return { mocked: false }
+  }
+
+  return { mocked: false }
+}
+
+async function safeRecordToolCall(tool: string, input: any, result: any) {
+  if (!(globalThis as any).__ELASTICDASH_WORKER__) return
+  try {
+    const { recordToolCall } = await import('elasticdash-sdk')
+    recordToolCall(tool, input, result)
+  } catch { /* tracing must never block business logic */ }
+}
+```
+
+#### Per-tool inline pattern
+
+Each exported tool function follows the same shape — mock check, real call, record:
+
+```ts
+// Replace `dataService`, `runSelectQuery`, and import path with your own
+import { runSelectQuery } from './services/dataService'
+
+export const dataService = async (input: any) => {
+  const mock = resolveMock('dataService')
+  if (mock.mocked) {
+    await safeRecordToolCall('dataService', input, mock.result)
+    return mock.result
+  }
+
+  const { query } = input as { query: string }
+  return await runSelectQuery(query)
+    .then(async (res: any) => {
+      await safeRecordToolCall('dataService', input, res)
+      return res
+    })
+    .catch(async (err: any) => {
+      await safeRecordToolCall('dataService', input, err)
+      throw err
+    })
+}
+```
+
+This pattern does **not** require a central `dispatchTool`. The workflow imports each tool directly from `ed_tools.ts`.
+
+#### When you add a new tool
+
+1. Add an import for the underlying service function at the top of `ed_tools.ts`
+2. Add an exported async function with the mock check and `safeRecordToolCall` pattern
+3. Add a matching entry in the LLM `TOOLS` array with the same name string
+
+---
+
+## `ed_workflows.ts` — Workflow adapter
+
+`ed_workflows.ts` re-exports workflow functions to give the ElasticDash runner a clean import surface. Every exported function must:
+
+- Accept only plain JSON-serialisable inputs (strings, numbers, arrays, plain objects)
+- Return only plain JSON-serialisable outputs
+- Not depend on framework runtime APIs, HTTP request context, or live service clients
+
+Simple case — direct re-export:
+
+```ts
+// Replace `generateAIResponse` and `./ai.ts` with your workflow function and source file
+export { generateAIResponse } from './ai.ts'
+```
+
+**Node.js / Next.js projects:** Workflow logic often lives inside a framework route handler. `ed_workflows.ts` is where you strip the framework types and expose a plain-value wrapper:
+
+```ts
+// ed_workflows.ts — Node.js/Next.js example
+// Replace names and import paths with your own
+import { chatHandler as _chatHandler } from './app/api/chat/route'
+
+export async function chatHandler(input: { message: string; sessionId: string }) {
+  return _chatHandler(input)
+}
+```
+
+If a workflow ever acquires a non-serialisable parameter such as a database client, instantiate it here instead of passing it in:
+
+```ts
+// ed_workflows.ts — adapter example if your workflow gained a db parameter
+import { generateAIResponse as _generateAIResponse } from './ai.ts'
+import { createServiceClient } from './supabase_client.ts'
+
+export async function generateAIResponse(
+  chatId: string,
+  userId: string,
+  history: { role: string; content: string }[],
+) {
+  const db = createServiceClient()  // instantiated here, not passed in
+  return _generateAIResponse(chatId, userId, history, db)
+}
+```
+
+---
+
+## `ed_agents.ts` — Structured plan/task execution
+
+`ed_agents.ts` exposes a three-function API for multi-step agents with mid-trace resumption.
+
+In your project, `ed_agents.ts` is typically a thin re-export pointing to wherever `plannerAgent` and `executorAgent` live in your codebase:
+
+```ts
+// Replace the import path with wherever your agent logic lives
+export { plannerAgent, executorAgent } from './utils/aiHandler'
+```
+
+If you haven't written your own planner/executor yet, use the SDK's reference implementations:
+
+```ts
+export { plannerAgent, executorAgent, resumeAgentFromTrace } from 'elasticdash-sdk'
+```
+
+### Preconditions
+
+**All agent tools must be exported from `ed_tools.ts`.** The executor resolves tool functions by string name from `ed_tools.ts` exports. Any tool not exported there will cause the task to fail with `Tool "x" not found in registry`.
+
+**Task inputs and outputs must be JSON-serialisable.** Non-serialisable values (class instances, functions, circular references) will be lost when tasks wire their outputs as inputs to subsequent tasks.
+
+### The three functions
+
+**`plannerAgent(userQuery, context)`**
+
+Converts a user query and optional context into an `AgentPlan` — a typed list of `AgentTask` objects. Each task declares:
+
+- `id` — unique string identifier
+- `tool` — the string name of the tool to call (must match an export in `ed_tools.ts`)
+- `input` — the input object, which may contain `$ref` placeholders resolved from earlier task outputs
+
+```ts
+{
+  id: 'task-2',
+  tool: 'taskSelectorService',   // replace with your tool name
+  input: {
+    queryEmbedding: { $ref: 'task-1.output.embedding' },  // filled from task-1's result
+    topK: 3,
+  },
+}
+```
+
+The default `plannerAgent` builds a static plan. Replace it with LLM-based planning logic to generate tasks dynamically.
+
+**`executorAgent(plan, resumeFrom?)`**
+
+Executes tasks sequentially. On any failure, marks the plan `'failed'` and stops — remaining tasks do not run.
+
+The `resumeFrom` parameter (default `0`) skips all tasks before that index, using their already-recorded outputs for `$ref` resolution. This enables mid-trace resumption from any task.
+
+**`resumeAgentFromTrace(state)`**
+
+Resumes a partially-completed run from a serialized agent state. Use this to restart a failed or paused agent from any task without replaying steps that already succeeded.
+
+### Relationship between the three files
+
+```text
+ed_agents.ts
+  └── executorAgent() resolves tools by name
+        │
+        ▼
+  ed_tools.ts  (all exported tool functions)
+    ├── yourTool1()  ─┐
+    ├── yourTool2()  ─┤─ all wrapped with withTrace() or safeRecordToolCall() → recordToolCall()
+    └── ...          ─┘
+          │
+          ▼
+      your-tools.ts / your service files  (pure business logic)
+
+ed_workflows.ts
+  └── re-exports your workflow function from your workflow file
+        └── workflow calls dispatchTool() or individual tools imported from ed_tools.ts
+```
+
+`ed_tools.ts` is the single instrumentation point. Whether a tool call originates from the LLM-driven agentic loop or from `executorAgent`, it flows through the tracing wrapper and is recorded identically.

package/docs/langfuse-trace-structure.md

@@ -0,0 +1,145 @@
+# Langfuse Trace Structure for ElasticDash Replay
+
+ElasticDash reads Langfuse traces to replay and compare workflow runs. For replay to work correctly, observations in the trace must follow two structural rules.
+
+> **Note — examples use demo names.**
+> Span names like `"generateAIResponse"`, `"tool-getPokemon"`, and `"tool-listPokemon"` are from a sample Pokémon demo project. Replace them with your own workflow function name and tool names.
+
+---
+
+## Rule A — Parent workflow span
+
+There must be a span whose `name` exactly matches the exported workflow function name. Its `input` must be the workflow function's arguments in the same order they appear in the function signature, arranged as an array when there are multiple parameters. Its `output` must be the workflow's return value.
+
+This is what ElasticDash uses to re-invoke the workflow with the original inputs and diff the new output against the recorded one.
+
+```ts
+// Replace "generateAIResponse" with your workflow function name
+const workflowSpan = trace.span({
+  name: "generateAIResponse",       // must equal the exported function name exactly
+  input: [chatId, userId, history], // all args in declaration order, wrapped in array
+  startTime: workflowStartTime,
+})
+
+// ... workflow executes ...
+const response = await generateAIResponse(chatId, userId, history);
+
+workflowSpan.end({ output: response }) // must be the actual return value
+```
+
+If the function takes a single argument, `input` can be the value directly rather than a one-element array. The only requirement is that the stored value is directly usable as the argument for re-invocation with no transformation.
+
+### Multiple parameters → always use an array
+
+```ts
+// Function signature (replace with your own):
+async function generateAIResponse(chatId: string, userId: string, history: ChatMessage[])
+
+// Correct span input — preserves argument identity and order:
+input: [chatId, userId, history]
+
+// Wrong — loses argument boundaries, cannot be used to re-invoke:
+input: { chatId, userId, history }
+
+// Wrong — loses all but the last argument:
+input: history
+```
+
+### Output must be set before `flushAsync`
+
+`workflowSpan.end({ output })` must be called and `await langfuse.flushAsync()` must complete before the function returns. If the span is flushed without an output, ElasticDash records the run with no output and output comparison is impossible.
+
+```ts
+// Correct — every return path ends the span and awaits flush
+workflowSpan.end({ output: response })
+await langfuse.flushAsync()
+return response
+```
+
+---
+
+## Rule B — Tool call spans
+
+Every tool invocation must produce a child span of the workflow span. The span must be identifiable as a tool call by satisfying at least one of:
+
+- Its `name` is in the format `tool-[functionName]` (e.g. `tool-getPokemon`, `tool-listPokemon`) — replace with your own tool names
+- Its observation `type` is `'TOOL'` and the name is the function name (`[functionName]`)
+
+The `[functionName]` part must match the string used in `dispatchTool` and in the LLM tool definitions array — the same name passed to `recordToolCall` in `ed_tools.ts`.
+
+### Input and output requirements
+
+The span's `input` must be the exact arguments object as passed to the tool function. The span's `output` must be the complete result returned by the tool. Neither may be truncated, summarised, or transformed.
+
+```ts
+// Replace "tool-getPokemon" and "getPokemon" with your tool name
+const toolSpan = workflowSpan.span({
+  name: `tool-${tc.function.name}`, // e.g. "tool-getPokemon"
+  input: args,                       // the parsed args object, unchanged
+})
+
+const toolResult = await dispatchTool(tc.function.name, args)
+const result = JSON.stringify(toolResult) // full result
+toolSpan.end({ output: result })          // must be complete
+```
+
+If the result is too large to display comfortably in the Langfuse UI, that is a display concern — the span data itself must be the full value. ElasticDash uses tool span outputs to replay individual tool calls in isolation and to verify that re-runs produce consistent results.
+
+### Span must be a child of the workflow span
+
+Tool spans must be created from `workflowSpan.span(...)`, not from `trace.span(...)`. A tool span attached directly to the trace root is not associated with the workflow invocation and will not be linked to the correct replay context.
+
+```ts
+// Correct — child of workflowSpan
+const toolSpan = workflowSpan.span({ name: `tool-${name}`, input: args })
+
+// Wrong — attached to trace root, not to the workflow invocation
+const toolSpan = trace.span({ name: `tool-${name}`, input: args })
+```
+
+---
+
+## What breaks if these rules are not followed
+
+| Violation | Effect |
+| --- | --- |
+| Workflow span name does not match the function name | ElasticDash cannot locate the workflow entry point in the trace; replay fails entirely |
+| Workflow span input is not the raw function arguments | Re-invocation uses wrong inputs; the output diff compares unrelated runs |
+| Workflow span input uses an object instead of an array for multiple params | Argument positions are lost; re-invocation cannot reconstruct the call correctly |
+| `workflowSpan.end()` is not called on every return path | Span is recorded as open; output is null; comparison is impossible |
+| `flushAsync` is not awaited before the function returns | Spans are still buffered when the process exits; trace is partially or fully lost |
+| Tool span name does not match `tool-[name]` and type is not `'TOOL'` | Tool calls are invisible to the replay engine; tool-level diffs and replays are lost |
+| Tool span is attached to the trace root instead of `workflowSpan` | Tool call is not associated with the workflow invocation; replay linkage breaks |
+| Tool span input or output is truncated or transformed | Replayed tool call uses different args or an incomplete result; comparison is invalid |
+
+---
+
+## Reference — example implementation
+
+The following example follows all rules. Replace `generateAIResponse`, `tc.function.name`, and `dispatchTool` with your own workflow function and tool dispatch logic:
+
+```ts
+// Parent workflow span — name matches function, input is arg array, output is return value
+const workflowSpan = trace.span({
+  name: "generateAIResponse",         // your workflow function name
+  input: [chatId, userId, history],   // your args in declaration order
+  startTime: workflowStartTime,
+})
+
+// Tool spans — child of workflowSpan, named tool-[functionName], full input/output
+const toolSpan = workflowSpan.span({
+  name: `tool-${tc.function.name}`,   // e.g. "tool-getPokemon" — use your tool name
+  input: args,
+  metadata: { tool_name: tc.function.name },
+})
+const toolResult = await dispatchTool(tc.function.name, args)
+const result = JSON.stringify(toolResult)
+toolSpan.end({ output: result })
+
+// The rest of the workflow logic...
+
+// Every return path ends the workflow span and awaits flush
+workflowSpan.end({ output: response })
+await langfuse.flushAsync()
+return response
+```