elasticdash-test 0.1.24 → 0.1.25-alpha-2

package/docs/observability_mode.md

@@ -0,0 +1,195 @@
+# Observability Mode
+
+Observability mode turns the ElasticDash SDK into an always-on tracing instrument. When enabled, every `wrapTool` and `wrapAI` call automatically records and streams trace events to your ElasticDash backend — no test runner required.
+
+## Quick Start
+
+### Option 1: Programmatic (recommended)
+
+Add a single call at your app's entry point:
+
+```typescript
+// instrumentation.ts (Next.js) or server entry point
+import { initObservability } from 'elasticdash-test/http'
+
+const obs = initObservability({
+  serverUrl: 'https://server.elasticdash.com',
+})
+
+// On shutdown (optional — auto-registered on process exit)
+// await obs.shutdown()
+```
+
+### Option 2: Environment Variables Only
+
+If your app already uses `wrapTool` / `wrapAI`, just set the env vars:
+
+```bash
+ELASTICDASH_API_URL=https://server.elasticdash.com \
+ELASTICDASH_API_KEY=ed_key_xxx \
+node server.js
+```
+
+### Option 3: CLI
+
+```bash
+elasticdash observe --server https://server.elasticdash.com
+```
+
+## Configuration
+
+### `initObservability(options?)`
+
+| Option | Env Variable | Default | Description |
+|--------|-------------|---------|-------------|
+| `serverUrl` | `ELASTICDASH_API_URL` | *required* | ElasticDash backend URL |
+| `apiKey` | `ELASTICDASH_API_KEY` | — | Project authentication token |
+| `sessionId` | `ELASTICDASH_SESSION_ID` | auto-generated UUID | Session identifier |
+| `batchIntervalMs` | — | `2000` | How often to flush events (ms) |
+| `maxBatchSize` | — | `50` | Max events per batch before auto-flush |
+| `heartbeatIntervalMs` | — | `30000` | Heartbeat interval (ms) |
+| `sampleRate` | — | `1.0` | Fraction of events to send (0.0–1.0) |
+| `redactKeys` | — | `[]` | Object keys to redact from input/output |
+
+### Return Value
+
+```typescript
+interface ObservabilityHandle {
+  sessionId: string        // The active session ID
+  shutdown: () => Promise<void>  // Graceful shutdown
+}
+```
+
+## Grouping Events by Workflow with `startTrace()`
+
+By default, the SDK discovers workflow names from `ed_workflows.ts`. If exactly one workflow is exported, its name is used as the traceId prefix automatically (e.g. `chatStreamHandler::1712851200000::a1b2c3d4`). If multiple workflows are exported, the traceId defaults to `unknown-workflow` until you call `startTrace()`.
+
+To explicitly group events under a specific workflow, call `startTrace(workflowName)` at the start of each request handler:
+
+```typescript
+import { startTrace } from 'elasticdash-test/http'
+
+// In your route handler, before any wrapTool/wrapAI calls:
+startTrace('chatStreamHandler')
+```
+
+This sets the traceId to `chatStreamHandler::1712851200000::a1b2c3d4`, so all subsequent `wrapTool` / `wrapAI` / `wrapDB` / fetch calls in that request are grouped under the `chatStreamHandler` workflow in the dashboard.
+
+**Important:** Call `startTrace()` before any tool/AI calls execute. If you're using a streaming `ReadableStream`, place it inside the `start()` callback:
+
+```typescript
+const stream = new ReadableStream({
+  async start(controller) {
+    startTrace('chatStreamHandler')
+    // ... workflow logic with wrapTool/wrapAI calls ...
+  },
+})
+```
+
+### Alternative: `wrapWorkflow()`
+
+If you control the workflow function directly, you can use `wrapWorkflow()` instead. It calls `startTrace()` automatically before each invocation:
+
+```typescript
+import { wrapWorkflow } from 'elasticdash-test/http'
+
+export const chatStreamHandler = wrapWorkflow('chatStreamHandler', async (input) => {
+  // All tool/AI calls here are automatically grouped under 'chatStreamHandler'
+  const result = await fetchUser(input.userId)
+  return generateReply(result)
+})
+```
+
+## How It Works
+
+1. **`initObservability()`** creates an `ObservabilityContext` and installs the AI interceptor
+2. Every `wrapTool(name, fn)` and `wrapAI(model, fn)` call checks for this context
+3. When active, the wrapper executes the real function, captures timing/input/output, and enqueues the event
+4. Events are batched and flushed to `POST /api/observability/events` on the backend
+5. A heartbeat event is sent every 30 seconds (configurable) so the backend knows the service is alive
+6. On process exit, remaining events are flushed and a `session_end` event is sent
+
+### Event Flow
+
+```
+wrapTool("searchDB", fn) called
+  → fn(...args) executes normally
+  → WorkflowEvent created: { type: 'tool', name: 'searchDB', input, output, durationMs }
+  → pushTelemetryEvent(event)
+  → TelemetryBatcher.enqueue(event)
+  → Batch flushed every 2s → POST /api/observability/events
+```
+
+## Sampling & Redaction
+
+### Sampling
+
+For high-throughput services, use `sampleRate` to reduce event volume:
+
+```typescript
+initObservability({
+  serverUrl: 'https://server.elasticdash.com',
+  sampleRate: 0.1, // Send only 10% of events
+})
+```
+
+### Redaction
+
+Strip sensitive fields from input/output before sending:
+
+```typescript
+initObservability({
+  serverUrl: 'https://server.elasticdash.com',
+  redactKeys: ['apiKey', 'password', 'ssn', 'credit_card'],
+})
+```
+
+This deep-clones and replaces matching keys (case-insensitive) with `"[REDACTED]"` before serialization.
+
+## Batching & Reliability
+
+Events are not sent individually — they are buffered and flushed in batches:
+
+- **Interval flush**: every `batchIntervalMs` (default 2 seconds)
+- **Size flush**: when the buffer reaches `maxBatchSize` (default 50 events)
+- **Exit flush**: on `beforeExit`, `SIGTERM`, or `SIGINT`
+
+Failed flushes are retried with exponential backoff (1s, 2s, 4s) up to 3 times. After max retries, events are dropped to prevent memory leaks.
+
+## Graceful Shutdown
+
+Shutdown happens automatically on process exit signals. For manual control (e.g., serverless functions):
+
+```typescript
+import { shutdownObservability } from 'elasticdash-test/http'
+
+// In your cleanup handler
+await shutdownObservability()
+```
+
+This flushes all buffered events, sends a `session_end` marker, and clears the context.
+
+## Debug Logging
+
+To see SDK internal logs (telemetry push status, batch flush counts, etc.), set:
+
+```bash
+ELASTICDASH_DEBUG=1 node server.js
+```
+
+All internal `console.log` calls are gated behind this flag and produce no output by default.
+
+## Comparison with Test Mode
+
+| Feature | Test Mode | Observability Mode |
+|---------|-----------|-------------------|
+| Requires test runner | Yes | No |
+| Mocking support | Yes | No |
+| Step replay/freezing | Yes | No |
+| Event delivery | Fire-and-forget per event | Batched with retry |
+| Sampling | No | Yes |
+| Redaction | No | Yes |
+| Heartbeat | No | Yes |
+| Graceful shutdown | No | Yes |
+
+Both modes use the same `wrapTool` / `wrapAI` wrappers — the SDK detects which context is active and routes events accordingly.