elasticdash-test 0.1.26 → 0.1.27

package/docs/observability_rerun_backend_plan.md

@@ -1,596 +0,0 @@
-# Backend Plan: Observability Rerun Pipeline
-
-This document describes everything the backend needs to implement so that the SDK's observability mode can receive rerun triggers, execute them, and return results for evaluation.
-
-The SDK side is complete. This plan covers only backend and dashboard work.
-
----
-
-## How It Works (end-to-end)
-
-```
-SDK (user's server)                              Backend
-────────────────────                             ────────
-
-1. SDK sends event batch
-   POST /api/observability/events ──────────>    Ingests events into observability_events
-                                                 Checks sampling_configs for matches
-                                                 If match: creates trigger, includes in response
-
-2. <── Response with trigger ──────────────      { ok: true, ingested: 5, trigger: {
-                                                     triggerId: 42, runCount: 3,
-                                                     steps: [{ eventType, eventName,
-                                                               input, model, provider,
-                                                               originalEventDbId }]
-                                                   }}
-
-3. SDK pre-validates each step:
-   - Tool: checks ed_tools.ts has it
-   - AI: checks API key env var exists
-   Executes available steps × runCount
-   Collects output, durationMs, token usage
-
-4. SDK POSTs results
-   POST /api/observability/triggers/42/results -> Receives results
-                                                  Runs evaluations per sampling config
-                                                  Stores results + evaluations as JSONB
-                                                  Alerts if thresholds breached
-```
-
----
-
-## Database Schema (2 tables)
-
-### Table: `sampling_configs`
-
-Persistent user-defined rules. One config can produce many triggers over time.
-
-```sql
-CREATE TABLE sampling_configs (
-  id            SERIAL PRIMARY KEY,
-  project_id    UUID NOT NULL REFERENCES projects(id),
-  name          VARCHAR(255) NOT NULL,
-  enabled       BOOLEAN NOT NULL DEFAULT true,
-
-  -- Which traces to match
-  trace_filter  JSONB NOT NULL,
-  -- Which steps within matched traces to rerun
-  step_selector JSONB NOT NULL,
-  -- How many times to re-execute each step
-  run_count     INT NOT NULL DEFAULT 1 CHECK (run_count BETWEEN 1 AND 50),
-  -- Fraction of matching traces that actually trigger (0.0–1.0)
-  sample_rate   FLOAT NOT NULL DEFAULT 1.0 CHECK (sample_rate BETWEEN 0.0 AND 1.0),
-  -- Evaluations to run on results
-  evaluations   JSONB NOT NULL DEFAULT '[]',
-  -- Alert configuration
-  alerting      JSONB NOT NULL DEFAULT '{}',
-
-  created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-  updated_at    TIMESTAMPTZ NOT NULL DEFAULT NOW()
-);
-
-CREATE INDEX idx_sampling_configs_project ON sampling_configs(project_id, enabled);
-```
-
-**`trace_filter` format:**
-
-```json
-{
-  "serviceId": "my-ai-app",
-  "eventTypes": ["tool", "ai"],
-  "eventNames": ["searchDB", "gpt-4o"]
-}
-```
-
-All fields are optional. Omitted fields match everything. Rules:
-- `serviceId` — exact match on session's service ID
-- `eventTypes` — match events of these types
-- `eventNames` — match events with these names (tool name or model name)
-
-**`step_selector` format:**
-
-```json
-{ "mode": "by_name", "names": ["searchDB", "gpt-4o"] }
-```
-Or:
-```json
-{ "mode": "all", "types": ["tool", "ai"] }
-```
-
-`by_name` selects specific steps. `all` selects all steps of the given types from the matched trace.
-
-**`evaluations` format:**
-
-```json
-[
-  { "type": "latency-budget", "maxDurationMs": 5000 },
-  { "type": "token-budget", "maxTokens": 1000 },
-  { "type": "output-contains", "containsText": "pikachu", "notContainsText": "error" },
-  { "type": "output-schema", "jsonSchema": { "type": "object", "required": ["results"] } },
-  { "type": "determinism", "similarityThreshold": 0.9 },
-  {
-    "type": "llm-judge",
-    "judgePrompt": "Rate the quality and accuracy of this output on a scale of 1-10. Output format: SCORE: <number>",
-    "threshold": 7.0,
-    "judgeModel": "gpt-4o",
-    "judgeProvider": "openai"
-  }
-]
-```
-
-**`alerting` format:**
-
-```json
-{
-  "onFailure": true,
-  "webhookUrl": "https://hooks.slack.com/...",
-  "emailTo": ["dev@company.com"]
-}
-```
-
-### Table: `triggers`
-
-One-time execution instance of a sampling config. Stores the full lifecycle: what was sent, what came back, and evaluation outcomes — all as JSONB.
-
-```sql
-CREATE TABLE triggers (
-  id                  SERIAL PRIMARY KEY,
-  project_id          UUID NOT NULL REFERENCES projects(id),
-  sampling_config_id  INT NOT NULL REFERENCES sampling_configs(id),
-  session_id          UUID NOT NULL,
-  status              VARCHAR(20) NOT NULL DEFAULT 'pending',
-    -- pending: created, not yet included in a batch response
-    -- sent: included in batch response to SDK, waiting for results
-    -- completed: results received and evaluations stored
-    -- timed_out: SDK didn't respond within 10 minutes
-  run_count           INT NOT NULL,
-
-  -- What was sent to the SDK
-  steps_sent          JSONB NOT NULL,
-
-  -- What came back from the SDK (null until completed)
-  results             JSONB,
-
-  -- Evaluation outcomes computed by the backend (null until completed)
-  evaluations         JSONB,
-
-  -- Summary counts (denormalized for list queries)
-  steps_count         INT,
-  steps_available     INT,
-  steps_unavailable   INT,
-  evaluations_passed  INT,
-  evaluations_failed  INT,
-
-  created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-  sent_at             TIMESTAMPTZ,
-  completed_at        TIMESTAMPTZ
-);
-
-CREATE INDEX idx_triggers_project_status ON triggers(project_id, status);
-CREATE INDEX idx_triggers_session ON triggers(session_id);
-CREATE INDEX idx_triggers_config ON triggers(sampling_config_id);
-CREATE INDEX idx_triggers_sent_timeout ON triggers(status, sent_at)
-  WHERE status = 'sent';
-```
-
-**`steps_sent` format (what goes to SDK):**
-
-```json
-[
-  {
-    "eventId": 1,
-    "eventType": "tool",
-    "eventName": "searchDB",
-    "originalEventDbId": 102,
-    "input": { "query": "pikachu" }
-  },
-  {
-    "eventId": 2,
-    "eventType": "ai",
-    "eventName": "gpt-4o",
-    "originalEventDbId": 103,
-    "input": { "messages": [{"role": "user", "content": "..."}] },
-    "model": "gpt-4o",
-    "provider": "openai"
-  }
-]
-```
-
-**`results` format (what comes back from SDK):**
-
-```json
-[
-  {
-    "originalEventDbId": 102,
-    "eventType": "tool",
-    "eventName": "searchDB",
-    "available": true,
-    "runs": [
-      { "runIndex": 0, "input": {...}, "output": {...}, "durationMs": 45 },
-      { "runIndex": 1, "input": {...}, "output": {...}, "durationMs": 52 }
-    ]
-  },
-  {
-    "originalEventDbId": 103,
-    "eventType": "ai",
-    "eventName": "gpt-4o",
-    "available": false,
-    "unavailableReason": "Missing API key for provider \"openai\". Expected: OPENAI_API_KEY",
-    "runs": []
-  }
-]
-```
-
-**`evaluations` format (computed by backend):**
-
-```json
-[
-  {
-    "originalEventDbId": 102,
-    "eventName": "searchDB",
-    "results": [
-      { "type": "latency-budget", "passed": true, "detail": { "maxDurationMs": 5000, "actualMaxMs": 52 } },
-      { "type": "determinism", "passed": true, "detail": { "threshold": 0.9, "similarity": 0.95 } }
-    ]
-  },
-  {
-    "originalEventDbId": 103,
-    "eventName": "gpt-4o",
-    "results": [
-      { "type": "availability", "passed": false, "detail": { "reason": "Missing API key..." } }
-    ]
-  }
-]
-```
-
----
-
-## API Endpoints
-
-### 1. `POST /api/observability/events` — Modify Existing
-
-After ingesting the event batch, check sampling configs:
-
-```
-function handleObservabilityEvents(req, res):
-  1. Validate & ingest events (existing logic)
-  2. responseBody = { ok: true, ingested: events.length }
-
-  3. Find matching sampling_configs:
-     SELECT * FROM sampling_configs
-     WHERE project_id = :projectId
-       AND enabled = true
-
-  4. For each config, check if trace_filter matches any event in this batch:
-     - serviceId matches session's serviceId?
-     - eventTypes includes any event.type in batch?
-     - eventNames includes any event.name in batch?
-
-  5. If match AND Math.random() < config.sample_rate:
-     a. Check no pending/sent trigger exists for this config + session:
-        SELECT 1 FROM triggers
-        WHERE sampling_config_id = :configId
-          AND session_id = :sessionId
-          AND status IN ('pending', 'sent')
-        → If exists, skip (don't double-trigger)
-
-     b. Select steps to rerun based on step_selector:
-        - mode "by_name": find events in batch matching names
-        - mode "all": find events in batch matching types
-        For each selected event, look up its full input from observability_events:
-          SELECT input, event_name, event_type
-          FROM observability_events
-          WHERE id = :eventDbId
-
-     c. Build trigger steps with full input data:
-        steps = [{
-          eventId: <sequential>,
-          eventType: event.event_type,
-          eventName: event.event_name,
-          originalEventDbId: event.id,
-          input: event.input,          // REQUIRED: SDK needs this to re-execute
-          model: event.event_name,     // For AI events
-          provider: <inferred>         // For AI events: infer from model name
-        }]
-
-     d. INSERT INTO triggers (
-          project_id, sampling_config_id, session_id,
-          status, run_count, steps_sent, steps_count, sent_at
-        ) VALUES (
-          :projectId, :configId, :sessionId,
-          'sent', :config.run_count, :steps, :steps.length, NOW()
-        )
-
-     e. Add trigger to response:
-        responseBody.trigger = {
-          triggerId: trigger.id,
-          runCount: config.run_count,
-          steps: steps
-        }
-
-  6. Return responseBody (status 202)
-```
-
-**Important:** Only one trigger per response. If multiple configs match, pick the highest priority or oldest. Queue others for the next batch.
-
-### 2. `POST /api/observability/triggers/:triggerId/results` — New
-
-Receives execution results from the SDK.
-
-**Request body (from SDK):**
-
-```json
-{
-  "steps": [
-    {
-      "originalEventDbId": 102,
-      "eventType": "tool",
-      "eventName": "searchDB",
-      "available": true,
-      "runs": [
-        { "runIndex": 0, "input": {...}, "output": {...}, "durationMs": 45 },
-        { "runIndex": 1, "input": {...}, "output": {...}, "durationMs": 52 }
-      ]
-    },
-    {
-      "originalEventDbId": 103,
-      "eventType": "ai",
-      "eventName": "gpt-4o",
-      "available": false,
-      "unavailableReason": "Missing API key for provider \"openai\". Expected: OPENAI_API_KEY",
-      "runs": []
-    }
-  ]
-}
-```
-
-**Processing logic:**
-
-```
-function handleTriggerResults(triggerId, body):
-  1. Find trigger:
-     SELECT * FROM triggers WHERE id = :triggerId
-     → 404 if not found
-     → 409 if status != 'sent' (already completed or timed out)
-
-  2. Load sampling config:
-     SELECT evaluations FROM sampling_configs WHERE id = trigger.sampling_config_id
-
-  3. Store raw results:
-     UPDATE triggers SET results = body.steps WHERE id = :triggerId
-
-  4. Run evaluations for each step:
-     allEvaluations = []
-     passedCount = 0
-     failedCount = 0
-     availableCount = 0
-     unavailableCount = 0
-
-     For each step in body.steps:
-       stepEvals = { originalEventDbId: step.originalEventDbId, eventName: step.eventName, results: [] }
-
-       If step.available == false:
-         unavailableCount++
-         stepEvals.results.push({ type: 'availability', passed: false, detail: { reason: step.unavailableReason } })
-         failedCount++
-       Else:
-         availableCount++
-         For each evaluation in sampling_config.evaluations:
-           result = evaluate(evaluation, step.runs)
-           stepEvals.results.push(result)
-           if result.passed: passedCount++ else: failedCount++
-
-       allEvaluations.push(stepEvals)
-
-  5. Update trigger with everything:
-     UPDATE triggers SET
-       status = 'completed',
-       completed_at = NOW(),
-       results = :body.steps,
-       evaluations = :allEvaluations,
-       steps_available = :availableCount,
-       steps_unavailable = :unavailableCount,
-       evaluations_passed = :passedCount,
-       evaluations_failed = :failedCount
-     WHERE id = :triggerId
-
-  6. If failedCount > 0: fire alert (see Alerting below)
-
-  7. Return { ok: true, triggerId, evaluationsRun: passedCount + failedCount, passed: passedCount, failed: failedCount }
-```
-
-### 3. `GET /api/observability/triggers` — New
-
-List triggers for a project.
-
-**Query params:** `samplingConfigId?`, `sessionId?`, `status?`, `from?`, `to?`, `limit=50`, `offset=0`
-
-**Response:**
-
-```json
-{
-  "triggers": [
-    {
-      "id": 42,
-      "samplingConfigId": 1,
-      "samplingConfigName": "Monitor searchDB latency",
-      "sessionId": "f47ac10b-...",
-      "status": "completed",
-      "runCount": 3,
-      "stepsCount": 2,
-      "stepsAvailable": 1,
-      "stepsUnavailable": 1,
-      "evaluationsPassed": 3,
-      "evaluationsFailed": 1,
-      "createdAt": "2026-04-11T10:00:00Z",
-      "completedAt": "2026-04-11T10:00:05Z"
-    }
-  ],
-  "total": 15
-}
-```
-
-Note: summary counts come from the denormalized columns on `triggers` — no need to parse JSONB for list queries.
-
-### 4. `GET /api/observability/triggers/:triggerId` — New
-
-Full detail. Returns the trigger row with `steps_sent`, `results`, and `evaluations` JSONB fields unpacked.
-
-### 5. Sampling Config CRUD — New
-
-| Endpoint | Method | Purpose |
-|----------|--------|---------|
-| `/api/observability/sampling-configs` | POST | Create config |
-| `/api/observability/sampling-configs` | GET | List configs for project |
-| `/api/observability/sampling-configs/:id` | GET | Get config detail |
-| `/api/observability/sampling-configs/:id` | PUT | Update config |
-| `/api/observability/sampling-configs/:id` | DELETE | Delete config |
-
-All scoped to project via API key auth.
-
----
-
-## Evaluation Engine
-
-Implement each evaluation type as a function:
-
-### `latency-budget`
-
-```
-Input: runs[], config: { maxDurationMs }
-Logic: passed = runs.every(r => r.durationMs <= maxDurationMs)
-Detail: { maxDurationMs, actualMaxMs: Math.max(...runs.map(r => r.durationMs)) }
-```
-
-### `token-budget`
-
-```
-Input: runs[], config: { maxTokens }
-Logic: passed = runs.every(r => (r.usageTotalTokens ?? 0) <= maxTokens)
-Detail: { maxTokens, actualMaxTokens: Math.max(...) }
-```
-
-### `output-contains`
-
-```
-Input: runs[], config: { containsText?, notContainsText? }
-Logic:
-  outputStr = JSON.stringify(run.output)
-  if containsText: passed &= outputStr.includes(containsText)
-  if notContainsText: passed &= !outputStr.includes(notContainsText)
-Detail: { containsText, notContainsText, failedRunIndices: [...] }
-```
-
-### `output-schema`
-
-```
-Input: runs[], config: { jsonSchema }
-Logic: validate each run.output against JSON schema (use ajv)
-  passed = runs.every(r => ajv.validate(jsonSchema, r.output))
-Detail: { failedRunIndices: [...], errors: [...] }
-```
-
-### `determinism`
-
-```
-Input: runs[] (needs runCount >= 2), config: { similarityThreshold }
-Logic:
-  Compare all pairs of run outputs
-  similarity = average pairwise similarity (cosine, jaccard, or string match ratio)
-  passed = similarity >= similarityThreshold
-Detail: { threshold, actualSimilarity, pairComparisons: [...] }
-```
-
-Simple approach: normalize outputs to strings, compare with Levenshtein distance / longest common subsequence ratio. For structured outputs, deep-equal check with tolerance.
-
-### `llm-judge`
-
-```
-Input: runs[], config: { judgePrompt, threshold, judgeModel, judgeProvider }
-Logic:
-  For each run:
-    prompt = judgePrompt + "\n\nOutput to evaluate:\n" + JSON.stringify(run.output)
-    response = callLLM(judgeModel, judgeProvider, prompt)
-    score = parseScore(response)  // extract number from "SCORE: 8.5"
-  avgScore = average(scores)
-  passed = avgScore >= threshold
-Detail: { threshold, avgScore, perRunScores: [...] }
-```
-
----
-
-## Background Jobs
-
-### Trigger Timeout Sweep
-
-Run every 5 minutes:
-
-```sql
-UPDATE triggers
-SET status = 'timed_out'
-WHERE status = 'sent'
-  AND sent_at < NOW() - INTERVAL '10 minutes';
-```
-
-Optionally alert on timeouts (SDK may have crashed or process exited before completing).
-
-### Stale Session Cleanup
-
-Run every hour:
-
-```sql
-UPDATE observability_sessions
-SET ended_at = last_heartbeat
-WHERE ended_at IS NULL
-  AND last_heartbeat < NOW() - INTERVAL '5 minutes';
-```
-
----
-
-## Alerting
-
-When evaluations fail, fire alerts:
-
-1. **Dashboard notification** — mark trigger as having failures, show badge in UI
-2. **Webhook** (future) — POST to `alerting.webhookUrl` with trigger ID + failed evaluations
-3. **Email** (future) — send to `alerting.emailTo` addresses
-
----
-
-## Dashboard Pages
-
-### Sampling Configs Page (`/observability/sampling`)
-
-- Table: name, enabled toggle, filter summary, run count, sample rate, evaluation count
-- Create/edit modal with form fields for all config options
-- Evaluation builder: add rows with type dropdown + type-specific fields
-
-### Trigger History Page (`/observability/triggers`)
-
-- Table: ID, config name, status badge, steps count, available/unavailable, pass/fail counts, timestamps
-- Filter by: config, status, date range
-- Click row → trigger detail page
-
-### Trigger Detail Page (`/observability/triggers/:id`)
-
-- Header: status, config name, run count, timestamps
-- Per-step cards:
-  - Tool/AI icon + name
-  - Available badge (green / red with reason)
-  - Run results table: runIndex, output preview, durationMs, tokens, error
-  - Evaluation results: type, passed badge, detail expandable
-- Collapsible JSON viewers for input/output
-
----
-
-## Implementation Order
-
-1. **Schema**: Create `sampling_configs` + `triggers` tables
-2. **Sampling configs CRUD**: endpoints + validation
-3. **Modify `POST /api/observability/events`**: add sampling config matching + trigger creation + response body
-4. **`POST /api/observability/triggers/:id/results`**: result storage + evaluation engine
-5. **Evaluation engine**: implement all 6 types
-6. **Query endpoints**: `GET /api/observability/triggers` list + detail
-7. **Timeout sweep**: cron job
-8. **Dashboard**: sampling configs page, trigger history, trigger detail
-9. **Alerting**: dashboard notifications, then webhook/email