@warlock.js/ai-panoptic 5.30.0 → 5.31.0

package/llms-full.txt

@@ -39,10 +39,10 @@ Absent `evaluate` ⇒ the drawer never renders the button and a `POST` to its ro
 
 ### `EvaluateConfig`
 
-| Field | Required | Notes |
-|---|---|---|
-| `model` | yes | A `ModelContract`, or a `() => ModelContract \| Promise<ModelContract>` factory — mirrors `panoptic.cache`'s literal-or-factory ergonomics. Resolved fresh on every evaluate click (a deliberate, infrequent, human-triggered action — not the collector's hot path — so no memoization, unlike the cache store's driver). |
-| `instructions` | no | Default grading rubric/criteria, verbatim — the same shape `ai.prompts().validate({ criteria })` accepts. Seeds the dashboard's editable textarea (baked into the served page at boot, so the operator sees and can extend the actual default, not a blank box); the built-in prompt-quality rubric is used when neither this nor a per-run override is supplied. |
+| Field          | Required | Notes                                                                                                                                                                                                                                                                                                                                                             |
+| -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `model`        | yes      | A `ModelContract`, or a `() => ModelContract \| Promise<ModelContract>` factory — mirrors `panoptic.cache`'s literal-or-factory ergonomics. Resolved fresh on every evaluate click (a deliberate, infrequent, human-triggered action — not the collector's hot path — so no memoization, unlike the cache store's driver).                                        |
+| `instructions` | no       | Default grading rubric/criteria, verbatim — the same shape `ai.prompts().validate({ criteria })` accepts. Seeds the dashboard's editable textarea (baked into the served page at boot, so the operator sees and can extend the actual default, not a blank box); the built-in prompt-quality rubric is used when neither this nor a per-run override is supplied. |
 
 ## The UI
 
@@ -53,10 +53,14 @@ Click it to open an inline panel: a textarea pre-filled with the configured `ins
 ## The building blocks (for embedding elsewhere)
 
 ```ts
-import { evaluateSystemPrompt, extractLastSystemPrompt, findSpanById } from "@warlock.js/ai-panoptic";
+import {
+  evaluateSystemPrompt,
+  extractLastSystemPrompt,
+  findSpanById,
+} from "@warlock.js/ai-panoptic";
 
-const span = findSpanById(trace.root, spanId);          // walk the span tree
-const prompt = extractLastSystemPrompt(span);            // undefined if none captured
+const span = findSpanById(trace.root, spanId); // walk the span tree
+const prompt = extractLastSystemPrompt(span); // undefined if none captured
 if (prompt) {
   const verdict = await evaluateSystemPrompt(prompt, evaluateConfig, "Per-run override, optional");
   // verdict: { score?: number; issues: string[] }
@@ -71,15 +75,15 @@ if (prompt) {
 
 `POST {basePath}api/traces/:traceId/spans/:spanId/evaluate`, body `{ instructions?: string }` (all optional — an empty body is valid). Gated by the same `authToken` / `allowedHosts` checks as every other route (S4) — off-loopback still requires a token, exactly like the rest of the dashboard. Body reads are capped at 64 KB (`413` past that) so a hostile/oversized payload can't hold the connection open.
 
-| Status | Meaning |
-|---|---|
-| `200` | `EvaluateVerdict` — `{ score?: number; issues: string[] }` |
-| `404` | trace or span not found |
-| `422` | the span carried no captured system prompt to grade |
-| `413` | request body over 64 KB |
-| `400` | request body isn't valid JSON |
-| `502` | `config.model` (a factory) threw resolving the judge model |
-| `405` | POST hit this path but `evaluate` isn't configured |
+| Status | Meaning                                                    |
+| ------ | ---------------------------------------------------------- |
+| `200`  | `EvaluateVerdict` — `{ score?: number; issues: string[] }` |
+| `404`  | trace or span not found                                    |
+| `422`  | the span carried no captured system prompt to grade        |
+| `413`  | request body over 64 KB                                    |
+| `400`  | request body isn't valid JSON                              |
+| `502`  | `config.model` (a factory) threw resolving the judge model |
+| `405`  | POST hit this path but `evaluate` isn't configured         |
 
 ```sh
 curl -X POST 'http://127.0.0.1:4319/api/traces/tr_1/spans/sp_1/evaluate' \
@@ -119,35 +123,35 @@ Four exporters ship today. Each is a **factory** returning an `ExporterContract`
 
 ## Pick an exporter
 
-| Exporter | Backend | Deps | Use it for |
-|---|---|---|---|
-| `consoleExporter()` | stdout/stderr | none | local dev, CI logs |
-| `fileExporter()` | JSON-Lines file | none | durable local sink, replay, out-of-band shipping |
-| `otelExporter()` | OpenTelemetry | `@opentelemetry/api` (optional peer) | OTel-native stacks, GenAI semantic conventions |
-| `langfuseExporter()` | Langfuse | `langfuse` (optional peer) | LLM-specific dashboards, generations |
+| Exporter             | Backend         | Deps                                 | Use it for                                       |
+| -------------------- | --------------- | ------------------------------------ | ------------------------------------------------ |
+| `consoleExporter()`  | stdout/stderr   | none                                 | local dev, CI logs                               |
+| `fileExporter()`     | JSON-Lines file | none                                 | durable local sink, replay, out-of-band shipping |
+| `otelExporter()`     | OpenTelemetry   | `@opentelemetry/api` (optional peer) | OTel-native stacks, GenAI semantic conventions   |
+| `langfuseExporter()` | Langfuse        | `langfuse` (optional peer)           | LLM-specific dashboards, generations             |
 
-> `@opentelemetry/api` and `langfuse` are **OPTIONAL peer dependencies**, lazily imported by their exporter. Importing `@warlock.js/ai-panoptic` never pulls them in; a missing SDK surfaces as a curated *"install this"* error on first export, never a boot-time stack trace.
+> `@opentelemetry/api` and `langfuse` are **OPTIONAL peer dependencies**, lazily imported by their exporter. Importing `@warlock.js/ai-panoptic` never pulls them in; a missing SDK surfaces as a curated _"install this"_ error on first export, never a boot-time stack trace.
 
 ## Console
 
 ```ts
 import { consoleExporter } from "@warlock.js/ai-panoptic";
 
-collector.use(consoleExporter());                 // one summary line per trace
-collector.use(consoleExporter({ tree: true }));   // full indented span tree
+collector.use(consoleExporter()); // one summary line per trace
+collector.use(consoleExporter({ tree: true })); // full indented span tree
 collector.use(consoleExporter({ streaming: true })); // also print each span as it finalizes
 ```
 
-Failed/cancelled spans route to `console.error`; everything else to `console.log`. Inject a sink (`{ console: mySink }`) to redirect or to capture in tests.
+Failed/cancelled/incomplete spans route to `console.error`; everything else to `console.log`. Inject a sink (`{ console: mySink }`) to redirect or to capture in tests.
 
 ## File
 
 ```ts
 import { fileExporter } from "@warlock.js/ai-panoptic";
 
-collector.use(fileExporter({ path: "storage/traces.jsonl" }));      // write every trace
-collector.use(fileExporter({ path: "...", flushEvery: 50 }));       // batch 50 traces per write
-collector.use(fileExporter({ path: "...", pretty: true }));         // multi-line records
+collector.use(fileExporter({ path: "storage/traces.jsonl" })); // write every trace
+collector.use(fileExporter({ path: "...", flushEvery: 50 })); // batch 50 traces per write
+collector.use(fileExporter({ path: "...", pretty: true })); // multi-line records
 ```
 
 Appends one JSON record per line (a `TraceRecord` = `{ type: "trace", exportedAt, trace }`). Buffers in memory and drains on `flush()` / `shutdown()` — call `collector.shutdown()` on teardown so the last batch lands. The parent directory is created on first write.
@@ -173,9 +177,9 @@ Each `TraceSpan` becomes one OTel span with the source start/end times and paren
 - `gen_ai.conversation.id` (from the span's `sessionId`)
 - `gen_ai.system` / `gen_ai.request.model` / `gen_ai.operation.name` — **never invented**: these appear only when the span's free-form `attributes` bag already carried that exact key. `otelExporter`'s `system` option backfills `gen_ai.system` when the span supplied none (and never overrides one it did).
 - `warlock.report.type` / `warlock.version` / `warlock.duration_ms` / `warlock.cost.usd`
-- the collector's own per-primitive keys ride along verbatim — `agent.model.name` / `agent.model.provider` / `agent.trips`, `workflow.steps`, `supervisor.iterations` / `supervisor.terminatedBy`, `tool.tripIndex`, `retries`, … — since `toGenAiAttributes` copies every scalar entry of the span's `attributes` bag.
+- the collector's own per-primitive keys ride along verbatim — `agent.model.name` / `agent.model.provider` / `agent.trips`, `workflow.steps`, `supervisor.iterations` / `supervisor.terminatedBy`, `tool.tripIndex`, `tool.callId`, `retries`, … — since `toGenAiAttributes` copies every scalar entry of the span's `attributes` bag.
 
-Failed/cancelled spans get OTel `ERROR` status plus a recorded exception.
+Failed/cancelled/incomplete spans get OTel `ERROR` status plus a recorded exception.
 
 ## Langfuse
 
@@ -198,11 +202,12 @@ Exporters surface `span.input` / `span.output` (console `in:` / `out:`, file JSO
 
 ```ts
 panoptic({
-  exporters: [langfuseExporter({ /* ... */ })],
-  captureContent: true,                 // copy prompts/responses + tool args/results onto spans
-  redactContent: (value, { field }) =>  // optional masker, per value
+  exporters: [langfuseExporter({/* ... */})],
+  captureContent: true, // copy prompts/responses + tool args/results onto spans
+  redactContent: (value, { field }) =>
+    // optional masker, per value
     field === "input" ? "[redacted]" : value,
-  fullHistory: true,                    // emit the WHOLE conversation as span.input (see below)
+  fullHistory: true, // emit the WHOLE conversation as span.input (see below)
 });
 ```
 
@@ -229,11 +234,18 @@ Reach for `fullHistory` when a backend (Langfuse) should show the full multi-tur
 Reusable when authoring a **custom exporter**:
 
 ```ts
-import { walkSpans, toGenAiAttributes, totalCostUsd, GEN_AI_ATTRIBUTES, WARLOCK_ATTRIBUTES } from "@warlock.js/ai-panoptic";
+import {
+  walkSpans,
+  toGenAiAttributes,
+  totalCostUsd,
+  GEN_AI_ATTRIBUTES,
+  WARLOCK_ATTRIBUTES,
+} from "@warlock.js/ai-panoptic";
 
-for (const span of walkSpans(trace.root)) {        // depth-first pre-order
-  const attributes = toGenAiAttributes(span);      // gen_ai.* / warlock.* attribute map
-  const cost = totalCostUsd(span.usage);           // single USD scalar, or undefined
+for (const span of walkSpans(trace.root)) {
+  // depth-first pre-order
+  const attributes = toGenAiAttributes(span); // gen_ai.* / warlock.* attribute map
+  const cost = totalCostUsd(span.usage); // single USD scalar, or undefined
 }
 ```
 
@@ -278,7 +290,7 @@ Reach for `reportToTrace` when you already hold a finished `result.report` and w
 ## See also
 
 - The `observe-with-panoptic` topic — the source end: wiring the `panoptic(...)` subscriber into a run (`attach` / `middleware` / `collect`). The `collector` these exporters register on comes from there.
-- The `query-traces` topic — the in-memory trace store: a sibling sink that *retains* traces (instead of shipping them out) so you can `query` / `aggregate` them after the fact.
+- The `query-traces` topic — the in-memory trace store: a sibling sink that _retains_ traces (instead of shipping them out) so you can `query` / `aggregate` them after the fact.
 - The `run-ai-agent` topic of the `warlock-js-ai` skill — running the agent and the `BaseReport` shape Panoptic projects into a `Trace`.
 
 
@@ -303,9 +315,9 @@ import "@warlock.js/ai-panoptic"; // side-effect wiring — do this once at boot
 
 ai.config({
   panoptic: {
-    exporters: [consoleExporter()],   // optional; a store-shaped exporter also feeds the dashboard
-    observeAll: true,                 // observe every flow by default
-    dashboard: true,                  // optional local viewer — see use-local-dashboard
+    exporters: [consoleExporter()], // optional; a store-shaped exporter also feeds the dashboard
+    observeAll: true, // observe every flow by default
+    dashboard: true, // optional local viewer — see use-local-dashboard
   },
 });
 ```
@@ -316,13 +328,13 @@ ai.config({
 
 Once a collector is registered, which flows actually route to it is controlled by core's `observe` seam:
 
-| Setting | Effect |
-|---|---|
-| `observeAll: true` (config) | Every `ai.agent` / `ai.workflow` / `ai.supervisor` / `ai.team` that left `observe` unset routes its completed report to the registered observers. |
-| `observe: true` (per-flow) | Force-observe **this** flow even when `observeAll` is off → the globally registered observers. |
-| `observe: false` (per-flow) | Opt **this** flow out entirely, even when `observeAll` is on. |
+| Setting                          | Effect                                                                                                                                                                                  |
+| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `observeAll: true` (config)      | Every `ai.agent` / `ai.workflow` / `ai.supervisor` / `ai.team` that left `observe` unset routes its completed report to the registered observers.                                       |
+| `observe: true` (per-flow)       | Force-observe **this** flow even when `observeAll` is off → the globally registered observers.                                                                                          |
+| `observe: false` (per-flow)      | Opt **this** flow out entirely, even when `observeAll` is on.                                                                                                                           |
 | `observe: <Observer>` (per-flow) | A **flow-local** collector — only this flow's report, routed only to it (the global observers are skipped). A panoptic flow-local collector implements `Observer`, so pass it directly. |
-| (unset, `observeAll: false`) | Not observed — opt-in default, identical to legacy behavior. |
+| (unset, `observeAll: false`)     | Not observed — opt-in default, identical to legacy behavior.                                                                                                                            |
 
 ```ts
 ai.agent({ model, observe: true });                       // always traced
@@ -350,11 +362,11 @@ const observe = panoptic({
 
 ## Three feed paths
 
-| Path | Use when | API |
-|---|---|---|
-| **Events** | You want every run captured automatically and the primitive emits a `*.completed` event (agent / workflow / supervisor). | `observe.attach(target)` |
-| **Middleware** | The app already composes cross-cutting concerns through the agent middleware pipeline. | `panoptic().middleware()` |
-| **Direct** | The primitive carries no result-bearing event (the orchestrator), or you captured a report out of band. | `observe.collect(report)` |
+| Path           | Use when                                                                                                                 | API                       |
+| -------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------- |
+| **Events**     | You want every run captured automatically and the primitive emits a `*.completed` event (agent / workflow / supervisor). | `observe.attach(target)`  |
+| **Middleware** | The app already composes cross-cutting concerns through the agent middleware pipeline.                                   | `panoptic().middleware()` |
+| **Direct**     | The primitive carries no result-bearing event (the orchestrator), or you captured a report out of band.                  | `observe.collect(report)` |
 
 ### 1. Attach to the event stream
 
@@ -382,7 +394,7 @@ panoptic({ exporters, completedEvents: ["agent.completed"] }); // agents only
 ### 2. Wire it as agent middleware
 
 ```ts
-const observe = panoptic({ exporters: [langfuseExporter({ /* ... */ })] });
+const observe = panoptic({ exporters: [langfuseExporter({/* ... */})] });
 
 const agent = ai.agent({
   model,
@@ -414,12 +426,12 @@ An observability fault never escapes onto the AI run's hot path:
 - The collector wraps every exporter invocation in `Promise.allSettled`, so a throwing exporter can't crash the run or block its siblings.
 - The event handler and the middleware additionally swallow the `collect` rejection, so even a collector-level fault stays contained.
 
-A broken backend degrades to *missing telemetry*, never a failed agent run.
+A broken backend degrades to _missing telemetry_, never a failed agent run.
 
 ## Lifecycle
 
 ```ts
-await observe.flush();    // drain buffered exporters (between batches / checkpoints)
+await observe.flush(); // drain buffered exporters (between batches / checkpoints)
 await observe.shutdown(); // flush + release every exporter (process teardown)
 ```
 
@@ -432,7 +444,7 @@ The in-memory trace store doubles as an `ExporterContract`, so the simplest way
 ```ts
 import { panoptic, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
 
-const store = createInMemoryTraceStore();        // an ExporterContract + queryable store
+const store = createInMemoryTraceStore(); // an ExporterContract + queryable store
 const observe = panoptic({ exporters: [store, consoleExporter()] });
 
 // store now receives every trace — query/aggregate it later:
@@ -470,7 +482,7 @@ description: 'Retain @warlock.js/ai-panoptic traces in a queryable store — in-
 
 # Query traces — the in-memory queryable store
 
-The collector fans every completed `Trace` out to its exporters. A **trace store** is one such sink that *retains* traces so you can answer observability questions after the fact: pull one run by id, list every run for a session, slice failed runs in a time window, or roll usage + cost across any of those slices.
+The collector fans every completed `Trace` out to its exporters. A **trace store** is one such sink that _retains_ traces so you can answer observability questions after the fact: pull one run by id, list every run for a session, slice failed runs in a time window, or roll usage + cost across any of those slices.
 
 `createInMemoryTraceStore()` returns an object that satisfies **both** `TraceStoreContract` and `ExporterContract` — so wiring it into a collector is a single `collector.use(store)` and the store fills as traces complete.
 
@@ -500,12 +512,12 @@ const spend = store.aggregate({ sessionId: "session-42" });
 
 ```ts
 interface TraceStoreContract {
-  add(trace: Trace): void;                 // ingest; overwrites same traceId
+  add(trace: Trace): void; // ingest; overwrites same traceId
   get(traceId: string): Trace | undefined; // one run by id
-  query(filter?: TraceQuery): Trace[];     // matching traces, newest-started first
+  query(filter?: TraceQuery): Trace[]; // matching traces, newest-started first
   aggregate(filter?: TraceQuery): TraceAggregate; // usage + cost + status rollup
-  readonly size: number;                   // retained count
-  clear(): void;                           // drop everything
+  readonly size: number; // retained count
+  clear(): void; // drop everything
 }
 ```
 
@@ -519,11 +531,11 @@ Every field is optional and **ANDed** together — an absent field is "don't car
 
 ```ts
 type TraceQuery = {
-  traceId?: string;                         // exact one run
-  sessionId?: string;                       // every run for a session
-  status?: ReportStatus | ReportStatus[];   // single status or membership set
-  startedAfter?: string | Date;             // inclusive lower bound on root startedAt
-  startedBefore?: string | Date;            // inclusive upper bound on root startedAt
+  traceId?: string; // exact one run
+  sessionId?: string; // every run for a session
+  status?: ReportStatus | ReportStatus[]; // single status or membership set
+  startedAfter?: string | Date; // inclusive lower bound on root startedAt
+  startedBefore?: string | Date; // inclusive upper bound on root startedAt
 };
 ```
 
@@ -552,12 +564,12 @@ Time bounds compare against each trace's root `startedAt`, **inclusive on both e
 
 ```ts
 type TraceAggregate = {
-  traces: number;        // matched count
-  completed: number;     // root status === "completed"
-  failed: number;        // root status === "failed"
-  cancelled: number;     // root status === "cancelled"
-  usage: Usage;          // summed token usage (input/output/total always present)
-  cost?: ModelPricing;   // merged per-channel USD, or undefined if no trace was priced
+  traces: number; // matched count
+  completed: number; // root status === "completed"
+  failed: number; // root status === "failed" or "incomplete"
+  cancelled: number; // root status === "cancelled"
+  usage: Usage; // summed token usage (input/output/total always present)
+  cost?: ModelPricing; // merged per-channel USD, or undefined if no trace was priced
   totalDuration: number; // summed wall-clock ms
 };
 ```
@@ -567,17 +579,14 @@ const stats = store.aggregate({ sessionId: "session-42" });
 
 console.log(`${stats.traces} runs, ${stats.usage.total} tokens, ${stats.failed} failed`);
 
-const totalUsd =
-  (stats.cost?.input ?? 0) +
-  (stats.cost?.output ?? 0) +
-  (stats.cost?.cachedInput ?? 0) +
-  (stats.cost?.cachedOutput ?? 0);
+// totalCost from @warlock.js/ai sums every channel (cache reads + writes included).
+const totalUsd = totalCost(stats.cost) ?? 0;
 ```
 
 Notes that matter when reading the numbers:
 
-- **`usage`** sums each trace's *root* `Usage` (already a rollup of own cost + children), so the totals reflect the whole run tree without re-walking spans. `input` / `output` / `total` are always present (zero when nothing matched); the optional `cachedTokens` / `cacheWriteTokens` / `reasoningTokens` channels are summed only when at least one matched trace reported them.
-- **`cost`** is a per-channel `ModelPricing` breakdown (not one opaque number), merged with the framework's own cost-rollup logic — an unpriced trace never erases the cost of priced ones. It stays `undefined` only when **no** matched trace carried pricing.
+- **`usage`** sums each trace's _root_ `Usage` (already a rollup of own cost + children), so the totals reflect the whole run tree without re-walking spans. `input` / `output` / `total` are always present (zero when nothing matched); the optional `cachedTokens` / `cacheWriteTokens` / `reasoningTokens` channels are summed only when at least one matched trace reported them.
+- **`cost`** is a per-channel `ModelPricing` breakdown (not one opaque number; `cacheWrite` holds cache-write spend), merged with the framework's own cost-rollup logic — an unpriced trace never erases the cost of priced ones. It stays `undefined` only when **no** matched trace carried pricing.
 - **`completed` + `failed` + `cancelled` need not equal `traces`** — non-terminal statuses (`awaiting-input`, `max-iterations`) count toward `traces` but none of the three headline counters.
 - An empty match returns a zeroed aggregate (`usage` all `0`, `cost` `undefined`, `totalDuration` `0`).
 
@@ -612,13 +621,13 @@ const store = createCacheTraceStore(async () => {
   return driver;
 });
 
-await store.ready();   // hydrate the in-memory mirror from a prior run
-collector.use(store);  // fills as traces complete; reads stay synchronous
+await store.ready(); // hydrate the in-memory mirror from a prior run
+collector.use(store); // fills as traces complete; reads stay synchronous
 
 const failed = store.query({ status: "failed" }); // queryable like any store
 ```
 
-**Write-through over a sync contract.** The store keeps the same insertion-ordered in-memory **mirror** the in-memory store uses and serves every read (`get` / `query` / `aggregate` / `size`) from it synchronously — the dashboard polls these and can't await a cache. Writes go **through**: `add` updates the mirror immediately, then *fire-and-forget* persists the trace + a newest-first index to the cache. So:
+**Write-through over a sync contract.** The store keeps the same insertion-ordered in-memory **mirror** the in-memory store uses and serves every read (`get` / `query` / `aggregate` / `size`) from it synchronously — the dashboard polls these and can't await a cache. Writes go **through**: `add` updates the mirror immediately, then _fire-and-forget_ persists the trace + a newest-first index to the cache. So:
 
 - **Reads never wait on the cache** — the mirror is the runtime source of truth; the cache is the durable backing store.
 - **A flaky cache never throws into the hot path** — a rejected write is routed to the optional `onError` hook and swallowed; the trace stays visible in the mirror for the life of the process, it just won't survive a restart.
@@ -626,20 +635,21 @@ const failed = store.query({ status: "failed" }); // queryable like any store
 
 ```ts
 type CacheDriverInput =
-  | CacheDriver           // a live driver
+  | CacheDriver // a live driver
   | (() => CacheDriver | Promise<CacheDriver>); // lazy (sync or async) factory
 
 type CacheTraceStoreOptions = {
-  prefix?: string;   // key namespace; per-trace keys `${prefix}:trace:${id}`,
-                     // index at `${prefix}:index`. Default "panoptic".
+  prefix?: string; // key namespace; per-trace keys `${prefix}:trace:${id}`,
+  // index at `${prefix}:index`. Default "panoptic".
   capacity?: number; // FIFO cap (oldest-ingested evicted from cache + mirror);
-                     // absent / 0 = unbounded.
+  // absent / 0 = unbounded.
   onError?: (error: unknown) => void; // swallowed cache-write failures land here.
 };
 
-type CacheTraceStoreHandle = TraceStoreContract & ExporterContract & {
-  ready(): Promise<void>; // hydrate the mirror from the cache on startup
-};
+type CacheTraceStoreHandle = TraceStoreContract &
+  ExporterContract & {
+    ready(): Promise<void>; // hydrate the mirror from the cache on startup
+  };
 ```
 
 Ordering uses an **internal monotonic counter**, not `Date.now()`, so the persisted index stays a stable total order even when many traces land in the same millisecond — FIFO eviction stays honest across a restart.
@@ -689,8 +699,8 @@ import "@warlock.js/ai-panoptic"; // side-effect: subscribes panoptic to the con
 
 ai.config({
   panoptic: {
-    dashboard: true,    // serve on http://127.0.0.1:4319/
-    observeAll: true,   // route every workflow/supervisor/team run to the collector
+    dashboard: true, // serve on http://127.0.0.1:4319/
+    observeAll: true, // route every workflow/supervisor/team run to the collector
   },
 });
 ```
@@ -721,7 +731,7 @@ So `dashboard: true` with no exporters Just Works — you get a store for free.
 The page is one self-contained, dependency-free vanilla-JS file (no bundler, no framework, works offline). It polls the JSON API every 2s and renders:
 
 - **Header** — the Warlock logo + your `title`, the `TraceAggregate` headline counts (traces / completed / failed / cancelled), tokens as **`↓input · ↑output · total`**, total cost, an **errors-only** toggle, a **search** box, and a **light / dark / system** theme switch (persisted to `localStorage`, so the choice survives a reload; `system` follows the OS `prefers-color-scheme`).
-- **Trace list** — newest-first clickable master rows, each led by a **colour-coded Title-case type label** (`Supervisor` / `Team` / `Workflow` / `Agent` / `Tool` / …; `ai.team` reports its own `team` type rather than `supervisor`, so it groups, filters, and labels separately). An optional **group-by-session** view collapses runs under collapsible session headers; a **group-by-prompt** view does the same over the `name@version` prompt key (see *Prompt-version linkage* below); a **group-by-type** view buckets the list by root type under collapsible `Type (N)` headers (`Agent (12)`, `Team (3)`, …). The three groupings are mutually exclusive.
+- **Trace list** — newest-first clickable master rows, each led by a **colour-coded Title-case type label** (`Supervisor` / `Team` / `Workflow` / `Agent` / `Tool` / …; `ai.team` reports its own `team` type rather than `supervisor`, so it groups, filters, and labels separately). An optional **group-by-session** view collapses runs under collapsible session headers; a **group-by-prompt** view does the same over the `name@version` prompt key (see _Prompt-version linkage_ below); a **group-by-type** view buckets the list by root type under collapsible `Type (N)` headers (`Agent (12)`, `Team (3)`, …). The three groupings are mutually exclusive.
 - **Two-pane drawer** (click a row) — a collapsible **nested call tree** on the left and the **selected node's detail** on the right: rich structured input/output (chat bubbles / key-value / Markdown), a per-node **token breakdown** (`↓in · ↑out · total`), the node's **rollup cost** (its own cost, else the sum of its children — so wrapper nodes that carry tokens but no cost don't double-count), and a **metadata panel** with humanized keys (`Supervisor Terminated By`, `Span ID` — not raw `supervisor.terminatedBy`), and **started** / **ended** wall-clock timestamps (`28 Jun 2026 03:16 PM` — named month, 12-hour, minute precision) alongside the elapsed duration. Zero-valued token/cost rows are hidden so the panel stays readable.
 - **Cost heatmap** — every tree node and trace row carries a left accent tinted by its rollup cost relative to the trace's most expensive node, with a small legend, so the spend hot-spots jump out at a glance.
 - **Timeline view** — the drawer's left pane toggles between the nested call **Tree** and a Gantt **Timeline** (each span drawn at its offset from the root start, width = duration, critical path highlighted).
@@ -743,8 +753,14 @@ The pure matching/grouping rules are unit-tested in `trace-filter.ts` and mirror
 
 ```ts
 import {
-  filterTraces, matchesFilter, groupBySession, rollupCost, maxNodeCost, heatIntensity,
-  type TraceFilter, type SessionGroup,
+  filterTraces,
+  matchesFilter,
+  groupBySession,
+  rollupCost,
+  maxNodeCost,
+  heatIntensity,
+  type TraceFilter,
+  type SessionGroup,
 } from "@warlock.js/ai-panoptic"; // dashboard helpers, re-exported
 
 const failed = filterTraces(traces, { errorsOnly: true });
@@ -759,7 +775,7 @@ The open trace + span are reflected in the URL hash (`#trace=<id>&span=<id>`) an
 
 ### Prompt-version linkage
 
-When an agent ran against a *named* `ai.prompts` builder, the collector stamps `agent.promptName` / `agent.promptVersion` onto the root span. The dashboard derives a `name@version` key from them (`tracePromptKey(trace)`) and offers **filter-by-prompt** and **group-by-prompt** beside session — so a reviewer sees every run of `support@2` together and compares its cost / failure rate against `support@3`. Runs with no named prompt bucket under `"(no prompt)"`.
+When an agent ran against a _named_ `ai.prompts` builder, the collector stamps `agent.promptName` / `agent.promptVersion` onto the root span. The dashboard derives a `name@version` key from them (`tracePromptKey(trace)`) and offers **filter-by-prompt** and **group-by-prompt** beside session — so a reviewer sees every run of `support@2` together and compares its cost / failure rate against `support@3`. Runs with no named prompt bucket under `"(no prompt)"`.
 
 ```ts
 import { tracePromptKey, groupByPrompt, type PromptGroup } from "@warlock.js/ai-panoptic";
@@ -803,7 +819,7 @@ import { dashboard, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
 const store = createInMemoryTraceStore();
 const handle = await dashboard(store, { port: 4319, open: true });
 
-console.log(handle.url);  // http://127.0.0.1:4319/
+console.log(handle.url); // http://127.0.0.1:4319/
 console.log(handle.port); // 4319 (the REAL OS port when port: 0 was passed)
 
 // ...on teardown:
@@ -816,23 +832,23 @@ The store is the same object the collector writes to — wire the collector to i
 
 Every field optional, every default safe:
 
-| Option | Default | Notes |
-|---|---|---|
-| `port` | `4319` | Pass `0` for an ephemeral port — the resolved port comes back on `handle.port`. A busy port rejects with a clear `Error` (`port N in use; pass { port: 0 }`), not raw `EADDRINUSE`. |
-| `host` | `"127.0.0.1"` | **Loopback only** by default, so prompt content is never exposed to the LAN. Binding a non-loopback interface is gated — it requires `authToken` (see below) or `dashboard()` rejects. |
-| `basePath` | `"/"` | Mount-path prefix for both the page and every `/api/...` route, e.g. `"/panoptic"`. Normalized to leading + trailing slash. |
-| `open` | `false` | Open the default browser at the resolved URL once listening. Best-effort and fully swallowed — failing to open a browser never rejects the start. |
-| `title` | `"Panoptic"` | Header title baked into the served page. |
-| `authToken` | _unset_ | Bearer token required on every request. Compared with `crypto.timingSafeEqual` (constant-time, length-checked first) against both the header and query-string forms — never a plain `===` — so a network-reachable dashboard doesn't leak the token to a byte-at-a-time timing attack. `Authorization: Bearer <token>` is accepted on every route; `?token=` is accepted **only on the HTML page route** (the JSON API routes are header-only), since the page load is the one request that structurally can't carry a header — this narrows the token's footprint in access/proxy logs and browser history. Missing/wrong → `401`. **Required** when `host` is non-loopback. |
-| `allowedHosts` | loopback names | `Host`-header allowlist (DNS-rebind guard); an unlisted `Host` → `403`. Defaults to `localhost` / `127.0.0.1` / `[::1]` for a loopback bind. |
+| Option         | Default        | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| -------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `port`         | `4319`         | Pass `0` for an ephemeral port — the resolved port comes back on `handle.port`. A busy port rejects with a clear `Error` (`port N in use; pass { port: 0 }`), not raw `EADDRINUSE`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| `host`         | `"127.0.0.1"`  | **Loopback only** by default, so prompt content is never exposed to the LAN. Binding a non-loopback interface is gated — it requires `authToken` (see below) or `dashboard()` rejects.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| `basePath`     | `"/"`          | Mount-path prefix for both the page and every `/api/...` route, e.g. `"/panoptic"`. Normalized to leading + trailing slash.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+| `open`         | `false`        | Open the default browser at the resolved URL once listening. Best-effort and fully swallowed — failing to open a browser never rejects the start.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| `title`        | `"Panoptic"`   | Header title baked into the served page.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| `authToken`    | _unset_        | Bearer token required on every request. Compared with `crypto.timingSafeEqual` (constant-time, length-checked first) against both the header and query-string forms — never a plain `===` — so a network-reachable dashboard doesn't leak the token to a byte-at-a-time timing attack. `Authorization: Bearer <token>` is accepted on every route; `?token=` is accepted **only on the HTML page route** (the JSON API routes are header-only), since the page load is the one request that structurally can't carry a header — this narrows the token's footprint in access/proxy logs and browser history. Missing/wrong → `401`. **Required** when `host` is non-loopback. |
+| `allowedHosts` | loopback names | `Host`-header allowlist (DNS-rebind guard); an unlisted `Host` → `403`. Defaults to `localhost` / `127.0.0.1` / `[::1]` for a loopback bind.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 
 ### `DashboardHandle`
 
 ```ts
 type DashboardHandle = {
-  readonly url: string;            // http://host:port{basePath}
-  readonly port: number;           // real OS port (when port: 0 was requested)
-  close(): Promise<void>;          // stop the server, release the port
+  readonly url: string; // http://host:port{basePath}
+  readonly port: number; // real OS port (when port: 0 was requested)
+  close(): Promise<void>; // stop the server, release the port
 };
 ```
 
@@ -840,13 +856,13 @@ type DashboardHandle = {
 
 Every route is **`GET`-only and read-only** (non-`GET` → `405`), mounted under `basePath` — EXCEPT the one opt-in `POST .../spans/:spanId/evaluate` route `DashboardOptions.evaluate` enables (see the `evaluate-system-prompt` topic); absent that config, it 405s like everything else. The page polls the `GET` routes; you can curl them too. Store shapes are already JSON-safe, so responses are a plain `JSON.stringify`.
 
-| Route | Returns |
-|---|---|
-| `GET /api/traces` | `store.query(...)` — the trace list, filtered by the query string (`id` / `sessionId` / `status` / time window, parsed by `parseQuery`). |
-| `GET /api/traces/:id` | `store.get(id)` — one full trace, or `404 { error: "trace_not_found" }`. |
-| `GET /api/aggregate` | `store.aggregate(...)` — the usage + cost + status rollup over the same query filter. |
-| `GET /` (basePath) | the self-contained HTML page. |
-| `POST /api/traces/:traceId/spans/:spanId/evaluate` | grades the span's last captured system prompt — only when `evaluate` is configured. |
+| Route                                              | Returns                                                                                                                                  |
+| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
+| `GET /api/traces`                                  | `store.query(...)` — the trace list, filtered by the query string (`id` / `sessionId` / `status` / time window, parsed by `parseQuery`). |
+| `GET /api/traces/:id`                              | `store.get(id)` — one full trace, or `404 { error: "trace_not_found" }`.                                                                 |
+| `GET /api/aggregate`                               | `store.aggregate(...)` — the usage + cost + status rollup over the same query filter.                                                    |
+| `GET /` (basePath)                                 | the self-contained HTML page.                                                                                                            |
+| `POST /api/traces/:traceId/spans/:spanId/evaluate` | grades the span's last captured system prompt — only when `evaluate` is configured.                                                      |
 
 Query-string filters map 1:1 onto `TraceQuery`:
 
@@ -880,6 +896,7 @@ createServer(handler).listen(4319, "127.0.0.1");
 The dashboard surfaces whatever the store holds — including captured prompt/response content when content capture is on (`captureContent`). It binds **loopback-only** by default precisely so that content never leaves the machine. Binding a non-loopback host is gated — it requires an `authToken` (and checks a `Host` allowlist, sending `nosniff` / CSP / `X-Frame-Options: DENY` on every response) — but still prefer loopback for dev, and for production ship to a real backend via an exporter rather than exposing the dashboard. The dashboard has exactly one write-capable route (`evaluate`, off by default) — everything else stays read-only regardless of configuration.
 
 **Trace content is untrusted input to the dashboard's own UI, and is treated that way:**
+
 - **Markdown links in captured content are sanitized.** `span.input`/`span.output` can contain model output or attacker-controlled tool results — i.e. prompt-injected content — and the drawer renders it as Markdown. Link URLs go through `sanitizeHref` (a scheme allowlist: `http:`, `https:`, `mailto:`, plus relative/anchor URLs) after normalizing the way a browser will — attribute-entity decode, strip of ignored control chars/whitespace, lowercase — so `javascript:`/`data:`/`vbscript:` payloads (including case/whitespace/entity-obfuscated variants, and scheme-relative `//host` links) never become a live `href`; a rejected link renders its label as plain text instead. This closes a stored-XSS path that could otherwise reach the in-page bearer token.
 - **The bearer token is no longer a page-global.** The client script's `TOKEN` is sealed inside its own inner IIFE that exposes only the `fetchAuthed` helper, so other code in the page's closure can't read the raw token by name.
 

package/package.json

@@ -19,9 +19,9 @@
   "peerDependencies": {
     "@opentelemetry/api": "*",
     "@opentelemetry/sdk-trace-base": "*",
-    "@warlock.js/ai": "5.30.0",
-    "@warlock.js/cache": "5.30.0",
-    "@warlock.js/logger": "5.30.0",
+    "@warlock.js/ai": "5.31.0",
+    "@warlock.js/cache": "5.31.0",
+    "@warlock.js/logger": "5.31.0",
     "langfuse": "*"
   },
   "peerDependenciesMeta": {
@@ -41,7 +41,7 @@
   "dependencies": {
     "langfuse": "^3.38.20"
   },
-  "version": "5.30.0",
+  "version": "5.31.0",
   "main": "./cjs/index.cjs",
   "module": "./esm/index.mjs",
   "types": "./esm/index.d.mts",

package/skills/evaluate-system-prompt/SKILL.md

@@ -31,10 +31,10 @@ Absent `evaluate` ⇒ the drawer never renders the button and a `POST` to its ro
 
 ### `EvaluateConfig`
 
-| Field | Required | Notes |
-|---|---|---|
-| `model` | yes | A `ModelContract`, or a `() => ModelContract \| Promise<ModelContract>` factory — mirrors `panoptic.cache`'s literal-or-factory ergonomics. Resolved fresh on every evaluate click (a deliberate, infrequent, human-triggered action — not the collector's hot path — so no memoization, unlike the cache store's driver). |
-| `instructions` | no | Default grading rubric/criteria, verbatim — the same shape `ai.prompts().validate({ criteria })` accepts. Seeds the dashboard's editable textarea (baked into the served page at boot, so the operator sees and can extend the actual default, not a blank box); the built-in prompt-quality rubric is used when neither this nor a per-run override is supplied. |
+| Field          | Required | Notes                                                                                                                                                                                                                                                                                                                                                             |
+| -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `model`        | yes      | A `ModelContract`, or a `() => ModelContract \| Promise<ModelContract>` factory — mirrors `panoptic.cache`'s literal-or-factory ergonomics. Resolved fresh on every evaluate click (a deliberate, infrequent, human-triggered action — not the collector's hot path — so no memoization, unlike the cache store's driver).                                        |
+| `instructions` | no       | Default grading rubric/criteria, verbatim — the same shape `ai.prompts().validate({ criteria })` accepts. Seeds the dashboard's editable textarea (baked into the served page at boot, so the operator sees and can extend the actual default, not a blank box); the built-in prompt-quality rubric is used when neither this nor a per-run override is supplied. |
 
 ## The UI
 
@@ -45,10 +45,14 @@ Click it to open an inline panel: a textarea pre-filled with the configured `ins
 ## The building blocks (for embedding elsewhere)
 
 ```ts
-import { evaluateSystemPrompt, extractLastSystemPrompt, findSpanById } from "@warlock.js/ai-panoptic";
-
-const span = findSpanById(trace.root, spanId);          // walk the span tree
-const prompt = extractLastSystemPrompt(span);            // undefined if none captured
+import {
+  evaluateSystemPrompt,
+  extractLastSystemPrompt,
+  findSpanById,
+} from "@warlock.js/ai-panoptic";
+
+const span = findSpanById(trace.root, spanId); // walk the span tree
+const prompt = extractLastSystemPrompt(span); // undefined if none captured
 if (prompt) {
   const verdict = await evaluateSystemPrompt(prompt, evaluateConfig, "Per-run override, optional");
   // verdict: { score?: number; issues: string[] }
@@ -63,15 +67,15 @@ if (prompt) {
 
 `POST {basePath}api/traces/:traceId/spans/:spanId/evaluate`, body `{ instructions?: string }` (all optional — an empty body is valid). Gated by the same `authToken` / `allowedHosts` checks as every other route (S4) — off-loopback still requires a token, exactly like the rest of the dashboard. Body reads are capped at 64 KB (`413` past that) so a hostile/oversized payload can't hold the connection open.
 
-| Status | Meaning |
-|---|---|
-| `200` | `EvaluateVerdict` — `{ score?: number; issues: string[] }` |
-| `404` | trace or span not found |
-| `422` | the span carried no captured system prompt to grade |
-| `413` | request body over 64 KB |
-| `400` | request body isn't valid JSON |
-| `502` | `config.model` (a factory) threw resolving the judge model |
-| `405` | POST hit this path but `evaluate` isn't configured |
+| Status | Meaning                                                    |
+| ------ | ---------------------------------------------------------- |
+| `200`  | `EvaluateVerdict` — `{ score?: number; issues: string[] }` |
+| `404`  | trace or span not found                                    |
+| `422`  | the span carried no captured system prompt to grade        |
+| `413`  | request body over 64 KB                                    |
+| `400`  | request body isn't valid JSON                              |
+| `502`  | `config.model` (a factory) threw resolving the judge model |
+| `405`  | POST hit this path but `evaluate` isn't configured         |
 
 ```sh
 curl -X POST 'http://127.0.0.1:4319/api/traces/tr_1/spans/sp_1/evaluate' \