@warlock.js/ai-panoptic 5.24.0 → 5.25.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/llms-full.txt +869 -869
- package/package.json +4 -4
- package/skills/evaluate-system-prompt/SKILL.md +86 -86
- package/skills/export-traces/SKILL.md +184 -184
- package/skills/observe-with-panoptic/SKILL.md +175 -175
- package/skills/query-traces/SKILL.md +203 -203
- package/skills/use-local-dashboard/SKILL.md +221 -221
package/llms-full.txt
CHANGED
|
@@ -6,890 +6,890 @@
|
|
|
6
6
|
|
|
7
7
|
## evaluate-system-prompt `@warlock.js/ai-panoptic/evaluate-system-prompt/SKILL.md`
|
|
8
8
|
|
|
9
|
-
---
|
|
10
|
-
name: evaluate-system-prompt
|
|
11
|
-
description: 'Grade a single trace''s LAST captured system prompt from the dashboard drawer — the dashboard''s only write-capable feature, config-gated via `DashboardOptions.evaluate` (`ai.config({ panoptic: { dashboard: { evaluate: { model, instructions } } } })`). Triggers: `evaluate`, `EvaluateConfig`, `EvaluateVerdict`, `evaluateSystemPrompt`, `extractLastSystemPrompt`, `findSpanById`, "Evaluate system prompt" button, `POST .../spans/:spanId/evaluate`, grading a prompt from a real trace, judging a captured system prompt, per-run instructions override, judge model factory; ''evaluate this trace''s prompt'', ''grade the system prompt from the dashboard'', ''judge a captured run'', ''score a live system prompt''. Skip: grading raw prompt TEXT you already have in code (no trace involved) — `@warlock.js/ai`''s `ai.prompts().validate({ criteria })`, the same judge machinery this wraps; a DATASET of test cases run through an agent — `@warlock.js/ai`''s `ai.dataset` + `agent.eval`; viewing/filtering/grouping traces without grading them — `@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`; capturing the content this feature reads — `@warlock.js/ai-panoptic/export-traces/SKILL.md` (`captureContent`).'
|
|
12
|
-
---
|
|
13
|
-
|
|
14
|
-
# Evaluate a trace's system prompt from the dashboard
|
|
15
|
-
|
|
16
|
-
A drawer button that grades the LAST `{role: "system"}` message captured on the selected span — not the whole conversation, not the tool code, just the system prompt that was actually in effect for that run. It's the dashboard's only write-capable feature: every other route is `GET`-only and read-only by design (see `use-local-dashboard/SKILL.md`'s security note); this one POSTs, and it's off unless you explicitly opt in.
|
|
17
|
-
|
|
18
|
-
## Enable it
|
|
19
|
-
|
|
20
|
-
```ts
|
|
21
|
-
import { ai } from "@warlock.js/ai";
|
|
22
|
-
import "@warlock.js/ai-panoptic";
|
|
23
|
-
|
|
24
|
-
ai.config({
|
|
25
|
-
panoptic: {
|
|
26
|
-
dashboard: {
|
|
27
|
-
evaluate: {
|
|
28
|
-
model: openai.model("gpt-4o-mini"),
|
|
29
|
-
instructions: "Must address the user by name. Never invent prices.",
|
|
30
|
-
},
|
|
31
|
-
},
|
|
32
|
-
observeAll: true,
|
|
33
|
-
captureContent: true, // required — evaluate reads the captured span.input
|
|
34
|
-
},
|
|
35
|
-
});
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
Absent `evaluate` ⇒ the drawer never renders the button and a `POST` to its route 405s like any other route would (the path is never even pattern-matched) — the dashboard stays fully read-only unless you opt in.
|
|
39
|
-
|
|
40
|
-
### `EvaluateConfig`
|
|
41
|
-
|
|
42
|
-
| Field | Required | Notes |
|
|
43
|
-
|---|---|---|
|
|
44
|
-
| `model` | yes | A `ModelContract`, or a `() => ModelContract \| Promise<ModelContract>` factory — mirrors `panoptic.cache`'s literal-or-factory ergonomics. Resolved fresh on every evaluate click (a deliberate, infrequent, human-triggered action — not the collector's hot path — so no memoization, unlike the cache store's driver). |
|
|
45
|
-
| `instructions` | no | Default grading rubric/criteria, verbatim — the same shape `ai.prompts().validate({ criteria })` accepts. Seeds the dashboard's editable textarea (baked into the served page at boot, so the operator sees and can extend the actual default, not a blank box); the built-in prompt-quality rubric is used when neither this nor a per-run override is supplied. |
|
|
46
|
-
|
|
47
|
-
## The UI
|
|
48
|
-
|
|
49
|
-
Open a trace, select the span whose prompt you want graded (any span carrying a captured `[system, user]` pair or `fullHistory` message array — an agent span, not a tool span). An "Evaluate system prompt" button appears under the input/output block **only when that span actually has a system prompt to grade** (`extractLastSystemPrompt` finds one) — no dead button on tool spans or system-prompt-less runs.
|
|
50
|
-
|
|
51
|
-
Click it to open an inline panel: a textarea pre-filled with the configured `instructions` (fully editable — type over it, append to it, or clear it to fall back to the built-in rubric) and a "Run" button. The result — `score` (0–1, shown as a percentage) and `issues` (the judge's reasoning, as a list) — renders inline once it lands. State (open/closed, typed text, last result) is kept per-span in client memory, so switching to a sibling span and back doesn't lose it; it does NOT persist to the trace store or survive a page reload.
|
|
52
|
-
|
|
53
|
-
## The building blocks (for embedding elsewhere)
|
|
54
|
-
|
|
55
|
-
```ts
|
|
56
|
-
import { evaluateSystemPrompt, extractLastSystemPrompt, findSpanById } from "@warlock.js/ai-panoptic";
|
|
57
|
-
|
|
58
|
-
const span = findSpanById(trace.root, spanId); // walk the span tree
|
|
59
|
-
const prompt = extractLastSystemPrompt(span); // undefined if none captured
|
|
60
|
-
if (prompt) {
|
|
61
|
-
const verdict = await evaluateSystemPrompt(prompt, evaluateConfig, "Per-run override, optional");
|
|
62
|
-
// verdict: { score?: number; issues: string[] }
|
|
63
|
-
}
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
- **`extractLastSystemPrompt(span)`** — reads `span.input` (only present under `captureContent`). Handles both shapes `report-to-span.ts` produces: the `[system, user]` first-trip pair, and — under `fullHistory` — the full `CapturedMessage[]` conversation, where it takes the LAST system-role turn (a long-running agent can carry more than one; the most recent one is the one actually in effect). Returns `undefined` for a tool span, an agent with no system prompt, or when content capture is off.
|
|
67
|
-
- **`findSpanById(root, spanId)`** — depth-first lookup inside a trace's span tree; the route uses it to resolve which node the drawer had selected.
|
|
68
|
-
- **`evaluateSystemPrompt(text, config, instructionsOverride?)`** — the grading call. `instructionsOverride` (trimmed; blank-after-trim is treated as absent, never as "grade against nothing") wins over `config.instructions`. Reuses `@warlock.js/ai`'s `judgePromptBody` verbatim — the SAME LLM-as-judge machinery `ai.prompts().validate({ criteria })` runs — so there's exactly one judging implementation in the whole framework, not two. `judgePromptBody` itself never throws (a broken judge degrades to an issues-only outcome with no `score`); the only step that CAN throw is resolving `config.model` when it's a factory (e.g. constructing an SDK client) — the dashboard route catches that and returns `502`.
|
|
69
|
-
|
|
70
|
-
## The route
|
|
71
|
-
|
|
72
|
-
`POST {basePath}api/traces/:traceId/spans/:spanId/evaluate`, body `{ instructions?: string }` (all optional — an empty body is valid). Gated by the same `authToken` / `allowedHosts` checks as every other route (S4) — off-loopback still requires a token, exactly like the rest of the dashboard. Body reads are capped at 64 KB (`413` past that) so a hostile/oversized payload can't hold the connection open.
|
|
73
|
-
|
|
74
|
-
| Status | Meaning |
|
|
75
|
-
|---|---|
|
|
76
|
-
| `200` | `EvaluateVerdict` — `{ score?: number; issues: string[] }` |
|
|
77
|
-
| `404` | trace or span not found |
|
|
78
|
-
| `422` | the span carried no captured system prompt to grade |
|
|
79
|
-
| `413` | request body over 64 KB |
|
|
80
|
-
| `400` | request body isn't valid JSON |
|
|
81
|
-
| `502` | `config.model` (a factory) threw resolving the judge model |
|
|
82
|
-
| `405` | POST hit this path but `evaluate` isn't configured |
|
|
83
|
-
|
|
84
|
-
```sh
|
|
85
|
-
curl -X POST 'http://127.0.0.1:4319/api/traces/tr_1/spans/sp_1/evaluate' \
|
|
86
|
-
-H 'content-type: application/json' \
|
|
87
|
-
-d '{"instructions": "Must never invent prices."}'
|
|
88
|
-
```
|
|
89
|
-
|
|
90
|
-
## See also
|
|
91
|
-
|
|
92
|
-
- [`@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md) — the dashboard this feature is a drawer action of; its security note now says "every route is GET-only and read-only EXCEPT the one opt-in write route `evaluate` enables."
|
|
93
|
-
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — `captureContent` (and `fullHistory`), the prerequisite for `span.input` to carry anything gradable.
|
|
94
|
-
- `ai.prompts().validate({ criteria })` in `@warlock.js/ai` — the sibling capability this reuses: grading raw prompt TEXT you already have in code, no trace or dashboard involved.
|
|
9
|
+
---
|
|
10
|
+
name: evaluate-system-prompt
|
|
11
|
+
description: 'Grade a single trace''s LAST captured system prompt from the dashboard drawer — the dashboard''s only write-capable feature, config-gated via `DashboardOptions.evaluate` (`ai.config({ panoptic: { dashboard: { evaluate: { model, instructions } } } })`). Triggers: `evaluate`, `EvaluateConfig`, `EvaluateVerdict`, `evaluateSystemPrompt`, `extractLastSystemPrompt`, `findSpanById`, "Evaluate system prompt" button, `POST .../spans/:spanId/evaluate`, grading a prompt from a real trace, judging a captured system prompt, per-run instructions override, judge model factory; ''evaluate this trace''s prompt'', ''grade the system prompt from the dashboard'', ''judge a captured run'', ''score a live system prompt''. Skip: grading raw prompt TEXT you already have in code (no trace involved) — `@warlock.js/ai`''s `ai.prompts().validate({ criteria })`, the same judge machinery this wraps; a DATASET of test cases run through an agent — `@warlock.js/ai`''s `ai.dataset` + `agent.eval`; viewing/filtering/grouping traces without grading them — `@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`; capturing the content this feature reads — `@warlock.js/ai-panoptic/export-traces/SKILL.md` (`captureContent`).'
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Evaluate a trace's system prompt from the dashboard
|
|
15
|
+
|
|
16
|
+
A drawer button that grades the LAST `{role: "system"}` message captured on the selected span — not the whole conversation, not the tool code, just the system prompt that was actually in effect for that run. It's the dashboard's only write-capable feature: every other route is `GET`-only and read-only by design (see `use-local-dashboard/SKILL.md`'s security note); this one POSTs, and it's off unless you explicitly opt in.
|
|
17
|
+
|
|
18
|
+
## Enable it
|
|
19
|
+
|
|
20
|
+
```ts
|
|
21
|
+
import { ai } from "@warlock.js/ai";
|
|
22
|
+
import "@warlock.js/ai-panoptic";
|
|
23
|
+
|
|
24
|
+
ai.config({
|
|
25
|
+
panoptic: {
|
|
26
|
+
dashboard: {
|
|
27
|
+
evaluate: {
|
|
28
|
+
model: openai.model("gpt-4o-mini"),
|
|
29
|
+
instructions: "Must address the user by name. Never invent prices.",
|
|
30
|
+
},
|
|
31
|
+
},
|
|
32
|
+
observeAll: true,
|
|
33
|
+
captureContent: true, // required — evaluate reads the captured span.input
|
|
34
|
+
},
|
|
35
|
+
});
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Absent `evaluate` ⇒ the drawer never renders the button and a `POST` to its route 405s like any other route would (the path is never even pattern-matched) — the dashboard stays fully read-only unless you opt in.
|
|
39
|
+
|
|
40
|
+
### `EvaluateConfig`
|
|
41
|
+
|
|
42
|
+
| Field | Required | Notes |
|
|
43
|
+
|---|---|---|
|
|
44
|
+
| `model` | yes | A `ModelContract`, or a `() => ModelContract \| Promise<ModelContract>` factory — mirrors `panoptic.cache`'s literal-or-factory ergonomics. Resolved fresh on every evaluate click (a deliberate, infrequent, human-triggered action — not the collector's hot path — so no memoization, unlike the cache store's driver). |
|
|
45
|
+
| `instructions` | no | Default grading rubric/criteria, verbatim — the same shape `ai.prompts().validate({ criteria })` accepts. Seeds the dashboard's editable textarea (baked into the served page at boot, so the operator sees and can extend the actual default, not a blank box); the built-in prompt-quality rubric is used when neither this nor a per-run override is supplied. |
|
|
46
|
+
|
|
47
|
+
## The UI
|
|
48
|
+
|
|
49
|
+
Open a trace, select the span whose prompt you want graded (any span carrying a captured `[system, user]` pair or `fullHistory` message array — an agent span, not a tool span). An "Evaluate system prompt" button appears under the input/output block **only when that span actually has a system prompt to grade** (`extractLastSystemPrompt` finds one) — no dead button on tool spans or system-prompt-less runs.
|
|
50
|
+
|
|
51
|
+
Click it to open an inline panel: a textarea pre-filled with the configured `instructions` (fully editable — type over it, append to it, or clear it to fall back to the built-in rubric) and a "Run" button. The result — `score` (0–1, shown as a percentage) and `issues` (the judge's reasoning, as a list) — renders inline once it lands. State (open/closed, typed text, last result) is kept per-span in client memory, so switching to a sibling span and back doesn't lose it; it does NOT persist to the trace store or survive a page reload.
|
|
52
|
+
|
|
53
|
+
## The building blocks (for embedding elsewhere)
|
|
54
|
+
|
|
55
|
+
```ts
|
|
56
|
+
import { evaluateSystemPrompt, extractLastSystemPrompt, findSpanById } from "@warlock.js/ai-panoptic";
|
|
57
|
+
|
|
58
|
+
const span = findSpanById(trace.root, spanId); // walk the span tree
|
|
59
|
+
const prompt = extractLastSystemPrompt(span); // undefined if none captured
|
|
60
|
+
if (prompt) {
|
|
61
|
+
const verdict = await evaluateSystemPrompt(prompt, evaluateConfig, "Per-run override, optional");
|
|
62
|
+
// verdict: { score?: number; issues: string[] }
|
|
63
|
+
}
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
- **`extractLastSystemPrompt(span)`** — reads `span.input` (only present under `captureContent`). Handles both shapes `report-to-span.ts` produces: the `[system, user]` first-trip pair, and — under `fullHistory` — the full `CapturedMessage[]` conversation, where it takes the LAST system-role turn (a long-running agent can carry more than one; the most recent one is the one actually in effect). Returns `undefined` for a tool span, an agent with no system prompt, or when content capture is off.
|
|
67
|
+
- **`findSpanById(root, spanId)`** — depth-first lookup inside a trace's span tree; the route uses it to resolve which node the drawer had selected.
|
|
68
|
+
- **`evaluateSystemPrompt(text, config, instructionsOverride?)`** — the grading call. `instructionsOverride` (trimmed; blank-after-trim is treated as absent, never as "grade against nothing") wins over `config.instructions`. Reuses `@warlock.js/ai`'s `judgePromptBody` verbatim — the SAME LLM-as-judge machinery `ai.prompts().validate({ criteria })` runs — so there's exactly one judging implementation in the whole framework, not two. `judgePromptBody` itself never throws (a broken judge degrades to an issues-only outcome with no `score`); the only step that CAN throw is resolving `config.model` when it's a factory (e.g. constructing an SDK client) — the dashboard route catches that and returns `502`.
|
|
69
|
+
|
|
70
|
+
## The route
|
|
71
|
+
|
|
72
|
+
`POST {basePath}api/traces/:traceId/spans/:spanId/evaluate`, body `{ instructions?: string }` (all optional — an empty body is valid). Gated by the same `authToken` / `allowedHosts` checks as every other route (S4) — off-loopback still requires a token, exactly like the rest of the dashboard. Body reads are capped at 64 KB (`413` past that) so a hostile/oversized payload can't hold the connection open.
|
|
73
|
+
|
|
74
|
+
| Status | Meaning |
|
|
75
|
+
|---|---|
|
|
76
|
+
| `200` | `EvaluateVerdict` — `{ score?: number; issues: string[] }` |
|
|
77
|
+
| `404` | trace or span not found |
|
|
78
|
+
| `422` | the span carried no captured system prompt to grade |
|
|
79
|
+
| `413` | request body over 64 KB |
|
|
80
|
+
| `400` | request body isn't valid JSON |
|
|
81
|
+
| `502` | `config.model` (a factory) threw resolving the judge model |
|
|
82
|
+
| `405` | POST hit this path but `evaluate` isn't configured |
|
|
83
|
+
|
|
84
|
+
```sh
|
|
85
|
+
curl -X POST 'http://127.0.0.1:4319/api/traces/tr_1/spans/sp_1/evaluate' \
|
|
86
|
+
-H 'content-type: application/json' \
|
|
87
|
+
-d '{"instructions": "Must never invent prices."}'
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## See also
|
|
91
|
+
|
|
92
|
+
- [`@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md) — the dashboard this feature is a drawer action of; its security note now says "every route is GET-only and read-only EXCEPT the one opt-in write route `evaluate` enables."
|
|
93
|
+
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — `captureContent` (and `fullHistory`), the prerequisite for `span.input` to carry anything gradable.
|
|
94
|
+
- `ai.prompts().validate({ criteria })` in `@warlock.js/ai` — the sibling capability this reuses: grading raw prompt TEXT you already have in code, no trace or dashboard involved.
|
|
95
95
|
|
|
96
96
|
|
|
97
97
|
## export-traces `@warlock.js/ai-panoptic/export-traces/SKILL.md`
|
|
98
98
|
|
|
99
|
-
---
|
|
100
|
-
name: export-traces
|
|
101
|
-
description: 'Send @warlock.js/ai-panoptic traces to an observability backend via pluggable exporters, including capturing prompt/response content onto spans. Triggers: `consoleExporter`, `fileExporter`, `otelExporter`, `langfuseExporter`, `ExporterContract`, `ContentCaptureOptions`, `captureContent`, `redactContent`, `fullHistory`, `ContentRedactor`, `toGenAiAttributes`, `walkSpans`, `totalCostUsd`, `GEN_AI_ATTRIBUTES`, `WARLOCK_ATTRIBUTES`, `reportToTrace`, `reportToSpan`, `extractSpanAttributes`, `normalizeError`, `formatSpanLine`, `createPanopticMiddleware`; ''export AI traces to OpenTelemetry'', ''send traces to Langfuse'', ''log traces to console / a file'', ''capture the full prompt/conversation onto a span'', ''gen_ai semantic conventions'', ''write a custom exporter''; typical import `import { otelExporter, langfuseExporter, consoleExporter, fileExporter } from "@warlock.js/ai-panoptic"`. Skip: wiring the subscriber/collector into a run — `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`; the core report shape — `@warlock.js/ai/run-ai-agent/SKILL.md`.'
|
|
102
|
-
---
|
|
103
|
-
|
|
104
|
-
# Export traces — the sink end of Panoptic
|
|
105
|
-
|
|
106
|
-
An **exporter** translates a vendor-neutral `Trace` (produced by the collector) into a backend's wire format. The collector fans every completed trace out to each registered exporter via the `ExporterContract`:
|
|
107
|
-
|
|
108
|
-
```ts
|
|
109
|
-
export interface ExporterContract {
|
|
110
|
-
readonly name: string;
|
|
111
|
-
export(trace: Trace): void | Promise<void>;
|
|
112
|
-
exportSpan?(span: TraceSpan): void | Promise<void>; // optional live streaming
|
|
113
|
-
flush?(): void | Promise<void>;
|
|
114
|
-
shutdown?(): void | Promise<void>;
|
|
115
|
-
}
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
Four exporters ship today. Each is a **factory** returning an `ExporterContract` — callers never `new`.
|
|
119
|
-
|
|
120
|
-
## Pick an exporter
|
|
121
|
-
|
|
122
|
-
| Exporter | Backend | Deps | Use it for |
|
|
123
|
-
|---|---|---|---|
|
|
124
|
-
| `consoleExporter()` | stdout/stderr | none | local dev, CI logs |
|
|
125
|
-
| `fileExporter()` | JSON-Lines file | none | durable local sink, replay, out-of-band shipping |
|
|
126
|
-
| `otelExporter()` | OpenTelemetry | `@opentelemetry/api` (optional peer) | OTel-native stacks, GenAI semantic conventions |
|
|
127
|
-
| `langfuseExporter()` | Langfuse | `langfuse` (optional peer) | LLM-specific dashboards, generations |
|
|
128
|
-
|
|
129
|
-
> `@opentelemetry/api` and `langfuse` are **OPTIONAL peer dependencies**, lazily imported by their exporter. Importing `@warlock.js/ai-panoptic` never pulls them in; a missing SDK surfaces as a curated *"install this"* error on first export, never a boot-time stack trace.
|
|
130
|
-
|
|
131
|
-
## Console
|
|
132
|
-
|
|
133
|
-
```ts
|
|
134
|
-
import { consoleExporter } from "@warlock.js/ai-panoptic";
|
|
135
|
-
|
|
136
|
-
collector.use(consoleExporter()); // one summary line per trace
|
|
137
|
-
collector.use(consoleExporter({ tree: true })); // full indented span tree
|
|
138
|
-
collector.use(consoleExporter({ streaming: true })); // also print each span as it finalizes
|
|
139
|
-
```
|
|
140
|
-
|
|
141
|
-
Failed/cancelled spans route to `console.error`; everything else to `console.log`. Inject a sink (`{ console: mySink }`) to redirect or to capture in tests.
|
|
142
|
-
|
|
143
|
-
## File
|
|
144
|
-
|
|
145
|
-
```ts
|
|
146
|
-
import { fileExporter } from "@warlock.js/ai-panoptic";
|
|
147
|
-
|
|
148
|
-
collector.use(fileExporter({ path: "storage/traces.jsonl" })); // write every trace
|
|
149
|
-
collector.use(fileExporter({ path: "...", flushEvery: 50 })); // batch 50 traces per write
|
|
150
|
-
collector.use(fileExporter({ path: "...", pretty: true })); // multi-line records
|
|
151
|
-
```
|
|
152
|
-
|
|
153
|
-
Appends one JSON record per line (a `TraceRecord` = `{ type: "trace", exportedAt, trace }`). Buffers in memory and drains on `flush()` / `shutdown()` — call `collector.shutdown()` on teardown so the last batch lands. The parent directory is created on first write.
|
|
154
|
-
|
|
155
|
-
## OpenTelemetry
|
|
156
|
-
|
|
157
|
-
The exporter emits spans onto a `Tracer`; **the host app owns the SDK wiring** (`TracerProvider`, processors, span exporters), exactly as with any OTel instrumentation.
|
|
158
|
-
|
|
159
|
-
```ts
|
|
160
|
-
import { otelExporter } from "@warlock.js/ai-panoptic";
|
|
161
|
-
|
|
162
|
-
// app already configured @opentelemetry/sdk-trace-base + a global provider
|
|
163
|
-
collector.use(otelExporter({ tracerName: "my-app", system: "openai" }));
|
|
164
|
-
|
|
165
|
-
// or pass a Tracer directly
|
|
166
|
-
collector.use(otelExporter({ tracer: myTracer }));
|
|
167
|
-
```
|
|
168
|
-
|
|
169
|
-
Each `TraceSpan` becomes one OTel span with the source start/end times and parent relationship reconstructed, so the emitted tree matches the original execution tree. Attributes follow the [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/):
|
|
170
|
-
|
|
171
|
-
- `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` / `gen_ai.usage.total_tokens` (from the span's typed `usage` rollup)
|
|
172
|
-
- `gen_ai.usage.cached_tokens` / `gen_ai.usage.reasoning_tokens` (when the span reported them)
|
|
173
|
-
- `gen_ai.conversation.id` (from the span's `sessionId`)
|
|
174
|
-
- `gen_ai.system` / `gen_ai.request.model` / `gen_ai.operation.name` — **never invented**: these appear only when the span's free-form `attributes` bag already carried that exact key. `otelExporter`'s `system` option backfills `gen_ai.system` when the span supplied none (and never overrides one it did).
|
|
175
|
-
- `warlock.report.type` / `warlock.version` / `warlock.duration_ms` / `warlock.cost.usd`
|
|
176
|
-
- the collector's own per-primitive keys ride along verbatim — `agent.model.name` / `agent.model.provider` / `agent.trips`, `workflow.steps`, `supervisor.iterations` / `supervisor.terminatedBy`, `tool.tripIndex`, `retries`, … — since `toGenAiAttributes` copies every scalar entry of the span's `attributes` bag.
|
|
177
|
-
|
|
178
|
-
Failed/cancelled spans get OTel `ERROR` status plus a recorded exception.
|
|
179
|
-
|
|
180
|
-
## Langfuse
|
|
181
|
-
|
|
182
|
-
```ts
|
|
183
|
-
import { langfuseExporter } from "@warlock.js/ai-panoptic";
|
|
184
|
-
|
|
185
|
-
collector.use(langfuseExporter({ publicKey: "pk-...", secretKey: "sk-...", baseUrl: "..." }));
|
|
186
|
-
// or reuse an existing Langfuse client
|
|
187
|
-
collector.use(langfuseExporter({ client: myLangfuse }));
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
- The root span becomes a Langfuse **trace**.
|
|
191
|
-
- A token-producing span (`usage.total > 0`) becomes a **generation** (carries `usage`); a pure tool/callback becomes a plain **span**.
|
|
192
|
-
- Children nest under their parent observation; timing, `version`, and status (failed → `ERROR` level) are mapped.
|
|
193
|
-
- `flush()` / `shutdown()` delegate to the client's `flushAsync()` / `shutdownAsync()` — call `collector.shutdown()` on teardown so buffered observations are sent.
|
|
194
|
-
|
|
195
|
-
## Content capture — what reaches the `input` / `output` fields
|
|
196
|
-
|
|
197
|
-
Exporters surface `span.input` / `span.output` (console `in:` / `out:`, file JSON, OTel `gen_ai.prompt` / `gen_ai.completion`, Langfuse native `input` / `output`) only when **content capture** is enabled on the collector or subscriber. It's `ContentCaptureOptions`, set on `createCollector(...)` or `panoptic(...)`:
|
|
198
|
-
|
|
199
|
-
```ts
|
|
200
|
-
panoptic({
|
|
201
|
-
exporters: [langfuseExporter({ /* ... */ })],
|
|
202
|
-
captureContent: true, // copy prompts/responses + tool args/results onto spans
|
|
203
|
-
redactContent: (value, { field }) => // optional masker, per value
|
|
204
|
-
field === "input" ? "[redacted]" : value,
|
|
205
|
-
fullHistory: true, // emit the WHOLE conversation as span.input (see below)
|
|
206
|
-
});
|
|
207
|
-
```
|
|
208
|
-
|
|
209
|
-
Off by default — payloads are large and frequently sensitive. With `captureContent`, an agent span's `input` is a `[system, user]` chat array (or the bare user string when there's no system prompt) and its `output` is the last non-empty trip's text; a tool span carries its call args + result.
|
|
210
|
-
|
|
211
|
-
### `fullHistory` — emit the entire `CapturedMessage[]`
|
|
212
|
-
|
|
213
|
-
`fullHistory: true` replaces the `[system, user]` first-trip array on `span.input` with the agent's **complete assembled conversation** — every trip, every role (system / user / assistant / tool). The `output` stays the final response text.
|
|
214
|
-
|
|
215
|
-
```ts
|
|
216
|
-
// upstream: the agent run MUST opt into capturing its messages
|
|
217
|
-
const result = await agent.execute(input, { captureMessages: true });
|
|
218
|
-
// ^ core @warlock.js/ai — makes report.messages present
|
|
219
|
-
|
|
220
|
-
panoptic({ exporters: [...], captureContent: true, fullHistory: true });
|
|
221
|
-
```
|
|
222
|
-
|
|
223
|
-
**Requires the upstream `captureMessages`.** `fullHistory` reads `report.messages` (the captured `CapturedMessage[]`). When the run did NOT opt into `captureMessages`, `report.messages` is absent and the branch **degrades gracefully** to today's first-trip `[system, user]` capture — never an error, just less history. The whole array is passed through `ContentRedactor` as a single value, so masking still applies to captured history.
|
|
224
|
-
|
|
225
|
-
Reach for `fullHistory` when a backend (Langfuse) should show the full multi-turn / tool-loop transcript a run actually sent, not just its opening prompt.
|
|
226
|
-
|
|
227
|
-
## Shared utilities
|
|
228
|
-
|
|
229
|
-
Reusable when authoring a **custom exporter**:
|
|
230
|
-
|
|
231
|
-
```ts
|
|
232
|
-
import { walkSpans, toGenAiAttributes, totalCostUsd, GEN_AI_ATTRIBUTES, WARLOCK_ATTRIBUTES } from "@warlock.js/ai-panoptic";
|
|
233
|
-
|
|
234
|
-
for (const span of walkSpans(trace.root)) { // depth-first pre-order
|
|
235
|
-
const attributes = toGenAiAttributes(span); // gen_ai.* / warlock.* attribute map
|
|
236
|
-
const cost = totalCostUsd(span.usage); // single USD scalar, or undefined
|
|
237
|
-
}
|
|
238
|
-
```
|
|
239
|
-
|
|
240
|
-
`toGenAiAttributes` copies only **scalar** entries from a span's free-form `attributes` bag (objects/arrays are skipped — backend attribute values must be primitives) and lets an explicit collector attribute override a derived one.
|
|
241
|
-
|
|
242
|
-
## Write a custom exporter
|
|
243
|
-
|
|
244
|
-
Implement `ExporterContract` directly — `name` + `export` are the only required members:
|
|
245
|
-
|
|
246
|
-
```ts
|
|
247
|
-
import type { ExporterContract, Trace } from "@warlock.js/ai-panoptic";
|
|
248
|
-
|
|
249
|
-
const myExporter: ExporterContract = {
|
|
250
|
-
name: "my-backend",
|
|
251
|
-
async export(trace: Trace) {
|
|
252
|
-
await sendSomewhere(trace);
|
|
253
|
-
},
|
|
254
|
-
};
|
|
255
|
-
|
|
256
|
-
collector.use(myExporter);
|
|
257
|
-
```
|
|
258
|
-
|
|
259
|
-
Wrapping a third-party SDK? Lazily `await import(...)` it **inside** the export path (never at module top) so it stays an optional peer — copy the loader pattern from `otelExporter` / `langfuseExporter`. The collector isolates a throwing exporter, so one backend's failure never crashes the originating AI run.
|
|
260
|
-
|
|
261
|
-
## Lower-level building blocks
|
|
262
|
-
|
|
263
|
-
Most apps only need `panoptic(...)`, an exporter, and the store. These lower-level exports exist for **custom collectors / exporters and bespoke pipelines** — each is a pure function over the report tree:
|
|
264
|
-
|
|
265
|
-
```ts
|
|
266
|
-
import {
|
|
267
|
-
reportToTrace, // (report: BaseReport) => Trace — project a finished report into a vendor-neutral Trace
|
|
268
|
-
reportToSpan, // (report: BaseReport) => TraceSpan — project one report node into a span (recurses into children)
|
|
269
|
-
extractSpanAttributes, // (report: BaseReport) => Record<string, unknown> | undefined — the per-primitive attribute bag (agent.model.name, workflow.steps, …)
|
|
270
|
-
normalizeError, // (error: unknown) => TraceSpanError | undefined — coerce any thrown value into the typed span-error shape
|
|
271
|
-
formatSpanLine, // (span: TraceSpan, depth?: number) => string — render one indented console line (consoleExporter's tree mode uses this)
|
|
272
|
-
createPanopticMiddleware, // (collector: CollectorContract, name?: string) => AgentMiddleware — build a middleware from a collector you already hold (panoptic(...) wires this for you)
|
|
273
|
-
} from "@warlock.js/ai-panoptic";
|
|
274
|
-
```
|
|
275
|
-
|
|
276
|
-
Reach for `reportToTrace` when you already hold a finished `result.report` and want a `Trace` without the live subscriber; `formatSpanLine` to build a custom console renderer; `createPanopticMiddleware` when you assemble the agent's `middleware` array by hand instead of letting `panoptic(...)` do it.
|
|
277
|
-
|
|
278
|
-
## See also
|
|
279
|
-
|
|
280
|
-
- [`@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) — the source end: wiring the `panoptic(...)` subscriber into a run (`attach` / `middleware` / `collect`). The `collector` these exporters register on comes from there.
|
|
281
|
-
- [`@warlock.js/ai-panoptic/query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md) — the in-memory trace store: a sibling sink that *retains* traces (instead of shipping them out) so you can `query` / `aggregate` them after the fact.
|
|
282
|
-
- [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md) — running the agent and the `BaseReport` shape Panoptic projects into a `Trace`.
|
|
99
|
+
---
|
|
100
|
+
name: export-traces
|
|
101
|
+
description: 'Send @warlock.js/ai-panoptic traces to an observability backend via pluggable exporters, including capturing prompt/response content onto spans. Triggers: `consoleExporter`, `fileExporter`, `otelExporter`, `langfuseExporter`, `ExporterContract`, `ContentCaptureOptions`, `captureContent`, `redactContent`, `fullHistory`, `ContentRedactor`, `toGenAiAttributes`, `walkSpans`, `totalCostUsd`, `GEN_AI_ATTRIBUTES`, `WARLOCK_ATTRIBUTES`, `reportToTrace`, `reportToSpan`, `extractSpanAttributes`, `normalizeError`, `formatSpanLine`, `createPanopticMiddleware`; ''export AI traces to OpenTelemetry'', ''send traces to Langfuse'', ''log traces to console / a file'', ''capture the full prompt/conversation onto a span'', ''gen_ai semantic conventions'', ''write a custom exporter''; typical import `import { otelExporter, langfuseExporter, consoleExporter, fileExporter } from "@warlock.js/ai-panoptic"`. Skip: wiring the subscriber/collector into a run — `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`; the core report shape — `@warlock.js/ai/run-ai-agent/SKILL.md`.'
|
|
102
|
+
---
|
|
103
|
+
|
|
104
|
+
# Export traces — the sink end of Panoptic
|
|
105
|
+
|
|
106
|
+
An **exporter** translates a vendor-neutral `Trace` (produced by the collector) into a backend's wire format. The collector fans every completed trace out to each registered exporter via the `ExporterContract`:
|
|
107
|
+
|
|
108
|
+
```ts
|
|
109
|
+
export interface ExporterContract {
|
|
110
|
+
readonly name: string;
|
|
111
|
+
export(trace: Trace): void | Promise<void>;
|
|
112
|
+
exportSpan?(span: TraceSpan): void | Promise<void>; // optional live streaming
|
|
113
|
+
flush?(): void | Promise<void>;
|
|
114
|
+
shutdown?(): void | Promise<void>;
|
|
115
|
+
}
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Four exporters ship today. Each is a **factory** returning an `ExporterContract` — callers never `new`.
|
|
119
|
+
|
|
120
|
+
## Pick an exporter
|
|
121
|
+
|
|
122
|
+
| Exporter | Backend | Deps | Use it for |
|
|
123
|
+
|---|---|---|---|
|
|
124
|
+
| `consoleExporter()` | stdout/stderr | none | local dev, CI logs |
|
|
125
|
+
| `fileExporter()` | JSON-Lines file | none | durable local sink, replay, out-of-band shipping |
|
|
126
|
+
| `otelExporter()` | OpenTelemetry | `@opentelemetry/api` (optional peer) | OTel-native stacks, GenAI semantic conventions |
|
|
127
|
+
| `langfuseExporter()` | Langfuse | `langfuse` (optional peer) | LLM-specific dashboards, generations |
|
|
128
|
+
|
|
129
|
+
> `@opentelemetry/api` and `langfuse` are **OPTIONAL peer dependencies**, lazily imported by their exporter. Importing `@warlock.js/ai-panoptic` never pulls them in; a missing SDK surfaces as a curated *"install this"* error on first export, never a boot-time stack trace.
|
|
130
|
+
|
|
131
|
+
## Console
|
|
132
|
+
|
|
133
|
+
```ts
|
|
134
|
+
import { consoleExporter } from "@warlock.js/ai-panoptic";
|
|
135
|
+
|
|
136
|
+
collector.use(consoleExporter()); // one summary line per trace
|
|
137
|
+
collector.use(consoleExporter({ tree: true })); // full indented span tree
|
|
138
|
+
collector.use(consoleExporter({ streaming: true })); // also print each span as it finalizes
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Failed/cancelled spans route to `console.error`; everything else to `console.log`. Inject a sink (`{ console: mySink }`) to redirect or to capture in tests.
|
|
142
|
+
|
|
143
|
+
## File
|
|
144
|
+
|
|
145
|
+
```ts
|
|
146
|
+
import { fileExporter } from "@warlock.js/ai-panoptic";
|
|
147
|
+
|
|
148
|
+
collector.use(fileExporter({ path: "storage/traces.jsonl" })); // write every trace
|
|
149
|
+
collector.use(fileExporter({ path: "...", flushEvery: 50 })); // batch 50 traces per write
|
|
150
|
+
collector.use(fileExporter({ path: "...", pretty: true })); // multi-line records
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
Appends one JSON record per line (a `TraceRecord` = `{ type: "trace", exportedAt, trace }`). Buffers in memory and drains on `flush()` / `shutdown()` — call `collector.shutdown()` on teardown so the last batch lands. The parent directory is created on first write.
|
|
154
|
+
|
|
155
|
+
## OpenTelemetry
|
|
156
|
+
|
|
157
|
+
The exporter emits spans onto a `Tracer`; **the host app owns the SDK wiring** (`TracerProvider`, processors, span exporters), exactly as with any OTel instrumentation.
|
|
158
|
+
|
|
159
|
+
```ts
|
|
160
|
+
import { otelExporter } from "@warlock.js/ai-panoptic";
|
|
161
|
+
|
|
162
|
+
// app already configured @opentelemetry/sdk-trace-base + a global provider
|
|
163
|
+
collector.use(otelExporter({ tracerName: "my-app", system: "openai" }));
|
|
164
|
+
|
|
165
|
+
// or pass a Tracer directly
|
|
166
|
+
collector.use(otelExporter({ tracer: myTracer }));
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Each `TraceSpan` becomes one OTel span with the source start/end times and parent relationship reconstructed, so the emitted tree matches the original execution tree. Attributes follow the [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/):
|
|
170
|
+
|
|
171
|
+
- `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` / `gen_ai.usage.total_tokens` (from the span's typed `usage` rollup)
|
|
172
|
+
- `gen_ai.usage.cached_tokens` / `gen_ai.usage.reasoning_tokens` (when the span reported them)
|
|
173
|
+
- `gen_ai.conversation.id` (from the span's `sessionId`)
|
|
174
|
+
- `gen_ai.system` / `gen_ai.request.model` / `gen_ai.operation.name` — **never invented**: these appear only when the span's free-form `attributes` bag already carried that exact key. `otelExporter`'s `system` option backfills `gen_ai.system` when the span supplied none (and never overrides one it did).
|
|
175
|
+
- `warlock.report.type` / `warlock.version` / `warlock.duration_ms` / `warlock.cost.usd`
|
|
176
|
+
- the collector's own per-primitive keys ride along verbatim — `agent.model.name` / `agent.model.provider` / `agent.trips`, `workflow.steps`, `supervisor.iterations` / `supervisor.terminatedBy`, `tool.tripIndex`, `retries`, … — since `toGenAiAttributes` copies every scalar entry of the span's `attributes` bag.
|
|
177
|
+
|
|
178
|
+
Failed/cancelled spans get OTel `ERROR` status plus a recorded exception.
|
|
179
|
+
|
|
180
|
+
## Langfuse
|
|
181
|
+
|
|
182
|
+
```ts
|
|
183
|
+
import { langfuseExporter } from "@warlock.js/ai-panoptic";
|
|
184
|
+
|
|
185
|
+
collector.use(langfuseExporter({ publicKey: "pk-...", secretKey: "sk-...", baseUrl: "..." }));
|
|
186
|
+
// or reuse an existing Langfuse client
|
|
187
|
+
collector.use(langfuseExporter({ client: myLangfuse }));
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
- The root span becomes a Langfuse **trace**.
|
|
191
|
+
- A token-producing span (`usage.total > 0`) becomes a **generation** (carries `usage`); a pure tool/callback becomes a plain **span**.
|
|
192
|
+
- Children nest under their parent observation; timing, `version`, and status (failed → `ERROR` level) are mapped.
|
|
193
|
+
- `flush()` / `shutdown()` delegate to the client's `flushAsync()` / `shutdownAsync()` — call `collector.shutdown()` on teardown so buffered observations are sent.
|
|
194
|
+
|
|
195
|
+
## Content capture — what reaches the `input` / `output` fields
|
|
196
|
+
|
|
197
|
+
Exporters surface `span.input` / `span.output` (console `in:` / `out:`, file JSON, OTel `gen_ai.prompt` / `gen_ai.completion`, Langfuse native `input` / `output`) only when **content capture** is enabled on the collector or subscriber. It's `ContentCaptureOptions`, set on `createCollector(...)` or `panoptic(...)`:
|
|
198
|
+
|
|
199
|
+
```ts
|
|
200
|
+
panoptic({
|
|
201
|
+
exporters: [langfuseExporter({ /* ... */ })],
|
|
202
|
+
captureContent: true, // copy prompts/responses + tool args/results onto spans
|
|
203
|
+
redactContent: (value, { field }) => // optional masker, per value
|
|
204
|
+
field === "input" ? "[redacted]" : value,
|
|
205
|
+
fullHistory: true, // emit the WHOLE conversation as span.input (see below)
|
|
206
|
+
});
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
Off by default — payloads are large and frequently sensitive. With `captureContent`, an agent span's `input` is a `[system, user]` chat array (or the bare user string when there's no system prompt) and its `output` is the last non-empty trip's text; a tool span carries its call args + result.
|
|
210
|
+
|
|
211
|
+
### `fullHistory` — emit the entire `CapturedMessage[]`
|
|
212
|
+
|
|
213
|
+
`fullHistory: true` replaces the `[system, user]` first-trip array on `span.input` with the agent's **complete assembled conversation** — every trip, every role (system / user / assistant / tool). The `output` stays the final response text.
|
|
214
|
+
|
|
215
|
+
```ts
|
|
216
|
+
// upstream: the agent run MUST opt into capturing its messages
|
|
217
|
+
const result = await agent.execute(input, { captureMessages: true });
|
|
218
|
+
// ^ core @warlock.js/ai — makes report.messages present
|
|
219
|
+
|
|
220
|
+
panoptic({ exporters: [...], captureContent: true, fullHistory: true });
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
**Requires the upstream `captureMessages`.** `fullHistory` reads `report.messages` (the captured `CapturedMessage[]`). When the run did NOT opt into `captureMessages`, `report.messages` is absent and the branch **degrades gracefully** to today's first-trip `[system, user]` capture — never an error, just less history. The whole array is passed through `ContentRedactor` as a single value, so masking still applies to captured history.
|
|
224
|
+
|
|
225
|
+
Reach for `fullHistory` when a backend (Langfuse) should show the full multi-turn / tool-loop transcript a run actually sent, not just its opening prompt.
|
|
226
|
+
|
|
227
|
+
## Shared utilities
|
|
228
|
+
|
|
229
|
+
Reusable when authoring a **custom exporter**:
|
|
230
|
+
|
|
231
|
+
```ts
|
|
232
|
+
import { walkSpans, toGenAiAttributes, totalCostUsd, GEN_AI_ATTRIBUTES, WARLOCK_ATTRIBUTES } from "@warlock.js/ai-panoptic";
|
|
233
|
+
|
|
234
|
+
for (const span of walkSpans(trace.root)) { // depth-first pre-order
|
|
235
|
+
const attributes = toGenAiAttributes(span); // gen_ai.* / warlock.* attribute map
|
|
236
|
+
const cost = totalCostUsd(span.usage); // single USD scalar, or undefined
|
|
237
|
+
}
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
`toGenAiAttributes` copies only **scalar** entries from a span's free-form `attributes` bag (objects/arrays are skipped — backend attribute values must be primitives) and lets an explicit collector attribute override a derived one.
|
|
241
|
+
|
|
242
|
+
## Write a custom exporter
|
|
243
|
+
|
|
244
|
+
Implement `ExporterContract` directly — `name` + `export` are the only required members:
|
|
245
|
+
|
|
246
|
+
```ts
|
|
247
|
+
import type { ExporterContract, Trace } from "@warlock.js/ai-panoptic";
|
|
248
|
+
|
|
249
|
+
const myExporter: ExporterContract = {
|
|
250
|
+
name: "my-backend",
|
|
251
|
+
async export(trace: Trace) {
|
|
252
|
+
await sendSomewhere(trace);
|
|
253
|
+
},
|
|
254
|
+
};
|
|
255
|
+
|
|
256
|
+
collector.use(myExporter);
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
Wrapping a third-party SDK? Lazily `await import(...)` it **inside** the export path (never at module top) so it stays an optional peer — copy the loader pattern from `otelExporter` / `langfuseExporter`. The collector isolates a throwing exporter, so one backend's failure never crashes the originating AI run.
|
|
260
|
+
|
|
261
|
+
## Lower-level building blocks
|
|
262
|
+
|
|
263
|
+
Most apps only need `panoptic(...)`, an exporter, and the store. These lower-level exports exist for **custom collectors / exporters and bespoke pipelines** — each is a pure function over the report tree:
|
|
264
|
+
|
|
265
|
+
```ts
|
|
266
|
+
import {
|
|
267
|
+
reportToTrace, // (report: BaseReport) => Trace — project a finished report into a vendor-neutral Trace
|
|
268
|
+
reportToSpan, // (report: BaseReport) => TraceSpan — project one report node into a span (recurses into children)
|
|
269
|
+
extractSpanAttributes, // (report: BaseReport) => Record<string, unknown> | undefined — the per-primitive attribute bag (agent.model.name, workflow.steps, …)
|
|
270
|
+
normalizeError, // (error: unknown) => TraceSpanError | undefined — coerce any thrown value into the typed span-error shape
|
|
271
|
+
formatSpanLine, // (span: TraceSpan, depth?: number) => string — render one indented console line (consoleExporter's tree mode uses this)
|
|
272
|
+
createPanopticMiddleware, // (collector: CollectorContract, name?: string) => AgentMiddleware — build a middleware from a collector you already hold (panoptic(...) wires this for you)
|
|
273
|
+
} from "@warlock.js/ai-panoptic";
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
Reach for `reportToTrace` when you already hold a finished `result.report` and want a `Trace` without the live subscriber; `formatSpanLine` to build a custom console renderer; `createPanopticMiddleware` when you assemble the agent's `middleware` array by hand instead of letting `panoptic(...)` do it.
|
|
277
|
+
|
|
278
|
+
## See also
|
|
279
|
+
|
|
280
|
+
- [`@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) — the source end: wiring the `panoptic(...)` subscriber into a run (`attach` / `middleware` / `collect`). The `collector` these exporters register on comes from there.
|
|
281
|
+
- [`@warlock.js/ai-panoptic/query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md) — the in-memory trace store: a sibling sink that *retains* traces (instead of shipping them out) so you can `query` / `aggregate` them after the fact.
|
|
282
|
+
- [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md) — running the agent and the `BaseReport` shape Panoptic projects into a `Trace`.
|
|
283
283
|
|
|
284
284
|
|
|
285
285
|
## observe-with-panoptic `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`
|
|
286
286
|
|
|
287
|
-
---
|
|
288
|
-
name: observe-with-panoptic
|
|
289
|
-
description: 'Wire @warlock.js/ai-panoptic into an agent/workflow/supervisor/orchestrator — declaratively via `ai.config({ panoptic })` (registers panoptic on core''s Observer seam) or with the one-call `panoptic({ exporters })` subscriber. Triggers: `panoptic`, `Panoptic`, `PanopticOptions`, `PanopticConfig`, `ai.config({ panoptic })`, `observeAll`, per-flow `observe`, `FlowObserveOption`, `registerObserver`, `Observer`, `.attach`, `.middleware`, `.collect`, `.toTrace`, `observe.attach(agent)`, `panoptic().middleware()`, `completedEvents`, `middlewareName`; ''observe a warlock agent'', ''trace an agent run'', ''attach observability to a workflow/supervisor'', ''observe every flow by default'', ''add a tracing middleware'', ''collect an orchestrator turn report'', ''wire OTel/Langfuse into my agent''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"` (or `import { panoptic } from "@warlock.js/ai-panoptic"`). Skip: the zero-setup local dashboard — `@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`; writing/choosing an exporter (the sink end) — `@warlock.js/ai-panoptic/export-traces/SKILL.md`; the core report shape / running the agent itself — `@warlock.js/ai/run-ai-agent/SKILL.md`.'
|
|
290
|
-
---
|
|
291
|
-
|
|
292
|
-
# Observe a run with Panoptic — the one-call subscriber
|
|
293
|
-
|
|
294
|
-
There are two levels. **Declarative** (`ai.config({ panoptic })`) is the high-level path: panoptic registers itself on core's generic `Observer` seam and every observed flow routes its report in automatically — no per-target wiring. **Imperative** (`panoptic(...)`) is the underlying subscriber, with three explicit feed paths. The declarative path is sugar over the imperative one; both reuse the same collection pipeline.
|
|
295
|
-
|
|
296
|
-
## Declarative — `ai.config({ panoptic })` + the Observer seam
|
|
297
|
-
|
|
298
|
-
The simplest wiring. Importing `@warlock.js/ai-panoptic` (even bare, for side effects) subscribes panoptic to core's config seam; a later `ai.config({ panoptic })` then builds the collector and **registers it once on core's `Observer` registry** via `registerObserver`. Core itself never imports panoptic — it depends only on the structural `Observer` interface (`{ collect(report) }`), the dependency inversion that keeps the two sides decoupled.
|
|
299
|
-
|
|
300
|
-
```ts
|
|
301
|
-
import { ai } from "@warlock.js/ai";
|
|
302
|
-
import "@warlock.js/ai-panoptic"; // side-effect wiring — do this once at boot
|
|
303
|
-
|
|
304
|
-
ai.config({
|
|
305
|
-
panoptic: {
|
|
306
|
-
exporters: [consoleExporter()], // optional; a store-shaped exporter also feeds the dashboard
|
|
307
|
-
observeAll: true, // observe every flow by default
|
|
308
|
-
dashboard: true, // optional local viewer — see use-local-dashboard
|
|
309
|
-
},
|
|
310
|
-
});
|
|
311
|
-
```
|
|
312
|
-
|
|
313
|
-
`PanopticConfig`: `exporters?` (the collector's sinks), `observeAll?` (default `false`), `dashboard?` (`boolean | DashboardOptions`). Applying it is **idempotent** — the observer registers once, the dashboard starts at most once; repeat `ai.config(...)` merges only refresh `observeAll`.
|
|
314
|
-
|
|
315
|
-
### `observeAll` + per-flow `observe`
|
|
316
|
-
|
|
317
|
-
Once a collector is registered, which flows actually route to it is controlled by core's `observe` seam:
|
|
318
|
-
|
|
319
|
-
| Setting | Effect |
|
|
320
|
-
|---|---|
|
|
321
|
-
| `observeAll: true` (config) | Every `ai.agent` / `ai.workflow` / `ai.supervisor` / `ai.team` that left `observe` unset routes its completed report to the registered observers. |
|
|
322
|
-
| `observe: true` (per-flow) | Force-observe **this** flow even when `observeAll` is off → the globally registered observers. |
|
|
323
|
-
| `observe: false` (per-flow) | Opt **this** flow out entirely, even when `observeAll` is on. |
|
|
324
|
-
| `observe: <Observer>` (per-flow) | A **flow-local** collector — only this flow's report, routed only to it (the global observers are skipped). A panoptic flow-local collector implements `Observer`, so pass it directly. |
|
|
325
|
-
| (unset, `observeAll: false`) | Not observed — opt-in default, identical to legacy behavior. |
|
|
326
|
-
|
|
327
|
-
```ts
|
|
328
|
-
ai.agent({ model, observe: true }); // always traced
|
|
329
|
-
ai.workflow({ name: "billing", steps, observe: true }); // always traced
|
|
330
|
-
ai.supervisor({ name: "triage", ..., observe: false }); // never traced, even under observeAll
|
|
331
|
-
|
|
332
|
-
const flowLocal = panoptic({ exporters: [fileExporter({ path: "billing.jsonl" })] });
|
|
333
|
-
ai.team({ name: "research", ..., observe: flowLocal }); // only this flow, only this collector
|
|
334
|
-
```
|
|
335
|
-
|
|
336
|
-
The flow awaits each observer's `collect`, and a throwing observer is **swallowed** — observability never breaks the run (mirrors `onUsage` / `onComplete`). `observe` lives on **all four** flow configs — `ai.agent`, `ai.workflow`, `ai.supervisor`, and `ai.team` (the agent additionally supports the imperative `attach()` / `middleware()` paths below).
|
|
337
|
-
|
|
338
|
-
## Imperative — `panoptic(...)`, the one-call subscriber
|
|
339
|
-
|
|
340
|
-
`panoptic(...)` builds a collector, registers your exporters on it, and hands back a subscriber with **three ways to feed traces** in. All three converge on the same collector, so each run reaches every exporter exactly once. Use it when you want explicit control, a flow-local collector, or to wire a target the declarative seam doesn't cover (the orchestrator).
|
|
341
|
-
|
|
342
|
-
```ts
|
|
343
|
-
import { panoptic } from "@warlock.js/ai-panoptic";
|
|
344
|
-
import { consoleExporter, otelExporter } from "@warlock.js/ai-panoptic";
|
|
345
|
-
|
|
346
|
-
const observe = panoptic({
|
|
347
|
-
exporters: [consoleExporter(), otelExporter({ tracerName: "my-app" })],
|
|
348
|
-
});
|
|
349
|
-
```
|
|
350
|
-
|
|
351
|
-
## Three feed paths
|
|
352
|
-
|
|
353
|
-
| Path | Use when | API |
|
|
354
|
-
|---|---|---|
|
|
355
|
-
| **Events** | You want every run captured automatically and the primitive emits a `*.completed` event (agent / workflow / supervisor). | `observe.attach(target)` |
|
|
356
|
-
| **Middleware** | The app already composes cross-cutting concerns through the agent middleware pipeline. | `panoptic().middleware()` |
|
|
357
|
-
| **Direct** | The primitive carries no result-bearing event (the orchestrator), or you captured a report out of band. | `observe.collect(report)` |
|
|
358
|
-
|
|
359
|
-
### 1. Attach to the event stream
|
|
360
|
-
|
|
361
|
-
`attach` subscribes to the terminal `*.completed` events and returns a **detach** function:
|
|
362
|
-
|
|
363
|
-
```ts
|
|
364
|
-
const agent = ai.agent({ model });
|
|
365
|
-
const detach = observe.attach(agent); // agent | workflow | supervisor
|
|
366
|
-
|
|
367
|
-
await agent.execute("Summarize this");
|
|
368
|
-
|
|
369
|
-
// ...later, on teardown:
|
|
370
|
-
detach();
|
|
371
|
-
await observe.shutdown();
|
|
372
|
-
```
|
|
373
|
-
|
|
374
|
-
By default Panoptic subscribes to `agent.completed`, `workflow.completed`, and `supervisor.completed` — the terminal events that carry the finalized `result.report`. These fire **once per run regardless of outcome** (the matching `*.error` fires first on failure, then `*.completed` still fires), so attaching captures completed, failed, and cancelled runs alike. Panoptic silently ignores the names a given target doesn't emit, so the same subscriber attaches to any of the three primitives.
|
|
375
|
-
|
|
376
|
-
Narrow or widen the surface with `completedEvents`:
|
|
377
|
-
|
|
378
|
-
```ts
|
|
379
|
-
panoptic({ exporters, completedEvents: ["agent.completed"] }); // agents only
|
|
380
|
-
```
|
|
381
|
-
|
|
382
|
-
### 2. Wire it as agent middleware
|
|
383
|
-
|
|
384
|
-
```ts
|
|
385
|
-
const observe = panoptic({ exporters: [langfuseExporter({ /* ... */ })] });
|
|
386
|
-
|
|
387
|
-
const agent = ai.agent({
|
|
388
|
-
model,
|
|
389
|
-
middleware: [observe.middleware()],
|
|
390
|
-
});
|
|
391
|
-
```
|
|
392
|
-
|
|
393
|
-
The middleware feeds the collector from the `execute`-level `after` (success) and `onError` (failure) hooks. It never returns a value, so it never mutates the agent's result. Register more than one Panoptic middleware on the same agent by passing a distinct `middlewareName`:
|
|
394
|
-
|
|
395
|
-
```ts
|
|
396
|
-
panoptic({ middlewareName: "panoptic-secondary" }).middleware();
|
|
397
|
-
```
|
|
398
|
-
|
|
399
|
-
### 3. Collect a report directly
|
|
400
|
-
|
|
401
|
-
The **orchestrator** emits `orchestrator.turn.*` events that carry only session identity — no result. Feed its report straight in:
|
|
402
|
-
|
|
403
|
-
```ts
|
|
404
|
-
const result = await orchestrator.execute(input, { sessionId });
|
|
405
|
-
await observe.collect(result.report);
|
|
406
|
-
```
|
|
407
|
-
|
|
408
|
-
`collect(report)` also covers any report captured out of band (e.g. replayed from a store).
|
|
409
|
-
|
|
410
|
-
## Failure isolation
|
|
411
|
-
|
|
412
|
-
An observability fault never escapes onto the AI run's hot path:
|
|
413
|
-
|
|
414
|
-
- The collector wraps every exporter invocation in `Promise.allSettled`, so a throwing exporter can't crash the run or block its siblings.
|
|
415
|
-
- The event handler and the middleware additionally swallow the `collect` rejection, so even a collector-level fault stays contained.
|
|
416
|
-
|
|
417
|
-
A broken backend degrades to *missing telemetry*, never a failed agent run.
|
|
418
|
-
|
|
419
|
-
## Lifecycle
|
|
420
|
-
|
|
421
|
-
```ts
|
|
422
|
-
await observe.flush(); // drain buffered exporters (between batches / checkpoints)
|
|
423
|
-
await observe.shutdown(); // flush + release every exporter (process teardown)
|
|
424
|
-
```
|
|
425
|
-
|
|
426
|
-
Call `shutdown()` on teardown so buffered exporters (`fileExporter` batches, the Langfuse client) send their last records before exit.
|
|
427
|
-
|
|
428
|
-
## Bring your own collector or store
|
|
429
|
-
|
|
430
|
-
The in-memory trace store doubles as an `ExporterContract`, so the simplest way to retain runs for later querying is to pass it as an exporter — it then receives every trace alongside your other sinks:
|
|
431
|
-
|
|
432
|
-
```ts
|
|
433
|
-
import { panoptic, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
|
|
434
|
-
|
|
435
|
-
const store = createInMemoryTraceStore(); // an ExporterContract + queryable store
|
|
436
|
-
const observe = panoptic({ exporters: [store, consoleExporter()] });
|
|
437
|
-
|
|
438
|
-
// store now receives every trace — query/aggregate it later:
|
|
439
|
-
store.query({ status: "failed" });
|
|
440
|
-
store.aggregate({ sessionId });
|
|
441
|
-
```
|
|
442
|
-
|
|
443
|
-
Pass a pre-built `collector` instead when you want to share one collector across several subscribers (each subscriber's `attach` / `middleware` feeds the same exporters):
|
|
444
|
-
|
|
445
|
-
```ts
|
|
446
|
-
import { panoptic, createCollector } from "@warlock.js/ai-panoptic";
|
|
447
|
-
|
|
448
|
-
const collector = createCollector().use(store);
|
|
449
|
-
const observe = panoptic({ collector, exporters: [consoleExporter()] });
|
|
450
|
-
```
|
|
451
|
-
|
|
452
|
-
Add exporters after construction with `observe.use(exporter)` (chainable, deduped by `ExporterContract.name`). `observe.toTrace(report)` projects a report into a `Trace` **without** dispatching — handy for inspection or tests.
|
|
453
|
-
|
|
454
|
-
> The store's full query / aggregate / capacity surface is its own skill — see [`query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md).
|
|
455
|
-
|
|
456
|
-
## See also
|
|
457
|
-
|
|
458
|
-
- [`@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md) — the zero-setup local viewer the `ai.config({ panoptic: { dashboard } })` switch starts over the collector's store.
|
|
459
|
-
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — the sink end: choosing/writing an exporter, the `gen_ai.*` / `warlock.*` attribute mapping.
|
|
460
|
-
- [`@warlock.js/ai-panoptic/query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md) — retaining traces in the queryable in-memory store and slicing them (`query` / `aggregate` / `capacity`).
|
|
461
|
-
- [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md) — running the agent and the `BaseReport` shape Panoptic consumes.
|
|
287
|
+
---
|
|
288
|
+
name: observe-with-panoptic
|
|
289
|
+
description: 'Wire @warlock.js/ai-panoptic into an agent/workflow/supervisor/orchestrator — declaratively via `ai.config({ panoptic })` (registers panoptic on core''s Observer seam) or with the one-call `panoptic({ exporters })` subscriber. Triggers: `panoptic`, `Panoptic`, `PanopticOptions`, `PanopticConfig`, `ai.config({ panoptic })`, `observeAll`, per-flow `observe`, `FlowObserveOption`, `registerObserver`, `Observer`, `.attach`, `.middleware`, `.collect`, `.toTrace`, `observe.attach(agent)`, `panoptic().middleware()`, `completedEvents`, `middlewareName`; ''observe a warlock agent'', ''trace an agent run'', ''attach observability to a workflow/supervisor'', ''observe every flow by default'', ''add a tracing middleware'', ''collect an orchestrator turn report'', ''wire OTel/Langfuse into my agent''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"` (or `import { panoptic } from "@warlock.js/ai-panoptic"`). Skip: the zero-setup local dashboard — `@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`; writing/choosing an exporter (the sink end) — `@warlock.js/ai-panoptic/export-traces/SKILL.md`; the core report shape / running the agent itself — `@warlock.js/ai/run-ai-agent/SKILL.md`.'
|
|
290
|
+
---
|
|
291
|
+
|
|
292
|
+
# Observe a run with Panoptic — the one-call subscriber
|
|
293
|
+
|
|
294
|
+
There are two levels. **Declarative** (`ai.config({ panoptic })`) is the high-level path: panoptic registers itself on core's generic `Observer` seam and every observed flow routes its report in automatically — no per-target wiring. **Imperative** (`panoptic(...)`) is the underlying subscriber, with three explicit feed paths. The declarative path is sugar over the imperative one; both reuse the same collection pipeline.
|
|
295
|
+
|
|
296
|
+
## Declarative — `ai.config({ panoptic })` + the Observer seam
|
|
297
|
+
|
|
298
|
+
The simplest wiring. Importing `@warlock.js/ai-panoptic` (even bare, for side effects) subscribes panoptic to core's config seam; a later `ai.config({ panoptic })` then builds the collector and **registers it once on core's `Observer` registry** via `registerObserver`. Core itself never imports panoptic — it depends only on the structural `Observer` interface (`{ collect(report) }`), the dependency inversion that keeps the two sides decoupled.
|
|
299
|
+
|
|
300
|
+
```ts
|
|
301
|
+
import { ai } from "@warlock.js/ai";
|
|
302
|
+
import "@warlock.js/ai-panoptic"; // side-effect wiring — do this once at boot
|
|
303
|
+
|
|
304
|
+
ai.config({
|
|
305
|
+
panoptic: {
|
|
306
|
+
exporters: [consoleExporter()], // optional; a store-shaped exporter also feeds the dashboard
|
|
307
|
+
observeAll: true, // observe every flow by default
|
|
308
|
+
dashboard: true, // optional local viewer — see use-local-dashboard
|
|
309
|
+
},
|
|
310
|
+
});
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
`PanopticConfig`: `exporters?` (the collector's sinks), `observeAll?` (default `false`), `dashboard?` (`boolean | DashboardOptions`). Applying it is **idempotent** — the observer registers once, the dashboard starts at most once; repeat `ai.config(...)` merges only refresh `observeAll`.
|
|
314
|
+
|
|
315
|
+
### `observeAll` + per-flow `observe`
|
|
316
|
+
|
|
317
|
+
Once a collector is registered, which flows actually route to it is controlled by core's `observe` seam:
|
|
318
|
+
|
|
319
|
+
| Setting | Effect |
|
|
320
|
+
|---|---|
|
|
321
|
+
| `observeAll: true` (config) | Every `ai.agent` / `ai.workflow` / `ai.supervisor` / `ai.team` that left `observe` unset routes its completed report to the registered observers. |
|
|
322
|
+
| `observe: true` (per-flow) | Force-observe **this** flow even when `observeAll` is off → the globally registered observers. |
|
|
323
|
+
| `observe: false` (per-flow) | Opt **this** flow out entirely, even when `observeAll` is on. |
|
|
324
|
+
| `observe: <Observer>` (per-flow) | A **flow-local** collector — only this flow's report, routed only to it (the global observers are skipped). A panoptic flow-local collector implements `Observer`, so pass it directly. |
|
|
325
|
+
| (unset, `observeAll: false`) | Not observed — opt-in default, identical to legacy behavior. |
|
|
326
|
+
|
|
327
|
+
```ts
|
|
328
|
+
ai.agent({ model, observe: true }); // always traced
|
|
329
|
+
ai.workflow({ name: "billing", steps, observe: true }); // always traced
|
|
330
|
+
ai.supervisor({ name: "triage", ..., observe: false }); // never traced, even under observeAll
|
|
331
|
+
|
|
332
|
+
const flowLocal = panoptic({ exporters: [fileExporter({ path: "billing.jsonl" })] });
|
|
333
|
+
ai.team({ name: "research", ..., observe: flowLocal }); // only this flow, only this collector
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
The flow awaits each observer's `collect`, and a throwing observer is **swallowed** — observability never breaks the run (mirrors `onUsage` / `onComplete`). `observe` lives on **all four** flow configs — `ai.agent`, `ai.workflow`, `ai.supervisor`, and `ai.team` (the agent additionally supports the imperative `attach()` / `middleware()` paths below).
|
|
337
|
+
|
|
338
|
+
## Imperative — `panoptic(...)`, the one-call subscriber
|
|
339
|
+
|
|
340
|
+
`panoptic(...)` builds a collector, registers your exporters on it, and hands back a subscriber with **three ways to feed traces** in. All three converge on the same collector, so each run reaches every exporter exactly once. Use it when you want explicit control, a flow-local collector, or to wire a target the declarative seam doesn't cover (the orchestrator).
|
|
341
|
+
|
|
342
|
+
```ts
|
|
343
|
+
import { panoptic } from "@warlock.js/ai-panoptic";
|
|
344
|
+
import { consoleExporter, otelExporter } from "@warlock.js/ai-panoptic";
|
|
345
|
+
|
|
346
|
+
const observe = panoptic({
|
|
347
|
+
exporters: [consoleExporter(), otelExporter({ tracerName: "my-app" })],
|
|
348
|
+
});
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
## Three feed paths
|
|
352
|
+
|
|
353
|
+
| Path | Use when | API |
|
|
354
|
+
|---|---|---|
|
|
355
|
+
| **Events** | You want every run captured automatically and the primitive emits a `*.completed` event (agent / workflow / supervisor). | `observe.attach(target)` |
|
|
356
|
+
| **Middleware** | The app already composes cross-cutting concerns through the agent middleware pipeline. | `panoptic().middleware()` |
|
|
357
|
+
| **Direct** | The primitive carries no result-bearing event (the orchestrator), or you captured a report out of band. | `observe.collect(report)` |
|
|
358
|
+
|
|
359
|
+
### 1. Attach to the event stream
|
|
360
|
+
|
|
361
|
+
`attach` subscribes to the terminal `*.completed` events and returns a **detach** function:
|
|
362
|
+
|
|
363
|
+
```ts
|
|
364
|
+
const agent = ai.agent({ model });
|
|
365
|
+
const detach = observe.attach(agent); // agent | workflow | supervisor
|
|
366
|
+
|
|
367
|
+
await agent.execute("Summarize this");
|
|
368
|
+
|
|
369
|
+
// ...later, on teardown:
|
|
370
|
+
detach();
|
|
371
|
+
await observe.shutdown();
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
By default Panoptic subscribes to `agent.completed`, `workflow.completed`, and `supervisor.completed` — the terminal events that carry the finalized `result.report`. These fire **once per run regardless of outcome** (the matching `*.error` fires first on failure, then `*.completed` still fires), so attaching captures completed, failed, and cancelled runs alike. Panoptic silently ignores the names a given target doesn't emit, so the same subscriber attaches to any of the three primitives.
|
|
375
|
+
|
|
376
|
+
Narrow or widen the surface with `completedEvents`:
|
|
377
|
+
|
|
378
|
+
```ts
|
|
379
|
+
panoptic({ exporters, completedEvents: ["agent.completed"] }); // agents only
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
### 2. Wire it as agent middleware
|
|
383
|
+
|
|
384
|
+
```ts
|
|
385
|
+
const observe = panoptic({ exporters: [langfuseExporter({ /* ... */ })] });
|
|
386
|
+
|
|
387
|
+
const agent = ai.agent({
|
|
388
|
+
model,
|
|
389
|
+
middleware: [observe.middleware()],
|
|
390
|
+
});
|
|
391
|
+
```
|
|
392
|
+
|
|
393
|
+
The middleware feeds the collector from the `execute`-level `after` (success) and `onError` (failure) hooks. It never returns a value, so it never mutates the agent's result. Register more than one Panoptic middleware on the same agent by passing a distinct `middlewareName`:
|
|
394
|
+
|
|
395
|
+
```ts
|
|
396
|
+
panoptic({ middlewareName: "panoptic-secondary" }).middleware();
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
### 3. Collect a report directly
|
|
400
|
+
|
|
401
|
+
The **orchestrator** emits `orchestrator.turn.*` events that carry only session identity — no result. Feed its report straight in:
|
|
402
|
+
|
|
403
|
+
```ts
|
|
404
|
+
const result = await orchestrator.execute(input, { sessionId });
|
|
405
|
+
await observe.collect(result.report);
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
`collect(report)` also covers any report captured out of band (e.g. replayed from a store).
|
|
409
|
+
|
|
410
|
+
## Failure isolation
|
|
411
|
+
|
|
412
|
+
An observability fault never escapes onto the AI run's hot path:
|
|
413
|
+
|
|
414
|
+
- The collector wraps every exporter invocation in `Promise.allSettled`, so a throwing exporter can't crash the run or block its siblings.
|
|
415
|
+
- The event handler and the middleware additionally swallow the `collect` rejection, so even a collector-level fault stays contained.
|
|
416
|
+
|
|
417
|
+
A broken backend degrades to *missing telemetry*, never a failed agent run.
|
|
418
|
+
|
|
419
|
+
## Lifecycle
|
|
420
|
+
|
|
421
|
+
```ts
|
|
422
|
+
await observe.flush(); // drain buffered exporters (between batches / checkpoints)
|
|
423
|
+
await observe.shutdown(); // flush + release every exporter (process teardown)
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
Call `shutdown()` on teardown so buffered exporters (`fileExporter` batches, the Langfuse client) send their last records before exit.
|
|
427
|
+
|
|
428
|
+
## Bring your own collector or store
|
|
429
|
+
|
|
430
|
+
The in-memory trace store doubles as an `ExporterContract`, so the simplest way to retain runs for later querying is to pass it as an exporter — it then receives every trace alongside your other sinks:
|
|
431
|
+
|
|
432
|
+
```ts
|
|
433
|
+
import { panoptic, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
|
|
434
|
+
|
|
435
|
+
const store = createInMemoryTraceStore(); // an ExporterContract + queryable store
|
|
436
|
+
const observe = panoptic({ exporters: [store, consoleExporter()] });
|
|
437
|
+
|
|
438
|
+
// store now receives every trace — query/aggregate it later:
|
|
439
|
+
store.query({ status: "failed" });
|
|
440
|
+
store.aggregate({ sessionId });
|
|
441
|
+
```
|
|
442
|
+
|
|
443
|
+
Pass a pre-built `collector` instead when you want to share one collector across several subscribers (each subscriber's `attach` / `middleware` feeds the same exporters):
|
|
444
|
+
|
|
445
|
+
```ts
|
|
446
|
+
import { panoptic, createCollector } from "@warlock.js/ai-panoptic";
|
|
447
|
+
|
|
448
|
+
const collector = createCollector().use(store);
|
|
449
|
+
const observe = panoptic({ collector, exporters: [consoleExporter()] });
|
|
450
|
+
```
|
|
451
|
+
|
|
452
|
+
Add exporters after construction with `observe.use(exporter)` (chainable, deduped by `ExporterContract.name`). `observe.toTrace(report)` projects a report into a `Trace` **without** dispatching — handy for inspection or tests.
|
|
453
|
+
|
|
454
|
+
> The store's full query / aggregate / capacity surface is its own skill — see [`query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md).
|
|
455
|
+
|
|
456
|
+
## See also
|
|
457
|
+
|
|
458
|
+
- [`@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md) — the zero-setup local viewer the `ai.config({ panoptic: { dashboard } })` switch starts over the collector's store.
|
|
459
|
+
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — the sink end: choosing/writing an exporter, the `gen_ai.*` / `warlock.*` attribute mapping.
|
|
460
|
+
- [`@warlock.js/ai-panoptic/query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md) — retaining traces in the queryable in-memory store and slicing them (`query` / `aggregate` / `capacity`).
|
|
461
|
+
- [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md) — running the agent and the `BaseReport` shape Panoptic consumes.
|
|
462
462
|
|
|
463
463
|
|
|
464
464
|
## query-traces `@warlock.js/ai-panoptic/query-traces/SKILL.md`
|
|
465
465
|
|
|
466
|
-
---
|
|
467
|
-
name: query-traces
|
|
468
|
-
description: 'Retain @warlock.js/ai-panoptic traces in a queryable store — in-memory, or cache-backed so they survive a process restart — and slice them after the fact: get one run by id, list a session, filter failed runs in a window, roll up usage + cost. Triggers: `createInMemoryTraceStore`, `createCacheTraceStore`, `CacheTraceStoreHandle`, `CacheTraceStoreOptions`, `CacheDriverInput`, `store.ready`, `TraceStoreContract`, `TraceQuery`, `TraceAggregate`, `InMemoryTraceStoreOptions`, `store.query`, `store.aggregate`, `store.get`, `store.add`, `store.clear`, `store.size`, `capacity`, `prefix`, `onError`, `sumUsage`, `emptyUsage`, `matchTrace`; ''query collected traces'', ''aggregate AI cost per session'', ''how much did this session spend'', ''list failed runs'', ''retain traces in memory'', ''persist traces across a restart'', ''cache-backed trace store'', ''Redis trace store'', ''roll up token usage''; typical import `import { createInMemoryTraceStore, createCacheTraceStore } from "@warlock.js/ai-panoptic"`. Skip: sending traces to an external backend (OTel/Langfuse/console/file) — `@warlock.js/ai-panoptic/export-traces/SKILL.md`; wiring the subscriber into a run — `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`.'
|
|
469
|
-
---
|
|
470
|
-
|
|
471
|
-
# Query traces — the in-memory queryable store
|
|
472
|
-
|
|
473
|
-
The collector fans every completed `Trace` out to its exporters. A **trace store** is one such sink that *retains* traces so you can answer observability questions after the fact: pull one run by id, list every run for a session, slice failed runs in a time window, or roll usage + cost across any of those slices.
|
|
474
|
-
|
|
475
|
-
`createInMemoryTraceStore()` returns an object that satisfies **both** `TraceStoreContract` and `ExporterContract` — so wiring it into a collector is a single `collector.use(store)` and the store fills as traces complete.
|
|
476
|
-
|
|
477
|
-
```ts
|
|
478
|
-
import { panoptic, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
|
|
479
|
-
|
|
480
|
-
const store = createInMemoryTraceStore();
|
|
481
|
-
|
|
482
|
-
// The store doubles as an exporter — register it like any other sink.
|
|
483
|
-
const observe = panoptic({ exporters: [store] });
|
|
484
|
-
|
|
485
|
-
const agent = ai.agent({ model });
|
|
486
|
-
observe.attach(agent);
|
|
487
|
-
|
|
488
|
-
await agent.execute("Summarize this");
|
|
489
|
-
|
|
490
|
-
// ...later, off the hot path:
|
|
491
|
-
const failed = store.query({ status: "failed" });
|
|
492
|
-
const spend = store.aggregate({ sessionId: "session-42" });
|
|
493
|
-
```
|
|
494
|
-
|
|
495
|
-
> The store is also a valid `collector.use(...)` target directly — `panoptic({ collector: createCollector().use(store) })` — see [`observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) for bring-your-own-collector wiring.
|
|
496
|
-
|
|
497
|
-
## The store contract
|
|
498
|
-
|
|
499
|
-
`TraceStoreContract` is the queryable surface; the in-memory implementation adds the `ExporterContract` `name` + `export` so it can be a sink.
|
|
500
|
-
|
|
501
|
-
```ts
|
|
502
|
-
interface TraceStoreContract {
|
|
503
|
-
add(trace: Trace): void; // ingest; overwrites same traceId
|
|
504
|
-
get(traceId: string): Trace | undefined; // one run by id
|
|
505
|
-
query(filter?: TraceQuery): Trace[]; // matching traces, newest-started first
|
|
506
|
-
aggregate(filter?: TraceQuery): TraceAggregate; // usage + cost + status rollup
|
|
507
|
-
readonly size: number; // retained count
|
|
508
|
-
clear(): void; // drop everything
|
|
509
|
-
}
|
|
510
|
-
```
|
|
511
|
-
|
|
512
|
-
- **`add` overwrites** any existing trace with the same `traceId` — re-collecting a run replaces rather than duplicates it.
|
|
513
|
-
- **`export(trace)` is an alias for `add`** — that is what makes the store a drop-in exporter (`store.name === "in-memory-trace-store"`).
|
|
514
|
-
- **`query` returns newest-started first** (sorted by root `startedAt`, descending). An empty / omitted filter returns every retained trace.
|
|
515
|
-
|
|
516
|
-
## Filtering with `TraceQuery`
|
|
517
|
-
|
|
518
|
-
Every field is optional and **ANDed** together — an absent field is "don't care", so `{}` matches everything. The same filter type drives both `query` and `aggregate`.
|
|
519
|
-
|
|
520
|
-
```ts
|
|
521
|
-
type TraceQuery = {
|
|
522
|
-
traceId?: string; // exact one run
|
|
523
|
-
sessionId?: string; // every run for a session
|
|
524
|
-
status?: ReportStatus | ReportStatus[]; // single status or membership set
|
|
525
|
-
startedAfter?: string | Date; // inclusive lower bound on root startedAt
|
|
526
|
-
startedBefore?: string | Date; // inclusive upper bound on root startedAt
|
|
527
|
-
};
|
|
528
|
-
```
|
|
529
|
-
|
|
530
|
-
```ts
|
|
531
|
-
// One run by id:
|
|
532
|
-
store.query({ traceId: "run-abc" });
|
|
533
|
-
|
|
534
|
-
// Every run for a conversation:
|
|
535
|
-
store.query({ sessionId: "session-42" });
|
|
536
|
-
|
|
537
|
-
// Failed or cancelled runs since midnight (status accepts an array):
|
|
538
|
-
store.query({
|
|
539
|
-
status: ["failed", "cancelled"],
|
|
540
|
-
startedAfter: "2026-06-18T00:00:00.000Z",
|
|
541
|
-
});
|
|
542
|
-
|
|
543
|
-
// A Date works anywhere a time bound is accepted:
|
|
544
|
-
store.query({ startedAfter: new Date(Date.now() - 3_600_000) }); // last hour
|
|
545
|
-
```
|
|
546
|
-
|
|
547
|
-
Time bounds compare against each trace's root `startedAt`, **inclusive on both ends**, accepting either an ISO-8601 string or a `Date`. Status is exact equality (or array membership). The pure matcher behind this is `matchTrace(trace, filter)` — exported for testing or filtering a trace list you hold yourself.
|
|
548
|
-
|
|
549
|
-
## Rolling up with `aggregate`
|
|
550
|
-
|
|
551
|
-
`aggregate(filter?)` answers the headline questions — how many runs, how many tokens, how much did it cost, how many failed — for whatever slice the filter selects:
|
|
552
|
-
|
|
553
|
-
```ts
|
|
554
|
-
type TraceAggregate = {
|
|
555
|
-
traces: number; // matched count
|
|
556
|
-
completed: number; // root status === "completed"
|
|
557
|
-
failed: number; // root status === "failed"
|
|
558
|
-
cancelled: number; // root status === "cancelled"
|
|
559
|
-
usage: Usage; // summed token usage (input/output/total always present)
|
|
560
|
-
cost?: ModelPricing; // merged per-channel USD, or undefined if no trace was priced
|
|
561
|
-
totalDuration: number; // summed wall-clock ms
|
|
562
|
-
};
|
|
563
|
-
```
|
|
564
|
-
|
|
565
|
-
```ts
|
|
566
|
-
const stats = store.aggregate({ sessionId: "session-42" });
|
|
567
|
-
|
|
568
|
-
console.log(`${stats.traces} runs, ${stats.usage.total} tokens, ${stats.failed} failed`);
|
|
569
|
-
|
|
570
|
-
const totalUsd =
|
|
571
|
-
(stats.cost?.input ?? 0) +
|
|
572
|
-
(stats.cost?.output ?? 0) +
|
|
573
|
-
(stats.cost?.cachedInput ?? 0) +
|
|
574
|
-
(stats.cost?.cachedOutput ?? 0);
|
|
575
|
-
```
|
|
576
|
-
|
|
577
|
-
Notes that matter when reading the numbers:
|
|
578
|
-
|
|
579
|
-
- **`usage`** sums each trace's *root* `Usage` (already a rollup of own cost + children), so the totals reflect the whole run tree without re-walking spans. `input` / `output` / `total` are always present (zero when nothing matched); the optional `cachedTokens` / `cacheWriteTokens` / `reasoningTokens` channels are summed only when at least one matched trace reported them.
|
|
580
|
-
- **`cost`** is a per-channel `ModelPricing` breakdown (not one opaque number), merged with the framework's own cost-rollup logic — an unpriced trace never erases the cost of priced ones. It stays `undefined` only when **no** matched trace carried pricing.
|
|
581
|
-
- **`completed` + `failed` + `cancelled` need not equal `traces`** — non-terminal statuses (`awaiting-input`, `max-iterations`) count toward `traces` but none of the three headline counters.
|
|
582
|
-
- An empty match returns a zeroed aggregate (`usage` all `0`, `cost` `undefined`, `totalDuration` `0`).
|
|
583
|
-
|
|
584
|
-
## Bounding retention with `capacity`
|
|
585
|
-
|
|
586
|
-
Unbounded by default (keep everything until `clear()`). For a long-lived process, cap it — the oldest-ingested trace is evicted FIFO once the cap is exceeded:
|
|
587
|
-
|
|
588
|
-
```ts
|
|
589
|
-
type InMemoryTraceStoreOptions = { capacity?: number }; // absent / 0 = unbounded
|
|
590
|
-
|
|
591
|
-
const store = createInMemoryTraceStore({ capacity: 1000 });
|
|
592
|
-
```
|
|
593
|
-
|
|
594
|
-
Overwriting an existing `traceId` refreshes its insertion position, so a re-collected run is treated as newest for eviction — eviction stays honest under overwrite.
|
|
595
|
-
|
|
596
|
-
> The in-memory store has no secondary indexes: `get` / `add` are O(1), `query` / `aggregate` are O(n) scans. That is the right trade-off for dev/test and modest-volume runtime use. For high-volume or durable retention, write a custom `TraceStoreContract` (or export to a backend with [`export-traces`](@warlock.js/ai-panoptic/export-traces/SKILL.md)) backed by a real datastore.
|
|
597
|
-
|
|
598
|
-
## Persisting across a restart — `createCacheTraceStore`
|
|
599
|
-
|
|
600
|
-
The in-memory store loses everything when the process exits. `createCacheTraceStore(cache, options?)` is a **durable** `TraceStoreContract` (and `ExporterContract`) backed by any `@warlock.js/cache` `CacheDriver` — Redis, file, Postgres, … — so traces survive a restart and the dashboard re-opens with history intact.
|
|
601
|
-
|
|
602
|
-
```ts
|
|
603
|
-
import { createCacheTraceStore } from "@warlock.js/ai-panoptic";
|
|
604
|
-
import { RedisCacheDriver } from "@warlock.js/cache";
|
|
605
|
-
|
|
606
|
-
// A driver instance, or a (possibly async) FACTORY resolved lazily on first
|
|
607
|
-
// use — the factory form defers the Redis connect until the first trace is
|
|
608
|
-
// collected, so module import stays connection-free.
|
|
609
|
-
const store = createCacheTraceStore(async () => {
|
|
610
|
-
const driver = new RedisCacheDriver();
|
|
611
|
-
await driver.connect();
|
|
612
|
-
return driver;
|
|
613
|
-
});
|
|
614
|
-
|
|
615
|
-
await store.ready(); // hydrate the in-memory mirror from a prior run
|
|
616
|
-
collector.use(store); // fills as traces complete; reads stay synchronous
|
|
617
|
-
|
|
618
|
-
const failed = store.query({ status: "failed" }); // queryable like any store
|
|
619
|
-
```
|
|
620
|
-
|
|
621
|
-
**Write-through over a sync contract.** The store keeps the same insertion-ordered in-memory **mirror** the in-memory store uses and serves every read (`get` / `query` / `aggregate` / `size`) from it synchronously — the dashboard polls these and can't await a cache. Writes go **through**: `add` updates the mirror immediately, then *fire-and-forget* persists the trace + a newest-first index to the cache. So:
|
|
622
|
-
|
|
623
|
-
- **Reads never wait on the cache** — the mirror is the runtime source of truth; the cache is the durable backing store.
|
|
624
|
-
- **A flaky cache never throws into the hot path** — a rejected write is routed to the optional `onError` hook and swallowed; the trace stays visible in the mirror for the life of the process, it just won't survive a restart.
|
|
625
|
-
- **`ready()` re-hydrates** the mirror from the persisted index on startup (await it once — `ai.config({ panoptic: { cache } })` does this for you before serving the dashboard).
|
|
626
|
-
|
|
627
|
-
```ts
|
|
628
|
-
type CacheDriverInput =
|
|
629
|
-
| CacheDriver // a live driver
|
|
630
|
-
| (() => CacheDriver | Promise<CacheDriver>); // lazy (sync or async) factory
|
|
631
|
-
|
|
632
|
-
type CacheTraceStoreOptions = {
|
|
633
|
-
prefix?: string; // key namespace; per-trace keys `${prefix}:trace:${id}`,
|
|
634
|
-
// index at `${prefix}:index`. Default "panoptic".
|
|
635
|
-
capacity?: number; // FIFO cap (oldest-ingested evicted from cache + mirror);
|
|
636
|
-
// absent / 0 = unbounded.
|
|
637
|
-
onError?: (error: unknown) => void; // swallowed cache-write failures land here.
|
|
638
|
-
};
|
|
639
|
-
|
|
640
|
-
type CacheTraceStoreHandle = TraceStoreContract & ExporterContract & {
|
|
641
|
-
ready(): Promise<void>; // hydrate the mirror from the cache on startup
|
|
642
|
-
};
|
|
643
|
-
```
|
|
644
|
-
|
|
645
|
-
Ordering uses an **internal monotonic counter**, not `Date.now()`, so the persisted index stays a stable total order even when many traces land in the same millisecond — FIFO eviction stays honest across a restart.
|
|
646
|
-
|
|
647
|
-
> The usual path is declarative: `ai.config({ panoptic: { cache, dashboard: true } })` constructs this store, awaits `ready()`, and wires it under the dashboard for you — see [`use-local-dashboard/SKILL.md`](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md). Reach for `createCacheTraceStore` directly only when you manage the store / collector yourself.
|
|
648
|
-
|
|
649
|
-
## Summing usage yourself
|
|
650
|
-
|
|
651
|
-
`sumUsage` / `emptyUsage` are the pure folds `aggregate` is built on — reuse them to roll a `Usage` set the store didn't produce (e.g. a list of traces you filtered by hand):
|
|
652
|
-
|
|
653
|
-
```ts
|
|
654
|
-
import { emptyUsage, sumUsage } from "@warlock.js/ai-panoptic";
|
|
655
|
-
|
|
656
|
-
let total = emptyUsage(); // { input: 0, output: 0, total: 0 } — optional channels absent
|
|
657
|
-
for (const trace of store.query({ sessionId })) {
|
|
658
|
-
total = sumUsage(total, trace.usage);
|
|
659
|
-
}
|
|
660
|
-
```
|
|
661
|
-
|
|
662
|
-
`sumUsage` is pure (returns a fresh `Usage`, mutates neither argument): token channels always sum; the optional cache/reasoning channels sum only when a side reported them (so a never-metered channel stays absent rather than fabricating a `0`); `cost` merges with the framework's `accumulateCost` so an unpriced contributor never erases a priced one.
|
|
663
|
-
|
|
664
|
-
## See also
|
|
665
|
-
|
|
666
|
-
- [`@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) — wiring the `panoptic(...)` subscriber and bring-your-own-collector so the store fills as runs complete.
|
|
667
|
-
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — sending traces to an external backend instead of (or alongside) retaining them in memory; writing a custom sink.
|
|
668
|
-
- [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md) — the `BaseReport` the collector projects into the `Trace` this store retains.
|
|
466
|
+
---
|
|
467
|
+
name: query-traces
|
|
468
|
+
description: 'Retain @warlock.js/ai-panoptic traces in a queryable store — in-memory, or cache-backed so they survive a process restart — and slice them after the fact: get one run by id, list a session, filter failed runs in a window, roll up usage + cost. Triggers: `createInMemoryTraceStore`, `createCacheTraceStore`, `CacheTraceStoreHandle`, `CacheTraceStoreOptions`, `CacheDriverInput`, `store.ready`, `TraceStoreContract`, `TraceQuery`, `TraceAggregate`, `InMemoryTraceStoreOptions`, `store.query`, `store.aggregate`, `store.get`, `store.add`, `store.clear`, `store.size`, `capacity`, `prefix`, `onError`, `sumUsage`, `emptyUsage`, `matchTrace`; ''query collected traces'', ''aggregate AI cost per session'', ''how much did this session spend'', ''list failed runs'', ''retain traces in memory'', ''persist traces across a restart'', ''cache-backed trace store'', ''Redis trace store'', ''roll up token usage''; typical import `import { createInMemoryTraceStore, createCacheTraceStore } from "@warlock.js/ai-panoptic"`. Skip: sending traces to an external backend (OTel/Langfuse/console/file) — `@warlock.js/ai-panoptic/export-traces/SKILL.md`; wiring the subscriber into a run — `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`.'
|
|
469
|
+
---
|
|
470
|
+
|
|
471
|
+
# Query traces — the in-memory queryable store
|
|
472
|
+
|
|
473
|
+
The collector fans every completed `Trace` out to its exporters. A **trace store** is one such sink that *retains* traces so you can answer observability questions after the fact: pull one run by id, list every run for a session, slice failed runs in a time window, or roll usage + cost across any of those slices.
|
|
474
|
+
|
|
475
|
+
`createInMemoryTraceStore()` returns an object that satisfies **both** `TraceStoreContract` and `ExporterContract` — so wiring it into a collector is a single `collector.use(store)` and the store fills as traces complete.
|
|
476
|
+
|
|
477
|
+
```ts
|
|
478
|
+
import { panoptic, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
|
|
479
|
+
|
|
480
|
+
const store = createInMemoryTraceStore();
|
|
481
|
+
|
|
482
|
+
// The store doubles as an exporter — register it like any other sink.
|
|
483
|
+
const observe = panoptic({ exporters: [store] });
|
|
484
|
+
|
|
485
|
+
const agent = ai.agent({ model });
|
|
486
|
+
observe.attach(agent);
|
|
487
|
+
|
|
488
|
+
await agent.execute("Summarize this");
|
|
489
|
+
|
|
490
|
+
// ...later, off the hot path:
|
|
491
|
+
const failed = store.query({ status: "failed" });
|
|
492
|
+
const spend = store.aggregate({ sessionId: "session-42" });
|
|
493
|
+
```
|
|
494
|
+
|
|
495
|
+
> The store is also a valid `collector.use(...)` target directly — `panoptic({ collector: createCollector().use(store) })` — see [`observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) for bring-your-own-collector wiring.
|
|
496
|
+
|
|
497
|
+
## The store contract
|
|
498
|
+
|
|
499
|
+
`TraceStoreContract` is the queryable surface; the in-memory implementation adds the `ExporterContract` `name` + `export` so it can be a sink.
|
|
500
|
+
|
|
501
|
+
```ts
|
|
502
|
+
interface TraceStoreContract {
|
|
503
|
+
add(trace: Trace): void; // ingest; overwrites same traceId
|
|
504
|
+
get(traceId: string): Trace | undefined; // one run by id
|
|
505
|
+
query(filter?: TraceQuery): Trace[]; // matching traces, newest-started first
|
|
506
|
+
aggregate(filter?: TraceQuery): TraceAggregate; // usage + cost + status rollup
|
|
507
|
+
readonly size: number; // retained count
|
|
508
|
+
clear(): void; // drop everything
|
|
509
|
+
}
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
- **`add` overwrites** any existing trace with the same `traceId` — re-collecting a run replaces rather than duplicates it.
|
|
513
|
+
- **`export(trace)` is an alias for `add`** — that is what makes the store a drop-in exporter (`store.name === "in-memory-trace-store"`).
|
|
514
|
+
- **`query` returns newest-started first** (sorted by root `startedAt`, descending). An empty / omitted filter returns every retained trace.
|
|
515
|
+
|
|
516
|
+
## Filtering with `TraceQuery`
|
|
517
|
+
|
|
518
|
+
Every field is optional and **ANDed** together — an absent field is "don't care", so `{}` matches everything. The same filter type drives both `query` and `aggregate`.
|
|
519
|
+
|
|
520
|
+
```ts
|
|
521
|
+
type TraceQuery = {
|
|
522
|
+
traceId?: string; // exact one run
|
|
523
|
+
sessionId?: string; // every run for a session
|
|
524
|
+
status?: ReportStatus | ReportStatus[]; // single status or membership set
|
|
525
|
+
startedAfter?: string | Date; // inclusive lower bound on root startedAt
|
|
526
|
+
startedBefore?: string | Date; // inclusive upper bound on root startedAt
|
|
527
|
+
};
|
|
528
|
+
```
|
|
529
|
+
|
|
530
|
+
```ts
|
|
531
|
+
// One run by id:
|
|
532
|
+
store.query({ traceId: "run-abc" });
|
|
533
|
+
|
|
534
|
+
// Every run for a conversation:
|
|
535
|
+
store.query({ sessionId: "session-42" });
|
|
536
|
+
|
|
537
|
+
// Failed or cancelled runs since midnight (status accepts an array):
|
|
538
|
+
store.query({
|
|
539
|
+
status: ["failed", "cancelled"],
|
|
540
|
+
startedAfter: "2026-06-18T00:00:00.000Z",
|
|
541
|
+
});
|
|
542
|
+
|
|
543
|
+
// A Date works anywhere a time bound is accepted:
|
|
544
|
+
store.query({ startedAfter: new Date(Date.now() - 3_600_000) }); // last hour
|
|
545
|
+
```
|
|
546
|
+
|
|
547
|
+
Time bounds compare against each trace's root `startedAt`, **inclusive on both ends**, accepting either an ISO-8601 string or a `Date`. Status is exact equality (or array membership). The pure matcher behind this is `matchTrace(trace, filter)` — exported for testing or filtering a trace list you hold yourself.
|
|
548
|
+
|
|
549
|
+
## Rolling up with `aggregate`
|
|
550
|
+
|
|
551
|
+
`aggregate(filter?)` answers the headline questions — how many runs, how many tokens, how much did it cost, how many failed — for whatever slice the filter selects:
|
|
552
|
+
|
|
553
|
+
```ts
|
|
554
|
+
type TraceAggregate = {
|
|
555
|
+
traces: number; // matched count
|
|
556
|
+
completed: number; // root status === "completed"
|
|
557
|
+
failed: number; // root status === "failed"
|
|
558
|
+
cancelled: number; // root status === "cancelled"
|
|
559
|
+
usage: Usage; // summed token usage (input/output/total always present)
|
|
560
|
+
cost?: ModelPricing; // merged per-channel USD, or undefined if no trace was priced
|
|
561
|
+
totalDuration: number; // summed wall-clock ms
|
|
562
|
+
};
|
|
563
|
+
```
|
|
564
|
+
|
|
565
|
+
```ts
|
|
566
|
+
const stats = store.aggregate({ sessionId: "session-42" });
|
|
567
|
+
|
|
568
|
+
console.log(`${stats.traces} runs, ${stats.usage.total} tokens, ${stats.failed} failed`);
|
|
569
|
+
|
|
570
|
+
const totalUsd =
|
|
571
|
+
(stats.cost?.input ?? 0) +
|
|
572
|
+
(stats.cost?.output ?? 0) +
|
|
573
|
+
(stats.cost?.cachedInput ?? 0) +
|
|
574
|
+
(stats.cost?.cachedOutput ?? 0);
|
|
575
|
+
```
|
|
576
|
+
|
|
577
|
+
Notes that matter when reading the numbers:
|
|
578
|
+
|
|
579
|
+
- **`usage`** sums each trace's *root* `Usage` (already a rollup of own cost + children), so the totals reflect the whole run tree without re-walking spans. `input` / `output` / `total` are always present (zero when nothing matched); the optional `cachedTokens` / `cacheWriteTokens` / `reasoningTokens` channels are summed only when at least one matched trace reported them.
|
|
580
|
+
- **`cost`** is a per-channel `ModelPricing` breakdown (not one opaque number), merged with the framework's own cost-rollup logic — an unpriced trace never erases the cost of priced ones. It stays `undefined` only when **no** matched trace carried pricing.
|
|
581
|
+
- **`completed` + `failed` + `cancelled` need not equal `traces`** — non-terminal statuses (`awaiting-input`, `max-iterations`) count toward `traces` but none of the three headline counters.
|
|
582
|
+
- An empty match returns a zeroed aggregate (`usage` all `0`, `cost` `undefined`, `totalDuration` `0`).
|
|
583
|
+
|
|
584
|
+
## Bounding retention with `capacity`
|
|
585
|
+
|
|
586
|
+
Unbounded by default (keep everything until `clear()`). For a long-lived process, cap it — the oldest-ingested trace is evicted FIFO once the cap is exceeded:
|
|
587
|
+
|
|
588
|
+
```ts
|
|
589
|
+
type InMemoryTraceStoreOptions = { capacity?: number }; // absent / 0 = unbounded
|
|
590
|
+
|
|
591
|
+
const store = createInMemoryTraceStore({ capacity: 1000 });
|
|
592
|
+
```
|
|
593
|
+
|
|
594
|
+
Overwriting an existing `traceId` refreshes its insertion position, so a re-collected run is treated as newest for eviction — eviction stays honest under overwrite.
|
|
595
|
+
|
|
596
|
+
> The in-memory store has no secondary indexes: `get` / `add` are O(1), `query` / `aggregate` are O(n) scans. That is the right trade-off for dev/test and modest-volume runtime use. For high-volume or durable retention, write a custom `TraceStoreContract` (or export to a backend with [`export-traces`](@warlock.js/ai-panoptic/export-traces/SKILL.md)) backed by a real datastore.
|
|
597
|
+
|
|
598
|
+
## Persisting across a restart — `createCacheTraceStore`
|
|
599
|
+
|
|
600
|
+
The in-memory store loses everything when the process exits. `createCacheTraceStore(cache, options?)` is a **durable** `TraceStoreContract` (and `ExporterContract`) backed by any `@warlock.js/cache` `CacheDriver` — Redis, file, Postgres, … — so traces survive a restart and the dashboard re-opens with history intact.
|
|
601
|
+
|
|
602
|
+
```ts
|
|
603
|
+
import { createCacheTraceStore } from "@warlock.js/ai-panoptic";
|
|
604
|
+
import { RedisCacheDriver } from "@warlock.js/cache";
|
|
605
|
+
|
|
606
|
+
// A driver instance, or a (possibly async) FACTORY resolved lazily on first
|
|
607
|
+
// use — the factory form defers the Redis connect until the first trace is
|
|
608
|
+
// collected, so module import stays connection-free.
|
|
609
|
+
const store = createCacheTraceStore(async () => {
|
|
610
|
+
const driver = new RedisCacheDriver();
|
|
611
|
+
await driver.connect();
|
|
612
|
+
return driver;
|
|
613
|
+
});
|
|
614
|
+
|
|
615
|
+
await store.ready(); // hydrate the in-memory mirror from a prior run
|
|
616
|
+
collector.use(store); // fills as traces complete; reads stay synchronous
|
|
617
|
+
|
|
618
|
+
const failed = store.query({ status: "failed" }); // queryable like any store
|
|
619
|
+
```
|
|
620
|
+
|
|
621
|
+
**Write-through over a sync contract.** The store keeps the same insertion-ordered in-memory **mirror** the in-memory store uses and serves every read (`get` / `query` / `aggregate` / `size`) from it synchronously — the dashboard polls these and can't await a cache. Writes go **through**: `add` updates the mirror immediately, then *fire-and-forget* persists the trace + a newest-first index to the cache. So:
|
|
622
|
+
|
|
623
|
+
- **Reads never wait on the cache** — the mirror is the runtime source of truth; the cache is the durable backing store.
|
|
624
|
+
- **A flaky cache never throws into the hot path** — a rejected write is routed to the optional `onError` hook and swallowed; the trace stays visible in the mirror for the life of the process, it just won't survive a restart.
|
|
625
|
+
- **`ready()` re-hydrates** the mirror from the persisted index on startup (await it once — `ai.config({ panoptic: { cache } })` does this for you before serving the dashboard).
|
|
626
|
+
|
|
627
|
+
```ts
|
|
628
|
+
type CacheDriverInput =
|
|
629
|
+
| CacheDriver // a live driver
|
|
630
|
+
| (() => CacheDriver | Promise<CacheDriver>); // lazy (sync or async) factory
|
|
631
|
+
|
|
632
|
+
type CacheTraceStoreOptions = {
|
|
633
|
+
prefix?: string; // key namespace; per-trace keys `${prefix}:trace:${id}`,
|
|
634
|
+
// index at `${prefix}:index`. Default "panoptic".
|
|
635
|
+
capacity?: number; // FIFO cap (oldest-ingested evicted from cache + mirror);
|
|
636
|
+
// absent / 0 = unbounded.
|
|
637
|
+
onError?: (error: unknown) => void; // swallowed cache-write failures land here.
|
|
638
|
+
};
|
|
639
|
+
|
|
640
|
+
type CacheTraceStoreHandle = TraceStoreContract & ExporterContract & {
|
|
641
|
+
ready(): Promise<void>; // hydrate the mirror from the cache on startup
|
|
642
|
+
};
|
|
643
|
+
```
|
|
644
|
+
|
|
645
|
+
Ordering uses an **internal monotonic counter**, not `Date.now()`, so the persisted index stays a stable total order even when many traces land in the same millisecond — FIFO eviction stays honest across a restart.
|
|
646
|
+
|
|
647
|
+
> The usual path is declarative: `ai.config({ panoptic: { cache, dashboard: true } })` constructs this store, awaits `ready()`, and wires it under the dashboard for you — see [`use-local-dashboard/SKILL.md`](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md). Reach for `createCacheTraceStore` directly only when you manage the store / collector yourself.
|
|
648
|
+
|
|
649
|
+
## Summing usage yourself
|
|
650
|
+
|
|
651
|
+
`sumUsage` / `emptyUsage` are the pure folds `aggregate` is built on — reuse them to roll a `Usage` set the store didn't produce (e.g. a list of traces you filtered by hand):
|
|
652
|
+
|
|
653
|
+
```ts
|
|
654
|
+
import { emptyUsage, sumUsage } from "@warlock.js/ai-panoptic";
|
|
655
|
+
|
|
656
|
+
let total = emptyUsage(); // { input: 0, output: 0, total: 0 } — optional channels absent
|
|
657
|
+
for (const trace of store.query({ sessionId })) {
|
|
658
|
+
total = sumUsage(total, trace.usage);
|
|
659
|
+
}
|
|
660
|
+
```
|
|
661
|
+
|
|
662
|
+
`sumUsage` is pure (returns a fresh `Usage`, mutates neither argument): token channels always sum; the optional cache/reasoning channels sum only when a side reported them (so a never-metered channel stays absent rather than fabricating a `0`); `cost` merges with the framework's `accumulateCost` so an unpriced contributor never erases a priced one.
|
|
663
|
+
|
|
664
|
+
## See also
|
|
665
|
+
|
|
666
|
+
- [`@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) — wiring the `panoptic(...)` subscriber and bring-your-own-collector so the store fills as runs complete.
|
|
667
|
+
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — sending traces to an external backend instead of (or alongside) retaining them in memory; writing a custom sink.
|
|
668
|
+
- [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md) — the `BaseReport` the collector projects into the `Trace` this store retains.
|
|
669
669
|
|
|
670
670
|
|
|
671
671
|
## use-local-dashboard `@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`
|
|
672
672
|
|
|
673
|
-
---
|
|
674
|
-
name: use-local-dashboard
|
|
675
|
-
description: 'Run the zero-setup local Panoptic dashboard — a loopback `node:http` server over a trace store served from `ai.config({ panoptic: { dashboard } })`, with a two-pane call-tree + metadata UI (light/dark/system theme, search/filter, session + prompt-version grouping, cost heatmap, Gantt timeline, errors filter, hash deep-links) and an optional cache-backed persistent store. Triggers: `dashboard`, `DashboardOptions`, `DashboardHandle`, `ai.config({ panoptic })`, `panoptic.dashboard`, `panoptic.cache`, `port: 4319`, `basePath`, `open`, `authToken`, `allowedHosts`, `/api/traces`, `/api/aggregate`, `/api/traces/:id`, `createRequestHandler`, `dashboardHtml`, `sanitizeHref`, `ServeConfig`, `TraceStoreContract`, `createCacheTraceStore`, theme, dark mode, call tree, metadata panel, cost heatmap, timeline / Gantt, errors only, deep-link, group by session, prompt version, group by type, type stats panel, started / ended timestamps, team type; ''open a local AI trace dashboard'', ''view agent traces in the browser'', ''self-hosted panoptic UI'', ''inspect traces without Langfuse'', ''spin up a dev observability server'', ''dark mode dashboard'', ''persist traces across restart'', ''group traces by session / prompt version / type'', ''cost heatmap'', ''timeline / waterfall view'', ''filter failed runs in the dashboard'', ''per-type stats panel'', ''is the dashboard bearer token safe'', ''markdown links XSS in trace content''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"`. Skip: shipping traces to a backend (OTel / Langfuse / file) — `@warlock.js/ai-panoptic/export-traces/SKILL.md`; the queryable store API the dashboard reads — `@warlock.js/ai-panoptic/query-traces/SKILL.md`; wiring the subscriber by hand — `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`.'
|
|
676
|
-
---
|
|
677
|
-
|
|
678
|
-
# The local Panoptic dashboard — zero-setup trace viewing
|
|
679
|
-
|
|
680
|
-
A loopback-only `node:http` server that serves one self-contained HTML page polling a read-only JSON API over a trace store. No external backend and no build step; it binds `127.0.0.1` with no auth by default, and gains optional bearer-token auth + a Host allowlist (`authToken` / `allowedHosts`, **required** off-loopback). The store is the live object the collector fills, so each poll reflects the latest completed traces.
|
|
681
|
-
|
|
682
|
-
## The documented path — `ai.config({ panoptic: { dashboard } })`
|
|
683
|
-
|
|
684
|
-
You almost never call `dashboard(...)` directly. Set it declaratively and the side-effect import wires the store, the collector, and the server for you:
|
|
685
|
-
|
|
686
|
-
```ts
|
|
687
|
-
import { ai } from "@warlock.js/ai";
|
|
688
|
-
import "@warlock.js/ai-panoptic"; // side-effect: subscribes panoptic to the config seam
|
|
689
|
-
|
|
690
|
-
ai.config({
|
|
691
|
-
panoptic: {
|
|
692
|
-
dashboard: true, // serve on http://127.0.0.1:4319/
|
|
693
|
-
observeAll: true, // route every workflow/supervisor/team run to the collector
|
|
694
|
-
},
|
|
695
|
-
});
|
|
696
|
-
```
|
|
697
|
-
|
|
698
|
-
`dashboard: true` uses the defaults. Pass an object to override:
|
|
699
|
-
|
|
700
|
-
```ts
|
|
701
|
-
ai.config({
|
|
702
|
-
panoptic: {
|
|
703
|
-
exporters: [createInMemoryTraceStore({ capacity: 5000 })],
|
|
704
|
-
dashboard: { port: 4319, open: true, title: "My App — AI traces" },
|
|
705
|
-
observeAll: true,
|
|
706
|
-
},
|
|
707
|
-
});
|
|
708
|
-
```
|
|
709
|
-
|
|
710
|
-
**Where the dashboard reads its traces from.** It needs a queryable store (a `TraceStoreContract`). Panoptic resolves one in this order:
|
|
711
|
-
|
|
712
|
-
1. The first **store-shaped exporter** in `config.exporters` — `createInMemoryTraceStore()` doubles as an `ExporterContract`, so a store passed in `exporters` is reused (captured AND served from the same object).
|
|
713
|
-
2. If none was supplied but `dashboard` is set, panoptic creates a fresh in-memory store and registers it as an exporter so the collector fills it.
|
|
714
|
-
|
|
715
|
-
So `dashboard: true` with no exporters Just Works — you get a store for free. Supply your own when you want a capacity cap or want to `query` / `aggregate` it from app code too.
|
|
716
|
-
|
|
717
|
-
**Idempotent.** Applying the config is safe on every `ai.config(...)` merge: the collector is registered as a core `Observer` exactly once and the dashboard is started at most once. Repeat calls only refresh the `observeAll` flag (latest config wins). `panoptic` undefined in the config is a no-op.
|
|
718
|
-
|
|
719
|
-
## What the UI shows
|
|
720
|
-
|
|
721
|
-
The page is one self-contained, dependency-free vanilla-JS file (no bundler, no framework, works offline). It polls the JSON API every 2s and renders:
|
|
722
|
-
|
|
723
|
-
- **Header** — the Warlock logo + your `title`, the `TraceAggregate` headline counts (traces / completed / failed / cancelled), tokens as **`↓input · ↑output · total`**, total cost, an **errors-only** toggle, a **search** box, and a **light / dark / system** theme switch (persisted to `localStorage`, so the choice survives a reload; `system` follows the OS `prefers-color-scheme`).
|
|
724
|
-
- **Trace list** — newest-first clickable master rows, each led by a **colour-coded Title-case type label** (`Supervisor` / `Team` / `Workflow` / `Agent` / `Tool` / …; `ai.team` reports its own `team` type rather than `supervisor`, so it groups, filters, and labels separately). An optional **group-by-session** view collapses runs under collapsible session headers; a **group-by-prompt** view does the same over the `name@version` prompt key (see *Prompt-version linkage* below); a **group-by-type** view buckets the list by root type under collapsible `Type (N)` headers (`Agent (12)`, `Team (3)`, …). The three groupings are mutually exclusive.
|
|
725
|
-
- **Two-pane drawer** (click a row) — a collapsible **nested call tree** on the left and the **selected node's detail** on the right: rich structured input/output (chat bubbles / key-value / Markdown), a per-node **token breakdown** (`↓in · ↑out · total`), the node's **rollup cost** (its own cost, else the sum of its children — so wrapper nodes that carry tokens but no cost don't double-count), and a **metadata panel** with humanized keys (`Supervisor Terminated By`, `Span ID` — not raw `supervisor.terminatedBy`), and **started** / **ended** wall-clock timestamps (`28 Jun 2026 03:16 PM` — named month, 12-hour, minute precision) alongside the elapsed duration. Zero-valued token/cost rows are hidden so the panel stays readable.
|
|
726
|
-
- **Cost heatmap** — every tree node and trace row carries a left accent tinted by its rollup cost relative to the trace's most expensive node, with a small legend, so the spend hot-spots jump out at a glance.
|
|
727
|
-
- **Timeline view** — the drawer's left pane toggles between the nested call **Tree** and a Gantt **Timeline** (each span drawn at its offset from the root start, width = duration, critical path highlighted).
|
|
728
|
-
- **Stats panel** — a header **Stats** toggle shows a per-type aggregate table above the trace list, one row per root type (`Agent` / `Team` / `Workflow` / …) with Count, Failed (rate), p50 / p95 latency, total Tokens, and total Cost, computed over the active filtered set. Distinct from the headline counts in the header.
|
|
729
|
-
|
|
730
|
-
All UI state (selected trace, selected span, collapsed nodes, theme, filters) lives in JS, **not** the DOM, so the 2s poll never disturbs an open drawer.
|
|
731
|
-
|
|
732
|
-
### Filtering & grouping (all client-side over the polled list)
|
|
733
|
-
|
|
734
|
-
The list narrows entirely in the browser — no server round-trip per keystroke:
|
|
735
|
-
|
|
736
|
-
- **Search** — free-text, case-insensitive, matched against root span name + session id.
|
|
737
|
-
- **Status / type chips** — restrict to selected terminal statuses or executable types. Chip, badge, drawer, and group-header labels are **Title-cased** everywhere (`Completed`, `Workflow`, `Team`); the underlying filter keys stay lowercase (`completed`, `workflow`, `team`).
|
|
738
|
-
- **Errors-only** — header toggle that keeps just `failed` / `cancelled` runs.
|
|
739
|
-
- **Session chip** — exact `sessionId`; **prompt chip** — exact `name@version` key.
|
|
740
|
-
- **Group-by-type** — buckets the list by root type under `Type (N)` headers; mutually exclusive with group-by-session and group-by-prompt.
|
|
741
|
-
|
|
742
|
-
The pure matching/grouping rules are unit-tested in `trace-filter.ts` and mirrored 1:1 by the inlined dashboard JS, exported for reuse / your own list views:
|
|
743
|
-
|
|
744
|
-
```ts
|
|
745
|
-
import {
|
|
746
|
-
filterTraces, matchesFilter, groupBySession, rollupCost, maxNodeCost, heatIntensity,
|
|
747
|
-
type TraceFilter, type SessionGroup,
|
|
748
|
-
} from "@warlock.js/ai-panoptic"; // dashboard helpers, re-exported
|
|
749
|
-
|
|
750
|
-
const failed = filterTraces(traces, { errorsOnly: true });
|
|
751
|
-
const bySession = groupBySession(failed); // SessionGroup[], sessionless → "(no session)"
|
|
752
|
-
```
|
|
753
|
-
|
|
754
|
-
`TraceFilter` is `{ text?, statuses?, types?, sessionId?, promptKey?, errorsOnly? }` — every field optional and ANDed, so `{}` matches everything. `rollupCost(span)` / `maxNodeCost(root)` / `heatIntensity(nodeCost, maxCost)` are the pure heatmap folds (intensity in `[0,1]`, `0` for a free trace rather than dividing by zero).
|
|
755
|
-
|
|
756
|
-
### Deep-links
|
|
757
|
-
|
|
758
|
-
The open trace + span are reflected in the URL hash (`#trace=<id>&span=<id>`) and re-opened from it on load — so a drawer view is **shareable / bookmarkable**. (A live socket tail is a noted follow-up; this pass stays on the 2s poll.)
|
|
759
|
-
|
|
760
|
-
### Prompt-version linkage
|
|
761
|
-
|
|
762
|
-
When an agent ran against a *named* `ai.prompts` builder, the collector stamps `agent.promptName` / `agent.promptVersion` onto the root span. The dashboard derives a `name@version` key from them (`tracePromptKey(trace)`) and offers **filter-by-prompt** and **group-by-prompt** beside session — so a reviewer sees every run of `support@2` together and compares its cost / failure rate against `support@3`. Runs with no named prompt bucket under `"(no prompt)"`.
|
|
763
|
-
|
|
764
|
-
```ts
|
|
765
|
-
import { tracePromptKey, groupByPrompt, type PromptGroup } from "@warlock.js/ai-panoptic";
|
|
766
|
-
|
|
767
|
-
const byPrompt = groupByPrompt(store.query()); // PromptGroup[] keyed by name@version
|
|
768
|
-
```
|
|
769
|
-
|
|
770
|
-
## Persistent traces — `cache` (survive a restart)
|
|
771
|
-
|
|
772
|
-
By default the dashboard reads a fresh **in-memory** store, so traces vanish on restart. Point `panoptic.cache` at a `@warlock.js/cache` `CacheDriver` (Redis, file, …) and panoptic serves the dashboard from a **cache-backed** store instead — traces persist and re-hydrate on the next boot:
|
|
773
|
-
|
|
774
|
-
```ts
|
|
775
|
-
import { ai } from "@warlock.js/ai";
|
|
776
|
-
import "@warlock.js/ai-panoptic";
|
|
777
|
-
import { RedisCacheDriver } from "@warlock.js/cache";
|
|
778
|
-
|
|
779
|
-
ai.config({
|
|
780
|
-
panoptic: {
|
|
781
|
-
dashboard: true,
|
|
782
|
-
observeAll: true,
|
|
783
|
-
// A driver, or a (possibly async) factory resolved lazily on first use —
|
|
784
|
-
// the factory form defers the Redis connect until the first trace lands.
|
|
785
|
-
cache: async () => {
|
|
786
|
-
const driver = new RedisCacheDriver();
|
|
787
|
-
await driver.connect();
|
|
788
|
-
return driver;
|
|
789
|
-
},
|
|
790
|
-
},
|
|
791
|
-
});
|
|
792
|
-
```
|
|
793
|
-
|
|
794
|
-
Panoptic awaits the store's `ready()` hydration **before** the server starts serving, so a restart surfaces previously-persisted traces immediately. Reads stay synchronous (served from an in-memory mirror); writes go through to the cache fire-and-forget, so a flaky cache never throws into the collector's hot path. A store-shaped exporter you pass via `exporters` still wins over `cache`. The store building block (`createCacheTraceStore`) and its durability contract live in [`query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md).
|
|
795
|
-
|
|
796
|
-
## The low-level building block — `dashboard(store, options)`
|
|
797
|
-
|
|
798
|
-
Reach for this directly only when you manage the store yourself (e.g. a long-lived store you also query elsewhere, or a non-default lifecycle). It returns a `Promise<DashboardHandle>`:
|
|
799
|
-
|
|
800
|
-
```ts
|
|
801
|
-
import { dashboard, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
|
|
802
|
-
|
|
803
|
-
const store = createInMemoryTraceStore();
|
|
804
|
-
const handle = await dashboard(store, { port: 4319, open: true });
|
|
805
|
-
|
|
806
|
-
console.log(handle.url); // http://127.0.0.1:4319/
|
|
807
|
-
console.log(handle.port); // 4319 (the REAL OS port when port: 0 was passed)
|
|
808
|
-
|
|
809
|
-
// ...on teardown:
|
|
810
|
-
await handle.close();
|
|
811
|
-
```
|
|
812
|
-
|
|
813
|
-
The store is the same object the collector writes to — wire the collector to it (pass it as an exporter to `panoptic({ exporters: [store] })` or via `ai.config`) and the dashboard reflects new traces on its next poll.
|
|
814
|
-
|
|
815
|
-
### `DashboardOptions`
|
|
816
|
-
|
|
817
|
-
Every field optional, every default safe:
|
|
818
|
-
|
|
819
|
-
| Option | Default | Notes |
|
|
820
|
-
|---|---|---|
|
|
821
|
-
| `port` | `4319` | Pass `0` for an ephemeral port — the resolved port comes back on `handle.port`. A busy port rejects with a clear `Error` (`port N in use; pass { port: 0 }`), not raw `EADDRINUSE`. |
|
|
822
|
-
| `host` | `"127.0.0.1"` | **Loopback only** by default, so prompt content is never exposed to the LAN. Binding a non-loopback interface is gated — it requires `authToken` (see below) or `dashboard()` rejects. |
|
|
823
|
-
| `basePath` | `"/"` | Mount-path prefix for both the page and every `/api/...` route, e.g. `"/panoptic"`. Normalized to leading + trailing slash. |
|
|
824
|
-
| `open` | `false` | Open the default browser at the resolved URL once listening. Best-effort and fully swallowed — failing to open a browser never rejects the start. |
|
|
825
|
-
| `title` | `"Panoptic"` | Header title baked into the served page. |
|
|
826
|
-
| `authToken` | _unset_ | Bearer token required on every request. Compared with `crypto.timingSafeEqual` (constant-time, length-checked first) against both the header and query-string forms — never a plain `===` — so a network-reachable dashboard doesn't leak the token to a byte-at-a-time timing attack. `Authorization: Bearer <token>` is accepted on every route; `?token=` is accepted **only on the HTML page route** (the JSON API routes are header-only), since the page load is the one request that structurally can't carry a header — this narrows the token's footprint in access/proxy logs and browser history. Missing/wrong → `401`. **Required** when `host` is non-loopback. |
|
|
827
|
-
| `allowedHosts` | loopback names | `Host`-header allowlist (DNS-rebind guard); an unlisted `Host` → `403`. Defaults to `localhost` / `127.0.0.1` / `[::1]` for a loopback bind. |
|
|
828
|
-
|
|
829
|
-
### `DashboardHandle`
|
|
830
|
-
|
|
831
|
-
```ts
|
|
832
|
-
type DashboardHandle = {
|
|
833
|
-
readonly url: string; // http://host:port{basePath}
|
|
834
|
-
readonly port: number; // real OS port (when port: 0 was requested)
|
|
835
|
-
close(): Promise<void>; // stop the server, release the port
|
|
836
|
-
};
|
|
837
|
-
```
|
|
838
|
-
|
|
839
|
-
## The JSON API
|
|
840
|
-
|
|
841
|
-
Every route is **`GET`-only and read-only** (non-`GET` → `405`), mounted under `basePath` — EXCEPT the one opt-in `POST .../spans/:spanId/evaluate` route `DashboardOptions.evaluate` enables (see [`evaluate-system-prompt/SKILL.md`](@warlock.js/ai-panoptic/evaluate-system-prompt/SKILL.md)); absent that config, it 405s like everything else. The page polls the `GET` routes; you can curl them too. Store shapes are already JSON-safe, so responses are a plain `JSON.stringify`.
|
|
842
|
-
|
|
843
|
-
| Route | Returns |
|
|
844
|
-
|---|---|
|
|
845
|
-
| `GET /api/traces` | `store.query(...)` — the trace list, filtered by the query string (`id` / `sessionId` / `status` / time window, parsed by `parseQuery`). |
|
|
846
|
-
| `GET /api/traces/:id` | `store.get(id)` — one full trace, or `404 { error: "trace_not_found" }`. |
|
|
847
|
-
| `GET /api/aggregate` | `store.aggregate(...)` — the usage + cost + status rollup over the same query filter. |
|
|
848
|
-
| `GET /` (basePath) | the self-contained HTML page. |
|
|
849
|
-
| `POST /api/traces/:traceId/spans/:spanId/evaluate` | grades the span's last captured system prompt — only when `evaluate` is configured. |
|
|
850
|
-
|
|
851
|
-
Query-string filters map 1:1 onto `TraceQuery`:
|
|
852
|
-
|
|
853
|
-
```sh
|
|
854
|
-
curl 'http://127.0.0.1:4319/api/traces?status=failed'
|
|
855
|
-
curl 'http://127.0.0.1:4319/api/aggregate?sessionId=sess_42'
|
|
856
|
-
```
|
|
857
|
-
|
|
858
|
-
(The exact filter keys and the aggregate shape live in [`query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md).)
|
|
859
|
-
|
|
860
|
-
## Advanced — build the handler yourself
|
|
861
|
-
|
|
862
|
-
`createRequestHandler(store, config)` is the bare `(req, res) => void` the server closes over — split out from the lifecycle so it unit-tests without binding a port, and so you can mount the same routes inside an existing server:
|
|
863
|
-
|
|
864
|
-
```ts
|
|
865
|
-
import { createRequestHandler } from "@warlock.js/ai-panoptic";
|
|
866
|
-
import { createServer } from "node:http";
|
|
867
|
-
|
|
868
|
-
const handler = createRequestHandler(store, {
|
|
869
|
-
basePath: "/",
|
|
870
|
-
title: "Panoptic",
|
|
871
|
-
allowedHosts: ["127.0.0.1", "localhost"],
|
|
872
|
-
});
|
|
873
|
-
createServer(handler).listen(4319, "127.0.0.1");
|
|
874
|
-
```
|
|
875
|
-
|
|
876
|
-
`dashboardHtml(basePath, title, evaluateEnabled?, evaluateDefaultInstructions?)` returns the served page as a string if you embed it elsewhere. `ServeConfig` is `{ basePath; title; allowedHosts; authToken?; evaluate? }`.
|
|
877
|
-
|
|
878
|
-
## Security note
|
|
879
|
-
|
|
880
|
-
The dashboard surfaces whatever the store holds — including captured prompt/response content when content capture is on (`captureContent`). It binds **loopback-only** by default precisely so that content never leaves the machine. Binding a non-loopback host is gated — it requires an `authToken` (and checks a `Host` allowlist, sending `nosniff` / CSP / `X-Frame-Options: DENY` on every response) — but still prefer loopback for dev, and for production ship to a real backend via an exporter rather than exposing the dashboard. The dashboard has exactly one write-capable route (`evaluate`, off by default) — everything else stays read-only regardless of configuration.
|
|
881
|
-
|
|
882
|
-
**Trace content is untrusted input to the dashboard's own UI, and is treated that way:**
|
|
883
|
-
- **Markdown links in captured content are sanitized.** `span.input`/`span.output` can contain model output or attacker-controlled tool results — i.e. prompt-injected content — and the drawer renders it as Markdown. Link URLs go through `sanitizeHref` (a scheme allowlist: `http:`, `https:`, `mailto:`, plus relative/anchor URLs) after normalizing the way a browser will — attribute-entity decode, strip of ignored control chars/whitespace, lowercase — so `javascript:`/`data:`/`vbscript:` payloads (including case/whitespace/entity-obfuscated variants, and scheme-relative `//host` links) never become a live `href`; a rejected link renders its label as plain text instead. This closes a stored-XSS path that could otherwise reach the in-page bearer token.
|
|
884
|
-
- **The bearer token is no longer a page-global.** The client script's `TOKEN` is sealed inside its own inner IIFE that exposes only the `fetchAuthed` helper, so other code in the page's closure can't read the raw token by name.
|
|
885
|
-
|
|
886
|
-
These three findings (link sanitization + the two token-handling items above) came from the same audit; none is independently exploitable today, but each narrowed the blast radius of a future bug in an operator-facing surface that renders untrusted model/tool content.
|
|
887
|
-
|
|
888
|
-
## See also
|
|
889
|
-
|
|
890
|
-
- [`@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) — the `ai.config({ panoptic })` wiring and the underlying `panoptic(...)` subscriber that fills the store.
|
|
891
|
-
- [`@warlock.js/ai-panoptic/query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md) — the `TraceStoreContract` (`query` / `get` / `aggregate`) the dashboard reads, the `TraceQuery` filter, and capacity eviction.
|
|
892
|
-
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — shipping traces to OTel / Langfuse / a file instead of (or alongside) the local dashboard.
|
|
893
|
-
- [`@warlock.js/ai-panoptic/evaluate-system-prompt/SKILL.md`](@warlock.js/ai-panoptic/evaluate-system-prompt/SKILL.md) — the drawer's "Evaluate system prompt" action, the dashboard's one write route.
|
|
673
|
+
---
|
|
674
|
+
name: use-local-dashboard
|
|
675
|
+
description: 'Run the zero-setup local Panoptic dashboard — a loopback `node:http` server over a trace store served from `ai.config({ panoptic: { dashboard } })`, with a two-pane call-tree + metadata UI (light/dark/system theme, search/filter, session + prompt-version grouping, cost heatmap, Gantt timeline, errors filter, hash deep-links) and an optional cache-backed persistent store. Triggers: `dashboard`, `DashboardOptions`, `DashboardHandle`, `ai.config({ panoptic })`, `panoptic.dashboard`, `panoptic.cache`, `port: 4319`, `basePath`, `open`, `authToken`, `allowedHosts`, `/api/traces`, `/api/aggregate`, `/api/traces/:id`, `createRequestHandler`, `dashboardHtml`, `sanitizeHref`, `ServeConfig`, `TraceStoreContract`, `createCacheTraceStore`, theme, dark mode, call tree, metadata panel, cost heatmap, timeline / Gantt, errors only, deep-link, group by session, prompt version, group by type, type stats panel, started / ended timestamps, team type; ''open a local AI trace dashboard'', ''view agent traces in the browser'', ''self-hosted panoptic UI'', ''inspect traces without Langfuse'', ''spin up a dev observability server'', ''dark mode dashboard'', ''persist traces across restart'', ''group traces by session / prompt version / type'', ''cost heatmap'', ''timeline / waterfall view'', ''filter failed runs in the dashboard'', ''per-type stats panel'', ''is the dashboard bearer token safe'', ''markdown links XSS in trace content''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"`. Skip: shipping traces to a backend (OTel / Langfuse / file) — `@warlock.js/ai-panoptic/export-traces/SKILL.md`; the queryable store API the dashboard reads — `@warlock.js/ai-panoptic/query-traces/SKILL.md`; wiring the subscriber by hand — `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`.'
|
|
676
|
+
---
|
|
677
|
+
|
|
678
|
+
# The local Panoptic dashboard — zero-setup trace viewing
|
|
679
|
+
|
|
680
|
+
A loopback-only `node:http` server that serves one self-contained HTML page polling a read-only JSON API over a trace store. No external backend and no build step; it binds `127.0.0.1` with no auth by default, and gains optional bearer-token auth + a Host allowlist (`authToken` / `allowedHosts`, **required** off-loopback). The store is the live object the collector fills, so each poll reflects the latest completed traces.
|
|
681
|
+
|
|
682
|
+
## The documented path — `ai.config({ panoptic: { dashboard } })`
|
|
683
|
+
|
|
684
|
+
You almost never call `dashboard(...)` directly. Set it declaratively and the side-effect import wires the store, the collector, and the server for you:
|
|
685
|
+
|
|
686
|
+
```ts
|
|
687
|
+
import { ai } from "@warlock.js/ai";
|
|
688
|
+
import "@warlock.js/ai-panoptic"; // side-effect: subscribes panoptic to the config seam
|
|
689
|
+
|
|
690
|
+
ai.config({
|
|
691
|
+
panoptic: {
|
|
692
|
+
dashboard: true, // serve on http://127.0.0.1:4319/
|
|
693
|
+
observeAll: true, // route every workflow/supervisor/team run to the collector
|
|
694
|
+
},
|
|
695
|
+
});
|
|
696
|
+
```
|
|
697
|
+
|
|
698
|
+
`dashboard: true` uses the defaults. Pass an object to override:
|
|
699
|
+
|
|
700
|
+
```ts
|
|
701
|
+
ai.config({
|
|
702
|
+
panoptic: {
|
|
703
|
+
exporters: [createInMemoryTraceStore({ capacity: 5000 })],
|
|
704
|
+
dashboard: { port: 4319, open: true, title: "My App — AI traces" },
|
|
705
|
+
observeAll: true,
|
|
706
|
+
},
|
|
707
|
+
});
|
|
708
|
+
```
|
|
709
|
+
|
|
710
|
+
**Where the dashboard reads its traces from.** It needs a queryable store (a `TraceStoreContract`). Panoptic resolves one in this order:
|
|
711
|
+
|
|
712
|
+
1. The first **store-shaped exporter** in `config.exporters` — `createInMemoryTraceStore()` doubles as an `ExporterContract`, so a store passed in `exporters` is reused (captured AND served from the same object).
|
|
713
|
+
2. If none was supplied but `dashboard` is set, panoptic creates a fresh in-memory store and registers it as an exporter so the collector fills it.
|
|
714
|
+
|
|
715
|
+
So `dashboard: true` with no exporters Just Works — you get a store for free. Supply your own when you want a capacity cap or want to `query` / `aggregate` it from app code too.
|
|
716
|
+
|
|
717
|
+
**Idempotent.** Applying the config is safe on every `ai.config(...)` merge: the collector is registered as a core `Observer` exactly once and the dashboard is started at most once. Repeat calls only refresh the `observeAll` flag (latest config wins). `panoptic` undefined in the config is a no-op.
|
|
718
|
+
|
|
719
|
+
## What the UI shows
|
|
720
|
+
|
|
721
|
+
The page is one self-contained, dependency-free vanilla-JS file (no bundler, no framework, works offline). It polls the JSON API every 2s and renders:
|
|
722
|
+
|
|
723
|
+
- **Header** — the Warlock logo + your `title`, the `TraceAggregate` headline counts (traces / completed / failed / cancelled), tokens as **`↓input · ↑output · total`**, total cost, an **errors-only** toggle, a **search** box, and a **light / dark / system** theme switch (persisted to `localStorage`, so the choice survives a reload; `system` follows the OS `prefers-color-scheme`).
|
|
724
|
+
- **Trace list** — newest-first clickable master rows, each led by a **colour-coded Title-case type label** (`Supervisor` / `Team` / `Workflow` / `Agent` / `Tool` / …; `ai.team` reports its own `team` type rather than `supervisor`, so it groups, filters, and labels separately). An optional **group-by-session** view collapses runs under collapsible session headers; a **group-by-prompt** view does the same over the `name@version` prompt key (see *Prompt-version linkage* below); a **group-by-type** view buckets the list by root type under collapsible `Type (N)` headers (`Agent (12)`, `Team (3)`, …). The three groupings are mutually exclusive.
|
|
725
|
+
- **Two-pane drawer** (click a row) — a collapsible **nested call tree** on the left and the **selected node's detail** on the right: rich structured input/output (chat bubbles / key-value / Markdown), a per-node **token breakdown** (`↓in · ↑out · total`), the node's **rollup cost** (its own cost, else the sum of its children — so wrapper nodes that carry tokens but no cost don't double-count), and a **metadata panel** with humanized keys (`Supervisor Terminated By`, `Span ID` — not raw `supervisor.terminatedBy`), and **started** / **ended** wall-clock timestamps (`28 Jun 2026 03:16 PM` — named month, 12-hour, minute precision) alongside the elapsed duration. Zero-valued token/cost rows are hidden so the panel stays readable.
|
|
726
|
+
- **Cost heatmap** — every tree node and trace row carries a left accent tinted by its rollup cost relative to the trace's most expensive node, with a small legend, so the spend hot-spots jump out at a glance.
|
|
727
|
+
- **Timeline view** — the drawer's left pane toggles between the nested call **Tree** and a Gantt **Timeline** (each span drawn at its offset from the root start, width = duration, critical path highlighted).
|
|
728
|
+
- **Stats panel** — a header **Stats** toggle shows a per-type aggregate table above the trace list, one row per root type (`Agent` / `Team` / `Workflow` / …) with Count, Failed (rate), p50 / p95 latency, total Tokens, and total Cost, computed over the active filtered set. Distinct from the headline counts in the header.
|
|
729
|
+
|
|
730
|
+
All UI state (selected trace, selected span, collapsed nodes, theme, filters) lives in JS, **not** the DOM, so the 2s poll never disturbs an open drawer.
|
|
731
|
+
|
|
732
|
+
### Filtering & grouping (all client-side over the polled list)
|
|
733
|
+
|
|
734
|
+
The list narrows entirely in the browser — no server round-trip per keystroke:
|
|
735
|
+
|
|
736
|
+
- **Search** — free-text, case-insensitive, matched against root span name + session id.
|
|
737
|
+
- **Status / type chips** — restrict to selected terminal statuses or executable types. Chip, badge, drawer, and group-header labels are **Title-cased** everywhere (`Completed`, `Workflow`, `Team`); the underlying filter keys stay lowercase (`completed`, `workflow`, `team`).
|
|
738
|
+
- **Errors-only** — header toggle that keeps just `failed` / `cancelled` runs.
|
|
739
|
+
- **Session chip** — exact `sessionId`; **prompt chip** — exact `name@version` key.
|
|
740
|
+
- **Group-by-type** — buckets the list by root type under `Type (N)` headers; mutually exclusive with group-by-session and group-by-prompt.
|
|
741
|
+
|
|
742
|
+
The pure matching/grouping rules are unit-tested in `trace-filter.ts` and mirrored 1:1 by the inlined dashboard JS, exported for reuse / your own list views:
|
|
743
|
+
|
|
744
|
+
```ts
|
|
745
|
+
import {
|
|
746
|
+
filterTraces, matchesFilter, groupBySession, rollupCost, maxNodeCost, heatIntensity,
|
|
747
|
+
type TraceFilter, type SessionGroup,
|
|
748
|
+
} from "@warlock.js/ai-panoptic"; // dashboard helpers, re-exported
|
|
749
|
+
|
|
750
|
+
const failed = filterTraces(traces, { errorsOnly: true });
|
|
751
|
+
const bySession = groupBySession(failed); // SessionGroup[], sessionless → "(no session)"
|
|
752
|
+
```
|
|
753
|
+
|
|
754
|
+
`TraceFilter` is `{ text?, statuses?, types?, sessionId?, promptKey?, errorsOnly? }` — every field optional and ANDed, so `{}` matches everything. `rollupCost(span)` / `maxNodeCost(root)` / `heatIntensity(nodeCost, maxCost)` are the pure heatmap folds (intensity in `[0,1]`, `0` for a free trace rather than dividing by zero).
|
|
755
|
+
|
|
756
|
+
### Deep-links
|
|
757
|
+
|
|
758
|
+
The open trace + span are reflected in the URL hash (`#trace=<id>&span=<id>`) and re-opened from it on load — so a drawer view is **shareable / bookmarkable**. (A live socket tail is a noted follow-up; this pass stays on the 2s poll.)
|
|
759
|
+
|
|
760
|
+
### Prompt-version linkage
|
|
761
|
+
|
|
762
|
+
When an agent ran against a *named* `ai.prompts` builder, the collector stamps `agent.promptName` / `agent.promptVersion` onto the root span. The dashboard derives a `name@version` key from them (`tracePromptKey(trace)`) and offers **filter-by-prompt** and **group-by-prompt** beside session — so a reviewer sees every run of `support@2` together and compares its cost / failure rate against `support@3`. Runs with no named prompt bucket under `"(no prompt)"`.
|
|
763
|
+
|
|
764
|
+
```ts
|
|
765
|
+
import { tracePromptKey, groupByPrompt, type PromptGroup } from "@warlock.js/ai-panoptic";
|
|
766
|
+
|
|
767
|
+
const byPrompt = groupByPrompt(store.query()); // PromptGroup[] keyed by name@version
|
|
768
|
+
```
|
|
769
|
+
|
|
770
|
+
## Persistent traces — `cache` (survive a restart)
|
|
771
|
+
|
|
772
|
+
By default the dashboard reads a fresh **in-memory** store, so traces vanish on restart. Point `panoptic.cache` at a `@warlock.js/cache` `CacheDriver` (Redis, file, …) and panoptic serves the dashboard from a **cache-backed** store instead — traces persist and re-hydrate on the next boot:
|
|
773
|
+
|
|
774
|
+
```ts
|
|
775
|
+
import { ai } from "@warlock.js/ai";
|
|
776
|
+
import "@warlock.js/ai-panoptic";
|
|
777
|
+
import { RedisCacheDriver } from "@warlock.js/cache";
|
|
778
|
+
|
|
779
|
+
ai.config({
|
|
780
|
+
panoptic: {
|
|
781
|
+
dashboard: true,
|
|
782
|
+
observeAll: true,
|
|
783
|
+
// A driver, or a (possibly async) factory resolved lazily on first use —
|
|
784
|
+
// the factory form defers the Redis connect until the first trace lands.
|
|
785
|
+
cache: async () => {
|
|
786
|
+
const driver = new RedisCacheDriver();
|
|
787
|
+
await driver.connect();
|
|
788
|
+
return driver;
|
|
789
|
+
},
|
|
790
|
+
},
|
|
791
|
+
});
|
|
792
|
+
```
|
|
793
|
+
|
|
794
|
+
Panoptic awaits the store's `ready()` hydration **before** the server starts serving, so a restart surfaces previously-persisted traces immediately. Reads stay synchronous (served from an in-memory mirror); writes go through to the cache fire-and-forget, so a flaky cache never throws into the collector's hot path. A store-shaped exporter you pass via `exporters` still wins over `cache`. The store building block (`createCacheTraceStore`) and its durability contract live in [`query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md).
|
|
795
|
+
|
|
796
|
+
## The low-level building block — `dashboard(store, options)`
|
|
797
|
+
|
|
798
|
+
Reach for this directly only when you manage the store yourself (e.g. a long-lived store you also query elsewhere, or a non-default lifecycle). It returns a `Promise<DashboardHandle>`:
|
|
799
|
+
|
|
800
|
+
```ts
|
|
801
|
+
import { dashboard, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
|
|
802
|
+
|
|
803
|
+
const store = createInMemoryTraceStore();
|
|
804
|
+
const handle = await dashboard(store, { port: 4319, open: true });
|
|
805
|
+
|
|
806
|
+
console.log(handle.url); // http://127.0.0.1:4319/
|
|
807
|
+
console.log(handle.port); // 4319 (the REAL OS port when port: 0 was passed)
|
|
808
|
+
|
|
809
|
+
// ...on teardown:
|
|
810
|
+
await handle.close();
|
|
811
|
+
```
|
|
812
|
+
|
|
813
|
+
The store is the same object the collector writes to — wire the collector to it (pass it as an exporter to `panoptic({ exporters: [store] })` or via `ai.config`) and the dashboard reflects new traces on its next poll.
|
|
814
|
+
|
|
815
|
+
### `DashboardOptions`
|
|
816
|
+
|
|
817
|
+
Every field optional, every default safe:
|
|
818
|
+
|
|
819
|
+
| Option | Default | Notes |
|
|
820
|
+
|---|---|---|
|
|
821
|
+
| `port` | `4319` | Pass `0` for an ephemeral port — the resolved port comes back on `handle.port`. A busy port rejects with a clear `Error` (`port N in use; pass { port: 0 }`), not raw `EADDRINUSE`. |
|
|
822
|
+
| `host` | `"127.0.0.1"` | **Loopback only** by default, so prompt content is never exposed to the LAN. Binding a non-loopback interface is gated — it requires `authToken` (see below) or `dashboard()` rejects. |
|
|
823
|
+
| `basePath` | `"/"` | Mount-path prefix for both the page and every `/api/...` route, e.g. `"/panoptic"`. Normalized to leading + trailing slash. |
|
|
824
|
+
| `open` | `false` | Open the default browser at the resolved URL once listening. Best-effort and fully swallowed — failing to open a browser never rejects the start. |
|
|
825
|
+
| `title` | `"Panoptic"` | Header title baked into the served page. |
|
|
826
|
+
| `authToken` | _unset_ | Bearer token required on every request. Compared with `crypto.timingSafeEqual` (constant-time, length-checked first) against both the header and query-string forms — never a plain `===` — so a network-reachable dashboard doesn't leak the token to a byte-at-a-time timing attack. `Authorization: Bearer <token>` is accepted on every route; `?token=` is accepted **only on the HTML page route** (the JSON API routes are header-only), since the page load is the one request that structurally can't carry a header — this narrows the token's footprint in access/proxy logs and browser history. Missing/wrong → `401`. **Required** when `host` is non-loopback. |
|
|
827
|
+
| `allowedHosts` | loopback names | `Host`-header allowlist (DNS-rebind guard); an unlisted `Host` → `403`. Defaults to `localhost` / `127.0.0.1` / `[::1]` for a loopback bind. |
|
|
828
|
+
|
|
829
|
+
### `DashboardHandle`
|
|
830
|
+
|
|
831
|
+
```ts
|
|
832
|
+
type DashboardHandle = {
|
|
833
|
+
readonly url: string; // http://host:port{basePath}
|
|
834
|
+
readonly port: number; // real OS port (when port: 0 was requested)
|
|
835
|
+
close(): Promise<void>; // stop the server, release the port
|
|
836
|
+
};
|
|
837
|
+
```
|
|
838
|
+
|
|
839
|
+
## The JSON API
|
|
840
|
+
|
|
841
|
+
Every route is **`GET`-only and read-only** (non-`GET` → `405`), mounted under `basePath` — EXCEPT the one opt-in `POST .../spans/:spanId/evaluate` route `DashboardOptions.evaluate` enables (see [`evaluate-system-prompt/SKILL.md`](@warlock.js/ai-panoptic/evaluate-system-prompt/SKILL.md)); absent that config, it 405s like everything else. The page polls the `GET` routes; you can curl them too. Store shapes are already JSON-safe, so responses are a plain `JSON.stringify`.
|
|
842
|
+
|
|
843
|
+
| Route | Returns |
|
|
844
|
+
|---|---|
|
|
845
|
+
| `GET /api/traces` | `store.query(...)` — the trace list, filtered by the query string (`id` / `sessionId` / `status` / time window, parsed by `parseQuery`). |
|
|
846
|
+
| `GET /api/traces/:id` | `store.get(id)` — one full trace, or `404 { error: "trace_not_found" }`. |
|
|
847
|
+
| `GET /api/aggregate` | `store.aggregate(...)` — the usage + cost + status rollup over the same query filter. |
|
|
848
|
+
| `GET /` (basePath) | the self-contained HTML page. |
|
|
849
|
+
| `POST /api/traces/:traceId/spans/:spanId/evaluate` | grades the span's last captured system prompt — only when `evaluate` is configured. |
|
|
850
|
+
|
|
851
|
+
Query-string filters map 1:1 onto `TraceQuery`:
|
|
852
|
+
|
|
853
|
+
```sh
|
|
854
|
+
curl 'http://127.0.0.1:4319/api/traces?status=failed'
|
|
855
|
+
curl 'http://127.0.0.1:4319/api/aggregate?sessionId=sess_42'
|
|
856
|
+
```
|
|
857
|
+
|
|
858
|
+
(The exact filter keys and the aggregate shape live in [`query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md).)
|
|
859
|
+
|
|
860
|
+
## Advanced — build the handler yourself
|
|
861
|
+
|
|
862
|
+
`createRequestHandler(store, config)` is the bare `(req, res) => void` the server closes over — split out from the lifecycle so it unit-tests without binding a port, and so you can mount the same routes inside an existing server:
|
|
863
|
+
|
|
864
|
+
```ts
|
|
865
|
+
import { createRequestHandler } from "@warlock.js/ai-panoptic";
|
|
866
|
+
import { createServer } from "node:http";
|
|
867
|
+
|
|
868
|
+
const handler = createRequestHandler(store, {
|
|
869
|
+
basePath: "/",
|
|
870
|
+
title: "Panoptic",
|
|
871
|
+
allowedHosts: ["127.0.0.1", "localhost"],
|
|
872
|
+
});
|
|
873
|
+
createServer(handler).listen(4319, "127.0.0.1");
|
|
874
|
+
```
|
|
875
|
+
|
|
876
|
+
`dashboardHtml(basePath, title, evaluateEnabled?, evaluateDefaultInstructions?)` returns the served page as a string if you embed it elsewhere. `ServeConfig` is `{ basePath; title; allowedHosts; authToken?; evaluate? }`.
|
|
877
|
+
|
|
878
|
+
## Security note
|
|
879
|
+
|
|
880
|
+
The dashboard surfaces whatever the store holds — including captured prompt/response content when content capture is on (`captureContent`). It binds **loopback-only** by default precisely so that content never leaves the machine. Binding a non-loopback host is gated — it requires an `authToken` (and checks a `Host` allowlist, sending `nosniff` / CSP / `X-Frame-Options: DENY` on every response) — but still prefer loopback for dev, and for production ship to a real backend via an exporter rather than exposing the dashboard. The dashboard has exactly one write-capable route (`evaluate`, off by default) — everything else stays read-only regardless of configuration.
|
|
881
|
+
|
|
882
|
+
**Trace content is untrusted input to the dashboard's own UI, and is treated that way:**
|
|
883
|
+
- **Markdown links in captured content are sanitized.** `span.input`/`span.output` can contain model output or attacker-controlled tool results — i.e. prompt-injected content — and the drawer renders it as Markdown. Link URLs go through `sanitizeHref` (a scheme allowlist: `http:`, `https:`, `mailto:`, plus relative/anchor URLs) after normalizing the way a browser will — attribute-entity decode, strip of ignored control chars/whitespace, lowercase — so `javascript:`/`data:`/`vbscript:` payloads (including case/whitespace/entity-obfuscated variants, and scheme-relative `//host` links) never become a live `href`; a rejected link renders its label as plain text instead. This closes a stored-XSS path that could otherwise reach the in-page bearer token.
|
|
884
|
+
- **The bearer token is no longer a page-global.** The client script's `TOKEN` is sealed inside its own inner IIFE that exposes only the `fetchAuthed` helper, so other code in the page's closure can't read the raw token by name.
|
|
885
|
+
|
|
886
|
+
These three findings (link sanitization + the two token-handling items above) came from the same audit; none is independently exploitable today, but each narrowed the blast radius of a future bug in an operator-facing surface that renders untrusted model/tool content.
|
|
887
|
+
|
|
888
|
+
## See also
|
|
889
|
+
|
|
890
|
+
- [`@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) — the `ai.config({ panoptic })` wiring and the underlying `panoptic(...)` subscriber that fills the store.
|
|
891
|
+
- [`@warlock.js/ai-panoptic/query-traces/SKILL.md`](@warlock.js/ai-panoptic/query-traces/SKILL.md) — the `TraceStoreContract` (`query` / `get` / `aggregate`) the dashboard reads, the `TraceQuery` filter, and capacity eviction.
|
|
892
|
+
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — shipping traces to OTel / Langfuse / a file instead of (or alongside) the local dashboard.
|
|
893
|
+
- [`@warlock.js/ai-panoptic/evaluate-system-prompt/SKILL.md`](@warlock.js/ai-panoptic/evaluate-system-prompt/SKILL.md) — the drawer's "Evaluate system prompt" action, the dashboard's one write route.
|
|
894
894
|
|
|
895
895
|
|