@warlock.js/ai-panoptic 5.27.0 → 5.29.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/llms-full.txt +29 -29
- package/llms.txt +5 -5
- package/package.json +4 -4
- package/skills/README.md +6 -6
- package/skills/evaluate-system-prompt/SKILL.md +4 -4
- package/skills/export-traces/SKILL.md +4 -4
- package/skills/observe-with-panoptic/SKILL.md +6 -6
- package/skills/query-traces/SKILL.md +7 -7
- package/skills/use-local-dashboard/SKILL.md +8 -8
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,18 @@ All notable changes to `@warlock.js/ai-panoptic` are documented in this file.
|
|
|
4
4
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `@warlock.js/*` packages are released in lockstep — every package shares the same version number, so a version below may list only the changes that affected this package.
|
|
6
6
|
|
|
7
|
+
## 5.29.0 - 2026-10-01
|
|
8
|
+
|
|
9
|
+
### Changed
|
|
10
|
+
|
|
11
|
+
- Lockstep release; package APIs are unchanged. Skills refer to other topics by name instead of file paths, so cross-references keep working after `agent-kit` copies them into an app.
|
|
12
|
+
|
|
13
|
+
## 5.28.0 - 2026-10-01
|
|
14
|
+
|
|
15
|
+
### Changed
|
|
16
|
+
|
|
17
|
+
- Lockstep release; package APIs are unchanged. `llms.txt` now lists every skill topic with its description (frontmatter in CRLF files was skipped).
|
|
18
|
+
|
|
7
19
|
## 5.27.0 - 2026-10-01
|
|
8
20
|
|
|
9
21
|
### Changed
|
package/llms-full.txt
CHANGED
|
@@ -8,12 +8,12 @@
|
|
|
8
8
|
|
|
9
9
|
---
|
|
10
10
|
name: evaluate-system-prompt
|
|
11
|
-
description: 'Grade a single trace''s LAST captured system prompt from the dashboard drawer — the dashboard''s only write-capable feature, config-gated via `DashboardOptions.evaluate` (`ai.config({ panoptic: { dashboard: { evaluate: { model, instructions } } } })`). Triggers: `evaluate`, `EvaluateConfig`, `EvaluateVerdict`, `evaluateSystemPrompt`, `extractLastSystemPrompt`, `findSpanById`, "Evaluate system prompt" button, `POST .../spans/:spanId/evaluate`, grading a prompt from a real trace, judging a captured system prompt, per-run instructions override, judge model factory; ''evaluate this trace''s prompt'', ''grade the system prompt from the dashboard'', ''judge a captured run'', ''score a live system prompt''. Skip: grading raw prompt TEXT you already have in code (no trace involved) — `@warlock.js/ai`''s `ai.prompts().validate({ criteria })`, the same judge machinery this wraps; a DATASET of test cases run through an agent — `@warlock.js/ai`''s `ai.dataset` + `agent.eval`; viewing/filtering/grouping traces without grading them —
|
|
11
|
+
description: 'Grade a single trace''s LAST captured system prompt from the dashboard drawer — the dashboard''s only write-capable feature, config-gated via `DashboardOptions.evaluate` (`ai.config({ panoptic: { dashboard: { evaluate: { model, instructions } } } })`). Triggers: `evaluate`, `EvaluateConfig`, `EvaluateVerdict`, `evaluateSystemPrompt`, `extractLastSystemPrompt`, `findSpanById`, "Evaluate system prompt" button, `POST .../spans/:spanId/evaluate`, grading a prompt from a real trace, judging a captured system prompt, per-run instructions override, judge model factory; ''evaluate this trace''s prompt'', ''grade the system prompt from the dashboard'', ''judge a captured run'', ''score a live system prompt''. Skip: grading raw prompt TEXT you already have in code (no trace involved) — `@warlock.js/ai`''s `ai.prompts().validate({ criteria })`, the same judge machinery this wraps; a DATASET of test cases run through an agent — `@warlock.js/ai`''s `ai.dataset` + `agent.eval`; viewing/filtering/grouping traces without grading them — the `use-local-dashboard` topic; capturing the content this feature reads — the `export-traces` topic (`captureContent`).'
|
|
12
12
|
---
|
|
13
13
|
|
|
14
14
|
# Evaluate a trace's system prompt from the dashboard
|
|
15
15
|
|
|
16
|
-
A drawer button that grades the LAST `{role: "system"}` message captured on the selected span — not the whole conversation, not the tool code, just the system prompt that was actually in effect for that run. It's the dashboard's only write-capable feature: every other route is `GET`-only and read-only by design (see `use-local-dashboard
|
|
16
|
+
A drawer button that grades the LAST `{role: "system"}` message captured on the selected span — not the whole conversation, not the tool code, just the system prompt that was actually in effect for that run. It's the dashboard's only write-capable feature: every other route is `GET`-only and read-only by design (see the `use-local-dashboard` topic's security note); this one POSTs, and it's off unless you explicitly opt in.
|
|
17
17
|
|
|
18
18
|
## Enable it
|
|
19
19
|
|
|
@@ -89,8 +89,8 @@ curl -X POST 'http://127.0.0.1:4319/api/traces/tr_1/spans/sp_1/evaluate' \
|
|
|
89
89
|
|
|
90
90
|
## See also
|
|
91
91
|
|
|
92
|
-
-
|
|
93
|
-
-
|
|
92
|
+
- The `use-local-dashboard` topic — the dashboard this feature is a drawer action of; its security note now says "every route is GET-only and read-only EXCEPT the one opt-in write route `evaluate` enables."
|
|
93
|
+
- The `export-traces` topic — `captureContent` (and `fullHistory`), the prerequisite for `span.input` to carry anything gradable.
|
|
94
94
|
- `ai.prompts().validate({ criteria })` in `@warlock.js/ai` — the sibling capability this reuses: grading raw prompt TEXT you already have in code, no trace or dashboard involved.
|
|
95
95
|
|
|
96
96
|
|
|
@@ -98,7 +98,7 @@ curl -X POST 'http://127.0.0.1:4319/api/traces/tr_1/spans/sp_1/evaluate' \
|
|
|
98
98
|
|
|
99
99
|
---
|
|
100
100
|
name: export-traces
|
|
101
|
-
description: 'Send @warlock.js/ai-panoptic traces to an observability backend via pluggable exporters, including capturing prompt/response content onto spans. Triggers: `consoleExporter`, `fileExporter`, `otelExporter`, `langfuseExporter`, `ExporterContract`, `ContentCaptureOptions`, `captureContent`, `redactContent`, `fullHistory`, `ContentRedactor`, `toGenAiAttributes`, `walkSpans`, `totalCostUsd`, `GEN_AI_ATTRIBUTES`, `WARLOCK_ATTRIBUTES`, `reportToTrace`, `reportToSpan`, `extractSpanAttributes`, `normalizeError`, `formatSpanLine`, `createPanopticMiddleware`; ''export AI traces to OpenTelemetry'', ''send traces to Langfuse'', ''log traces to console / a file'', ''capture the full prompt/conversation onto a span'', ''gen_ai semantic conventions'', ''write a custom exporter''; typical import `import { otelExporter, langfuseExporter, consoleExporter, fileExporter } from "@warlock.js/ai-panoptic"`. Skip: wiring the subscriber/collector into a run —
|
|
101
|
+
description: 'Send @warlock.js/ai-panoptic traces to an observability backend via pluggable exporters, including capturing prompt/response content onto spans. Triggers: `consoleExporter`, `fileExporter`, `otelExporter`, `langfuseExporter`, `ExporterContract`, `ContentCaptureOptions`, `captureContent`, `redactContent`, `fullHistory`, `ContentRedactor`, `toGenAiAttributes`, `walkSpans`, `totalCostUsd`, `GEN_AI_ATTRIBUTES`, `WARLOCK_ATTRIBUTES`, `reportToTrace`, `reportToSpan`, `extractSpanAttributes`, `normalizeError`, `formatSpanLine`, `createPanopticMiddleware`; ''export AI traces to OpenTelemetry'', ''send traces to Langfuse'', ''log traces to console / a file'', ''capture the full prompt/conversation onto a span'', ''gen_ai semantic conventions'', ''write a custom exporter''; typical import `import { otelExporter, langfuseExporter, consoleExporter, fileExporter } from "@warlock.js/ai-panoptic"`. Skip: wiring the subscriber/collector into a run — the `observe-with-panoptic` topic; the core report shape — the `run-ai-agent` topic of the `warlock-js-ai` skill.'
|
|
102
102
|
---
|
|
103
103
|
|
|
104
104
|
# Export traces — the sink end of Panoptic
|
|
@@ -277,16 +277,16 @@ Reach for `reportToTrace` when you already hold a finished `result.report` and w
|
|
|
277
277
|
|
|
278
278
|
## See also
|
|
279
279
|
|
|
280
|
-
-
|
|
281
|
-
-
|
|
282
|
-
-
|
|
280
|
+
- The `observe-with-panoptic` topic — the source end: wiring the `panoptic(...)` subscriber into a run (`attach` / `middleware` / `collect`). The `collector` these exporters register on comes from there.
|
|
281
|
+
- The `query-traces` topic — the in-memory trace store: a sibling sink that *retains* traces (instead of shipping them out) so you can `query` / `aggregate` them after the fact.
|
|
282
|
+
- The `run-ai-agent` topic of the `warlock-js-ai` skill — running the agent and the `BaseReport` shape Panoptic projects into a `Trace`.
|
|
283
283
|
|
|
284
284
|
|
|
285
285
|
## observe-with-panoptic `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`
|
|
286
286
|
|
|
287
287
|
---
|
|
288
288
|
name: observe-with-panoptic
|
|
289
|
-
description: 'Wire @warlock.js/ai-panoptic into an agent/workflow/supervisor/orchestrator — declaratively via `ai.config({ panoptic })` (registers panoptic on core''s Observer seam) or with the one-call `panoptic({ exporters })` subscriber. Triggers: `panoptic`, `Panoptic`, `PanopticOptions`, `PanopticConfig`, `ai.config({ panoptic })`, `observeAll`, per-flow `observe`, `FlowObserveOption`, `registerObserver`, `Observer`, `.attach`, `.middleware`, `.collect`, `.toTrace`, `observe.attach(agent)`, `panoptic().middleware()`, `completedEvents`, `middlewareName`; ''observe a warlock agent'', ''trace an agent run'', ''attach observability to a workflow/supervisor'', ''observe every flow by default'', ''add a tracing middleware'', ''collect an orchestrator turn report'', ''wire OTel/Langfuse into my agent''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"` (or `import { panoptic } from "@warlock.js/ai-panoptic"`). Skip: the zero-setup local dashboard —
|
|
289
|
+
description: 'Wire @warlock.js/ai-panoptic into an agent/workflow/supervisor/orchestrator — declaratively via `ai.config({ panoptic })` (registers panoptic on core''s Observer seam) or with the one-call `panoptic({ exporters })` subscriber. Triggers: `panoptic`, `Panoptic`, `PanopticOptions`, `PanopticConfig`, `ai.config({ panoptic })`, `observeAll`, per-flow `observe`, `FlowObserveOption`, `registerObserver`, `Observer`, `.attach`, `.middleware`, `.collect`, `.toTrace`, `observe.attach(agent)`, `panoptic().middleware()`, `completedEvents`, `middlewareName`; ''observe a warlock agent'', ''trace an agent run'', ''attach observability to a workflow/supervisor'', ''observe every flow by default'', ''add a tracing middleware'', ''collect an orchestrator turn report'', ''wire OTel/Langfuse into my agent''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"` (or `import { panoptic } from "@warlock.js/ai-panoptic"`). Skip: the zero-setup local dashboard — the `use-local-dashboard` topic; writing/choosing an exporter (the sink end) — the `export-traces` topic; the core report shape / running the agent itself — the `run-ai-agent` topic of the `warlock-js-ai` skill.'
|
|
290
290
|
---
|
|
291
291
|
|
|
292
292
|
# Observe a run with Panoptic — the one-call subscriber
|
|
@@ -451,21 +451,21 @@ const observe = panoptic({ collector, exporters: [consoleExporter()] });
|
|
|
451
451
|
|
|
452
452
|
Add exporters after construction with `observe.use(exporter)` (chainable, deduped by `ExporterContract.name`). `observe.toTrace(report)` projects a report into a `Trace` **without** dispatching — handy for inspection or tests.
|
|
453
453
|
|
|
454
|
-
> The store's full query / aggregate / capacity surface is its own skill — see
|
|
454
|
+
> The store's full query / aggregate / capacity surface is its own skill — see the `query-traces` topic.
|
|
455
455
|
|
|
456
456
|
## See also
|
|
457
457
|
|
|
458
|
-
-
|
|
459
|
-
-
|
|
460
|
-
-
|
|
461
|
-
-
|
|
458
|
+
- The `use-local-dashboard` topic — the zero-setup local viewer the `ai.config({ panoptic: { dashboard } })` switch starts over the collector's store.
|
|
459
|
+
- The `export-traces` topic — the sink end: choosing/writing an exporter, the `gen_ai.*` / `warlock.*` attribute mapping.
|
|
460
|
+
- The `query-traces` topic — retaining traces in the queryable in-memory store and slicing them (`query` / `aggregate` / `capacity`).
|
|
461
|
+
- The `run-ai-agent` topic of the `warlock-js-ai` skill — running the agent and the `BaseReport` shape Panoptic consumes.
|
|
462
462
|
|
|
463
463
|
|
|
464
464
|
## query-traces `@warlock.js/ai-panoptic/query-traces/SKILL.md`
|
|
465
465
|
|
|
466
466
|
---
|
|
467
467
|
name: query-traces
|
|
468
|
-
description: 'Retain @warlock.js/ai-panoptic traces in a queryable store — in-memory, or cache-backed so they survive a process restart — and slice them after the fact: get one run by id, list a session, filter failed runs in a window, roll up usage + cost. Triggers: `createInMemoryTraceStore`, `createCacheTraceStore`, `CacheTraceStoreHandle`, `CacheTraceStoreOptions`, `CacheDriverInput`, `store.ready`, `TraceStoreContract`, `TraceQuery`, `TraceAggregate`, `InMemoryTraceStoreOptions`, `store.query`, `store.aggregate`, `store.get`, `store.add`, `store.clear`, `store.size`, `capacity`, `prefix`, `onError`, `sumUsage`, `emptyUsage`, `matchTrace`; ''query collected traces'', ''aggregate AI cost per session'', ''how much did this session spend'', ''list failed runs'', ''retain traces in memory'', ''persist traces across a restart'', ''cache-backed trace store'', ''Redis trace store'', ''roll up token usage''; typical import `import { createInMemoryTraceStore, createCacheTraceStore } from "@warlock.js/ai-panoptic"`. Skip: sending traces to an external backend (OTel/Langfuse/console/file) —
|
|
468
|
+
description: 'Retain @warlock.js/ai-panoptic traces in a queryable store — in-memory, or cache-backed so they survive a process restart — and slice them after the fact: get one run by id, list a session, filter failed runs in a window, roll up usage + cost. Triggers: `createInMemoryTraceStore`, `createCacheTraceStore`, `CacheTraceStoreHandle`, `CacheTraceStoreOptions`, `CacheDriverInput`, `store.ready`, `TraceStoreContract`, `TraceQuery`, `TraceAggregate`, `InMemoryTraceStoreOptions`, `store.query`, `store.aggregate`, `store.get`, `store.add`, `store.clear`, `store.size`, `capacity`, `prefix`, `onError`, `sumUsage`, `emptyUsage`, `matchTrace`; ''query collected traces'', ''aggregate AI cost per session'', ''how much did this session spend'', ''list failed runs'', ''retain traces in memory'', ''persist traces across a restart'', ''cache-backed trace store'', ''Redis trace store'', ''roll up token usage''; typical import `import { createInMemoryTraceStore, createCacheTraceStore } from "@warlock.js/ai-panoptic"`. Skip: sending traces to an external backend (OTel/Langfuse/console/file) — the `export-traces` topic; wiring the subscriber into a run — the `observe-with-panoptic` topic.'
|
|
469
469
|
---
|
|
470
470
|
|
|
471
471
|
# Query traces — the in-memory queryable store
|
|
@@ -492,7 +492,7 @@ const failed = store.query({ status: "failed" });
|
|
|
492
492
|
const spend = store.aggregate({ sessionId: "session-42" });
|
|
493
493
|
```
|
|
494
494
|
|
|
495
|
-
> The store is also a valid `collector.use(...)` target directly — `panoptic({ collector: createCollector().use(store) })` — see
|
|
495
|
+
> The store is also a valid `collector.use(...)` target directly — `panoptic({ collector: createCollector().use(store) })` — see the `observe-with-panoptic` topic for bring-your-own-collector wiring.
|
|
496
496
|
|
|
497
497
|
## The store contract
|
|
498
498
|
|
|
@@ -593,7 +593,7 @@ const store = createInMemoryTraceStore({ capacity: 1000 });
|
|
|
593
593
|
|
|
594
594
|
Overwriting an existing `traceId` refreshes its insertion position, so a re-collected run is treated as newest for eviction — eviction stays honest under overwrite.
|
|
595
595
|
|
|
596
|
-
> The in-memory store has no secondary indexes: `get` / `add` are O(1), `query` / `aggregate` are O(n) scans. That is the right trade-off for dev/test and modest-volume runtime use. For high-volume or durable retention, write a custom `TraceStoreContract` (or export to a backend with
|
|
596
|
+
> The in-memory store has no secondary indexes: `get` / `add` are O(1), `query` / `aggregate` are O(n) scans. That is the right trade-off for dev/test and modest-volume runtime use. For high-volume or durable retention, write a custom `TraceStoreContract` (or export to a backend with the `export-traces` topic) backed by a real datastore.
|
|
597
597
|
|
|
598
598
|
## Persisting across a restart — `createCacheTraceStore`
|
|
599
599
|
|
|
@@ -644,7 +644,7 @@ type CacheTraceStoreHandle = TraceStoreContract & ExporterContract & {
|
|
|
644
644
|
|
|
645
645
|
Ordering uses an **internal monotonic counter**, not `Date.now()`, so the persisted index stays a stable total order even when many traces land in the same millisecond — FIFO eviction stays honest across a restart.
|
|
646
646
|
|
|
647
|
-
> The usual path is declarative: `ai.config({ panoptic: { cache, dashboard: true } })` constructs this store, awaits `ready()`, and wires it under the dashboard for you — see
|
|
647
|
+
> The usual path is declarative: `ai.config({ panoptic: { cache, dashboard: true } })` constructs this store, awaits `ready()`, and wires it under the dashboard for you — see the `use-local-dashboard` topic. Reach for `createCacheTraceStore` directly only when you manage the store / collector yourself.
|
|
648
648
|
|
|
649
649
|
## Summing usage yourself
|
|
650
650
|
|
|
@@ -663,16 +663,16 @@ for (const trace of store.query({ sessionId })) {
|
|
|
663
663
|
|
|
664
664
|
## See also
|
|
665
665
|
|
|
666
|
-
-
|
|
667
|
-
-
|
|
668
|
-
-
|
|
666
|
+
- The `observe-with-panoptic` topic — wiring the `panoptic(...)` subscriber and bring-your-own-collector so the store fills as runs complete.
|
|
667
|
+
- The `export-traces` topic — sending traces to an external backend instead of (or alongside) retaining them in memory; writing a custom sink.
|
|
668
|
+
- The `run-ai-agent` topic of the `warlock-js-ai` skill — the `BaseReport` the collector projects into the `Trace` this store retains.
|
|
669
669
|
|
|
670
670
|
|
|
671
671
|
## use-local-dashboard `@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md`
|
|
672
672
|
|
|
673
673
|
---
|
|
674
674
|
name: use-local-dashboard
|
|
675
|
-
description: 'Run the zero-setup local Panoptic dashboard — a loopback `node:http` server over a trace store served from `ai.config({ panoptic: { dashboard } })`, with a two-pane call-tree + metadata UI (light/dark/system theme, search/filter, session + prompt-version grouping, cost heatmap, Gantt timeline, errors filter, hash deep-links) and an optional cache-backed persistent store. Triggers: `dashboard`, `DashboardOptions`, `DashboardHandle`, `ai.config({ panoptic })`, `panoptic.dashboard`, `panoptic.cache`, `port: 4319`, `basePath`, `open`, `authToken`, `allowedHosts`, `/api/traces`, `/api/aggregate`, `/api/traces/:id`, `createRequestHandler`, `dashboardHtml`, `sanitizeHref`, `ServeConfig`, `TraceStoreContract`, `createCacheTraceStore`, theme, dark mode, call tree, metadata panel, cost heatmap, timeline / Gantt, errors only, deep-link, group by session, prompt version, group by type, type stats panel, started / ended timestamps, team type; ''open a local AI trace dashboard'', ''view agent traces in the browser'', ''self-hosted panoptic UI'', ''inspect traces without Langfuse'', ''spin up a dev observability server'', ''dark mode dashboard'', ''persist traces across restart'', ''group traces by session / prompt version / type'', ''cost heatmap'', ''timeline / waterfall view'', ''filter failed runs in the dashboard'', ''per-type stats panel'', ''is the dashboard bearer token safe'', ''markdown links XSS in trace content''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"`. Skip: shipping traces to a backend (OTel / Langfuse / file) —
|
|
675
|
+
description: 'Run the zero-setup local Panoptic dashboard — a loopback `node:http` server over a trace store served from `ai.config({ panoptic: { dashboard } })`, with a two-pane call-tree + metadata UI (light/dark/system theme, search/filter, session + prompt-version grouping, cost heatmap, Gantt timeline, errors filter, hash deep-links) and an optional cache-backed persistent store. Triggers: `dashboard`, `DashboardOptions`, `DashboardHandle`, `ai.config({ panoptic })`, `panoptic.dashboard`, `panoptic.cache`, `port: 4319`, `basePath`, `open`, `authToken`, `allowedHosts`, `/api/traces`, `/api/aggregate`, `/api/traces/:id`, `createRequestHandler`, `dashboardHtml`, `sanitizeHref`, `ServeConfig`, `TraceStoreContract`, `createCacheTraceStore`, theme, dark mode, call tree, metadata panel, cost heatmap, timeline / Gantt, errors only, deep-link, group by session, prompt version, group by type, type stats panel, started / ended timestamps, team type; ''open a local AI trace dashboard'', ''view agent traces in the browser'', ''self-hosted panoptic UI'', ''inspect traces without Langfuse'', ''spin up a dev observability server'', ''dark mode dashboard'', ''persist traces across restart'', ''group traces by session / prompt version / type'', ''cost heatmap'', ''timeline / waterfall view'', ''filter failed runs in the dashboard'', ''per-type stats panel'', ''is the dashboard bearer token safe'', ''markdown links XSS in trace content''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"`. Skip: shipping traces to a backend (OTel / Langfuse / file) — the `export-traces` topic; the queryable store API the dashboard reads — the `query-traces` topic; wiring the subscriber by hand — the `observe-with-panoptic` topic.'
|
|
676
676
|
---
|
|
677
677
|
|
|
678
678
|
# The local Panoptic dashboard — zero-setup trace viewing
|
|
@@ -791,7 +791,7 @@ ai.config({
|
|
|
791
791
|
});
|
|
792
792
|
```
|
|
793
793
|
|
|
794
|
-
Panoptic awaits the store's `ready()` hydration **before** the server starts serving, so a restart surfaces previously-persisted traces immediately. Reads stay synchronous (served from an in-memory mirror); writes go through to the cache fire-and-forget, so a flaky cache never throws into the collector's hot path. A store-shaped exporter you pass via `exporters` still wins over `cache`. The store building block (`createCacheTraceStore`) and its durability contract live in
|
|
794
|
+
Panoptic awaits the store's `ready()` hydration **before** the server starts serving, so a restart surfaces previously-persisted traces immediately. Reads stay synchronous (served from an in-memory mirror); writes go through to the cache fire-and-forget, so a flaky cache never throws into the collector's hot path. A store-shaped exporter you pass via `exporters` still wins over `cache`. The store building block (`createCacheTraceStore`) and its durability contract live in the `query-traces` topic.
|
|
795
795
|
|
|
796
796
|
## The low-level building block — `dashboard(store, options)`
|
|
797
797
|
|
|
@@ -838,7 +838,7 @@ type DashboardHandle = {
|
|
|
838
838
|
|
|
839
839
|
## The JSON API
|
|
840
840
|
|
|
841
|
-
Every route is **`GET`-only and read-only** (non-`GET` → `405`), mounted under `basePath` — EXCEPT the one opt-in `POST .../spans/:spanId/evaluate` route `DashboardOptions.evaluate` enables (see
|
|
841
|
+
Every route is **`GET`-only and read-only** (non-`GET` → `405`), mounted under `basePath` — EXCEPT the one opt-in `POST .../spans/:spanId/evaluate` route `DashboardOptions.evaluate` enables (see the `evaluate-system-prompt` topic); absent that config, it 405s like everything else. The page polls the `GET` routes; you can curl them too. Store shapes are already JSON-safe, so responses are a plain `JSON.stringify`.
|
|
842
842
|
|
|
843
843
|
| Route | Returns |
|
|
844
844
|
|---|---|
|
|
@@ -855,7 +855,7 @@ curl 'http://127.0.0.1:4319/api/traces?status=failed'
|
|
|
855
855
|
curl 'http://127.0.0.1:4319/api/aggregate?sessionId=sess_42'
|
|
856
856
|
```
|
|
857
857
|
|
|
858
|
-
(The exact filter keys and the aggregate shape live in
|
|
858
|
+
(The exact filter keys and the aggregate shape live in the `query-traces` topic.)
|
|
859
859
|
|
|
860
860
|
## Advanced — build the handler yourself
|
|
861
861
|
|
|
@@ -887,9 +887,9 @@ These three findings (link sanitization + the two token-handling items above) ca
|
|
|
887
887
|
|
|
888
888
|
## See also
|
|
889
889
|
|
|
890
|
-
-
|
|
891
|
-
-
|
|
892
|
-
-
|
|
893
|
-
-
|
|
890
|
+
- The `observe-with-panoptic` topic — the `ai.config({ panoptic })` wiring and the underlying `panoptic(...)` subscriber that fills the store.
|
|
891
|
+
- The `query-traces` topic — the `TraceStoreContract` (`query` / `get` / `aggregate`) the dashboard reads, the `TraceQuery` filter, and capacity eviction.
|
|
892
|
+
- The `export-traces` topic — shipping traces to OTel / Langfuse / a file instead of (or alongside) the local dashboard.
|
|
893
|
+
- The `evaluate-system-prompt` topic — the drawer's "Evaluate system prompt" action, the dashboard's one write route.
|
|
894
894
|
|
|
895
895
|
|
package/llms.txt
CHANGED
|
@@ -6,8 +6,8 @@
|
|
|
6
6
|
|
|
7
7
|
## Skills
|
|
8
8
|
|
|
9
|
-
- [evaluate-system-prompt](@warlock.js/ai-panoptic/evaluate-system-prompt/SKILL.md): (no
|
|
10
|
-
- [export-traces](@warlock.js/ai-panoptic/export-traces/SKILL.md):
|
|
11
|
-
- [observe-with-panoptic](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md): (
|
|
12
|
-
- [query-traces](@warlock.js/ai-panoptic/query-traces/SKILL.md): (
|
|
13
|
-
- [use-local-dashboard](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md): (
|
|
9
|
+
- [evaluate-system-prompt](@warlock.js/ai-panoptic/evaluate-system-prompt/SKILL.md): Grade a single trace's LAST captured system prompt from the dashboard drawer — the dashboard's only write-capable feature, config-gated via `DashboardOptions.evaluate` (`ai.config({ panoptic: { dashboard: { evaluate: { model, instructions } } } })`). Triggers: `evaluate`, `EvaluateConfig`, `EvaluateVerdict`, `evaluateSystemPrompt`, `extractLastSystemPrompt`, `findSpanById`, "Evaluate system prompt" button, `POST .../spans/:spanId/evaluate`, grading a prompt from a real trace, judging a captured system prompt, per-run instructions override, judge model factory; 'evaluate this trace's prompt', 'grade the system prompt from the dashboard', 'judge a captured run', 'score a live system prompt'. Skip: grading raw prompt TEXT you already have in code (no trace involved) — `@warlock.js/ai`'s `ai.prompts().validate({ criteria })`, the same judge machinery this wraps; a DATASET of test cases run through an agent — `@warlock.js/ai`'s `ai.dataset` + `agent.eval`; viewing/filtering/grouping traces without grading them — the `use-local-dashboard` topic; capturing the content this feature reads — the `export-traces` topic (`captureContent`).
|
|
10
|
+
- [export-traces](@warlock.js/ai-panoptic/export-traces/SKILL.md): Send @warlock.js/ai-panoptic traces to an observability backend via pluggable exporters, including capturing prompt/response content onto spans. Triggers: `consoleExporter`, `fileExporter`, `otelExporter`, `langfuseExporter`, `ExporterContract`, `ContentCaptureOptions`, `captureContent`, `redactContent`, `fullHistory`, `ContentRedactor`, `toGenAiAttributes`, `walkSpans`, `totalCostUsd`, `GEN_AI_ATTRIBUTES`, `WARLOCK_ATTRIBUTES`, `reportToTrace`, `reportToSpan`, `extractSpanAttributes`, `normalizeError`, `formatSpanLine`, `createPanopticMiddleware`; 'export AI traces to OpenTelemetry', 'send traces to Langfuse', 'log traces to console / a file', 'capture the full prompt/conversation onto a span', 'gen_ai semantic conventions', 'write a custom exporter'; typical import `import { otelExporter, langfuseExporter, consoleExporter, fileExporter } from "@warlock.js/ai-panoptic"`. Skip: wiring the subscriber/collector into a run — the `observe-with-panoptic` topic; the core report shape — the `run-ai-agent` topic of the `warlock-js-ai` skill.
|
|
11
|
+
- [observe-with-panoptic](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md): Wire @warlock.js/ai-panoptic into an agent/workflow/supervisor/orchestrator — declaratively via `ai.config({ panoptic })` (registers panoptic on core's Observer seam) or with the one-call `panoptic({ exporters })` subscriber. Triggers: `panoptic`, `Panoptic`, `PanopticOptions`, `PanopticConfig`, `ai.config({ panoptic })`, `observeAll`, per-flow `observe`, `FlowObserveOption`, `registerObserver`, `Observer`, `.attach`, `.middleware`, `.collect`, `.toTrace`, `observe.attach(agent)`, `panoptic().middleware()`, `completedEvents`, `middlewareName`; 'observe a warlock agent', 'trace an agent run', 'attach observability to a workflow/supervisor', 'observe every flow by default', 'add a tracing middleware', 'collect an orchestrator turn report', 'wire OTel/Langfuse into my agent'; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"` (or `import { panoptic } from "@warlock.js/ai-panoptic"`). Skip: the zero-setup local dashboard — the `use-local-dashboard` topic; writing/choosing an exporter (the sink end) — the `export-traces` topic; the core report shape / running the agent itself — the `run-ai-agent` topic of the `warlock-js-ai` skill.
|
|
12
|
+
- [query-traces](@warlock.js/ai-panoptic/query-traces/SKILL.md): Retain @warlock.js/ai-panoptic traces in a queryable store — in-memory, or cache-backed so they survive a process restart — and slice them after the fact: get one run by id, list a session, filter failed runs in a window, roll up usage + cost. Triggers: `createInMemoryTraceStore`, `createCacheTraceStore`, `CacheTraceStoreHandle`, `CacheTraceStoreOptions`, `CacheDriverInput`, `store.ready`, `TraceStoreContract`, `TraceQuery`, `TraceAggregate`, `InMemoryTraceStoreOptions`, `store.query`, `store.aggregate`, `store.get`, `store.add`, `store.clear`, `store.size`, `capacity`, `prefix`, `onError`, `sumUsage`, `emptyUsage`, `matchTrace`; 'query collected traces', 'aggregate AI cost per session', 'how much did this session spend', 'list failed runs', 'retain traces in memory', 'persist traces across a restart', 'cache-backed trace store', 'Redis trace store', 'roll up token usage'; typical import `import { createInMemoryTraceStore, createCacheTraceStore } from "@warlock.js/ai-panoptic"`. Skip: sending traces to an external backend (OTel/Langfuse/console/file) — the `export-traces` topic; wiring the subscriber into a run — the `observe-with-panoptic` topic.
|
|
13
|
+
- [use-local-dashboard](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md): Run the zero-setup local Panoptic dashboard — a loopback `node:http` server over a trace store served from `ai.config({ panoptic: { dashboard } })`, with a two-pane call-tree + metadata UI (light/dark/system theme, search/filter, session + prompt-version grouping, cost heatmap, Gantt timeline, errors filter, hash deep-links) and an optional cache-backed persistent store. Triggers: `dashboard`, `DashboardOptions`, `DashboardHandle`, `ai.config({ panoptic })`, `panoptic.dashboard`, `panoptic.cache`, `port: 4319`, `basePath`, `open`, `authToken`, `allowedHosts`, `/api/traces`, `/api/aggregate`, `/api/traces/:id`, `createRequestHandler`, `dashboardHtml`, `sanitizeHref`, `ServeConfig`, `TraceStoreContract`, `createCacheTraceStore`, theme, dark mode, call tree, metadata panel, cost heatmap, timeline / Gantt, errors only, deep-link, group by session, prompt version, group by type, type stats panel, started / ended timestamps, team type; 'open a local AI trace dashboard', 'view agent traces in the browser', 'self-hosted panoptic UI', 'inspect traces without Langfuse', 'spin up a dev observability server', 'dark mode dashboard', 'persist traces across restart', 'group traces by session / prompt version / type', 'cost heatmap', 'timeline / waterfall view', 'filter failed runs in the dashboard', 'per-type stats panel', 'is the dashboard bearer token safe', 'markdown links XSS in trace content'; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"`. Skip: shipping traces to a backend (OTel / Langfuse / file) — the `export-traces` topic; the queryable store API the dashboard reads — the `query-traces` topic; wiring the subscriber by hand — the `observe-with-panoptic` topic.
|
package/package.json
CHANGED
|
@@ -18,9 +18,9 @@
|
|
|
18
18
|
"peerDependencies": {
|
|
19
19
|
"@opentelemetry/api": "*",
|
|
20
20
|
"@opentelemetry/sdk-trace-base": "*",
|
|
21
|
-
"@warlock.js/ai": "5.
|
|
22
|
-
"@warlock.js/cache": "5.
|
|
23
|
-
"@warlock.js/logger": "5.
|
|
21
|
+
"@warlock.js/ai": "5.29.0",
|
|
22
|
+
"@warlock.js/cache": "5.29.0",
|
|
23
|
+
"@warlock.js/logger": "5.29.0",
|
|
24
24
|
"langfuse": "*"
|
|
25
25
|
},
|
|
26
26
|
"peerDependenciesMeta": {
|
|
@@ -40,7 +40,7 @@
|
|
|
40
40
|
"dependencies": {
|
|
41
41
|
"langfuse": "^3.38.20"
|
|
42
42
|
},
|
|
43
|
-
"version": "5.
|
|
43
|
+
"version": "5.29.0",
|
|
44
44
|
"main": "./cjs/index.cjs",
|
|
45
45
|
"module": "./esm/index.mjs",
|
|
46
46
|
"types": "./esm/index.d.mts",
|
package/skills/README.md
CHANGED
|
@@ -1,25 +1,25 @@
|
|
|
1
1
|
# `@warlock.js/ai-panoptic` — skills index
|
|
2
2
|
|
|
3
|
-
Per-task skills.
|
|
3
|
+
Per-task skills. Cross-references name the topic ("the `<topic>` topic"), and for another package add its skill ("the `<topic>` topic of the `warlock-js-<pkg>` skill").
|
|
4
4
|
|
|
5
5
|
## Skills
|
|
6
6
|
|
|
7
|
-
###
|
|
7
|
+
### `observe-with-panoptic`
|
|
8
8
|
|
|
9
9
|
Wire Panoptic into a run — declaratively via `ai.config({ panoptic })` (registers panoptic on core's `Observer` seam; `observeAll` + per-flow `observe` decide which workflow/supervisor/team runs route in), or imperatively with the one-call `panoptic({ exporters })` subscriber (`attach()` to an event stream, `middleware()` for the agent pipeline, or `collect()` a report directly — the orchestrator turn). Covers the side-effect import, the three imperative feed paths, the default `*.completed` event surface and `completedEvents` override, failure isolation, `flush()` / `shutdown()`, and bringing your own collector/store. Load when attaching observability to a primitive, observing every flow by default, adding a tracing middleware, or collecting an orchestrator turn report.
|
|
10
10
|
|
|
11
|
-
###
|
|
11
|
+
### `export-traces`
|
|
12
12
|
|
|
13
13
|
Send Panoptic traces to an observability backend via pluggable exporters — `consoleExporter` / `fileExporter` (zero-dep) and `otelExporter` (OpenTelemetry GenAI semantic conventions) / `langfuseExporter` (both lazily import their optional-peer SDK). Covers picking an exporter, the `ExporterContract`, the `gen_ai.*` / `warlock.*` attribute mapping, opt-in content capture (`captureContent` / `redactContent` / `fullHistory` to emit the whole `CapturedMessage[]`), the shared `walkSpans` / `toGenAiAttributes` / `totalCostUsd` utilities, and authoring a custom exporter. Load when wiring `collector.use(...)` with an exporter, exporting to OTel / Langfuse / console / a file, capturing prompt/response content onto spans, or writing a custom sink.
|
|
14
14
|
|
|
15
|
-
###
|
|
15
|
+
### `query-traces`
|
|
16
16
|
|
|
17
17
|
Retain traces in a queryable store — in-memory (`createInMemoryTraceStore`) or **cache-backed and restart-durable** (`createCacheTraceStore` over any `@warlock.js/cache` `CacheDriver` / async factory, with a synchronous in-memory read mirror, write-through persistence, `ready()` hydration, `prefix` / `capacity` / `onError`) — and slice them after the fact: `get` / `query` / `aggregate` / `clear` / `size`, the `TraceQuery` filter (id / session / status / time window), the `TraceAggregate` usage + cost + status rollup, `capacity` FIFO eviction, and the pure `sumUsage` / `emptyUsage` / `matchTrace` helpers. Load when querying collected traces, aggregating AI cost/usage per session, listing failed runs, retaining traces in memory, or persisting them across a restart.
|
|
18
18
|
|
|
19
|
-
###
|
|
19
|
+
### `use-local-dashboard`
|
|
20
20
|
|
|
21
21
|
Run the zero-setup local Panoptic dashboard — a loopback `node:http` server over a trace store served from `ai.config({ panoptic: { dashboard } })`. Covers the redesigned UI (light/dark/system theme, two-pane nested call tree + metadata panel, colour-coded Title-case type labels including the first-class `Team` type, started/ended wall-clock timestamps, arrow-coded `↓in · ↑out · total` tokens, per-node rollup cost, hide-zero rows, Warlock logo), the new dashboard features (client-side search/filter, group-by-session + group-by-prompt + group-by-type, per-type stats panel, cost heatmap, Gantt timeline, errors-only toggle, hash deep-links) and their pure `trace-filter.ts` helpers (`filterTraces` / `groupBySession` / `groupByPrompt` / `rollupCost` / `maxNodeCost` / `heatIntensity` / `tracePromptKey`), the persistent `cache` store wiring, the declarative path (which store it reads, idempotent wiring), the low-level `dashboard(store, options)` building block, `DashboardOptions` (port `4319`, loopback host, `basePath`, `open`, `title`), the `DashboardHandle`, the read-only `/api/traces` / `/api/aggregate` / `/api/traces/:id` JSON API, `createRequestHandler` / `dashboardHtml` for embedding, prompt-version linkage, and the loopback-only security posture. Load when opening a local AI trace viewer in the browser, theming/filtering it, inspecting traces without a backend, persisting them across a restart, or spinning up a dev observability server.
|
|
22
22
|
|
|
23
|
-
###
|
|
23
|
+
### `evaluate-system-prompt`
|
|
24
24
|
|
|
25
25
|
Grade a trace's LAST captured system prompt from the drawer — the dashboard's only write-capable feature, config-gated via `DashboardOptions.evaluate` (`{ model, instructions }`). Covers `evaluateSystemPrompt` / `extractLastSystemPrompt` / `findSpanById`, the "Evaluate system prompt" button + editable-instructions panel, the `POST .../spans/:spanId/evaluate` route and its status codes, and how it reuses `@warlock.js/ai`'s `judgePromptBody` (the same machinery `ai.prompts().validate()` runs) rather than a second judging implementation. Load when grading a live/captured system prompt from the dashboard, wiring `evaluate` into `ai.config`, or embedding the evaluate building blocks elsewhere.
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: evaluate-system-prompt
|
|
3
|
-
description: 'Grade a single trace''s LAST captured system prompt from the dashboard drawer — the dashboard''s only write-capable feature, config-gated via `DashboardOptions.evaluate` (`ai.config({ panoptic: { dashboard: { evaluate: { model, instructions } } } })`). Triggers: `evaluate`, `EvaluateConfig`, `EvaluateVerdict`, `evaluateSystemPrompt`, `extractLastSystemPrompt`, `findSpanById`, "Evaluate system prompt" button, `POST .../spans/:spanId/evaluate`, grading a prompt from a real trace, judging a captured system prompt, per-run instructions override, judge model factory; ''evaluate this trace''s prompt'', ''grade the system prompt from the dashboard'', ''judge a captured run'', ''score a live system prompt''. Skip: grading raw prompt TEXT you already have in code (no trace involved) — `@warlock.js/ai`''s `ai.prompts().validate({ criteria })`, the same judge machinery this wraps; a DATASET of test cases run through an agent — `@warlock.js/ai`''s `ai.dataset` + `agent.eval`; viewing/filtering/grouping traces without grading them —
|
|
3
|
+
description: 'Grade a single trace''s LAST captured system prompt from the dashboard drawer — the dashboard''s only write-capable feature, config-gated via `DashboardOptions.evaluate` (`ai.config({ panoptic: { dashboard: { evaluate: { model, instructions } } } })`). Triggers: `evaluate`, `EvaluateConfig`, `EvaluateVerdict`, `evaluateSystemPrompt`, `extractLastSystemPrompt`, `findSpanById`, "Evaluate system prompt" button, `POST .../spans/:spanId/evaluate`, grading a prompt from a real trace, judging a captured system prompt, per-run instructions override, judge model factory; ''evaluate this trace''s prompt'', ''grade the system prompt from the dashboard'', ''judge a captured run'', ''score a live system prompt''. Skip: grading raw prompt TEXT you already have in code (no trace involved) — `@warlock.js/ai`''s `ai.prompts().validate({ criteria })`, the same judge machinery this wraps; a DATASET of test cases run through an agent — `@warlock.js/ai`''s `ai.dataset` + `agent.eval`; viewing/filtering/grouping traces without grading them — the `use-local-dashboard` topic; capturing the content this feature reads — the `export-traces` topic (`captureContent`).'
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Evaluate a trace's system prompt from the dashboard
|
|
7
7
|
|
|
8
|
-
A drawer button that grades the LAST `{role: "system"}` message captured on the selected span — not the whole conversation, not the tool code, just the system prompt that was actually in effect for that run. It's the dashboard's only write-capable feature: every other route is `GET`-only and read-only by design (see `use-local-dashboard
|
|
8
|
+
A drawer button that grades the LAST `{role: "system"}` message captured on the selected span — not the whole conversation, not the tool code, just the system prompt that was actually in effect for that run. It's the dashboard's only write-capable feature: every other route is `GET`-only and read-only by design (see the `use-local-dashboard` topic's security note); this one POSTs, and it's off unless you explicitly opt in.
|
|
9
9
|
|
|
10
10
|
## Enable it
|
|
11
11
|
|
|
@@ -81,6 +81,6 @@ curl -X POST 'http://127.0.0.1:4319/api/traces/tr_1/spans/sp_1/evaluate' \
|
|
|
81
81
|
|
|
82
82
|
## See also
|
|
83
83
|
|
|
84
|
-
-
|
|
85
|
-
-
|
|
84
|
+
- The `use-local-dashboard` topic — the dashboard this feature is a drawer action of; its security note now says "every route is GET-only and read-only EXCEPT the one opt-in write route `evaluate` enables."
|
|
85
|
+
- The `export-traces` topic — `captureContent` (and `fullHistory`), the prerequisite for `span.input` to carry anything gradable.
|
|
86
86
|
- `ai.prompts().validate({ criteria })` in `@warlock.js/ai` — the sibling capability this reuses: grading raw prompt TEXT you already have in code, no trace or dashboard involved.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: export-traces
|
|
3
|
-
description: 'Send @warlock.js/ai-panoptic traces to an observability backend via pluggable exporters, including capturing prompt/response content onto spans. Triggers: `consoleExporter`, `fileExporter`, `otelExporter`, `langfuseExporter`, `ExporterContract`, `ContentCaptureOptions`, `captureContent`, `redactContent`, `fullHistory`, `ContentRedactor`, `toGenAiAttributes`, `walkSpans`, `totalCostUsd`, `GEN_AI_ATTRIBUTES`, `WARLOCK_ATTRIBUTES`, `reportToTrace`, `reportToSpan`, `extractSpanAttributes`, `normalizeError`, `formatSpanLine`, `createPanopticMiddleware`; ''export AI traces to OpenTelemetry'', ''send traces to Langfuse'', ''log traces to console / a file'', ''capture the full prompt/conversation onto a span'', ''gen_ai semantic conventions'', ''write a custom exporter''; typical import `import { otelExporter, langfuseExporter, consoleExporter, fileExporter } from "@warlock.js/ai-panoptic"`. Skip: wiring the subscriber/collector into a run —
|
|
3
|
+
description: 'Send @warlock.js/ai-panoptic traces to an observability backend via pluggable exporters, including capturing prompt/response content onto spans. Triggers: `consoleExporter`, `fileExporter`, `otelExporter`, `langfuseExporter`, `ExporterContract`, `ContentCaptureOptions`, `captureContent`, `redactContent`, `fullHistory`, `ContentRedactor`, `toGenAiAttributes`, `walkSpans`, `totalCostUsd`, `GEN_AI_ATTRIBUTES`, `WARLOCK_ATTRIBUTES`, `reportToTrace`, `reportToSpan`, `extractSpanAttributes`, `normalizeError`, `formatSpanLine`, `createPanopticMiddleware`; ''export AI traces to OpenTelemetry'', ''send traces to Langfuse'', ''log traces to console / a file'', ''capture the full prompt/conversation onto a span'', ''gen_ai semantic conventions'', ''write a custom exporter''; typical import `import { otelExporter, langfuseExporter, consoleExporter, fileExporter } from "@warlock.js/ai-panoptic"`. Skip: wiring the subscriber/collector into a run — the `observe-with-panoptic` topic; the core report shape — the `run-ai-agent` topic of the `warlock-js-ai` skill.'
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Export traces — the sink end of Panoptic
|
|
@@ -179,6 +179,6 @@ Reach for `reportToTrace` when you already hold a finished `result.report` and w
|
|
|
179
179
|
|
|
180
180
|
## See also
|
|
181
181
|
|
|
182
|
-
-
|
|
183
|
-
-
|
|
184
|
-
-
|
|
182
|
+
- The `observe-with-panoptic` topic — the source end: wiring the `panoptic(...)` subscriber into a run (`attach` / `middleware` / `collect`). The `collector` these exporters register on comes from there.
|
|
183
|
+
- The `query-traces` topic — the in-memory trace store: a sibling sink that *retains* traces (instead of shipping them out) so you can `query` / `aggregate` them after the fact.
|
|
184
|
+
- The `run-ai-agent` topic of the `warlock-js-ai` skill — running the agent and the `BaseReport` shape Panoptic projects into a `Trace`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: observe-with-panoptic
|
|
3
|
-
description: 'Wire @warlock.js/ai-panoptic into an agent/workflow/supervisor/orchestrator — declaratively via `ai.config({ panoptic })` (registers panoptic on core''s Observer seam) or with the one-call `panoptic({ exporters })` subscriber. Triggers: `panoptic`, `Panoptic`, `PanopticOptions`, `PanopticConfig`, `ai.config({ panoptic })`, `observeAll`, per-flow `observe`, `FlowObserveOption`, `registerObserver`, `Observer`, `.attach`, `.middleware`, `.collect`, `.toTrace`, `observe.attach(agent)`, `panoptic().middleware()`, `completedEvents`, `middlewareName`; ''observe a warlock agent'', ''trace an agent run'', ''attach observability to a workflow/supervisor'', ''observe every flow by default'', ''add a tracing middleware'', ''collect an orchestrator turn report'', ''wire OTel/Langfuse into my agent''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"` (or `import { panoptic } from "@warlock.js/ai-panoptic"`). Skip: the zero-setup local dashboard —
|
|
3
|
+
description: 'Wire @warlock.js/ai-panoptic into an agent/workflow/supervisor/orchestrator — declaratively via `ai.config({ panoptic })` (registers panoptic on core''s Observer seam) or with the one-call `panoptic({ exporters })` subscriber. Triggers: `panoptic`, `Panoptic`, `PanopticOptions`, `PanopticConfig`, `ai.config({ panoptic })`, `observeAll`, per-flow `observe`, `FlowObserveOption`, `registerObserver`, `Observer`, `.attach`, `.middleware`, `.collect`, `.toTrace`, `observe.attach(agent)`, `panoptic().middleware()`, `completedEvents`, `middlewareName`; ''observe a warlock agent'', ''trace an agent run'', ''attach observability to a workflow/supervisor'', ''observe every flow by default'', ''add a tracing middleware'', ''collect an orchestrator turn report'', ''wire OTel/Langfuse into my agent''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"` (or `import { panoptic } from "@warlock.js/ai-panoptic"`). Skip: the zero-setup local dashboard — the `use-local-dashboard` topic; writing/choosing an exporter (the sink end) — the `export-traces` topic; the core report shape / running the agent itself — the `run-ai-agent` topic of the `warlock-js-ai` skill.'
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Observe a run with Panoptic — the one-call subscriber
|
|
@@ -165,11 +165,11 @@ const observe = panoptic({ collector, exporters: [consoleExporter()] });
|
|
|
165
165
|
|
|
166
166
|
Add exporters after construction with `observe.use(exporter)` (chainable, deduped by `ExporterContract.name`). `observe.toTrace(report)` projects a report into a `Trace` **without** dispatching — handy for inspection or tests.
|
|
167
167
|
|
|
168
|
-
> The store's full query / aggregate / capacity surface is its own skill — see
|
|
168
|
+
> The store's full query / aggregate / capacity surface is its own skill — see the `query-traces` topic.
|
|
169
169
|
|
|
170
170
|
## See also
|
|
171
171
|
|
|
172
|
-
-
|
|
173
|
-
-
|
|
174
|
-
-
|
|
175
|
-
-
|
|
172
|
+
- The `use-local-dashboard` topic — the zero-setup local viewer the `ai.config({ panoptic: { dashboard } })` switch starts over the collector's store.
|
|
173
|
+
- The `export-traces` topic — the sink end: choosing/writing an exporter, the `gen_ai.*` / `warlock.*` attribute mapping.
|
|
174
|
+
- The `query-traces` topic — retaining traces in the queryable in-memory store and slicing them (`query` / `aggregate` / `capacity`).
|
|
175
|
+
- The `run-ai-agent` topic of the `warlock-js-ai` skill — running the agent and the `BaseReport` shape Panoptic consumes.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: query-traces
|
|
3
|
-
description: 'Retain @warlock.js/ai-panoptic traces in a queryable store — in-memory, or cache-backed so they survive a process restart — and slice them after the fact: get one run by id, list a session, filter failed runs in a window, roll up usage + cost. Triggers: `createInMemoryTraceStore`, `createCacheTraceStore`, `CacheTraceStoreHandle`, `CacheTraceStoreOptions`, `CacheDriverInput`, `store.ready`, `TraceStoreContract`, `TraceQuery`, `TraceAggregate`, `InMemoryTraceStoreOptions`, `store.query`, `store.aggregate`, `store.get`, `store.add`, `store.clear`, `store.size`, `capacity`, `prefix`, `onError`, `sumUsage`, `emptyUsage`, `matchTrace`; ''query collected traces'', ''aggregate AI cost per session'', ''how much did this session spend'', ''list failed runs'', ''retain traces in memory'', ''persist traces across a restart'', ''cache-backed trace store'', ''Redis trace store'', ''roll up token usage''; typical import `import { createInMemoryTraceStore, createCacheTraceStore } from "@warlock.js/ai-panoptic"`. Skip: sending traces to an external backend (OTel/Langfuse/console/file) —
|
|
3
|
+
description: 'Retain @warlock.js/ai-panoptic traces in a queryable store — in-memory, or cache-backed so they survive a process restart — and slice them after the fact: get one run by id, list a session, filter failed runs in a window, roll up usage + cost. Triggers: `createInMemoryTraceStore`, `createCacheTraceStore`, `CacheTraceStoreHandle`, `CacheTraceStoreOptions`, `CacheDriverInput`, `store.ready`, `TraceStoreContract`, `TraceQuery`, `TraceAggregate`, `InMemoryTraceStoreOptions`, `store.query`, `store.aggregate`, `store.get`, `store.add`, `store.clear`, `store.size`, `capacity`, `prefix`, `onError`, `sumUsage`, `emptyUsage`, `matchTrace`; ''query collected traces'', ''aggregate AI cost per session'', ''how much did this session spend'', ''list failed runs'', ''retain traces in memory'', ''persist traces across a restart'', ''cache-backed trace store'', ''Redis trace store'', ''roll up token usage''; typical import `import { createInMemoryTraceStore, createCacheTraceStore } from "@warlock.js/ai-panoptic"`. Skip: sending traces to an external backend (OTel/Langfuse/console/file) — the `export-traces` topic; wiring the subscriber into a run — the `observe-with-panoptic` topic.'
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Query traces — the in-memory queryable store
|
|
@@ -27,7 +27,7 @@ const failed = store.query({ status: "failed" });
|
|
|
27
27
|
const spend = store.aggregate({ sessionId: "session-42" });
|
|
28
28
|
```
|
|
29
29
|
|
|
30
|
-
> The store is also a valid `collector.use(...)` target directly — `panoptic({ collector: createCollector().use(store) })` — see
|
|
30
|
+
> The store is also a valid `collector.use(...)` target directly — `panoptic({ collector: createCollector().use(store) })` — see the `observe-with-panoptic` topic for bring-your-own-collector wiring.
|
|
31
31
|
|
|
32
32
|
## The store contract
|
|
33
33
|
|
|
@@ -128,7 +128,7 @@ const store = createInMemoryTraceStore({ capacity: 1000 });
|
|
|
128
128
|
|
|
129
129
|
Overwriting an existing `traceId` refreshes its insertion position, so a re-collected run is treated as newest for eviction — eviction stays honest under overwrite.
|
|
130
130
|
|
|
131
|
-
> The in-memory store has no secondary indexes: `get` / `add` are O(1), `query` / `aggregate` are O(n) scans. That is the right trade-off for dev/test and modest-volume runtime use. For high-volume or durable retention, write a custom `TraceStoreContract` (or export to a backend with
|
|
131
|
+
> The in-memory store has no secondary indexes: `get` / `add` are O(1), `query` / `aggregate` are O(n) scans. That is the right trade-off for dev/test and modest-volume runtime use. For high-volume or durable retention, write a custom `TraceStoreContract` (or export to a backend with the `export-traces` topic) backed by a real datastore.
|
|
132
132
|
|
|
133
133
|
## Persisting across a restart — `createCacheTraceStore`
|
|
134
134
|
|
|
@@ -179,7 +179,7 @@ type CacheTraceStoreHandle = TraceStoreContract & ExporterContract & {
|
|
|
179
179
|
|
|
180
180
|
Ordering uses an **internal monotonic counter**, not `Date.now()`, so the persisted index stays a stable total order even when many traces land in the same millisecond — FIFO eviction stays honest across a restart.
|
|
181
181
|
|
|
182
|
-
> The usual path is declarative: `ai.config({ panoptic: { cache, dashboard: true } })` constructs this store, awaits `ready()`, and wires it under the dashboard for you — see
|
|
182
|
+
> The usual path is declarative: `ai.config({ panoptic: { cache, dashboard: true } })` constructs this store, awaits `ready()`, and wires it under the dashboard for you — see the `use-local-dashboard` topic. Reach for `createCacheTraceStore` directly only when you manage the store / collector yourself.
|
|
183
183
|
|
|
184
184
|
## Summing usage yourself
|
|
185
185
|
|
|
@@ -198,6 +198,6 @@ for (const trace of store.query({ sessionId })) {
|
|
|
198
198
|
|
|
199
199
|
## See also
|
|
200
200
|
|
|
201
|
-
-
|
|
202
|
-
-
|
|
203
|
-
-
|
|
201
|
+
- The `observe-with-panoptic` topic — wiring the `panoptic(...)` subscriber and bring-your-own-collector so the store fills as runs complete.
|
|
202
|
+
- The `export-traces` topic — sending traces to an external backend instead of (or alongside) retaining them in memory; writing a custom sink.
|
|
203
|
+
- The `run-ai-agent` topic of the `warlock-js-ai` skill — the `BaseReport` the collector projects into the `Trace` this store retains.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: use-local-dashboard
|
|
3
|
-
description: 'Run the zero-setup local Panoptic dashboard — a loopback `node:http` server over a trace store served from `ai.config({ panoptic: { dashboard } })`, with a two-pane call-tree + metadata UI (light/dark/system theme, search/filter, session + prompt-version grouping, cost heatmap, Gantt timeline, errors filter, hash deep-links) and an optional cache-backed persistent store. Triggers: `dashboard`, `DashboardOptions`, `DashboardHandle`, `ai.config({ panoptic })`, `panoptic.dashboard`, `panoptic.cache`, `port: 4319`, `basePath`, `open`, `authToken`, `allowedHosts`, `/api/traces`, `/api/aggregate`, `/api/traces/:id`, `createRequestHandler`, `dashboardHtml`, `sanitizeHref`, `ServeConfig`, `TraceStoreContract`, `createCacheTraceStore`, theme, dark mode, call tree, metadata panel, cost heatmap, timeline / Gantt, errors only, deep-link, group by session, prompt version, group by type, type stats panel, started / ended timestamps, team type; ''open a local AI trace dashboard'', ''view agent traces in the browser'', ''self-hosted panoptic UI'', ''inspect traces without Langfuse'', ''spin up a dev observability server'', ''dark mode dashboard'', ''persist traces across restart'', ''group traces by session / prompt version / type'', ''cost heatmap'', ''timeline / waterfall view'', ''filter failed runs in the dashboard'', ''per-type stats panel'', ''is the dashboard bearer token safe'', ''markdown links XSS in trace content''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"`. Skip: shipping traces to a backend (OTel / Langfuse / file) —
|
|
3
|
+
description: 'Run the zero-setup local Panoptic dashboard — a loopback `node:http` server over a trace store served from `ai.config({ panoptic: { dashboard } })`, with a two-pane call-tree + metadata UI (light/dark/system theme, search/filter, session + prompt-version grouping, cost heatmap, Gantt timeline, errors filter, hash deep-links) and an optional cache-backed persistent store. Triggers: `dashboard`, `DashboardOptions`, `DashboardHandle`, `ai.config({ panoptic })`, `panoptic.dashboard`, `panoptic.cache`, `port: 4319`, `basePath`, `open`, `authToken`, `allowedHosts`, `/api/traces`, `/api/aggregate`, `/api/traces/:id`, `createRequestHandler`, `dashboardHtml`, `sanitizeHref`, `ServeConfig`, `TraceStoreContract`, `createCacheTraceStore`, theme, dark mode, call tree, metadata panel, cost heatmap, timeline / Gantt, errors only, deep-link, group by session, prompt version, group by type, type stats panel, started / ended timestamps, team type; ''open a local AI trace dashboard'', ''view agent traces in the browser'', ''self-hosted panoptic UI'', ''inspect traces without Langfuse'', ''spin up a dev observability server'', ''dark mode dashboard'', ''persist traces across restart'', ''group traces by session / prompt version / type'', ''cost heatmap'', ''timeline / waterfall view'', ''filter failed runs in the dashboard'', ''per-type stats panel'', ''is the dashboard bearer token safe'', ''markdown links XSS in trace content''; typical import `import { ai } from "@warlock.js/ai"` + `import "@warlock.js/ai-panoptic"`. Skip: shipping traces to a backend (OTel / Langfuse / file) — the `export-traces` topic; the queryable store API the dashboard reads — the `query-traces` topic; wiring the subscriber by hand — the `observe-with-panoptic` topic.'
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# The local Panoptic dashboard — zero-setup trace viewing
|
|
@@ -119,7 +119,7 @@ ai.config({
|
|
|
119
119
|
});
|
|
120
120
|
```
|
|
121
121
|
|
|
122
|
-
Panoptic awaits the store's `ready()` hydration **before** the server starts serving, so a restart surfaces previously-persisted traces immediately. Reads stay synchronous (served from an in-memory mirror); writes go through to the cache fire-and-forget, so a flaky cache never throws into the collector's hot path. A store-shaped exporter you pass via `exporters` still wins over `cache`. The store building block (`createCacheTraceStore`) and its durability contract live in
|
|
122
|
+
Panoptic awaits the store's `ready()` hydration **before** the server starts serving, so a restart surfaces previously-persisted traces immediately. Reads stay synchronous (served from an in-memory mirror); writes go through to the cache fire-and-forget, so a flaky cache never throws into the collector's hot path. A store-shaped exporter you pass via `exporters` still wins over `cache`. The store building block (`createCacheTraceStore`) and its durability contract live in the `query-traces` topic.
|
|
123
123
|
|
|
124
124
|
## The low-level building block — `dashboard(store, options)`
|
|
125
125
|
|
|
@@ -166,7 +166,7 @@ type DashboardHandle = {
|
|
|
166
166
|
|
|
167
167
|
## The JSON API
|
|
168
168
|
|
|
169
|
-
Every route is **`GET`-only and read-only** (non-`GET` → `405`), mounted under `basePath` — EXCEPT the one opt-in `POST .../spans/:spanId/evaluate` route `DashboardOptions.evaluate` enables (see
|
|
169
|
+
Every route is **`GET`-only and read-only** (non-`GET` → `405`), mounted under `basePath` — EXCEPT the one opt-in `POST .../spans/:spanId/evaluate` route `DashboardOptions.evaluate` enables (see the `evaluate-system-prompt` topic); absent that config, it 405s like everything else. The page polls the `GET` routes; you can curl them too. Store shapes are already JSON-safe, so responses are a plain `JSON.stringify`.
|
|
170
170
|
|
|
171
171
|
| Route | Returns |
|
|
172
172
|
|---|---|
|
|
@@ -183,7 +183,7 @@ curl 'http://127.0.0.1:4319/api/traces?status=failed'
|
|
|
183
183
|
curl 'http://127.0.0.1:4319/api/aggregate?sessionId=sess_42'
|
|
184
184
|
```
|
|
185
185
|
|
|
186
|
-
(The exact filter keys and the aggregate shape live in
|
|
186
|
+
(The exact filter keys and the aggregate shape live in the `query-traces` topic.)
|
|
187
187
|
|
|
188
188
|
## Advanced — build the handler yourself
|
|
189
189
|
|
|
@@ -215,7 +215,7 @@ These three findings (link sanitization + the two token-handling items above) ca
|
|
|
215
215
|
|
|
216
216
|
## See also
|
|
217
217
|
|
|
218
|
-
-
|
|
219
|
-
-
|
|
220
|
-
-
|
|
221
|
-
-
|
|
218
|
+
- The `observe-with-panoptic` topic — the `ai.config({ panoptic })` wiring and the underlying `panoptic(...)` subscriber that fills the store.
|
|
219
|
+
- The `query-traces` topic — the `TraceStoreContract` (`query` / `get` / `aggregate`) the dashboard reads, the `TraceQuery` filter, and capacity eviction.
|
|
220
|
+
- The `export-traces` topic — shipping traces to OTel / Langfuse / a file instead of (or alongside) the local dashboard.
|
|
221
|
+
- The `evaluate-system-prompt` topic — the drawer's "Evaluate system prompt" action, the dashboard's one write route.
|