@warlock.js/ai-panoptic 5.24.0 → 5.25.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/llms-full.txt +869 -869
- package/package.json +4 -4
- package/skills/evaluate-system-prompt/SKILL.md +86 -86
- package/skills/export-traces/SKILL.md +184 -184
- package/skills/observe-with-panoptic/SKILL.md +175 -175
- package/skills/query-traces/SKILL.md +203 -203
- package/skills/use-local-dashboard/SKILL.md +221 -221
|
@@ -1,203 +1,203 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: query-traces
|
|
3
|
-
description: 'Retain @warlock.js/ai-panoptic traces in a queryable store — in-memory, or cache-backed so they survive a process restart — and slice them after the fact: get one run by id, list a session, filter failed runs in a window, roll up usage + cost. Triggers: `createInMemoryTraceStore`, `createCacheTraceStore`, `CacheTraceStoreHandle`, `CacheTraceStoreOptions`, `CacheDriverInput`, `store.ready`, `TraceStoreContract`, `TraceQuery`, `TraceAggregate`, `InMemoryTraceStoreOptions`, `store.query`, `store.aggregate`, `store.get`, `store.add`, `store.clear`, `store.size`, `capacity`, `prefix`, `onError`, `sumUsage`, `emptyUsage`, `matchTrace`; ''query collected traces'', ''aggregate AI cost per session'', ''how much did this session spend'', ''list failed runs'', ''retain traces in memory'', ''persist traces across a restart'', ''cache-backed trace store'', ''Redis trace store'', ''roll up token usage''; typical import `import { createInMemoryTraceStore, createCacheTraceStore } from "@warlock.js/ai-panoptic"`. Skip: sending traces to an external backend (OTel/Langfuse/console/file) — `@warlock.js/ai-panoptic/export-traces/SKILL.md`; wiring the subscriber into a run — `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`.'
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Query traces — the in-memory queryable store
|
|
7
|
-
|
|
8
|
-
The collector fans every completed `Trace` out to its exporters. A **trace store** is one such sink that *retains* traces so you can answer observability questions after the fact: pull one run by id, list every run for a session, slice failed runs in a time window, or roll usage + cost across any of those slices.
|
|
9
|
-
|
|
10
|
-
`createInMemoryTraceStore()` returns an object that satisfies **both** `TraceStoreContract` and `ExporterContract` — so wiring it into a collector is a single `collector.use(store)` and the store fills as traces complete.
|
|
11
|
-
|
|
12
|
-
```ts
|
|
13
|
-
import { panoptic, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
|
|
14
|
-
|
|
15
|
-
const store = createInMemoryTraceStore();
|
|
16
|
-
|
|
17
|
-
// The store doubles as an exporter — register it like any other sink.
|
|
18
|
-
const observe = panoptic({ exporters: [store] });
|
|
19
|
-
|
|
20
|
-
const agent = ai.agent({ model });
|
|
21
|
-
observe.attach(agent);
|
|
22
|
-
|
|
23
|
-
await agent.execute("Summarize this");
|
|
24
|
-
|
|
25
|
-
// ...later, off the hot path:
|
|
26
|
-
const failed = store.query({ status: "failed" });
|
|
27
|
-
const spend = store.aggregate({ sessionId: "session-42" });
|
|
28
|
-
```
|
|
29
|
-
|
|
30
|
-
> The store is also a valid `collector.use(...)` target directly — `panoptic({ collector: createCollector().use(store) })` — see [`observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) for bring-your-own-collector wiring.
|
|
31
|
-
|
|
32
|
-
## The store contract
|
|
33
|
-
|
|
34
|
-
`TraceStoreContract` is the queryable surface; the in-memory implementation adds the `ExporterContract` `name` + `export` so it can be a sink.
|
|
35
|
-
|
|
36
|
-
```ts
|
|
37
|
-
interface TraceStoreContract {
|
|
38
|
-
add(trace: Trace): void; // ingest; overwrites same traceId
|
|
39
|
-
get(traceId: string): Trace | undefined; // one run by id
|
|
40
|
-
query(filter?: TraceQuery): Trace[]; // matching traces, newest-started first
|
|
41
|
-
aggregate(filter?: TraceQuery): TraceAggregate; // usage + cost + status rollup
|
|
42
|
-
readonly size: number; // retained count
|
|
43
|
-
clear(): void; // drop everything
|
|
44
|
-
}
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
- **`add` overwrites** any existing trace with the same `traceId` — re-collecting a run replaces rather than duplicates it.
|
|
48
|
-
- **`export(trace)` is an alias for `add`** — that is what makes the store a drop-in exporter (`store.name === "in-memory-trace-store"`).
|
|
49
|
-
- **`query` returns newest-started first** (sorted by root `startedAt`, descending). An empty / omitted filter returns every retained trace.
|
|
50
|
-
|
|
51
|
-
## Filtering with `TraceQuery`
|
|
52
|
-
|
|
53
|
-
Every field is optional and **ANDed** together — an absent field is "don't care", so `{}` matches everything. The same filter type drives both `query` and `aggregate`.
|
|
54
|
-
|
|
55
|
-
```ts
|
|
56
|
-
type TraceQuery = {
|
|
57
|
-
traceId?: string; // exact one run
|
|
58
|
-
sessionId?: string; // every run for a session
|
|
59
|
-
status?: ReportStatus | ReportStatus[]; // single status or membership set
|
|
60
|
-
startedAfter?: string | Date; // inclusive lower bound on root startedAt
|
|
61
|
-
startedBefore?: string | Date; // inclusive upper bound on root startedAt
|
|
62
|
-
};
|
|
63
|
-
```
|
|
64
|
-
|
|
65
|
-
```ts
|
|
66
|
-
// One run by id:
|
|
67
|
-
store.query({ traceId: "run-abc" });
|
|
68
|
-
|
|
69
|
-
// Every run for a conversation:
|
|
70
|
-
store.query({ sessionId: "session-42" });
|
|
71
|
-
|
|
72
|
-
// Failed or cancelled runs since midnight (status accepts an array):
|
|
73
|
-
store.query({
|
|
74
|
-
status: ["failed", "cancelled"],
|
|
75
|
-
startedAfter: "2026-06-18T00:00:00.000Z",
|
|
76
|
-
});
|
|
77
|
-
|
|
78
|
-
// A Date works anywhere a time bound is accepted:
|
|
79
|
-
store.query({ startedAfter: new Date(Date.now() - 3_600_000) }); // last hour
|
|
80
|
-
```
|
|
81
|
-
|
|
82
|
-
Time bounds compare against each trace's root `startedAt`, **inclusive on both ends**, accepting either an ISO-8601 string or a `Date`. Status is exact equality (or array membership). The pure matcher behind this is `matchTrace(trace, filter)` — exported for testing or filtering a trace list you hold yourself.
|
|
83
|
-
|
|
84
|
-
## Rolling up with `aggregate`
|
|
85
|
-
|
|
86
|
-
`aggregate(filter?)` answers the headline questions — how many runs, how many tokens, how much did it cost, how many failed — for whatever slice the filter selects:
|
|
87
|
-
|
|
88
|
-
```ts
|
|
89
|
-
type TraceAggregate = {
|
|
90
|
-
traces: number; // matched count
|
|
91
|
-
completed: number; // root status === "completed"
|
|
92
|
-
failed: number; // root status === "failed"
|
|
93
|
-
cancelled: number; // root status === "cancelled"
|
|
94
|
-
usage: Usage; // summed token usage (input/output/total always present)
|
|
95
|
-
cost?: ModelPricing; // merged per-channel USD, or undefined if no trace was priced
|
|
96
|
-
totalDuration: number; // summed wall-clock ms
|
|
97
|
-
};
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
```ts
|
|
101
|
-
const stats = store.aggregate({ sessionId: "session-42" });
|
|
102
|
-
|
|
103
|
-
console.log(`${stats.traces} runs, ${stats.usage.total} tokens, ${stats.failed} failed`);
|
|
104
|
-
|
|
105
|
-
const totalUsd =
|
|
106
|
-
(stats.cost?.input ?? 0) +
|
|
107
|
-
(stats.cost?.output ?? 0) +
|
|
108
|
-
(stats.cost?.cachedInput ?? 0) +
|
|
109
|
-
(stats.cost?.cachedOutput ?? 0);
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
Notes that matter when reading the numbers:
|
|
113
|
-
|
|
114
|
-
- **`usage`** sums each trace's *root* `Usage` (already a rollup of own cost + children), so the totals reflect the whole run tree without re-walking spans. `input` / `output` / `total` are always present (zero when nothing matched); the optional `cachedTokens` / `cacheWriteTokens` / `reasoningTokens` channels are summed only when at least one matched trace reported them.
|
|
115
|
-
- **`cost`** is a per-channel `ModelPricing` breakdown (not one opaque number), merged with the framework's own cost-rollup logic — an unpriced trace never erases the cost of priced ones. It stays `undefined` only when **no** matched trace carried pricing.
|
|
116
|
-
- **`completed` + `failed` + `cancelled` need not equal `traces`** — non-terminal statuses (`awaiting-input`, `max-iterations`) count toward `traces` but none of the three headline counters.
|
|
117
|
-
- An empty match returns a zeroed aggregate (`usage` all `0`, `cost` `undefined`, `totalDuration` `0`).
|
|
118
|
-
|
|
119
|
-
## Bounding retention with `capacity`
|
|
120
|
-
|
|
121
|
-
Unbounded by default (keep everything until `clear()`). For a long-lived process, cap it — the oldest-ingested trace is evicted FIFO once the cap is exceeded:
|
|
122
|
-
|
|
123
|
-
```ts
|
|
124
|
-
type InMemoryTraceStoreOptions = { capacity?: number }; // absent / 0 = unbounded
|
|
125
|
-
|
|
126
|
-
const store = createInMemoryTraceStore({ capacity: 1000 });
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
Overwriting an existing `traceId` refreshes its insertion position, so a re-collected run is treated as newest for eviction — eviction stays honest under overwrite.
|
|
130
|
-
|
|
131
|
-
> The in-memory store has no secondary indexes: `get` / `add` are O(1), `query` / `aggregate` are O(n) scans. That is the right trade-off for dev/test and modest-volume runtime use. For high-volume or durable retention, write a custom `TraceStoreContract` (or export to a backend with [`export-traces`](@warlock.js/ai-panoptic/export-traces/SKILL.md)) backed by a real datastore.
|
|
132
|
-
|
|
133
|
-
## Persisting across a restart — `createCacheTraceStore`
|
|
134
|
-
|
|
135
|
-
The in-memory store loses everything when the process exits. `createCacheTraceStore(cache, options?)` is a **durable** `TraceStoreContract` (and `ExporterContract`) backed by any `@warlock.js/cache` `CacheDriver` — Redis, file, Postgres, … — so traces survive a restart and the dashboard re-opens with history intact.
|
|
136
|
-
|
|
137
|
-
```ts
|
|
138
|
-
import { createCacheTraceStore } from "@warlock.js/ai-panoptic";
|
|
139
|
-
import { RedisCacheDriver } from "@warlock.js/cache";
|
|
140
|
-
|
|
141
|
-
// A driver instance, or a (possibly async) FACTORY resolved lazily on first
|
|
142
|
-
// use — the factory form defers the Redis connect until the first trace is
|
|
143
|
-
// collected, so module import stays connection-free.
|
|
144
|
-
const store = createCacheTraceStore(async () => {
|
|
145
|
-
const driver = new RedisCacheDriver();
|
|
146
|
-
await driver.connect();
|
|
147
|
-
return driver;
|
|
148
|
-
});
|
|
149
|
-
|
|
150
|
-
await store.ready(); // hydrate the in-memory mirror from a prior run
|
|
151
|
-
collector.use(store); // fills as traces complete; reads stay synchronous
|
|
152
|
-
|
|
153
|
-
const failed = store.query({ status: "failed" }); // queryable like any store
|
|
154
|
-
```
|
|
155
|
-
|
|
156
|
-
**Write-through over a sync contract.** The store keeps the same insertion-ordered in-memory **mirror** the in-memory store uses and serves every read (`get` / `query` / `aggregate` / `size`) from it synchronously — the dashboard polls these and can't await a cache. Writes go **through**: `add` updates the mirror immediately, then *fire-and-forget* persists the trace + a newest-first index to the cache. So:
|
|
157
|
-
|
|
158
|
-
- **Reads never wait on the cache** — the mirror is the runtime source of truth; the cache is the durable backing store.
|
|
159
|
-
- **A flaky cache never throws into the hot path** — a rejected write is routed to the optional `onError` hook and swallowed; the trace stays visible in the mirror for the life of the process, it just won't survive a restart.
|
|
160
|
-
- **`ready()` re-hydrates** the mirror from the persisted index on startup (await it once — `ai.config({ panoptic: { cache } })` does this for you before serving the dashboard).
|
|
161
|
-
|
|
162
|
-
```ts
|
|
163
|
-
type CacheDriverInput =
|
|
164
|
-
| CacheDriver // a live driver
|
|
165
|
-
| (() => CacheDriver | Promise<CacheDriver>); // lazy (sync or async) factory
|
|
166
|
-
|
|
167
|
-
type CacheTraceStoreOptions = {
|
|
168
|
-
prefix?: string; // key namespace; per-trace keys `${prefix}:trace:${id}`,
|
|
169
|
-
// index at `${prefix}:index`. Default "panoptic".
|
|
170
|
-
capacity?: number; // FIFO cap (oldest-ingested evicted from cache + mirror);
|
|
171
|
-
// absent / 0 = unbounded.
|
|
172
|
-
onError?: (error: unknown) => void; // swallowed cache-write failures land here.
|
|
173
|
-
};
|
|
174
|
-
|
|
175
|
-
type CacheTraceStoreHandle = TraceStoreContract & ExporterContract & {
|
|
176
|
-
ready(): Promise<void>; // hydrate the mirror from the cache on startup
|
|
177
|
-
};
|
|
178
|
-
```
|
|
179
|
-
|
|
180
|
-
Ordering uses an **internal monotonic counter**, not `Date.now()`, so the persisted index stays a stable total order even when many traces land in the same millisecond — FIFO eviction stays honest across a restart.
|
|
181
|
-
|
|
182
|
-
> The usual path is declarative: `ai.config({ panoptic: { cache, dashboard: true } })` constructs this store, awaits `ready()`, and wires it under the dashboard for you — see [`use-local-dashboard/SKILL.md`](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md). Reach for `createCacheTraceStore` directly only when you manage the store / collector yourself.
|
|
183
|
-
|
|
184
|
-
## Summing usage yourself
|
|
185
|
-
|
|
186
|
-
`sumUsage` / `emptyUsage` are the pure folds `aggregate` is built on — reuse them to roll a `Usage` set the store didn't produce (e.g. a list of traces you filtered by hand):
|
|
187
|
-
|
|
188
|
-
```ts
|
|
189
|
-
import { emptyUsage, sumUsage } from "@warlock.js/ai-panoptic";
|
|
190
|
-
|
|
191
|
-
let total = emptyUsage(); // { input: 0, output: 0, total: 0 } — optional channels absent
|
|
192
|
-
for (const trace of store.query({ sessionId })) {
|
|
193
|
-
total = sumUsage(total, trace.usage);
|
|
194
|
-
}
|
|
195
|
-
```
|
|
196
|
-
|
|
197
|
-
`sumUsage` is pure (returns a fresh `Usage`, mutates neither argument): token channels always sum; the optional cache/reasoning channels sum only when a side reported them (so a never-metered channel stays absent rather than fabricating a `0`); `cost` merges with the framework's `accumulateCost` so an unpriced contributor never erases a priced one.
|
|
198
|
-
|
|
199
|
-
## See also
|
|
200
|
-
|
|
201
|
-
- [`@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) — wiring the `panoptic(...)` subscriber and bring-your-own-collector so the store fills as runs complete.
|
|
202
|
-
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — sending traces to an external backend instead of (or alongside) retaining them in memory; writing a custom sink.
|
|
203
|
-
- [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md) — the `BaseReport` the collector projects into the `Trace` this store retains.
|
|
1
|
+
---
|
|
2
|
+
name: query-traces
|
|
3
|
+
description: 'Retain @warlock.js/ai-panoptic traces in a queryable store — in-memory, or cache-backed so they survive a process restart — and slice them after the fact: get one run by id, list a session, filter failed runs in a window, roll up usage + cost. Triggers: `createInMemoryTraceStore`, `createCacheTraceStore`, `CacheTraceStoreHandle`, `CacheTraceStoreOptions`, `CacheDriverInput`, `store.ready`, `TraceStoreContract`, `TraceQuery`, `TraceAggregate`, `InMemoryTraceStoreOptions`, `store.query`, `store.aggregate`, `store.get`, `store.add`, `store.clear`, `store.size`, `capacity`, `prefix`, `onError`, `sumUsage`, `emptyUsage`, `matchTrace`; ''query collected traces'', ''aggregate AI cost per session'', ''how much did this session spend'', ''list failed runs'', ''retain traces in memory'', ''persist traces across a restart'', ''cache-backed trace store'', ''Redis trace store'', ''roll up token usage''; typical import `import { createInMemoryTraceStore, createCacheTraceStore } from "@warlock.js/ai-panoptic"`. Skip: sending traces to an external backend (OTel/Langfuse/console/file) — `@warlock.js/ai-panoptic/export-traces/SKILL.md`; wiring the subscriber into a run — `@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`.'
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Query traces — the in-memory queryable store
|
|
7
|
+
|
|
8
|
+
The collector fans every completed `Trace` out to its exporters. A **trace store** is one such sink that *retains* traces so you can answer observability questions after the fact: pull one run by id, list every run for a session, slice failed runs in a time window, or roll usage + cost across any of those slices.
|
|
9
|
+
|
|
10
|
+
`createInMemoryTraceStore()` returns an object that satisfies **both** `TraceStoreContract` and `ExporterContract` — so wiring it into a collector is a single `collector.use(store)` and the store fills as traces complete.
|
|
11
|
+
|
|
12
|
+
```ts
|
|
13
|
+
import { panoptic, createInMemoryTraceStore } from "@warlock.js/ai-panoptic";
|
|
14
|
+
|
|
15
|
+
const store = createInMemoryTraceStore();
|
|
16
|
+
|
|
17
|
+
// The store doubles as an exporter — register it like any other sink.
|
|
18
|
+
const observe = panoptic({ exporters: [store] });
|
|
19
|
+
|
|
20
|
+
const agent = ai.agent({ model });
|
|
21
|
+
observe.attach(agent);
|
|
22
|
+
|
|
23
|
+
await agent.execute("Summarize this");
|
|
24
|
+
|
|
25
|
+
// ...later, off the hot path:
|
|
26
|
+
const failed = store.query({ status: "failed" });
|
|
27
|
+
const spend = store.aggregate({ sessionId: "session-42" });
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
> The store is also a valid `collector.use(...)` target directly — `panoptic({ collector: createCollector().use(store) })` — see [`observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) for bring-your-own-collector wiring.
|
|
31
|
+
|
|
32
|
+
## The store contract
|
|
33
|
+
|
|
34
|
+
`TraceStoreContract` is the queryable surface; the in-memory implementation adds the `ExporterContract` `name` + `export` so it can be a sink.
|
|
35
|
+
|
|
36
|
+
```ts
|
|
37
|
+
interface TraceStoreContract {
|
|
38
|
+
add(trace: Trace): void; // ingest; overwrites same traceId
|
|
39
|
+
get(traceId: string): Trace | undefined; // one run by id
|
|
40
|
+
query(filter?: TraceQuery): Trace[]; // matching traces, newest-started first
|
|
41
|
+
aggregate(filter?: TraceQuery): TraceAggregate; // usage + cost + status rollup
|
|
42
|
+
readonly size: number; // retained count
|
|
43
|
+
clear(): void; // drop everything
|
|
44
|
+
}
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
- **`add` overwrites** any existing trace with the same `traceId` — re-collecting a run replaces rather than duplicates it.
|
|
48
|
+
- **`export(trace)` is an alias for `add`** — that is what makes the store a drop-in exporter (`store.name === "in-memory-trace-store"`).
|
|
49
|
+
- **`query` returns newest-started first** (sorted by root `startedAt`, descending). An empty / omitted filter returns every retained trace.
|
|
50
|
+
|
|
51
|
+
## Filtering with `TraceQuery`
|
|
52
|
+
|
|
53
|
+
Every field is optional and **ANDed** together — an absent field is "don't care", so `{}` matches everything. The same filter type drives both `query` and `aggregate`.
|
|
54
|
+
|
|
55
|
+
```ts
|
|
56
|
+
type TraceQuery = {
|
|
57
|
+
traceId?: string; // exact one run
|
|
58
|
+
sessionId?: string; // every run for a session
|
|
59
|
+
status?: ReportStatus | ReportStatus[]; // single status or membership set
|
|
60
|
+
startedAfter?: string | Date; // inclusive lower bound on root startedAt
|
|
61
|
+
startedBefore?: string | Date; // inclusive upper bound on root startedAt
|
|
62
|
+
};
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
```ts
|
|
66
|
+
// One run by id:
|
|
67
|
+
store.query({ traceId: "run-abc" });
|
|
68
|
+
|
|
69
|
+
// Every run for a conversation:
|
|
70
|
+
store.query({ sessionId: "session-42" });
|
|
71
|
+
|
|
72
|
+
// Failed or cancelled runs since midnight (status accepts an array):
|
|
73
|
+
store.query({
|
|
74
|
+
status: ["failed", "cancelled"],
|
|
75
|
+
startedAfter: "2026-06-18T00:00:00.000Z",
|
|
76
|
+
});
|
|
77
|
+
|
|
78
|
+
// A Date works anywhere a time bound is accepted:
|
|
79
|
+
store.query({ startedAfter: new Date(Date.now() - 3_600_000) }); // last hour
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Time bounds compare against each trace's root `startedAt`, **inclusive on both ends**, accepting either an ISO-8601 string or a `Date`. Status is exact equality (or array membership). The pure matcher behind this is `matchTrace(trace, filter)` — exported for testing or filtering a trace list you hold yourself.
|
|
83
|
+
|
|
84
|
+
## Rolling up with `aggregate`
|
|
85
|
+
|
|
86
|
+
`aggregate(filter?)` answers the headline questions — how many runs, how many tokens, how much did it cost, how many failed — for whatever slice the filter selects:
|
|
87
|
+
|
|
88
|
+
```ts
|
|
89
|
+
type TraceAggregate = {
|
|
90
|
+
traces: number; // matched count
|
|
91
|
+
completed: number; // root status === "completed"
|
|
92
|
+
failed: number; // root status === "failed"
|
|
93
|
+
cancelled: number; // root status === "cancelled"
|
|
94
|
+
usage: Usage; // summed token usage (input/output/total always present)
|
|
95
|
+
cost?: ModelPricing; // merged per-channel USD, or undefined if no trace was priced
|
|
96
|
+
totalDuration: number; // summed wall-clock ms
|
|
97
|
+
};
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
```ts
|
|
101
|
+
const stats = store.aggregate({ sessionId: "session-42" });
|
|
102
|
+
|
|
103
|
+
console.log(`${stats.traces} runs, ${stats.usage.total} tokens, ${stats.failed} failed`);
|
|
104
|
+
|
|
105
|
+
const totalUsd =
|
|
106
|
+
(stats.cost?.input ?? 0) +
|
|
107
|
+
(stats.cost?.output ?? 0) +
|
|
108
|
+
(stats.cost?.cachedInput ?? 0) +
|
|
109
|
+
(stats.cost?.cachedOutput ?? 0);
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Notes that matter when reading the numbers:
|
|
113
|
+
|
|
114
|
+
- **`usage`** sums each trace's *root* `Usage` (already a rollup of own cost + children), so the totals reflect the whole run tree without re-walking spans. `input` / `output` / `total` are always present (zero when nothing matched); the optional `cachedTokens` / `cacheWriteTokens` / `reasoningTokens` channels are summed only when at least one matched trace reported them.
|
|
115
|
+
- **`cost`** is a per-channel `ModelPricing` breakdown (not one opaque number), merged with the framework's own cost-rollup logic — an unpriced trace never erases the cost of priced ones. It stays `undefined` only when **no** matched trace carried pricing.
|
|
116
|
+
- **`completed` + `failed` + `cancelled` need not equal `traces`** — non-terminal statuses (`awaiting-input`, `max-iterations`) count toward `traces` but none of the three headline counters.
|
|
117
|
+
- An empty match returns a zeroed aggregate (`usage` all `0`, `cost` `undefined`, `totalDuration` `0`).
|
|
118
|
+
|
|
119
|
+
## Bounding retention with `capacity`
|
|
120
|
+
|
|
121
|
+
Unbounded by default (keep everything until `clear()`). For a long-lived process, cap it — the oldest-ingested trace is evicted FIFO once the cap is exceeded:
|
|
122
|
+
|
|
123
|
+
```ts
|
|
124
|
+
type InMemoryTraceStoreOptions = { capacity?: number }; // absent / 0 = unbounded
|
|
125
|
+
|
|
126
|
+
const store = createInMemoryTraceStore({ capacity: 1000 });
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
Overwriting an existing `traceId` refreshes its insertion position, so a re-collected run is treated as newest for eviction — eviction stays honest under overwrite.
|
|
130
|
+
|
|
131
|
+
> The in-memory store has no secondary indexes: `get` / `add` are O(1), `query` / `aggregate` are O(n) scans. That is the right trade-off for dev/test and modest-volume runtime use. For high-volume or durable retention, write a custom `TraceStoreContract` (or export to a backend with [`export-traces`](@warlock.js/ai-panoptic/export-traces/SKILL.md)) backed by a real datastore.
|
|
132
|
+
|
|
133
|
+
## Persisting across a restart — `createCacheTraceStore`
|
|
134
|
+
|
|
135
|
+
The in-memory store loses everything when the process exits. `createCacheTraceStore(cache, options?)` is a **durable** `TraceStoreContract` (and `ExporterContract`) backed by any `@warlock.js/cache` `CacheDriver` — Redis, file, Postgres, … — so traces survive a restart and the dashboard re-opens with history intact.
|
|
136
|
+
|
|
137
|
+
```ts
|
|
138
|
+
import { createCacheTraceStore } from "@warlock.js/ai-panoptic";
|
|
139
|
+
import { RedisCacheDriver } from "@warlock.js/cache";
|
|
140
|
+
|
|
141
|
+
// A driver instance, or a (possibly async) FACTORY resolved lazily on first
|
|
142
|
+
// use — the factory form defers the Redis connect until the first trace is
|
|
143
|
+
// collected, so module import stays connection-free.
|
|
144
|
+
const store = createCacheTraceStore(async () => {
|
|
145
|
+
const driver = new RedisCacheDriver();
|
|
146
|
+
await driver.connect();
|
|
147
|
+
return driver;
|
|
148
|
+
});
|
|
149
|
+
|
|
150
|
+
await store.ready(); // hydrate the in-memory mirror from a prior run
|
|
151
|
+
collector.use(store); // fills as traces complete; reads stay synchronous
|
|
152
|
+
|
|
153
|
+
const failed = store.query({ status: "failed" }); // queryable like any store
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
**Write-through over a sync contract.** The store keeps the same insertion-ordered in-memory **mirror** the in-memory store uses and serves every read (`get` / `query` / `aggregate` / `size`) from it synchronously — the dashboard polls these and can't await a cache. Writes go **through**: `add` updates the mirror immediately, then *fire-and-forget* persists the trace + a newest-first index to the cache. So:
|
|
157
|
+
|
|
158
|
+
- **Reads never wait on the cache** — the mirror is the runtime source of truth; the cache is the durable backing store.
|
|
159
|
+
- **A flaky cache never throws into the hot path** — a rejected write is routed to the optional `onError` hook and swallowed; the trace stays visible in the mirror for the life of the process, it just won't survive a restart.
|
|
160
|
+
- **`ready()` re-hydrates** the mirror from the persisted index on startup (await it once — `ai.config({ panoptic: { cache } })` does this for you before serving the dashboard).
|
|
161
|
+
|
|
162
|
+
```ts
|
|
163
|
+
type CacheDriverInput =
|
|
164
|
+
| CacheDriver // a live driver
|
|
165
|
+
| (() => CacheDriver | Promise<CacheDriver>); // lazy (sync or async) factory
|
|
166
|
+
|
|
167
|
+
type CacheTraceStoreOptions = {
|
|
168
|
+
prefix?: string; // key namespace; per-trace keys `${prefix}:trace:${id}`,
|
|
169
|
+
// index at `${prefix}:index`. Default "panoptic".
|
|
170
|
+
capacity?: number; // FIFO cap (oldest-ingested evicted from cache + mirror);
|
|
171
|
+
// absent / 0 = unbounded.
|
|
172
|
+
onError?: (error: unknown) => void; // swallowed cache-write failures land here.
|
|
173
|
+
};
|
|
174
|
+
|
|
175
|
+
type CacheTraceStoreHandle = TraceStoreContract & ExporterContract & {
|
|
176
|
+
ready(): Promise<void>; // hydrate the mirror from the cache on startup
|
|
177
|
+
};
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
Ordering uses an **internal monotonic counter**, not `Date.now()`, so the persisted index stays a stable total order even when many traces land in the same millisecond — FIFO eviction stays honest across a restart.
|
|
181
|
+
|
|
182
|
+
> The usual path is declarative: `ai.config({ panoptic: { cache, dashboard: true } })` constructs this store, awaits `ready()`, and wires it under the dashboard for you — see [`use-local-dashboard/SKILL.md`](@warlock.js/ai-panoptic/use-local-dashboard/SKILL.md). Reach for `createCacheTraceStore` directly only when you manage the store / collector yourself.
|
|
183
|
+
|
|
184
|
+
## Summing usage yourself
|
|
185
|
+
|
|
186
|
+
`sumUsage` / `emptyUsage` are the pure folds `aggregate` is built on — reuse them to roll a `Usage` set the store didn't produce (e.g. a list of traces you filtered by hand):
|
|
187
|
+
|
|
188
|
+
```ts
|
|
189
|
+
import { emptyUsage, sumUsage } from "@warlock.js/ai-panoptic";
|
|
190
|
+
|
|
191
|
+
let total = emptyUsage(); // { input: 0, output: 0, total: 0 } — optional channels absent
|
|
192
|
+
for (const trace of store.query({ sessionId })) {
|
|
193
|
+
total = sumUsage(total, trace.usage);
|
|
194
|
+
}
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
`sumUsage` is pure (returns a fresh `Usage`, mutates neither argument): token channels always sum; the optional cache/reasoning channels sum only when a side reported them (so a never-metered channel stays absent rather than fabricating a `0`); `cost` merges with the framework's `accumulateCost` so an unpriced contributor never erases a priced one.
|
|
198
|
+
|
|
199
|
+
## See also
|
|
200
|
+
|
|
201
|
+
- [`@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md`](@warlock.js/ai-panoptic/observe-with-panoptic/SKILL.md) — wiring the `panoptic(...)` subscriber and bring-your-own-collector so the store fills as runs complete.
|
|
202
|
+
- [`@warlock.js/ai-panoptic/export-traces/SKILL.md`](@warlock.js/ai-panoptic/export-traces/SKILL.md) — sending traces to an external backend instead of (or alongside) retaining them in memory; writing a custom sink.
|
|
203
|
+
- [`@warlock.js/ai/run-ai-agent/SKILL.md`](@warlock.js/ai/run-ai-agent/SKILL.md) — the `BaseReport` the collector projects into the `Trace` this store retains.
|