@tangle-network/traces 0.23.1 → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +61 -1078
- package/dist/analyst-model-call-JVUOYZ5V.js +20 -0
- package/dist/ask-2KF4KSPB.js +39 -0
- package/dist/check-YF6ZHPGG.js +322 -0
- package/dist/chunk-2XXABIRL.js +25 -0
- package/dist/chunk-4VKPN432.js +138 -0
- package/dist/chunk-6GOG5KIK.js +53 -0
- package/dist/chunk-7USMM4MA.js +5695 -0
- package/dist/chunk-CG5QHQO4.js +492 -0
- package/dist/chunk-CQ6UYZG6.js +1217 -0
- package/dist/chunk-DU534HAX.js +263 -0
- package/dist/chunk-FC4V25CG.js +503 -0
- package/dist/chunk-GLUDGUGJ.js +232 -0
- package/dist/chunk-GNRXKQEQ.js +120 -0
- package/dist/chunk-K2KH2HHD.js +228 -0
- package/dist/chunk-M3W6TZHQ.js +381 -0
- package/dist/chunk-O52XK7OB.js +19 -0
- package/dist/chunk-PYH3PL3N.js +62 -0
- package/dist/chunk-QLVP37YB.js +782 -0
- package/dist/chunk-QUHUMZ4W.js +759 -0
- package/dist/chunk-REO2JQJY.js +535 -0
- package/dist/chunk-RZL5GAWH.js +60 -0
- package/dist/chunk-TOCNGML5.js +273 -0
- package/dist/chunk-UQSAGXPH.js +208 -0
- package/dist/chunk-VFOLC4OX.js +407 -0
- package/dist/chunk-WF7PTDCZ.js +3921 -0
- package/dist/chunk-YAX72TST.js +259 -0
- package/dist/cli.js +2086 -2050
- package/dist/diff-J4RAIOXO.js +16 -0
- package/dist/improvement-LCCVGGUS.js +25 -0
- package/dist/index.d.ts +1235 -4357
- package/dist/index.js +53 -1204
- package/dist/live-XXL7VZWB.js +14 -0
- package/dist/mcp-EYIDKBEK.js +85 -0
- package/dist/report-E3G2T5TJ.js +338 -0
- package/dist/run-watch-RAOOAXOF.js +649 -0
- package/dist/supervisor-report-RMPLLBGZ.js +29 -0
- package/dist/supervisor-run-context-WO6VOTGV.js +30 -0
- package/dist/upload-3R7GRSKS.js +20 -0
- package/dist/verify-safe-C5ZXZOZ2.js +8 -0
- package/docs/analyze.md +76 -0
- package/docs/ask.md +88 -0
- package/docs/bundle.md +40 -0
- package/docs/check.md +77 -0
- package/docs/convert.md +44 -0
- package/docs/develop.md +32 -0
- package/docs/diff.md +38 -0
- package/docs/facts.md +114 -0
- package/docs/mcp.md +36 -0
- package/docs/migrate-1.0.md +33 -0
- package/docs/parsers.md +66 -0
- package/docs/release-1.0.0.md +147 -0
- package/docs/release-1.0.1.md +4 -0
- package/docs/sdk.md +77 -0
- package/docs/select.md +70 -0
- package/docs/skills.md +46 -0
- package/docs/upload.md +47 -0
- package/docs/watch-jsonl.v1.example.jsonl +5 -0
- package/docs/watch.md +66 -0
- package/examples/README.md +9 -0
- package/examples/custom-analyst.ts +65 -74
- package/install.sh +22 -2
- package/package.json +10 -9
- package/skills/inspect-agent-traces/SKILL.md +47 -110
- package/dist/chunk-7DVZCCZI.js +0 -20983
- package/docs/replay-verify.md +0 -123
- package/docs/stream-findings.svg +0 -39
- package/docs/trace-analysts.md +0 -643
- package/examples/custom-backend.ts +0 -21
- package/examples/external-engines.ts +0 -62
- package/examples/improvement-config.mjs +0 -63
- package/examples/observe-and-alert.ts +0 -19
- package/examples/register-harness.ts +0 -26
- package/skills/build-trace-analyst/SKILL.md +0 -76
- package/skills/build-trace-analyst/agents/openai.yaml +0 -4
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# traces
|
|
2
2
|
|
|
3
|
-
>
|
|
3
|
+
> What your coding agents did, where they wasted time, and whether a run passes: exact answers from the sessions Claude Code, Codex, OpenCode, Pi, Kimi and Factory already write to disk, or from the OTLP spans your own system emits. A CLI and an SDK.
|
|
4
4
|
|
|
5
5
|

|
|
6
6
|
|
|
@@ -8,1103 +8,86 @@
|
|
|
8
8
|
[](./LICENSE)
|
|
9
9
|
[](https://nodejs.org)
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
There are two ways in, and they are not equal:
|
|
14
|
-
|
|
15
|
-
| | Use when | How |
|
|
16
|
-
|---|---|---|
|
|
17
|
-
| **Emit the contract** | You control the system | Emit [`@tangle-network/agent-trace-contract`](https://github.com/tangle-network/agent-sdk/tree/main/packages/agent-trace-contract) spans, then `traces analyze --otlp <file\|dir>` |
|
|
18
|
-
| **Adapters** | You do *not* control the format | `traces analyze --harness claude-code` reads a proprietary session store and translates it |
|
|
19
|
-
|
|
20
|
-
Emitting the contract is the supported way to integrate a new system. The adapters are the legacy edge — they exist because the coding agents write formats nobody standardised, and each one is a translation somebody has to maintain. If you can choose, choose `--otlp`.
|
|
21
|
-
|
|
22
|
-
## Contents
|
|
23
|
-
|
|
24
|
-
- [Install](#install)
|
|
25
|
-
- [Quick start](#quick-start)
|
|
26
|
-
- [Gate CI on a trace contract](#gate-ci-on-a-trace-contract)
|
|
27
|
-
- [Integrate your own system](#integrate-your-own-system)
|
|
28
|
-
- [What it finds](#what-it-finds)
|
|
29
|
-
- [Supported harnesses](#supported-harnesses)
|
|
30
|
-
- [CLI reference](#cli-reference)
|
|
31
|
-
- [Live stream](#live-stream)
|
|
32
|
-
- [Watch a run tree](#watch-a-run-tree)
|
|
33
|
-
- [Improvement engine](#improvement-engine)
|
|
34
|
-
- [Ask questions](#ask-questions)
|
|
35
|
-
- [Session facts](#session-facts)
|
|
36
|
-
- [Skill usage](#skill-usage)
|
|
37
|
-
- [Diff two runs](#diff-two-runs)
|
|
38
|
-
- [MCP server](#mcp-server)
|
|
39
|
-
- [Session index](#session-index)
|
|
40
|
-
- [Session bundle](#session-bundle) · [Two views](#two-views-two-consumers)
|
|
41
|
-
- [Policy-mining evidence](#policy-mining-evidence)
|
|
42
|
-
- [Upload to the Intelligence Platform](#upload-to-the-intelligence-platform) · [Check before sharing](#check-before-sharing)
|
|
43
|
-
- [Trace analysts](#trace-analysts)
|
|
44
|
-
- [Agent skills](#agent-skills)
|
|
45
|
-
- [Library (SDK)](#library-sdk)
|
|
46
|
-
- [Examples](#examples)
|
|
47
|
-
- [Audit benchmark](#audit-benchmark)
|
|
48
|
-
- [Develop](#develop)
|
|
11
|
+
Every command except `ask` runs locally, with no key and no cost. Agents: run traces, never parse a session store yourself; the [`inspect-agent-traces`](skills/inspect-agent-traces/SKILL.md) skill maps each question to its command.
|
|
49
12
|
|
|
50
13
|
## Install
|
|
51
14
|
|
|
52
15
|
```bash
|
|
53
16
|
curl -fsSL https://raw.githubusercontent.com/tangle-network/traces/main/install.sh | bash
|
|
54
|
-
traces
|
|
55
|
-
|
|
56
|
-
npx --yes @tangle-network/traces@latest analyze --harness claude-code --last 1 # run without installing
|
|
57
|
-
npm i -g @tangle-network/traces # install manually
|
|
58
|
-
npm i @tangle-network/traces # use it as a library
|
|
17
|
+
npm i @tangle-network/traces # as a library
|
|
59
18
|
```
|
|
60
19
|
|
|
61
|
-
Requires Node
|
|
20
|
+
Requires Node 22 or later. `install.sh` refuses to overwrite another program's `traces` and names it. Coming from 0.x: [Migrate to 1.0](docs/migrate-1.0.md).
|
|
62
21
|
|
|
63
22
|
## Quick start
|
|
64
23
|
|
|
65
24
|
```bash
|
|
66
|
-
traces
|
|
67
|
-
traces
|
|
68
|
-
traces
|
|
69
|
-
traces
|
|
70
|
-
traces
|
|
71
|
-
traces watch --all
|
|
72
|
-
traces stream --all --mode findings
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
The first command is shown in the demo above.
|
|
76
|
-
The **deterministic pass** checks stuck loops, token growth, output decay, missing self-verification, tool failures, and human corrections.
|
|
77
|
-
Each supported issue is returned as a finding with evidence, an action, confidence, and a validation plan.
|
|
78
|
-
It needs no API key and costs nothing.
|
|
79
|
-
|
|
80
|
-
Add `--llm` for the **agentic analysts**. `traces` first uses free local signals to choose the smallest useful set: a failure review always runs, while knowledge and edit reviews run only when failures, repeated calls, or corrective feedback support them. They call OpenAI and respect `--budget <usd>`.
|
|
81
|
-
|
|
82
|
-
Every run also writes a **canonical OpenInference JSONL artifact**.
|
|
83
|
-
Use it with [HALO](https://github.com/context-labs/halo), [Hodoscope](https://github.com/ManifoldRG/hodoscope), or any command that reads the file.
|
|
84
|
-
See [Trace analysts](#trace-analysts).
|
|
85
|
-
|
|
86
|
-
`traces improve` is the reviewable action path.
|
|
87
|
-
It writes one typed result, one report, flattened evidence rows, and the canonical OTLP trace.
|
|
88
|
-
Each finding already contains the claim, evidence, recommended action, confidence, and validation plan.
|
|
89
|
-
|
|
90
|
-
## Gate CI on a trace contract
|
|
91
|
-
|
|
92
|
-
Two runs can give the same answer while one of them calls a forbidden tool.
|
|
93
|
-
An output check passes both; a trace contract fails the wrong one.
|
|
94
|
-
|
|
95
|
-
```json
|
|
96
|
-
{
|
|
97
|
-
"name": "refund-desk",
|
|
98
|
-
"run": { "requireCompleted": true, "maxDurationMs": 300000 },
|
|
99
|
-
"tools": { "required": ["Read"], "forbidden": ["Bash"], "allowed": ["Read", "Grep"], "maxCalls": 10, "enforced": true },
|
|
100
|
-
"retries": { "reads": ["Read", "Grep"] },
|
|
101
|
-
"llm": { "maxCalls": 10 }
|
|
102
|
-
}
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
```bash
|
|
106
|
-
claude -p "$TASK" --tools Read,Grep --strict-mcp-config --output-format stream-json --verbose > run.jsonl
|
|
107
|
-
traces check run.jsonl --contract refund-desk.json --junit contract.xml
|
|
25
|
+
traces list --cwd . --last 5 # which sessions ran here
|
|
26
|
+
traces facts --current # what this session did: turns, PRs, first failure, cost, $0
|
|
27
|
+
traces facts --since 30d --match 'pnpm test' # did we already try X
|
|
28
|
+
traces analyze 019a3c7e # where it wasted time, ranked
|
|
29
|
+
traces check run.jsonl --contract contract.json # fail CI when a run took a forbidden path
|
|
108
30
|
```
|
|
109
31
|
|
|
110
|
-
|
|
111
|
-
Without `--strict-mcp-config`, the run above is also offered every MCP server the account or project configures.
|
|
112
|
-
`retries` fails when a tool repeats a call with the same arguments and nothing proves the repeat is safe.
|
|
113
|
-
|
|
114
|
-
The trace is `claude -p --output-format stream-json` output, a Claude Code or Codex session file, OTLP spans (a file or a directory), or trace evidence; `--format` names it when the file could be read as more than one.
|
|
115
|
-
Only stream-json output records the offered tools and the run's own result.
|
|
116
|
-
**A saved interactive session file carries no run status at all** (the `result` record above only exists in `-p` mode), so `run.requireCompleted`/`run.allowedStatuses` always fail on one — write a contract for a session file with no `run` key, like the `refund-desk` example above running against `run.jsonl`.
|
|
117
|
-
Every rule prints `pass`, `fail`, `error`, or `skipped` (a rule of an alternative path that did not decide the result).
|
|
118
|
-
|
|
119
|
-
| Exit | Meaning |
|
|
120
|
-
|---|---|
|
|
121
|
-
| 0 | Every rule passed. |
|
|
122
|
-
| 1 | A rule failed. |
|
|
123
|
-
| 2 | The contract is malformed or contradicts itself, or a rule could not be evaluated. |
|
|
124
|
-
| 3 | The trace could not be read. |
|
|
125
|
-
| 4 | The trace reference is ambiguous. |
|
|
126
|
-
|
|
127
|
-
`--junit` writes one test case per rule, and GitHub annotations print under `GITHUB_ACTIONS=true` or `--annotations`.
|
|
128
|
-
`traces check --contract <file> --explain` prints what each compiled rule checks and reads no trace.
|
|
129
|
-
For a trace that does not pass, the verdict, the contract, its plain-language statement, and the redacted spans the violations cite go to `--evidence` (default `.traces/check`).
|
|
130
|
-
The contract keys and rule semantics are in [agent-eval's trace-contract guide](https://github.com/tangle-network/agent-eval/blob/main/docs/trace-contracts.md).
|
|
131
|
-
|
|
132
|
-
## Integrate your own system
|
|
133
|
-
|
|
134
|
-
No adapter, no plugin. Emit the span contract and point `--otlp` at the file.
|
|
135
|
-
|
|
136
|
-
```bash
|
|
137
|
-
npm i @tangle-network/agent-trace-contract # zero runtime dependencies
|
|
138
|
-
```
|
|
139
|
-
|
|
140
|
-
You install it to PRODUCE spans. This package already depends on it to read
|
|
141
|
-
them, so `npx @tangle-network/traces@latest` arrives with the contract-backed
|
|
142
|
-
analyses working: trace conformance, `traces validate`, `--otlp` inbound
|
|
143
|
-
reading, round-over-round convergence and the steering chain, and OpenInference
|
|
144
|
-
/ intelligence-spans conversion.
|
|
145
|
-
|
|
146
|
-
```ts
|
|
147
|
-
import { llmSpan, loopSpan, steeredBy, toolSpan } from '@tangle-network/agent-trace-contract'
|
|
148
|
-
|
|
149
|
-
// One round of a loop, the model call and tool call inside it, then the next
|
|
150
|
-
// round — linked to the verdict that CAUSED it, not parented to it.
|
|
151
|
-
const round1 = loopSpan({ traceId, spanId: 'round-1', parentSpanId: null, name: 'round 1',
|
|
152
|
-
loopId: 'fix-loop', iteration: 1, startTime, endTime })
|
|
153
|
-
const call = llmSpan({ traceId, spanId: 'llm-1', parentSpanId: 'round-1', name: 'coder turn',
|
|
154
|
-
model: 'glm-5.2', inputTokens: 1200, outputTokens: 340, costUsd: 0.0042, startTime, endTime })
|
|
155
|
-
const tool = toolSpan({ traceId, spanId: 'tool-1', parentSpanId: 'round-1', name: 'bash',
|
|
156
|
-
toolName: 'bash', startTime, endTime })
|
|
157
|
-
const round2 = loopSpan({ traceId, spanId: 'round-2', parentSpanId: null, name: 'round 2',
|
|
158
|
-
loopId: 'fix-loop', iteration: 2, resumed: true, startTime, endTime,
|
|
159
|
-
links: [steeredBy('llm-1', traceId)] })
|
|
160
|
-
```
|
|
161
|
-
|
|
162
|
-
Write one span per line to a `.jsonl` file, then:
|
|
163
|
-
|
|
164
|
-
```bash
|
|
165
|
-
traces validate spans.otlp.jsonl # exit 1 only when the file is not a trace
|
|
166
|
-
traces analyze --otlp spans.otlp.jsonl # a directory reads the OTLP files under it
|
|
167
|
-
```
|
|
32
|
+
## Commands
|
|
168
33
|
|
|
169
|
-
|
|
170
|
-
next to raw event, stream and SDK logs that are also `*.jsonl`. Only the OTLP
|
|
171
|
-
files are read — every other JSONL is listed in the report with what it actually
|
|
172
|
-
holds, rather than counted as several hundred broken spans. Put the exports in an
|
|
173
|
-
`otlp/` subdirectory and that decides it outright.
|
|
174
|
-
|
|
175
|
-
Writing commands (`convert`, `export`, `evidence`, `upload`) take `--otlp-out`; they refuse `--otlp`.
|
|
176
|
-
|
|
177
|
-
`parent_span_id` is CONTAINMENT — this span happened inside that one. `links` is
|
|
178
|
-
CAUSALITY — that verdict caused this retry. Encoding causality as a parent claims
|
|
179
|
-
a nesting that never existed, so the two edges stay separate and both survive a
|
|
180
|
-
round-trip through `traces`.
|
|
181
|
-
|
|
182
|
-
### What a trace cannot tell you, said out loud
|
|
183
|
-
|
|
184
|
-
`validate` reports the conformance of any OTLP file, conforming or not — it never
|
|
185
|
-
throws on a foreign trace, it reports it:
|
|
186
|
-
|
|
187
|
-
- **findings** by severity, each naming the *consequence* of the defect and the capabilities it blocks
|
|
188
|
-
- **the capability table** — `token-accounting`, `cost-attribution`, `tool-usage`, `loop-convergence`, `tree-comparison`, `steering-chain`, `latency-analysis` — available or not, with the trace's own reason
|
|
189
|
-
- **span structure** — how much of the tree actually attaches, and how many rows could not be read at all
|
|
190
|
-
|
|
191
|
-
Exit code is 1 on exactly one condition: not one entry could be read as a span,
|
|
192
|
-
so the file is not a trace. Severity says what a finding claims about the INPUT,
|
|
193
|
-
not how loud it is — `error` is "nothing readable", `warn` is "readable but
|
|
194
|
-
degraded", `info` is "nothing measurable lost" — so a `warn` exits 0 and the
|
|
195
|
-
capability table, not the exit code, says what the degradation cost.
|
|
196
|
-
|
|
197
|
-
#### A re-export never validates cleaner than its source
|
|
198
|
-
|
|
199
|
-
`--otlp-out` writes the spans that were analysed. Anywhere this package
|
|
200
|
-
normalized a field to analyse it — an `end_time` that precedes its `start_time`,
|
|
201
|
-
a status code the contract does not recognise — the export writes what the
|
|
202
|
-
PRODUCER wrote and keeps the normalized value in an attribute beside it, so
|
|
203
|
-
`negative-duration` and `invalid-status` are still findings against the copy.
|
|
204
|
-
|
|
205
|
-
A row that never became a span is the one thing that cannot be re-emitted: the
|
|
206
|
-
artifact is also the file the analysts read, so a non-span line in it is garbage
|
|
207
|
-
fed to the analysis and a repaired one is invented work. The export refuses the
|
|
208
|
-
equivalence instead — it carries `traces.source.unreadable_rows` and the kinds
|
|
209
|
-
behind it, accumulating across every hop, and `validate` on a re-export leads
|
|
210
|
-
with how much of the original is missing from it. Round-tripping a defective
|
|
211
|
-
trace through `traces` cannot produce a clean bill of health.
|
|
212
|
-
|
|
213
|
-
`analyze` prints the same section, plus an **analyses skipped, and why** list —
|
|
214
|
-
and, crucially, carries that verdict INTO the report. A section whose inputs are
|
|
215
|
-
incomplete says so in its own heading and again directly above its table, so a
|
|
216
|
-
number and the reason it is wrong can never end up forty lines apart. When the
|
|
217
|
-
trace supports `tree-comparison`, the table names the command that compares its
|
|
218
|
-
arms: `traces diff <file>#branch=<a> <file>#branch=<b>`.
|
|
219
|
-
|
|
220
|
-
Two of those analyses are why the loop shape exists at all:
|
|
221
|
-
|
|
222
|
-
- **round-over-round convergence** — per `agent.loop.id`, the per-round outcome
|
|
223
|
-
and score series, and whether it improved, plateaued, regressed, or peaked and
|
|
224
|
-
fell back. The verdict is read from the round's own `agent.outcome` or from the
|
|
225
|
-
shallowest EVALUATOR span under it.
|
|
226
|
-
- **steering chain** — follows `links` (`agent.link.kind`) to report which verdict
|
|
227
|
-
caused which subsequent round. A link whose target is missing is kept and marked,
|
|
228
|
-
because a shorter chain than the producer recorded is the opposite of the truth.
|
|
229
|
-
|
|
230
|
-
Parent and verdict lookups use both trace and span IDs.
|
|
231
|
-
Steering edges retain `causeTraceId` and `effectTraceId`, including unresolved targets.
|
|
232
|
-
Generic chat transcripts report unknown execution status because message text does not establish completion.
|
|
233
|
-
|
|
234
|
-
Nothing is repaired on the way in. A span with an unreadable timestamp is
|
|
235
|
-
reported and excluded, because a synthesized value would put invented work into a
|
|
236
|
-
total.
|
|
237
|
-
|
|
238
|
-
## What it finds
|
|
239
|
-
|
|
240
|
-
The deterministic pass (free, no key) surfaces:
|
|
241
|
-
|
|
242
|
-
| Finding | Meaning |
|
|
243
|
-
|---|---|
|
|
244
|
-
| **Stuck loops** | the same tool called N× with identical args and no state change |
|
|
245
|
-
| **Monotonic input growth** | full history re-sent every step; context never compressed |
|
|
246
|
-
| **Output-length decay** | planning/reasoning per step shrinking as context grows |
|
|
247
|
-
| **No self-verification** | state-mutating actions never followed by an eval/inspect/check |
|
|
248
|
-
| **Tool mix / retry / error rates** | repeated, retried, and failed tool calls |
|
|
249
|
-
|
|
250
|
-
`--llm` adds agentic analysts that read the conversation and cluster higher-order failure and improvement signals. They receive the compact deterministic findings first, then share their earlier findings with later analysts. With `traces improve`, the selected analysts and reasons are saved in `result.json` and the report, so another agent can reuse the packet without rereading the raw trace.
|
|
251
|
-
|
|
252
|
-
## Supported harnesses
|
|
253
|
-
|
|
254
|
-
Adapters are the **legacy edge**: they exist for coding agents whose on-disk format we do not
|
|
255
|
-
control, and each is a translation from a proprietary session store onto the span model. They are
|
|
256
|
-
not the way to integrate a system you own — for that, [emit the contract](#integrate-your-own-system).
|
|
257
|
-
|
|
258
|
-
"Verified" = tested against real sessions; "fixture" = tested against schema-accurate fixtures (no real sessions available).
|
|
259
|
-
|
|
260
|
-
| Harness (aliases) | Reads from | Status |
|
|
34
|
+
| Job | Command | Page |
|
|
261
35
|
|---|---|---|
|
|
262
|
-
|
|
|
263
|
-
|
|
|
264
|
-
|
|
|
265
|
-
|
|
|
266
|
-
|
|
|
267
|
-
|
|
|
268
|
-
|
|
|
269
|
-
|
|
|
270
|
-
|
|
|
271
|
-
|
|
|
272
|
-
|
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
traces
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
traces
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
traces
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
traces validate spans.otlp.jsonl # conformance; exit 1 only when it is not a trace
|
|
305
|
-
traces validate results/sessions --out conformance.md # a whole directory of exports
|
|
306
|
-
traces analyze --otlp spans.otlp.jsonl # analyse foreign OTLP, no adapter
|
|
307
|
-
traces analyze --otlp results/sessions --out report.md
|
|
308
|
-
traces convert --harness claude-code --last 1 --otlp-out spans.jsonl # OTLP only
|
|
309
|
-
traces index --all --since 24h --out session-index.json
|
|
310
|
-
traces bundle --harness claude-code --session <id|path> --out bundle-dir # one session's durable evidence dir
|
|
311
|
-
traces bundle-view bundle-dir --view evidence-only --out writer-dir # same session, without its own words
|
|
312
|
-
traces bundle verify bundle-dir # every file against its manifest SHA-256
|
|
313
|
-
traces inspect session-index.json --out inspection-report.md
|
|
314
|
-
traces evidence --harness codex --last 20 --out policy-evidence.jsonl
|
|
315
|
-
traces evidence --harness codex-exec --session /tmp/codex.jsonl --cwd "$PWD" --out policy-evidence.jsonl
|
|
316
|
-
traces export policy-evidence.jsonl --out spans.openinference.jsonl
|
|
317
|
-
traces import-codetracebench verified.jsonl --trajectory-dir normalized --out traces --revision <40-or-64-character-hex>
|
|
318
|
-
traces watch --all # live observer; loops + semantic findings
|
|
319
|
-
traces watch runs/my-run # live run tree: status, budget, driver vs child spend
|
|
320
|
-
traces watch runs/my-run --once # one snapshot, then exit
|
|
321
|
-
traces watch traces.otlp.jsonl # any OTLP span file, any emitter
|
|
322
|
-
traces analyze --supervisor-run-dir runs/my-run # the full tree report
|
|
323
|
-
traces stream --all --mode findings # low-volume semantic feed
|
|
324
|
-
traces stream --all --mode agent # findings + deterministic report events
|
|
325
|
-
traces stream spans.openinference.jsonl --format openinference --no-spans
|
|
326
|
-
traces upload --since 1h --dry-run # redact + dedup + preview, no network
|
|
327
|
-
traces upload --since 24h # upload last day to the Intelligence Platform
|
|
328
|
-
traces verify-safe bundle-dir # SAFE / SAFE_WITH_WARNINGS / UNSAFE (exit 1) / UNKNOWN (exit 2)
|
|
329
|
-
traces replay-verify --steps steps.json --image <docker-image> --at 37 --cwd /home --out ./replay-out \
|
|
330
|
-
--fix-command "<corrected step>" # executed proof: replay prefix, reproduce failure, show fix
|
|
331
|
-
traces verify-findings --findings findings.json --out ./receipts \
|
|
332
|
-
--steps steps.json --image <replay-ready-image> --cwd /app # execute recorded analyst findings as proofs
|
|
333
|
-
traces analyze --last 1 --llm --verify-findings --replay-corpus h=labels.json::prepared/ # proof-carrying analyze
|
|
334
|
-
```
|
|
335
|
-
|
|
336
|
-
`replay-verify` replays a CodeTraceBench-style trajectory prefix in a real sandbox and executes step k twice — recorded (does the failure reproduce?) and corrected (does it vanish?).
|
|
337
|
-
`verify-findings` runs that proof per analyst finding and annotates each with `reproduced | fix-flipped | divergent | not-replayable` plus a receipt directory; see [Verified findings](./docs/trace-analysts.md#verified-findings-executed-replay).
|
|
338
|
-
See [Replay verification](./docs/replay-verify.md) for setup, semantics, and honest limits (SWE-style trajectories with a docker image only; commands run as the non-root sandbox user).
|
|
339
|
-
|
|
340
|
-
| Flag | Meaning |
|
|
341
|
-
|---|---|
|
|
342
|
-
| `--harness <id>` | Harness or alias (default: `claude-code`) |
|
|
343
|
-
| `--all` | Every known harness |
|
|
344
|
-
| `--last <n>` | Most-recent N sessions |
|
|
345
|
-
| `--current` | Active Codex session from `CODEX_THREAD_ID` |
|
|
346
|
-
| `--latest-turn` | Limit a resumed Codex or Claude Code session to its latest task turn |
|
|
347
|
-
| `--session <id\|path>` | One listed session ID or explicit session file |
|
|
348
|
-
| `--workflow` | Expand selected files through stable parent and child session IDs |
|
|
349
|
-
| `--max-workflow-sessions <n>` | Stop workflow expansion before parsing more than `n` files; default `100` |
|
|
350
|
-
| `--cwd <dir>` | Filter by working directory |
|
|
351
|
-
| `--since <t>` | `30m`/`2h`/`7d` before now, or an ISO date. Commands that select sessions read those modified at or after it; `upload`: its window (default 24h); `facts`: opens a time window — see [Session facts](#session-facts) |
|
|
352
|
-
| `--until <t>` | `facts`: closes the time window (exclusive), same forms as `--since` |
|
|
353
|
-
| `--out <path>` | Write the report to a file |
|
|
354
|
-
| `--dir <path>` | `improve`: write the full artifact pack to this directory; `ask`: write `answers.json` + `report.md` there |
|
|
355
|
-
| `--otlp <file\|dir>` | **READ** OTLP-JSONL from any system, skipping the adapters; a directory reads the OTLP files under it (only `otlp/` when the producer made one) and names the JSONL that is not OTLP. `validate`, `check`, `analyze`, `investigate`, `improve`, `ask`, `stream` |
|
|
356
|
-
| `--contract <file>` | `check`: the declarative trace contract (JSON) |
|
|
357
|
-
| `--junit <file>` | `check`: write JUnit XML, one test case per rule |
|
|
358
|
-
| `--evidence <dir>` | `check`: where evidence for a trace that did not pass goes (default `.traces/check`) |
|
|
359
|
-
| `--annotations` | `check`: print GitHub workflow annotations (on by default under `GITHUB_ACTIONS=true`) |
|
|
360
|
-
| `--otlp-out <path>` | **WRITE** the OTLP artifact here (also evidence provenance / dry-run upload preview) |
|
|
361
|
-
| `--format <kind>` | File `analyze`, `export`, or `stream`: `auto`, `policy-evidence`, `sandbox-events`, `openinference`, `intelligence-spans`, or `chat-trajectory` |
|
|
362
|
-
| `--source-bundle <dir>` | `analyze` / `investigate` / `improve` / `ask`: read a retained full bundle and explicitly grant source-field reads |
|
|
363
|
-
| `--llm` / `--budget <usd>` | Enable agentic analysts (needs `TANGLE_API_KEY` + Python with `agent-eval-rpc[dspy]`) / cap their spend |
|
|
364
|
-
| `--question <text>` | `ask`: one question, repeatable. Kept short so the engine sees it whole |
|
|
365
|
-
| `--questions <file>` | `ask`: JSON array of questions — strings, or `{ id?, question, instructions?, answerSchema? }` |
|
|
366
|
-
| `--question-budget <usd>` | `ask`: provider ceiling for ONE question; `--budget` is the ceiling shared by all of them |
|
|
367
|
-
| `--question-timeout <seconds>` | `ask`: wall-clock deadline for ONE question's investigation (default 600); raise it for questions over many large sessions |
|
|
368
|
-
| `--concurrency <n>` | `ask`: questions running at once (default 4); `import-codetracebench`: trajectories imported at once |
|
|
369
|
-
| `--config <path>` | `analyze` / `investigate` / `improve` / `stream`: load BYO analysts, live analysts, and external analyzers |
|
|
370
|
-
| `--interval <s>` / `--window <m>` | `watch` / live `stream`: poll seconds (sessions 5, run tree 2) / active-session window minutes (default 30) |
|
|
371
|
-
| `--min-loop <n>` | Identical repeated calls before flagging a loop (default 3) |
|
|
372
|
-
| `--mode <kind>` | `stream`: `visualizer` (spans + findings), `findings` (low-volume), or `agent` (findings + reports) |
|
|
373
|
-
| `--supervisor-run-dir <dir>` | `analyze`: report one run tree; `watch`: tail it live |
|
|
374
|
-
| `--verify-findings` | `analyze`: execute the findings as sandbox replay proofs; each is marked VERIFIED (receipt path) or UNVERIFIABLE (reason). Needs `--replay-corpus` and a running sandbox (`SANDBOX_API_KEY` / `SANDBOX_API_URL`) |
|
|
375
|
-
| `--replay-corpus name=<labels>::<prepared>` | Trajectory source for `--verify-findings` (repeatable) |
|
|
376
|
-
| `--verify-out <dir>` | Receipt root for `--verify-findings` (default: `<--out>.verify`) |
|
|
377
|
-
| `--replay` | `stream`: scan once, then exit |
|
|
378
|
-
| `--once` | `stream`: scan once; `watch <target>`: print ONE snapshot and exit |
|
|
379
|
-
| `--no-spans` / `--no-findings` | `stream`: suppress raw span rows / finding rows |
|
|
380
|
-
| `--no-content` | `facts`: omit conversation, commands, paths and assignment text; `upload`: send metadata only |
|
|
381
|
-
| `--dry-run` / `--yes` | `upload`: preview without sending / skip the confirm prompt |
|
|
382
|
-
|
|
383
|
-
## Live stream
|
|
384
|
-
|
|
385
|
-
`traces watch` is the human terminal view.
|
|
386
|
-
It prints repeated-tool loops and semantic findings while a coding agent is still running.
|
|
387
|
-
|
|
388
|
-
`traces stream` is the machine feed.
|
|
389
|
-
It emits newline-delimited JSON events that a dashboard, art visualizer, local watcher, or hosted product can consume without scraping terminal prose.
|
|
390
|
-
|
|
391
|
-
```bash
|
|
392
|
-
traces stream --all
|
|
393
|
-
traces stream --all --mode findings
|
|
394
|
-
traces stream --all --mode agent --config traces.config.mjs
|
|
395
|
-
traces stream spans.openinference.jsonl --format openinference --no-spans
|
|
396
|
-
```
|
|
397
|
-
|
|
398
|
-
Live streaming emits `session`, `span`, `analysis_batch`, `finding`, and `tick` events; `--mode agent` also emits `report` events.
|
|
399
|
-
The semantic findings currently cover repeated failing commands, verification churn without code/config changes, completion claims without later verification, and high tool-error rates.
|
|
400
|
-
Use `--mode findings` when you want the low-volume meaning layer; keep `visualizer` for real-time views that need motion, timing, and tool-call texture.
|
|
401
|
-
Use `--mode agent` when another agent needs deterministic loop/tool-use reports alongside the findings.
|
|
402
|
-
The same JSONL shape is used for live sessions and replayed trace files:
|
|
403
|
-
|
|
404
|
-

|
|
405
|
-
|
|
406
|
-
Custom online analysts use the same config file as `investigate` and `improve`:
|
|
407
|
-
|
|
408
|
-
```js
|
|
409
|
-
export default {
|
|
410
|
-
liveAnalysts: [{
|
|
411
|
-
id: 'my-live-rule',
|
|
412
|
-
analyze(context) {
|
|
413
|
-
return context.actions.some((action) => action.kind === 'claim') ? [{
|
|
414
|
-
schemaVersion: 1,
|
|
415
|
-
kind: 'traces.live_finding',
|
|
416
|
-
id: `live.my-live-rule.${context.session.sessionId}`,
|
|
417
|
-
ruleId: 'my-live-rule',
|
|
418
|
-
fingerprint: `my-live-rule.${context.session.sessionId}`,
|
|
419
|
-
severity: 'info',
|
|
420
|
-
title: 'Completion claim seen',
|
|
421
|
-
claim: 'A claim-like assistant message appeared in the live trace.',
|
|
422
|
-
action: 'Require the next stream batch to include a verification action.',
|
|
423
|
-
check: 'A later finding or batch should show a test/build/typecheck action.',
|
|
424
|
-
evidence: [{ kind: 'metric', label: 'actions', value: String(context.actions.length) }],
|
|
425
|
-
session: context.session,
|
|
426
|
-
observedAt: context.generatedAt,
|
|
427
|
-
}] : []
|
|
428
|
-
},
|
|
429
|
-
}],
|
|
430
|
-
}
|
|
431
|
-
```
|
|
432
|
-
|
|
433
|
-
## Watch a run tree
|
|
434
|
-
|
|
435
|
-
`traces watch <target>` tails ONE run that fans work out to other agents, instead of one harness session.
|
|
436
|
-
It prints the tree at any depth: per node the status, the runtime, the authored budget, and the actual spend.
|
|
437
|
-
Plain stdout, no cursor control, no extra dependency — it works over SSH, in a CI log, and through a pipe.
|
|
438
|
-
|
|
439
|
-
```bash
|
|
440
|
-
traces watch runs/my-run # live: reprints only when something changed
|
|
441
|
-
traces watch runs/my-run --once # one snapshot, for a script or an agent
|
|
442
|
-
traces watch traces.otlp.jsonl # spans only, for any emitter
|
|
443
|
-
```
|
|
444
|
-
|
|
445
|
-
Two sources sit behind the one verb, and the output always names which number came from which:
|
|
446
|
-
|
|
447
|
-
| Source | Reads | Gives you | Cannot give you |
|
|
448
|
-
|---|---|---|---|
|
|
449
|
-
| **General** — OTLP spans | `trace_id` / `span_id` / `parent_span_id` in any `*.otlp.jsonl`, `*.spans.jsonl`, `*.traces.jsonl` | tree shape, wall time, per-turn tokens, cost when the emitter priced it | the authored budget — no span vocabulary carries one |
|
|
450
|
-
| **Specific** — the spawn journal | `spawn-journal.jsonl` written by agent-runtime's `createFileRunContext` | authored budget, settled spend, settlement status, blocking questions, steers, the typed result | per-turn detail the journal does not tap |
|
|
451
|
-
|
|
452
|
-
The general source is the one that matters for a system nobody has written yet: anything that emits a parent span pointer gets a tree, whether it is a supervisor, a pi session, a Claude subagent fan-out, or a custom fanout.
|
|
453
|
-
Attributes are read through `@tangle-network/agent-eval/trace-attributes` alias lists, so both live vocabularies — OpenInference's `llm.token_count.*` and GenAI semconv's `gen_ai.usage.*` — already work, and no key is invented here.
|
|
454
|
-
|
|
455
|
-
Two rules the view will not bend on:
|
|
456
|
-
|
|
457
|
-
- **Driver inference and child work never merge.** A node that drove and the work it delegated are separate lines, because one total cannot answer "which half spent the budget" — the decisive question in three real runs.
|
|
458
|
-
- **Unknown is never rendered as zero.** A turn the harness did not price reads `cost unknown (n unpriced)`, never `$0.0000`. A recorded `$0` paid on real tokens is printed AND flagged, because "free" and "unmetered" are different facts.
|
|
459
|
-
|
|
460
|
-
```
|
|
461
|
-
run proof-harness-mcp-20260731i
|
|
462
|
-
2 node(s) · depth 1 · elapsed 9m59s
|
|
463
|
-
driver total in 639,632 · out 35,077 · $0.0000 (zero across 58 priced)
|
|
464
|
-
worker total in 47,965 · out 515 · cost unknown (1 unpriced)
|
|
465
|
-
|
|
466
|
-
● · root [pending] supervisor inline budget 12it/400,000tok/$2.00/15m00s
|
|
467
|
-
driver 58 turn(s) · in 639,632 · out 35,077 · $0.0000 (zero across 58 priced) · 169% tok of budget
|
|
468
|
-
children 1 settled · in 47,965 · out 515 · cost unknown (1 unpriced)
|
|
469
|
-
└─ ✗ artifact-writer-s0 [failed] worker pi budget 6it/80,000tok wall 20.8s
|
|
470
|
-
settled 3it · in 47,965 · out 515 · cost unknown (1 unpriced) · 50% it, 61% tok of budget
|
|
471
|
-
|
|
472
|
-
! spend recorded $0 with tokens spent: usdKnown was never set false, so this reads as FREE rather
|
|
473
|
-
than unmetered. A harness-billed brain (a CLI on a subscription) looks exactly like this.
|
|
474
|
-
|
|
475
|
-
RESULT no-winner · budget-exhausted
|
|
476
|
-
```
|
|
477
|
-
|
|
478
|
-
`traces analyze --supervisor-run-dir <dir>` writes the full report for the same run.
|
|
479
|
-
It also reads `failure.json` when Runtime fails before the first spawn.
|
|
480
|
-
Pass a parent directory to combine reports from its Runtime runs.
|
|
481
|
-
Eval owns discovery, normalization, and analysis; Traces adds recorded coordination requests and delivery receipts.
|
|
482
|
-
Old Loops directories are unsupported.
|
|
483
|
-
|
|
484
|
-
Replay and resume are a separate concern: they belong to the durable spawn journal, and nothing here routes them through telemetry.
|
|
485
|
-
|
|
486
|
-
## Improvement engine
|
|
487
|
-
|
|
488
|
-
`traces improve` turns observed sessions into a portable improvement packet:
|
|
489
|
-
|
|
490
|
-
```bash
|
|
491
|
-
traces improve --all --last 20 --dir .traces/improvement
|
|
492
|
-
```
|
|
493
|
-
|
|
494
|
-
The directory contains:
|
|
495
|
-
|
|
496
|
-
| File | Purpose |
|
|
497
|
-
|---|---|
|
|
498
|
-
| `result.json` | findings, actions, checks, execution facts, adoption data, and the chosen LLM analysis route |
|
|
499
|
-
| `evidence.jsonl` | one row per evidence ref, suitable for downstream mining |
|
|
500
|
-
| `report.md` | human-readable report rendered from the typed data |
|
|
501
|
-
| `traces.otlp.jsonl` | canonical trace used by the analysts and execution accounting |
|
|
502
|
-
|
|
503
|
-
Bring your own analysts with a config file:
|
|
504
|
-
|
|
505
|
-
```bash
|
|
506
|
-
traces improve --last 5 --config examples/improvement-config.mjs --dir .traces/improvement
|
|
507
|
-
```
|
|
508
|
-
|
|
509
|
-
The config can export:
|
|
510
|
-
|
|
511
|
-
- `analysts`: deterministic or LLM analysts that implement the `agent-eval` `Analyst` contract
|
|
512
|
-
- `liveAnalysts`: deterministic online analysts that implement the `TraceLiveAnalyst` contract for `traces stream`
|
|
513
|
-
- `registry`: a prebuilt `AnalystRegistry`
|
|
514
|
-
- `externalAnalyzers`: HALO or any command/model adapter that reads the OTLP artifact
|
|
515
|
-
|
|
516
|
-
Traces does not pretend that an action is a measured candidate.
|
|
517
|
-
Use `agent-eval` to propose and compare candidate changes, `agent-runtime` to package an approved improvement, and `agent-interface` to represent profile edits.
|
|
518
|
-
|
|
519
|
-
## Ask questions
|
|
520
|
-
|
|
521
|
-
`traces ask` answers free-form questions about the selected sessions.
|
|
522
|
-
Each question becomes its own recursive investigation over the same trace, and the questions run at the same time under one budget.
|
|
523
|
-
|
|
524
|
-
```bash
|
|
525
|
-
traces ask --harness codex --session <id> \
|
|
526
|
-
--question "Which shell commands exited non-zero, and what were they?" \
|
|
527
|
-
--question "What was the last thing the human asked for?" \
|
|
528
|
-
--dir .traces/ask
|
|
529
|
-
```
|
|
530
|
-
|
|
531
|
-
Ask many questions from a file, and hold an answer to a shape a scorer can compare:
|
|
532
|
-
|
|
533
|
-
```bash
|
|
534
|
-
traces ask --all --last 3 --questions questions.json --concurrency 6 --budget 2 --dir .traces/ask
|
|
535
|
-
```
|
|
536
|
-
|
|
537
|
-
```json
|
|
538
|
-
[
|
|
539
|
-
"Which pull requests did this session open?",
|
|
540
|
-
{
|
|
541
|
-
"id": "merged-prs",
|
|
542
|
-
"question": "Which pull requests were merged?",
|
|
543
|
-
"instructions": "Count only merges the trace records, not merges the agent said it would do.",
|
|
544
|
-
"answerSchema": { "type": "array", "items": { "type": "integer" } }
|
|
545
|
-
}
|
|
546
|
-
]
|
|
547
|
-
```
|
|
548
|
-
|
|
549
|
-
An entry is a question string, or an object with `question` and an optional `id`, `instructions`, and `answerSchema`.
|
|
550
|
-
The schema accepts a small JSON Schema subset: `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, and `const`.
|
|
551
|
-
Any other keyword is rejected when the run starts, because a constraint that is silently ignored would let a wrong answer pass as checked.
|
|
552
|
-
|
|
553
|
-
The command writes two artifacts to `--dir`:
|
|
554
|
-
|
|
555
|
-
| File | Contents |
|
|
556
|
-
|---|---|
|
|
557
|
-
| `answers.json` | per question: the answer text, the parsed answer, every citation with whether it resolves, accepted findings, evidence-gate rejections by reason, model calls, tool calls, cost with provenance, and latency |
|
|
558
|
-
| `report.md` | the same run as readable Markdown: a summary line, a per-question table, then each answer with its citation and rejection notes |
|
|
559
|
-
|
|
560
|
-
What it checks, and what it costs:
|
|
561
|
-
|
|
562
|
-
- **Facts first.** Every question receives the deterministic [session-facts sheet](#session-facts) as prepared context before its first model call, at $0. The sheet is not citable; it names the span ids behind each fact.
|
|
563
|
-
- **Citations.** Every `trace://<trace_id>/span/<span_id>` URI in an answer is looked up in the trace. An answer that cites a span the trace does not hold fails.
|
|
564
|
-
- **Budget.** `--budget` is one ceiling shared by every question. `--question-budget` bounds one question. The run refuses to start when the budget cannot cover a single model call, and warns when the budget admits fewer concurrent calls than `--concurrency`.
|
|
565
|
-
- **Cost.** Each cost carries its provenance: `observed` from a provider receipt, `estimated` from token counts, or `uncaptured`. An uncaptured cost stays null; it never becomes zero.
|
|
566
|
-
- **Exit code.** `ask` writes both artifacts first, then exits 1 when any question failed, returned no answer, broke its schema, or cited a span that does not exist. A failed question never costs the other answers, and neither does Ctrl-C: the answers already bought are recorded, the questions the run never reached are recorded as `aborted`, and both artifacts are still written.
|
|
567
|
-
- **Wall time.** `totals.wallTimeMs` covers the whole run, including writing and indexing the trace file; `totals.setupTimeMs` says how much of it that setup was. `totals.peakConcurrency` is how many questions actually overlapped, which is at most `min(--concurrency, questions)`.
|
|
568
|
-
|
|
569
|
-
`ask` uses the same engine and credentials as `--llm`, so it needs `TANGLE_API_KEY` and a Python interpreter with `agent-eval-rpc[dspy]`.
|
|
570
|
-
Do not pass `--llm`; the command is model-backed by definition.
|
|
571
|
-
|
|
572
|
-
## Session facts
|
|
573
|
-
|
|
574
|
-
The SDK also accepts verified retained captures through
|
|
575
|
-
`parseRetainedSession({ harness, nativeSessionId, files, signal? })`.
|
|
576
|
-
Pass files as `{ path, content }` with their original relative layout.
|
|
577
|
-
Runtime or the caller owns custody, digest verification and capture completeness.
|
|
578
|
-
The reader uses existing adapters and selects a file only when native records
|
|
579
|
-
establish the supplied session identity.
|
|
580
|
-
It does not inspect the host session catalog or infer identity from filenames.
|
|
581
|
-
Directory-only stores and formats without a recorded native identity remain unavailable.
|
|
582
|
-
The returned source files, duplicate snapshots and unselected files explain the selection.
|
|
583
|
-
Identical or append-only copies count once; divergent histories remain unavailable.
|
|
584
|
-
Parsing is bounded to 1024 text files, 16 MiB per file, and 64 MiB total.
|
|
585
|
-
No model calls occur.
|
|
586
|
-
Call `buildSessionFactsReport(selection.spans, { includeContent: false })` to
|
|
587
|
-
withhold native bodies from the resulting facts.
|
|
588
|
-
The reader itself returns spans containing native content, for local trusted consumers only.
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
The same facts sheet includes observed tool outcomes, captured input/output byte
|
|
592
|
-
volumes, call-to-result intervals, compaction boundaries and available context
|
|
593
|
-
sizes, spawn intervals, and existing skill invocation/document-read evidence.
|
|
594
|
-
Each measurement carries coverage; absent telemetry remains unknown.
|
|
595
|
-
`facts --no-content` withholds conversation (including each turn's `answered` text),
|
|
596
|
-
command, changed-file, working-directory and assignment text while retaining
|
|
597
|
-
measurement metadata. This is a local metadata view, not a
|
|
598
|
-
share-safety certificate.
|
|
599
|
-
|
|
600
|
-
SDK consumers use `buildSessionFactsReport(spans, { includeContent: false })`.
|
|
601
|
-
Pass `window: { since, until }` (ISO bounds) for the windowed report `facts --since/--until` prints, and use
|
|
602
|
-
`createSessionFactsReportBuilder(options)` (`add(spans)`, `addUnread(...)`, `build()`) to build it one session at a time.
|
|
603
|
-
Cache by `measurementVersion` and `sourceDigest`; `evidenceThrough` is the latest
|
|
604
|
-
recorded endpoint, not current liveness. Computing the sheet parses the supplied
|
|
605
|
-
spans; returning a stored sheet needs no model call or trace scan. Semantic
|
|
606
|
-
judgments remain attributed analyst results.
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
`traces facts` prints the deterministic session-facts sheet: the answers a session audit needs first, computed straight from the spans.
|
|
610
|
-
No model call, no engine, no budget — it costs $0 and always returns the same sheet for the same spans.
|
|
611
|
-
|
|
612
|
-
```bash
|
|
613
|
-
traces facts --harness codex --session <id> # JSON on stdout
|
|
614
|
-
traces facts --harness codex --last 5 --format text # the short readable form
|
|
615
|
-
traces facts --otlp spans.otlp.jsonl --out facts.json
|
|
616
|
-
traces facts --harness claude-code --since 7d # every session with a record in the last week
|
|
617
|
-
traces facts --all --since 2026-10-14 --until 2026-10-15 # one day, across every harness
|
|
618
|
-
```
|
|
619
|
-
|
|
620
|
-
`--since <t>` and `--until <t>` open a half-open time window `[since, until)`; each takes `30m`/`2h`/`7d` before now or an ISO date, and either may be left open.
|
|
621
|
-
With a window, `facts` selects **every** session with a record that starts or ends inside it, not only the newest (`--last <n>` still caps it to the newest `n` per harness, and `--session` or `--otlp` still name the input).
|
|
622
|
-
Sessions are read one at a time, and a session file last modified before `since` is not read, because it cannot hold a later record.
|
|
623
|
-
Within a selected session, `humanTurns`, `excludedTurns` and `turnsByActor` keep only turns that started inside the window, so a one-day run counts only that day's turns; every other fact covers the whole session.
|
|
624
|
-
The report's `selection` states the window, both rules, the sessions read whose records all fell outside it (`outsideWindow`), and every session that could not be read (`unreadSessions`, each with its reason).
|
|
625
|
-
A report built without a window has `selection: null`.
|
|
626
|
-
|
|
627
|
-
| Fact | What it is |
|
|
628
|
-
|---|---|
|
|
629
|
-
| `toolCalls` | TOOL spans **this session's agent** actually invoked. Synthesized lifecycle spans are excluded and counted in `synthesizedToolSpans`, so the total is not high by the number of subagents; calls a spawned subagent made are excluded and counted in `subagentToolSpans` |
|
|
630
|
-
| `toolCallsByName` | the same calls by tool name, so a category decision is the reader's, not a guess |
|
|
631
|
-
| `subagentToolSpans` | TOOL spans excluded because a subagent, not this agent, made the call. Zero on a trace that carries no subagent records |
|
|
632
|
-
| `subagents` | every subagent-spawning call with the task name the adapter recorded |
|
|
633
|
-
| `pullRequests` | the pull requests the commands created and merged, each named by number or head branch, with the command span and how the identity was joined. Scanned the way a shell reads the script, so a `gh pr create` inside a heredoc body is not a command that ran |
|
|
634
|
-
| `cwd` | the working directory the harness recorded for the session (`traces.session.cwd` on the root span): Claude Code's first record `cwd`, Codex's `session_meta` cwd, Pi's session header, OpenCode's session directory (the older JSON store: its messages' `path.cwd`), Factory's `session_start`. Never decoded from a directory name or inferred from paths; `null` with the reason when the source names none or its adapter does not read one |
|
|
635
|
-
| `humanTurns` | `user.prompt` turns a person typed into this session, in order, with the timestamp. Inherited fork or compaction history, harness-injected blocks, and a second record of the same turn are excluded — each listed in `excludedTurns` with its reason and span ids, never silently dropped. `turnsByActor` shows every turn by actor so the filter is checkable. Each turn's `answered` is the last message this session's own agent wrote before it (not a subagent's, not inherited history), clipped to 1,500 characters with a `… [+N chars]` marker, with its span in `answeredSpanId`; `null` when the agent had written no text yet in the spans read (a `--latest-turn` read starts at the latest task, so its first turn has none) |
|
|
636
|
-
| `finalMessages` | the last message of the session's own agent, and of each subagent task, kept apart |
|
|
637
|
-
| `changedFiles` | paths the harness recorded for the changes it applied, plus paths named by patch headers and file-editing tool arguments, with the operation |
|
|
638
|
-
| `firstRecordAt` / `lastRecordAt` | the earliest span start and latest span end among this session's own records — a subagent that outlives the session does not stretch its window |
|
|
639
|
-
| `unreadRecords` | records the session reader could not parse, from the session's integrity receipt |
|
|
640
|
-
| `tokenTotal` | the harness's own cumulative token total, when a span carries `traces.session.total_tokens` |
|
|
641
|
-
| `firstFailure` | the first failure among this session's own records, ranked by agent-eval's fixed precedence: the innermost ERROR span that ended first, else a span whose `agent.outcome` is `fail`, else `none` — spans were observed and none of them failed. When every span is UNSET (a capture that failed before any status was written), or a session has no records of its own at all (every span is `traces.span.subagent` — spawned work folded into this trace, with nothing this agent recorded itself), the status is `unknown`, not `none`: the record does not say whether the run failed. Failures that ended at the same instant are `ambiguous` and list the candidates instead of picking one. `blame` separates machine and provider failures from the agent's own, and is `unknown` rather than `agent` when no evidence names the agent |
|
|
642
|
-
|
|
643
|
-
Three rules hold for every field:
|
|
644
|
-
|
|
645
|
-
- **Every fact is about this session's own agent.** Some harnesses fold a spawned agent's transcript into the parent's trace, so a trace can carry both what this agent did and what a child it spawned did. The child's work is named in `subagents` and counted in `subagentToolSpans`; it is never added to the parent's totals.
|
|
646
|
-
- **Every fact names its span ids.** `spanIds` lists the spans the value was computed from, so any number here can be opened and checked. The sheet itself is not a span and cannot be cited.
|
|
647
|
-
- **A fact the spans cannot support is `null` with its reason.** It is never guessed, and never a silent zero. `partial` marks a measured value that is known to be incomplete — a truncated patch, or a list above the entry cap.
|
|
648
|
-
|
|
649
|
-
Without a window, `facts` exits non-zero when a selected session cannot be read at all: a session that produced no record spans would otherwise print a sheet of zeros stating, in the sheet's own voice, that the session did nothing.
|
|
650
|
-
With a window, a session that cannot be read — the parser failed, or no record could be read — is named in `selection.unreadSessions` with its reason and a warning on stderr, and the run still exits 0 with the sessions it could read; it is never counted as a session with no turns.
|
|
651
|
-
|
|
652
|
-
The same sheet reaches the model-backed analysts as prepared context, before their first model call — see [Trace analysts](docs/trace-analysts.md#session-facts-as-prepared-context).
|
|
653
|
-
|
|
654
|
-
## Skill usage
|
|
655
|
-
|
|
656
|
-
`traces skills` indexes skill records in Claude Code, Codex, and Kimi Code sessions on this machine.
|
|
657
|
-
It makes no model call.
|
|
658
|
-
|
|
659
|
-
```bash
|
|
660
|
-
traces skills # every stored event, with the installed catalog joined
|
|
661
|
-
traces skills --since 30d # one window
|
|
662
|
-
traces skills --unused # installed skills with no use in the window, one per line
|
|
663
|
-
traces skills --format json # rows, unused skills, refresh cost and sweep exclusions
|
|
664
|
-
traces skills --no-refresh # answer from the store without reading transcripts
|
|
665
|
-
```
|
|
666
|
-
|
|
667
|
-
A run reads only transcript bytes appended since the previous run.
|
|
668
|
-
It parses only lines that can carry a skill record and stores the events in `~/.local/state/traces/skill-usage.json` (`--store`).
|
|
669
|
-
The first run reads the whole corpus: about two minutes for 41 GB on an M-series Mac.
|
|
670
|
-
Later runs take seconds, mostly to stat every session file.
|
|
671
|
-
Stored events survive a harness deleting old transcripts.
|
|
672
|
-
|
|
673
|
-
| Column | Harness | Counted record |
|
|
674
|
-
| --- | --- | --- |
|
|
675
|
-
| `model` | Claude Code | A Skill tool result that loaded a skill. A failed call does not count. |
|
|
676
|
-
| `slash` | Claude Code | A `/name` command whose next record is its expanded prompt. Built-in commands such as `/clear` do not expand and do not count. |
|
|
677
|
-
| `read` | Codex | A completed command that reads `<skill>/SKILL.md`. Codex records no skill event, so a read is the closest observable act. It does not show that the skill shaped the outcome. |
|
|
678
|
-
| `read` | Kimi Code | A successful `ReadFile` call or shell command that reads `<skill>/SKILL.md`, including streamed arguments. Kimi records no dedicated skill event. |
|
|
679
|
-
|
|
680
|
-
Each event is keyed by its record ID, so a resumed session that copies earlier records counts once.
|
|
681
|
-
One Codex or Kimi command that reads three or more skills (`--sweep`) is a catalog sweep, such as an audit; its reads are excluded and reported separately.
|
|
682
|
-
`installed` names the harnesses whose catalog lists the skill now: `~/.claude/skills` and enabled plugins for Claude Code, and `~/.codex/skills` and `~/.agents/skills` for Codex.
|
|
683
|
-
OpenCode, Pi, Gemini, Factory, and other harnesses are not indexed.
|
|
684
|
-
|
|
685
|
-
## Diff two runs
|
|
686
|
-
|
|
687
|
-
`traces diff` pairs the steps of two runs of the same task and marks where they first stop agreeing.
|
|
688
|
-
|
|
689
|
-
```bash
|
|
690
|
-
traces diff runs/a/otlp runs/b/otlp # two OTLP exports
|
|
691
|
-
traces diff a.sdk-events.jsonl b.sdk-events.jsonl --kind TOOL # any file convert reads; tool calls only
|
|
692
|
-
traces diff spans.otlp.jsonl#<trace id> other.jsonl --format json
|
|
693
|
-
traces diff run.otlp.jsonl#branch=arm-a run.otlp.jsonl#branch=arm-b # two arms of one run
|
|
694
|
-
```
|
|
695
|
-
|
|
696
|
-
A side is a file or directory that holds one run, `file#<trace id>` for one run of several, or `file#branch=<id>` for one arm: the spans whose `agent.branch.id` or `agent.branch.arm` is `<id>` and have no matched ancestor (a root), and every span below them. Sibling arms can share a branch id or label; when `<id>` names more than one such root subtree, the command refuses (exit 2) and lists each root's span id, rather than silently comparing the union of both.
|
|
697
|
-
Steps pair by span id first, then by position with the same name and kind, then by name and kind anywhere.
|
|
698
|
-
One inserted step therefore does not mark every later step as changed.
|
|
699
|
-
Paired steps compare status, tool and model.
|
|
700
|
-
The first divergence is `changed`, `replaced`, `only-in-a`, `only-in-b` or `reordered`, with its reason, followed by every changed and unpaired step.
|
|
701
|
-
`--kind` keeps only steps of that span kind, so event noise in a raw stream does not decide the first divergence.
|
|
702
|
-
The diff is agent-eval's `diffSteps`; this command reads the runs and prints it.
|
|
703
|
-
It exits 0 when the runs agree on outcome, 1 when they diverge, 2 when a side cannot be read (including an ambiguous `#branch=` selector), and 3 when the steps line up with no divergence but there is no recorded outcome to call agreement: neither side has a step with a definitive OK/ERROR status (both are entirely UNSET, the common shape for a capture that failed before any status was written), or neither side has a TOOL step. That case's report and text output carry a `caveat` naming which reason applies; a CI or agent gate should treat exit 3 as unknown, not as a pass.
|
|
704
|
-
|
|
705
|
-
## MCP server
|
|
706
|
-
|
|
707
|
-
`traces mcp` serves the selected traces to an MCP client over stdio.
|
|
708
|
-
|
|
709
|
-
```bash
|
|
710
|
-
claude mcp add traces -- traces mcp --otlp ./spans.otlp.jsonl
|
|
711
|
-
claude mcp add traces -- traces mcp --harness claude-code --last 5
|
|
712
|
-
```
|
|
713
|
-
|
|
714
|
-
It serves agent-eval's seven trace tools: `getDatasetOverview`, `queryTraces`, `countTraces`, `viewTrace`, `viewSpans`, `searchTrace` and `searchSpan`.
|
|
715
|
-
Each tool declares `readOnlyHint` and `idempotentHint` in its MCP annotations, and its description says that returned trace text is untrusted.
|
|
716
|
-
|
|
717
|
-
- The spans are redacted before the store is built — every span's name, attributes, status message and causal links' attributes, every field the store reads — and each result is redacted again at the boundary. The server then reassesses the redacted spans and refuses to start when they are still UNSAFE or UNKNOWN, the same gate `upload` applies to what it sends.
|
|
718
|
-
- Each result carries an `untrusted` notice beside it.
|
|
719
|
-
- A result above 512 KiB is refused with an error rather than truncated.
|
|
720
|
-
- `readSpanSource` is not served. It reads original source bytes in windows the caller picks, and a secret split across two windows matches no redaction rule.
|
|
721
|
-
|
|
722
|
-
## Session index
|
|
723
|
-
|
|
724
|
-
`traces index` writes one general JSON catalog over the selected sessions.
|
|
725
|
-
It is meant for deeper investigation and joins with other local data, not for one specific workflow.
|
|
726
|
-
|
|
727
|
-
```bash
|
|
728
|
-
traces index --all --since 24h --out session-index.json
|
|
729
|
-
traces inspect session-index.json --out inspection-report.md
|
|
730
|
-
```
|
|
731
|
-
|
|
732
|
-
The index contains:
|
|
733
|
-
|
|
734
|
-
- selection metadata and aggregate totals
|
|
735
|
-
- one row per session with harness, session id, path, cwd, repo labels, and time bounds
|
|
736
|
-
- behavior metrics: spans, LLM turns, tool calls, tool errors, tokens, models, and tools
|
|
737
|
-
- signal summaries: stuck loops and tool error rate
|
|
738
|
-
- nearby context files for joins: `AGENTS.md`, `CLAUDE.md`, and `.evolve` artifacts — the JSONL ledgers, `scorecard.json` / `current.json`, reflections, `handoffs/` plus the flat `handoff-*.md` convention, and `progress.md` — with markdown heading/ToC and JSONL key summaries
|
|
739
|
-
|
|
740
|
-
`traces inspect` reads that index back and prints ranked improvement findings over the sessions and nearby context.
|
|
741
|
-
It is intentionally read-only: it points to repeated-call loops, high tool-error sessions, missing repo attribution, long docs without Contents, invalid JSONL rows, and skill-run rows that cannot be joined back to a session.
|
|
742
|
-
|
|
743
|
-
## Session bundle
|
|
744
|
-
|
|
745
|
-
`traces bundle` assembles ONE session's durable evidence directory — the input for any downstream consumer that must cite the session after the live stores rotate.
|
|
746
|
-
It composes commands this CLI already owns and never spends a model call.
|
|
747
|
-
|
|
748
|
-
```bash
|
|
749
|
-
traces bundle --harness claude-code --session <id|path> --out bundle-dir
|
|
750
|
-
```
|
|
751
|
-
|
|
752
|
-
The bundle directory:
|
|
753
|
-
|
|
754
|
-
- `session/` — the transcript byte-for-byte, plus the sibling subagents directory when one exists
|
|
755
|
-
- `derived/` — `session-index.json`, the deterministic `report.md`, `evidence.jsonl`, and the OTLP span artifact
|
|
756
|
-
- `ledger/` — the repo's `.evolve` slices for the session window: `experiments.jsonl` and `skill-runs.jsonl` rows inside the padded span window, `current.json` / `scorecard.json` / `progress.md` copied whole, the latest flat `handoff-*.md`, and reflections dated inside the window
|
|
757
|
-
- `repo/git-log.txt` — commits in the session window from the session's cwd
|
|
758
|
-
- `manifest.json` — `view`, SHA-256 + byte count per file, provenance (session id, harness, cwd, transcript hash, session window), every absent artifact with its reason, per-slice row counts, and the bundle's known limits
|
|
759
|
-
|
|
760
|
-
A missing transcript fails the assembly loudly.
|
|
761
|
-
Every optional input that is absent (no subagents, no `.evolve`, no git repo) is recorded in `manifest.absent` with the probed path — a fact, not an error.
|
|
762
|
-
|
|
763
|
-
### Two views, two consumers
|
|
764
|
-
|
|
765
|
-
A bundle has two kinds of reader, and they need opposite things.
|
|
766
|
-
|
|
767
|
-
- An **auditor** cites the session and needs every source byte. That is the FULL view — what `traces bundle` writes, `manifest.view: "full"`.
|
|
768
|
-
- A **writer** must reach its own conclusion about the session, so it must not be able to read the conclusion that session already reached. That is the EVIDENCE-ONLY view, `manifest.view: "evidence-only"`.
|
|
769
|
-
|
|
770
|
-
The full bundle cannot serve the writer. Its transcript holds every word the session wrote, including the shell call that authored an earlier report, so a check that looks for the report FILE passes while the report CONTENT is right there. Projecting is the fix, not grepping:
|
|
771
|
-
|
|
772
|
-
```bash
|
|
773
|
-
traces bundle --harness claude-code --session <id|path> --out bundle-dir
|
|
774
|
-
traces bundle-view bundle-dir --view evidence-only --out writer-dir
|
|
775
|
-
```
|
|
776
|
-
|
|
777
|
-
The evidence-only view carries an explicit allow-list — `derived/session-index.json`, `derived/evidence.jsonl`, and the structured `ledger/` records (`current.json`, `scorecard.json`, `skill-runs.jsonl`). Everything else is excluded **by name** in `manifest.excluded`, with its SHA-256 and one of four rules:
|
|
778
|
-
|
|
779
|
-
| rule | what it covers |
|
|
780
|
-
|---|---|
|
|
781
|
-
| `session-source` | `session/` — the transcript and every subagent transcript |
|
|
782
|
-
| `session-text-derived` | `derived/report.md` (quotes the first prompt line and evidence excerpts) and `derived/trace.otlp.jsonl` (spans carry the full prompt and response text) |
|
|
783
|
-
| `authored-prose` | `ledger/progress.md`, `ledger/handoff-*.md`, `ledger/reflections/`, `ledger/experiments.jsonl` free-text, `repo/git-log.txt` |
|
|
784
|
-
| `not-allow-listed` | any other path, including one a later version of `traces` adds — the view can only narrow by accident, never widen |
|
|
785
|
-
|
|
786
|
-
The allow-list is the structure. A content check is the proof it is right: before anything is written, every candidate is compared against every excluded file for shared 8-word runs of prose, with identifiers (paths, URLs, hashes) and this package's own constant strings scrubbed from both sides first. A shared run in a repo ledger file drops that file with `rule: "content-signature"` and the match count; a shared run in an artifact `traces` derives is a defect in the derivation and no view is written at all. `manifest.projection.leakCheck` records the width, the sources compared, and the result — always `matches: 0`, because a match means no view exists.
|
|
787
|
-
|
|
788
|
-
A projected view also carries the full bundle's `manifest.json` hash, so an auditor can prove which record it came from. Projecting a view of a view is refused.
|
|
789
|
-
|
|
790
|
-
### Verify a bundle
|
|
791
|
-
|
|
792
|
-
```bash
|
|
793
|
-
traces bundle verify bundle-dir # VERIFIED, or FAILED with each difference; exit 1 on failure
|
|
794
|
-
traces bundle verify writer-dir --format json
|
|
795
|
-
```
|
|
796
|
-
|
|
797
|
-
`bundle verify` checks every file the manifest lists for its recorded size and SHA-256.
|
|
798
|
-
It reports a listed file that is missing, a file on disk that the manifest does not list, a manifest path that is absolute or climbs out of the directory, and any symbolic link.
|
|
799
|
-
For a projected view it also checks that the full bundle it names still has the manifest bytes the view recorded; a moved or deleted source is `unavailable`, not a failure.
|
|
800
|
-
It proves the bytes are the bytes `traces` wrote. It does not prove who wrote them: anyone who can rewrite a file can rewrite the manifest.
|
|
801
|
-
|
|
802
|
-
## Policy-mining evidence
|
|
803
|
-
|
|
804
|
-
`traces` does **not** emit benchmark campaign cells. It emits normalized coding-agent session evidence that another system can mine.
|
|
805
|
-
|
|
806
|
-
```bash
|
|
807
|
-
traces evidence --all --since 24h --out policy-evidence.jsonl --otlp-out spans.otlp.jsonl
|
|
808
|
-
```
|
|
809
|
-
|
|
810
|
-
`--last` follows recent file activity and may select a child session in multi-agent work.
|
|
811
|
-
Use `--harness codex --current --latest-turn --workflow` for the invoking Codex turn plus its connected workers.
|
|
812
|
-
Omit `--latest-turn` when the full history of a resumed session is the intended unit.
|
|
813
|
-
Workflow expansion follows session IDs, never timestamps or worker names.
|
|
814
|
-
For a latest Codex child, it selects the last parent task that structurally spawned or targeted that child ID, then includes only workers referenced by that parent task.
|
|
815
|
-
This requires the parent log to record the child ID on a spawn, send, follow-up, or lifecycle event and to associate that event with a stable turn ID.
|
|
816
|
-
If either field is absent, the workflow is reported incomplete instead of widening to the parent's full history.
|
|
817
|
-
Missing files, duplicate IDs, contradictory parents, and cycles are reported explicitly.
|
|
818
|
-
Claude Code subagent files are already folded into their parent trace, so `--workflow` normally resolves one Claude session file.
|
|
819
|
-
With `--latest-turn`, ordinary Claude subagents are selected by their parent tool call ID.
|
|
820
|
-
Claude Workflow subagents are selected by the run ID and transcript directory returned by the parent `Workflow` call.
|
|
821
|
-
Each Workflow child attaches to the latest matching call that started before the child.
|
|
822
|
-
Task selection then keeps only children attached to calls in that task.
|
|
823
|
-
Returned directories from another resumed Claude session are included in parsing and source hashes.
|
|
824
|
-
For a copied Claude session, preserve `<session UUID>/subagents/workflows/<run ID>` beside the parent JSONL file.
|
|
825
|
-
Recorded native `.claude/projects/<project>/<session UUID>/subagents/workflows/<run ID>` paths resolve to that copied subtree when outside the selected store.
|
|
826
|
-
Copy every referenced Workflow directory, including runs stored under another resumed session UUID.
|
|
827
|
-
The original transcript stays unchanged, and child attribution uses its recorded Workflow identity.
|
|
828
|
-
Missing copied directories, conflicting source bindings, and symlinks that change the copied session or run identity stop parsing.
|
|
829
|
-
Copied Workflow directories contain regular transcript and metadata files; symlink entries stop parsing.
|
|
830
|
-
Missing or conflicting Workflow identities stop the parse instead of widening the selected history.
|
|
831
|
-
Cross-harness parent/child trees are not inferred.
|
|
832
|
-
For a reproducible report, run `traces list` first and pass its session ID with `--session`.
|
|
833
|
-
Ephemeral `codex exec --json` output has no discovery location, so select its file with `--harness codex-exec --session <path>`.
|
|
834
|
-
The adapter rejects empty, incomplete, or other JSONL formats instead of emitting zero-valued session evidence.
|
|
835
|
-
|
|
836
|
-
Each JSONL row is one session:
|
|
837
|
-
|
|
838
|
-
- session provenance: harness, session id, cwd, path, mtime
|
|
839
|
-
- explicit-session source binding: `provenance.sourceSha256` over every adapter-declared input file, including Claude worker and metadata files
|
|
840
|
-
- repo labels: `tangle.subject.key`, `git.repository`, branch, commit
|
|
841
|
-
- behavior metrics: span counts, LLM turns, tool calls, errored tool calls, tokens, models, tool histogram
|
|
842
|
-
- mining signals: stuck loops and tool error rate
|
|
843
|
-
- provenance marker: `notCampaignCell: true`
|
|
844
|
-
|
|
845
|
-
That boundary matters. `agent-lab` campaign `cells.jsonl` says "arm X beat arm Y on task Z." `traces evidence` says "this real agent session had this repo/model/tool/failure shape." A downstream policy compiler can cluster these rows, propose candidate policies, then validate those policies in a separate eval campaign.
|
|
846
|
-
|
|
847
|
-
### Export existing evidence/events to OpenInference
|
|
848
|
-
|
|
849
|
-
If you already have compact evidence or event captures on disk, convert them to the same OpenInference JSONL shape that `traces analyze --analyzer halo` uses:
|
|
850
|
-
|
|
851
|
-
```bash
|
|
852
|
-
traces export policy-evidence.jsonl --out spans.openinference.jsonl
|
|
853
|
-
traces export sandbox-events.json --format sandbox-events --out spans.openinference.jsonl
|
|
854
|
-
traces export trajectory.json --format chat-trajectory --out spans.openinference.jsonl
|
|
855
|
-
halo spans.openinference.jsonl --prompt "Analyze this trace slice" --max-turns 1
|
|
856
|
-
```
|
|
857
|
-
|
|
858
|
-
`traces export` accepts:
|
|
859
|
-
|
|
860
|
-
- compact `traces.policy_evidence.session` JSONL from `traces evidence`
|
|
861
|
-
- JSON arrays with `start`, `raw`, `result`, `done`, and `error` events. A tool call becomes one span, keyed by its message part id, however many events report on that part. The Sandbox SDK records each call as a `message.part.updated` event and again as the harness's raw `tool_use` event. The span takes its start from the part's first event, its final state from the last, and `ERROR` if any event reported one. The root span's `traces.event.tool_part_updates_merged` counts the events folded in.
|
|
862
|
-
- existing OpenInference JSONL, rewritten through the local redaction path
|
|
863
|
-
- chat message arrays and objects with a `messages` array
|
|
864
|
-
|
|
865
|
-
Run `traces export --help` for the full command reference.
|
|
866
|
-
|
|
867
|
-
## Upload to the Intelligence Platform
|
|
868
|
-
|
|
869
|
-
`upload` **redacts locally before anything leaves the machine**, dedups against already-uploaded sessions, and tags each with metadata (harness, cwd, git branch, host).
|
|
870
|
-
It then checks the final events of each session, exactly as they would be sent, with the share-safety verdict.
|
|
871
|
-
A session that is UNSAFE or UNKNOWN is refused: it is not sent, not written to the dry-run preview, and not marked uploaded, and the command exits 1.
|
|
872
|
-
|
|
873
|
-
```bash
|
|
874
|
-
traces upload --since 24h --dry-run # see exactly what would be sent; no network
|
|
875
|
-
traces upload --since 24h --no-content # send metadata only; drop all prompt/response text
|
|
876
|
-
traces upload --since 24h # send it
|
|
877
|
-
```
|
|
878
|
-
|
|
879
|
-
It needs `TANGLE_INGEST_URL` (or `TANGLE_ORCHESTRATOR_URL`), `TANGLE_INGEST_API_KEY` (or `TANGLE_API_KEY`), and `TANGLE_TENANT_ID`. Without them, `--dry-run` still works fully.
|
|
880
|
-
|
|
881
|
-
### Redaction scope: read this before uploading prose
|
|
882
|
-
|
|
883
|
-
Redaction is agent-eval's redaction core ([docs/redaction.md](https://github.com/tangle-network/agent-eval/blob/main/docs/redaction.md)), `default` profile.
|
|
884
|
-
It finds credentials by field name (`apiKey`, `x-api-key`, `client.secret`, `refreshToken`) and by value shape (provider keys, GitHub, Slack, AWS and Google tokens, JWTs, bearer headers, private keys, `KEY=secret` assignments, credentials in URLs), and replaces the whole string.
|
|
885
|
-
It replaces email, card, SSN and phone values in place.
|
|
886
|
-
Token counts such as `input_tokens` and `max_tokens` are kept.
|
|
887
|
-
It runs over every span attribute, including the captured prompt/response text.
|
|
888
|
-
|
|
889
|
-
It does **not** catch names, postal addresses or account numbers written in prose. Those need a context-aware model. Three postures, strongest first:
|
|
890
|
-
|
|
891
|
-
1. **`--no-content`**: upload metadata only (tool calls, tokens, timing, loop signal); no prose leaves the machine.
|
|
892
|
-
2. **`--redactor <cmd>`**: run an ML PII scrubber (e.g. [`openai/privacy-filter`](https://github.com/openai/privacy-filter)) after the redaction core.
|
|
893
|
-
3. Default: the redaction core.
|
|
894
|
-
|
|
895
|
-
Always `--dry-run` first to see exactly what would be sent.
|
|
896
|
-
|
|
897
|
-
### Check before sharing
|
|
898
|
-
|
|
899
|
-
`verify-safe` reads files and says whether they are safe to share. It never changes them.
|
|
900
|
-
|
|
901
|
-
```bash
|
|
902
|
-
traces verify-safe bundle-dir # share profile (default)
|
|
903
|
-
traces verify-safe session.jsonl --profile strict --format json
|
|
904
|
-
traces verify-safe out/ --known-secret-env TANGLE_API_KEY # also look for this exact key, in any encoding
|
|
905
|
-
```
|
|
906
|
-
|
|
907
|
-
| Status | Meaning | Exit |
|
|
908
|
-
|---|---|---|
|
|
909
|
-
| `SAFE` | Nothing found | 0 |
|
|
910
|
-
| `SAFE_WITH_WARNINGS` | Only warnings: raw content, identifiers, embedded media | 0 |
|
|
911
|
-
| `UNSAFE` | A credential remains, or personal data under `share` or `strict` | 1 |
|
|
912
|
-
| `UNKNOWN` | A file, line or byte range could not be read, so its safety is unknown | 2 |
|
|
913
|
-
|
|
914
|
-
`.json` files are parsed whole and `.jsonl` or `.ndjson` files line by line. Any other file is scanned as text.
|
|
915
|
-
Each finding names the file, line and JSON Pointer, never the matched value.
|
|
916
|
-
|
|
917
|
-
## Trace analysts
|
|
918
|
-
|
|
919
|
-
Use the local checks by default.
|
|
920
|
-
Add a model-backed engine only when the question needs semantic judgment or behavior discovery.
|
|
921
|
-
|
|
922
|
-
```bash
|
|
923
|
-
traces analyze --last 1
|
|
924
|
-
traces analyze --last 1 --llm --budget 0.50
|
|
925
|
-
traces analyze --last 1 --analyzer halo --analyzer-prompt "find token waste"
|
|
926
|
-
traces analyze --all --last 20 --analyzer hodoscope
|
|
927
|
-
traces analyze --last 1 --analyzer prime
|
|
928
|
-
traces analyze --last 1 --analyzer my-installed-command
|
|
929
|
-
```
|
|
930
|
-
|
|
931
|
-
To ask your own question instead of the built-in kinds, use [`traces ask`](#ask-questions).
|
|
932
|
-
|
|
933
|
-
HALO returns a diagnosis report.
|
|
934
|
-
Hodoscope samples distinct behaviors and marks every sample `needs_review`.
|
|
935
|
-
Prime posts the full span projection to an OpenAI-compatible bridge (`TRACES_PRIME_BRIDGE_URL`, default `http://localhost:4181`) and returns validated findings with span evidence.
|
|
936
|
-
An arbitrary command returns a raw report unless its SDK adapter explicitly parses a stricter output type.
|
|
937
|
-
|
|
938
|
-
Read [Trace analysts](./docs/trace-analysts.md) for the output contract, a minimal custom analyst, and labeled benchmark setup.
|
|
939
|
-
|
|
940
|
-
**Redactors** scrub prompt/response prose with an external PII model (catching names/addresses the regex pass can't), running *after* the built-in redaction:
|
|
941
|
-
|
|
942
|
-
```bash
|
|
943
|
-
# the command reads a JSON array of strings on stdin, writes the scrubbed array on stdout
|
|
944
|
-
traces upload --since 24h --dry-run --redactor "my-pii-scrubber"
|
|
945
|
-
```
|
|
946
|
-
|
|
947
|
-
In the SDK these are the `ExternalAnalyzer` and `Redactor` interfaces (`haloAnalyzer`, `hodoscopeAnalyzer`, `primeAnalyzer`, `commandAnalyzer`, `commandRedactor`, `applyRedactor`, `runExternalAnalyzers`).
|
|
948
|
-
See [`examples/external-engines.ts`](./examples/external-engines.ts).
|
|
949
|
-
|
|
950
|
-
> The built-in agentic analysts (`--llm`) run on the Tangle router by default: set `TANGLE_API_KEY`.
|
|
951
|
-
> `OPENAI_API_KEY` alone targets OpenAI instead, and `OPENAI_BASE_URL` overrides the endpoint for any other OpenAI-compatible gateway.
|
|
952
|
-
> `--llm` also needs a Python interpreter with `agent-eval-rpc[dspy]` installed, because agent-eval's model-backed analysts run through the DSPy RLM engine out of process; set `TRACES_PYTHON` to choose the interpreter.
|
|
953
|
-
> The bridge protocol is version-locked: install the exact version matching this package's `@tangle-network/agent-eval` dependency (`pip install "agent-eval-rpc[dspy]==$(npm view @tangle-network/traces dependencies.@tangle-network/agent-eval)"`) — a skewed bridge kills every agentic analyst at startup.
|
|
954
|
-
> When `--llm` was requested and every agentic analyst fails, `analyze`/`investigate`/`improve` still write the deterministic report, then exit 1 with each analyst's underlying error.
|
|
955
|
-
> A requested `--analyzer` behaves the same way: its error is written into the report, and `analyze` then exits 1 naming every external analyzer that failed.
|
|
956
|
-
> Every deterministic command — `list`, `analyze` without `--llm`, `convert`, `index`, `inspect`, `export`, `evidence`, `stream`, `watch`, `analyze --supervisor-run-dir` — needs neither a key nor Python.
|
|
957
|
-
|
|
958
|
-
## Agent skills
|
|
959
|
-
|
|
960
|
-
The npm package ships two AgentProfile-ready skills:
|
|
961
|
-
|
|
962
|
-
| Skill | Use |
|
|
963
|
-
|---|---|
|
|
964
|
-
| [`inspect-agent-traces`](./skills/inspect-agent-traces/SKILL.md) | select workflows and export deterministic findings |
|
|
965
|
-
| [`build-trace-analyst`](./skills/build-trace-analyst/SKILL.md) | write and calibrate a custom analyst |
|
|
966
|
-
|
|
967
|
-
Load both from GitHub:
|
|
968
|
-
|
|
969
|
-
```ts
|
|
970
|
-
import { defineGitHubResource } from '@tangle-network/agent-interface'
|
|
971
|
-
|
|
972
|
-
const ref = process.env.TRACES_SKILLS_REF
|
|
973
|
-
if (!ref) throw new Error('TRACES_SKILLS_REF must be an immutable commit')
|
|
974
|
-
|
|
975
|
-
const names = ['inspect-agent-traces', 'build-trace-analyst']
|
|
976
|
-
const skills = names.map((name) => defineGitHubResource(`skills/${name}/SKILL.md`, {
|
|
977
|
-
repository: 'tangle-network/traces',
|
|
978
|
-
ref,
|
|
979
|
-
name,
|
|
980
|
-
}))
|
|
981
|
-
const profile = { name: 'trace-reviewer', resources: { skills } }
|
|
982
|
-
```
|
|
983
|
-
|
|
984
|
-
## Release automation
|
|
985
|
-
|
|
986
|
-
Releases use a reviewed version commit because `main` is protected:
|
|
987
|
-
|
|
988
|
-
1. Update `package.json` and `pnpm-lock.yaml` to the intended version on a branch.
|
|
989
|
-
2. Merge that change through a pull request.
|
|
990
|
-
3. Tag the exact merge commit as `vX.Y.Z` and push the tag.
|
|
991
|
-
4. The Publish workflow verifies that the tag is merged into `main` and matches `package.json`, then builds, publishes to npm, and creates a GitHub release.
|
|
992
|
-
5. `pnpm check:package` proves the npm tarball contains the `traces` binary before release.
|
|
993
|
-
|
|
994
|
-
To retry a failed publish without changing source or version, manually run the Publish workflow with the existing tag.
|
|
995
|
-
|
|
996
|
-
## Library (SDK)
|
|
997
|
-
|
|
998
|
-
The CLI is a thin consumer of these exports.
|
|
999
|
-
|
|
1000
|
-
| Export | Signature | Use |
|
|
1001
|
-
|---|---|---|
|
|
1002
|
-
| `readOtlpInput` | `(file\|dir) → { files, skipped, rows, spans, validation, issues }` | ingest OTLP from **any** system — no adapter; never throws on a foreign trace |
|
|
1003
|
-
| `validateTraceSpans` | `(spans) → { ok, findings, capabilities }` | the contract's conformance check, re-exported |
|
|
1004
|
-
| `conformanceOfSpans` | `(spans) → TraceValidation` | grade spans this package produced, on the artifact a consumer receives |
|
|
1005
|
-
| `renderValidation` / `renderConformance` | `(validation, opts) → string` | the `validate` report / the section embedded in an analysis report |
|
|
1006
|
-
| `skippedAnalyses` | `(validation) → SkippedAnalysis[]` | which analyses a missing capability empties, and why |
|
|
1007
|
-
| `unavailableCapabilities` / `incompleteInputsNote` | `(validation) → Map` / `(map, name) → string` | the gate: what the trace cannot support, and the marker each affected section carries |
|
|
1008
|
-
| `summarizeSpanStructure` | `(spans) → SpanStructure` | how much of the tree attaches: rootless, orphans, links |
|
|
1009
|
-
| `analyzeLoopConvergence` | `(spans) → LoopConvergenceReport` | per loop, the per-round outcome/score series and whether it improved, plateaued, regressed or fell back |
|
|
1010
|
-
| `analyzeSteeringChain` | `(spans) → SteeringChainReport` | which verdict span caused which subsequent round, following `links` |
|
|
1011
|
-
| `llmSpan` / `toolSpan` / `loopSpan` / `branchSpan` / `steeredBy` | contract builders, re-exported | emit conforming spans from your own system |
|
|
1012
|
-
| `analyzeSpans` | `(spans, { registry?, ai?, budgetUsd? }) → AnalyzeResult` | run built-in analysts, or **your own** via `registry` |
|
|
1013
|
-
| `runTraceInvestigation` | `(TraceInvestigationOptions) → TraceInvestigationResult` | typed findings with actions/checks, execution facts, external analyzer output, and report |
|
|
1014
|
-
| `runTraceImprovement` | `(TraceImprovementOptions) → TraceImprovementResult` | writes the full findings, evidence, report, and trace artifact pack |
|
|
1015
|
-
| `runTraceQuestions` | `(TraceQuestionsOptions) → TraceQuestionsResult` | ask many free-form questions of one span list, concurrently, under one shared cost ledger; keeps each answer and checks its citations |
|
|
1016
|
-
| `buildTraceFindingPacket` | `({ findings }) → TraceFindingPacket` | render any `AnalystFinding[]` without changing its schema |
|
|
1017
|
-
| `runTraceStoreInvestigation` | `({ traceStore }) → TraceStoreInvestigationResult` | run the same packet layer over a hosted/custom `TraceAnalysisStore` |
|
|
1018
|
-
| `loadTracesConfig` | `(path?) → TracesConfig \| undefined` | load BYO analysts and external analyzers |
|
|
1019
|
-
| `watchSessions` | `(ObserverOptions) → Promise<void>` | live observer; `onLoop` / `onReport` / `signal` / `adapters` |
|
|
1020
|
-
| `streamSessions` | `(TraceStreamOptions) → Promise<void>` | live JSONL-ready event stream over active sessions |
|
|
1021
|
-
| `traceStreamEventsFromSpans` | `(spans, opts?) → TraceStreamEvent[]` | replay an existing span list as stream events |
|
|
1022
|
-
| `analyzeLiveBatch` | `(spans, opts?) → TraceLiveBatch` | compute semantic online findings for one batch |
|
|
1023
|
-
| `classifyLiveActions` | `(spans) → TraceLiveAction[]` | classify spans once as read/edit/verify/claim/tool/other |
|
|
1024
|
-
| `defaultTraceLiveAnalysts` | `TraceLiveAnalyst[]` | the built-in online analysts; extend or replace them |
|
|
1025
|
-
| `collectSessionIndex` | `(ScanOptions) → TraceSessionIndex` | scan sessions and return a reusable JSON-ready catalog |
|
|
1026
|
-
| `inspectSessionIndex` | `(TraceSessionIndex) → TraceInspectionReport` | rank improvement findings from an index without rescanning sessions |
|
|
1027
|
-
| `buildPolicyEvidenceRecord` | `(ref, spans, opts?) → PolicyEvidenceRecord` | summarize one session for downstream policy mining |
|
|
1028
|
-
| `collectPolicyEvidence` | `(ScanOptions) → PolicyEvidenceRecord[]` | scan harness sessions and emit policy-evidence rows |
|
|
1029
|
-
| `collectSessionWorkflow` | `(adapter, seeds, opts?) → SessionWorkflow` | resolve a bounded parent/child session tree with explicit relationship issues |
|
|
1030
|
-
| `collectSessionSelection` | `(groups, opts?) → SessionSelection` | parse, expand, and optionally byte-bind selected session groups |
|
|
1031
|
-
| `exportTraceEvidenceFile` | `(path, opts?) → { format, spans, redactionCount }` | convert compact evidence/events/OpenInference files to redacted OpenInference spans |
|
|
1032
|
-
| `scanSessions` | `(ScanOptions) → AsyncIterable<ScannedSession>` | the shared locate→parse iterator |
|
|
1033
|
-
| `collectSessions` | `(CollectOptions) → SessionBatch[]` | redacted per-session batches for your own pipeline |
|
|
1034
|
-
| `redactSpans` | `(spans, { profile?, knownSecrets? }) → { spans, report }` | redact spans with agent-eval's redaction core |
|
|
1035
|
-
| `assessSpans` / `verifySafe` | `(spans, profile?)` / `(path, { profile?, knownSecrets? })` → verdict | share-safety verdict for spans or files |
|
|
1036
|
-
| `planUpload` / `executeUpload` | `(…, { backend? }) → …` | redact + dedup + send to any sink |
|
|
1037
|
-
| `selectAdapters` / `listAdapters` / `resolveAdapter` | adapter selection + the harness registry |
|
|
1038
|
-
| `PiAdapter` | `new PiAdapter({ sessionsRoot? })` | parse Pi's default session tree or an exact `PI_CODING_AGENT_SESSION_DIR` |
|
|
1039
|
-
| `HarnessTraceAdapter` | interface (`locate` + `parse`) | implement to add a harness |
|
|
1040
|
-
| `ExternalAnalyzer` / `Redactor` | `haloAnalyzer` / `hodoscopeAnalyzer` / `commandAnalyzer` / `commandRedactor` | drive engines/models you install |
|
|
36
|
+
| Find a session | `list` | [Refs, selection and harnesses](docs/select.md) |
|
|
37
|
+
| What happened, exactly, for $0 | `facts` | [Session facts](docs/facts.md) |
|
|
38
|
+
| What went wrong, ranked, for $0 | `analyze` | [Analyze a session](docs/analyze.md) |
|
|
39
|
+
| A question facts cannot answer, with checked citations | `ask` | [Ask a question](docs/ask.md) |
|
|
40
|
+
| The one gate: contracts, conformance, share safety | `check` | [Check](docs/check.md) |
|
|
41
|
+
| Where two runs split | `diff` | [Diff two runs](docs/diff.md) |
|
|
42
|
+
| Live supervision of sessions or a run tree | `watch` | [Watch](docs/watch.md) |
|
|
43
|
+
| Hand spans to another tool | `convert` | [Convert](docs/convert.md) |
|
|
44
|
+
| Keep a session's evidence | `bundle` | [Session bundle](docs/bundle.md) |
|
|
45
|
+
| Send redacted sessions to Intelligence | `upload` | [Upload](docs/upload.md) |
|
|
46
|
+
| Which skills are used | `skills` | [Skill usage](docs/skills.md) |
|
|
47
|
+
| Span tools for an MCP client with no shell | `mcp` | [MCP server](docs/mcp.md) |
|
|
48
|
+
|
|
49
|
+
Each page embeds its command's `--help`, and `tests/docs-help.test.ts` holds them equal to the CLI.
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
traces: what your agents did, where they wasted time, whether a run passes
|
|
53
|
+
Start here for one session: traces facts <ref> (25 lines); one field: add --format json | jq '<path>'
|
|
54
|
+
|
|
55
|
+
Which sessions ran? traces list [--cwd .] [--last N]
|
|
56
|
+
What happened (turns, PRs, failure, cost)? $0 traces facts <ref> | --current | --since 1d
|
|
57
|
+
Did we already try X / what was said? $0 traces facts <ref> | --since 30d --match '<re>'
|
|
58
|
+
Where did it waste time or get interrupted? $0 traces analyze <ref>
|
|
59
|
+
A question facts can't answer (model, $) traces ask <ref> --question "…" --budget 1
|
|
60
|
+
Does this run pass? (CI) traces check <ref> --contract c.json|conformance
|
|
61
|
+
Is this safe to share? traces check <path> --share-safe
|
|
62
|
+
Where did two runs diverge? traces diff <a> <b>
|
|
63
|
+
What is happening right now? traces watch [<run-dir>] [--once]
|
|
64
|
+
Keep a session's evidence traces bundle <ref> --out <dir> · bundle verify <dir>
|
|
65
|
+
Hand spans to another tool traces convert <ref> --out <file>
|
|
66
|
+
Ship sessions to Intelligence traces upload <ref>|--since T [--dry-run]
|
|
67
|
+
Which skills are used? traces skills [--unused]
|
|
68
|
+
Serve span tools over MCP traces mcp <ref>|--last N
|
|
69
|
+
|
|
70
|
+
ref: session id, or 8+ char unique prefix or suffix; session file; OTLP file/dir; bundle dir; run dir; store root
|
|
71
|
+
Select: --current --last N --since/--until 2h|1d|ISO --cwd DIR --harness ID --latest-turn
|
|
72
|
+
Output: --format text|json|jsonl|junit|github --out PATH --no-content
|
|
73
|
+
Exit: 0 yes · 1 no (fail, diverged, unsafe) · 2 could not answer (stderr says why)
|
|
74
|
+
traces <command> --help for flags, jq paths and examples · moved from 0.x: traces help moved
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## Library
|
|
1041
78
|
|
|
1042
79
|
```ts
|
|
1043
|
-
import {
|
|
1044
|
-
AnalystRegistry, analyzeSpans, makeFinding, readOtlpInput, renderValidation,
|
|
1045
|
-
streamSessions, watchSessions,
|
|
1046
|
-
} from '@tangle-network/traces'
|
|
80
|
+
import { computeSessionFacts, readOtlpInput } from '@tangle-network/traces'
|
|
1047
81
|
|
|
1048
|
-
// Read OTLP any system emitted, and find out what it can answer before analysing:
|
|
1049
82
|
const input = await readOtlpInput('spans.otlp.jsonl') // or a directory
|
|
1050
|
-
console.log(
|
|
1051
|
-
|
|
1052
|
-
|
|
1053
|
-
// Observe live sessions, feed findings anywhere (read-only, cancellable):
|
|
1054
|
-
const c = new AbortController()
|
|
1055
|
-
await watchSessions({ all: true, signal: c.signal, onLoop: (l) => alert(`${l.toolName} x${l.occurrences}`) })
|
|
1056
|
-
|
|
1057
|
-
// Feed a visualizer or dashboard:
|
|
1058
|
-
await streamSessions({ all: true, signal: c.signal, includeSpans: false, onEvent: (event) => console.log(event) })
|
|
1059
|
-
|
|
1060
|
-
// Run your own analyst instead of the built-ins:
|
|
1061
|
-
const registry = new AnalystRegistry()
|
|
1062
|
-
registry.register({
|
|
1063
|
-
id: 'mine', description: '…', inputKind: 'trace-store', cost: { kind: 'deterministic' }, version: '1.0.0',
|
|
1064
|
-
async analyze() {
|
|
1065
|
-
return [makeFinding({ analyst_id: 'mine', area: 'custom', claim: '…', severity: 'info', evidence_refs: [], confidence: 0.9 })]
|
|
1066
|
-
},
|
|
1067
|
-
})
|
|
1068
|
-
await analyzeSpans(input.spans, { registry })
|
|
83
|
+
console.log(input.validation.capabilities) // what this trace can answer
|
|
84
|
+
const [facts] = computeSessionFacts(input.spans)
|
|
1069
85
|
```
|
|
1070
86
|
|
|
1071
|
-
|
|
1072
|
-
|
|
1073
|
-
Runnable, in [`examples/`](./examples):
|
|
1074
|
-
|
|
1075
|
-
| File | Shows |
|
|
1076
|
-
|---|---|
|
|
1077
|
-
| [`observe-and-alert.ts`](./examples/observe-and-alert.ts) | tail live sessions and alert on stuck loops |
|
|
1078
|
-
| [`custom-analyst.ts`](./examples/custom-analyst.ts) | register and run your own analyst |
|
|
1079
|
-
| [`improvement-config.mjs`](./examples/improvement-config.mjs) | plug in BYO analysts for `traces stream` / `traces improve` |
|
|
1080
|
-
| [`custom-backend.ts`](./examples/custom-backend.ts) | redact + dedup + upload to your own sink |
|
|
1081
|
-
| [`register-harness.ts`](./examples/register-harness.ts) | add a new harness by implementing `HarnessTraceAdapter` |
|
|
1082
|
-
| [`external-engines.ts`](./examples/external-engines.ts) | drive HALO, Hodoscope, and an external PII scrubber |
|
|
87
|
+
The SDK is 25 functions and 25 types, one engine per command: [Library (SDK)](docs/sdk.md). Runnable code is in [examples](examples/README.md).
|
|
1083
88
|
|
|
1084
|
-
##
|
|
89
|
+
## Docs
|
|
1085
90
|
|
|
1086
|
-
|
|
1087
|
-
|
|
1088
|
-
|
|
1089
|
-
single read cannot cover: counts over hundreds of records, timestamps of specific records,
|
|
1090
|
-
and facts linked across tool calls and sessions.
|
|
1091
|
-
|
|
1092
|
-
```bash
|
|
1093
|
-
pnpm bench:audit fixtures --out /tmp/audit-bench # the tree an arm may read
|
|
1094
|
-
pnpm bench:audit gold --out /tmp/audit-gold.json # the answer key, kept away from the arm
|
|
1095
|
-
pnpm bench:audit score /tmp/arm-answers.json --fixtures /tmp/audit-bench
|
|
1096
|
-
```
|
|
1097
|
-
|
|
1098
|
-
[`bench/audit/README.md`](./bench/audit/README.md) documents the questions, the scoring
|
|
1099
|
-
rules, the answers-file format, and which answers are re-derived from the generated bytes.
|
|
1100
|
-
The benchmark ships outside the published package; the model arms are run by hand.
|
|
1101
|
-
|
|
1102
|
-
## Develop
|
|
1103
|
-
|
|
1104
|
-
```bash
|
|
1105
|
-
pnpm install
|
|
1106
|
-
pnpm dev analyze --harness claude-code --last 1 # run from source via tsx
|
|
1107
|
-
pnpm test
|
|
1108
|
-
pnpm typecheck
|
|
1109
|
-
pnpm build # dist/index.js (SDK) + dist/cli.js (bin) + .d.ts
|
|
1110
|
-
```
|
|
91
|
+
- **Commands:** the pages in the table above.
|
|
92
|
+
- **Reference:** [Library (SDK)](docs/sdk.md) · [Migrate to 1.0](docs/migrate-1.0.md) · [Parsers outside traces](docs/parsers.md)
|
|
93
|
+
- **Contribute:** [Develop and release](docs/develop.md): local setup, releases and how quality is measured
|