@tangle-network/traces 0.23.1 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (74) hide show
  1. package/README.md +61 -1078
  2. package/dist/analyst-model-call-JVUOYZ5V.js +20 -0
  3. package/dist/ask-XLS3VANY.js +39 -0
  4. package/dist/check-YF6ZHPGG.js +322 -0
  5. package/dist/chunk-3I4PJ5Q7.js +25 -0
  6. package/dist/chunk-3T62R7ON.js +5695 -0
  7. package/dist/chunk-4VKPN432.js +138 -0
  8. package/dist/chunk-6CLC7DKP.js +492 -0
  9. package/dist/chunk-7CBV5RJ2.js +535 -0
  10. package/dist/chunk-B62DOFGI.js +273 -0
  11. package/dist/chunk-CL3OQOWX.js +407 -0
  12. package/dist/chunk-DLGAKCQJ.js +782 -0
  13. package/dist/chunk-DU534HAX.js +263 -0
  14. package/dist/chunk-GLUDGUGJ.js +232 -0
  15. package/dist/chunk-J2PM5SVN.js +503 -0
  16. package/dist/chunk-M3W6TZHQ.js +381 -0
  17. package/dist/chunk-NH7HCDLB.js +759 -0
  18. package/dist/chunk-NZ5Y7UEL.js +1217 -0
  19. package/dist/chunk-O52XK7OB.js +19 -0
  20. package/dist/chunk-PYH3PL3N.js +62 -0
  21. package/dist/chunk-RZL5GAWH.js +60 -0
  22. package/dist/chunk-TR6O4S4T.js +208 -0
  23. package/dist/chunk-TW43MR5D.js +259 -0
  24. package/dist/chunk-TYLFUX42.js +228 -0
  25. package/dist/chunk-UOLRCQHC.js +120 -0
  26. package/dist/chunk-UVE5ANVV.js +3921 -0
  27. package/dist/chunk-V2OCGX57.js +53 -0
  28. package/dist/cli.js +2086 -2050
  29. package/dist/diff-ZZ6KYBPP.js +16 -0
  30. package/dist/improvement-OJQ6365E.js +25 -0
  31. package/dist/index.d.ts +1235 -4357
  32. package/dist/index.js +53 -1204
  33. package/dist/live-Y64AJ6MA.js +14 -0
  34. package/dist/mcp-ZPGTJQYQ.js +85 -0
  35. package/dist/report-JAMXRTCW.js +338 -0
  36. package/dist/run-watch-PLWTIRSV.js +649 -0
  37. package/dist/supervisor-report-RMPLLBGZ.js +29 -0
  38. package/dist/supervisor-run-context-WO6VOTGV.js +30 -0
  39. package/dist/upload-R6NMKSGQ.js +20 -0
  40. package/dist/verify-safe-C5ZXZOZ2.js +8 -0
  41. package/docs/analyze.md +76 -0
  42. package/docs/ask.md +88 -0
  43. package/docs/bundle.md +40 -0
  44. package/docs/check.md +77 -0
  45. package/docs/convert.md +44 -0
  46. package/docs/develop.md +32 -0
  47. package/docs/diff.md +38 -0
  48. package/docs/facts.md +114 -0
  49. package/docs/mcp.md +36 -0
  50. package/docs/migrate-1.0.md +33 -0
  51. package/docs/parsers.md +66 -0
  52. package/docs/release-1.0.0.md +147 -0
  53. package/docs/sdk.md +77 -0
  54. package/docs/select.md +70 -0
  55. package/docs/skills.md +46 -0
  56. package/docs/upload.md +47 -0
  57. package/docs/watch-jsonl.v1.example.jsonl +5 -0
  58. package/docs/watch.md +66 -0
  59. package/examples/README.md +9 -0
  60. package/examples/custom-analyst.ts +65 -74
  61. package/install.sh +22 -2
  62. package/package.json +8 -7
  63. package/skills/inspect-agent-traces/SKILL.md +47 -110
  64. package/dist/chunk-7DVZCCZI.js +0 -20983
  65. package/docs/replay-verify.md +0 -123
  66. package/docs/stream-findings.svg +0 -39
  67. package/docs/trace-analysts.md +0 -643
  68. package/examples/custom-backend.ts +0 -21
  69. package/examples/external-engines.ts +0 -62
  70. package/examples/improvement-config.mjs +0 -63
  71. package/examples/observe-and-alert.ts +0 -19
  72. package/examples/register-harness.ts +0 -26
  73. package/skills/build-trace-analyst/SKILL.md +0 -76
  74. package/skills/build-trace-analyst/agents/openai.yaml +0 -4
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # traces
2
2
 
3
- > Point `traces` at any agent trace — OTLP spans your own system emits, or the session logs Claude Code, Codex, OpenCode, Pi, Kimi, Factory and Gemini already write to disk. Get failure-mode and efficiency findings, plus a straight answer about what the trace cannot tell you. A CLI *and* an SDK.
3
+ > What your coding agents did, where they wasted time, and whether a run passes: exact answers from the sessions Claude Code, Codex, OpenCode, Pi, Kimi and Factory already write to disk, or from the OTLP spans your own system emits. A CLI and an SDK.
4
4
 
5
5
  ![traces analyzing a real Claude Code session](https://raw.githubusercontent.com/tangle-network/traces/main/docs/demo.gif)
6
6
 
@@ -8,1103 +8,86 @@
8
8
  [![license](https://img.shields.io/npm/l/@tangle-network/traces.svg)](./LICENSE)
9
9
  [![node](https://img.shields.io/node/v/@tangle-network/traces.svg)](https://nodejs.org)
10
10
 
11
- It reads a run as spans and reports where the agent got stuck, burned tokens, or stopped checking its own work. The deterministic pass runs locally, with no API key and no cost.
12
-
13
- There are two ways in, and they are not equal:
14
-
15
- | | Use when | How |
16
- |---|---|---|
17
- | **Emit the contract** | You control the system | Emit [`@tangle-network/agent-trace-contract`](https://github.com/tangle-network/agent-sdk/tree/main/packages/agent-trace-contract) spans, then `traces analyze --otlp <file\|dir>` |
18
- | **Adapters** | You do *not* control the format | `traces analyze --harness claude-code` reads a proprietary session store and translates it |
19
-
20
- Emitting the contract is the supported way to integrate a new system. The adapters are the legacy edge — they exist because the coding agents write formats nobody standardised, and each one is a translation somebody has to maintain. If you can choose, choose `--otlp`.
21
-
22
- ## Contents
23
-
24
- - [Install](#install)
25
- - [Quick start](#quick-start)
26
- - [Gate CI on a trace contract](#gate-ci-on-a-trace-contract)
27
- - [Integrate your own system](#integrate-your-own-system)
28
- - [What it finds](#what-it-finds)
29
- - [Supported harnesses](#supported-harnesses)
30
- - [CLI reference](#cli-reference)
31
- - [Live stream](#live-stream)
32
- - [Watch a run tree](#watch-a-run-tree)
33
- - [Improvement engine](#improvement-engine)
34
- - [Ask questions](#ask-questions)
35
- - [Session facts](#session-facts)
36
- - [Skill usage](#skill-usage)
37
- - [Diff two runs](#diff-two-runs)
38
- - [MCP server](#mcp-server)
39
- - [Session index](#session-index)
40
- - [Session bundle](#session-bundle) · [Two views](#two-views-two-consumers)
41
- - [Policy-mining evidence](#policy-mining-evidence)
42
- - [Upload to the Intelligence Platform](#upload-to-the-intelligence-platform) · [Check before sharing](#check-before-sharing)
43
- - [Trace analysts](#trace-analysts)
44
- - [Agent skills](#agent-skills)
45
- - [Library (SDK)](#library-sdk)
46
- - [Examples](#examples)
47
- - [Audit benchmark](#audit-benchmark)
48
- - [Develop](#develop)
11
+ Every command except `ask` runs locally, with no key and no cost. Agents: run traces, never parse a session store yourself; the [`inspect-agent-traces`](skills/inspect-agent-traces/SKILL.md) skill maps each question to its command.
49
12
 
50
13
  ## Install
51
14
 
52
15
  ```bash
53
16
  curl -fsSL https://raw.githubusercontent.com/tangle-network/traces/main/install.sh | bash
54
- traces --version
55
-
56
- npx --yes @tangle-network/traces@latest analyze --harness claude-code --last 1 # run without installing
57
- npm i -g @tangle-network/traces # install manually
58
- npm i @tangle-network/traces # use it as a library
17
+ npm i @tangle-network/traces # as a library
59
18
  ```
60
19
 
61
- Requires Node ≥ 22.
20
+ Requires Node 22 or later. `install.sh` refuses to overwrite another program's `traces` and names it. Coming from 0.x: [Migrate to 1.0](docs/migrate-1.0.md).
62
21
 
63
22
  ## Quick start
64
23
 
65
24
  ```bash
66
- traces validate spans.otlp.jsonl # what can this trace answer?
67
- traces analyze --otlp spans.otlp.jsonl # analyse it, no adapter involved
68
- traces analyze --harness claude-code --last 1 # or read a coding agent's own log
69
- traces check session.jsonl --contract contract.json # gate CI on the path the run took
70
- traces improve --harness claude-code --last 5 --dir .traces/improvement
71
- traces watch --all
72
- traces stream --all --mode findings
73
- ```
74
-
75
- The first command is shown in the demo above.
76
- The **deterministic pass** checks stuck loops, token growth, output decay, missing self-verification, tool failures, and human corrections.
77
- Each supported issue is returned as a finding with evidence, an action, confidence, and a validation plan.
78
- It needs no API key and costs nothing.
79
-
80
- Add `--llm` for the **agentic analysts**. `traces` first uses free local signals to choose the smallest useful set: a failure review always runs, while knowledge and edit reviews run only when failures, repeated calls, or corrective feedback support them. They call OpenAI and respect `--budget <usd>`.
81
-
82
- Every run also writes a **canonical OpenInference JSONL artifact**.
83
- Use it with [HALO](https://github.com/context-labs/halo), [Hodoscope](https://github.com/ManifoldRG/hodoscope), or any command that reads the file.
84
- See [Trace analysts](#trace-analysts).
85
-
86
- `traces improve` is the reviewable action path.
87
- It writes one typed result, one report, flattened evidence rows, and the canonical OTLP trace.
88
- Each finding already contains the claim, evidence, recommended action, confidence, and validation plan.
89
-
90
- ## Gate CI on a trace contract
91
-
92
- Two runs can give the same answer while one of them calls a forbidden tool.
93
- An output check passes both; a trace contract fails the wrong one.
94
-
95
- ```json
96
- {
97
- "name": "refund-desk",
98
- "run": { "requireCompleted": true, "maxDurationMs": 300000 },
99
- "tools": { "required": ["Read"], "forbidden": ["Bash"], "allowed": ["Read", "Grep"], "maxCalls": 10, "enforced": true },
100
- "retries": { "reads": ["Read", "Grep"] },
101
- "llm": { "maxCalls": 10 }
102
- }
103
- ```
104
-
105
- ```bash
106
- claude -p "$TASK" --tools Read,Grep --strict-mcp-config --output-format stream-json --verbose > run.jsonl
107
- traces check run.jsonl --contract refund-desk.json --junit contract.xml
25
+ traces list --cwd . --last 5 # which sessions ran here
26
+ traces facts --current # what this session did: turns, PRs, first failure, cost, $0
27
+ traces facts --since 30d --match 'pnpm test' # did we already try X
28
+ traces analyze 019a3c7e # where it wasted time, ranked
29
+ traces check run.jsonl --contract contract.json # fail CI when a run took a forbidden path
108
30
  ```
109
31
 
110
- `tools.enforced` fails when the harness offered the model a tool the contract does not allow, which a flag such as `--tools` does not prove.
111
- Without `--strict-mcp-config`, the run above is also offered every MCP server the account or project configures.
112
- `retries` fails when a tool repeats a call with the same arguments and nothing proves the repeat is safe.
113
-
114
- The trace is `claude -p --output-format stream-json` output, a Claude Code or Codex session file, OTLP spans (a file or a directory), or trace evidence; `--format` names it when the file could be read as more than one.
115
- Only stream-json output records the offered tools and the run's own result.
116
- **A saved interactive session file carries no run status at all** (the `result` record above only exists in `-p` mode), so `run.requireCompleted`/`run.allowedStatuses` always fail on one — write a contract for a session file with no `run` key, like the `refund-desk` example above running against `run.jsonl`.
117
- Every rule prints `pass`, `fail`, `error`, or `skipped` (a rule of an alternative path that did not decide the result).
118
-
119
- | Exit | Meaning |
120
- |---|---|
121
- | 0 | Every rule passed. |
122
- | 1 | A rule failed. |
123
- | 2 | The contract is malformed or contradicts itself, or a rule could not be evaluated. |
124
- | 3 | The trace could not be read. |
125
- | 4 | The trace reference is ambiguous. |
126
-
127
- `--junit` writes one test case per rule, and GitHub annotations print under `GITHUB_ACTIONS=true` or `--annotations`.
128
- `traces check --contract <file> --explain` prints what each compiled rule checks and reads no trace.
129
- For a trace that does not pass, the verdict, the contract, its plain-language statement, and the redacted spans the violations cite go to `--evidence` (default `.traces/check`).
130
- The contract keys and rule semantics are in [agent-eval's trace-contract guide](https://github.com/tangle-network/agent-eval/blob/main/docs/trace-contracts.md).
131
-
132
- ## Integrate your own system
133
-
134
- No adapter, no plugin. Emit the span contract and point `--otlp` at the file.
135
-
136
- ```bash
137
- npm i @tangle-network/agent-trace-contract # zero runtime dependencies
138
- ```
139
-
140
- You install it to PRODUCE spans. This package already depends on it to read
141
- them, so `npx @tangle-network/traces@latest` arrives with the contract-backed
142
- analyses working: trace conformance, `traces validate`, `--otlp` inbound
143
- reading, round-over-round convergence and the steering chain, and OpenInference
144
- / intelligence-spans conversion.
145
-
146
- ```ts
147
- import { llmSpan, loopSpan, steeredBy, toolSpan } from '@tangle-network/agent-trace-contract'
148
-
149
- // One round of a loop, the model call and tool call inside it, then the next
150
- // round — linked to the verdict that CAUSED it, not parented to it.
151
- const round1 = loopSpan({ traceId, spanId: 'round-1', parentSpanId: null, name: 'round 1',
152
- loopId: 'fix-loop', iteration: 1, startTime, endTime })
153
- const call = llmSpan({ traceId, spanId: 'llm-1', parentSpanId: 'round-1', name: 'coder turn',
154
- model: 'glm-5.2', inputTokens: 1200, outputTokens: 340, costUsd: 0.0042, startTime, endTime })
155
- const tool = toolSpan({ traceId, spanId: 'tool-1', parentSpanId: 'round-1', name: 'bash',
156
- toolName: 'bash', startTime, endTime })
157
- const round2 = loopSpan({ traceId, spanId: 'round-2', parentSpanId: null, name: 'round 2',
158
- loopId: 'fix-loop', iteration: 2, resumed: true, startTime, endTime,
159
- links: [steeredBy('llm-1', traceId)] })
160
- ```
161
-
162
- Write one span per line to a `.jsonl` file, then:
163
-
164
- ```bash
165
- traces validate spans.otlp.jsonl # exit 1 only when the file is not a trace
166
- traces analyze --otlp spans.otlp.jsonl # a directory reads the OTLP files under it
167
- ```
32
+ ## Commands
168
33
 
169
- A directory is a run directory, not a span directory: it holds the span export
170
- next to raw event, stream and SDK logs that are also `*.jsonl`. Only the OTLP
171
- files are read — every other JSONL is listed in the report with what it actually
172
- holds, rather than counted as several hundred broken spans. Put the exports in an
173
- `otlp/` subdirectory and that decides it outright.
174
-
175
- Writing commands (`convert`, `export`, `evidence`, `upload`) take `--otlp-out`; they refuse `--otlp`.
176
-
177
- `parent_span_id` is CONTAINMENT — this span happened inside that one. `links` is
178
- CAUSALITY — that verdict caused this retry. Encoding causality as a parent claims
179
- a nesting that never existed, so the two edges stay separate and both survive a
180
- round-trip through `traces`.
181
-
182
- ### What a trace cannot tell you, said out loud
183
-
184
- `validate` reports the conformance of any OTLP file, conforming or not — it never
185
- throws on a foreign trace, it reports it:
186
-
187
- - **findings** by severity, each naming the *consequence* of the defect and the capabilities it blocks
188
- - **the capability table** — `token-accounting`, `cost-attribution`, `tool-usage`, `loop-convergence`, `tree-comparison`, `steering-chain`, `latency-analysis` — available or not, with the trace's own reason
189
- - **span structure** — how much of the tree actually attaches, and how many rows could not be read at all
190
-
191
- Exit code is 1 on exactly one condition: not one entry could be read as a span,
192
- so the file is not a trace. Severity says what a finding claims about the INPUT,
193
- not how loud it is — `error` is "nothing readable", `warn` is "readable but
194
- degraded", `info` is "nothing measurable lost" — so a `warn` exits 0 and the
195
- capability table, not the exit code, says what the degradation cost.
196
-
197
- #### A re-export never validates cleaner than its source
198
-
199
- `--otlp-out` writes the spans that were analysed. Anywhere this package
200
- normalized a field to analyse it — an `end_time` that precedes its `start_time`,
201
- a status code the contract does not recognise — the export writes what the
202
- PRODUCER wrote and keeps the normalized value in an attribute beside it, so
203
- `negative-duration` and `invalid-status` are still findings against the copy.
204
-
205
- A row that never became a span is the one thing that cannot be re-emitted: the
206
- artifact is also the file the analysts read, so a non-span line in it is garbage
207
- fed to the analysis and a repaired one is invented work. The export refuses the
208
- equivalence instead — it carries `traces.source.unreadable_rows` and the kinds
209
- behind it, accumulating across every hop, and `validate` on a re-export leads
210
- with how much of the original is missing from it. Round-tripping a defective
211
- trace through `traces` cannot produce a clean bill of health.
212
-
213
- `analyze` prints the same section, plus an **analyses skipped, and why** list —
214
- and, crucially, carries that verdict INTO the report. A section whose inputs are
215
- incomplete says so in its own heading and again directly above its table, so a
216
- number and the reason it is wrong can never end up forty lines apart. When the
217
- trace supports `tree-comparison`, the table names the command that compares its
218
- arms: `traces diff <file>#branch=<a> <file>#branch=<b>`.
219
-
220
- Two of those analyses are why the loop shape exists at all:
221
-
222
- - **round-over-round convergence** — per `agent.loop.id`, the per-round outcome
223
- and score series, and whether it improved, plateaued, regressed, or peaked and
224
- fell back. The verdict is read from the round's own `agent.outcome` or from the
225
- shallowest EVALUATOR span under it.
226
- - **steering chain** — follows `links` (`agent.link.kind`) to report which verdict
227
- caused which subsequent round. A link whose target is missing is kept and marked,
228
- because a shorter chain than the producer recorded is the opposite of the truth.
229
-
230
- Parent and verdict lookups use both trace and span IDs.
231
- Steering edges retain `causeTraceId` and `effectTraceId`, including unresolved targets.
232
- Generic chat transcripts report unknown execution status because message text does not establish completion.
233
-
234
- Nothing is repaired on the way in. A span with an unreadable timestamp is
235
- reported and excluded, because a synthesized value would put invented work into a
236
- total.
237
-
238
- ## What it finds
239
-
240
- The deterministic pass (free, no key) surfaces:
241
-
242
- | Finding | Meaning |
243
- |---|---|
244
- | **Stuck loops** | the same tool called N× with identical args and no state change |
245
- | **Monotonic input growth** | full history re-sent every step; context never compressed |
246
- | **Output-length decay** | planning/reasoning per step shrinking as context grows |
247
- | **No self-verification** | state-mutating actions never followed by an eval/inspect/check |
248
- | **Tool mix / retry / error rates** | repeated, retried, and failed tool calls |
249
-
250
- `--llm` adds agentic analysts that read the conversation and cluster higher-order failure and improvement signals. They receive the compact deterministic findings first, then share their earlier findings with later analysts. With `traces improve`, the selected analysts and reasons are saved in `result.json` and the report, so another agent can reuse the packet without rereading the raw trace.
251
-
252
- ## Supported harnesses
253
-
254
- Adapters are the **legacy edge**: they exist for coding agents whose on-disk format we do not
255
- control, and each is a translation from a proprietary session store onto the span model. They are
256
- not the way to integrate a system you own — for that, [emit the contract](#integrate-your-own-system).
257
-
258
- "Verified" = tested against real sessions; "fixture" = tested against schema-accurate fixtures (no real sessions available).
259
-
260
- | Harness (aliases) | Reads from | Status |
34
+ | Job | Command | Page |
261
35
  |---|---|---|
262
- | `claude-code` (`claude`, `claudish`, `openclaw`, `nanoclaw`) | `~/.claude/projects/<cwd>/*.jsonl` (+ subagent sidechains), or an explicit `claude -p --output-format stream-json --verbose` JSONL file | verified |
263
- | `codex` (`codex-acp`) | `~/.codex/sessions/**/rollout-*.jsonl` | verified |
264
- | `codex-exec` (`codex-json`) | explicit `codex exec --json` JSONL file | fixture |
265
- | `opencode` | `~/.local/share/opencode/opencode.db` (shared reader), plus historical `storage/` JSON sessions (legacy reader) | verified |
266
- | `gemini` (`gemini-cli`) | `~/.gemini/tmp/<hash>/chats/session-*.json` | verified |
267
- | `pi` | `~/.pi/agent/sessions/<cwd>/*.jsonl` | verified |
268
- | `kimi` (`kimi-code`) | `~/.kimi/sessions/<cwd hash>/<session id>/wire.jsonl` | verified |
269
- | `factory` (`factory-droids`, `droid`) | `~/.factory/sessions/<cwd>/*.jsonl` + `.settings.json` sidecar | verified |
270
- | `qwen` (`qwen-code`) | `~/.qwen/projects/<cwd>/chats/*.jsonl` | fixture |
271
- | `amp` | `~/.local/share/amp/threads/T-*.json` | fixture |
272
- | `github-copilot` (`copilot`) | `~/.copilot/session-state/<id>/events.jsonl` | fixture |
273
-
274
- Every adapter captures the conversation stored in one session file: the **user's prompt** and the **assistant's response** text, plus tool calls/results and token usage.
275
- Claude Code, Codex, current OpenCode, Pi, Kimi, and Factory read native records through [`@tangle-network/harness-sessions`](https://github.com/tangle-network/agent-sdk/tree/main/packages/harness-sessions); traces adds conversation and workflow spans over its model and tool calls. Historical OpenCode JSON sessions remain readable through the older adapter because their IDs do not overlap the SQLite store. Kimi's wire log has no served-model field, so its model remains unknown rather than being guessed.
276
- Claude Code's nested subagent files are folded into the parent trace.
277
- Claude Code's stream-json output also records the tools the harness offered the model and whether the run failed.
278
- The root span keeps them as `gen_ai.tool.definitions` and `run.status`, so a trace contract can check what the harness enforced.
279
- Exported native-session span IDs derive deterministically from the session and record IDs.
280
- The Claude transcript proves turns and tool calls, but does not prove loop iterations or causal links, so those fields remain absent.
281
- Codex stores each worker in a separate session file; add `--workflow` to resolve the connected coordinator and worker files.
282
- `github-copilot` is the one exception: its log format carries no user prompt.
283
- Factory stores token totals in `.settings.json`, not per turn.
284
- ACP-only bridges may not persist a local transcript.
285
-
286
- ## CLI reference
287
-
288
- ```bash
289
- traces list --harness claude-code --last 20 # discover sessions
290
- traces analyze --harness codex --session <id> # pin one ID printed by `list`
291
- traces analyze --harness codex --current # this session, even when a child wrote later
292
- traces analyze --harness codex --current --latest-turn --workflow # current turn plus workers
293
- traces analyze --harness claude-code --session <path> --latest-turn # latest task plus its subagents
294
- traces investigate --all --last 10 --out report.md # explicit investigation alias
295
- traces improve --all --last 10 --dir .traces/improvement
296
- traces ask --harness codex --session <id> --question "Which commands failed?"
297
- traces facts --harness codex --session <id> # the deterministic facts sheet, $0
298
- traces facts --all --since 2026-10-14 --until 2026-10-15 # every session with a record that day, that day's turns
299
- traces skills --since 30d # skill use across every local session, incremental, $0
300
- traces diff run-a.otlp.jsonl run-b.otlp.jsonl # where two runs of one task first diverge
301
- traces mcp --otlp spans.otlp.jsonl # read-only trace tools for an MCP client
302
- traces analyze --all --since 2026-06-18 --out report.md
303
- traces check spans.otlp.jsonl --contract c.json --junit c.xml # exit 0 pass, 1 fail, 2 bad contract, 3 unreadable, 4 ambiguous
304
- traces validate spans.otlp.jsonl # conformance; exit 1 only when it is not a trace
305
- traces validate results/sessions --out conformance.md # a whole directory of exports
306
- traces analyze --otlp spans.otlp.jsonl # analyse foreign OTLP, no adapter
307
- traces analyze --otlp results/sessions --out report.md
308
- traces convert --harness claude-code --last 1 --otlp-out spans.jsonl # OTLP only
309
- traces index --all --since 24h --out session-index.json
310
- traces bundle --harness claude-code --session <id|path> --out bundle-dir # one session's durable evidence dir
311
- traces bundle-view bundle-dir --view evidence-only --out writer-dir # same session, without its own words
312
- traces bundle verify bundle-dir # every file against its manifest SHA-256
313
- traces inspect session-index.json --out inspection-report.md
314
- traces evidence --harness codex --last 20 --out policy-evidence.jsonl
315
- traces evidence --harness codex-exec --session /tmp/codex.jsonl --cwd "$PWD" --out policy-evidence.jsonl
316
- traces export policy-evidence.jsonl --out spans.openinference.jsonl
317
- traces import-codetracebench verified.jsonl --trajectory-dir normalized --out traces --revision <40-or-64-character-hex>
318
- traces watch --all # live observer; loops + semantic findings
319
- traces watch runs/my-run # live run tree: status, budget, driver vs child spend
320
- traces watch runs/my-run --once # one snapshot, then exit
321
- traces watch traces.otlp.jsonl # any OTLP span file, any emitter
322
- traces analyze --supervisor-run-dir runs/my-run # the full tree report
323
- traces stream --all --mode findings # low-volume semantic feed
324
- traces stream --all --mode agent # findings + deterministic report events
325
- traces stream spans.openinference.jsonl --format openinference --no-spans
326
- traces upload --since 1h --dry-run # redact + dedup + preview, no network
327
- traces upload --since 24h # upload last day to the Intelligence Platform
328
- traces verify-safe bundle-dir # SAFE / SAFE_WITH_WARNINGS / UNSAFE (exit 1) / UNKNOWN (exit 2)
329
- traces replay-verify --steps steps.json --image <docker-image> --at 37 --cwd /home --out ./replay-out \
330
- --fix-command "<corrected step>" # executed proof: replay prefix, reproduce failure, show fix
331
- traces verify-findings --findings findings.json --out ./receipts \
332
- --steps steps.json --image <replay-ready-image> --cwd /app # execute recorded analyst findings as proofs
333
- traces analyze --last 1 --llm --verify-findings --replay-corpus h=labels.json::prepared/ # proof-carrying analyze
334
- ```
335
-
336
- `replay-verify` replays a CodeTraceBench-style trajectory prefix in a real sandbox and executes step k twice — recorded (does the failure reproduce?) and corrected (does it vanish?).
337
- `verify-findings` runs that proof per analyst finding and annotates each with `reproduced | fix-flipped | divergent | not-replayable` plus a receipt directory; see [Verified findings](./docs/trace-analysts.md#verified-findings-executed-replay).
338
- See [Replay verification](./docs/replay-verify.md) for setup, semantics, and honest limits (SWE-style trajectories with a docker image only; commands run as the non-root sandbox user).
339
-
340
- | Flag | Meaning |
341
- |---|---|
342
- | `--harness <id>` | Harness or alias (default: `claude-code`) |
343
- | `--all` | Every known harness |
344
- | `--last <n>` | Most-recent N sessions |
345
- | `--current` | Active Codex session from `CODEX_THREAD_ID` |
346
- | `--latest-turn` | Limit a resumed Codex or Claude Code session to its latest task turn |
347
- | `--session <id\|path>` | One listed session ID or explicit session file |
348
- | `--workflow` | Expand selected files through stable parent and child session IDs |
349
- | `--max-workflow-sessions <n>` | Stop workflow expansion before parsing more than `n` files; default `100` |
350
- | `--cwd <dir>` | Filter by working directory |
351
- | `--since <t>` | `30m`/`2h`/`7d` before now, or an ISO date. Commands that select sessions read those modified at or after it; `upload`: its window (default 24h); `facts`: opens a time window — see [Session facts](#session-facts) |
352
- | `--until <t>` | `facts`: closes the time window (exclusive), same forms as `--since` |
353
- | `--out <path>` | Write the report to a file |
354
- | `--dir <path>` | `improve`: write the full artifact pack to this directory; `ask`: write `answers.json` + `report.md` there |
355
- | `--otlp <file\|dir>` | **READ** OTLP-JSONL from any system, skipping the adapters; a directory reads the OTLP files under it (only `otlp/` when the producer made one) and names the JSONL that is not OTLP. `validate`, `check`, `analyze`, `investigate`, `improve`, `ask`, `stream` |
356
- | `--contract <file>` | `check`: the declarative trace contract (JSON) |
357
- | `--junit <file>` | `check`: write JUnit XML, one test case per rule |
358
- | `--evidence <dir>` | `check`: where evidence for a trace that did not pass goes (default `.traces/check`) |
359
- | `--annotations` | `check`: print GitHub workflow annotations (on by default under `GITHUB_ACTIONS=true`) |
360
- | `--otlp-out <path>` | **WRITE** the OTLP artifact here (also evidence provenance / dry-run upload preview) |
361
- | `--format <kind>` | File `analyze`, `export`, or `stream`: `auto`, `policy-evidence`, `sandbox-events`, `openinference`, `intelligence-spans`, or `chat-trajectory` |
362
- | `--source-bundle <dir>` | `analyze` / `investigate` / `improve` / `ask`: read a retained full bundle and explicitly grant source-field reads |
363
- | `--llm` / `--budget <usd>` | Enable agentic analysts (needs `TANGLE_API_KEY` + Python with `agent-eval-rpc[dspy]`) / cap their spend |
364
- | `--question <text>` | `ask`: one question, repeatable. Kept short so the engine sees it whole |
365
- | `--questions <file>` | `ask`: JSON array of questions — strings, or `{ id?, question, instructions?, answerSchema? }` |
366
- | `--question-budget <usd>` | `ask`: provider ceiling for ONE question; `--budget` is the ceiling shared by all of them |
367
- | `--question-timeout <seconds>` | `ask`: wall-clock deadline for ONE question's investigation (default 600); raise it for questions over many large sessions |
368
- | `--concurrency <n>` | `ask`: questions running at once (default 4); `import-codetracebench`: trajectories imported at once |
369
- | `--config <path>` | `analyze` / `investigate` / `improve` / `stream`: load BYO analysts, live analysts, and external analyzers |
370
- | `--interval <s>` / `--window <m>` | `watch` / live `stream`: poll seconds (sessions 5, run tree 2) / active-session window minutes (default 30) |
371
- | `--min-loop <n>` | Identical repeated calls before flagging a loop (default 3) |
372
- | `--mode <kind>` | `stream`: `visualizer` (spans + findings), `findings` (low-volume), or `agent` (findings + reports) |
373
- | `--supervisor-run-dir <dir>` | `analyze`: report one run tree; `watch`: tail it live |
374
- | `--verify-findings` | `analyze`: execute the findings as sandbox replay proofs; each is marked VERIFIED (receipt path) or UNVERIFIABLE (reason). Needs `--replay-corpus` and a running sandbox (`SANDBOX_API_KEY` / `SANDBOX_API_URL`) |
375
- | `--replay-corpus name=<labels>::<prepared>` | Trajectory source for `--verify-findings` (repeatable) |
376
- | `--verify-out <dir>` | Receipt root for `--verify-findings` (default: `<--out>.verify`) |
377
- | `--replay` | `stream`: scan once, then exit |
378
- | `--once` | `stream`: scan once; `watch <target>`: print ONE snapshot and exit |
379
- | `--no-spans` / `--no-findings` | `stream`: suppress raw span rows / finding rows |
380
- | `--no-content` | `facts`: omit conversation, commands, paths and assignment text; `upload`: send metadata only |
381
- | `--dry-run` / `--yes` | `upload`: preview without sending / skip the confirm prompt |
382
-
383
- ## Live stream
384
-
385
- `traces watch` is the human terminal view.
386
- It prints repeated-tool loops and semantic findings while a coding agent is still running.
387
-
388
- `traces stream` is the machine feed.
389
- It emits newline-delimited JSON events that a dashboard, art visualizer, local watcher, or hosted product can consume without scraping terminal prose.
390
-
391
- ```bash
392
- traces stream --all
393
- traces stream --all --mode findings
394
- traces stream --all --mode agent --config traces.config.mjs
395
- traces stream spans.openinference.jsonl --format openinference --no-spans
396
- ```
397
-
398
- Live streaming emits `session`, `span`, `analysis_batch`, `finding`, and `tick` events; `--mode agent` also emits `report` events.
399
- The semantic findings currently cover repeated failing commands, verification churn without code/config changes, completion claims without later verification, and high tool-error rates.
400
- Use `--mode findings` when you want the low-volume meaning layer; keep `visualizer` for real-time views that need motion, timing, and tool-call texture.
401
- Use `--mode agent` when another agent needs deterministic loop/tool-use reports alongside the findings.
402
- The same JSONL shape is used for live sessions and replayed trace files:
403
-
404
- ![traces stream findings mode terminal capture](./docs/stream-findings.svg)
405
-
406
- Custom online analysts use the same config file as `investigate` and `improve`:
407
-
408
- ```js
409
- export default {
410
- liveAnalysts: [{
411
- id: 'my-live-rule',
412
- analyze(context) {
413
- return context.actions.some((action) => action.kind === 'claim') ? [{
414
- schemaVersion: 1,
415
- kind: 'traces.live_finding',
416
- id: `live.my-live-rule.${context.session.sessionId}`,
417
- ruleId: 'my-live-rule',
418
- fingerprint: `my-live-rule.${context.session.sessionId}`,
419
- severity: 'info',
420
- title: 'Completion claim seen',
421
- claim: 'A claim-like assistant message appeared in the live trace.',
422
- action: 'Require the next stream batch to include a verification action.',
423
- check: 'A later finding or batch should show a test/build/typecheck action.',
424
- evidence: [{ kind: 'metric', label: 'actions', value: String(context.actions.length) }],
425
- session: context.session,
426
- observedAt: context.generatedAt,
427
- }] : []
428
- },
429
- }],
430
- }
431
- ```
432
-
433
- ## Watch a run tree
434
-
435
- `traces watch <target>` tails ONE run that fans work out to other agents, instead of one harness session.
436
- It prints the tree at any depth: per node the status, the runtime, the authored budget, and the actual spend.
437
- Plain stdout, no cursor control, no extra dependency — it works over SSH, in a CI log, and through a pipe.
438
-
439
- ```bash
440
- traces watch runs/my-run # live: reprints only when something changed
441
- traces watch runs/my-run --once # one snapshot, for a script or an agent
442
- traces watch traces.otlp.jsonl # spans only, for any emitter
443
- ```
444
-
445
- Two sources sit behind the one verb, and the output always names which number came from which:
446
-
447
- | Source | Reads | Gives you | Cannot give you |
448
- |---|---|---|---|
449
- | **General** — OTLP spans | `trace_id` / `span_id` / `parent_span_id` in any `*.otlp.jsonl`, `*.spans.jsonl`, `*.traces.jsonl` | tree shape, wall time, per-turn tokens, cost when the emitter priced it | the authored budget — no span vocabulary carries one |
450
- | **Specific** — the spawn journal | `spawn-journal.jsonl` written by agent-runtime's `createFileRunContext` | authored budget, settled spend, settlement status, blocking questions, steers, the typed result | per-turn detail the journal does not tap |
451
-
452
- The general source is the one that matters for a system nobody has written yet: anything that emits a parent span pointer gets a tree, whether it is a supervisor, a pi session, a Claude subagent fan-out, or a custom fanout.
453
- Attributes are read through `@tangle-network/agent-eval/trace-attributes` alias lists, so both live vocabularies — OpenInference's `llm.token_count.*` and GenAI semconv's `gen_ai.usage.*` — already work, and no key is invented here.
454
-
455
- Two rules the view will not bend on:
456
-
457
- - **Driver inference and child work never merge.** A node that drove and the work it delegated are separate lines, because one total cannot answer "which half spent the budget" — the decisive question in three real runs.
458
- - **Unknown is never rendered as zero.** A turn the harness did not price reads `cost unknown (n unpriced)`, never `$0.0000`. A recorded `$0` paid on real tokens is printed AND flagged, because "free" and "unmetered" are different facts.
459
-
460
- ```
461
- run proof-harness-mcp-20260731i
462
- 2 node(s) · depth 1 · elapsed 9m59s
463
- driver total in 639,632 · out 35,077 · $0.0000 (zero across 58 priced)
464
- worker total in 47,965 · out 515 · cost unknown (1 unpriced)
465
-
466
- ● · root [pending] supervisor inline budget 12it/400,000tok/$2.00/15m00s
467
- driver 58 turn(s) · in 639,632 · out 35,077 · $0.0000 (zero across 58 priced) · 169% tok of budget
468
- children 1 settled · in 47,965 · out 515 · cost unknown (1 unpriced)
469
- └─ ✗ artifact-writer-s0 [failed] worker pi budget 6it/80,000tok wall 20.8s
470
- settled 3it · in 47,965 · out 515 · cost unknown (1 unpriced) · 50% it, 61% tok of budget
471
-
472
- ! spend recorded $0 with tokens spent: usdKnown was never set false, so this reads as FREE rather
473
- than unmetered. A harness-billed brain (a CLI on a subscription) looks exactly like this.
474
-
475
- RESULT no-winner · budget-exhausted
476
- ```
477
-
478
- `traces analyze --supervisor-run-dir <dir>` writes the full report for the same run.
479
- It also reads `failure.json` when Runtime fails before the first spawn.
480
- Pass a parent directory to combine reports from its Runtime runs.
481
- Eval owns discovery, normalization, and analysis; Traces adds recorded coordination requests and delivery receipts.
482
- Old Loops directories are unsupported.
483
-
484
- Replay and resume are a separate concern: they belong to the durable spawn journal, and nothing here routes them through telemetry.
485
-
486
- ## Improvement engine
487
-
488
- `traces improve` turns observed sessions into a portable improvement packet:
489
-
490
- ```bash
491
- traces improve --all --last 20 --dir .traces/improvement
492
- ```
493
-
494
- The directory contains:
495
-
496
- | File | Purpose |
497
- |---|---|
498
- | `result.json` | findings, actions, checks, execution facts, adoption data, and the chosen LLM analysis route |
499
- | `evidence.jsonl` | one row per evidence ref, suitable for downstream mining |
500
- | `report.md` | human-readable report rendered from the typed data |
501
- | `traces.otlp.jsonl` | canonical trace used by the analysts and execution accounting |
502
-
503
- Bring your own analysts with a config file:
504
-
505
- ```bash
506
- traces improve --last 5 --config examples/improvement-config.mjs --dir .traces/improvement
507
- ```
508
-
509
- The config can export:
510
-
511
- - `analysts`: deterministic or LLM analysts that implement the `agent-eval` `Analyst` contract
512
- - `liveAnalysts`: deterministic online analysts that implement the `TraceLiveAnalyst` contract for `traces stream`
513
- - `registry`: a prebuilt `AnalystRegistry`
514
- - `externalAnalyzers`: HALO or any command/model adapter that reads the OTLP artifact
515
-
516
- Traces does not pretend that an action is a measured candidate.
517
- Use `agent-eval` to propose and compare candidate changes, `agent-runtime` to package an approved improvement, and `agent-interface` to represent profile edits.
518
-
519
- ## Ask questions
520
-
521
- `traces ask` answers free-form questions about the selected sessions.
522
- Each question becomes its own recursive investigation over the same trace, and the questions run at the same time under one budget.
523
-
524
- ```bash
525
- traces ask --harness codex --session <id> \
526
- --question "Which shell commands exited non-zero, and what were they?" \
527
- --question "What was the last thing the human asked for?" \
528
- --dir .traces/ask
529
- ```
530
-
531
- Ask many questions from a file, and hold an answer to a shape a scorer can compare:
532
-
533
- ```bash
534
- traces ask --all --last 3 --questions questions.json --concurrency 6 --budget 2 --dir .traces/ask
535
- ```
536
-
537
- ```json
538
- [
539
- "Which pull requests did this session open?",
540
- {
541
- "id": "merged-prs",
542
- "question": "Which pull requests were merged?",
543
- "instructions": "Count only merges the trace records, not merges the agent said it would do.",
544
- "answerSchema": { "type": "array", "items": { "type": "integer" } }
545
- }
546
- ]
547
- ```
548
-
549
- An entry is a question string, or an object with `question` and an optional `id`, `instructions`, and `answerSchema`.
550
- The schema accepts a small JSON Schema subset: `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, and `const`.
551
- Any other keyword is rejected when the run starts, because a constraint that is silently ignored would let a wrong answer pass as checked.
552
-
553
- The command writes two artifacts to `--dir`:
554
-
555
- | File | Contents |
556
- |---|---|
557
- | `answers.json` | per question: the answer text, the parsed answer, every citation with whether it resolves, accepted findings, evidence-gate rejections by reason, model calls, tool calls, cost with provenance, and latency |
558
- | `report.md` | the same run as readable Markdown: a summary line, a per-question table, then each answer with its citation and rejection notes |
559
-
560
- What it checks, and what it costs:
561
-
562
- - **Facts first.** Every question receives the deterministic [session-facts sheet](#session-facts) as prepared context before its first model call, at $0. The sheet is not citable; it names the span ids behind each fact.
563
- - **Citations.** Every `trace://<trace_id>/span/<span_id>` URI in an answer is looked up in the trace. An answer that cites a span the trace does not hold fails.
564
- - **Budget.** `--budget` is one ceiling shared by every question. `--question-budget` bounds one question. The run refuses to start when the budget cannot cover a single model call, and warns when the budget admits fewer concurrent calls than `--concurrency`.
565
- - **Cost.** Each cost carries its provenance: `observed` from a provider receipt, `estimated` from token counts, or `uncaptured`. An uncaptured cost stays null; it never becomes zero.
566
- - **Exit code.** `ask` writes both artifacts first, then exits 1 when any question failed, returned no answer, broke its schema, or cited a span that does not exist. A failed question never costs the other answers, and neither does Ctrl-C: the answers already bought are recorded, the questions the run never reached are recorded as `aborted`, and both artifacts are still written.
567
- - **Wall time.** `totals.wallTimeMs` covers the whole run, including writing and indexing the trace file; `totals.setupTimeMs` says how much of it that setup was. `totals.peakConcurrency` is how many questions actually overlapped, which is at most `min(--concurrency, questions)`.
568
-
569
- `ask` uses the same engine and credentials as `--llm`, so it needs `TANGLE_API_KEY` and a Python interpreter with `agent-eval-rpc[dspy]`.
570
- Do not pass `--llm`; the command is model-backed by definition.
571
-
572
- ## Session facts
573
-
574
- The SDK also accepts verified retained captures through
575
- `parseRetainedSession({ harness, nativeSessionId, files, signal? })`.
576
- Pass files as `{ path, content }` with their original relative layout.
577
- Runtime or the caller owns custody, digest verification and capture completeness.
578
- The reader uses existing adapters and selects a file only when native records
579
- establish the supplied session identity.
580
- It does not inspect the host session catalog or infer identity from filenames.
581
- Directory-only stores and formats without a recorded native identity remain unavailable.
582
- The returned source files, duplicate snapshots and unselected files explain the selection.
583
- Identical or append-only copies count once; divergent histories remain unavailable.
584
- Parsing is bounded to 1024 text files, 16 MiB per file, and 64 MiB total.
585
- No model calls occur.
586
- Call `buildSessionFactsReport(selection.spans, { includeContent: false })` to
587
- withhold native bodies from the resulting facts.
588
- The reader itself returns spans containing native content, for local trusted consumers only.
589
-
590
-
591
- The same facts sheet includes observed tool outcomes, captured input/output byte
592
- volumes, call-to-result intervals, compaction boundaries and available context
593
- sizes, spawn intervals, and existing skill invocation/document-read evidence.
594
- Each measurement carries coverage; absent telemetry remains unknown.
595
- `facts --no-content` withholds conversation (including each turn's `answered` text),
596
- command, changed-file, working-directory and assignment text while retaining
597
- measurement metadata. This is a local metadata view, not a
598
- share-safety certificate.
599
-
600
- SDK consumers use `buildSessionFactsReport(spans, { includeContent: false })`.
601
- Pass `window: { since, until }` (ISO bounds) for the windowed report `facts --since/--until` prints, and use
602
- `createSessionFactsReportBuilder(options)` (`add(spans)`, `addUnread(...)`, `build()`) to build it one session at a time.
603
- Cache by `measurementVersion` and `sourceDigest`; `evidenceThrough` is the latest
604
- recorded endpoint, not current liveness. Computing the sheet parses the supplied
605
- spans; returning a stored sheet needs no model call or trace scan. Semantic
606
- judgments remain attributed analyst results.
607
-
608
-
609
- `traces facts` prints the deterministic session-facts sheet: the answers a session audit needs first, computed straight from the spans.
610
- No model call, no engine, no budget — it costs $0 and always returns the same sheet for the same spans.
611
-
612
- ```bash
613
- traces facts --harness codex --session <id> # JSON on stdout
614
- traces facts --harness codex --last 5 --format text # the short readable form
615
- traces facts --otlp spans.otlp.jsonl --out facts.json
616
- traces facts --harness claude-code --since 7d # every session with a record in the last week
617
- traces facts --all --since 2026-10-14 --until 2026-10-15 # one day, across every harness
618
- ```
619
-
620
- `--since <t>` and `--until <t>` open a half-open time window `[since, until)`; each takes `30m`/`2h`/`7d` before now or an ISO date, and either may be left open.
621
- With a window, `facts` selects **every** session with a record that starts or ends inside it, not only the newest (`--last <n>` still caps it to the newest `n` per harness, and `--session` or `--otlp` still name the input).
622
- Sessions are read one at a time, and a session file last modified before `since` is not read, because it cannot hold a later record.
623
- Within a selected session, `humanTurns`, `excludedTurns` and `turnsByActor` keep only turns that started inside the window, so a one-day run counts only that day's turns; every other fact covers the whole session.
624
- The report's `selection` states the window, both rules, the sessions read whose records all fell outside it (`outsideWindow`), and every session that could not be read (`unreadSessions`, each with its reason).
625
- A report built without a window has `selection: null`.
626
-
627
- | Fact | What it is |
628
- |---|---|
629
- | `toolCalls` | TOOL spans **this session's agent** actually invoked. Synthesized lifecycle spans are excluded and counted in `synthesizedToolSpans`, so the total is not high by the number of subagents; calls a spawned subagent made are excluded and counted in `subagentToolSpans` |
630
- | `toolCallsByName` | the same calls by tool name, so a category decision is the reader's, not a guess |
631
- | `subagentToolSpans` | TOOL spans excluded because a subagent, not this agent, made the call. Zero on a trace that carries no subagent records |
632
- | `subagents` | every subagent-spawning call with the task name the adapter recorded |
633
- | `pullRequests` | the pull requests the commands created and merged, each named by number or head branch, with the command span and how the identity was joined. Scanned the way a shell reads the script, so a `gh pr create` inside a heredoc body is not a command that ran |
634
- | `cwd` | the working directory the harness recorded for the session (`traces.session.cwd` on the root span): Claude Code's first record `cwd`, Codex's `session_meta` cwd, Pi's session header, OpenCode's session directory (the older JSON store: its messages' `path.cwd`), Factory's `session_start`. Never decoded from a directory name or inferred from paths; `null` with the reason when the source names none or its adapter does not read one |
635
- | `humanTurns` | `user.prompt` turns a person typed into this session, in order, with the timestamp. Inherited fork or compaction history, harness-injected blocks, and a second record of the same turn are excluded — each listed in `excludedTurns` with its reason and span ids, never silently dropped. `turnsByActor` shows every turn by actor so the filter is checkable. Each turn's `answered` is the last message this session's own agent wrote before it (not a subagent's, not inherited history), clipped to 1,500 characters with a `… [+N chars]` marker, with its span in `answeredSpanId`; `null` when the agent had written no text yet in the spans read (a `--latest-turn` read starts at the latest task, so its first turn has none) |
636
- | `finalMessages` | the last message of the session's own agent, and of each subagent task, kept apart |
637
- | `changedFiles` | paths the harness recorded for the changes it applied, plus paths named by patch headers and file-editing tool arguments, with the operation |
638
- | `firstRecordAt` / `lastRecordAt` | the earliest span start and latest span end among this session's own records — a subagent that outlives the session does not stretch its window |
639
- | `unreadRecords` | records the session reader could not parse, from the session's integrity receipt |
640
- | `tokenTotal` | the harness's own cumulative token total, when a span carries `traces.session.total_tokens` |
641
- | `firstFailure` | the first failure among this session's own records, ranked by agent-eval's fixed precedence: the innermost ERROR span that ended first, else a span whose `agent.outcome` is `fail`, else `none` — spans were observed and none of them failed. When every span is UNSET (a capture that failed before any status was written), or a session has no records of its own at all (every span is `traces.span.subagent` — spawned work folded into this trace, with nothing this agent recorded itself), the status is `unknown`, not `none`: the record does not say whether the run failed. Failures that ended at the same instant are `ambiguous` and list the candidates instead of picking one. `blame` separates machine and provider failures from the agent's own, and is `unknown` rather than `agent` when no evidence names the agent |
642
-
643
- Three rules hold for every field:
644
-
645
- - **Every fact is about this session's own agent.** Some harnesses fold a spawned agent's transcript into the parent's trace, so a trace can carry both what this agent did and what a child it spawned did. The child's work is named in `subagents` and counted in `subagentToolSpans`; it is never added to the parent's totals.
646
- - **Every fact names its span ids.** `spanIds` lists the spans the value was computed from, so any number here can be opened and checked. The sheet itself is not a span and cannot be cited.
647
- - **A fact the spans cannot support is `null` with its reason.** It is never guessed, and never a silent zero. `partial` marks a measured value that is known to be incomplete — a truncated patch, or a list above the entry cap.
648
-
649
- Without a window, `facts` exits non-zero when a selected session cannot be read at all: a session that produced no record spans would otherwise print a sheet of zeros stating, in the sheet's own voice, that the session did nothing.
650
- With a window, a session that cannot be read — the parser failed, or no record could be read — is named in `selection.unreadSessions` with its reason and a warning on stderr, and the run still exits 0 with the sessions it could read; it is never counted as a session with no turns.
651
-
652
- The same sheet reaches the model-backed analysts as prepared context, before their first model call — see [Trace analysts](docs/trace-analysts.md#session-facts-as-prepared-context).
653
-
654
- ## Skill usage
655
-
656
- `traces skills` indexes skill records in Claude Code, Codex, and Kimi Code sessions on this machine.
657
- It makes no model call.
658
-
659
- ```bash
660
- traces skills # every stored event, with the installed catalog joined
661
- traces skills --since 30d # one window
662
- traces skills --unused # installed skills with no use in the window, one per line
663
- traces skills --format json # rows, unused skills, refresh cost and sweep exclusions
664
- traces skills --no-refresh # answer from the store without reading transcripts
665
- ```
666
-
667
- A run reads only transcript bytes appended since the previous run.
668
- It parses only lines that can carry a skill record and stores the events in `~/.local/state/traces/skill-usage.json` (`--store`).
669
- The first run reads the whole corpus: about two minutes for 41 GB on an M-series Mac.
670
- Later runs take seconds, mostly to stat every session file.
671
- Stored events survive a harness deleting old transcripts.
672
-
673
- | Column | Harness | Counted record |
674
- | --- | --- | --- |
675
- | `model` | Claude Code | A Skill tool result that loaded a skill. A failed call does not count. |
676
- | `slash` | Claude Code | A `/name` command whose next record is its expanded prompt. Built-in commands such as `/clear` do not expand and do not count. |
677
- | `read` | Codex | A completed command that reads `<skill>/SKILL.md`. Codex records no skill event, so a read is the closest observable act. It does not show that the skill shaped the outcome. |
678
- | `read` | Kimi Code | A successful `ReadFile` call or shell command that reads `<skill>/SKILL.md`, including streamed arguments. Kimi records no dedicated skill event. |
679
-
680
- Each event is keyed by its record ID, so a resumed session that copies earlier records counts once.
681
- One Codex or Kimi command that reads three or more skills (`--sweep`) is a catalog sweep, such as an audit; its reads are excluded and reported separately.
682
- `installed` names the harnesses whose catalog lists the skill now: `~/.claude/skills` and enabled plugins for Claude Code, and `~/.codex/skills` and `~/.agents/skills` for Codex.
683
- OpenCode, Pi, Gemini, Factory, and other harnesses are not indexed.
684
-
685
- ## Diff two runs
686
-
687
- `traces diff` pairs the steps of two runs of the same task and marks where they first stop agreeing.
688
-
689
- ```bash
690
- traces diff runs/a/otlp runs/b/otlp # two OTLP exports
691
- traces diff a.sdk-events.jsonl b.sdk-events.jsonl --kind TOOL # any file convert reads; tool calls only
692
- traces diff spans.otlp.jsonl#<trace id> other.jsonl --format json
693
- traces diff run.otlp.jsonl#branch=arm-a run.otlp.jsonl#branch=arm-b # two arms of one run
694
- ```
695
-
696
- A side is a file or directory that holds one run, `file#<trace id>` for one run of several, or `file#branch=<id>` for one arm: the spans whose `agent.branch.id` or `agent.branch.arm` is `<id>` and have no matched ancestor (a root), and every span below them. Sibling arms can share a branch id or label; when `<id>` names more than one such root subtree, the command refuses (exit 2) and lists each root's span id, rather than silently comparing the union of both.
697
- Steps pair by span id first, then by position with the same name and kind, then by name and kind anywhere.
698
- One inserted step therefore does not mark every later step as changed.
699
- Paired steps compare status, tool and model.
700
- The first divergence is `changed`, `replaced`, `only-in-a`, `only-in-b` or `reordered`, with its reason, followed by every changed and unpaired step.
701
- `--kind` keeps only steps of that span kind, so event noise in a raw stream does not decide the first divergence.
702
- The diff is agent-eval's `diffSteps`; this command reads the runs and prints it.
703
- It exits 0 when the runs agree on outcome, 1 when they diverge, 2 when a side cannot be read (including an ambiguous `#branch=` selector), and 3 when the steps line up with no divergence but there is no recorded outcome to call agreement: neither side has a step with a definitive OK/ERROR status (both are entirely UNSET, the common shape for a capture that failed before any status was written), or neither side has a TOOL step. That case's report and text output carry a `caveat` naming which reason applies; a CI or agent gate should treat exit 3 as unknown, not as a pass.
704
-
705
- ## MCP server
706
-
707
- `traces mcp` serves the selected traces to an MCP client over stdio.
708
-
709
- ```bash
710
- claude mcp add traces -- traces mcp --otlp ./spans.otlp.jsonl
711
- claude mcp add traces -- traces mcp --harness claude-code --last 5
712
- ```
713
-
714
- It serves agent-eval's seven trace tools: `getDatasetOverview`, `queryTraces`, `countTraces`, `viewTrace`, `viewSpans`, `searchTrace` and `searchSpan`.
715
- Each tool declares `readOnlyHint` and `idempotentHint` in its MCP annotations, and its description says that returned trace text is untrusted.
716
-
717
- - The spans are redacted before the store is built — every span's name, attributes, status message and causal links' attributes, every field the store reads — and each result is redacted again at the boundary. The server then reassesses the redacted spans and refuses to start when they are still UNSAFE or UNKNOWN, the same gate `upload` applies to what it sends.
718
- - Each result carries an `untrusted` notice beside it.
719
- - A result above 512 KiB is refused with an error rather than truncated.
720
- - `readSpanSource` is not served. It reads original source bytes in windows the caller picks, and a secret split across two windows matches no redaction rule.
721
-
722
- ## Session index
723
-
724
- `traces index` writes one general JSON catalog over the selected sessions.
725
- It is meant for deeper investigation and joins with other local data, not for one specific workflow.
726
-
727
- ```bash
728
- traces index --all --since 24h --out session-index.json
729
- traces inspect session-index.json --out inspection-report.md
730
- ```
731
-
732
- The index contains:
733
-
734
- - selection metadata and aggregate totals
735
- - one row per session with harness, session id, path, cwd, repo labels, and time bounds
736
- - behavior metrics: spans, LLM turns, tool calls, tool errors, tokens, models, and tools
737
- - signal summaries: stuck loops and tool error rate
738
- - nearby context files for joins: `AGENTS.md`, `CLAUDE.md`, and `.evolve` artifacts — the JSONL ledgers, `scorecard.json` / `current.json`, reflections, `handoffs/` plus the flat `handoff-*.md` convention, and `progress.md` — with markdown heading/ToC and JSONL key summaries
739
-
740
- `traces inspect` reads that index back and prints ranked improvement findings over the sessions and nearby context.
741
- It is intentionally read-only: it points to repeated-call loops, high tool-error sessions, missing repo attribution, long docs without Contents, invalid JSONL rows, and skill-run rows that cannot be joined back to a session.
742
-
743
- ## Session bundle
744
-
745
- `traces bundle` assembles ONE session's durable evidence directory — the input for any downstream consumer that must cite the session after the live stores rotate.
746
- It composes commands this CLI already owns and never spends a model call.
747
-
748
- ```bash
749
- traces bundle --harness claude-code --session <id|path> --out bundle-dir
750
- ```
751
-
752
- The bundle directory:
753
-
754
- - `session/` — the transcript byte-for-byte, plus the sibling subagents directory when one exists
755
- - `derived/` — `session-index.json`, the deterministic `report.md`, `evidence.jsonl`, and the OTLP span artifact
756
- - `ledger/` — the repo's `.evolve` slices for the session window: `experiments.jsonl` and `skill-runs.jsonl` rows inside the padded span window, `current.json` / `scorecard.json` / `progress.md` copied whole, the latest flat `handoff-*.md`, and reflections dated inside the window
757
- - `repo/git-log.txt` — commits in the session window from the session's cwd
758
- - `manifest.json` — `view`, SHA-256 + byte count per file, provenance (session id, harness, cwd, transcript hash, session window), every absent artifact with its reason, per-slice row counts, and the bundle's known limits
759
-
760
- A missing transcript fails the assembly loudly.
761
- Every optional input that is absent (no subagents, no `.evolve`, no git repo) is recorded in `manifest.absent` with the probed path — a fact, not an error.
762
-
763
- ### Two views, two consumers
764
-
765
- A bundle has two kinds of reader, and they need opposite things.
766
-
767
- - An **auditor** cites the session and needs every source byte. That is the FULL view — what `traces bundle` writes, `manifest.view: "full"`.
768
- - A **writer** must reach its own conclusion about the session, so it must not be able to read the conclusion that session already reached. That is the EVIDENCE-ONLY view, `manifest.view: "evidence-only"`.
769
-
770
- The full bundle cannot serve the writer. Its transcript holds every word the session wrote, including the shell call that authored an earlier report, so a check that looks for the report FILE passes while the report CONTENT is right there. Projecting is the fix, not grepping:
771
-
772
- ```bash
773
- traces bundle --harness claude-code --session <id|path> --out bundle-dir
774
- traces bundle-view bundle-dir --view evidence-only --out writer-dir
775
- ```
776
-
777
- The evidence-only view carries an explicit allow-list — `derived/session-index.json`, `derived/evidence.jsonl`, and the structured `ledger/` records (`current.json`, `scorecard.json`, `skill-runs.jsonl`). Everything else is excluded **by name** in `manifest.excluded`, with its SHA-256 and one of four rules:
778
-
779
- | rule | what it covers |
780
- |---|---|
781
- | `session-source` | `session/` — the transcript and every subagent transcript |
782
- | `session-text-derived` | `derived/report.md` (quotes the first prompt line and evidence excerpts) and `derived/trace.otlp.jsonl` (spans carry the full prompt and response text) |
783
- | `authored-prose` | `ledger/progress.md`, `ledger/handoff-*.md`, `ledger/reflections/`, `ledger/experiments.jsonl` free-text, `repo/git-log.txt` |
784
- | `not-allow-listed` | any other path, including one a later version of `traces` adds — the view can only narrow by accident, never widen |
785
-
786
- The allow-list is the structure. A content check is the proof it is right: before anything is written, every candidate is compared against every excluded file for shared 8-word runs of prose, with identifiers (paths, URLs, hashes) and this package's own constant strings scrubbed from both sides first. A shared run in a repo ledger file drops that file with `rule: "content-signature"` and the match count; a shared run in an artifact `traces` derives is a defect in the derivation and no view is written at all. `manifest.projection.leakCheck` records the width, the sources compared, and the result — always `matches: 0`, because a match means no view exists.
787
-
788
- A projected view also carries the full bundle's `manifest.json` hash, so an auditor can prove which record it came from. Projecting a view of a view is refused.
789
-
790
- ### Verify a bundle
791
-
792
- ```bash
793
- traces bundle verify bundle-dir # VERIFIED, or FAILED with each difference; exit 1 on failure
794
- traces bundle verify writer-dir --format json
795
- ```
796
-
797
- `bundle verify` checks every file the manifest lists for its recorded size and SHA-256.
798
- It reports a listed file that is missing, a file on disk that the manifest does not list, a manifest path that is absolute or climbs out of the directory, and any symbolic link.
799
- For a projected view it also checks that the full bundle it names still has the manifest bytes the view recorded; a moved or deleted source is `unavailable`, not a failure.
800
- It proves the bytes are the bytes `traces` wrote. It does not prove who wrote them: anyone who can rewrite a file can rewrite the manifest.
801
-
802
- ## Policy-mining evidence
803
-
804
- `traces` does **not** emit benchmark campaign cells. It emits normalized coding-agent session evidence that another system can mine.
805
-
806
- ```bash
807
- traces evidence --all --since 24h --out policy-evidence.jsonl --otlp-out spans.otlp.jsonl
808
- ```
809
-
810
- `--last` follows recent file activity and may select a child session in multi-agent work.
811
- Use `--harness codex --current --latest-turn --workflow` for the invoking Codex turn plus its connected workers.
812
- Omit `--latest-turn` when the full history of a resumed session is the intended unit.
813
- Workflow expansion follows session IDs, never timestamps or worker names.
814
- For a latest Codex child, it selects the last parent task that structurally spawned or targeted that child ID, then includes only workers referenced by that parent task.
815
- This requires the parent log to record the child ID on a spawn, send, follow-up, or lifecycle event and to associate that event with a stable turn ID.
816
- If either field is absent, the workflow is reported incomplete instead of widening to the parent's full history.
817
- Missing files, duplicate IDs, contradictory parents, and cycles are reported explicitly.
818
- Claude Code subagent files are already folded into their parent trace, so `--workflow` normally resolves one Claude session file.
819
- With `--latest-turn`, ordinary Claude subagents are selected by their parent tool call ID.
820
- Claude Workflow subagents are selected by the run ID and transcript directory returned by the parent `Workflow` call.
821
- Each Workflow child attaches to the latest matching call that started before the child.
822
- Task selection then keeps only children attached to calls in that task.
823
- Returned directories from another resumed Claude session are included in parsing and source hashes.
824
- For a copied Claude session, preserve `<session UUID>/subagents/workflows/<run ID>` beside the parent JSONL file.
825
- Recorded native `.claude/projects/<project>/<session UUID>/subagents/workflows/<run ID>` paths resolve to that copied subtree when outside the selected store.
826
- Copy every referenced Workflow directory, including runs stored under another resumed session UUID.
827
- The original transcript stays unchanged, and child attribution uses its recorded Workflow identity.
828
- Missing copied directories, conflicting source bindings, and symlinks that change the copied session or run identity stop parsing.
829
- Copied Workflow directories contain regular transcript and metadata files; symlink entries stop parsing.
830
- Missing or conflicting Workflow identities stop the parse instead of widening the selected history.
831
- Cross-harness parent/child trees are not inferred.
832
- For a reproducible report, run `traces list` first and pass its session ID with `--session`.
833
- Ephemeral `codex exec --json` output has no discovery location, so select its file with `--harness codex-exec --session <path>`.
834
- The adapter rejects empty, incomplete, or other JSONL formats instead of emitting zero-valued session evidence.
835
-
836
- Each JSONL row is one session:
837
-
838
- - session provenance: harness, session id, cwd, path, mtime
839
- - explicit-session source binding: `provenance.sourceSha256` over every adapter-declared input file, including Claude worker and metadata files
840
- - repo labels: `tangle.subject.key`, `git.repository`, branch, commit
841
- - behavior metrics: span counts, LLM turns, tool calls, errored tool calls, tokens, models, tool histogram
842
- - mining signals: stuck loops and tool error rate
843
- - provenance marker: `notCampaignCell: true`
844
-
845
- That boundary matters. `agent-lab` campaign `cells.jsonl` says "arm X beat arm Y on task Z." `traces evidence` says "this real agent session had this repo/model/tool/failure shape." A downstream policy compiler can cluster these rows, propose candidate policies, then validate those policies in a separate eval campaign.
846
-
847
- ### Export existing evidence/events to OpenInference
848
-
849
- If you already have compact evidence or event captures on disk, convert them to the same OpenInference JSONL shape that `traces analyze --analyzer halo` uses:
850
-
851
- ```bash
852
- traces export policy-evidence.jsonl --out spans.openinference.jsonl
853
- traces export sandbox-events.json --format sandbox-events --out spans.openinference.jsonl
854
- traces export trajectory.json --format chat-trajectory --out spans.openinference.jsonl
855
- halo spans.openinference.jsonl --prompt "Analyze this trace slice" --max-turns 1
856
- ```
857
-
858
- `traces export` accepts:
859
-
860
- - compact `traces.policy_evidence.session` JSONL from `traces evidence`
861
- - JSON arrays with `start`, `raw`, `result`, `done`, and `error` events. A tool call becomes one span, keyed by its message part id, however many events report on that part. The Sandbox SDK records each call as a `message.part.updated` event and again as the harness's raw `tool_use` event. The span takes its start from the part's first event, its final state from the last, and `ERROR` if any event reported one. The root span's `traces.event.tool_part_updates_merged` counts the events folded in.
862
- - existing OpenInference JSONL, rewritten through the local redaction path
863
- - chat message arrays and objects with a `messages` array
864
-
865
- Run `traces export --help` for the full command reference.
866
-
867
- ## Upload to the Intelligence Platform
868
-
869
- `upload` **redacts locally before anything leaves the machine**, dedups against already-uploaded sessions, and tags each with metadata (harness, cwd, git branch, host).
870
- It then checks the final events of each session, exactly as they would be sent, with the share-safety verdict.
871
- A session that is UNSAFE or UNKNOWN is refused: it is not sent, not written to the dry-run preview, and not marked uploaded, and the command exits 1.
872
-
873
- ```bash
874
- traces upload --since 24h --dry-run # see exactly what would be sent; no network
875
- traces upload --since 24h --no-content # send metadata only; drop all prompt/response text
876
- traces upload --since 24h # send it
877
- ```
878
-
879
- It needs `TANGLE_INGEST_URL` (or `TANGLE_ORCHESTRATOR_URL`), `TANGLE_INGEST_API_KEY` (or `TANGLE_API_KEY`), and `TANGLE_TENANT_ID`. Without them, `--dry-run` still works fully.
880
-
881
- ### Redaction scope: read this before uploading prose
882
-
883
- Redaction is agent-eval's redaction core ([docs/redaction.md](https://github.com/tangle-network/agent-eval/blob/main/docs/redaction.md)), `default` profile.
884
- It finds credentials by field name (`apiKey`, `x-api-key`, `client.secret`, `refreshToken`) and by value shape (provider keys, GitHub, Slack, AWS and Google tokens, JWTs, bearer headers, private keys, `KEY=secret` assignments, credentials in URLs), and replaces the whole string.
885
- It replaces email, card, SSN and phone values in place.
886
- Token counts such as `input_tokens` and `max_tokens` are kept.
887
- It runs over every span attribute, including the captured prompt/response text.
888
-
889
- It does **not** catch names, postal addresses or account numbers written in prose. Those need a context-aware model. Three postures, strongest first:
890
-
891
- 1. **`--no-content`**: upload metadata only (tool calls, tokens, timing, loop signal); no prose leaves the machine.
892
- 2. **`--redactor <cmd>`**: run an ML PII scrubber (e.g. [`openai/privacy-filter`](https://github.com/openai/privacy-filter)) after the redaction core.
893
- 3. Default: the redaction core.
894
-
895
- Always `--dry-run` first to see exactly what would be sent.
896
-
897
- ### Check before sharing
898
-
899
- `verify-safe` reads files and says whether they are safe to share. It never changes them.
900
-
901
- ```bash
902
- traces verify-safe bundle-dir # share profile (default)
903
- traces verify-safe session.jsonl --profile strict --format json
904
- traces verify-safe out/ --known-secret-env TANGLE_API_KEY # also look for this exact key, in any encoding
905
- ```
906
-
907
- | Status | Meaning | Exit |
908
- |---|---|---|
909
- | `SAFE` | Nothing found | 0 |
910
- | `SAFE_WITH_WARNINGS` | Only warnings: raw content, identifiers, embedded media | 0 |
911
- | `UNSAFE` | A credential remains, or personal data under `share` or `strict` | 1 |
912
- | `UNKNOWN` | A file, line or byte range could not be read, so its safety is unknown | 2 |
913
-
914
- `.json` files are parsed whole and `.jsonl` or `.ndjson` files line by line. Any other file is scanned as text.
915
- Each finding names the file, line and JSON Pointer, never the matched value.
916
-
917
- ## Trace analysts
918
-
919
- Use the local checks by default.
920
- Add a model-backed engine only when the question needs semantic judgment or behavior discovery.
921
-
922
- ```bash
923
- traces analyze --last 1
924
- traces analyze --last 1 --llm --budget 0.50
925
- traces analyze --last 1 --analyzer halo --analyzer-prompt "find token waste"
926
- traces analyze --all --last 20 --analyzer hodoscope
927
- traces analyze --last 1 --analyzer prime
928
- traces analyze --last 1 --analyzer my-installed-command
929
- ```
930
-
931
- To ask your own question instead of the built-in kinds, use [`traces ask`](#ask-questions).
932
-
933
- HALO returns a diagnosis report.
934
- Hodoscope samples distinct behaviors and marks every sample `needs_review`.
935
- Prime posts the full span projection to an OpenAI-compatible bridge (`TRACES_PRIME_BRIDGE_URL`, default `http://localhost:4181`) and returns validated findings with span evidence.
936
- An arbitrary command returns a raw report unless its SDK adapter explicitly parses a stricter output type.
937
-
938
- Read [Trace analysts](./docs/trace-analysts.md) for the output contract, a minimal custom analyst, and labeled benchmark setup.
939
-
940
- **Redactors** scrub prompt/response prose with an external PII model (catching names/addresses the regex pass can't), running *after* the built-in redaction:
941
-
942
- ```bash
943
- # the command reads a JSON array of strings on stdin, writes the scrubbed array on stdout
944
- traces upload --since 24h --dry-run --redactor "my-pii-scrubber"
945
- ```
946
-
947
- In the SDK these are the `ExternalAnalyzer` and `Redactor` interfaces (`haloAnalyzer`, `hodoscopeAnalyzer`, `primeAnalyzer`, `commandAnalyzer`, `commandRedactor`, `applyRedactor`, `runExternalAnalyzers`).
948
- See [`examples/external-engines.ts`](./examples/external-engines.ts).
949
-
950
- > The built-in agentic analysts (`--llm`) run on the Tangle router by default: set `TANGLE_API_KEY`.
951
- > `OPENAI_API_KEY` alone targets OpenAI instead, and `OPENAI_BASE_URL` overrides the endpoint for any other OpenAI-compatible gateway.
952
- > `--llm` also needs a Python interpreter with `agent-eval-rpc[dspy]` installed, because agent-eval's model-backed analysts run through the DSPy RLM engine out of process; set `TRACES_PYTHON` to choose the interpreter.
953
- > The bridge protocol is version-locked: install the exact version matching this package's `@tangle-network/agent-eval` dependency (`pip install "agent-eval-rpc[dspy]==$(npm view @tangle-network/traces dependencies.@tangle-network/agent-eval)"`) — a skewed bridge kills every agentic analyst at startup.
954
- > When `--llm` was requested and every agentic analyst fails, `analyze`/`investigate`/`improve` still write the deterministic report, then exit 1 with each analyst's underlying error.
955
- > A requested `--analyzer` behaves the same way: its error is written into the report, and `analyze` then exits 1 naming every external analyzer that failed.
956
- > Every deterministic command — `list`, `analyze` without `--llm`, `convert`, `index`, `inspect`, `export`, `evidence`, `stream`, `watch`, `analyze --supervisor-run-dir` — needs neither a key nor Python.
957
-
958
- ## Agent skills
959
-
960
- The npm package ships two AgentProfile-ready skills:
961
-
962
- | Skill | Use |
963
- |---|---|
964
- | [`inspect-agent-traces`](./skills/inspect-agent-traces/SKILL.md) | select workflows and export deterministic findings |
965
- | [`build-trace-analyst`](./skills/build-trace-analyst/SKILL.md) | write and calibrate a custom analyst |
966
-
967
- Load both from GitHub:
968
-
969
- ```ts
970
- import { defineGitHubResource } from '@tangle-network/agent-interface'
971
-
972
- const ref = process.env.TRACES_SKILLS_REF
973
- if (!ref) throw new Error('TRACES_SKILLS_REF must be an immutable commit')
974
-
975
- const names = ['inspect-agent-traces', 'build-trace-analyst']
976
- const skills = names.map((name) => defineGitHubResource(`skills/${name}/SKILL.md`, {
977
- repository: 'tangle-network/traces',
978
- ref,
979
- name,
980
- }))
981
- const profile = { name: 'trace-reviewer', resources: { skills } }
982
- ```
983
-
984
- ## Release automation
985
-
986
- Releases use a reviewed version commit because `main` is protected:
987
-
988
- 1. Update `package.json` and `pnpm-lock.yaml` to the intended version on a branch.
989
- 2. Merge that change through a pull request.
990
- 3. Tag the exact merge commit as `vX.Y.Z` and push the tag.
991
- 4. The Publish workflow verifies that the tag is merged into `main` and matches `package.json`, then builds, publishes to npm, and creates a GitHub release.
992
- 5. `pnpm check:package` proves the npm tarball contains the `traces` binary before release.
993
-
994
- To retry a failed publish without changing source or version, manually run the Publish workflow with the existing tag.
995
-
996
- ## Library (SDK)
997
-
998
- The CLI is a thin consumer of these exports.
999
-
1000
- | Export | Signature | Use |
1001
- |---|---|---|
1002
- | `readOtlpInput` | `(file\|dir) → { files, skipped, rows, spans, validation, issues }` | ingest OTLP from **any** system — no adapter; never throws on a foreign trace |
1003
- | `validateTraceSpans` | `(spans) → { ok, findings, capabilities }` | the contract's conformance check, re-exported |
1004
- | `conformanceOfSpans` | `(spans) → TraceValidation` | grade spans this package produced, on the artifact a consumer receives |
1005
- | `renderValidation` / `renderConformance` | `(validation, opts) → string` | the `validate` report / the section embedded in an analysis report |
1006
- | `skippedAnalyses` | `(validation) → SkippedAnalysis[]` | which analyses a missing capability empties, and why |
1007
- | `unavailableCapabilities` / `incompleteInputsNote` | `(validation) → Map` / `(map, name) → string` | the gate: what the trace cannot support, and the marker each affected section carries |
1008
- | `summarizeSpanStructure` | `(spans) → SpanStructure` | how much of the tree attaches: rootless, orphans, links |
1009
- | `analyzeLoopConvergence` | `(spans) → LoopConvergenceReport` | per loop, the per-round outcome/score series and whether it improved, plateaued, regressed or fell back |
1010
- | `analyzeSteeringChain` | `(spans) → SteeringChainReport` | which verdict span caused which subsequent round, following `links` |
1011
- | `llmSpan` / `toolSpan` / `loopSpan` / `branchSpan` / `steeredBy` | contract builders, re-exported | emit conforming spans from your own system |
1012
- | `analyzeSpans` | `(spans, { registry?, ai?, budgetUsd? }) → AnalyzeResult` | run built-in analysts, or **your own** via `registry` |
1013
- | `runTraceInvestigation` | `(TraceInvestigationOptions) → TraceInvestigationResult` | typed findings with actions/checks, execution facts, external analyzer output, and report |
1014
- | `runTraceImprovement` | `(TraceImprovementOptions) → TraceImprovementResult` | writes the full findings, evidence, report, and trace artifact pack |
1015
- | `runTraceQuestions` | `(TraceQuestionsOptions) → TraceQuestionsResult` | ask many free-form questions of one span list, concurrently, under one shared cost ledger; keeps each answer and checks its citations |
1016
- | `buildTraceFindingPacket` | `({ findings }) → TraceFindingPacket` | render any `AnalystFinding[]` without changing its schema |
1017
- | `runTraceStoreInvestigation` | `({ traceStore }) → TraceStoreInvestigationResult` | run the same packet layer over a hosted/custom `TraceAnalysisStore` |
1018
- | `loadTracesConfig` | `(path?) → TracesConfig \| undefined` | load BYO analysts and external analyzers |
1019
- | `watchSessions` | `(ObserverOptions) → Promise<void>` | live observer; `onLoop` / `onReport` / `signal` / `adapters` |
1020
- | `streamSessions` | `(TraceStreamOptions) → Promise<void>` | live JSONL-ready event stream over active sessions |
1021
- | `traceStreamEventsFromSpans` | `(spans, opts?) → TraceStreamEvent[]` | replay an existing span list as stream events |
1022
- | `analyzeLiveBatch` | `(spans, opts?) → TraceLiveBatch` | compute semantic online findings for one batch |
1023
- | `classifyLiveActions` | `(spans) → TraceLiveAction[]` | classify spans once as read/edit/verify/claim/tool/other |
1024
- | `defaultTraceLiveAnalysts` | `TraceLiveAnalyst[]` | the built-in online analysts; extend or replace them |
1025
- | `collectSessionIndex` | `(ScanOptions) → TraceSessionIndex` | scan sessions and return a reusable JSON-ready catalog |
1026
- | `inspectSessionIndex` | `(TraceSessionIndex) → TraceInspectionReport` | rank improvement findings from an index without rescanning sessions |
1027
- | `buildPolicyEvidenceRecord` | `(ref, spans, opts?) → PolicyEvidenceRecord` | summarize one session for downstream policy mining |
1028
- | `collectPolicyEvidence` | `(ScanOptions) → PolicyEvidenceRecord[]` | scan harness sessions and emit policy-evidence rows |
1029
- | `collectSessionWorkflow` | `(adapter, seeds, opts?) → SessionWorkflow` | resolve a bounded parent/child session tree with explicit relationship issues |
1030
- | `collectSessionSelection` | `(groups, opts?) → SessionSelection` | parse, expand, and optionally byte-bind selected session groups |
1031
- | `exportTraceEvidenceFile` | `(path, opts?) → { format, spans, redactionCount }` | convert compact evidence/events/OpenInference files to redacted OpenInference spans |
1032
- | `scanSessions` | `(ScanOptions) → AsyncIterable<ScannedSession>` | the shared locate→parse iterator |
1033
- | `collectSessions` | `(CollectOptions) → SessionBatch[]` | redacted per-session batches for your own pipeline |
1034
- | `redactSpans` | `(spans, { profile?, knownSecrets? }) → { spans, report }` | redact spans with agent-eval's redaction core |
1035
- | `assessSpans` / `verifySafe` | `(spans, profile?)` / `(path, { profile?, knownSecrets? })` → verdict | share-safety verdict for spans or files |
1036
- | `planUpload` / `executeUpload` | `(…, { backend? }) → …` | redact + dedup + send to any sink |
1037
- | `selectAdapters` / `listAdapters` / `resolveAdapter` | adapter selection + the harness registry |
1038
- | `PiAdapter` | `new PiAdapter({ sessionsRoot? })` | parse Pi's default session tree or an exact `PI_CODING_AGENT_SESSION_DIR` |
1039
- | `HarnessTraceAdapter` | interface (`locate` + `parse`) | implement to add a harness |
1040
- | `ExternalAnalyzer` / `Redactor` | `haloAnalyzer` / `hodoscopeAnalyzer` / `commandAnalyzer` / `commandRedactor` | drive engines/models you install |
36
+ | Find a session | `list` | [Refs, selection and harnesses](docs/select.md) |
37
+ | What happened, exactly, for $0 | `facts` | [Session facts](docs/facts.md) |
38
+ | What went wrong, ranked, for $0 | `analyze` | [Analyze a session](docs/analyze.md) |
39
+ | A question facts cannot answer, with checked citations | `ask` | [Ask a question](docs/ask.md) |
40
+ | The one gate: contracts, conformance, share safety | `check` | [Check](docs/check.md) |
41
+ | Where two runs split | `diff` | [Diff two runs](docs/diff.md) |
42
+ | Live supervision of sessions or a run tree | `watch` | [Watch](docs/watch.md) |
43
+ | Hand spans to another tool | `convert` | [Convert](docs/convert.md) |
44
+ | Keep a session's evidence | `bundle` | [Session bundle](docs/bundle.md) |
45
+ | Send redacted sessions to Intelligence | `upload` | [Upload](docs/upload.md) |
46
+ | Which skills are used | `skills` | [Skill usage](docs/skills.md) |
47
+ | Span tools for an MCP client with no shell | `mcp` | [MCP server](docs/mcp.md) |
48
+
49
+ Each page embeds its command's `--help`, and `tests/docs-help.test.ts` holds them equal to the CLI.
50
+
51
+ ```text
52
+ traces: what your agents did, where they wasted time, whether a run passes
53
+ Start here for one session: traces facts <ref> (25 lines); one field: add --format json | jq '<path>'
54
+
55
+ Which sessions ran? traces list [--cwd .] [--last N]
56
+ What happened (turns, PRs, failure, cost)? $0 traces facts <ref> | --current | --since 1d
57
+ Did we already try X / what was said? $0 traces facts <ref> | --since 30d --match '<re>'
58
+ Where did it waste time or get interrupted? $0 traces analyze <ref>
59
+ A question facts can't answer (model, $) traces ask <ref> --question "…" --budget 1
60
+ Does this run pass? (CI) traces check <ref> --contract c.json|conformance
61
+ Is this safe to share? traces check <path> --share-safe
62
+ Where did two runs diverge? traces diff <a> <b>
63
+ What is happening right now? traces watch [<run-dir>] [--once]
64
+ Keep a session's evidence traces bundle <ref> --out <dir> · bundle verify <dir>
65
+ Hand spans to another tool traces convert <ref> --out <file>
66
+ Ship sessions to Intelligence traces upload <ref>|--since T [--dry-run]
67
+ Which skills are used? traces skills [--unused]
68
+ Serve span tools over MCP traces mcp <ref>|--last N
69
+
70
+ ref: session id, or 8+ char unique prefix or suffix; session file; OTLP file/dir; bundle dir; run dir; store root
71
+ Select: --current --last N --since/--until 2h|1d|ISO --cwd DIR --harness ID --latest-turn
72
+ Output: --format text|json|jsonl|junit|github --out PATH --no-content
73
+ Exit: 0 yes · 1 no (fail, diverged, unsafe) · 2 could not answer (stderr says why)
74
+ traces <command> --help for flags, jq paths and examples · moved from 0.x: traces help moved
75
+ ```
76
+
77
+ ## Library
1041
78
 
1042
79
  ```ts
1043
- import {
1044
- AnalystRegistry, analyzeSpans, makeFinding, readOtlpInput, renderValidation,
1045
- streamSessions, watchSessions,
1046
- } from '@tangle-network/traces'
80
+ import { computeSessionFacts, readOtlpInput } from '@tangle-network/traces'
1047
81
 
1048
- // Read OTLP any system emitted, and find out what it can answer before analysing:
1049
82
  const input = await readOtlpInput('spans.otlp.jsonl') // or a directory
1050
- console.log(renderValidation(input.validation, { subject: 'spans.otlp.jsonl', spans: input.spans }))
1051
- if (input.validation.ok) await analyzeSpans(input.spans)
1052
-
1053
- // Observe live sessions, feed findings anywhere (read-only, cancellable):
1054
- const c = new AbortController()
1055
- await watchSessions({ all: true, signal: c.signal, onLoop: (l) => alert(`${l.toolName} x${l.occurrences}`) })
1056
-
1057
- // Feed a visualizer or dashboard:
1058
- await streamSessions({ all: true, signal: c.signal, includeSpans: false, onEvent: (event) => console.log(event) })
1059
-
1060
- // Run your own analyst instead of the built-ins:
1061
- const registry = new AnalystRegistry()
1062
- registry.register({
1063
- id: 'mine', description: '…', inputKind: 'trace-store', cost: { kind: 'deterministic' }, version: '1.0.0',
1064
- async analyze() {
1065
- return [makeFinding({ analyst_id: 'mine', area: 'custom', claim: '…', severity: 'info', evidence_refs: [], confidence: 0.9 })]
1066
- },
1067
- })
1068
- await analyzeSpans(input.spans, { registry })
83
+ console.log(input.validation.capabilities) // what this trace can answer
84
+ const [facts] = computeSessionFacts(input.spans)
1069
85
  ```
1070
86
 
1071
- ## Examples
1072
-
1073
- Runnable, in [`examples/`](./examples):
1074
-
1075
- | File | Shows |
1076
- |---|---|
1077
- | [`observe-and-alert.ts`](./examples/observe-and-alert.ts) | tail live sessions and alert on stuck loops |
1078
- | [`custom-analyst.ts`](./examples/custom-analyst.ts) | register and run your own analyst |
1079
- | [`improvement-config.mjs`](./examples/improvement-config.mjs) | plug in BYO analysts for `traces stream` / `traces improve` |
1080
- | [`custom-backend.ts`](./examples/custom-backend.ts) | redact + dedup + upload to your own sink |
1081
- | [`register-harness.ts`](./examples/register-harness.ts) | add a new harness by implementing `HarnessTraceAdapter` |
1082
- | [`external-engines.ts`](./examples/external-engines.ts) | drive HALO, Hodoscope, and an external PII scrubber |
87
+ The SDK is 25 functions and 25 types, one engine per command: [Library (SDK)](docs/sdk.md). Runnable code is in [examples](examples/README.md).
1083
88
 
1084
- ## Audit benchmark
89
+ ## Docs
1085
90
 
1086
- [`bench/audit/`](./bench/audit) asks nineteen questions about three synthetic agent
1087
- sessions and scores an arm's answers by exact match, with no model judging anything. It
1088
- measures whether reading a session with `traces` beats reading the raw files, on facts a
1089
- single read cannot cover: counts over hundreds of records, timestamps of specific records,
1090
- and facts linked across tool calls and sessions.
1091
-
1092
- ```bash
1093
- pnpm bench:audit fixtures --out /tmp/audit-bench # the tree an arm may read
1094
- pnpm bench:audit gold --out /tmp/audit-gold.json # the answer key, kept away from the arm
1095
- pnpm bench:audit score /tmp/arm-answers.json --fixtures /tmp/audit-bench
1096
- ```
1097
-
1098
- [`bench/audit/README.md`](./bench/audit/README.md) documents the questions, the scoring
1099
- rules, the answers-file format, and which answers are re-derived from the generated bytes.
1100
- The benchmark ships outside the published package; the model arms are run by hand.
1101
-
1102
- ## Develop
1103
-
1104
- ```bash
1105
- pnpm install
1106
- pnpm dev analyze --harness claude-code --last 1 # run from source via tsx
1107
- pnpm test
1108
- pnpm typecheck
1109
- pnpm build # dist/index.js (SDK) + dist/cli.js (bin) + .d.ts
1110
- ```
91
+ - **Commands:** the pages in the table above.
92
+ - **Reference:** [Library (SDK)](docs/sdk.md) · [Migrate to 1.0](docs/migrate-1.0.md) · [Parsers outside traces](docs/parsers.md)
93
+ - **Contribute:** [Develop and release](docs/develop.md): local setup, releases and how quality is measured