@tangle-network/traces 0.11.6 → 0.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +132 -3
- package/dist/{chunk-MEAEXKUV.js → chunk-7DOBU7BJ.js} +14750 -10886
- package/dist/cli.js +221 -169
- package/dist/index.d.ts +998 -58
- package/dist/index.js +113 -1
- package/docs/trace-analysts.md +307 -0
- package/install.sh +0 -0
- package/package.json +23 -19
- package/skills/build-trace-analyst/SKILL.md +3 -0
- package/skills/inspect-agent-traces/SKILL.md +22 -20
package/docs/trace-analysts.md
CHANGED
|
@@ -8,6 +8,8 @@ Keeping them separate prevents an exploratory model output from being reported a
|
|
|
8
8
|
| What happened? | Built-in local checks | Findings from explicit trace facts |
|
|
9
9
|
| Why might it have happened? | `--llm`, HALO, or a custom analyst | Findings or a diagnosis report with cited spans |
|
|
10
10
|
| What behavior should we inspect? | Hodoscope | Samples marked `needs_review` |
|
|
11
|
+
| What exactly does this session say about X? | `traces ask` | An answer per question, with checked `trace://` citations |
|
|
12
|
+
| What does this session state, exactly and for free? | `traces facts` | A deterministic facts sheet, no model call |
|
|
11
13
|
|
|
12
14
|
## Start here
|
|
13
15
|
|
|
@@ -34,6 +36,8 @@ Without those fields, the result is explicitly incomplete; names, nicknames, and
|
|
|
34
36
|
It reports missing files, duplicate IDs, contradictory parents, and cycles instead of guessing.
|
|
35
37
|
Use `--max-workflow-sessions <n>` to set a different bound.
|
|
36
38
|
Claude Code already folds nested subagent files into its parent trace.
|
|
39
|
+
The adapter preserves Claude source UUIDs in `traces.claude.source_*` attributes and derives valid OTLP IDs for the exported graph.
|
|
40
|
+
It emits tool spans from recorded tool calls, but does not invent loop iterations or causal links absent from the transcript.
|
|
37
41
|
Ordinary Claude subagents use their parent tool call ID.
|
|
38
42
|
Claude Workflow subagents use the run ID and transcript directory returned by the parent `Workflow` call.
|
|
39
43
|
If the same Workflow run is resumed, each child attaches to the latest matching call that started before it.
|
|
@@ -54,6 +58,262 @@ Returned directories from another resumed Claude session are parsed and included
|
|
|
54
58
|
The OpenInference file is the shared input for external engines.
|
|
55
59
|
The original trace and exact cited span remain available for review.
|
|
56
60
|
|
|
61
|
+
## Ask free-form questions
|
|
62
|
+
|
|
63
|
+
The built-in kinds ask fixed questions.
|
|
64
|
+
`traces ask` asks your own.
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
traces ask --harness codex --session <id> \
|
|
68
|
+
--question "Which shell commands exited non-zero?" \
|
|
69
|
+
--question "Which pull requests did the session open, and were they merged?" \
|
|
70
|
+
--dir .traces/ask
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Each question runs through agent-eval's `runTraceAnalyst` directly, not through the analyst registry.
|
|
74
|
+
That matters for three reasons.
|
|
75
|
+
|
|
76
|
+
- The registry keeps only findings, so it discards the engine's prose answer. `ask` keeps the answer text verbatim.
|
|
77
|
+
- The registry runs analysts one at a time. `ask` runs questions concurrently, so wall time falls toward the slowest question instead of the sum.
|
|
78
|
+
- One shared `CostLedger` bounds the whole run, so `--budget` means the same thing whatever the number of questions.
|
|
79
|
+
|
|
80
|
+
### Questions
|
|
81
|
+
|
|
82
|
+
A question stays short on purpose.
|
|
83
|
+
The engine shows the model a preview of each long input: it keeps the first 500 and last 500 characters and drops the middle.
|
|
84
|
+
A question inside the limit therefore reaches the model whole, and the answer rules sit in the first 500 characters of the instructions, which the preview always keeps.
|
|
85
|
+
`ask` rejects an over-long question before it starts a model call and names the limit.
|
|
86
|
+
Put the detail in the entry's `instructions` field instead, which the model reads after the rules.
|
|
87
|
+
|
|
88
|
+
Read many questions from a file:
|
|
89
|
+
|
|
90
|
+
```json
|
|
91
|
+
{
|
|
92
|
+
"questions": [
|
|
93
|
+
"What was the last thing the human asked for?",
|
|
94
|
+
{
|
|
95
|
+
"id": "failed-commands",
|
|
96
|
+
"question": "Which shell commands exited non-zero?",
|
|
97
|
+
"instructions": "Report the command line and the exit code. Ignore commands the agent only proposed.",
|
|
98
|
+
"answerSchema": {
|
|
99
|
+
"type": "array",
|
|
100
|
+
"items": {
|
|
101
|
+
"type": "object",
|
|
102
|
+
"properties": { "command": { "type": "string" }, "exit_code": { "type": "integer" } },
|
|
103
|
+
"required": ["command", "exit_code"]
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
]
|
|
108
|
+
}
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
An `answerSchema` makes the answer one JSON value a scorer can compare field by field.
|
|
112
|
+
The supported keywords are `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, `const`, `title`, and `description`.
|
|
113
|
+
A keyword that constrains one JSON type must declare it: `required`, `properties`, and `additionalProperties` need `"type": "object"`, and `items` needs `"type": "array"`, or the constraint would be skipped for an answer of another shape.
|
|
114
|
+
Any other keyword is rejected when the run starts.
|
|
115
|
+
This package carries no JSON Schema library, and a constraint that is quietly ignored would let a wrong answer pass as checked.
|
|
116
|
+
|
|
117
|
+
### What the run guarantees
|
|
118
|
+
|
|
119
|
+
- Every question receives the deterministic [session-facts sheet](#session-facts) as prepared context, before its first model call. It costs nothing and answers the counting questions the bounded trace tools cannot.
|
|
120
|
+
- Every `trace://<trace_id>/span/<span_id>` URI in an answer is resolved against the store. An unresolvable citation fails that question.
|
|
121
|
+
A citation the model wrapped in Markdown emphasis (`**...**`, `_..._`, `~~...~~`) or ended a sentence with resolves like a bare one: the delimiters are prose, not part of the span ID.
|
|
122
|
+
- Findings the answer submits still pass the same evidence gate as the built-in kinds. Refused findings are counted by reason in both artifacts.
|
|
123
|
+
- A failed question never stops the others. Its failure is recorded on its own answer, and the remaining answers are written.
|
|
124
|
+
- Ctrl-C keeps what the run already bought. The signal reaches the engine, not the checks that follow it: an answer that came back is kept with its citations resolved, and each question the run never reached is recorded as `aborted`.
|
|
125
|
+
- The artifacts are written before the exit code is decided. `ask` exits 1 when any question failed, returned no answer, broke its schema, or cited a missing span.
|
|
126
|
+
- `totals.wallTimeMs` covers the whole run, including writing and indexing the trace file; `totals.setupTimeMs` names that part. `result.effectiveConcurrency` is the number of workers the run created, `min(--concurrency, questions)`, and `totals.peakConcurrency` is how many actually overlapped.
|
|
127
|
+
|
|
128
|
+
### Budget under concurrency
|
|
129
|
+
|
|
130
|
+
`--budget` is the ceiling shared by every question; `--question-budget` is the provider ceiling for one question.
|
|
131
|
+
The ledger reserves each model call's maximum charge before the call runs, and releases the unused part when the call settles.
|
|
132
|
+
Two consequences follow.
|
|
133
|
+
|
|
134
|
+
- A budget below one call's reservation refuses the run before any model call, rather than failing every question.
|
|
135
|
+
- A budget that admits fewer concurrent reservations than `--concurrency` still runs, and the report carries a warning naming how many concurrent calls it covers.
|
|
136
|
+
|
|
137
|
+
A question the ledger refuses is reported as `budget-refused`, and the answers already produced are kept.
|
|
138
|
+
That kind is decided from the accounting, not only from the error text.
|
|
139
|
+
The refusal happens inside the model proxy, behind the DSPy bridge, whose HTTP error handling can replace the Node error with its own message.
|
|
140
|
+
So a failed question is reported as `budget-refused` whenever the shared ledger's settled spend left less than one model call's reservation at the time it failed and nothing else in the failure names its cause.
|
|
141
|
+
A failure whose own text names its cause keeps that cause: a bridge version mismatch stays `error`, so the reinstall hint still prints, and an empty answer from the bridge stays `no-answer`.
|
|
142
|
+
The message itself is kept verbatim on the answer.
|
|
143
|
+
|
|
144
|
+
### SDK
|
|
145
|
+
|
|
146
|
+
```ts
|
|
147
|
+
import { analysisEngineFromEnv, runTraceQuestions, writeTraceQuestionsArtifacts } from '@tangle-network/traces'
|
|
148
|
+
|
|
149
|
+
const result = await runTraceQuestions({
|
|
150
|
+
questions: [
|
|
151
|
+
{ id: 'failed-commands', question: 'Which shell commands exited non-zero?' },
|
|
152
|
+
{ id: 'last-ask', question: 'What was the last thing the human asked for?' },
|
|
153
|
+
],
|
|
154
|
+
spans,
|
|
155
|
+
engine: analysisEngineFromEnv({ model: 'gpt-5.6-luna', maxCostUsd: 1 }),
|
|
156
|
+
concurrency: 4,
|
|
157
|
+
budgetUsd: 2,
|
|
158
|
+
})
|
|
159
|
+
await writeTraceQuestionsArtifacts(result, '.traces/ask')
|
|
160
|
+
if (!result.ok) process.exitCode = 1
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
`result.questions[i].answer` is the engine's prose, unedited.
|
|
164
|
+
`result.totals` carries the wall time, the summed question time, the peak concurrency, and the cost with its provenance.
|
|
165
|
+
|
|
166
|
+
## Session facts
|
|
167
|
+
|
|
168
|
+
`traces facts` computes a fixed set of session facts straight from the spans.
|
|
169
|
+
It runs no model, opens no engine, and spends nothing.
|
|
170
|
+
The same spans always produce the same sheet.
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
traces facts --harness codex --session <id> # JSON on stdout
|
|
174
|
+
traces facts --harness codex --last 5 --format text # the short readable form
|
|
175
|
+
traces facts --otlp spans.otlp.jsonl --out facts.json
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
### Why it exists
|
|
179
|
+
|
|
180
|
+
The trace tools are bounded, and above their bounds they answer a different question than the one asked.
|
|
181
|
+
`viewTrace` returns a `≤20`-entry span-name histogram once a trace exceeds `perCallByteCeiling` (150,000 bytes).
|
|
182
|
+
`countTraces` counts traces, not spans.
|
|
183
|
+
`viewSpans` needs span ids the reader does not have yet.
|
|
184
|
+
`searchTrace` stops at 500 hits.
|
|
185
|
+
|
|
186
|
+
A model asked "how many tool calls ran?" therefore adds up a capped histogram and decides by eye which names belong.
|
|
187
|
+
Measured over twelve private audit sessions, the model-backed analyst arm scored a deterministic mean of **0.389**.
|
|
188
|
+
Extracting the same answers mechanically from the OTLP spans those runs already wrote scored **0.858** — the facts were present and exact the whole time.
|
|
189
|
+
The sheet is that extraction, made part of the tool.
|
|
190
|
+
|
|
191
|
+
### What it states
|
|
192
|
+
|
|
193
|
+
| Field | Value |
|
|
194
|
+
| --- | --- |
|
|
195
|
+
| `toolCalls` | TOOL spans **this session's agent** invoked. Synthesized lifecycle spans are excluded and counted in `synthesizedToolSpans`; calls made by a spawned subagent are excluded and counted in `subagentToolSpans` |
|
|
196
|
+
| `toolCallsByName` | the same calls by tool name |
|
|
197
|
+
| `subagentToolSpans` | TOOL spans excluded because a subagent, not this agent, made the call |
|
|
198
|
+
| `subagents` | every subagent-spawning call, with the task name the call recorded |
|
|
199
|
+
| `pullRequests` | pull requests the commands created and merged, each named by number or head branch, with the command span and the join evidence |
|
|
200
|
+
| `humanTurns` | `user.prompt` turns a person typed into this session, in order, with timestamps |
|
|
201
|
+
| `excludedTurns` | every `user.prompt` turn `humanTurns` left out, grouped by the reason, with the span ids |
|
|
202
|
+
| `turnsByActor` | every `user.prompt` turn by actor, so the human filter is checkable |
|
|
203
|
+
| `finalMessages` | the last message of the session's own agent, and of each subagent task, separately |
|
|
204
|
+
| `changedFiles` | paths the harness recorded for its own file changes, plus paths from `*** Add/Update/Delete/Move to File:` patch headers and from file-editing tool arguments |
|
|
205
|
+
| `firstRecordAt`, `lastRecordAt` | the earliest span start and latest span end among this session's own records |
|
|
206
|
+
| `unreadRecords` | records the session reader could not parse |
|
|
207
|
+
| `tokenTotal` | the harness's cumulative total, when a span carries `traces.session.total_tokens` |
|
|
208
|
+
|
|
209
|
+
Three rules hold for every field.
|
|
210
|
+
|
|
211
|
+
- **Every fact is about this session's own agent.** Some harnesses fold a spawned agent's transcript into the parent's trace — Claude Code writes one file per subagent under the session directory and the adapter reads them all — so a trace can carry both what this agent did and what a child it spawned did. The child's work is named in `subagents` and counted in `subagentToolSpans`, never added to the parent's totals. On a trace with no subagent records every one of those counts is zero.
|
|
212
|
+
- **Every fact names the span ids it came from.** A reader can open those spans and check the number. The sheet is not a span and cannot be cited.
|
|
213
|
+
- **A fact the spans cannot support is `null` with a stated reason.** It is never guessed and never a silent zero. `partial` marks a measured value known to be incomplete.
|
|
214
|
+
|
|
215
|
+
`facts` exits non-zero when a selected session produced no record spans, rather than printing a sheet of zeros for a session it could not read.
|
|
216
|
+
|
|
217
|
+
#### Pull requests
|
|
218
|
+
|
|
219
|
+
The command spans carry the script, the exit code and the output.
|
|
220
|
+
`pullRequests` scans each script the way a shell would, so a `gh pr create` inside a heredoc body or a commit message is not counted as a command that ran.
|
|
221
|
+
A pull request is named by its number when the command or an output that joins to it shows one, and by its head branch when neither does.
|
|
222
|
+
A create whose stdout was redirected away takes the number a later output states for the same branch on the same line, and names the span that stated it.
|
|
223
|
+
A failed `git push && gh pr create` counts only when the output shows `gh` itself answering; otherwise the shell never reached it.
|
|
224
|
+
A trace whose spans carry no executed command returns `null` with that reason, because "no pull requests" and "the spans cannot say" are different answers.
|
|
225
|
+
|
|
226
|
+
#### Human turns
|
|
227
|
+
|
|
228
|
+
A user message is one the human typed into this session.
|
|
229
|
+
Three filters run in order, and each excluded span is listed in `excludedTurns` with its reason, never silently dropped:
|
|
230
|
+
|
|
231
|
+
- history the session carries but did not receive — the prefix a fork copies from its parent, and the turns a compaction replays;
|
|
232
|
+
- turns whose actor is not a person — an instruction file, an environment-context block, a system reminder, a subagent notification, a turn-aborted marker, or a skill or slash-command expansion;
|
|
233
|
+
- a second record of the turn before it, meaning the same text at the same instant with no model call, tool call or assistant message between them.
|
|
234
|
+
|
|
235
|
+
For Codex the actor comes from the harness's own per-item labelling (`internal_chat_message_metadata_passthrough.content_item_kinds`) whenever the record carries it: a message is the person's exactly when every item in it is a `user.` kind.
|
|
236
|
+
|
|
237
|
+
### Session facts as prepared context
|
|
238
|
+
|
|
239
|
+
The same sheet is supplied to the model-backed analysts before their first model call, through `TraceAnalystDefinition.prepareContext`.
|
|
240
|
+
It reaches the built-in kinds run by `analyze --llm`, `investigate`, and `improve`, and it reaches every `traces ask` question inside `PREPARED CONTEXT:`.
|
|
241
|
+
|
|
242
|
+
- The sheet is bounded at `PREPARED_CONTEXT_BYTE_CEILING` (30,000 bytes), a fifth of the `perCallByteCeiling` of 150,000 the trace tools work to, so the rest of the budget stays available for the tool calls the model still makes.
|
|
243
|
+
- When the sheet does not fit, fields are shed from the largest downward and the shed is listed in `omitted_fields`. The tool-call counts are the last facts to go.
|
|
244
|
+
- Receiving the sheet changes an analyst's behavior, so its version carries `+session-facts.1`. `createTraceAnalyst` records `prepare_context` in the exact-run identity, and a changed prepared context must not hide behind an unchanged version.
|
|
245
|
+
- The sheet is not evidence. Citations still resolve against the raw spans, which is why every fact names its span ids rather than asking the model to trust the sheet.
|
|
246
|
+
|
|
247
|
+
Pass `sessionFactsContext: false` to `analyzeSpans` to run an analyst without it.
|
|
248
|
+
|
|
249
|
+
```ts
|
|
250
|
+
import { buildSessionFactsReport, computeSessionFacts, renderSessionFacts } from '@tangle-network/traces'
|
|
251
|
+
|
|
252
|
+
const [facts] = computeSessionFacts(spans)
|
|
253
|
+
console.log(facts.toolCalls.value, facts.toolCalls.spanIds)
|
|
254
|
+
console.log(renderSessionFacts(buildSessionFactsReport(spans)))
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
## Evidence-gate rejections
|
|
258
|
+
|
|
259
|
+
A model-backed analyst can submit a finding the evidence gate then refuses.
|
|
260
|
+
Without the reason, a report showing "0 findings" reads as "the model found nothing" when the truth may be "the gate refused everything it found".
|
|
261
|
+
|
|
262
|
+
`analyze`, `investigate`, `improve`, and `ask` now carry those refusals:
|
|
263
|
+
|
|
264
|
+
- the CLI log adds whatever the event carries beyond its own text to each `finding rejected` line: the cause (`reason` for the gate's rejections and the bridge-row rejection, `issues` for a schema failure), the offending URI, the citation counts, and the subject;
|
|
265
|
+
- the analyst table's Detail cell names the reasons and their counts;
|
|
266
|
+
- `result.findingRejections` (investigation and improvement) and `answers.json` (`ask`) hold the counts per analyst and reason.
|
|
267
|
+
|
|
268
|
+
The common reasons are an excerpt the cited span does not contain, a span the trace does not hold, and too few distinct citations for the kind.
|
|
269
|
+
|
|
270
|
+
## External analyzer failures exit non-zero
|
|
271
|
+
|
|
272
|
+
`--analyzer halo|hodoscope|prime|<command>` promises that engine's output.
|
|
273
|
+
An analyzer that fails now writes its error into the report as before, and then the command exits 1 naming every analyzer that failed.
|
|
274
|
+
Scripts that treated exit 0 as "the analyzer ran" were reading a report that said otherwise.
|
|
275
|
+
|
|
276
|
+
The check runs on `analyze`, `investigate`, and `improve`, and it covers every analyzer the run requested, not only the ones named on the command line.
|
|
277
|
+
`investigate` and `improve` load the default traces config file, so an analyzer declared in `externalAnalyzers` there is a requested analyzer too, and a flaky one now turns those two commands red.
|
|
278
|
+
`analyze` does not load a default config, so only its own `--analyzer` flags reach the check.
|
|
279
|
+
|
|
280
|
+
## Codex tool outcomes
|
|
281
|
+
|
|
282
|
+
Both function and custom tool outputs use the same status parser.
|
|
283
|
+
Explicit exit codes, error flags, and runner headers determine tool status.
|
|
284
|
+
Explicit failure takes precedence when structured fields conflict.
|
|
285
|
+
Text such as `error:`, `ENOENT`, or `command failed` inside stdout does not determine status.
|
|
286
|
+
An unmatched call, running process, or output without trustworthy status remains `UNSET`.
|
|
287
|
+
A recorded `wait_agent` timeout is a completed poll, not proof that the agent finished.
|
|
288
|
+
Tool output remains in `output.value`, with the existing size limit and truncation receipt.
|
|
289
|
+
|
|
290
|
+
Error and retry counts use explicit failures.
|
|
291
|
+
Successful follow-up counts require explicit success; unknown follow-up outcomes remain `null` in the detailed report.
|
|
292
|
+
The runtime span projection leaves its optional status absent for `UNSET` spans.
|
|
293
|
+
Use the execution report for terminal run outcomes.
|
|
294
|
+
The older runtime store's required run status cannot represent unknown and is not completion evidence.
|
|
295
|
+
|
|
296
|
+
## Model-assisted citations
|
|
297
|
+
|
|
298
|
+
agent-eval's evidence gate accepts a model finding only when each cited span exists and each excerpt appears in that span.
|
|
299
|
+
Before the gate runs, `traces` normalizes the citations of the built-in `--llm` analysts against the analyzed spans.
|
|
300
|
+
|
|
301
|
+
- An excerpt copied from a `searchTrace` hit carries JSON escapes such as `\"` and `\n`.
|
|
302
|
+
It is replaced by its decoded form only when the cited span contains that decoded text.
|
|
303
|
+
- A citation of a readable harness ID, such as `tool:<call_id>` or the Codex session ID, is replaced by the hex OTLP span ID.
|
|
304
|
+
The readable ID must name exactly one span through its `traces.<harness>.source_*` attributes.
|
|
305
|
+
- A subject from another analyst's vocabulary is removed, because the subject is optional.
|
|
306
|
+
The claim and its citations still go through the gate.
|
|
307
|
+
|
|
308
|
+
The gate itself is unchanged and checks every rewritten finding.
|
|
309
|
+
A fabricated excerpt, an unknown ID, or an ambiguous ID remains rejected.
|
|
310
|
+
Normalization never adds a citation, so a failure-mode finding still needs two distinct spans.
|
|
311
|
+
A subject outside the subject grammar is rejected before normalization runs.
|
|
312
|
+
Wrapped analysts report a version ending in `+traces-citations-1`.
|
|
313
|
+
|
|
314
|
+
A custom analyst built with `createTraceAnalyst` gets the same behavior through `normalizeAnalystCitations(definitions, spans)`.
|
|
315
|
+
Pass the spans that the analyst's trace store was written from.
|
|
316
|
+
|
|
57
317
|
## External engines
|
|
58
318
|
|
|
59
319
|
External engines are optional tools that you install separately.
|
|
@@ -287,3 +547,50 @@ The required review path is:
|
|
|
287
547
|
```text
|
|
288
548
|
production trace -> finding -> reviewed feedback -> eval case -> candidate change -> comparison
|
|
289
549
|
```
|
|
550
|
+
|
|
551
|
+
## Read retained source fields
|
|
552
|
+
|
|
553
|
+
Normalized conversation text and tool values remain capped.
|
|
554
|
+
Create a full bundle before the harness rotates its original files.
|
|
555
|
+
Explicitly authorize that bundle when analysis needs omitted text:
|
|
556
|
+
|
|
557
|
+
```bash
|
|
558
|
+
traces bundle --session <session-id> --out ./session-bundle
|
|
559
|
+
traces analyze --source-bundle ./session-bundle --llm
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
`investigate` and `improve` accept the same flag.
|
|
563
|
+
The SDK accepts `sourceBundle: { path, maxRecordBytes? }` in analysis and investigation options.
|
|
564
|
+
This authorization exposes `traces.readSpanSource` to local analysts through the existing trace store.
|
|
565
|
+
External analyzers receive normalized OTLP without this capability.
|
|
566
|
+
Ordinary OTLP input does not authorize source reads.
|
|
567
|
+
|
|
568
|
+
The tool selects a trace, span, attribute, and optional `source_index`.
|
|
569
|
+
It returns a UTF-8 byte window of that field's decoded value.
|
|
570
|
+
Adapters that extract message text blocks retain only those text leaves, excluding adjacent tool blocks.
|
|
571
|
+
Gemini also accepts structured message content; its source reference selects that entire content field.
|
|
572
|
+
Multiple records or text fragments use separate source indices.
|
|
573
|
+
Strings retain their decoded source text, including whitespace and JSON-looking strings.
|
|
574
|
+
Structured values use the existing sorted-key tool-value JSON encoder.
|
|
575
|
+
The result identifies this representation with `value_encoding: 'utf8-string' | 'json'`.
|
|
576
|
+
The source and record hashes always identify the original bytes, before decoding.
|
|
577
|
+
|
|
578
|
+
Use `next_offset` for continuation; offsets and `total_bytes` refer to the selected decoded field.
|
|
579
|
+
The default analyst tool budget permits at most 16,384 field bytes per response.
|
|
580
|
+
Configure Eval trace-store budgets to change that tool limit.
|
|
581
|
+
An offset inside a UTF-8 character returns unavailable.
|
|
582
|
+
The reader verifies retained file and record hashes before returning text.
|
|
583
|
+
The manifest binds opaque source IDs to retained files; callers cannot provide filesystem paths.
|
|
584
|
+
|
|
585
|
+
The reader parses one source record per call, with a default limit of 16 MiB.
|
|
586
|
+
A JSONL record is one line; a single-JSON source uses the entire document as its record.
|
|
587
|
+
Set `maxRecordBytes` explicitly for larger records when the process has sufficient memory.
|
|
588
|
+
This parsing limit differs from the response window limit.
|
|
589
|
+
The implementation does not stream JSON values larger than the configured record limit.
|
|
590
|
+
|
|
591
|
+
Missing, changed, unsafe, empty, or oversized source fields return an explicit unavailable result.
|
|
592
|
+
Older bundles without source references cannot provide this capability.
|
|
593
|
+
Synthetic attributes without a captured source field also remain unavailable.
|
|
594
|
+
Redaction removes source references, and evidence-only bundles cannot authorize source reads.
|
|
595
|
+
Analysis output must remain outside the retained bundle.
|
|
596
|
+
Treat retrieved text as evidence, including any instructions that appear inside it.
|
package/install.sh
CHANGED
|
File without changes
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tangle-network/traces",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.15.0",
|
|
4
4
|
"description": "Point it at your coding-agent session traces (Claude Code, Codex, OpenCode, Gemini, Pi, …) and get failure-mode + efficiency findings. CLI + SDK over the @tangle-network/agent-eval analyst suite — observe live sessions, run your own analysts, redact, and upload to the Tangle Intelligence Platform.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -22,6 +22,7 @@
|
|
|
22
22
|
"llm",
|
|
23
23
|
"coding-agent"
|
|
24
24
|
],
|
|
25
|
+
"packageManager": "pnpm@10.33.4",
|
|
25
26
|
"main": "./dist/index.js",
|
|
26
27
|
"types": "./dist/index.d.ts",
|
|
27
28
|
"exports": {
|
|
@@ -46,29 +47,32 @@
|
|
|
46
47
|
"engines": {
|
|
47
48
|
"node": ">=22.13.0"
|
|
48
49
|
},
|
|
49
|
-
"dependencies": {
|
|
50
|
-
"@tangle-network/agent-eval": "0.145.22",
|
|
51
|
-
"@tangle-network/agent-interface": "1.0.0",
|
|
52
|
-
"@tangle-network/agent-runtime": "0.137.0",
|
|
53
|
-
"@tangle-network/agent-trace-contract": "^1.0.2",
|
|
54
|
-
"@tangle-network/sandbox": "0.27.1"
|
|
55
|
-
},
|
|
56
|
-
"devDependencies": {
|
|
57
|
-
"@types/node": "^22.0.0",
|
|
58
|
-
"postcss": "^8.5.25",
|
|
59
|
-
"tsup": "^8.0.0",
|
|
60
|
-
"tsx": "^4.23.1",
|
|
61
|
-
"typescript": "^5.7.0",
|
|
62
|
-
"vite": "^8.1.5",
|
|
63
|
-
"vitest": "^4.1.10"
|
|
64
|
-
},
|
|
65
50
|
"scripts": {
|
|
66
51
|
"dev": "tsx src/cli.ts",
|
|
52
|
+
"bench:audit": "tsx bench/audit/cli.ts",
|
|
67
53
|
"build": "tsup src/index.ts src/cli.ts --format esm --target node22 --clean --dts",
|
|
68
54
|
"check:package": "node scripts/check-package-bin.mjs",
|
|
69
55
|
"check:source": "node scripts/check-source-text.mjs",
|
|
56
|
+
"prepare": "node scripts/check-source-text.mjs",
|
|
70
57
|
"test": "vitest run",
|
|
71
58
|
"test:watch": "vitest",
|
|
72
|
-
"typecheck": "tsc --noEmit"
|
|
59
|
+
"typecheck": "tsc --noEmit",
|
|
60
|
+
"prepublishOnly": "pnpm check:source && pnpm build && pnpm check:package"
|
|
61
|
+
},
|
|
62
|
+
"dependencies": {
|
|
63
|
+
"@tangle-network/agent-eval": "0.182.0",
|
|
64
|
+
"@tangle-network/agent-interface": "2.6.1",
|
|
65
|
+
"@tangle-network/agent-runtime": "0.231.1",
|
|
66
|
+
"@tangle-network/agent-trace-contract": "1.0.2",
|
|
67
|
+
"@tangle-network/sandbox": "0.40.2"
|
|
68
|
+
},
|
|
69
|
+
"devDependencies": {
|
|
70
|
+
"@types/node": "^22.0.0",
|
|
71
|
+
"postcss": "^8.5.26",
|
|
72
|
+
"tsup": "^8.0.0",
|
|
73
|
+
"tsx": "^4.23.13",
|
|
74
|
+
"typescript": "^5.7.0",
|
|
75
|
+
"vite": "^8.2.2",
|
|
76
|
+
"vitest": "^4.1.11"
|
|
73
77
|
}
|
|
74
|
-
}
|
|
78
|
+
}
|
|
@@ -17,6 +17,9 @@ Check live analysts separately only for `traces stream`.
|
|
|
17
17
|
Extend an existing analyst when it already emits the target with usable evidence.
|
|
18
18
|
Do not inspect bundled `dist` or `node_modules`.
|
|
19
19
|
|
|
20
|
+
Answer a one-off question with `traces ask --question "<text>"` instead.
|
|
21
|
+
Build an analyst when the question repeats and its answer must be scored, not read.
|
|
22
|
+
|
|
20
23
|
## Implement
|
|
21
24
|
|
|
22
25
|
- Use `Analyst`, `AnalystRegistry`, `TraceAnalysisStore`, and `makeFinding` from `@tangle-network/traces`.
|
|
@@ -5,13 +5,12 @@ description: Inspect real agent workflows with the published Traces CLI and expo
|
|
|
5
5
|
|
|
6
6
|
# Inspect agent traces
|
|
7
7
|
|
|
8
|
-
Use the deterministic CLI first.
|
|
9
|
-
Keep inspection local and read-only.
|
|
8
|
+
Use the deterministic CLI first; keep inspection local and read-only.
|
|
10
9
|
|
|
11
10
|
## Choose the way in
|
|
12
11
|
|
|
13
|
-
If the trace is already OTLP (
|
|
14
|
-
|
|
12
|
+
If the trace is already OTLP (any `@tangle-network/agent-trace-contract` emitter or other
|
|
13
|
+
conforming exporter), read it directly. No adapter is involved.
|
|
15
14
|
|
|
16
15
|
```bash
|
|
17
16
|
traces validate spans.otlp.jsonl # what can this trace answer? exit 1 on error findings
|
|
@@ -22,14 +21,13 @@ traces analyze --otlp results/sessions --out .traces/all.md # a directory of e
|
|
|
22
21
|
Use `--harness` only for coding agents whose on-disk format we do not control.
|
|
23
22
|
Those adapters are the legacy edge, not the way to integrate a system you own.
|
|
24
23
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
An `otlp/` subdirectory, when present, is read on its own.
|
|
24
|
+
In a run directory only the OTLP files are read; other `*.jsonl` logs are listed with
|
|
25
|
+
what they hold. An `otlp/` subdirectory, when present, is read on its own.
|
|
28
26
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
27
|
+
A section headed `inputs incomplete`, or carrying an `Inputs incomplete` line above its
|
|
28
|
+
table, is computed from a field the trace does not carry everywhere. Report those numbers
|
|
29
|
+
as uncaptured, never as zero spend. The `trace conformance` section names each such
|
|
30
|
+
capability once; the markers repeat it at the number.
|
|
33
31
|
|
|
34
32
|
For a loop trace, read `round-over-round convergence` (did round N+1 improve on N) and
|
|
35
33
|
`steering chain` (which verdict caused which retry) before drawing any conclusion about
|
|
@@ -48,7 +46,7 @@ traces analyze --harness codex --current --latest-turn --workflow \
|
|
|
48
46
|
- Use `--session <id-or-path>` to pin a listed session.
|
|
49
47
|
- Use `--latest-turn` for the current task in a resumed Codex or Claude Code session.
|
|
50
48
|
- Use `--workflow` to include workers linked by stable parent and child IDs.
|
|
51
|
-
- Use `--max-workflow-sessions <n>` only when the default 100-file bound is
|
|
49
|
+
- Use `--max-workflow-sessions <n>` only when the default 100-file bound is too small.
|
|
52
50
|
- For Claude Code, use `--harness claude-code --session <path> --latest-turn`; nested subagents are included.
|
|
53
51
|
|
|
54
52
|
Never join agents by display name or timestamp when Traces reports missing or conflicting IDs.
|
|
@@ -70,7 +68,7 @@ traces improve --harness codex --current --latest-turn --workflow \
|
|
|
70
68
|
```
|
|
71
69
|
|
|
72
70
|
`improve` writes findings, evidence, a report, and spans.
|
|
73
|
-
It
|
|
71
|
+
It edits no agent, repository, memory store, or knowledge base.
|
|
74
72
|
|
|
75
73
|
Write one session's durable evidence directory for a later reader:
|
|
76
74
|
|
|
@@ -80,12 +78,11 @@ traces bundle --harness claude-code --session <id-or-path> --out .traces/bundle
|
|
|
80
78
|
|
|
81
79
|
`bundle` copies the transcript, the derived report and spans, the `.evolve` ledger rows inside the session window, and a `manifest.json` with a SHA-256 per file.
|
|
82
80
|
It spends no model call.
|
|
83
|
-
A missing transcript stops the assembly.
|
|
84
|
-
An absent optional input is recorded in `manifest.absent` with the probed path.
|
|
81
|
+
A missing transcript stops the assembly; an absent optional input is recorded in `manifest.absent` with the probed path.
|
|
85
82
|
|
|
86
83
|
## Pick the view for the reader
|
|
87
84
|
|
|
88
|
-
That bundle is the FULL view (`manifest.view: "full"`)
|
|
85
|
+
That bundle is the FULL view (`manifest.view: "full"`) and holds the whole session transcript.
|
|
89
86
|
Never give it to a writer that must not see an earlier conclusion: the transcript holds the text of every file the session wrote, so a check for the earlier report FILE passes while its CONTENT is still readable.
|
|
90
87
|
|
|
91
88
|
Project the writer's copy instead:
|
|
@@ -94,17 +91,22 @@ Project the writer's copy instead:
|
|
|
94
91
|
traces bundle-view .traces/bundle --view evidence-only --out .traces/writer
|
|
95
92
|
```
|
|
96
93
|
|
|
97
|
-
|
|
98
|
-
It excludes the transcripts,
|
|
99
|
-
|
|
94
|
+
It carries `derived/session-index.json`, `derived/evidence.jsonl`, and the structured `ledger/` records.
|
|
95
|
+
It excludes the transcripts, report, spans, and every prose ledger file by name, with the rules and hashes in `manifest.excluded`.
|
|
96
|
+
It also drops any carried file that repeats an 8-word run of prose from an excluded one.
|
|
100
97
|
`manifest.view` names which copy you hold.
|
|
101
98
|
|
|
99
|
+
## Ask your own question
|
|
100
|
+
|
|
101
|
+
`traces ask --last 1 --question "<text>" --budget 2 --dir .traces/ask` keeps the engine's
|
|
102
|
+
prose answer, resolves every `trace://` citation in it, and spends model calls to do it.
|
|
103
|
+
|
|
102
104
|
## Report
|
|
103
105
|
|
|
104
106
|
- State the source (`--otlp <path>` or the harness), selected task boundary, session and span counts, and integrity warnings.
|
|
105
107
|
- State which capabilities the trace could not support, and name the analyses that reported nothing because of it.
|
|
106
108
|
- Cite each finding with its exact `trace://` reference.
|
|
107
|
-
- Mark missing outcome, cost, token, skill, or relationship data
|
|
109
|
+
- Mark missing outcome, cost, token, skill, or relationship data unknown.
|
|
108
110
|
- Do not infer task success from a completion message.
|
|
109
111
|
- Do not infer skill use from reading a `SKILL.md`.
|
|
110
112
|
- Do not upload traces unless explicitly requested.
|