@tangle-network/traces 0.23.1 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +61 -1078
- package/dist/analyst-model-call-JVUOYZ5V.js +20 -0
- package/dist/ask-XLS3VANY.js +39 -0
- package/dist/check-YF6ZHPGG.js +322 -0
- package/dist/chunk-3I4PJ5Q7.js +25 -0
- package/dist/chunk-3T62R7ON.js +5695 -0
- package/dist/chunk-4VKPN432.js +138 -0
- package/dist/chunk-6CLC7DKP.js +492 -0
- package/dist/chunk-7CBV5RJ2.js +535 -0
- package/dist/chunk-B62DOFGI.js +273 -0
- package/dist/chunk-CL3OQOWX.js +407 -0
- package/dist/chunk-DLGAKCQJ.js +782 -0
- package/dist/chunk-DU534HAX.js +263 -0
- package/dist/chunk-GLUDGUGJ.js +232 -0
- package/dist/chunk-J2PM5SVN.js +503 -0
- package/dist/chunk-M3W6TZHQ.js +381 -0
- package/dist/chunk-NH7HCDLB.js +759 -0
- package/dist/chunk-NZ5Y7UEL.js +1217 -0
- package/dist/chunk-O52XK7OB.js +19 -0
- package/dist/chunk-PYH3PL3N.js +62 -0
- package/dist/chunk-RZL5GAWH.js +60 -0
- package/dist/chunk-TR6O4S4T.js +208 -0
- package/dist/chunk-TW43MR5D.js +259 -0
- package/dist/chunk-TYLFUX42.js +228 -0
- package/dist/chunk-UOLRCQHC.js +120 -0
- package/dist/chunk-UVE5ANVV.js +3921 -0
- package/dist/chunk-V2OCGX57.js +53 -0
- package/dist/cli.js +2086 -2050
- package/dist/diff-ZZ6KYBPP.js +16 -0
- package/dist/improvement-OJQ6365E.js +25 -0
- package/dist/index.d.ts +1235 -4357
- package/dist/index.js +53 -1204
- package/dist/live-Y64AJ6MA.js +14 -0
- package/dist/mcp-ZPGTJQYQ.js +85 -0
- package/dist/report-JAMXRTCW.js +338 -0
- package/dist/run-watch-PLWTIRSV.js +649 -0
- package/dist/supervisor-report-RMPLLBGZ.js +29 -0
- package/dist/supervisor-run-context-WO6VOTGV.js +30 -0
- package/dist/upload-R6NMKSGQ.js +20 -0
- package/dist/verify-safe-C5ZXZOZ2.js +8 -0
- package/docs/analyze.md +76 -0
- package/docs/ask.md +88 -0
- package/docs/bundle.md +40 -0
- package/docs/check.md +77 -0
- package/docs/convert.md +44 -0
- package/docs/develop.md +32 -0
- package/docs/diff.md +38 -0
- package/docs/facts.md +114 -0
- package/docs/mcp.md +36 -0
- package/docs/migrate-1.0.md +33 -0
- package/docs/parsers.md +66 -0
- package/docs/release-1.0.0.md +147 -0
- package/docs/sdk.md +77 -0
- package/docs/select.md +70 -0
- package/docs/skills.md +46 -0
- package/docs/upload.md +47 -0
- package/docs/watch-jsonl.v1.example.jsonl +5 -0
- package/docs/watch.md +66 -0
- package/examples/README.md +9 -0
- package/examples/custom-analyst.ts +65 -74
- package/install.sh +22 -2
- package/package.json +8 -7
- package/skills/inspect-agent-traces/SKILL.md +47 -110
- package/dist/chunk-7DVZCCZI.js +0 -20983
- package/docs/replay-verify.md +0 -123
- package/docs/stream-findings.svg +0 -39
- package/docs/trace-analysts.md +0 -643
- package/examples/custom-backend.ts +0 -21
- package/examples/external-engines.ts +0 -62
- package/examples/improvement-config.mjs +0 -63
- package/examples/observe-and-alert.ts +0 -19
- package/examples/register-harness.ts +0 -26
- package/skills/build-trace-analyst/SKILL.md +0 -76
- package/skills/build-trace-analyst/agents/openai.yaml +0 -4
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Migrate to 1.0
|
|
2
|
+
|
|
3
|
+
[traces](../README.md#docs) › Migrate to 1.0
|
|
4
|
+
|
|
5
|
+
1.0 has 12 commands and 24 options. Every removed command and flag exits 2 and prints its replacement, with your argv rewritten where the rewrite is mechanical; `traces help moved` prints the whole table. 0.23.1 stays on npm: pin `@tangle-network/traces@0.23.1` to go back.
|
|
6
|
+
|
|
7
|
+
- **Commands.** `investigate` and `improve` are `analyze` (`--out DIR`). `validate` is `check --contract conformance`, and `verify-safe` is `check --share-safe`. `stream` is `watch --format jsonl`, with the same `traces.stream.*` v1 envelope minus batch events. `export` is `convert`. `index` is `list --format json`. `evidence` is `facts --format json`. `bundle-view` is `facts --no-content`. `inspect` is `analyze`. The CodeTraceBench and replay commands are removed; use `npx --yes @tangle-network/traces@0.23.1 <command>`.
|
|
8
|
+
- **Refs replace flags.** `--session`, `--otlp`, `--supervisor-run-dir` and `--source-bundle` become a positional ref. Ids, unique 8+ character prefixes or suffixes, files, directories and store roots are all refs ([refs and selection](select.md)).
|
|
9
|
+
- **Selection.**
|
|
10
|
+
- `--harness` defaults to every harness, and `--all` is gone.
|
|
11
|
+
- A reading command needs a ref, `--current`, `--last` or `--since`.
|
|
12
|
+
- `--last` is by mtime and never selects your own session.
|
|
13
|
+
- `--current` works in Claude Code too; in a Codex session started from Claude Code, the Codex session wins.
|
|
14
|
+
- Children are always included; the named session is `sessions[0]`.
|
|
15
|
+
- `upload` needs a ref or `--since`.
|
|
16
|
+
- **Output.** Text is the default, including for `facts`, so pass `--format json` to parse. `--dir`, `--otlp-out` and `--evidence` are `--out`. `--junit F` is `--format junit > F`. Nothing is written outside `--out` and `$XDG_STATE_HOME/traces`.
|
|
17
|
+
- **Errors and exit codes:** 0 yes, 1 no, 2 could not answer. check's 3 and 4, diff's 2 and 3, and verify-safe's UNKNOWN 2 all become 2. With `--format json|jsonl`, the error is one JSON object on stderr with `code`; stdout stays empty.
|
|
18
|
+
- **check needs a gate.** `check <ref>` alone exits 2. Use `--contract <file>`, `--contract conformance` or `--share-safe`. `--known-secret-env` is unchanged.
|
|
19
|
+
- **Model use.** Only `ask` calls a model; `analyze --llm` is gone. `ask` requires `--budget`, caps each question at min(budget, $2), and defaults to 1800 s and concurrency 4. A partial run writes `answers.json` and exits 2 with `code: partial`. Without `TRACES_PYTHON`, the bridge runs through `uvx`.
|
|
20
|
+
- **SDK.** 25 values and 25 types ([sdk](sdk.md)). Import agent-eval and agent-trace-contract names from their owners. `watchSessions` is now `streamSessions`. `runPipelines` is gone; `analyze --format json` carries its measures. `buildTraceFindingPacket` is unchanged.
|
|
21
|
+
- **Harnesses.** gemini, qwen, amp and copilot are removed; pin 0.23.1 if you read them. `codex-exec` stays.
|
|
22
|
+
- **mcp.** Unchanged except that its selection is refs and selection flags.
|
|
23
|
+
- **facts JSON.** `measurementVersion` is 5. Fields are added: `reply`, `interrupted`, `signals`, `author`, `cost`, `matches` and the waste measures. `tokens.input` is the whole prompt, cache included, on every harness. n = 1 distributions are removed. Every existing value path is unchanged.
|
|
24
|
+
- **Install.** `install.sh` refuses to overwrite a `traces` that is not this package's (Homebrew ships an unrelated one) and names it.
|
|
25
|
+
|
|
26
|
+
| 0.x argv (after `traces`) | 1.0 |
|
|
27
|
+
|---|---|
|
|
28
|
+
| `investigate --session X` | `traces analyze X` |
|
|
29
|
+
| `facts --harness codex --session P` | `traces facts P --harness codex --format json` |
|
|
30
|
+
| `convert --format sandbox-events --otlp-out f x.json` | `traces convert x.json --out f` |
|
|
31
|
+
| `stream --all --mode findings` | `traces watch --format jsonl` |
|
|
32
|
+
| `validate f` · `verify-safe p` | `traces check f --contract conformance` · `traces check p --share-safe` |
|
|
33
|
+
| `improve X --dir d --llm` | `traces analyze X --out d`, then `traces ask X --question … --budget 1` |
|
package/docs/parsers.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Parsers outside traces
|
|
2
|
+
|
|
3
|
+
[traces](../README.md#docs) › Parsers outside traces
|
|
4
|
+
|
|
5
|
+
The hill: agents and operators answer every question about an agent session with `traces` and never write a transcript parser. This census lists every hand-written session parser outside traces, the command that replaces it, and the step that deletes it. A consumer PR deletes a fork only when its row in [`bench/parity/forks.json`](../bench/parity/forks.json) passes its tolerance.
|
|
6
|
+
|
|
7
|
+
## How it ran (2026-10-17)
|
|
8
|
+
|
|
9
|
+
On each default branch after `git fetch`: `git grep -lE 'tool_use|response_item|event_msg' origin/<default> -- . ':!*test*' ':!*fixture*' ':!docs/*' ':!*.md' ':!*.json' ':!*.jsonl'`, then every hit read and classified. Branches: dotfiles `main` b9049ca, tangle-tools `main` 1cf9239, agent-dev-container `develop` ec55d0b319, discovery-lab `master` 8fbaa456c, agent-eval `main` 16307dc8, agent-runtime `main` 89b09c8b, agent-sdk `main` 08bf6e8, trustgate `main` f7f91bd (no hits).
|
|
10
|
+
`have claude project session parser` searched 198 repositories at their default branch and found nothing the grep missed. Two parsers (11, 12) read Kimi, Pi and OTLP files and match no pattern; they are listed by name.
|
|
11
|
+
80 hits: 33 parsers and 47 that are not. With the 2 found by name: **35 parsers today, 26 after the 1.0 consumer PRs (steps 22 to 30), 22 after 1.1.**
|
|
12
|
+
|
|
13
|
+
## Parsers
|
|
14
|
+
|
|
15
|
+
| # | Repo: path | What it reads | Retired by | Release (step) |
|
|
16
|
+
|---|---|---|---|---|
|
|
17
|
+
| 1 | dotfiles: `claude/skills/reflect/scripts/self-audit` | rework and rule breaches in Claude Code and Codex files | counts: `facts` waste (U4); rules: `check --contract` (U16) | 1.0 (23); 1.1 |
|
|
18
|
+
| 2 | dotfiles: `claude/tools/halo-extract.mjs` | HALO profile measures | `facts` (U4) | 1.0 (23) |
|
|
19
|
+
| 3 | dotfiles: `claude/tools/wf-status` | Claude Code workflow runs | `watch` on workflow runs (U17) | 1.1 |
|
|
20
|
+
| 4 | dotfiles: `claude/tools/skill_ledger/core.py` | PRs, human turns, tokens per skill run | `facts <session_id>` (U2, U3, U4) | 1.0 (23) |
|
|
21
|
+
| 5 | dotfiles: `claude/skills/voice-ledger/scripts/multi_extract.py` | every harness's turns | `facts --format jsonl` (U2, U3) | 1.0 (23) |
|
|
22
|
+
| 6 | dotfiles: `claude/analysis/wait-loss/scan_waits.py` | gaps between records | `facts` stalls (U4) | 1.x |
|
|
23
|
+
| 7 | dotfiles: `claude/analysis/wait-loss/loss.py` | wait loss per session | `facts` stalls (U4) | 1.x |
|
|
24
|
+
| 8 | tangle-tools: `trace-insights/mine/analyst/extract.mjs` | turns for the meta-analyst | `facts --format jsonl` (U3) | 1.0 (25) |
|
|
25
|
+
| 9 | tangle-tools: `trace-insights/trace_insights/claude_adapter.py` | Claude Code to spans | `convert` | 1.0 (25) |
|
|
26
|
+
| 10 | tangle-tools: `trace-insights/trace_insights/codex_adapter.py` | Codex to spans | `convert` | 1.0 (25) |
|
|
27
|
+
| 11 | tangle-tools: `trace-adapters/adapt.py` | Kimi and Pi wire logs to spans | `convert` | 1.0 (25) |
|
|
28
|
+
| 12 | tangle-tools: `trace-adapters/flatten.py` | collector OTLP files to rows | OTLP refs (U10) | 1.0 (25) |
|
|
29
|
+
| 13 | tangle-tools: `fleet/recurrence.py` | Codex homes: tokens, topics | `facts --format json` (U2, U3) | 1.0 (25) |
|
|
30
|
+
| 14 | tangle-tools: `dispatch-eval/extract.py` | dispatch rows | `facts` | 1.x |
|
|
31
|
+
| 15 | tangle-tools: `lineage/lineage_collect.py` | the lineage graph | `facts` | 1.x |
|
|
32
|
+
| 16 | tangle-tools: `trace-insights/mine/outputs/distill/harvest.py` | distilled turns | `facts --format jsonl` | 1.x |
|
|
33
|
+
| 17 | agent-runtime: `src/runtime/codex-rollout-store.ts` | Codex rollouts as spend receipts | one parser (U18) | 1.1 |
|
|
34
|
+
| 18 | tangle-tools: `fleet/deliver.py` | acknowledgements in agent records | `facts --match` | 1.x |
|
|
35
|
+
| 19 | tangle-tools: `fleet/messages.py` | delivery evidence in session records | `facts --match` | 1.x |
|
|
36
|
+
| 20 | tangle-tools: `lineage/lineage_outcomes.py` | Bash calls and results per session | `facts` | 1.x |
|
|
37
|
+
| 21 | tangle-tools: `lineage/lineage_timeline.py` | Claude Code and Codex timelines | `facts --format jsonl` | 1.x |
|
|
38
|
+
| 22 | tangle-tools: `lineage/lineage_workflows.py` | workflow lifecycle journals | `watch` on workflow runs (U17) | 1.1 |
|
|
39
|
+
| 23 | tangle-tools: `fleet/weather.py` | Codex rollouts per day | `facts --since` | 1.x |
|
|
40
|
+
| 24 | tangle-tools: `trace-archive/catalog.py` | tool and turn counts for the archive catalog | `list --format json`, `facts` over a store root | 1.x |
|
|
41
|
+
| 25 | discovery-lab: `runner/readout-transcripts.mjs` | each agent's steps from retained sessions | `facts --format jsonl` | 1.x |
|
|
42
|
+
| 26 | discovery-lab: `runner/skill-usage.mjs` | skill use in retained sessions | `facts` `measurements.skills` | 1.x |
|
|
43
|
+
| 27 | discovery-lab: `runner/scorecard.mjs` | tool calls and tokens from run streams and archives | `convert`, `facts` | 1.x |
|
|
44
|
+
| 28 | discovery-lab: `tools/trace-assess/science.mjs` | its assessor's required Bash calls | `check --contract` | 1.x |
|
|
45
|
+
| 29 | discovery-lab: `tools/trace-assess/pdf-source.mjs` | its assessor's required Read call | `check --contract` | 1.x |
|
|
46
|
+
| 30 | discovery-lab: `reports/fleet-observability/coverage.mjs` | Task spawns in run records | `facts` subagents | 1.x |
|
|
47
|
+
| 31 | discovery-lab: `experiments/wait-cost-smoke/measure.mjs` | a director's per-turn usage | `facts` tokens and stalls | 1.x |
|
|
48
|
+
| 32 | agent-dev-container: `products/sandbox/evals/src/local-agent-sessions.ts` | local Claude Code and Codex sessions for evals | `convert`, `facts` | 1.x |
|
|
49
|
+
| 33 | agent-eval: `src/contract/intake/code-agent-session.ts` | Claude Code and Codex records for contracts | harness-sessions, as U18 does for traces | 1.x |
|
|
50
|
+
| 34 | agent-eval: `src/contract/intake/code-agent-observation.ts` | the same, as observations | harness-sessions | 1.x |
|
|
51
|
+
| 35 | agent-eval: `src/supervisor-run/claude-code-reader.ts` | Claude Code supervision trees (U17 extends it) | harness-sessions | 1.x |
|
|
52
|
+
|
|
53
|
+
The 1.0 consumer PRs retire 2, 4, 5 and 8 to 13 (9 parsers) and the counting half of 1: 35 − 9 = 26. 1.1 retires 1 (rules), 3, 17 and 22: 22 remain, each a 1.x consumer PR for its owner after its parity row passes.
|
|
54
|
+
Against the spec's expectation (23 today, 8 after 1.0, 5 after 1.1): this census read seven repositories instead of two, and found 13 parsers no step yet retires (23 to 35), and 5 of the 6 candidates are parsers (18 to 22).
|
|
55
|
+
|
|
56
|
+
## Read and not a parser
|
|
57
|
+
|
|
58
|
+
- dotfiles `claude/hooks/secret_scan.py`: a PostToolUse hook's payload, not a store. tangle-tools `fleet/research_observer.py`: the output of its own `claude -p` runs.
|
|
59
|
+
- discovery-lab `tools/profile-ablation-report.mjs` (a comment), `reports/fleet-understanding-20261003/index.html` (a report page), `research/raw/sources/disco-research-*.txt` (source text), `prereg/factory-portfolio-20260929/sources/phony/…/{metrics,scorecard}.ts` (a frozen copy of another product), `reports/research-frontier-continuation-20260909/…/capture_usage.py` (a script inside frozen report evidence).
|
|
60
|
+
- agent-dev-container: the 18 files under `packages/sdk-provider-*`, `packages/sdk-telemetry/src/sse-parser.ts`, four under `products/sandbox/sdk/src`, `products/sandbox/cli/src/commands/agent-connect.ts`, `products/platform/api/src/lib/openai-compatible-backend.ts`, `products/intelligence/api/src/routes/proxy.ts` and `products/sandbox/evals/src/auto-optimization.ts` drive a harness's live stream or a model API's `tool_use` blocks.
|
|
61
|
+
- agent-eval `examples/_shared/openai-compatible-owner.ts` and `src/campaign/external-optimizer-anthropic.ts`; agent-runtime `scripts/lib/full-capacity-acceptance.mjs`, `src/runtime/profile-chat-client.ts`, `src/runtime/sandbox-events.ts` and `src/runtime/supervise/trace-source.ts`; agent-sdk `packages/agent-core` (3 files): live streams and model API blocks. agent-sdk `packages/harness-sessions/src/formats/*`: the shared reader itself.
|
|
62
|
+
|
|
63
|
+
## Definition differences
|
|
64
|
+
|
|
65
|
+
- **Tokens of a session with no usage record.** `fleet/recurrence.py` reports 0; `traces facts` reports `tokens.input: null` with its reason, because unknown is null, never 0. On the parity row of 2026-10-08..14 this is the whole disagreement (14 of 16 files equal; 2 Claude Code sessions with no usage record). It is a definition, not a defect: recurrence should read `null` as unknown when it moves to `facts` (step 25).
|
|
66
|
+
- **Commands in a Codex code-mode script.** `self-audit` counts each `cmd` inside an `exec` script; `facts` waste counts each tool call's script. Search rows (`--match`) split scripts into single commands.
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# traces 1.0.0
|
|
2
|
+
|
|
3
|
+
traces answers every question about an agent session with one command: 12 commands and 24 options, down from 27 commands, each with its own `--help`, one page, and a test that spawns it.
|
|
4
|
+
`facts` states what happened for $0, `analyze` ranks what went wrong for $0, `check` is the one gate, `watch` the one live view, and `ask` the only command that spends.
|
|
5
|
+
Selection finds every session on every harness (every Codex home, ids by 8-character prefix or suffix, `--current` in Claude Code too), `facts` owns every number with tokens and cost checked against the raw records, `--match` searches 30 days of sessions through an incremental index, nothing is written outside `--out` and `$XDG_STATE_HOME/traces`, and `install.sh` refuses to overwrite another program's `traces`.
|
|
6
|
+
The `inspect-agent-traces` skill maps each question an agent asks to the command that answers it.
|
|
7
|
+
|
|
8
|
+
## Migration
|
|
9
|
+
|
|
10
|
+
- **Commands.** `investigate` and `improve` are `analyze` (`--out DIR`). `validate` is `check --contract conformance`, and `verify-safe` is `check --share-safe`. `stream` is `watch --format jsonl`, with the same `traces.stream.*` v1 envelope minus batch events. `export` is `convert`. `index` is `list --format json`. `evidence` is `facts --format json`. `bundle-view` is `facts --no-content`. `inspect` is `analyze`. The CodeTraceBench and replay commands are removed; use `npx --yes @tangle-network/traces@0.23.1 <command>`.
|
|
11
|
+
- **Refs replace flags.** `--session`, `--otlp`, `--supervisor-run-dir` and `--source-bundle` become a positional ref. Ids, unique 8+ character prefixes or suffixes, files, directories and store roots are all refs ([refs and selection](select.md)).
|
|
12
|
+
- **Selection.**
|
|
13
|
+
- `--harness` defaults to every harness, and `--all` is gone.
|
|
14
|
+
- A reading command needs a ref, `--current`, `--last` or `--since`.
|
|
15
|
+
- `--last` is by mtime and never selects your own session.
|
|
16
|
+
- `--current` works in Claude Code too; in a Codex session started from Claude Code, the Codex session wins.
|
|
17
|
+
- Children are always included; the named session is `sessions[0]`.
|
|
18
|
+
- `upload` needs a ref or `--since`.
|
|
19
|
+
- **Output.** Text is the default, including for `facts`, so pass `--format json` to parse. `--dir`, `--otlp-out` and `--evidence` are `--out`. `--junit F` is `--format junit > F`. Nothing is written outside `--out` and `$XDG_STATE_HOME/traces`.
|
|
20
|
+
- **Errors and exit codes:** 0 yes, 1 no, 2 could not answer. check's 3 and 4, diff's 2 and 3, and verify-safe's UNKNOWN 2 all become 2. With `--format json|jsonl`, the error is one JSON object on stderr with `code`; stdout stays empty.
|
|
21
|
+
- **check needs a gate.** `check <ref>` alone exits 2. Use `--contract <file>`, `--contract conformance` or `--share-safe`. `--known-secret-env` is unchanged.
|
|
22
|
+
- **Model use.** Only `ask` calls a model; `analyze --llm` is gone. `ask` requires `--budget`, caps each question at min(budget, $2), and defaults to 1800 s and concurrency 4. A partial run writes `answers.json` and exits 2 with `code: partial`. Without `TRACES_PYTHON`, the bridge runs through `uvx`.
|
|
23
|
+
- **SDK.** 25 values and 25 types ([sdk](sdk.md)). Import agent-eval and agent-trace-contract names from their owners. `watchSessions` is now `streamSessions`. `runPipelines` is gone; `analyze --format json` carries its measures. `buildTraceFindingPacket` is unchanged.
|
|
24
|
+
- **Harnesses.** gemini, qwen, amp and copilot are removed; pin 0.23.1 if you read them. `codex-exec` stays.
|
|
25
|
+
- **mcp.** Unchanged except that its selection is refs and selection flags.
|
|
26
|
+
- **facts JSON.** `measurementVersion` is 5. Fields are added: `reply`, `interrupted`, `signals`, `author`, `cost`, `matches` and the waste measures. `tokens.input` is the whole prompt, cache included, on every harness. n = 1 distributions are removed. Every existing value path is unchanged.
|
|
27
|
+
- **Install.** `install.sh` refuses to overwrite a `traces` that is not this package's (Homebrew ships an unrelated one) and names it.
|
|
28
|
+
|
|
29
|
+
The full table: [docs/migrate-1.0.md](migrate-1.0.md), or `traces help moved`.
|
|
30
|
+
|
|
31
|
+
## A week of one host
|
|
32
|
+
|
|
33
|
+
`analyze`, `ask` and `mcp` read a `--since` sweep one session at a time and keep only what their reports need, so a week of a busy host is never held as spans.
|
|
34
|
+
On drew-gtr-pro, `traces analyze --since 7d --format json` over every harness (4,079 sessions, 1.54 million spans) exits 0 at the default heap with a peak RSS of 6.5 GB in 18 minutes, under `nice -n 10` at load 40 to 65. Before, it ran out of V8 heap at the default ~4.5 GB; with a 24 GB heap it held 7.5 GB and more, and skipped agent-eval's behavioral analyst (`4063 traces exceed maxTraces=1000`). Every analyst now reads every selected trace.
|
|
35
|
+
For small selections the output is unchanged: analyze JSON before and after differs only in the new fields.
|
|
36
|
+
|
|
37
|
+
`facts.measurements` gains three measures, and `analyze` pools them in `measures.operating`:
|
|
38
|
+
|
|
39
|
+
- `modelCalls`: the context each model call read (input + cache read + cache write), and the model's latency after a tool result by 100k-token context band. Claude Code on the GTR week: mean 366k, p50 298k tokens; reply p50 3.4 s at 0-100k and 5.3 s at 900k+.
|
|
40
|
+
- `guardRefusals`: tool calls a hook refused, by guard. GTR week: 1.7 per 1,000 tool calls, process-guard most.
|
|
41
|
+
- `limitStop`: the usage or weekly limit notice a session stopped on. GTR week: 81 of 129 terminal failures. Mac week: 11 of 12.
|
|
42
|
+
|
|
43
|
+
A Codex task that completes with an error (`You’ve hit your usage limit`) now ends its run in error, where it read as a success.
|
|
44
|
+
|
|
45
|
+
## Benchmark: traces against an agent reading the raw files (U11)
|
|
46
|
+
|
|
47
|
+
`bench/audit` asks 19 questions about three synthetic sessions, scored by exact match. Each model arm is one `claude -p` agent per question, model `sonnet` (served as `claude-sonnet-5-5`), $0.25 per question, tools `Bash`, `Read`, `Grep`, `Glob`; cost is the harness's own `total_cost_usd` at list price. The rule: the traces arm answers at least as many questions correctly as the raw-files arm, at lower cost.
|
|
48
|
+
|
|
49
|
+
**Step 20 (2026-10-17, Claude Code 2.1.301, traces on PATH, no skill):**
|
|
50
|
+
|
|
51
|
+
| Run | raw-files correct / partial / wrong | cost | traces correct / partial / wrong | cost |
|
|
52
|
+
|---|---|---|---|---|
|
|
53
|
+
| 1 | 17 / 1 / 1 | $0.6712 | 16 / 2 / 1 | $0.6069 |
|
|
54
|
+
| 2 | 17 / 1 / 1 | $0.6951 | 17 / 1 / 1 | $0.6693 |
|
|
55
|
+
| 3 | 16 / 2 / 1 | $0.7051 | 16 / 3 / 0 | $0.6974 |
|
|
56
|
+
| **Total** | **50 / 57** | **$2.0714** | **49 / 57** | **$1.9736** |
|
|
57
|
+
|
|
58
|
+
**Step 21 rerun (2026-10-17, the same questions, model and budget; the traces arm loads the skill as installed):**
|
|
59
|
+
|
|
60
|
+
| Run | raw-files correct / partial / wrong | cost | traces correct / partial / wrong | cost | traces arm: skill loaded / ran traces / only `--help` / no traces |
|
|
61
|
+
|---|---|---|---|---|---|
|
|
62
|
+
| 1 | 17 / 1 / 1 | $0.6515 | 16 / 2 / 1 | $0.7436 | 4 / 4 / 5 / 10 |
|
|
63
|
+
| 2 | 17 / 1 / 1 | $0.6574 | 17 / 1 / 1 | $0.9749 | 5 / 6 / 4 / 9 |
|
|
64
|
+
| 3 | 17 / 1 / 1 | $0.6789 | 17 / 2 / 0 | $0.9876 | 4 / 5 / 4 / 10 |
|
|
65
|
+
| **Total** | **51 / 57** | **$1.9878** | **50 / 57** | **$2.7061** | 13 / 15 / 13 / 29 of 57 |
|
|
66
|
+
|
|
67
|
+
The rule is **not met**: the traces arm answered one question fewer, at 36% more cost. The question it loses is `op.role` in all three runs: it counts runs a `labctl run` inside a code-mode script loop started, which no facts field records. Where the agent loaded the skill and ran `traces facts`, the answer cost more than a targeted `grep` on these small files: the operator session's JSON sheet is about 60 KB. A first traces run with Claude Code's 19 bundled skills also listed in the Skill tool scored 15 / 3 / 1 at $1.1506; the arm now turns them off (`bench/audit/README.md`).
|
|
68
|
+
|
|
69
|
+
**The facts arm** (`traces facts <ref> --format json` mapped by fixed rules, $0, in `pnpm test`): 12 of 19 on the canonical wordings and 12 of 19 held out since #215, which answers `op.runs` and `op.role` (10 of 19 at step 21, 5 before step 20's fixes).
|
|
70
|
+
|
|
71
|
+
**The large-session arm** (reported apart; it does not decide the rule): one synthetic Codex session of 50,124 spans (27 MB), four questions, one run per arm.
|
|
72
|
+
|
|
73
|
+
| Arm | correct / partial / wrong | cost | note |
|
|
74
|
+
|---|---|---|---|
|
|
75
|
+
| facts | 4 / 0 / 0 | $0 | 8 s on a laptop |
|
|
76
|
+
| raw-files | 3 / 1 / 0 | $0.1225 | counted `write_stdin` poll exits as command exits (5,252 against 4,297) |
|
|
77
|
+
| traces | 3 / 1 / 0 | $0.1522 | never loaded the skill; ran `traces --help` once; the same miss |
|
|
78
|
+
|
|
79
|
+
**U11 round (2026-10-17, after #215: code-mode processes by program, the `commands` sheet line, the facts-first skill).** The same questions, model and budget; at most 2 `claude -p` at a time on this Mac, each started at load1 ≤ 20 and free memory ≥ 30%.
|
|
80
|
+
|
|
81
|
+
| Run | raw-files correct / partial / wrong | cost | traces correct / partial / wrong | cost | traces arm: skill loaded / ran traces / only `--help` / no traces |
|
|
82
|
+
|---|---|---|---|---|---|
|
|
83
|
+
| 1, step 21's arm settings, before #215's two leak fixes | 17 / 1 / 1 | $0.6847 | 19 / 0 / 0 | $0.9958 | 14 / 15 / 1 / 3 |
|
|
84
|
+
| 1, account skills off | 17 / 1 / 1 | $0.6317 | 17 / 1 / 1 | $0.7530 | 15 / 15 / 0 / 4 |
|
|
85
|
+
| 2, account skills off | 14 / 4 / 1 | $0.6228 | 18 / 1 / 0 | $0.6707 | 15 / 15 / 0 / 4 |
|
|
86
|
+
| 3, account skills off | 17 / 2 / 0 | $0.6615 | 19 / 0 / 0 | $0.6826 | 16 / 16 / 0 / 3 |
|
|
87
|
+
| **Total, account skills off** | **48 / 57** | **$1.9160** | **54 / 57** | **$2.1063** | 46 / 46 / 0 / 11 of 57 |
|
|
88
|
+
|
|
89
|
+
The rule is **not met**: the traces arm answers six more questions correctly, but at 9.9% more cost. Under step 21's settings, Claude Code loaded the login account's claude.ai skills into the traces arm's Skill tool after the first request (24 `commands_changed` events in 19 questions), so each question's second turn rewrote the prompt cache: `op.tokens`, the same two calls in both arms, cost $0.0171 in raw-files and $0.0310 in traces. The arm now sets `syncClaudeAiSkills: false`: 0 such events in 57 questions, and the traces arm's median second-turn cache write fell from 4,037 to about 1,378 tokens (raw-files about 585; the rest is the skill's own text). What still costs more: `claude.bash` (+$0.097 over three runs; facts counts only the main session's Bash calls), `op.runs` (+$0.090; `--match` rows carry no exit status, so agents read the raw records), and `claude.tasks` (+$0.053; no facts field holds `subagent_type`).
|
|
90
|
+
|
|
91
|
+
Large session, account skills off, three paired runs: traces 12 / 12 at $0.3579 (4 / 4 each run), raw-files 9 / 12 at $0.5494 (3 / 4 each run, counting `write_stdin` poll exits as command exits, 5,252 against 4,297); facts 4 / 4 at $0.
|
|
92
|
+
|
|
93
|
+
**U11 rerun after #217 (2026-10-18, traces `e8c0d35`, on GTR).** The same questions, model (`claude -p` 2.1.302, `sonnet`), budget and arm settings; at most 2 `claude -p` at a time, each started at load1 ≤ 40 and 16 GiB available. The work trees sat under `/tmp`: under the login's home, both arms' agents loaded that home's skills, `AGENTS.md` and auto-memory.
|
|
94
|
+
|
|
95
|
+
| Run | raw-files correct / partial / wrong | cost | traces correct / partial / wrong | cost | traces arm: skill loaded / ran traces / only `--help` / no traces |
|
|
96
|
+
|---|---|---|---|---|---|
|
|
97
|
+
| 1 | 15 / 1 / 3 | $0.5731 | 19 / 0 / 0 | $0.6029 | 12 / 12 / 0 / 7 |
|
|
98
|
+
| 2 | 17 / 1 / 1 | $0.6931 | 19 / 0 / 0 | $0.6996 | 11 / 11 / 0 / 8 |
|
|
99
|
+
| 3 | 16 / 2 / 1 | $0.7169 | 17 / 1 / 1 | $0.6828 | 11 / 11 / 1 / 7 |
|
|
100
|
+
| **Total** | **48 / 57** | **$1.9830** | **55 / 57** | **$1.9853** | 34 / 34 / 1 / 22 of 57 |
|
|
101
|
+
|
|
102
|
+
The rule is **not met**, by $0.0023 (0.1%). #217's facts closed the three gaps it named: over the three runs, `claude.bash` went from +$0.097 to −$0.004, `op.runs` from +$0.090 to −$0.052, and `claude.tasks` from +$0.053 to $0.000. What still costs more is answers that cite or quote a record. The agent reads the fact, then searches the session file for its line: `op.exit-codes` +$0.091 (no exit codes by command), `op.local-copy` +$0.066, `claude.first-bash-error` +$0.035, `op.large-output` +$0.033 (`--match` showed no output) and `op.last-human-turn` +$0.027. Large session, three pairs: traces 10 / 12 at $0.3784, raw-files 9 / 12 at $0.4097; both missed `large.commands` in most runs.
|
|
103
|
+
|
|
104
|
+
## Fork parity (step 20, this Mac, 2026-10-08 to 2026-10-14)
|
|
105
|
+
|
|
106
|
+
Each parser traces retires, scored against the command that replaces it; a consumer PR deletes a fork only at or above its tolerance (90%).
|
|
107
|
+
|
|
108
|
+
| Fork (deleted in step) | Agreement | Disagreements and their cause |
|
|
109
|
+
|---|---|---|
|
|
110
|
+
| voice-ledger `multi_extract.py`, Codex (23) | 75.8%, below | Codex goal-context injections the old parser counts (traces follows Codex's own context list), and question-tool answers and screenshot turns traces dropped; **fixed in 1.0.0**: on the session that held 23 of the 45, facts now reads 19 turns instead of 14 |
|
|
111
|
+
| trace-mine `extract.mjs` (25) | 93.2%, pass | Claude Code `Warmup` prompts (old only), typed slash commands (traces only) |
|
|
112
|
+
| reflect `self-audit`, counting half (23) | 93.8%, pass | one code-mode session: per-`cmd` against per-script counting |
|
|
113
|
+
| HALO `halo-extract.mjs` (23) | 43.8%, below | HALO counts a script-launched session's prompt as a person's turn |
|
|
114
|
+
| skill_ledger `_pr_events` (23) | 93.8%, pass | `gh-drew pr create` inside Codex code-mode scripts; **fixed in 1.0.0**: that session now reads all 6 creates (#127, #130, #131, #134, #135, #136) |
|
|
115
|
+
| recurrence `scan()` tokens (25) | 87.5%, below | 2 sessions with no usage record: recurrence 0, facts null with its reason; a definition, not a defect ([parsers](parsers.md)) |
|
|
116
|
+
| trace_insights adapters (25) | 100%, pass | none |
|
|
117
|
+
| trace-adapters `flatten.py` (25) | 100%, pass | none |
|
|
118
|
+
|
|
119
|
+
## Correction signals (U21, step 10)
|
|
120
|
+
|
|
121
|
+
On voice-ledger's 150 blind calibration turns (148 joined, 44 corrections), against trace-mine Opus labels:
|
|
122
|
+
|
|
123
|
+
| Arm | Recall [95% CI] | Precision [95% CI] | F1 |
|
|
124
|
+
|---|---|---|---|
|
|
125
|
+
| interrupt | 0.091 [0.04, 0.21] | 0.400 [0.17, 0.69] | 0.148 |
|
|
126
|
+
| rejected-tool | 0.068 [0.02, 0.18] | 0.750 [0.30, 0.95] | 0.125 |
|
|
127
|
+
| after-claim | 0.091 [0.04, 0.21] | 0.190 [0.08, 0.40] | 0.123 |
|
|
128
|
+
| union | 0.205 [0.11, 0.34] | 0.290 [0.16, 0.47] | 0.240 |
|
|
129
|
+
| lexical (0.23.1 baseline) | 0.341 [0.22, 0.49] | 0.577 [0.39, 0.74] | 0.429 |
|
|
130
|
+
| lexical-any | 0.591 [0.44, 0.72] | 0.553 [0.41, 0.69] | 0.571 |
|
|
131
|
+
|
|
132
|
+
The union does not beat the lexical baseline, so analyze section 7 is "Interrupts and rejections", and nothing calls these signals corrections.
|
|
133
|
+
|
|
134
|
+
## Finding precision (U7, step 13)
|
|
135
|
+
|
|
136
|
+
HIGH and MED findings on 10 real sessions (5 Claude Code, 5 Codex), each labelled by reading the evidence:
|
|
137
|
+
|
|
138
|
+
| Finding kind | Severity labelled | Findings | True | Precision | Shipped |
|
|
139
|
+
|---|---|---|---|---|---|
|
|
140
|
+
| `terminal-failure` | HIGH | 3 | 3 | 1.00 | HIGH |
|
|
141
|
+
| `completion-claim-without-verification` | MED | 9 | 0 | 0.00 | LOW in step 13, removed in step 14 |
|
|
142
|
+
| `no-self-verification` | MED | 1 | 0 | 0.00 | LOW |
|
|
143
|
+
| the other six kinds | | 0 | | not measured | unchanged |
|
|
144
|
+
|
|
145
|
+
## Parsers outside traces
|
|
146
|
+
|
|
147
|
+
[docs/parsers.md](parsers.md) re-ran the census on seven default branches: 35 hand-written session parsers today, 26 after the 1.0 consumer PRs (steps 22 to 30), 22 after 1.1. The spec expected 23, 8 and 5: the census found 13 more parsers (discovery-lab 7, tangle-tools 2, agent-dev-container 1, agent-eval 3) that no step retires yet.
|
package/docs/sdk.md
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Library (SDK)
|
|
2
|
+
|
|
3
|
+
[traces](../README.md#docs) › Library (SDK)
|
|
4
|
+
|
|
5
|
+
The SDK is 25 functions and 25 types. Each function is the engine of a `traces` command, or the analyst extension point (`analyzeSpans` with your own `registry`).
|
|
6
|
+
[`src/index.ts`](../src/index.ts) lists them, and `tests/sdk-surface.test.ts` fails when the built package exports any other name.
|
|
7
|
+
Import other packages' names from their owners: `AnalystRegistry`, `makeFinding`, `checkTraceContracts` and the redaction core from `@tangle-network/agent-eval`, and the span builders from `@tangle-network/agent-trace-contract`.
|
|
8
|
+
|
|
9
|
+
| Export | Signature | Command it serves |
|
|
10
|
+
|---|---|---|
|
|
11
|
+
| `scanSessions` | `(ScanOptions) → AsyncGenerator<ScannedSession>` | every command: locate, then parse, each harness session |
|
|
12
|
+
| `parseSession` | `(adapter, SessionRef, opts?) → Promise<OtlpSpan[]>` | one session's spans |
|
|
13
|
+
| `knownHarnesses` | `() → string[]` | the ids `--harness` takes |
|
|
14
|
+
| `readOtlpInput` | `(file or dir) → Promise<OtlpInput>` | read OTLP from any producer; never throws on a foreign trace |
|
|
15
|
+
| `writeOtlpFile` | `(spans, path) → Promise<string>` | `traces convert <ref> --out <file>` |
|
|
16
|
+
| `computeSessionFacts` | `(spans, opts?) → SessionFacts[]` | `traces facts`: one sheet per session |
|
|
17
|
+
| `buildSessionFactsReport` | `(spans, opts?) → SessionFactsReport` | `traces facts <ref> --format json` |
|
|
18
|
+
| `renderSessionFacts` | `(SessionFactsReport) → string` | `traces facts` text |
|
|
19
|
+
| `analyzeSpans` | `(spans, AnalyzeOptions?) → Promise<AnalyzeResult>` | agent-eval's deterministic analyst, or yours through `{ registry }`; no model call |
|
|
20
|
+
| `runTraceInvestigation` | `(TraceInvestigationOptions) → Promise<TraceInvestigationResult>` | `traces analyze <ref> --format json`: facts, finding packet, measures, conformance, `cannotAnswer` |
|
|
21
|
+
| `buildTraceFindingPacket` | `(BuildTraceFindingPacketOptions) → TraceFindingPacket` | `findings.json` under `traces analyze <ref> --out <dir>`; renders any `AnalystFinding[]` unchanged |
|
|
22
|
+
| `runTraceQuestions` | `(options) → Promise<TraceQuestionsResult>` | `traces ask`: concurrent questions under one cost ledger, citations checked |
|
|
23
|
+
| `analysisEngineFromEnv` | `({ model, maxCostUsd?, timeoutMs?, onPermanentFailure?, env? }) → engine` | the model engine `traces ask` runs |
|
|
24
|
+
| `conformanceOfSpans` | `(spans) → TraceValidation` | `traces check <ref> --contract conformance` |
|
|
25
|
+
| `diffRuns` | `(a, b) → Promise<RunDiffReport>` | `traces diff`: two trace files or `file#<trace id>` refs |
|
|
26
|
+
| `renderRunDiff` | `(RunDiffReport) → string` | `traces diff` text |
|
|
27
|
+
| `streamSessions` | `(options) → Promise<void>` | `traces watch --format jsonl`: session, finding and tick events (`TraceStreamEvent`) |
|
|
28
|
+
| `redactSpans` | `(spans, { profile?, knownSecrets? }) → { spans, report }` | redaction before `upload` |
|
|
29
|
+
| `assessSpans` | `(spans, profile?) → ShareSafetyVerdict` | the share verdict for spans |
|
|
30
|
+
| `verifySafe` | `(path, { knownSecrets? }) → Promise<VerifySafeResult>` | `traces check <path> --share-safe` |
|
|
31
|
+
| `planUpload` | `(opts?) → Promise<UploadPlan>` | `traces upload <ref> --dry-run` |
|
|
32
|
+
| `executeUpload` | `(UploadPlan, { backend? }) → Promise<result>` | `traces upload`, to Intelligence or your own `UploadBackend` |
|
|
33
|
+
| `assembleSessionBundle` | `({ adapter, ref, outDir }) → Promise<result>` | `traces bundle` |
|
|
34
|
+
| `verifySessionBundle` | `(dir) → Promise<verification>` | `traces bundle verify <dir>` |
|
|
35
|
+
| `tracesVersion` | `() → string` | `traces --version` |
|
|
36
|
+
|
|
37
|
+
**Types:** `OtlpSpan`, `SessionRef`, `ScanOptions`, `ScannedSession`, `HarnessTraceAdapter`, `OtlpInput`, `SessionFact`, `SessionFacts`, `SessionFactsReport`, `SessionTurnFact`, `AnalyzeOptions`, `AnalyzeResult`, `TraceInvestigationOptions`, `TraceInvestigationResult`, `TraceFindingPacket`, `BuildTraceFindingPacketOptions`, `TraceQuestion`, `TraceQuestionsResult`, `RunDiffReport`, `TraceStreamEvent`, `TraceLiveFinding`, `UploadPlan`, `UploadBackend`, `VerifySafeResult`, `SessionBundleManifest`.
|
|
38
|
+
An options type that is not listed is still in `dist/index.d.ts`; name it as `Parameters<typeof runTraceQuestions>[0]`.
|
|
39
|
+
|
|
40
|
+
```ts
|
|
41
|
+
import { analyzeSpans, readOtlpInput, streamSessions } from '@tangle-network/traces'
|
|
42
|
+
|
|
43
|
+
const input = await readOtlpInput('spans.otlp.jsonl') // any producer; never throws on a foreign trace
|
|
44
|
+
console.log(input.validation.capabilities)
|
|
45
|
+
|
|
46
|
+
const stop = new AbortController()
|
|
47
|
+
await streamSessions({ signal: stop.signal, onEvent: (event) => { if (event.event === 'finding') console.log(event.finding.claim) } })
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Write your own analyst
|
|
51
|
+
|
|
52
|
+
An analyst receives a paged trace store and returns typed findings; start with deterministic logic when the trace holds the facts.
|
|
53
|
+
|
|
54
|
+
```ts
|
|
55
|
+
import { buildDefaultAnalystRegistry, defineTraceAnalyst, makeFinding } from '@tangle-network/agent-eval/analyst'
|
|
56
|
+
import { analyzeSpans } from '@tangle-network/traces'
|
|
57
|
+
|
|
58
|
+
const failedTools = defineTraceAnalyst({
|
|
59
|
+
id: 'failed-tools',
|
|
60
|
+
description: 'Reports repeated tool failure signatures.',
|
|
61
|
+
async analyze(store) {
|
|
62
|
+
const overview = await store.getOverview({ has_errors: true })
|
|
63
|
+
return overview.error_clusters.map((cluster) => makeFinding({
|
|
64
|
+
analyst_id: 'failed-tools', area: 'tool-use', subject: cluster.signature,
|
|
65
|
+
claim: `${cluster.span_count} spans share this failure`, severity: cluster.span_count >= 3 ? 'high' : 'medium',
|
|
66
|
+
evidence_refs: [{ kind: 'span', uri: `trace://${cluster.exemplar_trace_ids[0]}/span/${cluster.exemplar_span_ids[0]}` }],
|
|
67
|
+
confidence: 1, id_basis: cluster.signature,
|
|
68
|
+
}))
|
|
69
|
+
},
|
|
70
|
+
})
|
|
71
|
+
const registry = buildDefaultAnalystRegistry()
|
|
72
|
+
registry.register(failedTools)
|
|
73
|
+
const { result } = await analyzeSpans(spans, { registry })
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
The runnable version is [`examples/custom-analyst.ts`](../examples/custom-analyst.ts) (`pnpm tsx examples/custom-analyst.ts --fixture` prints its one finding).
|
|
77
|
+
Do not rewrite policy from an analyst's prose alone: attach a measured outcome or reviewed feedback, promote the failure into an eval case, change one thing, and compare on fresh cases.
|
package/docs/select.md
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Refs, selection and harnesses
|
|
2
|
+
|
|
3
|
+
[traces](../README.md#docs) › Refs, selection and harnesses
|
|
4
|
+
|
|
5
|
+
Every reading command takes refs, or a selection, and the same flags choose sessions everywhere.
|
|
6
|
+
`traces list` finds a session: one row per session, newest first, across every harness, never the caller's own.
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
traces list --cwd . --last 5
|
|
10
|
+
traces facts 019a3c7e # a ref list printed
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
## Refs
|
|
14
|
+
|
|
15
|
+
A ref is sniffed, so no flag names its type:
|
|
16
|
+
|
|
17
|
+
- an existing path, read by its content: a harness session file (its store root decides the harness, else its first record), a Sandbox live stream (`"backend":"codex"` envelopes, read as `codex-exec`), a sandbox-events recording, contract OTLP-JSONL, standard OTLP/JSON (`resourceSpans`, one document or one envelope per line), `claude -p` stream-json, OpenInference JSONL, a directory of OTLP files, a bundle directory, a Runtime run directory, or a **store root**: a directory that holds harness stores the way a home does, such as a trace archive's `/mnt/traces/<host>/home`;
|
|
18
|
+
- otherwise a session id, or a unique prefix or suffix of 8 or more characters, looked up across every harness. An ambiguous ref exits 2 with `code: ambiguous` and lists the candidates.
|
|
19
|
+
|
|
20
|
+
`file#<traceId>` and `file#branch=<id>` name one run of several. `--harness` with a file ref forces that reader.
|
|
21
|
+
|
|
22
|
+
## Selection
|
|
23
|
+
|
|
24
|
+
A reading command needs a ref, `--current`, `--last N` or `--since T`; `--cwd`, `--harness`, `--until` and `--latest-turn` only narrow. `list` defaults to `--last 20`, `watch` to `--since 30m`, `skills` to `--since 30d`.
|
|
25
|
+
|
|
26
|
+
- `--current` is the caller's own session: `CODEX_THREAD_ID` (Codex) wins, else `CLAUDE_CODE_SESSION_ID`. On Pi, OpenCode or Kimi, `traces list --cwd . --last 1` names it.
|
|
27
|
+
- `--last N` is the N session files with the newest mtime, never the caller's own session.
|
|
28
|
+
- `--since T` / `--until T` select every session with a record in the window (`30m`, `2h`, `7d`, or an ISO time).
|
|
29
|
+
- `--cwd DIR` matches the cwd the session recorded, on a path boundary; a live worktree of the same repo; or, for Codex, the recorded repo of a removed worktree.
|
|
30
|
+
- `--latest-turn` keeps a resumed Codex or Claude Code session to its latest task and the workers spawned from it.
|
|
31
|
+
|
|
32
|
+
A named session brings its spawned children, up to 100: it is `sessions[0]`, each child follows with `parentSessionId`, and no number of the named session includes a child's.
|
|
33
|
+
In a window, `--last` or a machine-wide selection, a session that cannot be read is skipped and named in `selection.unreadSessions`; a named ref that cannot be read exits 2 with `code: unreadable`.
|
|
34
|
+
|
|
35
|
+
## Harnesses
|
|
36
|
+
|
|
37
|
+
| Harness | Reads from |
|
|
38
|
+
|---|---|
|
|
39
|
+
| `claude-code` | `~/.claude/projects/<cwd>/*.jsonl` and subagent sidechains, or a `claude -p --output-format stream-json` file |
|
|
40
|
+
| `codex` | every Codex home: `~/.codex` and `~/.codex-*` siblings, `$CODEX_HOME`, and `~/.config/agent-accounts/codex/<account>`, each with `sessions/` and `archived_sessions/` |
|
|
41
|
+
| `codex-exec` | a `codex exec --json` file or a Sandbox live stream (a file ref) |
|
|
42
|
+
| `opencode` | `~/.local/share/opencode/opencode.db`, read in place, and older `storage/` JSON sessions |
|
|
43
|
+
| `pi`, `kimi`, `factory` | their session stores under the home |
|
|
44
|
+
|
|
45
|
+
Native stores are read through [`@tangle-network/harness-sessions`](https://github.com/tangle-network/agent-sdk/tree/main/packages/harness-sessions); `--latest-turn` uses traces' own Claude Code and Codex readers until one parser lands (1.1), and `tests/dual-path.test.ts` holds the two equal.
|
|
46
|
+
Claude Code subagent files fold into their parent trace; Codex workers are sessions of their own, joined by the session id or task path a child records.
|
|
47
|
+
Unknown is null with its reason, never 0: Kimi's wire log has no served model, so its model stays unknown.
|
|
48
|
+
|
|
49
|
+
## `traces list --help`
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
Usage: traces list [<store-root>] [--last N | --since T] [--cwd DIR] [--harness ID] [--format text|json]
|
|
53
|
+
Find a session: one row per session, newest first, across every harness, never the caller's own.
|
|
54
|
+
|
|
55
|
+
--last N the N newest sessions (default 20)
|
|
56
|
+
--since T every session with a record since T: 30m, 2h, 7d or an ISO time
|
|
57
|
+
--cwd DIR sessions recorded in DIR or under it, or in a worktree of its repo
|
|
58
|
+
--harness ID only this harness's sessions
|
|
59
|
+
--format text|json text (default), or json with id, path, bytes and parent id
|
|
60
|
+
|
|
61
|
+
jq paths (--format json):
|
|
62
|
+
.sessions[].ref the short ref any command takes
|
|
63
|
+
.sessions[] | {id, cwd, firstPrompt} which session is which
|
|
64
|
+
.selection.unreadSessions sessions that could not be read, and why
|
|
65
|
+
|
|
66
|
+
Examples:
|
|
67
|
+
traces list --cwd . --last 5
|
|
68
|
+
traces list --since 1d --harness codex
|
|
69
|
+
traces list --format json | jq -r '.sessions[0].id'
|
|
70
|
+
```
|
package/docs/skills.md
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# Skill usage
|
|
2
|
+
|
|
3
|
+
[traces](../README.md#docs) › Skill usage
|
|
4
|
+
|
|
5
|
+
`traces skills` says which skills are used, and which installed skills never are, across Claude Code, Codex and Kimi Code sessions on this machine. It calls no model.
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
traces skills # the last 30 days, with the installed catalog joined
|
|
9
|
+
traces skills --unused # installed skills with no use in the window
|
|
10
|
+
traces skills --format json # rows, unused skills, refresh cost and sweep exclusions
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
A run reads only transcript bytes appended since the previous run, and stores events in `$XDG_STATE_HOME/traces/skill-usage.json`. The first run reads the whole corpus (about two minutes for 41 GB); later runs take seconds. Stored events survive a harness deleting old transcripts.
|
|
14
|
+
|
|
15
|
+
| Column | Harness | Counted record |
|
|
16
|
+
| --- | --- | --- |
|
|
17
|
+
| `model` | Claude Code | a Skill tool result that loaded a skill; a failed call does not count |
|
|
18
|
+
| `slash` | Claude Code | a `/name` command whose next record is its expanded prompt |
|
|
19
|
+
| `read` | Codex, Kimi Code | a completed command that reads `<skill>/SKILL.md`, the closest observable act |
|
|
20
|
+
|
|
21
|
+
Each event is keyed by its record id, so a resumed session counts once. One command that reads three or more skills is a catalog sweep; its reads are excluded and reported apart.
|
|
22
|
+
`traces facts` counts one session's skill use with the same classifier (`measurements.skills`).
|
|
23
|
+
`installed` names the harnesses whose catalog lists the skill: `~/.claude/skills` and enabled plugins, `~/.codex/skills` and `~/.agents/skills`.
|
|
24
|
+
|
|
25
|
+
This package ships one skill, [`inspect-agent-traces`](../skills/inspect-agent-traces/SKILL.md): a table from each question an agent asks about a session to the command that answers it. Install it where your harness reads skills (`~/.claude/skills/inspect-agent-traces/`).
|
|
26
|
+
|
|
27
|
+
## `traces skills --help`
|
|
28
|
+
|
|
29
|
+
```text
|
|
30
|
+
Usage: traces skills [--since T] [--unused] [--format text|json]
|
|
31
|
+
Which skills are used, and which installed skills never are; no model call.
|
|
32
|
+
Reads Claude Code, Codex and Kimi Code sessions; each run reads only bytes appended since the last.
|
|
33
|
+
|
|
34
|
+
--since T count use since T: 30m, 2h, 7d or an ISO time (default 30d)
|
|
35
|
+
--unused only installed skills with no use in the window
|
|
36
|
+
--format text|json text (default), or json with the refresh cost and sweep exclusions
|
|
37
|
+
|
|
38
|
+
jq paths (--format json):
|
|
39
|
+
.rows[] | {skill, sessions} use per skill
|
|
40
|
+
.unused[].skill installed skills never used in the window
|
|
41
|
+
|
|
42
|
+
Examples:
|
|
43
|
+
traces skills --since 30d
|
|
44
|
+
traces skills --unused
|
|
45
|
+
traces skills --format json | jq '.rows[] | {skill, sessions}'
|
|
46
|
+
```
|
package/docs/upload.md
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Upload
|
|
2
|
+
|
|
3
|
+
[traces](../README.md#docs) › Upload
|
|
4
|
+
|
|
5
|
+
`traces upload` sends redacted sessions to the Tangle Intelligence Platform. It needs a ref or `--since`; with neither it exits 2 and prints `--since 24h`.
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
traces upload --since 24h --dry-run # see what would be sent; no network
|
|
9
|
+
traces upload --since 24h --no-content # metadata only
|
|
10
|
+
traces upload spans.otlp.jsonl --dry-run --out preview.otlp.jsonl
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
`upload` redacts locally before anything leaves the machine, dedups against sessions already sent, and tags each with its harness, cwd, branch and host.
|
|
14
|
+
It then checks each session's final events, exactly as they would be sent, with the share-safety verdict: an UNSAFE or UNKNOWN session is not sent, not previewed and not marked uploaded, and the command exits 1.
|
|
15
|
+
It needs `TANGLE_INGEST_URL`, `TANGLE_INGEST_API_KEY` (or `TANGLE_API_KEY`) and `TANGLE_TENANT_ID`; `--dry-run` works without them.
|
|
16
|
+
|
|
17
|
+
## What redaction catches
|
|
18
|
+
|
|
19
|
+
Redaction is agent-eval's redaction core, `default` profile: credentials by field name and by value shape (provider keys, GitHub, Slack, AWS and Google tokens, JWTs, bearer headers, private keys, `KEY=secret`, credentials in URLs), and email, card, SSN and phone values. It does not catch names, postal addresses or account numbers in prose. Three postures, strongest first:
|
|
20
|
+
|
|
21
|
+
1. `--no-content`: metadata only, the same rule as `traces convert <ref> --no-content`.
|
|
22
|
+
2. Your own scrubber on a file: `traces convert <ref> --out spans.jsonl`, scrub `spans.jsonl`, then `traces upload spans.jsonl`; upload still redacts and gates it.
|
|
23
|
+
3. The redaction core alone.
|
|
24
|
+
|
|
25
|
+
Run `--dry-run` first. `traces check <path> --share-safe` gives the same verdict for any file ([check](check.md)).
|
|
26
|
+
|
|
27
|
+
## `traces upload --help`
|
|
28
|
+
|
|
29
|
+
```text
|
|
30
|
+
Usage: traces upload <ref…> | --current | --since T [--dry-run] [--no-content] [--yes]
|
|
31
|
+
Send redacted sessions to Tangle Intelligence; one still UNSAFE or UNKNOWN once redacted is refused (exit 1).
|
|
32
|
+
Env: TANGLE_INGEST_URL, TANGLE_INGEST_API_KEY (or TANGLE_API_KEY), TANGLE_TENANT_ID.
|
|
33
|
+
|
|
34
|
+
--current the caller's own session: CODEX_THREAD_ID, else CLAUDE_CODE_SESSION_ID
|
|
35
|
+
--since T every session with a record since T: 30m, 2h, 7d or an ISO time
|
|
36
|
+
--cwd DIR sessions recorded in DIR or under it, or in a worktree of its repo
|
|
37
|
+
--harness ID read only this harness; with a file ref, force its reader
|
|
38
|
+
--dry-run redact and preview; send nothing
|
|
39
|
+
--no-content strip prompt and response text; send metadata only
|
|
40
|
+
--yes skip the confirmation prompt
|
|
41
|
+
--out PATH with --dry-run: write the redacted OTLP preview to PATH
|
|
42
|
+
|
|
43
|
+
Examples:
|
|
44
|
+
traces upload --since 24h --dry-run
|
|
45
|
+
traces upload 019a3c7e --yes
|
|
46
|
+
traces upload spans.otlp.jsonl --dry-run --out preview.otlp.jsonl
|
|
47
|
+
```
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
{"schemaVersion":1,"kind":"traces.stream.session","event":"session","generatedAt":"2026-10-16T10:40:00.000Z","session":{"harness":"claude-code","sessionId":"a11a1a11-aaaa-4aaa-8aaa-aaaaaaaaaaaa","cwd":"/w/analyzed-repo","path":"/tmp/traces-watch-fixture/home/.claude/projects/-w-analyzed-repo/a11a1a11-aaaa-4aaa-8aaa-aaaaaaaaaaaa.jsonl"},"spanCount":28,"startedAt":"2026-10-14T10:00:00.000Z","endedAt":"2026-10-16T10:29:20.000Z"}
|
|
2
|
+
{"schemaVersion":1,"kind":"traces.stream.finding","event":"finding","generatedAt":"2026-10-16T10:40:00.000Z","finding":{"schemaVersion":1,"kind":"traces.live_finding","id":"live.same-failing-command.2e9548f302d43a89","fingerprint":"2e9548f302d43a89","ruleId":"same-failing-command","severity":"high","title":"The same command failed again and again","claim":"`pnpm test` failed 3× in a row (5 min).","action":"Stop rerunning it; read the first failure, change state, or choose a different diagnostic path before another run.","check":"The next action should read the failing output, edit code or config, or run a narrower command before this one runs again.","evidence":[{"kind":"pattern","label":"command","value":"pnpm test","spanIds":["1299987ef6149612","87c74570a098898a","7d199fa0631ea55b"]},{"kind":"metric","label":"failed_runs_in_a_row","value":"3"},{"kind":"metric","label":"minutes","value":"5"}],"session":{"harness":"claude-code","sessionId":"a11a1a11-aaaa-4aaa-8aaa-aaaaaaaaaaaa","cwd":"/w/analyzed-repo","path":"/tmp/traces-watch-fixture/home/.claude/projects/-w-analyzed-repo/a11a1a11-aaaa-4aaa-8aaa-aaaaaaaaaaaa.jsonl"},"observedAt":"2026-10-16T10:40:00.000Z"}}
|
|
3
|
+
{"schemaVersion":1,"kind":"traces.stream.finding","event":"finding","generatedAt":"2026-10-16T10:40:00.000Z","finding":{"schemaVersion":1,"kind":"traces.live_finding","id":"live.verification-without-change.90335ccea7b19082","fingerprint":"90335ccea7b19082","ruleId":"verification-without-change","severity":"medium","title":"A check is rerun without a change","claim":"`pnpm test` ran 3× with no edit between.","action":"Do not run the check again until the agent edits state or says what new signal the rerun will reveal.","check":"The next action should be an edit, a targeted read, or a different, narrower check.","evidence":[{"kind":"pattern","label":"check","value":"pnpm test","spanIds":["1299987ef6149612","87c74570a098898a","7d199fa0631ea55b"]},{"kind":"metric","label":"runs_without_an_edit","value":"3"}],"session":{"harness":"claude-code","sessionId":"a11a1a11-aaaa-4aaa-8aaa-aaaaaaaaaaaa","cwd":"/w/analyzed-repo","path":"/tmp/traces-watch-fixture/home/.claude/projects/-w-analyzed-repo/a11a1a11-aaaa-4aaa-8aaa-aaaaaaaaaaaa.jsonl"},"observedAt":"2026-10-16T10:40:00.000Z"}}
|
|
4
|
+
{"schemaVersion":1,"kind":"traces.stream.finding","event":"finding","generatedAt":"2026-10-16T10:40:00.000Z","finding":{"schemaVersion":1,"kind":"traces.live_finding","id":"live.high-tool-error-rate.f58f47ed22ff6ead","fingerprint":"f58f47ed22ff6ead","ruleId":"high-tool-error-rate","severity":"medium","title":"Tool calls are mostly failing","claim":"3/6 tool call(s) ended with ERROR.","action":"Switch from broad execution to diagnosis: classify the dominant error, then run one targeted command to validate the fix path.","check":"The next calls should reduce the error ratio or show a new targeted diagnostic result.","evidence":[{"kind":"metric","label":"errored_tool_calls","value":"3/6","spanIds":["1299987ef6149612","87c74570a098898a","7d199fa0631ea55b"]}],"session":{"harness":"claude-code","sessionId":"a11a1a11-aaaa-4aaa-8aaa-aaaaaaaaaaaa","cwd":"/w/analyzed-repo","path":"/tmp/traces-watch-fixture/home/.claude/projects/-w-analyzed-repo/a11a1a11-aaaa-4aaa-8aaa-aaaaaaaaaaaa.jsonl"},"observedAt":"2026-10-16T10:40:00.000Z"}}
|
|
5
|
+
{"schemaVersion":1,"kind":"traces.stream.tick","event":"tick","generatedAt":"2026-10-16T10:40:00.000Z","sessions":1,"newSpans":28,"findings":3,"unreadSessions":[{"sessionId":"0199f9d0-9999-7999-8999-999999999999","source":"/tmp/traces-watch-fixture/home/.codex/sessions/2026/10/16/rollout-2026-10-16T10-30-00-0199f9d0-9999-7999-8999-999999999999.jsonl","reason":"codex adapter output contains duplicate span identity (3b14c728e785beeaa7fb4d0f5fa96b4f, 7d00cfb5b23dd5f4)"}]}
|
package/docs/watch.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Watch live sessions and run trees
|
|
2
|
+
|
|
3
|
+
[traces](../README.md#docs) › Watch
|
|
4
|
+
|
|
5
|
+
`traces watch` runs the analyze rules on every session active in the last 30 minutes, on every harness, while the agents still run: a command that failed 3 or more times in a row, a check rerun 3 or more times with no edit between, and half or more of a session's tool calls failing.
|
|
6
|
+
Sessions are read every 5 s and the window slides, so a feed left running never goes blind. A session that cannot be read is skipped and named.
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
traces watch --once # one pass over every active session
|
|
10
|
+
traces watch --since 2h --cwd . # this repo's sessions of the last two hours, live
|
|
11
|
+
traces watch --format jsonl # the machine feed
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Text prints one line per new finding:
|
|
15
|
+
|
|
16
|
+
```
|
|
17
|
+
10:06:00 HIGH a11a1a11 claude-code same-failing-command: `pnpm test` failed 3× in a row (5 min). (spans t1, t2, …)
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## The `traces.stream` envelope
|
|
21
|
+
|
|
22
|
+
`--format jsonl` prints the same events the text renders, at `schemaVersion` 1. It is a stable API: [`watch-jsonl.v1.example.jsonl`](./watch-jsonl.v1.example.jsonl) is the CLI's output on a fixture, and `tests/watch-jsonl-contract.test.ts` holds it and the fields consumers read.
|
|
23
|
+
|
|
24
|
+
| `kind` | When | Fields |
|
|
25
|
+
|---|---|---|
|
|
26
|
+
| `traces.stream.session` | the first pass that reads a session | `session` (`harness`, `sessionId`, `cwd`, `path`), `spanCount`, `startedAt`, `endedAt` |
|
|
27
|
+
| `traces.stream.finding` | the first pass that finds it | `finding`: `ruleId`, `fingerprint`, `severity`, `claim`, `action`, `check`, `evidence`, `session`, `observedAt` |
|
|
28
|
+
| `traces.stream.tick` | the end of every pass | `sessions`, `newSpans`, `findings`, `unreadSessions` |
|
|
29
|
+
|
|
30
|
+
Every event also carries `schemaVersion`, `event` and `generatedAt`. Custom live rules go through the SDK: `streamSessions({ extraAnalysts })`.
|
|
31
|
+
|
|
32
|
+
## A run tree
|
|
33
|
+
|
|
34
|
+
`traces watch <run-dir>` tails one run that fans work out to other agents: per node its status, runtime, authored budget and actual spend, at any depth. It is plain stdout, so it works over SSH, in a CI log and through a pipe. A run tree is read every 2 s until its result lands, and a run that ended with an error exits 1.
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
traces watch runs/my-run --once
|
|
38
|
+
traces watch traces.otlp.jsonl # spans only, from any emitter with parent span ids
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Two sources sit behind it, and the output names which number came from which: OTLP spans (tree shape, wall time, tokens, cost when priced) and agent-runtime's `spawn-journal.jsonl` (authored budget, settled spend, blocking questions, steers, the result).
|
|
42
|
+
A node's own driving and the work it delegated are separate lines, and an unpriced turn reads `cost unknown`, never `$0.0000`. `traces analyze <run-dir>` writes the full report.
|
|
43
|
+
|
|
44
|
+
## `traces watch --help`
|
|
45
|
+
|
|
46
|
+
```text
|
|
47
|
+
Usage: traces watch [<id…> | <store-root>] [--since T] [--once] | watch <run-dir|span file> [--once]
|
|
48
|
+
Live supervision: what the analyze rules find in every session active in the last 30 minutes, or one run tree.
|
|
49
|
+
Sessions are read every 5 s; a run tree every 2 s, until its result lands (exit 1 when the run ended with an error).
|
|
50
|
+
A run dir or span file prints its tree as text, and its findings with --format jsonl.
|
|
51
|
+
|
|
52
|
+
--since T sessions active in the last T (default 30m; it slides with time), or since an ISO time
|
|
53
|
+
--cwd DIR sessions recorded in DIR or under it, or in a worktree of its repo
|
|
54
|
+
--harness ID only this harness's sessions; with a file ref, force its reader
|
|
55
|
+
--once one pass, then exit
|
|
56
|
+
--format text|jsonl text (default): a line per new finding, or the run tree; jsonl: the traces.stream events
|
|
57
|
+
|
|
58
|
+
jq paths (--format jsonl):
|
|
59
|
+
select(.event == "finding") | .finding | {severity, ruleId, claim} each new finding
|
|
60
|
+
select(.event == "tick") | .unreadSessions sessions a pass could not read, and why
|
|
61
|
+
|
|
62
|
+
Examples:
|
|
63
|
+
traces watch --once
|
|
64
|
+
traces watch runs/my-run --once
|
|
65
|
+
traces watch --since 2h --format jsonl
|
|
66
|
+
```
|