@tangle-network/traces 0.15.4 → 0.16.0

package/README.md

@@ -23,6 +23,7 @@ Emitting the contract is the supported way to integrate a new system. The adapte
 
 - [Install](#install)
 - [Quick start](#quick-start)
+- [Gate CI on a trace contract](#gate-ci-on-a-trace-contract)
 - [Integrate your own system](#integrate-your-own-system)
 - [What it finds](#what-it-finds)
 - [Supported harnesses](#supported-harnesses)
@@ -32,10 +33,12 @@ Emitting the contract is the supported way to integrate a new system. The adapte
 - [Improvement engine](#improvement-engine)
 - [Ask questions](#ask-questions)
 - [Session facts](#session-facts)
+- [Diff two runs](#diff-two-runs)
+- [MCP server](#mcp-server)
 - [Session index](#session-index)
 - [Session bundle](#session-bundle) · [Two views](#two-views-two-consumers)
 - [Policy-mining evidence](#policy-mining-evidence)
-- [Upload to the Intelligence Platform](#upload-to-the-intelligence-platform)
+- [Upload to the Intelligence Platform](#upload-to-the-intelligence-platform) · [Check before sharing](#check-before-sharing)
 - [Trace analysts](#trace-analysts)
 - [Agent skills](#agent-skills)
 - [Library (SDK)](#library-sdk)
@@ -62,6 +65,7 @@ Requires Node ≥ 22.
 traces validate spans.otlp.jsonl                  # what can this trace answer?
 traces analyze  --otlp spans.otlp.jsonl           # analyse it, no adapter involved
 traces analyze --harness claude-code --last 1     # or read a coding agent's own log
+traces check session.jsonl --contract contract.json # gate CI on the path the run took
 traces improve --harness claude-code --last 5 --dir .traces/improvement
 traces watch --all
 traces stream --all --mode findings
@@ -82,6 +86,48 @@ See [Trace analysts](#trace-analysts).
 It writes one typed result, one report, flattened evidence rows, and the canonical OTLP trace.
 Each finding already contains the claim, evidence, recommended action, confidence, and validation plan.
 
+## Gate CI on a trace contract
+
+Two runs can give the same answer while one of them calls a forbidden tool.
+An output check passes both; a trace contract fails the wrong one.
+
+```json
+{
+  "name": "refund-desk",
+  "run": { "requireCompleted": true, "maxDurationMs": 300000 },
+  "tools": { "required": ["Read"], "forbidden": ["Bash"], "allowed": ["Read", "Grep"], "maxCalls": 10, "enforced": true },
+  "retries": { "reads": ["Read", "Grep"] },
+  "llm": { "maxCalls": 10 }
+}
+```
+
+```bash
+claude -p "$TASK" --tools Read,Grep --strict-mcp-config --output-format stream-json --verbose > run.jsonl
+traces check run.jsonl --contract refund-desk.json --junit contract.xml
+```
+
+`tools.enforced` fails when the harness offered the model a tool the contract does not allow, which a flag such as `--tools` does not prove.
+Without `--strict-mcp-config`, the run above is also offered every MCP server the account or project configures.
+`retries` fails when a tool repeats a call with the same arguments and nothing proves the repeat is safe.
+
+The trace is `claude -p --output-format stream-json` output, a Claude Code or Codex session file, OTLP spans (a file or a directory), or trace evidence; `--format` names it when the file could be read as more than one.
+Only stream-json output records the offered tools and the run's own result.
+**A saved interactive session file carries no run status at all** (the `result` record above only exists in `-p` mode), so `run.requireCompleted`/`run.allowedStatuses` always fail on one — write a contract for a session file with no `run` key, like the `refund-desk` example above running against `run.jsonl`.
+Every rule prints `pass`, `fail`, `error`, or `skipped` (a rule of an alternative path that did not decide the result).
+
+| Exit | Meaning |
+|---|---|
+| 0 | Every rule passed. |
+| 1 | A rule failed. |
+| 2 | The contract is malformed or contradicts itself, or a rule could not be evaluated. |
+| 3 | The trace could not be read. |
+| 4 | The trace reference is ambiguous. |
+
+`--junit` writes one test case per rule, and GitHub annotations print under `GITHUB_ACTIONS=true` or `--annotations`.
+`traces check --contract <file> --explain` prints what each compiled rule checks and reads no trace.
+For a trace that does not pass, the verdict, the contract, its plain-language statement, and the redacted spans the violations cite go to `--evidence` (default `.traces/check`).
+The contract keys and rule semantics are in [agent-eval's trace-contract guide](https://github.com/tangle-network/agent-eval/blob/main/docs/trace-contracts.md).
+
 ## Integrate your own system
 
 No adapter, no plugin. Emit the span contract and point `--otlp` at the file.
@@ -125,9 +171,7 @@ files are read — every other JSONL is listed in the report with what it actual
 holds, rather than counted as several hundred broken spans. Put the exports in an
 `otlp/` subdirectory and that decides it outright.
 
-On a writing command (`convert`, `export`, `evidence`, `upload`), `--otlp` is the
-DEPRECATED spelling of `--otlp-out`. It still works, with a warning, and is
-removed in 0.12.
+Writing commands (`convert`, `export`, `evidence`, `upload`) take `--otlp-out`; they refuse `--otlp`.
 
 `parent_span_id` is CONTAINMENT — this span happened inside that one. `links` is
 CAUSALITY — that verdict caused this retry. Encoding causality as a parent claims
@@ -168,9 +212,9 @@ trace through `traces` cannot produce a clean bill of health.
 `analyze` prints the same section, plus an **analyses skipped, and why** list —
 and, crucially, carries that verdict INTO the report. A section whose inputs are
 incomplete says so in its own heading and again directly above its table, so a
-number and the reason it is wrong can never end up forty lines apart. A capability
-the trace supports that this package has no analysis for yet is listed as
-`⚠️ available, unused` rather than quietly dropped.
+number and the reason it is wrong can never end up forty lines apart. When the
+trace supports `tree-comparison`, the table names the command that compares its
+arms: `traces diff <file>#branch=<a> <file>#branch=<b>`.
 
 Two of those analyses are why the loop shape exists at all:
 
@@ -250,7 +294,10 @@ traces investigate --all --last 10 --out report.md  # explicit investigation ali
 traces improve --all --last 10 --dir .traces/improvement
 traces ask --harness codex --session <id> --question "Which commands failed?"
 traces facts --harness codex --session <id>        # the deterministic facts sheet, $0
+traces diff  run-a.otlp.jsonl run-b.otlp.jsonl      # where two runs of one task first diverge
+traces mcp   --otlp spans.otlp.jsonl                # read-only trace tools for an MCP client
 traces analyze  --all --since 2026-06-18 --out report.md
+traces check   spans.otlp.jsonl --contract c.json --junit c.xml  # exit 0 pass, 1 fail, 2 bad contract, 3 unreadable, 4 ambiguous
 traces validate spans.otlp.jsonl                   # conformance; exit 1 only when it is not a trace
 traces validate results/sessions --out conformance.md  # a whole directory of exports
 traces analyze  --otlp spans.otlp.jsonl            # analyse foreign OTLP, no adapter
@@ -275,6 +322,7 @@ traces stream   --all --mode agent                 # findings + deterministic re
 traces stream   spans.openinference.jsonl --format openinference --no-spans
 traces upload   --since 1h --dry-run               # redact + dedup + preview, no network
 traces upload   --since 24h                        # upload last day to the Intelligence Platform
+traces verify-safe bundle-dir                      # SAFE / SAFE_WITH_WARNINGS / UNSAFE (exit 1) / UNKNOWN (exit 2)
 traces replay-verify --steps steps.json --image <docker-image> --at 37 --cwd /home --out ./replay-out \
   --fix-command "<corrected step>"                 # executed proof: replay prefix, reproduce failure, show fix
 traces verify-findings --findings findings.json --out ./receipts \
@@ -300,7 +348,11 @@ See [Replay verification](./docs/replay-verify.md) for setup, semantics, and hon
 | `--since <t>` | `upload`: window, `30m`/`2h`/`7d` or ISO (default 24h); `analyze`: ISO cutoff |
 | `--out <path>` | Write the report to a file |
 | `--dir <path>` | `improve`: write the full artifact pack to this directory; `ask`: write `answers.json` + `report.md` there |
-| `--otlp <file\|dir>` | **READ** OTLP-JSONL from any system, skipping the adapters; a directory reads the OTLP files under it (only `otlp/` when the producer made one) and names the JSONL that is not OTLP. `validate`, `analyze`, `investigate`, `improve`, `ask`, `stream` |
+| `--otlp <file\|dir>` | **READ** OTLP-JSONL from any system, skipping the adapters; a directory reads the OTLP files under it (only `otlp/` when the producer made one) and names the JSONL that is not OTLP. `validate`, `check`, `analyze`, `investigate`, `improve`, `ask`, `stream` |
+| `--contract <file>` | `check`: the declarative trace contract (JSON) |
+| `--junit <file>` | `check`: write JUnit XML, one test case per rule |
+| `--evidence <dir>` | `check`: where evidence for a trace that did not pass goes (default `.traces/check`) |
+| `--annotations` | `check`: print GitHub workflow annotations (on by default under `GITHUB_ACTIONS=true`) |
 | `--otlp-out <path>` | **WRITE** the OTLP artifact here (also evidence provenance / dry-run upload preview) |
 | `--format <kind>` | File `analyze`, `export`, or `stream`: `auto`, `policy-evidence`, `sandbox-events`, `openinference`, `intelligence-spans`, or `chat-trajectory` |
 | `--source-bundle <dir>` | `analyze` / `investigate` / `improve` / `ask`: read a retained full bundle and explicitly grant source-field reads |
@@ -536,6 +588,7 @@ traces facts --otlp spans.otlp.jsonl --out facts.json
 | `firstRecordAt` / `lastRecordAt` | the earliest span start and latest span end among this session's own records — a subagent that outlives the session does not stretch its window |
 | `unreadRecords` | records the session reader could not parse, from the session's integrity receipt |
 | `tokenTotal` | the harness's own cumulative token total, when a span carries `traces.session.total_tokens` |
+| `firstFailure` | the first failure among this session's own records, ranked by agent-eval's fixed precedence: the innermost ERROR span that ended first, else a span whose `agent.outcome` is `fail`, else `none` — spans were observed and none of them failed. When every span is UNSET (a capture that failed before any status was written), or a session has no records of its own at all (every span is `traces.span.subagent` — spawned work folded into this trace, with nothing this agent recorded itself), the status is `unknown`, not `none`: the record does not say whether the run failed. Failures that ended at the same instant are `ambiguous` and list the candidates instead of picking one. `blame` separates machine and provider failures from the agent's own, and is `unknown` rather than `agent` when no evidence names the agent |
 
 Three rules hold for every field:
 
@@ -547,6 +600,43 @@ Three rules hold for every field:
 
 The same sheet reaches the model-backed analysts as prepared context, before their first model call — see [Trace analysts](docs/trace-analysts.md#session-facts-as-prepared-context).
 
+## Diff two runs
+
+`traces diff` pairs the steps of two runs of the same task and marks where they first stop agreeing.
+
+```bash
+traces diff runs/a/otlp runs/b/otlp                     # two OTLP exports
+traces diff a.sdk-events.jsonl b.sdk-events.jsonl --kind TOOL   # any file convert reads; tool calls only
+traces diff spans.otlp.jsonl#<trace id> other.jsonl --format json
+traces diff run.otlp.jsonl#branch=arm-a run.otlp.jsonl#branch=arm-b    # two arms of one run
+```
+
+A side is a file or directory that holds one run, `file#<trace id>` for one run of several, or `file#branch=<id>` for one arm: the spans whose `agent.branch.id` or `agent.branch.arm` is `<id>` and have no matched ancestor (a root), and every span below them. Sibling arms can share a branch id or label; when `<id>` names more than one such root subtree, the command refuses (exit 2) and lists each root's span id, rather than silently comparing the union of both.
+Steps pair by span id first, then by position with the same name and kind, then by name and kind anywhere.
+One inserted step therefore does not mark every later step as changed.
+Paired steps compare status, tool and model.
+The first divergence is `changed`, `replaced`, `only-in-a`, `only-in-b` or `reordered`, with its reason, followed by every changed and unpaired step.
+`--kind` keeps only steps of that span kind, so event noise in a raw stream does not decide the first divergence.
+The diff is agent-eval's `diffSteps`; this command reads the runs and prints it.
+It exits 0 when the runs agree on outcome, 1 when they diverge, 2 when a side cannot be read (including an ambiguous `#branch=` selector), and 3 when the steps line up with no divergence but there is no recorded outcome to call agreement: neither side has a step with a definitive OK/ERROR status (both are entirely UNSET, the common shape for a capture that failed before any status was written), or neither side has a TOOL step. That case's report and text output carry a `caveat` naming which reason applies; a CI or agent gate should treat exit 3 as unknown, not as a pass.
+
+## MCP server
+
+`traces mcp` serves the selected traces to an MCP client over stdio.
+
+```bash
+claude mcp add traces -- traces mcp --otlp ./spans.otlp.jsonl
+claude mcp add traces -- traces mcp --harness claude-code --last 5
+```
+
+It serves agent-eval's seven trace tools: `getDatasetOverview`, `queryTraces`, `countTraces`, `viewTrace`, `viewSpans`, `searchTrace` and `searchSpan`.
+Each tool declares `readOnlyHint` and `idempotentHint` in its MCP annotations, and its description says that returned trace text is untrusted.
+
+- The spans are redacted before the store is built — every span's name, attributes, status message and causal links' attributes, every field the store reads — and each result is redacted again at the boundary. The server then reassesses the redacted spans and refuses to start when they are still UNSAFE or UNKNOWN, the same gate `upload` applies to what it sends.
+- Each result carries an `untrusted` notice beside it.
+- A result above 512 KiB is refused with an error rather than truncated.
+- `readSpanSource` is not served. It reads original source bytes in windows the caller picks, and a secret split across two windows matches no redaction rule.
+
 ## Session index
 
 `traces index` writes one general JSON catalog over the selected sessions.
@@ -689,6 +779,8 @@ Run `traces export --help` for the full command reference.
 ## Upload to the Intelligence Platform
 
 `upload` **redacts locally before anything leaves the machine**, dedups against already-uploaded sessions, and tags each with metadata (harness, cwd, git branch, host).
+It then checks the final events of each session, exactly as they would be sent, with the share-safety verdict.
+A session that is UNSAFE or UNKNOWN is refused: it is not sent, not written to the dry-run preview, and not marked uploaded, and the command exits 1.
 
 ```bash
 traces upload --since 24h --dry-run     # see exactly what would be sent; no network
@@ -700,16 +792,40 @@ It needs `TANGLE_INGEST_URL` (or `TANGLE_ORCHESTRATOR_URL`), `TANGLE_INGEST_API_
 
 ### Redaction scope: read this before uploading prose
 
-Redaction is **best-effort regex** for *structured* secrets and credentials: API keys, GitHub/cloud tokens, JWTs, bearer headers, private-key blocks, `KEY=secret` assignments, and credentials embedded in URLs. It runs over every span attribute, including the captured prompt/response text.
+Redaction is agent-eval's redaction core ([docs/redaction.md](https://github.com/tangle-network/agent-eval/blob/main/docs/redaction.md)), `default` profile.
+It finds credentials by field name (`apiKey`, `x-api-key`, `client.secret`, `refreshToken`) and by value shape (provider keys, GitHub, Slack, AWS and Google tokens, JWTs, bearer headers, private keys, `KEY=secret` assignments, credentials in URLs), and replaces the whole string.
+It replaces email, card, SSN and phone values in place.
+Token counts such as `input_tokens` and `max_tokens` are kept.
+It runs over every span attribute, including the captured prompt/response text.
 
-It does **not** catch free-form PII such as names, postal addresses, or phone numbers in prose. Those need a context-aware model. Three postures, strongest first:
+It does **not** catch names, postal addresses or account numbers written in prose. Those need a context-aware model. Three postures, strongest first:
 
 1. **`--no-content`**: upload metadata only (tool calls, tokens, timing, loop signal); no prose leaves the machine.
-2. Run an ML PII scrubber (e.g. [`openai/privacy-filter`](https://github.com/openai/privacy-filter)) on the platform ingest side as defense-in-depth.
-3. Default: regex redaction of structured secrets.
+2. **`--redactor <cmd>`**: run an ML PII scrubber (e.g. [`openai/privacy-filter`](https://github.com/openai/privacy-filter)) after the redaction core.
+3. Default: the redaction core.
 
 Always `--dry-run` first to see exactly what would be sent.
 
+### Check before sharing
+
+`verify-safe` reads files and says whether they are safe to share. It never changes them.
+
+```bash
+traces verify-safe bundle-dir                              # share profile (default)
+traces verify-safe session.jsonl --profile strict --format json
+traces verify-safe out/ --known-secret-env TANGLE_API_KEY  # also look for this exact key, in any encoding
+```
+
+| Status | Meaning | Exit |
+|---|---|---|
+| `SAFE` | Nothing found | 0 |
+| `SAFE_WITH_WARNINGS` | Only warnings: raw content, identifiers, embedded media | 0 |
+| `UNSAFE` | A credential remains, or personal data under `share` or `strict` | 1 |
+| `UNKNOWN` | A file, line or byte range could not be read, so its safety is unknown | 2 |
+
+`.json` files are parsed whole and `.jsonl` or `.ndjson` files line by line. Any other file is scanned as text.
+Each finding names the file, line and JSON Pointer, never the matched value.
+
 ## Trace analysts
 
 Use the local checks by default.
@@ -827,7 +943,8 @@ The CLI is a thin consumer of these exports.
 | `exportTraceEvidenceFile` | `(path, opts?) → { format, spans, redactionCount }` | convert compact evidence/events/OpenInference files to redacted OpenInference spans |
 | `scanSessions` | `(ScanOptions) → AsyncIterable<ScannedSession>` | the shared locate→parse iterator |
 | `collectSessions` | `(CollectOptions) → SessionBatch[]` | redacted per-session batches for your own pipeline |
-| `redactSpans` | `(spans, rules?) → { spans, report }` | PII/secret redaction (`TRACES_REDACTION_RULES`) |
+| `redactSpans` | `(spans, { profile?, knownSecrets? }) → { spans, report }` | redact spans with agent-eval's redaction core |
+| `assessSpans` / `verifySafe` | `(spans, profile?)` / `(path, { profile?, knownSecrets? })` → verdict | share-safety verdict for spans or files |
 | `planUpload` / `executeUpload` | `(…, { backend? }) → …` | redact + dedup + send to any sink |
 | `selectAdapters` / `listAdapters` / `resolveAdapter` | adapter selection + the harness registry |
 | `PiAdapter` | `new PiAdapter({ sessionsRoot? })` | parse Pi's default session tree or an exact `PI_CODING_AGENT_SESSION_DIR` |