agent-inspect 6.19.0 → 6.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/README.md +3 -1
- package/SECURITY.md +2 -0
- package/docs/ARCHITECTURE.md +28 -16
- package/docs/KNOWN-ISSUES.md +3 -3
- package/docs/LIMITATIONS.md +5 -4
- package/docs/PERFORMANCE.md +1 -1
- package/docs/SUPPORT-LEVELS.md +2 -1
- package/docs/TRACE-CONTRACTS.md +64 -52
- package/package.json +3 -2
- package/packages/cli/dist/{chunk-R7Y5SGH5.mjs → chunk-QPRJUVLH.mjs} +57 -17
- package/packages/cli/dist/chunk-QPRJUVLH.mjs.map +1 -0
- package/packages/cli/dist/index.cjs +69 -18
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +9 -6
- package/packages/cli/dist/index.mjs.map +1 -1
- package/packages/cli/dist/{src-T4EZERCX.mjs → src-6W5ZR3I7.mjs} +3 -3
- package/packages/cli/dist/{src-T4EZERCX.mjs.map → src-6W5ZR3I7.mjs.map} +1 -1
- package/packages/core/dist/advanced.cjs +118 -18
- package/packages/core/dist/advanced.cjs.map +1 -1
- package/packages/core/dist/advanced.d.cts +2 -2
- package/packages/core/dist/advanced.d.ts +2 -2
- package/packages/core/dist/advanced.mjs +8 -8
- package/packages/core/dist/checks.cjs +443 -9
- package/packages/core/dist/checks.cjs.map +1 -1
- package/packages/core/dist/checks.d.cts +78 -16
- package/packages/core/dist/checks.d.ts +78 -16
- package/packages/core/dist/checks.mjs +3 -3
- package/packages/core/dist/{chunk-VFO76UH3.mjs → chunk-3XS5O4JI.mjs} +3 -3
- package/packages/core/dist/{chunk-VFO76UH3.mjs.map → chunk-3XS5O4JI.mjs.map} +1 -1
- package/packages/core/dist/{chunk-ZWA5RWMV.mjs → chunk-5YFJJFXU.mjs} +4 -4
- package/packages/core/dist/{chunk-ZWA5RWMV.mjs.map → chunk-5YFJJFXU.mjs.map} +1 -1
- package/packages/core/dist/{chunk-NABMW5DX.mjs → chunk-G47KQB42.mjs} +445 -13
- package/packages/core/dist/chunk-G47KQB42.mjs.map +1 -0
- package/packages/core/dist/{chunk-SGL4JGCD.mjs → chunk-JBNRJLIV.mjs} +58 -18
- package/packages/core/dist/chunk-JBNRJLIV.mjs.map +1 -0
- package/packages/core/dist/{chunk-ULCGIRTC.mjs → chunk-U25RHRSU.mjs} +65 -5
- package/packages/core/dist/chunk-U25RHRSU.mjs.map +1 -0
- package/packages/core/dist/{context-Ca5yK8Cs.d.cts → context-C5ye_xcS.d.cts} +0 -4
- package/packages/core/dist/{context-D6cS4XIL.d.ts → context-CNsNSgE2.d.ts} +0 -4
- package/packages/core/dist/exporters.cjs +55 -15
- package/packages/core/dist/exporters.cjs.map +1 -1
- package/packages/core/dist/exporters.mjs +3 -3
- package/packages/core/dist/{index-DKwuGSe-.d.ts → index-Bg9ZdsdS.d.ts} +10 -0
- package/packages/core/dist/{index-DCR816Yt.d.cts → index-rVpX5PPh.d.cts} +10 -0
- package/packages/core/dist/index.cjs +63 -3
- package/packages/core/dist/index.cjs.map +1 -1
- package/packages/core/dist/index.d.cts +1 -1
- package/packages/core/dist/index.d.ts +1 -1
- package/packages/core/dist/index.mjs +4 -4
- package/packages/core/dist/writers.cjs +33 -2
- package/packages/core/dist/writers.cjs.map +1 -1
- package/packages/core/dist/writers.mjs +1 -1
- package/packages/cli/dist/chunk-R7Y5SGH5.mjs.map +0 -1
- package/packages/core/dist/chunk-NABMW5DX.mjs.map +0 -1
- package/packages/core/dist/chunk-SGL4JGCD.mjs.map +0 -1
- package/packages/core/dist/chunk-ULCGIRTC.mjs.map +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,17 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 6.20.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 43a4481: Flexible deterministic contracts: selectable `requiredOrderMode` (`first-occurrence` | `happens-before` | `all-occurrences`), one-level `alternatives.anyOf`, and `lintTraceContract` / `explainTraceContract` helpers. Includes MCP expected-rejection and local Promptfoo use-together recipes.
|
|
8
|
+
|
|
9
|
+
## 6.19.1
|
|
10
|
+
|
|
11
|
+
### Patch Changes
|
|
12
|
+
|
|
13
|
+
- e23b94b: 6.19.1 trust restoration: redact free-text credentials in persisted error messages before disk, refresh shipped docs to match 6.19.0 reality, and enforce all 18 fixed-group package READMEs (including root `agent-inspect`) via `package-readmes:check`.
|
|
14
|
+
|
|
3
15
|
## 6.19.0
|
|
4
16
|
|
|
5
17
|
### Minor Changes
|
package/README.md
CHANGED
|
@@ -33,6 +33,8 @@
|
|
|
33
33
|
<a href="package.json"><img src="https://img.shields.io/badge/node-%3E%3D20-brightgreen" alt="Node.js 20 or newer"></a>
|
|
34
34
|
</p>
|
|
35
35
|
|
|
36
|
+
**Support level:** Stable — see [SUPPORT-LEVELS.md](docs/SUPPORT-LEVELS.md). Network behavior: [NETWORK-BEHAVIOR.md](docs/NETWORK-BEHAVIOR.md).
|
|
37
|
+
|
|
36
38
|
```bash
|
|
37
39
|
npm install agent-inspect
|
|
38
40
|
```
|
|
@@ -210,7 +212,7 @@ The root package is enough for custom capture, the CLI, checks, and Evidence wor
|
|
|
210
212
|
|
|
211
213
|
## Status and documentation
|
|
212
214
|
|
|
213
|
-
**Current published baseline:** **6.
|
|
215
|
+
**Current published baseline:** **6.20.0** · persisted schema `1.0` · Node.js `>=20` · MIT.
|
|
214
216
|
|
|
215
217
|
Legacy v0.1 and v0.2 traces remain readable. Check the npm badge and [changelog](CHANGELOG.md) for the current published version.
|
|
216
218
|
|
package/SECURITY.md
CHANGED
|
@@ -82,6 +82,8 @@ For a practical pre-share workflow, see `docs/SAFE-TRACE-SHARING.md`. For schema
|
|
|
82
82
|
|
|
83
83
|
Pull requests run GitHub **Dependency review** (`.github/workflows/dependency-review.yml`) and fail on newly introduced **high** or **critical** advisory severity. Routine unit/typecheck/size CI also exercises redaction, path containment, and Evidence safety tests. These scanners do **not** replace manual review of share profiles or Evidence packages before you disclose traces.
|
|
84
84
|
|
|
85
|
+
For synthetic fixtures, dynamic-import boundaries, and an explicit “what scanners verify vs not” table, see [docs/SECURITY-SCANNERS.md](docs/SECURITY-SCANNERS.md).
|
|
86
|
+
|
|
85
87
|
## Optional surfaces (v2.6)
|
|
86
88
|
|
|
87
89
|
### Local viewer (`agent-inspect serve`)
|
package/docs/ARCHITECTURE.md
CHANGED
|
@@ -1,27 +1,41 @@
|
|
|
1
1
|
## Architecture
|
|
2
2
|
|
|
3
|
-
> **Adoption-focused overview:** see [TECHNICAL-GUIDE.md](./TECHNICAL-GUIDE.md) for a comprehensive technical guide
|
|
3
|
+
> **Adoption-focused overview:** see [TECHNICAL-GUIDE.md](./TECHNICAL-GUIDE.md) for a comprehensive technical guide suitable for blogs and outreach.
|
|
4
4
|
|
|
5
|
-
AgentInspect is a **local-first execution-tree debugger
|
|
5
|
+
AgentInspect is a **local-first execution-tree debugger and trajectory-test toolkit** for TypeScript AI agents: framework-native capture and manual traces produce inspectable step trees as local JSONL (default `.agent-inspect`), with CLI, TraceContract / TraceFacts checks, Evidence v2 packaging, and optional read-only MCP.
|
|
6
6
|
|
|
7
7
|
### Package layout
|
|
8
8
|
|
|
9
9
|
| Package | Published? | Role |
|
|
10
10
|
| -------- | ---------- | ---- |
|
|
11
11
|
| `agent-inspect` | Yes | Public tarball: core tracing APIs + CLI (`agent-inspect` binary) |
|
|
12
|
-
| `@agent-inspect/core` | No (private) | Tracing, storage,
|
|
12
|
+
| `@agent-inspect/core` | No (private) | Tracing, storage, readers/writers, checks, export, diff |
|
|
13
13
|
| `@agent-inspect/cli` | No (private) | Commander CLI implementation |
|
|
14
|
-
| `@agent-inspect/
|
|
15
|
-
| `@agent-inspect/
|
|
16
|
-
|
|
17
|
-
|
|
14
|
+
| `@agent-inspect/redact` | Yes | Standalone redaction engine (also used by CLI `redact`) |
|
|
15
|
+
| `@agent-inspect/ai-sdk` | Yes (optional) | Vercel AI SDK telemetry integration |
|
|
16
|
+
| `@agent-inspect/openai-agents` | Yes (optional) | OpenAI Agents JS `setTraceProcessors()` adapter |
|
|
17
|
+
| `@agent-inspect/langchain` | Yes (optional) | LangChain.js / LangGraph callback adapter |
|
|
18
|
+
| `@agent-inspect/mcp` | Yes (optional) | MCP client `tools/list` / `tools/call` tracing |
|
|
19
|
+
| `@agent-inspect/adapter-sdk` | Yes (optional) | Shared adapter helpers / plugin surface |
|
|
20
|
+
| `@agent-inspect/vitest` / `@agent-inspect/jest` | Yes (optional) | Test reporters and TraceContract matchers |
|
|
21
|
+
| `@agent-inspect/eval` | Yes (optional) | Deterministic local eval heuristics |
|
|
22
|
+
| `@agent-inspect/guardrails` / `@agent-inspect/circuit` | Yes (optional) | Local guardrail and circuit utilities |
|
|
23
|
+
| `@agent-inspect/harness` | Yes (optional) | Lightweight local harness helpers |
|
|
24
|
+
| `@agent-inspect/viewer` | Yes (optional) | Local HTML/viewer surface |
|
|
25
|
+
| `@agent-inspect/tui` | Yes (optional) | Ink/React terminal viewer |
|
|
26
|
+
| `@agent-inspect/mcp-server` | Yes (optional) | Read-only local MCP server over traces |
|
|
27
|
+
| `@agent-inspect/index-sqlite` | Yes (optional) | Optional local SQLite metadata index |
|
|
28
|
+
| `@agent-inspect/studio` | Yes (optional) | Local studio CLI / ingest surfaces |
|
|
29
|
+
|
|
30
|
+
Root `agent-inspect` uses **conditional exports** for ESM/CJS TypeScript consumers (`import.types` / `require.types`). Heavy framework dependencies stay in optional packages.
|
|
18
31
|
|
|
19
32
|
### Event model and schema
|
|
20
33
|
|
|
21
|
-
- Manual
|
|
22
|
-
- Failures use `step_completed` with `status: "error"` — there is no `step_failed` event.
|
|
34
|
+
- Manual global helpers (`inspectRun()` / `step()`) still write **`schemaVersion: "0.1"`** JSONL for compatibility (`run_started`, `step_started`, `step_completed`, `run_completed`).
|
|
35
|
+
- Failures use `step_completed` / `run_completed` with `status: "error"` — there is no `step_failed` event.
|
|
36
|
+
- **Persisted writer/runtime output targets schema 1.0.** `createInspector()` with built-in writers emits schema **1.0** persisted rows; v0.1 and v0.2 remain readable.
|
|
23
37
|
- Log-derived runs use confidence labels (`explicit`, `correlated`, `heuristic`, `unknown`) and conservative tree-building rules.
|
|
24
|
-
-
|
|
38
|
+
- Migration is explicit (`agent-inspect migrate …`); AgentInspect does not rewrite old traces in place.
|
|
25
39
|
|
|
26
40
|
See [SCHEMA.md](./SCHEMA.md) for field reference and [API.md](./API.md) for public surfaces (stable vs experimental).
|
|
27
41
|
|
|
@@ -29,21 +43,19 @@ See [SCHEMA.md](./SCHEMA.md) for field reference and [API.md](./API.md) for publ
|
|
|
29
43
|
|
|
30
44
|
- Instrumentation must **not throw into user code**; trace safety failures degrade gracefully.
|
|
31
45
|
- **Manual metadata** is redacted before disk by default; `redact: false` opts out.
|
|
46
|
+
- **Error messages** also run high-confidence free-text credential detectors before disk (provider keys, bearer tokens, JWTs, and related patterns), unless `redact: false`.
|
|
32
47
|
- **Size bounds** cap persisted event and metadata size.
|
|
33
48
|
- Log ingest: JSON first-class; log4js best-effort; no `eval` or JS object-literal parsing.
|
|
34
49
|
|
|
35
50
|
See [SECURITY.md](../SECURITY.md) and [LIMITATIONS.md](./LIMITATIONS.md).
|
|
36
51
|
|
|
37
|
-
### Optional adapters
|
|
38
|
-
|
|
39
|
-
- **`@agent-inspect/langchain`**: in-memory by default; `persist: true` writes local JSONL. Experimental.
|
|
40
|
-
- **`@agent-inspect/tui`**: interactive viewer; isolated from root deps. Experimental.
|
|
52
|
+
### Optional adapters and surfaces
|
|
41
53
|
|
|
42
|
-
|
|
54
|
+
Official adapters (`@agent-inspect/ai-sdk`, `@agent-inspect/openai-agents`, `@agent-inspect/langchain`) share a bounded preview-capture contract. MCP client tracing, reporters, eval, redact, viewer/TUI, studio, and read-only MCP are separate optional packages — see [ADAPTERS.md](./ADAPTERS.md) and [SUPPORT-LEVELS.md](./SUPPORT-LEVELS.md).
|
|
43
55
|
|
|
44
56
|
### Architecture proposals
|
|
45
57
|
|
|
46
|
-
Maintainer-owned RFCs and planning proposals are indexed in [proposals/README.md](./proposals/README.md). The active
|
|
58
|
+
Maintainer-owned RFCs and planning proposals are indexed in [proposals/README.md](./proposals/README.md). The active roadmap is [implementation/ROADMAP.md](./implementation/ROADMAP.md).
|
|
47
59
|
|
|
48
60
|
### Where to read next
|
|
49
61
|
|
package/docs/KNOWN-ISSUES.md
CHANGED
|
@@ -28,10 +28,10 @@ AgentInspect is **local-first** and **CLI-first**. These behaviors are intention
|
|
|
28
28
|
## Integrations
|
|
29
29
|
|
|
30
30
|
- **Vendor sinks** (hosted dashboards, Langfuse/Braintrust/New Relic/Datadog native uploads, OTLP gRPC streaming, etc.) are **not implemented** in the core packages described here.
|
|
31
|
-
- **AI SDK adapter** (`@agent-inspect/ai-sdk`) is
|
|
32
|
-
- **OpenAI Agents JS adapter** (`@agent-inspect/openai-agents`) is
|
|
31
|
+
- **AI SDK adapter** (`@agent-inspect/ai-sdk`) is Supported and metadata-first by default. It depends on explicit AI SDK telemetry configuration and requires `recordInputs: false` / `recordOutputs: false` for the documented safe path. Opt-in `capture: "preview"` uses the shared bounded preview helper.
|
|
32
|
+
- **OpenAI Agents JS adapter** (`@agent-inspect/openai-agents`) is Supported and metadata-first. Runtime metadata mapping is local-only; the safe install path is `setTraceProcessors()` rather than `addTraceProcessor()`. Bounded `capture: "preview"` parity ships with the other official adapters.
|
|
33
33
|
- **LangGraph support** is currently a documented boundary through `@agent-inspect/langchain`, not a dedicated package.
|
|
34
|
-
- **LangChain adapter** captures **metadata-oriented** signals by default
|
|
34
|
+
- **LangChain adapter** (`@agent-inspect/langchain`) is Supported; it captures **metadata-oriented** signals by default and shares the same preview contract when `capture: "preview"` is set.
|
|
35
35
|
- **LangChain `stream: true`** records chunk counts and timing only — not a full token replay. Per-token JSONL events are not emitted.
|
|
36
36
|
- **Deep swarm / nested RunnableSequence (6.14.2+):** capture resolves parents before registering child indexes and rejects `parentId === stepId`. Legacy traces that still contain self-parent edges are normalized in logical projection (`AI_LOGICAL_SELF_PARENT_REMOVED`) and rendered with visibility-first, cycle-safe trees so nested LLM/tool steps remain visible.
|
|
37
37
|
- **Correlation metadata** (`correlationId`, `requestId`, `decisionId`, `groupId`) is written on `run_started` but **CLI list/view does not filter by correlation fields** yet.
|
package/docs/LIMITATIONS.md
CHANGED
|
@@ -13,10 +13,10 @@ This document states what AgentInspect **does not** provide today. It complement
|
|
|
13
13
|
|
|
14
14
|
- **`list` / `view` / `export`** do not filter by correlation fields yet — `stats --correlation-id` / `--group-id` and `search` provide targeted read paths; full CLI filtering remains incremental.
|
|
15
15
|
|
|
16
|
-
## Persisted event model
|
|
16
|
+
## Persisted event model
|
|
17
17
|
|
|
18
18
|
- **Manual global tracing remains v0.1.** `inspectRun()` / `step()` still write `schemaVersion: "0.1"` JSONL for compatibility.
|
|
19
|
-
- **Persisted writer/runtime output targets schema 1.0.** `createInspector()` with built-in writers emits schema 1.0 persisted rows; v0.2
|
|
19
|
+
- **Persisted writer/runtime output targets schema 1.0.** `createInspector()` with built-in writers emits schema 1.0 persisted rows; v0.1 and v0.2 remain readable.
|
|
20
20
|
- **Migration is explicit, not automatic.** `agent-inspect migrate <input> --to 1.0 --dry-run` reports what would change, and `--output <file>` writes a separate file. AgentInspect does not rewrite old traces in place.
|
|
21
21
|
|
|
22
22
|
## Runtime writers and universal readers (v1.6)
|
|
@@ -32,7 +32,7 @@ This document states what AgentInspect **does not** provide today. It complement
|
|
|
32
32
|
|
|
33
33
|
- **AI SDK integration is explicit telemetry wiring.** Use `@agent-inspect/ai-sdk` through AI SDK `experimental_telemetry.integrations`; AgentInspect does not wrap providers, patch fetch, or enable telemetry globally.
|
|
34
34
|
- **AI SDK privacy settings are caller-owned.** Examples set `recordInputs: false` and `recordOutputs: false`; leaving those enabled in user code can cause the AI SDK telemetry layer to include richer data before AgentInspect receives events.
|
|
35
|
-
- **OpenAI Agents JS support is
|
|
35
|
+
- **OpenAI Agents JS support is Supported (metadata-first).** `@agent-inspect/openai-agents` maps runtime spans through the safe `setTraceProcessors()` boundary and does not capture raw payloads by default. Bounded `capture: "preview"` is available through the shared adapter capture contract.
|
|
36
36
|
- **LangGraph support is a boundary decision, not a separate package.** Initial support is expected through `@agent-inspect/langchain` callbacks unless no-network fixtures prove a separate package is needed.
|
|
37
37
|
- **No root/core adapter dependencies.** AI SDK, OpenAI Agents, LangGraph, OpenTelemetry, and LangChain remain outside the root/core runtime dependency graph.
|
|
38
38
|
- **Preview capture is bounded, not sanitized.** `capture: "preview"` persists truncated, key-redacted previews of framework-provided input/output fields. Key-based redaction cannot detect a secret embedded in free text, and there is no full-content capture mode. Fields the framework never exposes are reported as `AI_CAPTURE_FIELD_UNAVAILABLE` rather than reconstructed.
|
|
@@ -53,7 +53,8 @@ This document states what AgentInspect **does not** provide today. It complement
|
|
|
53
53
|
|
|
54
54
|
- **Redaction profiles** (`local`, `share`, `strict`) are key-based presets — not compliance-grade PII detection. Review exports before sharing even with `--redaction-profile strict`.
|
|
55
55
|
- **`@agent-inspect/redact` and `agent-inspect redact` create redacted copies.** They do not encrypt source traces, mutate originals, certify compliance, or guarantee every sensitive value is detected.
|
|
56
|
-
- **Default metadata redaction** covers common sensitive keys
|
|
56
|
+
- **Default metadata redaction** covers common sensitive keys (exact key match, case-insensitive). Custom secret field names are not redacted unless you add rules via `redact: { rules: [...] }`.
|
|
57
|
+
- **Error-message free text** also runs high-confidence credential detectors before disk (provider API keys, bearer tokens, JWTs, GitHub tokens, PEM private keys, and high-confidence `key=value` secrets). This is still not compliance-grade detection; `redact: false` opts out.
|
|
57
58
|
- **Metadata truncation** applies to string values and nested structures; very large metadata may be replaced with a truncation marker when `maxEventBytes` is exceeded (default 64 KiB per JSONL line).
|
|
58
59
|
- **Redaction is not encryption.** Local trace files remain readable on disk; treat `.agent-inspect-runs/` like any developer artifact that may contain operational data.
|
|
59
60
|
|
package/docs/PERFORMANCE.md
CHANGED
|
@@ -30,7 +30,7 @@ npx agent-inspect stats --dir fixtures/performance
|
|
|
30
30
|
|
|
31
31
|
All three are validated by `pnpm fixtures:check` and small enough for git (the largest is well under 1 MB).
|
|
32
32
|
|
|
33
|
-
## Comfortable ranges
|
|
33
|
+
## Comfortable ranges
|
|
34
34
|
|
|
35
35
|
| Workload | Comfortable | Warning | Not designed for |
|
|
36
36
|
| -------- | ----------- | ------- | ---------------- |
|
package/docs/SUPPORT-LEVELS.md
CHANGED
|
@@ -49,7 +49,8 @@ shows). `pnpm package-readmes:check` enforces the agreement and runs as part of
|
|
|
49
49
|
To promote or demote a surface:
|
|
50
50
|
|
|
51
51
|
1. Edit the row in the matrix above.
|
|
52
|
-
2. Edit the `**Support level:**` line in
|
|
52
|
+
2. Edit the `**Support level:**` line in the affected public README: `README.md`
|
|
53
|
+
for the root `agent-inspect` package, or `packages/*/README.md` for scoped packages.
|
|
53
54
|
3. Run `pnpm package-readmes:check`.
|
|
54
55
|
|
|
55
56
|
The check also reports packages whose level is **unenforced** — those the matrix
|
package/docs/TRACE-CONTRACTS.md
CHANGED
|
@@ -10,6 +10,9 @@ Contracts compile to deterministic check rules for common cases:
|
|
|
10
10
|
|
|
11
11
|
- run status / completion / max duration
|
|
12
12
|
- tool required / forbidden / allowed / maxCalls / order (`requiredTools` / `forbiddenTools` aliases)
|
|
13
|
+
- selectable `requiredOrderMode` (`first-occurrence` | `happens-before` | `all-occurrences`)
|
|
14
|
+
- `alternatives.anyOf` for one level of legitimate alternate paths
|
|
15
|
+
- `lintTraceContract` / `explainTraceContract` for brittle-contract diagnostics
|
|
13
16
|
- LLM maxCalls / maxTotalTokens / allowedModels
|
|
14
17
|
- evidence-bearing findings on failures
|
|
15
18
|
- evaluation over **logical** TraceFacts (raw events remain available)
|
|
@@ -24,13 +27,14 @@ Contracts compile to deterministic check rules for common cases:
|
|
|
24
27
|
→ contract.tool.order.1: B before C
|
|
25
28
|
```
|
|
26
29
|
|
|
27
|
-
|
|
30
|
+
`requiredOrderMode` selects one ordering relation for every generated pair:
|
|
28
31
|
|
|
29
32
|
- unlisted intermediate tools are allowed;
|
|
30
|
-
- later repetitions do not invalidate an earlier valid first-occurrence order;
|
|
31
33
|
- TraceContract `requiredOrder` **implies presence** — every listed name is added to the effective required-tool set;
|
|
32
|
-
-
|
|
33
|
-
-
|
|
34
|
+
- `first-occurrence` (default when omitted) compares first occurrences in start/encounter order; later repetitions do not invalidate an earlier valid order, and interval overlap emits a non-failing `tool.order.overlap` warning;
|
|
35
|
+
- `happens-before` requires the first `before` occurrence to finish before the first `after` occurrence starts;
|
|
36
|
+
- `all-occurrences` requires every `before` occurrence to finish before every `after` occurrence starts (`max(before.end) <= min(after.start)`);
|
|
37
|
+
- causal modes fail when a required interval boundary cannot be resolved instead of falling back to encounter order.
|
|
34
38
|
|
|
35
39
|
Examples for `requiredOrder: ["retrieve", "generate"]`:
|
|
36
40
|
|
|
@@ -38,12 +42,16 @@ Examples for `requiredOrder: ["retrieve", "generate"]`:
|
|
|
38
42
|
| --- | --- |
|
|
39
43
|
| `retrieve → generate` | PASS |
|
|
40
44
|
| `retrieve → rerank → generate` | PASS |
|
|
41
|
-
| `retrieve → generate → retrieve` | PASS
|
|
45
|
+
| `retrieve → generate → retrieve` | PASS under omitted / `first-occurrence`; FAIL under `all-occurrences` |
|
|
42
46
|
| `generate → retrieve` | FAIL (order) |
|
|
43
47
|
| `cache_lookup → generate` | FAIL (missing `retrieve` via implied presence) |
|
|
44
48
|
|
|
45
49
|
Low-level `createToolOrderingRule({ before, after })` alone may still pass when an endpoint is missing (compositional). TraceContract `requiredOrder` does not.
|
|
46
50
|
|
|
51
|
+
For overlapping first calls, omitted / `first-occurrence` warns while `happens-before` fails.
|
|
52
|
+
|
|
53
|
+
Immediate or positional `all-pairs` matching is not implemented.
|
|
54
|
+
|
|
47
55
|
### Experimental Vitest / Jest matchers (shipped)
|
|
48
56
|
|
|
49
57
|
| Package | Export | Matchers |
|
|
@@ -55,7 +63,7 @@ These are **Experimental** — API names may evolve. There is no `expectTrace(..
|
|
|
55
63
|
|
|
56
64
|
See [API.md](./API.md), [TRACE-FACTS.md](./TRACE-FACTS.md), and `packages/core/src/checks/contract.ts`.
|
|
57
65
|
|
|
58
|
-
## Rule kinds
|
|
66
|
+
## Rule kinds
|
|
59
67
|
|
|
60
68
|
TraceContract rules fall into distinct categories. Mixing them incorrectly is a common source of false failures (see GitHub #308 and #309).
|
|
61
69
|
|
|
@@ -65,65 +73,68 @@ Unconditional path invariant: every named tool must appear **at least once** in
|
|
|
65
73
|
|
|
66
74
|
- Use when the tool is always part of a valid execution path.
|
|
67
75
|
- **Do not** use for steps that legitimate shortcuts may skip (for example cache hits that bypass `retrieve`).
|
|
68
|
-
-
|
|
76
|
+
- Prefer `alternatives.anyOf` or `observations.required` when a shortcut is valid.
|
|
69
77
|
|
|
70
|
-
### `tools.requiredOrder` (shipped —
|
|
78
|
+
### `tools.requiredOrder` (shipped — selectable ordering modes)
|
|
71
79
|
|
|
72
|
-
|
|
80
|
+
The evaluator expands each list into adjacent pairs and applies one `requiredOrderMode` to every pair.
|
|
73
81
|
|
|
74
82
|
- TraceContract `requiredOrder` **implies presence** of every listed tool (unioned into `tools.required`).
|
|
75
|
-
-
|
|
76
|
-
-
|
|
77
|
-
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
83
|
+
- `requiredOrderMode: "first-occurrence"` is the default first-occurrence start/encounter relation; overlapping intervals emit a non-failing warning.
|
|
84
|
+
- `requiredOrderMode: "happens-before"` requires the first before to **end** before the first after **starts**; overlap fails.
|
|
85
|
+
- `requiredOrderMode: "all-occurrences"` requires every before to end before every after starts; any cross-boundary overlap or later before fails.
|
|
86
|
+
- Missing interval boundaries fail closed in the two causal modes.
|
|
87
|
+
|
|
88
|
+
### `alternatives.anyOf` (shipped)
|
|
89
|
+
|
|
90
|
+
One level of named deterministic branches. Base rules always apply. At least one complete branch must pass.
|
|
91
|
+
|
|
92
|
+
```ts
|
|
93
|
+
defineTraceContract({
|
|
94
|
+
run: { requireCompleted: true },
|
|
95
|
+
tools: { required: ["generate"] },
|
|
96
|
+
alternatives: {
|
|
97
|
+
anyOf: [
|
|
98
|
+
{
|
|
99
|
+
id: "cache-hit",
|
|
100
|
+
contract: {
|
|
101
|
+
tools: { required: ["cache_lookup"], forbidden: ["retrieve"] },
|
|
102
|
+
observations: { required: ["cache-hit-valid"] },
|
|
103
|
+
},
|
|
104
|
+
},
|
|
105
|
+
{
|
|
106
|
+
id: "retrieve",
|
|
107
|
+
contract: {
|
|
108
|
+
tools: { required: ["retrieve"], requiredOrder: ["retrieve", "generate"] },
|
|
109
|
+
observations: { required: ["retrieval-context-valid"] },
|
|
110
|
+
},
|
|
111
|
+
},
|
|
112
|
+
],
|
|
113
|
+
},
|
|
114
|
+
});
|
|
115
|
+
```
|
|
88
116
|
|
|
89
|
-
|
|
90
|
-
|---------------|---------|--------|
|
|
91
|
-
| `alternatives.anyOf` | One of several deterministic valid paths (one level, no nested groups, no predicates) | #309 |
|
|
92
|
-
| `requiredOrderMode: "happens-before"` | Causal completion-before-start ordering | #308 |
|
|
93
|
-
| `requiredOrderMode: "all-occurrences"` | Strict ordering across all tool occurrences | #308 |
|
|
117
|
+
Constraints:
|
|
94
118
|
|
|
95
|
-
|
|
119
|
+
- unique branch ids
|
|
120
|
+
- no nested `alternatives`
|
|
121
|
+
- no predicates / runtime DSL
|
|
122
|
+
- unused failed branches do not fail the contract when another branch passes
|
|
123
|
+
- if none pass → `contract.alternatives.none-satisfied`
|
|
96
124
|
|
|
97
|
-
|
|
125
|
+
### `observations.required` (shipped)
|
|
98
126
|
|
|
99
|
-
|
|
127
|
+
Requires externally observed or effect evidence (for example HTTP status, file write, cache key) rather than a specific tool call. Prefer this when the invariant is about **outcome** rather than **which tool ran**.
|
|
100
128
|
|
|
101
|
-
|
|
102
|
-
2. **Express** the verified outcome via `observations.required` when possible.
|
|
103
|
-
3. **Document** the cache-hit or alternate path in contract comments for reviewers.
|
|
129
|
+
### Lint and explain (shipped)
|
|
104
130
|
|
|
105
|
-
|
|
131
|
+
```ts
|
|
132
|
+
import { lintTraceContract, explainTraceContract } from "agent-inspect/checks";
|
|
106
133
|
|
|
107
|
-
|
|
108
|
-
contract
|
|
109
|
-
tools:
|
|
110
|
-
required: [generate] # not retrieve — cache may skip it
|
|
111
|
-
requiredOrder: [generate] # ordering only among tools that ran
|
|
112
|
-
observations:
|
|
113
|
-
required: [cache_hit_or_retrieve_evidence]
|
|
134
|
+
lintTraceContract(contract); // brittle / invalid shape diagnostics
|
|
135
|
+
explainTraceContract(contract); // human-readable intent lines
|
|
114
136
|
```
|
|
115
137
|
|
|
116
|
-
With first-occurrence ordering, `retrieve → generate → retrieve` still **passes** when both retrieves are present (see worked example below).
|
|
117
|
-
|
|
118
|
-
## What is not shipped (yet)
|
|
119
|
-
|
|
120
|
-
Do **not** document these as available:
|
|
121
|
-
|
|
122
|
-
- `expectTrace(...).toSatisfyTraceContract` (different API shape than the shipped matchers)
|
|
123
|
-
- Full workflow handoff / approval / MCP protocol contract rules
|
|
124
|
-
- Per-tool argument schema / regex validators on the contract surface
|
|
125
|
-
- Every structure rule (orphan/cycle/depth) exposed on the contract API (many exist as standalone check rules)
|
|
126
|
-
|
|
127
138
|
## CLI relationship
|
|
128
139
|
|
|
129
140
|
```bash
|
|
@@ -137,3 +148,4 @@ Suites and gates can consume check results; see [SUITES-COHORTS-GATES.md](./SUIT
|
|
|
137
148
|
- Experimental/Beta API — may evolve in minors
|
|
138
149
|
- Contract tests are smoke-level; prefer check-engine tests for deep rule coverage
|
|
139
150
|
- Always review findings before treating a green check as product proof
|
|
151
|
+
- No nested alternatives, all-pairs matching, or general temporal DSL
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-inspect",
|
|
3
|
-
"version": "6.
|
|
3
|
+
"version": "6.20.0",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"description": "Local evidence debugger and trajectory-test toolkit for TypeScript AI agents — execution trees, TraceContract checks, Evidence v2, and read-only MCP",
|
|
@@ -243,7 +243,8 @@
|
|
|
243
243
|
"public-truth:check": "node scripts/validate-public-truth.mjs",
|
|
244
244
|
"ai-assets:check": "node scripts/validate-ai-assets.mjs",
|
|
245
245
|
"package-readmes:check": "node scripts/validate-package-readmes.mjs",
|
|
246
|
-
"
|
|
246
|
+
"actions:check": "node scripts/check-github-actions-pins.mjs",
|
|
247
|
+
"docs:check": "pnpm run docs:commands && pnpm run docs:links && pnpm run public-truth:check && pnpm run ai-assets:check && pnpm run package-readmes:check && pnpm run repo:health && pnpm run actions:check && pnpm run build:cli && pnpm run demo:verify",
|
|
247
248
|
"build:cli": "pnpm exec tsup --config tsup.cli.config.ts",
|
|
248
249
|
"repo:health": "node scripts/validate-repo-health.mjs",
|
|
249
250
|
"demo:generate": "node scripts/demo-generate.mjs",
|
|
@@ -1714,9 +1714,63 @@ async function extractMetadata(filePath, _quickScan) {
|
|
|
1714
1714
|
createdAt: stats.birthtime
|
|
1715
1715
|
};
|
|
1716
1716
|
}
|
|
1717
|
+
var ROOT_STEP_DEPTH = 0;
|
|
1718
|
+
var MAX_RUN_SUMMARY_DEPTH = 1e3;
|
|
1717
1719
|
function isNonNegativeFiniteNumber(value) {
|
|
1718
1720
|
return typeof value === "number" && Number.isFinite(value) && value >= 0;
|
|
1719
1721
|
}
|
|
1722
|
+
function resolveParentStepId(step, steps) {
|
|
1723
|
+
const parentId = step.parentId;
|
|
1724
|
+
if (typeof parentId !== "string" || parentId.trim() === "") {
|
|
1725
|
+
return void 0;
|
|
1726
|
+
}
|
|
1727
|
+
return steps.has(parentId) ? parentId : void 0;
|
|
1728
|
+
}
|
|
1729
|
+
function computeStepDepth(stepId, steps, depthCache) {
|
|
1730
|
+
const cached = depthCache.get(stepId);
|
|
1731
|
+
if (cached !== void 0) return cached;
|
|
1732
|
+
const ancestry = [];
|
|
1733
|
+
const ancestryIndexes = /* @__PURE__ */ new Map();
|
|
1734
|
+
let currentStepId = stepId;
|
|
1735
|
+
let resolvedDepth;
|
|
1736
|
+
while (resolvedDepth === void 0) {
|
|
1737
|
+
const cachedDepth = depthCache.get(currentStepId);
|
|
1738
|
+
if (cachedDepth !== void 0) {
|
|
1739
|
+
resolvedDepth = cachedDepth;
|
|
1740
|
+
continue;
|
|
1741
|
+
}
|
|
1742
|
+
const cycleStart = ancestryIndexes.get(currentStepId);
|
|
1743
|
+
if (cycleStart !== void 0) {
|
|
1744
|
+
const cycleStepIds = ancestry.splice(cycleStart);
|
|
1745
|
+
for (const cycleStepId of cycleStepIds) {
|
|
1746
|
+
depthCache.set(cycleStepId, ROOT_STEP_DEPTH);
|
|
1747
|
+
}
|
|
1748
|
+
resolvedDepth = ROOT_STEP_DEPTH;
|
|
1749
|
+
continue;
|
|
1750
|
+
}
|
|
1751
|
+
const currentStep = steps.get(currentStepId);
|
|
1752
|
+
if (!currentStep) {
|
|
1753
|
+
resolvedDepth = ROOT_STEP_DEPTH;
|
|
1754
|
+
continue;
|
|
1755
|
+
}
|
|
1756
|
+
ancestryIndexes.set(currentStepId, ancestry.length);
|
|
1757
|
+
ancestry.push(currentStepId);
|
|
1758
|
+
const parentStepId = resolveParentStepId(currentStep, steps);
|
|
1759
|
+
if (parentStepId === void 0) {
|
|
1760
|
+
ancestry.pop();
|
|
1761
|
+
depthCache.set(currentStepId, ROOT_STEP_DEPTH);
|
|
1762
|
+
resolvedDepth = ROOT_STEP_DEPTH;
|
|
1763
|
+
continue;
|
|
1764
|
+
}
|
|
1765
|
+
currentStepId = parentStepId;
|
|
1766
|
+
}
|
|
1767
|
+
ancestry.reverse();
|
|
1768
|
+
for (const ancestorStepId of ancestry) {
|
|
1769
|
+
resolvedDepth = Math.min(MAX_RUN_SUMMARY_DEPTH, resolvedDepth + 1);
|
|
1770
|
+
depthCache.set(ancestorStepId, resolvedDepth);
|
|
1771
|
+
}
|
|
1772
|
+
return depthCache.get(stepId) ?? ROOT_STEP_DEPTH;
|
|
1773
|
+
}
|
|
1720
1774
|
function buildRunSummary(events) {
|
|
1721
1775
|
const started = events.find(
|
|
1722
1776
|
(e) => e.event === "run_started"
|
|
@@ -1770,27 +1824,13 @@ function buildRunSummary(events) {
|
|
|
1770
1824
|
let stepsWithKnownTotal = 0;
|
|
1771
1825
|
let hasCachedTokens = false;
|
|
1772
1826
|
const depthCache = /* @__PURE__ */ new Map();
|
|
1773
|
-
const computeDepth = (stepId) => {
|
|
1774
|
-
const cached = depthCache.get(stepId);
|
|
1775
|
-
if (cached !== void 0) return cached;
|
|
1776
|
-
const node = steps.get(stepId);
|
|
1777
|
-
if (!node) return 0;
|
|
1778
|
-
const parent = node.parentId;
|
|
1779
|
-
if (typeof parent !== "string" || parent.trim() === "" || !steps.has(parent)) {
|
|
1780
|
-
depthCache.set(stepId, 0);
|
|
1781
|
-
return 0;
|
|
1782
|
-
}
|
|
1783
|
-
const d = Math.min(1e3, computeDepth(parent) + 1);
|
|
1784
|
-
depthCache.set(stepId, d);
|
|
1785
|
-
return d;
|
|
1786
|
-
};
|
|
1787
1827
|
for (const [id, s] of steps.entries()) {
|
|
1788
1828
|
totalSteps += 1;
|
|
1789
1829
|
if (s.type === "llm") llmSteps += 1;
|
|
1790
1830
|
else if (s.type === "tool") toolSteps2 += 1;
|
|
1791
1831
|
else logicSteps += 1;
|
|
1792
1832
|
if (s.status === "error") errorSteps += 1;
|
|
1793
|
-
const depth =
|
|
1833
|
+
const depth = computeStepDepth(id, steps, depthCache);
|
|
1794
1834
|
if (depth > maxDepth) maxDepth = depth;
|
|
1795
1835
|
if (typeof s.durationMs === "number" && Number.isFinite(s.durationMs)) {
|
|
1796
1836
|
if (!longestStep || s.durationMs > longestStep.durationMs) {
|
|
@@ -11741,5 +11781,5 @@ function renderGateReport(result, options = {}) {
|
|
|
11741
11781
|
}
|
|
11742
11782
|
|
|
11743
11783
|
export { COHORT_METRIC_IDS, DEFAULT_SUITE_ARTIFACTS_DIR, EVIDENCE_FORMAT_VERSION, EVIDENCE_HTML_FILENAME, EVIDENCE_MANIFEST_FILENAME, Redactor, TraceDirectory, TraceReadError, TreeBuilder, aggregateBundleSafeStatus, aggregateSessionCheckResults, analyzeCohort, applyProfileMetadataCaps, assertBundlePathContained, assertEvidenceRelativePath, buildActivitySummary, buildBundleMetadata, buildBundleSummaryMarkdown, buildEvidenceCausalFailureViewHtml, buildEvidenceCiPackage, buildEvidenceCircuitViewHtml, buildEvidenceContractsViewHtml, buildEvidenceDiffViewHtml, buildEvidenceHtmlShell, buildEvidenceManifest, buildEvidenceOutcomesViewHtml, buildEvidenceProvenanceViewHtml, buildEvidenceSafetyViewHtml, buildEvidenceTimelineViewHtml, buildEvidenceToolsLlmViewHtml, buildEvidenceTreeViewHtml, buildLocalExplanation, buildPlaceholderArtifact, buildRunSummary, buildRunTimeline, buildRunWhatSummary, buildSessionIndex, buildTraceStats, buildZipArchive, bundleFailsOnSafety, bundleRunAssetRelativePath, collectTraceSchemaVersions, compactAttributes, createBaselineRegressionRule, createLlmUsageRule, createMaxStepDurationRule, createObservedOutcomeRule, createRequireCompletedRule, createRunDepthRule, createRunDurationRule, createRunStatusRule, createSafetyOversizedAttributeRule, createSafetyRawContentRule, createSafetyRedactionRule, createSafetySecretPatternRule, createStallDetectionRule, createStructureCycleRule, createStructureOrphanRule, createStructureParallelWidthRule, createStructureRelationshipRule, createToolUsageRule, defaultBundleOutputPath, defaultSuiteConfigTemplate, diffRuns, diffTraceEvents, enrichSessionRunRecord, escapeHtml, escapeMarkdown, extractMetadata, extractOutcomesFromTraceEvents, filterMetasBySessionScope, filterTraces, flattenTree, formatDuration2 as formatDuration, formatStepLabel, formatTimestamp, gateHasThresholds, getIndent, getTraceFilePath, inferEvidenceFileRole, isAgentInspectTrace, isPersistedInspectEvent, loadSessionRunRecords, loadSuiteConfig, loadTraceMetadataList, manualTraceEventsToComparableRun, nanoid, normalizeBundleOutputPath, openTrace, parseCohortMetricList, parseDuration, parseDurationFilter, parseGateList, parseTraceJsonl, persistedInspectEventsToTraceEvents, renderActivitySummaryHuman, renderCohortReport, renderErrorLine, renderGateReport, renderObservedOutcomesHtml, renderObservedOutcomesMarkdown, renderRunDiff, renderRunWhat, renderStepLine, renderSuiteReport, renderTimeline, renderTraceStats, resolveBundleRunIds, resolveRedactionProfile, resolveSuiteTemplate, resolveTraceDir, runGate, runSuite, runTraceChecks, safeString, sanitizeBundleRunId, searchTraces, serializeEvidenceManifest, sha256Hex, stableJson, summarizeObservedOutcomes, summarizeSemanticParity, traceEventToPersistedInspectEvent, truncateName, truncateStringForProfile, validateEvent, validateSuiteConfig, verifyEvidenceDirectory, zeroKinds };
|
|
11744
|
-
//# sourceMappingURL=chunk-
|
|
11745
|
-
//# sourceMappingURL=chunk-
|
|
11784
|
+
//# sourceMappingURL=chunk-QPRJUVLH.mjs.map
|
|
11785
|
+
//# sourceMappingURL=chunk-QPRJUVLH.mjs.map
|