agent-inspect 2.1.0 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +27 -0
- package/README.md +27 -3
- package/docs/ADAPTER-CONFORMANCE.md +25 -5
- package/docs/ADAPTERS.md +55 -11
- package/docs/API.md +29 -4
- package/docs/CLI.md +34 -7
- package/package.json +11 -1
- package/packages/cli/dist/index.cjs +461 -93
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +460 -92
- package/packages/cli/dist/index.mjs.map +1 -1
- package/packages/core/dist/reporters.cjs +185 -0
- package/packages/core/dist/reporters.cjs.map +1 -0
- package/packages/core/dist/reporters.d.cts +91 -0
- package/packages/core/dist/reporters.d.ts +91 -0
- package/packages/core/dist/reporters.mjs +175 -0
- package/packages/core/dist/reporters.mjs.map +1 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,32 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.3.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 22cad5a: Release v2.3.0 with hardened framework adapter paths.
|
|
8
|
+
|
|
9
|
+
This train strengthens the official AI SDK, OpenAI Agents JS, and LangChain/LangGraph integrations with no-network fixtures, clearer local-only defaults, adapter conformance evidence, and adoption-ready recipes. Mastra and NestJS framework packages remain demand-gated; NestJS stays on the structured-log ingestion recipe path for this release.
|
|
10
|
+
|
|
11
|
+
## 2.2.0
|
|
12
|
+
|
|
13
|
+
### Minor Changes
|
|
14
|
+
|
|
15
|
+
- efb3fef: Release v2.2.0 with local test reporter artifacts and CI summaries.
|
|
16
|
+
|
|
17
|
+
Adds the public optional `@agent-inspect/vitest` and `@agent-inspect/jest` reporter packages, the shared experimental `agent-inspect/reporters` helpers, and the `agent-inspect ci-summary` workflow for deterministic local reporter manifests and CI artifacts.
|
|
18
|
+
|
|
19
|
+
## Unreleased
|
|
20
|
+
|
|
21
|
+
### Draft v2.3.0 Notes
|
|
22
|
+
|
|
23
|
+
- Hardened the official adapter paths for AI SDK, OpenAI Agents JS, and LangChain/LangGraph with no-network recipes, local-only defaults, clearer lifecycle coverage, and executable adapter conformance evidence.
|
|
24
|
+
- AI SDK coverage now includes route-style telemetry factory guidance, per-request integration isolation, tool/stream/error/parallel fixtures, token metadata, and the required `recordInputs: false` / `recordOutputs: false` host settings.
|
|
25
|
+
- OpenAI Agents JS documentation and fixtures distinguish local-only replacement via `setTraceProcessors()` from advanced additional processor usage.
|
|
26
|
+
- LangGraph support remains through `@agent-inspect/langchain`, with graph/node identity, subgraphs, checkpoint/session IDs, stream modes, handoffs, and parallel branch hints covered through callback metadata.
|
|
27
|
+
- Mastra and NestJS framework packages remain explicitly deferred. NestJS support stays on structured-log ingestion unless future demand proves a narrow local-only helper is worth maintaining.
|
|
28
|
+
- No root/core framework dependency, hosted upload, provider call, schema change, or public breaking change is added in this train.
|
|
29
|
+
|
|
3
30
|
## 2.1.0
|
|
4
31
|
|
|
5
32
|
### Minor Changes
|
package/README.md
CHANGED
|
@@ -22,7 +22,7 @@ agent-inspect gives those runs **structure**: an **execution tree** you can read
|
|
|
22
22
|
|
|
23
23
|
## Install
|
|
24
24
|
|
|
25
|
-
Current npm release line: **2.
|
|
25
|
+
Current npm release line: **2.1.x** for the existing public packages. v2.1.0 adds deterministic local eval and reusable redaction utilities on top of the stable v2 trace contract: small root API, schema 1.0 persisted writer path, v0.1/v0.2/v1.0 read compatibility, and explicit non-destructive migration workflow.
|
|
26
26
|
|
|
27
27
|
```bash
|
|
28
28
|
npm install agent-inspect
|
|
@@ -240,6 +240,7 @@ AGENT_INSPECT=1 node eval-runner.mjs
|
|
|
240
240
|
- **Redact local files** with `agent-inspect redact` or `@agent-inspect/redact` before creating shareable copies.
|
|
241
241
|
- **Migrate explicitly** with `agent-inspect migrate <trace.jsonl> --to 1.0 --dry-run` or `--output <file>`; originals are never overwritten by default.
|
|
242
242
|
- **Export share-safe copies** — `export --redaction-profile share` (or `strict`) writes local Markdown/HTML/OpenInference/OTLP JSON only.
|
|
243
|
+
- **Create local CI artifacts** with `agent-inspect artifacts`, and summarize local test-reporter manifests with `agent-inspect ci-summary`.
|
|
243
244
|
- **Parse structured logs** you already emit (JSON first-class; log4js best-effort).
|
|
244
245
|
- **Optional LangChain adapter** — metadata-only by default; optional `persist: true` and `stream: true` streaming metadata (no full token capture by default).
|
|
245
246
|
- **Optional AI SDK adapter** — experimental `@agent-inspect/ai-sdk` telemetry integration for AI SDK v6; metadata-only by default with `recordInputs: false` and `recordOutputs: false`.
|
|
@@ -308,6 +309,7 @@ More detail: [docs/LOGS.md](docs/LOGS.md) · [docs/LOG-TO-TREE-QUICKSTART.md](do
|
|
|
308
309
|
| `report` | Markdown/HTML inspection report (what + timeline + tree) |
|
|
309
310
|
| `check` / `scan` / `verify-safe` | Deterministic local trace checks and best-effort safety verification |
|
|
310
311
|
| `artifacts` | Safe local CI artifact bundles and optional step-summary file output |
|
|
312
|
+
| `ci-summary` | Summarize local Vitest/Jest reporter artifact manifests for CI |
|
|
311
313
|
|
|
312
314
|

|
|
313
315
|
|
|
@@ -338,6 +340,12 @@ AgentInspect is the **local-first trace workbench** for TypeScript AI agents:
|
|
|
338
340
|
|
|
339
341
|
Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInspectRun` with `AGENT_INSPECT=1` to toggle tracing in eval or CI — see [docs/API.md](docs/API.md).
|
|
340
342
|
|
|
343
|
+
**v2.3 train status:** adapter hardening is ready for release prep. AI SDK, OpenAI Agents JS, and LangChain/LangGraph paths have no-network recipes and executable conformance coverage. Mastra and NestJS framework packages remain demand-gated; NestJS is covered through structured-log ingestion.
|
|
344
|
+
|
|
345
|
+
**Shipped in 2.2.0:** public optional Vitest/Jest reporter packages, shared `agent-inspect/reporters` helpers, and `agent-inspect ci-summary` for deterministic local reporter artifact summaries. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, `@agent-inspect/tui`, `@agent-inspect/openai-agents`, `@agent-inspect/redact`, `@agent-inspect/eval`, `@agent-inspect/vitest`, and `@agent-inspect/jest` at **2.2.0**.
|
|
346
|
+
|
|
347
|
+
**Shipped in 2.1.0:** deterministic local eval and redaction utilities. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, `@agent-inspect/tui`, `@agent-inspect/openai-agents`, `@agent-inspect/redact`, and `@agent-inspect/eval` at **2.1.0**.
|
|
348
|
+
|
|
341
349
|
**Shipped in 2.0.0:** stable root API contract, schema 1.0 persisted writer path, v0.1/v0.2/v1.0 read compatibility, and explicit trace migration workflow. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, `@agent-inspect/tui`, and `@agent-inspect/openai-agents` at **2.0.0**.
|
|
342
350
|
|
|
343
351
|
**Shipped in 1.9.0:** private harness workspace foundation, explain dry-run/local analysis, promoted adapter adoption paths, and the v2 root API slimming plan.
|
|
@@ -350,7 +358,7 @@ Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInsp
|
|
|
350
358
|
|
|
351
359
|
**Shipped in 1.5.0:** non-breaking subpath exports; `what` and `report` CLI; dual-format read path (v0.1 + v0.2 JSONL); [what-report-inspect recipe](examples/recipes/what-report-inspect/). Linked release aligns all three npm packages at **1.5.0**.
|
|
352
360
|
|
|
353
|
-
**Roadmap beyond current release work:** v2.
|
|
361
|
+
**Roadmap beyond current release work:** v2.4 adds sessions/MCP telemetry, followed by guardrails, optional viewer/IDE surfaces, and conditional v3 extensibility. See [ROADMAP.md](ROADMAP.md).
|
|
354
362
|
|
|
355
363
|
**Shipped in 1.4.0:** CI artifact recipe ([docs/CI-ARTIFACTS.md](docs/CI-ARTIFACTS.md)); `timeline`, `stats`, and `search` CLI; core helpers `buildRunTimeline`, `buildTraceStats`, `searchTraces`. Linked release aligns all three npm packages at **1.4.0**.
|
|
356
364
|
|
|
@@ -366,6 +374,16 @@ Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInsp
|
|
|
366
374
|
|
|
367
375
|
## Optional packages
|
|
368
376
|
|
|
377
|
+
### Framework adapters (`@agent-inspect/ai-sdk`, `@agent-inspect/openai-agents`, `@agent-inspect/langchain`)
|
|
378
|
+
|
|
379
|
+
Official framework adapters are optional packages and stay explicit, local-first, and metadata-only by default:
|
|
380
|
+
|
|
381
|
+
- **AI SDK:** pass `agentInspect(...)` through AI SDK telemetry with `recordInputs: false` and `recordOutputs: false`.
|
|
382
|
+
- **OpenAI Agents JS:** use `setTraceProcessors([agentInspectProcessor(...)])` for the documented local-only replacement path.
|
|
383
|
+
- **LangChain/LangGraph:** pass `new AgentInspectCallback(...)` through callbacks; LangGraph support rides through the LangChain callback boundary.
|
|
384
|
+
|
|
385
|
+
No-network recipes: [ai-sdk-local-telemetry](examples/recipes/ai-sdk-local-telemetry/), [ai-sdk-next-route](examples/recipes/ai-sdk-next-route/), [openai-agents-local-tracing](examples/recipes/openai-agents-local-tracing/), and [langgraph-callback-local](examples/recipes/langgraph-callback-local/). Conformance and limits are documented in [docs/ADAPTERS.md](docs/ADAPTERS.md) and [docs/ADAPTER-CONFORMANCE.md](docs/ADAPTER-CONFORMANCE.md).
|
|
386
|
+
|
|
369
387
|
### LangChain callback adapter (`@agent-inspect/langchain`)
|
|
370
388
|
|
|
371
389
|
Optional package: official **LangChain.js callbacks** (`BaseCallbackHandler`), **metadata-oriented by default**, **no monkey-patching**, **no vendor sink**. Optional **`stream: true`** records chunk counts and stream duration **without storing full token text by default**. The LangChain adapter ships with 1.x; its programmatic API remains experimental and may evolve independently of the stable core tracing API.
|
|
@@ -405,6 +423,12 @@ npx agent-inspect view <run-id> --tui
|
|
|
405
423
|
|
|
406
424
|
The TUI is available as a separate optional package; its programmatic API is experimental, while the CLI integration (`view --tui`) is the intended usage. Details: [docs/ADAPTERS.md](docs/ADAPTERS.md).
|
|
407
425
|
|
|
426
|
+
### Test reporter artifacts (`@agent-inspect/vitest`, `@agent-inspect/jest`)
|
|
427
|
+
|
|
428
|
+
Optional Vitest/Jest reporter packages are public as of v2.2. They write shared `schemaVersion: "0.1"` reporter manifests with safe relative artifact paths and bounded structural metadata. Use `agent-inspect ci-summary` to summarize those local manifests in CI without reading trace contents or calling GitHub APIs.
|
|
429
|
+
|
|
430
|
+
Reporter artifact behavior and API details are documented in [docs/API.md](docs/API.md) and [docs/CI-ARTIFACTS.md](docs/CI-ARTIFACTS.md).
|
|
431
|
+
|
|
408
432
|
## Examples and recipes
|
|
409
433
|
|
|
410
434
|
| Example | Shows |
|
|
@@ -431,7 +455,7 @@ The TUI is available as a separate optional package; its programmatic API is exp
|
|
|
431
455
|
| [examples/recipes/eval-local-checks](examples/recipes/eval-local-checks) | v2.1 deterministic local eval checks |
|
|
432
456
|
| [examples/recipes/redact-share-safe-file](examples/recipes/redact-share-safe-file) | v2.1 share-safe local redaction copy |
|
|
433
457
|
| [examples/recipes/eval-ci-artifacts](examples/recipes/eval-ci-artifacts) | v2.1 eval before safe CI artifacts |
|
|
434
|
-
| [examples/recipes/test-reporter-artifacts](examples/recipes/test-reporter-artifacts) |
|
|
458
|
+
| [examples/recipes/test-reporter-artifacts](examples/recipes/test-reporter-artifacts) | Vitest/Jest reporter artifact patterns |
|
|
435
459
|
| [examples/recipes/what-report-inspect](examples/recipes/what-report-inspect/) | `what` + `report` inspection |
|
|
436
460
|
| [examples/recipes/runtime-and-ingestion](examples/recipes/runtime-and-ingestion/) | v1.6 runtime writers + universal ingestion |
|
|
437
461
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
AgentInspect optional framework adapters must stay local-first, dependency-isolated, and metadata-only by default.
|
|
4
4
|
|
|
5
|
-
The machine-readable matrix lives at [docs/implementation/adapter-conformance-matrix.json](./implementation/adapter-conformance-matrix.json).
|
|
5
|
+
The machine-readable matrix lives at [docs/implementation/adapter-conformance-matrix.json](./implementation/adapter-conformance-matrix.json). The release-evidence fixture manifest lives at [docs/implementation/adapter-conformance-fixtures.json](./implementation/adapter-conformance-fixtures.json). These files track expected coverage for:
|
|
6
6
|
|
|
7
7
|
- run lifecycle
|
|
8
8
|
- generic steps
|
|
@@ -12,19 +12,39 @@ The machine-readable matrix lives at [docs/implementation/adapter-conformance-ma
|
|
|
12
12
|
- streaming metadata
|
|
13
13
|
- metadata bounds and privacy controls
|
|
14
14
|
|
|
15
|
-
The v1.7 matrix
|
|
15
|
+
The v1.7 matrix began as declarative coverage guidance. v1.8 made the core conformance path executable and requires canonical-reader round trips before adapter output is used by checks. v2.3 uses that foundation to harden official adapters without adding shallow new adapter packages.
|
|
16
16
|
|
|
17
17
|
Executable shared assertions live in `packages/core/test/adapter-executable-conformance.test.ts` and `packages/core/test/adapter-conformance-utils.ts`. Adapter-specific suites may add deeper fixture coverage, but the shared suite owns the cross-adapter defaults: local-only execution, no raw payload persistence, lifecycle identity, parentage, streaming summaries, token usage where exposed, and reader round trips.
|
|
18
18
|
|
|
19
|
+
Release evidence gate:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
pnpm exec vitest run packages/core/test/adapter-executable-conformance.test.ts packages/core/test/adapter-conformance-matrix.test.ts
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
This is an internal official-adapter gate, not a third-party certification program.
|
|
26
|
+
|
|
19
27
|
## Current matrix
|
|
20
28
|
|
|
21
29
|
| Adapter | Package | Status | Default install mode | Boundary |
|
|
22
30
|
| --- | --- | --- | --- | --- |
|
|
23
|
-
| AI SDK | `@agent-inspect/ai-sdk` | implemented experimental;
|
|
24
|
-
|
|
|
25
|
-
|
|
|
31
|
+
| AI SDK | `@agent-inspect/ai-sdk` | implemented experimental; v2.3 priority 1 | AI SDK telemetry integration | optional package peer dependency |
|
|
32
|
+
| OpenAI Agents JS | `@agent-inspect/openai-agents` | implemented experimental; v2.3 priority 2 | `setTraceProcessors()` replacement | optional package peer dependency |
|
|
33
|
+
| LangChain | `@agent-inspect/langchain` | implemented experimental; v2.3 priority 3 | explicit callback | optional package peer dependency |
|
|
26
34
|
| LangGraph | `@agent-inspect/langchain` | fixture-backed through LangChain callback | explicit LangChain callback | existing LangChain adapter first |
|
|
27
35
|
|
|
36
|
+
## v2.3 scorecard
|
|
37
|
+
|
|
38
|
+
| Adapter path | Current coverage | Hardening gap | Decision |
|
|
39
|
+
| ------------ | ---------------- | ------------- | -------- |
|
|
40
|
+
| AI SDK | Shared conformance marks run, step, tool, LLM, error, streaming, and metadata bounds covered. | More adoption-grade fixtures for `generateText`, `streamText`, tool calls, Next.js route usage, parallel calls, abort/error lifecycle, and token metadata. | Harden first. |
|
|
41
|
+
| OpenAI Agents JS | Shared conformance covers run, step, tool, LLM, error, metadata bounds, and local-only replacement metadata; adapter fixtures cover agents, generations, tools, handoffs, guardrails, response, MCP tools, custom, transcription, and speech shapes without provider calls. | Streaming remains planned in the matrix; `addTraceProcessor()` remains documented only as an advanced user-owned additional mode. | Hardened second. |
|
|
42
|
+
| LangChain/LangGraph | LangChain and LangGraph-through-LangChain fixtures cover shared signals plus graph/node identity, subgraphs, checkpoint/session IDs, stream modes, handoffs, and parallel branch hints through callback metadata. | Callback-surface limits remain documented; no separate package is justified yet. | Hardened third. |
|
|
43
|
+
| Mastra | No official package, no conformance fixture, no root dependency. | Demand and extension-point evidence are not yet sufficient for local-only useful traces. | Defer. |
|
|
44
|
+
| NestJS | Structured-log recipe exists; no official framework adapter or conformance fixture. | Current evidence supports log ingestion. Demand may justify a narrow harness/bootstrap helper, not a broad monkey-patching adapter. | Defer package; keep recipe/helper gate. |
|
|
45
|
+
|
|
46
|
+
`@agent-inspect/vitest` and `@agent-inspect/jest` are public reporter packages, not framework trace adapters. Their artifact-manifest behavior is covered by reporter tests and CI-summary tests rather than this adapter conformance matrix.
|
|
47
|
+
|
|
28
48
|
## Required defaults
|
|
29
49
|
|
|
30
50
|
- No network behavior.
|
package/docs/ADAPTERS.md
CHANGED
|
@@ -2,11 +2,25 @@
|
|
|
2
2
|
|
|
3
3
|
AgentInspect is **framework-agnostic** at its core. Optional adapter packages integrate specific frameworks without monkey-patching, vendor sinks, or network upload.
|
|
4
4
|
|
|
5
|
+
## v2.3 hardening scorecard
|
|
6
|
+
|
|
7
|
+
v2.3 hardens existing official adapters before adding new ones. Priority is based on the current package set, open issue/adoption signals, conformance coverage, and how directly a framework can produce useful local traces without root/core dependencies.
|
|
8
|
+
|
|
9
|
+
| Priority | Adapter path | Decision | v2.3 focus |
|
|
10
|
+
| -------- | ------------ | -------- | ---------- |
|
|
11
|
+
| 1 | AI SDK (`@agent-inspect/ai-sdk`) | Harden first | Improve low-friction `generateText`, `streamText`, tool-call, parallel-call, abort/error, token metadata, and Next.js route coverage while keeping `recordInputs: false` and `recordOutputs: false` as required host controls. |
|
|
12
|
+
| 2 | OpenAI Agents JS (`@agent-inspect/openai-agents`) | Harden second | Make local-only replacement vs additional processor modes unmistakable, with fixtures for agents, generations, tools, handoffs, guardrails, and no default upload confusion. |
|
|
13
|
+
| 3 | LangChain/LangGraph (`@agent-inspect/langchain`) | Harden third | Improve LangGraph-through-LangChain mapping for node identity, subgraphs, checkpoints, stream modes, branches, handoffs, and session/thread IDs without adding a separate package unless the callback surface proves insufficient. |
|
|
14
|
+
| Defer | Mastra | No package in v2.3 | Current evidence does not justify an official package. Revisit only when there is explicit user demand and a verified extension point that avoids hidden monkey-patching, hosted sinks, or root dependencies. |
|
|
15
|
+
| Defer | NestJS | No framework adapter in v2.3 | Keep the supported path at structured-log ingestion via the existing NestJS JSON logging recipe. Revisit a narrow harness/bootstrap helper only with concrete demand; do not add a package only to wrap app bootstrap. |
|
|
16
|
+
|
|
17
|
+
Reporters (`@agent-inspect/vitest` and `@agent-inspect/jest`) are public packages as of v2.2, but they are CI/test artifact reporters rather than framework trace adapters. They stay outside the v2.3 adapter-hardening priority order.
|
|
18
|
+
|
|
5
19
|
## Vercel AI SDK (`@agent-inspect/ai-sdk`)
|
|
6
20
|
|
|
7
|
-
**Status:** experimental adapter — optional package published in the aligned
|
|
21
|
+
**Status:** experimental adapter — optional package published in the aligned v2.2.0 package set and hardened in the v2.3 adapter train.
|
|
8
22
|
|
|
9
|
-
The
|
|
23
|
+
The adapter has hardened lifecycle identity and parallel integration isolation. It remains metadata-only: `capture: "preview"` and preview-only redaction options emit diagnostics and fall back to metadata-only capture until bounded free-text previews are implemented.
|
|
10
24
|
|
|
11
25
|
### Install
|
|
12
26
|
|
|
@@ -49,9 +63,11 @@ const result = await generateText({
|
|
|
49
63
|
|
|
50
64
|
[examples/recipes/ai-sdk-local-telemetry](../examples/recipes/ai-sdk-local-telemetry/) uses AI SDK test utilities only (`MockLanguageModelV3`, `simulateReadableStream`) and writes local v0.2 adapter events for `agent-inspect open`.
|
|
51
65
|
|
|
66
|
+
[examples/recipes/ai-sdk-next-route](../examples/recipes/ai-sdk-next-route/) shows a route-style telemetry factory that creates one AgentInspect integration per request while keeping the same no-network, metadata-only defaults.
|
|
67
|
+
|
|
52
68
|
### Common host shapes
|
|
53
69
|
|
|
54
|
-
Use the same explicit telemetry block for route handlers, streaming, and tool calls. The adapter does not wrap providers or change host-call settings.
|
|
70
|
+
Use the same explicit telemetry block shape for route handlers, streaming, and tool calls. Create a fresh `agentInspect(...)` integration per concurrent request/generation. The adapter does not wrap providers or change host-call settings.
|
|
55
71
|
|
|
56
72
|
```ts
|
|
57
73
|
const telemetry = {
|
|
@@ -83,7 +99,7 @@ Full API: [API.md](./API.md) §11.
|
|
|
83
99
|
|
|
84
100
|
## LangChain.js (`@agent-inspect/langchain`)
|
|
85
101
|
|
|
86
|
-
**Status:** experimental — programmatic API may evolve independently of stable core tracing.
|
|
102
|
+
**Status:** experimental adapter — optional package published in the aligned v2.2.0 package set; programmatic API may evolve independently of stable core tracing.
|
|
87
103
|
|
|
88
104
|
### Install
|
|
89
105
|
|
|
@@ -209,7 +225,7 @@ Full API: [API.md](./API.md) §9.
|
|
|
209
225
|
|
|
210
226
|
### LangGraph boundary
|
|
211
227
|
|
|
212
|
-
LangGraph support
|
|
228
|
+
LangGraph support rides through this same `@agent-inspect/langchain` callback boundary first. The v2.3 fixtures cover graph/node identity, subgraphs, checkpoint/session IDs, stream modes, handoffs, and parallel branch hints without adding a separate package. A dedicated LangGraph package remains deferred until fixtures prove that LangGraph exposes important lifecycle data unavailable through LangChain callbacks.
|
|
213
229
|
|
|
214
230
|
Future LangGraph examples must keep the same safety defaults: explicit callback installation, metadata-only capture, no raw prompt/output/tool payload capture by default, no hosted sink, and local persistence only when `persist: true` is set.
|
|
215
231
|
|
|
@@ -236,7 +252,7 @@ Requires an interactive terminal. See [API.md](./API.md) §10.
|
|
|
236
252
|
|
|
237
253
|
## Vitest (`@agent-inspect/vitest`)
|
|
238
254
|
|
|
239
|
-
**Status:** experimental
|
|
255
|
+
**Status:** experimental reporter package — optional package published in the aligned v2.2.0 package set.
|
|
240
256
|
|
|
241
257
|
```bash
|
|
242
258
|
npm install agent-inspect @agent-inspect/vitest vitest
|
|
@@ -285,7 +301,7 @@ Full API: [API.md](./API.md) §12.
|
|
|
285
301
|
|
|
286
302
|
## Jest (`@agent-inspect/jest`)
|
|
287
303
|
|
|
288
|
-
**Status:** experimental
|
|
304
|
+
**Status:** experimental reporter package — optional package published in the aligned v2.2.0 package set.
|
|
289
305
|
|
|
290
306
|
```bash
|
|
291
307
|
npm install agent-inspect @agent-inspect/jest jest
|
|
@@ -328,7 +344,7 @@ Full API: [API.md](./API.md) §13.
|
|
|
328
344
|
|
|
329
345
|
## OpenAI Agents JS (`@agent-inspect/openai-agents`)
|
|
330
346
|
|
|
331
|
-
**Status:** experimental adapter — optional package published in the aligned
|
|
347
|
+
**Status:** experimental adapter — optional package published in the aligned v2.2.0 package set.
|
|
332
348
|
|
|
333
349
|
The safe integration boundary is documented in [OPENAI-AGENTS-JS-TRACING.md](./proposals/OPENAI-AGENTS-JS-TRACING.md). Install the AgentInspect processor by replacing processors:
|
|
334
350
|
|
|
@@ -348,14 +364,15 @@ Do not use `addTraceProcessor()` as the default AgentInspect path; that preserve
|
|
|
348
364
|
|
|
349
365
|
Integration modes:
|
|
350
366
|
|
|
351
|
-
- **Local-only replacement:** `setTraceProcessors([agentInspectProcessor(...)])` replaces existing processors for the current process. This is the documented safe default when you want AgentInspect to own local trace output.
|
|
352
|
-
- **Additional processor:** `addTraceProcessor(agentInspectProcessor(...))` is an advanced, user-owned choice. It can preserve existing/default processors and any backend export behavior they already perform.
|
|
367
|
+
- **Local-only replacement:** `setTraceProcessors([agentInspectProcessor(...)])` replaces existing processors for the current process. This is the documented safe default when you want AgentInspect to own local trace output and avoid preserving the SDK default exporter.
|
|
368
|
+
- **Additional processor:** `addTraceProcessor(agentInspectProcessor(...))` is an advanced, user-owned choice. It can preserve existing/default processors and any backend export behavior they already perform; use it only when that is intentional.
|
|
353
369
|
|
|
354
370
|
- **No auto-install** — importing or constructing `agentInspectProcessor()` never calls `setTraceProcessors()` or `addTraceProcessor()`.
|
|
355
371
|
- **No upload behavior** — the processor writes only to an explicit local writer or `traceDir`.
|
|
356
372
|
- **Metadata-only by default** — records trace/span IDs, parentage, names, timing, status, errors, safe model/tool names, token counts, and bounded summaries.
|
|
357
373
|
- **No raw payload capture by default** — prompts, messages, generated text, function inputs/outputs, arbitrary custom data, trace exporter credentials, headers, request bodies, response bodies, and hosted tool payloads are not persisted.
|
|
358
374
|
- **Preview capture is not enabled yet** — `capture: "preview"`, `redactionProfile`, and `maxPreviewChars` are diagnosed through `getDiagnostics()` and do not persist raw previews.
|
|
375
|
+
- **Fixture-backed lifecycle coverage** — local tests and the recipe cover agent, generation, function tool, handoff, guardrail, response, MCP tools, custom, transcription, and speech span shapes without provider calls.
|
|
359
376
|
|
|
360
377
|
Full API: [API.md](./API.md) §14.
|
|
361
378
|
|
|
@@ -363,11 +380,38 @@ Runnable local recipe: [openai-agents-local-tracing](../examples/recipes/openai-
|
|
|
363
380
|
|
|
364
381
|
---
|
|
365
382
|
|
|
383
|
+
## Demand-gated framework decisions
|
|
384
|
+
|
|
385
|
+
### Mastra
|
|
386
|
+
|
|
387
|
+
**v2.3 decision:** no official package, recipe, or conformance fixture.
|
|
388
|
+
|
|
389
|
+
The v2.3 evidence review found no open adapter request and no verified extension point in this repository that would produce useful local AgentInspect traces without hidden framework patching or new root/core dependencies. A future Mastra path must first prove:
|
|
390
|
+
|
|
391
|
+
- explicit user demand, such as an issue, design-partner request, or retained external recipe;
|
|
392
|
+
- a stable, framework-native callback/export hook;
|
|
393
|
+
- metadata-only local trace output with no hosted upload or provider calls by default;
|
|
394
|
+
- no dependency leakage into `agent-inspect` root or `@agent-inspect/core`;
|
|
395
|
+
- no raw prompt, output, tool payload, header, or request/response capture by default.
|
|
396
|
+
|
|
397
|
+
Until those are true, Mastra stays outside the official adapter set rather than shipping a shallow package.
|
|
398
|
+
|
|
399
|
+
### NestJS
|
|
400
|
+
|
|
401
|
+
**v2.3 decision:** no official framework adapter package.
|
|
402
|
+
|
|
403
|
+
NestJS remains covered through structured-log ingestion, not app bootstrap wrapping. The supported recipe is [examples/recipes/nestjs-json-logging](../examples/recipes/nestjs-json-logging), which maps Nest-shaped JSON lines into local execution trees without importing `@nestjs/*`, starting an HTTP server, or changing application behavior.
|
|
404
|
+
|
|
405
|
+
A future Nest helper remains demand-gated and must be narrower than a framework adapter. Acceptable evidence would be a repeated need to reduce logging/harness setup friction while preserving explicit opt-in, local-only output, and zero root/core Nest dependency. Broad interceptors, automatic module scanning, monkey-patching, request body capture, or default telemetry upload remain out of scope.
|
|
406
|
+
|
|
407
|
+
---
|
|
408
|
+
|
|
366
409
|
## Future adapters (not shipped)
|
|
367
410
|
|
|
368
411
|
Direction only — see [ROADMAP.md](../ROADMAP.md):
|
|
369
412
|
|
|
370
|
-
- **NestJS
|
|
413
|
+
- **NestJS helper patterns** — only if demand proves a narrow harness/bootstrap helper is worth maintaining beyond [LOGGING-PLAYBOOK.md](./LOGGING-PLAYBOOK.md) and the current NestJS logging recipe.
|
|
414
|
+
- **Mastra** — deferred until demand and extension-point evidence justify a narrow explicit integration.
|
|
371
415
|
|
|
372
416
|
No automatic universal instrumentation. Integrations remain explicit and opt-in.
|
|
373
417
|
|
package/docs/API.md
CHANGED
|
@@ -222,9 +222,32 @@ No network writer, OpenTelemetry exporter, provider wrapper, or global monkey-pa
|
|
|
222
222
|
|
|
223
223
|
Recipe: [examples/recipes/ai-sdk-local-telemetry](../examples/recipes/ai-sdk-local-telemetry/).
|
|
224
224
|
|
|
225
|
+
## 11.1 Experimental `agent-inspect/reporters` APIs
|
|
226
|
+
|
|
227
|
+
`agent-inspect/reporters` contains shared, dependency-free helpers for local test reporter artifacts. The subpath does not import Vitest, Jest, GitHub SDKs, provider SDKs, or upload clients.
|
|
228
|
+
|
|
229
|
+
Import from `agent-inspect/reporters`:
|
|
230
|
+
|
|
231
|
+
```ts
|
|
232
|
+
import {
|
|
233
|
+
TRACE_ARTIFACT_MANIFEST_SCHEMA_VERSION,
|
|
234
|
+
createReporterArtifactPath,
|
|
235
|
+
createTraceArtifactManifest,
|
|
236
|
+
validateReporterArtifactPath,
|
|
237
|
+
type TraceArtifactManifest,
|
|
238
|
+
} from "agent-inspect/reporters";
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
- **`TRACE_ARTIFACT_MANIFEST_SCHEMA_VERSION`**: currently `"0.1"` for local reporter manifests.
|
|
242
|
+
- **`createTraceArtifactManifest(options)`**: clones, sorts, and deduplicates reporter results/artifacts into deterministic manifest JSON.
|
|
243
|
+
- **`createReporterArtifactPath(options)`**: creates a safe relative artifact path under a caller-provided output directory.
|
|
244
|
+
- **`validateReporterArtifactPath(options)`**: rejects empty, absolute, traversal, Windows-absolute, and symlink-escape style paths before reporters or `ci-summary` trust artifact links.
|
|
245
|
+
|
|
246
|
+
The manifest records framework, generation time, bounded test results, artifact descriptors, redaction profile, and diagnostics. It is an artifact index only; it should not contain raw trace contents, prompts, model outputs, request/response bodies, headers, API keys, secrets, or full tool payloads.
|
|
247
|
+
|
|
225
248
|
## 12. Experimental `@agent-inspect/vitest` APIs
|
|
226
249
|
|
|
227
|
-
`@agent-inspect/vitest` is an optional experimental workspace package for local Vitest failure artifacts. It remains private/unpublished. It does not add a Vitest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
|
|
250
|
+
`@agent-inspect/vitest` is an optional experimental workspace package for local Vitest failure artifacts. It remains private/unpublished pending maintainer first-publication setup. It does not add a Vitest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
|
|
228
251
|
|
|
229
252
|
Import from `@agent-inspect/vitest`:
|
|
230
253
|
|
|
@@ -237,6 +260,7 @@ import { createAgentInspectVitestReporter } from "@agent-inspect/vitest";
|
|
|
237
260
|
- **`githubSummary`**: optional GitHub step-summary file path. The reporter appends bounded structural counts only and does not use the GitHub API.
|
|
238
261
|
- **`retainSuccessful`**: `false`/undefined keeps no passing-test artifacts; `true` keeps up to `maxSuccessfulTraces`; a number keeps up to that many passing-test artifacts.
|
|
239
262
|
- **`maxSuccessfulTraces`**: upper bound for passing-test artifacts, capped by the reporter.
|
|
263
|
+
- **`redactionProfile`**: manifest artifact profile, `local` (default), `share`, or `strict`.
|
|
240
264
|
- **`resolveTrace(test)`**: optional explicit association resolver when task metadata is not convenient.
|
|
241
265
|
- **`onDiagnostic(diagnostic)`**: observes non-fatal reporter/artifact failures.
|
|
242
266
|
- **`getDiagnostics()`** and **`getArtifacts()`** expose reporter state for tests and custom harnesses.
|
|
@@ -252,11 +276,11 @@ ctx.task.meta.agentInspect = {
|
|
|
252
276
|
};
|
|
253
277
|
```
|
|
254
278
|
|
|
255
|
-
Artifacts are safe structural summaries.
|
|
279
|
+
Artifacts are safe structural summaries. The reporter writes a shared `schemaVersion: "0.1"` manifest wrapper with package metadata, generated time, framework, test results, artifact descriptors, relative paths, and redaction profile. It includes bounded test identity, status, trace run id, and trace filename, but it does not read or embed raw trace contents, prompts, generated outputs, request/response bodies, headers, API keys, secrets, or tool payloads. Reporter/artifact failures are diagnostics and do not replace original Vitest failures.
|
|
256
280
|
|
|
257
281
|
## 13. Experimental `@agent-inspect/jest` APIs
|
|
258
282
|
|
|
259
|
-
`@agent-inspect/jest` is an optional experimental workspace package for local Jest failure artifacts. It remains private/unpublished. It does not add a Jest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
|
|
283
|
+
`@agent-inspect/jest` is an optional experimental workspace package for local Jest failure artifacts. It remains private/unpublished pending maintainer first-publication setup. It does not add a Jest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
|
|
260
284
|
|
|
261
285
|
Import from `@agent-inspect/jest`:
|
|
262
286
|
|
|
@@ -270,6 +294,7 @@ import { AgentInspectJestReporter, createAgentInspectJestReporter } from "@agent
|
|
|
270
294
|
- **`githubSummary`**: optional GitHub step-summary file path. The reporter appends bounded structural counts only and does not use the GitHub API.
|
|
271
295
|
- **`retainSuccessful`**: `false`/undefined keeps no passing-test artifacts; `true` keeps up to `maxSuccessfulTraces`; a number keeps up to that many passing-test artifacts.
|
|
272
296
|
- **`maxSuccessfulTraces`**: upper bound for passing-test artifacts, capped by the reporter.
|
|
297
|
+
- **`redactionProfile`**: manifest artifact profile, `local` (default), `share`, or `strict`.
|
|
273
298
|
- **`associations`**: explicit trace associations keyed by `file::fullName`, `basename::fullName`, or `fullName`.
|
|
274
299
|
- **`resolveTrace(test)`**: optional explicit association resolver for normalized Jest assertion results.
|
|
275
300
|
- **`onDiagnostic(diagnostic)`**: observes non-fatal reporter/artifact failures.
|
|
@@ -294,7 +319,7 @@ reporters: [
|
|
|
294
319
|
],
|
|
295
320
|
```
|
|
296
321
|
|
|
297
|
-
Artifacts are safe structural summaries.
|
|
322
|
+
Artifacts are safe structural summaries. The reporter writes a shared `schemaVersion: "0.1"` manifest wrapper with package metadata, generated time, framework, test results, artifact descriptors, relative paths, and redaction profile. It includes bounded test identity, status, trace run id, and trace filename, but it does not read or embed raw trace contents, prompts, generated outputs, request/response bodies, headers, API keys, secrets, or tool payloads. Reporter/artifact failures are diagnostics and do not replace original Jest failures.
|
|
298
323
|
|
|
299
324
|
## 14. Experimental `@agent-inspect/openai-agents` APIs
|
|
300
325
|
|
package/docs/CLI.md
CHANGED
|
@@ -33,6 +33,7 @@ Core commands:
|
|
|
33
33
|
- `scan` — best-effort local safety scan for trace capture risks
|
|
34
34
|
- `verify-safe` — best-effort local trace safety verification
|
|
35
35
|
- `artifacts` — create safe local CI trace artifact bundles and optional step summaries
|
|
36
|
+
- `ci-summary` — summarize local reporter artifact manifests for CI
|
|
36
37
|
- `diff` — compare two manual traces (local, read-only)
|
|
37
38
|
- `timeline` — chronological view of one run (local JSONL)
|
|
38
39
|
- `stats` — local aggregate stats over a trace directory
|
|
@@ -489,7 +490,33 @@ npx agent-inspect artifacts candidate.jsonl --baseline baseline.jsonl --output-d
|
|
|
489
490
|
|
|
490
491
|
Recipe and sample workflow: [examples/recipes/deterministic-ci-checks](../examples/recipes/deterministic-ci-checks/README.md)
|
|
491
492
|
|
|
492
|
-
### 6.14 `
|
|
493
|
+
### 6.14 `ci-summary`
|
|
494
|
+
|
|
495
|
+
Summarize local Vitest/Jest reporter artifact manifests into deterministic Markdown or JSON. This command reads shared `schemaVersion: "0.1"` manifest JSON files only, including the reporter package wrapper emitted by the workspace reporters. It does not read trace contents, rerun tests, upload artifacts, call GitHub APIs, or mutate repository state. `--output` and `--github-summary` write local files.
|
|
496
|
+
|
|
497
|
+
```bash
|
|
498
|
+
agent-inspect ci-summary <manifest...> [options]
|
|
499
|
+
```
|
|
500
|
+
|
|
501
|
+
Options:
|
|
502
|
+
|
|
503
|
+
- `-o, --output <path>`: write the Markdown summary to a local file
|
|
504
|
+
- `--github-summary <path>`: append the Markdown summary to a local file, such as `$GITHUB_STEP_SUMMARY`
|
|
505
|
+
- `--json`: print deterministic JSON summary
|
|
506
|
+
|
|
507
|
+
Example:
|
|
508
|
+
|
|
509
|
+
```bash
|
|
510
|
+
npx agent-inspect ci-summary .agent-inspect/jest-artifacts/tests/**/report.json \
|
|
511
|
+
--output ./artifacts/reporter-summary.md \
|
|
512
|
+
--github-summary "$GITHUB_STEP_SUMMARY"
|
|
513
|
+
```
|
|
514
|
+
|
|
515
|
+
Reporter artifact paths in the summary are kept relative and validated conservatively. The summary includes bounded package/framework metadata, test identity, status counts, trace filenames, artifact paths, redaction profiles, and diagnostic counts only.
|
|
516
|
+
|
|
517
|
+
Recipe and sample workflow: [examples/recipes/github-actions-artifact](../examples/recipes/github-actions-artifact/README.md)
|
|
518
|
+
|
|
519
|
+
### 6.15 `diff`
|
|
493
520
|
|
|
494
521
|
Compare two manual trace runs. Diff is **local** and **read-only** (does not rerun agents).
|
|
495
522
|
|
|
@@ -553,7 +580,7 @@ Differences:
|
|
|
553
580
|
|
|
554
581
|
More examples, including timing-only and structure-only diffs, are in `docs/DIFF.md`.
|
|
555
582
|
|
|
556
|
-
### 6.
|
|
583
|
+
### 6.16 `timeline`
|
|
557
584
|
|
|
558
585
|
Chronological step list for one manual trace. Read-only; does not mutate JSONL files.
|
|
559
586
|
|
|
@@ -569,7 +596,7 @@ Options:
|
|
|
569
596
|
|
|
570
597
|

|
|
571
598
|
|
|
572
|
-
### 6.
|
|
599
|
+
### 6.17 `stats`
|
|
573
600
|
|
|
574
601
|
Local aggregate statistics over trace files in a directory. Read-only.
|
|
575
602
|
|
|
@@ -589,7 +616,7 @@ Options:
|
|
|
589
616
|
|
|
590
617
|
Use `--correlation-id` or `--group-id` to filter runs by `run_started` metadata (see [API.md](./API.md)).
|
|
591
618
|
|
|
592
|
-
### 6.
|
|
619
|
+
### 6.18 `search`
|
|
593
620
|
|
|
594
621
|
Deterministic search over local traces (substring / exact filters). No semantic search.
|
|
595
622
|
|
|
@@ -619,7 +646,7 @@ npx agent-inspect search --duration ">100ms" --json
|
|
|
619
646
|
|
|
620
647
|

|
|
621
648
|
|
|
622
|
-
### 6.
|
|
649
|
+
### 6.19 `what`
|
|
623
650
|
|
|
624
651
|
Concise human-readable summary of one local trace run. Read-only; accepts v0.1 manual JSONL and v0.2 persisted-event JSONL through the shared dual-format normalization path. Vocabulary: [TRACE-VOCABULARY-V1.5.md](./proposals/TRACE-VOCABULARY-V1.5.md).
|
|
625
652
|
|
|
@@ -648,7 +675,7 @@ Outcome: Completed successfully.
|
|
|
648
675
|
Slowest: plan (100ms, logic)
|
|
649
676
|
```
|
|
650
677
|
|
|
651
|
-
### 6.
|
|
678
|
+
### 6.20 `report`
|
|
652
679
|
|
|
653
680
|
Generate a local inspection report combining **what happened**, **timeline**, and **execution tree** sections. The command reads local v0.1 manual JSONL and v0.2 persisted-event JSONL through the shared dual-format normalization path without mutating them. Distinct from `export` (which targets shareable tree snapshots and standards formats).
|
|
654
681
|
|
|
@@ -673,7 +700,7 @@ Example:
|
|
|
673
700
|
npx agent-inspect report minimal-success --dir fixtures/traces --format html -o report.html
|
|
674
701
|
```
|
|
675
702
|
|
|
676
|
-
### 6.
|
|
703
|
+
### 6.21 `explain`
|
|
677
704
|
|
|
678
705
|
Explain a local trace using deterministic facts and local inference labels. This command reads through the same local reader pipeline as `open` / `check`; it does not call a model provider, upload traces, replay agents, or mutate input files.
|
|
679
706
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-inspect",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.3.0",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"description": "Local-first execution-tree debugger for TypeScript AI agents",
|
|
@@ -105,6 +105,16 @@
|
|
|
105
105
|
"types": "./packages/core/dist/checks.d.cts",
|
|
106
106
|
"default": "./packages/core/dist/checks.cjs"
|
|
107
107
|
}
|
|
108
|
+
},
|
|
109
|
+
"./reporters": {
|
|
110
|
+
"import": {
|
|
111
|
+
"types": "./packages/core/dist/reporters.d.ts",
|
|
112
|
+
"default": "./packages/core/dist/reporters.mjs"
|
|
113
|
+
},
|
|
114
|
+
"require": {
|
|
115
|
+
"types": "./packages/core/dist/reporters.d.cts",
|
|
116
|
+
"default": "./packages/core/dist/reporters.cjs"
|
|
117
|
+
}
|
|
108
118
|
}
|
|
109
119
|
},
|
|
110
120
|
"bin": {
|