agent-inspect 1.7.0 → 1.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +13 -1
- package/README.md +11 -6
- package/docs/ADAPTER-CONFORMANCE.md +7 -3
- package/docs/ADAPTERS.md +120 -5
- package/docs/API.md +123 -21
- package/docs/CLI.md +154 -6
- package/docs/KNOWN-ISSUES.md +7 -1
- package/docs/LIMITATIONS.md +7 -1
- package/docs/SCHEMA.md +1 -0
- package/package.json +12 -2
- package/packages/cli/dist/index.cjs +2057 -33
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +2057 -33
- package/packages/cli/dist/index.mjs.map +1 -1
- package/packages/core/dist/advanced.d.cts +4 -4
- package/packages/core/dist/advanced.d.ts +4 -4
- package/packages/core/dist/checks.cjs +1535 -0
- package/packages/core/dist/checks.cjs.map +1 -0
- package/packages/core/dist/checks.d.cts +585 -0
- package/packages/core/dist/checks.d.ts +585 -0
- package/packages/core/dist/checks.mjs +1512 -0
- package/packages/core/dist/checks.mjs.map +1 -0
- package/packages/core/dist/diff.d.cts +3 -3
- package/packages/core/dist/diff.d.ts +3 -3
- package/packages/core/dist/exporters.d.cts +3 -3
- package/packages/core/dist/exporters.d.ts +3 -3
- package/packages/core/dist/index.d.cts +6 -6
- package/packages/core/dist/index.d.ts +6 -6
- package/packages/core/dist/{inspect-event-Des4JDHo.d.cts → inspect-event-CevRYp58.d.cts} +1 -1
- package/packages/core/dist/{inspect-event-Des4JDHo.d.ts → inspect-event-CevRYp58.d.ts} +1 -1
- package/packages/core/dist/{log-config-C1GcJPIM.d.ts → log-config-BPHS4Sds.d.ts} +1 -1
- package/packages/core/dist/{log-config-BnH8Ykcb.d.cts → log-config-DanPV3P9.d.cts} +1 -1
- package/packages/core/dist/logs.d.cts +3 -3
- package/packages/core/dist/logs.d.ts +3 -3
- package/packages/core/dist/{persisted-inspect-event-DiFto0K2.d.ts → persisted-inspect-event-Cw7TeYGr.d.ts} +1 -1
- package/packages/core/dist/{persisted-inspect-event-0kaRADsp.d.cts → persisted-inspect-event-DHPfzUd8.d.cts} +1 -1
- package/packages/core/dist/persisted.d.cts +5 -5
- package/packages/core/dist/persisted.d.ts +5 -5
- package/packages/core/dist/readers.d.cts +2 -2
- package/packages/core/dist/readers.d.ts +2 -2
- package/packages/core/dist/{types-tSix7tfv.d.ts → types-Ap9uMdx_.d.ts} +1 -1
- package/packages/core/dist/{types-DB8jB6Jg.d.cts → types-B2-BU5CS.d.cts} +1 -1
- package/packages/core/dist/writers.d.cts +2 -2
- package/packages/core/dist/writers.d.ts +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,10 +1,22 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.8.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 0bee42c: Release v1.8.0 with OpenAI Agents trace processor support, optional Vitest/Jest reporter packages kept private, deterministic CI release checks, and the validated local-first reporting improvements from the v1.8 release train.
|
|
8
|
+
|
|
3
9
|
## 1.7.0
|
|
4
10
|
|
|
11
|
+
Released **2026-06-26**.
|
|
12
|
+
|
|
5
13
|
### Minor Changes
|
|
6
14
|
|
|
7
|
-
- 94a7220: Release v1.7.0 framework-native adoption with the experimental AI SDK telemetry adapter, adapter conformance
|
|
15
|
+
- 94a7220: Release v1.7.0 framework-native adoption with the experimental AI SDK telemetry adapter, declarative adapter conformance matrix, and local-first adapter documentation.
|
|
16
|
+
|
|
17
|
+
### Notes
|
|
18
|
+
|
|
19
|
+
- The v1.8 train carries the remaining adapter correctness work: AI SDK logical lifecycle identity, parallel integration isolation, explicit capture/redaction behavior, executable conformance fixtures, OpenAI Agents runtime mapping, and LangGraph no-network fixtures. v1.7.0 should not be read as claiming those deferred behaviors.
|
|
8
20
|
|
|
9
21
|
## 1.6.0
|
|
10
22
|
|
package/README.md
CHANGED
|
@@ -22,7 +22,7 @@ agent-inspect gives those runs **structure**: an **execution tree** you can read
|
|
|
22
22
|
|
|
23
23
|
## Install
|
|
24
24
|
|
|
25
|
-
Current npm release: **1.
|
|
25
|
+
Current npm release: **1.7.0** (`agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, `@agent-inspect/tui` — all aligned).
|
|
26
26
|
|
|
27
27
|
```bash
|
|
28
28
|
npm install agent-inspect
|
|
@@ -111,7 +111,7 @@ await maybeInspectRun("eval-case-42", async () => runAgent());
|
|
|
111
111
|
AGENT_INSPECT=1 node eval-runner.mjs
|
|
112
112
|
```
|
|
113
113
|
|
|
114
|
-
## What you can do today (v1.
|
|
114
|
+
## What you can do today (v1.7.0)
|
|
115
115
|
|
|
116
116
|
- **Trace manually** with `inspectRun`, `step`, `step.llm`, `step.tool`, and `observe` — local JSONL under `.agent-inspect/` by default.
|
|
117
117
|
- **Toggle tracing** with `maybeInspectRun` and `AGENT_INSPECT=1` in eval harnesses or CI.
|
|
@@ -121,6 +121,7 @@ AGENT_INSPECT=1 node eval-runner.mjs
|
|
|
121
121
|
- **Export share-safe copies** — `export --redaction-profile share` (or `strict`) writes local Markdown/HTML/OpenInference/OTLP JSON only.
|
|
122
122
|
- **Parse structured logs** you already emit (JSON first-class; log4js best-effort).
|
|
123
123
|
- **Optional LangChain adapter** — metadata-only by default; optional `persist: true` and `stream: true` streaming metadata (no full token capture by default).
|
|
124
|
+
- **Optional AI SDK adapter** — experimental `@agent-inspect/ai-sdk` telemetry integration for AI SDK v6; metadata-only by default with `recordInputs: false` and `recordOutputs: false`.
|
|
124
125
|
- **Optional TUI** — `view --tui` when `@agent-inspect/tui` is installed.
|
|
125
126
|
- **Persisted-event foundation (v1.2.0+)** — in-memory `PersistedInspectEvent` converters; manual writing stays `schemaVersion: "0.1"`.
|
|
126
127
|
- **Experimental v1.6.0 APIs** — `agent-inspect/writers`, `agent-inspect/readers`, `createInspector()`, and `agent-inspect open` for local AgentInspect/OpenInference/OTLP ingestion.
|
|
@@ -180,6 +181,8 @@ More detail: [docs/LOGS.md](docs/LOGS.md) · [docs/LOG-TO-TREE-QUICKSTART.md](do
|
|
|
180
181
|
| `search` | Deterministic search over local traces |
|
|
181
182
|
| `what` | Concise summary of one run |
|
|
182
183
|
| `report` | Markdown/HTML inspection report (what + timeline + tree) |
|
|
184
|
+
| `check` / `scan` / `verify-safe` | Deterministic local trace checks and best-effort safety verification |
|
|
185
|
+
| `artifacts` | Safe local CI artifact bundles and optional step-summary file output |
|
|
183
186
|
|
|
184
187
|

|
|
185
188
|
|
|
@@ -198,7 +201,7 @@ Full flags and behavior: [docs/CLI.md](docs/CLI.md).
|
|
|
198
201
|
|
|
199
202
|
## Stable foundation (AgentInspect 1.x)
|
|
200
203
|
|
|
201
|
-
**agent-inspect 1.x** (current: **1.
|
|
204
|
+
**agent-inspect 1.x** (current: **1.7.0**) is the **local-first trace workbench** for TypeScript AI agents:
|
|
202
205
|
|
|
203
206
|
- Instrument runs with `inspectRun` and `step`
|
|
204
207
|
- Write **local JSONL traces** (`schemaVersion: "0.1"` — compatibility retained)
|
|
@@ -208,13 +211,13 @@ Full flags and behavior: [docs/CLI.md](docs/CLI.md).
|
|
|
208
211
|
|
|
209
212
|
Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInspectRun` with `AGENT_INSPECT=1` to toggle tracing in eval or CI — see [docs/API.md](docs/API.md).
|
|
210
213
|
|
|
211
|
-
**Shipped in 1.
|
|
214
|
+
**Shipped in 1.7.0:** experimental `@agent-inspect/ai-sdk` telemetry integration for AI SDK v6 with a local no-network [ai-sdk-local-telemetry recipe](examples/recipes/ai-sdk-local-telemetry/), adapter conformance fixtures, OpenAI Agents/LangGraph support decisions, and local-first adapter docs. Examples keep `recordInputs: false`, `recordOutputs: false`, metadata-only capture, and no upload behavior. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, and `@agent-inspect/tui` at **1.7.0**.
|
|
212
215
|
|
|
213
|
-
**
|
|
216
|
+
**Shipped in 1.6.0:** experimental writer subpath (`agent-inspect/writers`), isolated `createInspector()` API via `agent-inspect/advanced`, local trace readers via `agent-inspect/readers`, OpenInference/OTLP JSON readers, universal `agent-inspect open`, and deterministic [runtime-and-ingestion recipe](examples/recipes/runtime-and-ingestion/). These remain local-only and do not add upload behavior. Linked release aligns all three then-published npm packages at **1.6.0**.
|
|
214
217
|
|
|
215
218
|
**Shipped in 1.5.0:** non-breaking subpath exports; `what` and `report` CLI; dual-format read path (v0.1 + v0.2 JSONL); [what-report-inspect recipe](examples/recipes/what-report-inspect/). Linked release aligns all three npm packages at **1.5.0**.
|
|
216
219
|
|
|
217
|
-
**Roadmap beyond current release work:** future work continues from the local runtime
|
|
220
|
+
**Roadmap beyond current release work:** future work continues from the local runtime, universal ingestion, and optional adapter foundations. See [ROADMAP.md](ROADMAP.md).
|
|
218
221
|
|
|
219
222
|
**Shipped in 1.4.0:** CI artifact recipe ([docs/CI-ARTIFACTS.md](docs/CI-ARTIFACTS.md)); `timeline`, `stats`, and `search` CLI; core helpers `buildRunTimeline`, `buildTraceStats`, `searchTraces`. Linked release aligns all three npm packages at **1.4.0**.
|
|
220
223
|
|
|
@@ -291,6 +294,8 @@ The TUI is available as a separate optional package; its programmatic API is exp
|
|
|
291
294
|
| [examples/recipes/retry-fallback](examples/recipes/retry-fallback) | Fallback pattern |
|
|
292
295
|
| [examples/recipes/parallel-tools](examples/recipes/parallel-tools) | Parallel tools |
|
|
293
296
|
| [examples/recipes/github-actions-artifact](examples/recipes/github-actions-artifact) | CI trace artifacts |
|
|
297
|
+
| [examples/recipes/deterministic-ci-checks](examples/recipes/deterministic-ci-checks) | v1.8 checks, baseline, and safe CI artifacts |
|
|
298
|
+
| [examples/recipes/test-reporter-artifacts](examples/recipes/test-reporter-artifacts) | v1.8 Vitest/Jest reporter artifact patterns |
|
|
294
299
|
| [examples/recipes/what-report-inspect](examples/recipes/what-report-inspect/) | `what` + `report` inspection |
|
|
295
300
|
| [examples/recipes/runtime-and-ingestion](examples/recipes/runtime-and-ingestion/) | v1.6 runtime writers + universal ingestion |
|
|
296
301
|
|
|
@@ -12,14 +12,18 @@ The machine-readable matrix lives at [docs/implementation/adapter-conformance-ma
|
|
|
12
12
|
- streaming metadata
|
|
13
13
|
- metadata bounds and privacy controls
|
|
14
14
|
|
|
15
|
+
The v1.7 matrix is declarative coverage guidance. v1.8 makes conformance executable and requires canonical-reader round trips before adapter output is used by checks.
|
|
16
|
+
|
|
17
|
+
Executable shared assertions live in `packages/core/test/adapter-executable-conformance.test.ts` and `packages/core/test/adapter-conformance-utils.ts`. Adapter-specific suites may add deeper fixture coverage, but the shared suite owns the cross-adapter defaults: local-only execution, no raw payload persistence, lifecycle identity, parentage, streaming summaries, token usage where exposed, and reader round trips.
|
|
18
|
+
|
|
15
19
|
## Current matrix
|
|
16
20
|
|
|
17
21
|
| Adapter | Package | Status | Default install mode | Boundary |
|
|
18
22
|
| --- | --- | --- | --- | --- |
|
|
19
|
-
| AI SDK | `@agent-inspect/ai-sdk` | implemented experimental | AI SDK telemetry integration | optional package peer dependency |
|
|
23
|
+
| AI SDK | `@agent-inspect/ai-sdk` | implemented experimental; v1.8 correctness hardening pending | AI SDK telemetry integration | optional package peer dependency |
|
|
20
24
|
| LangChain | `@agent-inspect/langchain` | implemented experimental | explicit callback | optional package peer dependency |
|
|
21
|
-
| OpenAI Agents JS | `@agent-inspect/openai-agents` |
|
|
22
|
-
| LangGraph | `@agent-inspect/langchain` |
|
|
25
|
+
| OpenAI Agents JS | `@agent-inspect/openai-agents` | implemented experimental; private until first publication gate | `setTraceProcessors()` replacement | optional package peer dependency |
|
|
26
|
+
| LangGraph | `@agent-inspect/langchain` | fixture-backed through LangChain callback | explicit LangChain callback | existing LangChain adapter first |
|
|
23
27
|
|
|
24
28
|
## Required defaults
|
|
25
29
|
|
package/docs/ADAPTERS.md
CHANGED
|
@@ -4,7 +4,9 @@ AgentInspect is **framework-agnostic** at its core. Optional adapter packages in
|
|
|
4
4
|
|
|
5
5
|
## Vercel AI SDK (`@agent-inspect/ai-sdk`)
|
|
6
6
|
|
|
7
|
-
**Status:** experimental v1.7
|
|
7
|
+
**Status:** experimental v1.7 adapter — optional package published in the v1.7.0 linked release.
|
|
8
|
+
|
|
9
|
+
The v1.8 train has hardened lifecycle identity and parallel integration isolation. The adapter remains metadata-only: `capture: "preview"` and preview-only redaction options emit diagnostics and fall back to metadata-only capture until bounded free-text previews are implemented.
|
|
8
10
|
|
|
9
11
|
### Install
|
|
10
12
|
|
|
@@ -41,6 +43,7 @@ const result = await generateText({
|
|
|
41
43
|
- **Metadata-only by default** — records model, finish reason, token usage, timing, and safe counts/summaries.
|
|
42
44
|
- **Required safe telemetry settings** — set `recordInputs: false` and `recordOutputs: false` on every AI SDK call using this adapter.
|
|
43
45
|
- **No raw payload capture by default** — prompts, messages, generated text, stream chunks, tool inputs/outputs, headers, request bodies, and response bodies are not persisted.
|
|
46
|
+
- **Preview capture is not enabled yet** — `capture: "preview"`, `redactionProfile`, and `maxPreviewChars` are diagnosed through `getDiagnostics()` and do not persist raw previews.
|
|
44
47
|
|
|
45
48
|
### Local no-network recipe
|
|
46
49
|
|
|
@@ -112,6 +115,16 @@ npx agent-inspect export <run-id> --format markdown --redaction-profile share
|
|
|
112
115
|
|
|
113
116
|
Written events use `schemaVersion: "0.1"` manual trace names.
|
|
114
117
|
|
|
118
|
+
### LangGraph through LangChain callbacks
|
|
119
|
+
|
|
120
|
+
LangGraph-shaped callback metadata is covered through the existing `@agent-inspect/langchain` callback boundary. No separate `@agent-inspect/langgraph` package is shipped.
|
|
121
|
+
|
|
122
|
+
- **Explicit callback only** — pass `new AgentInspectCallback(...)` through LangChain/LangGraph callback configuration.
|
|
123
|
+
- **Bounded graph metadata** — known graph, node, subgraph, task, branch, checkpoint, retry, handoff, thread, and session identifiers are copied into `attributes.langGraph` when present.
|
|
124
|
+
- **No full graph state** — checkpoint/task/branch containers are summarized by type/count; raw graph state, prompts, tool payloads, outputs, and stream tokens are not stored in `metadata-only` mode.
|
|
125
|
+
- **Conservative parent mapping** — in-memory events preserve framework `parentRunId`; persisted JSONL maps parents only when the parent callback was seen, and marks unresolved parent mappings in step metadata.
|
|
126
|
+
- **No hosted tracing requirement** — fixtures use structural no-network callback payloads and do not require LangSmith, provider calls, or LangGraph platform services.
|
|
127
|
+
|
|
115
128
|
### Capture modes
|
|
116
129
|
|
|
117
130
|
| `capture` | Behavior |
|
|
@@ -164,7 +177,7 @@ Full API: [API.md](./API.md) §9.
|
|
|
164
177
|
|
|
165
178
|
### LangGraph boundary
|
|
166
179
|
|
|
167
|
-
LangGraph support is expected to ride through this same `@agent-inspect/langchain` callback boundary first. A dedicated LangGraph package
|
|
180
|
+
LangGraph support is expected to ride through this same `@agent-inspect/langchain` callback boundary first. v1.8 adds executable no-network fixtures before claiming broader LangGraph support. A dedicated LangGraph package remains deferred until fixtures prove that LangGraph exposes important lifecycle data unavailable through LangChain callbacks.
|
|
168
181
|
|
|
169
182
|
Future LangGraph examples must keep the same safety defaults: explicit callback installation, metadata-only capture, no raw prompt/output/tool payload capture by default, no hosted sink, and local persistence only when `persist: true` is set.
|
|
170
183
|
|
|
@@ -187,11 +200,103 @@ Requires an interactive terminal. See [API.md](./API.md) §10.
|
|
|
187
200
|
|
|
188
201
|
---
|
|
189
202
|
|
|
203
|
+
## Vitest (`@agent-inspect/vitest`)
|
|
204
|
+
|
|
205
|
+
**Status:** experimental v1.8 reporter — optional workspace package, private/unpublished until release readiness.
|
|
206
|
+
|
|
207
|
+
```bash
|
|
208
|
+
npm install agent-inspect @agent-inspect/vitest vitest
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
After publication, the reporter creates safe, structural artifacts for failed tests that explicitly attach AgentInspect trace metadata. It never guesses trace files by timestamp and does not read trace contents into artifacts.
|
|
212
|
+
|
|
213
|
+
```ts
|
|
214
|
+
import { createAgentInspectVitestReporter } from "@agent-inspect/vitest";
|
|
215
|
+
|
|
216
|
+
export default {
|
|
217
|
+
test: {
|
|
218
|
+
reporters: [
|
|
219
|
+
"default",
|
|
220
|
+
createAgentInspectVitestReporter({
|
|
221
|
+
artifactDir: ".agent-inspect/vitest-artifacts",
|
|
222
|
+
retainSuccessful: 5,
|
|
223
|
+
}),
|
|
224
|
+
],
|
|
225
|
+
},
|
|
226
|
+
};
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
Attach explicit metadata from the test harness, or provide `resolveTrace(test)`:
|
|
230
|
+
|
|
231
|
+
```ts
|
|
232
|
+
test("agent workflow", async (ctx) => {
|
|
233
|
+
ctx.task.meta.agentInspect = {
|
|
234
|
+
runId: "support-agent",
|
|
235
|
+
tracePath: ".agent-inspect/support-agent.jsonl",
|
|
236
|
+
artifactLabel: "support-agent",
|
|
237
|
+
};
|
|
238
|
+
});
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
- **Explicit association only** — no timestamp matching or directory guessing.
|
|
242
|
+
- **Failure artifacts by default** — passing-test artifacts are kept only when `retainSuccessful` is configured.
|
|
243
|
+
- **Bounded success retention** — successful artifacts are capped by `maxSuccessfulTraces`.
|
|
244
|
+
- **Failure-preserving** — reporter/artifact errors are surfaced through diagnostics and do not replace original Vitest failures.
|
|
245
|
+
- **Local-only** — no network I/O, no hosted upload, no GitHub API, and no root/core Vitest dependency.
|
|
246
|
+
- **Safe rendering** — artifacts include structural test/run/file references only, not raw trace contents, prompts, outputs, request/response bodies, headers, API keys, secrets, or tool payloads.
|
|
247
|
+
|
|
248
|
+
Full API: [API.md](./API.md) §12.
|
|
249
|
+
|
|
250
|
+
---
|
|
251
|
+
|
|
252
|
+
## Jest (`@agent-inspect/jest`)
|
|
253
|
+
|
|
254
|
+
**Status:** experimental v1.8 reporter — optional workspace package, private/unpublished until release readiness.
|
|
255
|
+
|
|
256
|
+
```bash
|
|
257
|
+
npm install agent-inspect @agent-inspect/jest jest
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
After publication, the reporter creates safe, structural artifacts for failed Jest assertions that explicitly attach AgentInspect trace metadata through a map or resolver. It never guesses trace files by timestamp and does not read trace contents into artifacts.
|
|
261
|
+
|
|
262
|
+
```js
|
|
263
|
+
module.exports = {
|
|
264
|
+
reporters: [
|
|
265
|
+
"default",
|
|
266
|
+
[
|
|
267
|
+
"@agent-inspect/jest",
|
|
268
|
+
{
|
|
269
|
+
artifactDir: ".agent-inspect/jest-artifacts",
|
|
270
|
+
retainSuccessful: 5,
|
|
271
|
+
associations: {
|
|
272
|
+
"agent.test.cjs::agent suite agent workflow": {
|
|
273
|
+
runId: "support-agent",
|
|
274
|
+
tracePath: ".agent-inspect/support-agent.jsonl",
|
|
275
|
+
},
|
|
276
|
+
},
|
|
277
|
+
},
|
|
278
|
+
],
|
|
279
|
+
],
|
|
280
|
+
};
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
- **Explicit association only** — use `associations` or `resolveTrace(test)`; no timestamp matching or directory guessing.
|
|
284
|
+
- **Jest lifecycle** — processes assertion results from `onTestResult` and aggregated file results from `onRunComplete`.
|
|
285
|
+
- **Failure artifacts by default** — passing-test artifacts are kept only when `retainSuccessful` is configured.
|
|
286
|
+
- **Bounded success retention** — successful artifacts are capped by `maxSuccessfulTraces`.
|
|
287
|
+
- **Failure-preserving** — reporter/artifact errors are surfaced through diagnostics and do not replace original Jest failures.
|
|
288
|
+
- **Local-only** — no network I/O, no hosted upload, no GitHub API, and no root/core Jest dependency.
|
|
289
|
+
- **Safe rendering** — artifacts include structural test/run/file references only, not raw trace contents, prompts, outputs, request/response bodies, headers, API keys, secrets, or tool payloads.
|
|
290
|
+
|
|
291
|
+
Full API: [API.md](./API.md) §13.
|
|
292
|
+
|
|
293
|
+
---
|
|
294
|
+
|
|
190
295
|
## OpenAI Agents JS (`@agent-inspect/openai-agents`)
|
|
191
296
|
|
|
192
|
-
**Status:**
|
|
297
|
+
**Status:** experimental v1.8 adapter — optional workspace package remains private/unpublished until the manual first-publication gate.
|
|
193
298
|
|
|
194
|
-
The safe integration boundary is documented in [OPENAI-AGENTS-JS-TRACING.md](./proposals/OPENAI-AGENTS-JS-TRACING.md).
|
|
299
|
+
The safe integration boundary is documented in [OPENAI-AGENTS-JS-TRACING.md](./proposals/OPENAI-AGENTS-JS-TRACING.md). Install the AgentInspect processor by replacing processors:
|
|
195
300
|
|
|
196
301
|
```ts
|
|
197
302
|
import { setTraceProcessors } from "@openai/agents";
|
|
@@ -205,7 +310,17 @@ setTraceProcessors([
|
|
|
205
310
|
]);
|
|
206
311
|
```
|
|
207
312
|
|
|
208
|
-
Do not use `addTraceProcessor()` as the default AgentInspect path; that preserves the OpenAI default exporter in server runtimes. The
|
|
313
|
+
Do not use `addTraceProcessor()` as the default AgentInspect path; that preserves the OpenAI default exporter in server runtimes. The processor does not auto-install itself, does not upload, and does not add OpenAI Agents dependencies to root/core.
|
|
314
|
+
|
|
315
|
+
- **No auto-install** — importing or constructing `agentInspectProcessor()` never calls `setTraceProcessors()` or `addTraceProcessor()`.
|
|
316
|
+
- **No upload behavior** — the processor writes only to an explicit local writer or `traceDir`.
|
|
317
|
+
- **Metadata-only by default** — records trace/span IDs, parentage, names, timing, status, errors, safe model/tool names, token counts, and bounded summaries.
|
|
318
|
+
- **No raw payload capture by default** — prompts, messages, generated text, function inputs/outputs, arbitrary custom data, trace exporter credentials, headers, request bodies, response bodies, and hosted tool payloads are not persisted.
|
|
319
|
+
- **Preview capture is not enabled yet** — `capture: "preview"`, `redactionProfile`, and `maxPreviewChars` are diagnosed through `getDiagnostics()` and do not persist raw previews.
|
|
320
|
+
|
|
321
|
+
Full API: [API.md](./API.md) §14.
|
|
322
|
+
|
|
323
|
+
Runnable local recipe: [openai-agents-local-tracing](../examples/recipes/openai-agents-local-tracing).
|
|
209
324
|
|
|
210
325
|
---
|
|
211
326
|
|
package/docs/API.md
CHANGED
|
@@ -152,7 +152,7 @@ Rationale: v1.x includes one official adapter and **zero production sinks**, so
|
|
|
152
152
|
|
|
153
153
|
## 11. Experimental `@agent-inspect/ai-sdk` APIs
|
|
154
154
|
|
|
155
|
-
`@agent-inspect/ai-sdk` is an optional v1.7 adapter package for Vercel AI SDK v6 telemetry integrations. It is experimental
|
|
155
|
+
`@agent-inspect/ai-sdk` is an optional v1.7 adapter package for Vercel AI SDK v6 telemetry integrations. It is experimental and published as part of the v1.7.0 linked release.
|
|
156
156
|
|
|
157
157
|
Import from `@agent-inspect/ai-sdk`:
|
|
158
158
|
|
|
@@ -164,9 +164,10 @@ import { agentInspect } from "@agent-inspect/ai-sdk";
|
|
|
164
164
|
- **`writer`**: optional explicit local `TraceWriter` for tests, recipes, and controlled runtime integration.
|
|
165
165
|
- **`traceDir`**: optional local directory that creates a file writer inside the adapter package.
|
|
166
166
|
- **`runName`**: optional local run name.
|
|
167
|
-
- **`capture`**: `"metadata-only"` (default) or `"preview"`
|
|
168
|
-
- **`redactionProfile`** and **`maxPreviewChars`**:
|
|
169
|
-
- **`getDiagnostics()`**: exposes isolated adapter write failures without throwing into AI SDK callbacks.
|
|
167
|
+
- **`capture`**: `"metadata-only"` (default) or `"preview"`; `preview` currently emits a diagnostic and falls back to metadata-only.
|
|
168
|
+
- **`redactionProfile`** and **`maxPreviewChars`**: preview-only knobs; when preview capture is unsupported or not selected, they emit diagnostics instead of silently doing nothing.
|
|
169
|
+
- **`getDiagnostics()`**: exposes isolated adapter write, lifecycle/configuration, flush, and close failures without throwing into AI SDK callbacks.
|
|
170
|
+
- **`getWriterStats()`**, **`flush()`**, and **`close()`**: explicit writer lifecycle helpers. Failures are captured in diagnostics.
|
|
170
171
|
|
|
171
172
|
Every AI SDK call using the adapter must keep telemetry local and metadata-only:
|
|
172
173
|
|
|
@@ -179,15 +180,89 @@ experimental_telemetry: {
|
|
|
179
180
|
}
|
|
180
181
|
```
|
|
181
182
|
|
|
182
|
-
The adapter records local v0.2 persisted events for run, LLM step, and tool lifecycle metadata.
|
|
183
|
+
The adapter records local v0.2 persisted events for run, LLM step, and tool lifecycle metadata. It does not persist raw prompts, messages, generated text, stream chunks, tool inputs, tool outputs, headers, request bodies, response bodies, or user `experimental_context`. Unsupported preview capture options are explicit diagnostics and keep this metadata-only behavior.
|
|
183
184
|
|
|
184
185
|
No network writer, OpenTelemetry exporter, provider wrapper, or global monkey-patch is part of this package.
|
|
185
186
|
|
|
186
187
|
Recipe: [examples/recipes/ai-sdk-local-telemetry](../examples/recipes/ai-sdk-local-telemetry/).
|
|
187
188
|
|
|
188
|
-
## 12. Experimental `@agent-inspect/
|
|
189
|
+
## 12. Experimental `@agent-inspect/vitest` APIs
|
|
189
190
|
|
|
190
|
-
`@agent-inspect/
|
|
191
|
+
`@agent-inspect/vitest` is an optional experimental package for local Vitest failure artifacts. In the v1.8 train it remains private/unpublished until release readiness. It does not add a Vitest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
|
|
192
|
+
|
|
193
|
+
Import from `@agent-inspect/vitest`:
|
|
194
|
+
|
|
195
|
+
```ts
|
|
196
|
+
import { createAgentInspectVitestReporter } from "@agent-inspect/vitest";
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
- **`createAgentInspectVitestReporter(options?)`**: returns a structural Vitest reporter facade with `onTestCaseResult`, `onTaskUpdate`, and `onFinished` hooks.
|
|
200
|
+
- **`artifactDir`**: local output directory for safe artifacts; defaults to `.agent-inspect/vitest-artifacts`.
|
|
201
|
+
- **`githubSummary`**: optional GitHub step-summary file path. The reporter appends bounded structural counts only and does not use the GitHub API.
|
|
202
|
+
- **`retainSuccessful`**: `false`/undefined keeps no passing-test artifacts; `true` keeps up to `maxSuccessfulTraces`; a number keeps up to that many passing-test artifacts.
|
|
203
|
+
- **`maxSuccessfulTraces`**: upper bound for passing-test artifacts, capped by the reporter.
|
|
204
|
+
- **`resolveTrace(test)`**: optional explicit association resolver when task metadata is not convenient.
|
|
205
|
+
- **`onDiagnostic(diagnostic)`**: observes non-fatal reporter/artifact failures.
|
|
206
|
+
- **`getDiagnostics()`** and **`getArtifacts()`** expose reporter state for tests and custom harnesses.
|
|
207
|
+
- **`agentInspectVitestReporter`**: alias for `createAgentInspectVitestReporter`.
|
|
208
|
+
|
|
209
|
+
Trace association must be explicit. The default resolver reads `meta.agentInspect`, `meta["agent-inspect"]`, `meta.trace`, `context.meta.agentInspect`, or result metadata when present:
|
|
210
|
+
|
|
211
|
+
```ts
|
|
212
|
+
ctx.task.meta.agentInspect = {
|
|
213
|
+
runId: "support-agent",
|
|
214
|
+
tracePath: ".agent-inspect/support-agent.jsonl",
|
|
215
|
+
artifactLabel: "support-agent",
|
|
216
|
+
};
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
Artifacts are safe structural summaries. They include bounded test identity, status, trace run id, and trace filename, but they do not read or embed raw trace contents, prompts, generated outputs, request/response bodies, headers, API keys, secrets, or tool payloads. Reporter/artifact failures are diagnostics and do not replace original Vitest failures.
|
|
220
|
+
|
|
221
|
+
## 13. Experimental `@agent-inspect/jest` APIs
|
|
222
|
+
|
|
223
|
+
`@agent-inspect/jest` is an optional experimental package for local Jest failure artifacts. In the v1.8 train it remains private/unpublished until release readiness. It does not add a Jest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
|
|
224
|
+
|
|
225
|
+
Import from `@agent-inspect/jest`:
|
|
226
|
+
|
|
227
|
+
```ts
|
|
228
|
+
import { AgentInspectJestReporter, createAgentInspectJestReporter } from "@agent-inspect/jest";
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
- **`AgentInspectJestReporter`**: default Jest custom reporter class for `reporters: [["@agent-inspect/jest", options]]`.
|
|
232
|
+
- **`createAgentInspectJestReporter(options?)`**: returns a structural reporter facade with `onTestResult` and `onRunComplete` hooks for tests and custom harnesses.
|
|
233
|
+
- **`artifactDir`**: local output directory for safe artifacts; defaults to `.agent-inspect/jest-artifacts`.
|
|
234
|
+
- **`githubSummary`**: optional GitHub step-summary file path. The reporter appends bounded structural counts only and does not use the GitHub API.
|
|
235
|
+
- **`retainSuccessful`**: `false`/undefined keeps no passing-test artifacts; `true` keeps up to `maxSuccessfulTraces`; a number keeps up to that many passing-test artifacts.
|
|
236
|
+
- **`maxSuccessfulTraces`**: upper bound for passing-test artifacts, capped by the reporter.
|
|
237
|
+
- **`associations`**: explicit trace associations keyed by `file::fullName`, `basename::fullName`, or `fullName`.
|
|
238
|
+
- **`resolveTrace(test)`**: optional explicit association resolver for normalized Jest assertion results.
|
|
239
|
+
- **`onDiagnostic(diagnostic)`**: observes non-fatal reporter/artifact failures.
|
|
240
|
+
- **`getDiagnostics()`** and **`getArtifacts()`** expose reporter state for tests and custom harnesses.
|
|
241
|
+
- **`agentInspectJestReporter`**: alias for `createAgentInspectJestReporter`.
|
|
242
|
+
|
|
243
|
+
Jest association is explicit because Jest assertion results do not expose Vitest-style mutable task metadata:
|
|
244
|
+
|
|
245
|
+
```js
|
|
246
|
+
reporters: [
|
|
247
|
+
[
|
|
248
|
+
"@agent-inspect/jest",
|
|
249
|
+
{
|
|
250
|
+
associations: {
|
|
251
|
+
"agent.test.cjs::agent suite agent workflow": {
|
|
252
|
+
runId: "support-agent",
|
|
253
|
+
tracePath: ".agent-inspect/support-agent.jsonl",
|
|
254
|
+
},
|
|
255
|
+
},
|
|
256
|
+
},
|
|
257
|
+
],
|
|
258
|
+
],
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
Artifacts are safe structural summaries. They include bounded test identity, status, trace run id, and trace filename, but they do not read or embed raw trace contents, prompts, generated outputs, request/response bodies, headers, API keys, secrets, or tool payloads. Reporter/artifact failures are diagnostics and do not replace original Jest failures.
|
|
262
|
+
|
|
263
|
+
## 14. Experimental `@agent-inspect/openai-agents` APIs
|
|
264
|
+
|
|
265
|
+
`@agent-inspect/openai-agents` is an optional experimental package for OpenAI Agents JS tracing processor integration. In the v1.8 train it remains private/unpublished until the manual first-publication gate, but runtime metadata mapping is implemented locally.
|
|
191
266
|
|
|
192
267
|
Import from `@agent-inspect/openai-agents`:
|
|
193
268
|
|
|
@@ -195,11 +270,16 @@ Import from `@agent-inspect/openai-agents`:
|
|
|
195
270
|
import { agentInspectProcessor } from "@agent-inspect/openai-agents";
|
|
196
271
|
```
|
|
197
272
|
|
|
198
|
-
- **`agentInspectProcessor(options?)`**: returns a local-only
|
|
273
|
+
- **`agentInspectProcessor(options?)`**: returns a local-only OpenAI Agents `TracingProcessor`.
|
|
199
274
|
- **`installMode`**: always `"setTraceProcessors"` to document the safe replacement install path.
|
|
200
|
-
- **`localOnly`**: always `true`; the
|
|
201
|
-
- **`writer
|
|
202
|
-
- **`
|
|
275
|
+
- **`localOnly`**: always `true`; the processor performs no network I/O and does not install itself globally.
|
|
276
|
+
- **`writer`**: optional explicit local `TraceWriter` for tests, recipes, and controlled runtime integration.
|
|
277
|
+
- **`traceDir`**: optional local directory that creates a file writer inside the adapter package.
|
|
278
|
+
- **`workflowName`**: optional local run name overriding the SDK trace name.
|
|
279
|
+
- **`capture`**: `"metadata-only"` (default) or `"preview"`; `preview` currently emits a diagnostic and falls back to metadata-only.
|
|
280
|
+
- **`redactionProfile`** and **`maxPreviewChars`**: preview-only knobs; when preview capture is unsupported or not selected, they emit diagnostics instead of silently doing nothing.
|
|
281
|
+
- **`getDiagnostics()`**: exposes isolated processor write, lifecycle/configuration, flush, and shutdown failures without throwing into OpenAI Agents callbacks.
|
|
282
|
+
- **`getWriterStats()`**, **`forceFlush()`**, and **`shutdown()`**: explicit writer lifecycle helpers. Failures are captured in diagnostics.
|
|
203
283
|
|
|
204
284
|
Safe future usage must replace processors explicitly:
|
|
205
285
|
|
|
@@ -209,7 +289,9 @@ setTraceProcessors([agentInspectProcessor({ traceDir: "./.agent-inspect" })]);
|
|
|
209
289
|
|
|
210
290
|
Do not use `addTraceProcessor()` as the default AgentInspect path; that leaves existing/default processors in place and can preserve backend export behavior in server runtimes.
|
|
211
291
|
|
|
212
|
-
|
|
292
|
+
The processor records local v0.2 persisted events for trace/run, agent, generation/response, function/tool, handoff, guardrail, MCP tools, custom, transcription, and speech span metadata where safely representable. It does not persist raw prompts, messages, generated text, function inputs/outputs, arbitrary custom data, trace exporter credentials, headers, request bodies, response bodies, or hosted tool payloads by default.
|
|
293
|
+
|
|
294
|
+
## 15. Experimental persisted-event foundation (v1.2.0)
|
|
213
295
|
|
|
214
296
|
These helpers expose the **source-agnostic `PersistedInspectEvent` model** (`schemaVersion: "0.2"`). They are **local-only**, **in-memory**, and **do not change** storage write/read or CLI behavior in v1.2.0.
|
|
215
297
|
|
|
@@ -235,7 +317,7 @@ Related types: `PersistedInspectEvent`, `PersistedEventSourceType`, `PersistedEv
|
|
|
235
317
|
- v0.2 is **not written by default**; use converters and `fixtures/traces-v0.2/` samples for validation.
|
|
236
318
|
- Inspection read paths normalize v0.1 and v0.2 JSONL for local CLI/API use. v0.2 remains experimental as a persisted-event foundation and is not the default writer.
|
|
237
319
|
|
|
238
|
-
##
|
|
320
|
+
## 16. Local observability helpers (v1.4.0+)
|
|
239
321
|
|
|
240
322
|
Read-only helpers for timeline, stats, and search over local JSONL traces. v0.1 manual traces remain the default writer; v0.2 persisted-event files are accepted where the shared dual-format read path is used. Local files only.
|
|
241
323
|
|
|
@@ -245,7 +327,7 @@ Read-only helpers for timeline, stats, and search over local JSONL traces. v0.1
|
|
|
245
327
|
|
|
246
328
|
CLI wrappers: `agent-inspect timeline`, `stats`, `search` — see [CLI.md](./CLI.md).
|
|
247
329
|
|
|
248
|
-
##
|
|
330
|
+
## 17. Report and what helpers (v1.5.0+)
|
|
249
331
|
|
|
250
332
|
Read-only helpers for concise inspection summaries and local reports:
|
|
251
333
|
|
|
@@ -254,7 +336,7 @@ Read-only helpers for concise inspection summaries and local reports:
|
|
|
254
336
|
|
|
255
337
|
Report redaction profiles are key-based safeguards applied to the complete rendered report input, not only to the tree section. Review generated reports before sharing; this is not compliance-grade DLP.
|
|
256
338
|
|
|
257
|
-
##
|
|
339
|
+
## 18. Experimental trace writers (v1.6)
|
|
258
340
|
|
|
259
341
|
Trace writers are the first slice of the v1.6 runtime foundation. They are experimental during v1.x and intended for tests, adapters, and future `createInspector` work.
|
|
260
342
|
|
|
@@ -286,7 +368,7 @@ import type {
|
|
|
286
368
|
|
|
287
369
|
No network writer or vendor sink exists in this package.
|
|
288
370
|
|
|
289
|
-
##
|
|
371
|
+
## 19. Experimental inspector API/runtime (v1.6)
|
|
290
372
|
|
|
291
373
|
`createInspector()` is the experimental public instance API for local-first tracing with explicit writers. It owns an instance-specific runtime context, records v0.2 persisted inspect events, preserves application return values/errors, and exposes diagnostics plus deterministic `flush()`/`close()` lifecycle hooks.
|
|
292
374
|
|
|
@@ -328,7 +410,7 @@ Public methods:
|
|
|
328
410
|
|
|
329
411
|
These APIs are experimental during v1.x. They do not add a default network writer or vendor sink.
|
|
330
412
|
|
|
331
|
-
##
|
|
413
|
+
## 20. Experimental trace readers (v1.6)
|
|
332
414
|
|
|
333
415
|
`agent-inspect/readers` exposes the experimental local trace reader contract and detection pipeline. It includes AgentInspect JSONL for v0.1, v0.2, and mixed local trace files, plus local OpenInference JSON and OTLP JSON compatibility readers.
|
|
334
416
|
|
|
@@ -359,23 +441,43 @@ import type { TraceReader } from "agent-inspect/readers";
|
|
|
359
441
|
|
|
360
442
|
The reader contract does not silently accept arbitrary JSON and does not add OTel SDK, database, hosted ingestion, or network upload dependencies.
|
|
361
443
|
|
|
362
|
-
##
|
|
444
|
+
## 21. Experimental Checks
|
|
445
|
+
|
|
446
|
+
`agent-inspect/checks` exposes the experimental deterministic trace-check engine foundation. It consumes normalized reader output, runs supplied pure rules in stable order, and returns aggregate findings/diagnostics. It does not read files, discover config, call providers, perform network I/O, mutate inputs, or create a new persisted schema.
|
|
447
|
+
|
|
448
|
+
Import from `agent-inspect/checks`:
|
|
449
|
+
|
|
450
|
+
```ts
|
|
451
|
+
import { runTraceChecks } from "agent-inspect/checks";
|
|
452
|
+
import type { TraceCheckRule, TraceCheckResult } from "agent-inspect/checks";
|
|
453
|
+
```
|
|
454
|
+
|
|
455
|
+
- **`runTraceChecks({ read }, { rules?, select?, runId? })`**: executes provided rules against a `TraceReadResult` from `agent-inspect/readers`.
|
|
456
|
+
- **Built-in rule factories**: run, tool, LLM, structure, retrieval, guardrail, decision, safety, and baseline helpers including `createRunStatusRule`, `createToolUsageRule`, `createLlmUsageRule`, `createStructureOrphanRule`, `createStructureCycleRule`, `createStructureRelationshipRule`, `createRetrievalRule`, `createGuardrailRule`, `createDecisionRule`, `createSafetyRawContentRule`, `createSafetySecretPatternRule`, and `createBaselineRegressionRule`.
|
|
457
|
+
- **`TraceCheckRule`**: synchronous pure rule contract.
|
|
458
|
+
- **`TraceCheckResult`**: deterministic aggregate result with findings, evidence, summary counts, and execution diagnostics.
|
|
459
|
+
|
|
460
|
+
The checks API is experimental in v1.x. The `agent-inspect check` CLI uses this API for local reader-backed checks and deterministic JSON output; `agent-inspect artifacts` reuses the same safe findings for local CI artifact bundles and optional step-summary file output. Built-in rules operate on normalized event metadata, tree relationships, bounded summaries, token counts, and normalized baseline facts; safety and baseline findings identify event IDs and field paths rather than emitting raw prompts, outputs, secrets, headers, request/response bodies, or full tool payloads.
|
|
461
|
+
|
|
462
|
+
Recipes: [deterministic-ci-checks](../examples/recipes/deterministic-ci-checks/README.md) for check/baseline/artifact workflows, and [test-reporter-artifacts](../examples/recipes/test-reporter-artifacts/README.md) for Vitest/Jest reporter configuration patterns.
|
|
463
|
+
|
|
464
|
+
## 22. Deprecated APIs
|
|
363
465
|
|
|
364
466
|
No deprecated APIs are declared as of 1.4.0.
|
|
365
467
|
|
|
366
|
-
##
|
|
468
|
+
## 23. Removal / deprecation policy
|
|
367
469
|
|
|
368
470
|
- Stable APIs are not removed in v1.x.
|
|
369
471
|
- If removal is necessary, the API should be **deprecated** first, documented, and kept for a reasonable window (target: at least one minor line) unless security requires faster action.
|
|
370
472
|
|
|
371
|
-
##
|
|
473
|
+
## 24. Backward compatibility policy
|
|
372
474
|
|
|
373
475
|
- Manual trace JSONL (`schemaVersion: "0.1"`) remains readable.
|
|
374
476
|
- Additive schema changes are allowed in minor versions.
|
|
375
477
|
- Breaking changes require a major version.
|
|
376
478
|
- Unknown fields should be ignored where safe.
|
|
377
479
|
|
|
378
|
-
##
|
|
480
|
+
## 25. Examples
|
|
379
481
|
|
|
380
482
|
### Minimal manual trace
|
|
381
483
|
|