agent-inspect 6.17.7 → 6.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/README.md +1 -1
- package/SECURITY.md +5 -0
- package/docs/ADAPTERS.md +34 -4
- package/docs/AI-SDK-ADOPTION.md +25 -0
- package/docs/API.md +11 -9
- package/docs/CLI.md +6 -0
- package/docs/COMPARE.md +6 -0
- package/docs/FIRST-TRACE-IN-5-MINUTES.md +3 -1
- package/docs/LIMITATIONS.md +1 -0
- package/docs/OPENAI-AGENTS-LOCAL.md +1 -0
- package/docs/SAFE-TRACE-SHARING.md +2 -2
- package/docs/STANDARDS.md +1 -0
- package/docs/SUPPORT-LEVELS.md +1 -0
- package/docs/USE-CASES.md +1 -1
- package/docs/VSCODE.md +8 -0
- package/package.json +2 -2
- package/packages/cli/dist/{chunk-25LUDS4T.mjs → chunk-S4TLTLXN.mjs} +2 -2
- package/packages/cli/dist/{chunk-25LUDS4T.mjs.map → chunk-S4TLTLXN.mjs.map} +1 -1
- package/packages/cli/dist/index.cjs +1327 -930
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +1245 -857
- package/packages/cli/dist/index.mjs.map +1 -1
- package/packages/cli/dist/{src-2FS6BLR7.mjs → src-R6V5EZZH.mjs} +3 -3
- package/packages/cli/dist/{src-2FS6BLR7.mjs.map → src-R6V5EZZH.mjs.map} +1 -1
- package/packages/core/dist/advanced.cjs +180 -0
- package/packages/core/dist/advanced.cjs.map +1 -1
- package/packages/core/dist/advanced.d.cts +81 -2
- package/packages/core/dist/advanced.d.ts +81 -2
- package/packages/core/dist/advanced.mjs +176 -1
- package/packages/core/dist/advanced.mjs.map +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,30 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 6.18.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 4a1cd87: Bounded `preview` capture parity across official framework adapters through one shared helper (#311).
|
|
8
|
+
|
|
9
|
+
`@agent-inspect/ai-sdk` and `@agent-inspect/openai-agents` previously accepted `capture: "preview"` and fell back to metadata-only with an `AI_ADAPTER_PREVIEW_NOT_AVAILABLE` warning. Both now persist bounded `*Preview` attributes, and `@agent-inspect/langchain` resolves its existing preview support through the same helper, so `capture`, `redactionProfile`, `maxPreviewChars`, and `onDiagnostic` mean the same thing in every adapter. A cross-adapter conformance matrix enforces the contract.
|
|
10
|
+
|
|
11
|
+
Redaction runs on the structured value before a preview string is persisted, `maxPreviewChars` is a hard bound that the `share` and `strict` profiles cap further, and cycles, bigints, and throwing getters are handled without throwing into the traced application. Capture diagnostics are stable: `AI_CAPTURE_FIELD_UNAVAILABLE`, `AI_CAPTURE_PREVIEW_TRUNCATED`, and `AI_CAPTURE_PREVIEW_REDACTED`, reported through `onDiagnostic` and counted in `getDiagnostics().capture`.
|
|
12
|
+
|
|
13
|
+
`metadata-only` remains the default and stays silent, there is no full-content capture mode, no network I/O is added, and the helper ships on the existing `agent-inspect/advanced` subpath rather than the root API. Preview redaction is key-based and bounded — it is not a sanitization guarantee for secrets embedded in free text.
|
|
14
|
+
|
|
15
|
+
- ba794e1: Add a no-key packed-consumer golden path for `@agent-inspect/ai-sdk` (`scripts/packed-ai-sdk-e2e.mjs`) and wire it into `pack:smoke` (#307, #213).
|
|
16
|
+
- 4fcee12: Render `view --errors-only` as a pruned human error tree (ancestors + failed nodes) while keeping `--errors-only --json` as the filtered event list (#330).
|
|
17
|
+
- 8c65ee5: Surface residual safety assessment after CLI `redact` (#328) and add a bounded local `--policy` JSON for redact/verify-safe (#329).
|
|
18
|
+
- 63b9606: Improve the broken-agent starter so good and regression paths return the same final answer while TraceContract trajectory checks PASS vs FAIL (`prove-same-output-wrong-path.mjs`).
|
|
19
|
+
|
|
20
|
+
## 6.17.8
|
|
21
|
+
|
|
22
|
+
### Patch Changes
|
|
23
|
+
|
|
24
|
+
- ddea9ea: Strictly validate `clean --keep` as a complete positive decimal integer token before planning deletions, so malformed values like `1.5`, `1e2`, or `10oops` fail closed instead of partial-parsing (#339, #340).
|
|
25
|
+
- b855436: Clarify `doctor` remediations with doc links for packed-consumer install mistakes, and land contributor regression coverage for packed-adapter golden paths, TraceFacts schema parity, and MCP protocol-state fixtures (#296, #305, #294, #302).
|
|
26
|
+
- 18941d0: Treat trace-derived MCP content as untrusted application data: advertise `instructions` on initialize, warn on trace-bearing tool descriptions, and add adversarial no-execution coverage (#344).
|
|
27
|
+
|
|
3
28
|
## 6.17.7
|
|
4
29
|
|
|
5
30
|
### Patch Changes
|
package/README.md
CHANGED
|
@@ -210,7 +210,7 @@ The root package is enough for custom capture, the CLI, checks, and Evidence wor
|
|
|
210
210
|
|
|
211
211
|
## Status and documentation
|
|
212
212
|
|
|
213
|
-
**Current published baseline:** **6.
|
|
213
|
+
**Current published baseline:** **6.18.0** · persisted schema `1.0` · Node.js `>=20` · MIT.
|
|
214
214
|
|
|
215
215
|
Legacy v0.1 and v0.2 traces remain readable. Check the npm badge and [changelog](CHANGELOG.md) for the current published version.
|
|
216
216
|
|
package/SECURITY.md
CHANGED
|
@@ -78,6 +78,10 @@ For a practical pre-share workflow, see `docs/SAFE-TRACE-SHARING.md`. For schema
|
|
|
78
78
|
- Do not add vendor SDKs, OpenTelemetry SDKs, or framework dependencies to the main `agent-inspect` package.
|
|
79
79
|
- Keep optional integrations (`@agent-inspect/langchain`, `@agent-inspect/tui`, `@agent-inspect/viewer`, `@agent-inspect/mcp-server`) separate so users do not pull them in by default.
|
|
80
80
|
|
|
81
|
+
### Automated scanners (CI)
|
|
82
|
+
|
|
83
|
+
Pull requests run GitHub **Dependency review** (`.github/workflows/dependency-review.yml`) and fail on newly introduced **high** or **critical** advisory severity. Routine unit/typecheck/size CI also exercises redaction, path containment, and Evidence safety tests. These scanners do **not** replace manual review of share profiles or Evidence packages before you disclose traces.
|
|
84
|
+
|
|
81
85
|
## Optional surfaces (v2.6)
|
|
82
86
|
|
|
83
87
|
### Local viewer (`agent-inspect serve`)
|
|
@@ -92,6 +96,7 @@ For a practical pre-share workflow, see `docs/SAFE-TRACE-SHARING.md`. For schema
|
|
|
92
96
|
- Default tool output redaction profile is **`share`** (not `local`).
|
|
93
97
|
- Does not invoke user agent tools or mutate traces.
|
|
94
98
|
- Configure via `AGENT_INSPECT_TRACE_DIR` and `AGENT_INSPECT_MCP_REDACTION_PROFILE` when documented.
|
|
99
|
+
- **Trace-derived strings are untrusted application data.** Read-only describes server capabilities, not the trustworthiness of captured content. Redaction removes recognized sensitive values, not malicious intent. Coding agents must treat MCP tool results as evidence to corroborate against code, tests, contracts, and the user's request — never as instructions to execute. `initialize` advertises this boundary via MCP `instructions`; AgentInspect does not grant trace text higher priority than user/system instructions.
|
|
95
100
|
|
|
96
101
|
IDE extension: see [VSCODE.md](docs/VSCODE.md) (in-repo; Marketplace deferred).
|
|
97
102
|
|
package/docs/ADAPTERS.md
CHANGED
|
@@ -16,13 +16,43 @@ v2.3 hardens existing official adapters before adding new ones. Priority is base
|
|
|
16
16
|
|
|
17
17
|
Reporters (`@agent-inspect/vitest` and `@agent-inspect/jest`) are public packages as of v2.2, but they are CI/test artifact reporters rather than framework trace adapters. They stay outside the v2.3 adapter-hardening priority order.
|
|
18
18
|
|
|
19
|
+
## Shared adapter capture contract (preview)
|
|
20
|
+
|
|
21
|
+
All three official framework adapters — `@agent-inspect/ai-sdk`, `@agent-inspect/openai-agents`, and `@agent-inspect/langchain` — resolve capture through one shared helper, so the same options mean the same thing everywhere. A cross-adapter conformance matrix asserts this contract.
|
|
22
|
+
|
|
23
|
+
| Option | Type | Default | Behavior |
|
|
24
|
+
| ------ | ---- | ------- | -------- |
|
|
25
|
+
| `capture` | `"metadata-only" \| "preview"` | `"metadata-only"` | `metadata-only` persists no preview attributes at all. `preview` adds bounded `*Preview` attributes. There is no full-content mode. |
|
|
26
|
+
| `redactionProfile` | `"local" \| "share" \| "strict"` | `"local"` | Key-based redaction applied **before** a preview string is persisted. `share` and `strict` also lower the effective `maxPreviewChars` ceiling; `strict` replaces preview values entirely. |
|
|
27
|
+
| `maxPreviewChars` | `number` | adapter default, profile-capped | Hard upper bound on each persisted preview string, including the truncation marker. |
|
|
28
|
+
| `onDiagnostic` | `(diagnostic) => void` | — | Local callback for capture diagnostics. Listener failures are isolated and never reach the traced application. |
|
|
29
|
+
|
|
30
|
+
Capture diagnostic codes are stable:
|
|
31
|
+
|
|
32
|
+
| Code | Meaning |
|
|
33
|
+
| ---- | ------- |
|
|
34
|
+
| `AI_CAPTURE_FIELD_UNAVAILABLE` | `capture: "preview"` was requested but the framework did not provide the field, or the value could not be safely serialized. Nothing is persisted for that field. |
|
|
35
|
+
| `AI_CAPTURE_PREVIEW_TRUNCATED` | The preview string hit the `maxPreviewChars` bound and was truncated. |
|
|
36
|
+
| `AI_CAPTURE_PREVIEW_REDACTED` | Redaction replaced at least one value inside the preview before persistence. |
|
|
37
|
+
|
|
38
|
+
Counters for the same events are available without a listener via each adapter's `getDiagnostics().capture`.
|
|
39
|
+
|
|
40
|
+
What the contract guarantees:
|
|
41
|
+
|
|
42
|
+
- **Metadata-only stays the default and stays silent** — no preview attributes, no capture diagnostics.
|
|
43
|
+
- **Redaction runs before persistence** — key-based redaction is applied to the structured value, not to an already-serialized string.
|
|
44
|
+
- **Bounds are hard** — cycles, bigints, and getters that throw are handled without throwing into the traced application, and every persisted preview respects the resolved bound.
|
|
45
|
+
- **No network, no new root/core dependencies** — the helper lives in core and is exported from `agent-inspect/advanced`.
|
|
46
|
+
|
|
47
|
+
This is bounded preview capture, **not sanitization**: redaction is key-based, so a secret embedded in free text (for example inside a prompt string) can still appear in a preview. Treat preview traces as sensitive and run `agent-inspect redact` before sharing — see [SAFE-TRACE-SHARING.md](./SAFE-TRACE-SHARING.md).
|
|
48
|
+
|
|
19
49
|
## Vercel AI SDK (`@agent-inspect/ai-sdk`)
|
|
20
50
|
|
|
21
|
-
**Full guide:** [AI-SDK-ADOPTION.md](./AI-SDK-ADOPTION.md)
|
|
51
|
+
**Full guide:** [AI-SDK-ADOPTION.md](./AI-SDK-ADOPTION.md) · **Capture path:** [CHOOSE-YOUR-CAPTURE-PATH.md](./CHOOSE-YOUR-CAPTURE-PATH.md)
|
|
22
52
|
|
|
23
53
|
**Status:** experimental adapter — optional package published in the aligned v2.2.0 package set and hardened in the v2.3 adapter train.
|
|
24
54
|
|
|
25
|
-
The adapter has hardened lifecycle identity and parallel integration isolation.
|
|
55
|
+
The adapter has hardened lifecycle identity and parallel integration isolation. Capture is **metadata-only by default**. Opting into `capture: "preview"` persists bounded, redacted preview fields through the [shared adapter capture contract](#shared-adapter-capture-contract-preview); there is no full-content mode.
|
|
26
56
|
|
|
27
57
|
### Install
|
|
28
58
|
|
|
@@ -59,7 +89,7 @@ const result = await generateText({
|
|
|
59
89
|
- **Metadata-only by default** — records model, finish reason, token usage, timing, and safe counts/summaries.
|
|
60
90
|
- **Required safe telemetry settings** — set `recordInputs: false` and `recordOutputs: false` on every AI SDK call using this adapter.
|
|
61
91
|
- **No raw payload capture by default** — prompts, messages, generated text, stream chunks, tool inputs/outputs, headers, request bodies, and response bodies are not persisted.
|
|
62
|
-
- **
|
|
92
|
+
- **Opt-in bounded previews** — `capture: "preview"` adds `*Preview` attributes for prompt, message, text, tool input, and tool output fields, redacted and truncated by the [shared capture contract](#shared-adapter-capture-contract-preview). Headers, request bodies, response bodies, and `experimental_context` are never previewed.
|
|
63
93
|
|
|
64
94
|
### Local no-network recipe
|
|
65
95
|
|
|
@@ -375,7 +405,7 @@ Integration modes:
|
|
|
375
405
|
- **No upload behavior** — the processor writes only to an explicit local writer or `traceDir`.
|
|
376
406
|
- **Metadata-only by default** — records trace/span IDs, parentage, names, timing, status, errors, safe model/tool names, token counts, and bounded summaries.
|
|
377
407
|
- **No raw payload capture by default** — prompts, messages, generated text, function inputs/outputs, arbitrary custom data, trace exporter credentials, headers, request bodies, response bodies, and hosted tool payloads are not persisted.
|
|
378
|
-
- **
|
|
408
|
+
- **Opt-in bounded previews** — `capture: "preview"` adds `inputPreview` / `outputPreview` attributes for generation, function, response, custom, transcription, speech, and speech-group spans through the [shared capture contract](#shared-adapter-capture-contract-preview). Span types without a payload concept (handoff, guardrail, agent) stay metadata-only and report no unavailable fields.
|
|
379
409
|
- **Fixture-backed lifecycle coverage** — local tests and the recipe cover agent, generation, function tool, handoff, guardrail, response, MCP tools, custom, transcription, and speech span shapes without provider calls.
|
|
380
410
|
|
|
381
411
|
Full API: [API.md](./API.md) §14.
|
package/docs/AI-SDK-ADOPTION.md
CHANGED
|
@@ -84,6 +84,15 @@ See [examples/recipes/ai-sdk-next-route](../../examples/recipes/ai-sdk-next-rout
|
|
|
84
84
|
| `recordOutputs: false` | yes | Prevents model output capture in telemetry |
|
|
85
85
|
| `capture: "metadata-only"` | yes (adapter) | AgentInspect adapter redacts/bounds persisted fields |
|
|
86
86
|
|
|
87
|
+
Opting into `capture: "preview"` persists bounded, redacted `*Preview` attributes
|
|
88
|
+
for prompt, message, text, and tool payload fields. Tune it with
|
|
89
|
+
`maxPreviewChars`, raise `redactionProfile` to `share` or `strict` for stricter
|
|
90
|
+
bounds, and observe `AI_CAPTURE_FIELD_UNAVAILABLE` /
|
|
91
|
+
`AI_CAPTURE_PREVIEW_TRUNCATED` / `AI_CAPTURE_PREVIEW_REDACTED` through
|
|
92
|
+
`onDiagnostic` or `getDiagnostics().capture`. Preview traces can still contain
|
|
93
|
+
sensitive free text; redact before sharing. Full contract:
|
|
94
|
+
[ADAPTERS.md](./ADAPTERS.md#shared-adapter-capture-contract-preview).
|
|
95
|
+
|
|
87
96
|
## Inspect locally
|
|
88
97
|
|
|
89
98
|
```bash
|
|
@@ -106,4 +115,20 @@ npx agent-inspect check .agent-inspect/<run>.jsonl
|
|
|
106
115
|
- [ai-sdk-local-telemetry](../../examples/recipes/ai-sdk-local-telemetry/)
|
|
107
116
|
- [ai-sdk-next-route](../../examples/recipes/ai-sdk-next-route/)
|
|
108
117
|
|
|
118
|
+
## No-key packed consumer check
|
|
119
|
+
|
|
120
|
+
After building the repository, run the clean packed-consumer path directly:
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
pnpm build
|
|
124
|
+
node scripts/packed-ai-sdk-e2e.mjs
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
The same check participates in `pnpm pack:smoke`. It installs the packed root
|
|
128
|
+
and `@agent-inspect/ai-sdk` tarballs with the supported `ai` peer in a clean
|
|
129
|
+
temporary consumer, then verifies provider-independent telemetry integration
|
|
130
|
+
through a deterministic AI SDK mock model. It uses no provider package, API
|
|
131
|
+
key, or live provider call, and asserts that metadata-only evidence excludes
|
|
132
|
+
the fixture prompt and output.
|
|
133
|
+
|
|
109
134
|
See also [ADAPTERS.md](./ADAPTERS.md) and [ADAPTER-CONFORMANCE.md](./ADAPTER-CONFORMANCE.md).
|
package/docs/API.md
CHANGED
|
@@ -206,12 +206,13 @@ import { agentInspect } from "@agent-inspect/ai-sdk";
|
|
|
206
206
|
- **`writer`**: optional explicit local `TraceWriter` for tests, recipes, and controlled runtime integration.
|
|
207
207
|
- **`traceDir`**: optional local directory that creates a file writer inside the adapter package.
|
|
208
208
|
- **`runName`**: optional local run name.
|
|
209
|
-
- **`capture`**: `"metadata-only"` (default) or `"preview"`; `preview`
|
|
210
|
-
- **`redactionProfile`** and **`maxPreviewChars`**: preview-only knobs;
|
|
211
|
-
- **`
|
|
209
|
+
- **`capture`**: `"metadata-only"` (default) or `"preview"`; `preview` persists bounded, redacted `*Preview` attributes via the shared adapter capture contract.
|
|
210
|
+
- **`redactionProfile`** and **`maxPreviewChars`**: preview-only knobs; redaction runs before persistence and `maxPreviewChars` is a hard bound (further capped by `share` / `strict` profiles).
|
|
211
|
+
- **`onDiagnostic`**: optional local listener for capture diagnostics (`AI_CAPTURE_FIELD_UNAVAILABLE`, `AI_CAPTURE_PREVIEW_TRUNCATED`, `AI_CAPTURE_PREVIEW_REDACTED`). Listener failures are isolated.
|
|
212
|
+
- **`getDiagnostics()`**: exposes isolated adapter write, lifecycle/configuration, flush, and close failures without throwing into AI SDK callbacks, plus a `capture` block with the resolved capture mode and preview counters.
|
|
212
213
|
- **`getWriterStats()`**, **`flush()`**, and **`close()`**: explicit writer lifecycle helpers. Failures are captured in diagnostics.
|
|
213
214
|
|
|
214
|
-
Every AI SDK call using the adapter must keep telemetry local
|
|
215
|
+
Every AI SDK call using the adapter must keep telemetry local:
|
|
215
216
|
|
|
216
217
|
```ts
|
|
217
218
|
experimental_telemetry: {
|
|
@@ -222,7 +223,7 @@ experimental_telemetry: {
|
|
|
222
223
|
}
|
|
223
224
|
```
|
|
224
225
|
|
|
225
|
-
The adapter records local v0.2 persisted events for run, LLM step, and tool lifecycle metadata.
|
|
226
|
+
The adapter records local v0.2 persisted events for run, LLM step, and tool lifecycle metadata. In the default `metadata-only` mode it does not persist raw prompts, messages, generated text, stream chunks, tool inputs, tool outputs, headers, request bodies, response bodies, or user `experimental_context`. With `capture: "preview"`, prompt, message, text, tool input, and tool output fields are persisted as bounded, redacted previews; headers, request bodies, response bodies, and `experimental_context` are still never previewed. See [ADAPTERS.md](./ADAPTERS.md#shared-adapter-capture-contract-preview).
|
|
226
227
|
|
|
227
228
|
No network writer, OpenTelemetry exporter, provider wrapper, or global monkey-patch is part of this package.
|
|
228
229
|
|
|
@@ -343,9 +344,10 @@ import { agentInspectProcessor } from "@agent-inspect/openai-agents";
|
|
|
343
344
|
- **`writer`**: optional explicit local `TraceWriter` for tests, recipes, and controlled runtime integration.
|
|
344
345
|
- **`traceDir`**: optional local directory that creates a file writer inside the adapter package.
|
|
345
346
|
- **`workflowName`**: optional local run name overriding the SDK trace name.
|
|
346
|
-
- **`capture`**: `"metadata-only"` (default) or `"preview"`; `preview`
|
|
347
|
-
- **`redactionProfile`** and **`maxPreviewChars`**: preview-only knobs;
|
|
348
|
-
- **`
|
|
347
|
+
- **`capture`**: `"metadata-only"` (default) or `"preview"`; `preview` persists bounded, redacted `inputPreview` / `outputPreview` span attributes via the shared adapter capture contract.
|
|
348
|
+
- **`redactionProfile`** and **`maxPreviewChars`**: preview-only knobs; redaction runs before persistence and `maxPreviewChars` is a hard bound (further capped by `share` / `strict` profiles).
|
|
349
|
+
- **`onDiagnostic`**: optional local listener for capture diagnostics (`AI_CAPTURE_FIELD_UNAVAILABLE`, `AI_CAPTURE_PREVIEW_TRUNCATED`, `AI_CAPTURE_PREVIEW_REDACTED`). Listener failures are isolated.
|
|
350
|
+
- **`getDiagnostics()`**: exposes isolated processor write, lifecycle/configuration, flush, and shutdown failures without throwing into OpenAI Agents callbacks, plus a `capture` block with the resolved capture mode and preview counters.
|
|
349
351
|
- **`getWriterStats()`**, **`forceFlush()`**, and **`shutdown()`**: explicit writer lifecycle helpers. Failures are captured in diagnostics.
|
|
350
352
|
|
|
351
353
|
Safe future usage must replace processors explicitly:
|
|
@@ -356,7 +358,7 @@ setTraceProcessors([agentInspectProcessor({ traceDir: "./.agent-inspect" })]);
|
|
|
356
358
|
|
|
357
359
|
Do not use `addTraceProcessor()` as the default AgentInspect path; that leaves existing/default processors in place and can preserve backend export behavior in server runtimes.
|
|
358
360
|
|
|
359
|
-
The processor records local v0.2 persisted events for trace/run, agent, generation/response, function/tool, handoff, guardrail, MCP tools, custom, transcription, and speech span metadata where safely representable. It does not persist raw prompts, messages, generated text, function inputs/outputs, arbitrary custom data, trace exporter credentials, headers, request bodies, response bodies, or hosted tool payloads by default.
|
|
361
|
+
The processor records local v0.2 persisted events for trace/run, agent, generation/response, function/tool, handoff, guardrail, MCP tools, custom, transcription, and speech span metadata where safely representable. It does not persist raw prompts, messages, generated text, function inputs/outputs, arbitrary custom data, trace exporter credentials, headers, request bodies, response bodies, or hosted tool payloads by default. With `capture: "preview"`, span input/output fields are persisted as bounded, redacted previews; trace exporter credentials, headers, and request/response bodies are still never previewed. See [ADAPTERS.md](./ADAPTERS.md#shared-adapter-capture-contract-preview).
|
|
360
362
|
|
|
361
363
|
## 15. Experimental persisted-event foundation (v1.2.0)
|
|
362
364
|
|
package/docs/CLI.md
CHANGED
|
@@ -445,12 +445,17 @@ Options:
|
|
|
445
445
|
- `--profile <local|share|strict>`: redaction profile (default `share`)
|
|
446
446
|
- `-o, --output <path>`: write redacted content to a file
|
|
447
447
|
- `--json`: print deterministic JSON wrapper with findings
|
|
448
|
+
- `--policy <path>`: local JSON redaction policy (`extraKeys` + bounded `literal` / `prefix` / `typed` patterns). No remote fetch; no secrets on argv. See [SAFETY-POLICY.md](SAFETY-POLICY.md).
|
|
449
|
+
- `--fail-on-residual`: opt-in non-zero exit when residual safety is `UNSAFE` or `UNKNOWN` (default exit codes unchanged)
|
|
450
|
+
|
|
451
|
+
After redaction, the command surfaces a **residual safety assessment** using the same local detector pipeline as `verify-safe`. Human mode prints a concise stderr warning when residual status is not `SAFE`. JSON mode adds an additive `residualAssessment` field (`status`, finding counts, codes only — never matched secret values). Residual status uses `SAFE` | `SAFE_WITH_WARNINGS` | `UNSAFE` | `UNKNOWN`. Supported AgentInspect traces get a full assessment; arbitrary JSON that is not a supported trace yields `UNKNOWN`. Redact never certifies safe sharing — finish with `verify-safe` before publishing.
|
|
448
452
|
|
|
449
453
|
Examples:
|
|
450
454
|
|
|
451
455
|
```bash
|
|
452
456
|
npx agent-inspect redact trace.jsonl --profile share --json
|
|
453
457
|
npx agent-inspect redact trace.jsonl --profile strict -o trace.share.jsonl
|
|
458
|
+
npx agent-inspect redact trace.jsonl --policy ./redact-policy.json --fail-on-residual
|
|
454
459
|
```
|
|
455
460
|
|
|
456
461
|
Recipe: [redact-share-safe-file](../examples/recipes/redact-share-safe-file/README.md).
|
|
@@ -486,6 +491,7 @@ Options:
|
|
|
486
491
|
- `--max-array-length <number>`: unsafe threshold for array values
|
|
487
492
|
- `--max-object-keys <number>`: unsafe threshold for object key counts
|
|
488
493
|
- `--max-serialized-bytes <number>`: unsafe threshold for serialized values
|
|
494
|
+
- `--policy <path>`: local JSON redaction policy shared with `redact` (`extraKeys` + bounded patterns)
|
|
489
495
|
|
|
490
496
|
The scan looks for raw prompt/output-like capture paths, unredacted sensitive-looking keys, secret-like string patterns, and oversized values. It reports evidence paths rather than raw prompt, output, request/response, header, API key, secret, or full tool payload values. Secret detection is best-effort and should not be treated as exhaustive.
|
|
491
497
|
|
package/docs/COMPARE.md
CHANGED
|
@@ -11,12 +11,18 @@ AgentInspect is a local-first trace workbench for TypeScript AI agents: capture
|
|
|
11
11
|
Position AgentInspect as the **inner-loop local evidence** layer:
|
|
12
12
|
|
|
13
13
|
```text
|
|
14
|
+
Observability collects, searches, and monitors traces.
|
|
15
|
+
AgentInspect lets a local TypeScript test or CI job fail when the agent
|
|
16
|
+
takes an unacceptable path, then packages bounded reviewable evidence.
|
|
17
|
+
|
|
14
18
|
local evidence and inner-loop debugging → AgentInspect
|
|
15
19
|
production observability → LangSmith / Langfuse / MLflow / Phoenix / APM
|
|
16
20
|
prompt/output and red-team eval → Promptfoo / Evalite / other eval tools
|
|
17
21
|
generic OTel trace access via MCP → OTel MCP servers
|
|
18
22
|
```
|
|
19
23
|
|
|
24
|
+
Proof: `examples/starters/broken-agent-debugging` — good and regression variants return the **same** final answer; trajectory checks PASS vs FAIL (`node prove-same-output-wrong-path.mjs`).
|
|
25
|
+
|
|
20
26
|
Handoff story:
|
|
21
27
|
|
|
22
28
|
```text
|
|
@@ -31,6 +31,8 @@ npx agent-inspect init --yes
|
|
|
31
31
|
Creates `agent-inspect.config.ts`, `.agent-inspect/`, and `examples/agent-inspect-demo.mjs`.
|
|
32
32
|
`init` scaffolds files; it does **not** write a trace by itself.
|
|
33
33
|
|
|
34
|
+
Framework users: pick the correct capture path first — [CHOOSE-YOUR-CAPTURE-PATH.md](./CHOOSE-YOUR-CAPTURE-PATH.md).
|
|
35
|
+
|
|
34
36
|
## Minutes 1–2: Run
|
|
35
37
|
|
|
36
38
|
```bash
|
|
@@ -73,7 +75,7 @@ npx agent-inspect redact <run-id> --dir .agent-inspect --profile share -o redact
|
|
|
73
75
|
|
|
74
76
|
| If you use… | Go to |
|
|
75
77
|
| ----------- | ----- |
|
|
76
|
-
| Broken agent demo | [broken-agent-debugging starter](../examples/starters/broken-agent-debugging/README.md) |
|
|
78
|
+
| Broken agent demo (same answer, wrong path) | [broken-agent-debugging starter](../examples/starters/broken-agent-debugging/README.md) (`node prove-same-output-wrong-path.mjs`) |
|
|
77
79
|
| Coding-agent MCP loop | [CODING-AGENT-LOOP.md](./CODING-AGENT-LOOP.md) · [coding-agent-debug-loop](../examples/starters/coding-agent-debug-loop/README.md) |
|
|
78
80
|
| Contracts / CI gates | [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md) · [SUITES-COHORTS-GATES.md](./SUITES-COHORTS-GATES.md) |
|
|
79
81
|
| AI SDK | [AI SDK adoption](./AI-SDK-ADOPTION.md) |
|
package/docs/LIMITATIONS.md
CHANGED
|
@@ -35,6 +35,7 @@ This document states what AgentInspect **does not** provide today. It complement
|
|
|
35
35
|
- **OpenAI Agents JS support is experimental.** `@agent-inspect/openai-agents` maps metadata-only runtime spans through the safe `setTraceProcessors()` boundary and does not capture raw payloads by default. The v1.9 package publication retry is a separate maintainer npm automation task, not part of the v2 contract work.
|
|
36
36
|
- **LangGraph support is a boundary decision, not a separate package.** Initial support is expected through `@agent-inspect/langchain` callbacks unless no-network fixtures prove a separate package is needed.
|
|
37
37
|
- **No root/core adapter dependencies.** AI SDK, OpenAI Agents, LangGraph, OpenTelemetry, and LangChain remain outside the root/core runtime dependency graph.
|
|
38
|
+
- **Preview capture is bounded, not sanitized.** `capture: "preview"` persists truncated, key-redacted previews of framework-provided input/output fields. Key-based redaction cannot detect a secret embedded in free text, and there is no full-content capture mode. Fields the framework never exposes are reported as `AI_CAPTURE_FIELD_UNAVAILABLE` rather than reconstructed.
|
|
38
39
|
|
|
39
40
|
## LangChain streaming (v1.3.0)
|
|
40
41
|
|
|
@@ -36,6 +36,7 @@ setTraceProcessors([
|
|
|
36
36
|
- Default `metadata-only` capture
|
|
37
37
|
- No automatic upload from AgentInspect
|
|
38
38
|
- Replacing processors does not by itself redact OpenAI SDK network traffic — review OpenAI SDK settings separately
|
|
39
|
+
- `capture: "preview"` is opt-in and persists bounded, redacted `inputPreview` / `outputPreview` span attributes; tune with `maxPreviewChars` and `redactionProfile`, and watch `onDiagnostic` / `getDiagnostics().capture` for `AI_CAPTURE_FIELD_UNAVAILABLE`. There is no full-content mode. See [ADAPTERS.md](./ADAPTERS.md#shared-adapter-capture-contract-preview).
|
|
39
40
|
|
|
40
41
|
## Recipes
|
|
41
42
|
|
|
@@ -6,7 +6,7 @@ AgentInspect traces, log-ingest outputs, and exports are local files. They may s
|
|
|
6
6
|
|
|
7
7
|
This guide is practical sharing guidance, not a guarantee that any artifact is safe to publish. Redaction profiles are **best-effort transformation**, not compliance-grade DLP or a safety certification. Always finish with `verify-safe` before sharing.
|
|
8
8
|
|
|
9
|
-
Built-in profiles redact high-confidence credential forms (provider keys, JWTs, bearer tokens, and bounded `token=` / `api_key=` / `internal_token=` style key/value secrets). Broad or context-sensitive findings—such as private filesystem paths—may still appear under `verify-safe` and
|
|
9
|
+
Built-in profiles redact high-confidence credential forms (provider keys, JWTs, bearer tokens, and bounded `token=` / `api_key=` / `internal_token=` style key/value secrets). Broad or context-sensitive findings—such as private filesystem paths—may still appear under `verify-safe` and as residual status after `redact`. Org-specific patterns can use a local CLI `--policy` JSON file (`extraKeys` plus bounded literal/prefix/typed patterns) or programmatic custom detectors.
|
|
10
10
|
|
|
11
11
|
`strict` is a stricter **rule set** (more keys), not a promise that every input produces bytes different from `share`.
|
|
12
12
|
|
|
@@ -63,7 +63,7 @@ Replace sensitive data with clear placeholders such as `example.test`, `user@exa
|
|
|
63
63
|
- Redacted copies from `agent-inspect redact`: review the output file itself; findings show detector/path/action evidence but do not certify full safety.
|
|
64
64
|
- OpenInference / OTLP JSON exports: check attributes, span names, events, and resource metadata.
|
|
65
65
|
- Structured log ingest configs: confirm mapped keys do not pull in full request bodies, headers, raw prompts, or unbounded output fields.
|
|
66
|
-
-
|
|
66
|
+
- Framework adapter traces (ai-sdk, openai-agents, langchain): keep `capture: "metadata-only"` for shareable examples; review `capture: "preview"` traces carefully because previews can include prompt or output fragments. Preview redaction is key-based and bounded, not a sanitization guarantee.
|
|
67
67
|
- Third-party adapter packages: follow the [Adapter SDK privacy checklist](./ADAPTER-SDK-PRIVACY.md) before sharing adapter traces, examples, or registry submissions.
|
|
68
68
|
|
|
69
69
|
## When to use each profile
|
package/docs/STANDARDS.md
CHANGED
|
@@ -54,6 +54,7 @@ Run `pnpm public-truth:check` and `pnpm docs:check` after changing these sources
|
|
|
54
54
|
|
|
55
55
|
- [Phoenix / OpenInference](../examples/recipes/phoenix-openinference-import/)
|
|
56
56
|
- [Langfuse self-hosted](../examples/recipes/langfuse-local-import/)
|
|
57
|
+
- [Local OpenTelemetry Collector round-trip](./OTEL-COLLECTOR-ROUNDTRIP.md)
|
|
57
58
|
|
|
58
59
|
## Vendor graduation (manual)
|
|
59
60
|
|
package/docs/SUPPORT-LEVELS.md
CHANGED
|
@@ -75,6 +75,7 @@ See [INSTALL-KITS.md](./INSTALL-KITS.md).
|
|
|
75
75
|
- Persisted schema **1.0**; v0.1 / v0.2 / 1.0 traces remain readable
|
|
76
76
|
- Optional packages do not add root/core runtime dependencies
|
|
77
77
|
- Network behavior is explicit (see [NETWORK-BEHAVIOR.md](./NETWORK-BEHAVIOR.md))
|
|
78
|
+
- Every Supported official adapter honors the same capture contract — `metadata-only` default, opt-in bounded `preview`, redaction before persistence, shared diagnostic codes — enforced by a cross-adapter conformance matrix (see [ADAPTERS.md](./ADAPTERS.md#shared-adapter-capture-contract-preview))
|
|
78
79
|
|
|
79
80
|
## Promotion criteria
|
|
80
81
|
|
package/docs/USE-CASES.md
CHANGED
|
@@ -24,7 +24,7 @@ npx agent-inspect redact <run-id> --dir .agent-inspect --profile share -o safe.j
|
|
|
24
24
|
npx agent-inspect verify-safe <run-id> --dir .agent-inspect
|
|
25
25
|
```
|
|
26
26
|
|
|
27
|
-
**Starter:** [broken-agent-debugging](../examples/starters/broken-agent-debugging/README.md) (
|
|
27
|
+
**Starter:** [broken-agent-debugging](../examples/starters/broken-agent-debugging/README.md) (same final answer, wrong trajectory — `node prove-same-output-wrong-path.mjs`) or [custom-observe](../examples/starters/custom-observe/README.md)
|
|
28
28
|
|
|
29
29
|
**Not:** Live model replay or hosted trace UI.
|
|
30
30
|
|
package/docs/VSCODE.md
CHANGED
|
@@ -2,6 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
Read-only sidebar for local trace directories. The extension shells out to the published `agent-inspect` CLI (`list`, `view`, `timeline`, `report`, `check`, `doctor`, `verify-safe`).
|
|
4
4
|
|
|
5
|
+
**Support level:** Experimental (unpublished). Ignored by Changesets; not part of the fixed npm release group.
|
|
6
|
+
|
|
7
|
+
## Product scope decision (6.18.0-H)
|
|
8
|
+
|
|
9
|
+
**Disposition: defer (Option A).** Keep the in-repo extension unpublished. Core, official adapters, CLI, and Evidence take precedence over Marketplace packaging. Open PR #295 (sample trace command) and related issues (#66, #65) stay out of the active 6.18 implementation train until a later capacity window revisits VS Code.
|
|
10
|
+
|
|
11
|
+
Do not merge #295 merely to clear the contributor queue.
|
|
12
|
+
|
|
5
13
|
## Develop
|
|
6
14
|
|
|
7
15
|
```bash
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-inspect",
|
|
3
|
-
"version": "6.
|
|
3
|
+
"version": "6.18.0",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"description": "Local evidence debugger and trajectory-test toolkit for TypeScript AI agents — execution trees, TraceContract checks, Evidence v2, and read-only MCP",
|
|
@@ -235,7 +235,7 @@
|
|
|
235
235
|
"test:all": "pnpm run typecheck && pnpm run linked-versions:check && pnpm run build && pnpm run test && pnpm run size",
|
|
236
236
|
"prepublish:checks": "pnpm run typecheck && pnpm run test && pnpm run test:coverage && pnpm run build && pnpm run fixtures:check && pnpm run recipes:check && pnpm run size && pnpm run linked-versions:check && pnpm run repo:health && pnpm run pack:smoke",
|
|
237
237
|
"pack:dry-run": "pnpm run build && npm pack --dry-run",
|
|
238
|
-
"pack:smoke": "pnpm run build && node scripts/package-smoke.mjs && node scripts/packed-openai-agents-e2e.mjs && node scripts/packed-quickstart-e2e.mjs && node scripts/packed-semantic-loop-e2e.mjs && node scripts/packed-swarm-loop-e2e.mjs && node scripts/evidence-ci-golden-paths.mjs",
|
|
238
|
+
"pack:smoke": "pnpm run build && node scripts/package-smoke.mjs && node scripts/packed-openai-agents-e2e.mjs && node scripts/packed-ai-sdk-e2e.mjs && node scripts/packed-quickstart-e2e.mjs && node scripts/packed-semantic-loop-e2e.mjs && node scripts/packed-swarm-loop-e2e.mjs && node scripts/evidence-ci-golden-paths.mjs",
|
|
239
239
|
"linked-versions:check": "node scripts/check-linked-versions.mjs",
|
|
240
240
|
"docs:commands": "node scripts/validate-doc-commands.mjs",
|
|
241
241
|
"docs:links": "node scripts/validate-doc-links.mjs",
|
|
@@ -11420,5 +11420,5 @@ function renderGateReport(result, options = {}) {
|
|
|
11420
11420
|
}
|
|
11421
11421
|
|
|
11422
11422
|
export { COHORT_METRIC_IDS, DEFAULT_SUITE_ARTIFACTS_DIR, EVIDENCE_FORMAT_VERSION, EVIDENCE_HTML_FILENAME, EVIDENCE_MANIFEST_FILENAME, Redactor, TraceDirectory, TraceReadError, TreeBuilder, aggregateBundleSafeStatus, aggregateSessionCheckResults, analyzeCohort, applyProfileMetadataCaps, assertBundlePathContained, assertEvidenceRelativePath, buildActivitySummary, buildBundleMetadata, buildBundleSummaryMarkdown, buildEvidenceCausalFailureViewHtml, buildEvidenceCiPackage, buildEvidenceCircuitViewHtml, buildEvidenceContractsViewHtml, buildEvidenceDiffViewHtml, buildEvidenceHtmlShell, buildEvidenceManifest, buildEvidenceOutcomesViewHtml, buildEvidenceProvenanceViewHtml, buildEvidenceSafetyViewHtml, buildEvidenceTimelineViewHtml, buildEvidenceToolsLlmViewHtml, buildEvidenceTreeViewHtml, buildLocalExplanation, buildPlaceholderArtifact, buildRunSummary, buildRunTimeline, buildRunWhatSummary, buildSessionIndex, buildTraceStats, buildZipArchive, bundleFailsOnSafety, bundleRunAssetRelativePath, collectTraceSchemaVersions, compactAttributes, createBaselineRegressionRule, createLlmUsageRule, createMaxStepDurationRule, createObservedOutcomeRule, createRequireCompletedRule, createRunDepthRule, createRunDurationRule, createRunStatusRule, createSafetyOversizedAttributeRule, createSafetyRawContentRule, createSafetyRedactionRule, createSafetySecretPatternRule, createStallDetectionRule, createStructureCycleRule, createStructureOrphanRule, createStructureParallelWidthRule, createStructureRelationshipRule, createToolUsageRule, defaultBundleOutputPath, defaultSuiteConfigTemplate, diffRuns, diffTraceEvents, enrichSessionRunRecord, escapeHtml, escapeMarkdown, extractMetadata, extractOutcomesFromTraceEvents, filterMetasBySessionScope, filterTraces, flattenTree, formatDuration2 as formatDuration, formatStepLabel, formatTimestamp, gateHasThresholds, getIndent, getTraceFilePath, inferEvidenceFileRole, isAgentInspectTrace, isPersistedInspectEvent, loadSessionRunRecords, loadSuiteConfig, loadTraceMetadataList, manualTraceEventsToComparableRun, nanoid, normalizeBundleOutputPath, openTrace, parseCohortMetricList, parseDuration, parseDurationFilter, parseGateList, parseTraceJsonl, persistedInspectEventsToTraceEvents, renderActivitySummaryHuman, renderCohortReport, renderErrorLine, renderGateReport, renderObservedOutcomesHtml, renderObservedOutcomesMarkdown, renderRunDiff, renderRunWhat, renderStepLine, renderSuiteReport, renderTimeline, renderTraceStats, resolveBundleRunIds, resolveRedactionProfile, resolveSuiteTemplate, resolveTraceDir, runGate, runSuite, runTraceChecks, safeString, sanitizeBundleRunId, searchTraces, serializeEvidenceManifest, sha256Hex, stableJson, summarizeObservedOutcomes, summarizeSemanticParity, traceEventToPersistedInspectEvent, truncateName, truncateStringForProfile, validateEvent, validateSuiteConfig, verifyEvidenceDirectory, zeroKinds };
|
|
11423
|
-
//# sourceMappingURL=chunk-
|
|
11424
|
-
//# sourceMappingURL=chunk-
|
|
11423
|
+
//# sourceMappingURL=chunk-S4TLTLXN.mjs.map
|
|
11424
|
+
//# sourceMappingURL=chunk-S4TLTLXN.mjs.map
|