agent-inspect 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +46 -4
- package/README.md +27 -9
- package/SECURITY.md +1 -1
- package/docs/ADAPTERS.md +22 -1
- package/docs/API.md +45 -11
- package/docs/ARCHITECTURE.md +1 -0
- package/docs/CLI.md +56 -1
- package/docs/DIFF.md +127 -0
- package/docs/EXPORTS.md +14 -0
- package/docs/GETTING-STARTED.md +7 -2
- package/docs/KNOWN-ISSUES.md +53 -0
- package/docs/LIMITATIONS.md +17 -0
- package/docs/SCHEMA.md +41 -2
- package/package.json +2 -1
- package/packages/cli/dist/index.cjs +890 -109
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +884 -104
- package/packages/cli/dist/index.mjs.map +1 -1
- package/packages/core/dist/index.cjs +1627 -136
- package/packages/core/dist/index.cjs.map +1 -1
- package/packages/core/dist/index.d.cts +190 -3
- package/packages/core/dist/index.d.ts +190 -3
- package/packages/core/dist/index.mjs +1611 -133
- package/packages/core/dist/index.mjs.map +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,12 +1,54 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## 1.
|
|
3
|
+
## 1.3.0
|
|
4
4
|
|
|
5
5
|
### Minor Changes
|
|
6
6
|
|
|
7
|
-
-
|
|
7
|
+
- 503d240: Correlation metadata, redaction profiles for share-safe exports, and LangChain streaming metadata support.
|
|
8
|
+
|
|
9
|
+
## 1.3.0 — Unreleased
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- Added correlation metadata on `inspectRun` / `maybeInspectRun` (`correlationId`, `requestId`, `decisionId`, `groupId`) and `getCurrentCorrelationMetadata()`.
|
|
14
|
+
- Added redaction profiles (`local`, `share`, `strict`) for trace safety and share-safe exports.
|
|
15
|
+
- Added `redactionProfile` on `InspectRunOptions` and `ExportOptions`.
|
|
16
|
+
- Added `--redaction-profile` to `agent-inspect export`.
|
|
17
|
+
- Added LangChain streaming metadata support (`stream: true`) for token chunk counts and duration.
|
|
18
|
+
- Added bounded preview behavior for preview capture mode (`maxStreamPreviewChars`).
|
|
19
|
+
|
|
20
|
+
### Notes
|
|
21
|
+
|
|
22
|
+
- LangChain `capture: "metadata-only"` remains default; full stream text is not captured by default.
|
|
23
|
+
- LangChain streaming does not emit per-token JSONL events.
|
|
24
|
+
- Redaction profiles are key-based safeguards, not compliance-grade PII detection.
|
|
25
|
+
- Export redaction does not upload anywhere and does not mutate original traces.
|
|
26
|
+
- No vendor upload, hosted dashboard, or OTLP HTTP sink was added.
|
|
27
|
+
- Manual trace writing remains `schemaVersion: "0.1"`; v0.2 is not written by default.
|
|
28
|
+
|
|
29
|
+
## 1.2.0
|
|
30
|
+
|
|
31
|
+
Released **2026-06-11**. Changeset `5a7f785`.
|
|
32
|
+
|
|
33
|
+
### Added
|
|
34
|
+
|
|
35
|
+
- Added experimental `PersistedInspectEvent` model (`schemaVersion: "0.2"`) as a source-agnostic event foundation.
|
|
36
|
+
- Added validator for persisted events (`isPersistedInspectEvent`).
|
|
37
|
+
- Added converters from legacy `schemaVersion: "0.1"` manual trace events to persisted events.
|
|
38
|
+
- Added converters between `InspectEvent` and `PersistedInspectEvent`.
|
|
39
|
+
- Added in-memory helpers to build run trees from persisted events (`persistedInspectEventsToRunTrees`, `traceEventsToPersistedRunTrees`).
|
|
40
|
+
- Added canonical v0.2 fixture samples under `fixtures/traces-v0.2/`.
|
|
41
|
+
|
|
42
|
+
### Notes
|
|
43
|
+
|
|
44
|
+
- Existing manual trace writing remains `schemaVersion: "0.1"`.
|
|
45
|
+
- v0.2 is not written by default in this release.
|
|
46
|
+
- CLI read/write behavior is unchanged.
|
|
47
|
+
- No vendor upload, hosted dashboard, OTLP HTTP sink, replay engine, or cost engine was added.
|
|
48
|
+
|
|
49
|
+
## 1.1.0
|
|
8
50
|
|
|
9
|
-
|
|
51
|
+
Changeset `21ecc6f`: env-gated tracing, trace safety (redaction + size bounds), LangChain JSONL persistence, logging recipes, CJS/ESM type export compatibility, community docs.
|
|
10
52
|
|
|
11
53
|
### Added
|
|
12
54
|
|
|
@@ -34,7 +76,7 @@
|
|
|
34
76
|
### Documentation
|
|
35
77
|
|
|
36
78
|
- Added/updated logging playbook for structured JSON logs ([docs/LOGGING-PLAYBOOK.md](docs/LOGGING-PLAYBOOK.md)).
|
|
37
|
-
- Updated public roadmap
|
|
79
|
+
- Updated public roadmap after the 1.1.0 release (Released recently / Now / Next / Future).
|
|
38
80
|
- Updated contributor/community docs for package boundaries and optional packages.
|
|
39
81
|
- Added clearer community onboarding and issue-draft guidance.
|
|
40
82
|
|
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
agent-inspect helps you understand what happened inside an AI agent run — **locally**. It turns manual steps, tool calls, LLM calls, structured logs, failures, durations, and run metadata into **readable execution trees** you can inspect from the terminal.
|
|
6
6
|
|
|
7
|
-
It is built for TypeScript/Node.js developers and teams shipping real agentic products — not just toy demos. Use it **
|
|
7
|
+
It is built for TypeScript/Node.js developers and teams shipping real agentic products — not just toy demos. Use it for **local TypeScript agent debugging**, **eval iteration**, and **CI trace artifacts**. It **complements** production observability platforms; it does **not** replace them.
|
|
8
8
|
|
|
9
9
|
The tool starts with **manual traces** and **existing structured logs**, and extends into **optional framework callbacks** and **standards-aligned local export** — without turning the core into a SaaS or a vendor pipeline.
|
|
10
10
|
|
|
@@ -20,6 +20,8 @@ agent-inspect gives those runs **structure**: an **execution tree** you can read
|
|
|
20
20
|
|
|
21
21
|
## Install
|
|
22
22
|
|
|
23
|
+
Current npm release: **1.2.0**.
|
|
24
|
+
|
|
23
25
|
```bash
|
|
24
26
|
npm install agent-inspect
|
|
25
27
|
```
|
|
@@ -34,6 +36,8 @@ Verify the CLI is available:
|
|
|
34
36
|
npx agent-inspect --help
|
|
35
37
|
```
|
|
36
38
|
|
|
39
|
+
For a clean npm/pnpm install checklist with ESM, CJS, and CLI checks, see [Clean install smoke test](docs/INSTALL-SMOKE-TEST.md).
|
|
40
|
+
|
|
37
41
|
## 60-second quickstart
|
|
38
42
|
|
|
39
43
|
Create `demo.mjs`:
|
|
@@ -153,14 +157,14 @@ Full flags and behavior: [docs/CLI.md](docs/CLI.md).
|
|
|
153
157
|
|
|
154
158
|
- Debug a **failed tool call** or thrown error in a support or ops agent.
|
|
155
159
|
- See **which step dominated latency** in a multi-step planner or RAG pipeline.
|
|
156
|
-
- **Diff two runs** after a prompt, model, or routing change.
|
|
160
|
+
- **Diff two runs** after a prompt, model, or routing change (see [diff examples](docs/DIFF.md)).
|
|
157
161
|
- Point **`logs`** / **`tail`** at existing job or service logs to get a **local execution view** without shipping data upstream.
|
|
158
|
-
- **Export** a run to Markdown for a PR, postmortem, or internal thread — then review before sharing.
|
|
162
|
+
- **Export** a run to Markdown for a PR, postmortem, or internal thread — use `--redaction-profile share` for share-safe copies, then review before sharing.
|
|
159
163
|
- Keep traces **on disk** while still using enterprise observability elsewhere.
|
|
160
164
|
|
|
161
|
-
##
|
|
165
|
+
## Stable foundation (AgentInspect 1.x)
|
|
162
166
|
|
|
163
|
-
**agent-inspect 1.0** stabilizes the **local debugging foundation**:
|
|
167
|
+
**agent-inspect 1.x** (current: **1.2.0**) stabilizes the **local debugging foundation**:
|
|
164
168
|
|
|
165
169
|
- Instrument a run with `inspectRun` and `step`
|
|
166
170
|
- Write **local JSONL traces** (`schemaVersion: "0.1"` — compatibility retained)
|
|
@@ -173,7 +177,7 @@ Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInsp
|
|
|
173
177
|
|
|
174
178
|
**Stable CLI workflows:** `agent-inspect list`, `agent-inspect view`, `agent-inspect clean`.
|
|
175
179
|
|
|
176
|
-
**Also included in 1.
|
|
180
|
+
**Also included in 1.x** as local-first extensions:
|
|
177
181
|
|
|
178
182
|
- Structured log inspection: **`logs`**
|
|
179
183
|
- Live log tailing: **`tail`**
|
|
@@ -182,6 +186,7 @@ Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInsp
|
|
|
182
186
|
- Optional **`@agent-inspect/langchain`** callback adapter
|
|
183
187
|
- Optional **`@agent-inspect/tui`** terminal viewer
|
|
184
188
|
- **Fixtures** and **recipes** for deterministic checks and adoption patterns
|
|
189
|
+
- **v1.2.0** — experimental persisted-event foundation (`PersistedInspectEvent`, converters, in-memory tree bridge) for future source-agnostic local inspection. Manual trace writing remains **`schemaVersion: "0.1"`**; v0.2 is **not written by default**; CLI behavior unchanged; no vendor upload.
|
|
185
190
|
|
|
186
191
|
**Honest boundaries:** programmatic log parsing, export, and diff APIs; LangChain and TUI programmatic surfaces; and OpenInference/OTLP JSON exports are **experimental or compatibility-oriented**. Nothing performs **vendor upload** by default.
|
|
187
192
|
|
|
@@ -189,7 +194,7 @@ Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInsp
|
|
|
189
194
|
|
|
190
195
|
### LangChain callback adapter (`@agent-inspect/langchain`)
|
|
191
196
|
|
|
192
|
-
Optional package: official **LangChain.js callbacks** (`BaseCallbackHandler`), **metadata-oriented by default**, **no monkey-patching**, **no vendor sink**. The LangChain adapter
|
|
197
|
+
Optional package: official **LangChain.js callbacks** (`BaseCallbackHandler`), **metadata-oriented by default**, **no monkey-patching**, **no vendor sink**. Optional **`stream: true`** records chunk counts and stream duration **without storing full token text by default**. The LangChain adapter ships with 1.x; its programmatic API remains experimental and may evolve independently of the stable core tracing API.
|
|
193
198
|
|
|
194
199
|
```bash
|
|
195
200
|
pnpm add agent-inspect @agent-inspect/langchain @langchain/core
|
|
@@ -257,7 +262,7 @@ The TUI is available as a separate optional package; its programmatic API is exp
|
|
|
257
262
|
- **Manual metadata** is user-controlled. By default, common sensitive keys are **redacted before disk**; pass `redact: false` to opt out. Long metadata is truncated and events are capped at 64 KiB per JSONL line. Review traces and exports before sharing.
|
|
258
263
|
- **Review exports** before sharing (especially with richer attribute flags).
|
|
259
264
|
|
|
260
|
-
See [SECURITY.md](SECURITY.md).
|
|
265
|
+
See [SECURITY.md](SECURITY.md) and the [safe trace sharing checklist](docs/SAFE-TRACE-SHARING.md).
|
|
261
266
|
|
|
262
267
|
## agent-inspect comparison
|
|
263
268
|
|
|
@@ -268,6 +273,7 @@ For a detailed comparison, see [Compare with other tools](docs/COMPARE.md).
|
|
|
268
273
|
## Documentation
|
|
269
274
|
|
|
270
275
|
- [Getting started](docs/GETTING-STARTED.md)
|
|
276
|
+
- [Clean install smoke test](docs/INSTALL-SMOKE-TEST.md)
|
|
271
277
|
- [API stability & experimental surfaces](docs/API.md)
|
|
272
278
|
- [CLI reference](docs/CLI.md)
|
|
273
279
|
- [Schema (`schemaVersion: "0.1"`)](docs/SCHEMA.md)
|
|
@@ -280,11 +286,23 @@ For a detailed comparison, see [Compare with other tools](docs/COMPARE.md).
|
|
|
280
286
|
- [Adapters](docs/ADAPTERS.md)
|
|
281
287
|
- [Compare with other tools](docs/COMPARE.md)
|
|
282
288
|
- [Security](SECURITY.md)
|
|
283
|
-
- [
|
|
289
|
+
- [Safe trace sharing checklist](docs/SAFE-TRACE-SHARING.md)
|
|
290
|
+
- [Changelog](CHANGELOG.md) — **upcoming [1.3.0](CHANGELOG.md#130--unreleased)** on `main` (not yet on npm): correlation metadata, share-safe export profiles, LangChain streaming metadata
|
|
284
291
|
- [Known issues](docs/KNOWN-ISSUES.md)
|
|
285
292
|
- [Limitations](docs/LIMITATIONS.md)
|
|
286
293
|
- [Screenshot checklist (planned assets)](docs/SCREENSHOTS.md)
|
|
287
294
|
|
|
295
|
+
## Contributing
|
|
296
|
+
|
|
297
|
+
AgentInspect welcomes docs, fixtures, examples, and carefully scoped CLI improvements.
|
|
298
|
+
|
|
299
|
+
- **Good first issues:** [GOOD-FIRST-ISSUES.md](GOOD-FIRST-ISSUES.md) — live batches [#7–#14](https://github.com/rajudandigam/agent-inspect/issues?q=is%3Aissue+is%3Aopen) and [#18–#30](https://github.com/rajudandigam/agent-inspect/issues/18) (comment on an issue before opening a PR)
|
|
300
|
+
- **Discussions:** [github.com/rajudandigam/agent-inspect/discussions](https://github.com/rajudandigam/agent-inspect/discussions) — feedback, stack survey, integration ideas
|
|
301
|
+
- **Roadmap:** [ROADMAP.md](ROADMAP.md) — Now / Next / Future direction (non-committal)
|
|
302
|
+
- **Contributing guide:** [CONTRIBUTING.md](CONTRIBUTING.md) — validation commands, PR expectations, scope boundaries
|
|
303
|
+
|
|
304
|
+
**Security:** Traces and logs may contain secrets. **Redact before sharing** in issues, Discussions, PRs, or exports. See [SECURITY.md](SECURITY.md) and the [safe trace sharing checklist](docs/SAFE-TRACE-SHARING.md).
|
|
305
|
+
|
|
288
306
|
## Development
|
|
289
307
|
|
|
290
308
|
From a clone of this repo:
|
package/SECURITY.md
CHANGED
|
@@ -70,7 +70,7 @@ Important limitations:
|
|
|
70
70
|
- Always **review exported files** before sharing them externally.
|
|
71
71
|
- Avoid committing trace directories (`.agent-inspect-runs/`) to source control.
|
|
72
72
|
|
|
73
|
-
For schema-level redaction notes, see `docs/SCHEMA.md` (redaction section) and `docs/API.md`.
|
|
73
|
+
For a practical pre-share workflow, see `docs/SAFE-TRACE-SHARING.md`. For schema-level redaction notes, see `docs/SCHEMA.md` (redaction section) and `docs/API.md`.
|
|
74
74
|
|
|
75
75
|
## Dependency and security review process
|
|
76
76
|
|
package/docs/ADAPTERS.md
CHANGED
|
@@ -8,6 +8,7 @@ AgentInspect is framework-agnostic at its core, but can optionally integrate wit
|
|
|
8
8
|
- **In-memory by default** (`getEvents()` / `clear()`)
|
|
9
9
|
- **Optional JSONL persistence** (`persist: true`) maps callback lifecycle to schemaVersion `"0.1"` manual events (`run_started`, `step_started`, `step_completed`, `run_completed`) so `list` / `view` / `export` / `diff` work without a separate schema
|
|
10
10
|
- **Metadata-only capture by default**; `capture: "preview"` is opt-in with truncation
|
|
11
|
+
- **Optional streaming metadata** (`stream: true`) — chunk counts, timing, and bounded previews only; **no full token stream** captured by default
|
|
11
12
|
- **No monkey-patching**, **no vendor sinks**, **no network upload**
|
|
12
13
|
- **Interactive TUI** (optional): `@agent-inspect/tui`
|
|
13
14
|
- Documented as **experimental** in `docs/API.md`
|
|
@@ -36,7 +37,27 @@ npx agent-inspect list --dir ./.agent-inspect
|
|
|
36
37
|
npx agent-inspect view <run-id> --dir ./.agent-inspect
|
|
37
38
|
```
|
|
38
39
|
|
|
40
|
+
### LangChain streaming metadata (v1.3.0+)
|
|
41
|
+
|
|
42
|
+
When `stream: true` on `AgentInspectCallback`:
|
|
43
|
+
|
|
44
|
+
- `handleLLMNewToken` updates **in-memory streaming stats** only (no per-token JSONL lines).
|
|
45
|
+
- On LLM end/error, metadata such as `chunkCount`, `firstChunkAt`, `lastChunkAt`, `streamDurationMs`, and `streamedCharCount` attach to the LLM completion event.
|
|
46
|
+
- **`capture: "metadata-only"`** (default) does **not** store token text.
|
|
47
|
+
- **`capture: "preview"`** may include a **bounded** `streamPreview` via `maxStreamPreviewChars` (defaults to `maxPreviewChars`).
|
|
48
|
+
- With **`persist: true`**, streaming metadata is written on the LLM `step_started` metadata at completion time (deferred write for streaming LLM steps).
|
|
49
|
+
- Inspect persisted runs locally: `agent-inspect list`, `view`, `export --redaction-profile share`.
|
|
50
|
+
|
|
51
|
+
```ts
|
|
52
|
+
const callback = new AgentInspectCallback({
|
|
53
|
+
stream: true,
|
|
54
|
+
capture: "metadata-only",
|
|
55
|
+
persist: true,
|
|
56
|
+
});
|
|
57
|
+
```
|
|
58
|
+
|
|
39
59
|
See also:
|
|
40
60
|
- `docs/MIGRATION.md` (what changed from early versions)
|
|
41
|
-
- `docs/API.md` (LangChain adapter options: `persist`, `capture`, `traceDir`)
|
|
61
|
+
- `docs/API.md` (LangChain adapter options: `persist`, `capture`, `stream`, `traceDir`)
|
|
62
|
+
- `docs/SAFE-TRACE-SHARING.md` (review exports before sharing)
|
|
42
63
|
|
package/docs/API.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# API (AgentInspect 1.
|
|
1
|
+
# API (AgentInspect 1.x)
|
|
2
2
|
|
|
3
3
|
This document describes the **public TypeScript API surface** of AgentInspect and classifies each area as **stable** or **experimental**.
|
|
4
4
|
|
|
@@ -11,9 +11,9 @@ AgentInspect is a **local-first execution-tree debugger**. It is not a SaaS, not
|
|
|
11
11
|
|
|
12
12
|
Notes:
|
|
13
13
|
|
|
14
|
-
- The core guarantee of v1.
|
|
14
|
+
- The core guarantee of v1.x is **stable local debugging**: manual tracing + CLI inspection.
|
|
15
15
|
- Export formats (OpenInference / OTLP JSON) are **local-only** and **compatibility-oriented**. They do **not** upload anywhere.
|
|
16
|
-
- There are **zero production sinks** in v1.
|
|
16
|
+
- There are **zero production sinks** in v1.x; sink/uploader APIs are not stable.
|
|
17
17
|
|
|
18
18
|
## 2. Stable core APIs (manual tracing)
|
|
19
19
|
|
|
@@ -26,16 +26,21 @@ import { inspectRun, maybeInspectRun, step, observe } from "agent-inspect";
|
|
|
26
26
|
```
|
|
27
27
|
|
|
28
28
|
- **`inspectRun(name, fn, options?)`**: wraps a workflow in a local JSONL trace (`run_started` / `run_completed`), prints terminal progress, and swallows instrumentation failures (user errors are re-thrown). **Traces by default** when `enabled` is omitted or `true`. Pass **`enabled: false`** to run `fn` with no trace file, no execution context, and no terminal output.
|
|
29
|
-
- **`redact`**: default `true` — redacts sensitive metadata keys before disk (`authorization`, `cookie`, `token`, `apiKey`, `password`, `secret`, `email`). Pass `false` to persist metadata as-is. Pass `{ rules?: RedactionRule[] }` for custom rules (defaults still apply).
|
|
29
|
+
- **`redact`**: default `true` — redacts sensitive metadata keys before disk (`authorization`, `cookie`, `token`, `apiKey`, `password`, `secret`, `email`). Pass `false` to persist metadata as-is. Pass `{ rules?: RedactionRule[] }` for custom rules (defaults still apply). **`redact: false` wins** over `redactionProfile` for trace writing.
|
|
30
|
+
- **`redactionProfile`**: optional preset — `local` (default), `share`, or `strict`. Adds extra key-based redaction and tighter metadata string bounds for trace writing. Key-based only — not compliance-grade DLP.
|
|
30
31
|
- **`maxMetadataValueLength`**: max string length for metadata values (default `2000`).
|
|
31
32
|
- **`maxPreviewLength`**: max string length for preview-like keys containing `preview` (default `500`).
|
|
32
33
|
- **`maxEventBytes`**: max UTF-8 bytes per serialized JSONL event (default `65536`). Oversized events are truncated; instrumentation never throws into user code.
|
|
34
|
+
- **Correlation metadata (v1.3.0+):** optional `correlationId`, `requestId`, `decisionId`, and `groupId` strings. When set, they are written on `run_started.metadata` (not on every step). Top-level correlation options override the same keys in `options.metadata`. Useful for eval cases, CI job IDs, request tracing, and future stats/cohort views. They are **metadata only** — they do not replace `runId`. Treat sensitive IDs as trace data before sharing exports.
|
|
33
35
|
- **`maybeInspectRun(name, fn, options?)`**: same as `inspectRun` when tracing is enabled; otherwise passthrough. Enablement: explicit **`options.enabled`** wins; when omitted, reads **`AGENT_INSPECT`** (`1`, `true`, `yes`, `on`, `enabled` — case-insensitive). Unset or other values disable tracing. Use in eval harnesses, CI, or jobs where tracing should be toggled by environment.
|
|
34
36
|
- **`isAgentInspectEnabled(value?)`**: returns whether a string (or `process.env.AGENT_INSPECT`) matches an enable token.
|
|
35
37
|
- **`step(name, fn, options?)`**: traces a named unit of work inside `inspectRun` (`step_started` / `step_completed`). Step `metadata` inherits the parent run's redaction and size-bound settings.
|
|
36
38
|
- **`step.llm(model, fn)`**: convenience wrapper (`type: "llm"`, `metadata.model`).
|
|
37
39
|
- **`step.tool(toolName, fn)`**: convenience wrapper (`type: "tool"`, `metadata.toolName`).
|
|
38
40
|
- **`observe(agent, options?)`**: proxy wrapper that traces top-level `run` / `execute` / `invoke` methods via `inspectRun`.
|
|
41
|
+
- **`getCurrentCorrelationMetadata()`**: returns active run correlation fields (`correlationId`, `requestId`, `decisionId`, `groupId`) inside `inspectRun` / `maybeInspectRun`; `undefined` outside a traced run or when none were set.
|
|
42
|
+
- **`RedactionProfile`**: `"local" | "share" | "strict"` — see `redactionProfile` on `InspectRunOptions` and `ExportOptions`.
|
|
43
|
+
- **`resolveRedactionProfile(profile?)`**: resolves profile extra keys and metadata caps (for advanced integrations).
|
|
39
44
|
|
|
40
45
|
## 3. Stable local inspection APIs
|
|
41
46
|
|
|
@@ -95,7 +100,8 @@ The CLI `tail` workflow is supported. The programmatic accumulator is experiment
|
|
|
95
100
|
|
|
96
101
|
Exports are **read-only**, **local-only**, and **compatibility-oriented**. They do not upload, stream, or integrate vendor SDKs.
|
|
97
102
|
|
|
98
|
-
- **`exportRunTree`**
|
|
103
|
+
- **`exportRunTree`**, **`redactRunTreeForExport`**
|
|
104
|
+
- **`ExportOptions.redactionProfile`**: `local` (default), `share`, or `strict` — applies key-based redaction to an exported copy without mutating the source tree.
|
|
99
105
|
- **`exportMarkdown`**, **`exportHtml`**
|
|
100
106
|
- **`exportOpenInference`** (OpenInference-compatible JSON)
|
|
101
107
|
- **`exportOtlpJson`** (OTLP JSON, experimental until verified per backend)
|
|
@@ -119,12 +125,14 @@ Diff is local and read-only. Programmatic diff surfaces are experimental until t
|
|
|
119
125
|
- **`runName`**: default `"langchain-agent"` for standalone persisted runs
|
|
120
126
|
- **`traceDir`**: defaults via `resolveTraceDir` / `AGENT_INSPECT_TRACE_DIR`
|
|
121
127
|
- **`capture`**: `"none"` | `"metadata-only"` (default) | `"preview"` (truncated previews, opt-in)
|
|
128
|
+
- **`stream`**: default `false` — when `true`, records streaming lifecycle metadata (`chunkCount`, `streamDurationMs`, etc.) on LLM end/error; does **not** capture full token text by default
|
|
129
|
+
- **`maxStreamPreviewChars`**: bounds `streamPreview` when `capture: "preview"` and `stream: true` (defaults to `maxPreviewChars`)
|
|
122
130
|
- **`redact`**: custom `RedactionRule[]` applied before disk (core defaults still apply via shared redactor)
|
|
123
131
|
- **`runId`**: optional id for standalone persisted runs
|
|
124
132
|
- In-memory **`getEvents()`** / **`clear()`** unchanged when `persist` is false
|
|
125
133
|
- Metadata helpers: `extractModelName`, `extractTokenUsage`, `safePreview`, `toPlainMetadata`
|
|
126
134
|
|
|
127
|
-
Rationale: v1.
|
|
135
|
+
Rationale: v1.x includes one official adapter and **zero production sinks**, so adapter surfaces remain experimental.
|
|
128
136
|
|
|
129
137
|
## 10. Experimental `@agent-inspect/tui` APIs
|
|
130
138
|
|
|
@@ -132,23 +140,49 @@ Rationale: v1.0 includes one official adapter and **zero production sinks**, so
|
|
|
132
140
|
|
|
133
141
|
- `runTraceViewer`, `loadTraceForTui`, `buildTuiTraceModel`, etc.
|
|
134
142
|
|
|
135
|
-
## 11.
|
|
143
|
+
## 11. Experimental persisted-event foundation (v1.2.0)
|
|
136
144
|
|
|
137
|
-
|
|
145
|
+
These helpers expose the **source-agnostic `PersistedInspectEvent` model** (`schemaVersion: "0.2"`). They are **local-only**, **in-memory**, and **do not change** storage write/read or CLI behavior in v1.2.0.
|
|
138
146
|
|
|
139
|
-
|
|
147
|
+
Import from `agent-inspect`:
|
|
148
|
+
|
|
149
|
+
| API | Role |
|
|
150
|
+
| --- | ---- |
|
|
151
|
+
| `isPersistedInspectEvent` | Runtime validator for v0.2 persisted events |
|
|
152
|
+
| `traceEventToPersistedInspectEvent` | Convert one v0.1 `TraceEvent` |
|
|
153
|
+
| `traceEventsToPersistedInspectEvents` | Batch v0.1 → v0.2 |
|
|
154
|
+
| `inspectEventToPersistedInspectEvent` | Convert one in-memory `InspectEvent` |
|
|
155
|
+
| `inspectEventsToPersistedInspectEvents` | Batch `InspectEvent` → v0.2 |
|
|
156
|
+
| `persistedInspectEventToInspectEvent` | Convert one v0.2 event to `InspectEvent` |
|
|
157
|
+
| `persistedInspectEventsToInspectEvents` | Batch v0.2 → `InspectEvent` |
|
|
158
|
+
| `persistedInspectEventsToRunTrees` | Build `InspectRunTree[]` from v0.2 events (via `TreeBuilder`) |
|
|
159
|
+
| `traceEventsToPersistedRunTrees` | v0.1 `TraceEvent[]` → persisted model → trees |
|
|
160
|
+
|
|
161
|
+
Related types: `PersistedInspectEvent`, `PersistedEventSourceType`, `PersistedEventStatus`, `TraceEventToPersistedOptions`, `InspectEventToPersistedOptions`, `PersistedToInspectEventOptions`, `PersistedTreeBridgeOptions`.
|
|
162
|
+
|
|
163
|
+
**Notes:**
|
|
164
|
+
|
|
165
|
+
- Manual trace **writing** remains `schemaVersion: "0.1"`.
|
|
166
|
+
- v0.2 is **not written by default**; use converters and `fixtures/traces-v0.2/` samples for validation.
|
|
167
|
+
- Storage dual-read and CLI integration are future work.
|
|
168
|
+
|
|
169
|
+
## 12. Deprecated APIs
|
|
170
|
+
|
|
171
|
+
No deprecated APIs are declared as of 1.2.0.
|
|
172
|
+
|
|
173
|
+
## 13. Removal / deprecation policy
|
|
140
174
|
|
|
141
175
|
- Stable APIs are not removed in v1.x.
|
|
142
176
|
- If removal is necessary, the API should be **deprecated** first, documented, and kept for a reasonable window (target: at least one minor line) unless security requires faster action.
|
|
143
177
|
|
|
144
|
-
##
|
|
178
|
+
## 14. Backward compatibility policy
|
|
145
179
|
|
|
146
180
|
- Manual trace JSONL (`schemaVersion: "0.1"`) remains readable.
|
|
147
181
|
- Additive schema changes are allowed in minor versions.
|
|
148
182
|
- Breaking changes require a major version.
|
|
149
183
|
- Unknown fields should be ignored where safe.
|
|
150
184
|
|
|
151
|
-
##
|
|
185
|
+
## 15. Examples
|
|
152
186
|
|
|
153
187
|
### Minimal manual trace
|
|
154
188
|
|
package/docs/ARCHITECTURE.md
CHANGED
|
@@ -19,6 +19,7 @@ Root `agent-inspect` uses **conditional exports** for ESM/CJS TypeScript consume
|
|
|
19
19
|
- Manual traces use **`schemaVersion: "0.1"`** JSONL events (`run_started`, `step_started`, `step_completed`, `run_completed`).
|
|
20
20
|
- Failures use `step_completed` with `status: "error"` — there is no `step_failed` event.
|
|
21
21
|
- Log-derived runs use confidence labels (`explicit`, `correlated`, `heuristic`, `unknown`) and conservative tree-building rules.
|
|
22
|
+
- **v1.2.0 persisted-event foundation (released):** source-agnostic `PersistedInspectEvent` helpers (`schemaVersion: "0.2"`) — types, validators, converters, and in-memory tree bridge. Existing **`TreeBuilder`** remains the canonical tree builder. v0.2 bridge works **in memory only**; storage dual-read and CLI integration are v1.3+ / post-foundation work. See [proposals/UNIFIED-PERSISTED-INSPECT-EVENT.md](./proposals/UNIFIED-PERSISTED-INSPECT-EVENT.md) and [implementation/V1.3.0-RELEASE-TRAIN.md](./implementation/V1.3.0-RELEASE-TRAIN.md).
|
|
22
23
|
|
|
23
24
|
See [SCHEMA.md](./SCHEMA.md) for field reference and [API.md](./API.md) for public surfaces (stable vs experimental).
|
|
24
25
|
|
package/docs/CLI.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# CLI (AgentInspect 1.
|
|
1
|
+
# CLI (AgentInspect 1.x)
|
|
2
2
|
|
|
3
3
|
This document describes the **stable CLI surface** of AgentInspect.
|
|
4
4
|
|
|
@@ -178,6 +178,14 @@ Options:
|
|
|
178
178
|
- `--include-attributes`: include bounded attributes (review before sharing)
|
|
179
179
|
- `--no-metadata`: omit summary/metadata sections
|
|
180
180
|
- `--no-errors`: omit error sections
|
|
181
|
+
- `--redaction-profile <profile>`: redaction profile for exported copies — `local` (default), `share`, or `strict`. Key-based safety only; review exports before sharing.
|
|
182
|
+
|
|
183
|
+
Examples:
|
|
184
|
+
|
|
185
|
+
```bash
|
|
186
|
+
npx agent-inspect export <run-id> --format markdown --redaction-profile share
|
|
187
|
+
npx agent-inspect export <run-id> --format html --redaction-profile strict
|
|
188
|
+
```
|
|
181
189
|
|
|
182
190
|
### 6.7 `diff`
|
|
183
191
|
|
|
@@ -196,6 +204,53 @@ Options:
|
|
|
196
204
|
- `--focus <all|errors|structure|outputs>`
|
|
197
205
|
- `--check <all|structure|outputs|errors|timing>`
|
|
198
206
|
|
|
207
|
+
Fixture examples:
|
|
208
|
+
|
|
209
|
+
```bash
|
|
210
|
+
agent-inspect diff minimal-success minimal-error --dir fixtures/traces
|
|
211
|
+
agent-inspect diff minimal-success long-running --dir fixtures/traces --check timing --duration-threshold 1ms
|
|
212
|
+
agent-inspect diff minimal-success nested-3-levels --dir fixtures/traces --check structure --ignore-duration
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
**Simplified example output** (actual CLI formatting may differ slightly):
|
|
216
|
+
|
|
217
|
+
```text
|
|
218
|
+
Run diff
|
|
219
|
+
Left: minimal-success
|
|
220
|
+
Right: minimal-error
|
|
221
|
+
|
|
222
|
+
Summary:
|
|
223
|
+
Differences: 4
|
|
224
|
+
Errors: 0
|
|
225
|
+
Warnings: 3
|
|
226
|
+
Info: 1
|
|
227
|
+
|
|
228
|
+
First divergence:
|
|
229
|
+
run-status at (run)
|
|
230
|
+
left: success
|
|
231
|
+
right: error
|
|
232
|
+
|
|
233
|
+
Differences:
|
|
234
|
+
[warning] run-status
|
|
235
|
+
Run completion status differs
|
|
236
|
+
left: success
|
|
237
|
+
right: error
|
|
238
|
+
[info] duration
|
|
239
|
+
Run duration differs
|
|
240
|
+
left: 120
|
|
241
|
+
right: 70
|
|
242
|
+
[warning] step-removed plan
|
|
243
|
+
Step only in left run: plan
|
|
244
|
+
left: step_root
|
|
245
|
+
right: (undefined)
|
|
246
|
+
[warning] step-added failing-step
|
|
247
|
+
Step only in right run: failing-step
|
|
248
|
+
left: (undefined)
|
|
249
|
+
right: step_fail
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
More examples, including timing-only and structure-only diffs, are in `docs/DIFF.md`.
|
|
253
|
+
|
|
199
254
|
## 7. Optional TUI behavior
|
|
200
255
|
|
|
201
256
|
`view --tui` delegates to `@agent-inspect/tui` and requires an interactive terminal. If the package is not installed, the CLI prints a short install hint.
|
package/docs/DIFF.md
CHANGED
|
@@ -8,4 +8,131 @@ AgentInspect can compare two runs (manual traces) locally and render a diff summ
|
|
|
8
8
|
Notes:
|
|
9
9
|
- Programmatic diff APIs are documented as **experimental** in `docs/API.md`.
|
|
10
10
|
- `diff` is read-only and does **not** rerun agents; “differences” do not necessarily imply failure (command errors do).
|
|
11
|
+
- `diff` reads local trace files only. It is useful for local debugging, but it is not production APM or hosted observability.
|
|
11
12
|
|
|
13
|
+
## CLI examples
|
|
14
|
+
|
|
15
|
+
The examples below use the checked-in fixture traces, so they are safe to copy and run locally:
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
agent-inspect diff minimal-success minimal-error --dir fixtures/traces
|
|
19
|
+
agent-inspect diff minimal-success long-running --dir fixtures/traces --check timing --duration-threshold 1ms
|
|
20
|
+
agent-inspect diff minimal-success nested-3-levels --dir fixtures/traces --check structure --ignore-duration
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
If you are running from a source checkout before installing the package globally, use the built CLI path instead:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
node packages/cli/dist/index.cjs diff minimal-success minimal-error --dir fixtures/traces
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
### Success run vs error run
|
|
30
|
+
|
|
31
|
+
This compares a successful run with a run that ended in an error. It shows the run status change, a duration difference, and the step-level structure difference (`plan` removed, `failing-step` added).
|
|
32
|
+
|
|
33
|
+
**Simplified example output** (actual CLI formatting may differ slightly):
|
|
34
|
+
|
|
35
|
+
```text
|
|
36
|
+
Run diff
|
|
37
|
+
Left: minimal-success
|
|
38
|
+
Right: minimal-error
|
|
39
|
+
|
|
40
|
+
Summary:
|
|
41
|
+
Differences: 4
|
|
42
|
+
Errors: 0
|
|
43
|
+
Warnings: 3
|
|
44
|
+
Info: 1
|
|
45
|
+
|
|
46
|
+
First divergence:
|
|
47
|
+
run-status at (run)
|
|
48
|
+
left: success
|
|
49
|
+
right: error
|
|
50
|
+
|
|
51
|
+
Differences:
|
|
52
|
+
[warning] run-status
|
|
53
|
+
Run completion status differs
|
|
54
|
+
left: success
|
|
55
|
+
right: error
|
|
56
|
+
[info] duration
|
|
57
|
+
Run duration differs
|
|
58
|
+
left: 120
|
|
59
|
+
right: 70
|
|
60
|
+
[warning] step-removed plan
|
|
61
|
+
Step only in left run: plan
|
|
62
|
+
left: step_root
|
|
63
|
+
right: (undefined)
|
|
64
|
+
[warning] step-added failing-step
|
|
65
|
+
Step only in right run: failing-step
|
|
66
|
+
left: (undefined)
|
|
67
|
+
right: step_fail
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### Duration-only timing check
|
|
71
|
+
|
|
72
|
+
Use `--check timing` when you only want timing differences. `--duration-threshold` can hide tiny deltas.
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
agent-inspect diff minimal-success long-running --dir fixtures/traces --check timing --duration-threshold 1ms
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
**Simplified example output**:
|
|
79
|
+
|
|
80
|
+
```text
|
|
81
|
+
Run diff
|
|
82
|
+
Left: minimal-success
|
|
83
|
+
Right: long-running
|
|
84
|
+
|
|
85
|
+
Summary:
|
|
86
|
+
Differences: 1
|
|
87
|
+
Errors: 0
|
|
88
|
+
Warnings: 0
|
|
89
|
+
Info: 1
|
|
90
|
+
|
|
91
|
+
First divergence:
|
|
92
|
+
duration at (run)
|
|
93
|
+
left: 120
|
|
94
|
+
right: 45
|
|
95
|
+
|
|
96
|
+
Differences:
|
|
97
|
+
[info] duration
|
|
98
|
+
Run duration differs
|
|
99
|
+
left: 120
|
|
100
|
+
right: 45
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
### Step structure / rename-style changes
|
|
104
|
+
|
|
105
|
+
When a prompt, model, or routing change renames or restructures steps, the current diff output represents that as `step-removed` plus `step-added`. Use `--ignore-duration` if the structure is what matters.
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
agent-inspect diff minimal-success nested-3-levels --dir fixtures/traces --check structure --ignore-duration
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
**Simplified example output**:
|
|
112
|
+
|
|
113
|
+
```text
|
|
114
|
+
Run diff
|
|
115
|
+
Left: minimal-success
|
|
116
|
+
Right: nested-3-levels
|
|
117
|
+
|
|
118
|
+
Summary:
|
|
119
|
+
Differences: 2
|
|
120
|
+
Errors: 0
|
|
121
|
+
Warnings: 2
|
|
122
|
+
Info: 0
|
|
123
|
+
|
|
124
|
+
First divergence:
|
|
125
|
+
step-removed at plan
|
|
126
|
+
left: step_root
|
|
127
|
+
right: (undefined)
|
|
128
|
+
|
|
129
|
+
Differences:
|
|
130
|
+
[warning] step-removed plan
|
|
131
|
+
Step only in left run: plan
|
|
132
|
+
left: step_root
|
|
133
|
+
right: (undefined)
|
|
134
|
+
[warning] step-added outer
|
|
135
|
+
Step only in right run: outer
|
|
136
|
+
left: (undefined)
|
|
137
|
+
right: step_outer
|
|
138
|
+
```
|
package/docs/EXPORTS.md
CHANGED
|
@@ -6,6 +6,20 @@ AgentInspect supports local-only exports from run trees and traces. Exports are
|
|
|
6
6
|
- **Schema + compatibility guarantees**: `docs/SCHEMA.md`
|
|
7
7
|
- **Safety / redaction notes**: `docs/CLI.md`, `SECURITY.md`, and `docs/SCHEMA.md` (redaction section)
|
|
8
8
|
|
|
9
|
+
## Share-safe exports (v1.3.0+)
|
|
10
|
+
|
|
11
|
+
Use `--redaction-profile share` (PRs, issues, internal threads) or `strict` (external sharing) when generating an export copy:
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
agent-inspect export <run-id> --format markdown --redaction-profile share
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
- Profiles are **key-based** redaction presets — not compliance-grade PII detection.
|
|
18
|
+
- Export redaction operates on a **copy** of the run tree; original JSONL traces on disk are **not mutated**.
|
|
19
|
+
- **Review** every export before posting — especially when `--include-attributes` is set.
|
|
20
|
+
|
|
21
|
+
See [SAFE-TRACE-SHARING.md](./SAFE-TRACE-SHARING.md).
|
|
22
|
+
|
|
9
23
|
Notes:
|
|
10
24
|
- Export formats (Markdown/HTML/OpenInference/OTLP JSON) are documented as **experimental / compatibility-oriented** in `docs/API.md`.
|
|
11
25
|
- AgentInspect does **not** upload anywhere; exports write local strings/files only.
|
package/docs/GETTING-STARTED.md
CHANGED
|
@@ -14,6 +14,8 @@ The `agent-inspect` package includes the CLI binary via its `bin` field:
|
|
|
14
14
|
npx agent-inspect --help
|
|
15
15
|
```
|
|
16
16
|
|
|
17
|
+
For a clean install verification path covering npm, pnpm, ESM import, CJS require, and CLI help, see [Clean install smoke test](./INSTALL-SMOKE-TEST.md).
|
|
18
|
+
|
|
17
19
|
For local repo development (this monorepo), build and run the CLI from `packages/cli`:
|
|
18
20
|
|
|
19
21
|
```bash
|
|
@@ -33,7 +35,7 @@ await inspectRun("demo-agent", async () => {
|
|
|
33
35
|
});
|
|
34
36
|
```
|
|
35
37
|
|
|
36
|
-
This writes a local JSONL trace with stable
|
|
38
|
+
This writes a local JSONL trace with stable event names (`schemaVersion: "0.1"`):
|
|
37
39
|
|
|
38
40
|
- `run_started`, `run_completed`
|
|
39
41
|
- `step_started`, `step_completed`
|
|
@@ -58,6 +60,10 @@ AGENT_INSPECT=1 node eval-runner.mjs
|
|
|
58
60
|
|
|
59
61
|
Enable tokens: `1`, `true`, `yes`, `on`, `enabled` (case-insensitive). Explicit `enabled: true | false` in options overrides the env var.
|
|
60
62
|
|
|
63
|
+
### Install compatibility
|
|
64
|
+
|
|
65
|
+
If `import` or `require` fails after install, see [KNOWN-ISSUES.md — Common install/runtime compatibility checks](./KNOWN-ISSUES.md#common-installruntime-compatibility-checks).
|
|
66
|
+
|
|
61
67
|
To skip tracing in code without env vars: `inspectRun(name, fn, { enabled: false })`.
|
|
62
68
|
|
|
63
69
|
## 3. View runs
|
|
@@ -151,4 +157,3 @@ agent-inspect view <runId> --tui
|
|
|
151
157
|
- [docs/KNOWN-ISSUES.md](./KNOWN-ISSUES.md)
|
|
152
158
|
- [docs/LIMITATIONS.md](./LIMITATIONS.md)
|
|
153
159
|
- [ROADMAP.md](../ROADMAP.md)
|
|
154
|
-
|