agent-inspect 1.2.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +41 -4
- package/README.md +41 -38
- package/docs/ADAPTERS.md +122 -21
- package/docs/API.md +25 -7
- package/docs/ARCHITECTURE.md +1 -1
- package/docs/CLI.md +71 -2
- package/docs/EXPORTS.md +80 -8
- package/docs/GETTING-STARTED.md +29 -1
- package/docs/KNOWN-ISSUES.md +6 -0
- package/docs/LIMITATIONS.md +17 -0
- package/docs/SCHEMA.md +6 -4
- package/package.json +1 -1
- package/packages/cli/dist/index.cjs +1019 -14
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +1019 -14
- package/packages/cli/dist/index.mjs.map +1 -1
- package/packages/core/dist/index.cjs +890 -14
- package/packages/core/dist/index.cjs.map +1 -1
- package/packages/core/dist/index.d.cts +176 -3
- package/packages/core/dist/index.d.ts +176 -3
- package/packages/core/dist/index.mjs +881 -15
- package/packages/core/dist/index.mjs.map +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,12 +1,49 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## 1.
|
|
3
|
+
## 1.4.0
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- Added `docs/CI-ARTIFACTS.md` and `examples/recipes/github-actions-artifact/` for CI trace artifact workflows.
|
|
8
|
+
- Added `agent-inspect timeline <runId>` — chronological local run view (`--json`, `--focus slow`).
|
|
9
|
+
- Added `agent-inspect stats` — local aggregate stats (`--since`, `--correlation-id`, `--group-id`, `--json`).
|
|
10
|
+
- Added `agent-inspect search` — deterministic local trace search (`--status`, `--kind`, `--name`, `--tool`, `--duration`, `--json`).
|
|
11
|
+
- Added core helpers: `buildRunTimeline`, `buildTraceStats`, `searchTraces`.
|
|
12
|
+
|
|
13
|
+
### Notes
|
|
14
|
+
|
|
15
|
+
- CI artifact upload is configured in user CI (e.g. GitHub Actions `upload-artifact`) — AgentInspect does not upload.
|
|
16
|
+
- Search is exact/contains matching only — no semantic search or index database.
|
|
17
|
+
- Stats/search scan local files linearly — intended for developer-machine scale.
|
|
18
|
+
- No Vitest/Jest reporter package in this release.
|
|
19
|
+
- Manual trace writing remains `schemaVersion: "0.1"`.
|
|
20
|
+
- Linked release aligns `@agent-inspect/tui` with `agent-inspect` and `@agent-inspect/langchain` (all **1.4.0**).
|
|
21
|
+
|
|
22
|
+
## 1.3.0
|
|
4
23
|
|
|
5
|
-
|
|
24
|
+
Released **2026-06-12**.
|
|
6
25
|
|
|
7
|
-
|
|
26
|
+
### Added
|
|
27
|
+
|
|
28
|
+
- Added correlation metadata on `inspectRun` / `maybeInspectRun` (`correlationId`, `requestId`, `decisionId`, `groupId`) and `getCurrentCorrelationMetadata()`.
|
|
29
|
+
- Added redaction profiles (`local`, `share`, `strict`) for trace safety and share-safe exports.
|
|
30
|
+
- Added `redactionProfile` on `InspectRunOptions` and `ExportOptions`.
|
|
31
|
+
- Added `--redaction-profile` to `agent-inspect export`.
|
|
32
|
+
- Added LangChain streaming metadata support (`stream: true`) for token chunk counts and duration.
|
|
33
|
+
- Added bounded preview behavior for preview capture mode (`maxStreamPreviewChars`).
|
|
34
|
+
|
|
35
|
+
### Notes
|
|
36
|
+
|
|
37
|
+
- LangChain `capture: "metadata-only"` remains default; full stream text is not captured by default.
|
|
38
|
+
- LangChain streaming does not emit per-token JSONL events.
|
|
39
|
+
- Redaction profiles are key-based safeguards, not compliance-grade PII detection.
|
|
40
|
+
- Export redaction does not upload anywhere and does not mutate original traces.
|
|
41
|
+
- No vendor upload, hosted dashboard, or OTLP HTTP sink was added.
|
|
42
|
+
- Manual trace writing remains `schemaVersion: "0.1"`; v0.2 is not written by default.
|
|
43
|
+
|
|
44
|
+
## 1.2.0
|
|
8
45
|
|
|
9
|
-
|
|
46
|
+
Released **2026-06-11**. Changeset `5a7f785`.
|
|
10
47
|
|
|
11
48
|
### Added
|
|
12
49
|
|
package/README.md
CHANGED
|
@@ -20,7 +20,7 @@ agent-inspect gives those runs **structure**: an **execution tree** you can read
|
|
|
20
20
|
|
|
21
21
|
## Install
|
|
22
22
|
|
|
23
|
-
Current npm release: **1.1.
|
|
23
|
+
Current npm release: **1.3.0** (`agent-inspect`, `@agent-inspect/langchain`; `@agent-inspect/tui` is **1.2.1** — patch-only alignment release).
|
|
24
24
|
|
|
25
25
|
```bash
|
|
26
26
|
npm install agent-inspect
|
|
@@ -109,6 +109,21 @@ await maybeInspectRun("eval-case-42", async () => runAgent());
|
|
|
109
109
|
AGENT_INSPECT=1 node eval-runner.mjs
|
|
110
110
|
```
|
|
111
111
|
|
|
112
|
+
## What you can do today (v1.3.0)
|
|
113
|
+
|
|
114
|
+
- **Trace manually** with `inspectRun`, `step`, `step.llm`, `step.tool`, and `observe` — local JSONL under `.agent-inspect/` by default.
|
|
115
|
+
- **Toggle tracing** with `maybeInspectRun` and `AGENT_INSPECT=1` in eval harnesses or CI.
|
|
116
|
+
- **Correlate runs** with optional `correlationId`, `requestId`, `decisionId`, and `groupId` on `run_started` metadata.
|
|
117
|
+
- **Redact before disk** with default key-based redaction, or choose `redactionProfile`: `local`, `share`, or `strict`.
|
|
118
|
+
- **Inspect from the CLI** — `list`, `view`, `clean`, `logs`, `tail`, `export`, `diff`, `timeline`, `stats`, `search`.
|
|
119
|
+
- **Export share-safe copies** — `export --redaction-profile share` (or `strict`) writes local Markdown/HTML/OpenInference/OTLP JSON only.
|
|
120
|
+
- **Parse structured logs** you already emit (JSON first-class; log4js best-effort).
|
|
121
|
+
- **Optional LangChain adapter** — metadata-only by default; optional `persist: true` and `stream: true` streaming metadata (no full token capture by default).
|
|
122
|
+
- **Optional TUI** — `view --tui` when `@agent-inspect/tui` is installed.
|
|
123
|
+
- **Persisted-event foundation (v1.2.0+)** — in-memory `PersistedInspectEvent` converters; manual writing stays `schemaVersion: "0.1"`.
|
|
124
|
+
|
|
125
|
+
Nothing uploads traces by default. Review exports before sharing — see [safe trace sharing](docs/SAFE-TRACE-SHARING.md).
|
|
126
|
+
|
|
112
127
|
## What the trace shows
|
|
113
128
|
|
|
114
129
|
Each run produces a **JSONL** trace: `run_started` / `run_completed`, `step_started` / `step_completed`, with **nested steps**, **tool/LLM** types where you use `step.tool` / `step.llm`, and **durations** on completed steps. Failures are recorded on `step_completed` with `status: "error"` (there is no separate `step_failed` event). See [docs/SCHEMA.md](docs/SCHEMA.md).
|
|
@@ -150,6 +165,9 @@ More detail: [docs/LOGS.md](docs/LOGS.md) · [docs/LOG-TO-TREE-QUICKSTART.md](do
|
|
|
150
165
|
| `tail` | Watch structured logs while the app runs |
|
|
151
166
|
| `export` | Write Markdown / HTML / OpenInference-compatible JSON / OTLP JSON **locally** |
|
|
152
167
|
| `diff` | Compare two local runs (read-only) |
|
|
168
|
+
| `timeline` | Chronological view of one run |
|
|
169
|
+
| `stats` | Local aggregates over a trace directory |
|
|
170
|
+
| `search` | Deterministic search over local traces |
|
|
153
171
|
|
|
154
172
|
Full flags and behavior: [docs/CLI.md](docs/CLI.md).
|
|
155
173
|
|
|
@@ -159,41 +177,36 @@ Full flags and behavior: [docs/CLI.md](docs/CLI.md).
|
|
|
159
177
|
- See **which step dominated latency** in a multi-step planner or RAG pipeline.
|
|
160
178
|
- **Diff two runs** after a prompt, model, or routing change (see [diff examples](docs/DIFF.md)).
|
|
161
179
|
- Point **`logs`** / **`tail`** at existing job or service logs to get a **local execution view** without shipping data upstream.
|
|
162
|
-
- **Export** a run to Markdown for a PR, postmortem, or internal thread — then review before sharing.
|
|
180
|
+
- **Export** a run to Markdown for a PR, postmortem, or internal thread — use `--redaction-profile share` for share-safe copies, then review before sharing.
|
|
163
181
|
- Keep traces **on disk** while still using enterprise observability elsewhere.
|
|
164
182
|
|
|
165
183
|
## Stable foundation (AgentInspect 1.x)
|
|
166
184
|
|
|
167
|
-
**agent-inspect 1.x** (current: **1.
|
|
185
|
+
**agent-inspect 1.x** (current: **1.3.0**) is the **local-first trace workbench** for TypeScript AI agents:
|
|
168
186
|
|
|
169
|
-
- Instrument
|
|
187
|
+
- Instrument runs with `inspectRun` and `step`
|
|
170
188
|
- Write **local JSONL traces** (`schemaVersion: "0.1"` — compatibility retained)
|
|
171
|
-
- Inspect
|
|
172
|
-
- Safely remove old trace files with **`clean`**
|
|
189
|
+
- Inspect with **`list`**, **`view`**, **`clean`**, **`logs`**, **`tail`**, **`export`**, **`diff`**
|
|
173
190
|
|
|
174
|
-
**Stable APIs:** `inspectRun()`, `maybeInspectRun()`, `step()`, `step.llm()`, `step.tool()`, `observe()`.
|
|
191
|
+
**Stable APIs:** `inspectRun()`, `maybeInspectRun()`, `step()`, `step.llm()`, `step.tool()`, `observe()`, `getCurrentCorrelationMetadata()`.
|
|
175
192
|
|
|
176
|
-
Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInspectRun` with `AGENT_INSPECT=1`
|
|
193
|
+
Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInspectRun` with `AGENT_INSPECT=1` to toggle tracing in eval or CI — see [docs/API.md](docs/API.md).
|
|
177
194
|
|
|
178
|
-
**
|
|
195
|
+
**Shipped in 1.3.0:** correlation metadata; redaction profiles (`local` / `share` / `strict`); `export --redaction-profile`; LangChain `stream: true` metadata (chunk counts, duration — no full token capture by default).
|
|
179
196
|
|
|
180
|
-
**Also
|
|
197
|
+
**Also in 1.x** (local-first extensions):
|
|
181
198
|
|
|
182
|
-
-
|
|
183
|
-
-
|
|
184
|
-
-
|
|
185
|
-
- Local run comparison: **`diff`**
|
|
186
|
-
- Optional **`@agent-inspect/langchain`** callback adapter
|
|
187
|
-
- Optional **`@agent-inspect/tui`** terminal viewer
|
|
188
|
-
- **Fixtures** and **recipes** for deterministic checks and adoption patterns
|
|
199
|
+
- **v1.2.0** — experimental persisted-event foundation (`PersistedInspectEvent`, converters, in-memory tree bridge). Manual writing remains **`schemaVersion: "0.1"`**; v0.2 is **not written by default**.
|
|
200
|
+
- Optional **`@agent-inspect/langchain`** and **`@agent-inspect/tui`**
|
|
201
|
+
- **Fixtures** and **recipes** for deterministic adoption patterns
|
|
189
202
|
|
|
190
|
-
**Honest boundaries:**
|
|
203
|
+
**Honest boundaries:** log parsing, export, diff, LangChain/TUI programmatic APIs, and OpenInference/OTLP JSON exports are **experimental or compatibility-oriented**. Nothing performs **vendor upload** by default.
|
|
191
204
|
|
|
192
205
|
## Optional packages
|
|
193
206
|
|
|
194
207
|
### LangChain callback adapter (`@agent-inspect/langchain`)
|
|
195
208
|
|
|
196
|
-
Optional package: official **LangChain.js callbacks** (`BaseCallbackHandler`), **metadata-oriented by default**, **no monkey-patching**, **no vendor sink**. The LangChain adapter ships with 1.x; its programmatic API remains experimental and may evolve independently of the stable core tracing API.
|
|
209
|
+
Optional package: official **LangChain.js callbacks** (`BaseCallbackHandler`), **metadata-oriented by default**, **no monkey-patching**, **no vendor sink**. Optional **`stream: true`** records chunk counts and stream duration **without storing full token text by default**. The LangChain adapter ships with 1.x; its programmatic API remains experimental and may evolve independently of the stable core tracing API.
|
|
197
210
|
|
|
198
211
|
```bash
|
|
199
212
|
pnpm add agent-inspect @agent-inspect/langchain @langchain/core
|
|
@@ -271,25 +284,15 @@ For a detailed comparison, see [Compare with other tools](docs/COMPARE.md).
|
|
|
271
284
|
|
|
272
285
|
## Documentation
|
|
273
286
|
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
- [API
|
|
277
|
-
- [CLI
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
- [Exports](docs/EXPORTS.md)
|
|
284
|
-
- [Diff](docs/DIFF.md)
|
|
285
|
-
- [Adapters](docs/ADAPTERS.md)
|
|
286
|
-
- [Compare with other tools](docs/COMPARE.md)
|
|
287
|
-
- [Security](SECURITY.md)
|
|
288
|
-
- [Safe trace sharing checklist](docs/SAFE-TRACE-SHARING.md)
|
|
289
|
-
- [Changelog](CHANGELOG.md)
|
|
290
|
-
- [Known issues](docs/KNOWN-ISSUES.md)
|
|
291
|
-
- [Limitations](docs/LIMITATIONS.md)
|
|
292
|
-
- [Screenshot checklist (planned assets)](docs/SCREENSHOTS.md)
|
|
287
|
+
| Start here | Reference | Safety & boundaries |
|
|
288
|
+
| ---------- | --------- | ------------------- |
|
|
289
|
+
| [Getting started](docs/GETTING-STARTED.md) | [API](docs/API.md) | [Safe trace sharing](docs/SAFE-TRACE-SHARING.md) |
|
|
290
|
+
| [Install smoke test](docs/INSTALL-SMOKE-TEST.md) | [CLI](docs/CLI.md) | [Security](SECURITY.md) |
|
|
291
|
+
| [Log-to-tree quickstart](docs/LOG-TO-TREE-QUICKSTART.md) | [Schema](docs/SCHEMA.md) | [Limitations](docs/LIMITATIONS.md) |
|
|
292
|
+
| [Logging playbook](docs/LOGGING-PLAYBOOK.md) | [Exports](docs/EXPORTS.md) | [Known issues](docs/KNOWN-ISSUES.md) |
|
|
293
|
+
| [Examples](examples/README.md) | [Adapters](docs/ADAPTERS.md) | [Compare with other tools](docs/COMPARE.md) |
|
|
294
|
+
|
|
295
|
+
Also: [Architecture](docs/ARCHITECTURE.md) · [Logs & tail](docs/LOGS.md) · [Diff](docs/DIFF.md) · [Changelog](CHANGELOG.md) · [Roadmap](ROADMAP.md) · [Contributing](CONTRIBUTING.md) · [Good first issues](GOOD-FIRST-ISSUES.md)
|
|
293
296
|
|
|
294
297
|
## Contributing
|
|
295
298
|
|
package/docs/ADAPTERS.md
CHANGED
|
@@ -1,26 +1,38 @@
|
|
|
1
|
-
|
|
1
|
+
# Adapters
|
|
2
2
|
|
|
3
|
-
AgentInspect is framework-agnostic at its core
|
|
3
|
+
AgentInspect is **framework-agnostic** at its core. Optional adapter packages integrate specific frameworks without monkey-patching, vendor sinks, or network upload.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
- Documented as **experimental** in `docs/API.md`
|
|
7
|
-
- Requires `@langchain/core` as a peer dependency
|
|
8
|
-
- **In-memory by default** (`getEvents()` / `clear()`)
|
|
9
|
-
- **Optional JSONL persistence** (`persist: true`) maps callback lifecycle to schemaVersion `"0.1"` manual events (`run_started`, `step_started`, `step_completed`, `run_completed`) so `list` / `view` / `export` / `diff` work without a separate schema
|
|
10
|
-
- **Metadata-only capture by default**; `capture: "preview"` is opt-in with truncation
|
|
11
|
-
- **No monkey-patching**, **no vendor sinks**, **no network upload**
|
|
12
|
-
- **Interactive TUI** (optional): `@agent-inspect/tui`
|
|
13
|
-
- Documented as **experimental** in `docs/API.md`
|
|
14
|
-
- Intended for CLI integration; programmatic TUI APIs may change
|
|
5
|
+
## LangChain.js (`@agent-inspect/langchain`)
|
|
15
6
|
|
|
16
|
-
|
|
7
|
+
**Status:** experimental — programmatic API may evolve independently of stable core tracing.
|
|
17
8
|
|
|
18
|
-
|
|
9
|
+
### Install
|
|
19
10
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
11
|
+
```bash
|
|
12
|
+
npm install agent-inspect @agent-inspect/langchain @langchain/core
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
### Basic callback (in-memory)
|
|
16
|
+
|
|
17
|
+
```ts
|
|
18
|
+
import { AgentInspectCallback } from "@agent-inspect/langchain";
|
|
19
|
+
|
|
20
|
+
const callback = new AgentInspectCallback({
|
|
21
|
+
runName: "support-agent",
|
|
22
|
+
capture: "metadata-only", // default
|
|
23
|
+
});
|
|
24
|
+
|
|
25
|
+
await agent.invoke(input, { callbacks: [callback] });
|
|
26
|
+
|
|
27
|
+
const events = callback.getEvents();
|
|
28
|
+
callback.clear();
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
- **No monkey-patching** — pass the callback explicitly to LangChain.
|
|
32
|
+
- **Metadata-only by default** — does not capture full prompts/outputs unless you opt into `capture: "preview"`.
|
|
33
|
+
- **No vendor sink** — events stay in memory unless you set `persist: true`.
|
|
34
|
+
|
|
35
|
+
### Persist to local JSONL
|
|
24
36
|
|
|
25
37
|
```ts
|
|
26
38
|
const callback = new AgentInspectCallback({
|
|
@@ -29,14 +41,103 @@ const callback = new AgentInspectCallback({
|
|
|
29
41
|
persist: true,
|
|
30
42
|
capture: "metadata-only",
|
|
31
43
|
});
|
|
44
|
+
|
|
45
|
+
await agent.invoke(input, { callbacks: [callback] });
|
|
32
46
|
```
|
|
33
47
|
|
|
34
48
|
```bash
|
|
35
49
|
npx agent-inspect list --dir ./.agent-inspect
|
|
36
50
|
npx agent-inspect view <run-id> --dir ./.agent-inspect
|
|
51
|
+
npx agent-inspect export <run-id> --format markdown --redaction-profile share
|
|
37
52
|
```
|
|
38
53
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
54
|
+
**Persistence model (Strategy A):**
|
|
55
|
+
|
|
56
|
+
1. **Standalone session** — one AgentInspect run per callback instance until the root LangChain run completes.
|
|
57
|
+
2. **Inside `inspectRun`** — callback steps append to the active manual run (no extra `run_started` / `run_completed`).
|
|
58
|
+
3. **Parent mapping** — LangChain `parentRunId` maps to AgentInspect `parentId` when the parent step was persisted; unknown parents stay at run root.
|
|
59
|
+
4. **Step types** — `LLM` → `llm`, `TOOL` → `tool`, `DECISION` → `decision`, other kinds → `logic`.
|
|
60
|
+
|
|
61
|
+
Written events use `schemaVersion: "0.1"` manual trace names.
|
|
62
|
+
|
|
63
|
+
### Capture modes
|
|
64
|
+
|
|
65
|
+
| `capture` | Behavior |
|
|
66
|
+
| --------- | -------- |
|
|
67
|
+
| `none` | Minimal metadata |
|
|
68
|
+
| `metadata-only` | **Default** — model names, token usage, timing; no full text |
|
|
69
|
+
| `preview` | Truncated previews via `maxPreviewChars` — review before sharing |
|
|
70
|
+
|
|
71
|
+
### Streaming metadata (v1.3.0+)
|
|
72
|
+
|
|
73
|
+
```ts
|
|
74
|
+
const callback = new AgentInspectCallback({
|
|
75
|
+
stream: true,
|
|
76
|
+
capture: "metadata-only",
|
|
77
|
+
persist: true,
|
|
78
|
+
});
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
When `stream: true`:
|
|
82
|
+
|
|
83
|
+
- `handleLLMNewToken` updates **in-memory stats only** — no per-token JSONL lines.
|
|
84
|
+
- On LLM end/error: `chunkCount`, `firstChunkAt`, `lastChunkAt`, `streamDurationMs`, `streamedCharCount`.
|
|
85
|
+
- **`capture: "metadata-only"`** does **not** store raw token text.
|
|
86
|
+
- **`capture: "preview"`** may include bounded `streamPreview` via `maxStreamPreviewChars`.
|
|
87
|
+
- With **`persist: true`**, streaming metadata is written on the LLM step at completion (deferred write for streaming LLM steps).
|
|
88
|
+
|
|
89
|
+
Streaming metadata is for **local inspection and timing** — not replay or cassette playback.
|
|
90
|
+
|
|
91
|
+
### Correlation inside `inspectRun`
|
|
92
|
+
|
|
93
|
+
When the callback runs inside `inspectRun` / `maybeInspectRun`, correlation fields from `getCurrentCorrelationMetadata()` attach to LLM lifecycle events.
|
|
94
|
+
|
|
95
|
+
### Options reference
|
|
96
|
+
|
|
97
|
+
| Option | Default | Notes |
|
|
98
|
+
| ------ | ------- | ----- |
|
|
99
|
+
| `persist` | `false` | Write local JSONL |
|
|
100
|
+
| `runName` | `"langchain-agent"` | Standalone persisted run name |
|
|
101
|
+
| `traceDir` | from env / `.agent-inspect` | |
|
|
102
|
+
| `capture` | `"metadata-only"` | |
|
|
103
|
+
| `stream` | `false` | Streaming lifecycle metadata |
|
|
104
|
+
| `maxStreamPreviewChars` | `maxPreviewChars` | Bounds preview when `capture: "preview"` |
|
|
105
|
+
| `redact` | — | Custom `RedactionRule[]` before disk |
|
|
106
|
+
|
|
107
|
+
Full API: [API.md](./API.md) §9.
|
|
108
|
+
|
|
109
|
+
### Example
|
|
110
|
+
|
|
111
|
+
[examples/08-langchain-adapter](../examples/08-langchain-adapter/README.md)
|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
|
|
115
|
+
## TUI (`@agent-inspect/tui`)
|
|
116
|
+
|
|
117
|
+
**Status:** experimental programmatic API; CLI integration is the intended usage.
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
npm install agent-inspect @agent-inspect/tui
|
|
121
|
+
npx agent-inspect view <run-id> --tui
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Requires an interactive terminal. See [API.md](./API.md) §10.
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## Future adapters (not shipped)
|
|
129
|
+
|
|
130
|
+
Direction only — see [ROADMAP.md](../ROADMAP.md):
|
|
131
|
+
|
|
132
|
+
- **Vercel AI SDK** — optional callback-style adapter (metadata-first, no vendor sink)
|
|
133
|
+
- **NestJS / logging bridges** — deeper recipes or helper patterns beyond [LOGGING-PLAYBOOK.md](./LOGGING-PLAYBOOK.md)
|
|
134
|
+
|
|
135
|
+
No automatic universal instrumentation. Integrations remain explicit and opt-in.
|
|
136
|
+
|
|
137
|
+
---
|
|
138
|
+
|
|
139
|
+
## Related docs
|
|
42
140
|
|
|
141
|
+
- [API.md](./API.md) — adapter options and stability policy
|
|
142
|
+
- [SAFE-TRACE-SHARING.md](./SAFE-TRACE-SHARING.md) — review exports before sharing
|
|
143
|
+
- [LIMITATIONS.md](./LIMITATIONS.md) — LangChain streaming and metadata boundaries
|
package/docs/API.md
CHANGED
|
@@ -26,16 +26,21 @@ import { inspectRun, maybeInspectRun, step, observe } from "agent-inspect";
|
|
|
26
26
|
```
|
|
27
27
|
|
|
28
28
|
- **`inspectRun(name, fn, options?)`**: wraps a workflow in a local JSONL trace (`run_started` / `run_completed`), prints terminal progress, and swallows instrumentation failures (user errors are re-thrown). **Traces by default** when `enabled` is omitted or `true`. Pass **`enabled: false`** to run `fn` with no trace file, no execution context, and no terminal output.
|
|
29
|
-
- **`redact`**: default `true` — redacts sensitive metadata keys before disk (`authorization`, `cookie`, `token`, `apiKey`, `password`, `secret`, `email`). Pass `false` to persist metadata as-is. Pass `{ rules?: RedactionRule[] }` for custom rules (defaults still apply).
|
|
29
|
+
- **`redact`**: default `true` — redacts sensitive metadata keys before disk (`authorization`, `cookie`, `token`, `apiKey`, `password`, `secret`, `email`). Pass `false` to persist metadata as-is. Pass `{ rules?: RedactionRule[] }` for custom rules (defaults still apply). **`redact: false` wins** over `redactionProfile` for trace writing.
|
|
30
|
+
- **`redactionProfile`**: optional preset — `local` (default), `share`, or `strict`. Adds extra key-based redaction and tighter metadata string bounds for trace writing. Key-based only — not compliance-grade DLP.
|
|
30
31
|
- **`maxMetadataValueLength`**: max string length for metadata values (default `2000`).
|
|
31
32
|
- **`maxPreviewLength`**: max string length for preview-like keys containing `preview` (default `500`).
|
|
32
33
|
- **`maxEventBytes`**: max UTF-8 bytes per serialized JSONL event (default `65536`). Oversized events are truncated; instrumentation never throws into user code.
|
|
34
|
+
- **Correlation metadata (v1.3.0+):** optional `correlationId`, `requestId`, `decisionId`, and `groupId` strings. When set, they are written on `run_started.metadata` (not on every step). Top-level correlation options override the same keys in `options.metadata`. Useful for eval cases, CI job IDs, request tracing, and future stats/cohort views. They are **metadata only** — they do not replace `runId`. Treat sensitive IDs as trace data before sharing exports.
|
|
33
35
|
- **`maybeInspectRun(name, fn, options?)`**: same as `inspectRun` when tracing is enabled; otherwise passthrough. Enablement: explicit **`options.enabled`** wins; when omitted, reads **`AGENT_INSPECT`** (`1`, `true`, `yes`, `on`, `enabled` — case-insensitive). Unset or other values disable tracing. Use in eval harnesses, CI, or jobs where tracing should be toggled by environment.
|
|
34
36
|
- **`isAgentInspectEnabled(value?)`**: returns whether a string (or `process.env.AGENT_INSPECT`) matches an enable token.
|
|
35
37
|
- **`step(name, fn, options?)`**: traces a named unit of work inside `inspectRun` (`step_started` / `step_completed`). Step `metadata` inherits the parent run's redaction and size-bound settings.
|
|
36
38
|
- **`step.llm(model, fn)`**: convenience wrapper (`type: "llm"`, `metadata.model`).
|
|
37
39
|
- **`step.tool(toolName, fn)`**: convenience wrapper (`type: "tool"`, `metadata.toolName`).
|
|
38
40
|
- **`observe(agent, options?)`**: proxy wrapper that traces top-level `run` / `execute` / `invoke` methods via `inspectRun`.
|
|
41
|
+
- **`getCurrentCorrelationMetadata()`**: returns active run correlation fields (`correlationId`, `requestId`, `decisionId`, `groupId`) inside `inspectRun` / `maybeInspectRun`; `undefined` outside a traced run or when none were set.
|
|
42
|
+
- **`RedactionProfile`**: `"local" | "share" | "strict"` — see `redactionProfile` on `InspectRunOptions` and `ExportOptions`.
|
|
43
|
+
- **`resolveRedactionProfile(profile?)`**: resolves profile extra keys and metadata caps (for advanced integrations).
|
|
39
44
|
|
|
40
45
|
## 3. Stable local inspection APIs
|
|
41
46
|
|
|
@@ -95,7 +100,8 @@ The CLI `tail` workflow is supported. The programmatic accumulator is experiment
|
|
|
95
100
|
|
|
96
101
|
Exports are **read-only**, **local-only**, and **compatibility-oriented**. They do not upload, stream, or integrate vendor SDKs.
|
|
97
102
|
|
|
98
|
-
- **`exportRunTree`**
|
|
103
|
+
- **`exportRunTree`**, **`redactRunTreeForExport`**
|
|
104
|
+
- **`ExportOptions.redactionProfile`**: `local` (default), `share`, or `strict` — applies key-based redaction to an exported copy without mutating the source tree.
|
|
99
105
|
- **`exportMarkdown`**, **`exportHtml`**
|
|
100
106
|
- **`exportOpenInference`** (OpenInference-compatible JSON)
|
|
101
107
|
- **`exportOtlpJson`** (OTLP JSON, experimental until verified per backend)
|
|
@@ -119,6 +125,8 @@ Diff is local and read-only. Programmatic diff surfaces are experimental until t
|
|
|
119
125
|
- **`runName`**: default `"langchain-agent"` for standalone persisted runs
|
|
120
126
|
- **`traceDir`**: defaults via `resolveTraceDir` / `AGENT_INSPECT_TRACE_DIR`
|
|
121
127
|
- **`capture`**: `"none"` | `"metadata-only"` (default) | `"preview"` (truncated previews, opt-in)
|
|
128
|
+
- **`stream`**: default `false` — when `true`, records streaming lifecycle metadata (`chunkCount`, `streamDurationMs`, etc.) on LLM end/error; does **not** capture full token text by default
|
|
129
|
+
- **`maxStreamPreviewChars`**: bounds `streamPreview` when `capture: "preview"` and `stream: true` (defaults to `maxPreviewChars`)
|
|
122
130
|
- **`redact`**: custom `RedactionRule[]` applied before disk (core defaults still apply via shared redactor)
|
|
123
131
|
- **`runId`**: optional id for standalone persisted runs
|
|
124
132
|
- In-memory **`getEvents()`** / **`clear()`** unchanged when `persist` is false
|
|
@@ -158,23 +166,33 @@ Related types: `PersistedInspectEvent`, `PersistedEventSourceType`, `PersistedEv
|
|
|
158
166
|
- v0.2 is **not written by default**; use converters and `fixtures/traces-v0.2/` samples for validation.
|
|
159
167
|
- Storage dual-read and CLI integration are future work.
|
|
160
168
|
|
|
161
|
-
## 12.
|
|
169
|
+
## 12. Local observability helpers (v1.4.0)
|
|
162
170
|
|
|
163
|
-
|
|
171
|
+
Read-only helpers for timeline, stats, and search over manual `TraceEvent` JSONL. Local files only.
|
|
164
172
|
|
|
165
|
-
|
|
173
|
+
- **`buildRunTimeline`**, **`renderTimeline`** — chronological run view; types `RunTimeline`, `TimelineEntry`
|
|
174
|
+
- **`buildTraceStats`**, **`renderTraceStats`** — directory aggregates; type `TraceStats`
|
|
175
|
+
- **`searchTraces`**, **`parseDurationFilter`**, **`loadTraceMetadataList`** — deterministic search; types `TraceSearchResult`, `TraceSearchOptions`
|
|
176
|
+
|
|
177
|
+
CLI wrappers: `agent-inspect timeline`, `stats`, `search` — see [CLI.md](./CLI.md).
|
|
178
|
+
|
|
179
|
+
## 13. Deprecated APIs
|
|
180
|
+
|
|
181
|
+
No deprecated APIs are declared as of 1.4.0.
|
|
182
|
+
|
|
183
|
+
## 14. Removal / deprecation policy
|
|
166
184
|
|
|
167
185
|
- Stable APIs are not removed in v1.x.
|
|
168
186
|
- If removal is necessary, the API should be **deprecated** first, documented, and kept for a reasonable window (target: at least one minor line) unless security requires faster action.
|
|
169
187
|
|
|
170
|
-
##
|
|
188
|
+
## 15. Backward compatibility policy
|
|
171
189
|
|
|
172
190
|
- Manual trace JSONL (`schemaVersion: "0.1"`) remains readable.
|
|
173
191
|
- Additive schema changes are allowed in minor versions.
|
|
174
192
|
- Breaking changes require a major version.
|
|
175
193
|
- Unknown fields should be ignored where safe.
|
|
176
194
|
|
|
177
|
-
##
|
|
195
|
+
## 16. Examples
|
|
178
196
|
|
|
179
197
|
### Minimal manual trace
|
|
180
198
|
|
package/docs/ARCHITECTURE.md
CHANGED
|
@@ -19,7 +19,7 @@ Root `agent-inspect` uses **conditional exports** for ESM/CJS TypeScript consume
|
|
|
19
19
|
- Manual traces use **`schemaVersion: "0.1"`** JSONL events (`run_started`, `step_started`, `step_completed`, `run_completed`).
|
|
20
20
|
- Failures use `step_completed` with `status: "error"` — there is no `step_failed` event.
|
|
21
21
|
- Log-derived runs use confidence labels (`explicit`, `correlated`, `heuristic`, `unknown`) and conservative tree-building rules.
|
|
22
|
-
- **v1.2.0 persisted-event foundation (
|
|
22
|
+
- **v1.2.0 persisted-event foundation (released):** source-agnostic `PersistedInspectEvent` helpers (`schemaVersion: "0.2"`) — types, validators, converters, and in-memory tree bridge. Existing **`TreeBuilder`** remains the canonical tree builder. v0.2 bridge works **in memory only**; storage dual-read and CLI integration are v1.3+ / post-foundation work. See [proposals/UNIFIED-PERSISTED-INSPECT-EVENT.md](./proposals/UNIFIED-PERSISTED-INSPECT-EVENT.md) and [implementation/V1.3.0-RELEASE-TRAIN.md](./implementation/V1.3.0-RELEASE-TRAIN.md).
|
|
23
23
|
|
|
24
24
|
See [SCHEMA.md](./SCHEMA.md) for field reference and [API.md](./API.md) for public surfaces (stable vs experimental).
|
|
25
25
|
|
package/docs/CLI.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# CLI (AgentInspect 1.
|
|
1
|
+
# CLI (AgentInspect 1.x)
|
|
2
2
|
|
|
3
3
|
This document describes the **stable CLI surface** of AgentInspect.
|
|
4
4
|
|
|
@@ -26,6 +26,9 @@ Core commands:
|
|
|
26
26
|
- `tail` — live-tail logs into updating local trees
|
|
27
27
|
- `export` — export manual traces to Markdown/HTML/OpenInference/OTLP JSON (local only)
|
|
28
28
|
- `diff` — compare two manual traces (local, read-only)
|
|
29
|
+
- `timeline` — chronological view of one run (local JSONL)
|
|
30
|
+
- `stats` — local aggregate stats over a trace directory
|
|
31
|
+
- `search` — deterministic local search over traces
|
|
29
32
|
|
|
30
33
|
## 2. Environment variables
|
|
31
34
|
|
|
@@ -162,7 +165,7 @@ Important: `tail` is a local developer tool, not a production monitor.
|
|
|
162
165
|
|
|
163
166
|
### 6.6 `export`
|
|
164
167
|
|
|
165
|
-
Export a manual trace run to local formats. **No upload
|
|
168
|
+
Export a manual trace run to local formats. **No upload.** Export redaction operates on a **copy** of the run tree — original JSONL files are not modified. Review every export before sharing, even with `--redaction-profile strict`.
|
|
166
169
|
|
|
167
170
|
```bash
|
|
168
171
|
agent-inspect export <run-id> --format <markdown|html|openinference|otlp-json> [options]
|
|
@@ -178,6 +181,14 @@ Options:
|
|
|
178
181
|
- `--include-attributes`: include bounded attributes (review before sharing)
|
|
179
182
|
- `--no-metadata`: omit summary/metadata sections
|
|
180
183
|
- `--no-errors`: omit error sections
|
|
184
|
+
- `--redaction-profile <profile>`: redaction profile for exported copies — `local` (default), `share`, or `strict`. Key-based safety only; review exports before sharing.
|
|
185
|
+
|
|
186
|
+
Examples:
|
|
187
|
+
|
|
188
|
+
```bash
|
|
189
|
+
npx agent-inspect export <run-id> --format markdown --redaction-profile share
|
|
190
|
+
npx agent-inspect export <run-id> --format html --redaction-profile strict
|
|
191
|
+
```
|
|
181
192
|
|
|
182
193
|
### 6.7 `diff`
|
|
183
194
|
|
|
@@ -243,6 +254,64 @@ Differences:
|
|
|
243
254
|
|
|
244
255
|
More examples, including timing-only and structure-only diffs, are in `docs/DIFF.md`.
|
|
245
256
|
|
|
257
|
+
### 6.8 `timeline`
|
|
258
|
+
|
|
259
|
+
Chronological step list for one manual trace. Read-only; does not mutate JSONL files.
|
|
260
|
+
|
|
261
|
+
```bash
|
|
262
|
+
agent-inspect timeline <run-id> [options]
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
Options:
|
|
266
|
+
|
|
267
|
+
- `--dir <path>`
|
|
268
|
+
- `--json` — structured `RunTimeline` JSON
|
|
269
|
+
- `--focus slow` — show only the slowest steps by duration (top N)
|
|
270
|
+
|
|
271
|
+
### 6.9 `stats`
|
|
272
|
+
|
|
273
|
+
Local aggregate statistics over trace files in a directory. Read-only.
|
|
274
|
+
|
|
275
|
+
```bash
|
|
276
|
+
agent-inspect stats [options]
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
Options:
|
|
280
|
+
|
|
281
|
+
- `--dir <path>`
|
|
282
|
+
- `--since <duration>` — e.g. `7d`, `24h`
|
|
283
|
+
- `--correlation-id <id>` — filter by `run_started.metadata.correlationId`
|
|
284
|
+
- `--group-id <id>` — filter by `run_started.metadata.groupId`
|
|
285
|
+
- `--json`
|
|
286
|
+
|
|
287
|
+
### 6.10 `search`
|
|
288
|
+
|
|
289
|
+
Deterministic search over local traces (substring / exact filters). No semantic search.
|
|
290
|
+
|
|
291
|
+
```bash
|
|
292
|
+
agent-inspect search [options]
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
Options:
|
|
296
|
+
|
|
297
|
+
- `--dir <path>`
|
|
298
|
+
- `--since <duration>`
|
|
299
|
+
- `--status <success|error|running|unknown>`
|
|
300
|
+
- `--kind <kind>` / `--type <type>` — manual step type (`llm`, `tool`, `logic`, …)
|
|
301
|
+
- `--name <query>` — substring on run or step name
|
|
302
|
+
- `--tool <query>` — substring on tool step name or `metadata.toolName`
|
|
303
|
+
- `--duration <expr>` — e.g. `>5s`, `>=500ms`
|
|
304
|
+
- `--limit <number>` — default 50
|
|
305
|
+
- `--json`
|
|
306
|
+
|
|
307
|
+
Examples:
|
|
308
|
+
|
|
309
|
+
```bash
|
|
310
|
+
npx agent-inspect search --status error --dir ./.agent-inspect
|
|
311
|
+
npx agent-inspect search --kind tool --name search
|
|
312
|
+
npx agent-inspect search --duration ">100ms" --json
|
|
313
|
+
```
|
|
314
|
+
|
|
246
315
|
## 7. Optional TUI behavior
|
|
247
316
|
|
|
248
317
|
`view --tui` delegates to `@agent-inspect/tui` and requires an interactive terminal. If the package is not installed, the CLI prints a short install hint.
|
package/docs/EXPORTS.md
CHANGED
|
@@ -1,12 +1,84 @@
|
|
|
1
|
-
|
|
1
|
+
# Exports
|
|
2
2
|
|
|
3
|
-
AgentInspect
|
|
3
|
+
AgentInspect exports are **local-only**: they produce strings or files on disk. Nothing uploads to a vendor, collector, or hosted dashboard.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
- **Schema + compatibility guarantees**: `docs/SCHEMA.md`
|
|
7
|
-
- **Safety / redaction notes**: `docs/CLI.md`, `SECURITY.md`, and `docs/SCHEMA.md` (redaction section)
|
|
5
|
+
Use exports to share run summaries in PRs, postmortems, or internal threads — **after reviewing** the output. See [SAFE-TRACE-SHARING.md](./SAFE-TRACE-SHARING.md).
|
|
8
6
|
|
|
9
|
-
|
|
10
|
-
- Export formats (Markdown/HTML/OpenInference/OTLP JSON) are documented as **experimental / compatibility-oriented** in `docs/API.md`.
|
|
11
|
-
- AgentInspect does **not** upload anywhere; exports write local strings/files only.
|
|
7
|
+
## CLI
|
|
12
8
|
|
|
9
|
+
```bash
|
|
10
|
+
agent-inspect export <run-id> --format <markdown|html|openinference|otlp-json> [options]
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Full flags: [CLI.md](./CLI.md) §6.6.
|
|
14
|
+
|
|
15
|
+
## Formats
|
|
16
|
+
|
|
17
|
+
| Format | Purpose |
|
|
18
|
+
| ------ | ------- |
|
|
19
|
+
| `markdown` | Human-readable tree for PRs, issues, docs |
|
|
20
|
+
| `html` | Standalone HTML snapshot for local viewing |
|
|
21
|
+
| `openinference` | OpenInference-compatible JSON (compatibility-oriented) |
|
|
22
|
+
| `otlp-json` | OTLP JSON shape (experimental; validate per backend) |
|
|
23
|
+
|
|
24
|
+
All formats are **compatibility-oriented** for local inspection and handoff — not guaranteed to match every vendor schema version.
|
|
25
|
+
|
|
26
|
+
## Common options
|
|
27
|
+
|
|
28
|
+
| Flag | Notes |
|
|
29
|
+
| ---- | ----- |
|
|
30
|
+
| `--dir <path>` | Trace directory (default from `AGENT_INSPECT_TRACE_DIR` or `.agent-inspect`) |
|
|
31
|
+
| `-o, --output <path>` | Write to file instead of stdout |
|
|
32
|
+
| `--json` | JSON wrapper output (includes content when not writing to file) |
|
|
33
|
+
| `--validate` | Validate exported payload shape before writing |
|
|
34
|
+
| `--include-attributes` | Include bounded attributes — **review before sharing** |
|
|
35
|
+
| `--no-metadata` | Omit summary/metadata sections |
|
|
36
|
+
| `--no-errors` | Omit error sections |
|
|
37
|
+
| `--redaction-profile <local\|share\|strict>` | Key-based redaction on the **export copy** (v1.3.0+) |
|
|
38
|
+
|
|
39
|
+
## Redaction profiles (share-safe exports)
|
|
40
|
+
|
|
41
|
+
Profiles apply to the **exported copy** only. Original JSONL traces on disk are **not mutated**.
|
|
42
|
+
|
|
43
|
+
| Profile | Typical use |
|
|
44
|
+
| ------- | ------------- |
|
|
45
|
+
| `local` | Default — same key-based defaults as trace writing |
|
|
46
|
+
| `share` | PRs, GitHub issues, internal Slack/email threads |
|
|
47
|
+
| `strict` | External or public sharing — also redacts prompt/output/message-like keys |
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
npx agent-inspect export <run-id> --format markdown --redaction-profile share
|
|
51
|
+
npx agent-inspect export <run-id> --format html --redaction-profile strict
|
|
52
|
+
npx agent-inspect export <run-id> --format openinference --redaction-profile share --validate
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Redaction profiles are **key-based safeguards**, not compliance-grade PII detection. Always review exports manually.
|
|
56
|
+
|
|
57
|
+
## Programmatic API
|
|
58
|
+
|
|
59
|
+
Experimental helpers (local-only):
|
|
60
|
+
|
|
61
|
+
- `exportRunTree`, `redactRunTreeForExport`
|
|
62
|
+
- `exportMarkdown`, `exportHtml`, `exportOpenInference`, `exportOtlpJson`
|
|
63
|
+
- `validateExport`, `validateExportContent`
|
|
64
|
+
|
|
65
|
+
See [API.md](./API.md) §7.
|
|
66
|
+
|
|
67
|
+
## What not to share
|
|
68
|
+
|
|
69
|
+
Even with `--redaction-profile strict`, review exports for:
|
|
70
|
+
|
|
71
|
+
- API keys, tokens, cookies, session IDs
|
|
72
|
+
- Customer IDs, emails, phone numbers, internal hostnames
|
|
73
|
+
- Full prompts, completions, tool I/O (especially with `--include-attributes`)
|
|
74
|
+
- File paths that expose usernames or internal project names
|
|
75
|
+
|
|
76
|
+
Prefer the smallest useful artifact — a summary or synthetic reproduction beats a full production trace.
|
|
77
|
+
|
|
78
|
+
## Related docs
|
|
79
|
+
|
|
80
|
+
- [CLI.md](./CLI.md) — `export` command reference
|
|
81
|
+
- [SCHEMA.md](./SCHEMA.md) — trace event model and redaction notes
|
|
82
|
+
- [SAFE-TRACE-SHARING.md](./SAFE-TRACE-SHARING.md) — checklist before posting
|
|
83
|
+
- [API.md](./API.md) — `ExportOptions`, experimental export APIs
|
|
84
|
+
- [LIMITATIONS.md](./LIMITATIONS.md) — compatibility and boundary notes
|