agent-inspect 3.5.3 → 3.5.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -2
- package/README.md +83 -41
- package/docs/ADOPTION.md +3 -1
- package/docs/AI-SDK-ADOPTION.md +109 -0
- package/docs/ARCHITECTURE.md +2 -0
- package/docs/CI-ARTIFACTS.md +2 -0
- package/docs/COMPARE.md +2 -0
- package/docs/DEMO-SCRIPT.md +87 -0
- package/docs/DESIGN-PARTNER-GUIDE.md +71 -0
- package/docs/FIRST-TRACE-IN-5-MINUTES.md +3 -1
- package/docs/GETTING-STARTED.md +2 -0
- package/docs/NESTJS.md +46 -0
- package/docs/OPENAI-AGENTS-LOCAL.md +52 -0
- package/docs/SAFE-TRACE-SHARING.md +2 -0
- package/docs/SCREENSHOTS.md +13 -1
- package/docs/TEAM-WORKFLOWS.md +47 -0
- package/docs/TECHNICAL-GUIDE.md +526 -0
- package/docs/VSCODE.md +21 -0
- package/docs/assets/agent-inspect-logo-dark.svg +13 -0
- package/docs/assets/agent-inspect-logo.svg +15 -0
- package/docs/assets/readme-product-loop.svg +31 -0
- package/package.json +19 -5
- package/packages/cli/dist/index.cjs +1 -1
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +1 -1
- package/packages/cli/dist/index.mjs.map +1 -1
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# Safe trace sharing checklist
|
|
2
2
|
|
|
3
|
+
**Docs site:** [https://agentinspect.vercel.app/docs/safe-sharing/](https://agentinspect.vercel.app/docs/safe-sharing/)
|
|
4
|
+
|
|
3
5
|
AgentInspect traces, log-ingest outputs, and exports are local files. They may still contain sensitive metadata that you attached manually, collected from logs, or included through optional preview settings. Use this checklist before sharing an artifact in a GitHub issue, Discussion, PR, support thread, or public post.
|
|
4
6
|
|
|
5
7
|
This guide is practical sharing guidance, not a guarantee that any artifact is safe to publish. Redaction profiles are **key-based safeguards**, not compliance-grade DLP.
|
package/docs/SCREENSHOTS.md
CHANGED
|
@@ -4,7 +4,7 @@ Curated terminal recordings and static diagrams for AgentInspect **3.5.x**. They
|
|
|
4
4
|
|
|
5
5
|
**Synthetic output only:** demos use committed [fixtures](../fixtures/README.md), [examples](../examples/README.md), and recipes. No external LLM calls or API keys.
|
|
6
6
|
|
|
7
|
-
**npm note:** GIFs live in `docs/assets/demos/` for GitHub documentation. They are **not** shipped in the `agent-inspect` npm tarball.
|
|
7
|
+
**npm note:** GIFs live in `docs/assets/demos/` for GitHub documentation. They are **not** shipped in the `agent-inspect` npm tarball. Brand SVGs used by the root README **are** included in the npm package.
|
|
8
8
|
|
|
9
9
|
**Re-record guide:** [assets/demos/RECORDING.md](assets/demos/RECORDING.md)
|
|
10
10
|
**Maintainer audit:** [implementation/VISUAL-DEMO-AUDIT.md](implementation/VISUAL-DEMO-AUDIT.md)
|
|
@@ -12,6 +12,18 @@ Curated terminal recordings and static diagrams for AgentInspect **3.5.x**. They
|
|
|
12
12
|
|
|
13
13
|
---
|
|
14
14
|
|
|
15
|
+
## README brand assets (v3.5.4)
|
|
16
|
+
|
|
17
|
+
| Asset | Use |
|
|
18
|
+
| ----- | --- |
|
|
19
|
+
| [agent-inspect-logo.svg](assets/agent-inspect-logo.svg) | Light-mode wordmark |
|
|
20
|
+
| [agent-inspect-logo-dark.svg](assets/agent-inspect-logo-dark.svg) | Dark-mode wordmark |
|
|
21
|
+
| [readme-product-loop.svg](assets/readme-product-loop.svg) | Capture → inspect → check → redact |
|
|
22
|
+
|
|
23
|
+
Root README embeds the logo and product-loop SVG only. Terminal GIFs stay below.
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
15
27
|
## Adoption demo (v3.5.2)
|
|
16
28
|
|
|
17
29
|
Deterministic broken-agent flow — recommended for live demos and Show HN rehearsal.
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Team workflows
|
|
2
|
+
|
|
3
|
+
How teams adopt AgentInspect without a hosted platform.
|
|
4
|
+
|
|
5
|
+
## Solo developer (day 1)
|
|
6
|
+
|
|
7
|
+
1. `npm install agent-inspect && npx agent-inspect init --yes`
|
|
8
|
+
2. Run demo: `node examples/agent-inspect-demo.mjs`
|
|
9
|
+
3. `npx agent-inspect view <run-id>`
|
|
10
|
+
4. Before posting issues: `npx agent-inspect redact --profile share`
|
|
11
|
+
|
|
12
|
+
## PR / code review
|
|
13
|
+
|
|
14
|
+
- Add `@agent-inspect/vitest` or `@agent-inspect/jest` reporter
|
|
15
|
+
- Upload `.agent-inspect/**/*.jsonl` as CI artifact on failure
|
|
16
|
+
- Reviewer downloads artifact and runs `agent-inspect report` locally
|
|
17
|
+
|
|
18
|
+
See [CI artifacts](./CI-ARTIFACTS.md).
|
|
19
|
+
|
|
20
|
+
## Design partner sprint
|
|
21
|
+
|
|
22
|
+
| Week | Action |
|
|
23
|
+
| ---- | ------ |
|
|
24
|
+
| 1 | Pick one agent path (adapter or observe) |
|
|
25
|
+
| 2 | Add one deterministic check rule |
|
|
26
|
+
| 3 | Redact + share one artifact with the team |
|
|
27
|
+
|
|
28
|
+
Checklist: [DESIGN-PARTNER-GUIDE.md](./DESIGN-PARTNER-GUIDE.md).
|
|
29
|
+
|
|
30
|
+
## Enterprise / no-cloud constraint
|
|
31
|
+
|
|
32
|
+
- All traces stay on disk under `AGENT_INSPECT_TRACE_DIR`
|
|
33
|
+
- No account, no default upload
|
|
34
|
+
- `verify-safe` before any export
|
|
35
|
+
- Redaction profiles: `local` → `share` → `strict`
|
|
36
|
+
|
|
37
|
+
## Framework owners
|
|
38
|
+
|
|
39
|
+
- AI SDK: metadata-only telemetry; `recordInputs: false`, `recordOutputs: false`
|
|
40
|
+
- OpenAI Agents: local-only processor mode (see [OPENAI-AGENTS-LOCAL.md](./OPENAI-AGENTS-LOCAL.md))
|
|
41
|
+
- LangChain: callback with `persist: true`
|
|
42
|
+
|
|
43
|
+
## What teams should not expect
|
|
44
|
+
|
|
45
|
+
- Org-wide trace search SaaS
|
|
46
|
+
- LLM-as-judge eval platform
|
|
47
|
+
- Compliance certification from redaction alone
|
|
@@ -0,0 +1,526 @@
|
|
|
1
|
+
# AgentInspect Technical Guide
|
|
2
|
+
|
|
3
|
+
**Version:** 3.5.3 · **Audience:** TypeScript agent developers, platform engineers, tech leads
|
|
4
|
+
**Purpose:** Source-of-truth technical overview for adoption content (blogs, articles, emails, demos).
|
|
5
|
+
**Status:** Grounded in shipped code and docs as of v3.5.3. Not a compliance or marketing guarantee.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## 1. Thesis
|
|
10
|
+
|
|
11
|
+
AgentInspect is a **local-first TypeScript toolkit** for AI agents. It helps you:
|
|
12
|
+
|
|
13
|
+
1. **Trace** what happened — capture runs and nested steps as a local execution tree
|
|
14
|
+
2. **Check** what should have happened — deterministic rules and eval heuristics in CI
|
|
15
|
+
3. **Redact** what must not leave your machine — profiles before sharing artifacts
|
|
16
|
+
|
|
17
|
+
**North star:** install → one trace → one failure check → one share-safe artifact in under five minutes.
|
|
18
|
+
|
|
19
|
+
**Principles:** CLI-first · TypeScript-first · dependency-light at the root · safe by default · framework-aware but not framework-locked · **no account · no upload · no hosted dashboard** · metadata-only by default.
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## 2. The problem
|
|
24
|
+
|
|
25
|
+
AI agents are not linear scripts. A single user request can fan out into parallel tool calls, nested LLM rounds, retries, handoffs, and MCP invocations. `console.log` gives you a **flat stream** — you see lines, not structure.
|
|
26
|
+
|
|
27
|
+
| Pain | Why flat logs fail |
|
|
28
|
+
| ---- | ------------------ |
|
|
29
|
+
| Wrong tool selected | You see a tool name, not sibling steps or parent run context |
|
|
30
|
+
| Eval/test failure | Assertion message only; no step-level evidence |
|
|
31
|
+
| Silent stall | No duration boundaries or step tree to spot hung work |
|
|
32
|
+
| Incident handoff | Raw logs may contain secrets; no share-safe workflow |
|
|
33
|
+
| Multi-agent flows | Handoffs span runs; hard to reconstruct causality |
|
|
34
|
+
|
|
35
|
+
**Hosted observability** (LangSmith, Langfuse, Braintrust, OTel backends) excels at retention, dashboards, and fleet-scale monitoring. AgentInspect targets the **inner loop**: local debugging, PR artifacts, and deterministic checks **before** you need a vendor account or collector setup.
|
|
36
|
+
|
|
37
|
+
AgentInspect is **complementary**, not a replacement. See [COMPARE.md](./COMPARE.md).
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## 3. Core idea: the execution tree
|
|
42
|
+
|
|
43
|
+
AgentInspect models agent work as a **tree of runs and steps**, persisted as **JSONL files** on disk (default directory `.agent-inspect/` or `AGENT_INSPECT_TRACE_DIR`).
|
|
44
|
+
|
|
45
|
+
### 3.1 Manual trace events (schema 0.1)
|
|
46
|
+
|
|
47
|
+
Written by `inspectRun()`, `step()`, and `observe()`:
|
|
48
|
+
|
|
49
|
+
| Event | Meaning |
|
|
50
|
+
| ----- | ------- |
|
|
51
|
+
| `run_started` | Top-level workflow begins |
|
|
52
|
+
| `step_started` | Nested unit of work begins |
|
|
53
|
+
| `step_completed` | Step ends (success or error) |
|
|
54
|
+
| `run_completed` | Run ends |
|
|
55
|
+
|
|
56
|
+
**Important:** There is no `step_failed` event. Failures use `step_completed` with `status: "error"`.
|
|
57
|
+
|
|
58
|
+
Step types include **tool**, **LLM**, and custom names. Convenience helpers: `step.tool(name, fn)` and `step.llm(model, fn)`.
|
|
59
|
+
|
|
60
|
+
### 3.2 Persisted InspectEvent (schema 1.0)
|
|
61
|
+
|
|
62
|
+
Framework adapters and `createInspector()` / writers can emit **schema 1.0** persisted rows. Manual global tracing remains 0.1 for compatibility. Readers normalize multiple schema versions for CLI inspection.
|
|
63
|
+
|
|
64
|
+
### 3.3 Log-derived trees
|
|
65
|
+
|
|
66
|
+
Structured JSON logs can be ingested via `agent-inspect logs` / `tail` into normalized in-memory trees with **confidence labels** (`explicit`, `correlated`, `heuristic`, `unknown`). Log-derived views are not identical to manual JSONL fidelity.
|
|
67
|
+
|
|
68
|
+
### 3.4 Product loop
|
|
69
|
+
|
|
70
|
+
```mermaid
|
|
71
|
+
flowchart LR
|
|
72
|
+
subgraph capture [Capture]
|
|
73
|
+
observe[observe]
|
|
74
|
+
manual[inspectRun_step]
|
|
75
|
+
adapters[Framework_adapters]
|
|
76
|
+
logs[Log_ingest]
|
|
77
|
+
end
|
|
78
|
+
subgraph persist [Local_persistence]
|
|
79
|
+
jsonl[".agent-inspect/*.jsonl"]
|
|
80
|
+
end
|
|
81
|
+
subgraph inspect [Inspect_and_quality]
|
|
82
|
+
cli[CLI_list_view_report]
|
|
83
|
+
check[check_eval_circuit]
|
|
84
|
+
end
|
|
85
|
+
subgraph safety [Safe_sharing]
|
|
86
|
+
redact[redact_profiles]
|
|
87
|
+
verify[verify-safe]
|
|
88
|
+
end
|
|
89
|
+
capture --> jsonl
|
|
90
|
+
jsonl --> cli
|
|
91
|
+
jsonl --> check
|
|
92
|
+
jsonl --> redact
|
|
93
|
+
redact --> verify
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
## 4. System architecture
|
|
99
|
+
|
|
100
|
+
### 4.1 Package layout
|
|
101
|
+
|
|
102
|
+
| Package | Published | Role |
|
|
103
|
+
| ------- | --------- | ---- |
|
|
104
|
+
| `agent-inspect` | Yes | Core tracing APIs + CLI binary |
|
|
105
|
+
| `@agent-inspect/core` | No (private) | Tracing, storage, readers, checks, export |
|
|
106
|
+
| `@agent-inspect/cli` | No (private) | Commander CLI implementation |
|
|
107
|
+
|
|
108
|
+
The root `agent-inspect` tarball ships core + CLI. Heavy optional deps stay in scoped packages.
|
|
109
|
+
|
|
110
|
+
### 4.2 Public package ecosystem (v3.5.3)
|
|
111
|
+
|
|
112
|
+
Sixteen linked npm packages at **3.5.3**:
|
|
113
|
+
|
|
114
|
+
| Tier | Package | Purpose |
|
|
115
|
+
| ---- | ------- | ------- |
|
|
116
|
+
| **Core** | `agent-inspect` | `observe`, `inspectRun`, `step`, CLI |
|
|
117
|
+
| **Framework** | `@agent-inspect/ai-sdk` | Vercel AI SDK telemetry integration |
|
|
118
|
+
| | `@agent-inspect/openai-agents` | OpenAI Agents JS tracing processor |
|
|
119
|
+
| | `@agent-inspect/langchain` | LangChain callback adapter |
|
|
120
|
+
| **Real projects** | `@agent-inspect/harness` | Fixture runner (bootstrap → invoke → shutdown) |
|
|
121
|
+
| **Quality** | `@agent-inspect/eval` | Deterministic eval heuristics over traces |
|
|
122
|
+
| | `@agent-inspect/vitest` | Vitest reporter (artifacts on failure) |
|
|
123
|
+
| | `@agent-inspect/jest` | Jest reporter |
|
|
124
|
+
| | `@agent-inspect/guardrails` | Deterministic guardrail rules |
|
|
125
|
+
| | `@agent-inspect/circuit` | Loop/retry/stall analyzers |
|
|
126
|
+
| **Safety** | `@agent-inspect/redact` | JSON/trace redaction (`local` / `share` / `strict`) |
|
|
127
|
+
| **Inspect UX** | `@agent-inspect/viewer` | Localhost read-only viewer |
|
|
128
|
+
| | `@agent-inspect/tui` | Optional terminal UI (`view --tui`) |
|
|
129
|
+
| | `@agent-inspect/mcp-server` | Read-only MCP server for trace listing |
|
|
130
|
+
| **Extensibility** | `@agent-inspect/mcp` | MCP **client** tool-call tracing |
|
|
131
|
+
| | `@agent-inspect/adapter-sdk` | Third-party adapter authoring + conformance |
|
|
132
|
+
|
|
133
|
+
**In-repo (not on Marketplace yet):** `agent-inspect-vscode` — read-only trace explorer via F5 dev host. See [VSCODE.md](./VSCODE.md).
|
|
134
|
+
|
|
135
|
+
Each scoped package has its own README on npm.
|
|
136
|
+
|
|
137
|
+
### 4.3 Instrumentation safety
|
|
138
|
+
|
|
139
|
+
Tracing, stepping, and observation **must not throw into user code** in ways that break the agent. Failures in instrumentation degrade gracefully (warn, skip event, continue). User errors inside `fn` are re-thrown unchanged.
|
|
140
|
+
|
|
141
|
+
---
|
|
142
|
+
|
|
143
|
+
## 5. Five capture paths
|
|
144
|
+
|
|
145
|
+
| Path | When | Entry |
|
|
146
|
+
| ---- | ---- | ----- |
|
|
147
|
+
| **Observe** | Existing class with `run` / `execute` / `invoke` | `observe(agent, { traceDir })` |
|
|
148
|
+
| **Manual** | Custom control flow, explicit step names | `inspectRun` + `step` / `step.tool` / `step.llm` |
|
|
149
|
+
| **AI SDK** | Vercel AI SDK `generateText` / `streamText` | `@agent-inspect/ai-sdk` + `recordInputs: false`, `recordOutputs: false` |
|
|
150
|
+
| **OpenAI Agents** | OpenAI Agents JS | `@agent-inspect/openai-agents` local-only processor mode |
|
|
151
|
+
| **LangChain** | Callback-based chains / LangGraph-via-LangChain | `@agent-inspect/langchain` with `persist: true` |
|
|
152
|
+
| **Logs** | Structured JSON logs already emitted | `agent-inspect logs` / `tail` |
|
|
153
|
+
| **Harness** | Real app (NestJS, etc.) with fixtures | `@agent-inspect/harness` + starters |
|
|
154
|
+
|
|
155
|
+
**Blessed starters** (no API keys): [examples/starters](../examples/starters/README.md)
|
|
156
|
+
|
|
157
|
+
**Quick bootstrap:**
|
|
158
|
+
|
|
159
|
+
```bash
|
|
160
|
+
npm install agent-inspect
|
|
161
|
+
npx agent-inspect init --yes
|
|
162
|
+
node examples/agent-inspect-demo.mjs
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
---
|
|
166
|
+
|
|
167
|
+
## 6. CLI workbench
|
|
168
|
+
|
|
169
|
+
All commands run locally. No network required.
|
|
170
|
+
|
|
171
|
+
### 6.1 Onboarding
|
|
172
|
+
|
|
173
|
+
| Command | Purpose |
|
|
174
|
+
| ------- | ------- |
|
|
175
|
+
| `init` | Config, trace dir, deterministic demo |
|
|
176
|
+
| `doctor` | Node, permissions, optional packages |
|
|
177
|
+
|
|
178
|
+
### 6.2 Browse and understand
|
|
179
|
+
|
|
180
|
+
| Command | Purpose |
|
|
181
|
+
| ------- | ------- |
|
|
182
|
+
| `list` | Runs in a trace directory |
|
|
183
|
+
| `view` | Execution tree in terminal (optional `--tui`, `--serve`) |
|
|
184
|
+
| `timeline` | Chronological view (e.g. `--focus slow`) |
|
|
185
|
+
| `what` | Concise run summary |
|
|
186
|
+
| `report` | Markdown/HTML report |
|
|
187
|
+
| `explain` | Local analysis dry-run |
|
|
188
|
+
| `sessions` / `session` | Multi-run session grouping |
|
|
189
|
+
| `search` | Filter by status, kind, etc. |
|
|
190
|
+
| `stats` | Aggregate directory metrics |
|
|
191
|
+
| `serve` | Localhost viewer |
|
|
192
|
+
|
|
193
|
+
### 6.3 Compare and migrate
|
|
194
|
+
|
|
195
|
+
| Command | Purpose |
|
|
196
|
+
| ------- | ------- |
|
|
197
|
+
| `diff` | Compare two runs (baseline vs candidate) |
|
|
198
|
+
| `open` | Open trace with format detection |
|
|
199
|
+
| `migrate` | Explicit schema migration (non-destructive) |
|
|
200
|
+
|
|
201
|
+
### 6.4 Quality gates
|
|
202
|
+
|
|
203
|
+
| Command | Purpose |
|
|
204
|
+
| ------- | ------- |
|
|
205
|
+
| `check` | Deterministic trace rules (duration, stalls, tool usage, etc.) |
|
|
206
|
+
| `eval` | Eval heuristics over traces |
|
|
207
|
+
| `scan` | Safety scan |
|
|
208
|
+
| `verify-safe` | Verify artifact is share-safe |
|
|
209
|
+
| `artifacts` / `ci-summary` | CI-oriented summaries |
|
|
210
|
+
|
|
211
|
+
**Example CI check:**
|
|
212
|
+
|
|
213
|
+
```bash
|
|
214
|
+
npx agent-inspect check .agent-inspect/*.jsonl --require-completed --detect-stalls
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
### 6.5 Safe sharing
|
|
218
|
+
|
|
219
|
+
| Command | Purpose |
|
|
220
|
+
| ------- | ------- |
|
|
221
|
+
| `export` | Markdown, HTML, OpenInference, OTLP JSON (local files) |
|
|
222
|
+
| `redact` | Copy with redaction profile |
|
|
223
|
+
|
|
224
|
+
### 6.6 Logs and scale
|
|
225
|
+
|
|
226
|
+
| Command | Purpose |
|
|
227
|
+
| ------- | ------- |
|
|
228
|
+
| `logs` / `tail` | Ingest structured logs |
|
|
229
|
+
| `index build` / `status` / `clean` | Optional metadata index for large directories |
|
|
230
|
+
| `clean` | Verified trace deletion |
|
|
231
|
+
|
|
232
|
+
### 6.7 End-to-end debug flow
|
|
233
|
+
|
|
234
|
+
```bash
|
|
235
|
+
cd examples/starters/broken-agent-debugging && pnpm install && pnpm start
|
|
236
|
+
npx agent-inspect report <run-id> --dir .agent-inspect
|
|
237
|
+
npx agent-inspect check .agent-inspect/*.jsonl
|
|
238
|
+
npx agent-inspect redact .agent-inspect/*.jsonl --profile share -o safe.jsonl
|
|
239
|
+
npx agent-inspect verify-safe safe.jsonl
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
---
|
|
243
|
+
|
|
244
|
+
## 7. Programmatic API
|
|
245
|
+
|
|
246
|
+
### 7.1 Stable root exports
|
|
247
|
+
|
|
248
|
+
```ts
|
|
249
|
+
import {
|
|
250
|
+
createInspector,
|
|
251
|
+
observe,
|
|
252
|
+
inspectRun,
|
|
253
|
+
maybeInspectRun,
|
|
254
|
+
step,
|
|
255
|
+
getCurrentCorrelationMetadata,
|
|
256
|
+
} from "agent-inspect";
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
| API | Use |
|
|
260
|
+
| --- | --- |
|
|
261
|
+
| `inspectRun(name, fn, opts?)` | Wrap workflow; writes JSONL |
|
|
262
|
+
| `maybeInspectRun` | Env-gated via `AGENT_INSPECT` |
|
|
263
|
+
| `step(name, fn)` | Named step inside a run |
|
|
264
|
+
| `step.tool` / `step.llm` | Typed step shortcuts |
|
|
265
|
+
| `observe(agent, opts?)` | Proxy `run`/`execute`/`invoke` |
|
|
266
|
+
| `getCurrentCorrelationMetadata()` | Active correlation IDs inside a run |
|
|
267
|
+
|
|
268
|
+
**Defaults:** metadata redaction before disk; size bounds on events; `enabled: false` skips tracing entirely.
|
|
269
|
+
|
|
270
|
+
### 7.2 Subpath exports (advanced / experimental)
|
|
271
|
+
|
|
272
|
+
| Subpath | Examples |
|
|
273
|
+
| ------- | -------- |
|
|
274
|
+
| `agent-inspect/logs` | `parseLogsToTrees` |
|
|
275
|
+
| `agent-inspect/exporters` | Markdown, HTML, OpenInference, OTLP JSON export |
|
|
276
|
+
| `agent-inspect/writers` | `fileWriter`, custom persistence |
|
|
277
|
+
| `agent-inspect/readers` | `openTrace`, multi-format read |
|
|
278
|
+
| `agent-inspect/checks` | `runTraceChecks`, rule factories |
|
|
279
|
+
| `agent-inspect/diff` | `diffTraceEvents` |
|
|
280
|
+
| `agent-inspect/persisted` | Schema 1.0 types and converters |
|
|
281
|
+
| `agent-inspect/advanced` | Runtime helpers, redaction profiles |
|
|
282
|
+
| `agent-inspect/reporters` | Reporter utilities |
|
|
283
|
+
|
|
284
|
+
See [API.md](./API.md) for stability classification.
|
|
285
|
+
|
|
286
|
+
### 7.3 Check rule families (deterministic)
|
|
287
|
+
|
|
288
|
+
Built-in check rules include (non-exhaustive): run status/duration, max step duration, stall detection, require-completed, tool usage/ordering/failures, LLM usage, structure (orphans, cycles, parallel width), retrieval, guardrails, decision metadata, safety (redaction, raw content, secret patterns, oversized attributes), baseline regression.
|
|
289
|
+
|
|
290
|
+
**No LLM-as-judge** in core checks or `@agent-inspect/eval`.
|
|
291
|
+
|
|
292
|
+
---
|
|
293
|
+
|
|
294
|
+
## 8. Framework adapters (high level)
|
|
295
|
+
|
|
296
|
+
### 8.1 AI SDK (`@agent-inspect/ai-sdk`)
|
|
297
|
+
|
|
298
|
+
- Explicit `experimental_telemetry.integrations` wiring — no monkey-patching
|
|
299
|
+
- **Required:** `recordInputs: false`, `recordOutputs: false` on AI SDK calls
|
|
300
|
+
- Metadata-only: model, timing, token counts, tool names — not raw prompts/outputs
|
|
301
|
+
- Guide: [AI-SDK-ADOPTION.md](./AI-SDK-ADOPTION.md)
|
|
302
|
+
|
|
303
|
+
### 8.2 OpenAI Agents (`@agent-inspect/openai-agents`)
|
|
304
|
+
|
|
305
|
+
- `TracingProcessor` for local JSONL
|
|
306
|
+
- **Local-only mode:** use `setTraceProcessors` replacement, not `addTraceProcessor`, to avoid default cloud export
|
|
307
|
+
- Maps agents, tools, handoffs, guardrails as metadata
|
|
308
|
+
- Guide: [OPENAI-AGENTS-LOCAL.md](./OPENAI-AGENTS-LOCAL.md)
|
|
309
|
+
|
|
310
|
+
### 8.3 LangChain (`@agent-inspect/langchain`)
|
|
311
|
+
|
|
312
|
+
- `AgentInspectCallback` with `persist: true`
|
|
313
|
+
- LangGraph supported through LangChain callback surfaces
|
|
314
|
+
- Guide: [ADAPTERS.md](./ADAPTERS.md)
|
|
315
|
+
|
|
316
|
+
### 8.4 MCP
|
|
317
|
+
|
|
318
|
+
- `@agent-inspect/mcp` — trace **client** tool list/call lifecycle only
|
|
319
|
+
- `@agent-inspect/mcp-server` — read-only trace access for IDE agents; no tool invocation, no mutation
|
|
320
|
+
|
|
321
|
+
---
|
|
322
|
+
|
|
323
|
+
## 9. CI and test integration
|
|
324
|
+
|
|
325
|
+
| Surface | Behavior |
|
|
326
|
+
| ------- | -------- |
|
|
327
|
+
| `@agent-inspect/vitest` | On test failure, write trace artifacts; passing tests stay quiet |
|
|
328
|
+
| `@agent-inspect/jest` | Same pattern for Jest |
|
|
329
|
+
| `agent-inspect check` | Gate merges on trace shape, completion, stalls |
|
|
330
|
+
| `agent-inspect eval` | Heuristic findings with evidence paths |
|
|
331
|
+
| CI artifacts | Upload `.agent-inspect/**/*.jsonl`; reviewer runs `report` locally |
|
|
332
|
+
|
|
333
|
+
Workflow: [CI-ARTIFACTS.md](./CI-ARTIFACTS.md) · Starter: [ci-eval-redact](../examples/starters/ci-eval-redact/)
|
|
334
|
+
|
|
335
|
+
---
|
|
336
|
+
|
|
337
|
+
## 10. Safety and privacy model
|
|
338
|
+
|
|
339
|
+
| Property | Behavior |
|
|
340
|
+
| -------- | -------- |
|
|
341
|
+
| Storage | Local JSONL only |
|
|
342
|
+
| Upload | **None** by default; AgentInspect does not phone home |
|
|
343
|
+
| Capture default | Metadata-only; no raw prompts/outputs unless you opt in |
|
|
344
|
+
| Redaction before disk | Common sensitive keys stripped on manual traces |
|
|
345
|
+
| Profiles | `local` → `share` → `strict` for export/redact |
|
|
346
|
+
| Verification | `scan`, `verify-safe` before attaching to issues |
|
|
347
|
+
| Instrumentation | Failures degrade gracefully; never break user agent |
|
|
348
|
+
| Compliance | **Not** a DLP or compliance engine — human review still required |
|
|
349
|
+
|
|
350
|
+
Details: [SAFE-TRACE-SHARING.md](./SAFE-TRACE-SHARING.md) · [SECURITY.md](../SECURITY.md)
|
|
351
|
+
|
|
352
|
+
---
|
|
353
|
+
|
|
354
|
+
## 11. Performance and scale (honest limits)
|
|
355
|
+
|
|
356
|
+
AgentInspect optimizes for **developer workflows**, not production APM.
|
|
357
|
+
|
|
358
|
+
| Workload | Comfortable | Warning threshold |
|
|
359
|
+
| -------- | ----------- | ----------------- |
|
|
360
|
+
| Runs per directory | ≤ 1,000 | 1,000–10,000 (consider index/archive) |
|
|
361
|
+
| Events per run | ≤ 10,000 | 10,000–50,000 |
|
|
362
|
+
| Single trace file | ≤ 10 MB | 50 MB+ |
|
|
363
|
+
|
|
364
|
+
Optional: `agent-inspect index build` for large directories. See [PERFORMANCE.md](./PERFORMANCE.md) and [SCALE-LIMITS.md](./SCALE-LIMITS.md).
|
|
365
|
+
|
|
366
|
+
---
|
|
367
|
+
|
|
368
|
+
## 12. Positioning matrix
|
|
369
|
+
|
|
370
|
+
| Need | AgentInspect | Alternative |
|
|
371
|
+
| ---- | ------------ | ----------- |
|
|
372
|
+
| Local agent debugging | **Strong** | console.log (flat) |
|
|
373
|
+
| No-account CLI tracing | **Strong** | Hosted platforms |
|
|
374
|
+
| Deterministic CI checks | **Good** | Braintrust eval platform |
|
|
375
|
+
| Share-safe redacted copy | **Good** | Manual scrubbing |
|
|
376
|
+
| Production dashboards | Not the goal | Langfuse, LangSmith |
|
|
377
|
+
| Hosted eval datasets | Not the goal | Braintrust, LangSmith |
|
|
378
|
+
| Prompt registry | Not the goal | Langfuse, etc. |
|
|
379
|
+
| Fleet-wide OTel pipelines | Not the goal | OTel + collector + backend |
|
|
380
|
+
| Standards export | Partial (local OpenInference/OTLP JSON) | Phoenix, OTel |
|
|
381
|
+
|
|
382
|
+
Full comparison: [COMPARE.md](./COMPARE.md)
|
|
383
|
+
|
|
384
|
+
---
|
|
385
|
+
|
|
386
|
+
## 13. What AgentInspect is not
|
|
387
|
+
|
|
388
|
+
- Hosted SaaS or dashboard product
|
|
389
|
+
- Production APM replacement
|
|
390
|
+
- Eval dataset platform or LLM-as-judge service
|
|
391
|
+
- Prompt registry or pricing/cost engine
|
|
392
|
+
- Default telemetry uploader or replay/cassette engine
|
|
393
|
+
- Raw chain-of-thought recorder
|
|
394
|
+
- Universal monkey-patching framework
|
|
395
|
+
- Compliance certification from redaction alone
|
|
396
|
+
|
|
397
|
+
See [LIMITATIONS.md](./LIMITATIONS.md).
|
|
398
|
+
|
|
399
|
+
---
|
|
400
|
+
|
|
401
|
+
## 14. Real-world scenarios
|
|
402
|
+
|
|
403
|
+
| # | Problem | Mechanism | Starter / doc |
|
|
404
|
+
| - | ------- | --------- | ------------- |
|
|
405
|
+
| 1 | Wrong tool call | `view`, `report` | [broken-agent-debugging](../examples/starters/broken-agent-debugging/) |
|
|
406
|
+
| 2 | Eval failure — which step? | `check`, `@agent-inspect/vitest` | [ci-eval-redact](../examples/starters/ci-eval-redact/) |
|
|
407
|
+
| 3 | CI PR artifact | upload + `redact` + `verify-safe` | [CI-ARTIFACTS.md](./CI-ARTIFACTS.md) |
|
|
408
|
+
| 4 | Framework-native trace | adapter packages | [starters](../examples/starters/) |
|
|
409
|
+
| 5 | Safe incident handoff | `redact --profile share` | [SAFE-TRACE-SHARING.md](./SAFE-TRACE-SHARING.md) |
|
|
410
|
+
| 6 | Multi-agent / sessions | `sessions`, `search`, `diff` | [USE-CASES.md](./USE-CASES.md) |
|
|
411
|
+
| 7 | MCP tool tracing | `@agent-inspect/mcp` | [ADAPTERS.md](./ADAPTERS.md) |
|
|
412
|
+
| 8 | Team adoption sprint | one agent, one check, one artifact | [DESIGN-PARTNER-GUIDE.md](./DESIGN-PARTNER-GUIDE.md) |
|
|
413
|
+
| 9 | VS Code review | in-repo extension | [VSCODE.md](./VSCODE.md) |
|
|
414
|
+
| 10 | Existing logs only | `logs`, `tail` | [LOG-TO-TREE-QUICKSTART.md](./LOG-TO-TREE-QUICKSTART.md) |
|
|
415
|
+
|
|
416
|
+
Full index: [USE-CASES.md](./USE-CASES.md) · [TEAM-WORKFLOWS.md](./TEAM-WORKFLOWS.md)
|
|
417
|
+
|
|
418
|
+
---
|
|
419
|
+
|
|
420
|
+
## 15. Adoption personas
|
|
421
|
+
|
|
422
|
+
| Persona | Pain | AgentInspect answer |
|
|
423
|
+
| ------- | ---- | ------------------- |
|
|
424
|
+
| **Solo agent developer** | Flat logs, wrong tool | Local tree + `report` in 60 seconds |
|
|
425
|
+
| **CI maintainer** | Failed test, no evidence | Vitest/Jest reporter + `check` gates |
|
|
426
|
+
| **Platform / security** | No-cloud data constraint | Disk-only traces, explicit redact |
|
|
427
|
+
| **Framework owner** | Manual wrapping tedious | AI SDK / OpenAI Agents / LangChain adapters |
|
|
428
|
+
| **Design partner lead** | Team needs one win in a sprint | [DESIGN-PARTNER-GUIDE.md](./DESIGN-PARTNER-GUIDE.md) |
|
|
429
|
+
|
|
430
|
+
Daily workflow: capture → inspect → verify → share safely → scale. See [ADOPTION.md](./ADOPTION.md).
|
|
431
|
+
|
|
432
|
+
---
|
|
433
|
+
|
|
434
|
+
## 16. Content creation appendix
|
|
435
|
+
|
|
436
|
+
Reuse these blocks in blogs, emails, and comments. All statements are accurate to v3.5.3; adjust version numbers when publishing.
|
|
437
|
+
|
|
438
|
+
### 16.1 Elevator pitches
|
|
439
|
+
|
|
440
|
+
**30 seconds:**
|
|
441
|
+
AgentInspect is a local-first TypeScript toolkit for AI agents. It writes execution trees to JSONL on your machine — no account, no upload. Inspect with the CLI, run deterministic checks in CI, and redact before sharing. It complements LangSmith and Langfuse for the inner loop.
|
|
442
|
+
|
|
443
|
+
**60 seconds:**
|
|
444
|
+
When an agent calls the wrong tool or a test fails, flat logs are not enough. AgentInspect captures runs and nested steps locally, lets you `report` and `diff` failures, and gates CI with deterministic `check` rules — no LLM judge. Framework adapters exist for AI SDK, OpenAI Agents, and LangChain. Metadata-only by default; redact before posting traces anywhere.
|
|
445
|
+
|
|
446
|
+
**120 seconds (technical):**
|
|
447
|
+
Add `observe()` or a framework adapter, get JSONL under `.agent-inspect/`, then use `agent-inspect list`, `view`, `report`, `check`, and `redact --profile share`. Sixteen npm packages cover adapters, eval, redaction, MCP tracing, harness fixtures, and test reporters. Schema 1.0 persisted events; readers handle 0.1/0.2/1.0. Not production APM — local debug and share-safe artifacts first.
|
|
448
|
+
|
|
449
|
+
### 16.2 Pull quotes (factual)
|
|
450
|
+
|
|
451
|
+
- "Traces stay on your disk. AgentInspect does not upload them."
|
|
452
|
+
- "Metadata-only by default — not a chain-of-thought recorder."
|
|
453
|
+
- "Deterministic checks, not LLM-as-judge evals."
|
|
454
|
+
- "Wrong tool call? See the step tree, not just a log line."
|
|
455
|
+
- "Redact before Slack, GitHub, or email — `local`, `share`, `strict` profiles."
|
|
456
|
+
- "Works before you set up OpenTelemetry or a hosted dashboard."
|
|
457
|
+
- "Framework-aware: AI SDK, OpenAI Agents, LangChain — optional packages."
|
|
458
|
+
- "Instrumentation failures degrade gracefully; your agent keeps running."
|
|
459
|
+
|
|
460
|
+
### 16.3 Blog post outlines
|
|
461
|
+
|
|
462
|
+
**A. First trace in 5 minutes**
|
|
463
|
+
Hook (flat logs) → `npm install` → `init` → demo → `list` / `report` → `check` → `redact` → link to starters.
|
|
464
|
+
|
|
465
|
+
**B. Debug the wrong tool call**
|
|
466
|
+
broken-agent-debugging starter → `report` highlights error step → fix → `diff` before/after → redact for issue attachment.
|
|
467
|
+
|
|
468
|
+
**C. CI trace artifacts without a vendor account**
|
|
469
|
+
Vitest reporter → upload artifact → `verify-safe` → reviewer workflow → contrast with hosted APM.
|
|
470
|
+
|
|
471
|
+
### 16.4 FAQ (for comments and emails)
|
|
472
|
+
|
|
473
|
+
| Question | Answer |
|
|
474
|
+
| -------- | ------ |
|
|
475
|
+
| Does it upload traces? | No. Local files only. |
|
|
476
|
+
| Replace LangSmith? | No. Complementary for local inner loop. |
|
|
477
|
+
| Store full prompts? | Not by default. Metadata-only; opt-in is caller responsibility. |
|
|
478
|
+
| LLM judge evals? | No. Deterministic heuristics and rules only. |
|
|
479
|
+
| VS Code extension? | In-repo; F5 dev host. Marketplace not published yet. |
|
|
480
|
+
| Node version? | Node 20+ recommended; run `doctor`. |
|
|
481
|
+
| Monorepo? | `pnpm build` at root; starters use workspace packages. |
|
|
482
|
+
| OpenTelemetry? | Local OTLP JSON export possible; not a full OTel SDK. |
|
|
483
|
+
| Compliance? | Redaction helps; not certification or DLP. |
|
|
484
|
+
|
|
485
|
+
### 16.5 Objection handlers (engineering managers)
|
|
486
|
+
|
|
487
|
+
| Objection | Response |
|
|
488
|
+
| --------- | -------- |
|
|
489
|
+
| "We already use Langfuse" | Use AgentInspect locally for PR/debug artifacts; Langfuse for retention and dashboards. |
|
|
490
|
+
| "Another dependency" | Root package is dependency-light; adapters are optional scoped packages. |
|
|
491
|
+
| "Security review" | No network, no account, metadata defaults, redact profiles, SECURITY.md policy. |
|
|
492
|
+
| "Will it slow our agents?" | Instrumentation is bounded; failures are swallowed; disable with `AGENT_INSPECT=0`. |
|
|
493
|
+
|
|
494
|
+
### 16.6 Demo and launch assets
|
|
495
|
+
|
|
496
|
+
| Asset | Location |
|
|
497
|
+
| ----- | -------- |
|
|
498
|
+
| Live 3-min demo | [DEMO-SCRIPT.md](./DEMO-SCRIPT.md) |
|
|
499
|
+
| Video script | [VIDEO-WALKTHROUGH-SCRIPT.md](./VIDEO-WALKTHROUGH-SCRIPT.md) |
|
|
500
|
+
| Show HN draft | [SHOW-HN-DRAFT.md](./SHOW-HN-DRAFT.md) |
|
|
501
|
+
| GIF demos | [SCREENSHOTS.md](./SCREENSHOTS.md) |
|
|
502
|
+
| Pitch | [PITCH.md](./PITCH.md) |
|
|
503
|
+
|
|
504
|
+
---
|
|
505
|
+
|
|
506
|
+
## 17. References
|
|
507
|
+
|
|
508
|
+
| Topic | Doc |
|
|
509
|
+
| ----- | --- |
|
|
510
|
+
| Website | [https://agentinspect.vercel.app/](https://agentinspect.vercel.app/) |
|
|
511
|
+
| Docs site | [https://agentinspect.vercel.app/docs/](https://agentinspect.vercel.app/docs/) |
|
|
512
|
+
| Docs index | [README.md](./README.md) |
|
|
513
|
+
| Getting started | [GETTING-STARTED.md](./GETTING-STARTED.md) · [Web](https://agentinspect.vercel.app/docs/getting-started/) |
|
|
514
|
+
| 5-minute path | [FIRST-TRACE-IN-5-MINUTES.md](./FIRST-TRACE-IN-5-MINUTES.md) |
|
|
515
|
+
| API | [API.md](./API.md) |
|
|
516
|
+
| CLI | [CLI.md](./CLI.md) |
|
|
517
|
+
| Schema | [SCHEMA.md](./SCHEMA.md) |
|
|
518
|
+
| Adapters | [ADAPTERS.md](./ADAPTERS.md) |
|
|
519
|
+
| Examples | [examples/starters](../examples/starters/) · [recipes](../examples/recipes/) |
|
|
520
|
+
| Changelog | [CHANGELOG.md](../CHANGELOG.md) |
|
|
521
|
+
| Roadmap | [ROADMAP.md](../ROADMAP.md) |
|
|
522
|
+
| Discussions | [GitHub Discussions](https://github.com/rajudandigam/agent-inspect/discussions) |
|
|
523
|
+
|
|
524
|
+
---
|
|
525
|
+
|
|
526
|
+
*AgentInspect v3.5.3 — local-first trace workbench for TypeScript AI agents. MIT license.*
|
package/docs/VSCODE.md
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# AgentInspect VS Code extension
|
|
2
|
+
|
|
3
|
+
Read-only sidebar for local trace directories. The extension shells out to the published `agent-inspect` CLI (`list`, `view`, `timeline`, `report`, `check`, `doctor`, `verify-safe`).
|
|
4
|
+
|
|
5
|
+
## Develop
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
pnpm install
|
|
9
|
+
pnpm --filter agent-inspect-vscode run build
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
Open `packages/vscode` in VS Code and press F5 (Extension Development Host).
|
|
13
|
+
|
|
14
|
+
## Requirements
|
|
15
|
+
|
|
16
|
+
- Node 20+
|
|
17
|
+
- `agent-inspect` available via `npx` in the workspace (devDependency or global)
|
|
18
|
+
|
|
19
|
+
## Manual gate
|
|
20
|
+
|
|
21
|
+
First VS Code Marketplace publish requires maintainer credentials. See [VSCODE-EXTENSION-RFC.md](./proposals/VSCODE-EXTENSION-RFC.md).
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 280 72" role="img" aria-label="AgentInspect">
|
|
2
|
+
<defs>
|
|
3
|
+
<linearGradient id="ai-grad-d" x1="0%" y1="0%" x2="100%" y2="100%">
|
|
4
|
+
<stop offset="0%" stop-color="#818cf8"/>
|
|
5
|
+
<stop offset="100%" stop-color="#a5b4fc"/>
|
|
6
|
+
</linearGradient>
|
|
7
|
+
</defs>
|
|
8
|
+
<rect x="8" y="12" width="48" height="48" rx="12" fill="url(#ai-grad-d)"/>
|
|
9
|
+
<path d="M24 28h16M24 36h12M24 44h16" stroke="#0f172a" stroke-width="2.5" stroke-linecap="round"/>
|
|
10
|
+
<circle cx="44" cy="36" r="5" fill="none" stroke="#0f172a" stroke-width="2"/>
|
|
11
|
+
<path d="M47.5 39.5L52 44" stroke="#0f172a" stroke-width="2.5" stroke-linecap="round"/>
|
|
12
|
+
<text x="68" y="44" font-family="ui-sans-serif,system-ui,-apple-system,Segoe UI,sans-serif" font-size="22" font-weight="600" fill="#f8fafc">agent-inspect</text>
|
|
13
|
+
</svg>
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 280 72" role="img" aria-label="AgentInspect">
|
|
2
|
+
<defs>
|
|
3
|
+
<linearGradient id="ai-grad" x1="0%" y1="0%" x2="100%" y2="100%">
|
|
4
|
+
<stop offset="0%" stop-color="#4f46e5"/>
|
|
5
|
+
<stop offset="100%" stop-color="#6366f1"/>
|
|
6
|
+
</linearGradient>
|
|
7
|
+
</defs>
|
|
8
|
+
<!-- Tree / inspect mark -->
|
|
9
|
+
<rect x="8" y="12" width="48" height="48" rx="12" fill="url(#ai-grad)"/>
|
|
10
|
+
<path d="M24 28h16M24 36h12M24 44h16" stroke="#fff" stroke-width="2.5" stroke-linecap="round"/>
|
|
11
|
+
<circle cx="44" cy="36" r="5" fill="none" stroke="#fff" stroke-width="2"/>
|
|
12
|
+
<path d="M47.5 39.5L52 44" stroke="#fff" stroke-width="2.5" stroke-linecap="round"/>
|
|
13
|
+
<!-- Wordmark -->
|
|
14
|
+
<text x="68" y="44" font-family="ui-sans-serif,system-ui,-apple-system,Segoe UI,sans-serif" font-size="22" font-weight="600" fill="#0f172a">agent-inspect</text>
|
|
15
|
+
</svg>
|