agent-inspect 3.4.0 → 3.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +23 -3
- package/README.md +102 -557
- package/docs/ADOPTION.md +67 -0
- package/docs/CI-ARTIFACTS.md +108 -0
- package/docs/COMPARE.md +13 -0
- package/docs/FIRST-TRACE-IN-5-MINUTES.md +56 -0
- package/docs/GETTING-STARTED.md +4 -1
- package/docs/ONE-PAGE-QUICKSTART.md +15 -0
- package/docs/PERFORMANCE.md +54 -0
- package/docs/SAFE-TRACE-SHARING.md +89 -0
- package/docs/USE-CASES.md +116 -0
- package/docs/assets/agent-inspect-loop.svg +22 -0
- package/docs/assets/package-map.svg +31 -0
- package/docs/assets/trace-check-redact.svg +19 -0
- package/package.json +12 -2
- package/packages/cli/dist/index.cjs +1 -1
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +1 -1
- package/packages/cli/dist/index.mjs.map +1 -1
package/docs/ADOPTION.md
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# Adoption guide
|
|
2
|
+
|
|
3
|
+
AgentInspect is **local-first**: traces stay on disk, checks run in CI, and sharing is opt-in via redaction. This guide is the blessed path from zero to a working team workflow.
|
|
4
|
+
|
|
5
|
+
## Who this is for
|
|
6
|
+
|
|
7
|
+
- TypeScript/Node teams shipping AI agents (not toy demos)
|
|
8
|
+
- Developers who want **structure** without a hosted observability account
|
|
9
|
+
- Maintainers who need **deterministic CI artifacts** (check, eval, redact)
|
|
10
|
+
|
|
11
|
+
## 5-minute path
|
|
12
|
+
|
|
13
|
+
See [FIRST-TRACE-IN-5-MINUTES.md](./FIRST-TRACE-IN-5-MINUTES.md).
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
npm install agent-inspect
|
|
17
|
+
npx agent-inspect init --yes
|
|
18
|
+
node examples/agent-inspect-demo.mjs
|
|
19
|
+
npx agent-inspect list --dir .agent-inspect
|
|
20
|
+
npx agent-inspect view <run-id> --dir .agent-inspect
|
|
21
|
+
npx agent-inspect verify-safe --dir .agent-inspect
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
No API keys required for the init demo. See [examples/starters/](../examples/starters/README.md).
|
|
25
|
+
|
|
26
|
+
## Choose a starter
|
|
27
|
+
|
|
28
|
+
| Goal | Starter |
|
|
29
|
+
| ---- | ------- |
|
|
30
|
+
| Manual `inspectRun` / `step` | [custom-observe](../examples/starters/custom-observe/) |
|
|
31
|
+
| Vercel AI SDK | [ai-sdk](../examples/starters/ai-sdk/) — see [AI-SDK-ADOPTION.md](./AI-SDK-ADOPTION.md) |
|
|
32
|
+
| OpenAI Agents JS | [openai-agents](../examples/starters/openai-agents/) — see [OPENAI-AGENTS-LOCAL.md](./OPENAI-AGENTS-LOCAL.md) |
|
|
33
|
+
| LangChain | [langchain](../examples/starters/langchain/) |
|
|
34
|
+
| CI eval + redact | [ci-eval-redact](../examples/starters/ci-eval-redact/) |
|
|
35
|
+
| Broken agent debug | [broken-agent-debugging](../examples/starters/broken-agent-debugging/) |
|
|
36
|
+
| NestJS harness | [harness-nestjs](../examples/starters/harness-nestjs/) — see [NESTJS.md](./NESTJS.md) |
|
|
37
|
+
| VS Code (dev host) | [VSCODE.md](./VSCODE.md) — extension in-repo; Marketplace publish is manual |
|
|
38
|
+
|
|
39
|
+
## Daily workflow
|
|
40
|
+
|
|
41
|
+
1. **Capture** — `observe()` / adapter / manual steps → `.agent-inspect/*.jsonl`
|
|
42
|
+
2. **Inspect** — `list`, `view`, `timeline`, `report`, optional `serve` viewer
|
|
43
|
+
3. **Verify** — `check`, `eval`, `@agent-inspect/circuit` rules in CI
|
|
44
|
+
4. **Share safely** — `redact --profile share` before attaching traces to issues/PRs
|
|
45
|
+
5. **Scale** — when directories grow, see [SCALE-LIMITS.md](./SCALE-LIMITS.md) and `agent-inspect index build`
|
|
46
|
+
|
|
47
|
+
## CI checklist
|
|
48
|
+
|
|
49
|
+
- Persist `.agent-inspect/` or upload as CI artifacts ([CI-ARTIFACTS.md](./CI-ARTIFACTS.md))
|
|
50
|
+
- Run `agent-inspect check` with `--require-completed` on fixture traces
|
|
51
|
+
- Run `agent-inspect eval` / `verify-safe` where applicable
|
|
52
|
+
- Never commit raw traces with secrets — use `redact` profiles
|
|
53
|
+
|
|
54
|
+
## When not to use AgentInspect alone
|
|
55
|
+
|
|
56
|
+
Use a production observability platform when you need hosted dashboards, fleet-wide sampling, or org-wide retention. AgentInspect complements that stack for the **inner loop**. See [COMPARE.md](./COMPARE.md).
|
|
57
|
+
|
|
58
|
+
## Presenting AgentInspect
|
|
59
|
+
|
|
60
|
+
- Live demo: [DEMO-SCRIPT.md](./DEMO-SCRIPT.md)
|
|
61
|
+
- Recorded walkthrough: [VIDEO-WALKTHROUGH-SCRIPT.md](./VIDEO-WALKTHROUGH-SCRIPT.md)
|
|
62
|
+
- Elevator pitch: [PITCH.md](./PITCH.md)
|
|
63
|
+
- Design partners: [DESIGN-PARTNER-GUIDE.md](./DESIGN-PARTNER-GUIDE.md)
|
|
64
|
+
|
|
65
|
+
## Measuring adoption (no default telemetry)
|
|
66
|
+
|
|
67
|
+
Track funnel metrics manually or from npm/GitHub public signals. See [product/ADOPTION-METRICS.md](./product/ADOPTION-METRICS.md).
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
# CI trace artifacts
|
|
2
|
+
|
|
3
|
+
AgentInspect helps you **write and export traces locally** in CI. Uploading artifacts is done by **your CI platform** (e.g. GitHub Actions `upload-artifact`) — AgentInspect does not upload anywhere.
|
|
4
|
+
|
|
5
|
+
## Quick pattern
|
|
6
|
+
|
|
7
|
+
1. Install `agent-inspect` in CI.
|
|
8
|
+
2. Enable tracing with `AGENT_INSPECT=1` and `maybeInspectRun` (or `inspectRun` when always-on is intended).
|
|
9
|
+
3. Set `AGENT_INSPECT_TRACE_DIR` (default `.agent-inspect`).
|
|
10
|
+
4. Run your job/tests.
|
|
11
|
+
5. Create safe CI artifacts: `agent-inspect artifacts <run-id> --output-dir ./artifacts`.
|
|
12
|
+
6. For reporter manifests, create a local CI summary: `agent-inspect ci-summary <manifest...> --output ./artifacts/reporter-summary.md`.
|
|
13
|
+
7. Optional legacy exports: `agent-inspect export <run-id> --redaction-profile share`.
|
|
14
|
+
8. Optional inspection reports: `agent-inspect what <run-id>` and `agent-inspect report <run-id> --format html`.
|
|
15
|
+
9. Upload files with your CI artifact step.
|
|
16
|
+
|
|
17
|
+
## Environment variables
|
|
18
|
+
|
|
19
|
+
| Variable | Purpose |
|
|
20
|
+
| -------- | ------- |
|
|
21
|
+
| `AGENT_INSPECT=1` | Enables `maybeInspectRun` tracing |
|
|
22
|
+
| `AGENT_INSPECT_TRACE_DIR` | Trace output directory |
|
|
23
|
+
| `AGENT_INSPECT_SILENT=true` | Suppress live terminal tree in CI logs |
|
|
24
|
+
|
|
25
|
+
## Export before upload
|
|
26
|
+
|
|
27
|
+
Prefer **`--redaction-profile share`** for internal PR/issue attachments; use **`strict`** for wider sharing.
|
|
28
|
+
|
|
29
|
+
For a deterministic CI bundle with structural JSON, safe Markdown/HTML summaries, safety check output, optional baseline diff output, and optional GitHub step-summary output:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
npx agent-inspect artifacts <run-id> --dir ./.agent-inspect \
|
|
33
|
+
--output-dir ./artifacts --github-summary "$GITHUB_STEP_SUMMARY"
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
This command writes local files only. It does not call GitHub APIs or upload artifacts.
|
|
37
|
+
|
|
38
|
+
For Vitest/Jest reporter artifacts, summarize shared `schemaVersion: "0.1"` reporter manifests without reading trace contents:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
npx agent-inspect ci-summary .agent-inspect/jest-artifacts/tests/**/report.json \
|
|
42
|
+
--output ./artifacts/reporter-summary.md \
|
|
43
|
+
--github-summary "$GITHUB_STEP_SUMMARY"
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
`ci-summary` writes local files only. It validates reporter artifact paths as relative paths and includes bounded structural metadata: package/framework, test status counts, trace filenames, artifact paths, redaction profile, and diagnostic counts.
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
npx agent-inspect export <run-id> --dir ./.agent-inspect \
|
|
50
|
+
--format markdown --redaction-profile share -o ./artifacts/trace.md
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Formats: `markdown`, `html`, `openinference`, `otlp-json` — all local files only.
|
|
54
|
+
|
|
55
|
+
## What and report (v1.5)
|
|
56
|
+
|
|
57
|
+
For quick human review in CI logs or local debugging:
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
npx agent-inspect what <run-id> --dir ./.agent-inspect
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
For a fuller inspection artifact (what + timeline + execution tree):
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
npx agent-inspect report <run-id> --dir ./.agent-inspect \
|
|
67
|
+
--format html --redaction-profile share -o ./artifacts/report.html
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Recipe: [examples/recipes/what-report-inspect](../examples/recipes/what-report-inspect/README.md)
|
|
71
|
+
|
|
72
|
+
## GitHub Actions example
|
|
73
|
+
|
|
74
|
+
Recipes:
|
|
75
|
+
|
|
76
|
+
- [examples/recipes/deterministic-ci-checks](../examples/recipes/deterministic-ci-checks/README.md) for v1.8 `check`, baseline, safe artifact, and step-summary workflows.
|
|
77
|
+
- [examples/recipes/github-actions-artifact](../examples/recipes/github-actions-artifact/README.md) for share-safe trace exports and reporter manifest summaries.
|
|
78
|
+
|
|
79
|
+
Sample workflows: [deterministic checks workflow](../examples/recipes/deterministic-ci-checks/workflow-example.yml), [share-safe export workflow](../examples/recipes/github-actions-artifact/workflow-example.yml)
|
|
80
|
+
|
|
81
|
+
```yaml
|
|
82
|
+
- uses: actions/upload-artifact@v4
|
|
83
|
+
with:
|
|
84
|
+
name: agent-inspect-traces
|
|
85
|
+
path: |
|
|
86
|
+
./.agent-inspect
|
|
87
|
+
./artifacts
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## Inspect artifacts locally after download
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
npx agent-inspect list --dir ./.agent-inspect
|
|
94
|
+
npx agent-inspect view <run-id> --dir ./.agent-inspect
|
|
95
|
+
npx agent-inspect what <run-id> --dir ./.agent-inspect
|
|
96
|
+
npx agent-inspect report <run-id> --dir ./.agent-inspect --format markdown
|
|
97
|
+
npx agent-inspect timeline <run-id> --dir ./.agent-inspect
|
|
98
|
+
npx agent-inspect stats --dir ./.agent-inspect
|
|
99
|
+
npx agent-inspect search --dir ./.agent-inspect --status error
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
## Safety checklist
|
|
103
|
+
|
|
104
|
+
- Review exports — redaction profiles are key-based, not compliance-grade DLP.
|
|
105
|
+
- Do not commit trace directories or artifacts with secrets.
|
|
106
|
+
- AgentInspect does not replace production observability platforms.
|
|
107
|
+
|
|
108
|
+
See [SAFE-TRACE-SHARING.md](./SAFE-TRACE-SHARING.md).
|
package/docs/COMPARE.md
CHANGED
|
@@ -63,8 +63,21 @@ AgentInspect avoids SDK/collector setup for local debugging:
|
|
|
63
63
|
| No-account CLI tracing | Strong fit |
|
|
64
64
|
| Deterministic local eval heuristics | Good fit |
|
|
65
65
|
| Share-safe local redaction copy | Good fit |
|
|
66
|
+
| VS Code trace sidebar (in-repo / dev host) | Good fit — Marketplace listing separate |
|
|
66
67
|
| Production dashboards | Not the goal |
|
|
67
68
|
| Hosted eval datasets | Not the goal |
|
|
68
69
|
| Prompt management | Not the goal |
|
|
69
70
|
| Standards-aligned local export | Partial (compatibility-oriented) |
|
|
70
71
|
| Full observability platform | Use a dedicated platform |
|
|
72
|
+
|
|
73
|
+
## v3.5 positioning (local inner loop)
|
|
74
|
+
|
|
75
|
+
AgentInspect v3.5 is the **adoption release** — not new runtime surface. Use it when:
|
|
76
|
+
|
|
77
|
+
- You want **traces on disk** before/alongside hosted tools
|
|
78
|
+
- You need **CI gates** (`check`, `eval`, `circuit`) without a vendor account
|
|
79
|
+
- You want **metadata-only** defaults and explicit `redact` before sharing
|
|
80
|
+
|
|
81
|
+
Keep using LangSmith, Langfuse, Braintrust, Phoenix, or OTel when you need hosted retention, fleet dashboards, dataset management, or org-wide production pipelines. AgentInspect is complementary.
|
|
82
|
+
|
|
83
|
+
**Full adoption path:** [ADOPTION.md](./ADOPTION.md)
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# First trace in 5 minutes
|
|
2
|
+
|
|
3
|
+
Goal: install → one trace → one check → one share-safe artifact.
|
|
4
|
+
|
|
5
|
+
## Minutes 0–1: Install
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
mkdir my-agent-debug && cd my-agent-debug
|
|
9
|
+
npm install agent-inspect
|
|
10
|
+
npx agent-inspect init --yes
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Creates `agent-inspect.config.ts`, `.agent-inspect/`, and `examples/agent-inspect-demo.mjs`.
|
|
14
|
+
|
|
15
|
+
## Minutes 1–2: Run
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
node examples/agent-inspect-demo.mjs
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
No API keys. Deterministic local trace.
|
|
22
|
+
|
|
23
|
+
## Minutes 2–3: Inspect
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
npx agent-inspect list --dir .agent-inspect
|
|
27
|
+
npx agent-inspect view <run-id> --dir .agent-inspect
|
|
28
|
+
npx agent-inspect report <run-id> --dir .agent-inspect
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Minutes 3–4: Check
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
npx agent-inspect check .agent-inspect/*.jsonl
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Minutes 4–5: Share-safe artifact
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
npx agent-inspect redact --profile share --dir .agent-inspect
|
|
41
|
+
npx agent-inspect verify-safe --dir .agent-inspect
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Attach redacted JSONL to a PR or issue — not raw traces.
|
|
45
|
+
|
|
46
|
+
## Next steps
|
|
47
|
+
|
|
48
|
+
| If you use… | Go to |
|
|
49
|
+
| ----------- | ----- |
|
|
50
|
+
| AI SDK | [AI SDK adoption](./AI-SDK-ADOPTION.md) |
|
|
51
|
+
| OpenAI Agents | [OpenAI Agents local](./OPENAI-AGENTS-LOCAL.md) |
|
|
52
|
+
| LangChain | [Adapters](./ADAPTERS.md) |
|
|
53
|
+
| CI tests | [CI artifacts](./CI-ARTIFACTS.md) |
|
|
54
|
+
| Broken agent demo | [broken-agent-debugging starter](../examples/starters/broken-agent-debugging/README.md) |
|
|
55
|
+
|
|
56
|
+
Full index: [docs/README.md](./README.md)
|
package/docs/GETTING-STARTED.md
CHANGED
|
@@ -12,8 +12,11 @@ pnpm add agent-inspect
|
|
|
12
12
|
|
|
13
13
|
### Quick bootstrap (v3.1+)
|
|
14
14
|
|
|
15
|
+
See also [FIRST-TRACE-IN-5-MINUTES.md](./FIRST-TRACE-IN-5-MINUTES.md) and [ONE-PAGE-QUICKSTART.md](./ONE-PAGE-QUICKSTART.md).
|
|
16
|
+
|
|
15
17
|
```bash
|
|
16
|
-
npx agent-inspect init
|
|
18
|
+
npx agent-inspect init --yes
|
|
19
|
+
node examples/agent-inspect-demo.mjs
|
|
17
20
|
npx agent-inspect doctor
|
|
18
21
|
```
|
|
19
22
|
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# One-page quickstart
|
|
2
|
+
|
|
3
|
+
```bash
|
|
4
|
+
npm install agent-inspect
|
|
5
|
+
npx agent-inspect init --yes
|
|
6
|
+
node examples/agent-inspect-demo.mjs
|
|
7
|
+
npx agent-inspect list --dir .agent-inspect
|
|
8
|
+
npx agent-inspect view <run-id> --dir .agent-inspect
|
|
9
|
+
npx agent-inspect report <run-id> --dir .agent-inspect
|
|
10
|
+
npx agent-inspect verify-safe --dir .agent-inspect
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
**Paths:** [Getting started](./GETTING-STARTED.md) · [Adoption](./ADOPTION.md) · [Starters](https://github.com/rajudandigam/agent-inspect/tree/main/examples/starters)
|
|
14
|
+
|
|
15
|
+
**Safety:** Traces are local files. No upload by default. Redact before sharing.
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# Performance
|
|
2
|
+
|
|
3
|
+
AgentInspect is **local-first** and optimized for developer workflows, not production APM. Use this guide to set expectations and run the repo baseline.
|
|
4
|
+
|
|
5
|
+
## Quick baseline
|
|
6
|
+
|
|
7
|
+
After `pnpm build`:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
pnpm perf:baseline
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The script prints deterministic timings for parse, normalize, tree build, diff, export, and checks on synthetic data. Warnings appear when a step exceeds internal soft thresholds.
|
|
14
|
+
|
|
15
|
+
## Comfortable ranges (v3.4)
|
|
16
|
+
|
|
17
|
+
| Workload | Comfortable | Warning | Not designed for |
|
|
18
|
+
| -------- | ----------- | ------- | ---------------- |
|
|
19
|
+
| Trace runs in one directory | ≤ 1,000 | 1,000–10,000 | > 10,000 without archive/split |
|
|
20
|
+
| Events per run (check/view) | ≤ 10,000 | 10,000–50,000 | > 100,000 single JSONL |
|
|
21
|
+
| Single trace file size | ≤ 10 MB | 50 MB+ | Multi-GB logs |
|
|
22
|
+
| `list` / `search` / `stats` | Sub-second on SSD for ≤1k runs | Seconds+ above warn threshold | Full scans of huge dirs every request |
|
|
23
|
+
|
|
24
|
+
See [SCALE-LIMITS.md](./SCALE-LIMITS.md) for CLI warnings and indexer guidance.
|
|
25
|
+
|
|
26
|
+
## What affects performance
|
|
27
|
+
|
|
28
|
+
- **Directory size:** `list`, `search`, and `stats` read metadata from every trace file unless an optional index is used.
|
|
29
|
+
- **Event count:** `view`, `check`, `report`, and `timeline` scale with events in the selected run.
|
|
30
|
+
- **Exports:** OpenInference and OTLP export walk the full tree.
|
|
31
|
+
- **Viewer:** `agent-inspect serve` loads one run at a time; startup is dominated by read + render.
|
|
32
|
+
|
|
33
|
+
## Optional index
|
|
34
|
+
|
|
35
|
+
For large directories (≥ 1,000 runs), build a rebuildable metadata index:
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
agent-inspect index build --dir .agent-inspect
|
|
39
|
+
agent-inspect index status --dir .agent-inspect
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The index is optional, local, and safe to delete (`agent-inspect index clean`).
|
|
43
|
+
|
|
44
|
+
## Stall and timeout checks
|
|
45
|
+
|
|
46
|
+
Use deterministic check rules on completed traces:
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
agent-inspect check run_abc --dir .agent-inspect --max-step-duration 30s
|
|
50
|
+
agent-inspect check run_abc --dir .agent-inspect --require-completed
|
|
51
|
+
agent-inspect check run_abc --dir .agent-inspect --detect-stalls
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
These complement `@agent-inspect/circuit` analyzers for loop/retry detection.
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
# Safe trace sharing checklist
|
|
2
|
+
|
|
3
|
+
AgentInspect traces, log-ingest outputs, and exports are local files. They may still contain sensitive metadata that you attached manually, collected from logs, or included through optional preview settings. Use this checklist before sharing an artifact in a GitHub issue, Discussion, PR, support thread, or public post.
|
|
4
|
+
|
|
5
|
+
This guide is practical sharing guidance, not a guarantee that any artifact is safe to publish. Redaction profiles are **key-based safeguards**, not compliance-grade DLP.
|
|
6
|
+
|
|
7
|
+
## Quick presets (v1.3.0+)
|
|
8
|
+
|
|
9
|
+
- **`--redaction-profile share`** — PRs, GitHub issues, Slack/email threads inside your org. Redacts correlation IDs, customer/user IDs, and common contact fields.
|
|
10
|
+
- **`--redaction-profile strict`** — external sharing or public posts. Also redacts prompt/output/message-like metadata keys.
|
|
11
|
+
- **Default (`local`)** — same as before; still review before sharing.
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
npx agent-inspect export <run-id> --format markdown --redaction-profile share
|
|
15
|
+
npx agent-inspect export <run-id> --format html --redaction-profile strict
|
|
16
|
+
npx agent-inspect redact trace.jsonl --profile share --json
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Original trace files under `.agent-inspect-runs/` are **not modified** by export or `redact` copy workflows.
|
|
20
|
+
|
|
21
|
+
### Default redaction before disk
|
|
22
|
+
|
|
23
|
+

|
|
24
|
+
|
|
25
|
+
Manual traces redact common sensitive keys **before disk** by default. Pass `redact: false` only when you accept the risk locally.
|
|
26
|
+
|
|
27
|
+
## Before sharing
|
|
28
|
+
|
|
29
|
+
- Use **`--redaction-profile share`** for PR/issue attachments; use **`strict`** when sharing outside your team.
|
|
30
|
+
- **Review** the exported file — profiles do not detect all sensitive data.
|
|
31
|
+
- Treat traces written with `redact: false` as sensitive. Review every event before sharing them outside your team.
|
|
32
|
+
- Inspect manual metadata passed to `inspectRun()`, `step()`, `step.tool()`, `step.llm()`, or `observe()`.
|
|
33
|
+
- Inspect log-derived fields from `logs` / `tail` ingest configs, including custom `run-id`, `event`, `parent`, timestamp, and attribute mappings.
|
|
34
|
+
- Avoid posting raw prompts, completions, tool inputs, or tool outputs in public threads unless the content is approved for public disclosure.
|
|
35
|
+
- Prefer Markdown export for issue or PR sharing when a summarized tree is enough.
|
|
36
|
+
|
|
37
|
+
## Remove or replace sensitive values
|
|
38
|
+
|
|
39
|
+
Search the artifact for:
|
|
40
|
+
|
|
41
|
+
- API keys, bearer tokens, cookies, session IDs, JWTs, OAuth codes, and webhook secrets
|
|
42
|
+
- passwords, private keys, certificates, signing secrets, and database URLs
|
|
43
|
+
- email addresses, phone numbers, physical addresses, usernames, and customer IDs
|
|
44
|
+
- internal hostnames, service URLs, ticket IDs, order IDs, account IDs, and tenant IDs
|
|
45
|
+
- proprietary prompts, system instructions, model outputs, retrieved documents, and tool responses
|
|
46
|
+
- file paths that expose user names, project names, or internal directory layouts
|
|
47
|
+
|
|
48
|
+
Replace sensitive data with clear placeholders such as `example.test`, `user@example.test`, `CUSTOMER_ID_REDACTED`, or `TOKEN_REDACTED`.
|
|
49
|
+
|
|
50
|
+
## Export and ingest-specific checks
|
|
51
|
+
|
|
52
|
+
- Markdown / HTML exports: review rendered text and copied snippets, not only the source trace.
|
|
53
|
+
- Eval JSON / Markdown: review failed-rule messages, expected/actual summaries, source IDs, and evidence paths before attaching them to PRs or issues.
|
|
54
|
+
- Redacted copies from `agent-inspect redact`: review the output file itself; findings show detector/path/action evidence but do not certify full safety.
|
|
55
|
+
- OpenInference / OTLP JSON exports: check attributes, span names, events, and resource metadata.
|
|
56
|
+
- Structured log ingest configs: confirm mapped keys do not pull in full request bodies, headers, raw prompts, or unbounded output fields.
|
|
57
|
+
- LangChain adapter traces: keep `capture: "metadata-only"` for shareable examples; review `capture: "preview"` traces carefully because previews can include prompt or output fragments.
|
|
58
|
+
|
|
59
|
+
## When to use each profile
|
|
60
|
+
|
|
61
|
+
| Situation | Profile | Format tip |
|
|
62
|
+
| --------- | ------- | ---------- |
|
|
63
|
+
| Local debugging only | `local` (default) | Full CLI `view` is fine on your machine |
|
|
64
|
+
| PR or GitHub issue attachment | `share` | `export --format markdown --redaction-profile share` |
|
|
65
|
+
| External blog, public forum, customer-facing | `strict` | Review twice; prefer synthetic/minimal repro |
|
|
66
|
+
| Local JSON/JSONL copy for review | `share` or `strict` | `redact <file> --profile share --json` |
|
|
67
|
+
| Security incident or secret leak suspicion | — | Do not post traces publicly; use [SECURITY.md](../SECURITY.md) |
|
|
68
|
+
|
|
69
|
+
## Guardrails and circuits (v2.5 planning)
|
|
70
|
+
|
|
71
|
+
- Optional `@agent-inspect/guardrails` and `@agent-inspect/circuit` packages evaluate local text/JSON/trace patterns deterministically.
|
|
72
|
+
- They reuse `@agent-inspect/redact` for PII-style findings where applicable — same best-effort limits as redaction profiles.
|
|
73
|
+
- They are **not** compliance tools and do not guarantee an artifact is safe to publish.
|
|
74
|
+
- Circuits detect repetition, retries, and width in traces or explicit counters; they do not stop runaway agents unless your code enforces results.
|
|
75
|
+
|
|
76
|
+
## What this guide does not claim
|
|
77
|
+
|
|
78
|
+
- Redaction profiles are **not** GDPR/HIPAA/SOC2 compliance tools.
|
|
79
|
+
- Export redaction does **not** upload anywhere — it only shapes a local copy.
|
|
80
|
+
- No automated scan guarantees an artifact is safe to publish.
|
|
81
|
+
|
|
82
|
+
## Final review
|
|
83
|
+
|
|
84
|
+
- Open the exact file you plan to share and search for common sensitive strings: `token`, `secret`, `authorization`, `cookie`, `password`, `email`, `apiKey`, `key`, `jwt`, `bearer`.
|
|
85
|
+
- Confirm any screenshots do not show terminal history, environment variables, browser account data, or private repository names.
|
|
86
|
+
- Share the smallest useful artifact: a summary, selected excerpt, or minimized synthetic reproduction is better than a full trace.
|
|
87
|
+
- If the artifact includes production or customer data, do not share it publicly. Use a private security or support channel.
|
|
88
|
+
|
|
89
|
+
See [SECURITY.md](../SECURITY.md) for the security policy and redaction expectations.
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
# Use cases
|
|
2
|
+
|
|
3
|
+
Problem-oriented guide for AgentInspect. Each case links to a starter or doc.
|
|
4
|
+
|
|
5
|
+
## 1. Wrong tool call
|
|
6
|
+
|
|
7
|
+
**Problem:** The agent picked `search_docs` instead of `refund_policy`.
|
|
8
|
+
|
|
9
|
+
**Why logs alone are hard:** Provider logs show a tool name, not parent run context or sibling steps.
|
|
10
|
+
|
|
11
|
+
**Use:** `agent-inspect` CLI + local trace tree.
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
npx agent-inspect view <run-id> --dir .agent-inspect
|
|
15
|
+
npx agent-inspect report <run-id> --dir .agent-inspect
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
**Output:** Step tree with tool metadata (not raw payloads by default).
|
|
19
|
+
|
|
20
|
+
**Safety:** Metadata-only default. Redact before sharing: `npx agent-inspect redact --profile share`.
|
|
21
|
+
|
|
22
|
+
**Starter:** [custom-observe](../examples/starters/custom-observe/README.md)
|
|
23
|
+
|
|
24
|
+
**Not:** Live model replay or hosted trace UI.
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## 2. Eval failure — which step broke?
|
|
29
|
+
|
|
30
|
+
**Problem:** A Vitest eval failed; you need the failing step, not just the assertion message.
|
|
31
|
+
|
|
32
|
+
**Use:** `@agent-inspect/vitest` or `npx agent-inspect check`.
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
npx agent-inspect check .agent-inspect/*.jsonl --require-completed
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
**Starter:** [ci-eval-redact](../examples/starters/ci-eval-redact/README.md)
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## 3. CI trace artifact on a failed PR
|
|
43
|
+
|
|
44
|
+
**Problem:** Reviewers need safe evidence without cloning your laptop.
|
|
45
|
+
|
|
46
|
+
**Use:** CI upload + `redact --profile share` + `verify-safe`.
|
|
47
|
+
|
|
48
|
+
**Doc:** [CI artifacts](./CI-ARTIFACTS.md)
|
|
49
|
+
|
|
50
|
+
---
|
|
51
|
+
|
|
52
|
+
## 4. Framework-native trace (AI SDK / OpenAI Agents / LangChain)
|
|
53
|
+
|
|
54
|
+
**Problem:** Manual `step()` calls do not match framework lifecycle.
|
|
55
|
+
|
|
56
|
+
**Use:** `@agent-inspect/ai-sdk`, `@agent-inspect/openai-agents`, or `@agent-inspect/langchain`.
|
|
57
|
+
|
|
58
|
+
**Starters:** [examples/starters](../examples/starters/README.md)
|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## 5. Safe incident handoff
|
|
63
|
+
|
|
64
|
+
**Problem:** You need to paste a trace into Slack or a GitHub issue.
|
|
65
|
+
|
|
66
|
+
**Use:** `@agent-inspect/redact` or CLI `redact` / `scan`.
|
|
67
|
+
|
|
68
|
+
**Doc:** [Safe trace sharing](./SAFE-TRACE-SHARING.md)
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
## 6. Multi-agent / session debugging
|
|
73
|
+
|
|
74
|
+
**Problem:** Handoffs and retries span multiple runs.
|
|
75
|
+
|
|
76
|
+
**Use:** `agent-inspect sessions`, `search`, `diff`.
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
## 7. MCP tool tracing
|
|
81
|
+
|
|
82
|
+
**Problem:** Which MCP tools were listed, called, and where did they fail?
|
|
83
|
+
|
|
84
|
+
**Use:** `@agent-inspect/mcp` (client tracing only).
|
|
85
|
+
|
|
86
|
+
---
|
|
87
|
+
|
|
88
|
+
## 8. Design partner adoption
|
|
89
|
+
|
|
90
|
+
**Problem:** Team wants one real agent instrumented in one sprint.
|
|
91
|
+
|
|
92
|
+
**Doc:** [Design partner guide](./DESIGN-PARTNER-GUIDE.md)
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## 9. VS Code trace review
|
|
97
|
+
|
|
98
|
+
**Problem:** Browse `.agent-inspect/` from the editor.
|
|
99
|
+
|
|
100
|
+
**Use:** In-repo `packages/vscode` (F5 dev). Not on Marketplace yet.
|
|
101
|
+
|
|
102
|
+
**Doc:** [VS Code](./VSCODE.md)
|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
## 10. Existing structured logs
|
|
107
|
+
|
|
108
|
+
**Problem:** You cannot change app code; logs already exist.
|
|
109
|
+
|
|
110
|
+
**Use:** `agent-inspect/logs` entry + CLI import helpers.
|
|
111
|
+
|
|
112
|
+
**Doc:** [Log-to-tree quickstart](./LOG-TO-TREE-QUICKSTART.md)
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
See also: [Real-world scenarios](./REAL-WORLD-SCENARIOS.md) · [Team workflows](./TEAM-WORKFLOWS.md)
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 120" role="img" aria-label="AgentInspect loop: trace, check, redact">
|
|
2
|
+
<defs>
|
|
3
|
+
<marker id="arrow" markerWidth="8" markerHeight="8" refX="6" refY="3" orient="auto">
|
|
4
|
+
<path d="M0,0 L6,3 L0,6 Z" fill="#334155"/>
|
|
5
|
+
</marker>
|
|
6
|
+
</defs>
|
|
7
|
+
<rect width="720" height="120" fill="#f8fafc" rx="8"/>
|
|
8
|
+
<g font-family="system-ui,sans-serif" font-size="14" fill="#0f172a">
|
|
9
|
+
<rect x="24" y="36" width="140" height="48" rx="8" fill="#dbeafe" stroke="#3b82f6"/>
|
|
10
|
+
<text x="94" y="66" text-anchor="middle">Trace</text>
|
|
11
|
+
<rect x="220" y="36" width="140" height="48" rx="8" fill="#dcfce7" stroke="#22c55e"/>
|
|
12
|
+
<text x="290" y="66" text-anchor="middle">Check</text>
|
|
13
|
+
<rect x="416" y="36" width="140" height="48" rx="8" fill="#fef3c7" stroke="#f59e0b"/>
|
|
14
|
+
<text x="486" y="66" text-anchor="middle">Redact</text>
|
|
15
|
+
<rect x="572" y="36" width="124" height="48" rx="8" fill="#f1f5f9" stroke="#64748b"/>
|
|
16
|
+
<text x="634" y="66" text-anchor="middle">Share</text>
|
|
17
|
+
<line x1="164" y1="60" x2="212" y2="60" stroke="#334155" marker-end="url(#arrow)"/>
|
|
18
|
+
<line x1="360" y1="60" x2="408" y2="60" stroke="#334155" marker-end="url(#arrow)"/>
|
|
19
|
+
<line x1="556" y1="60" x2="564" y2="60" stroke="#334155" marker-end="url(#arrow)"/>
|
|
20
|
+
<text x="360" y="108" text-anchor="middle" font-size="12" fill="#64748b">local files only � no upload by default</text>
|
|
21
|
+
</g>
|
|
22
|
+
</svg>
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 280" role="img" aria-label="AgentInspect package map">
|
|
2
|
+
<rect width="720" height="280" fill="#f8fafc" rx="8"/>
|
|
3
|
+
<g font-family="system-ui,sans-serif" font-size="12" fill="#0f172a">
|
|
4
|
+
<rect x="280" y="20" width="160" height="40" rx="6" fill="#1e293b" stroke="#0f172a"/>
|
|
5
|
+
<text x="360" y="45" text-anchor="middle" fill="#fff" font-weight="600">agent-inspect</text>
|
|
6
|
+
<text x="360" y="72" text-anchor="middle" font-size="11" fill="#64748b">core + CLI</text>
|
|
7
|
+
<line x1="360" y1="60" x2="120" y2="110" stroke="#94a3b8"/>
|
|
8
|
+
<line x1="360" y1="60" x2="280" y2="110" stroke="#94a3b8"/>
|
|
9
|
+
<line x1="360" y1="60" x2="440" y2="110" stroke="#94a3b8"/>
|
|
10
|
+
<line x1="360" y1="60" x2="600" y2="110" stroke="#94a3b8"/>
|
|
11
|
+
<rect x="40" y="110" width="160" height="36" rx="4" fill="#dbeafe" stroke="#3b82f6"/>
|
|
12
|
+
<text x="120" y="133" text-anchor="middle">Framework adapters</text>
|
|
13
|
+
<rect x="40" y="156" width="48" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="64" y="174" text-anchor="middle" font-size="10">ai-sdk</text>
|
|
14
|
+
<rect x="96" y="156" width="48" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="120" y="174" text-anchor="middle" font-size="10">oai</text>
|
|
15
|
+
<rect x="152" y="156" width="48" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="176" y="174" text-anchor="middle" font-size="10">lc</text>
|
|
16
|
+
<rect x="220" y="110" width="120" height="36" rx="4" fill="#dcfce7" stroke="#22c55e"/>
|
|
17
|
+
<text x="280" y="133" text-anchor="middle">Quality</text>
|
|
18
|
+
<rect x="220" y="156" width="56" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="248" y="174" text-anchor="middle" font-size="10">eval</text>
|
|
19
|
+
<rect x="284" y="156" width="56" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="312" y="174" text-anchor="middle" font-size="10">vitest</text>
|
|
20
|
+
<rect x="380" y="110" width="120" height="36" rx="4" fill="#fef3c7" stroke="#f59e0b"/>
|
|
21
|
+
<text x="440" y="133" text-anchor="middle">Safety</text>
|
|
22
|
+
<rect x="380" y="156" width="56" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="408" y="174" text-anchor="middle" font-size="10">redact</text>
|
|
23
|
+
<rect x="444" y="156" width="56" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="472" y="174" text-anchor="middle" font-size="10">guard</text>
|
|
24
|
+
<rect x="520" y="110" width="160" height="36" rx="4" fill="#f3e8ff" stroke="#a855f7"/>
|
|
25
|
+
<text x="600" y="133" text-anchor="middle">Inspect UX</text>
|
|
26
|
+
<rect x="520" y="156" width="48" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="544" y="174" text-anchor="middle" font-size="10">viewer</text>
|
|
27
|
+
<rect x="576" y="156" width="48" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="600" y="174" text-anchor="middle" font-size="10">tui</text>
|
|
28
|
+
<rect x="632" y="156" width="48" height="28" rx="4" fill="#fff" stroke="#cbd5e1"/><text x="656" y="174" text-anchor="middle" font-size="10">mcp</text>
|
|
29
|
+
<text x="360" y="250" text-anchor="middle" font-size="11" fill="#64748b">All packages: local-first � no default upload</text>
|
|
30
|
+
</g>
|
|
31
|
+
</svg>
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 200" role="img" aria-label="Trace to check to redact workflow">
|
|
2
|
+
<rect width="640" height="200" fill="#fff" rx="8"/>
|
|
3
|
+
<g font-family="system-ui,sans-serif" font-size="13" fill="#0f172a">
|
|
4
|
+
<text x="32" y="32" font-weight="600">Workflow</text>
|
|
5
|
+
<rect x="32" y="48" width="160" height="56" rx="6" fill="#eff6ff" stroke="#2563eb"/>
|
|
6
|
+
<text x="112" y="72" text-anchor="middle">.agent-inspect/</text>
|
|
7
|
+
<text x="112" y="90" text-anchor="middle" font-size="11" fill="#475569">JSONL trace</text>
|
|
8
|
+
<text x="220" y="80" font-size="18">�</text>
|
|
9
|
+
<rect x="248" y="48" width="160" height="56" rx="6" fill="#f0fdf4" stroke="#16a34a"/>
|
|
10
|
+
<text x="328" y="72" text-anchor="middle">check / eval</text>
|
|
11
|
+
<text x="328" y="90" text-anchor="middle" font-size="11" fill="#475569">deterministic rules</text>
|
|
12
|
+
<text x="436" y="80" font-size="18">�</text>
|
|
13
|
+
<rect x="464" y="48" width="144" height="56" rx="6" fill="#fffbeb" stroke="#d97706"/>
|
|
14
|
+
<text x="536" y="72" text-anchor="middle">redact</text>
|
|
15
|
+
<text x="536" y="90" text-anchor="middle" font-size="11" fill="#475569">share profile</text>
|
|
16
|
+
<text x="32" y="140" font-size="12" fill="#64748b">CLI: list � view � report � verify-safe</text>
|
|
17
|
+
<text x="32" y="160" font-size="12" fill="#64748b">Packages: @agent-inspect/eval � @agent-inspect/redact</text>
|
|
18
|
+
</g>
|
|
19
|
+
</svg>
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-inspect",
|
|
3
|
-
"version": "3.
|
|
3
|
+
"version": "3.5.1",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"description": "Local-first execution-tree debugger for TypeScript AI agents",
|
|
@@ -128,6 +128,7 @@
|
|
|
128
128
|
"SECURITY.md",
|
|
129
129
|
"CHANGELOG.md",
|
|
130
130
|
"docs/GETTING-STARTED.md",
|
|
131
|
+
"docs/ADOPTION.md",
|
|
131
132
|
"docs/API.md",
|
|
132
133
|
"docs/CLI.md",
|
|
133
134
|
"docs/SCHEMA.md",
|
|
@@ -142,7 +143,16 @@
|
|
|
142
143
|
"docs/ADAPTER-CONFORMANCE.md",
|
|
143
144
|
"docs/COMPARE.md",
|
|
144
145
|
"docs/LOG-TO-TREE-QUICKSTART.md",
|
|
145
|
-
"docs/SCREENSHOTS.md"
|
|
146
|
+
"docs/SCREENSHOTS.md",
|
|
147
|
+
"docs/SAFE-TRACE-SHARING.md",
|
|
148
|
+
"docs/CI-ARTIFACTS.md",
|
|
149
|
+
"docs/PERFORMANCE.md",
|
|
150
|
+
"docs/FIRST-TRACE-IN-5-MINUTES.md",
|
|
151
|
+
"docs/ONE-PAGE-QUICKSTART.md",
|
|
152
|
+
"docs/USE-CASES.md",
|
|
153
|
+
"docs/assets/agent-inspect-loop.svg",
|
|
154
|
+
"docs/assets/trace-check-redact.svg",
|
|
155
|
+
"docs/assets/package-map.svg"
|
|
146
156
|
],
|
|
147
157
|
"sideEffects": false,
|
|
148
158
|
"keywords": [
|