agent-inspect 0.1.2 → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +99 -0
- package/README.md +254 -44
- package/SECURITY.md +77 -0
- package/docs/ADAPTERS.md +15 -0
- package/docs/API.md +152 -0
- package/docs/ARCHITECTURE.md +13 -0
- package/docs/CLI.md +212 -0
- package/docs/COMPARE.md +69 -0
- package/docs/DIFF.md +11 -0
- package/docs/EXPORTS.md +12 -0
- package/docs/GETTING-STARTED.md +128 -0
- package/docs/KNOWN-ISSUES.md +34 -0
- package/docs/LIMITATIONS.md +32 -0
- package/docs/LOG-TO-TREE-QUICKSTART.md +54 -0
- package/docs/LOGS.md +13 -0
- package/docs/MIGRATION.md +109 -0
- package/docs/SCHEMA.md +199 -0
- package/docs/SCREENSHOTS.md +14 -0
- package/package.json +41 -15
- package/packages/cli/dist/index.cjs +3620 -118
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +3620 -119
- package/packages/cli/dist/index.mjs.map +1 -1
- package/packages/core/dist/index.cjs +2732 -30
- package/packages/core/dist/index.cjs.map +1 -1
- package/packages/core/dist/index.d.cts +461 -6
- package/packages/core/dist/index.d.ts +461 -6
- package/packages/core/dist/index.mjs +2687 -30
- package/packages/core/dist/index.mjs.map +1 -1
package/docs/API.md
ADDED
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# API (AgentInspect 1.0)
|
|
2
|
+
|
|
3
|
+
This document describes the **public TypeScript API surface** of AgentInspect and classifies each area as **stable** or **experimental**.
|
|
4
|
+
|
|
5
|
+
AgentInspect is a **local-first execution-tree debugger**. It is not a SaaS, not a production APM, not a sink/uploader, and not a replay engine.
|
|
6
|
+
|
|
7
|
+
## 1. Stability policy
|
|
8
|
+
|
|
9
|
+
- **Stable**: intended to be compatible across v1.x. Breaking changes require v2.0.
|
|
10
|
+
- **Experimental**: available for adoption, but subject to refinement (including naming/shape changes) before a future stability declaration. Experimental APIs may change in v1.x.
|
|
11
|
+
|
|
12
|
+
Notes:
|
|
13
|
+
|
|
14
|
+
- The core guarantee of v1.0 is **stable local debugging**: manual tracing + CLI inspection.
|
|
15
|
+
- Export formats (OpenInference / OTLP JSON) are **local-only** and **compatibility-oriented**. They do **not** upload anywhere.
|
|
16
|
+
- There are **zero production sinks** in v1.0; sink/uploader APIs are not stable.
|
|
17
|
+
|
|
18
|
+
## 2. Stable core APIs (manual tracing)
|
|
19
|
+
|
|
20
|
+
These are the recommended entry points for manual instrumentation. They are designed to be dependency-light and safe-by-default.
|
|
21
|
+
|
|
22
|
+
Import from `agent-inspect`:
|
|
23
|
+
|
|
24
|
+
```ts
|
|
25
|
+
import { inspectRun, step, observe } from "agent-inspect";
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
- **`inspectRun(name, fn, options?)`**: wraps a workflow in a local JSONL trace (`run_started` / `run_completed`), prints terminal progress, and swallows instrumentation failures (user errors are re-thrown).
|
|
29
|
+
- **`step(name, fn, options?)`**: traces a named unit of work inside `inspectRun` (`step_started` / `step_completed`).
|
|
30
|
+
- **`step.llm(model, fn)`**: convenience wrapper (`type: "llm"`, `metadata.model`).
|
|
31
|
+
- **`step.tool(toolName, fn)`**: convenience wrapper (`type: "tool"`, `metadata.toolName`).
|
|
32
|
+
- **`observe(agent, options?)`**: proxy wrapper that traces top-level `run` / `execute` / `invoke` methods via `inspectRun`.
|
|
33
|
+
|
|
34
|
+
## 3. Stable local inspection APIs
|
|
35
|
+
|
|
36
|
+
These APIs support local workflows like listing traces, extracting metadata/summaries, and safety checks for deletion.
|
|
37
|
+
|
|
38
|
+
- **`TraceDirectory`**, **`resolveTraceDir`**
|
|
39
|
+
- **`extractMetadata`**, **`buildRunSummary`**
|
|
40
|
+
- **`filterTraces`**
|
|
41
|
+
- **`isAgentInspectTrace`** (conservative trace verifier for cleanup)
|
|
42
|
+
- **`parseDuration`**, **`formatDuration`**
|
|
43
|
+
- Types: **`TraceMetadata`**, **`RunSummary`**
|
|
44
|
+
|
|
45
|
+
## 4. Stable event/config model types
|
|
46
|
+
|
|
47
|
+
### Manual trace JSONL model (stable)
|
|
48
|
+
|
|
49
|
+
- **`TraceSchemaVersion`** (`"0.1"`)
|
|
50
|
+
- **`TraceEvent`** union and specific event types:
|
|
51
|
+
- `RunStartedEvent` (`event: "run_started"`)
|
|
52
|
+
- `RunCompletedEvent` (`event: "run_completed"`)
|
|
53
|
+
- `StepStartedEvent` (`event: "step_started"`)
|
|
54
|
+
- `StepCompletedEvent` (`event: "step_completed"`)
|
|
55
|
+
- Related types: `StepType`, `StepStatus`, `RunStatus`, `ErrorInfo`, `StepMetadata`, `TokenMetadata`
|
|
56
|
+
|
|
57
|
+
### Log-derived normalized model (stable as a model, not necessarily the parsing APIs)
|
|
58
|
+
|
|
59
|
+
- **`InspectKind`**
|
|
60
|
+
- **`AttributionConfidence`**
|
|
61
|
+
- **`InspectEvent`**, **`InspectNode`**, **`InspectRunTree`**
|
|
62
|
+
- **`EventSource`**
|
|
63
|
+
|
|
64
|
+
### Log ingest config types (stable)
|
|
65
|
+
|
|
66
|
+
- **`LogIngestConfig`**
|
|
67
|
+
- **`LogEventMapping`**
|
|
68
|
+
- **`RedactionRule`**, **`RedactionStrategy`**
|
|
69
|
+
|
|
70
|
+
## 5. Experimental log parsing APIs
|
|
71
|
+
|
|
72
|
+
These are compatibility-oriented utilities for turning structured logs into normalized `InspectEvent` and grouped trees. They remain conservative: **no eval**, **no parsing JS object literals**, JSON logs first-class, log4js best-effort.
|
|
73
|
+
|
|
74
|
+
- **`parseLogsToTrees`**
|
|
75
|
+
- **`JsonLogParser`**, **`Log4jsParser`**
|
|
76
|
+
- **`EventNormalizer`**
|
|
77
|
+
- **`TreeBuilder`**
|
|
78
|
+
- **`Redactor`**
|
|
79
|
+
- **`renderRunTree`**, **`renderRunTrees`**
|
|
80
|
+
- **`parseLogLine`**
|
|
81
|
+
|
|
82
|
+
## 6. Experimental live tail APIs
|
|
83
|
+
|
|
84
|
+
The CLI `tail` workflow is supported. The programmatic accumulator is experimental.
|
|
85
|
+
|
|
86
|
+
- **`LiveLogAccumulator`**
|
|
87
|
+
|
|
88
|
+
## 7. Experimental standards export APIs
|
|
89
|
+
|
|
90
|
+
Exports are **read-only**, **local-only**, and **compatibility-oriented**. They do not upload, stream, or integrate vendor SDKs.
|
|
91
|
+
|
|
92
|
+
- **`exportRunTree`**
|
|
93
|
+
- **`exportMarkdown`**, **`exportHtml`**
|
|
94
|
+
- **`exportOpenInference`** (OpenInference-compatible JSON)
|
|
95
|
+
- **`exportOtlpJson`** (OTLP JSON, experimental until verified per backend)
|
|
96
|
+
- **`validateExport`**, `validateExportContent` (validation helpers)
|
|
97
|
+
|
|
98
|
+
## 8. Experimental diff APIs
|
|
99
|
+
|
|
100
|
+
Diff is local and read-only. Programmatic diff surfaces are experimental until the comparison semantics are explicitly frozen.
|
|
101
|
+
|
|
102
|
+
- **`diffRuns`**
|
|
103
|
+
- **`diffTraceEvents`**
|
|
104
|
+
- **`renderRunDiff`**
|
|
105
|
+
- **`manualTraceEventsToComparableRun`**
|
|
106
|
+
|
|
107
|
+
## 9. Experimental `@agent-inspect/langchain` APIs
|
|
108
|
+
|
|
109
|
+
`@agent-inspect/langchain` is an optional adapter package.
|
|
110
|
+
|
|
111
|
+
- **`AgentInspectCallback`** (experimental)
|
|
112
|
+
- Metadata helpers: `extractModelName`, `extractTokenUsage`, `safePreview`, `toPlainMetadata`
|
|
113
|
+
|
|
114
|
+
Rationale: v1.0 includes one official adapter and **zero production sinks**, so adapter surfaces remain experimental.
|
|
115
|
+
|
|
116
|
+
## 10. Experimental `@agent-inspect/tui` APIs
|
|
117
|
+
|
|
118
|
+
`@agent-inspect/tui` is an optional package. CLI integration via `agent-inspect view --tui` is supported; programmatic TUI APIs remain experimental.
|
|
119
|
+
|
|
120
|
+
- `runTraceViewer`, `loadTraceForTui`, `buildTuiTraceModel`, etc.
|
|
121
|
+
|
|
122
|
+
## 11. Deprecated APIs
|
|
123
|
+
|
|
124
|
+
No deprecated APIs are declared in v1.0 Pass 1.
|
|
125
|
+
|
|
126
|
+
## 12. Removal / deprecation policy
|
|
127
|
+
|
|
128
|
+
- Stable APIs are not removed in v1.x.
|
|
129
|
+
- If removal is necessary, the API should be **deprecated** first, documented, and kept for a reasonable window (target: at least one minor line) unless security requires faster action.
|
|
130
|
+
|
|
131
|
+
## 13. Backward compatibility policy
|
|
132
|
+
|
|
133
|
+
- Manual trace JSONL (`schemaVersion: "0.1"`) remains readable.
|
|
134
|
+
- Additive schema changes are allowed in minor versions.
|
|
135
|
+
- Breaking changes require a major version.
|
|
136
|
+
- Unknown fields should be ignored where safe.
|
|
137
|
+
|
|
138
|
+
## 14. Examples
|
|
139
|
+
|
|
140
|
+
### Minimal manual trace
|
|
141
|
+
|
|
142
|
+
```ts
|
|
143
|
+
import { inspectRun, step } from "agent-inspect";
|
|
144
|
+
|
|
145
|
+
await inspectRun("demo-agent", async () => {
|
|
146
|
+
const plan = await step("plan", async () => "ok");
|
|
147
|
+
const hits = await step.tool("search", async () => ({ count: 2 }));
|
|
148
|
+
const answer = await step.llm("fixture-model", async () => "done");
|
|
149
|
+
return { plan, hits, answer };
|
|
150
|
+
});
|
|
151
|
+
```
|
|
152
|
+
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
## Architecture
|
|
2
|
+
|
|
3
|
+
This doc is a stable entry point for AgentInspect’s internal architecture and product boundaries.
|
|
4
|
+
|
|
5
|
+
- **Canonical architecture overview**: see `docs-local/architecture/ARCHITECTURE.md`
|
|
6
|
+
- **Event model**: `docs-local/architecture/EVENT-MODEL.md`
|
|
7
|
+
- **Tree building rules**: `docs-local/architecture/TREE-BUILDING-RULES.md`
|
|
8
|
+
- **Schema evolution policy**: `docs-local/architecture/SCHEMA-EVOLUTION.md`
|
|
9
|
+
- **Dependency policy**: `docs-local/architecture/DEPENDENCY-POLICY.md`
|
|
10
|
+
- **Redaction design**: `docs-local/architecture/REDACTION.md`
|
|
11
|
+
|
|
12
|
+
If you’re new, start with `docs/GETTING-STARTED.md`, then `docs/API.md`, `docs/CLI.md`, and `docs/SCHEMA.md`.
|
|
13
|
+
|
package/docs/CLI.md
ADDED
|
@@ -0,0 +1,212 @@
|
|
|
1
|
+
# CLI (AgentInspect 1.0)
|
|
2
|
+
|
|
3
|
+
This document describes the **stable CLI surface** of AgentInspect.
|
|
4
|
+
|
|
5
|
+
AgentInspect is **local-first** and **read-only by default** where possible:
|
|
6
|
+
|
|
7
|
+
- **No upload** (exports write local strings/files only)
|
|
8
|
+
- **No vendor sinks**
|
|
9
|
+
- **No external services required**
|
|
10
|
+
- **No replay/fork execution**
|
|
11
|
+
|
|
12
|
+
## 1. Overview
|
|
13
|
+
|
|
14
|
+
The CLI command is:
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
agent-inspect <command> [options]
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Core commands:
|
|
21
|
+
|
|
22
|
+
- `list` — list local runs
|
|
23
|
+
- `view` — render a single run
|
|
24
|
+
- `clean` — safely delete old runs (verified traces only)
|
|
25
|
+
- `logs` — parse structured logs into local execution trees
|
|
26
|
+
- `tail` — live-tail logs into updating local trees
|
|
27
|
+
- `export` — export manual traces to Markdown/HTML/OpenInference/OTLP JSON (local only)
|
|
28
|
+
- `diff` — compare two manual traces (local, read-only)
|
|
29
|
+
|
|
30
|
+
## 2. Environment variables
|
|
31
|
+
|
|
32
|
+
- **`AGENT_INSPECT_TRACE_DIR`**: default directory for manual trace files (`.jsonl`) when not passed via `--dir` (or API options).
|
|
33
|
+
- **`AGENT_INSPECT_SILENT`**: when `true`, suppresses live terminal tree output during manual tracing (`inspectRun` / `step`). Trace files are still written.
|
|
34
|
+
|
|
35
|
+
## 3. Exit code policy
|
|
36
|
+
|
|
37
|
+
- **0**: command succeeded (even if a diff reports “differences”)
|
|
38
|
+
- **1**: command error (invalid args, missing files, missing runs, parse failures, validation failures, etc.)
|
|
39
|
+
|
|
40
|
+
AgentInspect favors **human-readable errors without stack traces** for expected user mistakes.
|
|
41
|
+
|
|
42
|
+
## 4. JSON output policy
|
|
43
|
+
|
|
44
|
+
Many commands support `--json` for scripting. JSON output is intended to be:
|
|
45
|
+
|
|
46
|
+
- machine-parseable
|
|
47
|
+
- deterministic for the same input files
|
|
48
|
+
- local-only (no network)
|
|
49
|
+
|
|
50
|
+
## 5. Safety and redaction notes
|
|
51
|
+
|
|
52
|
+
- Log-derived output includes **confidence** labels and avoids inventing parent-child relationships.
|
|
53
|
+
- Redaction defaults are conservative (e.g. `authorization`, `cookie`, `token`, `apiKey`, `password`, `secret`, `email`).
|
|
54
|
+
- Exported payloads are **redacted by default** unless explicitly configured otherwise.
|
|
55
|
+
|
|
56
|
+
## 6. Command reference
|
|
57
|
+
|
|
58
|
+
### 6.1 `list`
|
|
59
|
+
|
|
60
|
+
List recent local runs (trace files).
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
agent-inspect list [options]
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Options:
|
|
67
|
+
|
|
68
|
+
- `--dir <path>`: trace directory
|
|
69
|
+
- `--limit <number>`: max runs to show (default 20, max 100)
|
|
70
|
+
- `--status <running|success|error|unknown>`: filter by status
|
|
71
|
+
- `--name <query>`: substring match on run id/name
|
|
72
|
+
- `--since <duration>`: only include recent runs (e.g. `30s`, `5m`, `2h`, `7d`)
|
|
73
|
+
- `--json`: print list as JSON
|
|
74
|
+
|
|
75
|
+
### 6.2 `view`
|
|
76
|
+
|
|
77
|
+
Render a single manual trace by run id.
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
agent-inspect view <run-id> [options]
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Options:
|
|
84
|
+
|
|
85
|
+
- `--dir <path>`: trace directory
|
|
86
|
+
- `--summary`: run summary (counts, max depth, longest step)
|
|
87
|
+
- `--metadata`: file path/size + timestamps
|
|
88
|
+
- `--errors-only`: only error events/failed steps
|
|
89
|
+
- `--verbose`: include extra detail (types, metadata, stacks)
|
|
90
|
+
- `--json`: print raw trace events as JSON
|
|
91
|
+
- `--tui`: open optional interactive TUI viewer (requires `@agent-inspect/tui`)
|
|
92
|
+
|
|
93
|
+
### 6.3 `clean`
|
|
94
|
+
|
|
95
|
+
Safely delete old local trace files. This is safety-critical: the CLI verifies trace files before deletion.
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
agent-inspect clean --older-than <duration> [--dry-run] [--yes]
|
|
99
|
+
agent-inspect clean --keep <count> [--dry-run] [--yes]
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Options:
|
|
103
|
+
|
|
104
|
+
- `--dir <path>`: trace directory
|
|
105
|
+
- `--older-than <duration>`: delete runs older than a duration
|
|
106
|
+
- `--keep <count>`: keep N most recent runs
|
|
107
|
+
- `--dry-run`: show what would be deleted
|
|
108
|
+
- `--yes`: skip confirmation prompt
|
|
109
|
+
|
|
110
|
+
Recommendation: run with `--dry-run` first.
|
|
111
|
+
|
|
112
|
+
### 6.4 `logs`
|
|
113
|
+
|
|
114
|
+
Parse structured logs into local execution trees.
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
agent-inspect logs <file> [options]
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Options:
|
|
121
|
+
|
|
122
|
+
- `--format <auto|json|log4js>`
|
|
123
|
+
- `--config <path>`: ingest config JSON (see `docs/SCHEMA.md` for config types)
|
|
124
|
+
- `--run-id-key <keys>`: override runId keys (comma-separated)
|
|
125
|
+
- `--event-key <key>`: override event key
|
|
126
|
+
- `--timestamp-key <key>`: override timestamp key
|
|
127
|
+
- `--message-key <key>`: override message key
|
|
128
|
+
- `--level-key <key>`: override level key
|
|
129
|
+
- `--parent-id-key <key>`: override parent id key
|
|
130
|
+
- `--duration-key <key>`: override duration key
|
|
131
|
+
- `--status-key <key>`: override status key
|
|
132
|
+
- `--json`: emit JSON payload (events/trees/warnings/summary)
|
|
133
|
+
- `--summary`: include summary section in human output
|
|
134
|
+
- `--warnings <none|summary|all>`: warning output mode
|
|
135
|
+
|
|
136
|
+
Example (fixtures):
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
agent-inspect logs fixtures/logs/proactive-json.log --format json --config fixtures/configs/proactive-agent-inspect.logs.json
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### 6.5 `tail`
|
|
143
|
+
|
|
144
|
+
Live-tail logs into updating execution trees in the terminal.
|
|
145
|
+
|
|
146
|
+
```bash
|
|
147
|
+
agent-inspect tail [options]
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
Options:
|
|
151
|
+
|
|
152
|
+
- `--file <path>`: tail a file (otherwise reads stdin)
|
|
153
|
+
- `--format <auto|json|log4js>`
|
|
154
|
+
- `--config <path>`
|
|
155
|
+
- `--once`: read once and exit (useful for CI/scripting with `--file`)
|
|
156
|
+
- `--warnings <none|summary|all>`
|
|
157
|
+
- `--refresh <ms>`: minimum time between renders
|
|
158
|
+
- `--json`: newline-delimited JSON updates
|
|
159
|
+
|
|
160
|
+
Important: `tail` is a local developer tool, not a production monitor.
|
|
161
|
+
|
|
162
|
+
### 6.6 `export`
|
|
163
|
+
|
|
164
|
+
Export a manual trace run to local formats. **No upload**.
|
|
165
|
+
|
|
166
|
+
```bash
|
|
167
|
+
agent-inspect export <run-id> --format <markdown|html|openinference|otlp-json> [options]
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
Options:
|
|
171
|
+
|
|
172
|
+
- `--dir <path>`
|
|
173
|
+
- `--format <format>`
|
|
174
|
+
- `-o, --output <path>`: write file
|
|
175
|
+
- `--json`: JSON wrapper output (includes content if not writing to file)
|
|
176
|
+
- `--validate`: validate exported payload shape
|
|
177
|
+
- `--include-attributes`: include bounded attributes (review before sharing)
|
|
178
|
+
- `--no-metadata`: omit summary/metadata sections
|
|
179
|
+
- `--no-errors`: omit error sections
|
|
180
|
+
|
|
181
|
+
### 6.7 `diff`
|
|
182
|
+
|
|
183
|
+
Compare two manual trace runs. Diff is **local** and **read-only** (does not rerun agents).
|
|
184
|
+
|
|
185
|
+
```bash
|
|
186
|
+
agent-inspect diff <left-run-id> <right-run-id> [options]
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
Options:
|
|
190
|
+
|
|
191
|
+
- `--dir <path>`
|
|
192
|
+
- `--json`
|
|
193
|
+
- `--ignore-duration`
|
|
194
|
+
- `--duration-threshold <duration>`
|
|
195
|
+
- `--focus <all|errors|structure|outputs>`
|
|
196
|
+
- `--check <all|structure|outputs|errors|timing>`
|
|
197
|
+
|
|
198
|
+
## 7. Optional TUI behavior
|
|
199
|
+
|
|
200
|
+
`view --tui` delegates to `@agent-inspect/tui` and requires an interactive terminal. If the package is not installed, the CLI prints a short install hint.
|
|
201
|
+
|
|
202
|
+
## 8. Warnings behavior
|
|
203
|
+
|
|
204
|
+
Log parsing emits warnings for malformed lines or missing required keys. `--warnings` controls whether warnings are hidden, summarized, or printed line-by-line.
|
|
205
|
+
|
|
206
|
+
## 9. Limitations (reminder)
|
|
207
|
+
|
|
208
|
+
See:
|
|
209
|
+
|
|
210
|
+
- `docs/KNOWN-ISSUES.md`
|
|
211
|
+
- `docs/LIMITATIONS.md`
|
|
212
|
+
|
package/docs/COMPARE.md
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
## Compare AgentInspect
|
|
2
|
+
|
|
3
|
+
AgentInspect is a local-first execution-tree debugger for TypeScript AI agents. It’s designed for inner-loop debugging and quick inspection — not as a replacement for hosted observability, evaluation, or production monitoring platforms.
|
|
4
|
+
|
|
5
|
+
## AgentInspect vs console.log
|
|
6
|
+
|
|
7
|
+
- **console.log is flat**: logs are a stream of lines without run grouping or step boundaries.
|
|
8
|
+
- **AgentInspect adds structure**: runs, nested steps, step types (tool/LLM), durations, status summaries, and local trace files you can inspect later.
|
|
9
|
+
- **console.log still matters**: use it for quick values or ad-hoc debugging inside a step.
|
|
10
|
+
- **AgentInspect helps you understand flow**: especially when your agent fans out, retries, or has nested tool/LLM calls.
|
|
11
|
+
|
|
12
|
+
## AgentInspect vs LangSmith
|
|
13
|
+
|
|
14
|
+
LangSmith is a hosted/platform workflow for tracing, evaluation, and observability in the LangChain ecosystem.
|
|
15
|
+
|
|
16
|
+
AgentInspect is local-first CLI debugging:
|
|
17
|
+
|
|
18
|
+
- Use AgentInspect to debug locally before/alongside LangSmith when iterating on agent logic.
|
|
19
|
+
- AgentInspect does not provide hosted dashboards, dataset/eval workflows, or production tracing pipelines.
|
|
20
|
+
|
|
21
|
+
## AgentInspect vs Langfuse
|
|
22
|
+
|
|
23
|
+
Langfuse provides broader LLM observability (tracing, prompt management, datasets/evals, dashboards).
|
|
24
|
+
|
|
25
|
+
AgentInspect focuses on local execution trees and CLI workflows:
|
|
26
|
+
|
|
27
|
+
- Complementary: use AgentInspect for quick local run understanding; use Langfuse for dashboards and longer-lived observability workflows.
|
|
28
|
+
- Not a replacement.
|
|
29
|
+
|
|
30
|
+
## AgentInspect vs Braintrust
|
|
31
|
+
|
|
32
|
+
Braintrust is strong for evals, regressions, datasets, and production AI quality workflows.
|
|
33
|
+
|
|
34
|
+
AgentInspect is lighter and local-first:
|
|
35
|
+
|
|
36
|
+
- Use AgentInspect to understand a single run locally.
|
|
37
|
+
- Use Braintrust when you want repeatable evals, comparisons at scale, and production quality workflows.
|
|
38
|
+
|
|
39
|
+
## AgentInspect vs Phoenix / OpenInference
|
|
40
|
+
|
|
41
|
+
Phoenix and the OpenInference ecosystem are standards-oriented and useful for trace visualization and interoperability.
|
|
42
|
+
|
|
43
|
+
AgentInspect can export **OpenInference-compatible JSON** and **OTLP JSON** as local files:
|
|
44
|
+
|
|
45
|
+
- Exports are **compatibility-oriented** and should be validated against your target backend/collector.
|
|
46
|
+
- AgentInspect does not claim that every backend will accept these exports without configuration.
|
|
47
|
+
|
|
48
|
+
## AgentInspect vs OpenTelemetry setup
|
|
49
|
+
|
|
50
|
+
OpenTelemetry is powerful, but setup can be heavier (SDK configuration, exporters, collectors, backend).
|
|
51
|
+
|
|
52
|
+
AgentInspect avoids SDK/collector setup for local debugging:
|
|
53
|
+
|
|
54
|
+
- Use AgentInspect when you want quick, local execution trees with no collector required.
|
|
55
|
+
- Use OpenTelemetry when you need organization-wide production telemetry pipelines.
|
|
56
|
+
- Exports can help bridge later, but AgentInspect is not an OpenTelemetry SDK replacement.
|
|
57
|
+
|
|
58
|
+
## Quick decision table
|
|
59
|
+
|
|
60
|
+
| Need | AgentInspect fit |
|
|
61
|
+
| --- | --- |
|
|
62
|
+
| Local agent debugging | Strong fit |
|
|
63
|
+
| No-account CLI tracing | Strong fit |
|
|
64
|
+
| Production dashboards | Not the goal |
|
|
65
|
+
| Hosted eval datasets | Not the goal |
|
|
66
|
+
| Prompt management | Not the goal |
|
|
67
|
+
| Standards-aligned local export | Partial (compatibility-oriented) |
|
|
68
|
+
| Full observability platform | Use a dedicated platform |
|
|
69
|
+
|
package/docs/DIFF.md
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
## Diff / compare
|
|
2
|
+
|
|
3
|
+
AgentInspect can compare two runs (manual traces) locally and render a diff summary.
|
|
4
|
+
|
|
5
|
+
- **CLI usage**: `docs/CLI.md` (`diff`)
|
|
6
|
+
- **Schema expectations for diff inputs**: `docs/SCHEMA.md` (manual trace JSONL, `schemaVersion: "0.1"`)
|
|
7
|
+
|
|
8
|
+
Notes:
|
|
9
|
+
- Programmatic diff APIs are documented as **experimental** in `docs/API.md`.
|
|
10
|
+
- `diff` is read-only and does **not** rerun agents; “differences” do not necessarily imply failure (command errors do).
|
|
11
|
+
|
package/docs/EXPORTS.md
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
## Exports
|
|
2
|
+
|
|
3
|
+
AgentInspect supports local-only exports from run trees and traces. Exports are intended for **compatibility and sharing**, but should be reviewed before sending to others.
|
|
4
|
+
|
|
5
|
+
- **CLI usage**: `docs/CLI.md` (`export`)
|
|
6
|
+
- **Schema + compatibility guarantees**: `docs/SCHEMA.md`
|
|
7
|
+
- **Safety / redaction notes**: `docs/CLI.md`, `SECURITY.md`, and `docs-local/architecture/REDACTION.md`
|
|
8
|
+
|
|
9
|
+
Notes:
|
|
10
|
+
- Export formats (Markdown/HTML/OpenInference/OTLP JSON) are documented as **experimental / compatibility-oriented** in `docs/API.md`.
|
|
11
|
+
- AgentInspect does **not** upload anywhere; exports write local strings/files only.
|
|
12
|
+
|
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
# Getting started
|
|
2
|
+
|
|
3
|
+
AgentInspect is a **local-first execution-tree debugger** for TypeScript AI agents. It helps you produce and inspect an execution tree of steps, safely and deterministically, without uploading data anywhere.
|
|
4
|
+
|
|
5
|
+
## 1. Install
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
pnpm add agent-inspect
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
The `agent-inspect` package includes the CLI binary via its `bin` field:
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
npx agent-inspect --help
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
For local repo development (this monorepo), build and run the CLI from `packages/cli`:
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
pnpm build
|
|
21
|
+
node packages/cli/dist/index.cjs --help
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## 2. Basic manual trace
|
|
25
|
+
|
|
26
|
+
```ts
|
|
27
|
+
import { inspectRun, step } from "agent-inspect";
|
|
28
|
+
|
|
29
|
+
await inspectRun("demo-agent", async () => {
|
|
30
|
+
await step("plan", async () => "ok");
|
|
31
|
+
await step.tool("search", async () => ({ count: 2 }));
|
|
32
|
+
await step.llm("fixture-model", async () => "done");
|
|
33
|
+
});
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
This writes a local JSONL trace with stable v1.0 event names:
|
|
37
|
+
|
|
38
|
+
- `run_started`, `run_completed`
|
|
39
|
+
- `step_started`, `step_completed`
|
|
40
|
+
|
|
41
|
+
## 3. View runs
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
agent-inspect list
|
|
45
|
+
agent-inspect view <runId>
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
## 4. Clean old runs (safely)
|
|
49
|
+
|
|
50
|
+
Always start with `--dry-run`:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
agent-inspect clean --older-than 7d --dry-run
|
|
54
|
+
agent-inspect clean --older-than 7d --yes
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## 5. Parse existing logs
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
agent-inspect logs fixtures/logs/proactive-json.log \
|
|
61
|
+
--format json \
|
|
62
|
+
--config fixtures/configs/proactive-agent-inspect.logs.json
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## 6. Tail logs
|
|
66
|
+
|
|
67
|
+
For scripting/CI-style usage, `--once` reads and exits:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
agent-inspect tail \
|
|
71
|
+
--file fixtures/logs/proactive-json.log \
|
|
72
|
+
--format json \
|
|
73
|
+
--config fixtures/configs/proactive-agent-inspect.logs.json \
|
|
74
|
+
--once
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## 7. Export a run
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
agent-inspect export minimal-success --dir fixtures/traces --format markdown
|
|
81
|
+
agent-inspect export minimal-success --dir fixtures/traces --format openinference --validate
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Exports are **local-only** and do not upload anywhere.
|
|
85
|
+
|
|
86
|
+
## 8. Diff two runs
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
agent-inspect diff minimal-success minimal-error --dir fixtures/traces
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## 9. Try recipes
|
|
93
|
+
|
|
94
|
+
See `examples/recipes/README.md`.
|
|
95
|
+
|
|
96
|
+
## 10. Optional LangChain adapter
|
|
97
|
+
|
|
98
|
+
`@agent-inspect/langchain` is optional and **experimental**:
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
pnpm add @agent-inspect/langchain
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
## 11. Optional TUI
|
|
105
|
+
|
|
106
|
+
`@agent-inspect/tui` is optional and **experimental**. The CLI can invoke it with:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
agent-inspect view <runId> --tui
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
## 12. Safety notes
|
|
113
|
+
|
|
114
|
+
- Redaction is on by default for log-derived attributes and exports.
|
|
115
|
+
- Confidence labels are required to keep attribution honest.
|
|
116
|
+
- AgentInspect is for local debugging, not production monitoring.
|
|
117
|
+
|
|
118
|
+
## 13. Next docs
|
|
119
|
+
|
|
120
|
+
- `docs/API.md`
|
|
121
|
+
- `docs/CLI.md`
|
|
122
|
+
- `docs/SCHEMA.md`
|
|
123
|
+
- `docs/COMPARE.md`
|
|
124
|
+
- `docs/LOG-TO-TREE-QUICKSTART.md`
|
|
125
|
+
- `docs/KNOWN-ISSUES.md`
|
|
126
|
+
- `docs/LIMITATIONS.md`
|
|
127
|
+
- `docs-local/V1-READINESS-CHECKLIST.md`
|
|
128
|
+
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Known issues
|
|
2
|
+
|
|
3
|
+
AgentInspect is **local-first** and **CLI-first**. These behaviors are intentional constraints or best-effort areas—not silent guarantees.
|
|
4
|
+
|
|
5
|
+
## Logs
|
|
6
|
+
|
|
7
|
+
- **log4js-style parsing** is **best-effort**: embedded JSON must be recoverable from the line; unusual layouts may lose fields or warn.
|
|
8
|
+
- **JavaScript object-style log payloads** (e.g. `{ msg: '...', meta: ... }` printed without JSON) are **not** a supported interchange format for ingestion.
|
|
9
|
+
|
|
10
|
+
## Exports
|
|
11
|
+
|
|
12
|
+
- **OpenInference** and **OTLP JSON** exports are **compatibility-oriented** and **experimental**. Validate against your target collector or backend before relying on them.
|
|
13
|
+
- Exports generate **strings/files locally** only—there is **no** automatic upload.
|
|
14
|
+
|
|
15
|
+
## Integrations
|
|
16
|
+
|
|
17
|
+
- **Vendor sinks** (hosted dashboards, Langfuse/Braintrust/New Relic/Datadog native uploads, OTLP gRPC streaming, etc.) are **not implemented** in the core packages described here.
|
|
18
|
+
- **LangChain adapter** captures **metadata-oriented** signals by default; it does not replace full framework observability.
|
|
19
|
+
|
|
20
|
+
## UI / replay
|
|
21
|
+
|
|
22
|
+
- **Optional TUI** (`@agent-inspect/tui`) is separate from the default CLI and requires an interactive terminal where documented.
|
|
23
|
+
- **Live streaming tree inside the TUI** is not the same as **live tail** over logs; product scope varies by version—see roadmap docs.
|
|
24
|
+
- **Full replay / fork execution** of historical traces is **out of scope** for current MVP-style releases.
|
|
25
|
+
|
|
26
|
+
## Cost / tokens
|
|
27
|
+
|
|
28
|
+
- **Token counting** and **cost calculation** are **not** core features.
|
|
29
|
+
|
|
30
|
+
## Confidence labels
|
|
31
|
+
|
|
32
|
+
- Log-derived trees carry **confidence** (`explicit`, `correlated`, `heuristic`, `unknown`). Labels explain **how** relationships were inferred—they are not semantic proof.
|
|
33
|
+
|
|
34
|
+
When in doubt, treat AgentInspect as a **debugging aid**, not a compliance or billing system of record.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Limitations
|
|
2
|
+
|
|
3
|
+
This document states what AgentInspect **does not** provide today. It complements [KNOWN-ISSUES.md](./KNOWN-ISSUES.md).
|
|
4
|
+
|
|
5
|
+
## Product boundaries
|
|
6
|
+
|
|
7
|
+
- **No SaaS dashboard** or hosted multi-tenant product in the open-source core workflow.
|
|
8
|
+
- **No production APM replacement**: no sampling agents, no fleet-wide aggregation, no uptime SLAs.
|
|
9
|
+
- **No vendor upload pipeline**: no built-in Langfuse/Braintrust/New Relic/Datadog direct exporters as live sinks.
|
|
10
|
+
- **No automatic universal instrumentation** of every framework: integration is explicit (manual traces, log ingest, optional adapters).
|
|
11
|
+
|
|
12
|
+
## Data fidelity
|
|
13
|
+
|
|
14
|
+
- **No full prompt/output capture by default** for manual traces (by design: safety and PII risk).
|
|
15
|
+
- **Log-derived runs** and **manual JSONL traces** may differ in fidelity (timestamps, nesting, confidence).
|
|
16
|
+
- **Confidence labels** qualify inferred relationships; they do not guarantee correctness.
|
|
17
|
+
|
|
18
|
+
## Execution semantics
|
|
19
|
+
|
|
20
|
+
- **No replay / fork** of past runs from traces alone.
|
|
21
|
+
- **No time-travel debugging** across arbitrary runtime state.
|
|
22
|
+
- **No multi-run statistical evaluation** built into core.
|
|
23
|
+
|
|
24
|
+
## Economics
|
|
25
|
+
|
|
26
|
+
- **No cost engine**: no pricing tables, invoice-grade usage, or provider billing reconciliation.
|
|
27
|
+
|
|
28
|
+
## Scale
|
|
29
|
+
|
|
30
|
+
- Designed for **developer machines** and **inner-loop debugging**, not petabyte log warehouses.
|
|
31
|
+
|
|
32
|
+
For roadmap intent, see `docs-local/roadmap/VERSION-ROADMAP.md` and `docs-local/prd/v1.0-STABLE-LOCAL-INSPECTOR-PRD.md` (planning docs; not a promise of future features).
|