agent-inspect 6.7.0 → 6.7.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -10
- package/README.md +148 -93
- package/docs/ADOPTION.md +1 -1
- package/docs/BUNDLES.md +67 -0
- package/docs/COMPARE.md +3 -2
- package/docs/DEMO-SCRIPT.md +6 -5
- package/docs/FIRST-TRACE-IN-5-MINUTES.md +28 -10
- package/docs/GOLDEN-PATH.md +40 -0
- package/docs/INDEX.md +69 -0
- package/docs/NETWORK-BEHAVIOR.md +32 -0
- package/docs/PRE-V7-PILOT-KIT.md +45 -0
- package/docs/SCREENSHOTS.md +7 -6
- package/docs/SELF-HOSTING.md +130 -0
- package/docs/SESSIONS-AND-OUTCOMES.md +19 -0
- package/docs/STANDARDS.md +45 -0
- package/docs/SUITES-COHORTS-GATES.md +34 -0
- package/docs/SUPPORT-LEVELS.md +47 -0
- package/docs/TRACE-CONTRACTS.md +39 -0
- package/docs/assets/readme-product-loop.svg +27 -18
- package/package.json +20 -4
- package/packages/cli/dist/index.cjs +1 -1
- package/packages/cli/dist/index.cjs.map +1 -1
- package/packages/cli/dist/index.mjs +1 -1
- package/packages/cli/dist/index.mjs.map +1 -1
package/docs/INDEX.md
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Optional Local Index (`@agent-inspect/index-sqlite`)
|
|
2
|
+
|
|
3
|
+
The optional SQLite index accelerates local queries over large AgentInspect trace directories. It is **experimental** (added in v4.1) and entirely opt-in.
|
|
4
|
+
|
|
5
|
+
- **Source of truth stays JSONL.** The index is derived from your trace files and is always safe to delete.
|
|
6
|
+
- **Never mutates traces.** Building, rebuilding, and cleaning the index only touch the index database file.
|
|
7
|
+
- **Local-only.** No network access. No upload. No hosted service.
|
|
8
|
+
- **Not required.** Every core CLI command works without it; the index only speeds up queries.
|
|
9
|
+
|
|
10
|
+
## Install
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
npm install @agent-inspect/index-sqlite
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
The package depends on `better-sqlite3` (a native module). The core `agent-inspect` package has no SQLite dependency.
|
|
17
|
+
|
|
18
|
+
## CLI
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
agent-inspect index sqlite build # build/rebuild trace-index.sqlite
|
|
22
|
+
agent-inspect index sqlite status --json # health, counts, staleness
|
|
23
|
+
agent-inspect index sqlite query --status error --tool search
|
|
24
|
+
agent-inspect index sqlite clean # delete the index (traces untouched)
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
See [CLI.md § 6.23](CLI.md) for the full flag reference.
|
|
28
|
+
|
|
29
|
+
## Programmatic API
|
|
30
|
+
|
|
31
|
+
```ts
|
|
32
|
+
import {
|
|
33
|
+
buildIndex,
|
|
34
|
+
queryRuns,
|
|
35
|
+
indexStatus,
|
|
36
|
+
isIndexStale,
|
|
37
|
+
resolveIndexDbPath,
|
|
38
|
+
} from "@agent-inspect/index-sqlite";
|
|
39
|
+
|
|
40
|
+
const { dbPath, runs } = await buildIndex({ traceDir: ".agent-inspect/runs" });
|
|
41
|
+
|
|
42
|
+
const failed = queryRuns(dbPath, { status: "error", tool: "search", limit: 50 });
|
|
43
|
+
|
|
44
|
+
const status = indexStatus(dbPath); // { healthy, runs, steps, builtAt, ... }
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
All read functions are non-throwing: a missing or corrupt index returns empty
|
|
48
|
+
results (for `queryRuns`), `healthy: false` (for `indexStatus`), or `true` (for
|
|
49
|
+
`isIndexStale`), so callers can transparently fall back to a directory scan.
|
|
50
|
+
|
|
51
|
+
## Layout
|
|
52
|
+
|
|
53
|
+
The index database defaults to `<traceDir>/trace-index.sqlite`. It stores derived
|
|
54
|
+
`runs`, `steps`, `errors`, and `sessions` tables plus a `meta` table recording the
|
|
55
|
+
schema version, source directory, and build time.
|
|
56
|
+
|
|
57
|
+
## Staleness and recovery
|
|
58
|
+
|
|
59
|
+
- **Staleness:** an index is stale when any trace file is newer than the recorded
|
|
60
|
+
build time. `status` reports this; rebuild to refresh.
|
|
61
|
+
- **Corruption:** a failed integrity check is treated as "absent". Queries return
|
|
62
|
+
empty and a rebuild recreates the database from scratch.
|
|
63
|
+
|
|
64
|
+
## Boundaries
|
|
65
|
+
|
|
66
|
+
- No vector or semantic search.
|
|
67
|
+
- No background daemon.
|
|
68
|
+
- No SQLite dependency in root/core.
|
|
69
|
+
- No trace schema changes; deleting the index is always safe.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Network behavior
|
|
2
|
+
|
|
3
|
+
AgentInspect is **local-first**. Core tracing writes local files and does **not** upload to AgentInspect. Some optional surfaces can use the network when **explicitly** enabled.
|
|
4
|
+
|
|
5
|
+
## Summary
|
|
6
|
+
|
|
7
|
+
| Surface | Default | Network? | Notes |
|
|
8
|
+
| ------- | ------- | -------- | ----- |
|
|
9
|
+
| Core CLI / writers | On | No | Local JSONL only |
|
|
10
|
+
| Framework adapters | Opt-in install | No AgentInspect upload | Provider SDKs may still call their own APIs |
|
|
11
|
+
| `explain` | Local facts | No provider calls in default mode | |
|
|
12
|
+
| Viewer | Localhost | Loopback only | Read-only |
|
|
13
|
+
| Studio CLI | Localhost bind | Loopback by default | Customer-owned |
|
|
14
|
+
| Studio file-drop ingest | Off / explicit | No | Local files |
|
|
15
|
+
| Studio GitHub artifact import | Off / explicit | Yes → GitHub | User credentials; customer-owned |
|
|
16
|
+
| Studio HTTP ingest | **Disabled by default** | Yes if enabled | Token + binding required |
|
|
17
|
+
| MCP client (`@agent-inspect/mcp`) | Opt-in | To your MCP servers | Tracing only; not a gateway |
|
|
18
|
+
| MCP server | Opt-in | Exposes local evidence to connected client | Preview; share-profile boundary |
|
|
19
|
+
| Standards export / collector | Opt-in | Only if you configure export | Known losses documented |
|
|
20
|
+
| Website / docs | N/A | Public site | No trace upload |
|
|
21
|
+
|
|
22
|
+
## Rules of thumb
|
|
23
|
+
|
|
24
|
+
1. **No default upload** of traces to AgentInspect maintainers.
|
|
25
|
+
2. **No hidden telemetry** from the library.
|
|
26
|
+
3. **Customer-owned** destinations only (your disk, your Studio, your collector).
|
|
27
|
+
4. Prefer **redact / verify-safe** before any share or ingest path.
|
|
28
|
+
5. Redaction is best-effort, not certification.
|
|
29
|
+
|
|
30
|
+
## Enabling Studio ingest
|
|
31
|
+
|
|
32
|
+
See [SELF-HOSTING.md](./SELF-HOSTING.md) and [`@agent-inspect/studio`](https://github.com/rajudandigam/agent-inspect/tree/main/packages/studio). HTTP ingest remains off until you configure token and binding.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Pre-v7 Pilot Kit
|
|
2
|
+
|
|
3
|
+
**Purpose:** External design-partner and three-team pilot evidence for the 6.7.x technical launch candidate.
|
|
4
|
+
**Do not fabricate results.** Record only real partner outcomes in [implementation/PRE-V7-ADOPTION-EVIDENCE.md](implementation/PRE-V7-ADOPTION-EVIDENCE.md).
|
|
5
|
+
|
|
6
|
+
## What is shipping for partners
|
|
7
|
+
|
|
8
|
+
| Item | Version / path |
|
|
9
|
+
|------|----------------|
|
|
10
|
+
| npm | `agent-inspect@6.7.1` (and fixed-group packages) |
|
|
11
|
+
| Quickstart | `npx agent-inspect init --yes` → demo → `list` → `verify-safe` |
|
|
12
|
+
| Packed E2E (maintainers) | `pnpm run pack:smoke` |
|
|
13
|
+
| Demo script | [DEMO-SCRIPT.md](DEMO-SCRIPT.md) |
|
|
14
|
+
| Design partner guide | [DESIGN-PARTNER-GUIDE.md](DESIGN-PARTNER-GUIDE.md) |
|
|
15
|
+
| Broken-agent starter | `examples/starters/broken-agent-debugging` |
|
|
16
|
+
| Studio (Beta) | `@agent-inspect/studio` — customer-owned, local |
|
|
17
|
+
|
|
18
|
+
## Partner trial checklist (copy per team)
|
|
19
|
+
|
|
20
|
+
1. Install `agent-inspect@6.7.1` on Node ≥ 20.
|
|
21
|
+
2. Complete five-minute quickstart (init → one run → verify-safe).
|
|
22
|
+
3. Run at least one framework path (AI SDK, OpenAI Agents, or LangChain) **or** observe/manual path.
|
|
23
|
+
4. Optionally run Studio against a local workspace.
|
|
24
|
+
5. Optionally retain a CI check/suite gate on a PR.
|
|
25
|
+
6. Return dated findings: blockers, what worked, whether they will keep using it.
|
|
26
|
+
|
|
27
|
+
## Evidence required before distinct 6.8.0
|
|
28
|
+
|
|
29
|
+
From the canonical roadmap:
|
|
30
|
+
|
|
31
|
+
- One design-partner Studio trial (sign-off row)
|
|
32
|
+
- Three **external** teams with golden-path trial
|
|
33
|
+
- At least one retained CI contract/gate per pilot set
|
|
34
|
+
- At least one Studio trial in the pilot set
|
|
35
|
+
- Dated findings — no internal-only rows
|
|
36
|
+
|
|
37
|
+
## Maintainer stop condition
|
|
38
|
+
|
|
39
|
+
When this kit is prepared and 6.7.1 is on npm:
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
trainStatus: blocked-on-external-pilot
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
Do **not** schedule or implement v7 until adoption gates in [implementation/release-trains/V7.0.0-READINESS-ASSESSMENT.md](implementation/release-trains/V7.0.0-READINESS-ASSESSMENT.md) are met and a maintainer explicitly authorizes a v7 train.
|
package/docs/SCREENSHOTS.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Visual demos
|
|
2
2
|
|
|
3
|
-
Curated terminal recordings and static diagrams for AgentInspect **
|
|
3
|
+
Curated terminal recordings and static diagrams for AgentInspect **6.7.x**. They show local capture → inspect → enforce → verify/bundle — without a maintainer-hosted dashboard or vendor upload.
|
|
4
4
|
|
|
5
5
|
**Synthetic output only:** demos use committed [fixtures](../fixtures/README.md), [examples](../examples/README.md), and recipes. No external LLM calls or API keys.
|
|
6
6
|
|
|
@@ -12,21 +12,21 @@ Curated terminal recordings and static diagrams for AgentInspect **3.5.x**. They
|
|
|
12
12
|
|
|
13
13
|
---
|
|
14
14
|
|
|
15
|
-
## README brand assets (
|
|
15
|
+
## README brand assets (6.7.x)
|
|
16
16
|
|
|
17
17
|
| Asset | Use |
|
|
18
18
|
| ----- | --- |
|
|
19
19
|
| [agent-inspect-logo.svg](assets/agent-inspect-logo.svg) | Light-mode wordmark |
|
|
20
20
|
| [agent-inspect-logo-dark.svg](assets/agent-inspect-logo-dark.svg) | Dark-mode wordmark |
|
|
21
|
-
| [readme-product-loop.svg](assets/readme-product-loop.svg) | Capture →
|
|
21
|
+
| [readme-product-loop.svg](assets/readme-product-loop.svg) | Capture → understand → enforce → verify/bundle → local/Studio |
|
|
22
22
|
|
|
23
23
|
Root README embeds the logo and product-loop SVG only. Terminal GIFs stay below.
|
|
24
24
|
|
|
25
25
|
---
|
|
26
26
|
|
|
27
|
-
## Adoption demo (
|
|
27
|
+
## Adoption demo (6.7.x)
|
|
28
28
|
|
|
29
|
-
Deterministic broken-agent flow — recommended for live demos
|
|
29
|
+
Deterministic broken-agent flow — recommended for live demos.
|
|
30
30
|
|
|
31
31
|
**Run:**
|
|
32
32
|
|
|
@@ -34,7 +34,8 @@ Deterministic broken-agent flow — recommended for live demos and Show HN rehea
|
|
|
34
34
|
cd examples/starters/broken-agent-debugging
|
|
35
35
|
pnpm install && pnpm start
|
|
36
36
|
npx agent-inspect report <run-id> --dir .agent-inspect
|
|
37
|
-
npx agent-inspect redact .agent-inspect
|
|
37
|
+
npx agent-inspect redact <run-id> --dir .agent-inspect --profile share -o safe.jsonl
|
|
38
|
+
npx agent-inspect verify-safe <run-id> --dir .agent-inspect
|
|
38
39
|
```
|
|
39
40
|
|
|
40
41
|
**GIF status:** re-record optional; use [error-handling.gif](assets/demos/error-handling.gif) as b-roll until a dedicated broken-agent GIF exists.
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
# Self-hosting AgentInspect Studio
|
|
2
|
+
|
|
3
|
+
AgentInspect Studio is a **customer-owned**, **read-only** analyzer for multi-project workspaces. It is not a maintainer-hosted service.
|
|
4
|
+
|
|
5
|
+
## Install
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
npm install agent-inspect @agent-inspect/studio
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
## Quick start (localhost)
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
npx agent-inspect studio --workspace ./studio-registry.json
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Defaults:
|
|
18
|
+
|
|
19
|
+
- bind: `127.0.0.1:7340`
|
|
20
|
+
- database: `./.agent-inspect/studio.db` (SQLite, disposable cache)
|
|
21
|
+
- JSONL + `workspace.json` remain canonical
|
|
22
|
+
|
|
23
|
+
## Studio registry
|
|
24
|
+
|
|
25
|
+
Create `studio-registry.json` beside your team workspace root:
|
|
26
|
+
|
|
27
|
+
```json
|
|
28
|
+
{
|
|
29
|
+
"schemaVersion": "1.0",
|
|
30
|
+
"name": "platform-team",
|
|
31
|
+
"projects": [
|
|
32
|
+
{ "id": "support-agent", "path": "/path/to/project", "label": "Support Agent" }
|
|
33
|
+
],
|
|
34
|
+
"import": {
|
|
35
|
+
"ciArtifactsDir": "./imports/ci",
|
|
36
|
+
"bundlesDir": "./imports/bundles"
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Each `projects[].path` must contain `.agent-inspect/workspace.json` (v4 layout).
|
|
42
|
+
|
|
43
|
+
Optional **file-drop ingest** (v6.1+, disabled by default):
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
npx agent-inspect studio import drop --workspace ./studio-registry.json
|
|
47
|
+
npx agent-inspect studio --ingest file-drop --workspace ./studio-registry.json
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Registry `import.fileDropDir` points at the watched folder; allowlisted files (`.jsonl`, `.suite.json`, `.tgz`, `.zip`) copy into `import.ciArtifactsDir` / `import.bundlesDir` with idempotent SQLite bookkeeping. Ingest stays off until you pass `--ingest file-drop` or run `studio import drop`.
|
|
51
|
+
|
|
52
|
+
**GitHub Actions artifacts** (v6.1+, operator-initiated pull only):
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
export GITHUB_TOKEN=... # actions:read on your repo
|
|
56
|
+
npx agent-inspect studio import github \
|
|
57
|
+
--repo owner/name \
|
|
58
|
+
--run-id 123456789 \
|
|
59
|
+
--artifact ci-artifacts \
|
|
60
|
+
--workspace ./studio-registry.json
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Downloads the artifact zip into `import.bundlesDir`, records idempotent ingest bookkeeping, and refreshes the studio project index. No maintainer GitHub App or AgentInspect proxy — CI tests use checked-in fixture archives only.
|
|
64
|
+
|
|
65
|
+
**Manual bundle upload** (v6.1+):
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
npx agent-inspect bundle <run-id> --profile share --out ./bundle-out
|
|
69
|
+
npx agent-inspect studio import bundle --path ./bundle-out --workspace ./studio-registry.json
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Validates `metadata.json` from a local share-safe bundle directory before copying into `import.bundlesDir`.
|
|
73
|
+
|
|
74
|
+
**HTTP ingest** (v6.1+, disabled by default):
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
export STUDIO_INGEST_TOKEN=$(openssl rand -hex 32)
|
|
78
|
+
npx agent-inspect studio --ingest http --ingest-token-env STUDIO_INGEST_TOKEN
|
|
79
|
+
# POST /api/ingest/bundle or /api/ingest/artifact with Authorization: Bearer $STUDIO_INGEST_TOKEN
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
HTTP ingest stays off until `--ingest http` or `ingest.http.enabled: true` in the registry. Token required on every POST.
|
|
83
|
+
|
|
84
|
+
## Ingestion security model (v6.1)
|
|
85
|
+
|
|
86
|
+
| Control | Behavior |
|
|
87
|
+
| ------- | -------- |
|
|
88
|
+
| Default | All ingest channels off |
|
|
89
|
+
| File-drop | Explicit CLI or `--ingest file-drop` only |
|
|
90
|
+
| GitHub | Operator-initiated pull with their token only |
|
|
91
|
+
| HTTP POST | Explicit enable + `STUDIO_INGEST_TOKEN` (constant-time validation) |
|
|
92
|
+
| Body size | Bounded (default 50MB) |
|
|
93
|
+
| Paths | Traversal guards; server chooses dest under registry `import.*` |
|
|
94
|
+
| Secrets | Tokens never logged; safe error messages only |
|
|
95
|
+
| Network | Studio never phones home; no maintainer upload target |
|
|
96
|
+
| Data | Imported files are read-only evidence; JSONL canonical |
|
|
97
|
+
|
|
98
|
+
Combine `--auth basic` when binding beyond localhost. Review imported artifacts with `scan` / `verify-safe` before sharing outside your network.
|
|
99
|
+
|
|
100
|
+
## Network exposure
|
|
101
|
+
|
|
102
|
+
- **Default:** localhost only.
|
|
103
|
+
- **`--server`:** binds `0.0.0.0` with an explicit startup warning.
|
|
104
|
+
- **`--auth basic --password-env STUDIO_PASSWORD`:** optional HTTP Basic auth (recommended for non-localhost).
|
|
105
|
+
|
|
106
|
+
Studio performs **no default upload**. Read routes are GET-only; optional **ingest POST routes** exist only when HTTP ingest is explicitly enabled (v6.1+).
|
|
107
|
+
|
|
108
|
+
## API surface (read-only)
|
|
109
|
+
|
|
110
|
+
| Route | Purpose |
|
|
111
|
+
| ----- | ------- |
|
|
112
|
+
| `GET /api/health` | Studio status |
|
|
113
|
+
| `GET /api/projects` | Registered projects |
|
|
114
|
+
| `GET /api/projects/:id/runs` | Run list |
|
|
115
|
+
| `GET /api/projects/:id/sessions` | Session index |
|
|
116
|
+
| `GET /api/projects/:id/suites` | Suite results |
|
|
117
|
+
| `GET /api/projects/:id/checks` | Check summaries |
|
|
118
|
+
| `GET /api/search?projectId=&q=` | Metadata search |
|
|
119
|
+
| `GET /api/diff?projectId=&left=&right=` | Regression diff |
|
|
120
|
+
| `GET /api/bundles/export?projectId=&runId=` | Bundle export hints (CLI assembly) |
|
|
121
|
+
|
|
122
|
+
## Postgres
|
|
123
|
+
|
|
124
|
+
Postgres URLs are reserved for team deployments and are **not required** for local use. v6.0 ships SQLite by default in `@agent-inspect/studio` only — never in root/core.
|
|
125
|
+
|
|
126
|
+
## Related docs
|
|
127
|
+
|
|
128
|
+
- [SELF-HOSTED-STUDIO-V6.0.md](./proposals/SELF-HOSTED-STUDIO-V6.0.md)
|
|
129
|
+
- [CLIENT-HOSTED-INGESTION-V6.1.md](./proposals/CLIENT-HOSTED-INGESTION-V6.1.md)
|
|
130
|
+
- [LOCAL-TRACE-WORKSPACE.md](./proposals/LOCAL-TRACE-WORKSPACE.md)
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# Sessions and observed outcomes
|
|
2
|
+
|
|
3
|
+
## Sessions
|
|
4
|
+
|
|
5
|
+
Workflow sessions group related runs (retries, handoffs, multi-agent activity) using **explicit metadata** — AgentInspect does not invent causal links from timestamps alone.
|
|
6
|
+
|
|
7
|
+
Useful CLI entry points: `sessions`, `search`, activity views (see [CLI.md](./CLI.md)).
|
|
8
|
+
|
|
9
|
+
## Observed outcomes
|
|
10
|
+
|
|
11
|
+
Outcomes record what the agent produced or decided at a high level for later review and gates. They remain local JSONL-derived evidence.
|
|
12
|
+
|
|
13
|
+
## Limitations
|
|
14
|
+
|
|
15
|
+
- Session indexing is not a full workflow contract engine
|
|
16
|
+
- Handoff / approval TraceContract rules are not fully wired — see [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md)
|
|
17
|
+
- Studio session pages may still be thinner than APIs — Studio is Beta
|
|
18
|
+
|
|
19
|
+
Related: [WORKSPACE.md](./WORKSPACE.md) · [USE-CASES.md](./USE-CASES.md)
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Standards interop (v6.4+)
|
|
2
|
+
|
|
3
|
+
AgentInspect persists **AgentInspect JSONL** locally. Standards exports are **compatibility copies** for review and optional import — not a replacement persisted schema.
|
|
4
|
+
|
|
5
|
+
## OpenInference (experimental)
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
npx agent-inspect export <run-id> --format openinference --profile share
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
Validate fixtures:
|
|
12
|
+
|
|
13
|
+
```ts
|
|
14
|
+
import {
|
|
15
|
+
validateOpenInferenceFixture,
|
|
16
|
+
validateOpenInferenceSemanticFixture,
|
|
17
|
+
} from "agent-inspect/exporters";
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Shape validation is **compatible**; semantic checks add field-level warnings for tested fixtures only.
|
|
21
|
+
|
|
22
|
+
Fixture: [fixtures/standards/openinference-basic.json](../fixtures/standards/openinference-basic.json)
|
|
23
|
+
|
|
24
|
+
## OTLP JSON (experimental)
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
npx agent-inspect export <run-id> --format otlp-json --profile share
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
GenAI attribute mapping follows `OTEL_GEN_AI_SEMCONV_PIN` (see exporters API). No gRPC collector included.
|
|
31
|
+
|
|
32
|
+
Fixture: [fixtures/standards/otlp-basic.json](../fixtures/standards/otlp-basic.json)
|
|
33
|
+
|
|
34
|
+
## Import recipes
|
|
35
|
+
|
|
36
|
+
- [Phoenix / OpenInference](../examples/recipes/phoenix-openinference-import/)
|
|
37
|
+
- [Langfuse self-hosted](../examples/recipes/langfuse-local-import/)
|
|
38
|
+
|
|
39
|
+
## Vendor graduation (manual)
|
|
40
|
+
|
|
41
|
+
- [New Relic](./vendors/NEW-RELIC.md)
|
|
42
|
+
- [Datadog](./vendors/DATADOG.md)
|
|
43
|
+
- [Honeycomb](./vendors/HONEYCOMB.md)
|
|
44
|
+
|
|
45
|
+
Review redacted exports before sharing. See [SAFE-TRACE-SHARING.md](./SAFE-TRACE-SHARING.md).
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Suites, cohorts, and gates
|
|
2
|
+
|
|
3
|
+
**Support level:** Beta
|
|
4
|
+
|
|
5
|
+
Deterministic regression tooling over local traces and workspaces.
|
|
6
|
+
|
|
7
|
+
## Suites
|
|
8
|
+
|
|
9
|
+
- Validate expectations against **existing** traces (`suite validate` / config-driven checks)
|
|
10
|
+
- Aggregate status must not pass when all cases are missing evidence (all-skipped semantics)
|
|
11
|
+
- Executable “fresh run” modes exist only where harness/suite run is implemented — do not assume every template executes live agents
|
|
12
|
+
|
|
13
|
+
## Cohorts
|
|
14
|
+
|
|
15
|
+
- Baseline vs candidate comparison with tolerances
|
|
16
|
+
- Sample counts, missing values, and insufficient-evidence / not-comparable states
|
|
17
|
+
- Not every positive delta is a regression
|
|
18
|
+
|
|
19
|
+
## Gates
|
|
20
|
+
|
|
21
|
+
- CI-oriented thresholds (error rate, duration, etc.)
|
|
22
|
+
- Invalid ranges (negative / NaN / out-of-range percentages) are rejected
|
|
23
|
+
- Exit codes are intended for CI fail-closed use
|
|
24
|
+
|
|
25
|
+
## Artifacts
|
|
26
|
+
|
|
27
|
+
Suites/gates can emit local reports and CI summaries. Prefer share profiles before attaching to PRs.
|
|
28
|
+
|
|
29
|
+
## Limitations
|
|
30
|
+
|
|
31
|
+
- Cross-platform consumer matrix rows may still be pending in adoption evidence
|
|
32
|
+
- Full golden-path automation (broken → contract → suite → Studio) is only partially covered by scripts — see [GOLDEN-PATH.md](./GOLDEN-PATH.md)
|
|
33
|
+
|
|
34
|
+
Related: [CI-ARTIFACTS.md](./CI-ARTIFACTS.md) · [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md) · [CLI.md](./CLI.md)
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Support levels
|
|
2
|
+
|
|
3
|
+
Canonical maturity labels for AgentInspect public packages and major surfaces (6.7.x).
|
|
4
|
+
|
|
5
|
+
## Definitions
|
|
6
|
+
|
|
7
|
+
| Level | Meaning |
|
|
8
|
+
| ----- | ------- |
|
|
9
|
+
| **Stable** | Core contracts intended for long-term use; breaking changes require a major version |
|
|
10
|
+
| **Supported** | Officially maintained; API may evolve with minors; documented and tested |
|
|
11
|
+
| **Beta** | Useful and tested; API or UX may change; known limitations disclosed |
|
|
12
|
+
| **Preview** | Early surface; expect gaps; not for production-critical workflows alone |
|
|
13
|
+
| **Experimental** | Research / extension; may be removed or redesigned |
|
|
14
|
+
|
|
15
|
+
## Package matrix (fixed release line)
|
|
16
|
+
|
|
17
|
+
| Package / surface | Level |
|
|
18
|
+
| ----------------- | ----- |
|
|
19
|
+
| `agent-inspect` core schema, readers, writers, inspection CLI | Stable |
|
|
20
|
+
| Redaction engine / `@agent-inspect/redact` | Stable |
|
|
21
|
+
| Deterministic checks (`agent-inspect/checks`) | Stable |
|
|
22
|
+
| Official adapters (ai-sdk, openai-agents, langchain) | Supported |
|
|
23
|
+
| Vitest / Jest reporters | Supported |
|
|
24
|
+
| `@agent-inspect/harness` | Supported |
|
|
25
|
+
| Workspace / bundles / observed outcomes | Supported |
|
|
26
|
+
| TraceContract API | Beta |
|
|
27
|
+
| Suites / cohorts / gates | Beta |
|
|
28
|
+
| `@agent-inspect/index-sqlite` | Beta |
|
|
29
|
+
| `@agent-inspect/viewer` | Beta |
|
|
30
|
+
| `@agent-inspect/adapter-sdk` / plugins | Beta |
|
|
31
|
+
| `@agent-inspect/studio` | Beta |
|
|
32
|
+
| Studio HTTP / GitHub ingest | Preview |
|
|
33
|
+
| `@agent-inspect/mcp-server` | Preview |
|
|
34
|
+
| Standards round-trip / Collector–Phoenix external proof | Preview |
|
|
35
|
+
| Vitest/Jest TraceContract matchers | **Not shipped** |
|
|
36
|
+
|
|
37
|
+
Part of the fixed AgentInspect release line — see the npm badge for the current version.
|
|
38
|
+
|
|
39
|
+
## Compatibility promise
|
|
40
|
+
|
|
41
|
+
- Persisted schema **1.0**; v0.1 / v0.2 / 1.0 traces remain readable
|
|
42
|
+
- Optional packages do not add root/core runtime dependencies
|
|
43
|
+
- Network behavior is explicit (see [NETWORK-BEHAVIOR.md](./NETWORK-BEHAVIOR.md))
|
|
44
|
+
|
|
45
|
+
## Promotion criteria
|
|
46
|
+
|
|
47
|
+
A surface moves up only with tests, docs, packed smoke where relevant, and honest limitation disclosure — not changelog marketing alone.
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# Trace contracts
|
|
2
|
+
|
|
3
|
+
**Support level:** Beta
|
|
4
|
+
|
|
5
|
+
Typed trajectory expectations over local AgentInspect traces via `defineTraceContract` / `evaluateTraceContract` (`agent-inspect/checks`).
|
|
6
|
+
|
|
7
|
+
## What is shipped
|
|
8
|
+
|
|
9
|
+
Contracts compile to deterministic check rules for common cases:
|
|
10
|
+
|
|
11
|
+
- run status / completion / max duration
|
|
12
|
+
- tool required / forbidden / allowed / maxCalls / order
|
|
13
|
+
- LLM maxCalls / maxTotalTokens / allowedModels
|
|
14
|
+
- evidence-bearing findings on failures
|
|
15
|
+
|
|
16
|
+
See [API.md](./API.md) and `packages/core/src/checks/contract.ts`.
|
|
17
|
+
|
|
18
|
+
## What is not shipped (yet)
|
|
19
|
+
|
|
20
|
+
Do **not** document these as available:
|
|
21
|
+
|
|
22
|
+
- Vitest / Jest `expectTrace(...).toSatisfyTraceContract` matchers
|
|
23
|
+
- Full workflow handoff / approval / MCP protocol contract rules
|
|
24
|
+
- Per-tool argument schema / regex validators on the contract surface
|
|
25
|
+
- Every structure rule (orphan/cycle/depth) exposed on the contract API (many exist as standalone check rules)
|
|
26
|
+
|
|
27
|
+
## CLI relationship
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
npx agent-inspect check <run-id> --dir .agent-inspect
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Suites and gates can consume check results; see [SUITES-COHORTS-GATES.md](./SUITES-COHORTS-GATES.md).
|
|
34
|
+
|
|
35
|
+
## Limitations
|
|
36
|
+
|
|
37
|
+
- Experimental/Beta API — may evolve in minors
|
|
38
|
+
- Contract tests are smoke-level; prefer check-engine tests for deep rule coverage
|
|
39
|
+
- Always review findings before treating a green check as product proof
|
|
@@ -1,26 +1,35 @@
|
|
|
1
|
-
<svg xmlns="http://www.w3.org/2000/svg" width="
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="150" viewBox="0 0 900 150" role="img" aria-label="AgentInspect evidence loop: capture or import, understand causality, enforce expectations, verify and bundle, review locally or in customer-owned Studio">
|
|
2
2
|
<defs>
|
|
3
3
|
<marker id="arr" markerWidth="8" markerHeight="8" refX="6" refY="3" orient="auto">
|
|
4
4
|
<path d="M0,0 L6,3 L0,6 Z" fill="#64748b"/>
|
|
5
5
|
</marker>
|
|
6
6
|
</defs>
|
|
7
|
-
<rect width="
|
|
7
|
+
<rect width="900" height="150" fill="#ffffff"/>
|
|
8
8
|
<g font-family="ui-sans-serif, system-ui, -apple-system, Segoe UI, sans-serif">
|
|
9
|
-
<rect x="
|
|
10
|
-
<text x="
|
|
11
|
-
<text x="
|
|
12
|
-
<
|
|
13
|
-
<
|
|
14
|
-
<
|
|
15
|
-
<text x="
|
|
16
|
-
<
|
|
17
|
-
<
|
|
18
|
-
<
|
|
19
|
-
<
|
|
20
|
-
<
|
|
21
|
-
<
|
|
22
|
-
<text x="
|
|
23
|
-
<
|
|
24
|
-
<
|
|
9
|
+
<rect x="10" y="24" width="150" height="78" rx="12" fill="#eef2ff" stroke="#6366f1" stroke-width="1.5"/>
|
|
10
|
+
<text x="85" y="52" text-anchor="middle" font-size="13" font-weight="600" fill="#312e81">Capture / import</text>
|
|
11
|
+
<text x="85" y="72" text-anchor="middle" font-size="10" fill="#475569">JSONL, adapters,</text>
|
|
12
|
+
<text x="85" y="86" text-anchor="middle" font-size="10" fill="#475569">logs, standards</text>
|
|
13
|
+
<path d="M168 63h22" stroke="#64748b" stroke-width="2" marker-end="url(#arr)"/>
|
|
14
|
+
<rect x="198" y="24" width="150" height="78" rx="12" fill="#ecfdf5" stroke="#10b981" stroke-width="1.5"/>
|
|
15
|
+
<text x="273" y="52" text-anchor="middle" font-size="13" font-weight="600" fill="#064e3b">Understand</text>
|
|
16
|
+
<text x="273" y="72" text-anchor="middle" font-size="10" fill="#475569">tree, timeline,</text>
|
|
17
|
+
<text x="273" y="86" text-anchor="middle" font-size="10" fill="#475569">report, diff</text>
|
|
18
|
+
<path d="M356 63h22" stroke="#64748b" stroke-width="2" marker-end="url(#arr)"/>
|
|
19
|
+
<rect x="386" y="24" width="150" height="78" rx="12" fill="#fff7ed" stroke="#f59e0b" stroke-width="1.5"/>
|
|
20
|
+
<text x="461" y="52" text-anchor="middle" font-size="13" font-weight="600" fill="#78350f">Enforce</text>
|
|
21
|
+
<text x="461" y="72" text-anchor="middle" font-size="10" fill="#475569">checks, contracts,</text>
|
|
22
|
+
<text x="461" y="86" text-anchor="middle" font-size="10" fill="#475569">suites, CI gates</text>
|
|
23
|
+
<path d="M544 63h22" stroke="#64748b" stroke-width="2" marker-end="url(#arr)"/>
|
|
24
|
+
<rect x="574" y="24" width="150" height="78" rx="12" fill="#f8fafc" stroke="#64748b" stroke-width="1.5"/>
|
|
25
|
+
<text x="649" y="52" text-anchor="middle" font-size="13" font-weight="600" fill="#0f172a">Verify / bundle</text>
|
|
26
|
+
<text x="649" y="72" text-anchor="middle" font-size="10" fill="#475569">redact, verify-safe,</text>
|
|
27
|
+
<text x="649" y="86" text-anchor="middle" font-size="10" fill="#475569">offline bundle</text>
|
|
28
|
+
<path d="M732 63h22" stroke="#64748b" stroke-width="2" marker-end="url(#arr)"/>
|
|
29
|
+
<rect x="762" y="24" width="128" height="78" rx="12" fill="#f5f3ff" stroke="#7c3aed" stroke-width="1.5"/>
|
|
30
|
+
<text x="826" y="52" text-anchor="middle" font-size="13" font-weight="600" fill="#4c1d95">Review</text>
|
|
31
|
+
<text x="826" y="72" text-anchor="middle" font-size="10" fill="#475569">local or Studio</text>
|
|
32
|
+
<text x="826" y="86" text-anchor="middle" font-size="10" fill="#475569">Beta (yours)</text>
|
|
33
|
+
<text x="450" y="132" text-anchor="middle" font-size="12" fill="#64748b">No account · no default upload · metadata-only by default · optional customer-owned Studio</text>
|
|
25
34
|
</g>
|
|
26
35
|
</svg>
|
package/package.json
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-inspect",
|
|
3
|
-
"version": "6.7.
|
|
3
|
+
"version": "6.7.2",
|
|
4
4
|
"license": "MIT",
|
|
5
5
|
"type": "module",
|
|
6
|
-
"description": "
|
|
6
|
+
"description": "Debug, regression-test, and safely share TypeScript AI-agent behavior locally — no account, no default upload, metadata-only by default",
|
|
7
7
|
"repository": {
|
|
8
8
|
"type": "git",
|
|
9
9
|
"url": "git+https://github.com/rajudandigam/agent-inspect.git"
|
|
@@ -164,6 +164,17 @@
|
|
|
164
164
|
"docs/DEMO-SCRIPT.md",
|
|
165
165
|
"docs/TECHNICAL-GUIDE.md",
|
|
166
166
|
"docs/WORKSPACE.md",
|
|
167
|
+
"docs/SELF-HOSTING.md",
|
|
168
|
+
"docs/STANDARDS.md",
|
|
169
|
+
"docs/BUNDLES.md",
|
|
170
|
+
"docs/INDEX.md",
|
|
171
|
+
"docs/SUPPORT-LEVELS.md",
|
|
172
|
+
"docs/NETWORK-BEHAVIOR.md",
|
|
173
|
+
"docs/TRACE-CONTRACTS.md",
|
|
174
|
+
"docs/SUITES-COHORTS-GATES.md",
|
|
175
|
+
"docs/SESSIONS-AND-OUTCOMES.md",
|
|
176
|
+
"docs/GOLDEN-PATH.md",
|
|
177
|
+
"docs/PRE-V7-PILOT-KIT.md",
|
|
167
178
|
"docs/AI-SDK-ADOPTION.md",
|
|
168
179
|
"docs/OPENAI-AGENTS-LOCAL.md",
|
|
169
180
|
"docs/NESTJS.md",
|
|
@@ -212,10 +223,15 @@
|
|
|
212
223
|
"test:watch": "vitest",
|
|
213
224
|
"test:coverage": "vitest run --coverage",
|
|
214
225
|
"size": "size-limit --config size-limit.config.mjs",
|
|
215
|
-
"test:all": "pnpm run typecheck && pnpm run test && pnpm run build && pnpm run size",
|
|
216
|
-
"prepublish:checks": "pnpm run typecheck && pnpm run test && pnpm run test:coverage && pnpm run build && pnpm run fixtures:check && pnpm run recipes:check && pnpm run size && pnpm run pack:smoke",
|
|
226
|
+
"test:all": "pnpm run typecheck && pnpm run linked-versions:check && pnpm run test && pnpm run build && pnpm run size",
|
|
227
|
+
"prepublish:checks": "pnpm run typecheck && pnpm run test && pnpm run test:coverage && pnpm run build && pnpm run fixtures:check && pnpm run recipes:check && pnpm run size && pnpm run linked-versions:check && pnpm run pack:smoke",
|
|
217
228
|
"pack:dry-run": "pnpm run build && npm pack --dry-run",
|
|
218
229
|
"pack:smoke": "pnpm run build && node scripts/package-smoke.mjs && node scripts/packed-quickstart-e2e.mjs",
|
|
230
|
+
"linked-versions:check": "node scripts/check-linked-versions.mjs",
|
|
231
|
+
"docs:commands": "node scripts/validate-doc-commands.mjs",
|
|
232
|
+
"docs:links": "node scripts/validate-doc-links.mjs",
|
|
233
|
+
"public-truth:check": "node scripts/validate-public-truth.mjs",
|
|
234
|
+
"docs:check": "pnpm run docs:commands && pnpm run docs:links && pnpm run public-truth:check",
|
|
219
235
|
"compat:smoke": "node scripts/compat-smoke.mjs",
|
|
220
236
|
"fixtures:check": "node scripts/validate-fixtures.mjs",
|
|
221
237
|
"recipes:check": "node scripts/validate-recipes.mjs",
|
|
@@ -10788,7 +10788,7 @@ var init_src = __esm({
|
|
|
10788
10788
|
});
|
|
10789
10789
|
|
|
10790
10790
|
// package.json
|
|
10791
|
-
var version = "6.7.
|
|
10791
|
+
var version = "6.7.2";
|
|
10792
10792
|
|
|
10793
10793
|
// packages/cli/src/list.ts
|
|
10794
10794
|
init_advanced();
|