agent-inspect 2.0.0 → 2.1.0

package/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 2.1.0
+
+### Minor Changes
+
+- 1e5e889: Release v2.1.0 with deterministic local eval and redaction utilities.
+
+  Adds the public optional `@agent-inspect/redact` and `@agent-inspect/eval` packages, root CLI redaction and eval workflows, shared redaction profiles/findings, deterministic local eval checks, and adoption recipes for local eval, share-safe traces, and CI artifacts.
+
 ## 2.0.0
 
 ### Major Changes

package/README.md

@@ -22,7 +22,7 @@ agent-inspect gives those runs **structure**: an **execution tree** you can read
 
 ## Install
 
-Current npm release line: **1.9.x** for the existing public packages. The v2 contract work on `main` is unreleased until the release-readiness gate completes. `@agent-inspect/openai-agents` v1.9 publication recovery is a separate maintainer npm setup item and is not retried by the v2 train.
+Current npm release line: **2.0.x** for the existing public packages. v2.0.0 is the stable trace-contract release: small root API, schema 1.0 persisted writer path, v0.1/v0.2/v1.0 read compatibility, and explicit non-destructive migration workflow.
 
 ```bash
 npm install agent-inspect
@@ -235,7 +235,9 @@ AGENT_INSPECT=1 node eval-runner.mjs
 - **Use an isolated inspector** with `createInspector()` and explicit local writers for tests/adapters.
 - **Correlate runs** with optional `correlationId`, `requestId`, `decisionId`, and `groupId` on `run_started` metadata.
 - **Redact before disk** with default key-based redaction, or choose `redactionProfile`: `local`, `share`, or `strict`.
-- **Inspect from the CLI** — `list`, `view`, `clean`, `logs`, `tail`, `export`, `open`, `migrate`, `diff`, `timeline`, `stats`, `search`, `what`, `report`.
+- **Inspect from the CLI** — `list`, `view`, `clean`, `logs`, `tail`, `export`, `open`, `migrate`, `eval`, `redact`, `diff`, `timeline`, `stats`, `search`, `what`, `report`.
+- **Run local evals** with `agent-inspect eval` or `@agent-inspect/eval`; built-in checks are deterministic heuristics over local traces, not model judges.
+- **Redact local files** with `agent-inspect redact` or `@agent-inspect/redact` before creating shareable copies.
 - **Migrate explicitly** with `agent-inspect migrate <trace.jsonl> --to 1.0 --dry-run` or `--output <file>`; originals are never overwritten by default.
 - **Export share-safe copies** — `export --redaction-profile share` (or `strict`) writes local Markdown/HTML/OpenInference/OTLP JSON only.
 - **Parse structured logs** you already emit (JSON first-class; log4js best-effort).
@@ -296,6 +298,8 @@ More detail: [docs/LOGS.md](docs/LOGS.md) · [docs/LOG-TO-TREE-QUICKSTART.md](do
 | `export` | Write Markdown / HTML / OpenInference-compatible JSON / OTLP JSON **locally** |
 | `open` | Read AgentInspect JSONL, OpenInference JSON, or OTLP JSON locally |
 | `migrate` | Convert a local AgentInspect JSONL file to schema 1.0 with dry-run or explicit output |
+| `eval` | Deterministic local evals over existing traces |
+| `redact` | Redact a local JSON/JSONL file or trace copy |
 | `diff` | Compare two local runs (read-only) |
 | `timeline` | Chronological view of one run |
 | `stats` | Local aggregates over a trace directory |
@@ -316,6 +320,8 @@ Full flags and behavior: [docs/CLI.md](docs/CLI.md).
 - Debug a **failed tool call** or thrown error in a support or ops agent.
 - See **which step dominated latency** in a multi-step planner or RAG pipeline.
 - **Diff two runs** after a prompt, model, or routing change (see [diff examples](docs/DIFF.md)).
+- Run **local eval checks** over a trace before sharing or creating CI artifacts.
+- **Redact** a local trace/file before attaching it to a PR, issue, or support thread.
 - Point **`logs`** / **`tail`** at existing job or service logs to get a **local execution view** without shipping data upstream.
 - **Export** a run to Markdown for a PR, postmortem, or internal thread — use `--redaction-profile share` for share-safe copies, then review before sharing.
 - Keep traces **on disk** while still using enterprise observability elsewhere.
@@ -332,6 +338,10 @@ AgentInspect is the **local-first trace workbench** for TypeScript AI agents:
 
 Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInspectRun` with `AGENT_INSPECT=1` to toggle tracing in eval or CI — see [docs/API.md](docs/API.md).
 
+**Shipped in 2.0.0:** stable root API contract, schema 1.0 persisted writer path, v0.1/v0.2/v1.0 read compatibility, and explicit trace migration workflow. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, `@agent-inspect/tui`, and `@agent-inspect/openai-agents` at **2.0.0**.
+
+**Shipped in 1.9.0:** private harness workspace foundation, explain dry-run/local analysis, promoted adapter adoption paths, and the v2 root API slimming plan.
+
 **Shipped in 1.8.0:** experimental deterministic checks (`agent-inspect/checks` and `agent-inspect check`), safe-sharing workflows (`scan`, `verify-safe`, safe artifacts), and first public `@agent-inspect/openai-agents` package. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, `@agent-inspect/tui`, and `@agent-inspect/openai-agents` at **1.8.0**.
 
 **Shipped in 1.7.0:** experimental `@agent-inspect/ai-sdk` telemetry integration for AI SDK v6 with a local no-network [ai-sdk-local-telemetry recipe](examples/recipes/ai-sdk-local-telemetry/), adapter conformance fixtures, OpenAI Agents/LangGraph support decisions, and local-first adapter docs. Examples keep `recordInputs: false`, `recordOutputs: false`, metadata-only capture, and no upload behavior. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, and `@agent-inspect/tui` at **1.7.0**.
@@ -340,7 +350,7 @@ Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInsp
 
 **Shipped in 1.5.0:** non-breaking subpath exports; `what` and `report` CLI; dual-format read path (v0.1 + v0.2 JSONL); [what-report-inspect recipe](examples/recipes/what-report-inspect/). Linked release aligns all three npm packages at **1.5.0**.
 
-**Roadmap beyond current release work:** future work continues from the local runtime, universal ingestion, and optional adapter foundations. See [ROADMAP.md](ROADMAP.md).
+**Roadmap beyond current release work:** v2.1 starts the eval/redact utility triangle, followed by reporters/CI, adapter hardening, sessions/MCP telemetry, guardrails, optional viewer/IDE surfaces, and conditional v3 extensibility. See [ROADMAP.md](ROADMAP.md).
 
 **Shipped in 1.4.0:** CI artifact recipe ([docs/CI-ARTIFACTS.md](docs/CI-ARTIFACTS.md)); `timeline`, `stats`, and `search` CLI; core helpers `buildRunTimeline`, `buildTraceStats`, `searchTraces`. Linked release aligns all three npm packages at **1.4.0**.
 
@@ -418,6 +428,9 @@ The TUI is available as a separate optional package; its programmatic API is exp
 | [examples/recipes/parallel-tools](examples/recipes/parallel-tools) | Parallel tools |
 | [examples/recipes/github-actions-artifact](examples/recipes/github-actions-artifact) | CI trace artifacts |
 | [examples/recipes/deterministic-ci-checks](examples/recipes/deterministic-ci-checks) | v1.8 checks, baseline, and safe CI artifacts |
+| [examples/recipes/eval-local-checks](examples/recipes/eval-local-checks) | v2.1 deterministic local eval checks |
+| [examples/recipes/redact-share-safe-file](examples/recipes/redact-share-safe-file) | v2.1 share-safe local redaction copy |
+| [examples/recipes/eval-ci-artifacts](examples/recipes/eval-ci-artifacts) | v2.1 eval before safe CI artifacts |
 | [examples/recipes/test-reporter-artifacts](examples/recipes/test-reporter-artifacts) | v1.8 Vitest/Jest reporter artifact patterns |
 | [examples/recipes/what-report-inspect](examples/recipes/what-report-inspect/) | `what` + `report` inspection |
 | [examples/recipes/runtime-and-ingestion](examples/recipes/runtime-and-ingestion/) | v1.6 runtime writers + universal ingestion |

package/docs/ADAPTERS.md

@@ -128,8 +128,12 @@ await agent.invoke(input, { callbacks: [callback] });
 npx agent-inspect list --dir ./.agent-inspect
 npx agent-inspect view <run-id> --dir ./.agent-inspect
 npx agent-inspect export <run-id> --format markdown --redaction-profile share
+npx agent-inspect eval <run-id> --dir ./.agent-inspect --require-success --json
+npx agent-inspect redact ./.agent-inspect/<trace-file>.jsonl --profile share --json
 ```
 
+`eval` and `redact` read local adapter traces only. They do not call model providers, upload traces, or loosen the adapter metadata-only capture defaults.
+
 ![LangChain callback with persist true writing inspectable JSONL](../assets/demos/langchain-persistence.gif)
 
 *Synthetic demo — [examples/08-langchain-adapter](../../examples/08-langchain-adapter/README.md).*

package/docs/API.md

@@ -497,7 +497,47 @@ The checks API is experimental in v1.x. The `agent-inspect check` CLI uses this
 
 Recipes: [deterministic-ci-checks](../examples/recipes/deterministic-ci-checks/README.md) for check/baseline/artifact workflows, and [test-reporter-artifacts](../examples/recipes/test-reporter-artifacts/README.md) for Vitest/Jest reporter configuration patterns.
 
-## 22. Experimental local explain APIs (v1.9)
+## 22. Experimental `@agent-inspect/eval` APIs (v2.1)
+
+`@agent-inspect/eval` is an optional package for deterministic local evals over existing traces. It consumes normalized reader output or local trace paths, returns stable JSON-compatible results, and does not call model providers, upload traces, replay agents, or create hosted datasets.
+
+Import from `@agent-inspect/eval`:
+
+```ts
+import { checks, evalRun, renderEvalMarkdown } from "@agent-inspect/eval";
+```
+
+- **`evalRun(input, options?)`**: runs selected eval rules over a local trace path or `TraceReadResult`.
+- **`checks`**: built-in deterministic rule factories for run status, tool usage, duration, depth, retries, token totals, failed steps, retrieval-before-generation, decision metadata, context overlap, quote overlap, citation presence, required source IDs, answer length bounds, and banned unsupported phrases.
+- **`renderEvalMarkdown(result)`**: renders a deterministic Markdown summary suitable for local CI logs, PR text, or artifact files after review.
+- Result types include **`EvalRunResult`**, **`EvalFinding`**, **`EvalDiagnostic`**, and **`EvalRule`**.
+
+Findings are designed for CI output: they include rule IDs, expected/actual structural summaries, and evidence paths. They should not include raw prompt, answer, context, request/response, header, API key, secret, or full tool payload values.
+
+CLI wrapper: `agent-inspect eval <trace-path-or-run-id> --require-success --json`.
+
+Recipes: [eval-local-checks](../examples/recipes/eval-local-checks/README.md) and [eval-ci-artifacts](../examples/recipes/eval-ci-artifacts/README.md).
+
+## 23. Experimental `@agent-inspect/redact` APIs (v2.1)
+
+`@agent-inspect/redact` is an optional package for reusable local redaction. It powers the root CLI `redact` workflow and shared trace-safety integrations. Redaction operates on local values/files and returns a redacted copy; it does not mutate the source object, upload content, or claim compliance-grade DLP.
+
+Import from `@agent-inspect/redact`:
+
+```ts
+import { createRedactor, redact } from "@agent-inspect/redact";
+```
+
+- **`redact(value, options?)`**: returns `{ value, findings, redacted, profile }` for a redacted copy.
+- **`createRedactor(options?)`**: creates a reusable redactor with profile, custom detectors, and custom rules.
+- **Profiles**: `local`, `share`, and `strict`.
+- **Findings**: detector id, path, action, severity, and bounded preview metadata where applicable.
+
+CLI wrapper: `agent-inspect redact <trace-or-file> --profile share --json`.
+
+Recipe: [redact-share-safe-file](../examples/recipes/redact-share-safe-file/README.md).
+
+## 24. Experimental local explain APIs (v1.9)
 
 `buildLocalExplanation()` creates a deterministic local explanation payload from a reader-selected `InspectRunTree`. It performs no network I/O, does not call model providers, and separates observed facts from deterministic inference labels.
 
@@ -522,7 +562,7 @@ Provider design gate:
 - Provider prompts must use redacted facts only, label inferred claims, and must not request raw chain-of-thought.
 - Provider packages or SDKs must not become root/core runtime dependencies.
 
-## 23. Experimental `@agent-inspect/harness` APIs
+## 25. Experimental `@agent-inspect/harness` APIs
 
 `@agent-inspect/harness` is a private experimental workspace package during the v1.9 release train. It provides a no-framework fixture runner for local targets and recipes; first public package publication remains a manual maintainer gate.
 
@@ -548,23 +588,23 @@ The harness package does not add root/core dependencies, does not upload traces,
 
 Recipes: [harness-basic](../examples/recipes/harness-basic/README.md) and [harness-adapter-local](../examples/recipes/harness-adapter-local/README.md).
 
-## 24. Deprecated APIs
+## 26. Deprecated APIs
 
 No deprecated APIs are declared as of 1.4.0.
 
-## 25. Removal / deprecation policy
+## 27. Removal / deprecation policy
 
 - Stable APIs are not removed within the current major version.
 - If removal is necessary, the API should be **deprecated** first, documented, and kept for a reasonable window (target: at least one minor line) unless security requires faster action.
 
-## 26. Backward compatibility policy
+## 28. Backward compatibility policy
 
 - Manual trace JSONL (`schemaVersion: "0.1"`) remains readable.
 - Additive schema changes are allowed in minor versions.
 - Breaking changes require a major version.
 - Unknown fields should be ignored where safe.
 
-## 25. Examples
+## 29. Examples
 
 ### Minimal manual trace
 

package/docs/CLI.md

@@ -28,6 +28,8 @@ Core commands:
 - `open` — read supported local trace files, directories, or stdin through the canonical reader pipeline
 - `migrate` — convert one local AgentInspect JSONL file to schema 1.0 with dry-run or explicit output
 - `check` — run deterministic local trace checks with stable JSON and exit codes
+- `eval` — run deterministic local evals over existing traces
+- `redact` — redact a local JSON/JSONL file or trace copy
 - `scan` — best-effort local safety scan for trace capture risks
 - `verify-safe` — best-effort local trace safety verification
 - `artifacts` — create safe local CI trace artifact bundles and optional step summaries
@@ -58,6 +60,12 @@ Exception: `check` uses CI-oriented semantic exit codes:
 - **3**: trace input could not be read
 - **4**: unsupported or ambiguous trace format
 
+Exception: `eval` uses local eval semantic exit codes:
+
+- **0**: all selected eval rules passed
+- **1**: eval ran and at least one error-severity rule failed
+- **2**: invalid arguments, invalid config, unreadable input, unsupported input, ambiguous input, or run-selection errors
+
 Exception: `scan` and `verify-safe` use local safety status exit codes:
 
 - **0**: status is SAFE or SAFE WITH WARNINGS
@@ -79,6 +87,8 @@ Many commands support `--json` for scripting. JSON output is intended to be:
 - Log-derived output includes **confidence** labels and avoids inventing parent-child relationships.
 - Redaction defaults are conservative (e.g. `authorization`, `cookie`, `token`, `apiKey`, `password`, `secret`, `email`).
 - Exported payloads are **redacted by default** unless explicitly configured otherwise.
+- `eval` is deterministic and local-only. It does not replay agents, call model providers, upload traces, or create hosted datasets.
+- `redact` writes or prints a redacted copy. It does not mutate source trace files.
 - `scan` and `verify-safe` are best-effort local checks, not compliance, privacy, security, or regulatory certifications.
 - `artifacts` renders structural summaries and check evidence only; it does not include raw prompt/output bodies, request/response bodies, headers, API keys, secrets, or full tool payloads.
 
@@ -322,7 +332,85 @@ npx agent-inspect check trace.jsonl --max-duration-ms 30000 --required-tool sear
 
 Recipe: [examples/recipes/deterministic-ci-checks](../examples/recipes/deterministic-ci-checks/README.md)
 
-### 6.10 `scan` and `verify-safe`
+### 6.10 `eval`
+
+Run deterministic local evals against an existing trace. This command reads through the same local reader pipeline as `open` and `check`; it does not rerun agents, call models, upload traces, mutate inputs, or create a hosted dataset.
+
+```bash
+agent-inspect eval <trace-path-or-run-id> [options]
+```
+
+Options:
+
+- `--dir <path>`: trace directory for run-id lookup
+- `--format <agent-inspect-jsonl|openinference-json|otlp-json>`: explicit trace input format
+- `--run <run-id>`: select a run when input contains multiple runs
+- `--config <path>`: eval config (`.json`, `.js`, `.mjs`, or `.cjs`); TypeScript configs are rejected until an explicit loader is approved
+- `--json`: print deterministic JSON eval result
+- `--markdown`: print deterministic Markdown eval summary
+- `--require-success`: require the selected run to complete successfully
+- `--required-tool <name>`: require a tool name (repeatable)
+- `--forbid-tool <name>` / `--forbidden-tool <name>`: forbid a tool name (repeatable)
+- `--max-duration-ms <number>`, `--max-depth <number>`, `--max-retries <number>`, `--max-total-tokens <number>`
+- `--require-retrieval-before-generation`
+- `--required-decision-metadata <key>`: require decision metadata (repeatable)
+- `--context-overlap`, `--min-context-overlap <number>`, `--min-shared-terms <number>`
+- `--quote-overlap`
+- `--citation-presence`
+- `--required-source-id <id>`: require a source id in context or citations (repeatable)
+- `--min-answer-characters <number>`, `--max-answer-characters <number>`, `--min-answer-words <number>`, `--max-answer-words <number>`
+- `--banned-phrase <text>`: ban unsupported-answer phrasing (repeatable)
+
+Example config:
+
+```json
+{
+  "eval": {
+    "requireSuccess": true,
+    "requiredTools": ["searchDocs"],
+    "forbiddenTools": ["deleteAccount"],
+    "citationPresence": true,
+    "contextOverlap": { "minOverlap": 0.2 },
+    "requiredSourceIds": ["policy-30-day"]
+  }
+}
+```
+
+Examples:
+
+```bash
+npx agent-inspect eval fixtures/traces-v0.2/manual-basic.jsonl --require-success --json
+npx agent-inspect eval trace.jsonl --forbid-tool deleteAccount --markdown
+npx agent-inspect eval trace.jsonl --config agent-inspect.eval.json --json
+```
+
+Recipes: [eval-local-checks](../examples/recipes/eval-local-checks/README.md) and [eval-ci-artifacts](../examples/recipes/eval-ci-artifacts/README.md).
+
+### 6.11 `redact`
+
+Redact a local JSON or JSONL trace/file. The command prints or writes a redacted copy and reports bounded findings; it does not mutate the source file or upload content.
+
+```bash
+agent-inspect redact <trace-or-file> [options]
+```
+
+Options:
+
+- `--dir <path>`: trace directory for run-id lookup
+- `--profile <local|share|strict>`: redaction profile (default `share`)
+- `-o, --output <path>`: write redacted content to a file
+- `--json`: print deterministic JSON wrapper with findings
+
+Examples:
+
+```bash
+npx agent-inspect redact trace.jsonl --profile share --json
+npx agent-inspect redact trace.jsonl --profile strict -o trace.share.jsonl
+```
+
+Recipe: [redact-share-safe-file](../examples/recipes/redact-share-safe-file/README.md).
+
+### 6.12 `scan` and `verify-safe`
 
 Run best-effort local safety verification for supported trace inputs. These commands are local and read-only: they do not rerun agents, call models, upload traces, mutate input files, or certify compliance.
 
@@ -361,7 +449,7 @@ npx agent-inspect verify-safe minimal-success --dir fixtures/traces
 npx agent-inspect verify-safe trace.jsonl --max-string-length 8192 --json
 ```
 
-### 6.11 `artifacts`
+### 6.13 `artifacts`
 
 Create deterministic local CI artifacts for supported trace inputs. This command is local and read-only for trace inputs: it does not rerun agents, call models, upload files, use GitHub APIs, or mutate repository state. It writes only to `--output-dir` and, when requested, a local step-summary file.
 
@@ -401,7 +489,7 @@ npx agent-inspect artifacts candidate.jsonl --baseline baseline.jsonl --output-d
 
 Recipe and sample workflow: [examples/recipes/deterministic-ci-checks](../examples/recipes/deterministic-ci-checks/README.md)
 
-### 6.12 `diff`
+### 6.14 `diff`
 
 Compare two manual trace runs. Diff is **local** and **read-only** (does not rerun agents).
 
@@ -465,7 +553,7 @@ Differences:
 
 More examples, including timing-only and structure-only diffs, are in `docs/DIFF.md`.
 
-### 6.13 `timeline`
+### 6.15 `timeline`
 
 Chronological step list for one manual trace. Read-only; does not mutate JSONL files.
 
@@ -481,7 +569,7 @@ Options:
 
 ![Timeline with slow-step focus](../assets/demos/timeline.gif)
 
-### 6.14 `stats`
+### 6.16 `stats`
 
 Local aggregate statistics over trace files in a directory. Read-only.
 
@@ -501,7 +589,7 @@ Options:
 
 Use `--correlation-id` or `--group-id` to filter runs by `run_started` metadata (see [API.md](./API.md)).
 
-### 6.15 `search`
+### 6.17 `search`
 
 Deterministic search over local traces (substring / exact filters). No semantic search.
 
@@ -531,7 +619,7 @@ npx agent-inspect search --duration ">100ms" --json
 
 ![Search traces by status error](../assets/demos/search.gif)
 
-### 6.16 `what`
+### 6.18 `what`
 
 Concise human-readable summary of one local trace run. Read-only; accepts v0.1 manual JSONL and v0.2 persisted-event JSONL through the shared dual-format normalization path. Vocabulary: [TRACE-VOCABULARY-V1.5.md](./proposals/TRACE-VOCABULARY-V1.5.md).
 
@@ -560,7 +648,7 @@ Outcome: Completed successfully.
 Slowest: plan (100ms, logic)
 ```
 
-### 6.17 `report`
+### 6.19 `report`
 
 Generate a local inspection report combining **what happened**, **timeline**, and **execution tree** sections. The command reads local v0.1 manual JSONL and v0.2 persisted-event JSONL through the shared dual-format normalization path without mutating them. Distinct from `export` (which targets shareable tree snapshots and standards formats).
 
@@ -585,7 +673,7 @@ Example:
 npx agent-inspect report minimal-success --dir fixtures/traces --format html -o report.html
 ```
 
-### 6.18 `explain`
+### 6.20 `explain`
 
 Explain a local trace using deterministic facts and local inference labels. This command reads through the same local reader pipeline as `open` / `check`; it does not call a model provider, upload traces, replay agents, or mutate input files.
 

package/docs/COMPARE.md

@@ -1,6 +1,6 @@
 ## Compare AgentInspect
 
-AgentInspect is a local-first execution-tree debugger for TypeScript AI agents. It’s designed for inner-loop debugging and quick inspection — not as a replacement for hosted observability, evaluation, or production monitoring platforms.
+AgentInspect is a local-first execution-tree debugger for TypeScript AI agents. It’s designed for inner-loop debugging, deterministic local eval heuristics, redaction, and quick inspection — not as a replacement for hosted observability, dataset evaluation, or production monitoring platforms.
 
 ## AgentInspect vs console.log
 
@@ -16,7 +16,7 @@ LangSmith is a hosted/platform workflow for tracing, evaluation, and observabili
 AgentInspect is local-first CLI debugging:
 
 - Use AgentInspect to debug locally before/alongside LangSmith when iterating on agent logic.
-- AgentInspect does not provide hosted dashboards, dataset/eval workflows, or production tracing pipelines.
+- AgentInspect does not provide hosted dashboards, dataset/eval management workflows, or production tracing pipelines.
 
 ## AgentInspect vs Langfuse
 
@@ -33,7 +33,7 @@ Braintrust is strong for evals, regressions, datasets, and production AI quality
 
 AgentInspect is lighter and local-first:
 
-- Use AgentInspect to understand a single run locally.
+- Use AgentInspect to understand a single run locally and run deterministic trace checks/eval heuristics before sharing artifacts.
 - Use Braintrust when you want repeatable evals, comparisons at scale, and production quality workflows.
 
 ## AgentInspect vs Phoenix / OpenInference
@@ -61,9 +61,10 @@ AgentInspect avoids SDK/collector setup for local debugging:
 | --- | --- |
 | Local agent debugging | Strong fit |
 | No-account CLI tracing | Strong fit |
+| Deterministic local eval heuristics | Good fit |
+| Share-safe local redaction copy | Good fit |
 | Production dashboards | Not the goal |
 | Hosted eval datasets | Not the goal |
 | Prompt management | Not the goal |
 | Standards-aligned local export | Partial (compatibility-oriented) |
 | Full observability platform | Use a dedicated platform |
-

package/docs/GETTING-STARTED.md

@@ -204,17 +204,34 @@ agent-inspect search --dir ./.agent-inspect --status error --limit 10
 
 For CI artifact workflows, see [CI-ARTIFACTS.md](./CI-ARTIFACTS.md) and [github-actions-artifact recipe](../examples/recipes/github-actions-artifact/).
 
-## 10. Diff two runs
+## 10. Run local evals and redact share copies
+
+After a trace exists, run deterministic eval checks without replaying the agent or calling a model provider:
+
+```bash
+agent-inspect eval minimal-success --dir fixtures/traces --require-success --json
+agent-inspect eval trace.jsonl --forbid-tool deleteAccount --citation-presence --json
+```
+
+Before attaching a trace or JSON artifact to a PR, issue, or support thread, create a redacted local copy:
+
+```bash
+agent-inspect redact trace.jsonl --profile share --json
+```
+
+Recipes: [eval-local-checks](../examples/recipes/eval-local-checks/), [redact-share-safe-file](../examples/recipes/redact-share-safe-file/), and [eval-ci-artifacts](../examples/recipes/eval-ci-artifacts/).
+
+## 11. Diff two runs
 
 ```bash
 agent-inspect diff minimal-success minimal-error --dir fixtures/traces
 ```
 
-## 11. Try recipes
+## 12. Try recipes
 
 See `examples/recipes/README.md`.
 
-## 12. Optional framework adapters
+## 13. Optional framework adapters
 
 See [ADAPTERS.md](./ADAPTERS.md) for AI SDK local telemetry, OpenAI Agents local-only processing, and LangChain callbacks.
 
@@ -228,7 +245,7 @@ pnpm add @agent-inspect/langchain
 
 See [examples/08-langchain-adapter](../examples/08-langchain-adapter/README.md) and [docs/ADAPTERS.md](./ADAPTERS.md).
 
-## 13. Optional TUI
+## 14. Optional TUI
 
 `@agent-inspect/tui` is optional and **experimental**. The CLI can invoke it with:
 
@@ -236,16 +253,17 @@ See [examples/08-langchain-adapter](../examples/08-langchain-adapter/README.md)
 agent-inspect view <runId> --tui
 ```
 
-## 14. Safety notes
+## 15. Safety notes
 
 - Nothing uploads by default; core tracing, readers, checks, and exports are local-first.
+- Eval and redaction commands read local inputs and do not call provider APIs or hosted services.
 - Redaction is on by default for log-derived attributes, **manual trace metadata (before disk)**, and exports. Pass `redact: false` to opt out of manual metadata redaction.
 - Export redaction shapes a local copy and does not mutate the source trace; review exported files before sharing.
 - Persisted events are size-bounded by default (see `docs/API.md`).
 - Confidence labels are required to keep attribution honest.
 - AgentInspect is for local debugging, not production monitoring.
 
-## 15. Next docs
+## 16. Next docs
 
 - [docs/API.md](./API.md)
 - [docs/CLI.md](./CLI.md)

package/docs/KNOWN-ISSUES.md

@@ -12,6 +12,12 @@ AgentInspect is **local-first** and **CLI-first**. These behaviors are intention
 - **OpenInference** and **OTLP JSON** exports are **compatibility-oriented** and **experimental**. Validate against your target collector or backend before relying on them.
 - Exports generate **strings/files locally** only—there is **no** automatic upload.
 
+## Eval and redaction
+
+- **`agent-inspect eval` is deterministic local eval, not hosted evaluation.** Built-in rules are structural checks and grounding heuristics over existing traces. They do not call model providers, manage datasets, replay agents, or perform semantic LLM judging.
+- **TypeScript eval configs are not loaded by default.** Use `.json`, `.js`, `.mjs`, or `.cjs` until an explicit loader path is approved.
+- **`agent-inspect redact` creates a local redacted copy.** It does not mutate source traces, encrypt files, or guarantee compliance-grade detection. Review the exact output before sharing.
+
 ## Readers and `open`
 
 - **OpenInference** and **OTLP JSON** readers are **compatibility-oriented** and **experimental**. They normalize local JSON payloads into AgentInspect inspection trees and may warn on unsupported semantic fields.
@@ -103,6 +109,7 @@ pnpm compat:smoke
 
 - `agent-inspect artifacts --github-summary` writes a local step-summary file only. It does not call GitHub APIs, open PR comments, upload artifacts, or mutate repository state.
 - Baseline checks compare normalized structural facts from explicit candidate and baseline inputs. They are useful for CI regression evidence, not replay or semantic eval scoring.
+- v2.1 eval/redact adoption recipes are local and deterministic: [eval-local-checks](../examples/recipes/eval-local-checks/), [redact-share-safe-file](../examples/recipes/redact-share-safe-file/), and [eval-ci-artifacts](../examples/recipes/eval-ci-artifacts/).
 
 ### What to include in a bug report
 

package/docs/LIMITATIONS.md

@@ -50,6 +50,7 @@ This document states what AgentInspect **does not** provide today. It complement
 ## Trace safety bounds
 
 - **Redaction profiles** (`local`, `share`, `strict`) are key-based presets — not compliance-grade PII detection. Review exports before sharing even with `--redaction-profile strict`.
+- **`@agent-inspect/redact` and `agent-inspect redact` create redacted copies.** They do not encrypt source traces, mutate originals, certify compliance, or guarantee every sensitive value is detected.
 - **Default metadata redaction** covers common sensitive keys only (exact key match, case-insensitive). Custom secret field names are not redacted unless you add rules via `redact: { rules: [...] }`.
 - **Metadata truncation** applies to string values and nested structures; very large metadata may be replaced with a truncation marker when `maxEventBytes` is exceeded (default 64 KiB per JSONL line).
 - **Redaction is not encryption.** Local trace files remain readable on disk; treat `.agent-inspect-runs/` like any developer artifact that may contain operational data.
@@ -57,6 +58,7 @@ This document states what AgentInspect **does not** provide today. It complement
 ## Checks, artifacts, and test reporters
 
 - **Checks are deterministic local rules, not compliance certification.** `check`, `scan`, and `verify-safe` surface bounded findings and diagnostics over supported local inputs; they do not prove a trace is safe for every sharing context.
+- **Eval is deterministic local heuristics.** `@agent-inspect/eval` and `agent-inspect eval` do not provide LLM-as-judge scoring, hosted datasets, replay, semantic grading, or production quality analytics.
 - **Safe CI artifacts are structural summaries.** They avoid raw prompt/output/request/response/header/tool payload content by default, but teams should still review generated files before sharing.
 - **Vitest/Jest reporters are optional package surfaces.** Recipes document config patterns and explicit associations; package publication is controlled by release readiness and maintainer authorization.
 
@@ -64,7 +66,7 @@ This document states what AgentInspect **does not** provide today. It complement
 
 - **No replay / fork** of past runs from traces alone.
 - **No time-travel debugging** across arbitrary runtime state.
-- **No multi-run statistical evaluation** built into core.
+- **No multi-run statistical evaluation** built into core. v2.1 eval checks are per selected trace/run unless user code aggregates results.
 
 ## Economics
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-inspect",
-  "version": "2.0.0",
+  "version": "2.1.0",
   "license": "MIT",
   "type": "module",
   "description": "Local-first execution-tree debugger for TypeScript AI agents",
@@ -168,7 +168,7 @@
   },
   "scripts": {
     "clean": "pnpm -r exec -- rm -rf dist",
-    "build": "pnpm exec tsup --config tsup.core.config.ts && pnpm exec tsup --config tsup.cli.config.ts && pnpm exec tsup --config tsup.langchain.config.ts && pnpm exec tsup --config tsup.tui.config.ts && pnpm exec tsup --config tsup.ai-sdk.config.ts && pnpm exec tsup --config tsup.vitest.config.ts && pnpm exec tsup --config tsup.jest.config.ts && pnpm exec tsup --config tsup.openai-agents.config.ts && pnpm exec tsup --config tsup.harness.config.ts",
+    "build": "pnpm exec tsup --config tsup.core.config.ts && pnpm exec tsup --config tsup.cli.config.ts && pnpm exec tsup --config tsup.langchain.config.ts && pnpm exec tsup --config tsup.tui.config.ts && pnpm exec tsup --config tsup.ai-sdk.config.ts && pnpm exec tsup --config tsup.vitest.config.ts && pnpm exec tsup --config tsup.jest.config.ts && pnpm exec tsup --config tsup.openai-agents.config.ts && pnpm exec tsup --config tsup.harness.config.ts && pnpm exec tsup --config tsup.redact.config.ts && pnpm exec tsup --config tsup.eval.config.ts",
     "typecheck": "tsc --noEmit",
     "test": "vitest run",
     "test:watch": "vitest",