agent-inspect 2.0.0 → 2.2.0

package/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## 2.2.0
+
+### Minor Changes
+
+- efb3fef: Release v2.2.0 with local test reporter artifacts and CI summaries.
+
+  Adds the public optional `@agent-inspect/vitest` and `@agent-inspect/jest` reporter packages, the shared experimental `agent-inspect/reporters` helpers, and the `agent-inspect ci-summary` workflow for deterministic local reporter manifests and CI artifacts.
+
+## Unreleased
+
+### Draft v2.2.0 Notes
+
+- Prepared the v2.2 reporter and CI workflow release notes for local Vitest/Jest reporter artifacts, the shared `agent-inspect/reporters` manifest helpers, and `agent-inspect ci-summary`.
+- Reporter artifacts remain local-only and metadata-bounded. `ci-summary` reads reporter manifest JSON only, validates relative artifact paths, and does not read trace contents, upload artifacts, call GitHub APIs, or mutate repository state.
+- `@agent-inspect/vitest` and `@agent-inspect/jest` remain private/unpublished until maintainer first-publication setup is explicitly cleared before release prep.
+
+## 2.1.0
+
+### Minor Changes
+
+- 1e5e889: Release v2.1.0 with deterministic local eval and redaction utilities.
+
+  Adds the public optional `@agent-inspect/redact` and `@agent-inspect/eval` packages, root CLI redaction and eval workflows, shared redaction profiles/findings, deterministic local eval checks, and adoption recipes for local eval, share-safe traces, and CI artifacts.
+
 ## 2.0.0
 
 ### Major Changes

package/README.md

@@ -22,7 +22,7 @@ agent-inspect gives those runs **structure**: an **execution tree** you can read
 
 ## Install
 
-Current npm release line: **1.9.x** for the existing public packages. The v2 contract work on `main` is unreleased until the release-readiness gate completes. `@agent-inspect/openai-agents` v1.9 publication recovery is a separate maintainer npm setup item and is not retried by the v2 train.
+Current npm release line: **2.1.x** for the existing public packages. v2.1.0 adds deterministic local eval and reusable redaction utilities on top of the stable v2 trace contract: small root API, schema 1.0 persisted writer path, v0.1/v0.2/v1.0 read compatibility, and explicit non-destructive migration workflow.
 
 ```bash
 npm install agent-inspect
@@ -235,9 +235,12 @@ AGENT_INSPECT=1 node eval-runner.mjs
 - **Use an isolated inspector** with `createInspector()` and explicit local writers for tests/adapters.
 - **Correlate runs** with optional `correlationId`, `requestId`, `decisionId`, and `groupId` on `run_started` metadata.
 - **Redact before disk** with default key-based redaction, or choose `redactionProfile`: `local`, `share`, or `strict`.
-- **Inspect from the CLI** — `list`, `view`, `clean`, `logs`, `tail`, `export`, `open`, `migrate`, `diff`, `timeline`, `stats`, `search`, `what`, `report`.
+- **Inspect from the CLI** — `list`, `view`, `clean`, `logs`, `tail`, `export`, `open`, `migrate`, `eval`, `redact`, `diff`, `timeline`, `stats`, `search`, `what`, `report`.
+- **Run local evals** with `agent-inspect eval` or `@agent-inspect/eval`; built-in checks are deterministic heuristics over local traces, not model judges.
+- **Redact local files** with `agent-inspect redact` or `@agent-inspect/redact` before creating shareable copies.
 - **Migrate explicitly** with `agent-inspect migrate <trace.jsonl> --to 1.0 --dry-run` or `--output <file>`; originals are never overwritten by default.
 - **Export share-safe copies** — `export --redaction-profile share` (or `strict`) writes local Markdown/HTML/OpenInference/OTLP JSON only.
+- **Create local CI artifacts** with `agent-inspect artifacts`, and summarize local test-reporter manifests with `agent-inspect ci-summary`.
 - **Parse structured logs** you already emit (JSON first-class; log4js best-effort).
 - **Optional LangChain adapter** — metadata-only by default; optional `persist: true` and `stream: true` streaming metadata (no full token capture by default).
 - **Optional AI SDK adapter** — experimental `@agent-inspect/ai-sdk` telemetry integration for AI SDK v6; metadata-only by default with `recordInputs: false` and `recordOutputs: false`.
@@ -296,6 +299,8 @@ More detail: [docs/LOGS.md](docs/LOGS.md) · [docs/LOG-TO-TREE-QUICKSTART.md](do
 | `export` | Write Markdown / HTML / OpenInference-compatible JSON / OTLP JSON **locally** |
 | `open` | Read AgentInspect JSONL, OpenInference JSON, or OTLP JSON locally |
 | `migrate` | Convert a local AgentInspect JSONL file to schema 1.0 with dry-run or explicit output |
+| `eval` | Deterministic local evals over existing traces |
+| `redact` | Redact a local JSON/JSONL file or trace copy |
 | `diff` | Compare two local runs (read-only) |
 | `timeline` | Chronological view of one run |
 | `stats` | Local aggregates over a trace directory |
@@ -304,6 +309,7 @@ More detail: [docs/LOGS.md](docs/LOGS.md) · [docs/LOG-TO-TREE-QUICKSTART.md](do
 | `report` | Markdown/HTML inspection report (what + timeline + tree) |
 | `check` / `scan` / `verify-safe` | Deterministic local trace checks and best-effort safety verification |
 | `artifacts` | Safe local CI artifact bundles and optional step-summary file output |
+| `ci-summary` | Summarize local Vitest/Jest reporter artifact manifests for CI |
 
 ![Timeline with slow-step focus for one run](https://raw.githubusercontent.com/rajudandigam/agent-inspect/main/docs/assets/demos/timeline.gif)
 
@@ -316,6 +322,8 @@ Full flags and behavior: [docs/CLI.md](docs/CLI.md).
 - Debug a **failed tool call** or thrown error in a support or ops agent.
 - See **which step dominated latency** in a multi-step planner or RAG pipeline.
 - **Diff two runs** after a prompt, model, or routing change (see [diff examples](docs/DIFF.md)).
+- Run **local eval checks** over a trace before sharing or creating CI artifacts.
+- **Redact** a local trace/file before attaching it to a PR, issue, or support thread.
 - Point **`logs`** / **`tail`** at existing job or service logs to get a **local execution view** without shipping data upstream.
 - **Export** a run to Markdown for a PR, postmortem, or internal thread — use `--redaction-profile share` for share-safe copies, then review before sharing.
 - Keep traces **on disk** while still using enterprise observability elsewhere.
@@ -332,6 +340,12 @@ AgentInspect is the **local-first trace workbench** for TypeScript AI agents:
 
 Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInspectRun` with `AGENT_INSPECT=1` to toggle tracing in eval or CI — see [docs/API.md](docs/API.md).
 
+**Shipped in 2.1.0:** deterministic local eval and redaction utilities. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, `@agent-inspect/tui`, `@agent-inspect/openai-agents`, `@agent-inspect/redact`, and `@agent-inspect/eval` at **2.1.0**.
+
+**Shipped in 2.0.0:** stable root API contract, schema 1.0 persisted writer path, v0.1/v0.2/v1.0 read compatibility, and explicit trace migration workflow. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, `@agent-inspect/tui`, and `@agent-inspect/openai-agents` at **2.0.0**.
+
+**Shipped in 1.9.0:** private harness workspace foundation, explain dry-run/local analysis, promoted adapter adoption paths, and the v2 root API slimming plan.
+
 **Shipped in 1.8.0:** experimental deterministic checks (`agent-inspect/checks` and `agent-inspect check`), safe-sharing workflows (`scan`, `verify-safe`, safe artifacts), and first public `@agent-inspect/openai-agents` package. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, `@agent-inspect/tui`, and `@agent-inspect/openai-agents` at **1.8.0**.
 
 **Shipped in 1.7.0:** experimental `@agent-inspect/ai-sdk` telemetry integration for AI SDK v6 with a local no-network [ai-sdk-local-telemetry recipe](examples/recipes/ai-sdk-local-telemetry/), adapter conformance fixtures, OpenAI Agents/LangGraph support decisions, and local-first adapter docs. Examples keep `recordInputs: false`, `recordOutputs: false`, metadata-only capture, and no upload behavior. Linked release aligns `agent-inspect`, `@agent-inspect/ai-sdk`, `@agent-inspect/langchain`, and `@agent-inspect/tui` at **1.7.0**.
@@ -340,7 +354,7 @@ Pass `enabled: false` to `inspectRun` for a no-trace passthrough. Use `maybeInsp
 
 **Shipped in 1.5.0:** non-breaking subpath exports; `what` and `report` CLI; dual-format read path (v0.1 + v0.2 JSONL); [what-report-inspect recipe](examples/recipes/what-report-inspect/). Linked release aligns all three npm packages at **1.5.0**.
 
-**Roadmap beyond current release work:** future work continues from the local runtime, universal ingestion, and optional adapter foundations. See [ROADMAP.md](ROADMAP.md).
+**Roadmap beyond current release work:** v2.2 prepares test reporters and CI workflows, followed by adapter hardening, sessions/MCP telemetry, guardrails, optional viewer/IDE surfaces, and conditional v3 extensibility. See [ROADMAP.md](ROADMAP.md).
 
 **Shipped in 1.4.0:** CI artifact recipe ([docs/CI-ARTIFACTS.md](docs/CI-ARTIFACTS.md)); `timeline`, `stats`, and `search` CLI; core helpers `buildRunTimeline`, `buildTraceStats`, `searchTraces`. Linked release aligns all three npm packages at **1.4.0**.
 
@@ -395,6 +409,12 @@ npx agent-inspect view <run-id> --tui
 
 The TUI is available as a separate optional package; its programmatic API is experimental, while the CLI integration (`view --tui`) is the intended usage. Details: [docs/ADAPTERS.md](docs/ADAPTERS.md).
 
+### Test reporter artifacts (`@agent-inspect/vitest`, `@agent-inspect/jest`)
+
+Optional Vitest/Jest reporter packages are implemented in the workspace for local failure artifacts, but remain private/unpublished until the maintainer clears first-publication setup for a v2.2 release. They write shared `schemaVersion: "0.1"` reporter manifests with safe relative artifact paths and bounded structural metadata. Use `agent-inspect ci-summary` to summarize those local manifests in CI without reading trace contents or calling GitHub APIs.
+
+Reporter artifact behavior and API details are documented in [docs/API.md](docs/API.md) and [docs/CI-ARTIFACTS.md](docs/CI-ARTIFACTS.md).
+
 ## Examples and recipes
 
 | Example | Shows |
@@ -418,7 +438,10 @@ The TUI is available as a separate optional package; its programmatic API is exp
 | [examples/recipes/parallel-tools](examples/recipes/parallel-tools) | Parallel tools |
 | [examples/recipes/github-actions-artifact](examples/recipes/github-actions-artifact) | CI trace artifacts |
 | [examples/recipes/deterministic-ci-checks](examples/recipes/deterministic-ci-checks) | v1.8 checks, baseline, and safe CI artifacts |
-| [examples/recipes/test-reporter-artifacts](examples/recipes/test-reporter-artifacts) | v1.8 Vitest/Jest reporter artifact patterns |
+| [examples/recipes/eval-local-checks](examples/recipes/eval-local-checks) | v2.1 deterministic local eval checks |
+| [examples/recipes/redact-share-safe-file](examples/recipes/redact-share-safe-file) | v2.1 share-safe local redaction copy |
+| [examples/recipes/eval-ci-artifacts](examples/recipes/eval-ci-artifacts) | v2.1 eval before safe CI artifacts |
+| [examples/recipes/test-reporter-artifacts](examples/recipes/test-reporter-artifacts) | Vitest/Jest reporter artifact patterns |
 | [examples/recipes/what-report-inspect](examples/recipes/what-report-inspect/) | `what` + `report` inspection |
 | [examples/recipes/runtime-and-ingestion](examples/recipes/runtime-and-ingestion/) | v1.6 runtime writers + universal ingestion |
 

package/docs/ADAPTERS.md

@@ -128,8 +128,12 @@ await agent.invoke(input, { callbacks: [callback] });
 npx agent-inspect list --dir ./.agent-inspect
 npx agent-inspect view <run-id> --dir ./.agent-inspect
 npx agent-inspect export <run-id> --format markdown --redaction-profile share
+npx agent-inspect eval <run-id> --dir ./.agent-inspect --require-success --json
+npx agent-inspect redact ./.agent-inspect/<trace-file>.jsonl --profile share --json
 ```
 
+`eval` and `redact` read local adapter traces only. They do not call model providers, upload traces, or loosen the adapter metadata-only capture defaults.
+
 ![LangChain callback with persist true writing inspectable JSONL](../assets/demos/langchain-persistence.gif)
 
 *Synthetic demo — [examples/08-langchain-adapter](../../examples/08-langchain-adapter/README.md).*

package/docs/API.md

@@ -222,9 +222,32 @@ No network writer, OpenTelemetry exporter, provider wrapper, or global monkey-pa
 
 Recipe: [examples/recipes/ai-sdk-local-telemetry](../examples/recipes/ai-sdk-local-telemetry/).
 
+## 11.1 Experimental `agent-inspect/reporters` APIs
+
+`agent-inspect/reporters` contains shared, dependency-free helpers for local test reporter artifacts. The subpath does not import Vitest, Jest, GitHub SDKs, provider SDKs, or upload clients.
+
+Import from `agent-inspect/reporters`:
+
+```ts
+import {
+  TRACE_ARTIFACT_MANIFEST_SCHEMA_VERSION,
+  createReporterArtifactPath,
+  createTraceArtifactManifest,
+  validateReporterArtifactPath,
+  type TraceArtifactManifest,
+} from "agent-inspect/reporters";
+```
+
+- **`TRACE_ARTIFACT_MANIFEST_SCHEMA_VERSION`**: currently `"0.1"` for local reporter manifests.
+- **`createTraceArtifactManifest(options)`**: clones, sorts, and deduplicates reporter results/artifacts into deterministic manifest JSON.
+- **`createReporterArtifactPath(options)`**: creates a safe relative artifact path under a caller-provided output directory.
+- **`validateReporterArtifactPath(options)`**: rejects empty, absolute, traversal, Windows-absolute, and symlink-escape style paths before reporters or `ci-summary` trust artifact links.
+
+The manifest records framework, generation time, bounded test results, artifact descriptors, redaction profile, and diagnostics. It is an artifact index only; it should not contain raw trace contents, prompts, model outputs, request/response bodies, headers, API keys, secrets, or full tool payloads.
+
 ## 12. Experimental `@agent-inspect/vitest` APIs
 
-`@agent-inspect/vitest` is an optional experimental workspace package for local Vitest failure artifacts. It remains private/unpublished. It does not add a Vitest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
+`@agent-inspect/vitest` is an optional experimental workspace package for local Vitest failure artifacts. It remains private/unpublished pending maintainer first-publication setup. It does not add a Vitest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
 
 Import from `@agent-inspect/vitest`:
 
@@ -237,6 +260,7 @@ import { createAgentInspectVitestReporter } from "@agent-inspect/vitest";
   - **`githubSummary`**: optional GitHub step-summary file path. The reporter appends bounded structural counts only and does not use the GitHub API.
   - **`retainSuccessful`**: `false`/undefined keeps no passing-test artifacts; `true` keeps up to `maxSuccessfulTraces`; a number keeps up to that many passing-test artifacts.
   - **`maxSuccessfulTraces`**: upper bound for passing-test artifacts, capped by the reporter.
+  - **`redactionProfile`**: manifest artifact profile, `local` (default), `share`, or `strict`.
   - **`resolveTrace(test)`**: optional explicit association resolver when task metadata is not convenient.
   - **`onDiagnostic(diagnostic)`**: observes non-fatal reporter/artifact failures.
   - **`getDiagnostics()`** and **`getArtifacts()`** expose reporter state for tests and custom harnesses.
@@ -252,11 +276,11 @@ ctx.task.meta.agentInspect = {
 };
 ```
 
-Artifacts are safe structural summaries. They include bounded test identity, status, trace run id, and trace filename, but they do not read or embed raw trace contents, prompts, generated outputs, request/response bodies, headers, API keys, secrets, or tool payloads. Reporter/artifact failures are diagnostics and do not replace original Vitest failures.
+Artifacts are safe structural summaries. The reporter writes a shared `schemaVersion: "0.1"` manifest wrapper with package metadata, generated time, framework, test results, artifact descriptors, relative paths, and redaction profile. It includes bounded test identity, status, trace run id, and trace filename, but it does not read or embed raw trace contents, prompts, generated outputs, request/response bodies, headers, API keys, secrets, or tool payloads. Reporter/artifact failures are diagnostics and do not replace original Vitest failures.
 
 ## 13. Experimental `@agent-inspect/jest` APIs
 
-`@agent-inspect/jest` is an optional experimental workspace package for local Jest failure artifacts. It remains private/unpublished. It does not add a Jest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
+`@agent-inspect/jest` is an optional experimental workspace package for local Jest failure artifacts. It remains private/unpublished pending maintainer first-publication setup. It does not add a Jest dependency to root/core, does not upload artifacts, and does not infer trace relationships by timestamp.
 
 Import from `@agent-inspect/jest`:
 
@@ -270,6 +294,7 @@ import { AgentInspectJestReporter, createAgentInspectJestReporter } from "@agent
   - **`githubSummary`**: optional GitHub step-summary file path. The reporter appends bounded structural counts only and does not use the GitHub API.
   - **`retainSuccessful`**: `false`/undefined keeps no passing-test artifacts; `true` keeps up to `maxSuccessfulTraces`; a number keeps up to that many passing-test artifacts.
   - **`maxSuccessfulTraces`**: upper bound for passing-test artifacts, capped by the reporter.
+  - **`redactionProfile`**: manifest artifact profile, `local` (default), `share`, or `strict`.
   - **`associations`**: explicit trace associations keyed by `file::fullName`, `basename::fullName`, or `fullName`.
   - **`resolveTrace(test)`**: optional explicit association resolver for normalized Jest assertion results.
   - **`onDiagnostic(diagnostic)`**: observes non-fatal reporter/artifact failures.
@@ -294,7 +319,7 @@ reporters: [
 ],
 ```
 
-Artifacts are safe structural summaries. They include bounded test identity, status, trace run id, and trace filename, but they do not read or embed raw trace contents, prompts, generated outputs, request/response bodies, headers, API keys, secrets, or tool payloads. Reporter/artifact failures are diagnostics and do not replace original Jest failures.
+Artifacts are safe structural summaries. The reporter writes a shared `schemaVersion: "0.1"` manifest wrapper with package metadata, generated time, framework, test results, artifact descriptors, relative paths, and redaction profile. It includes bounded test identity, status, trace run id, and trace filename, but it does not read or embed raw trace contents, prompts, generated outputs, request/response bodies, headers, API keys, secrets, or tool payloads. Reporter/artifact failures are diagnostics and do not replace original Jest failures.
 
 ## 14. Experimental `@agent-inspect/openai-agents` APIs
 
@@ -497,7 +522,47 @@ The checks API is experimental in v1.x. The `agent-inspect check` CLI uses this
 
 Recipes: [deterministic-ci-checks](../examples/recipes/deterministic-ci-checks/README.md) for check/baseline/artifact workflows, and [test-reporter-artifacts](../examples/recipes/test-reporter-artifacts/README.md) for Vitest/Jest reporter configuration patterns.
 
-## 22. Experimental local explain APIs (v1.9)
+## 22. Experimental `@agent-inspect/eval` APIs (v2.1)
+
+`@agent-inspect/eval` is an optional package for deterministic local evals over existing traces. It consumes normalized reader output or local trace paths, returns stable JSON-compatible results, and does not call model providers, upload traces, replay agents, or create hosted datasets.
+
+Import from `@agent-inspect/eval`:
+
+```ts
+import { checks, evalRun, renderEvalMarkdown } from "@agent-inspect/eval";
+```
+
+- **`evalRun(input, options?)`**: runs selected eval rules over a local trace path or `TraceReadResult`.
+- **`checks`**: built-in deterministic rule factories for run status, tool usage, duration, depth, retries, token totals, failed steps, retrieval-before-generation, decision metadata, context overlap, quote overlap, citation presence, required source IDs, answer length bounds, and banned unsupported phrases.
+- **`renderEvalMarkdown(result)`**: renders a deterministic Markdown summary suitable for local CI logs, PR text, or artifact files after review.
+- Result types include **`EvalRunResult`**, **`EvalFinding`**, **`EvalDiagnostic`**, and **`EvalRule`**.
+
+Findings are designed for CI output: they include rule IDs, expected/actual structural summaries, and evidence paths. They should not include raw prompt, answer, context, request/response, header, API key, secret, or full tool payload values.
+
+CLI wrapper: `agent-inspect eval <trace-path-or-run-id> --require-success --json`.
+
+Recipes: [eval-local-checks](../examples/recipes/eval-local-checks/README.md) and [eval-ci-artifacts](../examples/recipes/eval-ci-artifacts/README.md).
+
+## 23. Experimental `@agent-inspect/redact` APIs (v2.1)
+
+`@agent-inspect/redact` is an optional package for reusable local redaction. It powers the root CLI `redact` workflow and shared trace-safety integrations. Redaction operates on local values/files and returns a redacted copy; it does not mutate the source object, upload content, or claim compliance-grade DLP.
+
+Import from `@agent-inspect/redact`:
+
+```ts
+import { createRedactor, redact } from "@agent-inspect/redact";
+```
+
+- **`redact(value, options?)`**: returns `{ value, findings, redacted, profile }` for a redacted copy.
+- **`createRedactor(options?)`**: creates a reusable redactor with profile, custom detectors, and custom rules.
+- **Profiles**: `local`, `share`, and `strict`.
+- **Findings**: detector id, path, action, severity, and bounded preview metadata where applicable.
+
+CLI wrapper: `agent-inspect redact <trace-or-file> --profile share --json`.
+
+Recipe: [redact-share-safe-file](../examples/recipes/redact-share-safe-file/README.md).
+
+## 24. Experimental local explain APIs (v1.9)
 
 `buildLocalExplanation()` creates a deterministic local explanation payload from a reader-selected `InspectRunTree`. It performs no network I/O, does not call model providers, and separates observed facts from deterministic inference labels.
 
@@ -522,7 +587,7 @@ Provider design gate:
 - Provider prompts must use redacted facts only, label inferred claims, and must not request raw chain-of-thought.
 - Provider packages or SDKs must not become root/core runtime dependencies.
 
-## 23. Experimental `@agent-inspect/harness` APIs
+## 25. Experimental `@agent-inspect/harness` APIs
 
 `@agent-inspect/harness` is a private experimental workspace package during the v1.9 release train. It provides a no-framework fixture runner for local targets and recipes; first public package publication remains a manual maintainer gate.
 
@@ -548,23 +613,23 @@ The harness package does not add root/core dependencies, does not upload traces,
 
 Recipes: [harness-basic](../examples/recipes/harness-basic/README.md) and [harness-adapter-local](../examples/recipes/harness-adapter-local/README.md).
 
-## 24. Deprecated APIs
+## 26. Deprecated APIs
 
 No deprecated APIs are declared as of 1.4.0.
 
-## 25. Removal / deprecation policy
+## 27. Removal / deprecation policy
 
 - Stable APIs are not removed within the current major version.
 - If removal is necessary, the API should be **deprecated** first, documented, and kept for a reasonable window (target: at least one minor line) unless security requires faster action.
 
-## 26. Backward compatibility policy
+## 28. Backward compatibility policy
 
 - Manual trace JSONL (`schemaVersion: "0.1"`) remains readable.
 - Additive schema changes are allowed in minor versions.
 - Breaking changes require a major version.
 - Unknown fields should be ignored where safe.
 
-## 25. Examples
+## 29. Examples
 
 ### Minimal manual trace
 

package/docs/CLI.md

@@ -28,9 +28,12 @@ Core commands:
 - `open` — read supported local trace files, directories, or stdin through the canonical reader pipeline
 - `migrate` — convert one local AgentInspect JSONL file to schema 1.0 with dry-run or explicit output
 - `check` — run deterministic local trace checks with stable JSON and exit codes
+- `eval` — run deterministic local evals over existing traces
+- `redact` — redact a local JSON/JSONL file or trace copy
 - `scan` — best-effort local safety scan for trace capture risks
 - `verify-safe` — best-effort local trace safety verification
 - `artifacts` — create safe local CI trace artifact bundles and optional step summaries
+- `ci-summary` — summarize local reporter artifact manifests for CI
 - `diff` — compare two manual traces (local, read-only)
 - `timeline` — chronological view of one run (local JSONL)
 - `stats` — local aggregate stats over a trace directory
@@ -58,6 +61,12 @@ Exception: `check` uses CI-oriented semantic exit codes:
 - **3**: trace input could not be read
 - **4**: unsupported or ambiguous trace format
 
+Exception: `eval` uses local eval semantic exit codes:
+
+- **0**: all selected eval rules passed
+- **1**: eval ran and at least one error-severity rule failed
+- **2**: invalid arguments, invalid config, unreadable input, unsupported input, ambiguous input, or run-selection errors
+
 Exception: `scan` and `verify-safe` use local safety status exit codes:
 
 - **0**: status is SAFE or SAFE WITH WARNINGS
@@ -79,6 +88,8 @@ Many commands support `--json` for scripting. JSON output is intended to be:
 - Log-derived output includes **confidence** labels and avoids inventing parent-child relationships.
 - Redaction defaults are conservative (e.g. `authorization`, `cookie`, `token`, `apiKey`, `password`, `secret`, `email`).
 - Exported payloads are **redacted by default** unless explicitly configured otherwise.
+- `eval` is deterministic and local-only. It does not replay agents, call model providers, upload traces, or create hosted datasets.
+- `redact` writes or prints a redacted copy. It does not mutate source trace files.
 - `scan` and `verify-safe` are best-effort local checks, not compliance, privacy, security, or regulatory certifications.
 - `artifacts` renders structural summaries and check evidence only; it does not include raw prompt/output bodies, request/response bodies, headers, API keys, secrets, or full tool payloads.
 
@@ -322,7 +333,85 @@ npx agent-inspect check trace.jsonl --max-duration-ms 30000 --required-tool sear
 
 Recipe: [examples/recipes/deterministic-ci-checks](../examples/recipes/deterministic-ci-checks/README.md)
 
-### 6.10 `scan` and `verify-safe`
+### 6.10 `eval`
+
+Run deterministic local evals against an existing trace. This command reads through the same local reader pipeline as `open` and `check`; it does not rerun agents, call models, upload traces, mutate inputs, or create a hosted dataset.
+
+```bash
+agent-inspect eval <trace-path-or-run-id> [options]
+```
+
+Options:
+
+- `--dir <path>`: trace directory for run-id lookup
+- `--format <agent-inspect-jsonl|openinference-json|otlp-json>`: explicit trace input format
+- `--run <run-id>`: select a run when input contains multiple runs
+- `--config <path>`: eval config (`.json`, `.js`, `.mjs`, or `.cjs`); TypeScript configs are rejected until an explicit loader is approved
+- `--json`: print deterministic JSON eval result
+- `--markdown`: print deterministic Markdown eval summary
+- `--require-success`: require the selected run to complete successfully
+- `--required-tool <name>`: require a tool name (repeatable)
+- `--forbid-tool <name>` / `--forbidden-tool <name>`: forbid a tool name (repeatable)
+- `--max-duration-ms <number>`, `--max-depth <number>`, `--max-retries <number>`, `--max-total-tokens <number>`
+- `--require-retrieval-before-generation`
+- `--required-decision-metadata <key>`: require decision metadata (repeatable)
+- `--context-overlap`, `--min-context-overlap <number>`, `--min-shared-terms <number>`
+- `--quote-overlap`
+- `--citation-presence`
+- `--required-source-id <id>`: require a source id in context or citations (repeatable)
+- `--min-answer-characters <number>`, `--max-answer-characters <number>`, `--min-answer-words <number>`, `--max-answer-words <number>`
+- `--banned-phrase <text>`: ban unsupported-answer phrasing (repeatable)
+
+Example config:
+
+```json
+{
+  "eval": {
+    "requireSuccess": true,
+    "requiredTools": ["searchDocs"],
+    "forbiddenTools": ["deleteAccount"],
+    "citationPresence": true,
+    "contextOverlap": { "minOverlap": 0.2 },
+    "requiredSourceIds": ["policy-30-day"]
+  }
+}
+```
+
+Examples:
+
+```bash
+npx agent-inspect eval fixtures/traces-v0.2/manual-basic.jsonl --require-success --json
+npx agent-inspect eval trace.jsonl --forbid-tool deleteAccount --markdown
+npx agent-inspect eval trace.jsonl --config agent-inspect.eval.json --json
+```
+
+Recipes: [eval-local-checks](../examples/recipes/eval-local-checks/README.md) and [eval-ci-artifacts](../examples/recipes/eval-ci-artifacts/README.md).
+
+### 6.11 `redact`
+
+Redact a local JSON or JSONL trace/file. The command prints or writes a redacted copy and reports bounded findings; it does not mutate the source file or upload content.
+
+```bash
+agent-inspect redact <trace-or-file> [options]
+```
+
+Options:
+
+- `--dir <path>`: trace directory for run-id lookup
+- `--profile <local|share|strict>`: redaction profile (default `share`)
+- `-o, --output <path>`: write redacted content to a file
+- `--json`: print deterministic JSON wrapper with findings
+
+Examples:
+
+```bash
+npx agent-inspect redact trace.jsonl --profile share --json
+npx agent-inspect redact trace.jsonl --profile strict -o trace.share.jsonl
+```
+
+Recipe: [redact-share-safe-file](../examples/recipes/redact-share-safe-file/README.md).
+
+### 6.12 `scan` and `verify-safe`
 
 Run best-effort local safety verification for supported trace inputs. These commands are local and read-only: they do not rerun agents, call models, upload traces, mutate input files, or certify compliance.
 
@@ -361,7 +450,7 @@ npx agent-inspect verify-safe minimal-success --dir fixtures/traces
 npx agent-inspect verify-safe trace.jsonl --max-string-length 8192 --json
 ```
 
-### 6.11 `artifacts`
+### 6.13 `artifacts`
 
 Create deterministic local CI artifacts for supported trace inputs. This command is local and read-only for trace inputs: it does not rerun agents, call models, upload files, use GitHub APIs, or mutate repository state. It writes only to `--output-dir` and, when requested, a local step-summary file.
 
@@ -401,7 +490,33 @@ npx agent-inspect artifacts candidate.jsonl --baseline baseline.jsonl --output-d
 
 Recipe and sample workflow: [examples/recipes/deterministic-ci-checks](../examples/recipes/deterministic-ci-checks/README.md)
 
-### 6.12 `diff`
+### 6.14 `ci-summary`
+
+Summarize local Vitest/Jest reporter artifact manifests into deterministic Markdown or JSON. This command reads shared `schemaVersion: "0.1"` manifest JSON files only, including the reporter package wrapper emitted by the workspace reporters. It does not read trace contents, rerun tests, upload artifacts, call GitHub APIs, or mutate repository state. `--output` and `--github-summary` write local files.
+
+```bash
+agent-inspect ci-summary <manifest...> [options]
+```
+
+Options:
+
+- `-o, --output <path>`: write the Markdown summary to a local file
+- `--github-summary <path>`: append the Markdown summary to a local file, such as `$GITHUB_STEP_SUMMARY`
+- `--json`: print deterministic JSON summary
+
+Example:
+
+```bash
+npx agent-inspect ci-summary .agent-inspect/jest-artifacts/tests/**/report.json \
+  --output ./artifacts/reporter-summary.md \
+  --github-summary "$GITHUB_STEP_SUMMARY"
+```
+
+Reporter artifact paths in the summary are kept relative and validated conservatively. The summary includes bounded package/framework metadata, test identity, status counts, trace filenames, artifact paths, redaction profiles, and diagnostic counts only.
+
+Recipe and sample workflow: [examples/recipes/github-actions-artifact](../examples/recipes/github-actions-artifact/README.md)
+
+### 6.15 `diff`
 
 Compare two manual trace runs. Diff is **local** and **read-only** (does not rerun agents).
 
@@ -465,7 +580,7 @@ Differences:
 
 More examples, including timing-only and structure-only diffs, are in `docs/DIFF.md`.
 
-### 6.13 `timeline`
+### 6.16 `timeline`
 
 Chronological step list for one manual trace. Read-only; does not mutate JSONL files.
 
@@ -481,7 +596,7 @@ Options:
 
 ![Timeline with slow-step focus](../assets/demos/timeline.gif)
 
-### 6.14 `stats`
+### 6.17 `stats`
 
 Local aggregate statistics over trace files in a directory. Read-only.
 
@@ -501,7 +616,7 @@ Options:
 
 Use `--correlation-id` or `--group-id` to filter runs by `run_started` metadata (see [API.md](./API.md)).
 
-### 6.15 `search`
+### 6.18 `search`
 
 Deterministic search over local traces (substring / exact filters). No semantic search.
 
@@ -531,7 +646,7 @@ npx agent-inspect search --duration ">100ms" --json
 
 ![Search traces by status error](../assets/demos/search.gif)
 
-### 6.16 `what`
+### 6.19 `what`
 
 Concise human-readable summary of one local trace run. Read-only; accepts v0.1 manual JSONL and v0.2 persisted-event JSONL through the shared dual-format normalization path. Vocabulary: [TRACE-VOCABULARY-V1.5.md](./proposals/TRACE-VOCABULARY-V1.5.md).
 
@@ -560,7 +675,7 @@ Outcome: Completed successfully.
 Slowest: plan (100ms, logic)
 ```
 
-### 6.17 `report`
+### 6.20 `report`
 
 Generate a local inspection report combining **what happened**, **timeline**, and **execution tree** sections. The command reads local v0.1 manual JSONL and v0.2 persisted-event JSONL through the shared dual-format normalization path without mutating them. Distinct from `export` (which targets shareable tree snapshots and standards formats).
 
@@ -585,7 +700,7 @@ Example:
 npx agent-inspect report minimal-success --dir fixtures/traces --format html -o report.html
 ```
 
-### 6.18 `explain`
+### 6.21 `explain`
 
 Explain a local trace using deterministic facts and local inference labels. This command reads through the same local reader pipeline as `open` / `check`; it does not call a model provider, upload traces, replay agents, or mutate input files.
 

package/docs/COMPARE.md

@@ -1,6 +1,6 @@
 ## Compare AgentInspect
 
-AgentInspect is a local-first execution-tree debugger for TypeScript AI agents. It’s designed for inner-loop debugging and quick inspection — not as a replacement for hosted observability, evaluation, or production monitoring platforms.
+AgentInspect is a local-first execution-tree debugger for TypeScript AI agents. It’s designed for inner-loop debugging, deterministic local eval heuristics, redaction, and quick inspection — not as a replacement for hosted observability, dataset evaluation, or production monitoring platforms.
 
 ## AgentInspect vs console.log
 
@@ -16,7 +16,7 @@ LangSmith is a hosted/platform workflow for tracing, evaluation, and observabili
 AgentInspect is local-first CLI debugging:
 
 - Use AgentInspect to debug locally before/alongside LangSmith when iterating on agent logic.
-- AgentInspect does not provide hosted dashboards, dataset/eval workflows, or production tracing pipelines.
+- AgentInspect does not provide hosted dashboards, dataset/eval management workflows, or production tracing pipelines.
 
 ## AgentInspect vs Langfuse
 
@@ -33,7 +33,7 @@ Braintrust is strong for evals, regressions, datasets, and production AI quality
 
 AgentInspect is lighter and local-first:
 
-- Use AgentInspect to understand a single run locally.
+- Use AgentInspect to understand a single run locally and run deterministic trace checks/eval heuristics before sharing artifacts.
 - Use Braintrust when you want repeatable evals, comparisons at scale, and production quality workflows.
 
 ## AgentInspect vs Phoenix / OpenInference
@@ -61,9 +61,10 @@ AgentInspect avoids SDK/collector setup for local debugging:
 | --- | --- |
 | Local agent debugging | Strong fit |
 | No-account CLI tracing | Strong fit |
+| Deterministic local eval heuristics | Good fit |
+| Share-safe local redaction copy | Good fit |
 | Production dashboards | Not the goal |
 | Hosted eval datasets | Not the goal |
 | Prompt management | Not the goal |
 | Standards-aligned local export | Partial (compatibility-oriented) |
 | Full observability platform | Use a dedicated platform |
-

package/docs/GETTING-STARTED.md

@@ -204,17 +204,34 @@ agent-inspect search --dir ./.agent-inspect --status error --limit 10
 
 For CI artifact workflows, see [CI-ARTIFACTS.md](./CI-ARTIFACTS.md) and [github-actions-artifact recipe](../examples/recipes/github-actions-artifact/).
 
-## 10. Diff two runs
+## 10. Run local evals and redact share copies
+
+After a trace exists, run deterministic eval checks without replaying the agent or calling a model provider:
+
+```bash
+agent-inspect eval minimal-success --dir fixtures/traces --require-success --json
+agent-inspect eval trace.jsonl --forbid-tool deleteAccount --citation-presence --json
+```
+
+Before attaching a trace or JSON artifact to a PR, issue, or support thread, create a redacted local copy:
+
+```bash
+agent-inspect redact trace.jsonl --profile share --json
+```
+
+Recipes: [eval-local-checks](../examples/recipes/eval-local-checks/), [redact-share-safe-file](../examples/recipes/redact-share-safe-file/), and [eval-ci-artifacts](../examples/recipes/eval-ci-artifacts/).
+
+## 11. Diff two runs
 
 ```bash
 agent-inspect diff minimal-success minimal-error --dir fixtures/traces
 ```
 
-## 11. Try recipes
+## 12. Try recipes
 
 See `examples/recipes/README.md`.
 
-## 12. Optional framework adapters
+## 13. Optional framework adapters
 
 See [ADAPTERS.md](./ADAPTERS.md) for AI SDK local telemetry, OpenAI Agents local-only processing, and LangChain callbacks.
 
@@ -228,7 +245,7 @@ pnpm add @agent-inspect/langchain
 
 See [examples/08-langchain-adapter](../examples/08-langchain-adapter/README.md) and [docs/ADAPTERS.md](./ADAPTERS.md).
 
-## 13. Optional TUI
+## 14. Optional TUI
 
 `@agent-inspect/tui` is optional and **experimental**. The CLI can invoke it with:
 
@@ -236,16 +253,17 @@ See [examples/08-langchain-adapter](../examples/08-langchain-adapter/README.md)
 agent-inspect view <runId> --tui
 ```
 
-## 14. Safety notes
+## 15. Safety notes
 
 - Nothing uploads by default; core tracing, readers, checks, and exports are local-first.
+- Eval and redaction commands read local inputs and do not call provider APIs or hosted services.
 - Redaction is on by default for log-derived attributes, **manual trace metadata (before disk)**, and exports. Pass `redact: false` to opt out of manual metadata redaction.
 - Export redaction shapes a local copy and does not mutate the source trace; review exported files before sharing.
 - Persisted events are size-bounded by default (see `docs/API.md`).
 - Confidence labels are required to keep attribution honest.
 - AgentInspect is for local debugging, not production monitoring.
 
-## 15. Next docs
+## 16. Next docs
 
 - [docs/API.md](./API.md)
 - [docs/CLI.md](./CLI.md)

package/docs/KNOWN-ISSUES.md

@@ -12,6 +12,12 @@ AgentInspect is **local-first** and **CLI-first**. These behaviors are intention
 - **OpenInference** and **OTLP JSON** exports are **compatibility-oriented** and **experimental**. Validate against your target collector or backend before relying on them.
 - Exports generate **strings/files locally** only—there is **no** automatic upload.
 
+## Eval and redaction
+
+- **`agent-inspect eval` is deterministic local eval, not hosted evaluation.** Built-in rules are structural checks and grounding heuristics over existing traces. They do not call model providers, manage datasets, replay agents, or perform semantic LLM judging.
+- **TypeScript eval configs are not loaded by default.** Use `.json`, `.js`, `.mjs`, or `.cjs` until an explicit loader path is approved.
+- **`agent-inspect redact` creates a local redacted copy.** It does not mutate source traces, encrypt files, or guarantee compliance-grade detection. Review the exact output before sharing.
+
 ## Readers and `open`
 
 - **OpenInference** and **OTLP JSON** readers are **compatibility-oriented** and **experimental**. They normalize local JSON payloads into AgentInspect inspection trees and may warn on unsupported semantic fields.
@@ -103,6 +109,7 @@ pnpm compat:smoke
 
 - `agent-inspect artifacts --github-summary` writes a local step-summary file only. It does not call GitHub APIs, open PR comments, upload artifacts, or mutate repository state.
 - Baseline checks compare normalized structural facts from explicit candidate and baseline inputs. They are useful for CI regression evidence, not replay or semantic eval scoring.
+- v2.1 eval/redact adoption recipes are local and deterministic: [eval-local-checks](../examples/recipes/eval-local-checks/), [redact-share-safe-file](../examples/recipes/redact-share-safe-file/), and [eval-ci-artifacts](../examples/recipes/eval-ci-artifacts/).
 
 ### What to include in a bug report