agent-inspect 4.3.0 → 5.0.0

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 5.0.0
+
+### Minor Changes
+
+- c5e3b16: Add trace suite config (`agent-inspect.suite.json`) with `suite init`, `validate`, `list`, `run`, and `report` commands for local CI trajectory checks.
+
+## 4.4.0
+
+### Minor Changes
+
+- 42635d8: Add observed outcomes (`observeOutcome`) with OUTCOME events, report/check/search integration, and redaction-bounded evidence fields.
+
 ## 4.3.0
 
 ### Minor Changes

package/docs/API.md

@@ -84,6 +84,7 @@ import {
   - **`step.llm(model, fn)`**: convenience wrapper (`type: "llm"`, `metadata.model`).
   - **`step.tool(toolName, fn)`**: convenience wrapper (`type: "tool"`, `metadata.toolName`).
 - **`observe(agent, options?)`**: proxy wrapper that traces top-level `run` / `execute` / `invoke` methods via `inspectRun`.
+- **`observeOutcome(name, options)`** (v4.4.0+): records an observed outcome (`outcome_observed`) inside an active `inspectRun` context. Requires `expectation` and `status` (`passed` | `failed` | `unknown` | `skipped`); optional `method`, `actual`, and `evidence`. Outside a run → warn and no-op (never throws). `actual` / `evidence` are bounded and redacted before disk.
 - **`getCurrentCorrelationMetadata()`**: returns active run correlation fields (`correlationId`, `requestId`, `decisionId`, `groupId`) inside `inspectRun` / `maybeInspectRun`; `undefined` outside a traced run or when none were set.
 - **`RedactionProfile`**: `"local" | "share" | "strict"` — see `redactionProfile` on `InspectRunOptions` and `ExportOptions`.
 - **`resolveRedactionProfile(profile?)`**: advanced helper available from `agent-inspect/advanced`; resolves profile extra keys and metadata caps for integrations.
@@ -109,6 +110,7 @@ These APIs support local workflows like listing traces, extracting metadata/summ
   - `RunCompletedEvent` (`event: "run_completed"`)
   - `StepStartedEvent` (`event: "step_started"`)
   - `StepCompletedEvent` (`event: "step_completed"`)
+  - `OutcomeObservedEvent` (`event: "outcome_observed"`, v4.4.0+)
 - Related types: `StepType`, `StepStatus`, `RunStatus`, `ErrorInfo`, `StepMetadata`, `TokenMetadata`
 
 ### Log-derived normalized model (stable as a model, not necessarily the parsing APIs)
@@ -460,6 +462,7 @@ Public methods:
 - **`step(name, fn, options?)`**: writes nested step lifecycle events when called inside the same inspector's run context; outside a context it passes through.
 - **`tool(name, fn, options?)`** / **`llm(name, fn, options?)`**: convenience wrappers that set `type` and metadata.
 - **`observe(name, fn, options?)`**: returns an async wrapper that records the function call as an inspector step.
+- **`observeOutcome(name, options)`** (v4.4.0+): writes `outcome_observed` / `OUTCOME` events through the configured writer when inside an inspector run context.
 - **`getDiagnostics()`**: returns instrumentation error counts and writer stats without requiring direct runtime access.
 - **`flush()`** / **`close()`**: delegate to the configured writer through the runtime.
 

package/docs/CLI.md

@@ -317,6 +317,7 @@ Options:
 - `--correlate-group`: when using `--session`, also match synthetic `group:` session keys
 - `--guardrails <rule>`: optional deterministic guardrail rules (`banned-phrase`, `pii-leak`, `prompt-injection`, …); repeatable
 - `--circuit <rule>`: optional circuit analyzers (`same-tool-repetition`, `max-retries`, …); repeatable
+- `--fail-on-observation <status>`: add `outcome.status` rule; repeatable (`failed`, `passed`, `unknown`, `skipped`; default when flag present without value: `failed`)
 
 By default, `check` runs `run.status`. Additional built-in rules can be selected with `--rule` or config when their options are available.
 
@@ -667,6 +668,7 @@ Options:
 - `--limit <number>` — default 50
 - `--session <id>` — limit to runs in one workflow session
 - `--correlate-group` — when using `--session`, also match synthetic `group:` keys
+- `--observation <status>` — filter runs with observed outcomes matching status (`passed`, `failed`, `unknown`, `skipped`)
 - `--json`
 
 Examples:
@@ -779,6 +781,7 @@ Options:
 - `-o, --output <path>` — write to file
 - `--json` — JSON wrapper (includes `content` when writing to stdout)
 - `--include-attributes` — bounded attributes in the execution tree section
+- `--section <name>` — include report sections; repeatable (`what`, `timeline`, `tree`, `observations`; default: all except when narrowed)
 - `--no-errors` — omit error details from the execution tree section
 - `--no-correlation` — omit correlation ids from what section
 - `--redaction-profile <local|share|strict>` — key-based redaction profile applied to the complete report (default `local`); review output before sharing
@@ -893,6 +896,40 @@ npx agent-inspect bundle --since 24h --dir ./.agent-inspect --json
 
 Recipe: [shareable-bundle-basic](../examples/recipes/shareable-bundle-basic/README.md).
 
+### 6.25 `suite`
+
+Define and run **local trace suites** for CI trajectory checks (v5.0+). Suites read existing traces only — no agent replay, no model calls, no upload.
+
+```bash
+agent-inspect suite init
+agent-inspect suite validate [--config path]
+agent-inspect suite list [--config path]
+agent-inspect suite run [--config path] [--json] [--markdown] [-o dir]
+agent-inspect suite report --input <suite-run.json> [--format markdown|json]
+```
+
+Default config file: `agent-inspect.suite.json` in the current directory.
+
+Options (shared):
+
+- `--config <path>` — suite config (`.json`, `.js`, `.mjs`, `.cjs`)
+
+`run` also supports:
+
+- `-o, --output <dir>` — write JSON run artifact (default: `.agent-inspect/suite-runs`)
+- `--json` / `--markdown` — structured or readable output
+
+Case fields include `requireTools`, `forbidTools`, `maxDurationMs`, and `expectedObservations` (outcome names that must be `passed`).
+
+Example:
+
+```bash
+npx agent-inspect suite init
+npx agent-inspect suite run --config fixtures/configs/outcome-suite.suite.json --json
+```
+
+Recipe: [trace-suite-basic](../examples/recipes/trace-suite-basic/README.md).
+
 ## 7. Optional TUI behavior
 
 `view --tui` delegates to `@agent-inspect/tui` and requires an interactive terminal. If the package is not installed, the CLI prints a short install hint.

package/docs/SCHEMA.md

@@ -36,6 +36,7 @@ The stable event names are:
 - `run_completed`
 - `step_started`
 - `step_completed`
+- `outcome_observed` (v4.4.0+)
 
 There is **no** `step_failed` event. Failures are represented by `step_completed` with `status: "error"`.
 
@@ -111,6 +112,31 @@ Every trace line contains:
 }
 ```
 
+### 3.5 `outcome_observed` (v4.4.0+)
+
+Records whether an external side effect matched expectations. Distinct from step/tool success.
+
+```ts
+{
+  schemaVersion: "0.1",
+  event: "outcome_observed",
+  timestamp: number,
+  runId: string,
+  outcomeId: string,
+  parentId?: string,
+  name: string,
+  expectation: string,
+  status: "passed" | "failed" | "unknown" | "skipped",
+  method?: "dom" | "accessibility" | "snapshot" | "network" | "storage"
+    | "filesystem" | "database" | "queue" | "custom",
+  actual?: unknown,
+  evidence?: Record<string, unknown>,
+  observedAt: number
+}
+```
+
+`actual` and `evidence` are bounded and redacted before disk like step metadata. See [OBSERVED-OUTCOMES-V4.4.md](./proposals/OBSERVED-OUTCOMES-V4.4.md).
+
 ## 4. Error representation
 
 - Errors are structured as `{ message, stack? }`.
@@ -173,7 +199,7 @@ interface InspectEvent {
   runId: string;
   parentId?: string;
   name: string;
-  kind: "RUN" | "AGENT" | "LLM" | "TOOL" | "CHAIN" | "RETRIEVER" | "DECISION" | "RESULT" | "ERROR" | "LOGIC" | "LOG";
+  kind: "RUN" | "AGENT" | "LLM" | "TOOL" | "CHAIN" | "RETRIEVER" | "DECISION" | "RESULT" | "ERROR" | "LOGIC" | "LOG" | "OUTCOME";
   timestamp: number;
   status?: "running" | "ok" | "error";
   durationMs?: number;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-inspect",
-  "version": "4.3.0",
+  "version": "5.0.0",
   "license": "MIT",
   "type": "module",
   "description": "Trace, check, and safely share TypeScript AI-agent runs locally — no account, no upload, metadata-only by default",