agent-inspect 6.25.1 → 6.26.0

package/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 6.26.0
+
+### Minor Changes
+
+- ec2ebbf: Outcome-aware behavioral sessions: `--preset behavioral-session` scores OUTCOME events without collapsing graceful tool errors into run failure, plus a synthetic MCP dual-axis recipe (#362).
+
 ## 6.25.1
 
 ### Patch Changes

package/README.md

@@ -212,7 +212,7 @@ The root package is enough for custom capture, the CLI, checks, and Evidence wor
 
 ## Status and documentation
 
-**Current published baseline:** **6.25.1** · persisted schema `1.0` · Node.js `>=20` · MIT.
+**Current published baseline:** **6.26.0** · persisted schema `1.0` · Node.js `>=20` · MIT.
 
 Legacy v0.1 and v0.2 traces remain readable. Check the npm badge and [changelog](CHANGELOG.md) for the current published version.
 

package/docs/CLI.md

@@ -320,7 +320,8 @@ Options:
 - `--guardrails <rule>`: optional deterministic guardrail rules (`banned-phrase`, `pii-leak`, `prompt-injection`, …); repeatable
 - `--circuit <rule>`: optional circuit analyzers (`same-tool-repetition`, `max-retries`, …); repeatable
 - `--fail-on-observation <status>`: add `outcome.status` rule; repeatable (`failed`, `passed`, `unknown`, `skipped`; default when flag present without value: `failed`)
-- `--preset <trajectory|safety|comprehensive>`: additive check preset (does not change the default when omitted)
+- `--preset <trajectory|safety|comprehensive|behavioral-session>`: additive check preset (does not change the default when omitted)
+  - `behavioral-session` (6.26): require harness completion + score OUTCOME events (`--fail-on-observation failed` by default); does **not** treat every TOOL `error` as a failed run
   - `trajectory`: completion/structure/relationship focus; excludes share-safety findings
   - `safety`: raw-content / secret / redaction focus
   - `comprehensive`: union of trajectory and safety

package/docs/SESSIONS-AND-OUTCOMES.md

@@ -8,11 +8,26 @@ Useful CLI entry points: `sessions`, `search`, activity views (see [CLI.md](./CL
 
 Attempt identity for contracts (6.22+): `operationId`, `attemptId`, `attemptNumber`, `retryOf`, `fallbackOf`, `idempotencyKey`. Retry safety evaluation prefers this identity over “saw a prior ok” (see [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md), corrected in 6.25.1).
 
-## Observed outcomes
+## Observed outcomes (dual-axis)
 
 Outcomes record what the agent produced or decided at a high level for later review and gates. They remain local JSONL-derived evidence.
 
-**Dual-axis reminder:** tool/run **execution status** (`ok` / `error`) is independent of behavioral **outcome** (`passed` / `failed` / `unknown`). A graceful tool rejection can be `status: "error"` while the expected behavioral outcome is `passed`. Expanding this for MCP behavioral sessions is the focus of **6.26.0** (#362).
+| Axis | Where | Values | Meaning |
+| --- | --- | --- | --- |
+| Execution | TOOL / RUN `status` | `ok` / `error` / … | What happened at runtime (MCP `isError` stays `error`) |
+| Behavior | OUTCOME `outcomeStatus` | `passed` / `failed` / `unknown` / `skipped` | Whether the result matched the test expectation |
+
+A graceful tool rejection can be TOOL `status: "error"` while the expected behavioral OUTCOME is `passed`. Do **not** rewrite tool errors to `ok` to make a gate green.
+
+### CLI (6.26)
+
+```bash
+npx agent-inspect check <run> --preset behavioral-session --json
+```
+
+Preset selects harness completion + `outcome.status` and defaults `--fail-on-observation failed`. Recipe: [examples/recipes/mcp-behavioral-session](../examples/recipes/mcp-behavioral-session/).
+
+Issue **#362**: external sanitized fixtures remain `BLOCKED_ON_EXTERNAL_FIXTURE` until reviewed; the synthetic recipe ships first.
 
 ## Limitations
 
@@ -21,4 +36,4 @@ Outcomes record what the agent produced or decided at a high level for later rev
 - Studio session pages may still be thinner than APIs — Studio is Beta
 - AgentInspect does not execute retries or mutate source sessions
 
-Related: [WORKSPACE.md](./WORKSPACE.md) · [USE-CASES.md](./USE-CASES.md)
+Related: [WORKSPACE.md](./WORKSPACE.md) · [USE-CASES.md](./USE-CASES.md) · [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-inspect",
-  "version": "6.25.1",
+  "version": "6.26.0",
   "license": "MIT",
   "type": "module",
   "description": "Local evidence debugger and trajectory-test toolkit for TypeScript AI agents — execution trees, TraceContract checks, Evidence v2, and read-only MCP",

package/packages/cli/dist/index.cjs

@@ -13063,7 +13063,7 @@ var init_src = __esm({
 });
 
 // package.json
-var version = "6.25.1";
+var version = "6.26.0";
 
 // packages/cli/src/list.ts
 init_advanced();
@@ -19791,9 +19791,9 @@ function unionCheckSelect(input3) {
 function resolvePreset(preset, context = {}) {
   if (preset === void 0 || preset.trim() === "") return void 0;
   const name = preset.trim().toLowerCase();
-  if (name !== "trajectory" && name !== "safety" && name !== "comprehensive") {
+  if (name !== "trajectory" && name !== "safety" && name !== "comprehensive" && name !== "behavioral-session") {
     throw new Error(
-      `Unknown --preset "${preset}". Use trajectory, safety, or comprehensive.`
+      `Unknown --preset "${preset}". Use trajectory, safety, comprehensive, or behavioral-session.`
     );
   }
   const trajectorySelect = [
@@ -19828,6 +19828,15 @@ function resolvePreset(preset, context = {}) {
       select: safetySelect
     };
   }
+  if (name === "behavioral-session") {
+    return {
+      requireCompleted: true,
+      enableSafetyRedaction: false,
+      enableStructureRelationshipDefaults: false,
+      select: ["run.requireCompleted", "outcome.status", "structure.orphan"],
+      failOnObservation: "failed"
+    };
+  }
   const select = [.../* @__PURE__ */ new Set([...trajectorySelect, ...safetySelect])];
   return {
     requireCompleted: true,
@@ -20261,7 +20270,8 @@ function applyResolvedPreset(config, options, resolved) {
     config: { checks: checks2 },
     options: {
       ...options,
-      ...resolved.requireCompleted ? { requireCompleted: true } : {}
+      ...resolved.requireCompleted ? { requireCompleted: true } : {},
+      ...resolved.failOnObservation !== void 0 && (options.failOnObservation === void 0 || options.failOnObservation.trim() === "") ? { failOnObservation: resolved.failOnObservation } : {}
     }
   };
 }
@@ -20407,7 +20417,13 @@ function isSafetyFinding(ruleId) {
 }
 function printPresetClassSummary(result, preset) {
   const name = preset?.trim().toLowerCase();
-  if (name !== "trajectory" && name !== "safety" && name !== "comprehensive") {
+  if (name !== "trajectory" && name !== "safety" && name !== "comprehensive" && name !== "behavioral-session") {
+    return;
+  }
+  if (name === "behavioral-session") {
+    console.log(
+      `Behavioral session: ${result.status === "pass" ? "PASS" : "FAIL"} (outcomes scored; tool errors may be expected)`
+    );
     return;
   }
   const hasSafetyFindings = result.findings.some(
@@ -26176,7 +26192,7 @@ function createCliProgram() {
     "run optional circuit rules (repeatable): same-tool-repetition, max-retries, ...",
     (value, previous = []) => [...previous, value]
   ).addOption(
-    new commander.Option("--preset <name>", "additive check preset").choices(["trajectory", "safety", "comprehensive"])
+    new commander.Option("--preset <name>", "additive check preset").choices(["trajectory", "safety", "comprehensive", "behavioral-session"])
   ).addOption(
     new commander.Option(
       "--evidence-on <mode>",