agent-inspect 6.25.1 → 6.27.0

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 6.27.0
+
+### Minor Changes
+
+- 0f7c2e1: Bounded safe recovery contracts: additive TraceContract `retry.operations[]` oracles (read-first `retrieve_policy`), ADR-0011, and flagship recipe. Schema remains 1.0; AgentInspect still does not execute retries.
+
+## 6.26.0
+
+### Minor Changes
+
+- ec2ebbf: Outcome-aware behavioral sessions: `--preset behavioral-session` scores OUTCOME events without collapsing graceful tool errors into run failure, plus a synthetic MCP dual-axis recipe (#362).
+
 ## 6.25.1
 
 ### Patch Changes

package/README.md

@@ -212,7 +212,7 @@ The root package is enough for custom capture, the CLI, checks, and Evidence wor
 
 ## Status and documentation
 
-**Current published baseline:** **6.25.1** · persisted schema `1.0` · Node.js `>=20` · MIT.
+**Current published baseline:** **6.27.0** · persisted schema `1.0` · Node.js `>=20` · MIT.
 
 Legacy v0.1 and v0.2 traces remain readable. Check the npm badge and [changelog](CHANGELOG.md) for the current published version.
 

package/docs/CLI.md

@@ -320,7 +320,8 @@ Options:
 - `--guardrails <rule>`: optional deterministic guardrail rules (`banned-phrase`, `pii-leak`, `prompt-injection`, …); repeatable
 - `--circuit <rule>`: optional circuit analyzers (`same-tool-repetition`, `max-retries`, …); repeatable
 - `--fail-on-observation <status>`: add `outcome.status` rule; repeatable (`failed`, `passed`, `unknown`, `skipped`; default when flag present without value: `failed`)
-- `--preset <trajectory|safety|comprehensive>`: additive check preset (does not change the default when omitted)
+- `--preset <trajectory|safety|comprehensive|behavioral-session>`: additive check preset (does not change the default when omitted)
+  - `behavioral-session` (6.26): require harness completion + score OUTCOME events (`--fail-on-observation failed` by default); does **not** treat every TOOL `error` as a failed run
   - `trajectory`: completion/structure/relationship focus; excludes share-safety findings
   - `safety`: raw-content / secret / redaction focus
   - `comprehensive`: union of trajectory and safety

package/docs/LIMITATIONS.md

@@ -8,7 +8,7 @@ This document states what AgentInspect **does not** provide today. It complement
 - **No production APM replacement**: no sampling agents, no fleet-wide aggregation, no uptime SLAs.
 - **No vendor upload pipeline**: no built-in Langfuse/Braintrust/New Relic/Datadog direct exporters as live sinks.
 - **No automatic universal instrumentation** of every framework: integration is explicit (manual traces, log ingest, optional adapters).
-- **No retry execution engine:** TraceContract `retry` rules evaluate attempt identity and evidence only. AgentInspect does not retry tools, remediates nothing, and does not treat a client `idempotencyKey` as proof of exactly-once writes.
+- **No retry execution engine:** TraceContract `retry` rules (including additive `retry.operations[]` recovery oracles in 6.27) evaluate attempt identity and evidence only. AgentInspect does not retry tools, remediates nothing, and does not treat a client `idempotencyKey` as proof of exactly-once writes. Write-tool timeout/unknown completion is unevaluable without authoritative idempotency evidence; write-retry is not safe by default.
 - **Omitted-payload digests** (`createOmittedPayloadCommitment`) are bounded (1 MiB preflight). Digests prove omitted bytes existed; they are not redaction or authorization.
 
 ## Correlation metadata (v1.3.0)

package/docs/SESSIONS-AND-OUTCOMES.md

@@ -8,11 +8,26 @@ Useful CLI entry points: `sessions`, `search`, activity views (see [CLI.md](./CL
 
 Attempt identity for contracts (6.22+): `operationId`, `attemptId`, `attemptNumber`, `retryOf`, `fallbackOf`, `idempotencyKey`. Retry safety evaluation prefers this identity over “saw a prior ok” (see [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md), corrected in 6.25.1).
 
-## Observed outcomes
+## Observed outcomes (dual-axis)
 
 Outcomes record what the agent produced or decided at a high level for later review and gates. They remain local JSONL-derived evidence.
 
-**Dual-axis reminder:** tool/run **execution status** (`ok` / `error`) is independent of behavioral **outcome** (`passed` / `failed` / `unknown`). A graceful tool rejection can be `status: "error"` while the expected behavioral outcome is `passed`. Expanding this for MCP behavioral sessions is the focus of **6.26.0** (#362).
+| Axis | Where | Values | Meaning |
+| --- | --- | --- | --- |
+| Execution | TOOL / RUN `status` | `ok` / `error` / … | What happened at runtime (MCP `isError` stays `error`) |
+| Behavior | OUTCOME `outcomeStatus` | `passed` / `failed` / `unknown` / `skipped` | Whether the result matched the test expectation |
+
+A graceful tool rejection can be TOOL `status: "error"` while the expected behavioral OUTCOME is `passed`. Do **not** rewrite tool errors to `ok` to make a gate green.
+
+### CLI (6.26)
+
+```bash
+npx agent-inspect check <run> --preset behavioral-session --json
+```
+
+Preset selects harness completion + `outcome.status` and defaults `--fail-on-observation failed`. Recipe: [examples/recipes/mcp-behavioral-session](../examples/recipes/mcp-behavioral-session/).
+
+Issue **#362**: external sanitized fixtures remain `BLOCKED_ON_EXTERNAL_FIXTURE` until reviewed; the synthetic recipe ships first.
 
 ## Limitations
 
@@ -21,4 +36,4 @@ Outcomes record what the agent produced or decided at a high level for later rev
 - Studio session pages may still be thinner than APIs — Studio is Beta
 - AgentInspect does not execute retries or mutate source sessions
 
-Related: [WORKSPACE.md](./WORKSPACE.md) · [USE-CASES.md](./USE-CASES.md)
+Related: [WORKSPACE.md](./WORKSPACE.md) · [USE-CASES.md](./USE-CASES.md) · [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md)

package/docs/TRACE-CONTRACTS.md

@@ -14,7 +14,7 @@ Contracts compile to deterministic check rules for common cases:
 - additive `tools.orderRules` with per-rule occurrence modes
 - bounded `tools.arguments` JSON Pointer checks (`exists` | `type` | `equals` | `oneOf`)
 - `controls` declared-versus-enforced invariants
-- `retry` / side-effect safety using explicit attempt identity
+- `retry` / side-effect safety using explicit attempt identity (additive `retry.operations[]` recovery oracles in 6.27)
 - `alternatives.anyOf` for one level of legitimate alternate paths
 - actor `scope` selectors (`runId`, `subAgentId`, `groupId`, `workflowStep`, `rootEventId`)
 - observation `requireProvenance` (structural method / evidence / same-run event references)
@@ -169,9 +169,9 @@ defineTraceContract({
 
 Bounded evidence shapes: string event id, `{ eventId }`, or `{ eventIds }` (max 16). Method must be in the `ObservedOutcomeMethod` vocabulary. Omitting `requireProvenance` leaves prior observation behavior unchanged.
 
-### `tools.arguments` / `tools.orderRules` / `controls` / `retry` (shipped — experimental, 6.23; retry chronology corrected in 6.25.1)
+### `tools.arguments` / `tools.orderRules` / `controls` / `retry` (shipped — experimental, 6.23; retry chronology corrected in 6.25.1; `retry.operations` in 6.27)
 
-See [ADR-0010](./decisions/ADR-0010-structured-control-contracts.md).
+See [ADR-0010](./decisions/ADR-0010-structured-control-contracts.md) and [ADR-0011](./decisions/ADR-0011-bounded-safe-recovery.md).
 
 ```ts
 defineTraceContract({
@@ -203,12 +203,30 @@ defineTraceContract({
     requireIdempotencyEvidenceForRetry: true,
     requireRecoveredFailureVisible: true,
     fallbackOnlyAfterFailure: true,
+    operations: [
+      {
+        tool: "retrieve_policy",
+        sideEffectClass: "read",
+        maxAttempts: 2,
+        retryableErrors: { codes: ["TRANSIENT"] },
+        requireFailureBeforeRetry: true,
+        requireSameArguments: "structured-or-digest",
+        requireTerminalSuccess: true,
+        requireRecoveredFailureVisible: true,
+        successfulResultDependency: {
+          consumerKind: "LLM",
+          requireExplicitReference: true,
+        },
+      },
+    ],
   },
 });
 ```
 
 **Retry classification (6.25.1):** a genuine retry is detected from explicit identity preference — `attemptNumber > 1`, valid `retryOf` (target exists and precedes), distinct later `attemptId` under the same `operationId`, or a later finished attempt in an explicitly grouped operation — **not** only from a prior `ok`. `error → success` without `idempotencyKey` / `noSideEffect` evidence fails when `requireIdempotencyEvidenceForRetry` is set. `fallbackOnlyAfterFailure` and `requireRecoveredFailureVisible` require chronological earlier failure in the related chain. A client `idempotencyKey` is evidence of intent, not proof of exactly-once mutation. AgentInspect evaluates traces; it does not execute retries.
 
+**Bounded recovery operations (6.27):** `retry.operations[]` adds per-tool oracles for safe **read** recovery first (`retrieve_policy` recipe). Same-arguments checks accept structured payloads or matching digests and fail closed when both are missing. Write `sideEffectClass` treats timeout/`unknown`/`running` completion as unevaluable unless authoritative idempotency evidence is present. Write-retry is not safe by default.
+
 Missing structured argument evidence fails closed (`AI_CHECK_TOOL_ARGUMENT_EVIDENCE_UNAVAILABLE`). Findings never include full actual inputs.
 
 ### Capture capability matrix (tool-argument evidence)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-inspect",
-  "version": "6.25.1",
+  "version": "6.27.0",
   "license": "MIT",
   "type": "module",
   "description": "Local evidence debugger and trajectory-test toolkit for TypeScript AI agents — execution trees, TraceContract checks, Evidence v2, and read-only MCP",

package/packages/cli/dist/{chunk-HPR2OJPU.mjs → chunk-WQITHDXH.mjs}

@@ -11823,5 +11823,5 @@ function renderGateReport(result, options = {}) {
 }
 
 export { COHORT_METRIC_IDS, DEFAULT_SUITE_ARTIFACTS_DIR, EVIDENCE_FORMAT_VERSION, EVIDENCE_HTML_FILENAME, EVIDENCE_MANIFEST_FILENAME, Redactor, TraceDirectory, TraceReadError, TreeBuilder, aggregateBundleSafeStatus, aggregateSessionCheckResults, analyzeCohort, applyProfileMetadataCaps, assertBundlePathContained, assertEvidenceRelativePath, buildActivitySummary, buildBundleMetadata, buildBundleSummaryMarkdown, buildEvidenceCausalFailureViewHtml, buildEvidenceCiPackage, buildEvidenceCircuitViewHtml, buildEvidenceContractsViewHtml, buildEvidenceDiffViewHtml, buildEvidenceHtmlShell, buildEvidenceManifest, buildEvidenceOutcomesViewHtml, buildEvidenceProvenanceViewHtml, buildEvidenceSafetyViewHtml, buildEvidenceTimelineViewHtml, buildEvidenceToolsLlmViewHtml, buildEvidenceTreeViewHtml, buildLocalExplanation, buildPlaceholderArtifact, buildRunSummary, buildRunTimeline, buildRunWhatSummary, buildSessionIndex, buildTraceStats, buildZipArchive, bundleFailsOnSafety, bundleRunAssetRelativePath, collectTraceSchemaVersions, compactAttributes, createBaselineRegressionRule, createLlmUsageRule, createMaxStepDurationRule, createObservedOutcomeRule, createRequireCompletedRule, createRunDepthRule, createRunDurationRule, createRunStatusRule, createSafetyOversizedAttributeRule, createSafetyRawContentRule, createSafetyRedactionRule, createSafetySecretPatternRule, createStallDetectionRule, createStructureCycleRule, createStructureOrphanRule, createStructureParallelWidthRule, createStructureRelationshipRule, createToolUsageRule, defaultBundleOutputPath, defaultSuiteConfigTemplate, diffRuns, diffTraceEvents, enrichSessionRunRecord, escapeHtml, escapeMarkdown, extractMetadata, extractOutcomesFromTraceEvents, filterMetasBySessionScope, filterTraces, flattenTree, formatDuration2 as formatDuration, formatStepLabel, formatTimestamp, gateHasThresholds, getIndent, getTraceFilePath, inferEvidenceFileRole, isAgentInspectTrace, isPersistedInspectEvent, loadSessionRunRecords, loadSuiteConfig, loadTraceMetadataList, manualTraceEventsToComparableRun, nanoid, normalizeBundleOutputPath, openTrace, parseCohortMetricList, parseDuration, parseDurationFilter, parseGateList, parseTraceJsonl, persistedInspectEventsToTraceEvents, renderActivitySummaryHuman, renderCohortReport, renderErrorLine, renderGateReport, renderObservedOutcomesHtml, renderObservedOutcomesMarkdown, renderRunDiff, renderRunWhat, renderStepLine, renderSuiteReport, renderTimeline, renderTraceStats, resolveBundleRunIds, resolveRedactionProfile, resolveSuiteTemplate, resolveTraceDir, runGate, runSuite, runTraceChecks, safeString, sanitizeBundleRunId, searchTraces, serializeEvidenceManifest, sha256Hex, stableJson, summarizeObservedOutcomes, summarizeSemanticParity, traceEventToPersistedInspectEvent, truncateName, truncateStringForProfile, validateEvent, validateSuiteConfig, verifyEvidenceDirectory, zeroKinds };
-//# sourceMappingURL=chunk-HPR2OJPU.mjs.map
-//# sourceMappingURL=chunk-HPR2OJPU.mjs.map
+//# sourceMappingURL=chunk-WQITHDXH.mjs.map
+//# sourceMappingURL=chunk-WQITHDXH.mjs.map