agent-inspect 6.25.0 → 6.26.0

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 6.26.0
+
+### Minor Changes
+
+- ec2ebbf: Outcome-aware behavioral sessions: `--preset behavioral-session` scores OUTCOME events without collapsing graceful tool errors into run failure, plus a synthetic MCP dual-axis recipe (#362).
+
+## 6.25.1
+
+### Patch Changes
+
+- ca0ce77: Fix retry fail-open on error→success (identity-based retry classification, chronological fallback/recovery rules) and preflight the omitted-payload 1 MiB bound before copying oversized inputs.
+
+## Unreleased
+
 ## 6.25.0
 
 ### Minor Changes
@@ -135,12 +149,6 @@
 
 - 9aa0a80: Repository health and public-truth patch: permanent roadmap/active-plan structure, aggressive cleanup of archives/trains/proposals, ADRs, package-docs manifest, and repo:health CI gate. Docs/validators only — no schema or runtime product change.
 
-## Unreleased
-
-### Patch
-
-- Repository health and public-truth cleanup toward 6.16.1 (in progress).
-
 ## 6.16.0
 
 ### Minor Changes

package/README.md

@@ -212,7 +212,7 @@ The root package is enough for custom capture, the CLI, checks, and Evidence wor
 
 ## Status and documentation
 
-**Current published baseline:** **6.25.0** · persisted schema `1.0` · Node.js `>=20` · MIT.
+**Current published baseline:** **6.26.0** · persisted schema `1.0` · Node.js `>=20` · MIT.
 
 Legacy v0.1 and v0.2 traces remain readable. Check the npm badge and [changelog](CHANGELOG.md) for the current published version.
 

package/docs/CLI.md

@@ -320,7 +320,8 @@ Options:
 - `--guardrails <rule>`: optional deterministic guardrail rules (`banned-phrase`, `pii-leak`, `prompt-injection`, …); repeatable
 - `--circuit <rule>`: optional circuit analyzers (`same-tool-repetition`, `max-retries`, …); repeatable
 - `--fail-on-observation <status>`: add `outcome.status` rule; repeatable (`failed`, `passed`, `unknown`, `skipped`; default when flag present without value: `failed`)
-- `--preset <trajectory|safety|comprehensive>`: additive check preset (does not change the default when omitted)
+- `--preset <trajectory|safety|comprehensive|behavioral-session>`: additive check preset (does not change the default when omitted)
+  - `behavioral-session` (6.26): require harness completion + score OUTCOME events (`--fail-on-observation failed` by default); does **not** treat every TOOL `error` as a failed run
   - `trajectory`: completion/structure/relationship focus; excludes share-safety findings
   - `safety`: raw-content / secret / redaction focus
   - `comprehensive`: union of trajectory and safety

package/docs/KNOWN-ISSUES.md

@@ -2,6 +2,11 @@
 
 AgentInspect is **local-first** and **CLI-first**. These behaviors are intentional constraints or best-effort areas—not silent guarantees.
 
+## Contracts and safety
+
+- **Retry / side-effect rules (6.23+; chronology fixed in 6.25.1):** `requireIdempotencyEvidenceForRetry` applies to genuine retries including `error → success`, not only retries after a prior `ok`. Ambiguous attempt identity fails closed under that rule. Write timeouts remain **unknown completion** — safe write retry is not claimed.
+- **Omitted-payload digests:** inputs larger than 1 MiB are rejected after a length preflight; oversized rejection must not be treated as a free full-buffer copy path.
+
 ## Logs
 
 - **log4js-style parsing** is **best-effort**: embedded JSON must be recoverable from the line; unusual layouts may lose fields or warn.

package/docs/LIMITATIONS.md

@@ -8,6 +8,8 @@ This document states what AgentInspect **does not** provide today. It complement
 - **No production APM replacement**: no sampling agents, no fleet-wide aggregation, no uptime SLAs.
 - **No vendor upload pipeline**: no built-in Langfuse/Braintrust/New Relic/Datadog direct exporters as live sinks.
 - **No automatic universal instrumentation** of every framework: integration is explicit (manual traces, log ingest, optional adapters).
+- **No retry execution engine:** TraceContract `retry` rules evaluate attempt identity and evidence only. AgentInspect does not retry tools, remediates nothing, and does not treat a client `idempotencyKey` as proof of exactly-once writes.
+- **Omitted-payload digests** (`createOmittedPayloadCommitment`) are bounded (1 MiB preflight). Digests prove omitted bytes existed; they are not redaction or authorization.
 
 ## Correlation metadata (v1.3.0)
 

package/docs/SESSIONS-AND-OUTCOMES.md

@@ -6,14 +6,34 @@ Workflow sessions group related runs (retries, handoffs, multi-agent activity) u
 
 Useful CLI entry points: `sessions`, `search`, activity views (see [CLI.md](./CLI.md)).
 
-## Observed outcomes
+Attempt identity for contracts (6.22+): `operationId`, `attemptId`, `attemptNumber`, `retryOf`, `fallbackOf`, `idempotencyKey`. Retry safety evaluation prefers this identity over “saw a prior ok” (see [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md), corrected in 6.25.1).
+
+## Observed outcomes (dual-axis)
 
 Outcomes record what the agent produced or decided at a high level for later review and gates. They remain local JSONL-derived evidence.
 
+| Axis | Where | Values | Meaning |
+| --- | --- | --- | --- |
+| Execution | TOOL / RUN `status` | `ok` / `error` / … | What happened at runtime (MCP `isError` stays `error`) |
+| Behavior | OUTCOME `outcomeStatus` | `passed` / `failed` / `unknown` / `skipped` | Whether the result matched the test expectation |
+
+A graceful tool rejection can be TOOL `status: "error"` while the expected behavioral OUTCOME is `passed`. Do **not** rewrite tool errors to `ok` to make a gate green.
+
+### CLI (6.26)
+
+```bash
+npx agent-inspect check <run> --preset behavioral-session --json
+```
+
+Preset selects harness completion + `outcome.status` and defaults `--fail-on-observation failed`. Recipe: [examples/recipes/mcp-behavioral-session](../examples/recipes/mcp-behavioral-session/).
+
+Issue **#362**: external sanitized fixtures remain `BLOCKED_ON_EXTERNAL_FIXTURE` until reviewed; the synthetic recipe ships first.
+
 ## Limitations
 
 - Session indexing is not a full workflow contract engine
 - Handoff / approval TraceContract rules are not fully wired — see [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md)
 - Studio session pages may still be thinner than APIs — Studio is Beta
+- AgentInspect does not execute retries or mutate source sessions
 
-Related: [WORKSPACE.md](./WORKSPACE.md) · [USE-CASES.md](./USE-CASES.md)
+Related: [WORKSPACE.md](./WORKSPACE.md) · [USE-CASES.md](./USE-CASES.md) · [TRACE-CONTRACTS.md](./TRACE-CONTRACTS.md)

package/docs/TRACE-CONTRACTS.md

@@ -169,7 +169,7 @@ defineTraceContract({
 
 Bounded evidence shapes: string event id, `{ eventId }`, or `{ eventIds }` (max 16). Method must be in the `ObservedOutcomeMethod` vocabulary. Omitting `requireProvenance` leaves prior observation behavior unchanged.
 
-### `tools.arguments` / `tools.orderRules` / `controls` / `retry` (shipped — experimental, 6.23)
+### `tools.arguments` / `tools.orderRules` / `controls` / `retry` (shipped — experimental, 6.23; retry chronology corrected in 6.25.1)
 
 See [ADR-0010](./decisions/ADR-0010-structured-control-contracts.md).
 
@@ -202,10 +202,13 @@ defineTraceContract({
     nonIdempotentTools: ["charge"],
     requireIdempotencyEvidenceForRetry: true,
     requireRecoveredFailureVisible: true,
+    fallbackOnlyAfterFailure: true,
   },
 });
 ```
 
+**Retry classification (6.25.1):** a genuine retry is detected from explicit identity preference — `attemptNumber > 1`, valid `retryOf` (target exists and precedes), distinct later `attemptId` under the same `operationId`, or a later finished attempt in an explicitly grouped operation — **not** only from a prior `ok`. `error → success` without `idempotencyKey` / `noSideEffect` evidence fails when `requireIdempotencyEvidenceForRetry` is set. `fallbackOnlyAfterFailure` and `requireRecoveredFailureVisible` require chronological earlier failure in the related chain. A client `idempotencyKey` is evidence of intent, not proof of exactly-once mutation. AgentInspect evaluates traces; it does not execute retries.
+
 Missing structured argument evidence fails closed (`AI_CHECK_TOOL_ARGUMENT_EVIDENCE_UNAVAILABLE`). Findings never include full actual inputs.
 
 ### Capture capability matrix (tool-argument evidence)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-inspect",
-  "version": "6.25.0",
+  "version": "6.26.0",
   "license": "MIT",
   "type": "module",
   "description": "Local evidence debugger and trajectory-test toolkit for TypeScript AI agents — execution trees, TraceContract checks, Evidence v2, and read-only MCP",

package/packages/cli/dist/{chunk-NHVN5AYW.mjs → chunk-HPR2OJPU.mjs}

@@ -11823,5 +11823,5 @@ function renderGateReport(result, options = {}) {
 }
 
 export { COHORT_METRIC_IDS, DEFAULT_SUITE_ARTIFACTS_DIR, EVIDENCE_FORMAT_VERSION, EVIDENCE_HTML_FILENAME, EVIDENCE_MANIFEST_FILENAME, Redactor, TraceDirectory, TraceReadError, TreeBuilder, aggregateBundleSafeStatus, aggregateSessionCheckResults, analyzeCohort, applyProfileMetadataCaps, assertBundlePathContained, assertEvidenceRelativePath, buildActivitySummary, buildBundleMetadata, buildBundleSummaryMarkdown, buildEvidenceCausalFailureViewHtml, buildEvidenceCiPackage, buildEvidenceCircuitViewHtml, buildEvidenceContractsViewHtml, buildEvidenceDiffViewHtml, buildEvidenceHtmlShell, buildEvidenceManifest, buildEvidenceOutcomesViewHtml, buildEvidenceProvenanceViewHtml, buildEvidenceSafetyViewHtml, buildEvidenceTimelineViewHtml, buildEvidenceToolsLlmViewHtml, buildEvidenceTreeViewHtml, buildLocalExplanation, buildPlaceholderArtifact, buildRunSummary, buildRunTimeline, buildRunWhatSummary, buildSessionIndex, buildTraceStats, buildZipArchive, bundleFailsOnSafety, bundleRunAssetRelativePath, collectTraceSchemaVersions, compactAttributes, createBaselineRegressionRule, createLlmUsageRule, createMaxStepDurationRule, createObservedOutcomeRule, createRequireCompletedRule, createRunDepthRule, createRunDurationRule, createRunStatusRule, createSafetyOversizedAttributeRule, createSafetyRawContentRule, createSafetyRedactionRule, createSafetySecretPatternRule, createStallDetectionRule, createStructureCycleRule, createStructureOrphanRule, createStructureParallelWidthRule, createStructureRelationshipRule, createToolUsageRule, defaultBundleOutputPath, defaultSuiteConfigTemplate, diffRuns, diffTraceEvents, enrichSessionRunRecord, escapeHtml, escapeMarkdown, extractMetadata, extractOutcomesFromTraceEvents, filterMetasBySessionScope, filterTraces, flattenTree, formatDuration2 as formatDuration, formatStepLabel, formatTimestamp, gateHasThresholds, getIndent, getTraceFilePath, inferEvidenceFileRole, isAgentInspectTrace, isPersistedInspectEvent, loadSessionRunRecords, loadSuiteConfig, loadTraceMetadataList, manualTraceEventsToComparableRun, nanoid, normalizeBundleOutputPath, openTrace, parseCohortMetricList, parseDuration, parseDurationFilter, parseGateList, parseTraceJsonl, persistedInspectEventsToTraceEvents, renderActivitySummaryHuman, renderCohortReport, renderErrorLine, renderGateReport, renderObservedOutcomesHtml, renderObservedOutcomesMarkdown, renderRunDiff, renderRunWhat, renderStepLine, renderSuiteReport, renderTimeline, renderTraceStats, resolveBundleRunIds, resolveRedactionProfile, resolveSuiteTemplate, resolveTraceDir, runGate, runSuite, runTraceChecks, safeString, sanitizeBundleRunId, searchTraces, serializeEvidenceManifest, sha256Hex, stableJson, summarizeObservedOutcomes, summarizeSemanticParity, traceEventToPersistedInspectEvent, truncateName, truncateStringForProfile, validateEvent, validateSuiteConfig, verifyEvidenceDirectory, zeroKinds };
-//# sourceMappingURL=chunk-NHVN5AYW.mjs.map
-//# sourceMappingURL=chunk-NHVN5AYW.mjs.map
+//# sourceMappingURL=chunk-HPR2OJPU.mjs.map
+//# sourceMappingURL=chunk-HPR2OJPU.mjs.map