agent-inspect 6.29.4 → 6.29.6

package/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## 6.29.6
+
+### Patch Changes
+
+- 129f45a: Safe sharing and integration clarity for 6.29.6:
+
+  - Treat fully redacted raw-content fields as share-safe while retaining other safety checks (N-8).
+  - Clarify trajectory vs outcome CI gates and TOOL-only `requiredOrder` diagnostics in CLI help/docs.
+  - Recipe corrections: pairwise measured stage commitments for comparable-cohort-runs; browser observed-outcome identity/precondition/postcondition matrix.
+  - Jest reporter association remains explicit (`associations` / `resolveTrace`); private hotel capture (N-9) stays unverified without application checkouts.
+
+## 6.29.5
+
+### Patch Changes
+
+- 3e8750f: Trustworthy existing checks for 6.29.5:
+
+  - Validate `structure.minConfidence` against the canonical AttributionConfidence vocabulary (`unknown` | `heuristic` | `correlated` | `explicit`); reject obsolete values with exit 2.
+  - Fail closed on uncertain write completion: client idempotency keys / producer flags are not write-completion proof (ADR-0011).
+  - Circuit rules use logical-event projection, case-insensitive TOOL/LLM kind, and `--run` scoping so other runs cannot inflate counts.
+  - Selected `--circuit` / `--guardrails` rules appear in `ruleExecutions` / `rulesEvaluated`; unknown extension rule names fail closed instead of silent green.
+
 ## 6.29.4
 
 ### Patch Changes

package/README.md

@@ -212,7 +212,7 @@ The root package is enough for custom capture, the CLI, checks, and Evidence wor
 
 ## Status and documentation
 
-**Current published baseline:** **6.29.4** · persisted schema `1.0` · Node.js `>=20` · MIT.
+**Current published baseline:** **6.29.6** · persisted schema `1.0` · Node.js `>=20` · MIT.
 
 Legacy v0.1 and v0.2 traces remain readable. Check the npm badge and [changelog](CHANGELOG.md) for the current published version.
 

package/docs/CLI.md

@@ -337,7 +337,7 @@ Options:
 - `--correlate-group`: when using `--session`, also match synthetic `group:` session keys
 - `--guardrails <rule>`: optional deterministic guardrail rules (`banned-phrase`, `pii-leak`, `prompt-injection`, …); repeatable
 - `--circuit <rule>`: optional circuit analyzers (`same-tool-repetition`, `max-retries`, …); repeatable
-- `--fail-on-observation <status>`: add `outcome.status` rule; repeatable (`failed`, `passed`, `unknown`, `skipped`; default when flag present without value: `failed`)
+- `--fail-on-observation <status>`: **outcome gate** — adds `outcome.status` with `requireAny=true` (needs ≥1 OUTCOME event). Comma-separated statuses: `failed`, `passed`, `unknown`, `skipped`. Does **not** invent a passed outcome from tool/LLM/run success. For adapter structure without outcomes, use `--preset trajectory` (plus `--required-tool` etc.); a trajectory pass is not share safety (`verify-safe`).
 - `--preset <trajectory|safety|comprehensive|behavioral-session>`: additive check preset (does not change the default when omitted)
   - `behavioral-session` (6.26): require harness completion + score OUTCOME events (`--fail-on-observation failed` by default); does **not** treat every TOOL `error` as a failed run
   - `trajectory`: completion/structure/relationship focus; excludes share-safety findings

package/docs/LIMITATIONS.md

@@ -8,7 +8,7 @@ This document states what AgentInspect **does not** provide today. It complement
 - **No production APM replacement**: no sampling agents, no fleet-wide aggregation, no uptime SLAs.
 - **No vendor upload pipeline**: no built-in Langfuse/Braintrust/New Relic/Datadog direct exporters as live sinks.
 - **No automatic universal instrumentation** of every framework: integration is explicit (manual traces, log ingest, optional adapters).
-- **No retry execution engine:** TraceContract `retry` rules (including additive `retry.operations[]` recovery oracles in 6.27) evaluate attempt identity and evidence only. AgentInspect does not retry tools, remediates nothing, and does not treat a client `idempotencyKey` as proof of exactly-once writes. Write-tool timeout/unknown completion is unevaluable without authoritative idempotency evidence; write-retry is not safe by default.
+- **No retry execution engine:** TraceContract `retry` rules (including additive `retry.operations[]` recovery oracles in 6.27) evaluate attempt identity and evidence only. AgentInspect does not retry tools, remediates nothing, and does not treat a client `idempotencyKey` as proof of exactly-once writes. Write-tool timeout/unknown/running completion is unevaluable for `retry.operations` write recovery (client keys and producer no-side-effect flags do not clear that finding); write-retry is not safe by default.
 - **Transport facts are source-boundary only:** HTTP status, server-advertised retry delay, selected delay, and their sources must be captured at the transport/fetch boundary. An MCP wrapper cannot recover facts already collapsed into an SDK error message.
 - **Run comparability is optional guidance:** Ordinary trajectory checks do not fail when cohort/prompt/tool commitments are absent. See [RUN-COMPARABILITY.md](./RUN-COMPARABILITY.md). A machine-readable comparable-run profile is not a default gate in this line.
 - **Omitted-payload digests** (`createOmittedPayloadCommitment`) are bounded (1 MiB preflight). Digests prove omitted bytes existed; they are not redaction or authorization.

package/docs/TRACE-CONTRACTS.md

@@ -25,6 +25,8 @@ Contracts compile to deterministic check rules for common cases:
 
 ## `tools.requiredOrder` semantics
 
+`requiredOrder` / `orderRules` match **TOOL** events only (via canonical tool names). LLM / LOGIC / other kinds with the same display name do not satisfy the order and are not relabeled as tools. When a required name exists only under another kind, `tool.usage` reports that kind in the finding message.
+
 `requiredOrder` is expanded into **adjacent pair** ordering rules with unique ids:
 
 ```text
@@ -42,15 +44,16 @@ Contracts compile to deterministic check rules for common cases:
 - `all-occurrences` requires every `before` occurrence to finish before every `after` occurrence starts (`max(before.end) <= min(after.start)`);
 - causal modes fail when a required interval boundary cannot be resolved instead of falling back to encounter order.
 
-Examples for `requiredOrder: ["retrieve", "generate"]`:
+Examples for `requiredOrder: ["retrieve_policy", "send_email"]` (both TOOL-typed):
 
 | Trajectory | Result |
 | --- | --- |
-| `retrieve → generate` | PASS |
-| `retrieve → rerank → generate` | PASS |
-| `retrieve → generate → retrieve` | PASS under omitted / `first-occurrence`; FAIL under `all-occurrences` |
-| `generate → retrieve` | FAIL (order) |
-| `cache_lookup → generate` | FAIL (missing `retrieve` via implied presence) |
+| `retrieve_policy → send_email` | PASS |
+| `retrieve_policy → rerank_docs → send_email` | PASS |
+| `retrieve_policy → send_email → retrieve_policy` | PASS under omitted / `first-occurrence`; FAIL under `all-occurrences` |
+| `send_email → retrieve_policy` | FAIL (order) |
+| `cache_lookup → send_email` | FAIL (missing `retrieve_policy` via implied presence) |
+| LLM named `generate` present, no TOOL `generate` | FAIL presence with non-TOOL kind diagnostic; not treated as a tool |
 
 Low-level `createToolOrderingRule({ before, after })` alone may still pass when an endpoint is missing (compositional). TraceContract `requiredOrder` does not.
 
@@ -111,7 +114,7 @@ defineTraceContract({
       {
         id: "retrieve",
         contract: {
-          tools: { required: ["retrieve"], requiredOrder: ["retrieve", "generate"] },
+          tools: { required: ["retrieve_policy"], requiredOrder: ["retrieve_policy", "send_email"] },
           observations: { required: ["retrieval-context-valid"] },
         },
       },
@@ -225,7 +228,7 @@ defineTraceContract({
 
 **Retry classification (6.25.1):** a genuine retry is detected from explicit identity preference — `attemptNumber > 1`, valid `retryOf` (target exists and precedes), distinct later `attemptId` under the same `operationId`, or a later finished attempt in an explicitly grouped operation — **not** only from a prior `ok`. `error → success` without `idempotencyKey` / `noSideEffect` evidence fails when `requireIdempotencyEvidenceForRetry` is set. `fallbackOnlyAfterFailure` and `requireRecoveredFailureVisible` require chronological earlier failure in the related chain. A client `idempotencyKey` is evidence of intent, not proof of exactly-once mutation. AgentInspect evaluates traces; it does not execute retries.
 
-**Bounded recovery operations (6.27):** `retry.operations[]` adds per-tool oracles for safe **read** recovery first (`retrieve_policy` recipe). Same-arguments checks accept structured payloads or matching digests and fail closed when both are missing. Write `sideEffectClass` treats timeout/`unknown`/`running` completion as unevaluable unless authoritative idempotency evidence is present. Write-retry is not safe by default.
+**Bounded recovery operations (6.27 / 6.29.5):** `retry.operations[]` adds per-tool oracles for safe **read** recovery first (`retrieve_policy` recipe). Same-arguments checks accept structured payloads or matching digests and fail closed when both are missing. Write `sideEffectClass` treats timeout/`unknown`/`running` completion as unevaluable → fail; a client `idempotencyKey` or producer `noSideEffect`/`sideEffect:false` flag does **not** clear that finding. Write-retry is not safe by default.
 
 Missing structured argument evidence fails closed (`AI_CHECK_TOOL_ARGUMENT_EVIDENCE_UNAVAILABLE`). Findings never include full actual inputs.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-inspect",
-  "version": "6.29.4",
+  "version": "6.29.6",
   "license": "MIT",
   "type": "module",
   "description": "Local evidence debugger and trajectory-test toolkit for TypeScript AI agents — execution trees, TraceContract checks, Evidence v2, and read-only MCP",

package/packages/cli/dist/{chunk-FJRUJSST.mjs → chunk-DBRMI7TC.mjs}

@@ -9,6 +9,23 @@ import { createReadStream } from 'fs';
 import { createInterface } from 'readline';
 import { pathToFileURL } from 'url';
 
+// packages/core/src/types/inspect-event.ts
+var ATTRIBUTION_CONFIDENCES = [
+  "unknown",
+  "heuristic",
+  "correlated",
+  "explicit"
+];
+function isAttributionConfidence(value) {
+  return typeof value === "string" && ATTRIBUTION_CONFIDENCES.includes(value);
+}
+var ATTRIBUTION_CONFIDENCE_RANK = {
+  unknown: 0,
+  heuristic: 1,
+  correlated: 2,
+  explicit: 3
+};
+
 // packages/core/src/types/persisted-inspect-event.ts
 var INSPECT_KINDS = [
   "RUN",
@@ -24,12 +41,6 @@ var INSPECT_KINDS = [
   "LOG",
   "OUTCOME"
 ];
-var ATTRIBUTION_CONFIDENCES = [
-  "explicit",
-  "correlated",
-  "heuristic",
-  "unknown"
-];
 var PERSISTED_EVENT_SOURCE_TYPES = [
   "manual",
   "json-log",
@@ -65,9 +76,6 @@ function isOptionalNonNegativeNumber(value) {
 function isInspectKind(value) {
   return typeof value === "string" && INSPECT_KINDS.includes(value);
 }
-function isAttributionConfidence(value) {
-  return typeof value === "string" && ATTRIBUTION_CONFIDENCES.includes(value);
-}
 function isPersistedEventSourceType(value) {
   return typeof value === "string" && PERSISTED_EVENT_SOURCE_TYPES.includes(value);
 }
@@ -5192,12 +5200,7 @@ var STATUS_RANK = {
   warning: 1,
   pass: 2
 };
-var CONFIDENCE_RANK = {
-  unknown: 0,
-  heuristic: 1,
-  correlated: 2,
-  explicit: 3
-};
+var CONFIDENCE_RANK = ATTRIBUTION_CONFIDENCE_RANK;
 var DEFAULT_SENSITIVE_KEYS = DEFAULT_CREDENTIAL_SENSITIVE_KEYS;
 var DEFAULT_RAW_CONTENT_KEYS = [
   "body",
@@ -5671,6 +5674,18 @@ function limitFindings(findings, maxFindings) {
 function hasRedactionMarker(value, markers) {
   return markers.some((marker) => value.includes(marker)) || /^\[HASH:[A-Za-z0-9_-]+\]$/.test(value);
 }
+function isFullyRedactedValue(value, markers = ["[REDACTED]", "[REDACTED:"]) {
+  if (typeof value !== "string") return false;
+  const trimmed = value.trim();
+  if (trimmed.length === 0) return false;
+  if (/^\[HASH:[A-Za-z0-9_-]+\]$/.test(trimmed)) return true;
+  if (/^\[REDACTED:[^\]]*\]$/.test(trimmed)) return true;
+  for (const marker of markers) {
+    if (marker.endsWith(":")) continue;
+    if (trimmed === marker) return true;
+  }
+  return false;
+}
 function isSensitiveKey(key, sensitiveKeys) {
   return isCredentialSensitiveKey(key, sensitiveKeys);
 }
@@ -5989,8 +6004,26 @@ function createToolUsageRule(options) {
       const findings = [];
       for (const required of options.required ?? []) {
         if (!nameSet.has(required)) {
+          const otherKinds = [
+            ...new Set(
+              context.events.filter((event) => {
+                const kind = typeof event.kind === "string" ? event.kind.toUpperCase() : "";
+                if (kind === "TOOL") return false;
+                return resolveCanonicalToolName(event) === required || event.name === required || event.name === `tool:${required}` || event.name.endsWith(`:${required}`);
+              }).map(
+                (event) => typeof event.kind === "string" ? event.kind.toUpperCase() : "UNKNOWN"
+              )
+            )
+          ].sort();
+          const hint = otherKinds.length > 0 ? ` Required name appears under non-TOOL kind(s): ${otherKinds.join(", ")}. tools.required / requiredOrder match TOOL events only.` : "";
           findings.push(
-            failFinding("tool.usage", `Required tool ${required} did not appear.`, runEvidence(context.selectedRun), required, names)
+            failFinding(
+              "tool.usage",
+              `Required tool ${required} did not appear.${hint}`,
+              runEvidence(context.selectedRun),
+              required,
+              names
+            )
           );
         }
       }
@@ -6164,6 +6197,12 @@ function createStructureCycleRule() {
   };
 }
 function createStructureRelationshipRule(options = {}) {
+  if (options.minConfidence !== void 0 && !isAttributionConfidence(options.minConfidence)) {
+    throw new TypeError(
+      `createStructureRelationshipRule: minConfidence must be one of: ${ATTRIBUTION_CONFIDENCES.join(", ")}.`
+    );
+  }
+  const minConfidence = options.minConfidence;
   return {
     id: "structure.relationship",
     category: "structure",
@@ -6171,18 +6210,21 @@ function createStructureRelationshipRule(options = {}) {
     evaluate(context) {
       const byId = eventMap(semanticEvents(context));
       const findings = [];
-      const minConfidence = options.minConfidence;
       for (const event of semanticEvents(context)) {
-        if (minConfidence && CONFIDENCE_RANK[event.confidence] < CONFIDENCE_RANK[minConfidence]) {
-          findings.push(
-            failFinding(
-              "structure.relationship",
-              `Event confidence ${event.confidence} is below ${minConfidence}.`,
-              [eventEvidence(event, "confidence")],
-              { minConfidence },
-              event.confidence
-            )
-          );
+        if (minConfidence) {
+          const eventRank = CONFIDENCE_RANK[event.confidence] ?? CONFIDENCE_RANK.unknown;
+          const minRank = CONFIDENCE_RANK[minConfidence];
+          if (eventRank < minRank) {
+            findings.push(
+              failFinding(
+                "structure.relationship",
+                `Event confidence ${event.confidence} is below ${minConfidence}.`,
+                [eventEvidence(event, "confidence")],
+                { minConfidence },
+                event.confidence
+              )
+            );
+          }
         }
         if (!event.parentId) continue;
         if (event.parentId === event.eventId) {
@@ -6346,6 +6388,7 @@ function createSafetyRedactionRule(options = {}) {
 function createSafetyRawContentRule(options = {}) {
   const forbiddenKeys = options.forbiddenKeys ?? DEFAULT_RAW_CONTENT_KEYS;
   const safePathPrefixes = options.safePathPrefixes ?? DEFAULT_SAFE_RAW_CONTENT_PATH_PREFIXES;
+  const markers = options.redactedMarkers ?? ["[REDACTED]", "[REDACTED:"];
   return {
     id: "safety.rawPrompt",
     category: "safety",
@@ -6356,6 +6399,7 @@ function createSafetyRawContentRule(options = {}) {
         for (const entry of eventValueEntries(event, { includeSummaries: options.includeSummaries })) {
           const key = entry.key ?? lastPathSegment(entry.path);
           if (!isRawContentPath(entry.path, key, forbiddenKeys, safePathPrefixes)) continue;
+          if (isFullyRedactedValue(entry.value, markers)) continue;
           findings.push(
             failFinding(
               "safety.rawPrompt",
@@ -12488,6 +12532,6 @@ function renderGateReport(result, options = {}) {
   return renderGateSummaryMarkdown(result);
 }
 
-export { COHORT_METRIC_IDS, DEFAULT_SUITE_ARTIFACTS_DIR, EVIDENCE_FORMAT_VERSION, EVIDENCE_HTML_FILENAME, EVIDENCE_MANIFEST_FILENAME, Redactor, TraceDirectory, TraceReadError, TreeBuilder, aggregateBundleSafeStatus, aggregateSessionCheckResults, analyzeCohort, applyProfileMetadataCaps, assertBundlePathContained, assertEvidenceRelativePath, buildActivitySummary, buildBundleMetadata, buildBundleSummaryMarkdown, buildEvidenceCausalFailureViewHtml, buildEvidenceCiPackage, buildEvidenceCircuitViewHtml, buildEvidenceContractsViewHtml, buildEvidenceDiffViewHtml, buildEvidenceHtmlShell, buildEvidenceManifest, buildEvidenceOutcomesViewHtml, buildEvidenceProvenanceViewHtml, buildEvidenceSafetyViewHtml, buildEvidenceTimelineViewHtml, buildEvidenceToolsLlmViewHtml, buildEvidenceTreeViewHtml, buildLocalExplanation, buildPlaceholderArtifact, buildRunSummary, buildRunTimeline, buildRunWhatSummary, buildSessionIndex, buildTraceStats, buildZipArchive, bundleFailsOnSafety, bundleRunAssetRelativePath, collectTraceSchemaVersions, compactAttributes, createBaselineRegressionRule, createLlmUsageRule, createMaxStepDurationRule, createObservedOutcomeRule, createRequireCompletedRule, createRunDepthRule, createRunDurationRule, createRunStatusRule, createSafetyOversizedAttributeRule, createSafetyRawContentRule, createSafetyRedactionRule, createSafetySecretPatternRule, createStallDetectionRule, createStructureCycleRule, createStructureOrphanRule, createStructureParallelWidthRule, createStructureRelationshipRule, createToolUsageRule, defaultBundleOutputPath, defaultSuiteConfigTemplate, diffRuns, diffTraceEvents, enrichSessionRunRecord, escapeHtml, escapeMarkdown, extractMetadata, extractOutcomesFromTraceEvents, filterMetasBySessionScope, filterTraces, flattenTree, formatDuration2 as formatDuration, formatStepLabel, formatTimestamp, gateHasThresholds, getIndent, getTraceFilePath, inferEvidenceFileRole, isAgentInspectTrace, isPersistedInspectEvent, loadSessionRunRecords, loadSuiteConfig, loadTraceMetadataList, manualTraceEventsToComparableRun, nanoid, normalizeBundleOutputPath, openTrace, parseCohortMetricList, parseDuration, parseDurationFilter, parseGateList, parseTraceJsonl, persistedInspectEventsToTraceEvents, renderActivitySummaryHuman, renderCohortReport, renderErrorLine, renderGateReport, renderObservedOutcomesHtml, renderObservedOutcomesMarkdown, renderRunDiff, renderRunWhat, renderStepLine, renderSuiteReport, renderTimeline, renderTraceStats, resolveBundleRunIds, resolveRedactionProfile, resolveSuiteTemplate, resolveTraceDir, runGate, runSuite, runTraceChecks, safeString, sanitizeBundleRunId, searchTraces, serializeEvidenceManifest, sha256Hex, stableJson, summarizeObservedOutcomes, summarizeSemanticParity, traceEventToPersistedInspectEvent, truncateName, truncateStringForProfile, validateEvent, validateSuiteConfig, verifyEvidenceDirectory, zeroKinds };
-//# sourceMappingURL=chunk-FJRUJSST.mjs.map
-//# sourceMappingURL=chunk-FJRUJSST.mjs.map
+export { ATTRIBUTION_CONFIDENCES, COHORT_METRIC_IDS, DEFAULT_SUITE_ARTIFACTS_DIR, EVIDENCE_FORMAT_VERSION, EVIDENCE_HTML_FILENAME, EVIDENCE_MANIFEST_FILENAME, Redactor, TraceDirectory, TraceReadError, TreeBuilder, aggregateBundleSafeStatus, aggregateSessionCheckResults, analyzeCohort, applyProfileMetadataCaps, assertBundlePathContained, assertEvidenceRelativePath, buildActivitySummary, buildBundleMetadata, buildBundleSummaryMarkdown, buildEvidenceCausalFailureViewHtml, buildEvidenceCiPackage, buildEvidenceCircuitViewHtml, buildEvidenceContractsViewHtml, buildEvidenceDiffViewHtml, buildEvidenceHtmlShell, buildEvidenceManifest, buildEvidenceOutcomesViewHtml, buildEvidenceProvenanceViewHtml, buildEvidenceSafetyViewHtml, buildEvidenceTimelineViewHtml, buildEvidenceToolsLlmViewHtml, buildEvidenceTreeViewHtml, buildLocalExplanation, buildPlaceholderArtifact, buildRunSummary, buildRunTimeline, buildRunWhatSummary, buildSessionIndex, buildTraceStats, buildZipArchive, bundleFailsOnSafety, bundleRunAssetRelativePath, collectTraceSchemaVersions, compactAttributes, createBaselineRegressionRule, createLlmUsageRule, createMaxStepDurationRule, createObservedOutcomeRule, createRequireCompletedRule, createRunDepthRule, createRunDurationRule, createRunStatusRule, createSafetyOversizedAttributeRule, createSafetyRawContentRule, createSafetyRedactionRule, createSafetySecretPatternRule, createStallDetectionRule, createStructureCycleRule, createStructureOrphanRule, createStructureParallelWidthRule, createStructureRelationshipRule, createToolUsageRule, defaultBundleOutputPath, defaultSuiteConfigTemplate, diffRuns, diffTraceEvents, enrichSessionRunRecord, escapeHtml, escapeMarkdown, extractMetadata, extractOutcomesFromTraceEvents, filterMetasBySessionScope, filterTraces, flattenTree, formatDuration2 as formatDuration, formatStepLabel, formatTimestamp, gateHasThresholds, getIndent, getTraceFilePath, inferEvidenceFileRole, isAgentInspectTrace, isAttributionConfidence, isPersistedInspectEvent, loadSessionRunRecords, loadSuiteConfig, loadTraceMetadataList, manualTraceEventsToComparableRun, nanoid, normalizeBundleOutputPath, openTrace, parseCohortMetricList, parseDuration, parseDurationFilter, parseGateList, parseTraceJsonl, persistedInspectEventsToTraceEvents, projectLogicalEvents, renderActivitySummaryHuman, renderCohortReport, renderErrorLine, renderGateReport, renderObservedOutcomesHtml, renderObservedOutcomesMarkdown, renderRunDiff, renderRunWhat, renderStepLine, renderSuiteReport, renderTimeline, renderTraceStats, resolveBundleRunIds, resolveRedactionProfile, resolveSuiteTemplate, resolveTraceDir, runGate, runSuite, runTraceChecks, safeString, sanitizeBundleRunId, searchTraces, serializeEvidenceManifest, sha256Hex, stableJson, summarizeObservedOutcomes, summarizeSemanticParity, traceEventToPersistedInspectEvent, truncateName, truncateStringForProfile, validateEvent, validateSuiteConfig, verifyEvidenceDirectory, zeroKinds };
+//# sourceMappingURL=chunk-DBRMI7TC.mjs.map
+//# sourceMappingURL=chunk-DBRMI7TC.mjs.map