@leanandmean/coding-agent 0.74.1-scramjet.67 → 0.74.1-scramjet.69

package/docs/custom-provider.md

@@ -515,6 +515,26 @@ output.usage.totalTokens = output.usage.input + output.usage.output +
 calculateCost(model, output.usage);
 ```
 
+### Failure Evidence
+
+`@leanandmean/ai` exports `RequestFailureV1`, `RetrySuppressionV1`, `appendRequestFailure`, `validateRequestFailure`, and `inspectFailureEvidence`. Optional diagnostics persist in the existing assistant message envelope. A `request_failure` diagnostic has `details.schemaVersion: 1` and exactly one closed branch:
+
+- `kind: "http"`, numeric integer `status` from 400 through 599, and `reason: "status"` or a recognized semantic rejection.
+- `kind: "provider"` and a closed semantic `category` (for example `rate_limit`, `quota_exhausted`, or `context_overflow`).
+- `kind: "stream"` and `reason: "missing_body"`, `"missing_terminal_event"`, `"transport"`, `"timeout"`, or `"malformed_event"`.
+- `kind: "callback"` and `callback: "onPayload"` or `"onResponse"`.
+- `kind: "local"` and `reason: "request_preparation"`, `"request_validation"`, `"configuration"`, or `"context_overflow"`. The last is reserved for witnessed local context allocation failure, not arbitrary error prose.
+
+Use the exported TypeScript union as the complete vocabulary. AI's neutral evidence utility validates both primary families; the Responses validator remains available from its existing public subpath. Unknown keys, invalid values, duplicate recognized diagnostics, and simultaneous `request_failure` and Responses `provider_failure` evidence fail closed. HTTP 401/403/404 cannot authorize overflow or override their terminal status category; explicit HTTP overflow evidence is compatible only with 400/413/422. Previously accepted contradictory version-1 records remain readable in sessions but no longer validate for recovery; no journal migration is required. Do not emit two primary failure families for one failure. Valid evidence takes precedence over retry and overflow prose; absent evidence preserves legacy custom-provider behavior. Stream completeness failures may be retried within configured limits without asserting their cause was transient. The existing Responses EOF diagnostic remains `unknown`; session policy explicitly admits its canonical category.
+
+Separate `retry_suppression` evidence vetoes retries without changing the provider cause. Version 1 accepts `server_delay_exceeds_limit` with finite safe `requestedDelayMs > maxDelayMs > 0`, or `invalid_server_delay` with no raw value. It must accompany one valid transient primary failure. Orphaned, malformed, duplicate or inconsistent suppression blocks automatic recovery. These records are evidence, not durable retry queues; reopening a session does not restart retries.
+
+Built-in failure catches preserve an existing primary diagnosis, classify witnessed failures at their source, and emit explicit unknown evidence when available facts do not establish a category. Unknown, provider-error, and malformed-event evidence blocks both retry and failure-triggered compaction, even when preceding usage is high. This is not a universal enforcement guarantee for future adapters or trusted extension replacements. Independent new-prompt maintenance and evidence-absent legacy custom providers retain their existing behavior.
+
+Callback failures must never be inferred from provider prose or persist original callback exception details. Message origin describes who produced the message, not which operation failed. Retain partial provider output and mark the callback failure separately.
+
+Completeness coverage is route-qualified: shared Responses, Chat's existing finish-reason check, and started Anthropic streams detect missing terminals. Additional mandatory markers for Google/Vertex, Mistral, Bedrock and startless Anthropic SSE were not established from installed SDK contracts and are not enforced. In particular, Google prompt-feedback blocking can occur without a candidate finish reason; parser acceptance alone does not establish valid backend completion. The installed Google SDK exposes bodyless generation failures as plain errors without a supported raw-response hook; Google/Vertex therefore retain an explicit unknown diagnosis rather than inferring body absence from the error message. Chat and Mistral classify body absence from their available response objects. Responses and default Chat/Anthropic clients distinguish local SDK preparation from dispatch at the invocation-local fetch boundary, not from observational SDK-attempt diagnostics or a generic `TypeError`.
+
 ### Context Overflow Errors
 
 When a request exceeds the model's context window, scramjet can recover automatically by compacting the conversation and retrying. This recovery only kicks in if scramjet recognizes the failure as an overflow.
@@ -522,9 +542,9 @@ When a request exceeds the model's context window, scramjet can recover automati
 Detection runs on the finalized assistant message:
 
 - `stopReason === "error"`
-- Without a Responses provider-failure diagnostic, `errorMessage` matches one of scramjet's known overflow patterns (see [`packages/ai/src/utils/overflow.ts`](https://github.com/earendil-works/pi-mono/blob/main/packages/ai/src/utils/overflow.ts)). With a Responses diagnostic, its validated category determines overflow instead of message text.
+- Without authoritative failure evidence, `errorMessage` matches one of scramjet's known overflow patterns (see [`packages/ai/src/utils/overflow.ts`](https://github.com/earendil-works/pi-mono/blob/main/packages/ai/src/utils/overflow.ts)). With recognized failure evidence, its validated category determines overflow instead of message text.
 
-If your provider returns overflow errors without Responses diagnostics and with a message scramjet does not recognize, normalize the error from the same extension that registers the provider. Use a `message_end` handler to rewrite the assistant message so its `errorMessage` starts with a phrase scramjet recognizes. The generic fallback `context_length_exceeded` is the safest choice. For a provider that produces Responses diagnostics, classify overflow at the provider-error source; rewriting `errorMessage` alone cannot override a present diagnostic. Shared Responses adapters can also record a bounded local snapshot of unfamiliar error fields separately from the readable `errorMessage`; this snapshot is display-only and never grants retry or compaction.
+If your provider returns overflow errors without any authoritative failure evidence and with a message scramjet does not recognize, normalize the error from the same extension that registers the provider. Use a `message_end` handler to rewrite the assistant message so its `errorMessage` starts with a phrase scramjet recognizes. The generic fallback `context_length_exceeded` is the safest choice. For a provider that emits `request_failure` or Responses `provider_failure` diagnostics, classify overflow at the provider-error source; rewriting `errorMessage` alone cannot override present evidence. Shared Responses adapters can also record a bounded local snapshot of unfamiliar error fields separately from the readable `errorMessage`; this snapshot is display-only and never grants retry or compaction.
 
 ```typescript
 const MY_PROVIDER_OVERFLOW_PATTERN = /your provider's overflow phrase/i;
@@ -556,7 +576,7 @@ export default function (pi: ExtensionAPI) {
 }
 ```
 
-`message_end` runs before scramjet tracks the assistant message for auto-compaction. When there is no Responses diagnostic, the rewritten `errorMessage` is what scramjet checks. With this in place, scramjet will:
+`message_end` runs before scramjet tracks the assistant message for auto-compaction. When there is no authoritative failure diagnostic, the rewritten `errorMessage` is what scramjet checks. With this in place, scramjet will:
 
 1. Detect the overflow from `errorMessage`.
 2. Drop the failed assistant message from live context.

package/docs/extensions.md

@@ -1022,7 +1022,7 @@ if (usage && usage.tokens !== null && usage.tokens > usage.contextWindow * 0.8)
 
 ### ctx.compact()
 
-Trigger compaction without awaiting completion. Use `onComplete` and `onError` for follow-up actions.
+Trigger compaction without awaiting completion. Use `onComplete` and `onError` for follow-up actions. Competing manual/automatic compaction, branch summarization, or executing idle-origin harness work causes immediate refusal through `onError`; retry deliberately after conflicting work and its persistence settle. The admitted owner drains final `message_end` hooks and persistence while suppressing queued/new run/turn lifecycle continuation. New session harness invocations reject during manual ownership. See [manual admission and terminal drain](compaction.md#manual-admission-and-terminal-drain).
 
 ```typescript
 ctx.compact({
@@ -1628,6 +1628,8 @@ await pi.invokeHarnessTool("my_notice_tool", { text: "model changed to X" });
 
 The returned promise **resolves only after the resulting tool-result message has been persisted** to the session — a stronger guarantee than "the tool ran". A consumer that must not replace or tear down the session (e.g. before starting a fresh session) until the record row exists can safely `await` it. The promise **rejects** if no tool with `name` is registered, if a matching event fails to process/persist, if the session is disposed before it settles or is called after disposal, or if an explicit `options.toolCallId` is malformed or already pending. Rejection means the pipeline did not complete — it does **not** prove the artifact is absent (state and persistence may have partially completed), so do not blindly retry. A tool's own `execute()` error is not a rejection: it resolves normally as an `isError` result.
 
+New invocations also reject during manual compaction ownership, before allocating an id or persistence acknowledgement or executing a tool. Already-admitted active-run/queued invocations drain normally; manual compaction refuses existing idle-origin execution rather than pretending `Agent.waitForIdle()` covers it. This session-local admission policy does not guard direct underlying Agent calls.
+
 `options.toolCallId` (optional) sets an explicit tool-call id (must be provider-safe: 1-64 characters of `[a-zA-Z0-9_-]`); a provider-safe id is generated when omitted.
 
 Do not `await` this from work whose return is required for the agent to reach the mid-run drain point or for the event queue to persist the result (a model-callable `execute`, `beforeToolBatch`/`beforeToolCall`/`afterToolCall`/`prepareNextTurn`, or an awaited event handler) — that would deadlock. Start the promise, return from the gating callback, and await it from an independent continuation.
@@ -1884,7 +1886,13 @@ pi.registerTool({
 });
 ```
 
-**Signaling errors:** To mark a tool execution as failed (sets `isError: true` on the result and reports it to the LLM), throw an error from `execute`. Returning a value never sets the error flag regardless of what properties you include in the return object.
+**Signaling errors:** Throw an error from `execute`, or return `isError: true` alongside `content` and `details`. Returned failures retain their structured details and are reported to the LLM as errors. Omitted `isError` preserves normal success; `tool_result` hooks may still override the final error flag.
+
+**Reported cost:** Results and `onUpdate` partials may include `cost: number`, the invocation's cumulative reported USD including descendants but excluding the requesting assistant. Each finite, nonnegative report replaces the previous amount; zero is authoritative, omission is not a new report, and malformed values are ignored. The latest reported cost survives a final result that omits it and `tool_result` content/details patches; hooks cannot override cost through their patch interface. On an execution throw, the latest cost-bearing partial's details and cost accompany the error text, even after a content-only update. On an after-tool hook failure, executed accounting-bearing details and cost accompany the error text. Neither failure inherits a termination hint. Non-accounting throws retain their existing empty-details behavior.
+
+**Historical cost interpretation:** A tool definition may supply synchronous, read-only `getHistoricalCost(details: unknown): number | undefined` to recover costs from older saved details. It must tolerate untrusted historical data, return only a finite nonnegative amount or `undefined`, and never execute work or call a renderer. It stays on the definition, not the executable Agent tool; consumers own aggregation and prefer valid generic message cost, including zero, over this fallback. `AgentSession.getRecordedSessionCost()` and the default footer include all journal assistant/tool costs plus live reports awaiting final append. The interpreter remains available on discovered definitions even when a tool is inactive or excluded by the execution allowlist, following the same definition collision winners without enabling execution. Missing definitions cannot recover legacy schemas. Interpreter exceptions use the extension error channel and do not break presentation.
+
+Session accounting captures scalar reports before asynchronous event hooks. A final `message_end` replacement can change tool content/details but cannot override the captured cost. Live cost transfers to recorded cost only after a successful append; append failure retains the observation in memory and uses existing persistence/settlement error reporting, without automatic retry or a durability claim. Disposal clears live ownership; ordinary abort does not discard it. Compaction/tree navigation retain whole-journal recorded totals, and forks count only copied records. This does not redefine `/session`, RPC statistics, or custom footers.
 
 **Early termination:** Return `terminate: true` from `execute()` to hint that the automatic follow-up LLM call should be skipped after the current tool batch. This only takes effect when every finalized tool result in that batch is terminating. See [examples/extensions/structured-output.ts](../examples/extensions/structured-output.ts) for a minimal example where the agent ends on a final structured-output tool call.
 
@@ -2659,7 +2667,7 @@ const highlighted = highlightCode(code, lang, theme);
 
 - Extension errors are logged, agent continues
 - `tool_call` errors block the tool (fail-safe)
-- Tool `execute` errors must be signaled by throwing; the thrown error is caught, reported to the LLM with `isError: true`, and execution continues
+- Tool `execute` errors may be signaled by throwing or returning `isError: true`; failures are reported to the LLM and execution continues unless the finalized batch explicitly terminates
 
 ## Mode Behavior
 

package/docs/json.md

@@ -55,6 +55,8 @@ Extended messages from [`packages/coding-agent/src/core/messages.ts`](https://gi
 - `BranchSummaryMessage` (line 55)
 - `CompactionSummaryMessage` (line 62)
 
+Tool execution updates (`partialResult.cost`), execution-end results (`result.cost`), and final `toolResult` messages (`message.cost`) may carry optional finite, nonnegative cumulative invocation USD, including descendants but excluding the requesting assistant. Reports replace prior amounts rather than representing additional expenditures; zero is authoritative and omission is not a new report. Final costs remain present for reported failures (`isError: true`). These are additive fields in JSON and RPC events; closed-schema clients must allow them. Existing tools may omit them.
+
 ## Output Format
 
 Each line is a JSON object. The first line is the session header:

package/docs/sdk.md

@@ -96,6 +96,9 @@ interface AgentSession {
   // Harness-originated tool execution (resolves after the tool-result is persisted)
   invokeHarnessTool(name: string, args: unknown, options?: { toolCallId?: string }): Promise<void>;
 
+  // Whole-journal assistant/tool costs plus live reports awaiting final append
+  getRecordedSessionCost(): number;
+
   // State access
   agent: Agent;
   model: Model | undefined;
@@ -443,6 +446,12 @@ The `createAgentSession()` stream path checks the current request's estimated in
 
 OpenRouter's implicit aggregate output maximum is omitted because it can exclude the long-context endpoint; explicit summary output limits are retained. Vercel's Anthropic route still requires an output limit. Codex still omits `max_output_tokens`. Aggregate route compatibility is not guaranteed by scalar metadata. Anthropic/Bedrock simple adapters treat explicit `maxTokens` as a combined output ceiling including thinking, rather than increasing it for reasoning.
 
+### Provider failure recovery
+
+AgentSession classifies the finalized persisted assistant snapshot. Built-in adapters distinguish provider failures from payload/response callback failures without changing message origin; callback exceptions persist only a fixed stage-specific error and closed diagnostic, not private exception text. Typed local allocation overflows retain compaction recovery, while arbitrary runtime errors cannot request recovery through suggestive prose.
+
+Recognized incomplete responses can use the existing bounded agent retry loop; tools from a failed response do not execute. Provider/SDK retries remain a separate layer. See [settings.md](settings.md#retry) for numeric validation, zero semantics and Codex-only server-delay suppression, and [custom-provider.md](custom-provider.md#failure-evidence) for evidence precedence. Neither these attempt limits nor forwarding provider options guarantees a universal request-count or elapsed-time bound.
+
 ### API Keys and OAuth
 
 API key resolution priority (handled by AuthStorage):

package/docs/session-format.md

@@ -97,6 +97,7 @@ interface ToolResultMessage {
   toolName: string;
   content: (TextContent | ImageContent)[];
   details?: any;      // Tool-specific metadata
+  cost?: number;     // Cumulative reported invocation USD, including descendants
   isError: boolean;
   timestamp: number;
 }
@@ -117,6 +118,8 @@ interface Usage {
 }
 ```
 
+Tool-result `cost` is optional additive metadata: it excludes the requesting assistant's usage and is independent of `details` and error status. Valid amounts are finite and nonnegative; explicit zero is authoritative, while absence means no generic cost was recorded, not that execution was free. This addition does not change the session format version or rewrite older records. Closed-schema readers must allow the optional field.
+
 ### Extended Message Types (from `@leanandmean/coding-agent`)
 
 ```typescript

package/docs/settings.md

@@ -108,14 +108,18 @@ The upstream runtime can check `https://pi.dev/api/latest-version` for updates,
 
 | Setting | Type | Default | Description |
 |---------|------|---------|-------------|
-| `retry.enabled` | boolean | `true` | Enable automatic agent-level retry on transient errors |
+| `retry.enabled` | boolean | `true` | Enable bounded agent retries for transient and specifically recognized incomplete responses |
 | `retry.maxRetries` | number | `3` | Maximum agent-level retry attempts |
 | `retry.baseDelayMs` | number | `2000` | Base delay for agent-level exponential backoff (2s, 4s, 8s) |
 | `retry.provider.timeoutMs` | number | SDK default | Provider/SDK request timeout in milliseconds |
 | `retry.provider.maxRetries` | number | SDK default | Provider/SDK retry attempts |
 | `retry.provider.maxRetryDelayMs` | number | `60000` | Max server-requested delay before failing (60s) |
 
-When a provider requests a retry delay longer than `retry.provider.maxRetryDelayMs` (e.g., Google's "quota will reset after 5h"), the request fails immediately with an informative error instead of waiting silently. Set to `0` to disable the cap.
+Agent retries include validated missing-terminal Responses streams and accepted responses with missing bodies. Failed-turn tool calls never execute. Known quota/authentication/permission/content rejection, callback failures, and invalid evidence do not authorize retries. Provider/SDK request attempts are separate from the agent's consecutive and cumulative limits; these settings do not establish a universal HTTP-attempt or elapsed-time budget.
+
+Codex SSE honors `retry.provider.maxRetries` (default three) and rejects server-requested waits above `retry.provider.maxRetryDelayMs`, including after the final inner attempt. That rejection suppresses outer agent retries too. Set the delay cap to `0` to disable the policy cap; long representable waits remain cancellable, and unrepresentable waits fail without retry. Other SDKs do not necessarily implement this delay cap or these request controls; forwarding an option is not proof of enforcement.
+
+Counts must be nonnegative safe integers; millisecond settings must be integers from 0 through 2,147,483,647. Invalid values generate scoped settings diagnostics and inherit a valid lower-priority value or the default without rewriting the file. Nested `retry.provider` leaves merge individually. Zero retries disables that layer's retries; zero base delay removes intentional backoff; timeout zero retains provider/SDK semantics and does not universally disable timeouts. Codex direct API callers receive a local validation failure for invalid retry options.
 
 ```json
 {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@leanandmean/coding-agent",
-  "version": "0.74.1-scramjet.67",
+  "version": "0.74.1-scramjet.69",
   "publishConfig": {
     "access": "public"
   },
@@ -37,9 +37,9 @@
     "test": "vitest --run --passWithNoTests"
   },
   "dependencies": {
-    "@leanandmean/agent": "0.74.1-scramjet.46",
-    "@leanandmean/ai": "0.74.1-scramjet.42",
-    "@leanandmean/tui": "0.74.1-scramjet.39",
+    "@leanandmean/agent": "0.74.1-scramjet.48",
+    "@leanandmean/ai": "0.74.1-scramjet.44",
+    "@leanandmean/tui": "0.74.1-scramjet.41",
     "@silvia-odwyer/photon-node": "^0.3.4",
     "chalk": "^5.5.0",
     "dictionary-en": "^4.0.0",