smith-agents 0.12.1 → 0.12.2

data/docs/INSPECTION.md

@@ -0,0 +1,39 @@
+# Graph Inspection and Runtime Readiness
+
+Smith can inspect a workflow's declared graph without running agents or advancing state. This is useful for host apps that want to render, lint, or cache a workflow shape before execution.
+
+```ruby
+report = ReplyWorkflow.validate_graph
+
+report.valid?        # => true
+report.transitions   # => read-only transition snapshots
+report.diagnostics   # => errors and warnings for missing states or routes
+report.metrics       # => state, transition, reachability, and terminal-state counts
+```
+
+Graph inspection is static and diagnostic-only. Runtime execution, persistence, progress projection, retries, and recovery remain host-owned concerns.
+
+Smith also exposes a static runtime-readiness report for checks that require
+declared runtime bindings but still do not execute the workflow:
+
+```ruby
+readiness = ReplyWorkflow.runtime_readiness
+
+readiness.ready?       # => true when no readiness/topology errors exist
+readiness.status       # => :ready, :warning, or :not_ready
+readiness.diagnostics  # => topology diagnostics + runtime binding diagnostics
+```
+
+Runtime readiness checks graph topology, registered agent bindings, model
+requirements for structured runtime roles, lazy/uninspectable bindings, invalid
+non-agent bindings, nested workflow readiness, and fan-out branch binding counts.
+It does not call providers, resolve lazy container blocks, run tools, enqueue
+jobs, or verify host-owned durability.
+
+Readiness metrics include both direct graph counts and transitive counts folded
+in from nested workflows.
+
+Transition snapshots include runtime contracts for complex primitives where
+Smith owns executable semantics: heterogeneous fan-out, evaluator-optimizer, and
+orchestrator-worker transitions expose bounded settings, output shapes, and
+transition-level resume behavior for host renderers and compilers.

data/docs/PATTERNS.md

@@ -1,6 +1,6 @@
 # Patterns
 
-Working examples for each Smith workflow pattern. The table in the [README](../README.md#patterns) is the quick selection rule; this file is the depth.
+Working examples for each Smith workflow pattern. The steps table in the [README](../README.md#steps) is the quick selection rule; this file is the depth.
 
 ## Example 1: Single-Step Workflow
 
@@ -649,6 +649,6 @@ The yielded step object exposes a narrow, read-heavy surface:
 - **No output**: Deterministic steps produce no session message output. `last_output` continues to mean the last agent output.
 - **No budget**: No tokens or cost consumed.
 - **Persistence**: Context writes (always to a persisted key, see above) and written outcomes survive `to_state`/`from_state`. The block itself (a Proc) lives on the class-level Transition and is never serialized.
-- **Trace**: Emits `:deterministic_step` traces whose `outcome` is `:started`, `:success`, `:routed`, or `:failed` (`outcome`, not `result`, which the content policy strips). A failure carries `error_class`, `error_family`, and `retryable` when known, never the error message. When a step writes an outcome, the trace includes `outcome_kind`.
+- **Trace**: Emits `:deterministic_step` traces whose `outcome` is `:started`, `:success`, `:routed`, `:failed`, or `:interrupted` (an interrupting error, `Smith.config.interrupting_errors`, with no error fields) (`outcome`, not `result`, which the content policy strips). A failure carries `error_class`, `error_family`, and `retryable` when known, never the error message. When a step writes an outcome, the trace includes `outcome_kind`.
 - **Mutual exclusivity**: `compute` and `run` cannot be combined with `execute`, `route`, `workflow`, `optimize`, or `orchestrate`. A transition declares exactly one primary execution body.
 - **No hidden scheduler**: `compute` and `run` execute synchronously inside the current runner. Use them for bounded deterministic logic, not for durable polling or wake-up loops.

data/docs/PERSISTENCE.md

@@ -940,7 +940,7 @@ end
 
 ### Batch Waits
 
-A step whose mode is `:batch` (`execute` or `fan_out`, README "Batch mode")
+A step whose mode is `:batch` (`execute` or `fan_out`, [AGENTS.md](AGENTS.md), Batch mode)
 sends no request: it captures each call's request as a synchronous call would
 post it, hands it to `Smith.config.batch_queue` and waits.
 
@@ -998,7 +998,7 @@ post it, hands it to `Smith.config.batch_queue` and waits.
 
 ## Budget Ledgers
 
-A workflow's `budget_ledger` (README, "Budget ledgers") is never persisted,
+A workflow's `budget_ledger` ([BUDGETS.md](BUDGETS.md), Budget ledgers) is never persisted,
 and restoring a run never builds it: Smith builds it at the run's first
 request from the run's `budget_ledger_scope`, the values its scope keys had in
 its trusted context when it was created, kept with the run (a run stored
@@ -1110,13 +1110,17 @@ Smith.config.run_launcher = QueueLauncher.new
 ## Replay Safety: Idempotency Keys
 
 A replayed step's provider calls are paid for again unless
-`Smith.config.response_memo` keeps them (README, "Response memo"): each call
+`Smith.config.response_memo` keeps them ([ERRORS_AND_RETRIES.md](ERRORS_AND_RETRIES.md), Response memo): each call
 is keyed by its position's idempotency scope (so a settled failure's new key
 generation reaches new records), its attempt, its model in the fallback chain
 and its place among the attempt's calls. Nothing about a memo is persisted
 with the run.
 
 
+A step that met one of the host's interrupting errors (`Smith.config.interrupting_errors`) stopped where
+it was, and is replayed as one whose process died ([ERRORS_AND_RETRIES.md](ERRORS_AND_RETRIES.md),
+Interrupting errors).
+
 A step that is replayed, by a retry or by a restore after a crash, sends its
 outside requests again. Smith gives each request a key the step can pass along,
 so a receiver that honours idempotency keys can drop the repeat. Passing the key
@@ -1237,7 +1241,7 @@ negative, or non-integer versions fail closed instead of being overwritten.
 The default host model needs a unique key, a JSON/text payload, and Rails'
 standard optimistic-locking column. `bin/rails generate smith:install`
 writes its migration, model and initializer (`--state-model`,
-`--max-state-bytes`; see the README's Installation), with a CHECK on the
+`--max-state-bytes`), with a CHECK on the
 payload's size a third above `persistence_max_bytes`. Smith keeps a run
 within the bound: a stored run's step whose result would not fit fails with
 `Smith::PersistedStateTooLarge` before the result is kept (its usage
@@ -1426,3 +1430,170 @@ Why this matters:
   which ends with the step, so outside a run's steps `Smith.artifacts` is the
   configured store itself; a child run reads its ancestors' refs too
 - nested workflows inherit artifact scope correctly
+
+## Adapters and Host-Coordinated Execution
+
+```ruby
+# Persist after every advance
+result = ReplyWorkflow.run_persisted!(
+  context: { user_message: "..." },
+  adapter: Smith.persistence_adapter
+)
+
+# Resume later
+result = ReplyWorkflow.run_persisted!(
+  key: "ticket:T-1042",
+  adapter: Smith.persistence_adapter
+)
+
+# Admit host-supplied conversation input at a stable workflow boundary.
+workflow = ReplyWorkflow.restore("ticket:T-1042", adapter: Smith.persistence_adapter)
+admission = workflow.append_session_messages!(role: :user, content: "Continue")
+```
+
+`append_session_messages!` validates and owns a bounded canonical copy of the
+message batch, appends it under Smith's workflow lifecycle lock, and returns an
+immutable digest witness. The host still owns session identity, transaction
+coordination, persistence, idempotency, and resume policy.
+
+Built-in adapters (all except `ActiveRecord` support TTL; `ActiveRecord` has no
+expiry column, so its writes refuse a non-nil `persistence_ttl`, while reads and
+deletes keep working and `smith doctor --durability` flags the combination;
+`Redis`, `ActiveRecord`, and `Memory` support optimistic locking via
+`store_versioned` and exact dispatch claims via `replace_exact`; `ActiveRecord`
+participates in the caller's transaction for host-record coordination;
+`Redis` and `Memory` also support heartbeat probes via `record_heartbeat` /
+`last_heartbeat`):
+
+- `:memory` — in-process Hash, intended for tests and `test_mode = true`
+- `:redis` — Redis client; uses WATCH/MULTI/EXEC for CAS
+- `:rails_cache`, `:solid_cache` — Rails cache backends
+- `:cache_store` — any object responding to `write/read/delete`
+- `:active_record` — keyed ActiveRecord model with an enabled optimistic-locking
+  column for row-level CAS; Smith's logical version remains in the JSON payload
+
+The sections above cover schema versioning,
+seed-drift validation, the `idempotency_mode :strict` step-in-progress contract,
+host-coordinated message admission, and opt-in restart-safe prepared recovery.
+Hosts that need to commit a durable attempt before provider or tool work can
+separate verification from execution with
+`authorize_prepared_step_execution!`, commit their own attempt ledger, and then
+consume the exact process-local capability through
+`execute_authorized_prepared_step!`. Smith does not own the host lease, attempt
+record, scheduler, or unknown-outcome policy. Authorization discovers the exact
+agent bindings reachable by that transition with a bounded `O(V + E)` graph
+walk, then resolves the bounded binding set in one registry epoch. A concurrent
+registry writer therefore cannot produce a mixed authorization containing
+bindings from two registry states or replace a captured binding between
+authorization and work. Every public Dry::Container mutation path participates
+in that epoch. Execution authorization accepts only concrete registered
+`Smith::Agent` classes; lazy container callbacks are not executed while
+authority is captured.
+While the capability is active, a host may call `each_agent_binding` with a
+block to inspect the exact captured `(registry_name, agent_class)` pairs before
+external work. Enumeration is process-local, performs no dispatch, and is
+rejected after release or consumption; it deliberately does not return a lazy
+enumerator that could escape the authorization scope.
+Reachable nested workflow transition contracts are captured and revalidated
+before child execution. The typed result owns a bounded snapshot of JSON-like
+Ruby values; unsupported mutable output values fail closed.
+During authorized execution, Smith also seals its private execution path against
+workflow subclass method-name collisions, including nested child workflows.
+Ordinary non-authorized runs retain normal Ruby override behavior.
+
+For `parallel: true` and `fan_out` transitions, a host can split one confirmed
+dispatch into independently scheduled branch operations without invoking agent
+classes outside Smith:
+
+```ruby
+preparation = workflow.prepare_composite_step!
+
+# Persist preparation.plan and preparation.input in host-owned records.
+# Each worker recovers the same confirmed dispatch and receives one descriptor.
+branch = preparation.plan.branches.fetch(0)
+execution = preparation.plan.execution_for(branch)
+outcome = branch_workflow.execute_prepared_composite_branch!(
+  execution:,
+  input: preparation.input
+)
+
+# After every branch has a terminal outcome, use another fresh recovery to
+# complete the Smith transition exactly once.
+result = reducer_workflow.reduce_prepared_composite_step!(
+  plan: preparation.plan,
+  input: preparation.input,
+  outcomes: ordered_or_unordered_complete_outcome_set,
+  primary_failure: host_committed_failed_branch_key
+)
+```
+
+Plans bind the exact prepared dispatch, host-supplied complete workflow
+`definition_digest`, execution-semantics version, prepared input digest, budget
+baseline, one shared artifact execution namespace, ordered branch identities,
+and explicit policies. Each plan, input, outcome, or effect payload is
+canonical, immutable, bounded to 1 MiB after JSON encoding, rejects missing or
+unknown fields, and redacts exception messages and backtraces.
+Reduction additionally bounds cumulative effects to 1 MiB and final aggregate
+output to 4 MiB before consuming execution authority. Reduction accepts either
+an `Array` or an `Enumerator`, validates each outcome incrementally, and stops
+as soon as a cumulative bound is crossed. The transport hard limit is 10,000
+branches; every recovered worker also enforces the current
+`Smith.config.parallel_branch_limit` before work.
+
+Planning is `O(B * D)`, where `D` is the bounded number of budget dimensions.
+The coordinator derives a compact selected-branch execution envelope in
+`O(D)` without copying the full branch collection. The scoped branch execution
+entry point captures and validates one selected binding in `O(D)` and constant
+time with respect to total branch count; the reducer validates the complete plan
+once. Reduction is `O(B + P log P)` in
+the worst case because canonical nested Hash ownership sorts keys, and uses
+`O(B + P)` bounded space.
+Smith does not persist plans or outcomes, select scheduling slots, claim work,
+choose an uncertain-outcome policy, or infer the primary failure. The host
+durably owns those decisions, gives each branch and reduction a separately
+recovered workflow instance, and must pass a complete outcome set. Transport or
+encoding faults become typed branch failures when Smith can preserve accounting
+evidence; otherwise the host must treat the worker result as uncertain and must
+not infer safe retry. Resume policy is incomplete branches only; per-branch
+retry is intentionally unsupported in this contract version.
+Composite payload digests prove canonical content integrity, not worker
+authenticity. Before reduction, the host must authenticate every outcome against
+its own claimed operation identity and current fencing token; those durable
+receipts remain outside Smith.
+
+The supported composite API is the three scoped workflow methods above plus the
+immutable transport values they return or accept: `Preparation`, `Plan`,
+`Input`, `Branch`, `BranchExecution`, `BranchOutcome`, `BranchFailure`, `Error`,
+`Effects`, and `Reduction`. Other constants under `Workflow::Composite` are
+implementation collaborators, not extension points or compatibility promises.
+
+Every agent used by this durable composite boundary must declare a stable
+host-computed SHA-256 execution identity. The identity represents the exact
+sealed executable asset the host intends to dispatch; Smith records it in the
+branch descriptor and rejects registry replacement after planning. Identities
+are deliberately not inherited because a subclass is a distinct executable
+implementation.
+
+```ruby
+class Researcher < Smith::Agent
+  execution_identity Digest::SHA256.hexdigest(generated_asset_manifest)
+  register_as :researcher, publish: false
+end
+
+Researcher.freeze
+Researcher.publish_registration!
+```
+
+Hosts that generate agent classes dynamically may call
+`register_as :name, publish: false` while constructing the class. This records
+the class identity without exposing it through `Smith::Agent::Registry`. After
+the host has validated and sealed the class, it may publish the exact class with
+`agent_class.publish_registration!`. Publication always uses the staged,
+immutable identity; callers cannot provide a second key. Registry collision and
+mutation epoch guarantees remain unchanged.
+
+Hosts that persist or cache generated executable definitions should include
+`Smith::EXECUTION_SEMANTICS_VERSION` in their definition identity. The gem
+version describes the package release; the execution-semantics version changes
+when Smith's executable workflow contract changes and must participate in host
+cache invalidation.

data/docs/TESTING.md

@@ -0,0 +1,88 @@
+# Testing and the Doctor
+
+## Testing
+
+`require "smith/testing"` runs a host's workflows in its tests without a
+provider. A stubbed agent's requests are answered where they would go over
+the wire, so everything above runs as for a provider's reply: the tool loop,
+budget holds, the response memo, streaming, output checks and repairs, usage
+entries and budgets.
+
+```ruby
+# test/test_helper.rb
+require "smith/testing/minitest"   # or "smith/testing/rspec" for matchers
+
+class NoticeTest < ActiveSupport::TestCase
+  include Smith::Testing::Assertions
+
+  test "drafts with the venue's hours, then waits for approval" do
+    Smith::Testing.stub_agent(Writer, outputs: [Smith::Testing.tool_call(Lookup, venue: "the hall"),
+                                                "The hall opens at 8."],
+                                      usage: { input_tokens: 30, output_tokens: 6 })
+    run = Notice.new(context: {})
+
+    Smith::Testing.run_until_waiting(run)
+    assert_waiting_for run, :approval
+    Smith::Testing.resume_with(run, payload: { approved: true })
+    assert_transitioned_through run, :draft, :approve, :publish
+    assert_usage_recorded run, agent_name: Writer, stop_reason: "end_turn", input_tokens: 30
+  end
+end
+```
+
+- `stub_agent(agent, outputs:, usage:, stop_reason:, stream:)` answers the
+  agent's requests (its class, or its registered name) with `outputs` in
+  turn, the last answering every request after it: a String reply, a Hash as
+  JSON text (for an output schema), or `tool_call(ToolClass, **arguments)`
+  (an Array of them for one round), which runs the agent's own tools before
+  its next request. A tool request naming a tool the agent does not have
+  fails the call. `stop_reason: "max_tokens"` tests a truncated reply;
+  `stream: n` sends a streamed step's reply in pieces of n characters. The
+  stub keeps the messages each request sent (`calls`). Only the agent's own
+  requests are answered, never a tool's own provider call. The chat is still
+  built, so configure the provider (any key). An agent with no stub goes over
+  the wire; `Smith::Testing.unstubbed = :refuse` fails its call with
+  `Smith::Testing::UnstubbedCall` instead, so a mistyped stub never reaches a
+  paid provider.
+- `run_until_waiting(run)` runs until the run waits or ends and returns its
+  wait (a durable retry's backoff is ended at once unless `retries: false`);
+  `resume_with(run, payload:)` delivers the wait's event and runs on;
+  `expire(run)` ends the wait as if its deadline had passed and runs on.
+- A batch step's request is captured as ever (the batch queue gets it);
+  `resume_with(run, payload: Smith::Testing.batch_payload(run, Smith::Testing.batch_result("text")))`
+  answers it, and a fan-out's `batch_payload` takes results by branch key.
+- Minitest: `assert_waiting_for(run, :event)` (or `kind: :batch`),
+  `assert_transitioned_through(run, *transitions)` (in order; the steps the
+  instance completed, not a failed or timed-out one),
+  `assert_usage_recorded(run, count:, **usage_entry_attributes)`. RSpec:
+  `be_waiting_for`, `have_transitioned_through`, `have_recorded_usage`.
+- Stubs belong to the process; both helpers drop them after each test. An
+  in-process `retry_on` backoff still sleeps: stub `sleep` on the workflow,
+  or declare the retry `durable: true`.
+
+## Doctor
+
+After adding Smith, verify the integration:
+
+```bash
+# Plain Ruby
+smith doctor              # offline checks
+smith doctor --live       # live provider call
+smith doctor --durability # persistence round-trip
+smith install             # scaffold config/smith.rb
+
+# Rails
+bin/rails smith:doctor
+bin/rails smith:doctor:live
+bin/rails smith:doctor:durability
+bin/rails generate smith:install
+```
+
+Doctor verifies: Smith loads, RubyLLM loads, minimal workflow boots, configuration is non-empty, serialization round-trips, persistence adapter works, and (with `--live`) a real provider call succeeds.
+An `:active_record` persistence adapter's model is checked on every run: its
+`lock_version` (Rails' locking column), a text payload, and a string key the
+database keeps unique, at least 64 characters wide, read from schema
+metadata (skipped when the database cannot be reached; an encrypted payload
+is a warning). The durability probe writes as runs write (`store_versioned`
+where the adapter has it, and `persistence_max_bytes`), so it meets what a
+run would.

data/docs/TOOLS_AND_GUARDRAILS.md

@@ -387,7 +387,8 @@ receive an inspection-only argument Hash. The full admitted argument graph is
 Smith-owned and frozen, so policy inspection cannot rewrite the values that
 reach `perform`. Errors raised from inside `perform`, including
 `Smith::DeadlineExceeded`, do not prove that external work was absent and retain
-their original type. Process-fatal exceptions are never converted into receipt
+their original type. Process-fatal exceptions, and errors the host names in
+`Smith.config.interrupting_errors`, are never converted into receipt
 outcomes or terminal callbacks: an admitted receipt remains unresolved and the
 durable host must reconcile it before any resume decision. Smith never
 automatically replays that uncertain work. Failure-handler errors propagate
@@ -450,3 +451,60 @@ class GuardedWorkflow < Smith::Workflow
   end
 end
 ```
+
+## Shipped Tools and Tool Budgets
+
+Smith ships `Tools::WebSearch`, `Tools::UrlFetcher`, and `Tools::Think`. Tools declare provider compatibility via `compatible_with`; Smith's normalizer routes or drops them per-attempt.
+
+`Tools::WebSearch` and `Tools::UrlFetcher` are stubs: they declare each
+tool's parameters and description, and their `perform` raises
+`NotImplementedError`. A host subclasses them with its own search or fetch
+behind `perform`. `Tools::Think` works as shipped.
+
+```ruby
+class WebSearch < Smith::Tools::WebSearch
+  def perform(query:, max_results: 5) = SearchClient.search(query, limit: max_results)
+end
+
+class UrlFetcher < Smith::Tools::UrlFetcher
+  def perform(url:) = PageClient.fetch(url)
+end
+
+class SearchAgent < Smith::Agent
+  register_as :search_agent
+  model "claude-opus-4-7"
+  tools WebSearch, UrlFetcher # the host's subclasses
+  budget tool_calls: 3
+  tool_budget_exhaustion :complete
+end
+```
+
+Agent tool-call budgets default to raising `Smith::BudgetExceeded` when a tool
+tries to execute beyond the configured limit. Agents that declare
+`tool_budget_exhaustion :complete` must also declare a finite `tool_calls`
+budget. Smith then consumes each model-requested call before dispatch, executes
+only a complete batch admitted against both agent and workflow limits, and
+performs one tool-disabled completion when the allowance is exhausted or the
+provider returns an oversized batch. The bounded provider loop is iterative,
+not recursive. Calls to unavailable tools consume allowance as well, so a
+provider cannot create an unbounded correction loop. Workflow reservations are
+reconciled to actual Smith tool execution.
+
+Hosts may install `Smith::Tool::CallBudget` with exact per-tool limits through
+`Smith::Tool.with_call_budget`. Smith propagates that same root allowance into
+parallel and heterogeneous fan-out worker threads; agent budgets narrow the
+root and all branches share its atomic counters.
+
+Graceful exhaustion is an in-process protocol guarantee, not durable recovery
+for the model/tool loop. If the process exits after a tool begins and before the
+agent completes, the host must treat the outcome as uncertain unless it owns a
+durable per-invocation receipt and reconciliation contract. Smith also refuses
+to start a fresh fallback-model chat after tool execution has begun because that
+would discard the captured evidence and conversation protocol.
+
+Within a host invocation scope, `Smith::Tool.current_invocation` exposes the
+normalized tool-call ID and deterministic execution/batch ordinals. The ID is
+RubyLLM correlation metadata and must not be treated as a provider idempotency
+key. Durable hosts can pass an explicitly seeded
+`Smith::Tool::InvocationSequence` to `with_invocation_context`; Smith itself does
+not persist or reconcile operation receipts.

data/lib/smith/agent/completion.rb

@@ -15,8 +15,9 @@ module Smith
       attribute :response_facts, Types::Hash.default({}.freeze)
 
       # `stream` is set for a streamed call: the facts of each of its
-      # requests, read from the stream, and the request's output limit
-      # (Providers::StreamFacts). Each response's usage carries its facts and
+      # requests, read from the stream, the request's output limit, and, for
+      # a call that failed, how many of its replies a response memo kept
+      # ahead of them (Providers::StreamFacts). Each response's usage carries its facts and
       # `mode: :stream`. `mode` marks another call's usage (:batch).
       def self.from_messages(response:, messages:, stream: nil, mode: nil)
         assistant_messages = Array(messages).select do |message|
@@ -56,7 +57,8 @@ module Smith
       def self.response_facts(messages, stream)
         return messages.map { Providers::ResponseFacts.from_message(_1) } unless stream
 
-        Providers::StreamFacts.per_message(messages, stream.fetch(:rounds), stream.fetch(:output_limit))
+        Providers::StreamFacts.per_message(messages, stream.fetch(:rounds), stream.fetch(:output_limit),
+                                           kept: stream.fetch(:kept, 0))
       end
 
       def self.build_provider_usages(messages, streamed_facts, mode)

data/lib/smith/agent/provider_completion.rb

@@ -123,7 +123,7 @@ module Smith
         observed_reference ||= observed_model_reference(chat, fallback: model_reference)
         prefix_usages = account_completed_prefix(
           agent_class, observed_reference, new_chat_messages(chat, message_count),
-          attempt_id: facts[:attempt_id], stream_rounds: streamed_rounds(collected || [])
+          attempt_id: facts[:attempt_id], stream: streamed(collected || [])
         )
         attempt = failed_provider_attempt(e, observed_reference, timer, facts, prefix_usages)
         raise unless provider_failure?(e) || unsupported_content?(e)

data/lib/smith/agent/provider_streaming.rb

@@ -39,12 +39,13 @@ module Smith
       # call a response memo kept part of goes on from.
       def live_completion(chat, model_reference, collected, facts, resumed)
         ledger = run_budget_ledger
-        return wire_completion(chat, model_reference, collected) unless ledger
+        kept = resumed&.rounds || 0
+        return wire_completion(chat, model_reference, collected, kept) unless ledger
 
         start = resumed ? resumed.start : chat_message_count(chat)
-        holding = { pausable: budget_pausable?, rounds: resumed&.rounds || 0 }
+        holding = { pausable: budget_pausable?, rounds: kept }
         Providers::RequestHolds.within(chat_provider(chat), ledger, facts[:attempt_id], **holding) do |holder|
-          held_completion(holder, chat, start) { wire_completion(chat, model_reference, collected) }
+          held_completion(holder, chat, start) { wire_completion(chat, model_reference, collected, kept) }
         end
       end
 
@@ -82,21 +83,23 @@ module Smith
         false
       end
 
-      def wire_completion(chat, model_reference, collected)
+      # `kept`: the replies a response memo kept for the call, ahead of
+      # those its stream answers (a failed call's facts pair after them).
+      def wire_completion(chat, model_reference, collected, kept = 0)
         stream = @step_stream
         return [chat.complete, nil] unless stream
 
         stream.start(attempt: @step_attempt, model: model_reference.model_id, provider: model_reference.provider&.to_s)
-        response, rounds = streamed_completion(chat, stream, collected)
+        response, rounds = streamed_completion(chat, stream, collected, kept)
         [response, { rounds:, output_limit: request_output_limit(chat, model_reference) }]
       end
 
       # The chat's reply, its text sent as it comes (without <think> text),
       # and the facts of each of its requests.
-      def streamed_completion(chat, stream, collected)
+      def streamed_completion(chat, stream, collected, kept)
         response = nil
         thinking = Streaming::ThinkTags.new
-        rounds = Providers::StreamFacts.collect(chat_provider(chat)) do |collection|
+        rounds = Providers::StreamFacts.collect(chat_provider(chat), kept:) do |collection|
           collected << collection
           response = chat.complete { |chunk| stream.delta(thinking.strip(chunk.content, collection.rounds.length)) }
         end
@@ -138,8 +141,11 @@ module Smith
         tools.respond_to?(:empty?) && !tools.empty?
       end
 
-      # The facts of a streamed call that failed, if it streamed.
-      def streamed_rounds(collected) = collected.first&.facts
+      # What a streamed call that failed streamed, if it did.
+      def streamed(collected)
+        collection = collected.first
+        { rounds: collection.facts, output_limit: nil, kept: collection.kept } if collection
+      end
 
       def chat_provider(chat)
         llm_chat = chat.respond_to?(:to_llm) ? chat.to_llm : chat

data/lib/smith/agent/usage_tracking.rb

@@ -27,8 +27,7 @@ module Smith
       # billed: they are recorded and settled into the budget like a failed
       # attempt's usage. Returns the provider usages recorded, so the
       # attempt's trace carries the same totals its usage entries hold.
-      def account_completed_prefix(agent_class, model_reference, messages, attempt_id: nil, stream_rounds: nil)
-        stream = stream_rounds && { rounds: stream_rounds, output_limit: nil }
+      def account_completed_prefix(agent_class, model_reference, messages, attempt_id: nil, stream: nil)
         completion = Completion.from_messages(response: nil, messages:, stream:)
         record_billed_usages(agent_class, completion.provider_usages, :partial_attempt, model_reference, attempt_id:)
         completion.provider_usages

data/lib/smith/interruptions.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Smith
+  # The errors a host names in Smith.config.interrupting_errors: the host
+  # could not do its part (its database or queue out of reach), so the step
+  # did not go wrong, it was cut short. A step that meets one stops where it
+  # is, as if its worker had stopped. Smith takes no failure route, makes no
+  # retry, records no failure and notifies no tool failure handler; the
+  # error reaches the caller of advance!, run!, resume! or expire_wait! as it
+  # was raised, even when Smith or the host wrapped it in another error on
+  # the way (its `cause`). Smith does not count interruptions: a step that
+  # meets one every time is the host's to bound. A persisted run's state is as the step left it: a
+  # lax run takes the step again when it is next advanced, a strict run
+  # finds it under way.
+  module Interruptions
+    CAUSE_DEPTH = 16
+
+    module_function
+
+    # The interrupting error that `error` is, or was raised from, or nil:
+    # the error, or one of its first CAUSE_DEPTH - 1 causes (so a chain of
+    # causes that loops ends too). A provider's own failure (RubyLLM's, or
+    # its HTTP client's) is the provider's, whatever it was raised from: it
+    # interrupts only when its own class is named.
+    def find(error)
+      classes = Smith.config.interrupting_errors
+      return nil if classes.empty?
+
+      current = error
+      CAUSE_DEPTH.times do
+        return nil if current.nil?
+        return current if classes.any? { current.is_a?(_1) }
+        return nil if provider_failure?(current)
+
+        current = current.cause
+      end
+      nil
+    end
+
+    def provider_failure?(error)
+      error.is_a?(RubyLLM::Error) || (defined?(Faraday::Error) && error.is_a?(Faraday::Error))
+    end
+
+    # Raises the interrupting error `error` is or was raised from, if any.
+    def pass!(error)
+      interruption = find(error)
+      raise interruption, cause: interruption.cause if interruption
+    end
+  end
+end

data/lib/smith/launchers/inline.rb

@@ -26,6 +26,7 @@ module Smith
 
         ChildRun.result(ref: child.ref, run_result: run)
       rescue StandardError => e
+        Interruptions.pass!(e)
         failed(child, nil, DiagnosticText.error_class_name(e))
       end
 

data/lib/smith/providers/request_holds.rb

@@ -57,7 +57,7 @@ module Smith
         exchange = BatchExchange.exchange_for(provider)
         return capture(provider, exchange, args, kwargs, block, &request) if exchange
 
-        ResponseMemo.round(provider, args.first) { send_held(provider, args, kwargs, block, &request) }
+        ResponseMemo.round(provider, args.first, kwargs) { send_held(provider, args, kwargs, block, &request) }
       end
 
       def send_held(provider, args, kwargs, block, &request)