smith-agents 0.12.0 → 0.12.2

data/docs/ERRORS_AND_RETRIES.md

@@ -0,0 +1,175 @@
+# Errors and Retries
+
+```ruby
+Smith::Errors.retryable?(error)
+# AgentError, DeadlineExceeded => true (always)
+# OutputTruncated, OutputRefused, OutputInvalid (OutputFailure) => false
+# DeterministicStepFailure, ToolGuardrailFailed => honors error.retryable
+# everything else => false
+
+Smith::Errors.retryable_classes
+# => [Smith::AgentError, Smith::DeadlineExceeded]  (for ActiveJob retry_on)
+
+Smith::Errors.retry_forbidden?(error)
+# => true for terminal tool-evidence failures that must never be replayed
+```
+
+A nested workflow's step fails with the child's own error when the child
+raises it, so these answers carry through nesting. When a tool ran inside the
+child, the parent step's failure becomes `ToolOutcomeUncertain`, which is never
+retried.
+
+Workflow transitions can also declare a bounded local retry policy:
+
+```ruby
+transition :draft, from: :idle, to: :done do
+  execute :writer
+  retry_on Smith::AgentError, attempts: 3, backoff: 0.1, max_delay: 1.0
+end
+```
+
+When no classes are passed, `retry_on` uses `Smith::Errors.retryable?`. No
+attempt starts once the workflow's wall-clock deadline has passed.
+This is a bounded local transition retry policy. What a receiver does with an
+idempotency key remains host-owned.
+
+For backoffs too long to sleep through in a worker, `durable: true` makes each
+one a wait of the run instead:
+
+```ruby
+transition :charge, from: :approved, to: :charged do
+  compute { |step| Payments.charge(order, idempotency_key: step.idempotency_key) }
+  retry_on Payments::Timeout, attempts: 5, backoff: 60, max_delay: 3600, durable: true
+  on_failure :charge_failed
+end
+```
+
+A failed attempt the policy retries stores the run waiting (`awaiting.kind ==
+:retry`, due when the backoff ends); the host schedules `expire_wait!` for that
+time and then continues the run, which makes the next attempt under the step's
+keys (new ones only after an attempt declared settled). See
+[PERSISTENCE.md](PERSISTENCE.md), Durable Retry.
+
+Steps and tools get idempotency keys for the requests they send outside the
+system, the same on every attempt and after a restore:
+
+```ruby
+transition :charge, from: :approved, to: :charged do
+  compute { |step| Payments.charge(order, idempotency_key: step.idempotency_key) }
+  retry_on Payments::Timeout, Payments::Declined, attempts: 3,
+           new_idempotency_key_on: [Payments::Declined]
+end
+```
+
+A retry keeps the keys, so a receiver that honours them drops a request that
+took effect before a timeout. An error named in `new_idempotency_key_on:` (or
+`step.fail!(..., new_idempotency_key: true)`) shows the attempt's own requests
+took no effect, and the attempts after it use new keys. See
+[PERSISTENCE.md](PERSISTENCE.md), Replay Safety.
+
+## Interrupting errors
+
+Some errors say the host could not do its part, not that a step went wrong:
+its database or its queue out of reach for a moment. Named as interrupting,
+they stop a step where it is, as if its worker had stopped, instead of
+failing it:
+
+```ruby
+Smith.config.interrupting_errors = [
+  ActiveRecord::ConnectionNotEstablished,
+  ActiveRecord::ConnectionFailed,
+  Smith::PersistenceIOError
+]
+```
+
+A step meets one when it is raised anywhere in the step (a compute block, a
+tool, a tool failure handler, a guardrail, a budget ledger, a nested or
+inline child run, a parallel branch) or is one of the first 15 causes of an
+error that was. Smith then takes no failure route, makes no retry
+(`retry_on`, durable ones included), records no failure (`last_failed_step`,
+`StepFailed`; a deterministic step's trace says `outcome: :interrupted`),
+notifies no tool failure handler and keeps no failed branch outcome. The
+error itself, not one that wrapped it, reaches the caller of `advance!`,
+`advance_persisted!`, `run!`, `run_persisted!`, or of `resume!` when the
+delivery completes a batch step; among parallel branches and tools run
+together it wins over every other error but a process-fatal one. The run's
+stored state is as the step left it: a lax run takes the step again, under
+the same idempotency keys and with the response memo's answers, when it is
+next advanced; a strict run is found under way on restore
+(`Smith::StepInProgressOnRestore`), as after a crash. The instance that
+raised is not advanced again: restore the run.
+
+Name only errors that mean your own systems failed. A provider's own failure
+(`RubyLLM::Error`, or its HTTP client's `Faraday::Error`) is the provider's
+whatever it was raised from, so naming a low-level class such as
+`Errno::ECONNREFUSED` never turns a provider outage into an interruption;
+name the provider's class itself for that. Smith does not count
+interruptions: a step that meets one every time (a query of its own that
+always times out, if `ActiveRecord::StatementTimeout` were named) is taken
+again for as long as the host keeps advancing it, so the host bounds it, by
+its job's retries or a count of its own. A `wall_clock` budget is charged as
+after a crash: on restore, the time from the step's start, an outage
+included, at most what is left.
+
+A response memo, a trace adapter, a stream sink, an event handler, an
+`on_step` callback, a `capture_result` block that is not strict, and a budget
+ledger's `reconcile!` and `release!` log one they meet and go on, as they do
+any error of their own. Only subclasses of StandardError can be named, never
+StandardError itself. None are named by default, so every error takes its
+step's route.
+
+## Response memo
+
+A step replayed after its process died makes its provider calls again, and
+pays for them again. A host's response memo keeps what they got:
+
+```ruby
+class ResponseMemo
+  def fetch(key) = Rails.cache.read("smith-memo/#{key}")
+  def store(key, record) = Rails.cache.write("smith-memo/#{key}", record, expires_in: 2.days)
+end
+
+Smith.config.response_memo = ResponseMemo.new
+```
+
+Each agent call (its tool loop included) is kept under a key of its position
+(the run, the step, any branch or nested position, the key generation), the
+attempts it runs under (its step's, and each enclosing run's for a nested
+workflow, so a retry never takes the attempt before's answer), its attempt
+index (its model in the fallback chain, or a follow-up's place), its place
+among the attempt's calls, and a digest of its request (model, messages,
+tools with their descriptions and parameters, schema, params). A replay
+reaches the same key; a call whose request changed since (a deployed prompt or
+tool, another model, a date in the prompt) misses and is sent again, as a miss
+always may be.
+
+Its record (`{"version" => 1 | 2, "messages" => [...], "complete" => true |
+false}`, JSON) holds the messages the call added: the model's replies
+(structured ones too) with their usage, stop reason and message id (a
+stream's included), its tool calls and their results. It is stored when a
+reply calling the tools its request offered comes, before they run; before
+each later request, with their results; and when the call ends in the model's
+reply (a call a tool halted keeps only its progress). A replayed call that
+ended sends nothing: its reply is the step's (a streamed step's sink gets the
+reply's text at once, not the earlier rounds'), its usage is recorded as if it
+had just come, once, and its `:provider_call` trace says `memo: true`; a reply
+that was cut off or refused fails the step as it did. One that had not ended
+goes on from where it was kept. Kept after its tools' results, it continues
+from them: kept tool results count as tools the step ran, so no retry or
+fallback runs them again, and each is charged to the agent's `tool_calls`
+allowance and the workflow's `tool_calls` budget as it was. Kept while its
+tools ran (the step was cut inside a tool), the reply that called them is not
+asked for again: its tools run again, under the same idempotency keys, and the
+call goes on. Either way its live part goes on from the kept rounds (on a
+budget ledger, a later round, held only when the step can pause for budget).
+A call whose messages cannot be kept (an attachment, past 1 MiB) is not
+stored, and is logged; a record Smith does not keep (its shape, or tool
+results that do not answer the calls before them) is ignored, and an error
+from `fetch` or `store` is logged by class: a memo never fails a step. Only a
+record ending in a reply whose tools have not run is version 2, which a Smith
+before 0.12.2 ignores (paying for that reply again and running its tools, as
+it did); it reads every other record. Records for calls with tools that
+0.12.1 kept are not found after the upgrade, nor these by 0.12.1, since the
+digest now covers the tools' definitions. Keeping records and dropping them (a
+TTL) are the host's; batch requests are not memoized, as their `custom_id`
+already identifies them.

data/docs/INSPECTION.md

@@ -0,0 +1,39 @@
+# Graph Inspection and Runtime Readiness
+
+Smith can inspect a workflow's declared graph without running agents or advancing state. This is useful for host apps that want to render, lint, or cache a workflow shape before execution.
+
+```ruby
+report = ReplyWorkflow.validate_graph
+
+report.valid?        # => true
+report.transitions   # => read-only transition snapshots
+report.diagnostics   # => errors and warnings for missing states or routes
+report.metrics       # => state, transition, reachability, and terminal-state counts
+```
+
+Graph inspection is static and diagnostic-only. Runtime execution, persistence, progress projection, retries, and recovery remain host-owned concerns.
+
+Smith also exposes a static runtime-readiness report for checks that require
+declared runtime bindings but still do not execute the workflow:
+
+```ruby
+readiness = ReplyWorkflow.runtime_readiness
+
+readiness.ready?       # => true when no readiness/topology errors exist
+readiness.status       # => :ready, :warning, or :not_ready
+readiness.diagnostics  # => topology diagnostics + runtime binding diagnostics
+```
+
+Runtime readiness checks graph topology, registered agent bindings, model
+requirements for structured runtime roles, lazy/uninspectable bindings, invalid
+non-agent bindings, nested workflow readiness, and fan-out branch binding counts.
+It does not call providers, resolve lazy container blocks, run tools, enqueue
+jobs, or verify host-owned durability.
+
+Readiness metrics include both direct graph counts and transitive counts folded
+in from nested workflows.
+
+Transition snapshots include runtime contracts for complex primitives where
+Smith owns executable semantics: heterogeneous fan-out, evaluator-optimizer, and
+orchestrator-worker transitions expose bounded settings, output shapes, and
+transition-level resume behavior for host renderers and compilers.

data/docs/PATTERNS.md

@@ -1,6 +1,6 @@
 # Patterns
 
-Working examples for each Smith workflow pattern. The table in the [README](../README.md#patterns) is the quick selection rule; this file is the depth.
+Working examples for each Smith workflow pattern. The steps table in the [README](../README.md#steps) is the quick selection rule; this file is the depth.
 
 ## Example 1: Single-Step Workflow
 
@@ -649,6 +649,6 @@ The yielded step object exposes a narrow, read-heavy surface:
 - **No output**: Deterministic steps produce no session message output. `last_output` continues to mean the last agent output.
 - **No budget**: No tokens or cost consumed.
 - **Persistence**: Context writes (always to a persisted key, see above) and written outcomes survive `to_state`/`from_state`. The block itself (a Proc) lives on the class-level Transition and is never serialized.
-- **Trace**: Emits `:deterministic_step` traces whose `outcome` is `:started`, `:success`, `:routed`, or `:failed` (`outcome`, not `result`, which the content policy strips). A failure carries `error_class`, `error_family`, and `retryable` when known, never the error message. When a step writes an outcome, the trace includes `outcome_kind`.
+- **Trace**: Emits `:deterministic_step` traces whose `outcome` is `:started`, `:success`, `:routed`, `:failed`, or `:interrupted` (an interrupting error, `Smith.config.interrupting_errors`, with no error fields) (`outcome`, not `result`, which the content policy strips). A failure carries `error_class`, `error_family`, and `retryable` when known, never the error message. When a step writes an outcome, the trace includes `outcome_kind`.
 - **Mutual exclusivity**: `compute` and `run` cannot be combined with `execute`, `route`, `workflow`, `optimize`, or `orchestrate`. A transition declares exactly one primary execution body.
 - **No hidden scheduler**: `compute` and `run` execute synchronously inside the current runner. Use them for bounded deterministic logic, not for durable polling or wake-up loops.

data/docs/PERSISTENCE.md

@@ -940,7 +940,7 @@ end
 
 ### Batch Waits
 
-A step whose mode is `:batch` (`execute` or `fan_out`, README "Batch mode")
+A step whose mode is `:batch` (`execute` or `fan_out`, [AGENTS.md](AGENTS.md), Batch mode)
 sends no request: it captures each call's request as a synchronous call would
 post it, hands it to `Smith.config.batch_queue` and waits.
 
@@ -998,7 +998,7 @@ post it, hands it to `Smith.config.batch_queue` and waits.
 
 ## Budget Ledgers
 
-A workflow's `budget_ledger` (README, "Budget ledgers") is never persisted,
+A workflow's `budget_ledger` ([BUDGETS.md](BUDGETS.md), Budget ledgers) is never persisted,
 and restoring a run never builds it: Smith builds it at the run's first
 request from the run's `budget_ledger_scope`, the values its scope keys had in
 its trusted context when it was created, kept with the run (a run stored
@@ -1110,13 +1110,17 @@ Smith.config.run_launcher = QueueLauncher.new
 ## Replay Safety: Idempotency Keys
 
 A replayed step's provider calls are paid for again unless
-`Smith.config.response_memo` keeps them (README, "Response memo"): each call
+`Smith.config.response_memo` keeps them ([ERRORS_AND_RETRIES.md](ERRORS_AND_RETRIES.md), Response memo): each call
 is keyed by its position's idempotency scope (so a settled failure's new key
 generation reaches new records), its attempt, its model in the fallback chain
 and its place among the attempt's calls. Nothing about a memo is persisted
 with the run.
 
 
+A step that met one of the host's interrupting errors (`Smith.config.interrupting_errors`) stopped where
+it was, and is replayed as one whose process died ([ERRORS_AND_RETRIES.md](ERRORS_AND_RETRIES.md),
+Interrupting errors).
+
 A step that is replayed, by a retry or by a restore after a crash, sends its
 outside requests again. Smith gives each request a key the step can pass along,
 so a receiver that honours idempotency keys can drop the repeat. Passing the key
@@ -1237,7 +1241,7 @@ negative, or non-integer versions fail closed instead of being overwritten.
 The default host model needs a unique key, a JSON/text payload, and Rails'
 standard optimistic-locking column. `bin/rails generate smith:install`
 writes its migration, model and initializer (`--state-model`,
-`--max-state-bytes`; see the README's Installation), with a CHECK on the
+`--max-state-bytes`), with a CHECK on the
 payload's size a third above `persistence_max_bytes`. Smith keeps a run
 within the bound: a stored run's step whose result would not fit fails with
 `Smith::PersistedStateTooLarge` before the result is kept (its usage
@@ -1426,3 +1430,170 @@ Why this matters:
   which ends with the step, so outside a run's steps `Smith.artifacts` is the
   configured store itself; a child run reads its ancestors' refs too
 - nested workflows inherit artifact scope correctly
+
+## Adapters and Host-Coordinated Execution
+
+```ruby
+# Persist after every advance
+result = ReplyWorkflow.run_persisted!(
+  context: { user_message: "..." },
+  adapter: Smith.persistence_adapter
+)
+
+# Resume later
+result = ReplyWorkflow.run_persisted!(
+  key: "ticket:T-1042",
+  adapter: Smith.persistence_adapter
+)
+
+# Admit host-supplied conversation input at a stable workflow boundary.
+workflow = ReplyWorkflow.restore("ticket:T-1042", adapter: Smith.persistence_adapter)
+admission = workflow.append_session_messages!(role: :user, content: "Continue")
+```
+
+`append_session_messages!` validates and owns a bounded canonical copy of the
+message batch, appends it under Smith's workflow lifecycle lock, and returns an
+immutable digest witness. The host still owns session identity, transaction
+coordination, persistence, idempotency, and resume policy.
+
+Built-in adapters (all except `ActiveRecord` support TTL; `ActiveRecord` has no
+expiry column, so its writes refuse a non-nil `persistence_ttl`, while reads and
+deletes keep working and `smith doctor --durability` flags the combination;
+`Redis`, `ActiveRecord`, and `Memory` support optimistic locking via
+`store_versioned` and exact dispatch claims via `replace_exact`; `ActiveRecord`
+participates in the caller's transaction for host-record coordination;
+`Redis` and `Memory` also support heartbeat probes via `record_heartbeat` /
+`last_heartbeat`):
+
+- `:memory` — in-process Hash, intended for tests and `test_mode = true`
+- `:redis` — Redis client; uses WATCH/MULTI/EXEC for CAS
+- `:rails_cache`, `:solid_cache` — Rails cache backends
+- `:cache_store` — any object responding to `write/read/delete`
+- `:active_record` — keyed ActiveRecord model with an enabled optimistic-locking
+  column for row-level CAS; Smith's logical version remains in the JSON payload
+
+The sections above cover schema versioning,
+seed-drift validation, the `idempotency_mode :strict` step-in-progress contract,
+host-coordinated message admission, and opt-in restart-safe prepared recovery.
+Hosts that need to commit a durable attempt before provider or tool work can
+separate verification from execution with
+`authorize_prepared_step_execution!`, commit their own attempt ledger, and then
+consume the exact process-local capability through
+`execute_authorized_prepared_step!`. Smith does not own the host lease, attempt
+record, scheduler, or unknown-outcome policy. Authorization discovers the exact
+agent bindings reachable by that transition with a bounded `O(V + E)` graph
+walk, then resolves the bounded binding set in one registry epoch. A concurrent
+registry writer therefore cannot produce a mixed authorization containing
+bindings from two registry states or replace a captured binding between
+authorization and work. Every public Dry::Container mutation path participates
+in that epoch. Execution authorization accepts only concrete registered
+`Smith::Agent` classes; lazy container callbacks are not executed while
+authority is captured.
+While the capability is active, a host may call `each_agent_binding` with a
+block to inspect the exact captured `(registry_name, agent_class)` pairs before
+external work. Enumeration is process-local, performs no dispatch, and is
+rejected after release or consumption; it deliberately does not return a lazy
+enumerator that could escape the authorization scope.
+Reachable nested workflow transition contracts are captured and revalidated
+before child execution. The typed result owns a bounded snapshot of JSON-like
+Ruby values; unsupported mutable output values fail closed.
+During authorized execution, Smith also seals its private execution path against
+workflow subclass method-name collisions, including nested child workflows.
+Ordinary non-authorized runs retain normal Ruby override behavior.
+
+For `parallel: true` and `fan_out` transitions, a host can split one confirmed
+dispatch into independently scheduled branch operations without invoking agent
+classes outside Smith:
+
+```ruby
+preparation = workflow.prepare_composite_step!
+
+# Persist preparation.plan and preparation.input in host-owned records.
+# Each worker recovers the same confirmed dispatch and receives one descriptor.
+branch = preparation.plan.branches.fetch(0)
+execution = preparation.plan.execution_for(branch)
+outcome = branch_workflow.execute_prepared_composite_branch!(
+  execution:,
+  input: preparation.input
+)
+
+# After every branch has a terminal outcome, use another fresh recovery to
+# complete the Smith transition exactly once.
+result = reducer_workflow.reduce_prepared_composite_step!(
+  plan: preparation.plan,
+  input: preparation.input,
+  outcomes: ordered_or_unordered_complete_outcome_set,
+  primary_failure: host_committed_failed_branch_key
+)
+```
+
+Plans bind the exact prepared dispatch, host-supplied complete workflow
+`definition_digest`, execution-semantics version, prepared input digest, budget
+baseline, one shared artifact execution namespace, ordered branch identities,
+and explicit policies. Each plan, input, outcome, or effect payload is
+canonical, immutable, bounded to 1 MiB after JSON encoding, rejects missing or
+unknown fields, and redacts exception messages and backtraces.
+Reduction additionally bounds cumulative effects to 1 MiB and final aggregate
+output to 4 MiB before consuming execution authority. Reduction accepts either
+an `Array` or an `Enumerator`, validates each outcome incrementally, and stops
+as soon as a cumulative bound is crossed. The transport hard limit is 10,000
+branches; every recovered worker also enforces the current
+`Smith.config.parallel_branch_limit` before work.
+
+Planning is `O(B * D)`, where `D` is the bounded number of budget dimensions.
+The coordinator derives a compact selected-branch execution envelope in
+`O(D)` without copying the full branch collection. The scoped branch execution
+entry point captures and validates one selected binding in `O(D)` and constant
+time with respect to total branch count; the reducer validates the complete plan
+once. Reduction is `O(B + P log P)` in
+the worst case because canonical nested Hash ownership sorts keys, and uses
+`O(B + P)` bounded space.
+Smith does not persist plans or outcomes, select scheduling slots, claim work,
+choose an uncertain-outcome policy, or infer the primary failure. The host
+durably owns those decisions, gives each branch and reduction a separately
+recovered workflow instance, and must pass a complete outcome set. Transport or
+encoding faults become typed branch failures when Smith can preserve accounting
+evidence; otherwise the host must treat the worker result as uncertain and must
+not infer safe retry. Resume policy is incomplete branches only; per-branch
+retry is intentionally unsupported in this contract version.
+Composite payload digests prove canonical content integrity, not worker
+authenticity. Before reduction, the host must authenticate every outcome against
+its own claimed operation identity and current fencing token; those durable
+receipts remain outside Smith.
+
+The supported composite API is the three scoped workflow methods above plus the
+immutable transport values they return or accept: `Preparation`, `Plan`,
+`Input`, `Branch`, `BranchExecution`, `BranchOutcome`, `BranchFailure`, `Error`,
+`Effects`, and `Reduction`. Other constants under `Workflow::Composite` are
+implementation collaborators, not extension points or compatibility promises.
+
+Every agent used by this durable composite boundary must declare a stable
+host-computed SHA-256 execution identity. The identity represents the exact
+sealed executable asset the host intends to dispatch; Smith records it in the
+branch descriptor and rejects registry replacement after planning. Identities
+are deliberately not inherited because a subclass is a distinct executable
+implementation.
+
+```ruby
+class Researcher < Smith::Agent
+  execution_identity Digest::SHA256.hexdigest(generated_asset_manifest)
+  register_as :researcher, publish: false
+end
+
+Researcher.freeze
+Researcher.publish_registration!
+```
+
+Hosts that generate agent classes dynamically may call
+`register_as :name, publish: false` while constructing the class. This records
+the class identity without exposing it through `Smith::Agent::Registry`. After
+the host has validated and sealed the class, it may publish the exact class with
+`agent_class.publish_registration!`. Publication always uses the staged,
+immutable identity; callers cannot provide a second key. Registry collision and
+mutation epoch guarantees remain unchanged.
+
+Hosts that persist or cache generated executable definitions should include
+`Smith::EXECUTION_SEMANTICS_VERSION` in their definition identity. The gem
+version describes the package release; the execution-semantics version changes
+when Smith's executable workflow contract changes and must participate in host
+cache invalidation.

data/docs/TESTING.md

@@ -0,0 +1,88 @@
+# Testing and the Doctor
+
+## Testing
+
+`require "smith/testing"` runs a host's workflows in its tests without a
+provider. A stubbed agent's requests are answered where they would go over
+the wire, so everything above runs as for a provider's reply: the tool loop,
+budget holds, the response memo, streaming, output checks and repairs, usage
+entries and budgets.
+
+```ruby
+# test/test_helper.rb
+require "smith/testing/minitest"   # or "smith/testing/rspec" for matchers
+
+class NoticeTest < ActiveSupport::TestCase
+  include Smith::Testing::Assertions
+
+  test "drafts with the venue's hours, then waits for approval" do
+    Smith::Testing.stub_agent(Writer, outputs: [Smith::Testing.tool_call(Lookup, venue: "the hall"),
+                                                "The hall opens at 8."],
+                                      usage: { input_tokens: 30, output_tokens: 6 })
+    run = Notice.new(context: {})
+
+    Smith::Testing.run_until_waiting(run)
+    assert_waiting_for run, :approval
+    Smith::Testing.resume_with(run, payload: { approved: true })
+    assert_transitioned_through run, :draft, :approve, :publish
+    assert_usage_recorded run, agent_name: Writer, stop_reason: "end_turn", input_tokens: 30
+  end
+end
+```
+
+- `stub_agent(agent, outputs:, usage:, stop_reason:, stream:)` answers the
+  agent's requests (its class, or its registered name) with `outputs` in
+  turn, the last answering every request after it: a String reply, a Hash as
+  JSON text (for an output schema), or `tool_call(ToolClass, **arguments)`
+  (an Array of them for one round), which runs the agent's own tools before
+  its next request. A tool request naming a tool the agent does not have
+  fails the call. `stop_reason: "max_tokens"` tests a truncated reply;
+  `stream: n` sends a streamed step's reply in pieces of n characters. The
+  stub keeps the messages each request sent (`calls`). Only the agent's own
+  requests are answered, never a tool's own provider call. The chat is still
+  built, so configure the provider (any key). An agent with no stub goes over
+  the wire; `Smith::Testing.unstubbed = :refuse` fails its call with
+  `Smith::Testing::UnstubbedCall` instead, so a mistyped stub never reaches a
+  paid provider.
+- `run_until_waiting(run)` runs until the run waits or ends and returns its
+  wait (a durable retry's backoff is ended at once unless `retries: false`);
+  `resume_with(run, payload:)` delivers the wait's event and runs on;
+  `expire(run)` ends the wait as if its deadline had passed and runs on.
+- A batch step's request is captured as ever (the batch queue gets it);
+  `resume_with(run, payload: Smith::Testing.batch_payload(run, Smith::Testing.batch_result("text")))`
+  answers it, and a fan-out's `batch_payload` takes results by branch key.
+- Minitest: `assert_waiting_for(run, :event)` (or `kind: :batch`),
+  `assert_transitioned_through(run, *transitions)` (in order; the steps the
+  instance completed, not a failed or timed-out one),
+  `assert_usage_recorded(run, count:, **usage_entry_attributes)`. RSpec:
+  `be_waiting_for`, `have_transitioned_through`, `have_recorded_usage`.
+- Stubs belong to the process; both helpers drop them after each test. An
+  in-process `retry_on` backoff still sleeps: stub `sleep` on the workflow,
+  or declare the retry `durable: true`.
+
+## Doctor
+
+After adding Smith, verify the integration:
+
+```bash
+# Plain Ruby
+smith doctor              # offline checks
+smith doctor --live       # live provider call
+smith doctor --durability # persistence round-trip
+smith install             # scaffold config/smith.rb
+
+# Rails
+bin/rails smith:doctor
+bin/rails smith:doctor:live
+bin/rails smith:doctor:durability
+bin/rails generate smith:install
+```
+
+Doctor verifies: Smith loads, RubyLLM loads, minimal workflow boots, configuration is non-empty, serialization round-trips, persistence adapter works, and (with `--live`) a real provider call succeeds.
+An `:active_record` persistence adapter's model is checked on every run: its
+`lock_version` (Rails' locking column), a text payload, and a string key the
+database keeps unique, at least 64 characters wide, read from schema
+metadata (skipped when the database cannot be reached; an encrypted payload
+is a warning). The durability probe writes as runs write (`store_versioned`
+where the adapter has it, and `persistence_max_bytes`), so it meets what a
+run would.

data/docs/TOOLS_AND_GUARDRAILS.md

@@ -387,7 +387,8 @@ receive an inspection-only argument Hash. The full admitted argument graph is
 Smith-owned and frozen, so policy inspection cannot rewrite the values that
 reach `perform`. Errors raised from inside `perform`, including
 `Smith::DeadlineExceeded`, do not prove that external work was absent and retain
-their original type. Process-fatal exceptions are never converted into receipt
+their original type. Process-fatal exceptions, and errors the host names in
+`Smith.config.interrupting_errors`, are never converted into receipt
 outcomes or terminal callbacks: an admitted receipt remains unresolved and the
 durable host must reconcile it before any resume decision. Smith never
 automatically replays that uncertain work. Failure-handler errors propagate
@@ -450,3 +451,60 @@ class GuardedWorkflow < Smith::Workflow
   end
 end
 ```
+
+## Shipped Tools and Tool Budgets
+
+Smith ships `Tools::WebSearch`, `Tools::UrlFetcher`, and `Tools::Think`. Tools declare provider compatibility via `compatible_with`; Smith's normalizer routes or drops them per-attempt.
+
+`Tools::WebSearch` and `Tools::UrlFetcher` are stubs: they declare each
+tool's parameters and description, and their `perform` raises
+`NotImplementedError`. A host subclasses them with its own search or fetch
+behind `perform`. `Tools::Think` works as shipped.
+
+```ruby
+class WebSearch < Smith::Tools::WebSearch
+  def perform(query:, max_results: 5) = SearchClient.search(query, limit: max_results)
+end
+
+class UrlFetcher < Smith::Tools::UrlFetcher
+  def perform(url:) = PageClient.fetch(url)
+end
+
+class SearchAgent < Smith::Agent
+  register_as :search_agent
+  model "claude-opus-4-7"
+  tools WebSearch, UrlFetcher # the host's subclasses
+  budget tool_calls: 3
+  tool_budget_exhaustion :complete
+end
+```
+
+Agent tool-call budgets default to raising `Smith::BudgetExceeded` when a tool
+tries to execute beyond the configured limit. Agents that declare
+`tool_budget_exhaustion :complete` must also declare a finite `tool_calls`
+budget. Smith then consumes each model-requested call before dispatch, executes
+only a complete batch admitted against both agent and workflow limits, and
+performs one tool-disabled completion when the allowance is exhausted or the
+provider returns an oversized batch. The bounded provider loop is iterative,
+not recursive. Calls to unavailable tools consume allowance as well, so a
+provider cannot create an unbounded correction loop. Workflow reservations are
+reconciled to actual Smith tool execution.
+
+Hosts may install `Smith::Tool::CallBudget` with exact per-tool limits through
+`Smith::Tool.with_call_budget`. Smith propagates that same root allowance into
+parallel and heterogeneous fan-out worker threads; agent budgets narrow the
+root and all branches share its atomic counters.
+
+Graceful exhaustion is an in-process protocol guarantee, not durable recovery
+for the model/tool loop. If the process exits after a tool begins and before the
+agent completes, the host must treat the outcome as uncertain unless it owns a
+durable per-invocation receipt and reconciliation contract. Smith also refuses
+to start a fresh fallback-model chat after tool execution has begun because that
+would discard the captured evidence and conversation protocol.
+
+Within a host invocation scope, `Smith::Tool.current_invocation` exposes the
+normalized tool-call ID and deterministic execution/batch ordinals. The ID is
+RubyLLM correlation metadata and must not be treated as a provider idempotency
+key. Durable hosts can pass an explicitly seeded
+`Smith::Tool::InvocationSequence` to `with_invocation_context`; Smith itself does
+not persist or reconcile operation receipts.

data/lib/generators/smith/install/install_generator.rb

@@ -13,13 +13,17 @@ module Smith
     # The most a PostgreSQL text column holds (1 GiB less a byte), the
     # payload's limit when no bound is set.
     UNBOUNDED_PAYLOAD = 1_073_741_823
+    # The default bound: well above what one step can add to a state (a
+    # map's collected results, 4 MiB; a wait's payload or a paused tool
+    # loop, 1 MiB each), so it refuses runaway growth, not a run Smith built.
+    MAX_STATE_BYTES = 16 * 1024 * 1024
 
     source_root File.expand_path("templates", __dir__)
     desc "Writes Smith's initializer, and a table and model for workflow state"
 
     class_option :state_model, type: :string, default: "WorkflowState",
                                desc: "The model that keeps workflow state"
-    class_option :max_state_bytes, type: :numeric, default: 1_048_576,
+    class_option :max_state_bytes, type: :numeric, default: MAX_STATE_BYTES,
                                    desc: "persistence_max_bytes, and the payload CHECK above it (0: neither)"
     class_option :skip_state_model, type: :boolean, default: false,
                                     desc: "Write only the initializer"
@@ -77,7 +81,7 @@ module Smith
     # ignores it, as SQLite does.
     def payload_limit = [check_bytes || UNBOUNDED_PAYLOAD, UNBOUNDED_PAYLOAD].min
 
-    # An Integer as a Ruby literal: 1_048_576.
+    # An Integer as a Ruby literal: 16_777_216.
     def literal(number) = number.to_s.gsub(/\B(?=(\d{3})+(?!\d))/, "_")
   end
 end

data/lib/smith/agent/completion.rb

@@ -15,8 +15,9 @@ module Smith
       attribute :response_facts, Types::Hash.default({}.freeze)
 
       # `stream` is set for a streamed call: the facts of each of its
-      # requests, read from the stream, and the request's output limit
-      # (Providers::StreamFacts). Each response's usage carries its facts and
+      # requests, read from the stream, the request's output limit, and, for
+      # a call that failed, how many of its replies a response memo kept
+      # ahead of them (Providers::StreamFacts). Each response's usage carries its facts and
       # `mode: :stream`. `mode` marks another call's usage (:batch).
       def self.from_messages(response:, messages:, stream: nil, mode: nil)
         assistant_messages = Array(messages).select do |message|
@@ -56,7 +57,8 @@ module Smith
       def self.response_facts(messages, stream)
         return messages.map { Providers::ResponseFacts.from_message(_1) } unless stream
 
-        Providers::StreamFacts.per_message(messages, stream.fetch(:rounds), stream.fetch(:output_limit))
+        Providers::StreamFacts.per_message(messages, stream.fetch(:rounds), stream.fetch(:output_limit),
+                                           kept: stream.fetch(:kept, 0))
       end
 
       def self.build_provider_usages(messages, streamed_facts, mode)