smith-agents 0.10.0 → 0.12.0

data/README.md

@@ -17,13 +17,33 @@ environment before calling the slice complete.
 
 ```ruby
 # Gemfile
-gem "smith-agents", "0.5.0", require: "smith"
+gem "smith-agents", "0.12.0", require: "smith"
 ```
 
 ```bash
 bundle install
+bin/rails generate smith:install   # in a Rails app
+bin/rails db:migrate
 ```
 
+`smith:install` writes `config/initializers/smith.rb` and a table and model
+for workflow state (`WorkflowState`, Smith's `:active_record` persistence
+adapter): a unique string key (child runs are stored under 64-character
+keys of their own, so the key is not a host row's id), the state as JSON
+text, the `lock_version` Smith's versioned writes compare, and a CHECK on
+the payload's size. The initializer sets `persistence_max_bytes` (1 MiB): a
+step whose result would take a stored run past it fails with
+`Smith::PersistedStateTooLarge` before the result is kept, as with any
+error (its usage recorded, its `on_failure` taken), so the run stays
+storable; any other state past it is refused before anything is written.
+The answer that did not fit is not kept, so set the bound above the largest
+output a step keeps, or keep large outputs in the artifact store. The CHECK
+sits a third above the bound, room for a column the host encrypts (Rails
+`encrypts` holds runs, but not split-step exact writes; the doctor warns).
+Options: `--state-model=NAME` (a namespaced model gets its table name set),
+`--max-state-bytes=N` (`0` for no bound and no CHECK), and
+`--skip-state-model` (the initializer only).
+
 The Ruby module namespace stays `Smith::`; only the gem name is namespaced because `smith` on RubyGems is taken. The `require: "smith"` in the Gemfile tells bundler to load the actual file name.
 
 ## Quickstart
@@ -72,7 +92,7 @@ result.steps    # => [{ transition: :reply, from: :idle, to: :done, output: ...
 |---|---|
 | `Smith::Agent` | A `RubyLLM` agent plus model, instructions, output schema, tools, budget, and fallback models. Identifies itself to the workflow via `register_as :name`. |
 | `Smith::Workflow` | A state machine of named transitions. Each transition calls an agent, runs deterministic code, routes, or composes a nested workflow. |
-| `Smith::Context` | Declares which workflow context keys persist across restore, and how those keys become agent-visible input via `inject_state`. |
+| `Smith::Context` | Declares which workflow context keys persist across restore, how those keys become agent-visible input via `inject_state`, and which keys hold untrusted outside content (`untrusted`). |
 | `Smith::Tool` | A `RubyLLM` tool plus provider-compatibility metadata and guardrail hooks. |
 | Persistence adapters | Host-owned storage. Smith ships `Memory`, `RedisStore`, `CacheStore`, `RailsCache`, `ActiveRecordStore`. |
 | Trace adapters | Host-owned observability. Smith ships `Memory`, `Logger`, `OpenTelemetry`. |
@@ -100,6 +120,7 @@ end
 | Evaluator-Optimizer | `optimize generator:, evaluator:, ...` | Generate-then-critique refinement loops. |
 | Orchestrator-Worker | `orchestrate orchestrator:, worker:, ...` | Dynamic task fan-out with delegation rounds. |
 | Deterministic | `compute { |step| ... }` | Pure Ruby step inside the state machine. |
+| Wait | `await :event, into:, timeout:` / `delay seconds` | Durable wait for an event the host delivers, or until a time. |
 
 The full pattern guide with working examples for each lives in [`docs/PATTERNS.md`](docs/PATTERNS.md).
 
@@ -116,7 +137,272 @@ For bounded dynamic delegation, use the separate Orchestrator-Worker pattern.
 | Evaluator-Optimizer | Executable | `optimize`, bounded by `max_rounds` plus structured evaluator output. |
 | Deterministic repair | Not first-class yet | Can be handwritten with `compute` / `run` only when the workflow author owns the exact guard, repair, revalidation, and exit policy. Deterministic steps may declare inspectable `routes: [...]`, but that is not a native repair-loop contract. |
 | Guarded state re-entry | Not first-class yet | `compute` / `run` can declare and route to named transitions with `routes: [...]`, but Smith does not yet own persisted entry counts, mutation policy, or safe re-entry contracts. |
-| Polling / wait | Host-owned | Use the host app's queue/timer plus Smith persistence helpers. Smith must not model durable polling with busy-waits or sleep loops. |
+| Wait for an event or a time | Executable | `await` and `delay`, bounded by a required timeout and timeout route. Smith records the wait and its deadline; delivering the event (`resume!`), scheduling the expiry (`expire_wait!`) and waking the run stay host-owned. Smith never polls for an event or sleeps until a deadline. |
+
+### Isolation
+
+Each agent step can get its own messages and write its output to the context
+instead of a shared transcript:
+
+```ruby
+class ReviewContext < Smith::Context
+  persist :source, :fields
+  untrusted :source, :fields   # outside text, and what an agent drew from it
+end
+
+class DocumentReview < Smith::Workflow
+  context_manager ReviewContext
+  session :isolated   # no shared transcript; every agent step declares input:
+  initial_state :fetched
+  state :extracted
+
+  transition :extract, from: :fetched, to: :extracted do
+    execute :field_extractor,
+            input: ->(ctx) { [{ role: :user, content: Smith::Untrusted.wrap(ctx[:source], source: "document") }] },
+            output_to: :fields
+  end
+end
+```
+
+Context keys declared `untrusted` never reach `inject_state`, the
+`seed_messages` builder, or an agent's `inputs`, in the workflow or any
+workflow it nests. See [`docs/PERSISTENCE.md`](docs/PERSISTENCE.md), Step
+Isolation and Untrusted Content.
+
+A message in `input:` can carry content parts: text, and documents and images
+stored as artifacts of the run (a child run reads its ancestors' too), with
+`cache: true` marking a prefix Anthropic may cache; seed messages take text
+parts:
+
+```ruby
+transition :store, from: :fetched, to: :stored do
+  compute { |step| step.write_context(:contract_ref, Smith.artifacts.store(Contracts.pdf(step.read_context(:id)))) }
+end
+
+transition :extract, from: :stored, to: :extracted do
+  execute :field_extractor,
+          input: lambda { |ctx|
+            [Smith::Message.user([Smith::Content.document(ctx[:contract_ref], media_type: "application/pdf"),
+                                  Smith::Content.text("List the parties and the term.")])]
+          },
+          output_to: :fields
+end
+```
+
+Anthropic gets native blocks; other providers get the parts in order as text,
+with the files attached where they stand. Documents are read once, checked
+against their media type and bounds, and only from runs allowed to read them.
+See [`docs/PERSISTENCE.md`](docs/PERSISTENCE.md), Content Parts.
+
+### Waiting
+
+A step can wait, durably and bounded, for an event the host delivers or for a
+time to pass:
+
+```ruby
+class DocumentReview < Smith::Workflow
+  initial_state :requested
+
+  transition :fetch_source, from: :requested, to: :fetched do
+    await :source_ready, into: :source, timeout: 6 * 3600 do |step|
+      SourceClient.request(ref: step.read_context(:source_ref), idempotency_key: step.await_token)
+    end
+    on_timeout :source_unavailable
+  end
+  transition :source_unavailable, from: :requested, to: :unavailable
+
+  transition(:cool_off, from: :fetched, to: :reviewed) { delay 600 }
+end
+
+workflow.run_persisted!(key)       # stops when the run waits
+workflow.waiting?                  # => true
+workflow.awaiting                  # => Smith::Workflow::Wait (event, token, deadline_at, ...)
+
+DocumentReview.resume!(key, event: :source_ready, token:, payload: reply)  # => :resumed, :failed, :missing or :stale
+DocumentReview.expire_wait!(key, token:, now: Time.now.utc)                # => :expired, :not_due, :missing or :stale
+```
+
+The host indexes `awaiting` to route events and to schedule the expiry call.
+See [`docs/PERSISTENCE.md`](docs/PERSISTENCE.md), Waiting.
+
+### Child runs
+
+A step can start other workflows as runs of their own, which the host launches,
+persists and runs, and wait for their results:
+
+```ruby
+Smith.config.run_launcher = MyLauncher.new   # launch(children:, parent_key:, await_token:)
+
+class DocumentReview < Smith::Workflow
+  initial_state :split
+  state :failed
+
+  transition :review_sections, from: :split, to: :reviewed do
+    map :section_ids, workflow: SectionReview, max_items: 500, timeout: 6 * 3600,
+        input: ->(section_id, ctx) { { document_id: ctx[:document_id], section_id: } },
+        into: :section_reviews, collect: :outputs, on_item_failure: :collect
+    on_failure :fail
+  end
+
+  transition :summarize, from: :reviewed, to: :summarized do
+    call Summarize, input: ->(ctx) { { document_id: ctx[:document_id] } }, timeout: 3600, into: :summary
+    on_failure :fail
+  end
+
+  transition :notify, from: :summarized, to: :done do
+    spawn :watcher_ids, workflow: NotifyWatcher, max_items: 1_000, input: ->(id, _ctx) { { watcher_id: id } }
+  end
+end
+```
+
+`map` launches a run per item (more than `max_items` fails the step before
+anything launches; the launcher is idempotent on each child's `key`) and waits,
+bounded by `timeout:`, for the host to resume the parent with
+`resume!(key, event: :children, token:, payload:)`, one `Smith::ChildRun.result`
+entry per child. `call` is a map of one whose `into:`
+gets the child's output; `spawn` launches and continues. A child starts with
+only what `input:` returns, and keeps any of its parent's untrusted keys passed
+under the same name. With `on_item_failure: :fail` (the default) a failed child
+fails the step with `Smith::ChildRunsFailed`, so the step declares `on_failure`;
+these steps take no `retry_on`. `Smith::Launchers::Inline` runs the children in
+process, for tests and scripts. See
+[`docs/PERSISTENCE.md`](docs/PERSISTENCE.md), Child Runs.
+
+### Streaming
+
+An agent step can send its reply's text to the host as the model writes it:
+
+```ruby
+# Called when a streamed step starts; returns the step's sink, or nil to
+# stream nothing (nobody is watching). Each step execution gets its own.
+Smith.config.stream_sinks = {
+  answer: ->(info) { AnswerStream.new(info[:execution_key]) if Screens.watching?(info[:execution_key]) }
+}
+
+class AnswerStream
+  def initialize(key)
+    @key = key
+    @reply = Smith::Streaming::JsonField.new(:reply)  # the output schema's reply field, as it grows
+  end
+
+  def delta(text) = (piece = @reply.push(text)) && Screens.append(@key, piece)
+  def reset(_reason) = @reply.reset || Screens.clear(@key)
+  def finish(output) = Screens.final(@key, output["reply"])
+  def fail(_error_class) = Screens.failed(@key)
+end
+
+class Answer < Smith::Workflow
+  initial_state :asked
+  state :failed
+
+  transition :answer, from: :asked, to: :answered do
+    execute :answerer, stream: :answer, output_to: :answer
+    on_failure :fail
+  end
+end
+```
+
+A sink defines any of `start(info)` (each provider call: the step's info with
+the attempt, model and provider), `delta(text)` (each text chunk; tool calls,
+thinking and `<think>` text are not sent), `reset(reason)` (Smith discarded the
+text sent so far: `:retry`, `:fallback`, `:truncation` or `:repair`; and
+`:retry_wait`, the last word, while a durable retry waits), `finish(output)`
+(the output the step accepted, after guardrails, the output contract and
+`output_to:`, once the step is stored) and `fail(error_class)`. Streamed text is
+a draft: guardrails and output checks run on the whole reply, so nothing should
+be saved or acted on before `finish`, and a step with an output schema streams
+JSON text (`Smith::Streaming::JsonField` gives the text each delta adds to one
+string field). A sink that raises is logged and the step goes on; a sink name
+that is not registered fails the step before its call. Stop reasons are read
+from the stream, so cut-off and refused replies are handled as without it, and a
+stream that ends without one counts as cut off (`stream_incomplete`). Smith's
+OpenAI Responses path does not stream: its sink gets each reply as one delta.
+Streaming changes nothing durable (each usage entry keeps its response's stop
+reason and message id, with `mode: :stream`), and a replayed step streams again
+(unless a response memo kept its reply: then its sink gets the text at once).
+Graph inspection shows the stream, and runtime readiness reports a missing sink
+(`:stream_sink_missing`) and an agent whose calls go to the Responses API
+(`:stream_whole_replies`).
+
+### Batch mode
+
+An agent step, or a fan-out, can send its requests through a provider's batch
+API (half price, answered within 24 hours) instead of over the wire:
+
+```ruby
+# The host keeps each request until a scheduled job submits it (a table, in
+# production); enqueue is idempotent on custom_id.
+class HostBatchQueue
+  Entry = Struct.new(:custom_id, :await_token, :request, :run_key, :batch_id)
+
+  def initialize = @entries = {}
+  def enqueue(custom_id:, await_token:, request:, execution_key:)
+    @entries[custom_id] ||= Entry.new(custom_id, await_token, request, execution_key)
+  end
+  def unsubmitted = @entries.values.reject(&:batch_id)
+  def entry(custom_id) = @entries.fetch(custom_id)
+end
+Smith.config.batch_queue = HostBatchQueue.new
+
+class Triage < Smith::Workflow
+  initial_state :queued
+  state :failed
+
+  transition :triage, from: :queued, to: :triaged do
+    execute :triager, mode: ->(ctx) { ctx[:urgent] ? :sync : :batch }, output_to: :verdict
+    on_failure :fail
+  end
+end
+
+# One scheduled job submits what is queued; another collects an ended batch
+# and resumes each run once with every result of its wait.
+queue = Smith.config.batch_queue
+batches = Smith::Providers::Anthropic::Batches.new
+entries = queue.unsubmitted
+batch_id = batches.submit(entries.map { { custom_id: _1.custom_id, request: _1.request } })
+entries.each { _1.batch_id = batch_id }
+if batches.status(batch_id)[:status] == "ended"
+  payloads = Hash.new { |hash, wait| hash[wait] = {} }
+  batches.results(batch_id) do |custom_id, result|
+    entry = queue.entry(custom_id)
+    payloads[[entry.run_key, entry.await_token]][custom_id] = result
+  end
+  payloads.each { |(run_key, token), payload| Triage.resume!(run_key, event: :batch_result, token:, payload:) }
+end
+```
+
+`mode:` is `:sync` (the default), `:batch`, or a callable of the context and
+the step attempt returning one, on `execute` and `fan_out`; a batch step
+declares `on_failure`, and cannot stream or be `parallel:`. Smith prepares
+the call as a synchronous one, admits it against the budget, and hands the
+exact request (the provider payload and the call's own headers, never the
+connection's credentials) to the queue under a `custom_id` (the wait's token;
+each fan-out branch its own) once the step's attempt has ended without error,
+then the run waits on `:batch_result`. Smith never submits or polls, and waits
+without sleeping: `Smith::Providers::Anthropic::Batches` gives the host
+`submit`, `status`, `results` and `cancel`. `resume!` with
+`{custom_id => result}` runs the step again, each call taking its result as
+the reply it would have got. A call that renders another request by then (a
+date in its prompt, data read afresh, a prompt changed in a deploy) does what
+its step's `on_request_change:` says: `:keep` (the default) takes the result,
+which answers the request as sent, and its `:provider_call` trace says
+`request_changed: true`; `:resend` sends the request as rendered now over the
+wire instead; `:fail` fails the step with `Smith::BatchRequestChanged` (an
+`AgentError`, so `retry_on` can send it again, synchronously if the mode
+callable says so). A result the call does not use is charged all the same.
+Then the output checks, a follow-up call (over the
+wire), guardrails and `output_to:` as ever, usage recorded with `mode: :batch`
+and priced with the catalog's `batch_discount`, and charged even past a
+budget, as the money is spent. A request the provider refused fails the step
+(`Smith::BatchRequestInvalid`); a failed, canceled or expired result, or none
+by the wait's deadline (`expire_wait!`, 25 hours), raises
+`Smith::BatchResultFailed`, which `retry_on` sends again (synchronously, if the
+mode callable says so at the next attempt; an in-process `retry_on` backoff
+sleeps as on any step, while `durable: true` waits). Only Anthropic's batch interface is
+built in, and a batch request is one turn, so a batching agent has no tools;
+runtime readiness reports a missing queue (`:batch_queue_missing`) and an
+agent that cannot batch (`:batch_unavailable`), and graph inspection shows the
+mode.
 
 ## Workflow Graph Inspection
 
@@ -167,12 +453,14 @@ require "smith"
 Smith.configure do |config|
   config.logger = Logger.new($stdout)
   config.trace_adapter = Smith::Trace::Memory.new
-  config.artifact_store = Smith::Artifacts::Memory.new
+  config.artifact_store = :active_record                  # or :s3 (S3, R2), :file, :memory
+  config.artifact_options = { model: "SmithArtifact" }
 
   # Persistence
   config.persistence_adapter = :rails_cache
   config.persistence_options = { namespace: "smith" }
   config.persistence_ttl = 1.day.to_i
+  config.persistence_max_bytes = 1_048_576 # a stored state's bound (nil: none); see Installation
   config.persistence_retry_policy = { attempts: 3, base_delay: 0.1, max_delay: 1.0 }
 
   # Generic runtime resource bounds
@@ -184,13 +472,18 @@ Smith.configure do |config|
   # OpenAI /v1/responses routing for gpt-5 + tools + thinking. :auto (default) or :off.
   config.openai_api_mode = :auto
 
-  config.pricing = {
-    "gpt-4.1-nano" => { input_cost_per_token: 1.0e-7, output_cost_per_token: 4.0e-7 }
-  }
+  config.pricing = Smith::Pricing.validate_catalog!(
+    %w[openai gpt-4.1-nano] => {
+      input_cost_per_token: 1.0e-7, output_cost_per_token: 4.0e-7,
+      cache_read_cost_per_token: 2.5e-8, # optional: prompt-cache reads (0 when absent)
+      batch_discount: 0.5                # optional: off the cost of batch-mode calls (0 when absent)
+    }
+  )
 end
 ```
 
-All settings are optional for a first run. See [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md) for the full reference.
+All settings are optional for a first run, except that storing artifacts or
+attaching documents needs `artifact_store` (outside `test_mode`). See [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md) for the full reference.
 
 ## Persistence and Resume
 
@@ -217,10 +510,12 @@ message batch, appends it under Smith's workflow lifecycle lock, and returns an
 immutable digest witness. The host still owns session identity, transaction
 coordination, persistence, idempotency, and resume policy.
 
-Built-in adapters (all support TTL where the backend allows; `Redis`,
-`ActiveRecord`, and `Memory` support optimistic locking via `store_versioned`
-and exact dispatch claims via `replace_exact`; `ActiveRecord` participates in
-the caller's transaction for host-record coordination;
+Built-in adapters (all except `ActiveRecord` support TTL; `ActiveRecord` has no
+expiry column, so its writes refuse a non-nil `persistence_ttl`, while reads and
+deletes keep working and `smith doctor --durability` flags the combination;
+`Redis`, `ActiveRecord`, and `Memory` support optimistic locking via
+`store_versioned` and exact dispatch claims via `replace_exact`; `ActiveRecord`
+participates in the caller's transaction for host-record coordination;
 `Redis` and `Memory` also support heartbeat probes via `record_heartbeat` /
 `last_heartbeat`):
 
@@ -361,11 +656,24 @@ cache invalidation.
 
 Smith ships `Tools::WebSearch`, `Tools::UrlFetcher`, and `Tools::Think`. Tools declare provider compatibility via `compatible_with`; Smith's normalizer routes or drops them per-attempt.
 
+`Tools::WebSearch` and `Tools::UrlFetcher` are stubs: they declare each
+tool's parameters and description, and their `perform` raises
+`NotImplementedError`. A host subclasses them with its own search or fetch
+behind `perform`. `Tools::Think` works as shipped.
+
 ```ruby
+class WebSearch < Smith::Tools::WebSearch
+  def perform(query:, max_results: 5) = SearchClient.search(query, limit: max_results)
+end
+
+class UrlFetcher < Smith::Tools::UrlFetcher
+  def perform(url:) = PageClient.fetch(url)
+end
+
 class SearchAgent < Smith::Agent
   register_as :search_agent
   model "claude-opus-4-7"
-  tools Smith::Tools::WebSearch, Smith::Tools::UrlFetcher
+  tools WebSearch, UrlFetcher # the host's subclasses
   budget tool_calls: 3
   tool_budget_exhaustion :complete
 end
@@ -407,11 +715,25 @@ Guardrails run as input/output gates around agent calls. See [`docs/TOOLS_AND_GU
 
 ```ruby
 class BudgetedWorkflow < Smith::Workflow
-  budget total_tokens: 10_000, total_cost: 0.50, wall_clock_ms: 30_000
+  budget total_tokens: 10_000, total_cost: 0.50, wall_clock: 30
 end
 ```
 
-Budgets reserve serially at each step and reconcile after the agent call. Parallel branches reserve scoped envelopes that release back to the parent ledger. The `Workflow::RunResult` carries `total_tokens`, `total_cost`, and per-call `usage_entries`.
+`wall_clock` is in seconds. A workflow's `wall_clock` bounds the time its steps
+spend running, counted across restores in the persisted `active_seconds`, not
+the time since the run was created: time spent waiting, or between steps a host
+advances from a queue, does not count. A step that raised or whose process died
+is charged, on the next restore, the time from its start, at most what is left
+of the budget, so a run that keeps failing stays bounded across retries.
+Durable composite branches each start from what is left; their running time is
+not added back. An agent's `wall_clock` bounds each of
+its calls. A workflow budget accepts `total_tokens`,
+`token_limit`, `total_cost`, `tool_calls`, and `wall_clock`; an agent budget
+accepts `token_limit`, `cost`, `wall_clock`, `tool_calls`, `total_tokens`, and
+`total_cost`. Any other key raises `ArgumentError` naming the accepted keys. A
+restored run drops consumed amounts for keys the workflow no longer declares.
+
+Budgets reserve serially at each step and reconcile after the agent call. Parallel branches reserve scoped envelopes that release back to the parent ledger. The `Workflow::RunResult` carries `total_tokens`, `total_cost`, and per-call `usage_entries`. Token budgets and totals count prompt-cache reads and writes beside uncached input and output, and settle every billed response, including the completed rounds of a tool loop whose attempt then failed.
 
 Direct ledger integrations settle the exact receipt returned by reservation:
 
@@ -422,7 +744,9 @@ ledger.reconcile_many!(reservation, actual: { total_tokens: 640, total_cost: 0.0
 
 `release!` / `release_many!` accept the same receipt when work never starts.
 Receipts are ledger-owned and one-shot: another ledger, a replay, or an
-amount-only settlement is rejected.
+amount-only settlement is rejected. `ledger.remaining(key)` never reads back
+above the exact remaining amount, and `ledger.remaining_share(key, parts)`
+returns one of `parts` equal shares of it such that reserving all `parts` fits.
 
 Budget amounts accept finite, non-negative `Integer` and `Float` values. Smith
 uses an exact `BigDecimal` representation derived from each value's canonical
@@ -437,6 +761,91 @@ threads with `Thread#raise` or `Thread#kill`. Arbitrary asynchronous thread
 termination can make any in-process return value outcome-ambiguous; hosts must
 use cancellation signals, bounded I/O, and process isolation for hard stops.
 
+### Budget ledgers
+
+A run's `budget` bounds that run. A budget shared by many runs (an account's
+monthly spend, say) lives in the host, and a workflow names it with
+`budget_ledger`:
+
+```ruby
+class MonthlyLedger
+  def initialize(account) = @account = account
+
+  # Admit the hold (true; anything else refuses it) and keep it under key
+  # until expires_at, after which it counts for nothing.
+  def reserve!(amount, key:, expires_at:) = ...
+  def reconcile!(key, amount) = ...   # what the request spent, in place of its hold
+  def release!(key) = ...             # it spent nothing
+end
+
+class DocumentReview < Smith::Workflow
+  budget_ledger(:account_id) { |scope| MonthlyLedger.new(scope[:account_id]) }
+end
+```
+
+The block gets the run's scope: the values the named keys have in its trusted
+context when the run is created (JSON scalars, kept with the run), so nothing a
+step or a model writes later chooses another ledger, and an untrusted key is
+refused. Smith builds the ledger at the first request a step makes, never
+persists it, and never needs it to restore a run; nil means none. A nested
+workflow without a `budget_ledger` of its own charges the enclosing run's; a
+child run (`call`, `map`, `spawn`), inline or not, uses its own class's, so
+give each child workflow its own (runtime readiness warns of one that has none,
+`:child_run_unbudgeted`) and pass its scope keys in each child's input.
+
+Before a call's first request over the wire (each fallback model, each
+follow-up, streamed or not, OpenAI's Responses route included) Smith asks the
+ledger to admit its hold. If the ledger refuses, the step fails with
+`Smith::BudgetUnavailable` (a `BudgetExceeded`, not retried by default) before
+anything is sent; `retry_on Smith::BudgetUnavailable, durable: true, ...` waits
+for budget instead (the host ends the wait with `expire_wait!`, and ends it
+early, once there is budget, with a `now:` at its `deadline_at`). After the request, Smith reconciles what it
+spent: its usage, or its hold when it answered without usage or its stream
+failed part way; a request that failed before anything came back releases its
+hold. If building the ledger or `reserve!` raises, the step fails with
+`Smith::BudgetLedgerError` (nothing was sent, no fallback model is tried); one
+from `reconcile!` or `release!` is logged by class and the step goes on, as
+the money is spent.
+
+A tool loop's later rounds continue a call whose tools already ran, so
+stopping one would leave the tools' effects without their answer. A step whose
+`retry_on` names `BudgetUnavailable` holds them too: when the ledger refuses
+one, the loop pauses, keeping its tool calls and their results (at most 1 MiB)
+with the step's retry, and the retry continues the loop from them, running
+none of its tools again. Any other step's later rounds, or a loop too large to
+keep, go on unheld and are reconciled with what they spent, so give tool-using
+agents a `tool_calls` budget.
+
+A hold (`Smith::Budget::Amount`: `input_tokens`, `output_tokens`, `cost`, and
+`tokens`, their sum) is the request's output cap (its `max_tokens`, or its
+provider's name for it, else the model's) and an estimate of its input: text
+at one token per 3 bytes, an image at 1,600 tokens, a PDF at 2,000 tokens a
+page, 8 tokens a message for its framing, capped at the model's context window.
+Images and documents count so as RubyLLM attachments and in raw content blocks
+(base64 data or a data URL). `Smith.config.budget_hold_output_tokens`, when
+set, caps the output part (an agent with no `max_tokens` is otherwise held for
+the model's maximum, which can be tens of thousands of tokens, and a longer
+reply is reconciled at what it cost). It is priced from the pricing catalog
+under the id RubyLLM resolves the model to; `cost` is nil for an unpriced
+model, which runtime readiness warns of (`:budget_ledger_unpriced`, for every
+agent that charges the ledger), and its readiness metrics name the scope keys.
+
+A key is the provider attempt's `attempt_id` (as on its `:provider_call` trace)
+and the request's round, `"<attempt_id>:<n>"`, and for a batch request its
+`custom_id`. A batch request is held when its step captures it, at the batch
+discount, and settled when its result comes: a succeeded result, used or not,
+is reconciled at what it cost; one that failed, or never came, is released.
+A process that dies during a request never settles its hold, so every hold
+carries `expires_at`: `Smith.config.budget_hold_ttl` from now (an hour by
+default; it must outlast the longest request), and for a batch request its
+25-hour wait longer. Smith runs no timer; the ledger leaves holds past
+`expires_at` out of what it admits, and still records a reconcile that comes
+later (a batch delivered late). A key may come again after a crash (a batch
+request captured again, a delivery that ran twice): a reserve of it replaces
+its hold, a release ends it, and its reconcile is final, recorded once.
+Fan-out branches call the ledger from their own threads. RubyLLM's own retries
+(`max_retries`) resend a request under its one hold.
+
 ## Doctor
 
 After adding Smith, verify the integration:
@@ -456,6 +865,73 @@ bin/rails generate smith:install
 ```
 
 Doctor verifies: Smith loads, RubyLLM loads, minimal workflow boots, configuration is non-empty, serialization round-trips, persistence adapter works, and (with `--live`) a real provider call succeeds.
+An `:active_record` persistence adapter's model is checked on every run: its
+`lock_version` (Rails' locking column), a text payload, and a string key the
+database keeps unique, at least 64 characters wide, read from schema
+metadata (skipped when the database cannot be reached; an encrypted payload
+is a warning). The durability probe writes as runs write (`store_versioned`
+where the adapter has it, and `persistence_max_bytes`), so it meets what a
+run would.
+
+## Testing
+
+`require "smith/testing"` runs a host's workflows in its tests without a
+provider. A stubbed agent's requests are answered where they would go over
+the wire, so everything above runs as for a provider's reply: the tool loop,
+budget holds, the response memo, streaming, output checks and repairs, usage
+entries and budgets.
+
+```ruby
+# test/test_helper.rb
+require "smith/testing/minitest"   # or "smith/testing/rspec" for matchers
+
+class NoticeTest < ActiveSupport::TestCase
+  include Smith::Testing::Assertions
+
+  test "drafts with the venue's hours, then waits for approval" do
+    Smith::Testing.stub_agent(Writer, outputs: [Smith::Testing.tool_call(Lookup, venue: "the hall"),
+                                                "The hall opens at 8."],
+                                      usage: { input_tokens: 30, output_tokens: 6 })
+    run = Notice.new(context: {})
+
+    Smith::Testing.run_until_waiting(run)
+    assert_waiting_for run, :approval
+    Smith::Testing.resume_with(run, payload: { approved: true })
+    assert_transitioned_through run, :draft, :approve, :publish
+    assert_usage_recorded run, agent_name: Writer, stop_reason: "end_turn", input_tokens: 30
+  end
+end
+```
+
+- `stub_agent(agent, outputs:, usage:, stop_reason:, stream:)` answers the
+  agent's requests (its class, or its registered name) with `outputs` in
+  turn, the last answering every request after it: a String reply, a Hash as
+  JSON text (for an output schema), or `tool_call(ToolClass, **arguments)`
+  (an Array of them for one round), which runs the agent's own tools before
+  its next request. A tool request naming a tool the agent does not have
+  fails the call. `stop_reason: "max_tokens"` tests a truncated reply;
+  `stream: n` sends a streamed step's reply in pieces of n characters. The
+  stub keeps the messages each request sent (`calls`). Only the agent's own
+  requests are answered, never a tool's own provider call. The chat is still
+  built, so configure the provider (any key). An agent with no stub goes over
+  the wire; `Smith::Testing.unstubbed = :refuse` fails its call with
+  `Smith::Testing::UnstubbedCall` instead, so a mistyped stub never reaches a
+  paid provider.
+- `run_until_waiting(run)` runs until the run waits or ends and returns its
+  wait (a durable retry's backoff is ended at once unless `retries: false`);
+  `resume_with(run, payload:)` delivers the wait's event and runs on;
+  `expire(run)` ends the wait as if its deadline had passed and runs on.
+- A batch step's request is captured as ever (the batch queue gets it);
+  `resume_with(run, payload: Smith::Testing.batch_payload(run, Smith::Testing.batch_result("text")))`
+  answers it, and a fan-out's `batch_payload` takes results by branch key.
+- Minitest: `assert_waiting_for(run, :event)` (or `kind: :batch`),
+  `assert_transitioned_through(run, *transitions)` (in order; the steps the
+  instance completed, not a failed or timed-out one),
+  `assert_usage_recorded(run, count:, **usage_entry_attributes)`. RSpec:
+  `be_waiting_for`, `have_transitioned_through`, `have_recorded_usage`.
+- Stubs belong to the process; both helpers drop them after each test. An
+  in-process `retry_on` backoff still sleeps: stub `sleep` on the workflow,
+  or declare the retry `durable: true`.
 
 ## Capability-aware request shaping
 
@@ -486,11 +962,69 @@ fallback provider from the primary model or from a model id because the same id
 may be registered by more than one provider. Unqualified fallback entries fail
 closed during agent configuration.
 
+## Agent outputs
+
+Smith reads how each response ended from the provider's stop reason. A
+response cut off at its output limit, a refused one, and a structured output
+that is not what the agent declares are not accepted as the step's output:
+
+```ruby
+class Reviewer < Smith::Agent
+  model "claude-opus-5-5", provider: :anthropic
+  output_schema ReviewSchema                         # its root is an object
+  output_contract ReviewContract                     # call(hash) => true, false, or a result with success? and errors
+  output_repairs 1                                   # the default; 0 turns repair off
+  on_truncation retry_with: { max_tokens: 64_000 }   # or :fail (the default), or :accept
+  on_refusal fallback: "openai/gpt-5"                # or :fail (the default)
+end
+```
+
+- A cut-off response raises `Smith::OutputTruncated`, unless the agent
+  accepts it or retries it once on the same model with a larger output limit,
+  mapped to the provider's own parameter (Anthropic, OpenAI, Gemini, Vertex AI
+  and Bedrock; another provider is not retried). There is no retry when the
+  response ran out of context rather than output, or when the new limit is no
+  larger than the request had (the agent's own parameter, or for Anthropic the
+  model's maximum, which RubyLLM already sends); a later repair keeps the
+  larger limit.
+- A refused response (a Gemini prompt block included) raises
+  `Smith::OutputRefused`, unless the agent falls back once to another model.
+- Every other response then goes through the agent's `after_completion`. A
+  structured output (an `output_schema`, or a static RubyLLM `schema`) that is
+  not a JSON object, breaks a schema constraint, or fails the output contract
+  goes back to the same model up to `output_repairs` times: the step's messages
+  as the provider got them, the rejected reply as the assistant's, and the
+  problems as "path: problem" lines. It then raises `Smith::OutputInvalid`,
+  whose `problems` lists at most 20 of them. A blank output still raises
+  `Smith::BlankAgentOutputError`.
+- No follow-up call is made once a tool has run in the step (the tools would
+  run again; the step fails with `Smith::ToolOutcomeUncertain` around the output
+  error) or once the call has spent what its step reserved from a budget. A
+  follow-up that fails raises the output error it was making up for, with the
+  provider's error as its cause.
+- Anthropic's structured outputs reject numeric, string-length and most
+  array-size constraints: Smith sends Anthropic the schema without them, with
+  `additionalProperties: false` on every object, and checks them on the parsed
+  output for every provider. An object that allows other properties or
+  declares none cannot be sent to Anthropic; the step fails before the call
+  with a `Smith::WorkflowError`, and runtime readiness reports
+  `:output_schema_unsupported` for an agent declared for Anthropic.
+- An agent with no model and no fallback models fails its step with a
+  `Smith::WorkflowError`; runtime readiness reports it (`:agent_without_model`).
+
+Every response is billed and settles into budgets, whatever happens to its
+output, and each rejected one leaves an `:output_rejected` trace naming its
+attempt. The output errors share the failure family `output_failure`, persist
+with a failed step, and are never retried by `Smith::Errors.retryable?`, a job
+retry on `Smith::Errors.retryable_classes`, or a transition's
+`retry_on Smith::AgentError`: Smith's own retry, fallback and repairs have run.
+
 ## Errors and retry
 
 ```ruby
 Smith::Errors.retryable?(error)
 # AgentError, DeadlineExceeded => true (always)
+# OutputTruncated, OutputRefused, OutputInvalid (OutputFailure) => false
 # DeterministicStepFailure, ToolGuardrailFailed => honors error.retryable
 # everything else => false
 
@@ -501,6 +1035,11 @@ Smith::Errors.retry_forbidden?(error)
 # => true for terminal tool-evidence failures that must never be replayed
 ```
 
+A nested workflow's step fails with the child's own error when the child
+raises it, so these answers carry through nesting. When a tool ran inside the
+child, the parent step's failure becomes `ToolOutcomeUncertain`, which is never
+retried.
+
 Workflow transitions can also declare a bounded local retry policy:
 
 ```ruby
@@ -510,9 +1049,86 @@ transition :draft, from: :idle, to: :done do
 end
 ```
 
-When no classes are passed, `retry_on` uses `Smith::Errors.retryable?`.
-This is a bounded local transition retry policy. Durable scheduling, long waits,
-and external idempotency guarantees remain host-owned.
+When no classes are passed, `retry_on` uses `Smith::Errors.retryable?`. No
+attempt starts once the workflow's wall-clock deadline has passed.
+This is a bounded local transition retry policy. What a receiver does with an
+idempotency key remains host-owned.
+
+For backoffs too long to sleep through in a worker, `durable: true` makes each
+one a wait of the run instead:
+
+```ruby
+transition :charge, from: :approved, to: :charged do
+  compute { |step| Payments.charge(order, idempotency_key: step.idempotency_key) }
+  retry_on Payments::Timeout, attempts: 5, backoff: 60, max_delay: 3600, durable: true
+  on_failure :charge_failed
+end
+```
+
+A failed attempt the policy retries stores the run waiting (`awaiting.kind ==
+:retry`, due when the backoff ends); the host schedules `expire_wait!` for that
+time and then continues the run, which makes the next attempt under the step's
+keys (new ones only after an attempt declared settled). See
+`docs/PERSISTENCE.md`, Durable Retry.
+
+Steps and tools get idempotency keys for the requests they send outside the
+system, the same on every attempt and after a restore:
+
+```ruby
+transition :charge, from: :approved, to: :charged do
+  compute { |step| Payments.charge(order, idempotency_key: step.idempotency_key) }
+  retry_on Payments::Timeout, Payments::Declined, attempts: 3,
+           new_idempotency_key_on: [Payments::Declined]
+end
+```
+
+A retry keeps the keys, so a receiver that honours them drops a request that
+took effect before a timeout. An error named in `new_idempotency_key_on:` (or
+`step.fail!(..., new_idempotency_key: true)`) shows the attempt's own requests
+took no effect, and the attempts after it use new keys. See
+`docs/PERSISTENCE.md`, Replay Safety.
+
+### Response memo
+
+A step replayed after its process died makes its provider calls again, and
+pays for them again. A host's response memo keeps what they got:
+
+```ruby
+class ResponseMemo
+  def fetch(key) = Rails.cache.read("smith-memo/#{key}")
+  def store(key, record) = Rails.cache.write("smith-memo/#{key}", record, expires_in: 2.days)
+end
+
+Smith.config.response_memo = ResponseMemo.new
+```
+
+Each agent call (its tool loop included) is kept under a key of its position
+(the run, the step, any branch or nested position, the key generation), the
+attempts it runs under (its step's, and each enclosing run's for a nested
+workflow, so a retry never takes the attempt before's answer), its attempt
+index (its model in the fallback chain, or a follow-up's place), its place
+among the attempt's calls, and a digest of its request (model, messages,
+tools, schema, params). A replay reaches the same key; a call whose request
+changed since (a deployed prompt, another model, a date in the prompt) misses
+and is sent again, as a miss always may be.
+
+Its record (`{"version" => 1, "messages" => [...], "complete" => true | false}`,
+JSON) holds the messages the call added: the model's replies (structured ones
+too) with their usage, stop reason and message id (a stream's included), its
+tool calls and their results. It is stored after each round's tool results
+and when the call ends; a complete record ends with the model's reply. A
+replayed call that ended sends nothing: its reply is the step's (a streamed
+step's sink gets the reply's text at once, not the earlier rounds'), its usage
+is recorded as if it had just come, once, and its `:provider_call` trace says
+`memo: true`; a reply that was cut off or refused fails the step as it did.
+One that had not ended continues from its tools' results, its live part going
+on from the kept rounds (on a budget ledger, a later round). Kept tool results
+count as tools the step ran, so no retry or fallback runs them again. A call
+whose messages cannot be kept (an attachment, past 1 MiB) is not stored, and
+is logged; a record Smith does not keep is ignored, and an error from `fetch`
+or `store` is logged by class: a memo never fails a step. Keeping records and
+dropping them (a TTL) are the host's; batch requests are not memoized, as
+their `custom_id` already identifies them.
 
 ## Runtime Complexity
 
@@ -566,5 +1182,5 @@ bundle exec rspec
 bundle exec rubocop
 ```
 
-1,175 examples, MIT licensed. See [`CHANGELOG.md`](CHANGELOG.md) for the current
+2,420 examples, MIT licensed. See [`CHANGELOG.md`](CHANGELOG.md) for the current
 release surface.