smith-agents 0.10.0 → 0.12.0

data/docs/PERSISTENCE.md

@@ -41,9 +41,43 @@ What Smith does for you:
   original roles and order, while bounding only later workflow observations
 - injects a state summary message into that prepared input
 - persists declared workflow context keys
+- keeps one context shape everywhere. Top-level keys must be Symbols: `new`,
+  and the class-level helpers that derive a persistence key from a context
+  (`restore_or_initialize`, `persisted_state_exists?`,
+  `restorable_billing_state?`, `run_persisted!`), raise `Smith::WorkflowError`
+  for any other key, so a `persistence_key` builder sees the same context on
+  every path. Every Hash key inside a value becomes the Symbol of its String
+  form, at every depth, in `new`, at each deterministic write, after hooks
+  that write the live context, and on restore, so a nested read such as
+  `context.dig(:fields, :title)` is the same live and after a JSON round trip.
+  Integer and other keys become Symbols too (`101` becomes `:"101"`, as JSON
+  would make it); Hash subclasses become plain Hashes, without default procs
+  or identity comparison; colliding keys, self-containing values and nesting
+  deeper than 100 levels raise `Smith::WorkflowError`. Values are not
+  converted: only JSON-native values (Strings, numbers, booleans, nil, Hashes,
+  Arrays) read the same after a restore, while Symbols, Times and other
+  objects come back as Strings. `new` holds a normalized copy of the context,
+  not the caller's Hash
 - persists accepted session history
 - preserves chosen next transitions across persistence
 - supports JSON host round-trips through `to_state` and `.from_state`
+- refuses to restore a payload whose state, or whose pending transition, the
+  workflow class does not know (a deploy renamed or removed it, or the payload
+  has no state while the class has an initial state), raising
+  `Smith::UnknownStateOnRestore` or `Smith::UnknownTransitionOnRestore`
+  instead of restoring a run that would read as terminal. A state is known
+  when it is declared with `state` or `initial_state`, or named by any
+  transition's `from:` or `to:`, matched by its String form, so String-named
+  states restore as the Strings the class declares. The check runs after
+  `migrate_from` and after the `definition_digest` check, so a digest
+  mismatch is reported as one and a migration can map an old name: bump
+  `persistence_schema_version` and declare `migrate_from(<previous version>)`,
+  since migrations run only when the stored version is behind. Finished runs
+  are refused too, by `restore`, `restore_or_initialize`,
+  `restorable_billing_state?` and the class-level `run_persisted!`: to read the
+  outcome of a finished run whose terminal state was renamed, migrate it or
+  read the stored JSON directly. `stuck_for?` reports a stored run in an
+  unknown state as stuck, since it can neither advance nor restore
 
 `preserve_seed: true` is intended for workflows whose seed is durable external
 conversation and whose later accepted outputs are internal stage handoffs. Smith
@@ -150,6 +184,8 @@ appends the batch under the workflow lifecycle mutex, and returns an immutable
 `Smith::Workflow::MessageAdmission`. Its `message_digest` is the SHA-256 digest
 of the canonical message batch and can be correlated with host-owned evidence.
 
+A host-admitted message cannot have the system role: only Smith (its injected
+state) and the host's `seed_messages` builder put system messages in a session.
 Admission is bounded to 100 messages, 64 levels, 100,000 visited values, and one
 MiB of canonical JSON. Integers must fit a signed 64-bit value, Floats must be
 finite, Hash keys must be String or Symbol values, and cyclic or unsupported
@@ -534,7 +570,11 @@ before writing when an open transaction has no exact identity capability.
 If execution raises or the process dies after external work, the durable marker
 remains set. A strict restore therefore fails closed with
 `StepInProgressOnRestore`; the host must reconcile operation results or classify
-the run as uncertain rather than blindly replaying the transition. The same
+the run as uncertain rather than blindly replaying the transition. The error's
+`state` is the persisted state the interrupted step started from (a Symbol) and
+its `transition` is the next transition the payload records, such as a routed
+one, or nil when it records none, so the host can tell which step died without
+reading Smith's payload itself. The same
 in-memory workflow object cannot retry an attempted transition. If a host
 transaction rolls back after a successful `persist!`,
 `complete_persisted_step!` rejects the rolled-back checkpoint. When the adapter
@@ -563,6 +603,619 @@ entry points remain guarded even when a host prepends modules after class
 creation, and prepared execution bypasses host `advance!` wrappers so they
 cannot become transition execution authority.
 
+## Step Isolation and Untrusted Content
+
+By default a workflow keeps one transcript: each agent step sees the session
+(with `inject_state` and masking applied) and its accepted output is appended
+for the next step. Three declarations narrow that:
+
+```ruby
+class ReviewContext < Smith::Context
+  persist :document_id, :source, :fields
+  untrusted :source, :fields   # the document, and what an agent drew from it
+end
+
+class DocumentReview < Smith::Workflow
+  session :isolated
+  context_manager ReviewContext
+  initial_state :fetched
+  state :extracted
+
+  transition :extract, from: :fetched, to: :extracted do
+    execute :field_extractor,
+            input: ->(ctx) { [{ role: :user, content: Smith::Untrusted.wrap(ctx[:source], source: "document") }] },
+            output_to: :fields
+  end
+end
+```
+
+- `input:` (on `execute`, `route`, `fan_out`, `optimize` and `orchestrate`)
+  is a callable of a copy of the context that returns the step's messages: a
+  non-empty Array of at most 100 `{ role: :user or :assistant, content: }`
+  with content, bounded and JSON-shaped like session messages (values that are
+  not JSON are copied shallowly). They replace the transcript for that step;
+  the agent's own instructions still apply, and Smith adds its continuation
+  user turn only when they end with the output the run last added to its
+  transcript, as it does for the transcript. A fan-out's branches share one
+  `input:`. An optimizer's `input:` is its generator's, and also its
+  evaluator's under `evaluator_context: :inject_state`, which then sees the
+  input followed by the candidate and no injected state.
+- `output_to: :key` writes the step's output to `key` instead of the
+  transcript, once the step completes. The value is JSON-shaped (Symbols
+  become Strings) and in the context's key shape, so it reads the same after a
+  restore; a step with no output writes nil. An explicit `persist` list must
+  allow the key: graph validation reports `:output_to_not_persisted` (and
+  `:await_into_not_persisted` for an await's `into:`), and the step is refused
+  before it runs. The step's output, `RunResult#output` and a later compute
+  step's `last_output` are this output; the next shared agent step still reads
+  the transcript, whose last output gets the continuation turn.
+- `session :isolated` (per workflow; `:shared` is the default) keeps no
+  transcript for agents: every agent step must declare `input:` (it fails
+  otherwise, and graph validation reports `:isolated_step_without_input`),
+  accepted outputs are never appended, and `inject_state` never runs. A nested
+  workflow keeps its own session mode; validation warns
+  (`:isolated_workflow_nests_shared_session`) when an isolated workflow nests a
+  shared one.
+- `untrusted :key` in a `Smith::Context` marks values from outside the system,
+  and values a step derived from them: `inject_state` formatters and the
+  `seed_messages` builder never receive them, and an agent that names one in
+  `inputs` is refused when it runs, in this workflow and in any workflow it
+  nests (runtime readiness reports `:untrusted_agent_input` and
+  `:nested_untrusted_agent_input`). Reserved input names (`provider`,
+  `model_id`, `endpoint_mode`) are never read from the context, so they are not
+  refused. Untrustedness belongs to the key, not the value: a key that an
+  agent writes from untrusted content (`output_to: :fields` above) carries the
+  outside text's influence, so declare it untrusted too.
+- `Smith::Untrusted.wrap(text, source:)` places outside text between tags with
+  a boundary the text cannot guess. Inside a step the boundary derives from the
+  run's namespace, the step's position, the source and the text, so a step that
+  prepares its input again (a durable composite step on recovery) gets the same
+  messages; elsewhere it is random. Wrapping changes the text: an untrusted tag
+  in it (with spaces, invisible characters or full-width forms) becomes
+  `untrusted-text`, and a line that opens with a role label (`system:`,
+  `assistant:`, `user:`, `tool:`, `developer:`, `human:`, `ai:`, after any line
+  break and behind spaces, invisible characters, brackets, quotes, `#`, `>`,
+  `*`, `_` or `-`) gains "(quoted text)", ordinary text such as `user: alice`
+  included. Markdown headings, XML-like tags and look-alike letters are left
+  as they are. Wrapping marks where outside content starts and ends; it does
+  not make the content safe to act on.
+- A host admits session messages with the role `user`, `assistant` or `tool`
+  only; system messages come from Smith (injected state) and the
+  `seed_messages` builder. A restored session message, and a durable composite
+  step's prepared message, must have one of those roles or `system`. A payload
+  written by an earlier version may hold a system message the host admitted;
+  it still reaches agents, so rebuild such sessions before relying on this.
+
+### Content Parts
+
+A message in a step's `input:` (and, for text, in `seed_messages`) can carry
+content parts instead of a String:
+
+```ruby
+ref = Smith.artifacts.store(pdf_bytes, content_type: "application/pdf")   # inside a step
+Smith::Message.user([Smith::Content.document(ref, media_type: "application/pdf", cache: true),
+                     Smith::Content.text("Who signed?")])
+Smith::Message.system([Smith::Content.text(house_style, cache: true)])     # in seed_messages
+```
+
+- **Parts.** `Smith::Content.text(text, cache: false)`,
+  `Smith::Content.document(ref, media_type:, cache: false)` and
+  `Smith::Content.image(ref, media_type:, cache: false)` build frozen,
+  JSON-shaped parts: text of 1 byte to 1 MiB with more than whitespace, in
+  UTF-8; a document (`application/pdf`, `text/plain`) or image (`image/png`,
+  `image/jpeg`, `image/gif`, `image/webp`) as a ref from `Smith.artifacts.store`
+  (letters, digits, `-`, `_` and `:` only, at most 1 KiB), never bytes, so a
+  message with parts stays small in sessions, payloads and digests.
+- **Messages.** `Smith::Message.user`, `.assistant` and `.system` take a String
+  or 1 to 100 parts. Only a user message takes documents and images; system
+  and assistant messages take text parts. `input:` takes user and assistant
+  messages; `seed_messages` take text parts only, since a seed is built
+  before the run can store an artifact. A message with parts carries
+  `"parts" => true` beside its role and content; no agent output can carry
+  it and `append_session_messages!` refuses it, so a structured output that
+  looks like parts stays data and reaches providers as JSON text, and only
+  content Smith builds from parts for the call passes to RubyLLM unencoded.
+- **Whose artifacts.** A ref is read by the run whose step stored it, the
+  workflows it nests, and its child runs: a child run (map, call or spawn) may
+  read its parent's artifacts and those of the parent's nearest ancestors, up
+  to 16 generations, so a document stored once reaches every item of a map. A
+  run never reads an unrelated run's artifacts, nor its own children's; a
+  child returns content, or the parent stores what it needs. Content parts are
+  read only from a store that records which run stored each artifact
+  (`stored_by?`, as every built-in store does), so a ref built from untrusted
+  text cannot name another run's artifact or a file outside the store. A run
+  that waits, retries durably or resumes in another process needs a shared
+  store (S3 or R2, Active Record, or File on a shared disk) whose retention
+  outlives it; see Artifacts.
+- **Reading.** Before the first provider attempt Smith reads each document and
+  image once and checks it, or fails the step with `Smith::ContentUnavailable`
+  (a `WorkflowError`, never retried as a provider failure); the store's own
+  errors (a timeout, `Smith::ArtifactStoreMissing`) pass on as they are, so
+  `retry_on` can take a transient one. Each must be a
+  non-empty String whose bytes match its media type (a PDF, PNG, JPEG, GIF or
+  WebP by its signature; `text/plain` as UTF-8), a document at most 24 MiB and
+  an image at most 5 MiB, and the request's text with its documents and images
+  as they are sent (PDFs and images in base64) at most 32 MiB.
+- **Anthropic** gets native blocks (`RubyLLM::Content::Raw`): a PDF or image as
+  base64, a `text/plain` document as text, and `cache_control: { type:
+  "ephemeral" }` on a part marked `cache: true`. A cache breakpoint caches the
+  whole request prefix up to and including that block, in Anthropic's order:
+  tools, then system (the agent's instructions, then the seed system
+  messages), then the messages. So a cached seed prefix serves later calls of
+  the same agent (same tools and instructions), and a cached `input:` part
+  caches the agent's tools and instructions with it. Anthropic needs a minimum
+  prefix length (about 1,024 tokens on most models), keeps a cache about 5
+  minutes, charges a premium to write one, and takes at most 4 breakpoints per
+  request: Smith counts the chat's own (cached instructions) with its parts,
+  against every Anthropic model in the step's chain, and refuses more before
+  the first call. Concurrent branches (`parallel:`, `fan_out`) each write the
+  cache.
+- **Other providers** get the parts as one text in their order: text parts as
+  they are, a `text/plain` document inline (`<file name='document-2.txt'
+  mime_type='text/plain'>...</file>`), and `[attached: image-3.png]` where each
+  PDF or image stands, attached under that name. `cache:` adds nothing for
+  them (OpenAI and Gemini cache on their own; Bedrock and OpenRouter get no
+  cache markers from Smith). Each attempt builds its own chat, so a fallback
+  model on another provider gets its own form, and a model whose provider
+  takes no such content (RubyLLM raises `UnsupportedAttachmentError`) passes
+  the step to the next model in the chain, or fails it with
+  `Smith::ContentUnavailable` when none is left.
+- **Sessions.** Seed messages keep their text parts in the session, digests
+  included, and reach the agent steps that read the session: a step with
+  `input:` sees only its input and its agent's instructions. A restored
+  session may hold parts only among its seed messages, each valid (text parts
+  only), however the payload was parsed, or the restore raises
+  `Smith::SerializationError`. A document is outside content by nature and
+  cannot be wrapped like text; give a step that reads one an isolated session
+  or a narrow `input:`.
+
+## Waiting
+
+`await` and `delay` steps let a run wait, durably and bounded, without anything
+sleeping or polling:
+
+```ruby
+class DocumentReview < Smith::Workflow
+  initial_state :requested
+  state :failed
+
+  transition :fetch_source, from: :requested, to: :fetched do
+    await :source_ready, into: :source, timeout: 6 * 3600 do |step|
+      reply = SourceClient.request(ref: step.read_context(:source_ref), idempotency_key: step.await_token)
+      step.resolve!(reply.to_h) if reply.complete?   # finished inline: no wait
+    end
+    on_timeout :source_unavailable
+  end
+  transition :source_unavailable, from: :requested, to: :unavailable
+
+  transition(:cool_off, from: :fetched, to: :reviewed) { delay 600 }
+end
+```
+
+- `await event, into:, timeout:` waits for `event`. `into:` is the Symbol
+  context key its payload is written to; an explicit `persist` list must allow
+  it, which is checked before the block runs. `timeout:` is required: positive
+  seconds, or a callable that receives the context and returns them. An await
+  must also declare where its timeout goes, `on_timeout` or `on_failure`; one
+  with neither is refused when the class is defined. `delay seconds` waits for
+  the time alone and then continues to its `to:` state. Either is the
+  transition's one execution body.
+- The optional block runs when the step is entered, with a step object that
+  reads the context (`read_context`, `context`), names the wait (`await_token`,
+  the key whatever the block sends must carry), gives other keys
+  (`idempotency_key`), and may finish at once with `resolve!(payload)`, which
+  takes a copy of the payload. It cannot write the context. In lax mode it runs
+  again if the process dies before the wait is stored, under the same token, so
+  what it sends must be idempotent by that token; in strict mode the restore of
+  such a run raises `Smith::StepInProgressOnRestore` instead. `retry_on`
+  retries the block, never a timeout.
+- Entering a wait is not a step. `advance!` returns nil, `run!`,
+  `run_persisted!` and `advance_persisted!` stop with the wait stored, and the
+  step count and `max_transitions` do not move; `wall_clock` is charged only
+  for the time the block ran. `waiting?` and `awaiting` (a
+  `Smith::Workflow::Wait` with `kind`, `event`, `token`, `transition`, `into`,
+  `entered_at`, `deadline_at`, and `children`, `attempt` and `error` for the
+  waits that have them; the event and key as the transition declares them)
+  let the host index the wait, and `pending_transition_name` is the
+  awaited transition. While a run waits, `advance!` does nothing and the run is
+  never terminal, so no `clear:` policy deletes it.
+- `Workflow.resume!(key, event:, token:, payload:, adapter:)` restores the run,
+  checks that it awaits that event under that token, writes the payload to
+  `into:` (bounded and JSON-shaped like a session message, with the context's
+  key shape; it never enters the session), completes the step, and stores the
+  run with the version check. It returns `:resumed`, `:failed` when the
+  completion took the failure path, `:missing` when no run is stored, or
+  `:stale`, changing nothing, for any other event or token.
+- A reply can arrive before its wait is stored, when the block's request is
+  answered at once. If the stored run stands at that await and the token is
+  the one it is entering the wait with, `resume!` completes the step at once
+  (in strict mode too, where the run is caught mid-step) and the process still
+  entering the wait loses its next store with
+  `Smith::PersistenceVersionConflict`; restoring the run shows it past the
+  await. Any other delivery to a strict run caught mid-step raises
+  `Smith::StepInProgressOnRestore`.
+- `Workflow.expire_wait!(key, token:, now:, adapter:)` ends a wait whose
+  `deadline_at` has passed at `now` (a Time or an ISO 8601 String): a `delay`
+  continues to its `to:`, an `await` takes `on_timeout` (a `StepCompleted`
+  whose `to` is the await's own state) or else `on_failure` with
+  `Smith::WaitTimedOut`, which is never retried and restores as itself with the
+  wait it names; a durable retry's wait (`kind: :retry`) ends without a step,
+  and the host must then continue the run (see Durable Retry). It returns
+  `:expired`, `:not_due`, `:missing` or `:stale`. The host's scheduler decides
+  when to call it; Smith never wakes a run. `now:` is the host's clock: passing
+  a time at or past `deadline_at` ends a wait early, as a "retry now" control
+  may.
+- Only one completion of a wait wins. On one instance, a completion claims the
+  wait, so a delivery and an expiry racing there cannot both complete it. Across
+  processes, the class methods need an adapter with `store_versioned` (Memory,
+  Redis, Active Record), so the second store fails with
+  `Smith::PersistenceVersionConflict`; they refuse adapters without it.
+- The completion of an await, delay, map or call, by `resume!` or
+  `expire_wait!`, is the step: it has the step's position, so the token matches
+  the entry's, and it emits `StepCompleted` (or `StepFailed`). The end of a
+  retry wait is not a step and emits nothing; the attempt that follows is. In strict mode a waiting run is a
+  checkpoint, not a step in progress, and restores cleanly.
+- A stored wait is checked on restore: one that is malformed, whose deadline
+  precedes its entry or has no time zone, or that sits beside a pending
+  transition raises `Smith::SerializationError`; one whose transition the
+  class no longer declares exactly (kind, event, `into:`) or that does not
+  start from the run's state raises `Smith::UnknownTransitionOnRestore`. A
+  `migrate_from` that renames such things must rewrite `awaiting` too, and
+  `pending_retry` for a retry wait.
+- A wait that does not end within the store's persistence TTL is refused when
+  it is entered, before the block runs. `stuck_for?` reports a waiting run as stuck
+  only once its deadline has passed by the threshold.
+- Not in 0.12: split-step persistence refuses to prepare a step that waits or
+  launches runs (an `await`, `delay`, `map`, `call` or `spawn`, or a step with
+  a durable retry), or any step while the run waits, and a durable retry
+  cannot wait inside split-step execution. An in-process nested workflow that
+  reaches a wait fails its parent's step (runtime readiness reports
+  `:nested_workflow_waits`).
+
+### Durable Retry
+
+`retry_on ..., durable: true` waits out each backoff as a wait of the run
+instead of sleeping in the worker:
+
+```ruby
+transition :charge, from: :approved, to: :charged do
+  compute { |step| Payments.charge(order, idempotency_key: step.idempotency_key) }
+  retry_on Payments::Timeout, attempts: 5, backoff: 60, max_delay: 3600, durable: true
+  on_failure :charge_failed
+end
+```
+
+- An attempt that fails in a way the policy retries, with attempts left,
+  enters a wait of kind `:retry` for the transition (no event, no `into:`),
+  due when the backoff ends, under a token of the attempt it waits for. The
+  step does not complete or count, and the run records its pending retry (the
+  transition, the next attempt, the key generation, and the class name of the
+  error) as `pending_retry`; `awaiting.attempt` and `awaiting.error` show them.
+  Traces record `outcome: :retrying` (with the attempt, error and deadline)
+  and, when the wait ends, `:retry_due`.
+- `expire_wait!` past the deadline ends the wait without a step and returns
+  `:expired`; the transition becomes the run's next. The host continues the run
+  (`run_persisted!` or `advance_persisted!`), as after any wait, and the step
+  makes the recorded attempt under the recorded keys: the same as the failed
+  attempt's, unless that attempt was declared settled (see Replay Safety).
+  It may succeed, fail for good (`on_failure`, or the error), or wait again.
+  `resume!` never completes a retry wait.
+- The pending retry persists until the step ends without entering a wait:
+  it completes, fails, or raises. A crash during the attempt makes the same
+  attempt again under the same keys in lax mode; in strict mode the attempt
+  runs under the step-in-progress mark, so the restore of a run caught in it
+  raises `Smith::StepInProgressOnRestore`.
+- No retry wait is entered once the run's wall-clock deadline has passed, after
+  a tool ran (`Smith::ToolOutcomeUncertain` is never retried), or for a backoff
+  that does not end within the store's persistence TTL (the step fails with a
+  `Smith::WorkflowError` whose cause is the attempt's error); runtime readiness
+  reports a policy whose longest backoff, jitter included, reaches the TTL
+  (`:durable_retry_outlasts_ttl`). Time spent waiting does not count toward
+  `wall_clock`.
+- `durable:` needs at least 2 attempts, a positive backoff and `on_failure`,
+  and is refused on an await, delay, parallel, fan_out, optimize or
+  orchestrate step. `on_failure` is required because a last failure raised to
+  the host would leave the stored run before that attempt, as a crash does,
+  so a job that runs again would make the attempt again at once, past the
+  policy's attempts; through `on_failure` the failure settles the run; a map,
+  call or spawn takes no `retry_on` at all. A durable retry inside an
+  in-process nested workflow, or a child run that `Smith::Launchers::Inline`
+  runs, fails the parent's step (a nested workflow's with the child's error as
+  the cause), and runtime readiness reports it as `:nested_workflow_waits`.
+  Graph inspection shows `durable: true` in the retry policy.
+- A streamed step's sink (`stream:`) gets `reset(:retry_wait)` when the run
+  enters a retry wait, and holds nothing durable: the attempt the host's later
+  job makes streams to a new sink from the start. In a stored run, a streamed
+  step's `finish` comes once `persist!` has stored the step (a store that fails
+  gives the sink `fail` with the store's error instead), so a sink never shows
+  as final an output the store did not keep.
+- A stored `pending_retry` is checked on restore: it must name the transition
+  of the stored retry wait, or the run's next transition when the run does not
+  wait, with an attempt after the first, a key generation an earlier attempt
+  could reach, and an error class name; a stored retry wait needs one.
+  Anything else raises `Smith::SerializationError`, except a definition
+  change: a transition that no longer declares a durable retry, or now
+  declares fewer attempts than the recorded one, raises
+  `Smith::UnknownTransitionOnRestore`.
+
+### Batch Waits
+
+A step whose mode is `:batch` (`execute` or `fan_out`, README "Batch mode")
+sends no request: it captures each call's request as a synchronous call would
+post it, hands it to `Smith.config.batch_queue` and waits.
+
+- The requests are queued once the step's attempt has ended without error
+  (its context checks included), so a step that fails queues nothing. The
+  wait is of kind `:batch` (event `:batch_result`, `children` the number of
+  requests, `attempt` the step attempt), due 25 hours after entry (batches end
+  within 24). Its token is a key of the step's position and attempt, and is
+  the `custom_id` of an agent step's request; a fan-out branch's is
+  `Smith::Batch.custom_id(token, branch_key)`. The run records its pending
+  batch (`pending_batch`: transition, attempt, key generation, and the digest
+  of each request by its id). On restore a malformed one, an attempt past the
+  step's `retry_on attempts:`, a key generation without
+  `new_idempotency_key_on`, or ids the step's branches no longer give raise
+  `SerializationError`, and a wait the class no longer batches is refused
+  like an unknown position.
+- Enqueues are one call per request. If the queue raises after taking some
+  of a wait's requests, the step fails and those entries answer no wait: a
+  host submits only entries whose run waits under their `await_token`
+  (`awaiting.token`), and drops the rest.
+- The host resumes the run once, with every result of the wait:
+  `resume!(key, event: :batch_result, token:, payload: {custom_id => result})`,
+  each result as the provider's results file gives it (`succeeded` with its
+  `message`, `errored`, `canceled` or `expired`). A payload that names other
+  ids, or a result of an unknown shape, raises ArgumentError and leaves the
+  wait for a good one. `expire_wait!` past the deadline completes the step as
+  if every result were missing.
+- The step runs again at its attempt and key generation, each call taking its
+  result instead of going over the wire, then goes on as after a synchronous
+  call; the completion is the step. A call that renders another request than
+  the one sent (`pending_batch` keeps each request's digest to tell: a model
+  block, instructions or input that changed while the run waited, a date, data
+  read afresh) does what the step's `on_request_change:` says. `:keep` (the
+  default) takes the result, and the call's `:provider_call` trace says
+  `request_changed: true`; `:resend` sends the request as rendered now over
+  the wire; `:fail` raises `Smith::BatchRequestChanged`, an `AgentError` that
+  `retry_on` may retry. A result set aside so is charged as a
+  `:failed_attempt` with `mode: :batch`.
+  Only the batch's own attempt takes results; a succeeded result no call took
+  (a fan-out branch cancelled by a sibling's failure) is recorded and charged
+  as a `:failed_attempt` all the same. In strict mode a completion is stored as
+  a step under way before it runs, and a run caught in one is refused as
+  unfinished (`StepInProgressOnRestore`). A failed result raises
+  `Smith::BatchResultFailed`, which `retry_on` retries: in process, the next
+  attempt sends a new batch under a new token (or goes over the wire, if the
+  mode callable says :sync); a durable retry waits its backoff first, and the
+  batch its next attempt sends takes the place of its pending retry. A request
+  the provider refused raises `Smith::BatchRequestInvalid`, never retried.
+- Admission against the budget happens when the request is queued; nothing is
+  held while the run waits. The result's usage is charged when it comes, past
+  the limits if need be (`Budget::Ledger#reserve_spent!`).
+- A batch step cannot wait inside split-step execution, and a batch step in
+  an in-process child (a nested workflow, or a child run run inline) fails
+  before anything is captured or queued (readiness: `:nested_workflow_waits`).
+
+## Budget Ledgers
+
+A workflow's `budget_ledger` (README, "Budget ledgers") is never persisted,
+and restoring a run never builds it: Smith builds it at the run's first
+request from the run's `budget_ledger_scope`, the values its scope keys had in
+its trusted context when it was created, kept with the run (a run stored
+before its class named those keys takes them from its context when
+restored). A stored scope that is not of JSON scalars raises
+`SerializationError`.
+
+A batch request's hold is keyed by its `custom_id`, which the run's
+`pending_batch` already keeps, so the delivery that resumes the run (in any
+process) settles it on the same ledger: a succeeded result, used by the step
+or not, reconciles what it cost; a failed result, or an expired wait,
+releases it. A ledger that cannot be built then is logged, and its holds
+expire. Holds are taken while the step's attempt runs and released if it ends
+before its requests are queued, except one the queue already took. A process
+that dies during a request leaves its hold unsettled until its `expires_at`;
+a delivery that ran again after a crash settles the same keys again.
+
+A tool loop paused for budget keeps its messages (its tool calls, their
+results, its thinking and signatures; at most 1 MiB) in the step's
+`pending_retry` as `continuation`, and the retry continues from them. A
+continuation that is not of the shape Smith keeps raises `SerializationError`
+on restore.
+
+## Child Runs
+
+`map`, `call` and `spawn` steps start other workflows as runs of their own.
+Smith decides which runs to start and checks what comes back; the host's run
+launcher starts them, and the host persists them, runs them and reports their
+results. Smith never queues, polls or waits on a child.
+
+```ruby
+class QueueLauncher
+  # children: Smith::ChildRun (ref, key, workflow, input, untrusted_keys,
+  # artifact_namespaces, #build)
+  def launch(children:, parent_key:, await_token:)
+    children.each do |child|
+      ChildRunJob.perform_later(key: child.key, parent_key:, await_token:, ref: child.ref,
+                                workflow: child.workflow.name, input: child.input,
+                                untrusted_keys: child.untrusted_keys,
+                                artifact_namespaces: child.artifact_namespaces)
+    end
+  end
+end
+
+Smith.config.run_launcher = QueueLauncher.new
+```
+
+- A child's `ref` is the index of its item, and its `key` is the step's token
+  and the ref together, unique across every child-run step of every run. Items
+  come from the context (a persisted key holding an Array, or a callable of the
+  context, which must give the same list on a replay), so a replayed step
+  launches the same children under the same keys: the launcher must be
+  idempotent on `child.key`. Only a launcher whose `inline?` is true has its
+  return value taken as results (it must return them, as an Array); anything
+  another launcher returns is ignored. The job side rebuilds a
+  `Smith::ChildRun` from its arguments (input keys as Symbols) and calls
+  `#build`, which checks the untrusted key names it gets back as Strings and
+  the artifact namespaces the child may read (its parent's and the parent's
+  nearest ancestors', at most 16; a job that omits them gives the child none,
+  and its reads of the parent's documents fail).
+- A child gets only the context `input:` returns (JSON-shaped, bounded at 1 MiB
+  each and 16 MiB for a step's children together) and no session. A key the
+  parent treats as untrusted that `input:` passes under the same name stays
+  untrusted in the child: `ChildRun#build` constructs the child with those keys
+  in force, and they persist with it (`inherited_untrusted_keys`). The mark
+  goes by name: a value passed under another name, or inside another value, is
+  not marked, so pass untrusted content under its own key.
+- A map, call or spawn needs a persisted run (`run_persisted!` or
+  `advance_persisted!`), except with a launcher that runs the children itself
+  (`inline?` true, as `Smith::Launchers::Inline`): it returns their results and
+  the step completes at once. A map over no items completes at once and
+  launches nothing. In `idempotency_mode :strict` a map, call or spawn step
+  that crashed mid-launch runs again on restore, since launching is idempotent.
+- Otherwise the step waits like an `await` for the event `:children`, bounded by
+  its `timeout:` and routed by `on_timeout` or `on_failure`. Once every child is
+  terminal the host resumes the parent with one entry per ref,
+  `Smith::ChildRun.result(ref:, run_result:)`:
+  `{ ref:, status: "succeeded" | "failed", state:, output:, error: }`, where a
+  child failed when it ended in its failed state or any of its steps raised.
+  `state`, `output` and `error` may be left out (as nil); `state` and `error`
+  are text of at most 256 bytes. Results that do not match the number of
+  children the step launched (`Wait#children`) raise to the host and leave the
+  run waiting. They are bounded at 4 MiB together, after the outputs a map does
+  not collect are dropped, so `collect: :states` or `:none` suits a large map.
+- `collect: :outputs` writes the entries to `into:` in ref order, `:states`
+  drops the outputs, and `:none` writes `{ succeeded:, failed: }`. A `call`
+  writes its child's output itself. With `on_item_failure: :fail` (always for
+  a call) any failed child fails the step with `Smith::ChildRunsFailed`
+  (failure family `child_runs_failed`, up to 20 failed refs with the counts),
+  so such a step must declare `on_failure`.
+- No map, call or spawn takes `retry_on`: the launcher retries launching, and
+  a map's results arrive with the host's resume, outside any attempt. Each
+  child is a run with its own budget; the parent's budget does not cover it.
+- `Smith::Launchers::Inline` runs each child to its end under the parent's
+  deadline, apart from the parent's tool evidence and attribution, and reports
+  a child left waiting, or one that raised, as failed with its error.
+- A `spawn` launches and completes its step; nothing comes back.
+- Graph inspection shows the kind (`:map`, `:call`, `:spawn`) and the child
+  workflow, bound, items, collect and failure modes. Runtime readiness walks
+  the child workflow like a nested one (a child may wait: it is a run of its
+  own, and may map over its own workflow) and reports `:run_launcher_missing`
+  when no launcher is configured. An in-process nested workflow that reaches a
+  map, call or spawn is reported (`:nested_workflow_waits`) unless the launcher
+  runs children in process. An agent in a child run that reads a key the
+  parent marks untrusted is a warning (`:nested_untrusted_agent_input`), not
+  an error as in a nested workflow, since readiness cannot see whether
+  `input:` passes that key.
+
+## Replay Safety: Idempotency Keys
+
+A replayed step's provider calls are paid for again unless
+`Smith.config.response_memo` keeps them (README, "Response memo"): each call
+is keyed by its position's idempotency scope (so a settled failure's new key
+generation reaches new records), its attempt, its model in the fallback chain
+and its place among the attempt's calls. Nothing about a memo is persisted
+with the run.
+
+
+A step that is replayed, by a retry or by a restore after a crash, sends its
+outside requests again. Smith gives each request a key the step can pass along,
+so a receiver that honours idempotency keys can drop the repeat. Passing the key
+is the step's job, and what a receiver does with it stays the host's concern:
+
+```ruby
+transition :charge, from: :approved, to: :charged do
+  compute do |step|
+    Payments.charge(order, idempotency_key: step.idempotency_key)
+    Receipts.email(order, idempotency_key: step.idempotency_key(:receipt))
+  end
+  retry_on Payments::Timeout, Payments::Declined, attempts: 3,
+           new_idempotency_key_on: [Payments::Declined]
+end
+```
+
+- `step.idempotency_key(scope = nil)` in `compute` and `run` steps,
+  `idempotency_key(scope)` inside a `Smith::Tool`'s `perform`, and
+  `Smith::IdempotencyKey.current(scope = nil)` for host code called from a
+  running step. Each returns 64 lowercase hex characters (a SHA-256), which
+  fits receivers that cap keys at 64 characters.
+- A key is unique to the run, the step's position and the scope. The position
+  is the step number and transition, then any nested workflow step, parallel or
+  fan-out branch (durable composite branches included), optimizer round and
+  role (generator, `before_eval`, evaluator), or orchestrator round and worker
+  task. A worker task is named by its content and which copy of equal tasks it
+  is, so a replayed orchestrator that lists the same tasks in another order
+  keeps each task's key. A transition that runs again later in the run (a loop,
+  or an `on_failure` route back to it) has a new step number and so new keys.
+- Use a scope to tell apart several requests of one step. A tool can run several
+  times in a step, and a replayed model may word or order its calls
+  differently, so a tool's scope is required and should be the request's stable
+  identity (`idempotency_key(customer_id:, invoice_number:)`), not free text. A
+  tool's key also includes its `name`, so renaming a tool class changes its
+  keys; a tool without a name is refused. A plain RubyLLM tool has only
+  `Smith::IdempotencyKey.current`, so it should put its own name in the scope.
+- A scope is built from Strings, Symbols, numbers, `true`, `false`, `nil`,
+  Arrays and Hashes with String or Symbol keys. Symbols count as their Strings,
+  Hash order does not matter, and a Float with a whole value counts as the
+  Integer (a model's JSON may send `12` or `12.0`). NaN, Infinity and other
+  objects raise `ArgumentError`, as do scopes nested deeper than 32 levels or
+  holding more than 100,000 values, which bounds the work per key.
+- Keys exist only while a step does its own work: its body, agents, tools and
+  guardrails. Event handlers and `on_step` callbacks get none. The position is
+  thread-local state that Smith carries into its own branch threads and
+  concurrent tool calls; code in a Fiber or Thread the host creates has none,
+  while `step.idempotency_key` works anywhere.
+- The run part is the workflow's execution namespace, a UUID assigned when the
+  workflow is built and persisted with it. A payload written without one (by an
+  earlier version) gets one derived from its class, persistence key and
+  creation time, the same on every restore, until a checkpoint stores it. A
+  stored namespace that is not a UUID raises `Smith::SerializationError`. Runs
+  built with `new` never share keys, even when one reuses another's persistence
+  key after it was cleared; runs restored from copies of one payload share
+  them, and their artifact scope, so a payload must not be cloned into several
+  runs.
+- A replay keeps a step's keys when the checkpoint written before the step
+  holds the namespace and step count, as `advance_persisted!` and
+  `run_persisted!` write it. In `idempotency_mode :lax` a run restored after a
+  crash mid-step replays that step under its keys. In `:strict` mode the restore
+  raises `Smith::StepInProgressOnRestore` instead, and reconciling the step's
+  requests is the host's; a host-coordinated prepared step is never executed
+  again after dispatch. A run that is never persisted has no replay to protect.
+
+Every attempt of a step uses the same keys. A retry after an uncertain failure
+(a timeout, a dropped connection) then cannot repeat a request that took
+effect. Some receivers store a failed reply under its key and return it for
+every repeat. When an attempt fails in a way that shows its own requests took
+no effect, the attempts after it can use new keys:
+
+- `step.fail!("declined", retryable: true, new_idempotency_key: true)` in a
+  `compute` or `run` step with `retry_on`. It counts only for the step that
+  raised it: a nested workflow's step that raises it does not move the parent's
+  retry to new keys, because the child's earlier steps may have taken effect.
+  The flag is not kept in the persisted failure record.
+- `retry_on ..., new_idempotency_key_on: [Payments::Declined]`. The listed
+  classes must be ones the policy retries (any class, when the policy lists none
+  and Smith's retry classification decides), each listed once. Classes that may
+  hide a tool's outcome are refused, and so is the option on a nested workflow,
+  parallel, fan-out, optimize or orchestrate transition, where one error cannot
+  speak for every request of the attempt. A broad class (`StandardError`) moves
+  every retry, timeouts included, to new keys. Graph inspection shows the list
+  in the transition's retry policy.
+
+`step.attempt` is the attempt under way within one run of the step's
+`retry_on` attempts, from 1 (a nested workflow's step counts its own). A host
+that wants a different key on every attempt can pass it in the scope
+(`step.idempotency_key([:charge, step.attempt])`).
+
+The generation of keys lives only within one run of the step's retries (a
+durable retry persists it across its waits; see Durable Retry). When
+the attempts run out and the host retries the run later (a job that runs again),
+the step starts again from its first keys, so a receiver that stored a failed
+reply returns it again. A host that wants its own later retries to use new keys
+puts a counter it owns in the scope. After a crash, the step's retries run again
+from the first attempt (a durable retry's from the attempt it recorded); a replay reaches the newer keys again only if the
+receivers refuse the replayed requests as they did before, which a receiver that
+stores replies under their keys does.
+
 ## Active Record Optimistic Locking
 
 `ActiveRecordStore` keeps two version domains deliberately separate:
@@ -582,7 +1235,22 @@ Valid JSON scalars are not workflow-state documents, and explicit null,
 negative, or non-integer versions fail closed instead of being overwritten.
 
 The default host model needs a unique key, a JSON/text payload, and Rails'
-standard optimistic-locking column:
+standard optimistic-locking column. `bin/rails generate smith:install`
+writes its migration, model and initializer (`--state-model`,
+`--max-state-bytes`; see the README's Installation), with a CHECK on the
+payload's size a third above `persistence_max_bytes`. Smith keeps a run
+within the bound: a stored run's step whose result would not fit fails with
+`Smith::PersistedStateTooLarge` before the result is kept (its usage
+recorded, its `on_failure` taken, strict mode included), a tool loop never
+pauses into a state the bound would refuse (it goes on unheld), and any
+other state past the bound is refused before anything is written. The CHECK
+bounds the stored bytes, a column the host encrypts included; an encrypted
+payload holds runs, but `replace_exact` (split-step exact writes) compares
+payloads in the database and refuses one. The key is a string at least 64
+characters wide: child runs are stored under 64-character keys of their
+own, so it is never a host row's id. `smith doctor` checks the model on
+every run (skipping when the database cannot be reached), and its durability
+probe writes with `store_versioned`, as runs do. By hand:
 
 ```ruby
 create_table :workflow_states do |t|
@@ -680,17 +1348,81 @@ class LargeReportAgent < Smith::Agent
 end
 ```
 
-Configure a backend:
+Configure a store. Like workflow state, artifacts must outlive the process
+that stored them whenever a run waits, retries or resumes elsewhere, so Smith
+assumes none: outside `test_mode`, storing or reading an artifact with
+nothing configured raises `Smith::ArtifactStoreMissing` (a run that keeps no
+artifacts needs no store), and `smith doctor` warns. In a test environment,
+set `test_mode` (an in-process store) or `artifact_store = :memory`.
 
 ```ruby
 Smith.configure do |config|
-  config.artifact_store = Smith::Artifacts::Memory.new
-  config.artifact_retention = 3600
+  # Cloudflare R2, or any S3-compatible bucket (needs the aws-sdk-s3 gem, or client:)
+  config.artifact_store = :s3
+  config.artifact_options = {
+    bucket: "smith-artifacts", prefix: "artifacts",
+    endpoint: "https://<account>.r2.cloudflarestorage.com", region: "auto",
+    access_key_id: ENV.fetch("R2_ACCESS_KEY_ID"), secret_access_key: ENV.fetch("R2_SECRET_ACCESS_KEY"),
+    # Cloudflare's advice for aws-sdk-s3 1.178 and later:
+    request_checksum_calculation: "when_required", response_checksum_validation: "when_required"
+  }
+  config.artifact_retention = 30 * 86_400
 end
 ```
 
+- An artifact is bytes: `store` takes a String (serialize anything else,
+  for example with `JSON.generate`) and refuses other values with a
+  `TypeError`.
+- `:s3` keeps each artifact as one object under `prefix`, its key the ref (a
+  random UUID after the store's tenant), with the run that stored it in the
+  object's metadata; only keys of that shape are artifacts, so other objects
+  in the bucket are never read, listed or deleted. It builds an
+  `Aws::S3::Client` from the remaining options, or takes the host's as
+  `client:` (not both); a Rails app already using R2 through Active Storage
+  can pass that service's client. Smith does not depend on the SDK.
+- `:active_record` keeps each artifact as a row through the host's model
+  (`model:`, a class or its name):
+
+  ```ruby
+  create_table :smith_artifacts do |t|
+    t.string :ref, null: false, index: { unique: true }
+    t.binary :data, null: false, limit: 64.megabytes   # MySQL's default blob holds 64 KiB
+    t.string :content_type, null: false
+    t.string :execution_namespace, index: true
+    t.timestamps
+  end
+  ```
+
+- `:file` keeps files in a directory (`dir:`), shared only when the directory
+  is; `:memory` keeps them in the process, for one process or tests (its refs
+  are content hashes, so runs that store the same bytes share one, each with
+  its own claim). A store object (or a class or module whose own methods
+  store), or a store class built with the options, works too: it implements
+  `store` and `fetch`; `stored_by?(ref, execution_namespace)` lets runs read
+  only their own refs (content parts and a run's deletes require it), and
+  `expired` and `delete(ref, execution_namespace: nil)` serve cleanup. Every
+  built-in store implements them all.
+- Any built-in store takes `namespace:`, a tenant prefix of letters, digits,
+  `-` and `_` that its refs carry; a store reads no other tenant's refs, and
+  with `artifact_tenant_isolation` it refuses to work without one.
+  `artifact_retention` is the retention every built-in store's `expired`
+  uses when none is given.
+- Cleanup is the host's. `workflow.artifacts` is a run's view of the store
+  (what its steps see as `Smith.artifacts`): it reads the run's refs, lists
+  its expired ones (`expired(retention:)`) and deletes them (`delete(ref)`,
+  only the run's own, and in Memory only the run's claim). Across runs,
+  `Smith.artifact_backend.expired(retention:)` lists refs older than the
+  retention and `delete(ref)` removes them. A run that waits keeps its
+  artifacts for as long as it waits, so keep the retention longer than the
+  longest wait, or clean up per run once it is cleared. A child run reads its
+  ancestors' artifacts, so a parent's must outlive its spawned children. On
+  S3, listing one run's expired refs costs a HEAD request per expired object
+  under the prefix.
+
 Why this matters:
 
 - large payloads can move out of the inline workflow result
-- refs are execution-scoped
+- refs are execution-scoped: a step stores and reads under its run's scope,
+  which ends with the step, so outside a run's steps `Smith.artifacts` is the
+  configured store itself; a child run reads its ancestors' refs too
 - nested workflows inherit artifact scope correctly