squishling 0.4.0 → 0.5.0

data/docs/failures.md

@@ -8,26 +8,48 @@ A failed attempt moves on to the next one; once the last fails, the error is rai
 
 | Failure | What Squishling does |
 |---|---|
-| Rate limit, 5xx, overload, timeout, connection error | RubyLLM retries at the HTTP level first (`RubyLLM.config.max_retries`, default 3). If it still fails, moves to the next attempt in a fresh chat; on the last attempt, raises `Squishling::LLMError` with the original exception as its `cause`. |
+| Rate limit, 5xx, overload, timeout, connection error | The [adapter](adapters.md) retries at the HTTP level first (RubyLLM: `RubyLLM.config.max_retries`, default 3; OpenAI-compatible: `max_retries:`, default 2). If it still fails, moves to the next attempt in a fresh chat; on the last attempt, raises `Squishling::LLMError` with the original exception as its `cause` (when the adapter has one: the RubyLLM adapter always does; the OpenAI-compatible adapter for transport errors, not for HTTP statuses). |
 | Bad API key, unknown model, missing provider config | Raises `Squishling::ConfigurationError`. Never escalated, never sent to the fallback. |
-| Provider rejects the request (400 Bad Request), e.g. an unsupported `temperature` or a schema it won't accept | Raises `ConfigurationError` naming the error class and HTTP status, and the params in use; the provider's own message is on `cause`. Never escalated, never sent to the fallback, so a setup mistake can't be silently covered up on every call. |
-| Empty or `nil` response (e.g. a refusal or a max-tokens cutoff) | Moves to the next attempt, then raises `Squishling::InvalidOutputError` |
+| Provider rejects the request (400 Bad Request), e.g. an unsupported `temperature` or a schema it won't accept | Raises `Squishling::RequestRejectedError` (a `ConfigurationError`) naming the error class, the HTTP status, and the provider's error status/code when its body carries one (`INVALID_ARGUMENT`, `invalid_request_error`). They are also on `error.provider_status` and `error.provider_code` (nil when absent); only plain tokens are kept, never the provider's message. The message points at a param only when the provider named it, and otherwise says the params, output schema, or input may be invalid. The provider's own message is on `cause` when the adapter has one (RubyLLM's does; with the OpenAI-compatible adapter `cause` carries only the status). It is never retried, so if you know a given code means "this input was bad", rescue it and decide for yourself. Never escalated, never sent to the fallback, so a setup mistake can't be silently covered up on every call. |
+| Empty or `nil` response (e.g. a refusal or a max-tokens cutoff) | Moves to the next attempt, then raises `Squishling::InvalidOutputError`. When the provider reported a max-tokens cutoff, the error says so and [`truncated?`](#a-cutoff-or-a-wrong-answer) is true. |
 | Malformed or truncated JSON | Moves to the next attempt, then raises `InvalidOutputError`. JSON wrapped in a markdown code fence is accepted. |
 | JSON that doesn't match the schema (wrong types, missing or extra keys, `null` in a non-`optional` field, a broken [conditional rule](schemas.md#contracts-beyond-the-schema), root not an object) | Moves to the next attempt with the validation errors, then raises `InvalidOutputError` |
 | Schema-valid output that [`squish_validate`](#output-checks-squish_validate) rejects | Moves to the next attempt with your messages, then raises `InvalidOutputError` |
+| The call has used its [token budget](#capping-a-calls-cost-token-budgets) | Stops before the next attempt and raises `Squishling::BudgetExhaustedError` (an `InvalidOutputError`) |
 | With a [squishsum harness](harnesses.md): two valid samples that differ, with no judge or a judge that rejects both | Raises `Squishling::DisagreementError` (an `InvalidOutputError`), carrying both typed `candidates` |
 
 Squishling validates output itself with [json_schemer](https://github.com/davishmcclurg/json_schemer), because
-RubyLLM doesn't, and some providers don't enforce strict mode. When the next attempt is on the same step (same
-model, provider, params, and `forward_rejected:`), the re-ask happens in the same conversation, so the model sees
-what it got wrong. A different step gets a fresh chat with the original input plus the rejected output (truncated to
-4,000 characters) and its errors, so that output is sent to the next step's provider, which may not be the one that
-produced it. Set `forward_rejected: false` on a step to leave both out. `InvalidOutputError` exposes `errors`, `raw`
-(the last response), `attempts`, and `models` (the model tried on each attempt); for a
-[`DisagreementError`](harnesses.md#when-the-harness-fails), `raw` holds both candidates, `attempts` is `nil`, and
-`models` has one entry per role (sample a, sample b, then the judge). With a `logger` configured, every
+adapters don't, and some providers don't enforce strict mode. When the next attempt is on the same step (same
+model, provider, params, `forward_rejected:`, and `retry_temperature:`), the re-ask happens in the same conversation, so the model sees
+what it got wrong. If that re-ask returns the same reply it was just told was invalid, Squishling doesn't spend the
+step's remaining attempts: it moves straight to the next step (or raises `InvalidOutputError`, when this was the
+last), so size `attempts:` for replies that can differ. Set `retry_temperature:` on a step to vary the temperature
+of its same-chat retries. A different step gets a fresh chat with the original input plus the rejected output
+(truncated to 4,000 characters) and its errors, so that output is sent to the next step's provider, which may not be the one that
+produced it. Set `forward_rejected: false` on a step to leave both out. `InvalidOutputError` exposes `errors`,
+`error_details` (the same messages, each tagged with where it came from; see below), `raw`
+(the last response), `attempts`, `models` (the model tried on each attempt), and `finish_reason` (why the provider
+stopped on the last one). For a [`DisagreementError`](harnesses.md#when-the-harness-fails), `raw` holds both
+candidates, `attempts` and `finish_reason` are `nil`, and `models` has one entry per role (sample a, sample b, then
+the judge). With a `logger` configured, every
 escalation is logged as a warning.
 
+### A cutoff or a wrong answer
+
+An empty or half-written reply can mean the model was cut off at the output cap, not that it answered wrongly. The
+provider's stop reason tells them apart: `InvalidOutputError#finish_reason` and `squawk`'s `metadata[:finish_reason]`
+hold it as a Symbol (`:stop`, `:max_tokens`, `:content_filter`, or the provider's own spelling; `nil` when it reported
+none), and `truncated?` is true for `:max_tokens`. A truncated error's message and the log line say so.
+
+```ruby
+rescue Squishling::InvalidOutputError => e
+  e.truncated? # => true: raise max_output_tokens, rather than tuning the prompt or validator
+```
+
+Providers that count thinking against the cap need `max_output_tokens` to leave room for the thinking budget plus the
+answer: `thinking: { budget: 8192 }` with `max_output_tokens: 8192` leaves none, so every reply that uses its full
+budget is cut off before any JSON is written.
+
 ### What ends up in errors and logs
 
 Model output can echo sensitive input, so the raw response is kept in one place: `InvalidOutputError#raw` (plus the opt-in `squawk` hook below).
@@ -37,13 +59,14 @@ Model output can echo sensitive input, so the raw response is kept in one place:
   snippet of the response.
 - JSON Schema errors, which name the schema path and the rule that failed. They never contain the rejected value, but
   they do name an extra key the model added (`object property at /foo is a disallowed additional property`), and
-  that key name is chosen by the model.
+  that key name is chosen by the model. These are the only entries Squishling itself fills with model-chosen text; see
+  [telling the entries apart](#telling-the-entries-apart).
 - Your own `squish_validate` messages, verbatim. If you interpolate result values into them, those values reach
   the message, the logger, and the next attempt's prompt.
 - A fixed message when a response contains a value JSON can't represent (`1e400` parses to `Infinity`, or a string
   with invalid UTF-8).
 - For a failed provider request (`LLMError`, and the `ConfigurationError` for a 400, bad credentials, or a forbidden
-  key), only the error class and HTTP status. The provider's own message can echo request content or part of an API
+  key), only the error class and HTTP status, plus a 400's provider status and code when they are plain tokens. The provider's own message can echo request content or part of an API
   key, so it stays on `cause` (`error.cause.message`) for you to read or log deliberately.
 - At most the first 20 of these errors, plus a count of the rest, so a very large malformed response can't flood
   the retry message or the log.
@@ -51,14 +74,77 @@ Model output can echo sensitive input, so the raw response is kept in one place:
   choice and probability. A chat judge's `reason` is model-written, so it is available as `#reason` but never in the
   message or logs.
 
+#### Telling the entries apart
+
+`InvalidOutputError#errors` is an Array of Strings. `#error_details` is the same list as
+`Squishling::InvalidOutputError::Detail` values (`message`, `source`), so a sink with its own control syntax (a chat
+webhook, HTML, a markdown renderer) can escape only what needs it:
+
+| `source` | Where the message came from | `model_derived?` |
+|---|---|---|
+| `:schema` | A JSON Schema error. It can name a key the model chose. | `true` |
+| `:validator` | Your own [`squish_validate`](#output-checks-squish_validate) message, verbatim. Never flagged, so escape it too if you interpolate result values into it. | `false` |
+| `:parse` | A message about a response or return value that didn't parse (not valid JSON, empty, a value JSON can't represent). | `false` |
+| `:squishling` | Other text Squishling or a [custom harness](harnesses.md#when-the-harness-fails) wrote, such as the "... and N more errors" notice. | `false` |
+
+```ruby
+squish_fallback do |error, **|
+  trusted = error.error_details.reject(&:model_derived?).map(&:message)
+  notify_channel("squishling rejected output: #{trusted.first(2).join('; ')}")
+  # ...
+end
+```
+
 The same applies to anything you log yourself, such as `error.message` in a [fallback](#fallbacks). Use `error.raw`
 only where you are willing to store model output. To send the output of every attempt somewhere on purpose, use
 [`squawk`](#observing-every-attempt-squawk).
 
+## Capping a call's cost (token budgets)
+
+`attempts:` limits retries per step, but a call that fails every attempt still pays for all of them, and with
+reasoning models the failed attempts are often the longest. A token budget caps what one call may use across every
+request it makes, whichever [harness](harnesses.md) it runs:
+
+```ruby
+class TicketTriager
+  include Squishling
+
+  squishling escalation: [{ model: "claude-haiku-4-5", attempts: 2 }, { model: "claude-sonnet-5-5", attempts: 2 }],
+    max_total_output_tokens: 25_000,
+    max_total_input_tokens: 200_000
+end
+```
+
+Before each request, Squishling checks the tokens used so far by the call's earlier requests (the escalation's
+attempts, a squishsum harness's samples, and a judge's). Once either total has reached its limit, it makes no more
+requests and raises `Squishling::BudgetExhaustedError`. It is an `InvalidOutputError` (so
+[`squish_fallback`](#fallbacks) receives it, and existing `rescue InvalidOutputError` code keeps working), and
+`rescue Squishling::BudgetExhaustedError` tells it apart from a plain validation failure:
+
+- `limit` is `:max_total_output_tokens` or `:max_total_input_tokens`, `max` its value, and `used` the tokens spent
+  so far (`{ input_tokens:, output_tokens: }`).
+- `errors` starts with the budget line, followed by the validation errors of the last rejected output, which is in
+  `raw` (`nil` when no output was rejected, e.g. the samples were valid but there was no budget left for the judge);
+  `attempts` and `models` are the requests made.
+- Like any other error message, it carries no model output.
+
+Things to know:
+
+- The first request always runs, and a request already in flight is never cut short, so a call can end above the
+  limit by one request (for the squishsum harnesses, one round of samples).
+- Tokens are the counts the [adapter](adapters.md) reports (`output_tokens` includes a reasoning model's thinking
+  tokens for providers that count them that way). A request that failed reports none, and a provider that reports no
+  usage never exhausts a budget.
+- The limits are positive Integers. Set them with `Squishling.configure` (`default_max_total_output_tokens`,
+  `default_max_total_input_tokens`), `squishling`, `squish :method, ...`, or a [`squish!`](routing.md#handing-off-to-the-llm-with-squish) call.
+  The more specific level wins, limit by limit, and `false` lifts a limit set at a wider level.
+- To limit a single response instead, use the `max_output_tokens` [generation param](configuration.md#generation-params).
+
 ## Observing every attempt (`squawk`)
 
 `squawk` is an opt-in hook for sending what the model returned to an error tracker or tracing tool. It runs after
-every LLM attempt, accepted or rejected, and does nothing unless you set it:
+every LLM attempt, accepted or rejected (unless you narrow it with `squawk_on:`, below), and does nothing unless you
+set it:
 
 ```ruby
 Squishling.configure do |config|
@@ -72,11 +158,29 @@ end
 |---|---|
 | `output` | The raw response: a Hash or String, or `nil` when the provider call itself failed |
 | `error` | `nil` for an accepted attempt. Otherwise the `InvalidOutputError` (with `errors` and `raw`) or `LLMError` that ended the attempt |
-| `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length; under a harness, the sample's or judge's own attempts), `final` (the last attempt), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when RubyLLM reports them), `role` (`nil` for an escalation attempt, `:sample`, or `:judge`), `sample` (`"a"`, `"b"`, ... for a sample's attempt, else `nil`) |
+| `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length; `attempt` jumps past the attempts a repeated reply skips; under a harness, the sample's or judge's own attempts), `final` (the last attempt, or a rejection after which none follows), `repeated` (`true` when the attempt returned the same reply as the one before it, in the same chat, which skips the step's remaining attempts), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when the adapter reports them), `finish_reason` (why the provider stopped, e.g. `:stop` or `:max_tokens`; `nil` when unknown or the call failed), `role` (`nil` for an escalation attempt, `:sample`, or `:judge`), `sample` (`"a"`, `"b"`, ... for a sample's attempt, else `nil`) |
 
 - Set it globally with `config.squawk`, per class with `squishling squawk: ...`, or per method with
   `squish :triage, squawk: ...`. The method's hook wins over the class's, which wins over the configured one;
   `squawk: false` silences an inherited hook. Subclasses and `squish!` calls inherit it.
+- Limit which attempts reach the hook with `squawk_on:`: `:all` (the default), `:rejections` (only attempts that
+  errored), `:retries` (those plus any accepted attempt after the first), or `:final` (the last attempt the escalation
+  allows, accepted or not; `metadata[:final]`). A classifier that passes on the first try nearly every time can post
+  only when something went wrong:
+
+  ```ruby
+  squishling squawk: ->(error:, **) { Tracer.record(error) if error }, squawk_on: :retries
+  ```
+
+  Pass a list to combine conditions; the hook runs when any one holds. `squawk_on: %i[rejections final]` reports
+  every error, plus the last attempt when it is reached and succeeds. A call accepted on an earlier attempt never
+  reaches `:final`. A level's value replaces the one above it; lists are never merged across levels.
+
+  Set it globally with `config.squawk_on`, per class with `squishling squawk_on: ...`, or per method with
+  `squish :triage, squawk_on: ...`. It resolves on its own, method then class then configured, so
+  `config.squawk_on = :retries` filters every hook, including a class's own, unless that class or method overrides it.
+  Under a [harness](harnesses.md), `attempt`, `attempts` and `final` count per sample or judge, so a sample's
+  first-attempt success is skipped even while another sample retries, and `:final` fires once per sample and judge.
 - Any object that responds to `call` works. The hook receives only the keywords it declares, unless it takes `**`,
   so a lambda that wants just `error:` is fine and new metadata fields won't break it.
 - It runs inline, so keep it quick. Exceptions it raises propagate unchanged and fail the call, like any of your own
@@ -120,17 +224,18 @@ end
 | Error | Raised when |
 |---|---|
 | `Squishling::Error` | Base class for everything below. Raised directly for misuse at call time, e.g. `result` or `squish!` called outside a squished method |
-| `Squishling::ConfigurationError` | Missing purpose or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_to_purpose` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (400) |
+| `Squishling::ConfigurationError` | Missing purpose or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_to_purpose` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (400, as `RequestRejectedError`, with `provider_status` and `provider_code`) |
 | `Squishling::InvalidOutputError` | LLM output still invalid (schema or `squish_validate`) after every attempt in the escalation, or a deterministic/fallback return that doesn't match the schema |
 | `Squishling::DisagreementError` | A subclass of `InvalidOutputError`: a [squishsum harness](harnesses.md#when-the-harness-fails)'s samples disagreed and no judge accepted either. `candidates`, `verdict`, and `reason` describe what happened |
-| `Squishling::LLMError` | The provider call failed on the last attempt in the escalation, after RubyLLM's own retries, including context-length errors (`cause` holds the original) |
+| `Squishling::BudgetExhaustedError` | A subclass of `InvalidOutputError`: the call used up its [token budget](#capping-a-calls-cost-token-budgets) before an attempt was accepted. `limit`, `max`, and `used` describe the budget |
+| `Squishling::LLMError` | The provider call failed on the last attempt in the escalation, after the adapter's own retries, including context-length errors (`cause` holds the original, when the adapter has one) |
 
 Errors raised by your own Ruby code are not wrapped.
 
 ## Fallbacks
 
 Use `squish_fallback` to decide what happens when the LLM path fails with `InvalidOutputError` (including a
-`DisagreementError`) or `LLMError`. It's the one fallback for every [harness](harnesses.md).
+`DisagreementError` or `BudgetExhaustedError`) or `LLMError`. It's the one fallback for every [harness](harnesses.md).
 It receives the error plus the method's inputs as keywords, and runs against the instance:
 
 ```ruby

data/docs/harnesses.md

@@ -58,8 +58,8 @@ Both samples run on the escalation's first step, with its model, provider, and p
 - **They're independent.** Each sample has its own chat, and the two requests are sent concurrently on separate
   threads. Only the provider requests run on those threads. Parsing, schema validation, `squish_validate`,
   `compare:`, and the judge all run on the calling thread, so your Squishling callbacks never run concurrently.
-  RubyLLM instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for `chat.ruby_llm`) do see
-  each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
+  With the RubyLLM adapter, instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for
+  `chat.ruby_llm`) do see each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
 - **Each one is retried like an escalation step.** A sample with invalid output is re-asked in its own chat, up to
   the first step's `attempts:`, while a sample that already passed keeps its result. A failed request gets a fresh
   chat. A sample still failing after its attempts fails the call with `InvalidOutputError` or `LLMError`, as
@@ -83,7 +83,7 @@ whose typed result is returned (`squished?` is `true`), or it picks `neither`, w
 The judge is:
 
 1. the `judge:` step, if declared: a model name, or a Hash with `model:`, `provider:`, `params:`, and `attempts:`
-   (as in an escalation step, without `priority:` or `rotate:`), plus `type:`; or
+   (as in an escalation step, without `priority:`, `rotate:`, or `include_when:`), plus `type:`; or
 2. the escalation's next step after the sampled one, with its attempts. Steps tied on `priority:` rotate between calls
    (see [Rotating models](configuration.md#rotating-models-within-a-priority)), so which model that is can vary.
 
@@ -108,8 +108,10 @@ valid verdict raises `InvalidOutputError`. A chat judge's params are the method'
 ### System One judgment models (Jev)
 
 A judge can also be a System One decision model, such as TypeSafe's Jev, via
-[RubyLLM judgments](https://rubyllm.com/judgments/) (RubyLLM 2.1+). Decision models return calibrated
-probabilities rather than text, so they are much faster and cheaper than a chat judge.
+[RubyLLM judgments](https://rubyllm.com/judgments/) (RubyLLM 2.1+). This needs an [adapter](adapters.md) that
+supports judgments, which today means the `:ruby_llm` adapter; on any other, a `type: :judgment` judge raises
+`ConfigurationError` before a request is made (a `type: :chat` judge works with every adapter). Decision models return
+calibrated probabilities rather than text, so they are much faster and cheaper than a chat judge.
 
 ```ruby
 squishling harness: {
@@ -122,8 +124,8 @@ Squishling asks one `choice` question (`a`, `b`, or `neither`), with the judge p
 operation's purpose, input, output schema, and candidates as the judgment input. A pick of `a` or `b` is
 accepted only when its probability is at least `min_confidence:` (default `0.8`). Otherwise, as with `neither`,
 `DisagreementError` is raised with the choice and its probability in its `reason`. A judgment judge sends only its own
-`params:` (as RubyLLM `provider_options:`). `attempts:` retries a failed request. Configure the provider's
-credentials in RubyLLM as usual.
+`params:` (as RubyLLM `provider_options:`). `attempts:` retries a failed request. Judgments are a RubyLLM feature, so
+they need the `:ruby_llm` adapter, with the provider's credentials configured in RubyLLM as usual.
 
 ### The default judge prompt
 
@@ -166,6 +168,10 @@ end
 | `attempts` | `nil` |
 | `models` | The model behind each role: sample `a`, sample `b`, then the judge if one ran (one entry each, not one per attempt) |
 
+A [token budget](failures.md#capping-a-calls-cost-token-budgets) covers every request of the call, the samples
+and the judge's included. When it is spent before the next round of retries or before the judge can run, the call raises
+`BudgetExhaustedError` (also an `InvalidOutputError`, and also sent to the fallback) instead of a `DisagreementError`.
+
 Other failures are unchanged. A sample or judge that never produces valid output raises `InvalidOutputError`, a
 failed request on the last attempt raises `LLMError`, and setup mistakes (including a provider 400 on either
 sample) raise `ConfigurationError`, which is never sent to the fallback.
@@ -256,7 +262,7 @@ the harness is declared, before any request is made, and an unknown, missing, or
 
 | Type | Accepts | `call` receives |
 |---|---|---|
-| `:step` | A model name, or a Hash with `model:`, `provider:`, `params:`, `attempts:`, `forward_rejected:` | A normalized, frozen step; pass it to `run.escalate`, or in the Array given to `run.samples` |
+| `:step` | A model name, or a Hash with `model:`, `provider:`, `params:`, `attempts:`, `forward_rejected:`, `retry_temperature:` | A normalized, frozen step; pass it to `run.escalate`, or in the Array given to `run.samples` |
 | `:proc` | A Proc | The Proc |
 | `:string`, `:symbol`, `:integer`, `:boolean` | A String, a Symbol (or String, converted), an Integer, `true` or `false` | The value |
 
@@ -288,7 +294,7 @@ results, the same type the deterministic path returns.
 | `run.stage(declaration)` | A stage from a model name or a step Hash (a single step is a stage of one attempt). It keeps its own provider, and its `params:` merge over the method's, as a [judge](#the-judge) step does |
 | `run.escalate(*stages)` | Makes each attempt in order until an output passes the schema and `squish_validate`, in the same chat after invalid output and in a fresh one (told about the rejected output) on a new step. With no argument, it runs the escalation: exactly what `:escalation` does |
 | `run.samples(stages)` | Takes an Array of stages (so `run.samples([stage, stage])`, not `run.samples(stage)`; a bare step counts as a stage of one attempt): one sample per stage, each in its own chat and retried like an escalation step, with the requests of every round sent concurrently. Returns the outputs in order. Each stage must be one model's attempts: mixed steps raise `ConfigurationError`. Samples are named `"a"`, `"b"`, ... (up to 26) |
-| `run.judge(judge: nil, instructions: nil)` | Resolves a [judge](#the-judge) now (so a missing one raises `ConfigurationError` before any request) and returns it. `judge:` is a model name, or a [judge Hash](#the-judge) (`model:`, `provider:`, `params:`, `attempts:`, `type:`, `min_confidence:`; no `priority:` or `rotate:`), and `instructions:` a non-blank String or a Proc. `judge.verdict(first, second)` returns a `Squishling::Judge::Verdict` with `choice` (`:a`, `:b`, or `:neither`), `reason`, and `detail`, and `judge.step` is its first attempt. A chat judge's `reason` is model-written, so use `detail` (Squishling's own text, `nil` for a chat judge), not `reason`, in messages and logs. Without `judge:`, the judge is the escalation's second stage |
+| `run.judge(judge: nil, instructions: nil)` | Resolves a [judge](#the-judge) now (so a missing one raises `ConfigurationError` before any request) and returns it. `judge:` is a model name, or a [judge Hash](#the-judge) (`model:`, `provider:`, `params:`, `attempts:`, `type:`, `min_confidence:`; no `priority:`, `rotate:`, or `include_when:`), and `instructions:` a non-blank String or a Proc. `judge.verdict(first, second)` returns a `Squishling::Judge::Verdict` with `choice` (`:a`, `:b`, or `:neither`), `reason`, and `detail`, and `judge.step` is its first attempt. A chat judge's `reason` is model-written, so use `detail` (Squishling's own text, `nil` for a chat judge), not `reason`, in messages and logs. Without `judge:`, the judge is the escalation's second stage |
 | `run.agree?(first, second, compare = nil)` | `compare` (evaluated against the instance with both results and the method's inputs) or, without one, exact equality |
 | `run.log(message)` | A warning to `config.logger`, prefixed with `"[Squishling] Class#method"` |
 
@@ -302,8 +308,10 @@ A harness fails the way the built-ins do, so `squish_fallback` and the error cla
 - **Attempts run out.** `run.escalate` and `run.samples` raise `InvalidOutputError` (invalid output after the last
   attempt) or `LLMError` (a failed request on the last attempt) themselves, with the same attributes as always.
 - **Your rule rejects.** Raise one of those errors yourself. `Squishling::DisagreementError.new(candidates:,
-  models:)` is the one for samples that don't agree, and `Squishling::InvalidOutputError.new(errors: [...], source: "Harness")` for
-  anything else. Both go to [`squish_fallback`](failures.md#fallbacks).
+  models:)` is the one for samples that don't agree, and `Squishling::InvalidOutputError.new(errors: [...],
+  source: "Harness")` for anything else. Both go to [`squish_fallback`](failures.md#fallbacks). A String error is
+  tagged `:squishling` in [`error_details`](failures.md#telling-the-entries-apart); pass a
+  `Squishling::InvalidOutputError::Detail` to tag it yourself.
 - **Setup mistakes** raise `ConfigurationError`, which is never retried and never sent to the fallback.
 - **Your own exceptions** (from `call`, `compare:`, or `squish_validate`) propagate unchanged. Squishling doesn't wrap or
   swallow them.
@@ -313,10 +321,12 @@ available to your code as the results and in `InvalidOutputError#raw`, and to `s
 
 ### Reporting to `squawk`
 
-Every attempt a primitive makes is reported to the [`squawk`](failures.md#observing-every-attempt-squawk) hook, with
-usage, so [token measuring](measuring-tokens.md) works unchanged. `metadata[:role]` says what the attempt was for:
+Every attempt a primitive makes is reported to the [`squawk`](failures.md#observing-every-attempt-squawk) hook
+(unless narrowed with `squawk_on:`), with usage, so [token measuring](measuring-tokens.md) works unchanged. `metadata[:role]` says what the attempt was for:
 `nil` for an escalation attempt, `:sample` (with `metadata[:sample]` set to `"a"`, `"b"`, ...), or `:judge`. `attempt`
-and `attempts` count within that sample, judge, or escalation.
+and `attempts` count within that sample, judge, or escalation. `metadata[:repeated]` is `true` for an attempt that
+sent back the reply it was just rejected for; the primitive then skips the rest of that step's attempts (the next
+step's `attempt` jumps past them, and a sample ends with its `InvalidOutputError`).
 
 ### Threads
 

data/docs/measuring-tokens.md

@@ -6,11 +6,13 @@ first tells you when the second is worth paying.
 ## What Squishling gives you
 
 - `result.squished?` says which path served a call, so you can count LLM calls per class.
-- The [`squawk` hook](failures.md#observing-every-attempt-squawk) runs after every LLM attempt and receives the
+- The [`squawk` hook](failures.md#observing-every-attempt-squawk) runs after every LLM attempt (by default; see `squawk_on` below) and receives the
   method (`label`, e.g. `"ShipmentUpdate#call"`), the model and provider, which attempt it was, and the token usage
-  RubyLLM reported. That is enough to attribute spend to a squishling method without any other tooling.
-- Cost in dollars comes from RubyLLM's model pricing. Multiply the reported tokens by your model's price, or read it
-  from RubyLLM directly (see [OpenTelemetry](#opentelemetry), or RubyLLM's `config.instrumenter`).
+  the adapter reported. That is enough to attribute spend to a squishling method without any other tooling.
+- A [token budget](failures.md#capping-a-calls-cost-token-budgets) (`max_total_output_tokens:`, `max_total_input_tokens:`)
+  caps what a single call may use across all its attempts, so a call that keeps failing can't spend without limit.
+- Cost in dollars isn't reported by Squishling's adapters. Multiply the reported tokens by your model's price, or, with
+  the RubyLLM adapter, read it from RubyLLM directly (see [OpenTelemetry](#opentelemetry), or RubyLLM's `config.instrumenter`).
 
 ## Deciding when to write the code
 
@@ -56,8 +58,8 @@ template each call belongs to; see [Coolhand Labs](coolhand.md).
 
 ### Custom: the `squawk` hook
 
-`config.squawk` runs after every LLM attempt, accepted or rejected, and is handed the method's label, the model,
-and the token usage RubyLLM reported. A lambda can take just the keywords it needs:
+`config.squawk` runs after every LLM attempt, accepted or rejected (unless you narrow it with `squawk_on:`, which makes token totals undercount, so keep `:all` for measuring), and is handed the method's label, the model,
+and the token usage the adapter reported. A lambda can take just the keywords it needs:
 
 ```ruby
 Squishling.configure do |config|

data/docs/routing.md

@@ -61,6 +61,9 @@ This is the workflow from [Elastic Software](https://everythingengineer.substack
 
 Callers never change, because both paths return the same result class.
 
+To keep a particular model off certain inputs (a client that needs a BAA, say), give its escalation step an
+`include_when:` Proc; see [Excluding a model per call](configuration.md#excluding-a-model-per-call).
+
 ## Entry points
 
 Squish methods other than `call` with `squish`, optionally overriding the class-level settings per method:
@@ -91,6 +94,7 @@ end
 - `output_schema:` (or a schema block)
 - `model:` or `escalation:`, `provider:`, and `params:` (generation params; see [Configuration](configuration.md))
 - `harness:` (see [Harnesses](harnesses.md), including your own)
+- `max_total_output_tokens:` and `max_total_input_tokens:` (see [Capping a call's cost](failures.md#capping-a-calls-cost-token-budgets))
 - `when:`, a predicate proc
 - `validate:` and `fallback:` (see [Failure handling](failures.md))
 
@@ -134,6 +138,7 @@ end
 | `purpose:` | Replaces the purpose |
 | `model:` or `escalation:`, `provider:`, `params:` | E.g. send this call to a stronger model, or a whole [escalation](configuration.md#models-and-escalation), when Ruby fails. A `provider:` needs a `model:` or `escalation:`; `params:` merge key by key over the declared ones. |
 | `harness:` | Replaces the declared [harness](harnesses.md), e.g. `harness: :judged_squishsum` to double-check a hand-off |
+| `max_total_output_tokens:`, `max_total_input_tokens:` | Cap what this call may spend, e.g. a tighter budget for a hand-off; `false` lifts a declared cap. See [Capping a call's cost](failures.md#capping-a-calls-cost-token-budgets). |
 
 - **The output schema can't be overridden.** The call still returns the method's result type.
 - **Failures** go through the normal LLM path: a declared `squish_fallback` is used, otherwise

data/docs/schemas.md

@@ -49,8 +49,10 @@ end
 
 ## Strict schemas only
 
-Squishling only supports **strict** output schemas: it always asks RubyLLM for strict structured output, and a
-schema declaring `strict: false` raises `Squishling::ConfigurationError`. RubyLLM 2.0 on its own would send a
+Squishling only supports **strict** output schemas: it always asks the adapter for strict structured output (the
+OpenAI-compatible adapter's `structured_output: :json_object` and `:none` modes are the explicit exception, for servers
+that lack it, where the schema goes into the prompt instead), and a schema declaring `strict: false` raises
+`Squishling::ConfigurationError`. Squishling validates every answer against the schema either way. RubyLLM 2.0 on its own would send a
 schema with optional properties non-strict; Squishling always sets the flag explicitly, so it stays strict.
 
 With raw hashes, follow your provider's strict-mode rules (e.g. list every property in `required` and set
@@ -75,6 +77,13 @@ r.to_h                           # deep Hash with symbol keys
 r.squished?                      # => true when it came from the LLM
 ```
 
+A property can't share a name with a method the result depends on, because the `Data` member would replace it.
+Squishling raises `ConfigurationError` when the schema is built (before any request), naming the clashing
+fields; rename them (`class` to `ticket_class`). The reserved names are `class`, `is_a?`, `kind_of?`, `hash`, `inspect`, `to_s`,
+`to_h`, `members`, `with`, `dup`, `send`, `public_send`, `eql?`, `==`, `[]`, `squished?`, `object_id`,
+`deconstruct_keys`, and a few Ruby and Squishling internals (`initialize*`, `__send__`, `squishling_squished`).
+Other names, such as `method`, `format`, or `type`, work as usual. This applies to nested objects and array items too.
+
 On the deterministic path, return a `Hash` or build the result with `result(...)`. Either way it's validated
 against the schema, and so is anything else you return (`nil`, a string, another schema's result), which raises
 `Squishling::InvalidOutputError` unless it matches.

data/docs/testing.md

@@ -0,0 +1,183 @@
+# Testing
+
+`require "squishling/testing"` adds test support: a scripted [adapter](adapters.md) that answers in place of a
+provider and records what was sent to it, stand-ins for the callers of squished methods, replies derived from an output
+schema, and assertions. It is opt-in (`require "squishling"` doesn't load it), needs no test framework or mocking
+library, and works with RSpec (without rspec-mocks), Minitest, and Mocha alike. It makes no network request.
+
+```ruby
+# Minitest
+include Squishling::Testing::Minitest
+
+# RSpec
+RSpec.configure { |config| config.include Squishling::Testing::RSpec }
+```
+
+Both give your tests the helpers below and put back, after each test, whatever the helpers changed
+(`Squishling::Testing.reset!`). They touch only the adapter, observer, and stubs they installed (and only while they
+still hold what the helpers set), so the rest of your `Squishling.configure` survives.
+
+The helpers change global state (`Squishling.config` and the stubbed classes), so tests that use them must not run
+in parallel threads.
+
+## Scripting the model
+
+```ruby
+fake = fake_squishling do |llm|
+  llm.step("gemini-2.5-flash").reply({ regex: "[bad(" }, { regex: "Run.*" })  # in order; the last one repeats
+  llm.step("gemini-2.5-pro").fail(:rate_limit)
+  llm.step(:any).reply({ regex: "fallback.*" })                               # anything no rule above claims
+end
+
+result = RegexInferrer.call(sample_turns: turns)
+
+assert_equal "Run.*", result.regex
+assert_equal 2, fake.requests.size
+assert_sent_arguments fake, { sample_turns: turns }       # what left the process
+assert_includes fake.requests.last.errors, "regex is not a valid Ruby regex"   # the re-ask carried them
+```
+
+A reply is one of:
+
+| Reply | The model… |
+|---|---|
+| a `Hash` | answered with this object |
+| a `String` | returned this text (malformed or fenced JSON, truncated output) |
+| `nil` | returned nothing (a refusal or a cut-off) |
+| `Squishling::Testing::Reply.new(content, model: "m", usage: { input_tokens: 10, output_tokens: 3 })` | answered, and the provider reported this model and these token counts (`squawk`'s `metadata[:usage]`, the observer's `Usage`) |
+| an `Exception` | made the request raise it |
+| `.fail(intent)` | the provider failed: see below |
+
+`reply` takes several, `.fail` adds one, and `.once` makes running out of replies an error instead of repeating the last.
+
+**Provider failures by intent.** You don't need to know which SDK error maps to which Squishling error:
+
+| Intent | Raises |
+|---|---|
+| `:rate_limit`, `:server_error`, `:overloaded`, `:timeout`, `:context_length` | `LLMError` (the next attempt is tried) |
+| `:bad_request`, `:unauthorized`, `:forbidden` | `ConfigurationError` (never retried, never given to a fallback) |
+
+**Selecting what a rule answers.** `step(model)`, `step(provider: :vertexai)`, and `step(:any)` answer escalation
+attempts; `attempt: 2` narrows a step rule to one attempt of the escalation. Rules are matched most specific first
+(a sample or the judge, then the rule naming the most of model, provider and attempt, then `step(:any)`), and by
+declaration order within that.
+Scripts are keyed by model, so a rotated tier (`priority:`) gets the same answers whichever model goes first. Which
+model goes first still depends on how many calls the class has had, so don't assert the order of `fake.models` for a
+rotated tier unless the class is new to the test (or pin the tier with `rotate: false`).
+A request no rule covers raises `Squishling::Testing::UnscriptedRequestError` naming the squished method; it isn't
+a `Squishling::Error`, so `squish_fallback` can't hide it.
+
+### Harnesses
+
+```ruby
+fake_squishling do |llm|
+  llm.sample("a").reply(valid_reply_for(Triager))
+  llm.sample("b").reply(valid_reply_for(Triager, team: "billing"))
+  llm.judge.verdict(:a, reason: "a fits")                    # a chat judge
+  llm.judge.judgment(:b, confidence: 0.92)                   # a System One judge (type: :judgment, min_confidence:)
+  llm.judge.fail_judgment(:timeout)
+end
+```
+
+Each sample has its own script, so the order its worker thread happens to run in can't change what it gets. The
+scripts, the request log, and the conversation tracking are taken under one lock. `DisagreementError#candidates`
+and `#verdict` are the real ones; rescue them as you would in production.
+
+## Inspecting what was sent
+
+`fake.requests` is every provider request, in order, as `Squishling::Testing::Request`:
+
+| | |
+|---|---|
+| `arguments`, `context` | the method's arguments and the `squish_context` / `squish!` context the model was sent |
+| `system_prompt`, `output_schema` | the purpose (with `append_to_purpose` sections) and the strict schema |
+| `model`, `provider`, `params` | the step it went to |
+| `role`, `sample`, `attempt`, `attempts` | `nil`/`:sample`/`:judge`, `"a"`, and where it is in the escalation |
+| `conversation` | the chat it belongs to |
+| `errors` | the validation messages a retry or escalation carried back to the model |
+| `forwarded_rejected` | the rejected output forwarded to the next step (`nil` with `forward_rejected: false`) |
+| `candidates` | for a judge, the two samples |
+| `messages`, `reply`, `failed?` | the conversation as sent, and the `Adapter::Response` (or error) that came back |
+
+`fake.conversations` groups the requests by chat: a retry on the same step shares one, and an escalation to a new
+step starts a new one. `fake.requests_for(sample: "a")`, `fake.models`, and `fake.arguments` are shortcuts.
+
+## Asserting the LLM was not called
+
+```ruby
+assert_no_llm_call { RegexInferrer.call(sample_turns: easy_turns) }   # then: refute result.squished?
+assert_llm_calls fake, 2
+```
+
+`assert_no_llm_call` fails even if the code under test rescued the error that stopped the call.
+
+**A safety net for the whole suite.** In a suite-wide `before` hook, `forbid_provider_calls!` makes any request that
+reaches the provider raise `ProviderCallError` naming the squished method (`"RegexInferrer#call reached the provider…"`),
+so a missing script fails at the call, not far from it. Installing a fake replaces it.
+
+## Stubbing a squished class for its callers
+
+Callers of a squished class shouldn't run Squishling at all. These make `Klass.call` return a real, schema-validated
+typed result, or raise the real error class, without `allow(...).to receive`:
+
+```ruby
+stub_squishling_result(PatternGenerator, regex: "x")                          # squished? is true
+stub_squishling_result(PatternGenerator, { regex: "x" }, squished: false)     # as the Ruby path would return it
+stub_squishling_result(PatternGenerator, regex: "x") { ... }                  # for the block only
+
+stub_squishling_error(PatternGenerator, :invalid_output, errors: ["bad"], raw: "{}", attempts: 2, models: %w[a b])
+stub_squishling_error(PatternGenerator, :llm_error)           # also :configuration, :disagreement (candidates:, verdict:, reason:)
+```
+
+Attributes that don't satisfy the method's `output_schema` raise `Squishling::Testing::SetupError` when you stub, so a
+stand-in can't drift from the real result type. Pass `method: :triage` for another squished method. The stubs are
+removed after each test. A stub applies to that class only, not to its subclasses, which keep their own behavior. The
+hook that makes a stub possible stays on the class for the life of the process, so stub the classes of your suite, not
+one per test.
+
+## Running parts of a method without a model
+
+```ruby
+squishling_validate(Klass, { "regex" => "[bad(" }, text: "x")   # => ["regex is not a valid Ruby regex"]
+squishling_fallback_for(Klass, Squishling::LLMError.new("x"), text: "x")   # the typed fallback result
+squishling_preview(Klass, text: "x")
+# => { system_prompt:, user_message:, output_schema:, step: }   # for snapshot tests; makes no request
+```
+
+Arguments are keywords, or `args: [...]` for positional ones; `receiver:` supplies an instance for a class whose
+constructor takes arguments; `method:` names a squished method other than `call`. (A squished method with a parameter
+called `args`, `receiver`, or `method` can't be given it as a keyword here.) `squishling_preview` doesn't advance model rotation.
+
+## Replies derived from the schema
+
+```ruby
+valid_reply_for(Triager)                                  # => { "priority" => "example", "team" => "example" }
+valid_reply_for(Triager, team: "billing")                 # overrides
+invalid_reply_for(Triager, violating: :score)             # out of its range, enum, or type
+invalid_reply_for(Triager, violating: :score, how: :missing)   # :enum, :range, :type, :null, :missing
+```
+
+Both accept a squished class, a `Squishling::Schema`, or a schema Hash, and are checked against the schema before they
+are returned, so a test that doesn't care about the content stays valid when the schema changes. A schema it can't
+satisfy on its own (a `pattern`, an unsatisfiable conditional) raises `SetupError` and asks for overrides.
+
+## `squawk` and observers
+
+```ruby
+capture = with_squawk(capture_squawk) { Klass.call(text: "x") }
+capture.calls.map { |call| call.metadata[:attempt] }    # output:, error:, and metadata: of every attempt
+```
+
+`with_squawk` swaps `config.squawk` for the block and puts the previous hook back, so an app-wide hook that enqueues
+a job isn't run in tests. A `squawk:` set on a class or method still takes precedence over the configured one.
+
+An observer you configured *before* `fake_squishling` keeps working while the fake is installed (it is chained behind
+the fake's own). Configure it first: one assigned afterwards replaces the fake's, and then sample and judge scripts
+can't tell which request is which.
+
+## How it works
+
+`fake_squishling` assigns a `Squishling::Testing::Adapter` (a `Squishling::Adapter::Base`) as `config.adapter`, and an
+observer that tells it which role, sample, and attempt each request is. Because it sits behind the same interface as
+the real adapters, it needs no knowledge of RubyLLM or any provider, and it is covered by Squishling's own specs.
+Call `Squishling::Testing.install` and `.reset!` yourself if you use a test framework other than the two above.