squishling 0.4.0 → 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (70) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +149 -0
  3. data/README.md +26 -10
  4. data/docs/adapters.md +257 -0
  5. data/docs/configuration.md +54 -19
  6. data/docs/failures.md +170 -22
  7. data/docs/harnesses.md +39 -15
  8. data/docs/measuring-tokens.md +12 -6
  9. data/docs/observers.md +3 -3
  10. data/docs/routing.md +14 -3
  11. data/docs/schemas.md +24 -2
  12. data/docs/testing.md +187 -0
  13. data/docs/verifying.md +86 -0
  14. data/lib/squishling/adapter/error_body.rb +54 -0
  15. data/lib/squishling/adapter.rb +328 -0
  16. data/lib/squishling/adapters/openai_compatible.rb +311 -0
  17. data/lib/squishling/adapters/openai_options.rb +114 -0
  18. data/lib/squishling/adapters/openai_transport.rb +123 -0
  19. data/lib/squishling/adapters/ruby_llm.rb +229 -0
  20. data/lib/squishling/budget.rb +71 -0
  21. data/lib/squishling/class_methods.rb +43 -7
  22. data/lib/squishling/class_registry.rb +63 -0
  23. data/lib/squishling/configuration.rb +58 -6
  24. data/lib/squishling/definition.rb +90 -16
  25. data/lib/squishling/errors.rb +94 -7
  26. data/lib/squishling/harness/judged_squishsum.rb +2 -2
  27. data/lib/squishling/harness/option_types.rb +2 -1
  28. data/lib/squishling/harness/registry.rb +8 -36
  29. data/lib/squishling/invoker.rb +44 -14
  30. data/lib/squishling/judge.rb +16 -3
  31. data/lib/squishling/llm_client.rb +66 -132
  32. data/lib/squishling/model_path.rb +60 -13
  33. data/lib/squishling/named_registry.rb +64 -0
  34. data/lib/squishling/observer.rb +13 -4
  35. data/lib/squishling/output_check.rb +13 -9
  36. data/lib/squishling/params.rb +7 -24
  37. data/lib/squishling/railtie.rb +10 -0
  38. data/lib/squishling/result.rb +50 -16
  39. data/lib/squishling/rotation.rb +5 -0
  40. data/lib/squishling/router.rb +3 -2
  41. data/lib/squishling/rspec.rb +18 -0
  42. data/lib/squishling/run/messages.rb +40 -3
  43. data/lib/squishling/run/rejections.rb +43 -0
  44. data/lib/squishling/run/reports.rb +63 -0
  45. data/lib/squishling/run.rb +60 -53
  46. data/lib/squishling/schema.rb +50 -9
  47. data/lib/squishling/secret.rb +45 -0
  48. data/lib/squishling/squawk.rb +37 -4
  49. data/lib/squishling/tasks/squishling.rake +18 -0
  50. data/lib/squishling/testing/adapter.rb +194 -0
  51. data/lib/squishling/testing/checks.rb +39 -0
  52. data/lib/squishling/testing/direct.rb +41 -0
  53. data/lib/squishling/testing/errors.rb +18 -0
  54. data/lib/squishling/testing/fixtures.rb +206 -0
  55. data/lib/squishling/testing/helpers.rb +71 -0
  56. data/lib/squishling/testing/minitest.rb +32 -0
  57. data/lib/squishling/testing/observer.rb +31 -0
  58. data/lib/squishling/testing/reply.rb +60 -0
  59. data/lib/squishling/testing/request.rb +75 -0
  60. data/lib/squishling/testing/rspec.rb +30 -0
  61. data/lib/squishling/testing/script.rb +189 -0
  62. data/lib/squishling/testing/state.rb +74 -0
  63. data/lib/squishling/testing/stubs.rb +112 -0
  64. data/lib/squishling/testing.rb +126 -0
  65. data/lib/squishling/verification.rb +39 -0
  66. data/lib/squishling/verifier.rb +179 -0
  67. data/lib/squishling/version.rb +1 -1
  68. data/lib/squishling/wrapper.rb +19 -1
  69. data/lib/squishling.rb +60 -4
  70. metadata +43 -21
@@ -1,7 +1,9 @@
1
1
  # Configuration
2
2
 
3
- Squishling sends requests through [RubyLLM](https://rubyllm.com), so configure provider API keys there first
4
- (OpenAI, Anthropic Claude, Google Gemini, AWS Bedrock, OpenRouter, and any other provider RubyLLM supports):
3
+ Squishling sends requests through an [adapter](adapters.md): [RubyLLM](https://rubyllm.com) (OpenAI, Anthropic
4
+ Claude, Google Gemini, AWS Bedrock, OpenRouter, and any other provider RubyLLM supports), any OpenAI-compatible
5
+ endpoint, or your own. With none configured it uses RubyLLM when the `ruby_llm` gem is in your bundle. For RubyLLM,
6
+ configure provider API keys there first (`require "ruby_llm"` first if your app doesn't use `Bundler.require`):
5
7
 
6
8
  ```ruby
7
9
  RubyLLM.configure do |config|
@@ -14,6 +16,7 @@ Then configure Squishling:
14
16
 
15
17
  ```ruby
16
18
  Squishling.configure do |config|
19
+ config.adapter = :ruby_llm # or :openai_compatible, or your own: see Adapters
17
20
  config.default_model = "claude-sonnet-5-5" # or default_escalation, below
18
21
  config.default_provider = nil
19
22
  config.default_params = { temperature: 0 }
@@ -24,17 +27,20 @@ end
24
27
 
25
28
  | Option | Default | Description |
26
29
  |---|---|---|
27
- | `default_model` | `nil` | The universal model (a single attempt) for every squishling class that doesn't declare one. `nil` uses RubyLLM's `default_model`. |
30
+ | `adapter` | `nil` (RubyLLM, when its gem is available) | How requests reach a model: `:ruby_llm`, `:openai_compatible`, a [registered](adapters.md#writing-your-own) name, a Hash with `type:` and that adapter's options, or an adapter instance. See [Adapters](adapters.md). |
31
+ | `default_model` | `nil` | The universal model (a single attempt) for every squishling class that doesn't declare one. `nil` uses the adapter's own default (RubyLLM's `default_model`; the OpenAI-compatible adapter's `default_model:` option). |
28
32
  | `default_escalation` | `nil` | The universal [escalation](#models-and-escalation): models tried in order. Assigning it clears `default_model`, and vice versa. |
29
- | `default_provider` | `nil` | The provider for `default_model`/`default_escalation` steps that don't name one. Only needed for models missing from RubyLLM's registry (see below). |
33
+ | `default_provider` | `nil` | The provider for `default_model`/`default_escalation` steps that don't name one. What it means depends on the adapter: RubyLLM needs it only for models missing from its registry (see below); the OpenAI-compatible adapter ignores it. |
30
34
  | `default_params` | `{}` | Generation params for every call (temperature, reasoning effort, top_p, …), overridable per class, per method, and per escalation step. See [Generation params](#generation-params). |
31
- | `squawk` | `nil` | A callable run after every LLM attempt with the raw output, for sending it to an error tracker or tracing tool. Also settable per class and per method. See [Observing every attempt](failures.md#observing-every-attempt-squawk). |
35
+ | `default_max_total_output_tokens`, `default_max_total_input_tokens` | `nil` (no limit) | The most output (or input) tokens one call may use across all its requests; once reached, the call raises `BudgetExhaustedError` instead of making another attempt. Overridable per class (`squishling max_total_output_tokens: ...`), per method, and per `squish!` call. See [Capping a call's cost](failures.md#capping-a-calls-cost-token-budgets). |
36
+ | `squawk` | `nil` | A callable run after every LLM attempt (see `squawk_on`), and once more when a token budget or a disagreement stops the call, with the raw output, for sending it to an error tracker or tracing tool. Also settable per class and per method. See [Observing every attempt](failures.md#observing-every-attempt-squawk). |
37
+ | `squawk_on` | `:all` | Which attempts reach the `squawk` hook: `:all`, `:rejections` (errors only), `:retries` (errors plus any accepted attempt after the first), `:final` (the last attempt the escalation allows), or a list of them (any one holding runs the hook). A call [stopped](failures.md#a-call-that-is-stopped) by a token budget or a disagreement is always reported. Also settable per class and per method. |
32
38
  | `observer` | `nil` | An around-style hook for every provider request: `:coolhand`, any object responding to `around(call)`, or `false` for none. `nil` uses the [Coolhand observer](coolhand.md) when the `coolhand` gem is configured. See [Observers](observers.md). |
33
39
  | `default_harness` | `nil` (`:escalation`) | How every squishling class that doesn't declare its own uses its escalation: `:escalation`, `:squishsum`, `:judged_squishsum`, a [registered custom harness](harnesses.md#custom-harnesses), or a Hash with `type:` and options. See [Harnesses](harnesses.md). |
34
40
  | `logger` | `nil` | Any `Logger`. Debug lines when a call routes to the LLM, warnings on each escalation and when a fallback is used. Never includes the raw model response; see [what ends up in errors and logs](failures.md#what-ends-up-in-errors-and-logs). |
35
41
 
36
- Transport-level retries (rate limits, 5xx, timeouts) are configured on RubyLLM itself
37
- (`RubyLLM.config.max_retries`, `request_timeout`).
42
+ Transport-level retries (rate limits, 5xx, timeouts) belong to the adapter: RubyLLM's own
43
+ (`RubyLLM.config.max_retries`, `request_timeout`), or the OpenAI-compatible adapter's `max_retries:` option.
38
44
 
39
45
  > **`max_retries` was removed.** The escalation now decides how many attempts run, and setting
40
46
  > `config.max_retries` raises `ConfigurationError`. The old default (`max_retries = 1`, two attempts on one model) is
@@ -65,7 +71,9 @@ Each step is a model name, or a Hash with:
65
71
  | `attempts:` | `1` | How many attempts this step gets before escalating to the next one. |
66
72
  | `priority:` | position | An Integer; steps run lowest first. Give every step a `priority:` or none. Steps with the same priority are one tier (see [Rotating models](#rotating-models-within-a-priority)). Gaps are fine. `order:` is the deprecated name for it. |
67
73
  | `rotate:` | `true` | Set `false` on a step to keep its tier in the order written instead of rotating it. |
74
+ | `include_when:` | always | A Proc that leaves this step out of any call where it returns falsy (see [Excluding a model per call](#excluding-a-model-per-call)). |
68
75
  | `provider:` | the level's `provider:` | See [Providers](#providers-and-newly-released-models). |
76
+ | `retry_temperature:` | none | A Numeric `temperature` for this step's retries in the same chat (not its first attempt), to nudge a model that keeps sending the same rejected reply. Don't use it on a step that sets `thinking`, since reasoning models reject `temperature`. |
69
77
  | `forward_rejected:` | `true` | Set `false` to start this step from the original input alone, without the previous step's rejected output and its errors (see below). |
70
78
  | `params:` | `{}` | [Generation params](#generation-params) for this step, merged over the class and method params key by key. |
71
79
 
@@ -107,6 +115,25 @@ squishling escalation: [
107
115
  whole tier in the order written, whatever its other steps say.
108
116
  - Rotation also moves which stages the [`:squishsum` and `:judged_squishsum`](harnesses.md) harnesses sample first.
109
117
 
118
+ ### Excluding a model per call
119
+
120
+ `include_when:` takes a Proc, called like a `squish_when` predicate: against the instance, with the method's inputs as
121
+ keywords. A step whose Proc returns falsy is left out of that call's escalation, for example a model without a BAA:
122
+
123
+ ```ruby
124
+ squishling escalation: [
125
+ { model: "gpt-5", include_when: ->(client:, **) { !client.needs_a_baa_model? } },
126
+ "claude-sonnet-5-5"
127
+ ]
128
+ ```
129
+
130
+ - Steps are filtered before rotation, so a tier rotates only between the models still eligible for the call. A step
131
+ that is excluded takes no part in its tier, so an excluded `rotate: false` step no longer pins it.
132
+ - A call whose steps are all excluded raises `ConfigurationError`. It is never retried or passed to `squish_fallback`.
133
+ - It applies wherever an escalation is declared (class, method, `default_escalation`, or `squish!`), but not to a judge
134
+ or harness step, which are single steps.
135
+ - The Proc runs on the calling thread. An exception it raises propagates unwrapped.
136
+
110
137
  `params:` on a step let one step switch to a reasoning model that rejects sampling params:
111
138
 
112
139
  ```ruby
@@ -123,7 +150,7 @@ The first level that declares a `model:` or an `escalation:` supplies the whole
123
150
  2. per method: `squish :name, model: ...` or `escalation: ...`
124
151
  3. per class: `squishling model: ...` or `escalation: ...` (inherited by subclasses)
125
152
  4. universal: `Squishling.config.default_model` or `default_escalation`
126
- 5. RubyLLM's `default_model` (a single attempt)
153
+ 5. the adapter's default model (RubyLLM's `default_model`, or the OpenAI-compatible adapter's `default_model:`; a single attempt)
127
154
 
128
155
  ```ruby
129
156
  class InvoiceParser
@@ -139,16 +166,20 @@ end
139
166
  Passing both `model:` and `escalation:` in one declaration raises `ConfigurationError`, and so does a list passed as
140
167
  `model:`. Across separate declarations (a reopened class, or two config assignments), the latest one wins.
141
168
 
142
- Consecutive attempts of the same step (same model, provider, params, and `forward_rejected:`, e.g. `attempts: 2`)
169
+ Consecutive attempts of the same step (same model, provider, params, `forward_rejected:`, and `retry_temperature:`, e.g. `attempts: 2`)
143
170
  continue one conversation. Moving to a different step starts a fresh chat with the original input plus the previous
144
171
  output and why it was rejected, so no provider-specific history crosses providers. That rejected output is
145
172
  model-generated and is sent to the next step's provider, even when it is a different one, truncated to 4,000
146
173
  characters. If you don't want a step to see it (say, because the next step is a different provider), set
147
174
  `forward_rejected: false` on that step and it starts from the original input only. Configuration errors (bad
148
- credentials, an unknown model, a request the provider rejects) never move to the next attempt. See [Failure handling](failures.md).
175
+ credentials, an unknown model, a request the provider rejects) never move to the next attempt. A retry that returns
176
+ exactly the reply the step was just told was invalid skips the step's remaining attempts and moves to the next step. See [Failure handling](failures.md).
149
177
 
150
178
  ## Providers and newly released models
151
179
 
180
+ This section is about the RubyLLM adapter; the OpenAI-compatible adapter takes model names as given and ignores
181
+ `provider:`.
182
+
152
183
  RubyLLM looks models up in its bundled registry. To use a model that isn't there yet, such as a newly released
153
184
  OpenAI or Anthropic model, name its provider next to it. Squishling then tells RubyLLM to assume the model exists:
154
185
 
@@ -189,14 +220,17 @@ end
189
220
 
190
221
  | Key | Sent as |
191
222
  |---|---|
192
- | `temperature` | RubyLLM's `with_temperature` |
193
- | `max_output_tokens` | RubyLLM's `with_max_output_tokens` (translated to each provider's own field) |
194
- | `thinking` | RubyLLM's `with_thinking`: `true` (the model's default), `false` (off), or options such as `{ effort: :low }`, `{ budget: 1024 }`, `{ display: :omitted }` |
195
- | anything else (`top_p`, `seed`, `service_tier`, Gemini's `generationConfig`, …) | merged into the provider request as-is via `with_provider_options`, in that provider's own field names |
223
+ | `temperature` | RubyLLM's `with_temperature`; `temperature` in the request body for OpenAI-compatible endpoints |
224
+ | `max_output_tokens` | RubyLLM's `with_max_output_tokens` (translated to each provider's own field); `max_completion_tokens` (or `max_tokens`) for OpenAI-compatible endpoints |
225
+ | `thinking` | RubyLLM's `with_thinking`: `true` (the model's default), `false` (off), or options such as `{ effort: :low }`, `{ budget: 1024 }`, `{ display: :omitted }`. OpenAI-compatible endpoints take only `{ effort: }`, as `reasoning_effort`. |
226
+ | anything else (`top_p`, `seed`, `service_tier`, Gemini's `generationConfig`, …) | merged into the provider request as-is (via `with_provider_options` in RubyLLM), in that provider's own field names |
196
227
 
197
228
  - **No params means provider defaults.** Squishling sends nothing unless you set it. For structured
198
229
  extraction on non-reasoning models (e.g. Anthropic Claude Haiku), a low temperature reduces run-to-run
199
230
  variance.
231
+ - **Thinking counts against `max_output_tokens`** on providers that bill it as output: leave room for the thinking
232
+ budget plus the answer. With `thinking: { budget: 8192 }` and `max_output_tokens: 8192`, a reply that uses its full
233
+ budget is cut off before any JSON, which shows as [`InvalidOutputError#truncated?`](failures.md#a-cutoff-or-a-wrong-answer).
200
234
  - **Reasoning models** (e.g. OpenAI GPT-6 Luna) usually reject sampling params like `temperature` and `top_p`;
201
235
  tune `thinking: { effort: }` instead. Anthropic requires `temperature` to be 1 (or unset) when `thinking` is on.
202
236
  - **Unsetting an inherited key:** set it to `nil` to send nothing for that key, so the provider default applies.
@@ -206,9 +240,9 @@ end
206
240
  squish :triage, model: "gpt-6-luna", provider: :openai, params: { temperature: nil, thinking: { effort: :low } }
207
241
  ```
208
242
 
209
- - Keys that Squishling or RubyLLM control (`model`, `messages`, `input`, `instructions`, `system`, `stream`,
210
- `response_format`, `text`, `output_config`, `tools`, `tool_choice`, `schema`, and the camelCase and plural
211
- spellings other providers use, such as `systemInstruction`, `toolConfig`/`tool_config`, `outputConfig`, `inputs`, …)
243
+ - Keys that Squishling or the adapter control (`model`, `messages`, `input`, `instructions`, `system`, `stream`,
244
+ `response_format`, `text`, `output_config`, `tools`, `tool_choice`, `functions`, `function_call`, `schema`, and the camelCase and plural
245
+ spellings other providers use, such as `systemInstruction`, `cachedContent`/`cached_content`, `toolConfig`/`tool_config`, `outputConfig`, `inputs`, …)
212
246
  raise `ConfigurationError`, because they would override the model, the conversation, or the strict output format.
213
247
  - **Nested containers** such as Gemini's `generationConfig` (and `generation_config` for Gemini Interactions) and
214
248
  Mistral Conversations' `completion_args` accept ordinary settings, but reject the keys inside them that carry the
@@ -225,7 +259,7 @@ end
225
259
 
226
260
  ## Inheritance
227
261
 
228
- Subclasses inherit the model or escalation and its provider (as a pair), harness, generation params (merged key by key), purpose, output schema,
262
+ Subclasses inherit the model or escalation and its provider (as a pair), harness, generation params (merged key by key), token budgets (limit by limit), purpose, output schema,
229
263
  `squish_when` predicate, `squish_context` names, `squish_validate`, `squish_fallback`, and every `squish` declaration. Overrides in a subclass, including
230
264
  overridden methods, are routed the same way.
231
265
 
@@ -235,5 +269,6 @@ overridden methods, are routed the same way.
235
269
  ## Per-call overrides
236
270
 
237
271
  Inside a squished method, `squish!` sends the call to the LLM with its own `purpose:`,
238
- `append_to_purpose:`, `context:`, `model:` or `escalation:` (with `provider:`), `params:`, and `harness:`. Each layers over the method and class
272
+ `append_to_purpose:`, `context:`, `model:` or `escalation:` (with `provider:`), `params:`, `harness:`, and
273
+ `max_total_output_tokens:`/`max_total_input_tokens:`. Each layers over the method and class
239
274
  settings the same way they layer over each other. See [Handing off to the LLM](routing.md#handing-off-to-the-llm-with-squish).
data/docs/failures.md CHANGED
@@ -8,26 +8,48 @@ A failed attempt moves on to the next one; once the last fails, the error is rai
8
8
 
9
9
  | Failure | What Squishling does |
10
10
  |---|---|
11
- | Rate limit, 5xx, overload, timeout, connection error | RubyLLM retries at the HTTP level first (`RubyLLM.config.max_retries`, default 3). If it still fails, moves to the next attempt in a fresh chat; on the last attempt, raises `Squishling::LLMError` with the original exception as its `cause`. |
12
- | Bad API key, unknown model, missing provider config | Raises `Squishling::ConfigurationError`. Never escalated, never sent to the fallback. |
13
- | Provider rejects the request (400 Bad Request), e.g. an unsupported `temperature` or a schema it won't accept | Raises `ConfigurationError` naming the error class and HTTP status, and the params in use; the provider's own message is on `cause`. Never escalated, never sent to the fallback, so a setup mistake can't be silently covered up on every call. |
14
- | Empty or `nil` response (e.g. a refusal or a max-tokens cutoff) | Moves to the next attempt, then raises `Squishling::InvalidOutputError` |
11
+ | Rate limit, 5xx, overload, timeout, connection error | The [adapter](adapters.md) retries at the HTTP level first (RubyLLM: `RubyLLM.config.max_retries`, default 3; OpenAI-compatible: `max_retries:`, default 2). If it still fails, moves to the next attempt in a fresh chat; on the last attempt, raises `Squishling::LLMError` with a `cause` when the adapter has one: the RubyLLM adapter always does, a [sanitized stand-in](#the-cause-of-a-provider-failure); the OpenAI-compatible adapter only for transport errors (the error itself, which carries no response body), not for HTTP statuses. |
12
+ | Bad API key, billing or quota problem (402), unknown model or endpoint (404), missing provider config | Raises `Squishling::ConfigurationError`. Never escalated, never sent to the fallback. |
13
+ | Provider rejects the request (400 Bad Request, or another 4xx such as 409 or 422), e.g. an unsupported `temperature` or a schema it won't accept | Raises `Squishling::RequestRejectedError` (a `ConfigurationError`) naming the error class, the HTTP status, and the provider's error status/code when its body carries one (`INVALID_ARGUMENT`, `invalid_request_error`). They are also on `error.provider_status` and `error.provider_code` (nil when absent); only plain tokens are kept, never the provider's message. The message points at a param only when the provider named it, and otherwise says the params, output schema, or input may be invalid. The provider's own message is not on `cause` either unless you opt in (see [the cause of a provider failure](#the-cause-of-a-provider-failure)). It is never retried, so if you know a given code means "this input was bad", rescue it and decide for yourself. A 400 that reports an overlong input is the exception: the adapter raises an `LLMError` for it (the built-in adapters recognize the structured `context_length_exceeded` code; RubyLLM's own `ContextLengthExceededError` is honored too, and the OpenAI-compatible adapter also recognizes the usual wording of servers that send none), since a bigger model may take it. Which of the two a failure is, is the adapter's call (see [the error contract](adapters.md#the-error-contract)); a wording it doesn't recognize stays a rejected request. Never escalated, never sent to the fallback, so a setup mistake can't be silently covered up on every call. |
14
+ | Empty or `nil` response (e.g. a refusal or a max-tokens cutoff) | Moves to the next attempt, then raises `Squishling::InvalidOutputError`. When the provider reported a max-tokens cutoff, the error says so and [`truncated?`](#a-cutoff-or-a-wrong-answer) is true. |
15
15
  | Malformed or truncated JSON | Moves to the next attempt, then raises `InvalidOutputError`. JSON wrapped in a markdown code fence is accepted. |
16
16
  | JSON that doesn't match the schema (wrong types, missing or extra keys, `null` in a non-`optional` field, a broken [conditional rule](schemas.md#contracts-beyond-the-schema), root not an object) | Moves to the next attempt with the validation errors, then raises `InvalidOutputError` |
17
17
  | Schema-valid output that [`squish_validate`](#output-checks-squish_validate) rejects | Moves to the next attempt with your messages, then raises `InvalidOutputError` |
18
+ | The call has used its [token budget](#capping-a-calls-cost-token-budgets) | Stops before the next attempt and raises `Squishling::BudgetExhaustedError` (an `InvalidOutputError`) |
18
19
  | With a [squishsum harness](harnesses.md): two valid samples that differ, with no judge or a judge that rejects both | Raises `Squishling::DisagreementError` (an `InvalidOutputError`), carrying both typed `candidates` |
19
20
 
20
21
  Squishling validates output itself with [json_schemer](https://github.com/davishmcclurg/json_schemer), because
21
- RubyLLM doesn't, and some providers don't enforce strict mode. When the next attempt is on the same step (same
22
- model, provider, params, and `forward_rejected:`), the re-ask happens in the same conversation, so the model sees
23
- what it got wrong. A different step gets a fresh chat with the original input plus the rejected output (truncated to
24
- 4,000 characters) and its errors, so that output is sent to the next step's provider, which may not be the one that
25
- produced it. Set `forward_rejected: false` on a step to leave both out. `InvalidOutputError` exposes `errors`, `raw`
26
- (the last response), `attempts`, and `models` (the model tried on each attempt); for a
27
- [`DisagreementError`](harnesses.md#when-the-harness-fails), `raw` holds both candidates, `attempts` is `nil`, and
28
- `models` has one entry per role (sample a, sample b, then the judge). With a `logger` configured, every
22
+ adapters don't, and some providers don't enforce strict mode. When the next attempt is on the same step (same
23
+ model, provider, params, `forward_rejected:`, and `retry_temperature:`), the re-ask happens in the same conversation, so the model sees
24
+ what it got wrong. If that re-ask returns the same reply it was just told was invalid, Squishling doesn't spend the
25
+ step's remaining attempts: it moves straight to the next step (or raises `InvalidOutputError`, when this was the
26
+ last), so size `attempts:` for replies that can differ. Set `retry_temperature:` on a step to vary the temperature
27
+ of its same-chat retries. A different step gets a fresh chat with the original input plus the rejected output
28
+ (truncated to 4,000 characters) and its errors, so that output is sent to the next step's provider, which may not be the one that
29
+ produced it. Set `forward_rejected: false` on a step to leave both out. `InvalidOutputError` exposes `errors`,
30
+ `error_details` (the same messages, each tagged with where it came from; see below), `raw`
31
+ (the last response), `attempts`, `models` (the model tried on each attempt), and `finish_reason` (why the provider
32
+ stopped on the last one). For a [`DisagreementError`](harnesses.md#when-the-harness-fails), `raw` holds both
33
+ candidates, `attempts` and `finish_reason` are `nil`, and `models` has one entry per role (sample a, sample b, then
34
+ the judge). With a `logger` configured, every
29
35
  escalation is logged as a warning.
30
36
 
37
+ ### A cutoff or a wrong answer
38
+
39
+ An empty or half-written reply can mean the model was cut off at the output cap, not that it answered wrongly. The
40
+ provider's stop reason tells them apart: `InvalidOutputError#finish_reason` and `squawk`'s `metadata[:finish_reason]`
41
+ hold it as a Symbol (`:stop`, `:max_tokens`, `:content_filter`, or the provider's own spelling; `nil` when it reported
42
+ none), and `truncated?` is true for `:max_tokens`. A truncated error's message and the log line say so.
43
+
44
+ ```ruby
45
+ rescue Squishling::InvalidOutputError => e
46
+ e.truncated? # => true: raise max_output_tokens, rather than tuning the prompt or validator
47
+ ```
48
+
49
+ Providers that count thinking against the cap need `max_output_tokens` to leave room for the thinking budget plus the
50
+ answer: `thinking: { budget: 8192 }` with `max_output_tokens: 8192` leaves none, so every reply that uses its full
51
+ budget is cut off before any JSON is written.
52
+
31
53
  ### What ends up in errors and logs
32
54
 
33
55
  Model output can echo sensitive input, so the raw response is kept in one place: `InvalidOutputError#raw` (plus the opt-in `squawk` hook below).
@@ -37,28 +59,113 @@ Model output can echo sensitive input, so the raw response is kept in one place:
37
59
  snippet of the response.
38
60
  - JSON Schema errors, which name the schema path and the rule that failed. They never contain the rejected value, but
39
61
  they do name an extra key the model added (`object property at /foo is a disallowed additional property`), and
40
- that key name is chosen by the model.
62
+ that key name is chosen by the model. These are the only entries Squishling itself fills with model-chosen text; see
63
+ [telling the entries apart](#telling-the-entries-apart).
41
64
  - Your own `squish_validate` messages, verbatim. If you interpolate result values into them, those values reach
42
65
  the message, the logger, and the next attempt's prompt.
43
66
  - A fixed message when a response contains a value JSON can't represent (`1e400` parses to `Infinity`, or a string
44
67
  with invalid UTF-8).
45
- - For a failed provider request (`LLMError`, and the `ConfigurationError` for a 400, bad credentials, or a forbidden
46
- key), only the error class and HTTP status. The provider's own message can echo request content or part of an API
47
- key, so it stays on `cause` (`error.cause.message`) for you to read or log deliberately.
68
+ - For a failed provider request (`LLMError`, and the `ConfigurationError` for a rejected request, bad credentials, or a forbidden
69
+ key), only the error class and HTTP status, plus a rejection's provider status and code when they are plain tokens. The provider's own message can echo request content or part of an API
70
+ key, and error trackers report the whole `cause` chain, so it is not on `cause` either unless you
71
+ [opt in](#the-cause-of-a-provider-failure).
48
72
  - At most the first 20 of these errors, plus a count of the rest, so a very large malformed response can't flood
49
73
  the retry message or the log.
50
74
  - For a [`DisagreementError`](harnesses.md#when-the-harness-fails), only what Squishling wrote: a `:judgment` judge's
51
75
  choice and probability. A chat judge's `reason` is model-written, so it is available as `#reason` but never in the
52
76
  message or logs.
53
77
 
78
+ #### Telling the entries apart
79
+
80
+ `InvalidOutputError#errors` is an Array of Strings. `#error_details` is the same list as
81
+ `Squishling::InvalidOutputError::Detail` values (`message`, `source`), so a sink with its own control syntax (a chat
82
+ webhook, HTML, a markdown renderer) can escape only what needs it:
83
+
84
+ | `source` | Where the message came from | `model_derived?` |
85
+ |---|---|---|
86
+ | `:schema` | A JSON Schema error. It can name a key the model chose. | `true` |
87
+ | `:validator` | Your own [`squish_validate`](#output-checks-squish_validate) message, verbatim. Never flagged, so escape it too if you interpolate result values into it. | `false` |
88
+ | `:parse` | A message about a response or return value that didn't parse (not valid JSON, empty, a value JSON can't represent). | `false` |
89
+ | `:squishling` | Other text Squishling or a [custom harness](harnesses.md#when-the-harness-fails) wrote, such as the "... and N more errors" notice. | `false` |
90
+
91
+ ```ruby
92
+ squish_fallback do |error, **|
93
+ trusted = error.error_details.reject(&:model_derived?).map(&:message)
94
+ notify_channel("squishling rejected output: #{trusted.first(2).join('; ')}")
95
+ # ...
96
+ end
97
+ ```
98
+
54
99
  The same applies to anything you log yourself, such as `error.message` in a [fallback](#fallbacks). Use `error.raw`
55
100
  only where you are willing to store model output. To send the output of every attempt somewhere on purpose, use
56
101
  [`squawk`](#observing-every-attempt-squawk).
57
102
 
103
+ ### The cause of a provider failure
104
+
105
+ A provider's own exception (RubyLLM's, say) carries the provider's response body in its message and `response`, and the
106
+ body can echo your request. Error trackers such as Sentry report the whole `cause` chain, so an `LLMError`,
107
+ `ConfigurationError`, or `RequestRejectedError` raised for a failed request does not carry that exception as its
108
+ `cause`. The [RubyLLM adapter](adapters.md#ruby_llm) puts a `Squishling::Adapter::ProviderFailure` there instead: its
109
+ message is the original's class and HTTP status (`RubyLLM::RateLimitError, HTTP 429`), and it has the original's
110
+ backtrace, but none of its message, response, or other state. Transport errors that carry no body (the OpenAI-compatible
111
+ adapter's timeouts and connection failures) are passed through as they are.
112
+
113
+ To get the original back for debugging, accept that it reaches your tracker and set `keep_cause: true`:
114
+
115
+ ```ruby
116
+ Squishling.configure do |config|
117
+ config.adapter = { type: :ruby_llm, keep_cause: true }
118
+ end
119
+ ```
120
+
121
+ Then `error.cause` is the `RubyLLM::Error` itself, with its message and `response`.
122
+
123
+ ## Capping a call's cost (token budgets)
124
+
125
+ `attempts:` limits retries per step, but a call that fails every attempt still pays for all of them, and with
126
+ reasoning models the failed attempts are often the longest. A token budget caps what one call may use across every
127
+ request it makes, whichever [harness](harnesses.md) it runs:
128
+
129
+ ```ruby
130
+ class TicketTriager
131
+ include Squishling
132
+
133
+ squishling escalation: [{ model: "claude-haiku-4-5", attempts: 2 }, { model: "claude-sonnet-5-5", attempts: 2 }],
134
+ max_total_output_tokens: 25_000,
135
+ max_total_input_tokens: 200_000
136
+ end
137
+ ```
138
+
139
+ Before each request, Squishling checks the tokens used so far by the call's earlier requests (the escalation's
140
+ attempts, a squishsum harness's samples, and a judge's). Once either total has reached its limit, it makes no more
141
+ requests and raises `Squishling::BudgetExhaustedError`. It is an `InvalidOutputError` (so
142
+ [`squish_fallback`](#fallbacks) receives it, and existing `rescue InvalidOutputError` code keeps working), and
143
+ `rescue Squishling::BudgetExhaustedError` tells it apart from a plain validation failure:
144
+
145
+ - `limit` is `:max_total_output_tokens` or `:max_total_input_tokens`, `max` its value, and `used` the tokens spent
146
+ so far (`{ input_tokens:, output_tokens: }`).
147
+ - `errors` starts with the budget line, followed by the validation errors of the last rejected output, which is in
148
+ `raw` (`nil` when no output was rejected, e.g. the samples were valid but there was no budget left for the judge);
149
+ `attempts` and `models` are the requests made.
150
+ - Like any other error message, it carries no model output.
151
+
152
+ Things to know:
153
+
154
+ - The first request always runs, and a request already in flight is never cut short, so a call can end above the
155
+ limit by one request (for the squishsum harnesses, one round of samples).
156
+ - Tokens are the counts the [adapter](adapters.md) reports (`output_tokens` includes a reasoning model's thinking
157
+ tokens for providers that count them that way). A request that failed reports none, and a provider that reports no
158
+ usage never exhausts a budget.
159
+ - The limits are positive Integers. Set them with `Squishling.configure` (`default_max_total_output_tokens`,
160
+ `default_max_total_input_tokens`), `squishling`, `squish :method, ...`, or a [`squish!`](routing.md#handing-off-to-the-llm-with-squish) call.
161
+ The more specific level wins, limit by limit, and `false` lifts a limit set at a wider level.
162
+ - To limit a single response instead, use the `max_output_tokens` [generation param](configuration.md#generation-params).
163
+
58
164
  ## Observing every attempt (`squawk`)
59
165
 
60
166
  `squawk` is an opt-in hook for sending what the model returned to an error tracker or tracing tool. It runs after
61
- every LLM attempt, accepted or rejected, and does nothing unless you set it:
167
+ every LLM attempt, accepted or rejected (unless you narrow it with `squawk_on:`, below), and once more when a token
168
+ budget or a disagreement stops the call. It does nothing unless you set it:
62
169
 
63
170
  ```ruby
64
171
  Squishling.configure do |config|
@@ -71,12 +178,30 @@ end
71
178
  | Keyword | Value |
72
179
  |---|---|
73
180
  | `output` | The raw response: a Hash or String, or `nil` when the provider call itself failed |
74
- | `error` | `nil` for an accepted attempt. Otherwise the `InvalidOutputError` (with `errors` and `raw`) or `LLMError` that ended the attempt |
75
- | `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length; under a harness, the sample's or judge's own attempts), `final` (the last attempt), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when RubyLLM reports them), `role` (`nil` for an escalation attempt, `:sample`, or `:judge`), `sample` (`"a"`, `"b"`, ... for a sample's attempt, else `nil`) |
181
+ | `error` | `nil` for an accepted attempt. Otherwise the `InvalidOutputError` (with `errors` and `raw`) or `LLMError` that ended the attempt, or the `BudgetExhaustedError` or `DisagreementError` that stopped the call |
182
+ | `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length, or for a [stopped call](#a-call-that-is-stopped) the attempts made; `attempt` jumps past the attempts a repeated reply skips; under a harness, the sample's or judge's own attempts), `final` (the last attempt, or a rejection after which none follows), `repeated` (`true` when the attempt returned the same reply as the one before it, in the same chat, which skips the step's remaining attempts), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when the adapter reports them), `finish_reason` (why the provider stopped, e.g. `:stop` or `:max_tokens`; `nil` when unknown or the call failed), `role` (`nil` for an escalation attempt, `:sample`, or `:judge`), `sample` (`"a"`, `"b"`, ... for a sample's attempt, else `nil`), `stopped` (`nil` for an attempt; `:budget` or `:disagreement` for the [stopped call's report](#a-call-that-is-stopped)) |
76
183
 
77
184
  - Set it globally with `config.squawk`, per class with `squishling squawk: ...`, or per method with
78
185
  `squish :triage, squawk: ...`. The method's hook wins over the class's, which wins over the configured one;
79
186
  `squawk: false` silences an inherited hook. Subclasses and `squish!` calls inherit it.
187
+ - Limit which attempts reach the hook with `squawk_on:`: `:all` (the default), `:rejections` (only attempts that
188
+ errored), `:retries` (those plus any accepted attempt after the first), or `:final` (the last attempt the escalation
189
+ allows, accepted or not; `metadata[:final]`). A classifier that passes on the first try nearly every time can post
190
+ only when something went wrong:
191
+
192
+ ```ruby
193
+ squishling squawk: ->(error:, **) { Tracer.record(error) if error }, squawk_on: :retries
194
+ ```
195
+
196
+ Pass a list to combine conditions; the hook runs when any one holds. `squawk_on: %i[rejections final]` reports
197
+ every error, plus the last attempt when it is reached and succeeds. A call accepted on an earlier attempt never
198
+ reaches `:final`. A level's value replaces the one above it; lists are never merged across levels.
199
+
200
+ Set it globally with `config.squawk_on`, per class with `squishling squawk_on: ...`, or per method with
201
+ `squish :triage, squawk_on: ...`. It resolves on its own, method then class then configured, so
202
+ `config.squawk_on = :retries` filters every hook, including a class's own, unless that class or method overrides it.
203
+ Under a [harness](harnesses.md), `attempt`, `attempts` and `final` count per sample or judge, so a sample's
204
+ first-attempt success is skipped even while another sample retries, and `:final` fires once per sample and judge.
80
205
  - Any object that responds to `call` works. The hook receives only the keywords it declares, unless it takes `**`,
81
206
  so a lambda that wants just `error:` is fine and new metadata fields won't break it.
82
207
  - It runs inline, so keep it quick. Exceptions it raises propagate unchanged and fail the call, like any of your own
@@ -87,6 +212,28 @@ end
87
212
  - It isn't called for a `ConfigurationError`, for deterministic and fallback returns, or for an attempt where your own
88
213
  `squish_validate` raises (that exception propagates first).
89
214
 
215
+ ### A call that is stopped
216
+
217
+ A call that a [token budget](#capping-a-calls-cost-token-budgets) or a
218
+ [disagreement](harnesses.md#when-the-harness-fails) stops has no attempt of its own to report: the attempt before a
219
+ budget stop was reported as not final, and a disagreement's samples were both accepted. So the call gets one more
220
+ report, whatever `squawk_on:` is set to:
221
+
222
+ - `error` is the `BudgetExhaustedError` or `DisagreementError`, `output` is `nil` (nothing was sent), `metadata[:final]`
223
+ is `true`, and `metadata[:stopped]` is `:budget` or `:disagreement` (it is `nil` for every attempt's report).
224
+ - `attempt` and `attempts` are `error.attempts`, or `0` when that is `nil` (a disagreement, or a budget that ran out
225
+ before a sample or judge made its first attempt); a stop before a retry counts the attempts already made. `model` is
226
+ the last of `error.models`, and `provider`, `params`, `usage`, `role`, and `sample` are `nil`.
227
+ - It is sent once, after the attempts that led to it and before `squish_fallback` runs, for the built-in harnesses and
228
+ for a [custom one](harnesses.md#reporting-to-squawk) that raises either error. A call to another squished method
229
+ from your own code (a `squish_validate`, say) reports its own stop, and the outer call does not report that error
230
+ again. The attempts before it keep `final: false`, so `squawk_on: :final` sees a budget or disagreement ending only
231
+ through this report.
232
+ - Like any attempt's report, it runs your hook inline: an exception the hook raises propagates in place of the
233
+ `BudgetExhaustedError` or `DisagreementError` (which stays on `cause`), so `squish_fallback` does not run.
234
+ - `squawk: false`, or no hook, sends nothing. Ending on the last invalid attempt is not a stop: that attempt already
235
+ carried `final: true`.
236
+
90
237
  ## Output checks (`squish_validate`)
91
238
 
92
239
  The schema covers shape and types. For rules it can't express, such as cross-field arithmetic or a lookup against
@@ -120,17 +267,18 @@ end
120
267
  | Error | Raised when |
121
268
  |---|---|
122
269
  | `Squishling::Error` | Base class for everything below. Raised directly for misuse at call time, e.g. `result` or `squish!` called outside a squished method |
123
- | `Squishling::ConfigurationError` | Missing purpose or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_to_purpose` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (400) |
270
+ | `Squishling::ConfigurationError` | Missing purpose or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_to_purpose` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (a 400 or other rejected 4xx, as `RequestRejectedError`, with `provider_status` and `provider_code`) |
124
271
  | `Squishling::InvalidOutputError` | LLM output still invalid (schema or `squish_validate`) after every attempt in the escalation, or a deterministic/fallback return that doesn't match the schema |
125
272
  | `Squishling::DisagreementError` | A subclass of `InvalidOutputError`: a [squishsum harness](harnesses.md#when-the-harness-fails)'s samples disagreed and no judge accepted either. `candidates`, `verdict`, and `reason` describe what happened |
126
- | `Squishling::LLMError` | The provider call failed on the last attempt in the escalation, after RubyLLM's own retries, including context-length errors (`cause` holds the original) |
273
+ | `Squishling::BudgetExhaustedError` | A subclass of `InvalidOutputError`: the call used up its [token budget](#capping-a-calls-cost-token-budgets) before an attempt was accepted. `limit`, `max`, and `used` describe the budget |
274
+ | `Squishling::LLMError` | The provider call failed on the last attempt in the escalation, after the adapter's own retries, including context-length errors (`cause` holds a [sanitized stand-in](#the-cause-of-a-provider-failure), when the adapter has one) |
127
275
 
128
276
  Errors raised by your own Ruby code are not wrapped.
129
277
 
130
278
  ## Fallbacks
131
279
 
132
280
  Use `squish_fallback` to decide what happens when the LLM path fails with `InvalidOutputError` (including a
133
- `DisagreementError`) or `LLMError`. It's the one fallback for every [harness](harnesses.md).
281
+ `DisagreementError` or `BudgetExhaustedError`) or `LLMError`. It's the one fallback for every [harness](harnesses.md).
134
282
  It receives the error plus the method's inputs as keywords, and runs against the instance:
135
283
 
136
284
  ```ruby
data/docs/harnesses.md CHANGED
@@ -58,8 +58,8 @@ Both samples run on the escalation's first step, with its model, provider, and p
58
58
  - **They're independent.** Each sample has its own chat, and the two requests are sent concurrently on separate
59
59
  threads. Only the provider requests run on those threads. Parsing, schema validation, `squish_validate`,
60
60
  `compare:`, and the judge all run on the calling thread, so your Squishling callbacks never run concurrently.
61
- RubyLLM instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for `chat.ruby_llm`) do see
62
- each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
61
+ With the RubyLLM adapter, instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for
62
+ `chat.ruby_llm`) do see each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
63
63
  - **Each one is retried like an escalation step.** A sample with invalid output is re-asked in its own chat, up to
64
64
  the first step's `attempts:`, while a sample that already passed keeps its result. A failed request gets a fresh
65
65
  chat. A sample still failing after its attempts fails the call with `InvalidOutputError` or `LLMError`, as
@@ -83,7 +83,7 @@ whose typed result is returned (`squished?` is `true`), or it picks `neither`, w
83
83
  The judge is:
84
84
 
85
85
  1. the `judge:` step, if declared: a model name, or a Hash with `model:`, `provider:`, `params:`, and `attempts:`
86
- (as in an escalation step, without `priority:` or `rotate:`), plus `type:`; or
86
+ (as in an escalation step, without `priority:`, `rotate:`, or `include_when:`), plus `type:`; or
87
87
  2. the escalation's next step after the sampled one, with its attempts. Steps tied on `priority:` rotate between calls
88
88
  (see [Rotating models](configuration.md#rotating-models-within-a-priority)), so which model that is can vary.
89
89
 
@@ -91,6 +91,13 @@ A judged harness with neither raises `ConfigurationError` before any sample is r
91
91
  step uses only its own `provider:`; it doesn't inherit the class or method `provider:`, which belongs to their models
92
92
  (a chat judge's params, though, do merge over the method's).
93
93
 
94
+ **What the judge is sent.** The judge's provider receives the operation's purpose, the call's full input (the same
95
+ method arguments and named context the samples saw), and **both candidate outputs**, whatever they hold. None of it is
96
+ capped or filtered, and there is no opt-out short of not using a judged harness. When the judge is a different
97
+ provider than the sampled model (a `judge:` step with its own `provider:`, or an escalation's next step on another
98
+ provider), the input and both outputs go to that provider as well, so only name a judge you would trust with the
99
+ input itself.
100
+
94
101
  ### Chat judges
95
102
 
96
103
  By default the judge is a chat model, asked for a strict `{ "verdict": "a" | "b" | "neither", "reason": "..." }`.
@@ -108,8 +115,10 @@ valid verdict raises `InvalidOutputError`. A chat judge's params are the method'
108
115
  ### System One judgment models (Jev)
109
116
 
110
117
  A judge can also be a System One decision model, such as TypeSafe's Jev, via
111
- [RubyLLM judgments](https://rubyllm.com/judgments/) (RubyLLM 2.1+). Decision models return calibrated
112
- probabilities rather than text, so they are much faster and cheaper than a chat judge.
118
+ [RubyLLM judgments](https://rubyllm.com/judgments/) (RubyLLM 2.1+). This needs an [adapter](adapters.md) that
119
+ supports judgments, which today means the `:ruby_llm` adapter; on any other, a `type: :judgment` judge raises
120
+ `ConfigurationError` before a request is made (a `type: :chat` judge works with every adapter). Decision models return
121
+ calibrated probabilities rather than text, so they are much faster and cheaper than a chat judge.
113
122
 
114
123
  ```ruby
115
124
  squishling harness: {
@@ -122,8 +131,8 @@ Squishling asks one `choice` question (`a`, `b`, or `neither`), with the judge p
122
131
  operation's purpose, input, output schema, and candidates as the judgment input. A pick of `a` or `b` is
123
132
  accepted only when its probability is at least `min_confidence:` (default `0.8`). Otherwise, as with `neither`,
124
133
  `DisagreementError` is raised with the choice and its probability in its `reason`. A judgment judge sends only its own
125
- `params:` (as RubyLLM `provider_options:`). `attempts:` retries a failed request. Configure the provider's
126
- credentials in RubyLLM as usual.
134
+ `params:` (as RubyLLM `provider_options:`). `attempts:` retries a failed request. Judgments are a RubyLLM feature, so
135
+ they need the `:ruby_llm` adapter, with the provider's credentials configured in RubyLLM as usual.
127
136
 
128
137
  ### The default judge prompt
129
138
 
@@ -166,6 +175,10 @@ end
166
175
  | `attempts` | `nil` |
167
176
  | `models` | The model behind each role: sample `a`, sample `b`, then the judge if one ran (one entry each, not one per attempt) |
168
177
 
178
+ A [token budget](failures.md#capping-a-calls-cost-token-budgets) covers every request of the call, the samples
179
+ and the judge's included. When it is spent before the next round of retries or before the judge can run, the call raises
180
+ `BudgetExhaustedError` (also an `InvalidOutputError`, and also sent to the fallback) instead of a `DisagreementError`.
181
+
169
182
  Other failures are unchanged. A sample or judge that never produces valid output raises `InvalidOutputError`, a
170
183
  failed request on the last attempt raises `LLMError`, and setup mistakes (including a provider 400 on either
171
184
  sample) raise `ConfigurationError`, which is never sent to the fallback.
@@ -256,7 +269,7 @@ the harness is declared, before any request is made, and an unknown, missing, or
256
269
 
257
270
  | Type | Accepts | `call` receives |
258
271
  |---|---|---|
259
- | `:step` | A model name, or a Hash with `model:`, `provider:`, `params:`, `attempts:`, `forward_rejected:` | A normalized, frozen step; pass it to `run.escalate`, or in the Array given to `run.samples` |
272
+ | `:step` | A model name, or a Hash with `model:`, `provider:`, `params:`, `attempts:`, `forward_rejected:`, `retry_temperature:` | A normalized, frozen step; pass it to `run.escalate`, or in the Array given to `run.samples` |
260
273
  | `:proc` | A Proc | The Proc |
261
274
  | `:string`, `:symbol`, `:integer`, `:boolean` | A String, a Symbol (or String, converted), an Integer, `true` or `false` | The value |
262
275
 
@@ -288,12 +301,15 @@ results, the same type the deterministic path returns.
288
301
  | `run.stage(declaration)` | A stage from a model name or a step Hash (a single step is a stage of one attempt). It keeps its own provider, and its `params:` merge over the method's, as a [judge](#the-judge) step does |
289
302
  | `run.escalate(*stages)` | Makes each attempt in order until an output passes the schema and `squish_validate`, in the same chat after invalid output and in a fresh one (told about the rejected output) on a new step. With no argument, it runs the escalation: exactly what `:escalation` does |
290
303
  | `run.samples(stages)` | Takes an Array of stages (so `run.samples([stage, stage])`, not `run.samples(stage)`; a bare step counts as a stage of one attempt): one sample per stage, each in its own chat and retried like an escalation step, with the requests of every round sent concurrently. Returns the outputs in order. Each stage must be one model's attempts: mixed steps raise `ConfigurationError`. Samples are named `"a"`, `"b"`, ... (up to 26) |
291
- | `run.judge(judge: nil, instructions: nil)` | Resolves a [judge](#the-judge) now (so a missing one raises `ConfigurationError` before any request) and returns it. `judge:` is a model name, or a [judge Hash](#the-judge) (`model:`, `provider:`, `params:`, `attempts:`, `type:`, `min_confidence:`; no `priority:` or `rotate:`), and `instructions:` a non-blank String or a Proc. `judge.verdict(first, second)` returns a `Squishling::Judge::Verdict` with `choice` (`:a`, `:b`, or `:neither`), `reason`, and `detail`, and `judge.step` is its first attempt. A chat judge's `reason` is model-written, so use `detail` (Squishling's own text, `nil` for a chat judge), not `reason`, in messages and logs. Without `judge:`, the judge is the escalation's second stage |
304
+ | `run.judge(judge: nil, instructions: nil)` | Resolves a [judge](#the-judge) now (so a missing one raises `ConfigurationError` before any request) and returns it. `judge:` is a model name, or a [judge Hash](#the-judge) (`model:`, `provider:`, `params:`, `attempts:`, `type:`, `min_confidence:`; no `priority:`, `rotate:`, or `include_when:`), and `instructions:` a non-blank String or a Proc. `judge.verdict(first, second)` returns a `Squishling::Judge::Verdict` with `choice` (`:a`, `:b`, or `:neither`), `reason`, and `detail`, and `judge.step` is its first attempt. A chat judge's `reason` is model-written, so use `detail` (Squishling's own text, `nil` for a chat judge), not `reason`, in messages and logs. Without `judge:`, the judge is the escalation's second stage |
292
305
  | `run.agree?(first, second, compare = nil)` | `compare` (evaluated against the instance with both results and the method's inputs) or, without one, exact equality |
293
306
  | `run.log(message)` | A warning to `config.logger`, prefixed with `"[Squishling] Class#method"` |
294
307
 
295
308
  `call` returns the accepted result: one a primitive of this run returned. Returning anything else (a Hash, `nil`, a
296
- result you built or copied with `with`) raises `ConfigurationError`, so a harness can't skip validation.
309
+ result you built or copied with `with`) raises `ConfigurationError`, so a harness can't skip validation. The check is
310
+ by object identity, which suits the typed results of an object schema. When the schema isn't an object, a result is a
311
+ bare JSON value, and an immediate such as `1` or `true` counts as produced whatever its source, so there the check can't
312
+ catch a hand-built value.
297
313
 
298
314
  ### When a custom harness fails
299
315
 
@@ -302,8 +318,10 @@ A harness fails the way the built-ins do, so `squish_fallback` and the error cla
302
318
  - **Attempts run out.** `run.escalate` and `run.samples` raise `InvalidOutputError` (invalid output after the last
303
319
  attempt) or `LLMError` (a failed request on the last attempt) themselves, with the same attributes as always.
304
320
  - **Your rule rejects.** Raise one of those errors yourself. `Squishling::DisagreementError.new(candidates:,
305
- models:)` is the one for samples that don't agree, and `Squishling::InvalidOutputError.new(errors: [...], source: "Harness")` for
306
- anything else. Both go to [`squish_fallback`](failures.md#fallbacks).
321
+ models:)` is the one for samples that don't agree, and `Squishling::InvalidOutputError.new(errors: [...],
322
+ source: "Harness")` for anything else. Both go to [`squish_fallback`](failures.md#fallbacks). A String error is
323
+ tagged `:squishling` in [`error_details`](failures.md#telling-the-entries-apart); pass a
324
+ `Squishling::InvalidOutputError::Detail` to tag it yourself.
307
325
  - **Setup mistakes** raise `ConfigurationError`, which is never retried and never sent to the fallback.
308
326
  - **Your own exceptions** (from `call`, `compare:`, or `squish_validate`) propagate unchanged. Squishling doesn't wrap or
309
327
  swallow them.
@@ -313,10 +331,16 @@ available to your code as the results and in `InvalidOutputError#raw`, and to `s
313
331
 
314
332
  ### Reporting to `squawk`
315
333
 
316
- Every attempt a primitive makes is reported to the [`squawk`](failures.md#observing-every-attempt-squawk) hook, with
317
- usage, so [token measuring](measuring-tokens.md) works unchanged. `metadata[:role]` says what the attempt was for:
334
+ Every attempt a primitive makes is reported to the [`squawk`](failures.md#observing-every-attempt-squawk) hook
335
+ (unless narrowed with `squawk_on:`), with usage, so [token measuring](measuring-tokens.md) works unchanged. `metadata[:role]` says what the attempt was for:
318
336
  `nil` for an escalation attempt, `:sample` (with `metadata[:sample]` set to `"a"`, `"b"`, ...), or `:judge`. `attempt`
319
- and `attempts` count within that sample, judge, or escalation.
337
+ and `attempts` count within that sample, judge, or escalation. `metadata[:repeated]` is `true` for an attempt that
338
+ sent back the reply it was just rejected for; the primitive then skips the rest of that step's attempts (the next
339
+ step's `attempt` jumps past them, and a sample ends with its `InvalidOutputError`).
340
+
341
+ A call that ends on the token budget or on a `DisagreementError` (the built-in squishsum harnesses', or one your
342
+ `call` raises) is reported once more, as [a stopped call](failures.md#a-call-that-is-stopped), with `metadata[:stopped]`
343
+ set and no output. Squishling does it around your `call`, so a custom harness needs no code for it.
320
344
 
321
345
  ### Threads
322
346
 
@@ -6,11 +6,13 @@ first tells you when the second is worth paying.
6
6
  ## What Squishling gives you
7
7
 
8
8
  - `result.squished?` says which path served a call, so you can count LLM calls per class.
9
- - The [`squawk` hook](failures.md#observing-every-attempt-squawk) runs after every LLM attempt and receives the
9
+ - The [`squawk` hook](failures.md#observing-every-attempt-squawk) runs after every LLM attempt (by default; see `squawk_on` below) and receives the
10
10
  method (`label`, e.g. `"ShipmentUpdate#call"`), the model and provider, which attempt it was, and the token usage
11
- RubyLLM reported. That is enough to attribute spend to a squishling method without any other tooling.
12
- - Cost in dollars comes from RubyLLM's model pricing. Multiply the reported tokens by your model's price, or read it
13
- from RubyLLM directly (see [OpenTelemetry](#opentelemetry), or RubyLLM's `config.instrumenter`).
11
+ the adapter reported. That is enough to attribute spend to a squishling method without any other tooling.
12
+ - A [token budget](failures.md#capping-a-calls-cost-token-budgets) (`max_total_output_tokens:`, `max_total_input_tokens:`)
13
+ caps what a single call may use across all its attempts, so a call that keeps failing can't spend without limit.
14
+ - Cost in dollars isn't reported by Squishling's adapters. Multiply the reported tokens by your model's price, or, with
15
+ the RubyLLM adapter, read it from RubyLLM directly (see [OpenTelemetry](#opentelemetry), or RubyLLM's `config.instrumenter`).
14
16
 
15
17
  ## Deciding when to write the code
16
18
 
@@ -56,8 +58,8 @@ template each call belongs to; see [Coolhand Labs](coolhand.md).
56
58
 
57
59
  ### Custom: the `squawk` hook
58
60
 
59
- `config.squawk` runs after every LLM attempt, accepted or rejected, and is handed the method's label, the model,
60
- and the token usage RubyLLM reported. A lambda can take just the keywords it needs:
61
+ `config.squawk` runs after every LLM attempt, accepted or rejected (unless you narrow it with `squawk_on:`, which makes token totals undercount, so keep `:all` for measuring), and is handed the method's label, the model,
62
+ and the token usage the adapter reported. A lambda can take just the keywords it needs:
61
63
 
62
64
  ```ruby
63
65
  Squishling.configure do |config|
@@ -70,6 +72,10 @@ Squishling.configure do |config|
70
72
  end
71
73
  ```
72
74
 
75
+ A call that stops on its [token budget](failures.md#capping-a-calls-cost-token-budgets) or a disagreement is also
76
+ reported once, with `metadata[:stopped]` set (`:budget` or `:disagreement`) and no `usage`, so the hook above logs
77
+ it with empty counts; check `metadata[:stopped]` to count it apart from an attempt.
78
+
73
79
  Each line carries the `"Class#method"` that made the call, so grouping by label gives you tokens per squishling. A call
74
80
  that stays in Ruby makes no LLM request and never reaches the hook, so a label that stops appearing is a path that has
75
81
  been hardened. You can also set the hook per class (`squishling squawk: ...`) or per method. See