squishling 0.4.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +149 -0
- data/README.md +26 -10
- data/docs/adapters.md +257 -0
- data/docs/configuration.md +54 -19
- data/docs/failures.md +170 -22
- data/docs/harnesses.md +39 -15
- data/docs/measuring-tokens.md +12 -6
- data/docs/observers.md +3 -3
- data/docs/routing.md +14 -3
- data/docs/schemas.md +24 -2
- data/docs/testing.md +187 -0
- data/docs/verifying.md +86 -0
- data/lib/squishling/adapter/error_body.rb +54 -0
- data/lib/squishling/adapter.rb +328 -0
- data/lib/squishling/adapters/openai_compatible.rb +311 -0
- data/lib/squishling/adapters/openai_options.rb +114 -0
- data/lib/squishling/adapters/openai_transport.rb +123 -0
- data/lib/squishling/adapters/ruby_llm.rb +229 -0
- data/lib/squishling/budget.rb +71 -0
- data/lib/squishling/class_methods.rb +43 -7
- data/lib/squishling/class_registry.rb +63 -0
- data/lib/squishling/configuration.rb +58 -6
- data/lib/squishling/definition.rb +90 -16
- data/lib/squishling/errors.rb +94 -7
- data/lib/squishling/harness/judged_squishsum.rb +2 -2
- data/lib/squishling/harness/option_types.rb +2 -1
- data/lib/squishling/harness/registry.rb +8 -36
- data/lib/squishling/invoker.rb +44 -14
- data/lib/squishling/judge.rb +16 -3
- data/lib/squishling/llm_client.rb +66 -132
- data/lib/squishling/model_path.rb +60 -13
- data/lib/squishling/named_registry.rb +64 -0
- data/lib/squishling/observer.rb +13 -4
- data/lib/squishling/output_check.rb +13 -9
- data/lib/squishling/params.rb +7 -24
- data/lib/squishling/railtie.rb +10 -0
- data/lib/squishling/result.rb +50 -16
- data/lib/squishling/rotation.rb +5 -0
- data/lib/squishling/router.rb +3 -2
- data/lib/squishling/rspec.rb +18 -0
- data/lib/squishling/run/messages.rb +40 -3
- data/lib/squishling/run/rejections.rb +43 -0
- data/lib/squishling/run/reports.rb +63 -0
- data/lib/squishling/run.rb +60 -53
- data/lib/squishling/schema.rb +50 -9
- data/lib/squishling/secret.rb +45 -0
- data/lib/squishling/squawk.rb +37 -4
- data/lib/squishling/tasks/squishling.rake +18 -0
- data/lib/squishling/testing/adapter.rb +194 -0
- data/lib/squishling/testing/checks.rb +39 -0
- data/lib/squishling/testing/direct.rb +41 -0
- data/lib/squishling/testing/errors.rb +18 -0
- data/lib/squishling/testing/fixtures.rb +206 -0
- data/lib/squishling/testing/helpers.rb +71 -0
- data/lib/squishling/testing/minitest.rb +32 -0
- data/lib/squishling/testing/observer.rb +31 -0
- data/lib/squishling/testing/reply.rb +60 -0
- data/lib/squishling/testing/request.rb +75 -0
- data/lib/squishling/testing/rspec.rb +30 -0
- data/lib/squishling/testing/script.rb +189 -0
- data/lib/squishling/testing/state.rb +74 -0
- data/lib/squishling/testing/stubs.rb +112 -0
- data/lib/squishling/testing.rb +126 -0
- data/lib/squishling/verification.rb +39 -0
- data/lib/squishling/verifier.rb +179 -0
- data/lib/squishling/version.rb +1 -1
- data/lib/squishling/wrapper.rb +19 -1
- data/lib/squishling.rb +60 -4
- metadata +43 -21
data/docs/configuration.md
CHANGED
|
@@ -1,7 +1,9 @@
|
|
|
1
1
|
# Configuration
|
|
2
2
|
|
|
3
|
-
Squishling sends requests through [RubyLLM](https://rubyllm.com),
|
|
4
|
-
|
|
3
|
+
Squishling sends requests through an [adapter](adapters.md): [RubyLLM](https://rubyllm.com) (OpenAI, Anthropic
|
|
4
|
+
Claude, Google Gemini, AWS Bedrock, OpenRouter, and any other provider RubyLLM supports), any OpenAI-compatible
|
|
5
|
+
endpoint, or your own. With none configured it uses RubyLLM when the `ruby_llm` gem is in your bundle. For RubyLLM,
|
|
6
|
+
configure provider API keys there first (`require "ruby_llm"` first if your app doesn't use `Bundler.require`):
|
|
5
7
|
|
|
6
8
|
```ruby
|
|
7
9
|
RubyLLM.configure do |config|
|
|
@@ -14,6 +16,7 @@ Then configure Squishling:
|
|
|
14
16
|
|
|
15
17
|
```ruby
|
|
16
18
|
Squishling.configure do |config|
|
|
19
|
+
config.adapter = :ruby_llm # or :openai_compatible, or your own: see Adapters
|
|
17
20
|
config.default_model = "claude-sonnet-5-5" # or default_escalation, below
|
|
18
21
|
config.default_provider = nil
|
|
19
22
|
config.default_params = { temperature: 0 }
|
|
@@ -24,17 +27,20 @@ end
|
|
|
24
27
|
|
|
25
28
|
| Option | Default | Description |
|
|
26
29
|
|---|---|---|
|
|
27
|
-
| `
|
|
30
|
+
| `adapter` | `nil` (RubyLLM, when its gem is available) | How requests reach a model: `:ruby_llm`, `:openai_compatible`, a [registered](adapters.md#writing-your-own) name, a Hash with `type:` and that adapter's options, or an adapter instance. See [Adapters](adapters.md). |
|
|
31
|
+
| `default_model` | `nil` | The universal model (a single attempt) for every squishling class that doesn't declare one. `nil` uses the adapter's own default (RubyLLM's `default_model`; the OpenAI-compatible adapter's `default_model:` option). |
|
|
28
32
|
| `default_escalation` | `nil` | The universal [escalation](#models-and-escalation): models tried in order. Assigning it clears `default_model`, and vice versa. |
|
|
29
|
-
| `default_provider` | `nil` | The provider for `default_model`/`default_escalation` steps that don't name one.
|
|
33
|
+
| `default_provider` | `nil` | The provider for `default_model`/`default_escalation` steps that don't name one. What it means depends on the adapter: RubyLLM needs it only for models missing from its registry (see below); the OpenAI-compatible adapter ignores it. |
|
|
30
34
|
| `default_params` | `{}` | Generation params for every call (temperature, reasoning effort, top_p, …), overridable per class, per method, and per escalation step. See [Generation params](#generation-params). |
|
|
31
|
-
| `
|
|
35
|
+
| `default_max_total_output_tokens`, `default_max_total_input_tokens` | `nil` (no limit) | The most output (or input) tokens one call may use across all its requests; once reached, the call raises `BudgetExhaustedError` instead of making another attempt. Overridable per class (`squishling max_total_output_tokens: ...`), per method, and per `squish!` call. See [Capping a call's cost](failures.md#capping-a-calls-cost-token-budgets). |
|
|
36
|
+
| `squawk` | `nil` | A callable run after every LLM attempt (see `squawk_on`), and once more when a token budget or a disagreement stops the call, with the raw output, for sending it to an error tracker or tracing tool. Also settable per class and per method. See [Observing every attempt](failures.md#observing-every-attempt-squawk). |
|
|
37
|
+
| `squawk_on` | `:all` | Which attempts reach the `squawk` hook: `:all`, `:rejections` (errors only), `:retries` (errors plus any accepted attempt after the first), `:final` (the last attempt the escalation allows), or a list of them (any one holding runs the hook). A call [stopped](failures.md#a-call-that-is-stopped) by a token budget or a disagreement is always reported. Also settable per class and per method. |
|
|
32
38
|
| `observer` | `nil` | An around-style hook for every provider request: `:coolhand`, any object responding to `around(call)`, or `false` for none. `nil` uses the [Coolhand observer](coolhand.md) when the `coolhand` gem is configured. See [Observers](observers.md). |
|
|
33
39
|
| `default_harness` | `nil` (`:escalation`) | How every squishling class that doesn't declare its own uses its escalation: `:escalation`, `:squishsum`, `:judged_squishsum`, a [registered custom harness](harnesses.md#custom-harnesses), or a Hash with `type:` and options. See [Harnesses](harnesses.md). |
|
|
34
40
|
| `logger` | `nil` | Any `Logger`. Debug lines when a call routes to the LLM, warnings on each escalation and when a fallback is used. Never includes the raw model response; see [what ends up in errors and logs](failures.md#what-ends-up-in-errors-and-logs). |
|
|
35
41
|
|
|
36
|
-
Transport-level retries (rate limits, 5xx, timeouts)
|
|
37
|
-
(`RubyLLM.config.max_retries`, `request_timeout`).
|
|
42
|
+
Transport-level retries (rate limits, 5xx, timeouts) belong to the adapter: RubyLLM's own
|
|
43
|
+
(`RubyLLM.config.max_retries`, `request_timeout`), or the OpenAI-compatible adapter's `max_retries:` option.
|
|
38
44
|
|
|
39
45
|
> **`max_retries` was removed.** The escalation now decides how many attempts run, and setting
|
|
40
46
|
> `config.max_retries` raises `ConfigurationError`. The old default (`max_retries = 1`, two attempts on one model) is
|
|
@@ -65,7 +71,9 @@ Each step is a model name, or a Hash with:
|
|
|
65
71
|
| `attempts:` | `1` | How many attempts this step gets before escalating to the next one. |
|
|
66
72
|
| `priority:` | position | An Integer; steps run lowest first. Give every step a `priority:` or none. Steps with the same priority are one tier (see [Rotating models](#rotating-models-within-a-priority)). Gaps are fine. `order:` is the deprecated name for it. |
|
|
67
73
|
| `rotate:` | `true` | Set `false` on a step to keep its tier in the order written instead of rotating it. |
|
|
74
|
+
| `include_when:` | always | A Proc that leaves this step out of any call where it returns falsy (see [Excluding a model per call](#excluding-a-model-per-call)). |
|
|
68
75
|
| `provider:` | the level's `provider:` | See [Providers](#providers-and-newly-released-models). |
|
|
76
|
+
| `retry_temperature:` | none | A Numeric `temperature` for this step's retries in the same chat (not its first attempt), to nudge a model that keeps sending the same rejected reply. Don't use it on a step that sets `thinking`, since reasoning models reject `temperature`. |
|
|
69
77
|
| `forward_rejected:` | `true` | Set `false` to start this step from the original input alone, without the previous step's rejected output and its errors (see below). |
|
|
70
78
|
| `params:` | `{}` | [Generation params](#generation-params) for this step, merged over the class and method params key by key. |
|
|
71
79
|
|
|
@@ -107,6 +115,25 @@ squishling escalation: [
|
|
|
107
115
|
whole tier in the order written, whatever its other steps say.
|
|
108
116
|
- Rotation also moves which stages the [`:squishsum` and `:judged_squishsum`](harnesses.md) harnesses sample first.
|
|
109
117
|
|
|
118
|
+
### Excluding a model per call
|
|
119
|
+
|
|
120
|
+
`include_when:` takes a Proc, called like a `squish_when` predicate: against the instance, with the method's inputs as
|
|
121
|
+
keywords. A step whose Proc returns falsy is left out of that call's escalation, for example a model without a BAA:
|
|
122
|
+
|
|
123
|
+
```ruby
|
|
124
|
+
squishling escalation: [
|
|
125
|
+
{ model: "gpt-5", include_when: ->(client:, **) { !client.needs_a_baa_model? } },
|
|
126
|
+
"claude-sonnet-5-5"
|
|
127
|
+
]
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
- Steps are filtered before rotation, so a tier rotates only between the models still eligible for the call. A step
|
|
131
|
+
that is excluded takes no part in its tier, so an excluded `rotate: false` step no longer pins it.
|
|
132
|
+
- A call whose steps are all excluded raises `ConfigurationError`. It is never retried or passed to `squish_fallback`.
|
|
133
|
+
- It applies wherever an escalation is declared (class, method, `default_escalation`, or `squish!`), but not to a judge
|
|
134
|
+
or harness step, which are single steps.
|
|
135
|
+
- The Proc runs on the calling thread. An exception it raises propagates unwrapped.
|
|
136
|
+
|
|
110
137
|
`params:` on a step let one step switch to a reasoning model that rejects sampling params:
|
|
111
138
|
|
|
112
139
|
```ruby
|
|
@@ -123,7 +150,7 @@ The first level that declares a `model:` or an `escalation:` supplies the whole
|
|
|
123
150
|
2. per method: `squish :name, model: ...` or `escalation: ...`
|
|
124
151
|
3. per class: `squishling model: ...` or `escalation: ...` (inherited by subclasses)
|
|
125
152
|
4. universal: `Squishling.config.default_model` or `default_escalation`
|
|
126
|
-
5. RubyLLM's `default_model`
|
|
153
|
+
5. the adapter's default model (RubyLLM's `default_model`, or the OpenAI-compatible adapter's `default_model:`; a single attempt)
|
|
127
154
|
|
|
128
155
|
```ruby
|
|
129
156
|
class InvoiceParser
|
|
@@ -139,16 +166,20 @@ end
|
|
|
139
166
|
Passing both `model:` and `escalation:` in one declaration raises `ConfigurationError`, and so does a list passed as
|
|
140
167
|
`model:`. Across separate declarations (a reopened class, or two config assignments), the latest one wins.
|
|
141
168
|
|
|
142
|
-
Consecutive attempts of the same step (same model, provider, params, and `
|
|
169
|
+
Consecutive attempts of the same step (same model, provider, params, `forward_rejected:`, and `retry_temperature:`, e.g. `attempts: 2`)
|
|
143
170
|
continue one conversation. Moving to a different step starts a fresh chat with the original input plus the previous
|
|
144
171
|
output and why it was rejected, so no provider-specific history crosses providers. That rejected output is
|
|
145
172
|
model-generated and is sent to the next step's provider, even when it is a different one, truncated to 4,000
|
|
146
173
|
characters. If you don't want a step to see it (say, because the next step is a different provider), set
|
|
147
174
|
`forward_rejected: false` on that step and it starts from the original input only. Configuration errors (bad
|
|
148
|
-
credentials, an unknown model, a request the provider rejects) never move to the next attempt.
|
|
175
|
+
credentials, an unknown model, a request the provider rejects) never move to the next attempt. A retry that returns
|
|
176
|
+
exactly the reply the step was just told was invalid skips the step's remaining attempts and moves to the next step. See [Failure handling](failures.md).
|
|
149
177
|
|
|
150
178
|
## Providers and newly released models
|
|
151
179
|
|
|
180
|
+
This section is about the RubyLLM adapter; the OpenAI-compatible adapter takes model names as given and ignores
|
|
181
|
+
`provider:`.
|
|
182
|
+
|
|
152
183
|
RubyLLM looks models up in its bundled registry. To use a model that isn't there yet, such as a newly released
|
|
153
184
|
OpenAI or Anthropic model, name its provider next to it. Squishling then tells RubyLLM to assume the model exists:
|
|
154
185
|
|
|
@@ -189,14 +220,17 @@ end
|
|
|
189
220
|
|
|
190
221
|
| Key | Sent as |
|
|
191
222
|
|---|---|
|
|
192
|
-
| `temperature` | RubyLLM's `with_temperature` |
|
|
193
|
-
| `max_output_tokens` | RubyLLM's `with_max_output_tokens` (translated to each provider's own field) |
|
|
194
|
-
| `thinking` | RubyLLM's `with_thinking`: `true` (the model's default), `false` (off), or options such as `{ effort: :low }`, `{ budget: 1024 }`, `{ display: :omitted }` |
|
|
195
|
-
| anything else (`top_p`, `seed`, `service_tier`, Gemini's `generationConfig`, …) | merged into the provider request as-is via `with_provider_options
|
|
223
|
+
| `temperature` | RubyLLM's `with_temperature`; `temperature` in the request body for OpenAI-compatible endpoints |
|
|
224
|
+
| `max_output_tokens` | RubyLLM's `with_max_output_tokens` (translated to each provider's own field); `max_completion_tokens` (or `max_tokens`) for OpenAI-compatible endpoints |
|
|
225
|
+
| `thinking` | RubyLLM's `with_thinking`: `true` (the model's default), `false` (off), or options such as `{ effort: :low }`, `{ budget: 1024 }`, `{ display: :omitted }`. OpenAI-compatible endpoints take only `{ effort: }`, as `reasoning_effort`. |
|
|
226
|
+
| anything else (`top_p`, `seed`, `service_tier`, Gemini's `generationConfig`, …) | merged into the provider request as-is (via `with_provider_options` in RubyLLM), in that provider's own field names |
|
|
196
227
|
|
|
197
228
|
- **No params means provider defaults.** Squishling sends nothing unless you set it. For structured
|
|
198
229
|
extraction on non-reasoning models (e.g. Anthropic Claude Haiku), a low temperature reduces run-to-run
|
|
199
230
|
variance.
|
|
231
|
+
- **Thinking counts against `max_output_tokens`** on providers that bill it as output: leave room for the thinking
|
|
232
|
+
budget plus the answer. With `thinking: { budget: 8192 }` and `max_output_tokens: 8192`, a reply that uses its full
|
|
233
|
+
budget is cut off before any JSON, which shows as [`InvalidOutputError#truncated?`](failures.md#a-cutoff-or-a-wrong-answer).
|
|
200
234
|
- **Reasoning models** (e.g. OpenAI GPT-6 Luna) usually reject sampling params like `temperature` and `top_p`;
|
|
201
235
|
tune `thinking: { effort: }` instead. Anthropic requires `temperature` to be 1 (or unset) when `thinking` is on.
|
|
202
236
|
- **Unsetting an inherited key:** set it to `nil` to send nothing for that key, so the provider default applies.
|
|
@@ -206,9 +240,9 @@ end
|
|
|
206
240
|
squish :triage, model: "gpt-6-luna", provider: :openai, params: { temperature: nil, thinking: { effort: :low } }
|
|
207
241
|
```
|
|
208
242
|
|
|
209
|
-
- Keys that Squishling or
|
|
210
|
-
`response_format`, `text`, `output_config`, `tools`, `tool_choice`, `schema`, and the camelCase and plural
|
|
211
|
-
spellings other providers use, such as `systemInstruction`, `toolConfig`/`tool_config`, `outputConfig`, `inputs`, …)
|
|
243
|
+
- Keys that Squishling or the adapter control (`model`, `messages`, `input`, `instructions`, `system`, `stream`,
|
|
244
|
+
`response_format`, `text`, `output_config`, `tools`, `tool_choice`, `functions`, `function_call`, `schema`, and the camelCase and plural
|
|
245
|
+
spellings other providers use, such as `systemInstruction`, `cachedContent`/`cached_content`, `toolConfig`/`tool_config`, `outputConfig`, `inputs`, …)
|
|
212
246
|
raise `ConfigurationError`, because they would override the model, the conversation, or the strict output format.
|
|
213
247
|
- **Nested containers** such as Gemini's `generationConfig` (and `generation_config` for Gemini Interactions) and
|
|
214
248
|
Mistral Conversations' `completion_args` accept ordinary settings, but reject the keys inside them that carry the
|
|
@@ -225,7 +259,7 @@ end
|
|
|
225
259
|
|
|
226
260
|
## Inheritance
|
|
227
261
|
|
|
228
|
-
Subclasses inherit the model or escalation and its provider (as a pair), harness, generation params (merged key by key), purpose, output schema,
|
|
262
|
+
Subclasses inherit the model or escalation and its provider (as a pair), harness, generation params (merged key by key), token budgets (limit by limit), purpose, output schema,
|
|
229
263
|
`squish_when` predicate, `squish_context` names, `squish_validate`, `squish_fallback`, and every `squish` declaration. Overrides in a subclass, including
|
|
230
264
|
overridden methods, are routed the same way.
|
|
231
265
|
|
|
@@ -235,5 +269,6 @@ overridden methods, are routed the same way.
|
|
|
235
269
|
## Per-call overrides
|
|
236
270
|
|
|
237
271
|
Inside a squished method, `squish!` sends the call to the LLM with its own `purpose:`,
|
|
238
|
-
`append_to_purpose:`, `context:`, `model:` or `escalation:` (with `provider:`), `params:`,
|
|
272
|
+
`append_to_purpose:`, `context:`, `model:` or `escalation:` (with `provider:`), `params:`, `harness:`, and
|
|
273
|
+
`max_total_output_tokens:`/`max_total_input_tokens:`. Each layers over the method and class
|
|
239
274
|
settings the same way they layer over each other. See [Handing off to the LLM](routing.md#handing-off-to-the-llm-with-squish).
|
data/docs/failures.md
CHANGED
|
@@ -8,26 +8,48 @@ A failed attempt moves on to the next one; once the last fails, the error is rai
|
|
|
8
8
|
|
|
9
9
|
| Failure | What Squishling does |
|
|
10
10
|
|---|---|
|
|
11
|
-
| Rate limit, 5xx, overload, timeout, connection error |
|
|
12
|
-
| Bad API key, unknown model, missing provider config | Raises `Squishling::ConfigurationError`. Never escalated, never sent to the fallback. |
|
|
13
|
-
| Provider rejects the request (400 Bad Request), e.g. an unsupported `temperature` or a schema it won't accept | Raises `ConfigurationError` naming the error class
|
|
14
|
-
| Empty or `nil` response (e.g. a refusal or a max-tokens cutoff) | Moves to the next attempt, then raises `Squishling::InvalidOutputError` |
|
|
11
|
+
| Rate limit, 5xx, overload, timeout, connection error | The [adapter](adapters.md) retries at the HTTP level first (RubyLLM: `RubyLLM.config.max_retries`, default 3; OpenAI-compatible: `max_retries:`, default 2). If it still fails, moves to the next attempt in a fresh chat; on the last attempt, raises `Squishling::LLMError` with a `cause` when the adapter has one: the RubyLLM adapter always does, a [sanitized stand-in](#the-cause-of-a-provider-failure); the OpenAI-compatible adapter only for transport errors (the error itself, which carries no response body), not for HTTP statuses. |
|
|
12
|
+
| Bad API key, billing or quota problem (402), unknown model or endpoint (404), missing provider config | Raises `Squishling::ConfigurationError`. Never escalated, never sent to the fallback. |
|
|
13
|
+
| Provider rejects the request (400 Bad Request, or another 4xx such as 409 or 422), e.g. an unsupported `temperature` or a schema it won't accept | Raises `Squishling::RequestRejectedError` (a `ConfigurationError`) naming the error class, the HTTP status, and the provider's error status/code when its body carries one (`INVALID_ARGUMENT`, `invalid_request_error`). They are also on `error.provider_status` and `error.provider_code` (nil when absent); only plain tokens are kept, never the provider's message. The message points at a param only when the provider named it, and otherwise says the params, output schema, or input may be invalid. The provider's own message is not on `cause` either unless you opt in (see [the cause of a provider failure](#the-cause-of-a-provider-failure)). It is never retried, so if you know a given code means "this input was bad", rescue it and decide for yourself. A 400 that reports an overlong input is the exception: the adapter raises an `LLMError` for it (the built-in adapters recognize the structured `context_length_exceeded` code; RubyLLM's own `ContextLengthExceededError` is honored too, and the OpenAI-compatible adapter also recognizes the usual wording of servers that send none), since a bigger model may take it. Which of the two a failure is, is the adapter's call (see [the error contract](adapters.md#the-error-contract)); a wording it doesn't recognize stays a rejected request. Never escalated, never sent to the fallback, so a setup mistake can't be silently covered up on every call. |
|
|
14
|
+
| Empty or `nil` response (e.g. a refusal or a max-tokens cutoff) | Moves to the next attempt, then raises `Squishling::InvalidOutputError`. When the provider reported a max-tokens cutoff, the error says so and [`truncated?`](#a-cutoff-or-a-wrong-answer) is true. |
|
|
15
15
|
| Malformed or truncated JSON | Moves to the next attempt, then raises `InvalidOutputError`. JSON wrapped in a markdown code fence is accepted. |
|
|
16
16
|
| JSON that doesn't match the schema (wrong types, missing or extra keys, `null` in a non-`optional` field, a broken [conditional rule](schemas.md#contracts-beyond-the-schema), root not an object) | Moves to the next attempt with the validation errors, then raises `InvalidOutputError` |
|
|
17
17
|
| Schema-valid output that [`squish_validate`](#output-checks-squish_validate) rejects | Moves to the next attempt with your messages, then raises `InvalidOutputError` |
|
|
18
|
+
| The call has used its [token budget](#capping-a-calls-cost-token-budgets) | Stops before the next attempt and raises `Squishling::BudgetExhaustedError` (an `InvalidOutputError`) |
|
|
18
19
|
| With a [squishsum harness](harnesses.md): two valid samples that differ, with no judge or a judge that rejects both | Raises `Squishling::DisagreementError` (an `InvalidOutputError`), carrying both typed `candidates` |
|
|
19
20
|
|
|
20
21
|
Squishling validates output itself with [json_schemer](https://github.com/davishmcclurg/json_schemer), because
|
|
21
|
-
|
|
22
|
-
model, provider, params, and `
|
|
23
|
-
what it got wrong.
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
22
|
+
adapters don't, and some providers don't enforce strict mode. When the next attempt is on the same step (same
|
|
23
|
+
model, provider, params, `forward_rejected:`, and `retry_temperature:`), the re-ask happens in the same conversation, so the model sees
|
|
24
|
+
what it got wrong. If that re-ask returns the same reply it was just told was invalid, Squishling doesn't spend the
|
|
25
|
+
step's remaining attempts: it moves straight to the next step (or raises `InvalidOutputError`, when this was the
|
|
26
|
+
last), so size `attempts:` for replies that can differ. Set `retry_temperature:` on a step to vary the temperature
|
|
27
|
+
of its same-chat retries. A different step gets a fresh chat with the original input plus the rejected output
|
|
28
|
+
(truncated to 4,000 characters) and its errors, so that output is sent to the next step's provider, which may not be the one that
|
|
29
|
+
produced it. Set `forward_rejected: false` on a step to leave both out. `InvalidOutputError` exposes `errors`,
|
|
30
|
+
`error_details` (the same messages, each tagged with where it came from; see below), `raw`
|
|
31
|
+
(the last response), `attempts`, `models` (the model tried on each attempt), and `finish_reason` (why the provider
|
|
32
|
+
stopped on the last one). For a [`DisagreementError`](harnesses.md#when-the-harness-fails), `raw` holds both
|
|
33
|
+
candidates, `attempts` and `finish_reason` are `nil`, and `models` has one entry per role (sample a, sample b, then
|
|
34
|
+
the judge). With a `logger` configured, every
|
|
29
35
|
escalation is logged as a warning.
|
|
30
36
|
|
|
37
|
+
### A cutoff or a wrong answer
|
|
38
|
+
|
|
39
|
+
An empty or half-written reply can mean the model was cut off at the output cap, not that it answered wrongly. The
|
|
40
|
+
provider's stop reason tells them apart: `InvalidOutputError#finish_reason` and `squawk`'s `metadata[:finish_reason]`
|
|
41
|
+
hold it as a Symbol (`:stop`, `:max_tokens`, `:content_filter`, or the provider's own spelling; `nil` when it reported
|
|
42
|
+
none), and `truncated?` is true for `:max_tokens`. A truncated error's message and the log line say so.
|
|
43
|
+
|
|
44
|
+
```ruby
|
|
45
|
+
rescue Squishling::InvalidOutputError => e
|
|
46
|
+
e.truncated? # => true: raise max_output_tokens, rather than tuning the prompt or validator
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Providers that count thinking against the cap need `max_output_tokens` to leave room for the thinking budget plus the
|
|
50
|
+
answer: `thinking: { budget: 8192 }` with `max_output_tokens: 8192` leaves none, so every reply that uses its full
|
|
51
|
+
budget is cut off before any JSON is written.
|
|
52
|
+
|
|
31
53
|
### What ends up in errors and logs
|
|
32
54
|
|
|
33
55
|
Model output can echo sensitive input, so the raw response is kept in one place: `InvalidOutputError#raw` (plus the opt-in `squawk` hook below).
|
|
@@ -37,28 +59,113 @@ Model output can echo sensitive input, so the raw response is kept in one place:
|
|
|
37
59
|
snippet of the response.
|
|
38
60
|
- JSON Schema errors, which name the schema path and the rule that failed. They never contain the rejected value, but
|
|
39
61
|
they do name an extra key the model added (`object property at /foo is a disallowed additional property`), and
|
|
40
|
-
that key name is chosen by the model.
|
|
62
|
+
that key name is chosen by the model. These are the only entries Squishling itself fills with model-chosen text; see
|
|
63
|
+
[telling the entries apart](#telling-the-entries-apart).
|
|
41
64
|
- Your own `squish_validate` messages, verbatim. If you interpolate result values into them, those values reach
|
|
42
65
|
the message, the logger, and the next attempt's prompt.
|
|
43
66
|
- A fixed message when a response contains a value JSON can't represent (`1e400` parses to `Infinity`, or a string
|
|
44
67
|
with invalid UTF-8).
|
|
45
|
-
- For a failed provider request (`LLMError`, and the `ConfigurationError` for a
|
|
46
|
-
key), only the error class and HTTP status. The provider's own message can echo request content or part of an API
|
|
47
|
-
key,
|
|
68
|
+
- For a failed provider request (`LLMError`, and the `ConfigurationError` for a rejected request, bad credentials, or a forbidden
|
|
69
|
+
key), only the error class and HTTP status, plus a rejection's provider status and code when they are plain tokens. The provider's own message can echo request content or part of an API
|
|
70
|
+
key, and error trackers report the whole `cause` chain, so it is not on `cause` either unless you
|
|
71
|
+
[opt in](#the-cause-of-a-provider-failure).
|
|
48
72
|
- At most the first 20 of these errors, plus a count of the rest, so a very large malformed response can't flood
|
|
49
73
|
the retry message or the log.
|
|
50
74
|
- For a [`DisagreementError`](harnesses.md#when-the-harness-fails), only what Squishling wrote: a `:judgment` judge's
|
|
51
75
|
choice and probability. A chat judge's `reason` is model-written, so it is available as `#reason` but never in the
|
|
52
76
|
message or logs.
|
|
53
77
|
|
|
78
|
+
#### Telling the entries apart
|
|
79
|
+
|
|
80
|
+
`InvalidOutputError#errors` is an Array of Strings. `#error_details` is the same list as
|
|
81
|
+
`Squishling::InvalidOutputError::Detail` values (`message`, `source`), so a sink with its own control syntax (a chat
|
|
82
|
+
webhook, HTML, a markdown renderer) can escape only what needs it:
|
|
83
|
+
|
|
84
|
+
| `source` | Where the message came from | `model_derived?` |
|
|
85
|
+
|---|---|---|
|
|
86
|
+
| `:schema` | A JSON Schema error. It can name a key the model chose. | `true` |
|
|
87
|
+
| `:validator` | Your own [`squish_validate`](#output-checks-squish_validate) message, verbatim. Never flagged, so escape it too if you interpolate result values into it. | `false` |
|
|
88
|
+
| `:parse` | A message about a response or return value that didn't parse (not valid JSON, empty, a value JSON can't represent). | `false` |
|
|
89
|
+
| `:squishling` | Other text Squishling or a [custom harness](harnesses.md#when-the-harness-fails) wrote, such as the "... and N more errors" notice. | `false` |
|
|
90
|
+
|
|
91
|
+
```ruby
|
|
92
|
+
squish_fallback do |error, **|
|
|
93
|
+
trusted = error.error_details.reject(&:model_derived?).map(&:message)
|
|
94
|
+
notify_channel("squishling rejected output: #{trusted.first(2).join('; ')}")
|
|
95
|
+
# ...
|
|
96
|
+
end
|
|
97
|
+
```
|
|
98
|
+
|
|
54
99
|
The same applies to anything you log yourself, such as `error.message` in a [fallback](#fallbacks). Use `error.raw`
|
|
55
100
|
only where you are willing to store model output. To send the output of every attempt somewhere on purpose, use
|
|
56
101
|
[`squawk`](#observing-every-attempt-squawk).
|
|
57
102
|
|
|
103
|
+
### The cause of a provider failure
|
|
104
|
+
|
|
105
|
+
A provider's own exception (RubyLLM's, say) carries the provider's response body in its message and `response`, and the
|
|
106
|
+
body can echo your request. Error trackers such as Sentry report the whole `cause` chain, so an `LLMError`,
|
|
107
|
+
`ConfigurationError`, or `RequestRejectedError` raised for a failed request does not carry that exception as its
|
|
108
|
+
`cause`. The [RubyLLM adapter](adapters.md#ruby_llm) puts a `Squishling::Adapter::ProviderFailure` there instead: its
|
|
109
|
+
message is the original's class and HTTP status (`RubyLLM::RateLimitError, HTTP 429`), and it has the original's
|
|
110
|
+
backtrace, but none of its message, response, or other state. Transport errors that carry no body (the OpenAI-compatible
|
|
111
|
+
adapter's timeouts and connection failures) are passed through as they are.
|
|
112
|
+
|
|
113
|
+
To get the original back for debugging, accept that it reaches your tracker and set `keep_cause: true`:
|
|
114
|
+
|
|
115
|
+
```ruby
|
|
116
|
+
Squishling.configure do |config|
|
|
117
|
+
config.adapter = { type: :ruby_llm, keep_cause: true }
|
|
118
|
+
end
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Then `error.cause` is the `RubyLLM::Error` itself, with its message and `response`.
|
|
122
|
+
|
|
123
|
+
## Capping a call's cost (token budgets)
|
|
124
|
+
|
|
125
|
+
`attempts:` limits retries per step, but a call that fails every attempt still pays for all of them, and with
|
|
126
|
+
reasoning models the failed attempts are often the longest. A token budget caps what one call may use across every
|
|
127
|
+
request it makes, whichever [harness](harnesses.md) it runs:
|
|
128
|
+
|
|
129
|
+
```ruby
|
|
130
|
+
class TicketTriager
|
|
131
|
+
include Squishling
|
|
132
|
+
|
|
133
|
+
squishling escalation: [{ model: "claude-haiku-4-5", attempts: 2 }, { model: "claude-sonnet-5-5", attempts: 2 }],
|
|
134
|
+
max_total_output_tokens: 25_000,
|
|
135
|
+
max_total_input_tokens: 200_000
|
|
136
|
+
end
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Before each request, Squishling checks the tokens used so far by the call's earlier requests (the escalation's
|
|
140
|
+
attempts, a squishsum harness's samples, and a judge's). Once either total has reached its limit, it makes no more
|
|
141
|
+
requests and raises `Squishling::BudgetExhaustedError`. It is an `InvalidOutputError` (so
|
|
142
|
+
[`squish_fallback`](#fallbacks) receives it, and existing `rescue InvalidOutputError` code keeps working), and
|
|
143
|
+
`rescue Squishling::BudgetExhaustedError` tells it apart from a plain validation failure:
|
|
144
|
+
|
|
145
|
+
- `limit` is `:max_total_output_tokens` or `:max_total_input_tokens`, `max` its value, and `used` the tokens spent
|
|
146
|
+
so far (`{ input_tokens:, output_tokens: }`).
|
|
147
|
+
- `errors` starts with the budget line, followed by the validation errors of the last rejected output, which is in
|
|
148
|
+
`raw` (`nil` when no output was rejected, e.g. the samples were valid but there was no budget left for the judge);
|
|
149
|
+
`attempts` and `models` are the requests made.
|
|
150
|
+
- Like any other error message, it carries no model output.
|
|
151
|
+
|
|
152
|
+
Things to know:
|
|
153
|
+
|
|
154
|
+
- The first request always runs, and a request already in flight is never cut short, so a call can end above the
|
|
155
|
+
limit by one request (for the squishsum harnesses, one round of samples).
|
|
156
|
+
- Tokens are the counts the [adapter](adapters.md) reports (`output_tokens` includes a reasoning model's thinking
|
|
157
|
+
tokens for providers that count them that way). A request that failed reports none, and a provider that reports no
|
|
158
|
+
usage never exhausts a budget.
|
|
159
|
+
- The limits are positive Integers. Set them with `Squishling.configure` (`default_max_total_output_tokens`,
|
|
160
|
+
`default_max_total_input_tokens`), `squishling`, `squish :method, ...`, or a [`squish!`](routing.md#handing-off-to-the-llm-with-squish) call.
|
|
161
|
+
The more specific level wins, limit by limit, and `false` lifts a limit set at a wider level.
|
|
162
|
+
- To limit a single response instead, use the `max_output_tokens` [generation param](configuration.md#generation-params).
|
|
163
|
+
|
|
58
164
|
## Observing every attempt (`squawk`)
|
|
59
165
|
|
|
60
166
|
`squawk` is an opt-in hook for sending what the model returned to an error tracker or tracing tool. It runs after
|
|
61
|
-
every LLM attempt, accepted or rejected, and
|
|
167
|
+
every LLM attempt, accepted or rejected (unless you narrow it with `squawk_on:`, below), and once more when a token
|
|
168
|
+
budget or a disagreement stops the call. It does nothing unless you set it:
|
|
62
169
|
|
|
63
170
|
```ruby
|
|
64
171
|
Squishling.configure do |config|
|
|
@@ -71,12 +178,30 @@ end
|
|
|
71
178
|
| Keyword | Value |
|
|
72
179
|
|---|---|
|
|
73
180
|
| `output` | The raw response: a Hash or String, or `nil` when the provider call itself failed |
|
|
74
|
-
| `error` | `nil` for an accepted attempt. Otherwise the `InvalidOutputError` (with `errors` and `raw`) or `LLMError` that ended the attempt |
|
|
75
|
-
| `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length; under a harness, the sample's or judge's own attempts), `final` (the last attempt), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when
|
|
181
|
+
| `error` | `nil` for an accepted attempt. Otherwise the `InvalidOutputError` (with `errors` and `raw`) or `LLMError` that ended the attempt, or the `BudgetExhaustedError` or `DisagreementError` that stopped the call |
|
|
182
|
+
| `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length, or for a [stopped call](#a-call-that-is-stopped) the attempts made; `attempt` jumps past the attempts a repeated reply skips; under a harness, the sample's or judge's own attempts), `final` (the last attempt, or a rejection after which none follows), `repeated` (`true` when the attempt returned the same reply as the one before it, in the same chat, which skips the step's remaining attempts), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when the adapter reports them), `finish_reason` (why the provider stopped, e.g. `:stop` or `:max_tokens`; `nil` when unknown or the call failed), `role` (`nil` for an escalation attempt, `:sample`, or `:judge`), `sample` (`"a"`, `"b"`, ... for a sample's attempt, else `nil`), `stopped` (`nil` for an attempt; `:budget` or `:disagreement` for the [stopped call's report](#a-call-that-is-stopped)) |
|
|
76
183
|
|
|
77
184
|
- Set it globally with `config.squawk`, per class with `squishling squawk: ...`, or per method with
|
|
78
185
|
`squish :triage, squawk: ...`. The method's hook wins over the class's, which wins over the configured one;
|
|
79
186
|
`squawk: false` silences an inherited hook. Subclasses and `squish!` calls inherit it.
|
|
187
|
+
- Limit which attempts reach the hook with `squawk_on:`: `:all` (the default), `:rejections` (only attempts that
|
|
188
|
+
errored), `:retries` (those plus any accepted attempt after the first), or `:final` (the last attempt the escalation
|
|
189
|
+
allows, accepted or not; `metadata[:final]`). A classifier that passes on the first try nearly every time can post
|
|
190
|
+
only when something went wrong:
|
|
191
|
+
|
|
192
|
+
```ruby
|
|
193
|
+
squishling squawk: ->(error:, **) { Tracer.record(error) if error }, squawk_on: :retries
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
Pass a list to combine conditions; the hook runs when any one holds. `squawk_on: %i[rejections final]` reports
|
|
197
|
+
every error, plus the last attempt when it is reached and succeeds. A call accepted on an earlier attempt never
|
|
198
|
+
reaches `:final`. A level's value replaces the one above it; lists are never merged across levels.
|
|
199
|
+
|
|
200
|
+
Set it globally with `config.squawk_on`, per class with `squishling squawk_on: ...`, or per method with
|
|
201
|
+
`squish :triage, squawk_on: ...`. It resolves on its own, method then class then configured, so
|
|
202
|
+
`config.squawk_on = :retries` filters every hook, including a class's own, unless that class or method overrides it.
|
|
203
|
+
Under a [harness](harnesses.md), `attempt`, `attempts` and `final` count per sample or judge, so a sample's
|
|
204
|
+
first-attempt success is skipped even while another sample retries, and `:final` fires once per sample and judge.
|
|
80
205
|
- Any object that responds to `call` works. The hook receives only the keywords it declares, unless it takes `**`,
|
|
81
206
|
so a lambda that wants just `error:` is fine and new metadata fields won't break it.
|
|
82
207
|
- It runs inline, so keep it quick. Exceptions it raises propagate unchanged and fail the call, like any of your own
|
|
@@ -87,6 +212,28 @@ end
|
|
|
87
212
|
- It isn't called for a `ConfigurationError`, for deterministic and fallback returns, or for an attempt where your own
|
|
88
213
|
`squish_validate` raises (that exception propagates first).
|
|
89
214
|
|
|
215
|
+
### A call that is stopped
|
|
216
|
+
|
|
217
|
+
A call that a [token budget](#capping-a-calls-cost-token-budgets) or a
|
|
218
|
+
[disagreement](harnesses.md#when-the-harness-fails) stops has no attempt of its own to report: the attempt before a
|
|
219
|
+
budget stop was reported as not final, and a disagreement's samples were both accepted. So the call gets one more
|
|
220
|
+
report, whatever `squawk_on:` is set to:
|
|
221
|
+
|
|
222
|
+
- `error` is the `BudgetExhaustedError` or `DisagreementError`, `output` is `nil` (nothing was sent), `metadata[:final]`
|
|
223
|
+
is `true`, and `metadata[:stopped]` is `:budget` or `:disagreement` (it is `nil` for every attempt's report).
|
|
224
|
+
- `attempt` and `attempts` are `error.attempts`, or `0` when that is `nil` (a disagreement, or a budget that ran out
|
|
225
|
+
before a sample or judge made its first attempt); a stop before a retry counts the attempts already made. `model` is
|
|
226
|
+
the last of `error.models`, and `provider`, `params`, `usage`, `role`, and `sample` are `nil`.
|
|
227
|
+
- It is sent once, after the attempts that led to it and before `squish_fallback` runs, for the built-in harnesses and
|
|
228
|
+
for a [custom one](harnesses.md#reporting-to-squawk) that raises either error. A call to another squished method
|
|
229
|
+
from your own code (a `squish_validate`, say) reports its own stop, and the outer call does not report that error
|
|
230
|
+
again. The attempts before it keep `final: false`, so `squawk_on: :final` sees a budget or disagreement ending only
|
|
231
|
+
through this report.
|
|
232
|
+
- Like any attempt's report, it runs your hook inline: an exception the hook raises propagates in place of the
|
|
233
|
+
`BudgetExhaustedError` or `DisagreementError` (which stays on `cause`), so `squish_fallback` does not run.
|
|
234
|
+
- `squawk: false`, or no hook, sends nothing. Ending on the last invalid attempt is not a stop: that attempt already
|
|
235
|
+
carried `final: true`.
|
|
236
|
+
|
|
90
237
|
## Output checks (`squish_validate`)
|
|
91
238
|
|
|
92
239
|
The schema covers shape and types. For rules it can't express, such as cross-field arithmetic or a lookup against
|
|
@@ -120,17 +267,18 @@ end
|
|
|
120
267
|
| Error | Raised when |
|
|
121
268
|
|---|---|
|
|
122
269
|
| `Squishling::Error` | Base class for everything below. Raised directly for misuse at call time, e.g. `result` or `squish!` called outside a squished method |
|
|
123
|
-
| `Squishling::ConfigurationError` | Missing purpose or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_to_purpose` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (400) |
|
|
270
|
+
| `Squishling::ConfigurationError` | Missing purpose or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_to_purpose` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (a 400 or other rejected 4xx, as `RequestRejectedError`, with `provider_status` and `provider_code`) |
|
|
124
271
|
| `Squishling::InvalidOutputError` | LLM output still invalid (schema or `squish_validate`) after every attempt in the escalation, or a deterministic/fallback return that doesn't match the schema |
|
|
125
272
|
| `Squishling::DisagreementError` | A subclass of `InvalidOutputError`: a [squishsum harness](harnesses.md#when-the-harness-fails)'s samples disagreed and no judge accepted either. `candidates`, `verdict`, and `reason` describe what happened |
|
|
126
|
-
| `Squishling::
|
|
273
|
+
| `Squishling::BudgetExhaustedError` | A subclass of `InvalidOutputError`: the call used up its [token budget](#capping-a-calls-cost-token-budgets) before an attempt was accepted. `limit`, `max`, and `used` describe the budget |
|
|
274
|
+
| `Squishling::LLMError` | The provider call failed on the last attempt in the escalation, after the adapter's own retries, including context-length errors (`cause` holds a [sanitized stand-in](#the-cause-of-a-provider-failure), when the adapter has one) |
|
|
127
275
|
|
|
128
276
|
Errors raised by your own Ruby code are not wrapped.
|
|
129
277
|
|
|
130
278
|
## Fallbacks
|
|
131
279
|
|
|
132
280
|
Use `squish_fallback` to decide what happens when the LLM path fails with `InvalidOutputError` (including a
|
|
133
|
-
`DisagreementError`) or `LLMError`. It's the one fallback for every [harness](harnesses.md).
|
|
281
|
+
`DisagreementError` or `BudgetExhaustedError`) or `LLMError`. It's the one fallback for every [harness](harnesses.md).
|
|
134
282
|
It receives the error plus the method's inputs as keywords, and runs against the instance:
|
|
135
283
|
|
|
136
284
|
```ruby
|
data/docs/harnesses.md
CHANGED
|
@@ -58,8 +58,8 @@ Both samples run on the escalation's first step, with its model, provider, and p
|
|
|
58
58
|
- **They're independent.** Each sample has its own chat, and the two requests are sent concurrently on separate
|
|
59
59
|
threads. Only the provider requests run on those threads. Parsing, schema validation, `squish_validate`,
|
|
60
60
|
`compare:`, and the judge all run on the calling thread, so your Squishling callbacks never run concurrently.
|
|
61
|
-
RubyLLM instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for
|
|
62
|
-
each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
|
|
61
|
+
With the RubyLLM adapter, instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for
|
|
62
|
+
`chat.ruby_llm`) do see each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
|
|
63
63
|
- **Each one is retried like an escalation step.** A sample with invalid output is re-asked in its own chat, up to
|
|
64
64
|
the first step's `attempts:`, while a sample that already passed keeps its result. A failed request gets a fresh
|
|
65
65
|
chat. A sample still failing after its attempts fails the call with `InvalidOutputError` or `LLMError`, as
|
|
@@ -83,7 +83,7 @@ whose typed result is returned (`squished?` is `true`), or it picks `neither`, w
|
|
|
83
83
|
The judge is:
|
|
84
84
|
|
|
85
85
|
1. the `judge:` step, if declared: a model name, or a Hash with `model:`, `provider:`, `params:`, and `attempts:`
|
|
86
|
-
(as in an escalation step, without `priority
|
|
86
|
+
(as in an escalation step, without `priority:`, `rotate:`, or `include_when:`), plus `type:`; or
|
|
87
87
|
2. the escalation's next step after the sampled one, with its attempts. Steps tied on `priority:` rotate between calls
|
|
88
88
|
(see [Rotating models](configuration.md#rotating-models-within-a-priority)), so which model that is can vary.
|
|
89
89
|
|
|
@@ -91,6 +91,13 @@ A judged harness with neither raises `ConfigurationError` before any sample is r
|
|
|
91
91
|
step uses only its own `provider:`; it doesn't inherit the class or method `provider:`, which belongs to their models
|
|
92
92
|
(a chat judge's params, though, do merge over the method's).
|
|
93
93
|
|
|
94
|
+
**What the judge is sent.** The judge's provider receives the operation's purpose, the call's full input (the same
|
|
95
|
+
method arguments and named context the samples saw), and **both candidate outputs**, whatever they hold. None of it is
|
|
96
|
+
capped or filtered, and there is no opt-out short of not using a judged harness. When the judge is a different
|
|
97
|
+
provider than the sampled model (a `judge:` step with its own `provider:`, or an escalation's next step on another
|
|
98
|
+
provider), the input and both outputs go to that provider as well, so only name a judge you would trust with the
|
|
99
|
+
input itself.
|
|
100
|
+
|
|
94
101
|
### Chat judges
|
|
95
102
|
|
|
96
103
|
By default the judge is a chat model, asked for a strict `{ "verdict": "a" | "b" | "neither", "reason": "..." }`.
|
|
@@ -108,8 +115,10 @@ valid verdict raises `InvalidOutputError`. A chat judge's params are the method'
|
|
|
108
115
|
### System One judgment models (Jev)
|
|
109
116
|
|
|
110
117
|
A judge can also be a System One decision model, such as TypeSafe's Jev, via
|
|
111
|
-
[RubyLLM judgments](https://rubyllm.com/judgments/) (RubyLLM 2.1+).
|
|
112
|
-
|
|
118
|
+
[RubyLLM judgments](https://rubyllm.com/judgments/) (RubyLLM 2.1+). This needs an [adapter](adapters.md) that
|
|
119
|
+
supports judgments, which today means the `:ruby_llm` adapter; on any other, a `type: :judgment` judge raises
|
|
120
|
+
`ConfigurationError` before a request is made (a `type: :chat` judge works with every adapter). Decision models return
|
|
121
|
+
calibrated probabilities rather than text, so they are much faster and cheaper than a chat judge.
|
|
113
122
|
|
|
114
123
|
```ruby
|
|
115
124
|
squishling harness: {
|
|
@@ -122,8 +131,8 @@ Squishling asks one `choice` question (`a`, `b`, or `neither`), with the judge p
|
|
|
122
131
|
operation's purpose, input, output schema, and candidates as the judgment input. A pick of `a` or `b` is
|
|
123
132
|
accepted only when its probability is at least `min_confidence:` (default `0.8`). Otherwise, as with `neither`,
|
|
124
133
|
`DisagreementError` is raised with the choice and its probability in its `reason`. A judgment judge sends only its own
|
|
125
|
-
`params:` (as RubyLLM `provider_options:`). `attempts:` retries a failed request.
|
|
126
|
-
credentials in RubyLLM as usual.
|
|
134
|
+
`params:` (as RubyLLM `provider_options:`). `attempts:` retries a failed request. Judgments are a RubyLLM feature, so
|
|
135
|
+
they need the `:ruby_llm` adapter, with the provider's credentials configured in RubyLLM as usual.
|
|
127
136
|
|
|
128
137
|
### The default judge prompt
|
|
129
138
|
|
|
@@ -166,6 +175,10 @@ end
|
|
|
166
175
|
| `attempts` | `nil` |
|
|
167
176
|
| `models` | The model behind each role: sample `a`, sample `b`, then the judge if one ran (one entry each, not one per attempt) |
|
|
168
177
|
|
|
178
|
+
A [token budget](failures.md#capping-a-calls-cost-token-budgets) covers every request of the call, the samples
|
|
179
|
+
and the judge's included. When it is spent before the next round of retries or before the judge can run, the call raises
|
|
180
|
+
`BudgetExhaustedError` (also an `InvalidOutputError`, and also sent to the fallback) instead of a `DisagreementError`.
|
|
181
|
+
|
|
169
182
|
Other failures are unchanged. A sample or judge that never produces valid output raises `InvalidOutputError`, a
|
|
170
183
|
failed request on the last attempt raises `LLMError`, and setup mistakes (including a provider 400 on either
|
|
171
184
|
sample) raise `ConfigurationError`, which is never sent to the fallback.
|
|
@@ -256,7 +269,7 @@ the harness is declared, before any request is made, and an unknown, missing, or
|
|
|
256
269
|
|
|
257
270
|
| Type | Accepts | `call` receives |
|
|
258
271
|
|---|---|---|
|
|
259
|
-
| `:step` | A model name, or a Hash with `model:`, `provider:`, `params:`, `attempts:`, `forward_rejected:` | A normalized, frozen step; pass it to `run.escalate`, or in the Array given to `run.samples` |
|
|
272
|
+
| `:step` | A model name, or a Hash with `model:`, `provider:`, `params:`, `attempts:`, `forward_rejected:`, `retry_temperature:` | A normalized, frozen step; pass it to `run.escalate`, or in the Array given to `run.samples` |
|
|
260
273
|
| `:proc` | A Proc | The Proc |
|
|
261
274
|
| `:string`, `:symbol`, `:integer`, `:boolean` | A String, a Symbol (or String, converted), an Integer, `true` or `false` | The value |
|
|
262
275
|
|
|
@@ -288,12 +301,15 @@ results, the same type the deterministic path returns.
|
|
|
288
301
|
| `run.stage(declaration)` | A stage from a model name or a step Hash (a single step is a stage of one attempt). It keeps its own provider, and its `params:` merge over the method's, as a [judge](#the-judge) step does |
|
|
289
302
|
| `run.escalate(*stages)` | Makes each attempt in order until an output passes the schema and `squish_validate`, in the same chat after invalid output and in a fresh one (told about the rejected output) on a new step. With no argument, it runs the escalation: exactly what `:escalation` does |
|
|
290
303
|
| `run.samples(stages)` | Takes an Array of stages (so `run.samples([stage, stage])`, not `run.samples(stage)`; a bare step counts as a stage of one attempt): one sample per stage, each in its own chat and retried like an escalation step, with the requests of every round sent concurrently. Returns the outputs in order. Each stage must be one model's attempts: mixed steps raise `ConfigurationError`. Samples are named `"a"`, `"b"`, ... (up to 26) |
|
|
291
|
-
| `run.judge(judge: nil, instructions: nil)` | Resolves a [judge](#the-judge) now (so a missing one raises `ConfigurationError` before any request) and returns it. `judge:` is a model name, or a [judge Hash](#the-judge) (`model:`, `provider:`, `params:`, `attempts:`, `type:`, `min_confidence:`; no `priority
|
|
304
|
+
| `run.judge(judge: nil, instructions: nil)` | Resolves a [judge](#the-judge) now (so a missing one raises `ConfigurationError` before any request) and returns it. `judge:` is a model name, or a [judge Hash](#the-judge) (`model:`, `provider:`, `params:`, `attempts:`, `type:`, `min_confidence:`; no `priority:`, `rotate:`, or `include_when:`), and `instructions:` a non-blank String or a Proc. `judge.verdict(first, second)` returns a `Squishling::Judge::Verdict` with `choice` (`:a`, `:b`, or `:neither`), `reason`, and `detail`, and `judge.step` is its first attempt. A chat judge's `reason` is model-written, so use `detail` (Squishling's own text, `nil` for a chat judge), not `reason`, in messages and logs. Without `judge:`, the judge is the escalation's second stage |
|
|
292
305
|
| `run.agree?(first, second, compare = nil)` | `compare` (evaluated against the instance with both results and the method's inputs) or, without one, exact equality |
|
|
293
306
|
| `run.log(message)` | A warning to `config.logger`, prefixed with `"[Squishling] Class#method"` |
|
|
294
307
|
|
|
295
308
|
`call` returns the accepted result: one a primitive of this run returned. Returning anything else (a Hash, `nil`, a
|
|
296
|
-
result you built or copied with `with`) raises `ConfigurationError`, so a harness can't skip validation.
|
|
309
|
+
result you built or copied with `with`) raises `ConfigurationError`, so a harness can't skip validation. The check is
|
|
310
|
+
by object identity, which suits the typed results of an object schema. When the schema isn't an object, a result is a
|
|
311
|
+
bare JSON value, and an immediate such as `1` or `true` counts as produced whatever its source, so there the check can't
|
|
312
|
+
catch a hand-built value.
|
|
297
313
|
|
|
298
314
|
### When a custom harness fails
|
|
299
315
|
|
|
@@ -302,8 +318,10 @@ A harness fails the way the built-ins do, so `squish_fallback` and the error cla
|
|
|
302
318
|
- **Attempts run out.** `run.escalate` and `run.samples` raise `InvalidOutputError` (invalid output after the last
|
|
303
319
|
attempt) or `LLMError` (a failed request on the last attempt) themselves, with the same attributes as always.
|
|
304
320
|
- **Your rule rejects.** Raise one of those errors yourself. `Squishling::DisagreementError.new(candidates:,
|
|
305
|
-
models:)` is the one for samples that don't agree, and `Squishling::InvalidOutputError.new(errors: [...],
|
|
306
|
-
anything else. Both go to [`squish_fallback`](failures.md#fallbacks).
|
|
321
|
+
models:)` is the one for samples that don't agree, and `Squishling::InvalidOutputError.new(errors: [...],
|
|
322
|
+
source: "Harness")` for anything else. Both go to [`squish_fallback`](failures.md#fallbacks). A String error is
|
|
323
|
+
tagged `:squishling` in [`error_details`](failures.md#telling-the-entries-apart); pass a
|
|
324
|
+
`Squishling::InvalidOutputError::Detail` to tag it yourself.
|
|
307
325
|
- **Setup mistakes** raise `ConfigurationError`, which is never retried and never sent to the fallback.
|
|
308
326
|
- **Your own exceptions** (from `call`, `compare:`, or `squish_validate`) propagate unchanged. Squishling doesn't wrap or
|
|
309
327
|
swallow them.
|
|
@@ -313,10 +331,16 @@ available to your code as the results and in `InvalidOutputError#raw`, and to `s
|
|
|
313
331
|
|
|
314
332
|
### Reporting to `squawk`
|
|
315
333
|
|
|
316
|
-
Every attempt a primitive makes is reported to the [`squawk`](failures.md#observing-every-attempt-squawk) hook
|
|
317
|
-
usage, so [token measuring](measuring-tokens.md) works unchanged. `metadata[:role]` says what the attempt was for:
|
|
334
|
+
Every attempt a primitive makes is reported to the [`squawk`](failures.md#observing-every-attempt-squawk) hook
|
|
335
|
+
(unless narrowed with `squawk_on:`), with usage, so [token measuring](measuring-tokens.md) works unchanged. `metadata[:role]` says what the attempt was for:
|
|
318
336
|
`nil` for an escalation attempt, `:sample` (with `metadata[:sample]` set to `"a"`, `"b"`, ...), or `:judge`. `attempt`
|
|
319
|
-
and `attempts` count within that sample, judge, or escalation.
|
|
337
|
+
and `attempts` count within that sample, judge, or escalation. `metadata[:repeated]` is `true` for an attempt that
|
|
338
|
+
sent back the reply it was just rejected for; the primitive then skips the rest of that step's attempts (the next
|
|
339
|
+
step's `attempt` jumps past them, and a sample ends with its `InvalidOutputError`).
|
|
340
|
+
|
|
341
|
+
A call that ends on the token budget or on a `DisagreementError` (the built-in squishsum harnesses', or one your
|
|
342
|
+
`call` raises) is reported once more, as [a stopped call](failures.md#a-call-that-is-stopped), with `metadata[:stopped]`
|
|
343
|
+
set and no output. Squishling does it around your `call`, so a custom harness needs no code for it.
|
|
320
344
|
|
|
321
345
|
### Threads
|
|
322
346
|
|
data/docs/measuring-tokens.md
CHANGED
|
@@ -6,11 +6,13 @@ first tells you when the second is worth paying.
|
|
|
6
6
|
## What Squishling gives you
|
|
7
7
|
|
|
8
8
|
- `result.squished?` says which path served a call, so you can count LLM calls per class.
|
|
9
|
-
- The [`squawk` hook](failures.md#observing-every-attempt-squawk) runs after every LLM attempt and receives the
|
|
9
|
+
- The [`squawk` hook](failures.md#observing-every-attempt-squawk) runs after every LLM attempt (by default; see `squawk_on` below) and receives the
|
|
10
10
|
method (`label`, e.g. `"ShipmentUpdate#call"`), the model and provider, which attempt it was, and the token usage
|
|
11
|
-
|
|
12
|
-
-
|
|
13
|
-
|
|
11
|
+
the adapter reported. That is enough to attribute spend to a squishling method without any other tooling.
|
|
12
|
+
- A [token budget](failures.md#capping-a-calls-cost-token-budgets) (`max_total_output_tokens:`, `max_total_input_tokens:`)
|
|
13
|
+
caps what a single call may use across all its attempts, so a call that keeps failing can't spend without limit.
|
|
14
|
+
- Cost in dollars isn't reported by Squishling's adapters. Multiply the reported tokens by your model's price, or, with
|
|
15
|
+
the RubyLLM adapter, read it from RubyLLM directly (see [OpenTelemetry](#opentelemetry), or RubyLLM's `config.instrumenter`).
|
|
14
16
|
|
|
15
17
|
## Deciding when to write the code
|
|
16
18
|
|
|
@@ -56,8 +58,8 @@ template each call belongs to; see [Coolhand Labs](coolhand.md).
|
|
|
56
58
|
|
|
57
59
|
### Custom: the `squawk` hook
|
|
58
60
|
|
|
59
|
-
`config.squawk` runs after every LLM attempt, accepted or rejected, and is handed the method's label, the model,
|
|
60
|
-
and the token usage
|
|
61
|
+
`config.squawk` runs after every LLM attempt, accepted or rejected (unless you narrow it with `squawk_on:`, which makes token totals undercount, so keep `:all` for measuring), and is handed the method's label, the model,
|
|
62
|
+
and the token usage the adapter reported. A lambda can take just the keywords it needs:
|
|
61
63
|
|
|
62
64
|
```ruby
|
|
63
65
|
Squishling.configure do |config|
|
|
@@ -70,6 +72,10 @@ Squishling.configure do |config|
|
|
|
70
72
|
end
|
|
71
73
|
```
|
|
72
74
|
|
|
75
|
+
A call that stops on its [token budget](failures.md#capping-a-calls-cost-token-budgets) or a disagreement is also
|
|
76
|
+
reported once, with `metadata[:stopped]` set (`:budget` or `:disagreement`) and no `usage`, so the hook above logs
|
|
77
|
+
it with empty counts; check `metadata[:stopped]` to count it apart from an attempt.
|
|
78
|
+
|
|
73
79
|
Each line carries the `"Class#method"` that made the call, so grouping by label gives you tokens per squishling. A call
|
|
74
80
|
that stays in Ruby makes no LLM request and never reaches the hook, so a label that stops appearing is a path that has
|
|
75
81
|
been hardened. You can also set the hook per class (`squishling squawk: ...`) or per method. See
|