squishling 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +58 -1
- data/README.md +38 -6
- data/docs/configuration.md +36 -6
- data/docs/coolhand.md +66 -0
- data/docs/failures.md +9 -5
- data/docs/harnesses.md +170 -42
- data/docs/measuring-tokens.md +29 -37
- data/docs/observers.md +80 -0
- data/docs/routing.md +2 -2
- data/docs/schemas.md +22 -4
- data/lib/squishling/configuration.rb +15 -2
- data/lib/squishling/definition.rb +9 -5
- data/lib/squishling/errors.rb +5 -5
- data/lib/squishling/harness/base.rb +136 -0
- data/lib/squishling/harness/escalation.rb +22 -0
- data/lib/squishling/harness/judged_squishsum.rb +83 -0
- data/lib/squishling/harness/option_types.rb +93 -0
- data/lib/squishling/harness/registry.rb +69 -0
- data/lib/squishling/harness/squishsum.rb +22 -0
- data/lib/squishling/harness.rb +25 -99
- data/lib/squishling/invoker.rb +11 -228
- data/lib/squishling/judge.rb +66 -22
- data/lib/squishling/llm_client.rb +36 -6
- data/lib/squishling/model_path.rb +64 -21
- data/lib/squishling/observer.rb +100 -0
- data/lib/squishling/observers/coolhand.rb +86 -0
- data/lib/squishling/params.rb +10 -5
- data/lib/squishling/rotation.rb +26 -0
- data/lib/squishling/run/messages.rb +42 -0
- data/lib/squishling/run.rb +296 -0
- data/lib/squishling/version.rb +1 -1
- data/lib/squishling.rb +32 -0
- metadata +14 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 0fb55da97a0a032802afcd90b8e56d497a42194206ad90f07d71bc46c70a9e49
|
|
4
|
+
data.tar.gz: 79430c869cda0726357c42c1ad1fdc7cbf9108a00ebcd56814df4e45acf99db6
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 95f83f1f99d86b5b2fddcc0d5e7d845aff73dbb7cc85620eafb93f999edd69181e8a72c726409cdb6ec11fa1286264947c48ac5f3467c1fb368755ef8ad95b62
|
|
7
|
+
data.tar.gz: 4ad6fbec6185bb9e4369c972b9f561bce840db75a96ba02c619fe82d1726e15230330b8f457c740017b08e412b3da2c364fd40f657db3128b28cab9c01ba993b
|
data/CHANGELOG.md
CHANGED
|
@@ -7,7 +7,64 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
-
## [0.
|
|
10
|
+
## [0.4.0] - 2026-10-18
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- Custom harnesses: `Squishling.register_harness` (with `Squishling.harnesses` and `reset_harnesses!`) and
|
|
15
|
+
`Squishling::Harness::Base` let you write your own harness and use it anywhere a built-in works (`harness:`,
|
|
16
|
+
`squish!`, `default_harness`). A harness gets a `Squishling::Run` with `escalate`, `samples`, `judge`, `agree?`,
|
|
17
|
+
`stage`, and `log`, so it keeps model, provider, and params resolution, `squish_validate`, concurrent samples, and the
|
|
18
|
+
normal failure handling; it can only return typed results its `Run` produced, and its options are validated when it
|
|
19
|
+
is declared. The built-in `:escalation`, `:squishsum`, and `:judged_squishsum` now run on the same interface.
|
|
20
|
+
`squawk` metadata gains `role:` and `sample:`. See [Custom harnesses](docs/harnesses.md#custom-harnesses). (#31)
|
|
21
|
+
- Round-robin models: escalation steps that share a `priority:` form a tier. Each call rotates which model in the tier
|
|
22
|
+
goes first (counted per class and method, shared across threads), and the tier's other models are tried before the
|
|
23
|
+
next priority. `rotate: false` on a step pins its tier to the order written. (#38)
|
|
24
|
+
- `config.observer`: an around-style hook (`around(call) { ... }`) wrapped around every provider request, on the thread
|
|
25
|
+
that makes it, so a `:squishsum` sample's observer runs on its worker thread. It is told the request's `Observer::Call`
|
|
26
|
+
and gets back only an `Observer::Usage` (token counts and model), never the response. An observer that yields twice
|
|
27
|
+
or never raises `ConfigurationError`, and one that rescues the request's error has it raised again. See
|
|
28
|
+
[Observers](docs/observers.md). (#25, #40)
|
|
29
|
+
- A bundled Coolhand observer (`observer = :coolhand`) that declares each call's workload (`Class#method`) and template
|
|
30
|
+
(`squish:L<n>`, `sample:a`, or `judge`) through `Coolhand.with_context`. The workload's description is
|
|
31
|
+
`Class#method`, or `Class#method (rescue)` when a `squish!` call handed the call to the LLM (`Observer::Call#handed_off`),
|
|
32
|
+
never the purpose text. It needs coolhand 0.9.0 or newer. See
|
|
33
|
+
[Coolhand Labs](docs/coolhand.md). (#40)
|
|
34
|
+
- A docs page and landing-page section on the architecture, an explainer site in `site/` deployed to GitHub Pages, and
|
|
35
|
+
per-field `description:` documented as the best practice over `purpose` guidance. (#30, #33, #34, #36)
|
|
36
|
+
|
|
37
|
+
### Changed
|
|
38
|
+
|
|
39
|
+
- **Behavior change:** when the `coolhand` gem is loaded and configured (an API key, not disabled), Squishling now
|
|
40
|
+
declares each request's workload and template to it automatically. Nothing changes for apps without Coolhand. To
|
|
41
|
+
opt out, set `config.observer = false`.
|
|
42
|
+
- **Behavior change:** two escalation steps may now share a `priority:` (they used to raise `duplicate order:`), and
|
|
43
|
+
they rotate per call as described above. If you relied on a fixed order between tied steps, give them distinct
|
|
44
|
+
priorities or set `rotate: false`.
|
|
45
|
+
- `order:` on an escalation step is deprecated in favor of `priority:`. It still works, with a deprecation warning, and
|
|
46
|
+
giving both on one step raises `ConfigurationError`. Judge and harness steps take no `priority:` or `rotate:`. (#38)
|
|
47
|
+
- **Behavior change:** `LLMError` and the `ConfigurationError` for a rejected request (400), bad credentials, or a
|
|
48
|
+
forbidden key no longer repeat the provider's own message, which can echo request content or part of an API key into
|
|
49
|
+
logs and error trackers. They carry the error class and HTTP status (`RubyLLM::RateLimitError, HTTP 429`); the
|
|
50
|
+
provider's message is still on `cause`. If you matched on provider text in `e.message`, read `e.cause.message`
|
|
51
|
+
instead.
|
|
52
|
+
- `docs/measuring-tokens.md` now points to observers for bringing your own metrics instead of walking through a
|
|
53
|
+
hand-rolled one. (#40)
|
|
54
|
+
|
|
55
|
+
### Removed
|
|
56
|
+
|
|
57
|
+
- **Breaking change:** the `:ensemble` and `:judged_ensemble` harnesses added in 0.3.0 are gone. `harness: :ensemble`
|
|
58
|
+
or `:judged_ensemble` (and `default_harness`) now raise `ConfigurationError` (unknown harness). Use `:squishsum` or
|
|
59
|
+
`:judged_squishsum`, which sample the first escalation step twice with the judge defaulting to the next step, or
|
|
60
|
+
write a custom harness. (#22, #24)
|
|
61
|
+
|
|
62
|
+
### Internal
|
|
63
|
+
|
|
64
|
+
- CI gains an exhale duplication and complexity gate (`bin/exhale`), and dependabot bumps `rubocop` (1.92), `ruby/setup-ruby`,
|
|
65
|
+
and `actions/upload-artifact`. These are development-only changes. (#27, #28, #29, #41)
|
|
66
|
+
|
|
67
|
+
## [0.3.0] - 2026-10-10
|
|
11
68
|
|
|
12
69
|
### Added
|
|
13
70
|
|
data/README.md
CHANGED
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
<img src="assets/squishling-logo.png" alt="Squishling" width="420">
|
|
3
3
|
</h1>
|
|
4
4
|
|
|
5
|
+
<p align="center">
|
|
6
|
+
<strong><a href="https://coolhand-labs.github.io/squishling/">📖 Read the Squishling explainer →</a></strong>
|
|
7
|
+
</p>
|
|
8
|
+
|
|
5
9
|
[](https://github.com/Coolhand-Labs/squishling/actions/workflows/ci.yml)
|
|
6
10
|
[](https://badge.fury.io/rb/squishling)
|
|
7
11
|
|
|
@@ -14,7 +18,17 @@ Either way it returns the same strict-schema-validated, typed result, so callers
|
|
|
14
18
|
Think of code as a cache for AI judgment. Every line you write is something to maintain, so the LLM is the
|
|
15
19
|
default and code has to earn its place: you build it for the inputs that carry the volume, and the LLM keeps
|
|
16
20
|
handling the rest. New to the idea? Read
|
|
17
|
-
[Elastic Software](https://everythingengineer.substack.com/p/beginners-write-software-with-ai)
|
|
21
|
+
[Elastic Software](https://everythingengineer.substack.com/p/beginners-write-software-with-ai) or browse the
|
|
22
|
+
[talk slides](https://michael.carroll.io/talks/2026/elastic-software/).
|
|
23
|
+
|
|
24
|
+
Or try it on the [explainer site](https://coolhand-labs.github.io/squishling/):
|
|
25
|
+
|
|
26
|
+
- [Spectrum slider](https://coolhand-labs.github.io/squishling/#spectrum): drag from traditional software to fully
|
|
27
|
+
LLM and see the tradeoff at each point
|
|
28
|
+
- [Token-cost calculator](https://coolhand-labs.github.io/squishling/#number): price your tech debt in tokens
|
|
29
|
+
against code upkeep
|
|
30
|
+
- [Walkthrough](https://coolhand-labs.github.io/squishling/#squishling): describe it, tune it, pay for accuracy,
|
|
31
|
+
harden the common shapes
|
|
18
32
|
|
|
19
33
|
## Use cases
|
|
20
34
|
|
|
@@ -81,6 +95,12 @@ Both calls return the same result class. If the LLM can't deliver either, `squis
|
|
|
81
95
|
return, or the error is raised with the original `ParseError` as its cause. See
|
|
82
96
|
[Handing off to the LLM](docs/routing.md#handing-off-to-the-llm-with-squish).
|
|
83
97
|
|
|
98
|
+
### Retire code you no longer need
|
|
99
|
+
|
|
100
|
+
Go the other way too. Find the parsers and handlers that serve a thin tail of your traffic and squish them. When a
|
|
101
|
+
path's monthly token cost is lower than what its code costs to maintain, delete the code and relish the feeling of a
|
|
102
|
+
slimmer code base (or let your AI coding agent relish it).
|
|
103
|
+
|
|
84
104
|
### Measure your tech debt in tokens
|
|
85
105
|
|
|
86
106
|
Code you haven't written is debt you haven't taken on. A squished path has a running cost you can read off a
|
|
@@ -131,6 +151,11 @@ validated on both paths). A squished call goes to the LLM when:
|
|
|
131
151
|
|
|
132
152
|
Otherwise the Ruby runs, and whatever it returns is validated and typed like LLM output.
|
|
133
153
|
|
|
154
|
+
**Best practice: describe fields in the schema, not the purpose.** Put per-field guidance in a `description:` on
|
|
155
|
+
the field (`string :status, description: "..."`, or `optional(:eta, description: "...") { string }`). The model
|
|
156
|
+
sees it right where the value goes, and the purpose stays short and about the goal. See
|
|
157
|
+
[Describing fields](docs/schemas.md#describing-fields).
|
|
158
|
+
|
|
134
159
|
## Features
|
|
135
160
|
|
|
136
161
|
- **One contract, two paths**: Ruby returns and LLM output are validated against the same strict schema and
|
|
@@ -140,7 +165,7 @@ Otherwise the Ruby runs, and whatever it returns is validated and typed like LLM
|
|
|
140
165
|
- **Any RubyLLM provider and model**: OpenAI, Anthropic Claude, Google Gemini, AWS Bedrock, OpenRouter, and more.
|
|
141
166
|
Set a universal, per-class, per-method, or per-call model, plus layered generation params (temperature,
|
|
142
167
|
reasoning effort, top_p, …).
|
|
143
|
-
- **Model escalation**: declare an `escalation:` instead of a `model:`, with per-step `attempts:`, `
|
|
168
|
+
- **Model escalation**: declare an `escalation:` instead of a `model:`, with per-step `attempts:`, `priority:` (models tied on priority take turns going first), provider,
|
|
144
169
|
params, and `forward_rejected:`. Invalid output (with its errors fed back) or a provider failure moves on to the next attempt. Steps can
|
|
145
170
|
cross providers, e.g. a local Qwen model on Ollama, then Anthropic Claude Haiku on AWS Bedrock, then Claude Opus:
|
|
146
171
|
|
|
@@ -168,7 +193,8 @@ Otherwise the Ruby runs, and whatever it returns is validated and typed like LLM
|
|
|
168
193
|
- [`:escalation`](docs/configuration.md#models-and-escalation) (default): try each attempt until one passes
|
|
169
194
|
- [`:squishsum`](docs/harnesses.md#samples): two concurrent samples, accepted only if they agree
|
|
170
195
|
- [`:judged_squishsum`](docs/harnesses.md#the-judge): a chat or Jev judge picks between disagreeing samples
|
|
171
|
-
- [
|
|
196
|
+
- [Your own](docs/harnesses.md#custom-harnesses): `Squishling.register_harness` plugs a custom harness into the same
|
|
197
|
+
machinery (samples, judges, `squawk`, `squish_fallback`)
|
|
172
198
|
- **Output contracts**: beyond the strict schema, conditional rules (`given`) and Ruby checks (`squish_validate`)
|
|
173
199
|
reject bad output and trigger the next attempt.
|
|
174
200
|
- **Defined failure behavior**: provider errors become `Squishling::LLMError`, and `squish_fallback` lets you decide
|
|
@@ -182,15 +208,19 @@ Otherwise the Ruby runs, and whatever it returns is validated and typed like LLM
|
|
|
182
208
|
|
|
183
209
|
## Documentation
|
|
184
210
|
|
|
211
|
+
- [Explainer site](https://coolhand-labs.github.io/squishling/): interactive walkthrough, spectrum slider, and
|
|
212
|
+
token-cost calculator
|
|
185
213
|
- [Configuration](docs/configuration.md): options, models and escalation, providers, generation params, inheritance
|
|
186
214
|
- [Routing](docs/routing.md): when a call goes to the LLM, `squish!`, `append_to_purpose`, hardening a path,
|
|
187
215
|
what the LLM sees
|
|
188
216
|
- [Output schemas](docs/schemas.md): schema forms, strict mode, typed results, optional vs. empty, contracts
|
|
189
217
|
- [Failure handling](docs/failures.md): escalation, `squish_validate`, error classes, fallbacks
|
|
190
|
-
- [Harnesses](docs/harnesses.md): escalation, squishsum,
|
|
218
|
+
- [Harnesses](docs/harnesses.md): escalation, squishsum, judges (chat or Jev), and custom harnesses
|
|
191
219
|
- [Naming and collisions](docs/naming.md): the methods `include Squishling` adds and what happens when a name is taken
|
|
192
220
|
- [Measuring token spend](docs/measuring-tokens.md): see what each squished path costs, with Coolhand Labs,
|
|
193
|
-
the `squawk` hook,
|
|
221
|
+
the `squawk` hook, your own observer, or OpenTelemetry
|
|
222
|
+
- [Observers](docs/observers.md): wrap every provider request, and the built-in [Coolhand Labs](docs/coolhand.md) observer
|
|
223
|
+
that files each call under its workload and template
|
|
194
224
|
- [Live examples](examples/README.md): end-to-end tests against Anthropic Claude Haiku and OpenAI GPT-6 Luna
|
|
195
225
|
|
|
196
226
|
## Development
|
|
@@ -201,10 +231,12 @@ bundle exec rake # RSpec (offline; RubyLLM is stubbed) + RuboCop
|
|
|
201
231
|
```
|
|
202
232
|
|
|
203
233
|
See [AGENTS.md](AGENTS.md) for repo conventions, and [SECURITY.md](SECURITY.md) for reporting vulnerabilities.
|
|
234
|
+
Have a suggestion to improve Squishling? [Open an issue](https://github.com/Coolhand-Labs/squishling/issues).
|
|
204
235
|
|
|
205
236
|
## Credits
|
|
206
237
|
|
|
207
|
-
Inspired by [Elastic Software](https://everythingengineer.substack.com/p/beginners-write-software-with-ai)
|
|
238
|
+
Inspired by [Elastic Software](https://everythingengineer.substack.com/p/beginners-write-software-with-ai); see also
|
|
239
|
+
the [talk slides](https://michael.carroll.io/talks/2026/elastic-software/) from XO Ruby NYC 2026.
|
|
208
240
|
|
|
209
241
|
## License
|
|
210
242
|
|
data/docs/configuration.md
CHANGED
|
@@ -29,7 +29,8 @@ end
|
|
|
29
29
|
| `default_provider` | `nil` | The provider for `default_model`/`default_escalation` steps that don't name one. Only needed for models missing from RubyLLM's registry (see below). |
|
|
30
30
|
| `default_params` | `{}` | Generation params for every call (temperature, reasoning effort, top_p, …), overridable per class, per method, and per escalation step. See [Generation params](#generation-params). |
|
|
31
31
|
| `squawk` | `nil` | A callable run after every LLM attempt with the raw output, for sending it to an error tracker or tracing tool. Also settable per class and per method. See [Observing every attempt](failures.md#observing-every-attempt-squawk). |
|
|
32
|
-
| `
|
|
32
|
+
| `observer` | `nil` | An around-style hook for every provider request: `:coolhand`, any object responding to `around(call)`, or `false` for none. `nil` uses the [Coolhand observer](coolhand.md) when the `coolhand` gem is configured. See [Observers](observers.md). |
|
|
33
|
+
| `default_harness` | `nil` (`:escalation`) | How every squishling class that doesn't declare its own uses its escalation: `:escalation`, `:squishsum`, `:judged_squishsum`, a [registered custom harness](harnesses.md#custom-harnesses), or a Hash with `type:` and options. See [Harnesses](harnesses.md). |
|
|
33
34
|
| `logger` | `nil` | Any `Logger`. Debug lines when a call routes to the LLM, warnings on each escalation and when a fallback is used. Never includes the raw model response; see [what ends up in errors and logs](failures.md#what-ends-up-in-errors-and-logs). |
|
|
34
35
|
|
|
35
36
|
Transport-level retries (rate limits, 5xx, timeouts) are configured on RubyLLM itself
|
|
@@ -62,21 +63,50 @@ Each step is a model name, or a Hash with:
|
|
|
62
63
|
|---|---|---|
|
|
63
64
|
| `model:` | required | The model id. |
|
|
64
65
|
| `attempts:` | `1` | How many attempts this step gets before escalating to the next one. |
|
|
65
|
-
| `
|
|
66
|
+
| `priority:` | position | An Integer; steps run lowest first. Give every step a `priority:` or none. Steps with the same priority are one tier (see [Rotating models](#rotating-models-within-a-priority)). Gaps are fine. `order:` is the deprecated name for it. |
|
|
67
|
+
| `rotate:` | `true` | Set `false` on a step to keep its tier in the order written instead of rotating it. |
|
|
66
68
|
| `provider:` | the level's `provider:` | See [Providers](#providers-and-newly-released-models). |
|
|
67
69
|
| `forward_rejected:` | `true` | Set `false` to start this step from the original input alone, without the previous step's rejected output and its errors (see below). |
|
|
68
70
|
| `params:` | `{}` | [Generation params](#generation-params) for this step, merged over the class and method params key by key. |
|
|
69
71
|
|
|
70
|
-
Without `
|
|
72
|
+
Without `priority:`, steps run in the order written. With it, the order is explicit and doesn't depend on position:
|
|
71
73
|
|
|
72
74
|
```ruby
|
|
73
75
|
squishling escalation: [
|
|
74
|
-
{ model: "claude-opus-5-5",
|
|
75
|
-
{ model: "claude-haiku-4-5",
|
|
76
|
-
{ model: "
|
|
76
|
+
{ model: "claude-opus-5-5", priority: 30, params: { thinking: { effort: :high } } },
|
|
77
|
+
{ model: "claude-haiku-4-5", priority: 10, attempts: 2 },
|
|
78
|
+
{ model: "gemma-4b", priority: 10 },
|
|
79
|
+
{ model: "claude-sonnet-5-5", priority: 20 }
|
|
77
80
|
]
|
|
78
81
|
```
|
|
79
82
|
|
|
83
|
+
**Best practice:** number priorities 10, 20, 30, and so on, so a model can be slotted in between later (a `15`)
|
|
84
|
+
without renumbering the rest. Haiku and Gemma share the lowest priority here, so they take turns going first (see below).
|
|
85
|
+
|
|
86
|
+
### Rotating models within a priority
|
|
87
|
+
|
|
88
|
+
Give several models the same `priority:` and they share a tier. Each call rotates which of them goes first, so
|
|
89
|
+
every model gets an even share of the real work over many calls, a fair test of how well each handles it:
|
|
90
|
+
|
|
91
|
+
```ruby
|
|
92
|
+
squishling escalation: [
|
|
93
|
+
{ model: "claude-haiku-4-5", priority: 10 },
|
|
94
|
+
{ model: "gemma-4b", priority: 10 },
|
|
95
|
+
{ model: "claude-sonnet-5-5", priority: 20 }
|
|
96
|
+
]
|
|
97
|
+
# call 1: haiku, gemma-4b, sonnet call 2: gemma-4b, haiku, sonnet call 3: haiku, gemma-4b, sonnet
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
- If the model that goes first fails (invalid output or a provider failure, after its own `attempts:`), the other
|
|
101
|
+
models in the tier are tried, in rotated order, before the call escalates to the next priority.
|
|
102
|
+
- Each class and method counts its own calls (`AClass#a`, `AClass#b` and `BClass#a` rotate independently), shared across
|
|
103
|
+
threads, but not across processes: each process (a Puma worker, say) starts its own count at the first model, and it
|
|
104
|
+
restarts with the process. A `squish!` whose own escalation has a rotating tier counts against its method too.
|
|
105
|
+
- A tier rotates by its declared order, so ties keep the order they were written in. A tier with a single model
|
|
106
|
+
never rotates, and neither does an escalation without `priority:`. `rotate: false` on any step of a tier keeps the
|
|
107
|
+
whole tier in the order written, whatever its other steps say.
|
|
108
|
+
- Rotation also moves which stages the [`:squishsum` and `:judged_squishsum`](harnesses.md) harnesses sample first.
|
|
109
|
+
|
|
80
110
|
`params:` on a step let one step switch to a reasoning model that rejects sampling params:
|
|
81
111
|
|
|
82
112
|
```ruby
|
data/docs/coolhand.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Coolhand Labs
|
|
2
|
+
|
|
3
|
+
Squishling ships an [observer](observers.md) for [Coolhand Labs](https://coolhandlabs.com/). With it, every LLM call a
|
|
4
|
+
squished method makes is filed under a named **workload** and **template** in Coolhand. Squishling knows exactly
|
|
5
|
+
where each call sits (which method, which escalation level, which sample, which judge), so it declares that for you.
|
|
6
|
+
|
|
7
|
+
## Setup
|
|
8
|
+
|
|
9
|
+
Add and configure the [`coolhand`](https://github.com/Coolhand-Labs/coolhand-ruby) gem as described in
|
|
10
|
+
[Measuring token spend](measuring-tokens.md#coolhand-labs). **That's all.** Squishling defaults to the Coolhand
|
|
11
|
+
observer whenever Coolhand is already available in your codebase: loaded, with an `api_key`, and not disabled
|
|
12
|
+
(`config.enabled = false`). Nothing is added to Squishling's dependencies, and nothing happens when the gem isn't
|
|
13
|
+
there. It needs a version of the `coolhand` gem that has `Coolhand.with_context` (declared workloads); with an older
|
|
14
|
+
one, the observer stays off.
|
|
15
|
+
|
|
16
|
+
To turn it off, set `observer` to `false`:
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
Squishling.configure do |config|
|
|
20
|
+
config.observer = false # never declare, even though Coolhand is configured
|
|
21
|
+
end
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
The other settings:
|
|
25
|
+
|
|
26
|
+
```ruby
|
|
27
|
+
Squishling.configure do |config|
|
|
28
|
+
config.observer = :coolhand # always declare; raises ConfigurationError now if the gem is missing or too old
|
|
29
|
+
config.observer = nil # the default: declare when Coolhand is loaded and configured
|
|
30
|
+
end
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
To report somewhere else, or as well, use any [other observer](observers.md). `config.observer` holds one.
|
|
34
|
+
|
|
35
|
+
## What is declared
|
|
36
|
+
|
|
37
|
+
Each request runs inside `Coolhand.with_context(reporter: "squishling", workload: ..., template: ...)`, so Coolhand's
|
|
38
|
+
Net::HTTP interception tags it with the declaration.
|
|
39
|
+
|
|
40
|
+
| | Key | Name |
|
|
41
|
+
|---|---|---|
|
|
42
|
+
| Workload | `Class#method` | `Class#method` (its `description` is `Class#method`, or `Class#method (rescue)` when a [`squish!`](routing.md#handing-off-to-the-llm-with-squish) call handed it to the LLM) |
|
|
43
|
+
| Template | `Class#method/<role>` | `Class#method · <role>` |
|
|
44
|
+
|
|
45
|
+
`<role>` is where the request sits:
|
|
46
|
+
|
|
47
|
+
| Role | Request |
|
|
48
|
+
|---|---|
|
|
49
|
+
| `squish:L<n>` | Attempt *n* (1-based) of the [escalation](configuration.md#models-and-escalation), counting each attempt of each step (a [custom harness](harnesses.md#custom-harnesses) that escalates stage by stage restarts at L1 for each) |
|
|
50
|
+
| `sample:a`, `sample:b`, ... | A [squishsum](harnesses.md) sample (which runs on a worker thread; the declaration is made on that thread) |
|
|
51
|
+
| `judge` | A harness's judge, chat or System One (all of a judge's attempts share this one template) |
|
|
52
|
+
|
|
53
|
+
So `Parser#call/squish:L2` is the second attempt of `Parser#call`, e.g. `Parser#call · squish L2` in Coolhand.
|
|
54
|
+
|
|
55
|
+
- The model is deliberately not in the key. It is on the log, and the position is the stable identity.
|
|
56
|
+
- Repeated attempts on the same step count as separate levels, because the level is the attempt's position in the
|
|
57
|
+
expanded escalation.
|
|
58
|
+
- A class name long enough to push a key past Coolhand's 255-character limit is left undeclared rather than
|
|
59
|
+
breaking the call.
|
|
60
|
+
- Squishling's declaration is made inside any `Coolhand.with_context` you wrapped around the call, so for the fields it
|
|
61
|
+
sets, Squishling's wins.
|
|
62
|
+
- The description never contains the purpose text, which can be a proc that differs per call. The workload's key
|
|
63
|
+
and name stay `Class#method` either way, so calls that went through `squish!` and calls that didn't share one
|
|
64
|
+
workload.
|
|
65
|
+
- `observer = :coolhand` loads the gem when it is assigned, so assign it after `coolhand` is available (the default
|
|
66
|
+
needs no ordering).
|
data/docs/failures.md
CHANGED
|
@@ -10,12 +10,12 @@ A failed attempt moves on to the next one; once the last fails, the error is rai
|
|
|
10
10
|
|---|---|
|
|
11
11
|
| Rate limit, 5xx, overload, timeout, connection error | RubyLLM retries at the HTTP level first (`RubyLLM.config.max_retries`, default 3). If it still fails, moves to the next attempt in a fresh chat; on the last attempt, raises `Squishling::LLMError` with the original exception as its `cause`. |
|
|
12
12
|
| Bad API key, unknown model, missing provider config | Raises `Squishling::ConfigurationError`. Never escalated, never sent to the fallback. |
|
|
13
|
-
| Provider rejects the request (400 Bad Request), e.g. an unsupported `temperature` or a schema it won't accept | Raises `ConfigurationError`
|
|
13
|
+
| Provider rejects the request (400 Bad Request), e.g. an unsupported `temperature` or a schema it won't accept | Raises `ConfigurationError` naming the error class and HTTP status, and the params in use; the provider's own message is on `cause`. Never escalated, never sent to the fallback, so a setup mistake can't be silently covered up on every call. |
|
|
14
14
|
| Empty or `nil` response (e.g. a refusal or a max-tokens cutoff) | Moves to the next attempt, then raises `Squishling::InvalidOutputError` |
|
|
15
15
|
| Malformed or truncated JSON | Moves to the next attempt, then raises `InvalidOutputError`. JSON wrapped in a markdown code fence is accepted. |
|
|
16
16
|
| JSON that doesn't match the schema (wrong types, missing or extra keys, `null` in a non-`optional` field, a broken [conditional rule](schemas.md#contracts-beyond-the-schema), root not an object) | Moves to the next attempt with the validation errors, then raises `InvalidOutputError` |
|
|
17
17
|
| Schema-valid output that [`squish_validate`](#output-checks-squish_validate) rejects | Moves to the next attempt with your messages, then raises `InvalidOutputError` |
|
|
18
|
-
| With a [squishsum
|
|
18
|
+
| With a [squishsum harness](harnesses.md): two valid samples that differ, with no judge or a judge that rejects both | Raises `Squishling::DisagreementError` (an `InvalidOutputError`), carrying both typed `candidates` |
|
|
19
19
|
|
|
20
20
|
Squishling validates output itself with [json_schemer](https://github.com/davishmcclurg/json_schemer), because
|
|
21
21
|
RubyLLM doesn't, and some providers don't enforce strict mode. When the next attempt is on the same step (same
|
|
@@ -42,6 +42,9 @@ Model output can echo sensitive input, so the raw response is kept in one place:
|
|
|
42
42
|
the message, the logger, and the next attempt's prompt.
|
|
43
43
|
- A fixed message when a response contains a value JSON can't represent (`1e400` parses to `Infinity`, or a string
|
|
44
44
|
with invalid UTF-8).
|
|
45
|
+
- For a failed provider request (`LLMError`, and the `ConfigurationError` for a 400, bad credentials, or a forbidden
|
|
46
|
+
key), only the error class and HTTP status. The provider's own message can echo request content or part of an API
|
|
47
|
+
key, so it stays on `cause` (`error.cause.message`) for you to read or log deliberately.
|
|
45
48
|
- At most the first 20 of these errors, plus a count of the rest, so a very large malformed response can't flood
|
|
46
49
|
the retry message or the log.
|
|
47
50
|
- For a [`DisagreementError`](harnesses.md#when-the-harness-fails), only what Squishling wrote: a `:judgment` judge's
|
|
@@ -69,7 +72,7 @@ end
|
|
|
69
72
|
|---|---|
|
|
70
73
|
| `output` | The raw response: a Hash or String, or `nil` when the provider call itself failed |
|
|
71
74
|
| `error` | `nil` for an accepted attempt. Otherwise the `InvalidOutputError` (with `errors` and `raw`) or `LLMError` that ended the attempt |
|
|
72
|
-
| `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length; under a harness, the sample's or judge's own attempts), `final` (the last attempt), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when RubyLLM reports them) |
|
|
75
|
+
| `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length; under a harness, the sample's or judge's own attempts), `final` (the last attempt), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when RubyLLM reports them), `role` (`nil` for an escalation attempt, `:sample`, or `:judge`), `sample` (`"a"`, `"b"`, ... for a sample's attempt, else `nil`) |
|
|
73
76
|
|
|
74
77
|
- Set it globally with `config.squawk`, per class with `squishling squawk: ...`, or per method with
|
|
75
78
|
`squish :triage, squawk: ...`. The method's hook wins over the class's, which wins over the configured one;
|
|
@@ -79,7 +82,8 @@ end
|
|
|
79
82
|
- It runs inline, so keep it quick. Exceptions it raises propagate unchanged and fail the call, like any of your own
|
|
80
83
|
code, so rescue inside the hook if an outage in your tracing tool shouldn't.
|
|
81
84
|
- `output` is the same object the result is built from, so treat it as read-only.
|
|
82
|
-
- Under a [harness](harnesses.md), it runs for every sample and judge attempt too, each with its own `input
|
|
85
|
+
- Under a [harness](harnesses.md), it runs for every sample and judge attempt too, each with its own `input`, and
|
|
86
|
+
`role`/`sample` say which one it was. A [custom harness](harnesses.md#reporting-to-squawk) reports the same way.
|
|
83
87
|
- It isn't called for a `ConfigurationError`, for deterministic and fallback returns, or for an attempt where your own
|
|
84
88
|
`squish_validate` raises (that exception propagates first).
|
|
85
89
|
|
|
@@ -118,7 +122,7 @@ end
|
|
|
118
122
|
| `Squishling::Error` | Base class for everything below. Raised directly for misuse at call time, e.g. `result` or `squish!` called outside a squished method |
|
|
119
123
|
| `Squishling::ConfigurationError` | Missing purpose or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_to_purpose` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (400) |
|
|
120
124
|
| `Squishling::InvalidOutputError` | LLM output still invalid (schema or `squish_validate`) after every attempt in the escalation, or a deterministic/fallback return that doesn't match the schema |
|
|
121
|
-
| `Squishling::DisagreementError` | A subclass of `InvalidOutputError`: a [squishsum
|
|
125
|
+
| `Squishling::DisagreementError` | A subclass of `InvalidOutputError`: a [squishsum harness](harnesses.md#when-the-harness-fails)'s samples disagreed and no judge accepted either. `candidates`, `verdict`, and `reason` describe what happened |
|
|
122
126
|
| `Squishling::LLMError` | The provider call failed on the last attempt in the escalation, after RubyLLM's own retries, including context-length errors (`cause` holds the original) |
|
|
123
127
|
|
|
124
128
|
Errors raised by your own Ruby code are not wrapped.
|