squishling 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +140 -0
- data/README.md +118 -31
- data/docs/configuration.md +75 -21
- data/docs/coolhand.md +66 -0
- data/docs/failures.md +73 -9
- data/docs/harnesses.md +326 -0
- data/docs/measuring-tokens.md +140 -0
- data/docs/naming.md +43 -0
- data/docs/observers.md +80 -0
- data/docs/routing.md +36 -27
- data/docs/schemas.md +24 -5
- data/lib/squishling/appendices.rb +3 -3
- data/lib/squishling/class_methods.rb +60 -29
- data/lib/squishling/collisions.rb +54 -0
- data/lib/squishling/configuration.rb +32 -0
- data/lib/squishling/definition.rb +56 -27
- data/lib/squishling/errors.rb +22 -1
- data/lib/squishling/harness/base.rb +136 -0
- data/lib/squishling/harness/escalation.rb +22 -0
- data/lib/squishling/harness/judged_squishsum.rb +83 -0
- data/lib/squishling/harness/option_types.rb +93 -0
- data/lib/squishling/harness/registry.rb +69 -0
- data/lib/squishling/harness/squishsum.rb +22 -0
- data/lib/squishling/harness.rb +79 -0
- data/lib/squishling/invoker.rb +26 -188
- data/lib/squishling/judge.rb +186 -0
- data/lib/squishling/llm_client.rb +180 -0
- data/lib/squishling/model_path.rb +82 -25
- data/lib/squishling/observer.rb +100 -0
- data/lib/squishling/observers/coolhand.rb +86 -0
- data/lib/squishling/output_check.rb +109 -0
- data/lib/squishling/params.rb +47 -6
- data/lib/squishling/rotation.rb +26 -0
- data/lib/squishling/router.rb +8 -2
- data/lib/squishling/run/messages.rb +42 -0
- data/lib/squishling/run.rb +296 -0
- data/lib/squishling/source.rb +5 -5
- data/lib/squishling/squawk.rb +52 -0
- data/lib/squishling/version.rb +1 -1
- data/lib/squishling.rb +56 -7
- metadata +25 -3
data/docs/naming.md
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Naming and collisions
|
|
2
|
+
|
|
3
|
+
`include Squishling` adds a small set of methods to your class. This page lists them and explains what happens
|
|
4
|
+
when one of those names is already taken.
|
|
5
|
+
|
|
6
|
+
## What Squishling adds
|
|
7
|
+
|
|
8
|
+
| Where | Names |
|
|
9
|
+
|---|---|
|
|
10
|
+
| Class-level DSL | `squishling`, `purpose`, `append_to_purpose`, `output_schema`, `squish`, `squish_when`, `squish_fallback`, `squish_validate`, `squish_context`, `squished_methods`, `call`, and the `squishling_*` readers |
|
|
11
|
+
| Instance | `squishling_result`, `squish!`, and `result` (an alias for `squishling_result`) |
|
|
12
|
+
|
|
13
|
+
Methods your class defines itself always win over these, whether you define them before or after the `include`.
|
|
14
|
+
Only methods your class *inherits* can be shadowed.
|
|
15
|
+
|
|
16
|
+
## Inherited collisions raise
|
|
17
|
+
|
|
18
|
+
If a parent class or an earlier-included module already defines a class-level DSL method, `squish!` or
|
|
19
|
+
`squishling_result`, `include Squishling` raises `Squishling::ConfigurationError` listing every clash and where
|
|
20
|
+
it comes from:
|
|
21
|
+
|
|
22
|
+
```
|
|
23
|
+
ConfigurationError: MyApp inherits methods that Squishling would override: MyApp.call (from Sinatra::Base's
|
|
24
|
+
class methods). Include Squishling in a plain Ruby class instead ...
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
This is deliberate. Squishling's modules sit ahead of inherited ones, so without the check
|
|
28
|
+
`Sinatra::Base.call(env)` (the Rack entry point), or any `call` or `purpose` your parent class defines, would be
|
|
29
|
+
replaced silently. Put Squishling in a plain Ruby class and have your framework object call it instead.
|
|
30
|
+
|
|
31
|
+
## `result` is skipped when taken
|
|
32
|
+
|
|
33
|
+
`result` is a convenience for `squishling_result`. If the class already has a `result` (its own or inherited), it is
|
|
34
|
+
left alone and Squishling does not add the alias. Use `squishling_result(...)` instead; it is always available.
|
|
35
|
+
|
|
36
|
+
**ActiveRecord caveat.** Column readers are generated lazily, so Squishling can't see a `result` column when you
|
|
37
|
+
include it. Its `result` would then shadow the column reader. On a model with a `result` column, call
|
|
38
|
+
`squishling_result` (or keep Squishling out of the model and use a plain class).
|
|
39
|
+
|
|
40
|
+
## Not a collision
|
|
41
|
+
|
|
42
|
+
ActiveSupport adds `String#squish` and `String#squish!`. Those live on `String`; Squishling's `squish!` lives on
|
|
43
|
+
your class, so they never meet. They only look alike when reading code in a Rails app.
|
data/docs/observers.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# Observers
|
|
2
|
+
|
|
3
|
+
An observer wraps every provider request Squishling makes, so another tool can see it while it happens: open a
|
|
4
|
+
tracing span, tag the request, set a thread-local. It is the around-style counterpart to [`squawk`](failures.md#observing-every-attempt-squawk),
|
|
5
|
+
which reports each attempt only after it is over.
|
|
6
|
+
|
|
7
|
+
Squishling ships one, for [Coolhand Labs](coolhand.md), and uses it automatically when the `coolhand` gem is configured.
|
|
8
|
+
Any object that responds to `around(call)` is an observer:
|
|
9
|
+
|
|
10
|
+
```ruby
|
|
11
|
+
class TracingObserver
|
|
12
|
+
def around(call)
|
|
13
|
+
Tracer.in_span("squishling #{call.label}", attributes: { "model" => call.model, "role" => call.role.to_s }) do
|
|
14
|
+
yield
|
|
15
|
+
end
|
|
16
|
+
end
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
Squishling.configure { |config| config.observer = TracingObserver.new }
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
`around` must yield exactly once. The block makes the request, so the observer runs around it, on the thread that
|
|
23
|
+
makes it: for the concurrent samples of the squishsum harnesses, that is a worker thread, so thread-locals set by the
|
|
24
|
+
observer are visible to the HTTP call. What `around` returns is ignored. One that never yields, or yields twice,
|
|
25
|
+
raises `ConfigurationError`.
|
|
26
|
+
|
|
27
|
+
## What `yield` returns
|
|
28
|
+
|
|
29
|
+
The block returns a `Squishling::Observer::Usage`: the request's `input_tokens` and `output_tokens` (nil when the
|
|
30
|
+
provider didn't report them) and the `model` that answered. That is enough to meter spend:
|
|
31
|
+
|
|
32
|
+
```ruby
|
|
33
|
+
class SpendObserver
|
|
34
|
+
def around(call)
|
|
35
|
+
usage = yield
|
|
36
|
+
Metrics.increment("squishling.tokens.in", usage.input_tokens.to_i, tags: ["label:#{call.label}"])
|
|
37
|
+
Metrics.increment("squishling.tokens.out", usage.output_tokens.to_i, tags: ["label:#{call.label}"])
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Usage never includes the model's output; Squishling returns the real response to its own caller, whatever `around`
|
|
43
|
+
returns. If the request fails, the error is raised through `yield`, so a `begin/ensure` in `around` still runs.
|
|
44
|
+
|
|
45
|
+
## `config.observer`
|
|
46
|
+
|
|
47
|
+
| Value | Meaning |
|
|
48
|
+
|---|---|
|
|
49
|
+
| `nil` (default) | The Coolhand observer when `coolhand` is loaded and configured, otherwise none |
|
|
50
|
+
| `:coolhand` | The Coolhand observer, always. Raises `ConfigurationError` if the gem is missing or too old |
|
|
51
|
+
| any object responding to `around` | That observer |
|
|
52
|
+
| `false` | None, even when Coolhand is configured |
|
|
53
|
+
|
|
54
|
+
Anything else raises `ConfigurationError` when you assign it.
|
|
55
|
+
|
|
56
|
+
## The `call`
|
|
57
|
+
|
|
58
|
+
`around` receives a `Squishling::Observer::Call` (a `Data`) describing the request:
|
|
59
|
+
|
|
60
|
+
| Field | |
|
|
61
|
+
|---|---|
|
|
62
|
+
| `label` | `"Class#method"` |
|
|
63
|
+
| `purpose` | The purpose text sent as the system prompt (after `purpose` procs and `append_to_purpose` sections are resolved) |
|
|
64
|
+
| `handed_off` | `true` when a `squish!` call sent the call to the LLM (typically from a `rescue`), `false` when `squish_when` or a missing implementation did |
|
|
65
|
+
| `harness` | The harness type (`:escalation`, `:squishsum`, `:judged_squishsum`, or a [custom one](harnesses.md#custom-harnesses)) |
|
|
66
|
+
| `role` | `nil` for an escalation attempt, `:sample` or `:judge`, as in `squawk`'s `metadata` |
|
|
67
|
+
| `sample` | `"a"`, `"b"`, ... for a sample, else `nil` |
|
|
68
|
+
| `attempt`, `attempts` | This request's 1-based position in the steps it was asked on, and how many there are: the escalation's expanded steps (or, for a custom harness, those passed to `run.escalate`), the sample's stage, or the judge's steps |
|
|
69
|
+
| `model`, `provider` | The step's model and provider (`provider` is `nil` unless declared) |
|
|
70
|
+
| `request` | `:chat`, or `:judgment` for a System One judge |
|
|
71
|
+
|
|
72
|
+
It carries metadata only. The method's inputs, its context, and the model's output never reach an observer: it gets
|
|
73
|
+
the request's token [`Usage`](#what-yield-returns) back, and `squawk` is the route for output.
|
|
74
|
+
|
|
75
|
+
## Errors
|
|
76
|
+
|
|
77
|
+
An observer is your code: an exception it raises propagates unchanged, and is never retried, escalated, or handed to
|
|
78
|
+
`squish_fallback`. Squishling maps a failed request onto `LLMError` or `ConfigurationError` inside the block, so `around` sees those and
|
|
79
|
+
can rescue them or let them pass. If `around` rescues one and returns normally, Squishling raises it again: an
|
|
80
|
+
observer can watch a failure, not swallow it.
|
data/docs/routing.md
CHANGED
|
@@ -35,24 +35,27 @@ end
|
|
|
35
35
|
def summarize(text) = raise NotImplementedError # elastic until someone writes it
|
|
36
36
|
```
|
|
37
37
|
|
|
38
|
-
When the Ruby implementation runs,
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
38
|
+
When the Ruby implementation runs, whatever it returns (a `Hash`, `nil`, a string, another schema's result, …)
|
|
39
|
+
is validated against the schema and turned into the same typed result the LLM path produces.
|
|
40
|
+
`squishling_result(...)` does the same explicitly; `result(...)` is a convenience alias, skipped when the class
|
|
41
|
+
already has a `result` (see [Naming and collisions](naming.md)). Invalid deterministic output raises
|
|
42
|
+
`Squishling::InvalidOutputError` too, so a hardened path can't silently drift from the contract.
|
|
42
43
|
|
|
43
44
|
A `NotImplementedError` raised anywhere inside the method, including from code it calls, also routes to the
|
|
44
45
|
LLM.
|
|
45
46
|
|
|
46
|
-
Each call is routed on its own, including a squished method that calls itself on smaller inputs
|
|
47
|
+
Each call is routed on its own, including a squished method that calls itself on smaller inputs (those
|
|
48
|
+
recursive calls must return schema-valid values too). A subclass
|
|
47
49
|
override that calls `super` is one call: it's routed once, at the subclass. A call to the method from its own
|
|
48
|
-
`squish_when` or `squish_fallback` (or
|
|
49
|
-
implementation, so a fallback can hand the input back to Ruby with `call(**inputs)`.
|
|
50
|
+
`squish_when` or `squish_fallback` (or a purpose proc) isn't routed again: it runs the Ruby
|
|
51
|
+
implementation, so a fallback can hand the input back to Ruby with `call(**inputs)`. These inner calls
|
|
52
|
+
return a `Hash` as the typed result and any other value unchanged; only the outermost return is validated in full.
|
|
50
53
|
|
|
51
54
|
## Hardening a path
|
|
52
55
|
|
|
53
56
|
This is the workflow from [Elastic Software](https://everythingengineer.substack.com/p/beginners-write-software-with-ai):
|
|
54
57
|
|
|
55
|
-
1. Ship the class with
|
|
58
|
+
1. Ship the class with purpose and a schema but no implementation. Every call goes to the LLM.
|
|
56
59
|
2. Watch which inputs carry the volume. `squished?` on each result tells you which path served it.
|
|
57
60
|
3. Write Ruby for the high-volume cases and narrow `squish_when` so only the rest go to the LLM.
|
|
58
61
|
|
|
@@ -68,7 +71,7 @@ class TicketTriager
|
|
|
68
71
|
|
|
69
72
|
squish_context :customer_tier, :product # instance state sent alongside the arguments
|
|
70
73
|
|
|
71
|
-
squish :triage,
|
|
74
|
+
squish :triage, purpose: "Assign a priority and team.", model: "claude-haiku-4-5" do
|
|
72
75
|
string :priority, enum: %w[low med high]
|
|
73
76
|
string :team
|
|
74
77
|
end
|
|
@@ -83,10 +86,11 @@ end
|
|
|
83
86
|
|
|
84
87
|
`squish` accepts:
|
|
85
88
|
|
|
86
|
-
- `
|
|
87
|
-
- `
|
|
89
|
+
- `purpose:` (replaces the class's)
|
|
90
|
+
- `append_to_purpose:` (added to the class's, see [Appending to the purpose](#appending-to-the-purpose))
|
|
88
91
|
- `output_schema:` (or a schema block)
|
|
89
92
|
- `model:` or `escalation:`, `provider:`, and `params:` (generation params; see [Configuration](configuration.md))
|
|
93
|
+
- `harness:` (see [Harnesses](harnesses.md), including your own)
|
|
90
94
|
- `when:`, a predicate proc
|
|
91
95
|
- `validate:` and `fallback:` (see [Failure handling](failures.md))
|
|
92
96
|
|
|
@@ -103,8 +107,8 @@ from the method.
|
|
|
103
107
|
class InvoiceParser
|
|
104
108
|
include Squishling
|
|
105
109
|
|
|
106
|
-
|
|
107
|
-
|
|
110
|
+
purpose "Extract invoice fields from the client's raw data."
|
|
111
|
+
append_to_purpose "Here is the Ruby that parses well-formed invoices, for context on the logic and goals:",
|
|
108
112
|
self
|
|
109
113
|
output_schema do
|
|
110
114
|
string :invoice_number
|
|
@@ -115,7 +119,7 @@ class InvoiceParser
|
|
|
115
119
|
parsed = AcmeParser.parse(data)
|
|
116
120
|
result(invoice_number: parsed.id, total: parsed.sum)
|
|
117
121
|
rescue AcmeParser::ParseError => e
|
|
118
|
-
squish!(
|
|
122
|
+
squish!(append_to_purpose: "The Ruby parser above failed on this input; the error is in the context.",
|
|
119
123
|
context: { parse_error: e })
|
|
120
124
|
end
|
|
121
125
|
end
|
|
@@ -126,9 +130,10 @@ end
|
|
|
126
130
|
| Option | Effect |
|
|
127
131
|
|---|---|
|
|
128
132
|
| `context:` | A Hash sent under `"context"` with any `squish_context` values (a same-named key wins). Exceptions are sent as `{ "class", "message" }`, never their backtrace. |
|
|
129
|
-
| `
|
|
130
|
-
| `
|
|
133
|
+
| `append_to_purpose:` | Added to the declared sections; `false` (alone or first in an Array) drops them for this call |
|
|
134
|
+
| `purpose:` | Replaces the purpose |
|
|
131
135
|
| `model:` or `escalation:`, `provider:`, `params:` | E.g. send this call to a stronger model, or a whole [escalation](configuration.md#models-and-escalation), when Ruby fails. A `provider:` needs a `model:` or `escalation:`; `params:` merge key by key over the declared ones. |
|
|
136
|
+
| `harness:` | Replaces the declared [harness](harnesses.md), e.g. `harness: :judged_squishsum` to double-check a hand-off |
|
|
132
137
|
|
|
133
138
|
- **The output schema can't be overridden.** The call still returns the method's result type.
|
|
134
139
|
- **Failures** go through the normal LLM path: a declared `squish_fallback` is used, otherwise
|
|
@@ -148,9 +153,9 @@ end
|
|
|
148
153
|
or in a parent implementation reached through `super`. Calling it anywhere else, or from a `squish_fallback`
|
|
149
154
|
(which would loop), raises `Squishling::Error`.
|
|
150
155
|
|
|
151
|
-
## Appending to the
|
|
156
|
+
## Appending to the purpose
|
|
152
157
|
|
|
153
|
-
`
|
|
158
|
+
`append_to_purpose` adds sections to the system prompt after the purpose. It takes items, an Array of
|
|
154
159
|
them, or a block (treated as a Proc item). Each item is one of:
|
|
155
160
|
|
|
156
161
|
| Item | Sent as |
|
|
@@ -164,19 +169,19 @@ them, or a block (treated as a Proc item). Each item is one of:
|
|
|
164
169
|
class InvoiceParser
|
|
165
170
|
include Squishling
|
|
166
171
|
|
|
167
|
-
|
|
168
|
-
|
|
172
|
+
append_to_purpose "The Ruby that parses well-formed invoices:", self, AcmeParser
|
|
173
|
+
append_to_purpose -> { "This client's invoices are in #{currency}." }
|
|
169
174
|
|
|
170
|
-
squish :summarize,
|
|
175
|
+
squish :summarize, append_to_purpose: false do # no appendices for this method
|
|
171
176
|
string :summary
|
|
172
177
|
end
|
|
173
178
|
end
|
|
174
179
|
```
|
|
175
180
|
|
|
176
|
-
The same option goes in the class-wide call: `squishling
|
|
181
|
+
The same option goes in the class-wide call: `squishling append_to_purpose: ["...", self]`.
|
|
177
182
|
|
|
178
|
-
Sections are added down the chain: class, then subclass, then `squish :name,
|
|
179
|
-
`squish!(
|
|
183
|
+
Sections are added down the chain: class, then subclass, then `squish :name, append_to_purpose:`, then
|
|
184
|
+
`squish!(append_to_purpose:)`. `false` drops everything declared above it, so `[false, "Only this."]`
|
|
180
185
|
replaces it.
|
|
181
186
|
|
|
182
187
|
Source is read with Ruby's own parser (Prism) the first time it's needed and cached. Some limits:
|
|
@@ -195,8 +200,8 @@ Source is read with Ruby's own parser (Prism) the first time it's needed and cac
|
|
|
195
200
|
|
|
196
201
|
## What the LLM sees
|
|
197
202
|
|
|
198
|
-
- **System prompt:** your
|
|
199
|
-
`
|
|
203
|
+
- **System prompt:** your purpose (a String, or a Proc evaluated against the instance), then any
|
|
204
|
+
`append_to_purpose` sections, then a short note describing the input format.
|
|
200
205
|
- **User message:** JSON with the method's arguments, mapped to their parameter names. Any
|
|
201
206
|
`squish_context` values, and a `squish!` call's `context:`, go under `"context"`:
|
|
202
207
|
|
|
@@ -205,11 +210,15 @@ Source is read with Ruby's own parser (Prism) the first time it's needed and cac
|
|
|
205
210
|
"context": { "customer_tier": "enterprise", "product": "API" } }
|
|
206
211
|
```
|
|
207
212
|
|
|
213
|
+
- **Squishsum harnesses:** both samples get exactly this input, and a [judge](harnesses.md#the-judge) gets
|
|
214
|
+
the same purpose and input plus both samples' outputs.
|
|
208
215
|
- **Retries and escalation:** another attempt of the same step gets the validation errors in the same conversation. A
|
|
209
216
|
later [escalation](configuration.md#models-and-escalation) step, which may be a different provider (say, local
|
|
210
217
|
Ollama, then hosted Anthropic Claude), gets the same JSON plus the previous model's rejected output and the
|
|
211
218
|
errors, including any messages your [`squish_validate`](failures.md#output-checks-squish_validate) check
|
|
212
|
-
returned.
|
|
219
|
+
returned. The rejected output is model-generated and is truncated to 4,000 characters; set
|
|
220
|
+
`forward_rejected: false` on a step to start it from the original input only, without the rejected output or
|
|
221
|
+
its errors. Don't put data in those messages that you wouldn't send as an argument.
|
|
213
222
|
|
|
214
223
|
`squish_context` names are read from a method of that name if there is one, otherwise from the instance
|
|
215
224
|
variable. Only context you name is sent. Instance variables are never dumped wholesale, so API clients,
|
data/docs/schemas.md
CHANGED
|
@@ -30,6 +30,23 @@ output_schema(
|
|
|
30
30
|
)
|
|
31
31
|
```
|
|
32
32
|
|
|
33
|
+
## Describing fields
|
|
34
|
+
|
|
35
|
+
Every DSL type, and `optional`, takes a `description:`. It is sent to the provider as guidance for that field
|
|
36
|
+
and isn't enforced; validation is by type and shape only.
|
|
37
|
+
|
|
38
|
+
**Best practice:** put per-field guidance here rather than in the `purpose`. The description travels with the
|
|
39
|
+
field in the schema the model fills in, so the instruction sits right where the value goes and next to the type
|
|
40
|
+
or `enum` it explains. The purpose can then stay short and about the goal. For rules that must hold, use an
|
|
41
|
+
`enum` or `squish_validate`; a description is a request, not a guarantee.
|
|
42
|
+
|
|
43
|
+
```ruby
|
|
44
|
+
output_schema do
|
|
45
|
+
string :language, description: "Lowercase language name, e.g. 'ruby'."
|
|
46
|
+
optional(:regex, description: "The proposed regex source, without delimiters or flags.") { string }
|
|
47
|
+
end
|
|
48
|
+
```
|
|
49
|
+
|
|
33
50
|
## Strict schemas only
|
|
34
51
|
|
|
35
52
|
Squishling only supports **strict** output schemas: it always asks RubyLLM for strict structured output, and a
|
|
@@ -59,7 +76,8 @@ r.squished? # => true when it came from the LLM
|
|
|
59
76
|
```
|
|
60
77
|
|
|
61
78
|
On the deterministic path, return a `Hash` or build the result with `result(...)`. Either way it's validated
|
|
62
|
-
against the schema
|
|
79
|
+
against the schema, and so is anything else you return (`nil`, a string, another schema's result), which raises
|
|
80
|
+
`Squishling::InvalidOutputError` unless it matches.
|
|
63
81
|
|
|
64
82
|
## Optional vs. empty
|
|
65
83
|
|
|
@@ -69,7 +87,7 @@ field nullable with the DSL's `optional`. The key stays required, but its value
|
|
|
69
87
|
```ruby
|
|
70
88
|
output_schema do
|
|
71
89
|
array :symptoms, of: :string # always a list; [] means "none"
|
|
72
|
-
optional :allergies
|
|
90
|
+
optional :allergies, description: "[] if the record says none; null if it doesn't mention allergies" do
|
|
73
91
|
array of: :string
|
|
74
92
|
end
|
|
75
93
|
optional :vitals do # a typed Data object, or nil
|
|
@@ -83,8 +101,8 @@ end
|
|
|
83
101
|
`optional` produces `anyOf: [<schema>, {type: "null"}]`, which is strict-compatible. Squishling types the non-null
|
|
84
102
|
branch as usual: `vitals` is a `Data` object or `nil`, and a list of objects is a list of `Data` objects. In a raw
|
|
85
103
|
JSON Schema, `type: ["array", "null"]` works too. A union with more than one non-null branch is ambiguous, so its
|
|
86
|
-
values come back as plain hashes. If you need the model to tell "none" apart from "not mentioned", say so in
|
|
87
|
-
|
|
104
|
+
values come back as plain hashes. If you need the model to tell "none" apart from "not mentioned", say so in the field's
|
|
105
|
+
`description:`, as `allergies` does above.
|
|
88
106
|
|
|
89
107
|
## Contracts beyond the schema
|
|
90
108
|
|
|
@@ -110,7 +128,8 @@ Providers' strict modes don't support conditional keywords (`if`/`then`/`else` f
|
|
|
110
128
|
`dependentSchemas` from `dependent`), so Squishling keeps them out of the schema it sends and enforces them itself.
|
|
111
129
|
Output that breaks a rule is rejected like any other invalid output: the call moves on to the next attempt in its
|
|
112
130
|
[escalation](configuration.md#models-and-escalation) with the errors. The model doesn't see these rules
|
|
113
|
-
up front, so state them in your
|
|
131
|
+
up front, so state them in the `description:` of the field they govern (here `status`) or in your purpose. A
|
|
132
|
+
description declared inside a `given` block is local-only too, so it is never sent.
|
|
114
133
|
|
|
115
134
|
For anything a schema can't express (sums, lookups against your data), use
|
|
116
135
|
[`squish_validate`](failures.md#output-checks-squish_validate).
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module Squishling
|
|
4
|
-
#
|
|
4
|
+
# append_to_purpose: extra system-prompt sections placed after the purpose, layered
|
|
5
5
|
# class -> subclass -> method -> call. Each level adds to the levels above it; `false` drops them.
|
|
6
6
|
module Appendices
|
|
7
7
|
ITEM_TYPES = [String, Module, Method, UnboundMethod, Proc].freeze
|
|
@@ -29,7 +29,7 @@ module Squishling
|
|
|
29
29
|
|
|
30
30
|
values = receiver.instance_exec(&item)
|
|
31
31
|
(values.is_a?(Array) ? values : [values]).select(&:itself).map do |value|
|
|
32
|
-
check!(value, "#{label}
|
|
32
|
+
check!(value, "#{label} append_to_purpose proc", procs: false)
|
|
33
33
|
render_item(value)
|
|
34
34
|
end
|
|
35
35
|
end
|
|
@@ -43,7 +43,7 @@ module Squishling
|
|
|
43
43
|
allowed = procs ? ITEM_TYPES : ITEM_TYPES - [Proc]
|
|
44
44
|
return if allowed.any? { |type| item.is_a?(type) }
|
|
45
45
|
|
|
46
|
-
raise ConfigurationError, "#{label}:
|
|
46
|
+
raise ConfigurationError, "#{label}: append_to_purpose items must be Strings, classes or modules, " \
|
|
47
47
|
"#{procs ? 'methods, or procs' : 'or methods'} (got #{describe(item)})"
|
|
48
48
|
end
|
|
49
49
|
|
|
@@ -6,27 +6,49 @@ module Squishling
|
|
|
6
6
|
DEFAULT_METHOD = :call
|
|
7
7
|
|
|
8
8
|
# Set class-wide options in one call:
|
|
9
|
-
# squishling model: "claude-sonnet-5-5",
|
|
9
|
+
# squishling model: "claude-sonnet-5-5", purpose: "...", output_schema: MySchema
|
|
10
10
|
# model: is a single attempt; escalation: is a list of models tried in order (see ModelPath):
|
|
11
11
|
# squishling escalation: [{ model: "claude-haiku-4-5", attempts: 2 }, "claude-sonnet-5-5", "claude-opus-5-5"]
|
|
12
12
|
# Pass provider: alongside model: for models missing from RubyLLM's registry, e.g.
|
|
13
|
-
# squishling model: "gpt-
|
|
13
|
+
# squishling model: "gpt-7-preview", provider: :openai
|
|
14
|
+
# provider: needs a model: or escalation: (declared on the class), and a subclass's model never inherits its
|
|
15
|
+
# parent's provider.
|
|
14
16
|
# params: are generation params merged over the configured defaults (see Configuration#default_params):
|
|
15
17
|
# squishling params: { temperature: 0.1, top_p: 0.9 }
|
|
16
|
-
#
|
|
17
|
-
# squishling
|
|
18
|
-
|
|
19
|
-
|
|
18
|
+
# append_to_purpose: adds sections after the purpose (see #append_to_purpose):
|
|
19
|
+
# squishling append_to_purpose: ["The Ruby that handles well-formed input:", self]
|
|
20
|
+
# harness: chooses how the escalation is used (see Harness):
|
|
21
|
+
# squishling harness: :judged_squishsum
|
|
22
|
+
# squawk: is called after every LLM attempt with the raw output (see Configuration#squawk); false silences
|
|
23
|
+
# an inherited one:
|
|
24
|
+
# squishling squawk: ->(output:, metadata:, error:) { Tracer.record(output, metadata, error) }
|
|
25
|
+
def squishling(model: nil, escalation: nil, provider: nil, params: nil, harness: nil, purpose: nil,
|
|
26
|
+
append_to_purpose: nil, output_schema: nil, squawk: nil)
|
|
20
27
|
path = ModelPath.declare(model, escalation, to_s)
|
|
28
|
+
# A provider belongs to the model declared at the same level. Without a model here it would apply to
|
|
29
|
+
# nothing (the model comes from the config or a parent class), so it is an error unless this class already
|
|
30
|
+
# declared its own.
|
|
31
|
+
if provider && !path && !instance_variable_defined?(:@squishling_model_path)
|
|
32
|
+
raise ConfigurationError, "#{self}: provider: needs a model: or escalation: declared on this class"
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# Stored (even as nil) whenever a model is, so a subclass that declares its own model never inherits a
|
|
36
|
+
# provider meant for its parent's.
|
|
21
37
|
@squishling_model_path = path if path
|
|
22
|
-
@squishling_provider = provider if provider
|
|
38
|
+
@squishling_provider = provider if provider || path
|
|
23
39
|
@squishling_params = Params.normalize(params, "#{self} params") if params
|
|
24
|
-
|
|
25
|
-
self.
|
|
40
|
+
@squishling_harness = Harness.normalize(harness, to_s) unless harness.nil?
|
|
41
|
+
self.purpose(purpose) if purpose
|
|
42
|
+
self.append_to_purpose(append_to_purpose) unless append_to_purpose.nil?
|
|
26
43
|
self.output_schema(output_schema) if output_schema
|
|
44
|
+
@squishling_squawk = Squawk.validate(squawk, to_s) unless squawk.nil?
|
|
27
45
|
self
|
|
28
46
|
end
|
|
29
47
|
|
|
48
|
+
def squishling_squawk
|
|
49
|
+
squishling_lookup(:@squishling_squawk)
|
|
50
|
+
end
|
|
51
|
+
|
|
30
52
|
# The class's (or nearest ancestor's) model or escalation, as normalized steps.
|
|
31
53
|
def squishling_model_path
|
|
32
54
|
squishling_lookup(:@squishling_model_path)
|
|
@@ -36,33 +58,37 @@ module Squishling
|
|
|
36
58
|
squishling_lookup(:@squishling_provider)
|
|
37
59
|
end
|
|
38
60
|
|
|
61
|
+
def squishling_harness
|
|
62
|
+
squishling_lookup(:@squishling_harness)
|
|
63
|
+
end
|
|
64
|
+
|
|
39
65
|
# Generation params merged down the inheritance chain, so a subclass overrides individual keys.
|
|
40
66
|
def squishling_params
|
|
41
67
|
squishling_inherited(:squishling_params, {}).merge(@squishling_params || {})
|
|
42
68
|
end
|
|
43
69
|
|
|
44
70
|
# The system prompt. A String, or a Proc evaluated against the instance.
|
|
45
|
-
def
|
|
46
|
-
return squishling_lookup(:@
|
|
71
|
+
def purpose(text = nil, &block)
|
|
72
|
+
return squishling_lookup(:@squishling_purpose) if text.nil? && block.nil?
|
|
47
73
|
|
|
48
|
-
@
|
|
74
|
+
@squishling_purpose = block || text
|
|
49
75
|
end
|
|
50
76
|
|
|
51
|
-
# Sections appended to the system prompt after the
|
|
77
|
+
# Sections appended to the system prompt after the purpose, added to by subclasses, `squish`, and
|
|
52
78
|
# `squish!`. Items: Strings; a class or module (`self` for this class) or a method (`instance_method(:call)`),
|
|
53
79
|
# sent as its Ruby source; or a Proc evaluated against the instance. `false` drops inherited items.
|
|
54
|
-
#
|
|
55
|
-
#
|
|
56
|
-
def
|
|
80
|
+
# append_to_purpose "Here is the Ruby that parses well-formed invoices:", self
|
|
81
|
+
# append_to_purpose { "This client's invoices are in #{currency}." }
|
|
82
|
+
def append_to_purpose(*items, &block)
|
|
57
83
|
items = items.first if items.size == 1 && items.first.is_a?(Array)
|
|
58
84
|
items += [block] if block
|
|
59
|
-
(@
|
|
85
|
+
(@squishling_append_to_purpose ||= []).concat(Appendices.normalize(items, "#{self} append_to_purpose"))
|
|
60
86
|
self
|
|
61
87
|
end
|
|
62
88
|
|
|
63
89
|
# Every level's items in declaration order, `false` markers included (see Appendices.resolve).
|
|
64
|
-
def
|
|
65
|
-
squishling_inherited(:
|
|
90
|
+
def squishling_append_to_purpose
|
|
91
|
+
squishling_inherited(:squishling_append_to_purpose, []) + (@squishling_append_to_purpose || [])
|
|
66
92
|
end
|
|
67
93
|
|
|
68
94
|
# The output format: a Schematist::Schema subclass (RubyLLM::Schema with the ruby_llm-schema shim),
|
|
@@ -85,7 +111,7 @@ module Squishling
|
|
|
85
111
|
|
|
86
112
|
# Called with the error and the method's inputs (as keywords) when the LLM path fails with an
|
|
87
113
|
# InvalidOutputError or LLMError, evaluated against the instance. Its return value is used as
|
|
88
|
-
# the result (
|
|
114
|
+
# the result (validated against the schema and typed like a deterministic return); re-raise to propagate.
|
|
89
115
|
# squish_fallback { |error, **inputs| { priority: "medium", team: "support" } }
|
|
90
116
|
def squish_fallback(&block)
|
|
91
117
|
@squishling_fallback = block
|
|
@@ -118,21 +144,26 @@ module Squishling
|
|
|
118
144
|
end
|
|
119
145
|
|
|
120
146
|
# Make methods elastic. Each may override the class-level settings:
|
|
121
|
-
# squish :triage,
|
|
122
|
-
# fallback: ->(error, **) { { priority: "medium" } },
|
|
123
|
-
# validate: ->(result, **) { "team is required" if result.team.empty? }
|
|
147
|
+
# squish :triage, purpose: "...", escalation: %w[claude-haiku-4-5 claude-sonnet-5-5], when: ->(**) { true },
|
|
148
|
+
# fallback: ->(error, **) { { priority: "medium" } }, append_to_purpose: [...],
|
|
149
|
+
# validate: ->(result, **) { "team is required" if result.team.empty? },
|
|
150
|
+
# harness: :squishsum, squawk: ->(output:, error:, **) { Tracer.record(output, error) } do
|
|
124
151
|
# string :priority
|
|
125
152
|
# end
|
|
126
|
-
def squish(*names,
|
|
127
|
-
provider: nil, params: nil, when: nil, fallback: nil, validate: nil, &schema_block)
|
|
153
|
+
def squish(*names, purpose: nil, append_to_purpose: nil, output_schema: nil, model: nil, escalation: nil,
|
|
154
|
+
provider: nil, params: nil, harness: nil, when: nil, fallback: nil, validate: nil, squawk: nil, &schema_block)
|
|
128
155
|
schema = schema_block ? Schematist::Schema.create(&schema_block) : output_schema
|
|
129
156
|
model = ModelPath.declare(model, escalation, "#{self} squish")
|
|
157
|
+
raise ConfigurationError, "#{self} squish: provider: needs a model: or escalation:" if provider && !model
|
|
158
|
+
|
|
130
159
|
params &&= Params.normalize(params, "#{self} squish params")
|
|
131
|
-
unless
|
|
132
|
-
|
|
160
|
+
harness = Harness.normalize(harness, "#{self} squish") unless harness.nil?
|
|
161
|
+
unless append_to_purpose.nil?
|
|
162
|
+
append_to_purpose = Appendices.normalize(append_to_purpose, "#{self} squish append_to_purpose")
|
|
133
163
|
end
|
|
134
|
-
|
|
135
|
-
|
|
164
|
+
squawk = Squawk.validate(squawk, "#{self} squish")
|
|
165
|
+
options = { purpose:, append_to_purpose:, output_schema: schema, model:, provider:, params:, harness:,
|
|
166
|
+
predicate: binding.local_variable_get(:when), fallback:, validator: validate, squawk: }.compact
|
|
136
167
|
|
|
137
168
|
names.map(&:to_sym).each do |name|
|
|
138
169
|
(@squishling_methods ||= {})[name] = options
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Squishling
|
|
4
|
+
# `include Squishling` puts its methods ahead of inherited ones, so a class that inherits a method of the same
|
|
5
|
+
# name (Sinatra::Base.call, a parent's own `purpose`, ...) would be silently shadowed. Methods the class
|
|
6
|
+
# defines itself always win, so they are never reported.
|
|
7
|
+
module Collisions
|
|
8
|
+
INSTANCE_METHODS = %i[squish! squishling_result].freeze
|
|
9
|
+
|
|
10
|
+
module_function
|
|
11
|
+
|
|
12
|
+
# Raises ConfigurationError naming every inherited method Squishling would override. Call it before
|
|
13
|
+
# extending the class with ClassMethods, and after the module is in the class's ancestors.
|
|
14
|
+
def check!(base)
|
|
15
|
+
found = class_collisions(base) + instance_collisions(base)
|
|
16
|
+
return if found.empty?
|
|
17
|
+
|
|
18
|
+
raise ConfigurationError,
|
|
19
|
+
"#{base} inherits methods that Squishling would override: #{found.join(', ')}. " \
|
|
20
|
+
"Include Squishling in a plain Ruby class instead (compose the framework object rather than inheriting it)"
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def class_collisions(base)
|
|
24
|
+
ClassMethods.public_instance_methods(false).filter_map do |name|
|
|
25
|
+
next if name == :inherited || !base.respond_to?(name, true)
|
|
26
|
+
|
|
27
|
+
owner = base.method(name).owner
|
|
28
|
+
next if owner == base.singleton_class || owner == ClassMethods
|
|
29
|
+
|
|
30
|
+
"#{base}.#{name} (from #{describe(owner)})"
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def instance_collisions(base)
|
|
35
|
+
ancestors = base.ancestors
|
|
36
|
+
inherited = ancestors.drop(ancestors.index(Squishling) + 1)
|
|
37
|
+
|
|
38
|
+
INSTANCE_METHODS.filter_map do |name|
|
|
39
|
+
next if defined_in?(base, name)
|
|
40
|
+
|
|
41
|
+
owner = inherited.find { |mod| defined_in?(mod, name) }
|
|
42
|
+
"#{base}##{name} (from #{describe(owner)})" if owner
|
|
43
|
+
end
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def defined_in?(mod, name)
|
|
47
|
+
mod.method_defined?(name, false) || mod.private_method_defined?(name, false)
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def describe(owner)
|
|
51
|
+
owner.singleton_class? ? "#{owner.attached_object}'s class methods" : owner.to_s
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
end
|
|
@@ -23,16 +23,33 @@ module Squishling
|
|
|
23
23
|
# with_thinking; any other key (top_p, max_tokens, seed, ...) is passed to the provider as-is.
|
|
24
24
|
attr_reader :default_params
|
|
25
25
|
|
|
26
|
+
# How every squishling class that doesn't declare its own harness uses its escalation (see Harness):
|
|
27
|
+
# :escalation (the default), :squishsum, :judged_squishsum, a harness you registered, or a Hash with type: and
|
|
28
|
+
# its options.
|
|
29
|
+
attr_reader :default_harness
|
|
30
|
+
|
|
26
31
|
# Optional Logger for routing and escalation decisions.
|
|
27
32
|
attr_accessor :logger
|
|
28
33
|
|
|
34
|
+
# Optional callable run after every LLM attempt with the raw output (see Squawk), e.g.
|
|
35
|
+
# ->(output:, metadata:, error:) { Tracer.record(output, metadata, error) }
|
|
36
|
+
attr_reader :squawk
|
|
37
|
+
|
|
38
|
+
# Optional around-style hook wrapped around every provider request (see Observer): an object responding to
|
|
39
|
+
# around(call) { ... }, or :coolhand. nil (the default) uses the Coolhand observer when the coolhand gem is
|
|
40
|
+
# loaded and configured, and nothing otherwise; false never wraps.
|
|
41
|
+
attr_reader :observer
|
|
42
|
+
|
|
29
43
|
def initialize
|
|
44
|
+
@observer = nil
|
|
30
45
|
@default_model = nil
|
|
31
46
|
@default_escalation = nil
|
|
32
47
|
@default_model_path = nil
|
|
33
48
|
@default_provider = nil
|
|
34
49
|
@default_params = {}
|
|
50
|
+
@default_harness = nil
|
|
35
51
|
@logger = nil
|
|
52
|
+
@squawk = nil
|
|
36
53
|
end
|
|
37
54
|
|
|
38
55
|
def default_model=(model)
|
|
@@ -47,10 +64,25 @@ module Squishling
|
|
|
47
64
|
@default_escalation = escalation
|
|
48
65
|
end
|
|
49
66
|
|
|
67
|
+
def squawk=(hook)
|
|
68
|
+
@squawk = Squawk.validate(hook, "Squishling.configure")
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
def observer=(observer)
|
|
72
|
+
Observer.validate(observer, "Squishling.configure")
|
|
73
|
+
# A named observer is built now, so a missing gem fails at setup rather than on the first request.
|
|
74
|
+
Observer.resolve(observer) if observer.is_a?(Symbol)
|
|
75
|
+
@observer = observer
|
|
76
|
+
end
|
|
77
|
+
|
|
50
78
|
def default_params=(params)
|
|
51
79
|
@default_params = Params.normalize(params, "default_params")
|
|
52
80
|
end
|
|
53
81
|
|
|
82
|
+
def default_harness=(harness)
|
|
83
|
+
@default_harness = harness.nil? ? nil : Harness.normalize(harness, "default_harness")
|
|
84
|
+
end
|
|
85
|
+
|
|
54
86
|
MAX_RETRIES_REMOVED = "max_retries was removed; use default_escalation (or a class/method escalation:) " \
|
|
55
87
|
"with attempts:, e.g. [{ model: \"claude-haiku-4-5\", attempts: 2 }]"
|
|
56
88
|
|