squishling 0.2.0 → 0.4.0

data/docs/coolhand.md

@@ -0,0 +1,66 @@
+# Coolhand Labs
+
+Squishling ships an [observer](observers.md) for [Coolhand Labs](https://coolhandlabs.com/). With it, every LLM call a
+squished method makes is filed under a named **workload** and **template** in Coolhand. Squishling knows exactly
+where each call sits (which method, which escalation level, which sample, which judge), so it declares that for you.
+
+## Setup
+
+Add and configure the [`coolhand`](https://github.com/Coolhand-Labs/coolhand-ruby) gem as described in
+[Measuring token spend](measuring-tokens.md#coolhand-labs). **That's all.** Squishling defaults to the Coolhand
+observer whenever Coolhand is already available in your codebase: loaded, with an `api_key`, and not disabled
+(`config.enabled = false`). Nothing is added to Squishling's dependencies, and nothing happens when the gem isn't
+there. It needs a version of the `coolhand` gem that has `Coolhand.with_context` (declared workloads); with an older
+one, the observer stays off.
+
+To turn it off, set `observer` to `false`:
+
+```ruby
+Squishling.configure do |config|
+  config.observer = false # never declare, even though Coolhand is configured
+end
+```
+
+The other settings:
+
+```ruby
+Squishling.configure do |config|
+  config.observer = :coolhand # always declare; raises ConfigurationError now if the gem is missing or too old
+  config.observer = nil       # the default: declare when Coolhand is loaded and configured
+end
+```
+
+To report somewhere else, or as well, use any [other observer](observers.md). `config.observer` holds one.
+
+## What is declared
+
+Each request runs inside `Coolhand.with_context(reporter: "squishling", workload: ..., template: ...)`, so Coolhand's
+Net::HTTP interception tags it with the declaration.
+
+| | Key | Name |
+|---|---|---|
+| Workload | `Class#method` | `Class#method` (its `description` is `Class#method`, or `Class#method (rescue)` when a [`squish!`](routing.md#handing-off-to-the-llm-with-squish) call handed it to the LLM) |
+| Template | `Class#method/<role>` | `Class#method · <role>` |
+
+`<role>` is where the request sits:
+
+| Role | Request |
+|---|---|
+| `squish:L<n>` | Attempt *n* (1-based) of the [escalation](configuration.md#models-and-escalation), counting each attempt of each step (a [custom harness](harnesses.md#custom-harnesses) that escalates stage by stage restarts at L1 for each) |
+| `sample:a`, `sample:b`, ... | A [squishsum](harnesses.md) sample (which runs on a worker thread; the declaration is made on that thread) |
+| `judge` | A harness's judge, chat or System One (all of a judge's attempts share this one template) |
+
+So `Parser#call/squish:L2` is the second attempt of `Parser#call`, e.g. `Parser#call · squish L2` in Coolhand.
+
+- The model is deliberately not in the key. It is on the log, and the position is the stable identity.
+- Repeated attempts on the same step count as separate levels, because the level is the attempt's position in the
+  expanded escalation.
+- A class name long enough to push a key past Coolhand's 255-character limit is left undeclared rather than
+  breaking the call.
+- Squishling's declaration is made inside any `Coolhand.with_context` you wrapped around the call, so for the fields it
+  sets, Squishling's wins.
+- The description never contains the purpose text, which can be a proc that differs per call. The workload's key
+  and name stay `Class#method` either way, so calls that went through `squish!` and calls that didn't share one
+  workload.
+- `observer = :coolhand` loads the gem when it is assigned, so assign it after `coolhand` is available (the default
+  needs no ordering).

data/docs/failures.md

@@ -10,19 +10,82 @@ A failed attempt moves on to the next one; once the last fails, the error is rai
 |---|---|
 | Rate limit, 5xx, overload, timeout, connection error | RubyLLM retries at the HTTP level first (`RubyLLM.config.max_retries`, default 3). If it still fails, moves to the next attempt in a fresh chat; on the last attempt, raises `Squishling::LLMError` with the original exception as its `cause`. |
 | Bad API key, unknown model, missing provider config | Raises `Squishling::ConfigurationError`. Never escalated, never sent to the fallback. |
-| Provider rejects the request (400 Bad Request), e.g. an unsupported `temperature` or a schema it won't accept | Raises `ConfigurationError` with the provider's message and the params in use. Never escalated, never sent to the fallback, so a setup mistake can't be silently covered up on every call. |
+| Provider rejects the request (400 Bad Request), e.g. an unsupported `temperature` or a schema it won't accept | Raises `ConfigurationError` naming the error class and HTTP status, and the params in use; the provider's own message is on `cause`. Never escalated, never sent to the fallback, so a setup mistake can't be silently covered up on every call. |
 | Empty or `nil` response (e.g. a refusal or a max-tokens cutoff) | Moves to the next attempt, then raises `Squishling::InvalidOutputError` |
 | Malformed or truncated JSON | Moves to the next attempt, then raises `InvalidOutputError`. JSON wrapped in a markdown code fence is accepted. |
 | JSON that doesn't match the schema (wrong types, missing or extra keys, `null` in a non-`optional` field, a broken [conditional rule](schemas.md#contracts-beyond-the-schema), root not an object) | Moves to the next attempt with the validation errors, then raises `InvalidOutputError` |
 | Schema-valid output that [`squish_validate`](#output-checks-squish_validate) rejects | Moves to the next attempt with your messages, then raises `InvalidOutputError` |
+| With a [squishsum harness](harnesses.md): two valid samples that differ, with no judge or a judge that rejects both | Raises `Squishling::DisagreementError` (an `InvalidOutputError`), carrying both typed `candidates` |
 
 Squishling validates output itself with [json_schemer](https://github.com/davishmcclurg/json_schemer), because
 RubyLLM doesn't, and some providers don't enforce strict mode. When the next attempt is on the same step (same
-model, provider, and params), the re-ask happens in the same conversation, so the model sees what it got wrong. A
-different step gets a fresh chat with the original input plus the rejected output and its errors, so that output is
-sent to the next step's provider, which may not be the one that produced it. `InvalidOutputError` exposes `errors`,
-`raw` (the last response), `attempts`, and `models` (the model tried on each attempt). With a `logger` configured,
-every escalation is logged as a warning.
+model, provider, params, and `forward_rejected:`), the re-ask happens in the same conversation, so the model sees
+what it got wrong. A different step gets a fresh chat with the original input plus the rejected output (truncated to
+4,000 characters) and its errors, so that output is sent to the next step's provider, which may not be the one that
+produced it. Set `forward_rejected: false` on a step to leave both out. `InvalidOutputError` exposes `errors`, `raw`
+(the last response), `attempts`, and `models` (the model tried on each attempt); for a
+[`DisagreementError`](harnesses.md#when-the-harness-fails), `raw` holds both candidates, `attempts` is `nil`, and
+`models` has one entry per role (sample a, sample b, then the judge). With a `logger` configured, every
+escalation is logged as a warning.
+
+### What ends up in errors and logs
+
+Model output can echo sensitive input, so the raw response is kept in one place: `InvalidOutputError#raw` (plus the opt-in `squawk` hook below).
+`InvalidOutputError#message`, `#errors`, and `config.logger` warnings are built from these:
+
+- For unparseable JSON, only the position (`response was not valid JSON (at line 1 column 7)`), never the parser's
+  snippet of the response.
+- JSON Schema errors, which name the schema path and the rule that failed. They never contain the rejected value, but
+  they do name an extra key the model added (`object property at /foo is a disallowed additional property`), and
+  that key name is chosen by the model.
+- Your own `squish_validate` messages, verbatim. If you interpolate result values into them, those values reach
+  the message, the logger, and the next attempt's prompt.
+- A fixed message when a response contains a value JSON can't represent (`1e400` parses to `Infinity`, or a string
+  with invalid UTF-8).
+- For a failed provider request (`LLMError`, and the `ConfigurationError` for a 400, bad credentials, or a forbidden
+  key), only the error class and HTTP status. The provider's own message can echo request content or part of an API
+  key, so it stays on `cause` (`error.cause.message`) for you to read or log deliberately.
+- At most the first 20 of these errors, plus a count of the rest, so a very large malformed response can't flood
+  the retry message or the log.
+- For a [`DisagreementError`](harnesses.md#when-the-harness-fails), only what Squishling wrote: a `:judgment` judge's
+  choice and probability. A chat judge's `reason` is model-written, so it is available as `#reason` but never in the
+  message or logs.
+
+The same applies to anything you log yourself, such as `error.message` in a [fallback](#fallbacks). Use `error.raw`
+only where you are willing to store model output. To send the output of every attempt somewhere on purpose, use
+[`squawk`](#observing-every-attempt-squawk).
+
+## Observing every attempt (`squawk`)
+
+`squawk` is an opt-in hook for sending what the model returned to an error tracker or tracing tool. It runs after
+every LLM attempt, accepted or rejected, and does nothing unless you set it:
+
+```ruby
+Squishling.configure do |config|
+  config.squawk = lambda do |output:, metadata:, error:|
+    ObservabilitySolution.record(output, metadata, error) if error
+  end
+end
+```
+
+| Keyword | Value |
+|---|---|
+| `output` | The raw response: a Hash or String, or `nil` when the provider call itself failed |
+| `error` | `nil` for an accepted attempt. Otherwise the `InvalidOutputError` (with `errors` and `raw`) or `LLMError` that ended the attempt |
+| `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length; under a harness, the sample's or judge's own attempts), `final` (the last attempt), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when RubyLLM reports them), `role` (`nil` for an escalation attempt, `:sample`, or `:judge`), `sample` (`"a"`, `"b"`, ... for a sample's attempt, else `nil`) |
+
+- Set it globally with `config.squawk`, per class with `squishling squawk: ...`, or per method with
+  `squish :triage, squawk: ...`. The method's hook wins over the class's, which wins over the configured one;
+  `squawk: false` silences an inherited hook. Subclasses and `squish!` calls inherit it.
+- Any object that responds to `call` works. The hook receives only the keywords it declares, unless it takes `**`,
+  so a lambda that wants just `error:` is fine and new metadata fields won't break it.
+- It runs inline, so keep it quick. Exceptions it raises propagate unchanged and fail the call, like any of your own
+  code, so rescue inside the hook if an outage in your tracing tool shouldn't.
+- `output` is the same object the result is built from, so treat it as read-only.
+- Under a [harness](harnesses.md), it runs for every sample and judge attempt too, each with its own `input`, and
+  `role`/`sample` say which one it was. A [custom harness](harnesses.md#reporting-to-squawk) reports the same way.
+- It isn't called for a `ConfigurationError`, for deterministic and fallback returns, or for an attempt where your own
+  `squish_validate` raises (that exception propagates first).
 
 ## Output checks (`squish_validate`)
 
@@ -57,16 +120,17 @@ end
 | Error | Raised when |
 |---|---|
 | `Squishling::Error` | Base class for everything below. Raised directly for misuse at call time, e.g. `result` or `squish!` called outside a squished method |
-| `Squishling::ConfigurationError` | Missing instructions or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_instructions` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (400) |
+| `Squishling::ConfigurationError` | Missing purpose or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_to_purpose` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (400) |
 | `Squishling::InvalidOutputError` | LLM output still invalid (schema or `squish_validate`) after every attempt in the escalation, or a deterministic/fallback return that doesn't match the schema |
+| `Squishling::DisagreementError` | A subclass of `InvalidOutputError`: a [squishsum harness](harnesses.md#when-the-harness-fails)'s samples disagreed and no judge accepted either. `candidates`, `verdict`, and `reason` describe what happened |
 | `Squishling::LLMError` | The provider call failed on the last attempt in the escalation, after RubyLLM's own retries, including context-length errors (`cause` holds the original) |
 
 Errors raised by your own Ruby code are not wrapped.
 
 ## Fallbacks
 
-Use `squish_fallback` to decide what happens when every attempt fails, with `InvalidOutputError` or
-`LLMError`.
+Use `squish_fallback` to decide what happens when the LLM path fails with `InvalidOutputError` (including a
+`DisagreementError`) or `LLMError`. It's the one fallback for every [harness](harnesses.md).
 It receives the error plus the method's inputs as keywords, and runs against the instance:
 
 ```ruby

data/docs/harnesses.md

@@ -0,0 +1,326 @@
+# Harnesses: escalation, squishsum, and your own
+
+A squished call's **harness** decides how it uses its [escalation](configuration.md#models-and-escalation):
+
+| Harness | What it does | Requests per call |
+|---|---|---|
+| `:escalation` (default) | Tries each attempt in order until an output passes the schema and `squish_validate`. | 1 or more |
+| `:squishsum` | Asks the escalation's **first step** twice, concurrently. Identical outputs are accepted; different ones raise `Squishling::DisagreementError`. | 2 or more |
+| `:judged_squishsum` | Like `:squishsum`, but when the two outputs differ, a **judge** picks one or rejects both. | 2 or more, plus 1 or more judge requests when they differ |
+
+Need a different trade-off, such as two models plus a tiebreaker or a majority vote? Register your own with
+[`Squishling.register_harness`](#custom-harnesses).
+
+Use a squishsum harness when a confidently wrong answer costs more than a second request: one sample can make a
+mistake, but two independent samples rarely make the same one.
+
+```ruby
+class TicketTriager
+  include Squishling
+
+  squishling escalation: [{ model: "claude-haiku-4-5", attempts: 2 }, "claude-sonnet-5-5"],
+             harness: :judged_squishsum   # two Haiku samples; Sonnet judges when they differ
+  purpose "Assign a priority and a team."
+  output_schema do
+    string :priority, enum: %w[low medium high]
+    string :team
+  end
+end
+```
+
+## Declaring a harness
+
+`harness:` takes a type, or a Hash with `type:` and its options:
+
+```ruby
+Squishling.configure { |c| c.default_harness = :squishsum }      # every class without its own
+
+squishling harness: :judged_squishsum                            # this class (and subclasses)
+squish :triage, harness: { type: :squishsum,                     # one method
+                           compare: ->(a, b, **) { a.priority == b.priority } }
+squish!(harness: :escalation)                                    # one call (see routing.md)
+```
+
+The first level that declares one wins: `squish!`, then the method, then the class, then
+`Squishling.config.default_harness`, then `:escalation`. Options aren't merged across levels.
+
+| Option | Harnesses | Description |
+|---|---|---|
+| `type:` | all | `:escalation`, `:squishsum`, `:judged_squishsum`, or a [registered](#custom-harnesses) name (required in the Hash form). |
+| `compare:` | squishsum ones | `->(a, b, **inputs) { ... }`, evaluated against the instance with the two typed results and the method's inputs. Truthy means the samples agree. Default: the two outputs are exactly equal. |
+| `judge:` | judged ones | The judge step (see [The judge](#the-judge)). Default: the escalation's next step after the sampled one. |
+| `judge_instructions:` | judged ones | A String, or a Proc evaluated against the instance, that replaces the [default judge prompt](#the-default-judge-prompt). |
+
+## Samples
+
+Both samples run on the escalation's first step, with its model, provider, and params:
+
+- **They're independent.** Each sample has its own chat, and the two requests are sent concurrently on separate
+  threads. Only the provider requests run on those threads. Parsing, schema validation, `squish_validate`,
+  `compare:`, and the judge all run on the calling thread, so your Squishling callbacks never run concurrently.
+  RubyLLM instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for `chat.ruby_llm`) do see
+  each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
+- **Each one is retried like an escalation step.** A sample with invalid output is re-asked in its own chat, up to
+  the first step's `attempts:`, while a sample that already passed keeps its result. A failed request gets a fresh
+  chat. A sample still failing after its attempts fails the call with `InvalidOutputError` or `LLMError`, as
+  usual. Later escalation steps are never used for samples (a judge may use the next one, see
+  [The judge](#the-judge)).
+- **Agreement is exact by default.** Free-text fields seldom match word for word, so pass `compare:` to decide
+  which fields must match:
+
+  ```ruby
+  squishling harness: { type: :squishsum, compare: ->(a, b, **) { a.priority == b.priority && a.team == b.team } }
+  ```
+
+  When the samples agree, the first one is returned.
+
+## The judge
+
+With a judged harness, two valid samples that disagree go to a judge. The judge picks candidate `a` or `b`,
+whose typed result is returned (`squished?` is `true`), or it picks `neither`, which raises
+`DisagreementError`.
+
+The judge is:
+
+1. the `judge:` step, if declared: a model name, or a Hash with `model:`, `provider:`, `params:`, and `attempts:`
+   (as in an escalation step, without `priority:` or `rotate:`), plus `type:`; or
+2. the escalation's next step after the sampled one, with its attempts. Steps tied on `priority:` rotate between calls
+   (see [Rotating models](configuration.md#rotating-models-within-a-priority)), so which model that is can vary.
+
+A judged harness with neither raises `ConfigurationError` before any sample is requested. A `judge:`
+step uses only its own `provider:`; it doesn't inherit the class or method `provider:`, which belongs to their models
+(a chat judge's params, though, do merge over the method's).
+
+### Chat judges
+
+By default the judge is a chat model, asked for a strict `{ "verdict": "a" | "b" | "neither", "reason": "..." }`.
+It gets the same context the samples had, plus the samples themselves:
+
+- **System prompt:** the judge prompt, then the operation's own purpose (including any `append_to_purpose`
+  sections).
+- **User message:** JSON with the samples' `input` (`arguments` and `context`), the `output_schema`, and the
+  `candidates` `a` and `b`.
+
+A verdict that doesn't match its schema is retried per the judge step's `attempts:`; a judge that never returns a
+valid verdict raises `InvalidOutputError`. A chat judge's params are the method's
+[generation params](configuration.md#generation-params) with its own `params:` merged over them.
+
+### System One judgment models (Jev)
+
+A judge can also be a System One decision model, such as TypeSafe's Jev, via
+[RubyLLM judgments](https://rubyllm.com/judgments/) (RubyLLM 2.1+). Decision models return calibrated
+probabilities rather than text, so they are much faster and cheaper than a chat judge.
+
+```ruby
+squishling harness: {
+  type: :judged_squishsum,
+  judge: { model: "jev-latest", type: :judgment, min_confidence: 0.8 }
+}
+```
+
+Squishling asks one `choice` question (`a`, `b`, or `neither`), with the judge prompt as its instructions and the
+operation's purpose, input, output schema, and candidates as the judgment input. A pick of `a` or `b` is
+accepted only when its probability is at least `min_confidence:` (default `0.8`). Otherwise, as with `neither`,
+`DisagreementError` is raised with the choice and its probability in its `reason`. A judgment judge sends only its own
+`params:` (as RubyLLM `provider_options:`). `attempts:` retries a failed request. Configure the provider's
+credentials in RubyLLM as usual.
+
+### The default judge prompt
+
+```text
+You are an impartial judge. Two independent attempts at the same operation returned different outputs,
+candidate "a" and candidate "b". Both already match the required output format, so judge only their
+content. You are given the operation's purpose and its input. Decide which candidate correctly and
+faithfully carries out the purpose for this input. If both do (for example, they differ only in
+wording), choose either one, preferring "a". Choose "neither" only when both are wrong or you can't tell
+whether either is correct. Never combine them or invent a third answer.
+```
+
+It is available as `Squishling::Harness::DEFAULT_JUDGE_INSTRUCTIONS`. Replace it with `judge_instructions:`:
+
+```ruby
+squishling harness: { type: :judged_squishsum,
+                      judge_instructions: "Pick the candidate whose priority follows our SLA rules: ..." }
+```
+
+## When the harness fails
+
+A disagreement raises `Squishling::DisagreementError`, a subclass of `InvalidOutputError`. It's raised when the
+samples differ with no judge, or when the judge rejects both. Like any failure on the LLM path, it goes to
+[`squish_fallback`](failures.md#fallbacks), which can still return one of the candidates:
+
+```ruby
+squish_fallback do |error, **|
+  raise error unless error.is_a?(Squishling::DisagreementError)
+
+  error.candidates.first   # already typed and validated; squished? is true
+end
+```
+
+| Attribute | Value |
+|---|---|
+| `candidates` | The two typed results, `[a, b]` |
+| `verdict` | `nil` when there was no judge; `:neither` when the judge rejected both |
+| `reason` | The judge's reason (or, for a judgment model, the choice and its probability). A chat judge's reason is model-written, so it is not part of `message` or the logs; a judgment model's choice and probability are |
+| `raw` | Both candidates as Hashes, `[a.to_h, b.to_h]` |
+| `attempts` | `nil` |
+| `models` | The model behind each role: sample `a`, sample `b`, then the judge if one ran (one entry each, not one per attempt) |
+
+Other failures are unchanged. A sample or judge that never produces valid output raises `InvalidOutputError`, a
+failed request on the last attempt raises `LLMError`, and setup mistakes (including a provider 400 on either
+sample) raise `ConfigurationError`, which is never sent to the fallback.
+
+## Custom harnesses
+
+A harness is a class that decides how to use the escalation: which samples to ask for, how to compare them, who
+breaks a tie. Register one under a name and it works anywhere a built-in type does: `squishling harness: :name`,
+`squish :method, harness: { type: :name, ... }`, `squish!(harness: ...)`, and `config.default_harness`. It gets
+everything the built-ins get: model, provider, and params resolution, schema validation and
+[`squish_validate`](failures.md#output-checks-squish_validate), retries and escalation,
+[`squawk`](failures.md#observing-every-attempt-squawk) reports with token usage, concurrent samples, and the
+[failure behavior](#when-a-custom-harness-fails) (`squish_fallback`, `DisagreementError`).
+
+### Worked example: two models and a tiebreaker
+
+Ask two cheap models. If they agree, done. If they don't, ask a third:
+
+```ruby
+class Tiebreaker < Squishling::Harness::Base
+  option :tiebreaker, :step, required: true   # a model name, or a step Hash
+  option :compare, :proc                      # optional, like squishsum's compare:
+
+  def call(run, tiebreaker:, compare: nil)
+    raise Squishling::ConfigurationError, "#{run.label}: needs two models" if run.stages.size < 2
+
+    first, second = run.samples(run.stages.first(2))   # the escalation's first two steps, asked concurrently
+    return first if run.agree?(first, second, compare)
+
+    run.escalate(tiebreaker)                           # they differ: a third opinion
+  end
+end
+
+Squishling.register_harness(:tiebreaker, Tiebreaker)   # e.g. in an initializer
+
+class TicketTriager
+  include Squishling
+
+  squishling escalation: ["claude-haiku-4-5", "claude-sonnet-5-5"],
+             harness: { type: :tiebreaker, tiebreaker: "claude-opus-5-5",
+                        compare: ->(a, b, **) { a.priority == b.priority } }
+  purpose "Assign a priority and a team."
+  output_schema do
+    string :priority, enum: %w[low medium high]
+    string :team
+  end
+end
+```
+
+`TicketTriager.call(...)` sends the ticket to Haiku and Sonnet at the same time. A pair that agrees on `priority` returns
+Haiku's result; otherwise Opus decides. Either way you get the same typed, validated result as every other path.
+`run.stages` groups identical neighbouring steps into one stage (they are one model's attempts), so this harness needs
+two different models in the escalation.
+
+For small harnesses, pass a block instead of a class. It becomes the body of a `Squishling::Harness::Base` subclass:
+
+```ruby
+Squishling.register_harness(:best_of_three) do
+  def call(run)
+    results = run.samples([run.stages.first] * 3)
+    results.max_by { |result| results.count { |other| run.agree?(result, other) } }   # the most-agreed-with sample
+  end
+end
+```
+
+### Registering
+
+`Squishling.register_harness(name, klass = nil, &block)` takes a `Squishling::Harness::Base` subclass or a block, and
+returns the class. `Squishling.harnesses` lists every name, built-ins first.
+
+- The registry is global, and a harness is looked up when it is **declared**, so an unknown name fails up front.
+  Register in an initializer, before the classes that use the name load.
+- A name is lowercase letters, digits, and underscores. Registering a name twice, or one of `:escalation`,
+  `:squishsum`, `:judged_squishsum`, raises `ConfigurationError`.
+- A class must define `call` taking the run as its first argument and each declared option as a keyword (a keyword
+  for an option that may be left out needs a default), and be creatable without arguments. This is checked at
+  registration.
+- Subclass `Squishling::Harness::Base` directly. The built-in harness classes are implementation, not an extension
+  point; write against `Run` as they do.
+- `Squishling.reset_harnesses!` forgets the registered harnesses (the built-ins stay). It is meant for specs.
+  `Squishling.reset_config!` leaves them, since they're registered once at boot.
+
+### Options
+
+Declare each option with `option name, type, required: false`, before registering the class. They are checked when
+the harness is declared, before any request is made, and an unknown, missing, or mistyped option raises
+`ConfigurationError`. A `nil` counts as not given, and `call` receives only the options that were given.
+
+| Type | Accepts | `call` receives |
+|---|---|---|
+| `:step` | A model name, or a Hash with `model:`, `provider:`, `params:`, `attempts:`, `forward_rejected:` | A normalized, frozen step; pass it to `run.escalate`, or in the Array given to `run.samples` |
+| `:proc` | A Proc | The Proc |
+| `:string`, `:symbol`, `:integer`, `:boolean` | A String, a Symbol (or String, converted), an Integer, `true` or `false` | The value |
+
+An option with a shape of its own takes a callable as its type instead. It gets the value, the declaration site (a
+label for error messages), and the option's name, and returns the normalized value, or raises `ConfigurationError`. The
+Hashes, Arrays, and Strings in the result are copied and frozen:
+
+```ruby
+option :word, ->(value, label, name) do
+  raise Squishling::ConfigurationError, "#{label}: #{name}: must be a String" unless value.is_a?(String)
+
+  value.upcase
+end
+```
+
+To check options against each other, define `def self.validate_options!(options, label)` and raise
+`Squishling::ConfigurationError` (include the `label`) to reject.
+
+### The run
+
+`call(run, **options)` gets a `Squishling::Run`: one squished call, with these primitives. Each returns typed
+results, the same type the deterministic path returns.
+
+| Method | What it does |
+|---|---|
+| `run.receiver`, `run.inputs`, `run.label` | The instance, the method's inputs, and `"Class#method"` |
+| `run.steps` | The escalation, one step per attempt |
+| `run.stages` | The escalation grouped by model: `[[haiku, haiku], [sonnet]]` for `[{ model: haiku, attempts: 2 }, sonnet]`. A stage is the attempts of one model |
+| `run.stage(declaration)` | A stage from a model name or a step Hash (a single step is a stage of one attempt). It keeps its own provider, and its `params:` merge over the method's, as a [judge](#the-judge) step does |
+| `run.escalate(*stages)` | Makes each attempt in order until an output passes the schema and `squish_validate`, in the same chat after invalid output and in a fresh one (told about the rejected output) on a new step. With no argument, it runs the escalation: exactly what `:escalation` does |
+| `run.samples(stages)` | Takes an Array of stages (so `run.samples([stage, stage])`, not `run.samples(stage)`; a bare step counts as a stage of one attempt): one sample per stage, each in its own chat and retried like an escalation step, with the requests of every round sent concurrently. Returns the outputs in order. Each stage must be one model's attempts: mixed steps raise `ConfigurationError`. Samples are named `"a"`, `"b"`, ... (up to 26) |
+| `run.judge(judge: nil, instructions: nil)` | Resolves a [judge](#the-judge) now (so a missing one raises `ConfigurationError` before any request) and returns it. `judge:` is a model name, or a [judge Hash](#the-judge) (`model:`, `provider:`, `params:`, `attempts:`, `type:`, `min_confidence:`; no `priority:` or `rotate:`), and `instructions:` a non-blank String or a Proc. `judge.verdict(first, second)` returns a `Squishling::Judge::Verdict` with `choice` (`:a`, `:b`, or `:neither`), `reason`, and `detail`, and `judge.step` is its first attempt. A chat judge's `reason` is model-written, so use `detail` (Squishling's own text, `nil` for a chat judge), not `reason`, in messages and logs. Without `judge:`, the judge is the escalation's second stage |
+| `run.agree?(first, second, compare = nil)` | `compare` (evaluated against the instance with both results and the method's inputs) or, without one, exact equality |
+| `run.log(message)` | A warning to `config.logger`, prefixed with `"[Squishling] Class#method"` |
+
+`call` returns the accepted result: one a primitive of this run returned. Returning anything else (a Hash, `nil`, a
+result you built or copied with `with`) raises `ConfigurationError`, so a harness can't skip validation.
+
+### When a custom harness fails
+
+A harness fails the way the built-ins do, so `squish_fallback` and the error classes behave predictably:
+
+- **Attempts run out.** `run.escalate` and `run.samples` raise `InvalidOutputError` (invalid output after the last
+  attempt) or `LLMError` (a failed request on the last attempt) themselves, with the same attributes as always.
+- **Your rule rejects.** Raise one of those errors yourself. `Squishling::DisagreementError.new(candidates:,
+  models:)` is the one for samples that don't agree, and `Squishling::InvalidOutputError.new(errors: [...], source: "Harness")` for
+  anything else. Both go to [`squish_fallback`](failures.md#fallbacks).
+- **Setup mistakes** raise `ConfigurationError`, which is never retried and never sent to the fallback.
+- **Your own exceptions** (from `call`, `compare:`, or `squish_validate`) propagate unchanged. Squishling doesn't wrap or
+  swallow them.
+
+Keep model output out of the messages you raise and the lines you `run.log`: it can quote the input. The output is
+available to your code as the results and in `InvalidOutputError#raw`, and to `squawk` if you opt in.
+
+### Reporting to `squawk`
+
+Every attempt a primitive makes is reported to the [`squawk`](failures.md#observing-every-attempt-squawk) hook, with
+usage, so [token measuring](measuring-tokens.md) works unchanged. `metadata[:role]` says what the attempt was for:
+`nil` for an escalation attempt, `:sample` (with `metadata[:sample]` set to `"a"`, `"b"`, ...), or `:judge`. `attempt`
+and `attempts` count within that sample, judge, or escalation.
+
+### Threads
+
+Only the provider requests inside `run.samples` run on worker threads. Your `call`, parsing, validation,
+`squish_validate`, and `compare:` all run on the calling thread, and the routing state stays fiber-local. Call a run's
+primitives from the thread `call` runs on: from another one they raise `ConfigurationError`. Don't keep unsynchronized
+state at class level in a harness, since one instance is created per call but the class is shared.

data/docs/measuring-tokens.md

@@ -0,0 +1,140 @@
+# Measuring token spend
+
+A squished path costs tokens on every call. A Ruby path costs engineer time once, plus upkeep. Measuring the
+first tells you when the second is worth paying.
+
+## What Squishling gives you
+
+- `result.squished?` says which path served a call, so you can count LLM calls per class.
+- The [`squawk` hook](failures.md#observing-every-attempt-squawk) runs after every LLM attempt and receives the
+  method (`label`, e.g. `"ShipmentUpdate#call"`), the model and provider, which attempt it was, and the token usage
+  RubyLLM reported. That is enough to attribute spend to a squishling method without any other tooling.
+- Cost in dollars comes from RubyLLM's model pricing. Multiply the reported tokens by your model's price, or read it
+  from RubyLLM directly (see [OpenTelemetry](#opentelemetry), or RubyLLM's `config.instrumenter`).
+
+## Deciding when to write the code
+
+For each squished path, over a month:
+
+```
+token cost  =  calls × (input tokens × input price + output tokens × output price)
+code cost   =  time to write it  +  upkeep  (new cases, format changes, on-call)
+```
+
+Write the Ruby when the token cost keeps exceeding the code cost, and narrow `squish_when` so only the inputs
+Ruby can't handle still go to the LLM. See [Hardening a path](routing.md#hardening-a-path). If the token cost
+stays small, the debt you haven't taken on is cheaper than the code.
+
+## Tools
+
+| Option | Good for | Setup |
+|---|---|---|
+| [Coolhand Labs](#coolhand-labs) | Picking up your squishling calls automatically and measuring their cost and accuracy | The `coolhand` gem and an initializer |
+| [Custom: the `squawk` hook](#custom-the-squawk-hook) | Logging or metrics in your own stack, with no new dependency | A lambda and one config line |
+| [Your own observer](#your-own-observer) | Wrapping each request in your tracing or metrics tool, knowing which method, level, and sample it was | An object with `around(call)` |
+| [OpenTelemetry](#opentelemetry) | Sending traces to Datadog, Langfuse, Arize, Braintrust, LangSmith, or any OTel backend | RubyLLM's tracer and an exporter |
+
+### Coolhand Labs
+
+Add and initialize the [`coolhand`](https://github.com/Coolhand-Labs/coolhand-ruby) gem. It intercepts the LLM
+requests your app makes, so it picks up your squishling calls automatically and measures their cost, and their
+accuracy through Coolhand's feedback tools. No changes to your squishling classes.
+
+```ruby
+# Gemfile
+gem "coolhand"
+
+# config/initializers/coolhand.rb
+Coolhand.configure do |config|
+  config.api_key = ENV.fetch("COOLHAND_API_KEY")
+end
+```
+
+Get an API key at [coolhandlabs.com](https://coolhandlabs.com/). To keep the data in your own infrastructure, point
+`config.base_url` at a Coolhand-compatible endpoint of your own. Squishling also tells Coolhand which workload and
+template each call belongs to; see [Coolhand Labs](coolhand.md).
+
+### Custom: the `squawk` hook
+
+`config.squawk` runs after every LLM attempt, accepted or rejected, and is handed the method's label, the model,
+and the token usage RubyLLM reported. A lambda can take just the keywords it needs:
+
+```ruby
+Squishling.configure do |config|
+  config.squawk = lambda do |metadata:, **|
+    usage = metadata[:usage] || {}   # nil when the provider call failed; counts it didn't report are absent
+    Rails.logger.info("squishling #{metadata[:label]} model=#{metadata[:model]} " \
+                      "attempt=#{metadata[:attempt]}/#{metadata[:attempts]} " \
+                      "in=#{usage[:input_tokens]} out=#{usage[:output_tokens]}")
+  end
+end
+```
+
+Each line carries the `"Class#method"` that made the call, so grouping by label gives you tokens per squishling. A call
+that stays in Ruby makes no LLM request and never reaches the hook, so a label that stops appearing is a path that has
+been hardened. You can also set the hook per class (`squishling squawk: ...`) or per method. See
+[Observing every attempt](failures.md#observing-every-attempt-squawk) for every keyword and the rules for exceptions.
+Under a [harness](harnesses.md), including a [custom one](harnesses.md#custom-harnesses), every sample and judge
+request reaches the hook too. `metadata[:role]` (`nil`, `:sample`, or `:judge`) and `metadata[:sample]` (`"a"`, `"b"`, ...)
+split the spend by what each request was for.
+
+### Your own observer
+
+An [observer](observers.md) wraps each provider request and gets the request's token usage back from `yield`, so it
+can meter spend knowing which method, escalation level, sample, or judge made the request. Any object that responds to
+`around(call)` is one:
+
+```ruby
+class SpendObserver
+  def around(call)
+    usage = yield # input_tokens, output_tokens (nil if unreported), and model; never the content
+    tags = ["label:#{call.label}", "role:#{call.role || 'squish'}", "model:#{usage.model}"]
+    Metrics.increment("squishling.tokens.in", usage.input_tokens.to_i, tags:)
+    Metrics.increment("squishling.tokens.out", usage.output_tokens.to_i, tags:)
+  end
+end
+
+Squishling.configure { |config| config.observer = SpendObserver.new }
+```
+
+An observer sees usage, never the model's output or your inputs. See [Observers](observers.md) for the `call` fields
+and the rules.
+
+### OpenTelemetry
+
+RubyLLM 2.1+ ships its own OpenTelemetry tracer. Each request becomes a span with the model, request settings and
+token usage (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`), following the OpenTelemetry GenAI
+conventions. It does not record message text. Your app owns the SDK and exporters, and the spans work with any OTLP
+backend, [LangSmith](https://docs.smith.langchain.com/observability/how_to_guides/tracing/trace_with_opentelemetry)
+included.
+
+```ruby
+# Gemfile
+gem "opentelemetry-sdk"
+gem "opentelemetry-exporter-otlp"   # or your backend's exporter
+
+# config/initializers/opentelemetry.rb
+OpenTelemetry::SDK.configure        # reads the standard OTEL_* environment variables
+RubyLLM::OpenTelemetry.enable       # starts tracing RubyLLM requests
+```
+
+Spans don't say which squishling made the request. To attribute them, wrap the call in `RubyLLM.workflow("Name")`, or
+use an [observer](observers.md) that adds the label to the current span. RubyLLM's `config.instrumenter` also reports
+a `usage.ruby_llm` event per request, with tokens and cost, if you would rather meter than trace.
+
+## Privacy
+
+Squishling only sends the model what you name: the method arguments, any `squish_context` values, and any source you
+append. Tools that sit on the request path are a separate route out of your process, and they see both sides:
+
+- **Request interceptors, such as `coolhand`,** record the prompts (your purpose text and those inputs) and the
+  model's responses. Coolhand sends them to coolhandlabs.com unless you point `config.base_url` at an endpoint of your
+  own.
+- **A `squawk` hook that forwards `output:` or `metadata[:input]`** sends model output and your inputs wherever you
+  send them. That is the sanctioned route for model output; see
+  [Observing every attempt](failures.md#observing-every-attempt-squawk). The examples above log only metadata.
+- **OpenTelemetry** with RubyLLM's tracer records usage and request settings, not message text.
+- **An observer** sees timing, token usage, and the call's metadata (method, model, level), never the model's output or
+  your inputs.
+
+Check what your tool stores before pointing it at inputs that contain personal data.