squishling 0.3.0 → 0.4.0

data/docs/harnesses.md

@@ -1,4 +1,4 @@
-# Harnesses: escalation, squishsum, and ensemble
+# Harnesses: escalation, squishsum, and your own
 
 A squished call's **harness** decides how it uses its [escalation](configuration.md#models-and-escalation):
 
@@ -7,13 +7,12 @@ A squished call's **harness** decides how it uses its [escalation](configuration
 | `:escalation` (default) | Tries each attempt in order until an output passes the schema and `squish_validate`. | 1 or more |
 | `:squishsum` | Asks the escalation's **first step** twice, concurrently. Identical outputs are accepted; different ones raise `Squishling::DisagreementError`. | 2 or more |
 | `:judged_squishsum` | Like `:squishsum`, but when the two outputs differ, a **judge** picks one or rejects both. | 2 or more, plus 1 or more judge requests when they differ |
-| `:ensemble` | Asks the escalation's **first and second steps** once each, concurrently, so two different models are compared. Identical outputs are accepted; different ones raise `DisagreementError`. | 2 or more |
-| `:judged_ensemble` | Like `:ensemble`, but when the two outputs differ, a **judge** picks one or rejects both. | 2 or more, plus 1 or more judge requests when they differ |
+
+Need a different trade-off, such as two models plus a tiebreaker or a majority vote? Register your own with
+[`Squishling.register_harness`](#custom-harnesses).
 
 Use a squishsum harness when a confidently wrong answer costs more than a second request: one sample can make a
-mistake, but two independent samples rarely make the same one. Use an [ensemble](#ensembles) when the two samples
-should come from different models: a model rarely disagrees with itself at low temperature, but a cheap and a
-stronger model often disagree on exactly the cases the cheap one gets confidently wrong.
+mistake, but two independent samples rarely make the same one.
 
 ```ruby
 class TicketTriager
@@ -47,15 +46,14 @@ The first level that declares one wins: `squish!`, then the method, then the cla
 
 | Option | Harnesses | Description |
 |---|---|---|
-| `type:` | all | `:escalation`, `:squishsum`, `:judged_squishsum`, `:ensemble`, or `:judged_ensemble` (required in the Hash form). |
-| `compare:` | squishsum and ensemble ones | `->(a, b, **inputs) { ... }`, evaluated against the instance with the two typed results and the method's inputs. Truthy means the samples agree. Default: the two outputs are exactly equal. |
-| `judge:` | judged ones | The judge step (see [The judge](#the-judge)). Default: the escalation's next step after the sampled ones. |
+| `type:` | all | `:escalation`, `:squishsum`, `:judged_squishsum`, or a [registered](#custom-harnesses) name (required in the Hash form). |
+| `compare:` | squishsum ones | `->(a, b, **inputs) { ... }`, evaluated against the instance with the two typed results and the method's inputs. Truthy means the samples agree. Default: the two outputs are exactly equal. |
+| `judge:` | judged ones | The judge step (see [The judge](#the-judge)). Default: the escalation's next step after the sampled one. |
 | `judge_instructions:` | judged ones | A String, or a Proc evaluated against the instance, that replaces the [default judge prompt](#the-default-judge-prompt). |
 
 ## Samples
 
-Under `:squishsum`, both samples run on the escalation's first step, with its model, provider, and params
-(an [ensemble](#ensembles) runs them on its first and second steps):
+Both samples run on the escalation's first step, with its model, provider, and params:
 
 - **They're independent.** Each sample has its own chat, and the two requests are sent concurrently on separate
   threads. Only the provider requests run on those threads. Parsing, schema validation, `squish_validate`,
@@ -63,9 +61,10 @@ Under `:squishsum`, both samples run on the escalation's first step, with its mo
   RubyLLM instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for `chat.ruby_llm`) do see
   each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
 - **Each one is retried like an escalation step.** A sample with invalid output is re-asked in its own chat, up to
-  their step's `attempts:`, while a sample that already passed keeps its result. A failed request gets a fresh
+  the first step's `attempts:`, while a sample that already passed keeps its result. A failed request gets a fresh
   chat. A sample still failing after its attempts fails the call with `InvalidOutputError` or `LLMError`, as
-  usual. Under `:squishsum`, later escalation steps are never used for samples (see [Ensembles](#ensembles)).
+  usual. Later escalation steps are never used for samples (a judge may use the next one, see
+  [The judge](#the-judge)).
 - **Agreement is exact by default.** Free-text fields seldom match word for word, so pass `compare:` to decide
   which fields must match:
 
@@ -75,32 +74,6 @@ Under `:squishsum`, both samples run on the escalation's first step, with its mo
 
   When the samples agree, the first one is returned.
 
-## Ensembles
-
-`:ensemble` and `:judged_ensemble` sample two *different* steps instead of one step twice. Sample `a` is the
-escalation's first step and sample `b` its second, each with its own model, provider, and params:
-
-```ruby
-class TicketTriager
-  include Squishling
-
-  squishling escalation: [{ model: "claude-haiku-4-5", attempts: 2 }, "claude-sonnet-5-5", "claude-opus-5-5"],
-             harness: :judged_ensemble   # a = Haiku, b = Sonnet, Opus judges when they differ
-  # purpose, output_schema, ...
-end
-```
-
-- A step's `attempts:` are that sample's retries (here Haiku gets two attempts, Sonnet one), exactly as in
-  [Samples](#samples). Steps after the second are never sampled.
-- Everything else matches squishsum: independence and concurrency, `compare:`, `DisagreementError`, and the
-  judge. `DisagreementError#models` names the model behind each sample.
-- The judge defaults to the escalation's **third** step. With only two steps, a `:judged_ensemble` needs `judge:`.
-  An ensemble with fewer than two steps, or a judged one with neither a third step nor `judge:`, raises
-  `ConfigurationError` before any request.
-- A global `default_harness = :ensemble` therefore needs every class to declare at least two steps, and a
-  `squish!(model: ...)` override (one step) can't be combined with an ensemble harness.
-- Steps are told apart by value, so two adjacent steps with the same model, provider, and params count as one step.
-
 ## The judge
 
 With a judged harness, two valid samples that disagree go to a judge. The judge picks candidate `a` or `b`,
@@ -110,9 +83,9 @@ whose typed result is returned (`squished?` is `true`), or it picks `neither`, w
 The judge is:
 
 1. the `judge:` step, if declared: a model name, or a Hash with `model:`, `provider:`, `params:`, and `attempts:`
-   (as in an escalation step, without `order:`), plus `type:`; or
-2. the escalation's next step after the sampled ones (the second under `:judged_squishsum`, the third under
-   `:judged_ensemble`), with its attempts.
+   (as in an escalation step, without `priority:` or `rotate:`), plus `type:`; or
+2. the escalation's next step after the sampled one, with its attempts. Steps tied on `priority:` rotate between calls
+   (see [Rotating models](configuration.md#rotating-models-within-a-priority)), so which model that is can vary.
 
 A judged harness with neither raises `ConfigurationError` before any sample is requested. A `judge:`
 step uses only its own `provider:`; it doesn't inherit the class or method `provider:`, which belongs to their models
@@ -196,3 +169,158 @@ end
 Other failures are unchanged. A sample or judge that never produces valid output raises `InvalidOutputError`, a
 failed request on the last attempt raises `LLMError`, and setup mistakes (including a provider 400 on either
 sample) raise `ConfigurationError`, which is never sent to the fallback.
+
+## Custom harnesses
+
+A harness is a class that decides how to use the escalation: which samples to ask for, how to compare them, who
+breaks a tie. Register one under a name and it works anywhere a built-in type does: `squishling harness: :name`,
+`squish :method, harness: { type: :name, ... }`, `squish!(harness: ...)`, and `config.default_harness`. It gets
+everything the built-ins get: model, provider, and params resolution, schema validation and
+[`squish_validate`](failures.md#output-checks-squish_validate), retries and escalation,
+[`squawk`](failures.md#observing-every-attempt-squawk) reports with token usage, concurrent samples, and the
+[failure behavior](#when-a-custom-harness-fails) (`squish_fallback`, `DisagreementError`).
+
+### Worked example: two models and a tiebreaker
+
+Ask two cheap models. If they agree, done. If they don't, ask a third:
+
+```ruby
+class Tiebreaker < Squishling::Harness::Base
+  option :tiebreaker, :step, required: true   # a model name, or a step Hash
+  option :compare, :proc                      # optional, like squishsum's compare:
+
+  def call(run, tiebreaker:, compare: nil)
+    raise Squishling::ConfigurationError, "#{run.label}: needs two models" if run.stages.size < 2
+
+    first, second = run.samples(run.stages.first(2))   # the escalation's first two steps, asked concurrently
+    return first if run.agree?(first, second, compare)
+
+    run.escalate(tiebreaker)                           # they differ: a third opinion
+  end
+end
+
+Squishling.register_harness(:tiebreaker, Tiebreaker)   # e.g. in an initializer
+
+class TicketTriager
+  include Squishling
+
+  squishling escalation: ["claude-haiku-4-5", "claude-sonnet-5-5"],
+             harness: { type: :tiebreaker, tiebreaker: "claude-opus-5-5",
+                        compare: ->(a, b, **) { a.priority == b.priority } }
+  purpose "Assign a priority and a team."
+  output_schema do
+    string :priority, enum: %w[low medium high]
+    string :team
+  end
+end
+```
+
+`TicketTriager.call(...)` sends the ticket to Haiku and Sonnet at the same time. A pair that agrees on `priority` returns
+Haiku's result; otherwise Opus decides. Either way you get the same typed, validated result as every other path.
+`run.stages` groups identical neighbouring steps into one stage (they are one model's attempts), so this harness needs
+two different models in the escalation.
+
+For small harnesses, pass a block instead of a class. It becomes the body of a `Squishling::Harness::Base` subclass:
+
+```ruby
+Squishling.register_harness(:best_of_three) do
+  def call(run)
+    results = run.samples([run.stages.first] * 3)
+    results.max_by { |result| results.count { |other| run.agree?(result, other) } }   # the most-agreed-with sample
+  end
+end
+```
+
+### Registering
+
+`Squishling.register_harness(name, klass = nil, &block)` takes a `Squishling::Harness::Base` subclass or a block, and
+returns the class. `Squishling.harnesses` lists every name, built-ins first.
+
+- The registry is global, and a harness is looked up when it is **declared**, so an unknown name fails up front.
+  Register in an initializer, before the classes that use the name load.
+- A name is lowercase letters, digits, and underscores. Registering a name twice, or one of `:escalation`,
+  `:squishsum`, `:judged_squishsum`, raises `ConfigurationError`.
+- A class must define `call` taking the run as its first argument and each declared option as a keyword (a keyword
+  for an option that may be left out needs a default), and be creatable without arguments. This is checked at
+  registration.
+- Subclass `Squishling::Harness::Base` directly. The built-in harness classes are implementation, not an extension
+  point; write against `Run` as they do.
+- `Squishling.reset_harnesses!` forgets the registered harnesses (the built-ins stay). It is meant for specs.
+  `Squishling.reset_config!` leaves them, since they're registered once at boot.
+
+### Options
+
+Declare each option with `option name, type, required: false`, before registering the class. They are checked when
+the harness is declared, before any request is made, and an unknown, missing, or mistyped option raises
+`ConfigurationError`. A `nil` counts as not given, and `call` receives only the options that were given.
+
+| Type | Accepts | `call` receives |
+|---|---|---|
+| `:step` | A model name, or a Hash with `model:`, `provider:`, `params:`, `attempts:`, `forward_rejected:` | A normalized, frozen step; pass it to `run.escalate`, or in the Array given to `run.samples` |
+| `:proc` | A Proc | The Proc |
+| `:string`, `:symbol`, `:integer`, `:boolean` | A String, a Symbol (or String, converted), an Integer, `true` or `false` | The value |
+
+An option with a shape of its own takes a callable as its type instead. It gets the value, the declaration site (a
+label for error messages), and the option's name, and returns the normalized value, or raises `ConfigurationError`. The
+Hashes, Arrays, and Strings in the result are copied and frozen:
+
+```ruby
+option :word, ->(value, label, name) do
+  raise Squishling::ConfigurationError, "#{label}: #{name}: must be a String" unless value.is_a?(String)
+
+  value.upcase
+end
+```
+
+To check options against each other, define `def self.validate_options!(options, label)` and raise
+`Squishling::ConfigurationError` (include the `label`) to reject.
+
+### The run
+
+`call(run, **options)` gets a `Squishling::Run`: one squished call, with these primitives. Each returns typed
+results, the same type the deterministic path returns.
+
+| Method | What it does |
+|---|---|
+| `run.receiver`, `run.inputs`, `run.label` | The instance, the method's inputs, and `"Class#method"` |
+| `run.steps` | The escalation, one step per attempt |
+| `run.stages` | The escalation grouped by model: `[[haiku, haiku], [sonnet]]` for `[{ model: haiku, attempts: 2 }, sonnet]`. A stage is the attempts of one model |
+| `run.stage(declaration)` | A stage from a model name or a step Hash (a single step is a stage of one attempt). It keeps its own provider, and its `params:` merge over the method's, as a [judge](#the-judge) step does |
+| `run.escalate(*stages)` | Makes each attempt in order until an output passes the schema and `squish_validate`, in the same chat after invalid output and in a fresh one (told about the rejected output) on a new step. With no argument, it runs the escalation: exactly what `:escalation` does |
+| `run.samples(stages)` | Takes an Array of stages (so `run.samples([stage, stage])`, not `run.samples(stage)`; a bare step counts as a stage of one attempt): one sample per stage, each in its own chat and retried like an escalation step, with the requests of every round sent concurrently. Returns the outputs in order. Each stage must be one model's attempts: mixed steps raise `ConfigurationError`. Samples are named `"a"`, `"b"`, ... (up to 26) |
+| `run.judge(judge: nil, instructions: nil)` | Resolves a [judge](#the-judge) now (so a missing one raises `ConfigurationError` before any request) and returns it. `judge:` is a model name, or a [judge Hash](#the-judge) (`model:`, `provider:`, `params:`, `attempts:`, `type:`, `min_confidence:`; no `priority:` or `rotate:`), and `instructions:` a non-blank String or a Proc. `judge.verdict(first, second)` returns a `Squishling::Judge::Verdict` with `choice` (`:a`, `:b`, or `:neither`), `reason`, and `detail`, and `judge.step` is its first attempt. A chat judge's `reason` is model-written, so use `detail` (Squishling's own text, `nil` for a chat judge), not `reason`, in messages and logs. Without `judge:`, the judge is the escalation's second stage |
+| `run.agree?(first, second, compare = nil)` | `compare` (evaluated against the instance with both results and the method's inputs) or, without one, exact equality |
+| `run.log(message)` | A warning to `config.logger`, prefixed with `"[Squishling] Class#method"` |
+
+`call` returns the accepted result: one a primitive of this run returned. Returning anything else (a Hash, `nil`, a
+result you built or copied with `with`) raises `ConfigurationError`, so a harness can't skip validation.
+
+### When a custom harness fails
+
+A harness fails the way the built-ins do, so `squish_fallback` and the error classes behave predictably:
+
+- **Attempts run out.** `run.escalate` and `run.samples` raise `InvalidOutputError` (invalid output after the last
+  attempt) or `LLMError` (a failed request on the last attempt) themselves, with the same attributes as always.
+- **Your rule rejects.** Raise one of those errors yourself. `Squishling::DisagreementError.new(candidates:,
+  models:)` is the one for samples that don't agree, and `Squishling::InvalidOutputError.new(errors: [...], source: "Harness")` for
+  anything else. Both go to [`squish_fallback`](failures.md#fallbacks).
+- **Setup mistakes** raise `ConfigurationError`, which is never retried and never sent to the fallback.
+- **Your own exceptions** (from `call`, `compare:`, or `squish_validate`) propagate unchanged. Squishling doesn't wrap or
+  swallow them.
+
+Keep model output out of the messages you raise and the lines you `run.log`: it can quote the input. The output is
+available to your code as the results and in `InvalidOutputError#raw`, and to `squawk` if you opt in.
+
+### Reporting to `squawk`
+
+Every attempt a primitive makes is reported to the [`squawk`](failures.md#observing-every-attempt-squawk) hook, with
+usage, so [token measuring](measuring-tokens.md) works unchanged. `metadata[:role]` says what the attempt was for:
+`nil` for an escalation attempt, `:sample` (with `metadata[:sample]` set to `"a"`, `"b"`, ...), or `:judge`. `attempt`
+and `attempts` count within that sample, judge, or escalation.
+
+### Threads
+
+Only the provider requests inside `run.samples` run on worker threads. Your `call`, parsing, validation,
+`squish_validate`, and `compare:` all run on the calling thread, and the routing state stays fiber-local. Call a run's
+primitives from the thread `call` runs on: from another one they raise `ConfigurationError`. Don't keep unsynchronized
+state at class level in a harness, since one instance is created per call but the class is shared.

data/docs/measuring-tokens.md

@@ -10,7 +10,7 @@ first tells you when the second is worth paying.
   method (`label`, e.g. `"ShipmentUpdate#call"`), the model and provider, which attempt it was, and the token usage
   RubyLLM reported. That is enough to attribute spend to a squishling method without any other tooling.
 - Cost in dollars comes from RubyLLM's model pricing. Multiply the reported tokens by your model's price, or read it
-  from RubyLLM directly (see [the instrumenter](#lower-level-rubyllms-instrumenter)).
+  from RubyLLM directly (see [OpenTelemetry](#opentelemetry), or RubyLLM's `config.instrumenter`).
 
 ## Deciding when to write the code
 
@@ -31,8 +31,8 @@ stays small, the debt you haven't taken on is cheaper than the code.
 |---|---|---|
 | [Coolhand Labs](#coolhand-labs) | Picking up your squishling calls automatically and measuring their cost and accuracy | The `coolhand` gem and an initializer |
 | [Custom: the `squawk` hook](#custom-the-squawk-hook) | Logging or metrics in your own stack, with no new dependency | A lambda and one config line |
-| [OpenTelemetry](#opentelemetry) | Sending traces to Datadog, Langfuse, Arize, Braintrust, LangSmith, or any OTel backend | The OpenTelemetry SDK, an exporter, and one line |
-| [LangSmith](#langsmith) | Browsing traces and token usage in LangSmith | Through OpenTelemetry |
+| [Your own observer](#your-own-observer) | Wrapping each request in your tracing or metrics tool, knowing which method, level, and sample it was | An object with `around(call)` |
+| [OpenTelemetry](#opentelemetry) | Sending traces to Datadog, Langfuse, Arize, Braintrust, LangSmith, or any OTel backend | RubyLLM's tracer and an exporter |
 
 ### Coolhand Labs
 
@@ -51,7 +51,8 @@ end
 ```
 
 Get an API key at [coolhandlabs.com](https://coolhandlabs.com/). To keep the data in your own infrastructure, point
-`config.base_url` at a Coolhand-compatible endpoint of your own.
+`config.base_url` at a Coolhand-compatible endpoint of your own. Squishling also tells Coolhand which workload and
+template each call belongs to; see [Coolhand Labs](coolhand.md).
 
 ### Custom: the `squawk` hook
 
@@ -73,43 +74,39 @@ Each line carries the `"Class#method"` that made the call, so grouping by label
 that stays in Ruby makes no LLM request and never reaches the hook, so a label that stops appearing is a path that has
 been hardened. You can also set the hook per class (`squishling squawk: ...`) or per method. See
 [Observing every attempt](failures.md#observing-every-attempt-squawk) for every keyword and the rules for exceptions.
+Under a [harness](harnesses.md), including a [custom one](harnesses.md#custom-harnesses), every sample and judge
+request reaches the hook too. `metadata[:role]` (`nil`, `:sample`, or `:judge`) and `metadata[:sample]` (`"a"`, `"b"`, ...)
+split the spend by what each request was for.
 
-### Lower level: RubyLLM's instrumenter
+### Your own observer
 
-To meter every RubyLLM request in your app, not only Squishling's, use RubyLLM's own instrumentation. It reports a
-`usage.ruby_llm` event for each provider request, with `model`, `provider`, `status`, `tokens` (`input`, `output`,
-and more) and `cost`. Give it an object that responds to `instrument(name, payload)` and optionally takes a block. In
-Rails, `ActiveSupport::Notifications` is used automatically.
+An [observer](observers.md) wraps each provider request and gets the request's token usage back from `yield`, so it
+can meter spend knowing which method, escalation level, sample, or judge made the request. Any object that responds to
+`around(call)` is one:
 
 ```ruby
-class TokenMeter
-  def instrument(name, payload = {})
-    if name == "usage.ruby_llm"
-      Rails.logger.info("llm model=#{payload[:model]} workflow=#{payload[:workflow_name]} " \
-                        "in=#{payload[:tokens].input} out=#{payload[:tokens].output}")
-    end
-    block_given? ? yield : nil
+class SpendObserver
+  def around(call)
+    usage = yield # input_tokens, output_tokens (nil if unreported), and model; never the content
+    tags = ["label:#{call.label}", "role:#{call.role || 'squish'}", "model:#{usage.model}"]
+    Metrics.increment("squishling.tokens.in", usage.input_tokens.to_i, tags:)
+    Metrics.increment("squishling.tokens.out", usage.output_tokens.to_i, tags:)
   end
 end
 
-RubyLLM.configure { |config| config.instrumenter = TokenMeter.new }
+Squishling.configure { |config| config.observer = SpendObserver.new }
 ```
 
-These events don't know which squishling made the request. To attribute the spend, wrap the call in a RubyLLM
-workflow, and every event emitted inside the block carries its name, id and any `metadata:` you pass. (The
-`squawk` hook above needs no wrapping.)
-
-```ruby
-RubyLLM.workflow("ShipmentUpdate", metadata: { carrier: "acme-freight" }) do
-  ShipmentUpdate.call(carrier: "acme-freight", payload: body)
-end
-```
+An observer sees usage, never the model's output or your inputs. See [Observers](observers.md) for the `call` fields
+and the rules.
 
 ### OpenTelemetry
 
 RubyLLM 2.1+ ships its own OpenTelemetry tracer. Each request becomes a span with the model, request settings and
 token usage (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`), following the OpenTelemetry GenAI
-conventions. It does not record message text. Your app owns the SDK and exporters.
+conventions. It does not record message text. Your app owns the SDK and exporters, and the spans work with any OTLP
+backend, [LangSmith](https://docs.smith.langchain.com/observability/how_to_guides/tracing/trace_with_opentelemetry)
+included.
 
 ```ruby
 # Gemfile
@@ -121,16 +118,9 @@ OpenTelemetry::SDK.configure        # reads the standard OTEL_* environment vari
 RubyLLM::OpenTelemetry.enable       # starts tracing RubyLLM requests
 ```
 
-Spans don't say which squishling made the request, so pair this with a `RubyLLM.workflow` (see
-[above](#lower-level-rubyllms-instrumenter)) or group by model and prompt in your backend.
-
-### LangSmith
-
-LangSmith accepts OpenTelemetry traces, so the OpenTelemetry setup above covers it: point the OTLP exporter at
-LangSmith. See LangSmith's
-[OpenTelemetry docs](https://docs.smith.langchain.com/observability/how_to_guides/tracing/trace_with_opentelemetry)
-for the endpoint and headers. A community Ruby gem, [`langsmith-sdk`](https://github.com/felipekb/langsmith-ruby-sdk),
-also exists; it isn't published by LangChain.
+Spans don't say which squishling made the request. To attribute them, wrap the call in `RubyLLM.workflow("Name")`, or
+use an [observer](observers.md) that adds the label to the current span. RubyLLM's `config.instrumenter` also reports
+a `usage.ruby_llm` event per request, with tokens and cost, if you would rather meter than trace.
 
 ## Privacy
 
@@ -144,5 +134,7 @@ append. Tools that sit on the request path are a separate route out of your proc
   send them. That is the sanctioned route for model output; see
   [Observing every attempt](failures.md#observing-every-attempt-squawk). The examples above log only metadata.
 - **OpenTelemetry** with RubyLLM's tracer records usage and request settings, not message text.
+- **An observer** sees timing, token usage, and the call's metadata (method, model, level), never the model's output or
+  your inputs.
 
 Check what your tool stores before pointing it at inputs that contain personal data.

data/docs/observers.md

@@ -0,0 +1,80 @@
+# Observers
+
+An observer wraps every provider request Squishling makes, so another tool can see it while it happens: open a
+tracing span, tag the request, set a thread-local. It is the around-style counterpart to [`squawk`](failures.md#observing-every-attempt-squawk),
+which reports each attempt only after it is over.
+
+Squishling ships one, for [Coolhand Labs](coolhand.md), and uses it automatically when the `coolhand` gem is configured.
+Any object that responds to `around(call)` is an observer:
+
+```ruby
+class TracingObserver
+  def around(call)
+    Tracer.in_span("squishling #{call.label}", attributes: { "model" => call.model, "role" => call.role.to_s }) do
+      yield
+    end
+  end
+end
+
+Squishling.configure { |config| config.observer = TracingObserver.new }
+```
+
+`around` must yield exactly once. The block makes the request, so the observer runs around it, on the thread that
+makes it: for the concurrent samples of the squishsum harnesses, that is a worker thread, so thread-locals set by the
+observer are visible to the HTTP call. What `around` returns is ignored. One that never yields, or yields twice,
+raises `ConfigurationError`.
+
+## What `yield` returns
+
+The block returns a `Squishling::Observer::Usage`: the request's `input_tokens` and `output_tokens` (nil when the
+provider didn't report them) and the `model` that answered. That is enough to meter spend:
+
+```ruby
+class SpendObserver
+  def around(call)
+    usage = yield
+    Metrics.increment("squishling.tokens.in", usage.input_tokens.to_i, tags: ["label:#{call.label}"])
+    Metrics.increment("squishling.tokens.out", usage.output_tokens.to_i, tags: ["label:#{call.label}"])
+  end
+end
+```
+
+Usage never includes the model's output; Squishling returns the real response to its own caller, whatever `around`
+returns. If the request fails, the error is raised through `yield`, so a `begin/ensure` in `around` still runs.
+
+## `config.observer`
+
+| Value | Meaning |
+|---|---|
+| `nil` (default) | The Coolhand observer when `coolhand` is loaded and configured, otherwise none |
+| `:coolhand` | The Coolhand observer, always. Raises `ConfigurationError` if the gem is missing or too old |
+| any object responding to `around` | That observer |
+| `false` | None, even when Coolhand is configured |
+
+Anything else raises `ConfigurationError` when you assign it.
+
+## The `call`
+
+`around` receives a `Squishling::Observer::Call` (a `Data`) describing the request:
+
+| Field | |
+|---|---|
+| `label` | `"Class#method"` |
+| `purpose` | The purpose text sent as the system prompt (after `purpose` procs and `append_to_purpose` sections are resolved) |
+| `handed_off` | `true` when a `squish!` call sent the call to the LLM (typically from a `rescue`), `false` when `squish_when` or a missing implementation did |
+| `harness` | The harness type (`:escalation`, `:squishsum`, `:judged_squishsum`, or a [custom one](harnesses.md#custom-harnesses)) |
+| `role` | `nil` for an escalation attempt, `:sample` or `:judge`, as in `squawk`'s `metadata` |
+| `sample` | `"a"`, `"b"`, ... for a sample, else `nil` |
+| `attempt`, `attempts` | This request's 1-based position in the steps it was asked on, and how many there are: the escalation's expanded steps (or, for a custom harness, those passed to `run.escalate`), the sample's stage, or the judge's steps |
+| `model`, `provider` | The step's model and provider (`provider` is `nil` unless declared) |
+| `request` | `:chat`, or `:judgment` for a System One judge |
+
+It carries metadata only. The method's inputs, its context, and the model's output never reach an observer: it gets
+the request's token [`Usage`](#what-yield-returns) back, and `squawk` is the route for output.
+
+## Errors
+
+An observer is your code: an exception it raises propagates unchanged, and is never retried, escalated, or handed to
+`squish_fallback`. Squishling maps a failed request onto `LLMError` or `ConfigurationError` inside the block, so `around` sees those and
+can rescue them or let them pass. If `around` rescues one and returns normally, Squishling raises it again: an
+observer can watch a failure, not swallow it.

data/docs/routing.md

@@ -90,7 +90,7 @@ end
 - `append_to_purpose:` (added to the class's, see [Appending to the purpose](#appending-to-the-purpose))
 - `output_schema:` (or a schema block)
 - `model:` or `escalation:`, `provider:`, and `params:` (generation params; see [Configuration](configuration.md))
-- `harness:` (see [Harnesses](harnesses.md))
+- `harness:` (see [Harnesses](harnesses.md), including your own)
 - `when:`, a predicate proc
 - `validate:` and `fallback:` (see [Failure handling](failures.md))
 
@@ -210,7 +210,7 @@ Source is read with Ruby's own parser (Prism) the first time it's needed and cac
   "context":   { "customer_tier": "enterprise", "product": "API" } }
 ```
 
-- **Squishsum and ensemble harnesses:** both samples get exactly this input, and a [judge](harnesses.md#the-judge) gets
+- **Squishsum harnesses:** both samples get exactly this input, and a [judge](harnesses.md#the-judge) gets
   the same purpose and input plus both samples' outputs.
 - **Retries and escalation:** another attempt of the same step gets the validation errors in the same conversation. A
   later [escalation](configuration.md#models-and-escalation) step, which may be a different provider (say, local

data/docs/schemas.md

@@ -30,6 +30,23 @@ output_schema(
 )
 ```
 
+## Describing fields
+
+Every DSL type, and `optional`, takes a `description:`. It is sent to the provider as guidance for that field
+and isn't enforced; validation is by type and shape only.
+
+**Best practice:** put per-field guidance here rather than in the `purpose`. The description travels with the
+field in the schema the model fills in, so the instruction sits right where the value goes and next to the type
+or `enum` it explains. The purpose can then stay short and about the goal. For rules that must hold, use an
+`enum` or `squish_validate`; a description is a request, not a guarantee.
+
+```ruby
+output_schema do
+  string :language, description: "Lowercase language name, e.g. 'ruby'."
+  optional(:regex, description: "The proposed regex source, without delimiters or flags.") { string }
+end
+```
+
 ## Strict schemas only
 
 Squishling only supports **strict** output schemas: it always asks RubyLLM for strict structured output, and a
@@ -70,7 +87,7 @@ field nullable with the DSL's `optional`. The key stays required, but its value
 ```ruby
 output_schema do
   array :symptoms, of: :string      # always a list; [] means "none"
-  optional :allergies do            # [] means "none", nil means "not mentioned"
+  optional :allergies, description: "[] if the record says none; null if it doesn't mention allergies" do
     array of: :string
   end
   optional :vitals do               # a typed Data object, or nil
@@ -84,8 +101,8 @@ end
 `optional` produces `anyOf: [<schema>, {type: "null"}]`, which is strict-compatible. Squishling types the non-null
 branch as usual: `vitals` is a `Data` object or `nil`, and a list of objects is a list of `Data` objects. In a raw
 JSON Schema, `type: ["array", "null"]` works too. A union with more than one non-null branch is ambiguous, so its
-values come back as plain hashes. If you need the model to tell "none" apart from "not mentioned", say so in your
-purpose.
+values come back as plain hashes. If you need the model to tell "none" apart from "not mentioned", say so in the field's
+`description:`, as `allergies` does above.
 
 ## Contracts beyond the schema
 
@@ -111,7 +128,8 @@ Providers' strict modes don't support conditional keywords (`if`/`then`/`else` f
 `dependentSchemas` from `dependent`), so Squishling keeps them out of the schema it sends and enforces them itself.
 Output that breaks a rule is rejected like any other invalid output: the call moves on to the next attempt in its
 [escalation](configuration.md#models-and-escalation) with the errors. The model doesn't see these rules
-up front, so state them in your purpose too.
+up front, so state them in the `description:` of the field they govern (here `status`) or in your purpose. A
+description declared inside a `given` block is local-only too, so it is never sent.
 
 For anything a schema can't express (sums, lookups against your data), use
 [`squish_validate`](failures.md#output-checks-squish_validate).

data/lib/squishling/configuration.rb

@@ -24,8 +24,8 @@ module Squishling
     attr_reader :default_params
 
     # How every squishling class that doesn't declare its own harness uses its escalation (see Harness):
-    # :escalation (the default), :squishsum, :judged_squishsum, :ensemble, :judged_ensemble, or a Hash with
-    # type: and its options.
+    # :escalation (the default), :squishsum, :judged_squishsum, a harness you registered, or a Hash with type: and
+    # its options.
     attr_reader :default_harness
 
     # Optional Logger for routing and escalation decisions.
@@ -35,7 +35,13 @@ module Squishling
     #   ->(output:, metadata:, error:) { Tracer.record(output, metadata, error) }
     attr_reader :squawk
 
+    # Optional around-style hook wrapped around every provider request (see Observer): an object responding to
+    # around(call) { ... }, or :coolhand. nil (the default) uses the Coolhand observer when the coolhand gem is
+    # loaded and configured, and nothing otherwise; false never wraps.
+    attr_reader :observer
+
     def initialize
+      @observer = nil
       @default_model = nil
       @default_escalation = nil
       @default_model_path = nil
@@ -62,6 +68,13 @@ module Squishling
       @squawk = Squawk.validate(hook, "Squishling.configure")
     end
 
+    def observer=(observer)
+      Observer.validate(observer, "Squishling.configure")
+      # A named observer is built now, so a missing gem fails at setup rather than on the first request.
+      Observer.resolve(observer) if observer.is_a?(Symbol)
+      @observer = observer
+    end
+
     def default_params=(params)
       @default_params = Params.normalize(params, "default_params")
     end

data/lib/squishling/definition.rb

@@ -7,11 +7,11 @@ module Squishling
     # Matches the keys a squish! context: Hash may use.
     NAME_KEY = ->(key) { key.is_a?(String) || key.is_a?(Symbol) }
 
-    attr_reader :klass, :name, :call_context
+    attr_reader :klass, :name, :call_context, :handed_off
 
     def initialize(klass:, name:, purpose: nil, append_to_purpose: nil, output_schema: nil, model: nil,
       provider: nil, params: nil, harness: nil, predicate: nil, fallback: nil, validator: nil, squawk: nil,
-      call_context: {})
+      call_context: {}, handed_off: false)
       @klass = klass
       @name = name
       @purpose = purpose
@@ -26,6 +26,7 @@ module Squishling
       @validator = validator
       @squawk = squawk
       @call_context = call_context
+      @handed_off = handed_off
     end
 
     # This definition with one call's overrides on top. The output schema, predicate, fallback, and validator
@@ -50,7 +51,8 @@ module Squishling
         # merge, not Params.resolve: a nil at the method level must still unset the class's key.
         params: call_params ? (@params || {}).merge(call_params) : @params,
         harness: call_harness || @harness,
-        call_context: @call_context.merge((context || {}).transform_keys(&:to_sym))
+        call_context: @call_context.merge((context || {}).transform_keys(&:to_sym)),
+        handed_off: true # only squish! builds a definition this way; it is not part of a declared one
       )
     end
 
@@ -78,13 +80,15 @@ module Squishling
 
     # One ModelPath::Step per attempt, from the first level (method, class, or config) that declares a model
     # or escalation, or a single attempt on RubyLLM's default model. A provider travels with the model declared
-    # at the same level, so a per-method Anthropic model never inherits a class-level OpenAI provider.
+    # at the same level, so a per-method Anthropic model never inherits a class-level OpenAI provider. Models
+    # tied on priority are rotated by this method's call count (see Rotation).
     def escalation_path
       config = Squishling.config
       entries, provider = [[@model, @provider], [klass.squishling_model_path, klass.squishling_provider],
                            [config.default_model_path, config.default_provider]].find(&:first)
       entries ||= [{ model: nil, attempts: 1, forward_rejected: true }]
-      ModelPath.steps(entries, provider:, params:)
+      turn = ModelPath.rotating?(entries) ? Rotation.advance(klass, name) : 0
+      ModelPath.steps(entries, provider:, params:, turn:)
     end
 
     # How the escalation is used (see Harness): the method's, the class's, the configured default, or :escalation.

data/lib/squishling/errors.rb

@@ -23,11 +23,11 @@ module Squishling
     end
   end
 
-  # A sampling harness's samples (squishsum or ensemble) were valid but didn't agree, and either no judge was declared
-  # (verdict nil) or the judge rejected both (verdict :neither, with its reason). candidates holds the two
-  # typed results, so a squish_fallback can still return one of them, and raw both as Hashes. models names the
-  # model behind each role (sample a, sample b, then the judge if one ran), not one entry per attempt; attempts
-  # is nil.
+  # A harness's samples (the squishsum harnesses', or a custom one's) were valid but didn't agree, and either no
+  # judge was declared (verdict nil) or the judge rejected both (verdict :neither, with its reason). candidates
+  # holds the typed results, so a squish_fallback can still return one of them, and raw all of them as Hashes.
+  # models names the model behind each role (sample a, sample b, then the judge if one ran), not one entry per
+  # attempt; attempts is nil.
   class DisagreementError < InvalidOutputError
     attr_reader :candidates, :verdict, :reason