squishling 0.3.0 → 0.5.0

data/docs/harnesses.md

@@ -1,4 +1,4 @@
-# Harnesses: escalation, squishsum, and ensemble
+# Harnesses: escalation, squishsum, and your own
 
 A squished call's **harness** decides how it uses its [escalation](configuration.md#models-and-escalation):
 
@@ -7,13 +7,12 @@ A squished call's **harness** decides how it uses its [escalation](configuration
 | `:escalation` (default) | Tries each attempt in order until an output passes the schema and `squish_validate`. | 1 or more |
 | `:squishsum` | Asks the escalation's **first step** twice, concurrently. Identical outputs are accepted; different ones raise `Squishling::DisagreementError`. | 2 or more |
 | `:judged_squishsum` | Like `:squishsum`, but when the two outputs differ, a **judge** picks one or rejects both. | 2 or more, plus 1 or more judge requests when they differ |
-| `:ensemble` | Asks the escalation's **first and second steps** once each, concurrently, so two different models are compared. Identical outputs are accepted; different ones raise `DisagreementError`. | 2 or more |
-| `:judged_ensemble` | Like `:ensemble`, but when the two outputs differ, a **judge** picks one or rejects both. | 2 or more, plus 1 or more judge requests when they differ |
+
+Need a different trade-off, such as two models plus a tiebreaker or a majority vote? Register your own with
+[`Squishling.register_harness`](#custom-harnesses).
 
 Use a squishsum harness when a confidently wrong answer costs more than a second request: one sample can make a
-mistake, but two independent samples rarely make the same one. Use an [ensemble](#ensembles) when the two samples
-should come from different models: a model rarely disagrees with itself at low temperature, but a cheap and a
-stronger model often disagree on exactly the cases the cheap one gets confidently wrong.
+mistake, but two independent samples rarely make the same one.
 
 ```ruby
 class TicketTriager
@@ -47,25 +46,25 @@ The first level that declares one wins: `squish!`, then the method, then the cla
 
 | Option | Harnesses | Description |
 |---|---|---|
-| `type:` | all | `:escalation`, `:squishsum`, `:judged_squishsum`, `:ensemble`, or `:judged_ensemble` (required in the Hash form). |
-| `compare:` | squishsum and ensemble ones | `->(a, b, **inputs) { ... }`, evaluated against the instance with the two typed results and the method's inputs. Truthy means the samples agree. Default: the two outputs are exactly equal. |
-| `judge:` | judged ones | The judge step (see [The judge](#the-judge)). Default: the escalation's next step after the sampled ones. |
+| `type:` | all | `:escalation`, `:squishsum`, `:judged_squishsum`, or a [registered](#custom-harnesses) name (required in the Hash form). |
+| `compare:` | squishsum ones | `->(a, b, **inputs) { ... }`, evaluated against the instance with the two typed results and the method's inputs. Truthy means the samples agree. Default: the two outputs are exactly equal. |
+| `judge:` | judged ones | The judge step (see [The judge](#the-judge)). Default: the escalation's next step after the sampled one. |
 | `judge_instructions:` | judged ones | A String, or a Proc evaluated against the instance, that replaces the [default judge prompt](#the-default-judge-prompt). |
 
 ## Samples
 
-Under `:squishsum`, both samples run on the escalation's first step, with its model, provider, and params
-(an [ensemble](#ensembles) runs them on its first and second steps):
+Both samples run on the escalation's first step, with its model, provider, and params:
 
 - **They're independent.** Each sample has its own chat, and the two requests are sent concurrently on separate
   threads. Only the provider requests run on those threads. Parsing, schema validation, `squish_validate`,
   `compare:`, and the judge all run on the calling thread, so your Squishling callbacks never run concurrently.
-  RubyLLM instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for `chat.ruby_llm`) do see
-  each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
+  With the RubyLLM adapter, instrumentation subscribers (e.g. `ActiveSupport::Notifications` listeners for
+  `chat.ruby_llm`) do see each sample's request on its worker thread, outside the Rails executor, so keep them thread-safe.
 - **Each one is retried like an escalation step.** A sample with invalid output is re-asked in its own chat, up to
-  their step's `attempts:`, while a sample that already passed keeps its result. A failed request gets a fresh
+  the first step's `attempts:`, while a sample that already passed keeps its result. A failed request gets a fresh
   chat. A sample still failing after its attempts fails the call with `InvalidOutputError` or `LLMError`, as
-  usual. Under `:squishsum`, later escalation steps are never used for samples (see [Ensembles](#ensembles)).
+  usual. Later escalation steps are never used for samples (a judge may use the next one, see
+  [The judge](#the-judge)).
 - **Agreement is exact by default.** Free-text fields seldom match word for word, so pass `compare:` to decide
   which fields must match:
 
@@ -75,32 +74,6 @@ Under `:squishsum`, both samples run on the escalation's first step, with its mo
 
   When the samples agree, the first one is returned.
 
-## Ensembles
-
-`:ensemble` and `:judged_ensemble` sample two *different* steps instead of one step twice. Sample `a` is the
-escalation's first step and sample `b` its second, each with its own model, provider, and params:
-
-```ruby
-class TicketTriager
-  include Squishling
-
-  squishling escalation: [{ model: "claude-haiku-4-5", attempts: 2 }, "claude-sonnet-5-5", "claude-opus-5-5"],
-             harness: :judged_ensemble   # a = Haiku, b = Sonnet, Opus judges when they differ
-  # purpose, output_schema, ...
-end
-```
-
-- A step's `attempts:` are that sample's retries (here Haiku gets two attempts, Sonnet one), exactly as in
-  [Samples](#samples). Steps after the second are never sampled.
-- Everything else matches squishsum: independence and concurrency, `compare:`, `DisagreementError`, and the
-  judge. `DisagreementError#models` names the model behind each sample.
-- The judge defaults to the escalation's **third** step. With only two steps, a `:judged_ensemble` needs `judge:`.
-  An ensemble with fewer than two steps, or a judged one with neither a third step nor `judge:`, raises
-  `ConfigurationError` before any request.
-- A global `default_harness = :ensemble` therefore needs every class to declare at least two steps, and a
-  `squish!(model: ...)` override (one step) can't be combined with an ensemble harness.
-- Steps are told apart by value, so two adjacent steps with the same model, provider, and params count as one step.
-
 ## The judge
 
 With a judged harness, two valid samples that disagree go to a judge. The judge picks candidate `a` or `b`,
@@ -110,9 +83,9 @@ whose typed result is returned (`squished?` is `true`), or it picks `neither`, w
 The judge is:
 
 1. the `judge:` step, if declared: a model name, or a Hash with `model:`, `provider:`, `params:`, and `attempts:`
-   (as in an escalation step, without `order:`), plus `type:`; or
-2. the escalation's next step after the sampled ones (the second under `:judged_squishsum`, the third under
-   `:judged_ensemble`), with its attempts.
+   (as in an escalation step, without `priority:`, `rotate:`, or `include_when:`), plus `type:`; or
+2. the escalation's next step after the sampled one, with its attempts. Steps tied on `priority:` rotate between calls
+   (see [Rotating models](configuration.md#rotating-models-within-a-priority)), so which model that is can vary.
 
 A judged harness with neither raises `ConfigurationError` before any sample is requested. A `judge:`
 step uses only its own `provider:`; it doesn't inherit the class or method `provider:`, which belongs to their models
@@ -135,8 +108,10 @@ valid verdict raises `InvalidOutputError`. A chat judge's params are the method'
 ### System One judgment models (Jev)
 
 A judge can also be a System One decision model, such as TypeSafe's Jev, via
-[RubyLLM judgments](https://rubyllm.com/judgments/) (RubyLLM 2.1+). Decision models return calibrated
-probabilities rather than text, so they are much faster and cheaper than a chat judge.
+[RubyLLM judgments](https://rubyllm.com/judgments/) (RubyLLM 2.1+). This needs an [adapter](adapters.md) that
+supports judgments, which today means the `:ruby_llm` adapter; on any other, a `type: :judgment` judge raises
+`ConfigurationError` before a request is made (a `type: :chat` judge works with every adapter). Decision models return
+calibrated probabilities rather than text, so they are much faster and cheaper than a chat judge.
 
 ```ruby
 squishling harness: {
@@ -149,8 +124,8 @@ Squishling asks one `choice` question (`a`, `b`, or `neither`), with the judge p
 operation's purpose, input, output schema, and candidates as the judgment input. A pick of `a` or `b` is
 accepted only when its probability is at least `min_confidence:` (default `0.8`). Otherwise, as with `neither`,
 `DisagreementError` is raised with the choice and its probability in its `reason`. A judgment judge sends only its own
-`params:` (as RubyLLM `provider_options:`). `attempts:` retries a failed request. Configure the provider's
-credentials in RubyLLM as usual.
+`params:` (as RubyLLM `provider_options:`). `attempts:` retries a failed request. Judgments are a RubyLLM feature, so
+they need the `:ruby_llm` adapter, with the provider's credentials configured in RubyLLM as usual.
 
 ### The default judge prompt
 
@@ -193,6 +168,169 @@ end
 | `attempts` | `nil` |
 | `models` | The model behind each role: sample `a`, sample `b`, then the judge if one ran (one entry each, not one per attempt) |
 
+A [token budget](failures.md#capping-a-calls-cost-token-budgets) covers every request of the call, the samples
+and the judge's included. When it is spent before the next round of retries or before the judge can run, the call raises
+`BudgetExhaustedError` (also an `InvalidOutputError`, and also sent to the fallback) instead of a `DisagreementError`.
+
 Other failures are unchanged. A sample or judge that never produces valid output raises `InvalidOutputError`, a
 failed request on the last attempt raises `LLMError`, and setup mistakes (including a provider 400 on either
 sample) raise `ConfigurationError`, which is never sent to the fallback.
+
+## Custom harnesses
+
+A harness is a class that decides how to use the escalation: which samples to ask for, how to compare them, who
+breaks a tie. Register one under a name and it works anywhere a built-in type does: `squishling harness: :name`,
+`squish :method, harness: { type: :name, ... }`, `squish!(harness: ...)`, and `config.default_harness`. It gets
+everything the built-ins get: model, provider, and params resolution, schema validation and
+[`squish_validate`](failures.md#output-checks-squish_validate), retries and escalation,
+[`squawk`](failures.md#observing-every-attempt-squawk) reports with token usage, concurrent samples, and the
+[failure behavior](#when-a-custom-harness-fails) (`squish_fallback`, `DisagreementError`).
+
+### Worked example: two models and a tiebreaker
+
+Ask two cheap models. If they agree, done. If they don't, ask a third:
+
+```ruby
+class Tiebreaker < Squishling::Harness::Base
+  option :tiebreaker, :step, required: true   # a model name, or a step Hash
+  option :compare, :proc                      # optional, like squishsum's compare:
+
+  def call(run, tiebreaker:, compare: nil)
+    raise Squishling::ConfigurationError, "#{run.label}: needs two models" if run.stages.size < 2
+
+    first, second = run.samples(run.stages.first(2))   # the escalation's first two steps, asked concurrently
+    return first if run.agree?(first, second, compare)
+
+    run.escalate(tiebreaker)                           # they differ: a third opinion
+  end
+end
+
+Squishling.register_harness(:tiebreaker, Tiebreaker)   # e.g. in an initializer
+
+class TicketTriager
+  include Squishling
+
+  squishling escalation: ["claude-haiku-4-5", "claude-sonnet-5-5"],
+             harness: { type: :tiebreaker, tiebreaker: "claude-opus-5-5",
+                        compare: ->(a, b, **) { a.priority == b.priority } }
+  purpose "Assign a priority and a team."
+  output_schema do
+    string :priority, enum: %w[low medium high]
+    string :team
+  end
+end
+```
+
+`TicketTriager.call(...)` sends the ticket to Haiku and Sonnet at the same time. A pair that agrees on `priority` returns
+Haiku's result; otherwise Opus decides. Either way you get the same typed, validated result as every other path.
+`run.stages` groups identical neighbouring steps into one stage (they are one model's attempts), so this harness needs
+two different models in the escalation.
+
+For small harnesses, pass a block instead of a class. It becomes the body of a `Squishling::Harness::Base` subclass:
+
+```ruby
+Squishling.register_harness(:best_of_three) do
+  def call(run)
+    results = run.samples([run.stages.first] * 3)
+    results.max_by { |result| results.count { |other| run.agree?(result, other) } }   # the most-agreed-with sample
+  end
+end
+```
+
+### Registering
+
+`Squishling.register_harness(name, klass = nil, &block)` takes a `Squishling::Harness::Base` subclass or a block, and
+returns the class. `Squishling.harnesses` lists every name, built-ins first.
+
+- The registry is global, and a harness is looked up when it is **declared**, so an unknown name fails up front.
+  Register in an initializer, before the classes that use the name load.
+- A name is lowercase letters, digits, and underscores. Registering a name twice, or one of `:escalation`,
+  `:squishsum`, `:judged_squishsum`, raises `ConfigurationError`.
+- A class must define `call` taking the run as its first argument and each declared option as a keyword (a keyword
+  for an option that may be left out needs a default), and be creatable without arguments. This is checked at
+  registration.
+- Subclass `Squishling::Harness::Base` directly. The built-in harness classes are implementation, not an extension
+  point; write against `Run` as they do.
+- `Squishling.reset_harnesses!` forgets the registered harnesses (the built-ins stay). It is meant for specs.
+  `Squishling.reset_config!` leaves them, since they're registered once at boot.
+
+### Options
+
+Declare each option with `option name, type, required: false`, before registering the class. They are checked when
+the harness is declared, before any request is made, and an unknown, missing, or mistyped option raises
+`ConfigurationError`. A `nil` counts as not given, and `call` receives only the options that were given.
+
+| Type | Accepts | `call` receives |
+|---|---|---|
+| `:step` | A model name, or a Hash with `model:`, `provider:`, `params:`, `attempts:`, `forward_rejected:`, `retry_temperature:` | A normalized, frozen step; pass it to `run.escalate`, or in the Array given to `run.samples` |
+| `:proc` | A Proc | The Proc |
+| `:string`, `:symbol`, `:integer`, `:boolean` | A String, a Symbol (or String, converted), an Integer, `true` or `false` | The value |
+
+An option with a shape of its own takes a callable as its type instead. It gets the value, the declaration site (a
+label for error messages), and the option's name, and returns the normalized value, or raises `ConfigurationError`. The
+Hashes, Arrays, and Strings in the result are copied and frozen:
+
+```ruby
+option :word, ->(value, label, name) do
+  raise Squishling::ConfigurationError, "#{label}: #{name}: must be a String" unless value.is_a?(String)
+
+  value.upcase
+end
+```
+
+To check options against each other, define `def self.validate_options!(options, label)` and raise
+`Squishling::ConfigurationError` (include the `label`) to reject.
+
+### The run
+
+`call(run, **options)` gets a `Squishling::Run`: one squished call, with these primitives. Each returns typed
+results, the same type the deterministic path returns.
+
+| Method | What it does |
+|---|---|
+| `run.receiver`, `run.inputs`, `run.label` | The instance, the method's inputs, and `"Class#method"` |
+| `run.steps` | The escalation, one step per attempt |
+| `run.stages` | The escalation grouped by model: `[[haiku, haiku], [sonnet]]` for `[{ model: haiku, attempts: 2 }, sonnet]`. A stage is the attempts of one model |
+| `run.stage(declaration)` | A stage from a model name or a step Hash (a single step is a stage of one attempt). It keeps its own provider, and its `params:` merge over the method's, as a [judge](#the-judge) step does |
+| `run.escalate(*stages)` | Makes each attempt in order until an output passes the schema and `squish_validate`, in the same chat after invalid output and in a fresh one (told about the rejected output) on a new step. With no argument, it runs the escalation: exactly what `:escalation` does |
+| `run.samples(stages)` | Takes an Array of stages (so `run.samples([stage, stage])`, not `run.samples(stage)`; a bare step counts as a stage of one attempt): one sample per stage, each in its own chat and retried like an escalation step, with the requests of every round sent concurrently. Returns the outputs in order. Each stage must be one model's attempts: mixed steps raise `ConfigurationError`. Samples are named `"a"`, `"b"`, ... (up to 26) |
+| `run.judge(judge: nil, instructions: nil)` | Resolves a [judge](#the-judge) now (so a missing one raises `ConfigurationError` before any request) and returns it. `judge:` is a model name, or a [judge Hash](#the-judge) (`model:`, `provider:`, `params:`, `attempts:`, `type:`, `min_confidence:`; no `priority:`, `rotate:`, or `include_when:`), and `instructions:` a non-blank String or a Proc. `judge.verdict(first, second)` returns a `Squishling::Judge::Verdict` with `choice` (`:a`, `:b`, or `:neither`), `reason`, and `detail`, and `judge.step` is its first attempt. A chat judge's `reason` is model-written, so use `detail` (Squishling's own text, `nil` for a chat judge), not `reason`, in messages and logs. Without `judge:`, the judge is the escalation's second stage |
+| `run.agree?(first, second, compare = nil)` | `compare` (evaluated against the instance with both results and the method's inputs) or, without one, exact equality |
+| `run.log(message)` | A warning to `config.logger`, prefixed with `"[Squishling] Class#method"` |
+
+`call` returns the accepted result: one a primitive of this run returned. Returning anything else (a Hash, `nil`, a
+result you built or copied with `with`) raises `ConfigurationError`, so a harness can't skip validation.
+
+### When a custom harness fails
+
+A harness fails the way the built-ins do, so `squish_fallback` and the error classes behave predictably:
+
+- **Attempts run out.** `run.escalate` and `run.samples` raise `InvalidOutputError` (invalid output after the last
+  attempt) or `LLMError` (a failed request on the last attempt) themselves, with the same attributes as always.
+- **Your rule rejects.** Raise one of those errors yourself. `Squishling::DisagreementError.new(candidates:,
+  models:)` is the one for samples that don't agree, and `Squishling::InvalidOutputError.new(errors: [...],
+  source: "Harness")` for anything else. Both go to [`squish_fallback`](failures.md#fallbacks). A String error is
+  tagged `:squishling` in [`error_details`](failures.md#telling-the-entries-apart); pass a
+  `Squishling::InvalidOutputError::Detail` to tag it yourself.
+- **Setup mistakes** raise `ConfigurationError`, which is never retried and never sent to the fallback.
+- **Your own exceptions** (from `call`, `compare:`, or `squish_validate`) propagate unchanged. Squishling doesn't wrap or
+  swallow them.
+
+Keep model output out of the messages you raise and the lines you `run.log`: it can quote the input. The output is
+available to your code as the results and in `InvalidOutputError#raw`, and to `squawk` if you opt in.
+
+### Reporting to `squawk`
+
+Every attempt a primitive makes is reported to the [`squawk`](failures.md#observing-every-attempt-squawk) hook
+(unless narrowed with `squawk_on:`), with usage, so [token measuring](measuring-tokens.md) works unchanged. `metadata[:role]` says what the attempt was for:
+`nil` for an escalation attempt, `:sample` (with `metadata[:sample]` set to `"a"`, `"b"`, ...), or `:judge`. `attempt`
+and `attempts` count within that sample, judge, or escalation. `metadata[:repeated]` is `true` for an attempt that
+sent back the reply it was just rejected for; the primitive then skips the rest of that step's attempts (the next
+step's `attempt` jumps past them, and a sample ends with its `InvalidOutputError`).
+
+### Threads
+
+Only the provider requests inside `run.samples` run on worker threads. Your `call`, parsing, validation,
+`squish_validate`, and `compare:` all run on the calling thread, and the routing state stays fiber-local. Call a run's
+primitives from the thread `call` runs on: from another one they raise `ConfigurationError`. Don't keep unsynchronized
+state at class level in a harness, since one instance is created per call but the class is shared.

data/docs/measuring-tokens.md

@@ -6,11 +6,13 @@ first tells you when the second is worth paying.
 ## What Squishling gives you
 
 - `result.squished?` says which path served a call, so you can count LLM calls per class.
-- The [`squawk` hook](failures.md#observing-every-attempt-squawk) runs after every LLM attempt and receives the
+- The [`squawk` hook](failures.md#observing-every-attempt-squawk) runs after every LLM attempt (by default; see `squawk_on` below) and receives the
   method (`label`, e.g. `"ShipmentUpdate#call"`), the model and provider, which attempt it was, and the token usage
-  RubyLLM reported. That is enough to attribute spend to a squishling method without any other tooling.
-- Cost in dollars comes from RubyLLM's model pricing. Multiply the reported tokens by your model's price, or read it
-  from RubyLLM directly (see [the instrumenter](#lower-level-rubyllms-instrumenter)).
+  the adapter reported. That is enough to attribute spend to a squishling method without any other tooling.
+- A [token budget](failures.md#capping-a-calls-cost-token-budgets) (`max_total_output_tokens:`, `max_total_input_tokens:`)
+  caps what a single call may use across all its attempts, so a call that keeps failing can't spend without limit.
+- Cost in dollars isn't reported by Squishling's adapters. Multiply the reported tokens by your model's price, or, with
+  the RubyLLM adapter, read it from RubyLLM directly (see [OpenTelemetry](#opentelemetry), or RubyLLM's `config.instrumenter`).
 
 ## Deciding when to write the code
 
@@ -31,8 +33,8 @@ stays small, the debt you haven't taken on is cheaper than the code.
 |---|---|---|
 | [Coolhand Labs](#coolhand-labs) | Picking up your squishling calls automatically and measuring their cost and accuracy | The `coolhand` gem and an initializer |
 | [Custom: the `squawk` hook](#custom-the-squawk-hook) | Logging or metrics in your own stack, with no new dependency | A lambda and one config line |
-| [OpenTelemetry](#opentelemetry) | Sending traces to Datadog, Langfuse, Arize, Braintrust, LangSmith, or any OTel backend | The OpenTelemetry SDK, an exporter, and one line |
-| [LangSmith](#langsmith) | Browsing traces and token usage in LangSmith | Through OpenTelemetry |
+| [Your own observer](#your-own-observer) | Wrapping each request in your tracing or metrics tool, knowing which method, level, and sample it was | An object with `around(call)` |
+| [OpenTelemetry](#opentelemetry) | Sending traces to Datadog, Langfuse, Arize, Braintrust, LangSmith, or any OTel backend | RubyLLM's tracer and an exporter |
 
 ### Coolhand Labs
 
@@ -51,12 +53,13 @@ end
 ```
 
 Get an API key at [coolhandlabs.com](https://coolhandlabs.com/). To keep the data in your own infrastructure, point
-`config.base_url` at a Coolhand-compatible endpoint of your own.
+`config.base_url` at a Coolhand-compatible endpoint of your own. Squishling also tells Coolhand which workload and
+template each call belongs to; see [Coolhand Labs](coolhand.md).
 
 ### Custom: the `squawk` hook
 
-`config.squawk` runs after every LLM attempt, accepted or rejected, and is handed the method's label, the model,
-and the token usage RubyLLM reported. A lambda can take just the keywords it needs:
+`config.squawk` runs after every LLM attempt, accepted or rejected (unless you narrow it with `squawk_on:`, which makes token totals undercount, so keep `:all` for measuring), and is handed the method's label, the model,
+and the token usage the adapter reported. A lambda can take just the keywords it needs:
 
 ```ruby
 Squishling.configure do |config|
@@ -73,43 +76,39 @@ Each line carries the `"Class#method"` that made the call, so grouping by label
 that stays in Ruby makes no LLM request and never reaches the hook, so a label that stops appearing is a path that has
 been hardened. You can also set the hook per class (`squishling squawk: ...`) or per method. See
 [Observing every attempt](failures.md#observing-every-attempt-squawk) for every keyword and the rules for exceptions.
+Under a [harness](harnesses.md), including a [custom one](harnesses.md#custom-harnesses), every sample and judge
+request reaches the hook too. `metadata[:role]` (`nil`, `:sample`, or `:judge`) and `metadata[:sample]` (`"a"`, `"b"`, ...)
+split the spend by what each request was for.
 
-### Lower level: RubyLLM's instrumenter
+### Your own observer
 
-To meter every RubyLLM request in your app, not only Squishling's, use RubyLLM's own instrumentation. It reports a
-`usage.ruby_llm` event for each provider request, with `model`, `provider`, `status`, `tokens` (`input`, `output`,
-and more) and `cost`. Give it an object that responds to `instrument(name, payload)` and optionally takes a block. In
-Rails, `ActiveSupport::Notifications` is used automatically.
+An [observer](observers.md) wraps each provider request and gets the request's token usage back from `yield`, so it
+can meter spend knowing which method, escalation level, sample, or judge made the request. Any object that responds to
+`around(call)` is one:
 
 ```ruby
-class TokenMeter
-  def instrument(name, payload = {})
-    if name == "usage.ruby_llm"
-      Rails.logger.info("llm model=#{payload[:model]} workflow=#{payload[:workflow_name]} " \
-                        "in=#{payload[:tokens].input} out=#{payload[:tokens].output}")
-    end
-    block_given? ? yield : nil
+class SpendObserver
+  def around(call)
+    usage = yield # input_tokens, output_tokens (nil if unreported), and model; never the content
+    tags = ["label:#{call.label}", "role:#{call.role || 'squish'}", "model:#{usage.model}"]
+    Metrics.increment("squishling.tokens.in", usage.input_tokens.to_i, tags:)
+    Metrics.increment("squishling.tokens.out", usage.output_tokens.to_i, tags:)
   end
 end
 
-RubyLLM.configure { |config| config.instrumenter = TokenMeter.new }
+Squishling.configure { |config| config.observer = SpendObserver.new }
 ```
 
-These events don't know which squishling made the request. To attribute the spend, wrap the call in a RubyLLM
-workflow, and every event emitted inside the block carries its name, id and any `metadata:` you pass. (The
-`squawk` hook above needs no wrapping.)
-
-```ruby
-RubyLLM.workflow("ShipmentUpdate", metadata: { carrier: "acme-freight" }) do
-  ShipmentUpdate.call(carrier: "acme-freight", payload: body)
-end
-```
+An observer sees usage, never the model's output or your inputs. See [Observers](observers.md) for the `call` fields
+and the rules.
 
 ### OpenTelemetry
 
 RubyLLM 2.1+ ships its own OpenTelemetry tracer. Each request becomes a span with the model, request settings and
 token usage (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`), following the OpenTelemetry GenAI
-conventions. It does not record message text. Your app owns the SDK and exporters.
+conventions. It does not record message text. Your app owns the SDK and exporters, and the spans work with any OTLP
+backend, [LangSmith](https://docs.smith.langchain.com/observability/how_to_guides/tracing/trace_with_opentelemetry)
+included.
 
 ```ruby
 # Gemfile
@@ -121,16 +120,9 @@ OpenTelemetry::SDK.configure        # reads the standard OTEL_* environment vari
 RubyLLM::OpenTelemetry.enable       # starts tracing RubyLLM requests
 ```
 
-Spans don't say which squishling made the request, so pair this with a `RubyLLM.workflow` (see
-[above](#lower-level-rubyllms-instrumenter)) or group by model and prompt in your backend.
-
-### LangSmith
-
-LangSmith accepts OpenTelemetry traces, so the OpenTelemetry setup above covers it: point the OTLP exporter at
-LangSmith. See LangSmith's
-[OpenTelemetry docs](https://docs.smith.langchain.com/observability/how_to_guides/tracing/trace_with_opentelemetry)
-for the endpoint and headers. A community Ruby gem, [`langsmith-sdk`](https://github.com/felipekb/langsmith-ruby-sdk),
-also exists; it isn't published by LangChain.
+Spans don't say which squishling made the request. To attribute them, wrap the call in `RubyLLM.workflow("Name")`, or
+use an [observer](observers.md) that adds the label to the current span. RubyLLM's `config.instrumenter` also reports
+a `usage.ruby_llm` event per request, with tokens and cost, if you would rather meter than trace.
 
 ## Privacy
 
@@ -144,5 +136,7 @@ append. Tools that sit on the request path are a separate route out of your proc
   send them. That is the sanctioned route for model output; see
   [Observing every attempt](failures.md#observing-every-attempt-squawk). The examples above log only metadata.
 - **OpenTelemetry** with RubyLLM's tracer records usage and request settings, not message text.
+- **An observer** sees timing, token usage, and the call's metadata (method, model, level), never the model's output or
+  your inputs.
 
 Check what your tool stores before pointing it at inputs that contain personal data.

data/docs/observers.md

@@ -0,0 +1,80 @@
+# Observers
+
+An observer wraps every provider request Squishling makes, so another tool can see it while it happens: open a
+tracing span, tag the request, set a thread-local. It is the around-style counterpart to [`squawk`](failures.md#observing-every-attempt-squawk),
+which reports each attempt only after it is over.
+
+Squishling ships one, for [Coolhand Labs](coolhand.md), and uses it automatically when the `coolhand` gem is configured.
+Any object that responds to `around(call)` is an observer:
+
+```ruby
+class TracingObserver
+  def around(call)
+    Tracer.in_span("squishling #{call.label}", attributes: { "model" => call.model, "role" => call.role.to_s }) do
+      yield
+    end
+  end
+end
+
+Squishling.configure { |config| config.observer = TracingObserver.new }
+```
+
+`around` must yield exactly once. The block makes the request, so the observer runs around it, on the thread that
+makes it: for the concurrent samples of the squishsum harnesses, that is a worker thread, so thread-locals set by the
+observer are visible to the HTTP call. What `around` returns is ignored. One that never yields, or yields twice,
+raises `ConfigurationError`.
+
+## What `yield` returns
+
+The block returns a `Squishling::Observer::Usage`: the request's `input_tokens` and `output_tokens` (nil when the
+provider didn't report them) and the `model` that answered. That is enough to meter spend:
+
+```ruby
+class SpendObserver
+  def around(call)
+    usage = yield
+    Metrics.increment("squishling.tokens.in", usage.input_tokens.to_i, tags: ["label:#{call.label}"])
+    Metrics.increment("squishling.tokens.out", usage.output_tokens.to_i, tags: ["label:#{call.label}"])
+  end
+end
+```
+
+Usage never includes the model's output; Squishling returns the real response to its own caller, whatever `around`
+returns. If the request fails, the error is raised through `yield`, so a `begin/ensure` in `around` still runs.
+
+## `config.observer`
+
+| Value | Meaning |
+|---|---|
+| `nil` (default) | The Coolhand observer when `coolhand` is loaded and configured, otherwise none |
+| `:coolhand` | The Coolhand observer, always. Raises `ConfigurationError` if the gem is missing or too old |
+| any object responding to `around` | That observer |
+| `false` | None, even when Coolhand is configured |
+
+Anything else raises `ConfigurationError` when you assign it.
+
+## The `call`
+
+`around` receives a `Squishling::Observer::Call` (a `Data`) describing the request:
+
+| Field | |
+|---|---|
+| `label` | `"Class#method"` |
+| `purpose` | The purpose text sent as the system prompt (after `purpose` procs and `append_to_purpose` sections are resolved) |
+| `handed_off` | `true` when a `squish!` call sent the call to the LLM (typically from a `rescue`), `false` when `squish_when` or a missing implementation did |
+| `harness` | The harness type (`:escalation`, `:squishsum`, `:judged_squishsum`, or a [custom one](harnesses.md#custom-harnesses)) |
+| `role` | `nil` for an escalation attempt, `:sample` or `:judge`, as in `squawk`'s `metadata` |
+| `sample` | `"a"`, `"b"`, ... for a sample, else `nil` |
+| `attempt`, `attempts` | This request's 1-based position in the steps it was asked on, and how many there are: the escalation's expanded steps (or, for a custom harness, those passed to `run.escalate`), the sample's stage, or the judge's steps |
+| `model`, `provider` | The step's model and provider (`provider` is `nil` unless declared) |
+| `request` | `:chat`, or `:judgment` for a System One judge |
+
+It carries metadata only. The method's inputs, its context, and the model's output never reach an observer: it gets
+the request's token [`Usage`](#what-yield-returns) back, and `squawk` is the route for output.
+
+## Errors
+
+An observer is your code: an exception it raises propagates unchanged, and is never retried, escalated, or handed to
+`squish_fallback`. Squishling maps a failed request onto `LLMError` or `ConfigurationError` inside the block, so `around` sees those and
+can rescue them or let them pass. If `around` rescues one and returns normally, Squishling raises it again: an
+observer can watch a failure, not swallow it.

data/docs/routing.md

@@ -61,6 +61,9 @@ This is the workflow from [Elastic Software](https://everythingengineer.substack
 
 Callers never change, because both paths return the same result class.
 
+To keep a particular model off certain inputs (a client that needs a BAA, say), give its escalation step an
+`include_when:` Proc; see [Excluding a model per call](configuration.md#excluding-a-model-per-call).
+
 ## Entry points
 
 Squish methods other than `call` with `squish`, optionally overriding the class-level settings per method:
@@ -90,7 +93,8 @@ end
 - `append_to_purpose:` (added to the class's, see [Appending to the purpose](#appending-to-the-purpose))
 - `output_schema:` (or a schema block)
 - `model:` or `escalation:`, `provider:`, and `params:` (generation params; see [Configuration](configuration.md))
-- `harness:` (see [Harnesses](harnesses.md))
+- `harness:` (see [Harnesses](harnesses.md), including your own)
+- `max_total_output_tokens:` and `max_total_input_tokens:` (see [Capping a call's cost](failures.md#capping-a-calls-cost-token-budgets))
 - `when:`, a predicate proc
 - `validate:` and `fallback:` (see [Failure handling](failures.md))
 
@@ -134,6 +138,7 @@ end
 | `purpose:` | Replaces the purpose |
 | `model:` or `escalation:`, `provider:`, `params:` | E.g. send this call to a stronger model, or a whole [escalation](configuration.md#models-and-escalation), when Ruby fails. A `provider:` needs a `model:` or `escalation:`; `params:` merge key by key over the declared ones. |
 | `harness:` | Replaces the declared [harness](harnesses.md), e.g. `harness: :judged_squishsum` to double-check a hand-off |
+| `max_total_output_tokens:`, `max_total_input_tokens:` | Cap what this call may spend, e.g. a tighter budget for a hand-off; `false` lifts a declared cap. See [Capping a call's cost](failures.md#capping-a-calls-cost-token-budgets). |
 
 - **The output schema can't be overridden.** The call still returns the method's result type.
 - **Failures** go through the normal LLM path: a declared `squish_fallback` is used, otherwise
@@ -210,7 +215,7 @@ Source is read with Ruby's own parser (Prism) the first time it's needed and cac
   "context":   { "customer_tier": "enterprise", "product": "API" } }
 ```
 
-- **Squishsum and ensemble harnesses:** both samples get exactly this input, and a [judge](harnesses.md#the-judge) gets
+- **Squishsum harnesses:** both samples get exactly this input, and a [judge](harnesses.md#the-judge) gets
   the same purpose and input plus both samples' outputs.
 - **Retries and escalation:** another attempt of the same step gets the validation errors in the same conversation. A
   later [escalation](configuration.md#models-and-escalation) step, which may be a different provider (say, local

data/docs/schemas.md

@@ -30,10 +30,29 @@ output_schema(
 )
 ```
 
+## Describing fields
+
+Every DSL type, and `optional`, takes a `description:`. It is sent to the provider as guidance for that field
+and isn't enforced; validation is by type and shape only.
+
+**Best practice:** put per-field guidance here rather than in the `purpose`. The description travels with the
+field in the schema the model fills in, so the instruction sits right where the value goes and next to the type
+or `enum` it explains. The purpose can then stay short and about the goal. For rules that must hold, use an
+`enum` or `squish_validate`; a description is a request, not a guarantee.
+
+```ruby
+output_schema do
+  string :language, description: "Lowercase language name, e.g. 'ruby'."
+  optional(:regex, description: "The proposed regex source, without delimiters or flags.") { string }
+end
+```
+
 ## Strict schemas only
 
-Squishling only supports **strict** output schemas: it always asks RubyLLM for strict structured output, and a
-schema declaring `strict: false` raises `Squishling::ConfigurationError`. RubyLLM 2.0 on its own would send a
+Squishling only supports **strict** output schemas: it always asks the adapter for strict structured output (the
+OpenAI-compatible adapter's `structured_output: :json_object` and `:none` modes are the explicit exception, for servers
+that lack it, where the schema goes into the prompt instead), and a schema declaring `strict: false` raises
+`Squishling::ConfigurationError`. Squishling validates every answer against the schema either way. RubyLLM 2.0 on its own would send a
 schema with optional properties non-strict; Squishling always sets the flag explicitly, so it stays strict.
 
 With raw hashes, follow your provider's strict-mode rules (e.g. list every property in `required` and set
@@ -58,6 +77,13 @@ r.to_h                           # deep Hash with symbol keys
 r.squished?                      # => true when it came from the LLM
 ```
 
+A property can't share a name with a method the result depends on, because the `Data` member would replace it.
+Squishling raises `ConfigurationError` when the schema is built (before any request), naming the clashing
+fields; rename them (`class` to `ticket_class`). The reserved names are `class`, `is_a?`, `kind_of?`, `hash`, `inspect`, `to_s`,
+`to_h`, `members`, `with`, `dup`, `send`, `public_send`, `eql?`, `==`, `[]`, `squished?`, `object_id`,
+`deconstruct_keys`, and a few Ruby and Squishling internals (`initialize*`, `__send__`, `squishling_squished`).
+Other names, such as `method`, `format`, or `type`, work as usual. This applies to nested objects and array items too.
+
 On the deterministic path, return a `Hash` or build the result with `result(...)`. Either way it's validated
 against the schema, and so is anything else you return (`nil`, a string, another schema's result), which raises
 `Squishling::InvalidOutputError` unless it matches.
@@ -70,7 +96,7 @@ field nullable with the DSL's `optional`. The key stays required, but its value
 ```ruby
 output_schema do
   array :symptoms, of: :string      # always a list; [] means "none"
-  optional :allergies do            # [] means "none", nil means "not mentioned"
+  optional :allergies, description: "[] if the record says none; null if it doesn't mention allergies" do
     array of: :string
   end
   optional :vitals do               # a typed Data object, or nil
@@ -84,8 +110,8 @@ end
 `optional` produces `anyOf: [<schema>, {type: "null"}]`, which is strict-compatible. Squishling types the non-null
 branch as usual: `vitals` is a `Data` object or `nil`, and a list of objects is a list of `Data` objects. In a raw
 JSON Schema, `type: ["array", "null"]` works too. A union with more than one non-null branch is ambiguous, so its
-values come back as plain hashes. If you need the model to tell "none" apart from "not mentioned", say so in your
-purpose.
+values come back as plain hashes. If you need the model to tell "none" apart from "not mentioned", say so in the field's
+`description:`, as `allergies` does above.
 
 ## Contracts beyond the schema
 
@@ -111,7 +137,8 @@ Providers' strict modes don't support conditional keywords (`if`/`then`/`else` f
 `dependentSchemas` from `dependent`), so Squishling keeps them out of the schema it sends and enforces them itself.
 Output that breaks a rule is rejected like any other invalid output: the call moves on to the next attempt in its
 [escalation](configuration.md#models-and-escalation) with the errors. The model doesn't see these rules
-up front, so state them in your purpose too.
+up front, so state them in the `description:` of the field they govern (here `status`) or in your purpose. A
+description declared inside a `given` block is local-only too, so it is never sent.
 
 For anything a schema can't express (sums, lookups against your data), use
 [`squish_validate`](failures.md#output-checks-squish_validate).