squishling 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ad18b43555b041d5c0ef6eabda578fc6a2111ecfbf962b2003a7b502584f0c91
-  data.tar.gz: f4ce9bb54bd2c57abd73c5a2f0e50472c338e299f25fe7ed0dd2edb2f4854f06
+  metadata.gz: '0748f552c985521890d365ebc0818ec59c4336f93fb183d1769f49ed8cd32349'
+  data.tar.gz: '05128ecf8d7156e0c8ca084bf8a1860b754596017df2cdfdcc35e2242138fa7b'
 SHA512:
-  metadata.gz: 726d27bcc6067039d3bba294a923c302f49fc756ce0882541461525d8421be48d5cbb9c02e1d201eb1c062c9519353f62b66f9cf43795fc46a15c9834d7e24fe
-  data.tar.gz: 9e4093ad080e9742b38e6bfe3d59f2b5be21f0bf3ddfd5a8288b552bbe9d0abdffe3d47f6fea8d5d0c54afdb0172c69769451249e65f69cfc2665b96682b1b52
+  metadata.gz: 2be443b51d2dc1460a410fbf0850bed09e33fff5f7ad894d4bd92e3f47d428d22df7bfc0371036c090a6b806f466b1d930de3a9caf8d1b044185fd4000b765bf
+  data.tar.gz: 3a18d6e041f5807c966677fea5499de8af5d0f28146cb98bbe205141598ee6a01a2d598b1326df3b78227455dcc124c69cf95afacabf9a1689a1cc382a652414

data/CHANGELOG.md

@@ -7,6 +7,89 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-10-09
+
+### Added
+
+- Harnesses: a `harness:` option (on `squishling`, `squish`, `squish!`, and `default_harness` in
+  `Squishling.configure`) chooses how a call uses its escalation. `:escalation` is the existing behavior and stays
+  the default. `:squishsum` asks the first escalation step twice, concurrently, and accepts the result only when the
+  two samples agree (exact match, or a custom `compare:`). `:judged_squishsum` sends a disagreement to a judge: the
+  next escalation step or a dedicated `judge:` step, either a chat model or a Jev-style decision model through
+  `RubyLLM.judge`, with a default prompt you can override through `judge_instructions:`. Samples that still disagree
+  raise the new `Squishling::DisagreementError < InvalidOutputError`, which carries both typed results, and
+  `squish_fallback` remains the only fallback. See [Harnesses](docs/harnesses.md). (#16)
+- `:ensemble` and `:judged_ensemble` harnesses: like the squishsum ones, but the two concurrent samples come from the
+  escalation's first and second steps (a cross-model check) instead of the first step twice, each retrying within its
+  own step's `attempts:`. A judged ensemble's judge defaults to the third step, or `judge:`, and
+  `DisagreementError#models` names the model behind each sample. An ensemble needs at least two distinct steps (adjacent
+  identical steps count as one step's attempts) or it raises `ConfigurationError` before any request. (#20)
+- `squawk`, an opt-in hook for sending what the model returned to an error tracker or tracing tool. It is called after
+  every LLM attempt (every sample and judge attempt under a harness) with `output:`, `metadata:`, and `error:`, and
+  can be set on `Squishling.configure`, `squishling`, and `squish`. It does nothing by default, and exceptions it
+  raises propagate unchanged. (#15)
+- `forward_rejected:` on escalation steps. When an attempt is rejected and the next step starts, the previous
+  output is forwarded to that step's provider by default (unchanged behavior); set `forward_rejected: false` to start
+  that step from the original input alone. (#14)
+- `include Squishling` raises `ConfigurationError` when the class inherits a method Squishling would override (for
+  example `Sinatra::Base.call`) instead of silently shadowing it. See [Naming and collisions](docs/naming.md). (#17)
+
+### Documentation
+
+- A new guide, [Measuring token spend](docs/measuring-tokens.md), covers working out what each squished path costs
+  (with Coolhand Labs, the `squawk` hook, OpenTelemetry, LangSmith, or RubyLLM's instrumenter), and the README is
+  reworked around using Squishling to harden code paths as the economics justify it. (#21)
+
+### Changed
+
+- **Breaking:** the DSL method `instructions` is now `purpose`, and `append_instructions` is now `append_to_purpose`,
+  including the `instructions:`/`append_instructions:` keywords on `squishling`, `squish`, and `squish!`. There are no
+  aliases. Rename them in your classes: `instructions "..."` becomes `purpose "..."`, and
+  `append_instructions: [...]` becomes `append_to_purpose: [...]`. The error for a missing prompt now says
+  "has no purpose". (#17)
+- **Breaking:** `ruby_llm` must now be `~> 2.1` (was `~> 2.0`); run `bundle update ruby_llm`. (#16)
+- **Breaking:** every value a method with an `output_schema` returns is now validated and typed, not only Hashes. On
+  the Ruby path and in `squish_fallback`, a `nil`, String, Array, or other non-Hash return that used to pass through
+  now raises `InvalidOutputError`; return a Hash (or `result(...)`) that matches the schema. A result of the schema's
+  own class passes through, another schema's result is re-validated, and values JSON can't represent (`NaN`,
+  `Infinity`, cycles) raise `InvalidOutputError` instead of a raw JSON error. Inner calls (`super`, or the method
+  called from `squish_when` or a fallback) still only type Hashes, so overrides can reshape values. (#13)
+- **Behavior change:** a provider now travels with the model declared at the same level on a class too. A subclass
+  that declares its own `model:` or `escalation:` no longer inherits its parent's `provider:`, which could send the
+  whole payload to the parent's provider under another provider's model name. If you relied on that, set `provider:`
+  next to the subclass's model.
+- **Behavior change:** a `provider:` with no model beside it now raises `ConfigurationError` instead of being
+  silently ignored (the default provider was used). `squish ..., provider: ...` needs a `model:` or `escalation:` in
+  the same call, as `squish!` already did; `squishling provider: ...` needs one in the call or already declared on that
+  class. Move the `provider:` next to a model, or use `config.default_provider` with the default model.
+- The `result` alias for `squishling_result` is no longer added when the class already has a `result` (its own or
+  inherited); use `squishling_result` there. (#17)
+- `InvalidOutputError#message` and `config.logger` warnings no longer quote the model's response. For unparseable JSON
+  they report only the line and column; the raw output stays in `InvalidOutputError#raw`. (#15)
+
+### Fixed
+
+- A number a model writes outside the range JSON can represent (such as `1e400`, which parses to `Infinity`), or a
+  string with invalid UTF-8, was accepted as valid, while the same value from a Ruby method was rejected. It is now
+  invalid output on the LLM path too, and no longer raises a bare `JSON::GeneratorError` when `:squishsum` compares samples.
+
+### Security
+
+- Params can no longer replace the strict output format or tools through the containers that also hold ordinary
+  settings: `generationConfig`/`generation_config` (Google Gemini) and `completion_args` (Mistral Conversations) now
+  reject their format and tool keys, with symbol or string keys, and a non-Hash value for them is rejected.
+  `tool_config` is reserved at the top level. Settings such as `topK` still work. (#12)
+- The previous model's rejected output forwarded to the next escalation step is capped at 4,000 characters
+  (`Invoker::MAX_FORWARDED_CHARS`), with a truncation marker. (#14)
+- Model output is kept out of error messages and logs (see the `InvalidOutputError#message` change above); `squawk`
+  is the one sanctioned way for it to leave the process. (#15) A chat judge's `reason` is model-written, so
+  `DisagreementError#message` and the logs no longer include it; read it from `DisagreementError#reason`. A
+  `:judgment` judge's message still carries the choice and probability, and a choice other than `a`, `b`, or `neither`
+  is no longer repeated.
+- Only the first 20 validation errors (plus a count of the rest) are fed back to the model, logged, and put in
+  `InvalidOutputError#message` for one invalid output, so a very large malformed response can no longer produce a
+  megabyte-sized retry message or log line.
+
 ## [0.2.0] - 2026-10-09
 
 ### Added

data/README.md

@@ -1,46 +1,54 @@
-# Squishling
+<h1 align="center">
+  <img src="assets/squishling-logo.png" alt="Squishling" width="420">
+</h1>
 
 [![CI](https://github.com/Coolhand-Labs/squishling/actions/workflows/ci.yml/badge.svg)](https://github.com/Coolhand-Labs/squishling/actions/workflows/ci.yml)
+[![Gem Version](https://badge.fury.io/rb/squishling.svg)](https://badge.fury.io/rb/squishling)
 
-Elastic Ruby classes. A squished method either runs its Ruby implementation or sends its inputs through an LLM
-(via [RubyLLM](https://rubyllm.com)), and either way returns the same strict-schema-validated, typed result.
-Callers can't tell the difference.
+**Don't just use AI to write code. Use AI to not need code.** Write code only where it's worth maintaining.
 
-Inspired by [Elastic Software](https://everythingengineer.substack.com/p/beginners-write-software-with-ai):
-start flexible with AI, then harden high-volume paths into code as the economics justify it.
+A squished method either runs its Ruby implementation, with the LLM as a failover, or routes some callers
+through an LLM as a way of eliminating (or discovering) the cost of maintaining them as deterministic code.
+Either way it returns the same strict-schema-validated, typed result, so callers can't tell the difference.
+
+Think of code as a cache for AI judgment. Every line you write is something to maintain, so the LLM is the
+default and code has to earn its place: you build it for the inputs that carry the volume, and the LLM keeps
+handling the rest. New to the idea? Read
+[Elastic Software](https://everythingengineer.substack.com/p/beginners-write-software-with-ai).
 
 ## Use cases
 
-### Instant integration
+### Zero-code integrations
 
 Accept a new data source today, before anyone writes a parser. Declare what you want back and leave the
 method unimplemented: every call goes to the LLM, and its output is validated against your schema.
 
 ```ruby
-class PaymentWebhook
+class ShipmentUpdate
   include Squishling
 
-  instructions "Normalize this payment provider's webhook into our payment event."
+  purpose "Normalize this carrier's tracking webhook into our shipment update."
   output_schema do
-    string  :event, enum: %w[succeeded failed refunded disputed]
-    integer :amount_cents
-    string  :currency
-    string  :external_id
+    string :status, enum: %w[label_created in_transit out_for_delivery delivered exception]
+    string :tracking_number
+    string :location
   end
   # No `def call` yet, so every webhook goes to the LLM.
 end
 
-event = PaymentWebhook.call(provider: "adyen", payload: request.raw_post)
-event.event          # => "refunded"
-event.amount_cents   # => 4200
-event.squished?      # => true
+update = ShipmentUpdate.call(carrier: "acme-freight", payload: request.raw_post)
+update.status        # => "out_for_delivery"
+update.squished?     # => true
 ```
 
-When one provider carries the volume, write `def call` for it and add
-`squish_when { |provider:, **| provider != "stripe" }`. Stripe then runs in Ruby, everything else stays on the
+When one carrier carries the volume, write `def call` for it and add
+`squish_when { |carrier:, **| carrier != "ups" }`. UPS then runs in Ruby, everything else stays on the
 LLM, and callers don't change. See [Hardening a path](docs/routing.md#hardening-a-path).
 
-### Error recovery
+Validation guarantees the *shape* of the result, not that it's true. Where a wrong value is costly (money,
+identity), cross-check it against another source or keep that path in Ruby.
+
+### Rescue errors your code can't handle yet
 
 Keep the Ruby you have for the inputs it understands, and hand the rest to the LLM instead of failing. When the
 parser raises, `squish!` sends this call to the LLM with the error and the parser's own source as context.
@@ -49,8 +57,8 @@ parser raises, `squish!` sends this call to the LLM with the error and the parse
 class InvoiceParser
   include Squishling
 
-  instructions "Extract the invoice fields from the vendor's document."
-  append_instructions "The Ruby parser that handles well-formed invoices:", self   # this class's source
+  purpose "Extract the invoice fields from the vendor's document."
+  append_to_purpose "The Ruby parser that handles well-formed invoices:", self   # this class's source
   output_schema do
     string :invoice_number
     number :total
@@ -60,7 +68,7 @@ class InvoiceParser
     invoice = VendorFormats.fetch(vendor).parse(document)
     result(invoice_number: invoice.number, total: invoice.total)
   rescue VendorFormats::ParseError => e
-    squish!(append_instructions: "The parser failed on this document; the error is in the context.",
+    squish!(append_to_purpose: "The parser failed on this document; the error is in the context.",
             context: { parse_error: e })
   end
 end
@@ -73,13 +81,36 @@ Both calls return the same result class. If the LLM can't deliver either, `squis
 return, or the error is raised with the original `ParseError` as its cause. See
 [Handing off to the LLM](docs/routing.md#handing-off-to-the-llm-with-squish).
 
+### Measure your tech debt in tokens
+
+Code you haven't written is debt you haven't taken on. A squished path has a running cost you can read off a
+meter, so "should we write a parser for this?" becomes arithmetic: what the path costs in tokens each month,
+against what it costs to write and maintain the code. Write it when the first number is bigger.
+
+Add and initialize the [`coolhand`](https://github.com/Coolhand-Labs/coolhand-ruby) gem, and it picks up your
+squishling calls automatically and measures their accuracy and cost:
+
+```ruby
+# Gemfile
+gem "coolhand"
+
+# config/initializers/coolhand.rb
+Coolhand.configure do |config|
+  config.api_key = ENV.fetch("COOLHAND_API_KEY")
+end
+```
+
+`squished?` tells you which path served a call. Coolhand records your LLM requests and responses; see
+[what each tool sees](docs/measuring-tokens.md#privacy). Want to roll your own metrics? Check out
+[our guide](docs/measuring-tokens.md) for doing it with other tools.
+
 ## Installation
 
 ```ruby
 gem "squishling"
 ```
 
-Requires Ruby 3.3+ and RubyLLM 2.x. Configure your provider API keys in RubyLLM as usual, then optionally set a
+Requires Ruby 3.3+ and RubyLLM 2.1+. Configure your provider API keys in RubyLLM as usual, then optionally set a
 universal model, or an escalation of models to try in order:
 
 ```ruby
@@ -91,7 +122,7 @@ end
 
 ## How it works
 
-A squishling class needs **instructions** (the system prompt) and an **output schema** (the shape of the result,
+A squishling class needs a **purpose** (the system prompt) and an **output schema** (the shape of the result,
 validated on both paths). A squished call goes to the LLM when:
 
 - its `squish_when` predicate is truthy for these inputs,
@@ -104,13 +135,13 @@ Otherwise the Ruby runs, and whatever it returns is validated and typed like LLM
 
 - **One contract, two paths**: Ruby returns and LLM output are validated against the same strict schema and
   returned as the same typed `Data` objects. `squished?` tells you which path served a call.
-- **Your code as context**: `append_instructions` adds sections to the prompt, including a class's or method's
+- **Your code as context**: `append_to_purpose` adds sections to the prompt, including a class's or method's
   own Ruby source.
 - **Any RubyLLM provider and model**: OpenAI, Anthropic Claude, Google Gemini, AWS Bedrock, OpenRouter, and more.
   Set a universal, per-class, per-method, or per-call model, plus layered generation params (temperature,
   reasoning effort, top_p, …).
 - **Model escalation**: declare an `escalation:` instead of a `model:`, with per-step `attempts:`, `order:`, provider,
-  and params. Invalid output (with its errors fed back) or a provider failure moves on to the next attempt. Steps can
+  params, and `forward_rejected:`. Invalid output (with its errors fed back) or a provider failure moves on to the next attempt. Steps can
   cross providers, e.g. a local Qwen model on Ollama, then Anthropic Claude Haiku on AWS Bedrock, then Claude Opus:
 
   ```ruby
@@ -130,23 +161,36 @@ Otherwise the Ruby runs, and whatever it returns is validated and typed like LLM
       { model: "claude-haiku-4-5", provider: :bedrock },       # small hosted model
       { model: "claude-opus-5-5", provider: :anthropic }       # last resort
     ]
-    # instructions, output_schema, ...
+    # purpose, output_schema, ...
   end
   ```
+- **[Harnesses](docs/harnesses.md)**: choose how a call uses its escalation with `harness:`
+  - [`:escalation`](docs/configuration.md#models-and-escalation) (default): try each attempt until one passes
+  - [`:squishsum`](docs/harnesses.md#samples): two concurrent samples, accepted only if they agree
+  - [`:judged_squishsum`](docs/harnesses.md#the-judge): a chat or Jev judge picks between disagreeing samples
+  - [`:ensemble` and `:judged_ensemble`](docs/harnesses.md#ensembles): the same checks across two different models
 - **Output contracts**: beyond the strict schema, conditional rules (`given`) and Ruby checks (`squish_validate`)
   reject bad output and trigger the next attempt.
 - **Defined failure behavior**: provider errors become `Squishling::LLMError`, and `squish_fallback` lets you decide
   what to return when every attempt fails.
+- **Observability**: the opt-in [`squawk`](docs/failures.md#observing-every-attempt-squawk) hook sends every attempt's
+  raw output to your error tracker. Model output stays out of error messages and logs otherwise.
 - **Opt-in context**: only method arguments, the instance state you name with `squish_context`, the `context:` you
-  pass to `squish!`, and the source you choose to append are sent to the provider.
+  pass to `squish!`, and the source you choose to append are sent to the provider. The one addition: when
+  [escalation](docs/configuration.md#models-and-escalation) moves to the next step, that step also sees the previous
+  model's rejected output, unless the step sets `forward_rejected: false`.
 
 ## Documentation
 
 - [Configuration](docs/configuration.md): options, models and escalation, providers, generation params, inheritance
-- [Routing](docs/routing.md): when a call goes to the LLM, `squish!`, `append_instructions`, hardening a path,
+- [Routing](docs/routing.md): when a call goes to the LLM, `squish!`, `append_to_purpose`, hardening a path,
   what the LLM sees
 - [Output schemas](docs/schemas.md): schema forms, strict mode, typed results, optional vs. empty, contracts
 - [Failure handling](docs/failures.md): escalation, `squish_validate`, error classes, fallbacks
+- [Harnesses](docs/harnesses.md): escalation, squishsum, ensemble, and judges (chat or Jev)
+- [Naming and collisions](docs/naming.md): the methods `include Squishling` adds and what happens when a name is taken
+- [Measuring token spend](docs/measuring-tokens.md): see what each squished path costs, with Coolhand Labs,
+  the `squawk` hook, OpenTelemetry, LangSmith, or RubyLLM's instrumenter
 - [Live examples](examples/README.md): end-to-end tests against Anthropic Claude Haiku and OpenAI GPT-6 Luna
 
 ## Development
@@ -158,6 +202,17 @@ bundle exec rake        # RSpec (offline; RubyLLM is stubbed) + RuboCop
 
 See [AGENTS.md](AGENTS.md) for repo conventions, and [SECURITY.md](SECURITY.md) for reporting vulnerabilities.
 
+## Credits
+
+Inspired by [Elastic Software](https://everythingengineer.substack.com/p/beginners-write-software-with-ai).
+
 ## License
 
 Apache-2.0
+
+---
+
+<p align="center">
+  This open source project is supported by <a href="https://coolhandlabs.com/">Coolhand Labs</a>.<br><br>
+  <a href="https://coolhandlabs.com/"><img src="assets/coolhand-labs.png" alt="Coolhand Labs" width="220"></a>
+</p>

data/docs/configuration.md

@@ -17,6 +17,7 @@ Squishling.configure do |config|
   config.default_model = "claude-sonnet-5-5"   # or default_escalation, below
   config.default_provider = nil
   config.default_params = { temperature: 0 }
+  config.default_harness = :escalation
   config.logger = Rails.logger
 end
 ```
@@ -27,7 +28,9 @@ end
 | `default_escalation` | `nil` | The universal [escalation](#models-and-escalation): models tried in order. Assigning it clears `default_model`, and vice versa. |
 | `default_provider` | `nil` | The provider for `default_model`/`default_escalation` steps that don't name one. Only needed for models missing from RubyLLM's registry (see below). |
 | `default_params` | `{}` | Generation params for every call (temperature, reasoning effort, top_p, …), overridable per class, per method, and per escalation step. See [Generation params](#generation-params). |
-| `logger` | `nil` | Any `Logger`. Debug lines when a call routes to the LLM, warnings on each escalation and when a fallback is used. |
+| `squawk` | `nil` | A callable run after every LLM attempt with the raw output, for sending it to an error tracker or tracing tool. Also settable per class and per method. See [Observing every attempt](failures.md#observing-every-attempt-squawk). |
+| `default_harness` | `nil` (`:escalation`) | How every squishling class that doesn't declare its own uses its escalation: `:escalation`, `:squishsum`, `:judged_squishsum`, `:ensemble`, `:judged_ensemble`, or a Hash with `type:` and options. See [Harnesses](harnesses.md). |
+| `logger` | `nil` | Any `Logger`. Debug lines when a call routes to the LLM, warnings on each escalation and when a fallback is used. Never includes the raw model response; see [what ends up in errors and logs](failures.md#what-ends-up-in-errors-and-logs). |
 
 Transport-level retries (rate limits, 5xx, timeouts) are configured on RubyLLM itself
 (`RubyLLM.config.max_retries`, `request_timeout`).
@@ -61,6 +64,7 @@ Each step is a model name, or a Hash with:
 | `attempts:` | `1` | How many attempts this step gets before escalating to the next one. |
 | `order:` | position | An Integer; steps run lowest first. Give every step an `order:` or none (duplicates are rejected). Gaps are fine. |
 | `provider:` | the level's `provider:` | See [Providers](#providers-and-newly-released-models). |
+| `forward_rejected:` | `true` | Set `false` to start this step from the original input alone, without the previous step's rejected output and its errors (see below). |
 | `params:` | `{}` | [Generation params](#generation-params) for this step, merged over the class and method params key by key. |
 
 Without `order:`, steps run in the order written. With it, the order is explicit and doesn't depend on position:
@@ -105,10 +109,13 @@ end
 Passing both `model:` and `escalation:` in one declaration raises `ConfigurationError`, and so does a list passed as
 `model:`. Across separate declarations (a reopened class, or two config assignments), the latest one wins.
 
-Consecutive attempts of the same step (same model, provider, and params, e.g. `attempts: 2`) continue one
-conversation. Moving to a different step starts a fresh chat with the original input plus the previous output and why
-it was rejected, so no provider-specific history crosses providers. Configuration errors (bad credentials, an unknown model, a request the provider rejects) never move to the
-next attempt. See [Failure handling](failures.md).
+Consecutive attempts of the same step (same model, provider, params, and `forward_rejected:`, e.g. `attempts: 2`)
+continue one conversation. Moving to a different step starts a fresh chat with the original input plus the previous
+output and why it was rejected, so no provider-specific history crosses providers. That rejected output is
+model-generated and is sent to the next step's provider, even when it is a different one, truncated to 4,000
+characters. If you don't want a step to see it (say, because the next step is a different provider), set
+`forward_rejected: false` on that step and it starts from the original input only. Configuration errors (bad
+credentials, an unknown model, a request the provider rejects) never move to the next attempt. See [Failure handling](failures.md).
 
 ## Providers and newly released models
 
@@ -116,14 +123,21 @@ RubyLLM looks models up in its bundled registry. To use a model that isn't there
 OpenAI or Anthropic model, name its provider next to it. Squishling then tells RubyLLM to assume the model exists:
 
 ```ruby
-squishling model: "gpt-6-luna", provider: :openai
+squishling model: "gpt-7-preview", provider: :openai
 squish :triage, model: "claude-haiku-4-5", provider: :anthropic
-Squishling.configure { |c| c.default_model = "gpt-6-luna"; c.default_provider = :openai }
+Squishling.configure { |c| c.default_model = "gpt-7-preview"; c.default_provider = :openai }
 ```
 
 A provider is paired with the model or escalation declared at the same level, and applies to that escalation's steps
-that don't name their own. A per-method model never inherits a class-level provider meant for a different model. For models that are in the registry, a provider is optional
-and the normal registry lookup is kept.
+that don't name their own. A model never inherits a provider meant for a different model: a per-method model doesn't
+take the class's provider, and a subclass that declares its own `model:` or `escalation:` doesn't take its parent's
+(a subclass that only changes params or the purpose keeps the parent's model and provider together). For models that
+are in the registry, a provider is optional and the normal registry lookup is kept.
+
+A `provider:` with no model beside it would apply to nothing, so it raises `ConfigurationError`:
+`squish ..., provider: ...` needs a `model:` or `escalation:` in the same call (as `squish!` always has), and
+`squishling provider: ...` needs one in the call or already declared on that same class (not inherited). Set the
+provider next to the model, or, for the configured default model, use `config.default_provider`.
 
 ## Generation params
 
@@ -164,22 +178,32 @@ end
 
 - Keys that Squishling or RubyLLM control (`model`, `messages`, `input`, `instructions`, `system`, `stream`,
   `response_format`, `text`, `output_config`, `tools`, `tool_choice`, `schema`, and the camelCase and plural
-  spellings other providers use, such as `systemInstruction`, `toolConfig`, `outputConfig`, `inputs`, …) raise
-  `ConfigurationError`, because they would override the model, the conversation, or the strict output format.
+  spellings other providers use, such as `systemInstruction`, `toolConfig`/`tool_config`, `outputConfig`, `inputs`, …)
+  raise `ConfigurationError`, because they would override the model, the conversation, or the strict output format.
+- **Nested containers** such as Gemini's `generationConfig` (and `generation_config` for Gemini Interactions) and
+  Mistral Conversations' `completion_args` accept ordinary settings, but reject the keys inside them that carry the
+  output format or tools (`responseMimeType`, `responseSchema`, `responseJsonSchema`, `response_format`, `tools`,
+  `tool_choice`, `toolConfig`, and their snake_case spellings). Use `output_schema` instead:
+
+  ```ruby
+  squishling params: { generationConfig: { topK: 5 } }                      # ok
+  squishling params: { generationConfig: { responseMimeType: "text/plain" } } # ConfigurationError
+  ```
+
 - If a provider rejects a param, the call raises `Squishling::ConfigurationError` naming the params. It isn't
   retried or sent to `squish_fallback`. See [Failure handling](failures.md).
 
 ## Inheritance
 
-Subclasses inherit the model or escalation, provider, generation params (merged key by key), instructions, output schema,
+Subclasses inherit the model or escalation and its provider (as a pair), harness, generation params (merged key by key), purpose, output schema,
 `squish_when` predicate, `squish_context` names, `squish_validate`, `squish_fallback`, and every `squish` declaration. Overrides in a subclass, including
 overridden methods, are routed the same way.
 
-`append_instructions` sections are added to, not replaced: a subclass's sections follow its parent's, and
-`append_instructions false` drops the inherited ones. See [Appending to the instructions](routing.md#appending-to-the-instructions).
+`append_to_purpose` sections are added to, not replaced: a subclass's sections follow its parent's, and
+`append_to_purpose false` drops the inherited ones. See [Appending to the purpose](routing.md#appending-to-the-purpose).
 
 ## Per-call overrides
 
-Inside a squished method, `squish!` sends the call to the LLM with its own `instructions:`,
-`append_instructions:`, `context:`, `model:` or `escalation:` (with `provider:`), and `params:`. Each layers over the method and class
+Inside a squished method, `squish!` sends the call to the LLM with its own `purpose:`,
+`append_to_purpose:`, `context:`, `model:` or `escalation:` (with `provider:`), `params:`, and `harness:`. Each layers over the method and class
 settings the same way they layer over each other. See [Handing off to the LLM](routing.md#handing-off-to-the-llm-with-squish).

data/docs/failures.md

@@ -15,14 +15,73 @@ A failed attempt moves on to the next one; once the last fails, the error is rai
 | Malformed or truncated JSON | Moves to the next attempt, then raises `InvalidOutputError`. JSON wrapped in a markdown code fence is accepted. |
 | JSON that doesn't match the schema (wrong types, missing or extra keys, `null` in a non-`optional` field, a broken [conditional rule](schemas.md#contracts-beyond-the-schema), root not an object) | Moves to the next attempt with the validation errors, then raises `InvalidOutputError` |
 | Schema-valid output that [`squish_validate`](#output-checks-squish_validate) rejects | Moves to the next attempt with your messages, then raises `InvalidOutputError` |
+| With a [squishsum or ensemble harness](harnesses.md): two valid samples that differ, with no judge or a judge that rejects both | Raises `Squishling::DisagreementError` (an `InvalidOutputError`), carrying both typed `candidates` |
 
 Squishling validates output itself with [json_schemer](https://github.com/davishmcclurg/json_schemer), because
 RubyLLM doesn't, and some providers don't enforce strict mode. When the next attempt is on the same step (same
-model, provider, and params), the re-ask happens in the same conversation, so the model sees what it got wrong. A
-different step gets a fresh chat with the original input plus the rejected output and its errors, so that output is
-sent to the next step's provider, which may not be the one that produced it. `InvalidOutputError` exposes `errors`,
-`raw` (the last response), `attempts`, and `models` (the model tried on each attempt). With a `logger` configured,
-every escalation is logged as a warning.
+model, provider, params, and `forward_rejected:`), the re-ask happens in the same conversation, so the model sees
+what it got wrong. A different step gets a fresh chat with the original input plus the rejected output (truncated to
+4,000 characters) and its errors, so that output is sent to the next step's provider, which may not be the one that
+produced it. Set `forward_rejected: false` on a step to leave both out. `InvalidOutputError` exposes `errors`, `raw`
+(the last response), `attempts`, and `models` (the model tried on each attempt); for a
+[`DisagreementError`](harnesses.md#when-the-harness-fails), `raw` holds both candidates, `attempts` is `nil`, and
+`models` has one entry per role (sample a, sample b, then the judge). With a `logger` configured, every
+escalation is logged as a warning.
+
+### What ends up in errors and logs
+
+Model output can echo sensitive input, so the raw response is kept in one place: `InvalidOutputError#raw` (plus the opt-in `squawk` hook below).
+`InvalidOutputError#message`, `#errors`, and `config.logger` warnings are built from these:
+
+- For unparseable JSON, only the position (`response was not valid JSON (at line 1 column 7)`), never the parser's
+  snippet of the response.
+- JSON Schema errors, which name the schema path and the rule that failed. They never contain the rejected value, but
+  they do name an extra key the model added (`object property at /foo is a disallowed additional property`), and
+  that key name is chosen by the model.
+- Your own `squish_validate` messages, verbatim. If you interpolate result values into them, those values reach
+  the message, the logger, and the next attempt's prompt.
+- A fixed message when a response contains a value JSON can't represent (`1e400` parses to `Infinity`, or a string
+  with invalid UTF-8).
+- At most the first 20 of these errors, plus a count of the rest, so a very large malformed response can't flood
+  the retry message or the log.
+- For a [`DisagreementError`](harnesses.md#when-the-harness-fails), only what Squishling wrote: a `:judgment` judge's
+  choice and probability. A chat judge's `reason` is model-written, so it is available as `#reason` but never in the
+  message or logs.
+
+The same applies to anything you log yourself, such as `error.message` in a [fallback](#fallbacks). Use `error.raw`
+only where you are willing to store model output. To send the output of every attempt somewhere on purpose, use
+[`squawk`](#observing-every-attempt-squawk).
+
+## Observing every attempt (`squawk`)
+
+`squawk` is an opt-in hook for sending what the model returned to an error tracker or tracing tool. It runs after
+every LLM attempt, accepted or rejected, and does nothing unless you set it:
+
+```ruby
+Squishling.configure do |config|
+  config.squawk = lambda do |output:, metadata:, error:|
+    ObservabilitySolution.record(output, metadata, error) if error
+  end
+end
+```
+
+| Keyword | Value |
+|---|---|
+| `output` | The raw response: a Hash or String, or `nil` when the provider call itself failed |
+| `error` | `nil` for an accepted attempt. Otherwise the `InvalidOutputError` (with `errors` and `raw`) or `LLMError` that ended the attempt |
+| `metadata` | `label` (`"Class#method"`), `attempt`, `attempts` (the escalation's length; under a harness, the sample's or judge's own attempts), `final` (the last attempt), `model`, `provider`, `params`, `input` (the JSON sent: arguments and named context only), `usage` (token counts, when RubyLLM reports them) |
+
+- Set it globally with `config.squawk`, per class with `squishling squawk: ...`, or per method with
+  `squish :triage, squawk: ...`. The method's hook wins over the class's, which wins over the configured one;
+  `squawk: false` silences an inherited hook. Subclasses and `squish!` calls inherit it.
+- Any object that responds to `call` works. The hook receives only the keywords it declares, unless it takes `**`,
+  so a lambda that wants just `error:` is fine and new metadata fields won't break it.
+- It runs inline, so keep it quick. Exceptions it raises propagate unchanged and fail the call, like any of your own
+  code, so rescue inside the hook if an outage in your tracing tool shouldn't.
+- `output` is the same object the result is built from, so treat it as read-only.
+- Under a [harness](harnesses.md), it runs for every sample and judge attempt too, each with its own `input`.
+- It isn't called for a `ConfigurationError`, for deterministic and fallback returns, or for an attempt where your own
+  `squish_validate` raises (that exception propagates first).
 
 ## Output checks (`squish_validate`)
 
@@ -57,16 +116,17 @@ end
 | Error | Raised when |
 |---|---|
 | `Squishling::Error` | Base class for everything below. Raised directly for misuse at call time, e.g. `result` or `squish!` called outside a squished method |
-| `Squishling::ConfigurationError` | Missing instructions or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_instructions` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (400) |
+| `Squishling::ConfigurationError` | Missing purpose or schema, a non-strict schema, invalid or reserved params, an invalid `model:`/`escalation:` declaration, an invalid `append_to_purpose` item or unavailable source, bad credentials, an unknown model, a request the provider rejects (400) |
 | `Squishling::InvalidOutputError` | LLM output still invalid (schema or `squish_validate`) after every attempt in the escalation, or a deterministic/fallback return that doesn't match the schema |
+| `Squishling::DisagreementError` | A subclass of `InvalidOutputError`: a [squishsum or ensemble harness](harnesses.md#when-the-harness-fails)'s samples disagreed and no judge accepted either. `candidates`, `verdict`, and `reason` describe what happened |
 | `Squishling::LLMError` | The provider call failed on the last attempt in the escalation, after RubyLLM's own retries, including context-length errors (`cause` holds the original) |
 
 Errors raised by your own Ruby code are not wrapped.
 
 ## Fallbacks
 
-Use `squish_fallback` to decide what happens when every attempt fails, with `InvalidOutputError` or
-`LLMError`.
+Use `squish_fallback` to decide what happens when the LLM path fails with `InvalidOutputError` (including a
+`DisagreementError`) or `LLMError`. It's the one fallback for every [harness](harnesses.md).
 It receives the error plus the method's inputs as keywords, and runs against the instance:
 
 ```ruby