ruby_llm-contract 1.4.0 → 1.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b836138600866650ba9df05fce0a0ce02d148dc1819b9b215e5e47aa124e93cb
-  data.tar.gz: 237df37acef3aba9320acc723156dfdf33a928729c58b6e79ad54ea1ac33cf68
+  metadata.gz: 58ba70123cbc8bc4c7d3c9dc85ce430f1f1492ceb031df35ae0ea3f18f534dd8
+  data.tar.gz: 24fed341a96267536d66532926d0e19cb4eb59e2ffd95467be1ca861d9dbe4f9
 SHA512:
-  metadata.gz: 79ff71eea63a6ce371b0d183137d8b0c498b3dcb8f21fa2410cbd9793334120be140a15716080688569e96a0d7b543e2083e986ccb2c199153722a17ae0adffb
-  data.tar.gz: c0547247d8d84e2e4bfc03fc398ddee89d1b77330924487a68f606f363365df800dc2be0016a255227cf4bb004c2b3510d1bf3f5ee7f19a1d5d00a2896103a28
+  metadata.gz: 2cc04307951beb543adb0aae5876013767256e90501ed65be68cc9b4918e547568860bd377cf94651777faaa3159b5be47d3b054445223ee5d7f1a86a5aea697
+  data.tar.gz: 8c06562bc7093a78d50c3e42294875884e6537d326bf7a474dc69985f7a5798209cedbe2691b6acd83050857ef715d3ef963f5a08358bc3c79c2984e51a76014

data/CHANGELOG.md

@@ -1,5 +1,32 @@
 # Changelog
 
+## 1.4.1 (2026-10-11)
+
+### Fixed
+
+- **`compare_models` runs each candidate on a Step with a retry chain.** The
+  Step's `retry_policy` chain picked the model of every attempt, so without
+  `production_mode:` each candidate ran the chain's first model (and escalated
+  through the chain) under its own label: the scores and costs were all the
+  same model's, and `best_for` and `recommend` could name a model never run.
+  A candidate now runs without the chain, on its own model. This also holds
+  for the Steps of a Pipeline compared with `compare_models`. Without
+  `production_mode:`, a policy that only sets `attempts` picks no model, so
+  it stays: the candidate keeps its same-model retries, as before; and a
+  `retry_policy_override` passed in `context:` is still used as given. With
+  `production_mode:`, the candidate-and-fallback policy replaces both, as
+  before.
+- **An `evaluator:` that returns an `EvaluationResult` keeps its score.** The
+  result was taken as a truthy value, so every case passed with 1.0 - also one
+  the evaluator scored 0.0 with `passed: false`, as a judge built on another gem
+  does (the RubyLLM::Tribunal example in the docs). It is now used as returned.
+- **The `production_mode:` table draws no chain for a candidate run alone.**
+  Since 1.4.0 a candidate whose fallback would send the same request runs
+  single-shot, but the table still showed `candidate → fallback` with `0%`
+  when the two labels differed (`gpt-5-mini (effort: low)` with fallback
+  `gpt-5-mini`). It now shows the candidate alone with `—`, as for a fallback
+  equal to the candidate.
+
 ## 1.4.0 (2026-10-11)
 
 ### Added

data/README.md

@@ -15,12 +15,13 @@ gem "ruby_llm-contract"
 ```ruby
 RubyLLM.configure do |c|
   c.openai_api_key = ENV["OPENAI_API_KEY"]
-  c.default_model  = "gpt-4.1-mini"   # used when a Step has no explicit model
 end
 
 # Required: boots the gem so `Step.run` knows how to talk to your LLM.
-# Empty block is fine. Pass options here if you need them (e.g. `c.logger`).
-RubyLLM::Contract.configure { }
+# An empty block is fine; options such as `c.logger` go here too.
+RubyLLM::Contract.configure do |c|
+  c.default_model = "gpt-4.1-mini"   # used when a Step has no explicit model
+end
 ```
 
 Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc). Requires `ruby_llm ~> 2.0` and Ruby ≥ 3.2.
@@ -60,7 +61,7 @@ end
 Three things to notice in the snippet above:
 
 - **`{input}` is a gem-specific prompt placeholder** — at call time `SummarizeArticle.run(article_text)` interpolates `article_text` into that slot. For `input_type Hash`, each top-level key becomes its own placeholder (e.g. `{title}`, `{body}`). Plain text in, plain text out — no ERB, no `String#%`.
-- **`validate("...") { |o, _| ... }`** — each block receives `(parsed_output, context)`; the convention `|o, _|` reads "output, ignore context". Use `o[:key]` to assert on any field your schema declared. Return false → status `:validation_failed`, retry fires.
+- **`validate("...") { |o, _| ... }`** — each block receives `(parsed_output, input)`; the convention `|o, _|` reads "output, ignore input". Use `o[:key]` to assert on any field your schema declared. Return false → status `:validation_failed`, retry fires.
 - **`retry_policy do escalate(...) end`** — block form for mixed configs (strings + hashes with per-attempt options). For uniform model lists you can use the shorthand `retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]` (it's the same DSL — `models` is an alias for `escalate`).
 
 ```ruby
@@ -68,7 +69,7 @@ result = SummarizeArticle.run(article_text)
 result.status           # => :ok  (or :validation_failed if all steps fail)
 result.parsed_output    # => { tldr: "...", takeaways: [...], tone: "..." }
 result.trace[:model]    # => "gpt-4.1-mini"  (winning step)
-result.trace[:cost]     # => 0.000520        (total across all attempts)
+result.trace[:cost]     # => 0.000309        (total across all attempts)
 
 result.trace[:attempts]
 # => [
@@ -78,7 +79,7 @@ result.trace[:attempts]
 #        status: :validation_failed,
 #        usage: { input_tokens: 256, output_tokens: 84 },
 #        latency_ms: 45,
-#        cost: 0.000100
+#        cost: 0.000059
 #      },
 #      {
 #        attempt: 2,
@@ -86,7 +87,7 @@ result.trace[:attempts]
 #        status: :ok,
 #        usage: { input_tokens: 256, output_tokens: 92 },
 #        latency_ms: 92,
-#        cost: 0.000420
+#        cost: 0.000250
 #      }
 #    ]
 ```
@@ -114,7 +115,7 @@ end
 expect(SummarizeArticle).to pass_eval("regression").without_regressions
 ```
 
-A bad prompt edit or model swap that drops accuracy on the frozen dataset → red CI, blocked merge. The first CI run records a baseline; subsequent runs compare against it. Every production miss should become the next `add_case`. See [Prevent silent prompt regressions](docs/guide/eval_first.md) for the full flywheel.
+A bad prompt edit or model swap that drops accuracy on the frozen dataset → red CI, blocked merge. Save a baseline once (`report.save_baseline!`, or `t.save_baseline = true` in the rake task); until one exists, `without_regressions` has nothing to compare against and passes. Every production miss should become the next `add_case`. See [Prevent silent prompt regressions](docs/guide/eval_first.md) for the full flywheel.
 
 ## Do I need this?
 

data/docs/guide/eval_first.md

@@ -84,7 +84,8 @@ Compare two prompt versions on the same eval:
 diff = SummarizeArticleV2.compare_with(
   SummarizeArticleV1,
   eval: "regression",
-  model: "gpt-4.1-mini"
+  model: "gpt-4.1-mini",
+  context: { retry_policy_override: nil }   # run mini, not the step's retry chain
 )
 
 diff.safe_to_switch?  # => true if no cases regressed
@@ -103,7 +104,7 @@ end
 
 # Prompt iteration happens afterward
 diff = SummarizeArticleV2.compare_with(
-  SummarizeArticleV1, eval: "regression", model: "gpt-4.1-mini"
+  SummarizeArticleV1, eval: "regression", model: "gpt-4.1-mini", context: { retry_policy_override: nil }
 )
 ```
 
@@ -124,7 +125,7 @@ Good for: offline smoke tests, local development, testing evaluator wiring, chec
 
 Not enough for real prompt decisions. For those:
 
-- `run_eval(..., context: { model: "..." })` with a real model, or pass an explicit adapter via `context: { adapter: ... }`. An *adapter* here is the layer that actually executes the LLM call — `RubyLLM::Contract::Adapters::Test` returns canned data, the default RubyLlm adapter hits the real provider. `model:` chooses the model name; `adapter:` chooses whether the call goes live or stays canned.
+- `run_eval(..., context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new, model: "..." })` with a real model. An *adapter* here is the layer that actually executes the LLM call — `RubyLLM::Contract::Adapters::Test` returns canned data, the RubyLLM adapter hits the real provider. `model:` chooses the model name; `adapter:` chooses whether the call goes live or stays canned: an eval with `sample_response` stays canned until you pass an adapter, whatever `model:` says. On a step with a `retry_policy` chain, the chain picks each attempt's model, so add `retry_policy_override: nil` to measure the one you name.
 - `compare_with(...)` for prompt A/B.
 
 `compare_with` intentionally ignores `sample_response` — canned data would make both sides look the same.
@@ -135,11 +136,11 @@ For larger datasets, `run_eval` accepts a `concurrency:` argument — cases run
 
 ```ruby
 report = SummarizeArticle.run_eval("regression",
-  context: { model: "gpt-4.1-mini" },
+  context: { model: "gpt-4.1-mini", retry_policy_override: nil },
   concurrency: 8)
 ```
 
-Same accepted by `compare_models` and `optimize_retry_policy`. Thread count is a ceiling — dataset order of results is preserved. Keep it low enough to respect the provider's rate limits.
+`compare_models` and `optimize_retry_policy` do not take `concurrency:`; they run cases one at a time. Thread count is a ceiling — dataset order of results is preserved. Keep it low enough to respect the provider's rate limits.
 
 ## Budgeting an eval before you run it
 
@@ -148,7 +149,7 @@ Same accepted by `compare_models` and `optimize_retry_policy`. Thread count is a
 ```ruby
 SummarizeArticle.estimate_eval_cost("regression",
   models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
-# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }
+# => { "gpt-4.1-nano" => 0.003207, "gpt-4.1-mini" => 0.012829, "gpt-4.1" => 0.064142 }
 ```
 
 Use it in CI to decide which models are worth running regression on, or to cap worst-case spend per build.

data/docs/guide/getting_started.md

@@ -37,11 +37,11 @@ class SummarizeArticle < RubyLLM::Contract::Step::Base
   # 5. Refuse before calling the LLM if input is too large or estimated cost exceeds the cap
   max_input  2_000
   max_output 4_000
-  max_cost   0.01
+  max_cost   0.05
 end
 ```
 
-Each `validate` block receives `(parsed_output, context)` — the convention `|o, _|` reads "output, ignore context". Use `o[:key]` to assert on any field your `output_schema` declared. The block returns truthy/falsy: `false` → status `:validation_failed`, retry fires if `retry_policy` was set.
+Each `validate` block receives `(parsed_output, input)` — the convention `|o, _|` reads "output, ignore input". Use `o[:key]` to assert on any field your `output_schema` declared. The block returns truthy/falsy: `false` → status `:validation_failed`, retry fires if `retry_policy` was set.
 
 ## Validation and retry behavior
 
@@ -53,13 +53,13 @@ result = SummarizeArticle.run(article_text)
 result.status           # => :ok
 result.parsed_output    # => { tldr: "...", takeaways: [...], tone: "analytical" }
 result.trace[:model]    # => "gpt-4.1-mini"  (first model that passed)
-result.trace[:cost]     # => 0.00052  (sum of all attempts)
+result.trace[:cost]     # => 0.000309  (sum of all attempts)
 result.trace[:attempts]
 # => [
 #   { attempt: 1, model: "gpt-4.1-nano", status: :validation_failed,
-#     cost: 0.00010, latency_ms: 45, ... },
+#     cost: 0.000059, latency_ms: 45, ... },
 #   { attempt: 2, model: "gpt-4.1-mini", status: :ok,
-#     cost: 0.00042, latency_ms: 92, ... }
+#     cost: 0.000250, latency_ms: 92, ... }
 # ]
 ```
 
@@ -104,6 +104,8 @@ retry_policy models: [
 
 `reasoning_effort` is a gpt-5 family feature (gpt-4.1 is not a reasoning model). The per-attempt value is forwarded via `with_thinking` (provider-agnostic — OpenAI `reasoning_effort` and Anthropic extended-thinking budget both supported). Passing it alongside a non-reasoning model is forwarded unchanged to the provider, which will either ignore it or reject the request — the gem does not guard against this. An element without `reasoning_effort` inherits the call's or the step's; `{ model: "gpt-4.1-mini", reasoning_effort: :default }` sends none (see [thinking DSL note](optimizing_retry_policy.md#thinking-dsl-note)).
 
+On a step with a `retry_policy` chain, the chain picks the model of every attempt, so a `model:` in the context (of `run`, `run_eval`, `compare_with` or `pass_eval(...).with_context`) is not used. To measure or run one model, add `retry_policy_override: nil` to the context. `compare_models` does this for each candidate.
+
 ## Evals and CI gates
 
 An eval is a named scenario you can run to verify the step still works. `sample_response` makes it offline — zero API calls — so CI can run it on every merge without burning budget.
@@ -178,7 +180,7 @@ expect(SummarizeArticle).to pass_eval("regression").without_regressions
 `max_input`, `max_output`, and `max_cost` are preflight checks — the LLM is never called if an estimate exceeds the limit. Zero tokens spent, zero cost.
 
 ```ruby
-result = SummarizeArticle.run(giant_10mb_document)
+result = SummarizeArticle.run(forty_page_report)   # ~128k characters
 result.status            # => :limit_exceeded
 result.validation_errors
 # => ["Input token limit exceeded: estimated 32000 tokens (heuristic ±30%), max 2000"]
@@ -251,13 +253,13 @@ SummarizeArticle.estimate_cost(input: article_text)
 # => {
 #      model: "gpt-4.1-mini",
 #      input_tokens: 812, output_tokens_estimate: 4000,
-#      estimated_cost: 0.00243
+#      estimated_cost: 0.00672
 #    }
 
 # Estimate what a full eval would cost across candidate models
 SummarizeArticle.estimate_eval_cost("regression",
   models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
-# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }
+# => { "gpt-4.1-nano" => 0.003207, "gpt-4.1-mini" => 0.012829, "gpt-4.1" => 0.064142 }
 ```
 
 `estimate_cost` returns `nil` when pricing isn't registered. `estimate_eval_cost` silently treats unknown-pricing cases as `$0.00` and sums the rest — it does **not** fail closed the way `max_cost` does. Treat its output as a floor, not a guarantee; register pricing via `CostCalculator.register_model` before relying on it for budget decisions.

data/docs/guide/llm_judge.md

@@ -34,7 +34,7 @@ The judge is just another Step. The benefit of using `ruby_llm-contract`: you ge
 
 The judge takes **two** inputs in a single Hash: the SOURCE you want the answer to be faithful to (here: the article), and the ANSWER under review (here: the summary). Both are required — without the source, the judge has nothing to verify against.
 
-The `prompt do` DSL uses a small template syntax: `{key}` placeholders are interpolated from the input. With `input_type Hash`, every top-level key of the hash becomes available as `{key}` — here `{article}` and `{summary}` map to `input[:article]` and `input[:summary]` when you later call `AccuracyJudge.run(article: ..., summary: ...)`. With `input_type String`, you use the special `{input}` placeholder (one input, one slot). It is the gem's own syntax — not ERB, not `String#%`, not string interpolation — and it is what makes the judge inputs visible inside the prompt.
+The `prompt do` DSL uses a small template syntax: `{key}` placeholders are interpolated from the input. With `input_type Hash`, every top-level key of the hash becomes available as `{key}` — here `{article}` and `{summary}` map to `input[:article]` and `input[:summary]` when you later call `AccuracyJudge.run({ article: ..., summary: ... })` (one Hash argument, so the braces are needed). With `input_type String`, you use the special `{input}` placeholder (one input, one slot). It is the gem's own syntax — not ERB, not `String#%`, not string interpolation — and it is what makes the judge inputs visible inside the prompt.
 
 ```ruby
 # A cheap model reads the summary against its source and decides if the
@@ -66,11 +66,12 @@ class AccuracyJudge < RubyLLM::Contract::Step::Base
   end
 
   retry_policy models: %w[gpt-5-nano gpt-5-mini]
-  max_cost 0.001
+  max_output 2_000   # gpt-5 reasoning tokens count as output
+  max_cost   0.01
 end
 ```
 
-The `retry_policy` escalates to `gpt-5-mini` if the cheaper `gpt-5-nano` fails schema/validate (e.g., returns malformed JSON) — the judge stays reliable without paying for the bigger model on every call. `max_cost 0.001` bounds each call: the judge runs N× per eval suite (one call per case), so capping per-call cost prevents calibration runs from blowing the budget.
+The `retry_policy` escalates to `gpt-5-mini` if the cheaper `gpt-5-nano` fails schema/validate (e.g., returns malformed JSON) — the judge stays reliable without paying for the bigger model on every call. `max_cost 0.01` bounds each attempt (with `max_output 2_000`, up to roughly 24k input tokens on `gpt-5-mini`), so a case that falls back from nano to mini costs at most about $0.012: the judge runs N× per eval suite (one call per case), so capping per-call cost prevents calibration runs from blowing the budget.
 
 This is small on purpose: one boolean, one reason. Easy to read, easy to disagree with.
 
@@ -148,7 +149,7 @@ Only now, after the judge has earned trust on real data, do you use it as the or
 
 The lambda must return a `Float` in `[0.0, 1.0]` — the eval framework averages per-case scores into the suite score. Map `true → 1.0`, `false → 0.0`, or use weighted partial scores if your judge returns gradations.
 
-> ⚠️ **Heads-up before you copy the snippet below.** Smoke evals on `SummarizeArticle` (per [Getting Started](getting_started.md)) typically run with `RubyLLM::Contract::Adapters::Test` for determinism — canned summaries, no API calls. **The `AccuracyJudge.run(...)` inside the evaluator does NOT inherit that Test adapter.** Each Step picks its adapter independently (from `RubyLLM::Contract.configure`, `context: { adapter: }`, or `LIVE=1` env). Run accuracy-regression evals as live (`LIVE=1` or explicit `context:`), and rely on `max_cost` on the judge to bound the bill.
+> ⚠️ **Heads-up before you copy the snippet below.** Smoke evals on `SummarizeArticle` (per [Getting Started](getting_started.md)) typically run with `RubyLLM::Contract::Adapters::Test` for determinism — canned summaries, no API calls. **The `AccuracyJudge.run(...)` inside the evaluator does NOT inherit that Test adapter.** Each Step picks its adapter independently (from `context: { adapter: }`, or the `default_adapter` set in `RubyLLM::Contract.configure`). Run accuracy-regression evals live (an explicit `context:` or a configured default adapter), and rely on `max_cost` on the judge to bound the bill.
 >
 > This is intentional: the judge IS the oracle, so faking its verdicts would defeat the purpose. Keep the Test-adapter smoke evals for schema/wiring checks only.
 
@@ -161,7 +162,7 @@ SummarizeArticle.define_eval("accuracy_regression") do
                summary: output[:tldr],
                article: input
              })
-             verdict.parsed_output[:accurate] ? 1.0 : 0.0
+             verdict.ok? && verdict.parsed_output[:accurate] ? 1.0 : 0.0
            }
 
   # ... one case per real production article you want to defend

data/docs/guide/migration.md

@@ -153,6 +153,12 @@ The gem handles single calls. You handle parallelism:
 
 ```ruby
 class SummarizeBatch < RubyLLM::Contract::Step::Base
+  prompt <<~PROMPT
+    Give a one-line TL;DR for each article below, keyed by its id.
+
+    {input}
+  PROMPT
+
   output_schema do
     array :summaries do
       object do

data/docs/guide/multimodal_input.md

@@ -9,7 +9,7 @@
 ```ruby
 # app/contracts/extract_invoice_data.rb
 class ExtractInvoiceData < RubyLLM::Contract::Step::Base
-  prompt "Extract invoice fields from the attached PDF. Return JSON."
+  prompt "Extract invoice fields from the attached PDF. Return JSON. {input}"
 
   output_schema do
     string :vendor
@@ -24,7 +24,9 @@ class ExtractInvoiceData < RubyLLM::Contract::Step::Base
   # Conservative estimate of attachment input tokens (provider/model-specific).
   attachment_token_estimate 15_000   # ~12 PDF pages at ~1250 tokens/page
 
-  max_cost 0.10
+  max_output 4_000   # room for gpt-5's reasoning tokens, which count as output;
+                     # without it the cost check assumes an output as long as the input
+  max_cost   0.10
 
   retry_policy do
     escalate "gpt-4.1-mini",
@@ -64,7 +66,7 @@ end
 The same estimate applies to:
 
 - **Runtime** (`limit_checker`) — adds the estimate to `input_tokens` before checking `max_cost`/`max_input`. Refuses pre-flight if budget exceeded.
-- **Pre-flight** (`estimate_cost`) — accepts `attachment:` kwarg; same accounting. No drift between estimate and runtime decisions.
+- **Pre-flight** (`estimate_cost`) — accepts `attachment:` kwarg; same input accounting. The output side matches only when `max_output` is set: without it `estimate_cost` assumes 256 output tokens, while the runtime check assumes an output as long as the input.
 
 ### Fail-closed without estimate
 
@@ -78,7 +80,7 @@ end
 
 result = MyContract.run("text", context: { attachment: "doc.pdf" })
 result.status # => :limit_exceeded
-result.validation_errors # => ["attachment present but attachment_token_estimate not set; ..."]
+result.validation_errors # => ["attachment present but attachment_token_estimate not declared; cost cannot be bounded. ..."]
 ```
 
 ### Opting out per-step
@@ -104,12 +106,12 @@ ExtractInvoiceData.estimate_cost(
   attachment: "tmp/invoice.pdf"
 )
 # => { model: "gpt-4.1-mini",
-#      input_tokens: 15_320,
-#      output_tokens_estimate: 256,
-#      estimated_cost: 0.0123 }
+#      input_tokens: 15_022,
+#      output_tokens_estimate: 4000,
+#      estimated_cost: 0.012409 }
 ```
 
-The `input_tokens` figure includes both the text estimate (chars/4 heuristic) AND the declared `attachment_token_estimate`. Pre-flight refusal mirrors runtime: if `attachment` is passed and `attachment_token_estimate` is not declared, `estimate_cost` returns nil and emits the same fail-closed reason.
+The `input_tokens` figure includes both the text estimate (chars/4 heuristic) AND the declared `attachment_token_estimate`. Pre-flight refusal mirrors runtime: if `attachment` is passed and `attachment_token_estimate` is not declared, `estimate_cost` returns nil (with `on_unknown_attachment_size :warn` it warns and estimates without the attachment instead).
 
 **Note on output tokens.** `attachment_token_estimate` adds to `input_tokens` only — not to `output_tokens_estimate`. Vision-heavy responses (long image descriptions, transcribed paragraphs) may exceed the conservative `output_tokens_estimate` default. Treat `estimated_cost` as a floor for budget planning, not a precise predictor; inflate `max_output` or `max_cost` accordingly if your prompt routinely produces verbose descriptions.
 
@@ -130,7 +132,7 @@ Pick a value at or above the provider's worst-case. The estimate is a **floor fo
 
 ## Multi-turn caveat
 
-If your contract uses history (`add_history`), attachments from prior turns are **not** replayed (still true in 1.0.x). Single-turn multimodal works; follow-up questions on the same document require additional work that is deferred to a later release. The rationale is recorded in the project's internal ADR-0022, which is not shipped with the gem.
+A Step is single-turn: each `run` sends its prompt with the attachment and keeps no conversation history. A follow-up question on the same document is another `run` that passes the attachment again.
 
 ## Provider notes
 
@@ -149,8 +151,9 @@ RSpec.describe ExtractInvoiceData do
   it "forwards attachment to chat.ask" do
     expect_any_instance_of(RubyLLM::Chat).to receive(:ask)
       .with(anything, with: "fixtures/invoice.pdf")
-      .and_return(double(content: '{"vendor":"X",...}',
-                         tokens: RubyLLM::Tokens.new(input: 200, output: 50)))
+      .and_return(RubyLLM::Message.new(role: :assistant, content: '{"vendor":"X","invoice_number":"1","total_amount":10,"currency":"EUR"}',
+                                       model_id: "gpt-4.1-mini",
+                                       input_tokens: 200, output_tokens: 50))
 
     result = described_class.run("extract", context: { attachment: "fixtures/invoice.pdf" })
     expect(result.status).to eq(:ok)

data/docs/guide/optimizing_retry_policy.md

@@ -2,13 +2,13 @@
 
 > Read this when you have 2+ evals and want to know empirically — not by guessing — which models belong in `retry_policy` and in what order.
 
-You defined `SummarizeArticle` in the [README](../../README.md) with `retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]`. That list was a guess. `optimize_retry_policy` tells you which models your evals actually need, so you stop paying for the strong model when `nano` was enough — or stop shipping `nano` when the hardest eval proves it isn't.
+You defined `SummarizeArticle` in [Getting Started](getting_started.md) with `retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]`. That list was a guess. `optimize_retry_policy` tells you which models your evals actually need, so you stop paying for the strong model when `nano` was enough — or stop shipping `nano` when the hardest eval proves it isn't.
 
 ## Requirements
 
 - **`SummarizeArticle` already has `retry_policy`.** If your step has none, add one first ([getting started](getting_started.md)).
 - **2–3 evals per step.** One eval optimizes for one scenario; with only `smoke`, you get a recommendation that passes smoke but may miss production edge cases. See [eval-first](eval_first.md).
-- **Rake task `ruby_llm_contract:optimize` is registered when `lib/ruby_llm/contract/rake_task.rb` is loaded.** In Rails apps on **0.10.4+** the railtie auto-loads it lazily during `rake` invocation — no manual setup needed. On older Rails versions or in non-Rails projects, add `require "ruby_llm/contract/rake_task"` to your `Rakefile` (or any file under `lib/tasks/`) and set `EVAL_DIRS=...` on the command line.
+- **Rake task `ruby_llm_contract:optimize` is registered when `lib/ruby_llm/contract/rake_task.rb` is loaded.** In Rails apps the railtie auto-loads it lazily during `rake` invocation — no manual setup needed. In non-Rails projects, add `require "ruby_llm/contract/rake_task"` to your `Rakefile` (or any file under `lib/tasks/`) and set `EVAL_DIRS=...` on the command line.
 
 > **Two orthogonal dimensions to a retry chain.** A chain element is `{ model:, reasoning_effort: }` — model identity AND thinking budget. `optimize_retry_policy` explores both. You can pin the default thinking config at class level too — see the `thinking` DSL note at the end.
 
@@ -27,10 +27,12 @@ Run once offline to verify the wiring:
 ```bash
 rake ruby_llm_contract:optimize \
   STEP=SummarizeArticle \
-  CANDIDATES=gpt-4.1-nano,gpt-4.1-mini@low,gpt-4.1-mini,gpt-4.1
+  CANDIDATES=gpt-5-nano,gpt-5-mini@low,gpt-5-mini,gpt-5
 ```
 
-The `@low` (or `@medium`, `@high`) suffix on a candidate is CLI shorthand for `{ model: "gpt-4.1-mini", reasoning_effort: :low }` — saves you constructing hash strings on the command line. Plain model names without `@` use the Step's default thinking config (or none). `@default` runs the candidate with no effort at all, which is what a model without reasoning needs when the Step sets one: `CANDIDATES=gpt-4.1-mini@default,gpt-6-luna@none`.
+The `@low` (or `@medium`, `@high`) suffix on a candidate is CLI shorthand for `{ model: "gpt-5-mini", reasoning_effort: "low" }` — saves you constructing hash strings on the command line. Plain model names without `@` use the Step's default thinking config (or none). `@default` runs the candidate with no effort at all, which is what a model without reasoning needs when the Step sets one: `CANDIDATES=gpt-6-luna@none,gpt-4.1-mini@default`.
+
+List candidates from the cheapest to the most expensive. The optimizer takes that order as the cost order: the suggested chain tries them in it, and the first candidate that passes every eval is the safe fallback. It does not sort by price.
 
 Offline uses each eval's `sample_response` — zero API calls. **Every candidate gets the same score** because they all receive the canned response. That's fine for a smoke test (verifying evals load, candidates parse, output renders) but it doesn't compare model quality. For real optimization, go live.
 
@@ -39,7 +41,7 @@ Offline uses each eval's `sample_response` — zero API calls. **Every candidate
 ```bash
 LIVE=1 RUNS=3 rake ruby_llm_contract:optimize \
   STEP=SummarizeArticle \
-  CANDIDATES=gpt-4.1-nano,gpt-4.1-mini@low,gpt-4.1-mini,gpt-4.1
+  CANDIDATES=gpt-5-nano,gpt-5-mini@low,gpt-5-mini,gpt-5
 ```
 
 `LIVE=1` makes real API calls. `RUNS=3` averages each `(candidate, eval)` pair over three runs — necessary because OpenAI forces `temperature=1.0` on gpt-5 / o-series and the same pair can score `0.00` on one run and `1.00` on the next.
@@ -49,28 +51,35 @@ Output (illustrative):
 ```
 SummarizeArticle — fallback list optimization
 
-  eval             4.1-nano  4.1-mini@low  4.1-mini   4.1
-  ---------------------------------------------------------
-  smoke                1.00          1.00      1.00  1.00
-  dense_article        0.67 ←        1.00      1.00  1.00
-  critical_tone        0.50 ←        0.67 ←    1.00  1.00
+  eval               nano  mini@low      mini     gpt-5
+  -----------------------------------------------------
+  smoke            1.00      1.00      1.00      1.00
+  dense_article    0.67      1.00      1.00      1.00
+  critical_tone    0.50 ←    0.67 ←    1.00      1.00
 
   Hardest eval: critical_tone
 
   Suggested fallback list:
-    gpt-4.1-nano         — covers 1 eval(s)
-    gpt-4.1-mini         — passes all 3 evals
+    gpt-5-nano — covers 1 eval(s)
+    gpt-5-mini (effort: low) — covers 1 eval(s)
+    gpt-5-mini — passes all 3 evals
 
   DSL:
-    retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini]
+    retry_policy do
+      escalate(
+        { model: "gpt-5-nano" },
+        { model: "gpt-5-mini", reasoning_effort: "low" },
+        { model: "gpt-5-mini" }
+      )
+    end
 ```
 
 Reading the table:
 - **`←` marks scores below threshold in the hardest eval.** Not a selection hint — just "this candidate fails the row that matters most".
-- **Hardest eval** = the one that forces the strong fallback. Here, `critical_tone` demands `gpt-4.1-mini`.
-- **Suggested fallback list** = the shortest chain where each step covers more evals, built greedy-cheapest-first. Stops when all evals pass. Order matters: `gpt-4.1-nano` is tried first; on validation failure, the gem falls back to `gpt-4.1-mini`.
+- **Hardest eval** = the one that forces the strong fallback. Here, `critical_tone` demands `gpt-5-mini` at its default effort.
+- **Suggested fallback list** = the shortest chain where each step covers more evals, built greedy-cheapest-first. Stops when all evals pass. Order matters: `gpt-5-nano` is tried first; on validation failure, the gem falls back to `gpt-5-mini` at low effort, then at its default.
 
-Copy the DSL, paste into your step, verify with `rake ruby_llm_contract:eval`. You just dropped `gpt-4.1` from the chain — most requests finish on nano, mini handles what nano misses, and the strong model was never needed.
+Copy the DSL, paste into your step, verify with `rake ruby_llm_contract:eval` (defined by `RubyLLM::Contract::RakeTask.new`, see [Rails integration](rails_integration.md#7-ci-gate--block-regressions-before-merge)). You just dropped `gpt-5` from the chain — most requests finish on nano, mini handles what nano misses, and the strong model was never needed.
 
 ## Measure effective cost before shipping
 
@@ -81,8 +90,8 @@ Copy the DSL, paste into your step, verify with `rake ruby_llm_contract:eval`. Y
 ```ruby
 SummarizeArticle.compare_models(
   "dense_article",
-  candidates: [{ model: "gpt-5-nano" }, { model: "gpt-5-mini", reasoning_effort: "low" }],
-  production_mode: { fallback: "gpt-5-mini" }
+  candidates: [{ model: "gpt-5-nano" }, { model: "gpt-5-mini", reasoning_effort: "low" }, { model: "gpt-5" }],
+  production_mode: { fallback: "gpt-5" }
 ).print_summary
 ```
 
@@ -91,17 +100,20 @@ Output (live mode, illustrative):
 ```
 dense_article — model comparison
 
-  Chain                                    first-attempt  fallback %  effective cost  latency   score
-  ---------------------------------------------------------------------------------------------------
-  gpt-5-nano → gpt-5-mini                  $0.0010        33%         $0.0018         164ms     1.00
-  gpt-5-mini (effort: low) → gpt-5-mini    $0.0015         5%         $0.0016         210ms     1.00
-  gpt-5-mini                               $0.0030         —          $0.0030         220ms     1.00
+  Chain                             first-attempt  fallback %  effective cost  latency    score
+  ----------------------------------------------------------------------------------------------
+  gpt-5-nano → gpt-5                $0.0010        33%         $0.0030         164ms        1.00
+  gpt-5-mini (effort: low) → gpt-5  $0.0015        5%          $0.0018         210ms        1.00
+  gpt-5                             $0.0060        —           $0.0060         450ms        1.00
+
+  Best overall: gpt-5-mini (effort: low)
+  Cheapest at 100%: gpt-5-mini (effort: low)
 ```
 
 - **first-attempt** — cost of the first run alone.
 - **fallback %** — fraction of cases where the validator rejected and the fallback ran.
 - **effective cost** — total per successful output including retries.
-- **`—`** — candidate equals fallback, no chain to observe.
+- **`—`** — the fallback would send the candidate's own request, so the candidate runs alone and there is no chain to observe.
 
 Run this before finalizing: a candidate saving 3× on first-attempt but falling back 60% of the time may save only 1.2× in production.
 
@@ -135,10 +147,10 @@ Forwarded to `Chat#with_thinking(**)` through the adapter — works provider-agn
 `reasoning_effort: :default` - the same word as in the DSL - sends no effort for one call, candidate or chain element, and keeps a `thinking budget:`. Use it to run a Step set up for a reasoning model on one that takes no effort, which a provider otherwise rejects (OpenAI: "Unsupported parameter: 'reasoning.effort'"):
 
 ```ruby
-SummarizeArticle.run(text, context: { model: "gpt-4.1-mini", reasoning_effort: :default })
-SummarizeArticle.compare_models("regression", candidates: [{ model: "gpt-4.1-mini", reasoning_effort: :default },
-                                                           { model: "gpt-6-luna" }])
-retry_policy { escalate({ model: "gpt-4.1-mini", reasoning_effort: :default }, { model: "gpt-6-luna" }) }
+SummarizeArticle.run(text, context: { model: "gpt-4.1-mini", reasoning_effort: :default, retry_policy_override: nil })
+SummarizeArticle.compare_models("regression", candidates: [{ model: "gpt-6-luna" },
+                                                           { model: "gpt-4.1-mini", reasoning_effort: :default }])
+retry_policy { escalate({ model: "gpt-6-luna" }, { model: "gpt-4.1-mini", reasoning_effort: :default }) }
 ```
 
 `nil` is not `:default`: a nil effort is ignored and the class one is sent. The gem does not drop an effort by itself based on the model. A chain element without an effort runs with the call's (or the class's), and `trace.attempts[n][:config]` shows the effort each attempt was sent.

data/docs/guide/prompt_ast.md

@@ -29,7 +29,7 @@ The AST is immutable, diffable, and hashable. Useful for snapshot testing and au
 
 When input is a Hash, each key becomes a template variable. Concrete scenario: a multi-language newsletter product where the same article has to be summarised in Polish for EU subscribers, English for US, with different audiences per tier (Rails developers vs engineering managers). Hash inputs let one step cover all of these without forking the class.
 
-The `Types::Hash.schema(...)` form below declares a **typed contract on the input** — `run()` raises `TypeError` if a required key is missing or its value violates the declared type. Use it when the input shape is part of the Step's interface (multi-key inputs, multi-tenant variants); plain `input_type Hash` accepts any hash and is fine for prototypes or when the schema is enforced upstream.
+The `Types::Hash.schema(...)` form below declares a **typed contract on the input** — `run()` returns `status: :input_error` (with the reason in `validation_errors`, e.g. `":article is missing in Hash input"`) if a required key is missing or its value violates the declared type, and no model is called. Use it when the input shape is part of the Step's interface (multi-key inputs, multi-tenant variants); plain `input_type Hash` accepts any hash and is fine for prototypes or when the schema is enforced upstream.
 
 ```ruby
 class SummarizeArticle < RubyLLM::Contract::Step::Base
@@ -54,9 +54,9 @@ class SummarizeArticle < RubyLLM::Contract::Step::Base
 end
 ```
 
-Every `{key}` in a prompt node is pulled from the input hash at run time. Missing keys raise — making wire-up bugs loud, not silent.
+Every `{key}` in a prompt node is pulled from the input hash at run time. A key the input does not have is left in the prompt as literal `{key}` text, so declare the keys in a `Types::Hash.schema(...)` input type when a missing one should stop the call.
 
-> **Not the same as `RubyLLM::Agent.inputs`.** `Step.input_type` is a *runtime type check* on the positional argument passed to `run(input)` — it raises `TypeError` if the input violates the declared shape. `RubyLLM::Agent.inputs` is a list of *named template locals* injected into ERB instructions. They solve different problems (validation vs templating) and can coexist on the same project.
+> **Not the same as `RubyLLM::Agent.inputs`.** `Step.input_type` is a *runtime type check* on the positional argument passed to `run(input)` — a violating input returns `status: :input_error` instead of calling the model. `RubyLLM::Agent.inputs` is a list of *named template locals* injected into ERB instructions. They solve different problems (validation vs templating) and can coexist on the same project.
 
 > **Not the same as `RubyLLM::Agent` ERB templates either.** The `prompt do ... end` DSL above builds a multi-role message list (`system` / `user` / `assistant` / `example` nodes) into a node-AST. `Agent.instructions :name` loads a single-string ERB file from `app/prompts/<agent_path>/<name>.txt.erb` for the system prompt only. Different output shape, different scope. Variable interpolation here uses `{key}` substitution; ERB uses full `<%= ruby %>`.
 

data/docs/guide/rails_integration.md

@@ -141,16 +141,22 @@ Add to `spec/spec_helper.rb` (or `test_helper.rb`):
 require "ruby_llm/contract/rspec"    # or ruby_llm/contract/minitest
 ```
 
+With Minitest, also `include RubyLLM::Contract::MinitestHelpers` in `ActiveSupport::TestCase`.
+
 Then in specs:
 
 ```ruby
 RSpec.describe ArticlesController do
+  include ActiveJob::TestHelper
+
   it "saves the summary when the step passes" do
     stub_step(SummarizeArticle, response: {
       tldr: "...", takeaways: %w[a b c], tone: "analytical"
     })
 
-    post :summarize, params: { id: article.id }
+    # `summarize` enqueues SummarizeArticleJob; run it here, on this thread,
+    # where the stub applies (stub_step is thread-local).
+    perform_enqueued_jobs { post :summarize, params: { id: article.id } }
 
     # NOTE: jsonb/json column round-trips as string keys on reload.
     expect(article.reload.summary["tldr"]).to eq("...")

data/docs/guide/relation_to_agent.md

@@ -8,7 +8,8 @@
 
 | What you write | Where it lives |
 |---|---|
-| `model`, `temperature`, `schema`, `instructions`, `tools`, `thinking` | covered by both — same idea, different DSL surface |
+| `model`, `temperature`, `schema`, `instructions`, `thinking` | covered by both — same idea, different DSL surface |
+| `tools` | only in `RubyLLM::Agent` — a Step makes a single call without tools |
 | `validate :rule do ... end` business invariants on output | only in `ruby_llm-contract` |
 | `retry_policy escalate(...)` model escalation on validation failure | only here (different from RubyLLM's network-level retry) |
 | `max_cost` / `max_input` / `max_output` pre-flight refusal | only here |

data/docs/guide/relation_to_tribunal.md

@@ -13,7 +13,7 @@ Both gems sit on top of `ruby_llm`. The space they cover overlaps in vocabulary
 | Where the output lives at evaluation time | Already in your variable, returned to caller | Still inside the gem's runner, not yet released |
 | What "fail" means | Red test in CI | Trigger retry/escalate on a stronger model, or fail closed |
 | Strongest features | Rich LLM-as-judge (faithful, relevant, hallucination, refusal, bias, toxicity, jailbreak, PII), red-team adversarial prompts, deterministic helpers (`assert_contains`, `assert_levenshtein`, …), RSpec/Minitest matchers, HTML/JUnit/GH reporters | Schema DSL with constraints, `validate` business rules, `retry_policy escalate(...)` model escalation, `max_cost` pre-flight refusal, regression-eval framework (frozen dataset + baseline + min_score gate), pipeline composition |
-| What it does NOT cover | No retry, no model escalation, no pre-flight cost cap, no contract layer between LLM and your code | No 10-judge LLM-as-judge catalog, no red-team generation, no rich deterministic assertion library, no test-framework matchers |
+| What it does NOT cover | No retry, no model escalation, no pre-flight cost cap, no contract layer between LLM and your code | No 10-judge LLM-as-judge catalog, no red-team generation, no rich deterministic assertion library; its test-framework matchers (`satisfy_contract`, `pass_eval`) gate contracts and evals, not individual outputs |
 
 ## Visual: where each gem sits in your call
 
@@ -38,10 +38,10 @@ your code ──► Step.run ──► LLM ──► [schema + validate]
                                                                                  │
                                                               all attempts fail ─┘
                                                                                  ▼
-                                                                  Result(:validation_error)
+                                                                  status: :validation_failed
 ```
 
-The output **never reaches your code** until the contract passes. A failed validation either fixes itself (the retry policy escalates to a stronger model) or fails closed with `Result(:validation_error)` — your call site sees a deterministic failure status, never schema-invalid data.
+The output **never reaches your code** until the contract passes. A failed validation either fixes itself (the retry policy escalates to a stronger model) or fails closed with `status: :validation_failed` — your call site sees a deterministic failure status, never schema-invalid data.
 
 ### Both together (Tribunal in CI → then Contract in prod)
 
@@ -85,7 +85,7 @@ If you specifically need an **LLM-as-judge** (a second LLM grading the first one
   - **You need to iterate the judge prompt** because it over-flags on your data — Tribunal's prompts are fixed per assertion.
 - **Use both** for the same project even when your check is in Tribunal's catalog: Tribunal's `assert_faithful` for spec-time grade on individual responses, plus a calibrated custom judge wired as `evaluator:` in a regression `define_eval` over a frozen dataset for CI merge-gating. They cover different lifecycle stages.
 
-Either way, the **methodology** in [`llm_judge.md`](llm_judge.md) — calibrate the judge against human-labeled production samples before trusting any score, watch for the six anti-patterns, refine the prompt when it over-flags — applies equally to Tribunal's built-ins, Tribunal's custom registered judges, and Contract `Step::Base` judges. Tribunal's `default_threshold = 0.8` is a starting point, not a calibrated bar for your data.
+Either way, the **methodology** in [`llm_judge.md`](llm_judge.md) — calibrate the judge against human-labeled production samples before trusting any score, watch for the five anti-patterns, refine the prompt when it over-flags — applies equally to Tribunal's built-ins, Tribunal's custom registered judges, and Contract `Step::Base` judges. Tribunal's `default_threshold = 0.8` is a starting point, not a calibrated bar for your data.
 
 ## What Tribunal documents — and what it doesn't
 
@@ -111,7 +111,7 @@ These work today without any code changes in either gem — both use plain Ruby
 class ChatReply < RubyLLM::Contract::Step::Base
   prompt "Answer this question grounded in the docs:\n{input}"
 
-  validate("no PII in answer") do |output, _ctx|
+  validate("no PII in answer") do |output, _input|
     test_case = RubyLLM::Tribunal::TestCase.new(actual_output: output[:answer])
     RubyLLM::Tribunal::Assertions.evaluate(:pii, test_case, {}).first == :pass
   end

data/docs/guide/testing.md

@@ -33,7 +33,7 @@ result = SummarizeArticle.run("article text", context: { adapter: adapter })
 result.ok?  # => true
 ```
 
-Multi-step pipeline testing with per-step named responses (using `ArticleCardPipeline` from [Pipeline](pipeline.md)). The keys in `responses:` must match the step identifiers declared with `add_step :name, ...` in your pipeline definition — `:summarize`, `:tag`, `:card` here map to the three steps in that order.
+Multi-step pipeline testing with per-step named responses (using `ArticleCardPipeline` from [Pipeline](pipeline.md)). The keys in `responses:` must match the step identifiers declared with `step Klass, as: :name` in your pipeline definition — `:summarize`, `:tag`, `:card` here map to the three steps in that order.
 
 ```ruby
 result = ArticleCardPipeline.test("article text",
@@ -56,7 +56,7 @@ result.parsed_output[:tldr]     # => "..." ✓
 result.parsed_output["tldr"]    # => nil ✗
 ```
 
-The gem warns if a `validate` block (Step-level invariant check) or a `verify` block (eval-case evaluator declared via `verify(name) { |output| ... }` inside `define_eval`) returns `nil` — usually a sign of string-key access on symbol-keyed data.
+The gem warns if a `validate` block (Step-level invariant check) or a `verify` proc (eval-case evaluator declared via `verify "name", ->(output) { ... }` inside `define_eval`) returns `nil` — usually a sign of string-key access on symbol-keyed data.
 
 ## RSpec setup
 
@@ -128,7 +128,7 @@ In RSpec, non-block stubs auto-clean after each example. In Minitest, `teardown`
 
 ## Minitest
 
-Require `ruby_llm/contract/minitest` in your `test_helper.rb`. You get the same `satisfy_contract` / `pass_eval` assertions and `stub_step` helper, adapted for Minitest syntax.
+Require `ruby_llm/contract/minitest` in your `test_helper.rb` and `include RubyLLM::Contract::MinitestHelpers` in your test class (or `ActiveSupport::TestCase`). You get `assert_satisfies_contract` / `refute_satisfies_contract` and `assert_eval_passes` (the Minitest counterparts of `satisfy_contract` and `pass_eval`) and the same `stub_step` helper.
 
 ## RSpec matchers
 
@@ -158,7 +158,7 @@ end
 
 `pass_eval` supports a matcher chain — full reference lives in [Getting Started](getting_started.md) under Evals and CI gates. Quick summary:
 
-- `.with_context(model: "gpt-4.1-mini")` — pick model / pass adapter
+- `.with_context(model: "gpt-4.1-mini")` — pick model / pass adapter (on a step with a `retry_policy` chain, also pass `retry_policy_override: nil`, or the chain picks the model)
 - `.with_minimum_score(0.8)` — gate on average score
 - `.with_maximum_cost(0.01)` — gate on total cost; fails when a case's model has no pricing data unless you add `.on_unknown_pricing(:warn)`
 - `.without_regressions` — block any previously-passing case that now fails (reads the baseline)
@@ -168,18 +168,19 @@ end
 
 Evals run in one of two modes depending on how they're defined and what context is passed:
 
-| Has `sample_response`? | Context has adapter/model? | Mode | API calls |
+| Has `sample_response`? | `adapter:` in context? | Mode | API calls |
 |---|---|---|---|
-| Yes | No | **Offline** — uses `sample_response` as canned answer | Zero |
+| Yes | No | **Offline** — uses `sample_response` as canned answer (a `model:` alone does not change that) | Zero |
 | Yes | Yes | **Online** — ignores `sample_response`, calls real LLM | Real |
 | No | Yes | **Online** — calls real LLM | Real |
-| No | No | **Skipped** — returns `:skipped`, excluded from score | Zero |
+| No | No | **Online** with the default adapter, which `RubyLLM::Contract.configure` sets up; **Skipped** (`:skipped`, excluded from score) only when no adapter is configured | Real / Zero |
 
-Default is offline. To force online, pass adapter or model in context:
+Default is offline. To force online, pass an adapter in context:
 
 ```ruby
 # Online — real LLM call
-report = SummarizeArticle.run_eval("regression", context: { model: "gpt-4.1-nano" })
+report = SummarizeArticle.run_eval("regression",
+  context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new, model: "gpt-4.1-nano" })
 
 # Offline — uses sample_response
 report = SummarizeArticle.run_eval("smoke")
@@ -232,7 +233,7 @@ result.ok?           # => true (observe never fails the contract)
 result.observations  # => [{ description: "scores should differ", passed: false }]
 ```
 
-`observe` runs only after validation passes. Failed observations are logged via `RubyLLM::Contract.logger` — useful for "I want to know this happened without blocking the response".
+`observe` runs only after validation passes. Failed observations are logged via the logger set in `RubyLLM::Contract.configure` (`c.logger`) — useful for "I want to know this happened without blocking the response".
 
 > **Not the same as `Chat#on_end_message` / `on_tool_call`.** RubyLLM exposes anonymous, global callbacks attached to a chat instance — they receive the raw `Message` and have no inherent pass/fail concept. `Step.observe` is per-step, named with a description, runs against `parsed_output` (already JSON-parsed and schema-validated), and the pass/fail outcome is recorded in `result.observations`. The two can coexist: use Chat callbacks for transport/tool tracing, use `observe` for domain assertions you want captured in the step trace.
 
@@ -269,8 +270,8 @@ Baselines are JSON files in `.eval_baselines/` — commit them to git:
 ```
 .eval_baselines/
   SummarizeArticle/
-    regression_gpt-4_1-nano.json
-    regression_gpt-4_1-mini.json
+    regression.json                  # report.save_baseline!, the rake task, without_regressions
+    regression_gpt-4_1-mini.json     # report.save_baseline!(model: "gpt-4.1-mini"), one per model
 ```
 
 Each file contains dataset name, step name, score, and per-case results. No timestamps — re-saving an identical baseline produces no git diff. Baseline semantics (what counts as a regression, how `compare_with_baseline` works) are covered in [Getting Started](getting_started.md#evals-and-ci-gates) and [Eval-First](eval_first.md).

data/docs/guide/why.md

@@ -62,18 +62,18 @@ Nano misclassifies the tone on the first attempt → contract rejects → mini g
 
 **See it in 30 seconds:** `ruby examples/01_fallback_showcase.rb` — zero API keys required. The Test adapter simulates a tone/takeaways mismatch on the first attempt and a consistent sample on the retry, then prints the per-attempt trace.
 
-`retry_policy` has three other shapes beyond cross-model escalation — same-model `attempts: 3` (absorbs sampling variance without paying for a stronger tier), `reasoning_effort` escalation (low → medium → high on one model), and cross-provider fallback (Ollama → Anthropic → OpenAI — local first because it costs nothing, hosted last because it is the most accurate). `examples/06_retry_variants.rb` runs all three through the Test adapter with the trace printed.
+`retry_policy` has three other shapes beyond cross-model escalation — same-model `attempts: 3` (absorbs sampling variance without paying for a stronger tier), `reasoning_effort` escalation (low → medium → high on one model), and cross-provider fallback (OpenAI → Anthropic — ruby_llm picks the provider from the model name; a local Ollama model needs `provider: :ollama` for the whole call, so it cannot be one tier of such a chain). `examples/06_retry_variants.rb` runs all three through the Test adapter with the trace printed.
 
 ## Failure 4 — Runaway cost and no fallback policy
 
-Someone pastes a 40-page PDF into the endpoint that calls `SummarizeArticle`. The prompt expands to 80k tokens. Your provider bill jumps. Meanwhile, a separate team uses `gpt-4.1` for every single call because "quality matters" — even though 80% of their traffic is trivially handled by `gpt-4.1-nano` at 1/30th the cost. Neither situation is visible until the invoice arrives.
+Someone pastes a 40-page PDF into the endpoint that calls `SummarizeArticle`. The prompt expands to 80k tokens. Your provider bill jumps. Meanwhile, a separate team uses `gpt-4.1` for every single call because "quality matters" — even though 80% of their traffic is trivially handled by `gpt-4.1-nano` at 1/20th the cost. Neither situation is visible until the invoice arrives.
 
 Contracts make cost a first-class concern:
 
 ```ruby
 max_input  2_000   # refuses before calling the API if tokens exceed budget
 max_output 4_000
-max_cost   0.01    # refuses if estimated cost exceeds cap
+max_cost   0.05    # refuses if estimated cost exceeds cap (checked per attempt)
 retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]  # cheap first, escalate only on failure
 ```
 

data/examples/00_basics.rb

@@ -166,9 +166,15 @@ r.trace[:cost]       # => sum of per-attempt costs
 #   adapter = RubyLLM::Contract::Adapters::RubyLLM.new
 #   result  = SummarizeArticleWithRetry.run(article_text, context: { adapter: adapter })
 #
-# Switch provider per call — ruby_llm resolves the provider from the model name:
-#   SummarizeArticleWithRetry.run(article_text, context: { adapter: adapter, model: "claude-sonnet-4-6" })
-#   SummarizeArticleWithRetry.run(article_text, context: { adapter: adapter, model: "gemma3:4b" })  # local Ollama
+# Switch provider per call — ruby_llm resolves the provider from the model name.
+# The retry_policy chain picks the model of every attempt, so turn it off for
+# the call (retry_policy_override: nil) to run a model of your choice:
+#   (configure c.anthropic_api_key, or c.ollama_api_base for Ollama, first)
+#   SummarizeArticleWithRetry.run(article_text, context: { adapter: adapter, model: "claude-sonnet-4-6",
+#                                                          retry_policy_override: nil })
+#   SummarizeArticleWithRetry.run(article_text, context: { adapter: adapter, model: "gemma3:4b",
+#                                                          retry_policy_override: nil,
+#                                                          provider: :ollama, assume_model_exists: true })
 
 # =============================================================================
 # Where to go next

data/examples/01_fallback_showcase.rb

@@ -50,7 +50,7 @@
 #     tone:       "negative"
 #
 # See also: examples/06_retry_variants.rb — same-model retry, reasoning_effort
-# escalation, and cross-provider fallback (Ollama → Anthropic → OpenAI).
+# escalation, and cross-provider fallback (OpenAI → Anthropic).
 # =============================================================================
 
 require_relative "../lib/ruby_llm/contract"

data/examples/02_real_llm_minimal.rb

@@ -8,7 +8,8 @@
 # not change — same prompt, schema, validates, retry_policy.
 #
 # Requires: gem install ruby_llm; export OPENAI_API_KEY=sk-...
-# (Or an Anthropic / Gemini / Mistral key, or a local Ollama server.)
+# Another provider needs its own key in RubyLLM.configure below (e.g.
+# c.anthropic_api_key = ENV["ANTHROPIC_API_KEY"]) and its models in the step.
 #
 # Run: OPENAI_API_KEY=sk-... ruby examples/02_real_llm_minimal.rb
 # =============================================================================
@@ -40,6 +41,11 @@ puts "Tokens:      #{result.trace[:usage]}"          # real usage
 puts "Cost:        $#{result.trace[:cost]}"          # sum across retries
 puts "TL;DR:       #{result.parsed_output[:tldr]}"
 
-# Switch provider per call — ruby_llm resolves the provider from the model name:
-#   SummarizeArticle.run(article, context: { adapter: adapter, model: "claude-sonnet-4-6" })
-#   SummarizeArticle.run(article, context: { adapter: adapter, model: "gemma3:4b" })  # local Ollama
+# Switch provider per call — ruby_llm resolves the provider from the model name.
+# The step's retry_policy chain picks the model of every attempt, so turn it off
+# for the call (retry_policy_override: nil) to run a model of your choice:
+#   (configure c.anthropic_api_key, or c.ollama_api_base for Ollama, first)
+#   SummarizeArticle.run(article, context: { adapter: adapter, model: "claude-sonnet-4-6",
+#                                            retry_policy_override: nil })
+#   SummarizeArticle.run(article, context: { adapter: adapter, model: "gemma3:4b", retry_policy_override: nil,
+#                                            provider: :ollama, assume_model_exists: true })  # local Ollama

data/examples/06_retry_variants.rb

@@ -20,9 +20,9 @@
 #       attempt 1  effort=low     status=validation_failed
 #       attempt 3  effort=high    status=ok
 #
-#   C — cross-provider Ollama → Anthropic → OpenAI
-#       attempt 1  model=gemma3:4b          status=validation_failed
-#       attempt 3  model=gpt-5-nano         status=ok
+#   C — cross-provider OpenAI → Anthropic
+#       attempt 1  model=gpt-5-nano         status=validation_failed
+#       attempt 3  model=claude-haiku-4-5   status=ok
 # =============================================================================
 
 require_relative "../lib/ruby_llm/contract"
@@ -108,26 +108,30 @@ adapter = RubyLLM::Contract::Adapters::Test.new(responses: RESPONSES)
 print_trace("reasoning escalation", SummarizeArticleReasoningEscalation.run("article", context: { adapter: adapter }))
 
 # =============================================================================
-# VARIANT C — cross-provider fallback (Ollama → Anthropic → OpenAI)
+# VARIANT C — cross-provider fallback (OpenAI → Anthropic)
 #
-# When to use: you want to start on a local model (cheap, private, no quota)
-# and fall back to hosted providers only when the local one cannot satisfy
-# the contract. Each tier is a different provider — ruby_llm detects the
-# provider from the model name.
+# When to use: the cheap tiers live on one provider and the last resort on
+# another. ruby_llm detects the provider from the model name, so a chain of
+# models from its registry can cross providers.
 #
-# To run against real backends: configure ruby_llm for all three providers
-# (ollama_api_base + anthropic_api_key + openai_api_key) and swap the Test
-# adapter for Adapters::RubyLLM. The retry_policy itself is unchanged.
+# To run against real backends: configure ruby_llm for both providers
+# (openai_api_key + anthropic_api_key) and swap the Test adapter for
+# Adapters::RubyLLM. The retry_policy itself is unchanged.
 #
-# Order matters: local first (costs nothing); hosted last (most accurate).
+# A chain element carries only `model` and `reasoning_effort`. A local
+# Ollama model needs `provider: :ollama` (and `assume_model_exists: true`)
+# in the context, which applies to every attempt, so it cannot be one tier
+# of a chain that also reaches hosted providers.
+#
+# Order matters: cheapest first; the last tier is the safety net.
 # =============================================================================
 
 class SummarizeArticleCrossProvider < SummarizeArticle
-  retry_policy models: %w[gemma3:4b claude-haiku-4-5 gpt-5-nano]
+  retry_policy models: %w[gpt-5-nano gpt-5-mini claude-haiku-4-5]
 end
 
 puts "=" * 70
-puts "C — cross-provider fallback (Ollama → Anthropic → OpenAI)"
+puts "C — cross-provider fallback (OpenAI → Anthropic)"
 puts "=" * 70
 adapter = RubyLLM::Contract::Adapters::Test.new(responses: RESPONSES)
 print_trace("cross-provider", SummarizeArticleCrossProvider.run("article", context: { adapter: adapter }))
@@ -140,8 +144,8 @@ print_trace("cross-provider", SummarizeArticleCrossProvider.run("article", conte
 # 2. `reasoning_effort` escalation is cheaper than model escalation when the
 #    model is right but needs more thinking, not a stronger backbone.
 # 3. Cross-provider retry uses the same DSL — ruby_llm resolves the provider
-#    from the model name. Start cheapest (often a local Ollama model), end
-#    on the most accurate hosted provider.
+#    from the model name. Start cheapest, end on the most accurate model,
+#    whichever provider hosts it.
 # 4. The per-attempt trace (model, config, status, cost) is identical across
 #    variants — your logging does not care which retry shape you picked.
 # =============================================================================

data/examples/README.md

@@ -8,11 +8,11 @@ Pedagogical order: hook → activation → evolution → composition → quality
 |---|------|---------|
 | 00 | `00_basics.rb` | **"How do I start?"** — seven incremental layers: plain prompt → output_schema → validate → structured prompt → Hash input → cross-input validate → retry_policy → trace inspection, plus real-LLM swap pointer. |
 | 01 | `01_fallback_showcase.rb` | **"Show me the gem in 30 seconds."** — Part A: schema-only ships a flaky sample. Part B: full contract rejects it and retry_policy escalates to the next model. Per-attempt trace printed inline. |
-| 02 | `02_real_llm_minimal.rb` | **"How do I plug in a real LLM?"** — ~30 lines. `Adapters::RubyLLM.new` in context, same step. Also shows per-call provider switch (OpenAI → Anthropic → Ollama). |
+| 02 | `02_real_llm_minimal.rb` | **"How do I plug in a real LLM?"** — ~30 lines. `Adapters::RubyLLM.new` in context, same step. Also shows a per-call model switch (Anthropic, or a local Ollama model) with the step's retry chain turned off for that call. |
 | 03 | `03_summarize_with_keywords.rb` | **"How does the contract evolve when the product grows?"** — marketing wants a "topic pills" row, so `SummarizeArticle` gains a keywords field with probability and cross-validation. Prompt, schema, and validates stay in lockstep. |
 | 04 | `04_summarize_and_translate.rb` | **"How do steps compose into a pipeline?"** — 3 steps threaded by `Pipeline::Base`: English summary → translate to French → quality review. Fail-fast: a rejected summary means translate and review never run. |
 | 05 | `05_eval_dataset.rb` | **"How do I stop silent prompt regressions?"** — define_eval with real cases, baseline vs regressed adapter, regression detection signal, inline eval_case. |
-| 06 | `06_retry_variants.rb` | **"What retry shapes exist beyond cross-model?"** — `attempts: 3` (variance absorption), `reasoning_effort` escalation (low→medium→high), cross-provider fallback (Ollama → Anthropic → OpenAI). |
+| 06 | `06_retry_variants.rb` | **"What retry shapes exist beyond cross-model?"** — `attempts: 3` (variance absorption), `reasoning_effort` escalation (low→medium→high), cross-provider fallback (OpenAI → Anthropic). |
 
 Four of the seven carry an "Expected output" section in the file header (01, 03, 05, 06), so you can read what they print without running them. The other three differ: `00_basics.rb` prints nothing and annotates each result inline with `# =>`; `02_real_llm_minimal.rb` needs a live provider key, so its output is not deterministic; `04_summarize_and_translate.rb` prints but has no header block.
 

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -114,8 +114,12 @@ module RubyLLM
                 "call compare_models with production_mode: on individual Step classes instead."
         end
 
+        # A Step's retry chain would pick every attempt's model, so a candidate
+        # runs without it (same-model retries stay), unless production_mode or
+        # the caller passes a retry policy for the comparison.
         def build_candidate_context(context, config, fallback_config)
-          model_context = isolate_context(context).merge(model: config[:model])
+          model_context = { retry_policy_override: Step::RetryPolicy::WITHOUT_MODEL_CHAIN }
+                          .merge(isolate_context(context), model: config[:model])
           model_context[:reasoning_effort] = config[:reasoning_effort] if config[:reasoning_effort]
           return model_context unless fallback_config
 

data/lib/ruby_llm/contract/eval/evaluator/proc_evaluator.rb

@@ -39,6 +39,8 @@ module RubyLLM
 
           def build_evaluation_result(result)
             case result
+            when EvaluationResult
+              result
             when true
               EvaluationResult.new(score: 1.0, passed: true, details: PASSED_DETAILS)
             when false

data/lib/ruby_llm/contract/eval/model_comparison.rb

@@ -74,8 +74,8 @@ module RubyLLM
         def production_mode_table
           fallback_label = self.class.candidate_label(@fallback)
           rows = @reports.map do |label, report|
-            chain = chain_label(label, fallback_label)
-            { chain: chain, report: report, same: chain_same_as_fallback?(label, fallback_label) }
+            same = !report.production_mode?
+            { chain: same ? label : "#{label} → #{fallback_label}", report: report, same: same }
           end
 
           chain_width = [rows.map { |r| r[:chain].length }.max || 0, 20].max
@@ -92,14 +92,6 @@ module RubyLLM
 
         private
 
-        def chain_label(label, fallback_label)
-          label == fallback_label ? label : "#{label} → #{fallback_label}"
-        end
-
-        def chain_same_as_fallback?(label, fallback_label)
-          label == fallback_label
-        end
-
         def format_production_row(row, chain_width)
           report = row[:report]
           format("  %-#{chain_width}s  %-13s  %-10s  %-14s  %-9s  %6.2f",

data/lib/ruby_llm/contract/step/base.rb

@@ -195,8 +195,17 @@ module RubyLLM
             end
           end
 
+          def runtime_policy(context)
+            return retry_policy unless context.key?(:retry_policy_override)
+
+            override = context[:retry_policy_override]
+            return override unless override == RetryPolicy::WITHOUT_MODEL_CHAIN
+
+            retry_policy if retry_policy&.model_list&.empty?
+          end
+
           def runtime_settings(context)
-            policy = context.key?(:retry_policy_override) ? context[:retry_policy_override] : retry_policy
+            policy = runtime_policy(context)
             extra = context.slice(*ADAPTER_CONTEXT_KEYS).merge(merged_provider_options(context))
 
             # Always pass the class-level `thinking` config to the adapter when

data/lib/ruby_llm/contract/step/retry_policy.rb

@@ -15,6 +15,11 @@ module RubyLLM
         # transport retry could not).
         DEFAULT_RETRY_ON = %i[validation_failed parse_error].freeze
 
+        # Internal, not a documented override value: keeps a Step's own policy
+        # unless it picks models, so compare_models runs a candidate on its
+        # own model with the same-model retries the Step would give it.
+        WITHOUT_MODEL_CHAIN = :without_model_chain
+
         def initialize(models: nil, attempts: nil, retry_on: nil, &block)
           @configs = []
           @retryable_statuses = DEFAULT_RETRY_ON.dup

data/lib/ruby_llm/contract/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module Contract
-    VERSION = "1.4.0"
+    VERSION = "1.4.1"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-contract
 version: !ruby/object:Gem::Version
-  version: 1.4.0
+  version: 1.4.1
 platform: ruby
 authors:
 - Justyna