ruby_llm-contract 0.8.0 → 0.10.2

data/docs/guide/testing.md

@@ -0,0 +1,282 @@
+# Testing
+
+> Read this when writing deterministic unit specs for contract steps. Skip if you only ever run live evals in CI and never stub LLM calls.
+
+CI that hits a real LLM on every commit is slow and costly: tests take minutes instead of milliseconds, every rerun spends money, and flakes on provider hiccups block merges. The Test adapter + `stub_step` make LLM-backed specs run like ordinary unit tests — deterministic, offline, free. Live evals stay as an opt-in CI stage for quality gating (see [Eval-First](eval_first.md)).
+
+How to write deterministic specs and matchers for steps built on `ruby_llm-contract`. Examples use `SummarizeArticle` (the flagship step from the [README](../../README.md)).
+
+## Test adapter
+
+Ships deterministic specs with zero API calls. Accepts a String, Hash, or Array:
+
+```ruby
+# String JSON
+adapter = RubyLLM::Contract::Adapters::Test.new(
+  response: '{"tldr":"...","takeaways":["a","b","c"],"tone":"neutral"}'
+)
+
+# Hash — auto-converted to JSON
+adapter = RubyLLM::Contract::Adapters::Test.new(
+  response: { tldr: "...", takeaways: %w[a b c], tone: "neutral" }
+)
+
+# Multiple sequential responses (one per call)
+adapter = RubyLLM::Contract::Adapters::Test.new(
+  responses: [
+    { tldr: "...", takeaways: %w[a b c], tone: "neutral" },
+    { tldr: "...", takeaways: %w[x y z], tone: "analytical" }
+  ]
+)
+
+result = SummarizeArticle.run("article text", context: { adapter: adapter })
+result.ok?  # => true
+```
+
+Multi-step pipeline testing with per-step named responses (using `ArticleCardPipeline` from [Pipeline](pipeline.md)):
+
+```ruby
+result = ArticleCardPipeline.test("article text",
+  responses: {
+    summarize: { tldr: "...", takeaways: %w[a b c], tone: "analytical" },
+    tag:       { tldr: "...", takeaways: %w[a b c], tone: "analytical",
+                 hashtags: %w[#ruby #release] },
+    card:      { headline: "Ruby 3.4 ships", summary: "...",
+                 hashtags: %w[#ruby #release], sentiment_icon: "🧠" }
+  }
+)
+```
+
+## Output keys are always symbols
+
+Parsed output uses **symbol keys**, never strings:
+
+```ruby
+result.parsed_output[:tldr]     # => "..." ✓
+result.parsed_output["tldr"]    # => nil ✗
+```
+
+The gem warns if a `validate` or `verify` block returns `nil` — usually a sign of string-key access on symbol-keyed data.
+
+## RSpec setup
+
+In `spec_helper.rb`:
+
+```ruby
+require "ruby_llm/contract/rspec"
+```
+
+You get the `satisfy_contract` matcher, `pass_eval` matcher, and the `stub_step` helpers.
+
+## stub_step helpers
+
+`stub_step` canned-responses a single step; other steps run normally.
+
+```ruby
+RSpec.describe SummarizeArticle do
+  before do
+    stub_step(described_class,
+              response: { tldr: "...", takeaways: %w[a b c], tone: "neutral" })
+  end
+
+  it "satisfies its contract" do
+    result = described_class.run("article text")
+    expect(result).to satisfy_contract
+  end
+end
+```
+
+**Sequential responses:**
+
+```ruby
+stub_step(described_class, responses: [
+  { tldr: "...", takeaways: %w[a b c], tone: "neutral" },
+  { tldr: "...", takeaways: %w[x y z], tone: "analytical" }
+])
+```
+
+**Block form (auto-cleanup):**
+
+```ruby
+stub_step(SummarizeArticle, response: { tldr: "...", takeaways: %w[a b c], tone: "neutral" }) do
+  result = SummarizeArticle.run("article text")
+  # stub is active here
+end
+# stub gone — original adapter restored
+```
+
+**Multiple steps at once:**
+
+```ruby
+stub_steps(
+  SummarizeArticle => { response: { tldr: "...", takeaways: %w[a b c], tone: "neutral" } },
+  GenerateHashtags => { response: { tldr: "...", takeaways: %w[a b c],
+                                    tone: "neutral",
+                                    hashtags: %w[#ruby #release] } }
+) do
+  result = ArticleCardPipeline.run("article text")
+end
+```
+
+**Global stub for all steps:**
+
+```ruby
+stub_all_steps(response: { default: true })
+```
+
+In RSpec, non-block stubs auto-clean after each example. In Minitest, `teardown` restores the original adapter (via `MinitestHelpers`).
+
+## Minitest
+
+Require `ruby_llm/contract/minitest` in your `test_helper.rb`. You get the same `satisfy_contract` / `pass_eval` assertions and `stub_step` helper, adapted for Minitest syntax.
+
+## RSpec matchers
+
+```ruby
+RSpec.describe SummarizeArticle do
+  before do
+    stub_step(described_class,
+              response: { tldr: "...", takeaways: %w[a b c], tone: "neutral" })
+  end
+
+  it "satisfies its contract" do
+    result = described_class.run("article text")
+    expect(result).to satisfy_contract
+  end
+
+  it "rejects invalid output" do
+    stub_step(described_class, response: { tldr: "x" * 300, takeaways: %w[a b c], tone: "neutral" })
+    result = described_class.run("article text")
+    expect(result).not_to satisfy_contract  # TL;DR > 200 chars fails validate
+  end
+
+  it "passes its eval" do
+    expect(described_class).to pass_eval("smoke")
+  end
+end
+```
+
+`pass_eval` supports a matcher chain — full reference lives in [Getting Started](getting_started.md) under Evals and CI gates. Quick summary:
+
+- `.with_context(model: "gpt-4.1-mini")` — pick model / pass adapter
+- `.with_minimum_score(0.8)` — gate on average score
+- `.with_maximum_cost(0.01)` — gate on total cost
+- `.without_regressions` — block any previously-passing case that now fails (reads the baseline)
+- `.compared_with(SummarizeArticleV1)` — A/B against another step; implies regression check
+
+## Offline vs online eval
+
+Evals run in one of two modes depending on how they're defined and what context is passed:
+
+| Has `sample_response`? | Context has adapter/model? | Mode | API calls |
+|---|---|---|---|
+| Yes | No | **Offline** — uses `sample_response` as canned answer | Zero |
+| Yes | Yes | **Online** — ignores `sample_response`, calls real LLM | Real |
+| No | Yes | **Online** — calls real LLM | Real |
+| No | No | **Skipped** — returns `:skipped`, excluded from score | Zero |
+
+Default is offline. To force online, pass adapter or model in context:
+
+```ruby
+# Online — real LLM call
+report = SummarizeArticle.run_eval("regression", context: { model: "gpt-4.1-nano" })
+
+# Offline — uses sample_response
+report = SummarizeArticle.run_eval("smoke")
+```
+
+`compare_with` intentionally ignores `sample_response` because canned data would make both sides look identical. Always pass `model:` or an adapter to A/B.
+
+## Inspecting failures
+
+`run_eval` returns a `Report`. Drill into per-case failures:
+
+```ruby
+report = SummarizeArticle.run_eval("regression")
+
+report.score       # => 0.5
+report.pass_rate   # => "1/2"
+report.total_cost  # => 0.003
+
+report.failures.each do |result|
+  puts result.name        # => "critical review"
+  puts result.mismatches  # => { tone: { expected: "negative", got: "analytical" } }
+  puts result.output      # full parsed output hash
+  puts result.details     # human-readable explanation
+end
+```
+
+`mismatches` is a hash of keys where expected and actual output diverge — pinpoints which field the model got wrong.
+
+## Soft observations
+
+Log suspicious-but-not-invalid output without failing the contract:
+
+```ruby
+class CompareArticles < RubyLLM::Contract::Step::Base
+  prompt "Score the article pair for relevance. Return JSON: {score_a: 1-10, score_b: 1-10}.\n\n{input}"
+
+  output_schema do
+    integer :score_a, minimum: 1, maximum: 10
+    integer :score_b, minimum: 1, maximum: 10
+  end
+
+  validate("scores in range") { |o, _| (1..10).cover?(o[:score_a]) && (1..10).cover?(o[:score_b]) }
+  observe("scores should differ") { |o, _| o[:score_a] != o[:score_b] }
+end
+
+adapter = RubyLLM::Contract::Adapters::Test.new(response: { score_a: 5, score_b: 5 })
+result  = CompareArticles.run("two identical-looking articles", context: { adapter: adapter })
+
+result.ok?           # => true (observe never fails the contract)
+result.observations  # => [{ description: "scores should differ", passed: false }]
+```
+
+`observe` runs only after validation passes. Failed observations are logged via `RubyLLM::Contract.logger` — useful for "I want to know this happened without blocking the response".
+
+> **Not the same as `Chat#on_end_message` / `on_tool_call`.** RubyLLM exposes anonymous, global callbacks attached to a chat instance — they receive the raw `Message` and have no inherent pass/fail concept. `Step.observe` is per-step, named with a description, runs against `parsed_output` (already JSON-parsed and schema-validated), and the pass/fail outcome is recorded in `result.observations`. The two can coexist: use Chat callbacks for transport/tool tracing, use `observe` for domain assertions you want captured in the step trace.
+
+## Asserting on `around_call`
+
+`around_call` fires **once per run** with the final result (after retry fallback) and exceptions propagate. That makes it straightforward to test:
+
+```ruby
+class LoggedSummarize < RubyLLM::Contract::Step::Base
+  prompt "Summarize: {input}"
+  output_schema { string :tldr }
+
+  around_call do |_step, input, result|
+    CallLog.record(model: result.trace.model, cost: result.trace.cost, input_size: input.length)
+  end
+end
+
+RSpec.describe LoggedSummarize do
+  it "logs once per run, with final model + total cost" do
+    adapter = RubyLLM::Contract::Adapters::Test.new(response: { tldr: "ok" })
+    expect(CallLog).to receive(:record).once.with(hash_including(:model, :cost, :input_size))
+
+    LoggedSummarize.run("article text", context: { adapter: adapter })
+  end
+end
+```
+
+The callback receives `(step, input, result)` — the same `Result` the caller sees. Not invoked per-attempt inside a `retry_policy` chain; if you need per-attempt visibility, read `result.trace[:attempts]` inside the block.
+
+## Baseline file format
+
+Baselines are JSON files in `.eval_baselines/` — commit them to git:
+
+```
+.eval_baselines/
+  SummarizeArticle/
+    regression_gpt-4_1-nano.json
+    regression_gpt-4_1-mini.json
+```
+
+Each file contains dataset name, step name, score, and per-case results. No timestamps — re-saving an identical baseline produces no git diff. Baseline semantics (what counts as a regression, how `compare_with_baseline` works) are covered in [Getting Started](getting_started.md#evals-and-ci-gates) and [Eval-First](eval_first.md).
+
+## See also
+
+- [Getting Started](getting_started.md) — `pass_eval` matcher chain, threshold gating, Rake task, baseline regressions.
+- [Eval-First](eval_first.md) — `compare_with` prompt A/B workflow.
+- [Pipeline](pipeline.md) — pipeline-level testing with named step responses.

data/docs/guide/why.md

@@ -0,0 +1,103 @@
+# Why contracts?
+
+> Read this if you're not sure whether `ruby_llm-contract` solves a problem you actually have. It's the fastest way to recognise the production failure modes the gem exists for.
+
+LLMs return JSON that *looks* correct — valid shape, right types, right fields — while being silently wrong in ways that hurt users, burn budget, or break downstream code. Schema validation alone does not catch these. Contracts layer on business rules, retries, evals, and cost caps so the wrong output is caught at the boundary of your system instead of shipping to production.
+
+Below are four failure modes teams actually hit. If one looks familiar, the gem is probably worth 30 minutes of your time.
+
+## Failure 1 — Schema-valid, logically wrong
+
+A `SummarizeArticle` step produces `{ tldr: "...", takeaways: [...], tone: "analytical" }`. Schema passes. The TL;DR is 520 characters long and overflows the UI card. Or the takeaways are all variations of the same sentence. Or the article was a service-outage complaint and `tone` came back `"analytical"` instead of `"negative"` — so customer success's "critical feedback" filter never sees it.
+
+JSON schema enforces **shape**. It cannot enforce *length fits the card*, *takeaways are distinct*, or *tone matches content*. Those are business rules, and without them you find out from a Slack thread or a support ticket.
+
+```ruby
+validate("TL;DR fits the card")  { |o, _| o[:tldr].length <= 200 }
+validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }
+validate("negative tone requires concrete risk") do |o, _|
+  next true unless o[:tone] == "negative"
+  o[:takeaways].any? { |t| t.match?(/fail|break|crash|outage|risk/i) }
+end
+```
+
+Wrong output never reaches `Article.update!` — the contract refuses before it persists.
+
+## Failure 2 — Silent prompt regression
+
+`SummarizeArticle` ships and works. Two weeks later, someone tweaks the system prompt to emphasise negative sentiment because customer success complained about missed complaints. The tweak fixes that case and silently breaks three neutral product-update articles that now get labelled `"negative"`. Nobody knows for a week.
+
+Without evals, *every prompt change is a blind deploy.* Contracts invert this:
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "outage complaint", input: "...", expected: { tone: "negative" }
+  add_case "neutral product update", input: "...", expected: { tone: "neutral" }
+end
+
+# In CI — blocks merge when a prompt tweak regresses any previously-passing case
+expect(SummarizeArticle).to pass_eval("regression").without_regressions
+```
+
+The "tweak helped one case, broke three" scenario is caught at PR review. No Slack-thread surprises.
+
+## Failure 3 — Sampling variance on fixed-temperature models
+
+OpenAI's gpt-5 and o-series run with `temperature=1.0` server-side — you cannot lower it. That means the same prompt on the same model can produce different answers between calls. An outage complaint classified `tone: "negative"` on Monday may come back `tone: "positive"` on Tuesday, with no code change in between. Schema passes both. Your customer-success filter silently misroutes the Tuesday case.
+
+A `validate` block that cross-checks fields against each other turns a one-in-N flaky output into a deterministic retry:
+
+```ruby
+validate("tone matches severity keywords") do |o, _|
+  severity = /fail|crash|outage|broken|bug|error/i
+  flagged = o[:takeaways].any? { |t| t.match?(severity) }
+  next true unless flagged
+  %w[negative analytical].include?(o[:tone])
+end
+
+retry_policy models: %w[gpt-5-nano gpt-5-mini gpt-5]
+```
+
+Nano misclassifies the tone on the first attempt → contract rejects → mini gets the call and returns a different sample. Variance absorbed; the user never sees the flaky run. Your logs show the retry rate and the cost delta.
+
+**See it in 30 seconds:** `ruby examples/01_fallback_showcase.rb` — zero API keys required. The Test adapter simulates a tone/takeaways mismatch on the first attempt and a consistent sample on the retry, then prints the per-attempt trace.
+
+`retry_policy` has three other shapes beyond cross-model escalation — same-model `attempts: 3` (absorbs sampling variance without paying for a stronger tier), `reasoning_effort` escalation (low → medium → high on one model), and cross-provider fallback (Ollama → Anthropic → OpenAI — local first because it costs nothing, hosted last because it is the most accurate). `examples/06_retry_variants.rb` runs all three through the Test adapter with the trace printed.
+
+## Failure 4 — Runaway cost and no fallback policy
+
+Someone pastes a 40-page PDF into the endpoint that calls `SummarizeArticle`. The prompt expands to 80k tokens. Your provider bill jumps. Meanwhile, a separate team uses `gpt-4.1` for every single call because "quality matters" — even though 80% of their traffic is trivially handled by `gpt-4.1-nano` at 1/30th the cost. Neither situation is visible until the invoice arrives.
+
+Contracts make cost a first-class concern:
+
+```ruby
+max_input  2_000   # refuses before calling the API if tokens exceed budget
+max_output 4_000
+max_cost   0.01    # refuses if estimated cost exceeds cap
+retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]  # cheap first, escalate only on failure
+```
+
+The 40-page PDF returns `status: :limit_exceeded` — zero tokens spent. The 80/20 traffic pattern resolves on nano; only the hard 20% escalate. [`optimize_retry_policy`](optimizing_retry_policy.md) tells you empirically which fallback list is cheapest for *your* evals.
+
+## Also catches
+
+- **Leaked prompt placeholders** — model echoes `{article}` or `{audience}` into the output because the template string wasn't interpolated. Validate string-equality check stops it before a user sees it.
+- **Lazy models echoing input verbatim** — cheap model returns the article text as the "summary". 2-arity `validate("tldr shorter than input")` catches it.
+- **Tone mislabel breaking downstream routing** — content says "negative", model labels it "analytical", a customer success filter misses it. Cross-validate catches the label/content drift.
+
+## Failure → contract mechanism
+
+| Failure in production | Contract mechanism |
+|---|---|
+| Schema-valid but logically wrong output | `validate(...) { |o, i| ... }` with 2-arity for cross-checks |
+| Silent prompt regression after a tweak | `define_eval` + `pass_eval(...).without_regressions` in CI |
+| Sampling variance on fixed-temperature models (gpt-5 / o-series) | Cross-field `validate(...)` + `retry_policy models: [...]` |
+| Runaway cost on pathological inputs | `max_input`, `max_output`, `max_cost` preflight |
+| 80/20 traffic paying the premium model rate | `retry_policy` + `optimize_retry_policy` |
+| Leaked placeholder / input echo / tone drift | `validate` with content and cross-input checks |
+
+## What next
+
+- **If one of the failures above looks familiar** → [Getting Started](getting_started.md) walks through every feature in order on the same `SummarizeArticle` step.
+- **If you're adopting in an existing Rails app** → [Migration](migration.md) shows Before/After for replacing a raw `LlmClient.new.call` service.
+- **If you already ship LLM features and want to make them regression-safe** → [Eval-First](eval_first.md) is the workflow that prevents Failure 2 from ever happening again.

data/lib/ruby_llm/contract/adapters/ruby_llm.rb

@@ -13,7 +13,15 @@ module RubyLLM
           chat = build_chat(options, system_contents)
           add_history(chat, conversation[0..-2])
 
-          response = chat.ask(conversation.last&.fetch(:content, ""))
+          # `with: nil` is a documented no-op in RubyLLM (verified against
+          # 1.15.0: chat.rb:36-37 `build_content(message, nil)` -> content.rb:8-14
+          # `Content.new(text, nil)` keeps text-only path when attachments
+          # are empty; raise only fires when BOTH text and attachments are nil,
+          # and we always pass a non-nil string thanks to `&.fetch(:content, "")`).
+          response = chat.ask(
+            conversation.last&.fetch(:content, ""),
+            with: options[:attachment]
+          )
           build_response(response)
         end
 

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -190,16 +190,13 @@ module RubyLLM
           context.merge(adapter: sample_adapter)
         end
 
+        # `Class#subclasses` is available from Ruby 3.1; the gemspec requires
+        # `>= 3.2.0` so the legacy `ObjectSpace.each_object` fallback would be
+        # dead code on every supported runtime.
         def register_subclasses(klass)
-          if klass.respond_to?(:subclasses)
-            klass.subclasses.each do |sub|
-              Contract.register_eval_host(sub)
-              register_subclasses(sub)
-            end
-          else
-            ObjectSpace.each_object(Class) do |sub|
-              Contract.register_eval_host(sub) if sub < klass
-            end
+          klass.subclasses.each do |sub|
+            Contract.register_eval_host(sub)
+            register_subclasses(sub)
           end
         end
 

data/lib/ruby_llm/contract/concerns/stub_helpers.rb

@@ -0,0 +1,97 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Concerns
+      # Shared implementation of `stub_step`, `stub_steps`, and `stub_all_steps`.
+      # Included by both `RubyLLM::Contract::RSpec::Helpers` and
+      # `RubyLLM::Contract::MinitestHelpers` so the two test-framework
+      # adapters cannot drift on stub semantics (Codex DRY finding #1: the
+      # prior parallel implementations had already diverged on
+      # `normalize_test_response` — RSpec had it, Minitest didn't).
+      #
+      # Cleanup between examples is the responsibility of the host helper:
+      # - RSpec: `around(:each)` hook in `lib/ruby_llm/contract/rspec.rb`
+      #   restores `step_adapter_overrides`.
+      # - Minitest: `teardown` in `MinitestHelpers` clears overrides and
+      #   restores `default_adapter`.
+      module StubHelpers
+        # Stub a single step to return a canned response without API calls.
+        # Block form scopes the stub to the block; non-block form lives
+        # until the host's teardown/around hook fires.
+        def stub_step(step_class, response: nil, responses: nil, &block)
+          adapter = build_test_adapter(response: response, responses: responses)
+          overrides = RubyLLM::Contract.step_adapter_overrides
+
+          if block
+            previous = overrides[step_class]
+            overrides[step_class] = adapter
+            begin
+              yield
+            ensure
+              previous ? (overrides[step_class] = previous) : overrides.delete(step_class)
+            end
+          else
+            overrides[step_class] = adapter
+          end
+        end
+
+        # Stub multiple steps with different responses. Requires a block.
+        def stub_steps(stubs, &block)
+          raise ArgumentError, "stub_steps requires a block" unless block
+
+          overrides = RubyLLM::Contract.step_adapter_overrides
+          previous = {}
+
+          stubs.each do |step_class, opts|
+            opts = opts.transform_keys(&:to_sym)
+            previous[step_class] = overrides[step_class]
+            overrides[step_class] = build_test_adapter(**opts.slice(:response, :responses))
+          end
+
+          begin
+            yield
+          ensure
+            stubs.each_key do |step_class|
+              previous[step_class] ? (overrides[step_class] = previous[step_class]) : overrides.delete(step_class)
+            end
+          end
+        end
+
+        # Set a global test adapter for ALL steps. Block form restores the
+        # previous adapter on exit; non-block form persists until host cleanup.
+        def stub_all_steps(response: nil, responses: nil, &block)
+          adapter = build_test_adapter(response: response, responses: responses)
+
+          if block
+            previous = RubyLLM::Contract.configuration.default_adapter
+            begin
+              RubyLLM::Contract.configuration.default_adapter = adapter
+              yield
+            ensure
+              RubyLLM::Contract.configuration.default_adapter = previous
+            end
+          else
+            RubyLLM::Contract.configure { |c| c.default_adapter = adapter }
+          end
+        end
+
+        private
+
+        def build_test_adapter(response: nil, responses: nil)
+          if responses
+            Adapters::Test.new(responses: responses.map { |r| normalize_test_response(r) })
+          else
+            Adapters::Test.new(response: normalize_test_response(response))
+          end
+        end
+
+        # Hook for host frameworks to inject custom serialization (e.g.
+        # turning hashes into JSON strings). Default: identity.
+        def normalize_test_response(value)
+          value
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/contract/definition.rb

@@ -18,6 +18,8 @@ module RubyLLM
       end
 
       def invariant(description, &block)
+        raise ArgumentError, "invariant description must be a non-empty string", caller if description.to_s.empty?
+
         @invariants << Invariant.new(description, block)
       end
       alias validate invariant

data/lib/ruby_llm/contract/cost_calculator.rb

@@ -89,8 +89,13 @@ module RubyLLM
         (input_cost + output_cost).round(6)
       end
 
+      # Provider pricing is denominated per 1M tokens; divide here to get
+      # the dollar cost for the actual usage count. Named constant for
+      # consistency with how RubyLLM and provider docs express prices.
+      TOKENS_PER_MILLION = 1_000_000.0
+
       def self.token_cost(tokens, price_per_million)
-        (tokens || 0) * (price_per_million || 0) / 1_000_000.0
+        (tokens || 0) * (price_per_million || 0) / TOKENS_PER_MILLION
       end
 
       def self.find_model(model_name)
@@ -111,7 +116,11 @@ module RubyLLM
         end
       end
 
-      private_class_method :compute_cost, :token_cost, :find_model, :validate_price!
+      # `find_model` is intentionally public: `Step::Base#estimate_cost` needs
+      # to inspect model pricing before invoking `calculate` (e.g., to short-
+      # circuit estimate when the model is unknown). Exposing it removes
+      # `CostCalculator.send(:find_model)` workarounds at call sites.
+      private_class_method :compute_cost, :token_cost, :validate_price!
     end
   end
 end

data/lib/ruby_llm/contract/eval/recommender.rb

@@ -4,7 +4,9 @@ module RubyLLM
   module Contract
     module Eval
       class Recommender
-        def initialize(comparison:, min_score:, min_first_try_pass_rate: 0.8, current_config: nil)
+        def initialize(comparison:, min_score:,
+                       min_first_try_pass_rate: DEFAULT_MIN_FIRST_TRY_PASS_RATE,
+                       current_config: nil)
           @comparison = comparison
           @min_score = min_score
           @min_first_try_pass_rate = min_first_try_pass_rate

data/lib/ruby_llm/contract/eval/retry_optimizer.rb

@@ -98,7 +98,9 @@ module RubyLLM
           end
         end
 
-        def initialize(step:, candidates:, context: {}, min_score: 0.95, runs: 1, production_mode: nil)
+        def initialize(step:, candidates:, context: {},
+                       min_score: DEFAULT_MIN_SCORE,
+                       runs: 1, production_mode: nil)
           @step = step
           @candidates = candidates
           @context = context
@@ -113,10 +115,19 @@ module RubyLLM
 
           score_matrix = {}
           evals.each do |eval_name|
-            comparison = with_retry_disabled do
-              @step.compare_models(eval_name, candidates: @candidates, context: @context,
-                                              runs: @runs, production_mode: @production_mode)
-            end
+            # `retry_policy_override: nil` in context disables the step's
+            # class-level retry policy for this comparison run — see
+            # step/base.rb#runtime_settings, which honours the key when
+            # present (even when value is nil). Replaces the prior
+            # `define_singleton_method(:retry_policy)` mutation, which was
+            # not thread-safe across concurrent optimizer calls.
+            comparison = @step.compare_models(
+              eval_name,
+              candidates: @candidates,
+              context: @context.merge(retry_policy_override: nil),
+              runs: @runs,
+              production_mode: @production_mode
+            )
             score_matrix[eval_name] = extract_scores(comparison)
           end
 
@@ -203,14 +214,6 @@ module RubyLLM
           end
         end
 
-        def with_retry_disabled(&block)
-          original = @step.retry_policy if @step.respond_to?(:retry_policy)
-          @step.define_singleton_method(:retry_policy) { nil }
-          block.call
-        ensure
-          @step.define_singleton_method(:retry_policy) { original }
-        end
-
         def empty_result(evals)
           Result.new(
             step_name: @step.name || @step.to_s,

data/lib/ruby_llm/contract/eval.rb

@@ -33,3 +33,16 @@ require_relative "eval/eval_history"
 require_relative "eval/recommendation"
 require_relative "eval/recommender"
 require_relative "eval/retry_optimizer"
+
+module RubyLLM
+  module Contract
+    module Eval
+      # Default thresholds shared across `recommend`, `optimize_retry_policy`,
+      # `Recommender`, and `RetryOptimizer`. Centralised so a single change in
+      # what "viable" means (e.g. tightening from 0.95 to 0.97) propagates
+      # everywhere instead of needing the same edit in 4 places.
+      DEFAULT_MIN_SCORE = 0.95
+      DEFAULT_MIN_FIRST_TRY_PASS_RATE = 0.8
+    end
+  end
+end