ruby_llm-contract 0.5.2 → 0.6.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 359d08f8cf1e31b84f308c47c7f93c7cee7663054de3ab538a34c1f67873554f
-  data.tar.gz: 60d8728bed042277d40ec1d231b6712e258b658fd893a73afc6ed1f8e9cff8c8
+  metadata.gz: 2636c2e59f5fef27f929a94ac9e3194793ce6b51f86cce0c18ca6a5b0caa61ab
+  data.tar.gz: c2c81c7cc8fd281bf6c88738f0fb5bc4bdbb86b71d2fb59248e2b0ebb8d648fe
 SHA512:
-  metadata.gz: 4bd4d7cea9fde7281bf84e1283c4201f8c5e9425cb8357e40b85e5184f19f51eb57a88a35901eddf571defd93ff33ef790e24b5e2eb90add8ef6371e791d37e5
-  data.tar.gz: e68ca27fc2225224cd900b1afb2180cfd43929e0461420c7fd2987706a2ebaa282b1e659c8b5c14e69e30d1250ede547061e2d2ab74b5c9cc0bb7fdb77109f0a
+  metadata.gz: d9f2fca592fd3a183d987239dea0cdc2456eed639d6cccaea71e9b1ef3a3ff6e32f3e346f640c905168674e7401ccb85e5f342815a1b290cdebe05a9f7b5374f
+  data.tar.gz: 00b8d113564871db19f88d9061276a0aee0295da7638974804782fda7929cf893a0004aadfeffc2f25f13d421935e0e9d710e553d8e56b7b5d0229f62edd129e

data/CHANGELOG.md

@@ -1,5 +1,52 @@
 # Changelog
 
+## 0.6.2 (2026-04-18)
+
+### Features
+
+- **`Step.optimize_retry_policy`** — runs `compare_models` on ALL evals for the step, builds a score matrix, identifies the constraining eval, and suggests a retry chain. Chain's last model always passes all evals (safe fallback).
+- **`rake ruby_llm_contract:optimize`** — one-command retry chain optimization. Prints score table, constraining eval, suggested chain, and copy-paste DSL.
+- **Offline by default** — `optimize` uses `sample_response` (zero API calls) unless `LIVE=1` or `PROVIDER=` is set.
+- **`EVAL_DIRS=` support** — non-Rails setups can specify eval file directories.
+- **Guide: [Optimizing retry_policy](docs/guide/optimizing_retry_policy.md)** — full procedure with prerequisites, troubleshooting, and real-world example.
+
+### Fixes
+
+- Chain semantics aligned with `retry_executor` — retry fires on `validation_failed`/`parse_error`, not on low eval score. Disjoint eval coverage (A passes e1, B passes e2, neither passes both) correctly returns empty chain.
+- Removed ActiveSupport dependency from rake task (`.presence` → `.empty?`).
+- Added `require "set"` for non-Rails environments.
+
+## 0.6.1 (2026-04-17)
+
+### Features
+
+- **Multi-provider operator tooling** — rake tasks support `PROVIDER=openai|anthropic|ollama`, `CANDIDATES=model@effort,...`, and `REASONING_EFFORT=low|medium|high`.
+- **`rake ruby_llm_contract:recommend`** — wraps `Step.recommend` with CLI interface, prints best config, retry chain, DSL, rationale, and savings.
+- **Ollama support** — `PROVIDER=ollama` with configurable `OLLAMA_API_BASE`.
+
+## 0.6.0 (2026-04-12)
+
+"What should I do?" — model + configuration recommendation.
+
+### Features
+
+- **`Step.recommend`** — `ClassifyTicket.recommend("eval", candidates: [...], min_score: 0.95)` runs eval on all candidates and returns a `Recommendation` with optimal model, retry chain, rationale, savings vs current config, and `to_dsl` code output.
+- **Candidates as configurations** — `candidates:` accepts `{ model:, reasoning_effort: }` hashes, not just model name strings. `gpt-5-mini` with `reasoning_effort: "low"` is a different candidate than with `"high"`.
+- **`compare_models` extended** — new `candidates:` parameter alongside existing `models:` (backward compatible). Candidate labels include reasoning effort in output table.
+- **Per-attempt `reasoning_effort` in retry policies** — `escalate` accepts config hashes: `escalate({ model: "gpt-4.1-nano" }, { model: "gpt-5-mini", reasoning_effort: "high" })`. Each attempt gets its own reasoning_effort forwarded to the provider.
+- **`pass_rate_ratio`** — numeric float (0.0–1.0) on `Report` and `ReportStats`, complementing the string `pass_rate` (`"3/5"`).
+- **History entries enriched** — `save_history!` accepts `reasoning_effort:` and stores `model`, `reasoning_effort`, `pass_rate_ratio` in JSONL entries.
+
+### Game changer continuity
+
+```
+v0.2: "Which model?"          → compare_models (snapshot)
+v0.3: "Did it change?"        → baseline regression (binary)
+v0.4: "Show me the trend"     → eval history (time series)
+v0.5: "Which prompt is better?" → compare_with (A/B testing)
+v0.6: "What should I do?"     → recommend (actionable advice)
+```
+
 ## 0.5.2 (2026-04-06)
 
 ### Features
@@ -8,7 +55,7 @@
 
 ## 0.5.0 (2026-03-25)
 
-Data-Driven Prompt Engineering — see ADR-0015.
+Data-Driven Prompt Engineering.
 
 ### Features
 
@@ -55,7 +102,7 @@ Audit hardening — 18 bugs fixed across 4 audit rounds.
 
 ## 0.4.3 (2026-03-24)
 
-Production feedback release — driven by ADR-0016 (real Rails 8.1 deployment).
+Production feedback release.
 
 ### Features
 

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ruby_llm-contract (0.5.2)
+    ruby_llm-contract (0.6.1)
       dry-types (~> 1.7)
       ruby_llm (~> 1.0)
       ruby_llm-schema (~> 0.3)
@@ -258,7 +258,7 @@ CHECKSUMS
   rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   ruby_llm (1.14.0) sha256=57c6f7034fc4a44504ea137d70f853b07824f1c1cdbe774ab3ab3522e7098deb
-  ruby_llm-contract (0.5.2)
+  ruby_llm-contract (0.6.1)
   ruby_llm-schema (0.3.0) sha256=a591edc5ca1b7f0304f0e2261de61ba4b3bea17be09f5cf7558153adfda3dec6
   ruby_parser (3.22.0) sha256=1eb4937cd9eb220aa2d194e352a24dba90aef00751e24c8dfffdb14000f15d23
   rubycritic (4.12.0) sha256=024fed90fe656fa939f6ea80aab17569699ac3863d0b52fd72cb99892247abc8

data/README.md

@@ -1,24 +1,89 @@
 # ruby_llm-contract
 
-Contracts for LLM quality. Know which model to use, what it costs, and when accuracy drops.
+The missing link between LLM cost and quality. Stop choosing between "cheap but wrong" and "accurate but expensive" — get both. Contracts, model escalation, and data-driven recommendations for [ruby_llm](https://github.com/crmne/ruby_llm).
 
-Companion gem for [ruby_llm](https://github.com/crmne/ruby_llm).
+```
+  YOU WRITE                       THE GEM HANDLES                 YOU GET
+  ─────────                       ───────────────                 ───────
+
+  validate { |o| ... }            catch bad answers — combined     Zero garbage
+                                  with retry_policy, auto-retry   in production
 
-## The problem
+  retry_policy                    start cheap, escalate only      Pay for the cheapest
+  models: %w[nano mini full]      when validation fails           model that works
 
-Which model should you use? The expensive one is accurate but costs 4x more. The cheap one is fast but hallucinates on edge cases. You tweak a prompt — did accuracy improve or drop? You have no data. Just gut feeling.
+  max_cost 0.01                   estimate tokens, check price,   No surprise bills
+                                  refuse before calling LLM
 
-## The fix
+  output_schema { ... }           send JSON schema to provider,   Zero parsing code
+                                  validate response client-side
+
+  define_eval { ... }             test cases + baselines,          Regressions caught
+                                  run in CI with real LLM          before deploy
+
+  recommend(candidates: [...])    evaluate all configs, pick      Optimal model +
+                                  cheapest that passes            retry chain
+```
+
+## Before and after
+
+```
+  ┌─────────────────────────────────────────────────────────────────┐
+  │ BEFORE: pick one model, hope for the best                      │
+  │                                                                 │
+  │   expensive model → accurate, but you overpay on every call     │
+  │   cheap model     → fast, but wrong answers slip to production  │
+  │   prompt change   → "looks good to me" → deploy → users suffer │
+  └─────────────────────────────────────────────────────────────────┘
+
+                         ⬇  add ruby_llm-contract
+
+  ┌─────────────────────────────────────────────────────────────────┐
+  │ YOU DEFINE A CONTRACT                                            │
+  │                                                                 │
+  │   output_schema { string :priority }       ← valid structure   │
+  │   validate("valid priority") { |o| ... }   ← business rules    │
+  │   retry_policy models: %w[nano mini full]  ← escalation chain  │
+  │   max_cost 0.01                            ← budget cap         │
+  └───────────────────────────┬─────────────────────────────────────┘
+                              │
+                              ▼
+  ┌─────────────────────────────────────────────────────────────────┐
+  │ THE GEM HANDLES THE REST                                        │
+  │                                                                 │
+  │   request ──→ ┌──────┐   ┌──────────┐                           │
+  │               │ nano │─→ │ contract │──→ ✓ pass → done         │
+  │               └──────┘   └────┬─────┘                           │
+  │                               │ ✗ fail                          │
+  │                               ▼                                 │
+  │               ┌──────┐   ┌──────────┐                           │
+  │               │ mini │─→ │ contract │──→ ✓ pass → done         │
+  │               └──────┘   └────┬─────┘                           │
+  │                               │ ✗ fail                          │
+  │                               ▼                                 │
+  │               ┌──────┐   ┌──────────┐                           │
+  │               │ full │─→ │ contract │──→ ✓ pass → done         │
+  │               └──────┘   └──────────┘                           │
+  └───────────────────────────┬─────────────────────────────────────┘
+                              │
+                              ▼
+  ┌─────────────────────────────────────────────────────────────────┐
+  │ YOU GET                                                         │
+  │                                                                 │
+  │   ✓ Valid output guaranteed — schema + business rules enforced  │
+  │   ✓ Cheapest model that works — most requests stay on nano     │
+  │   ✓ Cost, latency, tokens — tracked on every call              │
+  │   ✓ Eval scores per model — data instead of gut feeling        │
+  │   ✓ Regressions caught — before deploy, not after              │
+  │   ✓ Recommendation — "use nano+mini, drop full, save $X/mo"   │
+  └─────────────────────────────────────────────────────────────────┘
+```
+
+## 30-second version
 
 ```ruby
 class ClassifyTicket < RubyLLM::Contract::Step::Base
-  prompt do
-    system "You are a support ticket classifier."
-    rule "Return valid JSON only, no markdown."
-    rule "Use exactly one priority: low, medium, high, urgent."
-    example input: "My invoice is wrong", output: '{"priority": "high"}'
-    user "{input}"
-  end
+  prompt "Classify this support ticket by priority and category.\n\n{input}"
 
   output_schema do
     string :priority, enum: %w[low medium high urgent]
@@ -30,29 +95,45 @@ class ClassifyTicket < RubyLLM::Contract::Step::Base
 end
 
 result = ClassifyTicket.run("I was charged twice")
-result.ok?               # => true
-result.parsed_output     # => {priority: "high", category: "billing"}
-result.trace[:cost]      # => 0.000032
-result.trace[:model]     # => "gpt-4.1-nano"
+result.parsed_output  # => {priority: "high", category: "billing"}
+result.trace[:model]  # => "gpt-4.1-nano" (first model that passed)
+result.trace[:cost]   # => 0.000032
 ```
 
-Bad JSON? Auto-retry. Wrong value? Escalate to a smarter model. Schema violated? Caught client-side even if the provider ignores it. All with cost tracking.
+Bad JSON? Retried automatically. Wrong answer? Escalated to a smarter model. Schema violated? Caught client-side. The contract guarantees every response meets your rules — you pay for the cheapest model that passes.
 
-## Start Here: Eval-First
+## Install
 
-The most powerful way to use this gem is simple:
+```ruby
+gem "ruby_llm-contract"
+```
+
+```ruby
+RubyLLM.configure { |c| c.openai_api_key = ENV["OPENAI_API_KEY"] }
+RubyLLM::Contract.configure { |c| c.default_model = "gpt-4.1-mini" }
+```
+
+Works with any ruby_llm provider (OpenAI, Anthropic, Gemini, etc).
 
-- define evals before changing prompts
-- compare prompt versions on the same dataset
-- merge only when the eval stays green
+## Save money with model escalation
+
+Without a contract, you use gpt-4.1 for everything because you can't tell when a cheaper model gets it wrong. With a contract, you start on nano and only escalate when the answer fails the contract:
+
+```ruby
+retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]
+```
 
-Read: [Eval-First](docs/guide/eval_first.md)
+```
+Attempt 1: gpt-4.1-nano  → contract failed  ($0.0001)
+Attempt 2: gpt-4.1-mini  → contract passed  ($0.0004)
+           gpt-4.1       → never called      ($0.00)
+```
 
-This is the workflow that gives prompt engineering teeth. No vibes, no cherry-picked examples, no "it felt better in the playground". Just cases, regressions, baselines, and measured wins.
+Most requests succeed on the cheapest model. You pay full price only for the ones that need it. How many? Run `compare_models` and find out.
 
-## Which model should I use?
+## Know which model to use — with data
 
-Define test cases. Compare models. Get data.
+Don't guess. Define test cases, compare models, get numbers:
 
 ```ruby
 ClassifyTicket.define_eval("regression") do
@@ -62,170 +143,127 @@ ClassifyTicket.define_eval("regression") do
 end
 
 comparison = ClassifyTicket.compare_models("regression",
-  models: %w[gpt-4.1-nano gpt-4.1-mini])
+  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
 ```
 
-Real output from real API calls:
-
 ```
-Model                      Score       Cost  Avg Latency
+Candidate                  Score       Cost  Avg Latency
 ---------------------------------------------------------
-gpt-4.1-nano                0.67    $0.000032      687ms
-gpt-4.1-mini                1.00    $0.000102     1070ms
+gpt-4.1-nano                0.67    $0.0001         48ms
+gpt-4.1-mini                1.00    $0.0004         92ms
+gpt-4.1                     1.00    $0.0021        210ms
 
 Cheapest at 100%: gpt-4.1-mini
 ```
 
-```ruby
-comparison.best_for(min_score: 0.95)  # => "gpt-4.1-mini"
-
-# Inspect failures
-comparison.reports["gpt-4.1-nano"].failures.each do |f|
-  puts "#{f.name}: expected #{f.expected}, got #{f.output}"
-  puts "  mismatches: #{f.mismatches}"
-  # => outage: expected {priority: "urgent"}, got {priority: "high"}
-  #      mismatches: {priority: {expected: "urgent", got: "high"}}
-end
-```
+Nano fails on edge cases. Mini and full both score 100% — but mini is **5x cheaper**. Now you know.
 
-## Pipeline
+## Let the gem tell you what to do
 
-Chain steps with fail-fast. Hallucination in step 1 stops before step 2 spends tokens.
+Don't read tables — get a recommendation. Supports `model + reasoning_effort` combinations:
 
 ```ruby
-class TicketPipeline < RubyLLM::Contract::Pipeline::Base
-  step ClassifyTicket,  as: :classify
-  step RouteToTeam,     as: :route
-  step DraftResponse,   as: :draft
-end
-
-result = TicketPipeline.run("I was charged twice")
-result.ok?                          # => true
-result.outputs_by_step[:classify]   # => {priority: "high", category: "billing"}
-result.trace.total_cost             # => 0.000128
+rec = ClassifyTicket.recommend("regression",
+  candidates: [
+    { model: "gpt-4.1-nano" },
+    { model: "gpt-4.1-mini" },
+    { model: "gpt-5-mini", reasoning_effort: "low" },
+    { model: "gpt-5-mini", reasoning_effort: "high" },
+  ],
+  min_score: 0.95
+)
+
+rec.best           # => { model: "gpt-4.1-mini" }
+rec.retry_chain    # => [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }]
+rec.to_dsl         # => "retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini]"
+rec.savings        # => savings vs your current model (if configured)
 ```
 
-## CI gate
-
-```ruby
-# RSpec — block merge if accuracy drops or cost spikes
-expect(ClassifyTicket).to pass_eval("regression")
-  .with_context(model: "gpt-4.1-mini")
-  .with_minimum_score(0.8)
-  .with_maximum_cost(0.01)
-
-# Rake — run all evals across all steps
-require "ruby_llm/contract/rake_task"
-RubyLLM::Contract::RakeTask.new do |t|
-  t.minimum_score = 0.8
-  t.maximum_cost = 0.05
-end
-# bundle exec rake ruby_llm_contract:eval
-```
+Copy `rec.to_dsl` into your step. Done.
 
-## Detect quality drops
+## Catch regressions before users do
 
-Save a baseline. Next run, see what regressed.
+A model update silently dropped your accuracy? A prompt tweak broke an edge case? You'll know before deploying:
 
 ```ruby
+# Save a baseline once:
 report = ClassifyTicket.run_eval("regression", context: { model: "gpt-4.1-nano" })
 report.save_baseline!(model: "gpt-4.1-nano")
 
-# Later — after prompt change, model update, or provider weight shift:
-report = ClassifyTicket.run_eval("regression", context: { model: "gpt-4.1-nano" })
-diff = report.compare_with_baseline(model: "gpt-4.1-nano")
-
-diff.regressed?    # => true
-diff.regressions   # => [{case: "outage", baseline: {passed: true}, current: {passed: false}}]
-diff.score_delta   # => -0.33
-```
-
-```ruby
-# CI: block merge if any previously-passing case now fails
+# In CI — block merge if anything regressed:
 expect(ClassifyTicket).to pass_eval("regression")
   .with_context(model: "gpt-4.1-nano")
   .without_regressions
 ```
 
-## Track quality over time
-
 ```ruby
-# Save every eval run
-report = ClassifyTicket.run_eval("regression", context: { model: "gpt-4.1-nano" })
-report.save_history!(model: "gpt-4.1-nano")
-
-# View trend
-history = report.eval_history(model: "gpt-4.1-nano")
-history.score_trend   # => :stable_or_improving | :declining
-history.drift?        # => true (score dropped > 10%)
+diff = report.compare_with_baseline(model: "gpt-4.1-nano")
+diff.regressed?    # => true
+diff.regressions   # => [{case: "outage", baseline: {passed: true}, current: {passed: false}}]
+diff.score_delta   # => -0.33
 ```
 
-## Run evals fast
-
-```ruby
-# 4x faster with parallel execution
-report = ClassifyTicket.run_eval("regression",
-  context: { model: "gpt-4.1-nano" },
-  concurrency: 4)
-```
+No more "it worked in the playground". Regressions are caught in CI, not production.
 
-## Prompt A/B testing
+## A/B test your prompts
 
-Changed a prompt? Compare old vs new with regression safety:
+Changed a prompt? Compare old vs new on the same dataset with regression safety:
 
 ```ruby
 diff = ClassifyTicketV2.compare_with(ClassifyTicketV1,
   eval: "regression", model: "gpt-4.1-mini")
 
-diff.safe_to_switch?  # => true (no regressions, no per-case score drops)
+diff.safe_to_switch?  # => true (no regressions)
 diff.improvements     # => [{case: "outage", ...}]
 diff.score_delta      # => +0.33
 ```
 
-Requires `model:` or `context: { adapter: ... }`.
-`compare_with` ignores `sample_response`; without a real model/adapter both sides are skipped and the A/B result is not meaningful.
-
-CI gate:
 ```ruby
+# CI gate:
 expect(ClassifyTicketV2).to pass_eval("regression")
   .compared_with(ClassifyTicketV1)
   .with_minimum_score(0.8)
 ```
 
-## Soft observations
+## Chain steps with fail-fast
 
-Log suspicious-but-not-invalid output without failing the contract:
+Pipeline stops at the first contract failure — no wasted tokens on downstream steps:
 
 ```ruby
-class EvaluateComparative < RubyLLM::Contract::Step::Base
-  validate("scores in range") { |o| (1..10).include?(o[:score_a]) }
-  observe("scores should differ") { |o| o[:score_a] != o[:score_b] }
+class TicketPipeline < RubyLLM::Contract::Pipeline::Base
+  step ClassifyTicket,  as: :classify
+  step RouteToTeam,     as: :route
+  step DraftResponse,   as: :draft
 end
 
-result = EvaluateComparative.run(input)
-result.ok?            # => true (observe never fails)
-result.observations   # => [{description: "scores should differ", passed: false}]
+result = TicketPipeline.run("I was charged twice")
+result.outputs_by_step[:classify]   # => {priority: "high", category: "billing"}
+result.trace.total_cost             # => $0.000128
 ```
 
-## Predict cost before running
+## Gate merges on quality and cost
 
 ```ruby
-ClassifyTicket.estimate_eval_cost("regression", models: %w[gpt-4.1-nano gpt-4.1-mini])
-# => { "gpt-4.1-nano" => 0.000024, "gpt-4.1-mini" => 0.000096 }
+# RSpec — block merge if accuracy drops or cost spikes
+expect(ClassifyTicket).to pass_eval("regression")
+  .with_minimum_score(0.8)
+  .with_maximum_cost(0.01)
+
+# Rake — run all evals across all steps
+RubyLLM::Contract::RakeTask.new do |t|
+  t.minimum_score = 0.8
+  t.maximum_cost = 0.05
+end
+# bundle exec rake ruby_llm_contract:eval
 ```
 
-## Install
+## Full power: data-driven retry chains
 
-```ruby
-gem "ruby_llm-contract"
-```
+The pieces above — evals, compare_models, recommend — combine into a workflow that replaces guesswork with measured optimization. You define evals for your step, run `recommend` against all of them, find the eval that actually needs the strongest model, and build a retry chain where each attempt is as cheap as the data allows.
 
-```ruby
-RubyLLM.configure { |c| c.openai_api_key = ENV["OPENAI_API_KEY"] }
-RubyLLM::Contract.configure { |c| c.default_model = "gpt-4.1-mini" }
-```
+The difference: instead of "gpt-5-mini seems to work, let's use it everywhere", you get "nano handles 4/6 scenarios, mini@low catches the 5th, full mini only fires on the hardest edge case — first attempt is 4× cheaper."
 
-Works with any ruby_llm provider (OpenAI, Anthropic, Gemini, etc).
+Full procedure with examples: **[Optimizing retry_policy](docs/guide/optimizing_retry_policy.md)**
 
 ## Docs
 
@@ -233,6 +271,7 @@ Works with any ruby_llm provider (OpenAI, Anthropic, Gemini, etc).
 |-------|-|
 | [Getting Started](docs/guide/getting_started.md) | Features walkthrough, model escalation, eval |
 | [Eval-First](docs/guide/eval_first.md) | Practical workflow for prompt engineering with datasets, baselines, and A/B gates |
+| [Optimizing retry_policy](docs/guide/optimizing_retry_policy.md) | Find the cheapest retry chain that passes all your evals |
 | [Best Practices](docs/guide/best_practices.md) | 6 patterns for bulletproof validates |
 | [Output Schema](docs/guide/output_schema.md) | Full schema reference + constraints |
 | [Pipeline](docs/guide/pipeline.md) | Multi-step composition, timeout, fail-fast |
@@ -241,13 +280,13 @@ Works with any ruby_llm provider (OpenAI, Anthropic, Gemini, etc).
 
 ## Roadmap
 
-**v0.5 (current):** Data-driven prompt engineering — `compare_with(OtherStep)` for prompt A/B testing with regression safety. `observe` DSL for soft observations that log but never fail.
+**v0.6 (current):** "What should I do?" — `Step.recommend` returns optimal model, reasoning effort, and retry chain. Per-attempt `reasoning_effort` in retry policies.
 
-**v0.4:** Observability & scale — eval history with trending, batch eval with concurrency, pipeline per-step eval, Minitest support, structured logging. Audit hardening (18 bugfixes).
+**v0.5:** Prompt A/B testing with `compare_with`. Soft observations with `observe`.
 
-**v0.3:** Baseline regression detection, migration guide, production hardening.
+**v0.4:** Eval history, batch concurrency, pipeline per-step eval, Minitest, structured logging.
 
-**v0.6:** Model recommendation based on eval history data. Cross-provider comparison docs.
+**v0.3:** Baseline regression detection, migration guide.
 
 ## License
 

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -70,18 +70,37 @@ module RubyLLM
           Eval::PromptDiff.new(candidate: my_report, baseline: other_report)
         end
 
-        def compare_models(eval_name, models:, context: {})
+        def compare_models(eval_name, models: [], candidates: [], context: {})
+          raise ArgumentError, "Pass either models: or candidates:, not both" if models.any? && candidates.any?
+
           context = safe_context(context)
-          models = models.uniq
-          reports = models.each_with_object({}) do |model, hash|
-            model_context = isolate_context(context).merge(model: model)
-            hash[model] = run_single_eval(eval_name, model_context)
+          candidate_configs = normalize_candidates(models, candidates)
+
+          reports = {}
+          configs = {}
+          candidate_configs.each do |config|
+            label = Eval::ModelComparison.candidate_label(config)
+            model_context = isolate_context(context).merge(model: config[:model])
+            model_context[:reasoning_effort] = config[:reasoning_effort] if config[:reasoning_effort]
+            reports[label] = run_single_eval(eval_name, model_context)
+            configs[label] = config
           end
-          Eval::ModelComparison.new(eval_name: eval_name, reports: reports)
+
+          Eval::ModelComparison.new(eval_name: eval_name, reports: reports, configs: configs)
         end
 
         private
 
+        def normalize_candidates(models, candidates)
+          if candidates.any?
+            candidates.map { |c| RubyLLM::Contract.normalize_candidate_config(c) }.uniq
+          elsif models.any?
+            models.uniq.map { |m| { model: m } }
+          else
+            raise ArgumentError, "Pass models: or candidates: with at least one entry"
+          end
+        end
+
         def comparison_context(context, model)
           base_context = safe_context(context)
           model ? base_context.merge(model: model) : base_context

data/lib/ruby_llm/contract/eval/model_comparison.rb

@@ -4,11 +4,17 @@ module RubyLLM
   module Contract
     module Eval
       class ModelComparison
-        attr_reader :eval_name, :reports
+        attr_reader :eval_name, :reports, :configs
 
-        def initialize(eval_name:, reports:)
+        def self.candidate_label(config)
+          effort = config[:reasoning_effort]
+          effort ? "#{config[:model]} (effort: #{effort})" : config[:model]
+        end
+
+        def initialize(eval_name:, reports:, configs: nil)
           @eval_name = eval_name
-          @reports = reports.dup.freeze # { "model_name" => Report }
+          @reports = reports.dup.freeze
+          @configs = (configs || default_configs_from_reports).freeze
           freeze
         end
 
@@ -16,12 +22,12 @@ module RubyLLM
           @reports.keys
         end
 
-        def score_for(model)
-          @reports[model]&.score
+        def score_for(candidate)
+          @reports[resolve_key(candidate)]&.score
         end
 
-        def cost_for(model)
-          @reports[model]&.total_cost
+        def cost_for(candidate)
+          @reports[resolve_key(candidate)]&.total_cost
         end
 
         def best_for(min_score: 0.0)
@@ -38,13 +44,14 @@ module RubyLLM
         end
 
         def table
-          lines = ["  Model                      Score       Cost  Avg Latency"]
-          lines << "  #{"-" * 57}"
+          max_label = [@reports.keys.map(&:length).max || 0, 25].max
+          lines = [format("  %-#{max_label}s  Score       Cost  Avg Latency", "Candidate")]
+          lines << "  #{"-" * (max_label + 36)}"
 
-          @reports.each do |model, report|
+          @reports.each do |label, report|
             latency = report.avg_latency_ms ? "#{report.avg_latency_ms.round}ms" : "n/a"
             cost = report.total_cost.positive? ? "$#{format("%.4f", report.total_cost)}" : "n/a"
-            lines << format("  %-25s %6.2f %10s %12s", model, report.score, cost, latency)
+            lines << format("  %-#{max_label}s %6.2f %10s %12s", label, report.score, cost, latency)
           end
 
           lines.join("\n")
@@ -70,10 +77,25 @@ module RubyLLM
               total_cost: report.total_cost,
               avg_latency_ms: report.avg_latency_ms,
               pass_rate: report.pass_rate,
+              pass_rate_ratio: report.pass_rate_ratio,
               passed: report.passed?
             }
           end
         end
+
+        private
+
+        def resolve_key(candidate)
+          case candidate
+          when String then candidate
+          when Hash then self.class.candidate_label(candidate)
+          else candidate.to_s
+          end
+        end
+
+        def default_configs_from_reports
+          @reports.each_with_object({}) { |(key, _), h| h[key] = { model: key } }
+        end
       end
     end
   end