ruby_llm-contract 0.6.3 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 75884386ae53ddf1985760afa2c94f64ce15274b0cb0829ea40e6711506e60cf
-  data.tar.gz: 203229c4ce8ab0b1ab9e6209871668b0713b114e412eba03e79e9964dc9a43bb
+  metadata.gz: 5decfb7456338baa05d0e6bb79287bb3fb3af0e0cc2c3f001e09122fa76ac298
+  data.tar.gz: be3b46ff015fd651e885c4c078dbdaf06623865f4e4b80b864b66e8dd9e16c34
 SHA512:
-  metadata.gz: 0c586ce70e71d8e77c262e0ae21bab2e29ef6ccdb16df4a0e56271aeb98efba59027618db24d4b59470fd3a9d340051201584038f8c67143f7ca1bec0e5e365b
-  data.tar.gz: 6a988c62f6b36a4da860b736c5523abcc568ed1384c1f20cd8ff9659124c12eeeddb36100d5b8542af07b363cef7ade8a1aca01250929ec2c23172d4f3faec5f
+  metadata.gz: fd3882613ac2b500c46dc6b1084d8f298db96800bf01932e3bc2638a7a3d2a8588610c1de8a1f3754a8e15fb7b1ef29c0c0a7bddd11709cd95bbf12fcd48e83e
+  data.tar.gz: 4cb584323a5575de4131b0eb82cdb1426743ca6651030708673d17264c9fec60619bb7d5d54f62eee4c1fa357b4022f57bf08b95d81f153eccf8e625ce3ef5e7

data/.rubocop.yml

@@ -42,14 +42,17 @@ AllCops:
     - 'internal/**/*'
 
 Metrics/ClassLength:
-  Max: 130
+  Max: 140
+
+Metrics/ModuleLength:
+  Max: 150
 
 Metrics/AbcSize:
   Max: 30
 
 Metrics/ParameterLists:
-  Max: 11
-  MaxOptionalParameters: 9
+  Max: 12
+  MaxOptionalParameters: 10
 
 Style/FormatStringToken:
   Enabled: false

data/CHANGELOG.md

@@ -1,5 +1,51 @@
 # Changelog
 
+## 0.7.0 (2026-04-21)
+
+### Breaking changes
+
+- **`:adapter_error` removed from `DEFAULT_RETRY_ON`.** New default: `[:validation_failed, :parse_error]`. `ruby_llm` already retries transport errors (`RateLimitError`, `ServerError`, `ServiceUnavailableError`, `OverloadedError`, timeouts) at the Faraday layer, so the previous default re-ran the same model on errors the HTTP middleware already retried with backoff. To restore pre-0.7 behavior: `retry_on :validation_failed, :parse_error, :adapter_error`. Recommended pattern: pair `:adapter_error` with `escalate "model_a", "model_b"` — a different model/provider can bypass what transport retry could not.
+- **`AdapterCaller` narrows `rescue` from `StandardError` to `RubyLLM::Error` + `Faraday::Error`.** Provider errors and transport errors that escape ruby_llm's Faraday retry middleware (`Faraday::TimeoutError`, `Faraday::ConnectionFailed`) still produce `:adapter_error` as before. Programmer errors that are neither (`NoMethodError`, adapter code bugs) now propagate instead of being silently converted to `:adapter_error` and retried. **Known limitation:** adapter code raising `ArgumentError` is still coerced into `:input_error` by `Step::Base#run_once` (which rescues `ArgumentError` for input-type validation). Disambiguating adapter-ArgumentError vs input-validation-ArgumentError requires a `run_once` refactor and is tracked as a follow-up.
+
+### Migration
+
+If you rely on the old behavior, opt in explicitly:
+
+```ruby
+retry_policy do
+  attempts 3
+  retry_on :validation_failed, :parse_error, :adapter_error
+end
+```
+
+Or better, with a model fallback chain:
+
+```ruby
+retry_policy do
+  escalate "gpt-4.1-nano", "gpt-4.1-mini"
+  retry_on :validation_failed, :parse_error, :adapter_error
+end
+```
+
+## 0.6.4 (2026-04-20)
+
+### Features
+
+- **`production_mode:` on `compare_models` and `optimize_retry_policy`** — measures retry-aware, end-to-end cost per successful output. Pass `production_mode: { fallback: "gpt-5-mini" }` and each candidate runs with a runtime-injected `[candidate, fallback]` retry chain. The report exposes `escalation_rate`, `single_shot_cost`, and `effective_cost` so "the cheaper candidate" decision matches production cost rather than first-attempt cost.
+- **New Report metrics** — `escalation_rate`, `single_shot_cost`, `effective_cost`, `single_shot_latency_ms`, `effective_latency_ms`, `latency_percentiles` (p50/p95/max). `AggregatedReport` averages all of them across `runs:`.
+- **Extended `ModelComparison#table`** — when `production_mode:` is set, renders a `Chain` column (`candidate → fallback`) with `single-shot`, `escalation`, `effective cost`, `latency`, `score`. Edge case `candidate == fallback` renders as a single model and `—` in the escalation column, with retry injection skipped entirely so `effective == single-shot` by construction, not by coincidence.
+- **`context[:retry_policy_override]`** — new context key that nullifies or replaces class-level `retry_policy` for a single call. Used internally by production-mode injection; safe to use directly when you need a transient override that doesn't mutate the step class.
+
+### Scope
+
+- Single-fallback (2-tier) chains only. Multi-tier chains can be inspected post-hoc via `trace.attempts` but aren't summarized in the optimize table.
+- Costs with `runs: 3 + production_mode: { fallback: "gpt-5-mini" }` are ≈3× a single-shot eval plus the actual retry attempts — not 6×. Production-mode metrics come from a single pass.
+- **Step-only.** Calling `compare_models` with `production_mode:` on a `Pipeline::Base` subclass raises `ArgumentError` — retry injection is Step-level and pipeline-wide fallback semantics aren't defined yet. Benchmark individual steps.
+
+### Documentation
+
+- **Guide: [Production-mode cost measurement](docs/guide/optimizing_retry_policy.md#production-mode-cost-measurement)** — API, metric interpretation, 2-tier scope note.
+
 ## 0.6.3 (2026-04-20)
 
 ### Features

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ruby_llm-contract (0.6.3)
+    ruby_llm-contract (0.7.0)
       dry-types (~> 1.7)
       ruby_llm (~> 1.0)
       ruby_llm-schema (~> 0.3)
@@ -258,7 +258,7 @@ CHECKSUMS
   rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   ruby_llm (1.14.0) sha256=57c6f7034fc4a44504ea137d70f853b07824f1c1cdbe774ab3ab3522e7098deb
-  ruby_llm-contract (0.6.3)
+  ruby_llm-contract (0.7.0)
   ruby_llm-schema (0.3.0) sha256=a591edc5ca1b7f0304f0e2261de61ba4b3bea17be09f5cf7558153adfda3dec6
   ruby_parser (3.22.0) sha256=1eb4937cd9eb220aa2d194e352a24dba90aef00751e24c8dfffdb14000f15d23
   rubycritic (4.12.0) sha256=024fed90fe656fa939f6ea80aab17569699ac3863d0b52fd72cb99892247abc8

data/README.md

@@ -160,6 +160,8 @@ Nano fails on edge cases. Mini and full both score 100% — but mini is **5x che
 
 Running live against gpt-5 / o-series? Pass `runs: 3` to average out sampling variance (OpenAI forces `temperature=1.0` server-side, so one unlucky run can misclassify a viable candidate). See [Reducing variance with `runs:`](docs/guide/optimizing_retry_policy.md#reducing-variance-with-runs).
 
+Want the *effective* cost — first-attempt plus retries — rather than the single-shot headline number? Pass `production_mode: { fallback: "gpt-5-mini" }` and the table gains `escalation`, `effective cost`, and a `Chain` column. See [Production-mode cost measurement](docs/guide/optimizing_retry_policy.md#production-mode-cost-measurement).
+
 ## Let the gem tell you what to do
 
 Don't read tables — get a recommendation. Supports `model + reasoning_effort` combinations:

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -5,6 +5,7 @@ module RubyLLM
     module Concerns
       module EvalHost
         include ContextHelpers
+        include ProductionModeContext
 
         SAMPLE_RESPONSE_COMPARE_WARNING = "[ruby_llm-contract] compare_with ignores sample_response. " \
                                           "Without model: or context: { adapter: ... }, both sides will be skipped " \
@@ -70,26 +71,29 @@ module RubyLLM
           Eval::PromptDiff.new(candidate: my_report, baseline: other_report)
         end
 
-        def compare_models(eval_name, models: [], candidates: [], context: {}, runs: 1)
+        def compare_models(eval_name, models: [], candidates: [], context: {}, runs: 1, production_mode: nil)
           raise ArgumentError, "Pass either models: or candidates:, not both" if models.any? && candidates.any?
 
           runs = coerce_runs(runs)
 
           context = safe_context(context)
           candidate_configs = normalize_candidates(models, candidates)
+          reject_production_mode_on_pipeline!(production_mode)
+          fallback_config = normalize_production_mode(production_mode)
 
           reports = {}
           configs = {}
           candidate_configs.each do |config|
             label = Eval::ModelComparison.candidate_label(config)
-            model_context = isolate_context(context).merge(model: config[:model])
-            model_context[:reasoning_effort] = config[:reasoning_effort] if config[:reasoning_effort]
+            model_context = build_candidate_context(context, config, fallback_config)
             per_run = Array.new(runs) { run_single_eval(eval_name, model_context) }
             reports[label] = runs == 1 ? per_run.first : Eval::AggregatedReport.new(per_run)
             configs[label] = config
           end
 
-          Eval::ModelComparison.new(eval_name: eval_name, reports: reports, configs: configs)
+          Eval::ModelComparison.new(
+            eval_name: eval_name, reports: reports, configs: configs, fallback: fallback_config
+          )
         end
 
         private
@@ -101,6 +105,24 @@ module RubyLLM
           runs
         end
 
+        def reject_production_mode_on_pipeline!(production_mode)
+          return if production_mode.nil? || production_mode == false
+          return unless defined?(Pipeline::Base) && self < Pipeline::Base
+
+          raise ArgumentError,
+                "production_mode: is not supported on Pipeline (#{self}). Retry injection happens at Step level; " \
+                "call compare_models with production_mode: on individual Step classes instead."
+        end
+
+        def build_candidate_context(context, config, fallback_config)
+          model_context = isolate_context(context).merge(model: config[:model])
+          model_context[:reasoning_effort] = config[:reasoning_effort] if config[:reasoning_effort]
+          return model_context unless fallback_config
+
+          model_context[:retry_policy_override] = production_mode_override(config, fallback_config)
+          model_context
+        end
+
         def normalize_candidates(models, candidates)
           if candidates.any?
             candidates.map { |c| RubyLLM::Contract.normalize_candidate_config(c) }.uniq

data/lib/ruby_llm/contract/concerns/production_mode_context.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Concerns
+      # Helpers for injecting a retry_policy_override into per-candidate eval
+      # context when compare_models runs in production-mode. When candidate
+      # == fallback, retry injection is skipped so the row degenerates into
+      # a single-shot eval by construction.
+      module ProductionModeContext
+        private
+
+        def normalize_production_mode(production_mode)
+          return nil if production_mode.nil? || production_mode == false
+
+          unless production_mode.is_a?(Hash) && production_mode[:fallback]
+            raise ArgumentError, "production_mode: must be a Hash with :fallback, got #{production_mode.inspect}"
+          end
+
+          RubyLLM::Contract.normalize_candidate_config(production_mode[:fallback])
+        end
+
+        def production_mode_override(config, fallback_config)
+          return nil if same_candidate?(config, fallback_config)
+
+          Step::RetryPolicy.new(models: [config, fallback_config])
+        end
+
+        def same_candidate?(first, second)
+          first[:model] == second[:model] && first[:reasoning_effort] == second[:reasoning_effort]
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/eval/aggregated_report.rb

@@ -86,6 +86,49 @@ module RubyLLM
         def failures
           @runs.flat_map(&:failures)
         end
+
+        def production_mode?
+          @runs.any?(&:production_mode?)
+        end
+
+        def escalation_rate
+          values = @runs.filter_map(&:escalation_rate)
+          return nil if values.empty?
+
+          values.sum / values.length.to_f
+        end
+
+        def single_shot_cost
+          values = @runs.filter_map(&:single_shot_cost)
+          return nil if values.empty?
+
+          values.sum / values.length.to_f
+        end
+
+        def effective_cost
+          total_cost
+        end
+
+        def single_shot_latency_ms
+          values = @runs.filter_map(&:single_shot_latency_ms)
+          return nil if values.empty?
+
+          values.sum / values.length.to_f
+        end
+
+        def effective_latency_ms
+          avg_latency_ms
+        end
+
+        def latency_percentiles
+          per_run = @runs.filter_map(&:latency_percentiles)
+          return nil if per_run.empty?
+
+          %i[p50 p95 max].each_with_object({}) do |key, acc|
+            values = per_run.filter_map { |h| h[key] }
+            acc[key] = values.empty? ? nil : values.sum / values.length.to_f
+          end
+        end
       end
     end
   end

data/lib/ruby_llm/contract/eval/case_result.rb

@@ -5,10 +5,10 @@ module RubyLLM
     module Eval
       class CaseResult
         attr_reader :name, :input, :output, :expected, :step_status,
-                    :score, :details, :duration_ms, :cost
+                    :score, :details, :duration_ms, :cost, :attempts
 
         def initialize(name:, input:, output:, expected:, step_status:,
-                       score:, passed:, label: nil, details: nil, duration_ms: nil, cost: nil)
+                       score:, passed:, label: nil, details: nil, duration_ms: nil, cost: nil, attempts: nil)
           @name = name
           @input = input
           @output = output
@@ -20,6 +20,7 @@ module RubyLLM
           @details = details
           @duration_ms = duration_ms
           @cost = cost
+          @attempts = attempts
           freeze
         end
 
@@ -58,7 +59,8 @@ module RubyLLM
             label: label,
             details: @details,
             duration_ms: @duration_ms,
-            cost: @cost
+            cost: @cost,
+            attempts: @attempts
           }
         end
 

data/lib/ruby_llm/contract/eval/case_result_builder.rb

@@ -18,7 +18,8 @@ module RubyLLM
             label: evaluation.label,
             details: evaluation.details,
             duration_ms: trace_metric(trace, :total_latency_ms, :latency_ms),
-            cost: trace_metric(trace, :total_cost, :cost)
+            cost: trace_metric(trace, :total_cost, :cost),
+            attempts: trace_attempts(trace)
           )
         end
 
@@ -29,6 +30,12 @@ module RubyLLM
 
           trace.respond_to?(pipeline_key) ? trace.public_send(pipeline_key) : trace[step_key]
         end
+
+        def trace_attempts(trace)
+          return nil unless trace
+
+          trace.respond_to?(:attempts) ? trace.attempts : nil
+        end
       end
     end
   end

data/lib/ruby_llm/contract/eval/model_comparison.rb

@@ -4,20 +4,25 @@ module RubyLLM
   module Contract
     module Eval
       class ModelComparison
-        attr_reader :eval_name, :reports, :configs
+        attr_reader :eval_name, :reports, :configs, :fallback
 
         def self.candidate_label(config)
           effort = config[:reasoning_effort]
           effort ? "#{config[:model]} (effort: #{effort})" : config[:model]
         end
 
-        def initialize(eval_name:, reports:, configs: nil)
+        def initialize(eval_name:, reports:, configs: nil, fallback: nil)
           @eval_name = eval_name
           @reports = reports.dup.freeze
           @configs = (configs || default_configs_from_reports).freeze
+          @fallback = fallback
           freeze
         end
 
+        def production_mode?
+          !@fallback.nil?
+        end
+
         def models
           @reports.keys
         end
@@ -44,6 +49,8 @@ module RubyLLM
         end
 
         def table
+          return production_mode_table if production_mode?
+
           max_label = [@reports.keys.map(&:length).max || 0, 25].max
           lines = [format("  %-#{max_label}s  Score       Cost  Avg Latency", "Candidate")]
           lines << "  #{"-" * (max_label + 36)}"
@@ -57,6 +64,62 @@ module RubyLLM
           lines.join("\n")
         end
 
+        def production_mode_table
+          fallback_label = self.class.candidate_label(@fallback)
+          rows = @reports.map do |label, report|
+            chain = chain_label(label, fallback_label)
+            { chain: chain, report: report, same: chain_same_as_fallback?(label, fallback_label) }
+          end
+
+          chain_width = [rows.map { |r| r[:chain].length }.max || 0, 20].max
+          lines = [format("  %-#{chain_width}s  %-11s  %-10s  %-14s  %-9s  %s",
+                          "Chain", "single-shot", "escalation", "effective cost", "latency", "score")]
+          lines << "  #{"-" * (chain_width + 60)}"
+
+          rows.each do |row|
+            lines << format_production_row(row, chain_width)
+          end
+
+          lines.join("\n")
+        end
+
+        private
+
+        def chain_label(label, fallback_label)
+          label == fallback_label ? label : "#{label} → #{fallback_label}"
+        end
+
+        def chain_same_as_fallback?(label, fallback_label)
+          label == fallback_label
+        end
+
+        def format_production_row(row, chain_width)
+          report = row[:report]
+          format("  %-#{chain_width}s  %-11s  %-10s  %-14s  %-9s  %6.2f",
+                 row[:chain],
+                 format_money(report.single_shot_cost || report.total_cost),
+                 format_escalation(row, report),
+                 format_money(report.effective_cost),
+                 format_latency(report.effective_latency_ms),
+                 report.score)
+        end
+
+        def format_money(value)
+          value&.positive? ? "$#{format("%.4f", value)}" : "n/a"
+        end
+
+        def format_latency(value)
+          value ? "#{value.round}ms" : "n/a"
+        end
+
+        def format_escalation(row, report)
+          return "—" if row[:same]
+
+          format("%d%%", ((report.escalation_rate || 0) * 100).round)
+        end
+
+        public
+
         def print_summary(io = $stdout)
           io.puts "#{@eval_name} — model comparison"
           io.puts
@@ -72,7 +135,7 @@ module RubyLLM
 
         def to_h
           @reports.transform_values do |report|
-            {
+            base = {
               score: report.score,
               total_cost: report.total_cost,
               avg_latency_ms: report.avg_latency_ms,
@@ -80,11 +143,25 @@ module RubyLLM
               pass_rate_ratio: report.pass_rate_ratio,
               passed: report.passed?
             }
+            production_mode_metrics(report, base)
           end
         end
 
         private
 
+        def production_mode_metrics(report, base)
+          return base unless report.respond_to?(:production_mode?) && report.production_mode?
+
+          base.merge(
+            escalation_rate: report.escalation_rate,
+            single_shot_cost: report.single_shot_cost,
+            effective_cost: report.effective_cost,
+            single_shot_latency_ms: report.single_shot_latency_ms,
+            effective_latency_ms: report.effective_latency_ms,
+            latency_percentiles: report.latency_percentiles
+          )
+        end
+
         def resolve_key(candidate)
           case candidate
           when String then candidate

data/lib/ruby_llm/contract/eval/report.rb

@@ -15,7 +15,9 @@ module RubyLLM
         BASELINE_DIR = ".eval_baselines"
 
         def_delegators :@stats, :score, :passed, :failed, :skipped, :failures, :pass_rate, :pass_rate_ratio,
-                       :total_cost, :avg_latency_ms, :passed?
+                       :total_cost, :avg_latency_ms, :passed?,
+                       :production_mode?, :escalation_rate, :single_shot_cost, :single_shot_latency_ms,
+                       :effective_cost, :effective_latency_ms, :latency_percentiles
         def_delegators :@presenter, :summary, :to_s, :print_summary
         def_delegators :@storage, :save_history!, :eval_history, :save_baseline!, :compare_with_baseline,
                        :baseline_exists?

data/lib/ruby_llm/contract/eval/report_stats.rb

@@ -65,6 +65,69 @@ module RubyLLM
         def evaluated_results_count
           evaluated_results.length
         end
+
+        def production_mode?
+          evaluated_results.any? { |r| r.respond_to?(:attempts) && r.attempts }
+        end
+
+        def escalation_rate
+          return nil unless production_mode?
+          return 0.0 if evaluated_results.empty?
+
+          escalated = evaluated_results.count { |r| (r.attempts || []).length > 1 }
+          escalated.to_f / evaluated_results.length
+        end
+
+        def single_shot_cost
+          return nil unless production_mode?
+
+          evaluated_results.sum { |r| first_attempt_cost(r) || r.cost || 0.0 }
+        end
+
+        def effective_cost
+          total_cost
+        end
+
+        def single_shot_latency_ms
+          return nil unless production_mode?
+
+          latencies = evaluated_results.filter_map { |r| first_attempt_latency(r) || r.duration_ms }
+          return nil if latencies.empty?
+
+          latencies.sum.to_f / latencies.length
+        end
+
+        def effective_latency_ms
+          avg_latency_ms
+        end
+
+        def latency_percentiles
+          return nil unless production_mode?
+
+          latencies = evaluated_results.filter_map(&:duration_ms).sort
+          return nil if latencies.empty?
+
+          { p50: percentile(latencies, 0.50), p95: percentile(latencies, 0.95), max: latencies.last.to_f }
+        end
+
+        private
+
+        def first_attempt_cost(result)
+          first = (result.attempts || []).first
+          first && first[:cost]
+        end
+
+        def first_attempt_latency(result)
+          first = (result.attempts || []).first
+          first && first[:latency_ms]
+        end
+
+        def percentile(sorted, fraction)
+          return nil if sorted.empty?
+
+          idx = (fraction * (sorted.length - 1)).round
+          sorted[idx].to_f
+        end
       end
     end
   end

data/lib/ruby_llm/contract/eval/retry_optimizer.rb

@@ -94,12 +94,13 @@ module RubyLLM
           end
         end
 
-        def initialize(step:, candidates:, context: {}, min_score: 0.95, runs: 1)
+        def initialize(step:, candidates:, context: {}, min_score: 0.95, runs: 1, production_mode: nil)
           @step = step
           @candidates = candidates
           @context = context
           @min_score = min_score
           @runs = runs
+          @production_mode = production_mode
         end
 
         def call
@@ -109,7 +110,8 @@ module RubyLLM
           score_matrix = {}
           evals.each do |eval_name|
             comparison = with_retry_disabled do
-              @step.compare_models(eval_name, candidates: @candidates, context: @context, runs: @runs)
+              @step.compare_models(eval_name, candidates: @candidates, context: @context,
+                                              runs: @runs, production_mode: @production_mode)
             end
             score_matrix[eval_name] = extract_scores(comparison)
           end

data/lib/ruby_llm/contract/step/adapter_caller.rb

@@ -1,9 +1,20 @@
 # frozen_string_literal: true
 
+require "faraday"
+
 module RubyLLM
   module Contract
     module Step
       class AdapterCaller
+        # Exceptions treated as :adapter_error (retryable when explicitly opted in).
+        # RubyLLM::Error covers provider-semantic errors (auth, bad request,
+        # rate limit, server error, context length). Faraday::Error covers
+        # transport failures that escape ruby_llm's Faraday retry middleware
+        # after exhaustion (Faraday::TimeoutError, Faraday::ConnectionFailed).
+        # Anything else (NoMethodError, programmer ArgumentError from adapter
+        # code, etc.) propagates — those are bugs, not retry candidates.
+        ADAPTER_ERRORS = [::RubyLLM::Error, ::Faraday::Error].freeze
+
         def initialize(adapter:, adapter_options:)
           @adapter = adapter
           @adapter_options = adapter_options
@@ -14,8 +25,14 @@ module RubyLLM
           response = @adapter.call(messages: messages, **@adapter_options)
           latency_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time) * 1000).round
           [response, latency_ms]
-        rescue StandardError => error
-          [Result.new(status: :adapter_error, raw_output: nil, parsed_output: nil, validation_errors: [error.message]), 0]
+        rescue *ADAPTER_ERRORS => e
+          result = Result.new(
+            status: :adapter_error,
+            raw_output: nil,
+            parsed_output: nil,
+            validation_errors: [e.message]
+          )
+          [result, 0]
         end
       end
     end

data/lib/ruby_llm/contract/step/base.rb

@@ -59,17 +59,19 @@ module RubyLLM
             ).recommend
           end
 
-          def optimize_retry_policy(candidates:, context: {}, min_score: 0.95, runs: 1)
+          def optimize_retry_policy(candidates:, context: {}, min_score: 0.95, runs: 1, production_mode: nil)
             Eval::RetryOptimizer.new(
               step: self,
               candidates: candidates,
               context: context,
               min_score: min_score,
-              runs: runs
+              runs: runs,
+              production_mode: production_mode
             ).call
           end
 
-          KNOWN_CONTEXT_KEYS = %i[adapter model temperature max_tokens provider assume_model_exists reasoning_effort].freeze
+          KNOWN_CONTEXT_KEYS = %i[adapter model temperature max_tokens provider assume_model_exists
+                                  reasoning_effort retry_policy_override].freeze
 
           include Concerns::ContextHelpers
 
@@ -156,11 +158,12 @@ module RubyLLM
           end
 
           def runtime_settings(context)
+            policy = context.key?(:retry_policy_override) ? context[:retry_policy_override] : retry_policy
             {
               model: context[:model] || model || RubyLLM::Contract.configuration.default_model,
               temperature: context[:temperature],
               extra_options: context.slice(:provider, :assume_model_exists, :max_tokens, :reasoning_effort),
-              policy: retry_policy
+              policy: policy
             }
           end
 

data/lib/ruby_llm/contract/step/retry_policy.rb

@@ -6,7 +6,14 @@ module RubyLLM
       class RetryPolicy
         attr_reader :max_attempts, :retryable_statuses
 
-        DEFAULT_RETRY_ON = %i[validation_failed parse_error adapter_error].freeze
+        # Breaking (0.7.0): :adapter_error removed from defaults. ruby_llm's Faraday
+        # middleware already retries transport errors (RateLimitError, ServerError,
+        # ServiceUnavailableError, OverloadedError, timeouts). Retrying on
+        # :adapter_error against the same model re-runs what transport already did.
+        # Opt in explicitly with `retry_on :adapter_error` — only meaningful paired
+        # with `escalate "model_a", "model_b"` (a different model may bypass what
+        # transport retry could not).
+        DEFAULT_RETRY_ON = %i[validation_failed parse_error].freeze
 
         def initialize(models: nil, attempts: nil, retry_on: nil, &block)
           @configs = []

data/lib/ruby_llm/contract/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module Contract
-    VERSION = "0.6.3"
+    VERSION = "0.7.0"
   end
 end

data/lib/ruby_llm/contract.rb

@@ -154,6 +154,7 @@ end
 require_relative "contract/concerns/context_helpers"
 require_relative "contract/concerns/deep_freeze"
 require_relative "contract/concerns/deep_symbolize"
+require_relative "contract/concerns/production_mode_context"
 require_relative "contract/concerns/eval_host"
 require_relative "contract/concerns/trace_equality"
 require_relative "contract/concerns/usage_aggregator"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-contract
 version: !ruby/object:Gem::Version
-  version: 0.6.3
+  version: 0.7.0
 platform: ruby
 authors:
 - Justyna
@@ -89,6 +89,7 @@ files:
 - lib/ruby_llm/contract/concerns/deep_freeze.rb
 - lib/ruby_llm/contract/concerns/deep_symbolize.rb
 - lib/ruby_llm/contract/concerns/eval_host.rb
+- lib/ruby_llm/contract/concerns/production_mode_context.rb
 - lib/ruby_llm/contract/concerns/trace_equality.rb
 - lib/ruby_llm/contract/concerns/usage_aggregator.rb
 - lib/ruby_llm/contract/configuration.rb