ruby_llm-contract 0.6.0 → 0.6.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c86dcf06a34e5ff934367708213a0852191b0be7bd61e61e8577161e32ccf807
-  data.tar.gz: 28a44dd4f4d4c0f74c5da7800d9fa1fd2b8e51242a056f5aaae5f6f9fcf1be69
+  metadata.gz: 75884386ae53ddf1985760afa2c94f64ce15274b0cb0829ea40e6711506e60cf
+  data.tar.gz: 203229c4ce8ab0b1ab9e6209871668b0713b114e412eba03e79e9964dc9a43bb
 SHA512:
-  metadata.gz: c2b6dc71a02519288ae4ee4f74f17d74e6c45d01e888036d18e4c821b8fc63ba43ac0ccee6b2948163597b19c93008a6e5e0375fd65c8f3ad8fcf1285f356e91
-  data.tar.gz: f08576e520ec7397c4b233c5907ce703890cf4616e969e25f4c980ac1b06013a62ff680b3c90878d76fd1da0980f04d4946a2fd11a8e304e43f5155e5c855e8e
+  metadata.gz: 0c586ce70e71d8e77c262e0ae21bab2e29ef6ccdb16df4a0e56271aeb98efba59027618db24d4b59470fd3a9d340051201584038f8c67143f7ca1bec0e5e365b
+  data.tar.gz: 6a988c62f6b36a4da860b736c5523abcc568ed1384c1f20cd8ff9659124c12eeeddb36100d5b8542af07b363cef7ade8a1aca01250929ec2c23172d4f3faec5f

data/CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## 0.6.3 (2026-04-20)
+
+### Features
+
+- **`runs:` parameter on `compare_models` and `optimize_retry_policy`** — runs each candidate N times per eval and aggregates the mean score, mean cost per run, and mean latency. Reduces sampling variance in live mode where LLM outputs are non-deterministic (gpt-5 family enforces `temperature=1.0` server-side, so a single unlucky sample can misclassify a viable candidate as "failing"). Default `runs: 1` — backward compatible.
+- **`RUNS=N` on `rake ruby_llm_contract:optimize`** — CLI flag for variance-aware optimization.
+- **`Eval::AggregatedReport`** — duck-type `Report` exposing `score` (mean), `score_min`/`score_max` (spread), `total_cost` (mean per run), `pass_rate` (clean-pass count x/N), and `clean_passes`.
+- **Guide: [Reducing variance with `runs:`](docs/guide/optimizing_retry_policy.md#reducing-variance-with-runs)** — when to use it and why.
+
+## 0.6.2 (2026-04-18)
+
+### Features
+
+- **`Step.optimize_retry_policy`** — runs `compare_models` on ALL evals for the step, builds a score matrix, identifies the constraining eval, and suggests a retry chain. Chain's last model always passes all evals (safe fallback).
+- **`rake ruby_llm_contract:optimize`** — one-command retry chain optimization. Prints score table, constraining eval, suggested chain, and copy-paste DSL.
+- **Offline by default** — `optimize` uses `sample_response` (zero API calls) unless `LIVE=1` or `PROVIDER=` is set.
+- **`EVAL_DIRS=` support** — non-Rails setups can specify eval file directories.
+- **Guide: [Optimizing retry_policy](docs/guide/optimizing_retry_policy.md)** — full procedure with prerequisites, troubleshooting, and real-world example.
+
+### Fixes
+
+- Chain semantics aligned with `retry_executor` — retry fires on `validation_failed`/`parse_error`, not on low eval score. Disjoint eval coverage (A passes e1, B passes e2, neither passes both) correctly returns empty chain.
+- Removed ActiveSupport dependency from rake task (`.presence` → `.empty?`).
+- Added `require "set"` for non-Rails environments.
+
+## 0.6.1 (2026-04-17)
+
+### Features
+
+- **Multi-provider operator tooling** — rake tasks support `PROVIDER=openai|anthropic|ollama`, `CANDIDATES=model@effort,...`, and `REASONING_EFFORT=low|medium|high`.
+- **`rake ruby_llm_contract:recommend`** — wraps `Step.recommend` with CLI interface, prints best config, retry chain, DSL, rationale, and savings.
+- **Ollama support** — `PROVIDER=ollama` with configurable `OLLAMA_API_BASE`.
+
 ## 0.6.0 (2026-04-12)
 
 "What should I do?" — model + configuration recommendation.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ruby_llm-contract (0.6.0)
+    ruby_llm-contract (0.6.3)
       dry-types (~> 1.7)
       ruby_llm (~> 1.0)
       ruby_llm-schema (~> 0.3)
@@ -258,7 +258,7 @@ CHECKSUMS
   rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   ruby_llm (1.14.0) sha256=57c6f7034fc4a44504ea137d70f853b07824f1c1cdbe774ab3ab3522e7098deb
-  ruby_llm-contract (0.6.0)
+  ruby_llm-contract (0.6.3)
   ruby_llm-schema (0.3.0) sha256=a591edc5ca1b7f0304f0e2261de61ba4b3bea17be09f5cf7558153adfda3dec6
   ruby_parser (3.22.0) sha256=1eb4937cd9eb220aa2d194e352a24dba90aef00751e24c8dfffdb14000f15d23
   rubycritic (4.12.0) sha256=024fed90fe656fa939f6ea80aab17569699ac3863d0b52fd72cb99892247abc8

data/README.md

@@ -158,6 +158,8 @@ Cheapest at 100%: gpt-4.1-mini
 
 Nano fails on edge cases. Mini and full both score 100% — but mini is **5x cheaper**. Now you know.
 
+Running live against gpt-5 / o-series? Pass `runs: 3` to average out sampling variance (OpenAI forces `temperature=1.0` server-side, so one unlucky run can misclassify a viable candidate). See [Reducing variance with `runs:`](docs/guide/optimizing_retry_policy.md#reducing-variance-with-runs).
+
 ## Let the gem tell you what to do
 
 Don't read tables — get a recommendation. Supports `model + reasoning_effort` combinations:
@@ -257,12 +259,21 @@ end
 # bundle exec rake ruby_llm_contract:eval
 ```
 
+## Full power: data-driven retry chains
+
+The pieces above — evals, compare_models, recommend — combine into a workflow that replaces guesswork with measured optimization. You define evals for your step, run `recommend` against all of them, find the eval that actually needs the strongest model, and build a retry chain where each attempt is as cheap as the data allows.
+
+The difference: instead of "gpt-5-mini seems to work, let's use it everywhere", you get "nano handles 4/6 scenarios, mini@low catches the 5th, full mini only fires on the hardest edge case — first attempt is 4× cheaper."
+
+Full procedure with examples: **[Optimizing retry_policy](docs/guide/optimizing_retry_policy.md)**
+
 ## Docs
 
 | Guide | |
 |-------|-|
 | [Getting Started](docs/guide/getting_started.md) | Features walkthrough, model escalation, eval |
 | [Eval-First](docs/guide/eval_first.md) | Practical workflow for prompt engineering with datasets, baselines, and A/B gates |
+| [Optimizing retry_policy](docs/guide/optimizing_retry_policy.md) | Find the cheapest retry chain that passes all your evals |
 | [Best Practices](docs/guide/best_practices.md) | 6 patterns for bulletproof validates |
 | [Output Schema](docs/guide/output_schema.md) | Full schema reference + constraints |
 | [Pipeline](docs/guide/pipeline.md) | Multi-step composition, timeout, fail-fast |

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -70,9 +70,11 @@ module RubyLLM
           Eval::PromptDiff.new(candidate: my_report, baseline: other_report)
         end
 
-        def compare_models(eval_name, models: [], candidates: [], context: {})
+        def compare_models(eval_name, models: [], candidates: [], context: {}, runs: 1)
           raise ArgumentError, "Pass either models: or candidates:, not both" if models.any? && candidates.any?
 
+          runs = coerce_runs(runs)
+
           context = safe_context(context)
           candidate_configs = normalize_candidates(models, candidates)
 
@@ -82,7 +84,8 @@ module RubyLLM
             label = Eval::ModelComparison.candidate_label(config)
             model_context = isolate_context(context).merge(model: config[:model])
             model_context[:reasoning_effort] = config[:reasoning_effort] if config[:reasoning_effort]
-            reports[label] = run_single_eval(eval_name, model_context)
+            per_run = Array.new(runs) { run_single_eval(eval_name, model_context) }
+            reports[label] = runs == 1 ? per_run.first : Eval::AggregatedReport.new(per_run)
             configs[label] = config
           end
 
@@ -91,6 +94,13 @@ module RubyLLM
 
         private
 
+        def coerce_runs(runs)
+          raise ArgumentError, "runs must be an Integer >= 1, got #{runs.inspect}" unless runs.is_a?(Integer)
+          raise ArgumentError, "runs must be >= 1, got #{runs.inspect}" if runs < 1
+
+          runs
+        end
+
         def normalize_candidates(models, candidates)
           if candidates.any?
             candidates.map { |c| RubyLLM::Contract.normalize_candidate_config(c) }.uniq

data/lib/ruby_llm/contract/eval/aggregated_report.rb

@@ -0,0 +1,92 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Eval
+      # Wraps N Reports from repeated runs of the same eval to reduce sampling
+      # variance in live mode (temperature=1 on gpt-5 family). Exposes the same
+      # duck-type as Report — mean score, mean cost per run, mean latency.
+      #
+      # pass_rate reports how many runs passed cleanly (x/N), not case-level
+      # pass rate, since the question is "does this candidate reliably pass?".
+      class AggregatedReport
+        attr_reader :runs, :results
+
+        def initialize(runs)
+          raise ArgumentError, "runs must not be empty" if runs.empty?
+
+          @runs = runs.freeze
+          @results = runs.flat_map(&:results).freeze
+          freeze
+        end
+
+        def dataset_name
+          @runs.first.dataset_name
+        end
+
+        def step_name
+          @runs.first.step_name
+        end
+
+        def score
+          @runs.sum(&:score) / @runs.length.to_f
+        end
+
+        def score_min
+          @runs.map(&:score).min
+        end
+
+        def score_max
+          @runs.map(&:score).max
+        end
+
+        def total_cost
+          @runs.sum(&:total_cost) / @runs.length.to_f
+        end
+
+        def avg_latency_ms
+          latencies = @runs.filter_map(&:avg_latency_ms)
+          return nil if latencies.empty?
+
+          latencies.sum / latencies.length.to_f
+        end
+
+        def pass_rate
+          "#{clean_passes}/#{@runs.length}"
+        end
+
+        def pass_rate_ratio
+          clean_passes.to_f / @runs.length
+        end
+
+        def each(&block)
+          @results.each(&block)
+        end
+
+        def summary
+          @runs.first.summary
+        end
+
+        def to_s
+          @runs.first.to_s
+        end
+
+        def print_summary(io = $stdout)
+          @runs.first.print_summary(io)
+        end
+
+        def passed?
+          @runs.all?(&:passed?)
+        end
+
+        def clean_passes
+          @runs.count(&:passed?)
+        end
+
+        def failures
+          @runs.flat_map(&:failures)
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/eval/retry_optimizer.rb

@@ -0,0 +1,222 @@
+# frozen_string_literal: true
+
+require "set"
+
+module RubyLLM
+  module Contract
+    module Eval
+      # Runs compare_models on ALL evals for a step, builds a score matrix,
+      # identifies the constraining eval, and suggests an escalation chain.
+      #
+      #   optimizer = RetryOptimizer.new(step: MyStep, candidates: [...], context: {})
+      #   result = optimizer.call
+      #   result.print_summary
+      #   result.to_dsl  # => copy-paste retry_policy
+      class RetryOptimizer
+        Result = Struct.new(:step_name, :eval_names, :candidate_labels, :score_matrix,
+                            :constraining_eval, :chain, :chain_details, keyword_init: true) do
+          def print_summary(io = $stdout)
+            io.puts "#{step_name} — retry chain optimization"
+            io.puts
+            print_table(io)
+            io.puts
+            print_chain(io)
+            io.puts
+            print_dsl(io)
+          end
+
+          def to_dsl
+            return "# No viable chain — no candidate passes all evals" if chain.empty?
+
+            if chain.all? { |c| c.keys == [:model] }
+              models_str = chain.map { |c| c[:model] }.join(" ")
+              "retry_policy models: %w[#{models_str}]"
+            else
+              args = chain.map { |c| config_to_ruby(c) }.join(",\n    ")
+              "retry_policy do\n  escalate(\n    #{args}\n  )\nend"
+            end
+          end
+
+          private
+
+          def print_table(io)
+            short_labels = candidate_labels.map { |l| short_candidate_label(l) }
+            col_width = [short_labels.map(&:length).max || 0, 8].max
+            eval_width = [eval_names.map { |e| e.to_s.length }.max || 0, 12].max
+
+            header = format("  %-#{eval_width}s", "eval") + short_labels.map { |l| format("  %#{col_width}s", l) }.join
+            io.puts header
+            io.puts "  #{"-" * (eval_width + (col_width + 2) * short_labels.size)}"
+
+            eval_names.each do |eval_name|
+              row = format("  %-#{eval_width}s", eval_name.to_s)
+              candidate_labels.each do |label|
+                score = score_matrix.dig(eval_name, label) || 0.0
+                marker = eval_name == constraining_eval && score < 1.0 ? " ←" : "  "
+                row += format("  %#{col_width - 2}.2f%s", score, marker)
+              end
+              io.puts row
+            end
+
+            io.puts
+            io.puts "  Constraining eval: #{constraining_eval}" if constraining_eval
+          end
+
+          def print_chain(io)
+            if chain.empty?
+              io.puts "  No viable chain."
+              return
+            end
+
+            io.puts "  Suggested chain:"
+            chain_details.each_with_index do |detail, i|
+              suffix = i == chain_details.size - 1 ? "passes all #{eval_names.size} evals" : "covers #{detail[:passes]} eval(s)"
+              io.puts "    #{detail[:label]} — #{suffix}"
+            end
+          end
+
+          def short_candidate_label(label)
+            label
+              .sub("gpt-5-", "")
+              .sub("gpt-4.1", "4.1")
+              .sub(" (effort: ", "@")
+              .sub(")", "")
+          end
+
+          def print_dsl(io)
+            io.puts "  DSL:"
+            to_dsl.each_line { |line| io.puts "    #{line}" }
+          end
+
+          def config_to_ruby(config)
+            pairs = config.map { |k, v| "#{k}: #{v.inspect}" }.join(", ")
+            "{ #{pairs} }"
+          end
+        end
+
+        def initialize(step:, candidates:, context: {}, min_score: 0.95, runs: 1)
+          @step = step
+          @candidates = candidates
+          @context = context
+          @min_score = min_score
+          @runs = runs
+        end
+
+        def call
+          evals = @step.eval_names
+          return empty_result(evals) if evals.empty?
+
+          score_matrix = {}
+          evals.each do |eval_name|
+            comparison = with_retry_disabled do
+              @step.compare_models(eval_name, candidates: @candidates, context: @context, runs: @runs)
+            end
+            score_matrix[eval_name] = extract_scores(comparison)
+          end
+
+          labels = score_matrix.values.flat_map(&:keys).uniq
+          constraining = find_constraining_eval(score_matrix, labels)
+          chain, details = build_chain(score_matrix, labels, evals)
+
+          Result.new(
+            step_name: @step.name || @step.to_s,
+            eval_names: evals,
+            candidate_labels: labels,
+            score_matrix: score_matrix,
+            constraining_eval: constraining,
+            chain: chain,
+            chain_details: details
+          )
+        end
+
+        private
+
+        def extract_scores(comparison)
+          comparison.reports.transform_values(&:score)
+        end
+
+        def find_constraining_eval(matrix, labels)
+          matrix.max_by do |_eval_name, scores|
+            cheapest_passing = labels.find { |l| (scores[l] || 0) >= @min_score }
+            cheapest_passing ? labels.index(cheapest_passing) : labels.size
+          end&.first
+        end
+
+        # Retry escalates on validation_failed/parse_error, NOT on low eval
+        # score. A model that returns :ok with semantically wrong output won't
+        # trigger retry. Therefore the LAST model in the chain must pass ALL
+        # evals — it's the safety net. Cheaper models are prepended as
+        # first-try optimization (they handle easy inputs cheaply; when they
+        # fail validation, retry escalates to the safe fallback).
+        #
+        # Known limitation: intermediate models are assumed safe if their eval
+        # failures correspond to validation failures (retryable). If an
+        # intermediate model returns :ok with semantically wrong output on
+        # some eval, retry won't fire and the safe fallback won't run. This
+        # requires step validates to cover the same semantics as eval verify
+        # checks. A future version could inspect per-case step_status from
+        # compare_models to verify failures are actually retryable.
+        def build_chain(matrix, labels, evals)
+          total = evals.size
+
+          # Find cheapest model that passes every eval — the safe fallback.
+          safe_fallback = labels.find { |l| evals.all? { |e| (matrix.dig(e, l) || 0) >= @min_score } }
+          return [[], []] unless safe_fallback
+
+          # Prepend cheaper models that pass a strict subset.
+          chain = []
+          details = []
+          covered_evals = Set.new
+
+          labels.each do |label|
+            break if label == safe_fallback
+
+            newly_covered = evals.select { |e| (matrix.dig(e, label) || 0) >= @min_score }
+            new_additions = newly_covered.to_set - covered_evals
+            next if new_additions.empty?
+
+            covered_evals.merge(new_additions)
+            chain << parse_label_to_config(label)
+            details << { label: label, passes: new_additions.size, cost: label }
+          end
+
+          # Always end with the safe fallback.
+          chain << parse_label_to_config(safe_fallback)
+          details << { label: safe_fallback, passes: total, cost: safe_fallback }
+
+          [chain, details]
+        end
+
+        def parse_label_to_config(label)
+          if label.match?(/\(effort: (\w+)\)/)
+            model = label.sub(/\s*\(effort:.*/, "").strip
+            effort = label.match(/\(effort: (\w+)\)/)[1]
+            { model: model, reasoning_effort: effort }
+          else
+            { model: label }
+          end
+        end
+
+        def with_retry_disabled(&block)
+          original = @step.retry_policy if @step.respond_to?(:retry_policy)
+          @step.define_singleton_method(:retry_policy) { nil }
+          block.call
+        ensure
+          @step.define_singleton_method(:retry_policy) { original }
+        end
+
+        def empty_result(evals)
+          Result.new(
+            step_name: @step.name || @step.to_s,
+            eval_names: evals,
+            candidate_labels: [],
+            score_matrix: {},
+            constraining_eval: nil,
+            chain: [],
+            chain_details: []
+          )
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/eval.rb

@@ -21,6 +21,7 @@ require_relative "eval/report_stats"
 require_relative "eval/report_presenter"
 require_relative "eval/report_storage"
 require_relative "eval/report"
+require_relative "eval/aggregated_report"
 require_relative "eval/eval_definition"
 require_relative "eval/model_comparison"
 require_relative "eval/baseline_diff"
@@ -31,3 +32,4 @@ require_relative "eval/prompt_diff"
 require_relative "eval/eval_history"
 require_relative "eval/recommendation"
 require_relative "eval/recommender"
+require_relative "eval/retry_optimizer"

data/lib/ruby_llm/contract/rake_task.rb

@@ -121,5 +121,98 @@ module RubyLLM
         defined?(::Rails) ? [:environment] : []
       end
     end
+
+    # Standalone task: runs all evals for one step across candidates,
+    # builds a score matrix, and suggests an optimal retry chain.
+    #
+    # Loaded automatically when `require "ruby_llm/contract/rake_task"`.
+    # Usage:
+    #   rake ruby_llm_contract:optimize \
+    #     STEP=MatchProblemsToPages \
+    #     CANDIDATES=gpt-5-nano,gpt-5-mini@low,gpt-5-mini
+    class OptimizeRakeTask < ::Rake::TaskLib
+      def initialize
+        super()
+        define_task
+      end
+
+      private
+
+      def define_task
+        desc "Run all evals for STEP with CANDIDATES and suggest an optimal retry chain"
+        task(:"ruby_llm_contract:optimize" => task_prerequisites) do
+          require "ruby_llm/contract"
+          eval_dirs = ENV["EVAL_DIRS"].to_s.split(",").map(&:strip).reject(&:empty?)
+          RubyLLM::Contract.load_evals!(*eval_dirs)
+
+          step_name = ENV["STEP"].to_s.strip
+          abort("STEP is required, e.g. STEP=MatchProblemsToPages") if step_name.empty?
+          raw_candidates = ENV["CANDIDATES"].to_s.strip
+          abort("CANDIDATES is required, e.g. CANDIDATES=gpt-5-nano,gpt-5-mini@low,gpt-5-mini") if raw_candidates.empty?
+          min_score = ENV.fetch("MIN_SCORE", "0.95").to_f
+          runs = parse_runs(ENV.fetch("RUNS", "1"))
+
+          host = RubyLLM::Contract.eval_hosts.find { |h| h.name == step_name }
+          unless host
+            available = RubyLLM::Contract.eval_hosts.filter_map(&:name).sort
+            abort "Unknown STEP=#{step_name}. Available: #{available.join(", ")}"
+          end
+
+          candidates = parse_candidates(raw_candidates)
+          context = build_context
+
+          result = host.optimize_retry_policy(
+            candidates: candidates,
+            context: context,
+            min_score: min_score,
+            runs: runs
+          )
+
+          result.print_summary
+        end
+      end
+
+      def parse_runs(raw)
+        runs = Integer(raw.to_s.strip, 10)
+        abort("RUNS must be an integer >= 1, e.g. RUNS=1") if runs < 1
+        runs
+      rescue ArgumentError
+        abort("RUNS must be an integer >= 1, e.g. RUNS=1")
+      end
+
+      def parse_candidates(raw)
+        entries = if raw.start_with?("[")
+                    Array(JSON.parse(raw))
+                  else
+                    raw.split(",").map(&:strip).reject(&:empty?).map do |entry|
+                      model, effort = entry.split("@", 2)
+                      config = { model: model.strip }
+                      config[:reasoning_effort] = effort.strip if effort && !effort.empty?
+                      config
+                    end
+                  end
+
+        entries.map { |e| RubyLLM::Contract.normalize_candidate_config(e) }.uniq
+      end
+
+      def build_context
+        ctx = {}
+        provider = ENV["PROVIDER"].to_s.strip
+        # Only inject real adapter when LIVE=1 or PROVIDER is set — otherwise
+        # evals use sample_response (offline mode, zero API calls).
+        if ENV["LIVE"] == "1" || !provider.empty?
+          ctx[:adapter] = RubyLLM::Contract::Adapters::RubyLLM.new
+          ctx[:provider] = provider.downcase.to_sym unless provider.empty?
+        end
+        ctx
+      end
+
+      def task_prerequisites
+        defined?(::Rails) ? [:environment] : []
+      end
+    end
+
+    # Auto-register the optimize task when this file is loaded
+    OptimizeRakeTask.new
   end
 end

data/lib/ruby_llm/contract/step/base.rb

@@ -59,6 +59,16 @@ module RubyLLM
             ).recommend
           end
 
+          def optimize_retry_policy(candidates:, context: {}, min_score: 0.95, runs: 1)
+            Eval::RetryOptimizer.new(
+              step: self,
+              candidates: candidates,
+              context: context,
+              min_score: min_score,
+              runs: runs
+            ).call
+          end
+
           KNOWN_CONTEXT_KEYS = %i[adapter model temperature max_tokens provider assume_model_exists reasoning_effort].freeze
 
           include Concerns::ContextHelpers

data/lib/ruby_llm/contract/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module Contract
-    VERSION = "0.6.0"
+    VERSION = "0.6.3"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-contract
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.6.3
 platform: ruby
 authors:
 - Justyna
@@ -109,6 +109,7 @@ files:
 - lib/ruby_llm/contract/dsl.rb
 - lib/ruby_llm/contract/errors.rb
 - lib/ruby_llm/contract/eval.rb
+- lib/ruby_llm/contract/eval/aggregated_report.rb
 - lib/ruby_llm/contract/eval/baseline_diff.rb
 - lib/ruby_llm/contract/eval/case_executor.rb
 - lib/ruby_llm/contract/eval/case_result.rb
@@ -136,6 +137,7 @@ files:
 - lib/ruby_llm/contract/eval/report_presenter.rb
 - lib/ruby_llm/contract/eval/report_stats.rb
 - lib/ruby_llm/contract/eval/report_storage.rb
+- lib/ruby_llm/contract/eval/retry_optimizer.rb
 - lib/ruby_llm/contract/eval/runner.rb
 - lib/ruby_llm/contract/eval/step_expectation_applier.rb
 - lib/ruby_llm/contract/eval/step_result_normalizer.rb