ruby_llm-contract 0.6.0 → 0.6.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c86dcf06a34e5ff934367708213a0852191b0be7bd61e61e8577161e32ccf807
-  data.tar.gz: 28a44dd4f4d4c0f74c5da7800d9fa1fd2b8e51242a056f5aaae5f6f9fcf1be69
+  metadata.gz: 2636c2e59f5fef27f929a94ac9e3194793ce6b51f86cce0c18ca6a5b0caa61ab
+  data.tar.gz: c2c81c7cc8fd281bf6c88738f0fb5bc4bdbb86b71d2fb59248e2b0ebb8d648fe
 SHA512:
-  metadata.gz: c2b6dc71a02519288ae4ee4f74f17d74e6c45d01e888036d18e4c821b8fc63ba43ac0ccee6b2948163597b19c93008a6e5e0375fd65c8f3ad8fcf1285f356e91
-  data.tar.gz: f08576e520ec7397c4b233c5907ce703890cf4616e969e25f4c980ac1b06013a62ff680b3c90878d76fd1da0980f04d4946a2fd11a8e304e43f5155e5c855e8e
+  metadata.gz: d9f2fca592fd3a183d987239dea0cdc2456eed639d6cccaea71e9b1ef3a3ff6e32f3e346f640c905168674e7401ccb85e5f342815a1b290cdebe05a9f7b5374f
+  data.tar.gz: 00b8d113564871db19f88d9061276a0aee0295da7638974804782fda7929cf893a0004aadfeffc2f25f13d421935e0e9d710e553d8e56b7b5d0229f62edd129e

data/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## 0.6.2 (2026-04-18)
+
+### Features
+
+- **`Step.optimize_retry_policy`** — runs `compare_models` on ALL evals for the step, builds a score matrix, identifies the constraining eval, and suggests a retry chain. Chain's last model always passes all evals (safe fallback).
+- **`rake ruby_llm_contract:optimize`** — one-command retry chain optimization. Prints score table, constraining eval, suggested chain, and copy-paste DSL.
+- **Offline by default** — `optimize` uses `sample_response` (zero API calls) unless `LIVE=1` or `PROVIDER=` is set.
+- **`EVAL_DIRS=` support** — non-Rails setups can specify eval file directories.
+- **Guide: [Optimizing retry_policy](docs/guide/optimizing_retry_policy.md)** — full procedure with prerequisites, troubleshooting, and real-world example.
+
+### Fixes
+
+- Chain semantics aligned with `retry_executor` — retry fires on `validation_failed`/`parse_error`, not on low eval score. Disjoint eval coverage (A passes e1, B passes e2, neither passes both) correctly returns empty chain.
+- Removed ActiveSupport dependency from rake task (`.presence` → `.empty?`).
+- Added `require "set"` for non-Rails environments.
+
+## 0.6.1 (2026-04-17)
+
+### Features
+
+- **Multi-provider operator tooling** — rake tasks support `PROVIDER=openai|anthropic|ollama`, `CANDIDATES=model@effort,...`, and `REASONING_EFFORT=low|medium|high`.
+- **`rake ruby_llm_contract:recommend`** — wraps `Step.recommend` with CLI interface, prints best config, retry chain, DSL, rationale, and savings.
+- **Ollama support** — `PROVIDER=ollama` with configurable `OLLAMA_API_BASE`.
+
 ## 0.6.0 (2026-04-12)
 
 "What should I do?" — model + configuration recommendation.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ruby_llm-contract (0.6.0)
+    ruby_llm-contract (0.6.1)
       dry-types (~> 1.7)
       ruby_llm (~> 1.0)
       ruby_llm-schema (~> 0.3)
@@ -258,7 +258,7 @@ CHECKSUMS
   rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   ruby_llm (1.14.0) sha256=57c6f7034fc4a44504ea137d70f853b07824f1c1cdbe774ab3ab3522e7098deb
-  ruby_llm-contract (0.6.0)
+  ruby_llm-contract (0.6.1)
   ruby_llm-schema (0.3.0) sha256=a591edc5ca1b7f0304f0e2261de61ba4b3bea17be09f5cf7558153adfda3dec6
   ruby_parser (3.22.0) sha256=1eb4937cd9eb220aa2d194e352a24dba90aef00751e24c8dfffdb14000f15d23
   rubycritic (4.12.0) sha256=024fed90fe656fa939f6ea80aab17569699ac3863d0b52fd72cb99892247abc8

data/README.md

@@ -257,12 +257,21 @@ end
 # bundle exec rake ruby_llm_contract:eval
 ```
 
+## Full power: data-driven retry chains
+
+The pieces above — evals, compare_models, recommend — combine into a workflow that replaces guesswork with measured optimization. You define evals for your step, run `recommend` against all of them, find the eval that actually needs the strongest model, and build a retry chain where each attempt is as cheap as the data allows.
+
+The difference: instead of "gpt-5-mini seems to work, let's use it everywhere", you get "nano handles 4/6 scenarios, mini@low catches the 5th, full mini only fires on the hardest edge case — first attempt is 4× cheaper."
+
+Full procedure with examples: **[Optimizing retry_policy](docs/guide/optimizing_retry_policy.md)**
+
 ## Docs
 
 | Guide | |
 |-------|-|
 | [Getting Started](docs/guide/getting_started.md) | Features walkthrough, model escalation, eval |
 | [Eval-First](docs/guide/eval_first.md) | Practical workflow for prompt engineering with datasets, baselines, and A/B gates |
+| [Optimizing retry_policy](docs/guide/optimizing_retry_policy.md) | Find the cheapest retry chain that passes all your evals |
 | [Best Practices](docs/guide/best_practices.md) | 6 patterns for bulletproof validates |
 | [Output Schema](docs/guide/output_schema.md) | Full schema reference + constraints |
 | [Pipeline](docs/guide/pipeline.md) | Multi-step composition, timeout, fail-fast |

data/lib/ruby_llm/contract/eval/retry_optimizer.rb

@@ -0,0 +1,221 @@
+# frozen_string_literal: true
+
+require "set"
+
+module RubyLLM
+  module Contract
+    module Eval
+      # Runs compare_models on ALL evals for a step, builds a score matrix,
+      # identifies the constraining eval, and suggests an escalation chain.
+      #
+      #   optimizer = RetryOptimizer.new(step: MyStep, candidates: [...], context: {})
+      #   result = optimizer.call
+      #   result.print_summary
+      #   result.to_dsl  # => copy-paste retry_policy
+      class RetryOptimizer
+        Result = Struct.new(:step_name, :eval_names, :candidate_labels, :score_matrix,
+                            :constraining_eval, :chain, :chain_details, keyword_init: true) do
+          def print_summary(io = $stdout)
+            io.puts "#{step_name} — retry chain optimization"
+            io.puts
+            print_table(io)
+            io.puts
+            print_chain(io)
+            io.puts
+            print_dsl(io)
+          end
+
+          def to_dsl
+            return "# No viable chain — no candidate passes all evals" if chain.empty?
+
+            if chain.all? { |c| c.keys == [:model] }
+              models_str = chain.map { |c| c[:model] }.join(" ")
+              "retry_policy models: %w[#{models_str}]"
+            else
+              args = chain.map { |c| config_to_ruby(c) }.join(",\n    ")
+              "retry_policy do\n  escalate(\n    #{args}\n  )\nend"
+            end
+          end
+
+          private
+
+          def print_table(io)
+            short_labels = candidate_labels.map { |l| short_candidate_label(l) }
+            col_width = [short_labels.map(&:length).max || 0, 8].max
+            eval_width = [eval_names.map { |e| e.to_s.length }.max || 0, 12].max
+
+            header = format("  %-#{eval_width}s", "eval") + short_labels.map { |l| format("  %#{col_width}s", l) }.join
+            io.puts header
+            io.puts "  #{"-" * (eval_width + (col_width + 2) * short_labels.size)}"
+
+            eval_names.each do |eval_name|
+              row = format("  %-#{eval_width}s", eval_name.to_s)
+              candidate_labels.each do |label|
+                score = score_matrix.dig(eval_name, label) || 0.0
+                marker = eval_name == constraining_eval && score < 1.0 ? " ←" : "  "
+                row += format("  %#{col_width - 2}.2f%s", score, marker)
+              end
+              io.puts row
+            end
+
+            io.puts
+            io.puts "  Constraining eval: #{constraining_eval}" if constraining_eval
+          end
+
+          def print_chain(io)
+            if chain.empty?
+              io.puts "  No viable chain."
+              return
+            end
+
+            io.puts "  Suggested chain:"
+            chain_details.each_with_index do |detail, i|
+              suffix = i == chain_details.size - 1 ? "passes all #{eval_names.size} evals" : "covers #{detail[:passes]} eval(s)"
+              io.puts "    #{detail[:label]} — #{suffix}"
+            end
+          end
+
+          def short_candidate_label(label)
+            label
+              .sub("gpt-5-", "")
+              .sub("gpt-4.1", "4.1")
+              .sub(" (effort: ", "@")
+              .sub(")", "")
+          end
+
+          def print_dsl(io)
+            io.puts "  DSL:"
+            to_dsl.each_line { |line| io.puts "    #{line}" }
+          end
+
+          def config_to_ruby(config)
+            pairs = config.map { |k, v| "#{k}: #{v.inspect}" }.join(", ")
+            "{ #{pairs} }"
+          end
+        end
+
+        def initialize(step:, candidates:, context: {}, min_score: 0.95)
+          @step = step
+          @candidates = candidates
+          @context = context
+          @min_score = min_score
+        end
+
+        def call
+          evals = @step.eval_names
+          return empty_result(evals) if evals.empty?
+
+          score_matrix = {}
+          evals.each do |eval_name|
+            comparison = with_retry_disabled do
+              @step.compare_models(eval_name, candidates: @candidates, context: @context)
+            end
+            score_matrix[eval_name] = extract_scores(comparison)
+          end
+
+          labels = score_matrix.values.flat_map(&:keys).uniq
+          constraining = find_constraining_eval(score_matrix, labels)
+          chain, details = build_chain(score_matrix, labels, evals)
+
+          Result.new(
+            step_name: @step.name || @step.to_s,
+            eval_names: evals,
+            candidate_labels: labels,
+            score_matrix: score_matrix,
+            constraining_eval: constraining,
+            chain: chain,
+            chain_details: details
+          )
+        end
+
+        private
+
+        def extract_scores(comparison)
+          comparison.reports.transform_values(&:score)
+        end
+
+        def find_constraining_eval(matrix, labels)
+          matrix.max_by do |_eval_name, scores|
+            cheapest_passing = labels.find { |l| (scores[l] || 0) >= @min_score }
+            cheapest_passing ? labels.index(cheapest_passing) : labels.size
+          end&.first
+        end
+
+        # Retry escalates on validation_failed/parse_error, NOT on low eval
+        # score. A model that returns :ok with semantically wrong output won't
+        # trigger retry. Therefore the LAST model in the chain must pass ALL
+        # evals — it's the safety net. Cheaper models are prepended as
+        # first-try optimization (they handle easy inputs cheaply; when they
+        # fail validation, retry escalates to the safe fallback).
+        #
+        # Known limitation: intermediate models are assumed safe if their eval
+        # failures correspond to validation failures (retryable). If an
+        # intermediate model returns :ok with semantically wrong output on
+        # some eval, retry won't fire and the safe fallback won't run. This
+        # requires step validates to cover the same semantics as eval verify
+        # checks. A future version could inspect per-case step_status from
+        # compare_models to verify failures are actually retryable.
+        def build_chain(matrix, labels, evals)
+          total = evals.size
+
+          # Find cheapest model that passes every eval — the safe fallback.
+          safe_fallback = labels.find { |l| evals.all? { |e| (matrix.dig(e, l) || 0) >= @min_score } }
+          return [[], []] unless safe_fallback
+
+          # Prepend cheaper models that pass a strict subset.
+          chain = []
+          details = []
+          covered_evals = Set.new
+
+          labels.each do |label|
+            break if label == safe_fallback
+
+            newly_covered = evals.select { |e| (matrix.dig(e, label) || 0) >= @min_score }
+            new_additions = newly_covered.to_set - covered_evals
+            next if new_additions.empty?
+
+            covered_evals.merge(new_additions)
+            chain << parse_label_to_config(label)
+            details << { label: label, passes: new_additions.size, cost: label }
+          end
+
+          # Always end with the safe fallback.
+          chain << parse_label_to_config(safe_fallback)
+          details << { label: safe_fallback, passes: total, cost: safe_fallback }
+
+          [chain, details]
+        end
+
+        def parse_label_to_config(label)
+          if label.match?(/\(effort: (\w+)\)/)
+            model = label.sub(/\s*\(effort:.*/, "").strip
+            effort = label.match(/\(effort: (\w+)\)/)[1]
+            { model: model, reasoning_effort: effort }
+          else
+            { model: label }
+          end
+        end
+
+        def with_retry_disabled(&block)
+          original = @step.retry_policy if @step.respond_to?(:retry_policy)
+          @step.define_singleton_method(:retry_policy) { nil }
+          block.call
+        ensure
+          @step.define_singleton_method(:retry_policy) { original }
+        end
+
+        def empty_result(evals)
+          Result.new(
+            step_name: @step.name || @step.to_s,
+            eval_names: evals,
+            candidate_labels: [],
+            score_matrix: {},
+            constraining_eval: nil,
+            chain: [],
+            chain_details: []
+          )
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/eval.rb

@@ -31,3 +31,4 @@ require_relative "eval/prompt_diff"
 require_relative "eval/eval_history"
 require_relative "eval/recommendation"
 require_relative "eval/recommender"
+require_relative "eval/retry_optimizer"

data/lib/ruby_llm/contract/rake_task.rb

@@ -121,5 +121,88 @@ module RubyLLM
         defined?(::Rails) ? [:environment] : []
       end
     end
+
+    # Standalone task: runs all evals for one step across candidates,
+    # builds a score matrix, and suggests an optimal retry chain.
+    #
+    # Loaded automatically when `require "ruby_llm/contract/rake_task"`.
+    # Usage:
+    #   rake ruby_llm_contract:optimize \
+    #     STEP=MatchProblemsToPages \
+    #     CANDIDATES=gpt-5-nano,gpt-5-mini@low,gpt-5-mini
+    class OptimizeRakeTask < ::Rake::TaskLib
+      def initialize
+        super()
+        define_task
+      end
+
+      private
+
+      def define_task
+        desc "Run all evals for STEP with CANDIDATES and suggest an optimal retry chain"
+        task(:"ruby_llm_contract:optimize" => task_prerequisites) do
+          require "ruby_llm/contract"
+          eval_dirs = ENV["EVAL_DIRS"].to_s.split(",").map(&:strip).reject(&:empty?)
+          RubyLLM::Contract.load_evals!(*eval_dirs)
+
+          step_name = ENV["STEP"].to_s.strip
+          abort("STEP is required, e.g. STEP=MatchProblemsToPages") if step_name.empty?
+          raw_candidates = ENV["CANDIDATES"].to_s.strip
+          abort("CANDIDATES is required, e.g. CANDIDATES=gpt-5-nano,gpt-5-mini@low,gpt-5-mini") if raw_candidates.empty?
+          min_score = ENV.fetch("MIN_SCORE", "0.95").to_f
+
+          host = RubyLLM::Contract.eval_hosts.find { |h| h.name == step_name }
+          unless host
+            available = RubyLLM::Contract.eval_hosts.filter_map(&:name).sort
+            abort "Unknown STEP=#{step_name}. Available: #{available.join(", ")}"
+          end
+
+          candidates = parse_candidates(raw_candidates)
+          context = build_context
+
+          result = host.optimize_retry_policy(
+            candidates: candidates,
+            context: context,
+            min_score: min_score
+          )
+
+          result.print_summary
+        end
+      end
+
+      def parse_candidates(raw)
+        entries = if raw.start_with?("[")
+                    Array(JSON.parse(raw))
+                  else
+                    raw.split(",").map(&:strip).reject(&:empty?).map do |entry|
+                      model, effort = entry.split("@", 2)
+                      config = { model: model.strip }
+                      config[:reasoning_effort] = effort.strip if effort && !effort.empty?
+                      config
+                    end
+                  end
+
+        entries.map { |e| RubyLLM::Contract.normalize_candidate_config(e) }.uniq
+      end
+
+      def build_context
+        ctx = {}
+        provider = ENV["PROVIDER"].to_s.strip
+        # Only inject real adapter when LIVE=1 or PROVIDER is set — otherwise
+        # evals use sample_response (offline mode, zero API calls).
+        if ENV["LIVE"] == "1" || !provider.empty?
+          ctx[:adapter] = RubyLLM::Contract::Adapters::RubyLLM.new
+          ctx[:provider] = provider.downcase.to_sym unless provider.empty?
+        end
+        ctx
+      end
+
+      def task_prerequisites
+        defined?(::Rails) ? [:environment] : []
+      end
+    end
+
+    # Auto-register the optimize task when this file is loaded
+    OptimizeRakeTask.new
   end
 end

data/lib/ruby_llm/contract/step/base.rb

@@ -59,6 +59,15 @@ module RubyLLM
             ).recommend
           end
 
+          def optimize_retry_policy(candidates:, context: {}, min_score: 0.95)
+            Eval::RetryOptimizer.new(
+              step: self,
+              candidates: candidates,
+              context: context,
+              min_score: min_score
+            ).call
+          end
+
           KNOWN_CONTEXT_KEYS = %i[adapter model temperature max_tokens provider assume_model_exists reasoning_effort].freeze
 
           include Concerns::ContextHelpers

data/lib/ruby_llm/contract/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module Contract
-    VERSION = "0.6.0"
+    VERSION = "0.6.2"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-contract
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.6.2
 platform: ruby
 authors:
 - Justyna
@@ -136,6 +136,7 @@ files:
 - lib/ruby_llm/contract/eval/report_presenter.rb
 - lib/ruby_llm/contract/eval/report_stats.rb
 - lib/ruby_llm/contract/eval/report_storage.rb
+- lib/ruby_llm/contract/eval/retry_optimizer.rb
 - lib/ruby_llm/contract/eval/runner.rb
 - lib/ruby_llm/contract/eval/step_expectation_applier.rb
 - lib/ruby_llm/contract/eval/step_result_normalizer.rb