ruby_llm-contract 1.0.0 → 1.1.1

data/README.md

@@ -168,7 +168,7 @@ Different layers, complementary. [`ruby_llm-tribunal`](https://github.com/Alqemi
 
 ## Status & versioning
 
-Stable at **1.0.0**. Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 1.0`.
+Stable since **1.0.0**. Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 1.0`.
 
 What 1.0 promises not to break inside the 1.x line: the documented DSL
 (`prompt`, `output_schema`, `validate`, `retry_policy`, `max_cost`, `define_eval`,

data/docs/guide/getting_started.md

@@ -145,7 +145,7 @@ report.save_baseline!
 expect(SummarizeArticle).to pass_eval("regression").without_regressions
 ```
 
-`without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality.
+`without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality. A previously-passing case that was skipped this run (no adapter configured) also fails the gate, listed as `SKIPPED` rather than as a regression.
 
 ## Budget caps
 
@@ -173,6 +173,8 @@ max_cost 0.01, on_unknown_pricing: :warn
 
 Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
 
+The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. The check sees only reported tokens: an adapter that returns no token counts looks like a zero-token run.
+
 ### Preflight cost estimates
 
 Check what a call is likely to cost before invoking it:

data/docs/guide/testing.md

@@ -160,7 +160,7 @@ end
 
 - `.with_context(model: "gpt-4.1-mini")` — pick model / pass adapter
 - `.with_minimum_score(0.8)` — gate on average score
-- `.with_maximum_cost(0.01)` — gate on total cost
+- `.with_maximum_cost(0.01)` — gate on total cost; fails when a case's model has no pricing data unless you add `.on_unknown_pricing(:warn)`
 - `.without_regressions` — block any previously-passing case that now fails (reads the baseline)
 - `.compared_with(SummarizeArticleV1)` — A/B against another step; implies regression check
 

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -16,13 +16,13 @@ module RubyLLM
           @file_sourced_evals ||= Set.new
           key = name.to_s
 
-          if @eval_definitions.key?(key) && !Thread.current[:ruby_llm_contract_reloading]
+          if @eval_definitions.key?(key) && !Contract.reloading?
             warn "[ruby_llm-contract] Redefining eval '#{key}' on #{self}. " \
                  "This replaces the previous definition."
           end
 
           @eval_definitions[key] = Eval::EvalDefinition.new(key, step_class: self, &)
-          @file_sourced_evals.add(key) if Thread.current[:ruby_llm_contract_reloading]
+          @file_sourced_evals.add(key) if Contract.reloading?
           Contract.register_eval_host(self)
           register_subclasses(self)
         end

data/lib/ruby_llm/contract/cost_calculator.rb

@@ -97,10 +97,11 @@ module RubyLLM
       # moved provider pricing into a nested value object and dropped those flat
       # readers, so it has to be walked: pricing -> text_tokens -> standard.
       #
-      # nil means "unknown pricing", which `max_cost` fails closed on. The guards
-      # keep the two known shapes explicit, but `calculate` rescues StandardError,
-      # so a future shape this walk cannot read still degrades to nil rather than
-      # surfacing - every budgeted call would be refused with no error.
+      # nil means "unknown pricing". The guards keep the two known shapes
+      # explicit, but `calculate` rescues StandardError, so a future shape this
+      # walk cannot read also degrades to nil: `max_cost` then refuses the call
+      # citing "no pricing data" rather than the real cause, and paths that floor
+      # unknown cost to 0.0 (eval cost estimates, report totals) undercount.
       def self.prices_for(model_info)
         if model_info.respond_to?(:input_price_per_million)
           return { input: model_info.input_price_per_million,

data/lib/ruby_llm/contract/eval/aggregated_report.rb

@@ -44,6 +44,10 @@ module RubyLLM
           @runs.sum(&:total_cost) / @runs.length.to_f
         end
 
+        def unknown_cost_results
+          @runs.flat_map(&:unknown_cost_results)
+        end
+
         def avg_latency_ms
           latencies = @runs.filter_map(&:avg_latency_ms)
           return nil if latencies.empty?

data/lib/ruby_llm/contract/eval/baseline_diff.rb

@@ -19,6 +19,7 @@ module RubyLLM
             current = @current[name]
             next unless current
             next unless baseline[:passed] && !current[:passed]
+            next if skipped?(current)
 
             {
               case: name,
@@ -47,8 +48,19 @@ module RubyLLM
           (current_score - baseline_score).round(4)
         end
 
+        # Passed in the baseline, not run now (no adapter): not evidence of a
+        # regression, but not evidence it still passes either, so the gate fails.
+        def skipped_passing_cases
+          @baseline.filter_map do |name, baseline|
+            current = @current[name]
+            next unless current && baseline[:passed] && skipped?(current)
+
+            { case: name, detail: current[:details] }
+          end
+        end
+
         def regressed?
-          regressions.any? || removed_passing_cases.any?
+          regressions.any? || removed_passing_cases.any? || skipped_passing_cases.any?
         end
 
         def removed_passing_cases
@@ -70,6 +82,7 @@ module RubyLLM
         def to_s
           lines = ["Score: #{baseline_score.round(2)} → #{current_score.round(2)} (#{format_delta})"]
           regressions.each { |r| lines << "  REGRESSED  #{r[:case]}: #{r[:detail]}" }
+          skipped_passing_cases.each { |s| lines << "  SKIPPED    #{s[:case]}: #{s[:detail]}" }
           improvements.each { |r| lines << "  IMPROVED   #{r[:case]}" }
           new_cases.each { |c| lines << "  NEW        #{c}" }
           removed_cases.each { |c| lines << "  REMOVED    #{c}" }
@@ -82,12 +95,16 @@ module RubyLLM
           # Asymmetric on purpose: the baseline side arrives already filtered by
           # ReportStats#evaluated_results, the current side does not, so this
           # string check is the only skip filter on one of the two sides.
-          evaluated = cases.reject { |c| c[:details]&.start_with?(CaseResult::SKIPPED_DETAILS_PREFIX) }
+          evaluated = cases.reject { |c| skipped?(c) }
           return 0.0 if evaluated.empty?
 
           evaluated.sum { |c| c[:score] } / evaluated.length
         end
 
+        def skipped?(serialized_case)
+          serialized_case[:details]&.start_with?(CaseResult::SKIPPED_DETAILS_PREFIX)
+        end
+
         def index_by_name(cases)
           cases.each_with_object({}) { |c, h| h[c[:name]] = c }
         end

data/lib/ruby_llm/contract/eval/case_executor.rb

@@ -39,7 +39,7 @@ module RubyLLM
             input: test_case.input,
             output: nil,
             expected: test_case.expected,
-            step_status: :skipped,
+            step_status: CaseResult::SKIPPED_STATUS,
             score: 0.0,
             passed: false,
             label: "SKIP",

data/lib/ruby_llm/contract/eval/case_result.rb

@@ -7,6 +7,9 @@ module RubyLLM
         # BaselineDiff#compute_score keys off this prefix to keep skipped cases
         # out of the score denominator. Both ends must read it from here.
         SKIPPED_DETAILS_PREFIX = "skipped:"
+        # Not run (no adapter). Readers compare step_status, so any result object
+        # exposing it works; kept out of score, pass rate and cost-per-call.
+        SKIPPED_STATUS = :skipped
 
         def self.skipped_details(reason)
           "#{SKIPPED_DETAILS_PREFIX} #{reason}"
@@ -15,8 +18,9 @@ module RubyLLM
         attr_reader :name, :input, :output, :expected, :step_status,
                     :score, :details, :duration_ms, :cost, :attempts
 
-        def initialize(name:, input:, output:, expected:, step_status:,
-                       score:, passed:, label: nil, details: nil, duration_ms: nil, cost: nil, attempts: nil)
+        def initialize(name:, input:, output:, expected:, step_status:, # rubocop:disable Metrics/ParameterLists
+                       score:, passed:, label: nil, details: nil, duration_ms: nil, cost: nil, attempts: nil,
+                       cost_unknown: false)
           @name = name
           @input = input
           @output = output
@@ -29,6 +33,7 @@ module RubyLLM
           @duration_ms = duration_ms
           @cost = cost
           @attempts = attempts
+          @cost_unknown = cost_unknown
           freeze
         end
 
@@ -40,6 +45,10 @@ module RubyLLM
           !@passed
         end
 
+        def cost_unknown?
+          @cost_unknown
+        end
+
         def label
           @label || (@passed ? "PASS" : "FAIL")
         end
@@ -69,7 +78,7 @@ module RubyLLM
             duration_ms: @duration_ms,
             cost: @cost,
             attempts: @attempts
-          }
+          }.tap { |hash| hash[:cost_unknown] = true if @cost_unknown }
         end
 
         private

data/lib/ruby_llm/contract/eval/case_result_builder.rb

@@ -19,7 +19,8 @@ module RubyLLM
             details: evaluation.details,
             duration_ms: trace_metric(trace, :total_latency_ms, :latency_ms),
             cost: trace_metric(trace, :total_cost, :cost),
-            attempts: trace_attempts(trace)
+            attempts: trace_attempts(trace),
+            cost_unknown: trace.respond_to?(:cost_unknown?) && trace.cost_unknown?
           )
         end
 

data/lib/ruby_llm/contract/eval/evaluator/proc_evaluator.rb

@@ -6,6 +6,10 @@ module RubyLLM
       module Evaluator
         # Adapts custom Ruby callables to the EvaluationResult contract.
         class ProcEvaluator
+          # Report hides these under a failure: they repeat the PASS/FAIL label.
+          PASSED_DETAILS = "passed"
+          FAILED_DETAILS = "not passed"
+
           def initialize(callable)
             @callable = callable
           end
@@ -36,9 +40,9 @@ module RubyLLM
           def build_evaluation_result(result)
             case result
             when true
-              EvaluationResult.new(score: 1.0, passed: true, details: "passed")
+              EvaluationResult.new(score: 1.0, passed: true, details: PASSED_DETAILS)
             when false
-              EvaluationResult.new(score: 0.0, passed: false, details: "not passed")
+              EvaluationResult.new(score: 0.0, passed: false, details: FAILED_DETAILS)
             when Numeric
               EvaluationResult.new(score: result, passed: result >= 0.5, details: "custom score: #{result}")
             else

data/lib/ruby_llm/contract/eval/prompt_diff_serializer.rb

@@ -5,7 +5,7 @@ module RubyLLM
     module Eval
       class PromptDiffSerializer
         def call(report)
-          report.results.reject { |result| result.step_status == :skipped }.map do |result|
+          report.results.reject { |result| result.step_status == CaseResult::SKIPPED_STATUS }.map do |result|
             {
               name: result.name,
               input: result.input,

data/lib/ruby_llm/contract/eval/recommender.rb

@@ -40,7 +40,7 @@ module RubyLLM
             report = @comparison.reports[label]
             next nil unless report
 
-            evaluated_count = report.results.count { |r| r.step_status != :skipped }
+            evaluated_count = report.results.count { |r| r.step_status != CaseResult::SKIPPED_STATUS }
             cases_count = [evaluated_count, 1].max
             cost_per_call = report.total_cost.to_f / cases_count
 
@@ -116,7 +116,7 @@ module RubyLLM
           current_report = @comparison.reports[current_label]
           return {} unless current_report
 
-          current_evaluated = current_report.results.count { |r| r.step_status != :skipped }
+          current_evaluated = current_report.results.count { |r| r.step_status != CaseResult::SKIPPED_STATUS }
           current_cases = [current_evaluated, 1].max
           current_cost = current_report.total_cost.to_f / current_cases
           diff = current_cost - best[:cost_per_call]

data/lib/ruby_llm/contract/eval/report.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "forwardable"
+require_relative "evaluator/proc_evaluator"
 
 module RubyLLM
   module Contract
@@ -10,12 +11,14 @@ module RubyLLM
 
         attr_reader :dataset_name, :results, :step_name
 
-        GENERIC_DETAILS = ["passed", "not passed"].freeze
+        GENERIC_DETAILS = [Evaluator::ProcEvaluator::PASSED_DETAILS, Evaluator::ProcEvaluator::FAILED_DETAILS].freeze
         HISTORY_DIR = ".eval_history"
+        HISTORY_EXT = "jsonl"
         BASELINE_DIR = ".eval_baselines"
+        BASELINE_EXT = "json"
 
         def_delegators :@stats, :score, :passed, :failed, :skipped, :failures, :pass_rate, :pass_rate_ratio,
-                       :total_cost, :avg_latency_ms, :passed?,
+                       :total_cost, :unknown_cost_results, :avg_latency_ms, :passed?,
                        :production_mode?, :escalation_rate, :single_shot_cost, :single_shot_latency_ms,
                        :effective_cost, :effective_latency_ms, :latency_percentiles
         def_delegators :@presenter, :summary, :to_s, :print_summary

data/lib/ruby_llm/contract/eval/report_stats.rb

@@ -23,7 +23,7 @@ module RubyLLM
         end
 
         def skipped
-          @results.count { |result| result.step_status == :skipped }
+          @results.count { |result| result.step_status == CaseResult::SKIPPED_STATUS }
         end
 
         def failures
@@ -44,6 +44,11 @@ module RubyLLM
           @results.sum { |result| result.cost || 0.0 }
         end
 
+        # Cases total_cost could not price; it counts them as 0.0.
+        def unknown_cost_results
+          evaluated_results.select(&:cost_unknown?)
+        end
+
         def avg_latency_ms
           latencies = @results.filter_map(&:duration_ms)
           return nil if latencies.empty?
@@ -58,7 +63,7 @@ module RubyLLM
         end
 
         def evaluated_results
-          @evaluated_results ||= @results.reject { |result| result.step_status == :skipped }
+          @evaluated_results ||= @results.reject { |result| result.step_status == CaseResult::SKIPPED_STATUS }
         end
 
         def evaluated_results_count

data/lib/ruby_llm/contract/eval/report_storage.rb

@@ -13,7 +13,7 @@ module RubyLLM
         end
 
         def save_history!(path: nil, model: nil, reasoning_effort: nil)
-          file = path || storage_path(Report::HISTORY_DIR, "jsonl", model: model, reasoning_effort: reasoning_effort)
+          file = path || storage_path(Report::HISTORY_DIR, Report::HISTORY_EXT, model: model, reasoning_effort: reasoning_effort)
           entry = history_entry
           entry[:model] = model if model
           entry[:reasoning_effort] = reasoning_effort if reasoning_effort
@@ -22,19 +22,19 @@ module RubyLLM
         end
 
         def eval_history(path: nil, model: nil, reasoning_effort: nil)
-          EvalHistory.load(path || storage_path(Report::HISTORY_DIR, "jsonl", model: model,
-                                                                              reasoning_effort: reasoning_effort))
+          EvalHistory.load(path || storage_path(Report::HISTORY_DIR, Report::HISTORY_EXT,
+                                                model: model, reasoning_effort: reasoning_effort))
         end
 
         def save_baseline!(path: nil, model: nil, reasoning_effort: nil)
-          file = path || storage_path(Report::BASELINE_DIR, "json", model: model, reasoning_effort: reasoning_effort)
+          file = path || storage_path(Report::BASELINE_DIR, Report::BASELINE_EXT, model: model, reasoning_effort: reasoning_effort)
           FileUtils.mkdir_p(File.dirname(file))
           File.write(file, JSON.pretty_generate(serialize_for_baseline))
           file
         end
 
         def compare_with_baseline(path: nil, model: nil, reasoning_effort: nil)
-          file = path || storage_path(Report::BASELINE_DIR, "json", model: model, reasoning_effort: reasoning_effort)
+          file = path || storage_path(Report::BASELINE_DIR, Report::BASELINE_EXT, model: model, reasoning_effort: reasoning_effort)
           raise ArgumentError, "No baseline found at #{file}" unless File.exist?(file)
 
           baseline_data = JSON.parse(File.read(file), symbolize_names: true)
@@ -47,7 +47,7 @@ module RubyLLM
         end
 
         def baseline_exists?(path: nil, model: nil, reasoning_effort: nil)
-          File.exist?(path || storage_path(Report::BASELINE_DIR, "json", model: model, reasoning_effort: reasoning_effort))
+          File.exist?(path || storage_path(Report::BASELINE_DIR, Report::BASELINE_EXT, model: model, reasoning_effort: reasoning_effort))
         end
 
         private

data/lib/ruby_llm/contract/eval/step_expectation_applier.rb

@@ -58,7 +58,8 @@ module RubyLLM
             label: "FAIL",
             details: "step expectations failed: #{failure_details}",
             duration_ms: result.duration_ms,
-            cost: result.cost
+            cost: result.cost,
+            cost_unknown: result.cost_unknown?
           )
         end
       end

data/lib/ruby_llm/contract/eval/unknown_cost_gate.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Eval
+      # Suite-level counterpart of the step-level `max_cost ... on_unknown_pricing:`.
+      # A report's total_cost counts an unpriced case as 0.0, so a maximum_cost
+      # gate that only compares totals passes whatever those cases really cost.
+      # Shared by the rake task, the pass_eval matcher and assert_eval_passes.
+      module UnknownCostGate
+        # Returns the failure message under :refuse, nil when the gate holds.
+        def self.check(reports, mode:)
+          names = reports.flat_map(&:unknown_cost_results).map(&:name).uniq
+          return nil if names.empty?
+
+          if mode == :warn
+            warn "[ruby_llm-contract] #{describe(names)} - maximum_cost checked against priced cases only"
+            return nil
+          end
+
+          "#{describe(names)}. Register pricing via CostCalculator.register_model " \
+            "or set on_unknown_pricing to :warn to check maximum_cost against priced cases only."
+        end
+
+        def self.describe(names)
+          "#{names.length} case(s) have unknown cost (model has no pricing data): #{names.join(", ")}"
+        end
+        private_class_method :describe
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/eval.rb

@@ -22,6 +22,7 @@ require_relative "eval/report_presenter"
 require_relative "eval/report_storage"
 require_relative "eval/report"
 require_relative "eval/aggregated_report"
+require_relative "eval/unknown_cost_gate"
 require_relative "eval/eval_definition"
 require_relative "eval/candidate_label"
 require_relative "eval/model_comparison"

data/lib/ruby_llm/contract/minitest.rb

@@ -30,7 +30,9 @@ module RubyLLM
         refute result.ok?, msg || "Expected step result NOT to satisfy contract, but it passed"
       end
 
-      def assert_eval_passes(step, eval_name, minimum_score: nil, maximum_cost: nil, context: {}, msg: nil)
+      def assert_eval_passes(step, eval_name, minimum_score: nil, maximum_cost: nil, context: {}, msg: nil,
+                             on_unknown_pricing: UnknownPolicy::DEFAULT)
+        UnknownPolicy.validate!("on_unknown_pricing", on_unknown_pricing)
         report = step.run_eval(eval_name, context: context)
 
         if minimum_score
@@ -42,6 +44,9 @@ module RubyLLM
         end
 
         if maximum_cost
+          unknown_cost_failure = Eval::UnknownCostGate.check([report], mode: on_unknown_pricing)
+          assert unknown_cost_failure.nil?,
+                 msg || "Expected #{eval_name} eval to stay within maximum cost, but #{unknown_cost_failure}"
           assert report.total_cost <= maximum_cost,
                  msg || "Expected #{eval_name} eval cost <= $#{format("%.4f", maximum_cost)}, got $#{format("%.4f", report.total_cost)}"
         end

data/lib/ruby_llm/contract/pipeline/trace.rb

@@ -33,6 +33,13 @@ module RubyLLM
           value.dig(*rest)
         end
 
+        # total_cost sums only the steps that could be priced.
+        def cost_unknown?
+          return false unless @step_traces.is_a?(Array)
+
+          @step_traces.any? { |step_trace| step_trace.respond_to?(:cost_unknown?) && step_trace.cost_unknown? }
+        end
+
         def to_h
           { trace_id: @trace_id, total_latency_ms: @total_latency_ms,
             total_usage: @total_usage, step_traces: @step_traces,

data/lib/ruby_llm/contract/railtie.rb

@@ -6,7 +6,7 @@ module RubyLLM
       # Ignore eval/ subdirs BEFORE Zeitwerk setup — eval files don't define
       # constants, they call define_eval on existing Step classes.
       initializer "ruby_llm_contract.ignore_eval_dirs", before: :set_autoload_paths do |app|
-        %w[app/contracts/eval app/steps/eval].each do |path|
+        RubyLLM::Contract::RAILS_EVAL_DIRS.each do |path|
           full = app.root.join(path)
           next unless full.exist?
 

data/lib/ruby_llm/contract/rake_task/suite_gate.rb

@@ -4,8 +4,9 @@ module RubyLLM
   module Contract
     class RakeTask < ::Rake::TaskLib
       # Gate ordering (preserved from pre-refactor behaviour):
-      #   1. cost gate runs FIRST — if `maximum_cost` set and exceeded, the
-      #      suite aborts before any score check; passed_reports is empty.
+      #   1. cost gate runs FIRST — if `maximum_cost` set and exceeded, or any
+      #      case has unknown cost under on_unknown_pricing :refuse, the suite
+      #      aborts before any score check; passed_reports is empty.
       #   2. score gate runs per-report; a report passes if
       #      `report_meets_score?` AND `!check_regression`.
       #   3. overall passed = ALL reports passed AND cost gate not tripped.
@@ -16,20 +17,24 @@ module RubyLLM
           end
         end
 
-        def self.evaluate(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:)
+        def self.evaluate(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:,
+                          on_unknown_pricing: UnknownPolicy::DEFAULT)
           new(host_reports: host_reports,
               minimum_score: minimum_score,
               maximum_cost: maximum_cost,
-              fail_on_regression: fail_on_regression).verdict
+              fail_on_regression: fail_on_regression,
+              on_unknown_pricing: on_unknown_pricing).verdict
         end
 
         attr_reader :verdict
 
-        def initialize(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:)
+        def initialize(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:,
+                       on_unknown_pricing: UnknownPolicy::DEFAULT)
           @host_reports = host_reports
           @minimum_score = minimum_score
           @maximum_cost = maximum_cost
           @fail_on_regression = fail_on_regression
+          @on_unknown_pricing = UnknownPolicy.validate!("on_unknown_pricing", on_unknown_pricing)
           @verdict = build_verdict
         end
 
@@ -37,11 +42,12 @@ module RubyLLM
 
         def build_verdict
           suite_cost = compute_suite_cost
+          cost_failure = cost_failure_message(suite_cost)
 
-          if cost_exceeded?(suite_cost)
+          if cost_failure
             return Verdict.new(
               passed: false,
-              abort_reason: cost_abort_message(suite_cost),
+              abort_reason: cost_failure,
               passed_reports: [],
               suite_cost: suite_cost
             )
@@ -50,7 +56,7 @@ module RubyLLM
           passed_reports, all_passed = score_each_report
           Verdict.new(
             passed: all_passed,
-            abort_reason: all_passed ? nil : "Eval suite FAILED",
+            abort_reason: all_passed ? nil : "one or more evals did not pass",
             passed_reports: passed_reports,
             suite_cost: suite_cost
           )
@@ -60,8 +66,11 @@ module RubyLLM
           @host_reports.sum { |_host, report| report.total_cost }
         end
 
-        def cost_exceeded?(suite_cost)
-          @maximum_cost && suite_cost > @maximum_cost
+        def cost_failure_message(suite_cost)
+          return nil unless @maximum_cost
+          return cost_abort_message(suite_cost) if suite_cost > @maximum_cost
+
+          Eval::UnknownCostGate.check(@host_reports.map(&:last), mode: @on_unknown_pricing)
         end
 
         def cost_abort_message(suite_cost)

data/lib/ruby_llm/contract/rake_task.rb

@@ -2,13 +2,15 @@
 
 require "rake"
 require "rake/tasklib"
+require_relative "unknown_policy"
 require_relative "rake_task/suite_gate"
 
 module RubyLLM
   module Contract
     class RakeTask < ::Rake::TaskLib
       attr_accessor :name, :context, :fail_on_empty, :minimum_score, :maximum_cost,
-                    :eval_dirs, :save_baseline, :fail_on_regression, :track_history
+                    :eval_dirs, :save_baseline, :fail_on_regression, :track_history,
+                    :on_unknown_pricing
 
       def initialize(name = :"ruby_llm_contract:eval", &block)
         super()
@@ -18,10 +20,13 @@ module RubyLLM
         @minimum_score = nil # nil = require 100%; float = threshold
         @maximum_cost = nil  # nil = no cost limit; float = budget cap (suite-level)
         @eval_dirs = []      # directories to load eval files from (non-Rails)
+        # With maximum_cost: :refuse fails on cases the budget cannot price; :warn gates priced cases only
+        @on_unknown_pricing = UnknownPolicy::DEFAULT
         @save_baseline = false
         @fail_on_regression = false
         @track_history = false
         block&.call(self)
+        UnknownPolicy.validate!("on_unknown_pricing", @on_unknown_pricing)
         define_task
       end
 
@@ -44,7 +49,8 @@ module RubyLLM
             host_reports: host_reports,
             minimum_score: @minimum_score,
             maximum_cost: @maximum_cost,
-            fail_on_regression: @fail_on_regression
+            fail_on_regression: @fail_on_regression,
+            on_unknown_pricing: @on_unknown_pricing
           )
 
           abort "\nEval suite FAILED: #{verdict.abort_reason}" unless verdict.passed?
@@ -128,7 +134,7 @@ module RubyLLM
           abort("STEP is required, e.g. STEP=MatchProblemsToPages") if step_name.empty?
           raw_candidates = ENV["CANDIDATES"].to_s.strip
           abort("CANDIDATES is required, e.g. CANDIDATES=gpt-5-nano,gpt-5-mini@low,gpt-5-mini") if raw_candidates.empty?
-          min_score = ENV.fetch("MIN_SCORE", "0.95").to_f
+          min_score = ENV.fetch("MIN_SCORE") { Eval::DEFAULT_MIN_SCORE }.to_f
           runs = parse_runs(ENV.fetch("RUNS", "1"))
 
           host = RubyLLM::Contract.eval_hosts.find { |h| h.name == step_name }