ruby_llm-contract 1.0.0 → 1.1.0

data/README.md

@@ -168,7 +168,7 @@ Different layers, complementary. [`ruby_llm-tribunal`](https://github.com/Alqemi
 
 ## Status & versioning
 
-Stable at **1.0.0**. Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 1.0`.
+Stable since **1.0.0**. Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 1.0`.
 
 What 1.0 promises not to break inside the 1.x line: the documented DSL
 (`prompt`, `output_schema`, `validate`, `retry_policy`, `max_cost`, `define_eval`,

data/docs/guide/getting_started.md

@@ -145,7 +145,7 @@ report.save_baseline!
 expect(SummarizeArticle).to pass_eval("regression").without_regressions
 ```
 
-`without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality.
+`without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality. A previously-passing case that was skipped this run (no adapter configured) also fails the gate, listed as `SKIPPED` rather than as a regression.
 
 ## Budget caps
 
@@ -173,6 +173,8 @@ max_cost 0.01, on_unknown_pricing: :warn
 
 Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
 
+The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. The check sees only reported tokens: an adapter that returns no token counts looks like a zero-token run.
+
 ### Preflight cost estimates
 
 Check what a call is likely to cost before invoking it:

data/docs/guide/testing.md

@@ -160,7 +160,7 @@ end
 
 - `.with_context(model: "gpt-4.1-mini")` — pick model / pass adapter
 - `.with_minimum_score(0.8)` — gate on average score
-- `.with_maximum_cost(0.01)` — gate on total cost
+- `.with_maximum_cost(0.01)` — gate on total cost; fails when a case's model has no pricing data unless you add `.on_unknown_pricing(:warn)`
 - `.without_regressions` — block any previously-passing case that now fails (reads the baseline)
 - `.compared_with(SummarizeArticleV1)` — A/B against another step; implies regression check
 

data/lib/ruby_llm/contract/cost_calculator.rb

@@ -97,10 +97,11 @@ module RubyLLM
       # moved provider pricing into a nested value object and dropped those flat
       # readers, so it has to be walked: pricing -> text_tokens -> standard.
       #
-      # nil means "unknown pricing", which `max_cost` fails closed on. The guards
-      # keep the two known shapes explicit, but `calculate` rescues StandardError,
-      # so a future shape this walk cannot read still degrades to nil rather than
-      # surfacing - every budgeted call would be refused with no error.
+      # nil means "unknown pricing". The guards keep the two known shapes
+      # explicit, but `calculate` rescues StandardError, so a future shape this
+      # walk cannot read also degrades to nil: `max_cost` then refuses the call
+      # citing "no pricing data" rather than the real cause, and paths that floor
+      # unknown cost to 0.0 (eval cost estimates, report totals) undercount.
       def self.prices_for(model_info)
         if model_info.respond_to?(:input_price_per_million)
           return { input: model_info.input_price_per_million,

data/lib/ruby_llm/contract/eval/aggregated_report.rb

@@ -44,6 +44,10 @@ module RubyLLM
           @runs.sum(&:total_cost) / @runs.length.to_f
         end
 
+        def unknown_cost_results
+          @runs.flat_map(&:unknown_cost_results)
+        end
+
         def avg_latency_ms
           latencies = @runs.filter_map(&:avg_latency_ms)
           return nil if latencies.empty?

data/lib/ruby_llm/contract/eval/baseline_diff.rb

@@ -19,6 +19,7 @@ module RubyLLM
             current = @current[name]
             next unless current
             next unless baseline[:passed] && !current[:passed]
+            next if skipped?(current)
 
             {
               case: name,
@@ -47,8 +48,19 @@ module RubyLLM
           (current_score - baseline_score).round(4)
         end
 
+        # Passed in the baseline, not run now (no adapter): not evidence of a
+        # regression, but not evidence it still passes either, so the gate fails.
+        def skipped_passing_cases
+          @baseline.filter_map do |name, baseline|
+            current = @current[name]
+            next unless current && baseline[:passed] && skipped?(current)
+
+            { case: name, detail: current[:details] }
+          end
+        end
+
         def regressed?
-          regressions.any? || removed_passing_cases.any?
+          regressions.any? || removed_passing_cases.any? || skipped_passing_cases.any?
         end
 
         def removed_passing_cases
@@ -70,6 +82,7 @@ module RubyLLM
         def to_s
           lines = ["Score: #{baseline_score.round(2)} → #{current_score.round(2)} (#{format_delta})"]
           regressions.each { |r| lines << "  REGRESSED  #{r[:case]}: #{r[:detail]}" }
+          skipped_passing_cases.each { |s| lines << "  SKIPPED    #{s[:case]}: #{s[:detail]}" }
           improvements.each { |r| lines << "  IMPROVED   #{r[:case]}" }
           new_cases.each { |c| lines << "  NEW        #{c}" }
           removed_cases.each { |c| lines << "  REMOVED    #{c}" }
@@ -82,12 +95,16 @@ module RubyLLM
           # Asymmetric on purpose: the baseline side arrives already filtered by
           # ReportStats#evaluated_results, the current side does not, so this
           # string check is the only skip filter on one of the two sides.
-          evaluated = cases.reject { |c| c[:details]&.start_with?(CaseResult::SKIPPED_DETAILS_PREFIX) }
+          evaluated = cases.reject { |c| skipped?(c) }
           return 0.0 if evaluated.empty?
 
           evaluated.sum { |c| c[:score] } / evaluated.length
         end
 
+        def skipped?(serialized_case)
+          serialized_case[:details]&.start_with?(CaseResult::SKIPPED_DETAILS_PREFIX)
+        end
+
         def index_by_name(cases)
           cases.each_with_object({}) { |c, h| h[c[:name]] = c }
         end

data/lib/ruby_llm/contract/eval/case_result.rb

@@ -15,8 +15,9 @@ module RubyLLM
         attr_reader :name, :input, :output, :expected, :step_status,
                     :score, :details, :duration_ms, :cost, :attempts
 
-        def initialize(name:, input:, output:, expected:, step_status:,
-                       score:, passed:, label: nil, details: nil, duration_ms: nil, cost: nil, attempts: nil)
+        def initialize(name:, input:, output:, expected:, step_status:, # rubocop:disable Metrics/ParameterLists
+                       score:, passed:, label: nil, details: nil, duration_ms: nil, cost: nil, attempts: nil,
+                       cost_unknown: false)
           @name = name
           @input = input
           @output = output
@@ -29,6 +30,7 @@ module RubyLLM
           @duration_ms = duration_ms
           @cost = cost
           @attempts = attempts
+          @cost_unknown = cost_unknown
           freeze
         end
 
@@ -40,6 +42,10 @@ module RubyLLM
           !@passed
         end
 
+        def cost_unknown?
+          @cost_unknown
+        end
+
         def label
           @label || (@passed ? "PASS" : "FAIL")
         end
@@ -69,7 +75,7 @@ module RubyLLM
             duration_ms: @duration_ms,
             cost: @cost,
             attempts: @attempts
-          }
+          }.tap { |hash| hash[:cost_unknown] = true if @cost_unknown }
         end
 
         private

data/lib/ruby_llm/contract/eval/case_result_builder.rb

@@ -19,7 +19,8 @@ module RubyLLM
             details: evaluation.details,
             duration_ms: trace_metric(trace, :total_latency_ms, :latency_ms),
             cost: trace_metric(trace, :total_cost, :cost),
-            attempts: trace_attempts(trace)
+            attempts: trace_attempts(trace),
+            cost_unknown: trace.respond_to?(:cost_unknown?) && trace.cost_unknown?
           )
         end
 

data/lib/ruby_llm/contract/eval/report.rb

@@ -15,7 +15,7 @@ module RubyLLM
         BASELINE_DIR = ".eval_baselines"
 
         def_delegators :@stats, :score, :passed, :failed, :skipped, :failures, :pass_rate, :pass_rate_ratio,
-                       :total_cost, :avg_latency_ms, :passed?,
+                       :total_cost, :unknown_cost_results, :avg_latency_ms, :passed?,
                        :production_mode?, :escalation_rate, :single_shot_cost, :single_shot_latency_ms,
                        :effective_cost, :effective_latency_ms, :latency_percentiles
         def_delegators :@presenter, :summary, :to_s, :print_summary

data/lib/ruby_llm/contract/eval/report_stats.rb

@@ -44,6 +44,11 @@ module RubyLLM
           @results.sum { |result| result.cost || 0.0 }
         end
 
+        # Cases total_cost could not price; it counts them as 0.0.
+        def unknown_cost_results
+          evaluated_results.select(&:cost_unknown?)
+        end
+
         def avg_latency_ms
           latencies = @results.filter_map(&:duration_ms)
           return nil if latencies.empty?

data/lib/ruby_llm/contract/eval/step_expectation_applier.rb

@@ -58,7 +58,8 @@ module RubyLLM
             label: "FAIL",
             details: "step expectations failed: #{failure_details}",
             duration_ms: result.duration_ms,
-            cost: result.cost
+            cost: result.cost,
+            cost_unknown: result.cost_unknown?
           )
         end
       end

data/lib/ruby_llm/contract/eval/unknown_cost_gate.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Eval
+      # Suite-level counterpart of the step-level `max_cost ... on_unknown_pricing:`.
+      # A report's total_cost counts an unpriced case as 0.0, so a maximum_cost
+      # gate that only compares totals passes whatever those cases really cost.
+      # Shared by the rake task, the pass_eval matcher and assert_eval_passes.
+      module UnknownCostGate
+        MODES = %i[refuse warn].freeze
+
+        def self.validate!(mode)
+          return mode if MODES.include?(mode)
+
+          raise ArgumentError, "on_unknown_pricing must be :refuse or :warn, got #{mode.inspect}"
+        end
+
+        # Returns the failure message under :refuse, nil when the gate holds.
+        def self.check(reports, mode:)
+          names = reports.flat_map(&:unknown_cost_results).map(&:name).uniq
+          return nil if names.empty?
+
+          if mode == :warn
+            warn "[ruby_llm-contract] #{describe(names)} - maximum_cost checked against priced cases only"
+            return nil
+          end
+
+          "#{describe(names)}. Register pricing via CostCalculator.register_model " \
+            "or set on_unknown_pricing to :warn to check maximum_cost against priced cases only."
+        end
+
+        def self.describe(names)
+          "#{names.length} case(s) have unknown cost (model has no pricing data): #{names.join(", ")}"
+        end
+        private_class_method :describe
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/eval.rb

@@ -22,6 +22,7 @@ require_relative "eval/report_presenter"
 require_relative "eval/report_storage"
 require_relative "eval/report"
 require_relative "eval/aggregated_report"
+require_relative "eval/unknown_cost_gate"
 require_relative "eval/eval_definition"
 require_relative "eval/candidate_label"
 require_relative "eval/model_comparison"

data/lib/ruby_llm/contract/minitest.rb

@@ -30,7 +30,9 @@ module RubyLLM
         refute result.ok?, msg || "Expected step result NOT to satisfy contract, but it passed"
       end
 
-      def assert_eval_passes(step, eval_name, minimum_score: nil, maximum_cost: nil, context: {}, msg: nil)
+      def assert_eval_passes(step, eval_name, minimum_score: nil, maximum_cost: nil, context: {}, msg: nil,
+                             on_unknown_pricing: :refuse)
+        Eval::UnknownCostGate.validate!(on_unknown_pricing)
         report = step.run_eval(eval_name, context: context)
 
         if minimum_score
@@ -42,6 +44,9 @@ module RubyLLM
         end
 
         if maximum_cost
+          unknown_cost_failure = Eval::UnknownCostGate.check([report], mode: on_unknown_pricing)
+          assert unknown_cost_failure.nil?,
+                 msg || "Expected #{eval_name} eval to stay within maximum cost, but #{unknown_cost_failure}"
           assert report.total_cost <= maximum_cost,
                  msg || "Expected #{eval_name} eval cost <= $#{format("%.4f", maximum_cost)}, got $#{format("%.4f", report.total_cost)}"
         end

data/lib/ruby_llm/contract/pipeline/trace.rb

@@ -33,6 +33,13 @@ module RubyLLM
           value.dig(*rest)
         end
 
+        # total_cost sums only the steps that could be priced.
+        def cost_unknown?
+          return false unless @step_traces.is_a?(Array)
+
+          @step_traces.any? { |step_trace| step_trace.respond_to?(:cost_unknown?) && step_trace.cost_unknown? }
+        end
+
         def to_h
           { trace_id: @trace_id, total_latency_ms: @total_latency_ms,
             total_usage: @total_usage, step_traces: @step_traces,

data/lib/ruby_llm/contract/rake_task/suite_gate.rb

@@ -4,8 +4,9 @@ module RubyLLM
   module Contract
     class RakeTask < ::Rake::TaskLib
       # Gate ordering (preserved from pre-refactor behaviour):
-      #   1. cost gate runs FIRST — if `maximum_cost` set and exceeded, the
-      #      suite aborts before any score check; passed_reports is empty.
+      #   1. cost gate runs FIRST — if `maximum_cost` set and exceeded, or any
+      #      case has unknown cost under on_unknown_pricing :refuse, the suite
+      #      aborts before any score check; passed_reports is empty.
       #   2. score gate runs per-report; a report passes if
       #      `report_meets_score?` AND `!check_regression`.
       #   3. overall passed = ALL reports passed AND cost gate not tripped.
@@ -16,20 +17,24 @@ module RubyLLM
           end
         end
 
-        def self.evaluate(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:)
+        def self.evaluate(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:,
+                          on_unknown_pricing: :refuse)
           new(host_reports: host_reports,
               minimum_score: minimum_score,
               maximum_cost: maximum_cost,
-              fail_on_regression: fail_on_regression).verdict
+              fail_on_regression: fail_on_regression,
+              on_unknown_pricing: on_unknown_pricing).verdict
         end
 
         attr_reader :verdict
 
-        def initialize(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:)
+        def initialize(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:,
+                       on_unknown_pricing: :refuse)
           @host_reports = host_reports
           @minimum_score = minimum_score
           @maximum_cost = maximum_cost
           @fail_on_regression = fail_on_regression
+          @on_unknown_pricing = Eval::UnknownCostGate.validate!(on_unknown_pricing)
           @verdict = build_verdict
         end
 
@@ -37,11 +42,12 @@ module RubyLLM
 
         def build_verdict
           suite_cost = compute_suite_cost
+          cost_failure = cost_failure_message(suite_cost)
 
-          if cost_exceeded?(suite_cost)
+          if cost_failure
             return Verdict.new(
               passed: false,
-              abort_reason: cost_abort_message(suite_cost),
+              abort_reason: cost_failure,
               passed_reports: [],
               suite_cost: suite_cost
             )
@@ -60,8 +66,11 @@ module RubyLLM
           @host_reports.sum { |_host, report| report.total_cost }
         end
 
-        def cost_exceeded?(suite_cost)
-          @maximum_cost && suite_cost > @maximum_cost
+        def cost_failure_message(suite_cost)
+          return nil unless @maximum_cost
+          return cost_abort_message(suite_cost) if suite_cost > @maximum_cost
+
+          Eval::UnknownCostGate.check(@host_reports.map(&:last), mode: @on_unknown_pricing)
         end
 
         def cost_abort_message(suite_cost)

data/lib/ruby_llm/contract/rake_task.rb

@@ -2,13 +2,15 @@
 
 require "rake"
 require "rake/tasklib"
+require_relative "eval/unknown_cost_gate"
 require_relative "rake_task/suite_gate"
 
 module RubyLLM
   module Contract
     class RakeTask < ::Rake::TaskLib
       attr_accessor :name, :context, :fail_on_empty, :minimum_score, :maximum_cost,
-                    :eval_dirs, :save_baseline, :fail_on_regression, :track_history
+                    :eval_dirs, :save_baseline, :fail_on_regression, :track_history,
+                    :on_unknown_pricing
 
       def initialize(name = :"ruby_llm_contract:eval", &block)
         super()
@@ -18,10 +20,13 @@ module RubyLLM
         @minimum_score = nil # nil = require 100%; float = threshold
         @maximum_cost = nil  # nil = no cost limit; float = budget cap (suite-level)
         @eval_dirs = []      # directories to load eval files from (non-Rails)
+        # With maximum_cost: :refuse fails on cases the budget cannot price; :warn gates priced cases only
+        @on_unknown_pricing = :refuse
         @save_baseline = false
         @fail_on_regression = false
         @track_history = false
         block&.call(self)
+        Eval::UnknownCostGate.validate!(@on_unknown_pricing)
         define_task
       end
 
@@ -44,7 +49,8 @@ module RubyLLM
             host_reports: host_reports,
             minimum_score: @minimum_score,
             maximum_cost: @maximum_cost,
-            fail_on_regression: @fail_on_regression
+            fail_on_regression: @fail_on_regression,
+            on_unknown_pricing: @on_unknown_pricing
           )
 
           abort "\nEval suite FAILED: #{verdict.abort_reason}" unless verdict.passed?

data/lib/ruby_llm/contract/rspec/pass_eval.rb

@@ -64,6 +64,10 @@ RSpec::Matchers.define :pass_eval do |eval_name|
     @maximum_cost = cost
   end
 
+  chain :on_unknown_pricing do |mode|
+    @on_unknown_pricing = RubyLLM::Contract::Eval::UnknownCostGate.validate!(mode)
+  end
+
   chain :without_regressions do
     @check_regressions = true
   end
@@ -78,6 +82,9 @@ RSpec::Matchers.define :pass_eval do |eval_name|
     @context ||= {}
     @minimum_score ||= nil
     @maximum_cost ||= nil
+    @on_unknown_pricing ||= :refuse
+    @unknown_cost_failure = nil
+    @unknown_cost_only = false
     @check_regressions ||= false
     @comparison_step ||= nil
     @error = nil
@@ -97,7 +104,11 @@ RSpec::Matchers.define :pass_eval do |eval_name|
                  @report.passed?
                end
 
-    cost_ok = @maximum_cost ? @report.total_cost <= @maximum_cost : true
+    cost_ok = true
+    if @maximum_cost
+      @unknown_cost_failure = RubyLLM::Contract::Eval::UnknownCostGate.check([@report], mode: @on_unknown_pricing)
+      cost_ok = @report.total_cost <= @maximum_cost && @unknown_cost_failure.nil?
+    end
 
     regression_ok = if @prompt_diff
                       @prompt_diff.safe_to_switch?
@@ -108,6 +119,7 @@ RSpec::Matchers.define :pass_eval do |eval_name|
                       true
                     end
 
+    @unknown_cost_only = score_ok && regression_ok && @report.total_cost <= @maximum_cost if @unknown_cost_failure
     score_ok && cost_ok && regression_ok
   rescue StandardError => e
     @error = e
@@ -115,71 +127,87 @@ RSpec::Matchers.define :pass_eval do |eval_name|
   end
 
   failure_message do
-    if @prompt_diff && !@prompt_diff.safe_to_switch?
-      msg = "expected #{@eval_name} eval to be safe to switch from baseline prompt\n"
-
-      # Check empty sides first — most fundamental problem
-      bl_empty = @prompt_diff.baseline_empty?
-      cd_empty = @prompt_diff.candidate_empty?
-      if bl_empty || cd_empty
-        msg += "  One side has no evaluated cases (all skipped or no adapter?)\n"
-        if sample_response_only_compare?
-          msg += "  compare_with ignores sample_response; pass model: or with_context(adapter: ...)\n"
+    other_failure_message = lambda do
+      if @prompt_diff && !@prompt_diff.safe_to_switch?
+        msg = "expected #{@eval_name} eval to be safe to switch from baseline prompt\n"
+
+        # Check empty sides first — most fundamental problem
+        bl_empty = @prompt_diff.baseline_empty?
+        cd_empty = @prompt_diff.candidate_empty?
+        if bl_empty || cd_empty
+          msg += "  One side has no evaluated cases (all skipped or no adapter?)\n"
+          if sample_response_only_compare?
+            msg += "  compare_with ignores sample_response; pass model: or with_context(adapter: ...)\n"
+          end
+          msg += "  Candidate score: #{@prompt_diff.candidate_score}, Baseline score: #{@prompt_diff.baseline_score}"
+          next msg
         end
-        msg += "  Candidate score: #{@prompt_diff.candidate_score}, Baseline score: #{@prompt_diff.baseline_score}"
-        next msg
-      end
 
-      # Check dataset comparability — names, inputs, AND expected must match
-      unless @prompt_diff.cases_comparable?
-        unless @prompt_diff.case_names_match?
-          mm = @prompt_diff.mismatched_cases
-          msg += "  Case set mismatch — candidate and baseline must have identical cases:\n"
-          mm[:only_in_baseline].each { |n| msg += "    only in baseline: #{n}\n" }
-          mm[:only_in_candidate].each { |n| msg += "    only in candidate: #{n}\n" }
-        end
-        @prompt_diff.input_mismatches.each do |m|
-          msg += "  Input mismatch for '#{m[:case]}' — same name but different inputs\n"
+        # Check dataset comparability — names, inputs, AND expected must match
+        unless @prompt_diff.cases_comparable?
+          unless @prompt_diff.case_names_match?
+            mm = @prompt_diff.mismatched_cases
+            msg += "  Case set mismatch — candidate and baseline must have identical cases:\n"
+            mm[:only_in_baseline].each { |n| msg += "    only in baseline: #{n}\n" }
+            mm[:only_in_candidate].each { |n| msg += "    only in candidate: #{n}\n" }
+          end
+          @prompt_diff.input_mismatches.each do |m|
+            msg += "  Input mismatch for '#{m[:case]}' — same name but different inputs\n"
+          end
+          @prompt_diff.expected_mismatches.each do |m|
+            msg += "  Expected mismatch for '#{m[:case]}' — same name/input but different expected values\n"
+          end
+          next msg
         end
-        @prompt_diff.expected_mismatches.each do |m|
-          msg += "  Expected mismatch for '#{m[:case]}' — same name/input but different expected values\n"
+
+        # Check per-case score regressions (even if global average is flat)
+        if @prompt_diff.score_regressions.any?
+          msg += "  Per-case score regressions (#{@prompt_diff.score_regressions.length}):\n"
+          @prompt_diff.score_regressions.each do |r|
+            msg += "    #{r[:case]}: #{r[:baseline_score]} -> #{r[:candidate_score]} (#{r[:delta]})\n"
+          end
+          msg += "  Score delta: #{@prompt_diff.score_delta}"
+          next msg
         end
-        next msg
-      end
 
-      # Check per-case score regressions (even if global average is flat)
-      if @prompt_diff.score_regressions.any?
-        msg += "  Per-case score regressions (#{@prompt_diff.score_regressions.length}):\n"
-        @prompt_diff.score_regressions.each do |r|
-          msg += "    #{r[:case]}: #{r[:baseline_score]} -> #{r[:candidate_score]} (#{r[:delta]})\n"
+        # Check pass/fail regressions and removed cases
+        removed = @prompt_diff.removed_passing_cases
+        reg_count = @prompt_diff.regressions.length + removed.length
+        msg += "  Found #{reg_count} regression(s):\n"
+        @prompt_diff.regressions.each do |r|
+          msg += "    #{r[:case]}: was PASS, now FAIL -- #{r[:detail]}\n"
+        end
+        removed.each do |name|
+          msg += "    #{name}: REMOVED (was passing in baseline)\n"
         end
         msg += "  Score delta: #{@prompt_diff.score_delta}"
         next msg
       end
 
-      # Check pass/fail regressions and removed cases
-      removed = @prompt_diff.removed_passing_cases
-      reg_count = @prompt_diff.regressions.length + removed.length
-      msg += "  Found #{reg_count} regression(s):\n"
-      @prompt_diff.regressions.each do |r|
-        msg += "    #{r[:case]}: was PASS, now FAIL -- #{r[:detail]}\n"
-      end
-      removed.each do |name|
-        msg += "    #{name}: REMOVED (was passing in baseline)\n"
+      msg = format_failure_message(@eval_name, @error, @report, @minimum_score, @maximum_cost)
+      if @diff&.regressed?
+        msg += "\n\nRegressions from baseline:\n"
+        @diff.regressions.each do |r|
+          msg += "  #{r[:case]}: was PASS, now FAIL -- #{r[:detail]}\n"
+        end
+        @diff.skipped_passing_cases.each do |s|
+          msg += "  #{s[:case]}: was PASS, now SKIPPED -- #{s[:detail]}\n"
+        end
+        msg += "  Score delta: #{@diff.score_delta}"
       end
-      msg += "  Score delta: #{@prompt_diff.score_delta}"
-      next msg
+      msg
     end
 
-    msg = format_failure_message(@eval_name, @error, @report, @minimum_score, @maximum_cost)
-    if @diff&.regressed?
-      msg += "\n\nRegressions from baseline:\n"
-      @diff.regressions.each do |r|
-        msg += "  #{r[:case]}: was PASS, now FAIL -- #{r[:detail]}\n"
-      end
-      msg += "  Score delta: #{@diff.score_delta}"
+    # Cost first, as in the rake suite gate: an unpriced total cannot be trusted.
+    # An exception outranks it, and any other failure is still listed.
+    if @unknown_cost_failure && @error.nil?
+      unknown_cost = "expected #{@eval_name} eval to stay within maximum cost, but #{@unknown_cost_failure}"
+      next unknown_cost if @unknown_cost_only
+
+      next "#{unknown_cost}\n\n#{other_failure_message.call}"
     end
-    msg
+
+    other_failure_message.call
   end
 
   failure_message_when_negated do

data/lib/ruby_llm/contract/step/trace.rb

@@ -50,6 +50,25 @@ module RubyLLM
           )
         end
 
+        # True when a call ran on a named model, used tokens, and still got no
+        # cost, so any total built from this trace undercounts. A zero-token call
+        # (Test adapter, sample_response) costs 0 whatever the pricing. A retried
+        # step is judged per attempt: a priced subtotal must not hide an unpriced
+        # attempt, and the merged trace may have re-priced on the last model.
+        def cost_unknown?
+          if @attempts.is_a?(Array) && !@attempts.empty?
+            @attempts.any? { |attempt| self.class.unpriced?(attempt[:model], attempt[:usage], attempt[:cost]) }
+          else
+            self.class.unpriced?(@model, @usage, @cost)
+          end
+        end
+
+        def self.unpriced?(model, usage, cost)
+          return false unless model && cost.nil? && usage.is_a?(Hash)
+
+          ((usage[:input_tokens] || 0) + (usage[:output_tokens] || 0)).positive?
+        end
+
         def to_h
           { messages: @messages, model: @model, latency_ms: @latency_ms,
             usage: @usage, attempts: @attempts, cost: @cost }.compact

data/lib/ruby_llm/contract/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module Contract
-    VERSION = "1.0.0"
+    VERSION = "1.1.0"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-contract
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - Justyna
@@ -59,6 +59,7 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- ".ruby-version"
 - CHANGELOG.md
 - LICENSE
 - README.md
@@ -152,6 +153,7 @@ files:
 - lib/ruby_llm/contract/eval/step_expectation_applier.rb
 - lib/ruby_llm/contract/eval/step_result_normalizer.rb
 - lib/ruby_llm/contract/eval/trait_evaluator.rb
+- lib/ruby_llm/contract/eval/unknown_cost_gate.rb
 - lib/ruby_llm/contract/minitest.rb
 - lib/ruby_llm/contract/pipeline.rb
 - lib/ruby_llm/contract/pipeline/base.rb