ruby_llm-contract 0.10.6 → 1.1.0

data/README.md

@@ -23,7 +23,7 @@ end
 RubyLLM::Contract.configure { }
 ```
 
-Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc). Requires `ruby_llm ~> 1.12` and Ruby ≥ 3.2.
+Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc). Requires `ruby_llm ~> 2.0` and Ruby ≥ 3.2.
 
 ## Example
 
@@ -132,6 +132,8 @@ Everything below is optional — the example above is a complete step. Reach for
 
 Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and per-step models.
 
+**Runnable companion repo:** [`ruby_llm-contract_demo`](https://github.com/justi/ruby_llm-contract_demo) — full LLM-as-judge lifecycle (v1 strict → v2 drift → judge calibration → v3/v4 iteration) on a returns-policy chatbot scenario, dual-language (EN default, `DEMO_LANG=pl` opt-in). Live OpenAI calls (`gpt-4.1-mini`, `temperature 0`), `LIVE=1` opt-in, ~$0.30 for the full lifecycle. See [llm_judge.md](docs/guide/llm_judge.md) for the methodology.
+
 ## Relation to `RubyLLM::Agent`
 
 `Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly — Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate(...)` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
@@ -162,10 +164,18 @@ Different layers, complementary. [`ruby_llm-tribunal`](https://github.com/Alqemi
 | [Multimodal input (PDF / image / audio)](docs/guide/multimodal_input.md) | Route attachments through the contract; `attachment_token_estimate`, fail-closed cost, calibration table |
 | [Schema DSL reference](docs/guide/output_schema.md) | Every constraint, nested objects, pattern table |
 | [Prompt DSL reference](docs/guide/prompt_ast.md) | `system` / `rule` / `section` / `example` / `user` nodes |
+| [Architecture overview](docs/architecture.md) | Class map: Pipeline, Step, Contract, Prompt, Eval |
 
 ## Status & versioning
 
-Pre-1.0 (currently **0.10.4**). Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 0.10.4` until 1.0 ships.
+Stable since **1.0.0**. Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 1.0`.
+
+What 1.0 promises not to break inside the 1.x line: the documented DSL
+(`prompt`, `output_schema`, `validate`, `retry_policy`, `max_cost`, `define_eval`,
+pipelines), the `Result` and `trace` shapes including `trace[:usage]`'s
+`{ input_tokens:, output_tokens: }` keys, and the adapter interface. A new major
+version of a runtime dependency may require a new major version here — that is
+what 1.0.0 itself was.
 
 ## FAQ
 

data/docs/guide/getting_started.md

@@ -145,7 +145,7 @@ report.save_baseline!
 expect(SummarizeArticle).to pass_eval("regression").without_regressions
 ```
 
-`without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality.
+`without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality. A previously-passing case that was skipped this run (no adapter configured) also fails the gate, listed as `SKIPPED` rather than as a regression.
 
 ## Budget caps
 
@@ -173,6 +173,8 @@ max_cost 0.01, on_unknown_pricing: :warn
 
 Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
 
+The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. The check sees only reported tokens: an adapter that returns no token counts looks like a zero-token run.
+
 ### Preflight cost estimates
 
 Check what a call is likely to cost before invoking it:

data/docs/guide/llm_judge.md

@@ -120,7 +120,7 @@ report = AccuracyJudge.run_eval("calibration")
 report.score   # => 0.92  → judge agrees with humans 92% of the time
 ```
 
-A reasonable bar is `score >= 0.85` — an empirical baseline from LLM-eval field practice ([Eugene Yan on evals](https://eugeneyan.com/writing/evals/), [Hamel Husain field guide](https://hamel.dev/blog/posts/field-guide/)). Below that, judge-human agreement drops into a band where the judge's confident percentages stop matching what a human reviewer would say, and gate decisions become unreliable. If your domain is high-stakes (medical, legal, financial), raise the bar to 0.90+. Iterate the judge's prompt, or escalate to a stronger model, until the score on real production data crosses your bar.
+A reasonable bar is `score >= 0.85` — a practical threshold synthesized from field practice. [Hamel Husain](https://hamel.dev/blog/posts/field-guide/) reports it took **three iterations** of the judge prompt to reach ">90% agreement" with humans in a Honeycomb case study; [Eugene Yan](https://eugeneyan.com/writing/evals/) notes RAG-grounded systems still hold a 5-10% factual inconsistency rate even after good prompt engineering. The specific 0.85 number is the author's summary — neither source quotes it directly. Below ~0.85, judge-human agreement drops into a band where the judge's confident percentages stop matching what a human reviewer would say, and gate decisions become unreliable. If your domain is high-stakes (medical, legal, financial), raise the bar to 0.90+. Iterate the judge's prompt, or escalate to a stronger model, until the score on real production data crosses your bar.
 
 **What "iterate the judge's prompt" actually looks like.** The first judge prompt almost always over-flags. Two or three iterations are normal before the judge is ready to gate anything. The common pattern:
 

data/docs/guide/multimodal_input.md

@@ -2,7 +2,7 @@
 
 > Read this when your contract needs to send a PDF, image, or audio file to the LLM — not just text.
 
-`ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy escalate(...)`, and trace observability still apply. The gem does **not** ship its own multimodal API — it forwards `with:` to `RubyLLM::Chat#ask`, which RubyLLM 1.15+ normalises per provider (Anthropic, OpenAI, Gemini).
+`ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy escalate(...)`, and trace observability still apply. The gem does **not** ship its own multimodal API — it forwards `with:` to `RubyLLM::Chat#ask`, which RubyLLM normalises per provider (Anthropic, OpenAI, Gemini).
 
 ## Minimal example
 
@@ -130,7 +130,7 @@ Pick a value at or above the provider's worst-case. The estimate is a **floor fo
 
 ## Multi-turn caveat
 
-If your contract uses history (`add_history`), attachments from prior turns are **not** replayed in 0.10.x. Single-turn multimodal works; follow-up questions on the same document require additional work that is deferred to a later release. See [ADR-0022](../decisions/ADR-0022-v09-multimodal-input.md) (internal) for the rationale.
+If your contract uses history (`add_history`), attachments from prior turns are **not** replayed (still true in 1.0.x). Single-turn multimodal works; follow-up questions on the same document require additional work that is deferred to a later release. The rationale is recorded in the project's internal ADR-0022, which is not shipped with the gem.
 
 ## Provider notes
 
@@ -149,7 +149,8 @@ RSpec.describe ExtractInvoiceData do
   it "forwards attachment to chat.ask" do
     expect_any_instance_of(RubyLLM::Chat).to receive(:ask)
       .with(anything, with: "fixtures/invoice.pdf")
-      .and_return(double(content: '{"vendor":"X",...}', input_tokens: 200, output_tokens: 50))
+      .and_return(double(content: '{"vendor":"X",...}',
+                         tokens: RubyLLM::Tokens.new(input: 200, output: 50)))
 
     result = described_class.run("extract", context: { attachment: "fixtures/invoice.pdf" })
     expect(result.status).to eq(:ok)

data/docs/guide/output_schema.md

@@ -2,12 +2,12 @@
 
 > Read this as a reference for the schema DSL — every constraint, nested objects, arrays of objects, the full pattern table.
 
-Declare the expected output structure using [ruby_llm-schema](https://github.com/danielfriis/ruby_llm-schema) DSL. The schema serves **two purposes**:
+Declare the expected output structure using the [schematist](https://github.com/crmne/schematist) DSL. The schema serves **two purposes**:
 
 1. **Output validation** — replaces type and shape checks (enums, ranges, required fields). One declaration instead of many.
 2. **Provider-side request** — with the RubyLLM adapter, the schema is sent to the LLM provider via `chat.with_schema(...)`, asking the model to return JSON matching the shape. Cheaper models sometimes ignore the request, which is why client-side validation (point 1) still matters.
 
-> **Same DSL `RubyLLM::Agent.schema` accepts.** Both use `RubyLLM::Schema.create(&block)` under the hood. `Step.output_schema` is eager-compiled at class load and drives client-side validation; `Agent.schema` accepts a `Proc` for dynamic per-call schemas. See [Relation to Agent](relation_to_agent.md) for the full comparison.
+> **Same DSL `RubyLLM::Agent.schema` accepts.** Both use `Schematist::Schema.create(&block)` under the hood (schematist is what the `ruby_llm-schema` gem became; RubyLLM 2.x depends on it directly). `Step.output_schema` is eager-compiled at class load and drives client-side validation; `Agent.schema` accepts a `Proc` for dynamic per-call schemas. See [Relation to Agent](relation_to_agent.md) for the full comparison.
 
 All examples below extend the `SummarizeArticle` step from the [README](../../README.md).
 

data/docs/guide/testing.md

@@ -160,7 +160,7 @@ end
 
 - `.with_context(model: "gpt-4.1-mini")` — pick model / pass adapter
 - `.with_minimum_score(0.8)` — gate on average score
-- `.with_maximum_cost(0.01)` — gate on total cost
+- `.with_maximum_cost(0.01)` — gate on total cost; fails when a case's model has no pricing data unless you add `.on_unknown_pricing(:warn)`
 - `.without_regressions` — block any previously-passing case that now fails (reads the baseline)
 - `.compared_with(SummarizeArticleV1)` — A/B against another step; implies regression check
 

data/examples/README.md

@@ -14,7 +14,7 @@ Pedagogical order: hook → activation → evolution → composition → quality
 | 05 | `05_eval_dataset.rb` | **"How do I stop silent prompt regressions?"** — define_eval with real cases, baseline vs regressed adapter, regression detection signal, inline eval_case. |
 | 06 | `06_retry_variants.rb` | **"What retry shapes exist beyond cross-model?"** — `attempts: 3` (variance absorption), `reasoning_effort` escalation (low→medium→high), cross-provider fallback (Ollama → Anthropic → OpenAI). |
 
-Every example has an "Expected output" section in the file header — you can read what each one prints without running it.
+Four of the seven carry an "Expected output" section in the file header (01, 03, 05, 06), so you can read what they print without running them. The other three differ: `00_basics.rb` prints nothing and annotates each result inline with `# =>`; `02_real_llm_minimal.rb` needs a live provider key, so its output is not deterministic; `04_summarize_and_translate.rb` prints but has no header block.
 
 ## Running
 

data/lib/ruby_llm/contract/adapters/ruby_llm.rb

@@ -25,7 +25,6 @@ module RubyLLM
           build_response(response)
         end
 
-        # Maps option keys to the RubyLLM chat method and argument form.
         CHAT_OPTION_METHODS = {
           temperature: :with_temperature,
           schema: :with_schema
@@ -70,12 +69,10 @@ module RubyLLM
           thinking_config = resolve_thinking_config(options)
           chat.with_thinking(**thinking_config) if thinking_config
 
-          # `with_params` carries only raw passthroughs (currently `max_tokens`).
-          # `reasoning_effort` is no longer forwarded here — it goes through
-          # `with_thinking` above, which is the canonical RubyLLM API.
-          params = {}
-          params[:max_tokens] = options[:max_tokens] if options[:max_tokens]
-          chat.with_params(**params) if params.any?
+          # RubyLLM 2.0 removed `with_params`; `with_max_output_tokens` is the
+          # replacement for this one passthrough. `reasoning_effort` is not
+          # forwarded here — it goes through `with_thinking` above.
+          chat.with_max_output_tokens(options[:max_tokens]) if options[:max_tokens]
         end
 
         # Returns merged `{ effort:, budget: }` or nil. `options[:reasoning_effort]`
@@ -91,11 +88,18 @@ module RubyLLM
           content = response.content
           content = content.to_s unless content.is_a?(Hash) || content.is_a?(Array)
 
+          # This is the ONLY place upstream token counts are read. RubyLLM 2.0
+          # replaced `Message#input_tokens`/`#output_tokens` with a `Tokens` value
+          # object. The `{ input_tokens:, output_tokens: }` shape below is this
+          # gem's own public contract (Step::Trace#usage, cost calculation,
+          # documented in the README) and deliberately does NOT change.
+          tokens = response.tokens
+
           Response.new(
             content: content,
             usage: {
-              input_tokens: response.input_tokens || 0,
-              output_tokens: response.output_tokens || 0
+              input_tokens: tokens&.input || 0,
+              output_tokens: tokens&.output || 0
             }
           )
         end

data/lib/ruby_llm/contract/concerns/context_helpers.rb

@@ -3,8 +3,6 @@
 module RubyLLM
   module Contract
     module Concerns
-      # Shared helpers for context hash manipulation.
-      # Used by EvalHost, Runner, Step::Base.
       module ContextHelpers
         private
 

data/lib/ruby_llm/contract/concerns/deep_symbolize.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     module Concerns
-      # Recursively converts Hash keys to symbols while preserving array shape.
       module DeepSymbolize
         def deep_symbolize(object)
           case object

data/lib/ruby_llm/contract/contract/parser.rb

@@ -38,7 +38,6 @@ module RubyLLM
       end
       private_class_method :parse_json_text
 
-      # Fallback: attempt to extract the first JSON object or array from prose
       def self.parse_json_with_extraction(text, raw_output)
         extracted = extract_json(text)
         unless extracted
@@ -72,8 +71,6 @@ module RubyLLM
         match ? match[1] : text
       end
 
-      # Extract the first JSON object or array from text that may contain prose.
-      # Uses bracket-matching to find the outermost balanced { } or [ ] block.
       JSON_START_PATTERN = /[{\[]/
 
       def self.extract_json(text)

data/lib/ruby_llm/contract/contract/schema_validator/bound_rule.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     class SchemaValidator
-      # Validates numeric and collection size bounds for a node.
       class BoundRule
         NUMERIC_LIMITS = [
           { key: :minimum, label: "minimum", relation: "below", invalid: ->(actual, limit) { actual < limit } },

data/lib/ruby_llm/contract/contract/schema_validator/enum_rule.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     class SchemaValidator
-      # Validates enum membership for a node when enum values are declared.
       class EnumRule
         def initialize(errors)
           @errors = errors

data/lib/ruby_llm/contract/contract/schema_validator/node.rb

@@ -21,10 +21,6 @@ module RubyLLM
           value.is_a?(Array)
         end
 
-        def numeric?
-          value.is_a?(Numeric)
-        end
-
         def properties
           schema[:properties] || {}
         end

data/lib/ruby_llm/contract/contract/schema_validator/scalar_rules.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     class SchemaValidator
-      # Applies scalar-only validation rules to a schema node.
       class ScalarRules
         def initialize(errors)
           @rules = [

data/lib/ruby_llm/contract/contract/schema_validator/type_rule.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     class SchemaValidator
-      # Validates the declared JSON schema type for a node.
       class TypeRule
         def initialize(errors)
           @errors = errors

data/lib/ruby_llm/contract/cost_calculator.rb

@@ -84,11 +84,37 @@ module RubyLLM
       end
 
       def self.compute_cost(model_info, usage)
-        input_cost = token_cost(usage[:input_tokens], model_info.input_price_per_million)
-        output_cost = token_cost(usage[:output_tokens], model_info.output_price_per_million)
+        prices = prices_for(model_info)
+        return nil unless prices
+
+        input_cost = token_cost(usage[:input_tokens], prices[:input])
+        output_cost = token_cost(usage[:output_tokens], prices[:output])
         (input_cost + output_cost).round(6)
       end
 
+      # Two shapes reach here. `RegisteredModel` (our own struct, from
+      # `register_model`) exposes flat `*_price_per_million` readers. RubyLLM 2.0
+      # moved provider pricing into a nested value object and dropped those flat
+      # readers, so it has to be walked: pricing -> text_tokens -> standard.
+      #
+      # nil means "unknown pricing". The guards keep the two known shapes
+      # explicit, but `calculate` rescues StandardError, so a future shape this
+      # walk cannot read also degrades to nil: `max_cost` then refuses the call
+      # citing "no pricing data" rather than the real cause, and paths that floor
+      # unknown cost to 0.0 (eval cost estimates, report totals) undercount.
+      def self.prices_for(model_info)
+        if model_info.respond_to?(:input_price_per_million)
+          return { input: model_info.input_price_per_million,
+                   output: model_info.output_price_per_million }
+        end
+
+        tier = model_info.pricing&.text_tokens&.standard if model_info.respond_to?(:pricing)
+        return unless tier.respond_to?(:input_per_million)
+
+        { input: tier.input_per_million, output: tier.output_per_million }
+      end
+      private_class_method :prices_for
+
       # Provider pricing is denominated per 1M tokens; divide here to get
       # the dollar cost for the actual usage count. Named constant for
       # consistency with how RubyLLM and provider docs express prices.

data/lib/ruby_llm/contract/eval/aggregated_report.rb

@@ -44,6 +44,10 @@ module RubyLLM
           @runs.sum(&:total_cost) / @runs.length.to_f
         end
 
+        def unknown_cost_results
+          @runs.flat_map(&:unknown_cost_results)
+        end
+
         def avg_latency_ms
           latencies = @runs.filter_map(&:avg_latency_ms)
           return nil if latencies.empty?

data/lib/ruby_llm/contract/eval/baseline_diff.rb

@@ -19,6 +19,7 @@ module RubyLLM
             current = @current[name]
             next unless current
             next unless baseline[:passed] && !current[:passed]
+            next if skipped?(current)
 
             {
               case: name,
@@ -47,8 +48,19 @@ module RubyLLM
           (current_score - baseline_score).round(4)
         end
 
+        # Passed in the baseline, not run now (no adapter): not evidence of a
+        # regression, but not evidence it still passes either, so the gate fails.
+        def skipped_passing_cases
+          @baseline.filter_map do |name, baseline|
+            current = @current[name]
+            next unless current && baseline[:passed] && skipped?(current)
+
+            { case: name, detail: current[:details] }
+          end
+        end
+
         def regressed?
-          regressions.any? || removed_passing_cases.any?
+          regressions.any? || removed_passing_cases.any? || skipped_passing_cases.any?
         end
 
         def removed_passing_cases
@@ -70,6 +82,7 @@ module RubyLLM
         def to_s
           lines = ["Score: #{baseline_score.round(2)} → #{current_score.round(2)} (#{format_delta})"]
           regressions.each { |r| lines << "  REGRESSED  #{r[:case]}: #{r[:detail]}" }
+          skipped_passing_cases.each { |s| lines << "  SKIPPED    #{s[:case]}: #{s[:detail]}" }
           improvements.each { |r| lines << "  IMPROVED   #{r[:case]}" }
           new_cases.each { |c| lines << "  NEW        #{c}" }
           removed_cases.each { |c| lines << "  REMOVED    #{c}" }
@@ -79,13 +92,19 @@ module RubyLLM
         private
 
         def compute_score(cases)
-          # Exclude skipped cases from score (consistent with Report#score)
-          evaluated = cases.reject { |c| c[:details]&.start_with?("skipped:") }
+          # Asymmetric on purpose: the baseline side arrives already filtered by
+          # ReportStats#evaluated_results, the current side does not, so this
+          # string check is the only skip filter on one of the two sides.
+          evaluated = cases.reject { |c| skipped?(c) }
           return 0.0 if evaluated.empty?
 
           evaluated.sum { |c| c[:score] } / evaluated.length
         end
 
+        def skipped?(serialized_case)
+          serialized_case[:details]&.start_with?(CaseResult::SKIPPED_DETAILS_PREFIX)
+        end
+
         def index_by_name(cases)
           cases.each_with_object({}) { |c, h| h[c[:name]] = c }
         end

data/lib/ruby_llm/contract/eval/candidate_label.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Eval
+      # The `model (effort: x)` notation, rendered into optimizer tables and parsed
+      # back to rebuild candidate configs. Render and parse live together because a
+      # format change on one side silently mis-parses on the other: the pattern is
+      # built from the same literal the renderer emits.
+      module CandidateLabel
+        OPEN = " (effort: "
+        CLOSE = ")"
+        PATTERN = /\s*#{Regexp.escape(OPEN.lstrip)}(\w+)#{Regexp.escape(CLOSE)}/
+
+        def self.render(config)
+          effort = config[:reasoning_effort]
+          return config[:model] unless effort
+
+          "#{config[:model]}#{OPEN}#{effort}#{CLOSE}"
+        end
+
+        def self.parse(label)
+          match = PATTERN.match(label)
+          return { model: label } unless match
+
+          { model: match.pre_match.strip, reasoning_effort: match[1] }
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/eval/case_executor.rb

@@ -30,7 +30,7 @@ module RubyLLM
         private
 
         def missing_adapter?(error)
-          error.message.include?("No adapter configured")
+          error.message.include?(Step::Base::NO_ADAPTER_MESSAGE)
         end
 
         def skipped_result(test_case, reason)
@@ -43,7 +43,7 @@ module RubyLLM
             score: 0.0,
             passed: false,
             label: "SKIP",
-            details: "skipped: #{reason}"
+            details: CaseResult.skipped_details(reason)
           )
         end
       end

data/lib/ruby_llm/contract/eval/case_result.rb

@@ -4,11 +4,20 @@ module RubyLLM
   module Contract
     module Eval
       class CaseResult
+        # BaselineDiff#compute_score keys off this prefix to keep skipped cases
+        # out of the score denominator. Both ends must read it from here.
+        SKIPPED_DETAILS_PREFIX = "skipped:"
+
+        def self.skipped_details(reason)
+          "#{SKIPPED_DETAILS_PREFIX} #{reason}"
+        end
+
         attr_reader :name, :input, :output, :expected, :step_status,
                     :score, :details, :duration_ms, :cost, :attempts
 
-        def initialize(name:, input:, output:, expected:, step_status:,
-                       score:, passed:, label: nil, details: nil, duration_ms: nil, cost: nil, attempts: nil)
+        def initialize(name:, input:, output:, expected:, step_status:, # rubocop:disable Metrics/ParameterLists
+                       score:, passed:, label: nil, details: nil, duration_ms: nil, cost: nil, attempts: nil,
+                       cost_unknown: false)
           @name = name
           @input = input
           @output = output
@@ -21,6 +30,7 @@ module RubyLLM
           @duration_ms = duration_ms
           @cost = cost
           @attempts = attempts
+          @cost_unknown = cost_unknown
           freeze
         end
 
@@ -32,6 +42,10 @@ module RubyLLM
           !@passed
         end
 
+        def cost_unknown?
+          @cost_unknown
+        end
+
         def label
           @label || (@passed ? "PASS" : "FAIL")
         end
@@ -61,7 +75,7 @@ module RubyLLM
             duration_ms: @duration_ms,
             cost: @cost,
             attempts: @attempts
-          }
+          }.tap { |hash| hash[:cost_unknown] = true if @cost_unknown }
         end
 
         private

data/lib/ruby_llm/contract/eval/case_result_builder.rb

@@ -19,7 +19,8 @@ module RubyLLM
             details: evaluation.details,
             duration_ms: trace_metric(trace, :total_latency_ms, :latency_ms),
             cost: trace_metric(trace, :total_cost, :cost),
-            attempts: trace_attempts(trace)
+            attempts: trace_attempts(trace),
+            cost_unknown: trace.respond_to?(:cost_unknown?) && trace.cost_unknown?
           )
         end
 

data/lib/ruby_llm/contract/eval/contract_detail_builder.rb

@@ -3,8 +3,6 @@
 module RubyLLM
   module Contract
     module Eval
-      # Extracted from Runner to reduce class length.
-      # Builds contract detail strings for contract-only evaluation.
       module ContractDetailBuilder
         private
 

data/lib/ruby_llm/contract/eval/model_comparison.rb

@@ -6,9 +6,10 @@ module RubyLLM
       class ModelComparison
         attr_reader :eval_name, :reports, :configs, :fallback
 
+        # Kept as the name six call sites already use; the format itself lives in
+        # CandidateLabel, which owns both directions.
         def self.candidate_label(config)
-          effort = config[:reasoning_effort]
-          effort ? "#{config[:model]} (effort: #{effort})" : config[:model]
+          CandidateLabel.render(config)
         end
 
         def initialize(eval_name:, reports:, configs: nil, fallback: nil)

data/lib/ruby_llm/contract/eval/pipeline_result_adapter.rb

@@ -3,8 +3,7 @@
 module RubyLLM
   module Contract
     module Eval
-      # Lightweight adapter that wraps a Pipeline::Result to look like a Step::Result.
-      # Replaces OpenStruct usage in Runner#normalize_pipeline_result.
+      # Replaces OpenStruct usage in StepResultNormalizer#normalize_pipeline_result.
       PipelineResultAdapter = Struct.new(:status, :ok_flag, :parsed_output, :validation_errors, :trace) do
         def ok?
           ok_flag

data/lib/ruby_llm/contract/eval/prompt_diff_comparator.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     module Eval
-      # Encapsulates the safety and mismatch rules for prompt A/B comparison.
       class PromptDiffComparator
         def initialize(candidate_cases:, baseline_cases:, diff:)
           @candidate_cases = candidate_cases

data/lib/ruby_llm/contract/eval/prompt_diff_presenter.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     module Eval
-      # Renders a prompt diff as a readable console summary.
       class PromptDiffPresenter
         VARIANT_LABEL_WIDTH = 12
         TABLE_WIDTH = 26

data/lib/ruby_llm/contract/eval/prompt_diff_serializer.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     module Eval
-      # Normalizes report results into comparable prompt-diff case hashes.
       class PromptDiffSerializer
         def call(report)
           report.results.reject { |result| result.step_status == :skipped }.map do |result|

data/lib/ruby_llm/contract/eval/report.rb

@@ -15,7 +15,7 @@ module RubyLLM
         BASELINE_DIR = ".eval_baselines"
 
         def_delegators :@stats, :score, :passed, :failed, :skipped, :failures, :pass_rate, :pass_rate_ratio,
-                       :total_cost, :avg_latency_ms, :passed?,
+                       :total_cost, :unknown_cost_results, :avg_latency_ms, :passed?,
                        :production_mode?, :escalation_rate, :single_shot_cost, :single_shot_latency_ms,
                        :effective_cost, :effective_latency_ms, :latency_percentiles
         def_delegators :@presenter, :summary, :to_s, :print_summary

data/lib/ruby_llm/contract/eval/report_presenter.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     module Eval
-      # Formats eval reports for console and string output.
       class ReportPresenter
         def initialize(report:, stats:)
           @report = report

data/lib/ruby_llm/contract/eval/report_stats.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     module Eval
-      # Computes aggregate metrics for an eval report.
       class ReportStats
         def initialize(results:)
           @results = results
@@ -45,6 +44,11 @@ module RubyLLM
           @results.sum { |result| result.cost || 0.0 }
         end
 
+        # Cases total_cost could not price; it counts them as 0.0.
+        def unknown_cost_results
+          evaluated_results.select(&:cost_unknown?)
+        end
+
         def avg_latency_ms
           latencies = @results.filter_map(&:duration_ms)
           return nil if latencies.empty?

data/lib/ruby_llm/contract/eval/report_storage.rb

@@ -6,7 +6,6 @@ require "fileutils"
 module RubyLLM
   module Contract
     module Eval
-      # Persists eval reports as history entries and regression baselines.
       class ReportStorage
         def initialize(report:, stats:)
           @report = report