ruby_llm-llm_judge 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 451771cadd6d75e7ea6361b43814d186bc0b9e3aeb0dea5f128fdc3c4a68e982
-  data.tar.gz: ca97b903a28238275417a96551abbf440198eb2dea95d52196cfb2e5d1b0d173
+  metadata.gz: 71f6bd771e7f6bec3a9dcdf7036ee1238110c63ee530b15d32c6dadcfb0d0eaa
+  data.tar.gz: e54b13efadda477c3d3713c66184dc4ce01061b740e06ab52e95df0b672ae7e9
 SHA512:
-  metadata.gz: ff30bd52e7b0dcb79f8804458e622d725ad3570351e00d180efb9ed040d4bb07c8551c37024f1730ca876125f52513e0e28a557fb9ed8f9d22211b286266aa6d
-  data.tar.gz: 25259a68bba8ffc259d343f076b7962788f236143ca8dabe43c03e626181f9bac24bfb50a98b50951130478be94951c56944775072614732c1341aa3218626eb
+  metadata.gz: 4e29ff7f1b47222dfc71f6f096cd058615d2517b7e16b29f9c2a68f094f2af95be8a99232d5ba72cc1d6b9a932fa5f9bf290e701a9744ee1a241fc6a0c710834
+  data.tar.gz: 942f4a7da0c27d1741d3e0e2b80ae719335123d6ddf276b84b0e4d200ed315d5f839cfc1ea998e708b6b54f83d7b7e97c718f6d4a070e17d2bfe0be23e9b3541

data/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## 0.1.2 (2026-09-28)
+
+- When a rating call fails, no new calls start and the error is raised once calls in flight finish. Previously the remaining calls kept running, and were billed, after `judge` raised.
+- `chat_provider_options` is merged over the defaults instead of replacing them. Setting one option no longer re-enables reasoning or drops `store: false` for Luna.
+- One-call JSON inside a Markdown code fence is accepted. Claude Haiku 4.5 now works with `strategy: :single_request`.
+- A one-call Choice whose top probabilities tie gets the corrective retry, then raises, instead of selecting the first option. `tie_breaker: :first` keeps the old behavior.
+- New `max_workers:` option (default 6) for concurrent rating calls.
+- Unknown provider options are named in the error.
+- A one-option Choice reports confidence 1.0.
+- RubyLLM 1.13 through 1.16: question validation matches RubyLLM 2.1 (duplicate or empty option names, duplicate probability outcomes, description types), and token totals keep cached and thinking tokens.
+
 ## 0.1.1 (2026-09-28)
 
 - Fix invalid scoring responses raising `NoMethodError` on RubyLLM 1.13, which also skipped the corrective retry. They now raise `RubyLLM::LLMJudge::Error`, a `RubyLLM::Error`, on every supported RubyLLM version.

data/README.md

@@ -19,7 +19,7 @@ The gem also runs on RubyLLM 1.13 through 1.16, which predate the Judge API. The
 What differs before 2.1:
 
 - Results are `RubyLLM::LLMJudge::Legacy` objects, not `RubyLLM::Judgment`, `RubyLLM::Choice`, and so on. Avoid class checks until you upgrade.
-- There is no `RubyLLM.judge`, `RubyLLM::Judge` class DSL, `cost`, `context:` instrumentation, or `metadata:`. Passing `metadata:` raises.
+- There is no `RubyLLM.judge`, `RubyLLM::Judge` class DSL, `cost`, instrumentation, or `metadata:`. Passing `metadata:` raises. `context:` is accepted and supplies its configuration.
 - `scoring_protocol` accepts only `:chat_completions`, the API RubyLLM 1.x uses for OpenAI.
 - The output limit and `chat_provider_options` are sent with `with_params`, using the field each provider expects (`max_completion_tokens` for OpenAI and Azure, `generationConfig.maxOutputTokens` for Gemini and Vertex AI, `inferenceConfig.maxTokens` for Bedrock, and `max_tokens` otherwise).
 
@@ -70,20 +70,23 @@ TicketTriage.judge('Please refund today.').urgent.probability
 
 ## Choose a model
 
-Pass the chat model as `model:` and its provider as `scoring_provider`:
+Pass the chat model as `model:` and its provider as `scoring_provider`. For example, Claude Haiku 4.5 through OpenRouter:
 
 ```ruby
 result = RubyLLM::LLMJudge.judge(
   'Please refund the duplicate charge today.',
-  model: 'claude-haiku-4-5',
+  model: 'anthropic/claude-haiku-4.5',
   questions: { urgent: { type: :probability, instructions: 'Does this need attention today?' } },
   provider_options: {
-    scoring_provider: :anthropic
+    scoring_provider: :openrouter,
+    temperature: 0
   }
 )
 ```
 
-The default Luna rating request uses Chat Completions, temperature zero, reasoning disabled, `store: false`, and a four-token output limit. You can tune requests with `scoring_protocol`, `temperature`, `max_output_tokens`, and `chat_provider_options`.
+Both strategies return valid judgments with this model. Its one-call answers arrive in a Markdown code fence, which the gem removes before parsing.
+
+The default Luna rating request uses Chat Completions, temperature zero, reasoning disabled, `store: false`, and a four-token output limit. Tune requests with `scoring_protocol`, `temperature`, `max_output_tokens`, and `chat_provider_options`. `chat_provider_options` is merged over those defaults, so setting one key, such as `service_tier: 'flex'`, keeps reasoning disabled; set a key explicitly to change a default.
 
 ## One-call typed decisions
 
@@ -107,7 +110,7 @@ result.urgent.probability
 result.department.probabilities
 ```
 
-The model returns one JSON object containing a distribution for each question. The gem checks that every question and option is present, validates each probability, normalizes each distribution, and builds RubyLLM's typed answers. `result.raw[:reported_probabilities]` and `result.raw[:reported_totals]` preserve the model's original numbers for inspection. With an OpenAI key configured, this example calls Luna directly through Chat Completions with reasoning disabled, temperature zero, and `store: false`. The 1024-token limit suits small judgments; omit it for large question sets to use the 8192-token default. It makes one corrective retry for malformed JSON or missing fields, counting both calls in `result.tokens` and `result.raw[:attempts]`. Set `malformed_retries: 0` to disable that retry. An invalid response after retries raises an error.
+The model returns one JSON object containing a distribution for each question. The gem checks that every question and option is present, validates each probability, normalizes each distribution, and builds RubyLLM's typed answers. `result.raw[:reported_probabilities]` and `result.raw[:reported_totals]` preserve the model's original numbers for inspection. With an OpenAI key configured, this example calls Luna directly through Chat Completions with reasoning disabled, temperature zero, and `store: false`. The 1024-token limit suits small judgments; omit it for large question sets to use the 8192-token default. It makes one corrective retry for malformed JSON, missing fields, or a Choice whose top probabilities tie, counting both calls in `result.tokens` and `result.raw[:attempts]`. A tie that remains raises instead of selecting the first option; set `tie_breaker: :first` to accept the first option. Set `malformed_retries: 0` to disable that retry. An invalid response after retries raises an error.
 
 If you use OpenRouter for Luna, prioritize the provider with the lowest observed latency:
 
@@ -139,11 +142,11 @@ OpenRouter's [latency sorting](https://openrouter.ai/docs/guides/routing/provide
 
 ## How scoring works
 
-The default `:ratings` strategy asks the model to rate every declared answer from 0 to 9. It makes one call per answer, with up to six calls running at once, then applies softmax to the ratings. It retries a malformed digit once. When the highest ratings tie on a Choice question, it makes a one-call JSON judgment for that question and uses its distribution. If that call also ties, it raises an error instead of choosing whichever option came first. `result.raw[:tie_breaks]` records each resolution, and token usage includes the extra call. Set `tie_breaker: :first` to use the original first-option rule, or `tie_break_max_output_tokens:` to change the tie-break response limit (default 8192).
+The default `:ratings` strategy asks the model to rate every declared answer from 0 to 9. It makes one call per answer, with up to six calls running at once (`max_workers:` changes this), then applies softmax to the ratings. If a call fails after its retry, no new calls start, calls in flight finish, and the error is raised. It retries a malformed digit once. When the highest ratings tie on a Choice question, it makes a one-call JSON judgment for that question and uses its distribution. If that call also ties, it raises an error instead of choosing whichever option came first. `result.raw[:tie_breaks]` records each resolution, and token usage includes the extra call. Set `tie_breaker: :first` to use the original first-option rule, or `tie_break_max_output_tokens:` to change the tie-break response limit (default 8192).
 
 The `:single_request` strategy asks for all distributions in one call. Both strategies build the same RubyLLM answer types: a `choice` returns the selected answer and distribution; a `score` returns the distribution and its weighted level; a `probability` returns the positive answer's share. The gem supports Judge's 1–255 choice options, 2–10 score levels, and multiple questions in one judgment.
 
-These probabilities compare the answers you supplied. The `:single_request` values are reported by the chat model; the default ratings values come from softmax over its 0–9 ratings. Neither strategy establishes calibration by itself. Include an `other` or `escalate` choice when the named answers may not cover the input. `confidence` measures how concentrated the returned distribution is. Use labeled examples to set any automation thresholds.
+These probabilities compare the answers you supplied. The `:single_request` values are reported by the chat model; the default ratings values come from softmax over its 0–9 ratings. Neither strategy establishes calibration by itself. Include an `other` or `escalate` choice when the named answers may not cover the input. `confidence` measures how concentrated the returned distribution is: 1.0 for a single option, 0.0 for a tie. Use labeled examples to set any automation thresholds.
 
 Set optional `max_arms` and `max_input_bytes` in `provider_options` to cap work per judgment. For example, `{ max_arms: 12, max_input_bytes: 32_768 }` rejects larger requests before scoring. Token usage is aggregated across calls, and `raw` contains the ratings and model metadata.
 
@@ -211,7 +214,7 @@ Latency sorting reduced median time by 20% on AG News and 28% on SST-2. It was f
 
 ## Development
 
-The gem uses RubyLLM's Judge API when it is present and the `Legacy` path otherwise. `test/llm_judge_test.rb` covers the Judge API and `test/legacy_test.rb` covers RubyLLM 1.13 through 1.16; each skips on the other side.
+The gem uses RubyLLM's Judge API when it is present and the `Legacy` path otherwise. `test/llm_judge_test.rb` covers the Judge API and `test/legacy_test.rb` covers RubyLLM 1.13 through 1.16; each skips on the other side. `test/engine_test.rb` and `test/http_test.rb` run on every version; the HTTP tests send real RubyLLM requests to WebMock stubs and check their bodies.
 
 ```bash
 bundle exec rake test                              # the latest released RubyLLM

data/lib/ruby_llm/llm_judge/engine.rb

@@ -17,8 +17,10 @@ module RubyLLM
         @config = config
         options = provider_options.transform_keys(&:to_sym)
         allowed = %i[scoring_provider scoring_protocol temperature max_output_tokens chat_provider_options
-                     max_arms max_input_bytes strategy malformed_retries tie_breaker tie_break_max_output_tokens]
-        raise ArgumentError, 'Unknown LLMJudge provider options' unless (options.keys - allowed).empty?
+                     max_arms max_input_bytes strategy malformed_retries tie_breaker tie_break_max_output_tokens
+                     max_workers]
+        unknown = options.keys - allowed
+        raise ArgumentError, "Unknown LLMJudge provider options: #{unknown.join(', ')}" unless unknown.empty?
 
         @strategy = options.fetch(:strategy, :ratings).to_sym
         raise ArgumentError, 'strategy must be :ratings or :single_request' unless %i[ratings single_request].include?(@strategy)
@@ -38,11 +40,16 @@ module RubyLLM
         @temperature = options.fetch(:temperature, luna_defaults ? 0 : nil)
         @max_output_tokens = options.fetch(:max_output_tokens, @strategy == :single_request ? 8192 : 4)
         @tie_break_max_output_tokens = options.fetch(:tie_break_max_output_tokens, 8192)
-        @chat_provider_options = options.fetch(:chat_provider_options, default_chat_options(luna_defaults))
+        chat_provider_options = options.fetch(:chat_provider_options, {})
+        raise ArgumentError, 'chat_provider_options must be a Hash' unless chat_provider_options.is_a?(Hash)
+
+        # Merged over the defaults, so tuning one option keeps the others (such as disabled reasoning).
+        @chat_provider_options = deep_merge(default_chat_options(luna_defaults), chat_provider_options)
         @max_arms = options[:max_arms]
         @max_input_bytes = options[:max_input_bytes]
         @malformed_retries = options.fetch(:malformed_retries, 1)
-        raise ArgumentError, 'chat_provider_options must be a Hash' unless @chat_provider_options.is_a?(Hash)
+        @max_workers = options.fetch(:max_workers, MAX_WORKERS)
+        raise ArgumentError, 'max_workers must be a positive integer' unless @max_workers.is_a?(Integer) && @max_workers.positive?
         raise ArgumentError, 'max_output_tokens must be positive' unless @max_output_tokens.is_a?(Integer) && @max_output_tokens.positive?
         unless @tie_break_max_output_tokens.is_a?(Integer) && @tie_break_max_output_tokens.positive?
           raise ArgumentError, 'tie_break_max_output_tokens must be positive'
@@ -70,12 +77,9 @@ module RubyLLM
           }
           chosen = answer(question, softmax(ratings))
           if question.type == :choice && ratings.count(ratings.max) > 1 && @tie_breaker == :single_request
+            # Raises if the one-call judgment also ties, after its corrective retry.
             judgment = judge_single_request(input, { question.name => question }, model)
             chosen = judgment.answers.fetch(question.name)
-            probabilities = chosen.probabilities.values
-            if probabilities.count(probabilities.max) > 1
-              raise Error, "Scoring model could not resolve the tie for #{question.name}"
-            end
             tie_breaks << { question: question.name, ratings:, judgment: }
           end
           [question.name, chosen]
@@ -127,7 +131,7 @@ module RubyLLM
       end
 
       def parse_single_request(content, specs)
-        parsed = JSON.parse(content)
+        parsed = JSON.parse(strip_code_fence(content))
         raise Error, 'Scoring model returned a non-object response' unless parsed.is_a?(Hash)
 
         reported = parsed.fetch('answers')
@@ -154,6 +158,12 @@ module RubyLLM
           total = values.sum.to_f
           raise Error, "Scoring model returned zero probability for #{question.name}" unless total.positive?
 
+          # A tied Choice would otherwise select whichever option was listed first.
+          if question.type == :choice && @tie_breaker == :single_request && values.count(values.max) > 1
+            raise Error, "Scoring model could not resolve the tie for #{question.name}; " \
+                         'one option must have the highest probability'
+          end
+
           normalization[question.name.to_s] = total
           [question.name, answer(question, values.map { |value| value / total })]
         end
@@ -230,28 +240,46 @@ module RubyLLM
         value.is_a?(String) ? value : JSON.generate(value)
       end
 
+      # Runs jobs on up to @max_workers threads. After a job fails, no new jobs
+      # start; calls already in flight finish, then the first error is raised.
       def run(jobs)
         queue = Queue.new
         jobs.each_with_index { |job, index| queue << [index, job] }
         results = Array.new(jobs.size)
-        workers = Array.new([jobs.size, MAX_WORKERS].min) do
+        errors = Queue.new
+        workers = Array.new([jobs.size, @max_workers].min) do
           Thread.new do
-            Thread.current.report_on_exception = false
-            loop do
+            while errors.empty?
               begin
                 index, job = queue.pop(true)
               rescue ThreadError
                 break
               end
-              results[index] = yield job
+              begin
+                results[index] = yield job
+              rescue StandardError => error
+                errors << error
+              end
             end
           end
         end
         workers.each(&:join)
-        workers.each(&:value)
+        raise errors.pop unless errors.empty?
+
         results
       end
 
+      # Some models wrap JSON in a Markdown code fence despite the instructions.
+      def strip_code_fence(content)
+        content.to_s.strip.sub(/\A```(?:json)?[ \t]*\n?/i, '').sub(/\n?```\z/, '')
+      end
+
+      def deep_merge(base, overrides)
+        base.merge(overrides) do |_key, old, new|
+          old.is_a?(Hash) && new.is_a?(Hash) ? deep_merge(old, new) : new
+        end
+      end
+
       def default_chat_options(luna_defaults)
         return {} unless @scoring_provider == :openai
 
@@ -292,7 +320,7 @@ module RubyLLM
           chat.with_max_output_tokens(max_output_tokens)
           chat.with_provider_options(@chat_provider_options)
         else
-          chat.with_params(**RubyLLM::Utils.deep_merge(max_output_tokens_param(max_output_tokens), @chat_provider_options))
+          chat.with_params(**deep_merge(max_output_tokens_param(max_output_tokens), @chat_provider_options))
         end
         chat
       end
@@ -331,11 +359,12 @@ module RubyLLM
       def softmax(ratings)
         peak = ratings.max
         weights = ratings.map { |rating| Math.exp(rating - peak) }
-        weights.map { |weight| weight / weights.sum }
+        total = weights.sum
+        weights.map { |weight| weight / total }
       end
 
       def concentration(probabilities)
-        return 0.0 if probabilities.one?
+        return 1.0 if probabilities.size == 1
         return 0.0 if probabilities.count(probabilities.max) > 1
 
         entropy = -probabilities.sum { |probability| probability.zero? ? 0.0 : probability * Math.log(probability) }

data/lib/ruby_llm/llm_judge/legacy.rb

@@ -26,7 +26,10 @@ module RubyLLM
         end
 
         def initialize(name, type:, instructions:, criteria:)
-          raise ArgumentError, 'A question name must be a String or Symbol' unless name.is_a?(String) || name.is_a?(Symbol)
+          unless name.is_a?(String) || name.is_a?(Symbol)
+            raise ArgumentError, 'A question name must be a String or Symbol'
+          end
+          raise ArgumentError, 'A question name cannot be empty' if name.to_s.empty?
 
           @name = name
           @type = type
@@ -38,20 +41,58 @@ module RubyLLM
 
         private
 
+        # Mirrors RubyLLM 2.1's Judge::Question validation.
         def validate!
+          raise ArgumentError, 'Question instructions must be text, a Hash, an Array, or nil' unless description?(instructions)
+
           case type
-          when :probability
-            return if criteria.nil?
-            unless criteria.is_a?(Hash) && (criteria.keys.map(&:to_s) - %w[yes no true false]).empty?
-              raise ArgumentError, 'Probability criteria must describe yes and no'
-            end
-          when :choice
-            raise ArgumentError, 'A choice needs a nonempty Hash of options' unless criteria.is_a?(Hash) && !criteria.empty?
-          when :score
-            unless criteria.is_a?(Array) && criteria.size >= 2 && criteria.none?(&:nil?)
-              raise ArgumentError, 'A score needs at least two non-nil levels'
-            end
+          when :probability then validate_probability!
+          when :choice then validate_choice!
+          when :score then validate_score!
+          end
+        end
+
+        def validate_probability!
+          return if criteria.nil?
+
+          unless criteria.is_a?(Hash) && (criteria.keys.map(&:to_s) - %w[yes no true false]).empty?
+            raise ArgumentError, 'Probability criteria must describe yes and no'
           end
+
+          positive = criteria.keys.map { |key| %w[yes true].include?(key.to_s) }
+          raise ArgumentError, 'Probability criteria contain duplicate outcomes' unless positive.uniq.size == positive.size
+
+          validate_descriptions!(criteria.values)
+        end
+
+        def validate_choice!
+          raise ArgumentError, 'A choice needs a nonempty Hash of options' unless criteria.is_a?(Hash) && !criteria.empty?
+          unless criteria.keys.all? { |key| (key.is_a?(String) || key.is_a?(Symbol)) && !key.to_s.empty? }
+            raise ArgumentError, 'Choice options must have nonempty String or Symbol names'
+          end
+
+          duplicate = criteria.keys.map(&:to_s).tally.find { |_, count| count > 1 }&.first
+          raise ArgumentError, "Duplicate judgment key: #{duplicate}" if duplicate
+
+          validate_descriptions!(criteria.values)
+        end
+
+        def validate_score!
+          unless criteria.is_a?(Array) && criteria.size >= 2 && criteria.none?(&:nil?)
+            raise ArgumentError, 'A score needs at least two non-nil levels'
+          end
+
+          validate_descriptions!(criteria)
+        end
+
+        def validate_descriptions!(values)
+          return if values.all? { |value| description?(value) }
+
+          raise ArgumentError, 'Descriptions must be text, a Hash, an Array, or nil'
+        end
+
+        def description?(value)
+          value.nil? || value.is_a?(String) || value.is_a?(Hash) || value.is_a?(Array)
         end
       end
 
@@ -140,10 +181,15 @@ module RubyLLM
 
       module_function
 
+      TOKEN_FIELDS = %i[input output cached cache_creation thinking].freeze
+
+      # Sums each field across calls; a field no call reported stays nil.
       def aggregate_tokens(tokens)
         tokens = tokens.compact
-        sum = ->(reader) { tokens.sum { |token| token.public_send(reader) || 0 } }
-        RubyLLM::Tokens.new(input: sum.(:input), output: sum.(:output))
+        RubyLLM::Tokens.new(**TOKEN_FIELDS.to_h do |field|
+          values = tokens.filter_map { |token| token.public_send(field) }
+          [field, values.empty? ? nil : values.sum]
+        end)
       end
     end
   end

data/lib/ruby_llm/llm_judge/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module LLMJudge
-    VERSION = '0.1.1'
+    VERSION = '0.1.2'
   end
 end

data/lib/ruby_llm/llm_judge.rb

@@ -14,12 +14,10 @@ module RubyLLM
     # Raised for invalid scoring responses. RubyLLM::Error takes (response, message)
     # before 1.16 and (message, response:) on main, so build it for either.
     class Error < RubyLLM::Error
+      MESSAGE_FIRST = RubyLLM::Error.instance_method(:initialize).parameters.include?(%i[key response])
+
       def initialize(message = nil)
-        if RubyLLM::Error.instance_method(:initialize).parameters.include?(%i[key response])
-          super(message)
-        else
-          super(nil, message)
-        end
+        MESSAGE_FIRST ? super(message) : super(nil, message)
       end
     end
   end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-llm_judge
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
 platform: ruby
 authors:
 - JP Camara