ruby_llm-llm_judge 0.1.1 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 451771cadd6d75e7ea6361b43814d186bc0b9e3aeb0dea5f128fdc3c4a68e982
-  data.tar.gz: ca97b903a28238275417a96551abbf440198eb2dea95d52196cfb2e5d1b0d173
+  metadata.gz: c438b1693068b5c47ada88cf13b23ed8786c597ed4c349d33d8c012f8edaedd2
+  data.tar.gz: ca228933f0f0772c7f523c788d9391edf012138b7f0faec84466da3026b1eb9b
 SHA512:
-  metadata.gz: ff30bd52e7b0dcb79f8804458e622d725ad3570351e00d180efb9ed040d4bb07c8551c37024f1730ca876125f52513e0e28a557fb9ed8f9d22211b286266aa6d
-  data.tar.gz: 25259a68bba8ffc259d343f076b7962788f236143ca8dabe43c03e626181f9bac24bfb50a98b50951130478be94951c56944775072614732c1341aa3218626eb
+  metadata.gz: b50f0831b1e86781c9d0aaac5d9d96b69284b0e527841119df6a73f7379bee194b67b7e09bad5e72778ea35851749bff9e0787664c815f90ff83e3d653a4f6a9
+  data.tar.gz: c49adc022376e4b959c73d48704b293a536fd6e8a9580eaf8b495676b93c236ad7c02d24f748814c5394edb26edc7668bfee91acaf1fec03a43cbabb39153a18

data/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## 0.1.3 (2026-09-29)
+
+- Setting a `chat_provider_options` key to `nil` removes that field, including the Luna defaults and, before RubyLLM 2.1, the output limit field. A nested hash emptied this way is removed too.
+- One-call answers given as a list of single-question objects are accepted when each question appears once. Claude Haiku 4.5 returns this shape on its first attempt, which previously cost a corrective retry on nearly every judgment.
+- The one-call instructions now say that `answers` is an object keyed by question ID. Haiku no longer needs corrective retries (59 of 96 benchmark cases before), and Luna scored at least as well in a back-to-back comparison. See [the Haiku benchmark](benchmarks/haiku.md).
+- Luna defaults apply to every OpenAI Luna model ID (`gpt-6-luna`, `gpt-5.6-luna`, `gpt-luna-latest`) and to OpenRouter's `openai/` Luna IDs, not only `gpt-6-luna` on OpenAI. On OpenRouter they send temperature zero and `reasoning: { enabled: false }`. With reasoning on, Luna's ratings often returned no digit. Pro variants and prefixed gateway IDs, such as `us.openai.gpt-5.6-luna` through `:openai`, are excluded.
+
+## 0.1.2 (2026-09-28)
+
+- When a rating call fails, no new calls start and the error is raised once calls in flight finish. Previously the remaining calls kept running, and were billed, after `judge` raised.
+- `chat_provider_options` is merged over the defaults instead of replacing them. Setting one option no longer re-enables reasoning or drops `store: false` for Luna.
+- One-call JSON inside a Markdown code fence is accepted. Claude Haiku 4.5 now works with `strategy: :single_request`.
+- A one-call Choice whose top probabilities tie gets the corrective retry, then raises, instead of selecting the first option. `tie_breaker: :first` keeps the old behavior.
+- New `max_workers:` option (default 6) for concurrent rating calls.
+- Unknown provider options are named in the error.
+- A one-option Choice reports confidence 1.0.
+- RubyLLM 1.13 through 1.16: question validation matches RubyLLM 2.1 (duplicate or empty option names, duplicate probability outcomes, description types), and token totals keep cached and thinking tokens.
+
 ## 0.1.1 (2026-09-28)
 
 - Fix invalid scoring responses raising `NoMethodError` on RubyLLM 1.13, which also skipped the corrective retry. They now raise `RubyLLM::LLMJudge::Error`, a `RubyLLM::Error`, on every supported RubyLLM version.

data/README.md

@@ -19,7 +19,7 @@ The gem also runs on RubyLLM 1.13 through 1.16, which predate the Judge API. The
 What differs before 2.1:
 
 - Results are `RubyLLM::LLMJudge::Legacy` objects, not `RubyLLM::Judgment`, `RubyLLM::Choice`, and so on. Avoid class checks until you upgrade.
-- There is no `RubyLLM.judge`, `RubyLLM::Judge` class DSL, `cost`, `context:` instrumentation, or `metadata:`. Passing `metadata:` raises.
+- There is no `RubyLLM.judge`, `RubyLLM::Judge` class DSL, `cost`, instrumentation, or `metadata:`. Passing `metadata:` raises. `context:` is accepted and supplies its configuration.
 - `scoring_protocol` accepts only `:chat_completions`, the API RubyLLM 1.x uses for OpenAI.
 - The output limit and `chat_provider_options` are sent with `with_params`, using the field each provider expects (`max_completion_tokens` for OpenAI and Azure, `generationConfig.maxOutputTokens` for Gemini and Vertex AI, `inferenceConfig.maxTokens` for Bedrock, and `max_tokens` otherwise).
 
@@ -70,20 +70,25 @@ TicketTriage.judge('Please refund today.').urgent.probability
 
 ## Choose a model
 
-Pass the chat model as `model:` and its provider as `scoring_provider`:
+Pass the chat model as `model:` and its provider as `scoring_provider`. For example, Claude Haiku 4.5 through OpenRouter:
 
 ```ruby
 result = RubyLLM::LLMJudge.judge(
   'Please refund the duplicate charge today.',
-  model: 'claude-haiku-4-5',
+  model: 'anthropic/claude-haiku-4.5',
   questions: { urgent: { type: :probability, instructions: 'Does this need attention today?' } },
   provider_options: {
-    scoring_provider: :anthropic
+    scoring_provider: :openrouter,
+    temperature: 0
   }
 )
 ```
 
-The default Luna rating request uses Chat Completions, temperature zero, reasoning disabled, `store: false`, and a four-token output limit. You can tune requests with `scoring_protocol`, `temperature`, `max_output_tokens`, and `chat_provider_options`.
+Both strategies return valid judgments with this model. Its one-call answers arrive in a Markdown code fence, which the gem removes before parsing. On the benchmark cases it scored 56–57/64 on AG News and 29–30/32 on SST-2 at about 1.1 s median, a little below Luna. [Haiku results](benchmarks/haiku.md).
+
+The default Luna rating request uses Chat Completions, temperature zero, reasoning disabled, `store: false`, and a four-token output limit. Tune requests with `scoring_protocol`, `temperature`, `max_output_tokens`, and `chat_provider_options`. `chat_provider_options` is merged over those defaults, so setting one key, such as `service_tier: 'flex'`, keeps reasoning disabled; set a key explicitly to change a default. Set a key to `nil` to remove that field from the request, whether it is a default or, before RubyLLM 2.1, the output limit field. For example, an app that routes RubyLLM 1.x OpenAI chats through the Responses API can send `chat_provider_options: { max_completion_tokens: nil, max_output_tokens: 256 }`.
+
+Luna defaults apply to each provider's own Luna model IDs: `gpt-6-luna`, `gpt-5.6-luna`, or `gpt-luna-latest` on `:openai`, which sends temperature zero, `reasoning_effort: 'none'`, and `store: false`; and `openai/gpt-6-luna` or `openai/gpt-5.6-luna` on `:openrouter`, which sends temperature zero and `reasoning: { enabled: false }`. Pro variants, other providers such as `:bedrock`, and prefixed IDs such as `us.openai.gpt-5.6-luna` sent through `:openai` to an OpenAI-compatible gateway get no Luna defaults, because their request format may differ; set `temperature:` and disable reasoning in that endpoint's format yourself. With reasoning left on, the ratings strategy's four-token output limit is spent on reasoning: on eight AG News cases, `gpt-6-luna` through OpenRouter returned no digit for all eight, and `gpt-5.6-luna` for four. RubyLLM before 2.1 sends temperature 1.0 for any OpenAI model whose ID starts with `gpt-5`, so `gpt-5.6-luna` on `:openai` runs at 1.0 there unless you pass `chat_provider_options: { temperature: 0 }`.
 
 ## One-call typed decisions
 
@@ -107,7 +112,7 @@ result.urgent.probability
 result.department.probabilities
 ```
 
-The model returns one JSON object containing a distribution for each question. The gem checks that every question and option is present, validates each probability, normalizes each distribution, and builds RubyLLM's typed answers. `result.raw[:reported_probabilities]` and `result.raw[:reported_totals]` preserve the model's original numbers for inspection. With an OpenAI key configured, this example calls Luna directly through Chat Completions with reasoning disabled, temperature zero, and `store: false`. The 1024-token limit suits small judgments; omit it for large question sets to use the 8192-token default. It makes one corrective retry for malformed JSON or missing fields, counting both calls in `result.tokens` and `result.raw[:attempts]`. Set `malformed_retries: 0` to disable that retry. An invalid response after retries raises an error.
+The model returns one JSON object containing a distribution for each question. The gem checks that every question and option is present, validates each probability, normalizes each distribution, and builds RubyLLM's typed answers. `result.raw[:reported_probabilities]` and `result.raw[:reported_totals]` preserve the model's original numbers for inspection. With an OpenAI key configured, this example calls Luna directly through Chat Completions with reasoning disabled, temperature zero, and `store: false`. The 1024-token limit suits small judgments; omit it for large question sets to use the 8192-token default. It makes one corrective retry for malformed JSON, missing fields, or a Choice whose top probabilities tie, counting both calls in `result.tokens` and `result.raw[:attempts]`. A tie that remains raises instead of selecting the first option; set `tie_breaker: :first` to accept the first option. Set `malformed_retries: 0` to disable that retry. An invalid response after retries raises an error.
 
 If you use OpenRouter for Luna, prioritize the provider with the lowest observed latency:
 
@@ -139,11 +144,11 @@ OpenRouter's [latency sorting](https://openrouter.ai/docs/guides/routing/provide
 
 ## How scoring works
 
-The default `:ratings` strategy asks the model to rate every declared answer from 0 to 9. It makes one call per answer, with up to six calls running at once, then applies softmax to the ratings. It retries a malformed digit once. When the highest ratings tie on a Choice question, it makes a one-call JSON judgment for that question and uses its distribution. If that call also ties, it raises an error instead of choosing whichever option came first. `result.raw[:tie_breaks]` records each resolution, and token usage includes the extra call. Set `tie_breaker: :first` to use the original first-option rule, or `tie_break_max_output_tokens:` to change the tie-break response limit (default 8192).
+The default `:ratings` strategy asks the model to rate every declared answer from 0 to 9. It makes one call per answer, with up to six calls running at once (`max_workers:` changes this), then applies softmax to the ratings. If a call fails after its retry, no new calls start, calls in flight finish, and the error is raised. It retries a malformed digit once. When the highest ratings tie on a Choice question, it makes a one-call JSON judgment for that question and uses its distribution. If that call also ties, it raises an error instead of choosing whichever option came first. `result.raw[:tie_breaks]` records each resolution, and token usage includes the extra call. Set `tie_breaker: :first` to use the original first-option rule, or `tie_break_max_output_tokens:` to change the tie-break response limit (default 8192).
 
 The `:single_request` strategy asks for all distributions in one call. Both strategies build the same RubyLLM answer types: a `choice` returns the selected answer and distribution; a `score` returns the distribution and its weighted level; a `probability` returns the positive answer's share. The gem supports Judge's 1–255 choice options, 2–10 score levels, and multiple questions in one judgment.
 
-These probabilities compare the answers you supplied. The `:single_request` values are reported by the chat model; the default ratings values come from softmax over its 0–9 ratings. Neither strategy establishes calibration by itself. Include an `other` or `escalate` choice when the named answers may not cover the input. `confidence` measures how concentrated the returned distribution is. Use labeled examples to set any automation thresholds.
+These probabilities compare the answers you supplied. The `:single_request` values are reported by the chat model; the default ratings values come from softmax over its 0–9 ratings. Neither strategy establishes calibration by itself. Include an `other` or `escalate` choice when the named answers may not cover the input. `confidence` measures how concentrated the returned distribution is: 1.0 for a single option, 0.0 for a tie. Use labeled examples to set any automation thresholds.
 
 Set optional `max_arms` and `max_input_bytes` in `provider_options` to cap work per judgment. For example, `{ max_arms: 12, max_input_bytes: 32_768 }` rejects larger requests before scoring. Token usage is aggregated across calls, and `raw` contains the ratings and model metadata.
 
@@ -211,7 +216,7 @@ Latency sorting reduced median time by 20% on AG News and 28% on SST-2. It was f
 
 ## Development
 
-The gem uses RubyLLM's Judge API when it is present and the `Legacy` path otherwise. `test/llm_judge_test.rb` covers the Judge API and `test/legacy_test.rb` covers RubyLLM 1.13 through 1.16; each skips on the other side.
+The gem uses RubyLLM's Judge API when it is present and the `Legacy` path otherwise. `test/llm_judge_test.rb` covers the Judge API and `test/legacy_test.rb` covers RubyLLM 1.13 through 1.16; each skips on the other side. `test/engine_test.rb` and `test/http_test.rb` run on every version; the HTTP tests send real RubyLLM requests to WebMock stubs and check their bodies.
 
 ```bash
 bundle exec rake test                              # the latest released RubyLLM

data/lib/ruby_llm/llm_judge/engine.rb

@@ -6,19 +6,30 @@ module RubyLLM
   module LLMJudge
     class Engine
       MAX_WORKERS = 6
+      # Luna model IDs by provider, which get temperature zero and reasoning disabled in that provider's
+      # format. Only each provider's own IDs match: an OpenAI-compatible gateway such as Bedrock, reached
+      # through :openai with an ID like "us.openai.gpt-5.6-luna", may use another request format. Pro
+      # variants are excluded: they exist to reason more.
+      LUNA_IDS = {
+        openai: /\Agpt-(?:\d+(?:\.\d+)?-)?luna(?:-latest)?\z/,
+        openrouter: %r{\A~?openai/gpt-(?:\d+(?:\.\d+)?-)?luna(?:-latest)?(?::\w+)?\z}
+      }.freeze
       SYSTEM_INSTRUCTIONS = 'Return exactly one ASCII digit 0-9. 9 means very likely to be the correct answer; 0 means very unlikely. No explanation.'
       SINGLE_REQUEST_INSTRUCTIONS = 'Answer all questions using only a JSON object with an "answers" field. ' \
                                     'For each question, return an object mapping every supplied option ID to a ' \
                                     'probability between 0 and 1. Include every option exactly once and make ' \
                                     'each question\'s probabilities sum to 1. Evaluate each question independently ' \
-                                    'against the same state. Return no explanations or markdown.'
+                                    'against the same state. Return no explanations or markdown. The "answers" value ' \
+                                    'must be an object whose keys are the question IDs.'
 
       def initialize(config:, model: DEFAULT_MODEL, provider_options: {}, scorer: nil, responder: nil)
         @config = config
         options = provider_options.transform_keys(&:to_sym)
         allowed = %i[scoring_provider scoring_protocol temperature max_output_tokens chat_provider_options
-                     max_arms max_input_bytes strategy malformed_retries tie_breaker tie_break_max_output_tokens]
-        raise ArgumentError, 'Unknown LLMJudge provider options' unless (options.keys - allowed).empty?
+                     max_arms max_input_bytes strategy malformed_retries tie_breaker tie_break_max_output_tokens
+                     max_workers]
+        unknown = options.keys - allowed
+        raise ArgumentError, "Unknown LLMJudge provider options: #{unknown.join(', ')}" unless unknown.empty?
 
         @strategy = options.fetch(:strategy, :ratings).to_sym
         raise ArgumentError, 'strategy must be :ratings or :single_request' unless %i[ratings single_request].include?(@strategy)
@@ -29,8 +40,9 @@ module RubyLLM
         @scoring_model = model
         raise ArgumentError, 'A model is required' unless @scoring_model.is_a?(String) && !@scoring_model.empty?
 
-        luna_defaults = @scoring_provider == :openai && @scoring_model == DEFAULT_MODEL
-        @scoring_protocol = options.fetch(:scoring_protocol, luna_defaults ? :chat_completions : nil)
+        luna_defaults = LUNA_IDS.fetch(@scoring_provider, nil)&.match?(@scoring_model) || false
+        @scoring_protocol = options.fetch(:scoring_protocol,
+                                          luna_defaults && @scoring_provider == :openai ? :chat_completions : nil)
         # Before 2.1, RubyLLM chats use each provider's one chat API (Chat Completions for OpenAI).
         if !NATIVE && ![nil, :chat_completions].include?(@scoring_protocol&.to_sym)
           raise ArgumentError, 'scoring_protocol requires RubyLLM 2.1'
@@ -38,11 +50,16 @@ module RubyLLM
         @temperature = options.fetch(:temperature, luna_defaults ? 0 : nil)
         @max_output_tokens = options.fetch(:max_output_tokens, @strategy == :single_request ? 8192 : 4)
         @tie_break_max_output_tokens = options.fetch(:tie_break_max_output_tokens, 8192)
-        @chat_provider_options = options.fetch(:chat_provider_options, default_chat_options(luna_defaults))
+        chat_provider_options = options.fetch(:chat_provider_options, {})
+        raise ArgumentError, 'chat_provider_options must be a Hash' unless chat_provider_options.is_a?(Hash)
+
+        # Merged over the defaults, so tuning one option keeps the others (such as disabled reasoning).
+        @chat_provider_options = deep_merge(default_chat_options(luna_defaults), chat_provider_options)
         @max_arms = options[:max_arms]
         @max_input_bytes = options[:max_input_bytes]
         @malformed_retries = options.fetch(:malformed_retries, 1)
-        raise ArgumentError, 'chat_provider_options must be a Hash' unless @chat_provider_options.is_a?(Hash)
+        @max_workers = options.fetch(:max_workers, MAX_WORKERS)
+        raise ArgumentError, 'max_workers must be a positive integer' unless @max_workers.is_a?(Integer) && @max_workers.positive?
         raise ArgumentError, 'max_output_tokens must be positive' unless @max_output_tokens.is_a?(Integer) && @max_output_tokens.positive?
         unless @tie_break_max_output_tokens.is_a?(Integer) && @tie_break_max_output_tokens.positive?
           raise ArgumentError, 'tie_break_max_output_tokens must be positive'
@@ -70,12 +87,9 @@ module RubyLLM
           }
           chosen = answer(question, softmax(ratings))
           if question.type == :choice && ratings.count(ratings.max) > 1 && @tie_breaker == :single_request
+            # Raises if the one-call judgment also ties, after its corrective retry.
             judgment = judge_single_request(input, { question.name => question }, model)
             chosen = judgment.answers.fetch(question.name)
-            probabilities = chosen.probabilities.values
-            if probabilities.count(probabilities.max) > 1
-              raise Error, "Scoring model could not resolve the tie for #{question.name}"
-            end
             tie_breaks << { question: question.name, ratings:, judgment: }
           end
           [question.name, chosen]
@@ -127,10 +141,15 @@ module RubyLLM
       end
 
       def parse_single_request(content, specs)
-        parsed = JSON.parse(content)
+        parsed = JSON.parse(strip_code_fence(content))
         raise Error, 'Scoring model returned a non-object response' unless parsed.is_a?(Hash)
 
         reported = parsed.fetch('answers')
+        # Some models mirror the prompt's question list: [{"id": {...}}, ...].
+        if reported.is_a?(Array) && reported.all? { |item| item.is_a?(Hash) }
+          ids = reported.flat_map(&:keys)
+          reported = reported.reduce({}, :merge) if ids.uniq.size == ids.size
+        end
         expected_questions = specs.map { |question, _| question.name.to_s }
         unless reported.is_a?(Hash) && reported.keys.sort == expected_questions.sort
           actual = reported.is_a?(Hash) ? reported.keys : reported.class.name
@@ -154,6 +173,12 @@ module RubyLLM
           total = values.sum.to_f
           raise Error, "Scoring model returned zero probability for #{question.name}" unless total.positive?
 
+          # A tied Choice would otherwise select whichever option was listed first.
+          if question.type == :choice && @tie_breaker == :single_request && values.count(values.max) > 1
+            raise Error, "Scoring model could not resolve the tie for #{question.name}; " \
+                         'one option must have the highest probability'
+          end
+
           normalization[question.name.to_s] = total
           [question.name, answer(question, values.map { |value| value / total })]
         end
@@ -230,32 +255,53 @@ module RubyLLM
         value.is_a?(String) ? value : JSON.generate(value)
       end
 
+      # Runs jobs on up to @max_workers threads. After a job fails, no new jobs
+      # start; calls already in flight finish, then the first error is raised.
       def run(jobs)
         queue = Queue.new
         jobs.each_with_index { |job, index| queue << [index, job] }
         results = Array.new(jobs.size)
-        workers = Array.new([jobs.size, MAX_WORKERS].min) do
+        errors = Queue.new
+        workers = Array.new([jobs.size, @max_workers].min) do
           Thread.new do
-            Thread.current.report_on_exception = false
-            loop do
+            while errors.empty?
               begin
                 index, job = queue.pop(true)
               rescue ThreadError
                 break
               end
-              results[index] = yield job
+              begin
+                results[index] = yield job
+              rescue StandardError => error
+                errors << error
+              end
             end
           end
         end
         workers.each(&:join)
-        workers.each(&:value)
+        raise errors.pop unless errors.empty?
+
         results
       end
 
-      def default_chat_options(luna_defaults)
-        return {} unless @scoring_provider == :openai
+      # Some models wrap JSON in a Markdown code fence despite the instructions.
+      def strip_code_fence(content)
+        content.to_s.strip.sub(/\A```(?:json)?[ \t]*\n?/i, '').sub(/\n?```\z/, '')
+      end
 
-        luna_defaults ? { store: false, reasoning_effort: 'none' } : { store: false }
+      def deep_merge(base, overrides)
+        base.merge(overrides) do |_key, old, new|
+          old.is_a?(Hash) && new.is_a?(Hash) ? deep_merge(old, new) : new
+        end
+      end
+
+      # Luna rates poorly with reasoning on: the output limit is spent before the answer.
+      def default_chat_options(luna_defaults)
+        case @scoring_provider
+        when :openai then luna_defaults ? { store: false, reasoning_effort: 'none' } : { store: false }
+        when :openrouter then luna_defaults ? { reasoning: { enabled: false } } : {}
+        else {}
+        end
       end
 
       def score_with_model(prompt)
@@ -290,13 +336,27 @@ module RubyLLM
         chat.with_temperature(@temperature) unless @temperature.nil?
         if NATIVE
           chat.with_max_output_tokens(max_output_tokens)
-          chat.with_provider_options(@chat_provider_options)
+          chat.with_provider_options(deep_compact(@chat_provider_options))
         else
-          chat.with_params(**RubyLLM::Utils.deep_merge(max_output_tokens_param(max_output_tokens), @chat_provider_options))
+          chat.with_params(**deep_compact(deep_merge(max_output_tokens_param(max_output_tokens), @chat_provider_options)))
         end
         chat
       end
 
+      # A nil in chat_provider_options removes that field: a default, or before 2.1
+      # the output limit field. A hash emptied by removal is removed too.
+      def deep_compact(hash)
+        hash.each_with_object({}) do |(key, value), compacted|
+          next if value.nil?
+
+          if value.is_a?(Hash) && !value.empty?
+            value = deep_compact(value)
+            next if value.empty?
+          end
+          compacted[key] = value
+        end
+      end
+
       # The request field each provider reads for the output limit, as RubyLLM 2.1 renders it.
       def max_output_tokens_param(limit)
         case @scoring_provider
@@ -331,11 +391,12 @@ module RubyLLM
       def softmax(ratings)
         peak = ratings.max
         weights = ratings.map { |rating| Math.exp(rating - peak) }
-        weights.map { |weight| weight / weights.sum }
+        total = weights.sum
+        weights.map { |weight| weight / total }
       end
 
       def concentration(probabilities)
-        return 0.0 if probabilities.one?
+        return 1.0 if probabilities.size == 1
         return 0.0 if probabilities.count(probabilities.max) > 1
 
         entropy = -probabilities.sum { |probability| probability.zero? ? 0.0 : probability * Math.log(probability) }

data/lib/ruby_llm/llm_judge/legacy.rb

@@ -26,7 +26,10 @@ module RubyLLM
         end
 
         def initialize(name, type:, instructions:, criteria:)
-          raise ArgumentError, 'A question name must be a String or Symbol' unless name.is_a?(String) || name.is_a?(Symbol)
+          unless name.is_a?(String) || name.is_a?(Symbol)
+            raise ArgumentError, 'A question name must be a String or Symbol'
+          end
+          raise ArgumentError, 'A question name cannot be empty' if name.to_s.empty?
 
           @name = name
           @type = type
@@ -38,20 +41,58 @@ module RubyLLM
 
         private
 
+        # Mirrors RubyLLM 2.1's Judge::Question validation.
         def validate!
+          raise ArgumentError, 'Question instructions must be text, a Hash, an Array, or nil' unless description?(instructions)
+
           case type
-          when :probability
-            return if criteria.nil?
-            unless criteria.is_a?(Hash) && (criteria.keys.map(&:to_s) - %w[yes no true false]).empty?
-              raise ArgumentError, 'Probability criteria must describe yes and no'
-            end
-          when :choice
-            raise ArgumentError, 'A choice needs a nonempty Hash of options' unless criteria.is_a?(Hash) && !criteria.empty?
-          when :score
-            unless criteria.is_a?(Array) && criteria.size >= 2 && criteria.none?(&:nil?)
-              raise ArgumentError, 'A score needs at least two non-nil levels'
-            end
+          when :probability then validate_probability!
+          when :choice then validate_choice!
+          when :score then validate_score!
+          end
+        end
+
+        def validate_probability!
+          return if criteria.nil?
+
+          unless criteria.is_a?(Hash) && (criteria.keys.map(&:to_s) - %w[yes no true false]).empty?
+            raise ArgumentError, 'Probability criteria must describe yes and no'
           end
+
+          positive = criteria.keys.map { |key| %w[yes true].include?(key.to_s) }
+          raise ArgumentError, 'Probability criteria contain duplicate outcomes' unless positive.uniq.size == positive.size
+
+          validate_descriptions!(criteria.values)
+        end
+
+        def validate_choice!
+          raise ArgumentError, 'A choice needs a nonempty Hash of options' unless criteria.is_a?(Hash) && !criteria.empty?
+          unless criteria.keys.all? { |key| (key.is_a?(String) || key.is_a?(Symbol)) && !key.to_s.empty? }
+            raise ArgumentError, 'Choice options must have nonempty String or Symbol names'
+          end
+
+          duplicate = criteria.keys.map(&:to_s).tally.find { |_, count| count > 1 }&.first
+          raise ArgumentError, "Duplicate judgment key: #{duplicate}" if duplicate
+
+          validate_descriptions!(criteria.values)
+        end
+
+        def validate_score!
+          unless criteria.is_a?(Array) && criteria.size >= 2 && criteria.none?(&:nil?)
+            raise ArgumentError, 'A score needs at least two non-nil levels'
+          end
+
+          validate_descriptions!(criteria)
+        end
+
+        def validate_descriptions!(values)
+          return if values.all? { |value| description?(value) }
+
+          raise ArgumentError, 'Descriptions must be text, a Hash, an Array, or nil'
+        end
+
+        def description?(value)
+          value.nil? || value.is_a?(String) || value.is_a?(Hash) || value.is_a?(Array)
         end
       end
 
@@ -140,10 +181,15 @@ module RubyLLM
 
       module_function
 
+      TOKEN_FIELDS = %i[input output cached cache_creation thinking].freeze
+
+      # Sums each field across calls; a field no call reported stays nil.
       def aggregate_tokens(tokens)
         tokens = tokens.compact
-        sum = ->(reader) { tokens.sum { |token| token.public_send(reader) || 0 } }
-        RubyLLM::Tokens.new(input: sum.(:input), output: sum.(:output))
+        RubyLLM::Tokens.new(**TOKEN_FIELDS.to_h do |field|
+          values = tokens.filter_map { |token| token.public_send(field) }
+          [field, values.empty? ? nil : values.sum]
+        end)
       end
     end
   end

data/lib/ruby_llm/llm_judge/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module LLMJudge
-    VERSION = '0.1.1'
+    VERSION = '0.1.3'
   end
 end

data/lib/ruby_llm/llm_judge.rb

@@ -14,12 +14,10 @@ module RubyLLM
     # Raised for invalid scoring responses. RubyLLM::Error takes (response, message)
     # before 1.16 and (message, response:) on main, so build it for either.
     class Error < RubyLLM::Error
+      MESSAGE_FIRST = RubyLLM::Error.instance_method(:initialize).parameters.include?(%i[key response])
+
       def initialize(message = nil)
-        if RubyLLM::Error.instance_method(:initialize).parameters.include?(%i[key response])
-          super(message)
-        else
-          super(nil, message)
-        end
+        MESSAGE_FIRST ? super(message) : super(nil, message)
       end
     end
   end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-llm_judge
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.3
 platform: ruby
 authors:
 - JP Camara