ruby_llm-llm_judge 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 71f6bd771e7f6bec3a9dcdf7036ee1238110c63ee530b15d32c6dadcfb0d0eaa
-  data.tar.gz: e54b13efadda477c3d3713c66184dc4ce01061b740e06ab52e95df0b672ae7e9
+  metadata.gz: c438b1693068b5c47ada88cf13b23ed8786c597ed4c349d33d8c012f8edaedd2
+  data.tar.gz: ca228933f0f0772c7f523c788d9391edf012138b7f0faec84466da3026b1eb9b
 SHA512:
-  metadata.gz: 4e29ff7f1b47222dfc71f6f096cd058615d2517b7e16b29f9c2a68f094f2af95be8a99232d5ba72cc1d6b9a932fa5f9bf290e701a9744ee1a241fc6a0c710834
-  data.tar.gz: 942f4a7da0c27d1741d3e0e2b80ae719335123d6ddf276b84b0e4d200ed315d5f839cfc1ea998e708b6b54f83d7b7e97c718f6d4a070e17d2bfe0be23e9b3541
+  metadata.gz: b50f0831b1e86781c9d0aaac5d9d96b69284b0e527841119df6a73f7379bee194b67b7e09bad5e72778ea35851749bff9e0787664c815f90ff83e3d653a4f6a9
+  data.tar.gz: c49adc022376e4b959c73d48704b293a536fd6e8a9580eaf8b495676b93c236ad7c02d24f748814c5394edb26edc7668bfee91acaf1fec03a43cbabb39153a18

data/CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## 0.1.3 (2026-09-29)
+
+- Setting a `chat_provider_options` key to `nil` removes that field, including the Luna defaults and, before RubyLLM 2.1, the output limit field. A nested hash emptied this way is removed too.
+- One-call answers given as a list of single-question objects are accepted when each question appears once. Claude Haiku 4.5 returns this shape on its first attempt, which previously cost a corrective retry on nearly every judgment.
+- The one-call instructions now say that `answers` is an object keyed by question ID. Haiku no longer needs corrective retries (59 of 96 benchmark cases before), and Luna scored at least as well in a back-to-back comparison. See [the Haiku benchmark](benchmarks/haiku.md).
+- Luna defaults apply to every OpenAI Luna model ID (`gpt-6-luna`, `gpt-5.6-luna`, `gpt-luna-latest`) and to OpenRouter's `openai/` Luna IDs, not only `gpt-6-luna` on OpenAI. On OpenRouter they send temperature zero and `reasoning: { enabled: false }`. With reasoning on, Luna's ratings often returned no digit. Pro variants and prefixed gateway IDs, such as `us.openai.gpt-5.6-luna` through `:openai`, are excluded.
+
 ## 0.1.2 (2026-09-28)
 
 - When a rating call fails, no new calls start and the error is raised once calls in flight finish. Previously the remaining calls kept running, and were billed, after `judge` raised.

data/README.md

@@ -84,9 +84,11 @@ result = RubyLLM::LLMJudge.judge(
 )
 ```
 
-Both strategies return valid judgments with this model. Its one-call answers arrive in a Markdown code fence, which the gem removes before parsing.
+Both strategies return valid judgments with this model. Its one-call answers arrive in a Markdown code fence, which the gem removes before parsing. On the benchmark cases it scored 56–57/64 on AG News and 29–30/32 on SST-2 at about 1.1 s median, a little below Luna. [Haiku results](benchmarks/haiku.md).
 
-The default Luna rating request uses Chat Completions, temperature zero, reasoning disabled, `store: false`, and a four-token output limit. Tune requests with `scoring_protocol`, `temperature`, `max_output_tokens`, and `chat_provider_options`. `chat_provider_options` is merged over those defaults, so setting one key, such as `service_tier: 'flex'`, keeps reasoning disabled; set a key explicitly to change a default.
+The default Luna rating request uses Chat Completions, temperature zero, reasoning disabled, `store: false`, and a four-token output limit. Tune requests with `scoring_protocol`, `temperature`, `max_output_tokens`, and `chat_provider_options`. `chat_provider_options` is merged over those defaults, so setting one key, such as `service_tier: 'flex'`, keeps reasoning disabled; set a key explicitly to change a default. Set a key to `nil` to remove that field from the request, whether it is a default or, before RubyLLM 2.1, the output limit field. For example, an app that routes RubyLLM 1.x OpenAI chats through the Responses API can send `chat_provider_options: { max_completion_tokens: nil, max_output_tokens: 256 }`.
+
+Luna defaults apply to each provider's own Luna model IDs: `gpt-6-luna`, `gpt-5.6-luna`, or `gpt-luna-latest` on `:openai`, which sends temperature zero, `reasoning_effort: 'none'`, and `store: false`; and `openai/gpt-6-luna` or `openai/gpt-5.6-luna` on `:openrouter`, which sends temperature zero and `reasoning: { enabled: false }`. Pro variants, other providers such as `:bedrock`, and prefixed IDs such as `us.openai.gpt-5.6-luna` sent through `:openai` to an OpenAI-compatible gateway get no Luna defaults, because their request format may differ; set `temperature:` and disable reasoning in that endpoint's format yourself. With reasoning left on, the ratings strategy's four-token output limit is spent on reasoning: on eight AG News cases, `gpt-6-luna` through OpenRouter returned no digit for all eight, and `gpt-5.6-luna` for four. RubyLLM before 2.1 sends temperature 1.0 for any OpenAI model whose ID starts with `gpt-5`, so `gpt-5.6-luna` on `:openai` runs at 1.0 there unless you pass `chat_provider_options: { temperature: 0 }`.
 
 ## One-call typed decisions
 

data/lib/ruby_llm/llm_judge/engine.rb

@@ -6,12 +6,21 @@ module RubyLLM
   module LLMJudge
     class Engine
       MAX_WORKERS = 6
+      # Luna model IDs by provider, which get temperature zero and reasoning disabled in that provider's
+      # format. Only each provider's own IDs match: an OpenAI-compatible gateway such as Bedrock, reached
+      # through :openai with an ID like "us.openai.gpt-5.6-luna", may use another request format. Pro
+      # variants are excluded: they exist to reason more.
+      LUNA_IDS = {
+        openai: /\Agpt-(?:\d+(?:\.\d+)?-)?luna(?:-latest)?\z/,
+        openrouter: %r{\A~?openai/gpt-(?:\d+(?:\.\d+)?-)?luna(?:-latest)?(?::\w+)?\z}
+      }.freeze
       SYSTEM_INSTRUCTIONS = 'Return exactly one ASCII digit 0-9. 9 means very likely to be the correct answer; 0 means very unlikely. No explanation.'
       SINGLE_REQUEST_INSTRUCTIONS = 'Answer all questions using only a JSON object with an "answers" field. ' \
                                     'For each question, return an object mapping every supplied option ID to a ' \
                                     'probability between 0 and 1. Include every option exactly once and make ' \
                                     'each question\'s probabilities sum to 1. Evaluate each question independently ' \
-                                    'against the same state. Return no explanations or markdown.'
+                                    'against the same state. Return no explanations or markdown. The "answers" value ' \
+                                    'must be an object whose keys are the question IDs.'
 
       def initialize(config:, model: DEFAULT_MODEL, provider_options: {}, scorer: nil, responder: nil)
         @config = config
@@ -31,8 +40,9 @@ module RubyLLM
         @scoring_model = model
         raise ArgumentError, 'A model is required' unless @scoring_model.is_a?(String) && !@scoring_model.empty?
 
-        luna_defaults = @scoring_provider == :openai && @scoring_model == DEFAULT_MODEL
-        @scoring_protocol = options.fetch(:scoring_protocol, luna_defaults ? :chat_completions : nil)
+        luna_defaults = LUNA_IDS.fetch(@scoring_provider, nil)&.match?(@scoring_model) || false
+        @scoring_protocol = options.fetch(:scoring_protocol,
+                                          luna_defaults && @scoring_provider == :openai ? :chat_completions : nil)
         # Before 2.1, RubyLLM chats use each provider's one chat API (Chat Completions for OpenAI).
         if !NATIVE && ![nil, :chat_completions].include?(@scoring_protocol&.to_sym)
           raise ArgumentError, 'scoring_protocol requires RubyLLM 2.1'
@@ -135,6 +145,11 @@ module RubyLLM
         raise Error, 'Scoring model returned a non-object response' unless parsed.is_a?(Hash)
 
         reported = parsed.fetch('answers')
+        # Some models mirror the prompt's question list: [{"id": {...}}, ...].
+        if reported.is_a?(Array) && reported.all? { |item| item.is_a?(Hash) }
+          ids = reported.flat_map(&:keys)
+          reported = reported.reduce({}, :merge) if ids.uniq.size == ids.size
+        end
         expected_questions = specs.map { |question, _| question.name.to_s }
         unless reported.is_a?(Hash) && reported.keys.sort == expected_questions.sort
           actual = reported.is_a?(Hash) ? reported.keys : reported.class.name
@@ -280,10 +295,13 @@ module RubyLLM
         end
       end
 
+      # Luna rates poorly with reasoning on: the output limit is spent before the answer.
       def default_chat_options(luna_defaults)
-        return {} unless @scoring_provider == :openai
-
-        luna_defaults ? { store: false, reasoning_effort: 'none' } : { store: false }
+        case @scoring_provider
+        when :openai then luna_defaults ? { store: false, reasoning_effort: 'none' } : { store: false }
+        when :openrouter then luna_defaults ? { reasoning: { enabled: false } } : {}
+        else {}
+        end
       end
 
       def score_with_model(prompt)
@@ -318,13 +336,27 @@ module RubyLLM
         chat.with_temperature(@temperature) unless @temperature.nil?
         if NATIVE
           chat.with_max_output_tokens(max_output_tokens)
-          chat.with_provider_options(@chat_provider_options)
+          chat.with_provider_options(deep_compact(@chat_provider_options))
         else
-          chat.with_params(**deep_merge(max_output_tokens_param(max_output_tokens), @chat_provider_options))
+          chat.with_params(**deep_compact(deep_merge(max_output_tokens_param(max_output_tokens), @chat_provider_options)))
         end
         chat
       end
 
+      # A nil in chat_provider_options removes that field: a default, or before 2.1
+      # the output limit field. A hash emptied by removal is removed too.
+      def deep_compact(hash)
+        hash.each_with_object({}) do |(key, value), compacted|
+          next if value.nil?
+
+          if value.is_a?(Hash) && !value.empty?
+            value = deep_compact(value)
+            next if value.empty?
+          end
+          compacted[key] = value
+        end
+      end
+
       # The request field each provider reads for the output limit, as RubyLLM 2.1 renders it.
       def max_output_tokens_param(limit)
         case @scoring_provider

data/lib/ruby_llm/llm_judge/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module LLMJudge
-    VERSION = '0.1.2'
+    VERSION = '0.1.3'
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-llm_judge
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.3
 platform: ruby
 authors:
 - JP Camara