ruby_llm-llm_judge 0.1.3 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c438b1693068b5c47ada88cf13b23ed8786c597ed4c349d33d8c012f8edaedd2
-  data.tar.gz: ca228933f0f0772c7f523c788d9391edf012138b7f0faec84466da3026b1eb9b
+  metadata.gz: fc81f9575b6468ad1a7ea81b19d777ea169eca0619cf1995c59c0aab01128254
+  data.tar.gz: 6052e01d0b02b576c25fe5baa0545a8f89e9060e12d42ae65f3fa99e8e14f5f7
 SHA512:
-  metadata.gz: b50f0831b1e86781c9d0aaac5d9d96b69284b0e527841119df6a73f7379bee194b67b7e09bad5e72778ea35851749bff9e0787664c815f90ff83e3d653a4f6a9
-  data.tar.gz: c49adc022376e4b959c73d48704b293a536fd6e8a9580eaf8b495676b93c236ad7c02d24f748814c5394edb26edc7668bfee91acaf1fec03a43cbabb39153a18
+  metadata.gz: ba27da7ce65adbb3d39d6447decaa6b2454d2bbee7086577f81798cb213ccf573efd81c250d46a5bc29d7b58d811892a73918b7f89cddd1dec778b50620d8406
+  data.tar.gz: 411fd92bb9f7700b6075559ed5644cbdad5485a555fc6601249ebfc1c28d657dbc221255bf308f4358c1730cd3c506b08c553a7e2359ddb0e449a2c0e5aedaf7

data/CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## 0.1.4 (2026-10-03)
+
+- `structured_output: true` sends a strict JSON Schema with one-call requests, requiring every question and option.
+- `strategy: :committed` asks for one option per question and returns it with probability 1.
+- The provider accepts RubyLLM main's `with:` attachments argument and raises `UnsupportedAttachmentError` when images are passed. Judgments on RubyLLM main failed with `ArgumentError` before.
+- The README explains when to enable reasoning: off suits classification, while rule-following questions need it. See [the decision-mode benchmarks](benchmarks/decision-modes.md).
+
 ## 0.1.3 (2026-09-29)
 
 - Setting a `chat_provider_options` key to `nil` removes that field, including the Luna defaults and, before RubyLLM 2.1, the output limit field. A nested hash emptied this way is removed too.

data/README.md

@@ -142,6 +142,31 @@ result = RubyLLM::LLMJudge.judge(
 
 OpenRouter's [latency sorting](https://openrouter.ai/docs/guides/routing/provider-selection) chooses among its available providers using their observed response times. It sped up OpenRouter in our test, while direct OpenAI was faster still. The measured speed and answer-quality tradeoffs are below.
 
+### Strict response schemas
+
+Set `structured_output: true` to send a strict JSON Schema with the one-call request, through RubyLLM's `with_schema`. The schema requires every question and every option, and nothing else, so a provider that supports structured outputs cannot return the wrong shape. Without it, the gem validates the reply and makes its corrective retry. In our runs it had little effect on accuracy, cut GPT-6 Luna's corrective retries from 3.3 to 0.7 per 96 judgments, and added 40–350 ms. Models and providers without structured-output support reject the request, so leave it off for them.
+
+### Committed answers
+
+Set `strategy: :committed` to ask for one option per question instead of a distribution. The reply is shorter, and each answer comes back with probability 1 for the chosen option and 0 for the rest, so `confidence` is always 1.0 and the probabilities carry no calibration. Use it when you only need the decision; use `:single_request` or `:ratings` when you threshold on probabilities. It accepts `structured_output: true`, which limits each answer to that question's option IDs. On classification it was 100–300 ms faster than probabilities and as accurate or up to three cases in 64 less; on rule-following questions it was the most accurate mode with reasoning off.
+
+### Reasoning
+
+The Luna defaults disable reasoning, which suits classification: on the benchmarks below, Luna is as accurate with reasoning off and much faster. Questions that apply explicit rules to a state are different. On [StreamDecisionBench](https://github.com/JacobLinCool/StreamDecisionBench), whose questions carry decision policies, GPT-5.6 Luna through LLM Judge answered 47% of states correctly with reasoning off and 90% with reasoning effort low, at about twice the latency. The ratings strategy cannot apply a policy across a state and should not be used for these questions. [Classification and rule-following results by mode](benchmarks/decision-modes.md).
+
+To enable reasoning, override the default and leave room for it in the output limit:
+
+```ruby
+provider_options: {
+  strategy: :single_request,
+  max_output_tokens: 4096,
+  chat_provider_options: { reasoning_effort: 'low' }          # OpenAI
+  # chat_provider_options: { reasoning: { effort: 'low' } }   # OpenRouter
+}
+```
+
+Reasoning does not fit the `:ratings` strategy, whose four-token output limit is meant for a single digit.
+
 ## How scoring works
 
 The default `:ratings` strategy asks the model to rate every declared answer from 0 to 9. It makes one call per answer, with up to six calls running at once (`max_workers:` changes this), then applies softmax to the ratings. If a call fails after its retry, no new calls start, calls in flight finish, and the error is raised. It retries a malformed digit once. When the highest ratings tie on a Choice question, it makes a one-call JSON judgment for that question and uses its distribution. If that call also ties, it raises an error instead of choosing whichever option came first. `result.raw[:tie_breaks]` records each resolution, and token usage includes the extra call. Set `tie_breaker: :first` to use the original first-option rule, or `tie_break_max_output_tokens:` to change the tie-break response limit (default 8192).

data/lib/ruby_llm/llm_judge/engine.rb

@@ -21,18 +21,23 @@ module RubyLLM
                                     'each question\'s probabilities sum to 1. Evaluate each question independently ' \
                                     'against the same state. Return no explanations or markdown. The "answers" value ' \
                                     'must be an object whose keys are the question IDs.'
+      COMMITTED_INSTRUCTIONS = 'Answer all questions using only a JSON object with an "answers" field whose keys are ' \
+                               'the question IDs. For each question, give the ID of the one option that best applies. ' \
+                               'Evaluate each question independently against the same state. Return no explanations ' \
+                               'or markdown.'
+      STRATEGIES = %i[ratings single_request committed].freeze
 
       def initialize(config:, model: DEFAULT_MODEL, provider_options: {}, scorer: nil, responder: nil)
         @config = config
         options = provider_options.transform_keys(&:to_sym)
         allowed = %i[scoring_provider scoring_protocol temperature max_output_tokens chat_provider_options
                      max_arms max_input_bytes strategy malformed_retries tie_breaker tie_break_max_output_tokens
-                     max_workers]
+                     max_workers structured_output]
         unknown = options.keys - allowed
         raise ArgumentError, "Unknown LLMJudge provider options: #{unknown.join(', ')}" unless unknown.empty?
 
         @strategy = options.fetch(:strategy, :ratings).to_sym
-        raise ArgumentError, 'strategy must be :ratings or :single_request' unless %i[ratings single_request].include?(@strategy)
+        raise ArgumentError, 'strategy must be :ratings, :single_request, or :committed' unless STRATEGIES.include?(@strategy)
         @tie_breaker = options.fetch(:tie_breaker, :single_request).to_sym
         raise ArgumentError, 'tie_breaker must be :single_request or :first' unless %i[single_request first].include?(@tie_breaker)
 
@@ -48,7 +53,7 @@ module RubyLLM
           raise ArgumentError, 'scoring_protocol requires RubyLLM 2.1'
         end
         @temperature = options.fetch(:temperature, luna_defaults ? 0 : nil)
-        @max_output_tokens = options.fetch(:max_output_tokens, @strategy == :single_request ? 8192 : 4)
+        @max_output_tokens = options.fetch(:max_output_tokens, @strategy == :ratings ? 4 : 8192)
         @tie_break_max_output_tokens = options.fetch(:tie_break_max_output_tokens, 8192)
         chat_provider_options = options.fetch(:chat_provider_options, {})
         raise ArgumentError, 'chat_provider_options must be a Hash' unless chat_provider_options.is_a?(Hash)
@@ -59,6 +64,8 @@ module RubyLLM
         @max_input_bytes = options[:max_input_bytes]
         @malformed_retries = options.fetch(:malformed_retries, 1)
         @max_workers = options.fetch(:max_workers, MAX_WORKERS)
+        @structured_output = options.fetch(:structured_output, false)
+        raise ArgumentError, 'structured_output must be true or false' unless [true, false].include?(@structured_output)
         raise ArgumentError, 'max_workers must be a positive integer' unless @max_workers.is_a?(Integer) && @max_workers.positive?
         raise ArgumentError, 'max_output_tokens must be positive' unless @max_output_tokens.is_a?(Integer) && @max_output_tokens.positive?
         unless @tie_break_max_output_tokens.is_a?(Integer) && @tie_break_max_output_tokens.positive?
@@ -72,11 +79,12 @@ module RubyLLM
         end
 
         @scorer = scorer || method(:score_with_model)
-        @responder = responder || method(:respond_with_model)
+        @responder = responder
       end
 
       def judge(input, questions:, model:)
-        return judge_single_request(input, questions, model) if @strategy == :single_request
+        return judge_one_call(input, questions, model, :probabilities) if @strategy == :single_request
+        return judge_one_call(input, questions, model, :committed) if @strategy == :committed
 
         jobs = build_jobs(input, questions)
         results = run(jobs) { |job| @scorer.call(job[:prompt]) }
@@ -88,7 +96,7 @@ module RubyLLM
           chosen = answer(question, softmax(ratings))
           if question.type == :choice && ratings.count(ratings.max) > 1 && @tie_breaker == :single_request
             # Raises if the one-call judgment also ties, after its corrective retry.
-            judgment = judge_single_request(input, { question.name => question }, model)
+            judgment = judge_one_call(input, { question.name => question }, model, :probabilities)
             chosen = judgment.answers.fetch(question.name)
             tie_breaks << { question: question.name, ratings:, judgment: }
           end
@@ -113,34 +121,45 @@ module RubyLLM
 
       private
 
-      def judge_single_request(input, questions, model)
+      # One request for every question. :probabilities asks for a distribution per question;
+      # :committed asks for one option per question and returns it with probability 1.
+      def judge_one_call(input, questions, model, mode)
         specs = validated_options(input, questions)
-        original_prompt = single_request_prompt(input, specs)
+        original_prompt = one_call_prompt(input, specs)
         prompt = original_prompt
         attempts = []
         loop do
-          result = @responder.call(prompt)
+          result = respond(prompt, mode, specs)
           attempts << result
           begin
-            answers, reported, normalization = parse_single_request(result.fetch(:content), specs)
+            answers, reported, normalization = mode == :committed ? parse_committed(result.fetch(:content), specs)
+                                                                  : parse_single_request(result.fetch(:content), specs)
+            raw = { method: mode == :committed ? 'committed_answers' : 'single_request_probabilities',
+                    scoring_provider: @scoring_provider, scoring_model: @scoring_model,
+                    reported_probabilities: reported, attempts: attempts.size }
+            raw[:reported_totals] = normalization if normalization
             return Types::Judgment.new(
-              answers:, model: model.id,
-              tokens: aggregate_tokens(attempts.map { |attempt| attempt.fetch(:tokens) }),
-              raw: { method: 'single_request_probabilities', scoring_provider: @scoring_provider,
-                     scoring_model: @scoring_model, reported_probabilities: reported,
-                     reported_totals: normalization, attempts: attempts.size }
+              answers:, model: model.id, raw:,
+              tokens: aggregate_tokens(attempts.map { |attempt| attempt.fetch(:tokens) })
             )
           rescue RubyLLM::Error => error
             raise if attempts.size > @malformed_retries
 
             prompt = "#{original_prompt}\n\nThe previous answer was invalid: #{error.message}. " \
                      "Previous answer: #{result.fetch(:content).to_s[0, 4000]}\n" \
-                     'Return a complete corrected JSON object with every question and option.'
+                     'Return a complete corrected JSON object with every question.'
           end
         end
       end
 
-      def parse_single_request(content, specs)
+      def respond(prompt, mode, specs)
+        return @responder.call(prompt) if @responder
+
+        respond_with_model(prompt, mode:, schema: @structured_output ? answer_schema(mode, specs) : nil)
+      end
+
+      # The answers object keyed by question ID, accepting a fenced reply or a list of one-question objects.
+      def reported_answers(content, specs)
         parsed = JSON.parse(strip_code_fence(content))
         raise Error, 'Scoring model returned a non-object response' unless parsed.is_a?(Hash)
 
@@ -156,6 +175,44 @@ module RubyLLM
           raise Error, "Question IDs must be #{expected_questions.inspect}; got #{actual.inspect}"
         end
 
+        reported
+      end
+
+      def parse_committed(content, specs)
+        reported = reported_answers(content, specs)
+        answers = specs.to_h do |question, options|
+          keys = options.map { |name, _| name.to_s }
+          value = reported.fetch(question.name.to_s)
+          value = { 'yes' => 'true', 'no' => 'false' }.fetch(value.to_s.downcase, value.to_s) if question.type == :probability
+          index = keys.index(value.to_s) unless value.is_a?(Hash) || value.is_a?(Array)
+          raise Error, "The answer for #{question.name} must be one of #{keys.inspect}; got #{value.inspect}" unless index
+
+          [question.name, answer(question, keys.each_index.map { |position| position == index ? 1.0 : 0.0 })]
+        end
+        [answers, reported, nil]
+      rescue JSON::ParserError, KeyError, TypeError => error
+        raise Error, "Scoring model returned invalid JSON answers: #{error.message}"
+      end
+
+      # A strict JSON Schema that admits only complete answers to these questions.
+      def answer_schema(mode, specs)
+        properties = specs.to_h do |question, options|
+          keys = options.map { |name, _| name.to_s }
+          schema = if mode == :committed
+                     { type: 'string', enum: keys }
+                   else
+                     { type: 'object', properties: keys.to_h { |key| [key, { type: 'number' }] },
+                       required: keys, additionalProperties: false }
+                   end
+          [question.name.to_s, schema]
+        end
+        answers = { type: 'object', properties:, required: properties.keys, additionalProperties: false }
+        { name: 'judgment', strict: true,
+          schema: { type: 'object', properties: { answers: }, required: ['answers'], additionalProperties: false } }
+      end
+
+      def parse_single_request(content, specs)
+        reported = reported_answers(content, specs)
         normalization = {}
         answers = specs.to_h do |question, options|
           distribution = reported.fetch(question.name.to_s)
@@ -187,7 +244,9 @@ module RubyLLM
         raise Error, "Scoring model returned invalid JSON probabilities: #{error.message}"
       end
 
-      def single_request_prompt(input, specs)
+      # The state comes first: listing the questions first made Luna 6 retry more and answer less
+      # accurately on the benchmark cases.
+      def one_call_prompt(input, specs)
         request = {
           state: input,
           questions: specs.map do |question, options|
@@ -318,13 +377,16 @@ module RubyLLM
         end
       end
 
-      def respond_with_model(prompt)
+      def respond_with_model(prompt, mode:, schema: nil)
         limit = @strategy == :ratings ? @tie_break_max_output_tokens : @max_output_tokens
-        response = scoring_chat(instructions: SINGLE_REQUEST_INSTRUCTIONS, max_output_tokens: limit).ask(prompt)
-        { content: response.content.to_s, tokens: response.tokens }
+        instructions = mode == :committed ? COMMITTED_INSTRUCTIONS : SINGLE_REQUEST_INSTRUCTIONS
+        response = scoring_chat(instructions:, max_output_tokens: limit, schema:).ask(prompt)
+        # With a schema, RubyLLM parses the JSON reply into a Hash.
+        content = response.content
+        { content: content.is_a?(String) ? content : JSON.generate(content), tokens: response.tokens }
       end
 
-      def scoring_chat(instructions: SYSTEM_INSTRUCTIONS, max_output_tokens: @max_output_tokens)
+      def scoring_chat(instructions: SYSTEM_INSTRUCTIONS, max_output_tokens: @max_output_tokens, schema: nil)
         context = RubyLLM::Context.new(@config)
         chat = if NATIVE
                  context.chat(model: @scoring_model, provider: @scoring_provider,
@@ -334,6 +396,7 @@ module RubyLLM
                end
         chat.with_instructions(instructions)
         chat.with_temperature(@temperature) unless @temperature.nil?
+        chat.with_schema(schema) if schema
         if NATIVE
           chat.with_max_output_tokens(max_output_tokens)
           chat.with_provider_options(deep_compact(@chat_provider_options))

data/lib/ruby_llm/llm_judge/provider.rb

@@ -7,7 +7,10 @@ module RubyLLM
         'http://127.0.0.1'
       end
 
-      def judge(input, questions:, model:, provider_options: {})
+      # RubyLLM passes images as +with:+; they are not supported yet.
+      def judge(input, questions:, model:, with: [], provider_options: {})
+        raise UnsupportedAttachmentError, Array(with).first.mime_type unless Array(with).empty?
+
         Engine.new(config:, model: model.id, provider_options:).judge(input, questions:, model:)
       end
 

data/lib/ruby_llm/llm_judge/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module LLMJudge
-    VERSION = '0.1.3'
+    VERSION = '0.1.4'
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-llm_judge
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 0.1.4
 platform: ruby
 authors:
 - JP Camara