ruby_llm-llm_judge 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: '009274010f75c4b80a06f6b13b94fe9de590c7adf61ee78f57a771b8b115a742'
+  data.tar.gz: e801d20e964135b0e17a3de5ad6c0d121974afb4f61aa888035cb96f30a7a2d8
+SHA512:
+  metadata.gz: 59f5ca2d328b42993d30168ebc528d58fb98f468b0d7600e4d9d059eacf7b3332d205ffe003175d05ebb6078cf2eda09f5d779309b17f5609b57e8a2363ae522
+  data.tar.gz: 9e0f82133b0b3fb1c295ac4b96a00b838c5d685df04964da90d49ffdd037ef156545cf3c2bf942d0a7e708f405ca0fecb07f810f71800e3015314e2c6f18f6da

data/CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+## 0.1.0 (2026-09-28)
+
+Initial release.
+
+- `RubyLLM::LLMJudge.judge` answers `probability`, `choice`, and `score` questions with any RubyLLM chat model. The default model is `gpt-6-luna`.
+- Registers the `:llm_judge` provider for RubyLLM 2.1's Judge API, so `RubyLLM.judge` and `RubyLLM::Judge` classes return native `Judgment` and typed answers.
+- Runs on RubyLLM 1.13 through 1.16, which predate the Judge API. `RubyLLM::LLMJudge.judge` returns `Legacy` answers with the same readers.
+- `:ratings` strategy (default): one 0–9 rating per answer, up to six at once, turned into a distribution with softmax. Retries a malformed digit once and breaks tied Choice ratings with a one-call judgment.
+- `:single_request` strategy: one call returns a distribution for every question. Validates and normalizes the distributions and makes one corrective retry for malformed JSON.
+- `max_arms` and `max_input_bytes` limits reject oversized judgments before any call.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 JP Camara
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,223 @@
+# ruby_llm-llm_judge
+
+Use a RubyLLM chat model to answer [RubyLLM Judge](https://rubyllm.com/next/judgments/) questions. Define `probability`, `choice`, or `score` questions and receive RubyLLM's native `Judgment` and typed answers. Choose between parallel per-answer ratings and one-call typed distributions.
+
+## Install
+
+```ruby
+gem 'ruby_llm-llm_judge'
+```
+
+The gem works with RubyLLM 1.13 and later. On RubyLLM 2.1 it plugs into the Judge API; on earlier versions see [RubyLLM 1.13 to 1.16](#rubyllm-113-to-116).
+
+Configure your chat provider's credentials through RubyLLM. The default scoring model is `gpt-6-luna`; you can choose another RubyLLM chat model for each judgment.
+
+### RubyLLM 1.13 to 1.16
+
+The gem also runs on RubyLLM 1.13 through 1.16, which predate the Judge API. There, `RubyLLM::LLMJudge.judge` runs the same scoring directly and accepts the same questions and `provider_options`. Answers have the same readers as RubyLLM's (`probability`, `choice`, `probabilities`, `confidence`, `score`, `levels`, and `model`, `tokens`, `raw`, `[]` and `fetch` on the judgment), so call sites keep working after you upgrade to 2.1.
+
+What differs before 2.1:
+
+- Results are `RubyLLM::LLMJudge::Legacy` objects, not `RubyLLM::Judgment`, `RubyLLM::Choice`, and so on. Avoid class checks until you upgrade.
+- There is no `RubyLLM.judge`, `RubyLLM::Judge` class DSL, `cost`, `context:` instrumentation, or `metadata:`. Passing `metadata:` raises.
+- `scoring_protocol` accepts only `:chat_completions`, the API RubyLLM 1.x uses for OpenAI.
+- The output limit and `chat_provider_options` are sent with `with_params`, using the field each provider expects (`max_completion_tokens` for OpenAI and Azure, `generationConfig.maxOutputTokens` for Gemini and Vertex AI, `inferenceConfig.maxTokens` for Bedrock, and `max_tokens` otherwise).
+
+## Use
+
+```ruby
+require 'ruby_llm/llm_judge'
+
+result = RubyLLM::LLMJudge.judge(
+  'Please refund the duplicate charge today.',
+  questions: {
+    urgent: {
+      type: :probability,
+      instructions: 'Does this need attention today?',
+      criteria: { yes: 'An explicit deadline today', no: 'No deadline today' }
+    },
+    department: {
+      type: :choice,
+      instructions: 'Which team should handle this?',
+      options: { billing: 'Payments and refunds', technical: 'Bugs and integrations' }
+    },
+    frustration: {
+      type: :score,
+      instructions: 'How frustrated is the customer?',
+      levels: ['Calm', 'Frustrated', 'Angry']
+    }
+  }
+)
+
+result.urgent.probability
+result.department.choice
+result.department.probabilities
+result.frustration.score
+result.model  # => "gpt-6-luna"
+result.tokens
+```
+
+You can also use a reusable Judge class:
+
+```ruby
+class TicketTriage < RubyLLM::Judge
+  model 'gpt-6-luna', provider: :llm_judge, assume_model_exists: true
+  probability :urgent, 'Does this need attention today?'
+end
+
+TicketTriage.judge('Please refund today.').urgent.probability
+```
+
+## Choose a model
+
+Pass the chat model as `model:` and its provider as `scoring_provider`:
+
+```ruby
+result = RubyLLM::LLMJudge.judge(
+  'Please refund the duplicate charge today.',
+  model: 'claude-haiku-4-5',
+  questions: { urgent: { type: :probability, instructions: 'Does this need attention today?' } },
+  provider_options: {
+    scoring_provider: :anthropic
+  }
+)
+```
+
+The default Luna rating request uses Chat Completions, temperature zero, reasoning disabled, `store: false`, and a four-token output limit. You can tune requests with `scoring_protocol`, `temperature`, `max_output_tokens`, and `chat_provider_options`.
+
+## One-call typed decisions
+
+Set `strategy: :single_request` to send the state and every Judge question to the model in one request:
+
+```ruby
+result = RubyLLM::LLMJudge.judge(
+  'Please refund the duplicate charge today.',
+  questions: {
+    urgent: { type: :probability, instructions: 'Does this need attention today?' },
+    department: {
+      type: :choice,
+      instructions: 'Which team should handle this?',
+      options: { billing: 'Payments and refunds', technical: 'Bugs and integrations' }
+    }
+  },
+  provider_options: { strategy: :single_request, max_output_tokens: 1024 }
+)
+
+result.urgent.probability
+result.department.probabilities
+```
+
+The model returns one JSON object containing a distribution for each question. The gem checks that every question and option is present, validates each probability, normalizes each distribution, and builds RubyLLM's typed answers. `result.raw[:reported_probabilities]` and `result.raw[:reported_totals]` preserve the model's original numbers for inspection. With an OpenAI key configured, this example calls Luna directly through Chat Completions with reasoning disabled, temperature zero, and `store: false`. The 1024-token limit suits small judgments; omit it for large question sets to use the 8192-token default. It makes one corrective retry for malformed JSON or missing fields, counting both calls in `result.tokens` and `result.raw[:attempts]`. Set `malformed_retries: 0` to disable that retry. An invalid response after retries raises an error.
+
+If you use OpenRouter for Luna, prioritize the provider with the lowest observed latency:
+
+```ruby
+result = RubyLLM::LLMJudge.judge(
+  'Please refund the duplicate charge today.',
+  model: 'openai/gpt-6-luna',
+  questions: {
+    department: {
+      type: :choice,
+      instructions: 'Which team should handle this?',
+      options: { billing: 'Payments and refunds', technical: 'Bugs and integrations' }
+    }
+  },
+  provider_options: {
+    strategy: :single_request,
+    scoring_provider: :openrouter,
+    temperature: 0,
+    max_output_tokens: 1024,
+    chat_provider_options: {
+      reasoning: { enabled: false },
+      provider: { sort: 'latency' }
+    }
+  }
+)
+```
+
+OpenRouter's [latency sorting](https://openrouter.ai/docs/guides/routing/provider-selection) chooses among its available providers using their observed response times. It sped up OpenRouter in our test, while direct OpenAI was faster still. The measured speed and answer-quality tradeoffs are below.
+
+## How scoring works
+
+The default `:ratings` strategy asks the model to rate every declared answer from 0 to 9. It makes one call per answer, with up to six calls running at once, then applies softmax to the ratings. It retries a malformed digit once. When the highest ratings tie on a Choice question, it makes a one-call JSON judgment for that question and uses its distribution. If that call also ties, it raises an error instead of choosing whichever option came first. `result.raw[:tie_breaks]` records each resolution, and token usage includes the extra call. Set `tie_breaker: :first` to use the original first-option rule, or `tie_break_max_output_tokens:` to change the tie-break response limit (default 8192).
+
+The `:single_request` strategy asks for all distributions in one call. Both strategies build the same RubyLLM answer types: a `choice` returns the selected answer and distribution; a `score` returns the distribution and its weighted level; a `probability` returns the positive answer's share. The gem supports Judge's 1–255 choice options, 2–10 score levels, and multiple questions in one judgment.
+
+These probabilities compare the answers you supplied. The `:single_request` values are reported by the chat model; the default ratings values come from softmax over its 0–9 ratings. Neither strategy establishes calibration by itself. Include an `other` or `escalate` choice when the named answers may not cover the input. `confidence` measures how concentrated the returned distribution is. Use labeled examples to set any automation thresholds.
+
+Set optional `max_arms` and `max_input_bytes` in `provider_options` to cap work per judgment. For example, `{ max_arms: 12, max_input_bytes: 32_768 }` rejects larger requests before scoring. Token usage is aggregated across calls, and `raw` contains the ratings and model metadata.
+
+The one-digit-per-answer approach was inspired by [@burkov's post on Jev](https://x.com/burkov/status/2102438687392797045). The one-call strategy follows the typed-question pattern in [TypeSafe's System One adapter](https://github.com/typesafe-ai/system-one-adapter-python).
+
+## Benchmarks
+
+On September 23, 2026, we tested GPT-6 Luna (OpenRouter), DeepSeek V4.1 Flash (Fireworks), [Celeris-1](https://docs.celeris.ai/making-requests), and Jev 1.13.0 on 64 balanced [AG News](https://huggingface.co/datasets/fancyzhx/ag_news) articles with four choices and 32 balanced [SST-2](https://huggingface.co/datasets/stanfordnlp/sst2) sentences. LLM runs used temperature zero with reasoning disabled. **Correct** counts all cases; **Usable** counts cases that returned a valid judgment. Brier scores and latency use usable cases only. Latency is the full client round trip, including retries.
+
+| AG News model | Method | Correct / 64 | Usable / 64 | Brier ↓ | Median / p90 latency |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Luna | One call | 59 | 64 | 0.1444 | 1,320 / 1,708 ms |
+| Luna | Parallel ratings | 53 | 64 | 0.1578 | 1,718 / 2,753 ms |
+| DeepSeek | One call | 59 | 64 | 0.1453 | 1,001 / 2,197 ms |
+| DeepSeek | Parallel ratings | 56 | 64 | 0.1513 | 2,577 / 3,531 ms |
+| Celeris | Plain one call | 54 | 58 | 0.1284 | 428 / 744 ms |
+| Celeris | Original ratings | 42 | 54 | 0.1998 | 505 / 861 ms |
+| Celeris | Original ratings, rerun | 47 | 58 | 0.1754 | 537 / 889 ms |
+| Celeris | Ratings + retry/tie-break, run 1 | 57 | 61 | 0.1284 | 545 / 1,137 ms |
+| Celeris | Ratings + retry/tie-break, run 2 | 56 | 59 | 0.1153 | 545 / 1,009 ms |
+| Celeris | JSON one call + retry, run 1 | 60 | 63 | 0.0958 | 453 / 852 ms |
+| Celeris | JSON one call + retry, run 2 | 59 | 62 | 0.1107 | 401 / 776 ms |
+| Jev | Luna paired run | 59 | 64 | 0.1032 | 365 / 586 ms |
+| Jev | DeepSeek paired run | 59 | 64 | 0.1040 | 505 / 1,060 ms |
+
+| SST-2 model | Method | Choice / 32 | Probability / 32 | Score / 32 | Usable / 32 | Median latency |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| Luna | One call | 29 | 29 | 29 | 32 | 1,858 ms |
+| Luna | Parallel ratings | 28 | 30 | 28 | 32 | 3,437 ms |
+| DeepSeek | One call | 29 | 29 | 29 | 32 | 1,443 ms |
+| DeepSeek | Parallel ratings | 30 | 26 | 28 | 32 | 3,009 ms |
+| Celeris | Plain one call | 29 | 29 | 29 | 32 | 282 ms |
+| Celeris | Original ratings | 28 | 24 | 26 | 30 | 530 ms |
+| Celeris | Ratings + retry/tie-break, run 1 | 28 | 24 | 28 | 32 | 577 ms |
+| Celeris | JSON one call + retry, run 1 | 28 | 28 | 28 | 32 | 248 ms |
+| Celeris | JSON one call + retry, run 2 | 28 | 28 | 28 | 32 | 264 ms |
+| Jev | Luna paired run | 30 | 30 | 30 | 32 | 522 ms |
+| Jev | DeepSeek paired run | 30 | 29 | 30 | 32 | 308 ms |
+
+The original 42/64 Celeris ratings result was not a one-off: the same behavior scored 47/64 in a fresh control run. In the original run, 10 cases returned no usable judgment and nine of the 12 incorrect usable choices had tied top ratings. With malformed-digit retry and Choice tie resolution, two runs scored 57/64 and 56/64. Those runs resolved nine of ten and seven of seven top ties to the reference label. Their observed median and p90 latencies were higher; tie-break calls add work. The control and revised run 2 covered only the 64 news cases; revised run 1 covered all 112 requests. Jev rows are saved earlier paired runs on the same cases, while Celeris was measured later.
+
+### Luna latency paths
+
+In a paired API-level run using the gem's one-call prompt, direct OpenAI was faster than latency-sorted OpenRouter. Both used temperature zero, reasoning disabled, a 1024-token output limit, and one corrective retry for malformed JSON. Four direct cases were also verified through the gem. All final judgments were usable.
+
+| Dataset | Luna path | Correct | Median | p90 | Brier ↓ |
+| --- | --- | ---: | ---: | ---: | ---: |
+| AG News (64) | Direct OpenAI | 58/64 | 823 ms | 1,241 ms | 0.1405 |
+| AG News (64) | OpenRouter latency sort | 59/64 | 1,379 ms | 1,938 ms | 0.1265 |
+| SST-2 (32) | Direct OpenAI | 28/32 | 843 ms | 1,745 ms | 0.1042 |
+| SST-2 (32) | OpenRouter latency sort | 29/32 | 1,423 ms | 3,829 ms | 0.0762 |
+
+In that paired run, direct OpenAI cut median latency by about 40% on both datasets, but lost one correct label on each and had worse Brier scores. Its SST-2 calls needed five corrective retries versus one through OpenRouter. A later direct-only repeat scored **59/64 news at 868 ms median** and **28/32 SST-2 at 1,022 ms median**, all usable. Just one news label and two sentiment labels changed across direct runs. OpenRouter was not rerun simultaneously with the repeat. [Methods and both runs' case-level results](benchmarks/luna-direct.md).
+
+A separate paired gem run sent the same one-call Luna judgments through OpenRouter's default route and `provider: { sort: 'latency' }`, rotating request order across cases. All responses were usable. Each AG News request asked one four-choice question; each SST-2 request asked Choice, Probability, and Score together. Both routes used temperature zero, reasoning disabled, and a 1024-token output limit. These results are separate from the runs above.
+
+| Dataset | OpenRouter route | Correct | Median | p90 | Brier ↓ |
+| --- | --- | ---: | ---: | ---: | ---: |
+| AG News (64) | Default | 61/64 | 1,569 ms | 2,805 ms | 0.0857 |
+| AG News (64) | Lowest latency | 60/64 | 1,260 ms | 1,812 ms | 0.1165 |
+| SST-2 (32) | Default | 29/32 | 2,093 ms | 8,693 ms | 0.0775 |
+| SST-2 (32) | Lowest latency | 30/32 | 1,507 ms | 1,944 ms | 0.0695 |
+
+Latency sorting reduced median time by 20% on AG News and 28% on SST-2. It was faster on 50/64 and 21/32 matched cases, respectively. AG News lost one correct label and its Brier score worsened; SST-2 gained one correct label. Check this route on your own labeled decisions before relying on its probabilities. [Methods and case-level data](benchmarks/luna-routing.md).
+
+## Development
+
+The gem uses RubyLLM's Judge API when it is present and the `Legacy` path otherwise. `test/llm_judge_test.rb` covers the Judge API and `test/legacy_test.rb` covers RubyLLM 1.13 through 1.16; each skips on the other side.
+
+```bash
+bundle exec rake test                              # the latest released RubyLLM
+RUBY_LLM_VERSION=1.13.2 bundle exec rake test      # a specific release
+RUBY_LLM_VERSION=main bundle exec rake test        # RubyLLM's main branch, with the Judge API
+RUBY_LLM_PATH=../ruby_llm bundle exec rake test    # a local checkout
+```
+
+CI runs Ruby 3.2 through 4.0 against RubyLLM 1.13.2 and 1.16.0, and Ruby 3.4 against RubyLLM main.

data/lib/ruby_llm/llm_judge/engine.rb

@@ -0,0 +1,346 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module RubyLLM
+  module LLMJudge
+    class Engine
+      MAX_WORKERS = 6
+      SYSTEM_INSTRUCTIONS = 'Return exactly one ASCII digit 0-9. 9 means very likely to be the correct answer; 0 means very unlikely. No explanation.'
+      SINGLE_REQUEST_INSTRUCTIONS = 'Answer all questions using only a JSON object with an "answers" field. ' \
+                                    'For each question, return an object mapping every supplied option ID to a ' \
+                                    'probability between 0 and 1. Include every option exactly once and make ' \
+                                    'each question\'s probabilities sum to 1. Evaluate each question independently ' \
+                                    'against the same state. Return no explanations or markdown.'
+
+      def initialize(config:, model: DEFAULT_MODEL, provider_options: {}, scorer: nil, responder: nil)
+        @config = config
+        options = provider_options.transform_keys(&:to_sym)
+        allowed = %i[scoring_provider scoring_protocol temperature max_output_tokens chat_provider_options
+                     max_arms max_input_bytes strategy malformed_retries tie_breaker tie_break_max_output_tokens]
+        raise ArgumentError, 'Unknown LLMJudge provider options' unless (options.keys - allowed).empty?
+
+        @strategy = options.fetch(:strategy, :ratings).to_sym
+        raise ArgumentError, 'strategy must be :ratings or :single_request' unless %i[ratings single_request].include?(@strategy)
+        @tie_breaker = options.fetch(:tie_breaker, :single_request).to_sym
+        raise ArgumentError, 'tie_breaker must be :single_request or :first' unless %i[single_request first].include?(@tie_breaker)
+
+        @scoring_provider = options.fetch(:scoring_provider, :openai).to_sym
+        @scoring_model = model
+        raise ArgumentError, 'A model is required' unless @scoring_model.is_a?(String) && !@scoring_model.empty?
+
+        luna_defaults = @scoring_provider == :openai && @scoring_model == DEFAULT_MODEL
+        @scoring_protocol = options.fetch(:scoring_protocol, luna_defaults ? :chat_completions : nil)
+        # Before 2.1, RubyLLM chats use each provider's one chat API (Chat Completions for OpenAI).
+        if !NATIVE && ![nil, :chat_completions].include?(@scoring_protocol&.to_sym)
+          raise ArgumentError, 'scoring_protocol requires RubyLLM 2.1'
+        end
+        @temperature = options.fetch(:temperature, luna_defaults ? 0 : nil)
+        @max_output_tokens = options.fetch(:max_output_tokens, @strategy == :single_request ? 8192 : 4)
+        @tie_break_max_output_tokens = options.fetch(:tie_break_max_output_tokens, 8192)
+        @chat_provider_options = options.fetch(:chat_provider_options, default_chat_options(luna_defaults))
+        @max_arms = options[:max_arms]
+        @max_input_bytes = options[:max_input_bytes]
+        @malformed_retries = options.fetch(:malformed_retries, 1)
+        raise ArgumentError, 'chat_provider_options must be a Hash' unless @chat_provider_options.is_a?(Hash)
+        raise ArgumentError, 'max_output_tokens must be positive' unless @max_output_tokens.is_a?(Integer) && @max_output_tokens.positive?
+        unless @tie_break_max_output_tokens.is_a?(Integer) && @tie_break_max_output_tokens.positive?
+          raise ArgumentError, 'tie_break_max_output_tokens must be positive'
+        end
+        unless [@max_arms, @max_input_bytes].all? { |limit| limit.nil? || (limit.is_a?(Integer) && limit.positive?) }
+          raise ArgumentError, 'LLMJudge limits must be positive integers'
+        end
+        unless @malformed_retries.is_a?(Integer) && @malformed_retries >= 0
+          raise ArgumentError, 'malformed_retries must be a nonnegative integer'
+        end
+
+        @scorer = scorer || method(:score_with_model)
+        @responder = responder || method(:respond_with_model)
+      end
+
+      def judge(input, questions:, model:)
+        return judge_single_request(input, questions, model) if @strategy == :single_request
+
+        jobs = build_jobs(input, questions)
+        results = run(jobs) { |job| @scorer.call(job[:prompt]) }
+        tie_breaks = []
+        answers = questions.values.to_h do |question|
+          ratings = jobs.each_index.filter_map { |index|
+            results[index].fetch(:digit) if jobs[index][:question] == question
+          }
+          chosen = answer(question, softmax(ratings))
+          if question.type == :choice && ratings.count(ratings.max) > 1 && @tie_breaker == :single_request
+            judgment = judge_single_request(input, { question.name => question }, model)
+            chosen = judgment.answers.fetch(question.name)
+            probabilities = chosen.probabilities.values
+            if probabilities.count(probabilities.max) > 1
+              raise RubyLLM::Error, "Scoring model could not resolve the tie for #{question.name}"
+            end
+            tie_breaks << { question: question.name, ratings:, judgment: }
+          end
+          [question.name, chosen]
+        end
+        tokens = aggregate_tokens(results.map { |result| result.fetch(:tokens) } +
+                                  tie_breaks.map { |item| item[:judgment].tokens })
+        Types::Judgment.new(
+          answers:, model: model.id, tokens:,
+          raw: { method: 'parallel_0_to_9_softmax', scoring_provider: @scoring_provider,
+                 scoring_model: @scoring_model,
+                 tie_breaks: tie_breaks.map { |item|
+                   { question: item[:question], ratings: item[:ratings],
+                     probabilities: item[:judgment].answers.fetch(item[:question]).probabilities,
+                     attempts: item[:judgment].raw[:attempts] }
+                 },
+                 ratings: jobs.each_with_index.map { |job, index|
+                   { question: job[:question].name, option: job[:name], digit: results[index][:digit] }
+                 } }
+        )
+      end
+
+      private
+
+      def judge_single_request(input, questions, model)
+        specs = validated_options(input, questions)
+        original_prompt = single_request_prompt(input, specs)
+        prompt = original_prompt
+        attempts = []
+        loop do
+          result = @responder.call(prompt)
+          attempts << result
+          begin
+            answers, reported, normalization = parse_single_request(result.fetch(:content), specs)
+            return Types::Judgment.new(
+              answers:, model: model.id,
+              tokens: aggregate_tokens(attempts.map { |attempt| attempt.fetch(:tokens) }),
+              raw: { method: 'single_request_probabilities', scoring_provider: @scoring_provider,
+                     scoring_model: @scoring_model, reported_probabilities: reported,
+                     reported_totals: normalization, attempts: attempts.size }
+            )
+          rescue RubyLLM::Error => error
+            raise if attempts.size > @malformed_retries
+
+            prompt = "#{original_prompt}\n\nThe previous answer was invalid: #{error.message}. " \
+                     "Previous answer: #{result.fetch(:content).to_s[0, 4000]}\n" \
+                     'Return a complete corrected JSON object with every question and option.'
+          end
+        end
+      end
+
+      def parse_single_request(content, specs)
+        parsed = JSON.parse(content)
+        raise RubyLLM::Error, 'Scoring model returned a non-object response' unless parsed.is_a?(Hash)
+
+        reported = parsed.fetch('answers')
+        expected_questions = specs.map { |question, _| question.name.to_s }
+        unless reported.is_a?(Hash) && reported.keys.sort == expected_questions.sort
+          actual = reported.is_a?(Hash) ? reported.keys : reported.class.name
+          raise RubyLLM::Error, "Question IDs must be #{expected_questions.inspect}; got #{actual.inspect}"
+        end
+
+        normalization = {}
+        answers = specs.to_h do |question, options|
+          distribution = reported.fetch(question.name.to_s)
+          keys = options.map { |name, _| name.to_s }
+          unless distribution.is_a?(Hash) && distribution.keys.sort == keys.sort
+            actual = distribution.is_a?(Hash) ? distribution.keys : distribution.class.name
+            raise RubyLLM::Error, "Option IDs for #{question.name} must be #{keys.inspect}; got #{actual.inspect}"
+          end
+
+          values = keys.map { |key| distribution.fetch(key) }
+          unless values.all? { |value| value.is_a?(Numeric) && value.finite? && (0..1).cover?(value) }
+            raise RubyLLM::Error, "Scoring model returned invalid probabilities for #{question.name}"
+          end
+
+          total = values.sum.to_f
+          raise RubyLLM::Error, "Scoring model returned zero probability for #{question.name}" unless total.positive?
+
+          normalization[question.name.to_s] = total
+          [question.name, answer(question, values.map { |value| value / total })]
+        end
+        [answers, reported, normalization]
+      rescue JSON::ParserError, KeyError, TypeError => error
+        raise RubyLLM::Error, "Scoring model returned invalid JSON probabilities: #{error.message}"
+      end
+
+      def single_request_prompt(input, specs)
+        request = {
+          state: input,
+          questions: specs.map do |question, options|
+            { id: question.name, type: question.type, instructions: question.instructions,
+              options: options.map { |name, description| { id: name.to_s, description: } } }
+          end
+        }
+        "Evaluate this decision request. Return only the requested JSON object.\n#{JSON.generate(request)}"
+      end
+
+      def build_jobs(input, questions)
+        jobs = validated_options(input, questions).flat_map do |question, options|
+          options.map do |name, description|
+            { question:, name:, description:, prompt: prompt(input, question.instructions, name, description) }
+          end
+        end
+        jobs
+      end
+
+      def validated_options(input, questions)
+        if @max_input_bytes
+          input_bytes = serialize(input).bytesize + questions.values.sum do |question|
+            serialize(question.instructions).bytesize + serialize(question.criteria).bytesize
+          end
+          raise ArgumentError, "LLMJudge input exceeds #{@max_input_bytes} bytes" if input_bytes > @max_input_bytes
+        end
+
+        specs = questions.values.map { |question| [question, options_for(question)] }
+        arms = specs.sum { |_, options| options.size }
+        raise ArgumentError, "LLMJudge exceeds #{@max_arms} scoring arms" if @max_arms && arms > @max_arms
+
+        specs
+      end
+
+      def options_for(question)
+        case question.type
+        when :probability
+          criteria = question.criteria || {}
+          yes_key = criteria.keys.find { |key| %w[yes true].include?(key.to_s) }
+          no_key = criteria.keys.find { |key| %w[no false].include?(key.to_s) }
+          yes = yes_key ? criteria[yes_key] : 'Yes, the condition holds'
+          no = no_key ? criteria[no_key] : 'No, the condition does not hold'
+          [['true', yes], ['false', no]]
+        when :choice
+          raise ArgumentError, 'A choice needs 1–255 options' unless (1..255).cover?(question.criteria.size)
+
+          question.criteria.to_a
+        when :score
+          raise ArgumentError, 'A score needs 2–10 levels' unless (2..10).cover?(question.criteria.size)
+
+          question.criteria.each_with_index.map { |description, index| [index, description] }
+        else
+          raise ArgumentError, "Unsupported question type: #{question.type}"
+        end
+      end
+
+      def prompt(input, instructions, name, description)
+        "Question: #{serialize(instructions || 'Which answer best applies?')}\n" \
+          "State: #{serialize(input)}\n\n" \
+          'Output a single number between 0 and 9 to tell how likely it is that the correct answer is: ' \
+          "#{name} — #{serialize(description)}"
+      end
+
+      def serialize(value)
+        value.is_a?(String) ? value : JSON.generate(value)
+      end
+
+      def run(jobs)
+        queue = Queue.new
+        jobs.each_with_index { |job, index| queue << [index, job] }
+        results = Array.new(jobs.size)
+        workers = Array.new([jobs.size, MAX_WORKERS].min) do
+          Thread.new do
+            Thread.current.report_on_exception = false
+            loop do
+              begin
+                index, job = queue.pop(true)
+              rescue ThreadError
+                break
+              end
+              results[index] = yield job
+            end
+          end
+        end
+        workers.each(&:join)
+        workers.each(&:value)
+        results
+      end
+
+      def default_chat_options(luna_defaults)
+        return {} unless @scoring_provider == :openai
+
+        luna_defaults ? { store: false, reasoning_effort: 'none' } : { store: false }
+      end
+
+      def score_with_model(prompt)
+        attempts = []
+        loop do
+          request = attempts.empty? ? prompt : "#{prompt}\n\nYour previous answer was invalid. Return exactly one ASCII digit 0-9."
+          response = scoring_chat.ask(request)
+          attempts << response
+          digit = response.content.to_s.strip
+          if /\A[0-9]\z/.match?(digit)
+            return { digit: digit.to_i, tokens: aggregate_tokens(attempts.map(&:tokens)) }
+          end
+          raise RubyLLM::Error, 'Scoring model returned no single digit' if attempts.size > @malformed_retries
+        end
+      end
+
+      def respond_with_model(prompt)
+        limit = @strategy == :ratings ? @tie_break_max_output_tokens : @max_output_tokens
+        response = scoring_chat(instructions: SINGLE_REQUEST_INSTRUCTIONS, max_output_tokens: limit).ask(prompt)
+        { content: response.content.to_s, tokens: response.tokens }
+      end
+
+      def scoring_chat(instructions: SYSTEM_INSTRUCTIONS, max_output_tokens: @max_output_tokens)
+        context = RubyLLM::Context.new(@config)
+        chat = if NATIVE
+                 context.chat(model: @scoring_model, provider: @scoring_provider,
+                              protocol: @scoring_protocol, assume_model_exists: true)
+               else
+                 context.chat(model: @scoring_model, provider: @scoring_provider, assume_model_exists: true)
+               end
+        chat.with_instructions(instructions)
+        chat.with_temperature(@temperature) unless @temperature.nil?
+        if NATIVE
+          chat.with_max_output_tokens(max_output_tokens)
+          chat.with_provider_options(@chat_provider_options)
+        else
+          chat.with_params(**RubyLLM::Utils.deep_merge(max_output_tokens_param(max_output_tokens), @chat_provider_options))
+        end
+        chat
+      end
+
+      # The request field each provider reads for the output limit, as RubyLLM 2.1 renders it.
+      def max_output_tokens_param(limit)
+        case @scoring_provider
+        when :openai, :azure then { max_completion_tokens: limit }
+        when :gemini, :vertexai then { generationConfig: { maxOutputTokens: limit } }
+        when :bedrock then { inferenceConfig: { maxTokens: limit } }
+        else { max_tokens: limit }
+        end
+      end
+
+      def aggregate_tokens(tokens)
+        NATIVE ? RubyLLM::Tokens.aggregate(tokens) : Legacy.aggregate_tokens(tokens)
+      end
+
+      def answer(question, probabilities)
+        case question.type
+        when :probability
+          Types::Probability.new(probability: probabilities.first)
+        when :choice
+          names = question.criteria.keys
+          distribution = names.zip(probabilities).to_h
+          Types::Choice.new(choice: names[probabilities.index(probabilities.max)], probabilities: distribution,
+                              confidence: concentration(probabilities))
+        when :score
+          Types::Score.new(score: probabilities.each_with_index.sum { |probability, index| probability * index },
+                             levels: question.criteria,
+                             probabilities: probabilities.each_with_index.to_h { |probability, index| [index, probability] },
+                             confidence: concentration(probabilities))
+        end
+      end
+
+      def softmax(ratings)
+        peak = ratings.max
+        weights = ratings.map { |rating| Math.exp(rating - peak) }
+        weights.map { |weight| weight / weights.sum }
+      end
+
+      def concentration(probabilities)
+        return 0.0 if probabilities.one?
+        return 0.0 if probabilities.count(probabilities.max) > 1
+
+        entropy = -probabilities.sum { |probability| probability.zero? ? 0.0 : probability * Math.log(probability) }
+        [[1.0 - entropy / Math.log(probabilities.size), 0.0].max, 1.0].min
+      end
+    end
+  end
+end

data/lib/ruby_llm/llm_judge/legacy.rb

@@ -0,0 +1,150 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module LLMJudge
+    # Stand-ins for RubyLLM 2.1's Judge types, used when RubyLLM predates the
+    # Judge API. They expose the same readers, so code written against
+    # RubyLLM::LLMJudge.judge keeps working after upgrading.
+    module Legacy
+      Model = Data.define(:id)
+
+      class Question
+        CRITERIA_KEYS = { probability: :criteria, choice: :options, score: :levels }.freeze
+
+        attr_reader :name, :type, :instructions, :criteria
+
+        def self.from_h(name, definition)
+          raise ArgumentError, 'Each question must be a Hash' unless definition.is_a?(Hash)
+
+          definition = definition.transform_keys(&:to_sym)
+          type = definition[:type]&.to_sym
+          key = CRITERIA_KEYS.fetch(type) { raise ArgumentError, "Unknown judgment type: #{type.inspect}" }
+          extra = definition.keys - [:type, :instructions, key]
+          raise ArgumentError, "Unknown question options: #{extra.join(', ')}" unless extra.empty?
+
+          new(name, type:, instructions: definition[:instructions], criteria: definition[key])
+        end
+
+        def initialize(name, type:, instructions:, criteria:)
+          raise ArgumentError, 'A question name must be a String or Symbol' unless name.is_a?(String) || name.is_a?(Symbol)
+
+          @name = name
+          @type = type
+          @instructions = instructions
+          @criteria = criteria
+          validate!
+          freeze
+        end
+
+        private
+
+        def validate!
+          case type
+          when :probability
+            return if criteria.nil?
+            unless criteria.is_a?(Hash) && (criteria.keys.map(&:to_s) - %w[yes no true false]).empty?
+              raise ArgumentError, 'Probability criteria must describe yes and no'
+            end
+          when :choice
+            raise ArgumentError, 'A choice needs a nonempty Hash of options' unless criteria.is_a?(Hash) && !criteria.empty?
+          when :score
+            unless criteria.is_a?(Array) && criteria.size >= 2 && criteria.none?(&:nil?)
+              raise ArgumentError, 'A score needs at least two non-nil levels'
+            end
+          end
+        end
+      end
+
+      class Probability
+        attr_reader :probability
+
+        def initialize(probability:)
+          @probability = probability
+          freeze
+        end
+
+        def type = :probability
+        def to_h = { type:, probability: }
+      end
+
+      class Choice
+        attr_reader :choice, :probabilities, :confidence
+
+        def initialize(choice:, probabilities:, confidence:)
+          @choice = choice
+          @probabilities = probabilities.dup.freeze
+          @confidence = confidence
+          freeze
+        end
+
+        def type = :choice
+        def to_h = { type:, choice:, probabilities:, confidence: }
+      end
+
+      class Score
+        attr_reader :score, :levels, :probabilities, :confidence
+
+        def initialize(score:, levels:, probabilities:, confidence:)
+          @score = score
+          @levels = levels
+          @probabilities = probabilities.dup.freeze
+          @confidence = confidence
+          freeze
+        end
+
+        def type = :score
+        def to_h = { type:, score:, levels:, probabilities:, confidence: }
+      end
+
+      class Judgment
+        include Enumerable
+
+        attr_reader :answers, :model, :tokens, :raw
+
+        def initialize(answers:, model:, tokens:, raw: nil)
+          @answers = answers.dup.freeze
+          @answer_keys = answers.keys.to_h { |key| [key.to_s, key] }.freeze
+          @model = model
+          @tokens = tokens
+          @raw = raw
+        end
+
+        def [](name)
+          answers[@answer_keys[name.to_s]]
+        end
+
+        def fetch(name)
+          answers.fetch(@answer_keys.fetch(name.to_s))
+        end
+
+        def each(&)
+          answers.each(&)
+        end
+
+        def to_h
+          { model:, answers: answers.transform_values(&:to_h), tokens: tokens.to_h }
+        end
+
+        private
+
+        def method_missing(name, *args, &block)
+          return super unless args.empty? && !block && @answer_keys.key?(name.to_s)
+
+          self[name]
+        end
+
+        def respond_to_missing?(name, include_private = false)
+          @answer_keys.key?(name.to_s) || super
+        end
+      end
+
+      module_function
+
+      def aggregate_tokens(tokens)
+        tokens = tokens.compact
+        sum = ->(reader) { tokens.sum { |token| token.public_send(reader) || 0 } }
+        RubyLLM::Tokens.new(input: sum.(:input), output: sum.(:output))
+      end
+    end
+  end
+end

data/lib/ruby_llm/llm_judge/provider.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module LLMJudge
+    class Provider < RubyLLM::Provider
+      def api_base
+        'http://127.0.0.1'
+      end
+
+      def judge(input, questions:, model:, provider_options: {})
+        Engine.new(config:, model: model.id, provider_options:).judge(input, questions:, model:)
+      end
+
+      def self.assume_models_exist?
+        true
+      end
+    end
+  end
+end

data/lib/ruby_llm/llm_judge/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module LLMJudge
+    VERSION = '0.1.0'
+  end
+end

data/lib/ruby_llm/llm_judge.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require 'ruby_llm'
+require_relative 'llm_judge/version'
+
+module RubyLLM
+  module LLMJudge
+    DEFAULT_MODEL = 'gpt-6-luna'
+
+    # RubyLLM 2.1 added the Judge API. Earlier versions run the engine directly
+    # and return Legacy stand-ins with the same readers.
+    NATIVE = defined?(RubyLLM::Judge) ? true : false
+  end
+end
+
+require_relative 'llm_judge/legacy' unless RubyLLM::LLMJudge::NATIVE
+require_relative 'llm_judge/engine'
+require_relative 'llm_judge/provider' if RubyLLM::LLMJudge::NATIVE
+
+module RubyLLM
+  module LLMJudge
+    # Answer and judgment classes: RubyLLM's own on 2.1+, Legacy otherwise.
+    Types = NATIVE ? RubyLLM : Legacy
+
+    def self.judge(input, questions:, model: DEFAULT_MODEL, **options)
+      if NATIVE
+        return RubyLLM.judge(input, questions:, model:, provider: :llm_judge,
+                                    assume_model_exists: true, **options)
+      end
+
+      provider_options = options.delete(:provider_options) || {}
+      context = options.delete(:context)
+      raise ArgumentError, "Unsupported before RubyLLM 2.1: #{options.keys.join(', ')}" unless options.empty?
+
+      built = questions.to_h { |name, definition| [name.to_s, Legacy::Question.from_h(name, definition)] }
+      Engine.new(config: context&.config || RubyLLM.config, model:, provider_options:)
+            .judge(input, questions: built, model: Legacy::Model.new(id: model))
+    end
+  end
+end
+
+RubyLLM::Provider.register(:llm_judge, RubyLLM::LLMJudge::Provider) if RubyLLM::LLMJudge::NATIVE

metadata

@@ -0,0 +1,72 @@
+--- !ruby/object:Gem::Specification
+name: ruby_llm-llm_judge
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- JP Camara
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: ruby_llm
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.13.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.13.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+description: Answers probability, choice, and score questions with any RubyLLM chat
+  model, using parallel per-answer ratings or one-call typed distributions. Plugs
+  into RubyLLM 2.1's Judge API and also runs on RubyLLM 1.13 through 1.16.
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/ruby_llm/llm_judge.rb
+- lib/ruby_llm/llm_judge/engine.rb
+- lib/ruby_llm/llm_judge/legacy.rb
+- lib/ruby_llm/llm_judge/provider.rb
+- lib/ruby_llm/llm_judge/version.rb
+homepage: https://github.com/jpcamara/ruby_llm-llm_judge
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/jpcamara/ruby_llm-llm_judge
+  changelog_uri: https://github.com/jpcamara/ruby_llm-llm_judge/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/jpcamara/ruby_llm-llm_judge/issues
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: Use RubyLLM chat models to answer RubyLLM Judge questions
+test_files: []