rubric_llm 0.6.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a1129b7b842324735b232caf6a116451cb04b599187aac713ce20c5befda4295
-  data.tar.gz: 33fc34b74d756ded10b00227cc9c46de94aa3da8a17dd798dc49e94edee5ba15
+  metadata.gz: fe5f964e023687c815cf95b56673d05f3103f15a679ebb6f5d272d287c396163
+  data.tar.gz: f082d306d7e50347dc6a53caf453dc7e24abbd0213012711506692516ff38a47
 SHA512:
-  metadata.gz: 19c150856bbb65ab6d1c85e0a80ab92c309fc4524f20ab2057323715f6dd506d24408e37263e37b13eb0d3b6410f7c56a47a2511cbb25d5d0a1d53872117e4ab
-  data.tar.gz: e3c83d54f10befc47c0a403f9784f33a634debae737201cf39b758784bc3a0c1f6760a08aab677d00c101eab6a5b67ee8d05626a39360d1d68fea5ab1304c767
+  metadata.gz: 8cdba0f5196529eed16d2c19edaca418f8feedca62244724a206e6b4baa1ab9ef13660a3cf24f1711ad52e43856f325648b88ee8443da0cce7bee9a9f9b637d7
+  data.tar.gz: c2828756fca3eb3395567bd61ebac9b80bdffd12dbdd4b410f000b5001b68169be14101a95bde4c0541f98526c9bdc7ff665e22e2d918fda75072af4dd1e4dc2

data/CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.8.0] - 2026-09-22
+
+### Changed
+
+- Calculate faithfulness, context precision, and context recall scores from the judge's item-level decisions instead of its suggested score
+- Define scoring criteria for correctness, relevance, and factual accuracy, with factual accuracy measuring contradictions against the reference rather than missing facts
+- Reject missing or malformed metric details, including missing reasoning, empty claim or fact lists, invalid context indices, and factual accuracy scores that conflict with their discrepancy lists. Context indices must be 1-based integers. Evaluation returns `nil` with an error for these responses; direct metric calls raise `JudgeError`
+
+## [0.7.0] - 2026-09-22
+
+### Added
+
+- Optional `thinking_effort` configuration and `RUBRIC_THINKING_EFFORT` environment variable for judge reasoning models, passed to RubyLLM's `with_thinking(effort: ...)`. The default leaves thinking controls unchanged; explicit `nil` ignores the environment variable
+
 ## [0.6.0] - 2026-09-18
 
 ### Changed

data/README.md

@@ -8,14 +8,14 @@ Lightweight LLM evaluation framework for Ruby, inspired by [DeepEval](https://gi
 
 Provider-agnostic evaluation with pluggable metrics, statistical A/B comparison, and test framework integration: no Rails, no ActiveRecord, no UI. Works anywhere Ruby runs.
 
-RubricLLM `0.6.0` supports RubyLLM `~> 2.0` and Ruby 3.4 or later. Applications that use RubyLLM 1.x should stay on RubricLLM `0.5.x`.
+RubricLLM `0.7.0` supports RubyLLM `~> 2.0` and Ruby 3.4 or later. Applications that use RubyLLM 1.x should stay on RubricLLM `0.5.x`.
 
 ## Installation
 
 Update both gem constraints together:
 
 ```ruby
-gem "rubric_llm", "~> 0.6.0"
+gem "rubric_llm", "~> 0.7.0"
 gem "ruby_llm", "~> 2.0"
 ```
 
@@ -30,7 +30,7 @@ If the application still has a RubyLLM 1.x constraint, update both constraints b
 Or install directly:
 
 ```bash
-gem install rubric_llm --version 0.6.0
+gem install rubric_llm --version 0.7.0
 ```
 
 ## Quick Start
@@ -92,6 +92,28 @@ end
 
 Structured output support depends on the selected provider and model. RubricLLM sends its schema when RubyLLM reports structured output support. Otherwise it requests JSON text and validates the response object and score locally. Check the target provider's support before relying on a schema or a specific protocol. See RubyLLM's [2.0 upgrade guide](https://rubyllm.com/next/upgrading/), [request control guide](https://rubyllm.com/next/chat-request-control/), and [structured output support](https://rubyllm.com/next/structured-output/).
 
+### Judge Thinking Effort
+
+Set `thinking_effort` when using a reasoning model to control the trade-off between judge quality, latency, and cost:
+
+```ruby
+RubricLLM.configure do |c|
+  c.judge_model = "gpt-6-astra"
+  c.judge_provider = :openai
+  c.temperature = nil
+  c.thinking_effort = :high
+end
+
+# Or configure an individual evaluation:
+config = RubricLLM::Config.new(judge_model: "gpt-6-astra", temperature: nil, thinking_effort: "high")
+```
+
+RubricLLM passes the value to RubyLLM's [`with_thinking(effort: ...)`](https://rubyllm.com/thinking/). Strings and symbols are accepted. Supported values depend on the model and provider; RubricLLM does not translate values or silently ignore unsupported settings. Request errors are reported as judge errors.
+
+The default is `nil`: RubricLLM does not call `with_thinking`, so existing RubyLLM and provider behavior is unchanged. This does **not** disable reasoning. An omitted setting reads `RUBRIC_THINKING_EFFORT`; an explicit `thinking_effort: nil` ignores that environment variable. Use `:none` only when the selected model supports it.
+
+Higher effort does not guarantee better scores. Compare results on your evaluation dataset. For providers that count reasoning tokens toward the output limit, increase `max_tokens` if the default 4,096 tokens leaves too little room for the JSON response.
+
 ### Environment Variables
 
 All config fields can be set via environment variables:
@@ -101,6 +123,7 @@ All config fields can be set via environment variables:
 | `RUBRIC_JUDGE_MODEL` | `gpt-4o` | Judge LLM model name |
 | `RUBRIC_JUDGE_PROVIDER` | `openai` | RubyLLM provider |
 | `RUBRIC_TEMPERATURE` | `0.0` | Judge temperature |
+| `RUBRIC_THINKING_EFFORT` | unset | Model-specific reasoning effort, such as `high` |
 | `RUBRIC_MAX_TOKENS` | `4096` | Max response tokens |
 | `RUBRIC_MAX_RETRIES` | `2` | Max retries on transient failures |
 | `RUBRIC_RETRY_BASE_DELAY` | `1.0` | Base delay (seconds) for exponential backoff |
@@ -142,14 +165,14 @@ RubyLLM classifies OpenAI's HTTP 429 `insufficient_quota` response as a rate-lim
 
 ### LLM-as-Judge Metrics
 
-These metrics use a judge LLM to evaluate quality. Each sends a structured prompt and parses a JSON response with a 0.0–1.0 score.
+These metrics use a judge LLM to evaluate quality. Each returns a 0.0–1.0 score. Faithfulness counts supported answer claims, context precision counts relevant non-empty chunks, and context recall counts covered reference facts. These three scores are calculated from the judge's item-level decisions, not its suggested score. An empty or malformed item list (including an answer with no factual claims) is an evaluation error, not a quality score. Correctness, relevance, and factual accuracy use judge scores with metric-specific scoring criteria. The judge still decides what counts as a claim, fact, or relevant chunk, so scores are not deterministic across models.
 
 | Metric | Question it answers | Requires |
 |--------|-------------------|----------|
 | **Correctness** | Does the answer match the known correct answer? | `ground_truth` |
 | **Relevance** | Does the answer address what was asked? | `question` |
 | **Context Precision** | Are the retrieved context chunks actually relevant? | `question`, `context` |
-| **Factual Accuracy** | Are there factual discrepancies between candidate and reference? | `ground_truth` |
+| **Factual Accuracy** | Does the candidate contradict the reference (not omit it)? | `ground_truth` |
 | **Context Recall** | Do the contexts cover the information in the ground truth? | `context`, `ground_truth` |
 | **Faithfulness** | Is every claim in the answer supported by the context? | `context` |
 
@@ -403,7 +426,7 @@ Deep dives live in the [project wiki](https://github.com/dpaluy/rubric_llm/wiki)
 ## Requirements
 
 - Ruby >= 3.4
-- [ruby_llm](https://github.com/crmne/ruby_llm) ~> 2.0 for RubricLLM 0.6.0
+- [ruby_llm](https://github.com/crmne/ruby_llm) ~> 2.0 for RubricLLM 0.7.0
 - An API key for your chosen LLM provider (set via RubyLLM configuration)
 
 ## Contributing

data/lib/rubric_llm/config.rb

@@ -3,14 +3,16 @@
 module RubricLLM
   class Config
     attr_accessor :judge_model, :judge_provider, :temperature, :max_tokens, :custom_prompt,
-                  :max_retries, :retry_base_delay, :concurrency
+                  :max_retries, :retry_base_delay, :concurrency, :thinking_effort
 
     def initialize(judge_model: nil, judge_provider: nil, # rubocop:disable Metrics/ParameterLists
                    temperature: Float(ENV.fetch("RUBRIC_TEMPERATURE", "0.0")), max_tokens: nil,
-                   custom_prompt: nil, max_retries: nil, retry_base_delay: nil, concurrency: nil, validate: false)
+                   custom_prompt: nil, max_retries: nil, retry_base_delay: nil, concurrency: nil, validate: false,
+                   thinking_effort: ENV.fetch("RUBRIC_THINKING_EFFORT", nil))
       @judge_model = judge_model || ENV.fetch("RUBRIC_JUDGE_MODEL", "gpt-4o")
       @judge_provider = (judge_provider || ENV.fetch("RUBRIC_JUDGE_PROVIDER", "openai")).to_sym
       @temperature = temperature
+      @thinking_effort = thinking_effort
       @max_tokens = max_tokens || Integer(ENV.fetch("RUBRIC_MAX_TOKENS", "4096"))
       @custom_prompt = custom_prompt
       @max_retries = max_retries || Integer(ENV.fetch("RUBRIC_MAX_RETRIES", "2"))
@@ -27,6 +29,7 @@ module RubricLLM
       validate_judge_model
       validate_judge_provider
       validate_temperature
+      validate_thinking_effort
       validate_max_tokens
       validate_max_retries
       validate_retry_base_delay
@@ -39,6 +42,7 @@ module RubricLLM
         judge_model:,
         judge_provider:,
         temperature:,
+        thinking_effort:,
         max_tokens:,
         custom_prompt:,
         max_retries:,
@@ -67,6 +71,13 @@ module RubricLLM
       raise ConfigurationError, "temperature must be nil or between 0.0 and 2.0"
     end
 
+    def validate_thinking_effort
+      return if thinking_effort.nil?
+      return if (thinking_effort.is_a?(String) || thinking_effort.is_a?(Symbol)) && !thinking_effort.to_s.strip.empty?
+
+      raise ConfigurationError, "thinking_effort must be nil or a non-empty string or symbol"
+    end
+
     def validate_max_tokens
       return if max_tokens.is_a?(Integer) && max_tokens.positive?
 

data/lib/rubric_llm/judge.rb

@@ -52,6 +52,7 @@ module RubricLLM
         chat = RubyLLM.chat(model: config.judge_model, provider: config.judge_provider)
         chat.with_temperature(config.temperature)
         chat.with_max_output_tokens(config.max_tokens)
+        chat.with_thinking(effort: config.thinking_effort) unless config.thinking_effort.nil?
         apply_response_schema(chat)
 
         full_system_prompt = build_system_prompt(system_prompt)

data/lib/rubric_llm/metrics/base.rb

@@ -35,6 +35,29 @@ module RubricLLM
       def judge_eval(system_prompt:, user_prompt:)
         judge.call(system_prompt:, user_prompt:)
       end
+
+      def checked_items(result, key, label:, value_key:, text_key:, count: nil)
+        items = result[key]
+        valid = items.is_a?(Array) && items.any? && (count.nil? || items.size == count)
+        valid &&= items.all? do |item|
+          item.is_a?(Hash) && item[text_key].is_a?(String) && !item[text_key].strip.empty? &&
+            [true, false].include?(item[value_key])
+        end
+        raise JudgeError, "Judge response has invalid #{label}" unless valid
+
+        items
+      end
+
+      def fraction(items, key)
+        items.count { |item| item[key] }.to_f / items.size
+      end
+
+      def reasoning_for(result)
+        reasoning = result["reasoning"]
+        raise JudgeError, "Judge response missing reasoning" unless reasoning.is_a?(String) && !reasoning.strip.empty?
+
+        reasoning
+      end
     end
   end
 end

data/lib/rubric_llm/metrics/context_precision.rb

@@ -5,7 +5,8 @@ module RubricLLM
     class ContextPrecision < Base
       SYSTEM_PROMPT = <<~PROMPT
         You are an evaluation judge. Assess whether the retrieved contexts are relevant to the question.
-        Context precision measures if the retrieved documents are useful for answering the question.
+        Context precision is the fraction of retrieved chunks useful for answering the question.
+        Classify every numbered chunk once, in order. Use the displayed 1-based index.
 
         Respond with JSON only:
         {
@@ -29,18 +30,20 @@ module RubricLLM
         PROMPT
 
         result = judge_eval(system_prompt: SYSTEM_PROMPT, user_prompt:)
-        normalize(result)
+        normalize(result, context_chunks.size)
       end
 
       private
 
-      def normalize(result)
+      def normalize(result, count)
+        context_scores = checked_items(result, "context_scores", label: "context scores", count:, value_key: "relevant", text_key: "reason")
+        unless context_scores.each_with_index.all? { |item, index| item["index"].is_a?(Integer) && item["index"] == index + 1 }
+          raise JudgeError, "Judge response has invalid context indices"
+        end
+
         {
-          score: Float(result["score"]),
-          details: {
-            context_scores: result["context_scores"],
-            reasoning: result["reasoning"]
-          }
+          score: fraction(context_scores, "relevant"),
+          details: { context_scores:, reasoning: reasoning_for(result) }
         }
       end
     end

data/lib/rubric_llm/metrics/context_recall.rb

@@ -5,7 +5,10 @@ module RubricLLM
     class ContextRecall < Base
       SYSTEM_PROMPT = <<~PROMPT
         You are an evaluation judge. Assess whether the provided contexts cover the information in the ground truth.
-        Context recall measures if the retrieved documents contain enough information to construct the ground truth answer.
+        Context recall is the fraction of facts in the ground truth supported by the contexts.
+        List every distinct factual claim in the ground truth. Mark it covered only if a numbered
+        context supports it, and use that context's 1-based index as source_context.
+        Use null for source_context when a fact is not covered.
 
         Respond with JSON only:
         {
@@ -31,18 +34,22 @@ module RubricLLM
         PROMPT
 
         result = judge_eval(system_prompt: SYSTEM_PROMPT, user_prompt:)
-        normalize(result)
+        normalize(result, context_chunks.size)
       end
 
       private
 
-      def normalize(result)
+      def normalize(result, count)
+        covered_facts = checked_items(result, "covered_facts", label: "covered facts", value_key: "covered", text_key: "fact")
+        valid_sources = covered_facts.all? do |item|
+          source = item["source_context"]
+          item["covered"] ? source.is_a?(Integer) && (1..count).cover?(source) : source.nil?
+        end
+        raise JudgeError, "Judge response has invalid source contexts" unless valid_sources
+
         {
-          score: Float(result["score"]),
-          details: {
-            covered_facts: result["covered_facts"],
-            reasoning: result["reasoning"]
-          }
+          score: fraction(covered_facts, "covered"),
+          details: { covered_facts:, reasoning: reasoning_for(result) }
         }
       end
     end

data/lib/rubric_llm/metrics/correctness.rb

@@ -5,7 +5,11 @@ module RubricLLM
     class Correctness < Base
       SYSTEM_PROMPT = <<~PROMPT
         You are an evaluation judge. Assess whether the answer matches the ground truth.
-        Consider semantic equivalence, not just exact string matching.
+        Consider semantic equivalence, not just exact string matching. Score the answer against
+        the question and reference: 1.0 if it fully answers with no material errors, 0.5 if it
+        is partly correct but misses required information or includes a material error, and 0.0
+        if it is wrong or does not answer. Use intermediate values for partial cases and explain
+        what is correct, missing, or wrong. Do not require irrelevant reference details.
 
         Respond with JSON only:
         {
@@ -36,7 +40,7 @@ module RubricLLM
       def normalize(result)
         {
           score: Float(result["score"]),
-          details: { reasoning: result["reasoning"] }
+          details: { reasoning: reasoning_for(result) }
         }
       end
     end

data/lib/rubric_llm/metrics/factual_accuracy.rb

@@ -5,7 +5,12 @@ module RubricLLM
     class FactualAccuracy < Base
       SYSTEM_PROMPT = <<~PROMPT
         You are an evaluation judge. Compare the factual claims in the candidate answer against the reference answer.
-        Identify any discrepancies where the candidate states something different from the reference.
+        Identify contradictions in candidate factual claims against the reference. Do not penalize
+        missing reference facts, which correctness measures. Score 1.0 if there are no contradictions,
+        0.5 if minor factual contradictions affect part of the answer, and 0.0 if major contradictions
+        undermine the answer. Use intermediate values for partial cases. Explain the score and list
+        each contradiction with severity minor or major. If the reference does not establish whether
+        a claim is true, do not call it a contradiction.
 
         Respond with JSON only:
         {
@@ -33,14 +38,25 @@ module RubricLLM
       private
 
       def normalize(result)
+        discrepancies = result["discrepancies"]
+        unless discrepancies.is_a?(Array) && discrepancies.all? { |item| valid_discrepancy?(item) }
+          raise JudgeError, "Judge response has invalid discrepancies"
+        end
+
+        score = Float(result["score"])
+        raise JudgeError, "Judge response score conflicts with no discrepancies" if discrepancies.empty? && score < 1.0
+        raise JudgeError, "Judge response score conflicts with discrepancies" if discrepancies.any? && score >= 1.0
+
         {
-          score: Float(result["score"]),
-          details: {
-            discrepancies: result["discrepancies"],
-            reasoning: result["reasoning"]
-          }
+          score:,
+          details: { discrepancies:, reasoning: reasoning_for(result) }
         }
       end
+
+      def valid_discrepancy?(item)
+        item.is_a?(Hash) && %w[claim reference].all? { |key| item[key].is_a?(String) && !item[key].strip.empty? } &&
+          %w[minor major].include?(item["severity"])
+      end
     end
   end
 end

data/lib/rubric_llm/metrics/faithfulness.rb

@@ -6,6 +6,8 @@ module RubricLLM
       SYSTEM_PROMPT = <<~PROMPT
         You are an evaluation judge. Assess whether the answer is faithful to the provided context.
         A faithful answer only contains information that is supported by the context.
+        List every factual claim in the answer. Mark a claim supported only if the context supports it.
+        The score is the fraction of supported claims. If there are no factual claims, do not invent claims.
 
         Respond with JSON only:
         {
@@ -36,11 +38,12 @@ module RubricLLM
       private
 
       def normalize(result)
+        claims = checked_items(result, "claims", label: "claims", value_key: "supported", text_key: "claim")
         {
-          score: Float(result["score"]),
+          score: fraction(claims, "supported"),
           details: {
-            claims: result["claims"],
-            reasoning: result["reasoning"]
+            claims:,
+            reasoning: reasoning_for(result)
           }
         }
       end

data/lib/rubric_llm/metrics/relevance.rb

@@ -5,7 +5,10 @@ module RubricLLM
     class Relevance < Base
       SYSTEM_PROMPT = <<~PROMPT
         You are an evaluation judge. Assess whether the answer is relevant to the question.
-        A relevant answer directly addresses what was asked.
+        A relevant answer directly addresses what was asked, regardless of its factual accuracy.
+        Score 1.0 if it directly addresses all parts of the question, 0.5 if it addresses only
+        part or is mostly tangential, and 0.0 if it does not address the question. Use intermediate
+        values for partial cases and explain the score. Do not score correctness here.
 
         Respond with JSON only:
         {
@@ -32,7 +35,7 @@ module RubricLLM
       def normalize(result)
         {
           score: Float(result["score"]),
-          details: { reasoning: result["reasoning"] }
+          details: { reasoning: reasoning_for(result) }
         }
       end
     end

data/lib/rubric_llm/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RubricLLM
-  VERSION = "0.6.0"
+  VERSION = "0.8.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rubric_llm
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.8.0
 platform: ruby
 authors:
 - David Paluy