rubric_llm 0.4.0 → 0.6.0.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 16ccb30778e4ace0b35900e0d488ac884042acf124e139181e4e364a6a5829ed
-  data.tar.gz: f5c1df6b47de48bd4c976a9986836aa2d2c66c6614dc94e4b6f4f8b12c090144
+  metadata.gz: 6d32ed4cba47a2fac15973d05bbf531317fa8d80f4c42b277dae5c302884ae57
+  data.tar.gz: a4bfbbc318593df77c33d36d41eda2e3d2f1b0cd4d0485fe9b8a61e72ee1519b
 SHA512:
-  metadata.gz: d267c7eceba64ebf8b9f5829b960ebf45aaec98bbd8545a393f16f8dfdf0070fb98ee8038e0fe9ddb1428164bbbcb677b6e8da0afa86513865187a358aad45a7
-  data.tar.gz: d0fff2333050e27bb785820f128ebaee7161f15e446f728beffbe2744078d95cd0de077b4a2674b26d8136391ee9c977f5cc87a1608f48ff96d9e0ed8cd04dd8
+  metadata.gz: a63b324d6f721297415feb55f6232b2771d58a7ca86a4a2ff0903abffec3c5fc5ad638c3d32632ed27bbf6cafea666a8247dcfa418d3454df61e3856c6c83b0a
+  data.tar.gz: 424912623b5f7f8157c91c10310c4e8ebe34e7406fcc146a3c146bd4e9de0cbe9b0100acc4b7a9a98174153f53178961f9c8233b120b3d3d23c11a5bf29bb360

data/CHANGELOG.md

@@ -5,6 +5,45 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0.rc1] - 2026-09-08
+
+This prerelease targets RubyLLM `2.0.0.rc1` and supports RubyLLM 2 only. Applications that use RubyLLM 1.x should stay on RubricLLM `0.5.x`. Stable `0.6.0` waits for the final RubyLLM 2.0 release and the complete Ruby 3.4 and 4.0 test matrix.
+
+### Changed
+
+- Require the exact RubyLLM `2.0.0.rc1` prerelease while the compatibility release is under review; Ruby >= 3.4 remains supported
+- Keep `max_tokens` and `RUBRIC_MAX_TOKENS` as the public RubricLLM setting while RubyLLM maps the limit to the selected provider and protocol
+- Treat explicit `temperature: nil` as an instruction to omit temperature from the provider request. Before 0.6.0.rc1, it selected `RUBRIC_TEMPERATURE` or `0.0`; an omitted temperature still reads `RUBRIC_TEMPERATURE` and defaults to `0.0`
+- Use OpenAI Responses by default through RubyLLM 2. OpenAI-compatible gateways that only accept Chat Completions can set `config.openai_protocol = :chat_completions`
+- Keep RubyLLM transport retries and RubricLLM judge retries separate. With RubyLLM's default `config.max_retries = 3` and RubricLLM's default `max_retries: 2`, a retryable metric failure can produce up to `(3 + 1) * (2 + 1) = 12` HTTP attempts
+- Keep Rails persistence migrations separate. Applications that use RubyLLM's Rails records must follow RubyLLM's upstream 2.0 migration guide
+
+## [0.5.0] - 2026-08-23
+
+### Fixed
+
+- The RSpec `hallucinate` matcher no longer reports a hallucination when the judge returns no score. A failed judge call previously read as a positive match and failed the build with a false verdict
+- `Comparison` pairs samples by `sample[:question]` instead of array position. Two reports built from the same dataset in a different order produced an invalid paired t-test. Questions present in only one report are dropped with a warning
+- `Judge#call` retries only transient failures (`RubyLLM::RateLimitError`, `ServerError`, `ServiceUnavailableError`, `OverloadedError`). A bad API key, an exhausted quota, an over-long prompt, or a judge contract violation now fails on the first attempt instead of sleeping through the full retry schedule. Transport timeouts and connection resets are left to RubyLLM's connection, which already retries them
+- `ContextPrecision` and `ContextRecall` drop blank and whitespace-only context chunks and return a `No context provided` error when nothing usable remains, matching `Faithfulness`
+- `Statistics.two_tailed_p` rescues only `Math::DomainError`, `ZeroDivisionError`, and `FloatDomainError`. It previously swallowed every `StandardError` and returned 1.0
+- A paired t-test over a constant difference returns 1.0 instead of a spurious near-zero p-value caused by float error in the variance
+- The RSpec `hallucinate` matcher renders a missing score as `nil` in its failure message instead of an empty string
+
+### Removed
+
+- The `Comparison` constructor no longer warns about reports of different sizes. Different sizes are fine when the questions match, and equal sizes can still drop every pair when they do not. The pairing warnings report what is actually dropped
+
+### Added
+
+- Holm-Bonferroni correction across metrics in `Comparison`. Each result carries `:p_value_adjusted` next to the raw `:p_value`, `#summary` prints a `p-adj` column, and `#significant_improvements` / `#significant_regressions` test the adjusted value. Six independent tests at alpha 0.05 gave a family-wise false-positive rate near 26%
+- `RubricLLM::Statistics`, a module holding the paired t-test, the two-tailed p-value, the Holm adjustment, and the incomplete beta function. No LLM calls, no state
+- `Metrics::Base.normalize_context`, the single definition of usable context
+
+### Changed
+
+- The Minitest `assert_faithful` and `refute_hallucination` helpers and the RSpec `be_faithful` and `hallucinate` matchers raise `ArgumentError` on an empty or blank context instead of returning a verdict that no judge produced
+
 ## [0.4.0] - 2026-07-11
 
 ### Added

data/README.md

@@ -6,20 +6,31 @@ Lightweight LLM evaluation framework for Ruby, inspired by [DeepEval](https://gi
 [![CI](https://github.com/dpaluy/rubric_llm/actions/workflows/ci.yml/badge.svg)](https://github.com/dpaluy/rubric_llm/actions/workflows/ci.yml)
 [![Wiki](https://img.shields.io/badge/docs-wiki-blue)](https://github.com/dpaluy/rubric_llm/wiki)
 
-Provider-agnostic evaluation with pluggable metrics, statistical A/B comparison, and test framework integration — no Rails, no ActiveRecord, no UI. Works anywhere Ruby runs.
+Provider-agnostic evaluation with pluggable metrics, statistical A/B comparison, and test framework integration: no Rails, no ActiveRecord, no UI. Works anywhere Ruby runs.
+
+`0.6.0.rc1` is a prerelease for RubyLLM `2.0.0.rc1`. It supports RubyLLM 2 only. Applications that use RubyLLM 1.x should stay on RubricLLM `0.5.x`. Stable `0.6.0` waits for the final RubyLLM 2.0 release and the complete Ruby 3.4 and 4.0 test matrix.
 
 ## Installation
 
-Add to your Gemfile:
+Install the release candidate with both exact prerelease constraints in the same Gemfile change:
 
 ```ruby
-gem "rubric_llm"
+gem "rubric_llm", "0.6.0.rc1"
+gem "ruby_llm", "2.0.0.rc1"
 ```
 
+Then resolve both gems together:
+
+```bash
+bundle update rubric_llm ruby_llm
+```
+
+RubyLLM's upstream prerelease pin command is `bundle add ruby_llm --version 2.0.0.rc1`. If the application still has a RubyLLM 1.x constraint, update both constraints before running Bundler.
+
 Or install directly:
 
 ```bash
-gem install rubric_llm
+gem install rubric_llm --version 0.6.0.rc1 --pre
 ```
 
 ## Quick Start
@@ -62,6 +73,25 @@ RubricLLM.configure do |c|
 end
 ```
 
+`max_tokens` remains RubricLLM's public setting, and `RUBRIC_MAX_TOKENS` remains its environment variable. RubyLLM 2 maps this shared limit to the selected provider and protocol.
+
+When `temperature` is omitted, RubricLLM reads `RUBRIC_TEMPERATURE` and uses `0.0` when the variable is not set. Before 0.6.0.rc1, an explicit `temperature: nil` selected that environment value or `0.0`; it now means that RubricLLM omits temperature from the provider request, so the provider chooses its default:
+
+```ruby
+RubricLLM::Config.new                  # RUBRIC_TEMPERATURE, otherwise 0.0
+RubricLLM::Config.new(temperature: nil) # omit temperature from the request
+```
+
+RubyLLM 2 uses the OpenAI Responses protocol by default when the selected model supports it. For an OpenAI-compatible gateway that only accepts Chat Completions, configure RubyLLM before evaluating:
+
+```ruby
+RubyLLM.configure do |config|
+  config.openai_protocol = :chat_completions
+end
+```
+
+Structured output support depends on the selected provider and model. RubricLLM sends its schema when RubyLLM reports structured output support. Otherwise it requests JSON text and validates the response object and score locally. Check the target provider's support before relying on a schema or a specific protocol. See RubyLLM's [2.0 upgrade guide](https://rubyllm.com/next/upgrading/), [request control guide](https://rubyllm.com/next/chat-request-control/), and [structured output support](https://rubyllm.com/next/structured-output/).
+
 ### Environment Variables
 
 All config fields can be set via environment variables:
@@ -100,6 +130,14 @@ RubricLLM.configure do |c|
 end
 ```
 
+RubricLLM has no Rails models or database migrations. If the application also uses RubyLLM's Rails persistence, follow RubyLLM's 2.0 upgrade guide and run its phased migrations separately.
+
+### Retries
+
+RubyLLM transport retries and RubricLLM judge retries remain separate. With RubyLLM's default `config.max_retries = 3` and RubricLLM's default `max_retries: 2`, one retryable metric failure can produce up to `(3 + 1) * (2 + 1) = 12` HTTP attempts. Set `RUBRIC_MAX_RETRIES` and `RUBRIC_RETRY_BASE_DELAY` for RubricLLM's layer, and set RubyLLM's `config.max_retries` and related transport settings for its layer. RubricLLM 0.6 does not combine or redesign these retry layers.
+
+RubyLLM classifies OpenAI's HTTP 429 `insufficient_quota` response as a rate-limit error, so an exhausted account uses both retry budgets and their delays before the error is returned.
+
 ## Metrics
 
 ### LLM-as-Judge Metrics
@@ -116,14 +154,14 @@ These metrics use a judge LLM to evaluate quality. Each sends a structured promp
 | **Faithfulness** | Is every claim in the answer supported by the context? | `context` |
 
 ```ruby
-# Only context — gets faithfulness, relevance, context_precision
+# Only context (gets faithfulness, relevance, context_precision)
 result = RubricLLM.evaluate(
   question: "How does photosynthesis work?",
   answer: "Plants convert sunlight into energy.",
   context: ["Photosynthesis is the process by which plants convert light energy into chemical energy."]
 )
 
-# With ground truth — gets all metrics
+# With ground truth (gets all metrics)
 result = RubricLLM.evaluate(
   question: "How does photosynthesis work?",
   answer: "Plants convert sunlight into energy.",
@@ -155,7 +193,7 @@ result.scores[:tone_metric]  # => 0.85
 
 ### Retrieval Metrics
 
-Pure math — no LLM calls, no API key needed.
+Pure math, no LLM calls, no API key needed.
 
 ```ruby
 result = RubricLLM.evaluate_retrieval(
@@ -164,9 +202,9 @@ result = RubricLLM.evaluate_retrieval(
 )
 
 result.precision_at_k(3)  # => 0.67
-result.recall_at_k(3)     # => 0.90
-result.mrr                # => 0.90
-result.ndcg               # => 0.86
+result.recall_at_k(3)     # => 1.0
+result.mrr                # => 1.0
+result.ndcg               # => 0.92
 result.hit_rate           # => 1.0
 ```
 
@@ -217,12 +255,14 @@ comparison = RubricLLM.compare(report_a, report_b)
 
 puts comparison.summary
 # A/B Comparison
-# ======================================================================
-# Metric                      A        B    Delta    p-value  Sig
-# ----------------------------------------------------------------------
-# faithfulness                0.880    0.920   +0.040     0.0230    *
-# relevance                   0.850    0.860   +0.010     0.4210
-# correctness                 0.910    0.940   +0.030     0.0089   **
+# ================================================================================
+# Metric                      A        B    Delta    p-value      p-adj  Sig
+# --------------------------------------------------------------------------------
+# faithfulness            0.880    0.920   +0.040     0.0023     0.0068   **
+# relevance               0.850    0.860   +0.010     0.3081     0.3081
+# correctness             0.910    0.940   +0.030     0.0240     0.0480    *
+#
+# p-adj: Holm-Bonferroni adjusted across 3 metrics. Significance uses p-adj.
 
 comparison.significant_improvements   # => [:faithfulness, :correctness]
 comparison.significant_regressions    # => []
@@ -230,6 +270,16 @@ comparison.significant_regressions    # => []
 
 Significance markers: `*` (p < 0.05), `**` (p < 0.01), `***` (p < 0.001)
 
+### Pairing
+
+A paired t-test needs the same sample on both sides. The comparison pairs results by `sample[:question]`, not by position, so a reordered dataset still gives a valid test. Questions present in only one report are dropped with a warning. If a question repeats an uneven number of times across the two reports, the extra occurrences are dropped with a warning.
+
+`evaluate_batch` requires a `:question` on every sample, so reports it produces always pair by identity. Hand-built reports whose results carry no `sample[:question]` fall back to position pairing and warn.
+
+### Multiple comparisons
+
+Every metric gets its own t-test. Six tests at alpha 0.05 give a family-wise false-positive rate near 26%, so each raw `p_value` is corrected with the Holm-Bonferroni step-down method and reported as `p_value_adjusted`. The significance markers and both `significant_*` methods read the adjusted value. The raw value stays in the result for reference.
+
 For the statistical reasoning behind paired t-tests and how to read these p-values, see [Understanding A/B Comparison](https://github.com/dpaluy/rubric_llm/wiki/Understanding-A-B-Comparison) on the wiki.
 
 ## Test Integration
@@ -298,7 +348,7 @@ rescue RubricLLM::Error => e
 end
 ```
 
-Individual metric failures are handled gracefully — a failed metric returns `nil` for the score and includes the error in details:
+Individual metric failures are handled gracefully: a failed metric returns `nil` for the score and includes the error in details:
 
 ```ruby
 result = RubricLLM.evaluate(question: "q", answer: "a")
@@ -311,18 +361,18 @@ result.overall                         # => mean of non-nil scores only
 
 ```bash
 bundle install
-bundle exec rake test
+bundle exec rake test test_contract
 bundle exec rubocop
 ```
 
 ## Limitations
 
-RubricLLM uses LLM-as-Judge — an LLM scores another LLM's output. This is the industry-standard approach (used by Ragas, DeepEval, ARES), but it means the judge shares the same class of failure modes as the system being evaluated. If the judge hallucinates that an answer is faithful, you get a false positive.
+RubricLLM uses LLM-as-Judge: an LLM scores another LLM's output. This is the industry-standard approach (used by Ragas, DeepEval, ARES), but it means the judge shares the same class of failure modes as the system being evaluated. If the judge hallucinates that an answer is faithful, you get a false positive.
 
 Mitigations built into the framework:
 
 - **Cross-model judging.** Configure a different model as judge than the one being evaluated. Don't let gpt-5.5 grade gpt-5.5.
-- **Retrieval metrics are pure math.** `precision_at_k`, `recall_at_k`, `mrr`, `ndcg` — no LLM involved, no judge bias. See [Why Retrieval Metrics Are Pure Math](https://github.com/dpaluy/rubric_llm/wiki/Why-Retrieval-Metrics-Are-Pure-Math).
+- **Retrieval metrics are pure math.** `precision_at_k`, `recall_at_k`, `mrr`, `ndcg` (no LLM involved, no judge bias). See [Why Retrieval Metrics Are Pure Math](https://github.com/dpaluy/rubric_llm/wiki/Why-Retrieval-Metrics-Are-Pure-Math).
 - **Custom non-LLM metrics.** Subclass `Metrics::Base` with regex checks, embedding similarity, or any deterministic logic.
 - **Statistical comparison.** A/B testing with paired t-tests surfaces systematic judge bias across runs.
 
@@ -338,7 +388,7 @@ Ruby has two LLM evaluation options today. Neither fits most use cases:
 | **LLM access** | Raw HTTP (OpenAI/Anthropic only) | You implement it | RubyLLM (any provider) |
 | **Rails required?** | No | Yes (engine + 6 migrations) | No |
 | **ActiveRecord?** | No | Yes | No |
-| **A/B comparison** | Basic | No | Paired t-test with p-values |
+| **A/B comparison** | Basic | No | Paired t-test with Holm-corrected p-values |
 | **Test assertions** | Minitest + RSpec | No | Minitest + RSpec |
 | **Pluggable metrics** | No (fixed set) | Yes | Yes |
 | **Retrieval metrics** | Yes | No | Yes |
@@ -347,13 +397,13 @@ Ruby has two LLM evaluation options today. Neither fits most use cases:
 
 Deep dives live in the [project wiki](https://github.com/dpaluy/rubric_llm/wiki):
 
-- [Understanding A/B Comparison](https://github.com/dpaluy/rubric_llm/wiki/Understanding-A-B-Comparison) — what paired t-tests and p-values mean for model comparison
-- [Why Retrieval Metrics Are Pure Math](https://github.com/dpaluy/rubric_llm/wiki/Why-Retrieval-Metrics-Are-Pure-Math) — why `precision_at_k`, `recall_at_k`, `mrr`, `ndcg`, and `hit_rate` are deterministic and bias-free
+- [Understanding A/B Comparison](https://github.com/dpaluy/rubric_llm/wiki/Understanding-A-B-Comparison): what paired t-tests and p-values mean for model comparison
+- [Why Retrieval Metrics Are Pure Math](https://github.com/dpaluy/rubric_llm/wiki/Why-Retrieval-Metrics-Are-Pure-Math): why `precision_at_k`, `recall_at_k`, `mrr`, `ndcg`, and `hit_rate` are deterministic and bias-free
 
 ## Requirements
 
 - Ruby >= 3.4
-- [ruby_llm](https://github.com/crmne/ruby_llm) ~> 1.0
+- [ruby_llm](https://github.com/crmne/ruby_llm) = 2.0.0.rc1 for RubricLLM 0.6.0.rc1
 - An API key for your chosen LLM provider (set via RubyLLM configuration)
 
 ## Contributing

data/lib/rubric_llm/comparison.rb

@@ -7,12 +7,6 @@ module RubricLLM
     def initialize(report_a, report_b)
       @report_a = report_a
       @report_b = report_b
-
-      return if report_a.results.size == report_b.results.size
-
-      warn "[RubricLLM] Comparison reports have different sizes " \
-           "(#{report_a.results.size} vs #{report_b.results.size}). " \
-           "Unmatched pairs will be dropped."
     end
 
     def results
@@ -21,24 +15,27 @@ module RubricLLM
 
     def summary
       lines = ["A/B Comparison"]
-      lines << ("=" * 70)
-      lines << "Metric                      A        B    Delta    p-value  Sig"
-      lines << ("-" * 70)
+      lines << ("=" * 80)
+      lines << "Metric                      A        B    Delta    p-value      p-adj  Sig"
+      lines << ("-" * 80)
 
       results.each do |metric, r|
-        lines << format("%-20s %8.3f %8.3f %+8.3f %10.4f %4s",
-                        metric, r[:mean_a], r[:mean_b], r[:delta], r[:p_value], r[:significance])
+        lines << format("%-20s %8.3f %8.3f %+8.3f %10.4f %10.4f %4s",
+                        metric, r[:mean_a], r[:mean_b], r[:delta], r[:p_value], r[:p_value_adjusted],
+                        r[:significance])
       end
 
+      lines << ""
+      lines << "p-adj: Holm-Bonferroni adjusted across #{results.size} metrics. Significance uses p-adj."
       lines.join("\n")
     end
 
     def significant_improvements(alpha: 0.05)
-      results.select { |_, r| r[:p_value] < alpha && r[:delta].positive? }.keys
+      results.select { |_, r| r[:p_value_adjusted] < alpha && r[:delta].positive? }.keys
     end
 
     def significant_regressions(alpha: 0.05)
-      results.select { |_, r| r[:p_value] < alpha && r[:delta].negative? }.keys
+      results.select { |_, r| r[:p_value_adjusted] < alpha && r[:delta].negative? }.keys
     end
 
     private
@@ -46,110 +43,98 @@ module RubricLLM
     def compute_results
       metrics = (report_a.metric_stats.keys | report_b.metric_stats.keys)
 
-      metrics.each_with_object({}) do |metric, hash|
-        paired_scores = report_a.scores_for(metric)
-                                .zip(report_b.scores_for(metric))
-                                .select { |score_a, score_b| !score_a.nil? && !score_b.nil? }
+      raw = metrics.each_with_object({}) do |metric, hash|
+        stats = metric_stats(metric)
+        hash[metric] = stats if stats
+      end
+
+      apply_holm_correction(raw)
+    end
 
-        next if paired_scores.empty?
+    def metric_stats(metric)
+      paired_scores = paired_results
+                      .map { |result_a, result_b| [result_a.scores[metric], result_b.scores[metric]] }
+                      .reject { |score_a, score_b| score_a.nil? || score_b.nil? }
 
-        scores_a, scores_b = paired_scores.transpose
+      return nil if paired_scores.empty?
 
-        mean_a = scores_a.sum / scores_a.size.to_f
-        mean_b = scores_b.sum / scores_b.size.to_f
-        delta = mean_b - mean_a
-        p_value = paired_t_test(scores_a, scores_b)
+      scores_a, scores_b = paired_scores.transpose
+      mean_a = scores_a.sum / scores_a.size.to_f
+      mean_b = scores_b.sum / scores_b.size.to_f
 
-        hash[metric] = {
-          mean_a:,
-          mean_b:,
-          delta:,
-          p_value:,
-          significance: significance_marker(p_value)
-        }
-      end
+      {
+        mean_a:,
+        mean_b:,
+        delta: mean_b - mean_a,
+        p_value: Statistics.paired_t_test(scores_a, scores_b)
+      }
     end
 
-    def paired_t_test(a, b)
-      n = [a.size, b.size].min
-      return 1.0 if n < 2
+    # A paired t-test requires the same sample on both sides. Pair by question
+    # instead of array position so a reordered dataset stays valid.
+    def paired_results
+      @paired_results ||= build_pairs
+    end
+
+    def build_pairs
+      groups_a = report_a.results.group_by { |result| pair_key(result) }
+      groups_b = report_b.results.group_by { |result| pair_key(result) }
 
-      diffs = a.first(n).zip(b.first(n)).map { |x, y| y - x }
-      mean_d = diffs.sum / n.to_f
-      var_d = diffs.sum { |d| (d - mean_d)**2 } / (n - 1).to_f
-      se = Math.sqrt(var_d / n)
+      warn_unkeyed(groups_a[nil].to_a.size + groups_b[nil].to_a.size)
 
-      return 1.0 if se.zero?
+      matched = groups_a.keys & groups_b.keys
+      warn_unmatched(((groups_a.keys | groups_b.keys) - matched).compact)
 
-      t = mean_d / se
-      df = n - 1
+      matched.flat_map do |key|
+        list_a = groups_a[key]
+        list_b = groups_b[key]
+        warn_uneven(key, list_a.size, list_b.size) unless list_a.size == list_b.size
 
-      # Two-tailed p-value approximation using Student's t-distribution
-      two_tailed_p(t.abs, df)
+        size = [list_a.size, list_b.size].min
+        list_a.first(size).zip(list_b.first(size))
+      end
     end
 
-    # Two-tailed p-value for Student's t-distribution.
-    # p = I_x(df/2, 1/2) where x = df/(df + t²)
-    def two_tailed_p(t, df)
-      x = df / (df + (t**2))
-      regularized_beta(x, df / 2.0, 0.5)
-    rescue StandardError
-      1.0
+    def pair_key(result)
+      sample = result.sample
+      sample.is_a?(Hash) ? sample[:question] : nil
     end
 
-    # Regularized incomplete beta function via continued fraction (Lentz's method).
-    def regularized_beta(x, a, b)
-      return 0.0 if x <= 0.0
-      return 1.0 if x >= 1.0
+    # Results with no sample[:question] share one bucket and pair by position,
+    # which is the behaviour identity pairing exists to replace. Say so.
+    def warn_unkeyed(count)
+      return if count.zero?
 
-      ln_beta = Math.lgamma(a + b)[0] - Math.lgamma(a)[0] - Math.lgamma(b)[0]
-      front = Math.exp(ln_beta + (a * Math.log(x)) + (b * Math.log(1.0 - x)))
+      warn "[RubricLLM] #{count} result(s) have no sample[:question]. They share one bucket and " \
+           "pair by position. Give every sample a :question to pair them reliably."
+    end
+
+    def warn_unmatched(keys)
+      return if keys.empty?
 
-      result = if x < ((a + 1.0) / (a + b + 2.0))
-                 front * beta_continued_fraction(a, b, x) / a
-               else
-                 1.0 - ((front * beta_continued_fraction(b, a, 1.0 - x)) / b)
-               end
+      warn "[RubricLLM] Comparison dropped #{keys.size} question(s) present in only one report. " \
+           "Paired tests need the same questions on both sides."
+    end
 
-      result.clamp(0.0, 1.0)
+    def warn_uneven(key, size_a, size_b)
+      warn "[RubricLLM] Question #{key.inspect} appears #{size_a} time(s) in report A and " \
+           "#{size_b} time(s) in report B. Extra occurrences are dropped."
     end
 
-    def beta_continued_fraction(a, b, x)
-      tiny = 1e-30
-      qab = a + b
-      qap = a + 1.0
-      qam = a - 1.0
-
-      c = 1.0
-      d = 1.0 - ((qab * x) / qap)
-      d = tiny if d.abs < tiny
-      d = 1.0 / d
-      fraction = d
-
-      (1..200).each do |m|
-        m2 = 2 * m
-
-        numerator = (m * (b - m) * x) / ((qam + m2) * (a + m2))
-        d = 1.0 + (numerator * d)
-        d = tiny if d.abs < tiny
-        c = 1.0 + (numerator / c)
-        c = tiny if c.abs < tiny
-        d = 1.0 / d
-        fraction *= c * d
-
-        numerator = -((a + m) * (qab + m) * x) / ((a + m2) * (qap + m2))
-        d = 1.0 + (numerator * d)
-        d = tiny if d.abs < tiny
-        c = 1.0 + (numerator / c)
-        c = tiny if c.abs < tiny
-        d = 1.0 / d
-        delta = c * d
-        fraction *= delta
-
-        break if (delta - 1.0).abs < 1e-12
+    # One t-test per metric inflates the family-wise error rate, so adjust
+    # before calling any metric significant.
+    def apply_holm_correction(results)
+      metrics = results.keys
+      adjusted = Statistics.holm_adjust(metrics.map { |metric| results[metric][:p_value] })
+
+      metrics.each_with_index do |metric, index|
+        results[metric] = results[metric].merge(
+          p_value_adjusted: adjusted[index],
+          significance: significance_marker(adjusted[index])
+        )
       end
 
-      fraction
+      results
     end
 
     def significance_marker(p)

data/lib/rubric_llm/config.rb

@@ -5,11 +5,12 @@ module RubricLLM
     attr_accessor :judge_model, :judge_provider, :temperature, :max_tokens, :custom_prompt,
                   :max_retries, :retry_base_delay, :concurrency
 
-    def initialize(judge_model: nil, judge_provider: nil, temperature: nil, max_tokens: nil, # rubocop:disable Metrics/ParameterLists
+    def initialize(judge_model: nil, judge_provider: nil, # rubocop:disable Metrics/ParameterLists
+                   temperature: Float(ENV.fetch("RUBRIC_TEMPERATURE", "0.0")), max_tokens: nil,
                    custom_prompt: nil, max_retries: nil, retry_base_delay: nil, concurrency: nil, validate: false)
       @judge_model = judge_model || ENV.fetch("RUBRIC_JUDGE_MODEL", "gpt-4o")
       @judge_provider = (judge_provider || ENV.fetch("RUBRIC_JUDGE_PROVIDER", "openai")).to_sym
-      @temperature = temperature || Float(ENV.fetch("RUBRIC_TEMPERATURE", "0.0"))
+      @temperature = temperature
       @max_tokens = max_tokens || Integer(ENV.fetch("RUBRIC_MAX_TOKENS", "4096"))
       @custom_prompt = custom_prompt
       @max_retries = max_retries || Integer(ENV.fetch("RUBRIC_MAX_RETRIES", "2"))
@@ -61,9 +62,9 @@ module RubricLLM
     end
 
     def validate_temperature
-      return if temperature.is_a?(Numeric) && temperature.between?(0.0, 2.0)
+      return if temperature.nil? || (temperature.is_a?(Numeric) && temperature.between?(0.0, 2.0))
 
-      raise ConfigurationError, "temperature must be between 0.0 and 2.0"
+      raise ConfigurationError, "temperature must be nil or between 0.0 and 2.0"
     end
 
     def validate_max_tokens

data/lib/rubric_llm/judge.rb

@@ -4,6 +4,20 @@ require "json"
 
 module RubricLLM
   class Judge
+    # Failures worth retrying: the same request may succeed later.
+    # Everything else (bad key, no credit, malformed request, prompt too long,
+    # contract violations in the judge response) fails on the first attempt.
+    #
+    # Transport failures are absent on purpose. RubyLLM's connection already
+    # retries timeouts and connection resets, so by the time one reaches us it
+    # has been tried several times and is not worth another round.
+    TRANSIENT_ERRORS = [
+      RubyLLM::RateLimitError,
+      RubyLLM::ServerError,
+      RubyLLM::ServiceUnavailableError,
+      RubyLLM::OverloadedError
+    ].freeze
+
     METRIC_RESPONSE_SCHEMA = {
       name: "rubric_llm_metric_response",
       strict: false,
@@ -37,20 +51,15 @@ module RubricLLM
         attempts += 1
         chat = RubyLLM.chat(model: config.judge_model, provider: config.judge_provider)
         chat.with_temperature(config.temperature)
-        chat.with_params(max_tokens: config.max_tokens)
+        chat.with_max_output_tokens(config.max_tokens)
         apply_response_schema(chat)
 
         full_system_prompt = build_system_prompt(system_prompt)
         chat.with_instructions(full_system_prompt)
         response = chat.ask(user_prompt)
-        content = response.content
-        validate_response!(content.is_a?(Hash) ? content : parse_json(content))
+        validate_response!(parse_json(response.content))
       rescue StandardError => e
-        if attempts > config.max_retries
-          raise e if e.is_a?(JudgeError)
-
-          raise JudgeError, "Judge call failed: #{e.message}"
-        end
+        raise wrap_error(e) unless transient?(e) && attempts <= config.max_retries
 
         sleep(config.retry_base_delay * (2**(attempts - 1)))
         retry
@@ -79,18 +88,24 @@ module RubricLLM
 
     private
 
+    def transient?(error)
+      TRANSIENT_ERRORS.any? { |klass| error.is_a?(klass) }
+    end
+
+    def wrap_error(error)
+      return error if error.is_a?(JudgeError)
+
+      JudgeError.new("Judge call failed: #{error.message}")
+    end
+
     def apply_response_schema(chat)
-      return chat unless chat.respond_to?(:with_schema)
       return chat unless structured_output_supported?(chat)
 
       chat.with_schema(METRIC_RESPONSE_SCHEMA)
     end
 
     def structured_output_supported?(chat)
-      return true unless chat.respond_to?(:model)
-      return true unless chat.model.respond_to?(:structured_output?)
-
-      chat.model.structured_output?
+      chat.model.supports?(:structured_output)
     end
 
     def validate_response!(response)

data/lib/rubric_llm/metrics/base.rb

@@ -3,6 +3,20 @@
 module RubricLLM
   module Metrics
     class Base
+      # Single source of truth for what counts as usable context.
+      # Blank and whitespace-only chunks are dropped.
+      def self.normalize_context(context)
+        Array(context).map { |chunk| chunk.to_s.strip }.reject(&:empty?)
+      end
+
+      # An empty context cannot produce a faithfulness score. Reject it as a caller
+      # error instead of letting a nil score read as a quality verdict.
+      def self.require_context!(context)
+        return unless normalize_context(context).empty?
+
+        raise ArgumentError, "context must contain at least one non-empty entry"
+      end
+
       attr_reader :judge
 
       def initialize(judge:)

data/lib/rubric_llm/metrics/context_precision.rb

@@ -16,13 +16,14 @@ module RubricLLM
       PROMPT
 
       def call(question:, context: [], **)
-        return { score: nil, details: { error: "No context provided" } } if Array(context).empty?
+        context_chunks = Base.normalize_context(context)
+        return { score: nil, details: { error: "No context provided" } } if context_chunks.empty?
 
         user_prompt = <<~PROMPT
           Question: #{question}
 
           Contexts:
-          #{Array(context).each_with_index.map { |c, i| "#{i + 1}. #{c}" }.join("\n")}
+          #{context_chunks.each_with_index.map { |c, i| "#{i + 1}. #{c}" }.join("\n")}
 
           Evaluate how relevant each context is to the question.
         PROMPT

data/lib/rubric_llm/metrics/context_recall.rb

@@ -17,11 +17,13 @@ module RubricLLM
 
       def call(context: [], ground_truth: nil, **)
         return { score: nil, details: { error: "No ground truth provided" } } if ground_truth.nil?
-        return { score: nil, details: { error: "No context provided" } } if Array(context).empty?
+
+        context_chunks = Base.normalize_context(context)
+        return { score: nil, details: { error: "No context provided" } } if context_chunks.empty?
 
         user_prompt = <<~PROMPT
           Contexts:
-          #{Array(context).each_with_index.map { |c, i| "#{i + 1}. #{c}" }.join("\n")}
+          #{context_chunks.each_with_index.map { |c, i| "#{i + 1}. #{c}" }.join("\n")}
 
           Ground Truth: #{ground_truth}
 

data/lib/rubric_llm/metrics/faithfulness.rb

@@ -16,7 +16,7 @@ module RubricLLM
       PROMPT
 
       def call(question:, answer:, context: [], **)
-        context_chunks = Array(context).map { |chunk| chunk.to_s.strip }.reject(&:empty?)
+        context_chunks = Base.normalize_context(context)
         return { score: nil, details: { error: "No context provided" } } if context_chunks.empty?
 
         user_prompt = <<~PROMPT

data/lib/rubric_llm/minitest.rb

@@ -4,7 +4,8 @@ require "rubric_llm"
 
 module RubricLLM
   module Assertions
-    def assert_faithful(answer, context, question: "", threshold: 0.8, config: RubricLLM.config)
+    def assert_faithful(answer, context, question: "", threshold: DEFAULT_THRESHOLD, config: RubricLLM.config)
+      require_context!(context)
       result = evaluate_metric(Metrics::Faithfulness, question:, answer:, context:, config:)
       score = result[:score]
 
@@ -12,7 +13,7 @@ module RubricLLM
              "Expected faithfulness >= #{threshold}, got #{score || "nil"}.#{failure_details(result)}"
     end
 
-    def assert_relevant(question, answer, threshold: 0.8, config: RubricLLM.config)
+    def assert_relevant(question, answer, threshold: DEFAULT_THRESHOLD, config: RubricLLM.config)
       result = evaluate_metric(Metrics::Relevance, question:, answer:, config:)
       score = result[:score]
 
@@ -20,7 +21,7 @@ module RubricLLM
              "Expected relevance >= #{threshold}, got #{score || "nil"}.#{failure_details(result)}"
     end
 
-    def assert_correct(answer, ground_truth, question: "", threshold: 0.8, config: RubricLLM.config)
+    def assert_correct(answer, ground_truth, question: "", threshold: DEFAULT_THRESHOLD, config: RubricLLM.config)
       result = evaluate_metric(Metrics::Correctness, question:, answer:, ground_truth:, config:)
       score = result[:score]
 
@@ -28,7 +29,8 @@ module RubricLLM
              "Expected correctness >= #{threshold}, got #{score || "nil"}.#{failure_details(result)}"
     end
 
-    def refute_hallucination(answer, context, question: "", threshold: 0.8, config: RubricLLM.config)
+    def refute_hallucination(answer, context, question: "", threshold: DEFAULT_THRESHOLD, config: RubricLLM.config)
+      require_context!(context)
       result = evaluate_metric(Metrics::Faithfulness, question:, answer:, context:, config:)
       score = result[:score]
 
@@ -38,6 +40,10 @@ module RubricLLM
 
     private
 
+    def require_context!(context)
+      Metrics::Base.require_context!(context)
+    end
+
     def evaluate_metric(metric_class, config:, **)
       judge = Judge.new(config:)
       metric = metric_class.new(judge:)

data/lib/rubric_llm/report.rb

@@ -19,7 +19,7 @@ module RubricLLM
       results.sort_by { |r| r.overall || -Float::INFINITY }.first(n)
     end
 
-    def failures(threshold: 0.8)
+    def failures(threshold: DEFAULT_THRESHOLD)
       results.reject { |r| r.pass?(threshold:) }
     end
 

data/lib/rubric_llm/result.rb

@@ -17,7 +17,7 @@ module RubricLLM
       valid.sum / valid.size.to_f
     end
 
-    def pass?(threshold: 0.8)
+    def pass?(threshold: DEFAULT_THRESHOLD)
       return false unless valid?
 
       score = overall

data/lib/rubric_llm/rspec.rb

@@ -24,7 +24,7 @@ module RubricLLM
       attr_reader :threshold, :config, :result
 
       def initialize
-        @threshold = 0.8
+        @threshold = DEFAULT_THRESHOLD
         @config = RubricLLM.config
       end
 
@@ -40,6 +40,10 @@ module RubricLLM
 
       private
 
+      def require_context!(context)
+        Metrics::Base.require_context!(context)
+      end
+
       def evaluate(metric_class, **)
         judge = Judge.new(config:)
         metric = metric_class.new(judge:)
@@ -51,6 +55,7 @@ module RubricLLM
     class FaithfulnessMatcher < BaseMatcher
       def initialize(context, question: nil)
         super()
+        require_context!(context)
         @context = context
         @question = question
       end
@@ -113,17 +118,19 @@ module RubricLLM
     class HallucinationMatcher < BaseMatcher
       def initialize(context, question: nil)
         super()
+        require_context!(context)
         @context = context
         @question = question
       end
 
+      # Fail closed: a missing score is not evidence of a hallucination.
       def matches?(answer)
         score = evaluate(Metrics::Faithfulness, question: @question || "", answer:, context: @context)
-        score.nil? || score < threshold
+        !score.nil? && score < threshold
       end
 
       def failure_message
-        "expected hallucination (faithfulness < #{threshold}), got #{result[:score]}"
+        "expected hallucination (faithfulness < #{threshold}), got #{result[:score] || "nil"}"
       end
 
       def failure_message_when_negated

data/lib/rubric_llm/statistics.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+module RubricLLM
+  # Pure statistical helpers. No LLM calls, no state.
+  module Statistics
+    module_function
+
+    # Two-tailed paired t-test. Both arrays must already be paired and equal length.
+    # Returns 1.0 when there is nothing to test.
+    def paired_t_test(scores_a, scores_b)
+      n = scores_a.size
+      return 1.0 if n < 2
+
+      diffs = scores_a.zip(scores_b).map { |x, y| y - x }
+      mean_d = diffs.sum / n.to_f
+      var_d = diffs.sum { |d| (d - mean_d)**2 } / (n - 1).to_f
+      se = Math.sqrt(var_d / n)
+
+      # A constant shift has no spread to test. Compare against the scale of the
+      # data rather than exact zero, or float error turns it into a huge t value.
+      return 1.0 if se <= Float::EPSILON * [mean_d.abs, 1.0].max
+
+      two_tailed_p((mean_d / se).abs, n - 1)
+    end
+
+    # Two-tailed p-value for Student's t-distribution.
+    # p = I_x(df/2, 1/2) where x = df/(df + t²)
+    def two_tailed_p(t, df)
+      x = df / (df + (t**2))
+      regularized_beta(x, df / 2.0, 0.5)
+    rescue Math::DomainError, ZeroDivisionError, FloatDomainError
+      1.0
+    end
+
+    # Holm-Bonferroni step-down adjustment.
+    # Takes p-values in any order, returns adjusted values in the same order.
+    def holm_adjust(p_values)
+      count = p_values.size
+      return p_values.dup if count.zero?
+
+      adjusted = Array.new(count)
+      running_max = 0.0
+
+      p_values.each_with_index.sort_by(&:first).each_with_index do |(p_value, position), rank|
+        running_max = ((count - rank) * p_value).clamp(running_max, 1.0)
+        adjusted[position] = running_max
+      end
+
+      adjusted
+    end
+
+    # Regularized incomplete beta function via continued fraction (Lentz's method).
+    def regularized_beta(x, a, b)
+      return 0.0 if x <= 0.0
+      return 1.0 if x >= 1.0
+
+      ln_beta = Math.lgamma(a + b)[0] - Math.lgamma(a)[0] - Math.lgamma(b)[0]
+      front = Math.exp(ln_beta + (a * Math.log(x)) + (b * Math.log(1.0 - x)))
+
+      result = if x < ((a + 1.0) / (a + b + 2.0))
+                 front * beta_continued_fraction(a, b, x) / a
+               else
+                 1.0 - ((front * beta_continued_fraction(b, a, 1.0 - x)) / b)
+               end
+
+      result.clamp(0.0, 1.0)
+    end
+
+    def beta_continued_fraction(a, b, x)
+      tiny = 1e-30
+      qab = a + b
+      qap = a + 1.0
+      qam = a - 1.0
+
+      c = 1.0
+      d = 1.0 - ((qab * x) / qap)
+      d = tiny if d.abs < tiny
+      d = 1.0 / d
+      fraction = d
+
+      (1..200).each do |m|
+        m2 = 2 * m
+
+        numerator = (m * (b - m) * x) / ((qam + m2) * (a + m2))
+        d = 1.0 + (numerator * d)
+        d = tiny if d.abs < tiny
+        c = 1.0 + (numerator / c)
+        c = tiny if c.abs < tiny
+        d = 1.0 / d
+        fraction *= c * d
+
+        numerator = -((a + m) * (qab + m) * x) / ((a + m2) * (qap + m2))
+        d = 1.0 + (numerator * d)
+        d = tiny if d.abs < tiny
+        c = 1.0 + (numerator / c)
+        c = tiny if c.abs < tiny
+        d = 1.0 / d
+        delta = c * d
+        fraction *= delta
+
+        break if (delta - 1.0).abs < 1e-12
+      end
+
+      fraction
+    end
+  end
+end

data/lib/rubric_llm/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RubricLLM
-  VERSION = "0.4.0"
+  VERSION = "0.6.0.rc1"
 end

data/lib/rubric_llm.rb

@@ -16,10 +16,13 @@ require_relative "rubric_llm/metrics/factual_accuracy"
 require_relative "rubric_llm/result"
 require_relative "rubric_llm/evaluator"
 require_relative "rubric_llm/report"
+require_relative "rubric_llm/statistics"
 require_relative "rubric_llm/comparison"
 require_relative "rubric_llm/retrieval_result"
 
 module RubricLLM
+  DEFAULT_THRESHOLD = 0.8
+
   class << self
     def config
       @config ||= Config.new
@@ -140,7 +143,7 @@ module RubricLLM
     def apply_custom_prompt(config, custom_prompt)
       return config unless custom_prompt
 
-      Config.new(**config.to_h.compact, custom_prompt:)
+      Config.new(**config.to_h, custom_prompt:)
     end
 
     def normalize_sample(sample)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rubric_llm
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.6.0.rc1
 platform: ruby
 authors:
 - David Paluy
@@ -27,16 +27,16 @@ dependencies:
   name: ruby_llm
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '1.16'
+        version: 2.0.0.rc1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '1.16'
+        version: 2.0.0.rc1
 description: Provider-agnostic LLM evaluation with pluggable metrics, statistical
   A/B comparison, and test framework integration. Ragas for Ruby, powered by RubyLLM.
 email:
@@ -69,6 +69,7 @@ files:
 - lib/rubric_llm/result.rb
 - lib/rubric_llm/retrieval_result.rb
 - lib/rubric_llm/rspec.rb
+- lib/rubric_llm/statistics.rb
 - lib/rubric_llm/version.rb
 homepage: https://github.com/dpaluy/rubric_llm
 licenses: