rubric_llm 0.3.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4afa554e2e9ae10520d0cc605f55d59cdc6330c7ba65f38c4d3e933fb11519a4
-  data.tar.gz: e371eaaf0ab368c46d930e352434664bca19315bfc45865601686b0a5525ca5a
+  metadata.gz: e4b3be22ad7ef0d4af66ddba820a505c75c29f3bf47e1259bdee1ed28cf87ae4
+  data.tar.gz: 2250b2b15b76203042befc1a86805827161c0df49efaf8dc240c5c594d5166b3
 SHA512:
-  metadata.gz: f635136ea65b903555493b377a14d5902b2be949ef7bc77b036450ac25d8ead70429039c5f47ffdc36c2216470e39409ceb50f35c609df8f4e5648b64569fa38
-  data.tar.gz: a66697e131141d27ff0c406fdd60b6ef103c4c69cce3b85a3c5cf481cbdf55cc489983b10e94ffe7557ab0a637a8c09530e9cd0f317e16e092496862574fed1c
+  metadata.gz: 9e7e813f4bf58b17b70b6da5b42f0e77e962cc2862a1b5a62e59c5152ebc8de0c25058842c1c92344704e5fd249846a8662bd9937ad40ef068464c3520c3fee0
+  data.tar.gz: aa9f1156c49f63cef0f00a7c649b14a6b11d9bec8904962e5a25af88bafd70447c797d7bb1f69ddffd5b17f13dda2e5e702fed5aede15c4cd9ef59c871c1be52

data/CHANGELOG.md

@@ -5,6 +5,43 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.0] - 2026-08-23
+
+### Fixed
+
+- The RSpec `hallucinate` matcher no longer reports a hallucination when the judge returns no score. A failed judge call previously read as a positive match and failed the build with a false verdict
+- `Comparison` pairs samples by `sample[:question]` instead of array position. Two reports built from the same dataset in a different order produced an invalid paired t-test. Questions present in only one report are dropped with a warning
+- `Judge#call` retries only transient failures (`RubyLLM::RateLimitError`, `ServerError`, `ServiceUnavailableError`, `OverloadedError`). A bad API key, an exhausted quota, an over-long prompt, or a judge contract violation now fails on the first attempt instead of sleeping through the full retry schedule. Transport timeouts and connection resets are left to RubyLLM's connection, which already retries them
+- `ContextPrecision` and `ContextRecall` drop blank and whitespace-only context chunks and return a `No context provided` error when nothing usable remains, matching `Faithfulness`
+- `Statistics.two_tailed_p` rescues only `Math::DomainError`, `ZeroDivisionError`, and `FloatDomainError`. It previously swallowed every `StandardError` and returned 1.0
+- A paired t-test over a constant difference returns 1.0 instead of a spurious near-zero p-value caused by float error in the variance
+- The RSpec `hallucinate` matcher renders a missing score as `nil` in its failure message instead of an empty string
+
+### Removed
+
+- The `Comparison` constructor no longer warns about reports of different sizes. Different sizes are fine when the questions match, and equal sizes can still drop every pair when they do not. The pairing warnings report what is actually dropped
+
+### Added
+
+- Holm-Bonferroni correction across metrics in `Comparison`. Each result carries `:p_value_adjusted` next to the raw `:p_value`, `#summary` prints a `p-adj` column, and `#significant_improvements` / `#significant_regressions` test the adjusted value. Six independent tests at alpha 0.05 gave a family-wise false-positive rate near 26%
+- `RubricLLM::Statistics`, a module holding the paired t-test, the two-tailed p-value, the Holm adjustment, and the incomplete beta function. No LLM calls, no state
+- `Metrics::Base.normalize_context`, the single definition of usable context
+
+### Changed
+
+- The Minitest `assert_faithful` and `refute_hallucination` helpers and the RSpec `be_faithful` and `hallucinate` matchers raise `ArgumentError` on an empty or blank context instead of returning a verdict that no judge produced
+
+## [0.4.0] - 2026-07-11
+
+### Added
+
+- `RubricLLM.evaluate_batch` validates every sample upfront (must be a Hash with non-nil `:question` and `:answer`, string or symbol keys) and raises `ArgumentError` with the offending index before any LLM call, so sequential and concurrent modes fail identically and without API spend
+
+### Changed
+
+- Add `csv` as a runtime dependency; `csv` moved from a default gem to a bundled gem in Ruby 3.4, so consumers previously hit a `LoadError` on `Report#export_csv`
+- Require `ruby_llm ~> 1.16`
+
 ## [0.3.0] - 2026-07-11
 
 ### Changed

data/README.md

@@ -6,7 +6,7 @@ Lightweight LLM evaluation framework for Ruby, inspired by [DeepEval](https://gi
 [![CI](https://github.com/dpaluy/rubric_llm/actions/workflows/ci.yml/badge.svg)](https://github.com/dpaluy/rubric_llm/actions/workflows/ci.yml)
 [![Wiki](https://img.shields.io/badge/docs-wiki-blue)](https://github.com/dpaluy/rubric_llm/wiki)
 
-Provider-agnostic evaluation with pluggable metrics, statistical A/B comparison, and test framework integration — no Rails, no ActiveRecord, no UI. Works anywhere Ruby runs.
+Provider-agnostic evaluation with pluggable metrics, statistical A/B comparison, and test framework integration: no Rails, no ActiveRecord, no UI. Works anywhere Ruby runs.
 
 ## Installation
 
@@ -116,14 +116,14 @@ These metrics use a judge LLM to evaluate quality. Each sends a structured promp
 | **Faithfulness** | Is every claim in the answer supported by the context? | `context` |
 
 ```ruby
-# Only context — gets faithfulness, relevance, context_precision
+# Only context (gets faithfulness, relevance, context_precision)
 result = RubricLLM.evaluate(
   question: "How does photosynthesis work?",
   answer: "Plants convert sunlight into energy.",
   context: ["Photosynthesis is the process by which plants convert light energy into chemical energy."]
 )
 
-# With ground truth — gets all metrics
+# With ground truth (gets all metrics)
 result = RubricLLM.evaluate(
   question: "How does photosynthesis work?",
   answer: "Plants convert sunlight into energy.",
@@ -155,7 +155,7 @@ result.scores[:tone_metric]  # => 0.85
 
 ### Retrieval Metrics
 
-Pure math — no LLM calls, no API key needed.
+Pure math, no LLM calls, no API key needed.
 
 ```ruby
 result = RubricLLM.evaluate_retrieval(
@@ -164,9 +164,9 @@ result = RubricLLM.evaluate_retrieval(
 )
 
 result.precision_at_k(3)  # => 0.67
-result.recall_at_k(3)     # => 0.90
-result.mrr                # => 0.90
-result.ndcg               # => 0.86
+result.recall_at_k(3)     # => 1.0
+result.mrr                # => 1.0
+result.ndcg               # => 0.92
 result.hit_rate           # => 1.0
 ```
 
@@ -217,12 +217,14 @@ comparison = RubricLLM.compare(report_a, report_b)
 
 puts comparison.summary
 # A/B Comparison
-# ======================================================================
-# Metric                      A        B    Delta    p-value  Sig
-# ----------------------------------------------------------------------
-# faithfulness                0.880    0.920   +0.040     0.0230    *
-# relevance                   0.850    0.860   +0.010     0.4210
-# correctness                 0.910    0.940   +0.030     0.0089   **
+# ================================================================================
+# Metric                      A        B    Delta    p-value      p-adj  Sig
+# --------------------------------------------------------------------------------
+# faithfulness            0.880    0.920   +0.040     0.0023     0.0068   **
+# relevance               0.850    0.860   +0.010     0.3081     0.3081
+# correctness             0.910    0.940   +0.030     0.0240     0.0480    *
+#
+# p-adj: Holm-Bonferroni adjusted across 3 metrics. Significance uses p-adj.
 
 comparison.significant_improvements   # => [:faithfulness, :correctness]
 comparison.significant_regressions    # => []
@@ -230,6 +232,16 @@ comparison.significant_regressions    # => []
 
 Significance markers: `*` (p < 0.05), `**` (p < 0.01), `***` (p < 0.001)
 
+### Pairing
+
+A paired t-test needs the same sample on both sides. The comparison pairs results by `sample[:question]`, not by position, so a reordered dataset still gives a valid test. Questions present in only one report are dropped with a warning. If a question repeats an uneven number of times across the two reports, the extra occurrences are dropped with a warning.
+
+`evaluate_batch` requires a `:question` on every sample, so reports it produces always pair by identity. Hand-built reports whose results carry no `sample[:question]` fall back to position pairing and warn.
+
+### Multiple comparisons
+
+Every metric gets its own t-test. Six tests at alpha 0.05 give a family-wise false-positive rate near 26%, so each raw `p_value` is corrected with the Holm-Bonferroni step-down method and reported as `p_value_adjusted`. The significance markers and both `significant_*` methods read the adjusted value. The raw value stays in the result for reference.
+
 For the statistical reasoning behind paired t-tests and how to read these p-values, see [Understanding A/B Comparison](https://github.com/dpaluy/rubric_llm/wiki/Understanding-A-B-Comparison) on the wiki.
 
 ## Test Integration
@@ -298,7 +310,7 @@ rescue RubricLLM::Error => e
 end
 ```
 
-Individual metric failures are handled gracefully — a failed metric returns `nil` for the score and includes the error in details:
+Individual metric failures are handled gracefully: a failed metric returns `nil` for the score and includes the error in details:
 
 ```ruby
 result = RubricLLM.evaluate(question: "q", answer: "a")
@@ -317,12 +329,12 @@ bundle exec rubocop
 
 ## Limitations
 
-RubricLLM uses LLM-as-Judge — an LLM scores another LLM's output. This is the industry-standard approach (used by Ragas, DeepEval, ARES), but it means the judge shares the same class of failure modes as the system being evaluated. If the judge hallucinates that an answer is faithful, you get a false positive.
+RubricLLM uses LLM-as-Judge: an LLM scores another LLM's output. This is the industry-standard approach (used by Ragas, DeepEval, ARES), but it means the judge shares the same class of failure modes as the system being evaluated. If the judge hallucinates that an answer is faithful, you get a false positive.
 
 Mitigations built into the framework:
 
 - **Cross-model judging.** Configure a different model as judge than the one being evaluated. Don't let gpt-5.5 grade gpt-5.5.
-- **Retrieval metrics are pure math.** `precision_at_k`, `recall_at_k`, `mrr`, `ndcg` — no LLM involved, no judge bias. See [Why Retrieval Metrics Are Pure Math](https://github.com/dpaluy/rubric_llm/wiki/Why-Retrieval-Metrics-Are-Pure-Math).
+- **Retrieval metrics are pure math.** `precision_at_k`, `recall_at_k`, `mrr`, `ndcg` (no LLM involved, no judge bias). See [Why Retrieval Metrics Are Pure Math](https://github.com/dpaluy/rubric_llm/wiki/Why-Retrieval-Metrics-Are-Pure-Math).
 - **Custom non-LLM metrics.** Subclass `Metrics::Base` with regex checks, embedding similarity, or any deterministic logic.
 - **Statistical comparison.** A/B testing with paired t-tests surfaces systematic judge bias across runs.
 
@@ -338,7 +350,7 @@ Ruby has two LLM evaluation options today. Neither fits most use cases:
 | **LLM access** | Raw HTTP (OpenAI/Anthropic only) | You implement it | RubyLLM (any provider) |
 | **Rails required?** | No | Yes (engine + 6 migrations) | No |
 | **ActiveRecord?** | No | Yes | No |
-| **A/B comparison** | Basic | No | Paired t-test with p-values |
+| **A/B comparison** | Basic | No | Paired t-test with Holm-corrected p-values |
 | **Test assertions** | Minitest + RSpec | No | Minitest + RSpec |
 | **Pluggable metrics** | No (fixed set) | Yes | Yes |
 | **Retrieval metrics** | Yes | No | Yes |
@@ -347,8 +359,8 @@ Ruby has two LLM evaluation options today. Neither fits most use cases:
 
 Deep dives live in the [project wiki](https://github.com/dpaluy/rubric_llm/wiki):
 
-- [Understanding A/B Comparison](https://github.com/dpaluy/rubric_llm/wiki/Understanding-A-B-Comparison) — what paired t-tests and p-values mean for model comparison
-- [Why Retrieval Metrics Are Pure Math](https://github.com/dpaluy/rubric_llm/wiki/Why-Retrieval-Metrics-Are-Pure-Math) — why `precision_at_k`, `recall_at_k`, `mrr`, `ndcg`, and `hit_rate` are deterministic and bias-free
+- [Understanding A/B Comparison](https://github.com/dpaluy/rubric_llm/wiki/Understanding-A-B-Comparison): what paired t-tests and p-values mean for model comparison
+- [Why Retrieval Metrics Are Pure Math](https://github.com/dpaluy/rubric_llm/wiki/Why-Retrieval-Metrics-Are-Pure-Math): why `precision_at_k`, `recall_at_k`, `mrr`, `ndcg`, and `hit_rate` are deterministic and bias-free
 
 ## Requirements
 

data/lib/rubric_llm/comparison.rb

@@ -7,12 +7,6 @@ module RubricLLM
     def initialize(report_a, report_b)
       @report_a = report_a
       @report_b = report_b
-
-      return if report_a.results.size == report_b.results.size
-
-      warn "[RubricLLM] Comparison reports have different sizes " \
-           "(#{report_a.results.size} vs #{report_b.results.size}). " \
-           "Unmatched pairs will be dropped."
     end
 
     def results
@@ -21,24 +15,27 @@ module RubricLLM
 
     def summary
       lines = ["A/B Comparison"]
-      lines << ("=" * 70)
-      lines << "Metric                      A        B    Delta    p-value  Sig"
-      lines << ("-" * 70)
+      lines << ("=" * 80)
+      lines << "Metric                      A        B    Delta    p-value      p-adj  Sig"
+      lines << ("-" * 80)
 
       results.each do |metric, r|
-        lines << format("%-20s %8.3f %8.3f %+8.3f %10.4f %4s",
-                        metric, r[:mean_a], r[:mean_b], r[:delta], r[:p_value], r[:significance])
+        lines << format("%-20s %8.3f %8.3f %+8.3f %10.4f %10.4f %4s",
+                        metric, r[:mean_a], r[:mean_b], r[:delta], r[:p_value], r[:p_value_adjusted],
+                        r[:significance])
       end
 
+      lines << ""
+      lines << "p-adj: Holm-Bonferroni adjusted across #{results.size} metrics. Significance uses p-adj."
       lines.join("\n")
     end
 
     def significant_improvements(alpha: 0.05)
-      results.select { |_, r| r[:p_value] < alpha && r[:delta].positive? }.keys
+      results.select { |_, r| r[:p_value_adjusted] < alpha && r[:delta].positive? }.keys
     end
 
     def significant_regressions(alpha: 0.05)
-      results.select { |_, r| r[:p_value] < alpha && r[:delta].negative? }.keys
+      results.select { |_, r| r[:p_value_adjusted] < alpha && r[:delta].negative? }.keys
     end
 
     private
@@ -46,110 +43,98 @@ module RubricLLM
     def compute_results
       metrics = (report_a.metric_stats.keys | report_b.metric_stats.keys)
 
-      metrics.each_with_object({}) do |metric, hash|
-        paired_scores = report_a.scores_for(metric)
-                                .zip(report_b.scores_for(metric))
-                                .select { |score_a, score_b| !score_a.nil? && !score_b.nil? }
+      raw = metrics.each_with_object({}) do |metric, hash|
+        stats = metric_stats(metric)
+        hash[metric] = stats if stats
+      end
+
+      apply_holm_correction(raw)
+    end
 
-        next if paired_scores.empty?
+    def metric_stats(metric)
+      paired_scores = paired_results
+                      .map { |result_a, result_b| [result_a.scores[metric], result_b.scores[metric]] }
+                      .reject { |score_a, score_b| score_a.nil? || score_b.nil? }
 
-        scores_a, scores_b = paired_scores.transpose
+      return nil if paired_scores.empty?
 
-        mean_a = scores_a.sum / scores_a.size.to_f
-        mean_b = scores_b.sum / scores_b.size.to_f
-        delta = mean_b - mean_a
-        p_value = paired_t_test(scores_a, scores_b)
+      scores_a, scores_b = paired_scores.transpose
+      mean_a = scores_a.sum / scores_a.size.to_f
+      mean_b = scores_b.sum / scores_b.size.to_f
 
-        hash[metric] = {
-          mean_a:,
-          mean_b:,
-          delta:,
-          p_value:,
-          significance: significance_marker(p_value)
-        }
-      end
+      {
+        mean_a:,
+        mean_b:,
+        delta: mean_b - mean_a,
+        p_value: Statistics.paired_t_test(scores_a, scores_b)
+      }
     end
 
-    def paired_t_test(a, b)
-      n = [a.size, b.size].min
-      return 1.0 if n < 2
+    # A paired t-test requires the same sample on both sides. Pair by question
+    # instead of array position so a reordered dataset stays valid.
+    def paired_results
+      @paired_results ||= build_pairs
+    end
+
+    def build_pairs
+      groups_a = report_a.results.group_by { |result| pair_key(result) }
+      groups_b = report_b.results.group_by { |result| pair_key(result) }
 
-      diffs = a.first(n).zip(b.first(n)).map { |x, y| y - x }
-      mean_d = diffs.sum / n.to_f
-      var_d = diffs.sum { |d| (d - mean_d)**2 } / (n - 1).to_f
-      se = Math.sqrt(var_d / n)
+      warn_unkeyed(groups_a[nil].to_a.size + groups_b[nil].to_a.size)
 
-      return 1.0 if se.zero?
+      matched = groups_a.keys & groups_b.keys
+      warn_unmatched(((groups_a.keys | groups_b.keys) - matched).compact)
 
-      t = mean_d / se
-      df = n - 1
+      matched.flat_map do |key|
+        list_a = groups_a[key]
+        list_b = groups_b[key]
+        warn_uneven(key, list_a.size, list_b.size) unless list_a.size == list_b.size
 
-      # Two-tailed p-value approximation using Student's t-distribution
-      two_tailed_p(t.abs, df)
+        size = [list_a.size, list_b.size].min
+        list_a.first(size).zip(list_b.first(size))
+      end
     end
 
-    # Two-tailed p-value for Student's t-distribution.
-    # p = I_x(df/2, 1/2) where x = df/(df + t²)
-    def two_tailed_p(t, df)
-      x = df / (df + (t**2))
-      regularized_beta(x, df / 2.0, 0.5)
-    rescue StandardError
-      1.0
+    def pair_key(result)
+      sample = result.sample
+      sample.is_a?(Hash) ? sample[:question] : nil
     end
 
-    # Regularized incomplete beta function via continued fraction (Lentz's method).
-    def regularized_beta(x, a, b)
-      return 0.0 if x <= 0.0
-      return 1.0 if x >= 1.0
+    # Results with no sample[:question] share one bucket and pair by position,
+    # which is the behaviour identity pairing exists to replace. Say so.
+    def warn_unkeyed(count)
+      return if count.zero?
 
-      ln_beta = Math.lgamma(a + b)[0] - Math.lgamma(a)[0] - Math.lgamma(b)[0]
-      front = Math.exp(ln_beta + (a * Math.log(x)) + (b * Math.log(1.0 - x)))
+      warn "[RubricLLM] #{count} result(s) have no sample[:question]. They share one bucket and " \
+           "pair by position. Give every sample a :question to pair them reliably."
+    end
+
+    def warn_unmatched(keys)
+      return if keys.empty?
 
-      result = if x < ((a + 1.0) / (a + b + 2.0))
-                 front * beta_continued_fraction(a, b, x) / a
-               else
-                 1.0 - ((front * beta_continued_fraction(b, a, 1.0 - x)) / b)
-               end
+      warn "[RubricLLM] Comparison dropped #{keys.size} question(s) present in only one report. " \
+           "Paired tests need the same questions on both sides."
+    end
 
-      result.clamp(0.0, 1.0)
+    def warn_uneven(key, size_a, size_b)
+      warn "[RubricLLM] Question #{key.inspect} appears #{size_a} time(s) in report A and " \
+           "#{size_b} time(s) in report B. Extra occurrences are dropped."
     end
 
-    def beta_continued_fraction(a, b, x)
-      tiny = 1e-30
-      qab = a + b
-      qap = a + 1.0
-      qam = a - 1.0
-
-      c = 1.0
-      d = 1.0 - ((qab * x) / qap)
-      d = tiny if d.abs < tiny
-      d = 1.0 / d
-      fraction = d
-
-      (1..200).each do |m|
-        m2 = 2 * m
-
-        numerator = (m * (b - m) * x) / ((qam + m2) * (a + m2))
-        d = 1.0 + (numerator * d)
-        d = tiny if d.abs < tiny
-        c = 1.0 + (numerator / c)
-        c = tiny if c.abs < tiny
-        d = 1.0 / d
-        fraction *= c * d
-
-        numerator = -((a + m) * (qab + m) * x) / ((a + m2) * (qap + m2))
-        d = 1.0 + (numerator * d)
-        d = tiny if d.abs < tiny
-        c = 1.0 + (numerator / c)
-        c = tiny if c.abs < tiny
-        d = 1.0 / d
-        delta = c * d
-        fraction *= delta
-
-        break if (delta - 1.0).abs < 1e-12
+    # One t-test per metric inflates the family-wise error rate, so adjust
+    # before calling any metric significant.
+    def apply_holm_correction(results)
+      metrics = results.keys
+      adjusted = Statistics.holm_adjust(metrics.map { |metric| results[metric][:p_value] })
+
+      metrics.each_with_index do |metric, index|
+        results[metric] = results[metric].merge(
+          p_value_adjusted: adjusted[index],
+          significance: significance_marker(adjusted[index])
+        )
       end
 
-      fraction
+      results
     end
 
     def significance_marker(p)

data/lib/rubric_llm/judge.rb

@@ -4,6 +4,20 @@ require "json"
 
 module RubricLLM
   class Judge
+    # Failures worth retrying: the same request may succeed later.
+    # Everything else (bad key, no credit, malformed request, prompt too long,
+    # contract violations in the judge response) fails on the first attempt.
+    #
+    # Transport failures are absent on purpose. RubyLLM's connection already
+    # retries timeouts and connection resets, so by the time one reaches us it
+    # has been tried several times and is not worth another round.
+    TRANSIENT_ERRORS = [
+      RubyLLM::RateLimitError,
+      RubyLLM::ServerError,
+      RubyLLM::ServiceUnavailableError,
+      RubyLLM::OverloadedError
+    ].freeze
+
     METRIC_RESPONSE_SCHEMA = {
       name: "rubric_llm_metric_response",
       strict: false,
@@ -46,11 +60,7 @@ module RubricLLM
         content = response.content
         validate_response!(content.is_a?(Hash) ? content : parse_json(content))
       rescue StandardError => e
-        if attempts > config.max_retries
-          raise e if e.is_a?(JudgeError)
-
-          raise JudgeError, "Judge call failed: #{e.message}"
-        end
+        raise wrap_error(e) unless transient?(e) && attempts <= config.max_retries
 
         sleep(config.retry_base_delay * (2**(attempts - 1)))
         retry
@@ -79,6 +89,16 @@ module RubricLLM
 
     private
 
+    def transient?(error)
+      TRANSIENT_ERRORS.any? { |klass| error.is_a?(klass) }
+    end
+
+    def wrap_error(error)
+      return error if error.is_a?(JudgeError)
+
+      JudgeError.new("Judge call failed: #{error.message}")
+    end
+
     def apply_response_schema(chat)
       return chat unless chat.respond_to?(:with_schema)
       return chat unless structured_output_supported?(chat)

data/lib/rubric_llm/metrics/base.rb

@@ -3,6 +3,20 @@
 module RubricLLM
   module Metrics
     class Base
+      # Single source of truth for what counts as usable context.
+      # Blank and whitespace-only chunks are dropped.
+      def self.normalize_context(context)
+        Array(context).map { |chunk| chunk.to_s.strip }.reject(&:empty?)
+      end
+
+      # An empty context cannot produce a faithfulness score. Reject it as a caller
+      # error instead of letting a nil score read as a quality verdict.
+      def self.require_context!(context)
+        return unless normalize_context(context).empty?
+
+        raise ArgumentError, "context must contain at least one non-empty entry"
+      end
+
       attr_reader :judge
 
       def initialize(judge:)

data/lib/rubric_llm/metrics/context_precision.rb

@@ -16,13 +16,14 @@ module RubricLLM
       PROMPT
 
       def call(question:, context: [], **)
-        return { score: nil, details: { error: "No context provided" } } if Array(context).empty?
+        context_chunks = Base.normalize_context(context)
+        return { score: nil, details: { error: "No context provided" } } if context_chunks.empty?
 
         user_prompt = <<~PROMPT
           Question: #{question}
 
           Contexts:
-          #{Array(context).each_with_index.map { |c, i| "#{i + 1}. #{c}" }.join("\n")}
+          #{context_chunks.each_with_index.map { |c, i| "#{i + 1}. #{c}" }.join("\n")}
 
           Evaluate how relevant each context is to the question.
         PROMPT

data/lib/rubric_llm/metrics/context_recall.rb

@@ -17,11 +17,13 @@ module RubricLLM
 
       def call(context: [], ground_truth: nil, **)
         return { score: nil, details: { error: "No ground truth provided" } } if ground_truth.nil?
-        return { score: nil, details: { error: "No context provided" } } if Array(context).empty?
+
+        context_chunks = Base.normalize_context(context)
+        return { score: nil, details: { error: "No context provided" } } if context_chunks.empty?
 
         user_prompt = <<~PROMPT
           Contexts:
-          #{Array(context).each_with_index.map { |c, i| "#{i + 1}. #{c}" }.join("\n")}
+          #{context_chunks.each_with_index.map { |c, i| "#{i + 1}. #{c}" }.join("\n")}
 
           Ground Truth: #{ground_truth}
 

data/lib/rubric_llm/metrics/faithfulness.rb

@@ -16,7 +16,7 @@ module RubricLLM
       PROMPT
 
       def call(question:, answer:, context: [], **)
-        context_chunks = Array(context).map { |chunk| chunk.to_s.strip }.reject(&:empty?)
+        context_chunks = Base.normalize_context(context)
         return { score: nil, details: { error: "No context provided" } } if context_chunks.empty?
 
         user_prompt = <<~PROMPT

data/lib/rubric_llm/minitest.rb

@@ -4,7 +4,8 @@ require "rubric_llm"
 
 module RubricLLM
   module Assertions
-    def assert_faithful(answer, context, question: "", threshold: 0.8, config: RubricLLM.config)
+    def assert_faithful(answer, context, question: "", threshold: DEFAULT_THRESHOLD, config: RubricLLM.config)
+      require_context!(context)
       result = evaluate_metric(Metrics::Faithfulness, question:, answer:, context:, config:)
       score = result[:score]
 
@@ -12,7 +13,7 @@ module RubricLLM
              "Expected faithfulness >= #{threshold}, got #{score || "nil"}.#{failure_details(result)}"
     end
 
-    def assert_relevant(question, answer, threshold: 0.8, config: RubricLLM.config)
+    def assert_relevant(question, answer, threshold: DEFAULT_THRESHOLD, config: RubricLLM.config)
       result = evaluate_metric(Metrics::Relevance, question:, answer:, config:)
       score = result[:score]
 
@@ -20,7 +21,7 @@ module RubricLLM
              "Expected relevance >= #{threshold}, got #{score || "nil"}.#{failure_details(result)}"
     end
 
-    def assert_correct(answer, ground_truth, question: "", threshold: 0.8, config: RubricLLM.config)
+    def assert_correct(answer, ground_truth, question: "", threshold: DEFAULT_THRESHOLD, config: RubricLLM.config)
       result = evaluate_metric(Metrics::Correctness, question:, answer:, ground_truth:, config:)
       score = result[:score]
 
@@ -28,7 +29,8 @@ module RubricLLM
              "Expected correctness >= #{threshold}, got #{score || "nil"}.#{failure_details(result)}"
     end
 
-    def refute_hallucination(answer, context, question: "", threshold: 0.8, config: RubricLLM.config)
+    def refute_hallucination(answer, context, question: "", threshold: DEFAULT_THRESHOLD, config: RubricLLM.config)
+      require_context!(context)
       result = evaluate_metric(Metrics::Faithfulness, question:, answer:, context:, config:)
       score = result[:score]
 
@@ -38,6 +40,10 @@ module RubricLLM
 
     private
 
+    def require_context!(context)
+      Metrics::Base.require_context!(context)
+    end
+
     def evaluate_metric(metric_class, config:, **)
       judge = Judge.new(config:)
       metric = metric_class.new(judge:)

data/lib/rubric_llm/report.rb

@@ -19,7 +19,7 @@ module RubricLLM
       results.sort_by { |r| r.overall || -Float::INFINITY }.first(n)
     end
 
-    def failures(threshold: 0.8)
+    def failures(threshold: DEFAULT_THRESHOLD)
       results.reject { |r| r.pass?(threshold:) }
     end
 
@@ -42,7 +42,7 @@ module RubricLLM
     end
 
     def export_csv(path)
-      require "csv" # optional dependency — add `gem "csv"` to your Gemfile if missing
+      require "csv"
       metrics = all_metric_names
       CSV.open(path, "w") do |csv|
         csv << ["question", "answer", "overall", *metrics]

data/lib/rubric_llm/result.rb

@@ -17,7 +17,7 @@ module RubricLLM
       valid.sum / valid.size.to_f
     end
 
-    def pass?(threshold: 0.8)
+    def pass?(threshold: DEFAULT_THRESHOLD)
       return false unless valid?
 
       score = overall

data/lib/rubric_llm/rspec.rb

@@ -24,7 +24,7 @@ module RubricLLM
       attr_reader :threshold, :config, :result
 
       def initialize
-        @threshold = 0.8
+        @threshold = DEFAULT_THRESHOLD
         @config = RubricLLM.config
       end
 
@@ -40,6 +40,10 @@ module RubricLLM
 
       private
 
+      def require_context!(context)
+        Metrics::Base.require_context!(context)
+      end
+
       def evaluate(metric_class, **)
         judge = Judge.new(config:)
         metric = metric_class.new(judge:)
@@ -51,6 +55,7 @@ module RubricLLM
     class FaithfulnessMatcher < BaseMatcher
       def initialize(context, question: nil)
         super()
+        require_context!(context)
         @context = context
         @question = question
       end
@@ -113,17 +118,19 @@ module RubricLLM
     class HallucinationMatcher < BaseMatcher
       def initialize(context, question: nil)
         super()
+        require_context!(context)
         @context = context
         @question = question
       end
 
+      # Fail closed: a missing score is not evidence of a hallucination.
       def matches?(answer)
         score = evaluate(Metrics::Faithfulness, question: @question || "", answer:, context: @context)
-        score.nil? || score < threshold
+        !score.nil? && score < threshold
       end
 
       def failure_message
-        "expected hallucination (faithfulness < #{threshold}), got #{result[:score]}"
+        "expected hallucination (faithfulness < #{threshold}), got #{result[:score] || "nil"}"
       end
 
       def failure_message_when_negated

data/lib/rubric_llm/statistics.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+module RubricLLM
+  # Pure statistical helpers. No LLM calls, no state.
+  module Statistics
+    module_function
+
+    # Two-tailed paired t-test. Both arrays must already be paired and equal length.
+    # Returns 1.0 when there is nothing to test.
+    def paired_t_test(scores_a, scores_b)
+      n = scores_a.size
+      return 1.0 if n < 2
+
+      diffs = scores_a.zip(scores_b).map { |x, y| y - x }
+      mean_d = diffs.sum / n.to_f
+      var_d = diffs.sum { |d| (d - mean_d)**2 } / (n - 1).to_f
+      se = Math.sqrt(var_d / n)
+
+      # A constant shift has no spread to test. Compare against the scale of the
+      # data rather than exact zero, or float error turns it into a huge t value.
+      return 1.0 if se <= Float::EPSILON * [mean_d.abs, 1.0].max
+
+      two_tailed_p((mean_d / se).abs, n - 1)
+    end
+
+    # Two-tailed p-value for Student's t-distribution.
+    # p = I_x(df/2, 1/2) where x = df/(df + t²)
+    def two_tailed_p(t, df)
+      x = df / (df + (t**2))
+      regularized_beta(x, df / 2.0, 0.5)
+    rescue Math::DomainError, ZeroDivisionError, FloatDomainError
+      1.0
+    end
+
+    # Holm-Bonferroni step-down adjustment.
+    # Takes p-values in any order, returns adjusted values in the same order.
+    def holm_adjust(p_values)
+      count = p_values.size
+      return p_values.dup if count.zero?
+
+      adjusted = Array.new(count)
+      running_max = 0.0
+
+      p_values.each_with_index.sort_by(&:first).each_with_index do |(p_value, position), rank|
+        running_max = ((count - rank) * p_value).clamp(running_max, 1.0)
+        adjusted[position] = running_max
+      end
+
+      adjusted
+    end
+
+    # Regularized incomplete beta function via continued fraction (Lentz's method).
+    def regularized_beta(x, a, b)
+      return 0.0 if x <= 0.0
+      return 1.0 if x >= 1.0
+
+      ln_beta = Math.lgamma(a + b)[0] - Math.lgamma(a)[0] - Math.lgamma(b)[0]
+      front = Math.exp(ln_beta + (a * Math.log(x)) + (b * Math.log(1.0 - x)))
+
+      result = if x < ((a + 1.0) / (a + b + 2.0))
+                 front * beta_continued_fraction(a, b, x) / a
+               else
+                 1.0 - ((front * beta_continued_fraction(b, a, 1.0 - x)) / b)
+               end
+
+      result.clamp(0.0, 1.0)
+    end
+
+    def beta_continued_fraction(a, b, x)
+      tiny = 1e-30
+      qab = a + b
+      qap = a + 1.0
+      qam = a - 1.0
+
+      c = 1.0
+      d = 1.0 - ((qab * x) / qap)
+      d = tiny if d.abs < tiny
+      d = 1.0 / d
+      fraction = d
+
+      (1..200).each do |m|
+        m2 = 2 * m
+
+        numerator = (m * (b - m) * x) / ((qam + m2) * (a + m2))
+        d = 1.0 + (numerator * d)
+        d = tiny if d.abs < tiny
+        c = 1.0 + (numerator / c)
+        c = tiny if c.abs < tiny
+        d = 1.0 / d
+        fraction *= c * d
+
+        numerator = -((a + m) * (qab + m) * x) / ((a + m2) * (qap + m2))
+        d = 1.0 + (numerator * d)
+        d = tiny if d.abs < tiny
+        c = 1.0 + (numerator / c)
+        c = tiny if c.abs < tiny
+        d = 1.0 / d
+        delta = c * d
+        fraction *= delta
+
+        break if (delta - 1.0).abs < 1e-12
+      end
+
+      fraction
+    end
+  end
+end

data/lib/rubric_llm/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RubricLLM
-  VERSION = "0.3.0"
+  VERSION = "0.5.0"
 end

data/lib/rubric_llm.rb

@@ -16,10 +16,13 @@ require_relative "rubric_llm/metrics/factual_accuracy"
 require_relative "rubric_llm/result"
 require_relative "rubric_llm/evaluator"
 require_relative "rubric_llm/report"
+require_relative "rubric_llm/statistics"
 require_relative "rubric_llm/comparison"
 require_relative "rubric_llm/retrieval_result"
 
 module RubricLLM
+  DEFAULT_THRESHOLD = 0.8
+
   class << self
     def config
       @config ||= Config.new
@@ -56,6 +59,7 @@ module RubricLLM
     #   report = RubricLLM.evaluate_batch(dataset)
     #   report = RubricLLM.evaluate_batch(dataset, concurrency: 4)
     def evaluate_batch(dataset, metrics: nil, config: self.config, custom_prompt: nil, concurrency: nil)
+      validate_dataset!(dataset)
       config = apply_custom_prompt(config, custom_prompt)
       pool_size = concurrency || config.concurrency
       start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
@@ -87,6 +91,22 @@ module RubricLLM
 
     private
 
+    def validate_dataset!(dataset)
+      dataset.each_with_index do |sample, index|
+        raise ArgumentError, "sample at index #{index} is not a Hash" unless sample.is_a?(Hash)
+
+        question_present = sample.key?(:question) || sample.key?("question")
+        answer_present = sample.key?(:answer) || sample.key?("answer")
+        raise ArgumentError, "sample at index #{index} is missing :question" unless question_present
+        raise ArgumentError, "sample at index #{index} is missing :answer" unless answer_present
+
+        question_provided = !sample[:question].nil? || !sample["question"].nil?
+        answer_provided = !sample[:answer].nil? || !sample["answer"].nil?
+        raise ArgumentError, "sample at index #{index} has nil :question" unless question_provided
+        raise ArgumentError, "sample at index #{index} has nil :answer" unless answer_provided
+      end
+    end
+
     def evaluate_sample(evaluator, sample)
       sample = normalize_sample(sample)
       evaluator.call(

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rubric_llm
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.5.0
 platform: ruby
 authors:
 - David Paluy
@@ -9,20 +9,34 @@ bindir: bin
 cert_chain: []
 date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: csv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: ruby_llm
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.13'
+        version: '1.16'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.13'
+        version: '1.16'
 description: Provider-agnostic LLM evaluation with pluggable metrics, statistical
   A/B comparison, and test framework integration. Ragas for Ruby, powered by RubyLLM.
 email:
@@ -55,6 +69,7 @@ files:
 - lib/rubric_llm/result.rb
 - lib/rubric_llm/retrieval_result.rb
 - lib/rubric_llm/rspec.rb
+- lib/rubric_llm/statistics.rb
 - lib/rubric_llm/version.rb
 homepage: https://github.com/dpaluy/rubric_llm
 licenses: