rubric_llm 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: a549a52da8585cfbdf8287315548389de6e93b1b0adca9b8fb32fedde3d966e5
+  data.tar.gz: 3de825ff22b9c4b3dc091cc17ef089e504f53238857d30393b1951d55884c38d
+SHA512:
+  metadata.gz: 004aff76b7b92d2266d75ec54b897d240cef7a86f424cbca8d1904d7e16eeb1821900c722e6df4f20be511936f90993e73b4e3aebd436e9d82f24ee558f12996
+  data.tar.gz: eacc3c28cb3d3504323dbbb491c3e9b8af6625520308cb319cab2d5164868ac56a52c3daed5a603e496c092e53a2c1a7b1f6f32754b9dc1c230ec0aebca30be9

data/CHANGELOG.md

@@ -0,0 +1,23 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-03-24
+
+### Added
+
+- LLM-as-Judge evaluation via RubyLLM (provider-agnostic)
+- Built-in metrics: Faithfulness, Relevance, Correctness, ContextPrecision, ContextRecall, FactualAccuracy
+- Pluggable metric interface (`RubricLLM::Metrics::Base`)
+- Single-sample evaluation (`RubricLLM.evaluate`)
+- Batch evaluation with reports (`RubricLLM.evaluate_batch`)
+- A/B model comparison with paired t-tests (`RubricLLM.compare`)
+- Retrieval metrics without LLM calls (`RubricLLM.evaluate_retrieval`)
+- Minitest assertions (`RubricLLM::Assertions`)
+- RSpec matchers (`RubricLLM::RSpecMatchers`)
+- CSV and JSON export for reports

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 David Paluy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,352 @@
+# RubricLLM
+
+Lightweight LLM evaluation framework for Ruby, inspired by [Ragas](https://github.com/vibrantlabsai/ragas), powered by [RubyLLM](https://github.com/crmne/ruby_llm).
+
+[![Gem Version](https://badge.fury.io/rb/rubric_llm.svg)](https://badge.fury.io/rb/rubric_llm)
+[![CI](https://github.com/dpaluy/rubric_llm/actions/workflows/ci.yml/badge.svg)](https://github.com/dpaluy/rubric_llm/actions/workflows/ci.yml)
+
+Provider-agnostic evaluation with pluggable metrics, statistical A/B comparison, and test framework integration — no Rails, no ActiveRecord, no UI. Works anywhere Ruby runs.
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "rubric_llm"
+```
+
+Or install directly:
+
+```bash
+gem install rubric_llm
+```
+
+## Quick Start
+
+```ruby
+require "rubric_llm"
+
+RubricLLM.configure do |c|
+  c.judge_model = "gpt-4o"
+  c.judge_provider = :openai
+end
+
+result = RubricLLM.evaluate(
+  question: "What is the capital of France?",
+  answer: "The capital of France is Paris, located on the Seine river.",
+  context: ["Paris is the capital and largest city of France."],
+  ground_truth: "Paris"
+)
+
+result.faithfulness      # => 0.95
+result.relevance         # => 0.92
+result.correctness       # => 0.98
+result.overall           # => 0.94
+result.pass?             # => true
+```
+
+## Configuration
+
+### Global
+
+```ruby
+RubricLLM.configure do |c|
+  c.judge_model = "gpt-4o"           # any model RubyLLM supports
+  c.judge_provider = :openai          # :openai, :anthropic, :gemini, etc.
+  c.temperature = 0.0                 # deterministic scoring (default)
+  c.max_tokens = 4096                 # max tokens for judge response
+end
+```
+
+### Environment Variables
+
+All config fields can be set via environment variables:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `RUBRIC_JUDGE_MODEL` | `gpt-4o` | Judge LLM model name |
+| `RUBRIC_JUDGE_PROVIDER` | `openai` | RubyLLM provider |
+| `RUBRIC_TEMPERATURE` | `0.0` | Judge temperature |
+| `RUBRIC_MAX_TOKENS` | `4096` | Max response tokens |
+| `RUBRIC_MAX_RETRIES` | `2` | Max retries on transient failures |
+| `RUBRIC_RETRY_BASE_DELAY` | `1.0` | Base delay (seconds) for exponential backoff |
+| `RUBRIC_CONCURRENCY` | `1` | Thread pool size for batch evaluation |
+
+```ruby
+# Reads all RUBRIC_* env vars automatically
+config = RubricLLM::Config.from_env
+```
+
+### Per-Evaluation Override
+
+```ruby
+custom = RubricLLM::Config.new(judge_model: "claude-haiku-4-5", judge_provider: :anthropic)
+
+result = RubricLLM.evaluate(question: "...", answer: "...", config: custom)
+report = RubricLLM.evaluate_batch(dataset, config: custom)
+```
+
+### Rails Setup
+
+```ruby
+# config/initializers/rubric_llm.rb
+RubricLLM.configure do |c|
+  c.judge_model = "gpt-4o"
+  c.judge_provider = :openai
+end
+```
+
+## Metrics
+
+### LLM-as-Judge Metrics
+
+These metrics use a judge LLM to evaluate quality. Each sends a structured prompt and parses a JSON response with a 0.0–1.0 score.
+
+| Metric | Question it answers | Requires |
+|--------|-------------------|----------|
+| **Faithfulness** | Is every claim in the answer supported by the context? | `context` |
+| **Relevance** | Does the answer address what was asked? | `question` |
+| **Correctness** | Does the answer match the known correct answer? | `ground_truth` |
+| **Context Precision** | Are the retrieved context chunks actually relevant? | `question`, `context` |
+| **Context Recall** | Do the contexts cover the information in the ground truth? | `context`, `ground_truth` |
+| **Factual Accuracy** | Are there factual discrepancies between candidate and reference? | `ground_truth` |
+
+```ruby
+# Only context — gets faithfulness, relevance, context_precision
+result = RubricLLM.evaluate(
+  question: "How does photosynthesis work?",
+  answer: "Plants convert sunlight into energy.",
+  context: ["Photosynthesis is the process by which plants convert light energy into chemical energy."]
+)
+
+# With ground truth — gets all metrics
+result = RubricLLM.evaluate(
+  question: "How does photosynthesis work?",
+  answer: "Plants convert sunlight into energy.",
+  context: ["Photosynthesis is the process by which plants convert light energy into chemical energy."],
+  ground_truth: "Plants use photosynthesis to convert sunlight, water, and CO2 into glucose and oxygen."
+)
+```
+
+### Custom Metrics
+
+```ruby
+class ToneMetric < RubricLLM::Metrics::Base
+  SYSTEM_PROMPT = "Rate professional tone from 0.0 to 1.0. Respond with JSON: {\"score\": 0.0, \"tone\": \"description\"}"
+
+  def call(answer:, **)
+    result = judge_eval(system_prompt: SYSTEM_PROMPT, user_prompt: "Answer: #{answer}")
+    return { score: nil, details: result } unless result.is_a?(Hash) && result["score"]
+
+    { score: Float(result["score"]), details: { tone: result["tone"] } }
+  end
+end
+
+result = RubricLLM.evaluate(
+  question: "q", answer: "a",
+  metrics: [RubricLLM::Metrics::Faithfulness, ToneMetric]
+)
+result.scores[:tone_metric]  # => 0.85
+```
+
+### Retrieval Metrics
+
+Pure math — no LLM calls, no API key needed.
+
+```ruby
+result = RubricLLM.evaluate_retrieval(
+  retrieved: ["doc_a", "doc_b", "doc_c", "doc_d"],
+  relevant: ["doc_a", "doc_c"]
+)
+
+result.precision_at_k(3)  # => 0.67
+result.recall_at_k(3)     # => 1.0
+result.mrr                # => 1.0
+result.ndcg               # => 0.86
+result.hit_rate           # => 1.0
+```
+
+## Batch Evaluation
+
+Evaluate a dataset and get aggregate statistics:
+
+```ruby
+dataset = [
+  { question: "What is Ruby?", answer: "A programming language.",
+    context: ["Ruby is a dynamic language."], ground_truth: "Ruby is a programming language." },
+  { question: "What is Rails?", answer: "A web framework.",
+    context: ["Rails is a web framework for Ruby."], ground_truth: "Rails is a Ruby web framework." },
+  # ...
+]
+
+report = RubricLLM.evaluate_batch(dataset)
+
+# Speed up with concurrent evaluation (thread pool)
+report = RubricLLM.evaluate_batch(dataset, concurrency: 4)
+
+puts report.summary
+# RubricLLM Evaluation Report
+# ========================================
+# Samples: 20
+# Duration: 45.2s
+#   faithfulness          mean=0.920  std=0.050  min=0.850  max=0.980  n=20
+
+report.worst(3)                    # 3 lowest-scoring results
+report.failures(threshold: 0.8)   # results below 0.8
+report.export_csv("results.csv")      # export to CSV
+report.export_json("results.json")    # export to JSON
+report.to_json                        # returns JSON string
+```
+
+## A/B Model Comparison
+
+Compare two models with statistical significance testing:
+
+```ruby
+config_a = RubricLLM::Config.new(judge_model: "gpt-4o")
+config_b = RubricLLM::Config.new(judge_model: "claude-sonnet-4-6")
+
+report_a = RubricLLM.evaluate_batch(dataset, config: config_a)
+report_b = RubricLLM.evaluate_batch(dataset, config: config_b)
+
+comparison = RubricLLM.compare(report_a, report_b)
+
+puts comparison.summary
+# A/B Comparison
+# ======================================================================
+# Metric                      A        B    Delta    p-value  Sig
+# ----------------------------------------------------------------------
+# faithfulness                0.880    0.920   +0.040     0.0230    *
+# relevance                   0.850    0.860   +0.010     0.4210
+# correctness                 0.910    0.940   +0.030     0.0089   **
+
+comparison.significant_improvements   # => [:faithfulness, :correctness]
+comparison.significant_regressions    # => []
+```
+
+Significance markers: `*` (p < 0.05), `**` (p < 0.01), `***` (p < 0.001)
+
+## Test Integration
+
+### Minitest
+
+```ruby
+require "rubric_llm/minitest"
+
+class AdvisorTest < Minitest::Test
+  include RubricLLM::Assertions
+
+  def test_answer_is_faithful
+    answer = my_llm.ask("What is Ruby?", context: docs)
+    assert_faithful answer, docs, threshold: 0.8
+  end
+
+  def test_answer_is_correct
+    answer = my_llm.ask("What is 2+2?")
+    assert_correct answer, "4", threshold: 0.9
+  end
+
+  def test_no_hallucination
+    answer = my_llm.ask("Summarize this", context: docs)
+    refute_hallucination answer, docs
+  end
+
+  def test_answer_is_relevant
+    answer = my_llm.ask("How do I deploy Rails?")
+    assert_relevant "How do I deploy Rails?", answer, threshold: 0.7
+  end
+end
+```
+
+### RSpec
+
+```ruby
+require "rubric_llm/rspec"
+
+RSpec.describe "My LLM" do
+  include RubricLLM::RSpecMatchers
+
+  let(:answer) { my_llm.ask(question, context: docs) }
+
+  it { expect(answer).to be_faithful_to(docs).with_threshold(0.8) }
+  it { expect(answer).to be_relevant_to(question) }
+  it { expect(answer).to be_correct_for(expected_answer) }
+  it { expect(answer).not_to hallucinate_from(docs) }
+end
+```
+
+## Error Handling
+
+```ruby
+begin
+  result = RubricLLM.evaluate(question: "q", answer: "a", context: ["c"])
+rescue RubricLLM::JudgeError => e
+  # LLM call failed (network, auth, rate limit)
+  puts "Judge error: #{e.message}"
+rescue RubricLLM::ConfigurationError => e
+  # Invalid configuration
+  puts "Config error: #{e.message}"
+rescue RubricLLM::Error => e
+  # Catch-all for any RubricLLM error
+  puts "Error: #{e.message}"
+end
+```
+
+Individual metric failures are handled gracefully — a failed metric returns `nil` for the score and includes the error in details:
+
+```ruby
+result = RubricLLM.evaluate(question: "q", answer: "a")
+result.scores[:faithfulness]           # => nil (if judge failed)
+result.details[:faithfulness][:error]  # => "Judge call failed: ..."
+result.overall                         # => mean of non-nil scores only
+```
+
+## Development
+
+```bash
+bundle install
+bundle exec rake test
+bundle exec rubocop
+```
+
+## Limitations
+
+RubricLLM uses LLM-as-Judge — an LLM scores another LLM's output. This is the industry-standard approach (used by Ragas, DeepEval, ARES), but it means the judge shares the same class of failure modes as the system being evaluated. If the judge hallucinates that an answer is faithful, you get a false positive.
+
+Mitigations built into the framework:
+
+- **Cross-model judging.** Configure a different model as judge than the one being evaluated. Don't let GPT-4o grade GPT-4o.
+- **Retrieval metrics are pure math.** `precision_at_k`, `recall_at_k`, `mrr`, `ndcg` — no LLM involved, no judge bias.
+- **Custom non-LLM metrics.** Subclass `Metrics::Base` with regex checks, embedding similarity, or any deterministic logic.
+- **Statistical comparison.** A/B testing with paired t-tests surfaces systematic judge bias across runs.
+
+For high-stakes evaluation, pair LLM-as-Judge metrics with retrieval metrics and periodic human review.
+
+## Why RubricLLM?
+
+Ruby has two LLM evaluation options today. Neither fits most use cases:
+
+| | [eval-ruby](https://github.com/johannesdwicahyo/eval-ruby) | [leva](https://github.com/kieranklaassen/leva) | RubricLLM |
+|---|---|---|---|
+| **What it is** | Generic RAG metrics | Rails engine with UI | Lightweight eval framework |
+| **LLM access** | Raw HTTP (OpenAI/Anthropic only) | You implement it | RubyLLM (any provider) |
+| **Rails required?** | No | Yes (engine + 6 migrations) | No |
+| **ActiveRecord?** | No | Yes | No |
+| **A/B comparison** | Basic | No | Paired t-test with p-values |
+| **Test assertions** | Minitest + RSpec | No | Minitest + RSpec |
+| **Pluggable metrics** | No (fixed set) | Yes | Yes |
+| **Retrieval metrics** | Yes | No | Yes |
+
+## Requirements
+
+- Ruby >= 3.4
+- [ruby_llm](https://github.com/crmne/ruby_llm) ~> 1.0
+- An API key for your chosen LLM provider (set via RubyLLM configuration)
+
+## Contributing
+
+Bug reports and pull requests are welcome on [GitHub](https://github.com/dpaluy/rubric_llm).
+
+## License
+
+[MIT](LICENSE.txt)

data/lib/rubric_llm/comparison.rb

@@ -0,0 +1,149 @@
+# frozen_string_literal: true
+
+module RubricLLM
+  class Comparison
+    attr_reader :report_a, :report_b
+
+    def initialize(report_a, report_b)
+      @report_a = report_a
+      @report_b = report_b
+
+      return if report_a.results.size == report_b.results.size
+
+      warn "[RubricLLM] Comparison reports have different sizes " \
+           "(#{report_a.results.size} vs #{report_b.results.size}). " \
+           "Unmatched pairs will be dropped."
+    end
+
+    def results
+      @results ||= compute_results
+    end
+
+    def summary
+      lines = ["A/B Comparison"]
+      lines << ("=" * 70)
+      lines << "Metric                      A        B    Delta    p-value  Sig"
+      lines << ("-" * 70)
+
+      results.each do |metric, r|
+        lines << format("%-20s %8.3f %8.3f %+8.3f %10.4f %4s",
+                        metric, r[:mean_a], r[:mean_b], r[:delta], r[:p_value], r[:significance])
+      end
+
+      lines.join("\n")
+    end
+
+    def significant_improvements(alpha: 0.05)
+      results.select { |_, r| r[:p_value] < alpha && r[:delta].positive? }.keys
+    end
+
+    def significant_regressions(alpha: 0.05)
+      results.select { |_, r| r[:p_value] < alpha && r[:delta].negative? }.keys
+    end
+
+    private
+
+    def compute_results
+      metrics = (report_a.metric_stats.keys | report_b.metric_stats.keys)
+
+      metrics.each_with_object({}) do |metric, hash|
+        paired_scores = report_a.scores_for(metric)
+                                .zip(report_b.scores_for(metric))
+                                .select { |score_a, score_b| !score_a.nil? && !score_b.nil? }
+
+        next if paired_scores.empty?
+
+        scores_a, scores_b = paired_scores.transpose
+
+        mean_a = scores_a.sum / scores_a.size.to_f
+        mean_b = scores_b.sum / scores_b.size.to_f
+        delta = mean_b - mean_a
+        p_value = paired_t_test(scores_a, scores_b)
+
+        hash[metric] = {
+          mean_a:,
+          mean_b:,
+          delta:,
+          p_value:,
+          significance: significance_marker(p_value)
+        }
+      end
+    end
+
+    def paired_t_test(a, b)
+      n = [a.size, b.size].min
+      return 1.0 if n < 2
+
+      diffs = a.first(n).zip(b.first(n)).map { |x, y| y - x }
+      mean_d = diffs.sum / n.to_f
+      var_d = diffs.sum { |d| (d - mean_d)**2 } / (n - 1).to_f
+      se = Math.sqrt(var_d / n)
+
+      return 1.0 if se.zero?
+
+      t = mean_d / se
+      df = n - 1
+
+      # Two-tailed p-value approximation using Student's t-distribution
+      two_tailed_p(t.abs, df)
+    end
+
+    # Two-tailed p-value for Student's t-distribution.
+    # p = I_x(df/2, 1/2) where x = df/(df + t²)
+    def two_tailed_p(t, df)
+      x = df / (df + (t**2))
+      regularized_beta(x, df / 2.0, 0.5)
+    rescue StandardError
+      1.0
+    end
+
+    # Regularized incomplete beta function via continued fraction (Lentz's method).
+    def regularized_beta(x, a, b)
+      return 0.0 if x <= 0.0
+      return 1.0 if x >= 1.0
+
+      ln_beta = Math.lgamma(a)[0] + Math.lgamma(b)[0] - Math.lgamma(a + b)[0]
+      front = Math.exp((a * Math.log(x)) + (b * Math.log(1.0 - x)) - ln_beta) / a
+
+      # Lentz's continued fraction
+      c = 1.0
+      d = 1.0 - ((a + b) * x / (a + 1.0))
+      d = 1.0 if d.abs < 1e-30
+      d = 1.0 / d
+      f = d
+
+      (1..200).each do |m|
+        # Even step
+        numerator = m * (b - m) * x / ((a + (2 * m) - 1) * (a + (2 * m)))
+        d = 1.0 + (numerator * d)
+        d = 1e-30 if d.abs < 1e-30
+        c = 1.0 + (numerator / c)
+        c = 1e-30 if c.abs < 1e-30
+        d = 1.0 / d
+        f *= c * d
+
+        # Odd step
+        numerator = -(a + m) * (a + b + m) * x / ((a + (2 * m)) * (a + (2 * m) + 1))
+        d = 1.0 + (numerator * d)
+        d = 1e-30 if d.abs < 1e-30
+        c = 1.0 + (numerator / c)
+        c = 1e-30 if c.abs < 1e-30
+        d = 1.0 / d
+        delta = c * d
+        f *= delta
+
+        break if (delta - 1.0).abs < 1e-10
+      end
+
+      front * f
+    end
+
+    def significance_marker(p)
+      if p < 0.001 then "***"
+      elsif p < 0.01 then "**"
+      elsif p < 0.05 then "*"
+      else ""
+      end
+    end
+  end
+end

data/lib/rubric_llm/config.rb

@@ -0,0 +1,93 @@
+# frozen_string_literal: true
+
+module RubricLLM
+  class Config
+    attr_accessor :judge_model, :judge_provider, :temperature, :max_tokens, :custom_prompt,
+                  :max_retries, :retry_base_delay, :concurrency
+
+    def initialize(judge_model: nil, judge_provider: nil, temperature: nil, max_tokens: nil, # rubocop:disable Metrics/ParameterLists
+                   custom_prompt: nil, max_retries: nil, retry_base_delay: nil, concurrency: nil, validate: false)
+      @judge_model = judge_model || ENV.fetch("RUBRIC_JUDGE_MODEL", "gpt-4o")
+      @judge_provider = (judge_provider || ENV.fetch("RUBRIC_JUDGE_PROVIDER", "openai")).to_sym
+      @temperature = temperature || Float(ENV.fetch("RUBRIC_TEMPERATURE", "0.0"))
+      @max_tokens = max_tokens || Integer(ENV.fetch("RUBRIC_MAX_TOKENS", "4096"))
+      @custom_prompt = custom_prompt
+      @max_retries = max_retries || Integer(ENV.fetch("RUBRIC_MAX_RETRIES", "2"))
+      @retry_base_delay = retry_base_delay || Float(ENV.fetch("RUBRIC_RETRY_BASE_DELAY", "1.0"))
+      @concurrency = concurrency || Integer(ENV.fetch("RUBRIC_CONCURRENCY", "1"))
+      validate! if validate
+    end
+
+    def self.from_env
+      new
+    end
+
+    def validate!
+      validate_judge_model
+      validate_judge_provider
+      validate_temperature
+      validate_max_tokens
+      validate_max_retries
+      validate_retry_base_delay
+      validate_concurrency
+      self
+    end
+
+    def to_h
+      {
+        judge_model:,
+        judge_provider:,
+        temperature:,
+        max_tokens:,
+        custom_prompt:,
+        max_retries:,
+        retry_base_delay:,
+        concurrency:
+      }
+    end
+
+    private
+
+    def validate_judge_model
+      return unless judge_model.nil? || judge_model.to_s.strip.empty?
+
+      raise ConfigurationError, "judge_model must be a non-empty string"
+    end
+
+    def validate_judge_provider
+      return if judge_provider.is_a?(Symbol)
+
+      raise ConfigurationError, "judge_provider must be a symbol"
+    end
+
+    def validate_temperature
+      return if temperature.is_a?(Numeric) && temperature.between?(0.0, 2.0)
+
+      raise ConfigurationError, "temperature must be between 0.0 and 2.0"
+    end
+
+    def validate_max_tokens
+      return if max_tokens.is_a?(Integer) && max_tokens.positive?
+
+      raise ConfigurationError, "max_tokens must be a positive integer"
+    end
+
+    def validate_max_retries
+      return if max_retries.is_a?(Integer) && max_retries >= 0
+
+      raise ConfigurationError, "max_retries must be a non-negative integer"
+    end
+
+    def validate_retry_base_delay
+      return if retry_base_delay.is_a?(Numeric) && retry_base_delay >= 0
+
+      raise ConfigurationError, "retry_base_delay must be a non-negative number"
+    end
+
+    def validate_concurrency
+      return if concurrency.is_a?(Integer) && concurrency.positive?
+
+      raise ConfigurationError, "concurrency must be a positive integer"
+    end
+  end
+end

data/lib/rubric_llm/errors.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module RubricLLM
+  class Error < StandardError; end
+
+  class ConfigurationError < Error; end
+
+  class JudgeError < Error; end
+end