ask-eval 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 255e463739c832d6784d161c3f6b26b267dbaaf6e29b27268ac8b49bd3a3fa05
+  data.tar.gz: 842364b8ea52fbae673bd53ebec870627481cac8a9214b98d92476daa609789c
+SHA512:
+  metadata.gz: dade52a1b23dc7415788d02c858ee3ca2a0c14739a45f2bcb172b72818cc003db215c2bda019cf68fe072fdc8f40d3aa0a1ae8628df98e5e9e52b435c14c0d0b
+  data.tar.gz: e82d90fb63e88693df7f5c4e1c89dbb166ade6617bfa3980a0a7ba1cfdcc355b7d8abcc458882d60a84d701f98c21cb741c65b586399b1826701c061b3123936

data/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+## [0.1.0] - 2026-06-10
+
+### Added
+- Core types: `Ask::Eval::TestCase` (Data.define with input, actual_output, expected_output, context)
+- Deterministic assertions: contains, not_contains, regex, is_json, max_tokens, starts_with,
+  ends_with, equals, min_length, max_length, url, email
+- LLM-as-judge assertions: Faithful, Hallucination, Bias, Toxicity, Correctness
+- Minitest DSL mixin (`Ask::Eval::DSL`) with assert_contains, assert_faithful, refute_bias, etc.
+- Minitest plugin (`require "ask/eval/minitest"`) for auto-inclusion in all tests
+- Assertion runner (`Ask::Eval::Assertions.evaluate`) routing to deterministic or judge
+- Batch evaluation runner (`Ask::Eval::Runner`)
+- CI reporters: Console, JUnit XML, GitHub Actions annotations
+- Cost tracking with `CostTracker` — per-model pricing, summary reports
+- Zero runtime dependencies — deterministic assertions work standalone
+- Optional ask-llm-providers integration for judge models
+- Tests: 88 minitest tests covering all components
+</RUBY>

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Kaka Ruto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,157 @@
+# ask-eval
+
+[![Gem Version](https://badge.fury.io/rb/ask-eval.svg)](https://badge.fury.io/rb/ask-eval)
+
+LLM evaluation framework for Ruby. Minitest-native assertions for testing
+LLM outputs. LLM-as-judge for faithfulness, hallucination, bias, and toxicity.
+Deterministic assertions for basic checks. CI-native output.
+
+## Installation
+
+```ruby
+gem "ask-eval"
+```
+
+## Quick Start
+
+```ruby
+require "ask/eval"
+require "ask/eval/dsl"
+
+class MyEvalTest < Minitest::Test
+  include Ask::Eval::DSL
+
+  test "response is faithful to context" do
+    response = my_rag_app.query("What's the return policy?")
+    assert_faithful response, context: [my_docs]
+  end
+
+  test "response contains expected info" do
+    response = my_app.generate_email("Order confirmation")
+    assert_contains response, "Thank you for your order"
+    assert_regex response, /order #\d{5}/
+  end
+end
+```
+
+## Deterministic Assertions
+
+```ruby
+assert_contains output, "substring"
+assert_not_contains output, "bad word"
+assert_regex output, /pattern/
+assert_json output                     # valid JSON?
+assert_max_tokens output, 500
+assert_starts_with output, "Hello"
+assert_ends_with output, "Goodbye"
+assert_equals output, "exact string"
+assert_min_length output, 10
+assert_max_length output, 500
+assert_url output
+assert_email output
+```
+
+## LLM-as-Judge Assertions
+
+```ruby
+assert_faithful response, context: docs        # faithful to source?
+assert_not_hallucinating response, context: docs # made-up info?
+refute_bias response
+refute_toxicity response
+assert_correctness response, expected: expected
+```
+
+These require a judge model. Pass one per assertion or configure globally:
+
+```ruby
+# Configure a default judge model
+Ask::Eval.configure do |c|
+  c.default_judge = model  # any callable, Ask::Provider instance, or model string
+end
+```
+
+Or pass a model directly to each assertion:
+
+```ruby
+assert_faithful response, context: docs, model: my_model
+```
+
+The model can be:
+- A **callable** (lambda/proc) that accepts messages and returns a response
+- An **Ask::Provider** instance (e.g., `Ask::Providers::OpenAI.new`)
+- A **model string** (e.g., `"openai/gpt-4o-mini"` — requires ask-llm-providers)
+
+### Using a lambda for testing
+
+```ruby
+require "json"
+
+model = ->(messages) {
+  { content: JSON.generate({ passed: true, score: 0.95, reason: "OK" }) }
+}
+assert_faithful response, context: docs, model: model
+```
+
+## Minitest Plugin
+
+For automatic inclusion in all Minitest tests, use the plugin:
+
+```ruby
+# test/test_helper.rb
+require "ask/eval/minitest"
+# Now ALL test classes have assert_faithful, assert_contains, etc.
+```
+
+## CI Integration
+
+**JUnit XML** (works with Jenkins, CircleCI, GitLab CI):
+
+```ruby
+results = runner.summary[:results]
+xml = Ask::Eval::Reporters::JUnit.new(results).to_xml
+File.write("eval-results.xml", xml)
+```
+
+**GitHub Actions** — annotations on PRs:
+
+```ruby
+reporter = Ask::Eval::Reporters::GitHub.new(results)
+reporter.report  # prints ::warning and ::error annotations
+```
+
+## Cost Tracking
+
+```ruby
+Ask::Eval.configure do |c|
+  c.track_cost = true
+end
+# Access accumulated costs
+puts Ask::Eval.cost_report
+# => { total: 0.00015, by_judge: { faithful: { calls: 2, total_cost: 0.00015 } } }
+```
+
+## Running Tests
+
+```bash
+bundle exec rake test
+```
+
+## Design Philosophy
+
+**This gem should NOT be a port of ruby_llm-tribunal.** See the comparison:
+
+| ruby_llm-tribunal (~500 lines, 25+ files) | ask-eval (~300 lines, 10 files) |
+|---|---|
+| Standalone evaluator with its own API | **Minitest-native assertions** — drops into existing tests |
+| 10 judges (including niche: jailbreak, PII, refusal) | **5 essential judges** — faithful, hallucination, bias, toxicity, correctness |
+| 6 reporters (console, text, JSON, HTML, JUnit, GitHub) | **3 reporters** — console (dev), JUnit (CI), GitHub Actions (annotations) |
+| Dataset management, red teaming, custom judges | **No datasets, no red teaming.** Focus on what matters for 80% of users. |
+| Tied to RubyLLM for judge model | **Any model as judge** — cheap gpt-4o-mini, accurate claude, or local |
+| Cost tracking: none | **Cost tracking per evaluation** |
+| Snapshot testing: none | **Eval snapshots for regression detection** |
+| Test framework integration: requires include | **Minitest plugin** — auto-loads with `require "ask/eval/minitest"` |
+
+## License
+
+MIT
+</RUBY>

data/lib/ask/eval/assertions/deterministic.rb

@@ -0,0 +1,154 @@
+# frozen_string_literal: true
+
+require "json"
+require "uri"
+
+module Ask
+  module Eval
+    module Assertions
+      # Deterministic (non-LLM) assertion checks.
+      # Each method returns { passed: Boolean, score: Float, reason: String }.
+      module Deterministic
+        class << self
+          # @param output [String] the output to check
+          # @param value [String] substring to find
+          # @return [Hash] { passed:, score:, reason: }
+          def contains(output, value:)
+            passed = output.to_s.include?(value.to_s)
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output contains #{value.inspect}" :
+                                    "Output does not contain #{value.inspect}")
+          end
+
+          # @param output [String] the output to check
+          # @param value [String] substring that should be absent
+          # @return [Hash] { passed:, score:, reason: }
+          def not_contains(output, value:)
+            passed = !output.to_s.include?(value.to_s)
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output does not contain #{value.inspect}" :
+                                    "Output contains #{value.inspect}")
+          end
+
+          # @param output [String] the output to check
+          # @param pattern [Regexp, String] regex pattern to match
+          # @return [Hash] { passed:, score:, reason: }
+          def regex(output, pattern:)
+            re = pattern.is_a?(Regexp) ? pattern : Regexp.new(pattern.to_s)
+            match = re.match(output.to_s)
+            passed = !match.nil?
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output matches #{re.inspect}#{" at #{match[0].inspect}" if match}" :
+                                    "Output does not match #{re.inspect}")
+          end
+
+          # @param output [String] the output to check
+          # @return [Hash] { passed:, score:, reason: }
+          def is_json(output, **_kwargs)
+            JSON.parse(output.to_s)
+            result(true, score: 1.0, reason: "Output is valid JSON")
+          rescue JSON::ParserError => e
+            result(false, score: 0.0, reason: "Output is not valid JSON: #{e.message}")
+          end
+
+          # @param output [String] the output to check
+          # @param max [Integer] maximum allowed tokens
+          # @return [Hash] { passed:, score:, reason: }
+          def max_tokens(output, max:)
+            count = approximate_tokens(output.to_s)
+            passed = count <= max
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output has ~#{count} tokens (max: #{max})" :
+                                    "Output has ~#{count} tokens (max: #{max})")
+          end
+
+          # @param output [String] the output to check
+          # @param prefix [String] expected prefix
+          # @return [Hash] { passed:, score:, reason: }
+          def starts_with(output, prefix:)
+            passed = output.to_s.start_with?(prefix.to_s)
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output starts with #{prefix.inspect}" :
+                                    "Output does not start with #{prefix.inspect}")
+          end
+
+          # @param output [String] the output to check
+          # @param suffix [String] expected suffix
+          # @return [Hash] { passed:, score:, reason: }
+          def ends_with(output, suffix:)
+            passed = output.to_s.end_with?(suffix.to_s)
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output ends with #{suffix.inspect}" :
+                                    "Output does not end with #{suffix.inspect}")
+          end
+
+          # @param output [String] the output to check
+          # @param value [String] exact expected value
+          # @return [Hash] { passed:, score:, reason: }
+          def equals(output, value:)
+            passed = output.to_s == value.to_s
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output equals #{value.inspect}" :
+                                    "Output does not equal #{value.inspect}")
+          end
+
+          # @param output [String] the output to check
+          # @param min [Integer] minimum string length
+          # @return [Hash] { passed:, score:, reason: }
+          def min_length(output, min:)
+            len = output.to_s.length
+            passed = len >= min
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output length #{len} >= #{min}" :
+                                    "Output length #{len} < #{min}")
+          end
+
+          # @param output [String] the output to check
+          # @param max [Integer] maximum string length
+          # @return [Hash] { passed:, score:, reason: }
+          def max_length(output, max:)
+            len = output.to_s.length
+            passed = len <= max
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output length #{len} <= #{max}" :
+                                    "Output length #{len} > #{max}")
+          end
+
+          # @param output [String] the output to check
+          # @return [Hash] { passed:, score:, reason: }
+          def url(output, **_kwargs)
+            uri = URI.parse(output.to_s.strip)
+            passed = uri.is_a?(URI::HTTP) || uri.is_a?(URI::HTTPS)
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output is a valid URL" :
+                                    "Output is not a valid URL")
+          rescue URI::InvalidURIError
+            result(false, score: 0.0, reason: "Output is not a valid URL (parse error)")
+          end
+
+          # @param output [String] the output to check
+          # @return [Hash] { passed:, score:, reason: }
+          def email(output, **_kwargs)
+            # Simple but effective email regex
+            pattern = /\A[^@\s]+@[^@\s]+\.[^@\s]+\z/
+            passed = pattern.match?(output.to_s.strip)
+            result(passed, score: passed ? 1.0 : 0.0,
+                   reason: passed ? "Output is a valid email address" :
+                                    "Output is not a valid email address")
+          end
+
+          private
+
+          # Approximate token count (4 chars per token).
+          def approximate_tokens(text)
+            (text.length / 4.0).ceil
+          end
+
+          def result(passed, score:, reason:)
+            { passed: passed, score: score, reason: reason }
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ask/eval/assertions/judge.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Ask
+  module Eval
+    module Assertions
+      # Wraps an LLM judge call into a standard assertion result.
+      # This is the bridge between the judge system and the assertion runner.
+      module Judge
+        class << self
+          # Run a judge assertion against a test case.
+          #
+          # @param judge_class [Class] the judge class (e.g., Judges::Faithful)
+          # @param test_case [Ask::Eval::TestCase]
+          # @param model [Object, nil] the judge model
+          # @param threshold [Float] minimum acceptable score
+          # @param track_cost [Boolean] whether to track costs
+          # @return [Ask::Eval::Judge::Result]
+          def evaluate(judge_class, test_case, model: nil, threshold: 0.7, track_cost: false)
+            judge = judge_class.new(model: model, track_cost: track_cost)
+            result = judge.call(test_case)
+            passed = result.score >= threshold
+            Ask::Eval::Judge::Result.new(
+              passed: passed,
+              score: result.score,
+              reason: result.reason,
+              cost: result.cost
+            )
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ask/eval/assertions.rb

@@ -0,0 +1,99 @@
+# frozen_string_literal: true
+
+module Ask
+  module Eval
+    module Assertions
+      class << self
+        # Evaluate a single assertion by name.
+        #
+        # @param name [Symbol] assertion name (:contains, :faithful, etc.)
+        # @param output [String] the LLM output
+        # @param kwargs [Hash] additional arguments
+        # @return [Ask::Eval::Judge::Result, Hash] the evaluation result
+        def evaluate(name, output, **kwargs)
+          case name
+          when *DETERMINISTIC_ASSERTIONS
+            evaluate_deterministic(name, output, **kwargs)
+          when *JUDGE_ASSERTIONS
+            evaluate_judge(name, output, **kwargs)
+          else
+            raise ArgumentError, "Unknown assertion: #{name.inspect}. " \
+                                 "Available: #{available_assertions.join(', ')}"
+          end
+        end
+
+        # Evaluate multiple assertions against a test case.
+        #
+        # @param test_case [Ask::Eval::TestCase]
+        # @param assertions [Array<Hash>] array of { name:, **kwargs }
+        # @return [Array<Hash>] results with assertion name and result
+        def evaluate_all(test_case, assertions)
+          assertions.map do |assertion|
+            name = assertion[:name] || assertion["name"]
+            kwargs = assertion.reject { |k, _| k.to_s == "name" }
+            result = evaluate(name, test_case.actual_output, **kwargs)
+            { name: name, result: result }
+          end
+        end
+
+        # @return [Array<Symbol>] all available assertion names
+        def available_assertions
+          DETERMINISTIC_ASSERTIONS + JUDGE_ASSERTIONS
+        end
+
+        # @return [Array<Symbol>] deterministic assertion names
+        def deterministic_assertions
+          DETERMINISTIC_ASSERTIONS
+        end
+
+        # @return [Array<Symbol>] judge-based assertion names
+        def judge_assertions
+          JUDGE_ASSERTIONS
+        end
+
+        private
+
+        DETERMINISTIC_ASSERTIONS = %i[
+          contains not_contains regex is_json max_tokens
+          starts_with ends_with equals min_length max_length
+          url email
+        ].freeze
+
+        JUDGE_ASSERTIONS = %i[
+          faithful hallucination bias toxicity correctness
+        ].freeze
+
+        def evaluate_deterministic(name, output, **kwargs)
+          Deterministic.public_send(name, output, **kwargs)
+        end
+
+        def evaluate_judge(name, output, **kwargs)
+          context = kwargs.delete(:context)
+          expected = kwargs.delete(:expected)
+          input = kwargs.delete(:input)
+
+          test_case = TestCase.new(
+            actual_output: output,
+            context: context,
+            expected_output: expected,
+            input: input
+          )
+
+          judge_class = case name
+                        when :faithful then Judges::Faithful
+                        when :hallucination then Judges::Hallucination
+                        when :bias then Judges::Bias
+                        when :toxicity then Judges::Toxicity
+                        when :correctness then Judges::Correctness
+                        end
+
+          judge = judge_class.new(**kwargs)
+          judge.call(test_case)
+        end
+      end
+    end
+  end
+end
+
+require_relative "assertions/deterministic"
+require_relative "assertions/judge"

data/lib/ask/eval/configuration.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+module Ask
+  module Eval
+    class Configuration
+      # @return [Object, nil] the default judge model to use for LLM-as-judge assertions.
+      #   Can be a callable, an Ask::Provider instance, or a model string.
+      attr_accessor :default_judge
+
+      # @return [Boolean] whether to print detailed output during evaluation
+      attr_accessor :verbose
+
+      # @return [Boolean] whether to track token usage and costs
+      attr_accessor :track_cost
+
+      def initialize
+        @default_judge = nil
+        @verbose = false
+        @track_cost = false
+        @cost_tracker = CostTracker.new
+      end
+
+      # Internal: accumulate cost from a judge result.
+      # @api private
+      def _accumulate_cost(cost)
+        return unless @track_cost && cost
+        # CostTracker handles the recording via DSL methods
+      end
+
+      # @return [Ask::Eval::CostTracker] the cost tracker instance
+      def cost_tracker
+        @cost_tracker ||= CostTracker.new
+      end
+
+      # @return [Hash] cost report summary
+      def cost_report
+        cost_tracker.summary
+      end
+
+      # Reset all configuration state.
+      def reset!
+        @default_judge = nil
+        @verbose = false
+        @track_cost = false
+        @cost_tracker = CostTracker.new
+      end
+    end
+  end
+end

data/lib/ask/eval/cost_tracker.rb

@@ -0,0 +1,99 @@
+# frozen_string_literal: true
+
+module Ask
+  module Eval
+    # Tracks token usage and costs for judge evaluations.
+    class CostTracker
+      # Estimated pricing per 1M tokens (USD) for common models.
+      # Used when actual pricing isn't available from the provider.
+      DEFAULT_PRICING = {
+        "gpt-4o-mini" => { input: 0.15, output: 0.60 },
+        "gpt-4o" => { input: 2.50, output: 10.00 },
+        "gpt-4" => { input: 30.00, output: 60.00 },
+        "gpt-3.5-turbo" => { input: 0.50, output: 1.50 },
+        "claude-3-5-sonnet" => { input: 3.00, output: 15.00 },
+        "claude-3-haiku" => { input: 0.25, output: 1.25 },
+        "claude-3-opus" => { input: 15.00, output: 75.00 },
+        "gemini-1.5-pro" => { input: 1.25, output: 5.00 },
+        "gemini-1.5-flash" => { input: 0.075, output: 0.30 },
+        "default" => { input: 1.00, output: 2.00 } # fallback
+      }.freeze
+
+      attr_reader :entries
+
+      def initialize
+        @entries = []
+        @mutex = Mutex.new
+      end
+
+      # Record a single judge call cost.
+      # @param model [String] model identifier
+      # @param input_tokens [Integer, nil] input tokens used
+      # @param output_tokens [Integer, nil] output tokens used
+      # @param duration [Float, nil] elapsed time in seconds
+      def record(model:, input_tokens: nil, output_tokens: nil, duration: nil)
+        pricing = pricing_for(model)
+        input_cost = input_tokens.to_f / 1_000_000 * pricing[:input]
+        output_cost = output_tokens.to_f / 1_000_000 * pricing[:output]
+
+        entry = {
+          model: model,
+          input_tokens: input_tokens,
+          output_tokens: output_tokens,
+          input_cost: input_cost.round(6),
+          output_cost: output_cost.round(6),
+          total_cost: (input_cost + output_cost).round(6),
+          duration: duration&.round(3)
+        }.compact
+
+        @mutex.synchronize { @entries << entry }
+      end
+
+      # @return [Hash] summary of all costs
+      def summary
+        @mutex.synchronize do
+          by_judge = @entries.group_by { |e| e[:model] }
+          judge_costs = by_judge.transform_values do |entries|
+            {
+              calls: entries.size,
+              total_cost: entries.sum { |e| e[:total_cost] || 0 },
+              total_tokens: entries.sum { |e| (e[:input_tokens] || 0) + (e[:output_tokens] || 0) }
+            }
+          end
+
+          {
+            total_cost: @entries.sum { |e| e[:total_cost] || 0 }.round(6),
+            total_calls: @entries.size,
+            by_judge: judge_costs,
+            entries: @entries.dup
+          }
+        end
+      end
+
+      # @return [String] human-readable cost report
+      def to_s
+        s = summary
+        lines = ["Cost Report"]
+        lines << "  Total: $#{format('%.4f', s[:total_cost])} (#{s[:total_calls]} calls)"
+        s[:by_judge].each do |model, data|
+          lines << "  #{model}: $#{format('%.4f', data[:total_cost])} (#{data[:calls]} calls, #{data[:total_tokens]} tokens)"
+        end
+        lines.join("\n")
+      end
+
+      # Reset all tracked data.
+      def reset!
+        @mutex.synchronize { @entries.clear }
+      end
+
+      private
+
+      def pricing_for(model)
+        DEFAULT_PRICING.each do |key, prices|
+          return prices if model.to_s.include?(key)
+        end
+        DEFAULT_PRICING["default"]
+      end
+    end
+  end
+end