ask-eval 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f3fd797bd764c1d47e8b95579fc37d9c4c3b3811c01fd858e2e83f9d76cc7b3a
-  data.tar.gz: 5f11f8a81f421a83b78ffbcf3518aff828719156cefe4f8f97399f35a7a92dbe
+  metadata.gz: d396e80eace74425011173335621532f318eb2921600aa8cfcdf1e79480f4df6
+  data.tar.gz: afc9c5a9873ba2b97766cd873211d4e2e94334c9adc8a39b372f4981a84e0fb1
 SHA512:
-  metadata.gz: cdc1212c80cfec4eeded7812f6346e1b398bec946eef6b40b0207c18215dd0350a6ef5aa93e6441343fa7c1013072678d8d645b5e7d059eb076a9277faf27db9
-  data.tar.gz: c5a3dfecc70f7d8d387279ff716ef5f9a5fd380058041daad08a0ab77a31602e451c87a7e6cbd844d5aa52634c0322c9390de21a0ff53404cb4737abbddaca76
+  metadata.gz: 88ca899713aecf0c9eec29f6470a7b0856e648f9cf54337d9252364c4ae01cd3df8f1040f5e534db7f2635e3ab8de365957858b01a11fd81d021acb81b82dde3
+  data.tar.gz: e05a44bd73f3f0236349cead4fa6f87bfe035a772f755b9ea379c3f165b01e78bb25f0cc28acbea443dc33d458031b62586bdeb61af7b35f14c897fa147998f6

data/CHANGELOG.md

@@ -1,3 +1,35 @@
+## [0.4.0] — 2026-08-07
+
+### Added
+
+- **Datasets & experiment runs — the A/B-testing loop for agents.**
+  - `Ask::Eval::Dataset` — a pinned set of fixed inputs (like fixtures, but
+    the items are prompts/tasks fed to a live agent). `add` items with
+    optional `expected` output, `context`, `tags`, and `metadata`;
+    `Dataset.save` / `Dataset.load` persist to JSON files.
+  - `Ask::Eval::Experiment` — runs a dataset against a single variant: a
+    `runner:` callable turns each input into the agent's output, an
+    optional `scorer:` callable scores it (0..1), and per-item results
+    record output, score, error, and duration. Runner errors are captured
+    per item and the run continues.
+  - `Experiment#summary` (total/passed/failed/avg_score/duration) and
+    `Experiment#compare(other)` — side-by-side per-item deltas plus an
+    aggregate verdict ("a" / "b" / "tie"), so changing a prompt or model
+    and re-running the same dataset shows the improvement.
+
+## [0.3.0] — 2026-08-03
+
+### Added
+
+- **Tool execution recording/replay.** The Recorder now tapes tool executions
+  alongside provider calls (`record_tool_call` / `replay_tool_call`), so a
+  multi-turn agent run replays as a faithful tape — tool results are replayed
+  instead of re-executed, which keeps the loop deterministic even when a tool
+  would behave differently on a second run (transient failures, changing
+  files, temp paths). Provider entries are tagged `type: "provider"` and
+  replay raises a clear "replay diverged" error if the run takes a different
+  path than the recording.
+
 ## [0.2.0] — 2026-07-21
 
 ### Added

data/README.md

@@ -2,9 +2,10 @@
 
 [![Gem Version](https://badge.fury.io/rb/ask-eval.svg)](https://badge.fury.io/rb/ask-eval)
 
-LLM evaluation framework for Ruby. Minitest-native assertions for testing
-LLM outputs. LLM-as-judge for faithfulness, hallucination, bias, and toxicity.
-Deterministic assertions for basic checks. CI-native output.
+LLM evaluation framework for Ruby. Minitest-native assertions for testing LLM
+outputs: deterministic checks plus LLM-as-judge for faithfulness,
+hallucination, bias, toxicity, and correctness. Includes session-level
+evaluation for agents, regression recording, and CI-native reporters.
 
 ## Installation
 
@@ -34,178 +35,70 @@ class MyEvalTest < Minitest::Test
 end
 ```
 
-## Deterministic Assertions
+Include `Ask::Eval::DSL` in test classes, or require `ask/eval/minitest` in
+`test_helper.rb` to get the assertions in every test automatically.
 
-```ruby
-assert_contains output, "substring"
-assert_not_contains output, "bad word"
-assert_regex output, /pattern/
-assert_json output                     # valid JSON?
-assert_max_tokens output, 500
-assert_starts_with output, "Hello"
-assert_ends_with output, "Goodbye"
-assert_equals output, "exact string"
-assert_min_length output, 10
-assert_max_length output, 500
-assert_url output
-assert_email output
-```
+## Assertions
 
-## LLM-as-Judge Assertions
+Deterministic: `assert_contains`, `assert_not_contains`, `assert_regex`,
+`assert_json`, `assert_max_tokens`, `assert_starts_with`, `assert_ends_with`,
+`assert_equals`, `assert_min_length`, `assert_max_length`, `assert_url`,
+`assert_email`.
 
-```ruby
-assert_faithful response, context: docs        # faithful to source?
-assert_not_hallucinating response, context: docs # made-up info?
-refute_bias response
-refute_toxicity response
-assert_correctness response, expected: expected
-```
+LLM-as-judge: `assert_faithful(output, context:)`,
+`assert_not_hallucinating(output, context:)`, `refute_bias`,
+`refute_toxicity`, `assert_correctness(output, expected:)`.
 
-These require a judge model. Pass one per assertion or configure globally:
+Judges need a model. Pass one per assertion (`model:`) or configure a default:
 
 ```ruby
-# Configure a default judge model
 Ask::Eval.configure do |c|
-  c.default_judge = model  # any callable, Ask::Provider instance, or model string
+  c.default_judge = "openai/gpt-4o-mini"  # any callable, Ask::Provider, or model string
 end
 ```
 
-Or pass a model directly to each assertion:
-
-```ruby
-assert_faithful response, context: docs, model: my_model
-```
-
-The model can be:
-- A **callable** (lambda/proc) that accepts messages and returns a response
-- An **Ask::Provider** instance (e.g., `Ask::Providers::OpenAI.new`)
-- A **model string** (e.g., `"openai/gpt-4o-mini"` — requires ask-llm-providers)
-
-### Using a lambda for testing
-
-```ruby
-require "json"
-
-model = ->(messages) {
-  { content: JSON.generate({ passed: true, score: 0.95, reason: "OK" }) }
-}
-assert_faithful response, context: docs, model: model
-```
-
-## Minitest Plugin
+## Agent Evaluation
 
-For automatic inclusion in all Minitest tests, use the plugin:
+Evaluate an `Ask::Agent::Session` with the `eval_session` DSL:
 
 ```ruby
-# test/test_helper.rb
-require "ask/eval/minitest"
-# Now ALL test classes have assert_faithful, assert_contains, etc.
+test "agent behavior" do
+  eval_session(model: "gpt-4o", tools: [Bash]) do |r|
+    r.run("Check health")
+    assert_tool_called "bash"
+    assert_cost_under 0.01
+  end
+end
 ```
 
-## CI Integration
+`eval_session` yields an `Ask::Eval::SessionEval` exposing `run(prompt)`,
+`tool_called?(name)`, `tool_names`, `total_cost`, and `last_response`.
 
-**JUnit XML** (works with Jenkins, CircleCI, GitLab CI):
+Interactions are recorded on first run and replayed instead of calling the LLM
+when `ASK_EVAL_MODE=replay` is set, so regression tests run without a model
+or API keys.
 
-```ruby
-results = runner.summary[:results]
-xml = Ask::Eval::Reporters::JUnit.new(results).to_xml
-File.write("eval-results.xml", xml)
-```
+## Reporters and Custom Judges
 
-**GitHub Actions** — annotations on PRs:
+Reporters consume `Ask::Eval::Runner` results: `Ask::Eval::Reporters::Console`
+(dev), `JUnit` (Jenkins, CircleCI, GitLab CI), and `GitHub` (`::warning` and
+`::error` annotations for pull requests).
 
-```ruby
-reporter = Ask::Eval::Reporters::GitHub.new(results)
-reporter.report  # prints ::warning and ::error annotations
-```
+Create your own judge by subclassing `Ask::Eval::Judge` and implementing
+`call`, `system_prompt`, and `user_message`; no registration needed.
 
-## Cost Tracking
+## Full documentation
 
-```ruby
-Ask::Eval.configure do |c|
-  c.track_cost = true
-end
-# Access accumulated costs
-puts Ask::Eval.cost_report
-# => { total: 0.00015, by_judge: { faithful: { calls: 2, total_cost: 0.00015 } } }
-```
+The full ask-rb documentation lives at https://ask-rb.github.io/ask-docs.
+https://ask-rb.github.io/ask-docs/production/evaluation covers ask-eval in
+depth, including custom judges, cost tracking, and CI integration. API
+reference: https://ask-rb.github.io/ask-docs/reference/api.
 
-## Running Tests
+## Development
 
-```bash
+bundle install
 bundle exec rake test
-```
-
-## Design Philosophy
-
-**This gem is NOT a port of ruby_llm-tribunal.** See the comparison below:
-
-| ruby_llm-tribunal | ask-eval |
-|---|---|
-| Standalone evaluator with its own API | **Minitest-native assertions** — drops into existing tests |
-| 10 judges (including niche: jailbreak, PII, refusal) | **5 essential judges** — faithful, hallucination, bias, toxicity, correctness |
-| 6 reporters (console, text, JSON, HTML, JUnit, GitHub) | **3 reporters** — console (dev), JUnit (CI), GitHub Actions (annotations) |
-| Dataset management, red teaming, custom judges | **No datasets, no red teaming.** Focus on what matters for 80% of users. |
-| Tied to RubyLLM for judge model | **Any model as judge** — cheap gpt-4o-mini, accurate claude, or local |
-| Cost tracking: none | **Cost tracking per evaluation** |
-| Snapshot testing: none | **Eval snapshots for regression detection** (v0.2.0) |
-| Test framework integration: requires include | **Minitest plugin** — auto-loads with `require "ask/eval/minitest"` |
-
-
 
 ## License
 
 MIT
-</RUBY>
-
-
-## Custom Judges
-
-The 5 built-in judges cover common cases, but you can create your own by
-subclassing `Ask::Eval::Judge`:
-
-```ruby
-class BrandVoiceJudge < Ask::Eval::Judge
-  def call(tc)
-    query_judge(tc)
-  end
-
-  private
-
-  def system_prompt
-    <<~PROMPT
-      You are a brand voice evaluator. Determine if the response matches our guidelines:
-      - Friendly but professional tone
-      - No jargon or technical terms
-      - Empathetic and helpful
-
-      Respond in JSON format:
-      { "passed": true/false, "score": 0.0-1.0, "reason": "..." }
-    PROMPT
-  end
-
-  def user_message(tc)
-    "Response to evaluate: " + tc.actual_output
-  end
-end
-
-# Use it directly
-judge = BrandVoiceJudge.new(model: my_model)
-result = judge.call(Ask::Eval::TestCase.new(actual_output: response))
-puts result.reason if result.passed?
-```
-
-### Using a lambda for custom evaluation
-
-For simple checks, pass a callable directly as the `model:` parameter --
-you do not need a full judge class:
-
-```ruby
-assert_faithful response, context: docs, model: ->(messages) {
-  { content: JSON.generate({ passed: true, score: 1.0, reason: "All good" }) }
-}
-```
-
-No registration system needed. Subclassing `Judge` and implementing
-`#call`, `#system_prompt`, and `#user_message` is the entire API.
-

data/lib/ask/eval/dataset.rb

@@ -0,0 +1,122 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Ask
+  module Eval
+    # A pinned set of fixed inputs for repeatable agent evaluation.
+    #
+    # Like Rails fixtures, a dataset gives you a deterministic baseline —
+    # but the items are *questions* (prompts/tasks fed to a live agent),
+    # not database records, and the point is running the same items against
+    # multiple variants (a changed prompt, a different model) and comparing
+    # the results.
+    #
+    # @example
+    #   dataset = Ask::Eval::Dataset.new("support-cases")
+    #   dataset.add(input: "The API returns 401 on stale tokens — how do I fix my auth flow?", tags: ["auth"])
+    #   dataset.add(input: "Summarize this thread and suggest next steps", expected: "A summary plus actions")
+    #   dataset.save("support-cases.json")
+    #
+    #   reloaded = Ask::Eval::Dataset.load("support-cases.json")
+    class Dataset
+      # A single dataset item: the input fed to the agent, optional expected
+      # output, optional context, and tags/metadata for filtering.
+      Item = Data.define(:id, :input, :expected, :context, :tags, :metadata) do
+        def to_h = { id: id, input: input, expected: expected, context: context, tags: tags, metadata: metadata }
+      end
+
+      # @param name [String] dataset name (used in reports)
+      # @param items [Array<Hash>] initial items with symbol or string keys
+      #   ({input:, expected:, context:, tags:, metadata:})
+      def initialize(name, items: [])
+        @name = name.to_s
+        @items = []
+        items.each { |item| add(**symbolize(item)) }
+      end
+
+      # @return [String] the dataset name
+      attr_reader :name
+
+      # Add an item.
+      #
+      # @param input [String] the prompt/task fed to the agent (required)
+      # @param expected [String, nil] expected/reference output
+      # @param context [String, nil] source context
+      # @param tags [Array<String>] labels for filtering
+      # @param metadata [Hash] free-form
+      # @return [Item]
+      # @raise [ArgumentError] on empty input
+      def add(input:, id: nil, expected: nil, context: nil, tags: [], metadata: {})
+        raise ArgumentError, "input is required" if input.to_s.strip.empty?
+
+        item = Item.new(
+          id: id || "item_#{@items.size + 1}",
+          input: input.to_s,
+          expected: expected,
+          context: context,
+          tags: Array(tags),
+          metadata: metadata
+        )
+        @items << item
+        item
+      end
+
+      # @return [Array<Item>] all items, in order
+      def items = @items.dup
+
+      # @return [Integer] number of items
+      def size = @items.size
+
+      def each(&block) = @items.each(&block)
+
+      # Create an experiment over this dataset (see {Experiment}).
+      #
+      # @param runner [Proc] called with the item's input; returns the
+      #   agent's output string
+      # @param scorer [Proc, nil] called with (input:, output:, expected:);
+      #   returns a score (0..1)
+      # @return [Ask::Eval::Experiment]
+      def experiment(runner:, scorer: nil)
+        Experiment.new(self, runner: runner, scorer: scorer)
+      end
+
+      # @return [Hash] serialized form (for {Dataset.save})
+      def to_h
+        { name: @name, items: @items.map(&:to_h) }
+      end
+
+      # Write the dataset to a JSON file.
+      #
+      # @param path [String]
+      # @return [void]
+      def save(path)
+        File.write(path, JSON.pretty_generate(to_h))
+        nil
+      end
+
+      # Load a dataset from a JSON file written by {Dataset.save} (or any
+      # compatible {name:, items:} shape).
+      #
+      # @param path [String]
+      # @return [Ask::Eval::Dataset]
+      def self.load(path)
+        data = JSON.parse(File.read(path))
+        new(data["name"] || File.basename(path, ".json"), items: data["items"] || [])
+      end
+
+      private
+
+      def symbolize(obj)
+        case obj
+        when Hash
+          obj.each_with_object({}) { |(k, v), h| h[k.to_sym] = symbolize(v) }
+        when Array
+          obj.map { |e| symbolize(e) }
+        else
+          obj
+        end
+      end
+    end
+  end
+end

data/lib/ask/eval/experiment.rb

@@ -0,0 +1,129 @@
+# frozen_string_literal: true
+
+module Ask
+  module Eval
+    # Runs a {Dataset} against a single variant and compares runs.
+    #
+    # An experiment executes every dataset item through a +runner+ (a
+    # callable that turns an input into the agent's output), optionally
+    # scores each output, and collects per-item results. Running the same
+    # dataset with two different runners (a changed prompt, a different
+    # model) and comparing the experiments is the A/B-testing loop of
+    # eval-driven development.
+    #
+    # @example
+    #   run_a = dataset.experiment(runner: ->(input) { agent(input, prompt: old) },
+    #                              scorer: judge_scorer)
+    #   run_b = dataset.experiment(runner: ->(input) { agent(input, prompt: new) },
+    #                              scorer: judge_scorer)
+    #   run_a.run
+    #   run_a.summary          # => {total:, passed:, failed:, avg_score:, ...}
+    #   run_a.compare(run_b)   # => per-item side-by-side + aggregate verdict
+    class Experiment
+      # One item's outcome: the produced output, optional score (0..1),
+      # error (when the runner raised), and wall-clock duration.
+      Result = Data.define(:item, :output, :score, :error, :duration_ms) do
+        # @return [Boolean] no error and, when scored, at least 0.5
+        def passed?
+          error.nil? && (score.nil? || score >= 0.5)
+        end
+      end
+
+      # @param dataset [Ask::Eval::Dataset]
+      # @param runner [Proc] called with the item's input; returns the
+      #   agent's output string
+      # @param scorer [Proc, nil] called with (input:, output:, expected:);
+      #   returns a score (0..1) or nil to skip scoring
+      def initialize(dataset, runner:, scorer: nil)
+        @dataset = dataset
+        @runner = runner
+        @scorer = scorer
+        @results = nil
+      end
+
+      # @return [Ask::Eval::Dataset]
+      attr_reader :dataset
+
+      # Execute every dataset item through the runner. A runner error is
+      # recorded on that item and the run continues.
+      #
+      # @return [self]
+      def run
+        @results = @dataset.items.map do |item|
+          start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          output = nil
+          error = nil
+          begin
+            output = @runner.call(item.input).to_s
+          rescue StandardError => e
+            error = e.message
+          end
+
+          score = nil
+          if error.nil? && @scorer
+            score = @scorer.call(input: item.input, output: output, expected: item.expected)
+          end
+
+          duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start) * 1000).round
+          Result.new(item: item, output: output, score: score, error: error, duration_ms: duration_ms)
+        end
+        self
+      end
+
+      # @return [Array<Result>] per-item results (runs first if needed)
+      def results
+        @results || run.results
+      end
+
+      # @return [Hash] aggregate statistics
+      def summary
+        rs = results
+        scores = rs.filter_map(&:score)
+        {
+          total: rs.size,
+          passed: rs.count(&:passed?),
+          failed: rs.count { |r| !r.passed? },
+          avg_score: scores.empty? ? nil : (scores.sum.fdiv(scores.size)).round(3),
+          total_duration_ms: rs.sum(&:duration_ms)
+        }
+      end
+
+      # Side-by-side comparison with another experiment over the same (or
+      # overlapping) dataset items.
+      #
+      # @param other [Ask::Eval::Experiment]
+      # @return [Hash] {deltas: [{item_id:, input:, a:, b:, delta:}],
+      #   a: summary, b: summary, verdict: "a"|"b"|"tie"}
+      def compare(other)
+        a = results
+        b = other.results
+        ha = a.to_h { |r| [r.item.id, r] }
+        hb = b.to_h { |r| [r.item.id, r] }
+
+        deltas = (ha.keys & hb.keys).map do |id|
+          ra = ha[id]
+          rb = hb[id]
+          {
+            item_id: id,
+            input: ra.item.input,
+            a: { output: ra.output, score: ra.score, error: ra.error },
+            b: { output: rb.output, score: rb.score, error: rb.error },
+            delta: (ra.score && rb.score) ? (rb.score - ra.score).round(3) : nil
+          }
+        end
+
+        { deltas: deltas, a: summary, b: other.summary, verdict: verdict(summary, other.summary) }
+      end
+
+      private
+
+      def verdict(a, b)
+        return nil if a[:avg_score].nil? || b[:avg_score].nil?
+        return "a" if a[:avg_score] > b[:avg_score]
+        return "b" if b[:avg_score] > a[:avg_score]
+
+        "tie"
+      end
+    end
+  end
+end

data/lib/ask/eval/recorder.rb

@@ -2,6 +2,7 @@
 
 require "json"
 require "fileutils"
+require "time" # Time#iso8601 for recorded_at timestamps
 
 module Ask
   module Eval
@@ -67,13 +68,27 @@ module Ask
       # Record a provider call with serialized result data.
       def record_call(args:, kwargs:, result_data:)
         @interactions << {
+          type: "provider",
           messages: scrub_messages(args.first),
           model: kwargs[:model],
           result: result_data
         }
       end
 
-      # Replay the next recorded call as an Ask::Message.
+      # Record a tool execution with serialized result data. Tool calls are
+      # interleaved with provider calls in the same order they happened, so a
+      # multi-turn agent run replays faithfully — tool results are replayed,
+      # not re-executed.
+      def record_tool_call(name:, args:, result_data:)
+        @interactions << {
+          type: "tool",
+          name: name,
+          args: args,
+          result: result_data
+        }
+      end
+
+      # Replay the next recorded provider call as an Ask::Message.
       def replay_as_message
         load_recording_if_needed
 
@@ -82,6 +97,11 @@ module Ask
           raise "No recorded interaction available. Delete #{recording_path} and re-record."
         end
 
+        if entry["type"] == "tool"
+          raise "Replay diverged: expected a provider call, but the next recorded interaction is a tool call. " \
+                "The run took a different path than the recording. Delete #{recording_path} and re-record."
+        end
+
         result = entry["result"]
 
         if result["type"] == "stream"
@@ -119,6 +139,32 @@ module Ask
         entry["result"]
       end
 
+      # Replay the next recorded tool execution, returning its original
+      # Ask::Result without executing the tool again.
+      def replay_tool_call
+        load_recording_if_needed
+
+        entry = @replay_queue.shift
+        unless entry
+          raise "No recorded tool interaction available. Delete #{recording_path} and re-record."
+        end
+        unless entry["type"] == "tool"
+          raise "Replay diverged: expected a tool call, but the next recorded interaction is a provider call. " \
+                "The run took a different path than the recording. Delete #{recording_path} and re-record."
+        end
+
+        result = entry["result"]
+        if result["type"] == "result"
+          if result["ok"]
+            Ask::Result.ok(data: result["output"], metadata: result["metadata"] || {})
+          else
+            Ask::Result.error(message: result["error"] || "", metadata: result["metadata"] || {})
+          end
+        else
+          result["data"]
+        end
+      end
+
       # Save recorded interactions to disk.
       def save
         return unless recording?
@@ -160,6 +206,15 @@ module Ask
         @replay_queue = (data["interactions"] || []).dup
       end
 
+      # Serialize a tool's Ask::Result so it can be replayed later.
+      def serialize_tool_result(result)
+        if result.is_a?(Ask::Result)
+          result.to_h.merge(type: "result")
+        else
+          { type: "raw", data: result }
+        end
+      end
+
       def serialize(result)
         if result.respond_to?(:chunks)
           {

data/lib/ask/eval/version.rb

@@ -1,5 +1,5 @@
 module Ask
   module Eval
-    VERSION = "0.2.0"
+    VERSION = "0.4.0"
   end
 end

data/lib/ask/eval.rb

@@ -14,6 +14,8 @@ module Ask
     autoload :Configuration, "ask/eval/configuration"
     autoload :Recorder, "ask/eval/recorder"
     autoload :SessionEval, "ask/eval/session_eval"
+    autoload :Dataset, "ask/eval/dataset"
+    autoload :Experiment, "ask/eval/experiment"
 
     # These are loaded eagerly since they define sub-modules with autoloads
     require_relative "eval/assertions"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ask-eval
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.4.0
 platform: ruby
 authors:
 - Kaka Ruto
@@ -51,6 +51,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: ask-core
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.1'
 description: 'Test LLM outputs with Minitest-native assertions. LLM-as-judge for faithfulness,
   hallucination, bias, toxicity. Deterministic assertions (contains, regex, JSON).
   CI-native: GitHub annotations, JUnit output, cost tracking.'
@@ -70,7 +84,9 @@ files:
 - lib/ask/eval/assertions/judge.rb
 - lib/ask/eval/configuration.rb
 - lib/ask/eval/cost_tracker.rb
+- lib/ask/eval/dataset.rb
 - lib/ask/eval/dsl.rb
+- lib/ask/eval/experiment.rb
 - lib/ask/eval/judge.rb
 - lib/ask/eval/judges/bias.rb
 - lib/ask/eval/judges/correctness.rb