ask-eval 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f3fd797bd764c1d47e8b95579fc37d9c4c3b3811c01fd858e2e83f9d76cc7b3a
-  data.tar.gz: 5f11f8a81f421a83b78ffbcf3518aff828719156cefe4f8f97399f35a7a92dbe
+  metadata.gz: 12e438efcef2038b328af35cdb14d6741a7730940f1d8f7014f4550c3b8ac543
+  data.tar.gz: c32aaf27235fdd33e206ebc4434ed647015c8fa6d81a8b16b88546c36597f4d2
 SHA512:
-  metadata.gz: cdc1212c80cfec4eeded7812f6346e1b398bec946eef6b40b0207c18215dd0350a6ef5aa93e6441343fa7c1013072678d8d645b5e7d059eb076a9277faf27db9
-  data.tar.gz: c5a3dfecc70f7d8d387279ff716ef5f9a5fd380058041daad08a0ab77a31602e451c87a7e6cbd844d5aa52634c0322c9390de21a0ff53404cb4737abbddaca76
+  metadata.gz: 015cd82d67ac7fcf88c1661d352c459845ffbee6dc373e2d9999db68d8256cb1e63988618e47263ee59eb887fa7381e42b30b0b3fcc4c1fac4cdc802891f08b2
+  data.tar.gz: 8daf2d3ccc31244b1e3748ee2f1b07970a9c1af12592e9e6bddfdafaea980816961f4099f4382f5704bb47554e8036185ce4131fe4494400b9ba0b59bec4b026

data/CHANGELOG.md

@@ -1,3 +1,16 @@
+## [0.3.0] — 2026-08-03
+
+### Added
+
+- **Tool execution recording/replay.** The Recorder now tapes tool executions
+  alongside provider calls (`record_tool_call` / `replay_tool_call`), so a
+  multi-turn agent run replays as a faithful tape — tool results are replayed
+  instead of re-executed, which keeps the loop deterministic even when a tool
+  would behave differently on a second run (transient failures, changing
+  files, temp paths). Provider entries are tagged `type: "provider"` and
+  replay raises a clear "replay diverged" error if the run takes a different
+  path than the recording.
+
 ## [0.2.0] — 2026-07-21
 
 ### Added

data/README.md

@@ -2,9 +2,10 @@
 
 [![Gem Version](https://badge.fury.io/rb/ask-eval.svg)](https://badge.fury.io/rb/ask-eval)
 
-LLM evaluation framework for Ruby. Minitest-native assertions for testing
-LLM outputs. LLM-as-judge for faithfulness, hallucination, bias, and toxicity.
-Deterministic assertions for basic checks. CI-native output.
+LLM evaluation framework for Ruby. Minitest-native assertions for testing LLM
+outputs: deterministic checks plus LLM-as-judge for faithfulness,
+hallucination, bias, toxicity, and correctness. Includes session-level
+evaluation for agents, regression recording, and CI-native reporters.
 
 ## Installation
 
@@ -34,178 +35,70 @@ class MyEvalTest < Minitest::Test
 end
 ```
 
-## Deterministic Assertions
+Include `Ask::Eval::DSL` in test classes, or require `ask/eval/minitest` in
+`test_helper.rb` to get the assertions in every test automatically.
 
-```ruby
-assert_contains output, "substring"
-assert_not_contains output, "bad word"
-assert_regex output, /pattern/
-assert_json output                     # valid JSON?
-assert_max_tokens output, 500
-assert_starts_with output, "Hello"
-assert_ends_with output, "Goodbye"
-assert_equals output, "exact string"
-assert_min_length output, 10
-assert_max_length output, 500
-assert_url output
-assert_email output
-```
+## Assertions
 
-## LLM-as-Judge Assertions
+Deterministic: `assert_contains`, `assert_not_contains`, `assert_regex`,
+`assert_json`, `assert_max_tokens`, `assert_starts_with`, `assert_ends_with`,
+`assert_equals`, `assert_min_length`, `assert_max_length`, `assert_url`,
+`assert_email`.
 
-```ruby
-assert_faithful response, context: docs        # faithful to source?
-assert_not_hallucinating response, context: docs # made-up info?
-refute_bias response
-refute_toxicity response
-assert_correctness response, expected: expected
-```
+LLM-as-judge: `assert_faithful(output, context:)`,
+`assert_not_hallucinating(output, context:)`, `refute_bias`,
+`refute_toxicity`, `assert_correctness(output, expected:)`.
 
-These require a judge model. Pass one per assertion or configure globally:
+Judges need a model. Pass one per assertion (`model:`) or configure a default:
 
 ```ruby
-# Configure a default judge model
 Ask::Eval.configure do |c|
-  c.default_judge = model  # any callable, Ask::Provider instance, or model string
+  c.default_judge = "openai/gpt-4o-mini"  # any callable, Ask::Provider, or model string
 end
 ```
 
-Or pass a model directly to each assertion:
-
-```ruby
-assert_faithful response, context: docs, model: my_model
-```
-
-The model can be:
-- A **callable** (lambda/proc) that accepts messages and returns a response
-- An **Ask::Provider** instance (e.g., `Ask::Providers::OpenAI.new`)
-- A **model string** (e.g., `"openai/gpt-4o-mini"` — requires ask-llm-providers)
-
-### Using a lambda for testing
-
-```ruby
-require "json"
-
-model = ->(messages) {
-  { content: JSON.generate({ passed: true, score: 0.95, reason: "OK" }) }
-}
-assert_faithful response, context: docs, model: model
-```
-
-## Minitest Plugin
+## Agent Evaluation
 
-For automatic inclusion in all Minitest tests, use the plugin:
+Evaluate an `Ask::Agent::Session` with the `eval_session` DSL:
 
 ```ruby
-# test/test_helper.rb
-require "ask/eval/minitest"
-# Now ALL test classes have assert_faithful, assert_contains, etc.
+test "agent behavior" do
+  eval_session(model: "gpt-4o", tools: [Bash]) do |r|
+    r.run("Check health")
+    assert_tool_called "bash"
+    assert_cost_under 0.01
+  end
+end
 ```
 
-## CI Integration
+`eval_session` yields an `Ask::Eval::SessionEval` exposing `run(prompt)`,
+`tool_called?(name)`, `tool_names`, `total_cost`, and `last_response`.
 
-**JUnit XML** (works with Jenkins, CircleCI, GitLab CI):
+Interactions are recorded on first run and replayed instead of calling the LLM
+when `ASK_EVAL_MODE=replay` is set, so regression tests run without a model
+or API keys.
 
-```ruby
-results = runner.summary[:results]
-xml = Ask::Eval::Reporters::JUnit.new(results).to_xml
-File.write("eval-results.xml", xml)
-```
+## Reporters and Custom Judges
 
-**GitHub Actions** — annotations on PRs:
+Reporters consume `Ask::Eval::Runner` results: `Ask::Eval::Reporters::Console`
+(dev), `JUnit` (Jenkins, CircleCI, GitLab CI), and `GitHub` (`::warning` and
+`::error` annotations for pull requests).
 
-```ruby
-reporter = Ask::Eval::Reporters::GitHub.new(results)
-reporter.report  # prints ::warning and ::error annotations
-```
+Create your own judge by subclassing `Ask::Eval::Judge` and implementing
+`call`, `system_prompt`, and `user_message`; no registration needed.
 
-## Cost Tracking
+## Full documentation
 
-```ruby
-Ask::Eval.configure do |c|
-  c.track_cost = true
-end
-# Access accumulated costs
-puts Ask::Eval.cost_report
-# => { total: 0.00015, by_judge: { faithful: { calls: 2, total_cost: 0.00015 } } }
-```
+The full ask-rb documentation lives at https://ask-rb.github.io/ask-docs.
+https://ask-rb.github.io/ask-docs/production/evaluation covers ask-eval in
+depth, including custom judges, cost tracking, and CI integration. API
+reference: https://ask-rb.github.io/ask-docs/reference/api.
 
-## Running Tests
+## Development
 
-```bash
+bundle install
 bundle exec rake test
-```
-
-## Design Philosophy
-
-**This gem is NOT a port of ruby_llm-tribunal.** See the comparison below:
-
-| ruby_llm-tribunal | ask-eval |
-|---|---|
-| Standalone evaluator with its own API | **Minitest-native assertions** — drops into existing tests |
-| 10 judges (including niche: jailbreak, PII, refusal) | **5 essential judges** — faithful, hallucination, bias, toxicity, correctness |
-| 6 reporters (console, text, JSON, HTML, JUnit, GitHub) | **3 reporters** — console (dev), JUnit (CI), GitHub Actions (annotations) |
-| Dataset management, red teaming, custom judges | **No datasets, no red teaming.** Focus on what matters for 80% of users. |
-| Tied to RubyLLM for judge model | **Any model as judge** — cheap gpt-4o-mini, accurate claude, or local |
-| Cost tracking: none | **Cost tracking per evaluation** |
-| Snapshot testing: none | **Eval snapshots for regression detection** (v0.2.0) |
-| Test framework integration: requires include | **Minitest plugin** — auto-loads with `require "ask/eval/minitest"` |
-
-
 
 ## License
 
 MIT
-</RUBY>
-
-
-## Custom Judges
-
-The 5 built-in judges cover common cases, but you can create your own by
-subclassing `Ask::Eval::Judge`:
-
-```ruby
-class BrandVoiceJudge < Ask::Eval::Judge
-  def call(tc)
-    query_judge(tc)
-  end
-
-  private
-
-  def system_prompt
-    <<~PROMPT
-      You are a brand voice evaluator. Determine if the response matches our guidelines:
-      - Friendly but professional tone
-      - No jargon or technical terms
-      - Empathetic and helpful
-
-      Respond in JSON format:
-      { "passed": true/false, "score": 0.0-1.0, "reason": "..." }
-    PROMPT
-  end
-
-  def user_message(tc)
-    "Response to evaluate: " + tc.actual_output
-  end
-end
-
-# Use it directly
-judge = BrandVoiceJudge.new(model: my_model)
-result = judge.call(Ask::Eval::TestCase.new(actual_output: response))
-puts result.reason if result.passed?
-```
-
-### Using a lambda for custom evaluation
-
-For simple checks, pass a callable directly as the `model:` parameter --
-you do not need a full judge class:
-
-```ruby
-assert_faithful response, context: docs, model: ->(messages) {
-  { content: JSON.generate({ passed: true, score: 1.0, reason: "All good" }) }
-}
-```
-
-No registration system needed. Subclassing `Judge` and implementing
-`#call`, `#system_prompt`, and `#user_message` is the entire API.
-

data/lib/ask/eval/recorder.rb

@@ -67,13 +67,27 @@ module Ask
       # Record a provider call with serialized result data.
       def record_call(args:, kwargs:, result_data:)
         @interactions << {
+          type: "provider",
           messages: scrub_messages(args.first),
           model: kwargs[:model],
           result: result_data
         }
       end
 
-      # Replay the next recorded call as an Ask::Message.
+      # Record a tool execution with serialized result data. Tool calls are
+      # interleaved with provider calls in the same order they happened, so a
+      # multi-turn agent run replays faithfully — tool results are replayed,
+      # not re-executed.
+      def record_tool_call(name:, args:, result_data:)
+        @interactions << {
+          type: "tool",
+          name: name,
+          args: args,
+          result: result_data
+        }
+      end
+
+      # Replay the next recorded provider call as an Ask::Message.
       def replay_as_message
         load_recording_if_needed
 
@@ -82,6 +96,11 @@ module Ask
           raise "No recorded interaction available. Delete #{recording_path} and re-record."
         end
 
+        if entry["type"] == "tool"
+          raise "Replay diverged: expected a provider call, but the next recorded interaction is a tool call. " \
+                "The run took a different path than the recording. Delete #{recording_path} and re-record."
+        end
+
         result = entry["result"]
 
         if result["type"] == "stream"
@@ -119,6 +138,32 @@ module Ask
         entry["result"]
       end
 
+      # Replay the next recorded tool execution, returning its original
+      # Ask::Result without executing the tool again.
+      def replay_tool_call
+        load_recording_if_needed
+
+        entry = @replay_queue.shift
+        unless entry
+          raise "No recorded tool interaction available. Delete #{recording_path} and re-record."
+        end
+        unless entry["type"] == "tool"
+          raise "Replay diverged: expected a tool call, but the next recorded interaction is a provider call. " \
+                "The run took a different path than the recording. Delete #{recording_path} and re-record."
+        end
+
+        result = entry["result"]
+        if result["type"] == "result"
+          if result["ok"]
+            Ask::Result.ok(data: result["output"], metadata: result["metadata"] || {})
+          else
+            Ask::Result.error(message: result["error"] || "", metadata: result["metadata"] || {})
+          end
+        else
+          result["data"]
+        end
+      end
+
       # Save recorded interactions to disk.
       def save
         return unless recording?
@@ -160,6 +205,15 @@ module Ask
         @replay_queue = (data["interactions"] || []).dup
       end
 
+      # Serialize a tool's Ask::Result so it can be replayed later.
+      def serialize_tool_result(result)
+        if result.is_a?(Ask::Result)
+          result.to_h.merge(type: "result")
+        else
+          { type: "raw", data: result }
+        end
+      end
+
       def serialize(result)
         if result.respond_to?(:chunks)
           {

data/lib/ask/eval/version.rb

@@ -1,5 +1,5 @@
 module Ask
   module Eval
-    VERSION = "0.2.0"
+    VERSION = "0.3.0"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ask-eval
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Kaka Ruto