omni_agent 0.1.5 → 0.1.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1513c46e356438c23dfd193f25c390df240f2d1d812813114e2e582222de2965
-  data.tar.gz: a9b05b4445128aea795b3f7f5ea80cb6bd668bf2b674c15651011ce6780e0c2e
+  metadata.gz: 53319cab79a557426f4a7a109a1627c425cffe49d56dc6f2d0baba58ab74b24d
+  data.tar.gz: 541ff86413a12fbf65f95c12945e45a74d9baac18f9f7d446f5dce3281ae93ce
 SHA512:
-  metadata.gz: 4ee226683fb316a82a18457b50bbe783712fe4ad9115befa4ab7c9421b31747acf2b3f609360c126a599ac8dded16a9cc4dda670e115f3bed3711147776956a3
-  data.tar.gz: 6ab849bcae377ccc919d6daf13b0556d9d6ed07b1cd16d645fe66fb026df4dd941d4f9a06b396ea044bb4ad0db886457c25ce1ca4a4c4d4de6306dc31edf47f2
+  metadata.gz: 996ba6f8c9679fea087dccc4fc202de1b4fb2e8debece3fa63438719447819590294b0e965228ff4fb2e93eb25556283cce000dde2f04bba8fb2ee14a3b3ccfd
+  data.tar.gz: 8620ac25e792aa0213f00abbf717b814c4b3cca8a09af45a8659f9e557d9df5c0e33363ff109615f139403595630832cf8ff0c370d88a1a26bdfc74bb97ccd6e

data/CHANGELOG.md

@@ -2,8 +2,29 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.1.7](https://github.com/ACR1209/omni_agent/compare/omni_agent-v0.1.6...omni_agent/v0.1.7) (2026-06-22)
+
+
+### Bug Fixes
+
+* **docs:** use release-please block marker for version badge ([b83ca8d](https://github.com/ACR1209/omni_agent/commit/b83ca8dcad4acba976125b2a844e9c67d5a4df73))
+
 ## [Unreleased]
 
+## [0.1.6] - 2026-06-21
+
+### Added
+- Added `OmniAgent::Eval` DSL for testing agent quality: `agent`, `eval_case`, `expect_tool_call`, `expect_output`, and LLM-as-judge `judge` assertions.
+- Added `golden_set` support for generating eval cases from a YAML/JSON dataset file.
+- Added `omni_agent:eval` rake task and `omni_agent:eval` generator for scaffolding and running evals.
+- Added `mock_judge` provider for deterministic judge-assertion testing.
+- Added `eval_judge_provider`/`eval_judge_model` configuration keys.
+- Added `run_alias` to `eval_case`, so evals can target an agent's `run_aliases` method (different prompt file) instead of plain `#run`.
+- Added `with:` to `input` for forwarding context into the agent run.
+- Added eval result caching (keyed on agent class, run_alias, input, context) to avoid re-spending tokens on unchanged cases, configurable via `eval_cache_enabled`/`eval_cache_path`.
+- Added `FRESH=1` env var and `fresh` task arg (`rake "omni_agent:eval[pattern,fresh]"`) to clear the eval cache before running, without manually deleting the cache file.
+- Added `omni_agent` executable (`bundle exec omni_agent eval [pattern] [--fresh]`), an rspec-like CLI alternative to the `omni_agent:eval` rake task.
+
 ## [0.1.5] - 2026-06-18
 
 ### Added

data/README.md

@@ -97,6 +97,90 @@ module ResearchAgent::Tools
 end
 ```
 
+## Evals
+
+`OmniAgent::Eval` lets you test agent quality: deterministic assertions (tool calls, output matching) and pluggable LLM-as-judge scoring.
+
+```ruby
+class ResearchAgentEval < OmniAgent::Eval
+	agent ResearchAgent
+
+	eval_case "answers weather question" do
+		input "What's the weather in Paris?"
+		expect_tool_call :get_weather, with: { city: "Paris" }
+		expect_output to_include: "Paris"
+	end
+
+	eval_case "is polite" do
+		input "Tell me a joke"
+		judge "Is the response friendly and on-topic?", threshold: 0.7
+	end
+
+	eval_case "summarizes via the :summarize run alias" do
+		run_alias :summarize
+		input "Some long article text...", with: { tone: "casual" }
+		expect_output to_include: "summary"
+	end
+end
+```
+
+* **`input text, with: {}`**: `with:` is forwarded as the agent's `context:`, bound to matching instance variables during the run (e.g. `with: { tone: "casual" }` sets `@tone`).
+* **`run_alias`**: Targets a method defined via `run_aliases` (or any zero-arg run entrypoint) instead of plain `#run` — useful when that alias renders a different prompt file (`<method_name>.md.erb`).
+
+Judge provider resolution order: explicit `provider:` kwarg on `judge` → `OmniAgent.configuration.eval_judge_provider`/`eval_judge_model` → the agent's own provider (zero-config default).
+
+### Caching
+
+Eval runs are cached by default, keyed on `(agent class, run_alias, input, context)`. Re-running the same case (e.g. iterating on assertions) replays the cached output instead of calling the provider again, saving tokens. Configure or disable it:
+
+```ruby
+OmniAgent.configure do |config|
+	config.eval_cache_enabled = true # default
+	config.eval_cache_path = "tmp/omni_agent_eval_cache.json" # default
+end
+```
+
+Bypass the cache for a run (clears it before running, no manual file deletion needed):
+
+```bash
+bundle exec omni_agent eval evals/research_agent_eval.rb --fresh
+```
+
+Sample output:
+
+```text
+[PASS] mentions lorem
+[FAIL] mentions something the mock never says
+  - output "Lorem ipsum dolor sit amet, consectetur adipiscing elit." does not include "this never appears"
+
+1/2 cases passed
+```
+
+Each case prints `[PASS]`/`[FAIL]` plus its name; failing cases list every unmet assertion's message. Exits non-zero if any case failed.
+
+For many input/expected-output pairs, load a YAML/JSON dataset instead of writing a `case` per row:
+
+```ruby
+golden_set "evals/golden/research_agent.yml" do |row|
+	expect_output to_include: row[:expected_output]
+end
+```
+
+Scaffold an eval and run it:
+
+```bash
+bundle exec rails generate omni_agent:eval ResearchAgent
+
+# run from your Rails app root, like running rspec
+bundle exec omni_agent eval
+bundle exec omni_agent eval evals/research_agent_eval.rb
+bundle exec omni_agent eval evals/research_agent_eval.rb --fresh
+```
+
+There's also an equivalent `rake omni_agent:eval` task (`rake "omni_agent:eval[pattern,fresh]"`) if you'd rather not use the binstub.
+
+Calls real LLM providers (cost, non-determinism) — deliberately **not** part of `bundle exec rspec` or CI.
+
 ## Configuration
 
 Global defaults can be configured through `OmniAgent.configure`:

data/Rakefile

@@ -6,6 +6,13 @@ load "rails/tasks/engine.rake"
 
 require "bundler/gem_tasks"
 
+# release-please already tags and pushes releases, and CI checks out a
+# detached HEAD, so the default `release` task's git tag/push step
+# (release:guard_clean, release:source_control_push) always fails there.
+# Redefine it to just build + push the gem.
+Rake::Task["release"].clear
+task release: [ "build", "release:rubygem_push" ]
+
 RSpec::Core::RakeTask.new(:spec)
 
 task default: :spec

data/exe/omni_agent

@@ -0,0 +1,42 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "optparse"
+
+options = { fresh: false }
+parser = OptionParser.new do |opts|
+  opts.banner = "Usage: omni_agent eval [pattern] [options]"
+  opts.on("--fresh", "Clear the eval cache before running") { options[:fresh] = true }
+end
+
+command = ARGV.shift
+
+case command
+when "eval"
+  parser.parse!(ARGV)
+  pattern = ARGV.shift
+
+  environment_file = File.expand_path("config/environment", Dir.pwd)
+  unless File.exist?("#{environment_file}.rb")
+    abort "Could not find config/environment.rb -- run `omni_agent eval` from your Rails app root."
+  end
+  require environment_file
+
+  status = OmniAgent::Eval::CLI.run(pattern: pattern, fresh: options[:fresh])
+
+  case status
+  when :no_files
+    abort "No eval files found matching #{(pattern || OmniAgent::Eval::CLI::DEFAULT_PATTERN).inspect}"
+  when :no_evals
+    abort "No OmniAgent::Eval subclasses found"
+  when :failed
+    exit 1
+  end
+when nil
+  warn parser.banner
+  exit 1
+else
+  warn "Unknown command: #{command.inspect}"
+  warn parser.banner
+  exit 1
+end

data/lib/generators/omni_agent/eval/eval_generator.rb

@@ -0,0 +1,37 @@
+require "rails/generators/named_base"
+require "fileutils"
+
+module OmniAgent
+  module Generators
+    class EvalGenerator < Rails::Generators::NamedBase
+      def create_eval_file
+        FileUtils.mkdir_p(evals_directory)
+
+        create_file(eval_file_path, <<~RUBY)
+          class #{eval_class_name} < OmniAgent::Eval
+            agent #{class_name}
+
+            eval_case "describe what #{class_name} should do" do
+              input "Say hello"
+              expect_output to_include: "hello"
+            end
+          end
+        RUBY
+      end
+
+      private
+
+      def evals_directory
+        File.join(destination_root, "evals")
+      end
+
+      def eval_file_path
+        File.join(evals_directory, "#{file_name}_eval.rb")
+      end
+
+      def eval_class_name
+        "#{class_name}Eval"
+      end
+    end
+  end
+end

data/lib/omni_agent/configuration.rb

@@ -1,6 +1,7 @@
 module OmniAgent
   class Configuration
-    attr_accessor :default_provider, :default_model, :max_retries, :retry_base_delay, :max_tool_iterations
+    attr_accessor :default_provider, :default_model, :max_retries, :retry_base_delay, :max_tool_iterations,
+                  :eval_judge_provider, :eval_judge_model, :eval_cache_enabled, :eval_cache_path
 
     def initialize
       @default_provider = :openai
@@ -8,6 +9,10 @@ module OmniAgent
       @max_retries = 3
       @retry_base_delay = 0.5
       @max_tool_iterations = 10
+      @eval_judge_provider = nil
+      @eval_judge_model = nil
+      @eval_cache_enabled = true
+      @eval_cache_path = "tmp/omni_agent_eval_cache.json"
     end
   end
-end
+end

data/lib/omni_agent/errors.rb

@@ -4,10 +4,12 @@ module OmniAgent
     class MissingDependencyError < Error; end
     class UnknownProviderError < Error; end
     class MaxToolIterationsError < Error; end
+    class EvalAssertionError < Error; end
   end
 
   Error = Errors::Error
   MissingDependencyError = Errors::MissingDependencyError
   UnknownProviderError = Errors::UnknownProviderError
   MaxToolIterationsError = Errors::MaxToolIterationsError
+  EvalAssertionError = Errors::EvalAssertionError
 end

data/lib/omni_agent/eval/cache.rb

@@ -0,0 +1,51 @@
+require "digest"
+require "fileutils"
+require "json"
+
+module OmniAgent
+  class Eval
+    module Cache
+      def self.fetch(key)
+        return yield unless OmniAgent.configuration.eval_cache_enabled
+
+        store = read_store
+        return store[key] if store.key?(key)
+
+        result = yield
+        store[key] = result
+        write_store(store)
+        result
+      end
+
+      def self.key_for(agent_class:, run_alias:, input:, context:)
+        Digest::SHA256.hexdigest(JSON.generate(
+          agent: agent_class.name,
+          run_alias: run_alias,
+          input: input,
+          context: context
+        ))
+      end
+
+      def self.clear!
+        File.delete(path) if File.exist?(path)
+      end
+
+      def self.path
+        OmniAgent.configuration.eval_cache_path
+      end
+
+      def self.read_store
+        return {} unless File.exist?(path)
+
+        JSON.parse(File.read(path))
+      rescue JSON::ParserError
+        {}
+      end
+
+      def self.write_store(store)
+        FileUtils.mkdir_p(File.dirname(path))
+        File.write(path, JSON.generate(store))
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/case.rb

@@ -0,0 +1,56 @@
+module OmniAgent
+  class Eval
+    class Case
+      attr_reader :name
+
+      def initialize(name, input: nil, context: {}, run_alias: nil, row: nil, &block)
+        @name = name
+        @input = input
+        @context = context || {}
+        @run_alias = run_alias
+        @assertions = []
+
+        if block
+          row.nil? ? instance_eval(&block) : instance_exec(row, &block)
+        end
+      end
+
+      def input(text, with: {})
+        @input = text
+        @context = with || {}
+      end
+
+      def run_alias(name)
+        @run_alias = name
+      end
+
+      def expect_tool_call(tool_name, with: nil)
+        @assertions << ToolCallAssertion.new(tool_name, with: with)
+      end
+
+      def expect_output(to_include: nil, to_match: nil, &block)
+        @assertions << OutputAssertion.new(to_include: to_include, to_match: to_match, &block)
+      end
+
+      def judge(criteria, threshold: 0.7, provider: nil, model: nil)
+        @assertions << JudgeAssertion.new(criteria, threshold: threshold, provider: provider, model: model)
+      end
+
+      def configured_input
+        @input
+      end
+
+      def configured_context
+        @context
+      end
+
+      def configured_run_alias
+        @run_alias
+      end
+
+      def configured_assertions
+        @assertions
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/case_result.rb

@@ -0,0 +1,16 @@
+module OmniAgent
+  class Eval
+    class CaseResult
+      attr_reader :case_name, :outcomes
+
+      def initialize(case_name:, outcomes:)
+        @case_name = case_name
+        @outcomes = outcomes
+      end
+
+      def passed?
+        outcomes.all?(&:passed?)
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/cli.rb

@@ -0,0 +1,24 @@
+module OmniAgent
+  class Eval
+    module CLI
+      DEFAULT_PATTERN = "evals/**/*_eval.rb"
+
+      def self.run(pattern: nil, fresh: false)
+        OmniAgent::Eval::Cache.clear! if fresh
+
+        files = Dir.glob(Rails.root.join(pattern || DEFAULT_PATTERN))
+        return :no_files if files.empty?
+
+        files.each { |file| require file }
+
+        eval_classes = ObjectSpace.each_object(Class).select { |klass| klass < OmniAgent::Eval }
+        return :no_evals if eval_classes.empty?
+
+        reports = eval_classes.map(&:run_all)
+        reports.each(&:print)
+
+        reports.all?(&:passed?) ? :passed : :failed
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/golden_set.rb

@@ -0,0 +1,24 @@
+require "yaml"
+require "json"
+
+module OmniAgent
+  class Eval
+    module GoldenSet
+      def self.load(path)
+        rows = path.to_s.end_with?(".json") ? JSON.parse(File.read(path)) : YAML.safe_load_file(path)
+        Array(rows).map { |row| deep_symbolize(row) }
+      end
+
+      def self.deep_symbolize(value)
+        case value
+        when Hash
+          value.each_with_object({}) { |(key, val), memo| memo[key.to_sym] = deep_symbolize(val) }
+        when Array
+          value.map { |item| deep_symbolize(item) }
+        else
+          value
+        end
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/judge.rb

@@ -0,0 +1,53 @@
+require "json"
+
+module OmniAgent
+  class Eval
+    class Judge
+      GRADING_PROMPT = <<~PROMPT
+        You are grading an AI agent's output against a single criterion.
+
+        Output: %<output>s
+
+        Criterion: %<criteria>s
+
+        Respond with strict JSON only, no other text: {"score": <float between 0 and 1>, "reason": "<short explanation>"}
+      PROMPT
+
+      def initialize(provider: nil, model: nil)
+        @provider = provider
+        @model = model
+      end
+
+      def call(criteria:, output:, fallback_provider: nil)
+        judge_provider = resolve_provider(fallback_provider)
+        prompt = format(GRADING_PROMPT, output: output, criteria: criteria)
+
+        response = judge_provider.chat(messages: [ { role: "user", content: prompt } ])
+        parse_score(response.content)
+      end
+
+      private
+
+      def resolve_provider(fallback_provider)
+        return @provider if @provider.is_a?(OmniAgent::Providers::Base)
+
+        provider_name = @provider || OmniAgent.configuration.eval_judge_provider
+        return fallback_provider if provider_name.nil?
+
+        provider_class = OmniAgent::Providers.registry[provider_name.to_sym]
+        unless provider_class
+          raise OmniAgent::UnknownProviderError, "Unknown judge provider #{provider_name.inspect}"
+        end
+
+        provider_class.new(model: @model || OmniAgent.configuration.eval_judge_model)
+      end
+
+      def parse_score(content)
+        data = JSON.parse(content.to_s)
+        { score: data["score"].to_f, reason: data["reason"] }
+      rescue JSON::ParserError
+        { score: 0.0, reason: "judge response was not valid JSON: #{content.inspect}" }
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/judge_assertion.rb

@@ -0,0 +1,22 @@
+module OmniAgent
+  class Eval
+    class JudgeAssertion
+      def initialize(criteria, threshold: 0.7, provider: nil, model: nil)
+        @criteria = criteria
+        @threshold = threshold
+        @judge = Judge.new(provider: provider, model: model)
+      end
+
+      def call(run)
+        verdict = @judge.call(criteria: @criteria, output: run.output, fallback_provider: run.agent.provider)
+        passed = verdict[:score] >= @threshold
+
+        Outcome.new(
+          passed: passed,
+          message: "judge score #{verdict[:score]} (threshold #{@threshold}) - #{verdict[:reason]}",
+          score: verdict[:score]
+        )
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/outcome.rb

@@ -0,0 +1,17 @@
+module OmniAgent
+  class Eval
+    class Outcome
+      attr_reader :message, :score
+
+      def initialize(passed:, message:, score: nil)
+        @passed = passed
+        @message = message
+        @score = score
+      end
+
+      def passed?
+        @passed == true
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/output_assertion.rb

@@ -0,0 +1,41 @@
+module OmniAgent
+  class Eval
+    class OutputAssertion
+      def initialize(to_include: nil, to_match: nil, &block)
+        @to_include = to_include
+        @to_match = to_match
+        @block = block
+      end
+
+      def call(run)
+        output = run.output
+
+        return call_block(output) if @block
+        return call_to_include(output) if @to_include
+        return call_to_match(output) if @to_match
+
+        raise ArgumentError, "expect_output requires to_include:, to_match:, or a block"
+      end
+
+      private
+
+      def call_block(output)
+        passed = !!@block.call(output)
+        message = passed ? "output matched custom block" : "output #{output.inspect} failed custom block"
+        Outcome.new(passed: passed, message: message)
+      end
+
+      def call_to_include(output)
+        passed = output.to_s.include?(@to_include.to_s)
+        message = passed ? "output includes #{@to_include.inspect}" : "output #{output.inspect} does not include #{@to_include.inspect}"
+        Outcome.new(passed: passed, message: message)
+      end
+
+      def call_to_match(output)
+        passed = !!(output.to_s =~ @to_match)
+        message = passed ? "output matches #{@to_match.inspect}" : "output #{output.inspect} does not match #{@to_match.inspect}"
+        Outcome.new(passed: passed, message: message)
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/report.rb

@@ -0,0 +1,37 @@
+module OmniAgent
+  class Eval
+    class Report
+      attr_reader :case_results
+
+      def initialize(case_results)
+        @case_results = case_results
+      end
+
+      def passed?
+        case_results.all?(&:passed?)
+      end
+
+      def raise_on_failure!
+        return if passed?
+
+        failures = case_results.reject(&:passed?).map(&:case_name)
+        raise OmniAgent::EvalAssertionError, "Eval cases failed: #{failures.join(', ')}"
+      end
+
+      def print(io: $stdout)
+        case_results.each do |case_result|
+          status = case_result.passed? ? "PASS" : "FAIL"
+          io.puts "[#{status}] #{case_result.case_name}"
+
+          case_result.outcomes.each do |outcome|
+            next if outcome.passed?
+            io.puts "  - #{outcome.message}"
+          end
+        end
+
+        passed_count = case_results.count(&:passed?)
+        io.puts "\n#{passed_count}/#{case_results.size} cases passed"
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/run.rb

@@ -0,0 +1,13 @@
+module OmniAgent
+  class Eval
+    class Run
+      attr_reader :output, :tool_calls, :agent
+
+      def initialize(output:, tool_calls:, agent:)
+        @output = output
+        @tool_calls = tool_calls
+        @agent = agent
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/runner.rb

@@ -0,0 +1,66 @@
+require "json"
+
+module OmniAgent
+  class Eval
+    module Runner
+      def self.run(eval_case, agent_class)
+        agent = agent_class.new
+
+        cache_key = Cache.key_for(
+          agent_class: agent_class,
+          run_alias: eval_case.configured_run_alias,
+          input: eval_case.configured_input,
+          context: eval_case.configured_context
+        )
+
+        cached = Cache.fetch(cache_key) { invoke_agent(agent, eval_case) }
+        run = Run.new(output: cached["output"], tool_calls: normalize_tool_calls(cached["tool_calls"]), agent: agent)
+
+        outcomes = eval_case.configured_assertions.map { |assertion| assertion.call(run) }
+        CaseResult.new(case_name: eval_case.name, outcomes: outcomes)
+      end
+
+      def self.invoke_agent(agent, eval_case)
+        response = if eval_case.configured_run_alias
+          agent.public_send(eval_case.configured_run_alias, eval_case.configured_input, context: eval_case.configured_context)
+        else
+          agent.run(eval_case.configured_input, context: eval_case.configured_context)
+        end
+
+        { "output" => response.answer.to_s, "tool_calls" => extract_tool_calls(response) }
+      end
+
+      def self.extract_tool_calls(response)
+        response.generated_messages
+          .select { |message| (message[:role] || message["role"]) == "assistant" }
+          .flat_map { |message| message[:tool_calls] || message["tool_calls"] || [] }
+          .map { |tool_call| extract_tool_call(tool_call) }
+      end
+
+      def self.extract_tool_call(tool_call)
+        function = tool_call[:function] || tool_call["function"] || {}
+        name = function[:name] || function["name"]
+        raw_arguments = function[:arguments] || function["arguments"]
+
+        { "name" => name, "arguments" => parse_arguments(raw_arguments) }
+      end
+
+      def self.parse_arguments(raw_arguments)
+        return {} if raw_arguments.nil?
+
+        parsed = raw_arguments.is_a?(String) ? JSON.parse(raw_arguments) : raw_arguments
+        parsed.transform_keys(&:to_s)
+      rescue JSON::ParserError
+        {}
+      end
+
+      def self.normalize_tool_calls(tool_calls)
+        Array(tool_calls).map do |tool_call|
+          { name: tool_call["name"], arguments: (tool_call["arguments"] || {}).transform_keys(&:to_sym) }
+        end
+      end
+
+      private_class_method :invoke_agent, :extract_tool_calls, :extract_tool_call, :parse_arguments, :normalize_tool_calls
+    end
+  end
+end

data/lib/omni_agent/eval/tool_call_assertion.rb

@@ -0,0 +1,28 @@
+module OmniAgent
+  class Eval
+    class ToolCallAssertion
+      def initialize(tool_name, with: nil)
+        @tool_name = tool_name.to_s
+        @expected_args = with
+      end
+
+      def call(run)
+        matches = run.tool_calls.select { |tool_call| tool_call[:name] == @tool_name }
+
+        if matches.empty?
+          return Outcome.new(passed: false, message: "expected tool `#{@tool_name}` to be called, but it wasn't")
+        end
+
+        return Outcome.new(passed: true, message: "tool `#{@tool_name}` was called") if @expected_args.nil?
+
+        expected = @expected_args.transform_keys(&:to_sym)
+        if matches.any? { |tool_call| expected.all? { |key, value| tool_call[:arguments][key] == value } }
+          Outcome.new(passed: true, message: "tool `#{@tool_name}` was called with #{expected}")
+        else
+          got = matches.map { |tool_call| tool_call[:arguments] }
+          Outcome.new(passed: false, message: "tool `#{@tool_name}` was called, but not with #{expected} (got #{got})")
+        end
+      end
+    end
+  end
+end

data/lib/omni_agent/eval.rb

@@ -0,0 +1,40 @@
+module OmniAgent
+  class Eval
+    class << self
+      def agent(klass = nil)
+        @agent_class = klass if klass
+        @agent_class
+      end
+
+      def eval_case(name, &block)
+        @configured_cases = configured_cases + [ Case.new(name, &block) ]
+      end
+
+      def golden_set(path, &block)
+        new_cases = GoldenSet.load(path).each_with_index.map do |row, index|
+          case_name = row[:name] || "row #{index}"
+          Case.new(case_name, input: row[:input], context: row[:context] || {}, run_alias: row[:run_alias], row: row, &block)
+        end
+
+        @configured_cases = configured_cases + new_cases
+      end
+
+      def configured_agent
+        @agent_class
+      end
+
+      def configured_cases
+        @configured_cases || []
+      end
+
+      def run_all
+        unless configured_agent
+          raise OmniAgent::Error, "#{name} must declare `agent SomeAgentClass` before running evals"
+        end
+
+        case_results = configured_cases.map { |eval_case| Runner.run(eval_case, configured_agent) }
+        Report.new(case_results)
+      end
+    end
+  end
+end

data/lib/omni_agent/providers/mock_judge.rb

@@ -0,0 +1,31 @@
+module OmniAgent
+  module Providers
+    class MockJudge < Base
+      CANNED_SCORE = 1.0
+      CANNED_REASON = "mock judge always approves"
+
+      def chat(messages:, tools: [], **_options)
+        validate_messages!(messages, allowed_roles: %i[system user assistant tool])
+
+        content = { score: CANNED_SCORE, reason: CANNED_REASON }.to_json
+
+        OmniAgent::Providers::Response.new(
+          content: content,
+          raw_request: { model: model, messages: messages, tools: tools },
+          raw_response: { "choices" => [ { "message" => { "content" => content } } ] },
+          tool_calls: []
+        )
+      end
+
+      protected
+
+      def default_api_key
+        nil
+      end
+
+      def default_model
+        "mock-judge"
+      end
+    end
+  end
+end

data/lib/omni_agent/providers.rb

@@ -4,7 +4,8 @@ module OmniAgent
     def self.registry
       {
         openai: OmniAgent::Providers::OpenAI,
-        mock: OmniAgent::Providers::Mock
+        mock: OmniAgent::Providers::Mock,
+        mock_judge: OmniAgent::Providers::MockJudge
       }
     end
   end

data/lib/omni_agent/tasks/omni_agent_tasks.rake

@@ -1,56 +1,56 @@
 require "fileutils"
 
 namespace :omni_agent do
-	desc "Set up OmniAgent directories in your Rails app"
-	task install: :environment do
-		agents_root = Rails.root.join("app", "agents")
-		FileUtils.mkdir_p(agents_root)
+  desc "Set up OmniAgent directories in your Rails app"
+  task install: :environment do
+    agents_root = Rails.root.join("app", "agents")
+    FileUtils.mkdir_p(agents_root)
 
-		puts "Created #{agents_root}"
-		puts "OmniAgent install complete."
-		puts "Next: bundle exec rake \"omni_agent:create_agent[ResearchAgent]\""
-	end
+    puts "Created #{agents_root}"
+    puts "OmniAgent install complete."
+    puts "Next: bundle exec rake \"omni_agent:create_agent[ResearchAgent]\""
+  end
 
-	desc "Create an agent scaffold. Usage: rake \"omni_agent:create_agent[ResearchAgent]\" [-- --with-tools]"
-	task :create_agent, [:name] => :environment do |_task, args|
-		agent_name = args[:name].to_s.strip
+  desc "Create an agent scaffold. Usage: rake \"omni_agent:create_agent[ResearchAgent]\" [-- --with-tools]"
+  task :create_agent, [ :name ] => :environment do |_task, args|
+    agent_name = args[:name].to_s.strip
 
-		if agent_name.empty?
-			abort "Please provide an agent name. Example: bundle exec rake \"omni_agent:create_agent[ResearchAgent]\""
-		end
+    if agent_name.empty?
+      abort "Please provide an agent name. Example: bundle exec rake \"omni_agent:create_agent[ResearchAgent]\""
+    end
 
-		with_tools = ARGV.include?("--with-tools")
+    with_tools = ARGV.include?("--with-tools")
 
-		agent_file_name = "#{agent_name.underscore}.rb"
-		agent_dir_name = agent_name.underscore
-		agent_dir = Rails.root.join("app", "agents", agent_dir_name)
-		tools_dir = agent_dir.join("tools")
-		prompt_file = agent_dir.join("prompt.md.erb")
-		agent_rb = agent_dir.join(agent_file_name)
+    agent_file_name = "#{agent_name.underscore}.rb"
+    agent_dir_name = agent_name.underscore
+    agent_dir = Rails.root.join("app", "agents", agent_dir_name)
+    tools_dir = agent_dir.join("tools")
+    prompt_file = agent_dir.join("prompt.md.erb")
+    agent_rb = agent_dir.join(agent_file_name)
 
-		FileUtils.mkdir_p(agent_dir)
+    FileUtils.mkdir_p(agent_dir)
 
-		if File.exist?(agent_rb)
-			abort "Agent file already exists: #{agent_rb}"
-		end
+    if File.exist?(agent_rb)
+      abort "Agent file already exists: #{agent_rb}"
+    end
 
-		File.write(agent_rb, <<~RUBY)
+    File.write(agent_rb, <<~RUBY)
 			class #{agent_name} < OmniAgent::Agent
 			end
-		RUBY
+    RUBY
 
-		unless File.exist?(prompt_file)
-			File.write(prompt_file, <<~PROMPT)
+    unless File.exist?(prompt_file)
+      File.write(prompt_file, <<~PROMPT)
 				You are #{agent_name}, a helpful assistant with access to local tools.
-			PROMPT
-		end
+      PROMPT
+    end
 
-		if with_tools
-			FileUtils.mkdir_p(tools_dir)
+    if with_tools
+      FileUtils.mkdir_p(tools_dir)
 
-			example_tool = tools_dir.join("example_tool.rb")
-			unless File.exist?(example_tool)
-				File.write(example_tool, <<~RUBY)
+      example_tool = tools_dir.join("example_tool.rb")
+      unless File.exist?(example_tool)
+        File.write(example_tool, <<~RUBY)
 					module #{agent_name}::Tools
 						class ExampleTool < OmniAgent::Tool
 							description "Example tool generated by omni_agent:create_agent"
@@ -66,13 +66,13 @@ namespace :omni_agent do
 							end
 						end
 					end
-				RUBY
-			end
-		end
+        RUBY
+      end
+    end
 
-		puts "Created #{agent_rb}"
-		puts "Created #{prompt_file}" if File.exist?(prompt_file)
-		puts "Created #{tools_dir}" if with_tools
-		puts "Created #{tools_dir.join("example_tool.rb")}" if with_tools && File.exist?(tools_dir.join("example_tool.rb"))
-	end
+    puts "Created #{agent_rb}"
+    puts "Created #{prompt_file}" if File.exist?(prompt_file)
+    puts "Created #{tools_dir}" if with_tools
+    puts "Created #{tools_dir.join("example_tool.rb")}" if with_tools && File.exist?(tools_dir.join("example_tool.rb"))
+  end
 end

data/lib/omni_agent/version.rb

@@ -1,3 +1,3 @@
 module OmniAgent
-  VERSION = "0.1.5"
+  VERSION = "0.1.7"
 end

data/lib/omni_agent.rb

@@ -3,7 +3,7 @@ require "omni_agent/engine" if defined?(Rails)
 require "zeitwerk"
 
 loader = Zeitwerk::Loader.for_gem
-loader.inflector.inflect("openai" => "OpenAI")
+loader.inflector.inflect("openai" => "OpenAI", "cli" => "CLI")
 loader.ignore(File.expand_path("generators", __dir__))
 loader.setup
 

data/lib/tasks/omni_agent_eval_tasks.rake

@@ -0,0 +1,19 @@
+namespace :omni_agent do
+  desc 'Run evals. Usage: rake omni_agent:eval or rake "omni_agent:eval[evals/research_agent_eval.rb,fresh]". Pass `fresh` as the 2nd arg, or set FRESH=1, to bypass the eval cache.'
+  task :eval, [ :pattern, :fresh ] => :environment do |_task, args|
+    fresh = ENV["FRESH"] == "1" || args[:fresh] == "fresh"
+    pattern = args[:pattern].to_s.strip
+    pattern = nil if pattern.empty?
+
+    status = OmniAgent::Eval::CLI.run(pattern: pattern, fresh: fresh)
+
+    case status
+    when :no_files
+      abort "No eval files found matching #{(pattern || OmniAgent::Eval::CLI::DEFAULT_PATTERN).inspect}"
+    when :no_evals
+      abort "No OmniAgent::Eval subclasses found"
+    when :failed
+      exit 1
+    end
+  end
+end

metadata

@@ -1,11 +1,11 @@
 --- !ruby/object:Gem::Specification
 name: omni_agent
 version: !ruby/object:Gem::Version
-  version: 0.1.5
+  version: 0.1.7
 platform: ruby
 authors:
 - ACR1209
-bindir: bin
+bindir: exe
 cert_chain: []
 date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
@@ -41,7 +41,8 @@ description: OmniAgent provides a Rails-native framework for defining AI agents,
   schemas, prompt templates, callbacks, and provider-backed generation workflows.
 email:
 - andrescoronel1209@gmail.com
-executables: []
+executables:
+- omni_agent
 extensions: []
 extra_rdoc_files: []
 files:
@@ -57,16 +58,33 @@ files:
 - app/models/omni_agent/application_record.rb
 - app/views/layouts/omni_agent/application.html.erb
 - config/routes.rb
+- exe/omni_agent
 - lib/generators/omni_agent/agent/agent_generator.rb
+- lib/generators/omni_agent/eval/eval_generator.rb
 - lib/generators/omni_agent/install/install_generator.rb
 - lib/omni_agent.rb
 - lib/omni_agent/agent.rb
 - lib/omni_agent/configuration.rb
 - lib/omni_agent/engine.rb
 - lib/omni_agent/errors.rb
+- lib/omni_agent/eval.rb
+- lib/omni_agent/eval/cache.rb
+- lib/omni_agent/eval/case.rb
+- lib/omni_agent/eval/case_result.rb
+- lib/omni_agent/eval/cli.rb
+- lib/omni_agent/eval/golden_set.rb
+- lib/omni_agent/eval/judge.rb
+- lib/omni_agent/eval/judge_assertion.rb
+- lib/omni_agent/eval/outcome.rb
+- lib/omni_agent/eval/output_assertion.rb
+- lib/omni_agent/eval/report.rb
+- lib/omni_agent/eval/run.rb
+- lib/omni_agent/eval/runner.rb
+- lib/omni_agent/eval/tool_call_assertion.rb
 - lib/omni_agent/providers.rb
 - lib/omni_agent/providers/base.rb
 - lib/omni_agent/providers/mock.rb
+- lib/omni_agent/providers/mock_judge.rb
 - lib/omni_agent/providers/openai.rb
 - lib/omni_agent/providers/response.rb
 - lib/omni_agent/railtie.rb
@@ -74,6 +92,7 @@ files:
 - lib/omni_agent/tool.rb
 - lib/omni_agent/tool/schema_builder.rb
 - lib/omni_agent/version.rb
+- lib/tasks/omni_agent_eval_tasks.rake
 homepage: https://github.com/ACR1209/omni_agent
 licenses:
 - MIT