omni_agent 0.1.4 → 0.1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 68e52e8d8eb8a4f59c708d32e7464b7526c92f8e5342d6a85fe3c1244bf643e7
-  data.tar.gz: 36d892f5d7593582f31cc15d957deab720c80c8416a48ba32df4253f24d10ac6
+  metadata.gz: d48868f19885179d74185a391e4b6687f3dc2d4320962eac162ac6f12d755892
+  data.tar.gz: 6b93ca1c7aa336226e64936e13839c862c4a05a301a42b9802042012fe2f157c
 SHA512:
-  metadata.gz: afaf69e6f85c7db759d6796477601945577550a80f86e53a694f52d07910ffe081e79aa18db0688a8aaf3b4cb5dae5e141beaefc43e4083c59e46af2f6496df3
-  data.tar.gz: efc7e291b645b15c0fe2d528737597948aaa28b9cc5a2f1bffd153313f071c0bd659d3d5b885932a607cf21f55dd217f907f85e82a667cbc797c8d8b85b9674d
+  metadata.gz: d0b1c7d3a1e9e0166f934d722b51c6f08c2771368b336c59efa46b67b2ba71ac173189b9d97b3431649b6171fe8eb309db6c49de5401b1fbd230cdb800008053
+  data.tar.gz: e337186410b4b79da024276338208520d95c48fe7ca2b9fa1731d37c2ad4836f469aa69d57230805f0e250c591727de9d6db0aef49a7502da1b065a403557d44

data/CHANGELOG.md

@@ -4,25 +4,41 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
-## [0.1.0] - 2026-06-09
+## [0.1.6] - 2026-06-21
 
 ### Added
-- Initial release of OmniAgent as a Rails engine for building application-native AI agents.
-- Agent runtime with provider abstraction, tool-calling loop, callbacks, and prompt rendering.
-- OpenAI provider support via the `openai` gem.
-- Tool DSL with JSON schema generation.
-- Rails generators and tasks for agent scaffolding.
-- RSpec coverage for agent runtime, providers, tools, and integration flows.
+- Added `OmniAgent::Eval` DSL for testing agent quality: `agent`, `eval_case`, `expect_tool_call`, `expect_output`, and LLM-as-judge `judge` assertions.
+- Added `golden_set` support for generating eval cases from a YAML/JSON dataset file.
+- Added `omni_agent:eval` rake task and `omni_agent:eval` generator for scaffolding and running evals.
+- Added `mock_judge` provider for deterministic judge-assertion testing.
+- Added `eval_judge_provider`/`eval_judge_model` configuration keys.
+- Added `run_alias` to `eval_case`, so evals can target an agent's `run_aliases` method (different prompt file) instead of plain `#run`.
+- Added `with:` to `input` for forwarding context into the agent run.
+- Added eval result caching (keyed on agent class, run_alias, input, context) to avoid re-spending tokens on unchanged cases, configurable via `eval_cache_enabled`/`eval_cache_path`.
+- Added `FRESH=1` env var and `fresh` task arg (`rake "omni_agent:eval[pattern,fresh]"`) to clear the eval cache before running, without manually deleting the cache file.
+- Added `omni_agent` executable (`bundle exec omni_agent eval [pattern] [--fresh]`), an rspec-like CLI alternative to the `omni_agent:eval` rake task.
 
-## [0.1.1] - 2026-06-09
+## [0.1.5] - 2026-06-18
 
-### Changed
-- Lowered Rails version needed.
+### Added
+- Added max tool iterations limit on the agent loop, with corresponding error raised when exceeded.
+- Added retry logic to the OpenAI provider, configurable via `Configuration`.
 
-## [0.1.2] - 2026-06-11
+### Fixed
+- Improved provider resolution error handling for unknown providers.
+- Forced eager load of error aliases to avoid Zeitwerk autoloading issues.
+
+# [0.1.4] - 2026-06-16
 
 ### Fixed
-- Fixed Zeitwerk issue with `OmniAgent::Errors`
+- Change invoke method to an instance method instead of a class method, to actually catch invocations of the `stop_generation!` method.
+
+### Changed
+- Remove agent file from directory to prevent needing to change Zeitwerk initializer as Railtie wasn't allowing to change behaviour without it. 
+
+### Added
+- Add Railtie to project to handle concerns.
+- Added docusaurus, the actual docs will be done later. 
 
 ## [0.1.3] - 2026-06-12
 
@@ -36,14 +52,22 @@ All notable changes to this project will be documented in this file.
 - Added `history` as a context variable which are the messages to be sent to the provider.
 - Added support for nested datatypes in the `array` schema for tools
 
-# [0.1.4] - 2026-06-16
+## [0.1.2] - 2026-06-11
 
 ### Fixed
-- Change invoke method to an instance method instead of a class method, to actually catch invocations of the `stop_generation!` method.
+- Fixed Zeitwerk issue with `OmniAgent::Errors`
+
+## [0.1.1] - 2026-06-09
 
 ### Changed
-- Remove agent file from directory to prevent needing to change Zeitwerk initializer as Railtie wasn't allowing to change behaviour without it. 
+- Lowered Rails version needed.
+
+## [0.1.0] - 2026-06-09
 
 ### Added
-- Add Railtie to project to handle concerns.
-- Added docusaurus, the actual docs will be done later. 
+- Initial release of OmniAgent as a Rails engine for building application-native AI agents.
+- Agent runtime with provider abstraction, tool-calling loop, callbacks, and prompt rendering.
+- OpenAI provider support via the `openai` gem.
+- Tool DSL with JSON schema generation.
+- Rails generators and tasks for agent scaffolding.
+- RSpec coverage for agent runtime, providers, tools, and integration flows.

data/README.md

@@ -97,6 +97,90 @@ module ResearchAgent::Tools
 end
 ```
 
+## Evals
+
+`OmniAgent::Eval` lets you test agent quality: deterministic assertions (tool calls, output matching) and pluggable LLM-as-judge scoring.
+
+```ruby
+class ResearchAgentEval < OmniAgent::Eval
+	agent ResearchAgent
+
+	eval_case "answers weather question" do
+		input "What's the weather in Paris?"
+		expect_tool_call :get_weather, with: { city: "Paris" }
+		expect_output to_include: "Paris"
+	end
+
+	eval_case "is polite" do
+		input "Tell me a joke"
+		judge "Is the response friendly and on-topic?", threshold: 0.7
+	end
+
+	eval_case "summarizes via the :summarize run alias" do
+		run_alias :summarize
+		input "Some long article text...", with: { tone: "casual" }
+		expect_output to_include: "summary"
+	end
+end
+```
+
+* **`input text, with: {}`**: `with:` is forwarded as the agent's `context:`, bound to matching instance variables during the run (e.g. `with: { tone: "casual" }` sets `@tone`).
+* **`run_alias`**: Targets a method defined via `run_aliases` (or any zero-arg run entrypoint) instead of plain `#run` — useful when that alias renders a different prompt file (`<method_name>.md.erb`).
+
+Judge provider resolution order: explicit `provider:` kwarg on `judge` → `OmniAgent.configuration.eval_judge_provider`/`eval_judge_model` → the agent's own provider (zero-config default).
+
+### Caching
+
+Eval runs are cached by default, keyed on `(agent class, run_alias, input, context)`. Re-running the same case (e.g. iterating on assertions) replays the cached output instead of calling the provider again, saving tokens. Configure or disable it:
+
+```ruby
+OmniAgent.configure do |config|
+	config.eval_cache_enabled = true # default
+	config.eval_cache_path = "tmp/omni_agent_eval_cache.json" # default
+end
+```
+
+Bypass the cache for a run (clears it before running, no manual file deletion needed):
+
+```bash
+bundle exec omni_agent eval evals/research_agent_eval.rb --fresh
+```
+
+Sample output:
+
+```text
+[PASS] mentions lorem
+[FAIL] mentions something the mock never says
+  - output "Lorem ipsum dolor sit amet, consectetur adipiscing elit." does not include "this never appears"
+
+1/2 cases passed
+```
+
+Each case prints `[PASS]`/`[FAIL]` plus its name; failing cases list every unmet assertion's message. Exits non-zero if any case failed.
+
+For many input/expected-output pairs, load a YAML/JSON dataset instead of writing a `case` per row:
+
+```ruby
+golden_set "evals/golden/research_agent.yml" do |row|
+	expect_output to_include: row[:expected_output]
+end
+```
+
+Scaffold an eval and run it:
+
+```bash
+bundle exec rails generate omni_agent:eval ResearchAgent
+
+# run from your Rails app root, like running rspec
+bundle exec omni_agent eval
+bundle exec omni_agent eval evals/research_agent_eval.rb
+bundle exec omni_agent eval evals/research_agent_eval.rb --fresh
+```
+
+There's also an equivalent `rake omni_agent:eval` task (`rake "omni_agent:eval[pattern,fresh]"`) if you'd rather not use the binstub.
+
+Calls real LLM providers (cost, non-determinism) — deliberately **not** part of `bundle exec rspec` or CI.
+
 ## Configuration
 
 Global defaults can be configured through `OmniAgent.configure`:

data/exe/omni_agent

@@ -0,0 +1,42 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "optparse"
+
+options = { fresh: false }
+parser = OptionParser.new do |opts|
+  opts.banner = "Usage: omni_agent eval [pattern] [options]"
+  opts.on("--fresh", "Clear the eval cache before running") { options[:fresh] = true }
+end
+
+command = ARGV.shift
+
+case command
+when "eval"
+  parser.parse!(ARGV)
+  pattern = ARGV.shift
+
+  environment_file = File.expand_path("config/environment", Dir.pwd)
+  unless File.exist?("#{environment_file}.rb")
+    abort "Could not find config/environment.rb -- run `omni_agent eval` from your Rails app root."
+  end
+  require environment_file
+
+  status = OmniAgent::Eval::CLI.run(pattern: pattern, fresh: options[:fresh])
+
+  case status
+  when :no_files
+    abort "No eval files found matching #{(pattern || OmniAgent::Eval::CLI::DEFAULT_PATTERN).inspect}"
+  when :no_evals
+    abort "No OmniAgent::Eval subclasses found"
+  when :failed
+    exit 1
+  end
+when nil
+  warn parser.banner
+  exit 1
+else
+  warn "Unknown command: #{command.inspect}"
+  warn parser.banner
+  exit 1
+end

data/lib/generators/omni_agent/agent/agent_generator.rb

@@ -46,7 +46,7 @@ module OmniAgent
       end
 
       def agent_file_path
-        File.join(destination_root, "#{file_name}.rb")
+        File.join(destination_root, "app", "agents", "#{file_name}.rb")
       end
 
       def prompt_file_path

data/lib/generators/omni_agent/eval/eval_generator.rb

@@ -0,0 +1,37 @@
+require "rails/generators/named_base"
+require "fileutils"
+
+module OmniAgent
+  module Generators
+    class EvalGenerator < Rails::Generators::NamedBase
+      def create_eval_file
+        FileUtils.mkdir_p(evals_directory)
+
+        create_file(eval_file_path, <<~RUBY)
+          class #{eval_class_name} < OmniAgent::Eval
+            agent #{class_name}
+
+            eval_case "describe what #{class_name} should do" do
+              input "Say hello"
+              expect_output to_include: "hello"
+            end
+          end
+        RUBY
+      end
+
+      private
+
+      def evals_directory
+        File.join(destination_root, "evals")
+      end
+
+      def eval_file_path
+        File.join(evals_directory, "#{file_name}_eval.rb")
+      end
+
+      def eval_class_name
+        "#{class_name}Eval"
+      end
+    end
+  end
+end

data/lib/omni_agent/agent.rb

@@ -159,8 +159,17 @@ module OmniAgent
       initial_messages_count = messages.length - 1
 
       filtered_tools = tool_filter(tools: available_tools, agent_tags: self.class.tags)
+      max_iterations = OmniAgent.configuration.max_tool_iterations
+      iterations = 0
 
       loop do
+        iterations += 1
+        if iterations > max_iterations
+          raise OmniAgent::MaxToolIterationsError,
+                "Exceeded max_tool_iterations (#{max_iterations}) without a final response. " \
+                "Increase OmniAgent.configuration.max_tool_iterations if more tool calls are expected."
+        end
+
         response = provider.chat(messages: messages, tools: filtered_tools, **@chat_options)
 
         if response.content && !response.tool_calls?
@@ -309,7 +318,15 @@ module OmniAgent
     end
 
     def resolve_provider(name, model)
-      OmniAgent::Providers.registry[name.to_sym].new(model: model)
+      provider_class = OmniAgent::Providers.registry[name.to_sym]
+
+      unless provider_class
+        known = OmniAgent::Providers.registry.keys.join(", ")
+        raise OmniAgent::UnknownProviderError,
+              "Unknown provider #{name.inspect}. Known providers: #{known}"
+      end
+
+      provider_class.new(model: model)
     end
 
     def run_before_generation_callbacks(input:, context:, messages:)

data/lib/omni_agent/configuration.rb

@@ -1,10 +1,18 @@
 module OmniAgent
   class Configuration
-    attr_accessor :default_provider, :default_model
+    attr_accessor :default_provider, :default_model, :max_retries, :retry_base_delay, :max_tool_iterations,
+                  :eval_judge_provider, :eval_judge_model, :eval_cache_enabled, :eval_cache_path
 
     def initialize
       @default_provider = :openai
       @default_model = "gpt-4o-mini"
+      @max_retries = 3
+      @retry_base_delay = 0.5
+      @max_tool_iterations = 10
+      @eval_judge_provider = nil
+      @eval_judge_model = nil
+      @eval_cache_enabled = true
+      @eval_cache_path = "tmp/omni_agent_eval_cache.json"
     end
   end
-end
+end

data/lib/omni_agent/errors.rb

@@ -2,8 +2,14 @@ module OmniAgent
   module Errors
     class Error < StandardError; end
     class MissingDependencyError < Error; end
+    class UnknownProviderError < Error; end
+    class MaxToolIterationsError < Error; end
+    class EvalAssertionError < Error; end
   end
 
   Error = Errors::Error
   MissingDependencyError = Errors::MissingDependencyError
+  UnknownProviderError = Errors::UnknownProviderError
+  MaxToolIterationsError = Errors::MaxToolIterationsError
+  EvalAssertionError = Errors::EvalAssertionError
 end

data/lib/omni_agent/eval/cache.rb

@@ -0,0 +1,51 @@
+require "digest"
+require "fileutils"
+require "json"
+
+module OmniAgent
+  class Eval
+    module Cache
+      def self.fetch(key)
+        return yield unless OmniAgent.configuration.eval_cache_enabled
+
+        store = read_store
+        return store[key] if store.key?(key)
+
+        result = yield
+        store[key] = result
+        write_store(store)
+        result
+      end
+
+      def self.key_for(agent_class:, run_alias:, input:, context:)
+        Digest::SHA256.hexdigest(JSON.generate(
+          agent: agent_class.name,
+          run_alias: run_alias,
+          input: input,
+          context: context
+        ))
+      end
+
+      def self.clear!
+        File.delete(path) if File.exist?(path)
+      end
+
+      def self.path
+        OmniAgent.configuration.eval_cache_path
+      end
+
+      def self.read_store
+        return {} unless File.exist?(path)
+
+        JSON.parse(File.read(path))
+      rescue JSON::ParserError
+        {}
+      end
+
+      def self.write_store(store)
+        FileUtils.mkdir_p(File.dirname(path))
+        File.write(path, JSON.generate(store))
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/case.rb

@@ -0,0 +1,56 @@
+module OmniAgent
+  class Eval
+    class Case
+      attr_reader :name
+
+      def initialize(name, input: nil, context: {}, run_alias: nil, row: nil, &block)
+        @name = name
+        @input = input
+        @context = context || {}
+        @run_alias = run_alias
+        @assertions = []
+
+        if block
+          row.nil? ? instance_eval(&block) : instance_exec(row, &block)
+        end
+      end
+
+      def input(text, with: {})
+        @input = text
+        @context = with || {}
+      end
+
+      def run_alias(name)
+        @run_alias = name
+      end
+
+      def expect_tool_call(tool_name, with: nil)
+        @assertions << ToolCallAssertion.new(tool_name, with: with)
+      end
+
+      def expect_output(to_include: nil, to_match: nil, &block)
+        @assertions << OutputAssertion.new(to_include: to_include, to_match: to_match, &block)
+      end
+
+      def judge(criteria, threshold: 0.7, provider: nil, model: nil)
+        @assertions << JudgeAssertion.new(criteria, threshold: threshold, provider: provider, model: model)
+      end
+
+      def configured_input
+        @input
+      end
+
+      def configured_context
+        @context
+      end
+
+      def configured_run_alias
+        @run_alias
+      end
+
+      def configured_assertions
+        @assertions
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/case_result.rb

@@ -0,0 +1,16 @@
+module OmniAgent
+  class Eval
+    class CaseResult
+      attr_reader :case_name, :outcomes
+
+      def initialize(case_name:, outcomes:)
+        @case_name = case_name
+        @outcomes = outcomes
+      end
+
+      def passed?
+        outcomes.all?(&:passed?)
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/cli.rb

@@ -0,0 +1,24 @@
+module OmniAgent
+  class Eval
+    module CLI
+      DEFAULT_PATTERN = "evals/**/*_eval.rb"
+
+      def self.run(pattern: nil, fresh: false)
+        OmniAgent::Eval::Cache.clear! if fresh
+
+        files = Dir.glob(Rails.root.join(pattern || DEFAULT_PATTERN))
+        return :no_files if files.empty?
+
+        files.each { |file| require file }
+
+        eval_classes = ObjectSpace.each_object(Class).select { |klass| klass < OmniAgent::Eval }
+        return :no_evals if eval_classes.empty?
+
+        reports = eval_classes.map(&:run_all)
+        reports.each(&:print)
+
+        reports.all?(&:passed?) ? :passed : :failed
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/golden_set.rb

@@ -0,0 +1,24 @@
+require "yaml"
+require "json"
+
+module OmniAgent
+  class Eval
+    module GoldenSet
+      def self.load(path)
+        rows = path.to_s.end_with?(".json") ? JSON.parse(File.read(path)) : YAML.safe_load_file(path)
+        Array(rows).map { |row| deep_symbolize(row) }
+      end
+
+      def self.deep_symbolize(value)
+        case value
+        when Hash
+          value.each_with_object({}) { |(key, val), memo| memo[key.to_sym] = deep_symbolize(val) }
+        when Array
+          value.map { |item| deep_symbolize(item) }
+        else
+          value
+        end
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/judge.rb

@@ -0,0 +1,53 @@
+require "json"
+
+module OmniAgent
+  class Eval
+    class Judge
+      GRADING_PROMPT = <<~PROMPT
+        You are grading an AI agent's output against a single criterion.
+
+        Output: %<output>s
+
+        Criterion: %<criteria>s
+
+        Respond with strict JSON only, no other text: {"score": <float between 0 and 1>, "reason": "<short explanation>"}
+      PROMPT
+
+      def initialize(provider: nil, model: nil)
+        @provider = provider
+        @model = model
+      end
+
+      def call(criteria:, output:, fallback_provider: nil)
+        judge_provider = resolve_provider(fallback_provider)
+        prompt = format(GRADING_PROMPT, output: output, criteria: criteria)
+
+        response = judge_provider.chat(messages: [ { role: "user", content: prompt } ])
+        parse_score(response.content)
+      end
+
+      private
+
+      def resolve_provider(fallback_provider)
+        return @provider if @provider.is_a?(OmniAgent::Providers::Base)
+
+        provider_name = @provider || OmniAgent.configuration.eval_judge_provider
+        return fallback_provider if provider_name.nil?
+
+        provider_class = OmniAgent::Providers.registry[provider_name.to_sym]
+        unless provider_class
+          raise OmniAgent::UnknownProviderError, "Unknown judge provider #{provider_name.inspect}"
+        end
+
+        provider_class.new(model: @model || OmniAgent.configuration.eval_judge_model)
+      end
+
+      def parse_score(content)
+        data = JSON.parse(content.to_s)
+        { score: data["score"].to_f, reason: data["reason"] }
+      rescue JSON::ParserError
+        { score: 0.0, reason: "judge response was not valid JSON: #{content.inspect}" }
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/judge_assertion.rb

@@ -0,0 +1,22 @@
+module OmniAgent
+  class Eval
+    class JudgeAssertion
+      def initialize(criteria, threshold: 0.7, provider: nil, model: nil)
+        @criteria = criteria
+        @threshold = threshold
+        @judge = Judge.new(provider: provider, model: model)
+      end
+
+      def call(run)
+        verdict = @judge.call(criteria: @criteria, output: run.output, fallback_provider: run.agent.provider)
+        passed = verdict[:score] >= @threshold
+
+        Outcome.new(
+          passed: passed,
+          message: "judge score #{verdict[:score]} (threshold #{@threshold}) - #{verdict[:reason]}",
+          score: verdict[:score]
+        )
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/outcome.rb

@@ -0,0 +1,17 @@
+module OmniAgent
+  class Eval
+    class Outcome
+      attr_reader :message, :score
+
+      def initialize(passed:, message:, score: nil)
+        @passed = passed
+        @message = message
+        @score = score
+      end
+
+      def passed?
+        @passed == true
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/output_assertion.rb

@@ -0,0 +1,41 @@
+module OmniAgent
+  class Eval
+    class OutputAssertion
+      def initialize(to_include: nil, to_match: nil, &block)
+        @to_include = to_include
+        @to_match = to_match
+        @block = block
+      end
+
+      def call(run)
+        output = run.output
+
+        return call_block(output) if @block
+        return call_to_include(output) if @to_include
+        return call_to_match(output) if @to_match
+
+        raise ArgumentError, "expect_output requires to_include:, to_match:, or a block"
+      end
+
+      private
+
+      def call_block(output)
+        passed = !!@block.call(output)
+        message = passed ? "output matched custom block" : "output #{output.inspect} failed custom block"
+        Outcome.new(passed: passed, message: message)
+      end
+
+      def call_to_include(output)
+        passed = output.to_s.include?(@to_include.to_s)
+        message = passed ? "output includes #{@to_include.inspect}" : "output #{output.inspect} does not include #{@to_include.inspect}"
+        Outcome.new(passed: passed, message: message)
+      end
+
+      def call_to_match(output)
+        passed = !!(output.to_s =~ @to_match)
+        message = passed ? "output matches #{@to_match.inspect}" : "output #{output.inspect} does not match #{@to_match.inspect}"
+        Outcome.new(passed: passed, message: message)
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/report.rb

@@ -0,0 +1,37 @@
+module OmniAgent
+  class Eval
+    class Report
+      attr_reader :case_results
+
+      def initialize(case_results)
+        @case_results = case_results
+      end
+
+      def passed?
+        case_results.all?(&:passed?)
+      end
+
+      def raise_on_failure!
+        return if passed?
+
+        failures = case_results.reject(&:passed?).map(&:case_name)
+        raise OmniAgent::EvalAssertionError, "Eval cases failed: #{failures.join(', ')}"
+      end
+
+      def print(io: $stdout)
+        case_results.each do |case_result|
+          status = case_result.passed? ? "PASS" : "FAIL"
+          io.puts "[#{status}] #{case_result.case_name}"
+
+          case_result.outcomes.each do |outcome|
+            next if outcome.passed?
+            io.puts "  - #{outcome.message}"
+          end
+        end
+
+        passed_count = case_results.count(&:passed?)
+        io.puts "\n#{passed_count}/#{case_results.size} cases passed"
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/run.rb

@@ -0,0 +1,13 @@
+module OmniAgent
+  class Eval
+    class Run
+      attr_reader :output, :tool_calls, :agent
+
+      def initialize(output:, tool_calls:, agent:)
+        @output = output
+        @tool_calls = tool_calls
+        @agent = agent
+      end
+    end
+  end
+end

data/lib/omni_agent/eval/runner.rb

@@ -0,0 +1,66 @@
+require "json"
+
+module OmniAgent
+  class Eval
+    module Runner
+      def self.run(eval_case, agent_class)
+        agent = agent_class.new
+
+        cache_key = Cache.key_for(
+          agent_class: agent_class,
+          run_alias: eval_case.configured_run_alias,
+          input: eval_case.configured_input,
+          context: eval_case.configured_context
+        )
+
+        cached = Cache.fetch(cache_key) { invoke_agent(agent, eval_case) }
+        run = Run.new(output: cached["output"], tool_calls: normalize_tool_calls(cached["tool_calls"]), agent: agent)
+
+        outcomes = eval_case.configured_assertions.map { |assertion| assertion.call(run) }
+        CaseResult.new(case_name: eval_case.name, outcomes: outcomes)
+      end
+
+      def self.invoke_agent(agent, eval_case)
+        response = if eval_case.configured_run_alias
+          agent.public_send(eval_case.configured_run_alias, eval_case.configured_input, context: eval_case.configured_context)
+        else
+          agent.run(eval_case.configured_input, context: eval_case.configured_context)
+        end
+
+        { "output" => response.answer.to_s, "tool_calls" => extract_tool_calls(response) }
+      end
+
+      def self.extract_tool_calls(response)
+        response.generated_messages
+          .select { |message| (message[:role] || message["role"]) == "assistant" }
+          .flat_map { |message| message[:tool_calls] || message["tool_calls"] || [] }
+          .map { |tool_call| extract_tool_call(tool_call) }
+      end
+
+      def self.extract_tool_call(tool_call)
+        function = tool_call[:function] || tool_call["function"] || {}
+        name = function[:name] || function["name"]
+        raw_arguments = function[:arguments] || function["arguments"]
+
+        { "name" => name, "arguments" => parse_arguments(raw_arguments) }
+      end
+
+      def self.parse_arguments(raw_arguments)
+        return {} if raw_arguments.nil?
+
+        parsed = raw_arguments.is_a?(String) ? JSON.parse(raw_arguments) : raw_arguments
+        parsed.transform_keys(&:to_s)
+      rescue JSON::ParserError
+        {}
+      end
+
+      def self.normalize_tool_calls(tool_calls)
+        Array(tool_calls).map do |tool_call|
+          { name: tool_call["name"], arguments: (tool_call["arguments"] || {}).transform_keys(&:to_sym) }
+        end
+      end
+
+      private_class_method :invoke_agent, :extract_tool_calls, :extract_tool_call, :parse_arguments, :normalize_tool_calls
+    end
+  end
+end

data/lib/omni_agent/eval/tool_call_assertion.rb

@@ -0,0 +1,28 @@
+module OmniAgent
+  class Eval
+    class ToolCallAssertion
+      def initialize(tool_name, with: nil)
+        @tool_name = tool_name.to_s
+        @expected_args = with
+      end
+
+      def call(run)
+        matches = run.tool_calls.select { |tool_call| tool_call[:name] == @tool_name }
+
+        if matches.empty?
+          return Outcome.new(passed: false, message: "expected tool `#{@tool_name}` to be called, but it wasn't")
+        end
+
+        return Outcome.new(passed: true, message: "tool `#{@tool_name}` was called") if @expected_args.nil?
+
+        expected = @expected_args.transform_keys(&:to_sym)
+        if matches.any? { |tool_call| expected.all? { |key, value| tool_call[:arguments][key] == value } }
+          Outcome.new(passed: true, message: "tool `#{@tool_name}` was called with #{expected}")
+        else
+          got = matches.map { |tool_call| tool_call[:arguments] }
+          Outcome.new(passed: false, message: "tool `#{@tool_name}` was called, but not with #{expected} (got #{got})")
+        end
+      end
+    end
+  end
+end

data/lib/omni_agent/eval.rb

@@ -0,0 +1,40 @@
+module OmniAgent
+  class Eval
+    class << self
+      def agent(klass = nil)
+        @agent_class = klass if klass
+        @agent_class
+      end
+
+      def eval_case(name, &block)
+        @configured_cases = configured_cases + [ Case.new(name, &block) ]
+      end
+
+      def golden_set(path, &block)
+        new_cases = GoldenSet.load(path).each_with_index.map do |row, index|
+          case_name = row[:name] || "row #{index}"
+          Case.new(case_name, input: row[:input], context: row[:context] || {}, run_alias: row[:run_alias], row: row, &block)
+        end
+
+        @configured_cases = configured_cases + new_cases
+      end
+
+      def configured_agent
+        @agent_class
+      end
+
+      def configured_cases
+        @configured_cases || []
+      end
+
+      def run_all
+        unless configured_agent
+          raise OmniAgent::Error, "#{name} must declare `agent SomeAgentClass` before running evals"
+        end
+
+        case_results = configured_cases.map { |eval_case| Runner.run(eval_case, configured_agent) }
+        Report.new(case_results)
+      end
+    end
+  end
+end

data/lib/omni_agent/providers/base.rb

@@ -49,6 +49,29 @@ module OmniAgent
         raise NotImplementedError, "Providers must define a default model"
       end
 
+      # Override in subclasses to flag exceptions worth retrying (rate limits, timeouts, 5xx).
+      def retryable_error?(_error)
+        false
+      end
+
+      def with_retries
+        max_retries = OmniAgent.configuration.max_retries
+        base_delay = OmniAgent.configuration.retry_base_delay
+        attempt = 0
+
+        begin
+          yield
+        rescue => e
+          raise unless retryable_error?(e)
+
+          attempt += 1
+          raise if attempt > max_retries
+
+          sleep(base_delay * (2**(attempt - 1)))
+          retry
+        end
+      end
+
       def validate_message_payload!(message, index, role_name)
         has_content = message.key?(:content) || message.key?("content")
         content_value = message[:content] || message["content"]

data/lib/omni_agent/providers/mock_judge.rb

@@ -0,0 +1,31 @@
+module OmniAgent
+  module Providers
+    class MockJudge < Base
+      CANNED_SCORE = 1.0
+      CANNED_REASON = "mock judge always approves"
+
+      def chat(messages:, tools: [], **_options)
+        validate_messages!(messages, allowed_roles: %i[system user assistant tool])
+
+        content = { score: CANNED_SCORE, reason: CANNED_REASON }.to_json
+
+        OmniAgent::Providers::Response.new(
+          content: content,
+          raw_request: { model: model, messages: messages, tools: tools },
+          raw_response: { "choices" => [ { "message" => { "content" => content } } ] },
+          tool_calls: []
+        )
+      end
+
+      protected
+
+      def default_api_key
+        nil
+      end
+
+      def default_model
+        "mock-judge"
+      end
+    end
+  end
+end

data/lib/omni_agent/providers/openai.rb

@@ -44,7 +44,7 @@ module OmniAgent
         payload[:tools] = openai_tools if openai_tools.any?
         payload.merge!(options) if options.any?
 
-        response = client.chat.completions.create(**payload)
+        response = with_retries { client.chat.completions.create(**payload) }
 
         parse_response(response, payload)
       rescue => e
@@ -53,6 +53,13 @@ module OmniAgent
 
       protected
 
+      def retryable_error?(error)
+        defined?(::OpenAI::Errors) &&
+          (error.is_a?(::OpenAI::Errors::RateLimitError) ||
+           error.is_a?(::OpenAI::Errors::InternalServerError) ||
+           error.is_a?(::OpenAI::Errors::APIConnectionError))
+      end
+
       def client
         @client ||= ::OpenAI::Client.new(api_key: @api_key)
       end

data/lib/omni_agent/providers.rb

@@ -4,7 +4,8 @@ module OmniAgent
     def self.registry
       {
         openai: OmniAgent::Providers::OpenAI,
-        mock: OmniAgent::Providers::Mock
+        mock: OmniAgent::Providers::Mock,
+        mock_judge: OmniAgent::Providers::MockJudge
       }
     end
   end

data/lib/omni_agent/tasks/omni_agent_tasks.rake

@@ -1,56 +1,56 @@
 require "fileutils"
 
 namespace :omni_agent do
-	desc "Set up OmniAgent directories in your Rails app"
-	task install: :environment do
-		agents_root = Rails.root.join("app", "agents")
-		FileUtils.mkdir_p(agents_root)
+  desc "Set up OmniAgent directories in your Rails app"
+  task install: :environment do
+    agents_root = Rails.root.join("app", "agents")
+    FileUtils.mkdir_p(agents_root)
 
-		puts "Created #{agents_root}"
-		puts "OmniAgent install complete."
-		puts "Next: bundle exec rake \"omni_agent:create_agent[ResearchAgent]\""
-	end
+    puts "Created #{agents_root}"
+    puts "OmniAgent install complete."
+    puts "Next: bundle exec rake \"omni_agent:create_agent[ResearchAgent]\""
+  end
 
-	desc "Create an agent scaffold. Usage: rake \"omni_agent:create_agent[ResearchAgent]\" [-- --with-tools]"
-	task :create_agent, [:name] => :environment do |_task, args|
-		agent_name = args[:name].to_s.strip
+  desc "Create an agent scaffold. Usage: rake \"omni_agent:create_agent[ResearchAgent]\" [-- --with-tools]"
+  task :create_agent, [ :name ] => :environment do |_task, args|
+    agent_name = args[:name].to_s.strip
 
-		if agent_name.empty?
-			abort "Please provide an agent name. Example: bundle exec rake \"omni_agent:create_agent[ResearchAgent]\""
-		end
+    if agent_name.empty?
+      abort "Please provide an agent name. Example: bundle exec rake \"omni_agent:create_agent[ResearchAgent]\""
+    end
 
-		with_tools = ARGV.include?("--with-tools")
+    with_tools = ARGV.include?("--with-tools")
 
-		agent_file_name = "#{agent_name.underscore}.rb"
-		agent_dir_name = agent_name.underscore
-		agent_dir = Rails.root.join("app", "agents", agent_dir_name)
-		tools_dir = agent_dir.join("tools")
-		prompt_file = agent_dir.join("prompt.md.erb")
-		agent_rb = agent_dir.join(agent_file_name)
+    agent_file_name = "#{agent_name.underscore}.rb"
+    agent_dir_name = agent_name.underscore
+    agent_dir = Rails.root.join("app", "agents", agent_dir_name)
+    tools_dir = agent_dir.join("tools")
+    prompt_file = agent_dir.join("prompt.md.erb")
+    agent_rb = agent_dir.join(agent_file_name)
 
-		FileUtils.mkdir_p(agent_dir)
+    FileUtils.mkdir_p(agent_dir)
 
-		if File.exist?(agent_rb)
-			abort "Agent file already exists: #{agent_rb}"
-		end
+    if File.exist?(agent_rb)
+      abort "Agent file already exists: #{agent_rb}"
+    end
 
-		File.write(agent_rb, <<~RUBY)
+    File.write(agent_rb, <<~RUBY)
 			class #{agent_name} < OmniAgent::Agent
 			end
-		RUBY
+    RUBY
 
-		unless File.exist?(prompt_file)
-			File.write(prompt_file, <<~PROMPT)
+    unless File.exist?(prompt_file)
+      File.write(prompt_file, <<~PROMPT)
 				You are #{agent_name}, a helpful assistant with access to local tools.
-			PROMPT
-		end
+      PROMPT
+    end
 
-		if with_tools
-			FileUtils.mkdir_p(tools_dir)
+    if with_tools
+      FileUtils.mkdir_p(tools_dir)
 
-			example_tool = tools_dir.join("example_tool.rb")
-			unless File.exist?(example_tool)
-				File.write(example_tool, <<~RUBY)
+      example_tool = tools_dir.join("example_tool.rb")
+      unless File.exist?(example_tool)
+        File.write(example_tool, <<~RUBY)
 					module #{agent_name}::Tools
 						class ExampleTool < OmniAgent::Tool
 							description "Example tool generated by omni_agent:create_agent"
@@ -66,13 +66,13 @@ namespace :omni_agent do
 							end
 						end
 					end
-				RUBY
-			end
-		end
+        RUBY
+      end
+    end
 
-		puts "Created #{agent_rb}"
-		puts "Created #{prompt_file}" if File.exist?(prompt_file)
-		puts "Created #{tools_dir}" if with_tools
-		puts "Created #{tools_dir.join("example_tool.rb")}" if with_tools && File.exist?(tools_dir.join("example_tool.rb"))
-	end
+    puts "Created #{agent_rb}"
+    puts "Created #{prompt_file}" if File.exist?(prompt_file)
+    puts "Created #{tools_dir}" if with_tools
+    puts "Created #{tools_dir.join("example_tool.rb")}" if with_tools && File.exist?(tools_dir.join("example_tool.rb"))
+  end
 end

data/lib/omni_agent/version.rb

@@ -1,3 +1,3 @@
 module OmniAgent
-  VERSION = "0.1.4"
+  VERSION = "0.1.6"
 end

data/lib/omni_agent.rb

@@ -3,10 +3,16 @@ require "omni_agent/engine" if defined?(Rails)
 require "zeitwerk"
 
 loader = Zeitwerk::Loader.for_gem
-loader.inflector.inflect("openai" => "OpenAI")
+loader.inflector.inflect("openai" => "OpenAI", "cli" => "CLI")
 loader.ignore(File.expand_path("generators", __dir__))
 loader.setup
 
+# errors.rb defines top-level aliases (OmniAgent::MissingDependencyError, etc.)
+# that Zeitwerk cannot autoload, since they don't map to a file path of their
+# own. Force it to load eagerly so the aliases exist regardless of which
+# constant is referenced first.
+require "omni_agent/errors"
+
 
 module OmniAgent
   class << self

data/lib/tasks/omni_agent_eval_tasks.rake

@@ -0,0 +1,19 @@
+namespace :omni_agent do
+  desc 'Run evals. Usage: rake omni_agent:eval or rake "omni_agent:eval[evals/research_agent_eval.rb,fresh]". Pass `fresh` as the 2nd arg, or set FRESH=1, to bypass the eval cache.'
+  task :eval, [ :pattern, :fresh ] => :environment do |_task, args|
+    fresh = ENV["FRESH"] == "1" || args[:fresh] == "fresh"
+    pattern = args[:pattern].to_s.strip
+    pattern = nil if pattern.empty?
+
+    status = OmniAgent::Eval::CLI.run(pattern: pattern, fresh: fresh)
+
+    case status
+    when :no_files
+      abort "No eval files found matching #{(pattern || OmniAgent::Eval::CLI::DEFAULT_PATTERN).inspect}"
+    when :no_evals
+      abort "No OmniAgent::Eval subclasses found"
+    when :failed
+      exit 1
+    end
+  end
+end

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: omni_agent
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.1.6
 platform: ruby
 authors:
 - ACR1209
-autorequire: 
-bindir: bin
+bindir: exe
 cert_chain: []
-date: 2026-06-17 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -42,7 +41,8 @@ description: OmniAgent provides a Rails-native framework for defining AI agents,
   schemas, prompt templates, callbacks, and provider-backed generation workflows.
 email:
 - andrescoronel1209@gmail.com
-executables: []
+executables:
+- omni_agent
 extensions: []
 extra_rdoc_files: []
 files:
@@ -58,16 +58,33 @@ files:
 - app/models/omni_agent/application_record.rb
 - app/views/layouts/omni_agent/application.html.erb
 - config/routes.rb
+- exe/omni_agent
 - lib/generators/omni_agent/agent/agent_generator.rb
+- lib/generators/omni_agent/eval/eval_generator.rb
 - lib/generators/omni_agent/install/install_generator.rb
 - lib/omni_agent.rb
 - lib/omni_agent/agent.rb
 - lib/omni_agent/configuration.rb
 - lib/omni_agent/engine.rb
 - lib/omni_agent/errors.rb
+- lib/omni_agent/eval.rb
+- lib/omni_agent/eval/cache.rb
+- lib/omni_agent/eval/case.rb
+- lib/omni_agent/eval/case_result.rb
+- lib/omni_agent/eval/cli.rb
+- lib/omni_agent/eval/golden_set.rb
+- lib/omni_agent/eval/judge.rb
+- lib/omni_agent/eval/judge_assertion.rb
+- lib/omni_agent/eval/outcome.rb
+- lib/omni_agent/eval/output_assertion.rb
+- lib/omni_agent/eval/report.rb
+- lib/omni_agent/eval/run.rb
+- lib/omni_agent/eval/runner.rb
+- lib/omni_agent/eval/tool_call_assertion.rb
 - lib/omni_agent/providers.rb
 - lib/omni_agent/providers/base.rb
 - lib/omni_agent/providers/mock.rb
+- lib/omni_agent/providers/mock_judge.rb
 - lib/omni_agent/providers/openai.rb
 - lib/omni_agent/providers/response.rb
 - lib/omni_agent/railtie.rb
@@ -75,6 +92,7 @@ files:
 - lib/omni_agent/tool.rb
 - lib/omni_agent/tool/schema_builder.rb
 - lib/omni_agent/version.rb
+- lib/tasks/omni_agent_eval_tasks.rake
 homepage: https://github.com/ACR1209/omni_agent
 licenses:
 - MIT
@@ -82,7 +100,6 @@ metadata:
   homepage_uri: https://github.com/ACR1209/omni_agent
   source_code_uri: https://github.com/ACR1209/omni_agent
   changelog_uri: https://github.com/ACR1209/omni_agent/blob/main/CHANGELOG.md
-post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
@@ -97,8 +114,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.11
-signing_key: 
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Rails engine for building AI agents with tools.
 test_files: []