ruby_llm-test 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6320355012e11036b37b8a13d6a7bd5782e573a1e7ab1a94601d67836e3e02ab
-  data.tar.gz: 566e774f1e9ac7fb636b5f05eeef52fc27565eb2d9fb9997f0f14b6630ba67c0
+  metadata.gz: ef753ca266479d9530fb44f0660cb2292bc916c2c949376c65686488bd41f5d9
+  data.tar.gz: f96175266a8866d9038b529b37ae0f099f1b20cdf677f591ab36acc941e566bb
 SHA512:
-  metadata.gz: 4a5b97108962cfa24230b4733e4357f2042ed6240dbe934d466567c102aadfee07224d0762b0105121c4092879d8ae5ef8f98ef7e76146170e366057aa09f15d
-  data.tar.gz: e30334a2adf81c0faeb482daf76a96dda06efa7664f567c2778a6aafd1e02c934686f33f9302f6533eae184f390c7f7bba7355e5b0fd09dc2408ff79276282eb
+  metadata.gz: e58becb7a4ae7b1d2248746f8aad685aa5a382307fe8c57887d0b6d2c170e35ffc4e97ae095cc0601ebf17e50de3374cab6060619c6ba57adfed4c85f5be576b
+  data.tar.gz: 76456e9ec3ed46ec76eafbae1bdebe6fd9598cce423f527c505028a43520dfa01c538341fd28c71fddfd70f36d7c47410b2cfb2fb8425fa6ba0abe6ccc6aac55

data/README.md

@@ -1,10 +1,27 @@
+[![Gem Version](https://badge.fury.io/rb/ruby_llm-test.svg)](https://badge.fury.io/rb/ruby_llm-test)
+[![Tests](https://github.com/RockSolt/ruby_llm-test/actions/workflows/test.yml/badge.svg)](https://github.com/RockSolt/ruby_llm-test/actions/workflows/test.yml)
+[![RuboCop](https://github.com/RockSolt/ruby_llm-test/actions/workflows/rubocop.yml/badge.svg)](https://github.com/RockSolt/ruby_llm-test/actions/workflows/rubocop.yml)
+
 # RubyLLM::Test
 
-This gem provides testing utilities for RubyLLM, a Ruby library for working with large language models (LLMs). It enables calls to LLM's to be stubbed so that the surrounding application logic can be tested without making actual calls to the LLM. This is particularly useful for testing code that interacts with LLMs, as it allows developers to simulate responses from the LLM without incurring the cost, latency, or randomness of real API calls.
+How do you test your business logic when it interacts with large language models (LLMs)? From a testing point of view, an LLM is just an external system. To keep tests fast and consistent, pick a boundary then stub or mock the external system.
+
+Part of the appeal of the RubyLLM library is that code using it does not need to know the specifics of provider APIs. The RubyLLM::Test gem brings the same benefit to your tests. No API or provider-specific knowledge is required to stub calls.
+
+Simple stubs make for simple tests. Keep your tests fast, consistent, and thorough with RubyLLM::Test!
+
+```ruby
+RubyLLM::Test.stub_response("Outlook good")
+
+chat = RubyLLM.chat
+response = chat.ask "What are the odds this works?"
+
+assert_equal "Outlook good", response.content
+```
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Add this line to your application's Gemfile in the test group:
 
 ```ruby
 gem 'ruby_llm-test'
@@ -20,7 +37,7 @@ Or install it yourself as:
 
 ## Usage
 
-Add the following lines to your `spec/spec_helper.rb` or `test/test_helper.rb`:
+Add the following lines to your `spec_helper.rb` or `test_helper.rb`:
 
 ```ruby
 require 'ruby_llm/test'
@@ -34,21 +51,22 @@ Then, in your tests, you can use the `stub_response` method to stub responses fr
 it 'returns a stubbed response' do
   RubyLLM::Test.stub_response('Hello, world!')
 
-  response = MyLLMClient.call('Hello?')
-  expect(response).to eq('Hello, world!')
+  response = RubyLLM.chat.ask 'Hello?'
+  expect(response.content).to eq('Hello, world!')
 end
 ```
 
-If you make multiple calls to the LLM, you can call `stub_response` more than once or use method `stub_responses` to stub multiple responses at once. For example:
+If you make multiple calls to the LLM, you can call `stub_response` more than once or use the `stub_responses` method to stub multiple responses at once. For example:
 
 ```ruby
 it 'returns multiple stubbed responses' do
-  RubyLLM::Test.stub_responses('Hello, world!', 'How are you?')
-  response1 = MyLLMClient.call('Hello?')
-  response2 = MyLLMClient.call('How are you?')
+  RubyLLM::Test.stub_responses('Blue.', 'No, yellow.')
+  chat = RubyLLM.chat
+  response1 = chat.ask 'What is your favorite color?'
+  response2 = chat.ask 'Are you sure?'
 
-  expect(response1).to eq('Hello, world!')
-  expect(response2).to eq('How are you?')
+  expect(response1.content).to eq('Blue.')
+  expect(response2.content).to eq('No, yellow.')
 end
 ```
 
@@ -61,7 +79,7 @@ it 'returns a stubbed message' do
   message = RubyLLM::Message.new(role: :assistant, content: 'Hello, world!')
   RubyLLM::Test.stub_response(message)
 
-  response = MyLLMClient.call('Hello?')
+  response = RubyLLM.chat.ask 'Hello?'
   expect(response).to eq(message)
 end
 ```
@@ -75,14 +93,14 @@ it 'returns a stubbed JSON message' do
   hash = { key: 'value' }
   RubyLLM::Test.stub_response(hash)
 
-  response = MyLLMClient.call('Hello?')
+  response = RubyLLM.chat.ask 'Hello?'
   expect(response.content).to eq(hash.to_json)
 end
 ```
 
 ### Resetting Stubs
 
-Make sure to reset stubs after each test to avoid interference before or between tests.
+Reset stubs before each test to ensure a clean slate.
 
 ```ruby
 RubyLLM::Test.reset
@@ -93,13 +111,29 @@ RubyLLM::Test.reset
 You can also stub responses in a block, which handles the setup and teardown of stubs automatically. For example:
 
 ```ruby
-
 RubyLLM::Test.with_responses('Hello, world!') do
-  response = MyLLMClient.call('Hello?')
-  expect(response).to eq('Hello, world!')
+  response = RubyLLM.chat.ask 'Hello?'
+  expect(response.content).to eq('Hello, world!')
 end
 ```
 
+### Testing Arguments
+
+You can verify arguments passed to the LLM by checking the requests received by the test provider with methods `requests` and `last_request`.
+
+```ruby
+RubyLLM::Test.stub_response('Hello, world!')
+chat = RubyLLM.chat(model: 'gpt-5-nano')
+chat.with_tools(GreeterTool)
+chat.ask('Hello?')
+request = RubyLLM::Test.last_request
+
+assert_equal 'gpt-5-nano', request.model
+assert_includes request.tool_classes, GreeterTool
+```
+
+Any parameters passed to the Provider's `complete` method are available on the request object. The `tool_classes` method is a helper that returns the classes of any tools included in the request.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/lib/ruby_llm/test/complete_parameters.rb

@@ -2,27 +2,69 @@
 
 module RubyLLM
   module Test
-    # This class encapsulates all the parameters that are passed to the `complete` method of the `Test` class. It
-    # serves as a structured way to manage and access these parameters throughout the testing process.
+    # This class encapsulates all the parameters that are passed to the `complete` method of the wrapped provider. It
+    # stores the raw arguments and uses the wrapped provider's actual method signature to expose named accessors for
+    # inspection in tests.
     class CompleteParameters
-      attr_reader :messages, :tools, :temperature, :model, :params, :headers, :schema, :thinking, :tool_prefs, :block
-
-      def initialize(messages:, tools:, temperature:, model:, params:, # rubocop:disable Metrics/ParameterLists
-                     headers:, schema:, thinking:, tool_prefs:, block:)
-        @messages = messages
-        @tools = tools
-        @temperature = temperature
-        @model = model
-        @params = params
-        @headers = headers
-        @schema = schema
-        @thinking = thinking
-        @tool_prefs = tool_prefs
+      attr_reader :args, :kwargs, :block, :parameter_definitions
+
+      def self.capture_from(provider, *args, **kwargs, &block)
+        parameters = provider.method(:complete).parameters
+        new(args:, kwargs:, block:, parameter_definitions: parameters)
+      end
+
+      def initialize(args:, kwargs:, block:, parameter_definitions:)
+        @args = args
+        @kwargs = kwargs
         @block = block
+        @parameter_definitions = parameter_definitions
       end
 
       def block_received?
-        !@block.nil?
+        !block.nil?
+      end
+
+      def [](name)
+        name = name.to_sym
+        return block if name == :block
+
+        positional_name_to_value[name] || kwargs[name]
+      end
+
+      def key?(name)
+        name = name.to_sym
+        name == :block || positional_name_to_value.key?(name) || kwargs.key?(name)
+      end
+
+      def to_h
+        positional_name_to_value.merge(kwargs).merge(block: block)
+      end
+
+      def method_missing(name, *call_args)
+        return super unless call_args.empty?
+        return self[name] if key?(name)
+
+        super
+      end
+
+      def respond_to_missing?(name, include_private = false)
+        key?(name) || super
+      end
+
+      def tool_classes
+        (kwargs[:tools] || {}).values.map(&:class)
+      end
+
+      private
+
+      def positional_name_to_value
+        @positional_name_to_value ||= begin
+          positional_names = parameter_definitions
+                             .select { |parameter| %i[req opt].include?(parameter.first) }
+                             .map(&:last)
+
+          positional_names.each_with_index.to_h { |param_name, index| [ param_name, args[index] ] }
+        end
       end
     end
   end

data/lib/ruby_llm/test/harness.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Test
+    # This class serves as a test harness for stubbing responses and recording requests sent to the provider.
+    # It allows tests to set up expected responses and then verify that the provider received the correct
+    # parameters.
+    class Harness
+      attr_reader :requests
+
+      def initialize
+        reset
+      end
+
+      # Pass in a RubyLLM::Message to have full control; strings and hashes will be wrapped in a message
+      def stub_response(body)
+        responses << body
+      end
+
+      def stub_responses(*bodies)
+        responses.concat(bodies)
+      end
+
+      def with_responses(*bodies)
+        previous_responses = responses.dup
+        @responses = []
+        stub_responses(*bodies)
+        yield
+      ensure
+        @responses = previous_responses
+      end
+
+      def next_response
+        responses.shift
+      end
+
+      def responses_empty?
+        responses.empty?
+      end
+
+      def record_request(request)
+        requests << request
+      end
+
+      def last_request
+        requests.last
+      end
+
+      def clear_requests
+        requests.clear
+      end
+
+      def reset
+        @responses = []
+        @requests = []
+      end
+
+      private
+
+      def responses
+        @responses ||= []
+      end
+    end
+  end
+end

data/lib/ruby_llm/test/resolve_with_test_provider.rb

@@ -12,7 +12,7 @@ module RubyLLM
     module ResolveWithTestProvider
       def resolve(...)
         model, provider_instance = super
-        [ model, Test::TestProvider.new(provider_instance, RubyLLM::Test) ]
+        [ model, Test::TestProvider.new(provider_instance, RubyLLM::Test.send(:harness)) ]
       end
     end
   end

data/lib/ruby_llm/test/test_provider.rb

@@ -6,24 +6,21 @@ module RubyLLM
     #  the `complete` method, allowing tests to assert that the correct parameters were passed and to simulate
     #  responses from the provider.
     class TestProvider < SimpleDelegator
-      extend Forwardable
-
-      attr_reader :complete_calls
-
-      def_delegators :last_call, :messages, :tools, :temperature, :model, :params, :headers, :schema, :thinking,
-                     :tool_prefs, :block_received?
-
-      def initialize(provider, test_harness = RubyLLM::Test)
+      def initialize(provider, test_harness)
         super(provider)
         @test_harness = test_harness
-        @complete_calls = []
       end
 
-      def complete(messages, tools:, temperature:, model:, params: {}, # rubocop:disable Metrics/ParameterLists
-                   headers: {}, schema: nil, thinking: nil, tool_prefs: nil, &block)
-        @complete_calls << CompleteParameters.new(messages:, tools:, temperature:, model:, params:, headers:, schema:,
-                                                  thinking:, tool_prefs:, block:)
-        raise Errors::NoResponseProvidedError, messages if @test_harness.responses_empty?
+      # There are a number of cases that access `provider.class.display_name`, which avoids the delegation. Adding
+      # dummy stub for this. TBD if this needs to be more sophisticated in the future.
+      def self.display_name
+        "TestProvider"
+      end
+
+      def complete(...)
+        parameters = CompleteParameters.capture_from(__getobj__, ...)
+        @test_harness.record_request(parameters)
+        raise Errors::NoResponseProvidedError, parameters.messages if @test_harness.responses_empty?
 
         response = @test_harness.next_response
         return response if response.is_a?(Message)
@@ -33,10 +30,6 @@ module RubyLLM
           content: response.is_a?(Hash) ? response.to_json : response
         )
       end
-
-      def last_call
-        @complete_calls.last
-      end
     end
   end
 end

data/lib/ruby_llm/test/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module Test
-    VERSION = "0.1.0"
+    VERSION = "0.3.0"
   end
 end

data/lib/ruby_llm/test.rb

@@ -7,43 +7,115 @@ loader = Zeitwerk::Loader.for_gem_extension(RubyLLM)
 loader.setup
 
 module RubyLLM
-  # The Test module provides a simple way to stub responses from an LLM for testing purposes. You can use it to set up
-  #  predetermined responses that your tests can rely on, allowing you to test your code's behavior without making
-  #  actual calls to an LLM.
+  # The Test module provides a simple way to stub responses from an LLM for testing purposes.
+  # You can use it to set up predetermined responses that your tests can rely on, allowing
+  # you to test your code's behavior without making actual calls to an LLM.
   module Test
     class << self
+      # Reset the test harness state.
+      #
+      # This clears all queued stubbed responses and all recorded requests.
+      #
+      # Use this at the start of a test, or whenever you want to ensure no state
+      # carries over from a previous example.
       def reset
-        @responses = nil
+        harness.reset
       end
 
-      # Pass in a RubyLLM::Message to have full control; strings and hashes will be wrapped in a message
+      # Queue a single stubbed response.
+      #
+      # Parameters:
+      #
+      # - `body`: The response to queue. Pass a `RubyLLM::Message` to control the
+      #   message directly. Strings and hashes are wrapped in a message
+      #   automatically.
+      #
+      # This is useful when your test only needs one response from the model.
+      #
+      # Example:
+      #
+      #     RubyLLM::Test.stub_response("Hello from the test harness!")
+      #
+      #     chat = RubyLLM.chat
+      #     response = chat.ask("Say hello")
+      #     response.content
+      #     # => "Hello from the test harness!"
       def stub_response(body)
-        responses << body
+        harness.stub_response(body)
       end
 
+      # Queue multiple stubbed responses.
+      #
+      # Parameters:
+      #
+      # - `*bodies`: One or more responses to queue.
+      #
+      # Responses are returned in the same order they are provided, making this
+      # useful for tests that perform multiple LLM calls in sequence.
+      #
+      # Example:
+      #
+      #     RubyLLM::Test.stub_responses("First reply", "Second reply")
+      #
+      #     chat = RubyLLM.chat
+      #     first_response = chat.ask("First question")
+      #     second_response = chat.ask("Second question")
+      #
+      #     first_response.content
+      #     # => "First reply"
+      #     second_response.content
+      #     # => "Second reply"
       def stub_responses(*bodies)
-        responses.concat(bodies)
+        harness.stub_responses(*bodies)
       end
 
-      def with_responses(*bodies)
-        previous_responses = responses.dup
-        @responses = []
-        stub_responses(*bodies)
-        yield
-      ensure
-        @responses = previous_responses
+      # Run a block with a temporary set of stubbed responses.
+      #
+      # Parameters:
+      #
+      # - `*bodies`: The responses to make available inside the block.
+      # - `&block`: The code to run while those responses are active.
+      #
+      # The provided responses are available only for the duration of the block.
+      # This is useful when you want to scope stubbed responses to a single part
+      # of a test without affecting later assertions.
+      #
+      # Example:
+      #
+      #     RubyLLM::Test.with_responses("Scoped reply") do
+      #       chat = RubyLLM.chat
+      #       chat.ask("Question")
+      #     end
+      def with_responses(*bodies, &)
+        harness.with_responses(*bodies, &)
       end
 
-      def next_response
-        responses.shift
+      # Return all recorded requests.
+      #
+      # This is useful for assertions about prompts, parameters, or other request
+      # details captured by the harness.
+      def requests
+        harness.requests
       end
 
-      def responses_empty?
-        responses.empty?
+      # Return the most recent request.
+      #
+      # Use this when you only care about the latest request made during a test.
+      def last_request
+        harness.last_request
       end
 
-      def responses
-        @responses ||= []
+      # Clear all recorded requests.
+      #
+      # This leaves queued responses unchanged.
+      def clear_requests
+        harness.clear_requests
+      end
+
+      private
+
+      def harness
+        @harness ||= Harness.new
       end
     end
   end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-test
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Todd Kummer
 bindir: bin
 cert_chain: []
-date: 2026-04-22 00:00:00.000000000 Z
+date: 2026-09-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ruby_llm
@@ -15,14 +15,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.14.0
+        version: 1.5.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.14.0
+        version: 1.5.0
 description: Provides a RubyLLM::Provider that allows you to stub responses for testing
   purposes. You can stub individual responses or a sequence of responses, and you
   can also temporarily stub responses within a block.
@@ -37,13 +37,16 @@ files:
 - lib/ruby_llm/test.rb
 - lib/ruby_llm/test/complete_parameters.rb
 - lib/ruby_llm/test/errors/no_response_provided_error.rb
+- lib/ruby_llm/test/harness.rb
 - lib/ruby_llm/test/resolve_with_test_provider.rb
 - lib/ruby_llm/test/test_provider.rb
 - lib/ruby_llm/test/version.rb
 homepage: https://github.com/RockSolt/ruby_llm-test
 licenses:
 - MIT
-metadata: {}
+metadata:
+  rubygems_mfa_required: 'true'
+  homepage_uri: https://github.com/RockSolt/ruby_llm-test
 rdoc_options: []
 require_paths:
 - lib