activeagent 1.7.0 → 1.7.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4aee77fd4f67a50824b7153b66f31aa5f19293bad4397e032ec0a9e674a759d0
-  data.tar.gz: e261e8ca9d221edbc849791ed28c67bc491ccc31e1e18f5c4cbf7a9a96a806c5
+  metadata.gz: 881ca4e27488bd719dda01c80e6241bf6d17082876039033c22f9265d1e559aa
+  data.tar.gz: 62526259e8a48d5243a480cf3ac6f795c1672b20e8569f7804fcf13af4d25ec4
 SHA512:
-  metadata.gz: f5779f6207e07b1aec0ef14f72a5b010075b953c9025a12f6231ba164522c8b785cd9ca1bb297361abedc42aa535c468ef642978ec9fb439c79c9c0fe71ebd79
-  data.tar.gz: 1d2720a47e5e8c2a6c17a74cff893f1da2e89dd430b295bdcda4b9bdfb966afa8e75fbf95c0f063f5bca0bc8406d782e08b99fa6bed99b20230fb793ac3e518b
+  metadata.gz: 0d135cc35623442cdf38df2d588e323fec82f7e16c7d9c8fcf90d2add91bbb55476e07fe548c70ddb2b22cc709e684ab8ccc53dd337c0650800f9d6dd268f547
+  data.tar.gz: 76a0b25ef5cf125e99399ff61cef131d2fe6ae6dade18173601f73f0ec7ecedde6bfc0738af99ac490444b1a067cd136dc2caac34cc1f1cd6c01996cbdddf841

data/CHANGELOG.md

@@ -7,6 +7,104 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.7.2] - 2026-09-29
+
+Releases `activeagent` and `actionagent` 1.7.2 from one tag. A patch on 1.7.1:
+the RubyLLM provider runs on ruby_llm 1.16 and 2.x, where 1.7.1 required 1.x.
+Apps on the RubyLLM provider need ruby_llm 1.16 or later; earlier 1.x
+releases lack the APIs the provider calls. No migrations.
+
+### Changed
+
+- **The RubyLLM provider supports ruby_llm 1.16 and 2.x** (`activeagent`).
+  The adapter handles 2.x's tool interface, token limits, embedding model
+  objects, usage and finish reasons while keeping the 1.16 API working.
+  Requiring `>= 1.16, < 3` prevents Bundler from selecting an older 1.x
+  release without the APIs the adapter calls. Rails main can now resolve
+  RubyLLM 2.x alongside Active Storage's Marcel 2 dependency. CI also runs
+  the full suite with RubyLLM 1.16 to retain coverage of that version.
+  Unsupported-version errors name the loaded version and both bounds (#508).
+
+## [1.7.1] - 2026-09-29
+
+Releases `activeagent` and `actionagent` 1.7.1 from one tag. A patch on 1.7.0:
+the Evaluations page picks the judge and compared models from the provider
+catalogs, `GET /api/provider_models` lists the host's RubyLLM registry, and
+the RubyLLM provider requires ruby_llm 1.x and sends tool calls and structured
+output in the shape ruby_llm reads. No migrations.
+
+### Added
+
+- **Model pickers on the Evaluations page** (`actionagent`). The judge model
+  and the models to compare, on the new-evaluation form and on a scenario
+  suite, are chosen from type-ahead suggestions, and a model the suggestions
+  lack can still be typed. The compare fields hold one removable chip per
+  model and still submit the same comma-separated list.
+  - With scenarios, the suggestions are the catalogs the agent builder uses,
+    for the providers the owner's runs have credentials for. A model whose
+    name alone would run elsewhere is offered with its provider in front, as
+    `ModelSpec.parse` reads it: `ollama/llama3.2`, or
+    `openrouter/anthropic/claude-sonnet-4.5` for OpenRouter's copy of an
+    Anthropic model.
+  - Without scenarios, a compared model selects the generations recorded
+    under its name, usually the provider's dated id, so the suggestions are
+    the names the agent's generations were recorded under, from the new
+    `GET /api/agents/:id/recorded_models`. Adding or clearing the scenarios
+    renames the catalog models already chosen to match.
+  - The judge field suggests the models of the provider the judge runs on
+    and says which provider that is, or that the credentials deciding it
+    could not be read.
+
+  `GET /api/evaluations` reports that provider as `judge_provider` and the
+  providers runs can use as `model_providers`
+  (`AgentExecutionService.available_providers`). Runs and their judge use
+  the evaluated agent's owner's credentials, so both fields describe that
+  owner's when `agent_id` scopes the list, and the signed-in owner's
+  otherwise. `judge_provider` is null when no provider has credentials, and
+  also when reading them raised, which `judge_provider_error: true` marks.
+  `model_providers` leaves out a provider whose credentials cannot be read.
+  Either way the list still loads.
+- **`GET /api/provider_models` lists the host's RubyLLM registry**
+  (`actionagent`). When the host app loads RubyLLM, the chat models its
+  registry lists for the provider that take and return text (RubyLLM's
+  bundled catalog, or the host's own model table) follow the live or curated
+  list, each once, so the builder's preselected default is unchanged. That
+  leaves out the speech, transcription, moderation and completion-only
+  models RubyLLM counts as chat models, which its registry lists with no
+  modalities or as taking audio, and with them the few chat models it lists
+  with no modalities, which can still be typed. A registry that raises is
+  logged and leaves the list as it was.
+
+### Changed
+
+- **The RubyLLM provider requires ruby_llm 1.x** (`activeagent`). ruby_llm
+  2.0 renamed the APIs `RubyLLMProvider` calls, and the open `>= 1.0`
+  requirement let `bundle update` install it. The provider now requires
+  `~> 1.0` until it supports 2.0 (#502). Loading it with an unsupported
+  version names the supported range and the loaded version, instead of
+  asking for a gem that is already in the Gemfile. Pin
+  `gem "ruby_llm", "~> 1.0"` if your bundle resolved 2.0.
+
+### Fixed
+
+- **Tool calls sent back through the RubyLLM provider** (`activeagent`).
+  After a tool ran, the follow-up request repeated the model's tool call
+  with its arguments as a JSON string where ruby_llm expects a Hash: OpenAI
+  received them JSON-encoded twice, and Anthropic received a string for
+  `tool_use.input`, which its API requires to be an object. The same
+  happened when a stored conversation containing a tool call was replayed.
+  The provider now hands ruby_llm the parsed arguments (#501).
+- **Structured output through the RubyLLM provider** (`activeagent`). A
+  `json_schema` response_format reached ruby_llm unchanged, but ruby_llm
+  reads `{ name:, schema:, strict: }`, so OpenAI received a schema with a
+  null name and body, and the Anthropic request raised inside ruby_llm
+  before it was sent. The provider now converts it, naming the schema
+  `response` and making it strict unless the format says otherwise, as
+  ruby_llm's own `with_schema` does. A `text` format asks for plain text;
+  `json_object`, which ruby_llm has no mode for, and a `json_schema`
+  without a schema now raise `ArgumentError` instead of sending a request
+  the API rejects (#501).
+
 ## [1.7.0] - 2026-09-24
 
 Releases `activeagent` and `actionagent` 1.7.0 from one tag. A minor release:

data/README.md

@@ -54,6 +54,9 @@ bundle add openai
 # OpenRouter (uses OpenAI-compatible API)
 bundle add openai
 
+# Requesty (uses OpenAI-compatible API)
+bundle add openai
+
 # RubyLLM (unified API for 15+ providers)
 bundle add ruby_llm
 ```

data/lib/active_agent/evals/model_spec.rb

@@ -20,6 +20,10 @@ module ActiveAgent
     class ModelSpec
       DEFAULT_PROVIDERS = %w[openai anthropic ollama openrouter].freeze
 
+      # The dashboard's model pickers resolve names with a JavaScript copy of
+      # these rules and of .parse (actionagent/frontend/utils/modelOptions.mjs).
+      # actionagent/test/fixtures/model_spec_cases.json holds the cases both
+      # are tested against, so a change here needs the same change there.
       DEFAULT_INFERENCE_RULES = [
         [ /\Aclaude/i, "anthropic" ],
         [ /\A(gpt-|o\d|chatgpt|text-embedding)/i, "openai" ],

data/lib/active_agent/evals/runner.rb

@@ -16,10 +16,10 @@ module ActiveAgent
     # faults in `refine_faults` (up to `judge_limit` calls per run).
     #
     #   Runner.new(
-    #     scenarios: suite.scenarios(groups: %w[blame]),
+    #     scenarios: suite.scenarios(groups: %w[history]),
     #     models: ModelSpec.parse_all(%w[gpt-5-mini claude-haiku-4-5], default_provider: "openai"),
     #     criteria: [{ "key" => "response_present", "type" => "response_present" }],
-    #     available_tools: { "find_records" => "Look up records" },
+    #     available_tools: { "lookup_order" => "Find an order by number" },
     #     instructions: agent.instructions,
     #     judge: Judge.new(label: "claude-opus-5") { |instructions:, prompt:| ... },
     #     replay: ->(scenario, spec) { ... },

data/lib/active_agent/evals/scenario.rb

@@ -7,11 +7,11 @@ module ActiveAgent
     # answer is expected to do.
     #
     # @!attribute key
-    #   @return [String] stable within a suite ("blame_3"), so results line up across runs
+    #   @return [String] stable within a suite ("history_3"), so results line up across runs
     # @!attribute group
-    #   @return [String, nil] the group key ("blame")
+    #   @return [String, nil] the group key ("history")
     # @!attribute group_name
-    #   @return [String, nil] the group's display name ("Blame / audit")
+    #   @return [String, nil] the group's display name ("History / audit")
     # @!attribute expected_tools
     #   @return [Array<String>] tool names a passing answer calls (any one of them)
     # @!attribute expected_patterns

data/lib/active_agent/evals/scenario_parser.rb

@@ -9,8 +9,8 @@ module ActiveAgent
     #   - `# Heading` or `**Heading**` lines, which start a group; so does a
     #     short unmarked line ending in a colon (`Find records:`)
     #   - a message in backticks at the start of the line, followed by notes,
-    #     as in an issue's question catalog:
-    #     `` 3. `Show me all providers with no license on file` — 1,060 locally ``
+    #     as in a list copied from an issue:
+    #     `` 3. `Show me all tickets with no assignee` — 12 in the sample data ``
     #   - trailing ` | tools: a, b | contains: x | not_contains: y | key: k`
     #     options on a line
     #   - a JSON array of strings, or of objects with `prompt` (or `message`),
@@ -19,7 +19,7 @@ module ActiveAgent
     #     group names, stable keys, notes and production-only flags
     #
     # Every scenario gets a key unique within the paste, derived from its group
-    # and position ("blame_3"), unless the line names one. The result is an
+    # and position ("history_3"), unless the line names one. The result is an
     # array of string-keyed hashes; `Scenario.from_hash` builds the structs.
     class ScenarioParser
       class ParseError < ArgumentError; end

data/lib/active_agent/evals/suite.rb

@@ -8,17 +8,17 @@ module ActiveAgent
     # suite. That is how an app keeps a shared suite and lets a deployment add or
     # reword questions.
     #
-    #   suite: assistant_dashboard
-    #   description: The V1 question catalog
+    #   suite: support_desk
+    #   description: Questions the support team asks every week
     #   groups:
-    #     - key: find_records
-    #       name: Find record(s)
+    #     - key: open_tickets
+    #       name: Open tickets
     #       scenarios:
-    #         - key: find_records_1
-    #           prompt: Which medical search terms are under client control?
+    #         - key: open_tickets_1
+    #           prompt: Which open tickets mention a refund?
     #           expect:
-    #             tools: [find_records, count_records]
-    #           notes: Uniform locally.
+    #             tools: [find_tickets, count_tickets]
+    #           notes: The sample data has three.
     #           production_only: false
     class Suite
       class NotFound < StandardError; end

data/lib/active_agent/evals.rb

@@ -50,7 +50,7 @@ require_relative "evals/publisher"
 #   report = ActiveAgent::Evals::Runner.new(
 #     scenarios: scenarios,
 #     models: models,
-#     available_tools: { "find_records" => "Look up records by filters" },
+#     available_tools: { "lookup_order" => "Find an order by number" },
 #     replay: ->(scenario, spec) { my_agent.run(scenario.prompt, model: spec.model, provider: spec.provider) }
 #   ).call
 #

data/lib/active_agent/providers/_base_provider.rb

@@ -10,7 +10,8 @@ require_relative "concerns/tool_choice_clearing"
 GEM_LOADERS = {
   anthropic: [ "anthropic", "~> 1.12", "anthropic" ],
   openai:    [ "openai",    "~> 0.34", "openai" ],
-  ruby_llm:  [ "ruby_llm",  ">= 1.0",  "ruby_llm" ]
+  # Keep a tested floor: 1.2 lacks the provider APIs this adapter uses.
+  ruby_llm:  [ "ruby_llm",  [ ">= 1.16", "< 3" ], "ruby_llm" ]
 }
 
 # Requires a provider's gem dependency.
@@ -18,15 +19,23 @@ GEM_LOADERS = {
 # @param type [Symbol] provider type (:anthropic, :openai)
 # @param file_name [String] for error context
 # @return [void]
-# @raise [LoadError] when required gem is not installed
+# @raise [LoadError] when the gem is not installed, or when the loaded
+#   version is outside the supported range
 def require_gem!(type, file_name)
   gem_name, requirement, package_name = GEM_LOADERS.fetch(type)
+  requirements = Array(requirement)
   provider_name = file_name.split("/").last.delete_suffix(".rb").camelize
 
   begin
-    gem(gem_name, requirement)
+    gem(gem_name, *requirements)
     require(package_name)
   rescue LoadError
+    loaded = Gem.loaded_specs[gem_name]
+    if loaded && !Gem::Requirement.new(requirement).satisfied_by?(loaded.version)
+      raise LoadError, "#{provider_name} supports the '#{gem_name}' gem #{requirements.join(', ')}, but #{loaded.version} is loaded. " \
+                       "Add `gem \"#{gem_name}\", #{requirements.map(&:inspect).join(', ')}` to your Gemfile and run `bundle update #{gem_name}`."
+    end
+
     raise LoadError, "The '#{gem_name}' gem is required for #{provider_name}. Please add it to your Gemfile and run `bundle install`."
   end
 end

data/lib/active_agent/providers/ruby_llm/tool_proxy.rb

@@ -27,6 +27,11 @@ module ActiveAgent
           {}
         end
 
+        # RubyLLM 2 names for the same tool interface.
+        alias_method :parameters_schema, :params_schema
+        alias_method :declared_parameters, :parameters
+        alias_method :provider_options, :provider_params
+
         private
 
         def deep_stringify(obj)

data/lib/active_agent/providers/ruby_llm_provider.rb

@@ -2,7 +2,7 @@
 
 require_relative "_base_provider"
 
-require_gem!(:ruby_llm, __FILE__) unless defined?(::RubyLLM)
+require_gem!(:ruby_llm, __FILE__)
 
 require_relative "ruby_llm/_types"
 require_relative "ruby_llm/tool_proxy"
@@ -59,16 +59,22 @@ module ActiveAgent
           tools: tools || {},
           temperature: parameters[:temperature]
         }
-        kwargs[:schema] = parameters[:response_format] if parameters[:response_format]
+        schema = ruby_llm_schema(parameters[:response_format])
+        kwargs[:schema] = schema if schema
 
-        # Pass extra params (max_tokens, etc.) via RubyLLM's params: deep-merge
+        # RubyLLM 2 renamed params: and exposes a provider-neutral token limit.
         max_tokens = parameters[:max_tokens] || options.max_tokens
         if max_tokens
-          kwargs[:params] = { max_tokens: max_tokens }
+          if ruby_llm_v2?
+            kwargs[:max_output_tokens] = max_tokens
+          else
+            kwargs[:params] = { max_tokens: max_tokens }
+          end
         end
 
         if parameters[:stream]
           stream_proc = parameters[:stream]
+          @stream_tool_calls = {}
 
           # For streaming, pass a block that forwards chunks
           @ruby_llm_provider.complete(messages, **kwargs) do |chunk|
@@ -94,7 +100,8 @@ module ActiveAgent
         inputs = input.is_a?(Array) ? input : [ input ]
 
         data = inputs.map.with_index do |text, index|
-          embedding = @ruby_llm_provider.embed(text, model: model_id, dimensions: parameters[:dimensions])
+          embedding_model = ruby_llm_v2? ? @ruby_llm_model : model_id
+          embedding = @ruby_llm_provider.embed(text, model: embedding_model, dimensions: parameters[:dimensions])
 
           {
             object: "embedding",
@@ -137,12 +144,15 @@ module ActiveAgent
         # Handle tool calls in chunk
         if chunk.tool_calls&.any?
           message[:tool_calls] ||= []
-          chunk.tool_calls.each do |_id, tool_call|
-            existing = message[:tool_calls].find { |tc| tc[:id] == tool_call.id }
+          chunk.tool_calls.each do |key, tool_call|
+            # RubyLLM 2 keys OpenAI deltas by index; only the first delta
+            # includes the call ID and name. Keep each index tied to its call.
+            existing = message[:tool_calls].find { |tc| tool_call.id && tc[:id] == tool_call.id }
+            existing ||= @stream_tool_calls[key]
             if existing
               existing[:function][:arguments] += tool_call.arguments.to_s if tool_call.arguments
             else
-              message[:tool_calls] << {
+              existing = {
                 id: tool_call.id,
                 type: "function",
                 function: {
@@ -150,7 +160,9 @@ module ActiveAgent
                   arguments: tool_call.arguments.to_s
                 }
               }
+              message[:tool_calls] << existing
             end
+            @stream_tool_calls[key] = existing
           end
         end
 
@@ -253,6 +265,10 @@ module ActiveAgent
 
       private
 
+      def ruby_llm_v2?
+        Gem.loaded_specs.fetch("ruby_llm").version.segments.first >= 2
+      end
+
       # Resolves and caches the RubyLLM provider for the given model.
       #
       # Reuses the cached provider if the model hasn't changed (e.g., during
@@ -345,12 +361,61 @@ module ActiveAgent
           call = ::RubyLLM::ToolCall.new(
             id: id,
             name: tc.dig(:function, :name) || tc[:name],
-            arguments: tc.dig(:function, :arguments) || tc[:input]&.to_json || "{}"
+            arguments: ruby_llm_tool_arguments(tc.dig(:function, :arguments) || tc[:input])
           )
           hash[id] = call
         end
       end
 
+      # ActiveAgent keeps a tool call's arguments as the JSON string the
+      # model sent; RubyLLM takes them as a Hash and renders them itself
+      # (JSON-encoded for OpenAI, as the input object for Anthropic).
+      #
+      # A string that is not valid JSON, such as arguments a model cut off
+      # mid-way in a stored conversation, is passed through unchanged.
+      #
+      # @param arguments [String, Hash, nil]
+      # @return [Hash, String]
+      def ruby_llm_tool_arguments(arguments)
+        case arguments
+        when Hash
+          arguments.deep_stringify_keys
+        when String
+          arguments.blank? ? {} : JSON.parse(arguments)
+        else
+          {}
+        end
+      rescue JSON::ParserError
+        arguments
+      end
+
+      # Converts ActiveAgent's response_format to the schema RubyLLM's
+      # complete takes, { name:, schema:, strict: }, filling the name and
+      # strict flag in the way RubyLLM::Chat#with_schema does.
+      #
+      # @param response_format [Hash, Symbol, String, nil] ActiveAgent common format
+      # @return [Hash, nil] nil when the response is plain text
+      # @raise [ArgumentError] for json_schema without a schema, and for any
+      #   other type, including json_object, which RubyLLM has no mode for
+      def ruby_llm_schema(response_format)
+        return nil if response_format.nil?
+
+        format = response_format.is_a?(Hash) ? response_format : { type: response_format.to_s }
+
+        case format[:type].to_s
+        when "text"
+          nil
+        when "json_schema"
+          json_schema = format[:json_schema] || {}
+          raise ArgumentError, "RubyLLMProvider needs a schema for a json_schema response_format" unless json_schema[:schema]
+
+          { name: json_schema[:name] || "response", schema: json_schema[:schema], strict: json_schema[:strict] != false }
+        else
+          raise ArgumentError, "RubyLLMProvider supports a json_schema or text response_format, not #{format[:type].inspect}; " \
+                               "ruby_llm has no JSON object mode, so give json_schema a schema instead"
+        end
+      end
+
       # Converts ActiveAgent tool definitions to RubyLLM ToolProxy objects.
       #
       # @param tools [Array<Hash>, nil] ActiveAgent tool definitions
@@ -397,6 +462,8 @@ module ActiveAgent
         # Add stop_reason if available
         if response.respond_to?(:stop_reason) && response.stop_reason
           hash[:stop_reason] = response.stop_reason
+        elsif response.respond_to?(:finish_reason) && response.finish_reason
+          hash[:stop_reason] = { stop: "end_turn", tool_calls: "tool_use" }.fetch(response.finish_reason, response.finish_reason.to_s)
         elsif response.tool_calls&.any?
           hash[:stop_reason] = "tool_use"
         else
@@ -404,7 +471,10 @@ module ActiveAgent
         end
 
         # Add usage info if available
-        if response.respond_to?(:input_tokens) && response.input_tokens
+        if response.respond_to?(:tokens)
+          tokens = response.tokens
+          hash[:usage] = { input_tokens: tokens.input, output_tokens: tokens.output } if tokens.input
+        elsif response.respond_to?(:input_tokens) && response.input_tokens
           hash[:usage] = {
             input_tokens: response.input_tokens,
             output_tokens: response.output_tokens

data/lib/active_agent/version.rb

@@ -1,3 +1,3 @@
 module ActiveAgent
-  VERSION = "1.7.0"
+  VERSION = "1.7.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: activeagent
 version: !ruby/object:Gem::Version
-  version: 1.7.0
+  version: 1.7.2
 platform: ruby
 authors:
 - Justin Bowen
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-24 00:00:00.000000000 Z
+date: 2026-09-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -206,14 +206,20 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '1.16'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '1.16'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3'
 - !ruby/object:Gem::Dependency
   name: capybara
   requirement: !ruby/object:Gem::Requirement