activeagent 1.7.0 → 1.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4aee77fd4f67a50824b7153b66f31aa5f19293bad4397e032ec0a9e674a759d0
-  data.tar.gz: e261e8ca9d221edbc849791ed28c67bc491ccc31e1e18f5c4cbf7a9a96a806c5
+  metadata.gz: 14e5b0cec7ba170c446d1ed32ad5c9d0705ac8eae4fdc6e4e4a7d17bdae502a2
+  data.tar.gz: 28e12f0ef3727a97fc231e5f7ace15ad4cb020b379fc76c757c2206c89f2b2e6
 SHA512:
-  metadata.gz: f5779f6207e07b1aec0ef14f72a5b010075b953c9025a12f6231ba164522c8b785cd9ca1bb297361abedc42aa535c468ef642978ec9fb439c79c9c0fe71ebd79
-  data.tar.gz: 1d2720a47e5e8c2a6c17a74cff893f1da2e89dd430b295bdcda4b9bdfb966afa8e75fbf95c0f063f5bca0bc8406d782e08b99fa6bed99b20230fb793ac3e518b
+  metadata.gz: 9cd766d568ab510b043e4902c1c5fa73ce877a268761168509f2772190ab0f2d6f259fe62e4654aa2a002d20228f876fce82c7c2f5f3e1018cd6cb01935aed27
+  data.tar.gz: 197cab65243e11561c7216f08e9b30369819e3ef73627d0f6e72092a7fc74c16a278638f5044e11f292149a1a39dbe11cab9593a41e7289049a3a90c6232a0a0

data/CHANGELOG.md

@@ -7,6 +7,86 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.7.1] - 2026-09-29
+
+Releases `activeagent` and `actionagent` 1.7.1 from one tag. A patch on 1.7.0:
+the Evaluations page picks the judge and compared models from the provider
+catalogs, `GET /api/provider_models` lists the host's RubyLLM registry, and
+the RubyLLM provider requires ruby_llm 1.x and sends tool calls and structured
+output in the shape ruby_llm reads. No migrations.
+
+### Added
+
+- **Model pickers on the Evaluations page** (`actionagent`). The judge model
+  and the models to compare, on the new-evaluation form and on a scenario
+  suite, are chosen from type-ahead suggestions, and a model the suggestions
+  lack can still be typed. The compare fields hold one removable chip per
+  model and still submit the same comma-separated list.
+  - With scenarios, the suggestions are the catalogs the agent builder uses,
+    for the providers the owner's runs have credentials for. A model whose
+    name alone would run elsewhere is offered with its provider in front, as
+    `ModelSpec.parse` reads it: `ollama/llama3.2`, or
+    `openrouter/anthropic/claude-sonnet-4.5` for OpenRouter's copy of an
+    Anthropic model.
+  - Without scenarios, a compared model selects the generations recorded
+    under its name, usually the provider's dated id, so the suggestions are
+    the names the agent's generations were recorded under, from the new
+    `GET /api/agents/:id/recorded_models`. Adding or clearing the scenarios
+    renames the catalog models already chosen to match.
+  - The judge field suggests the models of the provider the judge runs on
+    and says which provider that is, or that the credentials deciding it
+    could not be read.
+
+  `GET /api/evaluations` reports that provider as `judge_provider` and the
+  providers runs can use as `model_providers`
+  (`AgentExecutionService.available_providers`). Runs and their judge use
+  the evaluated agent's owner's credentials, so both fields describe that
+  owner's when `agent_id` scopes the list, and the signed-in owner's
+  otherwise. `judge_provider` is null when no provider has credentials, and
+  also when reading them raised, which `judge_provider_error: true` marks.
+  `model_providers` leaves out a provider whose credentials cannot be read.
+  Either way the list still loads.
+- **`GET /api/provider_models` lists the host's RubyLLM registry**
+  (`actionagent`). When the host app loads RubyLLM, the chat models its
+  registry lists for the provider that take and return text (RubyLLM's
+  bundled catalog, or the host's own model table) follow the live or curated
+  list, each once, so the builder's preselected default is unchanged. That
+  leaves out the speech, transcription, moderation and completion-only
+  models RubyLLM counts as chat models, which its registry lists with no
+  modalities or as taking audio, and with them the few chat models it lists
+  with no modalities, which can still be typed. A registry that raises is
+  logged and leaves the list as it was.
+
+### Changed
+
+- **The RubyLLM provider requires ruby_llm 1.x** (`activeagent`). ruby_llm
+  2.0 renamed the APIs `RubyLLMProvider` calls, and the open `>= 1.0`
+  requirement let `bundle update` install it. The provider now requires
+  `~> 1.0` until it supports 2.0 (#502). Loading it with an unsupported
+  version names the supported range and the loaded version, instead of
+  asking for a gem that is already in the Gemfile. Pin
+  `gem "ruby_llm", "~> 1.0"` if your bundle resolved 2.0.
+
+### Fixed
+
+- **Tool calls sent back through the RubyLLM provider** (`activeagent`).
+  After a tool ran, the follow-up request repeated the model's tool call
+  with its arguments as a JSON string where ruby_llm expects a Hash: OpenAI
+  received them JSON-encoded twice, and Anthropic received a string for
+  `tool_use.input`, which its API requires to be an object. The same
+  happened when a stored conversation containing a tool call was replayed.
+  The provider now hands ruby_llm the parsed arguments (#501).
+- **Structured output through the RubyLLM provider** (`activeagent`). A
+  `json_schema` response_format reached ruby_llm unchanged, but ruby_llm
+  reads `{ name:, schema:, strict: }`, so OpenAI received a schema with a
+  null name and body, and the Anthropic request raised inside ruby_llm
+  before it was sent. The provider now converts it, naming the schema
+  `response` and making it strict unless the format says otherwise, as
+  ruby_llm's own `with_schema` does. A `text` format asks for plain text;
+  `json_object`, which ruby_llm has no mode for, and a `json_schema`
+  without a schema now raise `ArgumentError` instead of sending a request
+  the API rejects (#501).
+
 ## [1.7.0] - 2026-09-24
 
 Releases `activeagent` and `actionagent` 1.7.0 from one tag. A minor release:

data/README.md

@@ -54,6 +54,9 @@ bundle add openai
 # OpenRouter (uses OpenAI-compatible API)
 bundle add openai
 
+# Requesty (uses OpenAI-compatible API)
+bundle add openai
+
 # RubyLLM (unified API for 15+ providers)
 bundle add ruby_llm
 ```

data/lib/active_agent/evals/model_spec.rb

@@ -20,6 +20,10 @@ module ActiveAgent
     class ModelSpec
       DEFAULT_PROVIDERS = %w[openai anthropic ollama openrouter].freeze
 
+      # The dashboard's model pickers resolve names with a JavaScript copy of
+      # these rules and of .parse (actionagent/frontend/utils/modelOptions.mjs).
+      # actionagent/test/fixtures/model_spec_cases.json holds the cases both
+      # are tested against, so a change here needs the same change there.
       DEFAULT_INFERENCE_RULES = [
         [ /\Aclaude/i, "anthropic" ],
         [ /\A(gpt-|o\d|chatgpt|text-embedding)/i, "openai" ],

data/lib/active_agent/evals/runner.rb

@@ -16,10 +16,10 @@ module ActiveAgent
     # faults in `refine_faults` (up to `judge_limit` calls per run).
     #
     #   Runner.new(
-    #     scenarios: suite.scenarios(groups: %w[blame]),
+    #     scenarios: suite.scenarios(groups: %w[history]),
     #     models: ModelSpec.parse_all(%w[gpt-5-mini claude-haiku-4-5], default_provider: "openai"),
     #     criteria: [{ "key" => "response_present", "type" => "response_present" }],
-    #     available_tools: { "find_records" => "Look up records" },
+    #     available_tools: { "lookup_order" => "Find an order by number" },
     #     instructions: agent.instructions,
     #     judge: Judge.new(label: "claude-opus-5") { |instructions:, prompt:| ... },
     #     replay: ->(scenario, spec) { ... },

data/lib/active_agent/evals/scenario.rb

@@ -7,11 +7,11 @@ module ActiveAgent
     # answer is expected to do.
     #
     # @!attribute key
-    #   @return [String] stable within a suite ("blame_3"), so results line up across runs
+    #   @return [String] stable within a suite ("history_3"), so results line up across runs
     # @!attribute group
-    #   @return [String, nil] the group key ("blame")
+    #   @return [String, nil] the group key ("history")
     # @!attribute group_name
-    #   @return [String, nil] the group's display name ("Blame / audit")
+    #   @return [String, nil] the group's display name ("History / audit")
     # @!attribute expected_tools
     #   @return [Array<String>] tool names a passing answer calls (any one of them)
     # @!attribute expected_patterns

data/lib/active_agent/evals/scenario_parser.rb

@@ -9,8 +9,8 @@ module ActiveAgent
     #   - `# Heading` or `**Heading**` lines, which start a group; so does a
     #     short unmarked line ending in a colon (`Find records:`)
     #   - a message in backticks at the start of the line, followed by notes,
-    #     as in an issue's question catalog:
-    #     `` 3. `Show me all providers with no license on file` — 1,060 locally ``
+    #     as in a list copied from an issue:
+    #     `` 3. `Show me all tickets with no assignee` — 12 in the sample data ``
     #   - trailing ` | tools: a, b | contains: x | not_contains: y | key: k`
     #     options on a line
     #   - a JSON array of strings, or of objects with `prompt` (or `message`),
@@ -19,7 +19,7 @@ module ActiveAgent
     #     group names, stable keys, notes and production-only flags
     #
     # Every scenario gets a key unique within the paste, derived from its group
-    # and position ("blame_3"), unless the line names one. The result is an
+    # and position ("history_3"), unless the line names one. The result is an
     # array of string-keyed hashes; `Scenario.from_hash` builds the structs.
     class ScenarioParser
       class ParseError < ArgumentError; end

data/lib/active_agent/evals/suite.rb

@@ -8,17 +8,17 @@ module ActiveAgent
     # suite. That is how an app keeps a shared suite and lets a deployment add or
     # reword questions.
     #
-    #   suite: assistant_dashboard
-    #   description: The V1 question catalog
+    #   suite: support_desk
+    #   description: Questions the support team asks every week
     #   groups:
-    #     - key: find_records
-    #       name: Find record(s)
+    #     - key: open_tickets
+    #       name: Open tickets
     #       scenarios:
-    #         - key: find_records_1
-    #           prompt: Which medical search terms are under client control?
+    #         - key: open_tickets_1
+    #           prompt: Which open tickets mention a refund?
     #           expect:
-    #             tools: [find_records, count_records]
-    #           notes: Uniform locally.
+    #             tools: [find_tickets, count_tickets]
+    #           notes: The sample data has three.
     #           production_only: false
     class Suite
       class NotFound < StandardError; end

data/lib/active_agent/evals.rb

@@ -50,7 +50,7 @@ require_relative "evals/publisher"
 #   report = ActiveAgent::Evals::Runner.new(
 #     scenarios: scenarios,
 #     models: models,
-#     available_tools: { "find_records" => "Look up records by filters" },
+#     available_tools: { "lookup_order" => "Find an order by number" },
 #     replay: ->(scenario, spec) { my_agent.run(scenario.prompt, model: spec.model, provider: spec.provider) }
 #   ).call
 #

data/lib/active_agent/providers/_base_provider.rb

@@ -10,7 +10,8 @@ require_relative "concerns/tool_choice_clearing"
 GEM_LOADERS = {
   anthropic: [ "anthropic", "~> 1.12", "anthropic" ],
   openai:    [ "openai",    "~> 0.34", "openai" ],
-  ruby_llm:  [ "ruby_llm",  ">= 1.0",  "ruby_llm" ]
+  # ruby_llm 2.0 renamed the APIs RubyLLMProvider calls.
+  ruby_llm:  [ "ruby_llm",  "~> 1.0",  "ruby_llm" ]
 }
 
 # Requires a provider's gem dependency.
@@ -18,7 +19,8 @@ GEM_LOADERS = {
 # @param type [Symbol] provider type (:anthropic, :openai)
 # @param file_name [String] for error context
 # @return [void]
-# @raise [LoadError] when required gem is not installed
+# @raise [LoadError] when the gem is not installed, or when the loaded
+#   version is outside the supported range
 def require_gem!(type, file_name)
   gem_name, requirement, package_name = GEM_LOADERS.fetch(type)
   provider_name = file_name.split("/").last.delete_suffix(".rb").camelize
@@ -27,6 +29,12 @@ def require_gem!(type, file_name)
     gem(gem_name, requirement)
     require(package_name)
   rescue LoadError
+    loaded = Gem.loaded_specs[gem_name]
+    if loaded && !Gem::Requirement.new(requirement).satisfied_by?(loaded.version)
+      raise LoadError, "#{provider_name} supports the '#{gem_name}' gem #{requirement}, but #{loaded.version} is loaded. " \
+                       "Add `gem \"#{gem_name}\", \"#{requirement}\"` to your Gemfile and run `bundle update #{gem_name}`."
+    end
+
     raise LoadError, "The '#{gem_name}' gem is required for #{provider_name}. Please add it to your Gemfile and run `bundle install`."
   end
 end

data/lib/active_agent/providers/ruby_llm_provider.rb

@@ -2,7 +2,7 @@
 
 require_relative "_base_provider"
 
-require_gem!(:ruby_llm, __FILE__) unless defined?(::RubyLLM)
+require_gem!(:ruby_llm, __FILE__)
 
 require_relative "ruby_llm/_types"
 require_relative "ruby_llm/tool_proxy"
@@ -59,7 +59,8 @@ module ActiveAgent
           tools: tools || {},
           temperature: parameters[:temperature]
         }
-        kwargs[:schema] = parameters[:response_format] if parameters[:response_format]
+        schema = ruby_llm_schema(parameters[:response_format])
+        kwargs[:schema] = schema if schema
 
         # Pass extra params (max_tokens, etc.) via RubyLLM's params: deep-merge
         max_tokens = parameters[:max_tokens] || options.max_tokens
@@ -345,12 +346,61 @@ module ActiveAgent
           call = ::RubyLLM::ToolCall.new(
             id: id,
             name: tc.dig(:function, :name) || tc[:name],
-            arguments: tc.dig(:function, :arguments) || tc[:input]&.to_json || "{}"
+            arguments: ruby_llm_tool_arguments(tc.dig(:function, :arguments) || tc[:input])
           )
           hash[id] = call
         end
       end
 
+      # ActiveAgent keeps a tool call's arguments as the JSON string the
+      # model sent; RubyLLM takes them as a Hash and renders them itself
+      # (JSON-encoded for OpenAI, as the input object for Anthropic).
+      #
+      # A string that is not valid JSON, such as arguments a model cut off
+      # mid-way in a stored conversation, is passed through unchanged.
+      #
+      # @param arguments [String, Hash, nil]
+      # @return [Hash, String]
+      def ruby_llm_tool_arguments(arguments)
+        case arguments
+        when Hash
+          arguments.deep_stringify_keys
+        when String
+          arguments.blank? ? {} : JSON.parse(arguments)
+        else
+          {}
+        end
+      rescue JSON::ParserError
+        arguments
+      end
+
+      # Converts ActiveAgent's response_format to the schema RubyLLM's
+      # complete takes, { name:, schema:, strict: }, filling the name and
+      # strict flag in the way RubyLLM::Chat#with_schema does.
+      #
+      # @param response_format [Hash, Symbol, String, nil] ActiveAgent common format
+      # @return [Hash, nil] nil when the response is plain text
+      # @raise [ArgumentError] for json_schema without a schema, and for any
+      #   other type, including json_object, which RubyLLM has no mode for
+      def ruby_llm_schema(response_format)
+        return nil if response_format.nil?
+
+        format = response_format.is_a?(Hash) ? response_format : { type: response_format.to_s }
+
+        case format[:type].to_s
+        when "text"
+          nil
+        when "json_schema"
+          json_schema = format[:json_schema] || {}
+          raise ArgumentError, "RubyLLMProvider needs a schema for a json_schema response_format" unless json_schema[:schema]
+
+          { name: json_schema[:name] || "response", schema: json_schema[:schema], strict: json_schema[:strict] != false }
+        else
+          raise ArgumentError, "RubyLLMProvider supports a json_schema or text response_format, not #{format[:type].inspect}; " \
+                               "ruby_llm has no JSON object mode, so give json_schema a schema instead"
+        end
+      end
+
       # Converts ActiveAgent tool definitions to RubyLLM ToolProxy objects.
       #
       # @param tools [Array<Hash>, nil] ActiveAgent tool definitions

data/lib/active_agent/version.rb

@@ -1,3 +1,3 @@
 module ActiveAgent
-  VERSION = "1.7.0"
+  VERSION = "1.7.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: activeagent
 version: !ruby/object:Gem::Version
-  version: 1.7.0
+  version: 1.7.1
 platform: ruby
 authors:
 - Justin Bowen
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-24 00:00:00.000000000 Z
+date: 2026-09-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -204,14 +204,14 @@ dependencies:
   name: ruby_llm
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.0'
 - !ruby/object:Gem::Dependency