activeagent 1.6.4 → 1.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a0332669c4ffee1ccbea3e1645e126acac6b4e957b980000e5d96baedc908638
-  data.tar.gz: 1d6062b56a6e7fc0e76b965d308df706c3be563292344cdf68b500046e838784
+  metadata.gz: 14e5b0cec7ba170c446d1ed32ad5c9d0705ac8eae4fdc6e4e4a7d17bdae502a2
+  data.tar.gz: 28e12f0ef3727a97fc231e5f7ace15ad4cb020b379fc76c757c2206c89f2b2e6
 SHA512:
-  metadata.gz: c0a940227737d94497c19e8760cf1241ea02f6078987f8cd27f78b0b1a1fb22ac90c8b924dff87e85a4bf8f13c952f6a36eec388dd806f56d230f5b38c9fe59b
-  data.tar.gz: df01fdb757be0bcfa549086dfe0082db0ada892fc552600087d07fe9c1299a4aa4d4c897987cbda897832d11df50f19368261413647f63db4e636a4bb5bb98e5
+  metadata.gz: 9cd766d568ab510b043e4902c1c5fa73ce877a268761168509f2772190ab0f2d6f259fe62e4654aa2a002d20228f876fce82c7c2f5f3e1018cd6cb01935aed27
+  data.tar.gz: 197cab65243e11561c7216f08e9b30369819e3ef73627d0f6e72092a7fc74c16a278638f5044e11f292149a1a39dbe11cab9593a41e7289049a3a90c6232a0a0

data/CHANGELOG.md

@@ -7,6 +7,303 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.7.1] - 2026-09-29
+
+Releases `activeagent` and `actionagent` 1.7.1 from one tag. A patch on 1.7.0:
+the Evaluations page picks the judge and compared models from the provider
+catalogs, `GET /api/provider_models` lists the host's RubyLLM registry, and
+the RubyLLM provider requires ruby_llm 1.x and sends tool calls and structured
+output in the shape ruby_llm reads. No migrations.
+
+### Added
+
+- **Model pickers on the Evaluations page** (`actionagent`). The judge model
+  and the models to compare, on the new-evaluation form and on a scenario
+  suite, are chosen from type-ahead suggestions, and a model the suggestions
+  lack can still be typed. The compare fields hold one removable chip per
+  model and still submit the same comma-separated list.
+  - With scenarios, the suggestions are the catalogs the agent builder uses,
+    for the providers the owner's runs have credentials for. A model whose
+    name alone would run elsewhere is offered with its provider in front, as
+    `ModelSpec.parse` reads it: `ollama/llama3.2`, or
+    `openrouter/anthropic/claude-sonnet-4.5` for OpenRouter's copy of an
+    Anthropic model.
+  - Without scenarios, a compared model selects the generations recorded
+    under its name, usually the provider's dated id, so the suggestions are
+    the names the agent's generations were recorded under, from the new
+    `GET /api/agents/:id/recorded_models`. Adding or clearing the scenarios
+    renames the catalog models already chosen to match.
+  - The judge field suggests the models of the provider the judge runs on
+    and says which provider that is, or that the credentials deciding it
+    could not be read.
+
+  `GET /api/evaluations` reports that provider as `judge_provider` and the
+  providers runs can use as `model_providers`
+  (`AgentExecutionService.available_providers`). Runs and their judge use
+  the evaluated agent's owner's credentials, so both fields describe that
+  owner's when `agent_id` scopes the list, and the signed-in owner's
+  otherwise. `judge_provider` is null when no provider has credentials, and
+  also when reading them raised, which `judge_provider_error: true` marks.
+  `model_providers` leaves out a provider whose credentials cannot be read.
+  Either way the list still loads.
+- **`GET /api/provider_models` lists the host's RubyLLM registry**
+  (`actionagent`). When the host app loads RubyLLM, the chat models its
+  registry lists for the provider that take and return text (RubyLLM's
+  bundled catalog, or the host's own model table) follow the live or curated
+  list, each once, so the builder's preselected default is unchanged. That
+  leaves out the speech, transcription, moderation and completion-only
+  models RubyLLM counts as chat models, which its registry lists with no
+  modalities or as taking audio, and with them the few chat models it lists
+  with no modalities, which can still be typed. A registry that raises is
+  logged and leaves the list as it was.
+
+### Changed
+
+- **The RubyLLM provider requires ruby_llm 1.x** (`activeagent`). ruby_llm
+  2.0 renamed the APIs `RubyLLMProvider` calls, and the open `>= 1.0`
+  requirement let `bundle update` install it. The provider now requires
+  `~> 1.0` until it supports 2.0 (#502). Loading it with an unsupported
+  version names the supported range and the loaded version, instead of
+  asking for a gem that is already in the Gemfile. Pin
+  `gem "ruby_llm", "~> 1.0"` if your bundle resolved 2.0.
+
+### Fixed
+
+- **Tool calls sent back through the RubyLLM provider** (`activeagent`).
+  After a tool ran, the follow-up request repeated the model's tool call
+  with its arguments as a JSON string where ruby_llm expects a Hash: OpenAI
+  received them JSON-encoded twice, and Anthropic received a string for
+  `tool_use.input`, which its API requires to be an object. The same
+  happened when a stored conversation containing a tool call was replayed.
+  The provider now hands ruby_llm the parsed arguments (#501).
+- **Structured output through the RubyLLM provider** (`activeagent`). A
+  `json_schema` response_format reached ruby_llm unchanged, but ruby_llm
+  reads `{ name:, schema:, strict: }`, so OpenAI received a schema with a
+  null name and body, and the Anthropic request raised inside ruby_llm
+  before it was sent. The provider now converts it, naming the schema
+  `response` and making it strict unless the format says otherwise, as
+  ruby_llm's own `with_schema` does. A `text` format asks for plain text;
+  `json_object`, which ruby_llm has no mode for, and a `json_schema`
+  without a schema now raise `ArgumentError` instead of sending a request
+  the API rejects (#501).
+
+## [1.7.0] - 2026-09-24
+
+Releases `activeagent` and `actionagent` 1.7.0 from one tag. A minor release:
+a mounted engine collects the evaluation reports applications publish with
+`ActiveAgent::Evals::Publisher`, whose failures now say what the collector
+refused and whether to retry; observed agents read their own traces in
+evaluation criteria, the Tools and Traces tabs and deploy markers; host apps
+extend the engine's models and controllers through concerns and mirror their
+agent classes into the dashboard; and the Evaluations page is rebuilt around
+runs. Run the install generator after upgrading (see Upgrading below).
+
+Upgrading: the install generator emits two new migrations, both guarded
+column by column: `ensure_agent_release_columns`, which adds the agent release
+columns an install generated fresh on 1.6.2-1.6.4 never got (and those on
+tables with a custom `table_name_prefix`), and `add_evaluation_report_identity`
+for the evaluation report collector. Re-run
+`bin/rails generate action_agent:install --skip` (`--skip` keeps your
+initializer) and `bin/rails db:migrate`. A traces-only install needs neither;
+if you re-run the generator there, pass `--traces_only` again, or it emits the
+whole dashboard schema. Nothing else changes until an application publishes a
+report to the mount.
+
+### Added
+
+- **Host concerns for the engine's models and controllers** (`actionagent`).
+  `ActionAgent.model_concerns` is included into
+  `ActionAgent::ApplicationRecord` as it loads, and so into every engine
+  model; `ActionAgent.controller_concerns` into
+  `ActionAgent::ApplicationController`, ahead of its own callbacks, and so
+  into every dashboard controller. (The ingest endpoint,
+  `Api::TracesController`, inherits `ActionController::API` and keeps its
+  own bearer-token authentication; it is not touched.) Entries are modules
+  or their names, resolved
+  when the class loads. A host that pins the engine's tables to one database
+  connection, or carries its session helpers onto the dashboard's
+  controllers, configures that here instead of reopening the classes from a
+  `to_prepare` block.
+- **The Evaluations page is rebuilt around runs** (`actionagent`). Evaluations
+  are the top level; every run is kept and listed with its movement against
+  the run before it (`+3 passed vs #2`, `partial run`, `#1 failed`), and a
+  sampling evaluation's run opens to a page of its own at
+  `<mount>/evaluations/:id/runs/:run_id` — a scorecard per model cohort, the
+  judge's verdict, the criteria × models matrix and what the run asks to fix.
+  A scenario suite's runs are the same full-width list; a row selects the run
+  the suite's model scorecards, fix items and scenario matrix show.
+- **What a run cost is two figures, not one.** The agent's spend — what the
+  replayed or sampled interactions cost to serve, with a `per_interaction`
+  rate, the operating cost a per-conversation budget is set against — is
+  reported apart from the judge's, the judge model's own calls, which run
+  agent-to-agent and offline. Every judge call is metered under what it was
+  for (`scores["_judge_usage"]`: calls, tokens, estimated cost and how many
+  calls scored, recommended, ruled or authored KPIs), and `EvaluationRun#usage`
+  carries both sides. The page shows them on every run row, on the run, on a
+  page tile and in the footer, so the cost of operating an agent is never
+  inflated by the cost of checking it.
+- A generation-sampling run records `scores["_cohorts"]`: per model, how many
+  generations were sampled, how many cleared every criterion, their latency
+  and tokens, and what those interactions cost to serve.
+- `GET /api/evaluations` carries `run_count` and a `previous_run` summary per
+  evaluation, and every serialized run its `number` in the evaluation's
+  history, oldest first.
+- The dashboard's object lists hold their metric columns in place: a trace,
+  interaction or evaluation run with nothing in a column prints a dash there
+  rather than sliding its neighbours over (`MetaStrip`).
+- `ActiveAgent::Base.rendered_instructions` renders an agent's instructions
+  outside a generation, for a dashboard mirroring the class and for tests
+  asserting what a model is told. Both otherwise reached a private renderer
+  through `send`.
+- `ActionAgent::AgentSync` mirrors host agent classes into dashboard `Agent`
+  records, setting the `agent_class_name` that `AgentRelease` already expects a
+  host to have written. The code owns what an agent is (name, description,
+  instructions, tools — rewritten every sync); the operator owns how it runs
+  (provider, model, status — set on create and preserved), so a model chosen in
+  the dashboard survives the next deploy.
+- `ActionAgent.run_host_agent_classes` (default `false`) runs an agent that
+  mirrors a host class as that class, rather than as one rebuilt from the
+  record's `tools` and `instructions` columns. Dashboard-authored agents, which
+  name no class, keep using the dynamic runtime either way; a class name that no
+  longer resolves falls back to it rather than failing the run.
+- MCP servers take `allowed_tools` and `require_approval` in the common
+  format. OpenAI's Responses API receives both as given; Anthropic receives
+  `allowed_tools` as an `mcp_toolset` entry in `tools` (every other tool of
+  the server disabled), beside any tools the request already declares
+  (#328, by @dark-panda).
+- **A mounted engine collects published evaluation reports** (`actionagent`).
+  An application that runs its agents itself and evaluates them in-process
+  publishes the finished report with `ActiveAgent::Evals::Publisher`; until
+  now a self-hosted install had nowhere to receive it. On a full install (not
+  one generated with `--traces_only`, which answers 501),
+  `POST <mount>/api/evaluation_reports` takes the version-1 envelope and
+  returns the receipt the publisher checks, and
+  `ActionAgent::EvaluationReportImport` stores it as the engine's own rows:
+  the observed agent for the report's `source` and `agent_name`, an evaluation
+  named for its suite and scope (`orders (eu, support)`), its scenarios, and a
+  complete run with a result per scenario and model. The Evaluations page
+  shows it the way it shows a run the dashboard executed, with the summary
+  recomputed from the stored results. The endpoint authenticates exactly as
+  trace ingest does (`ingest_api_key`, or the tenant's key in multi-tenant
+  mode), takes only `application/json`, and places a report's agent wherever
+  `trace_owner_resolver` puts that tenant's traced agents. A `run_id` is
+  stored once per tenant, or once per install, and compared exactly: 201 for a
+  new report, 200 for an identical retry, 409 for different content. Invalid
+  reports, and an evaluation name the report does not own, are 422; a cap an
+  operator has to lift (observed agents per owner, 100 evaluations per agent,
+  2,000 scenarios per evaluation) is 403; a new report over the new
+  `:evaluation_report` quota kind or past 30 new reports a minute from a key
+  is 429. An identical retry is never refused by the quota or the rate limit.
+  Bodies over 2 MiB are 413, and Rails never parses the body into params, so
+  nothing past the limit is read. `usage_recorder` is told
+  `:evaluation_report` for each stored report. Evaluation runs gain
+  `external_tenant`, `external_run_id` and `external_report_digest`, unique on
+  the first two (binary on MySQL); see Upgrading above.
+  `docs/evals/publication.md` documents the endpoint.
+
+### Changed
+
+- The engine's judge blocks take `ActiveAgent::Evals::Judge`'s `kind:`, so a
+  scenario run's score, recommendation and verdict calls are metered apart.
+- `ActionAgent::TelemetryTrace` inherits `ActionAgent::ApplicationRecord`
+  like every other engine model (`actionagent`), so it carries the model
+  concerns above, `AdapterAware` and the ownership API (`owner_association`,
+  `for_owner`) from the same place. Its table name is unchanged.
+- `Api::TracesController`'s bearer authentication and its 429 quota body
+  live in `ActionAgent::Api::IngestAuthentication` (`actionagent`), which the
+  evaluation report collector shares. A host subclass that overrides
+  `authenticate_api_key!` is unaffected. The tenant's
+  `increment_telemetry_usage!` is still called for each trace ingest request,
+  and not for a report post.
+- `add_agent_releases` reads `ActionAgent.table_name_prefix` for the tables
+  it alters (`actionagent`), so a newly generated copy works on an install
+  with a custom prefix. The trace table keeps its fixed name.
+- A collector's rejection of `ActiveAgent::Evals::Publisher` says what it
+  refused and whether to retry. The message carries the `error` string of a
+  JSON object response body beside the HTTP status — control characters and
+  runs of whitespace collapsed to one space, the API key replaced with
+  `[FILTERED]`, cut to 200 characters; nothing else from the body — and what
+  to do next: never retry the report under the same `run_id` on a 409,
+  publish a smaller selection on a 413, correct the report on a 422, retry
+  later on a 408, 429 or 5xx, and resolve the cause first on anything else,
+  such as a 401 or 403. `Publisher::Error` carries `status`, `detail` and
+  `retryable?`.
+
+### Deprecated
+
+- Assigning `ActionAgent.base_controller_class`, which has never been
+  consumed, warns through `ActionAgent.deprecator` and points at
+  `controller_concerns`. The accessor is removed in 2.0.
+
+### Fixed
+
+- Telemetry criteria (`trace_error_rate`, `trace_latency`) score an observed
+  agent from its own traces (`actionagent`). They selected traces by
+  `Agent#telemetry_agent_class`, which appends `Agent` to a class name
+  lacking it, so an agent observed from an application reporting `SupportBot`
+  found no traces and scored nothing, and observed agents of one class ending
+  in `Agent` read each other's actions. `Agent#telemetry_traces` selects the
+  traces `AgentRegistrar` attributed to the agent, plus unattributed ones with
+  its service, class and action. Deleting an observed agent leaves its traces
+  unattributed, so the agent registered again for them still reads them. The
+  agent's Traces tab, its Tools tab usage
+  columns and the Interactions list filtered to it use the same selection. The
+  Traces tab asks for it with `GET /api/traces?agent_id=`, which answers 404
+  for an agent the caller cannot see; `agent=` still filters by class. On the
+  Metrics page filtered to a class, an observed agent's deploy markers now
+  show under the class its traces report (`Agent#reported_agent_class`).
+  Authored and mirrored agents read the traces they did before.
+- `Agent.prompt(...).generate_later` and `Agent.embed(...).embed_later` run
+  their job instead of raising `ArgumentError: unknown keywords` in the
+  worker (#346).
+- The agent builder and editor can reach every model a provider serves: the
+  OpenRouter catalog is no longer cut to its first 100 ids, and the model
+  field is a type-ahead over the catalog that also takes an unlisted id
+  (`actionagent`, #427).
+- A rejected Create Agent shows its validation errors on the builder — a
+  summary and a message under each field — instead of leaving the form
+  silently in place. `POST`/`PATCH /api/agents` 422s carry `field_errors`
+  beside `errors` (`actionagent`, #426).
+- An engine agent is refused a provider whose client gem the host has not
+  installed (`openai` for OpenAI, Ollama and OpenRouter; `anthropic` for
+  Anthropic) when the provider is chosen, with a validation error naming
+  the gem, instead of failing on its first run (`actionagent`, #416).
+- A fresh `action_agent:install` creates the agent release columns with the
+  dashboard tables (`actionagent`): `release_digest` on agents,
+  `release_digest` and `revision` on agent versions, and `agent_version_id`
+  on agent runs and evaluation runs. The generator emits `add_agent_releases`
+  before the create-table migration, so on a fresh install it found none of
+  those tables and added nothing, and creating an agent run or an evaluation
+run raised `NoMethodError` on `agent_version_id`. An install generated
+that way on 1.6.2-1.6.4 gets the columns from the new
+`ensure_agent_release_columns` migration (see Upgrading above).
+- Every failure of `ActiveAgent::Evals::Publisher` to deliver a report
+  raises `Publisher::Error`. A malformed response (`Net::HTTPBadResponse`,
+  `Net::HTTPHeaderSyntaxError`, or a `Zlib::Error` from corrupt compression)
+  escaped as its own class and is now a retryable `Publisher::Error`. A
+  report that cannot be encoded as JSON (invalid UTF-8, `NaN`, nesting too
+  deep) is now a non-retryable one raised before anything is sent: it escaped
+  as `JSON::GeneratorError`, or was blamed on the collector as invalid JSON.
+  Only a network failure keeps its underlying error as `cause`, so a response
+  body or report content never reaches a log through the exception chain.
+  Invalid arguments raise `ArgumentError`, now also for a `report` that does
+  not convert to a hash (a string raised `NoMethodError` and `nil` published
+  an empty report) and a `nil` timeout (`TypeError`).
+- The publisher strips whitespace around its API key, so the key it sends is
+  the one it filters from a collector's explanation, and refuses a key with
+  characters other than visible ASCII.
+
+### Security
+
+- The dashboard's JSON API verifies the CSRF token (`actionagent`, #461). It
+  authenticates with the host's session cookie but had opted out of forgery
+  protection. The dashboard now sends the page's token with every mutating
+  request from one fetch shim; the MCP facade and trace ingest, which
+  authenticate by bearer token, stay exempt. A rejected request answers
+  `422` with `code: "invalid_csrf_token"`. Hosts that re-enabled protection
+  themselves (`ActionAgent::Api::BaseController.protect_from_forgery`) can
+  drop that line.
+
 ## [1.6.4] - 2026-09-22
 
 Releases `activeagent` and `actionagent` 1.6.4 from one tag. A patch on 1.6.3

data/README.md

@@ -54,6 +54,9 @@ bundle add openai
 # OpenRouter (uses OpenAI-compatible API)
 bundle add openai
 
+# Requesty (uses OpenAI-compatible API)
+bundle add openai
+
 # RubyLLM (unified API for 15+ providers)
 bundle add ruby_llm
 ```

data/lib/active_agent/concerns/view.rb

@@ -12,6 +12,29 @@ module ActiveAgent
       include ActionView::Layouts
     end
 
+    class_methods do
+      # The agent's rendered instructions, outside a generation.
+      #
+      # Two surfaces need the text an agent would run on without running it: a
+      # dashboard that mirrors the class (ActionAgent::AgentSync) and a test
+      # asserting what the model is told. Both otherwise reach a private
+      # renderer through `send`, which couples them to internals that can move
+      # without notice.
+      #
+      #   TicketAgent.rendered_instructions  # => "You are the Ticket agent..."
+      #
+      #   TicketAgent.rendered_instructions(topic: "tickets")
+      #
+      # @param template [String] template name, default "instructions"
+      # @param assigns [Hash] instance variables the template reads
+      # @return [String, nil] nil when the agent has no such template
+      def rendered_instructions(template: "instructions", **assigns)
+        agent = new
+        assigns.each { |name, value| agent.instance_variable_set(:"@#{name}", value) }
+        agent.send(:view_render_template, template)
+      end
+    end
+
     # Builds template lookup paths supporting both flat and nested directory structures.
     #
     # Templates are searched in priority order:

data/lib/active_agent/evals/model_spec.rb

@@ -20,6 +20,10 @@ module ActiveAgent
     class ModelSpec
       DEFAULT_PROVIDERS = %w[openai anthropic ollama openrouter].freeze
 
+      # The dashboard's model pickers resolve names with a JavaScript copy of
+      # these rules and of .parse (actionagent/frontend/utils/modelOptions.mjs).
+      # actionagent/test/fixtures/model_spec_cases.json holds the cases both
+      # are tested against, so a change here needs the same change there.
       DEFAULT_INFERENCE_RULES = [
         [ /\Aclaude/i, "anthropic" ],
         [ /\A(gpt-|o\d|chatgpt|text-embedding)/i, "openai" ],

data/lib/active_agent/evals/publisher.rb

@@ -4,6 +4,7 @@ require "json"
 require "net/http"
 require "openssl"
 require "uri"
+require "zlib"
 
 module ActiveAgent
   module Evals
@@ -11,11 +12,51 @@ module ActiveAgent
     # retain run_id when retrying: compatible collectors treat that identity as
     # immutable within the authenticated account. Delivery is blocking and does
     # not follow redirects with the account's bearer credential.
+    #
+    # Every failure to deliver raises Error. Invalid arguments raise
+    # ArgumentError before anything is sent.
     class Publisher
       DEFAULT_ENDPOINT = "https://api.activeagents.ai/v1/evaluations"
       MAX_BYTES = 2 * 1024 * 1024
-      class Error < StandardError; end
+      DETAIL_LIMIT = 200
 
+      # Raised for every failed delivery. Only a network failure keeps the
+      # underlying error as its +cause+, so neither the response nor the
+      # report reaches a log through the exception chain.
+      #
+      # @!attribute [r] status
+      #   @return [Integer, nil] the collector's HTTP status for a rejection, nil otherwise
+      # @!attribute [r] detail
+      #   @return [String, nil] the collector's sanitized explanation of a rejection, if it gave one
+      class Error < StandardError
+        attr_reader :status, :detail
+
+        def initialize(message = nil, status: nil, detail: nil, retryable: false)
+          super(message)
+          @status = status
+          @detail = detail
+          @retryable = retryable
+        end
+
+        # Returns true when delivering the same report under the same run_id
+        # again may succeed.
+        def retryable?
+          @retryable
+        end
+      end
+
+      # Whether each rejection status is retryable, and what the caller should
+      # do about it. Other statuses fall back to the rules in +rejection+.
+      REJECTIONS = {
+        409 => [ false, "the collector already holds a different report under this run_id; never retry this report with the same run_id" ],
+        413 => [ false, "the report exceeds the collector's size limit; publish a smaller selection" ],
+        422 => [ false, "correct what the collector refused before retrying" ],
+        429 => [ true, "the account is over its quota or rate limit; retain the report and run_id and retry later" ]
+      }.freeze
+
+      # The key is sent as a bearer token and filtered from the collector's
+      # explanation, so it is limited to visible ASCII: the sanitizer in
+      # +collector_detail+ never alters it, and an echo of it always matches.
       def initialize(api_key:, endpoint: DEFAULT_ENDPOINT, timeout: 10, open_timeout: 10)
         @uri = URI.parse(endpoint.to_s)
         unless @uri.is_a?(URI::HTTP) && @uri.host && !@uri.userinfo && !@uri.query && !@uri.fragment
@@ -24,12 +65,16 @@ module ActiveAgent
         unless @uri.scheme == "https" || %w[localhost 127.0.0.1 ::1].include?(@uri.hostname)
           raise ArgumentError, "Evaluation endpoint requires HTTPS except on loopback hosts"
         end
-        raise ArgumentError, "Evaluation API key is required" if api_key.to_s.strip.empty?
 
-        @api_key = api_key.to_s
-        @timeout = Float(timeout)
-        @open_timeout = Float(open_timeout)
-        unless [ @timeout, @open_timeout ].all? { |value| value.finite? && value.positive? }
+        @api_key = api_key.to_s.strip
+        raise ArgumentError, "Evaluation API key is required" if @api_key.empty?
+        unless @api_key.match?(/\A[\x21-\x7E]+\z/)
+          raise ArgumentError, "Evaluation API key must contain only visible ASCII characters"
+        end
+
+        @timeout = Float(timeout, exception: false)
+        @open_timeout = Float(open_timeout, exception: false)
+        unless [ @timeout, @open_timeout ].all? { |value| value&.finite? && value.positive? }
           raise ArgumentError, "Evaluation delivery timeouts must be positive and finite"
         end
       rescue URI::InvalidURIError
@@ -43,7 +88,7 @@ module ActiveAgent
         identities.each do |key, value|
           raise ArgumentError, "#{key} must be a nonempty string" unless value.is_a?(String) && !value.strip.empty?
         end
-        body = JSON.generate(identities.merge("version" => 1, "report" => report.to_h))
+        body = encode(identities.merge("version" => 1, "report" => report_hash(report)))
         raise Error, "Evaluation report exceeds the 2 MiB delivery limit; publish a smaller selection" if body.bytesize > MAX_BYTES
 
         http = Net::HTTP.new(@uri.hostname, @uri.port)
@@ -57,19 +102,70 @@ module ActiveAgent
         request["Accept"] = "application/json"
         request.body = body
         response = http.request(request)
-        unless %w[200 201].include?(response.code)
-          raise Error, "Evaluation delivery rejected (HTTP #{response.code}); retain the report and run_id for retry"
-        end
+        raise rejection(response) unless %w[200 201].include?(response.code)
 
-        receipt = JSON.parse(response.body)
+        receipt = JSON.parse(response.body.to_s)
         unless receipt.is_a?(Hash) && receipt["run_id"] == run_id && receipt["status"] == "complete" && receipt["id"] && receipt["evaluation_id"]
-          raise Error, "Evaluation collector returned an invalid completion receipt; retain the report and run_id for retry"
+          raise Error.new("Evaluation collector returned an invalid completion receipt; retain the report and run_id for retry", retryable: true)
         end
         receipt
       rescue JSON::ParserError
-        raise Error, "Evaluation collector returned invalid JSON; retain the report and run_id for retry"
+        # The parser's message quotes the body.
+        raise Error.new("Evaluation collector returned invalid JSON; retain the report and run_id for retry", retryable: true), cause: nil
+      rescue Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Zlib::Error => e
+        # These messages can quote the response's status line, headers or body.
+        raise Error.new("Evaluation collector returned a malformed response (#{e.class}); retain the report and run_id for retry", retryable: true), cause: nil
       rescue IOError, SocketError, SystemCallError, Timeout::Error, OpenSSL::SSL::SSLError => e
-        raise Error, "Evaluation delivery failed (#{e.class}); retain the report and run_id for retry"
+        raise Error.new("Evaluation delivery failed (#{e.class}); retain the report and run_id for retry", retryable: true)
+      end
+
+      private
+
+      def report_hash(report)
+        hash = report.to_h if report.respond_to?(:to_h) && !report.nil? && !report.is_a?(Array)
+        raise ArgumentError, "report must be a Report or its saved JSON hash" unless hash.is_a?(Hash)
+
+        hash
+      end
+
+      # Returns the envelope as JSON. A report holding invalid UTF-8, NaN,
+      # Infinity or nesting over JSON's depth limit raises a non-retryable
+      # Error, without the generator's message, which can quote the report.
+      def encode(envelope)
+        JSON.generate(envelope)
+      rescue JSON::JSONError, EncodingError => e
+        raise Error.new("Evaluation report cannot be encoded as JSON (#{e.class}); correct the report before publishing"), cause: nil
+      end
+
+      def rejection(response)
+        status = response.code.to_i
+        retryable, guidance = REJECTIONS.fetch(status) do
+          if status == 408 || (500..599).cover?(status)
+            [ true, "retain the report and run_id for retry" ]
+          else
+            [ false, "retain the report and run_id, and resolve the rejection before retrying" ]
+          end
+        end
+        detail = collector_detail(response.body)
+        reason = detail ? "HTTP #{status}: #{detail}" : "HTTP #{status}"
+        Error.new("Evaluation delivery rejected (#{reason}); #{guidance}", status: status, detail: detail, retryable: retryable)
+      end
+
+      # Returns the +error+ string of a JSON object body, or nil for any other
+      # body. Control characters and runs of whitespace become one space, the
+      # API key becomes [FILTERED], and the result is cut to DETAIL_LIMIT
+      # characters: `"answer\n\tis required"` → `"answer is required"`.
+      def collector_detail(body)
+        parsed = JSON.parse(body.to_s)
+        message = parsed["error"] if parsed.is_a?(Hash)
+        return unless message.is_a?(String)
+
+        detail = message.scrub.gsub(/[[:space:]\p{C}]+/, " ").strip.gsub(@api_key, "[FILTERED]")
+        return if detail.empty?
+
+        detail.length > DETAIL_LIMIT ? "#{detail[0, DETAIL_LIMIT - 1].rstrip}…" : detail
+      rescue JSON::ParserError, EncodingError
+        nil
       end
     end
   end

data/lib/active_agent/evals/runner.rb

@@ -16,10 +16,10 @@ module ActiveAgent
     # faults in `refine_faults` (up to `judge_limit` calls per run).
     #
     #   Runner.new(
-    #     scenarios: suite.scenarios(groups: %w[blame]),
+    #     scenarios: suite.scenarios(groups: %w[history]),
     #     models: ModelSpec.parse_all(%w[gpt-5-mini claude-haiku-4-5], default_provider: "openai"),
     #     criteria: [{ "key" => "response_present", "type" => "response_present" }],
-    #     available_tools: { "find_records" => "Look up records" },
+    #     available_tools: { "lookup_order" => "Find an order by number" },
     #     instructions: agent.instructions,
     #     judge: Judge.new(label: "claude-opus-5") { |instructions:, prompt:| ... },
     #     replay: ->(scenario, spec) { ... },

data/lib/active_agent/evals/scenario.rb

@@ -7,11 +7,11 @@ module ActiveAgent
     # answer is expected to do.
     #
     # @!attribute key
-    #   @return [String] stable within a suite ("blame_3"), so results line up across runs
+    #   @return [String] stable within a suite ("history_3"), so results line up across runs
     # @!attribute group
-    #   @return [String, nil] the group key ("blame")
+    #   @return [String, nil] the group key ("history")
     # @!attribute group_name
-    #   @return [String, nil] the group's display name ("Blame / audit")
+    #   @return [String, nil] the group's display name ("History / audit")
     # @!attribute expected_tools
     #   @return [Array<String>] tool names a passing answer calls (any one of them)
     # @!attribute expected_patterns

data/lib/active_agent/evals/scenario_parser.rb

@@ -9,8 +9,8 @@ module ActiveAgent
     #   - `# Heading` or `**Heading**` lines, which start a group; so does a
     #     short unmarked line ending in a colon (`Find records:`)
     #   - a message in backticks at the start of the line, followed by notes,
-    #     as in an issue's question catalog:
-    #     `` 3. `Show me all providers with no license on file` — 1,060 locally ``
+    #     as in a list copied from an issue:
+    #     `` 3. `Show me all tickets with no assignee` — 12 in the sample data ``
     #   - trailing ` | tools: a, b | contains: x | not_contains: y | key: k`
     #     options on a line
     #   - a JSON array of strings, or of objects with `prompt` (or `message`),
@@ -19,7 +19,7 @@ module ActiveAgent
     #     group names, stable keys, notes and production-only flags
     #
     # Every scenario gets a key unique within the paste, derived from its group
-    # and position ("blame_3"), unless the line names one. The result is an
+    # and position ("history_3"), unless the line names one. The result is an
     # array of string-keyed hashes; `Scenario.from_hash` builds the structs.
     class ScenarioParser
       class ParseError < ArgumentError; end

data/lib/active_agent/evals/suite.rb

@@ -8,17 +8,17 @@ module ActiveAgent
     # suite. That is how an app keeps a shared suite and lets a deployment add or
     # reword questions.
     #
-    #   suite: assistant_dashboard
-    #   description: The V1 question catalog
+    #   suite: support_desk
+    #   description: Questions the support team asks every week
     #   groups:
-    #     - key: find_records
-    #       name: Find record(s)
+    #     - key: open_tickets
+    #       name: Open tickets
     #       scenarios:
-    #         - key: find_records_1
-    #           prompt: Which medical search terms are under client control?
+    #         - key: open_tickets_1
+    #           prompt: Which open tickets mention a refund?
     #           expect:
-    #             tools: [find_records, count_records]
-    #           notes: Uniform locally.
+    #             tools: [find_tickets, count_tickets]
+    #           notes: The sample data has three.
     #           production_only: false
     class Suite
       class NotFound < StandardError; end

data/lib/active_agent/evals.rb

@@ -50,7 +50,7 @@ require_relative "evals/publisher"
 #   report = ActiveAgent::Evals::Runner.new(
 #     scenarios: scenarios,
 #     models: models,
-#     available_tools: { "find_records" => "Look up records by filters" },
+#     available_tools: { "lookup_order" => "Find an order by number" },
 #     replay: ->(scenario, spec) { my_agent.run(scenario.prompt, model: spec.model, provider: spec.provider) }
 #   ).call
 #

data/lib/active_agent/generation_job.rb

@@ -23,7 +23,21 @@ module ActiveAgent
     # argument, so a record arrives as the same record the caller passed and
     # the agent's authorization callbacks decide against a real user rather
     # than against nil.
-    def perform(agent, agent_method, generation_method, args:, kwargs: nil, params: nil, actor: nil)
+    #
+    # +direct_generation_type+ marks a job enqueued by Agent.prompt(...) or
+    # Agent.embed(...), which have no action to call: +agent_method+ is then
+    # the synthetic +__direct_*__+ name, so the generation is rebuilt from
+    # +direct_args+ and +direct_options+ instead.
+    def perform(agent, agent_method, generation_method, args:, kwargs: nil, params: nil, actor: nil,
+                direct_generation_type: nil, direct_args: nil, direct_options: nil)
+      if direct_generation_type
+        generation = ActiveAgent::Parameterized::DirectGeneration.new(
+          agent.constantize, direct_generation_type.to_sym, params || {},
+          *Array(direct_args), **(direct_options || {}).symbolize_keys
+        )
+        return generation.public_send(generation_method)
+      end
+
       agent_class = params ? agent.constantize.with(params) : agent.constantize
       agent_class = agent_class.as(actor) if actor
       prompt = if kwargs

data/lib/active_agent/providers/_base_provider.rb

@@ -10,7 +10,8 @@ require_relative "concerns/tool_choice_clearing"
 GEM_LOADERS = {
   anthropic: [ "anthropic", "~> 1.12", "anthropic" ],
   openai:    [ "openai",    "~> 0.34", "openai" ],
-  ruby_llm:  [ "ruby_llm",  ">= 1.0",  "ruby_llm" ]
+  # ruby_llm 2.0 renamed the APIs RubyLLMProvider calls.
+  ruby_llm:  [ "ruby_llm",  "~> 1.0",  "ruby_llm" ]
 }
 
 # Requires a provider's gem dependency.
@@ -18,7 +19,8 @@ GEM_LOADERS = {
 # @param type [Symbol] provider type (:anthropic, :openai)
 # @param file_name [String] for error context
 # @return [void]
-# @raise [LoadError] when required gem is not installed
+# @raise [LoadError] when the gem is not installed, or when the loaded
+#   version is outside the supported range
 def require_gem!(type, file_name)
   gem_name, requirement, package_name = GEM_LOADERS.fetch(type)
   provider_name = file_name.split("/").last.delete_suffix(".rb").camelize
@@ -27,6 +29,12 @@ def require_gem!(type, file_name)
     gem(gem_name, requirement)
     require(package_name)
   rescue LoadError
+    loaded = Gem.loaded_specs[gem_name]
+    if loaded && !Gem::Requirement.new(requirement).satisfied_by?(loaded.version)
+      raise LoadError, "#{provider_name} supports the '#{gem_name}' gem #{requirement}, but #{loaded.version} is loaded. " \
+                       "Add `gem \"#{gem_name}\", \"#{requirement}\"` to your Gemfile and run `bundle update #{gem_name}`."
+    end
+
     raise LoadError, "The '#{gem_name}' gem is required for #{provider_name}. Please add it to your Gemfile and run `bundle install`."
   end
 end

data/lib/active_agent/providers/anthropic/transforms.rb

@@ -38,10 +38,20 @@ module ActiveAgent
             end
 
             # Handle mcps parameter (common format) -> transforms to mcp_servers (provider format)
-            if params[:mcps]
-              params[:mcp_servers] = normalize_mcp_servers(params.delete(:mcps))
-            elsif params[:mcp_servers]
-              params[:mcp_servers] = normalize_mcp_servers(params[:mcp_servers])
+            if params[:mcps] || params[:mcp_servers]
+              mcps = if params[:mcps]
+                params.delete(:mcps)
+              else
+                params[:mcp_servers]
+              end
+
+              params[:mcp_servers] = normalize_mcp_servers(mcps)
+
+              # A server's allowed_tools become an mcp_toolset entry beside the
+              # request's own tools. Added only when there is one: an empty
+              # list would send `"tools": null` on every MCP request.
+              mcp_tools = normalize_mcp_tools(mcps)
+              params[:tools] = Array(params[:tools]) + mcp_tools if mcp_tools.present?
             end
 
             params
@@ -111,6 +121,49 @@ module ActiveAgent
             end
           end
 
+          # Builds Anthropic mcp_toolset tool entries from MCP server allowed_tools.
+          #
+          # Accepts allowed_tools entries as tool-name strings/symbols or hashes
+          # with a :name key; entries in any other format are ignored.
+          #
+          # @param mcp_servers [Array<Hash>]
+          # @return [Array<Hash>, nil] toolset entries, or nil when none were extracted
+          def normalize_mcp_tools(mcp_servers)
+            return nil unless mcp_servers.is_a?(Array)
+
+            result = mcp_servers.filter_map do |server|
+              next unless server.is_a?(Hash)
+
+              server_hash = server.deep_symbolize_keys
+              allowed_tools = server_hash[:allowed_tools]
+              next unless allowed_tools.is_a?(Array)
+
+              configs = allowed_tools.filter_map { |tool|
+                name = case tool
+                when String, Symbol
+                  tool.to_s
+                when Hash
+                  (tool[:name] || tool["name"]).to_s
+                end
+
+                [ name, { enabled: true } ] if name.present?
+              }.to_h
+
+              next if configs.empty?
+
+              {
+                type: "mcp_toolset",
+                mcp_server_name: server_hash[:name],
+                default_config: {
+                  enabled: false
+                },
+                configs: configs
+              }
+            end
+
+            result.presence
+          end
+
           # Normalizes tool_choice from common format to Anthropic gem model objects.
           #
           # The Anthropic gem expects tool_choice to be a model object (ToolChoiceAuto,

data/lib/active_agent/providers/open_ai/responses/transforms.rb

@@ -96,6 +96,14 @@ module ActiveAgent
                   server_url: server_hash[:url] || server_hash[:server_url]
                 }
 
+                if server_hash[:require_approval]
+                  result[:require_approval] = server_hash[:require_approval]
+                end
+
+                if server_hash[:allowed_tools]
+                  result[:allowed_tools] = server_hash[:allowed_tools]
+                end
+
                 # Keep authorization field (OpenAI uses 'authorization', not 'authorization_token')
                 if server_hash[:authorization]
                   result[:authorization] = server_hash[:authorization]

data/lib/active_agent/providers/ruby_llm_provider.rb

@@ -2,7 +2,7 @@
 
 require_relative "_base_provider"
 
-require_gem!(:ruby_llm, __FILE__) unless defined?(::RubyLLM)
+require_gem!(:ruby_llm, __FILE__)
 
 require_relative "ruby_llm/_types"
 require_relative "ruby_llm/tool_proxy"
@@ -59,7 +59,8 @@ module ActiveAgent
           tools: tools || {},
           temperature: parameters[:temperature]
         }
-        kwargs[:schema] = parameters[:response_format] if parameters[:response_format]
+        schema = ruby_llm_schema(parameters[:response_format])
+        kwargs[:schema] = schema if schema
 
         # Pass extra params (max_tokens, etc.) via RubyLLM's params: deep-merge
         max_tokens = parameters[:max_tokens] || options.max_tokens
@@ -345,12 +346,61 @@ module ActiveAgent
           call = ::RubyLLM::ToolCall.new(
             id: id,
             name: tc.dig(:function, :name) || tc[:name],
-            arguments: tc.dig(:function, :arguments) || tc[:input]&.to_json || "{}"
+            arguments: ruby_llm_tool_arguments(tc.dig(:function, :arguments) || tc[:input])
           )
           hash[id] = call
         end
       end
 
+      # ActiveAgent keeps a tool call's arguments as the JSON string the
+      # model sent; RubyLLM takes them as a Hash and renders them itself
+      # (JSON-encoded for OpenAI, as the input object for Anthropic).
+      #
+      # A string that is not valid JSON, such as arguments a model cut off
+      # mid-way in a stored conversation, is passed through unchanged.
+      #
+      # @param arguments [String, Hash, nil]
+      # @return [Hash, String]
+      def ruby_llm_tool_arguments(arguments)
+        case arguments
+        when Hash
+          arguments.deep_stringify_keys
+        when String
+          arguments.blank? ? {} : JSON.parse(arguments)
+        else
+          {}
+        end
+      rescue JSON::ParserError
+        arguments
+      end
+
+      # Converts ActiveAgent's response_format to the schema RubyLLM's
+      # complete takes, { name:, schema:, strict: }, filling the name and
+      # strict flag in the way RubyLLM::Chat#with_schema does.
+      #
+      # @param response_format [Hash, Symbol, String, nil] ActiveAgent common format
+      # @return [Hash, nil] nil when the response is plain text
+      # @raise [ArgumentError] for json_schema without a schema, and for any
+      #   other type, including json_object, which RubyLLM has no mode for
+      def ruby_llm_schema(response_format)
+        return nil if response_format.nil?
+
+        format = response_format.is_a?(Hash) ? response_format : { type: response_format.to_s }
+
+        case format[:type].to_s
+        when "text"
+          nil
+        when "json_schema"
+          json_schema = format[:json_schema] || {}
+          raise ArgumentError, "RubyLLMProvider needs a schema for a json_schema response_format" unless json_schema[:schema]
+
+          { name: json_schema[:name] || "response", schema: json_schema[:schema], strict: json_schema[:strict] != false }
+        else
+          raise ArgumentError, "RubyLLMProvider supports a json_schema or text response_format, not #{format[:type].inspect}; " \
+                               "ruby_llm has no JSON object mode, so give json_schema a schema instead"
+        end
+      end
+
       # Converts ActiveAgent tool definitions to RubyLLM ToolProxy objects.
       #
       # @param tools [Array<Hash>, nil] ActiveAgent tool definitions

data/lib/active_agent/version.rb

@@ -1,3 +1,3 @@
 module ActiveAgent
-  VERSION = "1.6.4"
+  VERSION = "1.7.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: activeagent
 version: !ruby/object:Gem::Version
-  version: 1.6.4
+  version: 1.7.1
 platform: ruby
 authors:
 - Justin Bowen
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-23 00:00:00.000000000 Z
+date: 2026-09-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -204,14 +204,14 @@ dependencies:
   name: ruby_llm
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.0'
 - !ruby/object:Gem::Dependency