activeagent 1.5.2 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 73e63c87b700b6dec4e3410ae0547ff1c02714fbe79b24175447fb5638a9fe3b
-  data.tar.gz: ef361dc3bb7dcab252c0fadf1fb526e689b97b21386a2d50b6ba6eb96a84100a
+  metadata.gz: 1838cd05b12c4bc56330e8c4b4ec98295e885990bc67b600fefaa401f2543593
+  data.tar.gz: dcd3590de72aef1a8059a6fb03d9303672e73260a59c33e9244ccf326d48b7b6
 SHA512:
-  metadata.gz: 7ed21c4a284e17d0014ef83b09fc56741b96cbb83f219f38dc3e8ad87f511f9ed5b3d902e1b1e9c7e4db11c83bce07ed0effbb2f44d85c35d0da4738598744b3
-  data.tar.gz: c3634cfbe8e532889e2ef82b7080b30f0233a0faf1e81b5c37995949a8c7a6d5e6ebf9e3800e39457c54de3803d22b3545d4606d448867a777915cabf1a39b15
+  metadata.gz: 1534a469b0eeec15765b71e7b43e8362660d9fc7610531c757acb882e130a95cc938a2033b2d0982233558123a0b162d6efb9c51da3e7038d0f0f1d6fcfe8d9e
+  data.tar.gz: eb849f183afd357f381836947891c8029d5ca4772f2e4bf1fda3fedb30ebb7f71db1021f9f94a8ba62337a99493b54b0124f9f0e28d8342fde4182d8ebf02411

data/CHANGELOG.md

@@ -7,6 +7,142 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.6.0] - 2026-09-14
+
+Releases `activeagent` and `actionagent` 1.6.0 from one tag.
+
+A minor, not a patch. The cycle that began after 1.5.2 gives an agent a
+caller — `current_user`, carried from whatever authenticated the call into
+every `before_action`, every tool, every delegated sub-agent, every run over
+MCP and every evaluation replay — so an authorization gem has something to
+decide against. Around it: schema tools defined at runtime rather than only
+in a file, a generator that writes the first one, those tools served
+directly over MCP, and an evaluation that calls a fabricated answer a fault
+instead of grading it as an honest gap. That is new public surface in both
+gems, which is a minor under semver even though 1.5.2 shipped a feature as a
+patch.
+
+Two notes for upgrades. `tools_succeeded` is now awarded only for a tool the
+scenario expected, so a suite that was quietly scoring wrong-tool runs as
+partial successes will report lower — read the first run as a corrected
+baseline. And `actor:` is now stripped from tool arguments and from
+`params[params][actor]`: the caller is a property of the run, set once by
+whatever authenticated it, and can no longer be named by the model or by a
+client.
+
+The engine's floor on the framework (`activeagent >= 1.4`) is unchanged and
+still correct: 1.6.0 satisfies it.
+
+### Added
+
+- **An evaluation replay runs as the evaluation's owner.** The scenario runner
+  handed `Agent#test_execute` no caller, so every tool a replay called ran
+  unattributed and a host scope answered empty — the suite graded an agent
+  that never saw a row. When agents are owned per user, a replay now runs as
+  the user who owns the evaluation, as a run over MCP runs as the key's
+  owner. A multi-tenant install still replays unattributed (an account is who
+  is billed, not who is allowed) unless a host adapter runs the suite itself.
+- **The MCP facade serves the host's schema tools directly.** `tools/list`
+  at `POST <mount>/mcp` now offers every tool the dashboard's discovered
+  `ActiveAgent::SchemaTools` classes generate — `find_<records>`,
+  `count_<records>`, `get_<record>` — beside the `run_<slug>` agents, each
+  with its own parameter schema, and `tools/call` runs one as the key's
+  caller through the host's own scope, exactly as it would inside an agent
+  run. A client that only needs the rows no longer has to ask an agent for
+  them. A boundary violation is a tool result with `isError`, a refusal from
+  the host's scope is a JSON-RPC `-32003`, and neither the execution switch
+  nor the execution quota applies, because nothing generates. Set
+  `ActionAgent.mcp_schema_tools = false` to keep the tools reachable only
+  through agents. Closes #439.
+- **A delegated run inherits its parent's caller.** `delegate_to` hands the
+  sub-agent the parent's `current_user` before its action runs, so its own
+  `before_action` callbacks and any scope its tools read through decide
+  against the same person. A parent authorized as one user no longer hands
+  its specialists an unattributed run — which a correctly written host scope
+  reads as "no access", a wrong answer wearing a right one's clothes. A
+  parent with no caller still delegates an unattributed run, never someone
+  else's.
+- **`rails generate active_agent:schema_tools Reservation`** writes a starter
+  `ActiveAgent::SchemaTools` class under `app/agent_tools`. It exposes nothing
+  beyond `id` until a column is moved into `filterable` or `returns`; every
+  column the model has is listed, commented out, with its type, so the
+  allowlist is a review step rather than a blank page, and columns that look
+  like secrets are left off the list. Reads are scoped through
+  `<Model>Policy::Scope` when it exists (`--policy` / `--no-policy` decide
+  explicitly). This is #440's second option: the roster is still declared,
+  once, but the declaration is no longer written from scratch. (#440)
+- **An agent knows who it is running for, so an authorization gem has
+  something to decide against.** `ActiveAgent::Base#current_user` carries the
+  caller, assigned by whatever authenticated the call
+  (`MyAgent.as(current_user).ask(...)`) and readable from every
+  `before_action`, so Pundit, CanCanCan or Action Policy authorize an agent
+  the way they authorize a controller. A refusal inside a tool call is
+  returned to the model as `{ error: ... }` so it can say it is not allowed;
+  a refusal anywhere else is raised to the caller. `denies_with` names a
+  gem's own error as a refusal.
+- **The dashboard fills that seam.** A run records the caller that started it
+  (a Global ID, so a worker on another machine authorizes as the same
+  person), passes it to every tool as `actor:`, and runs the agent through
+  `as(actor)` — which is what a `SchemaTools` `scope` block has been waiting
+  for since 1.5.0. `ActionAgent.agent_actor_resolver` overrides who that is.
+- **Agents reached over MCP run as the key's caller** rather than
+  unattributed, and an agent that refuses answers as a JSON-RPC error
+  (`-32003`) instead of as an empty result.
+- **Schema tools can be defined at runtime.** `ActiveAgent::SchemaTools.define(Reservation,
+  filterable:, returns:, scope: | policy:)` builds the same bounded roster a
+  file under `app/agent_tools` would — same allowlists, same `call`, named
+  `ReservationTools` for logs — from a declaration held anywhere: a table, a
+  dashboard form, a test. The class is registered under its model and a
+  redefinition replaces the previous one, so a registry rebuilt on every
+  change holds one class per model; `undefine` drops it. The dashboard
+  discovers registry entries beside the files and lets a runtime definition
+  supersede a file for the same model. What is persisted, and where, stays the
+  host's decision; this is the seam a persisted declaration builds on. (#441)
+- **A fabricated answer is now a fault.** `Diagnosis` raises `ungrounded_answer`
+  when an agent that had tools called none, did not say it could not answer,
+  and still stated specifics — a count, a record id, a date — that no tool
+  supplied. Where the scenario names an expected tool, `expected_tool_not_called`
+  says the same thing in its summary and carries `ungrounded: true`, so an
+  invented answer no longer reads like an honest gap. Both reach the judge,
+  which is what turns them into a suggested tool. An agent with no tools at all
+  is not flagged: it answers from its instructions by design, and whether that
+  is acceptable is the judge's grade, not a mechanical one. (#433)
+
+### Changed
+
+- **A wrong tool no longer outscores no tool.** `tools_succeeded` is awarded
+  only for a tool the scenario expected (or any tool when it expects none):
+  a tool that ran without erroring was evidence of the task only by accident,
+  and a scenario that called the wrong tool scored higher than one that called
+  nothing. (#433)
+- **The judge reads more of a scenario's notes** — 1,500 characters rather
+  than 300 — because a suite's notes are often its rubric and the "must not"
+  clause tends to come last. (#433)
+
+### Fixed
+
+- **The caller can no longer be named by the model, or by the client.**
+  `actor:` reached `AgentToolbox.call` in the same keyword namespace as the
+  arguments a provider parsed out of a model's tool call, and
+  `params[params][actor]` in an execute request would have won over the
+  controller's own. Both are now stripped: the caller is a property of the
+  run, set once by whatever authenticated it. Scoped `SchemaTools` reads are
+  also excluded from the tool-result cache, so one caller's rows are never
+  replayed for the next.
+- **A superseded runtime tool class is no longer offered twice.** Discovery
+  read every `SchemaTools` subclass out of `descendants`, where a class built
+  at runtime stays until it is collected, so rebuilding a model's tools
+  accumulated stale duplicates. Runtime-built classes are now read from the
+  registry only. (#441)
+- **A persisted model selection re-runs under the provider it ran under.**
+  `Evals::ModelSpec.parse_all` re-parsed a round-tripped spec from its label,
+  so `anthropic/claude-sonnet-4.5` run through OpenRouter came back as
+  Anthropic's own `claude-sonnet-4.5` as soon as that provider's gem was
+  installed — and the re-run failed for want of an Anthropic credential. A
+  hash naming both `provider` and `model` is now rebuilt as it was; a bare
+  label is still parsed. The dashboard's "re-run" of a saved selection is
+  the path this fixes.
+
 ## [1.5.2] - 2026-09-11
 
 Releases `activeagent` and `actionagent` 1.5.2 from one tag.

data/lib/active_agent/base.rb

@@ -4,6 +4,7 @@ require "active_support/core_ext/hash/except"
 require "active_support/core_ext/module/anonymous"
 require "active_support/core_ext/string/inflections"
 
+require "active_agent/concerns/authorization"
 require "active_agent/concerns/callbacks"
 require "active_agent/concerns/delegation"
 require "active_agent/concerns/observers"
@@ -43,6 +44,7 @@ module ActiveAgent
     include AbstractController::Caching
 
     include Callbacks
+    # After Rescue: the refusal handler is registered with rescue_from.
     include Delegation
     include Parameterized
     include Provider
@@ -52,6 +54,11 @@ module ActiveAgent
     include Tooling
     include View
 
+    # Last of the behaviour concerns: its rescue_from must sit on top of the
+    # handlers an agent registers for its own authorization gem, so a host
+    # mapping Pundit::NotAuthorizedError keeps deciding what a refusal means.
+    include Authorization
+
     include Observers
     include Previews
 

data/lib/active_agent/concerns/authorization.rb

@@ -0,0 +1,172 @@
+# frozen_string_literal: true
+
+module ActiveAgent
+  # Raised when an agent refuses a call on the current caller's behalf.
+  #
+  # Hosts usually raise their authorization gem's own error instead
+  # (+Pundit::NotAuthorizedError+, +CanCan::AccessDenied+) and name it with
+  # {Authorization::ClassMethods#denies_with}; this exists so an agent that
+  # has no gem still has something to raise, and so the framework has one
+  # class to describe a refusal with.
+  class NotAuthorized < StandardError
+    # @return [Symbol, String, nil] the action or tool that was refused
+    attr_reader :action
+    # @return [Object, nil] the caller the refusal was decided against
+    attr_reader :actor
+
+    def initialize(message = nil, action: nil, actor: nil)
+      @action = action
+      @actor = actor
+      super(message || default_message)
+    end
+
+    private
+
+    def default_message
+      who = actor.nil? ? "an unauthenticated caller" : "the current caller"
+      what = action ? "`#{action}`" : "this agent"
+      "#{who} is not allowed to call #{what}"
+    end
+  end
+
+  # Carries the caller an agent runs on behalf of, so an agent's callbacks can
+  # authorize with whatever the host app already uses.
+  #
+  # An agent reached over MCP, from a dashboard run, or from a controller is
+  # acting *for someone*. Without that someone, an authorization gem has
+  # nothing to decide against: a Pundit scope handed +nil+ correctly resolves
+  # to the empty set, so a perfectly wired agent answers "there are no
+  # tickets" instead of refusing — a wrong answer that reads like a true one.
+  #
+  # == The seam
+  #
+  # {#current_user} is assigned by whatever authenticated the call and is
+  # readable from every callback and action:
+  #
+  #   class TicketAgent < ApplicationAgent
+  #     before_action :authorize_tickets!
+  #
+  #     def find_tickets(**filters)
+  #       TicketTools.call("find_tickets", actor: current_user, **filters)
+  #     end
+  #
+  #     private
+  #
+  #     def authorize_tickets!
+  #       raise ActiveAgent::NotAuthorized.new(action: action_name, actor: current_user) unless
+  #         TicketPolicy.new(current_user, Ticket).index?
+  #     end
+  #   end
+  #
+  #   TicketAgent.as(current_user).find_tickets.generate_now
+  #
+  # Any gem works, because the framework never interprets the actor — Pundit's
+  # +authorize+/+policy_scope+, CanCanCan's +can?+, Action Policy's
+  # +authorize!+, or a plain predicate. +before_action+ is the same
+  # +AbstractController+ chain a controller uses, including +only:+/+except:+,
+  # so a roster can be authorized tool by tool.
+  #
+  # == It is never model input
+  #
+  # The actor is an attribute of the run, set out of band by the caller. It is
+  # deliberately not part of +params+ and not a tool argument: everything a
+  # model emits is attacker-reachable through the documents it reads, and an
+  # actor a model can name is not an authorization boundary. Assigning it is
+  # the caller's job, once, before the generation starts.
+  #
+  # == What a refusal does
+  #
+  # A refusal inside a *tool call* is returned to the model as an error result
+  # ({ error: ... }), so it can tell the user it is not allowed to look rather
+  # than dying mid-run or, worse, reporting an empty result set as fact. A
+  # refusal anywhere else — the action the caller asked for — is raised, so
+  # the MCP client or controller that asked gets an error instead of an
+  # answer that silently covers less ground than it appears to.
+  #
+  # {ClassMethods#denies_with} is how a gem's own error joins that rule:
+  #
+  #   class ApplicationAgent < ActiveAgent::Base
+  #     denies_with Pundit::NotAuthorizedError
+  #   end
+  module Authorization
+    extend ActiveSupport::Concern
+
+    included do
+      # The caller this generation runs on behalf of, or nil when it runs
+      # unattributed. Whatever the host uses as an actor: a User, an API
+      # key's owner, a service account.
+      attr_accessor :current_user
+
+      # Exception classes that mean "the caller may not do this". Declared
+      # rather than guessed: only the host knows which of its errors are a
+      # refusal and which are a bug.
+      class_attribute :authorization_errors, instance_writer: false,
+        default: [ ActiveAgent::NotAuthorized ].freeze
+    end
+
+    class_methods do
+      # Treats +classes+ as refusals, so raising one inside a tool call
+      # reports to the model instead of ending the run.
+      #
+      # @param classes [Array<Class>] exception classes from an authorization gem
+      # @return [Array<Class>] every class now treated as a refusal
+      #
+      # @example
+      #   denies_with Pundit::NotAuthorizedError, CanCan::AccessDenied
+      def denies_with(*classes)
+        self.authorization_errors = (authorization_errors | classes.flatten).freeze
+      end
+
+      # Runs the agent on behalf of +actor+.
+      #
+      # @param actor [Object, nil] the caller, or nil to run unattributed
+      # @return [ActiveAgent::Parameterized::Agent] a proxy carrying the actor
+      #
+      # @example
+      #   SupportAgent.as(current_user).answer(question).generate_now
+      #
+      # @example With parameters
+      #   SupportAgent.as(current_user).with(locale: :en).answer(question)
+      def as(actor)
+        ActiveAgent::Parameterized::Agent.new(self, {}, actor: actor)
+      end
+    end
+
+    # Whether a refusal right now would be reported to the model rather than
+    # raised to the caller.
+    # @return [Boolean]
+    def tool_call?
+      @_active_agent_tool_call ||= false
+    end
+
+    # Marks the block as a tool call, so a refusal inside it becomes a result
+    # the model can read. Nested calls keep the outer marking.
+    # @api private
+    def with_tool_call
+      previous = @_active_agent_tool_call
+      @_active_agent_tool_call = true
+
+      result = yield
+      # A host's own rescue_from handler runs inside `process`, and
+      # ActiveSupport::Rescuable answers with the exception itself — so a
+      # refusal arrives either raised or returned, and both mean the same
+      # thing here.
+      refusal?(result) ? refused(result) : result
+    rescue *authorization_errors => exception
+      refused(exception)
+    ensure
+      @_active_agent_tool_call = previous
+    end
+
+    private
+
+    def refusal?(value)
+      value.is_a?(Exception) && authorization_errors.any? { |klass| value.is_a?(klass) }
+    end
+
+    def refused(exception)
+      logger&.info("[#{self.class.name}] refused #{action_name}: #{exception.message}")
+      { error: exception.message }
+    end
+  end
+end

data/lib/active_agent/concerns/parameterized.rb

@@ -110,9 +110,28 @@ module ActiveAgent
     class Agent
       # @param agent [Class] the agent class to proxy
       # @param params [Hash] the parameters to pass to agent instances
-      def initialize(agent, params)
+      # @param actor [Object, nil] the caller the generation runs on behalf
+      #   of (see ActiveAgent::Authorization). Kept beside params rather than
+      #   in them: params reach templates and are part of what a generation
+      #   is *about*, while the actor is who it is *for*, and only the caller
+      #   may say.
+      def initialize(agent, params, actor: nil)
         @agent = agent
         @params = params
+        @actor = actor
+      end
+
+      # Adds parameters, keeping the actor. Lets +as+ and +with+ chain in
+      # either order.
+      # @return [ActiveAgent::Parameterized::Agent]
+      def with(params = {})
+        self.class.new(@agent, @params.merge(params), actor: @actor)
+      end
+
+      # Runs on behalf of +actor+, keeping any parameters.
+      # @return [ActiveAgent::Parameterized::Agent]
+      def as(actor)
+        self.class.new(@agent, @params, actor: actor)
       end
 
       # Intercepts calls to agent action methods and creates parameterized generations.
@@ -124,7 +143,9 @@ module ActiveAgent
       # @raise [NoMethodError] if the method doesn't exist on the agent class
       def method_missing(method_name, ...)
         if @agent.public_instance_methods.include?(method_name)
-          ActiveAgent::Parameterized::Generation.new(@agent, method_name, @params, ...)
+          ActiveAgent::Parameterized::Generation.new(@agent, method_name, @params, ...).tap do |generation|
+            generation.actor = @actor
+          end
         else
           super
         end
@@ -146,6 +167,17 @@ module ActiveAgent
     #
     # @api private
     class Generation < ActiveAgent::Generation
+      # @param agent_class [Class] the agent class
+      # @param action [Symbol, String] the action method name
+      # @param params [Hash] the parameters to set on the agent instance
+      # @param args [Array] additional arguments for the action method
+      # The caller the generation runs on behalf of. Assigned after
+      # construction rather than taken as a keyword, because the action's own
+      # arguments are forwarded here and an actor keyword would collide with
+      # one of the same name.
+      # @return [Object, nil]
+      attr_accessor :actor
+
       # @param agent_class [Class] the agent class
       # @param action [Symbol, String] the action method name
       # @param params [Hash] the parameters to set on the agent instance
@@ -163,6 +195,7 @@ module ActiveAgent
       def agent
         @agent ||= agent_class.new.tap do |agent|
           agent.params = @params
+          agent.current_user = @actor
           agent.process(action_name, *args, **kwargs)
         end
       end
@@ -180,8 +213,12 @@ module ActiveAgent
         if processed?
           super
         else
+          # The actor rides as an ordinary job argument, which ActiveJob
+          # serializes through GlobalID like any other record, so a worker on
+          # another machine authorizes as the same caller.
           agent_class.generation_job.set(job_options).perform_later(
-            agent_class.name, action_name.to_s, generation_method.to_s, params: @params, args: args, kwargs: kwargs
+            agent_class.name, action_name.to_s, generation_method.to_s,
+            params: @params, args: args, kwargs: kwargs, actor: @actor
           )
         end
       end

data/lib/active_agent/concerns/tooling.rb

@@ -16,7 +16,9 @@ module ActiveAgent
     # @return [Proc] callback proc that accepts (action_name, *args, **kwargs)
     def tools_function
       proc do |action_name, *args, **kwargs|
-        process(action_name, *args, **kwargs)
+        # Marked as a tool call so a refusal is reported to the model as a
+        # result rather than raised through the run (see Authorization).
+        with_tool_call { process(action_name, *args, **kwargs) }
       end
     end
   end

data/lib/active_agent/delegation/runner.rb

@@ -117,6 +117,7 @@ module ActiveAgent
       def generate(arguments)
         agent = definition.resolved_agent_class.new
         agent.params = resolved_params(arguments)
+        inherit_actor(agent)
         agent.process(definition.action, **arguments)
 
         definition.backend.apply(agent)
@@ -126,6 +127,22 @@ module ActiveAgent
         agent.process_prompt
       end
 
+      # A delegated generation runs on behalf of whoever the parent runs for.
+      # The sub-agent gets the parent's caller before its action runs, so its
+      # own before_action callbacks and any scope its tools read through decide
+      # against the same person — a parent authorized as one user must not
+      # hand its specialists an unattributed run, which a correctly written
+      # host scope reads as "no access". Hosts on an older framework, where an
+      # agent has no caller to carry, are left as they were.
+      #
+      # @param agent [ActiveAgent::Base]
+      # @return [void]
+      def inherit_actor(agent)
+        return unless owner.respond_to?(:current_user) && agent.respond_to?(:current_user=)
+
+        agent.current_user = owner.current_user
+      end
+
       # A delegated generation is part of its parent's work, so it carries the
       # parent's trace id — otherwise the sub-agent's tokens and latency land
       # in a separate trace and the budget you set has nothing to show for it.

data/lib/active_agent/evals/diagnosis.rb

@@ -12,6 +12,7 @@ module ActiveAgent
     #   tool_error               — a tool the agent called returned an error
     #   missing_capability       — the agent said no tool covers the task
     #   expected_tool_not_called — the scenario expects a tool the agent did not call
+    #   ungrounded_answer        — the answer states specifics no tool call supplied
     #   forbidden_content        — the answer contains a pattern the scenario forbids
     #   missing_content          — the answer lacks a pattern the scenario expects
     #   low_quality              — the answer scored below the threshold
@@ -21,7 +22,7 @@ module ActiveAgent
     # Returns nil for a passing result.
     class Diagnosis
       FAULTS = %w[
-        run_error tool_error missing_capability expected_tool_not_called
+        run_error tool_error missing_capability expected_tool_not_called ungrounded_answer
         forbidden_content missing_content low_quality judge_unavailable
       ].freeze
 
@@ -38,6 +39,19 @@ module ActiveAgent
         /\bcan(?:'|no)t (?:be )?(?:done|determined|answered) with (?:the|my) (?:current|available) tools\b/i
       ].freeze
 
+      # Phrasings that state a specific fact — a record id, a date, a count of
+      # things — which an agent that called no tool can only have invented.
+      # Deliberately narrow: a number inside prose ("here are three options",
+      # "within 30 days") is not a claim about data, and a false positive here
+      # fails a scenario that may have passed on its merits.
+      SPECIFIC_CLAIMS = [
+        /#\d+\b/,
+        /\b\d{4}-\d{2}-\d{2}\b/,
+        /\b(?:you have|there are|there is|we have|I found|found|showing|a total of)\s+(?:\*\*)?\d+\b/i,
+        /\b\d+\s+(?:\*\*)?(?:open|overdue|pending|active|closed|resolved|completed|unpaid|outstanding|new|matching|
+          records?|results?|rows?|entries|items?|tickets?|orders?|tasks?|issues?|invoices?|customers?|users?|milestones?)\b/ix
+      ].freeze
+
       Result = Struct.new(:fault, :summary, :recommendation, :evidence, keyword_init: true) do
         def to_h
           {
@@ -76,7 +90,7 @@ module ActiveAgent
       end
 
       def call
-        run_error || tool_error || missing_capability || expected_tool_not_called ||
+        run_error || tool_error || missing_capability || expected_tool_not_called || ungrounded_answer ||
           forbidden_content || missing_content || low_quality
       end
 
@@ -178,16 +192,58 @@ module ActiveAgent
             "#{agent} answered with #{called_tools.uniq.join(', ')} instead of #{expected.join(', ')}. Sharpen " \
             "both tools' descriptions so the model can tell them apart, or say in the instructions which tool " \
             "answers this kind of task."
+          elsif asserts_specifics?
+            "#{expected.join(', ')} is available but #{agent.downcase} answered without calling any tool and " \
+            "stated specifics it could not have looked up (\"#{claim_excerpt}\"). Treat the answer as invented: " \
+            "instruct it to answer this kind of task only from a tool result, and to say so when it has none."
           else
             "#{expected.join(', ')} is available but #{agent.downcase} answered without calling any tool. Tell " \
             "it in the instructions to prefer tool-backed answers for this kind of task, and check the tool's " \
             "description says what it returns."
           end
 
-        result("expected_tool_not_called",
-               "Expected #{expected.join(' or ')} to be called; #{agent.downcase} called " \
-               "#{called_tools.uniq.presence&.join(', ') || 'nothing'}.",
-               recommendation, "expected" => expected, "called" => called_tools, "unavailable" => unavailable)
+        summary = "Expected #{expected.join(' or ')} to be called; #{agent.downcase} called " \
+                  "#{called_tools.uniq.presence&.join(', ') || 'nothing'}"
+        summary += " and answered with specifics no tool supplied" if called_tools.empty? && asserts_specifics?
+
+        result("expected_tool_not_called", "#{summary}.", recommendation,
+               "expected" => expected, "called" => called_tools, "unavailable" => unavailable,
+               "ungrounded" => (called_tools.empty? && asserts_specifics?) || nil, "claim" => (claim_excerpt if called_tools.empty?))
+      end
+
+      # The answer states specifics — a count, an id, a date — that no tool
+      # call could have supplied. Reached only when the scenario names no
+      # expected tool (expected_tool_not_called reports the same fabrication
+      # otherwise) and only for an agent that had tools to call: one with
+      # none answers from its instructions by design, and whether that is
+      # acceptable is the judge's call, not a mechanical one.
+      def ungrounded_answer
+        return nil if @available_tools.empty? || called_tools.any?
+        return nil unless asserts_specifics?
+
+        result("ungrounded_answer",
+               "#{agent} stated specifics (\"#{claim_excerpt}\") without calling any tool that could have supplied them.",
+               "Nothing in the answer came from a tool, so the figures in it are invented. Tell #{agent.downcase} in its " \
+               "instructions to answer this kind of task only from a tool result and to say when it has none; if none of " \
+               "#{@available_tools.join(', ')} returns this data, add a tool that does.",
+               "claim" => claim_excerpt, "tools_available" => @available_tools)
+      end
+
+      def asserts_specifics?
+        specific_claim.present?
+      end
+
+      def specific_claim
+        return @specific_claim if defined?(@specific_claim)
+
+        @specific_claim = SPECIFIC_CLAIMS.lazy.filter_map { |pattern| answer.match(pattern) }.first
+      end
+
+      def claim_excerpt
+        match = specific_claim
+        return nil unless match
+
+        answer[[ match.begin(0) - 40, 0 ].max, 120].to_s.strip
       end
 
       def forbidden_content

data/lib/active_agent/evals/judge.rb

@@ -24,6 +24,11 @@ module ActiveAgent
       # @yieldparam instructions [String] the system prompt
       # @yieldparam prompt [String] the user prompt
       # @yieldreturn [String] the completion text
+      # How much of a scenario's notes the judge reads. Where a suite's notes
+      # are its grading rubric, a "Must not…" clause tends to come last, and a
+      # judge that never saw it recommends against it.
+      NOTES_LIMIT = 1_500
+
       def initialize(label:, &generate)
         raise ArgumentError, "Judge.new needs a block that returns the model's completion" unless generate
 
@@ -64,7 +69,7 @@ module ActiveAgent
           ---
           #{scenario.prompt}
           ---
-          #{"Context for the evaluator: #{scenario.notes.truncate(300)}\n" if scenario.notes.present?}
+          #{"Context for the evaluator: #{scenario.notes.truncate(NOTES_LIMIT)}\n" if scenario.notes.present?}
           The assistant answered:
           ---
           #{answer.to_s.truncate(4_000)}
@@ -99,7 +104,7 @@ module ActiveAgent
           Scenario (the user's message):
           #{scenario.prompt}
           #{"Expected tools: #{scenario.expected_tools.join(', ')}" if scenario.expected_tools.any?}
-          #{"Notes: #{scenario.notes.truncate(300)}" if scenario.notes.present?}
+          #{"Notes: #{scenario.notes.truncate(NOTES_LIMIT)}" if scenario.notes.present?}
 
           Tools the agent called:
           #{calls.presence || '(none)'}

data/lib/active_agent/evals/model_spec.rb

@@ -46,22 +46,38 @@ module ActiveAgent
       # duplicates by label.
       def self.parse_all(values, **options)
         values = values.to_s.split(",") unless values.is_a?(Array)
-        values.map { |value| coerce(value) }.reject(&:blank?).uniq.map { |value| parse(value, **options) }
+        values.filter_map { |value| from_value(value, **options) }.uniq(&:label)
       end
 
-      # The label of one requested model. A caller that round-trips a spec —
-      # the dashboard persists `specs.map(&:to_h)` and hands it back on a
-      # re-run — sends a Hash, whose `to_s` is the inspected Hash and reaches
-      # the provider as the model ID: `{"label" => "openrouter/gpt-4o-mini",
-      # ...} is not a valid model ID`.
+      # One requested model, from the text a user typed or from a spec handed
+      # back whole. The dashboard persists `specs.map(&:to_h)` and returns it on
+      # a re-run, so a value may be a Hash: one that names both `provider` and
+      # `model` is rebuilt exactly as it ran, because re-parsing its label
+      # would route a vendor-prefixed model the wrong way —
+      # `"anthropic/claude-sonnet-4.5"` run through OpenRouter came back as
+      # Anthropic's own `claude-sonnet-4.5` the moment that provider was
+      # installed. A Hash naming only a label or a model is parsed from that
+      # text; anything naming nothing is dropped.
       #
-      # @return [String, nil] nil when the value names no model
-      def self.coerce(value)
-        return value.to_h.stringify_keys.values_at("label", "model").compact.first.to_s.strip if value.respond_to?(:to_h) && !value.is_a?(String)
+      # @return [ModelSpec, nil]
+      def self.from_value(value, **options)
+        text =
+          if value.respond_to?(:to_h) && !value.is_a?(String)
+            hash = value.to_h.stringify_keys
+            if hash["provider"].present? && hash["model"].present?
+              return new(label: hash["label"].presence || hash["model"], provider: hash["provider"], model: hash["model"])
+            end
 
-        value.to_s.strip
+            hash.values_at("label", "model").compact.first.to_s.strip
+          else
+            value.to_s.strip
+          end
+
+        return nil if text.blank?
+
+        parse(text, **options)
       end
-      private_class_method :coerce
+      private_class_method :from_value
 
       # The provider a bare model name runs under. A rule whose provider the
       # caller does not offer is skipped, so an app without Ollama does not

data/lib/active_agent/evals/runner.rb

@@ -28,7 +28,7 @@ module ActiveAgent
     class Runner
       # Faults where a judge can add something the evidence alone cannot: what
       # tool to add, or how to change the instructions.
-      DEFAULT_REFINE_FAULTS = %w[missing_capability expected_tool_not_called low_quality missing_content].freeze
+      DEFAULT_REFINE_FAULTS = %w[missing_capability expected_tool_not_called ungrounded_answer low_quality missing_content].freeze
       DEFAULT_JUDGE_LIMIT = 25
 
       attr_reader :scenarios, :models, :criteria, :judge, :threshold

data/lib/active_agent/evals/scorer.rb

@@ -49,7 +49,10 @@ module ActiveAgent
           hit = scenario.forbidden_patterns.any? { |pattern| self.class.matches_pattern?(answer, pattern) }
           scores["forbidden_content"] = hit ? 0.0 : 1.0
         end
-        if replay.tool_calls.any?
+        # A tool that ran without erroring is evidence only when it is one the
+        # scenario expected: a wrong tool that succeeded used to outscore
+        # calling nothing at all.
+        if replay.tool_calls.any? && (scenario.expected_tools.empty? || (scenario.expected_tools & replay.tool_names).any?)
           scores["tools_succeeded"] = replay.failed_tool_calls.any? ? 0.0 : 1.0
         end
 

data/lib/active_agent/generation_job.rb

@@ -18,8 +18,14 @@ module ActiveAgent
 
     rescue_from StandardError, with: :handle_exception_with_agent_class
 
-    def perform(agent, agent_method, generation_method, args:, kwargs: nil, params: nil)
+    # +actor+ is the caller the generation runs on behalf of
+    # (ActiveAgent::Authorization). ActiveJob serializes it like any other
+    # argument, so a record arrives as the same record the caller passed and
+    # the agent's authorization callbacks decide against a real user rather
+    # than against nil.
+    def perform(agent, agent_method, generation_method, args:, kwargs: nil, params: nil, actor: nil)
       agent_class = params ? agent.constantize.with(params) : agent.constantize
+      agent_class = agent_class.as(actor) if actor
       prompt = if kwargs
         agent_class.public_send(agent_method, *args, **kwargs)
       else

data/lib/active_agent/schema_tools.rb

@@ -168,6 +168,71 @@ module ActiveAgent
         @scope = block
       end
 
+      # Runtime-defined tool classes, keyed by model name. A definition built
+      # by {.define} replaces the previous one for its model, so a registry
+      # that is rebuilt on every change — from a table, from a dashboard
+      # edit — holds one class per model rather than one per rebuild.
+      #
+      # @return [Hash{String => Class}]
+      def registry
+        @registry ||= {}
+      end
+
+      # Builds a tool class from a declaration rather than from a file:
+      #
+      #   ActiveAgent::SchemaTools.define(Reservation,
+      #     filterable: %i[status guest_id],
+      #     returns: %i[id status guest_id arrives_on],
+      #     policy: true)   # ReservationPolicy::Scope, as scope_by_policy would
+      #
+      # The class behaves exactly as a file-defined one — same roster, same
+      # allowlists, same +call+ — and is named "<Model>Tools" for logs and
+      # telemetry. It is marked runtime-built so discovery does not read it
+      # back out of +descendants+ (where every class ever built stays until
+      # collected), and registered under its model, replacing whatever the
+      # registry held: that is what keeps a rebuild from accumulating classes
+      # (#441). +scope:+ takes a lambda or proc receiving the actor; +policy:+
+      # resolves the model's policy by name; neither means unscoped, as for a
+      # file-defined class.
+      #
+      # @param model [Class] an ActiveRecord class
+      # @param filterable [Array<Symbol, String>]
+      # @param returns [Array<Symbol, String>]
+      # @param scope [Proc, nil]
+      # @param policy [Boolean, Class] true for the conventional policy, or the policy class
+      # @param name [String, nil] the class name, "<Model>Tools" by default
+      # @return [Class]
+      def define(model, filterable: [], returns: [], scope: nil, policy: false, name: nil)
+        klass = Class.new(self)
+        klass.instance_variable_set(:@runtime, true)
+        class_name = name || "#{model.name}Tools"
+        klass.define_singleton_method(:name) { class_name }
+        klass.model(model)
+        klass.filterable(*filterable) if filterable.present?
+        klass.returns(*returns) if returns.present?
+        klass.scope_by_policy(policy == true ? nil : policy) if policy
+        klass.scope(&scope) if scope
+
+        registry[model.name] = klass
+      end
+
+      # Drops the runtime definition for +model+; discovery stops offering it.
+      #
+      # @param model [Class, String]
+      # @return [Class, nil] the class that was registered
+      def undefine(model)
+        registry.delete(model.respond_to?(:name) ? model.name : model.to_s)
+      end
+
+      # Whether this class was built by {.define} rather than loaded from a
+      # file. Discovery reads runtime classes from {.registry}, never from
+      # +descendants+, so a superseded one is not offered twice.
+      #
+      # @return [Boolean]
+      def runtime?
+        @runtime == true
+      end
+
       # The full, fixed tool roster in provider function-calling format.
       #
       # @return [Array<Hash>] tool definitions with :name, :description, :parameters

data/lib/active_agent/version.rb

@@ -1,3 +1,3 @@
 module ActiveAgent
-  VERSION = "1.5.2"
+  VERSION = "1.6.0"
 end

data/lib/active_agent.rb

@@ -92,6 +92,9 @@ module ActiveAgent
   #
   # These components are loaded on-demand when first referenced.
   autoload :Base
+  # The refusal an agent raises, and the default in Base.authorization_errors.
+  # Reachable before any agent class has loaded, so the dashboard can name it.
+  autoload :NotAuthorized, "active_agent/concerns/authorization"
   autoload :Callbacks, "active_agent/concerns/callbacks"
   autoload :Delegation, "active_agent/concerns/delegation"
   autoload :Streaming, "active_agent/concerns/streaming"

data/lib/generators/active_agent/schema_tools/USAGE

@@ -0,0 +1,22 @@
+Description:
+    Writes a starter ActiveAgent::SchemaTools class for a model under
+    app/agent_tools. The class generates find_<records>, count_<records> and
+    get_<record> tools, and exposes nothing beyond `id` until you move a
+    column into `filterable` or `returns` — every column the model has is
+    listed, commented out, so the allowlist is a review step rather than a
+    blank page. Columns that look like secrets are left out of the list.
+
+    When <Model>Policy::Scope exists the class scopes every read through it
+    (`scope_by_policy`); otherwise a `scope` block is suggested. Pass
+    --policy or --no-policy to decide explicitly.
+
+Examples:
+    `bin/rails generate active_agent:schema_tools Reservation`
+
+    creates:
+        app/agent_tools/reservation_tools.rb
+
+    `bin/rails generate active_agent:schema_tools Reservation --policy`
+
+    scopes reads through ReservationPolicy::Scope even if it cannot be
+    found from here.

data/lib/generators/active_agent/schema_tools/schema_tools_generator.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+module ActiveAgent
+  module Generators
+    # Writes a starter ActiveAgent::SchemaTools class for one model:
+    #
+    #   bin/rails generate active_agent:schema_tools Reservation
+    #
+    # The class it writes exposes nothing beyond +id+ until someone uncomments
+    # a column, because which columns an agent may filter on and read back is
+    # a judgement about exposure, not a fact about the table (#440). Every
+    # column the model has is listed, commented out, minus the ones that look
+    # like secrets, so the allowlist is a review step rather than a blank page.
+    class SchemaToolsGenerator < ::Rails::Generators::NamedBase
+      source_root File.expand_path("templates", __dir__)
+
+      # Columns never suggested, whatever the model: reading one back would
+      # hand a model a credential, and filtering on one leaks it a character
+      # at a time through the row counts.
+      SECRET_COLUMNS = /password|digest|token|secret|api_key|otp|encrypted|ssn/i
+
+      class_option :policy, type: :boolean, default: nil,
+        desc: "Scope every read through <Model>Policy::Scope (default: when that policy exists)"
+
+      check_class_collision suffix: "Tools"
+
+      def create_tools_file
+        template "schema_tools.rb.tt", File.join("app/agent_tools", class_path, "#{file_name}_tools.rb")
+      end
+
+      private
+
+      # "Reservation" and "ReservationTools" both name the Reservation model.
+      def file_name # :doc:
+        @_file_name ||= super.sub(/_tools\z/i, "")
+      end
+
+      def model_class
+        @model_class ||= class_name.safe_constantize
+      end
+
+      def policy_class_name
+        "#{class_name}Policy"
+      end
+
+      def policy?
+        return options[:policy] unless options[:policy].nil?
+
+        "#{policy_class_name}::Scope".safe_constantize.present?
+      end
+
+      # [name, type] for every column the model has, or [] when the model or
+      # its table cannot be read from here (a model that does not exist yet,
+      # or a database that is not set up) — the file is still written.
+      def columns
+        @columns ||= begin
+          if model_class.respond_to?(:columns) && model_class.respond_to?(:table_exists?) && model_class.table_exists?
+            model_class.columns.map { |column| [ column.name, column.type ] }
+          else
+            []
+          end
+        rescue StandardError
+          []
+        end
+      end
+
+      def suggested_columns
+        columns.reject { |name, _type| name == "id" || name.match?(SECRET_COLUMNS) }
+      end
+
+      def secret_columns
+        columns.select { |name, _type| name.match?(SECRET_COLUMNS) }.map(&:first)
+      end
+
+      def collection_name
+        class_name.demodulize.underscore.pluralize
+      end
+
+      def resource_name
+        class_name.demodulize.underscore
+      end
+    end
+  end
+end

data/lib/generators/active_agent/schema_tools/templates/schema_tools.rb.tt

@@ -0,0 +1,48 @@
+<% module_namespacing do -%>
+# Bounded, read-only agent tools over <%= class_name %>:
+# find_<%= collection_name %>, count_<%= collection_name %>, get_<%= resource_name %>.
+#
+# Nothing is exposed until you say so. Move a column from the commented lists
+# into `filterable` (an agent may filter on it) or `returns` (an agent may
+# read it back). Each one is a judgement about exposure, not a fact about the
+# table: leave out anything a model should never see.
+class <%= class_name %>Tools < ActiveAgent::SchemaTools
+  model <%= class_name %>
+
+  filterable :id
+<% if suggested_columns.any? -%>
+  # filterable <%= suggested_columns.map { |name, _type| ":#{name}" }.join(", ") %>
+<% else -%>
+  # filterable :status, :owner_id   # columns an agent may filter on
+<% end -%>
+
+  returns :id
+<% if suggested_columns.any? -%>
+  # returns <%= suggested_columns.map { |name, _type| ":#{name}" }.join(", ") %>
+<% else -%>
+  # returns :id, :title, :status    # columns an agent may read back
+<% end -%>
+<% if suggested_columns.any? -%>
+  #
+  # Columns and their types:
+<% suggested_columns.each do |name, type| -%>
+  #   <%= name.ljust(24) %> <%= type %>
+<% end -%>
+<% end -%>
+<% if secret_columns.any? -%>
+  #
+  # Not suggested, and not to be added: <%= secret_columns.join(", ") %>.
+<% end -%>
+
+<% if policy? -%>
+  # Every read runs through <%= policy_class_name %>::Scope for the acting
+  # user, so an agent sees exactly what that user could see.
+  scope_by_policy
+<% else -%>
+  # Every read should run through a relation scoped to the acting user. Without
+  # a scope the tools read the whole table. Return <%= class_name %>.none for an
+  # actor with no access; never widen for a nil actor.
+  # scope { |actor| actor ? <%= class_name %>.where(owner: actor) : <%= class_name %>.none }
+<% end -%>
+end
+<% end -%>

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: activeagent
 version: !ruby/object:Gem::Version
-  version: 1.5.2
+  version: 1.6.0
 platform: ruby
 authors:
 - Justin Bowen
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-11 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -410,6 +409,7 @@ files:
 - lib/active_agent.rb
 - lib/active_agent/base.rb
 - lib/active_agent/collector.rb
+- lib/active_agent/concerns/authorization.rb
 - lib/active_agent/concerns/callbacks.rb
 - lib/active_agent/concerns/delegation.rb
 - lib/active_agent/concerns/observers.rb
@@ -578,6 +578,9 @@ files:
 - lib/generators/active_agent/agent/agent_generator.rb
 - lib/generators/active_agent/install/USAGE
 - lib/generators/active_agent/install/install_generator.rb
+- lib/generators/active_agent/schema_tools/USAGE
+- lib/generators/active_agent/schema_tools/schema_tools_generator.rb
+- lib/generators/active_agent/schema_tools/templates/schema_tools.rb.tt
 - lib/generators/active_agent/templates/active_agent.yml
 - lib/generators/active_agent/templates/agent.rb.tt
 - lib/generators/active_agent/templates/application_agent.rb.tt
@@ -602,7 +605,6 @@ metadata:
   documentation_uri: https://docs.activeagents.ai
   source_code_uri: https://github.com/activeagents/activeagent
   rubygems_mfa_required: 'true'
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -617,8 +619,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.22
-signing_key:
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Rails AI Agents Framework
 test_files: []