activeagent 1.6.2 → 1.6.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b14c71e437e0c5701b6300861adb5d8b79eba328602e255b38928e6030584a2b
-  data.tar.gz: c5eeeedce5730d88609b5131304571f9d7ecc12d3177ada460de7a4dd56ad61d
+  metadata.gz: 1611f71e0ff6a30b864e797b01cd20242a46acad9df3888c836799f0d297e3d1
+  data.tar.gz: 5f93c9a8c43a98cf7eb599d9efb37f2915c1d2e5ef6f4138464c33cd1f4d3af1
 SHA512:
-  metadata.gz: d76b024171c0a84e4ecc6f6618ccb92d1e94a215f15e15839e8b78b37c8bcd6a312626b8dda996cca23bb2ebcab325fc08fab289689d66143855fa0c372ffb40
-  data.tar.gz: 02605c62c4d22bc001462ee13451fce5f578cb43adb8f9380c8f116ef589629cafb175e10ce019fa073b9086382c2e8dd09bb03d9aab52813f75852a27e43aeb
+  metadata.gz: 7e27caea807dc1d32744260bf157086160d73935fba48aea9c4c59e3234c26a5aee707eb650104a76f2fdf2e59497b0408b6a80392b83ceeab2770b4c8eef3c0
+  data.tar.gz: e44dc91193d40a6fafa9bc79c3cfc3a8c4f2acc215fc0fae0229424218a8c1bc6ca5b96c5803553bb610f89b86eb519b2f0c6061018117b65a38d77eba61100b

data/CHANGELOG.md

@@ -7,6 +7,97 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.6.3] - 2026-09-18
+
+Releases `activeagent` and `actionagent` 1.6.3 from one tag.
+
+A release about telling the truth on the screens that report what happened.
+The context meter now divides the provider's own `prompt_tokens` among its
+segments instead of subtracting estimates from it, and sizes each piece —
+tool schemas, MCP schemas, instructions and the transcript — before the span
+clips it for storage; previously a trace with dense tool schemas showed a
+large message history that was never sent. An adapted replay is metered as
+one execution like any other, so a host that supplies its own runtime is no
+longer silently uncounted, and a spec naming a provider the agent cannot
+serve now fails before the replay rather than reaching it.
+
+Three seams hosts were reaching around become API. `ActiveAgent::Evals::Correlation`
+joins `Runner`'s `around_evaluation:` hook to a telemetry backend's trace
+scope, so a report row links back to the conversation behind it. A `Judge`
+block that accepts `kind:` is told whether it is scoring, recommending or
+writing the verdict, instead of matching on the gem's own instruction prose.
+`Agent#generations` replaces the polymorphic join hosts were copying out of a
+private service method.
+
+Upgrading: no migration, and nothing that already worked changes. The judge
+keyword reaches only a block that asks for it, so existing judges are
+untouched; `Evaluation#replace_scenarios!` keeps `:destroy` as its default.
+Adapters should drop any `ActionAgent.record_usage` call of their own, which
+now double-counts, and any provider allow-list check of their own, which is
+now dead code.
+
+### Added
+
+- The prompt span records how large the tool schemas actually are, as
+  `prompt.input.tools.tokens`, `prompt.input.mcp_tools.tokens`,
+  `prompt.input.instructions.tokens` and `prompt.input.messages.tokens`. The
+  transcript's size is measured before the span trims the history to the turns
+  that fit, the others before their content is clipped. The content attributes
+  beside them are previews clipped for storage — and on the SDK path the tool
+  attribute is a roster of names and parameter keys, several times smaller than
+  the schema the model is sent — so a reader that sized the context from one
+  understated tool pressure badly.
+- MCP tool schemas are attributed apart from the toolbox's, so the context meter
+  can name which half fills the window.
+- `Evaluation#replace_scenarios!` takes `on_removed:` — `:destroy` (the
+  default, unchanged) or `:disable`, which keeps a scenario the suite no
+  longer names as `enabled: false` so earlier runs' results still resolve.
+- **An evaluation's traces link back to the result that caused them.**
+  `ActiveAgent::Evals::Correlation` joins two APIs the module already had but
+  never connected: `Runner`'s `around_evaluation:` hook and its `metadata:`
+  run identity, and a telemetry backend's per-block agent scope. A run mints a
+  `run_id`, each evaluation a `result_id`, and both ride every trace opened
+  inside them as `eval.`-prefixed attributes; the trace ids travel the other
+  way onto `result.replay.metadata` — `trace_id` for the replay,
+  `judge_trace_ids` for the judge calls that graded it, with a run-level
+  verdict landing on the run metadata the Report carries rather than on
+  whichever result was evaluated last. The tracer is injected, so the module
+  takes on no telemetry dependency and `require "active_agent/evals"` still
+  loads on its own. Hand the object to `Runner.new(around_evaluation:)`
+  directly; a plain lambda there keeps working unchanged.
+- A `Judge` block that accepts `kind:` is told which of the judge's three calls
+  it is serving — `:score`, `:recommend` or `:verdict` — so a host can trace,
+  budget or model them separately. Previously the only signal was the
+  `instructions` string, so hosts matched against the gem's own
+  `RECOMMEND_INSTRUCTIONS` / `VERDICT_INSTRUCTIONS` constants; rewording one
+  then sent every such host quietly down its `else` branch, mislabelling traces
+  rather than failing. The keyword reaches only a block that names it or
+  collects `**`, so judges taking `instructions:` and `prompt:` are unaffected.
+  (#462)
+- `Agent#generations` reads the generations recorded against an agent, with
+  `Agent#agent_contexts` beside it. Generations hang off `AgentContext`
+  polymorphically, so reaching them meant hand-writing that join — the engine
+  did it itself in a private service method a host could not reuse, which now
+  uses the association instead. Destroying an agent still leaves its contexts
+  alone, as it always has. (#464)
+
+### Fixed
+
+- The dashboard's context meter divides the provider's own `prompt_tokens`
+  among its segments instead of subtracting its estimates from it. Charging the
+  difference to one segment made "Messages" absorb the whole approximation
+  error, so a trace with dense JSON tool schemas read as a large message history
+  that was never sent. The transcript is one of the divided segments: dividing
+  only the rest would hand its share to the segments that remained, so a long
+  conversation reported an enormous system prompt and no history at all.
+- A host that supplies a `scenario_evaluation_adapter_resolver` now has its
+  replays metered as executions, one per scenario x model, the same unit the
+  default path records. Previously an adapted replay was counted only if the
+  host remembered to call `ActionAgent.record_usage` itself.
+- A scenario evaluation whose selected models name a provider the agent cannot
+  serve fails with `ArgumentError` before any replay runs. A spec handed back
+  as a Hash naming both `provider` and `model` bypassed the `providers:`
+  allow-list, so the run reached the replay with a provider nothing serves.
 ## [1.6.2] - 2026-09-16
 
 Releases `activeagent` and `actionagent` 1.6.2 from one tag.

data/lib/active_agent/evals/correlation.rb

@@ -0,0 +1,178 @@
+# frozen_string_literal: true
+
+require "securerandom"
+require "active_support/isolated_execution_state"
+
+module ActiveAgent
+  module Evals
+    # Used to tie the traces an evaluation produces back to the run and the
+    # result that caused them, so a report row links to the exact conversation
+    # behind it.
+    #
+    # A run mints a `run_id`, each evaluation mints a `result_id`, and both ride
+    # every trace opened inside them as `eval.`-prefixed attributes. The trace
+    # ids travel the other way: a replay's trace id lands on
+    # `result.replay.metadata["trace_id"]`, and every judge call made while
+    # scoring that result appends to its `"judge_trace_ids"`. The verdict — a
+    # judge call made outside any evaluation — appends to the run metadata
+    # instead, which is the same Hash a Report carries as its `metadata`.
+    #
+    # A tracer is `(name, action:, attributes:, on_trace:) { ... }`: it opens a
+    # trace named for the agent, and calls `on_trace` with something answering
+    # to `#trace_id` once the trace is known.
+    #
+    #   correlation = ActiveAgent::Evals::Correlation.new(
+    #     agent_name: "SupportAgent",
+    #     judge_name: "SupportAgentJudge",
+    #     tracer: ->(name, action:, attributes:, on_trace:, &block) {
+    #       MyTelemetry.with_agent(name, action: action, attributes: attributes,
+    #                              on_trace: on_trace, synchronous: true, &block)
+    #     }
+    #   )
+    #
+    #   correlation.with_run("suite" => "support") do |metadata|
+    #     Runner.new(
+    #       scenarios: scenarios, models: models, metadata: metadata,
+    #       replay: ->(scenario, spec) { correlation.replay { agent.run(scenario.prompt) } },
+    #       judge: Judge.new(label: "judge-model") { |instructions:, prompt:|
+    #         correlation.judge("score") { chat.with_instructions(instructions).ask(prompt).content }
+    #       },
+    #       around_evaluation: correlation
+    #     ).call
+    #   end
+    #
+    # Without a tracer the correlation still mints ids and merges metadata. The
+    # blocks run untraced.
+    class Correlation
+      STATE_KEY = :active_agent_evals_correlation
+
+      # DEFAULT_TRACE_KEYS names the correlation metadata that rides a trace as
+      # `eval.`-prefixed attributes. Anything else a caller puts in the run
+      # metadata (a tenant, a role) stays on the report but off the traces.
+      DEFAULT_TRACE_KEYS = %w[run_id result_id suite scenario_key model_label model provider].freeze
+
+      attr_reader :agent_name, :judge_name, :trace_keys
+
+      # @param agent_name [String] the trace name for the agent under evaluation
+      # @param judge_name [String] the trace name for judge traffic, kept distinct
+      #   so grading calls do not read as the agent's own traffic
+      # @param tracer [#call, nil] `(name, action:, attributes:, on_trace:) { ... }`;
+      #   nil runs every block untraced
+      # @param replay_action [String] the action name recorded for a replay trace
+      # @param trace_keys [Array<String>] which correlation keys become attributes
+      def initialize(agent_name:, judge_name: "#{agent_name}Judge", tracer: nil, replay_action: "eval",
+                     trace_keys: DEFAULT_TRACE_KEYS)
+        @agent_name = agent_name
+        @judge_name = judge_name
+        @tracer = tracer
+        @replay_action = replay_action
+        @trace_keys = trace_keys.map(&:to_s)
+      end
+
+      # Opens a run. Mints `run_id` unless `metadata` carries one, and yields the
+      # metadata hash the traces will be correlated against — pass that same hash
+      # to `Runner.new(metadata:)` so the Report carries the run's identity and
+      # collects the verdict's trace id.
+      #
+      # The hash yielded is the caller's own, mutated in place, so a run
+      # reopened around a later verdict accumulates onto the metadata a Report
+      # already carries.
+      #
+      # @param metadata [Hash] opaque run metadata. A non-Hash is coerced with
+      #   `#to_h`, and non-String keys are stringified.
+      # @yieldparam metadata [Hash]
+      def with_run(metadata = {})
+        run = metadata.is_a?(Hash) ? metadata : metadata.to_h
+        run.transform_keys!(&:to_s) unless run.keys.all?(String)
+        run["run_id"] ||= SecureRandom.uuid
+        with_context({ run: run, result: nil }) { yield run }
+      end
+
+      # Wraps one evaluation, in the shape `Runner.new(around_evaluation:)` calls:
+      # `(scenario, spec) { ... } → Result`. Mints a `result_id`, merges the
+      # correlation onto the Result's replay metadata, and returns the Result.
+      def around_evaluation(scenario, spec)
+        result_metadata = {
+          "run_id" => run_metadata["run_id"],
+          "result_id" => SecureRandom.uuid,
+          "scenario_key" => scenario.key,
+          "model_label" => spec.label,
+          "model" => spec.model,
+          "provider" => spec.provider
+        }.compact
+
+        with_context(run: run_metadata, result: result_metadata) do
+          yield.tap { |result| result.replay.metadata.merge!(result_metadata) }
+        end
+      end
+
+      # Delegates to `around_evaluation`, so the object satisfies
+      # `Runner.new(around_evaluation:)` directly.
+      def call(scenario, spec, &)
+        around_evaluation(scenario, spec, &)
+      end
+
+      # Traces one replay of the agent under evaluation. The trace id lands on
+      # the current result's metadata, so a report row links to the conversation.
+      def replay(action = @replay_action, &)
+        trace(@agent_name, action, judge: false, &)
+      end
+
+      # Traces one judge call. Appends to the current result's `judge_trace_ids`,
+      # or the run's when no evaluation is open (the verdict).
+      def judge(action = "score", &)
+        trace(@judge_name, action, judge: true, &)
+      end
+
+      # The correlation metadata in scope, or nil outside a run. A result's
+      # values win over the run's.
+      def current
+        context = ActiveSupport::IsolatedExecutionState[STATE_KEY]
+        return nil unless context
+
+        context.fetch(:run, {}).merge(context[:result] || {})
+      end
+
+      private
+
+      def run_metadata
+        context = ActiveSupport::IsolatedExecutionState[STATE_KEY]
+        context&.fetch(:run, nil) || {}
+      end
+
+      def with_context(context)
+        previous = ActiveSupport::IsolatedExecutionState[STATE_KEY]
+        ActiveSupport::IsolatedExecutionState[STATE_KEY] = context
+        yield
+      ensure
+        ActiveSupport::IsolatedExecutionState[STATE_KEY] = previous
+      end
+
+      def trace(name, action, judge:, &block)
+        return block.call unless @tracer
+
+        context = ActiveSupport::IsolatedExecutionState[STATE_KEY] || {}
+        correlation = context.fetch(:run, {}).merge(context[:result] || {})
+        attributes = correlation.slice(*@trace_keys).transform_keys { |key| "eval.#{key}" }
+        # A replay belongs to the evaluation that opened it and nowhere else, so
+        # it records no trace id when called outside one. A judge call outside an
+        # evaluation is the verdict, which belongs to the run.
+        target = judge ? (context[:result] || context[:run]) : context[:result]
+
+        @tracer.call(name, action: action, attributes: attributes, on_trace: recorder(target, judge: judge), &block)
+      end
+
+      def recorder(target, judge:)
+        lambda do |trace|
+          next unless target
+
+          if judge
+            (target["judge_trace_ids"] ||= []) << trace.trace_id
+          else
+            target["trace_id"] = trace.trace_id
+          end
+        end
+      end
+    end
+  end
+end

data/lib/active_agent/evals/judge.rb

@@ -10,6 +10,21 @@ module ActiveAgent
     #     RubyLLM.chat(model: "claude-opus-5").with_instructions(instructions).ask(prompt).content
     #   end
     #
+    # A judge serves three different calls, and a block that accepts `kind:` is
+    # told which one it is serving — `:score`, `:recommend` or `:verdict` — so a
+    # host can trace them apart, budget them apart, or score with a cheaper model
+    # than it writes the verdict with:
+    #
+    #   Judge.new(label: "claude-opus-5") do |instructions:, prompt:, kind:|
+    #     model = kind == :score ? "claude-haiku-4-5" : "claude-opus-5"
+    #     RubyLLM.chat(model: model).with_instructions(instructions).ask(prompt).content
+    #   end
+    #
+    # The keyword is passed only to a block that names it (or collects `**`), so
+    # a two-keyword block written before this is unaffected. Without it the only
+    # signal is the `instructions` string, which means matching on the gem's own
+    # prose — and a reworded constant then mislabels silently instead of failing.
+    #
     # Every method returns nil when the judge fails or answers unusably, so an
     # evaluation degrades to rule scoring rather than aborting.
     class Judge
@@ -23,6 +38,8 @@ module ActiveAgent
       # @param label [String] how reports name the judge (usually its model)
       # @yieldparam instructions [String] the system prompt
       # @yieldparam prompt [String] the user prompt
+      # @yieldparam kind [Symbol] which call this is — `:score`, `:recommend` or
+      #   `:verdict`. Passed only to a block that accepts it.
       # @yieldreturn [String] the completion text
       # How much of a scenario's notes the judge reads. Where a suite's notes
       # are its grading rubric, a "Must not…" clause tends to come last, and a
@@ -41,7 +58,7 @@ module ActiveAgent
         return nil if answer.blank?
 
         guidance = criterion.dig("config", "prompt").presence || criterion["key"].to_s.humanize
-        parse_score(ask(SCORE_INSTRUCTIONS, <<~PROMPT))
+        parse_score(ask(SCORE_INSTRUCTIONS, <<~PROMPT, :score))
           Criterion: #{guidance}
 
           The user asked:
@@ -64,7 +81,7 @@ module ActiveAgent
       def score_task(scenario:, answer:)
         return nil if answer.blank?
 
-        parse_score(ask(SCORE_INSTRUCTIONS, <<~PROMPT))
+        parse_score(ask(SCORE_INSTRUCTIONS, <<~PROMPT, :score))
           A user asked an assistant:
           ---
           #{scenario.prompt}
@@ -90,7 +107,7 @@ module ActiveAgent
           "- #{call['name']}#{' (errored)' if call['error']}: #{call['arguments'].to_json.truncate(200)}"
         end.join("\n")
 
-        parsed = parse_object(ask(RECOMMEND_INSTRUCTIONS, <<~PROMPT))
+        parsed = parse_object(ask(RECOMMEND_INSTRUCTIONS, <<~PROMPT, :recommend))
           An AI agent failed one evaluation scenario. Recommend the fix.
 
           Agent instructions:
@@ -146,7 +163,7 @@ module ActiveAgent
             "#{", faults: #{faults}" if faults.present?}"
         end
 
-        parsed = parse_object(ask(VERDICT_INSTRUCTIONS, <<~PROMPT))
+        parsed = parse_object(ask(VERDICT_INSTRUCTIONS, <<~PROMPT, :verdict))
           An AI agent ran the same scenarios under several models. Its goals:
           ---
           #{instructions.to_s.truncate(1_000).presence || '(no instructions configured)'}
@@ -180,13 +197,34 @@ module ActiveAgent
         end
       end
 
-      def ask(instructions, prompt)
-        @generate.call(instructions: instructions, prompt: prompt).to_s
+      def ask(instructions, prompt, kind)
+        @generate.call(**ask_arguments(instructions, prompt, kind)).to_s
       rescue StandardError => e
         warn_failure(e)
         nil
       end
 
+      # The block signature is public API, and every judge written before `kind:`
+      # existed takes exactly `instructions:` and `prompt:` — passing a third
+      # keyword to one of those raises ArgumentError, which `ask` would swallow
+      # as a judge failure, degrading the run to rule scoring. So the kind goes
+      # only to a block that asked for it.
+      def ask_arguments(instructions, prompt, kind)
+        arguments = { instructions: instructions, prompt: prompt }
+        arguments[:kind] = kind if generate_accepts_kind?
+        arguments
+      end
+
+      # True for a block naming `kind:` or collecting `**`. Memoized because the
+      # answer cannot change for a given judge and `ask` runs per scored result.
+      def generate_accepts_kind?
+        return @generate_accepts_kind if defined?(@generate_accepts_kind)
+
+        @generate_accepts_kind = @generate.parameters.any? do |type, name|
+          type == :keyrest || (name == :kind && (type == :key || type == :keyreq))
+        end
+      end
+
       def warn_failure(error)
         message = "[ActiveAgent::Evals] judge #{label} failed: #{error.class}: #{error.message}"
         if defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger

data/lib/active_agent/evals.rb

@@ -25,6 +25,7 @@ require_relative "evals/design_tokens"
 require_relative "evals/report_html"
 require_relative "evals/report"
 require_relative "evals/runner"
+require_relative "evals/correlation"
 require_relative "evals/publisher"
 
 # Scenario evaluations for agents that answer with tools.
@@ -40,7 +41,8 @@ require_relative "evals/publisher"
 # fell short and what would fix it (Diagnosis, refined by an optional Judge),
 # and rolling everything up per model (Report). Runner ties them together
 # around one callable you supply: given a scenario and a model, run the
-# agent and return a Replay.
+# agent and return a Replay. Correlation is optional plumbing on top: it
+# links the traces a run emits back to the result that caused them.
 #
 #   scenarios = ActiveAgent::Evals::ScenarioParser.scenarios(pasted_text)
 #   models    = ActiveAgent::Evals::ModelSpec.parse_all(%w[gpt-5-mini qwen3:8b], default_provider: "openai")

data/lib/active_agent/telemetry/instrumentation.rb

@@ -101,7 +101,23 @@ module ActiveAgent
                     parameters: properties.is_a?(Hash) ? properties.keys : []
                   }.compact
                 }
-                prompt_span.set_attribute("prompt.input.tools", JSON.generate(roster)) if roster.any?
+                if roster.any?
+                  prompt_span.set_attribute("prompt.input.tools", JSON.generate(roster))
+                  # The roster is a reading aid: names, a truncated description
+                  # and parameter keys. What the model is sent is the full JSON
+                  # Schema — types, enums, nested objects, anyOf branches —
+                  # which for a twelve-tool agent is several times the roster,
+                  # so a context meter sizing tool pressure from the roster
+                  # understates it badly. The schemas come from the host and
+                  # need not be serializable; a size is worth less than the
+                  # generation it would otherwise take down.
+                  tools_size = begin
+                    JSON.generate(tools).length
+                  rescue StandardError
+                    nil
+                  end
+                  prompt_span.set_attribute("prompt.input.tools.tokens", (tools_size / 4.0).round) if tools_size
+                end
               end
 
               # prompt_options[:messages] holds the turns a caller passed

data/lib/active_agent/version.rb

@@ -1,3 +1,3 @@
 module ActiveAgent
-  VERSION = "1.6.2"
+  VERSION = "1.6.3"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: activeagent
 version: !ruby/object:Gem::Version
-  version: 1.6.2
+  version: 1.6.3
 platform: ruby
 authors:
 - Justin Bowen
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-17 00:00:00.000000000 Z
+date: 2026-09-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -434,6 +434,7 @@ files:
 - lib/active_agent/delegation/schema.rb
 - lib/active_agent/deprecator.rb
 - lib/active_agent/evals.rb
+- lib/active_agent/evals/correlation.rb
 - lib/active_agent/evals/design_tokens.rb
 - lib/active_agent/evals/diagnosis.rb
 - lib/active_agent/evals/judge.rb