activeagent 1.6.0 → 1.6.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1838cd05b12c4bc56330e8c4b4ec98295e885990bc67b600fefaa401f2543593
-  data.tar.gz: dcd3590de72aef1a8059a6fb03d9303672e73260a59c33e9244ccf326d48b7b6
+  metadata.gz: 1611f71e0ff6a30b864e797b01cd20242a46acad9df3888c836799f0d297e3d1
+  data.tar.gz: 5f93c9a8c43a98cf7eb599d9efb37f2915c1d2e5ef6f4138464c33cd1f4d3af1
 SHA512:
-  metadata.gz: 1534a469b0eeec15765b71e7b43e8362660d9fc7610531c757acb882e130a95cc938a2033b2d0982233558123a0b162d6efb9c51da3e7038d0f0f1d6fcfe8d9e
-  data.tar.gz: eb849f183afd357f381836947891c8029d5ca4772f2e4bf1fda3fedb30ebb7f71db1021f9f94a8ba62337a99493b54b0124f9f0e28d8342fde4182d8ebf02411
+  metadata.gz: 7e27caea807dc1d32744260bf157086160d73935fba48aea9c4c59e3234c26a5aee707eb650104a76f2fdf2e59497b0408b6a80392b83ceeab2770b4c8eef3c0
+  data.tar.gz: e44dc91193d40a6fafa9bc79c3cfc3a8c4f2acc215fc0fae0229424218a8c1bc6ca5b96c5803553bb610f89b86eb519b2f0c6061018117b65a38d77eba61100b

data/CHANGELOG.md

@@ -7,6 +7,228 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.6.3] - 2026-09-18
+
+Releases `activeagent` and `actionagent` 1.6.3 from one tag.
+
+A release about telling the truth on the screens that report what happened.
+The context meter now divides the provider's own `prompt_tokens` among its
+segments instead of subtracting estimates from it, and sizes each piece —
+tool schemas, MCP schemas, instructions and the transcript — before the span
+clips it for storage; previously a trace with dense tool schemas showed a
+large message history that was never sent. An adapted replay is metered as
+one execution like any other, so a host that supplies its own runtime is no
+longer silently uncounted, and a spec naming a provider the agent cannot
+serve now fails before the replay rather than reaching it.
+
+Three seams hosts were reaching around become API. `ActiveAgent::Evals::Correlation`
+joins `Runner`'s `around_evaluation:` hook to a telemetry backend's trace
+scope, so a report row links back to the conversation behind it. A `Judge`
+block that accepts `kind:` is told whether it is scoring, recommending or
+writing the verdict, instead of matching on the gem's own instruction prose.
+`Agent#generations` replaces the polymorphic join hosts were copying out of a
+private service method.
+
+Upgrading: no migration, and nothing that already worked changes. The judge
+keyword reaches only a block that asks for it, so existing judges are
+untouched; `Evaluation#replace_scenarios!` keeps `:destroy` as its default.
+Adapters should drop any `ActionAgent.record_usage` call of their own, which
+now double-counts, and any provider allow-list check of their own, which is
+now dead code.
+
+### Added
+
+- The prompt span records how large the tool schemas actually are, as
+  `prompt.input.tools.tokens`, `prompt.input.mcp_tools.tokens`,
+  `prompt.input.instructions.tokens` and `prompt.input.messages.tokens`. The
+  transcript's size is measured before the span trims the history to the turns
+  that fit, the others before their content is clipped. The content attributes
+  beside them are previews clipped for storage — and on the SDK path the tool
+  attribute is a roster of names and parameter keys, several times smaller than
+  the schema the model is sent — so a reader that sized the context from one
+  understated tool pressure badly.
+- MCP tool schemas are attributed apart from the toolbox's, so the context meter
+  can name which half fills the window.
+- `Evaluation#replace_scenarios!` takes `on_removed:` — `:destroy` (the
+  default, unchanged) or `:disable`, which keeps a scenario the suite no
+  longer names as `enabled: false` so earlier runs' results still resolve.
+- **An evaluation's traces link back to the result that caused them.**
+  `ActiveAgent::Evals::Correlation` joins two APIs the module already had but
+  never connected: `Runner`'s `around_evaluation:` hook and its `metadata:`
+  run identity, and a telemetry backend's per-block agent scope. A run mints a
+  `run_id`, each evaluation a `result_id`, and both ride every trace opened
+  inside them as `eval.`-prefixed attributes; the trace ids travel the other
+  way onto `result.replay.metadata` — `trace_id` for the replay,
+  `judge_trace_ids` for the judge calls that graded it, with a run-level
+  verdict landing on the run metadata the Report carries rather than on
+  whichever result was evaluated last. The tracer is injected, so the module
+  takes on no telemetry dependency and `require "active_agent/evals"` still
+  loads on its own. Hand the object to `Runner.new(around_evaluation:)`
+  directly; a plain lambda there keeps working unchanged.
+- A `Judge` block that accepts `kind:` is told which of the judge's three calls
+  it is serving — `:score`, `:recommend` or `:verdict` — so a host can trace,
+  budget or model them separately. Previously the only signal was the
+  `instructions` string, so hosts matched against the gem's own
+  `RECOMMEND_INSTRUCTIONS` / `VERDICT_INSTRUCTIONS` constants; rewording one
+  then sent every such host quietly down its `else` branch, mislabelling traces
+  rather than failing. The keyword reaches only a block that names it or
+  collects `**`, so judges taking `instructions:` and `prompt:` are unaffected.
+  (#462)
+- `Agent#generations` reads the generations recorded against an agent, with
+  `Agent#agent_contexts` beside it. Generations hang off `AgentContext`
+  polymorphically, so reaching them meant hand-writing that join — the engine
+  did it itself in a private service method a host could not reuse, which now
+  uses the association instead. Destroying an agent still leaves its contexts
+  alone, as it always has. (#464)
+
+### Fixed
+
+- The dashboard's context meter divides the provider's own `prompt_tokens`
+  among its segments instead of subtracting its estimates from it. Charging the
+  difference to one segment made "Messages" absorb the whole approximation
+  error, so a trace with dense JSON tool schemas read as a large message history
+  that was never sent. The transcript is one of the divided segments: dividing
+  only the rest would hand its share to the segments that remained, so a long
+  conversation reported an enormous system prompt and no history at all.
+- A host that supplies a `scenario_evaluation_adapter_resolver` now has its
+  replays metered as executions, one per scenario x model, the same unit the
+  default path records. Previously an adapted replay was counted only if the
+  host remembered to call `ActionAgent.record_usage` itself.
+- A scenario evaluation whose selected models name a provider the agent cannot
+  serve fails with `ArgumentError` before any replay runs. A spec handed back
+  as a Hash naming both `provider` and `model` bypassed the `providers:`
+  allow-list, so the run reached the replay with a provider nothing serves.
+## [1.6.2] - 2026-09-16
+
+Releases `activeagent` and `actionagent` 1.6.2 from one tag.
+
+Agents gain releases: a digest of everything the model is given, cut on
+deploy and pinned to every trace, run and evaluation run, so a score is a
+statement about a specific release and a regression is attributable to the
+change that caused it. Around it, five dashboard fixes: an evaluation
+created on MySQL can be run, the Tools tab reads the same `agent.tools` the
+runner does, a container-valued query parameter is coerced instead of
+raising, a recording's detail response no longer carries the visitor's
+cookies and web storage, and the MCP endpoint answers an unsupported method with 405
+instead of the dashboard page. `sign_in_path` and `sign_out_path` are now
+documented.
+
+Upgrading: the install generator emits a new `add_agent_releases` migration
+(guarded column by column); run it. Cutting a release is
+`rake action_agent:agents:release[REVISION]` in the deploy.
+
+### Added
+
+- **Agents have releases, and every trace, run and evaluation says which one
+  it ran under.** `ActiveAgent::Release` gives each agent class a digest of
+  what the model is given — provider and model, generation options minus
+  credentials, the actions, the prompt templates on disk, and the tools and
+  delegations it declares — so two deploys of the same agent share a digest
+  and any change to those inputs is a new one, with no number to bump.
+  `ActiveAgent::Release.revision` carries the deploy alongside (a git SHA;
+  read from `SERVICE_VERSION`, `GIT_SHA`, `KAMAL_VERSION` and friends when
+  not set). The instrumentation stamps `agent.version` and `agent.revision`
+  on every generation's root span, and every trace gets `service.version`.
+  In the dashboard, `rake action_agent:agents:release[REVISION]` cuts an
+  `AgentVersion` for each agent whose code changed since the last release —
+  idempotent, so it belongs in the deploy — `rake action_agent:agents:versions`
+  lists them, and `Agent#record_release!` is the call behind both for a host
+  that syncs agents its own way. Traces are pinned to the release their root
+  span names, runs and evaluation runs to the version current when they
+  started (`agent_version_id` on all three; the install generator emits the
+  migration). A version's JSON carries `release`, `release_digest` and
+  `revision`, so the Versions tab tells a deploy from an edit. For that to
+  reach a host's own agents, a trace from a class the host mirrors into the
+  dashboard is now attributed to that mirror — the registrar matched only on
+  service, class *and* action, so every code-path trace registered an
+  observed per-action twin beside the synced record and could never be
+  pinned to its release.
+
+### Fixed
+
+- **An evaluation created on MySQL can be run.** MySQL cannot give a JSON
+  column a default, so an evaluation saved there without `config` read it
+  back as `nil`, and `compare_models` raised before the runner did anything
+  else. `config` and `criteria` now read as the empty value their column
+  default supplies on other databases. (#417)
+- **The Tools tab now says which schema tools an agent is offered, and
+  lets you change it.** The editor listed every schema tool as enabled and
+  read-only whatever `agent.tools` held — *"a checkbox that cannot add or
+  remove the tool is a control that changes nothing"* — while evaluations,
+  dashboard runs and the MCP facade offered exactly what that column named.
+  An agent whose roster had been emptied over the API ran a suite with no
+  tools (1/8, `expected tool not called ×6`) under a tab reading "12
+  enabled". A schema tool's row now reads the roster and is switchable, and
+  every schema tool the host declares has a row, off unless the roster names
+  it — any agent may enable any of them, and a tool switched off has to keep
+  its row to be switched back on. A tool the agent class declares in code is
+  still reported rather than selected: the class offers it, and no checkbox
+  could change that.
+- **A container-valued query parameter no longer 500s the dashboard API.**
+  `page`, `per_page`, `days`, `minutes`, `limit` and `after_sequence` were
+  read with `to_i`, which neither an Array (`minutes[]=1&minutes[]=2`) nor a
+  nested object (`page[x]=1`) answers. `Api::BaseController` now coerces
+  them: a multi-valued parameter means its first value, a nested object falls
+  back to the default, and the clamps that bounded the number still apply.
+  `sandboxes#compare` answers a `providers` value that is not a list of
+  names with a 400 instead of a `NoMethodError`.
+- **A session recording's `show` no longer returns the visitor's cookies and
+  web storage.** Every other read path redacted the handoff state, but the
+  detail response carried `cookies`, `session_storage` and `local_storage`
+  unscrubbed, both as its own key and nested inside `metadata`. Both are now
+  stripped; only `#handoff` returns them, to the recording's owner. (#456)
+
+## [1.6.1] - 2026-09-16
+
+Releases `activeagent` and `actionagent` 1.6.1 from one tag.
+
+A patch for two defects that share a failure mode: each one turns a broken
+run into a plausible-looking success rather than an error. A date filter that
+matched nothing reported zero instead of raising, and an agent reported that
+zero as fact; telemetry that was enabled but never instrumented wrote no
+traces while every configuration signal read healthy. Neither surfaced in a
+test suite, because neither produces a failure — only a confident wrong
+answer and an empty table.
+
+No new public surface and no behaviour change for anything that was already
+working, so a patch under semver. Suites that filter on a date column will
+report different — correct — numbers after upgrading; read the first run as a
+corrected baseline.
+
+### Fixed
+
+- **A range filter on a `SchemaTools` column no longer matches nothing and
+  reports zero.** `permitted_filters!` validated the column against the
+  allowlist but passed the value through untouched, so a range hash reached
+  `where` unrecognized and Rails compiled `where(due_date: {"before" => x})`
+  to `due_date = NULL` — a predicate that matches no row. The tool returned
+  `{count: 0}` with no error and the model read it as a truthful empty
+  answer: "0 overdue tickets" against a database holding four. Equality
+  filters were unaffected, which is why this went unnoticed. Comparisons are
+  now built through Arel with the column's own type cast, under the operators
+  `before`, `after`, `lt`, `lte`, `gt`, `gte`, `on_or_before` and
+  `on_or_after`; two bounds may be given together to express a window; and an
+  operator outside that set raises `UnpermittedAttribute` rather than
+  returning zero, consistent with how an undeclared column is already
+  rejected. Ranges are offered for date, datetime, time and numeric columns
+  only — a lexical `>` on a name column answers a question nobody asked.
+- **A range filter is now discoverable.** `filter_properties` described a date
+  column as a bare `{type: "string", format: "date"}`, so the tool surface
+  could not express "before today" at all and a model asking the question
+  correctly still had no way to ask it. Comparable columns are now offered as
+  `anyOf: [scalar, range object]`, with the operator roster in the schema.
+- **Telemetry enabled from a host app's initializer now installs
+  instrumentation.** The railtie prepended `GenerationInstrumentation` only
+  when `Telemetry.enabled?` was already true as railties ran — before
+  `config/initializers/*.rb`. An app that configures telemetry in its own
+  initializer, which is what the documentation shows, was therefore never
+  instrumented: `enabled?` answered true, `local_storage` was on, the trace
+  model resolved and the store lambda worked when called directly, and no
+  generation ever produced a span to store. `configure` now installs as well
+  when the resulting configuration is enabled; `instrument_telemetry!` is
+  idempotent, so the railtie path and the configure path cannot
+  double-prepend and initializer order stops mattering.
+
 ## [1.6.0] - 2026-09-14
 
 Releases `activeagent` and `actionagent` 1.6.0 from one tag.
@@ -142,7 +364,6 @@ still correct: 1.6.0 satisfies it.
   hash naming both `provider` and `model` is now rebuilt as it was; a bare
   label is still parsed. The dashboard's "re-run" of a saved selection is
   the path this fixes.
-
 ## [1.5.2] - 2026-09-11
 
 Releases `activeagent` and `actionagent` 1.5.2 from one tag.

data/lib/active_agent/base.rb

@@ -12,6 +12,7 @@ require "active_agent/concerns/parameterized"
 require "active_agent/concerns/preview"
 require "active_agent/concerns/provider"
 require "active_agent/concerns/queueing"
+require "active_agent/concerns/release"
 require "active_agent/concerns/rescue"
 require "active_agent/concerns/streaming"
 require "active_agent/concerns/tooling"
@@ -49,6 +50,7 @@ module ActiveAgent
     include Parameterized
     include Provider
     include Queueing
+    include Release
     include Rescue
     include Streaming
     include Tooling

data/lib/active_agent/concerns/release.rb

@@ -0,0 +1,171 @@
+# frozen_string_literal: true
+
+require "digest"
+require "json"
+
+module ActiveAgent
+  # A release of an agent is what the model is given: the provider and model,
+  # the generation options, the prompt templates on disk, the actions and the
+  # tools the class declares. {ClassMethods#release_digest} names that
+  # deterministically, so two deploys that ship the same agent share a digest
+  # and a change to any of those inputs yields a new one — without anyone
+  # bumping a number by hand.
+  #
+  # The digest is what telemetry stamps on every generation (`agent.version`)
+  # and what a dashboard cuts an AgentVersion from on deploy, so a trace, a
+  # run and an evaluation can all say which release of the agent produced
+  # them. {Release.revision} carries the deploy itself (a git SHA or a release
+  # label) alongside, when the host knows it.
+  #
+  # Class-level and memoized: in development a reload replaces the class, so
+  # the next reference recomputes it.
+  module Release
+    extend ActiveSupport::Concern
+
+    # Option keys that never belong in a manifest: credentials, and per-call
+    # state the class does not own.
+    EXCLUDED_OPTION_KEYS = %i[
+      api_key access_token secret password token trace_id messages message instructions
+    ].freeze
+    SECRET_KEY_PATTERN = /key|token|secret|password|credential/i
+
+    # How the deploy identifies itself, when it does. A host sets
+    # `ActiveAgent::Release.revision = ENV["GIT_SHA"]` (or a proc) from an
+    # initializer; otherwise the conventional deploy variables are read.
+    class << self
+      attr_writer :revision
+
+      # @return [String, nil]
+      def revision
+        value = @revision.respond_to?(:call) ? @revision.call : @revision
+        value = value.presence || ENV.values_at("SERVICE_VERSION", "GIT_SHA", "KAMAL_VERSION", "SOURCE_VERSION", "HEROKU_SLUG_COMMIT").find(&:present?)
+        value&.to_s
+      end
+
+      # Canonical JSON: sorted keys at every level, so the digest does not
+      # depend on the order anything was declared in.
+      # @api private
+      def canonical(value)
+        case value
+        when Hash then value.map { |k, v| [ k.to_s, canonical(v) ] }.sort_by(&:first).to_h
+        when Array then value.map { |v| canonical(v) }
+        when Symbol then value.to_s
+        else value
+        end
+      end
+    end
+
+    class_methods do
+      # Everything about this class that shapes a generation, as data.
+      #
+      # @return [Hash]
+      def release_manifest
+        @release_manifest ||= Release.canonical(
+          agent: name,
+          provider: release_provider,
+          model: prompt_options&.dig(:model),
+          options: release_options,
+          actions: release_actions,
+          templates: release_templates,
+          tools: release_tools,
+          delegations: release_delegations
+        )
+      end
+
+      # A short, stable identifier for {#release_manifest}: the first twelve
+      # hex characters of its SHA-256.
+      #
+      # @return [String]
+      def release_digest
+        @release_digest ||= Digest::SHA256.hexdigest(JSON.generate(release_manifest))[0, 12]
+      end
+
+      # Forgets the memoized manifest and digest — for a host that edits
+      # templates at runtime, and for tests.
+      # @return [void]
+      def reset_release!
+        @release_manifest = nil
+        @release_digest = nil
+      end
+
+      private
+
+      def release_provider
+        provider = respond_to?(:prompt_provider) ? prompt_provider : nil
+        (provider || prompt_options&.dig(:service))&.to_s
+      end
+
+      # Generation options minus credentials and per-call state. Nested
+      # hashes are walked so a token under `options: { headers: … }` is
+      # dropped too.
+      def release_options
+        strip_secrets((prompt_options || {}).except(*EXCLUDED_OPTION_KEYS, :model, :service))
+      end
+
+      def strip_secrets(value)
+        case value
+        when Hash
+          value.each_with_object({}) do |(key, inner), kept|
+            next if EXCLUDED_OPTION_KEYS.include?(key.to_sym) || key.to_s.match?(SECRET_KEY_PATTERN)
+
+            kept[key] = strip_secrets(inner)
+          end
+        when Array then value.map { |inner| strip_secrets(inner) }
+        else value
+        end
+      end
+
+      # The public actions — the prompts a caller can invoke.
+      def release_actions
+        respond_to?(:action_methods) ? action_methods.to_a.sort : []
+      end
+
+      # Every template file under this agent's view prefixes, keyed by its
+      # path relative to the view root, with a digest of its contents. The
+      # prefixes mirror View#_prefixes without an action: `app/views/<agent>/`
+      # and `app/views/agents/<agent without suffix>/`.
+      def release_templates
+        return {} if anonymous? || !respond_to?(:view_paths)
+
+        base = name.underscore
+        prefixes = [ base, "agents/#{base.delete_suffix("_agent")}" ]
+        roots = Array(view_paths).map { |path| path.respond_to?(:to_path) ? path.to_path : path.to_s }
+
+        roots.each_with_object({}) do |root, templates|
+          prefixes.each do |prefix|
+            Dir.glob(File.join(root, prefix, "**", "*")).sort.each do |file|
+              next unless File.file?(file)
+
+              relative = file.delete_prefix("#{root}/")
+              templates[relative] = Digest::SHA256.hexdigest(File.binread(file))[0, 12]
+            end
+          end
+        end
+      end
+
+      # Tool definitions the class declares itself (a host convention such as
+      # schema-derived rosters), reduced to what identifies them.
+      def release_tools
+        return [] unless respond_to?(:tool_definitions)
+
+        Array(tool_definitions).map do |definition|
+          next definition.to_s unless definition.respond_to?(:to_h)
+
+          hash = definition.to_h
+          {
+            name: (hash[:name] || hash["name"]).to_s,
+            description: (hash[:description] || hash["description"]).to_s,
+            parameters: hash[:parameters] || hash["parameters"]
+          }
+        end.sort_by { |tool| tool.is_a?(Hash) ? tool[:name] : tool }
+      end
+
+      # The delegations this class declares, by tool name.
+      def release_delegations
+        return [] unless respond_to?(:delegations)
+
+        Array(delegations).map { |tool_name, _definition| tool_name.to_s }.sort
+      end
+    end
+  end
+end

data/lib/active_agent/evals/correlation.rb

@@ -0,0 +1,178 @@
+# frozen_string_literal: true
+
+require "securerandom"
+require "active_support/isolated_execution_state"
+
+module ActiveAgent
+  module Evals
+    # Used to tie the traces an evaluation produces back to the run and the
+    # result that caused them, so a report row links to the exact conversation
+    # behind it.
+    #
+    # A run mints a `run_id`, each evaluation mints a `result_id`, and both ride
+    # every trace opened inside them as `eval.`-prefixed attributes. The trace
+    # ids travel the other way: a replay's trace id lands on
+    # `result.replay.metadata["trace_id"]`, and every judge call made while
+    # scoring that result appends to its `"judge_trace_ids"`. The verdict — a
+    # judge call made outside any evaluation — appends to the run metadata
+    # instead, which is the same Hash a Report carries as its `metadata`.
+    #
+    # A tracer is `(name, action:, attributes:, on_trace:) { ... }`: it opens a
+    # trace named for the agent, and calls `on_trace` with something answering
+    # to `#trace_id` once the trace is known.
+    #
+    #   correlation = ActiveAgent::Evals::Correlation.new(
+    #     agent_name: "SupportAgent",
+    #     judge_name: "SupportAgentJudge",
+    #     tracer: ->(name, action:, attributes:, on_trace:, &block) {
+    #       MyTelemetry.with_agent(name, action: action, attributes: attributes,
+    #                              on_trace: on_trace, synchronous: true, &block)
+    #     }
+    #   )
+    #
+    #   correlation.with_run("suite" => "support") do |metadata|
+    #     Runner.new(
+    #       scenarios: scenarios, models: models, metadata: metadata,
+    #       replay: ->(scenario, spec) { correlation.replay { agent.run(scenario.prompt) } },
+    #       judge: Judge.new(label: "judge-model") { |instructions:, prompt:|
+    #         correlation.judge("score") { chat.with_instructions(instructions).ask(prompt).content }
+    #       },
+    #       around_evaluation: correlation
+    #     ).call
+    #   end
+    #
+    # Without a tracer the correlation still mints ids and merges metadata. The
+    # blocks run untraced.
+    class Correlation
+      STATE_KEY = :active_agent_evals_correlation
+
+      # DEFAULT_TRACE_KEYS names the correlation metadata that rides a trace as
+      # `eval.`-prefixed attributes. Anything else a caller puts in the run
+      # metadata (a tenant, a role) stays on the report but off the traces.
+      DEFAULT_TRACE_KEYS = %w[run_id result_id suite scenario_key model_label model provider].freeze
+
+      attr_reader :agent_name, :judge_name, :trace_keys
+
+      # @param agent_name [String] the trace name for the agent under evaluation
+      # @param judge_name [String] the trace name for judge traffic, kept distinct
+      #   so grading calls do not read as the agent's own traffic
+      # @param tracer [#call, nil] `(name, action:, attributes:, on_trace:) { ... }`;
+      #   nil runs every block untraced
+      # @param replay_action [String] the action name recorded for a replay trace
+      # @param trace_keys [Array<String>] which correlation keys become attributes
+      def initialize(agent_name:, judge_name: "#{agent_name}Judge", tracer: nil, replay_action: "eval",
+                     trace_keys: DEFAULT_TRACE_KEYS)
+        @agent_name = agent_name
+        @judge_name = judge_name
+        @tracer = tracer
+        @replay_action = replay_action
+        @trace_keys = trace_keys.map(&:to_s)
+      end
+
+      # Opens a run. Mints `run_id` unless `metadata` carries one, and yields the
+      # metadata hash the traces will be correlated against — pass that same hash
+      # to `Runner.new(metadata:)` so the Report carries the run's identity and
+      # collects the verdict's trace id.
+      #
+      # The hash yielded is the caller's own, mutated in place, so a run
+      # reopened around a later verdict accumulates onto the metadata a Report
+      # already carries.
+      #
+      # @param metadata [Hash] opaque run metadata. A non-Hash is coerced with
+      #   `#to_h`, and non-String keys are stringified.
+      # @yieldparam metadata [Hash]
+      def with_run(metadata = {})
+        run = metadata.is_a?(Hash) ? metadata : metadata.to_h
+        run.transform_keys!(&:to_s) unless run.keys.all?(String)
+        run["run_id"] ||= SecureRandom.uuid
+        with_context({ run: run, result: nil }) { yield run }
+      end
+
+      # Wraps one evaluation, in the shape `Runner.new(around_evaluation:)` calls:
+      # `(scenario, spec) { ... } → Result`. Mints a `result_id`, merges the
+      # correlation onto the Result's replay metadata, and returns the Result.
+      def around_evaluation(scenario, spec)
+        result_metadata = {
+          "run_id" => run_metadata["run_id"],
+          "result_id" => SecureRandom.uuid,
+          "scenario_key" => scenario.key,
+          "model_label" => spec.label,
+          "model" => spec.model,
+          "provider" => spec.provider
+        }.compact
+
+        with_context(run: run_metadata, result: result_metadata) do
+          yield.tap { |result| result.replay.metadata.merge!(result_metadata) }
+        end
+      end
+
+      # Delegates to `around_evaluation`, so the object satisfies
+      # `Runner.new(around_evaluation:)` directly.
+      def call(scenario, spec, &)
+        around_evaluation(scenario, spec, &)
+      end
+
+      # Traces one replay of the agent under evaluation. The trace id lands on
+      # the current result's metadata, so a report row links to the conversation.
+      def replay(action = @replay_action, &)
+        trace(@agent_name, action, judge: false, &)
+      end
+
+      # Traces one judge call. Appends to the current result's `judge_trace_ids`,
+      # or the run's when no evaluation is open (the verdict).
+      def judge(action = "score", &)
+        trace(@judge_name, action, judge: true, &)
+      end
+
+      # The correlation metadata in scope, or nil outside a run. A result's
+      # values win over the run's.
+      def current
+        context = ActiveSupport::IsolatedExecutionState[STATE_KEY]
+        return nil unless context
+
+        context.fetch(:run, {}).merge(context[:result] || {})
+      end
+
+      private
+
+      def run_metadata
+        context = ActiveSupport::IsolatedExecutionState[STATE_KEY]
+        context&.fetch(:run, nil) || {}
+      end
+
+      def with_context(context)
+        previous = ActiveSupport::IsolatedExecutionState[STATE_KEY]
+        ActiveSupport::IsolatedExecutionState[STATE_KEY] = context
+        yield
+      ensure
+        ActiveSupport::IsolatedExecutionState[STATE_KEY] = previous
+      end
+
+      def trace(name, action, judge:, &block)
+        return block.call unless @tracer
+
+        context = ActiveSupport::IsolatedExecutionState[STATE_KEY] || {}
+        correlation = context.fetch(:run, {}).merge(context[:result] || {})
+        attributes = correlation.slice(*@trace_keys).transform_keys { |key| "eval.#{key}" }
+        # A replay belongs to the evaluation that opened it and nowhere else, so
+        # it records no trace id when called outside one. A judge call outside an
+        # evaluation is the verdict, which belongs to the run.
+        target = judge ? (context[:result] || context[:run]) : context[:result]
+
+        @tracer.call(name, action: action, attributes: attributes, on_trace: recorder(target, judge: judge), &block)
+      end
+
+      def recorder(target, judge:)
+        lambda do |trace|
+          next unless target
+
+          if judge
+            (target["judge_trace_ids"] ||= []) << trace.trace_id
+          else
+            target["trace_id"] = trace.trace_id
+          end
+        end
+      end
+    end
+  end
+end

data/lib/active_agent/evals/judge.rb

@@ -10,6 +10,21 @@ module ActiveAgent
     #     RubyLLM.chat(model: "claude-opus-5").with_instructions(instructions).ask(prompt).content
     #   end
     #
+    # A judge serves three different calls, and a block that accepts `kind:` is
+    # told which one it is serving — `:score`, `:recommend` or `:verdict` — so a
+    # host can trace them apart, budget them apart, or score with a cheaper model
+    # than it writes the verdict with:
+    #
+    #   Judge.new(label: "claude-opus-5") do |instructions:, prompt:, kind:|
+    #     model = kind == :score ? "claude-haiku-4-5" : "claude-opus-5"
+    #     RubyLLM.chat(model: model).with_instructions(instructions).ask(prompt).content
+    #   end
+    #
+    # The keyword is passed only to a block that names it (or collects `**`), so
+    # a two-keyword block written before this is unaffected. Without it the only
+    # signal is the `instructions` string, which means matching on the gem's own
+    # prose — and a reworded constant then mislabels silently instead of failing.
+    #
     # Every method returns nil when the judge fails or answers unusably, so an
     # evaluation degrades to rule scoring rather than aborting.
     class Judge
@@ -23,6 +38,8 @@ module ActiveAgent
       # @param label [String] how reports name the judge (usually its model)
       # @yieldparam instructions [String] the system prompt
       # @yieldparam prompt [String] the user prompt
+      # @yieldparam kind [Symbol] which call this is — `:score`, `:recommend` or
+      #   `:verdict`. Passed only to a block that accepts it.
       # @yieldreturn [String] the completion text
       # How much of a scenario's notes the judge reads. Where a suite's notes
       # are its grading rubric, a "Must not…" clause tends to come last, and a
@@ -41,7 +58,7 @@ module ActiveAgent
         return nil if answer.blank?
 
         guidance = criterion.dig("config", "prompt").presence || criterion["key"].to_s.humanize
-        parse_score(ask(SCORE_INSTRUCTIONS, <<~PROMPT))
+        parse_score(ask(SCORE_INSTRUCTIONS, <<~PROMPT, :score))
           Criterion: #{guidance}
 
           The user asked:
@@ -64,7 +81,7 @@ module ActiveAgent
       def score_task(scenario:, answer:)
         return nil if answer.blank?
 
-        parse_score(ask(SCORE_INSTRUCTIONS, <<~PROMPT))
+        parse_score(ask(SCORE_INSTRUCTIONS, <<~PROMPT, :score))
           A user asked an assistant:
           ---
           #{scenario.prompt}
@@ -90,7 +107,7 @@ module ActiveAgent
           "- #{call['name']}#{' (errored)' if call['error']}: #{call['arguments'].to_json.truncate(200)}"
         end.join("\n")
 
-        parsed = parse_object(ask(RECOMMEND_INSTRUCTIONS, <<~PROMPT))
+        parsed = parse_object(ask(RECOMMEND_INSTRUCTIONS, <<~PROMPT, :recommend))
           An AI agent failed one evaluation scenario. Recommend the fix.
 
           Agent instructions:
@@ -146,7 +163,7 @@ module ActiveAgent
             "#{", faults: #{faults}" if faults.present?}"
         end
 
-        parsed = parse_object(ask(VERDICT_INSTRUCTIONS, <<~PROMPT))
+        parsed = parse_object(ask(VERDICT_INSTRUCTIONS, <<~PROMPT, :verdict))
           An AI agent ran the same scenarios under several models. Its goals:
           ---
           #{instructions.to_s.truncate(1_000).presence || '(no instructions configured)'}
@@ -180,13 +197,34 @@ module ActiveAgent
         end
       end
 
-      def ask(instructions, prompt)
-        @generate.call(instructions: instructions, prompt: prompt).to_s
+      def ask(instructions, prompt, kind)
+        @generate.call(**ask_arguments(instructions, prompt, kind)).to_s
       rescue StandardError => e
         warn_failure(e)
         nil
       end
 
+      # The block signature is public API, and every judge written before `kind:`
+      # existed takes exactly `instructions:` and `prompt:` — passing a third
+      # keyword to one of those raises ArgumentError, which `ask` would swallow
+      # as a judge failure, degrading the run to rule scoring. So the kind goes
+      # only to a block that asked for it.
+      def ask_arguments(instructions, prompt, kind)
+        arguments = { instructions: instructions, prompt: prompt }
+        arguments[:kind] = kind if generate_accepts_kind?
+        arguments
+      end
+
+      # True for a block naming `kind:` or collecting `**`. Memoized because the
+      # answer cannot change for a given judge and `ask` runs per scored result.
+      def generate_accepts_kind?
+        return @generate_accepts_kind if defined?(@generate_accepts_kind)
+
+        @generate_accepts_kind = @generate.parameters.any? do |type, name|
+          type == :keyrest || (name == :kind && (type == :key || type == :keyreq))
+        end
+      end
+
       def warn_failure(error)
         message = "[ActiveAgent::Evals] judge #{label} failed: #{error.class}: #{error.message}"
         if defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger

data/lib/active_agent/evals.rb

@@ -25,6 +25,7 @@ require_relative "evals/design_tokens"
 require_relative "evals/report_html"
 require_relative "evals/report"
 require_relative "evals/runner"
+require_relative "evals/correlation"
 require_relative "evals/publisher"
 
 # Scenario evaluations for agents that answer with tools.
@@ -40,7 +41,8 @@ require_relative "evals/publisher"
 # fell short and what would fix it (Diagnosis, refined by an optional Judge),
 # and rolling everything up per model (Report). Runner ties them together
 # around one callable you supply: given a scenario and a model, run the
-# agent and return a Replay.
+# agent and return a Replay. Correlation is optional plumbing on top: it
+# links the traces a run emits back to the result that caused them.
 #
 #   scenarios = ActiveAgent::Evals::ScenarioParser.scenarios(pasted_text)
 #   models    = ActiveAgent::Evals::ModelSpec.parse_all(%w[gpt-5-mini qwen3:8b], default_provider: "openai")

data/lib/active_agent/schema_tools.rb

@@ -68,6 +68,21 @@ module ActiveAgent
     # ask for 10_000; this is what stops that from becoming the prompt.
     MAX_LIMIT = 100
 
+    # Column types a range comparison is offered for. Strings and booleans
+    # are deliberately absent: a lexical `>` on a name column answers a
+    # question nobody asked.
+    RANGE_FILTERABLE_TYPES = %i[date datetime time integer float decimal].freeze
+
+    # The comparison operators a range filter may use, mapped to the Arel
+    # predicate that builds them. Names are the ones models reach for
+    # unprompted (`before`/`after` for dates, `lt`/`gte` for numbers), so a
+    # reasonable guess resolves instead of erroring.
+    RANGE_OPERATORS = {
+      "before" => :lt, "after" => :gt,
+      "lt" => :lt, "lte" => :lteq, "gt" => :gt, "gte" => :gteq,
+      "on_or_before" => :lteq, "on_or_after" => :gteq
+    }.freeze
+
     # Raised when a tool call names a column outside the declared allowlists,
     # or is otherwise outside the declared boundary.
     class UnpermittedAttribute < ArgumentError; end
@@ -300,8 +315,13 @@ module ActiveAgent
 
       # Validates and normalizes a filter hash against the allowlist.
       #
+      # A filter value is normally matched for equality. A Hash value instead
+      # declares a range — `{ "before" => "2026-01-01" }`, `{ "gte" => 10 }` —
+      # and may carry two bounds at once to express a window.
+      #
       # @api private
-      # @raise [UnpermittedAttribute] if any key is not declared filterable
+      # @raise [UnpermittedAttribute] if any key is not declared filterable,
+      #   or a range names an operator that does not exist
       def permitted_filters!(arguments)
         filters = arguments.each_with_object({}) do |(key, value), memo|
           next if value.nil?
@@ -318,6 +338,54 @@ module ActiveAgent
         filters
       end
 
+      # Splits filters into equality pairs and range predicates.
+      #
+      # Kept separate from {.permitted_filters!} because the two halves are
+      # applied differently: equality goes to `where(hash)`, ranges have to be
+      # built through Arel.
+      #
+      # @api private
+      # @return [Array(Hash, Array<Arel::Nodes::Node>)]
+      def partition_filters!(filters)
+        equality = {}
+        ranges   = []
+
+        filters.each do |column, value|
+          if value.is_a?(Hash)
+            ranges.concat(range_predicates!(column, value))
+          else
+            equality[column] = value
+          end
+        end
+
+        [ equality, ranges ]
+      end
+
+      # Builds Arel predicates for one column's range hash.
+      #
+      # Rails silently turns `where(col: { "before" => x })` into `col = NULL`,
+      # which matches nothing and reports zero rather than failing — the worst
+      # outcome for an agent, which reads it as a truthful empty answer. So an
+      # unknown operator is rejected loudly here instead.
+      #
+      # @api private
+      # @raise [UnpermittedAttribute] on an unknown operator
+      def range_predicates!(column, value)
+        arel = @model.arel_table[column]
+        type = @model.type_for_attribute(column)
+
+        value.map do |operator, operand|
+          predicate = RANGE_OPERATORS[operator.to_s]
+          unless predicate
+            raise UnpermittedAttribute,
+              "`#{operator}` is not a valid comparison for `#{column}`. " \
+              "Allowed comparisons: #{RANGE_OPERATORS.keys.join(", ")}"
+          end
+
+          arel.public_send(predicate, type.cast(operand))
+        end
+      end
+
       # Projects a record down to the declared return columns.
       #
       # The projection happens in SQL (+select+) as well as here, but the Ruby
@@ -386,7 +454,40 @@ module ActiveAgent
         )
         properties = schema[:schema][:properties]
 
-        filterable.index_with { |column| (properties[column] || { type: "string" }).deep_dup }
+        filterable.index_with do |column|
+          scalar = (properties[column] || { type: "string" }).deep_dup
+          range_filterable?(column) ? with_range_form(column, scalar) : scalar
+        end
+      end
+
+      # Dates, times and numbers are the columns a question like "overdue" or
+      # "more than 10" actually needs a comparison on.
+      def range_filterable?(column)
+        RANGE_FILTERABLE_TYPES.include?(@model.type_for_attribute(column).type)
+      end
+
+      # Offers a column as either a scalar (equality) or a range object.
+      #
+      # Without this the range form works but is undiscoverable: a model shown
+      # only `{type: "string", format: "date"}` has no way to know it may ask
+      # for `before`, and answers date questions with an equality match or no
+      # filter at all.
+      def with_range_form(column, scalar)
+        operand = scalar.slice(:type, :format)
+        description = scalar[:description]
+
+        {
+          description: [ description, "Accepts an exact value, or a range object such as " \
+            "{\"before\": ...} / {\"gte\": ...} (#{RANGE_OPERATORS.keys.join(", ")})." ].compact.join(" "),
+          anyOf: [
+            scalar.except(:description),
+            {
+              type: "object",
+              properties: RANGE_OPERATORS.keys.index_with { operand.dup },
+              additionalProperties: false
+            }
+          ]
+        }
       end
 
       def resource_name
@@ -415,10 +516,10 @@ module ActiveAgent
         )
 
         define_singleton_method(name) do |actor: nil, limit: nil, **arguments|
-          filters = permitted_filters!(arguments)
+          equality, ranges = partition_filters!(permitted_filters!(arguments))
           capped = normalize_limit(limit)
 
-          relation = relation_for(actor).where(filters)
+          relation = ranges.reduce(relation_for(actor).where(equality)) { |rel, p| rel.where(p) }
           # One extra row distinguishes "exactly at the limit" from "more than
           # the limit", without a second COUNT query.
           records = relation.limit(capped + 1).to_a
@@ -443,9 +544,10 @@ module ActiveAgent
         )
 
         define_singleton_method(name) do |actor: nil, **arguments|
-          filters = permitted_filters!(arguments)
+          equality, ranges = partition_filters!(permitted_filters!(arguments))
+          relation = ranges.reduce(relation_for(actor).where(equality)) { |rel, p| rel.where(p) }
 
-          { count: relation_for(actor).where(filters).count }
+          { count: relation.count }
         end
       end
 

data/lib/active_agent/telemetry/configuration.rb

@@ -24,6 +24,17 @@ module ActiveAgent
       # @return [Boolean] Whether to store traces in the app's own database
       attr_reader :local_storage
 
+      # The deploy every trace is stamped with (`service.version`). Unset, it
+      # is whatever ActiveAgent::Release.revision resolves — a git SHA from
+      # the conventional deploy variables — so traces from two deploys of
+      # the same service can be told apart without any host configuration.
+      attr_writer :service_version
+
+      def service_version
+        value = @service_version.respond_to?(:call) ? @service_version.call : @service_version
+        (value.presence || ActiveAgent::Release.revision)&.to_s
+      end
+
       def initialize
         super
         # The framework predates the shared gem and has always been opt-in;

data/lib/active_agent/telemetry/instrumentation.rb

@@ -58,6 +58,15 @@ module ActiveAgent
             span.set_attribute("agent.action", action_name.to_s)
             span.set_attribute("agent.provider", provider_name)
             span.set_attribute("agent.model", model_name)
+            # Which release of the agent ran: the digest of what the model was
+            # given (ActiveAgent::Release), so a dashboard can pin this trace
+            # to the version it cut on deploy.
+            if self.class.respond_to?(:release_digest)
+              span.set_attribute("agent.version", self.class.release_digest)
+              if (revision = ActiveAgent::Release.revision).present?
+                span.set_attribute("agent.revision", revision)
+              end
+            end
 
             # Add prompt span, carrying the prompt contents (instructions +
             # outbound messages) so dashboards can show what was sent.
@@ -92,7 +101,23 @@ module ActiveAgent
                     parameters: properties.is_a?(Hash) ? properties.keys : []
                   }.compact
                 }
-                prompt_span.set_attribute("prompt.input.tools", JSON.generate(roster)) if roster.any?
+                if roster.any?
+                  prompt_span.set_attribute("prompt.input.tools", JSON.generate(roster))
+                  # The roster is a reading aid: names, a truncated description
+                  # and parameter keys. What the model is sent is the full JSON
+                  # Schema — types, enums, nested objects, anyOf branches —
+                  # which for a twelve-tool agent is several times the roster,
+                  # so a context meter sizing tool pressure from the roster
+                  # understates it badly. The schemas come from the host and
+                  # need not be serializable; a size is worth less than the
+                  # generation it would otherwise take down.
+                  tools_size = begin
+                    JSON.generate(tools).length
+                  rescue StandardError
+                    nil
+                  end
+                  prompt_span.set_attribute("prompt.input.tools.tokens", (tools_size / 4.0).round) if tools_size
+                end
               end
 
               # prompt_options[:messages] holds the turns a caller passed

data/lib/active_agent/telemetry/tracer.rb

@@ -169,12 +169,15 @@ module ActiveAgent
 
       # Returns default attributes for all spans.
       def default_attributes
-        {
+        attributes = {
           "service.name" => configuration.resolved_service_name,
           "service.environment" => configuration.resolved_environment,
           "telemetry.sdk.name" => "activeagent",
           "telemetry.sdk.version" => ActiveAgent::VERSION
         }
+        version = configuration.respond_to?(:service_version) ? configuration.service_version : nil
+        attributes["service.version"] = version if version.present?
+        attributes
       end
     end
   end

data/lib/active_agent/telemetry.rb

@@ -78,9 +78,31 @@ module ActiveAgent
       #   end
       def configure
         yield configuration if block_given?
+        install_instrumentation! if enabled?
         configuration
       end
 
+      # Installs generation instrumentation on ActiveAgent::Base.
+      #
+      # The railtie also does this at boot, but only for configuration already
+      # loaded by then (activeagent.yml, config.active_agent.telemetry). A host
+      # app that enables telemetry from its own initializer runs *after*
+      # railties, so that check has already seen `enabled? == false` and
+      # skipped the install — leaving telemetry enabled but nothing
+      # instrumented, and so no traces despite a valid local_store. Calling it
+      # from {.configure} as well makes the install order-independent;
+      # `instrument_telemetry!` is idempotent, so the two paths cannot
+      # double-prepend.
+      #
+      # @api private
+      # @return [void]
+      def install_instrumentation!
+        return unless defined?(ActiveAgent::Base)
+
+        ActiveAgent::Base.include(Instrumentation)
+        ActiveAgent::Base.instrument_telemetry!
+      end
+
       # Resets the configuration to defaults.
       #
       # @return [Configuration] New default configuration

data/lib/active_agent/version.rb

@@ -1,3 +1,3 @@
 module ActiveAgent
-  VERSION = "1.6.0"
+  VERSION = "1.6.3"
 end

metadata

@@ -1,13 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: activeagent
 version: !ruby/object:Gem::Version
-  version: 1.6.0
+  version: 1.6.3
 platform: ruby
 authors:
 - Justin Bowen
+autorequire:
 bindir: bin
 cert_chain: []
-date: 1980-01-02 00:00:00.000000000 Z
+date: 2026-09-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -417,6 +418,7 @@ files:
 - lib/active_agent/concerns/preview.rb
 - lib/active_agent/concerns/provider.rb
 - lib/active_agent/concerns/queueing.rb
+- lib/active_agent/concerns/release.rb
 - lib/active_agent/concerns/rescue.rb
 - lib/active_agent/concerns/streaming.rb
 - lib/active_agent/concerns/tooling.rb
@@ -432,6 +434,7 @@ files:
 - lib/active_agent/delegation/schema.rb
 - lib/active_agent/deprecator.rb
 - lib/active_agent/evals.rb
+- lib/active_agent/evals/correlation.rb
 - lib/active_agent/evals/design_tokens.rb
 - lib/active_agent/evals/diagnosis.rb
 - lib/active_agent/evals/judge.rb
@@ -605,6 +608,7 @@ metadata:
   documentation_uri: https://docs.activeagents.ai
   source_code_uri: https://github.com/activeagents/activeagent
   rubygems_mfa_required: 'true'
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -619,7 +623,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.9
+rubygems_version: 3.5.22
+signing_key:
 specification_version: 4
 summary: Rails AI Agents Framework
 test_files: []