activeagent 1.9.1 → 1.9.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c47ac88022fb9c9fe8919efaaf8cfb6824dfa7ba102cfc31df2bf42d8f4b19ef
-  data.tar.gz: 9670044e2fd2fceae17068327a7f0b5e9695ae049f81cd2b00e7f56387c20286
+  metadata.gz: 2b52709687c94c3d6c989fc36bba7826b931a6d0aa6e193b1bd4482f916d945f
+  data.tar.gz: 1801338ff01afc21837db5382aee49eb4cde1f002b322b479cf2256c283e8668
 SHA512:
-  metadata.gz: a2ee2907d4449caef1fd7f22d0b5418dfb19aa0140aaa02b47d7a394671dc16653d315ee80bcd6699a9baa0085c6202726ccbbefe9514689d1b0c53b708482e5
-  data.tar.gz: f98937727b17855137cbf76434ebd5657dbd8d4c37a94119edafadd9da412e46ec44a068b4cbafbc19d04e2726dfa75e62f4eaad503e9db40709d9ca21aacc88
+  metadata.gz: 4d0299374c70e7e66b5edf3b141d171e79cb2f576ec32dcb6a2b4e11e9eb5bccaabbf2a5106fb5afd05f01ece8b6dc75e9a0cd956523c39ffdc4c3b5a17a0b22
+  data.tar.gz: b60f377af9288f08cd5166ba7d71edc1060ccddf23c1d2912ed4b9d105e75ce6a6325a1f0e855e2141ded3c2ebdffbb7cd0e2f37796e662cc37ef833054d21d1

data/CHANGELOG.md

@@ -5,7 +5,211 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [1.9.2] - 2026-10-20
+
+Releases `activeagent` and `actionagent` 1.9.2, with the matching browser
+sidecar: evaluations at scale (the scenario suite as a build page, model
+comparison with intervals and P95 latency, opt-in parallel scenario runs,
+the judge named after its model), provider streaming fixes (Anthropic
+citations, OpenAI Chat refusals and logprobs, Responses built-in tools),
+nested `has_many` schemas, and delegation contracts from `SchemaGenerator`
+classes.
+
+Upgrading: structured-output property names are now sent to the provider,
+and answered, as declared. A `response_format` schema with multiword
+property names (`first_name`) used to reach the provider camelized
+(`firstName`) while `required` stayed snake_case, which strict mode
+rejected; both are now as declared. `parsed_json` is unaffected; code that
+reads `message.content` itself, or matches on request payloads, sees
+`first_name` where it saw `firstName`. In a Rails app, `has_many` is
+available on every `ActiveModel::Model` class, since the railtie includes
+`SchemaGenerator` there, not only on classes that include it themselves.
+Scenario fan-out is off by default: `config.scenario_evaluation_fan_out =
+true` turns it on, and its jobs run on the `:agents` queue. No new
+migration; a host still on 1.9.0 needs 1.9.1's migration 027.
+
+### Added
+
+- **P50 and P95 latency per model in evaluation reports** (`activeagent`).
+  `Report#summary_by_model` adds `p50_duration_ms` and `p95_duration_ms`
+  beside `avg_duration_ms`, by nearest rank as the dashboard's Metrics view
+  computes them, and the Markdown and HTML reports show them next to the
+  mean, which hides the tail (#428).
+
+- **Cost per 1,000 scenario runs in the Markdown report** (`activeagent`).
+  The summary table gains `Cost / 1,000`, each model's mean cost per priced
+  replay on a common denominator, marked `~` when it is estimated or
+  covers only some replays (#428).
+
+- **`ActiveAgent::Evals::Comparison`: is the gap between models real?**
+  (`activeagent`). A 95% interval for each model's pass rate and for the
+  difference between every pair of models, from 5,000 seeded resamples of
+  whole scenarios (a paired bootstrap for the pairs, over the scenarios both
+  ran). A pair is decisive when its interval excludes zero, and
+  `#conclusion` says in one sentence which pairs are, or that the ranking
+  is not decisive evidence of superiority (#428).
+
+- **The model comparison in evaluation reports** (`activeagent`).
+  `Report#comparison` and `to_h["comparison"]` carry it for a report that
+  compares models, the Markdown report gains a Comparison section, and
+  `Report.new(comparison:)` renders one already recorded for the results
+  instead of resampling, as `verdict:` does (#428).
+
+- **A ranked interval plot in the HTML evaluation report** (`activeagent`).
+  Under the comparison table, each model's pass rate is a dot inside its
+  95% interval on a fixed 0 to 100% axis, best first, with the figures
+  beside it, the conclusion, and how the intervals were computed. Plain
+  CSS on the design tokens: no script, light and dark (#428).
+
+- **Each comparison run keeps its model comparison** (`actionagent`). A
+  scenario run that compares models records `Report#comparison` under
+  `scores["_comparison"]` when it completes; a published report's is
+  worked out from the stored results rather than taken from the payload;
+  and the run's report page renders the recorded intervals instead of
+  resampling on every view. A run recorded before gets one from its
+  results (#428).
+
+- **P50 and P95 latency in the dashboard's model comparison**
+  (`actionagent`). The comparison table gains a P50 / P95 column and each
+  model's scorecard a p50 / p95 figure, from the percentiles the run
+  recorded; a run recorded before shows neither (#428).
+
+- **The model comparison plotted in the dashboard** (`actionagent`). A
+  run that compared models shows, under its comparison table, each model's
+  pass rate inside its 95% interval, best first, with the run's
+  conclusion and how the intervals were computed, from the comparison the
+  run recorded (#428).
+
+- Parallel scenario evaluations in `actionagent`: with
+  `config.scenario_evaluation_fan_out = true`, a run enqueues one
+  `EvaluationReplayJob` per scenario × model, each stored replay is scored by
+  its own `EvaluationJudgeJob`, and the last result to be scored completes the
+  run with the scores a sequential run writes. Runs as wide as the host's
+  `:agents` queue. Off by default; browser and host-adapter runs stay
+  sequential (#422).
+- `ActiveAgent::Evals::Runner.new(refine_gate:)`, a callable asked before each
+  recommendation call so per-scenario runners share one run-wide judge limit.
+
+- **Nested arrays of objects in ActiveModel schemas** (`activeagent`).
+  `has_many :addresses, class_name: "Address"` on an ActiveModel class that
+  includes `ActiveAgent::SchemaGenerator` makes `to_json_schema` emit the
+  attribute as an array of `$ref`s to the item class's schema, which goes
+  under the root `$defs`. `strict:` and `additional_properties:` reach the
+  nested schema, a nested class's own definitions move up to the root, and
+  a class that nests itself raises `ArgumentError`. A length validation on
+  the array sets its item count (`min_items` / `max_items`) rather than a
+  string's length. `has_many` also declares the attribute with an
+  empty-array default, and is added only to a class without one, so
+  ActiveRecord's is untouched (#260).
+
+- **Evaluation runs record their judge** (`actionagent`), in
+  `scores["_judge"]`: the `EvaluationJudgeAgent` it runs as, its provider,
+  the model it asked for and where that setting came from (the evaluation,
+  the provider key, the host's config, or the provider's default), and the
+  models that answered, with call counts. The suite's Models panel shows
+  "judged by <model>" with an `auto` badge when no model is pinned, and
+  opens the judge's configuration, with a field that pins `judge_model` on
+  the evaluation through `PATCH /api/evaluations/:id` (#605).
+
+
+### Changed
+
+- **A scenario suite's page reads like a build page** (`actionagent`). The
+  page opened on a summary, every run, the costs, the models and every
+  scenario stacked down the page, so on a suite of 100 scenarios and 12 runs
+  the selected run's results started three screens down. It now opens on the
+  selected run: a run header with the run's passed count, models, agent and
+  judge spend, a to-fix count and a progress bar; beside it a rail of runs,
+  each with a pass bar per model, and a box of the run's scenarios that
+  scrolls on its own, one line per step (`[+]` passed, `[!]` failed, `[~]`
+  running, `[ ]` queued, `[?]` errored, `[-]` skipped) under a sticky header
+  and foldable group rows, with a step's detail opening in place. A toolbar
+  filters the steps to failed only, puts failed first, shows the tool cells
+  and follows the running step until the person scrolls. Models, what to fix
+  and the footer follow below; a failed run shows its error in the header
+  instead of models. Selecting a run keeps the page where it is, polling
+  backs off and stops in a hidden tab, and below 940px the rail becomes a
+  strip of the latest runs. The same suite is now 2,786px tall where it was
+  11,354px (#602).
+
+### Fixed
+
+- **`max_retries` sets the Anthropic `json_object` retries** (`activeagent`).
+  `AnthropicProvider` read the count from its constructor's arguments after
+  `BaseProvider` had already moved `max_retries` into the provider's options,
+  so an unparseable `json_object` answer was always retried twice, whatever
+  `max_retries` said, on Anthropic and on Bedrock, which inherits it. The
+  count now comes from the options: `max_retries: 0` sends the request once,
+  `max_retries: 5` up to six times, and the default is still two retries
+  (#531).
+- **A streamed Anthropic answer with citations no longer raises.** Claude
+  streams the citations of a text block as `citations_delta` deltas when it
+  cites a document or a web search result, and the Anthropic provider raised
+  `Unexpected delta type: citations_delta` on the first one. The delta is now
+  skipped like `input_json_delta`: the full block, citations included, arrives
+  at `content_block_stop`.
+- **A streamed OpenAI Chat refusal, or a stream with logprobs, no longer
+  raises.** The openai gem's chat stream emits `refusal.delta` and
+  `refusal.done` when the model refuses, and `logprobs.*` events when the
+  request sets `logprobs`, and the Chat provider failed on them with
+  `Unexpected Response Event Type`. They are now handled, and `refusal.done`
+  ends the generation, since a refused message never gets a `content.done`.
+- **A streamed Responses answer that used a built-in tool no longer raises.**
+  With `tools: [{ type: "web_search_preview" }]` (or file search, image
+  generation, code interpreter, remote MCP), the stream carries the tool's
+  call as an output item (`web_search_call`, ...), its progress events
+  (`response.web_search_call.searching`, ...), and
+  `response.output_text.annotation.added` for the citations, and the
+  Responses provider raised `Unexpected Item Type: web_search_call` on the
+  first of them. The tool calls are now kept like in a non-streamed response,
+  and their events are skipped.
+- **A streamed Responses answer cut short no longer raises.** When a response
+  hits `max_output_tokens`, the stream ends with `response.incomplete` instead
+  of `response.completed`, and the Responses provider raised `Unexpected
+  Response Chunk Type: response.incomplete`. It now ends the generation with
+  the text streamed so far, like `response.completed`.
+- **A streamed Responses refusal no longer raises.** The provider raised
+  `Unexpected Response Chunk Type: response.refusal.delta` when the model
+  refused; the refusal events are now skipped and the response completes.
+- **A streamed Responses generation reports its token usage.** The usage of a
+  streamed response is only on its final event, and the Responses provider
+  never read it, so `response.usage` was `nil`. It is now recorded from
+  `response.completed` or `response.incomplete`, as the Chat provider does
+  from its usage chunk.
+- **Delegation contracts built from a `SchemaGenerator` class**
+  (`activeagent`). `delegation ..., schema: TicketForm` and
+  `returns TicketForm` produced a contract with no properties, because
+  `to_json_schema` wraps the schema in `{ name:, schema:, strict: }` and
+  the contract read only a top-level `properties`. The wrapper is now
+  unwrapped, and `$defs` are kept so the `$ref`s that nested schemas use
+  still resolve. A contract built from a class is sent with the strict
+  setting the class declared: `Model.to_json_schema` is not strict unless
+  asked, and its `required` then names only the presence-validated
+  attributes, which OpenAI's strict mode refuses, so `returns Model` goes
+  out with `strict: false` and `returns Model.to_json_schema(strict: true)`
+  strict with every attribute required.
+
+- **Structured-output schema normalization preserves field names**
+  (`activeagent`). `prompt` converts JSON Schema keywords to camel case
+  without renaming properties or definitions, keeping multiword attributes
+  and `has_many` associations consistent with `required` and `$ref` in
+  strict schemas. Literal data in `default`, `const`, `enum` and `examples`
+  is preserved too (#260).
+
+  Property names now reach the provider as declared, so a `first_name`
+  attribute is sent as `first_name` rather than `firstName`, and the model
+  answers with that key. `parsed_json` underscores keys either way, so code
+  reading it is unaffected; code reading the raw `message.content`, or
+  matching on request payloads, sees snake_case where it saw camelCase.
+
+  A delegation's `returns` schema follows suit: its `required` entries are
+  no longer camelized, so a multiword required key such as `max_points`
+  matches its property in strict mode instead of being sent as
+  `maxPoints`.
+- **A judge with no `judge_model` is labelled with the model that
+  answered** (`actionagent`), not its provider ("judged by openrouter").
+  OpenRouter's default, `openrouter/auto`, routes each call, so a run judged
+  by more than one model is labelled "<provider> default" (#605).
 
 ## [1.9.1] - 2026-10-16
 

data/lib/active_agent/base.rb

@@ -397,13 +397,38 @@ module ActiveAgent
 
       if response_format[:type].to_sym == :json_schema
         response_format = nest_flat_json_schema(response_format)
-        response_format[:json_schema] = prompt_view_schema(response_format[:json_schema])
-        response_format[:json_schema].deep_transform_keys! { _1.to_s.camelize(:lower) }
+        response_format[:json_schema] = normalize_json_schema(prompt_view_schema(response_format[:json_schema]))
       end
 
       response_format
     end
 
+    # Convert Ruby-style schema keywords without changing property names,
+    # definition names, references or literal data (default, const, enum,
+    # examples). Only schema-valued keywords are traversed. Named maps keep
+    # their keys, so required entries and $ref paths still address them.
+    #
+    # @api private
+    def normalize_json_schema(schema)
+      return schema.map { |item| normalize_json_schema(item) } if schema.is_a?(Array)
+      return schema unless schema.is_a?(Hash)
+
+      schema.to_h do |key, value|
+        keyword = key.to_s.camelize(:lower)
+        normalized = case keyword
+        when "properties", "patternProperties", "$defs", "definitions", "dependentSchemas", "dependencies"
+          value.is_a?(Hash) ? value.transform_values { |definition| normalize_json_schema(definition) } : value
+        when "schema", "items", "prefixItems", "additionalItems", "contains", "additionalProperties",
+             "unevaluatedItems", "unevaluatedProperties", "propertyNames", "contentSchema",
+             "allOf", "anyOf", "oneOf", "not", "if", "then", "else"
+          normalize_json_schema(value)
+        else
+          value
+        end
+        [ keyword, normalized ]
+      end
+    end
+
     # Moves a json_schema given flat, with its name, schema and strict flag
     # beside +type+, under +json_schema+, so it is not mistaken for a
     # json_schema that names a view. Without a top-level schema, or with a

data/lib/active_agent/delegation/schema.rb

@@ -56,15 +56,25 @@ module ActiveAgent
           schema
         end
 
-        # @param hash [Hash] JSON Schema object
+        # @param hash [Hash] JSON Schema object, or the +{ name:, schema:, strict: }+
+        #   wrapper {ActiveAgent::SchemaGenerator} returns around one
         # @return [Schema]
         def from_hash(hash)
           hash = hash.deep_symbolize_keys
+          strict = nil
+          if !hash.key?(:properties) && hash[:schema].is_a?(Hash)
+            # SchemaGenerator's wrapper carries +strict: true+ only when the
+            # schema was built for strict mode; otherwise it is not strict.
+            strict = hash[:strict] == true
+            hash = hash[:schema]
+          end
 
           new.tap do |schema|
             (hash[:properties] || {}).each { |name, definition| schema.property(name, definition) }
             schema.required(*Array(hash[:required]))
             schema.additional_properties(hash.fetch(:additionalProperties, hash.fetch(:additional_properties, false)))
+            schema.definitions(hash[:$defs]) if hash[:$defs]
+            schema.strict(strict) unless strict.nil?
           end
         end
       end
@@ -73,6 +83,8 @@ module ActiveAgent
         @properties           = {}
         @required             = []
         @additional_properties = false
+        @definitions           = {}
+        @strict                = nil
       end
 
       # @return [Boolean] true when nothing has been declared
@@ -158,6 +170,29 @@ module ActiveAgent
         @additional_properties = value
       end
 
+      # Declares whether the schema was built for a provider's strict mode,
+      # as the +strict:+ of the wrapper {ActiveAgent::SchemaGenerator} returns
+      # says. Strict mode wants every property in +required+; a
+      # SchemaGenerator schema names only the attributes a presence
+      # validation covers unless it was built with +strict: true+, so a
+      # class is sent as it declared itself (see #to_response_format).
+      #
+      # @param value [Boolean]
+      # @return [void]
+      def strict(value = true)
+        @strict = value
+      end
+
+      # Declares the subschemas that properties reference with
+      # +"$ref": "#/$defs/name"+, such as the nested schemas
+      # {ActiveAgent::SchemaGenerator} emits for associations.
+      #
+      # @param definitions [Hash{Symbol => Hash}]
+      # @return [void]
+      def definitions(definitions)
+        @definitions = @definitions.merge(definitions.deep_symbolize_keys)
+      end
+
       # @return [Array<Symbol>] declared property names
       def keys
         @properties.keys
@@ -177,27 +212,30 @@ module ActiveAgent
           type: "object",
           properties: @properties.deep_dup,
           required: @required.map(&:to_s),
-          additionalProperties: @additional_properties
-        }
+          additionalProperties: @additional_properties,
+          "$defs": (@definitions.deep_dup if @definitions.any?)
+        }.compact
       end
       alias_method :to_h, :to_json_schema
 
       # JSON Schema for a +response_format+ payload.
       #
-      # ActiveAgent camelizes response-format schema *keys* on the way to the
-      # provider, but the +required+ array holds string *values* — so they are
-      # camelized here to keep the emitted schema internally consistent.
-      # {Assistant#parsed_json} underscores the keys again on the way back, so
-      # agent code only ever sees snake_case.
+      # ActiveAgent sends property names to the provider as declared, so the
+      # +required+ array names them as declared too.
+      # {Assistant#parsed_json} underscores the keys of the answer, so agent
+      # code only ever sees snake_case.
       #
       # @param name [String] schema name reported to the provider
-      # @param strict [Boolean]
+      # @param strict [Boolean, nil] nil follows the source: what a
+      #   SchemaGenerator class declared (not strict unless built with
+      #   +strict: true+, since its +required+ otherwise lists only the
+      #   presence-validated attributes, which strict mode refuses), else true
       # @return [Hash]
-      def to_response_format(name:, strict: true)
-        schema = to_json_schema
-        schema[:required] = schema[:required].map { |key| key.camelize(:lower) }
+      def to_response_format(name:, strict: nil)
+        strict = @strict if strict.nil?
+        strict = true if strict.nil?
 
-        { name: name, schema: schema, strict: strict }
+        { name: name, schema: to_json_schema, strict: strict }
       end
 
       # Validates a parsed payload against the declared required properties.

data/lib/active_agent/evals/comparison.rb

@@ -0,0 +1,169 @@
+# frozen_string_literal: true
+
+module ActiveAgent
+  module Evals
+    # Whether the gap between the models of a run is larger than the noise of
+    # its scenario sample: each model's pass rate with a confidence interval,
+    # and an interval for the difference between every pair of models.
+    #
+    # The scenario is the unit. A model's pass rate is the mean over its
+    # scenarios of the share of its results there that passed (1.0 or 0.0
+    # for one result per scenario), and the intervals come from resampling
+    # whole scenarios with replacement, +resamples+ times: the percentile
+    # bootstrap. A pair is compared on the scenarios both models ran, each
+    # resample drawing the same scenarios for both (a paired bootstrap of
+    # the per-scenario differences). A pair is decisive when its interval
+    # excludes zero.
+    #
+    # The resampling is seeded, so the same results always give the same
+    # intervals and a report renders the same page each time.
+    #
+    #   comparison = Comparison.new(results: report.results, models: report.models)
+    #   comparison.to_h["pairs"].first
+    #   # => { "models" => ["gpt-5-mini", "qwen3:8b"], "scenarios" => 42, "difference" => 7.1,
+    #   #      "interval" => [0.0, 16.7], "decisive" => false }
+    #   comparison.conclusion
+    #   # => "The pairwise difference interval includes zero: the ranking is not decisive evidence of superiority."
+    class Comparison
+      RESAMPLES = 5_000
+      CONFIDENCE = 0.95
+      SEED = 0
+      # Fewer scenarios than this give no interval: resampling one scenario
+      # only ever draws that scenario.
+      MINIMUM_SCENARIOS = 2
+
+      attr_reader :resamples, :confidence, :seed
+
+      # @param results [Array<Result>]
+      # @param models [Array<ModelSpec>] in the run's order, which breaks ties in the ranking
+      # @param resamples [Integer]
+      # @param confidence [Float] 0..1
+      # @param seed [Integer]
+      def initialize(results:, models:, resamples: RESAMPLES, confidence: CONFIDENCE, seed: SEED)
+        @labels = models.map(&:label)
+        @resamples = resamples
+        @confidence = confidence
+        @seed = seed
+        @outcomes = @labels.to_h do |label|
+          by_scenario = results.select { |result| result.label == label }.group_by { |result| result.scenario.key }
+          [ label, by_scenario.transform_values { |cohort| cohort.count(&:passed?).fdiv(cohort.size) } ]
+        end
+      end
+
+      # Per model label, in the run's order: `{ "scenarios", "pass_rate",
+      # "interval" }`, the pass rate and its interval in percent. The interval
+      # is nil below MINIMUM_SCENARIOS, and the pass rate nil for none.
+      def models
+        @models ||= @labels.to_h do |label|
+          values = @outcomes[label].values
+          [ label, {
+            "scenarios" => values.size,
+            "pass_rate" => percent(mean(values)),
+            "interval" => interval(values)
+          } ]
+        end
+      end
+
+      # The labels by pass rate, best first; ties keep the run's order.
+      def ranking
+        @ranking ||= @labels.sort_by.with_index { |label, index| [ -(models[label]["pass_rate"] || -1), index ] }
+      end
+
+      # One entry per pair of models, the better ranked first: `{ "models" =>
+      # [a, b], "scenarios", "difference", "interval", "decisive" }`, where
+      # "difference" is a's pass rate minus b's over the scenarios both ran,
+      # in percentage points.
+      def pairs
+        @pairs ||= ranking.combination(2).map do |a, b|
+          shared = @outcomes[a].keys & @outcomes[b].keys
+          differences = shared.map { |key| @outcomes[a][key] - @outcomes[b][key] }
+          range = interval(differences)
+
+          {
+            "models" => [ a, b ],
+            "scenarios" => shared.size,
+            "difference" => percent(mean(differences)),
+            "interval" => range,
+            "decisive" => !range.nil? && (range.first.positive? || range.last.negative?)
+          }
+        end
+      end
+
+      # The honest read of the ranking, in a sentence: which pairs differ by
+      # more than the noise, or that none does. nil for a single model.
+      def conclusion
+        return nil if pairs.empty?
+
+        compared, too_few = pairs.partition { |pair| pair["interval"] }
+        decisive, inconclusive = compared.partition { |pair| pair["decisive"] }
+        sentences =
+          if compared.empty?
+            [ "The models share too few scenarios to compare." ]
+          elsif decisive.empty?
+            [ "#{all_include_zero(compared.size)}: the ranking is not decisive evidence of superiority." ]
+          else
+            decisive.map { |pair| decisive_sentence(pair) } + (inconclusive.any? ? [ others_include_zero(inconclusive.size) ] : [])
+          end
+        sentences << "#{plural(too_few.size, 'pair')} share#{'s' if too_few.one?} too few scenarios to compare." if compared.any? && too_few.any?
+        sentences.join(" ")
+      end
+
+      # How the intervals were computed, for a caption.
+      def method_summary
+        { "statistic" => "pass rate", "interval" => "percentile bootstrap", "unit" => "scenario",
+          "resamples" => @resamples, "confidence" => @confidence, "seed" => @seed }
+      end
+
+      def to_h
+        { "method" => method_summary, "models" => models, "ranking" => ranking, "pairs" => pairs, "conclusion" => conclusion }
+      end
+
+      private
+
+      # The bootstrap interval of the mean of +values+, in percent, or nil
+      # below MINIMUM_SCENARIOS.
+      def interval(values)
+        return nil if values.size < MINIMUM_SCENARIOS
+
+        random = Random.new(@seed)
+        size = values.size
+        means = Array.new(@resamples) do
+          sum = 0.0
+          size.times { sum += values[random.rand(size)] }
+          sum / size
+        end
+        # A rational tail, so 95% leaves exactly 2.5: in floating point it
+        # is 2.5000000000000022, whose nearest rank is one too high.
+        tail = (1 - @confidence.rationalize) / 2 * 100
+        [ percent(Statistics.percentile(means, tail)), percent(Statistics.percentile(means, 100 - tail)) ]
+      end
+
+      def mean(values)
+        values.sum.fdiv(values.size) if values.any?
+      end
+
+      def percent(fraction)
+        (fraction * 100).round(1) if fraction
+      end
+
+      def all_include_zero(count)
+        count == 1 ? "The pairwise difference interval includes zero" : "All #{count} pairwise difference intervals include zero"
+      end
+
+      def others_include_zero(count)
+        count == 1 ? "The other pairwise interval includes zero." : "The other #{count} pairwise intervals include zero."
+      end
+
+      def decisive_sentence(pair)
+        better, worse = pair["difference"].positive? ? pair["models"] : pair["models"].reverse
+        low, high = pair["interval"].map(&:abs).sort
+        "#{better} passes more scenarios than #{worse}, by #{pair['difference'].abs} points " \
+          "(#{(@confidence * 100).round}% CI #{low} to #{high})."
+      end
+
+      def plural(count, word)
+        "#{count} #{word}#{'s' unless count == 1}"
+      end
+    end
+  end
+end

data/lib/active_agent/evals/report.rb

@@ -37,9 +37,12 @@ module ActiveAgent
       # @param release [Hash, nil] the release of the agent under evaluation,
       #   `{ "digest", "revision", "label" }`, so the report names the code
       #   it scored: in `to_h["release"]` and as a header chip.
+      # @param comparison [Hash, nil] a comparison already recorded for these
+      #   results (Comparison#to_h). A report rebuilt from a persisted run
+      #   renders the intervals that run recorded instead of resampling them.
       def initialize(results:, models:, judge: nil, instructions: nil, threshold: PASS_THRESHOLD, metadata: {},
                      tool_resolver: nil, agent_name: nil, links: {}, verdict: nil, judge_label: nil,
-                     judge_usage: nil, release: nil)
+                     judge_usage: nil, release: nil, comparison: nil)
         @results = results
         @models = models
         @judge = judge
@@ -53,6 +56,7 @@ module ActiveAgent
         @recorded_verdict = verdict.is_a?(Hash) ? verdict.to_h.stringify_keys.presence : nil
         @run_judge_usage = judge_usage.is_a?(Hash) ? judge_usage.to_h.stringify_keys.presence : nil
         @release = release.is_a?(Hash) ? release.to_h.stringify_keys.compact.presence : nil
+        @recorded_comparison = comparison.is_a?(Hash) ? comparison.to_h.deep_stringify_keys.presence : nil
       end
 
       # The release of the agent this report scored, `{ "digest",
@@ -64,7 +68,8 @@ module ActiveAgent
       end
 
       # Per model, keyed by label: scenario count, passes, errors, pass rate,
-      # mean score, mean latency, tokens, cost and fault counts. "cost" sums
+      # mean score, latency (mean, and the P50 and P95 by nearest rank,
+      # Statistics.percentile), tokens, cost and fault counts. "cost" sums
       # the replays that carried a cost — "priced" of the "scenarios", of
       # which "reported" came from the caller and "estimated" from tokens ×
       # a model rate (Result#cost_source) — and is nil when none did.
@@ -90,6 +95,8 @@ module ActiveAgent
             "avg_score" => scored.any? ? (scored.sum / scored.size).round(3) : nil,
             "avg_task_completion" => task_scores.any? ? (task_scores.sum / task_scores.size).round(3) : nil,
             "avg_duration_ms" => durations.any? ? (durations.sum.to_f / durations.size).round : nil,
+            "p50_duration_ms" => Statistics.percentile(durations, 50)&.round,
+            "p95_duration_ms" => Statistics.percentile(durations, 95)&.round,
             "input_tokens" => cohort.sum { |result| result.replay.input_tokens.to_i },
             "output_tokens" => cohort.sum { |result| result.replay.output_tokens.to_i },
             "cost" => costs.any? ? costs.sum.to_f.round(6) : nil,
@@ -267,14 +274,27 @@ module ActiveAgent
         verdict&.dig("winner")
       end
 
-      # Adds "judge_usage" and "release" only when there is one to add, so
-      # a report built the way it always was serializes exactly as before.
+      # Whether the gap between the models is real (Comparison#to_h): each
+      # model's pass rate with its interval, every pair's difference with
+      # its interval and whether it is decisive, and the conclusion. The
+      # comparison recorded for these results when one was handed in. nil
+      # for a single model.
+      def comparison
+        return nil unless comparing?
+
+        @comparison ||= @recorded_comparison || Comparison.new(results: @results, models: @models).to_h
+      end
+
+      # Adds "judge_usage" and "release" only when there is one to add, and
+      # "comparison" only when comparing models, so a single-model report
+      # built the way it always was serializes exactly as before.
       def to_h
         {
           "models" => summary_by_model,
           "criteria" => criterion_scores,
           "recommendations" => recommendations,
           "verdict" => verdict,
+          "comparison" => comparison,
           "judge" => @judge_label || @judge&.label,
           "judge_usage" => judge_usage,
           "release" => release,
@@ -298,6 +318,7 @@ module ActiveAgent
         lines.concat(total_lines)
         lines << "**Best model: #{winner}**" if winner
         lines << ""
+        lines.concat(comparison_lines)
         lines.concat(matrix_table)
         lines.concat(detail_lines)
         lines.concat(recommendation_lines)
@@ -525,22 +546,27 @@ module ActiveAgent
 
       # --- Markdown ----------------------------------------------------------
 
-      # The per-model table. A Judge column joins it when a judge spent
-      # anything, so the agent's cost and the evaluation's stay two numbers.
+      # The per-model table. Cost / 1,000 puts every model on one
+      # denominator however many of its scenarios ran. A Judge column joins
+      # it when a judge spent anything, so the agent's cost and the
+      # evaluation's stay two numbers.
       def summary_table
         judged = judge_usage.present?
-        columns = [ "Model", "Passed", "Mean score", "Mean latency", "Tokens in/out", "Cost", ("Judge" if judged), "Faults" ].compact
+        columns = [ "Model", "Passed", "Mean score", "Mean latency", "Median / P95", "Tokens in/out", "Cost", "Cost / 1,000", ("Judge" if judged), "Faults" ].compact
         header = [ "| #{columns.join(' | ')} |", "|#{columns.map { '---' }.join('|')}|" ]
         rows = summary_by_model.map do |label, stats|
           faults = stats["faults"].map { |fault, count| "#{fault.tr('_', ' ')} ×#{count}" }.join(", ")
           latency = stats["avg_duration_ms"] ? "#{stats['avg_duration_ms']} ms" : "—"
+          percentiles = stats["p50_duration_ms"] ? "#{stats['p50_duration_ms']} / #{stats['p95_duration_ms']} ms" : "—"
           cells = [
             "`#{label}`",
             Format.passes(stats["passed"], stats["scenarios"], style: :markdown),
             Format.score(stats["avg_score"]),
             latency,
+            percentiles,
             "#{stats['input_tokens']}/#{stats['output_tokens']}",
             Format.money(stats["cost"], estimated: estimated_cost?(stats)),
+            per_thousand_cell(stats),
             (judge_cell(stats) if judged),
             faults.presence || "—"
           ].compact
@@ -549,6 +575,14 @@ module ActiveAgent
         header + rows
       end
 
+      # "$3.6300": what 1,000 scenario runs cost at the model's mean cost
+      # per priced replay, marked "~" when that mean is estimated or covers
+      # only some of the replays. "—" when none was priced.
+      def per_thousand_cell(stats)
+        per = cost_per_priced(stats)
+        per ? Format.money(per * 1_000, estimated: estimated_cost?(stats)) : "—"
+      end
+
       # "~$0.0030 (3 calls)" for a model's judge spend, "—" when the judge
       # was not asked about its results.
       def judge_cell(stats)
@@ -572,6 +606,38 @@ module ActiveAgent
         lines << ""
       end
 
+      # Each model's pass rate with its interval, best first, then every
+      # pair's difference with its interval, the conclusion, and how the
+      # intervals were computed. Nothing for a single model.
+      def comparison_lines
+        return [] unless comparison
+
+        method = comparison["method"]
+        confidence = "#{(method['confidence'].to_f * 100).round}% CI"
+        lines = [ "## Comparison", "", "| Model | Pass rate | #{confidence} | Scenarios |", "|---|---|---|---|" ]
+        comparison["ranking"].each do |label|
+          stats = comparison["models"][label]
+          lines << "| `#{label}` | #{rate_text(stats['pass_rate'])} | #{interval_text(stats['interval'])} | #{stats['scenarios']} |"
+        end
+        lines += [ "", "| Pair | Difference | #{confidence} | Shared scenarios | Decisive |", "|---|---|---|---|---|" ]
+        comparison["pairs"].each do |pair|
+          a, b = pair["models"]
+          difference = pair["difference"] ? format("%+.1f points", pair["difference"]) : "—"
+          lines << "| `#{a}` vs `#{b}` | #{difference} | #{interval_text(pair['interval'])} | #{pair['scenarios']} | #{pair['decisive'] ? 'yes' : 'no'} |"
+        end
+        lines += [ "", comparison["conclusion"].to_s, "",
+                   "_#{confidence} from #{Format.delimited(method['resamples'])} resamples of whole scenarios (seed #{method['seed']})._", "" ]
+      end
+
+      def rate_text(rate)
+        rate ? "#{rate}%" : "—"
+      end
+
+      # "73.8 to 95.2", or "—" when there were too few scenarios.
+      def interval_text(interval)
+        interval ? "#{interval.first} to #{interval.last}" : "—"
+      end
+
       # The scenario matrix, with a trailing Cost column: what the scenario
       # cost across every model, and in brackets what the judge spent on it.
       def matrix_table

data/lib/active_agent/evals/report_html.rb

@@ -151,6 +151,14 @@ module ActiveAgent
         "#{format("%.#{n >= 10_000 ? 1 : 2}f", n / 1_000).sub(/\.?0+\z/, '')}s"
       end
 
+      # "4.02s / 22.2s": a model's P50 and P95 latency; "—" when no replay
+      # was timed.
+      def fmt_percentiles(stats)
+        return "—" if stats["p50_duration_ms"].nil?
+
+        "#{fmt_ms(stats['p50_duration_ms'])} / #{fmt_ms(stats['p95_duration_ms'])}"
+      end
+
       # Money reads "~$0.0243" when any part of it was estimated from tokens
       # × a model rate rather than reported (Format.money); the footer's
       # legend explains the mark once.
@@ -314,6 +322,7 @@ module ActiveAgent
           <div class="panel">
           <div class="panel-head"><span class="micro">Models</span><span class="right">judged by #{h(judged_by)}</span></div>
           #{html_comparison_table if comparing?}
+          #{html_intervals}
           #{blocks.join}
           #{verdict_row}
           </div>
@@ -335,12 +344,49 @@ module ActiveAgent
 
         <<~TABLE
           <div class="compare"><table>
-          <thead><tr><th>Model</th><th class="num">Passed</th><th class="num">Mean score</th><th class="num">Avg latency</th><th class="num" title="Average input + output tokens per scenario">Avg tokens</th><th class="num" title="Cohort spend, and per scenario">Cost</th>#{judge_head}<th class="fault">Typical fault</th></tr></thead>
+          <thead><tr><th>Model</th><th class="num">Passed</th><th class="num">Mean score</th><th class="num">Avg latency</th><th class="num" title="Median and 95th percentile latency, nearest rank">P50 / P95</th><th class="num" title="Average input + output tokens per scenario">Avg tokens</th><th class="num" title="Cohort spend, and per scenario">Cost</th>#{judge_head}<th class="fault">Typical fault</th></tr></thead>
           <tbody>#{rows.map { |label, stats| html_comparison_row(label, stats) }.join}</tbody>
           </table></div>
         TABLE
       end
 
+      # The ranked interval plot under the comparison table (Report#comparison):
+      # one row per model, best first, its pass rate a dot inside its interval
+      # on a fixed 0-100% axis, so a narrow gap is not magnified, with the
+      # figures written out beside it; then the conclusion and how the
+      # intervals were computed. Nothing for a single model. The rows and the
+      # axis share one grid (.iv-row is display: contents), so every track
+      # spans the same width and a tick sits under the dots it names.
+      def html_intervals
+        return "" unless comparison
+
+        method = comparison["method"]
+        confidence = "#{(method['confidence'].to_f * 100).round}% CI"
+        rows = comparison["ranking"].map { |label| html_interval_row(label, comparison["models"][label] || {}) }
+        ticks = [ 0, 25, 50, 75, 100 ].map { |tick| %(<span class="iv-tick" style="left:#{tick}%">#{tick}%</span>) }
+
+        <<~PLOT
+          <div class="intervals">
+          <div class="micro sm">Pass rate · #{h(confidence)}</div>
+          #{rows.join}
+          <div class="iv-row iv-axis" aria-hidden="true"><span></span><span class="iv-track">#{ticks.join}</span><span></span></div>
+          <p class="iv-conclusion">#{h(comparison['conclusion'])}</p>
+          <p class="iv-caption">#{h(confidence)} from #{h(Format.delimited(method['resamples']))} resamples of whole scenarios</p>
+          </div>
+        PLOT
+      end
+
+      def html_interval_row(label, stats)
+        rate = stats["pass_rate"]
+        low, high = stats["interval"]
+        range = high ? %(<span class="iv-range" style="left:#{low}%;width:#{(high - low).round(1)}%"></span>) : ""
+        point = rate ? %(<span class="iv-point" style="left:#{rate}%"></span>) : ""
+        figures = "<b>#{rate ? "#{rate}%" : '—'}</b> #{high ? "#{low} to #{high}" : 'no interval'} · #{h(plural(stats['scenarios'].to_i, 'scenario'))}"
+
+        %(<div class="iv-row"><span class="iv-name">#{h(short_name(model_by_label(label)))}</span>) +
+          %(<span class="iv-track">#{range}#{point}</span><span class="iv-figures">#{figures}</span></div>\n)
+      end
+
       def html_comparison_row(label, stats)
         short, provider = split_label(model_by_label(label))
         total = stats["scenarios"].to_i
@@ -362,6 +408,7 @@ module ActiveAgent
           <td class="num ratio tone-#{tone_for(ratio)}">#{h(fmt_passes(stats['passed'], total))}</td>
           <td class="num">#{h(fmt_score(stats['avg_score']))}</td>
           <td class="num">#{h(fmt_ms(stats['avg_duration_ms']))}</td>
+          <td class="num">#{h(fmt_percentiles(stats))}</td>
           <td class="num"#{tokens_title}>#{tokens_cell}</td>
           <td class="num"#{cost_title_attr(results_for(label), estimated: estimated)}>#{cost_cell}</td>
           #{judge_cell}<td class="fault">#{typical_fault_text(label, stats)}</td>
@@ -417,7 +464,7 @@ module ActiveAgent
         <<~BLOCK
           <div class="model">
           <div class="line"><span class="name">#{h(short)}</span><span class="provider">#{h(provider)}</span>#{pick}<span class="pass"><span class="bar bar-#{tone}"><span style="width:#{(ratio * 100).round}%"></span></span><span class="ratio tone-#{tone}">#{h(fmt_passes(stats['passed'], total))}</span></span></div>
-          <div class="stats-line"><span>score <b>#{h(fmt_score(stats['avg_score']))}</b></span><span>latency <b>#{h(fmt_ms(stats['avg_duration_ms']))}</b></span><span class="tok"><span class="in">in</span> #{h(fmt_k(stats['input_tokens']))} · <span class="out">out</span> #{h(fmt_k(stats['output_tokens']))}</span><span#{cost_title_attr(results_for(label), estimated: estimated)}>cost <b>#{h(fmt_cost(stats['cost'], estimated: estimated))}</b></span>#{judge_line}</div>
+          <div class="stats-line"><span>score <b>#{h(fmt_score(stats['avg_score']))}</b></span><span>latency <b>#{h(fmt_ms(stats['avg_duration_ms']))}</b></span><span>p50 / p95 <b>#{h(fmt_percentiles(stats))}</b></span><span class="tok"><span class="in">in</span> #{h(fmt_k(stats['input_tokens']))} · <span class="out">out</span> #{h(fmt_k(stats['output_tokens']))}</span><span#{cost_title_attr(results_for(label), estimated: estimated)}>cost <b>#{h(fmt_cost(stats['cost'], estimated: estimated))}</b></span>#{judge_line}</div>
           <div class="faults">#{faults_html}</div>
           </div>
         BLOCK
@@ -830,6 +877,21 @@ module ActiveAgent
         .pick-model input { position: absolute; opacity: 0; width: 0; height: 0; }
         .pick-model:has(input:checked) { border-color: var(--color-accent-ui); background: var(--color-accent-ui-tint); color: var(--color-accent-ui); }
         .clean { font-family: var(--font-mono); font-size: 11px; color: var(--color-success-text); }
+        .intervals { padding: 10px 12px; border-top: 1px solid var(--color-border-light); display: grid; grid-template-columns: minmax(120px, 200px) minmax(160px, 1fr) minmax(190px, auto); align-items: center; gap: 6px 12px; }
+        .intervals > .micro, .iv-conclusion, .iv-caption { grid-column: 1 / -1; }
+        .iv-row { display: contents; }
+        .iv-name { font-family: var(--font-mono); font-size: 12px; font-weight: 600; color: var(--color-text-primary); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+        .iv-track { position: relative; height: 14px; }
+        .iv-track::before { content: ""; position: absolute; left: 0; right: 0; top: 6px; height: 2px; background: var(--color-muted); }
+        .iv-range { position: absolute; top: 3px; height: 8px; box-sizing: border-box; border: 1px solid var(--color-info); border-radius: 999px; background: var(--color-info-soft); }
+        .iv-point { position: absolute; top: 2px; width: 10px; height: 10px; margin-left: -5px; border-radius: 50%; background: var(--color-info); }
+        .iv-figures { font-family: var(--font-mono); font-size: 11px; color: var(--color-text-secondary); white-space: nowrap; }
+        .iv-figures b { color: var(--color-text-primary); }
+        .iv-axis .iv-track::before { display: none; }
+        .iv-tick { position: absolute; top: 0; transform: translateX(-50%); font-family: var(--font-mono); font-size: 10px; color: var(--color-text-muted); }
+        .iv-conclusion { margin: 4px 0 0; font-size: 12px; line-height: 18px; color: var(--color-text-cell); }
+        .iv-caption { margin: 0; font-family: var(--font-mono); font-size: 10px; color: var(--color-text-muted); }
+        @media (max-width: 640px) { .intervals { grid-template-columns: minmax(0, 1fr); row-gap: 4px; } .iv-row + .iv-row .iv-name { margin-top: 6px; } .iv-axis { display: none; } }
         .verdict { padding: 10px 12px; border-top: 1px solid var(--color-border-light); font-size: 12px; line-height: 18px; color: var(--color-text-cell); }
         .verdict .micro { margin-right: 8px; }
         .fixes { display: grid; grid-template-columns: repeat(auto-fit, minmax(320px, 1fr)); gap: 12px; }

data/lib/active_agent/evals/runner.rb

@@ -54,10 +54,14 @@ module ActiveAgent
       # @param release [Hash, nil] the release of the agent under evaluation,
       #   `{ "digest", "revision", "label" }`, carried onto the Report so a
       #   published run names the code it scored (see Report.new)
+      # @param refine_gate [#call, nil] asked before each recommendation call,
+      #   after `judge_limit`; false skips it. A caller that scores each
+      #   scenario in its own Runner (a background job per scenario) shares one
+      #   run-wide limit this way, since `judge_limit` counts per instance.
       def initialize(scenarios:, models:, replay:, criteria: [], judge: nil, judge_task: true, available_tools: {},
                      instructions: nil, agent_name: "The agent", threshold: PASS_THRESHOLD,
                      refine_faults: DEFAULT_REFINE_FAULTS, judge_limit: DEFAULT_JUDGE_LIMIT, on_result: nil,
-                     around_evaluation: nil, require_judge_scores: false, metadata: {}, release: nil)
+                     around_evaluation: nil, require_judge_scores: false, metadata: {}, release: nil, refine_gate: nil)
         @scenarios = scenarios
         @models = models
         @replay = replay
@@ -75,6 +79,7 @@ module ActiveAgent
         @require_judge_scores = require_judge_scores
         @metadata = metadata
         @release = release
+        @refine_gate = refine_gate
         @judge_calls = 0
         @scorer = Scorer.new(criteria: criteria, judge: judge)
       end
@@ -187,6 +192,7 @@ module ActiveAgent
       def refine!(diagnosis, scenario, replay, result)
         return unless @judge && @refine_faults.include?(result.fault)
         return if @judge_calls >= @judge_limit
+        return if @refine_gate && !@refine_gate.call
 
         @judge_calls += 1
         refined = @judge.recommend(scenario: scenario, replay: replay, diagnosis: result,

data/lib/active_agent/evals/statistics.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module ActiveAgent
+  module Evals
+    # The summary statistics a report computes over its results.
+    module Statistics
+      module_function
+
+      # Nearest-rank percentile: the value at rank ceil(pct / 100 × n) of the
+      # values sorted ascending, so the result is always one of the values. The
+      # dashboard's Metrics view uses the same rule (ActionAgent::MetricsReport),
+      # so a model's P95 reads the same in both. nil for no values.
+      #
+      # @param values [Array<Numeric>] in any order; nils are ignored
+      # @param pct [Numeric] 0..100
+      # @return [Numeric, nil]
+      def percentile(values, pct)
+        sorted = values.compact.sort
+        return nil if sorted.empty?
+
+        rank = (sorted.size * pct / 100.0).ceil
+        sorted[rank.clamp(1, sorted.size) - 1]
+      end
+    end
+  end
+end

data/lib/active_agent/evals.rb

@@ -19,6 +19,8 @@ require_relative "evals/catalog"
 require_relative "evals/model_spec"
 require_relative "evals/replay"
 require_relative "evals/format"
+require_relative "evals/statistics"
+require_relative "evals/comparison"
 require_relative "evals/scorer"
 require_relative "evals/diagnosis"
 require_relative "evals/judge"

data/lib/active_agent/providers/anthropic_provider.rb

@@ -43,7 +43,7 @@ module ActiveAgent
       def initialize(kwargs = {})
         super
 
-        self.json_format_retry_count = kwargs[:max_retries] || ::Anthropic::Client::DEFAULT_MAX_RETRIES
+        self.json_format_retry_count = options.max_retries
       end
 
       # @todo Add support for Anthropic::BedrockClient and Anthropic::VertexClient
@@ -222,6 +222,8 @@ module ActiveAgent
             # No-Op; Wait for Function call to be complete
           when :thinking_delta, :signature_delta
             # TODO: Add with thinking rendering support
+          when :citations_delta
+            # No-Op; the citations arrive with the full block at content_block_stop
           else
             raise "Unexpected delta type: #{api_delta.type}"
           end

data/lib/active_agent/providers/open_ai/chat_provider.rb

@@ -143,6 +143,13 @@ module ActiveAgent
             # => {type: :"tool_calls.function.arguments.delta", name: "get_current_weather", index: 0, arguments: "", parsed: nil, arguments_delta: ""}
           when :"tool_calls.function.arguments.done"
             # => => {type: :"tool_calls.function.arguments.done", index: 0, name: "get_current_weather", arguments: "{\"location\":\"Boston, MA\"}", parsed: nil}
+          when :"refusal.delta"
+            # No-Op; the refusal is merged into the message from the :chunk event
+          when :"refusal.done"
+            # A refused message ends here, with no content.done
+            self.stream_completion_pending = true
+          when :"logprobs.content.delta", :"logprobs.content.done", :"logprobs.refusal.delta", :"logprobs.refusal.done"
+            # No-Op; sent when the request asks for logprobs
           else
             fail "Unexpected Response Event Type: #{api_response_event.type}"
           end

data/lib/active_agent/providers/open_ai/responses_provider.rb

@@ -15,6 +15,18 @@ module ActiveAgent
       class ResponsesProvider < Base
         include ToolChoiceClearing
 
+        # Output items of OpenAI's built-in tools (web search, file search,
+        # image generation, code interpreter, remote MCP).
+        BUILT_IN_TOOL_ITEM_TYPES = %i[
+          web_search_call file_search_call image_generation_call code_interpreter_call mcp_call mcp_list_tools
+        ].freeze
+
+        # Prefixes of the progress events those tools stream.
+        BUILT_IN_TOOL_EVENT_PREFIXES = %w[
+          response.web_search_call. response.file_search_call. response.image_generation_call.
+          response.code_interpreter_call response.mcp_call response.mcp_list_tools.
+        ].freeze
+
         # @return [Class]
         def self.options_klass
           Options
@@ -99,6 +111,7 @@ module ActiveAgent
         # - `:"response.content_part.done"` - content part completed
         # - `:"response.output_item.done"` - message, reasoning or function call completed
         # - `:"response.completed"` - response finished
+        # - `:"response.incomplete"` - response cut short (e.g. max_output_tokens)
         #
         # @param api_response_event [Hash] streaming chunk with :type key
         # @return [void]
@@ -106,6 +119,12 @@ module ActiveAgent
         def process_stream_chunk(api_response_event)
           instrument("stream_chunk.active_agent", chunk_type: api_response_event.type)
 
+          # Built-in tools report their progress with their own events; the
+          # finished tool call arrives as an output item.
+          if BUILT_IN_TOOL_EVENT_PREFIXES.any? { api_response_event.type.to_s.start_with?(_1) }
+            return
+          end
+
           case api_response_event.type
           # Response Created
           when :"response.created", :"response.in_progress"
@@ -124,12 +143,20 @@ module ActiveAgent
             message[:content] += api_response_event.delta
             broadcast_stream_update(message, api_response_event.delta)
 
+          # -> -> -> Content Text Citation (e.g. from web search)
+          when :"response.output_text.annotation.added"
+          # No-Op: The text is complete at response.output_text.done
+
           # -> -> -> Content Text Completed [Full Text]
           when :"response.output_text.done"
             message = message_stack.find { _1[:id] == api_response_event.item_id }
             message[:content] = api_response_event.text
             broadcast_stream_update(message, nil) # Don't double send content
 
+          # -> -> -> Refusal (the response still ends with response.completed)
+          when :"response.refusal.delta", :"response.refusal.done"
+          # No-Op: The message has no refusal field to stream it into
+
           # -> -> -> Content Function Call Append
           when :"response.function_call_arguments.delta", :"response.function_call_arguments.done"
           # No-Op: Wait for FC to Land
@@ -147,8 +174,13 @@ module ActiveAgent
           when :"response.output_item.done"
             process_stream_output_item_done(api_response_event)
 
-          # Response Completed
-          when :"response.completed"
+          # Response Completed, or cut short (e.g. by max_output_tokens) with
+          # the output it has so far
+          when :"response.completed", :"response.incomplete"
+            # The streamed response carries no body for resolve_prompt to read
+            # usage from, so it is recorded from the final event.
+            record_stream_usage(api_response_event.response.usage)
+
             # Completion, tool loop included, runs from stream_finished! once
             # the stream has drained, so each turn's tool calls run once.
             self.stream_completion_pending = true
@@ -174,7 +206,7 @@ module ActiveAgent
             # PATCH: API returns an empty array instead of empty string due to a bug in their serialization
             item_hash = Responses::Transforms.gem_to_hash(api_response_event.item).compact_blank
             message_stack << { content: "" }.merge(item_hash)
-          when :function_call, :reasoning
+          when :function_call, :reasoning, *BUILT_IN_TOOL_ITEM_TYPES
             # No-Op: Wait for the item to land (-> response.output_item.done)
           else
             raise "Unexpected Item Type: #{api_response_event.item.type}"
@@ -194,7 +226,8 @@ module ActiveAgent
           case api_response_event.item.type
           when :message
             # No-Op: Message Up to Date
-          when :function_call, :reasoning
+          when :function_call, :reasoning, *BUILT_IN_TOOL_ITEM_TYPES
+            # Kept like the non-streamed output, which holds every item
             item_hash = Responses::Transforms.gem_to_hash(api_response_event.item)
             message_stack << item_hash
           else

data/lib/active_agent/schema_generator.rb

@@ -34,6 +34,21 @@ module ActiveAgent
   #
   #   ContactForm.to_json_schema
   #
+  # @example ActiveModel with a nested array of objects
+  #   class Person
+  #     include ActiveModel::Model
+  #     include ActiveModel::Attributes
+  #     include ActiveAgent::SchemaGenerator
+  #
+  #     attribute :name, :string
+  #     has_many :addresses, class_name: "Address"
+  #   end
+  #
+  #   Person.to_json_schema
+  #   # => { name: "person", schema: { type: "object", properties: {
+  #   #      name: {...}, addresses: { type: "array", items: { "$ref": "#/$defs/address" } } },
+  #   #      ..., "$defs": { address: { type: "object", ... } } } }
+  #
   # @see Builder
   module SchemaGenerator
     extend ActiveSupport::Concern
@@ -47,6 +62,13 @@ module ActiveAgent
         # Fallback for any class that includes this module
         extend ActiveModelClassMethods
       end
+
+      # The railtie includes this module into ActiveRecord::Base too, whose
+      # has_many must stay ActiveRecord's.
+      unless respond_to?(:has_many)
+        extend ActiveModelAssociations
+        class_attribute :schema_associations, instance_accessor: false, default: {}.freeze
+      end
     end
 
     # Class methods added to ActiveRecord models.
@@ -106,6 +128,40 @@ module ActiveAgent
       end
     end
 
+    # Nested schemas for ActiveModel classes, which have no associations of
+    # their own. Added only to a class without a +has_many+ already.
+    module ActiveModelAssociations
+      # Declares an array of objects described by another schema class.
+      #
+      # {ActiveModelClassMethods#to_json_schema} emits the attribute as an
+      # array whose items are a +$ref+ to that class's schema, which goes under
+      # the root +$defs+ keyed by the singular name in lower camel case
+      # (+addresses+ becomes +address+, +line_items+ becomes +lineItem+).
+      # The prompt preserves both property and definition names.
+      #
+      # Also declares the attribute, defaulting to an empty array, so the
+      # model accepts the data the schema describes.
+      #
+      # @param name [Symbol] the attribute holding the array
+      # @param class_name [String] full constant name of the item class, which
+      #   generates JSON Schema too; resolved when the schema is generated
+      # @return [void]
+      # @raise [ArgumentError] when the class does not include
+      #   +ActiveModel::Attributes+, which declares the attribute
+      #
+      # @example
+      #   has_many :addresses, class_name: "Address"
+      def has_many(name, class_name:)
+        unless respond_to?(:attribute)
+          raise ArgumentError, "#{self.name || inspect}.has_many :#{name} needs ActiveModel::Attributes to declare the attribute; include it before calling has_many"
+        end
+
+        self.schema_associations = schema_associations.merge(name.to_sym => class_name).freeze
+
+        attribute name, default: -> { [] }
+      end
+    end
+
     # Internal builder class for constructing JSON Schemas from model classes.
     #
     # This class handles the actual schema generation logic, supporting both
@@ -122,24 +178,7 @@ module ActiveAgent
       #
       # @raise [ArgumentError] If model_class is not an ActiveRecord or ActiveModel class
       def self.json_schema_from_model(model_class, options = {})
-        schema = {
-          type: "object",
-          properties: {},
-          required: [],
-          additional_properties: options.fetch(:additional_properties, false)
-        }
-
-        if defined?(ActiveRecord::Base) && model_class < ActiveRecord::Base
-          schema = build_activerecord_schema(schema, model_class, options)
-        elsif defined?(ActiveModel::Model) && model_class.include?(ActiveModel::Model)
-          schema = build_activemodel_schema(schema, model_class, options)
-        else
-          raise ArgumentError, "Model must be an ActiveRecord or ActiveModel class"
-        end
-
-        # OpenAI strict mode requires all properties to be in the required array
-        # So we add all properties to required if strict mode is enabled
-        schema[:required] = schema[:properties].keys.sort if options[:strict]
+        schema = object_schema(model_class, options)
 
         {
           name: options[:name] || model_class.name.underscore,
@@ -151,6 +190,37 @@ module ActiveAgent
       class << self
         private
 
+        # Builds the object schema of a model class, without the name and
+        # strict flag {json_schema_from_model} wraps it in.
+        #
+        # @param model_class [Class] The ActiveRecord or ActiveModel class
+        # @param options [Hash] Schema generation options
+        # @param ancestors [Array<Class>] the classes nesting this one, outermost first
+        #
+        # @return [Hash] The object schema
+        def object_schema(model_class, options, ancestors = [])
+          schema = {
+            type: "object",
+            properties: {},
+            required: [],
+            additional_properties: options.fetch(:additional_properties, false)
+          }
+
+          if defined?(ActiveRecord::Base) && model_class < ActiveRecord::Base
+            schema = build_activerecord_schema(schema, model_class, options)
+          elsif defined?(ActiveModel::Model) && model_class.include?(ActiveModel::Model)
+            schema = build_activemodel_schema(schema, model_class, options, ancestors)
+          else
+            raise ArgumentError, "Model must be an ActiveRecord or ActiveModel class"
+          end
+
+          # OpenAI strict mode requires all properties to be in the required array
+          # So we add all properties to required if strict mode is enabled
+          schema[:required] = schema[:properties].keys.sort if options[:strict]
+
+          schema
+        end
+
         # Builds a JSON Schema from an ActiveRecord model.
         #
         # Extracts column information, associations, and validations to build
@@ -187,14 +257,16 @@ module ActiveAgent
 
         # Builds a JSON Schema from an ActiveModel class.
         #
-        # Extracts attribute types and validations to build a schema representation.
+        # Extracts attribute types, nested schemas and validations to build a
+        # schema representation.
         #
         # @param schema [Hash] The schema hash to populate
         # @param model_class [Class] The ActiveModel class
         # @param options [Hash] Schema generation options
+        # @param ancestors [Array<Class>] the classes nesting this one, outermost first
         #
         # @return [void]
-        def build_activemodel_schema(schema, model_class, options)
+        def build_activemodel_schema(schema, model_class, options, ancestors)
           if model_class.respond_to?(:attribute_types)
             model_class.attribute_types.each do |name, type|
               next if options[:exclude]&.include?(name.to_sym)
@@ -206,6 +278,10 @@ module ActiveAgent
             raise ArgumentError, "#{model_class.name} does not define any attributes. Use `attribute :name, :type` to define attributes."
           end
 
+          if model_class.respond_to?(:schema_associations)
+            schema = add_activemodel_associations_to_schema(schema, model_class, options, ancestors)
+          end
+
           if model_class.respond_to?(:validators)
             schema = add_validations_to_schema(schema, model_class, options)
           end
@@ -356,6 +432,51 @@ module ActiveAgent
           schema
         end
 
+        # Adds the arrays an ActiveModel class declares with
+        # {ActiveModelAssociations#has_many}, each item a +$ref+ to the nested
+        # class's schema under +$defs+.
+        #
+        # A +$ref+ resolves against the root of the schema, so the definitions
+        # a nested class collects move up into this schema's +$defs+.
+        #
+        # @param schema [Hash] The schema hash to populate
+        # @param model_class [Class] The ActiveModel class
+        # @param options [Hash] Schema generation options
+        # @param ancestors [Array<Class>] the classes nesting this one, outermost first
+        #
+        # @return [Hash]
+        # @raise [ArgumentError] when a class nests itself, or two different
+        #   schemas would share a definition name
+        def add_activemodel_associations_to_schema(schema, model_class, options, ancestors)
+          path        = ancestors + [ model_class ]
+          definitions = {}
+
+          model_class.schema_associations.each do |name, class_name|
+            next if options[:exclude]&.include?(name)
+
+            nested_class = class_name.constantize
+            if path.include?(nested_class)
+              raise ArgumentError, "#{model_class.name}.has_many :#{name} is recursive (#{(path + [ nested_class ]).map(&:name).join(' > ')}); recursive schemas are not supported"
+            end
+
+            key    = name.to_s.singularize.camelize(:lower).to_sym
+            nested = object_schema(nested_class, options.slice(:strict, :additional_properties), path)
+
+            [ [ key, nested ], *nested.delete(:$defs) ].each do |definition_key, definition|
+              if definitions.key?(definition_key) && definitions[definition_key] != definition
+                raise ArgumentError, "#{model_class.name} nests two different schemas under the definition #{definition_key.inspect}"
+              end
+
+              definitions[definition_key] = definition
+            end
+
+            schema[:properties][name] = { type: "array", items: { "$ref": "#/$defs/#{key}" } }
+          end
+
+          schema[:$defs] = definitions if definitions.any?
+          schema
+        end
+
         # Adds validation constraints to the schema.
         #
         # Translates ActiveModel validations (presence, length, numericality,
@@ -375,11 +496,14 @@ module ActiveAgent
               when ActiveModel::Validations::PresenceValidator
                 schema[:required] << attribute.to_sym unless schema[:required].include?(attribute.to_sym)
               when ActiveModel::Validations::LengthValidator
+                # An array, such as a has_many, is limited by its item count.
+                min_key, max_key = schema[:properties][attribute.to_sym][:type] == "array" ? [ :min_items, :max_items ] : [ :min_length, :max_length ]
+
                 if validator.options[:minimum]
-                  schema[:properties][attribute.to_sym][:min_length] = validator.options[:minimum]
+                  schema[:properties][attribute.to_sym][min_key] = validator.options[:minimum]
                 end
                 if validator.options[:maximum]
-                  schema[:properties][attribute.to_sym][:max_length] = validator.options[:maximum]
+                  schema[:properties][attribute.to_sym][max_key] = validator.options[:maximum]
                 end
               when ActiveModel::Validations::NumericalityValidator
                 if validator.options[:greater_than]

data/lib/active_agent/version.rb

@@ -1,3 +1,3 @@
 module ActiveAgent
-  VERSION = "1.9.1"
+  VERSION = "1.9.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: activeagent
 version: !ruby/object:Gem::Version
-  version: 1.9.1
+  version: 1.9.2
 platform: ruby
 authors:
 - Justin Bowen
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-10-17 00:00:00.000000000 Z
+date: 2026-10-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -456,6 +456,7 @@ files:
 - lib/active_agent/deprecator.rb
 - lib/active_agent/evals.rb
 - lib/active_agent/evals/catalog.rb
+- lib/active_agent/evals/comparison.rb
 - lib/active_agent/evals/correlation.rb
 - lib/active_agent/evals/design_tokens.rb
 - lib/active_agent/evals/diagnosis.rb
@@ -472,6 +473,7 @@ files:
 - lib/active_agent/evals/scenario.rb
 - lib/active_agent/evals/scenario_parser.rb
 - lib/active_agent/evals/scorer.rb
+- lib/active_agent/evals/statistics.rb
 - lib/active_agent/evals/suite.rb
 - lib/active_agent/generation.rb
 - lib/active_agent/generation_job.rb