activeagent 1.9.0 → 1.9.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: da954ef27a255c7c9b81c073b81a320e2f5d85bf89108d2282f23e926adefd1e
-  data.tar.gz: f01703dbd9f5114bbba7574a594e334eb03bab619e41f74c0e65726909d68c33
+  metadata.gz: 2b52709687c94c3d6c989fc36bba7826b931a6d0aa6e193b1bd4482f916d945f
+  data.tar.gz: 1801338ff01afc21837db5382aee49eb4cde1f002b322b479cf2256c283e8668
 SHA512:
-  metadata.gz: 1443ef8d4ef8f0ea234e8110c940f443cd02073859243489d471fd977ccb66fa4d4e3117d64a37b6062d981b109ef206158882623a94a445b36ff77712dfa66f
-  data.tar.gz: 60a3d87ffc1d8cdd5ef5ed02a7a9c9ae3f8e7fbda08db9d1622a26a0b179815b6cfc938cf1e4c9f582399bf1a049eea1121a2caf814a35b56546744a72a7353c
+  metadata.gz: 4d0299374c70e7e66b5edf3b141d171e79cb2f576ec32dcb6a2b4e11e9eb5bccaabbf2a5106fb5afd05f01ece8b6dc75e9a0cd956523c39ffdc4c3b5a17a0b22
+  data.tar.gz: b60f377af9288f08cd5166ba7d71edc1060ccddf23c1d2912ed4b9d105e75ce6a6325a1f0e855e2141ded3c2ebdffbb7cd0e2f37796e662cc37ef833054d21d1

data/CHANGELOG.md

@@ -5,7 +5,225 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [1.9.2] - 2026-10-20
+
+Releases `activeagent` and `actionagent` 1.9.2, with the matching browser
+sidecar: evaluations at scale (the scenario suite as a build page, model
+comparison with intervals and P95 latency, opt-in parallel scenario runs,
+the judge named after its model), provider streaming fixes (Anthropic
+citations, OpenAI Chat refusals and logprobs, Responses built-in tools),
+nested `has_many` schemas, and delegation contracts from `SchemaGenerator`
+classes.
+
+Upgrading: structured-output property names are now sent to the provider,
+and answered, as declared. A `response_format` schema with multiword
+property names (`first_name`) used to reach the provider camelized
+(`firstName`) while `required` stayed snake_case, which strict mode
+rejected; both are now as declared. `parsed_json` is unaffected; code that
+reads `message.content` itself, or matches on request payloads, sees
+`first_name` where it saw `firstName`. In a Rails app, `has_many` is
+available on every `ActiveModel::Model` class, since the railtie includes
+`SchemaGenerator` there, not only on classes that include it themselves.
+Scenario fan-out is off by default: `config.scenario_evaluation_fan_out =
+true` turns it on, and its jobs run on the `:agents` queue. No new
+migration; a host still on 1.9.0 needs 1.9.1's migration 027.
+
+### Added
+
+- **P50 and P95 latency per model in evaluation reports** (`activeagent`).
+  `Report#summary_by_model` adds `p50_duration_ms` and `p95_duration_ms`
+  beside `avg_duration_ms`, by nearest rank as the dashboard's Metrics view
+  computes them, and the Markdown and HTML reports show them next to the
+  mean, which hides the tail (#428).
+
+- **Cost per 1,000 scenario runs in the Markdown report** (`activeagent`).
+  The summary table gains `Cost / 1,000`, each model's mean cost per priced
+  replay on a common denominator, marked `~` when it is estimated or
+  covers only some replays (#428).
+
+- **`ActiveAgent::Evals::Comparison`: is the gap between models real?**
+  (`activeagent`). A 95% interval for each model's pass rate and for the
+  difference between every pair of models, from 5,000 seeded resamples of
+  whole scenarios (a paired bootstrap for the pairs, over the scenarios both
+  ran). A pair is decisive when its interval excludes zero, and
+  `#conclusion` says in one sentence which pairs are, or that the ranking
+  is not decisive evidence of superiority (#428).
+
+- **The model comparison in evaluation reports** (`activeagent`).
+  `Report#comparison` and `to_h["comparison"]` carry it for a report that
+  compares models, the Markdown report gains a Comparison section, and
+  `Report.new(comparison:)` renders one already recorded for the results
+  instead of resampling, as `verdict:` does (#428).
+
+- **A ranked interval plot in the HTML evaluation report** (`activeagent`).
+  Under the comparison table, each model's pass rate is a dot inside its
+  95% interval on a fixed 0 to 100% axis, best first, with the figures
+  beside it, the conclusion, and how the intervals were computed. Plain
+  CSS on the design tokens: no script, light and dark (#428).
+
+- **Each comparison run keeps its model comparison** (`actionagent`). A
+  scenario run that compares models records `Report#comparison` under
+  `scores["_comparison"]` when it completes; a published report's is
+  worked out from the stored results rather than taken from the payload;
+  and the run's report page renders the recorded intervals instead of
+  resampling on every view. A run recorded before gets one from its
+  results (#428).
+
+- **P50 and P95 latency in the dashboard's model comparison**
+  (`actionagent`). The comparison table gains a P50 / P95 column and each
+  model's scorecard a p50 / p95 figure, from the percentiles the run
+  recorded; a run recorded before shows neither (#428).
+
+- **The model comparison plotted in the dashboard** (`actionagent`). A
+  run that compared models shows, under its comparison table, each model's
+  pass rate inside its 95% interval, best first, with the run's
+  conclusion and how the intervals were computed, from the comparison the
+  run recorded (#428).
+
+- Parallel scenario evaluations in `actionagent`: with
+  `config.scenario_evaluation_fan_out = true`, a run enqueues one
+  `EvaluationReplayJob` per scenario × model, each stored replay is scored by
+  its own `EvaluationJudgeJob`, and the last result to be scored completes the
+  run with the scores a sequential run writes. Runs as wide as the host's
+  `:agents` queue. Off by default; browser and host-adapter runs stay
+  sequential (#422).
+- `ActiveAgent::Evals::Runner.new(refine_gate:)`, a callable asked before each
+  recommendation call so per-scenario runners share one run-wide judge limit.
+
+- **Nested arrays of objects in ActiveModel schemas** (`activeagent`).
+  `has_many :addresses, class_name: "Address"` on an ActiveModel class that
+  includes `ActiveAgent::SchemaGenerator` makes `to_json_schema` emit the
+  attribute as an array of `$ref`s to the item class's schema, which goes
+  under the root `$defs`. `strict:` and `additional_properties:` reach the
+  nested schema, a nested class's own definitions move up to the root, and
+  a class that nests itself raises `ArgumentError`. A length validation on
+  the array sets its item count (`min_items` / `max_items`) rather than a
+  string's length. `has_many` also declares the attribute with an
+  empty-array default, and is added only to a class without one, so
+  ActiveRecord's is untouched (#260).
+
+- **Evaluation runs record their judge** (`actionagent`), in
+  `scores["_judge"]`: the `EvaluationJudgeAgent` it runs as, its provider,
+  the model it asked for and where that setting came from (the evaluation,
+  the provider key, the host's config, or the provider's default), and the
+  models that answered, with call counts. The suite's Models panel shows
+  "judged by <model>" with an `auto` badge when no model is pinned, and
+  opens the judge's configuration, with a field that pins `judge_model` on
+  the evaluation through `PATCH /api/evaluations/:id` (#605).
+
+
+### Changed
+
+- **A scenario suite's page reads like a build page** (`actionagent`). The
+  page opened on a summary, every run, the costs, the models and every
+  scenario stacked down the page, so on a suite of 100 scenarios and 12 runs
+  the selected run's results started three screens down. It now opens on the
+  selected run: a run header with the run's passed count, models, agent and
+  judge spend, a to-fix count and a progress bar; beside it a rail of runs,
+  each with a pass bar per model, and a box of the run's scenarios that
+  scrolls on its own, one line per step (`[+]` passed, `[!]` failed, `[~]`
+  running, `[ ]` queued, `[?]` errored, `[-]` skipped) under a sticky header
+  and foldable group rows, with a step's detail opening in place. A toolbar
+  filters the steps to failed only, puts failed first, shows the tool cells
+  and follows the running step until the person scrolls. Models, what to fix
+  and the footer follow below; a failed run shows its error in the header
+  instead of models. Selecting a run keeps the page where it is, polling
+  backs off and stops in a hidden tab, and below 940px the rail becomes a
+  strip of the latest runs. The same suite is now 2,786px tall where it was
+  11,354px (#602).
+
+### Fixed
+
+- **`max_retries` sets the Anthropic `json_object` retries** (`activeagent`).
+  `AnthropicProvider` read the count from its constructor's arguments after
+  `BaseProvider` had already moved `max_retries` into the provider's options,
+  so an unparseable `json_object` answer was always retried twice, whatever
+  `max_retries` said, on Anthropic and on Bedrock, which inherits it. The
+  count now comes from the options: `max_retries: 0` sends the request once,
+  `max_retries: 5` up to six times, and the default is still two retries
+  (#531).
+- **A streamed Anthropic answer with citations no longer raises.** Claude
+  streams the citations of a text block as `citations_delta` deltas when it
+  cites a document or a web search result, and the Anthropic provider raised
+  `Unexpected delta type: citations_delta` on the first one. The delta is now
+  skipped like `input_json_delta`: the full block, citations included, arrives
+  at `content_block_stop`.
+- **A streamed OpenAI Chat refusal, or a stream with logprobs, no longer
+  raises.** The openai gem's chat stream emits `refusal.delta` and
+  `refusal.done` when the model refuses, and `logprobs.*` events when the
+  request sets `logprobs`, and the Chat provider failed on them with
+  `Unexpected Response Event Type`. They are now handled, and `refusal.done`
+  ends the generation, since a refused message never gets a `content.done`.
+- **A streamed Responses answer that used a built-in tool no longer raises.**
+  With `tools: [{ type: "web_search_preview" }]` (or file search, image
+  generation, code interpreter, remote MCP), the stream carries the tool's
+  call as an output item (`web_search_call`, ...), its progress events
+  (`response.web_search_call.searching`, ...), and
+  `response.output_text.annotation.added` for the citations, and the
+  Responses provider raised `Unexpected Item Type: web_search_call` on the
+  first of them. The tool calls are now kept like in a non-streamed response,
+  and their events are skipped.
+- **A streamed Responses answer cut short no longer raises.** When a response
+  hits `max_output_tokens`, the stream ends with `response.incomplete` instead
+  of `response.completed`, and the Responses provider raised `Unexpected
+  Response Chunk Type: response.incomplete`. It now ends the generation with
+  the text streamed so far, like `response.completed`.
+- **A streamed Responses refusal no longer raises.** The provider raised
+  `Unexpected Response Chunk Type: response.refusal.delta` when the model
+  refused; the refusal events are now skipped and the response completes.
+- **A streamed Responses generation reports its token usage.** The usage of a
+  streamed response is only on its final event, and the Responses provider
+  never read it, so `response.usage` was `nil`. It is now recorded from
+  `response.completed` or `response.incomplete`, as the Chat provider does
+  from its usage chunk.
+- **Delegation contracts built from a `SchemaGenerator` class**
+  (`activeagent`). `delegation ..., schema: TicketForm` and
+  `returns TicketForm` produced a contract with no properties, because
+  `to_json_schema` wraps the schema in `{ name:, schema:, strict: }` and
+  the contract read only a top-level `properties`. The wrapper is now
+  unwrapped, and `$defs` are kept so the `$ref`s that nested schemas use
+  still resolve. A contract built from a class is sent with the strict
+  setting the class declared: `Model.to_json_schema` is not strict unless
+  asked, and its `required` then names only the presence-validated
+  attributes, which OpenAI's strict mode refuses, so `returns Model` goes
+  out with `strict: false` and `returns Model.to_json_schema(strict: true)`
+  strict with every attribute required.
+
+- **Structured-output schema normalization preserves field names**
+  (`activeagent`). `prompt` converts JSON Schema keywords to camel case
+  without renaming properties or definitions, keeping multiword attributes
+  and `has_many` associations consistent with `required` and `$ref` in
+  strict schemas. Literal data in `default`, `const`, `enum` and `examples`
+  is preserved too (#260).
+
+  Property names now reach the provider as declared, so a `first_name`
+  attribute is sent as `first_name` rather than `firstName`, and the model
+  answers with that key. `parsed_json` underscores keys either way, so code
+  reading it is unaffected; code reading the raw `message.content`, or
+  matching on request payloads, sees snake_case where it saw camelCase.
+
+  A delegation's `returns` schema follows suit: its `required` entries are
+  no longer camelized, so a multiword required key such as `max_points`
+  matches its property in strict mode instead of being sent as
+  `maxPoints`.
+- **A judge with no `judge_model` is labelled with the model that
+  answered** (`actionagent`), not its provider ("judged by openrouter").
+  OpenRouter's default, `openrouter/auto`, routes each call, so a run judged
+  by more than one model is labelled "<provider> default" (#605).
+
+## [1.9.1] - 2026-10-16
+
+Releases `activeagent` and `actionagent` 1.9.1, with the matching browser
+sidecar version. Adds user feedback to the dashboard. Upgrade by running
+`bin/rails generate action_agent:install --skip` and `bin/rails db:migrate`
+before deploying the updated dashboard (migration 027).
+
+### Added
+
+- Interaction feedback in `actionagent`: submit thumbs up/down and optional
+  comments through the authenticated feedback API, then review and filter votes
+  in an agent’s Feedback tab. Includes response span attribution, retry-safe vote
+  updates, workspace scoping, an upgrade migration and user feedback docs (#591).
 
 ## [1.9.0] - 2026-10-14
 

data/lib/active_agent/base.rb

@@ -397,13 +397,38 @@ module ActiveAgent
 
       if response_format[:type].to_sym == :json_schema
         response_format = nest_flat_json_schema(response_format)
-        response_format[:json_schema] = prompt_view_schema(response_format[:json_schema])
-        response_format[:json_schema].deep_transform_keys! { _1.to_s.camelize(:lower) }
+        response_format[:json_schema] = normalize_json_schema(prompt_view_schema(response_format[:json_schema]))
       end
 
       response_format
     end
 
+    # Convert Ruby-style schema keywords without changing property names,
+    # definition names, references or literal data (default, const, enum,
+    # examples). Only schema-valued keywords are traversed. Named maps keep
+    # their keys, so required entries and $ref paths still address them.
+    #
+    # @api private
+    def normalize_json_schema(schema)
+      return schema.map { |item| normalize_json_schema(item) } if schema.is_a?(Array)
+      return schema unless schema.is_a?(Hash)
+
+      schema.to_h do |key, value|
+        keyword = key.to_s.camelize(:lower)
+        normalized = case keyword
+        when "properties", "patternProperties", "$defs", "definitions", "dependentSchemas", "dependencies"
+          value.is_a?(Hash) ? value.transform_values { |definition| normalize_json_schema(definition) } : value
+        when "schema", "items", "prefixItems", "additionalItems", "contains", "additionalProperties",
+             "unevaluatedItems", "unevaluatedProperties", "propertyNames", "contentSchema",
+             "allOf", "anyOf", "oneOf", "not", "if", "then", "else"
+          normalize_json_schema(value)
+        else
+          value
+        end
+        [ keyword, normalized ]
+      end
+    end
+
     # Moves a json_schema given flat, with its name, schema and strict flag
     # beside +type+, under +json_schema+, so it is not mistaken for a
     # json_schema that names a view. Without a top-level schema, or with a

data/lib/active_agent/delegation/schema.rb

@@ -56,15 +56,25 @@ module ActiveAgent
           schema
         end
 
-        # @param hash [Hash] JSON Schema object
+        # @param hash [Hash] JSON Schema object, or the +{ name:, schema:, strict: }+
+        #   wrapper {ActiveAgent::SchemaGenerator} returns around one
         # @return [Schema]
         def from_hash(hash)
           hash = hash.deep_symbolize_keys
+          strict = nil
+          if !hash.key?(:properties) && hash[:schema].is_a?(Hash)
+            # SchemaGenerator's wrapper carries +strict: true+ only when the
+            # schema was built for strict mode; otherwise it is not strict.
+            strict = hash[:strict] == true
+            hash = hash[:schema]
+          end
 
           new.tap do |schema|
             (hash[:properties] || {}).each { |name, definition| schema.property(name, definition) }
             schema.required(*Array(hash[:required]))
             schema.additional_properties(hash.fetch(:additionalProperties, hash.fetch(:additional_properties, false)))
+            schema.definitions(hash[:$defs]) if hash[:$defs]
+            schema.strict(strict) unless strict.nil?
           end
         end
       end
@@ -73,6 +83,8 @@ module ActiveAgent
         @properties           = {}
         @required             = []
         @additional_properties = false
+        @definitions           = {}
+        @strict                = nil
       end
 
       # @return [Boolean] true when nothing has been declared
@@ -158,6 +170,29 @@ module ActiveAgent
         @additional_properties = value
       end
 
+      # Declares whether the schema was built for a provider's strict mode,
+      # as the +strict:+ of the wrapper {ActiveAgent::SchemaGenerator} returns
+      # says. Strict mode wants every property in +required+; a
+      # SchemaGenerator schema names only the attributes a presence
+      # validation covers unless it was built with +strict: true+, so a
+      # class is sent as it declared itself (see #to_response_format).
+      #
+      # @param value [Boolean]
+      # @return [void]
+      def strict(value = true)
+        @strict = value
+      end
+
+      # Declares the subschemas that properties reference with
+      # +"$ref": "#/$defs/name"+, such as the nested schemas
+      # {ActiveAgent::SchemaGenerator} emits for associations.
+      #
+      # @param definitions [Hash{Symbol => Hash}]
+      # @return [void]
+      def definitions(definitions)
+        @definitions = @definitions.merge(definitions.deep_symbolize_keys)
+      end
+
       # @return [Array<Symbol>] declared property names
       def keys
         @properties.keys
@@ -177,27 +212,30 @@ module ActiveAgent
           type: "object",
           properties: @properties.deep_dup,
           required: @required.map(&:to_s),
-          additionalProperties: @additional_properties
-        }
+          additionalProperties: @additional_properties,
+          "$defs": (@definitions.deep_dup if @definitions.any?)
+        }.compact
       end
       alias_method :to_h, :to_json_schema
 
       # JSON Schema for a +response_format+ payload.
       #
-      # ActiveAgent camelizes response-format schema *keys* on the way to the
-      # provider, but the +required+ array holds string *values* — so they are
-      # camelized here to keep the emitted schema internally consistent.
-      # {Assistant#parsed_json} underscores the keys again on the way back, so
-      # agent code only ever sees snake_case.
+      # ActiveAgent sends property names to the provider as declared, so the
+      # +required+ array names them as declared too.
+      # {Assistant#parsed_json} underscores the keys of the answer, so agent
+      # code only ever sees snake_case.
       #
       # @param name [String] schema name reported to the provider
-      # @param strict [Boolean]
+      # @param strict [Boolean, nil] nil follows the source: what a
+      #   SchemaGenerator class declared (not strict unless built with
+      #   +strict: true+, since its +required+ otherwise lists only the
+      #   presence-validated attributes, which strict mode refuses), else true
       # @return [Hash]
-      def to_response_format(name:, strict: true)
-        schema = to_json_schema
-        schema[:required] = schema[:required].map { |key| key.camelize(:lower) }
+      def to_response_format(name:, strict: nil)
+        strict = @strict if strict.nil?
+        strict = true if strict.nil?
 
-        { name: name, schema: schema, strict: strict }
+        { name: name, schema: to_json_schema, strict: strict }
       end
 
       # Validates a parsed payload against the declared required properties.

data/lib/active_agent/evals/comparison.rb

@@ -0,0 +1,169 @@
+# frozen_string_literal: true
+
+module ActiveAgent
+  module Evals
+    # Whether the gap between the models of a run is larger than the noise of
+    # its scenario sample: each model's pass rate with a confidence interval,
+    # and an interval for the difference between every pair of models.
+    #
+    # The scenario is the unit. A model's pass rate is the mean over its
+    # scenarios of the share of its results there that passed (1.0 or 0.0
+    # for one result per scenario), and the intervals come from resampling
+    # whole scenarios with replacement, +resamples+ times: the percentile
+    # bootstrap. A pair is compared on the scenarios both models ran, each
+    # resample drawing the same scenarios for both (a paired bootstrap of
+    # the per-scenario differences). A pair is decisive when its interval
+    # excludes zero.
+    #
+    # The resampling is seeded, so the same results always give the same
+    # intervals and a report renders the same page each time.
+    #
+    #   comparison = Comparison.new(results: report.results, models: report.models)
+    #   comparison.to_h["pairs"].first
+    #   # => { "models" => ["gpt-5-mini", "qwen3:8b"], "scenarios" => 42, "difference" => 7.1,
+    #   #      "interval" => [0.0, 16.7], "decisive" => false }
+    #   comparison.conclusion
+    #   # => "The pairwise difference interval includes zero: the ranking is not decisive evidence of superiority."
+    class Comparison
+      RESAMPLES = 5_000
+      CONFIDENCE = 0.95
+      SEED = 0
+      # Fewer scenarios than this give no interval: resampling one scenario
+      # only ever draws that scenario.
+      MINIMUM_SCENARIOS = 2
+
+      attr_reader :resamples, :confidence, :seed
+
+      # @param results [Array<Result>]
+      # @param models [Array<ModelSpec>] in the run's order, which breaks ties in the ranking
+      # @param resamples [Integer]
+      # @param confidence [Float] 0..1
+      # @param seed [Integer]
+      def initialize(results:, models:, resamples: RESAMPLES, confidence: CONFIDENCE, seed: SEED)
+        @labels = models.map(&:label)
+        @resamples = resamples
+        @confidence = confidence
+        @seed = seed
+        @outcomes = @labels.to_h do |label|
+          by_scenario = results.select { |result| result.label == label }.group_by { |result| result.scenario.key }
+          [ label, by_scenario.transform_values { |cohort| cohort.count(&:passed?).fdiv(cohort.size) } ]
+        end
+      end
+
+      # Per model label, in the run's order: `{ "scenarios", "pass_rate",
+      # "interval" }`, the pass rate and its interval in percent. The interval
+      # is nil below MINIMUM_SCENARIOS, and the pass rate nil for none.
+      def models
+        @models ||= @labels.to_h do |label|
+          values = @outcomes[label].values
+          [ label, {
+            "scenarios" => values.size,
+            "pass_rate" => percent(mean(values)),
+            "interval" => interval(values)
+          } ]
+        end
+      end
+
+      # The labels by pass rate, best first; ties keep the run's order.
+      def ranking
+        @ranking ||= @labels.sort_by.with_index { |label, index| [ -(models[label]["pass_rate"] || -1), index ] }
+      end
+
+      # One entry per pair of models, the better ranked first: `{ "models" =>
+      # [a, b], "scenarios", "difference", "interval", "decisive" }`, where
+      # "difference" is a's pass rate minus b's over the scenarios both ran,
+      # in percentage points.
+      def pairs
+        @pairs ||= ranking.combination(2).map do |a, b|
+          shared = @outcomes[a].keys & @outcomes[b].keys
+          differences = shared.map { |key| @outcomes[a][key] - @outcomes[b][key] }
+          range = interval(differences)
+
+          {
+            "models" => [ a, b ],
+            "scenarios" => shared.size,
+            "difference" => percent(mean(differences)),
+            "interval" => range,
+            "decisive" => !range.nil? && (range.first.positive? || range.last.negative?)
+          }
+        end
+      end
+
+      # The honest read of the ranking, in a sentence: which pairs differ by
+      # more than the noise, or that none does. nil for a single model.
+      def conclusion
+        return nil if pairs.empty?
+
+        compared, too_few = pairs.partition { |pair| pair["interval"] }
+        decisive, inconclusive = compared.partition { |pair| pair["decisive"] }
+        sentences =
+          if compared.empty?
+            [ "The models share too few scenarios to compare." ]
+          elsif decisive.empty?
+            [ "#{all_include_zero(compared.size)}: the ranking is not decisive evidence of superiority." ]
+          else
+            decisive.map { |pair| decisive_sentence(pair) } + (inconclusive.any? ? [ others_include_zero(inconclusive.size) ] : [])
+          end
+        sentences << "#{plural(too_few.size, 'pair')} share#{'s' if too_few.one?} too few scenarios to compare." if compared.any? && too_few.any?
+        sentences.join(" ")
+      end
+
+      # How the intervals were computed, for a caption.
+      def method_summary
+        { "statistic" => "pass rate", "interval" => "percentile bootstrap", "unit" => "scenario",
+          "resamples" => @resamples, "confidence" => @confidence, "seed" => @seed }
+      end
+
+      def to_h
+        { "method" => method_summary, "models" => models, "ranking" => ranking, "pairs" => pairs, "conclusion" => conclusion }
+      end
+
+      private
+
+      # The bootstrap interval of the mean of +values+, in percent, or nil
+      # below MINIMUM_SCENARIOS.
+      def interval(values)
+        return nil if values.size < MINIMUM_SCENARIOS
+
+        random = Random.new(@seed)
+        size = values.size
+        means = Array.new(@resamples) do
+          sum = 0.0
+          size.times { sum += values[random.rand(size)] }
+          sum / size
+        end
+        # A rational tail, so 95% leaves exactly 2.5: in floating point it
+        # is 2.5000000000000022, whose nearest rank is one too high.
+        tail = (1 - @confidence.rationalize) / 2 * 100
+        [ percent(Statistics.percentile(means, tail)), percent(Statistics.percentile(means, 100 - tail)) ]
+      end
+
+      def mean(values)
+        values.sum.fdiv(values.size) if values.any?
+      end
+
+      def percent(fraction)
+        (fraction * 100).round(1) if fraction
+      end
+
+      def all_include_zero(count)
+        count == 1 ? "The pairwise difference interval includes zero" : "All #{count} pairwise difference intervals include zero"
+      end
+
+      def others_include_zero(count)
+        count == 1 ? "The other pairwise interval includes zero." : "The other #{count} pairwise intervals include zero."
+      end
+
+      def decisive_sentence(pair)
+        better, worse = pair["difference"].positive? ? pair["models"] : pair["models"].reverse
+        low, high = pair["interval"].map(&:abs).sort
+        "#{better} passes more scenarios than #{worse}, by #{pair['difference'].abs} points " \
+          "(#{(@confidence * 100).round}% CI #{low} to #{high})."
+      end
+
+      def plural(count, word)
+        "#{count} #{word}#{'s' unless count == 1}"
+      end
+    end
+  end
+end