activeagent 1.8.0 → 1.8.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3642068294052d483decf5260b5d7dd1793647f06920713ca658f7b89ba948b0
-  data.tar.gz: 1d0811bee23dc646e916d17abc711637c1e2d41055633ace8bb1c067615d26b4
+  metadata.gz: 1c7be2b31907ff2779ea3818516d887afecc6c540c89670c33f7ec1be242c01a
+  data.tar.gz: 0d7327bb8f94d2573f6e87b9191afbc7758b2fdb7d42a2ea896fbd25dfd71b17
 SHA512:
-  metadata.gz: 6d4492797c3d4b238bf86c49c48b9ec12e5876bbc9ef075c4cae7f369a3e3274b3db30781b9367643740bd9fa365eb8fe8ad81308b125f6843f3f828ccf5f339
-  data.tar.gz: 601676a4590808eef1469f3cfafc5136c43bccbd22e7b600c15940eb7a1fc957479c568676d699d7d41c10836fa621eb072a3ff5fd20d5f4cfa22a517e982416
+  metadata.gz: fd0f4bbab5439daabe85170c3fb57fe5c94260e0573fc35944e80cc8301bd35803fb5d89f2a6de9c6061c44752e67f125db9ad17d116058b444e92785c3819c6
+  data.tar.gz: 453ecb45269cc816fbea41378591e642a460a2a188e052745b87eed1fa9f7777281449a7adfe19ebef17e12e242009ad46ee6a773f0cc33660b161f6d4585197

data/CHANGELOG.md

@@ -7,6 +7,155 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.8.1] - 2026-10-01
+
+### Added
+
+- **Every evaluation cost is priced, and says how** (`actionagent`,
+  `activeagent`). A scenario result with no cost is priced down a chain —
+  the cost the publishing application reported, the estimate the engine
+  stored, the result's tokens × its model's rate, the tokens of the trace
+  it links to, or its text at four characters a token as a lower bound —
+  and a result that recorded no tokens costs `$0.00` (`cost_source:
+  no_usage`); only a result with no tokens, no trace and no text stays
+  unpriced. Results carry `cost` (the effective figure), `reported_cost`,
+  `cost_source`, `cost_rate` and `judge_usage`; a run's `usage` adds
+  `reported`, `estimated`, `cost_basis` and `total`; `scores._models` adds
+  per model `reported`, `estimated`, `judge_cost` and `judge_calls`; and
+  `GET /api/evaluations/:id/runs/:run_id` adds `costs` per scenario (judge
+  apart) and for the run (`ActionAgent::EvaluationRunCost`, cached per
+  finished run). The judge's spend is found the same way — the engine's
+  meter, the application's figures (`result.judge_usage`,
+  `report.judge_usage.run`, accepted by the report import) or the judge
+  traces priced on input and output tokens, never thinking tokens — within
+  the run's tenant. `ModelPricing` looks rates up under the provider the
+  model ran on, strips gateway and vendor prefixes and date suffixes, tries
+  dots and dashes both ways, prices `claude-sonnet-5` and the `gpt-5`
+  family by exact rows ahead of the family patterns, and reports where a
+  rate came from (`estimate_detailed`, `rate_detail`, `fingerprint`). The
+  engine's judge meter counts Anthropic's cached prompt tokens.
+- **One display format for passes, scores and costs** (`activeagent`
+  `ActiveAgent::Evals::Format`). A fraction always carries its percent
+  (`14/16 · 88%`; `14/16 (88%)` in Markdown; `—` for nothing scored), every
+  0..1 score reads as a whole percent (`93%`, `pass ≥ 70%`), and money
+  reads `$0.0243` when reported and `~$0.0243` when any part was estimated,
+  with one legend per surface. The HTML report gains a Cost tile, a Judge
+  column and per-model judge line, a trailing matrix Cost column with
+  group subtotals, a cost line per cell and in each result's details, a
+  judge chip with its calls and cost, a release chip, the pass mark and
+  the cost in its footer; the Markdown report gains a Judge column, a
+  matrix Cost column, the cost per answer and a total line. Diagnosis
+  text reads `Task completion scored 60% against a pass threshold of 70%`.
+- **Costs from replay metadata** (`activeagent`). A `Replay`'s metadata may
+  carry `cost_source`, `cost_rate` and `judge_usage`; `Report#summary_by_model`
+  adds `reported`, `estimated`, `judge_cost` and `judge_calls`,
+  `Report#scenario_costs` gives each scenario's cost across models, and
+  `Report#judge_usage` sums every result's judge calls with the run-level
+  part passed as `Report.new(judge_usage:)`. `Report.new(release:)` (and
+  `Runner.new(release:)`) names the release the run scored, in
+  `to_h["release"]` and a header chip. `to_h` is unchanged when neither is
+  given.
+- **An evaluation's standing against the agent as it is now**
+  (`actionagent`). Each evaluation reports its `headline_run_id` (its
+  newest complete run; a newer pending or failed run shows beside it), its
+  `standing` — `current`, `stale`, `unrecorded`, `archived` or `none`
+  (`ActionAgent::EvaluationStanding`) — its `archived_at` and `per_model`
+  passes, and every run its `agent_version` and `version_state`. Only
+  model-facing edits (instructions, action prompts, tools, MCP servers,
+  model config, response format) make a run stale. A published report's
+  `report.release` pins the run to that release, recorded as a version when
+  the dashboard has not seen the digest (`Agent#find_or_record_release!`,
+  which never moves a deploy's `release_digest` backwards); a report with
+  no release leaves the run unrecorded. `PATCH /api/evaluations/:id` with
+  `evaluation: { archived: true | false }` archives an evaluation or brings
+  it back; `GET /api/evaluations` leaves archived evaluations out before
+  its 50-row limit unless `?archived=1`, and returns `archived_count`. A
+  new run or a published report brings an archived evaluation back.
+
+- Codex code sessions in checkout sandboxes. Connect an OpenAI API key under
+  Settings → Integrations and select Codex in the code-session panel. The local
+  backend runs `codex exec` with JSONL events, workspace-write sandboxing, stdin
+  prompts, per-sandbox configuration, cancellation, timeout and diff capture.
+- Explicit code-runner capability checks for host backends. Existing adapters
+  continue to support Claude Code without implicitly receiving Codex credentials.
+
+### Changed
+
+- **The dashboard reads passes, scores and costs one way** (`actionagent`
+  frontend). Every fraction carries its percent (`14/16 · 88%`), every 0..1
+  score reads as a whole percent, and every cost reads `$0.0243` when
+  reported or `~$0.0243` when any part was estimated, with the legend once
+  per surface and the tokens × rate working in the figure's tooltip. The
+  Evaluations page's tiles pool only the headline run of each current or
+  unrecorded evaluation, show a pass line per model, and say how many
+  evaluations were left out as stale or archived; a card carries its
+  standing, an archive/unarchive control and, for a newer run still
+  pending or failed, that run's badge beside the headline's; *Show
+  archived (n)* lists the archived ones. The suite panel gains the spend
+  strip between Runs and Models, the matrix a cost line per cell
+  (`~$0.0243 · judge ~$0.0015`) and a trailing Cost column with group
+  subtotals, the model comparison a Judge column, the runs list a version
+  chip and a same-version / new-version / release-not-recorded line under
+  each delta, and the spend strip reads the judge's cost from the traces
+  and says "rules only" only when there is no judge. The agent cards'
+  Eval tile is the pooled pass rate with its fraction in the title.
+- **The agent card's Eval tile is the pooled pass rate** (`actionagent`
+  `AgentScorecard`) over the headline runs of the agent's current and
+  unrecorded evaluations — never a stale suite's or an archived one's —
+  rather than the mean criterion score of whichever run was last.
+  `eval_runs` and `eval_not_counted` say what was pooled and what was left
+  out.
+- **The partial-cost notes are retired** (`actionagent`, `activeagent`).
+  The `*` marker and the "k of n priced" notes of 1.8.0's partial-cost fix
+  give way to the `~` mark and its legend: a cost that covers only some of
+  a model's replays is a lower bound and reads as an estimate. The verdict
+  rationale reads `Passed 2 of 2 scenarios (100%) with a mean score of 100%
+  at ~$0.0010 (estimated)`, and the judge ruling on a comparison is told
+  each model's agent cost alone — never the judge's own spend, which never
+  enters the ranking either. The HTML report's header shows a chip per
+  scalar metadata value only: an array or object (the judge's trace ids)
+  is no longer rendered as one.
+- **What to fix comes after the scenario results** (`actionagent`,
+  `activeagent`). A run report now reads models, then the scenario results,
+  then What to fix. The scenario suite panel moves What to fix below the
+  scenario matrix. The standalone HTML report
+  (`ActiveAgent::Evals::ReportHtml`) moves its fix cards below the matrix and
+  the per-scenario details. `Report#to_markdown` moves its Recommendations
+  below its Answers. The sampling run detail already read in this order.
+  Each section's content is unchanged.
+
+### Fixed
+
+- Inherited Codex settings and credentials are removed from sandbox process
+  environments. Each Codex run receives only its owner's selected connection.
+
+- **An evaluation's cost when some interactions carried no cost estimate**
+  (`actionagent`, `activeagent`). A run's cost sums only the interactions that
+  were priced, but its per-interaction rate divided that partial sum by every
+  interaction, and nothing said part of the run was unpriced. The rate is now
+  over the priced interactions, and a run's `usage` counts them: `priced` and
+  `unpriced` beside `replays` (or `samples`). A run where nothing was priced
+  reports no `cost` or `per_interaction`, as before, with `priced: 0`. The
+  per-model summaries count them too: `Report#summary_by_model` adds `priced`
+  per model and a sampling run's `_cohorts` add `priced` per cohort. A
+  scenario run recorded before that has its `_models` counted from its
+  results when the API serves it (`EvaluationRun#model_summaries`); a
+  sampling cohort recorded before that reads as fully priced when it has a
+  cost. The dashboard shows a partial cost as an estimate — "estimated, 3 of
+  5 replays priced" — on the run's spend strip and footer, the model
+  scorecards and the Evaluations page's cost-per-interaction tile, and marks
+  it `*` in the runs list, the model comparison table and the spend strip's
+  total, whose titles give the count. The HTML
+  and Markdown reports and the pass-rate verdict name the priced count beside
+  a partial cost, the judge ruling on a comparison is told it, and a
+  pass-rate verdict breaks a tie on cost per priced scenario rather than on
+  the partial sum, so an unpriced replay no longer makes a model look
+  cheaper.
+
+Upgrade both gems together, then run `bin/rails generate action_agent:install
+--skip` and `bin/rails db:migrate`. The migration adds runner identity to code
+sessions; existing sessions remain Claude Code sessions.
+
 ## [1.8.0] - 2026-09-29
 
 Releases `activeagent` and `actionagent` 1.8.0 from one tag. A minor release.

data/lib/active_agent/evals/diagnosis.rb

@@ -285,11 +285,11 @@ module ActiveAgent
 
         weakest = (failed_grade ? grades : @scores.compact).min_by { |_, value| value }
         summary = if failed_grade
-          "#{graded_label(grades)} scored #{grade.round(2)} against a pass threshold of #{@threshold}"
+          "#{graded_label(grades)} scored #{Format.score(grade)} against a pass threshold of #{Format.percent(@threshold)}"
         else
-          "Scored #{@score.round(2)} against a pass threshold of #{@threshold}"
+          "Scored #{Format.score(@score)} against a pass threshold of #{Format.percent(@threshold)}"
         end
-        summary += ", weakest on #{weakest.first} (#{weakest.last.round(2)})" if weakest
+        summary += ", weakest on #{weakest.first} (#{Format.score(weakest.last)})" if weakest
         recommendation =
           if weakest
             "Read the answer against the #{weakest.first.to_s.humanize.downcase} criterion and adjust the " \

data/lib/active_agent/evals/format.rb

@@ -0,0 +1,115 @@
+# frozen_string_literal: true
+
+module ActiveAgent
+  module Evals
+    # How every evaluation surface writes a pass count, a score and a cost, so
+    # the HTML report, the Markdown report, a diagnosis and the dashboard read
+    # alike:
+    #
+    #   - a pass/fail fraction carries its percentage: "14/16 · 88%" (in
+    #     Markdown "14/16 (88%)"); nothing scored (0/0) reads "—"
+    #   - a 0..1 score — a mean score, a criterion score, task completion, a
+    #     judge's confidence, the pass threshold — reads as a whole percent:
+    #     "93%", "pass ≥ 70%"
+    #   - money has four decimals, six below $0.001, "$0.00" for an explicit
+    #     zero, and a leading "~" when any part of it was estimated from
+    #     tokens × model rates rather than reported; LEGEND explains the mark
+    #
+    # Percentages round half up to a whole number. Stored values stay 0..1
+    # and USD; only their display changes here. The dashboard's
+    # actionagent/frontend/utils/evalFormat.mjs is the JavaScript copy of
+    # these rules, so a change here needs the same change there.
+    module Format
+      EMPTY = "—"
+      # Shown once per surface where a "~" figure is visible.
+      LEGEND = "~ estimated from tokens × model rates"
+      # Rate sources that are a guess at the model's price rather than a
+      # catalog entry for it (see the dashboard's ModelPricing).
+      FALLBACK_RATE_SOURCES = %w[pattern default].freeze
+
+      module_function
+
+      # 0.875 → "88%". "—" for a missing value.
+      def percent(fraction)
+        return EMPTY unless finite?(fraction)
+
+        "#{whole(fraction.to_f * 100)}%"
+      end
+
+      # 14 of 16 → "14/16 · 88%", or "14/16 (88%)" with `style: :markdown`,
+      # where " · " would read as a table cell separator next to a pipe.
+      # "—" when nothing was scored.
+      def passes(passed, total, style: :text)
+        count = total.to_i
+        return EMPTY unless count.positive?
+
+        done = passed.to_i
+        pct = "#{whole(done * 100.0 / count)}%"
+        style == :markdown ? "#{done}/#{count} (#{pct})" : "#{done}/#{count} · #{pct}"
+      end
+
+      # A 0..1 score as a percent: 0.93 → "93%".
+      def score(value)
+        percent(value)
+      end
+
+      # 0.7 → "pass ≥ 70%".
+      def threshold(value)
+        "pass ≥ #{percent(value)}"
+      end
+
+      # 0.0243 → "$0.0243", 0.000697 → "$0.000697", 0 → "$0.00"; with
+      # `estimated`, "~$0.0243". "—" for a missing value.
+      def money(value, estimated: false)
+        return EMPTY unless finite?(value)
+
+        amount = value.to_f
+        digits = if amount.zero? then 2
+        elsif amount.abs < 0.001 then 6
+        else 4
+        end
+        "#{'~' if estimated}$#{format("%.#{digits}f", amount)}"
+      end
+
+      # A $/M token rate with at least two decimals and no trailing noise:
+      # 5 → "$5.00/M", 0.075 → "$0.075/M".
+      def per_million(rate)
+        whole, fraction = format("%.4f", rate.to_f).split(".")
+        fraction = fraction.sub(/0+\z/, "")
+        "$#{whole}.#{fraction.ljust(2, '0')}/M"
+      end
+
+      # The tooltip of an estimated figure: how it was worked out, e.g.
+      # "estimated: 2,328 in × $5.00/M + 423 out × $30.00/M · catalog rate".
+      # `rate` is `{ "input", "output", "source" }` in $ per million tokens;
+      # a rate from the name-pattern table or the default appends
+      # "(fallback rate)". Without a rate it names the method only.
+      def cost_title(input_tokens: nil, output_tokens: nil, rate: nil)
+        rate = rate.to_h.transform_keys(&:to_s) if rate.respond_to?(:to_h)
+        return "estimated from tokens × model rates" unless rate.is_a?(Hash) && finite?(rate["input"]) && finite?(rate["output"])
+
+        source = rate["source"].presence || "catalog"
+        title = "estimated: #{delimited(input_tokens)} in × #{per_million(rate['input'])} + " \
+                "#{delimited(output_tokens)} out × #{per_million(rate['output'])} · #{source} rate"
+        title += " (fallback rate)" if FALLBACK_RATE_SOURCES.include?(source.to_s)
+        title
+      end
+
+      # Half up, on the decimal value: 14.5 → 15, even when the float arrived
+      # as 14.499999999999998.
+      def whole(value)
+        value.to_f.round(9).round
+      end
+
+      def delimited(count)
+        count.to_i.to_s.gsub(/(\d)(?=(\d{3})+\z)/, '\1,')
+      end
+
+      def finite?(value)
+        return false if value.nil? || value == ""
+
+        Float(value, exception: false)&.finite? || false
+      end
+    end
+  end
+end

data/lib/active_agent/evals/judge.rb

@@ -158,6 +158,8 @@ module ActiveAgent
       def verdict(summaries, instructions: nil)
         lines = summaries.map do |label, stats|
           faults = (stats["faults"] || {}).map { |fault, count| "#{fault}×#{count}" }.join(", ")
+          # The agent's cost only: what the judge itself spent scoring a
+          # cohort says nothing about the model under comparison.
           "#{label}: pass rate #{stats['pass_rate']}%, mean score #{stats['avg_score'] || 'n/a'}, " \
             "avg latency #{stats['avg_duration_ms'] || 'n/a'}ms, cost $#{stats['cost'] || 'n/a'}" \
             "#{", faults: #{faults}" if faults.present?}"