activeagent 1.9.6 → 1.9.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9f73d1c3c9a6dffc5bf68d07e59a21a88e40eef1ab92da228688e4f1e8148658
-  data.tar.gz: caf1a01b8e53a7a6e9685e39162c38f88fed355fe9bef46f8850a1c4b313a09b
+  metadata.gz: a2e216123a2bdec63446cf6f6af685c0c61b285a318f5a98845e09b188838427
+  data.tar.gz: 434fbcc22e9a7ffa10434c93ba4b3d63306e6f0ae5502742eb2c83a31abb9338
 SHA512:
-  metadata.gz: '02886d6a6c6d30366ae19b1a2877c3969a39b814896a942f67e4c6beca14fa53c5169dff447adcb956039e8c6358d2903969d9c254e8475c8c0af467b66770e6'
-  data.tar.gz: 5cf3b2847e796674db3033a0745ab1e549a54ede83ddee48fb6f3edb8bd33d7256db17267a3ff42935aba39f1c07e0f2e8832bf9e7578216825176919271898d
+  metadata.gz: 442971fa937ce0d4f44c6add3d3e8841f48789da87de99363e8fcc0a891bcf4bc49461b914c5816ee2754a8c8b22ed92d42b0a26cff3423f3c5de8fbdd3e653c
+  data.tar.gz: 971714f2043a08e213f3a646706da229d77f289f2a98e7674169406ee1b5767235b98988a71113868dcd93d680da72dd9721859d49c30577f812ac902e84704f

data/CHANGELOG.md

@@ -5,6 +5,413 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [1.9.7] - 2026-11-04
+
+Evaluations an application runs in its own test suite, and the dashboard
+pieces around them: a suite declares the pass rate each group of questions
+must reach (`pass`), and `assert_meets_pass_gates` or RSpec's
+`meet_its_pass_gates` holds a run to it; scenarios replay a set number of
+trials; the judge can skip an answer that already failed its expectations;
+`Patterns` builds the regex for a count or a share a seeded scenario
+expects; scenario keys select strictly; named criteria lists layer per
+audience; `ActiveAgent::Evals::RubyLLM.verify!` checks every model's
+credentials before a run; and the publisher builds itself from the
+environment and returns links that open. Alongside them: Ollama Cloud,
+scoring leases, catalog scenario extensions kept, evaluation catalogs
+without Rails, ranked comparison tables, and users' own remote MCP servers.
+
+Upgrading: three dashboard migrations, 030 `add_evaluation_scoring_leases`
+(the scoring lease columns on `evaluation_scenario_results`), 031
+`add_catalog_scenario_metadata` (the `metadata` column on
+`catalog_scenarios`) and 032 `create_active_agent_mcp_server_configs` (the
+table behind a user's own MCP servers): run `bin/rails generate
+action_agent:install --skip` and `bin/rails db:migrate`. Until they run,
+fanned-out runs keep scoring with the claim 1.9.3 uses, catalog imports
+keep working but still drop scenario extensions (extensions discarded
+before the upgrade need the original document imported again), and
+the MCP Services page cannot list or save servers of your own.
+
+The pricing tables changed too: a trace recorded under Ollama that priced
+at the default rate before prices at Ollama Cloud's listed rate now, and a
+finished evaluation run's cached figures are worked out again.
+
+A suite's `pass` block is read and checked as the suite loads: a key other
+than `threshold` and `min_pass_rate` (and, on a group, `min_pass_rate` and
+`gated`) raises `ArgumentError` naming it, and in a catalog a product's or
+set's `pass` raises `Catalog::InvalidDocument`. 1.9.6 ignored a suite's
+`pass` and kept a product's as metadata, so a document that already carried
+one under its own keys needs them renamed.
+
+### Added
+
+- **Ollama Cloud** (`activeagent`). `generate_with :ollama, cloud: true,
+  model: "gpt-oss:120b"` sends requests to `https://ollama.com/v1` with the
+  key from `OLLAMA_API_KEY` (or `api_key:`), and a request to the cloud
+  without a key fails before it is sent, naming what to set. The host can
+  also be given as `host:` or through `OLLAMA_HOST`, read the way the
+  `ollama` command reads it: a bare origin is completed to its `/v1`
+  endpoint, and a scheme-less `host[:port]` is http on 11434. A local server
+  signed in to the cloud needs none of this — `model: "gpt-oss:120b-cloud"`
+  works as before — and the provider knows that spelling names the cloud's
+  model (`ActiveAgent::Providers::Ollama::Cloud`).
+- **Ollama model lookup** (`activeagent`). `provider.models` lists the
+  models the configured host serves and `provider.model_info(name)`
+  describes one — its details, capabilities, context length, and whether
+  the cloud serves it — from the host's own API (`/api/tags`, `/api/show`),
+  local server or Ollama Cloud alike; a host with only the OpenAI-compatible
+  API is listed through `/v1/models` (`ActiveAgent::Providers::Ollama::Catalog`).
+- **Ollama Cloud rates on traces and evaluations** (`actionagent`). A model
+  run under Ollama is priced from Ollama Cloud's listing by the name the
+  cloud uses: RubyLLM's registry carries it under its `ollama_cloud`
+  provider, and an install without RubyLLM has the same rows bundled
+  (`ModelPricing::OLLAMA_CLOUD_RATES`), each a `catalog` rate rather than a
+  guess at the family's. `gpt-oss:120b-cloud` through a local server and
+  `gpt-oss:120b` through the cloud price alike, and `deepseek-v4-flash`
+  under Ollama is Ollama's price, not DeepSeek's.
+- **The model lookup says which host is the cloud** (`actionagent`).
+  `POST /api/provider_keys/test` and `GET /api/provider_models?provider=ollama`
+  carry `cloud` and, per listed model a price list names, `rates`
+  (`{ input, output, source }` in $ per million tokens); Settings → Provider
+  API Keys shows both with the connection test.
+
+- **Atlas Cloud as an OpenAI-compatible provider** (`activeagent`).
+  `service: "AtlasCloud"` generates through Atlas Cloud's compatibility
+  endpoint, with the key from `ATLASCLOUD_API_KEY` and an overridable
+  `base_url`; chat only, no embeddings (docs/providers/atlas_cloud.md).
+  Contributed in #352.
+
+- **Cheaper Inference as an OpenAI-compatible provider** (`activeagent`).
+  `service: "CheaperInference"` generates and embeds through
+  `https://api.cheaperinference.com/v1`, with the key from
+  `CHEAPER_INFERENCE_API_KEY`; a thin subclass of the OpenAI provider, like
+  Opper. Contributed in #629 and landed in #666.
+
+- **Your own MCP servers** (`actionagent`). A dashboard user adds a remote
+  MCP server by URL on the MCP Services page (*Your MCP servers*), with
+  optional request headers, tests it (`tools/list`, recording the tools it
+  serves) and enables it on agents from the Tools tab, where it is listed
+  beside the catalog as `custom:<id>`. Agent runs and scenario evaluation runs
+  call its tools, and evaluation fix items attribute them to it. Each server
+  belongs to an owner like a provider key (`MCPServerConfig`, install
+  migration 032), and every lookup, including a run's, is scoped to that
+  owner. Header values are encrypted at rest, never returned to the browser
+  (names with masked values; a blank value on update keeps the stored one),
+  and scrubbed from recorded output. Writes and tests ask for
+  `:manage_credentials`. A URL that reaches a loopback, private or
+  link-local address is refused when saved and when called (`MCPHostGuard`),
+  unless the host sets `ActionAgent.allow_private_mcp_hosts`, e.g. for local
+  development. API: `/api/mcp_server_configs` with a `test` member action.
+
+- **Evals in the test suite** (`activeagent`). An eval is an ordinary test
+  under `test/evals`, collected by `rails test` like any other, whose
+  `ActiveAgent::Evals::TestCase` skips it unless `EVALS` is on, so the
+  suite never calls a model by accident. `bin/rails test:evals` switches
+  evals on and runs them through the same runner (`TEST=path` narrows,
+  `TESTOPTS` passes through); `EVALS=1 bin/rails test ...` runs one file,
+  one line, or the whole suite with the evals in. The same switch gates an
+  RSpec suite (`filter_run_excluding eval: true unless
+  ActiveAgent::Evals::TestHelper.enabled?`, with `expect_with :minitest`
+  for the assertions). `require "active_agent/evals/test_helper"`
+  after `rails/test_help` gives `ActiveAgent::Evals::TestCase` and the
+  `TestHelper` module: `run_eval MyAgent.with(...).action` returns an
+  `ActiveAgent::Evals::Replay` — the answer, every tool call with its
+  arguments, result or error and timing, duration and tokens — and
+  `assert_answer_present`, `assert_output_contains` /
+  `refute_output_contains`, `assert_tool_called` (with an argument subset)
+  / `refute_tool_called` and `assert_tools_succeeded` read it, failing with
+  what the agent did. `assert_eval_passes` (and `score_eval`) scores the
+  replay as a dashboard run does and fails with the fault and the fix;
+  `run_eval_suite` replays a YAML suite across models and returns the
+  `Report`, which `assert_pass_rate` gates and `to_html` writes.
+  `ActiveAgent::Evals::Capture` is the assertion-free core: it runs a
+  generation into a `Replay`, capturing tool calls on the agent's
+  `tools_function` (where the telemetry tool spans are; tools served by a
+  client-side MCP server are recorded through the agent's `tool_observer`),
+  replays under another `model:` or `provider:` the way a comparison run
+  does, and takes a block for an agent run some other way. The framework's railtie now
+  loads the gem's rake tasks; the empty `activeagent_tasks.rake`
+  placeholder is gone.
+
+- **A suite declares its pass gates** (`activeagent`). A suite's `pass`
+  block sets the score every answer must reach (`threshold`) and the share
+  of a model's answers that must pass (`min_pass_rate`), for the suite and
+  for any group with its own bar; `gated: false` keeps a group in the report
+  and out of `Report#passed?`. A catalog's products and sets take the
+  same block, and later documents layer it key by key. `Suite#pass_thresholds`
+  reads it as a `PassThresholds`, `Runner.new(pass_thresholds:)` takes it in
+  place of `threshold:` and hands it to the report, and `Report#gates` is
+  then one row per group × model with its pass rate against its bar, plus an
+  `All` row per model over its gated results; `Report#passed?` is every gate
+  met, and a report with no gate (a selection that matched nothing, or every
+  scenario dropped as `production_only`) does not pass;
+  `Report#failures(group:)` lists the results that did not pass. The
+  Markdown report adds a *Pass rate by group* table (`gates_markdown`;
+  `failures_markdown(group:)` lists the failures for a test message), the
+  HTML page tones each group's pass count by its gate, and `to_h` gains
+  `pass_thresholds`, `groups` and `gates`. A report built without pass
+  thresholds renders as it did. The command line's
+  `evals run --min-pass-rate` stays a gate on the whole run.
+- **The pass-gate assertion, for Minitest and RSpec** (`activeagent`).
+  `TestHelper#assert_meets_pass_gates(report)` passes when every gate of
+  the report is met, each group held to the bar the suite declares for it
+  and ungated groups left out, where `assert_pass_rate` holds every model
+  to one rate the test names; it takes a `PassThresholds` to hold the
+  report to another bar, and returns the check. `run_eval_suite` hands a
+  suite's bars to the runner when it declares any (a `threshold` other
+  than 0.7, a `min_pass_rate` below 1, or any key in a group's `pass`), so
+  the report it returns carries them; a suite with no `pass` block, or
+  only a suite-level one that restates the defaults, runs under the
+  defaults and its report carries no bars, and a `pass_thresholds:` or
+  `threshold:` given to `run_eval_suite` wins over the suite's. In RSpec the
+  assertion works through `expect_with :minitest` like the rest of the
+  helper, and `require "active_agent/evals/rspec"`, with
+  `ActiveAgent::Evals::RSpec` included beside the helper, adds the first
+  eval matcher, `expect(report).to meet_its_pass_gates`; that file loads
+  nothing from rspec, and `require "active_agent/evals"` loads neither it
+  nor the helper. Both fail with one message, `GateCheck#message`: how many
+  of the gates were not met, each unmet gate as its sentence, then the
+  failures to read under a heading per group with an unmet gate (one
+  `Report#failure_markdown(result)` line each, the line `failures_markdown`
+  is built from) — and, for an `All` row that is short, the failures it
+  counts: under `Failures:` when no group row is short with it, else the
+  ones outside the groups already listed under `Other failures:`; an
+  ungated group's failures are never listed. `GateCheck.new(report)`
+  answers `met?`, `unmet` and `gates` for a script that reports on its own.
+- **Pattern builders for counts and shares** (`activeagent`).
+  `ActiveAgent::Evals::Patterns.number(count)` and
+  `Patterns.share(count, total)` return a regex string for a scenario's
+  `contains` or `not_contains` entry: `number(10)` matches the figure on its
+  own and not the `10` in `2026-10-01`, `10/1`, `10:00`, `1–10`, `Oct 10`,
+  `10 October`, `10.5`, `110`, `10,000` or `10%`; `share(3, 4)` matches
+  `75%`, `75 %`, `75.0%` and `75.00%` and not `175%` or `7.5%`.
+  `Scorer.matches_pattern?` tries an entry as a substring and then as a
+  regex, so the string goes into a suite document or a scenario hash as it
+  is, and a scenario seeded with known data can expect the figure that data
+  implies. A fractional or negative count, a total of zero or a count above
+  the total raise `ArgumentError`.
+- **Strict scenario selection by key** (`activeagent`). `Suite#scenarios`
+  and `Catalog#scenarios` take `strict: true` with `keys:`: every key must
+  name a scenario the suite or catalog holds, or the call raises
+  `Suite::NotFound` listing the missing keys in the order given, so a
+  misspelled key from an environment variable or a command line fails the
+  run, named in the error. A key the suite holds outside the chosen
+  `groups:` returns an empty selection. `Suite.select` is the same check
+  over any scenarios, with `known:` the wider set the keys must come from,
+  for a runner that takes keys across several suites. Without `strict:` the
+  selection narrows as before: the scenario a misspelled key meant is left
+  out without notice, and only when every key misses does a run held to
+  `Report#passed?` (or `assert_meets_pass_gates`) fail, without naming the
+  key.
+- **Skip the judge for a replay that already failed its expectations**
+  (`activeagent`). `Runner.new(judge_failed_expectations: false)` asks the
+  judge about an answered replay only when every expectation its scenario
+  declared was met. A replay that called none of its expected tools, printed
+  a forbidden phrase, lacks expected content or had a tool call fail is a
+  failed result whatever a judge would say — `Diagnosis` assigns one of its
+  evidence faults (`tool_error`, `missing_capability`,
+  `expected_tool_not_called`, `ungrounded_answer`, `forbidden_content` or
+  `missing_content`) before it looks at quality — so it fails on whichever
+  of those it assigns first, with no `task_completion` score, its
+  `llm_judge` criteria unscored and no judge-written recommendation (a
+  refusal or invented figure the judge skipped loses the judge's suggested
+  tool too), and the calls saved stay out of the `judge_limit` budget.
+  `Scorer.new(judge_failed_expectations:)` and
+  `Scorer.expectations_met?(scores)` are the pieces underneath, and the
+  scores hash keeps its key order either way. The default (`true`) grades
+  every answered replay as it did; a blank answer is never graded, and a
+  blank or errored replay is never refined under the default
+  `refine_faults`, with or without the flag.
+- **Replay each scenario several times** (`activeagent`, `actionagent`).
+  `Runner.new(trials: 3)` replays every scenario under every model that
+  many times, scenario by scenario and model by model, so a flaky scenario
+  gets a pass share instead of one coin flip. Scenario keys stay as they
+  are; `Result#trial` tells the repeats apart, reads 1 on a result built
+  without one, and lands in `to_h` as `"trial"`. A model's `"scenarios"`
+  count and pass rate cover every replay, with `"unique_scenarios"` beside
+  them, and the verdict then reads "Passed 5 of 6 replays of 2 scenarios";
+  a matrix cell shows one mark per trial with the mean score and the most
+  frequent fault; the Answers section lists every trial; a failure line
+  (`failure_markdown`, `failures_markdown`, the pass-gate message) names
+  the trial (`gpt-5-mini` · trial 2) when its scenario × model was
+  replayed more than once; the Markdown, JSON and HTML reports state the
+  trial count only when it is above 1, so a one-trial Markdown or HTML
+  report is exactly what it was, and its JSON differs only by `"trial"` on
+  each result and `"unique_scenarios"` on each model. `Comparison` pools a
+  scenario's replays into its pass share, so its intervals still resample
+  scenarios. The collector accepts the repeats of one scenario × model when
+  each carries its `trial`, stores every one, and rebuilds them into the
+  run's report; the dashboard's own runs replay once.
+- **The evaluation publisher reads `run_id` from the report, returns an
+  absolute receipt `url` and builds itself from `ENV`** (`activeagent`).
+  `ActiveAgent::Evals::Publisher#call` and `#progress` take `run_id:` as
+  optional and read it from the report's `metadata.run_id`, where
+  `Correlation#with_run` records it, from a `Report` or its saved JSON hash;
+  an explicit `run_id:` wins, and a report with neither raises
+  `ArgumentError` before anything is sent. The receipt's `url` comes back
+  absolute: the path a collector sends
+  (`<mount>/evaluations/:evaluation_id/runs/:id` from the engine) is joined
+  onto the publisher's `dashboard_url`, by default the endpoint's origin
+  (`https://activeagents.ai` for the hosted collector on its API host), or
+  the HTTP(S) URL `Publisher.new(dashboard_url:)` is given, so the link opens
+  in a browser; an absolute `url` and a receipt without one are left as they
+  are. `Publisher.from_env` builds the publisher from
+  `ACTIVEAGENTS_EVALUATIONS_ENDPOINT` (the hosted default when blank),
+  `ACTIVEAGENTS_API_KEY` (required) and the new, optional
+  `ACTIVEAGENTS_DASHBOARD_URL`, for a dashboard people reach at another
+  address than the collector's:
+  `Publisher.from_env.call(report: report, source: "support-app", agent_name: "SupportBot", suite: "orders")`.
+- **Verify RubyLLM models and the judge before a run** (`activeagent`).
+  `ActiveAgent::Evals::RubyLLM.verify!(models, context:)` takes a run's
+  `ModelSpec`s (or one) and, for the first that cannot run, raises
+  `RubyLLM::ConfigurationError` saying why: a provider RubyLLM does not know
+  (the message lists the ones it does), one outside `providers:` (the names
+  a host has keys for), or one its context cannot build —
+  `claude-sonnet-5 runs on anthropic, which has no API key. Set
+  ANTHROPIC_API_KEY.`, naming the provider's `configuration_requirements`
+  the context leaves unset (`which is not configured. Set OLLAMA_API_BASE.`
+  when they are not keys), or carrying RubyLLM's own message when those are
+  all set and it refused for another reason, with RubyLLM's error as the
+  cause. It builds each chat with `assume_model_exists: true`, which is
+  where RubyLLM checks its keys, and sends nothing. `judge(..., verify: true)`
+  runs the same check on the judge's own model when the judge is built,
+  with the same `providers:` when it is given them, so a judge whose
+  provider has no key refuses before the first scenario is replayed rather
+  than at its first grade; the default stays lazy.
+- **Named criteria lists on a suite** (`activeagent`). A suite document
+  carries a top-level `criteria` beside `groups`: a list of the
+  `{ key, type, config }` hashes `Runner.new(criteria:)` takes, which reads
+  as the one list named `default`, or a mapping of name → list, one per
+  audience the same questions are graded for. `Suite#criteria(*names)`
+  joins the named lists in the order given, a later list's criterion
+  replacing an earlier one with the same key, ready for the runner;
+  with no names it returns every list in document order, and a name no
+  document declares contributes nothing, so a deployment asks for `shared`
+  plus an audience that may have no list of its own. `Suite#criteria_names`
+  lists the names. Later documents layer each list by criterion key, as they
+  layer scenarios: a key already present is replaced in place, a new key
+  appended. A catalog product carries the same mapping, which `suite_for`
+  hands its suite and `to_yaml` writes back. A `criteria` value that is
+  neither a list nor a mapping of lists, or a criterion that is not a
+  mapping or has no key, is refused: `ArgumentError` from a suite,
+  `Catalog::InvalidDocument` from a catalog product (a set keeps a criterion
+  without a key; see Changed) and `ScenarioParser::ParseError` from a pasted
+  suite. Earlier releases ignored a suite document's top-level `criteria`;
+  a document that carries one in another shape is refused.
+
+### Changed
+
+- **Every trace cost is priced under its provider** (`actionagent`). The
+  trace list, Metrics, the agent scorecard, the interactions list and an
+  evaluation result priced from its trace all hand `ModelPricing` the
+  provider the llm span recorded, so `gpt-5.5` through OpenRouter is
+  OpenRouter's rate, and a trace's `OpenAI::Chat` tag reads as `openai`
+  (`ModelPricing.provider_slug`). `TelemetryTrace.pluck_with_llm_model_and_provider`
+  plucks the provider beside the model for a caller that prices tokens.
+
+- **Ranked comparison tables and value pills** (`actionagent`). Wherever the
+  dashboard compares rows, they now read through one pattern: a
+  `ComparisonTable` with two-line mono headers that sort on click and a
+  rank-tinted `ValuePill` in every numeric cell, banded by tercile in the
+  column's better direction or by fixed thresholds where the domain has them
+  (pass rates at 85 and 50), with missing values unranked and sorted last,
+  and a visually hidden rank word so colour is never the only signal. Applied
+  to a run's model comparison table and the Evaluations leaderboard, the
+  scenario matrix (the key leads each step with the prompt under it, one
+  score pill per model, cost and time pills per scenario, the tools level
+  stacking tool call, time, tokens and cost pills), each opened result's
+  score, time, tokens and cost strip, the Metrics overview's change pills and
+  the home page's Top Agents list. Under 600px a table stacks each row to
+  labelled pills.
+
+- **A set's `criteria` layer across catalog documents** (`activeagent`). A
+  later document's criterion replaces the one with its key in place and a
+  new key is appended, as a product's named lists and a suite's scenarios
+  layer; a later document used to replace the set's whole list. A criterion
+  without a key is still accepted and appended as it is, once: a later
+  document that restates it adds nothing. A criterion's `key` is written as
+  a string, as scenario keys are, so a set criterion keyed by a number reads
+  back with a quoted key and the catalog's digest changes with it.
+
+### Fixed
+
+- **The dashboard engine loads in a host without Action Cable**
+  (`actionagent`). The engine's class body camelizes its inflection
+  overrides with `Zeitwerk::Inflector` and did not require Zeitwerk itself.
+  Rails requires it only once the application sets up its autoloaders,
+  after `Bundler.require` has loaded the gem; until then the constant
+  exists only because Action Cable's engine loads it as a side effect. An
+  app generated with `--skip-action-cable`, or any API-only host, died at
+  boot with `uninitialized constant ActionAgent::Engine::Zeitwerk` (seen on
+  Rails 8.1.4; the require order is the same on every Rails since 7.0). The
+  engine requires Zeitwerk, a dependency of Active Support, before reading
+  it, and a test loads the engine in a fresh process right after `rails`,
+  with nothing in between.
+
+- **An evaluation judge job delivered twice scores its result once**
+  (`actionagent`). The claim on a pending result was a conditional write on
+  its timestamp, which turned away a stale reader but not a duplicate
+  delivered after the claim: that one read the new timestamp and asked the
+  judge to score the result again. A claim now takes a 30-minute lease
+  (`scoring_token`, `scoring_expires_at`) that a fresh reader cannot take
+  while it is live, and a worker that finishes after its lease was
+  reclaimed cannot overwrite its successor's result. A delivery that finds
+  a live lease, such as the queue's redelivery after a worker died
+  mid-judge, enqueues the job again for when the lease expires, so the run
+  still completes. Without migration 030 the judge job keeps the timestamp
+  claim instead of failing on the missing columns.
+
+- **Keep scenario catalog extension fields** (`activeagent`, `actionagent`).
+  Opaque scenario fields survive parsing, YAML/JSON export, database reload,
+  standard-field edits and storage restore. Their changes affect import
+  previews and catalog digests, including when only an extension changes.
+  Empty values and literal `metadata` keys are preserved; canonical format
+  fields cannot be replaced through the metadata container.
+
+- **Bound trace console list limits** (`actionagent`). The console defaults
+  to 50 recent matching traces, clamps integer limits to 1–1000, and uses
+  the default for malformed values. Its documentation describes the bounded
+  recent list rather than claiming pagination.
+
+- **Select trace time windows in dark mode** (`actionagent`). The dark
+  Traces header includes the shared time-window selector and zoom controls,
+  including when embedded on an agent page or when the window is empty.
+
+- **Show tool usage in both dashboard themes** (`actionagent`). The Traces
+  page renders the same tool counts, durations, failures and empty state
+  in dark and light mode, using the shared telemetry colors.
+
+- **Load evaluation catalogs without Rails** (`activeagent`). The standalone
+  `active_agent/evals` entry point loads the ActiveSupport array extension
+  used to construct, parse and load catalogs, without loading the framework.
+
+- **Client-side MCP tool calls appear in an eval replay** (`activeagent`).
+  `Evals::Capture` recorded the tools that reach the agent's
+  `tools_function`, and a tool served by a client-side MCP server is
+  dispatched before it, so a replay of an agent with `mcps:` showed none of
+  those calls and `assert_tool_called` could not find them. The provider
+  now runs such a call inside the agent's `tool_observer`, which Capture
+  sets, so it is recorded once with its arguments, result preview and
+  timing. A result the server marks as an error, which the model receives
+  as a result and not an exception, is recorded as a failed call, so
+  `failed_tool_calls` and `assert_tools_succeeded` see it. A server the
+  provider runs itself (Anthropic's or OpenAI's hosted MCP) executes on the
+  provider's side and is still not recorded.
+
+- **An agent run is executed by one job** (`actionagent`). The execution
+  job read a run's status, then wrote `running`, as two steps, and only
+  turned away a finished or paused run. A job delivered twice, or two
+  workers that both loaded the run as pending, each started a generation
+  for it, and a duplicate that arrived while the run was running started
+  another. The job now moves the run from pending to running in one
+  conditional update, as `CodeSessionJob` claims a session, and starts no
+  generation unless it was the one to do so. The database decides at the
+  write, so this holds on SQLite under Rails 7.2 too, where a row-lock read
+  followed by a write is refused once another worker has committed. Nothing
+  is held over the generation, so a cancel does not wait for one. A run
+  whose worker died after the claim stays running and is not run again.
+
 ## [1.9.6] - 2026-10-29
 
 The dashboard on a phone, and the small finished pieces: the sidebar as a
@@ -1386,6 +1793,35 @@ finished must handle. Session Replay is now **Sessions** in the sidebar.
   do for `generate_later`. `InputRequest#arguments` holds the arguments of the
   paused call.
 
+- **`ActiveAgent::Evals::RubyLLM`** — the RubyLLM side of an evaluation,
+  which hosts driving RubyLLM conversations had been writing for themselves.
+  `require "active_agent/evals/ruby_llm"` (it requires `ruby_llm`;
+  `require "active_agent/evals"` alone still does not) gives two helpers, for
+  RubyLLM 1.16 and later and for 2.x:
+  - `RubyLLM.judge(label:, model:, provider:, context:, correlation:)`, a
+    `Judge` that answers from `context.chat(...)` and, with a `Correlation`,
+    traces each call under the kind it serves. `temperature:` and a
+    `configure:` hook set the chat up; any other keyword must be one
+    `RubyLLM::Chat.new` takes, and anything else raises when the judge is
+    built rather than failing every call. `on_usage:` reports each call's
+    model, tokens and cost.
+  - `RubyLLM.replay(messages)`, a `Replay` from a conversation's messages.
+    `acts_as_chat` records of either RubyLLM generation are read through
+    their own `to_llm`, so a 2.x table, or one between 2.x's upgrade and its
+    cleanup, replays the same as `RubyLLM::Message` values. Tool calls keep
+    the order the model made them in, each paired with the result that
+    answers it; one is errored when that result reports an error at its top
+    level — JSON whose `"error"` is a non-empty string, object or array, or
+    `true`; an MCP result with `"isError": true`; or the inspected
+    `{ error: "..." }` RubyLLM 1.x stores for a tool's error Hash with a
+    String error — with the error (JSON-encoded, at most 1,200 bytes) as its
+    detail. `input_tokens` counts the whole prompt, cache reads and writes
+    included. Cost is RubyLLM's own price, left nil when any message is
+    unpriced. The answer is the last reply after the last user message; a
+    conversation that stopped at a tool call or on the user's own message
+    has no answer and says so in its `error` (on RubyLLM 1.x, pass `answer:`
+    for a tool that ends the turn with `halt`).
+
 ### Changed
 
 - **Confirm destructive actions in the dashboard's own dialog**
@@ -2100,34 +2536,6 @@ output in the shape ruby_llm reads. No migrations.
   `json_object`, which ruby_llm has no mode for, and a `json_schema`
   without a schema now raise `ArgumentError` instead of sending a request
   the API rejects (#501).
-- **`ActiveAgent::Evals::RubyLLM`** — the RubyLLM side of an evaluation,
-  which hosts driving RubyLLM conversations had been writing for themselves.
-  `require "active_agent/evals/ruby_llm"` (it requires `ruby_llm`;
-  `require "active_agent/evals"` alone still does not) gives two helpers, for
-  RubyLLM 1.16 and later and for 2.x:
-  - `RubyLLM.judge(label:, model:, provider:, context:, correlation:)`, a
-    `Judge` that answers from `context.chat(...)` and, with a `Correlation`,
-    traces each call under the kind it serves. `temperature:` and a
-    `configure:` hook set the chat up; any other keyword must be one
-    `RubyLLM::Chat.new` takes, and anything else raises when the judge is
-    built rather than failing every call. `on_usage:` reports each call's
-    model, tokens and cost.
-  - `RubyLLM.replay(messages)`, a `Replay` from a conversation's messages.
-    `acts_as_chat` records of either RubyLLM generation are read through
-    their own `to_llm`, so a 2.x table, or one between 2.x's upgrade and its
-    cleanup, replays the same as `RubyLLM::Message` values. Tool calls keep
-    the order the model made them in, each paired with the result that
-    answers it; one is errored when that result reports an error at its top
-    level — JSON whose `"error"` is a non-empty string, object or array, or
-    `true`; an MCP result with `"isError": true`; or the inspected
-    `{ error: "..." }` RubyLLM 1.x stores for a tool's error Hash with a
-    String error — with the error (JSON-encoded, at most 1,200 bytes) as its
-    detail. `input_tokens` counts the whole prompt, cache reads and writes
-    included. Cost is RubyLLM's own price, left nil when any message is
-    unpriced. The answer is the last reply after the last user message; a
-    conversation that stopped at a tool call or on the user's own message
-    has no answer and says so in its `error` (on RubyLLM 1.x, pass `answer:`
-    for a tool that ends the turn with `halt`).
 - **`ActionAgent::ProviderKey.credentials_for(owner)` and
   `.apply_to(config, owner:)`** (`actionagent`) hand an owner's API keys to
   code outside the engine — `{ "openai" => "sk-..." }`, or written through

data/README.md

@@ -48,6 +48,9 @@ bundle add openai
 # Anthropic
 bundle add anthropic
 
+# Atlas Cloud (uses OpenAI-compatible API)
+bundle add openai
+
 # Ollama (uses OpenAI-compatible API)
 bundle add openai
 
@@ -60,6 +63,9 @@ bundle add openai
 # Opper (uses OpenAI-compatible API)
 bundle add openai
 
+# Cheaper Inference (uses OpenAI-compatible API)
+bundle add openai
+
 # DeepSeek (uses OpenAI-compatible API)
 bundle add openai
 

data/lib/active_agent/base.rb

@@ -336,6 +336,7 @@ module ActiveAgent
         exception_handler:,
         stream_broadcaster:,
         tools_function:,
+        tool_observer:,
         action_name: generation_action_name,
         input_request_resume: @_input_request_resume,
         announce_input_requests: !input_requests_silenced?

data/lib/active_agent/cli/eval_commands.rb

@@ -138,11 +138,12 @@ module ActiveAgent
 
       # Posts a saved report to the collector. FILE is a version-1 envelope
       # (what `evals report --format json` writes) or a bare Report#to_h
-      # hash, whose identities then come from the options.
+      # hash, whose identities then come from the options; the publisher
+      # reads the run_id from the report's metadata when neither names one.
       def evals_import(argv)
         identities = {}
         argv = parse(argv, "evals import FILE [--run-id ID --source NAME --agent NAME --suite NAME]") do |p|
-          p.on("--run-id ID", "The report's run_id (kept across retries)") { |v| identities["run_id"] = v }
+          p.on("--run-id ID", "The report's run_id, kept across retries; by default the one its metadata records") { |v| identities["run_id"] = v }
           p.on("--source NAME", "The application that ran it") { |v| identities["source"] = v }
           p.on("--agent NAME", "The agent's name") { |v| identities["agent_name"] = v }
           p.on("--suite NAME", "The suite's name") { |v| identities["suite"] = v }
@@ -156,7 +157,6 @@ module ActiveAgent
           raise Error.new("#{path} is a version #{envelope["version"].inspect} envelope; this CLI sends version 1", exit_code: EXIT_USAGE)
         end
         envelope = envelope.merge(identities)
-        envelope["run_id"] ||= envelope["report"].is_a?(Hash) ? envelope["report"].dig("metadata", "run_id") : nil
 
         receipt = client.publisher.call(report: envelope["report"], run_id: envelope["run_id"], source: envelope["source"],
                                         agent_name: envelope["agent_name"], suite: envelope["suite"])

data/lib/active_agent/concerns/tooling.rb

@@ -26,5 +26,14 @@ module ActiveAgent
         raise_without_input_request_secrets(error)
       end
     end
+
+    # A callback the provider runs the calls it serves from an MCP server
+    # around, which never reach {#tools_function}. It receives the tool name,
+    # its arguments and a block that makes the call, and returns the block's
+    # result. Nil unless a host that records a generation's tool calls (see
+    # Evals::Capture) sets it on the instance.
+    #
+    # @return [Proc, nil]
+    def tool_observer = nil
   end
 end