activeagent 1.3.1 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e041e8f2080b82e5b01648a83edfcfb7efa66cdfdb44fe814c4cca03e268816a
-  data.tar.gz: d6bcf308c34a8eb7522137275beaa286571fdc82c13cb91a0d8de9fa4609313f
+  metadata.gz: 7401cdff6a7895238a082383f2ef32683b743946445e11dacbf377470bf31071
+  data.tar.gz: b0f847d04eedf620c7f5320950d65e5ffc712296ef7db2b7ea6be25f3bc433b7
 SHA512:
-  metadata.gz: a9010a8e0f091c79757f282fd4f233b19468bfeb979fdecd4f00ad3b239bd60c3e39d159e17187444d56804fadde65ad748dd84fd7fb25fde9ecdfe3b081d257
-  data.tar.gz: 17c642804ffd9dc781bbb7a55fc5f68207fbb44ad68e836509aa3657d81767fee8d0138a29e53193fde58926e2c889dfa77461ba31372e95e5782f55adfb4b29
+  metadata.gz: 0a29cb99c3139f9c264eb6e86677d47ed592fb165c39489be2efc8ea2bb2dd1614bc8802643b812b04319b7356bc37177943bc9c2e0174de276555ba811ba426
+  data.tar.gz: 6fb392e7cb2f4381d79ae16567edca55d62c585b06f04a6a3da6b31cd0bb7e6149b2e67ceb7e5e143b04bc41123dd1d5bba14f38b33be6e21813e9923308d524

data/CHANGELOG.md

@@ -5,6 +5,427 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [1.5.0] - 2026-09-10
+
+Releases `activeagent` 1.5.0 and `actionagent` 1.5.0 from one tag.
+
+`actionagent` goes from 1.3.0 to 1.5.0, skipping 1.4: the two gems are
+released together from this repository and from one tag, and carrying one
+version number across both is less confusing than explaining which
+dashboard version pairs with which framework. `actionagent` 1.4 does not
+exist and never will. The engine's floor on the framework
+(`activeagent >= 1.4`) is unchanged and still correct.
+
+### Added
+
+- **`ActiveAgent::Evals::Publisher` delivers a finished report to a
+  collector.** A run that already happened — in CI, in a host app's own
+  runtime, anywhere the evaluation core runs — can be sent to an
+  ActiveAgents-compatible collector without replaying the agent:
+  `Publisher.new(api_key:, endpoint:).call(report:, run_id:, source:,
+  agent_name:, suite:)` posts a version-1 envelope wrapping `Report#to_h`
+  (or the saved JSON hash of an earlier run) and returns the collector's
+  receipt. Delivery is synchronous, requires HTTPS outside loopback, does
+  not follow a redirect carrying the bearer credential, caps a request at
+  2 MiB, and raises `Publisher::Error` on anything but a receipt naming the
+  same `run_id` — so a retry with that same `run_id` and the saved report
+  re-delivers rather than re-runs. Publication is strictly opt-in and
+  happens only where an application writes the call: no configuration flag,
+  no callback, no default credential, and `api_key:` supplied explicitly at
+  the call site. That is deliberate, because the payload is the report
+  itself — every scenario's prompt, the agent's answers, and the tool calls
+  and their results — and whether that may leave the application is the
+  application's decision to make. Installing the gem sends nothing
+  anywhere. `docs/evals/publication.md` documents the envelope, the receipt
+  and the retry rules. (#414)
+
+- **`Runner` takes `around_evaluation:` and `require_judge_scores:`.**
+  `around_evaluation:` is called with `(scenario, spec)` and a block, and
+  wraps the whole evaluation — the replay, the scoring, the judge calls
+  behind a recommendation — so a host can establish one trace context
+  across all of it and correlate a replay with the judging it triggered. It
+  must return the block's result; `on_result` runs after it returns, an
+  error it raises propagates to the caller, and `#evaluate` called directly
+  bypasses it, for a host doing its own scheduling. `require_judge_scores:`
+  (default `false`) settles what an unusable judge means. A judge that
+  raises or answers unscorably is skipped, and the scenario is then decided
+  on its rule scores alone — which reads as "the agent passed" when the
+  truth is "nobody graded the answer". Set it, and an otherwise passing
+  result whose `task_completion` or declared `llm_judge` criterion has no
+  usable score fails instead, with the new `judge_unavailable` fault naming
+  the unscored criteria and pointing at the judge's credentials, model and
+  JSON reply. A run with no judged criteria is unaffected. (#414)
+
+- **A grouped suite imports as YAML or JSON, whole.** `ScenarioParser` read
+  a pasted list or a JSON array of scenarios; it now also reads the grouped
+  document `Suite` loads — `groups:` with per-group keys and display names,
+  scenarios carrying `key`, `prompt`, `notes`, `expect` and
+  `production_only` — from YAML or JSON, keeping every part of it.
+  `ScenarioParser.parse` and `.scenarios` gain `include_production_only:`,
+  which defaults to `true` to match `Suite`. The dashboard defaults it the
+  other way: post the document as `scenarios_text` and the production-only
+  questions stay out unless `include_production_only` is sent alongside it,
+  because those prompts run against a live agent. The choice is made at
+  import — the engine stores the scenarios it selected, not the source
+  document — so changing it means importing the document again. (#414)
+
+- **`actionagent`: `ActionAgent.scenario_evaluation_adapter_resolver`.** A
+  host application with its own agent runtime can now run an evaluation
+  itself while keeping the dashboard's catalog, selection, jobs, result
+  persistence and report pages. The resolver is called with the persisted
+  evaluation and returns `nil` for the engine's normal `Agent#test_execute`
+  path, or a callable — `evaluation:`, `owner:`, `scenarios:`, `models:`,
+  `on_result:` — that runs the host's own agent and judge, yields every
+  result as it lands, and returns an `ActiveAgent::Evals::Report`. The
+  engine holds it to that contract: anything other than a `Report`, or a
+  report that omits or duplicates one of the selected scenario × model
+  pairs, fails the run rather than completing it with rows missing, and an
+  exception leaves the results already written in place. Dashboard
+  authentication, execution enablement and the host's execution quota still
+  apply. (#414)
+
+- **`actionagent`: the Run Agent page is a conversation workbench.** Testing
+  an agent used to mean one prompt in, one output out, with no way to see —
+  or shape — what the model was given. The page now works the way a user
+  would work the agent: it pins a persisted conversation (a solid_agent
+  context) and every run sends that conversation's user and assistant turns
+  ahead of the new message, so follow-up questions actually follow up. The
+  context is editable in place — edit or delete a turn, seed a user or
+  assistant message without running, start a new conversation — and every
+  run is a fresh `AgentRun` with its own trace, so Traces and Interactions
+  see exactly what the model saw. Files attach to a message and ride along
+  through Active Storage (`AgentRun has_many_attached :attachments`, guarded
+  for hosts without it): images reach the model as vision input, PDFs as
+  documents, and text-like files (CSV, Markdown, JSON, plain text) are
+  inlined into the message; the persisted user message keeps an attachment
+  manifest so the conversation shows thumbnails afterwards. Assistant replies
+  can render **generative UI** — cards, stats, tables, charts, lists,
+  progress, forms, choice buttons, images, callouts and code — from a fenced
+  ```` ```ui ```` JSON block in prose, a JSON reply whose top level is
+  `ui`/`blocks`, or the new `render_ui` tool (enable the **Generative UI**
+  tool on the agent). Forms and choices post their answer back into the
+  conversation as the next user message. New engine API: `GET/POST
+  /api/agents/:id/conversations`, message create/update/delete under
+  `/api/interactions/:id/messages`, multipart `POST /api/agents/:id/execute`
+  with `attachments[]` and `params[context_id]`, and attachment metadata on
+  run and message JSON. The reference host (`test/dummy`) gained the Active
+  Storage tables so the attachment path is exercised by the engine's tests.
+  Two notes for anyone driving that API directly: `execute`/`test` now answer
+  422 unless the request carries a prompt or a file, and per-run overrides in
+  `params` can no longer name `attachments` or `action` — those stay the
+  controller's to set. Model-supplied images in generative UI load on sight
+  only when they are inline data or this app's own URL; any other host is
+  offered as a click-to-load, since fetching one tells that host whatever the
+  model put in the URL.
+
+### Fixed
+
+- **A scenario passes only if it completed the task.** A scenario's verdict
+  was the mean of everything scored for it, and the judge's
+  `task_completion` grade was one number in that mean: an answer that
+  called the expected tool and contained the expected string could carry a
+  task grade of 0.2 to a mean of 0.73 and pass at the default threshold of
+  0.7. `task_completion` is a gate now — it has to reach `threshold` on its
+  own, and no number of passing tool and content checks can lift it — and
+  the fault names the number that failed: "Task completion scored 0.2
+  against a pass threshold of 0.7". An evaluation that configures its own
+  `llm_judge` criteria rather than relying on the implicit grade — which is
+  what the dashboard does — is gated the same way, on the mean of those
+  grades, so one soft dimension among strong ones still passes while an
+  answer the judge marked down cannot be carried by its mechanics. A
+  scenario the judge could not grade at all is unchanged, still falling
+  back to the rule scores. `score` and `avg_score` still mean the aggregate
+  they always did, and each model's
+  summary gains `avg_task_completion` so the judge's grade reads separately
+  from it. **This can turn a suite that passed on 1.4.0 red; see the note
+  on upgrading below.** (#414)
+
+- **A judge's score is read as a JSON number.** The score was pulled out of
+  the judge's reply by regular expression, matching the first run of digits
+  after `"score":`. It read `{"score": 9e-2}` — 0.09 — as 9, clamped to a
+  perfect 1.0; it read the string `{"score": "0.9"}` and the truncated
+  `{"score": 0.9oops}` as a confident 0.9 rather than as unusable. The
+  score now comes from the parsed JSON object and has to be a finite
+  number, so exponent notation is read as written and a string, a boolean,
+  `null`, `NaN` or `1e999` is unscorable — which the runner already knows
+  how to handle. Fenced ```` ```json ```` replies still parse. The
+  dashboard's generation-sampling evaluations score through the engine's own
+  judge rather than the framework's, and read a score by the same rule now,
+  so the two halves of the dashboard no longer disagree about the same
+  reply. (#414)
+
+- **A judge that answers with the wrong types cannot put junk in the fix
+  list.** `suggested_tool` and `instruction_change` were coerced rather
+  than checked, so a reply of `"suggested_tool": {"name": true}` added a
+  tool literally named `true` to the report's suggested tools, and
+  `"instruction_change": ["invalid"]` became a fix card asking someone to
+  add `["invalid"]` to the agent's instructions. Both fields must now be
+  nonempty strings and are dropped when they are not, so a malformed reply
+  loses only the malformed part: the judge's recommendation still reaches
+  the result, the report and every rendering of it. (#414)
+
+- **A grouped suite pasted into the dashboard keeps its keys and
+  expectations.** Only a pasted list or JSON was recognised, so a YAML suite
+  went to the line parser and was read as prose: a document describing three
+  scenarios became eighteen, with prompts like `tools: [lookup_order]` and
+  `production_only: true`, groups named `expect`, generated keys in place of
+  the document's own, and every expectation dropped — a suite that looked
+  imported and scored nothing real. Such a document is now parsed as the
+  suite it is, and one that is not valid, or a selection that matches no
+  scenarios, returns an import error (`ScenarioParser::ParseError`, HTTP
+  422) instead of a suite of nonsense or a sampling evaluation nobody asked
+  for. (#414)
+
+- **`actionagent`: run and result metadata survive persistence.** A run
+  rebuilt from the database was rebuilt without it: `Report#metadata` came
+  back holding only the four keys the engine writes itself, and each
+  result's replay metadata was gone entirely, so a host's own run and result
+  IDs, response trace IDs and judge trace IDs did not survive the round trip
+  and its reports could not be joined to its telemetry. Run metadata is now
+  kept in `scores["_metadata"]` and per-result metadata in
+  `diagnosis["_replay_metadata"]`, restored by `EvaluationRun#to_report` and
+  served as `metadata` on result JSON. Both are reserved storage keys that
+  the public diagnosis excludes, so nothing migrates and `diagnosis` still
+  means what it did. (#414)
+
+- **`actionagent`: refreshing a catalog does not rewrite what an earlier run
+  asked.** A saved run rendered its scenarios from the catalog rows as they
+  are now, so rewording a question, retagging its group or changing its
+  expectations silently rewrote history — last month's report showed this
+  month's prompt above last month's answers, and judged them against
+  expectations that were not in force when they were given. Each result now
+  records the scenario it was actually evaluated against in
+  `diagnosis["_scenario_snapshot"]`, and the report, the API and the
+  scenario matrix read that snapshot, in the order the run itself used, with
+  the dashboard noting on a scenario whose catalog entry has since changed
+  that re-running uses the current one. A run records its judge the same
+  way, in `scores["_judge_label"]`: `Report#to_h` and `#to_markdown` now
+  name the judge a rebuilt run was given instead of reporting "No judge" for
+  every run reconstructed from the database. Results saved before this
+  release carry no snapshot and still render from the current catalog, and a
+  link to a saved report (`?evaluation=:id&run=:run_id`) now opens its
+  evaluation even when it is no longer on the first page of the index.
+  (#414)
+
+- **`actionagent`: an observed agent cannot be made executable.** An agent
+  discovered from telemetry has no configuration to run, and `execute` and
+  `test` refused one — but `update` and `restore` did not, so an observed
+  record could be flipped to `active`, given instructions and then run; and
+  a run queued against an agent that became observed afterwards still
+  reached a provider when its job came up. The refusal now covers `update`
+  and `restore` as well, and it is enforced under the API rather than only
+  in front of it: `Agent#execute`, `#test_execute` and
+  `AgentExecutionService#call` raise
+  `ActionAgent::Agent::ObservedAgentError`, so that queued job fails its run
+  without a provider call or a trace. Duplicating the agent still gives you
+  an executable copy, and an evaluation whose host explicitly resolves an
+  adapter for it remains the one path that replays an observed agent's
+  scenarios. (#414)
+
+- **The OpenAI Responses API keeps images and documents on a message with a
+  role.** `{ role: "user", text: "…", image: "…" }` — the shorthand the Chat
+  API and Anthropic transforms accept, and the only provider-neutral way to
+  send history followed by a multimodal turn — lost its `image:` or
+  `document:` on the provider the framework defaults to, because the
+  Responses transform kept only `content` from a role-bearing hash. It now
+  builds `input_text` / `input_image` / `input_file` parts for it, and a
+  media-only `{ role: "user", image: "…" }` becomes a message with one part.
+  The shorthand keys always come off the message, so a hash that carries
+  `content` *and* `image:` no longer sends `image` as an unknown parameter,
+  and a blank `image:`/`document:` contributes no part rather than an empty
+  one. A nil `document:` alongside a role was the unknown-parameter case; the
+  crash needed the role-less `{ document: nil }` inside a content array, which
+  called `start_with?` on nil.
+
+- **`actionagent`: a dashboard run's trace is attributed to the agent that
+  ran it.** Every locally stored run used to register an "observed" twin of
+  its own agent, because a run's class and action match no authored record.
+  The service that ran the agent now names it when it records the trace.
+  It is named by that caller and never read from the payload: resource
+  attributes are whatever the reporter sent, and single-tenant ingest is
+  unauthenticated unless `ActionAgent.ingest_api_key` is set, so an id taken
+  from there would let any reporter bind its traces to any authored agent by
+  guessing a primary key. A host that swaps in its own `trace_model` should
+  add the `agent:` keyword to its `create_from_payload`; without it the
+  dashboard logs the error and records no trace for its own runs. (#405)
+
+- **`actionagent`: an observed agent's history cannot be authored.** The
+  runner's conversation workbench writes an agent's history without running
+  it, and those two endpoints — starting a conversation, and seeding, editing
+  or deleting a turn — did not answer to the read-only rule execution does.
+  A turn typed into a telemetry mirror would be a fabrication attributed to
+  an agent whose whole point is that it only reports what really happened.
+  Both refuse an observed agent now, with the same message and status
+  `execute` gives. Reading that history is unchanged. (#405)
+
+### Note on upgrading from 1.4.0
+
+A scenario suite that passed on 1.4.0 can fail on this release with nothing
+about your agent, your models or your suite having changed. Nothing has
+regressed: the numbers those runs passed on were wrong, and this release
+stops averaging them away.
+
+A scenario's score was the mean of every criterion scored for it, and the
+judge's `task_completion` grade — its answer to "did this actually do what
+was asked" — was one term in that mean, alongside the rule checks. An answer
+that called the expected tool, called it successfully, and contained the
+expected string scored 1.0, 1.0 and 0.2 for a mean of 0.73, and passed at
+the default threshold of 0.7: the mechanics carried the answer. That is the
+wrong answer to the question an evaluation exists to ask. The agent called
+`lookup_order`, said "ABC-123", and still never told the customer where the
+order was — and the suite went green.
+
+From this release `task_completion` has to clear `threshold` on its own.
+Expect the first run after upgrading to show fewer passes than the run
+before it, concentrated in the scenarios whose answers were thin, evasive or
+wrong while their mechanics were right. Each of those now carries the
+`low_quality` fault with a summary naming the grade that failed — "Task
+completion scored 0.2 against a pass threshold of 0.7" — and the judge's
+recommendation for it, and each model's summary reports
+`avg_task_completion` beside `avg_score`, so a drop in pass rate can be read
+against the grade that caused it. Nothing else about scoring moved: a
+scenario the judge could not grade still falls back to its rule scores
+unless you opt into `require_judge_scores: true`, and a suite meant to be
+scored on mechanics alone can run without a judge or at a lower `threshold`.
+Read that first run as a new baseline rather than a regression — it is
+measuring something the runs before it were not.
+
+## [1.4.0] - 2026-09-09
+
+Releases `activeagent` 1.4.0 and `actionagent` 1.3.0 from one tag.
+
+### Added
+
+- **`ActiveAgent::Evals`, the evaluation core, in the framework.** Pasted-list
+  and YAML suite parsing, model resolution, rule and expectation scoring, the
+  fault taxonomy with its recommendations, the optional judge, and the
+  per-model report live in `lib/active_agent/evals`, loadable on their own
+  with `require "active_agent/evals"`. Any app can replay a list of tasks
+  across models against its own agent through one `replay` callable and get
+  the same faults, recommendations and verdict the dashboard shows;
+  `actionagent` keeps only what the dashboard adds — persistence, the job,
+  the API and the UI.
+- **Scenario evaluations in the dashboard.** An evaluation can now carry a
+  suite of scenarios — a pasted list of user messages, grouped with
+  `# Heading` lines and annotated with the tool each should call — and a run
+  replays every selected scenario through the agent once per candidate model
+  (`compare_models`, or a per-run `models` selection) instead of sampling
+  recorded generations. Each scenario × model result records the answer, the
+  tools called, its score, and, when it falls short, one fault
+  (`run_error`, `tool_error`, `missing_capability`,
+  `expected_tool_not_called`, `forbidden_content`, `missing_content`,
+  `low_quality`) with a recommendation; a configured judge model refines the
+  recommendation with the tool to add or the instruction to change. Runs can
+  be narrowed to a group or to single scenarios, and the run summary ranks
+  the models by pass rate with a verdict. New tables
+  `evaluation_scenarios` and `evaluation_scenario_results` ship in
+  `create_active_agent_evaluation_scenarios`, which
+  `rails generate action_agent:install` emits for new and existing installs.
+- **`ActionAgent.mcp_catalog`.** A host app registers the MCP servers it
+  serves or connects itself — `[{ key:, name:, tool_hints: [...] }, …]` —
+  and they join the built-in catalog: listed in the MCP Services view, with
+  telemetry traffic for their bare tool names attributed to them.
+  `MCPCatalog.keys` lists built-ins and registrations together;
+  `MCPCatalog::BY_KEY` still holds the built-ins alone.
+- **Suite results rebuilt around what to do next.** An expanded scenario
+  suite used to be a summary and a matrix; it now opens on the three
+  questions a run is actually asked. **Runs** numbers every run of the suite
+  from the oldest and scores it against the one before — `+3 passed vs #7`,
+  or `partial run` when the two covered different scenarios or models and the
+  numbers do not compare — so progress is legible without reading two runs
+  side by side; selecting an older run re-derives the models, the fix list,
+  the matrix and every drill-down, and the collapsed header keeps reporting
+  the latest. **Models** marks the best candidate with an info-toned
+  `judge's pick` badge and the verdict that justifies it, rather than a green
+  *winner* — losing a comparison by one scenario is not a failing grade.
+  **What to fix** turns each fault into a card with the tools involved
+  (deduplicated to one chip each), the MCP server that serves them, whether
+  this agent has it enabled, and a button that deep-links to MCP Services,
+  Tools or the agent's instructions: the fix, not just the finding. The
+  scenario × model matrix shows the tools each model actually called against
+  the tools the scenario expected, coloured by whether they match, and a row
+  opens onto every model's answer, timing, cost and diagnosis.
+- **The evaluation report is a designed page.** `Report#to_html(theme:)`
+  renders a run on the dashboard's design system — stat tiles, a panel per
+  model with the judge's pick and verdict, the what-to-fix cards, the
+  scenario × model matrix and a disclosure per scenario — still one
+  self-contained file with inline styles and no external assets, so it
+  archives next to a CI run. `theme:` pins `"light"` or `"dark"`; without it
+  the page follows the viewer's `prefers-color-scheme`, and the dashboard
+  passes its own theme through when it frames the report at
+  `/api/evaluations/:id/runs/:run_id/report`. The new `Report#fix_items`
+  builds the what-to-fix list — faults grouped with the tools each implicates
+  and the action that addresses it — for the page, the engine's API and any
+  app that wants the backlog as JSON; `tool_resolver:`, `agent_name:` and
+  `links:` on `Report.new` let a host name the MCP server behind a tool, the
+  agent, and the routes an action should point at, so a CI job gets the same
+  cards the dashboard shows.
+- **An APM-style service overview on the Metrics page.** The page answered
+  "how much traffic in the last 24 hours"; it now answers "is this healthy
+  right now, and since when". A `1h` / `24h` / `7d` range fixes the bucket
+  size the whole page is drawn at (60 × 1 min, 96 × 15 min, 84 × 2 h); five
+  golden signals — requests, latency, error rate, tokens, cost — carry a
+  sparkline and a delta against the period just before the window; six panels
+  plot requests stacked by agent, latency percentiles, errors by class,
+  tokens, spend and tool calls, with markers for the agent versions deployed
+  inside the window and for an error spike when one stands out; and a rail
+  ranks the agents, models, slowest actions, tools and error classes behind
+  them. Filtering to an agent — from the select, or by clicking its rail
+  row — narrows every one of those together. `GET /api/metrics` gains
+  `range` and `agent` params and the keys that feed it (`totals`, `deltas`,
+  `series`, `agents`, `models`, `actions`, `tools`, `errors_by_type`,
+  `markers`) from the new `ActionAgent::MetricsReport`: one pass over the
+  window, with bucketing, nearest-rank percentiles and error classification
+  done in Ruby so PostgreSQL and SQLite report the same numbers. Every
+  earlier key and param still means what it did.
+- **A design token layer under the dashboard.** Colors, fonts and the type
+  scale live in `actionagent/frontend/tokens.css` as CSS variables scoped to
+  the mounted dashboard (`.aa-dashboard`, with the dark palette under
+  `.theme-dark`), and the views draw from a set of shared primitives —
+  badges, chips, panels, cards, pass bars, stat tiles, segmented controls —
+  instead of each restating the same hex codes and paddings. Dark mode is
+  then one class rather than a conditional at every call site, and a host
+  app's own stylesheet cannot bleed into the engine's. The framework carries
+  the same values in `ActiveAgent::Evals::DesignTokens` so the standalone
+  HTML report matches the dashboard it came from, with a test that fails when
+  the two drift apart.
+
+- **RubyLLM backend pinning via `platform:`.** RubyLLM resolves which of its
+  providers serves a request from the model ID, and a model served by more
+  than one — `gemini-2.5-flash` exists on both the Gemini API and Vertex
+  AI — lands on whichever RubyLLM's registry prefers, with no way to say
+  otherwise from ActiveAgent. The new `platform:` option
+  (`generate_with :ruby_llm, model: "gemini-2.5-flash", platform: :vertexai`)
+  forwards to RubyLLM's `provider:` and pins the backend, for embeddings as
+  well as prompts. It is not named `provider:` because a provider reference
+  is already the first argument to `generate_with`. Omitting it keeps
+  model-based routing unchanged. (#373)
+
+### Fixed
+
+- **A run report is readable in the dashboard.** The report was framed at a
+  fixed viewport height, so everything past the first screen — including
+  every fix item — sat behind a nested scrollbar. The frame is sized to the
+  report's own content, and a fix action targets the top window so it
+  navigates the dashboard instead of loading it into the frame. (#410, #411)
+
+- **Provider credentials store on a host that skipped `db:encryption:init`.**
+  Encryption keys derived from `secret_key_base` were installed after Rails
+  had already configured `ActiveRecord::Encryption`, so the config read back
+  correct while every credential write raised `Errors::Configuration` — in
+  the dashboard, the Settings API Keys tab failed to render and provider
+  keys failed to save. (#412)
+
+- **`service: "RubyLLM"` loads when the ruby_llm railtie has run.** The
+  ruby_llm gem registers `RubyLLM` as an inflector acronym in Rails apps,
+  which turns `"RubyLLM".underscore` into `rubyllm` — so provider loading
+  required a nonexistent `rubyllm_provider.rb` and failed with
+  `cannot load such file`. An alias file now covers that require path, the
+  same fix `openai_provider.rb` applies for `OpenAI`. (#371, fixed in #372
+  by @aoki-ryusei; regression tests in #374)
+
 ## [1.3.1] - 2026-08-19
 
 ### Fixed

data/lib/active_agent/evals/design_tokens.rb

@@ -0,0 +1,130 @@
+# frozen_string_literal: true
+
+module ActiveAgent
+  module Evals
+    # The dashboard's design tokens as Ruby, so the self-contained HTML report
+    # paints with the same palette as the mounted dashboard without loading
+    # its stylesheet.
+    #
+    # Mirrors actionagent/frontend/tokens.css: LIGHT is the `.aa-dashboard`
+    # block, DARK the overrides in `.aa-dashboard.theme-dark`. A test asserts
+    # the two files stay in step — change a value in both places.
+    module DesignTokens
+      FONT_TEXT = '"Inter Variable", Inter, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif'
+      FONT_MONO = '"JetBrains Mono", "SF Mono", "Fira Code", Menlo, Consolas, "Courier New", monospace'
+
+      LIGHT = {
+        # Brand
+        "--color-accent" => "#FA343B",
+        "--color-accent-hover" => "#E02D33",
+        "--color-accent-b" => "#FA343B29",
+        "--color-accent-b-hover" => "#FA343B33",
+        "--color-on-accent" => "#ffffff",
+        "--color-accent-ui" => "#ef4444",
+        "--color-accent-ui-hover" => "#dc2626",
+        "--color-accent-ui-muted" => "rgba(239, 68, 68, 0.15)",
+        "--color-accent-ui-tint" => "#fef2f2",
+
+        # Surfaces
+        "--color-background" => "#f9fafb",
+        "--color-background-page" => "#ffffff",
+        "--color-surface" => "#ffffff",
+        "--color-card" => "#ffffff",
+        "--color-muted" => "#f3f4f6",
+        "--color-hover" => "#f3f4f6",
+        "--color-background-blur" => "rgba(255,255,255,0.9)",
+
+        # Borders
+        "--color-border" => "#e5e7eb",
+        "--color-border-light" => "#f3f4f6",
+        "--color-border-strong" => "#d1d5db",
+
+        # Text
+        "--color-text-primary" => "#111827",
+        "--color-text-secondary" => "#6b7280",
+        "--color-text-muted" => "#9ca3af",
+        "--color-text-cell" => "#4b5563",
+
+        # Semantic status
+        "--color-success" => "#16a34a",
+        "--color-success-soft" => "#dcfce7",
+        "--color-success-text" => "#166534",
+        "--color-warning" => "#eab308",
+        "--color-warning-soft" => "#fef9c3",
+        "--color-warning-text" => "#854d0e",
+        "--color-error" => "#dc2626",
+        "--color-error-soft" => "#fee2e2",
+        "--color-error-text" => "#991b1b",
+        "--color-info" => "#3b82f6",
+        "--color-info-soft" => "#dbeafe",
+        "--color-info-text" => "#1e40af",
+
+        # Trace span colors (observability)
+        "--span-root" => "#9ca3af",
+        "--span-prompt" => "#60a5fa",
+        "--span-generate" => "#a855f7",
+        "--span-llm" => "#ef4444",
+        "--span-thinking" => "#fbbf24",
+        "--span-tool" => "#22c55e",
+        "--span-response" => "#2dd4bf",
+
+        # Token flow colors
+        "--color-token-in" => "#2563eb",
+        "--color-token-out" => "#7c3aed",
+
+        # Chart / agent palette
+        "--chart-1" => "#6366f1",
+        "--chart-2" => "#10b981",
+        "--chart-3" => "#f59e0b",
+        "--chart-4" => "#ec4899",
+        "--chart-5" => "#3b82f6",
+
+        # Type
+        "--font-text" => FONT_TEXT,
+        "--font-mono" => FONT_MONO
+      }.freeze
+
+      DARK = {
+        "--color-accent-ui-tint" => "rgba(239, 68, 68, 0.15)",
+
+        "--color-background" => "#0f0f0f",
+        "--color-background-page" => "#0f0f0f",
+        "--color-surface" => "#1a1a1a",
+        "--color-card" => "rgba(255,255,255,0.05)",
+        "--color-muted" => "rgba(255,255,255,0.05)",
+        "--color-hover" => "#252525",
+        "--color-background-blur" => "rgba(15,15,15,0.9)",
+
+        "--color-border" => "rgba(255,255,255,0.1)",
+        "--color-border-light" => "rgba(255,255,255,0.05)",
+        "--color-border-strong" => "rgba(255,255,255,0.2)",
+
+        "--color-text-primary" => "#ffffff",
+        "--color-text-secondary" => "rgba(255,255,255,0.6)",
+        "--color-text-muted" => "rgba(255,255,255,0.4)",
+        "--color-text-cell" => "rgba(255,255,255,0.7)",
+
+        "--color-success-soft" => "rgba(22,163,74,0.15)",
+        "--color-success-text" => "#4ade80",
+        "--color-warning-soft" => "rgba(234,179,8,0.15)",
+        "--color-warning-text" => "#facc15",
+        "--color-error-soft" => "rgba(220,38,38,0.15)",
+        "--color-error-text" => "#f87171",
+        "--color-info-soft" => "rgba(59,130,246,0.15)",
+        "--color-info-text" => "#93c5fd"
+      }.freeze
+
+      # One CSS rule declaring `tokens` as custom properties on `scope`, with
+      # an optional `color-scheme` so form controls and scrollbars follow:
+      #
+      #   DesignTokens.css(scope: ":root")                                        # light
+      #   DesignTokens.css(scope: ":root.theme-dark", tokens: DesignTokens::DARK,
+      #                    color_scheme: "dark")                                  # dark overrides
+      def self.css(scope:, tokens: LIGHT, color_scheme: nil)
+        declarations = tokens.map { |name, value| "  #{name}: #{value};" }
+        declarations.unshift("  color-scheme: #{color_scheme};") if color_scheme
+        "#{scope} {\n#{declarations.join("\n")}\n}"
+      end
+    end
+  end
+end