activeagent 1.4.0 → 1.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9cf80ba8d14c538e4f9193bf423cb82c0d50223362547df2bd3bfb2c66c62f86
-  data.tar.gz: 9f3bb57f1f22de2ee5c74350d601b459f4864a9daf2aa7198900ad0f3e44589a
+  metadata.gz: 73e63c87b700b6dec4e3410ae0547ff1c02714fbe79b24175447fb5638a9fe3b
+  data.tar.gz: ef361dc3bb7dcab252c0fadf1fb526e689b97b21386a2d50b6ba6eb96a84100a
 SHA512:
-  metadata.gz: 6f817eeba3d15db4b49e924457aa510d0c4623031994e96a72ffb566aa4b800c43209682333f90aa06dc49915e03fb640f03aed7a7d8b23bae30c8cc2e945d17
-  data.tar.gz: 788b6ad2472a4faf7e8a81bfc36452268f14c7d4958af283d937417686ebd9c652c9f0c43504d16c6eb07e5852a9b2b158dd0b2b4a3038e48296efbeb2025e58
+  metadata.gz: 7ed21c4a284e17d0014ef83b09fc56741b96cbb83f219f38dc3e8ad87f511f9ed5b3d902e1b1e9c7e4db11c83bce07ed0effbb2f44d85c35d0da4738598744b3
+  data.tar.gz: c3634cfbe8e532889e2ef82b7080b30f0233a0faf1e81b5c37995949a8c7a6d5e6ebf9e3800e39457c54de3803d22b3545d4606d448867a777915cabf1a39b15

data/CHANGELOG.md

@@ -7,6 +7,370 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.5.2] - 2026-09-11
+
+Releases `activeagent` and `actionagent` 1.5.2 from one tag.
+
+**1.5.1 was never tagged.** Its version bump reached `main`, but three PRs
+that change `actionagent` merged alongside it, and that release deliberately
+held `actionagent` at 1.5.0 — publishing it would have shipped the fix below
+while leaving every dashboard change of this cycle unpublished, because
+`release.yml` skips a version already on RubyGems. 1.5.2 supersedes it and
+carries both gems. The 1.5.1 notes are kept below as the record of what that
+bump contained.
+
+### Added
+
+- **Schema-derived agent tools.** `ActiveAgent::SchemaTools` turns an
+  ActiveRecord model plus a declared boundary into a bounded, enumerable tool
+  roster — `find_*`, `count_*`, `get_*` — with `filterable` and `returns`
+  allowlists. An undeclared column is rejected rather than silently dropped:
+  ignoring an unknown filter answers a broader question than was asked while
+  still looking like success. Results are capped (25 default, 100 max) with a
+  truncation marker the model can see. (#435)
+
+- **Host tools reach the dashboard.** `ActionAgent.schema_tools` offers each
+  generated tool beside `AgentToolbox`'s built-ins: individually selectable in
+  the agent editor, dispatched by name at execution, and nameable in an
+  evaluation's `tools:` expectation. Previously a declared schema tool was
+  invisible — `definitions_for` returned nothing, the model received no
+  schemas and invented tool names in prose while the run scored 0.0 for what
+  looked like a model failure. (#435, closes #438)
+
+- **Tools are discovered, not declared twice.** Leave `schema_tools` unset and
+  every subclass under `schema_tools_path` (`app/agent_tools`) is offered.
+  Adding a tool is adding a file. Anonymous classes are excluded from
+  discovery: a runtime-built class cannot supersede itself, so it would
+  accumulate one per reload. (#435, refs #440)
+
+- **`scope_by_policy`** resolves a model's policy by name —
+  `Reservation` → `ReservationPolicy::Scope` — instead of hand-writing the
+  block. Opt-in, because silently scoping a class that declared none would
+  change what an existing tool returns; a missing policy raises at declaration
+  rather than quietly reading the whole table. (#435)
+
+- **A model's agent starts with that model's tools.** `ReservationAgent` is
+  seeded from `ReservationTools` on create. A default, never a restriction:
+  any agent may enable any tool, and an explicit selection — including a
+  deliberate empty one — is never overwritten. (#435)
+
+### Fixed
+
+- **MCP tool-discovery failures are reported instead of running tool-less and
+  silent.** `MCPToolDispatcher#tool_definitions` rescued a failed `tools/list`
+  to `[]`, so a server that 401s and one that legitimately serves no tools
+  were indistinguishable: the agent ran without tools, the model fabricated,
+  and the report offered prompt advice for what was a transport failure.
+  `discovery_errors` now names the server, its URL and the underlying error,
+  and `all_servers_failed?` lets a caller fail loudly rather than grade an
+  invented answer. (#434, closes #425)
+
+- **Nil VCR filters no longer flake replays**, and the MCP plural is spelled
+  correctly. (#436)
+
+## [1.5.1] - 2026-09-11 [UNRELEASED — superseded by 1.5.2]
+
+Bumped `activeagent` to 1.5.1 and held `actionagent` at 1.5.0. Never tagged;
+its contents ship in 1.5.2.
+
+### Fixed
+
+- **A spec hash names its model rather than reaching the provider as an
+  inspected Hash.** `Evals::ModelSpec.parse_all` called `to_s` on each value,
+  so a Hash travelled as the model ID and the provider answered
+  `{"label" => "openrouter/openai/gpt-4o-mini", ...} is not a valid model ID`
+  — every scenario of the run failing before it reached the model. A caller
+  passing a plain string was unaffected, which is why a single run worked
+  while a whole suite failed. The path is reachable by design rather than by
+  misuse: a run persists its models as `specs.map(&:to_h)`, so re-running that
+  selection hands the hashes back. `parse_all` now reads a hash's `label`,
+  then its `model`, and leaves strings alone.
+
+## [1.5.0] - 2026-09-10
+
+Releases `activeagent` 1.5.0 and `actionagent` 1.5.0 from one tag.
+
+`actionagent` goes from 1.3.0 to 1.5.0, skipping 1.4: the two gems are
+released together from this repository and from one tag, and carrying one
+version number across both is less confusing than explaining which
+dashboard version pairs with which framework. `actionagent` 1.4 does not
+exist and never will. The engine's floor on the framework
+(`activeagent >= 1.4`) is unchanged and still correct.
+
+### Added
+
+- **`ActiveAgent::Evals::Publisher` delivers a finished report to a
+  collector.** A run that already happened — in CI, in a host app's own
+  runtime, anywhere the evaluation core runs — can be sent to an
+  ActiveAgents-compatible collector without replaying the agent:
+  `Publisher.new(api_key:, endpoint:).call(report:, run_id:, source:,
+  agent_name:, suite:)` posts a version-1 envelope wrapping `Report#to_h`
+  (or the saved JSON hash of an earlier run) and returns the collector's
+  receipt. Delivery is synchronous, requires HTTPS outside loopback, does
+  not follow a redirect carrying the bearer credential, caps a request at
+  2 MiB, and raises `Publisher::Error` on anything but a receipt naming the
+  same `run_id` — so a retry with that same `run_id` and the saved report
+  re-delivers rather than re-runs. Publication is strictly opt-in and
+  happens only where an application writes the call: no configuration flag,
+  no callback, no default credential, and `api_key:` supplied explicitly at
+  the call site. That is deliberate, because the payload is the report
+  itself — every scenario's prompt, the agent's answers, and the tool calls
+  and their results — and whether that may leave the application is the
+  application's decision to make. Installing the gem sends nothing
+  anywhere. `docs/evals/publication.md` documents the envelope, the receipt
+  and the retry rules. (#414)
+
+- **`Runner` takes `around_evaluation:` and `require_judge_scores:`.**
+  `around_evaluation:` is called with `(scenario, spec)` and a block, and
+  wraps the whole evaluation — the replay, the scoring, the judge calls
+  behind a recommendation — so a host can establish one trace context
+  across all of it and correlate a replay with the judging it triggered. It
+  must return the block's result; `on_result` runs after it returns, an
+  error it raises propagates to the caller, and `#evaluate` called directly
+  bypasses it, for a host doing its own scheduling. `require_judge_scores:`
+  (default `false`) settles what an unusable judge means. A judge that
+  raises or answers unscorably is skipped, and the scenario is then decided
+  on its rule scores alone — which reads as "the agent passed" when the
+  truth is "nobody graded the answer". Set it, and an otherwise passing
+  result whose `task_completion` or declared `llm_judge` criterion has no
+  usable score fails instead, with the new `judge_unavailable` fault naming
+  the unscored criteria and pointing at the judge's credentials, model and
+  JSON reply. A run with no judged criteria is unaffected. (#414)
+
+- **A grouped suite imports as YAML or JSON, whole.** `ScenarioParser` read
+  a pasted list or a JSON array of scenarios; it now also reads the grouped
+  document `Suite` loads — `groups:` with per-group keys and display names,
+  scenarios carrying `key`, `prompt`, `notes`, `expect` and
+  `production_only` — from YAML or JSON, keeping every part of it.
+  `ScenarioParser.parse` and `.scenarios` gain `include_production_only:`,
+  which defaults to `true` to match `Suite`. The dashboard defaults it the
+  other way: post the document as `scenarios_text` and the production-only
+  questions stay out unless `include_production_only` is sent alongside it,
+  because those prompts run against a live agent. The choice is made at
+  import — the engine stores the scenarios it selected, not the source
+  document — so changing it means importing the document again. (#414)
+
+- **`actionagent`: `ActionAgent.scenario_evaluation_adapter_resolver`.** A
+  host application with its own agent runtime can now run an evaluation
+  itself while keeping the dashboard's catalog, selection, jobs, result
+  persistence and report pages. The resolver is called with the persisted
+  evaluation and returns `nil` for the engine's normal `Agent#test_execute`
+  path, or a callable — `evaluation:`, `owner:`, `scenarios:`, `models:`,
+  `on_result:` — that runs the host's own agent and judge, yields every
+  result as it lands, and returns an `ActiveAgent::Evals::Report`. The
+  engine holds it to that contract: anything other than a `Report`, or a
+  report that omits or duplicates one of the selected scenario × model
+  pairs, fails the run rather than completing it with rows missing, and an
+  exception leaves the results already written in place. Dashboard
+  authentication, execution enablement and the host's execution quota still
+  apply. (#414)
+
+- **`actionagent`: the Run Agent page is a conversation workbench.** Testing
+  an agent used to mean one prompt in, one output out, with no way to see —
+  or shape — what the model was given. The page now works the way a user
+  would work the agent: it pins a persisted conversation (a solid_agent
+  context) and every run sends that conversation's user and assistant turns
+  ahead of the new message, so follow-up questions actually follow up. The
+  context is editable in place — edit or delete a turn, seed a user or
+  assistant message without running, start a new conversation — and every
+  run is a fresh `AgentRun` with its own trace, so Traces and Interactions
+  see exactly what the model saw. Files attach to a message and ride along
+  through Active Storage (`AgentRun has_many_attached :attachments`, guarded
+  for hosts without it): images reach the model as vision input, PDFs as
+  documents, and text-like files (CSV, Markdown, JSON, plain text) are
+  inlined into the message; the persisted user message keeps an attachment
+  manifest so the conversation shows thumbnails afterwards. Assistant replies
+  can render **generative UI** — cards, stats, tables, charts, lists,
+  progress, forms, choice buttons, images, callouts and code — from a fenced
+  ```` ```ui ```` JSON block in prose, a JSON reply whose top level is
+  `ui`/`blocks`, or the new `render_ui` tool (enable the **Generative UI**
+  tool on the agent). Forms and choices post their answer back into the
+  conversation as the next user message. New engine API: `GET/POST
+  /api/agents/:id/conversations`, message create/update/delete under
+  `/api/interactions/:id/messages`, multipart `POST /api/agents/:id/execute`
+  with `attachments[]` and `params[context_id]`, and attachment metadata on
+  run and message JSON. The reference host (`test/dummy`) gained the Active
+  Storage tables so the attachment path is exercised by the engine's tests.
+  Two notes for anyone driving that API directly: `execute`/`test` now answer
+  422 unless the request carries a prompt or a file, and per-run overrides in
+  `params` can no longer name `attachments` or `action` — those stay the
+  controller's to set. Model-supplied images in generative UI load on sight
+  only when they are inline data or this app's own URL; any other host is
+  offered as a click-to-load, since fetching one tells that host whatever the
+  model put in the URL.
+
+### Fixed
+
+- **A scenario passes only if it completed the task.** A scenario's verdict
+  was the mean of everything scored for it, and the judge's
+  `task_completion` grade was one number in that mean: an answer that
+  called the expected tool and contained the expected string could carry a
+  task grade of 0.2 to a mean of 0.73 and pass at the default threshold of
+  0.7. `task_completion` is a gate now — it has to reach `threshold` on its
+  own, and no number of passing tool and content checks can lift it — and
+  the fault names the number that failed: "Task completion scored 0.2
+  against a pass threshold of 0.7". An evaluation that configures its own
+  `llm_judge` criteria rather than relying on the implicit grade — which is
+  what the dashboard does — is gated the same way, on the mean of those
+  grades, so one soft dimension among strong ones still passes while an
+  answer the judge marked down cannot be carried by its mechanics. A
+  scenario the judge could not grade at all is unchanged, still falling
+  back to the rule scores. `score` and `avg_score` still mean the aggregate
+  they always did, and each model's
+  summary gains `avg_task_completion` so the judge's grade reads separately
+  from it. **This can turn a suite that passed on 1.4.0 red; see the note
+  on upgrading below.** (#414)
+
+- **A judge's score is read as a JSON number.** The score was pulled out of
+  the judge's reply by regular expression, matching the first run of digits
+  after `"score":`. It read `{"score": 9e-2}` — 0.09 — as 9, clamped to a
+  perfect 1.0; it read the string `{"score": "0.9"}` and the truncated
+  `{"score": 0.9oops}` as a confident 0.9 rather than as unusable. The
+  score now comes from the parsed JSON object and has to be a finite
+  number, so exponent notation is read as written and a string, a boolean,
+  `null`, `NaN` or `1e999` is unscorable — which the runner already knows
+  how to handle. Fenced ```` ```json ```` replies still parse. The
+  dashboard's generation-sampling evaluations score through the engine's own
+  judge rather than the framework's, and read a score by the same rule now,
+  so the two halves of the dashboard no longer disagree about the same
+  reply. (#414)
+
+- **A judge that answers with the wrong types cannot put junk in the fix
+  list.** `suggested_tool` and `instruction_change` were coerced rather
+  than checked, so a reply of `"suggested_tool": {"name": true}` added a
+  tool literally named `true` to the report's suggested tools, and
+  `"instruction_change": ["invalid"]` became a fix card asking someone to
+  add `["invalid"]` to the agent's instructions. Both fields must now be
+  nonempty strings and are dropped when they are not, so a malformed reply
+  loses only the malformed part: the judge's recommendation still reaches
+  the result, the report and every rendering of it. (#414)
+
+- **A grouped suite pasted into the dashboard keeps its keys and
+  expectations.** Only a pasted list or JSON was recognised, so a YAML suite
+  went to the line parser and was read as prose: a document describing three
+  scenarios became eighteen, with prompts like `tools: [lookup_order]` and
+  `production_only: true`, groups named `expect`, generated keys in place of
+  the document's own, and every expectation dropped — a suite that looked
+  imported and scored nothing real. Such a document is now parsed as the
+  suite it is, and one that is not valid, or a selection that matches no
+  scenarios, returns an import error (`ScenarioParser::ParseError`, HTTP
+  422) instead of a suite of nonsense or a sampling evaluation nobody asked
+  for. (#414)
+
+- **`actionagent`: run and result metadata survive persistence.** A run
+  rebuilt from the database was rebuilt without it: `Report#metadata` came
+  back holding only the four keys the engine writes itself, and each
+  result's replay metadata was gone entirely, so a host's own run and result
+  IDs, response trace IDs and judge trace IDs did not survive the round trip
+  and its reports could not be joined to its telemetry. Run metadata is now
+  kept in `scores["_metadata"]` and per-result metadata in
+  `diagnosis["_replay_metadata"]`, restored by `EvaluationRun#to_report` and
+  served as `metadata` on result JSON. Both are reserved storage keys that
+  the public diagnosis excludes, so nothing migrates and `diagnosis` still
+  means what it did. (#414)
+
+- **`actionagent`: refreshing a catalog does not rewrite what an earlier run
+  asked.** A saved run rendered its scenarios from the catalog rows as they
+  are now, so rewording a question, retagging its group or changing its
+  expectations silently rewrote history — last month's report showed this
+  month's prompt above last month's answers, and judged them against
+  expectations that were not in force when they were given. Each result now
+  records the scenario it was actually evaluated against in
+  `diagnosis["_scenario_snapshot"]`, and the report, the API and the
+  scenario matrix read that snapshot, in the order the run itself used, with
+  the dashboard noting on a scenario whose catalog entry has since changed
+  that re-running uses the current one. A run records its judge the same
+  way, in `scores["_judge_label"]`: `Report#to_h` and `#to_markdown` now
+  name the judge a rebuilt run was given instead of reporting "No judge" for
+  every run reconstructed from the database. Results saved before this
+  release carry no snapshot and still render from the current catalog, and a
+  link to a saved report (`?evaluation=:id&run=:run_id`) now opens its
+  evaluation even when it is no longer on the first page of the index.
+  (#414)
+
+- **`actionagent`: an observed agent cannot be made executable.** An agent
+  discovered from telemetry has no configuration to run, and `execute` and
+  `test` refused one — but `update` and `restore` did not, so an observed
+  record could be flipped to `active`, given instructions and then run; and
+  a run queued against an agent that became observed afterwards still
+  reached a provider when its job came up. The refusal now covers `update`
+  and `restore` as well, and it is enforced under the API rather than only
+  in front of it: `Agent#execute`, `#test_execute` and
+  `AgentExecutionService#call` raise
+  `ActionAgent::Agent::ObservedAgentError`, so that queued job fails its run
+  without a provider call or a trace. Duplicating the agent still gives you
+  an executable copy, and an evaluation whose host explicitly resolves an
+  adapter for it remains the one path that replays an observed agent's
+  scenarios. (#414)
+
+- **The OpenAI Responses API keeps images and documents on a message with a
+  role.** `{ role: "user", text: "…", image: "…" }` — the shorthand the Chat
+  API and Anthropic transforms accept, and the only provider-neutral way to
+  send history followed by a multimodal turn — lost its `image:` or
+  `document:` on the provider the framework defaults to, because the
+  Responses transform kept only `content` from a role-bearing hash. It now
+  builds `input_text` / `input_image` / `input_file` parts for it, and a
+  media-only `{ role: "user", image: "…" }` becomes a message with one part.
+  The shorthand keys always come off the message, so a hash that carries
+  `content` *and* `image:` no longer sends `image` as an unknown parameter,
+  and a blank `image:`/`document:` contributes no part rather than an empty
+  one. A nil `document:` alongside a role was the unknown-parameter case; the
+  crash needed the role-less `{ document: nil }` inside a content array, which
+  called `start_with?` on nil.
+
+- **`actionagent`: a dashboard run's trace is attributed to the agent that
+  ran it.** Every locally stored run used to register an "observed" twin of
+  its own agent, because a run's class and action match no authored record.
+  The service that ran the agent now names it when it records the trace.
+  It is named by that caller and never read from the payload: resource
+  attributes are whatever the reporter sent, and single-tenant ingest is
+  unauthenticated unless `ActionAgent.ingest_api_key` is set, so an id taken
+  from there would let any reporter bind its traces to any authored agent by
+  guessing a primary key. A host that swaps in its own `trace_model` should
+  add the `agent:` keyword to its `create_from_payload`; without it the
+  dashboard logs the error and records no trace for its own runs. (#405)
+
+- **`actionagent`: an observed agent's history cannot be authored.** The
+  runner's conversation workbench writes an agent's history without running
+  it, and those two endpoints — starting a conversation, and seeding, editing
+  or deleting a turn — did not answer to the read-only rule execution does.
+  A turn typed into a telemetry mirror would be a fabrication attributed to
+  an agent whose whole point is that it only reports what really happened.
+  Both refuse an observed agent now, with the same message and status
+  `execute` gives. Reading that history is unchanged. (#405)
+
+### Note on upgrading from 1.4.0
+
+A scenario suite that passed on 1.4.0 can fail on this release with nothing
+about your agent, your models or your suite having changed. Nothing has
+regressed: the numbers those runs passed on were wrong, and this release
+stops averaging them away.
+
+A scenario's score was the mean of every criterion scored for it, and the
+judge's `task_completion` grade — its answer to "did this actually do what
+was asked" — was one term in that mean, alongside the rule checks. An answer
+that called the expected tool, called it successfully, and contained the
+expected string scored 1.0, 1.0 and 0.2 for a mean of 0.73, and passed at
+the default threshold of 0.7: the mechanics carried the answer. That is the
+wrong answer to the question an evaluation exists to ask. The agent called
+`lookup_order`, said "ABC-123", and still never told the customer where the
+order was — and the suite went green.
+
+From this release `task_completion` has to clear `threshold` on its own.
+Expect the first run after upgrading to show fewer passes than the run
+before it, concentrated in the scenarios whose answers were thin, evasive or
+wrong while their mechanics were right. Each of those now carries the
+`low_quality` fault with a summary naming the grade that failed — "Task
+completion scored 0.2 against a pass threshold of 0.7" — and the judge's
+recommendation for it, and each model's summary reports
+`avg_task_completion` beside `avg_score`, so a drop in pass rate can be read
+against the grade that caused it. Nothing else about scoring moved: a
+scenario the judge could not grade still falls back to its rule scores
+unless you opt into `require_judge_scores: true`, and a suite meant to be
+scored on mechanics alone can run without a judge or at a lower `threshold`.
+Read that first run as a new baseline rather than a regression — it is
+measuring something the runs before it were not.
+
 ## [1.4.0] - 2026-09-09
 
 Releases `activeagent` 1.4.0 and `actionagent` 1.3.0 from one tag.

data/lib/active_agent/evals/diagnosis.rb

@@ -22,7 +22,7 @@ module ActiveAgent
     class Diagnosis
       FAULTS = %w[
         run_error tool_error missing_capability expected_tool_not_called
-        forbidden_content missing_content low_quality
+        forbidden_content missing_content low_quality judge_unavailable
       ].freeze
 
       # Phrasings an agent uses when nothing in its toolset covers the task.
@@ -56,11 +56,15 @@ module ActiveAgent
       # @param available_tools [Array<String>] tool names the agent could call
       # @param threshold [Float] the pass threshold for `score`
       # @param agent_name [String] how the recommendations refer to the agent
-      def self.call(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent")
-        new(scenario:, replay:, scores:, score:, available_tools:, threshold:, agent_name:).call
+      # @param judge_keys [Array] keys in `scores` a judge graded the answer on
+      #   (llm_judge criteria); `task_completion` always counts as one
+      def self.call(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent",
+                    judge_keys: [])
+        new(scenario:, replay:, scores:, score:, available_tools:, threshold:, agent_name:, judge_keys:).call
       end
 
-      def initialize(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent")
+      def initialize(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent",
+                     judge_keys: [])
         @scenario = scenario
         @replay = replay
         @scores = scores || {}
@@ -68,6 +72,7 @@ module ActiveAgent
         @available_tools = Array(available_tools).map(&:to_s)
         @threshold = threshold
         @agent_name = agent_name
+        @judge_keys = Array(judge_keys) | [ "task_completion" ]
       end
 
       def call
@@ -212,11 +217,22 @@ module ActiveAgent
                "missing" => missing)
       end
 
+      # A judge grade — the implicit task_completion score, or the llm_judge
+      # criteria the evaluation configured — measures the answer itself, so
+      # its mean has to reach the threshold on its own. Rule checks (a tool
+      # was called, a phrase is present) cannot carry a badly graded answer.
       def low_quality
-        return nil if @score.nil? || @score >= @threshold
-
-        weakest = @scores.compact.min_by { |_, value| value }
-        summary = "Scored #{@score.round(2)} against a pass threshold of #{@threshold}"
+        grades = @scores.slice(*@judge_keys).compact
+        grade = grades.any? ? (grades.values.sum / grades.size).round(3) : nil
+        failed_grade = grade && grade < @threshold
+        return nil unless failed_grade || (@score && @score < @threshold)
+
+        weakest = (failed_grade ? grades : @scores.compact).min_by { |_, value| value }
+        summary = if failed_grade
+          "#{graded_label(grades)} scored #{grade.round(2)} against a pass threshold of #{@threshold}"
+        else
+          "Scored #{@score.round(2)} against a pass threshold of #{@threshold}"
+        end
         summary += ", weakest on #{weakest.first} (#{weakest.last.round(2)})" if weakest
         recommendation =
           if weakest
@@ -230,6 +246,10 @@ module ActiveAgent
         result("low_quality", "#{summary}.", recommendation, "scores" => @scores)
       end
 
+      def graded_label(grades)
+        grades.keys == [ "task_completion" ] ? "Task completion" : "Judged quality"
+      end
+
       def result(fault, summary, recommendation, evidence = {})
         Result.new(fault: fault, summary: summary, recommendation: recommendation, evidence: evidence.compact)
       end

data/lib/active_agent/evals/judge.rb

@@ -122,6 +122,9 @@ module ActiveAgent
         parsed = parsed&.slice("recommendation", "suggested_tool", "instruction_change")&.compact
         return nil if parsed.blank?
 
+        %w[recommendation instruction_change].each do |key|
+          parsed.delete(key) unless parsed[key].is_a?(String) && parsed[key].present?
+        end
         parsed["suggested_tool"] = suggested_tool(parsed["suggested_tool"]) if parsed.key?("suggested_tool")
         parsed.compact.presence
       end
@@ -164,7 +167,9 @@ module ActiveAgent
       def suggested_tool(tool)
         case tool
         when Hash
-          { "name" => tool["name"].to_s, "description" => tool["description"].to_s } if tool["name"].present?
+          if tool["name"].is_a?(String) && tool["name"].present?
+            { "name" => tool["name"], "description" => tool["description"].is_a?(String) ? tool["description"] : "" }
+          end
         when String
           { "name" => tool, "description" => "" } if tool.present?
         end
@@ -187,8 +192,10 @@ module ActiveAgent
       end
 
       def parse_score(content)
-        match = content.to_s.match(/"score"\s*:\s*(\d+(?:\.\d+)?)/)
-        match && match[1].to_f.clamp(0.0, 1.0)
+        value = parse_object(content)&.dig("score")
+        return nil unless value.is_a?(Numeric) && value.finite?
+
+        value.to_f.clamp(0.0, 1.0)
       end
 
       def parse_object(content)

data/lib/active_agent/evals/model_spec.rb

@@ -46,9 +46,23 @@ module ActiveAgent
       # duplicates by label.
       def self.parse_all(values, **options)
         values = values.to_s.split(",") unless values.is_a?(Array)
-        values.map { |value| value.to_s.strip }.reject(&:blank?).uniq.map { |value| parse(value, **options) }
+        values.map { |value| coerce(value) }.reject(&:blank?).uniq.map { |value| parse(value, **options) }
       end
 
+      # The label of one requested model. A caller that round-trips a spec —
+      # the dashboard persists `specs.map(&:to_h)` and hands it back on a
+      # re-run — sends a Hash, whose `to_s` is the inspected Hash and reaches
+      # the provider as the model ID: `{"label" => "openrouter/gpt-4o-mini",
+      # ...} is not a valid model ID`.
+      #
+      # @return [String, nil] nil when the value names no model
+      def self.coerce(value)
+        return value.to_h.stringify_keys.values_at("label", "model").compact.first.to_s.strip if value.respond_to?(:to_h) && !value.is_a?(String)
+
+        value.to_s.strip
+      end
+      private_class_method :coerce
+
       # The provider a bare model name runs under. A rule whose provider the
       # caller does not offer is skipped, so an app without Ollama does not
       # route `name:tag` there.

data/lib/active_agent/evals/publisher.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require "json"
+require "net/http"
+require "openssl"
+require "uri"
+
+module ActiveAgent
+  module Evals
+    # Publishes a completed report without replaying the agent. The caller must
+    # retain run_id when retrying: compatible collectors treat that identity as
+    # immutable within the authenticated account. Delivery is blocking and does
+    # not follow redirects with the account's bearer credential.
+    class Publisher
+      DEFAULT_ENDPOINT = "https://api.activeagents.ai/v1/evaluations"
+      MAX_BYTES = 2 * 1024 * 1024
+      class Error < StandardError; end
+
+      def initialize(api_key:, endpoint: DEFAULT_ENDPOINT, timeout: 10, open_timeout: 10)
+        @uri = URI.parse(endpoint.to_s)
+        unless @uri.is_a?(URI::HTTP) && @uri.host && !@uri.userinfo && !@uri.query && !@uri.fragment
+          raise ArgumentError, "Evaluation endpoint must be an HTTP(S) URL without credentials, query or fragment"
+        end
+        unless @uri.scheme == "https" || %w[localhost 127.0.0.1 ::1].include?(@uri.hostname)
+          raise ArgumentError, "Evaluation endpoint requires HTTPS except on loopback hosts"
+        end
+        raise ArgumentError, "Evaluation API key is required" if api_key.to_s.strip.empty?
+
+        @api_key = api_key.to_s
+        @timeout = Float(timeout)
+        @open_timeout = Float(open_timeout)
+        unless [ @timeout, @open_timeout ].all? { |value| value.finite? && value.positive? }
+          raise ArgumentError, "Evaluation delivery timeouts must be positive and finite"
+        end
+      rescue URI::InvalidURIError
+        raise ArgumentError, "Evaluation endpoint is not a valid URL"
+      end
+
+      # report may be a Report or its saved JSON hash. Full prompts, answers and
+      # tool results are included; applications should make publication opt-in.
+      def call(report:, run_id:, source:, agent_name:, suite:)
+        identities = { "run_id" => run_id, "source" => source, "agent_name" => agent_name, "suite" => suite }
+        identities.each do |key, value|
+          raise ArgumentError, "#{key} must be a nonempty string" unless value.is_a?(String) && !value.strip.empty?
+        end
+        body = JSON.generate(identities.merge("version" => 1, "report" => report.to_h))
+        raise Error, "Evaluation report exceeds the 2 MiB delivery limit; publish a smaller selection" if body.bytesize > MAX_BYTES
+
+        http = Net::HTTP.new(@uri.hostname, @uri.port)
+        http.use_ssl = @uri.scheme == "https"
+        http.open_timeout = @open_timeout
+        http.read_timeout = @timeout
+        http.write_timeout = @timeout
+        request = Net::HTTP::Post.new(@uri.request_uri)
+        request["Authorization"] = "Bearer #{@api_key}"
+        request["Content-Type"] = "application/json"
+        request["Accept"] = "application/json"
+        request.body = body
+        response = http.request(request)
+        unless %w[200 201].include?(response.code)
+          raise Error, "Evaluation delivery rejected (HTTP #{response.code}); retain the report and run_id for retry"
+        end
+
+        receipt = JSON.parse(response.body)
+        unless receipt.is_a?(Hash) && receipt["run_id"] == run_id && receipt["status"] == "complete" && receipt["id"] && receipt["evaluation_id"]
+          raise Error, "Evaluation collector returned an invalid completion receipt; retain the report and run_id for retry"
+        end
+        receipt
+      rescue JSON::ParserError
+        raise Error, "Evaluation collector returned invalid JSON; retain the report and run_id for retry"
+      rescue IOError, SocketError, SystemCallError, Timeout::Error, OpenSSL::SSL::SSLError => e
+        raise Error, "Evaluation delivery failed (#{e.class}); retain the report and run_id for retry"
+      end
+    end
+  end
+end

data/lib/active_agent/evals/report.rb

@@ -54,6 +54,7 @@ module ActiveAgent
         @summary_by_model ||= @models.to_h do |spec|
           cohort = @results.select { |result| result.label == spec.label }
           scored = cohort.filter_map(&:score)
+          task_scores = cohort.filter_map { |result| result.scores["task_completion"] }
           durations = cohort.filter_map { |result| result.replay.duration_ms }
           costs = cohort.filter_map { |result| result.replay.cost }
 
@@ -65,6 +66,7 @@ module ActiveAgent
             "errored" => cohort.count(&:errored?),
             "pass_rate" => cohort.any? ? (cohort.count(&:passed?) * 100.0 / cohort.size).round(1) : 0.0,
             "avg_score" => scored.any? ? (scored.sum / scored.size).round(3) : nil,
+            "avg_task_completion" => task_scores.any? ? (task_scores.sum / task_scores.size).round(3) : nil,
             "avg_duration_ms" => durations.any? ? (durations.sum.to_f / durations.size).round : nil,
             "input_tokens" => cohort.sum { |result| result.replay.input_tokens.to_i },
             "output_tokens" => cohort.sum { |result| result.replay.output_tokens.to_i },
@@ -159,7 +161,7 @@ module ActiveAgent
           "criteria" => criterion_scores,
           "recommendations" => recommendations,
           "verdict" => verdict,
-          "judge" => @judge&.label,
+          "judge" => @judge_label || @judge&.label,
           "metadata" => @metadata.presence,
           "results" => @results.map(&:to_h)
         }.compact
@@ -172,7 +174,8 @@ module ActiveAgent
       def to_markdown
         scenario_count = @results.map { |result| result.scenario.key }.uniq.size
         lines = [ "# Evaluation — #{scenario_count} scenario#{'s' unless scenario_count == 1} × #{@models.size} model#{'s' unless @models.size == 1}", "" ]
-        lines << (@judge ? "Judged by `#{@judge.label}`." : "No judge; scored on rules and expectations alone.")
+        label = @judge_label || @judge&.label
+        lines << (label ? "Judged by `#{label}`." : "No judge; scored on rules and expectations alone.")
         lines << ""
         lines.concat(summary_table)
         lines << ""

data/lib/active_agent/evals/report_html.rb

@@ -308,8 +308,8 @@ module ActiveAgent
         %(<div class="tools"><span class="micro sm">#{h(item['tools_label'])}</span><div class="list">#{chips.join}</div></div>)
       end
 
-      # "available · not enabled for Clara", "unknown · not enabled for
-      # Clara" — every status but "enabled" leads with the status word, the
+      # "available · not enabled for Assistant", "unknown · not enabled for
+      # Assistant" — every status but "enabled" leads with the status word, the
       # way the dashboard's fix list reads it.
       def html_fix_server(server)
         badge =