activeagent 1.4.0 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9cf80ba8d14c538e4f9193bf423cb82c0d50223362547df2bd3bfb2c66c62f86
-  data.tar.gz: 9f3bb57f1f22de2ee5c74350d601b459f4864a9daf2aa7198900ad0f3e44589a
+  metadata.gz: 7401cdff6a7895238a082383f2ef32683b743946445e11dacbf377470bf31071
+  data.tar.gz: b0f847d04eedf620c7f5320950d65e5ffc712296ef7db2b7ea6be25f3bc433b7
 SHA512:
-  metadata.gz: 6f817eeba3d15db4b49e924457aa510d0c4623031994e96a72ffb566aa4b800c43209682333f90aa06dc49915e03fb640f03aed7a7d8b23bae30c8cc2e945d17
-  data.tar.gz: 788b6ad2472a4faf7e8a81bfc36452268f14c7d4958af283d937417686ebd9c652c9f0c43504d16c6eb07e5852a9b2b158dd0b2b4a3038e48296efbeb2025e58
+  metadata.gz: 0a29cb99c3139f9c264eb6e86677d47ed592fb165c39489be2efc8ea2bb2dd1614bc8802643b812b04319b7356bc37177943bc9c2e0174de276555ba811ba426
+  data.tar.gz: 6fb392e7cb2f4381d79ae16567edca55d62c585b06f04a6a3da6b31cd0bb7e6149b2e67ceb7e5e143b04bc41123dd1d5bba14f38b33be6e21813e9923308d524

data/CHANGELOG.md

@@ -7,6 +7,291 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.5.0] - 2026-09-10
+
+Releases `activeagent` 1.5.0 and `actionagent` 1.5.0 from one tag.
+
+`actionagent` goes from 1.3.0 to 1.5.0, skipping 1.4: the two gems are
+released together from this repository and from one tag, and carrying one
+version number across both is less confusing than explaining which
+dashboard version pairs with which framework. `actionagent` 1.4 does not
+exist and never will. The engine's floor on the framework
+(`activeagent >= 1.4`) is unchanged and still correct.
+
+### Added
+
+- **`ActiveAgent::Evals::Publisher` delivers a finished report to a
+  collector.** A run that already happened — in CI, in a host app's own
+  runtime, anywhere the evaluation core runs — can be sent to an
+  ActiveAgents-compatible collector without replaying the agent:
+  `Publisher.new(api_key:, endpoint:).call(report:, run_id:, source:,
+  agent_name:, suite:)` posts a version-1 envelope wrapping `Report#to_h`
+  (or the saved JSON hash of an earlier run) and returns the collector's
+  receipt. Delivery is synchronous, requires HTTPS outside loopback, does
+  not follow a redirect carrying the bearer credential, caps a request at
+  2 MiB, and raises `Publisher::Error` on anything but a receipt naming the
+  same `run_id` — so a retry with that same `run_id` and the saved report
+  re-delivers rather than re-runs. Publication is strictly opt-in and
+  happens only where an application writes the call: no configuration flag,
+  no callback, no default credential, and `api_key:` supplied explicitly at
+  the call site. That is deliberate, because the payload is the report
+  itself — every scenario's prompt, the agent's answers, and the tool calls
+  and their results — and whether that may leave the application is the
+  application's decision to make. Installing the gem sends nothing
+  anywhere. `docs/evals/publication.md` documents the envelope, the receipt
+  and the retry rules. (#414)
+
+- **`Runner` takes `around_evaluation:` and `require_judge_scores:`.**
+  `around_evaluation:` is called with `(scenario, spec)` and a block, and
+  wraps the whole evaluation — the replay, the scoring, the judge calls
+  behind a recommendation — so a host can establish one trace context
+  across all of it and correlate a replay with the judging it triggered. It
+  must return the block's result; `on_result` runs after it returns, an
+  error it raises propagates to the caller, and `#evaluate` called directly
+  bypasses it, for a host doing its own scheduling. `require_judge_scores:`
+  (default `false`) settles what an unusable judge means. A judge that
+  raises or answers unscorably is skipped, and the scenario is then decided
+  on its rule scores alone — which reads as "the agent passed" when the
+  truth is "nobody graded the answer". Set it, and an otherwise passing
+  result whose `task_completion` or declared `llm_judge` criterion has no
+  usable score fails instead, with the new `judge_unavailable` fault naming
+  the unscored criteria and pointing at the judge's credentials, model and
+  JSON reply. A run with no judged criteria is unaffected. (#414)
+
+- **A grouped suite imports as YAML or JSON, whole.** `ScenarioParser` read
+  a pasted list or a JSON array of scenarios; it now also reads the grouped
+  document `Suite` loads — `groups:` with per-group keys and display names,
+  scenarios carrying `key`, `prompt`, `notes`, `expect` and
+  `production_only` — from YAML or JSON, keeping every part of it.
+  `ScenarioParser.parse` and `.scenarios` gain `include_production_only:`,
+  which defaults to `true` to match `Suite`. The dashboard defaults it the
+  other way: post the document as `scenarios_text` and the production-only
+  questions stay out unless `include_production_only` is sent alongside it,
+  because those prompts run against a live agent. The choice is made at
+  import — the engine stores the scenarios it selected, not the source
+  document — so changing it means importing the document again. (#414)
+
+- **`actionagent`: `ActionAgent.scenario_evaluation_adapter_resolver`.** A
+  host application with its own agent runtime can now run an evaluation
+  itself while keeping the dashboard's catalog, selection, jobs, result
+  persistence and report pages. The resolver is called with the persisted
+  evaluation and returns `nil` for the engine's normal `Agent#test_execute`
+  path, or a callable — `evaluation:`, `owner:`, `scenarios:`, `models:`,
+  `on_result:` — that runs the host's own agent and judge, yields every
+  result as it lands, and returns an `ActiveAgent::Evals::Report`. The
+  engine holds it to that contract: anything other than a `Report`, or a
+  report that omits or duplicates one of the selected scenario × model
+  pairs, fails the run rather than completing it with rows missing, and an
+  exception leaves the results already written in place. Dashboard
+  authentication, execution enablement and the host's execution quota still
+  apply. (#414)
+
+- **`actionagent`: the Run Agent page is a conversation workbench.** Testing
+  an agent used to mean one prompt in, one output out, with no way to see —
+  or shape — what the model was given. The page now works the way a user
+  would work the agent: it pins a persisted conversation (a solid_agent
+  context) and every run sends that conversation's user and assistant turns
+  ahead of the new message, so follow-up questions actually follow up. The
+  context is editable in place — edit or delete a turn, seed a user or
+  assistant message without running, start a new conversation — and every
+  run is a fresh `AgentRun` with its own trace, so Traces and Interactions
+  see exactly what the model saw. Files attach to a message and ride along
+  through Active Storage (`AgentRun has_many_attached :attachments`, guarded
+  for hosts without it): images reach the model as vision input, PDFs as
+  documents, and text-like files (CSV, Markdown, JSON, plain text) are
+  inlined into the message; the persisted user message keeps an attachment
+  manifest so the conversation shows thumbnails afterwards. Assistant replies
+  can render **generative UI** — cards, stats, tables, charts, lists,
+  progress, forms, choice buttons, images, callouts and code — from a fenced
+  ```` ```ui ```` JSON block in prose, a JSON reply whose top level is
+  `ui`/`blocks`, or the new `render_ui` tool (enable the **Generative UI**
+  tool on the agent). Forms and choices post their answer back into the
+  conversation as the next user message. New engine API: `GET/POST
+  /api/agents/:id/conversations`, message create/update/delete under
+  `/api/interactions/:id/messages`, multipart `POST /api/agents/:id/execute`
+  with `attachments[]` and `params[context_id]`, and attachment metadata on
+  run and message JSON. The reference host (`test/dummy`) gained the Active
+  Storage tables so the attachment path is exercised by the engine's tests.
+  Two notes for anyone driving that API directly: `execute`/`test` now answer
+  422 unless the request carries a prompt or a file, and per-run overrides in
+  `params` can no longer name `attachments` or `action` — those stay the
+  controller's to set. Model-supplied images in generative UI load on sight
+  only when they are inline data or this app's own URL; any other host is
+  offered as a click-to-load, since fetching one tells that host whatever the
+  model put in the URL.
+
+### Fixed
+
+- **A scenario passes only if it completed the task.** A scenario's verdict
+  was the mean of everything scored for it, and the judge's
+  `task_completion` grade was one number in that mean: an answer that
+  called the expected tool and contained the expected string could carry a
+  task grade of 0.2 to a mean of 0.73 and pass at the default threshold of
+  0.7. `task_completion` is a gate now — it has to reach `threshold` on its
+  own, and no number of passing tool and content checks can lift it — and
+  the fault names the number that failed: "Task completion scored 0.2
+  against a pass threshold of 0.7". An evaluation that configures its own
+  `llm_judge` criteria rather than relying on the implicit grade — which is
+  what the dashboard does — is gated the same way, on the mean of those
+  grades, so one soft dimension among strong ones still passes while an
+  answer the judge marked down cannot be carried by its mechanics. A
+  scenario the judge could not grade at all is unchanged, still falling
+  back to the rule scores. `score` and `avg_score` still mean the aggregate
+  they always did, and each model's
+  summary gains `avg_task_completion` so the judge's grade reads separately
+  from it. **This can turn a suite that passed on 1.4.0 red; see the note
+  on upgrading below.** (#414)
+
+- **A judge's score is read as a JSON number.** The score was pulled out of
+  the judge's reply by regular expression, matching the first run of digits
+  after `"score":`. It read `{"score": 9e-2}` — 0.09 — as 9, clamped to a
+  perfect 1.0; it read the string `{"score": "0.9"}` and the truncated
+  `{"score": 0.9oops}` as a confident 0.9 rather than as unusable. The
+  score now comes from the parsed JSON object and has to be a finite
+  number, so exponent notation is read as written and a string, a boolean,
+  `null`, `NaN` or `1e999` is unscorable — which the runner already knows
+  how to handle. Fenced ```` ```json ```` replies still parse. The
+  dashboard's generation-sampling evaluations score through the engine's own
+  judge rather than the framework's, and read a score by the same rule now,
+  so the two halves of the dashboard no longer disagree about the same
+  reply. (#414)
+
+- **A judge that answers with the wrong types cannot put junk in the fix
+  list.** `suggested_tool` and `instruction_change` were coerced rather
+  than checked, so a reply of `"suggested_tool": {"name": true}` added a
+  tool literally named `true` to the report's suggested tools, and
+  `"instruction_change": ["invalid"]` became a fix card asking someone to
+  add `["invalid"]` to the agent's instructions. Both fields must now be
+  nonempty strings and are dropped when they are not, so a malformed reply
+  loses only the malformed part: the judge's recommendation still reaches
+  the result, the report and every rendering of it. (#414)
+
+- **A grouped suite pasted into the dashboard keeps its keys and
+  expectations.** Only a pasted list or JSON was recognised, so a YAML suite
+  went to the line parser and was read as prose: a document describing three
+  scenarios became eighteen, with prompts like `tools: [lookup_order]` and
+  `production_only: true`, groups named `expect`, generated keys in place of
+  the document's own, and every expectation dropped — a suite that looked
+  imported and scored nothing real. Such a document is now parsed as the
+  suite it is, and one that is not valid, or a selection that matches no
+  scenarios, returns an import error (`ScenarioParser::ParseError`, HTTP
+  422) instead of a suite of nonsense or a sampling evaluation nobody asked
+  for. (#414)
+
+- **`actionagent`: run and result metadata survive persistence.** A run
+  rebuilt from the database was rebuilt without it: `Report#metadata` came
+  back holding only the four keys the engine writes itself, and each
+  result's replay metadata was gone entirely, so a host's own run and result
+  IDs, response trace IDs and judge trace IDs did not survive the round trip
+  and its reports could not be joined to its telemetry. Run metadata is now
+  kept in `scores["_metadata"]` and per-result metadata in
+  `diagnosis["_replay_metadata"]`, restored by `EvaluationRun#to_report` and
+  served as `metadata` on result JSON. Both are reserved storage keys that
+  the public diagnosis excludes, so nothing migrates and `diagnosis` still
+  means what it did. (#414)
+
+- **`actionagent`: refreshing a catalog does not rewrite what an earlier run
+  asked.** A saved run rendered its scenarios from the catalog rows as they
+  are now, so rewording a question, retagging its group or changing its
+  expectations silently rewrote history — last month's report showed this
+  month's prompt above last month's answers, and judged them against
+  expectations that were not in force when they were given. Each result now
+  records the scenario it was actually evaluated against in
+  `diagnosis["_scenario_snapshot"]`, and the report, the API and the
+  scenario matrix read that snapshot, in the order the run itself used, with
+  the dashboard noting on a scenario whose catalog entry has since changed
+  that re-running uses the current one. A run records its judge the same
+  way, in `scores["_judge_label"]`: `Report#to_h` and `#to_markdown` now
+  name the judge a rebuilt run was given instead of reporting "No judge" for
+  every run reconstructed from the database. Results saved before this
+  release carry no snapshot and still render from the current catalog, and a
+  link to a saved report (`?evaluation=:id&run=:run_id`) now opens its
+  evaluation even when it is no longer on the first page of the index.
+  (#414)
+
+- **`actionagent`: an observed agent cannot be made executable.** An agent
+  discovered from telemetry has no configuration to run, and `execute` and
+  `test` refused one — but `update` and `restore` did not, so an observed
+  record could be flipped to `active`, given instructions and then run; and
+  a run queued against an agent that became observed afterwards still
+  reached a provider when its job came up. The refusal now covers `update`
+  and `restore` as well, and it is enforced under the API rather than only
+  in front of it: `Agent#execute`, `#test_execute` and
+  `AgentExecutionService#call` raise
+  `ActionAgent::Agent::ObservedAgentError`, so that queued job fails its run
+  without a provider call or a trace. Duplicating the agent still gives you
+  an executable copy, and an evaluation whose host explicitly resolves an
+  adapter for it remains the one path that replays an observed agent's
+  scenarios. (#414)
+
+- **The OpenAI Responses API keeps images and documents on a message with a
+  role.** `{ role: "user", text: "…", image: "…" }` — the shorthand the Chat
+  API and Anthropic transforms accept, and the only provider-neutral way to
+  send history followed by a multimodal turn — lost its `image:` or
+  `document:` on the provider the framework defaults to, because the
+  Responses transform kept only `content` from a role-bearing hash. It now
+  builds `input_text` / `input_image` / `input_file` parts for it, and a
+  media-only `{ role: "user", image: "…" }` becomes a message with one part.
+  The shorthand keys always come off the message, so a hash that carries
+  `content` *and* `image:` no longer sends `image` as an unknown parameter,
+  and a blank `image:`/`document:` contributes no part rather than an empty
+  one. A nil `document:` alongside a role was the unknown-parameter case; the
+  crash needed the role-less `{ document: nil }` inside a content array, which
+  called `start_with?` on nil.
+
+- **`actionagent`: a dashboard run's trace is attributed to the agent that
+  ran it.** Every locally stored run used to register an "observed" twin of
+  its own agent, because a run's class and action match no authored record.
+  The service that ran the agent now names it when it records the trace.
+  It is named by that caller and never read from the payload: resource
+  attributes are whatever the reporter sent, and single-tenant ingest is
+  unauthenticated unless `ActionAgent.ingest_api_key` is set, so an id taken
+  from there would let any reporter bind its traces to any authored agent by
+  guessing a primary key. A host that swaps in its own `trace_model` should
+  add the `agent:` keyword to its `create_from_payload`; without it the
+  dashboard logs the error and records no trace for its own runs. (#405)
+
+- **`actionagent`: an observed agent's history cannot be authored.** The
+  runner's conversation workbench writes an agent's history without running
+  it, and those two endpoints — starting a conversation, and seeding, editing
+  or deleting a turn — did not answer to the read-only rule execution does.
+  A turn typed into a telemetry mirror would be a fabrication attributed to
+  an agent whose whole point is that it only reports what really happened.
+  Both refuse an observed agent now, with the same message and status
+  `execute` gives. Reading that history is unchanged. (#405)
+
+### Note on upgrading from 1.4.0
+
+A scenario suite that passed on 1.4.0 can fail on this release with nothing
+about your agent, your models or your suite having changed. Nothing has
+regressed: the numbers those runs passed on were wrong, and this release
+stops averaging them away.
+
+A scenario's score was the mean of every criterion scored for it, and the
+judge's `task_completion` grade — its answer to "did this actually do what
+was asked" — was one term in that mean, alongside the rule checks. An answer
+that called the expected tool, called it successfully, and contained the
+expected string scored 1.0, 1.0 and 0.2 for a mean of 0.73, and passed at
+the default threshold of 0.7: the mechanics carried the answer. That is the
+wrong answer to the question an evaluation exists to ask. The agent called
+`lookup_order`, said "ABC-123", and still never told the customer where the
+order was — and the suite went green.
+
+From this release `task_completion` has to clear `threshold` on its own.
+Expect the first run after upgrading to show fewer passes than the run
+before it, concentrated in the scenarios whose answers were thin, evasive or
+wrong while their mechanics were right. Each of those now carries the
+`low_quality` fault with a summary naming the grade that failed — "Task
+completion scored 0.2 against a pass threshold of 0.7" — and the judge's
+recommendation for it, and each model's summary reports
+`avg_task_completion` beside `avg_score`, so a drop in pass rate can be read
+against the grade that caused it. Nothing else about scoring moved: a
+scenario the judge could not grade still falls back to its rule scores
+unless you opt into `require_judge_scores: true`, and a suite meant to be
+scored on mechanics alone can run without a judge or at a lower `threshold`.
+Read that first run as a new baseline rather than a regression — it is
+measuring something the runs before it were not.
+
 ## [1.4.0] - 2026-09-09
 
 Releases `activeagent` 1.4.0 and `actionagent` 1.3.0 from one tag.

data/lib/active_agent/evals/diagnosis.rb

@@ -22,7 +22,7 @@ module ActiveAgent
     class Diagnosis
       FAULTS = %w[
         run_error tool_error missing_capability expected_tool_not_called
-        forbidden_content missing_content low_quality
+        forbidden_content missing_content low_quality judge_unavailable
       ].freeze
 
       # Phrasings an agent uses when nothing in its toolset covers the task.
@@ -56,11 +56,15 @@ module ActiveAgent
       # @param available_tools [Array<String>] tool names the agent could call
       # @param threshold [Float] the pass threshold for `score`
       # @param agent_name [String] how the recommendations refer to the agent
-      def self.call(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent")
-        new(scenario:, replay:, scores:, score:, available_tools:, threshold:, agent_name:).call
+      # @param judge_keys [Array] keys in `scores` a judge graded the answer on
+      #   (llm_judge criteria); `task_completion` always counts as one
+      def self.call(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent",
+                    judge_keys: [])
+        new(scenario:, replay:, scores:, score:, available_tools:, threshold:, agent_name:, judge_keys:).call
       end
 
-      def initialize(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent")
+      def initialize(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent",
+                     judge_keys: [])
         @scenario = scenario
         @replay = replay
         @scores = scores || {}
@@ -68,6 +72,7 @@ module ActiveAgent
         @available_tools = Array(available_tools).map(&:to_s)
         @threshold = threshold
         @agent_name = agent_name
+        @judge_keys = Array(judge_keys) | [ "task_completion" ]
       end
 
       def call
@@ -212,11 +217,22 @@ module ActiveAgent
                "missing" => missing)
       end
 
+      # A judge grade — the implicit task_completion score, or the llm_judge
+      # criteria the evaluation configured — measures the answer itself, so
+      # its mean has to reach the threshold on its own. Rule checks (a tool
+      # was called, a phrase is present) cannot carry a badly graded answer.
       def low_quality
-        return nil if @score.nil? || @score >= @threshold
-
-        weakest = @scores.compact.min_by { |_, value| value }
-        summary = "Scored #{@score.round(2)} against a pass threshold of #{@threshold}"
+        grades = @scores.slice(*@judge_keys).compact
+        grade = grades.any? ? (grades.values.sum / grades.size).round(3) : nil
+        failed_grade = grade && grade < @threshold
+        return nil unless failed_grade || (@score && @score < @threshold)
+
+        weakest = (failed_grade ? grades : @scores.compact).min_by { |_, value| value }
+        summary = if failed_grade
+          "#{graded_label(grades)} scored #{grade.round(2)} against a pass threshold of #{@threshold}"
+        else
+          "Scored #{@score.round(2)} against a pass threshold of #{@threshold}"
+        end
         summary += ", weakest on #{weakest.first} (#{weakest.last.round(2)})" if weakest
         recommendation =
           if weakest
@@ -230,6 +246,10 @@ module ActiveAgent
         result("low_quality", "#{summary}.", recommendation, "scores" => @scores)
       end
 
+      def graded_label(grades)
+        grades.keys == [ "task_completion" ] ? "Task completion" : "Judged quality"
+      end
+
       def result(fault, summary, recommendation, evidence = {})
         Result.new(fault: fault, summary: summary, recommendation: recommendation, evidence: evidence.compact)
       end

data/lib/active_agent/evals/judge.rb

@@ -122,6 +122,9 @@ module ActiveAgent
         parsed = parsed&.slice("recommendation", "suggested_tool", "instruction_change")&.compact
         return nil if parsed.blank?
 
+        %w[recommendation instruction_change].each do |key|
+          parsed.delete(key) unless parsed[key].is_a?(String) && parsed[key].present?
+        end
         parsed["suggested_tool"] = suggested_tool(parsed["suggested_tool"]) if parsed.key?("suggested_tool")
         parsed.compact.presence
       end
@@ -164,7 +167,9 @@ module ActiveAgent
       def suggested_tool(tool)
         case tool
         when Hash
-          { "name" => tool["name"].to_s, "description" => tool["description"].to_s } if tool["name"].present?
+          if tool["name"].is_a?(String) && tool["name"].present?
+            { "name" => tool["name"], "description" => tool["description"].is_a?(String) ? tool["description"] : "" }
+          end
         when String
           { "name" => tool, "description" => "" } if tool.present?
         end
@@ -187,8 +192,10 @@ module ActiveAgent
       end
 
       def parse_score(content)
-        match = content.to_s.match(/"score"\s*:\s*(\d+(?:\.\d+)?)/)
-        match && match[1].to_f.clamp(0.0, 1.0)
+        value = parse_object(content)&.dig("score")
+        return nil unless value.is_a?(Numeric) && value.finite?
+
+        value.to_f.clamp(0.0, 1.0)
       end
 
       def parse_object(content)

data/lib/active_agent/evals/publisher.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require "json"
+require "net/http"
+require "openssl"
+require "uri"
+
+module ActiveAgent
+  module Evals
+    # Publishes a completed report without replaying the agent. The caller must
+    # retain run_id when retrying: compatible collectors treat that identity as
+    # immutable within the authenticated account. Delivery is blocking and does
+    # not follow redirects with the account's bearer credential.
+    class Publisher
+      DEFAULT_ENDPOINT = "https://api.activeagents.ai/v1/evaluations"
+      MAX_BYTES = 2 * 1024 * 1024
+      class Error < StandardError; end
+
+      def initialize(api_key:, endpoint: DEFAULT_ENDPOINT, timeout: 10, open_timeout: 10)
+        @uri = URI.parse(endpoint.to_s)
+        unless @uri.is_a?(URI::HTTP) && @uri.host && !@uri.userinfo && !@uri.query && !@uri.fragment
+          raise ArgumentError, "Evaluation endpoint must be an HTTP(S) URL without credentials, query or fragment"
+        end
+        unless @uri.scheme == "https" || %w[localhost 127.0.0.1 ::1].include?(@uri.hostname)
+          raise ArgumentError, "Evaluation endpoint requires HTTPS except on loopback hosts"
+        end
+        raise ArgumentError, "Evaluation API key is required" if api_key.to_s.strip.empty?
+
+        @api_key = api_key.to_s
+        @timeout = Float(timeout)
+        @open_timeout = Float(open_timeout)
+        unless [ @timeout, @open_timeout ].all? { |value| value.finite? && value.positive? }
+          raise ArgumentError, "Evaluation delivery timeouts must be positive and finite"
+        end
+      rescue URI::InvalidURIError
+        raise ArgumentError, "Evaluation endpoint is not a valid URL"
+      end
+
+      # report may be a Report or its saved JSON hash. Full prompts, answers and
+      # tool results are included; applications should make publication opt-in.
+      def call(report:, run_id:, source:, agent_name:, suite:)
+        identities = { "run_id" => run_id, "source" => source, "agent_name" => agent_name, "suite" => suite }
+        identities.each do |key, value|
+          raise ArgumentError, "#{key} must be a nonempty string" unless value.is_a?(String) && !value.strip.empty?
+        end
+        body = JSON.generate(identities.merge("version" => 1, "report" => report.to_h))
+        raise Error, "Evaluation report exceeds the 2 MiB delivery limit; publish a smaller selection" if body.bytesize > MAX_BYTES
+
+        http = Net::HTTP.new(@uri.hostname, @uri.port)
+        http.use_ssl = @uri.scheme == "https"
+        http.open_timeout = @open_timeout
+        http.read_timeout = @timeout
+        http.write_timeout = @timeout
+        request = Net::HTTP::Post.new(@uri.request_uri)
+        request["Authorization"] = "Bearer #{@api_key}"
+        request["Content-Type"] = "application/json"
+        request["Accept"] = "application/json"
+        request.body = body
+        response = http.request(request)
+        unless %w[200 201].include?(response.code)
+          raise Error, "Evaluation delivery rejected (HTTP #{response.code}); retain the report and run_id for retry"
+        end
+
+        receipt = JSON.parse(response.body)
+        unless receipt.is_a?(Hash) && receipt["run_id"] == run_id && receipt["status"] == "complete" && receipt["id"] && receipt["evaluation_id"]
+          raise Error, "Evaluation collector returned an invalid completion receipt; retain the report and run_id for retry"
+        end
+        receipt
+      rescue JSON::ParserError
+        raise Error, "Evaluation collector returned invalid JSON; retain the report and run_id for retry"
+      rescue IOError, SocketError, SystemCallError, Timeout::Error, OpenSSL::SSL::SSLError => e
+        raise Error, "Evaluation delivery failed (#{e.class}); retain the report and run_id for retry"
+      end
+    end
+  end
+end

data/lib/active_agent/evals/report.rb

@@ -54,6 +54,7 @@ module ActiveAgent
         @summary_by_model ||= @models.to_h do |spec|
           cohort = @results.select { |result| result.label == spec.label }
           scored = cohort.filter_map(&:score)
+          task_scores = cohort.filter_map { |result| result.scores["task_completion"] }
           durations = cohort.filter_map { |result| result.replay.duration_ms }
           costs = cohort.filter_map { |result| result.replay.cost }
 
@@ -65,6 +66,7 @@ module ActiveAgent
             "errored" => cohort.count(&:errored?),
             "pass_rate" => cohort.any? ? (cohort.count(&:passed?) * 100.0 / cohort.size).round(1) : 0.0,
             "avg_score" => scored.any? ? (scored.sum / scored.size).round(3) : nil,
+            "avg_task_completion" => task_scores.any? ? (task_scores.sum / task_scores.size).round(3) : nil,
             "avg_duration_ms" => durations.any? ? (durations.sum.to_f / durations.size).round : nil,
             "input_tokens" => cohort.sum { |result| result.replay.input_tokens.to_i },
             "output_tokens" => cohort.sum { |result| result.replay.output_tokens.to_i },
@@ -159,7 +161,7 @@ module ActiveAgent
           "criteria" => criterion_scores,
           "recommendations" => recommendations,
           "verdict" => verdict,
-          "judge" => @judge&.label,
+          "judge" => @judge_label || @judge&.label,
           "metadata" => @metadata.presence,
           "results" => @results.map(&:to_h)
         }.compact
@@ -172,7 +174,8 @@ module ActiveAgent
       def to_markdown
         scenario_count = @results.map { |result| result.scenario.key }.uniq.size
         lines = [ "# Evaluation — #{scenario_count} scenario#{'s' unless scenario_count == 1} × #{@models.size} model#{'s' unless @models.size == 1}", "" ]
-        lines << (@judge ? "Judged by `#{@judge.label}`." : "No judge; scored on rules and expectations alone.")
+        label = @judge_label || @judge&.label
+        lines << (label ? "Judged by `#{label}`." : "No judge; scored on rules and expectations alone.")
         lines << ""
         lines.concat(summary_table)
         lines << ""

data/lib/active_agent/evals/report_html.rb

@@ -308,8 +308,8 @@ module ActiveAgent
         %(<div class="tools"><span class="micro sm">#{h(item['tools_label'])}</span><div class="list">#{chips.join}</div></div>)
       end
 
-      # "available · not enabled for Clara", "unknown · not enabled for
-      # Clara" — every status but "enabled" leads with the status word, the
+      # "available · not enabled for Assistant", "unknown · not enabled for
+      # Assistant" — every status but "enabled" leads with the status word, the
       # way the dashboard's fix list reads it.
       def html_fix_server(server)
         badge =

data/lib/active_agent/evals/runner.rb

@@ -8,8 +8,10 @@ module ActiveAgent
     # The one thing the runner does not know is how to talk to your agent;
     # `replay` is a callable `(scenario, model_spec) → Replay` (a Hash with the
     # same keys is accepted, and an exception becomes an errored Replay). A
-    # scenario passes when its replay completed, met its expectations, and its
-    # mean score reached `threshold`; anything else carries exactly one fault
+    # scenario passes when its replay completed, met its expectations, and both
+    # its mean score and the mean of its judge grades (task completion, or the
+    # configured llm_judge criteria) reached `threshold`;
+    # anything else carries exactly one fault
     # and a recommendation from Diagnosis, refined by the `judge` for the
     # faults in `refine_faults` (up to `judge_limit` calls per run).
     #
@@ -42,9 +44,17 @@ module ActiveAgent
       # @param instructions [String, nil] the agent's instructions, for the judge
       # @param agent_name [String] how recommendations refer to the agent
       # @param on_result [#call, nil] called with each Result as it lands
+      # @param around_evaluation [#call, nil] called with (scenario, spec) and a
+      #   block that returns the Result. Establishes context for replay, scoring
+      #   and recommendations; must return the block's result. Wrapper errors
+      #   propagate to the caller. Applies to #call, not direct #evaluate calls.
+      # @param require_judge_scores [Boolean] fail an otherwise passing result
+      #   when a requested task/LLM grade is unavailable, rather than falling
+      #   back to rule scores. Does not require a judge for rules-only runs.
       def initialize(scenarios:, models:, replay:, criteria: [], judge: nil, judge_task: true, available_tools: {},
                      instructions: nil, agent_name: "The agent", threshold: PASS_THRESHOLD,
-                     refine_faults: DEFAULT_REFINE_FAULTS, judge_limit: DEFAULT_JUDGE_LIMIT, on_result: nil, metadata: {})
+                     refine_faults: DEFAULT_REFINE_FAULTS, judge_limit: DEFAULT_JUDGE_LIMIT, on_result: nil,
+                     around_evaluation: nil, require_judge_scores: false, metadata: {})
         @scenarios = scenarios
         @models = models
         @replay = replay
@@ -58,6 +68,8 @@ module ActiveAgent
         @refine_faults = refine_faults
         @judge_limit = judge_limit
         @on_result = on_result
+        @around_evaluation = around_evaluation
+        @require_judge_scores = require_judge_scores
         @metadata = metadata
         @judge_calls = 0
         @scorer = Scorer.new(criteria: criteria, judge: judge)
@@ -66,7 +78,7 @@ module ActiveAgent
       def call
         results = @scenarios.flat_map do |scenario|
           @models.map do |spec|
-            evaluate(scenario, spec).tap { |result| @on_result&.call(result) }
+            evaluate_with_context(scenario, spec).tap { |result| @on_result&.call(result) }
           end
         end
 
@@ -85,7 +97,9 @@ module ActiveAgent
         score = Scorer.mean(scores)
 
         diagnosis = Diagnosis.call(scenario: scenario, replay: replay, scores: scores, score: score,
-                                   available_tools: @available_tools.keys, threshold: @threshold, agent_name: @agent_name)
+                                   available_tools: @available_tools.keys, threshold: @threshold, agent_name: @agent_name,
+                                   judge_keys: llm_judge_keys)
+        diagnosis ||= unavailable_judge_diagnosis(scores)
         diagnosis_hash = diagnosis&.to_h
         refine!(diagnosis_hash, scenario, replay, diagnosis) if diagnosis_hash
 
@@ -102,10 +116,51 @@ module ActiveAgent
 
       private
 
+      def evaluate_with_context(scenario, spec)
+        return evaluate(scenario, spec) unless @around_evaluation
+
+        result = @around_evaluation.call(scenario, spec) { evaluate(scenario, spec) }
+        # A wrapper written the natural way — do something, yield, do something
+        # after — returns that last value rather than the Result. Left alone it
+        # reaches on_result and the Report, and fails somewhere far from the
+        # wrapper that caused it. Name the wrapper here instead.
+        unless result.is_a?(Result)
+          raise ArgumentError, "around_evaluation must return the Result its block yields, got #{result.class}"
+        end
+
+        result
+      end
+
       def judge_task?
         @judge && @judge_task && @criteria.none? { |criterion| criterion.to_h.stringify_keys["type"] == "llm_judge" }
       end
 
+      # The keys in `scores` a judge graded, so a low grade is not averaged
+      # away against rule checks.
+      def llm_judge_keys
+        @llm_judge_keys ||= @criteria.filter_map do |criterion|
+          value = criterion.to_h.stringify_keys
+          value["key"] if value["type"] == "llm_judge"
+        end
+      end
+
+      def unavailable_judge_diagnosis(scores)
+        return unless @require_judge_scores
+
+        keys = llm_judge_keys.dup
+        keys << "task_completion" if judge_task?
+        missing = keys.select { |key| scores[key].nil? }
+        return if missing.empty?
+
+        Diagnosis::Result.new(
+          fault: "judge_unavailable",
+          summary: "The evaluation judge did not return a usable score for #{missing.join(', ')}.",
+          recommendation: "Check the judge's credentials, model availability and JSON response, then re-run this evaluation. " \
+                          "The available rule scores do not establish answer quality.",
+          evidence: { "unscored_criteria" => missing }
+        )
+      end
+
       # Whatever the callable raises becomes an errored Replay, so one model
       # rejecting a parameter fails its scenario rather than the whole run.
       # A return value that is neither a Replay nor a Hash is the caller's

data/lib/active_agent/evals/scenario_parser.rb

@@ -15,11 +15,15 @@ module ActiveAgent
     #     options on a line
     #   - a JSON array of strings, or of objects with `prompt` (or `message`),
     #     `group`, `key`, `notes`, `tools`, `contains`, `not_contains`
+    #   - a grouped Suite document in YAML or JSON, retaining its expectations,
+    #     group names, stable keys, notes and production-only flags
     #
     # Every scenario gets a key unique within the paste, derived from its group
     # and position ("blame_3"), unless the line names one. The result is an
     # array of string-keyed hashes; `Scenario.from_hash` builds the structs.
     class ScenarioParser
+      class ParseError < ArgumentError; end
+
       LIST_MARKER = /\A\s*(?:[-*•]|\d+[.)])\s+/
       HEADING = /\A\s*#+\s+(.+?)\s*\z/
       BOLD_HEADING = /\A\s*\*\*(.+?)\*\*:?\s*(?:—.*)?\z/
@@ -28,13 +32,13 @@ module ActiveAgent
       BACKTICK_PROMPT = /\A`([^`]+)`/
       OPTION_KEYS = %w[tools contains not_contains key group notes].freeze
 
-      def self.parse(text)
-        new(text).parse
+      def self.parse(text, include_production_only: true)
+        new(text).parse(include_production_only: include_production_only)
       end
 
       # Parses and builds Scenario structs in one step.
-      def self.scenarios(text)
-        parse(text).map { |attrs| Scenario.from_hash(attrs) }
+      def self.scenarios(text, include_production_only: true)
+        parse(text, include_production_only: include_production_only).map { |attrs| Scenario.from_hash(attrs) }
       end
 
       def initialize(text)
@@ -42,12 +46,19 @@ module ActiveAgent
       end
 
       # @return [Array<Hash>] scenario attributes with string keys
-      def parse
+      def parse(include_production_only: true)
         stripped = @text.strip
         return [] if stripped.empty?
 
-        scenarios = json?(stripped) ? parse_json(stripped) : parse_lines(stripped)
-        assign_keys(scenarios)
+        scenarios = if json?(stripped)
+          parse_json(stripped)
+        elsif stripped.match?(/^(?:suite|groups):(?:\s|$)/)
+          parse_suite_yaml(stripped)
+        else
+          parse_lines(stripped)
+        end
+        assigned = assign_keys(scenarios)
+        include_production_only ? assigned : assigned.reject { |entry| entry["production_only"] }
       end
 
       private
@@ -58,6 +69,8 @@ module ActiveAgent
 
       def parse_json(text)
         parsed = JSON.parse(text)
+        return parse_suite(parsed) if parsed.is_a?(Hash) && parsed.key?("groups")
+
         parsed = parsed["scenarios"] if parsed.is_a?(Hash) && parsed.key?("scenarios")
         parsed = [ parsed ] if parsed.is_a?(Hash)
 
@@ -71,6 +84,39 @@ module ActiveAgent
         parse_lines(text)
       end
 
+      def parse_suite_yaml(text)
+        document = YAML.safe_load(text, aliases: true)
+        raise ParseError, "evaluation suite must contain a groups array" unless document.is_a?(Hash) && document["groups"].is_a?(Array)
+
+        parse_suite(document)
+      rescue Psych::Exception => e
+        raise ParseError, "invalid evaluation suite YAML: #{e.message}"
+      end
+
+      def parse_suite(document)
+        raise ParseError, "evaluation suite must contain a groups array" unless document["groups"].is_a?(Array)
+
+        document["groups"].each do |group|
+          unless group.is_a?(Hash) && (group["scenarios"].nil? || group["scenarios"].is_a?(Array))
+            raise ParseError, "each evaluation group must contain a scenarios array"
+          end
+          Array(group["scenarios"]).each do |entry|
+            unless entry.is_a?(Hash) && entry["prompt"].is_a?(String) && entry["prompt"].present?
+              raise ParseError, "each evaluation scenario must contain a prompt"
+            end
+            expectations = entry["expectations"] || entry["expect"]
+            if expectations && !expectations.is_a?(Hash)
+              raise ParseError, "scenario expectations must be an object"
+            end
+          end
+        end
+
+        Suite.new([ document ]).all_scenarios.map do |item|
+          scenario(prompt: item.prompt, group: item.group, group_name: item.group_name, key: item.key,
+                   notes: item.notes, expectations: item.expectations, production_only: item.production_only?)
+        end
+      end
+
       def scenario_from_hash(entry)
         entry = entry.stringify_keys
         prompt = entry["prompt"] || entry["message"] || entry["input"] || entry["question"]
@@ -84,9 +130,11 @@ module ActiveAgent
         scenario(
           prompt: prompt,
           group: entry["group"],
+          group_name: entry["group_name"],
           key: entry["key"],
           notes: entry["notes"],
-          expectations: expectations
+          expectations: expectations,
+          production_only: entry["production_only"] == true
         )
       end
 
@@ -172,13 +220,15 @@ module ActiveAgent
         text.to_s.gsub(/\*\*|__|`/, "").strip
       end
 
-      def scenario(prompt:, group: nil, key: nil, notes: nil, expectations: {})
+      def scenario(prompt:, group: nil, group_name: nil, key: nil, notes: nil, expectations: {}, production_only: false)
         {
           "prompt" => prompt.to_s.strip,
           "group" => group.presence&.to_s&.strip,
+          "group_name" => group_name.presence&.to_s&.strip,
           "key" => key.presence&.to_s&.strip,
           "notes" => notes.presence,
-          "expectations" => (expectations || {}).reject { |_, value| value.blank? }
+          "expectations" => (expectations || {}).reject { |_, value| value.blank? },
+          "production_only" => production_only
         }
       end
 

data/lib/active_agent/evals.rb

@@ -25,6 +25,7 @@ require_relative "evals/design_tokens"
 require_relative "evals/report_html"
 require_relative "evals/report"
 require_relative "evals/runner"
+require_relative "evals/publisher"
 
 # Scenario evaluations for agents that answer with tools.
 #

data/lib/active_agent/providers/_base_provider.rb

@@ -59,7 +59,8 @@ module ActiveAgent
                     :tools_function,                 # Callback (Tools)
                     :usage_stack,                    # Usage Tracking
                     :stream_usage_index,             # Usage Tracking (Streams)
-                    :max_tool_turns, :tool_turns     # Tool-loop safety
+                    :max_tool_turns, :tool_turns,    # Tool-loop safety
+                    :instrumentation_enabled        # Per-generation privacy
 
       # Upper bound on tool-calling round-trips within one generation. A
       # model that keeps emitting tool calls otherwise recurses until the
@@ -117,6 +118,7 @@ module ActiveAgent
         self.tools_function     = kwargs.delete(:tools_function)
         self.max_tool_turns     = kwargs.delete(:max_tool_turns) || DEFAULT_MAX_TOOL_TURNS
         self.tool_turns         = 0
+        self.instrumentation_enabled = kwargs.delete(:instrumentation) != false
         self.options            = options_klass.new(kwargs.extract!(*options_klass.keys))
         self.context            = kwargs
         self.message_stack      = []
@@ -175,6 +177,8 @@ module ActiveAgent
       # @yield block to instrument
       # @return [Object] block result
       def instrument(name, payload = {}, &block)
+        return block&.call(payload) unless instrumentation_enabled
+
         full_payload = { provider: service_name, provider_module: tag_name, trace_id: }.merge(payload)
         ActiveSupport::Notifications.instrument(name, full_payload, &block)
       end

data/lib/active_agent/providers/mock/messages/base.rb

@@ -21,8 +21,10 @@ module ActiveAgent
               if content_type == :text
                 self.content = value
               else
-                # For image/document, MockProvider doesn't support these, so ignore
-                # (or could raise an error)
+                # No vision here: an image/document stands in as a text
+                # marker, so a media-only turn still has content to validate
+                # and to concatenate with its neighbours when serialized.
+                self.content ||= "[#{content_type}]"
               end
             end
           end

data/lib/active_agent/providers/open_ai/responses/transforms.rb

@@ -295,25 +295,43 @@ module ActiveAgent
               if message.respond_to?(:serialize)
                 message.serialize
               elsif message.is_a?(Hash)
-                # If it has a role, it's a message - convert :text to :content
+                # If it has a role, it's a message. Its :text becomes :content,
+                # and an :image / :document alongside it becomes a content
+                # part — the same `{role:, text:, image:}` shorthand the Chat
+                # API and Anthropic transforms accept, so a caller sending
+                # history plus a multimodal turn gets the same request shape
+                # from every provider.
                 if message.key?(:role)
                   normalized = message.dup
-                  if normalized.key?(:text) && !normalized.key?(:content)
-                    normalized[:content] = normalized.delete(:text)
+                  # The shorthand keys always come off the message: left on,
+                  # they reach the request body as unknown parameters and the
+                  # API rejects the whole call. A blank one contributes no
+                  # part rather than an empty input_image the API would
+                  # refuse (or a nil document, which has no URL to send).
+                  text = normalized.delete(:text)
+                  image = normalized.delete(:image)
+                  document = normalized.delete(:document)
+
+                  unless normalized.key?(:content)
+                    parts = []
+                    parts << { type: "input_text", text: text } if text.present?
+                    parts << { type: "input_image", image_url: image } if image.present?
+                    parts << document_part(document) if document.present?
+
+                    if parts.size == 1 && parts.first[:type] == "input_text"
+                      normalized[:content] = parts.first[:text]
+                    elsif parts.any?
+                      normalized[:content] = parts
+                    end
                   end
                   return normalized
                 end
 
                 # Expand shorthand formats to full structures for content items
-                if message.key?(:image)
+                if message[:image].present?
                   { type: "input_image", image_url: message[:image] }
-                elsif message.key?(:document)
-                  document_value = message[:document]
-                  if document_value.start_with?("data:")
-                    { type: "input_file", filename: "document.pdf", file_data: document_value }
-                  else
-                    { type: "input_file", file_url: document_value }
-                  end
+                elsif message[:document].present?
+                  document_part(message[:document])
                 elsif message.key?(:text) && message.size == 1
                   # Single :text key without :role - treat as user message
                   { role: "user", content: message[:text] }
@@ -336,6 +354,19 @@ module ActiveAgent
               end
             end
 
+            # An input_file part for a document given as a URL or a data URI.
+            #
+            # @param document_value [String] URL or data URI
+            # @return [Hash] input_file content part
+            def document_part(document_value)
+              document_value = document_value.to_s
+              if document_value.start_with?("data:")
+                { type: "input_file", filename: "document.pdf", file_data: document_value }
+              else
+                { type: "input_file", file_url: document_value }
+              end
+            end
+
             # Cleans up serialized request for API submission
             #
             # Removes default values and simplifies input where possible.

data/lib/active_agent/telemetry/instrumentation.rb

@@ -39,6 +39,7 @@ module ActiveAgent
       module GenerationInstrumentation
         # Wraps process_prompt with telemetry tracing.
         def process_prompt
+          return super if respond_to?(:prompt_options) && prompt_options.is_a?(Hash) && prompt_options[:instrumentation] == false
           return super unless Telemetry.enabled?
 
           # Reuse (or mint) the generation's trace id so the telemetry trace
@@ -210,6 +211,7 @@ module ActiveAgent
         # don't expose tool calls.
         def tools_function
           base = super
+          return base if respond_to?(:prompt_options) && prompt_options.is_a?(Hash) && prompt_options[:instrumentation] == false
           return base unless Telemetry.enabled?
 
           agent = self
@@ -247,6 +249,7 @@ module ActiveAgent
 
         # Wraps process_embed with telemetry tracing.
         def process_embed
+          return super if respond_to?(:embed_options) && embed_options.is_a?(Hash) && embed_options[:instrumentation] == false
           return super unless Telemetry.enabled?
 
           Telemetry.trace("#{self.class.name}.embed", span_type: :embedding) do |span|

data/lib/active_agent/version.rb

@@ -1,3 +1,3 @@
 module ActiveAgent
-  VERSION = "1.4.0"
+  VERSION = "1.5.0"
 end

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: activeagent
 version: !ruby/object:Gem::Version
-  version: 1.4.0
+  version: 1.5.0
 platform: ruby
 authors:
 - Justin Bowen
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-09 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -436,6 +435,7 @@ files:
 - lib/active_agent/evals/diagnosis.rb
 - lib/active_agent/evals/judge.rb
 - lib/active_agent/evals/model_spec.rb
+- lib/active_agent/evals/publisher.rb
 - lib/active_agent/evals/replay.rb
 - lib/active_agent/evals/report.rb
 - lib/active_agent/evals/report_html.rb
@@ -600,7 +600,6 @@ metadata:
   documentation_uri: https://docs.activeagents.ai
   source_code_uri: https://github.com/activeagents/activeagent
   rubygems_mfa_required: 'true'
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -615,8 +614,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.22
-signing_key:
+rubygems_version: 4.0.16
 specification_version: 4
 summary: Rails AI Agents Framework
 test_files: []