activeagent 1.3.1 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +136 -0
- data/lib/active_agent/evals/design_tokens.rb +130 -0
- data/lib/active_agent/evals/diagnosis.rb +238 -0
- data/lib/active_agent/evals/judge.rb +205 -0
- data/lib/active_agent/evals/model_spec.rb +80 -0
- data/lib/active_agent/evals/replay.rb +63 -0
- data/lib/active_agent/evals/report.rb +447 -0
- data/lib/active_agent/evals/report_html.rb +634 -0
- data/lib/active_agent/evals/result.rb +78 -0
- data/lib/active_agent/evals/runner.rb +149 -0
- data/lib/active_agent/evals/scenario.rb +68 -0
- data/lib/active_agent/evals/scenario_parser.rb +215 -0
- data/lib/active_agent/evals/scorer.rb +118 -0
- data/lib/active_agent/evals/suite.rb +99 -0
- data/lib/active_agent/evals.rb +60 -0
- data/lib/active_agent/providers/ruby_llm/options.rb +4 -0
- data/lib/active_agent/providers/ruby_llm_provider.rb +14 -1
- data/lib/active_agent/providers/rubyllm_provider.rb +1 -0
- data/lib/active_agent/telemetry/configuration.rb +11 -0
- data/lib/active_agent/telemetry/instrumentation.rb +26 -6
- data/lib/active_agent/version.rb +1 -1
- data/lib/active_agent.rb +1 -0
- metadata +22 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9cf80ba8d14c538e4f9193bf423cb82c0d50223362547df2bd3bfb2c66c62f86
|
|
4
|
+
data.tar.gz: 9f3bb57f1f22de2ee5c74350d601b459f4864a9daf2aa7198900ad0f3e44589a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 6f817eeba3d15db4b49e924457aa510d0c4623031994e96a72ffb566aa4b800c43209682333f90aa06dc49915e03fb640f03aed7a7d8b23bae30c8cc2e945d17
|
|
7
|
+
data.tar.gz: 788b6ad2472a4faf7e8a81bfc36452268f14c7d4958af283d937417686ebd9c652c9f0c43504d16c6eb07e5852a9b2b158dd0b2b4a3038e48296efbeb2025e58
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,142 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [1.4.0] - 2026-09-09
|
|
11
|
+
|
|
12
|
+
Releases `activeagent` 1.4.0 and `actionagent` 1.3.0 from one tag.
|
|
13
|
+
|
|
14
|
+
### Added
|
|
15
|
+
|
|
16
|
+
- **`ActiveAgent::Evals`, the evaluation core, in the framework.** Pasted-list
|
|
17
|
+
and YAML suite parsing, model resolution, rule and expectation scoring, the
|
|
18
|
+
fault taxonomy with its recommendations, the optional judge, and the
|
|
19
|
+
per-model report live in `lib/active_agent/evals`, loadable on their own
|
|
20
|
+
with `require "active_agent/evals"`. Any app can replay a list of tasks
|
|
21
|
+
across models against its own agent through one `replay` callable and get
|
|
22
|
+
the same faults, recommendations and verdict the dashboard shows;
|
|
23
|
+
`actionagent` keeps only what the dashboard adds — persistence, the job,
|
|
24
|
+
the API and the UI.
|
|
25
|
+
- **Scenario evaluations in the dashboard.** An evaluation can now carry a
|
|
26
|
+
suite of scenarios — a pasted list of user messages, grouped with
|
|
27
|
+
`# Heading` lines and annotated with the tool each should call — and a run
|
|
28
|
+
replays every selected scenario through the agent once per candidate model
|
|
29
|
+
(`compare_models`, or a per-run `models` selection) instead of sampling
|
|
30
|
+
recorded generations. Each scenario × model result records the answer, the
|
|
31
|
+
tools called, its score, and, when it falls short, one fault
|
|
32
|
+
(`run_error`, `tool_error`, `missing_capability`,
|
|
33
|
+
`expected_tool_not_called`, `forbidden_content`, `missing_content`,
|
|
34
|
+
`low_quality`) with a recommendation; a configured judge model refines the
|
|
35
|
+
recommendation with the tool to add or the instruction to change. Runs can
|
|
36
|
+
be narrowed to a group or to single scenarios, and the run summary ranks
|
|
37
|
+
the models by pass rate with a verdict. New tables
|
|
38
|
+
`evaluation_scenarios` and `evaluation_scenario_results` ship in
|
|
39
|
+
`create_active_agent_evaluation_scenarios`, which
|
|
40
|
+
`rails generate action_agent:install` emits for new and existing installs.
|
|
41
|
+
- **`ActionAgent.mcp_catalog`.** A host app registers the MCP servers it
|
|
42
|
+
serves or connects itself — `[{ key:, name:, tool_hints: [...] }, …]` —
|
|
43
|
+
and they join the built-in catalog: listed in the MCP Services view, with
|
|
44
|
+
telemetry traffic for their bare tool names attributed to them.
|
|
45
|
+
`MCPCatalog.keys` lists built-ins and registrations together;
|
|
46
|
+
`MCPCatalog::BY_KEY` still holds the built-ins alone.
|
|
47
|
+
- **Suite results rebuilt around what to do next.** An expanded scenario
|
|
48
|
+
suite used to be a summary and a matrix; it now opens on the three
|
|
49
|
+
questions a run is actually asked. **Runs** numbers every run of the suite
|
|
50
|
+
from the oldest and scores it against the one before — `+3 passed vs #7`,
|
|
51
|
+
or `partial run` when the two covered different scenarios or models and the
|
|
52
|
+
numbers do not compare — so progress is legible without reading two runs
|
|
53
|
+
side by side; selecting an older run re-derives the models, the fix list,
|
|
54
|
+
the matrix and every drill-down, and the collapsed header keeps reporting
|
|
55
|
+
the latest. **Models** marks the best candidate with an info-toned
|
|
56
|
+
`judge's pick` badge and the verdict that justifies it, rather than a green
|
|
57
|
+
*winner* — losing a comparison by one scenario is not a failing grade.
|
|
58
|
+
**What to fix** turns each fault into a card with the tools involved
|
|
59
|
+
(deduplicated to one chip each), the MCP server that serves them, whether
|
|
60
|
+
this agent has it enabled, and a button that deep-links to MCP Services,
|
|
61
|
+
Tools or the agent's instructions: the fix, not just the finding. The
|
|
62
|
+
scenario × model matrix shows the tools each model actually called against
|
|
63
|
+
the tools the scenario expected, coloured by whether they match, and a row
|
|
64
|
+
opens onto every model's answer, timing, cost and diagnosis.
|
|
65
|
+
- **The evaluation report is a designed page.** `Report#to_html(theme:)`
|
|
66
|
+
renders a run on the dashboard's design system — stat tiles, a panel per
|
|
67
|
+
model with the judge's pick and verdict, the what-to-fix cards, the
|
|
68
|
+
scenario × model matrix and a disclosure per scenario — still one
|
|
69
|
+
self-contained file with inline styles and no external assets, so it
|
|
70
|
+
archives next to a CI run. `theme:` pins `"light"` or `"dark"`; without it
|
|
71
|
+
the page follows the viewer's `prefers-color-scheme`, and the dashboard
|
|
72
|
+
passes its own theme through when it frames the report at
|
|
73
|
+
`/api/evaluations/:id/runs/:run_id/report`. The new `Report#fix_items`
|
|
74
|
+
builds the what-to-fix list — faults grouped with the tools each implicates
|
|
75
|
+
and the action that addresses it — for the page, the engine's API and any
|
|
76
|
+
app that wants the backlog as JSON; `tool_resolver:`, `agent_name:` and
|
|
77
|
+
`links:` on `Report.new` let a host name the MCP server behind a tool, the
|
|
78
|
+
agent, and the routes an action should point at, so a CI job gets the same
|
|
79
|
+
cards the dashboard shows.
|
|
80
|
+
- **An APM-style service overview on the Metrics page.** The page answered
|
|
81
|
+
"how much traffic in the last 24 hours"; it now answers "is this healthy
|
|
82
|
+
right now, and since when". A `1h` / `24h` / `7d` range fixes the bucket
|
|
83
|
+
size the whole page is drawn at (60 × 1 min, 96 × 15 min, 84 × 2 h); five
|
|
84
|
+
golden signals — requests, latency, error rate, tokens, cost — carry a
|
|
85
|
+
sparkline and a delta against the period just before the window; six panels
|
|
86
|
+
plot requests stacked by agent, latency percentiles, errors by class,
|
|
87
|
+
tokens, spend and tool calls, with markers for the agent versions deployed
|
|
88
|
+
inside the window and for an error spike when one stands out; and a rail
|
|
89
|
+
ranks the agents, models, slowest actions, tools and error classes behind
|
|
90
|
+
them. Filtering to an agent — from the select, or by clicking its rail
|
|
91
|
+
row — narrows every one of those together. `GET /api/metrics` gains
|
|
92
|
+
`range` and `agent` params and the keys that feed it (`totals`, `deltas`,
|
|
93
|
+
`series`, `agents`, `models`, `actions`, `tools`, `errors_by_type`,
|
|
94
|
+
`markers`) from the new `ActionAgent::MetricsReport`: one pass over the
|
|
95
|
+
window, with bucketing, nearest-rank percentiles and error classification
|
|
96
|
+
done in Ruby so PostgreSQL and SQLite report the same numbers. Every
|
|
97
|
+
earlier key and param still means what it did.
|
|
98
|
+
- **A design token layer under the dashboard.** Colors, fonts and the type
|
|
99
|
+
scale live in `actionagent/frontend/tokens.css` as CSS variables scoped to
|
|
100
|
+
the mounted dashboard (`.aa-dashboard`, with the dark palette under
|
|
101
|
+
`.theme-dark`), and the views draw from a set of shared primitives —
|
|
102
|
+
badges, chips, panels, cards, pass bars, stat tiles, segmented controls —
|
|
103
|
+
instead of each restating the same hex codes and paddings. Dark mode is
|
|
104
|
+
then one class rather than a conditional at every call site, and a host
|
|
105
|
+
app's own stylesheet cannot bleed into the engine's. The framework carries
|
|
106
|
+
the same values in `ActiveAgent::Evals::DesignTokens` so the standalone
|
|
107
|
+
HTML report matches the dashboard it came from, with a test that fails when
|
|
108
|
+
the two drift apart.
|
|
109
|
+
|
|
110
|
+
- **RubyLLM backend pinning via `platform:`.** RubyLLM resolves which of its
|
|
111
|
+
providers serves a request from the model ID, and a model served by more
|
|
112
|
+
than one — `gemini-2.5-flash` exists on both the Gemini API and Vertex
|
|
113
|
+
AI — lands on whichever RubyLLM's registry prefers, with no way to say
|
|
114
|
+
otherwise from ActiveAgent. The new `platform:` option
|
|
115
|
+
(`generate_with :ruby_llm, model: "gemini-2.5-flash", platform: :vertexai`)
|
|
116
|
+
forwards to RubyLLM's `provider:` and pins the backend, for embeddings as
|
|
117
|
+
well as prompts. It is not named `provider:` because a provider reference
|
|
118
|
+
is already the first argument to `generate_with`. Omitting it keeps
|
|
119
|
+
model-based routing unchanged. (#373)
|
|
120
|
+
|
|
121
|
+
### Fixed
|
|
122
|
+
|
|
123
|
+
- **A run report is readable in the dashboard.** The report was framed at a
|
|
124
|
+
fixed viewport height, so everything past the first screen — including
|
|
125
|
+
every fix item — sat behind a nested scrollbar. The frame is sized to the
|
|
126
|
+
report's own content, and a fix action targets the top window so it
|
|
127
|
+
navigates the dashboard instead of loading it into the frame. (#410, #411)
|
|
128
|
+
|
|
129
|
+
- **Provider credentials store on a host that skipped `db:encryption:init`.**
|
|
130
|
+
Encryption keys derived from `secret_key_base` were installed after Rails
|
|
131
|
+
had already configured `ActiveRecord::Encryption`, so the config read back
|
|
132
|
+
correct while every credential write raised `Errors::Configuration` — in
|
|
133
|
+
the dashboard, the Settings API Keys tab failed to render and provider
|
|
134
|
+
keys failed to save. (#412)
|
|
135
|
+
|
|
136
|
+
- **`service: "RubyLLM"` loads when the ruby_llm railtie has run.** The
|
|
137
|
+
ruby_llm gem registers `RubyLLM` as an inflector acronym in Rails apps,
|
|
138
|
+
which turns `"RubyLLM".underscore` into `rubyllm` — so provider loading
|
|
139
|
+
required a nonexistent `rubyllm_provider.rb` and failed with
|
|
140
|
+
`cannot load such file`. An alias file now covers that require path, the
|
|
141
|
+
same fix `openai_provider.rb` applies for `OpenAI`. (#371, fixed in #372
|
|
142
|
+
by @aoki-ryusei; regression tests in #374)
|
|
143
|
+
|
|
8
144
|
## [1.3.1] - 2026-08-19
|
|
9
145
|
|
|
10
146
|
### Fixed
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ActiveAgent
|
|
4
|
+
module Evals
|
|
5
|
+
# The dashboard's design tokens as Ruby, so the self-contained HTML report
|
|
6
|
+
# paints with the same palette as the mounted dashboard without loading
|
|
7
|
+
# its stylesheet.
|
|
8
|
+
#
|
|
9
|
+
# Mirrors actionagent/frontend/tokens.css: LIGHT is the `.aa-dashboard`
|
|
10
|
+
# block, DARK the overrides in `.aa-dashboard.theme-dark`. A test asserts
|
|
11
|
+
# the two files stay in step — change a value in both places.
|
|
12
|
+
module DesignTokens
|
|
13
|
+
FONT_TEXT = '"Inter Variable", Inter, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif'
|
|
14
|
+
FONT_MONO = '"JetBrains Mono", "SF Mono", "Fira Code", Menlo, Consolas, "Courier New", monospace'
|
|
15
|
+
|
|
16
|
+
LIGHT = {
|
|
17
|
+
# Brand
|
|
18
|
+
"--color-accent" => "#FA343B",
|
|
19
|
+
"--color-accent-hover" => "#E02D33",
|
|
20
|
+
"--color-accent-b" => "#FA343B29",
|
|
21
|
+
"--color-accent-b-hover" => "#FA343B33",
|
|
22
|
+
"--color-on-accent" => "#ffffff",
|
|
23
|
+
"--color-accent-ui" => "#ef4444",
|
|
24
|
+
"--color-accent-ui-hover" => "#dc2626",
|
|
25
|
+
"--color-accent-ui-muted" => "rgba(239, 68, 68, 0.15)",
|
|
26
|
+
"--color-accent-ui-tint" => "#fef2f2",
|
|
27
|
+
|
|
28
|
+
# Surfaces
|
|
29
|
+
"--color-background" => "#f9fafb",
|
|
30
|
+
"--color-background-page" => "#ffffff",
|
|
31
|
+
"--color-surface" => "#ffffff",
|
|
32
|
+
"--color-card" => "#ffffff",
|
|
33
|
+
"--color-muted" => "#f3f4f6",
|
|
34
|
+
"--color-hover" => "#f3f4f6",
|
|
35
|
+
"--color-background-blur" => "rgba(255,255,255,0.9)",
|
|
36
|
+
|
|
37
|
+
# Borders
|
|
38
|
+
"--color-border" => "#e5e7eb",
|
|
39
|
+
"--color-border-light" => "#f3f4f6",
|
|
40
|
+
"--color-border-strong" => "#d1d5db",
|
|
41
|
+
|
|
42
|
+
# Text
|
|
43
|
+
"--color-text-primary" => "#111827",
|
|
44
|
+
"--color-text-secondary" => "#6b7280",
|
|
45
|
+
"--color-text-muted" => "#9ca3af",
|
|
46
|
+
"--color-text-cell" => "#4b5563",
|
|
47
|
+
|
|
48
|
+
# Semantic status
|
|
49
|
+
"--color-success" => "#16a34a",
|
|
50
|
+
"--color-success-soft" => "#dcfce7",
|
|
51
|
+
"--color-success-text" => "#166534",
|
|
52
|
+
"--color-warning" => "#eab308",
|
|
53
|
+
"--color-warning-soft" => "#fef9c3",
|
|
54
|
+
"--color-warning-text" => "#854d0e",
|
|
55
|
+
"--color-error" => "#dc2626",
|
|
56
|
+
"--color-error-soft" => "#fee2e2",
|
|
57
|
+
"--color-error-text" => "#991b1b",
|
|
58
|
+
"--color-info" => "#3b82f6",
|
|
59
|
+
"--color-info-soft" => "#dbeafe",
|
|
60
|
+
"--color-info-text" => "#1e40af",
|
|
61
|
+
|
|
62
|
+
# Trace span colors (observability)
|
|
63
|
+
"--span-root" => "#9ca3af",
|
|
64
|
+
"--span-prompt" => "#60a5fa",
|
|
65
|
+
"--span-generate" => "#a855f7",
|
|
66
|
+
"--span-llm" => "#ef4444",
|
|
67
|
+
"--span-thinking" => "#fbbf24",
|
|
68
|
+
"--span-tool" => "#22c55e",
|
|
69
|
+
"--span-response" => "#2dd4bf",
|
|
70
|
+
|
|
71
|
+
# Token flow colors
|
|
72
|
+
"--color-token-in" => "#2563eb",
|
|
73
|
+
"--color-token-out" => "#7c3aed",
|
|
74
|
+
|
|
75
|
+
# Chart / agent palette
|
|
76
|
+
"--chart-1" => "#6366f1",
|
|
77
|
+
"--chart-2" => "#10b981",
|
|
78
|
+
"--chart-3" => "#f59e0b",
|
|
79
|
+
"--chart-4" => "#ec4899",
|
|
80
|
+
"--chart-5" => "#3b82f6",
|
|
81
|
+
|
|
82
|
+
# Type
|
|
83
|
+
"--font-text" => FONT_TEXT,
|
|
84
|
+
"--font-mono" => FONT_MONO
|
|
85
|
+
}.freeze
|
|
86
|
+
|
|
87
|
+
DARK = {
|
|
88
|
+
"--color-accent-ui-tint" => "rgba(239, 68, 68, 0.15)",
|
|
89
|
+
|
|
90
|
+
"--color-background" => "#0f0f0f",
|
|
91
|
+
"--color-background-page" => "#0f0f0f",
|
|
92
|
+
"--color-surface" => "#1a1a1a",
|
|
93
|
+
"--color-card" => "rgba(255,255,255,0.05)",
|
|
94
|
+
"--color-muted" => "rgba(255,255,255,0.05)",
|
|
95
|
+
"--color-hover" => "#252525",
|
|
96
|
+
"--color-background-blur" => "rgba(15,15,15,0.9)",
|
|
97
|
+
|
|
98
|
+
"--color-border" => "rgba(255,255,255,0.1)",
|
|
99
|
+
"--color-border-light" => "rgba(255,255,255,0.05)",
|
|
100
|
+
"--color-border-strong" => "rgba(255,255,255,0.2)",
|
|
101
|
+
|
|
102
|
+
"--color-text-primary" => "#ffffff",
|
|
103
|
+
"--color-text-secondary" => "rgba(255,255,255,0.6)",
|
|
104
|
+
"--color-text-muted" => "rgba(255,255,255,0.4)",
|
|
105
|
+
"--color-text-cell" => "rgba(255,255,255,0.7)",
|
|
106
|
+
|
|
107
|
+
"--color-success-soft" => "rgba(22,163,74,0.15)",
|
|
108
|
+
"--color-success-text" => "#4ade80",
|
|
109
|
+
"--color-warning-soft" => "rgba(234,179,8,0.15)",
|
|
110
|
+
"--color-warning-text" => "#facc15",
|
|
111
|
+
"--color-error-soft" => "rgba(220,38,38,0.15)",
|
|
112
|
+
"--color-error-text" => "#f87171",
|
|
113
|
+
"--color-info-soft" => "rgba(59,130,246,0.15)",
|
|
114
|
+
"--color-info-text" => "#93c5fd"
|
|
115
|
+
}.freeze
|
|
116
|
+
|
|
117
|
+
# One CSS rule declaring `tokens` as custom properties on `scope`, with
|
|
118
|
+
# an optional `color-scheme` so form controls and scrollbars follow:
|
|
119
|
+
#
|
|
120
|
+
# DesignTokens.css(scope: ":root") # light
|
|
121
|
+
# DesignTokens.css(scope: ":root.theme-dark", tokens: DesignTokens::DARK,
|
|
122
|
+
# color_scheme: "dark") # dark overrides
|
|
123
|
+
def self.css(scope:, tokens: LIGHT, color_scheme: nil)
|
|
124
|
+
declarations = tokens.map { |name, value| " #{name}: #{value};" }
|
|
125
|
+
declarations.unshift(" color-scheme: #{color_scheme};") if color_scheme
|
|
126
|
+
"#{scope} {\n#{declarations.join("\n")}\n}"
|
|
127
|
+
end
|
|
128
|
+
end
|
|
129
|
+
end
|
|
130
|
+
end
|
|
@@ -0,0 +1,238 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ActiveAgent
|
|
4
|
+
module Evals
|
|
5
|
+
# Explains why a scenario did not pass and what would fix it.
|
|
6
|
+
#
|
|
7
|
+
# The fault is assigned from the replay's evidence, the most mechanical
|
|
8
|
+
# cause first, so a run that crashed is a `run_error` even if its empty
|
|
9
|
+
# answer would also have failed a content check:
|
|
10
|
+
#
|
|
11
|
+
# run_error — the run raised, or the agent returned nothing
|
|
12
|
+
# tool_error — a tool the agent called returned an error
|
|
13
|
+
# missing_capability — the agent said no tool covers the task
|
|
14
|
+
# expected_tool_not_called — the scenario expects a tool the agent did not call
|
|
15
|
+
# forbidden_content — the answer contains a pattern the scenario forbids
|
|
16
|
+
# missing_content — the answer lacks a pattern the scenario expects
|
|
17
|
+
# low_quality — the answer scored below the threshold
|
|
18
|
+
#
|
|
19
|
+
# Each fault carries a recommendation written from the evidence; a Judge
|
|
20
|
+
# can replace it with one that names the tool to add (Runner does this).
|
|
21
|
+
# Returns nil for a passing result.
|
|
22
|
+
class Diagnosis
|
|
23
|
+
FAULTS = %w[
|
|
24
|
+
run_error tool_error missing_capability expected_tool_not_called
|
|
25
|
+
forbidden_content missing_content low_quality
|
|
26
|
+
].freeze
|
|
27
|
+
|
|
28
|
+
# Phrasings an agent uses when nothing in its toolset covers the task.
|
|
29
|
+
# "find" and "see" are deliberately absent: "I can't find any…" and
|
|
30
|
+
# "I don't see…" report a negative result, not a missing capability.
|
|
31
|
+
REFUSED_VERBS = "have|access|retrieve|look up|query|check|view|search"
|
|
32
|
+
CAPABILITY_REFUSALS = [
|
|
33
|
+
/\bI(?:'m| am)? (?:do not |don't |cannot |can't |unable to |not able to )(?:currently )?(?:#{REFUSED_VERBS})\b/i,
|
|
34
|
+
/\b(?:no|don't have (?:a|any)) tools? (?:is |are )?(?:available|that can|to)\b/i,
|
|
35
|
+
/\bnot (?:something|able|possible) (?:I|to) (?:can|am able to )?(?:do|access|retrieve|look up)\b/i,
|
|
36
|
+
/\bI (?:don't|do not) have (?:the ability|a way|access|visibility|the tools?)\b/i,
|
|
37
|
+
/\boutside (?:of )?(?:my|the) (?:capabilities|available tools|scope)\b/i,
|
|
38
|
+
/\bcan(?:'|no)t (?:be )?(?:done|determined|answered) with (?:the|my) (?:current|available) tools\b/i
|
|
39
|
+
].freeze
|
|
40
|
+
|
|
41
|
+
Result = Struct.new(:fault, :summary, :recommendation, :evidence, keyword_init: true) do
|
|
42
|
+
def to_h
|
|
43
|
+
{
|
|
44
|
+
"fault" => fault,
|
|
45
|
+
"summary" => summary,
|
|
46
|
+
"recommendation" => recommendation,
|
|
47
|
+
"evidence" => evidence
|
|
48
|
+
}
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# @param scenario [Scenario]
|
|
53
|
+
# @param replay [Replay]
|
|
54
|
+
# @param scores [Hash] criterion key => 0.0..1.0 (nil when unscorable)
|
|
55
|
+
# @param score [Float, nil] the mean score
|
|
56
|
+
# @param available_tools [Array<String>] tool names the agent could call
|
|
57
|
+
# @param threshold [Float] the pass threshold for `score`
|
|
58
|
+
# @param agent_name [String] how the recommendations refer to the agent
|
|
59
|
+
def self.call(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent")
|
|
60
|
+
new(scenario:, replay:, scores:, score:, available_tools:, threshold:, agent_name:).call
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def initialize(scenario:, replay:, scores:, score:, available_tools:, threshold: PASS_THRESHOLD, agent_name: "The agent")
|
|
64
|
+
@scenario = scenario
|
|
65
|
+
@replay = replay
|
|
66
|
+
@scores = scores || {}
|
|
67
|
+
@score = score
|
|
68
|
+
@available_tools = Array(available_tools).map(&:to_s)
|
|
69
|
+
@threshold = threshold
|
|
70
|
+
@agent_name = agent_name
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def call
|
|
74
|
+
run_error || tool_error || missing_capability || expected_tool_not_called ||
|
|
75
|
+
forbidden_content || missing_content || low_quality
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
private
|
|
79
|
+
|
|
80
|
+
def answer
|
|
81
|
+
@replay.answer.to_s
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def called_tools
|
|
85
|
+
@replay.tool_names
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
def agent
|
|
89
|
+
@agent_name
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
def run_error
|
|
93
|
+
if @replay.errored?
|
|
94
|
+
message = @replay.error.to_s
|
|
95
|
+
return result("run_error", "The run failed before #{agent.downcase} answered: #{message.truncate(200)}",
|
|
96
|
+
run_error_recommendation(message), "error" => message.truncate(1_000))
|
|
97
|
+
end
|
|
98
|
+
return nil if answer.present?
|
|
99
|
+
|
|
100
|
+
result("run_error", "#{agent} returned an empty answer.",
|
|
101
|
+
"The provider returned no content. Check the model name is one the provider serves and that the " \
|
|
102
|
+
"output budget leaves room for an answer after the tool calls.",
|
|
103
|
+
"error" => "empty answer")
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
def run_error_recommendation(message)
|
|
107
|
+
case message
|
|
108
|
+
when /credentials|api.?key|access_token|unauthori[sz]ed|401/i
|
|
109
|
+
"Add credentials for the provider this model runs on before comparing it."
|
|
110
|
+
when /model.*(not found|does not exist|unknown|unsupported)|404/i
|
|
111
|
+
"The provider rejected the model name. Check the spelling against the provider's catalog, or prefix " \
|
|
112
|
+
"it with the provider (`ollama/qwen3:8b`) so it runs where it exists."
|
|
113
|
+
when /rate limit|429|overloaded|529/i
|
|
114
|
+
"The provider throttled the run. Re-run the failed scenarios; if it recurs, run fewer scenarios per " \
|
|
115
|
+
"batch or compare fewer models at once."
|
|
116
|
+
else
|
|
117
|
+
"Inspect the run's error. A failure here is infrastructure — it says nothing about the answer yet."
|
|
118
|
+
end
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
def tool_error
|
|
122
|
+
failed = @replay.failed_tool_calls
|
|
123
|
+
return nil if failed.empty?
|
|
124
|
+
|
|
125
|
+
names = failed.map { |call| call["name"] }.uniq
|
|
126
|
+
detail = failed.first["detail"].to_s.truncate(300)
|
|
127
|
+
result("tool_error", "Tool #{names.join(', ')} returned an error while answering.",
|
|
128
|
+
"Fix the failing tool before judging the answer: #{names.join(', ')} errored with \"#{detail}\". " \
|
|
129
|
+
"If the arguments look wrong, tighten the tool's parameter descriptions so the model calls it " \
|
|
130
|
+
"correctly; if the tool itself broke, fix its implementation.",
|
|
131
|
+
"tools" => names, "detail" => detail, "arguments" => failed.first["arguments"])
|
|
132
|
+
end
|
|
133
|
+
|
|
134
|
+
def missing_capability
|
|
135
|
+
return nil unless CAPABILITY_REFUSALS.any? { |pattern| answer.match?(pattern) }
|
|
136
|
+
return nil if called_tools.any? && @score.to_f >= @threshold
|
|
137
|
+
|
|
138
|
+
missing = @scenario.expected_tools - @available_tools
|
|
139
|
+
recommendation =
|
|
140
|
+
if missing.any?
|
|
141
|
+
"#{agent} said it cannot do this, and the expected tool(s) #{missing.join(', ')} are not in its " \
|
|
142
|
+
"toolset. Add or enable them."
|
|
143
|
+
elsif @available_tools.empty?
|
|
144
|
+
"#{agent} has no tools, so it can only answer from its instructions. Give it a tool that reads the " \
|
|
145
|
+
"data this task needs."
|
|
146
|
+
else
|
|
147
|
+
"None of the available tools (#{@available_tools.join(', ')}) covers this task. Add a tool that does, " \
|
|
148
|
+
"or, if one of them should, rewrite its description so the model recognises when to use it."
|
|
149
|
+
end
|
|
150
|
+
|
|
151
|
+
result("missing_capability", "#{agent} said it lacks the ability to perform this task.", recommendation,
|
|
152
|
+
"refusal" => refusal_excerpt, "tools_available" => @available_tools, "tools_called" => called_tools)
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
def refusal_excerpt
|
|
156
|
+
pattern = CAPABILITY_REFUSALS.find { |candidate| answer.match?(candidate) }
|
|
157
|
+
match = answer.match(pattern)
|
|
158
|
+
return nil unless match
|
|
159
|
+
|
|
160
|
+
answer[[ match.begin(0) - 80, 0 ].max, 260].to_s.strip
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
def expected_tool_not_called
|
|
164
|
+
expected = @scenario.expected_tools
|
|
165
|
+
return nil if expected.empty? || (expected & called_tools).any?
|
|
166
|
+
|
|
167
|
+
unavailable = expected - @available_tools
|
|
168
|
+
recommendation =
|
|
169
|
+
if unavailable.any?
|
|
170
|
+
"The scenario expects #{unavailable.join(', ')}, which #{agent.downcase} does not have. Enable the " \
|
|
171
|
+
"tool (or add the server that provides it) and re-run."
|
|
172
|
+
elsif called_tools.any?
|
|
173
|
+
"#{agent} answered with #{called_tools.uniq.join(', ')} instead of #{expected.join(', ')}. Sharpen " \
|
|
174
|
+
"both tools' descriptions so the model can tell them apart, or say in the instructions which tool " \
|
|
175
|
+
"answers this kind of task."
|
|
176
|
+
else
|
|
177
|
+
"#{expected.join(', ')} is available but #{agent.downcase} answered without calling any tool. Tell " \
|
|
178
|
+
"it in the instructions to prefer tool-backed answers for this kind of task, and check the tool's " \
|
|
179
|
+
"description says what it returns."
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
result("expected_tool_not_called",
|
|
183
|
+
"Expected #{expected.join(' or ')} to be called; #{agent.downcase} called " \
|
|
184
|
+
"#{called_tools.uniq.presence&.join(', ') || 'nothing'}.",
|
|
185
|
+
recommendation, "expected" => expected, "called" => called_tools, "unavailable" => unavailable)
|
|
186
|
+
end
|
|
187
|
+
|
|
188
|
+
def forbidden_content
|
|
189
|
+
matched = @scenario.forbidden_patterns.select { |pattern| Scorer.matches_pattern?(answer, pattern) }
|
|
190
|
+
return nil if matched.empty?
|
|
191
|
+
|
|
192
|
+
result("forbidden_content", "The answer contains content the scenario forbids: #{matched.join(', ')}.",
|
|
193
|
+
"Add an explicit instruction against \"#{matched.first}\" and, if the phrase comes from a tool " \
|
|
194
|
+
"result, filter it in the tool rather than relying on the model to omit it.",
|
|
195
|
+
"matched" => matched)
|
|
196
|
+
end
|
|
197
|
+
|
|
198
|
+
def missing_content
|
|
199
|
+
missing = @scenario.expected_patterns.reject { |pattern| Scorer.matches_pattern?(answer, pattern) }
|
|
200
|
+
return nil if missing.empty?
|
|
201
|
+
|
|
202
|
+
recommendation =
|
|
203
|
+
if called_tools.empty? && @available_tools.any?
|
|
204
|
+
"#{agent} answered without calling a tool, so it could not have found \"#{missing.first}\". " \
|
|
205
|
+
"Instruct it to use its tools for this kind of task."
|
|
206
|
+
else
|
|
207
|
+
"The answer never mentions \"#{missing.first}\". Check whether the tool result contained it — if it " \
|
|
208
|
+
"did, the instructions should ask for it explicitly; if not, the tool needs to return it."
|
|
209
|
+
end
|
|
210
|
+
|
|
211
|
+
result("missing_content", "The answer is missing expected content: #{missing.join(', ')}.", recommendation,
|
|
212
|
+
"missing" => missing)
|
|
213
|
+
end
|
|
214
|
+
|
|
215
|
+
def low_quality
|
|
216
|
+
return nil if @score.nil? || @score >= @threshold
|
|
217
|
+
|
|
218
|
+
weakest = @scores.compact.min_by { |_, value| value }
|
|
219
|
+
summary = "Scored #{@score.round(2)} against a pass threshold of #{@threshold}"
|
|
220
|
+
summary += ", weakest on #{weakest.first} (#{weakest.last.round(2)})" if weakest
|
|
221
|
+
recommendation =
|
|
222
|
+
if weakest
|
|
223
|
+
"Read the answer against the #{weakest.first.to_s.humanize.downcase} criterion and adjust the " \
|
|
224
|
+
"instructions where it falls short. A criterion that keeps scoring low across scenarios points at " \
|
|
225
|
+
"the instructions; one that fails on one scenario points at that task's tooling."
|
|
226
|
+
else
|
|
227
|
+
"Compare this answer with a passing one for a similar scenario and adjust the instructions."
|
|
228
|
+
end
|
|
229
|
+
|
|
230
|
+
result("low_quality", "#{summary}.", recommendation, "scores" => @scores)
|
|
231
|
+
end
|
|
232
|
+
|
|
233
|
+
def result(fault, summary, recommendation, evidence = {})
|
|
234
|
+
Result.new(fault: fault, summary: summary, recommendation: recommendation, evidence: evidence.compact)
|
|
235
|
+
end
|
|
236
|
+
end
|
|
237
|
+
end
|
|
238
|
+
end
|