activeagent 1.9.7 → 1.9.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +259 -1
- data/README.md +3 -0
- data/lib/active_agent/cli/eval_commands.rb +53 -4
- data/lib/active_agent/concerns/tooling.rb +4 -2
- data/lib/active_agent/evals/capture.rb +14 -3
- data/lib/active_agent/evals/catalog.rb +5 -2
- data/lib/active_agent/evals/gate_check.rb +4 -1
- data/lib/active_agent/model_capabilities.rb +1 -1
- data/lib/active_agent/providers/_base_provider.rb +1 -1
- data/lib/active_agent/providers/anthropic_provider.rb +10 -0
- data/lib/active_agent/providers/bedrock/bearer_client.rb +2 -1
- data/lib/active_agent/providers/common/usage.rb +6 -1
- data/lib/active_agent/providers/open_ai/responses_provider.rb +7 -0
- data/lib/active_agent/providers/requesty_provider.rb +8 -0
- data/lib/active_agent/providers/ruby_llm/tool_proxy.rb +15 -0
- data/lib/active_agent/providers/ruby_llm_provider.rb +62 -5
- data/lib/active_agent/providers/runcrate/_types.rb +17 -0
- data/lib/active_agent/providers/runcrate/options.rb +39 -0
- data/lib/active_agent/providers/runcrate_provider.rb +72 -0
- data/lib/active_agent/schema_tools.rb +50 -11
- data/lib/active_agent/version.rb +1 -1
- metadata +13 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e5cf4f1304cb0d939f96e270b950dbe8bc3505e6e4d8dacba9f0f16083c1b485
|
|
4
|
+
data.tar.gz: 7716fbc50b2e06ac508cf88a63b8515bbec0f17d6ab632b57a05706f0fe1187b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f237aff2404b739fa66c77e0d5a6e2925be257423fc2072ae7b1387c00a144cf0fdbb95466237981170bcf1f2f9b468c0f592bdc5d3442522300fc0dce6b581a
|
|
7
|
+
data.tar.gz: ef1161bd9b4d39338594946354ce0f455f212f9508ff0a8f0119fd2762fb965a7b32798a3913a2473a27424c6c918e7102d3a14e78233d4b67a96f1771aac108
|
data/CHANGELOG.md
CHANGED
|
@@ -5,7 +5,265 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
-
## [
|
|
8
|
+
## [1.9.9] - 2026-11-18
|
|
9
|
+
|
|
10
|
+
Schema tools can ask whether a column is set: a filter takes
|
|
11
|
+
`{"present": false}` (IS NULL) or `{"present": true}` (IS NOT NULL), so a
|
|
12
|
+
model can find the tickets with no assignee without a sentinel range in
|
|
13
|
+
the instructions. Around that, fixes to evaluations and providers: an
|
|
14
|
+
evaluation run and a fanned-out replay are each claimed before they
|
|
15
|
+
execute, so a job delivered twice no longer replays the suite or calls
|
|
16
|
+
the provider twice; a catalog document safe loading refuses is a 422
|
|
17
|
+
instead of a 500; a run of only ungated groups no longer reads as "every
|
|
18
|
+
gate met"; and the RubyLLM provider refuses media it would drop.
|
|
19
|
+
|
|
20
|
+
Upgrading: no migrations. The RubyLLM provider now raises
|
|
21
|
+
`ActiveAgent::Providers::Errors::InvalidRequest` for a message carrying an
|
|
22
|
+
image, audio or a file, where it used to send the text alone; route such
|
|
23
|
+
messages to a provider that takes media.
|
|
24
|
+
|
|
25
|
+
### Added
|
|
26
|
+
|
|
27
|
+
- **Schema tools can ask whether a column is set** (`activeagent`). A
|
|
28
|
+
filter value of `null` means "no filter", so a model had no way to ask
|
|
29
|
+
for the tickets with no assignee or the requests not yet approved, and
|
|
30
|
+
hosts wrote sentinel ranges such as `{"after": "2000-01-01"}` into their
|
|
31
|
+
instructions instead. A filter now takes `{"present": false}` (IS NULL)
|
|
32
|
+
or `{"present": true}` (IS NOT NULL) on any filterable column, alone or
|
|
33
|
+
beside a range bound. The tool schema offers it in the range object of
|
|
34
|
+
nullable date, time and number columns, so foreign keys get it too, and
|
|
35
|
+
an enum accepts it though it takes no comparison. Anything but true or
|
|
36
|
+
false is rejected rather than guessed.
|
|
37
|
+
|
|
38
|
+
### Fixed
|
|
39
|
+
|
|
40
|
+
- **An evaluation run is executed by one job** (`actionagent`). The run job
|
|
41
|
+
read a run's status, then the scenario runner wrote `running`, as two
|
|
42
|
+
steps, so a job delivered twice, or two workers that both loaded the run
|
|
43
|
+
as pending, each replayed the whole suite, or with
|
|
44
|
+
`scenario_evaluation_fan_out` on each enqueued a full set of replay jobs.
|
|
45
|
+
The runner now moves the run from pending to running in one conditional
|
|
46
|
+
update, as the execution job claims an agent run, and replays nothing
|
|
47
|
+
unless it was the one to do so; this covers `ProjectEvaluationJob` too.
|
|
48
|
+
The claim comes before the selection is read, so a selection that cannot
|
|
49
|
+
be resolved (an unsupported provider in a re-run's models) now fails the
|
|
50
|
+
run instead of leaving it pending forever.
|
|
51
|
+
- **A fanned-out replay is claimed before the agent runs** (`actionagent`).
|
|
52
|
+
`EvaluationReplayJob` checked for a stored result, replayed, and only
|
|
53
|
+
then took the run's lock to store, so two deliveries of one replay job
|
|
54
|
+
both called the provider (and the agent's tools) and the second answer
|
|
55
|
+
was dropped. A delivery that failed after another had stored the pair's
|
|
56
|
+
replay also marked that good result errored. The replay now takes a
|
|
57
|
+
claim, kept with the run's fan-out bookkeeping (no migration), before
|
|
58
|
+
the agent runs. A delivery that finds the pair claimed comes back when
|
|
59
|
+
the claim expires (30 minutes) and replays only if nothing was stored,
|
|
60
|
+
so a worker that stopped mid-replay is still covered. A failure is
|
|
61
|
+
stored only over the delivery's own pair. A worker that stops after the
|
|
62
|
+
provider answered and before the result is stored is still replayed
|
|
63
|
+
again: this is one active replay per pair, not exactly once.
|
|
64
|
+
- **A catalog document YAML refuses to load is an invalid document**
|
|
65
|
+
(`activeagent`, `actionagent`). `Catalog.parse` turned only a syntax
|
|
66
|
+
error into `Catalog::InvalidDocument`, while safe loading refuses a tag
|
|
67
|
+
naming a Ruby class (`!ruby/object:...`) or an alias to an undefined
|
|
68
|
+
anchor with other `Psych` errors. Those escaped the dashboard's catalog
|
|
69
|
+
import and preview as a 500, and the `catalogs_import` and
|
|
70
|
+
`catalogs_preview` tools as an internal error. They are now refused like
|
|
71
|
+
any other invalid catalog: 422 with "the catalog is not valid YAML: ...".
|
|
72
|
+
What safe loading permits is unchanged.
|
|
73
|
+
- **A run of only ungated groups no longer reads as "every gate met"**
|
|
74
|
+
(`activeagent`). An ungated group's row is met without being held to a
|
|
75
|
+
bar, so a report whose every result sat in ungated groups said "every
|
|
76
|
+
pass gate met" (`GateCheck`, `assert_meets_pass_gates`, the RSpec
|
|
77
|
+
matcher) and `evals run --wait` printed `Gates: every gate met`, even
|
|
78
|
+
when every answer had failed. The verdict is unchanged, since an ungated
|
|
79
|
+
group never fails the run, but the message now says that every group
|
|
80
|
+
covered is ungated and nothing was held to a bar.
|
|
81
|
+
- **The RubyLLM provider refuses media it would drop** (`activeagent`). It
|
|
82
|
+
sends RubyLLM the text of each message and silently left out every other
|
|
83
|
+
content block, so a message with an image (`image_url`, `image`,
|
|
84
|
+
`input_image`), audio or a file reached the model as its text alone, or
|
|
85
|
+
as an empty message, and the model answered as if it had seen it. Such
|
|
86
|
+
a message now raises `ActiveAgent::Providers::Errors::InvalidRequest`,
|
|
87
|
+
naming the block types, before any provider call. Blocks that are not
|
|
88
|
+
media (an earlier turn's reasoning) are still left out, as before.
|
|
89
|
+
|
|
90
|
+
## [1.9.8] - 2026-11-12
|
|
91
|
+
|
|
92
|
+
The dashboard holds a run to its catalog's pass bars: a product's and a
|
|
93
|
+
set's `pass` blocks and a product's named criteria lists survive import, a
|
|
94
|
+
run records the bars it started under and shows each group's gate beside
|
|
95
|
+
its pass rate, and `evals run` exits 4 when a gated group misses, so CI
|
|
96
|
+
fails on the group that slipped rather than on the whole-run average. A
|
|
97
|
+
catalog that declares a `threshold` now scores at it instead of the fixed
|
|
98
|
+
0.7. Around that: the empty agent list shows the ways an agent arrives,
|
|
99
|
+
telemetry first; the RubyLLM provider works with ruby_llm 2.2 and the
|
|
100
|
+
Anthropic provider with the `anthropic` gem 2.x; Runcrate joins the
|
|
101
|
+
OpenAI-compatible providers; and smaller fixes to Anthropic usage counts,
|
|
102
|
+
Requesty embeddings, `top_k` for thinking-first models, late cancels,
|
|
103
|
+
`tool_observer` under Capture and Responses streaming errors.
|
|
104
|
+
|
|
105
|
+
Upgrading: two dashboard migrations, 034 `add_catalog_product_criteria`
|
|
106
|
+
(the `criteria` column on scenario products) and 035
|
|
107
|
+
`add_catalog_pass_bars` (the `pass` column on scenario products and sets):
|
|
108
|
+
run `bin/rails generate action_agent:install --skip` and `bin/rails
|
|
109
|
+
db:migrate`. Until they run, catalog imports keep working but still drop
|
|
110
|
+
named criteria lists and pass bars, so dashboard runs stay ungated; a
|
|
111
|
+
catalog imported before the upgrade needs importing again to gain them.
|
|
112
|
+
033 is left free for a later release. A finished run that carries gates
|
|
113
|
+
now fails `evals run --wait` with exit 4 when any gated group missed its
|
|
114
|
+
bar; a run without gates is reported as before. ruby_llm 2.2.0 (2026-11-10)
|
|
115
|
+
breaks the RubyLLM provider in 1.9.7 and earlier (every tool call reads as
|
|
116
|
+
`nil`); upgrade, or hold `ruby_llm < 2.2` until you do.
|
|
117
|
+
|
|
118
|
+
### Added
|
|
119
|
+
|
|
120
|
+
- **The dashboard shows a run's per-group pass gates** (`actionagent`). A
|
|
121
|
+
published report's `pass_thresholds` are kept with the run, so the
|
|
122
|
+
rebuilt report, the export and the run page gate the stored results as
|
|
123
|
+
the application's report did, and a run started from the dashboard on a
|
|
124
|
+
catalog set is held to the set's and product's `pass` bars, recorded as
|
|
125
|
+
the run starts so a later catalog change does not regrade it. The run
|
|
126
|
+
page adds a **Pass rate by group** table, and the run's badge reads
|
|
127
|
+
`gate missed` when a gate was not met even if the whole-run pass rate
|
|
128
|
+
looks fine. The run's JSON adds `gates` and `gates_met`; a run without
|
|
129
|
+
bars reads as before (#678).
|
|
130
|
+
- **An empty agent list shows how agents arrive** (`actionagent`). With no
|
|
131
|
+
agents, the Agents page offered only Browse Templates and Create From
|
|
132
|
+
Scratch. It now shows three ways in, telemetry first:
|
|
133
|
+
- **Connect your app** asks what runs the agents. For ActiveAgent it shows
|
|
134
|
+
the `config/active_agent.yml` telemetry block. For RubyLLM it shows the
|
|
135
|
+
`activeagents-telemetry-ruby_llm` Gemfile line and the initializer that
|
|
136
|
+
subscribes it. Both point at this dashboard's endpoint and read the key
|
|
137
|
+
from `ACTIVEAGENTS_API_KEY`. The first trace registers the agent, and
|
|
138
|
+
**Check for agents** reloads the list.
|
|
139
|
+
- **Link a GitHub repository** opens New project.
|
|
140
|
+
- **Build one here** keeps the template library and the builder.
|
|
141
|
+
- **Runcrate as an OpenAI-compatible provider** (`activeagent`).
|
|
142
|
+
`service: "Runcrate"` generates and embeds through
|
|
143
|
+
`https://api.runcrate.ai/v1` (open-weight models such as
|
|
144
|
+
`deepseek-ai/DeepSeek-V4.1-Flash` and `Qwen/Qwen3-Embedding-8B`), with the
|
|
145
|
+
key from `RUNCRATE_API_KEY`; a thin subclass of the OpenAI provider, like
|
|
146
|
+
Cheaper Inference.
|
|
147
|
+
|
|
148
|
+
### Fixed
|
|
149
|
+
|
|
150
|
+
- **The RubyLLM provider works with ruby_llm 2.2** (`activeagent`). RubyLLM
|
|
151
|
+
2.2.0 (2026-11-10) changed three things the provider leaned on. A
|
|
152
|
+
message's `tool_calls` became a `ToolCall::Collection` that yields the
|
|
153
|
+
calls rather than ID-and-call pairs, so every tool call read as `nil`; a
|
|
154
|
+
streamed delta now carries the provider's call ID in `provider_call_id`
|
|
155
|
+
and leaves `id` empty, so every fragment joined the latest call and
|
|
156
|
+
arguments doubled up; and a chat asks each tool for its `namespace`,
|
|
157
|
+
`qualified_name` and `deferred?`, which ActiveAgent's tool proxy did not
|
|
158
|
+
answer. The provider now iterates either shape, keys streamed fragments
|
|
159
|
+
by the provider's ID, and the proxy answers the three. ruby_llm 1.16
|
|
160
|
+
through 2.1 behave as before. The evals conversation check also learned
|
|
161
|
+
2.2's `acts_as_chat` modules, columns and reworded unavailable-tool error.
|
|
162
|
+
- **`evals run` fails CI on a missed group gate** (`activeagent`). A run
|
|
163
|
+
whose critical group slipped to 50% passed `evals run --min-pass-rate`
|
|
164
|
+
whenever the other groups lifted the whole-run pass rate over the bar.
|
|
165
|
+
A finished run that the dashboard holds to per-group pass gates now fails
|
|
166
|
+
(exit 4) when any gated group missed its bar, with `--wait` alone or with
|
|
167
|
+
`--min-pass-rate`, which stays the whole-run bar; each missed gate prints
|
|
168
|
+
as one line, and `--json` adds `gates`. A run without gates is gated as
|
|
169
|
+
before (#677).
|
|
170
|
+
- **Responses streaming failures retain their error details** (`activeagent`).
|
|
171
|
+
`error` and `response.failed` events raise a provider error with the API
|
|
172
|
+
code and message instead of an unexpected chunk type error.
|
|
173
|
+
- **The dashboard keeps a catalog's pass bars** (`actionagent`). A
|
|
174
|
+
product's and a set's `pass` blocks (`min_pass_rate`, `threshold`,
|
|
175
|
+
`gated`) parsed in `ActiveAgent::Evals::Catalog`, but neither
|
|
176
|
+
`ScenarioProduct` nor `ScenarioSet` kept them: an import dropped the
|
|
177
|
+
bars, the export left them out, the stored catalog's `pass_thresholds`
|
|
178
|
+
fell back to the defaults (every group gated at 100%), and the import
|
|
179
|
+
preview reported the product and its sets as `update` on every re-import
|
|
180
|
+
of an unchanged file. The bars are now stored and exported as imported,
|
|
181
|
+
and both import modes take the document's. Existing installs add the
|
|
182
|
+
columns with `bin/rails generate action_agent:install --skip` and
|
|
183
|
+
`bin/rails db:migrate` (`add_catalog_pass_bars`); imports keep working,
|
|
184
|
+
without the bars, until it runs. Bars dropped by an earlier version come
|
|
185
|
+
back when the original document is imported again.
|
|
186
|
+
- **The dashboard keeps a catalog product's named criteria lists**
|
|
187
|
+
(`actionagent`). A product's `criteria: { shared: [...], strict: [...] }`
|
|
188
|
+
parsed and round-tripped in `ActiveAgent::Evals::Catalog`, but
|
|
189
|
+
`ScenarioProduct` had nowhere to keep it: an import dropped the lists, the
|
|
190
|
+
export left them out, the stored digest never matched the records'
|
|
191
|
+
export, and the import preview reported the product as `update` on every
|
|
192
|
+
re-import of an unchanged file. The lists are now stored on the product
|
|
193
|
+
and exported as imported, so the documents and digests match and an
|
|
194
|
+
unchanged file previews as unchanged. Both import modes take the
|
|
195
|
+
document's lists, as they take a set's `criteria`. A set still runs on
|
|
196
|
+
its own `criteria`; which product lists a run uses is left for later.
|
|
197
|
+
Existing installs add the column with
|
|
198
|
+
`bin/rails generate action_agent:install --skip` and `bin/rails db:migrate`
|
|
199
|
+
(`add_catalog_product_criteria`); imports keep working, without the lists,
|
|
200
|
+
until it runs. Lists dropped by an earlier version come back when the
|
|
201
|
+
original document is imported again (#687).
|
|
202
|
+
- **A self-hosted dashboard shows its own trace endpoint** (`actionagent`).
|
|
203
|
+
The Organization page's Telemetry card showed the platform's `/v1/traces`.
|
|
204
|
+
A self-hosted engine takes traces at `<mount>/api/traces`, and the card
|
|
205
|
+
now shows that. Multi-tenant installs are unchanged.
|
|
206
|
+
- **The agent list's filter arrows sit inside the select** (`actionagent`).
|
|
207
|
+
The browser's own arrow ignores padding and sat against the right border.
|
|
208
|
+
The three filters now draw an inset chevron (`aa-select`).
|
|
209
|
+
- **The Anthropic provider loads the `anthropic` gem 2.x** (`activeagent`).
|
|
210
|
+
The provider pinned its client gem to `~> 1.12`, so a bundle holding
|
|
211
|
+
anthropic 2.0 or later raised `LoadError` the moment the provider loaded,
|
|
212
|
+
although the provider builds its requests as hashes and the 2.0 SDK
|
|
213
|
+
(role-union `MessageParam`, one-value enums as constants, Text Completions
|
|
214
|
+
removed) breaks none of them. The provider and the gemspec's development
|
|
215
|
+
dependency now accept anthropic 1.12 through 2.x, and CI runs the provider
|
|
216
|
+
tests against 2.3 beside the 1.12, 1.14 and 1.55 legs. Two 2.0 changes do
|
|
217
|
+
reach the code: Text Completions went away with
|
|
218
|
+
`Anthropic::Resources::Completions`, which the Bedrock bearer client now
|
|
219
|
+
sets up only when it exists, and the client sends its key as
|
|
220
|
+
`Authorization: Bearer` instead of `x-api-key`. The 2.0 SDK also dropped
|
|
221
|
+
the `?beta=true` query string from its beta Messages requests, so the test
|
|
222
|
+
suite matches recorded cassettes with or without it.
|
|
223
|
+
- **A streamed Anthropic response reports its token usage**
|
|
224
|
+
(`activeagent`). With `stream: true`, `response.usage` was nil, so a
|
|
225
|
+
streamed generation counted 0 tokens and cost nothing. The provider now
|
|
226
|
+
records the usage `message_start` reports and the cumulative counts
|
|
227
|
+
`message_delta` carries, as the OpenAI providers do from their chunks.
|
|
228
|
+
- **Anthropic's `total_tokens` counts cached tokens** (`activeagent`).
|
|
229
|
+
Anthropic's `input_tokens` leaves out cache reads and writes, which it
|
|
230
|
+
reports separately, so `total_tokens` undercounted any prompt that used
|
|
231
|
+
the cache. It is now input + output + cache read + cache creation. The
|
|
232
|
+
other counts are unchanged, and so is cost: pricing still reads
|
|
233
|
+
`input_tokens` and `output_tokens` only.
|
|
234
|
+
- **Added usage keeps its duration** (`activeagent`). `Usage#+` dropped
|
|
235
|
+
`duration_ms`, so the usage of a tool-calling loop (Ollama reports a
|
|
236
|
+
duration) came back without one. It is now summed like the other
|
|
237
|
+
optional counts.
|
|
238
|
+
- **Requesty embeddings work** (`activeagent`). The Requesty provider
|
|
239
|
+
declared an embedding request but never sent one, so `embed` raised
|
|
240
|
+
`NotImplementedError`. It now calls Requesty's OpenAI-compatible
|
|
241
|
+
`/v1/embeddings`.
|
|
242
|
+
- **`top_k` is stripped for thinking-first models** (`activeagent`).
|
|
243
|
+
`ModelCapabilities` removed `temperature` and `top_p` for the models that
|
|
244
|
+
reject sampling parameters, but not `top_k`, which they reject too, so an
|
|
245
|
+
agent configured with it still got a 400.
|
|
246
|
+
- **A late cancel leaves a finished run as it finished** (`actionagent`).
|
|
247
|
+
`AgentRun#cancel!` checked the status of its own copy of the run and
|
|
248
|
+
then wrote `cancelled` unconditionally. A cancel request that loaded the
|
|
249
|
+
run while it was running, and wrote after the worker had recorded its
|
|
250
|
+
result, turned a complete run into a cancelled one that still carried
|
|
251
|
+
its output, and replaced a failed run's error with "Cancelled by user".
|
|
252
|
+
The cancel now moves the run to `cancelled` in one conditional update,
|
|
253
|
+
as the execution job claims a pending run, and only from pending,
|
|
254
|
+
running or awaiting input. A run that finished first keeps its status,
|
|
255
|
+
output, error and completion time; `cancel!` returns false and reloads
|
|
256
|
+
the copy, so the API answers with the status the run ended in. A cancel
|
|
257
|
+
is broadcast once, where the callback and the method each sent it
|
|
258
|
+
before. Cancelling still does not stop a provider call already in
|
|
259
|
+
flight; it keeps that call's result from being recorded.
|
|
260
|
+
- **Capture keeps an agent's own `tool_observer`** (`activeagent`). Since
|
|
261
|
+
1.9.7, `Evals::Capture` records client-side MCP calls through the
|
|
262
|
+
agent's `tool_observer`, and it replaced an observer the agent defined
|
|
263
|
+
itself, so that observer stopped running whenever the agent was
|
|
264
|
+
captured. Capture now runs the agent's observer around the call it
|
|
265
|
+
records: the server is still called once, both see the same result or
|
|
266
|
+
exception, and a call the observer withholds is not recorded as made.
|
|
9
267
|
|
|
10
268
|
## [1.9.7] - 2026-11-04
|
|
11
269
|
|
data/README.md
CHANGED
|
@@ -86,8 +86,8 @@ module ActiveAgent
|
|
|
86
86
|
p.on("--keys LIST", "Replay only these scenario keys, comma-separated") { |v| arguments[:keys] = list_option(v) }
|
|
87
87
|
p.on("--group GROUP", "Replay only this scenario group") { |v| arguments[:group] = v }
|
|
88
88
|
p.on("--sandbox ID", "Replay against this ready checkout sandbox") { |v| arguments[:sandbox_id] = v }
|
|
89
|
-
p.on("--wait", "Wait for the run to finish") { wait = true }
|
|
90
|
-
p.on("--min-pass-rate RATE", "
|
|
89
|
+
p.on("--wait", "Wait for the run to finish; fail (exit #{EXIT_GATE}) when it failed or missed a pass gate") { wait = true }
|
|
90
|
+
p.on("--min-pass-rate RATE", "Also fail below this whole-run pass rate: 0.8 or 80%; implies --wait") do |v|
|
|
91
91
|
min_pass_rate = pass_rate_option(v)
|
|
92
92
|
wait = true
|
|
93
93
|
end
|
|
@@ -199,18 +199,28 @@ module ActiveAgent
|
|
|
199
199
|
end
|
|
200
200
|
end
|
|
201
201
|
|
|
202
|
+
# A finished run fails the gate when it failed, when its whole pass
|
|
203
|
+
# rate is under --min-pass-rate, or when it was held to per-group pass
|
|
204
|
+
# gates (the run's "gates", from the suite's or catalog set's `pass`
|
|
205
|
+
# bars) and the dashboard says it did not meet every one. A run
|
|
206
|
+
# without gates is gated on its status and rate alone, as it was.
|
|
202
207
|
def gate(payload, min_pass_rate)
|
|
203
208
|
run = payload.fetch("run", {})
|
|
204
209
|
rate = pass_rate(run)
|
|
205
210
|
failed = run["status"] == "failed"
|
|
206
211
|
below = min_pass_rate && (rate.nil? || rate < min_pass_rate)
|
|
207
|
-
|
|
212
|
+
gates = run_gates(run)
|
|
213
|
+
missed = !gates.nil? && run["gates_met"] != true
|
|
214
|
+
verdict = failed || below || missed ? "fail" : "pass"
|
|
208
215
|
failures = payload.fetch("results", [])
|
|
216
|
+
summary = { "pass_rate" => rate, "min_pass_rate" => min_pass_rate }
|
|
217
|
+
summary["gates"] = gates if gates
|
|
209
218
|
|
|
210
|
-
emit(payload.merge("
|
|
219
|
+
emit(payload.merge(summary).merge("verdict" => verdict)) do
|
|
211
220
|
@out.puts "Run #{run["id"]} #{run["status"]}: #{passed(run) || "no results"}" \
|
|
212
221
|
"#{rate ? format(" (%.1f%%)", rate * 100) : ""}#{min_pass_rate ? format(", threshold %.1f%%", min_pass_rate * 100) : ""}."
|
|
213
222
|
@out.puts "Error: #{run["error_message"]}" if run["error_message"]
|
|
223
|
+
print_gates(gates, missed) if gates
|
|
214
224
|
unless failures.empty?
|
|
215
225
|
@out.puts "Failed results:"
|
|
216
226
|
table(failures, {
|
|
@@ -227,6 +237,45 @@ module ActiveAgent
|
|
|
227
237
|
verdict == "pass" ? EXIT_SUCCESS : EXIT_GATE
|
|
228
238
|
end
|
|
229
239
|
|
|
240
|
+
# The run's per-group pass gates as the dashboard worked them out
|
|
241
|
+
# (EvaluationRun#gates), or nil for a run held to none.
|
|
242
|
+
def run_gates(run)
|
|
243
|
+
rows = run["gates"]
|
|
244
|
+
rows.select { |row| row.is_a?(Hash) } if rows.is_a?(Array)
|
|
245
|
+
end
|
|
246
|
+
|
|
247
|
+
def print_gates(gates, missed)
|
|
248
|
+
unmet = gates.reject { |gate| gate["met"] == true }
|
|
249
|
+
unless missed
|
|
250
|
+
# Ungated rows are met without being held to a bar, so a run that
|
|
251
|
+
# reached only those says so rather than "every gate met".
|
|
252
|
+
if gates.none? { |gate| gate["gated"] != false }
|
|
253
|
+
@out.puts "Gates: every group reached is ungated (#{gates.size}); nothing was held to a bar."
|
|
254
|
+
else
|
|
255
|
+
@out.puts "Gates: every gate met (#{gates.size})."
|
|
256
|
+
end
|
|
257
|
+
return
|
|
258
|
+
end
|
|
259
|
+
|
|
260
|
+
@out.puts(unmet.empty? ? "Gates: no gate was reached." : "Gates not met:")
|
|
261
|
+
unmet.each { |gate| @out.puts " #{gate_sentence(gate)}" }
|
|
262
|
+
end
|
|
263
|
+
|
|
264
|
+
# A gate the run missed, as ActiveAgent::Evals::Report::Gate#to_s
|
|
265
|
+
# reads it: "Orders · gpt-5-mini: 1/2 (50%) passed, below 100%".
|
|
266
|
+
def gate_sentence(gate)
|
|
267
|
+
passed = gate["passed"].to_i
|
|
268
|
+
total = gate["total"].to_i
|
|
269
|
+
passes = total.positive? ? "#{passed}/#{total} (#{whole_percent(passed.to_f / total)})" : "no results"
|
|
270
|
+
label = cell(gate["label"]) unless gate["label"].to_s.empty?
|
|
271
|
+
"#{cell(gate["name"] || gate["group"] || "All")}#{label ? " · #{label}" : ""}: #{passes} passed, below #{whole_percent(gate["min_pass_rate"])}"
|
|
272
|
+
end
|
|
273
|
+
|
|
274
|
+
def whole_percent(fraction)
|
|
275
|
+
value = Float(fraction, exception: false)
|
|
276
|
+
value ? "#{(value * 100).round}%" : "?"
|
|
277
|
+
end
|
|
278
|
+
|
|
230
279
|
def pass_rate(run)
|
|
231
280
|
evaluated = run["samples_evaluated"].to_i
|
|
232
281
|
evaluated.positive? ? run["samples_passed"].to_f / evaluated : nil
|
|
@@ -30,8 +30,10 @@ module ActiveAgent
|
|
|
30
30
|
# A callback the provider runs the calls it serves from an MCP server
|
|
31
31
|
# around, which never reach {#tools_function}. It receives the tool name,
|
|
32
32
|
# its arguments and a block that makes the call, and returns the block's
|
|
33
|
-
# result. Nil unless
|
|
34
|
-
#
|
|
33
|
+
# result. Nil unless the agent defines one, to audit its MCP calls say,
|
|
34
|
+
# or a host that records a generation's tool calls sets it on the
|
|
35
|
+
# instance. Evals::Capture keeps an agent's own observer and records the
|
|
36
|
+
# call that observer makes.
|
|
35
37
|
#
|
|
36
38
|
# @return [Proc, nil]
|
|
37
39
|
def tool_observer = nil
|
|
@@ -19,8 +19,10 @@ module ActiveAgent
|
|
|
19
19
|
# (`mcp_strategy: :client`, or a provider with no MCP of its own) is
|
|
20
20
|
# dispatched before that proc, so the provider reports it through the
|
|
21
21
|
# agent's `tool_observer` instead, with a failure the server reported
|
|
22
|
-
# marked as one.
|
|
23
|
-
#
|
|
22
|
+
# marked as one. An observer the agent defines itself still runs, around
|
|
23
|
+
# the call that is recorded. A tool a provider runs on its own side
|
|
24
|
+
# (Anthropic's or OpenAI's hosted MCP) never reaches this process and is
|
|
25
|
+
# not recorded.
|
|
24
26
|
#
|
|
25
27
|
# `model:` and `provider:` replay the same action under another model,
|
|
26
28
|
# the way a comparison run does. A model is an option on the agent
|
|
@@ -190,9 +192,15 @@ module ActiveAgent
|
|
|
190
192
|
# the provider runs that call inside the agent's `tool_observer`, which
|
|
191
193
|
# records it the same way. The two paths are disjoint, so a call is
|
|
192
194
|
# recorded once, and the observer sees the call without making it again.
|
|
195
|
+
#
|
|
196
|
+
# An observer the agent already has is kept, as tools_function's `base`
|
|
197
|
+
# is: it runs around the call this records, so it sees the same name,
|
|
198
|
+
# arguments, result and exception, and a call it does not make is not
|
|
199
|
+
# recorded as made.
|
|
193
200
|
def record_tool_calls(agent)
|
|
194
201
|
calls = @tool_calls
|
|
195
202
|
base = agent.tools_function
|
|
203
|
+
observer = agent.tool_observer
|
|
196
204
|
capture = self.class
|
|
197
205
|
|
|
198
206
|
agent.define_singleton_method(:tools_function) do
|
|
@@ -203,7 +211,10 @@ module ActiveAgent
|
|
|
203
211
|
end
|
|
204
212
|
|
|
205
213
|
agent.define_singleton_method(:tool_observer) do
|
|
206
|
-
proc { |name, arguments, &call| capture.record_call(calls, name, arguments.presence, &call) }
|
|
214
|
+
record = proc { |name, arguments, &call| capture.record_call(calls, name, arguments.presence, &call) }
|
|
215
|
+
next record unless observer
|
|
216
|
+
|
|
217
|
+
proc { |name, arguments, &call| observer.call(name, arguments) { record.call(name, arguments, &call) } }
|
|
207
218
|
end
|
|
208
219
|
end
|
|
209
220
|
|
|
@@ -81,13 +81,16 @@ module ActiveAgent
|
|
|
81
81
|
end
|
|
82
82
|
|
|
83
83
|
# Parses YAML text. Raises InvalidDocument for YAML that is not a
|
|
84
|
-
# document (a list, a scalar) or that YAML cannot read
|
|
84
|
+
# document (a list, a scalar) or that YAML cannot read: a syntax error,
|
|
85
|
+
# and what safe loading refuses — a tag naming a Ruby class other than
|
|
86
|
+
# Symbol (Psych::DisallowedClass) or an alias to an undefined anchor
|
|
87
|
+
# (Psych::BadAlias), which are not Psych::SyntaxError.
|
|
85
88
|
def self.parse(text, name: nil)
|
|
86
89
|
document = YAML.safe_load(text.to_s, aliases: true, permitted_classes: [ Symbol ])
|
|
87
90
|
raise InvalidDocument, "a scenario catalog is a YAML mapping" unless document.is_a?(Hash)
|
|
88
91
|
|
|
89
92
|
new([ document ], name: name)
|
|
90
|
-
rescue Psych::
|
|
93
|
+
rescue Psych::Exception => e
|
|
91
94
|
raise InvalidDocument, "the catalog is not valid YAML: #{e.message}"
|
|
92
95
|
end
|
|
93
96
|
|
|
@@ -45,7 +45,9 @@ module ActiveAgent
|
|
|
45
45
|
end
|
|
46
46
|
|
|
47
47
|
# The text a failing assertion prints. The first line counts the gates
|
|
48
|
-
# not met, or is the one line "every pass gate met" when none is
|
|
48
|
+
# not met, or is the one line "every pass gate met" when none is (a
|
|
49
|
+
# report whose every row is an ungated group's says that instead: its
|
|
50
|
+
# rows are met because nothing held them to a bar); then
|
|
49
51
|
# each unmet gate as Gate#to_s; then, after a blank line, the failures
|
|
50
52
|
# to read: for each group with an unmet gate, in gates order, "<group>
|
|
51
53
|
# failures:" over Report#failures_markdown for that group. An unmet All
|
|
@@ -59,6 +61,7 @@ module ActiveAgent
|
|
|
59
61
|
return "no result reached a pass gate; the run evaluated nothing" if all.empty?
|
|
60
62
|
|
|
61
63
|
unmet = all.reject(&:met?)
|
|
64
|
+
return "every group the run covered is ungated; no answer was held to a bar" if all.none?(&:gated)
|
|
62
65
|
return "every pass gate met" if unmet.empty?
|
|
63
66
|
|
|
64
67
|
lines = [ "#{unmet.size} of #{all.size} pass gates not met", *unmet.map { |gate| "- #{gate}" } ]
|
|
@@ -17,7 +17,7 @@ module ActiveAgent
|
|
|
17
17
|
# @example Disable sanitization entirely
|
|
18
18
|
# ActiveAgent::ModelCapabilities.enabled = false
|
|
19
19
|
module ModelCapabilities
|
|
20
|
-
SAMPLING_PARAMS = [ :temperature, :top_p ].freeze
|
|
20
|
+
SAMPLING_PARAMS = [ :temperature, :top_p, :top_k ].freeze
|
|
21
21
|
|
|
22
22
|
# Model families that reject sampling parameters with a 400:
|
|
23
23
|
# - Anthropic thinking-first models (Opus 4.7+, Opus 5, Sonnet 5,
|
|
@@ -10,7 +10,7 @@ require_relative "concerns/tool_choice_clearing"
|
|
|
10
10
|
|
|
11
11
|
# @private
|
|
12
12
|
GEM_LOADERS = {
|
|
13
|
-
anthropic: [ "anthropic", "
|
|
13
|
+
anthropic: [ "anthropic", [ ">= 1.12", "< 3" ], "anthropic" ],
|
|
14
14
|
openai: [ "openai", "~> 0.34", "openai" ],
|
|
15
15
|
# Keep a tested floor: 1.2 lacks the provider APIs this adapter uses.
|
|
16
16
|
ruby_llm: [ "ruby_llm", [ ">= 1.16", "< 3" ], "ruby_llm" ]
|
|
@@ -197,6 +197,7 @@ module ActiveAgent
|
|
|
197
197
|
when :message_start
|
|
198
198
|
api_message = Anthropic::Transforms.gem_to_hash(api_response_chunk.message)
|
|
199
199
|
self.message_stack.push(api_message)
|
|
200
|
+
record_stream_usage(api_message[:usage])
|
|
200
201
|
broadcast_stream_update(message_stack.last)
|
|
201
202
|
|
|
202
203
|
# -> Content Block Create
|
|
@@ -239,6 +240,15 @@ module ActiveAgent
|
|
|
239
240
|
self.message_stack.last.merge!(delta)
|
|
240
241
|
self.stream_finish_reason = delta[:stop_reason] if delta[:stop_reason]
|
|
241
242
|
|
|
243
|
+
# message_delta carries the cumulative output tokens (and, on newer
|
|
244
|
+
# API versions, refreshed input counts); layer them over the usage
|
|
245
|
+
# message_start reported so the turn keeps a single running total.
|
|
246
|
+
delta_usage = Anthropic::Transforms.gem_to_hash(api_response_chunk.usage)&.compact
|
|
247
|
+
if delta_usage.present?
|
|
248
|
+
message_stack.last[:usage] = (message_stack.last[:usage] || {}).merge(delta_usage)
|
|
249
|
+
record_stream_usage(message_stack.last[:usage])
|
|
250
|
+
end
|
|
251
|
+
|
|
242
252
|
# Message Completed [Full Message]
|
|
243
253
|
when :message_stop
|
|
244
254
|
api_message = Anthropic::Transforms.gem_to_hash(api_response_chunk.message)
|
|
@@ -51,7 +51,8 @@ module ActiveAgent
|
|
|
51
51
|
)
|
|
52
52
|
|
|
53
53
|
@messages = ::Anthropic::Resources::Messages.new(client: self)
|
|
54
|
-
|
|
54
|
+
# The anthropic gem 2.0 removed Text Completions along with this resource.
|
|
55
|
+
@completions = ::Anthropic::Resources::Completions.new(client: self) if defined?(::Anthropic::Resources::Completions)
|
|
55
56
|
@beta = ::Anthropic::Resources::Beta.new(client: self)
|
|
56
57
|
end
|
|
57
58
|
|
|
@@ -148,7 +148,8 @@ module ActiveAgent
|
|
|
148
148
|
cached_tokens: sum_optional(self.cached_tokens, other.cached_tokens),
|
|
149
149
|
cache_creation_tokens: sum_optional(self.cache_creation_tokens, other.cache_creation_tokens),
|
|
150
150
|
reasoning_tokens: sum_optional(self.reasoning_tokens, other.reasoning_tokens),
|
|
151
|
-
audio_tokens: sum_optional(self.audio_tokens, other.audio_tokens)
|
|
151
|
+
audio_tokens: sum_optional(self.audio_tokens, other.audio_tokens),
|
|
152
|
+
duration_ms: sum_optional(self.duration_ms, other.duration_ms)
|
|
152
153
|
)
|
|
153
154
|
end
|
|
154
155
|
|
|
@@ -263,6 +264,10 @@ module ActiveAgent
|
|
|
263
264
|
**usage.slice(:input_tokens, :output_tokens, :service_tier),
|
|
264
265
|
input_tokens: usage[:input_tokens] || 0,
|
|
265
266
|
output_tokens: usage[:output_tokens] || 0,
|
|
267
|
+
# Anthropic's input_tokens excludes cache reads and writes, which are
|
|
268
|
+
# reported separately, so they are added to get every token processed.
|
|
269
|
+
total_tokens: (usage[:input_tokens] || 0) + (usage[:output_tokens] || 0) +
|
|
270
|
+
(usage[:cache_read_input_tokens] || 0) + (usage[:cache_creation_input_tokens] || 0),
|
|
266
271
|
cached_tokens: usage[:cache_read_input_tokens],
|
|
267
272
|
cache_creation_tokens: usage[:cache_creation_input_tokens],
|
|
268
273
|
provider_details: usage.slice(:cache_creation, :server_tool_use).compact
|
|
@@ -110,6 +110,7 @@ module ActiveAgent
|
|
|
110
110
|
# reasoning progress, kept from the reasoning item once it is done
|
|
111
111
|
# - `:"response.content_part.done"` - content part completed
|
|
112
112
|
# - `:"response.output_item.done"` - message, reasoning or function call completed
|
|
113
|
+
# - `:error`, `:"response.failed"` - provider failure
|
|
113
114
|
# - `:"response.completed"` - response finished
|
|
114
115
|
# - `:"response.incomplete"` - response cut short (e.g. max_output_tokens)
|
|
115
116
|
#
|
|
@@ -174,6 +175,12 @@ module ActiveAgent
|
|
|
174
175
|
when :"response.output_item.done"
|
|
175
176
|
process_stream_output_item_done(api_response_event)
|
|
176
177
|
|
|
178
|
+
# Errors arrive as events even though the HTTP request succeeded.
|
|
179
|
+
when :error, :"response.failed"
|
|
180
|
+
error = api_response_event.type == :error ? api_response_event : api_response_event.response.error
|
|
181
|
+
message = [ error&.code, error&.message ].compact.join(": ")
|
|
182
|
+
raise Errors::ProviderError.new(message.presence || "OpenAI response failed", provider_tag: tag_name)
|
|
183
|
+
|
|
177
184
|
# Response Completed, or cut short (e.g. by max_output_tokens) with
|
|
178
185
|
# the output it has so far
|
|
179
186
|
when :"response.completed", :"response.incomplete"
|
|
@@ -50,6 +50,14 @@ module ActiveAgent
|
|
|
50
50
|
|
|
51
51
|
protected
|
|
52
52
|
|
|
53
|
+
# Executes an embedding request via Requesty's OpenAI-compatible API.
|
|
54
|
+
#
|
|
55
|
+
# @param parameters [Hash]
|
|
56
|
+
# @return [Hash] symbolized API response
|
|
57
|
+
def api_embed_execute(parameters)
|
|
58
|
+
client.embeddings.create(**parameters).as_json.deep_symbolize_keys
|
|
59
|
+
end
|
|
60
|
+
|
|
53
61
|
# @see BaseProvider#api_response_normalize
|
|
54
62
|
# @param api_response [OpenAI::Models::ChatCompletion]
|
|
55
63
|
# @return [Hash] normalized response hash
|
|
@@ -32,6 +32,21 @@ module ActiveAgent
|
|
|
32
32
|
alias_method :declared_parameters, :parameters
|
|
33
33
|
alias_method :provider_options, :provider_params
|
|
34
34
|
|
|
35
|
+
# RubyLLM 2.2 registers a tool under its qualified name (namespace
|
|
36
|
+
# plus name) and asks each tool whether it defers. ActiveAgent's
|
|
37
|
+
# tools have no namespace and run when called.
|
|
38
|
+
def namespace
|
|
39
|
+
nil
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def qualified_name
|
|
43
|
+
name
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def deferred?
|
|
47
|
+
false
|
|
48
|
+
end
|
|
49
|
+
|
|
35
50
|
private
|
|
36
51
|
|
|
37
52
|
def deep_stringify(obj)
|
|
@@ -152,15 +152,16 @@ module ActiveAgent
|
|
|
152
152
|
# Handle tool calls in chunk
|
|
153
153
|
if chunk.tool_calls&.any?
|
|
154
154
|
message[:tool_calls] ||= []
|
|
155
|
-
chunk.tool_calls
|
|
155
|
+
each_tool_call(chunk.tool_calls) do |key, tool_call|
|
|
156
156
|
# Only the first delta of a call has its ID and name; the rest
|
|
157
157
|
# are fragments of its arguments. A fragment belongs to the call
|
|
158
158
|
# that shares its stream key (the index RubyLLM 2 numbers calls
|
|
159
159
|
# by), or, without one as RubyLLM 1.16 leaves it for OpenAI, to
|
|
160
160
|
# the latest call. RubyLLM 1.16 cannot preserve interleaved
|
|
161
161
|
# OpenAI calls: its parser drops their indices and fragments.
|
|
162
|
-
|
|
163
|
-
|
|
162
|
+
call_id = streamed_tool_call_id(tool_call)
|
|
163
|
+
existing = if call_id
|
|
164
|
+
message[:tool_calls].find { |tc| tc[:id] == call_id }
|
|
164
165
|
else
|
|
165
166
|
@stream_tool_calls[key] || message[:tool_calls].last
|
|
166
167
|
end
|
|
@@ -169,7 +170,7 @@ module ActiveAgent
|
|
|
169
170
|
existing[:function][:arguments] += fragment
|
|
170
171
|
else
|
|
171
172
|
existing = {
|
|
172
|
-
id:
|
|
173
|
+
id: call_id,
|
|
173
174
|
type: "function",
|
|
174
175
|
function: {
|
|
175
176
|
name: tool_call.name,
|
|
@@ -380,6 +381,7 @@ module ActiveAgent
|
|
|
380
381
|
when String
|
|
381
382
|
content
|
|
382
383
|
when Array
|
|
384
|
+
refuse_media_blocks!(content)
|
|
383
385
|
content.select { |block| block.is_a?(Hash) && block[:type] == "text" }
|
|
384
386
|
.map { |block| block[:text] }
|
|
385
387
|
.join("\n")
|
|
@@ -388,6 +390,26 @@ module ActiveAgent
|
|
|
388
390
|
end
|
|
389
391
|
end
|
|
390
392
|
|
|
393
|
+
# Content block types that carry media rather than text, in the shapes
|
|
394
|
+
# the OpenAI (Chat and Responses) and Anthropic transforms write.
|
|
395
|
+
MEDIA_BLOCK_TYPES = %w[image image_url input_image audio input_audio file input_file document video].freeze
|
|
396
|
+
|
|
397
|
+
# This provider sends RubyLLM text only, so an image, audio clip or
|
|
398
|
+
# file would be dropped while the model answered as if it had seen
|
|
399
|
+
# it. Such a request is refused before any provider call. Other
|
|
400
|
+
# blocks that are not text (an earlier turn's reasoning) are still
|
|
401
|
+
# left out.
|
|
402
|
+
def refuse_media_blocks!(content)
|
|
403
|
+
types = content.filter_map { |block| block[:type].to_s if block.is_a?(Hash) && MEDIA_BLOCK_TYPES.include?(block[:type].to_s) }.uniq
|
|
404
|
+
return if types.empty?
|
|
405
|
+
|
|
406
|
+
raise Errors::InvalidRequest.new(
|
|
407
|
+
"The RubyLLM provider sends text only; this message has #{types.join(', ')} content, which it would drop. " \
|
|
408
|
+
"Send it through a provider that accepts it, or as text.",
|
|
409
|
+
provider_tag: "RubyLLM"
|
|
410
|
+
)
|
|
411
|
+
end
|
|
412
|
+
|
|
391
413
|
# Converts ActiveAgent tool_calls to RubyLLM's ToolCall format.
|
|
392
414
|
#
|
|
393
415
|
# @param tool_calls [Array<Hash>] ActiveAgent format tool calls
|
|
@@ -492,7 +514,7 @@ module ActiveAgent
|
|
|
492
514
|
|
|
493
515
|
# Handle tool calls
|
|
494
516
|
if response.tool_calls&.any?
|
|
495
|
-
hash[:tool_calls] = response.tool_calls.map do |id, tc|
|
|
517
|
+
hash[:tool_calls] = each_tool_call(response.tool_calls).map do |id, tc|
|
|
496
518
|
{
|
|
497
519
|
id: id,
|
|
498
520
|
type: "function",
|
|
@@ -523,6 +545,41 @@ module ActiveAgent
|
|
|
523
545
|
hash
|
|
524
546
|
end
|
|
525
547
|
|
|
548
|
+
# Yields each of a message's tool calls with its key. RubyLLM 1.x and
|
|
549
|
+
# 2.0 to 2.1 keep them in a Hash keyed by call ID (or, on a streamed
|
|
550
|
+
# chunk, by stream index), so iterating yields pairs. RubyLLM 2.2's
|
|
551
|
+
# `ToolCall::Collection` yields the calls themselves and keeps the
|
|
552
|
+
# keys behind `each_key`, in the same order.
|
|
553
|
+
#
|
|
554
|
+
# @param calls [Hash, #each_key, nil]
|
|
555
|
+
# @return [Enumerator, Array] key and call pairs
|
|
556
|
+
def each_tool_call(calls, &block)
|
|
557
|
+
pairs = if calls.nil?
|
|
558
|
+
[]
|
|
559
|
+
elsif calls.is_a?(Hash)
|
|
560
|
+
calls.to_a
|
|
561
|
+
elsif calls.respond_to?(:each_key)
|
|
562
|
+
calls.each_key.to_a.zip(calls.each.to_a)
|
|
563
|
+
else
|
|
564
|
+
calls.each.to_a.map { |call| [ call.id, call ] }
|
|
565
|
+
end
|
|
566
|
+
block ? pairs.each(&block) : pairs
|
|
567
|
+
end
|
|
568
|
+
|
|
569
|
+
# The provider's ID for a streamed tool call, present on its first
|
|
570
|
+
# delta and nil on the fragments. RubyLLM 2.2 keeps it in
|
|
571
|
+
# `provider_call_id` and leaves `id` (its own identifier, assigned when
|
|
572
|
+
# the message completes) nil on a chunk; earlier releases put the
|
|
573
|
+
# provider's ID in `id`.
|
|
574
|
+
#
|
|
575
|
+
# @param tool_call [RubyLLM::ToolCall]
|
|
576
|
+
# @return [String, nil]
|
|
577
|
+
def streamed_tool_call_id(tool_call)
|
|
578
|
+
provider_id = tool_call.provider_call_id if tool_call.respond_to?(:provider_call_id)
|
|
579
|
+
provider_id = nil if provider_id.to_s.empty?
|
|
580
|
+
provider_id || tool_call.id
|
|
581
|
+
end
|
|
582
|
+
|
|
526
583
|
# A streamed tool call's arguments as the text they add to the JSON
|
|
527
584
|
# string the call builds up. Most APIs stream that string; Anthropic
|
|
528
585
|
# opens a call with an empty Hash for its input, and Gemini sends the
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "../open_ai/chat/_types"
|
|
4
|
+
require_relative "options"
|
|
5
|
+
|
|
6
|
+
module ActiveAgent
|
|
7
|
+
module Providers
|
|
8
|
+
module Runcrate
|
|
9
|
+
# ActiveModel type for casting and serializing Runcrate requests.
|
|
10
|
+
#
|
|
11
|
+
# Runcrate is OpenAI-compatible, so requests use the same shape as the
|
|
12
|
+
# OpenAI Chat API. This delegates entirely to
|
|
13
|
+
# OpenAI::Chat::RequestType.
|
|
14
|
+
RequestType = ActiveAgent::Providers::OpenAI::Chat::RequestType
|
|
15
|
+
end
|
|
16
|
+
end
|
|
17
|
+
end
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "../open_ai/options"
|
|
4
|
+
|
|
5
|
+
module ActiveAgent
|
|
6
|
+
module Providers
|
|
7
|
+
module Runcrate
|
|
8
|
+
# Configuration options for the Runcrate provider.
|
|
9
|
+
#
|
|
10
|
+
# Extends OpenAI::Options, overriding the base URL to point at
|
|
11
|
+
# Runcrate's OpenAI-compatible inference API and resolving the API key
|
|
12
|
+
# from RUNCRATE_API_KEY. Runcrate does not use organization or project
|
|
13
|
+
# identifiers.
|
|
14
|
+
#
|
|
15
|
+
# @example Basic configuration
|
|
16
|
+
# options = Options.new(api_key: ENV["RUNCRATE_API_KEY"])
|
|
17
|
+
#
|
|
18
|
+
# @see https://runcrate.ai/docs/models/overview
|
|
19
|
+
# @see https://runcrate.ai/dashboard/api-keys Runcrate API Keys
|
|
20
|
+
class Options < ActiveAgent::Providers::OpenAI::Options
|
|
21
|
+
# @!attribute base_url
|
|
22
|
+
# @return [String] API endpoint (default: "https://api.runcrate.ai/v1")
|
|
23
|
+
attribute :base_url, :string, as: "https://api.runcrate.ai/v1"
|
|
24
|
+
|
|
25
|
+
private
|
|
26
|
+
|
|
27
|
+
def resolve_api_key(kwargs)
|
|
28
|
+
kwargs[:api_key] ||
|
|
29
|
+
kwargs[:access_token] ||
|
|
30
|
+
ENV["RUNCRATE_API_KEY"]
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
# Not used as part of Runcrate
|
|
34
|
+
def resolve_organization_id(kwargs) = nil
|
|
35
|
+
def resolve_project_id(kwargs) = nil
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
end
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
require_relative "_base_provider"
|
|
2
|
+
|
|
3
|
+
require_gem!(:openai, __FILE__)
|
|
4
|
+
|
|
5
|
+
require_relative "open_ai_provider"
|
|
6
|
+
require_relative "runcrate/_types"
|
|
7
|
+
|
|
8
|
+
module ActiveAgent
|
|
9
|
+
module Providers
|
|
10
|
+
# Provides access to Runcrate's OpenAI-compatible inference API for
|
|
11
|
+
# open-weight models.
|
|
12
|
+
#
|
|
13
|
+
# Extends the OpenAI provider to work with Runcrate's OpenAI-compatible
|
|
14
|
+
# API. Model ids are the vendor-prefixed names Runcrate lists at
|
|
15
|
+
# +/v1/models+ (e.g. +deepseek-ai/DeepSeek-V4.1-Flash+).
|
|
16
|
+
#
|
|
17
|
+
# Runcrate is a plain OpenAI-compatible API: requests, responses and
|
|
18
|
+
# transforms are identical to the OpenAI Chat API, so this provider
|
|
19
|
+
# reuses OpenAI::Chat::RequestType and OpenAI::Chat::Transforms directly.
|
|
20
|
+
# The only Runcrate-specific configuration is the base URL and API key,
|
|
21
|
+
# which live in Runcrate::Options.
|
|
22
|
+
#
|
|
23
|
+
# @example Configuration in active_agent.yml
|
|
24
|
+
# runcrate:
|
|
25
|
+
# service: "Runcrate"
|
|
26
|
+
# api_key: <%= ENV["RUNCRATE_API_KEY"] %>
|
|
27
|
+
# model: "deepseek-ai/DeepSeek-V4.1-Flash"
|
|
28
|
+
#
|
|
29
|
+
# @see OpenAI::ChatProvider
|
|
30
|
+
# @see https://runcrate.ai/docs/models/overview
|
|
31
|
+
class RuncrateProvider < OpenAI::ChatProvider
|
|
32
|
+
# @return [String]
|
|
33
|
+
def self.service_name
|
|
34
|
+
"Runcrate"
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# @return [Class]
|
|
38
|
+
def self.options_klass
|
|
39
|
+
Runcrate::Options
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# @return [ActiveModel::Type::Value]
|
|
43
|
+
def self.prompt_request_type
|
|
44
|
+
OpenAI::Chat::RequestType.new
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
# @return [ActiveModel::Type::Value]
|
|
48
|
+
def self.embed_request_type
|
|
49
|
+
OpenAI::Embedding::RequestType.new
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
protected
|
|
53
|
+
|
|
54
|
+
# Executes an embedding request via Runcrate's OpenAI-compatible API.
|
|
55
|
+
#
|
|
56
|
+
# @param parameters [Hash]
|
|
57
|
+
# @return [Hash] symbolized API response
|
|
58
|
+
def api_embed_execute(parameters)
|
|
59
|
+
client.embeddings.create(**parameters).as_json.deep_symbolize_keys
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
# @see BaseProvider#api_response_normalize
|
|
63
|
+
# @param api_response [OpenAI::Models::ChatCompletion]
|
|
64
|
+
# @return [Hash] normalized response hash
|
|
65
|
+
def api_response_normalize(api_response)
|
|
66
|
+
return api_response unless api_response
|
|
67
|
+
|
|
68
|
+
OpenAI::Chat::Transforms.gem_to_hash(api_response)
|
|
69
|
+
end
|
|
70
|
+
end
|
|
71
|
+
end
|
|
72
|
+
end
|
|
@@ -83,6 +83,13 @@ module ActiveAgent
|
|
|
83
83
|
"on_or_before" => :lteq, "on_or_after" => :gteq
|
|
84
84
|
}.freeze
|
|
85
85
|
|
|
86
|
+
# Asks whether a column holds a value at all: `{ "present" => false }` is
|
|
87
|
+
# IS NULL, `{ "present" => true }` IS NOT NULL. A nil filter value cannot
|
|
88
|
+
# ask it, because nil means "no filter" (models send null for the
|
|
89
|
+
# optional parameters they leave out), so without this "tickets with no
|
|
90
|
+
# assignee" or "requests not yet approved" has no filter that says so.
|
|
91
|
+
PRESENCE_OPERATOR = "present"
|
|
92
|
+
|
|
86
93
|
# Column names that look like they hold a credential. The schema tools
|
|
87
94
|
# generator never suggests one, and the dashboard never lets one be
|
|
88
95
|
# chosen: reading one back would hand a model a credential, and
|
|
@@ -329,7 +336,8 @@ module ActiveAgent
|
|
|
329
336
|
#
|
|
330
337
|
# A filter value is normally matched for equality. A Hash value instead
|
|
331
338
|
# declares a range — `{ "before" => "2026-01-01" }`, `{ "gte" => 10 }` —
|
|
332
|
-
# and may carry two bounds at once to express a window
|
|
339
|
+
# and may carry two bounds at once to express a window, or asks whether
|
|
340
|
+
# the column is set at all (`{ "present" => false }`).
|
|
333
341
|
#
|
|
334
342
|
# @api private
|
|
335
343
|
# @raise [UnpermittedAttribute] if any key is not declared filterable,
|
|
@@ -389,17 +397,37 @@ module ActiveAgent
|
|
|
389
397
|
type = @model.type_for_attribute(column)
|
|
390
398
|
|
|
391
399
|
value.map do |operator, operand|
|
|
400
|
+
next presence_predicate!(column, arel, operand) if operator.to_s == PRESENCE_OPERATOR
|
|
401
|
+
|
|
392
402
|
predicate = RANGE_OPERATORS[operator.to_s]
|
|
393
403
|
unless predicate
|
|
394
404
|
raise UnpermittedAttribute,
|
|
395
405
|
"`#{operator}` is not a valid comparison for `#{column}`. " \
|
|
396
|
-
"Allowed comparisons: #{RANGE_OPERATORS.keys.join(", ")}"
|
|
406
|
+
"Allowed comparisons: #{[ *RANGE_OPERATORS.keys, PRESENCE_OPERATOR ].join(", ")}"
|
|
397
407
|
end
|
|
398
408
|
|
|
399
409
|
arel.public_send(predicate, type.cast(operand))
|
|
400
410
|
end
|
|
401
411
|
end
|
|
402
412
|
|
|
413
|
+
# Builds IS NULL / IS NOT NULL for `{ "present" => ... }`.
|
|
414
|
+
#
|
|
415
|
+
# Only true and false (or their strings) are accepted: anything else
|
|
416
|
+
# would have to be guessed into one of the two, and a wrong guess
|
|
417
|
+
# answers the opposite question.
|
|
418
|
+
#
|
|
419
|
+
# @api private
|
|
420
|
+
# @raise [UnpermittedAttribute] on any other operand
|
|
421
|
+
def presence_predicate!(column, arel, operand)
|
|
422
|
+
case operand.to_s
|
|
423
|
+
when "true" then arel.not_eq(nil)
|
|
424
|
+
when "false" then arel.eq(nil)
|
|
425
|
+
else
|
|
426
|
+
raise UnpermittedAttribute,
|
|
427
|
+
"`#{PRESENCE_OPERATOR}` for `#{column}` takes true or false, not #{operand.inspect}"
|
|
428
|
+
end
|
|
429
|
+
end
|
|
430
|
+
|
|
403
431
|
# A value outside a Rails enum reaches `where` as an unmatched name and
|
|
404
432
|
# returns zero rows — the same silent-nothing that {.range_predicates!}
|
|
405
433
|
# rejects for an unknown operator, and just as bad here: an agent reads
|
|
@@ -415,9 +443,12 @@ module ActiveAgent
|
|
|
415
443
|
return if values.blank?
|
|
416
444
|
|
|
417
445
|
if value.is_a?(Hash)
|
|
446
|
+
# Whether it is set at all is a question an enum can answer.
|
|
447
|
+
return if value.keys.map(&:to_s) == [ PRESENCE_OPERATOR ]
|
|
448
|
+
|
|
418
449
|
raise UnpermittedAttribute,
|
|
419
450
|
"`#{column}` is an enum and cannot be compared as a range. " \
|
|
420
|
-
"Allowed values: #{values.keys.join(", ")}"
|
|
451
|
+
"Allowed values: #{values.keys.join(", ")}, or {\"#{PRESENCE_OPERATOR}\": true|false}"
|
|
421
452
|
end
|
|
422
453
|
|
|
423
454
|
# An Array is an IN filter, which `where` silently narrows to its
|
|
@@ -550,20 +581,28 @@ module ActiveAgent
|
|
|
550
581
|
# only `{type: "string", format: "date"}` has no way to know it may ask
|
|
551
582
|
# for `before`, and answers date questions with an equality match or no
|
|
552
583
|
# filter at all.
|
|
584
|
+
#
|
|
585
|
+
# A nullable column's range object also takes `present`, so "has no
|
|
586
|
+
# due date" is asked in the same shape as "due before Friday". It is
|
|
587
|
+
# offered only here, on the columns that already carry an object form,
|
|
588
|
+
# rather than wrapping every nullable string and enum in one; a call
|
|
589
|
+
# may still send it for any filterable column.
|
|
553
590
|
def with_range_form(column, scalar)
|
|
554
591
|
operand = scalar.slice(:type, :format)
|
|
555
|
-
|
|
592
|
+
nullable = @model.columns_hash[column.to_s]&.null
|
|
593
|
+
properties = RANGE_OPERATORS.keys.index_with { operand.dup }
|
|
594
|
+
properties[PRESENCE_OPERATOR] = { type: "boolean" } if nullable
|
|
595
|
+
|
|
596
|
+
description = [ scalar[:description], "Accepts an exact value, or a range object such as " \
|
|
597
|
+
"{\"before\": ...} / {\"gte\": ...} (#{RANGE_OPERATORS.keys.join(", ")})." ]
|
|
598
|
+
description << "{\"#{PRESENCE_OPERATOR}\": false} matches records where it is empty (null), " \
|
|
599
|
+
"{\"#{PRESENCE_OPERATOR}\": true} where it is set." if nullable
|
|
556
600
|
|
|
557
601
|
{
|
|
558
|
-
description:
|
|
559
|
-
"{\"before\": ...} / {\"gte\": ...} (#{RANGE_OPERATORS.keys.join(", ")})." ].compact.join(" "),
|
|
602
|
+
description: description.compact.join(" "),
|
|
560
603
|
anyOf: [
|
|
561
604
|
scalar.except(:description),
|
|
562
|
-
{
|
|
563
|
-
type: "object",
|
|
564
|
-
properties: RANGE_OPERATORS.keys.index_with { operand.dup },
|
|
565
|
-
additionalProperties: false
|
|
566
|
-
}
|
|
605
|
+
{ type: "object", properties: properties, additionalProperties: false }
|
|
567
606
|
]
|
|
568
607
|
}
|
|
569
608
|
end
|
data/lib/active_agent/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: activeagent
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.9.
|
|
4
|
+
version: 1.9.9
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Justin Bowen
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-11-
|
|
11
|
+
date: 2026-11-19 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: actionpack
|
|
@@ -162,16 +162,22 @@ dependencies:
|
|
|
162
162
|
name: anthropic
|
|
163
163
|
requirement: !ruby/object:Gem::Requirement
|
|
164
164
|
requirements:
|
|
165
|
-
- - "
|
|
165
|
+
- - ">="
|
|
166
166
|
- !ruby/object:Gem::Version
|
|
167
167
|
version: '1.12'
|
|
168
|
+
- - "<"
|
|
169
|
+
- !ruby/object:Gem::Version
|
|
170
|
+
version: '3'
|
|
168
171
|
type: :development
|
|
169
172
|
prerelease: false
|
|
170
173
|
version_requirements: !ruby/object:Gem::Requirement
|
|
171
174
|
requirements:
|
|
172
|
-
- - "
|
|
175
|
+
- - ">="
|
|
173
176
|
- !ruby/object:Gem::Version
|
|
174
177
|
version: '1.12'
|
|
178
|
+
- - "<"
|
|
179
|
+
- !ruby/object:Gem::Version
|
|
180
|
+
version: '3'
|
|
175
181
|
- !ruby/object:Gem::Dependency
|
|
176
182
|
name: aws-sdk-bedrockruntime
|
|
177
183
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -630,6 +636,9 @@ files:
|
|
|
630
636
|
- lib/active_agent/providers/ruby_llm/tool_proxy.rb
|
|
631
637
|
- lib/active_agent/providers/ruby_llm_provider.rb
|
|
632
638
|
- lib/active_agent/providers/rubyllm_provider.rb
|
|
639
|
+
- lib/active_agent/providers/runcrate/_types.rb
|
|
640
|
+
- lib/active_agent/providers/runcrate/options.rb
|
|
641
|
+
- lib/active_agent/providers/runcrate_provider.rb
|
|
633
642
|
- lib/active_agent/railtie.rb
|
|
634
643
|
- lib/active_agent/railtie/schema_generator_extension.rb
|
|
635
644
|
- lib/active_agent/schema_generator.rb
|