activeagent 1.9.8 → 1.9.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +82 -0
- data/lib/active_agent/cli/eval_commands.rb +7 -1
- data/lib/active_agent/evals/catalog.rb +5 -2
- data/lib/active_agent/evals/gate_check.rb +4 -1
- data/lib/active_agent/providers/ruby_llm_provider.rb +21 -0
- data/lib/active_agent/schema_tools.rb +50 -11
- data/lib/active_agent/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e5cf4f1304cb0d939f96e270b950dbe8bc3505e6e4d8dacba9f0f16083c1b485
|
|
4
|
+
data.tar.gz: 7716fbc50b2e06ac508cf88a63b8515bbec0f17d6ab632b57a05706f0fe1187b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f237aff2404b739fa66c77e0d5a6e2925be257423fc2072ae7b1387c00a144cf0fdbb95466237981170bcf1f2f9b468c0f592bdc5d3442522300fc0dce6b581a
|
|
7
|
+
data.tar.gz: ef1161bd9b4d39338594946354ce0f455f212f9508ff0a8f0119fd2762fb965a7b32798a3913a2473a27424c6c918e7102d3a14e78233d4b67a96f1771aac108
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,88 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [1.9.9] - 2026-11-18
|
|
9
|
+
|
|
10
|
+
Schema tools can ask whether a column is set: a filter takes
|
|
11
|
+
`{"present": false}` (IS NULL) or `{"present": true}` (IS NOT NULL), so a
|
|
12
|
+
model can find the tickets with no assignee without a sentinel range in
|
|
13
|
+
the instructions. Around that, fixes to evaluations and providers: an
|
|
14
|
+
evaluation run and a fanned-out replay are each claimed before they
|
|
15
|
+
execute, so a job delivered twice no longer replays the suite or calls
|
|
16
|
+
the provider twice; a catalog document safe loading refuses is a 422
|
|
17
|
+
instead of a 500; a run of only ungated groups no longer reads as "every
|
|
18
|
+
gate met"; and the RubyLLM provider refuses media it would drop.
|
|
19
|
+
|
|
20
|
+
Upgrading: no migrations. The RubyLLM provider now raises
|
|
21
|
+
`ActiveAgent::Providers::Errors::InvalidRequest` for a message carrying an
|
|
22
|
+
image, audio or a file, where it used to send the text alone; route such
|
|
23
|
+
messages to a provider that takes media.
|
|
24
|
+
|
|
25
|
+
### Added
|
|
26
|
+
|
|
27
|
+
- **Schema tools can ask whether a column is set** (`activeagent`). A
|
|
28
|
+
filter value of `null` means "no filter", so a model had no way to ask
|
|
29
|
+
for the tickets with no assignee or the requests not yet approved, and
|
|
30
|
+
hosts wrote sentinel ranges such as `{"after": "2000-01-01"}` into their
|
|
31
|
+
instructions instead. A filter now takes `{"present": false}` (IS NULL)
|
|
32
|
+
or `{"present": true}` (IS NOT NULL) on any filterable column, alone or
|
|
33
|
+
beside a range bound. The tool schema offers it in the range object of
|
|
34
|
+
nullable date, time and number columns, so foreign keys get it too, and
|
|
35
|
+
an enum accepts it though it takes no comparison. Anything but true or
|
|
36
|
+
false is rejected rather than guessed.
|
|
37
|
+
|
|
38
|
+
### Fixed
|
|
39
|
+
|
|
40
|
+
- **An evaluation run is executed by one job** (`actionagent`). The run job
|
|
41
|
+
read a run's status, then the scenario runner wrote `running`, as two
|
|
42
|
+
steps, so a job delivered twice, or two workers that both loaded the run
|
|
43
|
+
as pending, each replayed the whole suite, or with
|
|
44
|
+
`scenario_evaluation_fan_out` on each enqueued a full set of replay jobs.
|
|
45
|
+
The runner now moves the run from pending to running in one conditional
|
|
46
|
+
update, as the execution job claims an agent run, and replays nothing
|
|
47
|
+
unless it was the one to do so; this covers `ProjectEvaluationJob` too.
|
|
48
|
+
The claim comes before the selection is read, so a selection that cannot
|
|
49
|
+
be resolved (an unsupported provider in a re-run's models) now fails the
|
|
50
|
+
run instead of leaving it pending forever.
|
|
51
|
+
- **A fanned-out replay is claimed before the agent runs** (`actionagent`).
|
|
52
|
+
`EvaluationReplayJob` checked for a stored result, replayed, and only
|
|
53
|
+
then took the run's lock to store, so two deliveries of one replay job
|
|
54
|
+
both called the provider (and the agent's tools) and the second answer
|
|
55
|
+
was dropped. A delivery that failed after another had stored the pair's
|
|
56
|
+
replay also marked that good result errored. The replay now takes a
|
|
57
|
+
claim, kept with the run's fan-out bookkeeping (no migration), before
|
|
58
|
+
the agent runs. A delivery that finds the pair claimed comes back when
|
|
59
|
+
the claim expires (30 minutes) and replays only if nothing was stored,
|
|
60
|
+
so a worker that stopped mid-replay is still covered. A failure is
|
|
61
|
+
stored only over the delivery's own pair. A worker that stops after the
|
|
62
|
+
provider answered and before the result is stored is still replayed
|
|
63
|
+
again: this is one active replay per pair, not exactly once.
|
|
64
|
+
- **A catalog document YAML refuses to load is an invalid document**
|
|
65
|
+
(`activeagent`, `actionagent`). `Catalog.parse` turned only a syntax
|
|
66
|
+
error into `Catalog::InvalidDocument`, while safe loading refuses a tag
|
|
67
|
+
naming a Ruby class (`!ruby/object:...`) or an alias to an undefined
|
|
68
|
+
anchor with other `Psych` errors. Those escaped the dashboard's catalog
|
|
69
|
+
import and preview as a 500, and the `catalogs_import` and
|
|
70
|
+
`catalogs_preview` tools as an internal error. They are now refused like
|
|
71
|
+
any other invalid catalog: 422 with "the catalog is not valid YAML: ...".
|
|
72
|
+
What safe loading permits is unchanged.
|
|
73
|
+
- **A run of only ungated groups no longer reads as "every gate met"**
|
|
74
|
+
(`activeagent`). An ungated group's row is met without being held to a
|
|
75
|
+
bar, so a report whose every result sat in ungated groups said "every
|
|
76
|
+
pass gate met" (`GateCheck`, `assert_meets_pass_gates`, the RSpec
|
|
77
|
+
matcher) and `evals run --wait` printed `Gates: every gate met`, even
|
|
78
|
+
when every answer had failed. The verdict is unchanged, since an ungated
|
|
79
|
+
group never fails the run, but the message now says that every group
|
|
80
|
+
covered is ungated and nothing was held to a bar.
|
|
81
|
+
- **The RubyLLM provider refuses media it would drop** (`activeagent`). It
|
|
82
|
+
sends RubyLLM the text of each message and silently left out every other
|
|
83
|
+
content block, so a message with an image (`image_url`, `image`,
|
|
84
|
+
`input_image`), audio or a file reached the model as its text alone, or
|
|
85
|
+
as an empty message, and the model answered as if it had seen it. Such
|
|
86
|
+
a message now raises `ActiveAgent::Providers::Errors::InvalidRequest`,
|
|
87
|
+
naming the block types, before any provider call. Blocks that are not
|
|
88
|
+
media (an earlier turn's reasoning) are still left out, as before.
|
|
89
|
+
|
|
8
90
|
## [1.9.8] - 2026-11-12
|
|
9
91
|
|
|
10
92
|
The dashboard holds a run to its catalog's pass bars: a product's and a
|
|
@@ -247,7 +247,13 @@ module ActiveAgent
|
|
|
247
247
|
def print_gates(gates, missed)
|
|
248
248
|
unmet = gates.reject { |gate| gate["met"] == true }
|
|
249
249
|
unless missed
|
|
250
|
-
|
|
250
|
+
# Ungated rows are met without being held to a bar, so a run that
|
|
251
|
+
# reached only those says so rather than "every gate met".
|
|
252
|
+
if gates.none? { |gate| gate["gated"] != false }
|
|
253
|
+
@out.puts "Gates: every group reached is ungated (#{gates.size}); nothing was held to a bar."
|
|
254
|
+
else
|
|
255
|
+
@out.puts "Gates: every gate met (#{gates.size})."
|
|
256
|
+
end
|
|
251
257
|
return
|
|
252
258
|
end
|
|
253
259
|
|
|
@@ -81,13 +81,16 @@ module ActiveAgent
|
|
|
81
81
|
end
|
|
82
82
|
|
|
83
83
|
# Parses YAML text. Raises InvalidDocument for YAML that is not a
|
|
84
|
-
# document (a list, a scalar) or that YAML cannot read
|
|
84
|
+
# document (a list, a scalar) or that YAML cannot read: a syntax error,
|
|
85
|
+
# and what safe loading refuses — a tag naming a Ruby class other than
|
|
86
|
+
# Symbol (Psych::DisallowedClass) or an alias to an undefined anchor
|
|
87
|
+
# (Psych::BadAlias), which are not Psych::SyntaxError.
|
|
85
88
|
def self.parse(text, name: nil)
|
|
86
89
|
document = YAML.safe_load(text.to_s, aliases: true, permitted_classes: [ Symbol ])
|
|
87
90
|
raise InvalidDocument, "a scenario catalog is a YAML mapping" unless document.is_a?(Hash)
|
|
88
91
|
|
|
89
92
|
new([ document ], name: name)
|
|
90
|
-
rescue Psych::
|
|
93
|
+
rescue Psych::Exception => e
|
|
91
94
|
raise InvalidDocument, "the catalog is not valid YAML: #{e.message}"
|
|
92
95
|
end
|
|
93
96
|
|
|
@@ -45,7 +45,9 @@ module ActiveAgent
|
|
|
45
45
|
end
|
|
46
46
|
|
|
47
47
|
# The text a failing assertion prints. The first line counts the gates
|
|
48
|
-
# not met, or is the one line "every pass gate met" when none is
|
|
48
|
+
# not met, or is the one line "every pass gate met" when none is (a
|
|
49
|
+
# report whose every row is an ungated group's says that instead: its
|
|
50
|
+
# rows are met because nothing held them to a bar); then
|
|
49
51
|
# each unmet gate as Gate#to_s; then, after a blank line, the failures
|
|
50
52
|
# to read: for each group with an unmet gate, in gates order, "<group>
|
|
51
53
|
# failures:" over Report#failures_markdown for that group. An unmet All
|
|
@@ -59,6 +61,7 @@ module ActiveAgent
|
|
|
59
61
|
return "no result reached a pass gate; the run evaluated nothing" if all.empty?
|
|
60
62
|
|
|
61
63
|
unmet = all.reject(&:met?)
|
|
64
|
+
return "every group the run covered is ungated; no answer was held to a bar" if all.none?(&:gated)
|
|
62
65
|
return "every pass gate met" if unmet.empty?
|
|
63
66
|
|
|
64
67
|
lines = [ "#{unmet.size} of #{all.size} pass gates not met", *unmet.map { |gate| "- #{gate}" } ]
|
|
@@ -381,6 +381,7 @@ module ActiveAgent
|
|
|
381
381
|
when String
|
|
382
382
|
content
|
|
383
383
|
when Array
|
|
384
|
+
refuse_media_blocks!(content)
|
|
384
385
|
content.select { |block| block.is_a?(Hash) && block[:type] == "text" }
|
|
385
386
|
.map { |block| block[:text] }
|
|
386
387
|
.join("\n")
|
|
@@ -389,6 +390,26 @@ module ActiveAgent
|
|
|
389
390
|
end
|
|
390
391
|
end
|
|
391
392
|
|
|
393
|
+
# Content block types that carry media rather than text, in the shapes
|
|
394
|
+
# the OpenAI (Chat and Responses) and Anthropic transforms write.
|
|
395
|
+
MEDIA_BLOCK_TYPES = %w[image image_url input_image audio input_audio file input_file document video].freeze
|
|
396
|
+
|
|
397
|
+
# This provider sends RubyLLM text only, so an image, audio clip or
|
|
398
|
+
# file would be dropped while the model answered as if it had seen
|
|
399
|
+
# it. Such a request is refused before any provider call. Other
|
|
400
|
+
# blocks that are not text (an earlier turn's reasoning) are still
|
|
401
|
+
# left out.
|
|
402
|
+
def refuse_media_blocks!(content)
|
|
403
|
+
types = content.filter_map { |block| block[:type].to_s if block.is_a?(Hash) && MEDIA_BLOCK_TYPES.include?(block[:type].to_s) }.uniq
|
|
404
|
+
return if types.empty?
|
|
405
|
+
|
|
406
|
+
raise Errors::InvalidRequest.new(
|
|
407
|
+
"The RubyLLM provider sends text only; this message has #{types.join(', ')} content, which it would drop. " \
|
|
408
|
+
"Send it through a provider that accepts it, or as text.",
|
|
409
|
+
provider_tag: "RubyLLM"
|
|
410
|
+
)
|
|
411
|
+
end
|
|
412
|
+
|
|
392
413
|
# Converts ActiveAgent tool_calls to RubyLLM's ToolCall format.
|
|
393
414
|
#
|
|
394
415
|
# @param tool_calls [Array<Hash>] ActiveAgent format tool calls
|
|
@@ -83,6 +83,13 @@ module ActiveAgent
|
|
|
83
83
|
"on_or_before" => :lteq, "on_or_after" => :gteq
|
|
84
84
|
}.freeze
|
|
85
85
|
|
|
86
|
+
# Asks whether a column holds a value at all: `{ "present" => false }` is
|
|
87
|
+
# IS NULL, `{ "present" => true }` IS NOT NULL. A nil filter value cannot
|
|
88
|
+
# ask it, because nil means "no filter" (models send null for the
|
|
89
|
+
# optional parameters they leave out), so without this "tickets with no
|
|
90
|
+
# assignee" or "requests not yet approved" has no filter that says so.
|
|
91
|
+
PRESENCE_OPERATOR = "present"
|
|
92
|
+
|
|
86
93
|
# Column names that look like they hold a credential. The schema tools
|
|
87
94
|
# generator never suggests one, and the dashboard never lets one be
|
|
88
95
|
# chosen: reading one back would hand a model a credential, and
|
|
@@ -329,7 +336,8 @@ module ActiveAgent
|
|
|
329
336
|
#
|
|
330
337
|
# A filter value is normally matched for equality. A Hash value instead
|
|
331
338
|
# declares a range — `{ "before" => "2026-01-01" }`, `{ "gte" => 10 }` —
|
|
332
|
-
# and may carry two bounds at once to express a window
|
|
339
|
+
# and may carry two bounds at once to express a window, or asks whether
|
|
340
|
+
# the column is set at all (`{ "present" => false }`).
|
|
333
341
|
#
|
|
334
342
|
# @api private
|
|
335
343
|
# @raise [UnpermittedAttribute] if any key is not declared filterable,
|
|
@@ -389,17 +397,37 @@ module ActiveAgent
|
|
|
389
397
|
type = @model.type_for_attribute(column)
|
|
390
398
|
|
|
391
399
|
value.map do |operator, operand|
|
|
400
|
+
next presence_predicate!(column, arel, operand) if operator.to_s == PRESENCE_OPERATOR
|
|
401
|
+
|
|
392
402
|
predicate = RANGE_OPERATORS[operator.to_s]
|
|
393
403
|
unless predicate
|
|
394
404
|
raise UnpermittedAttribute,
|
|
395
405
|
"`#{operator}` is not a valid comparison for `#{column}`. " \
|
|
396
|
-
"Allowed comparisons: #{RANGE_OPERATORS.keys.join(", ")}"
|
|
406
|
+
"Allowed comparisons: #{[ *RANGE_OPERATORS.keys, PRESENCE_OPERATOR ].join(", ")}"
|
|
397
407
|
end
|
|
398
408
|
|
|
399
409
|
arel.public_send(predicate, type.cast(operand))
|
|
400
410
|
end
|
|
401
411
|
end
|
|
402
412
|
|
|
413
|
+
# Builds IS NULL / IS NOT NULL for `{ "present" => ... }`.
|
|
414
|
+
#
|
|
415
|
+
# Only true and false (or their strings) are accepted: anything else
|
|
416
|
+
# would have to be guessed into one of the two, and a wrong guess
|
|
417
|
+
# answers the opposite question.
|
|
418
|
+
#
|
|
419
|
+
# @api private
|
|
420
|
+
# @raise [UnpermittedAttribute] on any other operand
|
|
421
|
+
def presence_predicate!(column, arel, operand)
|
|
422
|
+
case operand.to_s
|
|
423
|
+
when "true" then arel.not_eq(nil)
|
|
424
|
+
when "false" then arel.eq(nil)
|
|
425
|
+
else
|
|
426
|
+
raise UnpermittedAttribute,
|
|
427
|
+
"`#{PRESENCE_OPERATOR}` for `#{column}` takes true or false, not #{operand.inspect}"
|
|
428
|
+
end
|
|
429
|
+
end
|
|
430
|
+
|
|
403
431
|
# A value outside a Rails enum reaches `where` as an unmatched name and
|
|
404
432
|
# returns zero rows — the same silent-nothing that {.range_predicates!}
|
|
405
433
|
# rejects for an unknown operator, and just as bad here: an agent reads
|
|
@@ -415,9 +443,12 @@ module ActiveAgent
|
|
|
415
443
|
return if values.blank?
|
|
416
444
|
|
|
417
445
|
if value.is_a?(Hash)
|
|
446
|
+
# Whether it is set at all is a question an enum can answer.
|
|
447
|
+
return if value.keys.map(&:to_s) == [ PRESENCE_OPERATOR ]
|
|
448
|
+
|
|
418
449
|
raise UnpermittedAttribute,
|
|
419
450
|
"`#{column}` is an enum and cannot be compared as a range. " \
|
|
420
|
-
"Allowed values: #{values.keys.join(", ")}"
|
|
451
|
+
"Allowed values: #{values.keys.join(", ")}, or {\"#{PRESENCE_OPERATOR}\": true|false}"
|
|
421
452
|
end
|
|
422
453
|
|
|
423
454
|
# An Array is an IN filter, which `where` silently narrows to its
|
|
@@ -550,20 +581,28 @@ module ActiveAgent
|
|
|
550
581
|
# only `{type: "string", format: "date"}` has no way to know it may ask
|
|
551
582
|
# for `before`, and answers date questions with an equality match or no
|
|
552
583
|
# filter at all.
|
|
584
|
+
#
|
|
585
|
+
# A nullable column's range object also takes `present`, so "has no
|
|
586
|
+
# due date" is asked in the same shape as "due before Friday". It is
|
|
587
|
+
# offered only here, on the columns that already carry an object form,
|
|
588
|
+
# rather than wrapping every nullable string and enum in one; a call
|
|
589
|
+
# may still send it for any filterable column.
|
|
553
590
|
def with_range_form(column, scalar)
|
|
554
591
|
operand = scalar.slice(:type, :format)
|
|
555
|
-
|
|
592
|
+
nullable = @model.columns_hash[column.to_s]&.null
|
|
593
|
+
properties = RANGE_OPERATORS.keys.index_with { operand.dup }
|
|
594
|
+
properties[PRESENCE_OPERATOR] = { type: "boolean" } if nullable
|
|
595
|
+
|
|
596
|
+
description = [ scalar[:description], "Accepts an exact value, or a range object such as " \
|
|
597
|
+
"{\"before\": ...} / {\"gte\": ...} (#{RANGE_OPERATORS.keys.join(", ")})." ]
|
|
598
|
+
description << "{\"#{PRESENCE_OPERATOR}\": false} matches records where it is empty (null), " \
|
|
599
|
+
"{\"#{PRESENCE_OPERATOR}\": true} where it is set." if nullable
|
|
556
600
|
|
|
557
601
|
{
|
|
558
|
-
description:
|
|
559
|
-
"{\"before\": ...} / {\"gte\": ...} (#{RANGE_OPERATORS.keys.join(", ")})." ].compact.join(" "),
|
|
602
|
+
description: description.compact.join(" "),
|
|
560
603
|
anyOf: [
|
|
561
604
|
scalar.except(:description),
|
|
562
|
-
{
|
|
563
|
-
type: "object",
|
|
564
|
-
properties: RANGE_OPERATORS.keys.index_with { operand.dup },
|
|
565
|
-
additionalProperties: false
|
|
566
|
-
}
|
|
605
|
+
{ type: "object", properties: properties, additionalProperties: false }
|
|
567
606
|
]
|
|
568
607
|
}
|
|
569
608
|
end
|
data/lib/active_agent/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: activeagent
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.9.
|
|
4
|
+
version: 1.9.9
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Justin Bowen
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-11-
|
|
11
|
+
date: 2026-11-19 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: actionpack
|