activeagent 1.9.8 → 1.9.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 560139e6515a8b3568aabc383ec0ae4219764a2d8ca91b14b4defc4fdec3439d
4
- data.tar.gz: b1d8651cb0a058b8922e579374720efd42c3babcadfddc3c123c5b7c6c2afd8b
3
+ metadata.gz: e5cf4f1304cb0d939f96e270b950dbe8bc3505e6e4d8dacba9f0f16083c1b485
4
+ data.tar.gz: 7716fbc50b2e06ac508cf88a63b8515bbec0f17d6ab632b57a05706f0fe1187b
5
5
  SHA512:
6
- metadata.gz: 722dcbfe0e8bc2254871264f42ddf5394eb4f26acfc47f53ce17f2c9b382db627248d90ded1c9164a30bd176f908ee4510db11332e26bd6e25c91d485b251fde
7
- data.tar.gz: e97465f43ad9dfee0eecfa0e3b3207b7c3a67dc620a1fa656c451800b9e49ca89259a75dd7ce7ad7066f05bfeece5d719fdf869e30a3cf89ae11f6e6f9de42dd
6
+ metadata.gz: f237aff2404b739fa66c77e0d5a6e2925be257423fc2072ae7b1387c00a144cf0fdbb95466237981170bcf1f2f9b468c0f592bdc5d3442522300fc0dce6b581a
7
+ data.tar.gz: ef1161bd9b4d39338594946354ce0f455f212f9508ff0a8f0119fd2762fb965a7b32798a3913a2473a27424c6c918e7102d3a14e78233d4b67a96f1771aac108
data/CHANGELOG.md CHANGED
@@ -5,6 +5,88 @@ All notable changes to this project will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [1.9.9] - 2026-11-18
9
+
10
+ Schema tools can ask whether a column is set: a filter takes
11
+ `{"present": false}` (IS NULL) or `{"present": true}` (IS NOT NULL), so a
12
+ model can find the tickets with no assignee without a sentinel range in
13
+ the instructions. Around that, fixes to evaluations and providers: an
14
+ evaluation run and a fanned-out replay are each claimed before they
15
+ execute, so a job delivered twice no longer replays the suite or calls
16
+ the provider twice; a catalog document safe loading refuses is a 422
17
+ instead of a 500; a run of only ungated groups no longer reads as "every
18
+ gate met"; and the RubyLLM provider refuses media it would drop.
19
+
20
+ Upgrading: no migrations. The RubyLLM provider now raises
21
+ `ActiveAgent::Providers::Errors::InvalidRequest` for a message carrying an
22
+ image, audio or a file, where it used to send the text alone; route such
23
+ messages to a provider that takes media.
24
+
25
+ ### Added
26
+
27
+ - **Schema tools can ask whether a column is set** (`activeagent`). A
28
+ filter value of `null` means "no filter", so a model had no way to ask
29
+ for the tickets with no assignee or the requests not yet approved, and
30
+ hosts wrote sentinel ranges such as `{"after": "2000-01-01"}` into their
31
+ instructions instead. A filter now takes `{"present": false}` (IS NULL)
32
+ or `{"present": true}` (IS NOT NULL) on any filterable column, alone or
33
+ beside a range bound. The tool schema offers it in the range object of
34
+ nullable date, time and number columns, so foreign keys get it too, and
35
+ an enum accepts it though it takes no comparison. Anything but true or
36
+ false is rejected rather than guessed.
37
+
38
+ ### Fixed
39
+
40
+ - **An evaluation run is executed by one job** (`actionagent`). The run job
41
+ read a run's status, then the scenario runner wrote `running`, as two
42
+ steps, so a job delivered twice, or two workers that both loaded the run
43
+ as pending, each replayed the whole suite, or with
44
+ `scenario_evaluation_fan_out` on each enqueued a full set of replay jobs.
45
+ The runner now moves the run from pending to running in one conditional
46
+ update, as the execution job claims an agent run, and replays nothing
47
+ unless it was the one to do so; this covers `ProjectEvaluationJob` too.
48
+ The claim comes before the selection is read, so a selection that cannot
49
+ be resolved (an unsupported provider in a re-run's models) now fails the
50
+ run instead of leaving it pending forever.
51
+ - **A fanned-out replay is claimed before the agent runs** (`actionagent`).
52
+ `EvaluationReplayJob` checked for a stored result, replayed, and only
53
+ then took the run's lock to store, so two deliveries of one replay job
54
+ both called the provider (and the agent's tools) and the second answer
55
+ was dropped. A delivery that failed after another had stored the pair's
56
+ replay also marked that good result errored. The replay now takes a
57
+ claim, kept with the run's fan-out bookkeeping (no migration), before
58
+ the agent runs. A delivery that finds the pair claimed comes back when
59
+ the claim expires (30 minutes) and replays only if nothing was stored,
60
+ so a worker that stopped mid-replay is still covered. A failure is
61
+ stored only over the delivery's own pair. A worker that stops after the
62
+ provider answered and before the result is stored is still replayed
63
+ again: this is one active replay per pair, not exactly once.
64
+ - **A catalog document YAML refuses to load is an invalid document**
65
+ (`activeagent`, `actionagent`). `Catalog.parse` turned only a syntax
66
+ error into `Catalog::InvalidDocument`, while safe loading refuses a tag
67
+ naming a Ruby class (`!ruby/object:...`) or an alias to an undefined
68
+ anchor with other `Psych` errors. Those escaped the dashboard's catalog
69
+ import and preview as a 500, and the `catalogs_import` and
70
+ `catalogs_preview` tools as an internal error. They are now refused like
71
+ any other invalid catalog: 422 with "the catalog is not valid YAML: ...".
72
+ What safe loading permits is unchanged.
73
+ - **A run of only ungated groups no longer reads as "every gate met"**
74
+ (`activeagent`). An ungated group's row is met without being held to a
75
+ bar, so a report whose every result sat in ungated groups said "every
76
+ pass gate met" (`GateCheck`, `assert_meets_pass_gates`, the RSpec
77
+ matcher) and `evals run --wait` printed `Gates: every gate met`, even
78
+ when every answer had failed. The verdict is unchanged, since an ungated
79
+ group never fails the run, but the message now says that every group
80
+ covered is ungated and nothing was held to a bar.
81
+ - **The RubyLLM provider refuses media it would drop** (`activeagent`). It
82
+ sends RubyLLM the text of each message and silently left out every other
83
+ content block, so a message with an image (`image_url`, `image`,
84
+ `input_image`), audio or a file reached the model as its text alone, or
85
+ as an empty message, and the model answered as if it had seen it. Such
86
+ a message now raises `ActiveAgent::Providers::Errors::InvalidRequest`,
87
+ naming the block types, before any provider call. Blocks that are not
88
+ media (an earlier turn's reasoning) are still left out, as before.
89
+
8
90
  ## [1.9.8] - 2026-11-12
9
91
 
10
92
  The dashboard holds a run to its catalog's pass bars: a product's and a
@@ -247,7 +247,13 @@ module ActiveAgent
247
247
  def print_gates(gates, missed)
248
248
  unmet = gates.reject { |gate| gate["met"] == true }
249
249
  unless missed
250
- @out.puts "Gates: every gate met (#{gates.size})."
250
+ # Ungated rows are met without being held to a bar, so a run that
251
+ # reached only those says so rather than "every gate met".
252
+ if gates.none? { |gate| gate["gated"] != false }
253
+ @out.puts "Gates: every group reached is ungated (#{gates.size}); nothing was held to a bar."
254
+ else
255
+ @out.puts "Gates: every gate met (#{gates.size})."
256
+ end
251
257
  return
252
258
  end
253
259
 
@@ -81,13 +81,16 @@ module ActiveAgent
81
81
  end
82
82
 
83
83
  # Parses YAML text. Raises InvalidDocument for YAML that is not a
84
- # document (a list, a scalar) or that YAML cannot read.
84
+ # document (a list, a scalar) or that YAML cannot read: a syntax error,
85
+ # and what safe loading refuses — a tag naming a Ruby class other than
86
+ # Symbol (Psych::DisallowedClass) or an alias to an undefined anchor
87
+ # (Psych::BadAlias), which are not Psych::SyntaxError.
85
88
  def self.parse(text, name: nil)
86
89
  document = YAML.safe_load(text.to_s, aliases: true, permitted_classes: [ Symbol ])
87
90
  raise InvalidDocument, "a scenario catalog is a YAML mapping" unless document.is_a?(Hash)
88
91
 
89
92
  new([ document ], name: name)
90
- rescue Psych::SyntaxError => e
93
+ rescue Psych::Exception => e
91
94
  raise InvalidDocument, "the catalog is not valid YAML: #{e.message}"
92
95
  end
93
96
 
@@ -45,7 +45,9 @@ module ActiveAgent
45
45
  end
46
46
 
47
47
  # The text a failing assertion prints. The first line counts the gates
48
- # not met, or is the one line "every pass gate met" when none is; then
48
+ # not met, or is the one line "every pass gate met" when none is (a
49
+ # report whose every row is an ungated group's says that instead: its
50
+ # rows are met because nothing held them to a bar); then
49
51
  # each unmet gate as Gate#to_s; then, after a blank line, the failures
50
52
  # to read: for each group with an unmet gate, in gates order, "<group>
51
53
  # failures:" over Report#failures_markdown for that group. An unmet All
@@ -59,6 +61,7 @@ module ActiveAgent
59
61
  return "no result reached a pass gate; the run evaluated nothing" if all.empty?
60
62
 
61
63
  unmet = all.reject(&:met?)
64
+ return "every group the run covered is ungated; no answer was held to a bar" if all.none?(&:gated)
62
65
  return "every pass gate met" if unmet.empty?
63
66
 
64
67
  lines = [ "#{unmet.size} of #{all.size} pass gates not met", *unmet.map { |gate| "- #{gate}" } ]
@@ -381,6 +381,7 @@ module ActiveAgent
381
381
  when String
382
382
  content
383
383
  when Array
384
+ refuse_media_blocks!(content)
384
385
  content.select { |block| block.is_a?(Hash) && block[:type] == "text" }
385
386
  .map { |block| block[:text] }
386
387
  .join("\n")
@@ -389,6 +390,26 @@ module ActiveAgent
389
390
  end
390
391
  end
391
392
 
393
+ # Content block types that carry media rather than text, in the shapes
394
+ # the OpenAI (Chat and Responses) and Anthropic transforms write.
395
+ MEDIA_BLOCK_TYPES = %w[image image_url input_image audio input_audio file input_file document video].freeze
396
+
397
+ # This provider sends RubyLLM text only, so an image, audio clip or
398
+ # file would be dropped while the model answered as if it had seen
399
+ # it. Such a request is refused before any provider call. Other
400
+ # blocks that are not text (an earlier turn's reasoning) are still
401
+ # left out.
402
+ def refuse_media_blocks!(content)
403
+ types = content.filter_map { |block| block[:type].to_s if block.is_a?(Hash) && MEDIA_BLOCK_TYPES.include?(block[:type].to_s) }.uniq
404
+ return if types.empty?
405
+
406
+ raise Errors::InvalidRequest.new(
407
+ "The RubyLLM provider sends text only; this message has #{types.join(', ')} content, which it would drop. " \
408
+ "Send it through a provider that accepts it, or as text.",
409
+ provider_tag: "RubyLLM"
410
+ )
411
+ end
412
+
392
413
  # Converts ActiveAgent tool_calls to RubyLLM's ToolCall format.
393
414
  #
394
415
  # @param tool_calls [Array<Hash>] ActiveAgent format tool calls
@@ -83,6 +83,13 @@ module ActiveAgent
83
83
  "on_or_before" => :lteq, "on_or_after" => :gteq
84
84
  }.freeze
85
85
 
86
+ # Asks whether a column holds a value at all: `{ "present" => false }` is
87
+ # IS NULL, `{ "present" => true }` IS NOT NULL. A nil filter value cannot
88
+ # ask it, because nil means "no filter" (models send null for the
89
+ # optional parameters they leave out), so without this "tickets with no
90
+ # assignee" or "requests not yet approved" has no filter that says so.
91
+ PRESENCE_OPERATOR = "present"
92
+
86
93
  # Column names that look like they hold a credential. The schema tools
87
94
  # generator never suggests one, and the dashboard never lets one be
88
95
  # chosen: reading one back would hand a model a credential, and
@@ -329,7 +336,8 @@ module ActiveAgent
329
336
  #
330
337
  # A filter value is normally matched for equality. A Hash value instead
331
338
  # declares a range — `{ "before" => "2026-01-01" }`, `{ "gte" => 10 }` —
332
- # and may carry two bounds at once to express a window.
339
+ # and may carry two bounds at once to express a window, or asks whether
340
+ # the column is set at all (`{ "present" => false }`).
333
341
  #
334
342
  # @api private
335
343
  # @raise [UnpermittedAttribute] if any key is not declared filterable,
@@ -389,17 +397,37 @@ module ActiveAgent
389
397
  type = @model.type_for_attribute(column)
390
398
 
391
399
  value.map do |operator, operand|
400
+ next presence_predicate!(column, arel, operand) if operator.to_s == PRESENCE_OPERATOR
401
+
392
402
  predicate = RANGE_OPERATORS[operator.to_s]
393
403
  unless predicate
394
404
  raise UnpermittedAttribute,
395
405
  "`#{operator}` is not a valid comparison for `#{column}`. " \
396
- "Allowed comparisons: #{RANGE_OPERATORS.keys.join(", ")}"
406
+ "Allowed comparisons: #{[ *RANGE_OPERATORS.keys, PRESENCE_OPERATOR ].join(", ")}"
397
407
  end
398
408
 
399
409
  arel.public_send(predicate, type.cast(operand))
400
410
  end
401
411
  end
402
412
 
413
+ # Builds IS NULL / IS NOT NULL for `{ "present" => ... }`.
414
+ #
415
+ # Only true and false (or their strings) are accepted: anything else
416
+ # would have to be guessed into one of the two, and a wrong guess
417
+ # answers the opposite question.
418
+ #
419
+ # @api private
420
+ # @raise [UnpermittedAttribute] on any other operand
421
+ def presence_predicate!(column, arel, operand)
422
+ case operand.to_s
423
+ when "true" then arel.not_eq(nil)
424
+ when "false" then arel.eq(nil)
425
+ else
426
+ raise UnpermittedAttribute,
427
+ "`#{PRESENCE_OPERATOR}` for `#{column}` takes true or false, not #{operand.inspect}"
428
+ end
429
+ end
430
+
403
431
  # A value outside a Rails enum reaches `where` as an unmatched name and
404
432
  # returns zero rows — the same silent-nothing that {.range_predicates!}
405
433
  # rejects for an unknown operator, and just as bad here: an agent reads
@@ -415,9 +443,12 @@ module ActiveAgent
415
443
  return if values.blank?
416
444
 
417
445
  if value.is_a?(Hash)
446
+ # Whether it is set at all is a question an enum can answer.
447
+ return if value.keys.map(&:to_s) == [ PRESENCE_OPERATOR ]
448
+
418
449
  raise UnpermittedAttribute,
419
450
  "`#{column}` is an enum and cannot be compared as a range. " \
420
- "Allowed values: #{values.keys.join(", ")}"
451
+ "Allowed values: #{values.keys.join(", ")}, or {\"#{PRESENCE_OPERATOR}\": true|false}"
421
452
  end
422
453
 
423
454
  # An Array is an IN filter, which `where` silently narrows to its
@@ -550,20 +581,28 @@ module ActiveAgent
550
581
  # only `{type: "string", format: "date"}` has no way to know it may ask
551
582
  # for `before`, and answers date questions with an equality match or no
552
583
  # filter at all.
584
+ #
585
+ # A nullable column's range object also takes `present`, so "has no
586
+ # due date" is asked in the same shape as "due before Friday". It is
587
+ # offered only here, on the columns that already carry an object form,
588
+ # rather than wrapping every nullable string and enum in one; a call
589
+ # may still send it for any filterable column.
553
590
  def with_range_form(column, scalar)
554
591
  operand = scalar.slice(:type, :format)
555
- description = scalar[:description]
592
+ nullable = @model.columns_hash[column.to_s]&.null
593
+ properties = RANGE_OPERATORS.keys.index_with { operand.dup }
594
+ properties[PRESENCE_OPERATOR] = { type: "boolean" } if nullable
595
+
596
+ description = [ scalar[:description], "Accepts an exact value, or a range object such as " \
597
+ "{\"before\": ...} / {\"gte\": ...} (#{RANGE_OPERATORS.keys.join(", ")})." ]
598
+ description << "{\"#{PRESENCE_OPERATOR}\": false} matches records where it is empty (null), " \
599
+ "{\"#{PRESENCE_OPERATOR}\": true} where it is set." if nullable
556
600
 
557
601
  {
558
- description: [ description, "Accepts an exact value, or a range object such as " \
559
- "{\"before\": ...} / {\"gte\": ...} (#{RANGE_OPERATORS.keys.join(", ")})." ].compact.join(" "),
602
+ description: description.compact.join(" "),
560
603
  anyOf: [
561
604
  scalar.except(:description),
562
- {
563
- type: "object",
564
- properties: RANGE_OPERATORS.keys.index_with { operand.dup },
565
- additionalProperties: false
566
- }
605
+ { type: "object", properties: properties, additionalProperties: false }
567
606
  ]
568
607
  }
569
608
  end
@@ -1,3 +1,3 @@
1
1
  module ActiveAgent
2
- VERSION = "1.9.8"
2
+ VERSION = "1.9.9"
3
3
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: activeagent
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.9.8
4
+ version: 1.9.9
5
5
  platform: ruby
6
6
  authors:
7
7
  - Justin Bowen
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2026-11-12 00:00:00.000000000 Z
11
+ date: 2026-11-19 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: actionpack