agent-harness 0.42.0 → 0.43.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/CHANGELOG.md +8 -0
- data/docs/provider-neutral-api-execution-contract.md +70 -16
- data/lib/agent_harness/api/chat_transport.rb +54 -6
- data/lib/agent_harness/api/ruby_llm_chat_adapter.rb +66 -11
- data/lib/agent_harness/api/schema_response.rb +43 -0
- data/lib/agent_harness/version.rb +1 -1
- data/lib/agent_harness.rb +1 -0
- metadata +20 -11
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 40d84099230c75237b23dca08330a46b739029768ce345b57cb232ba0e66a935
|
|
4
|
+
data.tar.gz: 47d36a8e160bcd29f9d50f6eabc1d0ded6795d6e05f2e45e93038d1987cd352c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f059d918fed14691fe444dba2e5e93aaeb2cdb185785a580b8a7341db8198616a0a830fd43461b4dfceffc9258f71ce25a8cbb69351747af46980ba0bd783757
|
|
7
|
+
data.tar.gz: 2f09db993da48a0ca0ed852eee9653898f474e7c61a6e0603dde3588457d226c8f51de71a8b322e7eba46b7e64203cc61a7607aa31764ed63188ef8b38f7c1e5
|
data/CHANGELOG.md
CHANGED
|
@@ -2,9 +2,17 @@
|
|
|
2
2
|
|
|
3
3
|
### Features
|
|
4
4
|
|
|
5
|
+
* expose normalized API chat transport and schema-constrained responses with preserved JSON text, locally validated parsed values, and explicit refusal, truncation, invalid JSON, schema mismatch, and unsupported-mode outcomes; this combined delivery supersedes the separate chat-transport work item ([#433](https://github.com/viamin/agent-harness/issues/433), [#434](https://github.com/viamin/agent-harness/issues/434)).
|
|
5
6
|
* add runner model compatibility contract (`AgentHarness.model_compatibility`) with structured `ModelCompatibility::Result` outcomes. Codex exposes static facts for CLI-gated models (e.g. `gpt-5.5` requires Codex CLI `>= 0.116.0`), a baseline supported-model list, supported auth modes, and a `DEFAULT_COMPATIBLE_MODEL_ID` fallback so downstream orchestrators can validate tier/model assignments before scheduling agent runs ([#259](https://github.com/viamin/agent-harness/issues/259)).
|
|
6
7
|
* **auth:** add provider-owned PKCE code-exchange API for Claude OAuth (`AgentHarness::Authentication.exchange_code`). Takes an authorization code plus PKCE verifier (and `redirect_uri`/`client_id`), posts an `authorization_code` grant to the Claude token endpoint, and persists the resulting access/refresh tokens in the native `claudeAiOauth` shape. Adds `exchange_code_supported?` and a `code_exchange` key to `auth_capabilities` ([#266](https://github.com/viamin/agent-harness/issues/266)).
|
|
7
8
|
|
|
9
|
+
## [0.43.0](https://github.com/viamin/agent-harness/compare/agent-harness/v0.42.0...agent-harness/v0.43.0) (2026-09-25)
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
### Features
|
|
13
|
+
|
|
14
|
+
* Expose Schema-Constrained Parsed Responses (RDR-072) ([#442](https://github.com/viamin/agent-harness/issues/442)) ([3d05cfa](https://github.com/viamin/agent-harness/commit/3d05cfac68624c218eda5f6f9871a3a4d1044ec0))
|
|
15
|
+
|
|
8
16
|
## [0.42.0](https://github.com/viamin/agent-harness/compare/agent-harness/v0.41.0...agent-harness/v0.42.0) (2026-09-25)
|
|
9
17
|
|
|
10
18
|
|
|
@@ -8,8 +8,8 @@ usage, and optional conversation-persistence work.
|
|
|
8
8
|
## Status and rollout boundary
|
|
9
9
|
|
|
10
10
|
RDR-072's rollout guard was **docs-only** for the design phase. The normalized
|
|
11
|
-
chat
|
|
12
|
-
|
|
11
|
+
chat, attempt-accounting, schema, and embedding capabilities described below
|
|
12
|
+
are now implemented. Other capabilities remain design contracts and each still needs
|
|
13
13
|
its own failing-first contract tests, implementation, release evidence, and
|
|
14
14
|
downstream adoption evidence before a caller enables it.
|
|
15
15
|
|
|
@@ -70,11 +70,13 @@ Responses API must select `:chat_completions`; the transport never probes and
|
|
|
70
70
|
silently switches protocols. Only `authentication_mode: :api_key` is currently
|
|
71
71
|
supported.
|
|
72
72
|
|
|
73
|
-
Credentials, endpoint, custom headers, timeout, and RubyLLM configuration
|
|
74
|
-
isolated with a request-local `RubyLLM::Context`.
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
73
|
+
Credentials, endpoint, custom headers, read timeout, and RubyLLM configuration
|
|
74
|
+
are isolated with a request-local `RubyLLM::Context`. Only
|
|
75
|
+
`timeout.read_seconds` is supported; request-local connection timeouts are not.
|
|
76
|
+
RubyLLM middleware retries are disabled; `retry.max_attempts` is the total
|
|
77
|
+
physical-attempt limit owned by the harness. Authentication headers cannot be
|
|
78
|
+
overridden by custom headers. `max_output_tokens` is forwarded without changing
|
|
79
|
+
it.
|
|
78
80
|
|
|
79
81
|
Unknown model IDs are allowed only because a complete provider and protocol
|
|
80
82
|
are explicit in every candidate (`assume_model_exists: true` in the RubyLLM
|
|
@@ -84,6 +86,41 @@ endpoints retain the selected provider's wire protocol and authentication
|
|
|
84
86
|
shape. Custom provider types, authentication modes, media content, and
|
|
85
87
|
automatic protocol discovery are not supported by this capability.
|
|
86
88
|
|
|
89
|
+
## Shipped schema-constrained response surface
|
|
90
|
+
|
|
91
|
+
The same transport accepts `operation: :schema` with a JSON Schema and an
|
|
92
|
+
optional name:
|
|
93
|
+
|
|
94
|
+
```ruby
|
|
95
|
+
result = transport.call(request.merge(
|
|
96
|
+
operation: :schema,
|
|
97
|
+
schema_name: "person",
|
|
98
|
+
schema: {
|
|
99
|
+
type: "object",
|
|
100
|
+
properties: {name: {type: "string"}, age: {type: "integer"}},
|
|
101
|
+
required: %w[name age],
|
|
102
|
+
additionalProperties: false
|
|
103
|
+
}
|
|
104
|
+
))
|
|
105
|
+
|
|
106
|
+
result[:content] # => '{"name":"Ada","age":37}'
|
|
107
|
+
result[:parsed] # => {"name" => "Ada", "age" => 37}
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Schema operations use provider-native JSON Schema output and the same verified
|
|
111
|
+
Anthropic Messages, OpenAI Responses, and OpenAI Chat Completions scopes as
|
|
112
|
+
normalized chat. Provider adapters infer strictness from the schema; callers
|
|
113
|
+
using a `{schema:, strict:}` envelope can explicitly select strictness.
|
|
114
|
+
`schema_mode: :json_schema` is the only supported mode. JSON-only mode returns
|
|
115
|
+
`unsupported/structured_output_not_supported`.
|
|
116
|
+
|
|
117
|
+
The harness parses the exact provider text and validates it locally. It does
|
|
118
|
+
not remove Markdown fences or repair malformed JSON. Invalid JSON, schema
|
|
119
|
+
mismatch, refusal, and output-limit truncation return non-retryable
|
|
120
|
+
`invalid_response` outcomes with codes `invalid_json`, `invalid_schema`,
|
|
121
|
+
`refusal`, and `truncated_output`. They retain the original `content` and leave
|
|
122
|
+
`parsed` as `nil`.
|
|
123
|
+
|
|
87
124
|
## Ownership boundary
|
|
88
125
|
|
|
89
126
|
AgentHarness owns protocol translation and one bounded provider request:
|
|
@@ -139,7 +176,7 @@ request = {
|
|
|
139
176
|
}
|
|
140
177
|
],
|
|
141
178
|
fallback: {on_error_categories: [:transient]},
|
|
142
|
-
timeout: {
|
|
179
|
+
timeout: {read_seconds: 60},
|
|
143
180
|
retry: {max_attempts: 3, base_delay_seconds: 0.25, max_delay_seconds: 2},
|
|
144
181
|
cancellation: cancellation_token,
|
|
145
182
|
metadata: {tenant_id: "tenant-123", workflow_id: "workflow-456"}
|
|
@@ -160,9 +197,11 @@ part of a serializable request document.
|
|
|
160
197
|
gets a distinct `attempt_id`. Redelivering an already reported attempt retains
|
|
161
198
|
its `attempt_id`; initiating another outbound request does not.
|
|
162
199
|
|
|
163
|
-
Credentials, endpoint, headers,
|
|
164
|
-
request-local.
|
|
165
|
-
|
|
200
|
+
Credentials, endpoint, headers, the read timeout, retry limits, and cancellation
|
|
201
|
+
are request-local. Only `timeout.read_seconds` is supported; supplying a
|
|
202
|
+
connection timeout returns `unsupported/unsupported_capability` before any
|
|
203
|
+
provider request. Implementations MUST prevent concurrent requests from
|
|
204
|
+
observing one another's credentials or headers. They MUST reject reserved header
|
|
166
205
|
overrides that would conflict with the selected protocol's authentication.
|
|
167
206
|
Logs and errors MUST NOT contain credentials, authorization headers, message
|
|
168
207
|
bodies, tool arguments, or full provider responses.
|
|
@@ -504,7 +543,7 @@ records.
|
|
|
504
543
|
| Contract area | RubyLLM 2.0 mapping | Decision or gap |
|
|
505
544
|
| --- | --- | --- |
|
|
506
545
|
| Chat/protocols | `RubyLLM.chat`, messages, tools, stream callbacks | Candidate adapter; normalize all values and errors |
|
|
507
|
-
| Schema | `with_schema
|
|
546
|
+
| Schema | `with_schema`; harness JSON parsing and validation | Implemented for the verified chat scopes; JSON-only mode stays distinct |
|
|
508
547
|
| Embeddings | `RubyLLM.embed` and normalized vectors/usage | Candidate first capability; persistence remains in Paid |
|
|
509
548
|
| Custom headers | `with_headers` | Candidate; contract-test merging and secret redaction |
|
|
510
549
|
| Endpoint/credentials | provider configuration | Global mutable configuration is unsuitable; require request-local isolation or an upstream-supported client boundary |
|
|
@@ -607,11 +646,26 @@ types stay behind the harness boundary.
|
|
|
607
646
|
embeddings, and token trackers remain outside this ledger and require
|
|
608
647
|
separate migration issues.
|
|
609
648
|
|
|
649
|
+
### Schema capability release evidence
|
|
650
|
+
|
|
651
|
+
- Publication: unreleased; record the first installable version before
|
|
652
|
+
downstream adoption.
|
|
653
|
+
- Verified scopes: `:schema` with Anthropic Messages, OpenAI Responses, and
|
|
654
|
+
OpenAI Chat Completions using API-key authentication, including compatible
|
|
655
|
+
endpoints that explicitly select Chat Completions.
|
|
656
|
+
- Contract coverage: valid and required-field schemas, classified failures,
|
|
657
|
+
bounded retries, cancellation, refusal, truncation, malformed JSON, and
|
|
658
|
+
schema mismatch.
|
|
659
|
+
- Retained paths: CLI and subscription execution remain on existing provider
|
|
660
|
+
interfaces. JSON-only mode and model-specific capability discovery are not
|
|
661
|
+
migrated.
|
|
662
|
+
|
|
610
663
|
AgentHarness currently supports Ruby 3.2 and later and must remain usable as a
|
|
611
|
-
plain Ruby gem. RubyLLM 2.0.0 itself supports Ruby 3.1.3 and later
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
dependency set
|
|
664
|
+
plain Ruby gem. RubyLLM 2.0.0 itself supports Ruby 3.1.3 and later and adds
|
|
665
|
+
Faraday, event-stream parsing, Schematist, Marcel, and Zeitwerk runtime
|
|
666
|
+
dependencies. Local response validation adds `json_schemer` and its bounded
|
|
667
|
+
dependency set. A capability issue must test the harness minimum Ruby version
|
|
668
|
+
before adoption.
|
|
615
669
|
|
|
616
670
|
Rails and Active Record remain optional. Requiring `agent_harness` in a process
|
|
617
671
|
without Rails MUST NOT load Active Record, connect to a database, or require
|
|
@@ -4,6 +4,7 @@ require "securerandom"
|
|
|
4
4
|
require "ruby_llm"
|
|
5
5
|
require_relative "attempt_report"
|
|
6
6
|
require_relative "ruby_llm_chat_adapter"
|
|
7
|
+
require_relative "schema_response"
|
|
7
8
|
|
|
8
9
|
module AgentHarness
|
|
9
10
|
module Api
|
|
@@ -54,6 +55,7 @@ module AgentHarness
|
|
|
54
55
|
end
|
|
55
56
|
|
|
56
57
|
def call
|
|
58
|
+
return unsupported_schema_result unless schema_mode_supported?
|
|
57
59
|
return cancelled_result unless active?
|
|
58
60
|
|
|
59
61
|
candidates.each_with_index do |candidate, candidate_index|
|
|
@@ -95,6 +97,7 @@ module AgentHarness
|
|
|
95
97
|
max_output_tokens: request[:max_output_tokens],
|
|
96
98
|
temperature: request[:temperature],
|
|
97
99
|
timeout: request[:timeout],
|
|
100
|
+
schema: schema_payload,
|
|
98
101
|
on_accounting: ->(facts) { accounting = facts },
|
|
99
102
|
provider_usage: provider_usage
|
|
100
103
|
)
|
|
@@ -113,6 +116,7 @@ module AgentHarness
|
|
|
113
116
|
stream: request[:stream] == true,
|
|
114
117
|
timeout: request[:timeout],
|
|
115
118
|
cancellation: request[:cancellation],
|
|
119
|
+
schema: schema_payload,
|
|
116
120
|
on_accounting: ->(facts) { accounting = facts },
|
|
117
121
|
prepared_chat: prepared_chat
|
|
118
122
|
) do |event|
|
|
@@ -138,19 +142,29 @@ module AgentHarness
|
|
|
138
142
|
|
|
139
143
|
def success(candidate, attempt_id, started_at, adapter_result, accounting)
|
|
140
144
|
accounting ||= {usage: adapter_result[:usage], provider_reported: !adapter_result[:usage].nil?}
|
|
141
|
-
|
|
145
|
+
schema_result = normalize_schema_response(adapter_result)
|
|
146
|
+
status = schema_result[:error] ? :failed : :succeeded
|
|
147
|
+
append_attempt(candidate, attempt_id, started_at, status, error: schema_result[:error], **accounting)
|
|
142
148
|
result = base_result(candidate).merge(
|
|
143
|
-
status:
|
|
144
|
-
content:
|
|
149
|
+
status: status,
|
|
150
|
+
content: schema_result[:content],
|
|
151
|
+
parsed: schema_result[:parsed],
|
|
145
152
|
tool_calls: normalize_tool_calls(adapter_result[:tool_calls]),
|
|
146
153
|
finish_reason: adapter_result[:finish_reason],
|
|
147
154
|
usage: aggregate_usage,
|
|
148
|
-
error:
|
|
155
|
+
error: schema_result[:error]
|
|
149
156
|
)
|
|
150
|
-
|
|
157
|
+
event = (status == :succeeded) ? :response_completed : :response_failed
|
|
158
|
+
emit(event, attempt_id:, result: result)
|
|
151
159
|
result
|
|
152
160
|
end
|
|
153
161
|
|
|
162
|
+
def normalize_schema_response(adapter_result)
|
|
163
|
+
return {content: adapter_result[:content] || "", parsed: nil, error: nil} unless schema_operation?
|
|
164
|
+
|
|
165
|
+
SchemaResponse.new(schema_definition).call(adapter_result)
|
|
166
|
+
end
|
|
167
|
+
|
|
154
168
|
def failure(candidate, attempt_id, started_at, error, partial:, accounting:)
|
|
155
169
|
status = failure_status(error, partial)
|
|
156
170
|
error = error.merge(retryable: false) if partial
|
|
@@ -343,14 +357,48 @@ module AgentHarness
|
|
|
343
357
|
end
|
|
344
358
|
|
|
345
359
|
def validate!
|
|
346
|
-
|
|
360
|
+
unless %i[chat schema].include?(request[:operation]&.to_sym)
|
|
361
|
+
raise ArgumentError, "operation must be :chat or :schema"
|
|
362
|
+
end
|
|
347
363
|
raise ArgumentError, "request_id is required" if request[:request_id].to_s.empty?
|
|
348
364
|
raise ArgumentError, "candidates must not be empty" if !request[:candidates].is_a?(Array) || request[:candidates].empty?
|
|
349
365
|
raise ArgumentError, "messages must be an array" unless request[:messages].is_a?(Array)
|
|
366
|
+
raise ArgumentError, "schema is required for a schema operation" if schema_operation? && !request[:schema].is_a?(Hash)
|
|
350
367
|
validate_attempt_limit!
|
|
351
368
|
candidates.each { |candidate| validate_candidate!(candidate) }
|
|
352
369
|
end
|
|
353
370
|
|
|
371
|
+
def schema_operation?
|
|
372
|
+
request[:operation].to_sym == :schema
|
|
373
|
+
end
|
|
374
|
+
|
|
375
|
+
def schema_mode_supported?
|
|
376
|
+
!schema_operation? || !request[:schema_mode] || request[:schema_mode].to_sym == :json_schema
|
|
377
|
+
end
|
|
378
|
+
|
|
379
|
+
def schema_payload
|
|
380
|
+
return unless schema_operation?
|
|
381
|
+
|
|
382
|
+
raw = request[:schema]
|
|
383
|
+
return raw.merge(name: request[:schema_name] || raw[:name]) if raw[:schema]
|
|
384
|
+
|
|
385
|
+
{name: request[:schema_name] || raw[:title] || "response", schema: raw}
|
|
386
|
+
end
|
|
387
|
+
|
|
388
|
+
def schema_definition
|
|
389
|
+
schema_payload.fetch(:schema)
|
|
390
|
+
end
|
|
391
|
+
|
|
392
|
+
def unsupported_schema_result
|
|
393
|
+
error = {
|
|
394
|
+
category: :unsupported,
|
|
395
|
+
code: :structured_output_not_supported,
|
|
396
|
+
retryable: false,
|
|
397
|
+
message: "Schema request failed (unsupported/structured_output_not_supported)"
|
|
398
|
+
}
|
|
399
|
+
failed_result(error, candidates.first)
|
|
400
|
+
end
|
|
401
|
+
|
|
354
402
|
def validate_attempt_limit!
|
|
355
403
|
limit = retry_config[:max_attempts]
|
|
356
404
|
raise ArgumentError, "retry.max_attempts must be a positive integer" unless limit.is_a?(Integer) && limit.positive?
|
|
@@ -5,6 +5,22 @@ require "ruby_llm"
|
|
|
5
5
|
|
|
6
6
|
module AgentHarness
|
|
7
7
|
module Api
|
|
8
|
+
# RubyLLM 2.0.0 flattens Responses API refusal deltas into ordinary text
|
|
9
|
+
# chunks and offers no public access to their event type. The gemspec pins
|
|
10
|
+
# that exact release so this compatibility shim cannot silently outlive the
|
|
11
|
+
# private parser shape it targets. Remove it when RubyLLM exposes refusals.
|
|
12
|
+
module RubyLlmResponsesStreamingRefusal
|
|
13
|
+
module RefusalChunk; end
|
|
14
|
+
|
|
15
|
+
def build_chunk(data)
|
|
16
|
+
super.tap do |chunk|
|
|
17
|
+
chunk.extend(RefusalChunk) if data["type"] == "response.refusal.delta"
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
RubyLLM::Protocols::Responses.prepend(RubyLlmResponsesStreamingRefusal)
|
|
23
|
+
|
|
8
24
|
# Translates the normalized public chat values to RubyLLM public objects.
|
|
9
25
|
class RubyLlmChatAdapter
|
|
10
26
|
class UnsupportedOptionError < StandardError; end
|
|
@@ -18,24 +34,24 @@ module AgentHarness
|
|
|
18
34
|
OPENAI_UNSUPPLIED_CONFIG = %i[openai_organization_id openai_project_id openai_use_system_role].freeze
|
|
19
35
|
|
|
20
36
|
def call(candidate:, messages:, tools:, max_output_tokens:, temperature:, stream:, timeout:, cancellation:,
|
|
21
|
-
on_accounting: nil, prepared_chat: nil, &on_event)
|
|
37
|
+
schema: nil, on_accounting: nil, prepared_chat: nil, &on_event)
|
|
22
38
|
provider_usage_reported = false
|
|
23
39
|
chat = prepared_chat || prepare(candidate:, messages:, tools:, max_output_tokens:, temperature:, timeout:,
|
|
24
|
-
on_accounting:, provider_usage: -> { provider_usage_reported })
|
|
25
|
-
response = generate(chat, stream, cancellation) do |event|
|
|
40
|
+
schema:, on_accounting:, provider_usage: -> { provider_usage_reported })
|
|
41
|
+
response, streamed_refusal = generate(chat, stream, cancellation) do |event|
|
|
26
42
|
provider_usage_reported = true if event[:type] == :usage_updated
|
|
27
43
|
on_event&.call(event)
|
|
28
44
|
end
|
|
29
45
|
emit_completed_tool_calls(response, &on_event) if stream
|
|
30
|
-
normalize_response(response)
|
|
46
|
+
normalize_response(response, streamed_refusal: streamed_refusal)
|
|
31
47
|
end
|
|
32
48
|
|
|
33
49
|
def prepare(candidate:, messages:, tools:, max_output_tokens:, temperature:, timeout:, on_accounting: nil,
|
|
34
|
-
provider_usage: -> { false })
|
|
50
|
+
schema: nil, provider_usage: -> { false })
|
|
35
51
|
context = build_context(candidate, timeout, on_accounting, provider_usage)
|
|
36
52
|
context.chat(model: candidate[:model], provider: candidate[:provider], protocol: ruby_llm_protocol(candidate),
|
|
37
53
|
assume_model_exists: true).tap do |chat|
|
|
38
|
-
configure_chat(chat, candidate, messages, tools, max_output_tokens, temperature)
|
|
54
|
+
configure_chat(chat, candidate, messages, tools, max_output_tokens, temperature, schema)
|
|
39
55
|
end
|
|
40
56
|
end
|
|
41
57
|
|
|
@@ -45,6 +61,7 @@ module AgentHarness
|
|
|
45
61
|
# duplicate cumulative reports are not re-emitted.
|
|
46
62
|
class StreamState
|
|
47
63
|
attr_accessor :input_tokens, :output_tokens
|
|
64
|
+
attr_reader :refusal
|
|
48
65
|
|
|
49
66
|
def initialize
|
|
50
67
|
@provider_id_by_key = {}
|
|
@@ -52,6 +69,7 @@ module AgentHarness
|
|
|
52
69
|
@latest_provider_id = nil
|
|
53
70
|
@input_tokens = nil
|
|
54
71
|
@output_tokens = nil
|
|
72
|
+
@refusal = false
|
|
55
73
|
end
|
|
56
74
|
|
|
57
75
|
# Links a stream chunk key to its provider call id, returning true
|
|
@@ -72,6 +90,10 @@ module AgentHarness
|
|
|
72
90
|
total = (input_tokens + output_tokens) if input_tokens && output_tokens
|
|
73
91
|
{input_tokens: input_tokens, output_tokens: output_tokens, total_tokens: total}
|
|
74
92
|
end
|
|
93
|
+
|
|
94
|
+
def observe(chunk)
|
|
95
|
+
@refusal ||= chunk.is_a?(RubyLlmResponsesStreamingRefusal::RefusalChunk)
|
|
96
|
+
end
|
|
75
97
|
end
|
|
76
98
|
|
|
77
99
|
private
|
|
@@ -150,19 +172,20 @@ module AgentHarness
|
|
|
150
172
|
config.request_timeout = timeout&.dig(:read_seconds) || DEFAULT_REQUEST_TIMEOUT
|
|
151
173
|
end
|
|
152
174
|
|
|
153
|
-
def configure_chat(chat, candidate, messages, tools, max_output_tokens, temperature)
|
|
175
|
+
def configure_chat(chat, candidate, messages, tools, max_output_tokens, temperature, schema)
|
|
154
176
|
chat.messages = normalize_messages(messages)
|
|
155
177
|
chat.with_tools(tools.map { |tool| normalized_tool(tool) }) unless tools.empty?
|
|
156
178
|
chat.with_headers(candidate[:headers] || {})
|
|
157
179
|
chat.with_max_output_tokens(max_output_tokens) if max_output_tokens
|
|
158
180
|
chat.with_temperature(temperature) unless temperature.nil?
|
|
181
|
+
chat.with_schema(schema) if schema
|
|
159
182
|
end
|
|
160
183
|
|
|
161
184
|
def generate(chat, stream, cancellation)
|
|
162
|
-
return generate_without_events(chat, cancellation) unless stream
|
|
185
|
+
return [generate_without_events(chat, cancellation), false] unless stream
|
|
163
186
|
|
|
164
187
|
state = StreamState.new
|
|
165
|
-
chat.generate do |chunk|
|
|
188
|
+
response = chat.generate do |chunk|
|
|
166
189
|
if cancelled?(cancellation)
|
|
167
190
|
chat.cancel
|
|
168
191
|
raise RubyLLM::CancelledError
|
|
@@ -170,6 +193,7 @@ module AgentHarness
|
|
|
170
193
|
|
|
171
194
|
stream_events(chunk, state).each { |event| yield event }
|
|
172
195
|
end
|
|
196
|
+
[response, state.refusal]
|
|
173
197
|
end
|
|
174
198
|
|
|
175
199
|
def generate_without_events(chat, cancellation)
|
|
@@ -235,6 +259,7 @@ module AgentHarness
|
|
|
235
259
|
end
|
|
236
260
|
|
|
237
261
|
def stream_events(chunk, state)
|
|
262
|
+
state.observe(chunk)
|
|
238
263
|
[text_event(chunk), *tool_call_events(chunk, state), usage_event(chunk, state)].compact
|
|
239
264
|
end
|
|
240
265
|
|
|
@@ -300,16 +325,46 @@ module AgentHarness
|
|
|
300
325
|
end
|
|
301
326
|
end
|
|
302
327
|
|
|
303
|
-
def normalize_response(response)
|
|
328
|
+
def normalize_response(response, streamed_refusal: false)
|
|
304
329
|
{
|
|
305
330
|
content: response.content || "",
|
|
306
331
|
model: response.model,
|
|
307
332
|
finish_reason: response.finish_reason,
|
|
308
333
|
usage: normalize_usage(response.tokens),
|
|
309
|
-
tool_calls: Array(response.tool_calls&.values).map { |call| normalize_tool_call(call) }
|
|
334
|
+
tool_calls: Array(response.tool_calls&.values).map { |call| normalize_tool_call(call) },
|
|
335
|
+
refusal: streamed_refusal || refusal?(response)
|
|
310
336
|
}
|
|
311
337
|
end
|
|
312
338
|
|
|
339
|
+
def refusal?(response)
|
|
340
|
+
return true if response.finish_reason == :content_filter
|
|
341
|
+
return false unless response.respond_to?(:raw)
|
|
342
|
+
|
|
343
|
+
body = response.raw&.body
|
|
344
|
+
return false unless body.is_a?(Hash)
|
|
345
|
+
|
|
346
|
+
responses_api_refusal?(body) || chat_completions_refusal?(body)
|
|
347
|
+
end
|
|
348
|
+
|
|
349
|
+
def responses_api_refusal?(body)
|
|
350
|
+
Array(body["output"] || body[:output]).any? do |item|
|
|
351
|
+
next false unless item.is_a?(Hash)
|
|
352
|
+
|
|
353
|
+
Array(item["content"] || item[:content]).any? do |part|
|
|
354
|
+
part.is_a?(Hash) && (part["type"] || part[:type]) == "refusal"
|
|
355
|
+
end
|
|
356
|
+
end
|
|
357
|
+
end
|
|
358
|
+
|
|
359
|
+
def chat_completions_refusal?(body)
|
|
360
|
+
Array(body["choices"] || body[:choices]).any? do |choice|
|
|
361
|
+
next false unless choice.is_a?(Hash)
|
|
362
|
+
|
|
363
|
+
message = choice["message"] || choice[:message]
|
|
364
|
+
message.is_a?(Hash) && !(message["refusal"] || message[:refusal]).nil?
|
|
365
|
+
end
|
|
366
|
+
end
|
|
367
|
+
|
|
313
368
|
def normalize_tool_call(call)
|
|
314
369
|
{provider_id: call.id, name: call.name, arguments_json: JSON.generate(call.arguments || {})}
|
|
315
370
|
end
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require "json_schemer"
|
|
5
|
+
|
|
6
|
+
module AgentHarness
|
|
7
|
+
module Api
|
|
8
|
+
# Parses and validates one schema-constrained provider response.
|
|
9
|
+
class SchemaResponse
|
|
10
|
+
def initialize(schema)
|
|
11
|
+
@validator = JSONSchemer.schema(schema)
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def call(response)
|
|
15
|
+
content = response[:content].to_s
|
|
16
|
+
return failure(content, :refusal) if response[:refusal]
|
|
17
|
+
return failure(content, :truncated_output) if response[:finish_reason]&.to_sym == :max_tokens
|
|
18
|
+
|
|
19
|
+
parsed = JSON.parse(content)
|
|
20
|
+
return failure(content, :invalid_schema) unless @validator.valid?(parsed)
|
|
21
|
+
|
|
22
|
+
{content: content, parsed: parsed, error: nil}
|
|
23
|
+
rescue JSON::ParserError
|
|
24
|
+
failure(content, :invalid_json)
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
private
|
|
28
|
+
|
|
29
|
+
def failure(content, code)
|
|
30
|
+
{
|
|
31
|
+
content: content,
|
|
32
|
+
parsed: nil,
|
|
33
|
+
error: {
|
|
34
|
+
category: :invalid_response,
|
|
35
|
+
code: code,
|
|
36
|
+
retryable: false,
|
|
37
|
+
message: "Schema response failed (invalid_response/#{code})"
|
|
38
|
+
}
|
|
39
|
+
}
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
end
|
data/lib/agent_harness.rb
CHANGED
|
@@ -473,6 +473,7 @@ require_relative "agent_harness/openai_compatible_transport"
|
|
|
473
473
|
require_relative "agent_harness/conversation"
|
|
474
474
|
require_relative "agent_harness/api/attempt_report"
|
|
475
475
|
require_relative "agent_harness/api/chat_transport"
|
|
476
|
+
require_relative "agent_harness/api/schema_response"
|
|
476
477
|
require_relative "agent_harness/quota_status"
|
|
477
478
|
require_relative "agent_harness/authentication"
|
|
478
479
|
require_relative "agent_harness/provider_health_check"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: agent-harness
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.43.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Bart Agapinan
|
|
@@ -30,25 +30,33 @@ dependencies:
|
|
|
30
30
|
- !ruby/object:Gem::Version
|
|
31
31
|
version: '2.0'
|
|
32
32
|
- !ruby/object:Gem::Dependency
|
|
33
|
-
name:
|
|
33
|
+
name: json_schemer
|
|
34
34
|
requirement: !ruby/object:Gem::Requirement
|
|
35
35
|
requirements:
|
|
36
|
-
- - "
|
|
37
|
-
- !ruby/object:Gem::Version
|
|
38
|
-
version: '2.0'
|
|
39
|
-
- - "<"
|
|
36
|
+
- - "~>"
|
|
40
37
|
- !ruby/object:Gem::Version
|
|
41
|
-
version: '
|
|
38
|
+
version: '2.4'
|
|
42
39
|
type: :runtime
|
|
43
40
|
prerelease: false
|
|
44
41
|
version_requirements: !ruby/object:Gem::Requirement
|
|
45
42
|
requirements:
|
|
46
|
-
- - "
|
|
43
|
+
- - "~>"
|
|
47
44
|
- !ruby/object:Gem::Version
|
|
48
|
-
version: '2.
|
|
49
|
-
|
|
45
|
+
version: '2.4'
|
|
46
|
+
- !ruby/object:Gem::Dependency
|
|
47
|
+
name: ruby_llm
|
|
48
|
+
requirement: !ruby/object:Gem::Requirement
|
|
49
|
+
requirements:
|
|
50
|
+
- - '='
|
|
50
51
|
- !ruby/object:Gem::Version
|
|
51
|
-
version:
|
|
52
|
+
version: 2.0.0
|
|
53
|
+
type: :runtime
|
|
54
|
+
prerelease: false
|
|
55
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
56
|
+
requirements:
|
|
57
|
+
- - '='
|
|
58
|
+
- !ruby/object:Gem::Version
|
|
59
|
+
version: 2.0.0
|
|
52
60
|
- !ruby/object:Gem::Dependency
|
|
53
61
|
name: rake
|
|
54
62
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -138,6 +146,7 @@ files:
|
|
|
138
146
|
- lib/agent_harness/api/attempt_report.rb
|
|
139
147
|
- lib/agent_harness/api/chat_transport.rb
|
|
140
148
|
- lib/agent_harness/api/ruby_llm_chat_adapter.rb
|
|
149
|
+
- lib/agent_harness/api/schema_response.rb
|
|
141
150
|
- lib/agent_harness/authentication.rb
|
|
142
151
|
- lib/agent_harness/command_executor.rb
|
|
143
152
|
- lib/agent_harness/configuration.rb
|