agent-harness 0.22.5 → 0.23.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/CHANGELOG.md +12 -0
- data/README.md +48 -0
- data/lib/agent_harness/model_compatibility.rb +145 -0
- data/lib/agent_harness/providers/adapter.rb +57 -0
- data/lib/agent_harness/providers/codex.rb +174 -0
- data/lib/agent_harness/providers/registry.rb +23 -0
- data/lib/agent_harness/version.rb +1 -1
- data/lib/agent_harness.rb +28 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: '0874add2f55550c3c9f9b723c84a4036a2dd79dd717dd62069062cfbde630820'
|
|
4
|
+
data.tar.gz: 53c6335c3a47e4e0e18ea3151dad061f9302519013e27591c7541ebf6dfe51db
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a7297580e4a5a34631421730cab4c0fc5712448864bacefe8c8f3c43c7652b5a42cb9f7c2a603f585f308d2a54117dff918804a45a30c3bb302ec223e545eed4
|
|
7
|
+
data.tar.gz: 36428c8e91bee3395d235cd4ae309f6a43c177acc6991cdf3cd33ed4f0cc433a566849a8b282137466a455da5becdc63820a87a01a36eb8399f857ecbe457ef7
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,17 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
### Features
|
|
4
|
+
|
|
5
|
+
* add runner model compatibility contract (`AgentHarness.model_compatibility`) with structured `ModelCompatibility::Result` outcomes. Codex exposes static facts for CLI-gated models (e.g. `gpt-5.5` requires Codex CLI `>= 0.116.0`), a baseline supported-model list, supported auth modes, and a `DEFAULT_COMPATIBLE_MODEL_ID` fallback so downstream orchestrators can validate tier/model assignments before scheduling agent runs ([#259](https://github.com/viamin/agent-harness/issues/259)).
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
## [0.23.0](https://github.com/viamin/agent-harness/compare/agent-harness/v0.22.5...agent-harness/v0.23.0) (2026-06-15)
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
### Features
|
|
12
|
+
|
|
13
|
+
* Add runner model compatibility contracts for Codex and CLI-gated models ([#260](https://github.com/viamin/agent-harness/issues/260)) ([4c192a7](https://github.com/viamin/agent-harness/commit/4c192a72f323ecaaea96e87e8ead6634e33669c0))
|
|
14
|
+
|
|
3
15
|
## [0.22.5](https://github.com/viamin/agent-harness/compare/agent-harness/v0.22.4...agent-harness/v0.22.5) (2026-06-12)
|
|
4
16
|
|
|
5
17
|
|
data/README.md
CHANGED
|
@@ -326,6 +326,54 @@ tests assert that the expected binary remains aligned with that contract.
|
|
|
326
326
|
`source_type`, `package_name`, version fields, and install commands so
|
|
327
327
|
downstream apps do not need provider-specific branching.
|
|
328
328
|
|
|
329
|
+
### Runner Model Compatibility
|
|
330
|
+
|
|
331
|
+
Downstream orchestrators that map tiers or task requirements to concrete
|
|
332
|
+
models should consult the runner/model compatibility contract before
|
|
333
|
+
scheduling a run. The contract surfaces the runtime dimensions that
|
|
334
|
+
materially affect whether a model will run — runner identity, model id,
|
|
335
|
+
auth mode, and installed CLI version — and returns a structured result
|
|
336
|
+
instead of forcing callers to parse error strings:
|
|
337
|
+
|
|
338
|
+
```ruby
|
|
339
|
+
result = AgentHarness.model_compatibility(
|
|
340
|
+
runner: :codex,
|
|
341
|
+
model_id: "gpt-5.5",
|
|
342
|
+
auth_mode: :subscription,
|
|
343
|
+
cli_version: "0.115.0"
|
|
344
|
+
)
|
|
345
|
+
|
|
346
|
+
result.supported? # => false
|
|
347
|
+
result.reason # => :cli_version_too_old
|
|
348
|
+
result.minimum_cli_version # => "0.116.0"
|
|
349
|
+
result.fallback_model_id # => "gpt-5-codex"
|
|
350
|
+
result.source # => :static_contract
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
Outcomes follow three explicit shapes:
|
|
354
|
+
|
|
355
|
+
- **Supported** — `result.supported?` is `true`. The runner contract
|
|
356
|
+
confirms the model can be driven under the given constraints.
|
|
357
|
+
- **Unsupported** — `result.unsupported?` is `true` with a stable
|
|
358
|
+
symbolic `reason` such as `:cli_version_too_old` or
|
|
359
|
+
`:auth_mode_not_supported`, plus any version requirement and a
|
|
360
|
+
contract-owned `fallback_model_id` to use instead.
|
|
361
|
+
- **Unknown** — `result.unknown?` is `true`. The runner cannot answer
|
|
362
|
+
definitively — either the model is not in the runner's static
|
|
363
|
+
contract (`reason: :unknown_model`), or the model is CLI-gated and the
|
|
364
|
+
caller did not supply a comparable `cli_version`
|
|
365
|
+
(`reason: :cli_version_unknown`, with `minimum_cli_version` attached).
|
|
366
|
+
Callers must treat unknown as "ask the provider" — never as implicit
|
|
367
|
+
approval — so a missing CLI version cannot silently schedule a run
|
|
368
|
+
onto an older CLI that the model does not support.
|
|
369
|
+
|
|
370
|
+
Providers that have not opted in to a static contract return an unknown
|
|
371
|
+
result by default, so callers always get a normalized
|
|
372
|
+
`AgentHarness::ModelCompatibility::Result` without provider-specific
|
|
373
|
+
branching. The Codex runner ships static facts for CLI-gated models
|
|
374
|
+
(for example `gpt-5.5` requires Codex CLI `>= 0.116.0`) and a baseline
|
|
375
|
+
list of always-supported models.
|
|
376
|
+
|
|
329
377
|
### Custom Providers
|
|
330
378
|
|
|
331
379
|
```ruby
|
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module AgentHarness
|
|
4
|
+
# Structured runner/model compatibility contract.
|
|
5
|
+
#
|
|
6
|
+
# ModelCompatibility is the single source of truth for whether a given
|
|
7
|
+
# runner (provider) can execute a particular model under specific runtime
|
|
8
|
+
# constraints such as authentication mode or installed CLI version.
|
|
9
|
+
# Downstream orchestrators (for example Paid's RDR-040 tier-to-model
|
|
10
|
+
# mapping) consume this contract before validating tier models, selecting
|
|
11
|
+
# runners, and starting agent runs — instead of inferring compatibility
|
|
12
|
+
# from scattered facts like CLI version pins, smoke-test overrides, or
|
|
13
|
+
# observed runtime errors.
|
|
14
|
+
#
|
|
15
|
+
# @example Query Codex compatibility for a CLI-gated model
|
|
16
|
+
# AgentHarness.model_compatibility(
|
|
17
|
+
# runner: :codex,
|
|
18
|
+
# model_id: "gpt-5.5",
|
|
19
|
+
# auth_mode: :subscription,
|
|
20
|
+
# cli_version: "0.116.0"
|
|
21
|
+
# )
|
|
22
|
+
# # => #<AgentHarness::ModelCompatibility::Result supported=true ...>
|
|
23
|
+
module ModelCompatibility
|
|
24
|
+
# Sentinel for "the runner declares no opinion about this model."
|
|
25
|
+
UNKNOWN_REASON = :unknown
|
|
26
|
+
# Issued when a runner does not advertise the requested model at all.
|
|
27
|
+
UNKNOWN_MODEL_REASON = :unknown_model
|
|
28
|
+
# Issued when the runner needs a comparable CLI version to answer
|
|
29
|
+
# definitively for a CLI-gated model but the caller did not supply one
|
|
30
|
+
# (or it could not be parsed). Pairs with :minimum_cli_version on the
|
|
31
|
+
# result. Distinct from :unknown_model — the runner *does* know the
|
|
32
|
+
# model; it just cannot confirm the installed CLI is new enough.
|
|
33
|
+
UNKNOWN_CLI_VERSION_REASON = :cli_version_unknown
|
|
34
|
+
# Issued when the runner supports the model but the installed CLI is too
|
|
35
|
+
# old. Pairs with :minimum_cli_version on the result.
|
|
36
|
+
UNSUPPORTED_CLI_VERSION_REASON = :cli_version_too_old
|
|
37
|
+
# Issued when the runner supports the model but the requested auth mode
|
|
38
|
+
# is not part of the runner's contract for it.
|
|
39
|
+
UNSUPPORTED_AUTH_MODE_REASON = :auth_mode_not_supported
|
|
40
|
+
# Default supported reason.
|
|
41
|
+
SUPPORTED_REASON = :supported
|
|
42
|
+
|
|
43
|
+
# Sources for a compatibility decision. Static contracts are baked into
|
|
44
|
+
# the provider; live probes hit the provider CLI/API; entitlement checks
|
|
45
|
+
# gate on subscription state. Downstream orchestrators can use this to
|
|
46
|
+
# decide whether to cache the answer.
|
|
47
|
+
SOURCES = %i[static_contract live_provider_probe entitlement_check unknown].freeze
|
|
48
|
+
|
|
49
|
+
# Structured compatibility outcome.
|
|
50
|
+
#
|
|
51
|
+
# The struct intentionally exposes whatever facts the runner contract
|
|
52
|
+
# knows. Callers should treat unknown fields as nil rather than fail.
|
|
53
|
+
Result = Struct.new(
|
|
54
|
+
:runner,
|
|
55
|
+
:model_id,
|
|
56
|
+
:auth_mode,
|
|
57
|
+
:cli_version,
|
|
58
|
+
:supported,
|
|
59
|
+
:reason,
|
|
60
|
+
:minimum_cli_version,
|
|
61
|
+
:cli_version_requirement,
|
|
62
|
+
:fallback_model_id,
|
|
63
|
+
:source,
|
|
64
|
+
:details,
|
|
65
|
+
keyword_init: true
|
|
66
|
+
) do
|
|
67
|
+
# @return [Boolean] true when compatibility is known to be supported
|
|
68
|
+
def supported? = supported == true
|
|
69
|
+
|
|
70
|
+
# @return [Boolean] true when compatibility is known to be unsupported
|
|
71
|
+
def unsupported? = supported == false
|
|
72
|
+
|
|
73
|
+
# @return [Boolean] true when the runner did not return a definite answer
|
|
74
|
+
def unknown? = supported.nil?
|
|
75
|
+
|
|
76
|
+
def to_h
|
|
77
|
+
super.compact
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
class << self
|
|
82
|
+
# Build a Result with sensible defaults from a partial Hash.
|
|
83
|
+
#
|
|
84
|
+
# Provider implementations call this helper to return a normalized
|
|
85
|
+
# Result without having to instantiate Struct fields by hand.
|
|
86
|
+
#
|
|
87
|
+
# @param runner [Symbol] canonical provider/runner name
|
|
88
|
+
# @param model_id [String, Symbol, nil] requested model id
|
|
89
|
+
# @param attributes [Hash] additional Result attributes
|
|
90
|
+
# @return [Result]
|
|
91
|
+
def build_result(runner:, model_id:, **attributes)
|
|
92
|
+
normalized_model_id = model_id.is_a?(Symbol) ? model_id.to_s : model_id
|
|
93
|
+
source = attributes.fetch(:source, :static_contract)
|
|
94
|
+
supported = attributes[:supported]
|
|
95
|
+
|
|
96
|
+
if supported == false && attributes[:reason].nil?
|
|
97
|
+
raise ArgumentError,
|
|
98
|
+
"AgentHarness::ModelCompatibility.build_result requires an explicit " \
|
|
99
|
+
"`reason:` when `supported: false`. Pass a specific reason " \
|
|
100
|
+
"(e.g. :cli_version_too_old, :auth_mode_not_supported) so " \
|
|
101
|
+
"callers can distinguish unsupported from :unknown."
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
reason = attributes[:reason] || default_reason_for(supported)
|
|
105
|
+
|
|
106
|
+
Result.new(
|
|
107
|
+
runner: runner.to_sym,
|
|
108
|
+
model_id: normalized_model_id,
|
|
109
|
+
auth_mode: attributes[:auth_mode],
|
|
110
|
+
cli_version: attributes[:cli_version],
|
|
111
|
+
supported: supported,
|
|
112
|
+
reason: reason,
|
|
113
|
+
minimum_cli_version: attributes[:minimum_cli_version],
|
|
114
|
+
cli_version_requirement: attributes[:cli_version_requirement],
|
|
115
|
+
fallback_model_id: attributes[:fallback_model_id],
|
|
116
|
+
source: source,
|
|
117
|
+
details: attributes[:details]
|
|
118
|
+
)
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
# Build an explicit "unknown" Result. Use when the runner has no
|
|
122
|
+
# static contract for the requested model and cannot probe.
|
|
123
|
+
def unknown_result(runner:, model_id:, fallback_model_id: nil, **attributes)
|
|
124
|
+
build_result(
|
|
125
|
+
runner: runner,
|
|
126
|
+
model_id: model_id,
|
|
127
|
+
supported: nil,
|
|
128
|
+
reason: attributes.delete(:reason) || UNKNOWN_REASON,
|
|
129
|
+
fallback_model_id: fallback_model_id,
|
|
130
|
+
source: attributes.delete(:source) || :unknown,
|
|
131
|
+
**attributes
|
|
132
|
+
)
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
private
|
|
136
|
+
|
|
137
|
+
def default_reason_for(supported)
|
|
138
|
+
case supported
|
|
139
|
+
when true then SUPPORTED_REASON
|
|
140
|
+
else UNKNOWN_REASON
|
|
141
|
+
end
|
|
142
|
+
end
|
|
143
|
+
end
|
|
144
|
+
end
|
|
145
|
+
end
|
|
@@ -709,6 +709,63 @@ module AgentHarness
|
|
|
709
709
|
nil
|
|
710
710
|
end
|
|
711
711
|
|
|
712
|
+
# Runner/model compatibility contract.
|
|
713
|
+
#
|
|
714
|
+
# Returns a structured {AgentHarness::ModelCompatibility::Result}
|
|
715
|
+
# describing whether this runner can execute +model_id+ under the
|
|
716
|
+
# given +auth_mode+ and installed +cli_version+. Providers that own
|
|
717
|
+
# CLI-gated or entitlement-gated models should override this to
|
|
718
|
+
# expose stable, structured facts so downstream orchestrators do not
|
|
719
|
+
# have to parse error strings or rediscover provider knowledge.
|
|
720
|
+
#
|
|
721
|
+
# The default implementation returns an explicit +:unknown+ result
|
|
722
|
+
# rather than collapsing to "supported" — callers are expected to
|
|
723
|
+
# handle unknown outcomes explicitly.
|
|
724
|
+
#
|
|
725
|
+
# @param model_id [String, Symbol] the requested model identifier
|
|
726
|
+
# @param auth_mode [Symbol, nil] caller's auth mode (e.g. :api_key,
|
|
727
|
+
# :subscription)
|
|
728
|
+
# @param cli_version [String, Gem::Version, nil] installed CLI
|
|
729
|
+
# version, when known
|
|
730
|
+
# @return [AgentHarness::ModelCompatibility::Result]
|
|
731
|
+
def model_compatibility(model_id:, auth_mode: nil, cli_version: nil)
|
|
732
|
+
AgentHarness::ModelCompatibility.unknown_result(
|
|
733
|
+
runner: provider_name,
|
|
734
|
+
model_id: model_id,
|
|
735
|
+
auth_mode: auth_mode,
|
|
736
|
+
cli_version: normalize_cli_version_for_compatibility(cli_version),
|
|
737
|
+
fallback_model_id: default_compatible_model_id
|
|
738
|
+
)
|
|
739
|
+
end
|
|
740
|
+
|
|
741
|
+
# Optional default model the runner can recommend as a fallback when
|
|
742
|
+
# a requested model is unsupported or unknown. Providers override
|
|
743
|
+
# this when they own a stable default model for smoke tests or
|
|
744
|
+
# downstream tier fallback.
|
|
745
|
+
#
|
|
746
|
+
# @return [String, nil]
|
|
747
|
+
def default_compatible_model_id
|
|
748
|
+
nil
|
|
749
|
+
end
|
|
750
|
+
|
|
751
|
+
# Helper for {#model_compatibility} implementations: coerce a CLI
|
|
752
|
+
# version string into a +Gem::Version+ when possible, returning the
|
|
753
|
+
# raw value otherwise. Keeps Result#cli_version stable across
|
|
754
|
+
# callers that pass either form.
|
|
755
|
+
def normalize_cli_version_for_compatibility(value)
|
|
756
|
+
return nil if value.nil?
|
|
757
|
+
return value if value.is_a?(Gem::Version)
|
|
758
|
+
|
|
759
|
+
str = value.respond_to?(:strip) ? value.strip : value.to_s
|
|
760
|
+
return nil if str.empty?
|
|
761
|
+
|
|
762
|
+
begin
|
|
763
|
+
Gem::Version.new(str)
|
|
764
|
+
rescue ArgumentError
|
|
765
|
+
str
|
|
766
|
+
end
|
|
767
|
+
end
|
|
768
|
+
|
|
712
769
|
private
|
|
713
770
|
|
|
714
771
|
def versioned_installation_contract(version)
|
|
@@ -20,6 +20,43 @@ module AgentHarness
|
|
|
20
20
|
|
|
21
21
|
SUPPORTED_CLI_VERSION = "0.122.0"
|
|
22
22
|
SUPPORTED_CLI_REQUIREMENT = Gem::Requirement.new(">= #{SUPPORTED_CLI_VERSION}", "< 0.123.0").freeze
|
|
23
|
+
|
|
24
|
+
# Default model recommended by the Codex runner contract when callers
|
|
25
|
+
# have no explicit preference. Used as the {AgentHarness::ModelCompatibility::Result#fallback_model_id}
|
|
26
|
+
# for unsupported/unknown model lookups so downstream orchestrators
|
|
27
|
+
# (smoke tests, tier fallback) have a stable, contract-backed choice
|
|
28
|
+
# instead of relying on whichever model the CLI's default points at.
|
|
29
|
+
DEFAULT_COMPATIBLE_MODEL_ID = "gpt-5-codex"
|
|
30
|
+
|
|
31
|
+
# Known CLI-gated model facts. Each entry expresses the minimum Codex
|
|
32
|
+
# CLI version required to drive that model. Keep entries here only when
|
|
33
|
+
# the requirement is durable runner contract knowledge — not
|
|
34
|
+
# provider-side experiments or one-off CLI defaults.
|
|
35
|
+
#
|
|
36
|
+
# The +gpt-5.5+ entry tracks the failure class observed in
|
|
37
|
+
# viamin/agent-harness#245 and viamin/agent-harness#250: older Codex
|
|
38
|
+
# CLI builds (e.g. 0.115.x) could not drive the +gpt-5.5+ family.
|
|
39
|
+
MODEL_COMPATIBILITY_FACTS = {
|
|
40
|
+
"gpt-5.5" => {minimum_cli_version: "0.116.0"},
|
|
41
|
+
"gpt-5.5-codex" => {minimum_cli_version: "0.116.0"}
|
|
42
|
+
}.each_value(&:freeze).freeze
|
|
43
|
+
|
|
44
|
+
# Models that the runner contract considers supported on every Codex
|
|
45
|
+
# CLI release we ship. Used by {.model_compatibility} so callers can
|
|
46
|
+
# distinguish "unknown to the contract" from "explicitly supported."
|
|
47
|
+
BASELINE_SUPPORTED_MODELS = %w[
|
|
48
|
+
gpt-5
|
|
49
|
+
gpt-5-codex
|
|
50
|
+
gpt-5-mini
|
|
51
|
+
gpt-4o
|
|
52
|
+
gpt-4o-mini
|
|
53
|
+
o4-mini
|
|
54
|
+
].freeze
|
|
55
|
+
|
|
56
|
+
# Auth modes the Codex runner accepts. Compatibility checks for an
|
|
57
|
+
# unrecognised auth mode return :auth_mode_not_supported rather than
|
|
58
|
+
# silently approving the request.
|
|
59
|
+
SUPPORTED_AUTH_MODES = %i[api_key subscription].freeze
|
|
23
60
|
OAUTH_REFRESH_FAILURE_PATTERNS = [
|
|
24
61
|
/refresh_token_reused/i,
|
|
25
62
|
/failed to refresh token\b.*\b401\b/im,
|
|
@@ -232,6 +269,143 @@ module AgentHarness
|
|
|
232
269
|
Base::DEFAULT_SMOKE_TEST_CONTRACT
|
|
233
270
|
end
|
|
234
271
|
|
|
272
|
+
def default_compatible_model_id
|
|
273
|
+
DEFAULT_COMPATIBLE_MODEL_ID
|
|
274
|
+
end
|
|
275
|
+
|
|
276
|
+
# Structured Codex compatibility contract.
|
|
277
|
+
#
|
|
278
|
+
# Returns an {AgentHarness::ModelCompatibility::Result} for the
|
|
279
|
+
# combination of +model_id+, +auth_mode+, and installed
|
|
280
|
+
# +cli_version+. The contract surfaces three concrete outcomes:
|
|
281
|
+
#
|
|
282
|
+
# 1. **Supported** — the model is in {BASELINE_SUPPORTED_MODELS} (or
|
|
283
|
+
# a known CLI-gated model whose minimum CLI version is met).
|
|
284
|
+
# 2. **Unsupported due to CLI version** — the model is known to need
|
|
285
|
+
# a newer Codex CLI than was supplied; the result carries
|
|
286
|
+
# +:minimum_cli_version+ so callers can act on it.
|
|
287
|
+
# 3. **Unknown / dynamic** — the model is not in this static
|
|
288
|
+
# contract. Callers must treat this as "ask the provider" rather
|
|
289
|
+
# than as approval.
|
|
290
|
+
#
|
|
291
|
+
# @param model_id [String, Symbol]
|
|
292
|
+
# @param auth_mode [Symbol, nil] :api_key or :subscription
|
|
293
|
+
# @param cli_version [String, Gem::Version, nil]
|
|
294
|
+
# @return [AgentHarness::ModelCompatibility::Result]
|
|
295
|
+
def model_compatibility(model_id:, auth_mode: nil, cli_version: nil)
|
|
296
|
+
normalized_model_id = model_id.to_s
|
|
297
|
+
normalized_auth_mode = auth_mode&.to_sym
|
|
298
|
+
normalized_cli_version = normalize_cli_version_for_compatibility(cli_version)
|
|
299
|
+
|
|
300
|
+
if normalized_auth_mode && !SUPPORTED_AUTH_MODES.include?(normalized_auth_mode)
|
|
301
|
+
return AgentHarness::ModelCompatibility.build_result(
|
|
302
|
+
runner: provider_name,
|
|
303
|
+
model_id: normalized_model_id,
|
|
304
|
+
auth_mode: normalized_auth_mode,
|
|
305
|
+
cli_version: normalized_cli_version,
|
|
306
|
+
supported: false,
|
|
307
|
+
reason: AgentHarness::ModelCompatibility::UNSUPPORTED_AUTH_MODE_REASON,
|
|
308
|
+
fallback_model_id: DEFAULT_COMPATIBLE_MODEL_ID,
|
|
309
|
+
source: :static_contract,
|
|
310
|
+
details: {supported_auth_modes: SUPPORTED_AUTH_MODES}
|
|
311
|
+
)
|
|
312
|
+
end
|
|
313
|
+
|
|
314
|
+
gated_fact = MODEL_COMPATIBILITY_FACTS[normalized_model_id]
|
|
315
|
+
if gated_fact
|
|
316
|
+
minimum_version = gated_fact[:minimum_cli_version]
|
|
317
|
+
requirement = Gem::Requirement.new(">= #{minimum_version}")
|
|
318
|
+
comparable_version = comparable_cli_version(normalized_cli_version)
|
|
319
|
+
|
|
320
|
+
# A CLI-gated model without a comparable installed version must
|
|
321
|
+
# stay explicit. Returning :supported here would re-introduce the
|
|
322
|
+
# exact `gpt-5.5` failure class this contract is designed to
|
|
323
|
+
# prevent — a caller that cannot supply a version would get
|
|
324
|
+
# `supported? == true` and may still schedule a run onto an old
|
|
325
|
+
# CLI (e.g. 0.115.x). Surface :unknown with the requirement
|
|
326
|
+
# attached so callers can decide deliberately.
|
|
327
|
+
if comparable_version.nil?
|
|
328
|
+
return AgentHarness::ModelCompatibility.unknown_result(
|
|
329
|
+
runner: provider_name,
|
|
330
|
+
model_id: normalized_model_id,
|
|
331
|
+
auth_mode: normalized_auth_mode,
|
|
332
|
+
cli_version: normalized_cli_version,
|
|
333
|
+
reason: AgentHarness::ModelCompatibility::UNKNOWN_CLI_VERSION_REASON,
|
|
334
|
+
minimum_cli_version: minimum_version,
|
|
335
|
+
cli_version_requirement: requirement.to_s,
|
|
336
|
+
fallback_model_id: DEFAULT_COMPATIBLE_MODEL_ID,
|
|
337
|
+
source: :static_contract
|
|
338
|
+
)
|
|
339
|
+
end
|
|
340
|
+
|
|
341
|
+
unless requirement.satisfied_by?(comparable_version)
|
|
342
|
+
return AgentHarness::ModelCompatibility.build_result(
|
|
343
|
+
runner: provider_name,
|
|
344
|
+
model_id: normalized_model_id,
|
|
345
|
+
auth_mode: normalized_auth_mode,
|
|
346
|
+
cli_version: normalized_cli_version,
|
|
347
|
+
supported: false,
|
|
348
|
+
reason: AgentHarness::ModelCompatibility::UNSUPPORTED_CLI_VERSION_REASON,
|
|
349
|
+
minimum_cli_version: minimum_version,
|
|
350
|
+
cli_version_requirement: requirement.to_s,
|
|
351
|
+
fallback_model_id: DEFAULT_COMPATIBLE_MODEL_ID,
|
|
352
|
+
source: :static_contract
|
|
353
|
+
)
|
|
354
|
+
end
|
|
355
|
+
|
|
356
|
+
return AgentHarness::ModelCompatibility.build_result(
|
|
357
|
+
runner: provider_name,
|
|
358
|
+
model_id: normalized_model_id,
|
|
359
|
+
auth_mode: normalized_auth_mode,
|
|
360
|
+
cli_version: normalized_cli_version,
|
|
361
|
+
supported: true,
|
|
362
|
+
reason: AgentHarness::ModelCompatibility::SUPPORTED_REASON,
|
|
363
|
+
minimum_cli_version: minimum_version,
|
|
364
|
+
cli_version_requirement: requirement.to_s,
|
|
365
|
+
source: :static_contract
|
|
366
|
+
)
|
|
367
|
+
end
|
|
368
|
+
|
|
369
|
+
if BASELINE_SUPPORTED_MODELS.include?(normalized_model_id)
|
|
370
|
+
return AgentHarness::ModelCompatibility.build_result(
|
|
371
|
+
runner: provider_name,
|
|
372
|
+
model_id: normalized_model_id,
|
|
373
|
+
auth_mode: normalized_auth_mode,
|
|
374
|
+
cli_version: normalized_cli_version,
|
|
375
|
+
supported: true,
|
|
376
|
+
reason: AgentHarness::ModelCompatibility::SUPPORTED_REASON,
|
|
377
|
+
source: :static_contract
|
|
378
|
+
)
|
|
379
|
+
end
|
|
380
|
+
|
|
381
|
+
AgentHarness::ModelCompatibility.unknown_result(
|
|
382
|
+
runner: provider_name,
|
|
383
|
+
model_id: normalized_model_id,
|
|
384
|
+
auth_mode: normalized_auth_mode,
|
|
385
|
+
cli_version: normalized_cli_version,
|
|
386
|
+
reason: AgentHarness::ModelCompatibility::UNKNOWN_MODEL_REASON,
|
|
387
|
+
fallback_model_id: DEFAULT_COMPATIBLE_MODEL_ID
|
|
388
|
+
)
|
|
389
|
+
end
|
|
390
|
+
|
|
391
|
+
# Coerce normalized CLI version into a +Gem::Version+ usable for
|
|
392
|
+
# requirement comparison. Returns +nil+ when the value is missing or
|
|
393
|
+
# cannot be parsed — callers treat that as "no installed-version
|
|
394
|
+
# signal," not as a failure.
|
|
395
|
+
def comparable_cli_version(value)
|
|
396
|
+
return value if value.is_a?(Gem::Version)
|
|
397
|
+
return nil if value.nil?
|
|
398
|
+
|
|
399
|
+
str = value.respond_to?(:strip) ? value.strip : value.to_s
|
|
400
|
+
return nil if str.empty?
|
|
401
|
+
|
|
402
|
+
begin
|
|
403
|
+
Gem::Version.new(str)
|
|
404
|
+
rescue ArgumentError
|
|
405
|
+
nil
|
|
406
|
+
end
|
|
407
|
+
end
|
|
408
|
+
|
|
235
409
|
def parse_cli_jsonl_transcript(raw_output, max_events: nil)
|
|
236
410
|
return parser_instance.send(:parse_jsonl_output, "") if max_events && max_events <= 0
|
|
237
411
|
|
|
@@ -181,6 +181,29 @@ module AgentHarness
|
|
|
181
181
|
end
|
|
182
182
|
end
|
|
183
183
|
|
|
184
|
+
# Query runner/model compatibility contract for a provider.
|
|
185
|
+
#
|
|
186
|
+
# Delegates to the provider class's +.model_compatibility+ method,
|
|
187
|
+
# which returns a structured {AgentHarness::ModelCompatibility::Result}.
|
|
188
|
+
# The default implementation in {AgentHarness::Providers::Adapter}
|
|
189
|
+
# returns +:unknown+ for providers that have not opted in to a
|
|
190
|
+
# static contract.
|
|
191
|
+
#
|
|
192
|
+
# @param name [Symbol, String] the provider name or alias
|
|
193
|
+
# @param model_id [String, Symbol] requested model identifier
|
|
194
|
+
# @param auth_mode [Symbol, nil] caller's auth mode
|
|
195
|
+
# @param cli_version [String, Gem::Version, nil] installed CLI version
|
|
196
|
+
# @return [AgentHarness::ModelCompatibility::Result]
|
|
197
|
+
# @raise [ConfigurationError] if the provider name is not registered
|
|
198
|
+
def model_compatibility(name, model_id:, auth_mode: nil, cli_version: nil)
|
|
199
|
+
klass = get(name)
|
|
200
|
+
klass.model_compatibility(
|
|
201
|
+
model_id: model_id,
|
|
202
|
+
auth_mode: auth_mode,
|
|
203
|
+
cli_version: cli_version
|
|
204
|
+
)
|
|
205
|
+
end
|
|
206
|
+
|
|
184
207
|
# Get smoke-test metadata for a provider.
|
|
185
208
|
#
|
|
186
209
|
# @param name [Symbol, String] the provider name
|
data/lib/agent_harness.rb
CHANGED
|
@@ -257,6 +257,33 @@ module AgentHarness
|
|
|
257
257
|
Providers::Registry.instance.smoke_test_contracts
|
|
258
258
|
end
|
|
259
259
|
|
|
260
|
+
# Query runner/model compatibility contract.
|
|
261
|
+
#
|
|
262
|
+
# Returns a structured {AgentHarness::ModelCompatibility::Result}
|
|
263
|
+
# describing whether the named runner can execute +model_id+ under
|
|
264
|
+
# the requested runtime constraints. Downstream orchestrators should
|
|
265
|
+
# consume this contract before validating tier models, selecting a
|
|
266
|
+
# runner, or scheduling work — rather than inferring compatibility
|
|
267
|
+
# from scattered CLI version pins, smoke-test overrides, or runtime
|
|
268
|
+
# error strings.
|
|
269
|
+
#
|
|
270
|
+
# @param runner [Symbol, String] provider/runner name (e.g. :codex)
|
|
271
|
+
# @param model_id [String, Symbol] requested model identifier
|
|
272
|
+
# @param auth_mode [Symbol, nil] caller's auth mode (e.g. :api_key,
|
|
273
|
+
# :subscription)
|
|
274
|
+
# @param cli_version [String, Gem::Version, nil] installed CLI
|
|
275
|
+
# version, when known
|
|
276
|
+
# @return [AgentHarness::ModelCompatibility::Result]
|
|
277
|
+
# @raise [ConfigurationError] if the runner is not registered
|
|
278
|
+
def model_compatibility(runner:, model_id:, auth_mode: nil, cli_version: nil)
|
|
279
|
+
Providers::Registry.instance.model_compatibility(
|
|
280
|
+
runner,
|
|
281
|
+
model_id: model_id,
|
|
282
|
+
auth_mode: auth_mode,
|
|
283
|
+
cli_version: cli_version
|
|
284
|
+
)
|
|
285
|
+
end
|
|
286
|
+
|
|
260
287
|
# Check if authentication is valid for a provider
|
|
261
288
|
# @param provider_name [Symbol] the provider name
|
|
262
289
|
# @return [Boolean] true if auth is valid
|
|
@@ -395,6 +422,7 @@ require_relative "agent_harness/authentication"
|
|
|
395
422
|
require_relative "agent_harness/provider_health_check"
|
|
396
423
|
require_relative "agent_harness/release_registry"
|
|
397
424
|
require_relative "agent_harness/dependency_updater"
|
|
425
|
+
require_relative "agent_harness/model_compatibility"
|
|
398
426
|
|
|
399
427
|
# Provider layer
|
|
400
428
|
require_relative "agent_harness/providers/registry"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: agent-harness
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.23.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Bart Agapinan
|
|
@@ -113,6 +113,7 @@ files:
|
|
|
113
113
|
- lib/agent_harness/mcp_config_loader.rb
|
|
114
114
|
- lib/agent_harness/mcp_config_translator.rb
|
|
115
115
|
- lib/agent_harness/mcp_server.rb
|
|
116
|
+
- lib/agent_harness/model_compatibility.rb
|
|
116
117
|
- lib/agent_harness/openai_compatible_transport.rb
|
|
117
118
|
- lib/agent_harness/orchestration/circuit_breaker.rb
|
|
118
119
|
- lib/agent_harness/orchestration/conductor.rb
|