ruby_llm-contract 1.4.1 → 1.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 58ba70123cbc8bc4c7d3c9dc85ce430f1f1492ceb031df35ae0ea3f18f534dd8
4
- data.tar.gz: 24fed341a96267536d66532926d0e19cb4eb59e2ffd95467be1ca861d9dbe4f9
3
+ metadata.gz: cfebb3e04475038261441557a159265c51e6000104b495c193b526a0e6aab0b3
4
+ data.tar.gz: 86cd96a083faf0a9e7c2c310bc474e5981e33bd947cae5ca2e680f90fc8148d7
5
5
  SHA512:
6
- metadata.gz: 2cc04307951beb543adb0aae5876013767256e90501ed65be68cc9b4918e547568860bd377cf94651777faaa3159b5be47d3b054445223ee5d7f1a86a5aea697
7
- data.tar.gz: 8c06562bc7093a78d50c3e42294875884e6537d326bf7a474dc69985f7a5798209cedbe2691b6acd83050857ef715d3ef963f5a08358bc3c79c2984e51a76014
6
+ metadata.gz: 200c74207f9f91503ec60e505cbbc2bea4c0e8d83b09a3cbe6dc1c9a85a0afa8e657af81dce4b3444e5477e0a06261508161fb88514a734fa394d014b31382c4
7
+ data.tar.gz: 20bd3f86e2fbe26b7855055e7d2cdd5847db80341ef5d17f561917fd1dfeea7bbe1f33383c06e42bd7d9d54466dfb884cd1ec93e7c3ad1a7de63a2d1520ebaed
data/CHANGELOG.md CHANGED
@@ -1,5 +1,67 @@
1
1
  # Changelog
2
2
 
3
+ ## 1.5.1 (2026-11-17)
4
+
5
+ ### Changed
6
+
7
+ - **Checked against `ruby_llm` 2.2.2.** The offline suite and the live specs
8
+ pass on it, and no code changes are needed: the gemspec already allows it.
9
+ Run `bundle update ruby_llm` to pick it up. Two things change with the
10
+ upgrade itself, below.
11
+ - **With no model set anywhere, `ruby_llm` 2.2 runs `gpt-6.1-sol`.** A call
12
+ takes its model from the retry chain, then `context: { model: }`, then the
13
+ Step's `model`, then `default_model` in `RubyLLM::Contract.configure`. With
14
+ none of these the adapter passes no model, and `ruby_llm` uses
15
+ `RubyLLM.config.default_model`, which 2.2 changed from `gpt-5.6` to
16
+ `gpt-6.1-sol` unless you set it. A Step with `max_cost` refuses such a call
17
+ with `:limit_exceeded`, as before, since it cannot price a model it does not
18
+ know; with `on_unknown_pricing: :warn` it sends it with a warning and no cost
19
+ limit. Set `default_model` to keep the model you tested.
20
+ - **On `ruby_llm` 2.2 a network failure is a `RubyLLM::ConnectionError`.**
21
+ 2.1 raised the Faraday error itself (`Faraday::ConnectionFailed`,
22
+ `Faraday::TimeoutError`); 2.2 wraps it, with the Faraday error as its
23
+ `cause`. A failed completion request still returns `:adapter_error`, but
24
+ `trace[:error_class]` changes, and a `retry_on` that names a Faraday class
25
+ no longer matches. Name `RubyLLM::ConnectionError` instead after upgrading.
26
+ A provider or network error while `token_count :exact` counts the input
27
+ ends as `:limit_exceeded`, as before, so a `retry_on` naming an error class
28
+ does not retry it.
29
+
30
+ ### Fixed
31
+
32
+ - **A `max_cost` refusal with no model says no model is set.** It read
33
+ `model '' has no pricing data` and advised registering pricing. It now says
34
+ `no model is set for this Step` and advises setting `default_model` in
35
+ `RubyLLM::Contract.configure`, a model on the Step, or `context: { model: }`
36
+ (`RubyLLM.config.default_model` does not count). The
37
+ `on_unknown_pricing: :warn` warning also says no model is set.
38
+ - **The guides match the code again.** Among others: the RubyLLM chat
39
+ callbacks are `after_message` and `before_tool_call` (`testing.md` named
40
+ methods that do not exist); measuring quality with `run_eval` needs an
41
+ explicit adapter, since a `model:` alone keeps the `sample_response`
42
+ (`eval_first.md`); `max_output` caps the reply rather than refusing a call;
43
+ `optimize` shows scores, not cost, and builds its chain greedily; a
44
+ pipeline eval can assert on intermediate steps with `step_expectations:`;
45
+ `estimate_cost` does not read the retry chain; `RubyLLM::Message.new` takes
46
+ `model:`, not `model_id:`; and `retry_policy` takes `escalate` in a block.
47
+ - **The live `compare_models` specs name a Sonnet model that exists.** They
48
+ used `claude-sonnet-4-5-20250514`, which no `ruby_llm` registry lists, so
49
+ with `ANTHROPIC_API_KEY` set the three of them raised `ModelNotFoundError`.
50
+ They now use `claude-sonnet-4-5-20250929`.
51
+
52
+ ## 1.5.0 (2026-10-13)
53
+
54
+ ### Changed
55
+
56
+ - **Requires `ruby_llm` 2.1.1 or later in 2.x (was `~> 2.0`).** faraday-net_http
57
+ 3.5.0 (2026-10-11) declares no Faraday dependency, so a fresh install of ruby_llm
58
+ 2.0.0 to 2.1.0 resolves Faraday 2.0.0, which loads no default adapter, and every
59
+ call fails with `:net_http is not registered on Faraday::Adapter`. ruby_llm 2.1.1
60
+ excludes Faraday 2.0.0. An app whose lockfile already has a working Faraday ran
61
+ fine on the older releases, but CI now tests 2.1.1 and the latest 2.x, so the
62
+ gemspec allows what is tested. Run `bundle update ruby_llm` if you are on an
63
+ older 2.x. No code changes are needed.
64
+
3
65
  ## 1.4.1 (2026-10-11)
4
66
 
5
67
  ### Fixed
data/README.md CHANGED
@@ -24,7 +24,9 @@ RubyLLM::Contract.configure do |c|
24
24
  end
25
25
  ```
26
26
 
27
- Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc). Requires `ruby_llm ~> 2.0` and Ruby ≥ 3.2.
27
+ Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc). Requires `ruby_llm` 2.1.1 or later in 2.x and Ruby ≥ 3.2.
28
+
29
+ A call takes its model from the retry chain, then `context: { model: }`, then the Step's `model`, then `default_model` here. With none of these it goes to `ruby_llm`'s own `RubyLLM.config.default_model`, which `ruby_llm` 2.2 changed from `gpt-5.6` to `gpt-6.1-sol` unless you set it. A Step with `max_cost` refuses such a call with `:limit_exceeded`, since it cannot price a model it does not know; with `on_unknown_pricing: :warn` it sends it with a warning and no cost limit. Set `default_model` to keep the model you tested.
28
30
 
29
31
  ## Example
30
32
 
@@ -115,7 +117,7 @@ end
115
117
  expect(SummarizeArticle).to pass_eval("regression").without_regressions
116
118
  ```
117
119
 
118
- A bad prompt edit or model swap that drops accuracy on the frozen dataset → red CI, blocked merge. Save a baseline once (`report.save_baseline!`, or `t.save_baseline = true` in the rake task); until one exists, `without_regressions` has nothing to compare against and passes. Every production miss should become the next `add_case`. See [Prevent silent prompt regressions](docs/guide/eval_first.md) for the full flywheel.
120
+ A bad prompt edit or model swap that drops accuracy on the frozen dataset → red CI, blocked merge. Save a baseline once (`report.save_baseline!`, or `t.save_baseline = true` in the rake task); until one exists, `without_regressions` has nothing to compare against, and the matcher checks only that the eval passes. Every production miss should become the next `add_case`. See [Prevent silent prompt regressions](docs/guide/eval_first.md) for the full flywheel.
119
121
 
120
122
  ## Do I need this?
121
123
 
@@ -144,7 +146,7 @@ Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and
144
146
 
145
147
  ## Relation to `RubyLLM::Agent`
146
148
 
147
- `Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly - Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate(...)` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. RubyLLM 2.1's `RubyLLM::Evaluation` grades answers; `define_eval` guards against regressions and cost in CI - the mapping covers both. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
149
+ `Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly - Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy { escalate ... }` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. RubyLLM 2.1's `RubyLLM::Evaluation` grades answers; `define_eval` guards against regressions and cost in CI - the mapping covers both. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
148
150
 
149
151
  ## Relation to `ruby_llm-tribunal`
150
152
 
data/docs/architecture.md CHANGED
@@ -9,7 +9,7 @@ RubyLLM::Contract::Step::Base # single contracted step
9
9
  ├── Step::Dsl # DSL macros (prompt, validate, output_schema, etc.)
10
10
  ├── Step::RetryPolicy # attempts, models, reasoning_effort, retry_on
11
11
  ├── Step::RetryExecutor # retry loop driven by RetryPolicy
12
- ├── Step::LimitChecker # preflight cost / input / output checks
12
+ ├── Step::LimitChecker # preflight cost / input checks (max_output feeds the cost estimate)
13
13
  ├── Step::Runner # runtime flow (single attempt)
14
14
  ├── Step::Result # status + outputs + errors + trace
15
15
  ├── Step::Trace # model, latency, tokens, cost, attempts
@@ -33,13 +33,13 @@ RubyLLM::Contract::Eval # quality measurement
33
33
  ├── Eval::Dataset # test cases
34
34
  ├── Eval::Runner # sequential or concurrent execution
35
35
  ├── Eval::Report # score, pass_rate, per-case results
36
- ├── Eval::AggregatedReport # merged reports across models or runs
36
+ ├── Eval::AggregatedReport # one candidate's repeated runs (runs: N), averaged
37
37
  ├── Eval::CaseResult # value object (name, passed?, output, expected, mismatches, cost)
38
38
  ├── Eval::ExpectationEvaluator # expected / expected_traits / evaluator proc
39
39
  ├── Eval::ModelComparison # compare_models result (table, best_for, candidate configs)
40
40
  ├── Eval::Recommender # model recommendation algorithm (candidates → optimal config)
41
41
  ├── Eval::Recommendation # recommendation result (best, retry_chain, savings, to_dsl)
42
- ├── Eval::RetryOptimizer # optimize_retry_policy result (per-eval breakdown, fallback list)
42
+ ├── Eval::RetryOptimizer # optimize_retry_policy; its Result has the per-eval breakdown, fallback list
43
43
  ├── Eval::BaselineDiff # save_baseline! + without_regressions comparison
44
44
  ├── Eval::PromptDiffComparator # compare_with prompt A/B diff
45
45
  └── Eval::EvalHistory # time-series view across saved reports
@@ -34,7 +34,7 @@ SummarizeArticle.define_eval("regression") do
34
34
  end
35
35
  ```
36
36
 
37
- "Partial match" means `expected` is treated as a subset of `parsed_output` — each key in `expected` must equal the corresponding key in the output, but extra keys in the output are ignored. So the `ruby release` case passes whenever the model produces `tone: "analytical"`, regardless of what shows up in `tldr` or `takeaways`. To assert on multiple keys, list them all in `expected`.
37
+ "Partial match" means `expected` is treated as a subset of `parsed_output` — each key in `expected` must equal the corresponding key in the output, but extra keys in the output are ignored. A Regexp in `expected` matches the value as a string, a Range must include it, and a key whose output value is `nil` counts as missing. So the `ruby release` case passes whenever the step's own schema and `validate` rules pass and the model produces `tone: "analytical"`, regardless of what shows up in `tldr` or `takeaways`. To assert on multiple keys, list them all in `expected`.
38
38
 
39
39
  Only after the eval exists, touch the prompt-shaping building blocks of the Step: `system`, `rule`, and `example` (blocks inside `prompt do`), `validate` (class-level invariant checks that run after the model returns), or the prompt version itself. See [Getting Started](getting_started.md) for the DSL surface.
40
40
 
@@ -88,7 +88,7 @@ diff = SummarizeArticleV2.compare_with(
88
88
  context: { retry_policy_override: nil } # run mini, not the step's retry chain
89
89
  )
90
90
 
91
- diff.safe_to_switch? # => true if no cases regressed
91
+ diff.safe_to_switch? # => true when both ran the same cases and none scored lower
92
92
  ```
93
93
 
94
94
  This is the cleanest eval-first move: same eval, same cases, two prompt versions, one answer.
@@ -152,7 +152,7 @@ SummarizeArticle.estimate_eval_cost("regression",
152
152
  # => { "gpt-4.1-nano" => 0.003207, "gpt-4.1-mini" => 0.012829, "gpt-4.1" => 0.064142 }
153
153
  ```
154
154
 
155
- Use it in CI to decide which models are worth running regression on, or to cap worst-case spend per build.
155
+ Use it in CI to decide which models are worth running regression on, or to budget a build. It is a floor, not a worst case: it counts one attempt per case, `max_output` (or 256) output tokens, and $0 for a model it cannot price.
156
156
 
157
157
  ## Team workflow
158
158
 
@@ -184,7 +184,8 @@ Don't optimize cost before you stabilize quality:
184
184
  ```ruby
185
185
  comparison = SummarizeArticle.compare_models(
186
186
  "regression",
187
- candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }, { model: "gpt-4.1" }]
187
+ candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }, { model: "gpt-4.1" }],
188
+ context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new } # live; without it a sample_response answers
188
189
  )
189
190
 
190
191
  comparison.best_for(min_score: 0.95)
@@ -202,7 +203,7 @@ comparison.best_for(min_score: 0.95)
202
203
 
203
204
  1. Write `define_eval` before touching the prompt.
204
205
  2. Treat `sample_response` as smoke only.
205
- 3. Use `run_eval("name", context: { model: "..." })` for real quality measurement.
206
+ 3. Use `run_eval("name", context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new, model: "..." })` for real quality measurement — a `model:` alone keeps the canned `sample_response`, and a Step with a retry chain needs `retry_policy_override: nil` too, or the chain picks the model.
206
207
  4. Use `compare_with` for every serious prompt change.
207
208
  5. Gate merges with `pass_eval`.
208
209
  6. Feed every production miss back into the dataset.
@@ -27,7 +27,7 @@ class SummarizeArticle < RubyLLM::Contract::Step::Base
27
27
  string :tone, enum: %w[neutral positive negative analytical]
28
28
  end
29
29
 
30
- # 3. Business rules — things JSON Schema cannot express
30
+ # 3. Business rules - checks this schema does not make
31
31
  validate("TL;DR fits the card") { |o, _| o[:tldr].length <= 200 }
32
32
  validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }
33
33
 
@@ -45,7 +45,7 @@ Each `validate` block receives `(parsed_output, input)` — the convention `|o,
45
45
 
46
46
  ## Validation and retry behavior
47
47
 
48
- When the cheap model returns output that fails a `validate` block (status `:validation_failed`) or can't be parsed as JSON (status `:parse_error`), retry falls back to the next model in `models:` and tries again. Transport errors (timeouts, 5xx) retry independently via Faraday, not via `retry_policy`. See [Optimizing retry_policy](optimizing_retry_policy.md) for tuning the chain.
48
+ When the cheap model returns output that fails a `validate` block (status `:validation_failed`) or can't be parsed as JSON (status `:parse_error`), retry falls back to the next model in `models:` and tries again. Transport errors (timeouts, 5xx) retry inside RubyLLM, and by default not via `retry_policy`; to retry them there too, name them in `retry_on` (see below). See [Optimizing retry_policy](optimizing_retry_policy.md) for tuning the chain.
49
49
 
50
50
  ```ruby
51
51
  result = SummarizeArticle.run(article_text)
@@ -76,7 +76,7 @@ retry_policy do
76
76
  end
77
77
  ```
78
78
 
79
- A class matches its subclasses; `:adapter_error` in the list retries every adapter error. As with statuses, the list replaces the defaults. Switching to a fallback model on transport errors inside one call is RubyLLM's `chat.with_fallbacks`; the contract's adapter does not call it, so fallbacks across models belong in `escalate`.
79
+ A class matches its subclasses; `:adapter_error` in the list retries every adapter error. A network failure (refused connection, timeout) is `RubyLLM::ConnectionError` on `ruby_llm` 2.2; 2.1 raised the Faraday error itself (`Faraday::ConnectionFailed`, `Faraday::TimeoutError`), so a `retry_on` naming a Faraday class stops matching after the upgrade. A provider or network error while `token_count :exact` counts the input is not an adapter error: the Step ends as `:limit_exceeded`, so a `retry_on` naming an error class does not retry it. As with statuses, the list replaces the defaults. Switching to a fallback model on transport errors inside one call is RubyLLM's `chat.with_fallbacks`; the contract's adapter does not call it, so fallbacks across models belong in `escalate`.
80
80
 
81
81
  ### Responses cut off by a token limit
82
82
 
@@ -152,7 +152,7 @@ SummarizeArticle.define_eval("regression") do
152
152
  end
153
153
  ```
154
154
 
155
- `expected:` is treated as a partial match against `parsed_output` — extra output keys are ignored, listed keys must equal. So this case passes whenever the model produces `tone: "analytical"`, regardless of `tldr` content. To assert on multiple keys, list them all in `expected`.
155
+ `expected:` is treated as a partial match against `parsed_output` — extra output keys are ignored, listed keys must equal (a Regexp or Range in `expected` matches instead). So the first case passes whenever the step's own schema and `validate` rules pass and the model produces `tone: "analytical"`, regardless of `tldr` content. To assert on multiple keys, list them all in `expected`.
156
156
 
157
157
  Gate CI on score and cost thresholds:
158
158
 
@@ -173,11 +173,11 @@ report.save_baseline!
173
173
  expect(SummarizeArticle).to pass_eval("regression").without_regressions
174
174
  ```
175
175
 
176
- `without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality. A previously-passing case that was skipped this run (no adapter configured) also fails the gate, listed as `SKIPPED` rather than as a regression.
176
+ `pass_eval` always requires the eval to pass (every case that ran, or the score set by `with_minimum_score`; a case skipped for want of an adapter does not count, but at least one case must run). `without_regressions` adds a check against the saved baseline: it also fails the build if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality — or was removed from the eval. A previously-passing case that was skipped this run (no adapter configured) also fails the gate, listed as `SKIPPED` rather than as a regression.
177
177
 
178
178
  ## Budget caps
179
179
 
180
- `max_input`, `max_output`, and `max_cost` are preflight checks — the LLM is never called if an estimate exceeds the limit. Zero tokens spent, zero cost.
180
+ `max_input` and `max_cost` are preflight checks: an attempt whose estimate exceeds the limit is never sent, so it generates nothing and costs nothing. `max_output` caps the reply at the provider and is the output side of the `max_cost` estimate. With `token_count :exact` the provider counts the input before the check, and in a retry chain a refused attempt does not undo the attempts before it.
181
181
 
182
182
  ```ruby
183
183
  result = SummarizeArticle.run(forty_page_report) # ~128k characters
@@ -201,7 +201,7 @@ max_cost 0.01, on_unknown_pricing: :warn
201
201
 
202
202
  Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
203
203
 
204
- The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. Since 1.1.2 a real call whose provider sent no token counts is flagged too, instead of passing as a free zero-token run.
204
+ The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total leaves out what it cannot price (a case with no priced attempt counts as $0), so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced costs and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. Since 1.1.2 a real call whose provider sent no token counts is flagged too, instead of passing as a free zero-token run.
205
205
 
206
206
  ### What a trace's usage and cost count
207
207
 
@@ -230,7 +230,7 @@ class SummarizeArticle < RubyLLM::Contract::Step::Base
230
230
  end
231
231
  ```
232
232
 
233
- The count comes from RubyLLM's `chat.count_tokens` (OpenAI Responses, Anthropic, Gemini, Vertex AI, Bedrock Converse) and covers the prompt and attachments, so `attachment_token_estimate` is not needed. What else a provider counts is its own: OpenAI's and Anthropic's counts include the output schema, Gemini's leaves it out. It costs one extra request per attempt, and RubyLLM leaves `provider_options` out of it. Where no count can be had - a provider without the endpoint (Ollama, for one), a failed request, or an adapter without `count_tokens` - the call is refused as `:limit_exceeded` with the reason, never measured with the heuristic under the name "exact". The `Test` adapter counts with the heuristic, since it has no provider to ask. The output side of `max_cost` is still an estimate (`max_output`, or the input size without one).
233
+ The count comes from RubyLLM's `chat.count_tokens` (OpenAI Responses, Anthropic, Gemini, Vertex AI, Bedrock Converse) and covers the prompt and attachments, so `attachment_token_estimate` is not needed. What else a provider counts is its own: OpenAI's and Anthropic's counts include the output schema, Gemini's leaves it out. It costs one extra request per attempt, and RubyLLM leaves `provider_options` out of it. Where no count can be had - a provider without the endpoint (Ollama, for one), a request that fails with a RubyLLM or Faraday error, or an adapter without `count_tokens` - the call is refused as `:limit_exceeded` with the reason, never measured with the heuristic under the name "exact". The `Test` adapter counts with the heuristic, since it has no provider to ask. The output side of `max_cost` is still an estimate (`max_output`, or the input size without one).
234
234
 
235
235
  ### Local and custom models
236
236
 
@@ -262,7 +262,7 @@ SummarizeArticle.estimate_eval_cost("regression",
262
262
  # => { "gpt-4.1-nano" => 0.003207, "gpt-4.1-mini" => 0.012829, "gpt-4.1" => 0.064142 }
263
263
  ```
264
264
 
265
- `estimate_cost` returns `nil` when pricing isn't registered. `estimate_eval_cost` silently treats unknown-pricing cases as `$0.00` and sums the rest — it does **not** fail closed the way `max_cost` does. Treat its output as a floor, not a guarantee; register pricing via `CostCalculator.register_model` before relying on it for budget decisions.
265
+ `estimate_cost` returns `nil` when it does not know the model, and a hash with `estimated_cost: nil` when it knows the model but not its price. `estimate_eval_cost` silently treats unknown-pricing cases as `$0.00` and sums the rest — it does **not** fail closed the way `max_cost` does. Treat its output as a floor, not a guarantee; register pricing via `CostCalculator.register_model` before relying on it for budget decisions.
266
266
 
267
267
  ## Provider options
268
268
 
@@ -71,7 +71,7 @@ class AccuracyJudge < RubyLLM::Contract::Step::Base
71
71
  end
72
72
  ```
73
73
 
74
- The `retry_policy` escalates to `gpt-5-mini` if the cheaper `gpt-5-nano` fails schema/validate (e.g., returns malformed JSON) — the judge stays reliable without paying for the bigger model on every call. `max_cost 0.01` bounds each attempt (with `max_output 2_000`, up to roughly 24k input tokens on `gpt-5-mini`), so a case that falls back from nano to mini costs at most about $0.012: the judge runs N× per eval suite (one call per case), so capping per-call cost prevents calibration runs from blowing the budget.
74
+ The `retry_policy` escalates to `gpt-5-mini` if the cheaper `gpt-5-nano` fails schema/validate (e.g., returns malformed JSON) — the judge stays reliable without paying for the bigger model on every call. `max_cost 0.01` bounds each attempt's estimated cost (with `max_output 2_000`, up to roughly 24k input tokens on `gpt-5-mini`; the input side is a heuristic estimate, not the billed amount), so a case that falls back from nano to mini is estimated at no more than about $0.012: the judge runs N× per eval suite (one call per case), so capping per-call cost prevents calibration runs from blowing the budget.
75
75
 
76
76
  This is small on purpose: one boolean, one reason. Easy to read, easy to disagree with.
77
77
 
@@ -147,9 +147,9 @@ Only now, after the judge has earned trust on real data, do you use it as the or
147
147
  - `output` — `parsed_output` of the Step under test. Here `output[:tldr]` is the summary `SummarizeArticle` produced (the same field declared in its `output_schema`).
148
148
  - `input` — whatever you passed to `add_case input:`. Here it's the original article string.
149
149
 
150
- The lambda must return a `Float` in `[0.0, 1.0]` — the eval framework averages per-case scores into the suite score. Map `true → 1.0`, `false → 0.0`, or use weighted partial scores if your judge returns gradations.
150
+ Return a `Float` in `[0.0, 1.0]` — the eval framework averages per-case scores into the suite score, clamping a number outside that range, and a number of 0.5 or more passes the case. `true` and `false` count as 1.0 and 0.0, and an `EvaluationResult` is used as returned; use weighted partial scores if your judge returns gradations.
151
151
 
152
- > ⚠️ **Heads-up before you copy the snippet below.** Smoke evals on `SummarizeArticle` (per [Getting Started](getting_started.md)) typically run with `RubyLLM::Contract::Adapters::Test` for determinism — canned summaries, no API calls. **The `AccuracyJudge.run(...)` inside the evaluator does NOT inherit that Test adapter.** Each Step picks its adapter independently (from `context: { adapter: }`, or the `default_adapter` set in `RubyLLM::Contract.configure`). Run accuracy-regression evals live (an explicit `context:` or a configured default adapter), and rely on `max_cost` on the judge to bound the bill.
152
+ > ⚠️ **Heads-up before you copy the snippet below.** Smoke evals on `SummarizeArticle` (per [Getting Started](getting_started.md)) typically run with `RubyLLM::Contract::Adapters::Test` for determinism — canned summaries, no API calls. **The `AccuracyJudge.run(...)` inside the evaluator does NOT inherit that Test adapter.** Each Step picks its adapter independently (from `context: { adapter: }`, or the `default_adapter` set in `RubyLLM::Contract.configure`). Run accuracy-regression evals live (an explicit `context:` or a configured default adapter), and rely on `max_cost` on the judge to bound each call's estimated cost.
153
153
  >
154
154
  > This is intentional: the judge IS the oracle, so faking its verdicts would defeat the purpose. Keep the Test-adapter smoke evals for schema/wiring checks only.
155
155
 
@@ -49,7 +49,7 @@ end
49
49
  ```ruby
50
50
  # Before
51
51
  parsed = ArticleSummaryService.new.call(article_text)
52
- Article.update!(summary: parsed["tldr"])
52
+ Article.update!(summary: parsed[:tldr])
53
53
 
54
54
  # After
55
55
  result = SummarizeArticle.run(article_text)
@@ -2,7 +2,7 @@
2
2
 
3
3
  > Read this when your contract needs to send a PDF, image, or audio file to the LLM — not just text.
4
4
 
5
- `ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy escalate(...)`, and trace observability still apply. The gem does **not** ship its own multimodal API — it forwards `with:` to `RubyLLM::Chat#ask`, which RubyLLM normalises per provider (Anthropic, OpenAI, Gemini).
5
+ `ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy { escalate ... }`, and trace observability still apply. The gem does **not** ship its own multimodal API — it forwards `with:` to `RubyLLM::Chat#ask`, which RubyLLM normalises per provider (Anthropic, OpenAI, Gemini).
6
6
 
7
7
  ## Minimal example
8
8
 
@@ -48,7 +48,7 @@ result.trace[:cost] # => 0.0042 (total across attempts)
48
48
 
49
49
  1. **Input vs attachment.** The `input` argument to `Step.run` is the text prompt. The attachment travels via `context: { attachment: ... }` — opaque to the contract layer, forwarded to the adapter.
50
50
  2. **Adapter forwards `with:`.** `RubyLLM::Contract::Adapters::RubyLLM` reads `options[:attachment]` and passes it to `chat.ask(content, with: attachment)`. RubyLLM picks the right wire format per provider.
51
- 3. **Multi-attachment supported.** `with: [pdf1, pdf2]` or `with: { images: [...], pdfs: [...] }` works natively (`RubyLLM::Content#process_attachments`).
51
+ 3. **Multi-attachment supported.** `with: [pdf1, pdf2]` or `with: { images: [...], pdfs: [...] }` works natively; RubyLLM groups the attachments itself.
52
52
  4. **`with: nil` is a no-op.** Text-only contracts unaffected — the kwarg defaults to nil.
53
53
 
54
54
  ## Cost: `attachment_token_estimate` is required
@@ -70,7 +70,7 @@ The same estimate applies to:
70
70
 
71
71
  ### Fail-closed without estimate
72
72
 
73
- If your contract has `max_cost` or `max_input` set, receives an attachment, and `attachment_token_estimate` is **not declared**, the call fails with `:limit_exceeded` — the gem refuses to spend money on cost it cannot bound.
73
+ If your contract has `max_cost` or `max_input` set, receives an attachment, and `attachment_token_estimate` is **not declared**, the call fails with `:limit_exceeded` — the gem refuses to spend money on cost it cannot bound. This applies to the default heuristic count; with `token_count :exact` the provider counts the attachment instead, and the call fails closed only when it cannot.
74
74
 
75
75
  ```ruby
76
76
  class MyContract < RubyLLM::Contract::Step::Base
@@ -103,6 +103,7 @@ end
103
103
  ```ruby
104
104
  ExtractInvoiceData.estimate_cost(
105
105
  input: "Look for vendor, amount...",
106
+ model: "gpt-4.1-mini", # estimate_cost does not read the retry chain
106
107
  attachment: "tmp/invoice.pdf"
107
108
  )
108
109
  # => { model: "gpt-4.1-mini",
@@ -138,9 +139,9 @@ A Step is single-turn: each `run` sends its prompt with the attachment and keeps
138
139
 
139
140
  - **OpenAI** — PDFs sent as `type: 'file'` with `file_data` (base64). Images as `image_url`. Audio as `input_audio`. Vision pricing varies by image detail; check the model card.
140
141
  - **Anthropic** — PDFs sent as `type: 'document'` with `source.type: 'base64'` or `'url'` (auto-selected). Images same. Page limit ~100 per call.
141
- - **Gemini** — Everything via `inline_data` with `mime_type`. Multimodal token counting is unified.
142
+ - **Gemini** — Media sent inline via `inline_data` with `mime_type`; a file already uploaded to the provider goes as `file_data`. Multimodal token counting is unified.
142
143
 
143
- RubyLLM dispatches on `attachment.type` (`:image`, `:pdf`, `:audio`, `:text`, `:unknown`). Tempfiles must have the right extension (`.pdf`, `.png`, etc.) — RubyLLM detects MIME from the filename; an unsuffixed tempfile becomes `:unknown` and is rejected by the provider.
144
+ RubyLLM dispatches on `attachment.type` (`:image`, `:pdf`, `:audio`, `:video`, `:text`, `:document`, `:unknown`). Give tempfiles the right extension (`.pdf`, `.png`, etc.): RubyLLM detects the MIME type from the filename or the content, and a file it cannot place becomes `:unknown`, which the provider rejects.
144
145
 
145
146
  ## Testing contracts with attachments
146
147
 
@@ -152,7 +153,7 @@ RSpec.describe ExtractInvoiceData do
152
153
  expect_any_instance_of(RubyLLM::Chat).to receive(:ask)
153
154
  .with(anything, with: "fixtures/invoice.pdf")
154
155
  .and_return(RubyLLM::Message.new(role: :assistant, content: '{"vendor":"X","invoice_number":"1","total_amount":10,"currency":"EUR"}',
155
- model_id: "gpt-4.1-mini",
156
+ model: "gpt-4.1-mini",
156
157
  input_tokens: 200, output_tokens: 50))
157
158
 
158
159
  result = described_class.run("extract", context: { attachment: "fixtures/invoice.pdf" })
@@ -6,7 +6,7 @@ You defined `SummarizeArticle` in [Getting Started](getting_started.md) with `re
6
6
 
7
7
  ## Requirements
8
8
 
9
- - **`SummarizeArticle` already has `retry_policy`.** If your step has none, add one first ([getting started](getting_started.md)).
9
+ - **A `retry_policy` is optional.** The optimizer runs each candidate without the Step's own chain, and the DSL it prints is the policy to add.
10
10
  - **2–3 evals per step.** One eval optimizes for one scenario; with only `smoke`, you get a recommendation that passes smoke but may miss production edge cases. See [eval-first](eval_first.md).
11
11
  - **Rake task `ruby_llm_contract:optimize` is registered when `lib/ruby_llm/contract/rake_task.rb` is loaded.** In Rails apps the railtie auto-loads it lazily during `rake` invocation — no manual setup needed. In non-Rails projects, add `require "ruby_llm/contract/rake_task"` to your `Rakefile` (or any file under `lib/tasks/`) and set `EVAL_DIRS=...` on the command line.
12
12
 
@@ -32,9 +32,9 @@ rake ruby_llm_contract:optimize \
32
32
 
33
33
  The `@low` (or `@medium`, `@high`) suffix on a candidate is CLI shorthand for `{ model: "gpt-5-mini", reasoning_effort: "low" }` — saves you constructing hash strings on the command line. Plain model names without `@` use the Step's default thinking config (or none). `@default` runs the candidate with no effort at all, which is what a model without reasoning needs when the Step sets one: `CANDIDATES=gpt-6-luna@none,gpt-4.1-mini@default`.
34
34
 
35
- List candidates from the cheapest to the most expensive. The optimizer takes that order as the cost order: the suggested chain tries them in it, and the first candidate that passes every eval is the safe fallback. It does not sort by price.
35
+ List candidates from the cheapest to the most expensive. The optimizer takes that order as the cost order: the suggested chain tries them in it, and the first candidate that passes every eval is the safe fallback. It does not sort by price. A candidate passes an eval when it scores at least `MIN_SCORE`, 0.95 by default; set `MIN_SCORE=1` to require full scores.
36
36
 
37
- Offline uses each eval's `sample_response` — zero API calls. **Every candidate gets the same score** because they all receive the canned response. That's fine for a smoke test (verifying evals load, candidates parse, output renders) but it doesn't compare model quality. For real optimization, go live.
37
+ Offline, each eval with a `sample_response` uses it — zero API calls. An eval without one runs on the default adapter, which `RubyLLM::Contract.configure` sets up, so it calls the provider; give every eval a `sample_response` for a fully offline run. **When every eval has one, every candidate receives the same canned response.** That's fine for a smoke test (verifying evals load, candidates parse, output renders) but it doesn't compare model quality. For real optimization, go live.
38
38
 
39
39
  ## Optimize against real models
40
40
 
@@ -75,23 +75,24 @@ SummarizeArticle — fallback list optimization
75
75
  ```
76
76
 
77
77
  Reading the table:
78
- - **`←` marks scores below threshold in the hardest eval.** Not a selection hint — just "this candidate fails the row that matters most".
78
+ - **`←` marks scores below 1.0 in the hardest eval.** Not a selection hint — just "this candidate fails the row that matters most".
79
79
  - **Hardest eval** = the one that forces the strong fallback. Here, `critical_tone` demands `gpt-5-mini` at its default effort.
80
- - **Suggested fallback list** = the shortest chain where each step covers more evals, built greedy-cheapest-first. Stops when all evals pass. Order matters: `gpt-5-nano` is tried first; on validation failure, the gem falls back to `gpt-5-mini` at low effort, then at its default.
80
+ - **Suggested fallback list** = a greedy chain, not a searched-for shortest one: walking the candidates cheapest first, it keeps each one that passes an eval no earlier candidate passed, and ends on the first candidate that passes every eval. Order matters: `gpt-5-nano` is tried first; on validation failure, the gem falls back to `gpt-5-mini` at low effort, then at its default. Retry fires on `:validation_failed` and `:parse_error`, not on a low eval score, so a cheaper candidate's eval failures must be ones your `validate` rules catch, or the fallback never runs for them.
81
81
 
82
82
  Copy the DSL, paste into your step, verify with `rake ruby_llm_contract:eval` (defined by `RubyLLM::Contract::RakeTask.new`, see [Rails integration](rails_integration.md#7-ci-gate--block-regressions-before-merge)). You just dropped `gpt-5` from the chain — most requests finish on nano, mini handles what nano misses, and the strong model was never needed.
83
83
 
84
84
  ## Measure effective cost before shipping
85
85
 
86
- `optimize` shows **first-attempt** cost. In production, a candidate whose validator rejects 20% of outputs actually costs `first_try_cost + fallback_cost × 0.20` per successful output. The first-attempt number hides this.
86
+ `optimize` shows scores, not cost, and `compare_models` on its own reports what each candidate's run cost with no fallback (its same-model retries included). In production, a candidate whose validator rejects 20% of outputs actually costs about `first_try_cost + fallback_cost × 0.20` per request. A run without the fallback hides this.
87
87
 
88
- `production_mode:` is an optional `compare_models` kwarg. With `{ fallback: "model" }` it runs each candidate with a runtime `[candidate, fallback]` chain and reports **effective cost** (first-try + fallback cost weighted by rejection rate). Without it, you get only first-attempt metrics (the default). A fallback without a `reasoning_effort` runs with the candidate's, so a fallback naming the candidate's model sends the same request and is skipped; `{ model: "...", reasoning_effort: :default }` falls back with no effort.
88
+ `production_mode:` is an optional `compare_models` kwarg. With `{ fallback: "model" }` it runs each candidate with a runtime `[candidate, fallback]` chain and reports **effective cost**: the eval's total, every first attempt plus every fallback that ran. Without it, you get only first-attempt metrics (the default). A fallback without a `reasoning_effort` runs with the candidate's, so a fallback naming the candidate's model sends the same request and is skipped; `{ model: "...", reasoning_effort: :default }` falls back with no effort.
89
89
 
90
90
  ```ruby
91
91
  SummarizeArticle.compare_models(
92
92
  "dense_article",
93
93
  candidates: [{ model: "gpt-5-nano" }, { model: "gpt-5-mini", reasoning_effort: "low" }, { model: "gpt-5" }],
94
- production_mode: { fallback: "gpt-5" }
94
+ production_mode: { fallback: "gpt-5" },
95
+ context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new } # live; without it a sample_response answers
95
96
  ).print_summary
96
97
  ```
97
98
 
@@ -110,9 +111,9 @@ dense_article — model comparison
110
111
  Cheapest at 100%: gpt-5-mini (effort: low)
111
112
  ```
112
113
 
113
- - **first-attempt** — cost of the first run alone.
114
- - **fallback %** — fraction of cases where the validator rejected and the fallback ran.
115
- - **effective cost** — total per successful output including retries.
114
+ - **first-attempt** — cost of the first attempts alone, summed over the eval's cases.
115
+ - **fallback %** - fraction of cases that needed more than one attempt, so the fallback ran.
116
+ - **effective cost** — total over the eval's cases, fallbacks included.
116
117
  - **`—`** — the fallback would send the candidate's own request, so the candidate runs alone and there is no chain to observe.
117
118
 
118
119
  Run this before finalizing: a candidate saving 3× on first-attempt but falling back 60% of the time may save only 1.2× in production.
@@ -123,7 +124,7 @@ Run this before finalizing: a candidate saving 3× on first-attempt but falling
123
124
 
124
125
  - **"No viable chain" from a single live run.** Re-run with `RUNS=3`. If scores jump, the first run was noise. Never trust single-run results with gpt-5 / o-series in the pool — `temperature=1.0` is server-enforced.
125
126
  - **Every candidate fails the same eval**, including the strongest. The eval is rejecting correct answers. Run the step directly (`context: { retry_policy_override: nil, model: "gpt-4.1" }`), inspect the output, compare with the `verify` block. Loosen the eval if the output is correct but not one of the accepted values.
126
- - **Testing one specific hypothesis.** (e.g. "does `mini@medium` help on `critical_tone`?") Use `SummarizeArticle.compare_models("critical_tone", candidates: [{ model: "gpt-5-mini", reasoning_effort: "medium" }], runs: 3)` directly — three calls instead of rerunning the whole optimize pass.
127
+ - **Testing one specific hypothesis.** (e.g. "does `mini@medium` help on `critical_tone`?") Use `SummarizeArticle.compare_models("critical_tone", candidates: [{ model: "gpt-5-mini", reasoning_effort: "medium" }], runs: 3, context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new })` directly — three calls instead of rerunning the whole optimize pass.
127
128
 
128
129
  ## Programmatic API names
129
130
 
@@ -131,7 +132,7 @@ Metrics exposed on `Report` / `AggregatedReport` keep their original names: `sin
131
132
 
132
133
  ## thinking DSL note
133
134
 
134
- Set the default reasoning effort once on the Step class — mirrors `RubyLLM::Agent.thinking` exactly:
135
+ Set the default reasoning effort once on the Step class, with the `effort:` and `budget:` of `RubyLLM::Agent.thinking` (not its other options):
135
136
 
136
137
  ```ruby
137
138
  class SummarizeArticle < RubyLLM::Contract::Step::Base
@@ -153,4 +154,4 @@ SummarizeArticle.compare_models("regression", candidates: [{ model: "gpt-6-luna"
153
154
  retry_policy { escalate({ model: "gpt-6-luna" }, { model: "gpt-4.1-mini", reasoning_effort: :default }) }
154
155
  ```
155
156
 
156
- `nil` is not `:default`: a nil effort is ignored and the class one is sent. The gem does not drop an effort by itself based on the model. A chain element without an effort runs with the call's (or the class's), and `trace.attempts[n][:config]` shows the effort each attempt was sent.
157
+ `nil` is not `:default`: a nil effort is ignored and the class one is sent. The gem does not drop an effort by itself based on the model. A chain element without an effort runs with the call's (or the class's), and `trace.attempts[n][:config]` shows the effort an attempt was sent; it is absent when the attempt was sent no effort.
@@ -42,7 +42,7 @@ class GenerateHashtags < RubyLLM::Contract::Step::Base
42
42
 
43
43
  prompt do
44
44
  rule "Preserve tldr / takeaways / tone exactly as given."
45
- user "Article summary: {tldr}\nTone: {tone}\nGenerate 2 to 5 concise hashtags."
45
+ user "Article summary: {tldr}\nTakeaways: {takeaways}\nTone: {tone}\nGenerate 2 to 5 concise hashtags."
46
46
  end
47
47
 
48
48
  validate("tone preserved") { |o, input| o[:tone] == input[:tone] }
@@ -101,13 +101,14 @@ adapter = RubyLLM::Contract::Adapters::Test.new(response: {
101
101
 
102
102
  result = ArticleCardPipeline.run("article text", context: { adapter: adapter })
103
103
  result.failed? # => true
104
- result.failed_step # => :summarize (validate rejected; retries exhausted)
104
+ result.failed_step # => :summarize (validate rejected; no retry_policy, so one attempt)
105
105
  # tag and card never run — no downstream tokens spent on garbage
106
106
  ```
107
107
 
108
108
  ## Per-step model override
109
109
 
110
110
  ```ruby
111
+ # In the pipeline's definition: `step` appends, so reopening the class would add three more steps.
111
112
  class ArticleCardPipeline < RubyLLM::Contract::Pipeline::Base
112
113
  step SummarizeArticle, as: :summarize, model: "gpt-4.1-mini"
113
114
  step GenerateHashtags, as: :tag, model: "gpt-4.1-nano"
@@ -123,13 +124,14 @@ result = ArticleCardPipeline.run(article_text, timeout_ms: 30_000)
123
124
 
124
125
  ## Pipeline eval
125
126
 
126
- Pipeline `run_eval` matches only the **final step's output** against `expected:` (`BuildArticleCard` here). Intermediate step outputs (`SummarizeArticle`, `GenerateHashtags`) are not routed through `expected:` — assertions on `:tone` or `:hashtags` here would silently never match. For per-step assertions, run evals on each step individually.
127
+ Pipeline `run_eval` matches only the **final step's output** against `expected:` (`BuildArticleCard` here). Intermediate step outputs (`SummarizeArticle`, `GenerateHashtags`) are not routed through `expected:` — an assertion on `:tone` there would silently never match, and one on `:hashtags` would check the card's hashtags, not `GenerateHashtags`'. For per-step assertions, use `step_expectations:`, keyed by the step's alias, or run evals on each step individually.
127
128
 
128
129
  ```ruby
129
130
  ArticleCardPipeline.define_eval("e2e") do
130
131
  add_case "ruby 3.4 release",
131
132
  input: "Ruby 3.4 ships with frozen string literals by default and better YJIT...",
132
- expected: { sentiment_icon: "🧠" }
133
+ expected: { sentiment_icon: "🧠" },
134
+ step_expectations: { summarize: { tone: "analytical" } }
133
135
  end
134
136
 
135
137
  report = ArticleCardPipeline.run_eval("e2e", context: { model: "gpt-4.1-mini" })
@@ -23,7 +23,7 @@ Or just a plain string (wraps as a single user message):
23
23
  prompt "Summarize this article for a UI card. {input}"
24
24
  ```
25
25
 
26
- The AST is immutable, diffable, and hashable. Useful for snapshot testing and auditing prompt changes.
26
+ The AST is frozen, diffable, and compares by value with `==`. Useful for snapshot testing and auditing prompt changes.
27
27
 
28
28
  ## Hash inputs with variable interpolation
29
29
 
@@ -54,7 +54,7 @@ class SummarizeArticle < RubyLLM::Contract::Step::Base
54
54
  end
55
55
  ```
56
56
 
57
- Every `{key}` in a prompt node is pulled from the input hash at run time. A key the input does not have is left in the prompt as literal `{key}` text, so declare the keys in a `Types::Hash.schema(...)` input type when a missing one should stop the call.
57
+ Every `{key}` in a node's text (a `section`'s body, not its name) is pulled from the input hash at run time. A key the input does not have is left in the prompt as literal `{key}` text, so declare the keys in a `Types::Hash.schema(...)` input type when a missing one should stop the call.
58
58
 
59
59
  > **Not the same as `RubyLLM::Agent.inputs`.** `Step.input_type` is a *runtime type check* on the positional argument passed to `run(input)` — a violating input returns `status: :input_error` instead of calling the model. `RubyLLM::Agent.inputs` is a list of *named template locals* injected into ERB instructions. They solve different problems (validation vs templating) and can coexist on the same project.
60
60
 
@@ -131,7 +131,7 @@ RubyLLM 2.1 emits its own events and OpenTelemetry spans for every call. To grou
131
131
  RubyLLM::Contract.configure { |c| c.workflow_instrumentation = true }
132
132
  ```
133
133
 
134
- A pipeline then runs as one `RubyLLM.workflow` named after its class, with each step as `workflow.step(alias)`; a step run on its own is a workflow named after its class. RubyLLM's events carry `workflow_name` and `workflow_step_name`, and a workflow your code opened around the call becomes the parent. Off by default. In a Rails app RubyLLM also records each call in its usage ledger when configured; that figure and `trace[:cost]` are the same RubyLLM price, unless `CostCalculator.register_model` overrides it for the model.
134
+ A pipeline then runs as one `RubyLLM.workflow` named after its class, with each step as `workflow.step(alias)`; a step run on its own is a workflow named after its class. RubyLLM's events carry `workflow_name` and `workflow_step_name`, and a workflow your code opened around the call becomes the parent. Off by default. In a Rails app RubyLLM also records each call in its usage ledger when configured; for a single attempt that figure and `trace[:cost]` (rounded to six decimals) are the same RubyLLM price, unless `CostCalculator.register_model` sets a price for the model and the provider did not report what it billed. With a retry chain each attempt is its own RubyLLM call, while `trace[:cost]` sums the attempts it could price (`trace[:cost_complete]` says whether it priced them all).
135
135
 
136
136
  ## 5. Testing — RSpec and Minitest
137
137
 
@@ -191,7 +191,7 @@ class ArticlesController < ApplicationController
191
191
  end
192
192
  ```
193
193
 
194
- When `retry_policy` exhausts and all models fail, `result.failed?` is true but `result.parsed_output` still contains the last attempt's output — useful for logging what the model *did* return before the validate rejected it.
194
+ When `retry_policy` exhausts and all models fail, `result.failed?` is true but `result.parsed_output` still contains the last attempt's output when it parsed — useful for logging what the model *did* return before the validate rejected it. After a parse error it is `nil`; `result.raw_output` keeps the text.
195
195
 
196
196
  ## 7. CI gate — block regressions before merge
197
197
 
@@ -217,7 +217,7 @@ Then wire it in GitHub Actions:
217
217
  run: bundle exec rake ruby_llm_contract:eval
218
218
  ```
219
219
 
220
- The job fails when a previously-passing eval case now fails, when the average score drops below the threshold, or when total cost exceeds the cap. That is the signal that blocks a prompt regression or an accidental model upgrade from shipping.
220
+ The job fails when a previously-passing eval case now fails, when an eval's average score drops below `minimum_score`, or when total cost exceeds the cap. The first needs a baseline: with `save_baseline = false` the task creates none, so save one (`report.save_baseline!`, or a run with `save_baseline = true`) and commit it. That is the signal that blocks a prompt regression or an accidental model upgrade from shipping.
221
221
 
222
222
  **Two practical notes:**
223
223
 
@@ -9,13 +9,13 @@
9
9
  | What you write | Where it lives |
10
10
  |---|---|
11
11
  | `model`, `temperature`, `schema`, `instructions`, `thinking` | covered by both — same idea, different DSL surface |
12
- | `tools` | only in `RubyLLM::Agent` — a Step makes a single call without tools |
12
+ | `tools` | only in `RubyLLM::Agent` - each Step attempt is one call without tools |
13
13
  | `validate :rule do ... end` business invariants on output | only in `ruby_llm-contract` |
14
- | `retry_policy escalate(...)` model escalation on validation failure | only here (different from RubyLLM's network-level retry) |
15
- | `max_cost` / `max_input` / `max_output` pre-flight refusal | only here |
14
+ | `retry_policy { escalate ... }` model escalation on validation failure | only here (different from RubyLLM's network-level retry) |
15
+ | `max_cost` / `max_input` pre-flight refusal, `max_output` reply cap | only here |
16
16
  | `define_eval` + baseline regression + `compare_models` + `optimize_retry_policy` | only here; RubyLLM 2.1's `RubyLLM::Evaluation` grades answers (see below) but keeps no baseline and has no cost gate |
17
17
  | Pipeline composition with `step SomeStep, as: :alias` | only here; `RubyLLM.workflow` (2.1) groups calls for tracing, without typed step outputs, fail-fast or budgets |
18
- | `around_call`, named `observe` hooks with pass/fail recorded in trace | only here |
18
+ | `around_call`, named `observe` hooks with pass/fail recorded in `result.observations` | only here |
19
19
 
20
20
  ## Runtime relationship
21
21
 
@@ -50,7 +50,7 @@ A common pattern: simple ad-hoc prompts as `Agent`, contracts on the LLM feature
50
50
 
51
51
  ## On retry strategies
52
52
 
53
- The retry-strategy framing in this gem favours `retry_policy escalate(model_2, ...)` (model escalation, addresses model bias) over same-model `retry_policy attempts: N` (variance retry).
53
+ The retry-strategy framing in this gem favours `retry_policy { escalate model_1, model_2 }` (model escalation, addresses model bias) over same-model `retry_policy attempts: N` (variance retry).
54
54
 
55
55
  This is grounded in empirical comparison across PDF quiz generation, GSM8K math (n=30 + n=120), and multi-constraint schedule generation: same-model retry produced no useful lift for nano-class models on tasks with clear correctness criteria. Model escalation did move the needle when same-model retry could not.
56
56
 
@@ -12,7 +12,7 @@ Both gems sit on top of `ruby_llm`. The space they cover overlaps in vocabulary
12
12
  | When it runs | After the LLM call returns, typically in a spec | Before the LLM result reaches your code |
13
13
  | Where the output lives at evaluation time | Already in your variable, returned to caller | Still inside the gem's runner, not yet released |
14
14
  | What "fail" means | Red test in CI | Trigger retry/escalate on a stronger model, or fail closed |
15
- | Strongest features | Rich LLM-as-judge (faithful, relevant, hallucination, refusal, bias, toxicity, jailbreak, PII), red-team adversarial prompts, deterministic helpers (`assert_contains`, `assert_levenshtein`, …), RSpec/Minitest matchers, HTML/JUnit/GH reporters | Schema DSL with constraints, `validate` business rules, `retry_policy escalate(...)` model escalation, `max_cost` pre-flight refusal, regression-eval framework (frozen dataset + baseline + min_score gate), pipeline composition |
15
+ | Strongest features | Rich LLM-as-judge (faithful, relevant, hallucination, refusal, bias, toxicity, jailbreak, PII), red-team adversarial prompts, deterministic helpers (`assert_contains`, `assert_levenshtein`, …), RSpec/Minitest matchers, HTML/JUnit/GH reporters | Schema DSL with constraints, `validate` business rules, `retry_policy { escalate ... }` model escalation, `max_cost` pre-flight refusal, regression-eval framework (frozen dataset + baseline + min_score gate), pipeline composition |
16
16
  | What it does NOT cover | No retry, no model escalation, no pre-flight cost cap, no contract layer between LLM and your code | No 10-judge LLM-as-judge catalog, no red-team generation, no rich deterministic assertion library; its test-framework matchers (`satisfy_contract`, `pass_eval`) gate contracts and evals, not individual outputs |
17
17
 
18
18
  ## Visual: where each gem sits in your call
@@ -41,7 +41,7 @@ your code ──► Step.run ──► LLM ──► [schema + validate]
41
41
  status: :validation_failed
42
42
  ```
43
43
 
44
- The output **never reaches your code** until the contract passes. A failed validation either fixes itself (the retry policy escalates to a stronger model) or fails closed with `status: :validation_failed` — your call site sees a deterministic failure status, never schema-invalid data.
44
+ The output reaches your code **as a passing result** only once the contract passes. A failed validation either fixes itself (a retry policy escalates to a stronger model) or the call fails closed with the last attempt's status, such as `:validation_failed` — your call site sees a deterministic failure status. Check `result.ok?` before using the output: a failed result still carries it in `parsed_output` and `raw_output`, for logging.
45
45
 
46
46
  ### Both together (Tribunal in CI → then Contract in prod)
47
47
 
@@ -67,7 +67,7 @@ Tribunal grades **a fixed set of cases on every PR** to catch quality regression
67
67
 
68
68
  ## When to use which
69
69
 
70
- **Just Contract.** You ship LLM features whose output drives downstream code, money, or user trust. You need the bad-output-doesn't-reach-prod guarantee, retry escalation, and budget refusal. You are happy to write your own `validate` blocks for content checks; you don't need a 10-judge catalog yet.
70
+ **Just Contract.** You ship LLM features whose output drives downstream code, money, or user trust. You need a failure status instead of bad output (your code checks `result.ok?`), retry escalation, and budget refusal. You are happy to write your own `validate` blocks for content checks; you don't need a 10-judge catalog yet.
71
71
 
72
72
  **Just Tribunal.** You have a stable production path you don't want to wrap, but you want a CI safety net that grades LLM output for faithfulness, hallucination, PII, jailbreak resistance, etc. You're testing a RAG pipeline or chatbot whose runtime is owned by other code.
73
73
 
@@ -111,6 +111,14 @@ These work today without any code changes in either gem — both use plain Ruby
111
111
  class ChatReply < RubyLLM::Contract::Step::Base
112
112
  prompt "Answer this question grounded in the docs:\n{input}"
113
113
 
114
+ output_schema do
115
+ string :answer
116
+ end
117
+
118
+ retry_policy do
119
+ escalate "gpt-4.1-mini", "gpt-4.1"
120
+ end
121
+
114
122
  validate("no PII in answer") do |output, _input|
115
123
  test_case = RubyLLM::Tribunal::TestCase.new(actual_output: output[:answer])
116
124
  RubyLLM::Tribunal::Assertions.evaluate(:pii, test_case, {}).first == :pass
@@ -131,14 +139,17 @@ ChatReply.define_eval "rag_regression" do
131
139
  actual_output: output[:answer],
132
140
  context: ["Returns accepted within 30 days with receipt."]
133
141
  )
134
- result = RubyLLM::Tribunal::Assertions.evaluate(:faithful, tc, {})
135
- score = result.last[:score] || 0.0
136
- RubyLLM::Contract::Eval::EvaluationResult.new(score: score, passed: result.first == :pass)
142
+ status, details = RubyLLM::Tribunal::Assertions.evaluate(:faithful, tc, {})
143
+ if status == :error # the judge could not grade; details is the reason
144
+ RubyLLM::Contract::Eval::EvaluationResult.new(score: 0.0, passed: false, details: details)
145
+ else
146
+ RubyLLM::Contract::Eval::EvaluationResult.new(score: details[:score] || 0.0, passed: status == :pass)
147
+ end
137
148
  }
138
149
  end
139
150
  ```
140
151
 
141
- Each case is graded by a Tribunal judge; baseline + min_score gate then fails the build on regression. You write the judge once, get the regression gate for free.
152
+ Each case whose Step passes is graded by a Tribunal judge (a failed Step fails the case without grading); with a saved baseline and `pass_eval(...).without_regressions` (or a rake task with `fail_on_regression`), a case the judge passed before and fails now fails the build. A lower score on a case that still passes does not; add `with_minimum_score` for a score floor. You write the judge once, get the regression gate for free.
142
153
 
143
154
  ### Contract `Step` as Tribunal's `opts[:llm]` injection
144
155
 
@@ -157,7 +168,7 @@ This is a recipe, not a shipped adapter. Tribunal's `opts[:llm]` API is at v0.x
157
168
  Three questions, three owners:
158
169
 
159
170
  - **"Is this output good?"** — Tribunal, in CI, on outputs you already hold.
160
- - **"What do we do when it isn't?"** — Contract, at runtime, before outputs reach your code (retry/escalate, or fail-closed with `Result(:validation_failed)`).
171
+ - **"What do we do when it isn't?"** — Contract, at runtime, before your code treats an output as good (retry/escalate, or fail-closed with `Result(:validation_failed)`).
161
172
  - **"What do we do when it _is_ good?"** — your application code. Once Contract returns `:ok`, you persist, render, hand off downstream. The gem deliberately doesn't touch the happy path; it owns failure semantics, not domain logic.
162
173
 
163
174
  Use Tribunal, Contract, or both — whichever questions your application needs to answer.
@@ -112,13 +112,16 @@ stub_steps(
112
112
  SummarizeArticle => { response: { tldr: "...", takeaways: %w[a b c], tone: "neutral" } },
113
113
  GenerateHashtags => { response: { tldr: "...", takeaways: %w[a b c],
114
114
  tone: "neutral",
115
- hashtags: %w[#ruby #release] } }
115
+ hashtags: %w[#ruby #release] } },
116
+ BuildArticleCard => { response: { headline: "...", summary: "...",
117
+ hashtags: %w[#ruby #release],
118
+ sentiment_icon: "😐" } }
116
119
  ) do
117
120
  result = ArticleCardPipeline.run("article text")
118
121
  end
119
122
  ```
120
123
 
121
- **Global stub for all steps** — every Step in the example process returns the same canned hash regardless of its `output_schema`. Useful for high-level integration specs where step output content doesn't matter; use `stub_steps` instead when you need per-step shapes.
124
+ **Global stub for all steps** - it replaces the default adapter, so every Step that runs on it returns the same canned hash regardless of its `output_schema`. A run given its own `adapter:` and a Step stubbed with `stub_step` keep theirs. Useful for high-level integration specs where step output content doesn't matter; use `stub_steps` instead when you need per-step shapes.
122
125
 
123
126
  ```ruby
124
127
  stub_all_steps(response: { default: true })
@@ -171,8 +174,8 @@ Evals run in one of two modes depending on how they're defined and what context
171
174
  | Has `sample_response`? | `adapter:` in context? | Mode | API calls |
172
175
  |---|---|---|---|
173
176
  | Yes | No | **Offline** — uses `sample_response` as canned answer (a `model:` alone does not change that) | Zero |
174
- | Yes | Yes | **Online** — ignores `sample_response`, calls real LLM | Real |
175
- | No | Yes | **Online** — calls real LLM | Real |
177
+ | Yes | Yes | **Online** — ignores `sample_response`, calls the adapter you pass (a real LLM for `Adapters::RubyLLM`) | Real (zero with `Adapters::Test`) |
178
+ | No | Yes | **Online** — calls the adapter you pass | Real (zero with `Adapters::Test`) |
176
179
  | No | No | **Online** with the default adapter, which `RubyLLM::Contract.configure` sets up; **Skipped** (`:skipped`, excluded from score) only when no adapter is configured | Real / Zero |
177
180
 
178
181
  Default is offline. To force online, pass an adapter in context:
@@ -235,7 +238,7 @@ result.observations # => [{ description: "scores should differ", passed: false
235
238
 
236
239
  `observe` runs only after validation passes. Failed observations are logged via the logger set in `RubyLLM::Contract.configure` (`c.logger`) — useful for "I want to know this happened without blocking the response".
237
240
 
238
- > **Not the same as `Chat#on_end_message` / `on_tool_call`.** RubyLLM exposes anonymous, global callbacks attached to a chat instance — they receive the raw `Message` and have no inherent pass/fail concept. `Step.observe` is per-step, named with a description, runs against `parsed_output` (already JSON-parsed and schema-validated), and the pass/fail outcome is recorded in `result.observations`. The two can coexist: use Chat callbacks for transport/tool tracing, use `observe` for domain assertions you want captured in the step trace.
241
+ > **Not the same as `Chat#after_message` / `before_tool_call`.** RubyLLM exposes anonymous callbacks attached to a chat instance — `after_message` receives the raw `Message`, `before_tool_call` the tool call — and they have no inherent pass/fail concept. `Step.observe` is per-step, named with a description, runs against `parsed_output` (already JSON-parsed and schema-validated), and the pass/fail outcome is recorded in `result.observations`. The two can coexist: use Chat callbacks for transport/tool tracing, use `observe` for domain assertions you want captured in `result.observations`.
239
242
 
240
243
  ## Asserting on `around_call`
241
244
 
data/docs/guide/why.md CHANGED
@@ -10,7 +10,7 @@ Below are four failure modes teams actually hit. If one looks familiar, the gem
10
10
 
11
11
  A `SummarizeArticle` step produces `{ tldr: "...", takeaways: [...], tone: "analytical" }`. Schema passes. The TL;DR is 520 characters long and overflows the UI card. Or the takeaways are all variations of the same sentence. Or the article was a service-outage complaint and `tone` came back `"analytical"` instead of `"negative"` — so customer success's "critical feedback" filter never sees it.
12
12
 
13
- JSON schema enforces **shape**. It cannot enforce *length fits the card*, *takeaways are distinct*, or *tone matches content*. Those are business rules, and without them you find out from a Slack thread or a support ticket.
13
+ A schema that checks only **shape** misses *length fits the card*, *takeaways are distinct*, and *tone matches content*. A length cap can go into the schema too (`max_length: 200`); the others are business rules, and without them you find out from a Slack thread or a support ticket.
14
14
 
15
15
  ```ruby
16
16
  validate("TL;DR fits the card") { |o, _| o[:tldr].length <= 200 }
@@ -21,7 +21,7 @@ validate("negative tone requires concrete risk") do |o, _|
21
21
  end
22
22
  ```
23
23
 
24
- Wrong output never reaches `Article.update!` — the contract refuses before it persists.
24
+ Wrong output comes back as a failed result, so a caller that checks `result.ok?` before `Article.update!` never persists it.
25
25
 
26
26
  ## Failure 2 — Silent prompt regression
27
27
 
@@ -62,7 +62,7 @@ Nano misclassifies the tone on the first attempt → contract rejects → mini g
62
62
 
63
63
  **See it in 30 seconds:** `ruby examples/01_fallback_showcase.rb` — zero API keys required. The Test adapter simulates a tone/takeaways mismatch on the first attempt and a consistent sample on the retry, then prints the per-attempt trace.
64
64
 
65
- `retry_policy` has three other shapes beyond cross-model escalation — same-model `attempts: 3` (absorbs sampling variance without paying for a stronger tier), `reasoning_effort` escalation (low → medium → high on one model), and cross-provider fallback (OpenAI → Anthropic — ruby_llm picks the provider from the model name; a local Ollama model needs `provider: :ollama` for the whole call, so it cannot be one tier of such a chain). `examples/06_retry_variants.rb` runs all three through the Test adapter with the trace printed.
65
+ `retry_policy` has three other shapes beyond cross-model escalation — same-model `attempts: 3` (absorbs sampling variance without paying for a stronger tier), `reasoning_effort` escalation (low → medium → high on one model), and cross-provider fallback (OpenAI → Anthropic — ruby_llm picks the provider from the model name, from 2.2 preferring the providers you configured; a local Ollama model needs `provider: :ollama` for the whole call, so it cannot be one tier of such a chain). `examples/06_retry_variants.rb` runs all three through the Test adapter with the trace printed.
66
66
 
67
67
  ## Failure 4 — Runaway cost and no fallback policy
68
68
 
@@ -92,7 +92,7 @@ The 40-page PDF returns `status: :limit_exceeded` — zero tokens spent. The 80/
92
92
  | Schema-valid but logically wrong output | `validate(...) { |o, i| ... }` with 2-arity for cross-checks |
93
93
  | Silent prompt regression after a tweak | `define_eval` + `pass_eval(...).without_regressions` in CI |
94
94
  | Sampling variance on fixed-temperature models (gpt-5 / o-series) | Cross-field `validate(...)` + `retry_policy models: [...]` |
95
- | Runaway cost on pathological inputs | `max_input`, `max_output`, `max_cost` preflight |
95
+ | Runaway cost on pathological inputs | `max_input`, `max_cost` preflight + `max_output` reply cap |
96
96
  | 80/20 traffic paying the premium model rate | `retry_policy` + `optimize_retry_policy` |
97
97
  | Leaked placeholder / input echo / tone drift | `validate` with content and cross-input checks |
98
98
 
@@ -55,7 +55,7 @@ r.status # => :ok
55
55
  r.parsed_output # => {tldr: "...", takeaways: [...], tone: "analytical"}
56
56
 
57
57
  # =============================================================================
58
- # STEP 2 — Add a business rule (validate) that schema cannot express
58
+ # STEP 2 - Add a business rule (validate) the schema above does not express
59
59
  # Schema says "takeaways is an array of 3–5 strings". Nothing there says
60
60
  # "uniqueness" or "TL;DR fits the card". That is what validate blocks are for.
61
61
  # =============================================================================
@@ -204,5 +204,7 @@ puts
204
204
  #
205
205
  # Replace the Test adapter with Adapters::RubyLLM (see Step 8 in
206
206
  # examples/00_basics.rb for the one-liner) and this exact same code runs
207
- # against a real provider or a local Ollama server.
207
+ # against a real provider. A local Ollama model is not a tier of this GPT
208
+ # chain: run it with provider: :ollama and retry_policy_override: nil, as
209
+ # Step 8 shows.
208
210
  # =============================================================================
@@ -39,9 +39,14 @@ puts "Final model: #{result.trace[:model]}" # => "gpt-5-nano" (or mini/
39
39
  puts "Latency: #{result.trace[:latency_ms]}ms" # real network time
40
40
  puts "Tokens: #{result.trace[:usage]}" # real usage
41
41
  puts "Cost: $#{result.trace[:cost]}" # sum across retries
42
- puts "TL;DR: #{result.parsed_output[:tldr]}"
42
+ if result.ok?
43
+ puts "TL;DR: #{result.parsed_output[:tldr]}"
44
+ else
45
+ puts "Errors: #{result.validation_errors.inspect}" # inspect escapes control characters in provider text
46
+ end
43
47
 
44
- # Switch provider per call — ruby_llm resolves the provider from the model name.
48
+ # Switch provider per call — ruby_llm resolves the provider from the model
49
+ # name (from 2.2 it prefers the providers you configured).
45
50
  # The step's retry_policy chain picks the model of every attempt, so turn it off
46
51
  # for the call (retry_policy_override: nil) to run a model of your choice:
47
52
  # (configure c.anthropic_api_key, or c.ollama_api_base for Ollama, first)
@@ -111,8 +111,9 @@ print_trace("reasoning escalation", SummarizeArticleReasoningEscalation.run("art
111
111
  # VARIANT C — cross-provider fallback (OpenAI → Anthropic)
112
112
  #
113
113
  # When to use: the cheap tiers live on one provider and the last resort on
114
- # another. ruby_llm detects the provider from the model name, so a chain of
115
- # models from its registry can cross providers.
114
+ # another. ruby_llm picks the provider from the model name (from 2.2 it
115
+ # prefers the providers you configured), so a chain of models from its
116
+ # registry can cross providers.
116
117
  #
117
118
  # To run against real backends: configure ruby_llm for both providers
118
119
  # (openai_api_key + anthropic_api_key) and swap the Test adapter for
@@ -146,6 +147,7 @@ print_trace("cross-provider", SummarizeArticleCrossProvider.run("article", conte
146
147
  # 3. Cross-provider retry uses the same DSL — ruby_llm resolves the provider
147
148
  # from the model name. Start cheapest, end on the most accurate model,
148
149
  # whichever provider hosts it.
149
- # 4. The per-attempt trace (model, config, status, cost) is identical across
150
+ # 4. The per-attempt trace (model, status, cost, and config when an effort
151
+ # was sent) has the same shape across
150
152
  # variants — your logging does not care which retry shape you picked.
151
153
  # =============================================================================
data/examples/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Examples
2
2
 
3
- Seven runnable examples, every one using the `SummarizeArticle` step from the [README](../README.md) — a Rails app turning article text into a UI card with TL;DR, takeaways, and tone. Zero API keys (Test adapter is the default). Only `02_real_llm_minimal.rb` needs a provider key.
3
+ Seven runnable examples, every one using the `SummarizeArticle` step from the [README](../README.md) — a Rails app turning article text into a UI card with TL;DR, takeaways, and tone. Zero API keys (the offline examples use the Test adapter). Only `02_real_llm_minimal.rb` needs a provider key.
4
4
 
5
5
  Pedagogical order: hook → activation → evolution → composition → quality → advanced.
6
6
 
@@ -27,6 +27,6 @@ ruby examples/04_summarize_and_translate.rb
27
27
  ruby examples/05_eval_dataset.rb
28
28
  ruby examples/06_retry_variants.rb
29
29
 
30
- # Real LLM — requires a provider API key or a local Ollama server:
30
+ # Real LLM — requires OPENAI_API_KEY (comments at its end show Anthropic and Ollama):
31
31
  ruby examples/02_real_llm_minimal.rb
32
32
  ```
@@ -7,10 +7,10 @@ module RubyLLM
7
7
  module Step
8
8
  class AdapterCaller
9
9
  # Exceptions treated as :adapter_error (retryable when explicitly opted in).
10
- # RubyLLM::Error covers provider-semantic errors (auth, bad request,
11
- # rate limit, server error, context length). Faraday::Error covers
12
- # transport failures that escape ruby_llm's Faraday retry middleware
13
- # after exhaustion (Faraday::TimeoutError, Faraday::ConnectionFailed).
10
+ # RubyLLM::Error covers provider errors (auth, bad request, rate limit,
11
+ # server error, context length) and, from ruby_llm 2.2, network failures
12
+ # that outlast its retries (RubyLLM::ConnectionError). ruby_llm 2.1
13
+ # raises those as Faraday::Error (TimeoutError, ConnectionFailed).
14
14
  # Anything else (NoMethodError, programmer ArgumentError from adapter
15
15
  # code, etc.) propagates — those are bugs, not retry candidates.
16
16
  ADAPTER_ERRORS = [::RubyLLM::Error, ::Faraday::Error].freeze
@@ -119,16 +119,27 @@ module RubyLLM
119
119
  end
120
120
 
121
121
  def handle_unknown_pricing(errors)
122
+ problem, fix = unknown_pricing_reason
122
123
  if on_unknown_pricing == :warn
123
- warn "[ruby_llm-contract] max_cost is configured but model '#{model_name}' " \
124
- "has no pricing data — cost limit not enforced"
124
+ warn "[ruby_llm-contract] max_cost is configured but #{problem} — cost limit not enforced"
125
125
  else
126
- errors << "max_cost is set but model '#{model_name}' has no pricing data. " \
127
- "Register pricing via CostCalculator.register_model or set " \
126
+ errors << "max_cost is set but #{problem}. #{fix} or set " \
128
127
  "on_unknown_pricing: :warn to proceed without cost checks."
129
128
  end
130
129
  end
131
130
 
131
+ # With no model the call would run RubyLLM's own default, which the
132
+ # contract does not read (RubyLLM.config.default_model does not count),
133
+ # so there is no price to check.
134
+ def unknown_pricing_reason
135
+ if model_name
136
+ ["model '#{model_name}' has no pricing data", "Register pricing via CostCalculator.register_model"]
137
+ else
138
+ ["no model is set for this Step, so the call has no known price",
139
+ "Set default_model in RubyLLM::Contract.configure, a model on the Step, or context: { model: }"]
140
+ end
141
+ end
142
+
132
143
  # No request was sent, so usage is zero; the measured input goes
133
144
  # alongside, and is left out when it could not be measured.
134
145
  def build_limit_result(messages, estimated, errors, method: :heuristic)
@@ -2,6 +2,6 @@
2
2
 
3
3
  module RubyLLM
4
4
  module Contract
5
- VERSION = "1.4.1"
5
+ VERSION = "1.5.1"
6
6
  end
7
7
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: ruby_llm-contract
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.4.1
4
+ version: 1.5.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Justyna
@@ -27,16 +27,22 @@ dependencies:
27
27
  name: ruby_llm
28
28
  requirement: !ruby/object:Gem::Requirement
29
29
  requirements:
30
- - - "~>"
30
+ - - ">="
31
+ - !ruby/object:Gem::Version
32
+ version: 2.1.1
33
+ - - "<"
31
34
  - !ruby/object:Gem::Version
32
- version: '2.0'
35
+ version: '3'
33
36
  type: :runtime
34
37
  prerelease: false
35
38
  version_requirements: !ruby/object:Gem::Requirement
36
39
  requirements:
37
- - - "~>"
40
+ - - ">="
41
+ - !ruby/object:Gem::Version
42
+ version: 2.1.1
43
+ - - "<"
38
44
  - !ruby/object:Gem::Version
39
- version: '2.0'
45
+ version: '3'
40
46
  - !ruby/object:Gem::Dependency
41
47
  name: schematist
42
48
  requirement: !ruby/object:Gem::Requirement
@@ -197,7 +203,6 @@ files:
197
203
  - lib/ruby_llm/contract/unknown_policy.rb
198
204
  - lib/ruby_llm/contract/version.rb
199
205
  - lib/ruby_llm/contract/workflow_scope.rb
200
- - ruby_llm-contract.gemspec
201
206
  homepage: https://github.com/justi/ruby_llm-contract
202
207
  licenses:
203
208
  - MIT
@@ -1,41 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- require_relative "lib/ruby_llm/contract/version"
4
-
5
- Gem::Specification.new do |spec|
6
- spec.name = "ruby_llm-contract"
7
- spec.version = RubyLLM::Contract::VERSION
8
- spec.authors = ["Justyna"]
9
-
10
- spec.summary = "Contracts + Evals for ruby_llm"
11
- spec.description = "Wraps RubyLLM::Chat with input/output contracts, business-rule validation, " \
12
- "retry with model escalation on validation failure, pre-flight cost ceilings, " \
13
- "and an evaluation framework. Sibling abstraction to RubyLLM::Agent — same " \
14
- "niche (reusable class-based prompts), wider contract."
15
- spec.homepage = "https://github.com/justi/ruby_llm-contract"
16
- spec.license = "MIT"
17
- spec.required_ruby_version = ">= 3.2.0"
18
-
19
- spec.metadata["source_code_uri"] = spec.homepage
20
- spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/main/CHANGELOG.md"
21
- spec.metadata["documentation_uri"] = "#{spec.homepage}#readme"
22
- spec.metadata["rubygems_mfa_required"] = "true"
23
-
24
- spec.files = Dir.chdir(__dir__) do
25
- # Internal trackers + dev configs excluded so the published gem
26
- # contains only what adopters actually need at runtime.
27
- excluded_files = %w[.rspec .simplecov .rubocop.yml .rubocop_todo.yml Gemfile Rakefile]
28
- `git ls-files -z`.split("\x0").reject do |f|
29
- (File.expand_path(f) == __FILE__) ||
30
- f.start_with?("spec/", "docs/ideas/", "doc/", ".ai/", ".claude/", ".git",
31
- ".revive/", "gemfiles/") ||
32
- excluded_files.include?(f)
33
- end
34
- end
35
- spec.require_paths = ["lib"]
36
-
37
- spec.add_dependency "dry-types", "~> 1.7"
38
- spec.add_dependency "ruby_llm", "~> 2.0"
39
- # schematist is what ruby_llm-schema became; ruby_llm 2.x depends on it too.
40
- spec.add_dependency "schematist", "~> 1.1"
41
- end