ruby_llm-contract 1.4.1 → 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +62 -0
- data/README.md +5 -3
- data/docs/architecture.md +3 -3
- data/docs/guide/eval_first.md +6 -5
- data/docs/guide/getting_started.md +9 -9
- data/docs/guide/llm_judge.md +3 -3
- data/docs/guide/migration.md +1 -1
- data/docs/guide/multimodal_input.md +7 -6
- data/docs/guide/optimizing_retry_policy.md +15 -14
- data/docs/guide/pipeline.md +6 -4
- data/docs/guide/prompt_ast.md +2 -2
- data/docs/guide/rails_integration.md +3 -3
- data/docs/guide/relation_to_agent.md +5 -5
- data/docs/guide/relation_to_tribunal.md +19 -8
- data/docs/guide/testing.md +8 -5
- data/docs/guide/why.md +4 -4
- data/examples/00_basics.rb +1 -1
- data/examples/01_fallback_showcase.rb +3 -1
- data/examples/02_real_llm_minimal.rb +7 -2
- data/examples/06_retry_variants.rb +5 -3
- data/examples/README.md +2 -2
- data/lib/ruby_llm/contract/step/adapter_caller.rb +4 -4
- data/lib/ruby_llm/contract/step/limit_checker.rb +15 -4
- data/lib/ruby_llm/contract/version.rb +1 -1
- metadata +11 -6
- data/ruby_llm-contract.gemspec +0 -41
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: cfebb3e04475038261441557a159265c51e6000104b495c193b526a0e6aab0b3
|
|
4
|
+
data.tar.gz: 86cd96a083faf0a9e7c2c310bc474e5981e33bd947cae5ca2e680f90fc8148d7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 200c74207f9f91503ec60e505cbbc2bea4c0e8d83b09a3cbe6dc1c9a85a0afa8e657af81dce4b3444e5477e0a06261508161fb88514a734fa394d014b31382c4
|
|
7
|
+
data.tar.gz: 20bd3f86e2fbe26b7855055e7d2cdd5847db80341ef5d17f561917fd1dfeea7bbe1f33383c06e42bd7d9d54466dfb884cd1ec93e7c3ad1a7de63a2d1520ebaed
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,67 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.5.1 (2026-11-17)
|
|
4
|
+
|
|
5
|
+
### Changed
|
|
6
|
+
|
|
7
|
+
- **Checked against `ruby_llm` 2.2.2.** The offline suite and the live specs
|
|
8
|
+
pass on it, and no code changes are needed: the gemspec already allows it.
|
|
9
|
+
Run `bundle update ruby_llm` to pick it up. Two things change with the
|
|
10
|
+
upgrade itself, below.
|
|
11
|
+
- **With no model set anywhere, `ruby_llm` 2.2 runs `gpt-6.1-sol`.** A call
|
|
12
|
+
takes its model from the retry chain, then `context: { model: }`, then the
|
|
13
|
+
Step's `model`, then `default_model` in `RubyLLM::Contract.configure`. With
|
|
14
|
+
none of these the adapter passes no model, and `ruby_llm` uses
|
|
15
|
+
`RubyLLM.config.default_model`, which 2.2 changed from `gpt-5.6` to
|
|
16
|
+
`gpt-6.1-sol` unless you set it. A Step with `max_cost` refuses such a call
|
|
17
|
+
with `:limit_exceeded`, as before, since it cannot price a model it does not
|
|
18
|
+
know; with `on_unknown_pricing: :warn` it sends it with a warning and no cost
|
|
19
|
+
limit. Set `default_model` to keep the model you tested.
|
|
20
|
+
- **On `ruby_llm` 2.2 a network failure is a `RubyLLM::ConnectionError`.**
|
|
21
|
+
2.1 raised the Faraday error itself (`Faraday::ConnectionFailed`,
|
|
22
|
+
`Faraday::TimeoutError`); 2.2 wraps it, with the Faraday error as its
|
|
23
|
+
`cause`. A failed completion request still returns `:adapter_error`, but
|
|
24
|
+
`trace[:error_class]` changes, and a `retry_on` that names a Faraday class
|
|
25
|
+
no longer matches. Name `RubyLLM::ConnectionError` instead after upgrading.
|
|
26
|
+
A provider or network error while `token_count :exact` counts the input
|
|
27
|
+
ends as `:limit_exceeded`, as before, so a `retry_on` naming an error class
|
|
28
|
+
does not retry it.
|
|
29
|
+
|
|
30
|
+
### Fixed
|
|
31
|
+
|
|
32
|
+
- **A `max_cost` refusal with no model says no model is set.** It read
|
|
33
|
+
`model '' has no pricing data` and advised registering pricing. It now says
|
|
34
|
+
`no model is set for this Step` and advises setting `default_model` in
|
|
35
|
+
`RubyLLM::Contract.configure`, a model on the Step, or `context: { model: }`
|
|
36
|
+
(`RubyLLM.config.default_model` does not count). The
|
|
37
|
+
`on_unknown_pricing: :warn` warning also says no model is set.
|
|
38
|
+
- **The guides match the code again.** Among others: the RubyLLM chat
|
|
39
|
+
callbacks are `after_message` and `before_tool_call` (`testing.md` named
|
|
40
|
+
methods that do not exist); measuring quality with `run_eval` needs an
|
|
41
|
+
explicit adapter, since a `model:` alone keeps the `sample_response`
|
|
42
|
+
(`eval_first.md`); `max_output` caps the reply rather than refusing a call;
|
|
43
|
+
`optimize` shows scores, not cost, and builds its chain greedily; a
|
|
44
|
+
pipeline eval can assert on intermediate steps with `step_expectations:`;
|
|
45
|
+
`estimate_cost` does not read the retry chain; `RubyLLM::Message.new` takes
|
|
46
|
+
`model:`, not `model_id:`; and `retry_policy` takes `escalate` in a block.
|
|
47
|
+
- **The live `compare_models` specs name a Sonnet model that exists.** They
|
|
48
|
+
used `claude-sonnet-4-5-20250514`, which no `ruby_llm` registry lists, so
|
|
49
|
+
with `ANTHROPIC_API_KEY` set the three of them raised `ModelNotFoundError`.
|
|
50
|
+
They now use `claude-sonnet-4-5-20250929`.
|
|
51
|
+
|
|
52
|
+
## 1.5.0 (2026-10-13)
|
|
53
|
+
|
|
54
|
+
### Changed
|
|
55
|
+
|
|
56
|
+
- **Requires `ruby_llm` 2.1.1 or later in 2.x (was `~> 2.0`).** faraday-net_http
|
|
57
|
+
3.5.0 (2026-10-11) declares no Faraday dependency, so a fresh install of ruby_llm
|
|
58
|
+
2.0.0 to 2.1.0 resolves Faraday 2.0.0, which loads no default adapter, and every
|
|
59
|
+
call fails with `:net_http is not registered on Faraday::Adapter`. ruby_llm 2.1.1
|
|
60
|
+
excludes Faraday 2.0.0. An app whose lockfile already has a working Faraday ran
|
|
61
|
+
fine on the older releases, but CI now tests 2.1.1 and the latest 2.x, so the
|
|
62
|
+
gemspec allows what is tested. Run `bundle update ruby_llm` if you are on an
|
|
63
|
+
older 2.x. No code changes are needed.
|
|
64
|
+
|
|
3
65
|
## 1.4.1 (2026-10-11)
|
|
4
66
|
|
|
5
67
|
### Fixed
|
data/README.md
CHANGED
|
@@ -24,7 +24,9 @@ RubyLLM::Contract.configure do |c|
|
|
|
24
24
|
end
|
|
25
25
|
```
|
|
26
26
|
|
|
27
|
-
Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc). Requires `ruby_llm
|
|
27
|
+
Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc). Requires `ruby_llm` 2.1.1 or later in 2.x and Ruby ≥ 3.2.
|
|
28
|
+
|
|
29
|
+
A call takes its model from the retry chain, then `context: { model: }`, then the Step's `model`, then `default_model` here. With none of these it goes to `ruby_llm`'s own `RubyLLM.config.default_model`, which `ruby_llm` 2.2 changed from `gpt-5.6` to `gpt-6.1-sol` unless you set it. A Step with `max_cost` refuses such a call with `:limit_exceeded`, since it cannot price a model it does not know; with `on_unknown_pricing: :warn` it sends it with a warning and no cost limit. Set `default_model` to keep the model you tested.
|
|
28
30
|
|
|
29
31
|
## Example
|
|
30
32
|
|
|
@@ -115,7 +117,7 @@ end
|
|
|
115
117
|
expect(SummarizeArticle).to pass_eval("regression").without_regressions
|
|
116
118
|
```
|
|
117
119
|
|
|
118
|
-
A bad prompt edit or model swap that drops accuracy on the frozen dataset → red CI, blocked merge. Save a baseline once (`report.save_baseline!`, or `t.save_baseline = true` in the rake task); until one exists, `without_regressions` has nothing to compare against and passes. Every production miss should become the next `add_case`. See [Prevent silent prompt regressions](docs/guide/eval_first.md) for the full flywheel.
|
|
120
|
+
A bad prompt edit or model swap that drops accuracy on the frozen dataset → red CI, blocked merge. Save a baseline once (`report.save_baseline!`, or `t.save_baseline = true` in the rake task); until one exists, `without_regressions` has nothing to compare against, and the matcher checks only that the eval passes. Every production miss should become the next `add_case`. See [Prevent silent prompt regressions](docs/guide/eval_first.md) for the full flywheel.
|
|
119
121
|
|
|
120
122
|
## Do I need this?
|
|
121
123
|
|
|
@@ -144,7 +146,7 @@ Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and
|
|
|
144
146
|
|
|
145
147
|
## Relation to `RubyLLM::Agent`
|
|
146
148
|
|
|
147
|
-
`Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly - Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate
|
|
149
|
+
`Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly - Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy { escalate ... }` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. RubyLLM 2.1's `RubyLLM::Evaluation` grades answers; `define_eval` guards against regressions and cost in CI - the mapping covers both. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
|
|
148
150
|
|
|
149
151
|
## Relation to `ruby_llm-tribunal`
|
|
150
152
|
|
data/docs/architecture.md
CHANGED
|
@@ -9,7 +9,7 @@ RubyLLM::Contract::Step::Base # single contracted step
|
|
|
9
9
|
├── Step::Dsl # DSL macros (prompt, validate, output_schema, etc.)
|
|
10
10
|
├── Step::RetryPolicy # attempts, models, reasoning_effort, retry_on
|
|
11
11
|
├── Step::RetryExecutor # retry loop driven by RetryPolicy
|
|
12
|
-
├── Step::LimitChecker # preflight cost / input
|
|
12
|
+
├── Step::LimitChecker # preflight cost / input checks (max_output feeds the cost estimate)
|
|
13
13
|
├── Step::Runner # runtime flow (single attempt)
|
|
14
14
|
├── Step::Result # status + outputs + errors + trace
|
|
15
15
|
├── Step::Trace # model, latency, tokens, cost, attempts
|
|
@@ -33,13 +33,13 @@ RubyLLM::Contract::Eval # quality measurement
|
|
|
33
33
|
├── Eval::Dataset # test cases
|
|
34
34
|
├── Eval::Runner # sequential or concurrent execution
|
|
35
35
|
├── Eval::Report # score, pass_rate, per-case results
|
|
36
|
-
├── Eval::AggregatedReport #
|
|
36
|
+
├── Eval::AggregatedReport # one candidate's repeated runs (runs: N), averaged
|
|
37
37
|
├── Eval::CaseResult # value object (name, passed?, output, expected, mismatches, cost)
|
|
38
38
|
├── Eval::ExpectationEvaluator # expected / expected_traits / evaluator proc
|
|
39
39
|
├── Eval::ModelComparison # compare_models result (table, best_for, candidate configs)
|
|
40
40
|
├── Eval::Recommender # model recommendation algorithm (candidates → optimal config)
|
|
41
41
|
├── Eval::Recommendation # recommendation result (best, retry_chain, savings, to_dsl)
|
|
42
|
-
├── Eval::RetryOptimizer # optimize_retry_policy
|
|
42
|
+
├── Eval::RetryOptimizer # optimize_retry_policy; its Result has the per-eval breakdown, fallback list
|
|
43
43
|
├── Eval::BaselineDiff # save_baseline! + without_regressions comparison
|
|
44
44
|
├── Eval::PromptDiffComparator # compare_with prompt A/B diff
|
|
45
45
|
└── Eval::EvalHistory # time-series view across saved reports
|
data/docs/guide/eval_first.md
CHANGED
|
@@ -34,7 +34,7 @@ SummarizeArticle.define_eval("regression") do
|
|
|
34
34
|
end
|
|
35
35
|
```
|
|
36
36
|
|
|
37
|
-
"Partial match" means `expected` is treated as a subset of `parsed_output` — each key in `expected` must equal the corresponding key in the output, but extra keys in the output are ignored. So the `ruby release` case passes whenever the model produces `tone: "analytical"`, regardless of what shows up in `tldr` or `takeaways`. To assert on multiple keys, list them all in `expected`.
|
|
37
|
+
"Partial match" means `expected` is treated as a subset of `parsed_output` — each key in `expected` must equal the corresponding key in the output, but extra keys in the output are ignored. A Regexp in `expected` matches the value as a string, a Range must include it, and a key whose output value is `nil` counts as missing. So the `ruby release` case passes whenever the step's own schema and `validate` rules pass and the model produces `tone: "analytical"`, regardless of what shows up in `tldr` or `takeaways`. To assert on multiple keys, list them all in `expected`.
|
|
38
38
|
|
|
39
39
|
Only after the eval exists, touch the prompt-shaping building blocks of the Step: `system`, `rule`, and `example` (blocks inside `prompt do`), `validate` (class-level invariant checks that run after the model returns), or the prompt version itself. See [Getting Started](getting_started.md) for the DSL surface.
|
|
40
40
|
|
|
@@ -88,7 +88,7 @@ diff = SummarizeArticleV2.compare_with(
|
|
|
88
88
|
context: { retry_policy_override: nil } # run mini, not the step's retry chain
|
|
89
89
|
)
|
|
90
90
|
|
|
91
|
-
diff.safe_to_switch? # => true
|
|
91
|
+
diff.safe_to_switch? # => true when both ran the same cases and none scored lower
|
|
92
92
|
```
|
|
93
93
|
|
|
94
94
|
This is the cleanest eval-first move: same eval, same cases, two prompt versions, one answer.
|
|
@@ -152,7 +152,7 @@ SummarizeArticle.estimate_eval_cost("regression",
|
|
|
152
152
|
# => { "gpt-4.1-nano" => 0.003207, "gpt-4.1-mini" => 0.012829, "gpt-4.1" => 0.064142 }
|
|
153
153
|
```
|
|
154
154
|
|
|
155
|
-
Use it in CI to decide which models are worth running regression on, or to
|
|
155
|
+
Use it in CI to decide which models are worth running regression on, or to budget a build. It is a floor, not a worst case: it counts one attempt per case, `max_output` (or 256) output tokens, and $0 for a model it cannot price.
|
|
156
156
|
|
|
157
157
|
## Team workflow
|
|
158
158
|
|
|
@@ -184,7 +184,8 @@ Don't optimize cost before you stabilize quality:
|
|
|
184
184
|
```ruby
|
|
185
185
|
comparison = SummarizeArticle.compare_models(
|
|
186
186
|
"regression",
|
|
187
|
-
candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }, { model: "gpt-4.1" }]
|
|
187
|
+
candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }, { model: "gpt-4.1" }],
|
|
188
|
+
context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new } # live; without it a sample_response answers
|
|
188
189
|
)
|
|
189
190
|
|
|
190
191
|
comparison.best_for(min_score: 0.95)
|
|
@@ -202,7 +203,7 @@ comparison.best_for(min_score: 0.95)
|
|
|
202
203
|
|
|
203
204
|
1. Write `define_eval` before touching the prompt.
|
|
204
205
|
2. Treat `sample_response` as smoke only.
|
|
205
|
-
3. Use `run_eval("name", context: { model: "..." })` for real quality measurement.
|
|
206
|
+
3. Use `run_eval("name", context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new, model: "..." })` for real quality measurement — a `model:` alone keeps the canned `sample_response`, and a Step with a retry chain needs `retry_policy_override: nil` too, or the chain picks the model.
|
|
206
207
|
4. Use `compare_with` for every serious prompt change.
|
|
207
208
|
5. Gate merges with `pass_eval`.
|
|
208
209
|
6. Feed every production miss back into the dataset.
|
|
@@ -27,7 +27,7 @@ class SummarizeArticle < RubyLLM::Contract::Step::Base
|
|
|
27
27
|
string :tone, enum: %w[neutral positive negative analytical]
|
|
28
28
|
end
|
|
29
29
|
|
|
30
|
-
# 3. Business rules
|
|
30
|
+
# 3. Business rules - checks this schema does not make
|
|
31
31
|
validate("TL;DR fits the card") { |o, _| o[:tldr].length <= 200 }
|
|
32
32
|
validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }
|
|
33
33
|
|
|
@@ -45,7 +45,7 @@ Each `validate` block receives `(parsed_output, input)` — the convention `|o,
|
|
|
45
45
|
|
|
46
46
|
## Validation and retry behavior
|
|
47
47
|
|
|
48
|
-
When the cheap model returns output that fails a `validate` block (status `:validation_failed`) or can't be parsed as JSON (status `:parse_error`), retry falls back to the next model in `models:` and tries again. Transport errors (timeouts, 5xx) retry
|
|
48
|
+
When the cheap model returns output that fails a `validate` block (status `:validation_failed`) or can't be parsed as JSON (status `:parse_error`), retry falls back to the next model in `models:` and tries again. Transport errors (timeouts, 5xx) retry inside RubyLLM, and by default not via `retry_policy`; to retry them there too, name them in `retry_on` (see below). See [Optimizing retry_policy](optimizing_retry_policy.md) for tuning the chain.
|
|
49
49
|
|
|
50
50
|
```ruby
|
|
51
51
|
result = SummarizeArticle.run(article_text)
|
|
@@ -76,7 +76,7 @@ retry_policy do
|
|
|
76
76
|
end
|
|
77
77
|
```
|
|
78
78
|
|
|
79
|
-
A class matches its subclasses; `:adapter_error` in the list retries every adapter error. As with statuses, the list replaces the defaults. Switching to a fallback model on transport errors inside one call is RubyLLM's `chat.with_fallbacks`; the contract's adapter does not call it, so fallbacks across models belong in `escalate`.
|
|
79
|
+
A class matches its subclasses; `:adapter_error` in the list retries every adapter error. A network failure (refused connection, timeout) is `RubyLLM::ConnectionError` on `ruby_llm` 2.2; 2.1 raised the Faraday error itself (`Faraday::ConnectionFailed`, `Faraday::TimeoutError`), so a `retry_on` naming a Faraday class stops matching after the upgrade. A provider or network error while `token_count :exact` counts the input is not an adapter error: the Step ends as `:limit_exceeded`, so a `retry_on` naming an error class does not retry it. As with statuses, the list replaces the defaults. Switching to a fallback model on transport errors inside one call is RubyLLM's `chat.with_fallbacks`; the contract's adapter does not call it, so fallbacks across models belong in `escalate`.
|
|
80
80
|
|
|
81
81
|
### Responses cut off by a token limit
|
|
82
82
|
|
|
@@ -152,7 +152,7 @@ SummarizeArticle.define_eval("regression") do
|
|
|
152
152
|
end
|
|
153
153
|
```
|
|
154
154
|
|
|
155
|
-
`expected:` is treated as a partial match against `parsed_output` — extra output keys are ignored, listed keys must equal. So
|
|
155
|
+
`expected:` is treated as a partial match against `parsed_output` — extra output keys are ignored, listed keys must equal (a Regexp or Range in `expected` matches instead). So the first case passes whenever the step's own schema and `validate` rules pass and the model produces `tone: "analytical"`, regardless of `tldr` content. To assert on multiple keys, list them all in `expected`.
|
|
156
156
|
|
|
157
157
|
Gate CI on score and cost thresholds:
|
|
158
158
|
|
|
@@ -173,11 +173,11 @@ report.save_baseline!
|
|
|
173
173
|
expect(SummarizeArticle).to pass_eval("regression").without_regressions
|
|
174
174
|
```
|
|
175
175
|
|
|
176
|
-
`without_regressions` fails the build
|
|
176
|
+
`pass_eval` always requires the eval to pass (every case that ran, or the score set by `with_minimum_score`; a case skipped for want of an adapter does not count, but at least one case must run). `without_regressions` adds a check against the saved baseline: it also fails the build if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality — or was removed from the eval. A previously-passing case that was skipped this run (no adapter configured) also fails the gate, listed as `SKIPPED` rather than as a regression.
|
|
177
177
|
|
|
178
178
|
## Budget caps
|
|
179
179
|
|
|
180
|
-
`max_input
|
|
180
|
+
`max_input` and `max_cost` are preflight checks: an attempt whose estimate exceeds the limit is never sent, so it generates nothing and costs nothing. `max_output` caps the reply at the provider and is the output side of the `max_cost` estimate. With `token_count :exact` the provider counts the input before the check, and in a retry chain a refused attempt does not undo the attempts before it.
|
|
181
181
|
|
|
182
182
|
```ruby
|
|
183
183
|
result = SummarizeArticle.run(forty_page_report) # ~128k characters
|
|
@@ -201,7 +201,7 @@ max_cost 0.01, on_unknown_pricing: :warn
|
|
|
201
201
|
|
|
202
202
|
Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
|
|
203
203
|
|
|
204
|
-
The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total
|
|
204
|
+
The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total leaves out what it cannot price (a case with no priced attempt counts as $0), so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced costs and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. Since 1.1.2 a real call whose provider sent no token counts is flagged too, instead of passing as a free zero-token run.
|
|
205
205
|
|
|
206
206
|
### What a trace's usage and cost count
|
|
207
207
|
|
|
@@ -230,7 +230,7 @@ class SummarizeArticle < RubyLLM::Contract::Step::Base
|
|
|
230
230
|
end
|
|
231
231
|
```
|
|
232
232
|
|
|
233
|
-
The count comes from RubyLLM's `chat.count_tokens` (OpenAI Responses, Anthropic, Gemini, Vertex AI, Bedrock Converse) and covers the prompt and attachments, so `attachment_token_estimate` is not needed. What else a provider counts is its own: OpenAI's and Anthropic's counts include the output schema, Gemini's leaves it out. It costs one extra request per attempt, and RubyLLM leaves `provider_options` out of it. Where no count can be had - a provider without the endpoint (Ollama, for one), a
|
|
233
|
+
The count comes from RubyLLM's `chat.count_tokens` (OpenAI Responses, Anthropic, Gemini, Vertex AI, Bedrock Converse) and covers the prompt and attachments, so `attachment_token_estimate` is not needed. What else a provider counts is its own: OpenAI's and Anthropic's counts include the output schema, Gemini's leaves it out. It costs one extra request per attempt, and RubyLLM leaves `provider_options` out of it. Where no count can be had - a provider without the endpoint (Ollama, for one), a request that fails with a RubyLLM or Faraday error, or an adapter without `count_tokens` - the call is refused as `:limit_exceeded` with the reason, never measured with the heuristic under the name "exact". The `Test` adapter counts with the heuristic, since it has no provider to ask. The output side of `max_cost` is still an estimate (`max_output`, or the input size without one).
|
|
234
234
|
|
|
235
235
|
### Local and custom models
|
|
236
236
|
|
|
@@ -262,7 +262,7 @@ SummarizeArticle.estimate_eval_cost("regression",
|
|
|
262
262
|
# => { "gpt-4.1-nano" => 0.003207, "gpt-4.1-mini" => 0.012829, "gpt-4.1" => 0.064142 }
|
|
263
263
|
```
|
|
264
264
|
|
|
265
|
-
`estimate_cost` returns `nil` when
|
|
265
|
+
`estimate_cost` returns `nil` when it does not know the model, and a hash with `estimated_cost: nil` when it knows the model but not its price. `estimate_eval_cost` silently treats unknown-pricing cases as `$0.00` and sums the rest — it does **not** fail closed the way `max_cost` does. Treat its output as a floor, not a guarantee; register pricing via `CostCalculator.register_model` before relying on it for budget decisions.
|
|
266
266
|
|
|
267
267
|
## Provider options
|
|
268
268
|
|
data/docs/guide/llm_judge.md
CHANGED
|
@@ -71,7 +71,7 @@ class AccuracyJudge < RubyLLM::Contract::Step::Base
|
|
|
71
71
|
end
|
|
72
72
|
```
|
|
73
73
|
|
|
74
|
-
The `retry_policy` escalates to `gpt-5-mini` if the cheaper `gpt-5-nano` fails schema/validate (e.g., returns malformed JSON) — the judge stays reliable without paying for the bigger model on every call. `max_cost 0.01` bounds each attempt (with `max_output 2_000`, up to roughly 24k input tokens on `gpt-5-mini
|
|
74
|
+
The `retry_policy` escalates to `gpt-5-mini` if the cheaper `gpt-5-nano` fails schema/validate (e.g., returns malformed JSON) — the judge stays reliable without paying for the bigger model on every call. `max_cost 0.01` bounds each attempt's estimated cost (with `max_output 2_000`, up to roughly 24k input tokens on `gpt-5-mini`; the input side is a heuristic estimate, not the billed amount), so a case that falls back from nano to mini is estimated at no more than about $0.012: the judge runs N× per eval suite (one call per case), so capping per-call cost prevents calibration runs from blowing the budget.
|
|
75
75
|
|
|
76
76
|
This is small on purpose: one boolean, one reason. Easy to read, easy to disagree with.
|
|
77
77
|
|
|
@@ -147,9 +147,9 @@ Only now, after the judge has earned trust on real data, do you use it as the or
|
|
|
147
147
|
- `output` — `parsed_output` of the Step under test. Here `output[:tldr]` is the summary `SummarizeArticle` produced (the same field declared in its `output_schema`).
|
|
148
148
|
- `input` — whatever you passed to `add_case input:`. Here it's the original article string.
|
|
149
149
|
|
|
150
|
-
|
|
150
|
+
Return a `Float` in `[0.0, 1.0]` — the eval framework averages per-case scores into the suite score, clamping a number outside that range, and a number of 0.5 or more passes the case. `true` and `false` count as 1.0 and 0.0, and an `EvaluationResult` is used as returned; use weighted partial scores if your judge returns gradations.
|
|
151
151
|
|
|
152
|
-
> ⚠️ **Heads-up before you copy the snippet below.** Smoke evals on `SummarizeArticle` (per [Getting Started](getting_started.md)) typically run with `RubyLLM::Contract::Adapters::Test` for determinism — canned summaries, no API calls. **The `AccuracyJudge.run(...)` inside the evaluator does NOT inherit that Test adapter.** Each Step picks its adapter independently (from `context: { adapter: }`, or the `default_adapter` set in `RubyLLM::Contract.configure`). Run accuracy-regression evals live (an explicit `context:` or a configured default adapter), and rely on `max_cost` on the judge to bound
|
|
152
|
+
> ⚠️ **Heads-up before you copy the snippet below.** Smoke evals on `SummarizeArticle` (per [Getting Started](getting_started.md)) typically run with `RubyLLM::Contract::Adapters::Test` for determinism — canned summaries, no API calls. **The `AccuracyJudge.run(...)` inside the evaluator does NOT inherit that Test adapter.** Each Step picks its adapter independently (from `context: { adapter: }`, or the `default_adapter` set in `RubyLLM::Contract.configure`). Run accuracy-regression evals live (an explicit `context:` or a configured default adapter), and rely on `max_cost` on the judge to bound each call's estimated cost.
|
|
153
153
|
>
|
|
154
154
|
> This is intentional: the judge IS the oracle, so faking its verdicts would defeat the purpose. Keep the Test-adapter smoke evals for schema/wiring checks only.
|
|
155
155
|
|
data/docs/guide/migration.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
> Read this when your contract needs to send a PDF, image, or audio file to the LLM — not just text.
|
|
4
4
|
|
|
5
|
-
`ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy escalate
|
|
5
|
+
`ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy { escalate ... }`, and trace observability still apply. The gem does **not** ship its own multimodal API — it forwards `with:` to `RubyLLM::Chat#ask`, which RubyLLM normalises per provider (Anthropic, OpenAI, Gemini).
|
|
6
6
|
|
|
7
7
|
## Minimal example
|
|
8
8
|
|
|
@@ -48,7 +48,7 @@ result.trace[:cost] # => 0.0042 (total across attempts)
|
|
|
48
48
|
|
|
49
49
|
1. **Input vs attachment.** The `input` argument to `Step.run` is the text prompt. The attachment travels via `context: { attachment: ... }` — opaque to the contract layer, forwarded to the adapter.
|
|
50
50
|
2. **Adapter forwards `with:`.** `RubyLLM::Contract::Adapters::RubyLLM` reads `options[:attachment]` and passes it to `chat.ask(content, with: attachment)`. RubyLLM picks the right wire format per provider.
|
|
51
|
-
3. **Multi-attachment supported.** `with: [pdf1, pdf2]` or `with: { images: [...], pdfs: [...] }` works natively
|
|
51
|
+
3. **Multi-attachment supported.** `with: [pdf1, pdf2]` or `with: { images: [...], pdfs: [...] }` works natively; RubyLLM groups the attachments itself.
|
|
52
52
|
4. **`with: nil` is a no-op.** Text-only contracts unaffected — the kwarg defaults to nil.
|
|
53
53
|
|
|
54
54
|
## Cost: `attachment_token_estimate` is required
|
|
@@ -70,7 +70,7 @@ The same estimate applies to:
|
|
|
70
70
|
|
|
71
71
|
### Fail-closed without estimate
|
|
72
72
|
|
|
73
|
-
If your contract has `max_cost` or `max_input` set, receives an attachment, and `attachment_token_estimate` is **not declared**, the call fails with `:limit_exceeded` — the gem refuses to spend money on cost it cannot bound.
|
|
73
|
+
If your contract has `max_cost` or `max_input` set, receives an attachment, and `attachment_token_estimate` is **not declared**, the call fails with `:limit_exceeded` — the gem refuses to spend money on cost it cannot bound. This applies to the default heuristic count; with `token_count :exact` the provider counts the attachment instead, and the call fails closed only when it cannot.
|
|
74
74
|
|
|
75
75
|
```ruby
|
|
76
76
|
class MyContract < RubyLLM::Contract::Step::Base
|
|
@@ -103,6 +103,7 @@ end
|
|
|
103
103
|
```ruby
|
|
104
104
|
ExtractInvoiceData.estimate_cost(
|
|
105
105
|
input: "Look for vendor, amount...",
|
|
106
|
+
model: "gpt-4.1-mini", # estimate_cost does not read the retry chain
|
|
106
107
|
attachment: "tmp/invoice.pdf"
|
|
107
108
|
)
|
|
108
109
|
# => { model: "gpt-4.1-mini",
|
|
@@ -138,9 +139,9 @@ A Step is single-turn: each `run` sends its prompt with the attachment and keeps
|
|
|
138
139
|
|
|
139
140
|
- **OpenAI** — PDFs sent as `type: 'file'` with `file_data` (base64). Images as `image_url`. Audio as `input_audio`. Vision pricing varies by image detail; check the model card.
|
|
140
141
|
- **Anthropic** — PDFs sent as `type: 'document'` with `source.type: 'base64'` or `'url'` (auto-selected). Images same. Page limit ~100 per call.
|
|
141
|
-
- **Gemini** —
|
|
142
|
+
- **Gemini** — Media sent inline via `inline_data` with `mime_type`; a file already uploaded to the provider goes as `file_data`. Multimodal token counting is unified.
|
|
142
143
|
|
|
143
|
-
RubyLLM dispatches on `attachment.type` (`:image`, `:pdf`, `:audio`, `:text`, `:unknown`).
|
|
144
|
+
RubyLLM dispatches on `attachment.type` (`:image`, `:pdf`, `:audio`, `:video`, `:text`, `:document`, `:unknown`). Give tempfiles the right extension (`.pdf`, `.png`, etc.): RubyLLM detects the MIME type from the filename or the content, and a file it cannot place becomes `:unknown`, which the provider rejects.
|
|
144
145
|
|
|
145
146
|
## Testing contracts with attachments
|
|
146
147
|
|
|
@@ -152,7 +153,7 @@ RSpec.describe ExtractInvoiceData do
|
|
|
152
153
|
expect_any_instance_of(RubyLLM::Chat).to receive(:ask)
|
|
153
154
|
.with(anything, with: "fixtures/invoice.pdf")
|
|
154
155
|
.and_return(RubyLLM::Message.new(role: :assistant, content: '{"vendor":"X","invoice_number":"1","total_amount":10,"currency":"EUR"}',
|
|
155
|
-
|
|
156
|
+
model: "gpt-4.1-mini",
|
|
156
157
|
input_tokens: 200, output_tokens: 50))
|
|
157
158
|
|
|
158
159
|
result = described_class.run("extract", context: { attachment: "fixtures/invoice.pdf" })
|
|
@@ -6,7 +6,7 @@ You defined `SummarizeArticle` in [Getting Started](getting_started.md) with `re
|
|
|
6
6
|
|
|
7
7
|
## Requirements
|
|
8
8
|
|
|
9
|
-
-
|
|
9
|
+
- **A `retry_policy` is optional.** The optimizer runs each candidate without the Step's own chain, and the DSL it prints is the policy to add.
|
|
10
10
|
- **2–3 evals per step.** One eval optimizes for one scenario; with only `smoke`, you get a recommendation that passes smoke but may miss production edge cases. See [eval-first](eval_first.md).
|
|
11
11
|
- **Rake task `ruby_llm_contract:optimize` is registered when `lib/ruby_llm/contract/rake_task.rb` is loaded.** In Rails apps the railtie auto-loads it lazily during `rake` invocation — no manual setup needed. In non-Rails projects, add `require "ruby_llm/contract/rake_task"` to your `Rakefile` (or any file under `lib/tasks/`) and set `EVAL_DIRS=...` on the command line.
|
|
12
12
|
|
|
@@ -32,9 +32,9 @@ rake ruby_llm_contract:optimize \
|
|
|
32
32
|
|
|
33
33
|
The `@low` (or `@medium`, `@high`) suffix on a candidate is CLI shorthand for `{ model: "gpt-5-mini", reasoning_effort: "low" }` — saves you constructing hash strings on the command line. Plain model names without `@` use the Step's default thinking config (or none). `@default` runs the candidate with no effort at all, which is what a model without reasoning needs when the Step sets one: `CANDIDATES=gpt-6-luna@none,gpt-4.1-mini@default`.
|
|
34
34
|
|
|
35
|
-
List candidates from the cheapest to the most expensive. The optimizer takes that order as the cost order: the suggested chain tries them in it, and the first candidate that passes every eval is the safe fallback. It does not sort by price.
|
|
35
|
+
List candidates from the cheapest to the most expensive. The optimizer takes that order as the cost order: the suggested chain tries them in it, and the first candidate that passes every eval is the safe fallback. It does not sort by price. A candidate passes an eval when it scores at least `MIN_SCORE`, 0.95 by default; set `MIN_SCORE=1` to require full scores.
|
|
36
36
|
|
|
37
|
-
Offline
|
|
37
|
+
Offline, each eval with a `sample_response` uses it — zero API calls. An eval without one runs on the default adapter, which `RubyLLM::Contract.configure` sets up, so it calls the provider; give every eval a `sample_response` for a fully offline run. **When every eval has one, every candidate receives the same canned response.** That's fine for a smoke test (verifying evals load, candidates parse, output renders) but it doesn't compare model quality. For real optimization, go live.
|
|
38
38
|
|
|
39
39
|
## Optimize against real models
|
|
40
40
|
|
|
@@ -75,23 +75,24 @@ SummarizeArticle — fallback list optimization
|
|
|
75
75
|
```
|
|
76
76
|
|
|
77
77
|
Reading the table:
|
|
78
|
-
- **`←` marks scores below
|
|
78
|
+
- **`←` marks scores below 1.0 in the hardest eval.** Not a selection hint — just "this candidate fails the row that matters most".
|
|
79
79
|
- **Hardest eval** = the one that forces the strong fallback. Here, `critical_tone` demands `gpt-5-mini` at its default effort.
|
|
80
|
-
- **Suggested fallback list** =
|
|
80
|
+
- **Suggested fallback list** = a greedy chain, not a searched-for shortest one: walking the candidates cheapest first, it keeps each one that passes an eval no earlier candidate passed, and ends on the first candidate that passes every eval. Order matters: `gpt-5-nano` is tried first; on validation failure, the gem falls back to `gpt-5-mini` at low effort, then at its default. Retry fires on `:validation_failed` and `:parse_error`, not on a low eval score, so a cheaper candidate's eval failures must be ones your `validate` rules catch, or the fallback never runs for them.
|
|
81
81
|
|
|
82
82
|
Copy the DSL, paste into your step, verify with `rake ruby_llm_contract:eval` (defined by `RubyLLM::Contract::RakeTask.new`, see [Rails integration](rails_integration.md#7-ci-gate--block-regressions-before-merge)). You just dropped `gpt-5` from the chain — most requests finish on nano, mini handles what nano misses, and the strong model was never needed.
|
|
83
83
|
|
|
84
84
|
## Measure effective cost before shipping
|
|
85
85
|
|
|
86
|
-
`optimize` shows
|
|
86
|
+
`optimize` shows scores, not cost, and `compare_models` on its own reports what each candidate's run cost with no fallback (its same-model retries included). In production, a candidate whose validator rejects 20% of outputs actually costs about `first_try_cost + fallback_cost × 0.20` per request. A run without the fallback hides this.
|
|
87
87
|
|
|
88
|
-
`production_mode:` is an optional `compare_models` kwarg. With `{ fallback: "model" }` it runs each candidate with a runtime `[candidate, fallback]` chain and reports **effective cost
|
|
88
|
+
`production_mode:` is an optional `compare_models` kwarg. With `{ fallback: "model" }` it runs each candidate with a runtime `[candidate, fallback]` chain and reports **effective cost**: the eval's total, every first attempt plus every fallback that ran. Without it, you get only first-attempt metrics (the default). A fallback without a `reasoning_effort` runs with the candidate's, so a fallback naming the candidate's model sends the same request and is skipped; `{ model: "...", reasoning_effort: :default }` falls back with no effort.
|
|
89
89
|
|
|
90
90
|
```ruby
|
|
91
91
|
SummarizeArticle.compare_models(
|
|
92
92
|
"dense_article",
|
|
93
93
|
candidates: [{ model: "gpt-5-nano" }, { model: "gpt-5-mini", reasoning_effort: "low" }, { model: "gpt-5" }],
|
|
94
|
-
production_mode: { fallback: "gpt-5" }
|
|
94
|
+
production_mode: { fallback: "gpt-5" },
|
|
95
|
+
context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new } # live; without it a sample_response answers
|
|
95
96
|
).print_summary
|
|
96
97
|
```
|
|
97
98
|
|
|
@@ -110,9 +111,9 @@ dense_article — model comparison
|
|
|
110
111
|
Cheapest at 100%: gpt-5-mini (effort: low)
|
|
111
112
|
```
|
|
112
113
|
|
|
113
|
-
- **first-attempt** — cost of the first
|
|
114
|
-
- **fallback %**
|
|
115
|
-
- **effective cost** — total
|
|
114
|
+
- **first-attempt** — cost of the first attempts alone, summed over the eval's cases.
|
|
115
|
+
- **fallback %** - fraction of cases that needed more than one attempt, so the fallback ran.
|
|
116
|
+
- **effective cost** — total over the eval's cases, fallbacks included.
|
|
116
117
|
- **`—`** — the fallback would send the candidate's own request, so the candidate runs alone and there is no chain to observe.
|
|
117
118
|
|
|
118
119
|
Run this before finalizing: a candidate saving 3× on first-attempt but falling back 60% of the time may save only 1.2× in production.
|
|
@@ -123,7 +124,7 @@ Run this before finalizing: a candidate saving 3× on first-attempt but falling
|
|
|
123
124
|
|
|
124
125
|
- **"No viable chain" from a single live run.** Re-run with `RUNS=3`. If scores jump, the first run was noise. Never trust single-run results with gpt-5 / o-series in the pool — `temperature=1.0` is server-enforced.
|
|
125
126
|
- **Every candidate fails the same eval**, including the strongest. The eval is rejecting correct answers. Run the step directly (`context: { retry_policy_override: nil, model: "gpt-4.1" }`), inspect the output, compare with the `verify` block. Loosen the eval if the output is correct but not one of the accepted values.
|
|
126
|
-
- **Testing one specific hypothesis.** (e.g. "does `mini@medium` help on `critical_tone`?") Use `SummarizeArticle.compare_models("critical_tone", candidates: [{ model: "gpt-5-mini", reasoning_effort: "medium" }], runs: 3)` directly — three calls instead of rerunning the whole optimize pass.
|
|
127
|
+
- **Testing one specific hypothesis.** (e.g. "does `mini@medium` help on `critical_tone`?") Use `SummarizeArticle.compare_models("critical_tone", candidates: [{ model: "gpt-5-mini", reasoning_effort: "medium" }], runs: 3, context: { adapter: RubyLLM::Contract::Adapters::RubyLLM.new })` directly — three calls instead of rerunning the whole optimize pass.
|
|
127
128
|
|
|
128
129
|
## Programmatic API names
|
|
129
130
|
|
|
@@ -131,7 +132,7 @@ Metrics exposed on `Report` / `AggregatedReport` keep their original names: `sin
|
|
|
131
132
|
|
|
132
133
|
## thinking DSL note
|
|
133
134
|
|
|
134
|
-
Set the default reasoning effort once on the Step class
|
|
135
|
+
Set the default reasoning effort once on the Step class, with the `effort:` and `budget:` of `RubyLLM::Agent.thinking` (not its other options):
|
|
135
136
|
|
|
136
137
|
```ruby
|
|
137
138
|
class SummarizeArticle < RubyLLM::Contract::Step::Base
|
|
@@ -153,4 +154,4 @@ SummarizeArticle.compare_models("regression", candidates: [{ model: "gpt-6-luna"
|
|
|
153
154
|
retry_policy { escalate({ model: "gpt-6-luna" }, { model: "gpt-4.1-mini", reasoning_effort: :default }) }
|
|
154
155
|
```
|
|
155
156
|
|
|
156
|
-
`nil` is not `:default`: a nil effort is ignored and the class one is sent. The gem does not drop an effort by itself based on the model. A chain element without an effort runs with the call's (or the class's), and `trace.attempts[n][:config]` shows the effort
|
|
157
|
+
`nil` is not `:default`: a nil effort is ignored and the class one is sent. The gem does not drop an effort by itself based on the model. A chain element without an effort runs with the call's (or the class's), and `trace.attempts[n][:config]` shows the effort an attempt was sent; it is absent when the attempt was sent no effort.
|
data/docs/guide/pipeline.md
CHANGED
|
@@ -42,7 +42,7 @@ class GenerateHashtags < RubyLLM::Contract::Step::Base
|
|
|
42
42
|
|
|
43
43
|
prompt do
|
|
44
44
|
rule "Preserve tldr / takeaways / tone exactly as given."
|
|
45
|
-
user "Article summary: {tldr}\nTone: {tone}\nGenerate 2 to 5 concise hashtags."
|
|
45
|
+
user "Article summary: {tldr}\nTakeaways: {takeaways}\nTone: {tone}\nGenerate 2 to 5 concise hashtags."
|
|
46
46
|
end
|
|
47
47
|
|
|
48
48
|
validate("tone preserved") { |o, input| o[:tone] == input[:tone] }
|
|
@@ -101,13 +101,14 @@ adapter = RubyLLM::Contract::Adapters::Test.new(response: {
|
|
|
101
101
|
|
|
102
102
|
result = ArticleCardPipeline.run("article text", context: { adapter: adapter })
|
|
103
103
|
result.failed? # => true
|
|
104
|
-
result.failed_step # => :summarize (validate rejected;
|
|
104
|
+
result.failed_step # => :summarize (validate rejected; no retry_policy, so one attempt)
|
|
105
105
|
# tag and card never run — no downstream tokens spent on garbage
|
|
106
106
|
```
|
|
107
107
|
|
|
108
108
|
## Per-step model override
|
|
109
109
|
|
|
110
110
|
```ruby
|
|
111
|
+
# In the pipeline's definition: `step` appends, so reopening the class would add three more steps.
|
|
111
112
|
class ArticleCardPipeline < RubyLLM::Contract::Pipeline::Base
|
|
112
113
|
step SummarizeArticle, as: :summarize, model: "gpt-4.1-mini"
|
|
113
114
|
step GenerateHashtags, as: :tag, model: "gpt-4.1-nano"
|
|
@@ -123,13 +124,14 @@ result = ArticleCardPipeline.run(article_text, timeout_ms: 30_000)
|
|
|
123
124
|
|
|
124
125
|
## Pipeline eval
|
|
125
126
|
|
|
126
|
-
Pipeline `run_eval` matches only the **final step's output** against `expected:` (`BuildArticleCard` here). Intermediate step outputs (`SummarizeArticle`, `GenerateHashtags`) are not routed through `expected:` —
|
|
127
|
+
Pipeline `run_eval` matches only the **final step's output** against `expected:` (`BuildArticleCard` here). Intermediate step outputs (`SummarizeArticle`, `GenerateHashtags`) are not routed through `expected:` — an assertion on `:tone` there would silently never match, and one on `:hashtags` would check the card's hashtags, not `GenerateHashtags`'. For per-step assertions, use `step_expectations:`, keyed by the step's alias, or run evals on each step individually.
|
|
127
128
|
|
|
128
129
|
```ruby
|
|
129
130
|
ArticleCardPipeline.define_eval("e2e") do
|
|
130
131
|
add_case "ruby 3.4 release",
|
|
131
132
|
input: "Ruby 3.4 ships with frozen string literals by default and better YJIT...",
|
|
132
|
-
expected: { sentiment_icon: "🧠" }
|
|
133
|
+
expected: { sentiment_icon: "🧠" },
|
|
134
|
+
step_expectations: { summarize: { tone: "analytical" } }
|
|
133
135
|
end
|
|
134
136
|
|
|
135
137
|
report = ArticleCardPipeline.run_eval("e2e", context: { model: "gpt-4.1-mini" })
|
data/docs/guide/prompt_ast.md
CHANGED
|
@@ -23,7 +23,7 @@ Or just a plain string (wraps as a single user message):
|
|
|
23
23
|
prompt "Summarize this article for a UI card. {input}"
|
|
24
24
|
```
|
|
25
25
|
|
|
26
|
-
The AST is
|
|
26
|
+
The AST is frozen, diffable, and compares by value with `==`. Useful for snapshot testing and auditing prompt changes.
|
|
27
27
|
|
|
28
28
|
## Hash inputs with variable interpolation
|
|
29
29
|
|
|
@@ -54,7 +54,7 @@ class SummarizeArticle < RubyLLM::Contract::Step::Base
|
|
|
54
54
|
end
|
|
55
55
|
```
|
|
56
56
|
|
|
57
|
-
Every `{key}` in a
|
|
57
|
+
Every `{key}` in a node's text (a `section`'s body, not its name) is pulled from the input hash at run time. A key the input does not have is left in the prompt as literal `{key}` text, so declare the keys in a `Types::Hash.schema(...)` input type when a missing one should stop the call.
|
|
58
58
|
|
|
59
59
|
> **Not the same as `RubyLLM::Agent.inputs`.** `Step.input_type` is a *runtime type check* on the positional argument passed to `run(input)` — a violating input returns `status: :input_error` instead of calling the model. `RubyLLM::Agent.inputs` is a list of *named template locals* injected into ERB instructions. They solve different problems (validation vs templating) and can coexist on the same project.
|
|
60
60
|
|
|
@@ -131,7 +131,7 @@ RubyLLM 2.1 emits its own events and OpenTelemetry spans for every call. To grou
|
|
|
131
131
|
RubyLLM::Contract.configure { |c| c.workflow_instrumentation = true }
|
|
132
132
|
```
|
|
133
133
|
|
|
134
|
-
A pipeline then runs as one `RubyLLM.workflow` named after its class, with each step as `workflow.step(alias)`; a step run on its own is a workflow named after its class. RubyLLM's events carry `workflow_name` and `workflow_step_name`, and a workflow your code opened around the call becomes the parent. Off by default. In a Rails app RubyLLM also records each call in its usage ledger when configured; that figure and `trace[:cost]` are the same RubyLLM price, unless `CostCalculator.register_model`
|
|
134
|
+
A pipeline then runs as one `RubyLLM.workflow` named after its class, with each step as `workflow.step(alias)`; a step run on its own is a workflow named after its class. RubyLLM's events carry `workflow_name` and `workflow_step_name`, and a workflow your code opened around the call becomes the parent. Off by default. In a Rails app RubyLLM also records each call in its usage ledger when configured; for a single attempt that figure and `trace[:cost]` (rounded to six decimals) are the same RubyLLM price, unless `CostCalculator.register_model` sets a price for the model and the provider did not report what it billed. With a retry chain each attempt is its own RubyLLM call, while `trace[:cost]` sums the attempts it could price (`trace[:cost_complete]` says whether it priced them all).
|
|
135
135
|
|
|
136
136
|
## 5. Testing — RSpec and Minitest
|
|
137
137
|
|
|
@@ -191,7 +191,7 @@ class ArticlesController < ApplicationController
|
|
|
191
191
|
end
|
|
192
192
|
```
|
|
193
193
|
|
|
194
|
-
When `retry_policy` exhausts and all models fail, `result.failed?` is true but `result.parsed_output` still contains the last attempt's output — useful for logging what the model *did* return before the validate rejected it.
|
|
194
|
+
When `retry_policy` exhausts and all models fail, `result.failed?` is true but `result.parsed_output` still contains the last attempt's output when it parsed — useful for logging what the model *did* return before the validate rejected it. After a parse error it is `nil`; `result.raw_output` keeps the text.
|
|
195
195
|
|
|
196
196
|
## 7. CI gate — block regressions before merge
|
|
197
197
|
|
|
@@ -217,7 +217,7 @@ Then wire it in GitHub Actions:
|
|
|
217
217
|
run: bundle exec rake ruby_llm_contract:eval
|
|
218
218
|
```
|
|
219
219
|
|
|
220
|
-
The job fails when a previously-passing eval case now fails, when
|
|
220
|
+
The job fails when a previously-passing eval case now fails, when an eval's average score drops below `minimum_score`, or when total cost exceeds the cap. The first needs a baseline: with `save_baseline = false` the task creates none, so save one (`report.save_baseline!`, or a run with `save_baseline = true`) and commit it. That is the signal that blocks a prompt regression or an accidental model upgrade from shipping.
|
|
221
221
|
|
|
222
222
|
**Two practical notes:**
|
|
223
223
|
|
|
@@ -9,13 +9,13 @@
|
|
|
9
9
|
| What you write | Where it lives |
|
|
10
10
|
|---|---|
|
|
11
11
|
| `model`, `temperature`, `schema`, `instructions`, `thinking` | covered by both — same idea, different DSL surface |
|
|
12
|
-
| `tools` | only in `RubyLLM::Agent`
|
|
12
|
+
| `tools` | only in `RubyLLM::Agent` - each Step attempt is one call without tools |
|
|
13
13
|
| `validate :rule do ... end` business invariants on output | only in `ruby_llm-contract` |
|
|
14
|
-
| `retry_policy escalate
|
|
15
|
-
| `max_cost` / `max_input`
|
|
14
|
+
| `retry_policy { escalate ... }` model escalation on validation failure | only here (different from RubyLLM's network-level retry) |
|
|
15
|
+
| `max_cost` / `max_input` pre-flight refusal, `max_output` reply cap | only here |
|
|
16
16
|
| `define_eval` + baseline regression + `compare_models` + `optimize_retry_policy` | only here; RubyLLM 2.1's `RubyLLM::Evaluation` grades answers (see below) but keeps no baseline and has no cost gate |
|
|
17
17
|
| Pipeline composition with `step SomeStep, as: :alias` | only here; `RubyLLM.workflow` (2.1) groups calls for tracing, without typed step outputs, fail-fast or budgets |
|
|
18
|
-
| `around_call`, named `observe` hooks with pass/fail recorded in
|
|
18
|
+
| `around_call`, named `observe` hooks with pass/fail recorded in `result.observations` | only here |
|
|
19
19
|
|
|
20
20
|
## Runtime relationship
|
|
21
21
|
|
|
@@ -50,7 +50,7 @@ A common pattern: simple ad-hoc prompts as `Agent`, contracts on the LLM feature
|
|
|
50
50
|
|
|
51
51
|
## On retry strategies
|
|
52
52
|
|
|
53
|
-
The retry-strategy framing in this gem favours `retry_policy escalate
|
|
53
|
+
The retry-strategy framing in this gem favours `retry_policy { escalate model_1, model_2 }` (model escalation, addresses model bias) over same-model `retry_policy attempts: N` (variance retry).
|
|
54
54
|
|
|
55
55
|
This is grounded in empirical comparison across PDF quiz generation, GSM8K math (n=30 + n=120), and multi-constraint schedule generation: same-model retry produced no useful lift for nano-class models on tasks with clear correctness criteria. Model escalation did move the needle when same-model retry could not.
|
|
56
56
|
|
|
@@ -12,7 +12,7 @@ Both gems sit on top of `ruby_llm`. The space they cover overlaps in vocabulary
|
|
|
12
12
|
| When it runs | After the LLM call returns, typically in a spec | Before the LLM result reaches your code |
|
|
13
13
|
| Where the output lives at evaluation time | Already in your variable, returned to caller | Still inside the gem's runner, not yet released |
|
|
14
14
|
| What "fail" means | Red test in CI | Trigger retry/escalate on a stronger model, or fail closed |
|
|
15
|
-
| Strongest features | Rich LLM-as-judge (faithful, relevant, hallucination, refusal, bias, toxicity, jailbreak, PII), red-team adversarial prompts, deterministic helpers (`assert_contains`, `assert_levenshtein`, …), RSpec/Minitest matchers, HTML/JUnit/GH reporters | Schema DSL with constraints, `validate` business rules, `retry_policy escalate
|
|
15
|
+
| Strongest features | Rich LLM-as-judge (faithful, relevant, hallucination, refusal, bias, toxicity, jailbreak, PII), red-team adversarial prompts, deterministic helpers (`assert_contains`, `assert_levenshtein`, …), RSpec/Minitest matchers, HTML/JUnit/GH reporters | Schema DSL with constraints, `validate` business rules, `retry_policy { escalate ... }` model escalation, `max_cost` pre-flight refusal, regression-eval framework (frozen dataset + baseline + min_score gate), pipeline composition |
|
|
16
16
|
| What it does NOT cover | No retry, no model escalation, no pre-flight cost cap, no contract layer between LLM and your code | No 10-judge LLM-as-judge catalog, no red-team generation, no rich deterministic assertion library; its test-framework matchers (`satisfy_contract`, `pass_eval`) gate contracts and evals, not individual outputs |
|
|
17
17
|
|
|
18
18
|
## Visual: where each gem sits in your call
|
|
@@ -41,7 +41,7 @@ your code ──► Step.run ──► LLM ──► [schema + validate]
|
|
|
41
41
|
status: :validation_failed
|
|
42
42
|
```
|
|
43
43
|
|
|
44
|
-
The output
|
|
44
|
+
The output reaches your code **as a passing result** only once the contract passes. A failed validation either fixes itself (a retry policy escalates to a stronger model) or the call fails closed with the last attempt's status, such as `:validation_failed` — your call site sees a deterministic failure status. Check `result.ok?` before using the output: a failed result still carries it in `parsed_output` and `raw_output`, for logging.
|
|
45
45
|
|
|
46
46
|
### Both together (Tribunal in CI → then Contract in prod)
|
|
47
47
|
|
|
@@ -67,7 +67,7 @@ Tribunal grades **a fixed set of cases on every PR** to catch quality regression
|
|
|
67
67
|
|
|
68
68
|
## When to use which
|
|
69
69
|
|
|
70
|
-
**Just Contract.** You ship LLM features whose output drives downstream code, money, or user trust. You need
|
|
70
|
+
**Just Contract.** You ship LLM features whose output drives downstream code, money, or user trust. You need a failure status instead of bad output (your code checks `result.ok?`), retry escalation, and budget refusal. You are happy to write your own `validate` blocks for content checks; you don't need a 10-judge catalog yet.
|
|
71
71
|
|
|
72
72
|
**Just Tribunal.** You have a stable production path you don't want to wrap, but you want a CI safety net that grades LLM output for faithfulness, hallucination, PII, jailbreak resistance, etc. You're testing a RAG pipeline or chatbot whose runtime is owned by other code.
|
|
73
73
|
|
|
@@ -111,6 +111,14 @@ These work today without any code changes in either gem — both use plain Ruby
|
|
|
111
111
|
class ChatReply < RubyLLM::Contract::Step::Base
|
|
112
112
|
prompt "Answer this question grounded in the docs:\n{input}"
|
|
113
113
|
|
|
114
|
+
output_schema do
|
|
115
|
+
string :answer
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
retry_policy do
|
|
119
|
+
escalate "gpt-4.1-mini", "gpt-4.1"
|
|
120
|
+
end
|
|
121
|
+
|
|
114
122
|
validate("no PII in answer") do |output, _input|
|
|
115
123
|
test_case = RubyLLM::Tribunal::TestCase.new(actual_output: output[:answer])
|
|
116
124
|
RubyLLM::Tribunal::Assertions.evaluate(:pii, test_case, {}).first == :pass
|
|
@@ -131,14 +139,17 @@ ChatReply.define_eval "rag_regression" do
|
|
|
131
139
|
actual_output: output[:answer],
|
|
132
140
|
context: ["Returns accepted within 30 days with receipt."]
|
|
133
141
|
)
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
142
|
+
status, details = RubyLLM::Tribunal::Assertions.evaluate(:faithful, tc, {})
|
|
143
|
+
if status == :error # the judge could not grade; details is the reason
|
|
144
|
+
RubyLLM::Contract::Eval::EvaluationResult.new(score: 0.0, passed: false, details: details)
|
|
145
|
+
else
|
|
146
|
+
RubyLLM::Contract::Eval::EvaluationResult.new(score: details[:score] || 0.0, passed: status == :pass)
|
|
147
|
+
end
|
|
137
148
|
}
|
|
138
149
|
end
|
|
139
150
|
```
|
|
140
151
|
|
|
141
|
-
Each case is graded by a Tribunal judge; baseline
|
|
152
|
+
Each case whose Step passes is graded by a Tribunal judge (a failed Step fails the case without grading); with a saved baseline and `pass_eval(...).without_regressions` (or a rake task with `fail_on_regression`), a case the judge passed before and fails now fails the build. A lower score on a case that still passes does not; add `with_minimum_score` for a score floor. You write the judge once, get the regression gate for free.
|
|
142
153
|
|
|
143
154
|
### Contract `Step` as Tribunal's `opts[:llm]` injection
|
|
144
155
|
|
|
@@ -157,7 +168,7 @@ This is a recipe, not a shipped adapter. Tribunal's `opts[:llm]` API is at v0.x
|
|
|
157
168
|
Three questions, three owners:
|
|
158
169
|
|
|
159
170
|
- **"Is this output good?"** — Tribunal, in CI, on outputs you already hold.
|
|
160
|
-
- **"What do we do when it isn't?"** — Contract, at runtime, before
|
|
171
|
+
- **"What do we do when it isn't?"** — Contract, at runtime, before your code treats an output as good (retry/escalate, or fail-closed with `Result(:validation_failed)`).
|
|
161
172
|
- **"What do we do when it _is_ good?"** — your application code. Once Contract returns `:ok`, you persist, render, hand off downstream. The gem deliberately doesn't touch the happy path; it owns failure semantics, not domain logic.
|
|
162
173
|
|
|
163
174
|
Use Tribunal, Contract, or both — whichever questions your application needs to answer.
|
data/docs/guide/testing.md
CHANGED
|
@@ -112,13 +112,16 @@ stub_steps(
|
|
|
112
112
|
SummarizeArticle => { response: { tldr: "...", takeaways: %w[a b c], tone: "neutral" } },
|
|
113
113
|
GenerateHashtags => { response: { tldr: "...", takeaways: %w[a b c],
|
|
114
114
|
tone: "neutral",
|
|
115
|
-
hashtags: %w[#ruby #release] } }
|
|
115
|
+
hashtags: %w[#ruby #release] } },
|
|
116
|
+
BuildArticleCard => { response: { headline: "...", summary: "...",
|
|
117
|
+
hashtags: %w[#ruby #release],
|
|
118
|
+
sentiment_icon: "😐" } }
|
|
116
119
|
) do
|
|
117
120
|
result = ArticleCardPipeline.run("article text")
|
|
118
121
|
end
|
|
119
122
|
```
|
|
120
123
|
|
|
121
|
-
**Global stub for all steps**
|
|
124
|
+
**Global stub for all steps** - it replaces the default adapter, so every Step that runs on it returns the same canned hash regardless of its `output_schema`. A run given its own `adapter:` and a Step stubbed with `stub_step` keep theirs. Useful for high-level integration specs where step output content doesn't matter; use `stub_steps` instead when you need per-step shapes.
|
|
122
125
|
|
|
123
126
|
```ruby
|
|
124
127
|
stub_all_steps(response: { default: true })
|
|
@@ -171,8 +174,8 @@ Evals run in one of two modes depending on how they're defined and what context
|
|
|
171
174
|
| Has `sample_response`? | `adapter:` in context? | Mode | API calls |
|
|
172
175
|
|---|---|---|---|
|
|
173
176
|
| Yes | No | **Offline** — uses `sample_response` as canned answer (a `model:` alone does not change that) | Zero |
|
|
174
|
-
| Yes | Yes | **Online** — ignores `sample_response`, calls real LLM | Real |
|
|
175
|
-
| No | Yes | **Online** — calls
|
|
177
|
+
| Yes | Yes | **Online** — ignores `sample_response`, calls the adapter you pass (a real LLM for `Adapters::RubyLLM`) | Real (zero with `Adapters::Test`) |
|
|
178
|
+
| No | Yes | **Online** — calls the adapter you pass | Real (zero with `Adapters::Test`) |
|
|
176
179
|
| No | No | **Online** with the default adapter, which `RubyLLM::Contract.configure` sets up; **Skipped** (`:skipped`, excluded from score) only when no adapter is configured | Real / Zero |
|
|
177
180
|
|
|
178
181
|
Default is offline. To force online, pass an adapter in context:
|
|
@@ -235,7 +238,7 @@ result.observations # => [{ description: "scores should differ", passed: false
|
|
|
235
238
|
|
|
236
239
|
`observe` runs only after validation passes. Failed observations are logged via the logger set in `RubyLLM::Contract.configure` (`c.logger`) — useful for "I want to know this happened without blocking the response".
|
|
237
240
|
|
|
238
|
-
> **Not the same as `Chat#
|
|
241
|
+
> **Not the same as `Chat#after_message` / `before_tool_call`.** RubyLLM exposes anonymous callbacks attached to a chat instance — `after_message` receives the raw `Message`, `before_tool_call` the tool call — and they have no inherent pass/fail concept. `Step.observe` is per-step, named with a description, runs against `parsed_output` (already JSON-parsed and schema-validated), and the pass/fail outcome is recorded in `result.observations`. The two can coexist: use Chat callbacks for transport/tool tracing, use `observe` for domain assertions you want captured in `result.observations`.
|
|
239
242
|
|
|
240
243
|
## Asserting on `around_call`
|
|
241
244
|
|
data/docs/guide/why.md
CHANGED
|
@@ -10,7 +10,7 @@ Below are four failure modes teams actually hit. If one looks familiar, the gem
|
|
|
10
10
|
|
|
11
11
|
A `SummarizeArticle` step produces `{ tldr: "...", takeaways: [...], tone: "analytical" }`. Schema passes. The TL;DR is 520 characters long and overflows the UI card. Or the takeaways are all variations of the same sentence. Or the article was a service-outage complaint and `tone` came back `"analytical"` instead of `"negative"` — so customer success's "critical feedback" filter never sees it.
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
A schema that checks only **shape** misses *length fits the card*, *takeaways are distinct*, and *tone matches content*. A length cap can go into the schema too (`max_length: 200`); the others are business rules, and without them you find out from a Slack thread or a support ticket.
|
|
14
14
|
|
|
15
15
|
```ruby
|
|
16
16
|
validate("TL;DR fits the card") { |o, _| o[:tldr].length <= 200 }
|
|
@@ -21,7 +21,7 @@ validate("negative tone requires concrete risk") do |o, _|
|
|
|
21
21
|
end
|
|
22
22
|
```
|
|
23
23
|
|
|
24
|
-
Wrong output
|
|
24
|
+
Wrong output comes back as a failed result, so a caller that checks `result.ok?` before `Article.update!` never persists it.
|
|
25
25
|
|
|
26
26
|
## Failure 2 — Silent prompt regression
|
|
27
27
|
|
|
@@ -62,7 +62,7 @@ Nano misclassifies the tone on the first attempt → contract rejects → mini g
|
|
|
62
62
|
|
|
63
63
|
**See it in 30 seconds:** `ruby examples/01_fallback_showcase.rb` — zero API keys required. The Test adapter simulates a tone/takeaways mismatch on the first attempt and a consistent sample on the retry, then prints the per-attempt trace.
|
|
64
64
|
|
|
65
|
-
`retry_policy` has three other shapes beyond cross-model escalation — same-model `attempts: 3` (absorbs sampling variance without paying for a stronger tier), `reasoning_effort` escalation (low → medium → high on one model), and cross-provider fallback (OpenAI → Anthropic — ruby_llm picks the provider from the model name; a local Ollama model needs `provider: :ollama` for the whole call, so it cannot be one tier of such a chain). `examples/06_retry_variants.rb` runs all three through the Test adapter with the trace printed.
|
|
65
|
+
`retry_policy` has three other shapes beyond cross-model escalation — same-model `attempts: 3` (absorbs sampling variance without paying for a stronger tier), `reasoning_effort` escalation (low → medium → high on one model), and cross-provider fallback (OpenAI → Anthropic — ruby_llm picks the provider from the model name, from 2.2 preferring the providers you configured; a local Ollama model needs `provider: :ollama` for the whole call, so it cannot be one tier of such a chain). `examples/06_retry_variants.rb` runs all three through the Test adapter with the trace printed.
|
|
66
66
|
|
|
67
67
|
## Failure 4 — Runaway cost and no fallback policy
|
|
68
68
|
|
|
@@ -92,7 +92,7 @@ The 40-page PDF returns `status: :limit_exceeded` — zero tokens spent. The 80/
|
|
|
92
92
|
| Schema-valid but logically wrong output | `validate(...) { |o, i| ... }` with 2-arity for cross-checks |
|
|
93
93
|
| Silent prompt regression after a tweak | `define_eval` + `pass_eval(...).without_regressions` in CI |
|
|
94
94
|
| Sampling variance on fixed-temperature models (gpt-5 / o-series) | Cross-field `validate(...)` + `retry_policy models: [...]` |
|
|
95
|
-
| Runaway cost on pathological inputs | `max_input`, `
|
|
95
|
+
| Runaway cost on pathological inputs | `max_input`, `max_cost` preflight + `max_output` reply cap |
|
|
96
96
|
| 80/20 traffic paying the premium model rate | `retry_policy` + `optimize_retry_policy` |
|
|
97
97
|
| Leaked placeholder / input echo / tone drift | `validate` with content and cross-input checks |
|
|
98
98
|
|
data/examples/00_basics.rb
CHANGED
|
@@ -55,7 +55,7 @@ r.status # => :ok
|
|
|
55
55
|
r.parsed_output # => {tldr: "...", takeaways: [...], tone: "analytical"}
|
|
56
56
|
|
|
57
57
|
# =============================================================================
|
|
58
|
-
# STEP 2
|
|
58
|
+
# STEP 2 - Add a business rule (validate) the schema above does not express
|
|
59
59
|
# Schema says "takeaways is an array of 3–5 strings". Nothing there says
|
|
60
60
|
# "uniqueness" or "TL;DR fits the card". That is what validate blocks are for.
|
|
61
61
|
# =============================================================================
|
|
@@ -204,5 +204,7 @@ puts
|
|
|
204
204
|
#
|
|
205
205
|
# Replace the Test adapter with Adapters::RubyLLM (see Step 8 in
|
|
206
206
|
# examples/00_basics.rb for the one-liner) and this exact same code runs
|
|
207
|
-
# against a real provider
|
|
207
|
+
# against a real provider. A local Ollama model is not a tier of this GPT
|
|
208
|
+
# chain: run it with provider: :ollama and retry_policy_override: nil, as
|
|
209
|
+
# Step 8 shows.
|
|
208
210
|
# =============================================================================
|
|
@@ -39,9 +39,14 @@ puts "Final model: #{result.trace[:model]}" # => "gpt-5-nano" (or mini/
|
|
|
39
39
|
puts "Latency: #{result.trace[:latency_ms]}ms" # real network time
|
|
40
40
|
puts "Tokens: #{result.trace[:usage]}" # real usage
|
|
41
41
|
puts "Cost: $#{result.trace[:cost]}" # sum across retries
|
|
42
|
-
|
|
42
|
+
if result.ok?
|
|
43
|
+
puts "TL;DR: #{result.parsed_output[:tldr]}"
|
|
44
|
+
else
|
|
45
|
+
puts "Errors: #{result.validation_errors.inspect}" # inspect escapes control characters in provider text
|
|
46
|
+
end
|
|
43
47
|
|
|
44
|
-
# Switch provider per call — ruby_llm resolves the provider from the model
|
|
48
|
+
# Switch provider per call — ruby_llm resolves the provider from the model
|
|
49
|
+
# name (from 2.2 it prefers the providers you configured).
|
|
45
50
|
# The step's retry_policy chain picks the model of every attempt, so turn it off
|
|
46
51
|
# for the call (retry_policy_override: nil) to run a model of your choice:
|
|
47
52
|
# (configure c.anthropic_api_key, or c.ollama_api_base for Ollama, first)
|
|
@@ -111,8 +111,9 @@ print_trace("reasoning escalation", SummarizeArticleReasoningEscalation.run("art
|
|
|
111
111
|
# VARIANT C — cross-provider fallback (OpenAI → Anthropic)
|
|
112
112
|
#
|
|
113
113
|
# When to use: the cheap tiers live on one provider and the last resort on
|
|
114
|
-
# another. ruby_llm
|
|
115
|
-
#
|
|
114
|
+
# another. ruby_llm picks the provider from the model name (from 2.2 it
|
|
115
|
+
# prefers the providers you configured), so a chain of models from its
|
|
116
|
+
# registry can cross providers.
|
|
116
117
|
#
|
|
117
118
|
# To run against real backends: configure ruby_llm for both providers
|
|
118
119
|
# (openai_api_key + anthropic_api_key) and swap the Test adapter for
|
|
@@ -146,6 +147,7 @@ print_trace("cross-provider", SummarizeArticleCrossProvider.run("article", conte
|
|
|
146
147
|
# 3. Cross-provider retry uses the same DSL — ruby_llm resolves the provider
|
|
147
148
|
# from the model name. Start cheapest, end on the most accurate model,
|
|
148
149
|
# whichever provider hosts it.
|
|
149
|
-
# 4. The per-attempt trace (model,
|
|
150
|
+
# 4. The per-attempt trace (model, status, cost, and config when an effort
|
|
151
|
+
# was sent) has the same shape across
|
|
150
152
|
# variants — your logging does not care which retry shape you picked.
|
|
151
153
|
# =============================================================================
|
data/examples/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Examples
|
|
2
2
|
|
|
3
|
-
Seven runnable examples, every one using the `SummarizeArticle` step from the [README](../README.md) — a Rails app turning article text into a UI card with TL;DR, takeaways, and tone. Zero API keys (
|
|
3
|
+
Seven runnable examples, every one using the `SummarizeArticle` step from the [README](../README.md) — a Rails app turning article text into a UI card with TL;DR, takeaways, and tone. Zero API keys (the offline examples use the Test adapter). Only `02_real_llm_minimal.rb` needs a provider key.
|
|
4
4
|
|
|
5
5
|
Pedagogical order: hook → activation → evolution → composition → quality → advanced.
|
|
6
6
|
|
|
@@ -27,6 +27,6 @@ ruby examples/04_summarize_and_translate.rb
|
|
|
27
27
|
ruby examples/05_eval_dataset.rb
|
|
28
28
|
ruby examples/06_retry_variants.rb
|
|
29
29
|
|
|
30
|
-
# Real LLM — requires
|
|
30
|
+
# Real LLM — requires OPENAI_API_KEY (comments at its end show Anthropic and Ollama):
|
|
31
31
|
ruby examples/02_real_llm_minimal.rb
|
|
32
32
|
```
|
|
@@ -7,10 +7,10 @@ module RubyLLM
|
|
|
7
7
|
module Step
|
|
8
8
|
class AdapterCaller
|
|
9
9
|
# Exceptions treated as :adapter_error (retryable when explicitly opted in).
|
|
10
|
-
# RubyLLM::Error covers provider
|
|
11
|
-
#
|
|
12
|
-
#
|
|
13
|
-
#
|
|
10
|
+
# RubyLLM::Error covers provider errors (auth, bad request, rate limit,
|
|
11
|
+
# server error, context length) and, from ruby_llm 2.2, network failures
|
|
12
|
+
# that outlast its retries (RubyLLM::ConnectionError). ruby_llm 2.1
|
|
13
|
+
# raises those as Faraday::Error (TimeoutError, ConnectionFailed).
|
|
14
14
|
# Anything else (NoMethodError, programmer ArgumentError from adapter
|
|
15
15
|
# code, etc.) propagates — those are bugs, not retry candidates.
|
|
16
16
|
ADAPTER_ERRORS = [::RubyLLM::Error, ::Faraday::Error].freeze
|
|
@@ -119,16 +119,27 @@ module RubyLLM
|
|
|
119
119
|
end
|
|
120
120
|
|
|
121
121
|
def handle_unknown_pricing(errors)
|
|
122
|
+
problem, fix = unknown_pricing_reason
|
|
122
123
|
if on_unknown_pricing == :warn
|
|
123
|
-
warn "[ruby_llm-contract] max_cost is configured but
|
|
124
|
-
"has no pricing data — cost limit not enforced"
|
|
124
|
+
warn "[ruby_llm-contract] max_cost is configured but #{problem} — cost limit not enforced"
|
|
125
125
|
else
|
|
126
|
-
errors << "max_cost is set but
|
|
127
|
-
"Register pricing via CostCalculator.register_model or set " \
|
|
126
|
+
errors << "max_cost is set but #{problem}. #{fix} or set " \
|
|
128
127
|
"on_unknown_pricing: :warn to proceed without cost checks."
|
|
129
128
|
end
|
|
130
129
|
end
|
|
131
130
|
|
|
131
|
+
# With no model the call would run RubyLLM's own default, which the
|
|
132
|
+
# contract does not read (RubyLLM.config.default_model does not count),
|
|
133
|
+
# so there is no price to check.
|
|
134
|
+
def unknown_pricing_reason
|
|
135
|
+
if model_name
|
|
136
|
+
["model '#{model_name}' has no pricing data", "Register pricing via CostCalculator.register_model"]
|
|
137
|
+
else
|
|
138
|
+
["no model is set for this Step, so the call has no known price",
|
|
139
|
+
"Set default_model in RubyLLM::Contract.configure, a model on the Step, or context: { model: }"]
|
|
140
|
+
end
|
|
141
|
+
end
|
|
142
|
+
|
|
132
143
|
# No request was sent, so usage is zero; the measured input goes
|
|
133
144
|
# alongside, and is left out when it could not be measured.
|
|
134
145
|
def build_limit_result(messages, estimated, errors, method: :heuristic)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ruby_llm-contract
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.5.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Justyna
|
|
@@ -27,16 +27,22 @@ dependencies:
|
|
|
27
27
|
name: ruby_llm
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
29
29
|
requirements:
|
|
30
|
-
- - "
|
|
30
|
+
- - ">="
|
|
31
|
+
- !ruby/object:Gem::Version
|
|
32
|
+
version: 2.1.1
|
|
33
|
+
- - "<"
|
|
31
34
|
- !ruby/object:Gem::Version
|
|
32
|
-
version: '
|
|
35
|
+
version: '3'
|
|
33
36
|
type: :runtime
|
|
34
37
|
prerelease: false
|
|
35
38
|
version_requirements: !ruby/object:Gem::Requirement
|
|
36
39
|
requirements:
|
|
37
|
-
- - "
|
|
40
|
+
- - ">="
|
|
41
|
+
- !ruby/object:Gem::Version
|
|
42
|
+
version: 2.1.1
|
|
43
|
+
- - "<"
|
|
38
44
|
- !ruby/object:Gem::Version
|
|
39
|
-
version: '
|
|
45
|
+
version: '3'
|
|
40
46
|
- !ruby/object:Gem::Dependency
|
|
41
47
|
name: schematist
|
|
42
48
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -197,7 +203,6 @@ files:
|
|
|
197
203
|
- lib/ruby_llm/contract/unknown_policy.rb
|
|
198
204
|
- lib/ruby_llm/contract/version.rb
|
|
199
205
|
- lib/ruby_llm/contract/workflow_scope.rb
|
|
200
|
-
- ruby_llm-contract.gemspec
|
|
201
206
|
homepage: https://github.com/justi/ruby_llm-contract
|
|
202
207
|
licenses:
|
|
203
208
|
- MIT
|
data/ruby_llm-contract.gemspec
DELETED
|
@@ -1,41 +0,0 @@
|
|
|
1
|
-
# frozen_string_literal: true
|
|
2
|
-
|
|
3
|
-
require_relative "lib/ruby_llm/contract/version"
|
|
4
|
-
|
|
5
|
-
Gem::Specification.new do |spec|
|
|
6
|
-
spec.name = "ruby_llm-contract"
|
|
7
|
-
spec.version = RubyLLM::Contract::VERSION
|
|
8
|
-
spec.authors = ["Justyna"]
|
|
9
|
-
|
|
10
|
-
spec.summary = "Contracts + Evals for ruby_llm"
|
|
11
|
-
spec.description = "Wraps RubyLLM::Chat with input/output contracts, business-rule validation, " \
|
|
12
|
-
"retry with model escalation on validation failure, pre-flight cost ceilings, " \
|
|
13
|
-
"and an evaluation framework. Sibling abstraction to RubyLLM::Agent — same " \
|
|
14
|
-
"niche (reusable class-based prompts), wider contract."
|
|
15
|
-
spec.homepage = "https://github.com/justi/ruby_llm-contract"
|
|
16
|
-
spec.license = "MIT"
|
|
17
|
-
spec.required_ruby_version = ">= 3.2.0"
|
|
18
|
-
|
|
19
|
-
spec.metadata["source_code_uri"] = spec.homepage
|
|
20
|
-
spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/main/CHANGELOG.md"
|
|
21
|
-
spec.metadata["documentation_uri"] = "#{spec.homepage}#readme"
|
|
22
|
-
spec.metadata["rubygems_mfa_required"] = "true"
|
|
23
|
-
|
|
24
|
-
spec.files = Dir.chdir(__dir__) do
|
|
25
|
-
# Internal trackers + dev configs excluded so the published gem
|
|
26
|
-
# contains only what adopters actually need at runtime.
|
|
27
|
-
excluded_files = %w[.rspec .simplecov .rubocop.yml .rubocop_todo.yml Gemfile Rakefile]
|
|
28
|
-
`git ls-files -z`.split("\x0").reject do |f|
|
|
29
|
-
(File.expand_path(f) == __FILE__) ||
|
|
30
|
-
f.start_with?("spec/", "docs/ideas/", "doc/", ".ai/", ".claude/", ".git",
|
|
31
|
-
".revive/", "gemfiles/") ||
|
|
32
|
-
excluded_files.include?(f)
|
|
33
|
-
end
|
|
34
|
-
end
|
|
35
|
-
spec.require_paths = ["lib"]
|
|
36
|
-
|
|
37
|
-
spec.add_dependency "dry-types", "~> 1.7"
|
|
38
|
-
spec.add_dependency "ruby_llm", "~> 2.0"
|
|
39
|
-
# schematist is what ruby_llm-schema became; ruby_llm 2.x depends on it too.
|
|
40
|
-
spec.add_dependency "schematist", "~> 1.1"
|
|
41
|
-
end
|