ruby_llm-contract 1.1.1 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 390af91f8200f1098440872d22515fa6232888d13d21a149c0b11f0bf3c15d48
-  data.tar.gz: d39d8a0c2e8914d08e4d996e6a3eb4d2c53613304cabcc6e077af2d9963ad4f5
+  metadata.gz: fead48c5516f8e0aa738d3ab1633ca2d07f7173aba6466a6d29e37634263b857
+  data.tar.gz: 4ae4c314cd6365c5f5de455f9dc5ddc0741abd26fa751804b22c5ffded0e6cb1
 SHA512:
-  metadata.gz: 81e03a20720536700acfe5bc840720a84675ca738e248fbdb490dac1dc63479c93924923d16cb28433329b53d24b796a42332736d29ec6f4d42a41240dfd4367
-  data.tar.gz: a433acb982c831c7ee1e2db5675135c4a42762d75c78e53708673b8ca50f5f165fe2cba9a088828b95738f24f29afb01e2033f694e92c69182662a6307492aa4
+  metadata.gz: 9b66df1e490e238fdacc84a728b874a008b80c8b44cdd882b444a84389bb77996ee6132efb6adaad79c91a04bee30eb7351241a4989ff1a4ed5df8d7a6cff619
+  data.tar.gz: a055fd3d06ee4fa9591fa2a6cfed6872142febd950fe3a455daa6e588f04e157a585701456f4815483ecc3accb803cf3bbcffa9308c6dfa5011c00f6a7143274

data/CHANGELOG.md

@@ -1,5 +1,108 @@
 # Changelog
 
+## 1.2.0 (2026-10-10)
+
+Everything new is opt-in, except the temperature fix below.
+
+### Added
+
+- **`token_count :exact`** on a step: `max_input` and `max_cost` measure the input with
+  the provider's count (RubyLLM's `chat.count_tokens`: OpenAI Responses, Anthropic,
+  Gemini, Vertex AI, Bedrock Converse) instead of the chars/4 heuristic, attachments
+  included, so `attachment_token_estimate` is not needed. One extra request per attempt.
+  A provider without the endpoint, a failed count or an adapter without `count_tokens`
+  refuses the call (`:limit_exceeded`) rather than falling back to the heuristic; the
+  `Test` adapter counts with the heuristic. What a provider counts beyond prompt and
+  attachments is its own (Gemini's count leaves the output schema out).
+  `Adapters::RubyLLM#count_tokens`.
+- **`on_incomplete_output :refuse`** on a step: a response the provider stopped at a token
+  limit fails as `:output_truncated`, one stopped by a content filter as
+  `:content_filtered`, before validation, so it is never `:ok`. Both statuses are outside
+  the default `retry_on`. The default, `:accept`, keeps 1.1.2's behaviour.
+- **`retry_on` takes exception classes**: `retry_on :validation_failed,
+  RubyLLM::RateLimitError` retries an adapter error only of that class or a subclass;
+  `:adapter_error` still retries any. An adapter error's trace (and attempt) has
+  `error_class`.
+- **`provider_options`** on a step (inherited, `:default` resets) and per call
+  (`context: { provider_options: }`, merged over the step's), passed to RubyLLM's
+  `with_provider_options`. Keys the step controls raise `ArgumentError`.
+- **`workflow_instrumentation`** in `RubyLLM::Contract.configure`: a pipeline runs as one
+  `RubyLLM.workflow` with a `workflow.step` per alias, a step on its own as a workflow
+  named after its class, so RubyLLM's events and OpenTelemetry spans carry the step.
+- **`register_model` takes cache prices**: `cache_read_per_1m:` and `cache_write_per_1m:`,
+  both optional.
+- Guides: retry by error class, cut-off responses, exact token counts, local and custom
+  models (Ollama), provider options, workflow instrumentation, and `RubyLLM::Judge` (2.1)
+  in the LLM-judge guide. The README lists the new options and answers how to run local
+  models.
+
+### Changed
+
+- **A temperature is no longer sent to a model RubyLLM's registry marks as taking none**
+  (OpenAI's o-series and gpt-5 family). RubyLLM 1.x quietly sent 1.0 to those models;
+  2.x sends the value as given, so a step with `temperature` that escalated to one failed
+  with an adapter error. The adapter now leaves it out and warns once per model; a model
+  the registry does not know still gets it.
+- **`retry_on` raises on a condition it cannot match.** A status given as a String (or
+  any value that is neither a Symbol nor an exception class) never matched a result, so
+  it silently turned retries off; it now raises `ArgumentError` when the step is defined.
+
+## 1.1.2 (2026-10-10)
+
+### Changed - costs and gates may move
+
+- **Costs now count the prompt cache, and may go up.** RubyLLM 2.x reports cached prompt
+  tokens apart from `tokens.input`, and the adapter read only input and output: 1,000
+  uncached + 9,000 cached + 200 output tokens on gpt-4.1-mini came out at $0.00072
+  instead of $0.00162. `trace[:usage][:input_tokens]` is now the whole prompt and
+  `trace[:cost]` is the cost RubyLLM puts on the response: the price of the provider the
+  call went to, cache and long-context prices, the amount the provider reported when it
+  reports one, and RubyLLM's own retries of the request. Eval totals, history and
+  `compare_models` figures can rise against older baselines; baselines compare scores,
+  not costs, so they do not regress on this.
+- **A call without token counts is no longer free.** When the provider sends no usage
+  (for the call or for any of RubyLLM's attempts at it), the counts still read 0 but
+  `trace[:usage_complete]` is `false` and, unless the provider reported what it billed,
+  the cost is unknown, so the eval cost gates refuse (opt out with
+  `on_unknown_pricing: :warn`).
+- **A missing price is unknown, not $0.** A model priced for input but not output has no
+  cost instead of a partial one, so `max_cost` refuses it pre-flight. A call that wrote
+  to the prompt cache on a model without a cache-write price has an unknown cost
+  afterwards (pre-flight estimates do not predict cache writes).
+
+### Fixed
+
+- `CostCalculator` prices RubyLLM's models through `Model#cost_for`, so pre-flight
+  estimates use cache and long-context prices, and the price of the provider the call
+  goes to (12 model ids carry different prices per provider). `register_model` keeps
+  overriding the price for its id; cache reads are charged at its input price, cache
+  writes have no price.
+- `compare_models` no longer names a model with unpriced cases the cheapest
+  (`best_for`), gives it an infinite `cost_per_point`, and `recommend` neither ranks it
+  on cost nor promises savings against it.
+- `single_shot_cost` reported a later attempt's cost when the first attempt was unpriced.
+- The step log printed an unknown cost as `$0.000000`; it now prints `cost=unknown`.
+- Docs no longer say RubyLLM has no evaluation framework: `relation_to_agent.md` now
+  places `RubyLLM::Evaluation` (2.1) next to `define_eval`. The README and the getting
+  started guide describe what a trace's usage and cost count, and the README's 1.x
+  promise now says new trace keys may be added.
+
+### Added
+
+- `trace[:usage]` breakdown keys when reported: `cache_read_tokens`, `cache_write_tokens`,
+  `thinking_tokens` (parts of `input_tokens` / `output_tokens`, not extra tokens).
+- `trace[:finish_reason]`, also on each retry attempt. A failed result whose response
+  stopped at a token limit or a content filter gets a last validation error saying so,
+  with the `max_output` the call ran with. Statuses and retries are unchanged.
+- An adapter may report its own cost: `Adapters::Response.new(content:, usage:, cost:,
+  cost_complete:, usage_complete:, finish_reason:)`, where `cost: nil` means unknown.
+  A response without `cost:` is priced from `usage` as before.
+- `provider:` on `CostCalculator.calculate`, `CostCalculator.find_model`,
+  `estimate_cost` and `estimate_eval_cost`.
+- Baselines and history mark unpriced cases and runs with `cost_unknown: true`.
+- A pipeline with a `token_budget` warns when a step's provider left out token counts;
+  `Pipeline::Trace#usage_complete?`.
+
 ## 1.1.1 (2026-10-10)
 
 ### Fixed

data/README.md

@@ -91,6 +91,8 @@ result.trace[:attempts]
 #    ]
 ```
 
+`usage[:input_tokens]` is the whole prompt, prompt-cache tokens included (broken out as `cache_read_tokens` / `cache_write_tokens` when the provider reports them), and `cost` is what RubyLLM prices the response at: the provider's price, cache and long-context rates, or the amount the provider reported. A cost the gem cannot fully price is marked unknown rather than counted as $0 - see [what a trace's usage and cost count](docs/guide/getting_started.md#what-a-traces-usage-and-cost-count).
+
 If the response is malformed, the TL;DR overflows the card, or the takeaway count is off, the gem moves to the next step. This is model **escalation**, not a fallback list — each step is an independent config (`model`, `reasoning_effort`), so the retry policy spends more compute only when the cheaper one couldn't satisfy the contract.
 
 ### Add a CI gate in 6 lines
@@ -128,6 +130,11 @@ Everything below is optional — the example above is a complete step. Reach for
 - **[Find the cheapest viable fallback list](docs/guide/optimizing_retry_policy.md)** — empirically pick the cheapest model chain that still passes your evals.
 - **[A/B test prompts](docs/guide/eval_first.md)** — measure whether a new prompt is safe to ship before merging.
 - **[Budget caps](docs/guide/getting_started.md)** — refuse the request pre-flight when an estimate exceeds the limit.
+- **[Cost tracking](docs/guide/getting_started.md#what-a-traces-usage-and-cost-count)** - per-call cost with prompt-cache and per-provider prices; eval cost gates fail closed when a cost is unknown.
+- **[Exact token counts](docs/guide/getting_started.md#counting-input-tokens-exactly)** - `token_count :exact` checks `max_input` / `max_cost` against the provider's own count instead of a heuristic.
+- **[Cut-off responses](docs/guide/getting_started.md#responses-cut-off-by-a-token-limit)** - `on_incomplete_output :refuse` fails an answer the provider stopped at a token limit, even one that would validate.
+- **[Retry by error class](docs/guide/getting_started.md#retrying-provider-errors-by-class)** - `retry_on RubyLLM::RateLimitError` retries a rate limit without retrying an exhausted balance.
+- **[Provider options](docs/guide/getting_started.md#provider-options)** - send `service_tier`, `seed` and other provider settings from the step or per call.
 - **[Reasoning effort / thinking config](docs/guide/optimizing_retry_policy.md)** — Anthropic / OpenAI thinking configuration on the Step class.
 
 Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and per-step models.
@@ -136,7 +143,7 @@ Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and
 
 ## Relation to `RubyLLM::Agent`
 
-`Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly — Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate(...)` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
+`Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly - Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate(...)` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. RubyLLM 2.1's `RubyLLM::Evaluation` grades answers; `define_eval` guards against regressions and cost in CI - the mapping covers both. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
 
 ## Relation to `ruby_llm-tribunal`
 
@@ -153,7 +160,7 @@ Different layers, complementary. [`ruby_llm-tribunal`](https://github.com/Alqemi
 | [Why contracts?](docs/guide/why.md) | Recognise the four production failures the gem exists for |
 | [Relation to RubyLLM::Agent](docs/guide/relation_to_agent.md) | Sibling abstractions; what each adds; runtime call path; coexistence patterns |
 | [Relation to ruby_llm-tribunal](docs/guide/relation_to_tribunal.md) | Different layers (test framework vs runtime contract); visual flows; integration recipes |
-| [Getting Started](docs/guide/getting_started.md) | Walk the full feature set on one concrete step |
+| [Getting Started](docs/guide/getting_started.md) | Walk the full feature set on one concrete step, including what usage and cost count |
 | [Rails integration](docs/guide/rails_integration.md) | Directory, initializer, jobs, logging, specs, CI gate — 7 FAQs for Rails devs |
 | [Adopt in an existing Rails app](docs/guide/migration.md) | Replace raw `LlmClient.call` with a contract, Before/After |
 | [Prevent silent prompt regressions](docs/guide/eval_first.md) | Evals, baselines, CI gates that block quality drift |
@@ -173,7 +180,11 @@ Stable since **1.0.0**. Semver tracked; breaking changes flagged in [CHANGELOG](
 What 1.0 promises not to break inside the 1.x line: the documented DSL
 (`prompt`, `output_schema`, `validate`, `retry_policy`, `max_cost`, `define_eval`,
 pipelines), the `Result` and `trace` shapes including `trace[:usage]`'s
-`{ input_tokens:, output_tokens: }` keys, and the adapter interface. A new major
+`{ input_tokens:, output_tokens: }` keys, and the adapter interface. New keys may be
+added to both (1.1.2 added `cache_read_tokens`, `cache_write_tokens` and
+`thinking_tokens` to `trace[:usage]`, and `usage_complete`, `cost_complete` and
+`finish_reason` to the trace), and a figure may change when it was wrong: since 1.1.2
+`input_tokens` counts prompt-cache tokens. A new major
 version of a runtime dependency may require a new major version here — that is
 what 1.0.0 itself was.
 
@@ -185,6 +196,10 @@ what 1.0.0 itself was.
 
 **Where in a Rails app?** Default `app/contracts/`. The Railtie reloads `app/contracts/eval/` and `app/steps/eval/` in development; any autoloaded directory also works. See [Rails integration](docs/guide/rails_integration.md).
 
+**Costs went up, or an eval cost gate turned red, after upgrading to 1.1.2?** Earlier versions left prompt-cache tokens out of `input_tokens` and the cost, priced every model at its default provider, counted a missing price as $0, and treated a call without token counts as free. The [CHANGELOG](CHANGELOG.md) lists each change; `on_unknown_pricing: :warn` turns the unknown-cost refusal into a warning.
+
+**Local models (Ollama)?** Point RubyLLM at the server (`c.ollama_api_base = "http://localhost:11434/v1"`) and run steps with `context: { provider: :ollama, model: "...", assume_model_exists: true }`. Local models have no price in RubyLLM's registry, so register one (`CostCalculator.register_model("gemma-fast:latest", input_per_1m: 0, output_per_1m: 0)`) or the eval cost gates treat their cost as unknown. Ollama has no token-counting endpoint, so leave `token_count` at `:estimate`. See [local and custom models](docs/guide/getting_started.md#local-and-custom-models).
+
 **Upgraded from pre-0.10.0 and getting `:limit_exceeded` with attachments?** Multimodal contracts with `max_cost`/`max_input` need `attachment_token_estimate`. See [multimodal input guide](docs/guide/multimodal_input.md#cost-attachment_token_estimate-is-required) for setup, fail-closed behaviour, and `on_unknown_attachment_size :warn` opt-out.
 
 ## License

data/docs/guide/getting_started.md

@@ -65,6 +65,32 @@ result.trace[:attempts]
 
 If the whole chain exhausts, `result.status` is the status of the last attempt (`:validation_failed` or `:parse_error`) and `result.parsed_output` is the last attempt's output. The caller decides what to do — ship it anyway, fall back to a template, or raise.
 
+### Retrying provider errors by class
+
+A provider error that RubyLLM's own retries could not get past ends as `:adapter_error`, with the exception class in `trace[:error_class]` (e.g. `"RubyLLM::RateLimitError"`). Adapter errors are not retried by default. List exception classes in `retry_on` to retry only those, typically together with `escalate` so the next attempt goes to another model:
+
+```ruby
+retry_policy do
+  escalate "gpt-4.1-mini", "claude-haiku-4-5"
+  retry_on :validation_failed, :parse_error, RubyLLM::RateLimitError, RubyLLM::OverloadedError
+end
+```
+
+A class matches its subclasses; `:adapter_error` in the list retries every adapter error. As with statuses, the list replaces the defaults. Switching to a fallback model on transport errors inside one call is RubyLLM's `chat.with_fallbacks`; the contract's adapter does not call it, so fallbacks across models belong in `escalate`.
+
+### Responses cut off by a token limit
+
+When a provider stops at a token limit or a content filter, `trace[:finish_reason]` says so (`:max_tokens`, `:content_filter`), and a failed result gets a last validation error naming it. A cut-off answer that still parses and validates stays `:ok` unless the step refuses it:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  max_output 400
+  on_incomplete_output :refuse   # default :accept
+end
+```
+
+With `:refuse`, such a response fails before validation as `:output_truncated` (token limit) or `:content_filtered`, so `validate` blocks and observers never see it. Raw output, usage and cost stay on the result. Neither status is retried by default; `retry_on :output_truncated` opts in (useful when a later attempt has a larger `max_output`). Anthropic reports an exhausted context window as a token-limit stop too, so `:output_truncated` means "stopped on tokens", not necessarily "hit max_output".
+
 ### Per-attempt reasoning effort
 
 `models:` accepts config hashes as well as model-name strings, so a fallback can "try harder" (more reasoning) on retry, not just switch model:
@@ -173,7 +199,48 @@ max_cost 0.01, on_unknown_pricing: :warn
 
 Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
 
-The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. The check sees only reported tokens: an adapter that returns no token counts looks like a zero-token run.
+The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. Since 1.1.2 a real call whose provider sent no token counts is flagged too, instead of passing as a free zero-token run.
+
+### What a trace's usage and cost count
+
+`trace[:usage][:input_tokens]` is the whole prompt the model took in, including tokens served from or written to the provider's prompt cache. When the provider reports them, the trace adds a breakdown: `cache_read_tokens`, `cache_write_tokens` and `thinking_tokens`. These are parts of `input_tokens` / `output_tokens`, not extra tokens; reasoning tokens are already in `output_tokens` when the provider bills them as output, and a model with a separate reasoning price is charged that price for them.
+
+```ruby
+result.trace[:usage]
+# => { input_tokens: 10_000, output_tokens: 200, cache_read_tokens: 9_000 }
+result.trace[:cost]    # => 0.00162  (gpt-4.1-mini: cache reads at the cache price)
+```
+
+With the RubyLLM adapter, `trace[:cost]` is the cost RubyLLM puts on the response (`response.cost.total`): the price of the provider the call went to, cache and long-context prices, and the amount the provider reported when it reports one (OpenRouter, for example). A price set with `CostCalculator.register_model` still overrides it for that model id, unless the provider reported what it billed; there, cache reads are charged at the input price and a call that wrote to the cache has no price, since cache writes can cost more than input.
+
+When a cost does not cover the whole call, `result.trace.cost_unknown?` is true and the eval gates above refuse. That happens when pricing is missing for a token category the call used, or when the provider sent no token counts for the call or for one of RubyLLM's attempts at it (`trace[:usage_complete]` is then `false` and missing counts read as 0) and did not report what it billed either. On a retried step `trace[:cost]` is the sum of the attempts that could be priced, and `cost_unknown?` is true if any attempt could not. `max_cost` itself is a pre-flight check on an estimate; it does not inspect the finished call.
+
+A custom adapter returning `Response.new(content:, usage:)` is priced from `usage` as before. To report its own cost, pass `cost:` - `nil` means unknown - with `cost_complete:` and `usage_complete:`.
+
+### Counting input tokens exactly
+
+`max_input` and `max_cost` measure the input with a chars/4 heuristic (±30%). With `token_count :exact` they ask the provider instead:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  token_count :exact   # default :estimate
+  max_input 20_000
+end
+```
+
+The count comes from RubyLLM's `chat.count_tokens` (OpenAI Responses, Anthropic, Gemini, Vertex AI, Bedrock Converse) and covers the prompt and attachments, so `attachment_token_estimate` is not needed. What else a provider counts is its own: OpenAI's and Anthropic's counts include the output schema, Gemini's leaves it out. It costs one extra request per attempt, and RubyLLM leaves `provider_options` out of it. Where no count can be had - a provider without the endpoint (Ollama, for one), a failed request, or an adapter without `count_tokens` - the call is refused as `:limit_exceeded` with the reason, never measured with the heuristic under the name "exact". The `Test` adapter counts with the heuristic, since it has no provider to ask. The output side of `max_cost` is still an estimate (`max_output`, or the input size without one).
+
+### Local and custom models
+
+A local model (Ollama) or a fine-tuned one has no price in RubyLLM's registry, so its cost is unknown and the eval cost gates refuse. Register a price - zero for a model that costs nothing per token:
+
+```ruby
+RubyLLM::Contract::CostCalculator.register_model("gemma-fast:latest", input_per_1m: 0, output_per_1m: 0)
+RubyLLM::Contract::CostCalculator.register_model("ft:gpt-4o-custom",
+  input_per_1m: 3.0, output_per_1m: 6.0, cache_read_per_1m: 1.5, cache_write_per_1m: 3.75)
+```
+
+`register_model` sets a price for a model id, not a model RubyLLM can call: the step still needs `provider:` (and `assume_model_exists: true` for an id RubyLLM does not list). Cache prices are optional; without them cache reads are charged at the input price and a call that wrote to the cache has no price. A price does not make up for missing counts: a provider that sends no token counts still leaves the cost unknown.
 
 ### Preflight cost estimates
 
@@ -195,6 +262,24 @@ SummarizeArticle.estimate_eval_cost("regression",
 
 `estimate_cost` returns `nil` when pricing isn't registered. `estimate_eval_cost` silently treats unknown-pricing cases as `$0.00` and sums the rest — it does **not** fail closed the way `max_cost` does. Treat its output as a floor, not a guarantee; register pricing via `CostCalculator.register_model` before relying on it for budget decisions.
 
+## Provider options
+
+Settings RubyLLM has no method for go to the provider as-is through `provider_options`, on the step or per call:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  provider_options service_tier: "flex"
+end
+
+SummarizeArticle.run(text, context: { provider_options: { service_tier: "priority", seed: 7 } })
+```
+
+The call's keys win over the step's; a subclass replaces its parent's hash, and `provider_options :default` stops inheriting it. Keys the step itself sets (`model`, `temperature`, the max-token keys, the messages, and on a step with `output_schema` the structured-output keys `response_format` and `text`) raise `ArgumentError`, so a call cannot differ from what its limits and schema were checked against. Only top-level keys are checked. `compare_models` and `optimize_retry_policy` candidates do not carry provider options.
+
+## Temperature on reasoning models
+
+RubyLLM 1.x sent temperature 1.0 to OpenAI reasoning models (o-series, gpt-5), which reject other values; RubyLLM 2.x sends what it is given. A step with `temperature` therefore leaves it out for a model RubyLLM's registry marks as taking none, and warns once per model - so `temperature 0` with `escalate "gpt-4.1-mini", "gpt-5-mini"` still works. A model the registry does not know gets the temperature as set.
+
 ## `output_schema` vs `with_schema`
 
 `with_schema` in `ruby_llm` tells the provider to force a specific JSON structure. `output_schema` in this gem does the same thing (calls `with_schema` under the hood) **plus** validates the response client-side. Cheaper models sometimes ignore schema constraints — `with_schema` is a request; `output_schema` is a request plus verification.

data/docs/guide/llm_judge.md

@@ -195,6 +195,20 @@ What to do operationally:
 
 The drop is your signal to refine the judge's prompt, not to lower the gate.
 
+## `RubyLLM::Judge` (RubyLLM 2.1)
+
+RubyLLM 2.1 adds `RubyLLM::Judge` for typed questions - `probability`, `choice` and `score` - answered by **decision models**: OpenAI's judgment models, TypeSafe, or local decision models through Ollama. It does not run on ordinary chat models; those raise that they do not support judgments. Where you have such a model, a judge class can stand in for the judge Step in step 1 above:
+
+```ruby
+class SummaryFaithfulness < RubyLLM::Judge
+  probability :faithful, "Is every claim in the summary supported by the article?"
+end
+
+SummaryFaithfulness.judge(article_and_summary).faithful.probability   # => 0.93
+```
+
+Calibrate it against human labels exactly as in step 2; the threshold is still yours to set. On an ordinary chat model, keep the judge as a Contract Step - the pattern this guide builds.
+
 ## When to reach for Tribunal instead
 
 [`ruby_llm-tribunal`](https://github.com/Alqemist-labs/ruby_llm-tribunal) ships an off-the-shelf catalog of common LLM-as-judge assertions (`assert_faithful`, `assert_hallucination`, `assert_refusal`, `assert_no_pii`, etc.) — a shortcut when your check matches one of those domain-general categories. The methodology in this guide still applies: calibrate the judge against your human-labeled production data **before** trusting Tribunal's `default_threshold = 0.8`, refine the prompt when it over-flags, watch for the anti-patterns above. See [Relation to Tribunal](relation_to_tribunal.md) for the full positioning — what each gem documents (and doesn't), a concrete decision tree on catalog-vs-custom-judge, and three working integration patterns.

data/docs/guide/rails_integration.md

@@ -125,6 +125,14 @@ end
 
 Trace inspection in an admin UI: `result.trace[:attempts]` gives you per-attempt model, status, cost, latency — render it in a partial to debug production failures without re-running.
 
+RubyLLM 2.1 emits its own events and OpenTelemetry spans for every call. To group them by step, turn on workflow instrumentation in the initializer:
+
+```ruby
+RubyLLM::Contract.configure { |c| c.workflow_instrumentation = true }
+```
+
+A pipeline then runs as one `RubyLLM.workflow` named after its class, with each step as `workflow.step(alias)`; a step run on its own is a workflow named after its class. RubyLLM's events carry `workflow_name` and `workflow_step_name`, and a workflow your code opened around the call becomes the parent. Off by default. In a Rails app RubyLLM also records each call in its usage ledger when configured; that figure and `trace[:cost]` are the same RubyLLM price, unless `CostCalculator.register_model` overrides it for the model.
+
 ## 5. Testing — RSpec and Minitest
 
 Add to `spec/spec_helper.rb` (or `test_helper.rb`):

data/docs/guide/relation_to_agent.md

@@ -12,8 +12,8 @@
 | `validate :rule do ... end` business invariants on output | only in `ruby_llm-contract` |
 | `retry_policy escalate(...)` model escalation on validation failure | only here (different from RubyLLM's network-level retry) |
 | `max_cost` / `max_input` / `max_output` pre-flight refusal | only here |
-| `define_eval` + baseline regression + `compare_models` + `optimize_retry_policy` | only here (RubyLLM does not ship an evaluation framework) |
-| Pipeline composition with `step SomeStep, as: :alias` | only here (RubyLLM intentionally leaves workflows as plain Ruby) |
+| `define_eval` + baseline regression + `compare_models` + `optimize_retry_policy` | only here; RubyLLM 2.1's `RubyLLM::Evaluation` grades answers (see below) but keeps no baseline and has no cost gate |
+| Pipeline composition with `step SomeStep, as: :alias` | only here; `RubyLLM.workflow` (2.1) groups calls for tracing, without typed step outputs, fail-fast or budgets |
 | `around_call`, named `observe` hooks with pass/fail recorded in trace | only here |
 
 ## Runtime relationship
@@ -32,6 +32,12 @@ Step.run(input)
 
 This may change in a future release if upstream APIs make a layered design natural. The decision is not committed; it depends on adopter signal.
 
+## Relation to `RubyLLM::Evaluation`
+
+RubyLLM 2.1 ships `RubyLLM::Evaluation`: a dataset of cases run through your `perform` method, graded for correctness by an LLM or by your own criteria and Ruby assertions, with RSpec/Minitest integration and a report of verdicts, tokens and cost. Use it to answer "is this answer right?".
+
+`define_eval` answers a different question: did this prompt or model get worse than last time, and does it stay within budget? It keeps baselines and flags regressions, compares prompts (`compare_with`) and models (`compare_models`, `recommend`), runs offline with `sample_response`, and gates CI on score and cost. The two can share a project; an Evaluation's `perform` can call `Step.run`.
+
 ## Coexistence on the same project
 
 The two abstractions can live in the same Rails (or non-Rails) project. Pick one per use case:

data/lib/ruby_llm/contract/adapters/response.rb

@@ -3,16 +3,42 @@
 module RubyLLM
   module Contract
     module Adapters
+      # `content` and `usage` are the whole interface an adapter has to fill.
+      # The rest is optional accounting: an adapter that leaves `cost` out gets
+      # its usage priced from the registry, as before; one that passes `cost`
+      # (nil included) owns the figure, and `cost_complete` / `usage_complete`
+      # say whether it and the token counts cover the whole call.
       class Response
         include Concerns::DeepFreeze
 
-        attr_reader :content, :usage
+        attr_reader :content, :usage, :usage_complete, :cost_complete, :finish_reason
 
-        def initialize(content:, usage: {})
+        def initialize(content:, usage: {}, cost: Step::Trace::COST_UNSET, usage_complete: nil,
+                       cost_complete: nil, finish_reason: nil)
           @content = deep_dup_freeze(content)
           @usage = deep_dup_freeze(usage)
+          @cost = cost
+          @usage_complete = usage_complete
+          @cost_complete = cost_complete
+          @finish_reason = finish_reason
           freeze
         end
+
+        def cost
+          cost_provided? ? @cost : nil
+        end
+
+        def cost_provided?
+          !@cost.equal?(Step::Trace::COST_UNSET)
+        end
+
+        # Keyword arguments for Step::Trace.new. Omits what the adapter did not
+        # report, so a plain Response keeps the registry pricing path.
+        def trace_accounting
+          fields = { usage_complete: @usage_complete, cost_complete: @cost_complete,
+                     finish_reason: @finish_reason }.compact
+          cost_provided? ? fields.merge(cost: @cost) : fields
+        end
       end
     end
   end

data/lib/ruby_llm/contract/adapters/ruby_llm.rb

@@ -6,32 +6,59 @@ module RubyLLM
   module Contract
     module Adapters
       class RubyLLM < Base
+        # `with: nil` adds no attachment. The text is never nil (`fetch(:content, "")`),
+        # and RubyLLM raises only when both text and attachments are nil.
         def call(messages:, **options)
-          system_contents, conversation = partition_messages(messages)
-          conversation = fallback_conversation(system_contents, conversation)
-
-          chat = build_chat(options, system_contents)
-          add_history(chat, conversation[0..-2])
+          chat, text = prepared_chat(messages, options)
+          response = chat.ask(text, with: options[:attachment])
+          build_response(response, options[:model])
+        end
 
-          # `with: nil` is a documented no-op in RubyLLM (verified against
-          # 1.15.0: chat.rb:36-37 `build_content(message, nil)` -> content.rb:8-14
-          # `Content.new(text, nil)` keeps text-only path when attachments
-          # are empty; raise only fires when BOTH text and attachments are nil,
-          # and we always pass a non-nil string thanks to `&.fetch(:content, "")`).
-          response = chat.ask(
-            conversation.last&.fetch(:content, ""),
-            with: options[:attachment]
-          )
-          build_response(response)
+        # Input tokens of the request `call` would send, counted by the
+        # provider (`chat.count_tokens`), attachments included. `ask` is
+        # `ask_later` plus `complete` in RubyLLM, so the staged message is the
+        # one `call` sends. RubyLLM leaves provider_options out of the count.
+        # Raises RubyLLM::Error where the provider has no counting endpoint.
+        def count_tokens(messages:, **options)
+          chat, text = prepared_chat(messages, options)
+          chat.ask_later(text, with: options[:attachment]).count_tokens
+        rescue KeyError => e
+          # RubyLLM reads OpenAI's count with `fetch`; a reply without the
+          # field is a failed count, not a bug in the caller.
+          raise ::RubyLLM::Error, "count response without input_tokens (#{e.message})"
         end
 
         CHAT_OPTION_METHODS = {
-          temperature: :with_temperature,
           schema: :with_schema
         }.freeze
 
+        @dropped_temperature_models = Set.new
+        @dropped_temperature_lock = Mutex.new
+
+        # Warns once per provider and model, across threads and subclasses
+        # (called on this class, whose state subclasses do not inherit).
+        def self.warn_dropped_temperature(model)
+          key = "#{model.provider}:#{model.id}"
+          first = @dropped_temperature_lock.synchronize { @dropped_temperature_models.add?(key) }
+          return unless first
+
+          warn "[ruby_llm-contract] #{model.id} (#{model.provider}) does not accept temperature " \
+               "according to RubyLLM's model registry; the step's temperature is not sent"
+        end
+
         private
 
+        # The configured chat with the history added, and the text of the last
+        # message, which sending and counting both stage themselves.
+        def prepared_chat(messages, options)
+          system_contents, conversation = partition_messages(messages)
+          conversation = fallback_conversation(system_contents, conversation)
+
+          chat = build_chat(options, system_contents)
+          add_history(chat, conversation[0..-2])
+          [chat, conversation.last&.fetch(:content, "")]
+        end
+
         # When prompt has only system/section/rule nodes and no user message,
         # pop the last system message and use it as the user ask.
         def fallback_conversation(system_contents, conversation)
@@ -59,13 +86,13 @@ module RubyLLM
           CHAT_OPTION_METHODS.each do |key, method_name|
             chat.public_send(method_name, options[key]) if options[key]
           end
+          apply_temperature(chat, options[:temperature]) unless options[:temperature].nil?
 
           # Resolve thinking config from BOTH sources, with `:reasoning_effort`
           # taking precedence over `:thinking[:effort]`. This is the per-attempt
           # override path used by `retry_policy { escalate({model:, reasoning_effort:}) }`
           # — the attempt-specific effort must win over the class-level default.
-          # Forwarded provider-agnostically via `chat.with_thinking(**)` —
-          # available since RubyLLM 1.12 (gemspec enforces this minimum).
+          # Forwarded provider-agnostically via `chat.with_thinking(**)`.
           thinking_config = resolve_thinking_config(options)
           chat.with_thinking(**thinking_config) if thinking_config
 
@@ -73,6 +100,29 @@ module RubyLLM
           # replacement for this one passthrough. `reasoning_effort` is not
           # forwarded here — it goes through `with_thinking` above.
           chat.with_max_output_tokens(options[:max_tokens]) if options[:max_tokens]
+          apply_provider_options(chat, options)
+        end
+
+        def apply_provider_options(chat, options)
+          provider_options = options[:provider_options]
+          return if provider_options.nil? || provider_options.empty?
+
+          ProviderOptions.validate!(provider_options, schema: !options[:schema].nil?)
+          chat.with_provider_options(provider_options)
+        end
+
+        # RubyLLM 1.x set temperature to 1.0 for OpenAI reasoning models, which
+        # reject any other value; 2.x sends it as given. A step escalating from
+        # a sampling model to one of those would fail, so a temperature is left
+        # out where the registry says the model takes none. A model RubyLLM does
+        # not know (assume_model_exists) gets it as before.
+        def apply_temperature(chat, temperature)
+          model = chat.respond_to?(:model) ? chat.model : nil
+          if model.respond_to?(:metadata) && model.metadata.is_a?(Hash) && model.metadata[:temperature] == false
+            Adapters::RubyLLM.warn_dropped_temperature(model)
+          else
+            chat.with_temperature(temperature)
+          end
         end
 
         # Returns merged `{ effort:, budget: }` or nil. `options[:reasoning_effort]`
@@ -84,26 +134,84 @@ module RubyLLM
           base.empty? ? nil : base
         end
 
-        def build_response(response)
+        def build_response(response, model)
           content = response.content
           content = content.to_s unless content.is_a?(Hash) || content.is_a?(Array)
 
-          # This is the ONLY place upstream token counts are read. RubyLLM 2.0
-          # replaced `Message#input_tokens`/`#output_tokens` with a `Tokens` value
-          # object. The `{ input_tokens:, output_tokens: }` shape below is this
-          # gem's own public contract (Step::Trace#usage, cost calculation,
-          # documented in the README) and deliberately does NOT change.
+          # This is the ONLY place upstream token counts are read. The
+          # `{ input_tokens:, output_tokens: }` shape is this gem's public
+          # contract (Step::Trace#usage, README) and does not change; RubyLLM's
+          # own counts go into it as described in `usage_from`.
           tokens = response.tokens
+          usage_complete = every_attempt_counted?(response, tokens)
+          reported = tokens&.reported_cost
+          usage = usage_from(tokens)
+          cost = call_cost(response, model, usage, reported)
 
           Response.new(
             content: content,
-            usage: {
-              input_tokens: tokens&.input || 0,
-              output_tokens: tokens&.output || 0
-            }
+            usage: usage,
+            cost: cost,
+            usage_complete: usage_complete,
+            # A cost RubyLLM priced from partial counts covers part of the call;
+            # one the provider reported covers all of it.
+            cost_complete: !cost.nil? && (usage_complete || !reported.nil?),
+            finish_reason: response.finish_reason
           )
         end
 
+        # `response.tokens` sums RubyLLM's attempts at the request, skipping
+        # counts an attempt never got, so the sum alone can look complete.
+        # RubyLLM records each attempt; one that failed after the provider may
+        # have billed it keeps nil counts, one never sent or refused gets zeros.
+        def every_attempt_counted?(response, tokens)
+          entries = response.respond_to?(:ruby_llm_usage_entries) ? Array(response.ruby_llm_usage_entries) : []
+          return entries.all? { |entry| reported_both_counts?(entry.tokens) } if entries.any?
+
+          reported_both_counts?(tokens)
+        end
+
+        # Zero is a reported count (a full cache hit leaves uncached input at 0);
+        # nil is a count the provider did not send.
+        def reported_both_counts?(tokens)
+          !tokens.nil? && !tokens.input.nil? && !tokens.output.nil?
+        end
+
+        # RubyLLM 2.x counts `input` without the prompt-cache tokens, which it
+        # reports apart as `cache_read` / `cache_write`. Our input_tokens is the
+        # whole prompt, so they are added back and also kept as a breakdown.
+        # `output` already includes thinking tokens when the provider bills them
+        # as output, so `thinking_tokens` is a breakdown, never added. A count
+        # the provider did not report is 0 here; `usage_complete` says so.
+        def usage_from(tokens)
+          return { input_tokens: 0, output_tokens: 0 } unless tokens
+
+          cache_read = tokens.cache_read.to_i
+          cache_write = tokens.cache_write.to_i
+          usage = { input_tokens: tokens.input.to_i + cache_read + cache_write, output_tokens: tokens.output.to_i }
+          usage[:cache_read_tokens] = cache_read if cache_read.positive?
+          usage[:cache_write_tokens] = cache_write if cache_write.positive?
+          usage[:thinking_tokens] = tokens.thinking.to_i if tokens.thinking.to_i.positive?
+          usage
+        end
+
+        # RubyLLM prices the response itself: per provider, with cache and
+        # long-context prices, the amount the provider reported when it did,
+        # and its own retries of the request. It returns nil when it cannot
+        # price every attempt. A price set with `register_model` still wins
+        # for its id, unless the provider reported what it billed; it prices
+        # the summed counts, so it is complete only when every attempt was
+        # counted (`cost_complete` in build_response).
+        def call_cost(response, model, usage, reported)
+          if reported.nil? && CostCalculator.registered?(model)
+            CostCalculator.calculate(model_name: model, usage: usage)
+          else
+            response.cost.total&.round(6)
+          end
+        rescue StandardError
+          nil
+        end
+
         def partition_messages(messages)
           system_contents = []
           conversation = []

data/lib/ruby_llm/contract/adapters/test.rb

@@ -43,6 +43,12 @@ module RubyLLM
           end
         end
 
+        # There is no provider to ask, so `token_count :exact` steps measure
+        # their input with the same heuristic as `:estimate` under this adapter.
+        def count_tokens(messages:, **_options)
+          TokenEstimator.estimate(messages)
+        end
+
         def call(messages:, **_options) # rubocop:disable Lint/UnusedMethodArgument
           content = if @responses
                       c = @responses[@index] || @responses.last