ruby_llm-contract 1.1.0 → 1.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4620ae344425d6b9473b5265821d9bc16e25c7ade812f2c0d89a7926859a2e2e
-  data.tar.gz: 68565f25b5aa97b9804c9e91cec01cf0253602686e709f94ed26a07fe2bf3ce9
+  metadata.gz: 7ae19c7459f4999409748c604e8b80aa756e5b2333863c8cc16b4070bf57dff9
+  data.tar.gz: 3be220b3be5c0c313f4b3f5160da5416dc9be8c35629c0012d756e0d027aa646
 SHA512:
-  metadata.gz: 3c1b4e7ef0f06065bfecd68742c07e40c50e7f08c53760a24097ed5a7fe9eaa84dbf9796fab40dd1096791c623c14bdc48eb3a44e39ebd5858bbc21bd888c24c
-  data.tar.gz: b2f3aaa92a43c8e638bcbe30d46cf0629cca8f201416cf848fc2d9b93ed7d9a393e92973ec2e3dc69ce6951b0d4c436955d8a5fd976161f93c1fd587be437877
+  metadata.gz: 16bc0227706b8e1cc16f5d93c7771f82e7835c40c20888cec9163a9537d30f80a86f761c47fd10bd778df8f7023ce041bd0d574248950e11bdad5dd3ac4860da
+  data.tar.gz: 4d4374cc2d65aa555f30656c2475bae261b0af1be0656b7449c22c38467b7f1e32f7ba159d215adc17254d0c624f7144f91087644f578b5cb2727fc8f9087c91

data/CHANGELOG.md

@@ -1,5 +1,91 @@
 # Changelog
 
+## 1.1.2 (2026-10-10)
+
+### Changed - costs and gates may move
+
+- **Costs now count the prompt cache, and may go up.** RubyLLM 2.x reports cached prompt
+  tokens apart from `tokens.input`, and the adapter read only input and output: 1,000
+  uncached + 9,000 cached + 200 output tokens on gpt-4.1-mini came out at $0.00072
+  instead of $0.00162. `trace[:usage][:input_tokens]` is now the whole prompt and
+  `trace[:cost]` is the cost RubyLLM puts on the response: the price of the provider the
+  call went to, cache and long-context prices, the amount the provider reported when it
+  reports one, and RubyLLM's own retries of the request. Eval totals, history and
+  `compare_models` figures can rise against older baselines; baselines compare scores,
+  not costs, so they do not regress on this.
+- **A call without token counts is no longer free.** When the provider sends no usage
+  (for the call or for any of RubyLLM's attempts at it), the counts still read 0 but
+  `trace[:usage_complete]` is `false` and, unless the provider reported what it billed,
+  the cost is unknown, so the eval cost gates refuse (opt out with
+  `on_unknown_pricing: :warn`).
+- **A missing price is unknown, not $0.** A model priced for input but not output has no
+  cost instead of a partial one, so `max_cost` refuses it pre-flight. A call that wrote
+  to the prompt cache on a model without a cache-write price has an unknown cost
+  afterwards (pre-flight estimates do not predict cache writes).
+
+### Fixed
+
+- `CostCalculator` prices RubyLLM's models through `Model#cost_for`, so pre-flight
+  estimates use cache and long-context prices, and the price of the provider the call
+  goes to (12 model ids carry different prices per provider). `register_model` keeps
+  overriding the price for its id; cache reads are charged at its input price, cache
+  writes have no price.
+- `compare_models` no longer names a model with unpriced cases the cheapest
+  (`best_for`), gives it an infinite `cost_per_point`, and `recommend` neither ranks it
+  on cost nor promises savings against it.
+- `single_shot_cost` reported a later attempt's cost when the first attempt was unpriced.
+- The step log printed an unknown cost as `$0.000000`; it now prints `cost=unknown`.
+- Docs no longer say RubyLLM has no evaluation framework: `relation_to_agent.md` now
+  places `RubyLLM::Evaluation` (2.1) next to `define_eval`. The README and the getting
+  started guide describe what a trace's usage and cost count, and the README's 1.x
+  promise now says new trace keys may be added.
+
+### Added
+
+- `trace[:usage]` breakdown keys when reported: `cache_read_tokens`, `cache_write_tokens`,
+  `thinking_tokens` (parts of `input_tokens` / `output_tokens`, not extra tokens).
+- `trace[:finish_reason]`, also on each retry attempt. A failed result whose response
+  stopped at a token limit or a content filter gets a last validation error saying so,
+  with the `max_output` the call ran with. Statuses and retries are unchanged.
+- An adapter may report its own cost: `Adapters::Response.new(content:, usage:, cost:,
+  cost_complete:, usage_complete:, finish_reason:)`, where `cost: nil` means unknown.
+  A response without `cost:` is priced from `usage` as before.
+- `provider:` on `CostCalculator.calculate`, `CostCalculator.find_model`,
+  `estimate_cost` and `estimate_eval_cost`.
+- Baselines and history mark unpriced cases and runs with `cost_unknown: true`.
+- A pipeline with a `token_budget` warns when a step's provider left out token counts;
+  `Pipeline::Trace#usage_complete?`.
+
+## 1.1.1 (2026-10-10)
+
+### Fixed
+
+- **A failed rake eval gate printed "Eval suite FAILED: Eval suite FAILED".** The
+  message now reads "Eval suite FAILED: one or more evals did not pass".
+
+### Changed (internal, no API change)
+
+- Values that two or more places must agree on now have one owner, so one side can no
+  longer drift alone: the `:refuse`/`:warn` modes and their default for every
+  `on_unknown_*` option (`UnknownPolicy`), the Rails contract and eval dirs that the
+  Railtie ignores and `load_evals!` loads, the reload flag, the `:skipped` step status
+  (`CaseResult::SKIPPED_STATUS`), baseline and history file extensions, the context keys
+  forwarded to the adapter, the generic "passed"/"not passed" details the report hides,
+  and the optimize task's `MIN_SCORE` default, which now reads `Eval::DEFAULT_MIN_SCORE`.
+  The unknown-context-key warning lists the known keys in a different order, and in a
+  Rails app `load_evals!` now loads `app/contracts/eval` before `app/steps/eval` (the order
+  the Railtie and eager loading already used); this matters only if both dirs define an
+  eval of the same name on the same class, where the later file wins.
+
+### Tests
+
+- A real attachment through the RubyLLM 2.x adapter with HTTP intercepted (webmock):
+  pins the `input_image` part on the wire and the usage read back.
+- A minimal Rails app booted in a child process: pins that Zeitwerk ignores the
+  `eval/` dirs, that evals register after boot, and that the rake tasks appear.
+- `rubocop` is pinned to `~> 1.92.0`: CI resolves without a lockfile, so a floating
+  version let new cops land on CI before they ran locally.
+
 ## 1.1.0 (2026-10-10)
 
 ### Changed - may turn a green CI red

data/README.md

@@ -91,6 +91,8 @@ result.trace[:attempts]
 #    ]
 ```
 
+`usage[:input_tokens]` is the whole prompt, prompt-cache tokens included (broken out as `cache_read_tokens` / `cache_write_tokens` when the provider reports them), and `cost` is what RubyLLM prices the response at: the provider's price, cache and long-context rates, or the amount the provider reported. A cost the gem cannot fully price is marked unknown rather than counted as $0 - see [what a trace's usage and cost count](docs/guide/getting_started.md#what-a-traces-usage-and-cost-count).
+
 If the response is malformed, the TL;DR overflows the card, or the takeaway count is off, the gem moves to the next step. This is model **escalation**, not a fallback list — each step is an independent config (`model`, `reasoning_effort`), so the retry policy spends more compute only when the cheaper one couldn't satisfy the contract.
 
 ### Add a CI gate in 6 lines
@@ -128,6 +130,7 @@ Everything below is optional — the example above is a complete step. Reach for
 - **[Find the cheapest viable fallback list](docs/guide/optimizing_retry_policy.md)** — empirically pick the cheapest model chain that still passes your evals.
 - **[A/B test prompts](docs/guide/eval_first.md)** — measure whether a new prompt is safe to ship before merging.
 - **[Budget caps](docs/guide/getting_started.md)** — refuse the request pre-flight when an estimate exceeds the limit.
+- **[Cost tracking](docs/guide/getting_started.md#what-a-traces-usage-and-cost-count)** - per-call cost with prompt-cache and per-provider prices; eval cost gates fail closed when a cost is unknown.
 - **[Reasoning effort / thinking config](docs/guide/optimizing_retry_policy.md)** — Anthropic / OpenAI thinking configuration on the Step class.
 
 Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and per-step models.
@@ -136,7 +139,7 @@ Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and
 
 ## Relation to `RubyLLM::Agent`
 
-`Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly — Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate(...)` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
+`Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly - Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate(...)` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. RubyLLM 2.1's `RubyLLM::Evaluation` grades answers; `define_eval` guards against regressions and cost in CI - the mapping covers both. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
 
 ## Relation to `ruby_llm-tribunal`
 
@@ -153,7 +156,7 @@ Different layers, complementary. [`ruby_llm-tribunal`](https://github.com/Alqemi
 | [Why contracts?](docs/guide/why.md) | Recognise the four production failures the gem exists for |
 | [Relation to RubyLLM::Agent](docs/guide/relation_to_agent.md) | Sibling abstractions; what each adds; runtime call path; coexistence patterns |
 | [Relation to ruby_llm-tribunal](docs/guide/relation_to_tribunal.md) | Different layers (test framework vs runtime contract); visual flows; integration recipes |
-| [Getting Started](docs/guide/getting_started.md) | Walk the full feature set on one concrete step |
+| [Getting Started](docs/guide/getting_started.md) | Walk the full feature set on one concrete step, including what usage and cost count |
 | [Rails integration](docs/guide/rails_integration.md) | Directory, initializer, jobs, logging, specs, CI gate — 7 FAQs for Rails devs |
 | [Adopt in an existing Rails app](docs/guide/migration.md) | Replace raw `LlmClient.call` with a contract, Before/After |
 | [Prevent silent prompt regressions](docs/guide/eval_first.md) | Evals, baselines, CI gates that block quality drift |
@@ -173,7 +176,11 @@ Stable since **1.0.0**. Semver tracked; breaking changes flagged in [CHANGELOG](
 What 1.0 promises not to break inside the 1.x line: the documented DSL
 (`prompt`, `output_schema`, `validate`, `retry_policy`, `max_cost`, `define_eval`,
 pipelines), the `Result` and `trace` shapes including `trace[:usage]`'s
-`{ input_tokens:, output_tokens: }` keys, and the adapter interface. A new major
+`{ input_tokens:, output_tokens: }` keys, and the adapter interface. New keys may be
+added to both (1.1.2 added `cache_read_tokens`, `cache_write_tokens` and
+`thinking_tokens` to `trace[:usage]`, and `usage_complete`, `cost_complete` and
+`finish_reason` to the trace), and a figure may change when it was wrong: since 1.1.2
+`input_tokens` counts prompt-cache tokens. A new major
 version of a runtime dependency may require a new major version here — that is
 what 1.0.0 itself was.
 
@@ -185,6 +192,8 @@ what 1.0.0 itself was.
 
 **Where in a Rails app?** Default `app/contracts/`. The Railtie reloads `app/contracts/eval/` and `app/steps/eval/` in development; any autoloaded directory also works. See [Rails integration](docs/guide/rails_integration.md).
 
+**Costs went up, or an eval cost gate turned red, after upgrading to 1.1.2?** Earlier versions left prompt-cache tokens out of `input_tokens` and the cost, priced every model at its default provider, counted a missing price as $0, and treated a call without token counts as free. The [CHANGELOG](CHANGELOG.md) lists each change; `on_unknown_pricing: :warn` turns the unknown-cost refusal into a warning.
+
 **Upgraded from pre-0.10.0 and getting `:limit_exceeded` with attachments?** Multimodal contracts with `max_cost`/`max_input` need `attachment_token_estimate`. See [multimodal input guide](docs/guide/multimodal_input.md#cost-attachment_token_estimate-is-required) for setup, fail-closed behaviour, and `on_unknown_attachment_size :warn` opt-out.
 
 ## License

data/docs/guide/getting_started.md

@@ -173,7 +173,23 @@ max_cost 0.01, on_unknown_pricing: :warn
 
 Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
 
-The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. The check sees only reported tokens: an adapter that returns no token counts looks like a zero-token run.
+The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. Since 1.1.2 a real call whose provider sent no token counts is flagged too, instead of passing as a free zero-token run.
+
+### What a trace's usage and cost count
+
+`trace[:usage][:input_tokens]` is the whole prompt the model took in, including tokens served from or written to the provider's prompt cache. When the provider reports them, the trace adds a breakdown: `cache_read_tokens`, `cache_write_tokens` and `thinking_tokens`. These are parts of `input_tokens` / `output_tokens`, not extra tokens; reasoning tokens are already in `output_tokens` when the provider bills them as output, and a model with a separate reasoning price is charged that price for them.
+
+```ruby
+result.trace[:usage]
+# => { input_tokens: 10_000, output_tokens: 200, cache_read_tokens: 9_000 }
+result.trace[:cost]    # => 0.00162  (gpt-4.1-mini: cache reads at the cache price)
+```
+
+With the RubyLLM adapter, `trace[:cost]` is the cost RubyLLM puts on the response (`response.cost.total`): the price of the provider the call went to, cache and long-context prices, and the amount the provider reported when it reports one (OpenRouter, for example). A price set with `CostCalculator.register_model` still overrides it for that model id, unless the provider reported what it billed; there, cache reads are charged at the input price and a call that wrote to the cache has no price, since cache writes can cost more than input.
+
+When a cost does not cover the whole call, `result.trace.cost_unknown?` is true and the eval gates above refuse. That happens when pricing is missing for a token category the call used, or when the provider sent no token counts for the call or for one of RubyLLM's attempts at it (`trace[:usage_complete]` is then `false` and missing counts read as 0) and did not report what it billed either. On a retried step `trace[:cost]` is the sum of the attempts that could be priced, and `cost_unknown?` is true if any attempt could not. `max_cost` itself is a pre-flight check on an estimate; it does not inspect the finished call.
+
+A custom adapter returning `Response.new(content:, usage:)` is priced from `usage` as before. To report its own cost, pass `cost:` - `nil` means unknown - with `cost_complete:` and `usage_complete:`.
 
 ### Preflight cost estimates
 

data/docs/guide/relation_to_agent.md

@@ -12,8 +12,8 @@
 | `validate :rule do ... end` business invariants on output | only in `ruby_llm-contract` |
 | `retry_policy escalate(...)` model escalation on validation failure | only here (different from RubyLLM's network-level retry) |
 | `max_cost` / `max_input` / `max_output` pre-flight refusal | only here |
-| `define_eval` + baseline regression + `compare_models` + `optimize_retry_policy` | only here (RubyLLM does not ship an evaluation framework) |
-| Pipeline composition with `step SomeStep, as: :alias` | only here (RubyLLM intentionally leaves workflows as plain Ruby) |
+| `define_eval` + baseline regression + `compare_models` + `optimize_retry_policy` | only here; RubyLLM 2.1's `RubyLLM::Evaluation` grades answers (see below) but keeps no baseline and has no cost gate |
+| Pipeline composition with `step SomeStep, as: :alias` | only here; `RubyLLM.workflow` (2.1) groups calls for tracing, without typed step outputs, fail-fast or budgets |
 | `around_call`, named `observe` hooks with pass/fail recorded in trace | only here |
 
 ## Runtime relationship
@@ -32,6 +32,12 @@ Step.run(input)
 
 This may change in a future release if upstream APIs make a layered design natural. The decision is not committed; it depends on adopter signal.
 
+## Relation to `RubyLLM::Evaluation`
+
+RubyLLM 2.1 ships `RubyLLM::Evaluation`: a dataset of cases run through your `perform` method, graded for correctness by an LLM or by your own criteria and Ruby assertions, with RSpec/Minitest integration and a report of verdicts, tokens and cost. Use it to answer "is this answer right?".
+
+`define_eval` answers a different question: did this prompt or model get worse than last time, and does it stay within budget? It keeps baselines and flags regressions, compares prompts (`compare_with`) and models (`compare_models`, `recommend`), runs offline with `sample_response`, and gates CI on score and cost. The two can share a project; an Evaluation's `perform` can call `Step.run`.
+
 ## Coexistence on the same project
 
 The two abstractions can live in the same Rails (or non-Rails) project. Pick one per use case:

data/lib/ruby_llm/contract/adapters/response.rb

@@ -3,16 +3,42 @@
 module RubyLLM
   module Contract
     module Adapters
+      # `content` and `usage` are the whole interface an adapter has to fill.
+      # The rest is optional accounting: an adapter that leaves `cost` out gets
+      # its usage priced from the registry, as before; one that passes `cost`
+      # (nil included) owns the figure, and `cost_complete` / `usage_complete`
+      # say whether it and the token counts cover the whole call.
       class Response
         include Concerns::DeepFreeze
 
-        attr_reader :content, :usage
+        attr_reader :content, :usage, :usage_complete, :cost_complete, :finish_reason
 
-        def initialize(content:, usage: {})
+        def initialize(content:, usage: {}, cost: Step::Trace::COST_UNSET, usage_complete: nil,
+                       cost_complete: nil, finish_reason: nil)
           @content = deep_dup_freeze(content)
           @usage = deep_dup_freeze(usage)
+          @cost = cost
+          @usage_complete = usage_complete
+          @cost_complete = cost_complete
+          @finish_reason = finish_reason
           freeze
         end
+
+        def cost
+          cost_provided? ? @cost : nil
+        end
+
+        def cost_provided?
+          !@cost.equal?(Step::Trace::COST_UNSET)
+        end
+
+        # Keyword arguments for Step::Trace.new. Omits what the adapter did not
+        # report, so a plain Response keeps the registry pricing path.
+        def trace_accounting
+          fields = { usage_complete: @usage_complete, cost_complete: @cost_complete,
+                     finish_reason: @finish_reason }.compact
+          cost_provided? ? fields.merge(cost: @cost) : fields
+        end
       end
     end
   end

data/lib/ruby_llm/contract/adapters/ruby_llm.rb

@@ -13,16 +13,13 @@ module RubyLLM
           chat = build_chat(options, system_contents)
           add_history(chat, conversation[0..-2])
 
-          # `with: nil` is a documented no-op in RubyLLM (verified against
-          # 1.15.0: chat.rb:36-37 `build_content(message, nil)` -> content.rb:8-14
-          # `Content.new(text, nil)` keeps text-only path when attachments
-          # are empty; raise only fires when BOTH text and attachments are nil,
-          # and we always pass a non-nil string thanks to `&.fetch(:content, "")`).
+          # `with: nil` adds no attachment. The text is never nil (`fetch(:content, "")`),
+          # and RubyLLM raises only when both text and attachments are nil.
           response = chat.ask(
             conversation.last&.fetch(:content, ""),
             with: options[:attachment]
           )
-          build_response(response)
+          build_response(response, options[:model])
         end
 
         CHAT_OPTION_METHODS = {
@@ -64,8 +61,7 @@ module RubyLLM
           # taking precedence over `:thinking[:effort]`. This is the per-attempt
           # override path used by `retry_policy { escalate({model:, reasoning_effort:}) }`
           # — the attempt-specific effort must win over the class-level default.
-          # Forwarded provider-agnostically via `chat.with_thinking(**)` —
-          # available since RubyLLM 1.12 (gemspec enforces this minimum).
+          # Forwarded provider-agnostically via `chat.with_thinking(**)`.
           thinking_config = resolve_thinking_config(options)
           chat.with_thinking(**thinking_config) if thinking_config
 
@@ -84,26 +80,84 @@ module RubyLLM
           base.empty? ? nil : base
         end
 
-        def build_response(response)
+        def build_response(response, model)
           content = response.content
           content = content.to_s unless content.is_a?(Hash) || content.is_a?(Array)
 
-          # This is the ONLY place upstream token counts are read. RubyLLM 2.0
-          # replaced `Message#input_tokens`/`#output_tokens` with a `Tokens` value
-          # object. The `{ input_tokens:, output_tokens: }` shape below is this
-          # gem's own public contract (Step::Trace#usage, cost calculation,
-          # documented in the README) and deliberately does NOT change.
+          # This is the ONLY place upstream token counts are read. The
+          # `{ input_tokens:, output_tokens: }` shape is this gem's public
+          # contract (Step::Trace#usage, README) and does not change; RubyLLM's
+          # own counts go into it as described in `usage_from`.
           tokens = response.tokens
+          usage_complete = every_attempt_counted?(response, tokens)
+          reported = tokens&.reported_cost
+          usage = usage_from(tokens)
+          cost = call_cost(response, model, usage, reported)
 
           Response.new(
             content: content,
-            usage: {
-              input_tokens: tokens&.input || 0,
-              output_tokens: tokens&.output || 0
-            }
+            usage: usage,
+            cost: cost,
+            usage_complete: usage_complete,
+            # A cost RubyLLM priced from partial counts covers part of the call;
+            # one the provider reported covers all of it.
+            cost_complete: !cost.nil? && (usage_complete || !reported.nil?),
+            finish_reason: response.finish_reason
           )
         end
 
+        # `response.tokens` sums RubyLLM's attempts at the request, skipping
+        # counts an attempt never got, so the sum alone can look complete.
+        # RubyLLM records each attempt; one that failed after the provider may
+        # have billed it keeps nil counts, one never sent or refused gets zeros.
+        def every_attempt_counted?(response, tokens)
+          entries = response.respond_to?(:ruby_llm_usage_entries) ? Array(response.ruby_llm_usage_entries) : []
+          return entries.all? { |entry| reported_both_counts?(entry.tokens) } if entries.any?
+
+          reported_both_counts?(tokens)
+        end
+
+        # Zero is a reported count (a full cache hit leaves uncached input at 0);
+        # nil is a count the provider did not send.
+        def reported_both_counts?(tokens)
+          !tokens.nil? && !tokens.input.nil? && !tokens.output.nil?
+        end
+
+        # RubyLLM 2.x counts `input` without the prompt-cache tokens, which it
+        # reports apart as `cache_read` / `cache_write`. Our input_tokens is the
+        # whole prompt, so they are added back and also kept as a breakdown.
+        # `output` already includes thinking tokens when the provider bills them
+        # as output, so `thinking_tokens` is a breakdown, never added. A count
+        # the provider did not report is 0 here; `usage_complete` says so.
+        def usage_from(tokens)
+          return { input_tokens: 0, output_tokens: 0 } unless tokens
+
+          cache_read = tokens.cache_read.to_i
+          cache_write = tokens.cache_write.to_i
+          usage = { input_tokens: tokens.input.to_i + cache_read + cache_write, output_tokens: tokens.output.to_i }
+          usage[:cache_read_tokens] = cache_read if cache_read.positive?
+          usage[:cache_write_tokens] = cache_write if cache_write.positive?
+          usage[:thinking_tokens] = tokens.thinking.to_i if tokens.thinking.to_i.positive?
+          usage
+        end
+
+        # RubyLLM prices the response itself: per provider, with cache and
+        # long-context prices, the amount the provider reported when it did,
+        # and its own retries of the request. It returns nil when it cannot
+        # price every attempt. A price set with `register_model` still wins
+        # for its id, unless the provider reported what it billed; it prices
+        # the summed counts, so it is complete only when every attempt was
+        # counted (`cost_complete` in build_response).
+        def call_cost(response, model, usage, reported)
+          if reported.nil? && CostCalculator.registered?(model)
+            CostCalculator.calculate(model_name: model, usage: usage)
+          else
+            response.cost.total&.round(6)
+          end
+        rescue StandardError
+          nil
+        end
+
         def partition_messages(messages)
           system_contents = []
           conversation = []

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -16,13 +16,13 @@ module RubyLLM
           @file_sourced_evals ||= Set.new
           key = name.to_s
 
-          if @eval_definitions.key?(key) && !Thread.current[:ruby_llm_contract_reloading]
+          if @eval_definitions.key?(key) && !Contract.reloading?
             warn "[ruby_llm-contract] Redefining eval '#{key}' on #{self}. " \
                  "This replaces the previous definition."
           end
 
           @eval_definitions[key] = Eval::EvalDefinition.new(key, step_class: self, &)
-          @file_sourced_evals.add(key) if Thread.current[:ruby_llm_contract_reloading]
+          @file_sourced_evals.add(key) if Contract.reloading?
           Contract.register_eval_host(self)
           register_subclasses(self)
         end

data/lib/ruby_llm/contract/concerns/usage_aggregator.rb

@@ -4,6 +4,10 @@ module RubyLLM
   module Contract
     module Concerns
       module UsageAggregator
+        # Breakdown keys an adapter may add next to input/output. They are parts
+        # of input_tokens/output_tokens, not extra tokens, so sum_tokens skips them.
+        USAGE_DETAIL_KEYS = %i[cache_read_tokens cache_write_tokens thinking_tokens].freeze
+
         private
 
         def extract_usage(trace_entry)
@@ -24,18 +28,17 @@ module RubyLLM
         end
 
         def aggregate_usage(traces)
-          input_total = 0
-          output_total = 0
+          totals = Hash.new(0)
 
           traces.each do |trace_entry|
             usage = extract_usage(trace_entry)
             next unless usage.is_a?(Hash)
 
-            input_total += usage[:input_tokens] || 0
-            output_total += usage[:output_tokens] || 0
+            [:input_tokens, :output_tokens, *USAGE_DETAIL_KEYS].each { |key| totals[key] += usage[key] || 0 }
           end
 
-          { input_tokens: input_total, output_tokens: output_total }
+          { input_tokens: totals[:input_tokens], output_tokens: totals[:output_tokens] }
+            .merge(totals.slice(*USAGE_DETAIL_KEYS).select { |_, count| count.positive? })
         end
       end
     end

data/lib/ruby_llm/contract/cost_calculator.rb

@@ -7,20 +7,22 @@ module RubyLLM
     # **What this module does (public surface):**
     #
     # 1. **Fine-tune / custom-model pricing registry** — `register_model`
-    #    fills the gap left by RubyLLM 1.14's models.json: there is no
-    #    upstream `RubyLLM::Models.register` API, so fine-tuned models
-    #    (e.g. `ft:gpt-4o-custom`) need their pricing supplied locally.
+    #    fills a gap RubyLLM still has in 2.x: there is no simple
+    #    upstream API to price a model its registry lacks, so fine-tuned
+    #    models (e.g. `ft:gpt-4o-custom`) need their pricing supplied
+    #    locally. A registered price also overrides RubyLLM's for that id.
     # 2. **Lookup with fallback chain** — `calculate(model_name:, usage:)`
     #    checks the custom registry first, falls back to
     #    `RubyLLM.models.find(model_name)`, returns `nil` on miss.
     #
     # **What this module is NOT:**
     #
-    # - Not a "cost calculator" feature — the math itself
-    #   (`tokens × price_per_million / 1_000_000`) is trivial and lives
-    #   in `private_class_method :compute_cost` for internal use only.
-    # - Not a substitute for RubyLLM's pricing data — for any model in
-    #   `RubyLLM.models`, this module simply queries it.
+    # - Not a substitute for RubyLLM's pricing — for any model in
+    #   `RubyLLM.models`, RubyLLM's own `Model#cost_for` does the math.
+    # - Not the source of a finished call's cost: the RubyLLM adapter takes
+    #   that from the response (`response.cost`), which also covers what
+    #   the provider reported and RubyLLM's own retries. This module prices
+    #   pre-flight estimates and calls from adapters that report no cost.
     #
     # The reason this module exists at all is the registry + retry usage
     # aggregation across attempts (the latter sits in `Step::RetryExecutor`,
@@ -60,8 +62,9 @@ module RubyLLM
 
       # Look up cost for a single model + usage hash.
       # Returns nil if model is unknown (custom registry miss + RubyLLM miss),
-      # so callers can decide whether to refuse the call or proceed (see
-      # `on_unknown_pricing:` step option for the budget-gating policy).
+      # or if any used token category has no price, so callers can decide
+      # whether to refuse the call or proceed (see `on_unknown_pricing:` step
+      # option for the budget-gating policy).
       #
       #   CostCalculator.calculate(
       #     model_name: "gpt-4o-mini",
@@ -69,13 +72,17 @@ module RubyLLM
       #   )
       #   # => 0.00069 (or nil if model not registered)
       #
-      # Math is intentionally simple and private — this method is the
-      # primary public entry point. Aggregating across retry attempts is
-      # done in `Step::RetryExecutor`, not here.
-      def self.calculate(model_name:, usage:)
+      # `usage[:input_tokens]` is the whole prompt; `:cache_read_tokens` and
+      # `:cache_write_tokens`, when present, are the parts of it served from or
+      # written to the provider's prompt cache. `provider:` picks the provider's
+      # price for a model id several providers list at different prices.
+      #
+      # Aggregating across retry attempts is done in `Step::RetryExecutor`,
+      # not here.
+      def self.calculate(model_name:, usage:, provider: nil)
         return nil unless model_name && usage.is_a?(Hash)
 
-        model_info = find_model(model_name)
+        model_info = find_model(model_name, provider: provider)
         return nil unless model_info
 
         compute_cost(model_info, usage)
@@ -83,55 +90,76 @@ module RubyLLM
         nil
       end
 
-      def self.compute_cost(model_info, usage)
-        prices = prices_for(model_info)
-        return nil unless prices
-
-        input_cost = token_cost(usage[:input_tokens], prices[:input])
-        output_cost = token_cost(usage[:output_tokens], prices[:output])
-        (input_cost + output_cost).round(6)
+      def self.registered?(model_name)
+        @custom_models.key?(model_name)
       end
 
       # Two shapes reach here. `RegisteredModel` (our own struct, from
-      # `register_model`) exposes flat `*_price_per_million` readers. RubyLLM 2.0
-      # moved provider pricing into a nested value object and dropped those flat
-      # readers, so it has to be walked: pricing -> text_tokens -> standard.
+      # `register_model`) exposes flat `*_price_per_million` readers and is
+      # priced here. A RubyLLM model is priced by RubyLLM itself
+      # (`Model#cost_for`), which knows cache, long-context and per-provider
+      # prices and returns nil when a used category has no price.
       #
-      # nil means "unknown pricing". The guards keep the two known shapes
-      # explicit, but `calculate` rescues StandardError, so a future shape this
-      # walk cannot read also degrades to nil: `max_cost` then refuses the call
-      # citing "no pricing data" rather than the real cause, and paths that floor
-      # unknown cost to 0.0 (eval cost estimates, report totals) undercount.
-      def self.prices_for(model_info)
+      # nil means "unknown pricing". `calculate` rescues StandardError, so a
+      # future shape neither branch can read also degrades to nil: `max_cost`
+      # then refuses the call citing "no pricing data" rather than the real
+      # cause, and paths that floor unknown cost to 0.0 (eval cost estimates,
+      # report totals) undercount.
+      def self.compute_cost(model_info, usage)
         if model_info.respond_to?(:input_price_per_million)
-          return { input: model_info.input_price_per_million,
-                   output: model_info.output_price_per_million }
+          registered_cost(model_info, usage)
+        elsif model_info.respond_to?(:cost_for)
+          model_info.cost_for(tokens_for(usage)).total&.round(6)
         end
+      end
 
-        tier = model_info.pricing&.text_tokens&.standard if model_info.respond_to?(:pricing)
-        return unless tier.respond_to?(:input_per_million)
+      # Our registry knows input and output prices only. Cache reads are
+      # charged at the input price, which assumes they cost no more than
+      # uncached input (true of every provider in RubyLLM's registry, not
+      # guaranteed for a custom one). Cache writes can cost more than input
+      # (Anthropic charges 1.25x-2x), so a call that wrote to the cache has
+      # no price here rather than an undercount.
+      def self.registered_cost(model_info, usage)
+        return nil if (usage[:cache_write_tokens] || 0).positive?
+
+        input_cost = token_cost(usage[:input_tokens], model_info.input_price_per_million)
+        output_cost = token_cost(usage[:output_tokens], model_info.output_price_per_million)
+        (input_cost + output_cost).round(6)
+      end
 
-        { input: tier.input_per_million, output: tier.output_per_million }
+      # RubyLLM counts uncached input apart from cache reads and writes; our
+      # usage counts them all in input_tokens.
+      def self.tokens_for(usage)
+        cache_read = usage[:cache_read_tokens] || 0
+        cache_write = usage[:cache_write_tokens] || 0
+        ::RubyLLM::Tokens.new(
+          input: [(usage[:input_tokens] || 0) - cache_read - cache_write, 0].max,
+          output: usage[:output_tokens] || 0,
+          cache_read: cache_read.positive? ? cache_read : nil,
+          cache_write: cache_write.positive? ? cache_write : nil,
+          # Priced by RubyLLM only for models with a separate reasoning price.
+          thinking: usage[:thinking_tokens]
+        )
       end
-      private_class_method :prices_for
 
       # Provider pricing is denominated per 1M tokens; divide here to get
       # the dollar cost for the actual usage count. Named constant for
       # consistency with how RubyLLM and provider docs express prices.
       TOKENS_PER_MILLION = 1_000_000.0
 
+      # `register_model` validates both prices, so neither is nil here.
       def self.token_cost(tokens, price_per_million)
-        (tokens || 0) * (price_per_million || 0) / TOKENS_PER_MILLION
+        (tokens || 0) * price_per_million / TOKENS_PER_MILLION
       end
 
-      def self.find_model(model_name)
+      def self.find_model(model_name, provider: nil)
         # Check custom registry first
         custom = @custom_models[model_name]
         return custom if custom
 
         return nil unless defined?(RubyLLM)
 
-        RubyLLM.models.find(model_name)
+        provider ? RubyLLM.models.find(model_name, provider: provider) : RubyLLM.models.find(model_name)
       rescue StandardError
         nil
       end
@@ -146,7 +174,7 @@ module RubyLLM
       # to inspect model pricing before invoking `calculate` (e.g., to short-
       # circuit estimate when the model is unknown). Exposing it removes
       # `CostCalculator.send(:find_model)` workarounds at call sites.
-      private_class_method :compute_cost, :token_cost, :validate_price!
+      private_class_method :compute_cost, :registered_cost, :tokens_for, :token_cost, :validate_price!
     end
   end
 end

data/lib/ruby_llm/contract/eval/case_executor.rb

@@ -39,7 +39,7 @@ module RubyLLM
             input: test_case.input,
             output: nil,
             expected: test_case.expected,
-            step_status: :skipped,
+            step_status: CaseResult::SKIPPED_STATUS,
             score: 0.0,
             passed: false,
             label: "SKIP",

data/lib/ruby_llm/contract/eval/case_result.rb

@@ -7,6 +7,9 @@ module RubyLLM
         # BaselineDiff#compute_score keys off this prefix to keep skipped cases
         # out of the score denominator. Both ends must read it from here.
         SKIPPED_DETAILS_PREFIX = "skipped:"
+        # Not run (no adapter). Readers compare step_status, so any result object
+        # exposing it works; kept out of score, pass rate and cost-per-call.
+        SKIPPED_STATUS = :skipped
 
         def self.skipped_details(reason)
           "#{SKIPPED_DETAILS_PREFIX} #{reason}"