ruby_llm-contract 1.1.1 → 1.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 390af91f8200f1098440872d22515fa6232888d13d21a149c0b11f0bf3c15d48
-  data.tar.gz: d39d8a0c2e8914d08e4d996e6a3eb4d2c53613304cabcc6e077af2d9963ad4f5
+  metadata.gz: 7ae19c7459f4999409748c604e8b80aa756e5b2333863c8cc16b4070bf57dff9
+  data.tar.gz: 3be220b3be5c0c313f4b3f5160da5416dc9be8c35629c0012d756e0d027aa646
 SHA512:
-  metadata.gz: 81e03a20720536700acfe5bc840720a84675ca738e248fbdb490dac1dc63479c93924923d16cb28433329b53d24b796a42332736d29ec6f4d42a41240dfd4367
-  data.tar.gz: a433acb982c831c7ee1e2db5675135c4a42762d75c78e53708673b8ca50f5f165fe2cba9a088828b95738f24f29afb01e2033f694e92c69182662a6307492aa4
+  metadata.gz: 16bc0227706b8e1cc16f5d93c7771f82e7835c40c20888cec9163a9537d30f80a86f761c47fd10bd778df8f7023ce041bd0d574248950e11bdad5dd3ac4860da
+  data.tar.gz: 4d4374cc2d65aa555f30656c2475bae261b0af1be0656b7449c22c38467b7f1e32f7ba159d215adc17254d0c624f7144f91087644f578b5cb2727fc8f9087c91

data/CHANGELOG.md

@@ -1,5 +1,61 @@
 # Changelog
 
+## 1.1.2 (2026-10-10)
+
+### Changed - costs and gates may move
+
+- **Costs now count the prompt cache, and may go up.** RubyLLM 2.x reports cached prompt
+  tokens apart from `tokens.input`, and the adapter read only input and output: 1,000
+  uncached + 9,000 cached + 200 output tokens on gpt-4.1-mini came out at $0.00072
+  instead of $0.00162. `trace[:usage][:input_tokens]` is now the whole prompt and
+  `trace[:cost]` is the cost RubyLLM puts on the response: the price of the provider the
+  call went to, cache and long-context prices, the amount the provider reported when it
+  reports one, and RubyLLM's own retries of the request. Eval totals, history and
+  `compare_models` figures can rise against older baselines; baselines compare scores,
+  not costs, so they do not regress on this.
+- **A call without token counts is no longer free.** When the provider sends no usage
+  (for the call or for any of RubyLLM's attempts at it), the counts still read 0 but
+  `trace[:usage_complete]` is `false` and, unless the provider reported what it billed,
+  the cost is unknown, so the eval cost gates refuse (opt out with
+  `on_unknown_pricing: :warn`).
+- **A missing price is unknown, not $0.** A model priced for input but not output has no
+  cost instead of a partial one, so `max_cost` refuses it pre-flight. A call that wrote
+  to the prompt cache on a model without a cache-write price has an unknown cost
+  afterwards (pre-flight estimates do not predict cache writes).
+
+### Fixed
+
+- `CostCalculator` prices RubyLLM's models through `Model#cost_for`, so pre-flight
+  estimates use cache and long-context prices, and the price of the provider the call
+  goes to (12 model ids carry different prices per provider). `register_model` keeps
+  overriding the price for its id; cache reads are charged at its input price, cache
+  writes have no price.
+- `compare_models` no longer names a model with unpriced cases the cheapest
+  (`best_for`), gives it an infinite `cost_per_point`, and `recommend` neither ranks it
+  on cost nor promises savings against it.
+- `single_shot_cost` reported a later attempt's cost when the first attempt was unpriced.
+- The step log printed an unknown cost as `$0.000000`; it now prints `cost=unknown`.
+- Docs no longer say RubyLLM has no evaluation framework: `relation_to_agent.md` now
+  places `RubyLLM::Evaluation` (2.1) next to `define_eval`. The README and the getting
+  started guide describe what a trace's usage and cost count, and the README's 1.x
+  promise now says new trace keys may be added.
+
+### Added
+
+- `trace[:usage]` breakdown keys when reported: `cache_read_tokens`, `cache_write_tokens`,
+  `thinking_tokens` (parts of `input_tokens` / `output_tokens`, not extra tokens).
+- `trace[:finish_reason]`, also on each retry attempt. A failed result whose response
+  stopped at a token limit or a content filter gets a last validation error saying so,
+  with the `max_output` the call ran with. Statuses and retries are unchanged.
+- An adapter may report its own cost: `Adapters::Response.new(content:, usage:, cost:,
+  cost_complete:, usage_complete:, finish_reason:)`, where `cost: nil` means unknown.
+  A response without `cost:` is priced from `usage` as before.
+- `provider:` on `CostCalculator.calculate`, `CostCalculator.find_model`,
+  `estimate_cost` and `estimate_eval_cost`.
+- Baselines and history mark unpriced cases and runs with `cost_unknown: true`.
+- A pipeline with a `token_budget` warns when a step's provider left out token counts;
+  `Pipeline::Trace#usage_complete?`.
+
 ## 1.1.1 (2026-10-10)
 
 ### Fixed

data/README.md

@@ -91,6 +91,8 @@ result.trace[:attempts]
 #    ]
 ```
 
+`usage[:input_tokens]` is the whole prompt, prompt-cache tokens included (broken out as `cache_read_tokens` / `cache_write_tokens` when the provider reports them), and `cost` is what RubyLLM prices the response at: the provider's price, cache and long-context rates, or the amount the provider reported. A cost the gem cannot fully price is marked unknown rather than counted as $0 - see [what a trace's usage and cost count](docs/guide/getting_started.md#what-a-traces-usage-and-cost-count).
+
 If the response is malformed, the TL;DR overflows the card, or the takeaway count is off, the gem moves to the next step. This is model **escalation**, not a fallback list — each step is an independent config (`model`, `reasoning_effort`), so the retry policy spends more compute only when the cheaper one couldn't satisfy the contract.
 
 ### Add a CI gate in 6 lines
@@ -128,6 +130,7 @@ Everything below is optional — the example above is a complete step. Reach for
 - **[Find the cheapest viable fallback list](docs/guide/optimizing_retry_policy.md)** — empirically pick the cheapest model chain that still passes your evals.
 - **[A/B test prompts](docs/guide/eval_first.md)** — measure whether a new prompt is safe to ship before merging.
 - **[Budget caps](docs/guide/getting_started.md)** — refuse the request pre-flight when an estimate exceeds the limit.
+- **[Cost tracking](docs/guide/getting_started.md#what-a-traces-usage-and-cost-count)** - per-call cost with prompt-cache and per-provider prices; eval cost gates fail closed when a cost is unknown.
 - **[Reasoning effort / thinking config](docs/guide/optimizing_retry_policy.md)** — Anthropic / OpenAI thinking configuration on the Step class.
 
 Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and per-step models.
@@ -136,7 +139,7 @@ Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and
 
 ## Relation to `RubyLLM::Agent`
 
-`Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly — Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate(...)` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
+`Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly - Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate(...)` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. RubyLLM 2.1's `RubyLLM::Evaluation` grades answers; `define_eval` guards against regressions and cost in CI - the mapping covers both. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
 
 ## Relation to `ruby_llm-tribunal`
 
@@ -153,7 +156,7 @@ Different layers, complementary. [`ruby_llm-tribunal`](https://github.com/Alqemi
 | [Why contracts?](docs/guide/why.md) | Recognise the four production failures the gem exists for |
 | [Relation to RubyLLM::Agent](docs/guide/relation_to_agent.md) | Sibling abstractions; what each adds; runtime call path; coexistence patterns |
 | [Relation to ruby_llm-tribunal](docs/guide/relation_to_tribunal.md) | Different layers (test framework vs runtime contract); visual flows; integration recipes |
-| [Getting Started](docs/guide/getting_started.md) | Walk the full feature set on one concrete step |
+| [Getting Started](docs/guide/getting_started.md) | Walk the full feature set on one concrete step, including what usage and cost count |
 | [Rails integration](docs/guide/rails_integration.md) | Directory, initializer, jobs, logging, specs, CI gate — 7 FAQs for Rails devs |
 | [Adopt in an existing Rails app](docs/guide/migration.md) | Replace raw `LlmClient.call` with a contract, Before/After |
 | [Prevent silent prompt regressions](docs/guide/eval_first.md) | Evals, baselines, CI gates that block quality drift |
@@ -173,7 +176,11 @@ Stable since **1.0.0**. Semver tracked; breaking changes flagged in [CHANGELOG](
 What 1.0 promises not to break inside the 1.x line: the documented DSL
 (`prompt`, `output_schema`, `validate`, `retry_policy`, `max_cost`, `define_eval`,
 pipelines), the `Result` and `trace` shapes including `trace[:usage]`'s
-`{ input_tokens:, output_tokens: }` keys, and the adapter interface. A new major
+`{ input_tokens:, output_tokens: }` keys, and the adapter interface. New keys may be
+added to both (1.1.2 added `cache_read_tokens`, `cache_write_tokens` and
+`thinking_tokens` to `trace[:usage]`, and `usage_complete`, `cost_complete` and
+`finish_reason` to the trace), and a figure may change when it was wrong: since 1.1.2
+`input_tokens` counts prompt-cache tokens. A new major
 version of a runtime dependency may require a new major version here — that is
 what 1.0.0 itself was.
 
@@ -185,6 +192,8 @@ what 1.0.0 itself was.
 
 **Where in a Rails app?** Default `app/contracts/`. The Railtie reloads `app/contracts/eval/` and `app/steps/eval/` in development; any autoloaded directory also works. See [Rails integration](docs/guide/rails_integration.md).
 
+**Costs went up, or an eval cost gate turned red, after upgrading to 1.1.2?** Earlier versions left prompt-cache tokens out of `input_tokens` and the cost, priced every model at its default provider, counted a missing price as $0, and treated a call without token counts as free. The [CHANGELOG](CHANGELOG.md) lists each change; `on_unknown_pricing: :warn` turns the unknown-cost refusal into a warning.
+
 **Upgraded from pre-0.10.0 and getting `:limit_exceeded` with attachments?** Multimodal contracts with `max_cost`/`max_input` need `attachment_token_estimate`. See [multimodal input guide](docs/guide/multimodal_input.md#cost-attachment_token_estimate-is-required) for setup, fail-closed behaviour, and `on_unknown_attachment_size :warn` opt-out.
 
 ## License

data/docs/guide/getting_started.md

@@ -173,7 +173,23 @@ max_cost 0.01, on_unknown_pricing: :warn
 
 Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
 
-The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. The check sees only reported tokens: an adapter that returns no token counts looks like a zero-token run.
+The eval-level gates - `with_maximum_cost` on `pass_eval`, `maximum_cost` on the rake task, and `maximum_cost:` on `assert_eval_passes` - follow the same rule since 1.1.0. A report's total counts a case it cannot price as $0, so when any case ran on a model without pricing data the gate fails and names the cases instead of comparing an undercounted total. Opt out with `.on_unknown_pricing(:warn)`, `t.on_unknown_pricing = :warn` or `on_unknown_pricing: :warn`, which checks the budget against the priced cases and prints a warning. Offline runs (`sample_response`, a `Test` adapter without `usage:`) report zero tokens, cost $0 with or without pricing, and are never flagged. Since 1.1.2 a real call whose provider sent no token counts is flagged too, instead of passing as a free zero-token run.
+
+### What a trace's usage and cost count
+
+`trace[:usage][:input_tokens]` is the whole prompt the model took in, including tokens served from or written to the provider's prompt cache. When the provider reports them, the trace adds a breakdown: `cache_read_tokens`, `cache_write_tokens` and `thinking_tokens`. These are parts of `input_tokens` / `output_tokens`, not extra tokens; reasoning tokens are already in `output_tokens` when the provider bills them as output, and a model with a separate reasoning price is charged that price for them.
+
+```ruby
+result.trace[:usage]
+# => { input_tokens: 10_000, output_tokens: 200, cache_read_tokens: 9_000 }
+result.trace[:cost]    # => 0.00162  (gpt-4.1-mini: cache reads at the cache price)
+```
+
+With the RubyLLM adapter, `trace[:cost]` is the cost RubyLLM puts on the response (`response.cost.total`): the price of the provider the call went to, cache and long-context prices, and the amount the provider reported when it reports one (OpenRouter, for example). A price set with `CostCalculator.register_model` still overrides it for that model id, unless the provider reported what it billed; there, cache reads are charged at the input price and a call that wrote to the cache has no price, since cache writes can cost more than input.
+
+When a cost does not cover the whole call, `result.trace.cost_unknown?` is true and the eval gates above refuse. That happens when pricing is missing for a token category the call used, or when the provider sent no token counts for the call or for one of RubyLLM's attempts at it (`trace[:usage_complete]` is then `false` and missing counts read as 0) and did not report what it billed either. On a retried step `trace[:cost]` is the sum of the attempts that could be priced, and `cost_unknown?` is true if any attempt could not. `max_cost` itself is a pre-flight check on an estimate; it does not inspect the finished call.
+
+A custom adapter returning `Response.new(content:, usage:)` is priced from `usage` as before. To report its own cost, pass `cost:` - `nil` means unknown - with `cost_complete:` and `usage_complete:`.
 
 ### Preflight cost estimates
 

data/docs/guide/relation_to_agent.md

@@ -12,8 +12,8 @@
 | `validate :rule do ... end` business invariants on output | only in `ruby_llm-contract` |
 | `retry_policy escalate(...)` model escalation on validation failure | only here (different from RubyLLM's network-level retry) |
 | `max_cost` / `max_input` / `max_output` pre-flight refusal | only here |
-| `define_eval` + baseline regression + `compare_models` + `optimize_retry_policy` | only here (RubyLLM does not ship an evaluation framework) |
-| Pipeline composition with `step SomeStep, as: :alias` | only here (RubyLLM intentionally leaves workflows as plain Ruby) |
+| `define_eval` + baseline regression + `compare_models` + `optimize_retry_policy` | only here; RubyLLM 2.1's `RubyLLM::Evaluation` grades answers (see below) but keeps no baseline and has no cost gate |
+| Pipeline composition with `step SomeStep, as: :alias` | only here; `RubyLLM.workflow` (2.1) groups calls for tracing, without typed step outputs, fail-fast or budgets |
 | `around_call`, named `observe` hooks with pass/fail recorded in trace | only here |
 
 ## Runtime relationship
@@ -32,6 +32,12 @@ Step.run(input)
 
 This may change in a future release if upstream APIs make a layered design natural. The decision is not committed; it depends on adopter signal.
 
+## Relation to `RubyLLM::Evaluation`
+
+RubyLLM 2.1 ships `RubyLLM::Evaluation`: a dataset of cases run through your `perform` method, graded for correctness by an LLM or by your own criteria and Ruby assertions, with RSpec/Minitest integration and a report of verdicts, tokens and cost. Use it to answer "is this answer right?".
+
+`define_eval` answers a different question: did this prompt or model get worse than last time, and does it stay within budget? It keeps baselines and flags regressions, compares prompts (`compare_with`) and models (`compare_models`, `recommend`), runs offline with `sample_response`, and gates CI on score and cost. The two can share a project; an Evaluation's `perform` can call `Step.run`.
+
 ## Coexistence on the same project
 
 The two abstractions can live in the same Rails (or non-Rails) project. Pick one per use case:

data/lib/ruby_llm/contract/adapters/response.rb

@@ -3,16 +3,42 @@
 module RubyLLM
   module Contract
     module Adapters
+      # `content` and `usage` are the whole interface an adapter has to fill.
+      # The rest is optional accounting: an adapter that leaves `cost` out gets
+      # its usage priced from the registry, as before; one that passes `cost`
+      # (nil included) owns the figure, and `cost_complete` / `usage_complete`
+      # say whether it and the token counts cover the whole call.
       class Response
         include Concerns::DeepFreeze
 
-        attr_reader :content, :usage
+        attr_reader :content, :usage, :usage_complete, :cost_complete, :finish_reason
 
-        def initialize(content:, usage: {})
+        def initialize(content:, usage: {}, cost: Step::Trace::COST_UNSET, usage_complete: nil,
+                       cost_complete: nil, finish_reason: nil)
           @content = deep_dup_freeze(content)
           @usage = deep_dup_freeze(usage)
+          @cost = cost
+          @usage_complete = usage_complete
+          @cost_complete = cost_complete
+          @finish_reason = finish_reason
           freeze
         end
+
+        def cost
+          cost_provided? ? @cost : nil
+        end
+
+        def cost_provided?
+          !@cost.equal?(Step::Trace::COST_UNSET)
+        end
+
+        # Keyword arguments for Step::Trace.new. Omits what the adapter did not
+        # report, so a plain Response keeps the registry pricing path.
+        def trace_accounting
+          fields = { usage_complete: @usage_complete, cost_complete: @cost_complete,
+                     finish_reason: @finish_reason }.compact
+          cost_provided? ? fields.merge(cost: @cost) : fields
+        end
       end
     end
   end

data/lib/ruby_llm/contract/adapters/ruby_llm.rb

@@ -13,16 +13,13 @@ module RubyLLM
           chat = build_chat(options, system_contents)
           add_history(chat, conversation[0..-2])
 
-          # `with: nil` is a documented no-op in RubyLLM (verified against
-          # 1.15.0: chat.rb:36-37 `build_content(message, nil)` -> content.rb:8-14
-          # `Content.new(text, nil)` keeps text-only path when attachments
-          # are empty; raise only fires when BOTH text and attachments are nil,
-          # and we always pass a non-nil string thanks to `&.fetch(:content, "")`).
+          # `with: nil` adds no attachment. The text is never nil (`fetch(:content, "")`),
+          # and RubyLLM raises only when both text and attachments are nil.
           response = chat.ask(
             conversation.last&.fetch(:content, ""),
             with: options[:attachment]
           )
-          build_response(response)
+          build_response(response, options[:model])
         end
 
         CHAT_OPTION_METHODS = {
@@ -64,8 +61,7 @@ module RubyLLM
           # taking precedence over `:thinking[:effort]`. This is the per-attempt
           # override path used by `retry_policy { escalate({model:, reasoning_effort:}) }`
           # — the attempt-specific effort must win over the class-level default.
-          # Forwarded provider-agnostically via `chat.with_thinking(**)` —
-          # available since RubyLLM 1.12 (gemspec enforces this minimum).
+          # Forwarded provider-agnostically via `chat.with_thinking(**)`.
           thinking_config = resolve_thinking_config(options)
           chat.with_thinking(**thinking_config) if thinking_config
 
@@ -84,26 +80,84 @@ module RubyLLM
           base.empty? ? nil : base
         end
 
-        def build_response(response)
+        def build_response(response, model)
           content = response.content
           content = content.to_s unless content.is_a?(Hash) || content.is_a?(Array)
 
-          # This is the ONLY place upstream token counts are read. RubyLLM 2.0
-          # replaced `Message#input_tokens`/`#output_tokens` with a `Tokens` value
-          # object. The `{ input_tokens:, output_tokens: }` shape below is this
-          # gem's own public contract (Step::Trace#usage, cost calculation,
-          # documented in the README) and deliberately does NOT change.
+          # This is the ONLY place upstream token counts are read. The
+          # `{ input_tokens:, output_tokens: }` shape is this gem's public
+          # contract (Step::Trace#usage, README) and does not change; RubyLLM's
+          # own counts go into it as described in `usage_from`.
           tokens = response.tokens
+          usage_complete = every_attempt_counted?(response, tokens)
+          reported = tokens&.reported_cost
+          usage = usage_from(tokens)
+          cost = call_cost(response, model, usage, reported)
 
           Response.new(
             content: content,
-            usage: {
-              input_tokens: tokens&.input || 0,
-              output_tokens: tokens&.output || 0
-            }
+            usage: usage,
+            cost: cost,
+            usage_complete: usage_complete,
+            # A cost RubyLLM priced from partial counts covers part of the call;
+            # one the provider reported covers all of it.
+            cost_complete: !cost.nil? && (usage_complete || !reported.nil?),
+            finish_reason: response.finish_reason
           )
         end
 
+        # `response.tokens` sums RubyLLM's attempts at the request, skipping
+        # counts an attempt never got, so the sum alone can look complete.
+        # RubyLLM records each attempt; one that failed after the provider may
+        # have billed it keeps nil counts, one never sent or refused gets zeros.
+        def every_attempt_counted?(response, tokens)
+          entries = response.respond_to?(:ruby_llm_usage_entries) ? Array(response.ruby_llm_usage_entries) : []
+          return entries.all? { |entry| reported_both_counts?(entry.tokens) } if entries.any?
+
+          reported_both_counts?(tokens)
+        end
+
+        # Zero is a reported count (a full cache hit leaves uncached input at 0);
+        # nil is a count the provider did not send.
+        def reported_both_counts?(tokens)
+          !tokens.nil? && !tokens.input.nil? && !tokens.output.nil?
+        end
+
+        # RubyLLM 2.x counts `input` without the prompt-cache tokens, which it
+        # reports apart as `cache_read` / `cache_write`. Our input_tokens is the
+        # whole prompt, so they are added back and also kept as a breakdown.
+        # `output` already includes thinking tokens when the provider bills them
+        # as output, so `thinking_tokens` is a breakdown, never added. A count
+        # the provider did not report is 0 here; `usage_complete` says so.
+        def usage_from(tokens)
+          return { input_tokens: 0, output_tokens: 0 } unless tokens
+
+          cache_read = tokens.cache_read.to_i
+          cache_write = tokens.cache_write.to_i
+          usage = { input_tokens: tokens.input.to_i + cache_read + cache_write, output_tokens: tokens.output.to_i }
+          usage[:cache_read_tokens] = cache_read if cache_read.positive?
+          usage[:cache_write_tokens] = cache_write if cache_write.positive?
+          usage[:thinking_tokens] = tokens.thinking.to_i if tokens.thinking.to_i.positive?
+          usage
+        end
+
+        # RubyLLM prices the response itself: per provider, with cache and
+        # long-context prices, the amount the provider reported when it did,
+        # and its own retries of the request. It returns nil when it cannot
+        # price every attempt. A price set with `register_model` still wins
+        # for its id, unless the provider reported what it billed; it prices
+        # the summed counts, so it is complete only when every attempt was
+        # counted (`cost_complete` in build_response).
+        def call_cost(response, model, usage, reported)
+          if reported.nil? && CostCalculator.registered?(model)
+            CostCalculator.calculate(model_name: model, usage: usage)
+          else
+            response.cost.total&.round(6)
+          end
+        rescue StandardError
+          nil
+        end
+
         def partition_messages(messages)
           system_contents = []
           conversation = []

data/lib/ruby_llm/contract/concerns/usage_aggregator.rb

@@ -4,6 +4,10 @@ module RubyLLM
   module Contract
     module Concerns
       module UsageAggregator
+        # Breakdown keys an adapter may add next to input/output. They are parts
+        # of input_tokens/output_tokens, not extra tokens, so sum_tokens skips them.
+        USAGE_DETAIL_KEYS = %i[cache_read_tokens cache_write_tokens thinking_tokens].freeze
+
         private
 
         def extract_usage(trace_entry)
@@ -24,18 +28,17 @@ module RubyLLM
         end
 
         def aggregate_usage(traces)
-          input_total = 0
-          output_total = 0
+          totals = Hash.new(0)
 
           traces.each do |trace_entry|
             usage = extract_usage(trace_entry)
             next unless usage.is_a?(Hash)
 
-            input_total += usage[:input_tokens] || 0
-            output_total += usage[:output_tokens] || 0
+            [:input_tokens, :output_tokens, *USAGE_DETAIL_KEYS].each { |key| totals[key] += usage[key] || 0 }
           end
 
-          { input_tokens: input_total, output_tokens: output_total }
+          { input_tokens: totals[:input_tokens], output_tokens: totals[:output_tokens] }
+            .merge(totals.slice(*USAGE_DETAIL_KEYS).select { |_, count| count.positive? })
         end
       end
     end

data/lib/ruby_llm/contract/cost_calculator.rb

@@ -7,20 +7,22 @@ module RubyLLM
     # **What this module does (public surface):**
     #
     # 1. **Fine-tune / custom-model pricing registry** — `register_model`
-    #    fills the gap left by RubyLLM 1.14's models.json: there is no
-    #    upstream `RubyLLM::Models.register` API, so fine-tuned models
-    #    (e.g. `ft:gpt-4o-custom`) need their pricing supplied locally.
+    #    fills a gap RubyLLM still has in 2.x: there is no simple
+    #    upstream API to price a model its registry lacks, so fine-tuned
+    #    models (e.g. `ft:gpt-4o-custom`) need their pricing supplied
+    #    locally. A registered price also overrides RubyLLM's for that id.
     # 2. **Lookup with fallback chain** — `calculate(model_name:, usage:)`
     #    checks the custom registry first, falls back to
     #    `RubyLLM.models.find(model_name)`, returns `nil` on miss.
     #
     # **What this module is NOT:**
     #
-    # - Not a "cost calculator" feature — the math itself
-    #   (`tokens × price_per_million / 1_000_000`) is trivial and lives
-    #   in `private_class_method :compute_cost` for internal use only.
-    # - Not a substitute for RubyLLM's pricing data — for any model in
-    #   `RubyLLM.models`, this module simply queries it.
+    # - Not a substitute for RubyLLM's pricing — for any model in
+    #   `RubyLLM.models`, RubyLLM's own `Model#cost_for` does the math.
+    # - Not the source of a finished call's cost: the RubyLLM adapter takes
+    #   that from the response (`response.cost`), which also covers what
+    #   the provider reported and RubyLLM's own retries. This module prices
+    #   pre-flight estimates and calls from adapters that report no cost.
     #
     # The reason this module exists at all is the registry + retry usage
     # aggregation across attempts (the latter sits in `Step::RetryExecutor`,
@@ -60,8 +62,9 @@ module RubyLLM
 
       # Look up cost for a single model + usage hash.
       # Returns nil if model is unknown (custom registry miss + RubyLLM miss),
-      # so callers can decide whether to refuse the call or proceed (see
-      # `on_unknown_pricing:` step option for the budget-gating policy).
+      # or if any used token category has no price, so callers can decide
+      # whether to refuse the call or proceed (see `on_unknown_pricing:` step
+      # option for the budget-gating policy).
       #
       #   CostCalculator.calculate(
       #     model_name: "gpt-4o-mini",
@@ -69,13 +72,17 @@ module RubyLLM
       #   )
       #   # => 0.00069 (or nil if model not registered)
       #
-      # Math is intentionally simple and private — this method is the
-      # primary public entry point. Aggregating across retry attempts is
-      # done in `Step::RetryExecutor`, not here.
-      def self.calculate(model_name:, usage:)
+      # `usage[:input_tokens]` is the whole prompt; `:cache_read_tokens` and
+      # `:cache_write_tokens`, when present, are the parts of it served from or
+      # written to the provider's prompt cache. `provider:` picks the provider's
+      # price for a model id several providers list at different prices.
+      #
+      # Aggregating across retry attempts is done in `Step::RetryExecutor`,
+      # not here.
+      def self.calculate(model_name:, usage:, provider: nil)
         return nil unless model_name && usage.is_a?(Hash)
 
-        model_info = find_model(model_name)
+        model_info = find_model(model_name, provider: provider)
         return nil unless model_info
 
         compute_cost(model_info, usage)
@@ -83,55 +90,76 @@ module RubyLLM
         nil
       end
 
-      def self.compute_cost(model_info, usage)
-        prices = prices_for(model_info)
-        return nil unless prices
-
-        input_cost = token_cost(usage[:input_tokens], prices[:input])
-        output_cost = token_cost(usage[:output_tokens], prices[:output])
-        (input_cost + output_cost).round(6)
+      def self.registered?(model_name)
+        @custom_models.key?(model_name)
       end
 
       # Two shapes reach here. `RegisteredModel` (our own struct, from
-      # `register_model`) exposes flat `*_price_per_million` readers. RubyLLM 2.0
-      # moved provider pricing into a nested value object and dropped those flat
-      # readers, so it has to be walked: pricing -> text_tokens -> standard.
+      # `register_model`) exposes flat `*_price_per_million` readers and is
+      # priced here. A RubyLLM model is priced by RubyLLM itself
+      # (`Model#cost_for`), which knows cache, long-context and per-provider
+      # prices and returns nil when a used category has no price.
       #
-      # nil means "unknown pricing". The guards keep the two known shapes
-      # explicit, but `calculate` rescues StandardError, so a future shape this
-      # walk cannot read also degrades to nil: `max_cost` then refuses the call
-      # citing "no pricing data" rather than the real cause, and paths that floor
-      # unknown cost to 0.0 (eval cost estimates, report totals) undercount.
-      def self.prices_for(model_info)
+      # nil means "unknown pricing". `calculate` rescues StandardError, so a
+      # future shape neither branch can read also degrades to nil: `max_cost`
+      # then refuses the call citing "no pricing data" rather than the real
+      # cause, and paths that floor unknown cost to 0.0 (eval cost estimates,
+      # report totals) undercount.
+      def self.compute_cost(model_info, usage)
         if model_info.respond_to?(:input_price_per_million)
-          return { input: model_info.input_price_per_million,
-                   output: model_info.output_price_per_million }
+          registered_cost(model_info, usage)
+        elsif model_info.respond_to?(:cost_for)
+          model_info.cost_for(tokens_for(usage)).total&.round(6)
         end
+      end
 
-        tier = model_info.pricing&.text_tokens&.standard if model_info.respond_to?(:pricing)
-        return unless tier.respond_to?(:input_per_million)
+      # Our registry knows input and output prices only. Cache reads are
+      # charged at the input price, which assumes they cost no more than
+      # uncached input (true of every provider in RubyLLM's registry, not
+      # guaranteed for a custom one). Cache writes can cost more than input
+      # (Anthropic charges 1.25x-2x), so a call that wrote to the cache has
+      # no price here rather than an undercount.
+      def self.registered_cost(model_info, usage)
+        return nil if (usage[:cache_write_tokens] || 0).positive?
+
+        input_cost = token_cost(usage[:input_tokens], model_info.input_price_per_million)
+        output_cost = token_cost(usage[:output_tokens], model_info.output_price_per_million)
+        (input_cost + output_cost).round(6)
+      end
 
-        { input: tier.input_per_million, output: tier.output_per_million }
+      # RubyLLM counts uncached input apart from cache reads and writes; our
+      # usage counts them all in input_tokens.
+      def self.tokens_for(usage)
+        cache_read = usage[:cache_read_tokens] || 0
+        cache_write = usage[:cache_write_tokens] || 0
+        ::RubyLLM::Tokens.new(
+          input: [(usage[:input_tokens] || 0) - cache_read - cache_write, 0].max,
+          output: usage[:output_tokens] || 0,
+          cache_read: cache_read.positive? ? cache_read : nil,
+          cache_write: cache_write.positive? ? cache_write : nil,
+          # Priced by RubyLLM only for models with a separate reasoning price.
+          thinking: usage[:thinking_tokens]
+        )
       end
-      private_class_method :prices_for
 
       # Provider pricing is denominated per 1M tokens; divide here to get
       # the dollar cost for the actual usage count. Named constant for
       # consistency with how RubyLLM and provider docs express prices.
       TOKENS_PER_MILLION = 1_000_000.0
 
+      # `register_model` validates both prices, so neither is nil here.
       def self.token_cost(tokens, price_per_million)
-        (tokens || 0) * (price_per_million || 0) / TOKENS_PER_MILLION
+        (tokens || 0) * price_per_million / TOKENS_PER_MILLION
       end
 
-      def self.find_model(model_name)
+      def self.find_model(model_name, provider: nil)
         # Check custom registry first
         custom = @custom_models[model_name]
         return custom if custom
 
         return nil unless defined?(RubyLLM)
 
-        RubyLLM.models.find(model_name)
+        provider ? RubyLLM.models.find(model_name, provider: provider) : RubyLLM.models.find(model_name)
       rescue StandardError
         nil
       end
@@ -146,7 +174,7 @@ module RubyLLM
       # to inspect model pricing before invoking `calculate` (e.g., to short-
       # circuit estimate when the model is unknown). Exposing it removes
       # `CostCalculator.send(:find_model)` workarounds at call sites.
-      private_class_method :compute_cost, :token_cost, :validate_price!
+      private_class_method :compute_cost, :registered_cost, :tokens_for, :token_cost, :validate_price!
     end
   end
 end

data/lib/ruby_llm/contract/eval/eval_history.rb

@@ -69,7 +69,8 @@ module RubyLLM
 
           lines = ["#{runs.length} runs"]
           runs.last(5).each do |r|
-            lines << "  #{r[:date]} score=#{r[:score].round(2)} cost=$#{format("%.6f", r[:total_cost] || r[:cost] || 0)}"
+            cost = format("%.6f", r[:total_cost] || r[:cost] || 0)
+            lines << "  #{r[:date]} score=#{r[:score].round(2)} cost=$#{cost}#{" + unknown" if r[:cost_unknown]}"
           end
           lines.join("\n")
         end

data/lib/ruby_llm/contract/eval/model_comparison.rb

@@ -36,8 +36,12 @@ module RubyLLM
           @reports[resolve_key(candidate)]&.total_cost
         end
 
+        # A report with unpriced cases has only a subtotal, so it cannot be
+        # called the cheapest.
         def best_for(min_score: 0.0)
-          eligible = @reports.select { |_, report| report.score > 0.0 && report.score >= min_score }
+          eligible = @reports.select do |_, report|
+            report.score > 0.0 && report.score >= min_score && !cost_unknown?(report)
+          end
           return nil if eligible.empty?
 
           eligible.min_by { |_, report| report.total_cost }&.first
@@ -45,6 +49,8 @@ module RubyLLM
 
         def cost_per_point
           @reports.transform_values do |report|
+            next Float::INFINITY if cost_unknown?(report)
+
             report.score.positive? ? report.total_cost / report.score : Float::INFINITY
           end
         end
@@ -163,6 +169,10 @@ module RubyLLM
           )
         end
 
+        def cost_unknown?(report)
+          report.respond_to?(:unknown_cost_results) && report.unknown_cost_results.any?
+        end
+
         def resolve_key(candidate)
           case candidate
           when String then candidate

data/lib/ruby_llm/contract/eval/recommender.rb

@@ -51,7 +51,8 @@ module RubyLLM
               cost_per_call: cost_per_call,
               latency: report.avg_latency_ms || Float::INFINITY,
               pass_rate_ratio: report.pass_rate_ratio,
-              total_cost: report.total_cost
+              total_cost: report.total_cost,
+              cost_unknown: unknown_cost?(report)
             }
           end
         end
@@ -114,7 +115,7 @@ module RubyLLM
 
           current_label = ModelComparison.candidate_label(@current_config)
           current_report = @comparison.reports[current_label]
-          return {} unless current_report
+          return {} if current_report.nil? || unknown_cost?(current_report)
 
           current_evaluated = current_report.results.count { |r| r.step_status != CaseResult::SKIPPED_STATUS }
           current_cases = [current_evaluated, 1].max
@@ -125,8 +126,14 @@ module RubyLLM
           { per_call: diff.round(6), monthly_at: { 10_000 => (diff * 10_000).round(2) } }
         end
 
+        # A subtotal over unpriced cases is not a cost to rank by. Zero stays
+        # "unknown pricing" too: offline runs report $0 without any call.
         def cost_known?(scored_candidate)
-          scored_candidate[:cost_per_call]&.positive?
+          !scored_candidate[:cost_unknown] && scored_candidate[:cost_per_call]&.positive?
+        end
+
+        def unknown_cost?(report)
+          report.respond_to?(:unknown_cost_results) && report.unknown_cost_results.any?
         end
       end
     end

data/lib/ruby_llm/contract/eval/report_stats.rb

@@ -85,7 +85,7 @@ module RubyLLM
         def single_shot_cost
           return nil unless production_mode?
 
-          evaluated_results.sum { |r| first_attempt_cost(r) || r.cost || 0.0 }
+          evaluated_results.sum { |r| first_attempt_cost(r) || 0.0 }
         end
 
         def effective_cost
@@ -116,9 +116,14 @@ module RubyLLM
 
         private
 
+        # With attempts, only the first one's cost: falling back to the case
+        # cost when it is unpriced would report a later attempt's spend as the
+        # single-shot cost.
         def first_attempt_cost(result)
-          first = (result.attempts || []).first
-          first && first[:cost]
+          attempts = result.attempts || []
+          return result.cost if attempts.empty?
+
+          attempts.first[:cost]
         end
 
         def first_attempt_latency(result)

data/lib/ruby_llm/contract/eval/report_storage.rb

@@ -52,8 +52,10 @@ module RubyLLM
 
         private
 
+        # cost_unknown is written only when true, so older files and runs that
+        # priced every case keep the same shape.
         def history_entry
-          {
+          entry = {
             date: Time.now.strftime("%Y-%m-%d"),
             score: @stats.score,
             total_cost: @stats.total_cost,
@@ -61,6 +63,8 @@ module RubyLLM
             pass_rate_ratio: @stats.pass_rate_ratio,
             cases_count: @stats.evaluated_results_count
           }
+          entry[:cost_unknown] = true if @stats.unknown_cost_results.any?
+          entry
         end
 
         def serialize_for_baseline
@@ -74,13 +78,15 @@ module RubyLLM
         end
 
         def serialize_case(result)
-          {
+          serialized = {
             name: result.name,
             passed: result.passed?,
             score: result.score,
             details: result.details,
             cost: result.cost
           }
+          serialized[:cost_unknown] = true if result.cost_unknown?
+          serialized
         end
 
         def storage_path(root_dir, extension, model:, reasoning_effort: nil)

data/lib/ruby_llm/contract/pipeline/runner.rb

@@ -83,6 +83,7 @@ module RubyLLM
             total_usage: aggregate_usage(traces),
             step_traces: traces
           )
+          warn_incomplete_budget_usage(trace)
 
           Result.new(
             status: execution.status, step_results: execution.step_results,
@@ -91,6 +92,13 @@ module RubyLLM
           )
         end
 
+        def warn_incomplete_budget_usage(trace)
+          return unless @token_budget && !trace.usage_complete?
+
+          warn "[ruby_llm-contract] token_budget checked against incomplete token counts: " \
+               "a provider left out usage for at least one step, so the total undercounts"
+        end
+
         def elapsed_ms(start_time)
           ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time) * 1000).round
         end

data/lib/ruby_llm/contract/pipeline/trace.rb

@@ -40,6 +40,16 @@ module RubyLLM
           @step_traces.any? { |step_trace| step_trace.respond_to?(:cost_unknown?) && step_trace.cost_unknown? }
         end
 
+        # False when any step's provider left out token counts, so total_usage
+        # (and a token_budget checked against it) undercounts.
+        def usage_complete?
+          return true unless @step_traces.is_a?(Array)
+
+          @step_traces.none? do |step_trace|
+            step_trace.respond_to?(:usage_complete) && step_trace.usage_complete == false
+          end
+        end
+
         def to_h
           { trace_id: @trace_id, total_latency_ms: @total_latency_ms,
             total_usage: @total_usage, step_traces: @step_traces,

data/lib/ruby_llm/contract/step/base.rb

@@ -26,9 +26,9 @@ module RubyLLM
                              context: context).results.first
           end
 
-          def estimate_cost(input:, model: nil, attachment: nil)
+          def estimate_cost(input:, model: nil, attachment: nil, provider: nil)
             model_name = estimated_model_name(model)
-            model_info = CostCalculator.find_model(model_name)
+            model_info = CostCalculator.find_model(model_name, provider: provider)
             return nil unless model_info
 
             text_tokens = TokenEstimator.estimate(build_messages(input))
@@ -48,12 +48,13 @@ module RubyLLM
               output_tokens_estimate: output_tokens,
               estimated_cost: CostCalculator.calculate(
                 model_name: model_name,
-                usage: { input_tokens: input_tokens, output_tokens: output_tokens }
+                usage: { input_tokens: input_tokens, output_tokens: output_tokens },
+                provider: provider
               )
             }
           end
 
-          def estimate_eval_cost(eval_name, models: nil)
+          def estimate_eval_cost(eval_name, models: nil, provider: nil)
             defn = send(:all_eval_definitions)[eval_name.to_s]
             raise ArgumentError, "No eval '#{eval_name}' defined" unless defn
 
@@ -61,7 +62,7 @@ module RubyLLM
             cases = defn.build_dataset.cases
 
             model_list.each_with_object({}) do |model_name, result|
-              result[model_name] = estimate_eval_cost_for_model(cases, model_name)
+              result[model_name] = estimate_eval_cost_for_model(cases, model_name, provider)
             end
           end
 
@@ -145,9 +146,9 @@ module RubyLLM
             [0, true]
           end
 
-          def estimate_eval_cost_for_model(cases, model_name)
+          def estimate_eval_cost_for_model(cases, model_name, provider)
             cases.sum do |test_case|
-              estimate = estimate_cost(input: test_case.input, model: model_name)
+              estimate = estimate_cost(input: test_case.input, model: model_name, provider: provider)
               # Two misses floor to 0.0, not one: model absent from the registry
               # (nil estimate), or present with unreadable pricing (hash whose
               # estimated_cost is nil). Documented as a floor, not a fail-closed.
@@ -287,12 +288,18 @@ module RubyLLM
                   "model=#{trace.model} status=#{result.status} " \
                   "latency=#{trace.latency_ms}ms " \
                   "tokens=#{trace.usage&.dig(:input_tokens) || 0}+#{trace.usage&.dig(:output_tokens) || 0} " \
-                  "cost=$#{format("%.6f", trace.cost || 0)}"
+                  "cost=#{log_cost(trace)}"
             logger.info(msg)
 
             log_failed_observations(result, logger)
           end
 
+          def log_cost(trace)
+            return "unknown" if trace.respond_to?(:cost_unknown?) && trace.cost_unknown?
+
+            "$#{format("%.6f", trace.cost || 0)}"
+          end
+
           def log_failed_observations(result, logger)
             failed = result.observations.select { |o| !o[:passed] }
             return if failed.empty?

data/lib/ruby_llm/contract/step/limit_checker.rb

@@ -70,7 +70,8 @@ module RubyLLM
           estimated_output = effective_max_output || (estimated * DEFAULT_OUTPUT_RATIO)
           estimated_cost = CostCalculator.calculate(
             model_name: model_name,
-            usage: { input_tokens: estimated, output_tokens: estimated_output }
+            usage: { input_tokens: estimated, output_tokens: estimated_output },
+            provider: pricing_provider
           )
 
           if estimated_cost.nil?

data/lib/ruby_llm/contract/step/result_builder.rb

@@ -4,12 +4,21 @@ module RubyLLM
   module Contract
     module Step
       class ResultBuilder
-        def initialize(contract_definition:, output_type:, output_schema:, model:, observers:)
+        # Why the provider stopped, worded for a failed result. Anthropic also
+        # reports an exhausted context window as :max_tokens, so the message
+        # does not claim max_output was the limit hit.
+        FINISH_REASON_ERRORS = {
+          max_tokens: "provider reported token-limit termination",
+          content_filter: "provider reported content filtering or refusal"
+        }.freeze
+
+        def initialize(contract_definition:, output_type:, output_schema:, model:, observers:, max_output: nil)
           @contract_definition = contract_definition
           @output_type = output_type
           @output_schema = output_schema
           @model = model
           @observers = observers
+          @max_output = max_output
         end
 
         def error_result(error_result:, messages:)
@@ -25,13 +34,14 @@ module RubyLLM
         def success_result(response:, messages:, latency_ms:, input:)
           raw_output = response.content
           validation_result = validate_output(raw_output, input)
-          trace = Trace.new(messages: messages, model: @model, latency_ms: latency_ms, usage: response.usage)
+          trace = Trace.new(messages: messages, model: @model, latency_ms: latency_ms, usage: response.usage,
+                            **response_accounting(response))
 
           Result.new(
             status: validation_result[:status],
             raw_output: raw_output,
             parsed_output: validation_result[:parsed_output],
-            validation_errors: validation_result[:errors],
+            validation_errors: validation_errors(validation_result, response),
             trace: trace,
             observations: observations_for(validation_result, input)
           )
@@ -39,6 +49,25 @@ module RubyLLM
 
         private
 
+        # Only a failed result says why the provider stopped; a result that
+        # passed validation keeps finish_reason in its trace alone.
+        def validation_errors(validation_result, response)
+          errors = validation_result[:errors]
+          return errors if validation_result[:status] == :ok
+
+          reason = response.respond_to?(:finish_reason) ? response.finish_reason : nil
+          message = FINISH_REASON_ERRORS[reason]
+          return errors unless message
+
+          message += "; configured max_output: #{@max_output}" if reason == :max_tokens && @max_output
+          [*errors, message]
+        end
+
+        # Custom adapters may return any object with `content` and `usage`.
+        def response_accounting(response)
+          response.respond_to?(:trace_accounting) ? response.trace_accounting : {}
+        end
+
         def observations_for(validation_result, input)
           return [] unless validation_result[:status] == :ok && @observers.any?
 

data/lib/ruby_llm/contract/step/retry_executor.rb

@@ -6,6 +6,9 @@ module RubyLLM
       module RetryExecutor
         include Concerns::UsageAggregator
 
+        # Copied from each attempt's trace into its attempts entry when present.
+        ATTEMPT_TRACE_FIELDS = %i[usage latency_ms cost finish_reason].freeze
+
         private
 
         def run_with_retry(input, adapter:, default_model:, policy:, context_temperature: nil, extra_options: {})
@@ -39,7 +42,9 @@ module RubyLLM
             observations: last.observations,
             trace: last.trace.merge(
               attempts: attempt_log, usage: aggregated_usage,
-              cost: total_cost, latency_ms: total_latency
+              cost: total_cost, latency_ms: total_latency,
+              usage_complete: aggregate_flag(all_attempts, :usage_complete),
+              cost_complete: aggregate_cost_complete(all_attempts)
             )
           )
         end
@@ -53,12 +58,38 @@ module RubyLLM
         end
 
         def append_trace_fields(entry, trace)
-          entry[:usage] = trace.usage if trace.respond_to?(:usage) && trace.usage
-          entry[:latency_ms] = trace.latency_ms if trace.respond_to?(:latency_ms) && trace.latency_ms
-          entry[:cost] = trace.cost if trace.respond_to?(:cost) && trace.cost
+          ATTEMPT_TRACE_FIELDS.each do |field|
+            value = trace_value(trace, field)
+            entry[field] = value if value
+          end
+          entry[:cost_unknown] = true if trace_value(trace, :cost_unknown?)
+          entry[:usage_complete] = false if trace_value(trace, :usage_complete) == false
           entry
         end
 
+        def trace_value(trace, method)
+          trace.public_send(method) if trace.respond_to?(method)
+        end
+
+        # The subtotal is complete only if every attempt was; the last attempt's
+        # flag must not speak for it. nil when no attempt reported accounting
+        # state, so a retried Test-adapter step serializes exactly as before.
+        def aggregate_cost_complete(all_attempts)
+          traces = all_attempts.map { |a| a[:result].trace }
+          return false if traces.any? { |trace| trace.respond_to?(:cost_unknown?) && trace.cost_unknown? }
+
+          aggregate_flag(all_attempts, :cost_complete)
+        end
+
+        def aggregate_flag(all_attempts, flag)
+          flags = all_attempts.map { |a| a[:result].trace }
+                              .select { |trace| trace.respond_to?(flag) }
+                              .map { |trace| trace.public_send(flag) }.compact
+          return nil if flags.empty?
+
+          flags.all?
+        end
+
         def sum_attempt_costs(all_attempts)
           costs = extract_trace_values(all_attempts, :cost)
           return nil if costs.empty?

data/lib/ruby_llm/contract/step/runner.rb

@@ -59,7 +59,8 @@ module RubyLLM
             output_type: @config.output_type,
             output_schema: @config.output_schema,
             model: @config.model,
-            observers: @config.observers
+            observers: @config.observers,
+            max_output: @config.effective_max_output
           )
         end
 
@@ -87,6 +88,11 @@ module RubyLLM
           @config.on_unknown_attachment_size
         end
 
+        # The provider the adapter will call, so the estimate uses its price.
+        def pricing_provider
+          @config.extra_options&.dig(:provider)
+        end
+
         def attachment_present?
           opts = @config.extra_options
           !opts.nil? && !opts[:attachment].nil?

data/lib/ruby_llm/contract/step/trace.rb

@@ -7,19 +7,34 @@ module RubyLLM
         include Concerns::TraceEquality
         include Concerns::DeepFreeze
 
-        attr_reader :messages, :model, :latency_ms, :usage, :attempts, :cost
-
-        def initialize(messages: nil, model: nil, latency_ms: nil, usage: nil, attempts: nil, cost: nil)
+        attr_reader :messages, :model, :latency_ms, :usage, :attempts, :cost,
+                    :usage_complete, :cost_complete, :finish_reason
+
+        # Marks `cost:` as not given, as distinct from an explicit nil. Only an
+        # omitted cost is priced from the registry; nil from an adapter that
+        # reports its own cost means "unknown" and must stay unknown.
+        COST_UNSET = Object.new.freeze
+
+        # usage_complete / cost_complete are nil for traces that carry no
+        # accounting state (Test adapter, older adapters, hand-built traces);
+        # those keep the token-count rule in `unpriced?`.
+        def initialize(messages: nil, model: nil, latency_ms: nil, usage: nil, attempts: nil, cost: COST_UNSET,
+                       usage_complete: nil, cost_complete: nil, finish_reason: nil)
           @messages = deep_dup_freeze(messages)
           @model = model.frozen? ? model : model&.dup&.freeze
           @latency_ms = latency_ms
           @usage = deep_dup_freeze(usage)
           @attempts = deep_dup_freeze(attempts)
-          @cost = cost || CostCalculator.calculate(model_name: model, usage: usage)
+          @usage_complete = usage_complete
+          @cost_complete = cost_complete
+          @finish_reason = finish_reason
+          @cost = resolve_cost(cost)
+          @keep_nil_cost = keep_nil_cost?(cost)
           freeze
         end
 
-        KNOWN_KEYS = %i[messages model latency_ms usage attempts cost].freeze
+        KNOWN_KEYS = %i[messages model latency_ms usage attempts cost
+                        usage_complete cost_complete finish_reason].freeze
 
         def [](key)
           return nil unless KNOWN_KEYS.include?(key.to_sym)
@@ -39,27 +54,27 @@ module RubyLLM
         end
         alias has_key? key?
 
+        # Never re-prices: a retry merges a subtotal and an aggregate usage that
+        # no single model's pricing describes.
         def merge(**overrides)
-          self.class.new(
-            messages: overrides.fetch(:messages, @messages),
-            model: overrides.fetch(:model, @model),
-            latency_ms: overrides.fetch(:latency_ms, @latency_ms),
-            usage: overrides.fetch(:usage, @usage),
-            attempts: overrides.fetch(:attempts, @attempts),
-            cost: overrides.fetch(:cost, @cost)
-          )
+          self.class.new(**KNOWN_KEYS.to_h { |key| [key, overrides.fetch(key) { public_send(key) }] })
         end
 
-        # True when a call ran on a named model, used tokens, and still got no
-        # cost, so any total built from this trace undercounts. A zero-token call
+        # True when a total built from this trace undercounts: the adapter said
+        # its cost does not cover the call, or (no accounting state) a call ran
+        # on a named model, used tokens, and still got no cost. A zero-token call
         # (Test adapter, sample_response) costs 0 whatever the pricing. A retried
         # step is judged per attempt: a priced subtotal must not hide an unpriced
-        # attempt, and the merged trace may have re-priced on the last model.
+        # attempt.
         def cost_unknown?
+          return true if @cost_complete == false
+
           if @attempts.is_a?(Array) && !@attempts.empty?
-            @attempts.any? { |attempt| self.class.unpriced?(attempt[:model], attempt[:usage], attempt[:cost]) }
+            @attempts.any? do |attempt|
+              attempt[:cost_unknown] || self.class.unpriced?(attempt[:model], attempt[:usage], attempt[:cost])
+            end
           else
-            self.class.unpriced?(@model, @usage, @cost)
+            @cost_complete.nil? && self.class.unpriced?(@model, @usage, @cost)
           end
         end
 
@@ -70,8 +85,11 @@ module RubyLLM
         end
 
         def to_h
-          { messages: @messages, model: @model, latency_ms: @latency_ms,
-            usage: @usage, attempts: @attempts, cost: @cost }.compact
+          hash = { messages: @messages, model: @model, latency_ms: @latency_ms,
+                   usage: @usage, attempts: @attempts, cost: @cost, usage_complete: @usage_complete,
+                   cost_complete: @cost_complete, finish_reason: @finish_reason }.compact
+          hash[:cost] = nil if @keep_nil_cost
+          hash
         end
 
         def to_s
@@ -80,6 +98,22 @@ module RubyLLM
 
         private
 
+        # A given nil cost that the registry would price must stay `cost: nil` in
+        # to_h, or `Trace.new(**to_h)` would come back priced. Otherwise it is
+        # left out, so a merged trace compares equal to the one it came from.
+        def keep_nil_cost?(cost)
+          return false unless cost.nil? && @cost_complete.nil?
+
+          !CostCalculator.calculate(model_name: @model, usage: @usage).nil?
+        end
+
+        def resolve_cost(cost)
+          return cost unless cost.equal?(COST_UNSET)
+          return nil unless @cost_complete.nil?
+
+          CostCalculator.calculate(model_name: @model, usage: @usage)
+        end
+
         def build_summary_parts
           parts = [@model || "no-model"]
           parts << "#{@latency_ms}ms" if @latency_ms

data/lib/ruby_llm/contract/token_estimator.rb

@@ -11,9 +11,11 @@ module RubyLLM
     #   - Worse for non-English text, code, structured data, and unusual scripts
     #   - Useless for models with very different tokenizers (e.g. some open-source models)
     #
-    # RubyLLM 1.14 ships no pre-flight tokenizer either; once the API call
-    # returns, `RubyLLM::Tokens` provides accurate counts from provider usage
-    # data. This estimator is for the *pre-flight refusal* path only — its job
+    # RubyLLM 2.x can count tokens exactly before a call (`chat.count_tokens`),
+    # but only on some providers and at the price of an extra request, so this
+    # local estimate stays the default; once the API call returns,
+    # `RubyLLM::Tokens` provides accurate counts from provider usage data.
+    # This estimator is for the *pre-flight refusal* path only - its job
     # is to answer "is this call almost certainly within budget?" with enough
     # accuracy that runaway prompts get caught, while accepting that the
     # boundary cases will be wrong.

data/lib/ruby_llm/contract/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module Contract
-    VERSION = "1.1.1"
+    VERSION = "1.1.2"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-contract
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.1.2
 platform: ruby
 authors:
 - Justyna