ruby_llm-contract 1.3.1 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b3d6985e56b8e410f796207f23b52e910b925f1bf4263353da65ca3f540c9535
-  data.tar.gz: 4cb5f7471dc5d97f3d3bd78fe41110cb6a9f28139e90029872fa612ed4ef3e9e
+  metadata.gz: b836138600866650ba9df05fce0a0ce02d148dc1819b9b215e5e47aa124e93cb
+  data.tar.gz: 237df37acef3aba9320acc723156dfdf33a928729c58b6e79ad54ea1ac33cf68
 SHA512:
-  metadata.gz: 62e15223ae05cfa64fd64959eb62858ace305a6f24e23e2c3f466219681eae4a1cd4c08addaf9d7d309f554c4ead57e1051d9b4b1eb55bcea4542ce8df23d789
-  data.tar.gz: e4c07a0a8b4706cbc062ca713ab9c8a2e87906b2fa4126a3b6d10f72ef30625ce6c8c5ea0a29ac30fef0b5251a06fd451bd1462c45289a0ab1cc939cdc25b989
+  metadata.gz: 79ff71eea63a6ce371b0d183137d8b0c498b3dcb8f21fa2410cbd9793334120be140a15716080688569e96a0d7b543e2083e986ccb2c199153722a17ae0adffb
+  data.tar.gz: c0547247d8d84e2e4bfc03fc398ddee89d1b77330924487a68f606f363365df800dc2be0016a255227cf4bb004c2b3510d1bf3f5ee7f19a1d5d00a2896103a28

data/CHANGELOG.md

@@ -1,5 +1,36 @@
 # Changelog
 
+## 1.4.0 (2026-10-11)
+
+### Added
+
+- **`reasoning_effort: :default` sends no effort for one call, candidate or
+  retry-chain element.** A Step with `reasoning_effort "none"` sent that effort to
+  every model, also one given in the context, a `compare_models` candidate or a
+  retry-chain element without an effort of its own, and a model without reasoning
+  rejected the call (OpenAI: "Unsupported parameter: 'reasoning.effort'"). Nothing
+  could take the class effort out short of a subclass with `reasoning_effort :default`,
+  and `reasoning_effort: nil` was ignored. `:default` - the word the DSL already
+  uses - now works in `context:`, in candidates (`{ model: "gpt-4.1-mini",
+  reasoning_effort: :default }`), in a `production_mode:` fallback, in `escalate`
+  / `retry_policy models:` elements and as `CANDIDATES=gpt-4.1-mini@default` in
+  `rake ruby_llm_contract:optimize`. A `thinking budget:` stays. `nil` still means
+  "not set", and the gem does not drop an effort by itself based on the model.
+
+### Changed
+
+- **A retry attempt's `trace.attempts[n][:config]` shows the effort it was sent.**
+  It showed the chain element, so an element without an effort, which runs with the
+  call's or the class's, looked as if it had none. Such an attempt now has
+  `config: { model:, reasoning_effort: }`; an attempt sent no effort has no
+  `config`, as before. Code comparing these hashes may need the extra key.
+- **`production_mode:` skips a fallback that would send the candidate's own request.**
+  A fallback without an effort runs with the candidate's, so `{ model: "gpt-4.1-mini",
+  reasoning_effort: "low" }` with `fallback: "gpt-4.1-mini"` retried the same call.
+  It now runs single-shot, as a fallback equal to the candidate already did. The
+  comparison counts the effort each would send: inherited from the call or the
+  Step, and none for `:default`.
+
 ## 1.3.1 (2026-10-11)
 
 ### Fixed
@@ -8,11 +39,12 @@
   comparison.** With `runs > 1` each report is an `AggregatedReport`: its results
   hold the cases of all N runs, while `total_cost` is the mean cost of one run. The
   cost per call divided one run's cost by the cases of every run, so with `runs: 3`
-  a $0.002 call showed as $0.0007. That skewed `best`, the retry chain, the rationale
-  and `savings`, and a candidate measured over more runs looked cheaper than one
-  measured once. Cases are now counted per run. `Step.recommend` always compares
-  with one run, so only a `Recommender` built by hand on a multi-run comparison
-  was affected.
+  a $0.002 call showed as $0.0007. Every candidate of one `compare_models` call runs
+  the same number of times, so the ranking held: `cost_per_call`, the rationale and
+  `savings` were wrong, `best` and the retry chain were not. Only a comparison put
+  together from reports with different run counts could pick the wrong model, since
+  a candidate measured over more runs looked cheaper. Cases are now counted per run.
+  `Step.recommend` always compares with one run and was not affected.
 
 ## 1.3.0 (2026-10-11)
 

data/docs/guide/getting_started.md

@@ -102,7 +102,7 @@ retry_policy models: [
 ]
 ```
 
-`reasoning_effort` is a gpt-5 family feature (gpt-4.1 is not a reasoning model). The per-attempt value is forwarded via `with_thinking` (provider-agnostic — OpenAI `reasoning_effort` and Anthropic extended-thinking budget both supported). Passing it alongside a non-reasoning model is forwarded unchanged to the provider, which will either ignore it or reject the request — the gem does not guard against this.
+`reasoning_effort` is a gpt-5 family feature (gpt-4.1 is not a reasoning model). The per-attempt value is forwarded via `with_thinking` (provider-agnostic — OpenAI `reasoning_effort` and Anthropic extended-thinking budget both supported). Passing it alongside a non-reasoning model is forwarded unchanged to the provider, which will either ignore it or reject the request — the gem does not guard against this. An element without `reasoning_effort` inherits the call's or the step's; `{ model: "gpt-4.1-mini", reasoning_effort: :default }` sends none (see [thinking DSL note](optimizing_retry_policy.md#thinking-dsl-note)).
 
 ## Evals and CI gates
 

data/docs/guide/optimizing_retry_policy.md

@@ -30,7 +30,7 @@ rake ruby_llm_contract:optimize \
   CANDIDATES=gpt-4.1-nano,gpt-4.1-mini@low,gpt-4.1-mini,gpt-4.1
 ```
 
-The `@low` (or `@medium`, `@high`) suffix on a candidate is CLI shorthand for `{ model: "gpt-4.1-mini", reasoning_effort: :low }` — saves you constructing hash strings on the command line. Plain model names without `@` use the Step's default thinking config (or none).
+The `@low` (or `@medium`, `@high`) suffix on a candidate is CLI shorthand for `{ model: "gpt-4.1-mini", reasoning_effort: :low }` — saves you constructing hash strings on the command line. Plain model names without `@` use the Step's default thinking config (or none). `@default` runs the candidate with no effort at all, which is what a model without reasoning needs when the Step sets one: `CANDIDATES=gpt-4.1-mini@default,gpt-6-luna@none`.
 
 Offline uses each eval's `sample_response` — zero API calls. **Every candidate gets the same score** because they all receive the canned response. That's fine for a smoke test (verifying evals load, candidates parse, output renders) but it doesn't compare model quality. For real optimization, go live.
 
@@ -76,7 +76,7 @@ Copy the DSL, paste into your step, verify with `rake ruby_llm_contract:eval`. Y
 
 `optimize` shows **first-attempt** cost. In production, a candidate whose validator rejects 20% of outputs actually costs `first_try_cost + fallback_cost × 0.20` per successful output. The first-attempt number hides this.
 
-`production_mode:` is an optional `compare_models` kwarg. With `{ fallback: "model" }` it runs each candidate with a runtime `[candidate, fallback]` chain and reports **effective cost** (first-try + fallback cost weighted by rejection rate). Without it, you get only first-attempt metrics (the default).
+`production_mode:` is an optional `compare_models` kwarg. With `{ fallback: "model" }` it runs each candidate with a runtime `[candidate, fallback]` chain and reports **effective cost** (first-try + fallback cost weighted by rejection rate). Without it, you get only first-attempt metrics (the default). A fallback without a `reasoning_effort` runs with the candidate's, so a fallback naming the candidate's model sends the same request and is skipped; `{ model: "...", reasoning_effort: :default }` falls back with no effort.
 
 ```ruby
 SummarizeArticle.compare_models(
@@ -131,3 +131,14 @@ end
 ```
 
 Forwarded to `Chat#with_thinking(**)` through the adapter — works provider-agnostically (OpenAI `reasoning_effort`, Anthropic extended-thinking budget). A per-call override via `context: { reasoning_effort: :high }` still wins over the class default.
+
+`reasoning_effort: :default` - the same word as in the DSL - sends no effort for one call, candidate or chain element, and keeps a `thinking budget:`. Use it to run a Step set up for a reasoning model on one that takes no effort, which a provider otherwise rejects (OpenAI: "Unsupported parameter: 'reasoning.effort'"):
+
+```ruby
+SummarizeArticle.run(text, context: { model: "gpt-4.1-mini", reasoning_effort: :default })
+SummarizeArticle.compare_models("regression", candidates: [{ model: "gpt-4.1-mini", reasoning_effort: :default },
+                                                           { model: "gpt-6-luna" }])
+retry_policy { escalate({ model: "gpt-4.1-mini", reasoning_effort: :default }, { model: "gpt-6-luna" }) }
+```
+
+`nil` is not `:default`: a nil effort is ignored and the class one is sent. The gem does not drop an effort by itself based on the model. A chain element without an effort runs with the call's (or the class's), and `trace.attempts[n][:config]` shows the effort each attempt was sent.

data/lib/ruby_llm/contract/adapters/ruby_llm.rb

@@ -129,6 +129,7 @@ module RubyLLM
         # overrides any inherited `options[:thinking][:effort]`; budget is
         # taken from `options[:thinking][:budget]` only.
         def resolve_thinking_config(options)
+          options = ::RubyLLM::Contract.without_default_effort(options)
           base = options[:thinking].is_a?(Hash) ? options[:thinking].dup : {}
           base[:effort] = options[:reasoning_effort] if options[:reasoning_effort]
           base.empty? ? nil : base

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -119,7 +119,8 @@ module RubyLLM
           model_context[:reasoning_effort] = config[:reasoning_effort] if config[:reasoning_effort]
           return model_context unless fallback_config
 
-          model_context[:retry_policy_override] = production_mode_override(config, fallback_config)
+          inherited_effort = context[:reasoning_effort] || (thinking[:effort] if respond_to?(:thinking) && thinking)
+          model_context[:retry_policy_override] = production_mode_override(config, fallback_config, inherited_effort)
           model_context
         end
 

data/lib/ruby_llm/contract/concerns/production_mode_context.rb

@@ -20,14 +20,23 @@ module RubyLLM
           RubyLLM::Contract.normalize_candidate_config(production_mode[:fallback])
         end
 
-        def production_mode_override(config, fallback_config)
-          return nil if same_candidate?(config, fallback_config)
+        def production_mode_override(config, fallback_config, inherited_effort = nil)
+          return nil if same_request?(config, fallback_config, inherited_effort)
 
           Step::RetryPolicy.new(models: [config, fallback_config])
         end
 
-        def same_candidate?(first, second)
-          first[:model] == second[:model] && first[:reasoning_effort] == second[:reasoning_effort]
+        # Compared by what each would send: a candidate without an effort runs
+        # with the call's or the Step's, a fallback without one with the
+        # candidate's, and :default sends none.
+        def same_request?(candidate, fallback, inherited_effort)
+          candidate_effort = sent_effort(candidate[:reasoning_effort] || inherited_effort)
+          fallback_effort = fallback[:reasoning_effort] ? sent_effort(fallback[:reasoning_effort]) : candidate_effort
+          candidate[:model] == fallback[:model] && candidate_effort == fallback_effort
+        end
+
+        def sent_effort(effort)
+          RubyLLM::Contract.normalize_effort(effort) == :default ? nil : effort
         end
       end
     end

data/lib/ruby_llm/contract/eval/candidate_label.rb

@@ -23,7 +23,7 @@ module RubyLLM
           match = PATTERN.match(label)
           return { model: label } unless match
 
-          { model: match.pre_match.strip, reasoning_effort: match[1] }
+          { model: match.pre_match.strip, reasoning_effort: RubyLLM::Contract.normalize_effort(match[1]) }
         end
       end
     end

data/lib/ruby_llm/contract/step/base.rb

@@ -261,6 +261,7 @@ module RubyLLM
           # programmer bugs (NoMethodError, adapter-code ArgumentError) must propagate
           # instead of being silently masked as :input_error.
           def run_once(input, adapter:, model:, context_temperature: nil, extra_options: {})
+            extra_options = RubyLLM::Contract.without_default_effort(extra_options)
             runner =
               begin
                 Runner.new(config: build_runner_config(adapter, model, context_temperature, extra_options))

data/lib/ruby_llm/contract/step/retry_executor.rb

@@ -18,11 +18,12 @@ module RubyLLM
           policy.max_attempts.times do |attempt_index|
             config = policy.config_for_attempt(attempt_index, default_config)
             model = config[:model]
-            attempt_extra = extra_options.merge(config.except(:model))
+            attempt_extra = RubyLLM::Contract.without_default_effort(extra_options.merge(config.except(:model)))
 
             result = run_once(input, adapter: adapter, model: model,
                               context_temperature: context_temperature, extra_options: attempt_extra)
-            all_attempts << { attempt: attempt_index + 1, model: model, config: config, result: result }
+            all_attempts << { attempt: attempt_index + 1, model: model, config: sent_config(model, attempt_extra),
+                              result: result }
             break unless policy.retryable?(result)
           end
 
@@ -49,6 +50,14 @@ module RubyLLM
           )
         end
 
+        # What the attempt was sent, not the chain element it came from: an
+        # element without an effort still gets the class or context one
+        # (runtime_settings copies the class effort into :reasoning_effort).
+        def sent_config(model, options)
+          effort = options[:reasoning_effort]
+          effort ? { model: model, reasoning_effort: effort } : { model: model }
+        end
+
         def build_attempt_entry(attempt)
           trace = attempt[:result].trace
           entry = { attempt: attempt[:attempt], model: attempt[:model], status: attempt[:result].status }

data/lib/ruby_llm/contract/version.rb

@@ -2,6 +2,6 @@
 
 module RubyLLM
   module Contract
-    VERSION = "1.3.1"
+    VERSION = "1.4.0"
   end
 end

data/lib/ruby_llm/contract.rb

@@ -91,6 +91,23 @@ module RubyLLM
         Thread.current[RELOADING_KEY] ? true : false
       end
 
+      # `reasoning_effort: :default`, the DSL's word for "no effort", leaves the
+      # class-level effort out of one call, candidate or retry-chain element.
+      # Rake and JSON give it as a String.
+      def normalize_effort(effort)
+        effort.to_s == "default" ? :default : effort
+      end
+
+      # The options as an adapter takes them: `reasoning_effort: :default` goes,
+      # and with it the effort in `thinking`; a thinking budget stays.
+      def without_default_effort(options)
+        return options unless normalize_effort(options[:reasoning_effort]) == :default
+
+        thinking = options[:thinking].is_a?(Hash) ? options[:thinking].except(:effort) : {}
+        resolved = options.except(:reasoning_effort, :thinking)
+        thinking.empty? ? resolved : resolved.merge(thinking: thinking)
+      end
+
       def normalize_candidate_config(entry)
         case entry
         when String
@@ -104,7 +121,7 @@ module RubyLLM
           end
 
           normalized = { model: model.strip }
-          effort = entry[:reasoning_effort] || entry["reasoning_effort"]
+          effort = normalize_effort(entry[:reasoning_effort] || entry["reasoning_effort"])
           normalized[:reasoning_effort] = effort if effort
           normalized
         else

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby_llm-contract
 version: !ruby/object:Gem::Version
-  version: 1.3.1
+  version: 1.4.0
 platform: ruby
 authors:
 - Justyna