ruby_llm-contract 0.10.1 → 0.10.3

data/docs/guide/getting_started.md

@@ -0,0 +1,199 @@
+# Getting Started
+
+> Read this to walk through every feature on one concrete step for the first time.
+
+The README shows a minimal `SummarizeArticle` step. This guide walks through the features you reach for as production requirements grow: budget caps so runaway inputs don't drain your LLM provider budget, evals so you catch regressions in CI, and CI gating so a merge that lowers accuracy gets blocked.
+
+## The walkthrough
+
+Start with the README example, then add features one layer at a time. Each is optional — use what you need.
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  # 1. Prompt (required)
+  prompt <<~PROMPT
+    Summarize this article for a UI card. Return a short TL;DR,
+    3 to 5 key takeaways, and a tone label.
+
+    {input}
+  PROMPT
+
+  # 2. Schema — sent to the provider via with_schema, validated client-side
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
+  end
+
+  # 3. Business rules — things JSON Schema cannot express
+  validate("TL;DR fits the card")  { |o, _| o[:tldr].length <= 200 }
+  validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }
+
+  # 4. Retry with model fallback on validation_failed / parse_error
+  retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]
+
+  # 5. Refuse before calling the LLM if input is too large or estimated cost exceeds the cap
+  max_input  2_000
+  max_output 4_000
+  max_cost   0.01
+end
+```
+
+## Validation and retry behavior
+
+When the cheap model returns output that fails a `validate` block or can't be parsed, retry falls back to the next model in `models:` and tries again.
+
+```ruby
+result = SummarizeArticle.run(article_text)
+
+result.status           # => :ok
+result.parsed_output    # => { tldr: "...", takeaways: [...], tone: "analytical" }
+result.trace[:model]    # => "gpt-4.1-mini"  (first model that passed)
+result.trace[:cost]     # => 0.00052  (sum of all attempts)
+result.trace[:attempts]
+# => [
+#   { attempt: 1, model: "gpt-4.1-nano", status: :validation_failed,
+#     cost: 0.00010, latency_ms: 45, ... },
+#   { attempt: 2, model: "gpt-4.1-mini", status: :ok,
+#     cost: 0.00042, latency_ms: 92, ... }
+# ]
+```
+
+If the whole chain exhausts, `result.status` is the status of the last attempt (`:validation_failed` or `:parse_error`) and `result.parsed_output` is the last attempt's output. The caller decides what to do — ship it anyway, fall back to a template, or raise.
+
+### Per-attempt reasoning effort
+
+`models:` accepts config hashes as well as model-name strings, so a fallback can "try harder" (more reasoning) on retry, not just switch model:
+
+```ruby
+retry_policy models: [
+  "gpt-5-nano",                                      # attempt 1: cheap + fast
+  { model: "gpt-5-mini", reasoning_effort: "high" }  # attempt 2: stronger + more reasoning
+]
+```
+
+`reasoning_effort` is a gpt-5 family feature (gpt-4.1 is not a reasoning model). The per-attempt value is forwarded via `with_thinking` (provider-agnostic — OpenAI `reasoning_effort` and Anthropic extended-thinking budget both supported). Passing it alongside a non-reasoning model is forwarded unchanged to the provider, which will either ignore it or reject the request — the gem does not guard against this.
+
+## Evals and CI gates
+
+An eval is a named scenario you can run to verify the step still works. `sample_response` makes it offline — zero API calls — so CI can run it on every merge without burning budget.
+
+```ruby
+SummarizeArticle.define_eval("smoke") do
+  default_input <<~ARTICLE
+    Ruby 3.4 ships with frozen string literals on by default, measurable YJIT
+    speedups on Rails workloads, and tightened Warning.warn category filtering.
+    The release notes also mention several parser fixes and faster keyword
+    argument handling.
+  ARTICLE
+
+  sample_response({
+    tldr: "Ruby 3.4 brings frozen string literals by default, YJIT speedups, and parser fixes.",
+    takeaways: [
+      "Frozen string literals are the default",
+      "YJIT adds measurable speedups on Rails workloads",
+      "Warning.warn category filtering is tighter"
+    ],
+    tone: "analytical"
+  })
+end
+
+report = SummarizeArticle.run_eval("smoke")
+report.passed?  # => true — schema + validates pass on the canned response
+report.score    # => 1.0
+report.print_summary
+```
+
+For real regression testing, define cases with expected output (online — calls the LLM):
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "ruby release",
+           input: "Ruby 3.4 was released...",
+           expected: { tone: "analytical" }  # partial match
+
+  add_case "critical review",
+           input: "The new mesh networking hardware failed under load...",
+           expected: { tone: "negative" }
+end
+```
+
+Gate CI on score and cost thresholds:
+
+```ruby
+# RSpec — blocks merge if accuracy drops or cost spikes
+expect(SummarizeArticle).to pass_eval("regression")
+  .with_minimum_score(0.8)
+  .with_maximum_cost(0.01)
+```
+
+Save a baseline once, then block regressions automatically:
+
+```ruby
+report = SummarizeArticle.run_eval("regression")
+report.save_baseline!
+
+# In CI:
+expect(SummarizeArticle).to pass_eval("regression").without_regressions
+```
+
+`without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality.
+
+## Budget caps
+
+`max_input`, `max_output`, and `max_cost` are preflight checks — the LLM is never called if an estimate exceeds the limit. Zero tokens spent, zero cost.
+
+```ruby
+result = SummarizeArticle.run(giant_10mb_document)
+result.status            # => :limit_exceeded
+result.validation_errors
+# => ["Input token limit exceeded: estimated 32000 tokens (heuristic ±30%), max 2000"]
+```
+
+`max_cost` fails closed when the model's pricing isn't known — register custom or fine-tuned models explicitly:
+
+```ruby
+RubyLLM::Contract::CostCalculator.register_model("ft:gpt-4o-custom",
+  input_per_1m: 3.0, output_per_1m: 6.0)
+```
+
+Or opt into a soft warning instead of a refusal when pricing is missing:
+
+```ruby
+max_cost 0.01, on_unknown_pricing: :warn
+```
+
+Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
+
+### Preflight cost estimates
+
+Check what a call is likely to cost before invoking it:
+
+```ruby
+SummarizeArticle.estimate_cost(input: article_text)
+# => {
+#      model: "gpt-4.1-mini",
+#      input_tokens: 812, output_tokens_estimate: 4000,
+#      estimated_cost: 0.00243
+#    }
+
+# Estimate what a full eval would cost across candidate models
+SummarizeArticle.estimate_eval_cost("regression",
+  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
+# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }
+```
+
+`estimate_cost` returns `nil` when pricing isn't registered. `estimate_eval_cost` silently treats unknown-pricing cases as `$0.00` and sums the rest — it does **not** fail closed the way `max_cost` does. Treat its output as a floor, not a guarantee; register pricing via `CostCalculator.register_model` before relying on it for budget decisions.
+
+## `output_schema` vs `with_schema`
+
+`with_schema` in `ruby_llm` tells the provider to force a specific JSON structure. `output_schema` in this gem does the same thing (calls `with_schema` under the hood) **plus** validates the response client-side. Cheaper models sometimes ignore schema constraints — `with_schema` is a request; `output_schema` is a request plus verification.
+
+## See also
+
+- [Prompt AST](prompt_ast.md) — prompt DSL variants: `system`, `rule`, `section`, `example`, `user`, and dynamic prompts with `|input|`.
+- [Eval-First](eval_first.md) — datasets, baselines, A/B gates, the workflow that makes the above evals useful.
+- [Optimizing retry_policy](optimizing_retry_policy.md) — find the cheapest viable fallback list with `compare_models` and `optimize_retry_policy`.
+- [Testing](testing.md) — test adapter, `stub_step`, full RSpec + Minitest matcher reference.
+- [Output Schema](output_schema.md) — nested objects in arrays, constraints, pattern reference.
+- [Rails integration](rails_integration.md) — where step classes live, initializer, jobs, logging, specs, CI gate.

data/docs/guide/migration.md

@@ -0,0 +1,185 @@
+# Migration Guide
+
+> Read this when adopting the gem in an existing Rails app with a raw `LlmClient.call` service you want to replace. Skip if starting fresh — go to [Getting Started](getting_started.md) instead.
+
+How to adopt `ruby_llm-contract` in an existing Rails app. Examples use `SummarizeArticle` — the flagship step from the [README](../../README.md) — but the pattern applies to any single-input / structured-output service.
+
+## Step 1: Start with the simplest service
+
+Pick the LLM service with: single input → JSON output → DB save. Don't start with parallel batches or complex pipelines.
+
+## Step 2: Define the contract
+
+**Before — raw HTTP:**
+
+```ruby
+class ArticleSummaryService
+  def call(article_text)
+    response = LlmClient.new(model: "gpt-4o-mini").call(prompt(article_text))
+    JSON.parse(response[:content], symbolize_names: true)
+  end
+end
+```
+
+**After — contract:**
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  model "gpt-4.1-mini"
+
+  prompt do
+    system "You summarize articles for a UI card."
+    rule "Return valid JSON only."
+    user "{input}"
+  end
+
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
+  end
+
+  validate("TL;DR fits the card") { |o, _| o[:tldr].length <= 200 }
+  retry_policy models: %w[gpt-4.1-mini gpt-4.1]
+end
+```
+
+## Step 3: Replace the caller
+
+```ruby
+# Before
+parsed = ArticleSummaryService.new.call(article_text)
+Article.update!(summary: parsed["tldr"])
+
+# After
+result = SummarizeArticle.run(article_text)
+if result.ok?
+  Article.update!(summary: result.parsed_output[:tldr])
+else
+  Rails.logger.warn "Summary failed: #{result.status}"
+end
+```
+
+## Step 4: Add logging via around_call
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  # ... prompt, schema, validates ...
+
+  around_call do |step, input, result|
+    AiCallLog.create!(
+      ai_model: result.trace.model,
+      duration_ms: result.trace.latency_ms,
+      input_tokens: result.trace.usage&.dig(:input_tokens),
+      output_tokens: result.trace.usage&.dig(:output_tokens),
+      cost: result.trace.cost,
+      status: result.status.to_s
+    )
+  end
+end
+```
+
+## Step 5: Add eval cases
+
+Use real inputs from production logs:
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "short news",
+           input: "Ruby 3.4 ships with frozen string literals by default...",
+           expected: { tone: "analytical" }
+
+  add_case "critical review",
+           input: "The new mesh networking hardware failed under load...",
+           expected: { tone: "negative" }
+end
+```
+
+## Step 6: Find the cheapest model
+
+```ruby
+comparison = SummarizeArticle.compare_models("regression",
+  candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }])
+
+comparison.print_summary
+comparison.best_for(min_score: 0.95)  # => cheapest model at >= 95%
+```
+
+Full optimization workflow — multi-eval, fallback list, production-mode cost — in [Optimizing retry_policy](optimizing_retry_policy.md).
+
+## Step 7: Add CI gate
+
+```ruby
+# Rakefile
+require "ruby_llm/contract/rake_task"
+RubyLLM::Contract::RakeTask.new do |t|
+  t.minimum_score = 0.8
+  t.maximum_cost = 0.05
+  t.fail_on_regression = true
+  t.save_baseline = true
+end
+```
+
+**Rails apps:** if your adapter is configured in an initializer, use a Proc so context is resolved after Rails boots:
+
+```ruby
+RubyLLM::Contract::RakeTask.new do |t|
+  t.context = -> { { adapter: RubyLLM::Contract.configuration.default_adapter } }
+  t.minimum_score = 0.8
+end
+```
+
+## Common patterns
+
+| Old pattern | New pattern |
+|---|---|
+| `LlmClient.new(model:).call(prompt)` | `MyStep.run(input)` |
+| `JSON.parse(response[:content])` | `result.parsed_output` |
+| `begin; rescue; retry; end` | `retry_policy models: [...]` |
+| `body[:temperature] = 0.7` | `temperature 0.7` |
+| `AiCallLog.create(...)` | `around_call { \|s, i, r\| ... }` |
+| `response_format: JsonSchema.build(...)` | `output_schema do...end` |
+| `stub_request(:post, ...)` | `stub_step(MyStep, response: {...})` |
+
+## Anti-patterns
+
+- **Don't migrate markdown/text output services.** The gem is for structured JSON. Prose output gets no benefit from schema validation.
+- **Don't put parallelism in the gem.** Thread management is your app's concern. The gem provides the contract; you call it however you want.
+- **Don't migrate all services at once.** Start with one. Validate the pattern. Then migrate the next.
+
+## Parallel batch generation
+
+The gem handles single calls. You handle parallelism:
+
+```ruby
+class SummarizeBatch < RubyLLM::Contract::Step::Base
+  output_schema do
+    array :summaries do
+      object do
+        string :article_id
+        string :tldr
+      end
+    end
+  end
+  retry_policy models: %w[gpt-4.1-mini gpt-4.1]
+end
+
+# Your orchestrator
+threads = 10.times.map do |i|
+  Thread.new { Rails.application.executor.wrap { SummarizeBatch.run(input(i)) } }
+end
+results = threads.map(&:value)
+```
+
+**Note:** in tests, `stub_step` overrides are thread-local. If your orchestrator spawns threads, propagate overrides manually:
+
+```ruby
+overrides = RubyLLM::Contract.step_adapter_overrides.dup
+Thread.new { RubyLLM::Contract.step_adapter_overrides = overrides; SummarizeBatch.run(input) }
+```
+
+## See also
+
+- [Getting Started](getting_started.md) — the full walkthrough of every feature `SummarizeArticle` uses.
+- [Testing](testing.md) — `stub_step` reference for migrating your test adapter mocks.
+- [Eval-First](eval_first.md) — how to build the "regression" eval from production logs.

data/docs/guide/multimodal_input.md

@@ -0,0 +1,160 @@
+# Multimodal input
+
+> Read this when your contract needs to send a PDF, image, or audio file to the LLM — not just text.
+
+`ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy escalate(...)`, and trace observability still apply. The gem does **not** ship its own multimodal API — it forwards `with:` to `RubyLLM::Chat#ask`, which RubyLLM 1.15+ normalises per provider (Anthropic, OpenAI, Gemini).
+
+## Minimal example
+
+```ruby
+# app/contracts/extract_invoice_data.rb
+class ExtractInvoiceData < RubyLLM::Contract::Step::Base
+  prompt "Extract invoice fields from the attached PDF. Return JSON."
+
+  output_schema do
+    string :vendor
+    string :invoice_number
+    number :total_amount
+    string :currency, enum: %w[USD EUR PLN GBP]
+  end
+
+  validate("currency present") { |o, _| !o[:currency].nil? }
+
+  # REQUIRED when max_cost is set and the contract receives an attachment.
+  # Conservative estimate of attachment input tokens (provider/model-specific).
+  attachment_token_estimate 15_000   # ~12 PDF pages at ~1250 tokens/page
+
+  max_cost 0.10
+
+  retry_policy do
+    escalate "gpt-4.1-mini",
+             { model: "gpt-5", reasoning_effort: "high" }
+  end
+end
+
+result = ExtractInvoiceData.run(
+  "Look for vendor, amount, currency.",
+  context: { attachment: "tmp/invoice.pdf" }
+)
+
+result.status         # => :ok
+result.parsed_output  # => { vendor: "...", invoice_number: "...", ... }
+result.trace[:cost]   # => 0.0042  (total across attempts)
+```
+
+## How it works
+
+1. **Input vs attachment.** The `input` argument to `Step.run` is the text prompt. The attachment travels via `context: { attachment: ... }` — opaque to the contract layer, forwarded to the adapter.
+2. **Adapter forwards `with:`.** `RubyLLM::Contract::Adapters::RubyLLM` reads `options[:attachment]` and passes it to `chat.ask(content, with: attachment)`. RubyLLM picks the right wire format per provider.
+3. **Multi-attachment supported.** `with: [pdf1, pdf2]` or `with: { images: [...], pdfs: [...] }` works natively (`RubyLLM::Content#process_attachments`).
+4. **`with: nil` is a no-op.** Text-only contracts unaffected — the kwarg defaults to nil.
+
+## Cost: `attachment_token_estimate` is required
+
+The gem cannot count attachment input tokens precisely — vision/PDF token cost depends on model, image resolution, page count, and provider. To keep `max_cost` and `max_input` fail-closed, you declare a **conservative estimate** of attachment input tokens at the class level:
+
+```ruby
+class TranscribePDF < RubyLLM::Contract::Step::Base
+  # ...
+  attachment_token_estimate 15_000   # safe upper bound for ~12-page docs
+  max_cost 0.05
+end
+```
+
+The same estimate applies to:
+
+- **Runtime** (`limit_checker`) — adds the estimate to `input_tokens` before checking `max_cost`/`max_input`. Refuses pre-flight if budget exceeded.
+- **Pre-flight** (`estimate_cost`) — accepts `attachment:` kwarg; same accounting. No drift between estimate and runtime decisions.
+
+### Fail-closed without estimate
+
+If your contract has `max_cost` or `max_input` set, receives an attachment, and `attachment_token_estimate` is **not declared**, the call fails with `:limit_exceeded` — the gem refuses to spend money on cost it cannot bound.
+
+```ruby
+class MyContract < RubyLLM::Contract::Step::Base
+  max_cost 0.05
+  # no attachment_token_estimate declared
+end
+
+result = MyContract.run("text", context: { attachment: "doc.pdf" })
+result.status # => :limit_exceeded
+result.validation_errors # => ["attachment present but attachment_token_estimate not set; ..."]
+```
+
+### Opting out per-step
+
+If you do not want fail-closed (e.g., experimental or development contracts), set:
+
+```ruby
+class FlexibleContract < RubyLLM::Contract::Step::Base
+  on_unknown_attachment_size :warn   # log a warning instead of refusing
+  max_cost 0.05
+end
+```
+
+`:warn` is per-step. There is no global opt-out — the same invariant as `on_unknown_pricing`.
+
+## Pre-flight cost estimation
+
+`estimate_cost` accepts an optional `attachment:` kwarg for budget planning:
+
+```ruby
+ExtractInvoiceData.estimate_cost(
+  input: "Look for vendor, amount...",
+  attachment: "tmp/invoice.pdf"
+)
+# => { model: "gpt-4.1-mini",
+#      input_tokens: 15_320,
+#      output_tokens_estimate: 256,
+#      estimated_cost: 0.0123 }
+```
+
+The `input_tokens` figure includes both the text estimate (chars/4 heuristic) AND the declared `attachment_token_estimate`. Pre-flight refusal mirrors runtime: if `attachment` is passed and `attachment_token_estimate` is not declared, `estimate_cost` returns nil and emits the same fail-closed reason.
+
+**Note on output tokens.** `attachment_token_estimate` adds to `input_tokens` only — not to `output_tokens_estimate`. Vision-heavy responses (long image descriptions, transcribed paragraphs) may exceed the conservative `output_tokens_estimate` default. Treat `estimated_cost` as a floor for budget planning, not a precise predictor; inflate `max_output` or `max_cost` accordingly if your prompt routinely produces verbose descriptions.
+
+## Calibrating `attachment_token_estimate`
+
+The number depends on provider, model, and content shape. Some baselines:
+
+| Content                       | Provider | Approx tokens |
+|-------------------------------|----------|---------------|
+| 1 PDF page (text-heavy)       | OpenAI   | ~1000-1500    |
+| 1 PDF page (text-heavy)       | Anthropic | ~1000-1500   |
+| 1 image (1024x1024, low res)  | OpenAI   | ~85           |
+| 1 image (1024x1024, high res) | OpenAI   | ~765          |
+| 1 image                       | Anthropic | ~1500 max    |
+| 1 image                       | Gemini   | ~258 (fixed)  |
+
+Pick a value at or above the provider's worst-case. The estimate is a **floor for safety**, not a precise count — use it to gate budget refusal, not to predict exact cost.
+
+## Multi-turn caveat
+
+If your contract uses history (`add_history`), attachments from prior turns are **not** replayed in 0.10.x. Single-turn multimodal works; follow-up questions on the same document require additional work that is deferred to a later release. See [ADR-0022](../decisions/ADR-0022-v09-multimodal-input.md) (internal) for the rationale.
+
+## Provider notes
+
+- **OpenAI** — PDFs sent as `type: 'file'` with `file_data` (base64). Images as `image_url`. Audio as `input_audio`. Vision pricing varies by image detail; check the model card.
+- **Anthropic** — PDFs sent as `type: 'document'` with `source.type: 'base64'` or `'url'` (auto-selected). Images same. Page limit ~100 per call.
+- **Gemini** — Everything via `inline_data` with `mime_type`. Multimodal token counting is unified.
+
+RubyLLM dispatches on `attachment.type` (`:image`, `:pdf`, `:audio`, `:text`, `:unknown`). Tempfiles must have the right extension (`.pdf`, `.png`, etc.) — RubyLLM detects MIME from the filename; an unsuffixed tempfile becomes `:unknown` and is rejected by the provider.
+
+## Testing contracts with attachments
+
+The Test adapter ignores the `attachment` context key (deterministic responses by step). To verify your adapter call shape, stub `RubyLLM::Chat` directly:
+
+```ruby
+RSpec.describe ExtractInvoiceData do
+  it "forwards attachment to chat.ask" do
+    expect_any_instance_of(RubyLLM::Chat).to receive(:ask)
+      .with(anything, with: "fixtures/invoice.pdf")
+      .and_return(double(content: '{"vendor":"X",...}', input_tokens: 200, output_tokens: 50))
+
+    result = described_class.run("extract", context: { attachment: "fixtures/invoice.pdf" })
+    expect(result.status).to eq(:ok)
+  end
+end
+```
+
+For pre-flight estimate tests, just call `.estimate_cost(input: ..., attachment: ...)` — no adapter stub needed.

data/docs/guide/optimizing_retry_policy.md

@@ -0,0 +1,131 @@
+# Find the cheapest viable fallback list
+
+> Read this when you have 2+ evals and want to know empirically — not by guessing — which models belong in `retry_policy` and in what order.
+
+You defined `SummarizeArticle` in the [README](../../README.md) with `retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]`. That list was a guess. `optimize_retry_policy` tells you which models your evals actually need, so you stop paying for the strong model when `nano` was enough — or stop shipping `nano` when the hardest eval proves it isn't.
+
+## Requirements
+
+- **`SummarizeArticle` already has `retry_policy`.** If your step has none, add one first ([getting started](getting_started.md)).
+- **2–3 evals per step.** One eval optimizes for one scenario; with only `smoke`, you get a recommendation that passes smoke but may miss production edge cases. See [eval-first](eval_first.md).
+- **Rake tasks.** The standard `RubyLLM::Contract::RakeTask` includes `ruby_llm_contract:optimize`. Non-Rails projects: set `EVAL_DIRS=...`.
+
+> **Two orthogonal dimensions to a retry chain.** A chain element is `{ model:, reasoning_effort: }` — model identity AND thinking budget. `optimize_retry_policy` explores both. You can also fix the thinking config at class level via `thinking effort: :low` (or alias `reasoning_effort :low`) on the Step — it becomes the default for every chain element unless an override is passed. See the `thinking` DSL note at the bottom of this guide.
+
+For this guide, assume `SummarizeArticle` has three evals:
+
+```ruby
+SummarizeArticle.define_eval("smoke")          { ... }  # short news article
+SummarizeArticle.define_eval("dense_article")  { ... }  # long form, 5 takeaways required
+SummarizeArticle.define_eval("critical_tone")  { ... }  # negative review, tone must match
+```
+
+## Offline check first
+
+Run once offline to verify the wiring:
+
+```bash
+rake ruby_llm_contract:optimize \
+  STEP=SummarizeArticle \
+  CANDIDATES=gpt-4.1-nano,gpt-4.1-mini@low,gpt-4.1-mini,gpt-4.1
+```
+
+Offline uses each eval's `sample_response` — zero API calls. **Every candidate gets the same score** because they all receive the canned response. That's fine for a smoke test (verifying evals load, candidates parse, output renders) but it doesn't compare model quality. For real optimization, go live.
+
+## Optimize against real models
+
+```bash
+LIVE=1 RUNS=3 rake ruby_llm_contract:optimize \
+  STEP=SummarizeArticle \
+  CANDIDATES=gpt-4.1-nano,gpt-4.1-mini@low,gpt-4.1-mini,gpt-4.1
+```
+
+`LIVE=1` makes real API calls. `RUNS=3` averages each `(candidate, eval)` pair over three runs — necessary because OpenAI forces `temperature=1.0` on gpt-5 / o-series and the same pair can score `0.00` on one run and `1.00` on the next.
+
+Output (illustrative):
+
+```
+SummarizeArticle — fallback list optimization
+
+  eval             4.1-nano  4.1-mini@low  4.1-mini   4.1
+  ---------------------------------------------------------
+  smoke                1.00          1.00      1.00  1.00
+  dense_article        0.67 ←        1.00      1.00  1.00
+  critical_tone        0.50 ←        0.67 ←    1.00  1.00
+
+  Hardest eval: critical_tone
+
+  Suggested fallback list:
+    gpt-4.1-nano         — covers 1 eval(s)
+    gpt-4.1-mini         — passes all 3 evals
+
+  DSL:
+    retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini]
+```
+
+Reading the table:
+- **`←` marks scores below threshold in the hardest eval.** Not a selection hint — just "this candidate fails the row that matters most".
+- **Hardest eval** = the one that forces the strong fallback. Here, `critical_tone` demands `gpt-4.1-mini`.
+- **Suggested fallback list** = the shortest chain where each step covers more evals, built greedy-cheapest-first. Stops when all evals pass. Order matters: `gpt-4.1-nano` is tried first; on validation failure, the gem falls back to `gpt-4.1-mini`.
+
+Copy the DSL, paste into your step, verify with `rake ruby_llm_contract:eval`. You just dropped `gpt-4.1` from the chain — most requests finish on nano, mini handles what nano misses, and the strong model was never needed.
+
+## Measure effective cost before shipping
+
+`optimize` shows **first-attempt** cost. In production, a candidate whose validator rejects 20% of outputs actually costs `first_try_cost + fallback_cost × 0.20` per successful output. The first-attempt number hides this.
+
+`production_mode: { fallback: "..." }` runs each candidate with a runtime `[candidate, fallback]` chain and reports effective cost:
+
+```ruby
+SummarizeArticle.compare_models(
+  "dense_article",
+  candidates: [{ model: "gpt-5-nano" }, { model: "gpt-5-mini", reasoning_effort: "low" }],
+  production_mode: { fallback: "gpt-5-mini" }
+).print_summary
+```
+
+Output (live mode, illustrative):
+
+```
+dense_article — model comparison
+
+  Chain                                    first-attempt  fallback %  effective cost  latency   score
+  ---------------------------------------------------------------------------------------------------
+  gpt-5-nano → gpt-5-mini                  $0.0010        33%         $0.0018         164ms     1.00
+  gpt-5-mini (effort: low) → gpt-5-mini    $0.0015         5%         $0.0016         210ms     1.00
+  gpt-5-mini                               $0.0030         —          $0.0030         220ms     1.00
+```
+
+- **first-attempt** — cost of the first run alone.
+- **fallback %** — fraction of cases where the validator rejected and the fallback ran.
+- **effective cost** — total per successful output including retries.
+- **`—`** — candidate equals fallback, no chain to observe.
+
+Run this before finalizing: a candidate saving 3× on first-attempt but falling back 60% of the time may save only 1.2× in production.
+
+**Scope.** Single-fallback (2-tier) chains only. Multi-tier inspect via `trace.attempts`. Step-level — calling on `Pipeline::Base` raises `ArgumentError`.
+
+## When results look wrong
+
+- **"No viable chain" from a single live run.** Re-run with `RUNS=3`. If scores jump, the first run was noise. Never trust single-run results with gpt-5 / o-series in the pool — `temperature=1.0` is server-enforced.
+- **Every candidate fails the same eval**, including the strongest. The eval is rejecting correct answers. Run the step directly (`context: { retry_policy_override: nil, model: "gpt-4.1" }`), inspect the output, compare with the `verify` block. Loosen the eval if the output is correct but not one of the accepted values.
+- **Testing one specific hypothesis.** (e.g. "does `mini@medium` help on `critical_tone`?") Use `SummarizeArticle.compare_models("critical_tone", candidates: [{ model: "gpt-5-mini", reasoning_effort: "medium" }], runs: 3)` directly — three calls instead of rerunning the whole optimize pass.
+
+## Programmatic API names
+
+Metrics exposed on `Report` / `AggregatedReport` keep their original names: `single_shot_cost`, `single_shot_latency_ms`, `escalation_rate`. The optimize Result struct also exposes `hardest_eval` as an alias for `constraining_eval`.
+
+## thinking DSL note
+
+Set the default reasoning effort once on the Step class — mirrors `RubyLLM::Agent.thinking` exactly:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  model "gpt-5-nano"
+  thinking effort: :low          # canonical
+  # or
+  reasoning_effort :low          # alias for thinking(effort: :low)
+end
+```
+
+Forwarded to `Chat#with_thinking(**)` through the adapter — works provider-agnostically (OpenAI `reasoning_effort`, Anthropic extended-thinking budget). A per-call override via `context: { reasoning_effort: :high }` still wins over the class default.