ruby_llm-contract 0.8.0 → 0.10.2

data/docs/guide/multimodal_input.md

@@ -0,0 +1,160 @@
+# Multimodal input
+
+> Read this when your contract needs to send a PDF, image, or audio file to the LLM — not just text.
+
+`ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy escalate(...)`, and trace observability still apply. The gem does **not** ship its own multimodal API — it forwards `with:` to `RubyLLM::Chat#ask`, which RubyLLM 1.15+ normalises per provider (Anthropic, OpenAI, Gemini).
+
+## Minimal example
+
+```ruby
+# app/contracts/extract_invoice_data.rb
+class ExtractInvoiceData < RubyLLM::Contract::Step::Base
+  prompt "Extract invoice fields from the attached PDF. Return JSON."
+
+  output_schema do
+    string :vendor
+    string :invoice_number
+    number :total_amount
+    string :currency, enum: %w[USD EUR PLN GBP]
+  end
+
+  validate("currency present") { |o, _| !o[:currency].nil? }
+
+  # REQUIRED when max_cost is set and the contract receives an attachment.
+  # Conservative estimate of attachment input tokens (provider/model-specific).
+  attachment_token_estimate 15_000   # ~12 PDF pages at ~1250 tokens/page
+
+  max_cost 0.10
+
+  retry_policy do
+    escalate "gpt-4.1-mini",
+             { model: "gpt-5", reasoning_effort: "high" }
+  end
+end
+
+result = ExtractInvoiceData.run(
+  "Look for vendor, amount, currency.",
+  context: { attachment: "tmp/invoice.pdf" }
+)
+
+result.status         # => :ok
+result.parsed_output  # => { vendor: "...", invoice_number: "...", ... }
+result.trace[:cost]   # => 0.0042  (total across attempts)
+```
+
+## How it works
+
+1. **Input vs attachment.** The `input` argument to `Step.run` is the text prompt. The attachment travels via `context: { attachment: ... }` — opaque to the contract layer, forwarded to the adapter.
+2. **Adapter forwards `with:`.** `RubyLLM::Contract::Adapters::RubyLLM` reads `options[:attachment]` and passes it to `chat.ask(content, with: attachment)`. RubyLLM picks the right wire format per provider.
+3. **Multi-attachment supported.** `with: [pdf1, pdf2]` or `with: { images: [...], pdfs: [...] }` works natively (`RubyLLM::Content#process_attachments`).
+4. **`with: nil` is a no-op.** Text-only contracts unaffected — the kwarg defaults to nil.
+
+## Cost: `attachment_token_estimate` is required
+
+The gem cannot count attachment input tokens precisely — vision/PDF token cost depends on model, image resolution, page count, and provider. To keep `max_cost` and `max_input` fail-closed, you declare a **conservative estimate** of attachment input tokens at the class level:
+
+```ruby
+class TranscribePDF < RubyLLM::Contract::Step::Base
+  # ...
+  attachment_token_estimate 15_000   # safe upper bound for ~12-page docs
+  max_cost 0.05
+end
+```
+
+The same estimate applies to:
+
+- **Runtime** (`limit_checker`) — adds the estimate to `input_tokens` before checking `max_cost`/`max_input`. Refuses pre-flight if budget exceeded.
+- **Pre-flight** (`estimate_cost`) — accepts `attachment:` kwarg; same accounting. No drift between estimate and runtime decisions.
+
+### Fail-closed without estimate
+
+If your contract has `max_cost` or `max_input` set, receives an attachment, and `attachment_token_estimate` is **not declared**, the call fails with `:limit_exceeded` — the gem refuses to spend money on cost it cannot bound.
+
+```ruby
+class MyContract < RubyLLM::Contract::Step::Base
+  max_cost 0.05
+  # no attachment_token_estimate declared
+end
+
+result = MyContract.run("text", context: { attachment: "doc.pdf" })
+result.status # => :limit_exceeded
+result.validation_errors # => ["attachment present but attachment_token_estimate not set; ..."]
+```
+
+### Opting out per-step
+
+If you do not want fail-closed (e.g., experimental or development contracts), set:
+
+```ruby
+class FlexibleContract < RubyLLM::Contract::Step::Base
+  on_unknown_attachment_size :warn   # log a warning instead of refusing
+  max_cost 0.05
+end
+```
+
+`:warn` is per-step. There is no global opt-out — the same invariant as `on_unknown_pricing`.
+
+## Pre-flight cost estimation
+
+`estimate_cost` accepts an optional `attachment:` kwarg for budget planning:
+
+```ruby
+ExtractInvoiceData.estimate_cost(
+  input: "Look for vendor, amount...",
+  attachment: "tmp/invoice.pdf"
+)
+# => { model: "gpt-4.1-mini",
+#      input_tokens: 15_320,
+#      output_tokens_estimate: 256,
+#      estimated_cost: 0.0123 }
+```
+
+The `input_tokens` figure includes both the text estimate (chars/4 heuristic) AND the declared `attachment_token_estimate`. Pre-flight refusal mirrors runtime: if `attachment` is passed and `attachment_token_estimate` is not declared, `estimate_cost` returns nil and emits the same fail-closed reason.
+
+**Note on output tokens.** `attachment_token_estimate` adds to `input_tokens` only — not to `output_tokens_estimate`. Vision-heavy responses (long image descriptions, transcribed paragraphs) may exceed the conservative `output_tokens_estimate` default. Treat `estimated_cost` as a floor for budget planning, not a precise predictor; inflate `max_output` or `max_cost` accordingly if your prompt routinely produces verbose descriptions.
+
+## Calibrating `attachment_token_estimate`
+
+The number depends on provider, model, and content shape. Some baselines:
+
+| Content                       | Provider | Approx tokens |
+|-------------------------------|----------|---------------|
+| 1 PDF page (text-heavy)       | OpenAI   | ~1000-1500    |
+| 1 PDF page (text-heavy)       | Anthropic | ~1000-1500   |
+| 1 image (1024x1024, low res)  | OpenAI   | ~85           |
+| 1 image (1024x1024, high res) | OpenAI   | ~765          |
+| 1 image                       | Anthropic | ~1500 max    |
+| 1 image                       | Gemini   | ~258 (fixed)  |
+
+Pick a value at or above the provider's worst-case. The estimate is a **floor for safety**, not a precise count — use it to gate budget refusal, not to predict exact cost.
+
+## Multi-turn caveat
+
+If your contract uses history (`add_history`), attachments from prior turns are **not** replayed in 0.10.x. Single-turn multimodal works; follow-up questions on the same document require additional work that is deferred to a later release. See [ADR-0022](../decisions/ADR-0022-v09-multimodal-input.md) (internal) for the rationale.
+
+## Provider notes
+
+- **OpenAI** — PDFs sent as `type: 'file'` with `file_data` (base64). Images as `image_url`. Audio as `input_audio`. Vision pricing varies by image detail; check the model card.
+- **Anthropic** — PDFs sent as `type: 'document'` with `source.type: 'base64'` or `'url'` (auto-selected). Images same. Page limit ~100 per call.
+- **Gemini** — Everything via `inline_data` with `mime_type`. Multimodal token counting is unified.
+
+RubyLLM dispatches on `attachment.type` (`:image`, `:pdf`, `:audio`, `:text`, `:unknown`). Tempfiles must have the right extension (`.pdf`, `.png`, etc.) — RubyLLM detects MIME from the filename; an unsuffixed tempfile becomes `:unknown` and is rejected by the provider.
+
+## Testing contracts with attachments
+
+The Test adapter ignores the `attachment` context key (deterministic responses by step). To verify your adapter call shape, stub `RubyLLM::Chat` directly:
+
+```ruby
+RSpec.describe ExtractInvoiceData do
+  it "forwards attachment to chat.ask" do
+    expect_any_instance_of(RubyLLM::Chat).to receive(:ask)
+      .with(anything, with: "fixtures/invoice.pdf")
+      .and_return(double(content: '{"vendor":"X",...}', input_tokens: 200, output_tokens: 50))
+
+    result = described_class.run("extract", context: { attachment: "fixtures/invoice.pdf" })
+    expect(result.status).to eq(:ok)
+  end
+end
+```
+
+For pre-flight estimate tests, just call `.estimate_cost(input: ..., attachment: ...)` — no adapter stub needed.

data/docs/guide/optimizing_retry_policy.md

@@ -0,0 +1,131 @@
+# Find the cheapest viable fallback list
+
+> Read this when you have 2+ evals and want to know empirically — not by guessing — which models belong in `retry_policy` and in what order.
+
+You defined `SummarizeArticle` in the [README](../../README.md) with `retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]`. That list was a guess. `optimize_retry_policy` tells you which models your evals actually need, so you stop paying for the strong model when `nano` was enough — or stop shipping `nano` when the hardest eval proves it isn't.
+
+## Requirements
+
+- **`SummarizeArticle` already has `retry_policy`.** If your step has none, add one first ([getting started](getting_started.md)).
+- **2–3 evals per step.** One eval optimizes for one scenario; with only `smoke`, you get a recommendation that passes smoke but may miss production edge cases. See [eval-first](eval_first.md).
+- **Rake tasks.** The standard `RubyLLM::Contract::RakeTask` includes `ruby_llm_contract:optimize`. Non-Rails projects: set `EVAL_DIRS=...`.
+
+> **Two orthogonal dimensions to a retry chain.** A chain element is `{ model:, reasoning_effort: }` — model identity AND thinking budget. `optimize_retry_policy` explores both. You can also fix the thinking config at class level via `thinking effort: :low` (or alias `reasoning_effort :low`) on the Step — it becomes the default for every chain element unless an override is passed. See the `thinking` DSL note at the bottom of this guide.
+
+For this guide, assume `SummarizeArticle` has three evals:
+
+```ruby
+SummarizeArticle.define_eval("smoke")          { ... }  # short news article
+SummarizeArticle.define_eval("dense_article")  { ... }  # long form, 5 takeaways required
+SummarizeArticle.define_eval("critical_tone")  { ... }  # negative review, tone must match
+```
+
+## Offline check first
+
+Run once offline to verify the wiring:
+
+```bash
+rake ruby_llm_contract:optimize \
+  STEP=SummarizeArticle \
+  CANDIDATES=gpt-4.1-nano,gpt-4.1-mini@low,gpt-4.1-mini,gpt-4.1
+```
+
+Offline uses each eval's `sample_response` — zero API calls. **Every candidate gets the same score** because they all receive the canned response. That's fine for a smoke test (verifying evals load, candidates parse, output renders) but it doesn't compare model quality. For real optimization, go live.
+
+## Optimize against real models
+
+```bash
+LIVE=1 RUNS=3 rake ruby_llm_contract:optimize \
+  STEP=SummarizeArticle \
+  CANDIDATES=gpt-4.1-nano,gpt-4.1-mini@low,gpt-4.1-mini,gpt-4.1
+```
+
+`LIVE=1` makes real API calls. `RUNS=3` averages each `(candidate, eval)` pair over three runs — necessary because OpenAI forces `temperature=1.0` on gpt-5 / o-series and the same pair can score `0.00` on one run and `1.00` on the next.
+
+Output (illustrative):
+
+```
+SummarizeArticle — fallback list optimization
+
+  eval             4.1-nano  4.1-mini@low  4.1-mini   4.1
+  ---------------------------------------------------------
+  smoke                1.00          1.00      1.00  1.00
+  dense_article        0.67 ←        1.00      1.00  1.00
+  critical_tone        0.50 ←        0.67 ←    1.00  1.00
+
+  Hardest eval: critical_tone
+
+  Suggested fallback list:
+    gpt-4.1-nano         — covers 1 eval(s)
+    gpt-4.1-mini         — passes all 3 evals
+
+  DSL:
+    retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini]
+```
+
+Reading the table:
+- **`←` marks scores below threshold in the hardest eval.** Not a selection hint — just "this candidate fails the row that matters most".
+- **Hardest eval** = the one that forces the strong fallback. Here, `critical_tone` demands `gpt-4.1-mini`.
+- **Suggested fallback list** = the shortest chain where each step covers more evals, built greedy-cheapest-first. Stops when all evals pass. Order matters: `gpt-4.1-nano` is tried first; on validation failure, the gem falls back to `gpt-4.1-mini`.
+
+Copy the DSL, paste into your step, verify with `rake ruby_llm_contract:eval`. You just dropped `gpt-4.1` from the chain — most requests finish on nano, mini handles what nano misses, and the strong model was never needed.
+
+## Measure effective cost before shipping
+
+`optimize` shows **first-attempt** cost. In production, a candidate whose validator rejects 20% of outputs actually costs `first_try_cost + fallback_cost × 0.20` per successful output. The first-attempt number hides this.
+
+`production_mode: { fallback: "..." }` runs each candidate with a runtime `[candidate, fallback]` chain and reports effective cost:
+
+```ruby
+SummarizeArticle.compare_models(
+  "dense_article",
+  candidates: [{ model: "gpt-5-nano" }, { model: "gpt-5-mini", reasoning_effort: "low" }],
+  production_mode: { fallback: "gpt-5-mini" }
+).print_summary
+```
+
+Output (live mode, illustrative):
+
+```
+dense_article — model comparison
+
+  Chain                                    first-attempt  fallback %  effective cost  latency   score
+  ---------------------------------------------------------------------------------------------------
+  gpt-5-nano → gpt-5-mini                  $0.0010        33%         $0.0018         164ms     1.00
+  gpt-5-mini (effort: low) → gpt-5-mini    $0.0015         5%         $0.0016         210ms     1.00
+  gpt-5-mini                               $0.0030         —          $0.0030         220ms     1.00
+```
+
+- **first-attempt** — cost of the first run alone.
+- **fallback %** — fraction of cases where the validator rejected and the fallback ran.
+- **effective cost** — total per successful output including retries.
+- **`—`** — candidate equals fallback, no chain to observe.
+
+Run this before finalizing: a candidate saving 3× on first-attempt but falling back 60% of the time may save only 1.2× in production.
+
+**Scope.** Single-fallback (2-tier) chains only. Multi-tier inspect via `trace.attempts`. Step-level — calling on `Pipeline::Base` raises `ArgumentError`.
+
+## When results look wrong
+
+- **"No viable chain" from a single live run.** Re-run with `RUNS=3`. If scores jump, the first run was noise. Never trust single-run results with gpt-5 / o-series in the pool — `temperature=1.0` is server-enforced.
+- **Every candidate fails the same eval**, including the strongest. The eval is rejecting correct answers. Run the step directly (`context: { retry_policy_override: nil, model: "gpt-4.1" }`), inspect the output, compare with the `verify` block. Loosen the eval if the output is correct but not one of the accepted values.
+- **Testing one specific hypothesis.** (e.g. "does `mini@medium` help on `critical_tone`?") Use `SummarizeArticle.compare_models("critical_tone", candidates: [{ model: "gpt-5-mini", reasoning_effort: "medium" }], runs: 3)` directly — three calls instead of rerunning the whole optimize pass.
+
+## Programmatic API names
+
+Metrics exposed on `Report` / `AggregatedReport` keep their original names: `single_shot_cost`, `single_shot_latency_ms`, `escalation_rate`. The optimize Result struct also exposes `hardest_eval` as an alias for `constraining_eval`.
+
+## thinking DSL note
+
+Set the default reasoning effort once on the Step class — mirrors `RubyLLM::Agent.thinking` exactly:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  model "gpt-5-nano"
+  thinking effort: :low          # canonical
+  # or
+  reasoning_effort :low          # alias for thinking(effort: :low)
+end
+```
+
+Forwarded to `Chat#with_thinking(**)` through the adapter — works provider-agnostically (OpenAI `reasoning_effort`, Anthropic extended-thinking budget). A per-call override via `context: { reasoning_effort: :high }` still wins over the class default.

data/docs/guide/output_schema.md

@@ -0,0 +1,93 @@
+# Output Schema
+
+> Read this as a reference for the schema DSL — every constraint, nested objects, arrays of objects, the full pattern table.
+
+Declare the expected output structure using [ruby_llm-schema](https://github.com/danielfriis/ruby_llm-schema) DSL. The schema serves **two purposes**:
+
+1. **Output validation** — replaces type and shape checks (enums, ranges, required fields). One declaration instead of many.
+2. **Provider-side request** — with the RubyLLM adapter, the schema is sent to the LLM provider via `chat.with_schema(...)`, asking the model to return JSON matching the shape. Cheaper models sometimes ignore the request, which is why client-side validation (point 1) still matters.
+
+> **Same DSL `RubyLLM::Agent.schema` accepts.** `output_schema do ... end` here is a wrapper around `RubyLLM::Schema.create(&block)` plus a client-side validation step. `RubyLLM::Agent.schema` accepts the same block — choosing one over the other does not change the schema language. The difference: `Agent.schema` lets you pass a `Proc` evaluated in runtime context (dynamic per-call schema); `Step.output_schema` is eager-compiled at class load and additionally drives `output_type` inference and `Validator.validate`. Both can coexist.
+
+All examples below extend the `SummarizeArticle` step from the [README](../../README.md).
+
+## Schema replaces type and shape checks
+
+```ruby
+# WITHOUT schema — many validates:
+validate("tldr must be a string")          { |o| o[:tldr].is_a?(String) }
+validate("takeaways must be an array")     { |o| o[:takeaways].is_a?(Array) }
+validate("takeaways 3 to 5")               { |o| (3..5).cover?(o[:takeaways].size) }
+ALLOWED_TONES = %w[neutral positive negative analytical].freeze
+validate("tone must be an allowed label") { |o| ALLOWED_TONES.include?(o[:tone]) }
+
+# WITH schema — one declaration:
+output_schema do
+  string :tldr
+  array  :takeaways, of: :string, min_items: 3, max_items: 5
+  string :tone, enum: %w[neutral positive negative analytical]
+end
+```
+
+## Nested objects in arrays
+
+Use `object do...end` inside `array` when you need more than a primitive per element. Concrete scenario: the UI card grows a "confidence bar" next to each takeaway so editors can see which points the model was sure about vs guessing. That requires `confidence` paired with `text`, not two parallel arrays that could desync. Nested objects make the pairing a schema invariant:
+
+```ruby
+output_schema do
+  string :tldr
+  array :takeaways, min_items: 3, max_items: 5 do
+    object do
+      string :text
+      number :confidence, minimum: 0.0, maximum: 1.0
+    end
+  end
+  string :tone, enum: %w[neutral positive negative analytical]
+end
+```
+
+## Schema pattern reference
+
+| Your output looks like | Schema pattern | Example |
+|---|---|---|
+| `{"tldr": "...", "tone": "positive"}` | Flat fields | `string :tldr; string :tone, enum: [...]` |
+| `{"takeaways": ["...", "..."]}` | Array of primitives | `array :takeaways, of: :string, min_items: 3, max_items: 5` |
+| `{"takeaways": [{"text": "...", "confidence": 0.9}]}` | Array of objects | `array :takeaways do; object do; string :text; number :confidence; end; end` |
+
+Without `object do...end`, `array :takeaways do; string :text; end` tells the provider "takeaways is an array of strings" — not objects. That's what you get back.
+
+## Why schema alone is not enough
+
+Schema validates **shape** — correct types, allowed values, field presence. But LLMs can return structurally valid JSON that is **logically wrong**. Validates catch what schema can't:
+
+```ruby
+output_schema do
+  string :tldr
+  array  :takeaways, of: :string, min_items: 3, max_items: 5
+  string :tone, enum: %w[neutral positive negative analytical]
+end
+
+# Schema allows any string for :tldr — but a 500-char "summary" breaks the UI card.
+validate("TL;DR fits the card") { |o, _| o[:tldr].length <= 200 }
+
+# Schema enforces 3–5 takeaways — but says nothing about them being distinct.
+validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }
+
+# Schema can't express cross-field rules.
+validate("critical tone requires at least one concrete risk") do |o, _|
+  next true unless o[:tone] == "negative"
+  o[:takeaways].any? { |t| t.match?(/fail|break|crash|outage|vulnerab/i) }
+end
+```
+
+## Supported constraints
+
+| Constraint | Types | Example |
+|---|---|---|
+| `enum` | string, integer | `string :tone, enum: %w[neutral positive negative analytical]` |
+| `minimum` / `maximum` | number, integer | `number :confidence, minimum: 0.0, maximum: 1.0` |
+| `min_length` / `max_length` | string | `string :tldr, min_length: 1, max_length: 200` |
+| `min_items` / `max_items` | array | `array :takeaways, of: :string, min_items: 3, max_items: 5` |
+| `additional_properties` | object | Set to `false` in the schema to reject extra keys |
+
+Keyword args use Ruby snake_case (`min_length`, `min_items`). The DSL converts them internally to JSON Schema's camelCase (`minLength`, `minItems`) before sending the schema to the provider — you don't need to write camelCase in Ruby.

data/docs/guide/pipeline.md

@@ -0,0 +1,154 @@
+# Pipeline
+
+> Read this when one step isn't enough — you need multi-step with fail-fast, automatic data threading, and per-step models.
+
+Chain multiple steps with automatic data threading, fail-fast, per-step models, trace, and timeout.
+
+A pipeline needs more than one step to be interesting. This guide grows the `SummarizeArticle` step from the [README](../../README.md) into a three-step content pipeline that tags and routes the summary to a UI card.
+
+## Full example: article → summary → hashtags → card
+
+```ruby
+# Step 1 — the flagship step from README, unchanged.
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  prompt <<~PROMPT
+    Summarize this article for a UI card. Return a short TL;DR,
+    3 to 5 key takeaways, and a tone label.
+
+    {input}
+  PROMPT
+
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
+  end
+
+  validate("TL;DR fits the card") { |o, _| o[:tldr].length <= 200 }
+end
+
+# Step 2 — reads SummarizeArticle's output, produces hashtags suitable for social posts.
+class GenerateHashtags < RubyLLM::Contract::Step::Base
+  input_type Hash
+
+  output_schema do
+    # Carry through the summary fields downstream consumers (and the next step) need.
+    string :tldr
+    array  :takeaways, of: :string
+    string :tone, enum: %w[neutral positive negative analytical]
+    # Add new field.
+    array  :hashtags, of: :string, min_items: 2, max_items: 5
+  end
+
+  prompt do
+    rule "Preserve tldr / takeaways / tone exactly as given."
+    user "Article summary: {tldr}\nTone: {tone}\nGenerate 2 to 5 concise hashtags."
+  end
+
+  validate("tone preserved") { |o, input| o[:tone] == input[:tone] }
+end
+
+# Step 3 — final shape the UI card consumes.
+class BuildArticleCard < RubyLLM::Contract::Step::Base
+  input_type Hash
+
+  output_schema do
+    string :headline
+    string :summary
+    array  :hashtags, of: :string
+    string :sentiment_icon, enum: %w[😐 🙂 ⚠️ 🧠]
+  end
+
+  prompt do
+    rule "Headline <= 70 chars. Summary is the incoming tldr reprinted verbatim."
+    rule "Pick sentiment_icon from: 😐 neutral, 🙂 positive, ⚠️ negative, 🧠 analytical."
+    user "TL;DR: {tldr}\nTone: {tone}\nHashtags: {hashtags}"
+  end
+
+  validate("summary is the tldr verbatim") { |o, input| o[:summary] == input[:tldr] }
+end
+
+# Pipeline: summarize → hashtags → card
+class ArticleCardPipeline < RubyLLM::Contract::Pipeline::Base
+  step SummarizeArticle, as: :summarize
+  step GenerateHashtags, as: :tag
+  step BuildArticleCard, as: :card
+end
+```
+
+## Running and inspecting
+
+```ruby
+result = ArticleCardPipeline.run(article_text, context: { adapter: adapter })
+result.ok?                          # => true
+result.outputs_by_step[:summarize]  # => { tldr: "...", takeaways: [...], tone: "analytical" }
+result.outputs_by_step[:card]       # => { headline: "...", summary: "...", ... }
+result.trace.total_cost             # => 0.000128 (all steps combined)
+result.trace.total_latency_ms       # => 2340
+```
+
+## Fail-fast behavior
+
+When a step's schema, validate, or preflight check rejects the output, the pipeline stops there — downstream steps never run:
+
+```ruby
+# Summarize returns a TL;DR over 200 chars → the "TL;DR fits the card" validate fails
+adapter = RubyLLM::Contract::Adapters::Test.new(response: {
+  tldr: "x" * 500,
+  takeaways: %w[one two three],
+  tone: "neutral"
+})
+
+result = ArticleCardPipeline.run("article text", context: { adapter: adapter })
+result.failed?        # => true
+result.failed_step    # => :summarize (validate rejected; retries exhausted)
+# tag and card never run — no downstream tokens spent on garbage
+```
+
+## Per-step model override
+
+```ruby
+class ArticleCardPipeline < RubyLLM::Contract::Pipeline::Base
+  step SummarizeArticle, as: :summarize, model: "gpt-4.1-mini"
+  step GenerateHashtags, as: :tag,       model: "gpt-4.1-nano"
+  step BuildArticleCard, as: :card,      model: "gpt-4.1-nano"
+end
+```
+
+## Timeout
+
+```ruby
+result = ArticleCardPipeline.run(article_text, timeout_ms: 30_000)
+```
+
+## Pipeline eval
+
+```ruby
+ArticleCardPipeline.define_eval("e2e") do
+  add_case "ruby 3.4 release",
+    input: "Ruby 3.4 ships with frozen string literals by default and better YJIT...",
+    expected: { sentiment_icon: "🧠" }
+end
+
+report = ArticleCardPipeline.run_eval("e2e", context: { model: "gpt-4.1-mini" })
+report.print_summary
+```
+
+## Pretty print
+
+```ruby
+puts result
+# Pipeline: ok  3 steps  1234ms  450+120 tokens  trace=abc12345
+
+result.pretty_print
+# Full ASCII table with per-step outputs (Pipeline::Result)
+
+# For eval reports, use print_summary instead:
+report.print_summary
+# Tabular pass/fail breakdown (Eval::Report)
+```
+
+## See also
+
+- [Testing](testing.md) — `ArticleCardPipeline.test(..., responses: { summarize: ..., tag: ..., card: ... })` for pipeline-level spec adapters.
+- [Optimizing retry_policy](optimizing_retry_policy.md) — `optimize_retry_policy` runs per-step; pipelines benchmark one step at a time.

data/docs/guide/prompt_ast.md

@@ -0,0 +1,76 @@
+# Prompt AST
+
+> Read this when your prompt has more than one shape (per-tenant, per-language, per-audience) and string concatenation is starting to drift.
+
+Prompts are structured data, not strings. That matters the moment `SummarizeArticle` has to ship in more than one shape — a different audience per tenant, a different language per region, a different tone template for a B2B vs consumer card. Building those variants by string-concatenating a monolithic prompt leads to silent drift across environments. The AST gives you typed nodes (`system`, `rule`, `section`, `example`, `user`) that compose, diff, and snapshot-test cleanly.
+
+Available node types:
+
+```ruby
+prompt do
+  system  "You summarize articles for a UI card."     # system message
+  rule    "Return valid JSON only."                   # appended as separate system message
+  section "AUDIENCE", "Rails developers"              # labeled system message: [AUDIENCE]\n...
+  example input:  "Ruby 3.4 ships frozen strings...", # user/assistant few-shot pair
+          output: '{"tldr":"...","takeaways":[...],"tone":"analytical"}'
+  user    "{input}"                                   # user message with interpolation
+end
+```
+
+Or just a plain string (wraps as a single user message):
+
+```ruby
+prompt "Summarize this article for a UI card. {input}"
+```
+
+The AST is immutable, diffable, and hashable. Useful for snapshot testing and auditing prompt changes.
+
+## Hash inputs with variable interpolation
+
+When input is a Hash, each key becomes a template variable. Concrete scenario: a multi-language newsletter product where the same article has to be summarised in Polish for EU subscribers, English for US, with different audiences per tier (Rails developers vs engineering managers). Hash inputs let one step cover all of these without forking the class:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  input_type RubyLLM::Contract::Types::Hash.schema(
+    article:  RubyLLM::Contract::Types::String,
+    audience: RubyLLM::Contract::Types::String,
+    language: RubyLLM::Contract::Types::String
+  )
+
+  prompt do
+    system  "You summarize articles for a UI card."
+    rule    "Write the TL;DR and takeaways in {language}."
+    section "AUDIENCE", "{audience}"
+    user    "{article}"
+  end
+
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
+  end
+end
+```
+
+Every `{key}` in a prompt node is pulled from the input hash at run time. Missing keys raise — making wire-up bugs loud, not silent.
+
+> **Not the same as `RubyLLM::Agent.inputs`.** `Step.input_type` is a *runtime type check* on the positional argument passed to `run(input)` — it raises `TypeError` if the input violates the declared shape. `RubyLLM::Agent.inputs` is a list of *named template locals* injected into ERB instructions. They solve different problems (validation vs templating) and can coexist on the same project.
+
+> **Not the same as `RubyLLM::Agent` ERB templates either.** The `prompt do ... end` DSL above builds a multi-role message list (`system` / `user` / `assistant` / `example` nodes) into a node-AST. `Agent.instructions :name` loads a single-string ERB file from `app/prompts/<agent_path>/<name>.txt.erb` for the system prompt only. Different output shape, different scope. Variable interpolation here uses `{key}` substitution; ERB uses full `<%= ruby %>`.
+
+## Cross-validating output against input
+
+Validate blocks support 2-arity `|output, input|` so you can check that the model's answer stays faithful to the request:
+
+```ruby
+validate("tldr is not just the article reprinted") do |output, input|
+  # Guard against lazy models that return the input verbatim.
+  output[:tldr].length < input[:article].length / 2
+end
+
+validate("no takeaway repeats the TL;DR") do |output, _input|
+  output[:takeaways].none? { |t| t == output[:tldr] }
+end
+```
+
+The first example uses `input`; the second ignores it. Both are legal 2-arity signatures — Ruby accepts the unused `_input` parameter naming convention.