ruby_llm-contract 0.7.1 → 0.7.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a165b7d8436d99e564fd5e774c601bae38562c0827c226995f1922e75f9987cf
-  data.tar.gz: 4124a3a13caba843448529a55eb1420601cbde70873f5924c792d9ed0c97161b
+  metadata.gz: c9b5adda942a6e1f3393d2d05e8226d210b25088c65c2c20160b2ffc6a493533
+  data.tar.gz: 91110ddd5d2fc39a8e3f6cd9ec912c707ceef7da2ee40e8805598a787ae1c067
 SHA512:
-  metadata.gz: e5f17ef5241d9f7ddc047a16608b607d5fefedeff207cc6c3fb969c937192527c5b49c4213b3a3635db4e6aa68784f10f2363a5d99ed8528740b0d4ecebf3790
-  data.tar.gz: e58336f998f4df510b707534fd5fecdd87c8e4d04187f6ed9512e2b391dd359e3c9c4cfab106022a9a34ed8dfc27669156f5fc4214320ffe9dec3a0176aab181
+  metadata.gz: 543779ceacb617909e7e45355473fb300d9a5f81206ee51019d4f868b3ea66d43272b04be78aa95d0a014ea16619ab57c933eb58f1d17d8fa3845a957d6c750a
+  data.tar.gz: eea6b49a1cb01de6b501bddec6fcfcccbc221a5448a72a435f4c04e80131a7994757b3c337390e1b6212a2709ae082a5d55c82158535a5789e606b7ef13d27f9

data/CHANGELOG.md

@@ -1,5 +1,65 @@
 # Changelog
 
+## 0.7.3 (2026-04-24)
+
+Adoption-friction release. No runtime behavior changes — every delta is in `docs/`, `examples/`, or `spec/integration/` (plus the `version.rb` / Gemfile.lock bumps). Upgrading from 0.7.2 picks up the expanded guide set, the new runnable showcases, and one extra integration spec.
+
+### Documentation
+
+- **New guide: `docs/guide/why.md`** — four production failure modes the gem exists for (schema-valid logically wrong, silent prompt regression, sampling variance on fixed-temperature models, runaway cost). Opens from a concrete incident each time; designed for readers who have not yet felt the pain the gem relieves.
+- **New guide: `docs/guide/rails_integration.md`** — seven Rails-specific FAQs with runnable snippets: where step classes live (`app/contracts/`), initializer setup, background jobs, `around_call` observability, RSpec/Minitest stubs, error handling in controllers, CI gate wiring.
+- **README adoption-friction pass** — added a short "Do I need this?" block after Install, a reading-order hint (`README → why.md → getting_started.md`), and outcome-based labels in the docs index ("Prevent silent prompt regressions" instead of "Eval-First", etc.).
+- **TL;DR box at the top of every guide** — single-sentence orientation for readers who land via search; "Skip if" clause added where real confusion exists (`eval_first.md`, `testing.md`, `migration.md`).
+- **API coverage gaps closed** — `estimate_cost` / `estimate_eval_cost`, `max_cost on_unknown_pricing: :warn`, `run_eval(..., concurrency:)`, `around_call` testing patterns now documented in `getting_started.md`, `eval_first.md`, `testing.md`.
+- **Industry-standard terminology** — `temperature-locked` → `fixed-temperature`, `variance-induced` → `sampling variance`, `severity signals` → `severity keywords`, `takeaway drift` → `tone/takeaways mismatch`.
+- **`docs/architecture.md` refresh** — diagram now reflects the current class layout: added `Step::RetryPolicy`, `Pipeline::Result`, `Eval::AggregatedReport`, `Eval::BaselineDiff`, `Eval::PromptDiffComparator`, `Eval::EvalHistory`, `Eval::RetryOptimizer`, `OptimizeRakeTask`. Replaced the outdated `Eval::TraitEvaluator` entry with `Eval::ExpectationEvaluator`.
+- **Business framing added to guides** — every guide opens with a concrete production scenario or "why it matters" hook before the API reference.
+
+### Examples — consolidated on `SummarizeArticle`, renumbered 00-06
+
+The previous 12-file set mixed a private Reddit promo planner, customer support, meetings, keyword extraction, and translation. The new set is seven runnable files, each answering one adopter question on the README's `SummarizeArticle` case.
+
+| # | File | Answers |
+|---|------|---------|
+| 00 | `00_basics.rb` | How do I start? (seven incremental layers + real-LLM pointer) |
+| 01 | `01_fallback_showcase.rb` | Show me the gem in 30 seconds (zero API keys) |
+| 02 | `02_real_llm_minimal.rb` | How do I plug in a real LLM? (~30 lines) |
+| 03 | `03_summarize_with_keywords.rb` | How does the contract evolve? (growing prompt) |
+| 04 | `04_summarize_and_translate.rb` | Pipeline composition + pipeline-level `run_eval` |
+| 05 | `05_eval_dataset.rb` | How do I stop silent prompt regressions? |
+| 06 | `06_retry_variants.rb` | `attempts: 3`, `reasoning_effort` escalation, cross-provider (Ollama → Anthropic → OpenAI) |
+
+Every file carries an "Expected output" block in its header so readers see the result without running the script. The `docs/ideas/` directory is now fully untracked (already in `.gitignore`; one stray file removed from version control).
+
+### Examples — bug fixes carried along
+
+- **Schema pitfall fixed in 5 files** — `array :x do; string :y; ...; end` silently produces `items: string` and drops every declaration after the first, matching the documented pitfall in `spec/ruby_llm/contract/nested_schema_spec.rb:71`. Every affected array block is now wrapped in `object do...end`.
+- **`examples/05_eval_dataset.rb` (pre-renumber: `09_eval_dataset.rb`) `result[:passed]` → `result.passed?`** — the previous code called `[]` on an `Eval::CaseResult` and raised `NoMethodError` at runtime.
+
+### Testing
+
+- **New `spec/integration/pipeline_eval_spec.rb`** — three cases guaranteeing pipeline-level `run_eval` stays functional: happy path, final-step mismatch, and fail-fast propagation when an intermediate `validate` rejects. Closes the "09 STEP 5 pipeline evaluation" known issue flagged in the 0.7.2 release. The fail-fast case asserts `step_status == :validation_failed` and the validate's label in `details`, so a regression that short-circuits on schema instead of validate would fail loudly.
+
+### Deleted (private-project cleanup)
+
+- `examples/01_classify_threads.rb`, `02_generate_comment.rb`, `03_target_audience.rb`, `10_reddit_full_showcase.rb`, `spec/integration/reddit_pipeline_spec.rb` — Reddit Promo Planner was a separate private project; its examples do not belong in the gem's public repo.
+- `examples/02_output_schema.rb` — fully covered by `docs/guide/output_schema.md`; deleting avoids duplication.
+
+## 0.7.2 (2026-04-22)
+
+### Changed
+
+- **Terminal output labels renamed for consistency with README narrative.** `print_summary` now prints `Hardest eval` (was `Constraining eval`), `Suggested fallback list` (was `Suggested chain`), and the production-mode table uses `first-attempt` / `fallback %` as column headers (was `single-shot` / `escalation`). Programmatic metric names unchanged: `single_shot_cost`, `single_shot_latency_ms`, `escalation_rate`. `RetryOptimizer::Result` exposes `hardest_eval` as an alias for `constraining_eval`.
+
+### Documentation
+
+- **`docs/guide/optimizing_retry_policy.md` rewritten** — 17.7k → 6.4k characters. Continues the `SummarizeArticle` narrative from README. Offline mode clearly positioned as wiring-check; real optimization runs via `LIVE=1 RUNS=3`. Output samples match actual `print_summary` format.
+- **`docs/guide/getting_started.md` rewritten** — 8.7k → 6.1k. Every example uses `SummarizeArticle`. Evals + CI gates section moved before Budget caps. Structured Prompts / Dynamic Prompts / "Already using ruby_llm?" / Reasoning effort sections removed; content delegated to `prompt_ast.md` and README.
+- **`docs/guide/eval_first.md` refined** — 6.3k → 5.0k. Switched to `SummarizeArticle` case. Team workflow section compressed with links back to `getting_started.md` for the matcher chain.
+- **`docs/guide/testing.md` refined** — 10.7k → 7.4k. Switched to `SummarizeArticle` case. Threshold gating / Rake task / baseline walkthrough / prompt A/B sections delegated back to `getting_started.md` and `eval_first.md`.
+- **`docs/guide/output_schema.md` DSL bug fix** — the Supported constraints table documented JSON Schema camelCase keys (`minLength`, `minItems`, `additionalProperties`) that are not valid DSL arguments. Every copy-paste from the previous table would have raised `ArgumentError`. Switched to snake_case (`min_length`, `min_items`, `additional_properties`) as the DSL actually expects; added a short note on the internal camelCase conversion.
+- **`docs/guide/best_practices.md`, `pipeline.md`, `migration.md` sanity pass** — terminology alignment (model escalation → model fallback where narrative; `escalate` DSL method unchanged) and `SummarizeArticle` case where the guide is not inherently multi-step.
+
 ## 0.7.1 (2026-04-22)
 
 ### Changed (behavioral, follow-up to v0.7.0)

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ruby_llm-contract (0.7.1)
+    ruby_llm-contract (0.7.3)
       dry-types (~> 1.7)
       ruby_llm (~> 1.0)
       ruby_llm-schema (~> 0.3)
@@ -258,7 +258,7 @@ CHECKSUMS
   rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   ruby_llm (1.14.0) sha256=57c6f7034fc4a44504ea137d70f853b07824f1c1cdbe774ab3ab3522e7098deb
-  ruby_llm-contract (0.7.1)
+  ruby_llm-contract (0.7.3)
   ruby_llm-schema (0.3.0) sha256=a591edc5ca1b7f0304f0e2261de61ba4b3bea17be09f5cf7558153adfda3dec6
   ruby_parser (3.22.0) sha256=1eb4937cd9eb220aa2d194e352a24dba90aef00751e24c8dfffdb14000f15d23
   rubycritic (4.12.0) sha256=024fed90fe656fa939f6ea80aab17569699ac3863d0b52fd72cb99892247abc8

data/README.md

@@ -1,121 +1,8 @@
 # ruby_llm-contract
 
-**Handle LLM output variance for [ruby_llm](https://github.com/crmne/ruby_llm).** Transport is a solved problem — ruby_llm already retries rate limits, timeouts, and server errors at the Faraday layer. What it can't do: retry when the model returns malformed JSON or a wrong answer, escalate to a smarter model when the cheap one fails the rules, measure variance on your dataset, and gate CI on regressions. That's what this gem adds.
+**Validate and retry LLM outputs for [ruby_llm](https://github.com/crmne/ruby_llm).** Describe the answer you expect (JSON schema + business rules). If the model returns something that doesn't match, retry — optionally falling back to a stronger model — until it passes or you hit the budget.
 
-## Where the boundary sits
-
-| Concern | Handled by |
-|---|---|
-| Rate limits, timeouts, 5xx, connection errors | `ruby_llm` (Faraday retry middleware) |
-| Streaming, tool calls, embeddings, images, transcription | `ruby_llm` |
-| Chat history persistence (`acts_as_chat`) | `ruby_llm` |
-| **Malformed JSON / parse errors** | **`ruby_llm-contract`** |
-| **Business rule violations (invariants, schema)** | **`ruby_llm-contract`** |
-| **Retry with model escalation on bad output** | **`ruby_llm-contract`** |
-| **Measuring output variance on datasets** | **`ruby_llm-contract`** |
-| **Regression detection + CI gates** | **`ruby_llm-contract`** |
-
-Put together: `ruby_llm` owns the wire, this gem owns what the model *said*.
-
-```
-  YOU WRITE                       THE GEM HANDLES                 YOU GET
-  ─────────                       ───────────────                 ───────
-
-  validate { |o| ... }            catch bad answers — combined     Zero garbage
-                                  with retry_policy, auto-retry   in production
-
-  retry_policy                    start cheap, escalate only      Pay for the cheapest
-  models: %w[nano mini full]      when validation fails           model that works
-
-  max_cost 0.01                   estimate tokens, check price,   No surprise bills
-                                  refuse before calling LLM
-
-  output_schema { ... }           send JSON schema to provider,   Zero parsing code
-                                  validate response client-side
-
-  define_eval { ... }             test cases + baselines,          Regressions caught
-                                  run in CI with real LLM          before deploy
-
-  recommend(candidates: [...])    evaluate all configs, pick      Optimal model +
-                                  cheapest that passes            retry chain
-```
-
-## Before and after
-
-```
-  ┌─────────────────────────────────────────────────────────────────┐
-  │ BEFORE: pick one model, hope for the best                      │
-  │                                                                 │
-  │   expensive model → accurate, but you overpay on every call     │
-  │   cheap model     → fast, but wrong answers slip to production  │
-  │   prompt change   → "looks good to me" → deploy → users suffer │
-  └─────────────────────────────────────────────────────────────────┘
-
-                         ⬇  add ruby_llm-contract
-
-  ┌─────────────────────────────────────────────────────────────────┐
-  │ YOU DEFINE A CONTRACT                                            │
-  │                                                                 │
-  │   output_schema { string :priority }       ← valid structure   │
-  │   validate("valid priority") { |o| ... }   ← business rules    │
-  │   retry_policy models: %w[nano mini full]  ← escalation chain  │
-  │   max_cost 0.01                            ← budget cap         │
-  └───────────────────────────┬─────────────────────────────────────┘
-                              │
-                              ▼
-  ┌─────────────────────────────────────────────────────────────────┐
-  │ THE GEM HANDLES THE REST                                        │
-  │                                                                 │
-  │   request ──→ ┌──────┐   ┌──────────┐                           │
-  │               │ nano │─→ │ contract │──→ ✓ pass → done         │
-  │               └──────┘   └────┬─────┘                           │
-  │                               │ ✗ fail                          │
-  │                               ▼                                 │
-  │               ┌──────┐   ┌──────────┐                           │
-  │               │ mini │─→ │ contract │──→ ✓ pass → done         │
-  │               └──────┘   └────┬─────┘                           │
-  │                               │ ✗ fail                          │
-  │                               ▼                                 │
-  │               ┌──────┐   ┌──────────┐                           │
-  │               │ full │─→ │ contract │──→ ✓ pass → done         │
-  │               └──────┘   └──────────┘                           │
-  └───────────────────────────┬─────────────────────────────────────┘
-                              │
-                              ▼
-  ┌─────────────────────────────────────────────────────────────────┐
-  │ YOU GET                                                         │
-  │                                                                 │
-  │   ✓ Valid output guaranteed — schema + business rules enforced  │
-  │   ✓ Cheapest model that works — most requests stay on nano     │
-  │   ✓ Cost, latency, tokens — tracked on every call              │
-  │   ✓ Eval scores per model — data instead of gut feeling        │
-  │   ✓ Regressions caught — before deploy, not after              │
-  │   ✓ Recommendation — "use nano+mini, drop full, save $X/mo"   │
-  └─────────────────────────────────────────────────────────────────┘
-```
-
-## 30-second version
-
-```ruby
-class ClassifyTicket < RubyLLM::Contract::Step::Base
-  prompt "Classify this support ticket by priority and category.\n\n{input}"
-
-  output_schema do
-    string :priority, enum: %w[low medium high urgent]
-    string :category
-  end
-
-  validate("urgent needs justification") { |o, input| o[:priority] != "urgent" || input.length > 20 }
-  retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]
-end
-
-result = ClassifyTicket.run("I was charged twice")
-result.parsed_output  # => {priority: "high", category: "billing"}
-result.trace[:model]  # => "gpt-4.1-nano" (first model that passed)
-result.trace[:cost]   # => 0.000032
-```
-
-Bad JSON? Retried automatically. Wrong answer? Escalated to a smarter model. Schema violated? Caught client-side. The contract guarantees every response meets your rules — you pay for the cheapest model that passes.
+`ruby_llm` handles the HTTP side (rate limits, timeouts, streaming, tool calls, embeddings). This gem handles what the model *returned*: schema validation, business rules, retry with model fallback, datasets, regression tests.
 
 ## Install
 
@@ -128,239 +15,81 @@ RubyLLM.configure { |c| c.openai_api_key = ENV["OPENAI_API_KEY"] }
 RubyLLM::Contract.configure { |c| c.default_model = "gpt-4.1-mini" }
 ```
 
-Works with any ruby_llm provider (OpenAI, Anthropic, Gemini, etc).
+Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc).
 
-## Handle output variance with model escalation
+## Do I need this?
 
-Models are non-deterministic. A prompt that works on 95% of inputs can break on the edge case sitting in your production traffic right now. The defensive response is to pick the strongest model and pay for it on every call. The measured response is to define a contract and let the gem escalate only when the cheaper model's output actually fails the rules:
+Use this if LLM output affects production behaviour, money, user trust, or downstream code. You probably don't need it if you have one low-risk prompt, manually inspect every result, or only generate best-effort prose.
 
-```ruby
-retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]
-```
-
-```
-Attempt 1: gpt-4.1-nano  → contract failed  ($0.0001)
-Attempt 2: gpt-4.1-mini  → contract passed  ($0.0004)
-           gpt-4.1       → never called      ($0.00)
-```
+Already using structured outputs from your provider? This gem adds business-rule validation, retry with model fallback, evals, regression gating, and test stubs on top of them — the layer that stops schema-valid-but-wrong output from reaching users. See [Why contracts?](docs/guide/why.md) for the four production failure modes the gem exists for, or run `ruby examples/01_fallback_showcase.rb` to see the fallback loop in 30 seconds (no API key needed).
 
-Most requests succeed on the cheapest model. The expensive ones fire only when output variance demands it. The cost win is a consequence of measuring variance correctly — not the primary goal. Want to know how often each tier triggers? Run `compare_models` against your dataset.
+## Example
 
-Default retry statuses (since 0.7.0) are `:validation_failed` and `:parse_error` — the two flavors of LLM output variance. Transport errors (rate limits, timeouts, 5xx) are retried by ruby_llm at the HTTP layer and intentionally not duplicated here. If you want `:adapter_error` in retry too, opt in explicitly — it's meaningful paired with an escalation chain.
-
-## Soft delivery: retry for variance, ship the last attempt
-
-Sometimes validation is a **soft quality check** — "options balanced", "style consistent", "tone friendly" — and a partial output is better than none. The same model generating the same prompt produces different samples run-to-run (OpenAI forces `temperature=1.0` on gpt-5/o-series), so a single unlucky draw shouldn't fail the user. Use `attempts:` to retry on the SAME model — no escalation — and get the last attempt back even if it still failed the contract:
+A Rails app takes article text extracted from a user-submitted URL and wants to show a summary card: a short TL;DR, 3–5 key takeaways, and a tone label. The output has to fit the UI (TL;DR under 200 chars) and the schema has to be strict enough to render without conditionals.
 
 ```ruby
-class GenerateQuiz < RubyLLM::Contract::Step::Base
-  output_schema do
-    # ... 15 questions × 4 options ...
-  end
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  prompt <<~PROMPT
+    Summarize this article for a UI card. Return a short TL;DR,
+    3 to 5 key takeaways, and a tone label.
 
-  validate("answer options balanced") do |out, _|
-    out[:questions].all? do |q|
-      lens = q[:answer_options].map(&:length)
-      next false if lens.empty? || lens.min.zero?
+    {input}
+  PROMPT
 
-      lens.min >= 15 && (lens.max.to_f / lens.min) <= 1.7
-    end
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
   end
 
-  retry_policy attempts: 3
-end
-
-result = GenerateQuiz.run(document)
-if result.ok?
-  publish(result.parsed_output)
-else
-  # Three unlucky draws in a row — ship the last one anyway, log the miss.
-  Rails.logger.warn "Quiz delivered with soft-validation miss: #{result.validation_errors.join('; ')}"
-  publish(result.parsed_output)
-end
-```
-
-How this differs from the escalation chain:
-
-- `retry_policy models: %w[nano mini full]` — **document hardness.** Retry means "the cheap model isn't enough, use a smarter one."
-- `retry_policy attempts: 3` — **sampling variance.** Retry means "same model, different random seed — the model can do better on a second try."
-
-After all `attempts` are exhausted (`attempts: 3` means 3 total tries, not 3 retries on top of the first), the Result carries `status: :validation_failed` plus the last attempt's `parsed_output`. The caller decides: ship anyway, fall back to a template, or surface an error. The gem does not raise on exhaustion — your application policy, your choice.
-
-Combine both when helpful:
-
-```ruby
-retry_policy do
-  escalate({ model: "gpt-4.1-mini" }, { model: "gpt-4.1-mini" }, { model: "gpt-4.1" })
-end
-```
-
-Two tries on mini (variance retry) before paying for full-fat gpt-4.1.
-
-## Know which model to use — with data
+  validate("TL;DR fits the card")  { |o, _| o[:tldr].length <= 200 }
+  validate("takeaways are unique") { |o, _| o[:takeaways].uniq.size == o[:takeaways].size }
 
-Don't guess. Define test cases, compare models, get numbers:
-
-```ruby
-ClassifyTicket.define_eval("regression") do
-  add_case "billing", input: "I was charged twice", expected: { priority: "high" }
-  add_case "feature", input: "Add dark mode please", expected: { priority: "low" }
-  add_case "outage",  input: "Database is down",    expected: { priority: "urgent" }
-end
-
-comparison = ClassifyTicket.compare_models("regression",
-  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
-```
-
-```
-Candidate                  Score       Cost  Avg Latency
----------------------------------------------------------
-gpt-4.1-nano                0.67    $0.0001         48ms
-gpt-4.1-mini                1.00    $0.0004         92ms
-gpt-4.1                     1.00    $0.0021        210ms
-
-Cheapest at 100%: gpt-4.1-mini
-```
-
-Nano fails on edge cases. Mini and full both score 100% — but mini is **5x cheaper**. Now you know.
-
-Running live against gpt-5 / o-series? Pass `runs: 3` to average out sampling variance (OpenAI forces `temperature=1.0` server-side, so one unlucky run can misclassify a viable candidate). See [Reducing variance with `runs:`](docs/guide/optimizing_retry_policy.md#reducing-variance-with-runs).
-
-Want the *effective* cost — first-attempt plus retries — rather than the single-shot headline number? Pass `production_mode: { fallback: "gpt-5-mini" }` and the table gains `escalation`, `effective cost`, and a `Chain` column. See [Production-mode cost measurement](docs/guide/optimizing_retry_policy.md#production-mode-cost-measurement).
-
-## Let the gem tell you what to do
-
-Don't read tables — get a recommendation. Supports `model + reasoning_effort` combinations:
-
-```ruby
-rec = ClassifyTicket.recommend("regression",
-  candidates: [
-    { model: "gpt-4.1-nano" },
-    { model: "gpt-4.1-mini" },
-    { model: "gpt-5-mini", reasoning_effort: "low" },
-    { model: "gpt-5-mini", reasoning_effort: "high" },
-  ],
-  min_score: 0.95
-)
-
-rec.best           # => { model: "gpt-4.1-mini" }
-rec.retry_chain    # => [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }]
-rec.to_dsl         # => "retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini]"
-rec.savings        # => savings vs your current model (if configured)
-```
-
-Copy `rec.to_dsl` into your step. Done.
-
-## Catch regressions before users do
-
-A model update silently dropped your accuracy? A prompt tweak broke an edge case? You'll know before deploying:
-
-```ruby
-# Save a baseline once:
-report = ClassifyTicket.run_eval("regression", context: { model: "gpt-4.1-nano" })
-report.save_baseline!(model: "gpt-4.1-nano")
-
-# In CI — block merge if anything regressed:
-expect(ClassifyTicket).to pass_eval("regression")
-  .with_context(model: "gpt-4.1-nano")
-  .without_regressions
-```
-
-```ruby
-diff = report.compare_with_baseline(model: "gpt-4.1-nano")
-diff.regressed?    # => true
-diff.regressions   # => [{case: "outage", baseline: {passed: true}, current: {passed: false}}]
-diff.score_delta   # => -0.33
-```
-
-No more "it worked in the playground". Regressions are caught in CI, not production.
-
-## A/B test your prompts
-
-Changed a prompt? Compare old vs new on the same dataset with regression safety:
-
-```ruby
-diff = ClassifyTicketV2.compare_with(ClassifyTicketV1,
-  eval: "regression", model: "gpt-4.1-mini")
-
-diff.safe_to_switch?  # => true (no regressions)
-diff.improvements     # => [{case: "outage", ...}]
-diff.score_delta      # => +0.33
-```
-
-```ruby
-# CI gate:
-expect(ClassifyTicketV2).to pass_eval("regression")
-  .compared_with(ClassifyTicketV1)
-  .with_minimum_score(0.8)
-```
-
-## Chain steps with fail-fast
-
-Pipeline stops at the first contract failure — no wasted tokens on downstream steps:
-
-```ruby
-class TicketPipeline < RubyLLM::Contract::Pipeline::Base
-  step ClassifyTicket,  as: :classify
-  step RouteToTeam,     as: :route
-  step DraftResponse,   as: :draft
+  retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]
 end
 
-result = TicketPipeline.run("I was charged twice")
-result.outputs_by_step[:classify]   # => {priority: "high", category: "billing"}
-result.trace.total_cost             # => $0.000128
+result = SummarizeArticle.run(article_text)
+result.parsed_output    # => { tldr: "...", takeaways: [...], tone: "analytical" }
+result.trace[:model]    # => "gpt-4.1-nano"  (first model that passed)
+result.trace[:cost]     # => 0.000032
 ```
 
-## Gate merges on quality and cost
+The model returns JSON matching the schema. If the response is malformed, the TL;DR overflows the card, or the takeaway count is off, the gem retries — moving to the next model in `models:` only when the cheaper one can't satisfy the rules. In this setup cheaper models are tried first and the expensive ones are used only when cheaper models fail.
 
-```ruby
-# RSpec — block merge if accuracy drops or cost spikes
-expect(ClassifyTicket).to pass_eval("regression")
-  .with_minimum_score(0.8)
-  .with_maximum_cost(0.01)
+You could write this loop yourself once. The gem gives you the loop, a trace of every attempt (model, status, cost, latency), fallback policy, evals, baselines, and CI checks as one contract object — tracked per-step so adding a new LLM feature to your app is one class, not one-off scaffolding.
 
-# Rake — run all evals across all steps
-RubyLLM::Contract::RakeTask.new do |t|
-  t.minimum_score = 0.8
-  t.maximum_cost = 0.05
-end
-# bundle exec rake ruby_llm_contract:eval
-```
+## Most useful next
 
-## Full power: data-driven retry chains
+Everything below is optional — the example above is a complete step. Reach for these when one step isn't enough.
 
-The pieces above — evals, compare_models, recommend — combine into a workflow that replaces guesswork with measured optimization. You define evals for your step, run `recommend` against all of them, find the eval that actually needs the strongest model, and build a retry chain where each attempt is as cheap as the data allows.
+- **[CI regression gates](docs/guide/getting_started.md)** — `define_eval` + `save_baseline!` + `pass_eval(...).without_regressions` blocks CI when accuracy drops on a model update or prompt tweak.
+- **[Find the cheapest viable fallback list](docs/guide/optimizing_retry_policy.md)** — `Step.recommend("regression", candidates: [...], min_score: 0.95)` returns the cheapest list of models that still passes your evals. `production_mode:` measures retry-aware cost.
+- **[A/B test prompts](docs/guide/eval_first.md)** — `SummarizeArticleV2.compare_with(SummarizeArticleV1, eval: "regression")` reports whether the new prompt is safe to ship.
+- **[Budget caps](docs/guide/getting_started.md)** — `max_cost`, `max_input`, `max_output` refuse the request before calling the API when an estimate exceeds the limit.
 
-The difference: instead of "gpt-5-mini seems to work, let's use it everywhere", you get "nano handles 4/6 scenarios, mini@low catches the 5th, full mini only fires on the hardest edge case — first attempt is 4× cheaper."
-
-Full procedure with examples: **[Optimizing retry_policy](docs/guide/optimizing_retry_policy.md)**
+Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and [best-effort retries without fallback](docs/guide/best_practices.md) (`retry_policy attempts: 3` for sampling variance).
 
 ## Docs
 
-| Guide | |
-|-------|-|
-| [Getting Started](docs/guide/getting_started.md) | Features walkthrough, model escalation, eval |
-| [Eval-First](docs/guide/eval_first.md) | Practical workflow for prompt engineering with datasets, baselines, and A/B gates |
-| [Optimizing retry_policy](docs/guide/optimizing_retry_policy.md) | Find the cheapest retry chain that passes all your evals |
-| [Best Practices](docs/guide/best_practices.md) | 6 patterns for bulletproof validates |
-| [Output Schema](docs/guide/output_schema.md) | Full schema reference + constraints |
-| [Pipeline](docs/guide/pipeline.md) | Multi-step composition, timeout, fail-fast |
-| [Testing](docs/guide/testing.md) | Test adapter, RSpec matchers |
-| [Migration](docs/guide/migration.md) | Adopting the gem in existing Rails apps |
+**New here?** Read in order: this README → [Why contracts?](docs/guide/why.md) → [Getting Started](docs/guide/getting_started.md).
+
+| Guide | What it does for your app |
+|-------|---------------------------|
+| [Why contracts?](docs/guide/why.md) | Recognise the four production failures the gem exists for |
+| [Getting Started](docs/guide/getting_started.md) | Walk the full feature set on one concrete step |
+| [Rails integration](docs/guide/rails_integration.md) | Directory, initializer, jobs, logging, specs, CI gate — 7 FAQs for Rails devs |
+| [Adopt in an existing Rails app](docs/guide/migration.md) | Replace raw `LlmClient.call` with a contract, Before/After |
+| [Prevent silent prompt regressions](docs/guide/eval_first.md) | Evals, baselines, CI gates that block quality drift |
+| [Control retry cost and fallback behaviour](docs/guide/optimizing_retry_policy.md) | Find the cheapest viable fallback list empirically |
+| [Write validate rules that catch real bugs](docs/guide/best_practices.md) | Patterns for cross-input checks and content-quality rules |
+| [Stub LLM calls in tests](docs/guide/testing.md) | Deterministic specs, RSpec + Minitest matchers |
+| [Chain LLM calls into a pipeline](docs/guide/pipeline.md) | Multi-step with fail-fast and per-step models |
+| [Schema DSL reference](docs/guide/output_schema.md) | Every constraint, nested objects, pattern table |
+| [Prompt DSL reference](docs/guide/prompt_ast.md) | `system` / `rule` / `section` / `example` / `user` nodes |
 
 ## Roadmap
 
-**v0.7.1 (current):** Follow-up — `Step::Base#run_once` no longer masks adapter-phase `ArgumentError` as `:input_error`. Programmer bugs in adapter code now propagate; DSL misconfiguration still becomes `:input_error` via narrower rescue.
-
-**v0.7.0:** Sharpened retry semantics. `DEFAULT_RETRY_ON` now targets LLM output variance only (`:validation_failed`, `:parse_error`); transport errors are delegated to ruby_llm's Faraday retry. `AdapterCaller` narrowed to let programmer errors propagate instead of masking them as retries. Breaking change — see [CHANGELOG](CHANGELOG.md) for migration.
-
-**v0.6:** "What should I do?" — `Step.recommend` returns optimal model, reasoning effort, and retry chain. Per-attempt `reasoning_effort` in retry policies.
-
-**v0.5:** Prompt A/B testing with `compare_with`. Soft observations with `observe`.
-
-**v0.4:** Eval history, batch concurrency, pipeline per-step eval, Minitest, structured logging.
-
-**v0.3:** Baseline regression detection, migration guide.
+Latest: **v0.7.2** — terminal output labels and guides aligned with the fallback narrative; `output_schema.md` DSL bug fix. See [CHANGELOG](CHANGELOG.md) for history.
 
 ## License