ruby_llm-contract 0.10.1 → 0.10.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 66d3692be182ac8958132d06b53c1d47e01d07769d0e7a1942f81b71eb23a759
-  data.tar.gz: f254fcbc6bc8c2a42a78cff9ea3b24b23f04d860c60d79084bbd8a5d6ea6ec0b
+  metadata.gz: ac8c19463285a3e2c0f9050e835125454b21d2b1500dcb5bad2dd4a114666bd0
+  data.tar.gz: 339c0609e3dccbf55da649c3470ed234c5abafa55c8649aee5d8dd41f9bd82ec
 SHA512:
-  metadata.gz: f035aca56a5c0e2ca0607a06431cbb76a6ff4cc45f2f35fe9309d534b881910fa05d63c5c581a6e9cbac5374377f6fc187e521a844a653da5b53d612e9a23a89
-  data.tar.gz: 31bb34333a28224e7c00ce2dea7b3905dce878c320fc0fb8a0e5db1313259a0e12e136ab6cc74126b25f0a47984830ae2172e3f824ccfdcb28f847929f40080a
+  metadata.gz: c04fea66393868fbdb369f5a75eaeb66a28172c8f9ddabd0a8c053fd55e5daa06174aeeb00c4bb7e9aabaf613bd1ef4ea354ff7d0f4b817f2ec41af1a4c61667
+  data.tar.gz: 71f502ebe497ede22df508d3b56396660fbd1eba109066885d5e905724bd82d30a0d46fddfe4f1c1e7efe47f95ea41b4f61341a4698c12d3f3a397cab2510068

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.10.2 (2026-06-10)
+
+Patch release: ship the `docs/guide/` directory inside the gem so adopters and LLM integration agents can read the manuals locally (via `bundle show ruby_llm-contract` or `gem unpack`) without an internet round-trip to GitHub. No code behavior change.
+
+### Added
+
+- **`docs/guide/*` is now packaged with the gem** (14 files, ~120 KB). Previously the README's "See also" links pointed at `docs/guide/getting_started.md`, `docs/guide/optimizing_retry_policy.md`, etc., but those files were stripped from the published gem - LLM integration agents (Cursor, Claude Code, Copilot) reported "no documentation in the gem" because the links 404'd locally. `docs/ideas/` and `doc/decisions/` remain excluded.
+
+### Fixed
+
+- **`models:` keyword form documented + pinned for hash configs.** `retry_policy models: ["gpt-5-nano", { model: "gpt-5-mini", reasoning_effort: "high" }]` is now covered by a spec and shown in [getting_started.md](docs/guide/getting_started.md). The block form (`escalate(...)`) and the keyword form share the same `@configs` storage; both forms accept config hashes.
+- **`reasoning_effort` examples corrected to gpt-5 family.** Pre-0.10.2 docs and specs paired `reasoning_effort` with `gpt-4.1-*` model names, which is incorrect: `gpt-4.1` is not a reasoning model. Updated across `docs/guide/getting_started.md`, `docs/guide/optimizing_retry_policy.md`, `CHANGELOG.md`, and 7 spec files. Non-reasoning `gpt-4.1` examples (model fallback chains without `reasoning_effort`) are unchanged.
+- **Version mentions corrected in README and multimodal guide.** README FAQ ("Upgraded to 0.9.0 - why?") now reads "Upgraded to 0.10.0 from 0.8.x"; `docs/guide/multimodal_input.md` references `0.10.0+` and `0.10.x` instead of `0.9.0`. 0.9.0 and 0.9.1 were tagged but never published to rubygems; adopters jump from 0.8.0 directly to 0.10.x.
+
 ## 0.10.1 (2026-06-01)
 
 Patch release fixing gem packaging. 0.10.0 was yanked from rubygems.org due to the issue documented below; 0.10.1 is the recommended upgrade target. No code behavior change vs 0.10.0.
@@ -254,7 +268,7 @@ end
 - **`Step.recommend`** — `ClassifyTicket.recommend("eval", candidates: [...], min_score: 0.95)` runs eval on all candidates and returns a `Recommendation` with optimal model, retry chain, rationale, savings vs current config, and `to_dsl` code output.
 - **Candidates as configurations** — `candidates:` accepts `{ model:, reasoning_effort: }` hashes, not just model name strings. `gpt-5-mini` with `reasoning_effort: "low"` is a different candidate than with `"high"`.
 - **`compare_models` extended** — new `candidates:` parameter alongside existing `models:` (backward compatible). Candidate labels include reasoning effort in output table.
-- **Per-attempt `reasoning_effort` in retry policies** — `escalate` accepts config hashes: `escalate({ model: "gpt-4.1-nano" }, { model: "gpt-5-mini", reasoning_effort: "high" })`. Each attempt gets its own reasoning_effort forwarded to the provider.
+- **Per-attempt `reasoning_effort` in retry policies** — `escalate` accepts config hashes: `escalate({ model: "gpt-5-nano" }, { model: "gpt-5-mini", reasoning_effort: "high" })`. Each attempt gets its own reasoning_effort forwarded to the provider.
 - **`pass_rate_ratio`** — numeric float (0.0–1.0) on `Report` and `ReportStats`, complementing the string `pass_rate` (`"3/5"`).
 - **History entries enriched** — `save_history!` accepts `reasoning_effort:` and stores `model`, `reasoning_effort`, `pass_rate_ratio` in JSONL entries.
 

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ruby_llm-contract (0.10.1)
+    ruby_llm-contract (0.10.2)
       dry-types (~> 1.7)
       ruby_llm (~> 1.12)
       ruby_llm-schema (~> 0.3)
@@ -258,7 +258,7 @@ CHECKSUMS
   rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   ruby_llm (1.14.0) sha256=57c6f7034fc4a44504ea137d70f853b07824f1c1cdbe774ab3ab3522e7098deb
-  ruby_llm-contract (0.10.1)
+  ruby_llm-contract (0.10.2)
   ruby_llm-schema (0.3.0) sha256=a591edc5ca1b7f0304f0e2261de61ba4b3bea17be09f5cf7558153adfda3dec6
   ruby_parser (3.22.0) sha256=1eb4937cd9eb220aa2d194e352a24dba90aef00751e24c8dfffdb14000f15d23
   rubycritic (4.12.0) sha256=024fed90fe656fa939f6ea80aab17569699ac3863d0b52fd72cb99892247abc8

data/README.md

@@ -157,7 +157,7 @@ Different layers, complementary. [`ruby_llm-tribunal`](https://github.com/Alqemi
 
 ## Status & versioning
 
-Pre-1.0 (currently **0.10.1**). Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 0.10.1` until 1.0 ships.
+Pre-1.0 (currently **0.10.2**). Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 0.10.2` until 1.0 ships.
 
 ## FAQ
 
@@ -167,7 +167,7 @@ Pre-1.0 (currently **0.10.1**). Semver tracked; breaking changes flagged in [CHA
 
 **Where in a Rails app?** Default `app/contracts/`. The Railtie reloads `app/contracts/eval/` and `app/steps/eval/` in development; any autoloaded directory also works. See [Rails integration](docs/guide/rails_integration.md).
 
-**Upgraded to 0.9.0 and my contract started refusing — why?** 0.9.0 added multimodal input. If your contract has `max_cost` or `max_input` set AND now receives `context: { attachment: ... }`, you must declare `attachment_token_estimate(n)` (conservative input-token budget for the attachment) — otherwise the call fails closed with `:limit_exceeded`. The gem cannot bound vision/PDF cost without your estimate. Opt out per-step with `on_unknown_attachment_size :warn`. Text-only contracts are unaffected. See [multimodal input guide](docs/guide/multimodal_input.md).
+**Upgraded to 0.10.0 from 0.8.x and my contract started refusing — why?** 0.10.0 added multimodal input. If your contract has `max_cost` or `max_input` set AND now receives `context: { attachment: ... }`, you must declare `attachment_token_estimate(n)` (conservative input-token budget for the attachment) — otherwise the call fails closed with `:limit_exceeded`. The gem cannot bound vision/PDF cost without your estimate. Opt out per-step with `on_unknown_attachment_size :warn`. Text-only contracts are unaffected. See [multimodal input guide](docs/guide/multimodal_input.md).
 
 ## License
 

data/docs/architecture.md

@@ -0,0 +1,50 @@
+# Architecture
+
+```
+RubyLLM::Contract::Pipeline::Base   # optional: compose steps
+  ├── Pipeline::Runner           # sequential execution, fail-fast, trace, timeout
+  └── Pipeline::Result           # per-step outputs + aggregated trace
+
+RubyLLM::Contract::Step::Base       # single contracted step
+  ├── Step::Dsl                  # DSL macros (prompt, validate, output_schema, etc.)
+  ├── Step::RetryPolicy          # attempts, models, reasoning_effort, retry_on
+  ├── Step::RetryExecutor        # retry loop driven by RetryPolicy
+  ├── Step::LimitChecker         # preflight cost / input / output checks
+  ├── Step::Runner               # runtime flow (single attempt)
+  ├── Step::Result               # status + outputs + errors + trace
+  ├── Step::Trace                # model, latency, tokens, cost, attempts
+  ├── Prompt::AST                # structured prompt (immutable)
+  │     ├── Prompt::Builder      # DSL: system, rule, example, user, section
+  │     └── Prompt::Renderer     # AST → messages array
+  ├── Contract::Definition       # parse strategy + validates
+  │     ├── Contract::Parser     # :json / :text (auto-inferred from output type)
+  │     ├── Contract::Validator  # runs parse + schema + validates + observations
+  │     └── Contract::SchemaValidator  # JSON Schema validation (nested)
+  ├── CostCalculator             # per-step cost estimation from model pricing
+  ├── TokenEstimator             # input-token count estimation for limit checks
+  ├── estimate_cost              # single-call cost estimate (class method)
+  ├── estimate_eval_cost         # cost estimate for a full eval across models
+  └── Adapters::Base             # provider interface
+        ├── Adapters::RubyLLM    # real LLM calls via ruby_llm (any provider)
+        └── Adapters::Test       # canned responses for specs and examples
+
+RubyLLM::Contract::Eval             # quality measurement
+  ├── Eval::EvalDefinition       # define_eval DSL (verify, add_case, default_input, sample_response)
+  ├── Eval::Dataset              # test cases
+  ├── Eval::Runner               # sequential or concurrent execution
+  ├── Eval::Report               # score, pass_rate, per-case results
+  ├── Eval::AggregatedReport     # merged reports across models or runs
+  ├── Eval::CaseResult           # value object (name, passed?, output, expected, mismatches, cost)
+  ├── Eval::ExpectationEvaluator # expected / expected_traits / evaluator proc
+  ├── Eval::ModelComparison      # compare_models result (table, best_for, candidate configs)
+  ├── Eval::Recommender          # model recommendation algorithm (candidates → optimal config)
+  ├── Eval::Recommendation       # recommendation result (best, retry_chain, savings, to_dsl)
+  ├── Eval::RetryOptimizer       # optimize_retry_policy result (per-eval breakdown, fallback list)
+  ├── Eval::BaselineDiff         # save_baseline! + without_regressions comparison
+  ├── Eval::PromptDiffComparator # compare_with prompt A/B diff
+  └── Eval::EvalHistory          # time-series view across saved reports
+
+RubyLLM::Contract::RakeTask         # rake ruby_llm_contract:eval
+RubyLLM::Contract::OptimizeRakeTask # rake ruby_llm_contract:optimize
+RubyLLM::Contract::Railtie          # auto-loads eval files in Rails
+```

data/docs/guide/best_practices.md

@@ -0,0 +1,136 @@
+# Best Practices
+
+> Read this when writing your first `validate` rules and you want patterns that catch real bugs instead of restating schema.
+
+Schema guarantees valid JSON structure. An LLM can still return structurally perfect JSON that is **semantically wrong**. Schema handles _shape_, validates handle _meaning_.
+
+All examples extend the `SummarizeArticle` step from the [README](../../README.md).
+
+## 1. Guard against empty / placeholder output
+
+**Why it matters:** a cheap model that answers `{"tldr": "This article discusses...", "takeaways": ["This article discusses X", ...]}` passes the schema but renders a broken UI card that tells the user nothing. The validate catches it before `Article.update!` persists it.
+
+```ruby
+output_schema do
+  string :tldr
+  array  :takeaways, of: :string, min_items: 3, max_items: 5
+  string :tone, enum: %w[neutral positive negative analytical]
+end
+
+validate("tldr is substantive") do |o, _|
+  o[:tldr].to_s.split.length >= 5   # at least five words
+end
+
+validate("no boilerplate takeaways") do |o, _|
+  o[:takeaways].none? { |t| t.downcase.start_with?("this article") }
+end
+```
+
+## 2. Cross-validate output against input
+
+**Why it matters:** a lazy model will return the article text verbatim as the "summary", or invent takeaways about topics the article never mentions. The 2-arity form is how you catch answers that are internally consistent but unfaithful to the actual input.
+
+`validate` blocks with 2-arity `|output, input|` compare the model's answer against what was asked:
+
+```ruby
+validate("tldr is shorter than the article") do |output, input|
+  output[:tldr].length < input.length / 2
+end
+
+validate("every takeaway appears, in spirit, in the article") do |output, input|
+  output[:takeaways].all? { |t|
+    # cheap keyword overlap heuristic
+    t.downcase.split.any? { |w| input.downcase.include?(w) && w.length > 4 }
+  }
+end
+```
+
+## 3. Conditional logic schema can't express
+
+**Why it matters:** customer success filters on `tone == "negative"` to route angry users to a human. If the model labels an outage complaint "negative" but the takeaways are all positive-sounding, the filter runs on a label that doesn't match the content — the routing breaks silently.
+
+```ruby
+validate("negative tone requires at least one concrete concern") do |output, _input|
+  next true unless output[:tone] == "negative"
+  output[:takeaways].any? { |t| t.match?(/fail|break|crash|outage|vulnerab|risk/i) }
+end
+```
+
+A model that picks `tone: "negative"` but gives three upbeat takeaways fails this check. Schema can't catch it because each takeaway is, individually, a valid string.
+
+## 4. Content quality
+
+**Why it matters:** a TL;DR with `## Summary` leaks markdown into a plain-text card. A one-word takeaway ("Fast.") wastes a UI slot. A leaked `{article}` placeholder reveals the prompt template to end users. All pass schema; all embarrass you in front of customers.
+
+```ruby
+validate("no markdown headings in the TL;DR") do |o, _|
+  !o[:tldr].match?(/^\#{1,6}\s/)
+end
+
+validate("takeaways aren't single words") do |o, _|
+  o[:takeaways].all? { |t| t.split.length >= 3 }
+end
+
+validate("no template placeholders leaked") do |o, _|
+  !(o[:tldr] + o[:takeaways].join(" ")).include?("{")
+end
+```
+
+## 5. Pipeline: preserve data between steps
+
+In a pipeline, each step only sees the previous step's output. If a later step needs original article metadata, an intermediate step must carry it through. Suppose a pipeline `SummarizeArticle → GenerateHashtags`, where `GenerateHashtags` needs the `tone` from the summary:
+
+```ruby
+class GenerateHashtags < RubyLLM::Contract::Step::Base
+  input_type Hash
+
+  output_schema do
+    # Carry through the fields a downstream step (or the caller) might need
+    string :tone, enum: %w[neutral positive negative analytical]
+    array  :hashtags, of: :string, min_items: 2, max_items: 5
+  end
+
+  prompt do
+    rule "Preserve the tone label from the input unchanged."
+    user "Summary: {tldr}\nTone: {tone}\nTakeaways: {takeaways}"
+  end
+
+  validate("tone preserved") { |o, input| o[:tone] == input[:tone] }
+end
+```
+
+The explicit `validate("tone preserved")` catches the case where the model silently rewrites the tone during a downstream transform.
+
+## 6. Model fallback
+
+**Why it matters:** 80% of production articles are short and simple — `gpt-4.1-nano` handles them for ~$0.0001. The remaining 20% are dense, critical, or multi-topic — those need `gpt-4.1-mini` or `gpt-4.1`. Paying `gpt-4.1` rates for every call when nano is enough for most is throwing money away. Contracts tell you when nano wasn't enough, so fallback is cost-aware, not hope-based.
+
+Small models are cheap but hallucinate. Big models are accurate but expensive. Start cheap, fall back only when validates catch a failure:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
+  end
+
+  validate("TL;DR fits the card")  { |o, _| o[:tldr].length <= 200 }
+  validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }
+
+  retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]
+end
+```
+
+**Key insight:** without contracts, you can't do model fallback — you'd have no way to know if the cheap model's output is good enough. Validates are the quality gate that makes cost optimization possible. See [Optimizing retry_policy](optimizing_retry_policy.md) for how to find the cheapest viable fallback list for your step.
+
+## Summary
+
+| What to validate | Use |
+|---|---|
+| Field types, enums, ranges, required fields | `output_schema` |
+| Output makes sense given the input | `validate` (2-arity `\|output, input\|`) |
+| Conditional business rules | `validate` |
+| Content quality (not empty, not template) | `validate` |
+| Data preserved across pipeline steps | `validate` + schema carry-through |
+| Cost optimization via model fallback | `retry_policy` + `validate` as quality gate |

data/docs/guide/eval_first.md

@@ -0,0 +1,192 @@
+# Eval-First
+
+> Read this when you need to prevent silent prompt regressions in CI. Skip if your LLM output is evaluated only by humans and never gated in an automated pipeline.
+
+If you change prompts by feel, you ship regressions by feel.
+
+Concrete scenario: `SummarizeArticle` has been running in production for two weeks. Customer success notices that complaints about service outages keep getting `tone: "analytical"` instead of `"negative"` — so their "critical feedback" filter silently misses angry users. Someone tweaks the system prompt to emphasise negative sentiment. It fixes the outage article but now three neutral product-update articles get misclassified as `"negative"`. You find out from a Slack thread.
+
+That is the cost of prompt-by-feel. Evals are how you stop it.
+
+`ruby_llm-contract` works best when you treat evals as the source of truth:
+
+1. Capture real failures from production (the outage article, verbatim).
+2. Turn them into eval cases (`add_case "service outage complaint"`).
+3. Change the prompt.
+4. Re-run the same eval — plus all previously-passing cases.
+5. Merge only if the eval says quality improved or stayed safe on every case.
+
+## Core rule
+
+**Do not start with the prompt. Start with the eval.**
+
+Using the `SummarizeArticle` step from the [README](../../README.md):
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "ruby release",
+           input: "Ruby 3.4 shipped with frozen string literals...",
+           expected: { tone: "analytical" }  # partial match
+
+  add_case "critical review",
+           input: "Mesh networking hardware failed under load...",
+           expected: { tone: "negative" }
+end
+```
+
+Only after the eval exists, touch: `system`, `rule`, `example`, `validate`, prompt versions.
+
+## Three eval kinds
+
+### 1. `smoke` — wiring check, offline
+
+```ruby
+SummarizeArticle.define_eval("smoke") do
+  default_input "Ruby 3.4 shipped with frozen string literals..."
+  sample_response({
+    tldr: "...",
+    takeaways: ["point one", "point two", "point three"],
+    tone: "analytical"
+  })
+end
+```
+
+`sample_response` returns canned data. Zero API calls. Verifies schema + validates parse and the step wiring is intact. **Not a quality signal.**
+
+### 2. `regression` — real quality measurement
+
+Represent real traffic and known failures. Good sources: production logs, bad completions, incidents, QA edge cases, cases a human had to correct.
+
+Every production failure becomes `add_case`. That's the flywheel.
+
+### 3. `ab` — prompt iteration
+
+Compare two prompt versions on the same eval:
+
+```ruby
+diff = SummarizeArticleV2.compare_with(
+  SummarizeArticleV1,
+  eval: "regression",
+  model: "gpt-4.1-mini"
+)
+
+diff.safe_to_switch?  # => true if no cases regressed
+```
+
+This is the cleanest eval-first move: same eval, same cases, two prompt versions, one answer.
+
+## What counts as eval-first
+
+**Good** — eval exists before the prompt change:
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "short article", input: "...", expected: { tone: "neutral" }
+end
+
+# Prompt iteration happens afterward
+diff = SummarizeArticleV2.compare_with(
+  SummarizeArticleV1, eval: "regression", model: "gpt-4.1-mini"
+)
+```
+
+**Bad**:
+
+```ruby
+# Tweak prompt for an hour
+# Maybe add an example
+# Maybe tighten a rule
+# Then eyeball one or two responses
+```
+
+That's prompt guessing, not eval-first.
+
+## `sample_response`: useful, but not the main thing
+
+Good for: offline smoke tests, local development, testing evaluator wiring, checking schema + validate behavior with zero API calls.
+
+Not enough for real prompt decisions. For those:
+
+- `run_eval(..., context: { model: "..." })` with a real model, or pass an explicit adapter.
+- `compare_with(...)` for prompt A/B.
+
+`compare_with` intentionally ignores `sample_response` — canned data would make both sides look the same.
+
+## Parallel eval runs
+
+For larger datasets, `run_eval` accepts a `concurrency:` argument — cases run in parallel using a thread pool:
+
+```ruby
+report = SummarizeArticle.run_eval("regression",
+  context: { model: "gpt-4.1-mini" },
+  concurrency: 8)
+```
+
+Same accepted by `compare_models` and `optimize_retry_policy`. Thread count is a ceiling — dataset order of results is preserved. Keep it low enough to respect the provider's rate limits.
+
+## Budgeting an eval before you run it
+
+`estimate_eval_cost` gives you a cost projection without calling the LLM:
+
+```ruby
+SummarizeArticle.estimate_eval_cost("regression",
+  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
+# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }
+```
+
+Use it in CI to decide which models are worth running regression on, or to cap worst-case spend per build.
+
+## Team workflow
+
+1. **Build one eval that matters** — 10–30 cases representing real mistakes and important business paths.
+2. **Gate CI** — `pass_eval("regression").with_context(model: "...").with_minimum_score(0.8)`. See [Getting Started](getting_started.md) for the full matcher chain.
+3. **Save a baseline** — `report.save_baseline!` makes quality drift visible.
+4. **Change prompts only through comparison** — `pass_eval(...).compared_with(SummarizeArticleV1)` in CI so any regression blocks the merge.
+5. **Feed production failures back** — every miss in prod → new `add_case`, then fix. The eval gets stronger over time.
+
+## Few-shot examples fit naturally
+
+Adding `example input: ..., output: ...` inside the prompt is still a prompt change. The eval-first way:
+
+1. Add examples to the prompt.
+2. Rerun the existing regression eval.
+3. `compare_with` against the old prompt.
+
+Few-shot isn't the proof. The eval is.
+
+## Model selection comes after prompt stability
+
+Don't optimize cost before you stabilize quality:
+
+1. Build `regression`.
+2. Improve the prompt with `compare_with`.
+3. Lock quality in CI.
+4. Then run `compare_models` (see [Optimizing retry_policy](optimizing_retry_policy.md)).
+
+```ruby
+comparison = SummarizeArticle.compare_models(
+  "regression",
+  candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }, { model: "gpt-4.1" }]
+)
+
+comparison.best_for(min_score: 0.95)
+```
+
+## Strong defaults for teams
+
+- `smoke` uses `sample_response`.
+- `regression` uses real model calls.
+- Every prompt change uses `compare_with`.
+- Every merge runs `pass_eval`.
+- Every production failure becomes a new `add_case`.
+
+## Short version
+
+1. Write `define_eval` before touching the prompt.
+2. Treat `sample_response` as smoke only.
+3. Use `run_eval("name", context: { model: "..." })` for real quality measurement.
+4. Use `compare_with` for every serious prompt change.
+5. Gate merges with `pass_eval`.
+6. Feed every production miss back into the dataset.
+
+Prompts stop being vibes and start being engineering.

data/docs/guide/getting_started.md

@@ -0,0 +1,199 @@
+# Getting Started
+
+> Read this to walk through every feature on one concrete step for the first time.
+
+The README shows a minimal `SummarizeArticle` step. This guide walks through the features you reach for as production requirements grow: budget caps so runaway inputs don't drain your LLM provider budget, evals so you catch regressions in CI, and CI gating so a merge that lowers accuracy gets blocked.
+
+## The walkthrough
+
+Start with the README example, then add features one layer at a time. Each is optional — use what you need.
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  # 1. Prompt (required)
+  prompt <<~PROMPT
+    Summarize this article for a UI card. Return a short TL;DR,
+    3 to 5 key takeaways, and a tone label.
+
+    {input}
+  PROMPT
+
+  # 2. Schema — sent to the provider via with_schema, validated client-side
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
+  end
+
+  # 3. Business rules — things JSON Schema cannot express
+  validate("TL;DR fits the card")  { |o, _| o[:tldr].length <= 200 }
+  validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }
+
+  # 4. Retry with model fallback on validation_failed / parse_error
+  retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]
+
+  # 5. Refuse before calling the LLM if input is too large or estimated cost exceeds the cap
+  max_input  2_000
+  max_output 4_000
+  max_cost   0.01
+end
+```
+
+## Validation and retry behavior
+
+When the cheap model returns output that fails a `validate` block or can't be parsed, retry falls back to the next model in `models:` and tries again.
+
+```ruby
+result = SummarizeArticle.run(article_text)
+
+result.status           # => :ok
+result.parsed_output    # => { tldr: "...", takeaways: [...], tone: "analytical" }
+result.trace[:model]    # => "gpt-4.1-mini"  (first model that passed)
+result.trace[:cost]     # => 0.00052  (sum of all attempts)
+result.trace[:attempts]
+# => [
+#   { attempt: 1, model: "gpt-4.1-nano", status: :validation_failed,
+#     cost: 0.00010, latency_ms: 45, ... },
+#   { attempt: 2, model: "gpt-4.1-mini", status: :ok,
+#     cost: 0.00042, latency_ms: 92, ... }
+# ]
+```
+
+If the whole chain exhausts, `result.status` is the status of the last attempt (`:validation_failed` or `:parse_error`) and `result.parsed_output` is the last attempt's output. The caller decides what to do — ship it anyway, fall back to a template, or raise.
+
+### Per-attempt reasoning effort
+
+`models:` accepts config hashes as well as model-name strings, so a fallback can "try harder" (more reasoning) on retry, not just switch model:
+
+```ruby
+retry_policy models: [
+  "gpt-5-nano",                                      # attempt 1: cheap + fast
+  { model: "gpt-5-mini", reasoning_effort: "high" }  # attempt 2: stronger + more reasoning
+]
+```
+
+`reasoning_effort` is a gpt-5 family feature (gpt-4.1 is not a reasoning model). The per-attempt value is forwarded via `with_thinking` (provider-agnostic — OpenAI `reasoning_effort` and Anthropic extended-thinking budget both supported). Passing it alongside a non-reasoning model is forwarded unchanged to the provider, which will either ignore it or reject the request — the gem does not guard against this.
+
+## Evals and CI gates
+
+An eval is a named scenario you can run to verify the step still works. `sample_response` makes it offline — zero API calls — so CI can run it on every merge without burning budget.
+
+```ruby
+SummarizeArticle.define_eval("smoke") do
+  default_input <<~ARTICLE
+    Ruby 3.4 ships with frozen string literals on by default, measurable YJIT
+    speedups on Rails workloads, and tightened Warning.warn category filtering.
+    The release notes also mention several parser fixes and faster keyword
+    argument handling.
+  ARTICLE
+
+  sample_response({
+    tldr: "Ruby 3.4 brings frozen string literals by default, YJIT speedups, and parser fixes.",
+    takeaways: [
+      "Frozen string literals are the default",
+      "YJIT adds measurable speedups on Rails workloads",
+      "Warning.warn category filtering is tighter"
+    ],
+    tone: "analytical"
+  })
+end
+
+report = SummarizeArticle.run_eval("smoke")
+report.passed?  # => true — schema + validates pass on the canned response
+report.score    # => 1.0
+report.print_summary
+```
+
+For real regression testing, define cases with expected output (online — calls the LLM):
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "ruby release",
+           input: "Ruby 3.4 was released...",
+           expected: { tone: "analytical" }  # partial match
+
+  add_case "critical review",
+           input: "The new mesh networking hardware failed under load...",
+           expected: { tone: "negative" }
+end
+```
+
+Gate CI on score and cost thresholds:
+
+```ruby
+# RSpec — blocks merge if accuracy drops or cost spikes
+expect(SummarizeArticle).to pass_eval("regression")
+  .with_minimum_score(0.8)
+  .with_maximum_cost(0.01)
+```
+
+Save a baseline once, then block regressions automatically:
+
+```ruby
+report = SummarizeArticle.run_eval("regression")
+report.save_baseline!
+
+# In CI:
+expect(SummarizeArticle).to pass_eval("regression").without_regressions
+```
+
+`without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality.
+
+## Budget caps
+
+`max_input`, `max_output`, and `max_cost` are preflight checks — the LLM is never called if an estimate exceeds the limit. Zero tokens spent, zero cost.
+
+```ruby
+result = SummarizeArticle.run(giant_10mb_document)
+result.status            # => :limit_exceeded
+result.validation_errors
+# => ["Input token limit exceeded: estimated 32000 tokens (heuristic ±30%), max 2000"]
+```
+
+`max_cost` fails closed when the model's pricing isn't known — register custom or fine-tuned models explicitly:
+
+```ruby
+RubyLLM::Contract::CostCalculator.register_model("ft:gpt-4o-custom",
+  input_per_1m: 3.0, output_per_1m: 6.0)
+```
+
+Or opt into a soft warning instead of a refusal when pricing is missing:
+
+```ruby
+max_cost 0.01, on_unknown_pricing: :warn
+```
+
+Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
+
+### Preflight cost estimates
+
+Check what a call is likely to cost before invoking it:
+
+```ruby
+SummarizeArticle.estimate_cost(input: article_text)
+# => {
+#      model: "gpt-4.1-mini",
+#      input_tokens: 812, output_tokens_estimate: 4000,
+#      estimated_cost: 0.00243
+#    }
+
+# Estimate what a full eval would cost across candidate models
+SummarizeArticle.estimate_eval_cost("regression",
+  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
+# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }
+```
+
+`estimate_cost` returns `nil` when pricing isn't registered. `estimate_eval_cost` silently treats unknown-pricing cases as `$0.00` and sums the rest — it does **not** fail closed the way `max_cost` does. Treat its output as a floor, not a guarantee; register pricing via `CostCalculator.register_model` before relying on it for budget decisions.
+
+## `output_schema` vs `with_schema`
+
+`with_schema` in `ruby_llm` tells the provider to force a specific JSON structure. `output_schema` in this gem does the same thing (calls `with_schema` under the hood) **plus** validates the response client-side. Cheaper models sometimes ignore schema constraints — `with_schema` is a request; `output_schema` is a request plus verification.
+
+## See also
+
+- [Prompt AST](prompt_ast.md) — prompt DSL variants: `system`, `rule`, `section`, `example`, `user`, and dynamic prompts with `|input|`.
+- [Eval-First](eval_first.md) — datasets, baselines, A/B gates, the workflow that makes the above evals useful.
+- [Optimizing retry_policy](optimizing_retry_policy.md) — find the cheapest viable fallback list with `compare_models` and `optimize_retry_policy`.
+- [Testing](testing.md) — test adapter, `stub_step`, full RSpec + Minitest matcher reference.
+- [Output Schema](output_schema.md) — nested objects in arrays, constraints, pattern reference.
+- [Rails integration](rails_integration.md) — where step classes live, initializer, jobs, logging, specs, CI gate.