ruby_llm-contract 0.10.1 → 0.10.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 66d3692be182ac8958132d06b53c1d47e01d07769d0e7a1942f81b71eb23a759
-  data.tar.gz: f254fcbc6bc8c2a42a78cff9ea3b24b23f04d860c60d79084bbd8a5d6ea6ec0b
+  metadata.gz: 4dc75c6a5ec293047bd2e2c480cf0aaff0fb3f5043f668a9291553a30002ed53
+  data.tar.gz: 4bd5960d8ff68b8708992755a95e8b265f28cdc3734db80dbdeada251eebfba9
 SHA512:
-  metadata.gz: f035aca56a5c0e2ca0607a06431cbb76a6ff4cc45f2f35fe9309d534b881910fa05d63c5c581a6e9cbac5374377f6fc187e521a844a653da5b53d612e9a23a89
-  data.tar.gz: 31bb34333a28224e7c00ce2dea7b3905dce878c320fc0fb8a0e5db1313259a0e12e136ab6cc74126b25f0a47984830ae2172e3f824ccfdcb28f847929f40080a
+  metadata.gz: aa5741f4728be169c42ee861aa6c8366481b21d49e2ff66105028d22d027ba7939959c784b13f7f8707ae0483608a0360cd035aee2ba4eadd6c2e4b30edfe680
+  data.tar.gz: 875ffe28d243189193d8dad0a4d2201329b92d9701f9eec49349719ac20d678b4de21a15db7db7f827fa19bb5d1adbdf22496186a17802147156cc88e1664a9d

data/CHANGELOG.md

@@ -1,5 +1,34 @@
 # Changelog
 
+## 0.10.3 (2026-06-10)
+
+Hot-fix release: the schema validator now correctly accepts `nil` on **required-but-nullable** fields. This unblocks OpenAI structured-output strict mode, where every property has to be in `required` and "nullable" is expressed as a `null` branch in `anyOf`/`oneOf` or as an array `type` — exactly the combination the prior validator wrongly rejected.
+
+### Fixed
+
+- **`SchemaValidator` no longer rejects legal `nil` on required-nullable fields.** Pre-0.10.3 (every published version 0.2.x–0.10.2) conflated "required" (must be present) with "non-nullable" (cannot be null) — JSON Schema treats those as orthogonal. The validator now honours all three idioms for nullability:
+  - `type: ["string", "null"]` (array form)
+  - `type: "null"` (degenerate scalar form)
+  - `anyOf: [{type: "string"}, {type: "null"}]` and `oneOf` equivalents
+
+  **Adopter impact:** if you bypassed the bug by setting `required: false` + disabling OpenAI `strict: true` on every nullable field, you can now restore `strict: true` and keep the field required (= OpenAI's standard nullable idiom). Non-nullable required fields still reject `nil` exactly as before; this is a strictly additive fix.
+
+  Discovered via dogfooding in a production adopter using OpenAI strict structured output with 17 nullable fields (`"set the rest to null"` prompt). Smoking-gun mutation: remove `nullable_schema?` short-circuit from `validate_nil_field` — the three new spec cases in `spec/ruby_llm/contract/contract/schema_validator_spec.rb` fail.
+
+## 0.10.2 (2026-06-10)
+
+Patch release: ship the `docs/guide/` directory inside the gem so adopters and LLM integration agents can read the manuals locally (via `bundle show ruby_llm-contract` or `gem unpack`) without an internet round-trip to GitHub. No code behavior change.
+
+### Added
+
+- **`docs/guide/*` is now packaged with the gem** (14 files, ~120 KB). Previously the README's "See also" links pointed at `docs/guide/getting_started.md`, `docs/guide/optimizing_retry_policy.md`, etc., but those files were stripped from the published gem - LLM integration agents (Cursor, Claude Code, Copilot) reported "no documentation in the gem" because the links 404'd locally. `docs/ideas/` and `doc/decisions/` remain excluded.
+
+### Fixed
+
+- **`models:` keyword form documented + pinned for hash configs.** `retry_policy models: ["gpt-5-nano", { model: "gpt-5-mini", reasoning_effort: "high" }]` is now covered by a spec and shown in [getting_started.md](docs/guide/getting_started.md). The block form (`escalate(...)`) and the keyword form share the same `@configs` storage; both forms accept config hashes.
+- **`reasoning_effort` examples corrected to gpt-5 family.** Pre-0.10.2 docs and specs paired `reasoning_effort` with `gpt-4.1-*` model names, which is incorrect: `gpt-4.1` is not a reasoning model. Updated across `docs/guide/getting_started.md`, `docs/guide/optimizing_retry_policy.md`, `CHANGELOG.md`, and 7 spec files. Non-reasoning `gpt-4.1` examples (model fallback chains without `reasoning_effort`) are unchanged.
+- **Version mentions corrected in README and multimodal guide.** README FAQ ("Upgraded to 0.9.0 - why?") now reads "Upgraded to 0.10.0 from 0.8.x"; `docs/guide/multimodal_input.md` references `0.10.0+` and `0.10.x` instead of `0.9.0`. 0.9.0 and 0.9.1 were tagged but never published to rubygems; adopters jump from 0.8.0 directly to 0.10.x.
+
 ## 0.10.1 (2026-06-01)
 
 Patch release fixing gem packaging. 0.10.0 was yanked from rubygems.org due to the issue documented below; 0.10.1 is the recommended upgrade target. No code behavior change vs 0.10.0.
@@ -254,7 +283,7 @@ end
 - **`Step.recommend`** — `ClassifyTicket.recommend("eval", candidates: [...], min_score: 0.95)` runs eval on all candidates and returns a `Recommendation` with optimal model, retry chain, rationale, savings vs current config, and `to_dsl` code output.
 - **Candidates as configurations** — `candidates:` accepts `{ model:, reasoning_effort: }` hashes, not just model name strings. `gpt-5-mini` with `reasoning_effort: "low"` is a different candidate than with `"high"`.
 - **`compare_models` extended** — new `candidates:` parameter alongside existing `models:` (backward compatible). Candidate labels include reasoning effort in output table.
-- **Per-attempt `reasoning_effort` in retry policies** — `escalate` accepts config hashes: `escalate({ model: "gpt-4.1-nano" }, { model: "gpt-5-mini", reasoning_effort: "high" })`. Each attempt gets its own reasoning_effort forwarded to the provider.
+- **Per-attempt `reasoning_effort` in retry policies** — `escalate` accepts config hashes: `escalate({ model: "gpt-5-nano" }, { model: "gpt-5-mini", reasoning_effort: "high" })`. Each attempt gets its own reasoning_effort forwarded to the provider.
 - **`pass_rate_ratio`** — numeric float (0.0–1.0) on `Report` and `ReportStats`, complementing the string `pass_rate` (`"3/5"`).
 - **History entries enriched** — `save_history!` accepts `reasoning_effort:` and stores `model`, `reasoning_effort`, `pass_rate_ratio` in JSONL entries.
 

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ruby_llm-contract (0.10.1)
+    ruby_llm-contract (0.10.3)
       dry-types (~> 1.7)
       ruby_llm (~> 1.12)
       ruby_llm-schema (~> 0.3)
@@ -258,7 +258,7 @@ CHECKSUMS
   rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   ruby_llm (1.14.0) sha256=57c6f7034fc4a44504ea137d70f853b07824f1c1cdbe774ab3ab3522e7098deb
-  ruby_llm-contract (0.10.1)
+  ruby_llm-contract (0.10.3)
   ruby_llm-schema (0.3.0) sha256=a591edc5ca1b7f0304f0e2261de61ba4b3bea17be09f5cf7558153adfda3dec6
   ruby_parser (3.22.0) sha256=1eb4937cd9eb220aa2d194e352a24dba90aef00751e24c8dfffdb14000f15d23
   rubycritic (4.12.0) sha256=024fed90fe656fa939f6ea80aab17569699ac3863d0b52fd72cb99892247abc8

data/README.md

@@ -157,7 +157,7 @@ Different layers, complementary. [`ruby_llm-tribunal`](https://github.com/Alqemi
 
 ## Status & versioning
 
-Pre-1.0 (currently **0.10.1**). Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 0.10.1` until 1.0 ships.
+Pre-1.0 (currently **0.10.3**). Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 0.10.3` until 1.0 ships.
 
 ## FAQ
 
@@ -167,7 +167,7 @@ Pre-1.0 (currently **0.10.1**). Semver tracked; breaking changes flagged in [CHA
 
 **Where in a Rails app?** Default `app/contracts/`. The Railtie reloads `app/contracts/eval/` and `app/steps/eval/` in development; any autoloaded directory also works. See [Rails integration](docs/guide/rails_integration.md).
 
-**Upgraded to 0.9.0 and my contract started refusing — why?** 0.9.0 added multimodal input. If your contract has `max_cost` or `max_input` set AND now receives `context: { attachment: ... }`, you must declare `attachment_token_estimate(n)` (conservative input-token budget for the attachment) — otherwise the call fails closed with `:limit_exceeded`. The gem cannot bound vision/PDF cost without your estimate. Opt out per-step with `on_unknown_attachment_size :warn`. Text-only contracts are unaffected. See [multimodal input guide](docs/guide/multimodal_input.md).
+**Upgraded to 0.10.0 from 0.8.x and my contract started refusing — why?** 0.10.0 added multimodal input. If your contract has `max_cost` or `max_input` set AND now receives `context: { attachment: ... }`, you must declare `attachment_token_estimate(n)` (conservative input-token budget for the attachment) — otherwise the call fails closed with `:limit_exceeded`. The gem cannot bound vision/PDF cost without your estimate. Opt out per-step with `on_unknown_attachment_size :warn`. Text-only contracts are unaffected. See [multimodal input guide](docs/guide/multimodal_input.md).
 
 ## License
 

data/docs/architecture.md

@@ -0,0 +1,50 @@
+# Architecture
+
+```
+RubyLLM::Contract::Pipeline::Base   # optional: compose steps
+  ├── Pipeline::Runner           # sequential execution, fail-fast, trace, timeout
+  └── Pipeline::Result           # per-step outputs + aggregated trace
+
+RubyLLM::Contract::Step::Base       # single contracted step
+  ├── Step::Dsl                  # DSL macros (prompt, validate, output_schema, etc.)
+  ├── Step::RetryPolicy          # attempts, models, reasoning_effort, retry_on
+  ├── Step::RetryExecutor        # retry loop driven by RetryPolicy
+  ├── Step::LimitChecker         # preflight cost / input / output checks
+  ├── Step::Runner               # runtime flow (single attempt)
+  ├── Step::Result               # status + outputs + errors + trace
+  ├── Step::Trace                # model, latency, tokens, cost, attempts
+  ├── Prompt::AST                # structured prompt (immutable)
+  │     ├── Prompt::Builder      # DSL: system, rule, example, user, section
+  │     └── Prompt::Renderer     # AST → messages array
+  ├── Contract::Definition       # parse strategy + validates
+  │     ├── Contract::Parser     # :json / :text (auto-inferred from output type)
+  │     ├── Contract::Validator  # runs parse + schema + validates + observations
+  │     └── Contract::SchemaValidator  # JSON Schema validation (nested)
+  ├── CostCalculator             # per-step cost estimation from model pricing
+  ├── TokenEstimator             # input-token count estimation for limit checks
+  ├── estimate_cost              # single-call cost estimate (class method)
+  ├── estimate_eval_cost         # cost estimate for a full eval across models
+  └── Adapters::Base             # provider interface
+        ├── Adapters::RubyLLM    # real LLM calls via ruby_llm (any provider)
+        └── Adapters::Test       # canned responses for specs and examples
+
+RubyLLM::Contract::Eval             # quality measurement
+  ├── Eval::EvalDefinition       # define_eval DSL (verify, add_case, default_input, sample_response)
+  ├── Eval::Dataset              # test cases
+  ├── Eval::Runner               # sequential or concurrent execution
+  ├── Eval::Report               # score, pass_rate, per-case results
+  ├── Eval::AggregatedReport     # merged reports across models or runs
+  ├── Eval::CaseResult           # value object (name, passed?, output, expected, mismatches, cost)
+  ├── Eval::ExpectationEvaluator # expected / expected_traits / evaluator proc
+  ├── Eval::ModelComparison      # compare_models result (table, best_for, candidate configs)
+  ├── Eval::Recommender          # model recommendation algorithm (candidates → optimal config)
+  ├── Eval::Recommendation       # recommendation result (best, retry_chain, savings, to_dsl)
+  ├── Eval::RetryOptimizer       # optimize_retry_policy result (per-eval breakdown, fallback list)
+  ├── Eval::BaselineDiff         # save_baseline! + without_regressions comparison
+  ├── Eval::PromptDiffComparator # compare_with prompt A/B diff
+  └── Eval::EvalHistory          # time-series view across saved reports
+
+RubyLLM::Contract::RakeTask         # rake ruby_llm_contract:eval
+RubyLLM::Contract::OptimizeRakeTask # rake ruby_llm_contract:optimize
+RubyLLM::Contract::Railtie          # auto-loads eval files in Rails
+```

data/docs/guide/best_practices.md

@@ -0,0 +1,136 @@
+# Best Practices
+
+> Read this when writing your first `validate` rules and you want patterns that catch real bugs instead of restating schema.
+
+Schema guarantees valid JSON structure. An LLM can still return structurally perfect JSON that is **semantically wrong**. Schema handles _shape_, validates handle _meaning_.
+
+All examples extend the `SummarizeArticle` step from the [README](../../README.md).
+
+## 1. Guard against empty / placeholder output
+
+**Why it matters:** a cheap model that answers `{"tldr": "This article discusses...", "takeaways": ["This article discusses X", ...]}` passes the schema but renders a broken UI card that tells the user nothing. The validate catches it before `Article.update!` persists it.
+
+```ruby
+output_schema do
+  string :tldr
+  array  :takeaways, of: :string, min_items: 3, max_items: 5
+  string :tone, enum: %w[neutral positive negative analytical]
+end
+
+validate("tldr is substantive") do |o, _|
+  o[:tldr].to_s.split.length >= 5   # at least five words
+end
+
+validate("no boilerplate takeaways") do |o, _|
+  o[:takeaways].none? { |t| t.downcase.start_with?("this article") }
+end
+```
+
+## 2. Cross-validate output against input
+
+**Why it matters:** a lazy model will return the article text verbatim as the "summary", or invent takeaways about topics the article never mentions. The 2-arity form is how you catch answers that are internally consistent but unfaithful to the actual input.
+
+`validate` blocks with 2-arity `|output, input|` compare the model's answer against what was asked:
+
+```ruby
+validate("tldr is shorter than the article") do |output, input|
+  output[:tldr].length < input.length / 2
+end
+
+validate("every takeaway appears, in spirit, in the article") do |output, input|
+  output[:takeaways].all? { |t|
+    # cheap keyword overlap heuristic
+    t.downcase.split.any? { |w| input.downcase.include?(w) && w.length > 4 }
+  }
+end
+```
+
+## 3. Conditional logic schema can't express
+
+**Why it matters:** customer success filters on `tone == "negative"` to route angry users to a human. If the model labels an outage complaint "negative" but the takeaways are all positive-sounding, the filter runs on a label that doesn't match the content — the routing breaks silently.
+
+```ruby
+validate("negative tone requires at least one concrete concern") do |output, _input|
+  next true unless output[:tone] == "negative"
+  output[:takeaways].any? { |t| t.match?(/fail|break|crash|outage|vulnerab|risk/i) }
+end
+```
+
+A model that picks `tone: "negative"` but gives three upbeat takeaways fails this check. Schema can't catch it because each takeaway is, individually, a valid string.
+
+## 4. Content quality
+
+**Why it matters:** a TL;DR with `## Summary` leaks markdown into a plain-text card. A one-word takeaway ("Fast.") wastes a UI slot. A leaked `{article}` placeholder reveals the prompt template to end users. All pass schema; all embarrass you in front of customers.
+
+```ruby
+validate("no markdown headings in the TL;DR") do |o, _|
+  !o[:tldr].match?(/^\#{1,6}\s/)
+end
+
+validate("takeaways aren't single words") do |o, _|
+  o[:takeaways].all? { |t| t.split.length >= 3 }
+end
+
+validate("no template placeholders leaked") do |o, _|
+  !(o[:tldr] + o[:takeaways].join(" ")).include?("{")
+end
+```
+
+## 5. Pipeline: preserve data between steps
+
+In a pipeline, each step only sees the previous step's output. If a later step needs original article metadata, an intermediate step must carry it through. Suppose a pipeline `SummarizeArticle → GenerateHashtags`, where `GenerateHashtags` needs the `tone` from the summary:
+
+```ruby
+class GenerateHashtags < RubyLLM::Contract::Step::Base
+  input_type Hash
+
+  output_schema do
+    # Carry through the fields a downstream step (or the caller) might need
+    string :tone, enum: %w[neutral positive negative analytical]
+    array  :hashtags, of: :string, min_items: 2, max_items: 5
+  end
+
+  prompt do
+    rule "Preserve the tone label from the input unchanged."
+    user "Summary: {tldr}\nTone: {tone}\nTakeaways: {takeaways}"
+  end
+
+  validate("tone preserved") { |o, input| o[:tone] == input[:tone] }
+end
+```
+
+The explicit `validate("tone preserved")` catches the case where the model silently rewrites the tone during a downstream transform.
+
+## 6. Model fallback
+
+**Why it matters:** 80% of production articles are short and simple — `gpt-4.1-nano` handles them for ~$0.0001. The remaining 20% are dense, critical, or multi-topic — those need `gpt-4.1-mini` or `gpt-4.1`. Paying `gpt-4.1` rates for every call when nano is enough for most is throwing money away. Contracts tell you when nano wasn't enough, so fallback is cost-aware, not hope-based.
+
+Small models are cheap but hallucinate. Big models are accurate but expensive. Start cheap, fall back only when validates catch a failure:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
+  end
+
+  validate("TL;DR fits the card")  { |o, _| o[:tldr].length <= 200 }
+  validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }
+
+  retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]
+end
+```
+
+**Key insight:** without contracts, you can't do model fallback — you'd have no way to know if the cheap model's output is good enough. Validates are the quality gate that makes cost optimization possible. See [Optimizing retry_policy](optimizing_retry_policy.md) for how to find the cheapest viable fallback list for your step.
+
+## Summary
+
+| What to validate | Use |
+|---|---|
+| Field types, enums, ranges, required fields | `output_schema` |
+| Output makes sense given the input | `validate` (2-arity `\|output, input\|`) |
+| Conditional business rules | `validate` |
+| Content quality (not empty, not template) | `validate` |
+| Data preserved across pipeline steps | `validate` + schema carry-through |
+| Cost optimization via model fallback | `retry_policy` + `validate` as quality gate |

data/docs/guide/eval_first.md

@@ -0,0 +1,192 @@
+# Eval-First
+
+> Read this when you need to prevent silent prompt regressions in CI. Skip if your LLM output is evaluated only by humans and never gated in an automated pipeline.
+
+If you change prompts by feel, you ship regressions by feel.
+
+Concrete scenario: `SummarizeArticle` has been running in production for two weeks. Customer success notices that complaints about service outages keep getting `tone: "analytical"` instead of `"negative"` — so their "critical feedback" filter silently misses angry users. Someone tweaks the system prompt to emphasise negative sentiment. It fixes the outage article but now three neutral product-update articles get misclassified as `"negative"`. You find out from a Slack thread.
+
+That is the cost of prompt-by-feel. Evals are how you stop it.
+
+`ruby_llm-contract` works best when you treat evals as the source of truth:
+
+1. Capture real failures from production (the outage article, verbatim).
+2. Turn them into eval cases (`add_case "service outage complaint"`).
+3. Change the prompt.
+4. Re-run the same eval — plus all previously-passing cases.
+5. Merge only if the eval says quality improved or stayed safe on every case.
+
+## Core rule
+
+**Do not start with the prompt. Start with the eval.**
+
+Using the `SummarizeArticle` step from the [README](../../README.md):
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "ruby release",
+           input: "Ruby 3.4 shipped with frozen string literals...",
+           expected: { tone: "analytical" }  # partial match
+
+  add_case "critical review",
+           input: "Mesh networking hardware failed under load...",
+           expected: { tone: "negative" }
+end
+```
+
+Only after the eval exists, touch: `system`, `rule`, `example`, `validate`, prompt versions.
+
+## Three eval kinds
+
+### 1. `smoke` — wiring check, offline
+
+```ruby
+SummarizeArticle.define_eval("smoke") do
+  default_input "Ruby 3.4 shipped with frozen string literals..."
+  sample_response({
+    tldr: "...",
+    takeaways: ["point one", "point two", "point three"],
+    tone: "analytical"
+  })
+end
+```
+
+`sample_response` returns canned data. Zero API calls. Verifies schema + validates parse and the step wiring is intact. **Not a quality signal.**
+
+### 2. `regression` — real quality measurement
+
+Represent real traffic and known failures. Good sources: production logs, bad completions, incidents, QA edge cases, cases a human had to correct.
+
+Every production failure becomes `add_case`. That's the flywheel.
+
+### 3. `ab` — prompt iteration
+
+Compare two prompt versions on the same eval:
+
+```ruby
+diff = SummarizeArticleV2.compare_with(
+  SummarizeArticleV1,
+  eval: "regression",
+  model: "gpt-4.1-mini"
+)
+
+diff.safe_to_switch?  # => true if no cases regressed
+```
+
+This is the cleanest eval-first move: same eval, same cases, two prompt versions, one answer.
+
+## What counts as eval-first
+
+**Good** — eval exists before the prompt change:
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "short article", input: "...", expected: { tone: "neutral" }
+end
+
+# Prompt iteration happens afterward
+diff = SummarizeArticleV2.compare_with(
+  SummarizeArticleV1, eval: "regression", model: "gpt-4.1-mini"
+)
+```
+
+**Bad**:
+
+```ruby
+# Tweak prompt for an hour
+# Maybe add an example
+# Maybe tighten a rule
+# Then eyeball one or two responses
+```
+
+That's prompt guessing, not eval-first.
+
+## `sample_response`: useful, but not the main thing
+
+Good for: offline smoke tests, local development, testing evaluator wiring, checking schema + validate behavior with zero API calls.
+
+Not enough for real prompt decisions. For those:
+
+- `run_eval(..., context: { model: "..." })` with a real model, or pass an explicit adapter.
+- `compare_with(...)` for prompt A/B.
+
+`compare_with` intentionally ignores `sample_response` — canned data would make both sides look the same.
+
+## Parallel eval runs
+
+For larger datasets, `run_eval` accepts a `concurrency:` argument — cases run in parallel using a thread pool:
+
+```ruby
+report = SummarizeArticle.run_eval("regression",
+  context: { model: "gpt-4.1-mini" },
+  concurrency: 8)
+```
+
+Same accepted by `compare_models` and `optimize_retry_policy`. Thread count is a ceiling — dataset order of results is preserved. Keep it low enough to respect the provider's rate limits.
+
+## Budgeting an eval before you run it
+
+`estimate_eval_cost` gives you a cost projection without calling the LLM:
+
+```ruby
+SummarizeArticle.estimate_eval_cost("regression",
+  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
+# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }
+```
+
+Use it in CI to decide which models are worth running regression on, or to cap worst-case spend per build.
+
+## Team workflow
+
+1. **Build one eval that matters** — 10–30 cases representing real mistakes and important business paths.
+2. **Gate CI** — `pass_eval("regression").with_context(model: "...").with_minimum_score(0.8)`. See [Getting Started](getting_started.md) for the full matcher chain.
+3. **Save a baseline** — `report.save_baseline!` makes quality drift visible.
+4. **Change prompts only through comparison** — `pass_eval(...).compared_with(SummarizeArticleV1)` in CI so any regression blocks the merge.
+5. **Feed production failures back** — every miss in prod → new `add_case`, then fix. The eval gets stronger over time.
+
+## Few-shot examples fit naturally
+
+Adding `example input: ..., output: ...` inside the prompt is still a prompt change. The eval-first way:
+
+1. Add examples to the prompt.
+2. Rerun the existing regression eval.
+3. `compare_with` against the old prompt.
+
+Few-shot isn't the proof. The eval is.
+
+## Model selection comes after prompt stability
+
+Don't optimize cost before you stabilize quality:
+
+1. Build `regression`.
+2. Improve the prompt with `compare_with`.
+3. Lock quality in CI.
+4. Then run `compare_models` (see [Optimizing retry_policy](optimizing_retry_policy.md)).
+
+```ruby
+comparison = SummarizeArticle.compare_models(
+  "regression",
+  candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }, { model: "gpt-4.1" }]
+)
+
+comparison.best_for(min_score: 0.95)
+```
+
+## Strong defaults for teams
+
+- `smoke` uses `sample_response`.
+- `regression` uses real model calls.
+- Every prompt change uses `compare_with`.
+- Every merge runs `pass_eval`.
+- Every production failure becomes a new `add_case`.
+
+## Short version
+
+1. Write `define_eval` before touching the prompt.
+2. Treat `sample_response` as smoke only.
+3. Use `run_eval("name", context: { model: "..." })` for real quality measurement.
+4. Use `compare_with` for every serious prompt change.
+5. Gate merges with `pass_eval`.
+6. Feed every production miss back into the dataset.
+
+Prompts stop being vibes and start being engineering.