ruby_llm-contract 0.8.0 → 0.10.2

data/docs/guide/eval_first.md

@@ -0,0 +1,192 @@
+# Eval-First
+
+> Read this when you need to prevent silent prompt regressions in CI. Skip if your LLM output is evaluated only by humans and never gated in an automated pipeline.
+
+If you change prompts by feel, you ship regressions by feel.
+
+Concrete scenario: `SummarizeArticle` has been running in production for two weeks. Customer success notices that complaints about service outages keep getting `tone: "analytical"` instead of `"negative"` — so their "critical feedback" filter silently misses angry users. Someone tweaks the system prompt to emphasise negative sentiment. It fixes the outage article but now three neutral product-update articles get misclassified as `"negative"`. You find out from a Slack thread.
+
+That is the cost of prompt-by-feel. Evals are how you stop it.
+
+`ruby_llm-contract` works best when you treat evals as the source of truth:
+
+1. Capture real failures from production (the outage article, verbatim).
+2. Turn them into eval cases (`add_case "service outage complaint"`).
+3. Change the prompt.
+4. Re-run the same eval — plus all previously-passing cases.
+5. Merge only if the eval says quality improved or stayed safe on every case.
+
+## Core rule
+
+**Do not start with the prompt. Start with the eval.**
+
+Using the `SummarizeArticle` step from the [README](../../README.md):
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "ruby release",
+           input: "Ruby 3.4 shipped with frozen string literals...",
+           expected: { tone: "analytical" }  # partial match
+
+  add_case "critical review",
+           input: "Mesh networking hardware failed under load...",
+           expected: { tone: "negative" }
+end
+```
+
+Only after the eval exists, touch: `system`, `rule`, `example`, `validate`, prompt versions.
+
+## Three eval kinds
+
+### 1. `smoke` — wiring check, offline
+
+```ruby
+SummarizeArticle.define_eval("smoke") do
+  default_input "Ruby 3.4 shipped with frozen string literals..."
+  sample_response({
+    tldr: "...",
+    takeaways: ["point one", "point two", "point three"],
+    tone: "analytical"
+  })
+end
+```
+
+`sample_response` returns canned data. Zero API calls. Verifies schema + validates parse and the step wiring is intact. **Not a quality signal.**
+
+### 2. `regression` — real quality measurement
+
+Represent real traffic and known failures. Good sources: production logs, bad completions, incidents, QA edge cases, cases a human had to correct.
+
+Every production failure becomes `add_case`. That's the flywheel.
+
+### 3. `ab` — prompt iteration
+
+Compare two prompt versions on the same eval:
+
+```ruby
+diff = SummarizeArticleV2.compare_with(
+  SummarizeArticleV1,
+  eval: "regression",
+  model: "gpt-4.1-mini"
+)
+
+diff.safe_to_switch?  # => true if no cases regressed
+```
+
+This is the cleanest eval-first move: same eval, same cases, two prompt versions, one answer.
+
+## What counts as eval-first
+
+**Good** — eval exists before the prompt change:
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "short article", input: "...", expected: { tone: "neutral" }
+end
+
+# Prompt iteration happens afterward
+diff = SummarizeArticleV2.compare_with(
+  SummarizeArticleV1, eval: "regression", model: "gpt-4.1-mini"
+)
+```
+
+**Bad**:
+
+```ruby
+# Tweak prompt for an hour
+# Maybe add an example
+# Maybe tighten a rule
+# Then eyeball one or two responses
+```
+
+That's prompt guessing, not eval-first.
+
+## `sample_response`: useful, but not the main thing
+
+Good for: offline smoke tests, local development, testing evaluator wiring, checking schema + validate behavior with zero API calls.
+
+Not enough for real prompt decisions. For those:
+
+- `run_eval(..., context: { model: "..." })` with a real model, or pass an explicit adapter.
+- `compare_with(...)` for prompt A/B.
+
+`compare_with` intentionally ignores `sample_response` — canned data would make both sides look the same.
+
+## Parallel eval runs
+
+For larger datasets, `run_eval` accepts a `concurrency:` argument — cases run in parallel using a thread pool:
+
+```ruby
+report = SummarizeArticle.run_eval("regression",
+  context: { model: "gpt-4.1-mini" },
+  concurrency: 8)
+```
+
+Same accepted by `compare_models` and `optimize_retry_policy`. Thread count is a ceiling — dataset order of results is preserved. Keep it low enough to respect the provider's rate limits.
+
+## Budgeting an eval before you run it
+
+`estimate_eval_cost` gives you a cost projection without calling the LLM:
+
+```ruby
+SummarizeArticle.estimate_eval_cost("regression",
+  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
+# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }
+```
+
+Use it in CI to decide which models are worth running regression on, or to cap worst-case spend per build.
+
+## Team workflow
+
+1. **Build one eval that matters** — 10–30 cases representing real mistakes and important business paths.
+2. **Gate CI** — `pass_eval("regression").with_context(model: "...").with_minimum_score(0.8)`. See [Getting Started](getting_started.md) for the full matcher chain.
+3. **Save a baseline** — `report.save_baseline!` makes quality drift visible.
+4. **Change prompts only through comparison** — `pass_eval(...).compared_with(SummarizeArticleV1)` in CI so any regression blocks the merge.
+5. **Feed production failures back** — every miss in prod → new `add_case`, then fix. The eval gets stronger over time.
+
+## Few-shot examples fit naturally
+
+Adding `example input: ..., output: ...` inside the prompt is still a prompt change. The eval-first way:
+
+1. Add examples to the prompt.
+2. Rerun the existing regression eval.
+3. `compare_with` against the old prompt.
+
+Few-shot isn't the proof. The eval is.
+
+## Model selection comes after prompt stability
+
+Don't optimize cost before you stabilize quality:
+
+1. Build `regression`.
+2. Improve the prompt with `compare_with`.
+3. Lock quality in CI.
+4. Then run `compare_models` (see [Optimizing retry_policy](optimizing_retry_policy.md)).
+
+```ruby
+comparison = SummarizeArticle.compare_models(
+  "regression",
+  candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }, { model: "gpt-4.1" }]
+)
+
+comparison.best_for(min_score: 0.95)
+```
+
+## Strong defaults for teams
+
+- `smoke` uses `sample_response`.
+- `regression` uses real model calls.
+- Every prompt change uses `compare_with`.
+- Every merge runs `pass_eval`.
+- Every production failure becomes a new `add_case`.
+
+## Short version
+
+1. Write `define_eval` before touching the prompt.
+2. Treat `sample_response` as smoke only.
+3. Use `run_eval("name", context: { model: "..." })` for real quality measurement.
+4. Use `compare_with` for every serious prompt change.
+5. Gate merges with `pass_eval`.
+6. Feed every production miss back into the dataset.
+
+Prompts stop being vibes and start being engineering.

data/docs/guide/getting_started.md

@@ -0,0 +1,199 @@
+# Getting Started
+
+> Read this to walk through every feature on one concrete step for the first time.
+
+The README shows a minimal `SummarizeArticle` step. This guide walks through the features you reach for as production requirements grow: budget caps so runaway inputs don't drain your LLM provider budget, evals so you catch regressions in CI, and CI gating so a merge that lowers accuracy gets blocked.
+
+## The walkthrough
+
+Start with the README example, then add features one layer at a time. Each is optional — use what you need.
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  # 1. Prompt (required)
+  prompt <<~PROMPT
+    Summarize this article for a UI card. Return a short TL;DR,
+    3 to 5 key takeaways, and a tone label.
+
+    {input}
+  PROMPT
+
+  # 2. Schema — sent to the provider via with_schema, validated client-side
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
+  end
+
+  # 3. Business rules — things JSON Schema cannot express
+  validate("TL;DR fits the card")  { |o, _| o[:tldr].length <= 200 }
+  validate("takeaways are unique") { |o, _| o[:takeaways] == o[:takeaways].uniq }
+
+  # 4. Retry with model fallback on validation_failed / parse_error
+  retry_policy models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1]
+
+  # 5. Refuse before calling the LLM if input is too large or estimated cost exceeds the cap
+  max_input  2_000
+  max_output 4_000
+  max_cost   0.01
+end
+```
+
+## Validation and retry behavior
+
+When the cheap model returns output that fails a `validate` block or can't be parsed, retry falls back to the next model in `models:` and tries again.
+
+```ruby
+result = SummarizeArticle.run(article_text)
+
+result.status           # => :ok
+result.parsed_output    # => { tldr: "...", takeaways: [...], tone: "analytical" }
+result.trace[:model]    # => "gpt-4.1-mini"  (first model that passed)
+result.trace[:cost]     # => 0.00052  (sum of all attempts)
+result.trace[:attempts]
+# => [
+#   { attempt: 1, model: "gpt-4.1-nano", status: :validation_failed,
+#     cost: 0.00010, latency_ms: 45, ... },
+#   { attempt: 2, model: "gpt-4.1-mini", status: :ok,
+#     cost: 0.00042, latency_ms: 92, ... }
+# ]
+```
+
+If the whole chain exhausts, `result.status` is the status of the last attempt (`:validation_failed` or `:parse_error`) and `result.parsed_output` is the last attempt's output. The caller decides what to do — ship it anyway, fall back to a template, or raise.
+
+### Per-attempt reasoning effort
+
+`models:` accepts config hashes as well as model-name strings, so a fallback can "try harder" (more reasoning) on retry, not just switch model:
+
+```ruby
+retry_policy models: [
+  "gpt-5-nano",                                      # attempt 1: cheap + fast
+  { model: "gpt-5-mini", reasoning_effort: "high" }  # attempt 2: stronger + more reasoning
+]
+```
+
+`reasoning_effort` is a gpt-5 family feature (gpt-4.1 is not a reasoning model). The per-attempt value is forwarded via `with_thinking` (provider-agnostic — OpenAI `reasoning_effort` and Anthropic extended-thinking budget both supported). Passing it alongside a non-reasoning model is forwarded unchanged to the provider, which will either ignore it or reject the request — the gem does not guard against this.
+
+## Evals and CI gates
+
+An eval is a named scenario you can run to verify the step still works. `sample_response` makes it offline — zero API calls — so CI can run it on every merge without burning budget.
+
+```ruby
+SummarizeArticle.define_eval("smoke") do
+  default_input <<~ARTICLE
+    Ruby 3.4 ships with frozen string literals on by default, measurable YJIT
+    speedups on Rails workloads, and tightened Warning.warn category filtering.
+    The release notes also mention several parser fixes and faster keyword
+    argument handling.
+  ARTICLE
+
+  sample_response({
+    tldr: "Ruby 3.4 brings frozen string literals by default, YJIT speedups, and parser fixes.",
+    takeaways: [
+      "Frozen string literals are the default",
+      "YJIT adds measurable speedups on Rails workloads",
+      "Warning.warn category filtering is tighter"
+    ],
+    tone: "analytical"
+  })
+end
+
+report = SummarizeArticle.run_eval("smoke")
+report.passed?  # => true — schema + validates pass on the canned response
+report.score    # => 1.0
+report.print_summary
+```
+
+For real regression testing, define cases with expected output (online — calls the LLM):
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "ruby release",
+           input: "Ruby 3.4 was released...",
+           expected: { tone: "analytical" }  # partial match
+
+  add_case "critical review",
+           input: "The new mesh networking hardware failed under load...",
+           expected: { tone: "negative" }
+end
+```
+
+Gate CI on score and cost thresholds:
+
+```ruby
+# RSpec — blocks merge if accuracy drops or cost spikes
+expect(SummarizeArticle).to pass_eval("regression")
+  .with_minimum_score(0.8)
+  .with_maximum_cost(0.01)
+```
+
+Save a baseline once, then block regressions automatically:
+
+```ruby
+report = SummarizeArticle.run_eval("regression")
+report.save_baseline!
+
+# In CI:
+expect(SummarizeArticle).to pass_eval("regression").without_regressions
+```
+
+`without_regressions` fails the build only if a previously-passing case now fails — a new model version, a prompt tweak, or an upstream change that silently lowered quality.
+
+## Budget caps
+
+`max_input`, `max_output`, and `max_cost` are preflight checks — the LLM is never called if an estimate exceeds the limit. Zero tokens spent, zero cost.
+
+```ruby
+result = SummarizeArticle.run(giant_10mb_document)
+result.status            # => :limit_exceeded
+result.validation_errors
+# => ["Input token limit exceeded: estimated 32000 tokens (heuristic ±30%), max 2000"]
+```
+
+`max_cost` fails closed when the model's pricing isn't known — register custom or fine-tuned models explicitly:
+
+```ruby
+RubyLLM::Contract::CostCalculator.register_model("ft:gpt-4o-custom",
+  input_per_1m: 3.0, output_per_1m: 6.0)
+```
+
+Or opt into a soft warning instead of a refusal when pricing is missing:
+
+```ruby
+max_cost 0.01, on_unknown_pricing: :warn
+```
+
+Default is `:refuse`. Use `:warn` only when you accept running without a cost ceiling (fine-tuned models you trust, private endpoints).
+
+### Preflight cost estimates
+
+Check what a call is likely to cost before invoking it:
+
+```ruby
+SummarizeArticle.estimate_cost(input: article_text)
+# => {
+#      model: "gpt-4.1-mini",
+#      input_tokens: 812, output_tokens_estimate: 4000,
+#      estimated_cost: 0.00243
+#    }
+
+# Estimate what a full eval would cost across candidate models
+SummarizeArticle.estimate_eval_cost("regression",
+  models: %w[gpt-4.1-nano gpt-4.1-mini gpt-4.1])
+# => { "gpt-4.1-nano" => 0.00041, "gpt-4.1-mini" => 0.0018, "gpt-4.1" => 0.0092 }
+```
+
+`estimate_cost` returns `nil` when pricing isn't registered. `estimate_eval_cost` silently treats unknown-pricing cases as `$0.00` and sums the rest — it does **not** fail closed the way `max_cost` does. Treat its output as a floor, not a guarantee; register pricing via `CostCalculator.register_model` before relying on it for budget decisions.
+
+## `output_schema` vs `with_schema`
+
+`with_schema` in `ruby_llm` tells the provider to force a specific JSON structure. `output_schema` in this gem does the same thing (calls `with_schema` under the hood) **plus** validates the response client-side. Cheaper models sometimes ignore schema constraints — `with_schema` is a request; `output_schema` is a request plus verification.
+
+## See also
+
+- [Prompt AST](prompt_ast.md) — prompt DSL variants: `system`, `rule`, `section`, `example`, `user`, and dynamic prompts with `|input|`.
+- [Eval-First](eval_first.md) — datasets, baselines, A/B gates, the workflow that makes the above evals useful.
+- [Optimizing retry_policy](optimizing_retry_policy.md) — find the cheapest viable fallback list with `compare_models` and `optimize_retry_policy`.
+- [Testing](testing.md) — test adapter, `stub_step`, full RSpec + Minitest matcher reference.
+- [Output Schema](output_schema.md) — nested objects in arrays, constraints, pattern reference.
+- [Rails integration](rails_integration.md) — where step classes live, initializer, jobs, logging, specs, CI gate.

data/docs/guide/migration.md

@@ -0,0 +1,185 @@
+# Migration Guide
+
+> Read this when adopting the gem in an existing Rails app with a raw `LlmClient.call` service you want to replace. Skip if starting fresh — go to [Getting Started](getting_started.md) instead.
+
+How to adopt `ruby_llm-contract` in an existing Rails app. Examples use `SummarizeArticle` — the flagship step from the [README](../../README.md) — but the pattern applies to any single-input / structured-output service.
+
+## Step 1: Start with the simplest service
+
+Pick the LLM service with: single input → JSON output → DB save. Don't start with parallel batches or complex pipelines.
+
+## Step 2: Define the contract
+
+**Before — raw HTTP:**
+
+```ruby
+class ArticleSummaryService
+  def call(article_text)
+    response = LlmClient.new(model: "gpt-4o-mini").call(prompt(article_text))
+    JSON.parse(response[:content], symbolize_names: true)
+  end
+end
+```
+
+**After — contract:**
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  model "gpt-4.1-mini"
+
+  prompt do
+    system "You summarize articles for a UI card."
+    rule "Return valid JSON only."
+    user "{input}"
+  end
+
+  output_schema do
+    string :tldr
+    array  :takeaways, of: :string, min_items: 3, max_items: 5
+    string :tone, enum: %w[neutral positive negative analytical]
+  end
+
+  validate("TL;DR fits the card") { |o, _| o[:tldr].length <= 200 }
+  retry_policy models: %w[gpt-4.1-mini gpt-4.1]
+end
+```
+
+## Step 3: Replace the caller
+
+```ruby
+# Before
+parsed = ArticleSummaryService.new.call(article_text)
+Article.update!(summary: parsed["tldr"])
+
+# After
+result = SummarizeArticle.run(article_text)
+if result.ok?
+  Article.update!(summary: result.parsed_output[:tldr])
+else
+  Rails.logger.warn "Summary failed: #{result.status}"
+end
+```
+
+## Step 4: Add logging via around_call
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  # ... prompt, schema, validates ...
+
+  around_call do |step, input, result|
+    AiCallLog.create!(
+      ai_model: result.trace.model,
+      duration_ms: result.trace.latency_ms,
+      input_tokens: result.trace.usage&.dig(:input_tokens),
+      output_tokens: result.trace.usage&.dig(:output_tokens),
+      cost: result.trace.cost,
+      status: result.status.to_s
+    )
+  end
+end
+```
+
+## Step 5: Add eval cases
+
+Use real inputs from production logs:
+
+```ruby
+SummarizeArticle.define_eval("regression") do
+  add_case "short news",
+           input: "Ruby 3.4 ships with frozen string literals by default...",
+           expected: { tone: "analytical" }
+
+  add_case "critical review",
+           input: "The new mesh networking hardware failed under load...",
+           expected: { tone: "negative" }
+end
+```
+
+## Step 6: Find the cheapest model
+
+```ruby
+comparison = SummarizeArticle.compare_models("regression",
+  candidates: [{ model: "gpt-4.1-nano" }, { model: "gpt-4.1-mini" }])
+
+comparison.print_summary
+comparison.best_for(min_score: 0.95)  # => cheapest model at >= 95%
+```
+
+Full optimization workflow — multi-eval, fallback list, production-mode cost — in [Optimizing retry_policy](optimizing_retry_policy.md).
+
+## Step 7: Add CI gate
+
+```ruby
+# Rakefile
+require "ruby_llm/contract/rake_task"
+RubyLLM::Contract::RakeTask.new do |t|
+  t.minimum_score = 0.8
+  t.maximum_cost = 0.05
+  t.fail_on_regression = true
+  t.save_baseline = true
+end
+```
+
+**Rails apps:** if your adapter is configured in an initializer, use a Proc so context is resolved after Rails boots:
+
+```ruby
+RubyLLM::Contract::RakeTask.new do |t|
+  t.context = -> { { adapter: RubyLLM::Contract.configuration.default_adapter } }
+  t.minimum_score = 0.8
+end
+```
+
+## Common patterns
+
+| Old pattern | New pattern |
+|---|---|
+| `LlmClient.new(model:).call(prompt)` | `MyStep.run(input)` |
+| `JSON.parse(response[:content])` | `result.parsed_output` |
+| `begin; rescue; retry; end` | `retry_policy models: [...]` |
+| `body[:temperature] = 0.7` | `temperature 0.7` |
+| `AiCallLog.create(...)` | `around_call { \|s, i, r\| ... }` |
+| `response_format: JsonSchema.build(...)` | `output_schema do...end` |
+| `stub_request(:post, ...)` | `stub_step(MyStep, response: {...})` |
+
+## Anti-patterns
+
+- **Don't migrate markdown/text output services.** The gem is for structured JSON. Prose output gets no benefit from schema validation.
+- **Don't put parallelism in the gem.** Thread management is your app's concern. The gem provides the contract; you call it however you want.
+- **Don't migrate all services at once.** Start with one. Validate the pattern. Then migrate the next.
+
+## Parallel batch generation
+
+The gem handles single calls. You handle parallelism:
+
+```ruby
+class SummarizeBatch < RubyLLM::Contract::Step::Base
+  output_schema do
+    array :summaries do
+      object do
+        string :article_id
+        string :tldr
+      end
+    end
+  end
+  retry_policy models: %w[gpt-4.1-mini gpt-4.1]
+end
+
+# Your orchestrator
+threads = 10.times.map do |i|
+  Thread.new { Rails.application.executor.wrap { SummarizeBatch.run(input(i)) } }
+end
+results = threads.map(&:value)
+```
+
+**Note:** in tests, `stub_step` overrides are thread-local. If your orchestrator spawns threads, propagate overrides manually:
+
+```ruby
+overrides = RubyLLM::Contract.step_adapter_overrides.dup
+Thread.new { RubyLLM::Contract.step_adapter_overrides = overrides; SummarizeBatch.run(input) }
+```
+
+## See also
+
+- [Getting Started](getting_started.md) — the full walkthrough of every feature `SummarizeArticle` uses.
+- [Testing](testing.md) — `stub_step` reference for migrating your test adapter mocks.
+- [Eval-First](eval_first.md) — how to build the "regression" eval from production logs.