ruby_llm-contract 0.8.0 → 0.10.2

data/docs/guide/rails_integration.md

@@ -0,0 +1,218 @@
+# Rails integration
+
+> Read this when you've seen the `SummarizeArticle` example and want to know where contract steps fit in an actual Rails app — directory, initializer, jobs, logging, tests, CI. Skip if you're writing a non-Rails script.
+
+Seven pre-emptive answers to the questions that come up first.
+
+## 1. Where do step classes live?
+
+**Recommended: `app/contracts/`.** The gem's Railtie auto-reloads eval files under `app/contracts/eval/` and `app/steps/eval/` in development, so picking `app/contracts/` aligns with the default reload paths.
+
+```
+app/contracts/summarize_article.rb   # class SummarizeArticle
+```
+
+Any autoloaded directory works (`app/llm_steps/`, `app/services/llm/`, etc.) — Rails 7/8 autoloading resolves them all, and the step class itself does not depend on the path. Pick the default if you have no stronger convention.
+
+Keep evals in the same file as the step (`define_eval` block at the bottom of the class) — one source of truth per contract. If your evals grow too large for the class file, move them to `app/contracts/eval/summarize_article_eval.rb` — the Railtie reloads that directory explicitly in development.
+
+## 2. Initializer configuration
+
+```ruby
+# config/initializers/ruby_llm_contract.rb
+RubyLLM.configure do |c|
+  c.openai_api_key    = ENV.fetch("OPENAI_API_KEY", nil)
+  c.anthropic_api_key = ENV.fetch("ANTHROPIC_API_KEY", nil)
+end
+
+RubyLLM::Contract.configure do |c|
+  c.default_model   = Rails.env.production? ? "gpt-5-mini" : "gpt-5-nano"
+  c.default_adapter = RubyLLM::Contract::Adapters::RubyLLM.new
+end
+```
+
+In specs, override the default adapter to `Adapters::Test` in `spec_helper.rb` (or use `stub_step` per-example — see §5).
+
+Evals defined inside a step class (the recommended pattern) are picked up as soon as Rails autoloads the class — you do not need the eval file in any special directory. If you move evals into separate files under `app/contracts/eval/` or `app/steps/eval/`, the gem's Railtie reloads those two directories explicitly on each request in development; other directories follow standard Rails autoloading rules.
+
+## 3. Background jobs — never call LLMs inline in a controller
+
+LLM calls take 0.8–5 seconds and can fail. Wrap every step invocation in an ActiveJob:
+
+```ruby
+class SummarizeArticleJob < ApplicationJob
+  queue_as :llm
+
+  def perform(article_id)
+    article = Article.find(article_id)
+    result  = SummarizeArticle.run(article.body)
+
+    if result.ok?
+      # parsed_output uses symbol keys in memory. jsonb/json columns round-trip
+      # as strings on reload, so either use deep_stringify_keys before write or
+      # access downstream with string keys — pick one convention and stick to it.
+      article.update!(summary: result.parsed_output.deep_stringify_keys)
+    else
+      article.update!(summary_error: result.validation_errors.join("; "))
+    end
+  end
+end
+```
+
+`SummarizeArticleJob.perform_later(article.id)` returns in milliseconds; the controller stays responsive. If you use Sidekiq, pair `queue_as :llm` with a dedicated concurrency cap in `sidekiq.yml` so long-running LLM calls do not starve other job queues (mailers, webhooks, cleanups).
+
+## 4. Logging and observability
+
+`around_call` runs once per `run()` with the final `Result` (after all retries). Use it to write one row per LLM call:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  # ... prompt, schema, validates ...
+
+  around_call do |step, input, result|
+    AiCallLog.create!(
+      step: step.name,
+      model: result.trace[:model],
+      status: result.status.to_s,
+      latency_ms: result.trace[:latency_ms],
+      input_tokens: result.trace[:usage]&.dig(:input_tokens),
+      output_tokens: result.trace[:usage]&.dig(:output_tokens),
+      cost: result.trace[:cost],
+      validation_errors: result.validation_errors
+    )
+  end
+end
+```
+
+The `AiCallLog` model assumed above is a thin audit record. One possible migration:
+
+```ruby
+# rails g model AiCallLog step:string model:string status:string ...
+create_table :ai_call_logs do |t|
+  t.string  :step, null: false
+  t.string  :model
+  t.string  :status, null: false
+  t.integer :latency_ms
+  t.integer :input_tokens
+  t.integer :output_tokens
+  t.decimal :cost, precision: 10, scale: 6
+  t.jsonb   :validation_errors, default: []
+  t.timestamps
+end
+add_index :ai_call_logs, :step
+add_index :ai_call_logs, :status
+```
+
+For Appsignal / Honeybadger / Datadog, emit an `ActiveSupport::Notifications` event from inside the same `around_call` and subscribe in an initializer:
+
+```ruby
+class SummarizeArticle < RubyLLM::Contract::Step::Base
+  # ... prompt, schema, validates ...
+
+  around_call do |step, _input, result|
+    ActiveSupport::Notifications.instrument(
+      "ruby_llm_contract.run",
+      step: step.name, model: result.trace[:model], status: result.status
+    )
+  end
+end
+
+# config/initializers/observability.rb
+ActiveSupport::Notifications.subscribe("ruby_llm_contract.run") do |*, payload|
+  Appsignal.increment_counter("llm.run.#{payload[:status]}", 1, step: payload[:step])
+end
+```
+
+Trace inspection in an admin UI: `result.trace[:attempts]` gives you per-attempt model, status, cost, latency — render it in a partial to debug production failures without re-running.
+
+## 5. Testing — RSpec and Minitest
+
+Add to `spec/spec_helper.rb` (or `test_helper.rb`):
+
+```ruby
+require "ruby_llm/contract/rspec"    # or ruby_llm/contract/minitest
+```
+
+Then in specs:
+
+```ruby
+RSpec.describe ArticlesController do
+  it "saves the summary when the step passes" do
+    stub_step(SummarizeArticle, response: {
+      tldr: "...", takeaways: %w[a b c], tone: "analytical"
+    })
+
+    post :summarize, params: { id: article.id }
+
+    # NOTE: jsonb/json column round-trips as string keys on reload.
+    expect(article.reload.summary["tldr"]).to eq("...")
+  end
+end
+```
+
+For the step itself, use the `satisfy_contract` and `pass_eval` matchers — details in the [Testing guide](testing.md).
+
+## 6. Error handling in controllers
+
+Never raise on `result.failed?` — that crashes the request. Branch instead:
+
+```ruby
+class ArticlesController < ApplicationController
+  def summarize
+    SummarizeArticleJob.perform_later(params[:id])
+    head :accepted
+  end
+
+  # For synchronous cases (admin tools, small content):
+  def preview
+    result = SummarizeArticle.run(@article.body)
+
+    if result.ok?
+      render json: result.parsed_output
+    else
+      Rails.logger.warn "[llm] #{SummarizeArticle.name} failed: #{result.status}"
+      render json: { error: "Could not summarize; try again shortly." }, status: :service_unavailable
+    end
+  end
+end
+```
+
+When `retry_policy` exhausts and all models fail, `result.failed?` is true but `result.parsed_output` still contains the last attempt's output — useful for logging what the model *did* return before the validate rejected it.
+
+## 7. CI gate — block regressions before merge
+
+Add to your `Rakefile`:
+
+```ruby
+require "ruby_llm/contract/rake_task"
+
+RubyLLM::Contract::RakeTask.new do |t|
+  t.minimum_score      = 0.8
+  t.maximum_cost       = 0.05
+  t.fail_on_regression = true
+  t.save_baseline      = false # read-only in CI; refresh baselines manually
+end
+```
+
+Then wire it in GitHub Actions:
+
+```yaml
+- name: LLM contract evals
+  env:
+    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+  run: bundle exec rake ruby_llm_contract:eval
+```
+
+The job fails when a previously-passing eval case now fails, when the average score drops below the threshold, or when total cost exceeds the cap. That is the signal that blocks a prompt regression or an accidental model upgrade from shipping.
+
+**Two practical notes:**
+
+- **Live evals spend real money on every run** — provider tokens per case × number of cases × every merge. Keep the dataset small and targeted (5–15 high-value cases), use cheap models where quality allows, and rely on offline `sample_response` smoke tests in the bulk of CI runs. Live evals belong on merge-candidate branches and scheduled nightly runs, not on every commit.
+- **Baselines are checkout-managed** — commit them to git under `.eval_baselines/`. Refresh them in a separate manual workflow (or locally + a dedicated PR) rather than from the merge gate, which would dirty the checkout and race with the regression check it is supposed to run.
+
+## See also
+
+- [Getting Started](getting_started.md) — the feature walkthrough the step above is built on
+- [Migration](migration.md) — before/after for replacing a raw `LlmClient.new.call` service with a contract
+- [Eval-First](eval_first.md) — the workflow behind the CI gate above
+- [Testing](testing.md) — `satisfy_contract` and `pass_eval` matcher chains

data/docs/guide/relation_to_agent.md

@@ -0,0 +1,52 @@
+# Relation to `RubyLLM::Agent`
+
+> Read this when you already use `RubyLLM::Agent` (or are about to) and want to understand where `ruby_llm-contract` sits in the same project.
+
+`RubyLLM::Agent` shipped in RubyLLM 1.12. `Step::Base` from this gem and `Agent` target the **same niche**: reusable, class-based prompts. They are **siblings**, not foundation-and-roof.
+
+## Feature mapping
+
+| What you write | Where it lives |
+|---|---|
+| `model`, `temperature`, `schema`, `instructions`, `tools`, `thinking` | covered by both — same idea, different DSL surface |
+| `validate :rule do ... end` business invariants on output | only in `ruby_llm-contract` |
+| `retry_policy escalate(...)` model escalation on validation failure | only here (different from RubyLLM's network-level retry) |
+| `max_cost` / `max_input` / `max_output` pre-flight refusal | only here |
+| `define_eval` + baseline regression + `compare_models` + `optimize_retry_policy` | only here (RubyLLM does not ship an evaluation framework) |
+| Pipeline composition with `step SomeStep, as: :alias` | only here (RubyLLM intentionally leaves workflows as plain Ruby) |
+| `around_call`, named `observe` hooks with pass/fail recorded in trace | only here |
+
+## Runtime relationship
+
+`Step::Base` does **not** use `Agent` internally today. The actual call path is:
+
+```
+Step.run(input)
+  → Runner
+  → Adapters::RubyLLM
+  → RubyLLM.chat(model:, ...)
+  → ... .ask(prompt)
+```
+
+`Agent` is a sibling abstraction calling into `RubyLLM::Chat` through its own `apply_configuration` path. Both end up at `Chat`. They do not share the macro-storage layer.
+
+This may change in a future release if upstream APIs make a layered design natural. The decision is not committed; it depends on adopter signal.
+
+## Coexistence on the same project
+
+The two abstractions can live in the same Rails (or non-Rails) project. Pick one per use case:
+
+- **`RubyLLM::Agent`** when you want a reusable prompt with `model` + `instructions` + `schema` + `tools` and that is enough — no retry-on-validation-failure, no business invariants, no eval framework, no budget gating.
+- **`ruby_llm-contract`'s `Step::Base`** when you need any of: invariants (`validate`), retry with model escalation on validation failure, pre-flight cost ceilings, an evaluation framework with baseline regression, or pipeline composition.
+
+A common pattern: simple ad-hoc prompts as `Agent`, contracts on the LLM features that touch production behaviour or money as `Step`.
+
+## On retry strategies
+
+The retry-strategy framing in this gem favours `retry_policy escalate(model_2, ...)` (model escalation, addresses model bias) over same-model `retry_policy attempts: N` (variance retry).
+
+This is grounded in empirical comparison across PDF quiz generation, GSM8K math (n=30 + n=120), and multi-constraint schedule generation: same-model retry produced no useful lift for nano-class models on tasks with clear correctness criteria. Model escalation did move the needle when same-model retry could not.
+
+`attempts: N` stays in the gem API (backward compat + niche cases like subjective-criteria tasks, multi-step pipelines, weaker open-source models) but is not marketed as a default retry strategy.
+
+See [Optimize retry policy](optimizing_retry_policy.md) for the empirical tooling.

data/docs/guide/relation_to_tribunal.md

@@ -0,0 +1,135 @@
+# Relation to `ruby_llm-tribunal`
+
+> Read this when you've seen [`ruby_llm-tribunal`](https://github.com/Alqemist-labs/ruby_llm-tribunal) and want to know how it relates to `ruby_llm-contract` — and which one (or both) you need.
+
+Both gems sit on top of `ruby_llm`. The space they cover overlaps in vocabulary (both talk about "evals") but they live in different layers and answer different questions. They are not alternatives — they compose.
+
+## The core distinction
+
+| | `ruby_llm-tribunal` | `ruby_llm-contract` |
+|---|---|---|
+| Layer | **Test framework** | **Runtime contract** |
+| When it runs | After the LLM call returns, typically in a spec | Before the LLM result reaches your code |
+| Where the output lives at evaluation time | Already in your variable, returned to caller | Still inside the gem's runner, not yet released |
+| What "fail" means | Red test in CI | Trigger retry/escalate on a stronger model, or fail closed |
+| Strongest features | Rich LLM-as-judge (faithful, relevant, hallucination, refusal, bias, toxicity, jailbreak, PII), red-team adversarial prompts, deterministic helpers (`assert_contains`, `assert_levenshtein`, …), RSpec/Minitest matchers, HTML/JUnit/GH reporters | Schema DSL with constraints, `validate` business rules, `retry_policy escalate(...)` model escalation, `max_cost` pre-flight refusal, regression-eval framework (frozen dataset + baseline + min_score gate), pipeline composition |
+| What it does NOT cover | No retry, no model escalation, no pre-flight cost cap, no contract layer between LLM and your code | No 10-judge LLM-as-judge catalog, no red-team generation, no rich deterministic assertion library, no test-framework matchers |
+
+## Visual: where each gem sits in your call
+
+### Tribunal alone (test-time, in CI)
+
+```
+your code ──► LLM ──► output ──► variable ──► [Tribunal assert_*] ──► ✅ / ❌ red test
+                                                       ▲
+                                              runs in your spec, not in prod
+```
+
+The output **already exists in your code** by the time Tribunal sees it. Tribunal grades it after the fact. A failed grade is a red test — production is unaffected, you fix the prompt or model and re-run.
+
+### Contract alone (runtime, in prod)
+
+```
+your code ──► Step.run ──► LLM ──► [schema + validate]
+                                          │
+                                          ├── valid ────────────► output ──► your code
+                                          │
+                                          └── invalid ──► retry/escalate ──► next model
+                                                                                 │
+                                                              all attempts fail ─┘
+                                                                                 ▼
+                                                                  Result(:validation_error)
+```
+
+The output **never reaches your code** until the contract passes. A failed validation either fixes itself (the retry policy escalates to a stronger model) or fails closed with `Result(:validation_error)` — your call site sees a deterministic failure status, never schema-invalid data.
+
+### Both together (Tribunal in CI → then Contract in prod)
+
+```
+CI (before merge):
+  define_eval(frozen dataset) ──► run Step ──► [Tribunal grades each case]
+                                                       │
+                                                       ▼
+                                                regression gate
+                                           (prevents quality drift over time)
+                                                       │
+                                                       ▼
+                                              ✅ merge allowed
+                                                       │
+                                                       ▼
+PROD (every request):
+  your code ──► Step.run ──► LLM ──► [contract] ──► output ──► your code
+                                         ▲
+                                         │ keeps bad outputs out of prod
+```
+
+Tribunal grades **a fixed set of cases on every PR** to catch quality regressions before merge. Once merged, Contract gates **every production call** to keep bad outputs from reaching users. Each gem owns the layer it is best at — and they run in the order developers experience them.
+
+## When to use which
+
+**Just Contract.** You ship LLM features whose output drives downstream code, money, or user trust. You need the bad-output-doesn't-reach-prod guarantee, retry escalation, and budget refusal. You are happy to write your own `validate` blocks for content checks; you don't need a 10-judge catalog yet.
+
+**Just Tribunal.** You have a stable production path you don't want to wrap, but you want a CI safety net that grades LLM output for faithfulness, hallucination, PII, jailbreak resistance, etc. You're testing a RAG pipeline or chatbot whose runtime is owned by other code.
+
+**Both.** You ship contracts in prod (Contract) AND want stronger CI signal beyond schema regression — judge-quality grading on a frozen dataset, plus adversarial red-team probes. Use Contract's `Step` to make the call, run it in `define_eval` over your dataset, and grade each case with Tribunal helpers in your spec or via the dataset's `evaluator:` proc.
+
+## Integration patterns
+
+These work today without any code changes in either gem — both use plain Ruby blocks/procs as extension points.
+
+### Tribunal helpers inside Contract `validate`
+
+```ruby
+class ChatReply < RubyLLM::Contract::Step::Base
+  prompt "Answer this question grounded in the docs:\n{input}"
+
+  validate("no PII in answer") do |output, _ctx|
+    test_case = RubyLLM::Tribunal::TestCase.new(actual_output: output[:answer])
+    RubyLLM::Tribunal::Assertions.evaluate(:pii, test_case, {}).first == :pass
+  end
+end
+```
+
+A failed Tribunal grade triggers Contract's retry/escalate just like any other validation failure. You get LLM-as-judge **runtime gating**, not just CI testing.
+
+### Tribunal as `evaluator:` in a Contract dataset
+
+```ruby
+ChatReply.define_eval "rag_regression" do
+  add_case "policy",
+    input: "What is the return policy?",
+    evaluator: ->(output, _expected, _input) {
+      tc = RubyLLM::Tribunal::TestCase.new(
+        actual_output: output[:answer],
+        context: ["Returns accepted within 30 days with receipt."]
+      )
+      result = RubyLLM::Tribunal::Assertions.evaluate(:faithful, tc, {})
+      score = result.last[:score] || 0.0
+      RubyLLM::Contract::Eval::EvaluationResult.new(score: score, passed: result.first == :pass)
+    }
+end
+```
+
+Each case is graded by a Tribunal judge; baseline + min_score gate then fails the build on regression. You write the judge once, get the regression gate for free.
+
+### Contract `Step` as Tribunal's `opts[:llm]` injection
+
+Tribunal's built-in judges call `RubyLLM.chat(...).ask(...)` and naively `JSON.parse` the result. If you want **schema-validated, retried, cost-capped judge calls**, inject a Contract `Step` as the LLM caller via `opts[:llm]`. This is an advanced pattern; sketch it from `Tribunal::Assertions::Judge#run_judge`'s injection point and your own judge wrapping a `Step.run`.
+
+This is a recipe, not a shipped adapter. Tribunal's `opts[:llm]` API is at v0.x — recipes survive minor changes; a shipped adapter would not.
+
+## What we are NOT doing
+
+- **No `Contract::ContainsAssertion` or similar 16-helper deterministic library.** Tribunal owns that layer well. Contract's evaluator surface is intentionally minimal (`Exact`, `Regex`, `JsonIncludes`, `ProcEvaluator`, `TraitEvaluator`); for richer deterministic checks, drop a Tribunal helper into your `evaluator:` proc.
+- **No built-in LLM-as-judge catalog.** `Faithful`, `Hallucination`, `Refusal`, etc. are Tribunal's domain. We provide the runtime contract; they provide the grading vocabulary.
+- **No Tribunal as a hard or soft dependency.** Both gems work standalone. Recipes above are documentation, not code in this gem.
+
+## Summary
+
+Three questions, three owners:
+
+- **"Is this output good?"** — Tribunal, in CI, on outputs you already hold.
+- **"What do we do when it isn't?"** — Contract, at runtime, before outputs reach your code (retry/escalate, or fail-closed with `Result(:validation_failed)`).
+- **"What do we do when it _is_ good?"** — your application code. Once Contract returns `:ok`, you persist, render, hand off downstream. The gem deliberately doesn't touch the happy path; it owns failure semantics, not domain logic.
+
+Use Tribunal, Contract, or both — whichever questions your application needs to answer.