ruby_llm-contract 0.10.5 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 233b114efc88c168573b434e9adc17163401e747920a4fc6b587e7a834721723
-  data.tar.gz: fd2d57bd72cd72a7b4a405fda8bfcb0865d737a16a78db11e69d25026c89d1a2
+  metadata.gz: d7d18fbcf79a634c5f577d8965bc82ed10c12a8593b244ad3f07c28f21033035
+  data.tar.gz: '009718225fb17d5b34ea3fe3bf290b812fb17321faa1278d027ae8bdc66e9c2c'
 SHA512:
-  metadata.gz: ff1564d786bc8ab8b391d3dfe0e7e9be406dd4bcaca56fc891fa3ead2a3b8c76cfb6f26c3ad81bc2ffd798e3b9a63a90ab1c4a847e1ad2be1efc1170c6abee83
-  data.tar.gz: 33a4a6aa6798f55d6cfe9c4a1684ee3080e1ea42043d7f53ab54b40c5cc927a9dd607a266a3130206b43822ae7c06739856d29517237432d6cdcfda2f1b08ff3
+  metadata.gz: 2db5376a4aefe02acd3ce2a2debc645d2c2b14d185877da6cbccd9810b04aea8f1a3c830d48d693a66ae8268774ff5041ee81d3c3eba2e33cdc22e5d0ec15d0e
+  data.tar.gz: f560b145abc0b5114240edec2cc01fe563b9d51f48ef90c80a72e9b6b121e3246b7c9ac3196027471bd5450b3b3225f53f566d2d15c50976b8106fda889a458a

data/CHANGELOG.md

@@ -1,5 +1,124 @@
 # Changelog
 
+## 1.0.0 (2026-09-27)
+
+**Requires `ruby_llm ~> 2.0`.** This is the only breaking change: it is a
+dependency requirement, not a change to this gem's API. Your call sites,
+`output_schema` blocks, `trace[:usage]` keys and adapter interface are all
+untouched — if you are on ruby_llm 2.x, upgrading should need no code edits.
+
+1.0 also states what is now stable: the documented DSL, the `Result`/`trace`
+shapes including `trace[:usage]`'s `{ input_tokens:, output_tokens: }` keys, and
+the adapter interface. A future major version of a runtime dependency may again
+require a major version here — that is what this release is.
+
+### Added
+
+- **CI, for the first time.** Matrix over Ruby 3.2 and 3.4 against both
+  ruby_llm 2.0.0 (pinned, so a regression at the major boundary stays
+  distinguishable) and the newest 2.x the gemspec allows. The job also runs every
+  offline example, builds the gem and fails on any `gem build` warning, and
+  installs the built gem to prove it `require`s. The seven live-key specs are
+  bounded explicitly: CI asserts the pending count is exactly 7 and that all
+  seven come from `spec/integration/cost_of_quality_real_spec.rb`, so a failing
+  spec cannot be quietly downgraded to pending.
+- **`.rubocop_todo.yml`** so lint can be enforced in CI at zero offences while the
+  pre-existing backlog stays visible and shrinkable, rather than blocking the gate.
+- **A boundary spec for the RubyLLM 2.x seam**
+  (`spec/ruby_llm/contract/adapters/ruby_llm_2x_boundary_spec.rb`). Every
+  upstream-version-specific detail is supposed to live in `Adapters::RubyLLM` and
+  `CostCalculator`; these tests pin that, including that `max_cost` still fails
+  closed if provider pricing ever becomes unreachable again.
+
+### Changed
+
+- **`ruby_llm` requirement raised to `~> 2.0`** (from `~> 1.12`).
+- **`ruby_llm-schema` replaced by `schematist ~> 1.1`.** `ruby_llm-schema` 1.0.0 is
+  31 lines that emit a deprecation warning and alias `RubyLLM::Schema =
+  Schematist::Schema`; ruby_llm 2.x depends on schematist directly. The
+  `output_schema do ... end` DSL is unchanged — same object behind a different
+  constant.
+- **Adapter reads token counts from `Message#tokens`.** RubyLLM 2.0 replaced
+  `Message#input_tokens`/`#output_tokens` with a `Tokens` value object. This is the
+  only place in the gem that touches upstream token counts, and the normalised
+  `{ input_tokens:, output_tokens: }` hash it publishes is deliberately unchanged.
+- **`max_tokens` now forwards via `with_max_output_tokens`**; `Chat#with_params` no
+  longer exists in 2.0.
+
+### Fixed
+
+- **`estimate_eval_cost` raised `TypeError` instead of flooring at $0.00.** With
+  ruby_llm 2.x, `CostCalculator#calculate` returns nil for two distinct misses -
+  model absent from the registry, and model present but with unreadable pricing -
+  and the summation guarded only the first. Measured against the real 2.0.0
+  registry, 669 of 1672 models take the second path, so any eval estimate touching
+  one of them crashed. `getting_started.md` documents this method as a floor, not
+  a fail-closed, and it behaves that way again.
+- **The optimizer's candidate table could mangle a model name containing
+  brackets.** The short label was built by cutting the rendered label apart, so
+  `"custom (beta) (effort: high)"` came out as `"custom (beta@high)"` - the
+  model's own closing bracket was consumed. The notation now has one owner
+  (`Eval::CandidateLabel`, which both renders and parses), and the short form is
+  composed from parsed parts instead of string surgery.
+- **A reworded skipped-case reason could have reported a phantom regression.**
+  `BaselineDiff` excludes skipped cases from the score denominator by matching a
+  string prefix that `CaseExecutor` writes, and the two sides are filtered
+  differently - the baseline side structurally, the current side only by that
+  string. Both ends now read one constant, and a baseline file written by 0.10.6
+  is covered by its own spec (the fixture was generated by installing 0.10.6, not
+  written by hand).
+- **Cost calculation against ruby_llm 2.x.** 2.0 dropped the flat
+  `input_price_per_million` / `output_price_per_million` readers on `Model` for a
+  nested `pricing -> text_tokens -> standard` walk. Because `CostCalculator`
+  rescued `StandardError` into a nil cost, every priced model silently became
+  "unknown pricing" — and since `max_cost` fails closed on unknown pricing, every
+  budgeted call would have been refused. Pricing extraction is now explicit for
+  both shapes (upstream models and locally `register_model`-ed ones), and an
+  unrecognised shape returns nil rather than a swallowed exception.
+- **A shipped doc example taught the wrong test double.**
+  `docs/guide/multimodal_input.md` showed `double(content:, input_tokens:,
+  output_tokens:)`, which a verified double rejects against ruby_llm 2.0.
+- **`FINDING 7` in the audit specs asserted a constraint on `ruby_llm-schema`**, a
+  dependency this release removes. Generalised to "no runtime dependency is
+  declared without a version constraint", so the check cannot be retired by
+  renaming the gem it happened to be about.
+- Stale version and dependency claims in `README.md`, `docs/guide/output_schema.md`
+  and `docs/guide/multimodal_input.md`.
+
+### Housekeeping (was staged as 0.11.0)
+
+Result of a 20-pass systematic audit of the whole repository. No API changes and no behaviour change inside `Step`/`Pipeline`/`Eval`; the user-visible differences are in what the gem **packages**, what the RubyGems page **links**, and what the docs **claim**.
+
+### Changed
+
+- **The published gem no longer ships development-only files.** Measured by building the gem and unpacking it: `.rubocop.yml`, `Gemfile`, `Gemfile.lock` and `Rakefile` were being packaged, contradicting the gemspec's own stated policy ("dev configs excluded so the published gem contains only what adopters actually need at runtime"). The shipped `Rakefile` was unusable anyway — it `require`s rspec, which is not a runtime dependency, and defines a task over `spec/`, which the same build excludes. Package contents: **137 → 133 files**; `lib/` (105) and `docs/` (16) unchanged.
+- **`Gemfile.lock` is no longer tracked.** It was in the index despite being matched by `.gitignore:10`, and was repacked on every release. A library should not pin its adopters' dependency versions.
+- **The RubyGems page will now show a "Source Code" link.** `metadata["homepage_uri"]` duplicated `spec.homepage` and, being listed first, suppressed `source_code_uri` in the UI — `gem build` warned about this on every build. Dropped the duplicate; `source_code_uri` stays, because `bundle info` and the RubyGems API read it as a distinct field.
+
+### Fixed
+
+- **`.rubocop.yml` declared `AllCops` twice.** Psych applies last-key-wins, so `TargetRubyVersion: 3.2`, `NewCops: enable` and `SuggestExtensions: false` were silently discarded, and the second block's `Exclude` **replaced** RuboCop's defaults instead of adding to them. Effect: the linter scanned 353 files / 2423 offences, 147 of those files untracked under `tmp/` (they carried 2155 of the offences). Merging the blocks alone does not fix it — `inherit_mode: merge` is what restores the defaults. Now 206 files / 300 offences, all pre-existing formatting on real code. (Both counts measured with rubocop 1.88.0; `Gemfile.lock` is no longer tracked, so pin the tool when reproducing.)
+- **`docs/guide/multimodal_input.md` linked to a file that could never resolve** — `../decisions/ADR-0022-*.md` points at `docs/decisions/` (no such directory; the ADR lives in `doc/decisions/`, singular) and that path is gitignored. Since `docs/guide/*` is packaged, this reproduced exactly the "links 404'd locally" defect that 0.10.2 set out to fix. Same dangling reference removed from the 0.9.0 CHANGELOG entry.
+- **Two documented rake-task environment variables never existed.** `REASONING_EFFORT=` and `OLLAMA_API_BASE=` were listed in the 0.5.0 CHANGELOG entry, but `git log --all -S<name> -- lib/` is empty for both — they were never read at any point in the project's history, and ruby_llm's own configuration only seeds `RUBYLLM_DEBUG` / `RUBYLLM_STREAM_DEBUG` from the environment. Setting either was a silent no-op. Reasoning effort is selected with `CANDIDATES=model@effort`; Ollama's base URL goes through `RubyLLM.configure`.
+- **README stated the wrong current version** (0.10.4 while `version.rb` was 0.10.6) — the one sentence that has to change every release.
+- **`examples/README.md` claimed every example carries an "Expected output" section**; four of seven do, for three different reasons, so the sentence was the error rather than the files.
+- **`docs/architecture.md` was reachable from nowhere** — the only one of 16 tracked docs with no inbound link. Added to the README index.
+
+### Removed
+
+- Dead code with a full evidence battery behind each cut: `SchemaValidator::Node#numeric?` (zero readers; the one place needing the check inlines `value.is_a?(Numeric)`), an abandoned twin test helper whose sibling has 14 call sites, three unused `let`/helper definitions, four useless assignments, seven redundant `require`s in specs, and a `SimpleCov.start` that was a measured no-op because `require "simplecov"` already autoloads `.simplecov`.
+- 31 WHAT comment blocks (51 lines) in `lib/` plus 20 comment lines in `spec/`, all restating the line or class below them - including all six `Extracted from ...` comments, five of them the "to reduce class length" boilerplate (refactor history git already records) and ten one-line class docstrings paraphrasing their own class name. WHY comments — invariants, external-behaviour notes, anti-duplication markers — were deliberately kept; so were the three evaluator banners that name a comparison semantic the class name cannot convey.
+- Private-project residue: a spec block still labelled "reddit promo planner shape" although that project was removed as private in #25, and four dangling `(Batch N / TODO)` references to a `TODO.md` deleted in 0.10.1.
+
+## 0.10.6 (2026-06-11)
+
+Docs accuracy patch: correct the positioning of `llm_judge.md` against `ruby_llm-tribunal` after a deeper audit of Tribunal's documented scope. The previous wording ("you are reinventing what `ruby_llm-tribunal` ships as a built-in catalog") read as if Tribunal made `llm_judge.md` redundant — incorrect. Tribunal's README ships an off-the-shelf implementation catalog (`assert_faithful`, `assert_hallucination`, `assert_refusal`, `assert_no_pii`, etc.) but does **not** document calibration workflow, per-claim breakdown, judge-prompt iteration, or judge-as-`evaluator:` integration — exactly the methodology `llm_judge.md` covers. The two are complementary layers, not alternatives. No code behaviour change.
+
+### Fixed
+
+- **`docs/guide/relation_to_tribunal.md`** — added the **"Tribunal's catalog vs Contract's `llm_judge.md` — concrete decision tree"** sub-section under "When to use which", giving adopters a sharp three-way fork: reach for Tribunal's catalog when the check is domain-general (faithfulness vs context, hallucination, refusal, PII, jailbreak, toxicity, bias); build a custom judge per `llm_judge.md` when the criterion is domain-specific, when the judge needs to live inside a `define_eval` regression gate as the `evaluator:` lambda, when per-claim sentence-level debug output is required, or when the judge prompt itself needs to be iterated against your data; use both for the same project at different lifecycle stages (Tribunal at spec-time, calibrated custom judge at CI merge gate). Added the **"What Tribunal documents — and what it doesn't"** sub-section with a five-row comparison table making explicit which methodology gaps `llm_judge.md` covers that Tribunal's README leaves to the adopter (calibration against humans, prompt iteration on over-flag, per-claim breakdown, evaluator-lambda integration, the six anti-patterns).
+- **`docs/guide/llm_judge.md`** — rewrote the closing "When to escalate to Tribunal's catalog" section as **"When to reach for Tribunal instead"**: shorter, accurate (Tribunal is a complementary catalog, not a replacement), points to `relation_to_tribunal.md` for the full decision tree and integration patterns. The previous wording implied that building any of the four standard judges (faithful / hallucination / refusal / PII) was "reinvention" — true for the **implementation** (Tribunal ships them), false for the **methodology** (Tribunal's README doesn't document calibration, anti-patterns, or per-claim breakdown). The methodology applies equally to Tribunal's built-ins, Tribunal's custom registered judges, and Contract `Step::Base` judges.
+
 ## 0.10.5 (2026-06-11)
 
 Docs release: new `llm_judge.md` guide + comprehensive clarity audit across all 16 shipping documentation files. No code behaviour change.
@@ -94,7 +213,7 @@ validate("score in range 0-100") { |o| o[:score].between?(0, 100) }
 
 ### Added
 
-- **Multimodal input via `context: { attachment: ... }`** — pass a file/IO/URL through `Step.run(input, context: { attachment: path })`; the adapter forwards it to `RubyLLM::Chat#ask(content, with: attachment)`. RubyLLM normalises wire format per provider (Anthropic url/base64, OpenAI `image_url`/`file`, Gemini `inline_data`). Multi-attachment supported natively (`with: [pdf1, pdf2]` or `with: { images: [...], pdfs: [...] }`). See [multimodal input guide](docs/guide/multimodal_input.md) and [ADR-0022](doc/decisions/ADR-0022-v09-multimodal-input.md).
+- **Multimodal input via `context: { attachment: ... }`** — pass a file/IO/URL through `Step.run(input, context: { attachment: path })`; the adapter forwards it to `RubyLLM::Chat#ask(content, with: attachment)`. RubyLLM normalises wire format per provider (Anthropic url/base64, OpenAI `image_url`/`file`, Gemini `inline_data`). Multi-attachment supported natively (`with: [pdf1, pdf2]` or `with: { images: [...], pdfs: [...] }`). See [multimodal input guide](docs/guide/multimodal_input.md) (rationale in the internal ADR-0022, not shipped).
 - **`attachment_token_estimate(n)` class macro** — adopter-declared conservative estimate of attachment input tokens. Applied to BOTH runtime (`limit_checker`) and pre-flight (`estimate_cost`) — same source of truth, no estimate/runtime drift.
 - **`on_unknown_attachment_size(:refuse | :warn)` class macro** — mirrors `on_unknown_pricing` opt-out semantics. Defaults to `:refuse`. Never settable as global default — same invariant as `max_cost` fail-closed.
 
@@ -274,7 +393,7 @@ end
 
 ### Documentation
 
-- **Guide: [Production-mode cost measurement](docs/guide/optimizing_retry_policy.md#production-mode-cost-measurement)** — API, metric interpretation, 2-tier scope note.
+- **Guide: [Production-mode cost measurement](docs/guide/optimizing_retry_policy.md#measure-effective-cost-before-shipping)** — API, metric interpretation, 2-tier scope note.
 
 ## 0.6.3 (2026-04-20)
 
@@ -283,7 +402,7 @@ end
 - **`runs:` parameter on `compare_models` and `optimize_retry_policy`** — runs each candidate N times per eval and aggregates the mean score, mean cost per run, and mean latency. Reduces sampling variance in live mode where LLM outputs are non-deterministic (gpt-5 family enforces `temperature=1.0` server-side, so a single unlucky sample can misclassify a viable candidate as "failing"). Default `runs: 1` — backward compatible.
 - **`RUNS=N` on `rake ruby_llm_contract:optimize`** — CLI flag for variance-aware optimization.
 - **`Eval::AggregatedReport`** — duck-type `Report` exposing `score` (mean), `score_min`/`score_max` (spread), `total_cost` (mean per run), `pass_rate` (clean-pass count x/N), and `clean_passes`.
-- **Guide: [Reducing variance with `runs:`](docs/guide/optimizing_retry_policy.md#reducing-variance-with-runs)** — when to use it and why.
+- **Guide: [Reducing variance with `runs:`](docs/guide/optimizing_retry_policy.md)** — when to use it and why.
 
 ## 0.6.2 (2026-04-18)
 
@@ -305,9 +424,9 @@ end
 
 ### Features
 
-- **Multi-provider operator tooling** — rake tasks support `PROVIDER=openai|anthropic|ollama`, `CANDIDATES=model@effort,...`, and `REASONING_EFFORT=low|medium|high`.
+- **Multi-provider operator tooling** — rake tasks support `PROVIDER=openai|anthropic|ollama` and `CANDIDATES=model@effort,...`.
 - **`rake ruby_llm_contract:recommend`** — wraps `Step.recommend` with CLI interface, prints best config, retry chain, DSL, rationale, and savings.
-- **Ollama support** — `PROVIDER=ollama` with configurable `OLLAMA_API_BASE`.
+- **Ollama support** — `PROVIDER=ollama` (base URL via `RubyLLM.configure { |c| c.ollama_api_base = ... }`).
 
 ## 0.6.0 (2026-04-12)
 

data/README.md

@@ -23,7 +23,7 @@ end
 RubyLLM::Contract.configure { }
 ```
 
-Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc). Requires `ruby_llm ~> 1.12` and Ruby ≥ 3.2.
+Works with any `ruby_llm` provider (OpenAI, Anthropic, Gemini, etc). Requires `ruby_llm ~> 2.0` and Ruby ≥ 3.2.
 
 ## Example
 
@@ -132,6 +132,8 @@ Everything below is optional — the example above is a complete step. Reach for
 
 Also supports [multi-step pipelines](docs/guide/pipeline.md) with fail-fast and per-step models.
 
+**Runnable companion repo:** [`ruby_llm-contract_demo`](https://github.com/justi/ruby_llm-contract_demo) — full LLM-as-judge lifecycle (v1 strict → v2 drift → judge calibration → v3/v4 iteration) on a returns-policy chatbot scenario, dual-language (EN default, `DEMO_LANG=pl` opt-in). Live OpenAI calls (`gpt-4.1-mini`, `temperature 0`), `LIVE=1` opt-in, ~$0.30 for the full lifecycle. See [llm_judge.md](docs/guide/llm_judge.md) for the methodology.
+
 ## Relation to `RubyLLM::Agent`
 
 `Step::Base` and `RubyLLM::Agent` (since RubyLLM 1.12) are **siblings** targeting the same niche: reusable, class-based prompts. Both call into `RubyLLM::Chat` directly — Step does not wrap Agent. Step adds the contract layer: `validate` (business invariants), `retry_policy escalate(...)` (model escalation on validation failure), `max_cost` pre-flight refusal, regression-eval framework, pipeline composition. **[Full feature mapping →](docs/guide/relation_to_agent.md)**
@@ -162,10 +164,18 @@ Different layers, complementary. [`ruby_llm-tribunal`](https://github.com/Alqemi
 | [Multimodal input (PDF / image / audio)](docs/guide/multimodal_input.md) | Route attachments through the contract; `attachment_token_estimate`, fail-closed cost, calibration table |
 | [Schema DSL reference](docs/guide/output_schema.md) | Every constraint, nested objects, pattern table |
 | [Prompt DSL reference](docs/guide/prompt_ast.md) | `system` / `rule` / `section` / `example` / `user` nodes |
+| [Architecture overview](docs/architecture.md) | Class map: Pipeline, Step, Contract, Prompt, Eval |
 
 ## Status & versioning
 
-Pre-1.0 (currently **0.10.4**). Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 0.10.4` until 1.0 ships.
+Stable at **1.0.0**. Semver tracked; breaking changes flagged in [CHANGELOG](CHANGELOG.md). Pin `~> 1.0`.
+
+What 1.0 promises not to break inside the 1.x line: the documented DSL
+(`prompt`, `output_schema`, `validate`, `retry_policy`, `max_cost`, `define_eval`,
+pipelines), the `Result` and `trace` shapes including `trace[:usage]`'s
+`{ input_tokens:, output_tokens: }` keys, and the adapter interface. A new major
+version of a runtime dependency may require a new major version here — that is
+what 1.0.0 itself was.
 
 ## FAQ
 

data/docs/guide/llm_judge.md

@@ -120,7 +120,7 @@ report = AccuracyJudge.run_eval("calibration")
 report.score   # => 0.92  → judge agrees with humans 92% of the time
 ```
 
-A reasonable bar is `score >= 0.85` — an empirical baseline from LLM-eval field practice ([Eugene Yan on evals](https://eugeneyan.com/writing/evals/), [Hamel Husain field guide](https://hamel.dev/blog/posts/field-guide/)). Below that, judge-human agreement drops into a band where the judge's confident percentages stop matching what a human reviewer would say, and gate decisions become unreliable. If your domain is high-stakes (medical, legal, financial), raise the bar to 0.90+. Iterate the judge's prompt, or escalate to a stronger model, until the score on real production data crosses your bar.
+A reasonable bar is `score >= 0.85` — a practical threshold synthesized from field practice. [Hamel Husain](https://hamel.dev/blog/posts/field-guide/) reports it took **three iterations** of the judge prompt to reach ">90% agreement" with humans in a Honeycomb case study; [Eugene Yan](https://eugeneyan.com/writing/evals/) notes RAG-grounded systems still hold a 5-10% factual inconsistency rate even after good prompt engineering. The specific 0.85 number is the author's summary — neither source quotes it directly. Below ~0.85, judge-human agreement drops into a band where the judge's confident percentages stop matching what a human reviewer would say, and gate decisions become unreliable. If your domain is high-stakes (medical, legal, financial), raise the bar to 0.90+. Iterate the judge's prompt, or escalate to a stronger model, until the score on real production data crosses your bar.
 
 **What "iterate the judge's prompt" actually looks like.** The first judge prompt almost always over-flags. Two or three iterations are normal before the judge is ready to gate anything. The common pattern:
 
@@ -195,9 +195,9 @@ What to do operationally:
 
 The drop is your signal to refine the judge's prompt, not to lower the gate.
 
-## When to escalate to Tribunal's catalog
+## When to reach for Tribunal instead
 
-If you find yourself building three or four judges that all rhyme — "faithful?", "hallucination?", "refusal?", "PII leakage?" — you are reinventing what [`ruby_llm-tribunal`](https://github.com/Alqemist-labs/ruby_llm-tribunal) ships as a built-in catalog. See [Relation to Tribunal](relation_to_tribunal.md) for the integration recipe: Contract Steps make the LLM calls, Tribunal supplies the grading vocabulary, and the two compose in a single `define_eval`.
+[`ruby_llm-tribunal`](https://github.com/Alqemist-labs/ruby_llm-tribunal) ships an off-the-shelf catalog of common LLM-as-judge assertions (`assert_faithful`, `assert_hallucination`, `assert_refusal`, `assert_no_pii`, etc.) — a shortcut when your check matches one of those domain-general categories. The methodology in this guide still applies: calibrate the judge against your human-labeled production data **before** trusting Tribunal's `default_threshold = 0.8`, refine the prompt when it over-flags, watch for the anti-patterns above. See [Relation to Tribunal](relation_to_tribunal.md) for the full positioning — what each gem documents (and doesn't), a concrete decision tree on catalog-vs-custom-judge, and three working integration patterns.
 
 ## See also
 

data/docs/guide/multimodal_input.md

@@ -2,7 +2,7 @@
 
 > Read this when your contract needs to send a PDF, image, or audio file to the LLM — not just text.
 
-`ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy escalate(...)`, and trace observability still apply. The gem does **not** ship its own multimodal API — it forwards `with:` to `RubyLLM::Chat#ask`, which RubyLLM 1.15+ normalises per provider (Anthropic, OpenAI, Gemini).
+`ruby_llm-contract` 0.10.0+ routes attachments through the contract layer, so `max_cost`, `validate`, `retry_policy escalate(...)`, and trace observability still apply. The gem does **not** ship its own multimodal API — it forwards `with:` to `RubyLLM::Chat#ask`, which RubyLLM normalises per provider (Anthropic, OpenAI, Gemini).
 
 ## Minimal example
 
@@ -130,7 +130,7 @@ Pick a value at or above the provider's worst-case. The estimate is a **floor fo
 
 ## Multi-turn caveat
 
-If your contract uses history (`add_history`), attachments from prior turns are **not** replayed in 0.10.x. Single-turn multimodal works; follow-up questions on the same document require additional work that is deferred to a later release. See [ADR-0022](../decisions/ADR-0022-v09-multimodal-input.md) (internal) for the rationale.
+If your contract uses history (`add_history`), attachments from prior turns are **not** replayed (still true in 1.0.x). Single-turn multimodal works; follow-up questions on the same document require additional work that is deferred to a later release. The rationale is recorded in the project's internal ADR-0022, which is not shipped with the gem.
 
 ## Provider notes
 
@@ -149,7 +149,8 @@ RSpec.describe ExtractInvoiceData do
   it "forwards attachment to chat.ask" do
     expect_any_instance_of(RubyLLM::Chat).to receive(:ask)
       .with(anything, with: "fixtures/invoice.pdf")
-      .and_return(double(content: '{"vendor":"X",...}', input_tokens: 200, output_tokens: 50))
+      .and_return(double(content: '{"vendor":"X",...}',
+                         tokens: RubyLLM::Tokens.new(input: 200, output: 50)))
 
     result = described_class.run("extract", context: { attachment: "fixtures/invoice.pdf" })
     expect(result.status).to eq(:ok)

data/docs/guide/output_schema.md

@@ -2,12 +2,12 @@
 
 > Read this as a reference for the schema DSL — every constraint, nested objects, arrays of objects, the full pattern table.
 
-Declare the expected output structure using [ruby_llm-schema](https://github.com/danielfriis/ruby_llm-schema) DSL. The schema serves **two purposes**:
+Declare the expected output structure using the [schematist](https://github.com/crmne/schematist) DSL. The schema serves **two purposes**:
 
 1. **Output validation** — replaces type and shape checks (enums, ranges, required fields). One declaration instead of many.
 2. **Provider-side request** — with the RubyLLM adapter, the schema is sent to the LLM provider via `chat.with_schema(...)`, asking the model to return JSON matching the shape. Cheaper models sometimes ignore the request, which is why client-side validation (point 1) still matters.
 
-> **Same DSL `RubyLLM::Agent.schema` accepts.** Both use `RubyLLM::Schema.create(&block)` under the hood. `Step.output_schema` is eager-compiled at class load and drives client-side validation; `Agent.schema` accepts a `Proc` for dynamic per-call schemas. See [Relation to Agent](relation_to_agent.md) for the full comparison.
+> **Same DSL `RubyLLM::Agent.schema` accepts.** Both use `Schematist::Schema.create(&block)` under the hood (schematist is what the `ruby_llm-schema` gem became; RubyLLM 2.x depends on it directly). `Step.output_schema` is eager-compiled at class load and drives client-side validation; `Agent.schema` accepts a `Proc` for dynamic per-call schemas. See [Relation to Agent](relation_to_agent.md) for the full comparison.
 
 All examples below extend the `SummarizeArticle` step from the [README](../../README.md).
 

data/docs/guide/relation_to_tribunal.md

@@ -73,6 +73,34 @@ Tribunal grades **a fixed set of cases on every PR** to catch quality regression
 
 **Both.** You ship contracts in prod (Contract) AND want stronger CI signal beyond schema regression — judge-quality grading on a frozen dataset, plus adversarial red-team probes. Use Contract's `Step` to make the call, run it in `define_eval` over your dataset, and grade each case with Tribunal helpers in your spec or via the dataset's `evaluator:` proc.
 
+### Tribunal's catalog vs Contract's `llm_judge.md` — concrete decision tree
+
+If you specifically need an **LLM-as-judge** (a second LLM grading the first one's output), the decision is:
+
+- **Reach for Tribunal's catalog** when your check is one of the well-defined, domain-general categories Tribunal ships: *"is this faithful to the retrieved context?"*, *"is this a refusal?"*, *"does this contain PII?"*, *"is this jailbreak-resistant?"*, *"hallucinated?"*, *"toxic?"*, *"biased?"*. One line in a spec, default threshold, no judge code to write or maintain. The judge prompt is baked into the gem.
+- **Build a custom judge per [`llm_judge.md`](llm_judge.md)** when:
+  - **Your criterion is domain-specific** — *"does this medical advice match our internal safety policy?"*, *"is this reply in our brand voice?"*, *"does this summary preserve the legal disclaimer verbatim?"*. No off-the-shelf judge knows your policy; you write the prompt.
+  - **You need the verdict inside a `define_eval` regression gate** (the `evaluator:` lambda pattern) — Tribunal's surface is spec-time assertions, not eval-framework evaluators.
+  - **You need a per-claim breakdown** (sentence-level *"this claim → unsupported, that claim → contradicted"* output) for PR debugging — Tribunal returns one score per assertion.
+  - **You need to iterate the judge prompt** because it over-flags on your data — Tribunal's prompts are fixed per assertion.
+- **Use both** for the same project even when your check is in Tribunal's catalog: Tribunal's `assert_faithful` for spec-time grade on individual responses, plus a calibrated custom judge wired as `evaluator:` in a regression `define_eval` over a frozen dataset for CI merge-gating. They cover different lifecycle stages.
+
+Either way, the **methodology** in [`llm_judge.md`](llm_judge.md) — calibrate the judge against human-labeled production samples before trusting any score, watch for the six anti-patterns, refine the prompt when it over-flags — applies equally to Tribunal's built-ins, Tribunal's custom registered judges, and Contract `Step::Base` judges. Tribunal's `default_threshold = 0.8` is a starting point, not a calibrated bar for your data.
+
+## What Tribunal documents — and what it doesn't
+
+Tribunal's README ships an **implementation catalog** (`assert_faithful`, `assert_hallucination`, `assert_refusal`, `assert_no_pii`, `assert_no_toxicity`, `assert_no_bias`, `assert_jailbreak_resistant`, etc., plus a `register_judge` API for custom ones). What it currently leaves to the adopter:
+
+| Tribunal ships | Tribunal's README doesn't document (Contract's [`llm_judge.md`](llm_judge.md) does) |
+|---|---|
+| `default_threshold = 0.8` (fixed) | How to **calibrate** the threshold against your human-labeled production data |
+| Judge prompt baked in per assertion | How to **iterate the judge prompt** when it over-flags stylistic courtesy as drift |
+| Single score per assertion | **Per-claim breakdown** schema for sentence-level PR debugging |
+| `assert_faithful` in a spec | **Judge as `evaluator:` lambda** in a Contract `define_eval` regression gate |
+| Custom Judge mechanism (`register_judge`) | **Anti-patterns** (stubbing the verdict, calibrating on synthetic data, calibrating once and shipping) |
+
+This is a **complementary gap**, not a competition. Tribunal owns the implementation catalog; Contract's `llm_judge.md` owns the methodology. A typical production setup uses both layers: pick (or build) the implementation, then calibrate it against your humans **before** trusting any score.
+
 ## Integration patterns
 
 These work today without any code changes in either gem — both use plain Ruby blocks/procs as extension points.

data/examples/README.md

@@ -14,7 +14,7 @@ Pedagogical order: hook → activation → evolution → composition → quality
 | 05 | `05_eval_dataset.rb` | **"How do I stop silent prompt regressions?"** — define_eval with real cases, baseline vs regressed adapter, regression detection signal, inline eval_case. |
 | 06 | `06_retry_variants.rb` | **"What retry shapes exist beyond cross-model?"** — `attempts: 3` (variance absorption), `reasoning_effort` escalation (low→medium→high), cross-provider fallback (Ollama → Anthropic → OpenAI). |
 
-Every example has an "Expected output" section in the file header — you can read what each one prints without running it.
+Four of the seven carry an "Expected output" section in the file header (01, 03, 05, 06), so you can read what they print without running them. The other three differ: `00_basics.rb` prints nothing and annotates each result inline with `# =>`; `02_real_llm_minimal.rb` needs a live provider key, so its output is not deterministic; `04_summarize_and_translate.rb` prints but has no header block.
 
 ## Running
 

data/lib/ruby_llm/contract/adapters/ruby_llm.rb

@@ -25,7 +25,6 @@ module RubyLLM
           build_response(response)
         end
 
-        # Maps option keys to the RubyLLM chat method and argument form.
         CHAT_OPTION_METHODS = {
           temperature: :with_temperature,
           schema: :with_schema
@@ -70,12 +69,10 @@ module RubyLLM
           thinking_config = resolve_thinking_config(options)
           chat.with_thinking(**thinking_config) if thinking_config
 
-          # `with_params` carries only raw passthroughs (currently `max_tokens`).
-          # `reasoning_effort` is no longer forwarded here — it goes through
-          # `with_thinking` above, which is the canonical RubyLLM API.
-          params = {}
-          params[:max_tokens] = options[:max_tokens] if options[:max_tokens]
-          chat.with_params(**params) if params.any?
+          # RubyLLM 2.0 removed `with_params`; `with_max_output_tokens` is the
+          # replacement for this one passthrough. `reasoning_effort` is not
+          # forwarded here — it goes through `with_thinking` above.
+          chat.with_max_output_tokens(options[:max_tokens]) if options[:max_tokens]
         end
 
         # Returns merged `{ effort:, budget: }` or nil. `options[:reasoning_effort]`
@@ -91,11 +88,18 @@ module RubyLLM
           content = response.content
           content = content.to_s unless content.is_a?(Hash) || content.is_a?(Array)
 
+          # This is the ONLY place upstream token counts are read. RubyLLM 2.0
+          # replaced `Message#input_tokens`/`#output_tokens` with a `Tokens` value
+          # object. The `{ input_tokens:, output_tokens: }` shape below is this
+          # gem's own public contract (Step::Trace#usage, cost calculation,
+          # documented in the README) and deliberately does NOT change.
+          tokens = response.tokens
+
           Response.new(
             content: content,
             usage: {
-              input_tokens: response.input_tokens || 0,
-              output_tokens: response.output_tokens || 0
+              input_tokens: tokens&.input || 0,
+              output_tokens: tokens&.output || 0
             }
           )
         end

data/lib/ruby_llm/contract/concerns/context_helpers.rb

@@ -3,8 +3,6 @@
 module RubyLLM
   module Contract
     module Concerns
-      # Shared helpers for context hash manipulation.
-      # Used by EvalHost, Runner, Step::Base.
       module ContextHelpers
         private
 

data/lib/ruby_llm/contract/concerns/deep_symbolize.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     module Concerns
-      # Recursively converts Hash keys to symbols while preserving array shape.
       module DeepSymbolize
         def deep_symbolize(object)
           case object

data/lib/ruby_llm/contract/contract/parser.rb

@@ -38,7 +38,6 @@ module RubyLLM
       end
       private_class_method :parse_json_text
 
-      # Fallback: attempt to extract the first JSON object or array from prose
       def self.parse_json_with_extraction(text, raw_output)
         extracted = extract_json(text)
         unless extracted
@@ -72,8 +71,6 @@ module RubyLLM
         match ? match[1] : text
       end
 
-      # Extract the first JSON object or array from text that may contain prose.
-      # Uses bracket-matching to find the outermost balanced { } or [ ] block.
       JSON_START_PATTERN = /[{\[]/
 
       def self.extract_json(text)

data/lib/ruby_llm/contract/contract/schema_validator/bound_rule.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     class SchemaValidator
-      # Validates numeric and collection size bounds for a node.
       class BoundRule
         NUMERIC_LIMITS = [
           { key: :minimum, label: "minimum", relation: "below", invalid: ->(actual, limit) { actual < limit } },

data/lib/ruby_llm/contract/contract/schema_validator/enum_rule.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     class SchemaValidator
-      # Validates enum membership for a node when enum values are declared.
       class EnumRule
         def initialize(errors)
           @errors = errors

data/lib/ruby_llm/contract/contract/schema_validator/node.rb

@@ -21,10 +21,6 @@ module RubyLLM
           value.is_a?(Array)
         end
 
-        def numeric?
-          value.is_a?(Numeric)
-        end
-
         def properties
           schema[:properties] || {}
         end

data/lib/ruby_llm/contract/contract/schema_validator/scalar_rules.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     class SchemaValidator
-      # Applies scalar-only validation rules to a schema node.
       class ScalarRules
         def initialize(errors)
           @rules = [

data/lib/ruby_llm/contract/contract/schema_validator/type_rule.rb

@@ -3,7 +3,6 @@
 module RubyLLM
   module Contract
     class SchemaValidator
-      # Validates the declared JSON schema type for a node.
       class TypeRule
         def initialize(errors)
           @errors = errors

data/lib/ruby_llm/contract/cost_calculator.rb

@@ -84,11 +84,36 @@ module RubyLLM
       end
 
       def self.compute_cost(model_info, usage)
-        input_cost = token_cost(usage[:input_tokens], model_info.input_price_per_million)
-        output_cost = token_cost(usage[:output_tokens], model_info.output_price_per_million)
+        prices = prices_for(model_info)
+        return nil unless prices
+
+        input_cost = token_cost(usage[:input_tokens], prices[:input])
+        output_cost = token_cost(usage[:output_tokens], prices[:output])
         (input_cost + output_cost).round(6)
       end
 
+      # Two shapes reach here. `RegisteredModel` (our own struct, from
+      # `register_model`) exposes flat `*_price_per_million` readers. RubyLLM 2.0
+      # moved provider pricing into a nested value object and dropped those flat
+      # readers, so it has to be walked: pricing -> text_tokens -> standard.
+      #
+      # nil means "unknown pricing", which `max_cost` fails closed on. The guards
+      # keep the two known shapes explicit, but `calculate` rescues StandardError,
+      # so a future shape this walk cannot read still degrades to nil rather than
+      # surfacing - every budgeted call would be refused with no error.
+      def self.prices_for(model_info)
+        if model_info.respond_to?(:input_price_per_million)
+          return { input: model_info.input_price_per_million,
+                   output: model_info.output_price_per_million }
+        end
+
+        tier = model_info.pricing&.text_tokens&.standard if model_info.respond_to?(:pricing)
+        return unless tier.respond_to?(:input_per_million)
+
+        { input: tier.input_per_million, output: tier.output_per_million }
+      end
+      private_class_method :prices_for
+
       # Provider pricing is denominated per 1M tokens; divide here to get
       # the dollar cost for the actual usage count. Named constant for
       # consistency with how RubyLLM and provider docs express prices.

data/lib/ruby_llm/contract/eval/baseline_diff.rb

@@ -79,8 +79,10 @@ module RubyLLM
         private
 
         def compute_score(cases)
-          # Exclude skipped cases from score (consistent with Report#score)
-          evaluated = cases.reject { |c| c[:details]&.start_with?("skipped:") }
+          # Asymmetric on purpose: the baseline side arrives already filtered by
+          # ReportStats#evaluated_results, the current side does not, so this
+          # string check is the only skip filter on one of the two sides.
+          evaluated = cases.reject { |c| c[:details]&.start_with?(CaseResult::SKIPPED_DETAILS_PREFIX) }
           return 0.0 if evaluated.empty?
 
           evaluated.sum { |c| c[:score] } / evaluated.length

data/lib/ruby_llm/contract/eval/candidate_label.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Eval
+      # The `model (effort: x)` notation, rendered into optimizer tables and parsed
+      # back to rebuild candidate configs. Render and parse live together because a
+      # format change on one side silently mis-parses on the other: the pattern is
+      # built from the same literal the renderer emits.
+      module CandidateLabel
+        OPEN = " (effort: "
+        CLOSE = ")"
+        PATTERN = /\s*#{Regexp.escape(OPEN.lstrip)}(\w+)#{Regexp.escape(CLOSE)}/
+
+        def self.render(config)
+          effort = config[:reasoning_effort]
+          return config[:model] unless effort
+
+          "#{config[:model]}#{OPEN}#{effort}#{CLOSE}"
+        end
+
+        def self.parse(label)
+          match = PATTERN.match(label)
+          return { model: label } unless match
+
+          { model: match.pre_match.strip, reasoning_effort: match[1] }
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/eval/case_executor.rb

@@ -30,7 +30,7 @@ module RubyLLM
         private
 
         def missing_adapter?(error)
-          error.message.include?("No adapter configured")
+          error.message.include?(Step::Base::NO_ADAPTER_MESSAGE)
         end
 
         def skipped_result(test_case, reason)
@@ -43,7 +43,7 @@ module RubyLLM
             score: 0.0,
             passed: false,
             label: "SKIP",
-            details: "skipped: #{reason}"
+            details: CaseResult.skipped_details(reason)
           )
         end
       end

data/lib/ruby_llm/contract/eval/case_result.rb

@@ -4,6 +4,14 @@ module RubyLLM
   module Contract
     module Eval
       class CaseResult
+        # BaselineDiff#compute_score keys off this prefix to keep skipped cases
+        # out of the score denominator. Both ends must read it from here.
+        SKIPPED_DETAILS_PREFIX = "skipped:"
+
+        def self.skipped_details(reason)
+          "#{SKIPPED_DETAILS_PREFIX} #{reason}"
+        end
+
         attr_reader :name, :input, :output, :expected, :step_status,
                     :score, :details, :duration_ms, :cost, :attempts
 

data/lib/ruby_llm/contract/eval/contract_detail_builder.rb

@@ -3,8 +3,6 @@
 module RubyLLM
   module Contract
     module Eval
-      # Extracted from Runner to reduce class length.
-      # Builds contract detail strings for contract-only evaluation.
       module ContractDetailBuilder
         private
 

data/lib/ruby_llm/contract/eval/model_comparison.rb

@@ -6,9 +6,10 @@ module RubyLLM
       class ModelComparison
         attr_reader :eval_name, :reports, :configs, :fallback
 
+        # Kept as the name six call sites already use; the format itself lives in
+        # CandidateLabel, which owns both directions.
         def self.candidate_label(config)
-          effort = config[:reasoning_effort]
-          effort ? "#{config[:model]} (effort: #{effort})" : config[:model]
+          CandidateLabel.render(config)
         end
 
         def initialize(eval_name:, reports:, configs: nil, fallback: nil)