ruby_llm-contract 0.4.2 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e7f3a23f6ba5f67352d32b67ba0737c60d0112dc1db866c9988bbd5d894dcb4f
-  data.tar.gz: 7b94ae8959fd87e695063b4e94a3e7217d8bb9b93a860c574f3f1d626bb4a94d
+  metadata.gz: 2e6861a314beadad8064d5fe08b85dc0f94032987ba41c27fc9a640788f10c28
+  data.tar.gz: 7efb88142ef8ac8287ed58f6cc9bc93ca78130084e14f8310981e7f90faf9943
 SHA512:
-  metadata.gz: 4abcaf1069f7e97072375027cadbcbcb2b4970a48a5d0a21cb25bc992f53cba6a3c8c6061cc24749f3162600a99017213a58e91f30b52a690deae4102ec153c9
-  data.tar.gz: 6ba8118f0ba2b14b43a5908afc5118f96a585026257a3fef97d07a232ac35442b88d6210e2a0c155396dde487f85d9ebb50f65ab104e645abd6ea38128300877
+  metadata.gz: 1bd259ab22d13b2e7cc9848c401b91ebfaa135dc7502b72074fd28e71e6120f7ea183c09361948bf8d68df9cd5190e9450a3f03954bbd2927f8091c987368bb7
+  data.tar.gz: 904185a06d1def6513268033cbe87b30e3af9b92e0a26ec3cf89c93ad88450c4d49aa13d79c2428b1f7f6cc1cbffefeccf146b5a5c7ddf0057ab3db2f1b7dc8d

data/.rubycritic.yml

@@ -0,0 +1,8 @@
+paths:
+  - lib
+
+formats:
+  - console
+
+minimum_score: 80
+no_browser: true

data/.simplecov

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require "simplecov"
+
+SimpleCov.start do
+  enable_coverage :branch
+  primary_coverage :branch
+
+  add_filter "/spec/"
+  add_filter "/examples/"
+  add_filter "/internal/"
+  add_filter "/tmp/"
+
+  track_files "lib/**/*.rb"
+
+  if ENV["CI"] == "true" || ENV["SIMPLECOV_STRICT"] == "1"
+    minimum_coverage line: 89
+    minimum_coverage branch: 75
+  end
+
+  command_name "RSpec"
+end

data/CHANGELOG.md

@@ -1,5 +1,64 @@
 # Changelog
 
+## 0.5.0 (2026-03-25)
+
+Data-Driven Prompt Engineering — see ADR-0015.
+
+### Features
+
+- **`observe` DSL** — soft observations that log but never fail. `observe("scores differ") { |o| o[:a] != o[:b] }`. Results in `result.observations`. Logged via `Contract.logger` when they fail. Runs only when validation passes.
+- **`compare_with`** — prompt A/B testing. `StepV2.compare_with(StepV1, eval: "regression", model: "nano")` returns `PromptDiff` with `improvements`, `regressions`, `score_delta`, `safe_to_switch?`. Reuses `BaselineDiff` internally.
+- **RSpec `compared_with` chain** — `expect(StepV2).to pass_eval("x").compared_with(StepV1).without_regressions` blocks merge if new prompt regresses any case.
+
+### Game changer continuity
+
+```
+v0.2: "Which model?"          → compare_models (snapshot)
+v0.3: "Did it change?"        → baseline regression (binary)
+v0.4: "Show me the trend"     → eval history (time series)
+v0.5: "Which prompt is better?" → compare_with (A/B testing)
+```
+
+## 0.4.5 (2026-03-24)
+
+Audit hardening — 18 bugs fixed across 4 audit rounds.
+
+### Fixes
+
+- **RakeTask history before abort** — `track_history` now saves all reports (pass and fail) before gating, so failed runs appear in eval history.
+- **RSpec/Minitest stub scoping** — block form `stub_step` uses thread-local overrides with real cleanup. Non-block `stub_all_steps` auto-restored by RSpec `around(:each)` hook and Minitest `setup`/`teardown`.
+- **StepAdapterOverride** — handles `context: nil` and respects string key `"adapter"`. Moved to `contract.rb` so both test frameworks share one mechanism.
+- **max_cost fail closed output estimate** — preflight uses 1x input tokens as output estimate when `max_output` not set, preventing cost bypass for output-expensive models.
+- **reset_configuration! clears overrides** — `step_adapter_overrides` now cleared on reset.
+- **CostCalculator.register_model** — validates `Numeric`, `finite?`, non-negative. Rejects NaN, Infinity, strings, nil.
+- **Pipeline token_budget** — rejects negative and zero values (parity with `timeout_ms`).
+- **track_history model fallback** — uses step DSL `model`, then `default_model` when context has no model. Handles string key `"model"`.
+- **estimate_cost / estimate_eval_cost** — falls back to step DSL model when no explicit model arg given.
+- **stub_steps string keys** — both RSpec and Minitest normalize string-keyed options with `transform_keys(:to_sym)`.
+- **DSL `:default` reset** — `model(:default)`, `temperature(:default)`, `max_cost(:default)` reset inherited parent values.
+
+## 0.4.4 (2026-03-24)
+
+- **`stub_steps` (plural)** — stub multiple steps with different responses in one block. No nesting needed. Works in RSpec and Minitest:
+  ```ruby
+  stub_steps(
+    ClassifyTicket => { response: { priority: "high" } },
+    RouteToTeam => { response: { team: "billing" } }
+  ) { TicketPipeline.run("test") }
+  ```
+
+## 0.4.3 (2026-03-24)
+
+Production feedback release — driven by ADR-0016 (real Rails 8.1 deployment).
+
+### Features
+
+- **`stub_step` block form** — `stub_step(Step, response: x) { test }` auto-resets adapter after block. Works in RSpec and Minitest. Eliminates leaked test state.
+- **Minitest per-step routing** — `stub_step(StepA, ...)` now actually routes to StepA only (was setting global adapter, ignoring step class).
+- **`track_history` in RakeTask** — `t.track_history = true` auto-appends every eval run (pass and fail) to `.eval_history/`. Drift detection without manual `save_history!` calls.
+- **`max_cost` fail closed** — unknown model pricing now refuses the call instead of silently skipping. Set `on_unknown_pricing: :warn` for old behavior.
+- **`CostCalculator.register_model`** — register pricing for custom/fine-tuned models: `register_model("ft:gpt-4o", input_per_1m: 3.0, output_per_1m: 6.0)`.
+
 ## 0.4.2 (2026-03-24)
 
 - **RakeTask lazy context** — `t.context` now accepts a Proc, resolved at task runtime (after `:environment`). Fixes adapter not being available at Rake load time in Rails apps.

data/Gemfile

@@ -8,4 +8,6 @@ group :development, :test do
   gem "rake", "~> 13.0"
   gem "rspec", "~> 3.13"
   gem "rubocop", "~> 1.75"
+  gem "rubycritic", "~> 4.9"
+  gem "simplecov", "~> 0.22"
 end

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ruby_llm-contract (0.4.2)
+    ruby_llm-contract (0.5.0)
       dry-types (~> 1.7)
       ruby_llm (~> 1.0)
       ruby_llm-schema (~> 0.3)
@@ -12,20 +12,43 @@ GEM
     addressable (2.8.9)
       public_suffix (>= 2.0.2, < 8.0)
     ast (2.4.3)
+    axiom-types (0.1.1)
+      descendants_tracker (~> 0.0.4)
+      ice_nine (~> 0.11.0)
+      thread_safe (~> 0.3, >= 0.3.1)
     base64 (0.3.0)
     bigdecimal (4.0.1)
+    childprocess (5.1.0)
+      logger (~> 1.5)
+    coercible (1.0.0)
+      descendants_tracker (~> 0.0.1)
     concurrent-ruby (1.3.6)
+    descendants_tracker (0.0.4)
+      thread_safe (~> 0.3, >= 0.3.1)
     diff-lcs (1.6.2)
+    docile (1.4.1)
+    dry-configurable (1.3.0)
+      dry-core (~> 1.1)
+      zeitwerk (~> 2.6)
     dry-core (1.2.0)
       concurrent-ruby (~> 1.0)
       logger
       zeitwerk (~> 2.6)
     dry-inflector (1.3.1)
+    dry-initializer (3.2.0)
     dry-logic (1.6.0)
       bigdecimal
       concurrent-ruby (~> 1.0)
       dry-core (~> 1.1)
       zeitwerk (~> 2.6)
+    dry-schema (1.16.0)
+      concurrent-ruby (~> 1.0)
+      dry-configurable (~> 1.0, >= 1.0.1)
+      dry-core (~> 1.1)
+      dry-initializer (~> 3.2)
+      dry-logic (~> 1.6)
+      dry-types (~> 1.9, >= 1.9.1)
+      zeitwerk (~> 2.6)
     dry-types (1.9.1)
       bigdecimal (>= 3.0)
       concurrent-ruby (~> 1.0)
@@ -33,6 +56,7 @@ GEM
       dry-inflector (~> 1.0)
       dry-logic (~> 1.4)
       zeitwerk (~> 2.6)
+    erubi (1.13.1)
     event_stream_parser (1.0.0)
     faraday (2.14.1)
       faraday-net_http (>= 2.0, < 3.5)
@@ -44,11 +68,25 @@ GEM
       net-http (~> 0.5)
     faraday-retry (2.4.0)
       faraday (~> 2.0)
+    flay (2.14.3)
+      erubi (~> 1.10)
+      path_expander (~> 2.0)
+      prism (~> 1.7)
+      sexp_processor (~> 4.0)
+    flog (4.9.4)
+      path_expander (~> 2.0)
+      prism (~> 1.7)
+      sexp_processor (~> 4.8)
+    ice_nine (0.11.2)
     json (2.19.2)
     json-schema (6.2.0)
       addressable (~> 2.8)
       bigdecimal (>= 3.1, < 5)
     language_server-protocol (3.17.0.5)
+    launchy (3.1.1)
+      addressable (~> 2.8)
+      childprocess (~> 5.0)
+      logger (~> 1.6)
     lint_roller (1.1.0)
     logger (1.7.0)
     marcel (1.1.0)
@@ -61,12 +99,20 @@ GEM
     parser (3.3.10.2)
       ast (~> 2.4.1)
       racc
+    path_expander (2.0.1)
     prism (1.9.0)
     public_suffix (7.0.5)
     racc (1.8.1)
     rainbow (3.1.1)
     rake (13.3.1)
+    reek (6.5.0)
+      dry-schema (~> 1.13)
+      logger (~> 1.6)
+      parser (~> 3.3.0)
+      rainbow (>= 2.0, < 4.0)
+      rexml (~> 3.1)
     regexp_parser (2.11.3)
+    rexml (3.4.4)
     rspec (3.13.2)
       rspec-core (~> 3.13.0)
       rspec-expectations (~> 3.13.0)
@@ -107,10 +153,39 @@ GEM
       ruby_llm-schema (~> 0)
       zeitwerk (~> 2)
     ruby_llm-schema (0.3.0)
+    ruby_parser (3.22.0)
+      racc (~> 1.5)
+      sexp_processor (~> 4.16)
+    rubycritic (4.12.0)
+      flay (~> 2.13)
+      flog (~> 4.7)
+      launchy (>= 2.5.2)
+      parser (>= 3.3.0.5)
+      prism (>= 1.6.0)
+      rainbow (~> 3.1.1)
+      reek (~> 6.5.0, < 7.0)
+      rexml
+      ruby_parser (~> 3.21)
+      simplecov (>= 0.22.0)
+      tty-which (~> 0.5.0)
+      virtus (~> 2.0)
+    sexp_processor (4.17.5)
+    simplecov (0.22.0)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+      simplecov_json_formatter (~> 0.1)
+    simplecov-html (0.13.2)
+    simplecov_json_formatter (0.1.4)
+    thread_safe (0.3.6)
+    tty-which (0.5.0)
     unicode-display_width (3.2.0)
       unicode-emoji (~> 4.1)
     unicode-emoji (4.2.0)
     uri (1.1.1)
+    virtus (2.0.0)
+      axiom-types (~> 0.1)
+      coercible (~> 1.0)
+      descendants_tracker (~> 0.0, >= 0.0.3)
     zeitwerk (2.7.5)
 
 PLATFORMS
@@ -122,26 +197,41 @@ DEPENDENCIES
   rspec (~> 3.13)
   rubocop (~> 1.75)
   ruby_llm-contract!
+  rubycritic (~> 4.9)
+  simplecov (~> 0.22)
 
 CHECKSUMS
   addressable (2.8.9) sha256=cc154fcbe689711808a43601dee7b980238ce54368d23e127421753e46895485
   ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383
+  axiom-types (0.1.1) sha256=c1ff113f3de516fa195b2db7e0a9a95fd1b08475a502ff660d04507a09980383
   base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b
   bigdecimal (4.0.1) sha256=8b07d3d065a9f921c80ceaea7c9d4ae596697295b584c296fe599dd0ad01c4a7
+  childprocess (5.1.0) sha256=9a8d484be2fd4096a0e90a0cd3e449a05bc3aa33f8ac9e4d6dcef6ac1455b6ec
+  coercible (1.0.0) sha256=5081ad24352cc8435ce5472bc2faa30260c7ea7f2102cc6a9f167c4d9bffaadc
   concurrent-ruby (1.3.6) sha256=6b56837e1e7e5292f9864f34b69c5a2cbc75c0cf5338f1ce9903d10fa762d5ab
+  descendants_tracker (0.0.4) sha256=e9c41dd4cfbb85829a9301ea7e7c48c2a03b26f09319db230e6479ccdc780897
   diff-lcs (1.6.2) sha256=9ae0d2cba7d4df3075fe8cd8602a8604993efc0dfa934cff568969efb1909962
+  docile (1.4.1) sha256=96159be799bfa73cdb721b840e9802126e4e03dfc26863db73647204c727f21e
+  dry-configurable (1.3.0) sha256=882d862858567fc1210d2549d4c090f34370fc1bb7c5c1933de3fe792e18afa8
   dry-core (1.2.0) sha256=0cc5a7da88df397f153947eeeae42e876e999c1e30900f3c536fb173854e96a1
   dry-inflector (1.3.1) sha256=7fb0c2bb04f67638f25c52e7ba39ab435d922a3a5c3cd196120f63accb682dcc
+  dry-initializer (3.2.0) sha256=37d59798f912dc0a1efe14a4db4a9306989007b302dcd5f25d0a2a20c166c4e3
   dry-logic (1.6.0) sha256=da6fedbc0f90fc41f9b0cc7e6f05f5d529d1efaef6c8dcc8e0733f685745cea2
+  dry-schema (1.16.0) sha256=cd3aaeabc0f1af66ec82a29096d4c4fb92a0a58b9dae29a22b1bbceb78985727
   dry-types (1.9.1) sha256=baebeecdb9f8395d6c9d227b62011279440943e3ef2468fe8ccc1ba11467f178
+  erubi (1.13.1) sha256=a082103b0885dbc5ecf1172fede897f9ebdb745a4b97a5e8dc63953db1ee4ad9
   event_stream_parser (1.0.0) sha256=a2683bab70126286f8184dc88f7968ffc4028f813161fb073ec90d171f7de3c8
   faraday (2.14.1) sha256=a43cceedc1e39d188f4d2cdd360a8aaa6a11da0c407052e426ba8d3fb42ef61c
   faraday-multipart (1.2.0) sha256=7d89a949693714176f612323ca13746a2ded204031a6ba528adee788694ef757
   faraday-net_http (3.4.2) sha256=f147758260d3526939bf57ecf911682f94926a3666502e24c69992765875906c
   faraday-retry (2.4.0) sha256=7b79c48fb7e56526faf247b12d94a680071ff40c9fda7cf1ec1549439ad11ebe
+  flay (2.14.3) sha256=7f96a495f4bde880460e77e7345464e752bd44f09f5cd30c80af02afe0ed3f29
+  flog (4.9.4) sha256=12cc054fab7a2cbd2a906514397c4d7788954d530564782d6f14939dc2dfbcbb
+  ice_nine (0.11.2) sha256=5d506a7d2723d5592dc121b9928e4931742730131f22a1a37649df1c1e2e63db
   json (2.19.2) sha256=e7e1bd318b2c37c4ceee2444841c86539bc462e81f40d134cf97826cb14e83cf
   json-schema (6.2.0) sha256=e8bff46ed845a22c1ab2bd0d7eccf831c01fe23bb3920caa4c74db4306813666
   language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc
+  launchy (3.1.1) sha256=72b847b5cc961589dde2c395af0108c86ff0119f42d4648d25b5440ebb10059e
   lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87
   logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203
   marcel (1.1.0) sha256=fdcfcfa33cc52e93c4308d40e4090a5d4ea279e160a7f6af988260fa970e0bee
@@ -150,12 +240,15 @@ CHECKSUMS
   net-http (0.9.1) sha256=25ba0b67c63e89df626ed8fac771d0ad24ad151a858af2cc8e6a716ca4336996
   parallel (1.27.0) sha256=4ac151e1806b755fb4e2dc2332cbf0e54f2e24ba821ff2d3dcf86bf6dc4ae130
   parser (3.3.10.2) sha256=6f60c84aa4bdcedb6d1a2434b738fe8a8136807b6adc8f7f53b97da9bc4e9357
+  path_expander (2.0.1) sha256=2de201164bff4719cc4d0b3767286e9977cc832a59c4d70abab571ec86cb41e4
   prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85
   public_suffix (7.0.5) sha256=1a8bb08f1bbea19228d3bed6e5ed908d1cb4f7c2726d18bd9cadf60bc676f623
   racc (1.8.1) sha256=4a7f6929691dbec8b5209a0b373bc2614882b55fc5d2e447a21aaa691303d62f
   rainbow (3.1.1) sha256=039491aa3a89f42efa1d6dec2fc4e62ede96eb6acd95e52f1ad581182b79bc6a
   rake (13.3.1) sha256=8c9e89d09f66a26a01264e7e3480ec0607f0c497a861ef16063604b1b08eb19c
+  reek (6.5.0) sha256=d26d3a492773b2bbc228888067a21afe33ac07954a17dbd64cdeae42c4c69be1
   regexp_parser (2.11.3) sha256=ca13f381a173b7a93450e53459075c9b76a10433caadcb2f1180f2c741fc55a4
+  rexml (3.4.4) sha256=19e0a2c3425dfbf2d4fc1189747bdb2f849b6c5e74180401b15734bc97b5d142
   rspec (3.13.2) sha256=206284a08ad798e61f86d7ca3e376718d52c0bc944626b2349266f239f820587
   rspec-core (3.13.6) sha256=a8823c6411667b60a8bca135364351dda34cd55e44ff94c4be4633b37d828b2d
   rspec-expectations (3.13.5) sha256=33a4d3a1d95060aea4c94e9f237030a8f9eae5615e9bd85718fe3a09e4b58836
@@ -165,11 +258,20 @@ CHECKSUMS
   rubocop-ast (1.49.1) sha256=4412f3ee70f6fe4546cc489548e0f6fcf76cafcfa80fa03af67098ffed755035
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   ruby_llm (1.14.0) sha256=57c6f7034fc4a44504ea137d70f853b07824f1c1cdbe774ab3ab3522e7098deb
-  ruby_llm-contract (0.4.2)
+  ruby_llm-contract (0.5.0)
   ruby_llm-schema (0.3.0) sha256=a591edc5ca1b7f0304f0e2261de61ba4b3bea17be09f5cf7558153adfda3dec6
+  ruby_parser (3.22.0) sha256=1eb4937cd9eb220aa2d194e352a24dba90aef00751e24c8dfffdb14000f15d23
+  rubycritic (4.12.0) sha256=024fed90fe656fa939f6ea80aab17569699ac3863d0b52fd72cb99892247abc8
+  sexp_processor (4.17.5) sha256=ae2b48ba98353d5d465ce8759836b7a05f2e12c5879fcd14d7815b026de32f0e
+  simplecov (0.22.0) sha256=fe2622c7834ff23b98066bb0a854284b2729a569ac659f82621fc22ef36213a5
+  simplecov-html (0.13.2) sha256=bd0b8e54e7c2d7685927e8d6286466359b6f16b18cb0df47b508e8d73c777246
+  simplecov_json_formatter (0.1.4) sha256=529418fbe8de1713ac2b2d612aa3daa56d316975d307244399fa4838c601b428
+  thread_safe (0.3.6) sha256=9ed7072821b51c57e8d6b7011a8e282e25aeea3a4065eab326e43f66f063b05a
+  tty-which (0.5.0) sha256=5824055f0d6744c97e7c4426544f01d519c40d1806ef2ef47d9854477993f466
   unicode-display_width (3.2.0) sha256=0cdd96b5681a5949cdbc2c55e7b420facae74c4aaf9a9815eee1087cb1853c42
   unicode-emoji (4.2.0) sha256=519e69150f75652e40bf736106cfbc8f0f73aa3fb6a65afe62fefa7f80b0f80f
   uri (1.1.1) sha256=379fa58d27ffb1387eaada68c749d1426738bd0f654d812fcc07e7568f5c57c6
+  virtus (2.0.0) sha256=8841dae4eb7fcc097320ba5ea516bf1839e5d056c61ee27138aa4bddd6e3d1c2
   zeitwerk (2.7.5) sha256=d8da92128c09ea6ec62c949011b00ed4a20242b255293dd66bf41545398f73dd
 
 BUNDLED WITH

data/README.md

@@ -157,6 +157,44 @@ report = ClassifyTicket.run_eval("regression",
   concurrency: 4)
 ```
 
+## Prompt A/B testing
+
+Changed a prompt? Compare old vs new with regression safety:
+
+```ruby
+diff = ClassifyTicketV2.compare_with(ClassifyTicketV1,
+  eval: "regression", model: "gpt-4.1-mini")
+
+diff.safe_to_switch?  # => true (no regressions, no per-case score drops)
+diff.improvements     # => [{case: "outage", ...}]
+diff.score_delta      # => +0.33
+```
+
+Requires `model:` or `context: { adapter: ... }`.
+`compare_with` ignores `sample_response`; without a real model/adapter both sides are skipped and the A/B result is not meaningful.
+
+CI gate:
+```ruby
+expect(ClassifyTicketV2).to pass_eval("regression")
+  .compared_with(ClassifyTicketV1)
+  .with_minimum_score(0.8)
+```
+
+## Soft observations
+
+Log suspicious-but-not-invalid output without failing the contract:
+
+```ruby
+class EvaluateComparative < RubyLLM::Contract::Step::Base
+  validate("scores in range") { |o| (1..10).include?(o[:score_a]) }
+  observe("scores should differ") { |o| o[:score_a] != o[:score_b] }
+end
+
+result = EvaluateComparative.run(input)
+result.ok?            # => true (observe never fails)
+result.observations   # => [{description: "scores should differ", passed: false}]
+```
+
 ## Predict cost before running
 
 ```ruby
@@ -190,11 +228,13 @@ Works with any ruby_llm provider (OpenAI, Anthropic, Gemini, etc).
 
 ## Roadmap
 
-**v0.4 (current):** Observability & scale — eval history with trending, batch eval with concurrency, pipeline per-step eval, Minitest support, structured logging.
+**v0.5 (current):** Data-driven prompt engineering — `compare_with(OtherStep)` for prompt A/B testing with regression safety. `observe` DSL for soft observations that log but never fail.
+
+**v0.4:** Observability & scale — eval history with trending, batch eval with concurrency, pipeline per-step eval, Minitest support, structured logging. Audit hardening (18 bugfixes).
 
 **v0.3:** Baseline regression detection, migration guide, production hardening.
 
-**v0.5:** Prompt A/B testing — `compare_with(OtherStep)` for data-driven prompt engineering with regression safety. Cross-provider comparison docs.
+**v0.6:** Model recommendation based on eval history data. Cross-provider comparison docs.
 
 ## License
 

data/lib/ruby_llm/contract/concerns/context_helpers.rb

@@ -9,22 +9,23 @@ module RubyLLM
         private
 
         def safe_context(context)
-          (context || {}).transform_keys { |k| k.respond_to?(:to_sym) ? k.to_sym : k }
+          (context || {}).transform_keys { |key| key.respond_to?(:to_sym) ? key.to_sym : key }
         end
 
         def isolate_context(context)
-          context.transform_values do |v|
-            if v.respond_to?(:clone_for_concurrency)
-              v.clone_for_concurrency
-            elsif v.respond_to?(:dup)
-              v.dup
-            else
-              v
-            end
+          context.transform_values do |value|
+            duplicate_context_value(value)
           rescue TypeError
-            v
+            value
           end
         end
+
+        def duplicate_context_value(value)
+          return value.clone_for_concurrency if value.respond_to?(:clone_for_concurrency)
+          return value.dup if value.respond_to?(:dup)
+
+          value
+        end
       end
     end
   end

data/lib/ruby_llm/contract/concerns/deep_freeze.rb

@@ -8,15 +8,21 @@ module RubyLLM
       module DeepFreeze
         private
 
-        def deep_dup_freeze(obj)
-          case obj
-          when NilClass, Integer, Float, Symbol, TrueClass, FalseClass then obj
-          when Hash then obj.transform_values { |v| deep_dup_freeze(v) }.freeze
-          when Array then obj.map { |v| deep_dup_freeze(v) }.freeze
-          when String then obj.frozen? ? obj : obj.dup.freeze
-          else obj.frozen? ? obj : obj.dup.freeze
+        IMMUTABLE_TYPES = [NilClass, Integer, Float, Symbol, TrueClass, FalseClass].freeze
+
+        def deep_dup_freeze(object)
+          case object
+          when *IMMUTABLE_TYPES then object
+          when Hash then object.transform_values { |value| deep_dup_freeze(value) }.freeze
+          when Array then object.map { |value| deep_dup_freeze(value) }.freeze
+          else
+            frozen_copy(object)
           end
         end
+
+        def frozen_copy(object)
+          object.frozen? ? object : object.dup.freeze
+        end
       end
     end
   end

data/lib/ruby_llm/contract/concerns/deep_symbolize.rb

@@ -3,12 +3,22 @@
 module RubyLLM
   module Contract
     module Concerns
+      # Recursively converts Hash keys to symbols while preserving array shape.
       module DeepSymbolize
-        def deep_symbolize(obj)
-          case obj
-          when Hash then obj.transform_keys(&:to_sym).transform_values { |val| deep_symbolize(val) }
-          when Array then obj.map { |val| deep_symbolize(val) }
-          else obj
+        def deep_symbolize(object)
+          case object
+          when Hash then symbolize_hash(object)
+          when Array then object.map { |value| deep_symbolize(value) }
+          else
+            object
+          end
+        end
+
+        private
+
+        def symbolize_hash(hash)
+          hash.each_with_object({}) do |(key, value), symbolized|
+            symbolized[key.to_sym] = deep_symbolize(value)
           end
         end
       end

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -5,6 +5,11 @@ module RubyLLM
     module Concerns
       module EvalHost
         include ContextHelpers
+
+        SAMPLE_RESPONSE_COMPARE_WARNING = "[ruby_llm-contract] compare_with ignores sample_response. " \
+                                          "Without model: or context: { adapter: ... }, both sides will be skipped " \
+                                          "and the A/B comparison is not meaningful.".freeze
+
         def define_eval(name, &)
           @eval_definitions ||= {}
           @file_sourced_evals ||= Set.new
@@ -45,6 +50,26 @@ module RubyLLM
           end
         end
 
+        # Compare this step (candidate) with another step (baseline) using the
+        # baseline's eval definition as single source of truth.
+        #
+        # Requires a real adapter or model in context. sample_response is
+        # intentionally NOT used, because A/B testing with canned data
+        # gives identical results for both sides rather than a real comparison.
+        def compare_with(other_step, eval:, model: nil, context: {})
+          ctx = comparison_context(context, model)
+          baseline_defn = baseline_eval_definition(other_step, eval)
+          raise ArgumentError, "No eval '#{eval}' on baseline step #{other_step}" unless baseline_defn
+
+          dataset = baseline_defn.build_dataset
+          warn_sample_response_compare(ctx, baseline_defn)
+
+          my_report = Eval::Runner.run(step: self, dataset: dataset, context: isolate_context(ctx))
+          other_report = Eval::Runner.run(step: other_step, dataset: dataset, context: isolate_context(ctx))
+
+          Eval::PromptDiff.new(candidate: my_report, baseline: other_report)
+        end
+
         def compare_models(eval_name, models:, context: {})
           context = safe_context(context)
           models = models.uniq
@@ -57,6 +82,21 @@ module RubyLLM
 
         private
 
+        def comparison_context(context, model)
+          base_context = safe_context(context)
+          model ? base_context.merge(model: model) : base_context
+        end
+
+        def baseline_eval_definition(other_step, eval_name)
+          other_step.send(:all_eval_definitions)[eval_name.to_s]
+        end
+
+        def warn_sample_response_compare(context, baseline_defn)
+          return if context[:adapter] || context[:model] || !baseline_defn.build_adapter
+
+          warn SAMPLE_RESPONSE_COMPARE_WARNING
+        end
+
         def all_eval_definitions
           inherited = if superclass.respond_to?(:all_eval_definitions, true)
                         superclass.send(:all_eval_definitions)
@@ -71,20 +111,24 @@ module RubyLLM
           defn = all_eval_definitions[name.to_s]
           raise ArgumentError, "No eval '#{name}' defined. Available: #{all_eval_definitions.keys}" unless defn
 
-          effective_context = eval_context(defn, context)
-          Eval::Runner.run(step: self, dataset: defn.build_dataset, context: effective_context,
-                           concurrency: concurrency)
+          run_eval_definition(defn, context, concurrency: concurrency)
         end
 
         def run_all_own_evals(context, concurrency: nil)
           all_eval_definitions.transform_values do |defn|
-            isolated_context = isolate_context(context)
-            effective_context = eval_context(defn, isolated_context)
-            Eval::Runner.run(step: self, dataset: defn.build_dataset, context: effective_context,
-                             concurrency: concurrency)
+            run_eval_definition(defn, isolate_context(context), concurrency: concurrency)
           end
         end
 
+        def run_eval_definition(defn, context, concurrency: nil)
+          Eval::Runner.run(
+            step: self,
+            dataset: defn.build_dataset,
+            context: eval_context(defn, context),
+            concurrency: concurrency
+          )
+        end
+
         def eval_context(defn, context)
           context = safe_context(context)
           return context if context[:adapter]

data/lib/ruby_llm/contract/contract/schema_validator/bound_rule.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    class SchemaValidator
+      # Validates numeric and collection size bounds for a node.
+      class BoundRule
+        NUMERIC_LIMITS = [
+          { key: :minimum, label: "minimum", relation: "below", invalid: ->(actual, limit) { actual < limit } },
+          { key: :maximum, label: "maximum", relation: "above", invalid: ->(actual, limit) { actual > limit } }
+        ].freeze
+        SIZE_LIMITS = [
+          { bound: :min_key, relation: "below", invalid: ->(actual, limit) { actual < limit } },
+          { bound: :max_key, relation: "above", invalid: ->(actual, limit) { actual > limit } }
+        ].freeze
+
+        def initialize(errors)
+          @errors = errors
+        end
+
+        def validate(node)
+          value = node.value
+          schema = node.schema
+          path = node.path
+
+          append_numeric_bound_errors(path, value, schema) if value.is_a?(Numeric)
+          append_size_bound_errors(path, value, schema)
+        end
+
+        private
+
+        def append_numeric_bound_errors(path, value, schema)
+          NUMERIC_LIMITS.each do |limit_config|
+            append_bound_error(
+              path: path,
+              actual: value,
+              limit: schema[limit_config[:key]],
+              label: limit_config[:label],
+              relation: limit_config[:relation],
+              invalid: limit_config[:invalid]
+            )
+          end
+        end
+
+        def append_size_bound_errors(path, value, schema)
+          bounds = size_bounds_for(value)
+          return unless bounds
+
+          actual_size = value.length
+          metric = bounds[:metric]
+
+          SIZE_LIMITS.each do |limit_config|
+            label = bounds[limit_config[:bound]]
+            append_bound_error(
+              path: path,
+              actual: actual_size,
+              limit: schema[label],
+              label: label,
+              relation: limit_config[:relation],
+              metric: metric,
+              invalid: limit_config[:invalid]
+            )
+          end
+        end
+
+        def append_bound_error(path:, actual:, limit:, label:, relation:, invalid:, metric: nil)
+          return unless limit
+          return unless invalid.call(actual, limit)
+
+          subject = metric ? "#{metric} #{actual}" : actual
+          @errors << "#{path}: #{subject} is #{relation} #{label} #{limit}"
+        end
+
+        def size_bounds_for(value)
+          case value
+          when String
+            SchemaValidator::SIZE_BOUNDS[:string]
+          when Array
+            SchemaValidator::SIZE_BOUNDS[:array]
+          end
+        end
+      end
+    end
+  end
+end