rubric_llm 0.5.0 → 0.6.0.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e4b3be22ad7ef0d4af66ddba820a505c75c29f3bf47e1259bdee1ed28cf87ae4
-  data.tar.gz: 2250b2b15b76203042befc1a86805827161c0df49efaf8dc240c5c594d5166b3
+  metadata.gz: 6d32ed4cba47a2fac15973d05bbf531317fa8d80f4c42b277dae5c302884ae57
+  data.tar.gz: a4bfbbc318593df77c33d36d41eda2e3d2f1b0cd4d0485fe9b8a61e72ee1519b
 SHA512:
-  metadata.gz: 9e7e813f4bf58b17b70b6da5b42f0e77e962cc2862a1b5a62e59c5152ebc8de0c25058842c1c92344704e5fd249846a8662bd9937ad40ef068464c3520c3fee0
-  data.tar.gz: aa9f1156c49f63cef0f00a7c649b14a6b11d9bec8904962e5a25af88bafd70447c797d7bb1f69ddffd5b17f13dda2e5e702fed5aede15c4cd9ef59c871c1be52
+  metadata.gz: a63b324d6f721297415feb55f6232b2771d58a7ca86a4a2ff0903abffec3c5fc5ad638c3d32632ed27bbf6cafea666a8247dcfa418d3454df61e3856c6c83b0a
+  data.tar.gz: 424912623b5f7f8157c91c10310c4e8ebe34e7406fcc146a3c146bd4e9de0cbe9b0100acc4b7a9a98174153f53178961f9c8233b120b3d3d23c11a5bf29bb360

data/CHANGELOG.md

@@ -5,6 +5,19 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0.rc1] - 2026-09-08
+
+This prerelease targets RubyLLM `2.0.0.rc1` and supports RubyLLM 2 only. Applications that use RubyLLM 1.x should stay on RubricLLM `0.5.x`. Stable `0.6.0` waits for the final RubyLLM 2.0 release and the complete Ruby 3.4 and 4.0 test matrix.
+
+### Changed
+
+- Require the exact RubyLLM `2.0.0.rc1` prerelease while the compatibility release is under review; Ruby >= 3.4 remains supported
+- Keep `max_tokens` and `RUBRIC_MAX_TOKENS` as the public RubricLLM setting while RubyLLM maps the limit to the selected provider and protocol
+- Treat explicit `temperature: nil` as an instruction to omit temperature from the provider request. Before 0.6.0.rc1, it selected `RUBRIC_TEMPERATURE` or `0.0`; an omitted temperature still reads `RUBRIC_TEMPERATURE` and defaults to `0.0`
+- Use OpenAI Responses by default through RubyLLM 2. OpenAI-compatible gateways that only accept Chat Completions can set `config.openai_protocol = :chat_completions`
+- Keep RubyLLM transport retries and RubricLLM judge retries separate. With RubyLLM's default `config.max_retries = 3` and RubricLLM's default `max_retries: 2`, a retryable metric failure can produce up to `(3 + 1) * (2 + 1) = 12` HTTP attempts
+- Keep Rails persistence migrations separate. Applications that use RubyLLM's Rails records must follow RubyLLM's upstream 2.0 migration guide
+
 ## [0.5.0] - 2026-08-23
 
 ### Fixed

data/README.md

@@ -8,18 +8,29 @@ Lightweight LLM evaluation framework for Ruby, inspired by [DeepEval](https://gi
 
 Provider-agnostic evaluation with pluggable metrics, statistical A/B comparison, and test framework integration: no Rails, no ActiveRecord, no UI. Works anywhere Ruby runs.
 
+`0.6.0.rc1` is a prerelease for RubyLLM `2.0.0.rc1`. It supports RubyLLM 2 only. Applications that use RubyLLM 1.x should stay on RubricLLM `0.5.x`. Stable `0.6.0` waits for the final RubyLLM 2.0 release and the complete Ruby 3.4 and 4.0 test matrix.
+
 ## Installation
 
-Add to your Gemfile:
+Install the release candidate with both exact prerelease constraints in the same Gemfile change:
 
 ```ruby
-gem "rubric_llm"
+gem "rubric_llm", "0.6.0.rc1"
+gem "ruby_llm", "2.0.0.rc1"
+```
+
+Then resolve both gems together:
+
+```bash
+bundle update rubric_llm ruby_llm
 ```
 
+RubyLLM's upstream prerelease pin command is `bundle add ruby_llm --version 2.0.0.rc1`. If the application still has a RubyLLM 1.x constraint, update both constraints before running Bundler.
+
 Or install directly:
 
 ```bash
-gem install rubric_llm
+gem install rubric_llm --version 0.6.0.rc1 --pre
 ```
 
 ## Quick Start
@@ -62,6 +73,25 @@ RubricLLM.configure do |c|
 end
 ```
 
+`max_tokens` remains RubricLLM's public setting, and `RUBRIC_MAX_TOKENS` remains its environment variable. RubyLLM 2 maps this shared limit to the selected provider and protocol.
+
+When `temperature` is omitted, RubricLLM reads `RUBRIC_TEMPERATURE` and uses `0.0` when the variable is not set. Before 0.6.0.rc1, an explicit `temperature: nil` selected that environment value or `0.0`; it now means that RubricLLM omits temperature from the provider request, so the provider chooses its default:
+
+```ruby
+RubricLLM::Config.new                  # RUBRIC_TEMPERATURE, otherwise 0.0
+RubricLLM::Config.new(temperature: nil) # omit temperature from the request
+```
+
+RubyLLM 2 uses the OpenAI Responses protocol by default when the selected model supports it. For an OpenAI-compatible gateway that only accepts Chat Completions, configure RubyLLM before evaluating:
+
+```ruby
+RubyLLM.configure do |config|
+  config.openai_protocol = :chat_completions
+end
+```
+
+Structured output support depends on the selected provider and model. RubricLLM sends its schema when RubyLLM reports structured output support. Otherwise it requests JSON text and validates the response object and score locally. Check the target provider's support before relying on a schema or a specific protocol. See RubyLLM's [2.0 upgrade guide](https://rubyllm.com/next/upgrading/), [request control guide](https://rubyllm.com/next/chat-request-control/), and [structured output support](https://rubyllm.com/next/structured-output/).
+
 ### Environment Variables
 
 All config fields can be set via environment variables:
@@ -100,6 +130,14 @@ RubricLLM.configure do |c|
 end
 ```
 
+RubricLLM has no Rails models or database migrations. If the application also uses RubyLLM's Rails persistence, follow RubyLLM's 2.0 upgrade guide and run its phased migrations separately.
+
+### Retries
+
+RubyLLM transport retries and RubricLLM judge retries remain separate. With RubyLLM's default `config.max_retries = 3` and RubricLLM's default `max_retries: 2`, one retryable metric failure can produce up to `(3 + 1) * (2 + 1) = 12` HTTP attempts. Set `RUBRIC_MAX_RETRIES` and `RUBRIC_RETRY_BASE_DELAY` for RubricLLM's layer, and set RubyLLM's `config.max_retries` and related transport settings for its layer. RubricLLM 0.6 does not combine or redesign these retry layers.
+
+RubyLLM classifies OpenAI's HTTP 429 `insufficient_quota` response as a rate-limit error, so an exhausted account uses both retry budgets and their delays before the error is returned.
+
 ## Metrics
 
 ### LLM-as-Judge Metrics
@@ -323,7 +361,7 @@ result.overall                         # => mean of non-nil scores only
 
 ```bash
 bundle install
-bundle exec rake test
+bundle exec rake test test_contract
 bundle exec rubocop
 ```
 
@@ -365,7 +403,7 @@ Deep dives live in the [project wiki](https://github.com/dpaluy/rubric_llm/wiki)
 ## Requirements
 
 - Ruby >= 3.4
-- [ruby_llm](https://github.com/crmne/ruby_llm) ~> 1.0
+- [ruby_llm](https://github.com/crmne/ruby_llm) = 2.0.0.rc1 for RubricLLM 0.6.0.rc1
 - An API key for your chosen LLM provider (set via RubyLLM configuration)
 
 ## Contributing

data/lib/rubric_llm/config.rb

@@ -5,11 +5,12 @@ module RubricLLM
     attr_accessor :judge_model, :judge_provider, :temperature, :max_tokens, :custom_prompt,
                   :max_retries, :retry_base_delay, :concurrency
 
-    def initialize(judge_model: nil, judge_provider: nil, temperature: nil, max_tokens: nil, # rubocop:disable Metrics/ParameterLists
+    def initialize(judge_model: nil, judge_provider: nil, # rubocop:disable Metrics/ParameterLists
+                   temperature: Float(ENV.fetch("RUBRIC_TEMPERATURE", "0.0")), max_tokens: nil,
                    custom_prompt: nil, max_retries: nil, retry_base_delay: nil, concurrency: nil, validate: false)
       @judge_model = judge_model || ENV.fetch("RUBRIC_JUDGE_MODEL", "gpt-4o")
       @judge_provider = (judge_provider || ENV.fetch("RUBRIC_JUDGE_PROVIDER", "openai")).to_sym
-      @temperature = temperature || Float(ENV.fetch("RUBRIC_TEMPERATURE", "0.0"))
+      @temperature = temperature
       @max_tokens = max_tokens || Integer(ENV.fetch("RUBRIC_MAX_TOKENS", "4096"))
       @custom_prompt = custom_prompt
       @max_retries = max_retries || Integer(ENV.fetch("RUBRIC_MAX_RETRIES", "2"))
@@ -61,9 +62,9 @@ module RubricLLM
     end
 
     def validate_temperature
-      return if temperature.is_a?(Numeric) && temperature.between?(0.0, 2.0)
+      return if temperature.nil? || (temperature.is_a?(Numeric) && temperature.between?(0.0, 2.0))
 
-      raise ConfigurationError, "temperature must be between 0.0 and 2.0"
+      raise ConfigurationError, "temperature must be nil or between 0.0 and 2.0"
     end
 
     def validate_max_tokens

data/lib/rubric_llm/judge.rb

@@ -51,14 +51,13 @@ module RubricLLM
         attempts += 1
         chat = RubyLLM.chat(model: config.judge_model, provider: config.judge_provider)
         chat.with_temperature(config.temperature)
-        chat.with_params(max_tokens: config.max_tokens)
+        chat.with_max_output_tokens(config.max_tokens)
         apply_response_schema(chat)
 
         full_system_prompt = build_system_prompt(system_prompt)
         chat.with_instructions(full_system_prompt)
         response = chat.ask(user_prompt)
-        content = response.content
-        validate_response!(content.is_a?(Hash) ? content : parse_json(content))
+        validate_response!(parse_json(response.content))
       rescue StandardError => e
         raise wrap_error(e) unless transient?(e) && attempts <= config.max_retries
 
@@ -100,17 +99,13 @@ module RubricLLM
     end
 
     def apply_response_schema(chat)
-      return chat unless chat.respond_to?(:with_schema)
       return chat unless structured_output_supported?(chat)
 
       chat.with_schema(METRIC_RESPONSE_SCHEMA)
     end
 
     def structured_output_supported?(chat)
-      return true unless chat.respond_to?(:model)
-      return true unless chat.model.respond_to?(:structured_output?)
-
-      chat.model.structured_output?
+      chat.model.supports?(:structured_output)
     end
 
     def validate_response!(response)

data/lib/rubric_llm/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RubricLLM
-  VERSION = "0.5.0"
+  VERSION = "0.6.0.rc1"
 end

data/lib/rubric_llm.rb

@@ -143,7 +143,7 @@ module RubricLLM
     def apply_custom_prompt(config, custom_prompt)
       return config unless custom_prompt
 
-      Config.new(**config.to_h.compact, custom_prompt:)
+      Config.new(**config.to_h, custom_prompt:)
     end
 
     def normalize_sample(sample)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rubric_llm
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.6.0.rc1
 platform: ruby
 authors:
 - David Paluy
@@ -27,16 +27,16 @@ dependencies:
   name: ruby_llm
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '1.16'
+        version: 2.0.0.rc1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '1.16'
+        version: 2.0.0.rc1
 description: Provider-agnostic LLM evaluation with pluggable metrics, statistical
   A/B comparison, and test framework integration. Ragas for Ruby, powered by RubyLLM.
 email: