riffer 0.38.1 → 0.39.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 81a97541ff82791dbf24d5bcad216678db7db1262d3f0e18e4dfe18d77169b20
-  data.tar.gz: 294d8493c2e6349e6005c190deb15c7b56c1d1cfc49bcdafa2d99ea39fb4b1e3
+  metadata.gz: b1bf92e3bef065a4c07aadb273a7f6770b711c0f15f268726133c841cb3fd510
+  data.tar.gz: d7361c029b98ca819932e2c3d2d53113710bb434ad60092ee0578179fa844c5e
 SHA512:
-  metadata.gz: 7004eefa26eaa6bd7a02ff31ae04a4596c063f591b0375f7089e8f39037254945b32a5e57ef6595ddc3f0845173b07d5e020b01c553b573df2a9b535f21d15a2
-  data.tar.gz: c0444136536b7034241212e9bca19ad634c98080a1491af4cb32e51a2dcecf0566dcc0e5cd798e2fd433a0bebf1aff02fee46f231454500afaa1a8fb7e5675a3
+  metadata.gz: 7f503377743f6629a1423234afb158570dc33ee225c82c0bc46e28938a839fe25b5bdfad6f61da274a8bf81e1e8dd6c505ad826593c5eb269251d47667fe9d7c
+  data.tar.gz: eddabf85e5416c777c3a9a441ce99cd1cb902ec1c339fa7338890ae8cb7e78c5e78628e1b886aea44cdc6228b99299fa3983816e10075d8c28b59229e4c0b376

data/.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "0.38.1"
+  ".": "0.39.0"
 }

data/CHANGELOG.md

@@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.39.0](https://github.com/janeapp/riffer/compare/riffer/v0.38.1...riffer/v0.39.0) (2026-07-29)
+
+
+### Features
+
+* **evals:** expose token usage on evaluation results ([#380](https://github.com/janeapp/riffer/issues/380)) ([32d576e](https://github.com/janeapp/riffer/commit/32d576e492809845c36f44ebf37de3af3b6785b0))
+
 ## [0.38.1](https://github.com/janeapp/riffer/compare/riffer/v0.38.0...riffer/v0.38.1) (2026-07-29)
 
 

data/docs/11_EVALS.md

@@ -108,9 +108,11 @@ Per-scenario `context` overrides the top-level value. Scenarios without their ow
 The runner returns a `Riffer::Evals::RunResult`:
 
 ```ruby
-result.scores             # => { EvaluatorClass => avg_score } across all scenarios
-result.scenario_results   # => Array of ScenarioResult objects
-result.to_h               # => Hash representation
+result.scores                  # => { EvaluatorClass => avg_score } across all scenarios
+result.scenario_results        # => Array of ScenarioResult objects
+result.token_usage             # => TokenUsage the agent under test spent, summed across scenarios (nil if none reported)
+result.evaluator_token_usage   # => TokenUsage the judges spent, summed across scenarios (nil if no judge ran)
+result.to_h                    # => Hash representation
 ```
 
 ### ScenarioResult
@@ -119,13 +121,15 @@ Each scenario produces a `Riffer::Evals::ScenarioResult`:
 
 ```ruby
 scenario = result.scenario_results.first
-scenario.input        # => "What is the capital of France?"
-scenario.output       # => "The capital of France is Paris."
-scenario.ground_truth # => "Paris"
-scenario.scores       # => { EvaluatorClass => score } for this scenario
-scenario.results      # => Array of Result objects
-scenario.messages     # => Array of Message objects (system, user, assistant, tool)
-scenario.to_h         # => Hash representation
+scenario.input                  # => "What is the capital of France?"
+scenario.output                 # => "The capital of France is Paris."
+scenario.ground_truth           # => "Paris"
+scenario.scores                 # => { EvaluatorClass => score } for this scenario
+scenario.results                # => Array of Result objects
+scenario.messages               # => Array of Message objects (system, user, assistant, tool)
+scenario.token_usage            # => TokenUsage the agent under test spent on this scenario (nil if not reported)
+scenario.evaluator_token_usage  # => TokenUsage the judges spent on this scenario (nil if no judge ran)
+scenario.to_h                   # => Hash representation
 ```
 
 ### Result
@@ -138,8 +142,13 @@ r.evaluator        # => AnswerRelevancyEvaluator
 r.score            # => 0.92
 r.reason           # => "The response directly addresses..."
 r.higher_is_better # => true
+r.token_usage      # => TokenUsage for the judge call (nil for rule-based evaluators)
 ```
 
+`token_usage` is a `Riffer::Providers::TokenUsage`, the same type the agent exposes on `Agent::Response`, so eval usage accumulates with the `+` operator just like agent usage. On a `Result` it's populated for LLM-as-judge evaluators and `nil` for rule-based ones that never call an LLM.
+
+`ScenarioResult` and `RunResult` keep the two sources separate rather than exposing a single total: `token_usage` is what the agent under test spent generating the output, and `evaluator_token_usage` is what the judges spent scoring it. Add them yourself if you want a combined figure.
+
 ## Defining Custom Evaluators
 
 Create evaluators by subclassing `Riffer::Evals::Evaluator`. The simplest approach uses the `instructions` DSL — the base class handles calling the judge automatically:
@@ -190,7 +199,7 @@ Instance methods:
 
 - `evaluate(input:, output:, ground_truth:, messages:)` - Override for custom logic; default calls judge with `instructions`
 - `judge` - Returns a Judge instance for LLM-as-judge calls
-- `result(score:, reason:, metadata:)` - Helper to build Result objects
+- `result(score:, reason:, metadata:, token_usage:)` - Helper to build Result objects
 
 ### Advanced: Custom Evaluate Override
 

data/lib/riffer/evals/evaluator.rb

@@ -66,7 +66,7 @@ class Riffer::Evals::Evaluator
       ground_truth: ground_truth
     )
 
-    result(score: evaluation[:score], reason: evaluation[:reason])
+    result(score: evaluation[:score], reason: evaluation[:reason], token_usage: evaluation[:token_usage])
   end
 
   private
@@ -105,14 +105,15 @@ class Riffer::Evals::Evaluator
 
   # Builds a Result for this evaluator.
   #--
-  #: (score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped]) -> Riffer::Evals::Result
-  def result(score:, reason: nil, metadata: {})
+  #: (score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped], ?token_usage: Riffer::Providers::TokenUsage?) -> Riffer::Evals::Result
+  def result(score:, reason: nil, metadata: {}, token_usage: nil)
     Riffer::Evals::Result.new(
       evaluator: self.class,
       score: score,
       reason: reason,
       metadata: metadata,
-      higher_is_better: self.class.higher_is_better
+      higher_is_better: self.class.higher_is_better,
+      token_usage: token_usage
     )
   end
 end

data/lib/riffer/evals/evaluator_runner.rb

@@ -62,6 +62,7 @@ module Riffer::Evals::EvaluatorRunner
     response = agent.generate(input, context: resolved_context)
     output = response.content
     messages = response.messages
+    token_usage = response.token_usage
 
     results = evaluators.map do |evaluator_class|
       evaluator_class.new.evaluate(input: input, output: output, ground_truth: ground_truth, messages: messages)
@@ -72,7 +73,8 @@ module Riffer::Evals::EvaluatorRunner
       output: output,
       ground_truth: ground_truth,
       results: results,
-      messages: messages
+      messages: messages,
+      token_usage: token_usage
     )
   end
 end

data/lib/riffer/evals/judge.rb

@@ -121,7 +121,8 @@ class Riffer::Evals::Judge
 
     {
       score: score.to_f,
-      reason: reason
+      reason: reason,
+      token_usage: response.token_usage
     }
   rescue JSON::ParserError => e
     raise Riffer::Error, "Invalid judge response: #{e.message}"

data/lib/riffer/evals/result.rb

@@ -18,16 +18,21 @@ class Riffer::Evals::Result
   # Whether higher scores are better for this evaluator.
   attr_reader :higher_is_better #: bool
 
+  # Token usage for the judge call that produced this result, when the
+  # evaluator used an LLM. Nil for rule-based evaluators.
+  attr_reader :token_usage #: Riffer::Providers::TokenUsage?
+
   # Raises Riffer::ArgumentError if +score+ is not between 0.0 and 1.0.
   #--
-  #: (evaluator: singleton(Riffer::Evals::Evaluator), score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped], ?higher_is_better: bool) -> void
-  def initialize(evaluator:, score:, reason: nil, metadata: {}, higher_is_better: true)
+  #: (evaluator: singleton(Riffer::Evals::Evaluator), score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped], ?higher_is_better: bool, ?token_usage: Riffer::Providers::TokenUsage?) -> void
+  def initialize(evaluator:, score:, reason: nil, metadata: {}, higher_is_better: true, token_usage: nil)
     @evaluator = evaluator
     @score = score.to_f
     validate_score!
     @reason = reason
     @metadata = metadata
     @higher_is_better = higher_is_better
+    @token_usage = token_usage
   end
 
   # Returns a hash representation of the result.
@@ -40,7 +45,8 @@ class Riffer::Evals::Result
       score: score,
       reason: reason,
       metadata: metadata,
-      higher_is_better: higher_is_better
+      higher_is_better: higher_is_better,
+      token_usage: token_usage&.to_h
     }
   end
 

data/lib/riffer/evals/run_result.rb

@@ -35,6 +35,24 @@ class Riffer::Evals::RunResult
     end
   end
 
+  # Returns the summed token usage the agents under test spent across every
+  # scenario, or nil when none reported usage.
+  #
+  #--
+  #: () -> Riffer::Providers::TokenUsage?
+  def token_usage
+    scenario_results.map(&:token_usage).compact.reduce(:+)
+  end
+
+  # Returns the summed token usage across every scenario's LLM-as-judge
+  # evaluators, or nil when none reported usage.
+  #
+  #--
+  #: () -> Riffer::Providers::TokenUsage?
+  def evaluator_token_usage
+    scenario_results.map(&:evaluator_token_usage).compact.reduce(:+)
+  end
+
   # Returns a hash representation of the run result.
   #
   #--
@@ -42,7 +60,9 @@ class Riffer::Evals::RunResult
   def to_h
     {
       scores: scores.transform_keys(&:name),
-      scenario_results: scenario_results.map(&:to_h)
+      scenario_results: scenario_results.map(&:to_h),
+      token_usage: token_usage&.to_h,
+      evaluator_token_usage: evaluator_token_usage&.to_h
     }
   end
 end

data/lib/riffer/evals/scenario_result.rb

@@ -18,14 +18,18 @@ class Riffer::Evals::ScenarioResult
   # The full message history from the agent conversation.
   attr_reader :messages #: Array[Riffer::Messages::Base]
 
+  # Token usage the agent under test spent generating this scenario's output.
+  attr_reader :token_usage #: Riffer::Providers::TokenUsage?
+
   #--
-  #: (input: String, output: String, ground_truth: String?, results: Array[Riffer::Evals::Result], ?messages: Array[Riffer::Messages::Base]) -> void
-  def initialize(input:, output:, ground_truth:, results:, messages: [])
+  #: (input: String, output: String, ground_truth: String?, results: Array[Riffer::Evals::Result], ?messages: Array[Riffer::Messages::Base], ?token_usage: Riffer::Providers::TokenUsage?) -> void
+  def initialize(input:, output:, ground_truth:, results:, messages: [], token_usage: nil)
     @input = input
     @output = output
     @ground_truth = ground_truth
     @results = results
     @messages = messages
+    @token_usage = token_usage
   end
 
   # Returns scores keyed by evaluator class.
@@ -39,6 +43,15 @@ class Riffer::Evals::ScenarioResult
     end
   end
 
+  # Returns the summed token usage across this scenario's LLM-as-judge
+  # evaluators, or nil when none reported usage.
+  #
+  #--
+  #: () -> Riffer::Providers::TokenUsage?
+  def evaluator_token_usage
+    results.map(&:token_usage).compact.reduce(:+)
+  end
+
   # Returns a hash representation of the scenario result.
   #
   #--
@@ -50,7 +63,9 @@ class Riffer::Evals::ScenarioResult
       ground_truth: ground_truth,
       scores: scores.transform_keys(&:name),
       results: results.map(&:to_h),
-      messages: messages.map(&:to_h)
+      messages: messages.map(&:to_h),
+      token_usage: token_usage&.to_h,
+      evaluator_token_usage: evaluator_token_usage&.to_h
     }
   end
 end

data/lib/riffer/version.rb

@@ -2,5 +2,5 @@
 # rbs_inline: enabled
 
 module Riffer
-  VERSION = "0.38.1" #: String
+  VERSION = "0.39.0" #: String
 end

data/sig/generated/riffer/evals/evaluator.rbs

@@ -57,6 +57,6 @@ class Riffer::Evals::Evaluator
 
   # Builds a Result for this evaluator.
   # --
-  # : (score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped]) -> Riffer::Evals::Result
-  def result: (score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped]) -> Riffer::Evals::Result
+  # : (score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped], ?token_usage: Riffer::Providers::TokenUsage?) -> Riffer::Evals::Result
+  def result: (score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped], ?token_usage: Riffer::Providers::TokenUsage?) -> Riffer::Evals::Result
 end

data/sig/generated/riffer/evals/result.rbs

@@ -17,10 +17,14 @@ class Riffer::Evals::Result
   # Whether higher scores are better for this evaluator.
   attr_reader higher_is_better: bool
 
+  # Token usage for the judge call that produced this result, when the
+  # evaluator used an LLM. Nil for rule-based evaluators.
+  attr_reader token_usage: Riffer::Providers::TokenUsage?
+
   # Raises Riffer::ArgumentError if +score+ is not between 0.0 and 1.0.
   # --
-  # : (evaluator: singleton(Riffer::Evals::Evaluator), score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped], ?higher_is_better: bool) -> void
-  def initialize: (evaluator: singleton(Riffer::Evals::Evaluator), score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped], ?higher_is_better: bool) -> void
+  # : (evaluator: singleton(Riffer::Evals::Evaluator), score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped], ?higher_is_better: bool, ?token_usage: Riffer::Providers::TokenUsage?) -> void
+  def initialize: (evaluator: singleton(Riffer::Evals::Evaluator), score: Float, ?reason: String?, ?metadata: Hash[Symbol, untyped], ?higher_is_better: bool, ?token_usage: Riffer::Providers::TokenUsage?) -> void
 
   # Returns a hash representation of the result.
   #

data/sig/generated/riffer/evals/run_result.rbs

@@ -15,6 +15,20 @@ class Riffer::Evals::RunResult
   # : () -> Hash[singleton(Riffer::Evals::Evaluator), Float]
   def scores: () -> Hash[singleton(Riffer::Evals::Evaluator), Float]
 
+  # Returns the summed token usage the agents under test spent across every
+  # scenario, or nil when none reported usage.
+  #
+  # --
+  # : () -> Riffer::Providers::TokenUsage?
+  def token_usage: () -> Riffer::Providers::TokenUsage?
+
+  # Returns the summed token usage across every scenario's LLM-as-judge
+  # evaluators, or nil when none reported usage.
+  #
+  # --
+  # : () -> Riffer::Providers::TokenUsage?
+  def evaluator_token_usage: () -> Riffer::Providers::TokenUsage?
+
   # Returns a hash representation of the run result.
   #
   # --

data/sig/generated/riffer/evals/scenario_result.rbs

@@ -17,9 +17,12 @@ class Riffer::Evals::ScenarioResult
   # The full message history from the agent conversation.
   attr_reader messages: Array[Riffer::Messages::Base]
 
+  # Token usage the agent under test spent generating this scenario's output.
+  attr_reader token_usage: Riffer::Providers::TokenUsage?
+
   # --
-  # : (input: String, output: String, ground_truth: String?, results: Array[Riffer::Evals::Result], ?messages: Array[Riffer::Messages::Base]) -> void
-  def initialize: (input: String, output: String, ground_truth: String?, results: Array[Riffer::Evals::Result], ?messages: Array[Riffer::Messages::Base]) -> void
+  # : (input: String, output: String, ground_truth: String?, results: Array[Riffer::Evals::Result], ?messages: Array[Riffer::Messages::Base], ?token_usage: Riffer::Providers::TokenUsage?) -> void
+  def initialize: (input: String, output: String, ground_truth: String?, results: Array[Riffer::Evals::Result], ?messages: Array[Riffer::Messages::Base], ?token_usage: Riffer::Providers::TokenUsage?) -> void
 
   # Returns scores keyed by evaluator class.
   #
@@ -27,6 +30,13 @@ class Riffer::Evals::ScenarioResult
   # : () -> Hash[singleton(Riffer::Evals::Evaluator), Float]
   def scores: () -> Hash[singleton(Riffer::Evals::Evaluator), Float]
 
+  # Returns the summed token usage across this scenario's LLM-as-judge
+  # evaluators, or nil when none reported usage.
+  #
+  # --
+  # : () -> Riffer::Providers::TokenUsage?
+  def evaluator_token_usage: () -> Riffer::Providers::TokenUsage?
+
   # Returns a hash representation of the scenario result.
   #
   # --

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: riffer
 version: !ruby/object:Gem::Version
-  version: 0.38.1
+  version: 0.39.0
 platform: ruby
 authors:
 - Jake Bottrall