riffer 0.11.0 → 0.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f5f2d3838897b17320d17d3cc95b1db37ae8af023d6dcac8c5c9ce0fa5a6e847
-  data.tar.gz: e7e75acdd198f147747d06bcaaddeba27f3491bcc12edd2ac0e35f464d47e2f0
+  metadata.gz: 0cf5738c3a643db6e1626862456336f6b4bba78bbb1e0efa599ebf61e0b80ee5
+  data.tar.gz: 683586055701aa2ffb62dd88559ceac640b6e0221120dee6fa589ef3e44a6cb4
 SHA512:
-  metadata.gz: 557d945daf22b6a45cb838532a5df0c811d2ac73377b2a5c445efb0dc06a8a2e9c234d14691333d699c6d0dddda7299a5e3bb5a676e4ae72d32b1f631c9694b4
-  data.tar.gz: aa35cbff64ef8c11e72e76c3c931589e8163f751b96c066cfef74a5f7779fabb8891565c4d215b2ea9b29fca45fb991ebd49260462c203aa9b73328f39d913ca
+  metadata.gz: 2c3fed6e20f275474ac0e9388a9b5200eab9b899c32534c7ba67b2df5e804097774c35821045e1dfbdaf65db426f7d56a827cd86de2eb32cbd8b65cb7aa8d343
+  data.tar.gz: 3db7dd0aa5fde3a20f97bd462b634ef01d8221db5f9c1d6c71a6430d915dc17b2ddebdd625d38514731ba63d6803cab6e94f5f61205e69cbf8d8d47237a3ee42

data/.agents/code-style.md

@@ -8,10 +8,11 @@
 
 ## Required Header
 
-All Ruby files must include:
+All Ruby files in `lib/` must include:
 
 ```ruby
 # frozen_string_literal: true
+# rbs_inline: enabled
 ```
 
 ## Error Handling

data/.agents/rbs-inline.md

@@ -0,0 +1,123 @@
+# RBS Inline
+
+Type annotations are added directly in Ruby source files using [rbs-inline](https://github.com/soutaro/rbs-inline).
+
+## Magic Comment
+
+Every `lib/**/*.rb` file must include the `rbs_inline: enabled` comment on line 2:
+
+```ruby
+# frozen_string_literal: true
+# rbs_inline: enabled
+```
+
+## Annotation Syntax
+
+The **`#:`** prefix is used — standalone lines above methods (type signatures) or inline on the same line (attributes, constants).
+
+### Method Parameters and Return Types
+
+Use a single `#:` line above the method with the RBS method signature:
+
+```ruby
+#: (String, Integer) -> bool
+def valid?(name, age)
+```
+
+#### Parameter Mapping
+
+| Ruby param                   | RBS signature            |
+| ---------------------------- | ------------------------ |
+| `def foo(x)`                 | `(Type)`                 |
+| `def foo(x = nil)`           | `(?Type?)`               |
+| `def foo(x = val)`           | `(?Type)`                |
+| `def foo(x:)`                | `(x: Type)`              |
+| `def foo(x: nil)`            | `(?x: Type?)`            |
+| `def foo(x: val)`            | `(?x: Type)`             |
+| `def foo(*args)`             | `(*untyped)`             |
+| `def foo(**kwargs)`          | `(**untyped)`            |
+| `def foo(&block)` (required) | `() { (Type) -> void }`  |
+| `def foo(&block)` (optional) | `() ?{ (Type) -> void }` |
+| `def foo(...)`               | `(*untyped, **untyped)`  |
+
+#### Examples
+
+```ruby
+# No parameters
+#: () -> String
+def name
+
+# Positional parameters
+#: (String, Integer) -> bool
+def valid?(name, age)
+
+# Optional positional parameter
+#: (?String?) -> String
+def self.identifier(value = nil)
+
+# Required keyword parameters
+#: (input: String, output: String) -> Riffer::Evals::Result
+def evaluate(input:, output:)
+
+# Mixed keyword parameters (required + optional)
+#: (input: String, output: String, ?context: Hash[Symbol, untyped]?) -> Riffer::Evals::Result
+def evaluate(input:, output:, context: nil)
+
+# Positional + keyword parameters
+#: (String, ?tool_context: Hash[Symbol, untyped]?) -> String
+def generate(prompt, tool_context: nil)
+
+# Splat/double-splat
+#: (**untyped) -> void
+def initialize(**options)
+
+# Forward arguments
+#: (*untyped, **untyped) -> String
+def self.generate(...)
+
+# Block parameter (required)
+#: () { (Riffer::Messages::Base) -> void } -> self
+def on_message(&block)
+
+# Block parameter (optional)
+#: () ?{ (Riffer::Config) -> void } -> void
+def configure(&block)
+```
+
+### Attributes
+
+```ruby
+attr_reader :name #: String
+attr_reader :items #: Array[String]
+```
+
+### Constants
+
+```ruby
+VERSION = "1.0.0" #: String
+DEFAULTS = {}.freeze #: Hash[Symbol, untyped]
+```
+
+## Common Type Patterns
+
+| Pattern                   | Meaning                     |
+| ------------------------- | --------------------------- |
+| `String?`                 | Optional (String or nil)    |
+| `(String \| Integer)`     | Union type                  |
+| `Array[String]`           | Typed array                 |
+| `Hash[Symbol, untyped]`   | Typed hash                  |
+| `^(String) -> void`       | Block/proc type             |
+| `singleton(Riffer::Tool)` | Class object (not instance) |
+| `bool`                    | Boolean (true or false)     |
+| `untyped`                 | Any type                    |
+| `void`                    | No meaningful return        |
+
+## Workflow
+
+After changing type annotations:
+
+1. Run `bundle exec rake rbs:generate` to regenerate `sig/generated/` files
+2. Commit both the source changes and the generated `.rbs` files
+3. CI checks for drift between source annotations and committed `.rbs` files
+
+Use `bundle exec rake rbs:watch` during development to auto-regenerate on file changes.

data/.agents/rdoc.md

@@ -1,24 +1,30 @@
 # RDoc Documentation
 
-Use pure RDoc comments for public APIs (not YARD).
+Use RDoc prose comments for public API descriptions and RBS inline annotations for types.
 
-## Parameters
+## Parameters and Return Types
 
-Use definition list syntax (`::`):
+Describe parameters in the RDoc prose comment. Use a single `#:` line for the RBS method signature (see [rbs-inline.md](rbs-inline.md) for the full type annotation syntax):
 
 ```ruby
 # Creates a new agent.
 #
-# name:: String - the agent name
-# options:: Hash - optional configuration
+# +name+ - the agent name.
+# +options+ - optional configuration.
+#
+#: (String, ?options: Hash[Symbol, untyped]) -> void
+def initialize(name, options: {})
 ```
 
-## Return Values
+## Attributes and Constants
 
-Document with prose:
+Use `#:` inline syntax (on the same line) for attribute and constant types:
 
 ```ruby
-# Returns String - the agent identifier.
+# The agent name.
+attr_reader :name #: String
+
+DEFAULT_TIMEOUT = 10 #: Integer
 ```
 
 ## Exceptions

data/.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "0.11.0"
+  ".": "0.12.0"
 }

data/AGENTS.md

@@ -16,6 +16,7 @@ Ruby gem framework for building AI-powered agents with LLM provider adapters.
 - [Code Style](.agents/code-style.md) - StandardRB and comment conventions
 - [RDoc](.agents/rdoc.md) - Documentation format for public APIs
 - [Providers](.agents/providers.md) - Adding new LLM provider adapters
+- [RBS Inline](.agents/rbs-inline.md) - Type annotations with rbs-inline
 
 ## Commands
 
@@ -25,4 +26,6 @@ Ruby gem framework for building AI-powered agents with LLM provider adapters.
 | `bundle exec rake test` | Run tests only |
 | `bundle exec rake standard` | Check code style |
 | `bundle exec rake standard:fix` | Auto-fix style issues |
+| `bundle exec rake rbs:generate` | Generate RBS type signatures |
+| `bundle exec rake rbs:watch` | Watch and regenerate RBS files |
 | `bin/console` | Interactive console |

data/CHANGELOG.md

@@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.12.0](https://github.com/janeapp/riffer/compare/riffer/v0.11.0...riffer/v0.12.0) (2026-02-11)
+
+
+### ⚠ BREAKING CHANGES
+
+* Agent#generate now returns Riffer::Agent::Response instead of String. Use response.content or response.to_s for the text.
+
+### Features
+
+* add Claude Code Review GitHub Action ([#108](https://github.com/janeapp/riffer/issues/108)) ([f4b281c](https://github.com/janeapp/riffer/commit/f4b281c43e6ad50430c38323bcb876b60efc994a))
+* add evals primitive for LLM-as-judge evaluations ([#101](https://github.com/janeapp/riffer/issues/101)) ([8fd7b36](https://github.com/janeapp/riffer/commit/8fd7b369f2bd0236ea4c7d30cc12e71b960211dd))
+* add guardrails primitive for input/output processing ([#100](https://github.com/janeapp/riffer/issues/100)) ([48d8bad](https://github.com/janeapp/riffer/commit/48d8badce98c0bf9110bafebd3097e25f46c8444))
+* add inline RBS type annotations with Steep type checking ([#103](https://github.com/janeapp/riffer/issues/103)) ([02ae559](https://github.com/janeapp/riffer/commit/02ae559fa580ef4353bd969f2e50e056ab538e2d))
+
+
+### Bug Fixes
+
+* correct RBS inline annotations and remove ivar declarations ([#109](https://github.com/janeapp/riffer/issues/109)) ([d59076d](https://github.com/janeapp/riffer/commit/d59076d40b88f581b51ddbb9ee3d50ed57e84451))
+
+
+### Miscellaneous Chores
+
+* set next version ([#111](https://github.com/janeapp/riffer/issues/111)) ([faf41b9](https://github.com/janeapp/riffer/commit/faf41b92032e302c3f0d2d06ab93140137c1b199))
+
 ## [0.11.0](https://github.com/janeapp/riffer/compare/riffer/v0.10.0...riffer/v0.11.0) (2026-02-04)
 
 

data/CLAUDE.md

@@ -0,0 +1 @@
+AGENTS.md

data/Guardfile

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+guard :shell do
+  watch(%r{^lib/.*\.rb$}) do
+    system("bundle exec rake rbs:generate")
+  end
+end

data/Rakefile

@@ -23,4 +23,26 @@ end
 
 task docs: :rdoc
 
-task default: %i[test standard]
+namespace :rbs do
+  desc "Generate RBS type signatures from inline annotations"
+  task :generate do
+    sh "bundle exec rbs-inline --output sig/generated lib"
+  end
+
+  desc "Watch lib/ for changes and regenerate RBS files"
+  task :watch do
+    require "guard"
+    require "guard/commander"
+
+    Guard.start(no_interactions: true)
+  end
+end
+
+namespace :steep do
+  desc "Run Steep type checker"
+  task :check do
+    sh "bundle exec steep check"
+  end
+end
+
+task default: %i[test standard steep:check]

data/Steepfile

@@ -0,0 +1,14 @@
+D = Steep::Diagnostic
+
+target :lib do
+  signature "sig/generated"
+
+  check "lib"
+
+  library "logger"
+  library "anthropic"
+  library "openai"
+  library "aws-sdk-bedrockruntime"
+
+  configure_code_diagnostics(D::Ruby.lenient)
+end

data/docs/03_AGENTS.md

@@ -100,15 +100,38 @@ class MyAgent < Riffer::Agent
 end
 ```
 
+### guardrail
+
+Registers guardrails for pre/post processing of messages. Pass the guardrail class and any options:
+
+```ruby
+class MyAgent < Riffer::Agent
+  model 'openai/gpt-4o'
+
+  # Input-only guardrail
+  guardrail :before, with: InputValidator
+
+  # Output-only guardrail
+  guardrail :after, with: ResponseFilter
+
+  # Both input and output, with options
+  guardrail :around, with: Riffer::Guardrails::MaxLength, max: 1000
+end
+```
+
+See [Guardrails](09_GUARDRAILS.md) for detailed documentation.
+
 ## Instance Methods
 
 ### generate
 
-Generates a response synchronously:
+Generates a response synchronously. Returns a `Riffer::Agent::Response` object:
 
 ```ruby
 # Class method (recommended for simple calls)
 response = MyAgent.generate('Hello')
+puts response.content     # Access the response text
+puts response.blocked?    # Check if guardrail blocked (always false without guardrails)
 
 # Instance method (when you need message history or callbacks)
 agent = MyAgent.new

data/docs/06_STREAM_EVENTS.md

@@ -109,6 +109,40 @@ event.role     # => "assistant"
 event.content  # => "Let me think about this step by step..."
 ```
 
+### GuardrailTripwire
+
+Emitted when a guardrail blocks execution during streaming:
+
+```ruby
+agent.stream("Hello").each do |event|
+  case event
+  when Riffer::StreamEvents::GuardrailTripwire
+    puts "Blocked by: #{event.guardrail_id}"
+    puts "Reason: #{event.reason}"
+    puts "Phase: #{event.phase}"  # :before or :after
+  end
+end
+```
+
+See [Guardrails](09_GUARDRAILS.md) for more information.
+
+### GuardrailModification
+
+Emitted when a guardrail transforms data during streaming:
+
+```ruby
+agent.stream("Hello").each do |event|
+  case event
+  when Riffer::StreamEvents::GuardrailModification
+    puts "Modified by: #{event.guardrail_id}"
+    puts "Phase: #{event.phase}"              # :before or :after
+    puts "Changed: #{event.message_indices}"  # Array of affected indices
+  end
+end
+```
+
+See [Guardrails](09_GUARDRAILS.md) for more information.
+
 ### TokenUsageDone
 
 Emitted when token usage data is available at the end of a response:

data/docs/08_EVALS.md

@@ -0,0 +1,316 @@
+# Evals
+
+Evals let you measure the quality of agent outputs using LLM-as-judge evaluations.
+
+## Overview
+
+Riffer Evals provides a framework for evaluating agent responses against configurable quality metrics. It uses an LLM-as-judge approach where a separate model evaluates the outputs of your agents.
+
+Key concepts:
+
+- **Evaluators** - Classes that evaluate input/output pairs and return scores
+- **Metrics** - Evaluator configurations with pass/fail thresholds
+- **Profiles** - Collections of metrics that can be included in agents
+- **Results** - Individual evaluation scores and aggregate pass/fail status
+
+## Quick Start
+
+```ruby
+# 1. Configure the judge model
+Riffer.config.evals.judge_model = "anthropic/claude-opus-4-5-20251101"
+
+# 2. Define an eval profile
+module QualityEvals
+  include Riffer::Evals::Profile
+
+  ai_evals do
+    metric :answer_relevancy, min: 0.85
+  end
+end
+
+# 3. Include in your agent
+class MyAgent < Riffer::Agent
+  include QualityEvals
+  model "anthropic/claude-haiku-4-5-20251001"
+  instructions "You are a helpful assistant."
+end
+
+# 4. Run evals
+result = MyAgent.run_eval(input: "What is Ruby?")
+result.passed?         # => true/false
+result.aggregate_score # => 0.91
+```
+
+## Configuration
+
+Before using evals, configure the judge model:
+
+```ruby
+Riffer.config.evals.judge_model = "anthropic/claude-opus-4-5-20251101"
+```
+
+The judge model is the LLM that evaluates agent outputs. You can use any configured provider.
+
+## Built-in Evaluators
+
+### answer_relevancy
+
+Evaluates how well a response addresses the input question.
+
+- **higher_is_better**: true
+- **Score range**: 0.0 to 1.0
+- **1.0**: Perfectly relevant, directly addresses the question
+- **0.7-0.9**: Mostly relevant with minor tangents
+- **0.4-0.6**: Partially relevant, some off-topic content
+- **0.1-0.3**: Mostly irrelevant
+- **0.0**: Completely irrelevant
+
+```ruby
+ai_evals do
+  metric :answer_relevancy, min: 0.85
+end
+```
+
+## Eval Profiles
+
+Eval profiles define which evaluators to run and their pass/fail thresholds.
+
+### Defining a Profile
+
+```ruby
+module QualityEvals
+  include Riffer::Evals::Profile
+
+  ai_evals do
+    metric :answer_relevancy, min: 0.85
+  end
+end
+```
+
+### Metric Options
+
+- `min` - Minimum score to pass (for higher_is_better evaluators)
+- `max` - Maximum score to pass (for lower_is_better evaluators)
+- `weight` - Weight for aggregate scoring (default: 1.0)
+
+```ruby
+ai_evals do
+  metric :answer_relevancy, min: 0.85, weight: 2.0  # Weighted more heavily
+end
+```
+
+### Including in Agents
+
+```ruby
+class MyAgent < Riffer::Agent
+  include QualityEvals
+  model "anthropic/claude-haiku-4-5-20251001"
+end
+```
+
+## Running Evals
+
+Once a profile is included, call `.eval` on the agent class:
+
+```ruby
+result = MyAgent.run_eval(
+  input: "What is the capital of France?",
+  context: { ground_truth: "Paris" }  # Optional context
+)
+```
+
+### RunResult Object
+
+The eval method returns a `Riffer::Evals::RunResult`:
+
+```ruby
+result.passed?          # => true if all metrics pass thresholds
+result.aggregate_score  # => Weighted average of normalized scores (0.0-1.0)
+result.failures         # => Array of Result objects that failed
+result.results          # => Array of all Result objects
+result.input            # => The input that was evaluated
+result.output           # => The agent's output
+result.to_h             # => Hash representation
+```
+
+### Result Object
+
+Individual evaluation results:
+
+```ruby
+result.results.first.evaluator       # => "answer_relevancy"
+result.results.first.score           # => 0.92
+result.results.first.reason          # => "The response directly addresses..."
+result.results.first.higher_is_better # => true
+```
+
+## Defining Custom Evaluators
+
+Create evaluators by subclassing `Riffer::Evals::Evaluator`:
+
+```ruby
+# app/evals/medical_accuracy_evaluator.rb
+class MedicalAccuracyEvaluator < Riffer::Evals::Evaluator
+  identifier "medical_accuracy"
+  description "Evaluates medical information accuracy"
+  higher_is_better true
+  judge_model "anthropic/claude-opus-4-5-20251101"  # Optional override
+
+  SYSTEM_PROMPT = <<~PROMPT
+    You are an evaluation assistant that assesses medical accuracy.
+
+    Use the evaluation tool to submit your score (0.0-1.0) and reasoning.
+  PROMPT
+
+  def evaluate(input:, output:, context: nil)
+    user_prompt = <<~PROMPT
+      Question: #{input}
+      Response: #{output}
+      Ground truth: #{context[:ground_truth]}
+    PROMPT
+
+    evaluation = judge.evaluate(
+      system_prompt: SYSTEM_PROMPT,
+      user_prompt: user_prompt
+    )
+
+    result(score: evaluation[:score], reason: evaluation[:reason])
+  end
+end
+```
+
+### Registering Custom Evaluators
+
+Register custom evaluators in your app initialization. Built-in evaluators are always available.
+
+```ruby
+# config/initializers/riffer.rb
+Riffer::Evals::Evaluators::Repository.register(:medical_accuracy, MedicalAccuracyEvaluator)
+```
+
+### Evaluator DSL
+
+Class methods:
+
+- `identifier(value)` - Set the evaluator identifier (defaults to snake_case class name)
+- `description(value)` - Human-readable description
+- `higher_is_better(value)` - Whether higher scores are better (default: true)
+- `judge_model(value)` - Override the global judge model
+
+Instance methods:
+
+- `evaluate(input:, output:, context:)` - Must be implemented, returns a Result
+- `judge` - Returns a Judge instance for LLM-as-judge calls
+- `result(score:, reason:, metadata:)` - Helper to build Result objects
+
+### Judge Options
+
+The `judge.evaluate` method accepts either `system_prompt:` and `user_prompt:` or a `messages:` array:
+
+```ruby
+# Using system_prompt and user_prompt
+evaluation = judge.evaluate(
+  system_prompt: "You are a judge.",
+  user_prompt: "Evaluate this response."
+)
+
+# Using messages array (for more control)
+evaluation = judge.evaluate(
+  messages: [
+    { role: "system", content: "You are a judge." },
+    { role: "user", content: "Evaluate this response." }
+  ]
+)
+```
+
+The Judge uses tool calling internally to get structured output. An `evaluation` tool with `score` (Float) and `reason` (String) parameters is automatically provided to the judge model, so your prompts should instruct the model to use the evaluation tool rather than respond with raw JSON.
+
+### Rule-Based Evaluators
+
+Evaluators don't have to use LLM-as-judge:
+
+```ruby
+class LengthEvaluator < Riffer::Evals::Evaluator
+  identifier "response_length"
+  description "Checks response is within expected length"
+  higher_is_better true
+
+  def evaluate(input:, output:, context: nil)
+    min_length = context&.dig(:min_length) || 50
+    max_length = context&.dig(:max_length) || 500
+
+    length = output.length
+
+    if length < min_length
+      score = length.to_f / min_length
+      reason = "Response too short (#{length} < #{min_length})"
+    elsif length > max_length
+      score = max_length.to_f / length
+      reason = "Response too long (#{length} > #{max_length})"
+    else
+      score = 1.0
+      reason = "Response length is appropriate"
+    end
+
+    result(score: score, reason: reason)
+  end
+end
+```
+
+## Aggregate Scoring
+
+The `aggregate_score` normalizes all scores so higher is always better:
+
+- For `higher_is_better` evaluators: score is used directly
+- For `lower_is_better` evaluators: score is inverted (1.0 - score)
+
+Scores are then weighted:
+
+```ruby
+# With weights: relevancy=2.0, toxicity=1.0
+# relevancy score: 0.9 (higher_is_better)
+# toxicity score: 0.1 (lower_is_better)
+
+# Normalized: relevancy=0.9, toxicity=0.9 (1.0 - 0.1)
+# Weighted average: (0.9 * 2.0 + 0.9 * 1.0) / (2.0 + 1.0) = 0.9
+```
+
+## Example: Full Integration
+
+```ruby
+# config/initializers/riffer.rb
+Riffer.configure do |config|
+  config.anthropic.api_key = ENV["ANTHROPIC_API_KEY"]
+  config.evals.judge_model = "anthropic/claude-opus-4-5-20251101"
+end
+
+# app/evals/quality_evals.rb
+module QualityEvals
+  include Riffer::Evals::Profile
+
+  ai_evals do
+    metric :answer_relevancy, min: 0.85, weight: 2.0
+  end
+end
+
+# app/agents/support_agent.rb
+class SupportAgent < Riffer::Agent
+  include QualityEvals
+
+  model "anthropic/claude-opus-4-5-20251101"
+  instructions "You are a helpful customer support agent."
+end
+
+# test/agents/support_agent_test.rb
+class SupportAgentTest < Minitest::Test
+  def test_response_quality
+    result = SupportAgent.run_eval
+      input: "How do I reset my password?",
+      context: {}
+    )
+
+    assert result.passed?, "Expected eval to pass: #{result.failures.map(&:reason)}"
+    assert result.aggregate_score >= 0.85
+  end
+end
+```