ruby_llm 2.0.0 → 2.1.0

data/lib/ruby_llm.rb

@@ -1,7 +1,6 @@
 # frozen_string_literal: true
 
 require 'base64'
-require 'event_stream_parser'
 require 'faraday'
 require 'faraday/multipart'
 require 'faraday/retry'
@@ -25,14 +24,21 @@ loader.inflector.inflect(
   'deepseek' => 'DeepSeek',
   'elevenlabs' => 'ElevenLabs',
   'gpustack' => 'GPUStack',
+  'hetzner' => 'Hetzner',
+  'http' => 'HTTP',
   'llm' => 'LLM',
+  'mcp' => 'MCP',
+  'oauth' => 'OAuth',
   'mistral' => 'Mistral',
   'ocr' => 'OCR',
   'openai' => 'OpenAI',
+  'open_telemetry' => 'OpenTelemetry',
   'openrouter' => 'OpenRouter',
   'pdf' => 'PDF',
   'perplexity' => 'Perplexity',
+  'rspec' => 'RSpec',
   'ruby_llm' => 'RubyLLM',
+  'typesafe' => 'TypeSafe',
   'vertexai' => 'VertexAI',
   'xai' => 'XAI'
 )
@@ -75,9 +81,14 @@ loader.setup
 #
 # Subclass Tool and implement +execute+ to give the model an application
 # action. Tool.requires_approval pauses execution for a human decision;
-# Chat#approve and Chat#deny record it. Chat#with_provider_tools enables
-# provider-executed tools such as web search, code execution, and remote MCP.
+# Chat#approve and Chat#deny record it. Tool#progress reports what a
+# running tool is doing to Chat#after_tool_progress as a Progress.
+# Chat#with_provider_tools enables provider-executed tools such as web
+# search, code execution, and remote MCP.
 # Their calls appear as ServerToolCall values, with Citation values for sources.
+# MCP is a Model Context Protocol client: describe a server to connect to
+# in an MCP class, and Chat#with_mcp gives the model its tools. RubyLLM.mcp
+# connects to one inline.
 #
 # Agent defines a reusable configuration with model, instructions, tools,
 # schema, and runtime inputs. Chat#ask_later, Chat#generate, Chat#run_tools,
@@ -132,6 +143,41 @@ loader.setup
 # returns a ResearchJob for polling and cancellation. Hosted agent identities
 # are selected separately from model IDs.
 #
+# == Evaluations
+#
+# Evaluation runs your application against a dataset of cases and grades each
+# result. Implement Evaluation#perform to return a value, Message, Chat, or
+# Agent. Without declared criteria, a model checks each result against the
+# case's +expected_output+. Evaluations can also use an Agent or a Judge as
+# their evaluator, and Ruby assertions. Datasets live under app/evals, named
+# after the class:
+#
+#   class SupportEvaluation < RubyLLM::Evaluation
+#     def perform(input)
+#       SupportAgent.new.ask(input)
+#     end
+#   end
+#   SupportEvaluation.run.passed?
+#
+# Evaluation.run returns an Evaluation::Report with each trial's verdicts,
+# tokens, and cost. The +ruby_llm:eval+ Rake task, Evaluation::RSpec, and
+# Evaluation::Minitest run the same cases from the command line and tests.
+#
+# == Typed judgments
+#
+# Judge defines reusable probability, choice, and score questions. Its +judge+
+# method evaluates supplied text, structured data, or images passed as +with:+
+# and returns a Judgment:
+#
+#   class Urgency < RubyLLM::Judge
+#     probability :urgent, "Does this need attention today?"
+#   end
+#   Urgency.judge("Please help today.").urgent.probability
+#
+# RubyLLM.judge accepts question definitions as a Hash. Probability, Choice,
+# and Score answers retain uncertainty; Judgment reports the model, tokens,
+# and cost. Questions share the supplied input and do not retain chat history.
+#
 # == Batches, usage, and configuration
 #
 # RubyLLM.batch submits staged chats or EmbeddingRequest objects for
@@ -142,6 +188,10 @@ loader.setup
 # Tokens and Cost report usage and pricing. Chat totals include retries
 # and attempts that produced no message. Provider-reported costs take
 # precedence over estimates; unknown usage and prices remain +nil+.
+# RubyLLM.with_usage_owner attributes the usage of operations to a record.
+# OpenTelemetry.enable enables optional tracing alongside existing subscribers.
+# Your application configures the OpenTelemetry SDK and exporters.
+#
 # RubyLLM.workflow groups instrumentation from ordinary Ruby code into
 # named Workflow steps.
 #
@@ -149,7 +199,8 @@ loader.setup
 # isolated settings for a request or tenant. Models finds, filters, and
 # describes the model catalog. Provider supplies endpoints, authentication,
 # and protocol selection; Protocol implements request and response formats.
-# Error subclasses normalize provider failures.
+# Error subclasses normalize provider failures, and Error#request_shape
+# describes a refused conversation request as a RequestShape.
 #
 # == Rails integration
 #
@@ -160,7 +211,8 @@ loader.setup
 # Approvals and cancellation survive requests and processes.
 #
 # Your application owns chats and messages; RubyLLM owns usage, tool calls,
-# models, and batches. Agent can create and reload your chat records through
+# models, batches, MCP credentials, and provider files. ActiveRecord::Record configures their shared database
+# connection. Agent can create and reload your chat records through
 # Agent.chat_model. Individual operations also work directly in Rails
 # services and jobs.
 module RubyLLM
@@ -208,6 +260,23 @@ module RubyLLM
       Workflow.new(name, id:, metadata:, config: config).run(&)
     end
 
+    # Attributes the usage of every operation inside the block to +owner+,
+    # such as a user or an account record, and returns the block's value.
+    # In Rails, the usage ledger stores the owner alongside any chat and
+    # message associations; everywhere, the +usage.ruby_llm+ event carries
+    # it as +owner+. An operation's own +owner:+ keyword wins, and nested blocks
+    # restore the outer owner when they end. Each fiber and thread keeps its
+    # own owner, and fibers and threads started inside the block inherit it.
+    #
+    #   RubyLLM.with_usage_owner(current_user) do
+    #     RubyLLM.transcribe("memo.m4a")
+    #     RubyLLM.paint("a paper boat in the rain")
+    #   end
+    #
+    def with_usage_owner(owner, &)
+      Accounting::Usage.with_owner(owner, &)
+    end
+
     # Creates a Chat conversation. Arguments are forwarded to Chat.new:
     # +model:+, +provider:+, +protocol:+, +assume_model_exists:+, and
     # +context:+. With no arguments, uses the configured default model.
@@ -221,6 +290,25 @@ module RubyLLM
       Chat.new(...)
     end
 
+    # Connects to a Model Context Protocol server without writing an MCP
+    # class. Pass +url:+ for a Streamable HTTP server, +command:+ for a
+    # local server that speaks over stdio, or +transport:+ and +name:+ for
+    # a server reached any other way, as MCP.transport describes. Also
+    # accepts +name:+, +bearer_token:+, +headers:+, +env:+, +directory:+,
+    # +timeout:+, +prefix:+, +input_requests:+, +log_level:+,
+    # +extensions:+, which takes a name or a Hash of names to the settings
+    # of MCP.extension, and +oauth:+, which takes +true+ or the options of
+    # MCP.oauth.
+    #
+    #   docs = RubyLLM.mcp(url: "https://learn.microsoft.com/api/mcp")
+    #   files = RubyLLM.mcp(command: ["npx", "-y", "@modelcontextprotocol/server-filesystem", "."])
+    #   RubyLLM.mcp(url: server.endpoint, prefix: "mcp_#{server.id}", oauth: { owner: server })
+    #
+    # Returns an MCP.
+    def mcp(...)
+      MCP.define(...).new
+    end
+
     # Counts the tokens +text+ would consume as a single user message,
     # without requesting a completion. Builds a minimal Chat and delegates
     # to Chat#count_tokens. Returns an Integer.
@@ -287,6 +375,17 @@ module RubyLLM
       Moderation.moderate(...)
     end
 
+    # Judges text, structured data, or images against typed questions and
+    # returns a Judgment. Accepts the same arguments as Judge.judge. Subclass
+    # Judge to define reusable questions with probability, choice, and score.
+    # Uses Configuration#default_judgment_model unless a model is supplied.
+    #
+    #   RubyLLM.judge("Please help today",
+    #     questions: { urgent: { type: :probability, instructions: "Is this urgent?" } })
+    def judge(...)
+      Judge.judge(...)
+    end
+
     # Runs a hosted research task and returns its report as a Message.
     # Requires explicit +provider:+ and +agent:+. See ResearchJob.research.
     def research(...)
@@ -480,12 +579,14 @@ RubyLLM::Provider.register :deepseek, RubyLLM::Providers::DeepSeek
 RubyLLM::Provider.register :elevenlabs, RubyLLM::Providers::ElevenLabs
 RubyLLM::Provider.register :gemini, RubyLLM::Providers::Gemini
 RubyLLM::Provider.register :gpustack, RubyLLM::Providers::GPUStack
+RubyLLM::Provider.register :hetzner, RubyLLM::Providers::Hetzner
 RubyLLM::Provider.register :mistral, RubyLLM::Providers::Mistral
 RubyLLM::Provider.register :ollama, RubyLLM::Providers::Ollama
 RubyLLM::Provider.register :ollama_cloud, RubyLLM::Providers::OllamaCloud
 RubyLLM::Provider.register :openai, RubyLLM::Providers::OpenAI
 RubyLLM::Provider.register :openrouter, RubyLLM::Providers::OpenRouter
 RubyLLM::Provider.register :perplexity, RubyLLM::Providers::Perplexity
+RubyLLM::Provider.register :typesafe, RubyLLM::Providers::TypeSafe
 RubyLLM::Provider.register :vertexai, RubyLLM::Providers::VertexAI
 RubyLLM::Provider.register :xai, RubyLLM::Providers::XAI
 

data/lib/tasks/ruby_llm/evaluations.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+require 'fileutils'
+
+module RubyLLM
+  module Tasks
+    module Evaluations # :nodoc:
+      module_function
+
+      def run(name = nil, only: nil)
+        files = discover(name)
+        repetitions = Integer(ENV.fetch('EVAL_REPETITIONS', '1'))
+        reports = files.map do |file, class_name|
+          require file.to_s
+          evaluation = Object.const_get(class_name)
+          raise ArgumentError, "#{class_name} must inherit from RubyLLM::Evaluation" unless evaluation < Evaluation
+
+          report = evaluation.run(only:, repetitions:)
+          path = Pathname.new(ENV.fetch('EVAL_OUTPUT', 'tmp/evaluations')).join("#{class_name.gsub('::', '/')}.json")
+          FileUtils.mkdir_p(path.dirname)
+          report.save(path)
+          puts report
+          puts "Report: #{path}"
+          report
+        end
+        abort 'Evaluations failed' unless reports.all?(&:passed?)
+      end
+
+      def discover(name)
+        root = Prompt.root.parent.join('evals')
+        inflector = Zeitwerk::Inflector.new
+        files = root.glob('**/*_evaluation.rb').sort.to_h do |file|
+          parts = file.relative_path_from(root).to_s.delete_suffix('.rb').split('/')
+          [file, parts.map { |part| inflector.camelize(part, file.to_s) }.join('::')]
+        end
+        files.select! { |_file, class_name| class_name == name } if name
+        raise ArgumentError, "No evaluations found in #{root}#{" matching #{name}" if name}" if files.empty?
+
+        files
+      end
+    end
+  end
+end

data/lib/tasks/ruby_llm.rake

@@ -1,16 +1,13 @@
 # frozen_string_literal: true
 
 namespace :ruby_llm do
-  namespace :upgrade do
-    %i[rollback resume finalize].each do |action|
-      desc "#{action.to_s.capitalize} a copy-mode RubyLLM upgrade with affected processes stopped"
-      task action => :environment do
-        require 'generators/ruby_llm/upgrade/upgrade_migration'
+  desc 'Run app/evals evaluations (optionally select an evaluation and case)'
+  task :eval, %i[evaluation case] do |_task, args|
+    Rake::Task[:environment].invoke if Rake::Task.task_defined?(:environment)
+    require 'ruby_llm'
+    require 'tasks/ruby_llm/evaluations'
 
-        RubyLLM::Generators::UpgradeMigration.for.public_send(action)
-        puts "RubyLLM copy upgrade: #{action} completed"
-      end
-    end
+    RubyLLM::Tasks::Evaluations.run(args[:evaluation], only: args[:case])
   end
 
   desc 'Load the selected model registry into the database'

data/skills/rubyllm/SKILL.md

@@ -1,14 +1,14 @@
 ---
 name: rubyllm
-description: Build and maintain Ruby or Rails applications with the RubyLLM AI framework. Use for chats, agents, tools, structured output, media generation, transcription, OCR, moderation, embeddings, reranking, Rails integration, and RubyLLM upgrades; not for contributing to the framework itself.
+description: Build and maintain Ruby or Rails applications with the RubyLLM AI framework. Use for chats, agents, tools, MCP servers, structured output, judgments, evaluations, media generation, transcription, OCR, moderation, embeddings, reranking, tracing, Rails integration, and RubyLLM upgrades; not for contributing to the framework itself.
 license: MIT
 metadata:
-  api_version: "2.0"
+  api_version: "2.1"
 ---
 
 # Build with RubyLLM
 
-Use the application's installed RubyLLM version and existing conventions. This skill describes the 2.0 API. Read `Gemfile.lock`, or inspect the installed gem before choosing an API:
+Use the application's installed RubyLLM version and existing conventions. This skill describes the 2.x API; features marked 2.1 need RubyLLM 2.1 or later. Read `Gemfile.lock`, or inspect the installed gem before choosing an API:
 
 ```bash
 bundle exec ruby -rruby_llm -e 'puts RubyLLM::VERSION'
@@ -19,7 +19,7 @@ For a 1.x application, use its version's documentation and keep the requested ch
 
 ## Choose the API for the task
 
-RubyLLM has two public API families: conversations use chats, messages, tools, agents, structured output, streaming, and loop control; individual operations use `paint`, `animate`, `speak`, `transcribe`, `ocr`, `moderate`, `embed`, and `rerank`. Individual operations return typed results without requiring a chat. Some stream or have an asynchronous lifecycle.
+RubyLLM has two public API families: conversations use chats, messages, tools, agents, structured output, streaming, and loop control; individual operations use `paint`, `animate`, `speak`, `transcribe`, `ocr`, `moderate`, `embed`, `rerank`, `judge`, `tokenize`, and `research`. Individual operations return typed results without requiring a chat. Some stream or have an asynchronous lifecycle.
 
 Providers supply service endpoints, authentication, catalogs, and protocol selection. Protocols translate requests and responses. Both API families use this integration layer and shared services for model selection, configuration, accounting, and instrumentation. Build against the public API and let RubyLLM handle the service formats.
 
@@ -27,7 +27,7 @@ Rails integration adds persistence, attachments, streaming UI support, jobs, and
 
 ## Find the relevant API
 
-The installed gem's public method signatures and RDoc describe the code the application actually runs. Use them when online documentation differs. Find the matching version through [RubyLLM's documentation](https://rubyllm.com/). During the 2.0 release-candidate period, the 2.0 guides and Markdown API reference are under [rubyllm.com/next/](https://rubyllm.com/next/).
+The installed gem's public method signatures and RDoc describe the code the application actually runs. Use them when online documentation differs. Find the matching version through [RubyLLM's documentation](https://rubyllm.com/). The root site documents the latest stable release. Unreleased changes on `main` are under [rubyllm.com/next/](https://rubyllm.com/next/), and 1.x under [rubyllm.com/v1/](https://rubyllm.com/v1/).
 
 Read the relevant page instead of loading the entire documentation site:
 
@@ -37,44 +37,65 @@ Read the relevant page instead of loading the entire documentation site:
 | Configure a provider or select a model | `configuration/`, `configuration-providers/`, `models/` |
 | Chat, stream, or attach files | `chat/`, `streaming/`, `attachments/` |
 | Declare tools or require approval | `tools/`, `tool-parameters/`, `tool-execution/` |
+| Use tools from MCP servers (2.1) | `mcp/` |
+| Use provider-hosted tools such as web search | `provider-tools/`, `citations/` |
 | Parse structured output | `structured-output/` |
 | Build reusable agents | `agents/`, `prompt-rendering/` |
+| Enable thinking or prompt caching | `thinking/`, `prompt-caching/` |
 | Generate images or video | `image-generation/`, `video-generation/` |
 | Generate speech or transcribe audio | `text-to-speech/`, `audio-transcription/` |
 | Extract document text or moderate content | `ocr/`, `moderation/` |
-| Embed text or rank search results | `embeddings/`, `rerank/` |
-| Persist chats, attach files, or stream with Hotwire | `rails/`, `rails-persistence/`, `rails-streaming/`, `rails-generators/` |
+| Embed text or rank search results | `embeddings/`, `rerank/`, `rag/` |
+| Classify, score, or decide with typed answers (2.1) | `judgments/` |
+| Measure answer quality against a dataset (2.1) | `evaluations/`, `evaluation-datasets/`, `evaluation-evaluators/`, `evaluation-running/`, `evaluation-examples/` |
+| Count tokens, upload provider files, or run hosted research | `tokenization/`, `files/`, `hosted-research/` |
+| Persist chats, attach files, or stream with Hotwire | `rails/`, `rails-persistence/`, `rails-streaming/`, `rails-generators/`, `rails-advanced-config/` |
 | Resume agents in background jobs | `durable-agents/` |
 | Track spend or handle failures | `cost-and-usage-tracking/`, `error-handling/` |
+| Observe calls, group workflows, or trace with OpenTelemetry | `instrumentation/`, `opentelemetry/` (2.1) |
 | Submit batches or drive the conversation loop | `batches/`, `agentic-workflows/` |
 | Upgrade an existing application | `upgrading/` |
 
 Markdown guides use the same name with `.md`, such as `chat.md`. The API reference starts at `api/RubyLLM.md`; class references include `api/RubyLLM/Chat.md` and `api/RubyLLM/Agent.md`. If fetching documentation is unavailable, inspect the installed source and identify any remaining uncertainty.
 
-## Use the 2.0 API consistently
+## Use the 2.x API consistently
 
 - Configure credentials through `RubyLLM.configure`, following the application's secret storage. Select an existing configured provider. Verify explicit model IDs with `RubyLLM.models.find(id, provider: ...)`; do not invent IDs or use `assume_model_exists` to hide a typo.
 - Start with `RubyLLM.chat` for a conversation. Set options with chainable `with_*` methods. Use `RubyLLM::Agent` when instructions, tools, or options should be reused. Ordinary Ruby methods and jobs compose workflows.
 - Use individual operations directly for media, document processing, moderation, embeddings, and reranking. Read their typed results and use `save` on generated images, video, and speech. Check each operation's provider, model, and credentials; a configured chat provider may not offer every operation. `animate` waits for completion; use `animate_later` when the application needs to manage the job.
 - `ask` runs the conversation and returns a message. `ask_later` stages input and returns the chat. For explicit loop control, use `generate`, `run_tools`, `step`, and `complete?`. Approval can park the loop before completion; inspect `awaiting_approval?` and resume after `approve` or `deny`.
 - Read text through `message.content` and structured output through `message.parsed`. Declare schemas with `Schematist::Schema` or the schema DSL. Do not assume `content` is a parsed Hash.
-- Subclass `RubyLLM::Tool`. Use `description`, `parameter`, or a `parameters` block, and keyword arguments on `execute`. Register tools with `with_tools`; configure choice and concurrency with `with_tool_options`. Keep application authorization in the tool or underlying service. Use `requires_approval` when the product needs a user's decision before execution.
+- Subclass `RubyLLM::Tool`. Use `description`, `parameter`, or a `parameters` block, and keyword arguments on `execute`. Register tools with `with_tools`; configure choice and concurrency with `with_tool_options`. Keep application authorization in the tool or underlying service. Use `requires_approval` when the product needs a user's decision before execution. With large tool sets, pass `defer: true` to `with_tools` or `with_mcp` (or declare `defer` on the class) so the model searches for tools instead of receiving them all (2.1).
 - Use `with_thinking`, `with_caching`, `with_compaction`, and `with_citations` without arguments to enable them and `false` to disable them. They reject `nil`. Read the relevant guide for supported options and model requirements.
+- Connect MCP servers by subclassing `RubyLLM::MCP` (`url` or `command`, then `only` and `requires_approval` to shape tools) and attach them with `chat.with_mcp(...)` or the agent `mcp` macro (2.1). Do not add a separate MCP client gem.
+- For a yes/no probability, a choice, or a score about application data, subclass `RubyLLM::Judge` and call `.judge(input)` instead of parsing free text from a chat (2.1). Backends are TypeSafe's Jev, OpenAI Decisions, and local decision models such as Clef with `provider: :ollama`.
+- Use `with_fallbacks` for alternate models, `with_provider_tools` for provider-hosted tools, and `progress` inside a long-running tool with `after_tool_progress` on the chat (2.1) to report it.
+- Use `RubyLLM.context` for per-tenant credentials or endpoints, and an agent `context` block when the configuration depends on the agent's inputs (2.1).
 - Use shared RubyLLM options when available. Reserve `provider_options` for provider-specific request fields. Do not add a second provider SDK for functionality RubyLLM already supplies.
-- Read tokens through `message.tokens.input`, `.output`, `.cache_read`, `.cache_write`, and `.thinking`; read cost through `message.cost.total`. Unknown cost is `nil`, not zero. The ledger includes individual attempts; a successful answer can include earlier failed attempts.
+- Read tokens through `message.tokens.input`, `.output`, `.cache_read`, `.cache_write`, and `.thinking`; read cost through `message.cost.total`. Unknown cost is `nil`, not zero. The ledger includes individual attempts; a successful answer can include earlier failed attempts. Attribute usage with `owner:` on individual operations or `RubyLLM.with_usage_owner` (2.1). Persisted chat usage keeps both its chat and owner associations.
+- Group related calls with `RubyLLM.workflow` and `workflow.step` for instrumentation. To send traces to an existing OpenTelemetry setup, call `RubyLLM::OpenTelemetry.enable` after configuring the SDK (2.1). RubyLLM never configures the SDK or exporters, and does not export prompts or responses.
 - RubyLLM enums are Symbols: `finish_reason == :stop`, thinking effort `:high`. Provider slugs and model IDs remain Strings.
 
 ## Integrate with Rails
 
 For a new integration, use `bin/rails generate ruby_llm:install`, review its migrations, and follow the application's migration workflow. Load packaged model data with `bin/rails ruby_llm:load_models`. `RubyLLM.models.refresh` performs a network refresh and persists the result.
 
-Applications own their chat and message models, declared with `acts_as_chat` and `acts_as_message`. RubyLLM owns the supporting model, tool-call, usage, and batch tables. Do not generate application `Model` or `ToolCall` classes from 1.x examples. Pass custom chat and message mappings to the generators when the application uses different names.
+Applications own their chat and message models, declared with `acts_as_chat` and `acts_as_message`. RubyLLM owns the supporting model, tool-call, usage, and batch tables, plus MCP credential and provider file tables from 2.1. Do not generate application `Model` or `ToolCall` classes from 1.x examples. Pass custom chat and message mappings to the generators when the application uses different names.
 
 Use the same chat API on persisted records. Persisted `ask` returns the application's message record. Use the existing job backend for background work; retrieve and configure agents through their documented persistence API when a job resumes in another process. Prompt templates live under `app/prompts` by convention.
 
 Pass Active Storage attachments through `with:` and retain the message attachment association set up by the install generator. For a Hotwire UI, use the Rails streaming guide or `ruby_llm:chat_ui` generator: persisted messages provide the targets for Turbo Streams. Individual operations work directly in Rails services and jobs with the same methods as plain Ruby.
 
-For an upgrade, read the complete matching upgrade guide before editing migrations. The 2.0 cutover requires preparation, backfill, finish, and application-specific reconciliation before affected activity resumes. Legacy-column cleanup is a later phase. Checkpoints make a stopped backfill resumable; retained columns do not make a 1.x code rollback safe. Rehearse against an isolated snapshot and respect the application's deployment and recovery process.
+For an upgrade, read the complete matching upgrade guide before editing migrations. Each release's upgrade covers only the changes since the previous release, so upgrade one release at a time; a 1.x application completes the 2.0 upgrade first. The 2.0 cutover requires preparation, backfill, finish, and application-specific reconciliation before affected activity resumes. Legacy-column cleanup is a later phase. Checkpoints make a stopped backfill resumable; retained columns do not make a 1.x code rollback safe. Rehearse against an isolated snapshot and respect the application's deployment and recovery process.
+
+## Evaluate AI behavior
+
+Specs check code; evaluations check how often the model gets the answer right (2.1). Put a `RubyLLM::Evaluation` subclass and its dataset in `app/evals`, named after the class (`support_evaluation.rb` and `support_evaluation.yml`). `perform(input)` runs the application and returns a string, Message, Chat, or Agent. Each case has `name`, `inputs`, and usually `expected_output`.
+
+- Without declared criteria, a model checks each answer against `expected_output`, which every case must then supply. `evaluation :name, "statement"` declares criteria and replaces that default; `evaluation :correctness` without a description keeps it.
+- `assertions` uses Minitest assertions with `output`, `tool_calls`, `messages`, `input`, `expected_output`, and `metadata`. Assertions do not turn off grading; `evaluator false` does. Outside Rails, assertions need `gem "minitest"`.
+- Choose the grader with `evaluator model: ...`, an Agent class, or a Judge class with `evaluation :name, minimum: 0.8`.
+- Run with `bin/rails "ruby_llm:eval[SupportEvaluation]"`, `SupportEvaluation.run`, or `evaluates SupportEvaluation` in RSpec or Minitest after extending `RubyLLM::Evaluation::RSpec` or `RubyLLM::Evaluation::Minitest`. Model grading costs money on every run, so keep graded evaluations out of the default test run unless the application records requests.
 
 ## Verify the application change