ruby_llm-contract 0.2.0

data/examples/README.md

@@ -0,0 +1,140 @@
+# Examples
+
+## 00_basics.rb — From zero to ruby_llm-contract
+
+Step-by-step tutorial covering every feature. Start here.
+
+| Step | Feature | What it shows |
+|------|---------|---------------|
+| 1 | Plain string prompt | Simplest case — `user "{input}"` and nothing else |
+| 2 | System + user | Separate instructions from data |
+| 3 | Rules + output_schema | Requirements as statements + declarative output structure |
+| 4 | Invariants | Custom business logic on top of schema |
+| 5 | Examples | Few-shot (example input/output pairs) |
+| 6 | Sections | Labeled context blocks (heredoc replacement, with before/after) |
+| 7 | Hash input | Multiple fields with auto-interpolation |
+| 8 | 2-arity invariants | Cross-validate output against input |
+| 9 | Context override | Per-run adapter and model switching |
+| 10 | StepResult | Full inspection: status, output, errors, trace |
+| 11 | Pipeline | Chain steps with fail-fast data threading |
+
+Every step has a corresponding test in `spec/integration/examples_00_basics_spec.rb`.
+
+## 01_classify_threads.rb — Thread classification
+
+Real-world before/after: classify Reddit threads as PROMO/FILLER/SKIP.
+Shows ID matching, enum validation, score consistency invariants.
+
+## 02_generate_comment.rb — Comment generation
+
+Real-world before/after: generate Reddit comments with persona.
+Shows sections, banned openings, link presence, length constraints, 2-arity invariants.
+
+## 03_target_audience.rb — Audience profiling
+
+Real-world before/after: generate target audience profiles.
+Shows cascade failure prevention, locale validation, structural invariants.
+
+## 04_real_llm.rb — Real LLM calls via ruby_llm
+
+Connect to real LLM providers (OpenAI, Anthropic, Google, etc.) using Adapters::RubyLLM.
+Shows configuration, model switching, temperature/max_tokens control, provider-agnostic steps.
+
+| Step | Feature | What it shows |
+|------|---------|---------------|
+| 1 | Configure ruby_llm | Set API keys for your provider |
+| 2 | Set RubyLLM adapter | Swap Test adapter for production |
+| 3 | Define a step | Identical to Test adapter — provider-agnostic |
+| 4 | Run with real LLM | Real call, real tokens, full contract enforcement |
+| 5 | Compare models | A/B test different models per call |
+| 6 | Generation params | Temperature, max_tokens forwarding |
+| 7 | Switch providers | Same step, different provider — just change model name |
+| 8 | Error handling | Contract enforcement with real LLM responses |
+| 9 | Full power | Every feature combined in AnalyzeTicket |
+| 10 | Pipeline | Chain steps with real LLM calls |
+
+**Requires:** `export OPENAI_API_KEY=sk-...` (or another provider key)
+
+## 05_output_schema.rb — Declarative output schema
+
+Replace manual invariants with a schema DSL (ruby_llm-schema).
+
+| Step | Feature | What it shows |
+|------|---------|---------------|
+| 1 | Before (invariants) | Manual enum, range, required checks |
+| 2 | After (schema) | Same constraints in declarative DSL |
+| 3 | Schema + invariants | Schema for structure, invariants for business logic |
+| 4 | Complex schema | Nested objects, arrays, constraints |
+| 5 | Provider-agnostic | Same schema works with Test and RubyLLM adapters |
+| 6 | Pipeline + schemas | Fully typed multi-step composition |
+
+## Running
+
+```bash
+# Test adapter — no API keys needed:
+ruby examples/00_basics.rb
+ruby examples/01_classify_threads.rb
+ruby examples/02_generate_comment.rb
+ruby examples/03_target_audience.rb
+ruby examples/05_output_schema.rb
+
+# Real LLM — requires API key:
+ruby examples/04_real_llm.rb
+```
+
+## 06_reddit_promo.rb — Real-world Reddit promo pipeline
+
+3-step pipeline from the reddit_promo_planner case study:
+
+| Step | Role | Invariants catch |
+|------|------|------------------|
+| 1 | TargetAudience | `locale: "USA"` instead of `"en"`, vague summary |
+| 2 | ClassifyThreads | PROMO with score 2, SKIP with score 8 |
+| 3 | GenerateComment | `{PRODUCT}` instead of URL, banned openings |
+
+Runs with test adapter by default. `REAL_LLM=1` for Ollama, `MODEL=gemma:latest` to pick model.
+
+## 07_keyword_extraction.rb — Keyword extraction with probability
+
+Extract up to 15 keywords from an article, each with relevance probability.
+
+| Feature | What it shows |
+|---------|---------------|
+| Array schema | `min_items: 1, max_items: 15` with nested objects |
+| Number range | `probability: 0.0–1.0` |
+| Sorting invariant | Schema can't express "sorted descending" |
+| Uniqueness invariant | Schema can't express "no duplicates" |
+| Cross-validation | Keywords must appear in source text (catches hallucination) |
+| Pipeline | Keywords → Related Topics |
+
+## 08_translation.rb — Translation pipeline with quality review
+
+3-step pipeline: extract segments → translate → review quality.
+
+| Step | LLM Skill | Invariants catch |
+|------|-----------|------------------|
+| Extract | Analysis | Duplicate keys, wrong target_lang |
+| Translate | Creative | Missing segments, too long, echoed back untranslated |
+| Review | Evaluation | Inconsistent counts, failed reviews without issues |
+
+## Running
+
+```bash
+# Test adapter — no API keys needed:
+ruby examples/00_basics.rb
+ruby examples/01_classify_threads.rb
+ruby examples/02_generate_comment.rb
+ruby examples/03_target_audience.rb
+ruby examples/05_output_schema.rb
+ruby examples/06_reddit_promo.rb
+ruby examples/07_keyword_extraction.rb
+ruby examples/08_translation.rb
+
+# Real LLM — requires Ollama or API key:
+ruby examples/04_real_llm.rb
+REAL_LLM=1 ruby examples/06_reddit_promo.rb
+REAL_LLM=1 MODEL=llama3.2:3b ruby examples/06_reddit_promo.rb
+```
+
+Examples 00–03, 05–06 use the test adapter by default — no API keys needed.
+Example 04 and 06 with `REAL_LLM=1` require Ollama or an API key.

data/lib/ruby_llm/contract/adapters/base.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Adapters
+      class Base
+        def call(messages:, **_options)
+          raise NotImplementedError, "Subclasses must implement #call"
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/adapters/response.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Adapters
+      class Response
+        attr_reader :content, :usage
+
+        def initialize(content:, usage: {})
+          @content = content
+          @usage = usage
+          freeze
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/adapters/ruby_llm.rb

@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+require "ruby_llm"
+
+module RubyLLM
+  module Contract
+    module Adapters
+      class RubyLLM < Base
+        def call(messages:, **options)
+          system_contents, conversation = partition_messages(messages)
+          conversation = fallback_conversation(system_contents, conversation)
+
+          chat = build_chat(options, system_contents)
+          add_history(chat, conversation[0..-2])
+
+          response = chat.ask(conversation.last&.fetch(:content, ""))
+          build_response(response)
+        end
+
+        # Maps option keys to the RubyLLM chat method and argument form.
+        CHAT_OPTION_METHODS = {
+          temperature: :with_temperature,
+          schema: :with_schema
+        }.freeze
+
+        private
+
+        # When prompt has only system/section/rule nodes and no user message,
+        # pop the last system message and use it as the user ask.
+        def fallback_conversation(system_contents, conversation)
+          return conversation unless conversation.empty?
+
+          content = system_contents.any? ? system_contents.pop : ""
+          [{ role: :user, content: content }]
+        end
+
+        def build_chat(options, system_contents)
+          chat = ::RubyLLM.chat(**chat_constructor_options(options))
+          chat.with_instructions(system_contents.join("\n\n")) if system_contents.any?
+          apply_chat_options(chat, options)
+          chat
+        end
+
+        def chat_constructor_options(options)
+          opts = { model: options[:model] }
+          opts[:provider] = options[:provider] if options[:provider]
+          opts[:assume_model_exists] = options[:assume_model_exists] if options[:assume_model_exists]
+          opts
+        end
+
+        def apply_chat_options(chat, options)
+          CHAT_OPTION_METHODS.each do |key, method_name|
+            chat.public_send(method_name, options[key]) if options[key]
+          end
+          chat.with_params(max_tokens: options[:max_tokens]) if options[:max_tokens]
+        end
+
+        def build_response(response)
+          content = response.content
+          content = content.to_s unless content.is_a?(Hash)
+
+          Response.new(
+            content: content,
+            usage: {
+              input_tokens: response.input_tokens || 0,
+              output_tokens: response.output_tokens || 0
+            }
+          )
+        end
+
+        def partition_messages(messages)
+          system_contents = []
+          conversation = []
+
+          messages.each do |msg|
+            if msg[:role] == :system
+              system_contents << msg[:content]
+            else
+              conversation << msg
+            end
+          end
+
+          [system_contents, conversation]
+        end
+
+        def add_history(chat, messages)
+          messages&.each do |msg|
+            chat.add_message(role: msg[:role], content: msg[:content])
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/adapters/test.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Adapters
+      class Test < Base
+        def initialize(response: nil, responses: nil)
+          super()
+          if responses
+            raise ArgumentError, "responses: must not be empty (use response: nil for nil content)" if responses.empty?
+
+            @responses = responses.map { |r| normalize_response(r) }
+            @index = 0
+          else
+            @response = normalize_response(response)
+          end
+        end
+
+        private
+
+        def normalize_response(response)
+          case response
+          when Hash, Array then response
+          when nil then ""
+          else response.to_s
+          end
+        end
+
+        public
+
+        def call(messages:, **_options) # rubocop:disable Lint/UnusedMethodArgument
+          content = if @responses
+                      c = @responses[@index] || @responses.last
+                      @index += 1
+                      c
+                    else
+                      @response
+                    end
+          Response.new(content: content, usage: { input_tokens: 0, output_tokens: 0 })
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/adapters.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require_relative "adapters/base"
+require_relative "adapters/response"
+require_relative "adapters/test"
+require_relative "adapters/ruby_llm"

data/lib/ruby_llm/contract/concerns/deep_symbolize.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Concerns
+      module DeepSymbolize
+        def deep_symbolize(obj)
+          case obj
+          when Hash then obj.transform_keys(&:to_sym).transform_values { |val| deep_symbolize(val) }
+          when Array then obj.map { |val| deep_symbolize(val) }
+          else obj
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/concerns/eval_host.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Concerns
+      module EvalHost
+        def define_eval(name, &)
+          @eval_definitions ||= {}
+          key = name.to_s
+
+          if @eval_definitions.key?(key) && !Thread.current[:ruby_llm_contract_reloading]
+            warn "[ruby_llm-contract] Redefining eval '#{key}' on #{self}. " \
+                 "This replaces the previous definition."
+          end
+
+          @eval_definitions[key] = Eval::EvalDefinition.new(key, step_class: self, &)
+          Contract.register_eval_host(self)
+          register_subclasses(self)
+        end
+
+        def clear_eval_definitions!
+          @eval_definitions = {}
+        end
+
+        def eval_names
+          all_eval_definitions.keys
+        end
+
+        def eval_defined?
+          !all_eval_definitions.empty?
+        end
+
+        def run_eval(name = nil, context: {})
+          if name
+            run_single_eval(name, context)
+          else
+            run_all_own_evals(context)
+          end
+        end
+
+        def compare_models(eval_name, models:, context: {})
+          reports = models.each_with_object({}) do |model, hash|
+            model_context = deep_dup_context(context).merge(model: model)
+            hash[model] = run_single_eval(eval_name, model_context)
+          end
+          Eval::ModelComparison.new(eval_name: eval_name, reports: reports)
+        end
+
+        private
+
+        def all_eval_definitions
+          inherited = if superclass.respond_to?(:all_eval_definitions, true)
+                        superclass.send(:all_eval_definitions)
+                      else
+                        {}
+                      end
+          own = defined?(@eval_definitions) ? @eval_definitions : {}
+          inherited.merge(own)
+        end
+
+        def run_single_eval(name, context)
+          defn = all_eval_definitions[name.to_s]
+          raise ArgumentError, "No eval '#{name}' defined. Available: #{all_eval_definitions.keys}" unless defn
+
+          effective_context = eval_context(defn, context)
+          Eval::Runner.run(step: self, dataset: defn.build_dataset, context: effective_context)
+        end
+
+        def run_all_own_evals(context)
+          all_eval_definitions.transform_values do |defn|
+            isolated_context = deep_dup_context(context)
+            effective_context = eval_context(defn, isolated_context)
+            Eval::Runner.run(step: self, dataset: defn.build_dataset, context: effective_context)
+          end
+        end
+
+        def eval_context(defn, context)
+          return context if context[:adapter]
+
+          sample_adapter = defn.build_adapter
+          return context unless sample_adapter
+
+          context.merge(adapter: sample_adapter)
+        end
+
+        def register_subclasses(klass)
+          if klass.respond_to?(:subclasses)
+            klass.subclasses.each do |sub|
+              Contract.register_eval_host(sub)
+              register_subclasses(sub)
+            end
+          else
+            ObjectSpace.each_object(Class) do |sub|
+              Contract.register_eval_host(sub) if sub < klass
+            end
+          end
+        end
+
+        def deep_dup_context(context)
+          context.transform_values do |v|
+            v.respond_to?(:dup) ? v.dup : v
+          rescue TypeError
+            v
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/concerns/trace_equality.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Concerns
+      module TraceEquality
+        def ==(other)
+          return to_h == other if other.is_a?(Hash)
+
+          other.is_a?(self.class) && to_h == other.to_h
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/concerns/usage_aggregator.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    module Concerns
+      module UsageAggregator
+        private
+
+        def extract_usage(trace_entry)
+          if trace_entry.respond_to?(:usage)
+            trace_entry.usage
+          elsif trace_entry.respond_to?(:[])
+            trace_entry[:usage]
+          end
+        end
+
+        def sum_tokens(traces)
+          traces.sum do |trace_entry|
+            usage = extract_usage(trace_entry)
+            next 0 unless usage.is_a?(Hash)
+
+            (usage[:input_tokens] || 0) + (usage[:output_tokens] || 0)
+          end
+        end
+
+        def aggregate_usage(traces)
+          input_total = 0
+          output_total = 0
+
+          traces.each do |trace_entry|
+            usage = extract_usage(trace_entry)
+            next unless usage.is_a?(Hash)
+
+            input_total += usage[:input_tokens] || 0
+            output_total += usage[:output_tokens] || 0
+          end
+
+          { input_tokens: input_total, output_tokens: output_total }
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/configuration.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    # Configuration for ruby_llm-contract.
+    #
+    # API keys should be configured directly via RubyLLM:
+    #   RubyLLM.configure { |c| c.openai_api_key = ENV["OPENAI_API_KEY"] }
+    #
+    # Then configure contract-specific options:
+    #   RubyLLM::Contract.configure { |c| c.default_model = "gpt-4.1-mini" }
+    class Configuration
+      attr_accessor :default_adapter, :default_model
+
+      def initialize
+        @default_adapter = nil
+        @default_model = nil
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/contract/definition.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    class Definition
+      attr_reader :parse_strategy, :invariants
+
+      def initialize(&block)
+        @parse_strategy = :text
+        @invariants = []
+        instance_eval(&block) if block
+        @invariants = @invariants.freeze
+        freeze
+      end
+
+      def parse(strategy)
+        @parse_strategy = strategy
+      end
+
+      def invariant(description, &block)
+        @invariants << Invariant.new(description, block)
+      end
+      alias validate invariant
+
+      def self.build(&)
+        new(&)
+      end
+
+      def self.merge(base, extra_invariants: [], parse_override: nil)
+        new do
+          parse(parse_override || base.parse_strategy)
+          (base.invariants + extra_invariants).each do |inv|
+            @invariants << inv
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ruby_llm/contract/contract/invariant.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Contract
+    class Invariant
+      attr_reader :description
+
+      def initialize(description, block)
+        @description = description
+        @block = block
+        freeze
+      end
+
+      def call(parsed_output, input: nil)
+        if @block.arity >= 2
+          @block.call(parsed_output, input)
+        else
+          @block.call(parsed_output)
+        end
+      end
+    end
+  end
+end