dspy 0.30.0 → 0.31.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5e59fde98eb918cca54822bc6af704100c51b71081d8ac77e17e8df3cddcb016
-  data.tar.gz: f8ff81ef1873fc1bffac8a209ddb4172586295e6ef1f46d6bb598045148fc3eb
+  metadata.gz: 972e09f00d8d5417d5c1af255eb01503fc33ab370264345b4c7de880b4f99fda
+  data.tar.gz: 21f6f7952a9caaf8398a24a69516147bf68c88e510b88aa2f45239786bbfd31b
 SHA512:
-  metadata.gz: dc880712e317a8e88c188527b5e1ff906da8c03fe1cbc4251e265bd87c141e52a16b4498f2f9316c33543f3428fb3067fd80a0728ce33de81575f111b93d4553
-  data.tar.gz: 7d1899b7e8a61b5d2c80fb8f237085171e9b64407f99135b22c341076e11867ee0c323d15c67c5428337606c869702ec9e6ada13edf27c38a6a40d8c83ef4f3b
+  metadata.gz: 780f786797df9d50950c1526296c8bc7a0db87dab29078e14de2d32a0ec608ae30585963a4dda3a5261b60530b3dd59e0b9e5915eab6c8d60360c4d1b6e1d8af
+  data.tar.gz: 3fb5d69bb58d9b57905a6f9985ecdd5d792be0aa91fb5e980f437f5d8887144564697d1151ed9bb8ec742b4ae4c1efff5303b53d937603ac7719959ffb11cfd9

data/README.md

@@ -4,6 +4,7 @@
 [![Total Downloads](https://img.shields.io/gem/dt/dspy)](https://rubygems.org/gems/dspy)
 [![Build Status](https://img.shields.io/github/actions/workflow/status/vicentereig/dspy.rb/ruby.yml?branch=main&label=build)](https://github.com/vicentereig/dspy.rb/actions/workflows/ruby.yml)
 [![Documentation](https://img.shields.io/badge/docs-vicentereig.github.io%2Fdspy.rb-blue)](https://vicentereig.github.io/dspy.rb/)
+[![Discord](https://img.shields.io/discord/1161519468141355160?label=discord&logo=discord&logoColor=white)](https://discord.gg/zWBhrMqn)
 
 > [!NOTE]
 > The core Prompt Engineering Framework is production-ready with
@@ -17,18 +18,13 @@
 
 **Build reliable LLM applications in idiomatic Ruby using composable, type-safe modules.**
 
-The Ruby framework for programming with large language models. DSPy.rb brings structured LLM programming to Ruby developers, programmatic Prompt Engineering and Context Engineering. 
-Instead of wrestling with prompt strings and parsing responses, you define typed signatures using idiomatic Ruby to compose and decompose AI Worklows and AI Agents.
+DSPy.rb is the Ruby-first surgical port of Stanford's [DSPy framework](https://github.com/stanfordnlp/dspy). It delivers structured LLM programming, prompt engineering, and context engineering in the language we love. Instead of wrestling with brittle prompt strings, you define typed signatures in idiomatic Ruby and compose workflows and agents that actually behave.
 
-**Prompts are the just Functions.** Traditional prompting is like writing code with string concatenation: it works until it doesn't. DSPy.rb brings you 
-the programming approach pioneered by [dspy.ai](https://dspy.ai/): instead of crafting fragile prompts, you define modular 
-signatures and let the framework handle the messy details.
+**Prompts are just functions.** Traditional prompting is like writing code with string concatenation: it works until it doesn't. DSPy.rb brings you the programming approach pioneered by [dspy.ai](https://dspy.ai/): define modular signatures and let the framework deal with the messy bits.
 
-DSPy.rb is an idiomatic Ruby surgical port of Stanford's [DSPy framework](https://github.com/stanfordnlp/dspy). While implementing 
-the core concepts of signatures, predictors, and the main optimization algorithms from the original Python library, DSPy.rb embraces Ruby 
-conventions and adds Ruby-specific innovations like Sorbet-base Typed system, ReAct loops, and production-ready integrations like non-blocking Open Telemetry Instrumentation.
+While we implement the same signatures, predictors, and optimization algorithms as the original library, DSPy.rb leans hard into Ruby conventions with Sorbet-based typing, ReAct loops, and production-ready integrations like non-blocking OpenTelemetry instrumentation.
 
-**What you get?** Ruby LLM applications that actually scale and don't break when you sneeze.
+**What you get?** Ruby LLM applications that scale and don't break when you sneeze.
 
 Check the [examples](examples/) and take them for a spin!
 
@@ -47,28 +43,12 @@ and
 bundle install
 ```
 
-### Optional Sibling Gems
-
-DSPy.rb ships multiple gems from this monorepo so you only install what you need. Add these alongside `dspy`:
-
-| Gem | Description | Status |
-| --- | --- | --- |
-| `dspy-schema` | Exposes `DSPy::TypeSystem::SorbetJsonSchema` for downstream reuse. | **Stable** (v1.0.0) |
-| `dspy-code_act` | Think-Code-Observe agents that synthesize and execute Ruby safely. | Preview (0.x) |
-| `dspy-datasets` | Dataset helpers plus Parquet/Polars tooling for richer evaluation corpora. | Preview (0.x) |
-| `dspy-evals` | High-throughput evaluation harness with metrics, callbacks, and regression fixtures. | Preview (0.x) |
-| `dspy-miprov2` | Bayesian optimization + Gaussian Process backend for the MIPROv2 teleprompter. | Preview (0.x) |
-| `dspy-gepa` | `DSPy::Teleprompt::GEPA`, reflection loops, experiment tracking, telemetry adapters. | Preview (mirrors `dspy` version) |
-| `gepa` | GEPA optimizer core (Pareto engine, telemetry, reflective proposer). | Preview (mirrors `dspy` version) |
-| `dspy-o11y` | Core observability APIs: `DSPy::Observability`, async span processor, observation types. | **Stable** (v1.0.0) |
-| `dspy-o11y-langfuse` | Auto-configures DSPy observability to stream spans to Langfuse via OTLP. | **Stable** (v1.0.0) |
-
-Set the matching `DSPY_WITH_*` environment variables (see `Gemfile`) to include or exclude each sibling gem when running Bundler locally (for example `DSPY_WITH_GEPA=1` or `DSPY_WITH_O11Y_LANGFUSE=1`). Refer to `docs/core-concepts/dependency-tree.md` for the full dependency map and roadmap.
 ### Your First Reliable Predictor
 
 ```ruby
+require 'dspy'
 
-# Configure DSPy globablly to use your fave LLM - you can override this on an instance levle. 
+# Configure DSPy globally to use your fave LLM (you can override per predictor).
 DSPy.configure do |c|
   c.lm = DSPy::LM.new('openai/gpt-4o-mini',
                       api_key: ENV['OPENAI_API_KEY'],
@@ -99,7 +79,7 @@ class Classify < DSPy::Signature
   end
 end
 
-# Wire it to the simplest prompting technique - a Predictn.
+# Wire it to the simplest prompting technique: a prediction loop.
 classify = DSPy::Predict.new(Classify)
 # it may raise an error if you mess the inputs or your LLM messes the outputs.
 result = classify.call(sentence: "This book was super fun to read!")
@@ -108,6 +88,37 @@ puts result.sentiment    # => #<Sentiment::Positive>
 puts result.confidence   # => 0.85
 ```
 
+Save this as `examples/first_predictor.rb` and run it with:
+
+```bash
+bundle exec ruby examples/first_predictor.rb
+```
+
+### Sibling Gems
+
+DSPy.rb ships multiple gems from this monorepo so you can opt into features with heavier dependency trees (e.g., datasets pull in Polars/Arrow, MIPROv2 requires `numo-*` BLAS bindings) only when you need them. Add these alongside `dspy`:
+
+| Gem | Description | Status |
+| --- | --- | --- |
+| `dspy-schema` | Exposes `DSPy::TypeSystem::SorbetJsonSchema` for downstream reuse. (Still required by the core `dspy` gem; extraction lets other projects depend on it directly.) | **Stable** (v1.0.0) |
+| `dspy-openai` | Packages the OpenAI/OpenRouter/Ollama adapters plus the official SDK guardrails. Install whenever you call `openai/*`, `openrouter/*`, or `ollama/*`. [Adapter README](https://github.com/vicentereig/dspy.rb/blob/main/lib/dspy/openai/README.md) | **Stable** (v1.0.0) |
+| `dspy-anthropic` | Claude adapters, streaming, and structured-output helpers behind the official `anthropic` SDK. [Adapter README](https://github.com/vicentereig/dspy.rb/blob/main/lib/dspy/anthropic/README.md) | **Stable** (v1.0.0) |
+| `dspy-gemini` | Gemini adapters with multimodal + tool-call support via `gemini-ai`. [Adapter README](https://github.com/vicentereig/dspy.rb/blob/main/lib/dspy/gemini/README.md) | **Stable** (v1.0.0) |
+| `dspy-code_act` | Think-Code-Observe agents that synthesize and execute Ruby safely. (Add the gem or set `DSPY_WITH_CODE_ACT=1` before requiring `dspy/code_act`.) | **Stable** (v1.0.0) |
+| `dspy-datasets` | Dataset helpers plus Parquet/Polars tooling for richer evaluation corpora. (Toggle via `DSPY_WITH_DATASETS`.) | **Stable** (v1.0.0) |
+| `dspy-evals` | High-throughput evaluation harness with metrics, callbacks, and regression fixtures. (Toggle via `DSPY_WITH_EVALS`.) | **Stable** (v1.0.0) |
+| `dspy-miprov2` | Bayesian optimization + Gaussian Process backend for the MIPROv2 teleprompter. (Install or export `DSPY_WITH_MIPROV2=1` before requiring the teleprompter.) | **Stable** (v1.0.0) |
+| `dspy-gepa` | `DSPy::Teleprompt::GEPA`, reflection loops, experiment tracking, telemetry adapters. (Install or set `DSPY_WITH_GEPA=1`.) | **Stable** (v1.0.0) |
+| `gepa` | GEPA optimizer core (Pareto engine, telemetry, reflective proposer). | **Stable** (v1.0.0) |
+| `dspy-o11y` | Core observability APIs: `DSPy::Observability`, async span processor, observation types. (Install or set `DSPY_WITH_O11Y=1`.) | **Stable** (v1.0.0) |
+| `dspy-o11y-langfuse` | Auto-configures DSPy observability to stream spans to Langfuse via OTLP. (Install or set `DSPY_WITH_O11Y_LANGFUSE=1`.) | **Stable** (v1.0.0) |
+| `dspy-deep_search` | Production DeepSearch loop with Exa-backed search/read, token budgeting, and instrumentation (Issue #163). | **Stable** (v1.0.0) |
+| `dspy-deep_research` | Planner/QA orchestration atop DeepSearch plus the memory supervisor used by the CLI example. | **Stable** (v1.0.0) |
+| `sorbet-toon` | Token-Oriented Object Notation (TOON) codec, prompt formatter, and Sorbet mixins for BAML/TOON Enhanced Prompting. [Sorbet::Toon README](https://github.com/vicentereig/dspy.rb/blob/main/lib/sorbet/toon/README.md) | **Alpha** (v0.1.0) |
+
+**Provider adapters:** Add `dspy-openai`, `dspy-anthropic`, and/or `dspy-gemini` next to `dspy` in your Gemfile depending on which `DSPy::LM` providers you call. Each gem already depends on the official SDK (`openai`, `anthropic`, `gemini-ai`), and DSPy auto-loads the adapters when the gem is present—no extra `require` needed.
+
+Set the matching `DSPY_WITH_*` environment variables (see `Gemfile`) to include or exclude each sibling gem when running Bundler locally (for example `DSPY_WITH_GEPA=1` or `DSPY_WITH_O11Y_LANGFUSE=1`). Refer to `adr/013-dependency-tree.md` for the full dependency map and roadmap.
 ### Access to 200+ Models Across 5 Providers
 
 DSPy.rb provides unified access to major LLM providers with provider-specific optimizations:
@@ -148,7 +159,10 @@ end
 
 ## What You Get
 
-**Developer Experience:**
+**Developer Experience:** Official clients, multimodal coverage, and observability baked in.
+<details>
+<summary>Expand for everything included</summary>
+
 - LLM provider support using official Ruby clients:
   - [OpenAI Ruby](https://github.com/openai/openai-ruby) with vision model support
   - [Anthropic Ruby SDK](https://github.com/anthropics/anthropic-sdk-ruby) with multimodal capabilities
@@ -158,21 +172,33 @@ end
 - Runtime type checking with [Sorbet](https://sorbet.org/) including T::Enum and union types
 - Type-safe tool definitions for ReAct agents
 - Comprehensive instrumentation and observability
+</details>
+
+**Core Building Blocks:** Predictors, agents, and pipelines wired through type-safe signatures.
+<details>
+<summary>Expand for everything included</summary>
 
-**Core Building Blocks:**
 - **Signatures** - Define input/output schemas using Sorbet types with T::Enum and union type support
 - **Predict** - LLM completion with structured data extraction and multimodal support
 - **Chain of Thought** - Step-by-step reasoning for complex problems with automatic prompt optimization
 - **ReAct** - Tool-using agents with type-safe tool definitions and error recovery
 - **Module Composition** - Combine multiple LLM calls into production-ready workflows
+</details>
+
+**Optimization & Evaluation:** Treat prompt optimization like a real ML workflow.
+<details>
+<summary>Expand for everything included</summary>
 
-**Optimization & Evaluation:**
 - **Prompt Objects** - Manipulate prompts as first-class objects instead of strings
 - **Typed Examples** - Type-safe training data with automatic validation
 - **Evaluation Framework** - Advanced metrics beyond simple accuracy with error-resilient pipelines
 - **MIPROv2 Optimization** - Advanced Bayesian optimization with Gaussian Processes, multiple optimization strategies, auto-config presets, and storage persistence
+</details>
+
+**Production Features:** Hardened behaviors for teams shipping actual products.
+<details>
+<summary>Expand for everything included</summary>
 
-**Production Features:**
 - **Reliable JSON Extraction** - Native structured outputs for OpenAI and Gemini, Anthropic tool-based extraction, and automatic strategy selection with fallback
 - **Type-Safe Configuration** - Strategy enums with automatic provider optimization (Strict/Compatible modes)
 - **Smart Retry Logic** - Progressive fallback with exponential backoff for handling transient failures
@@ -180,10 +206,13 @@ end
 - **Performance Caching** - Schema and capability caching for faster repeated operations
 - **File-based Storage** - Optimization result persistence with versioning
 - **Structured Logging** - JSON and key=value formats with span tracking
+</details>
 
 ## Recent Achievements
 
-DSPy.rb has rapidly evolved from experimental to production-ready:
+DSPy.rb has gone from experimental to production-ready in three fast releases.
+<details>
+<summary>Expand for the full changelog highlights</summary>
 
 ### Foundation
 - ✅ **JSON Parsing Reliability** - Native OpenAI structured outputs with adaptive retry logic and schema-aware fallbacks
@@ -199,8 +228,11 @@ DSPy.rb has rapidly evolved from experimental to production-ready:
 - ✅ **Optimizer Utilities Parity (v0.29.0)** - Bootstrap strategies, dataset summaries, and Layer 3 utilities unlock multi-predictor programs on Ruby
 - ✅ **Observability Hardening (v0.29.0)** - OTLP exporter runs on a single-thread executor preventing frozen SSL contexts without blocking spans
 - ✅ **Documentation Refresh (v0.29.x)** - New GEPA guide plus ADE optimization docs covering presets, stratified splits, and error-handling defaults
+</details>
 
-**Current Focus Areas:**
+**Current Focus Areas:** Closing the loop on production patterns and community adoption ahead of v1.0.
+<details>
+<summary>Expand for the roadmap</summary>
 
 ### Production Readiness
 - 🚧 **Production Patterns** - Real-world usage validation and performance optimization
@@ -210,10 +242,9 @@ DSPy.rb has rapidly evolved from experimental to production-ready:
 - 🚧 **Community Examples** - Real-world applications and case studies
 - 🚧 **Contributor Experience** - Making it easier to contribute and extend
 - 🚧 **Performance Benchmarks** - Comparative analysis vs other frameworks
+</details>
 
-**v1.0 Philosophy:**
-v1.0 will be released after extensive production battle-testing, not after checking off features.
-The API is already stable - v1.0 represents confidence in production reliability backed by real-world validation.
+**v1.0 Philosophy:** v1.0 lands after battle-testing, not checkbox bingo. The API is already stable; the milestone marks production confidence.
 
 
 ## Documentation

data/lib/dspy/callbacks.rb

@@ -259,15 +259,34 @@ module DSPy
     # Executes method with around callbacks
     def execute_with_around_callbacks(method_name, original_method, *args, **kwargs, &block)
       callbacks = self.class.send(:callbacks_for, method_name)[:around]
+      args_copy = args.dup
+      kwargs_copy = kwargs.dup
 
       # Build callback chain from innermost (original method) to outermost
       chain = callbacks.reverse.inject(
         -> { original_method.bind(self).call(*args, **kwargs, &block) }
       ) do |inner, callback|
         if callback.is_a?(Proc)
-          -> { instance_exec(-> { inner.call }, &callback) }
+          -> do
+            next_proc = -> { inner.call }
+            proc_arity = callback.arity
+            expects_extra = proc_arity.abs > 1
+
+            if expects_extra
+              instance_exec(next_proc, args_copy, kwargs_copy, &callback)
+            else
+              instance_exec(next_proc, &callback)
+            end
+          end
         else
-          -> { send(callback) { inner.call } }
+          -> do
+            method_obj = method(callback)
+            if method_obj.arity.zero?
+              send(callback) { inner.call }
+            else
+              send(callback, args_copy, kwargs_copy) { inner.call }
+            end
+          end
         end
       end
 

data/lib/dspy/context.rb

@@ -33,7 +33,8 @@ module DSPy
         context = {
           trace_id: SecureRandom.uuid,
           span_stack: [],
-          otel_span_stack: []
+          otel_span_stack: [],
+          module_stack: []
         }
         
         # Set in both Thread and Fiber storage
@@ -158,6 +159,48 @@ module DSPy
           end
         end
       end
+
+      def with_module(module_instance, label: nil)
+        stack = module_stack
+        entry = build_module_entry(module_instance, label)
+        stack.push(entry)
+        yield
+      ensure
+        if stack.last.equal?(entry)
+          stack.pop
+        else
+          stack.delete(entry)
+        end
+      end
+
+      def module_stack
+        current[:module_stack] ||= []
+      end
+
+      def module_context_attributes
+        stack = module_stack
+        return {} if stack.empty?
+
+        path = stack.map do |entry|
+          {
+            id: entry[:id],
+            class: entry[:class],
+            label: entry[:label]
+          }
+        end
+
+        ancestry_token = path.map { |node| node[:id] }.join('>')
+
+        {
+          module_path: path,
+          module_root: path.first,
+          module_leaf: path.last,
+          module_scope: {
+            ancestry_token: ancestry_token,
+            depth: path.length
+          }
+        }
+      end
       
       def clear!
         # Clear both the thread-specific key and the legacy key
@@ -218,6 +261,14 @@ module DSPy
           end
         end
       end
+
+      def build_module_entry(module_instance, explicit_label)
+        {
+          id: (module_instance.respond_to?(:module_scope_id) ? module_instance.module_scope_id : SecureRandom.uuid),
+          class: module_instance.class.name,
+          label: explicit_label || (module_instance.respond_to?(:module_scope_label) ? module_instance.module_scope_label : nil)
+        }
+      end
     end
   end
 end

data/lib/dspy/evals.rb

@@ -1,7 +1,6 @@
 # frozen_string_literal: true
 
 require 'json'
-require 'polars'
 require 'concurrent'
 require 'sorbet-runtime'
 require_relative 'example'
@@ -111,8 +110,14 @@ module DSPy
         }
       end
 
-      sig { returns(Polars::DataFrame) }
+      if defined?(Polars::DataFrame)
+        sig { returns(Polars::DataFrame) }
+      else
+        sig { returns(T.untyped) }
+      end
       def to_polars
+        ensure_polars!
+
         rows = @results.each_with_index.map do |result, index|
           {
             "index" => index,
@@ -130,6 +135,20 @@ module DSPy
 
       private
 
+      POLARS_MISSING_ERROR = <<~MSG
+        Polars is required to export evaluation results. Add `gem 'polars'`
+        (or enable the `dspy-datasets` gem / `DSPY_WITH_DATASETS=1`) before
+        calling `DSPy::Evals::BatchEvaluationResult#to_polars`.
+      MSG
+
+      def ensure_polars!
+        return if defined?(Polars::DataFrame)
+
+        require 'polars'
+      rescue LoadError => e
+        raise LoadError, "#{POLARS_MISSING_ERROR}\n\n#{e.message}"
+      end
+
       def serialize_for_polars(value)
         case value
         when NilClass, TrueClass, FalseClass, Numeric, String

data/lib/dspy/lm/adapter_factory.rb

@@ -6,15 +6,27 @@ module DSPy
     class AdapterFactory
       # Maps provider prefixes to adapter classes
       ADAPTER_MAP = {
-        'openai' => 'OpenAIAdapter',
-        'anthropic' => 'AnthropicAdapter',
-        'ollama' => 'OllamaAdapter',
-        'gemini' => 'GeminiAdapter',
-        'openrouter' => 'OpenrouterAdapter'
+        'openai' => { class_name: 'DSPy::OpenAI::LM::Adapters::OpenAIAdapter', gem_name: 'dspy-openai' },
+        'anthropic' => { class_name: 'DSPy::Anthropic::LM::Adapters::AnthropicAdapter', gem_name: 'dspy-anthropic' },
+        'ollama' => { class_name: 'DSPy::OpenAI::LM::Adapters::OllamaAdapter', gem_name: 'dspy-openai' },
+        'gemini' => { class_name: 'DSPy::Gemini::LM::Adapters::GeminiAdapter', gem_name: 'dspy-gemini' },
+        'openrouter' => { class_name: 'DSPy::OpenAI::LM::Adapters::OpenRouterAdapter', gem_name: 'dspy-openai' }
       }.freeze
 
       PROVIDERS_WITH_EXTRA_OPTIONS = %w[openai anthropic ollama gemini openrouter].freeze
 
+      class AdapterData < Data.define(:class_name, :gem_name)
+        def self.from_prefix(provider_prefix)
+          if ADAPTER_MAP.key?(provider_prefix)
+            new(**ADAPTER_MAP[provider_prefix])
+          end
+        end
+
+        def require_path
+          gem_name.tr('-', '/')
+        end
+      end
+
       class << self
         # Creates an adapter instance based on model_id
         # @param model_id [String] Full model identifier (e.g., "openai/gpt-4")
@@ -46,21 +58,32 @@ module DSPy
         end
 
         def get_adapter_class(provider)
-          adapter_class_name = ADAPTER_MAP[provider]
-          
-          unless adapter_class_name
+          ensure_adapter_supported!(provider)
+          ensure_adapter_loaded!(provider)
+
+          Object.const_get(adapter_data(provider).class_name)
+        end
+
+        def adapter_data(provider)
+          AdapterData.from_prefix(provider)
+        end
+
+        def ensure_adapter_supported!(provider)
+          if adapter_data(provider).nil?
             available_providers = ADAPTER_MAP.keys.join(', ')
-            raise UnsupportedProviderError, 
-                  "Unsupported provider: #{provider}. Available: #{available_providers}"
+            raise UnsupportedProviderError, "Unsupported provider: #{provider}. Available: #{available_providers}"
           end
+        end
 
-          begin
-            Object.const_get("DSPy::LM::#{adapter_class_name}")
-          rescue NameError
-            raise UnsupportedProviderError, 
-                  "Adapter not found: DSPy::LM::#{adapter_class_name}. " \
-                  "Make sure the corresponding gem is installed."
-          end
+        def ensure_adapter_loaded!(provider)
+          adapter_data = adapter_data(provider)
+          require adapter_data.require_path
+          msg = <<~ERROR
+            Adapter not found: #{adapter_data.class_name}.
+            Install the #{adapter_data.gem_name} gem and try again.
+          ERROR
+        rescue LoadError
+          raise MissingAdapterError, msg
         end
       end
     end

data/lib/dspy/lm/errors.rb

@@ -6,6 +6,9 @@ module DSPy
     class AdapterError < Error; end
     class UnsupportedProviderError < Error; end
     class ConfigurationError < Error; end
+    class MissingAdapterError < Error; end
+    class UnsupportedVersionError < Error; end
+    class MissingOfficialSDKError < Error; end
     
     # Raised when API key is missing or invalid
     class MissingAPIKeyError < Error

data/lib/dspy/lm/json_strategy.rb

@@ -1,8 +1,6 @@
 # frozen_string_literal: true
 
 require "sorbet-runtime"
-require_relative "adapters/openai/schema_converter"
-require_relative "adapters/gemini/schema_converter"
 
 module DSPy
   class LM
@@ -72,10 +70,19 @@ module DSPy
       # OpenAI/Ollama preparation
       sig { params(request_params: T::Hash[Symbol, T.untyped]).void }
       def prepare_openai_request(request_params)
+        begin
+          require "dspy/openai"
+        rescue LoadError
+          msg = <<~MSG
+            OpenAI adapter is optional; structured output helpers will be unavailable until the gem is installed.
+            Add `gem 'dspy-openai'` to your Gemfile and run `bundle install`.
+          MSG
+          raise DSPy::LM::MissingAdapterError, msg
+        end
+
         # Check if structured outputs are supported
-        if adapter.instance_variable_get(:@structured_outputs_enabled) &&
-           DSPy::LM::Adapters::OpenAI::SchemaConverter.supports_structured_outputs?(adapter.model)
-          response_format = DSPy::LM::Adapters::OpenAI::SchemaConverter.to_openai_format(signature_class)
+        if adapter.instance_variable_get(:@structured_outputs_enabled) && DSPy::OpenAI::LM::SchemaConverter.supports_structured_outputs?(adapter.model)
+          response_format = DSPy::OpenAI::LM::SchemaConverter.to_openai_format(signature_class)
           request_params[:response_format] = response_format
         end
       end
@@ -107,10 +114,19 @@ module DSPy
       # Gemini preparation
       sig { params(request_params: T::Hash[Symbol, T.untyped]).void }
       def prepare_gemini_request(request_params)
+        begin
+          require "dspy/gemini"
+        rescue LoadError
+          msg = <<~MSG
+            Gemini adapter is optional; structured output helpers will be unavailable until the gem is installed.
+            Add `gem 'dspy-gemini'` to your Gemfile and run `bundle install`.
+          MSG
+          raise DSPy::LM::MissingAdapterError, msg
+        end
+
         # Check if structured outputs are supported
-        if adapter.instance_variable_get(:@structured_outputs_enabled) &&
-           DSPy::LM::Adapters::Gemini::SchemaConverter.supports_structured_outputs?(adapter.model)
-          schema = DSPy::LM::Adapters::Gemini::SchemaConverter.to_gemini_format(signature_class)
+        if adapter.instance_variable_get(:@structured_outputs_enabled) && DSPy::Gemini::LM::SchemaConverter.supports_structured_outputs?(adapter.model)
+          schema = DSPy::Gemini::LM::SchemaConverter.to_gemini_format(signature_class)
 
           request_params[:generation_config] = {
             response_mime_type: "application/json",

data/lib/dspy/lm.rb

@@ -12,13 +12,6 @@ require_relative 'lm/adapter_factory'
 
 # Load instrumentation
 
-# Load adapters
-require_relative 'lm/adapters/openai_adapter'
-require_relative 'lm/adapters/anthropic_adapter'
-require_relative 'lm/adapters/ollama_adapter'
-require_relative 'lm/adapters/gemini_adapter'
-require_relative 'lm/adapters/openrouter_adapter'
-
 # Load strategy system
 require_relative 'lm/chat_strategy'
 require_relative 'lm/json_strategy'
@@ -27,16 +20,18 @@ require_relative 'lm/json_strategy'
 require_relative 'lm/message'
 require_relative 'lm/message_builder'
 require_relative 'structured_outputs_prompt'
+require_relative 'schema/sorbet_toon_adapter'
 
 module DSPy
   class LM
     extend T::Sig
-    attr_reader :model_id, :api_key, :model, :provider, :adapter, :schema_format
+    attr_reader :model_id, :api_key, :model, :provider, :adapter, :schema_format, :data_format
 
-    def initialize(model_id, api_key: nil, schema_format: :json, **options)
+    def initialize(model_id, api_key: nil, schema_format: :json, data_format: :json, **options)
       @model_id = model_id
       @api_key = api_key
       @schema_format = schema_format
+      @data_format = data_format
 
       # Parse provider and model from model_id
       @provider, @model = parse_model_id(model_id)
@@ -209,12 +204,42 @@ module DSPy
       adapter_class_name = adapter.class.name
 
       if adapter_class_name.include?('OpenAIAdapter') || adapter_class_name.include?('OllamaAdapter')
+        begin
+          require "dspy/openai"
+        rescue LoadError
+          msg = <<~MSG
+            Install the openai gem to enable support for this adapter.
+            Add `gem 'dspy-openai'` to your Gemfile and run `bundle install`.
+          MSG
+          raise DSPy::LM::MissingAdapterError, msg
+        end
+
         adapter.instance_variable_get(:@structured_outputs_enabled) &&
-          DSPy::LM::Adapters::OpenAI::SchemaConverter.supports_structured_outputs?(adapter.model)
+          DSPy::OpenAI::LM::SchemaConverter.supports_structured_outputs?(adapter.model)
       elsif adapter_class_name.include?('GeminiAdapter')
+        begin
+          require "dspy/gemini"
+        rescue LoadError
+          msg = <<~MSG
+            Install the gem to enable Gemini support.
+            Add `gem 'dspy-gemini'` to your Gemfile and run `bundle install`.
+          MSG
+          raise DSPy::LM::MissingAdapterError, msg
+        end
+
         adapter.instance_variable_get(:@structured_outputs_enabled) &&
-          DSPy::LM::Adapters::Gemini::SchemaConverter.supports_structured_outputs?(adapter.model)
+          DSPy::Gemini::LM::SchemaConverter.supports_structured_outputs?(adapter.model)
       elsif adapter_class_name.include?('AnthropicAdapter')
+        begin
+          require "dspy/anthropic"
+        rescue LoadError
+          msg = <<~MSG
+            Install the gem to enable Claude support.
+            Add `gem 'dspy-anthropic'` to your Gemfile and run `bundle install`.
+          MSG
+          raise DSPy::LM::MissingAdapterError, msg
+        end
+
         structured_outputs_enabled = adapter.instance_variable_get(:@structured_outputs_enabled)
         structured_outputs_enabled.nil? ? true : structured_outputs_enabled
       else
@@ -223,28 +248,46 @@ module DSPy
     end
 
     def parse_response(response, input_values, signature_class)
-      # Try to parse the response as JSON
+      if data_format == :toon
+        payload = DSPy::Schema::SorbetToonAdapter.parse_output(signature_class, response.content.to_s)
+        return normalize_output_payload(payload)
+      end
+
       content = response.content
 
       begin
-        json_payload = JSON.parse(content)
-
-        # For Sorbet signatures, just return the parsed JSON
-        # The Predict will handle validation
-        json_payload
+        JSON.parse(content)
       rescue JSON::ParserError => e
-        # Enhanced error message with debugging information
         error_details = {
           original_content: response.content,
           provider: provider,
           model: model
         }
-        
+
         DSPy.logger.debug("JSON parsing failed: #{error_details}")
         raise "Failed to parse LLM response as JSON: #{e.message}. Original content length: #{response.content&.length || 0} chars"
       end
     end
 
+    def normalize_output_payload(payload)
+      case payload
+      when T::Struct
+        payload.class.props.each_with_object({}) do |(name, _), memo|
+          memo[name.to_s] = normalize_output_payload(payload.send(name))
+        end
+      when Hash
+        payload.each_with_object({}) do |(key, value), memo|
+          memo[key.to_s] = normalize_output_payload(value)
+        end
+      when Array
+        payload.map { |item| normalize_output_payload(item) }
+      when Set
+        payload.map { |item| normalize_output_payload(item) }
+      else
+        payload
+      end
+    end
+
     # Common instrumentation method for LM requests
     def instrument_lm_request(messages, signature_class_name, &execution_block)
       # Prepare input for tracing - convert messages to JSON for input tracking