dspy-evals 1.0.0 → 1.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2b829eb6518603189ba468cbd12bdfb8cf28569805219e0cf6402912e9f70aea
-  data.tar.gz: 3d20d190d99d337d3fc1acec66360eb7abbccab343ced186cc7af0e1072a2f88
+  metadata.gz: d6e4ef06553cd53f974d9813c83807ce43a4222132f6200837a15087722e6483
+  data.tar.gz: 077d72f3f4db1122e749248b29d27206f277cef03954ef58483dc4452328fbac
 SHA512:
-  metadata.gz: 89ea669cb6ae3a7f4443a9cab042ece8c5a50f39ededed884bd44d12454c55fad6d3c3a580fc70e6824a12dea44378325fab9e4fc2a66e3483c262d76dbf3e56
-  data.tar.gz: 848ddef8ad63facc5a08b0c79c138c7da4283b25207c670a74f328bdc3cafcff1563515b1a1f12b0b264f5b812d746dcd6219354c192d6b99b2b109d097a1055
+  metadata.gz: 73aa76a6904812e98cf7ded086a9642ca9fc696c3e1c017b5fec91a63a4887e54d61b961d79d6cb3d1fda2fc44707007e67849c71e61185af14d7146dc50ef0a
+  data.tar.gz: 4270e116956e4ba8960f958302f817a3236d82922224a8cdfd066d172693602e99dfe2da3a68c4699c91fb3a01d9f0e5da0ad6bec86b3bb4b3a802193b269d41

data/README.md

@@ -3,81 +3,97 @@
 [![Gem Version](https://img.shields.io/gem/v/dspy)](https://rubygems.org/gems/dspy)
 [![Total Downloads](https://img.shields.io/gem/dt/dspy)](https://rubygems.org/gems/dspy)
 [![Build Status](https://img.shields.io/github/actions/workflow/status/vicentereig/dspy.rb/ruby.yml?branch=main&label=build)](https://github.com/vicentereig/dspy.rb/actions/workflows/ruby.yml)
-[![Documentation](https://img.shields.io/badge/docs-vicentereig.github.io%2Fdspy.rb-blue)](https://vicentereig.github.io/dspy.rb/)
+[![Documentation](https://img.shields.io/badge/docs-oss.vicente.services%2Fdspy.rb-blue)](https://oss.vicente.services/dspy.rb/)
+[![Discord](https://img.shields.io/discord/1161519468141355160?label=discord&logo=discord&logoColor=white)](https://discord.gg/zWBhrMqn)
 
-> [!NOTE]
-> The core Prompt Engineering Framework is production-ready with
-> comprehensive documentation. I am focusing now on educational content on systematic Prompt Optimization and Context Engineering.
-> Your feedback is invaluable. if you encounter issues, please open an [issue](https://github.com/vicentereig/dspy.rb/issues). If you have suggestions, open a [new thread](https://github.com/vicentereig/dspy.rb/discussions).  
-> 
-> If you want to contribute, feel free to reach out to me to coordinate efforts: hey at vicente.services
->
-> And, yes, this is 100% a legit project. :) 
+**Build reliable LLM applications in idiomatic Ruby using composable, type-safe modules.**
 
+DSPy.rb is the Ruby port of Stanford's [DSPy](https://dspy.ai). Instead of wrestling with brittle prompt strings, you define typed signatures and let the framework handle the rest. Prompts become functions. LLM calls become predictable.
 
-**Build reliable LLM applications in idiomatic Ruby using composable, type-safe modules.**
+```ruby
+require 'dspy'
 
-The Ruby framework for programming with large language models. DSPy.rb brings structured LLM programming to Ruby developers, programmatic Prompt Engineering and Context Engineering. 
-Instead of wrestling with prompt strings and parsing responses, you define typed signatures using idiomatic Ruby to compose and decompose AI Worklows and AI Agents.
+DSPy.configure do |c|
+  c.lm = DSPy::LM.new('openai/gpt-4o-mini', api_key: ENV['OPENAI_API_KEY'])
+end
 
-**Prompts are the just Functions.** Traditional prompting is like writing code with string concatenation: it works until it doesn't. DSPy.rb brings you 
-the programming approach pioneered by [dspy.ai](https://dspy.ai/): instead of crafting fragile prompts, you define modular 
-signatures and let the framework handle the messy details.
+class Summarize < DSPy::Signature
+  description "Summarize the given text in one sentence."
 
-DSPy.rb is an idiomatic Ruby surgical port of Stanford's [DSPy framework](https://github.com/stanfordnlp/dspy). While implementing 
-the core concepts of signatures, predictors, and the main optimization algorithms from the original Python library, DSPy.rb embraces Ruby 
-conventions and adds Ruby-specific innovations like Sorbet-base Typed system, ReAct loops, and production-ready integrations like non-blocking Open Telemetry Instrumentation.
+  input do
+    const :text, String
+  end
 
-**What you get?** Ruby LLM applications that actually scale and don't break when you sneeze.
+  output do
+    const :summary, String
+  end
+end
 
-Check the [examples](examples/) and take them for a spin!
+summarizer = DSPy::Predict.new(Summarize)
+result = summarizer.call(text: "DSPy.rb brings structured LLM programming to Ruby...")
+puts result.summary
+```
 
-## Your First DSPy Program
-### Installation
+That's it. No prompt templates. No JSON parsing. No prayer-based error handling.
 
-Add to your Gemfile:
+## Installation
 
 ```ruby
+# Gemfile
 gem 'dspy'
+gem 'dspy-openai'     # For OpenAI, OpenRouter, or Ollama
+# gem 'dspy-anthropic' # For Claude
+# gem 'dspy-gemini'    # For Gemini
+# gem 'dspy-ruby_llm'  # For 12+ providers via RubyLLM
 ```
 
-and
-
 ```bash
 bundle install
 ```
 
-### Optional Sibling Gems
-
-DSPy.rb ships multiple gems from this monorepo so you only install what you need. Add these alongside `dspy`:
-
-| Gem | Description | Status |
-| --- | --- | --- |
-| `dspy-schema` | Exposes `DSPy::TypeSystem::SorbetJsonSchema` for downstream reuse. | **Stable** (v1.0.0) |
-| `dspy-code_act` | Think-Code-Observe agents that synthesize and execute Ruby safely. | Preview (0.x) |
-| `dspy-datasets` | Dataset helpers plus Parquet/Polars tooling for richer evaluation corpora. | Preview (0.x) |
-| `dspy-evals` | High-throughput evaluation harness with metrics, callbacks, and regression fixtures. | Preview (0.x) |
-| `dspy-miprov2` | Bayesian optimization + Gaussian Process backend for the MIPROv2 teleprompter. | Preview (0.x) |
-| `dspy-gepa` | `DSPy::Teleprompt::GEPA`, reflection loops, experiment tracking, telemetry adapters. | Preview (mirrors `dspy` version) |
-| `gepa` | GEPA optimizer core (Pareto engine, telemetry, reflective proposer). | Preview (mirrors `dspy` version) |
-| `dspy-o11y` | Core observability APIs: `DSPy::Observability`, async span processor, observation types. | **Stable** (v1.0.0) |
-| `dspy-o11y-langfuse` | Auto-configures DSPy observability to stream spans to Langfuse via OTLP. | **Stable** (v1.0.0) |
+## Quick Start
 
-Set the matching `DSPY_WITH_*` environment variables (see `Gemfile`) to include or exclude each sibling gem when running Bundler locally (for example `DSPY_WITH_GEPA=1` or `DSPY_WITH_O11Y_LANGFUSE=1`). Refer to `docs/core-concepts/dependency-tree.md` for the full dependency map and roadmap.
-### Your First Reliable Predictor
+### Configure Your LLM
 
 ```ruby
-
-# Configure DSPy globablly to use your fave LLM - you can override this on an instance levle. 
+# OpenAI
 DSPy.configure do |c|
   c.lm = DSPy::LM.new('openai/gpt-4o-mini',
                       api_key: ENV['OPENAI_API_KEY'],
-                      structured_outputs: true)  # Enable OpenAI's native JSON mode
+                      structured_outputs: true)
+end
+
+# Anthropic Claude
+DSPy.configure do |c|
+  c.lm = DSPy::LM.new('anthropic/claude-sonnet-4-20250514',
+                      api_key: ENV['ANTHROPIC_API_KEY'])
 end
 
-# Define a signature for sentiment classification - instead of writing a full prompt!
+# Google Gemini
+DSPy.configure do |c|
+  c.lm = DSPy::LM.new('gemini/gemini-2.5-flash',
+                      api_key: ENV['GEMINI_API_KEY'])
+end
+
+# Ollama (local, free)
+DSPy.configure do |c|
+  c.lm = DSPy::LM.new('ollama/llama3.2')
+end
+
+# OpenRouter (200+ models)
+DSPy.configure do |c|
+  c.lm = DSPy::LM.new('openrouter/deepseek/deepseek-chat-v3.1:free',
+                      api_key: ENV['OPENROUTER_API_KEY'])
+end
+```
+
+### Define a Signature
+
+Signatures are typed contracts for LLM operations. Define inputs, outputs, and let DSPy handle the prompt:
+
+```ruby
 class Classify < DSPy::Signature
-  description "Classify sentiment of a given sentence." # sets the goal of the underlying prompt
+  description "Classify sentiment of a given sentence."
 
   class Sentiment < T::Enum
     enums do
@@ -86,182 +102,130 @@ class Classify < DSPy::Signature
       Neutral = new('neutral')
     end
   end
-  
-  # Structured Inputs: makes sure you are sending only valid prompt inputs to your model
+
   input do
     const :sentence, String, description: 'The sentence to analyze'
   end
 
-  # Structured Outputs: your predictor will validate the output of the model too.
   output do
-    const :sentiment, Sentiment, description: 'The sentiment of the sentence'
-    const :confidence, Float, description: 'A number between 0.0 and 1.0'
+    const :sentiment, Sentiment
+    const :confidence, Float
   end
 end
 
-# Wire it to the simplest prompting technique - a Predictn.
-classify = DSPy::Predict.new(Classify)
-# it may raise an error if you mess the inputs or your LLM messes the outputs.
-result = classify.call(sentence: "This book was super fun to read!")
+classifier = DSPy::Predict.new(Classify)
+result = classifier.call(sentence: "This book was super fun to read!")
 
-puts result.sentiment    # => #<Sentiment::Positive>  
-puts result.confidence   # => 0.85
+result.sentiment    # => #<Sentiment::Positive>
+result.confidence   # => 0.92
 ```
 
-### Access to 200+ Models Across 5 Providers
+### Chain of Thought
 
-DSPy.rb provides unified access to major LLM providers with provider-specific optimizations:
+For complex reasoning, use `ChainOfThought` to get step-by-step explanations:
 
 ```ruby
-# OpenAI (GPT-4, GPT-4o, GPT-4o-mini, GPT-5, etc.)
-DSPy.configure do |c|
-  c.lm = DSPy::LM.new('openai/gpt-4o-mini',
-                      api_key: ENV['OPENAI_API_KEY'],
-                      structured_outputs: true)  # Native JSON mode
-end
+solver = DSPy::ChainOfThought.new(MathProblem)
+result = solver.call(problem: "If a train travels 120km in 2 hours, what's its speed?")
 
-# Google Gemini (Gemini 1.5 Pro, Flash, Gemini 2.0, etc.)
-DSPy.configure do |c|
-  c.lm = DSPy::LM.new('gemini/gemini-2.5-flash',
-                      api_key: ENV['GEMINI_API_KEY'],
-                      structured_outputs: true)  # Native structured outputs
-end
+result.reasoning  # => "Speed = Distance / Time = 120km / 2h = 60km/h"
+result.answer     # => "60 km/h"
+```
 
-# Anthropic Claude (Claude 3.5, Claude 4, etc.)
-DSPy.configure do |c|
-  c.lm = DSPy::LM.new('anthropic/claude-sonnet-4-5-20250929',
-                      api_key: ENV['ANTHROPIC_API_KEY'],
-                      structured_outputs: true)  # Tool-based extraction (default)
-end
+### ReAct Agents
 
-# Ollama - Run any local model (Llama, Mistral, Gemma, etc.)
-DSPy.configure do |c|
-  c.lm = DSPy::LM.new('ollama/llama3.2')  # Free, runs locally, no API key needed
-end
+Build agents that use tools to accomplish tasks:
 
-# OpenRouter - Access to 200+ models from multiple providers
-DSPy.configure do |c|
-  c.lm = DSPy::LM.new('openrouter/deepseek/deepseek-chat-v3.1:free',
-                      api_key: ENV['OPENROUTER_API_KEY'])
+```ruby
+class SearchTool < DSPy::Tools::Tool
+  tool_name "search"
+  description "Search for information"
+
+  input do
+    const :query, String
+  end
+
+  output do
+    const :results, T::Array[String]
+  end
+
+  def call(query:)
+    # Your search implementation
+    { results: ["Result 1", "Result 2"] }
+  end
 end
+
+toolset = DSPy::Tools::Toolset.new(tools: [SearchTool.new])
+agent = DSPy::ReAct.new(signature: ResearchTask, tools: toolset, max_iterations: 5)
+result = agent.call(question: "What's the latest on Ruby 3.4?")
 ```
 
-## What You Get
-
-**Developer Experience:**
-- LLM provider support using official Ruby clients:
-  - [OpenAI Ruby](https://github.com/openai/openai-ruby) with vision model support
-  - [Anthropic Ruby SDK](https://github.com/anthropics/anthropic-sdk-ruby) with multimodal capabilities
-  - [Google Gemini API](https://ai.google.dev/) with native structured outputs
-  - [Ollama](https://ollama.com/) via OpenAI compatibility layer for local models
-- **Multimodal Support** - Complete image analysis with DSPy::Image, type-safe bounding boxes, vision-capable models
-- Runtime type checking with [Sorbet](https://sorbet.org/) including T::Enum and union types
-- Type-safe tool definitions for ReAct agents
-- Comprehensive instrumentation and observability
-
-**Core Building Blocks:**
-- **Signatures** - Define input/output schemas using Sorbet types with T::Enum and union type support
-- **Predict** - LLM completion with structured data extraction and multimodal support
-- **Chain of Thought** - Step-by-step reasoning for complex problems with automatic prompt optimization
-- **ReAct** - Tool-using agents with type-safe tool definitions and error recovery
-- **Module Composition** - Combine multiple LLM calls into production-ready workflows
-
-**Optimization & Evaluation:**
-- **Prompt Objects** - Manipulate prompts as first-class objects instead of strings
-- **Typed Examples** - Type-safe training data with automatic validation
-- **Evaluation Framework** - Advanced metrics beyond simple accuracy with error-resilient pipelines
-- **MIPROv2 Optimization** - Advanced Bayesian optimization with Gaussian Processes, multiple optimization strategies, auto-config presets, and storage persistence
-
-**Production Features:**
-- **Reliable JSON Extraction** - Native structured outputs for OpenAI and Gemini, Anthropic tool-based extraction, and automatic strategy selection with fallback
-- **Type-Safe Configuration** - Strategy enums with automatic provider optimization (Strict/Compatible modes)
-- **Smart Retry Logic** - Progressive fallback with exponential backoff for handling transient failures
-- **Zero-Config Langfuse Integration** - Set env vars and get automatic OpenTelemetry traces in Langfuse
-- **Performance Caching** - Schema and capability caching for faster repeated operations
-- **File-based Storage** - Optimization result persistence with versioning
-- **Structured Logging** - JSON and key=value formats with span tracking
-
-## Recent Achievements
-
-DSPy.rb has rapidly evolved from experimental to production-ready:
-
-### Foundation
-- ✅ **JSON Parsing Reliability** - Native OpenAI structured outputs with adaptive retry logic and schema-aware fallbacks
-- ✅ **Type-Safe Strategy Configuration** - Provider-optimized strategy selection and enum-backed optimizer presets
-- ✅ **Core Module System** - Predict, ChainOfThought, ReAct with type safety (add `dspy-code_act` for Think-Code-Observe agents)
-- ✅ **Production Observability** - OpenTelemetry, New Relic, and Langfuse integration
-- ✅ **Advanced Optimization** - MIPROv2 with Bayesian optimization, Gaussian Processes, and multi-mode search
-
-### Recent Advances
-- ✅ **MIPROv2 ADE Integrity (v0.29.1)** - Stratified train/val/test splits, honest precision accounting, and enum-driven `--auto` presets with integration coverage
-- ✅ **Instruction Deduplication (v0.29.1)** - Candidate generation now filters repeated programs so optimization logs highlight unique strategies
-- ✅ **GEPA Teleprompter (v0.29.0)** - Genetic-Pareto reflective prompt evolution with merge proposer scheduling, reflective mutation, and ADE demo parity
-- ✅ **Optimizer Utilities Parity (v0.29.0)** - Bootstrap strategies, dataset summaries, and Layer 3 utilities unlock multi-predictor programs on Ruby
-- ✅ **Observability Hardening (v0.29.0)** - OTLP exporter runs on a single-thread executor preventing frozen SSL contexts without blocking spans
-- ✅ **Documentation Refresh (v0.29.x)** - New GEPA guide plus ADE optimization docs covering presets, stratified splits, and error-handling defaults
-
-**Current Focus Areas:**
-
-### Production Readiness
-- 🚧 **Production Patterns** - Real-world usage validation and performance optimization
-- 🚧 **Ruby Ecosystem Integration** - Rails integration, Sidekiq compatibility, deployment patterns
-
-### Community & Adoption
-- 🚧 **Community Examples** - Real-world applications and case studies
-- 🚧 **Contributor Experience** - Making it easier to contribute and extend
-- 🚧 **Performance Benchmarks** - Comparative analysis vs other frameworks
-
-**v1.0 Philosophy:**
-v1.0 will be released after extensive production battle-testing, not after checking off features.
-The API is already stable - v1.0 represents confidence in production reliability backed by real-world validation.
+## What's Included
+
+**Core Modules**: Predict, ChainOfThought, ReAct agents, and composable pipelines.
+
+**Type Safety**: Sorbet-based runtime validation. Enums, unions, nested structs—all work.
+
+**Multimodal**: Image analysis with `DSPy::Image` for vision-capable models.
 
+**Observability**: Zero-config Langfuse integration via OpenTelemetry. Non-blocking, production-ready.
+
+**Optimization**: MIPROv2 (Bayesian optimization) and GEPA (genetic evolution) for prompt tuning.
+
+**Provider Support**: OpenAI, Anthropic, Gemini, Ollama, and OpenRouter via official SDKs.
 
 ## Documentation
 
-📖 **[Complete Documentation Website](https://vicentereig.github.io/dspy.rb/)**
+**[Full Documentation](https://oss.vicente.services/dspy.rb/)** — Getting started, core concepts, advanced patterns.
 
-### LLM-Friendly Documentation
+**[llms.txt](https://oss.vicente.services/dspy.rb/llms.txt)** — LLM-friendly reference for AI assistants.
 
-For LLMs and AI assistants working with DSPy.rb:
-- **[llms.txt](https://vicentereig.github.io/dspy.rb/llms.txt)** - Concise reference optimized for LLMs
-- **[llms-full.txt](https://vicentereig.github.io/dspy.rb/llms-full.txt)** - Comprehensive API documentation
+### Claude Skill
 
-### Getting Started
-- **[Installation & Setup](docs/src/getting-started/installation.md)** - Detailed installation and configuration
-- **[Quick Start Guide](docs/src/getting-started/quick-start.md)** - Your first DSPy programs
-- **[Core Concepts](docs/src/getting-started/core-concepts.md)** - Understanding signatures, predictors, and modules
+A [Claude Skill](https://github.com/vicentereig/dspy-rb-skill) is available to help you build DSPy.rb applications:
 
-### Prompt Engineering
-- **[Signatures & Types](docs/src/core-concepts/signatures.md)** - Define typed interfaces for LLM operations
-- **[Predictors](docs/src/core-concepts/predictors.md)** - Predict, ChainOfThought, ReAct, and more
-- **[Modules & Pipelines](docs/src/core-concepts/modules.md)** - Compose complex multi-stage workflows
-- **[Multimodal Support](docs/src/core-concepts/multimodal.md)** - Image analysis with vision-capable models
-- **[Examples & Validation](docs/src/core-concepts/examples.md)** - Type-safe training data
-- **[Rich Types](docs/src/advanced/complex-types.md)** - Sorbet type integration with automatic coercion for structs, enums, and arrays
-- **[Composable Pipelines](docs/src/advanced/pipelines.md)** - Manual module composition patterns
+```bash
+# Claude Code
+git clone https://github.com/vicentereig/dspy-rb-skill ~/.claude/skills/dspy-rb
+```
 
-### Prompt Optimization
-- **[Evaluation Framework](docs/src/optimization/evaluation.md)** - Advanced metrics beyond simple accuracy
-- **[Prompt Optimization](docs/src/optimization/prompt-optimization.md)** - Manipulate prompts as objects
-- **[MIPROv2 Optimizer](docs/src/optimization/miprov2.md)** - Advanced Bayesian optimization with Gaussian Processes
-- **[GEPA Optimizer](docs/src/optimization/gepa.md)** *(beta)* - Reflective mutation with optional reflection LMs
+For Claude.ai Pro/Max, download the [skill ZIP](https://github.com/vicentereig/dspy-rb-skill/archive/refs/heads/main.zip) and upload via Settings > Skills.
 
-### Context Engineering
-- **[Tools](docs/src/core-concepts/toolsets.md)** - Tool wieldint agents.
-- **[Agentic Memory](docs/src/core-concepts/memory.md)** - Memory Tools & Agentic Loops
-- **[RAG Patterns](docs/src/advanced/rag.md)** - Manual RAG implementation with external services
+## Examples
 
-### Production Features
-- **[Observability](docs/src/production/observability.md)** - Zero-config Langfuse integration with a dedicated export worker that never blocks your LLMs
-- **[Storage System](docs/src/production/storage.md)** - Persistence and optimization result storage
-- **[Custom Metrics](docs/src/advanced/custom-metrics.md)** - Proc-based evaluation logic 
+The [examples/](examples/) directory has runnable code for common patterns:
 
+- Sentiment classification
+- ReAct agents with tools
+- Image analysis
+- Prompt optimization
 
+```bash
+bundle exec ruby examples/first_predictor.rb
+```
+
+## Optional Gems
 
+DSPy.rb ships sibling gems for features with heavier dependencies. Add them as needed:
 
+| Gem | What it does |
+| --- | --- |
+| `dspy-datasets` | Dataset helpers, Parquet/Polars tooling |
+| `dspy-evals` | Evaluation harness with metrics and callbacks |
+| `dspy-miprov2` | Bayesian optimization for prompt tuning |
+| `dspy-gepa` | Genetic-Pareto prompt evolution |
+| `dspy-o11y-langfuse` | Auto-configure Langfuse tracing |
+| `dspy-code_act` | Think-Code-Observe agents |
+| `dspy-deep_search` | Production DeepSearch with Exa |
 
+See [the full list](https://oss.vicente.services/dspy.rb/getting-started/installation/) in the docs.
 
+## Contributing
 
+Feedback is invaluable. If you encounter issues, [open an issue](https://github.com/vicentereig/dspy.rb/issues). For suggestions, [start a discussion](https://github.com/vicentereig/dspy.rb/discussions).
+
+Want to contribute code? Reach out: hey at vicente.services
 
 ## License
-This project is licensed under the MIT License.
+
+MIT License.

data/lib/dspy/evals/version.rb

@@ -2,6 +2,6 @@
 
 module DSPy
   class Evals
-    VERSION = '1.0.0'
+    VERSION = '1.0.2'
   end
 end

data/lib/dspy/evals.rb

@@ -1,7 +1,6 @@
 # frozen_string_literal: true
 
 require 'json'
-require 'polars'
 require 'concurrent'
 require 'sorbet-runtime'
 require_relative 'example'
@@ -111,8 +110,14 @@ module DSPy
         }
       end
 
-      sig { returns(Polars::DataFrame) }
+      if defined?(Polars::DataFrame)
+        sig { returns(Polars::DataFrame) }
+      else
+        sig { returns(T.untyped) }
+      end
       def to_polars
+        ensure_polars!
+
         rows = @results.each_with_index.map do |result, index|
           {
             "index" => index,
@@ -130,6 +135,20 @@ module DSPy
 
       private
 
+      POLARS_MISSING_ERROR = <<~MSG
+        Polars is required to export evaluation results. Add `gem 'polars'`
+        (or enable the `dspy-datasets` gem / `DSPY_WITH_DATASETS=1`) before
+        calling `DSPy::Evals::BatchEvaluationResult#to_polars`.
+      MSG
+
+      def ensure_polars!
+        return if defined?(Polars::DataFrame)
+
+        require 'polars'
+      rescue LoadError => e
+        raise LoadError, "#{POLARS_MISSING_ERROR}\n\n#{e.message}"
+      end
+
       def serialize_for_polars(value)
         case value
         when NilClass, TrueClass, FalseClass, Numeric, String
@@ -172,6 +191,12 @@ module DSPy
     sig { returns(T.nilable(BatchEvaluationResult)) }
     attr_reader :last_batch_result
 
+    sig { returns(T::Boolean) }
+    attr_reader :export_scores
+
+    sig { returns(String) }
+    attr_reader :score_name
+
     include DSPy::Callbacks
 
     create_before_callback :call, wrap: false
@@ -208,16 +233,20 @@ module DSPy
         num_threads: T.nilable(Integer),
         max_errors: T.nilable(Integer),
         failure_score: T.nilable(Numeric),
-        provide_traceback: T::Boolean
+        provide_traceback: T::Boolean,
+        export_scores: T::Boolean,
+        score_name: String
       ).void
     end
-    def initialize(program, metric: nil, num_threads: 1, max_errors: 5, failure_score: 0.0, provide_traceback: true)
+    def initialize(program, metric: nil, num_threads: 1, max_errors: 5, failure_score: 0.0, provide_traceback: true, export_scores: false, score_name: 'evaluation')
       @program = program
       @metric = metric
       @num_threads = num_threads || 1
       @max_errors = max_errors || 5
       @provide_traceback = provide_traceback
       @failure_score = failure_score ? failure_score.to_f : 0.0
+      @export_scores = export_scores
+      @score_name = score_name
       @last_example_result = nil
       @last_batch_result = nil
     end
@@ -225,25 +254,7 @@ module DSPy
     # Evaluate program on a single example
     sig { params(example: T.untyped, trace: T.nilable(T.untyped)).returns(EvaluationResult) }
     def call(example, trace: nil)
-      run_callbacks(:before, :call, example: example)
-
-      DSPy::Context.with_span(
-        operation: 'evaluation.example',
-        'dspy.module' => 'Evaluator',
-        'evaluation.program' => @program.class.name,
-        'evaluation.has_metric' => !@metric.nil?
-      ) do
-        begin
-          perform_call(example, trace: trace)
-        rescue => e
-          build_error_result(example, e, trace: trace)
-        end
-      end.then do |result|
-        @last_example_result = result
-        emit_example_observation(example, result)
-        run_callbacks(:after, :call, example: example, result: result)
-        result
-      end
+      call_with_program(@program, example, trace: trace, track_state: true)
     end
 
     # Evaluate program on multiple examples
@@ -374,8 +385,9 @@ module DSPy
 
         futures = batch.map do |item|
           Concurrent::Promises.future_on(executor) do
-            [:ok, item[:index], safe_call(item[:example])]
-          rescue => e
+            program_for_thread = fork_program_for_thread
+            [:ok, item[:index], safe_call(item[:example], program: program_for_thread, track_state: false)]
+          rescue StandardError => e
             [:error, item[:index], e]
           end
         end
@@ -412,18 +424,18 @@ module DSPy
       results.compact
     end
 
-    def safe_call(example)
-      call(example)
-    rescue => e
+    def safe_call(example, program: @program, track_state: true)
+      call_with_program(program, example, track_state: track_state)
+    rescue StandardError => e
       build_error_result(example, e)
     end
 
-    def perform_call(example, trace:)
+    def perform_call(example, trace:, program:)
       # Extract input from example - support both hash and object formats
       input_values = extract_input_values(example)
 
       # Run prediction
-      prediction = @program.call(**input_values)
+      prediction = program.call(**input_values)
 
       # Calculate metrics if provided
       metrics = {}
@@ -440,7 +452,7 @@ module DSPy
             passed = !!metric_result
             metrics[:passed] = passed
           end
-        rescue => e
+        rescue StandardError => e
           passed = false
           metrics[:error] = e.message
           metrics[:passed] = false
@@ -461,6 +473,34 @@ module DSPy
       )
     end
 
+    def call_with_program(program, example, trace: nil, track_state: true)
+      run_callbacks(:before, :call, example: example)
+
+      DSPy::Context.with_span(
+        operation: 'evaluation.example',
+        'dspy.module' => 'Evaluator',
+        'evaluation.program' => program.class.name,
+        'evaluation.has_metric' => !@metric.nil?
+      ) do
+        begin
+          perform_call(example, trace: trace, program: program)
+        rescue StandardError => e
+          build_error_result(example, e, trace: trace)
+        end
+      end.then do |result|
+        @last_example_result = result if track_state
+        emit_example_observation(example, result)
+        run_callbacks(:after, :call, example: example, result: result)
+        result
+      end
+    end
+
+    def fork_program_for_thread
+      return @program if @program.nil?
+      return @program.dup_for_thread if @program.respond_to?(:dup_for_thread)
+      @program.dup
+    end
+
     def build_error_result(example, error, trace: nil)
       metrics = {
         error: error.message,
@@ -646,7 +686,12 @@ module DSPy
         score: result.metrics[:score],
         error: result.metrics[:error]
       })
-    rescue => e
+
+      # Export score to Langfuse if enabled
+      if @export_scores
+        export_example_score(example, result)
+      end
+    rescue StandardError => e
       DSPy.log('evals.example.observation_error', error: e.message)
     end
 
@@ -659,10 +704,38 @@ module DSPy
         pass_rate: batch_result.pass_rate,
         score: batch_result.score
       })
-    rescue => e
+
+      # Export batch score to Langfuse if enabled
+      if @export_scores
+        export_batch_score(batch_result)
+      end
+    rescue StandardError => e
       DSPy.log('evals.batch.observation_error', error: e.message)
     end
 
+    def export_example_score(example, result)
+      score_value = result.metrics[:score] || (result.passed ? 1.0 : 0.0)
+      example_id = extract_example_id(example)
+
+      DSPy.score(
+        @score_name,
+        score_value,
+        comment: "Example: #{example_id || 'unknown'}, passed: #{result.passed}"
+      )
+    rescue StandardError => e
+      DSPy.log('evals.score_export_error', error: e.message)
+    end
+
+    def export_batch_score(batch_result)
+      DSPy.score(
+        "#{@score_name}_batch",
+        batch_result.pass_rate,
+        comment: "Batch: #{batch_result.passed_examples}/#{batch_result.total_examples} passed"
+      )
+    rescue StandardError => e
+      DSPy.log('evals.batch_score_export_error', error: e.message)
+    end
+
     def extract_example_id(example)
       if example.respond_to?(:id)
         example.id

metadata

@@ -1,29 +1,28 @@
 --- !ruby/object:Gem::Specification
 name: dspy-evals
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.0.2
 platform: ruby
 authors:
 - Vicente Reig Rincón de Arellano
-autorequire: 
 bindir: bin
 cert_chain: []
-date: 2025-10-25 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: dspy
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '='
+    - - ">="
       - !ruby/object:Gem::Version
-        version: 0.30.0
+        version: '0.30'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '='
+    - - ">="
       - !ruby/object:Gem::Version
-        version: 0.30.0
+        version: '0.30'
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
   requirement: !ruby/object:Gem::Requirement
@@ -69,7 +68,6 @@ licenses:
 - MIT
 metadata:
   github_repo: git@github.com:vicentereig/dspy.rb
-post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
@@ -84,8 +82,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3.1
-signing_key: 
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Evaluation utilities for DSPy.rb programs.
 test_files: []