dspy-miprov2 0.29.1 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +120 -100
- data/lib/dspy/miprov2/version.rb +1 -1
- data/lib/dspy/teleprompt/mipro_v2.rb +22 -23
- metadata +8 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 6c48ce619eb3b88c83c570f560457bb4ab97d273e53bcbb438418a8a158b4496
|
|
4
|
+
data.tar.gz: 063a655057176ea88d9cfbfe74e8d3d19603b24919258c9cf80a958666607aa4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 52d33c957780e5eb01796ce51bcdbfb786e8b05a3fda7b44e1e58d928045452d77060108e43b7113c6bce96f02ecb9fd25826cb3aeb78684df6783937aed29f5
|
|
7
|
+
data.tar.gz: 10460b0582588436f478cab0250349eee7ef4f7bb18394f8fb387eda0a59295d26718065bdc5500ea952522fa0ebac3598ae049c8737d21c682922b500df0882
|
data/README.md
CHANGED
|
@@ -5,26 +5,79 @@
|
|
|
5
5
|
[](https://github.com/vicentereig/dspy.rb/actions/workflows/ruby.yml)
|
|
6
6
|
[](https://vicentereig.github.io/dspy.rb/)
|
|
7
7
|
|
|
8
|
+
> [!NOTE]
|
|
9
|
+
> The core Prompt Engineering Framework is production-ready with
|
|
10
|
+
> comprehensive documentation. I am focusing now on educational content on systematic Prompt Optimization and Context Engineering.
|
|
11
|
+
> Your feedback is invaluable. if you encounter issues, please open an [issue](https://github.com/vicentereig/dspy.rb/issues). If you have suggestions, open a [new thread](https://github.com/vicentereig/dspy.rb/discussions).
|
|
12
|
+
>
|
|
13
|
+
> If you want to contribute, feel free to reach out to me to coordinate efforts: hey at vicente.services
|
|
14
|
+
>
|
|
15
|
+
> And, yes, this is 100% a legit project. :)
|
|
16
|
+
|
|
17
|
+
|
|
8
18
|
**Build reliable LLM applications in idiomatic Ruby using composable, type-safe modules.**
|
|
9
19
|
|
|
10
|
-
The Ruby framework for programming with large language models. DSPy.rb brings structured LLM programming to Ruby developers
|
|
20
|
+
The Ruby framework for programming with large language models. DSPy.rb brings structured LLM programming to Ruby developers, programmatic Prompt Engineering and Context Engineering.
|
|
21
|
+
Instead of wrestling with prompt strings and parsing responses, you define typed signatures using idiomatic Ruby to compose and decompose AI Worklows and AI Agents.
|
|
11
22
|
|
|
12
23
|
**Prompts are the just Functions.** Traditional prompting is like writing code with string concatenation: it works until it doesn't. DSPy.rb brings you
|
|
13
24
|
the programming approach pioneered by [dspy.ai](https://dspy.ai/): instead of crafting fragile prompts, you define modular
|
|
14
25
|
signatures and let the framework handle the messy details.
|
|
15
26
|
|
|
16
27
|
DSPy.rb is an idiomatic Ruby surgical port of Stanford's [DSPy framework](https://github.com/stanfordnlp/dspy). While implementing
|
|
17
|
-
the core concepts of signatures, predictors, and optimization from the original Python library, DSPy.rb embraces Ruby
|
|
18
|
-
conventions and adds Ruby-specific innovations like
|
|
28
|
+
the core concepts of signatures, predictors, and the main optimization algorithms from the original Python library, DSPy.rb embraces Ruby
|
|
29
|
+
conventions and adds Ruby-specific innovations like Sorbet-base Typed system, ReAct loops, and production-ready integrations like non-blocking Open Telemetry Instrumentation.
|
|
19
30
|
|
|
20
|
-
|
|
31
|
+
**What you get?** Ruby LLM applications that actually scale and don't break when you sneeze.
|
|
32
|
+
|
|
33
|
+
Check the [examples](examples/) and take them for a spin!
|
|
21
34
|
|
|
22
35
|
## Your First DSPy Program
|
|
36
|
+
### Installation
|
|
37
|
+
|
|
38
|
+
Add to your Gemfile:
|
|
39
|
+
|
|
40
|
+
```ruby
|
|
41
|
+
gem 'dspy'
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
and
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
bundle install
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
### Optional Sibling Gems
|
|
51
|
+
|
|
52
|
+
DSPy.rb ships multiple gems from this monorepo so you only install what you need. Add these alongside `dspy`:
|
|
53
|
+
|
|
54
|
+
| Gem | Description | Status |
|
|
55
|
+
| --- | --- | --- |
|
|
56
|
+
| `dspy-schema` | Exposes `DSPy::TypeSystem::SorbetJsonSchema` for downstream reuse. | **Stable** (v1.0.0) |
|
|
57
|
+
| `dspy-code_act` | Think-Code-Observe agents that synthesize and execute Ruby safely. | Preview (0.x) |
|
|
58
|
+
| `dspy-datasets` | Dataset helpers plus Parquet/Polars tooling for richer evaluation corpora. | Preview (0.x) |
|
|
59
|
+
| `dspy-evals` | High-throughput evaluation harness with metrics, callbacks, and regression fixtures. | Preview (0.x) |
|
|
60
|
+
| `dspy-miprov2` | Bayesian optimization + Gaussian Process backend for the MIPROv2 teleprompter. | Preview (0.x) |
|
|
61
|
+
| `dspy-gepa` | `DSPy::Teleprompt::GEPA`, reflection loops, experiment tracking, telemetry adapters. | Preview (mirrors `dspy` version) |
|
|
62
|
+
| `gepa` | GEPA optimizer core (Pareto engine, telemetry, reflective proposer). | Preview (mirrors `dspy` version) |
|
|
63
|
+
| `dspy-o11y` | Core observability APIs: `DSPy::Observability`, async span processor, observation types. | **Stable** (v1.0.0) |
|
|
64
|
+
| `dspy-o11y-langfuse` | Auto-configures DSPy observability to stream spans to Langfuse via OTLP. | **Stable** (v1.0.0) |
|
|
65
|
+
|
|
66
|
+
Set the matching `DSPY_WITH_*` environment variables (see `Gemfile`) to include or exclude each sibling gem when running Bundler locally (for example `DSPY_WITH_GEPA=1` or `DSPY_WITH_O11Y_LANGFUSE=1`). Refer to `docs/core-concepts/dependency-tree.md` for the full dependency map and roadmap.
|
|
67
|
+
### Your First Reliable Predictor
|
|
23
68
|
|
|
24
69
|
```ruby
|
|
25
|
-
|
|
70
|
+
|
|
71
|
+
# Configure DSPy globablly to use your fave LLM - you can override this on an instance levle.
|
|
72
|
+
DSPy.configure do |c|
|
|
73
|
+
c.lm = DSPy::LM.new('openai/gpt-4o-mini',
|
|
74
|
+
api_key: ENV['OPENAI_API_KEY'],
|
|
75
|
+
structured_outputs: true) # Enable OpenAI's native JSON mode
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# Define a signature for sentiment classification - instead of writing a full prompt!
|
|
26
79
|
class Classify < DSPy::Signature
|
|
27
|
-
description "Classify sentiment of a given sentence."
|
|
80
|
+
description "Classify sentiment of a given sentence." # sets the goal of the underlying prompt
|
|
28
81
|
|
|
29
82
|
class Sentiment < T::Enum
|
|
30
83
|
enums do
|
|
@@ -33,26 +86,22 @@ class Classify < DSPy::Signature
|
|
|
33
86
|
Neutral = new('neutral')
|
|
34
87
|
end
|
|
35
88
|
end
|
|
36
|
-
|
|
89
|
+
|
|
90
|
+
# Structured Inputs: makes sure you are sending only valid prompt inputs to your model
|
|
37
91
|
input do
|
|
38
|
-
const :sentence, String
|
|
92
|
+
const :sentence, String, description: 'The sentence to analyze'
|
|
39
93
|
end
|
|
40
94
|
|
|
95
|
+
# Structured Outputs: your predictor will validate the output of the model too.
|
|
41
96
|
output do
|
|
42
|
-
const :sentiment, Sentiment
|
|
43
|
-
const :confidence, Float
|
|
97
|
+
const :sentiment, Sentiment, description: 'The sentiment of the sentence'
|
|
98
|
+
const :confidence, Float, description: 'A number between 0.0 and 1.0'
|
|
44
99
|
end
|
|
45
100
|
end
|
|
46
101
|
|
|
47
|
-
#
|
|
48
|
-
DSPy.configure do |c|
|
|
49
|
-
c.lm = DSPy::LM.new('openai/gpt-4o-mini',
|
|
50
|
-
api_key: ENV['OPENAI_API_KEY'],
|
|
51
|
-
structured_outputs: true) # Enable OpenAI's native JSON mode
|
|
52
|
-
end
|
|
53
|
-
|
|
54
|
-
# Create the predictor and run inference
|
|
102
|
+
# Wire it to the simplest prompting technique - a Predictn.
|
|
55
103
|
classify = DSPy::Predict.new(Classify)
|
|
104
|
+
# it may raise an error if you mess the inputs or your LLM messes the outputs.
|
|
56
105
|
result = classify.call(sentence: "This book was super fun to read!")
|
|
57
106
|
|
|
58
107
|
puts result.sentiment # => #<Sentiment::Positive>
|
|
@@ -99,12 +148,22 @@ end
|
|
|
99
148
|
|
|
100
149
|
## What You Get
|
|
101
150
|
|
|
151
|
+
**Developer Experience:**
|
|
152
|
+
- LLM provider support using official Ruby clients:
|
|
153
|
+
- [OpenAI Ruby](https://github.com/openai/openai-ruby) with vision model support
|
|
154
|
+
- [Anthropic Ruby SDK](https://github.com/anthropics/anthropic-sdk-ruby) with multimodal capabilities
|
|
155
|
+
- [Google Gemini API](https://ai.google.dev/) with native structured outputs
|
|
156
|
+
- [Ollama](https://ollama.com/) via OpenAI compatibility layer for local models
|
|
157
|
+
- **Multimodal Support** - Complete image analysis with DSPy::Image, type-safe bounding boxes, vision-capable models
|
|
158
|
+
- Runtime type checking with [Sorbet](https://sorbet.org/) including T::Enum and union types
|
|
159
|
+
- Type-safe tool definitions for ReAct agents
|
|
160
|
+
- Comprehensive instrumentation and observability
|
|
161
|
+
|
|
102
162
|
**Core Building Blocks:**
|
|
103
163
|
- **Signatures** - Define input/output schemas using Sorbet types with T::Enum and union type support
|
|
104
164
|
- **Predict** - LLM completion with structured data extraction and multimodal support
|
|
105
165
|
- **Chain of Thought** - Step-by-step reasoning for complex problems with automatic prompt optimization
|
|
106
166
|
- **ReAct** - Tool-using agents with type-safe tool definitions and error recovery
|
|
107
|
-
- **CodeAct** - Dynamic code execution agents for programming tasks
|
|
108
167
|
- **Module Composition** - Combine multiple LLM calls into production-ready workflows
|
|
109
168
|
|
|
110
169
|
**Optimization & Evaluation:**
|
|
@@ -122,24 +181,40 @@ end
|
|
|
122
181
|
- **File-based Storage** - Optimization result persistence with versioning
|
|
123
182
|
- **Structured Logging** - JSON and key=value formats with span tracking
|
|
124
183
|
|
|
125
|
-
|
|
126
|
-
- LLM provider support using official Ruby clients:
|
|
127
|
-
- [OpenAI Ruby](https://github.com/openai/openai-ruby) with vision model support
|
|
128
|
-
- [Anthropic Ruby SDK](https://github.com/anthropics/anthropic-sdk-ruby) with multimodal capabilities
|
|
129
|
-
- [Google Gemini API](https://ai.google.dev/) with native structured outputs
|
|
130
|
-
- [Ollama](https://ollama.com/) via OpenAI compatibility layer for local models
|
|
131
|
-
- **Multimodal Support** - Complete image analysis with DSPy::Image, type-safe bounding boxes, vision-capable models
|
|
132
|
-
- Runtime type checking with [Sorbet](https://sorbet.org/) including T::Enum and union types
|
|
133
|
-
- Type-safe tool definitions for ReAct agents
|
|
134
|
-
- Comprehensive instrumentation and observability
|
|
184
|
+
## Recent Achievements
|
|
135
185
|
|
|
136
|
-
|
|
186
|
+
DSPy.rb has rapidly evolved from experimental to production-ready:
|
|
137
187
|
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
188
|
+
### Foundation
|
|
189
|
+
- ✅ **JSON Parsing Reliability** - Native OpenAI structured outputs with adaptive retry logic and schema-aware fallbacks
|
|
190
|
+
- ✅ **Type-Safe Strategy Configuration** - Provider-optimized strategy selection and enum-backed optimizer presets
|
|
191
|
+
- ✅ **Core Module System** - Predict, ChainOfThought, ReAct with type safety (add `dspy-code_act` for Think-Code-Observe agents)
|
|
192
|
+
- ✅ **Production Observability** - OpenTelemetry, New Relic, and Langfuse integration
|
|
193
|
+
- ✅ **Advanced Optimization** - MIPROv2 with Bayesian optimization, Gaussian Processes, and multi-mode search
|
|
194
|
+
|
|
195
|
+
### Recent Advances
|
|
196
|
+
- ✅ **MIPROv2 ADE Integrity (v0.29.1)** - Stratified train/val/test splits, honest precision accounting, and enum-driven `--auto` presets with integration coverage
|
|
197
|
+
- ✅ **Instruction Deduplication (v0.29.1)** - Candidate generation now filters repeated programs so optimization logs highlight unique strategies
|
|
198
|
+
- ✅ **GEPA Teleprompter (v0.29.0)** - Genetic-Pareto reflective prompt evolution with merge proposer scheduling, reflective mutation, and ADE demo parity
|
|
199
|
+
- ✅ **Optimizer Utilities Parity (v0.29.0)** - Bootstrap strategies, dataset summaries, and Layer 3 utilities unlock multi-predictor programs on Ruby
|
|
200
|
+
- ✅ **Observability Hardening (v0.29.0)** - OTLP exporter runs on a single-thread executor preventing frozen SSL contexts without blocking spans
|
|
201
|
+
- ✅ **Documentation Refresh (v0.29.x)** - New GEPA guide plus ADE optimization docs covering presets, stratified splits, and error-handling defaults
|
|
202
|
+
|
|
203
|
+
**Current Focus Areas:**
|
|
204
|
+
|
|
205
|
+
### Production Readiness
|
|
206
|
+
- 🚧 **Production Patterns** - Real-world usage validation and performance optimization
|
|
207
|
+
- 🚧 **Ruby Ecosystem Integration** - Rails integration, Sidekiq compatibility, deployment patterns
|
|
208
|
+
|
|
209
|
+
### Community & Adoption
|
|
210
|
+
- 🚧 **Community Examples** - Real-world applications and case studies
|
|
211
|
+
- 🚧 **Contributor Experience** - Making it easier to contribute and extend
|
|
212
|
+
- 🚧 **Performance Benchmarks** - Comparative analysis vs other frameworks
|
|
213
|
+
|
|
214
|
+
**v1.0 Philosophy:**
|
|
215
|
+
v1.0 will be released after extensive production battle-testing, not after checking off features.
|
|
216
|
+
The API is already stable - v1.0 represents confidence in production reliability backed by real-world validation.
|
|
141
217
|
|
|
142
|
-
Real-world usage feedback is invaluable - if you encounter issues or have suggestions, please open a GitHub issue!
|
|
143
218
|
|
|
144
219
|
## Documentation
|
|
145
220
|
|
|
@@ -156,92 +231,37 @@ For LLMs and AI assistants working with DSPy.rb:
|
|
|
156
231
|
- **[Quick Start Guide](docs/src/getting-started/quick-start.md)** - Your first DSPy programs
|
|
157
232
|
- **[Core Concepts](docs/src/getting-started/core-concepts.md)** - Understanding signatures, predictors, and modules
|
|
158
233
|
|
|
159
|
-
###
|
|
234
|
+
### Prompt Engineering
|
|
160
235
|
- **[Signatures & Types](docs/src/core-concepts/signatures.md)** - Define typed interfaces for LLM operations
|
|
161
236
|
- **[Predictors](docs/src/core-concepts/predictors.md)** - Predict, ChainOfThought, ReAct, and more
|
|
162
237
|
- **[Modules & Pipelines](docs/src/core-concepts/modules.md)** - Compose complex multi-stage workflows
|
|
163
238
|
- **[Multimodal Support](docs/src/core-concepts/multimodal.md)** - Image analysis with vision-capable models
|
|
164
239
|
- **[Examples & Validation](docs/src/core-concepts/examples.md)** - Type-safe training data
|
|
240
|
+
- **[Rich Types](docs/src/advanced/complex-types.md)** - Sorbet type integration with automatic coercion for structs, enums, and arrays
|
|
241
|
+
- **[Composable Pipelines](docs/src/advanced/pipelines.md)** - Manual module composition patterns
|
|
165
242
|
|
|
166
|
-
### Optimization
|
|
243
|
+
### Prompt Optimization
|
|
167
244
|
- **[Evaluation Framework](docs/src/optimization/evaluation.md)** - Advanced metrics beyond simple accuracy
|
|
168
245
|
- **[Prompt Optimization](docs/src/optimization/prompt-optimization.md)** - Manipulate prompts as objects
|
|
169
246
|
- **[MIPROv2 Optimizer](docs/src/optimization/miprov2.md)** - Advanced Bayesian optimization with Gaussian Processes
|
|
170
247
|
- **[GEPA Optimizer](docs/src/optimization/gepa.md)** *(beta)* - Reflective mutation with optional reflection LMs
|
|
171
248
|
|
|
172
|
-
###
|
|
173
|
-
- **[
|
|
174
|
-
- **[
|
|
175
|
-
|
|
176
|
-
### Advanced Usage
|
|
177
|
-
- **[Complex Types](docs/src/advanced/complex-types.md)** - Sorbet type integration with automatic coercion for structs, enums, and arrays
|
|
178
|
-
- **[Manual Pipelines](docs/src/advanced/pipelines.md)** - Manual module composition patterns
|
|
249
|
+
### Context Engineering
|
|
250
|
+
- **[Tools](docs/src/core-concepts/toolsets.md)** - Tool wieldint agents.
|
|
251
|
+
- **[Agentic Memory](docs/src/core-concepts/memory.md)** - Memory Tools & Agentic Loops
|
|
179
252
|
- **[RAG Patterns](docs/src/advanced/rag.md)** - Manual RAG implementation with external services
|
|
180
|
-
- **[Custom Metrics](docs/src/advanced/custom-metrics.md)** - Proc-based evaluation logic
|
|
181
|
-
|
|
182
|
-
## Quick Start
|
|
183
|
-
|
|
184
|
-
### Installation
|
|
185
|
-
|
|
186
|
-
Add to your Gemfile:
|
|
187
|
-
|
|
188
|
-
```ruby
|
|
189
|
-
gem 'dspy'
|
|
190
|
-
```
|
|
191
|
-
|
|
192
|
-
Then run:
|
|
193
|
-
|
|
194
|
-
```bash
|
|
195
|
-
bundle install
|
|
196
|
-
```
|
|
197
|
-
|
|
198
|
-
## Recent Achievements
|
|
199
|
-
|
|
200
|
-
DSPy.rb has rapidly evolved from experimental to production-ready:
|
|
201
|
-
|
|
202
|
-
### Foundation
|
|
203
|
-
- ✅ **JSON Parsing Reliability** - Native OpenAI structured outputs, strategy selection, retry logic
|
|
204
|
-
- ✅ **Type-Safe Strategy Configuration** - Provider-optimized automatic strategy selection
|
|
205
|
-
- ✅ **Core Module System** - Predict, ChainOfThought, ReAct, CodeAct with type safety
|
|
206
|
-
- ✅ **Production Observability** - OpenTelemetry, New Relic, and Langfuse integration
|
|
207
|
-
- ✅ **Advanced Optimization** - MIPROv2 with Bayesian optimization, Gaussian Processes, and multiple strategies
|
|
208
253
|
|
|
209
|
-
###
|
|
210
|
-
-
|
|
211
|
-
-
|
|
212
|
-
-
|
|
213
|
-
- ✅ **Production-Ready Evaluation** - Multi-factor metrics beyond accuracy, error-resilient evaluation pipelines
|
|
214
|
-
- ✅ **Documentation Ecosystem** - `llms.txt` for AI assistants, ADRs, blog articles, comprehensive examples
|
|
215
|
-
- ✅ **API Maturation** - Simplified idiomatic patterns, better error handling, production-proven designs
|
|
254
|
+
### Production Features
|
|
255
|
+
- **[Observability](docs/src/production/observability.md)** - Zero-config Langfuse integration with a dedicated export worker that never blocks your LLMs
|
|
256
|
+
- **[Storage System](docs/src/production/storage.md)** - Persistence and optimization result storage
|
|
257
|
+
- **[Custom Metrics](docs/src/advanced/custom-metrics.md)** - Proc-based evaluation logic
|
|
216
258
|
|
|
217
|
-
## Roadmap - Production Battle-Testing Toward v1.0
|
|
218
259
|
|
|
219
|
-
DSPy.rb has transitioned from **feature building** to **production validation**. The core framework is
|
|
220
|
-
feature-complete and stable - now I'm focusing on real-world usage patterns, performance optimization,
|
|
221
|
-
and ecosystem integration.
|
|
222
260
|
|
|
223
|
-
**Current Focus Areas:**
|
|
224
261
|
|
|
225
|
-
### Production Readiness
|
|
226
|
-
- 🚧 **Production Patterns** - Real-world usage validation and performance optimization
|
|
227
|
-
- 🚧 **Ruby Ecosystem Integration** - Rails integration, Sidekiq compatibility, deployment patterns
|
|
228
|
-
- 🚧 **Scale Testing** - High-volume usage, memory management, connection pooling
|
|
229
|
-
- 🚧 **Error Recovery** - Robust failure handling patterns for production environments
|
|
230
262
|
|
|
231
|
-
### Ecosystem Expansion
|
|
232
|
-
- 🚧 **Model Context Protocol (MCP)** - Integration with MCP ecosystem
|
|
233
|
-
- 🚧 **Additional Provider Support** - Azure OpenAI, local models beyond Ollama
|
|
234
|
-
- 🚧 **Tool Ecosystem** - Expanded tool integrations for ReAct agents
|
|
235
263
|
|
|
236
|
-
### Community & Adoption
|
|
237
|
-
- 🚧 **Community Examples** - Real-world applications and case studies
|
|
238
|
-
- 🚧 **Contributor Experience** - Making it easier to contribute and extend
|
|
239
|
-
- 🚧 **Performance Benchmarks** - Comparative analysis vs other frameworks
|
|
240
264
|
|
|
241
|
-
**v1.0 Philosophy:**
|
|
242
|
-
v1.0 will be released after extensive production battle-testing, not after checking off features.
|
|
243
|
-
The API is already stable - v1.0 represents confidence in production reliability backed by real-world validation.
|
|
244
265
|
|
|
245
266
|
## License
|
|
246
|
-
|
|
247
267
|
This project is licensed under the MIT License.
|
data/lib/dspy/miprov2/version.rb
CHANGED
|
@@ -9,6 +9,7 @@ require 'securerandom'
|
|
|
9
9
|
require 'set'
|
|
10
10
|
require_relative 'teleprompter'
|
|
11
11
|
require_relative 'utils'
|
|
12
|
+
require_relative 'instruction_updates'
|
|
12
13
|
require_relative '../propose/grounded_proposer'
|
|
13
14
|
require_relative '../optimizers/gaussian_process'
|
|
14
15
|
|
|
@@ -272,7 +273,7 @@ module DSPy
|
|
|
272
273
|
sig { returns(T::Hash[Symbol, T.untyped]) }
|
|
273
274
|
attr_reader :proposal_statistics
|
|
274
275
|
|
|
275
|
-
sig { returns(T.nilable(DSPy::
|
|
276
|
+
sig { returns(T.nilable(DSPy::Evals::BatchEvaluationResult)) }
|
|
276
277
|
attr_reader :best_evaluation_result
|
|
277
278
|
|
|
278
279
|
sig do
|
|
@@ -287,7 +288,7 @@ module DSPy
|
|
|
287
288
|
best_score_name: T.nilable(String),
|
|
288
289
|
best_score_value: T.nilable(Float),
|
|
289
290
|
metadata: T::Hash[Symbol, T.untyped],
|
|
290
|
-
best_evaluation_result: T.nilable(DSPy::
|
|
291
|
+
best_evaluation_result: T.nilable(DSPy::Evals::BatchEvaluationResult)
|
|
291
292
|
).void
|
|
292
293
|
end
|
|
293
294
|
def initialize(optimized_program:, scores:, history:, evaluated_candidates:, optimization_trace:, bootstrap_statistics:, proposal_statistics:, best_score_name: nil, best_score_value: nil, metadata: {}, best_evaluation_result: nil)
|
|
@@ -950,15 +951,6 @@ module DSPy
|
|
|
950
951
|
end
|
|
951
952
|
end
|
|
952
953
|
|
|
953
|
-
sig { params(predictor: T.untyped, examples: T::Array[DSPy::FewShotExample]).void }
|
|
954
|
-
def assign_predictor_examples(predictor, examples)
|
|
955
|
-
predictor.demos = examples if predictor.respond_to?(:demos=)
|
|
956
|
-
return unless predictor.respond_to?(:prompt)
|
|
957
|
-
|
|
958
|
-
cloned_examples = examples.map { |ex| ex }
|
|
959
|
-
predictor.prompt.instance_variable_set(:@few_shot_examples, cloned_examples.freeze)
|
|
960
|
-
end
|
|
961
|
-
|
|
962
954
|
# Initialize optimization state for candidate selection
|
|
963
955
|
sig { params(candidates: T::Array[EvaluatedCandidate]).returns(T::Hash[Symbol, T.untyped]) }
|
|
964
956
|
def initialize_optimization_state(candidates)
|
|
@@ -1131,7 +1123,7 @@ module DSPy
|
|
|
1131
1123
|
program: T.untyped,
|
|
1132
1124
|
candidate: EvaluatedCandidate,
|
|
1133
1125
|
evaluation_set: T::Array[DSPy::Example]
|
|
1134
|
-
).returns([Float, T.untyped, DSPy::
|
|
1126
|
+
).returns([Float, T.untyped, DSPy::Evals::BatchEvaluationResult])
|
|
1135
1127
|
end
|
|
1136
1128
|
def evaluate_candidate(program, candidate, evaluation_set)
|
|
1137
1129
|
# Apply candidate configuration to program
|
|
@@ -1163,7 +1155,7 @@ module DSPy
|
|
|
1163
1155
|
params(
|
|
1164
1156
|
modified_program: T.untyped,
|
|
1165
1157
|
evaluation_set: T::Array[DSPy::Example]
|
|
1166
|
-
).returns(DSPy::
|
|
1158
|
+
).returns(DSPy::Evals::BatchEvaluationResult)
|
|
1167
1159
|
end
|
|
1168
1160
|
def evaluate_candidate_concurrently(modified_program, evaluation_set)
|
|
1169
1161
|
chunk_size = T.must(config.minibatch_size)
|
|
@@ -1190,16 +1182,16 @@ module DSPy
|
|
|
1190
1182
|
end
|
|
1191
1183
|
|
|
1192
1184
|
sig do
|
|
1193
|
-
params(batch_results: T::Array[DSPy::
|
|
1185
|
+
params(batch_results: T::Array[DSPy::Evals::BatchEvaluationResult]).returns(DSPy::Evals::BatchEvaluationResult)
|
|
1194
1186
|
end
|
|
1195
1187
|
def combine_batch_results(batch_results)
|
|
1196
|
-
return DSPy::
|
|
1188
|
+
return DSPy::Evals::BatchEvaluationResult.new(results: [], aggregated_metrics: {}) if batch_results.empty?
|
|
1197
1189
|
|
|
1198
1190
|
combined_results = batch_results.flat_map(&:results)
|
|
1199
1191
|
total_examples = batch_results.sum(&:total_examples)
|
|
1200
1192
|
aggregated_metrics = merge_aggregated_metrics(batch_results, total_examples)
|
|
1201
1193
|
|
|
1202
|
-
DSPy::
|
|
1194
|
+
DSPy::Evals::BatchEvaluationResult.new(
|
|
1203
1195
|
results: combined_results,
|
|
1204
1196
|
aggregated_metrics: aggregated_metrics
|
|
1205
1197
|
)
|
|
@@ -1207,7 +1199,7 @@ module DSPy
|
|
|
1207
1199
|
|
|
1208
1200
|
sig do
|
|
1209
1201
|
params(
|
|
1210
|
-
batch_results: T::Array[DSPy::
|
|
1202
|
+
batch_results: T::Array[DSPy::Evals::BatchEvaluationResult],
|
|
1211
1203
|
total_examples: Integer
|
|
1212
1204
|
).returns(T::Hash[Symbol, T.untyped])
|
|
1213
1205
|
end
|
|
@@ -1251,28 +1243,35 @@ module DSPy
|
|
|
1251
1243
|
modified_program = modified_program.clone
|
|
1252
1244
|
modified_program.predictors.each_with_index do |predictor, idx|
|
|
1253
1245
|
if instructions_map.key?(idx)
|
|
1254
|
-
|
|
1255
|
-
|
|
1256
|
-
|
|
1246
|
+
modified_program, predictor = InstructionUpdates.apply_instruction(
|
|
1247
|
+
modified_program,
|
|
1248
|
+
predictor,
|
|
1249
|
+
instructions_map[idx]
|
|
1250
|
+
)
|
|
1257
1251
|
end
|
|
1258
1252
|
|
|
1259
1253
|
if demos_map.key?(idx)
|
|
1260
1254
|
normalized_examples = normalize_few_shot_examples(demos_map[idx])
|
|
1261
|
-
|
|
1255
|
+
modified_program, predictor = InstructionUpdates.apply_examples(
|
|
1256
|
+
modified_program,
|
|
1257
|
+
predictor,
|
|
1258
|
+
normalized_examples
|
|
1259
|
+
)
|
|
1262
1260
|
end
|
|
1263
1261
|
end
|
|
1264
1262
|
end
|
|
1265
1263
|
|
|
1266
1264
|
# Apply instruction if provided (top-level programs still respect with_instruction)
|
|
1267
|
-
if !candidate.instruction.empty?
|
|
1265
|
+
if !candidate.instruction.empty?
|
|
1266
|
+
InstructionUpdates.ensure_instruction_capability!(modified_program)
|
|
1268
1267
|
modified_program = modified_program.with_instruction(candidate.instruction)
|
|
1269
1268
|
end
|
|
1270
1269
|
|
|
1271
1270
|
should_apply_global_examples = candidate.few_shot_examples.any? &&
|
|
1272
|
-
modified_program.respond_to?(:with_examples) &&
|
|
1273
1271
|
(demos_map.empty? || !modified_program.respond_to?(:predictors))
|
|
1274
1272
|
|
|
1275
1273
|
if should_apply_global_examples
|
|
1274
|
+
InstructionUpdates.ensure_examples_capability!(modified_program)
|
|
1276
1275
|
normalized_few_shot = normalize_few_shot_examples(candidate.few_shot_examples)
|
|
1277
1276
|
modified_program = modified_program.with_examples(normalized_few_shot)
|
|
1278
1277
|
end
|
metadata
CHANGED
|
@@ -1,13 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: dspy-miprov2
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 1.0.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Vicente Reig Rincón de Arellano
|
|
8
|
+
autorequire:
|
|
8
9
|
bindir: bin
|
|
9
10
|
cert_chain: []
|
|
10
|
-
date: 2025-10-
|
|
11
|
+
date: 2025-10-25 00:00:00.000000000 Z
|
|
11
12
|
dependencies:
|
|
12
13
|
- !ruby/object:Gem::Dependency
|
|
13
14
|
name: dspy
|
|
@@ -15,14 +16,14 @@ dependencies:
|
|
|
15
16
|
requirements:
|
|
16
17
|
- - '='
|
|
17
18
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: 0.
|
|
19
|
+
version: 0.30.0
|
|
19
20
|
type: :runtime
|
|
20
21
|
prerelease: false
|
|
21
22
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
23
|
requirements:
|
|
23
24
|
- - '='
|
|
24
25
|
- !ruby/object:Gem::Version
|
|
25
|
-
version: 0.
|
|
26
|
+
version: 0.30.0
|
|
26
27
|
- !ruby/object:Gem::Dependency
|
|
27
28
|
name: numo-narray-alt
|
|
28
29
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -70,6 +71,7 @@ licenses:
|
|
|
70
71
|
- MIT
|
|
71
72
|
metadata:
|
|
72
73
|
github_repo: git@github.com:vicentereig/dspy.rb
|
|
74
|
+
post_install_message:
|
|
73
75
|
rdoc_options: []
|
|
74
76
|
require_paths:
|
|
75
77
|
- lib
|
|
@@ -84,7 +86,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
84
86
|
- !ruby/object:Gem::Version
|
|
85
87
|
version: '0'
|
|
86
88
|
requirements: []
|
|
87
|
-
rubygems_version: 3.
|
|
89
|
+
rubygems_version: 3.0.3.1
|
|
90
|
+
signing_key:
|
|
88
91
|
specification_version: 4
|
|
89
92
|
summary: MIPROv2 optimizer and Bayesian tooling for DSPy.rb.
|
|
90
93
|
test_files: []
|