dspy-o11y-langfuse 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +84 -43
- data/lib/dspy/o11y/langfuse/scores_exporter.rb +222 -0
- data/lib/dspy/o11y/langfuse/version.rb +1 -1
- data/lib/dspy/o11y/langfuse.rb +3 -0
- metadata +8 -7
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 681cd7bc2d72f935e3e86ba63e8f7846ff385cf98c485e49ed8c67065eb5e254
|
|
4
|
+
data.tar.gz: '03792ca1ab58794ab07be57da2ff73c1f522212114b0fec18db8cea674d74e22'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 5ae475d8c8d502806ea6c7edb647dc8378b9a68d35ebe9972717a0aac821a5d769e30147b0796efa7d94ead8e836e6f2169bfa0bb767384a14ad1190fc9bf12f
|
|
7
|
+
data.tar.gz: 3d9d35c62ead40d58d87ccf4eca2e0a23c811a864ed3e44f0c8ba23bab3c3e50e3406a7888012925c7f74af014ab9299ce33bfa3f6b66549424ca63c6f817413
|
data/README.md
CHANGED
|
@@ -3,7 +3,8 @@
|
|
|
3
3
|
[](https://rubygems.org/gems/dspy)
|
|
4
4
|
[](https://rubygems.org/gems/dspy)
|
|
5
5
|
[](https://github.com/vicentereig/dspy.rb/actions/workflows/ruby.yml)
|
|
6
|
-
[](https://oss.vicente.services/dspy.rb/)
|
|
7
|
+
[](https://discord.gg/zWBhrMqn)
|
|
7
8
|
|
|
8
9
|
> [!NOTE]
|
|
9
10
|
> The core Prompt Engineering Framework is production-ready with
|
|
@@ -12,23 +13,16 @@
|
|
|
12
13
|
>
|
|
13
14
|
> If you want to contribute, feel free to reach out to me to coordinate efforts: hey at vicente.services
|
|
14
15
|
>
|
|
15
|
-
> And, yes, this is 100% a legit project. :)
|
|
16
|
-
|
|
17
16
|
|
|
18
17
|
**Build reliable LLM applications in idiomatic Ruby using composable, type-safe modules.**
|
|
19
18
|
|
|
20
|
-
|
|
21
|
-
Instead of wrestling with prompt strings and parsing responses, you define typed signatures using idiomatic Ruby to compose and decompose AI Worklows and AI Agents.
|
|
19
|
+
DSPy.rb is the Ruby-first surgical port of Stanford's [DSPy paradigm](https://github.com/stanfordnlp/dspy). It delivers structured LLM programming, prompt engineering, and context engineering in the language we love. Instead of wrestling with brittle prompt strings, you define typed signatures in idiomatic Ruby and compose workflows and agents that actually behave.
|
|
22
20
|
|
|
23
|
-
**Prompts are
|
|
24
|
-
the programming approach pioneered by [dspy.ai](https://dspy.ai/): instead of crafting fragile prompts, you define modular
|
|
25
|
-
signatures and let the framework handle the messy details.
|
|
21
|
+
**Prompts are just functions.** Traditional prompting is like writing code with string concatenation: it works until it doesn't. DSPy.rb brings you the programming approach pioneered by [dspy.ai](https://dspy.ai/): define modular signatures and let the framework deal with the messy bits.
|
|
26
22
|
|
|
27
|
-
DSPy.rb
|
|
28
|
-
the core concepts of signatures, predictors, and the main optimization algorithms from the original Python library, DSPy.rb embraces Ruby
|
|
29
|
-
conventions and adds Ruby-specific innovations like Sorbet-base Typed system, ReAct loops, and production-ready integrations like non-blocking Open Telemetry Instrumentation.
|
|
23
|
+
While we implement the same signatures, predictors, and optimization algorithms as the original library, DSPy.rb leans hard into Ruby conventions with Sorbet-based typing, ReAct loops, and production-ready integrations like non-blocking OpenTelemetry instrumentation.
|
|
30
24
|
|
|
31
|
-
**What you get?** Ruby LLM applications that
|
|
25
|
+
**What you get?** Ruby LLM applications that scale and don't break when you sneeze.
|
|
32
26
|
|
|
33
27
|
Check the [examples](examples/) and take them for a spin!
|
|
34
28
|
|
|
@@ -47,28 +41,12 @@ and
|
|
|
47
41
|
bundle install
|
|
48
42
|
```
|
|
49
43
|
|
|
50
|
-
### Optional Sibling Gems
|
|
51
|
-
|
|
52
|
-
DSPy.rb ships multiple gems from this monorepo so you only install what you need. Add these alongside `dspy`:
|
|
53
|
-
|
|
54
|
-
| Gem | Description | Status |
|
|
55
|
-
| --- | --- | --- |
|
|
56
|
-
| `dspy-schema` | Exposes `DSPy::TypeSystem::SorbetJsonSchema` for downstream reuse. | **Stable** (v1.0.0) |
|
|
57
|
-
| `dspy-code_act` | Think-Code-Observe agents that synthesize and execute Ruby safely. | Preview (0.x) |
|
|
58
|
-
| `dspy-datasets` | Dataset helpers plus Parquet/Polars tooling for richer evaluation corpora. | Preview (0.x) |
|
|
59
|
-
| `dspy-evals` | High-throughput evaluation harness with metrics, callbacks, and regression fixtures. | Preview (0.x) |
|
|
60
|
-
| `dspy-miprov2` | Bayesian optimization + Gaussian Process backend for the MIPROv2 teleprompter. | Preview (0.x) |
|
|
61
|
-
| `dspy-gepa` | `DSPy::Teleprompt::GEPA`, reflection loops, experiment tracking, telemetry adapters. | Preview (mirrors `dspy` version) |
|
|
62
|
-
| `gepa` | GEPA optimizer core (Pareto engine, telemetry, reflective proposer). | Preview (mirrors `dspy` version) |
|
|
63
|
-
| `dspy-o11y` | Core observability APIs: `DSPy::Observability`, async span processor, observation types. | **Stable** (v1.0.0) |
|
|
64
|
-
| `dspy-o11y-langfuse` | Auto-configures DSPy observability to stream spans to Langfuse via OTLP. | **Stable** (v1.0.0) |
|
|
65
|
-
|
|
66
|
-
Set the matching `DSPY_WITH_*` environment variables (see `Gemfile`) to include or exclude each sibling gem when running Bundler locally (for example `DSPY_WITH_GEPA=1` or `DSPY_WITH_O11Y_LANGFUSE=1`).
|
|
67
44
|
### Your First Reliable Predictor
|
|
68
45
|
|
|
69
46
|
```ruby
|
|
47
|
+
require 'dspy'
|
|
70
48
|
|
|
71
|
-
# Configure DSPy
|
|
49
|
+
# Configure DSPy globally to use your fave LLM (you can override per predictor).
|
|
72
50
|
DSPy.configure do |c|
|
|
73
51
|
c.lm = DSPy::LM.new('openai/gpt-4o-mini',
|
|
74
52
|
api_key: ENV['OPENAI_API_KEY'],
|
|
@@ -99,7 +77,7 @@ class Classify < DSPy::Signature
|
|
|
99
77
|
end
|
|
100
78
|
end
|
|
101
79
|
|
|
102
|
-
# Wire it to the simplest prompting technique
|
|
80
|
+
# Wire it to the simplest prompting technique: a prediction loop.
|
|
103
81
|
classify = DSPy::Predict.new(Classify)
|
|
104
82
|
# it may raise an error if you mess the inputs or your LLM messes the outputs.
|
|
105
83
|
result = classify.call(sentence: "This book was super fun to read!")
|
|
@@ -108,6 +86,38 @@ puts result.sentiment # => #<Sentiment::Positive>
|
|
|
108
86
|
puts result.confidence # => 0.85
|
|
109
87
|
```
|
|
110
88
|
|
|
89
|
+
Save this as `examples/first_predictor.rb` and run it with:
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
bundle exec ruby examples/first_predictor.rb
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
### Sibling Gems
|
|
96
|
+
|
|
97
|
+
DSPy.rb ships multiple gems from this monorepo so you can opt into features with heavier dependency trees (e.g., datasets pull in Polars/Arrow, MIPROv2 requires `numo-*` BLAS bindings) only when you need them. Add these alongside `dspy`:
|
|
98
|
+
|
|
99
|
+
| Gem | Description | Status |
|
|
100
|
+
| --- | --- | --- |
|
|
101
|
+
| `dspy-schema` | Exposes `DSPy::TypeSystem::SorbetJsonSchema` for downstream reuse. (Still required by the core `dspy` gem; extraction lets other projects depend on it directly.) | **Stable** (v1.0.0) |
|
|
102
|
+
| `dspy-openai` | Packages the OpenAI/OpenRouter/Ollama adapters plus the official SDK guardrails. Install whenever you call `openai/*`, `openrouter/*`, or `ollama/*`. [Adapter README](https://github.com/vicentereig/dspy.rb/blob/main/lib/dspy/openai/README.md) | **Stable** (v1.0.0) |
|
|
103
|
+
| `dspy-anthropic` | Claude adapters, streaming, and structured-output helpers behind the official `anthropic` SDK. [Adapter README](https://github.com/vicentereig/dspy.rb/blob/main/lib/dspy/anthropic/README.md) | **Stable** (v1.0.0) |
|
|
104
|
+
| `dspy-gemini` | Gemini adapters with multimodal + tool-call support via `gemini-ai`. [Adapter README](https://github.com/vicentereig/dspy.rb/blob/main/lib/dspy/gemini/README.md) | **Stable** (v1.0.0) |
|
|
105
|
+
| `dspy-ruby_llm` | Unified access to 12+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Ollama, DeepSeek, etc.) via [RubyLLM](https://rubyllm.com). [Adapter README](https://github.com/vicentereig/dspy.rb/blob/main/lib/dspy/ruby_llm/README.md) | **Stable** (v0.1.0) |
|
|
106
|
+
| `dspy-code_act` | Think-Code-Observe agents that synthesize and execute Ruby safely. (Add the gem or set `DSPY_WITH_CODE_ACT=1` before requiring `dspy/code_act`.) | **Stable** (v1.0.0) |
|
|
107
|
+
| `dspy-datasets` | Dataset helpers plus Parquet/Polars tooling for richer evaluation corpora. (Toggle via `DSPY_WITH_DATASETS`.) | **Stable** (v1.0.0) |
|
|
108
|
+
| `dspy-evals` | High-throughput evaluation harness with metrics, callbacks, and regression fixtures. (Toggle via `DSPY_WITH_EVALS`.) | **Stable** (v1.0.0) |
|
|
109
|
+
| `dspy-miprov2` | Bayesian optimization + Gaussian Process backend for the MIPROv2 teleprompter. (Install or export `DSPY_WITH_MIPROV2=1` before requiring the teleprompter.) | **Stable** (v1.0.0) |
|
|
110
|
+
| `dspy-gepa` | `DSPy::Teleprompt::GEPA`, reflection loops, experiment tracking, telemetry adapters. (Install or set `DSPY_WITH_GEPA=1`.) | **Stable** (v1.0.0) |
|
|
111
|
+
| `gepa` | GEPA optimizer core (Pareto engine, telemetry, reflective proposer). | **Stable** (v1.0.0) |
|
|
112
|
+
| `dspy-o11y` | Core observability APIs: `DSPy::Observability`, async span processor, observation types. (Install or set `DSPY_WITH_O11Y=1`.) | **Stable** (v1.0.0) |
|
|
113
|
+
| `dspy-o11y-langfuse` | Auto-configures DSPy observability to stream spans to Langfuse via OTLP. (Install or set `DSPY_WITH_O11Y_LANGFUSE=1`.) | **Stable** (v1.0.0) |
|
|
114
|
+
| `dspy-deep_search` | Production DeepSearch loop with Exa-backed search/read, token budgeting, and instrumentation (Issue #163). | **Stable** (v1.0.0) |
|
|
115
|
+
| `dspy-deep_research` | Planner/QA orchestration atop DeepSearch plus the memory supervisor used by the CLI example. | **Stable** (v1.0.0) |
|
|
116
|
+
| `sorbet-toon` | Token-Oriented Object Notation (TOON) codec, prompt formatter, and Sorbet mixins for BAML/TOON Enhanced Prompting. [Sorbet::Toon README](https://github.com/vicentereig/dspy.rb/blob/main/lib/sorbet/toon/README.md) | **Alpha** (v0.1.0) |
|
|
117
|
+
|
|
118
|
+
**Provider adapters:** Add `dspy-openai`, `dspy-anthropic`, and/or `dspy-gemini` next to `dspy` in your Gemfile depending on which `DSPy::LM` providers you call. Each gem already depends on the official SDK (`openai`, `anthropic`, `gemini-ai`), and DSPy auto-loads the adapters when the gem is present—no extra `require` needed.
|
|
119
|
+
|
|
120
|
+
Set the matching `DSPY_WITH_*` environment variables (see `Gemfile`) to include or exclude each sibling gem when running Bundler locally (for example `DSPY_WITH_GEPA=1` or `DSPY_WITH_O11Y_LANGFUSE=1`). Refer to `adr/013-dependency-tree.md` for the full dependency map and roadmap.
|
|
111
121
|
### Access to 200+ Models Across 5 Providers
|
|
112
122
|
|
|
113
123
|
DSPy.rb provides unified access to major LLM providers with provider-specific optimizations:
|
|
@@ -148,7 +158,10 @@ end
|
|
|
148
158
|
|
|
149
159
|
## What You Get
|
|
150
160
|
|
|
151
|
-
**Developer Experience:**
|
|
161
|
+
**Developer Experience:** Official clients, multimodal coverage, and observability baked in.
|
|
162
|
+
<details>
|
|
163
|
+
<summary>Expand for everything included</summary>
|
|
164
|
+
|
|
152
165
|
- LLM provider support using official Ruby clients:
|
|
153
166
|
- [OpenAI Ruby](https://github.com/openai/openai-ruby) with vision model support
|
|
154
167
|
- [Anthropic Ruby SDK](https://github.com/anthropics/anthropic-sdk-ruby) with multimodal capabilities
|
|
@@ -158,21 +171,33 @@ end
|
|
|
158
171
|
- Runtime type checking with [Sorbet](https://sorbet.org/) including T::Enum and union types
|
|
159
172
|
- Type-safe tool definitions for ReAct agents
|
|
160
173
|
- Comprehensive instrumentation and observability
|
|
174
|
+
</details>
|
|
175
|
+
|
|
176
|
+
**Core Building Blocks:** Predictors, agents, and pipelines wired through type-safe signatures.
|
|
177
|
+
<details>
|
|
178
|
+
<summary>Expand for everything included</summary>
|
|
161
179
|
|
|
162
|
-
**Core Building Blocks:**
|
|
163
180
|
- **Signatures** - Define input/output schemas using Sorbet types with T::Enum and union type support
|
|
164
181
|
- **Predict** - LLM completion with structured data extraction and multimodal support
|
|
165
182
|
- **Chain of Thought** - Step-by-step reasoning for complex problems with automatic prompt optimization
|
|
166
183
|
- **ReAct** - Tool-using agents with type-safe tool definitions and error recovery
|
|
167
184
|
- **Module Composition** - Combine multiple LLM calls into production-ready workflows
|
|
185
|
+
</details>
|
|
186
|
+
|
|
187
|
+
**Optimization & Evaluation:** Treat prompt optimization like a real ML workflow.
|
|
188
|
+
<details>
|
|
189
|
+
<summary>Expand for everything included</summary>
|
|
168
190
|
|
|
169
|
-
**Optimization & Evaluation:**
|
|
170
191
|
- **Prompt Objects** - Manipulate prompts as first-class objects instead of strings
|
|
171
192
|
- **Typed Examples** - Type-safe training data with automatic validation
|
|
172
193
|
- **Evaluation Framework** - Advanced metrics beyond simple accuracy with error-resilient pipelines
|
|
173
194
|
- **MIPROv2 Optimization** - Advanced Bayesian optimization with Gaussian Processes, multiple optimization strategies, auto-config presets, and storage persistence
|
|
195
|
+
</details>
|
|
196
|
+
|
|
197
|
+
**Production Features:** Hardened behaviors for teams shipping actual products.
|
|
198
|
+
<details>
|
|
199
|
+
<summary>Expand for everything included</summary>
|
|
174
200
|
|
|
175
|
-
**Production Features:**
|
|
176
201
|
- **Reliable JSON Extraction** - Native structured outputs for OpenAI and Gemini, Anthropic tool-based extraction, and automatic strategy selection with fallback
|
|
177
202
|
- **Type-Safe Configuration** - Strategy enums with automatic provider optimization (Strict/Compatible modes)
|
|
178
203
|
- **Smart Retry Logic** - Progressive fallback with exponential backoff for handling transient failures
|
|
@@ -180,10 +205,13 @@ end
|
|
|
180
205
|
- **Performance Caching** - Schema and capability caching for faster repeated operations
|
|
181
206
|
- **File-based Storage** - Optimization result persistence with versioning
|
|
182
207
|
- **Structured Logging** - JSON and key=value formats with span tracking
|
|
208
|
+
</details>
|
|
183
209
|
|
|
184
210
|
## Recent Achievements
|
|
185
211
|
|
|
186
|
-
DSPy.rb has
|
|
212
|
+
DSPy.rb has gone from experimental to production-ready in three fast releases.
|
|
213
|
+
<details>
|
|
214
|
+
<summary>Expand for the full changelog highlights</summary>
|
|
187
215
|
|
|
188
216
|
### Foundation
|
|
189
217
|
- ✅ **JSON Parsing Reliability** - Native OpenAI structured outputs with adaptive retry logic and schema-aware fallbacks
|
|
@@ -199,8 +227,11 @@ DSPy.rb has rapidly evolved from experimental to production-ready:
|
|
|
199
227
|
- ✅ **Optimizer Utilities Parity (v0.29.0)** - Bootstrap strategies, dataset summaries, and Layer 3 utilities unlock multi-predictor programs on Ruby
|
|
200
228
|
- ✅ **Observability Hardening (v0.29.0)** - OTLP exporter runs on a single-thread executor preventing frozen SSL contexts without blocking spans
|
|
201
229
|
- ✅ **Documentation Refresh (v0.29.x)** - New GEPA guide plus ADE optimization docs covering presets, stratified splits, and error-handling defaults
|
|
230
|
+
</details>
|
|
202
231
|
|
|
203
|
-
**Current Focus Areas:**
|
|
232
|
+
**Current Focus Areas:** Closing the loop on production patterns and community adoption ahead of v1.0.
|
|
233
|
+
<details>
|
|
234
|
+
<summary>Expand for the roadmap</summary>
|
|
204
235
|
|
|
205
236
|
### Production Readiness
|
|
206
237
|
- 🚧 **Production Patterns** - Real-world usage validation and performance optimization
|
|
@@ -210,21 +241,31 @@ DSPy.rb has rapidly evolved from experimental to production-ready:
|
|
|
210
241
|
- 🚧 **Community Examples** - Real-world applications and case studies
|
|
211
242
|
- 🚧 **Contributor Experience** - Making it easier to contribute and extend
|
|
212
243
|
- 🚧 **Performance Benchmarks** - Comparative analysis vs other frameworks
|
|
244
|
+
</details>
|
|
213
245
|
|
|
214
|
-
**v1.0 Philosophy:**
|
|
215
|
-
v1.0 will be released after extensive production battle-testing, not after checking off features.
|
|
216
|
-
The API is already stable - v1.0 represents confidence in production reliability backed by real-world validation.
|
|
246
|
+
**v1.0 Philosophy:** v1.0 lands after battle-testing, not checkbox bingo. The API is already stable; the milestone marks production confidence.
|
|
217
247
|
|
|
218
248
|
|
|
219
249
|
## Documentation
|
|
220
250
|
|
|
221
|
-
📖 **[Complete Documentation Website](https://
|
|
251
|
+
📖 **[Complete Documentation Website](https://oss.vicente.services/dspy.rb/)**
|
|
222
252
|
|
|
223
253
|
### LLM-Friendly Documentation
|
|
224
254
|
|
|
225
255
|
For LLMs and AI assistants working with DSPy.rb:
|
|
226
|
-
- **[llms.txt](https://
|
|
227
|
-
- **[llms-full.txt](https://
|
|
256
|
+
- **[llms.txt](https://oss.vicente.services/dspy.rb/llms.txt)** - Concise reference optimized for LLMs
|
|
257
|
+
- **[llms-full.txt](https://oss.vicente.services/dspy.rb/llms-full.txt)** - Comprehensive API documentation
|
|
258
|
+
|
|
259
|
+
### Claude Skill
|
|
260
|
+
|
|
261
|
+
A [Claude Skill](https://github.com/vicentereig/dspy-rb-skill) is available to help you build DSPy.rb applications with Claude Code or claude.ai.
|
|
262
|
+
|
|
263
|
+
**Claude Code:**
|
|
264
|
+
```bash
|
|
265
|
+
git clone https://github.com/vicentereig/dspy-rb-skill ~/.claude/skills/dspy-rb
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
**Claude.ai (Pro/Max):** Download the [skill as a ZIP](https://github.com/vicentereig/dspy-rb-skill/archive/refs/heads/main.zip) and upload via Settings > Skills.
|
|
228
269
|
|
|
229
270
|
### Getting Started
|
|
230
271
|
- **[Installation & Setup](docs/src/getting-started/installation.md)** - Detailed installation and configuration
|
|
@@ -0,0 +1,222 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'sorbet-runtime'
|
|
4
|
+
require 'net/http'
|
|
5
|
+
require 'json'
|
|
6
|
+
require 'base64'
|
|
7
|
+
|
|
8
|
+
module DSPy
|
|
9
|
+
class Observability
|
|
10
|
+
module Adapters
|
|
11
|
+
module Langfuse
|
|
12
|
+
# Async exporter for sending scores to Langfuse REST API
|
|
13
|
+
# Uses a background thread to avoid blocking the main application
|
|
14
|
+
class ScoresExporter
|
|
15
|
+
extend T::Sig
|
|
16
|
+
|
|
17
|
+
DEFAULT_HOST = 'https://cloud.langfuse.com'
|
|
18
|
+
SCORES_ENDPOINT = '/api/public/scores'
|
|
19
|
+
DEFAULT_MAX_RETRIES = 3
|
|
20
|
+
DEFAULT_TIMEOUT = 10
|
|
21
|
+
|
|
22
|
+
attr_reader :host
|
|
23
|
+
|
|
24
|
+
sig do
|
|
25
|
+
params(
|
|
26
|
+
public_key: String,
|
|
27
|
+
secret_key: String,
|
|
28
|
+
host: String,
|
|
29
|
+
max_retries: Integer,
|
|
30
|
+
timeout: Integer
|
|
31
|
+
).void
|
|
32
|
+
end
|
|
33
|
+
def initialize(
|
|
34
|
+
public_key:,
|
|
35
|
+
secret_key:,
|
|
36
|
+
host: DEFAULT_HOST,
|
|
37
|
+
max_retries: DEFAULT_MAX_RETRIES,
|
|
38
|
+
timeout: DEFAULT_TIMEOUT
|
|
39
|
+
)
|
|
40
|
+
@public_key = public_key
|
|
41
|
+
@secret_key = secret_key
|
|
42
|
+
@host = host.chomp('/')
|
|
43
|
+
@max_retries = max_retries
|
|
44
|
+
@timeout = timeout
|
|
45
|
+
@queue = Thread::Queue.new
|
|
46
|
+
@running = false
|
|
47
|
+
@worker_thread = nil
|
|
48
|
+
@subscription_id = nil
|
|
49
|
+
@mutex = Mutex.new
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# Factory method that creates, starts, and subscribes to events
|
|
53
|
+
sig do
|
|
54
|
+
params(
|
|
55
|
+
public_key: String,
|
|
56
|
+
secret_key: String,
|
|
57
|
+
host: String,
|
|
58
|
+
max_retries: Integer,
|
|
59
|
+
timeout: Integer
|
|
60
|
+
).returns(ScoresExporter)
|
|
61
|
+
end
|
|
62
|
+
def self.configure(
|
|
63
|
+
public_key:,
|
|
64
|
+
secret_key:,
|
|
65
|
+
host: DEFAULT_HOST,
|
|
66
|
+
max_retries: DEFAULT_MAX_RETRIES,
|
|
67
|
+
timeout: DEFAULT_TIMEOUT
|
|
68
|
+
)
|
|
69
|
+
exporter = new(
|
|
70
|
+
public_key: public_key,
|
|
71
|
+
secret_key: secret_key,
|
|
72
|
+
host: host,
|
|
73
|
+
max_retries: max_retries,
|
|
74
|
+
timeout: timeout
|
|
75
|
+
)
|
|
76
|
+
exporter.start
|
|
77
|
+
exporter.subscribe_to_events
|
|
78
|
+
exporter
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
sig { void }
|
|
82
|
+
def start
|
|
83
|
+
@mutex.synchronize do
|
|
84
|
+
return if @running
|
|
85
|
+
|
|
86
|
+
@running = true
|
|
87
|
+
@worker_thread = Thread.new { process_queue }
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
sig { returns(T::Boolean) }
|
|
92
|
+
def running?
|
|
93
|
+
@mutex.synchronize { @running }
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
sig { params(score_event: DSPy::Scores::ScoreEvent).void }
|
|
97
|
+
def export(score_event)
|
|
98
|
+
return unless running?
|
|
99
|
+
|
|
100
|
+
@queue.push(score_event)
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
sig { returns(Integer) }
|
|
104
|
+
def queue_size
|
|
105
|
+
@queue.size
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
sig { void }
|
|
109
|
+
def subscribe_to_events
|
|
110
|
+
@subscription_id = DSPy.events.subscribe('score.create') do |_name, attrs|
|
|
111
|
+
# Reconstruct ScoreEvent from event attributes
|
|
112
|
+
score_event = DSPy::Scores::ScoreEvent.new(
|
|
113
|
+
id: attrs[:score_id],
|
|
114
|
+
name: attrs[:score_name],
|
|
115
|
+
value: attrs[:score_value],
|
|
116
|
+
data_type: DSPy::Scores::DataType.deserialize(attrs[:score_data_type]),
|
|
117
|
+
comment: attrs[:score_comment],
|
|
118
|
+
trace_id: attrs[:trace_id],
|
|
119
|
+
observation_id: attrs[:observation_id]
|
|
120
|
+
)
|
|
121
|
+
export(score_event)
|
|
122
|
+
end
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
sig { params(timeout: Integer).void }
|
|
126
|
+
def shutdown(timeout: 5)
|
|
127
|
+
@mutex.synchronize do
|
|
128
|
+
return unless @running
|
|
129
|
+
|
|
130
|
+
@running = false
|
|
131
|
+
|
|
132
|
+
# Unsubscribe from events
|
|
133
|
+
DSPy.events.unsubscribe(@subscription_id) if @subscription_id
|
|
134
|
+
@subscription_id = nil
|
|
135
|
+
|
|
136
|
+
# Signal worker to stop
|
|
137
|
+
@queue.push(:stop)
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
# Wait for worker thread to finish
|
|
141
|
+
@worker_thread&.join(timeout)
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
private
|
|
145
|
+
|
|
146
|
+
sig { void }
|
|
147
|
+
def process_queue
|
|
148
|
+
while running? || !@queue.empty?
|
|
149
|
+
item = @queue.pop
|
|
150
|
+
|
|
151
|
+
break if item == :stop
|
|
152
|
+
|
|
153
|
+
begin
|
|
154
|
+
send_with_retry(item)
|
|
155
|
+
rescue StandardError => e
|
|
156
|
+
DSPy.log('scores.export_error', error: e.message, score_name: item.name)
|
|
157
|
+
end
|
|
158
|
+
end
|
|
159
|
+
end
|
|
160
|
+
|
|
161
|
+
sig { params(score_event: DSPy::Scores::ScoreEvent).void }
|
|
162
|
+
def send_with_retry(score_event)
|
|
163
|
+
retries = 0
|
|
164
|
+
|
|
165
|
+
begin
|
|
166
|
+
send_to_langfuse(score_event)
|
|
167
|
+
rescue StandardError => e
|
|
168
|
+
retries += 1
|
|
169
|
+
if retries <= @max_retries
|
|
170
|
+
sleep(exponential_backoff(retries))
|
|
171
|
+
retry
|
|
172
|
+
else
|
|
173
|
+
raise e
|
|
174
|
+
end
|
|
175
|
+
end
|
|
176
|
+
end
|
|
177
|
+
|
|
178
|
+
sig { params(attempt: Integer).returns(Float) }
|
|
179
|
+
def exponential_backoff(attempt)
|
|
180
|
+
# 0.1s, 0.2s, 0.4s, 0.8s... with jitter
|
|
181
|
+
base_delay = 0.1 * (2 ** (attempt - 1))
|
|
182
|
+
base_delay + rand * 0.1
|
|
183
|
+
end
|
|
184
|
+
|
|
185
|
+
sig { params(score_event: DSPy::Scores::ScoreEvent).void }
|
|
186
|
+
def send_to_langfuse(score_event)
|
|
187
|
+
uri = URI("#{@host}#{SCORES_ENDPOINT}")
|
|
188
|
+
payload = build_payload(score_event)
|
|
189
|
+
|
|
190
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
|
191
|
+
http.use_ssl = uri.scheme == 'https'
|
|
192
|
+
http.open_timeout = @timeout
|
|
193
|
+
http.read_timeout = @timeout
|
|
194
|
+
|
|
195
|
+
request = Net::HTTP::Post.new(uri.path)
|
|
196
|
+
request['Content-Type'] = 'application/json'
|
|
197
|
+
request['Authorization'] = "Basic #{auth_token}"
|
|
198
|
+
request.body = JSON.generate(payload)
|
|
199
|
+
|
|
200
|
+
response = http.request(request)
|
|
201
|
+
|
|
202
|
+
unless response.is_a?(Net::HTTPSuccess)
|
|
203
|
+
raise "Langfuse API error: #{response.code} - #{response.body}"
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
DSPy.log('scores.exported', score_name: score_event.name, score_id: score_event.id)
|
|
207
|
+
end
|
|
208
|
+
|
|
209
|
+
sig { params(score_event: DSPy::Scores::ScoreEvent).returns(T::Hash[Symbol, T.untyped]) }
|
|
210
|
+
def build_payload(score_event)
|
|
211
|
+
score_event.to_langfuse_payload
|
|
212
|
+
end
|
|
213
|
+
|
|
214
|
+
sig { returns(String) }
|
|
215
|
+
def auth_token
|
|
216
|
+
Base64.strict_encode64("#{@public_key}:#{@secret_key}")
|
|
217
|
+
end
|
|
218
|
+
end
|
|
219
|
+
end
|
|
220
|
+
end
|
|
221
|
+
end
|
|
222
|
+
end
|
data/lib/dspy/o11y/langfuse.rb
CHANGED
metadata
CHANGED
|
@@ -1,28 +1,28 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: dspy-o11y-langfuse
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.1.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Vicente Reig Rincón de Arellano
|
|
8
8
|
bindir: bin
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
11
|
dependencies:
|
|
12
12
|
- !ruby/object:Gem::Dependency
|
|
13
13
|
name: dspy-o11y
|
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
|
15
15
|
requirements:
|
|
16
|
-
- -
|
|
16
|
+
- - ">="
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version:
|
|
18
|
+
version: '0.30'
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
|
-
- -
|
|
23
|
+
- - ">="
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version:
|
|
25
|
+
version: '0.30'
|
|
26
26
|
- !ruby/object:Gem::Dependency
|
|
27
27
|
name: opentelemetry-sdk
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -62,6 +62,7 @@ files:
|
|
|
62
62
|
- LICENSE
|
|
63
63
|
- README.md
|
|
64
64
|
- lib/dspy/o11y/langfuse.rb
|
|
65
|
+
- lib/dspy/o11y/langfuse/scores_exporter.rb
|
|
65
66
|
- lib/dspy/o11y/langfuse/version.rb
|
|
66
67
|
homepage: https://github.com/vicentereig/dspy.rb
|
|
67
68
|
licenses:
|
|
@@ -82,7 +83,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
82
83
|
- !ruby/object:Gem::Version
|
|
83
84
|
version: '0'
|
|
84
85
|
requirements: []
|
|
85
|
-
rubygems_version: 3.6.
|
|
86
|
+
rubygems_version: 3.6.9
|
|
86
87
|
specification_version: 4
|
|
87
88
|
summary: Langfuse auto-configuration adapter for DSPy observability.
|
|
88
89
|
test_files: []
|