dspy 0.4.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +15 -20
- data/lib/dspy/chain_of_thought.rb +84 -104
- data/lib/dspy/instrumentation/token_tracker.rb +6 -6
- data/lib/dspy/instrumentation.rb +109 -17
- data/lib/dspy/lm.rb +63 -29
- data/lib/dspy/mixins/instrumentation_helpers.rb +119 -0
- data/lib/dspy/mixins/struct_builder.rb +133 -0
- data/lib/dspy/mixins/type_coercion.rb +67 -0
- data/lib/dspy/predict.rb +51 -94
- data/lib/dspy/re_act.rb +242 -173
- data/lib/dspy/subscribers/logger_subscriber.rb +60 -5
- data/lib/dspy/version.rb +1 -1
- data/lib/dspy.rb +87 -0
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 84036f6f01324f792e9de3e85fb90092747a3a021072cb65ef02d0911832dd23
|
4
|
+
data.tar.gz: 368240d7747e0e381e39d117061aefa55161ec99b9a2dd4813b798f854e907b2
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: fe445ba22336550e234359eea194aa105f204332f117b44c560a9b170ec5f126192647f9edcba09a991fda1f1bcec8144239362cb95698b32b0526efd343d587
|
7
|
+
data.tar.gz: b47bbd52e43b765009778c110dd58a69eb75873ce1e8a17dc0fe33c388e7c7dfd8cfcb2f848b0bc81102487acba5230a47d7a001501462ea7f9c7a1b0605fc05
|
data/README.md
CHANGED
@@ -14,23 +14,19 @@ The result? LLM applications that actually scale and don't break when you sneeze
|
|
14
14
|
- **Signatures** - Define input/output schemas using Sorbet types
|
15
15
|
- **Predict** - Basic LLM completion with structured data
|
16
16
|
- **Chain of Thought** - Step-by-step reasoning for complex problems
|
17
|
-
- **ReAct** - Tool-using agents
|
18
|
-
- **
|
19
|
-
- **Multi-stage Pipelines** - Compose multiple LLM calls into workflows
|
17
|
+
- **ReAct** - Tool-using agents with basic tool integration
|
18
|
+
- **Manual Composition** - Combine multiple LLM calls into workflows
|
20
19
|
|
21
20
|
**Optimization & Evaluation:**
|
22
21
|
- **Prompt Objects** - Manipulate prompts as first-class objects instead of strings
|
23
22
|
- **Typed Examples** - Type-safe training data with automatic validation
|
24
|
-
- **Evaluation Framework** -
|
25
|
-
- **
|
26
|
-
- **Simple Optimizer** - Random/grid search for quick experimentation
|
23
|
+
- **Evaluation Framework** - Basic testing with simple metrics
|
24
|
+
- **Basic Optimization** - Simple prompt optimization techniques
|
27
25
|
|
28
26
|
**Production Features:**
|
29
|
-
- **Storage
|
30
|
-
- **Registry System** - Version control for optimized signatures with deployment tracking
|
27
|
+
- **File-based Storage** - Basic optimization result persistence
|
31
28
|
- **Multi-Platform Observability** - OpenTelemetry, New Relic, and Langfuse integration
|
32
|
-
- **
|
33
|
-
- **Rollback Protection** - Automatic rollback on performance degradation
|
29
|
+
- **Basic Instrumentation** - Event tracking and logging
|
34
30
|
|
35
31
|
**Developer Experience:**
|
36
32
|
- LLM provider support using official Ruby clients:
|
@@ -106,21 +102,20 @@ puts result.confidence # => 0.85
|
|
106
102
|
- **[Examples & Validation](docs/core-concepts/examples.md)** - Type-safe training data
|
107
103
|
|
108
104
|
### Optimization
|
109
|
-
- **[Evaluation Framework](docs/optimization/evaluation.md)** -
|
105
|
+
- **[Evaluation Framework](docs/optimization/evaluation.md)** - Basic testing with simple metrics
|
110
106
|
- **[Prompt Optimization](docs/optimization/prompt-optimization.md)** - Manipulate prompts as objects
|
111
|
-
- **[MIPROv2 Optimizer](docs/optimization/miprov2.md)** -
|
112
|
-
- **[Simple Optimizer](docs/optimization/simple-optimizer.md)** -
|
107
|
+
- **[MIPROv2 Optimizer](docs/optimization/miprov2.md)** - Basic automatic optimization
|
108
|
+
- **[Simple Optimizer](docs/optimization/simple-optimizer.md)** - Random search experimentation
|
113
109
|
|
114
110
|
### Production Features
|
115
|
-
- **[Storage System](docs/
|
116
|
-
- **[
|
117
|
-
- **[Observability](docs/enterprise/observability.md)** - Multi-platform monitoring and metrics
|
111
|
+
- **[Storage System](docs/production/storage.md)** - Basic file-based persistence
|
112
|
+
- **[Observability](docs/production/observability.md)** - Multi-platform monitoring and metrics
|
118
113
|
|
119
114
|
### Advanced Usage
|
120
|
-
- **[Complex Types](docs/advanced/complex-types.md)** -
|
121
|
-
- **[
|
122
|
-
- **[RAG
|
123
|
-
- **[Custom Metrics](docs/advanced/custom-metrics.md)** -
|
115
|
+
- **[Complex Types](docs/advanced/complex-types.md)** - Basic Sorbet type integration
|
116
|
+
- **[Manual Pipelines](docs/advanced/pipelines.md)** - Manual module composition patterns
|
117
|
+
- **[RAG Patterns](docs/advanced/rag.md)** - Manual RAG implementation with external services
|
118
|
+
- **[Custom Metrics](docs/advanced/custom-metrics.md)** - Proc-based evaluation logic
|
124
119
|
|
125
120
|
## What's Next
|
126
121
|
|
@@ -5,49 +5,22 @@ require 'sorbet-runtime'
|
|
5
5
|
require_relative 'predict'
|
6
6
|
require_relative 'signature'
|
7
7
|
require_relative 'instrumentation'
|
8
|
+
require_relative 'mixins/struct_builder'
|
8
9
|
|
9
10
|
module DSPy
|
10
11
|
# Enhances prediction by encouraging step-by-step reasoning
|
11
12
|
# before providing a final answer using Sorbet signatures.
|
12
13
|
class ChainOfThought < Predict
|
13
14
|
extend T::Sig
|
15
|
+
include Mixins::StructBuilder
|
14
16
|
|
15
17
|
FieldDescriptor = DSPy::Signature::FieldDescriptor
|
16
18
|
|
17
19
|
sig { params(signature_class: T.class_of(DSPy::Signature)).void }
|
18
20
|
def initialize(signature_class)
|
19
21
|
@original_signature = signature_class
|
20
|
-
|
21
|
-
|
22
|
-
enhanced_output_struct = create_enhanced_output_struct(signature_class)
|
23
|
-
|
24
|
-
# Create enhanced signature class
|
25
|
-
enhanced_signature = Class.new(DSPy::Signature) do
|
26
|
-
# Set the description
|
27
|
-
description "#{signature_class.description} Think step by step."
|
28
|
-
|
29
|
-
# Use the same input struct and copy field descriptors
|
30
|
-
@input_struct_class = signature_class.input_struct_class
|
31
|
-
@input_field_descriptors = signature_class.instance_variable_get(:@input_field_descriptors) || {}
|
32
|
-
|
33
|
-
# Use the enhanced output struct and create field descriptors for it
|
34
|
-
@output_struct_class = enhanced_output_struct
|
35
|
-
|
36
|
-
# Create field descriptors for the enhanced output struct
|
37
|
-
@output_field_descriptors = {}
|
38
|
-
|
39
|
-
# Copy original output field descriptors
|
40
|
-
original_output_descriptors = signature_class.instance_variable_get(:@output_field_descriptors) || {}
|
41
|
-
@output_field_descriptors.merge!(original_output_descriptors)
|
42
|
-
|
43
|
-
# Add reasoning field descriptor (ChainOfThought always provides this)
|
44
|
-
@output_field_descriptors[:reasoning] = FieldDescriptor.new(String, "Step by step reasoning process")
|
45
|
-
|
46
|
-
class << self
|
47
|
-
attr_reader :input_struct_class, :output_struct_class
|
48
|
-
end
|
49
|
-
end
|
50
|
-
|
22
|
+
enhanced_signature = build_enhanced_signature(signature_class)
|
23
|
+
|
51
24
|
# Call parent constructor with enhanced signature
|
52
25
|
super(enhanced_signature)
|
53
26
|
@signature_class = enhanced_signature
|
@@ -81,9 +54,8 @@ module DSPy
|
|
81
54
|
|
82
55
|
sig { override.params(instruction: String).returns(ChainOfThought) }
|
83
56
|
def with_instruction(instruction)
|
84
|
-
|
85
|
-
|
86
|
-
super(cot_instruction)
|
57
|
+
enhanced_instruction = ensure_chain_of_thought_instruction(instruction)
|
58
|
+
super(enhanced_instruction)
|
87
59
|
end
|
88
60
|
|
89
61
|
sig { override.params(examples: T::Array[FewShotExample]).returns(ChainOfThought) }
|
@@ -113,43 +85,96 @@ module DSPy
|
|
113
85
|
# Override forward_untyped to add ChainOfThought-specific instrumentation
|
114
86
|
sig { override.params(input_values: T.untyped).returns(T.untyped) }
|
115
87
|
def forward_untyped(**input_values)
|
116
|
-
|
117
|
-
input_fields = input_values.keys.map(&:to_s)
|
118
|
-
|
119
|
-
# Instrument ChainOfThought lifecycle
|
120
|
-
result = Instrumentation.instrument('dspy.chain_of_thought', {
|
121
|
-
signature_class: @original_signature.name,
|
122
|
-
model: lm.model,
|
123
|
-
provider: lm.provider,
|
124
|
-
input_fields: input_fields
|
125
|
-
}) do
|
88
|
+
instrument_prediction('dspy.chain_of_thought', @original_signature, input_values) do
|
126
89
|
# Call parent prediction logic
|
127
90
|
prediction_result = super(**input_values)
|
128
91
|
|
129
92
|
# Analyze reasoning if present
|
130
|
-
|
131
|
-
reasoning_content = prediction_result.reasoning.to_s
|
132
|
-
reasoning_length = reasoning_content.length
|
133
|
-
reasoning_steps = count_reasoning_steps(reasoning_content)
|
134
|
-
|
135
|
-
# Emit reasoning analysis event
|
136
|
-
Instrumentation.emit('dspy.chain_of_thought.reasoning_complete', {
|
137
|
-
signature_class: @original_signature.name,
|
138
|
-
reasoning_steps: reasoning_steps,
|
139
|
-
reasoning_length: reasoning_length,
|
140
|
-
has_reasoning: !reasoning_content.empty?
|
141
|
-
})
|
142
|
-
end
|
93
|
+
analyze_reasoning(prediction_result)
|
143
94
|
|
144
95
|
prediction_result
|
145
96
|
end
|
146
|
-
|
147
|
-
result
|
148
97
|
end
|
149
98
|
|
150
99
|
private
|
151
100
|
|
101
|
+
# Builds enhanced signature with reasoning capabilities
|
102
|
+
sig { params(signature_class: T.class_of(DSPy::Signature)).returns(T.class_of(DSPy::Signature)) }
|
103
|
+
def build_enhanced_signature(signature_class)
|
104
|
+
enhanced_output_struct = create_enhanced_output_struct(signature_class)
|
105
|
+
create_signature_class(signature_class, enhanced_output_struct)
|
106
|
+
end
|
107
|
+
|
108
|
+
# Creates signature class with enhanced description and reasoning field
|
109
|
+
sig { params(signature_class: T.class_of(DSPy::Signature), enhanced_output_struct: T.class_of(T::Struct)).returns(T.class_of(DSPy::Signature)) }
|
110
|
+
def create_signature_class(signature_class, enhanced_output_struct)
|
111
|
+
Class.new(DSPy::Signature) do
|
112
|
+
description "#{signature_class.description} Think step by step."
|
113
|
+
|
114
|
+
# Use the same input struct and copy field descriptors
|
115
|
+
@input_struct_class = signature_class.input_struct_class
|
116
|
+
@input_field_descriptors = signature_class.instance_variable_get(:@input_field_descriptors) || {}
|
117
|
+
|
118
|
+
# Use the enhanced output struct and create field descriptors for it
|
119
|
+
@output_struct_class = enhanced_output_struct
|
120
|
+
|
121
|
+
# Create field descriptors for the enhanced output struct
|
122
|
+
@output_field_descriptors = {}
|
123
|
+
|
124
|
+
# Copy original output field descriptors
|
125
|
+
original_output_descriptors = signature_class.instance_variable_get(:@output_field_descriptors) || {}
|
126
|
+
@output_field_descriptors.merge!(original_output_descriptors)
|
127
|
+
|
128
|
+
# Add reasoning field descriptor (ChainOfThought always provides this)
|
129
|
+
@output_field_descriptors[:reasoning] = FieldDescriptor.new(String, "Step by step reasoning process")
|
130
|
+
|
131
|
+
class << self
|
132
|
+
attr_reader :input_struct_class, :output_struct_class
|
133
|
+
end
|
134
|
+
end
|
135
|
+
end
|
136
|
+
|
137
|
+
# Creates enhanced output struct with reasoning field
|
138
|
+
sig { params(signature_class: T.class_of(DSPy::Signature)).returns(T.class_of(T::Struct)) }
|
139
|
+
def create_enhanced_output_struct(signature_class)
|
140
|
+
output_props = signature_class.output_struct_class.props
|
141
|
+
|
142
|
+
build_enhanced_struct(
|
143
|
+
{ output: output_props },
|
144
|
+
{ reasoning: [String, "Step by step reasoning process"] }
|
145
|
+
)
|
146
|
+
end
|
147
|
+
|
148
|
+
# Ensures instruction includes chain of thought prompt
|
149
|
+
sig { params(instruction: String).returns(String) }
|
150
|
+
def ensure_chain_of_thought_instruction(instruction)
|
151
|
+
instruction.include?("Think step by step") ? instruction : "#{instruction} Think step by step."
|
152
|
+
end
|
153
|
+
|
154
|
+
# Analyzes reasoning in prediction result and emits instrumentation events
|
155
|
+
sig { params(prediction_result: T.untyped).void }
|
156
|
+
def analyze_reasoning(prediction_result)
|
157
|
+
return unless prediction_result.respond_to?(:reasoning) && prediction_result.reasoning
|
158
|
+
|
159
|
+
reasoning_content = prediction_result.reasoning.to_s
|
160
|
+
return if reasoning_content.empty?
|
161
|
+
|
162
|
+
emit_reasoning_analysis(reasoning_content)
|
163
|
+
end
|
164
|
+
|
165
|
+
# Emits reasoning analysis instrumentation event
|
166
|
+
sig { params(reasoning_content: String).void }
|
167
|
+
def emit_reasoning_analysis(reasoning_content)
|
168
|
+
Instrumentation.emit('dspy.chain_of_thought.reasoning_complete', {
|
169
|
+
signature_class: @original_signature.name,
|
170
|
+
reasoning_steps: count_reasoning_steps(reasoning_content),
|
171
|
+
reasoning_length: reasoning_content.length,
|
172
|
+
has_reasoning: true
|
173
|
+
})
|
174
|
+
end
|
175
|
+
|
152
176
|
# Count reasoning steps by looking for step indicators
|
177
|
+
sig { params(reasoning_text: String).returns(Integer) }
|
153
178
|
def count_reasoning_steps(reasoning_text)
|
154
179
|
return 0 if reasoning_text.nil? || reasoning_text.empty?
|
155
180
|
|
@@ -170,50 +195,5 @@ module DSPy
|
|
170
195
|
# Fallback: count sentences if no clear steps
|
171
196
|
max_count > 0 ? max_count : reasoning_text.split(/[.!?]+/).reject(&:empty?).length
|
172
197
|
end
|
173
|
-
|
174
|
-
sig { params(signature_class: T.class_of(DSPy::Signature)).returns(T.class_of(T::Struct)) }
|
175
|
-
def create_enhanced_output_struct(signature_class)
|
176
|
-
# Get original output props
|
177
|
-
original_props = signature_class.output_struct_class.props
|
178
|
-
|
179
|
-
# Create new struct class with reasoning added
|
180
|
-
Class.new(T::Struct) do
|
181
|
-
# Add all original fields
|
182
|
-
original_props.each do |name, prop|
|
183
|
-
# Extract the type and other options
|
184
|
-
type = prop[:type]
|
185
|
-
options = prop.except(:type, :type_object, :accessor_key, :sensitivity, :redaction)
|
186
|
-
|
187
|
-
# Handle default values
|
188
|
-
if options[:default]
|
189
|
-
const name, type, default: options[:default]
|
190
|
-
elsif options[:factory]
|
191
|
-
const name, type, factory: options[:factory]
|
192
|
-
else
|
193
|
-
const name, type
|
194
|
-
end
|
195
|
-
end
|
196
|
-
|
197
|
-
# Add reasoning field (ChainOfThought always provides this)
|
198
|
-
const :reasoning, String
|
199
|
-
|
200
|
-
# Add to_h method to serialize the struct to a hash
|
201
|
-
define_method :to_h do
|
202
|
-
hash = {}
|
203
|
-
|
204
|
-
# Start with input values if available
|
205
|
-
if self.instance_variable_defined?(:@input_values)
|
206
|
-
hash.merge!(self.instance_variable_get(:@input_values))
|
207
|
-
end
|
208
|
-
|
209
|
-
# Then add output properties
|
210
|
-
self.class.props.keys.each do |key|
|
211
|
-
hash[key] = self.send(key)
|
212
|
-
end
|
213
|
-
|
214
|
-
hash
|
215
|
-
end
|
216
|
-
end
|
217
|
-
end
|
218
198
|
end
|
219
199
|
end
|
@@ -28,9 +28,9 @@ module DSPy
|
|
28
28
|
return {} unless usage.is_a?(Hash)
|
29
29
|
|
30
30
|
{
|
31
|
-
|
32
|
-
|
33
|
-
|
31
|
+
input_tokens: usage[:prompt_tokens] || usage['prompt_tokens'],
|
32
|
+
output_tokens: usage[:completion_tokens] || usage['completion_tokens'],
|
33
|
+
total_tokens: usage[:total_tokens] || usage['total_tokens']
|
34
34
|
}
|
35
35
|
end
|
36
36
|
|
@@ -44,9 +44,9 @@ module DSPy
|
|
44
44
|
output_tokens = usage[:output_tokens] || usage['output_tokens'] || 0
|
45
45
|
|
46
46
|
{
|
47
|
-
|
48
|
-
|
49
|
-
|
47
|
+
input_tokens: input_tokens,
|
48
|
+
output_tokens: output_tokens,
|
49
|
+
total_tokens: input_tokens + output_tokens
|
50
50
|
}
|
51
51
|
end
|
52
52
|
end
|
data/lib/dspy/instrumentation.rb
CHANGED
@@ -9,27 +9,27 @@ module DSPy
|
|
9
9
|
# Provides extension points for logging, OpenTelemetry, New Relic, Langfuse, and custom monitoring
|
10
10
|
module Instrumentation
|
11
11
|
# Get a logger subscriber instance (creates new instance each time)
|
12
|
-
def self.logger_subscriber
|
12
|
+
def self.logger_subscriber(**options)
|
13
13
|
require_relative 'subscribers/logger_subscriber'
|
14
|
-
DSPy::Subscribers::LoggerSubscriber.new
|
14
|
+
DSPy::Subscribers::LoggerSubscriber.new(**options)
|
15
15
|
end
|
16
16
|
|
17
17
|
# Get an OpenTelemetry subscriber instance (creates new instance each time)
|
18
|
-
def self.otel_subscriber
|
18
|
+
def self.otel_subscriber(**options)
|
19
19
|
require_relative 'subscribers/otel_subscriber'
|
20
|
-
DSPy::Subscribers::OtelSubscriber.new
|
20
|
+
DSPy::Subscribers::OtelSubscriber.new(**options)
|
21
21
|
end
|
22
22
|
|
23
23
|
# Get a New Relic subscriber instance (creates new instance each time)
|
24
|
-
def self.newrelic_subscriber
|
24
|
+
def self.newrelic_subscriber(**options)
|
25
25
|
require_relative 'subscribers/newrelic_subscriber'
|
26
|
-
DSPy::Subscribers::NewrelicSubscriber.new
|
26
|
+
DSPy::Subscribers::NewrelicSubscriber.new(**options)
|
27
27
|
end
|
28
28
|
|
29
29
|
# Get a Langfuse subscriber instance (creates new instance each time)
|
30
|
-
def self.langfuse_subscriber
|
30
|
+
def self.langfuse_subscriber(**options)
|
31
31
|
require_relative 'subscribers/langfuse_subscriber'
|
32
|
-
DSPy::Subscribers::LangfuseSubscriber.new
|
32
|
+
DSPy::Subscribers::LangfuseSubscriber.new(**options)
|
33
33
|
end
|
34
34
|
|
35
35
|
def self.notifications
|
@@ -115,9 +115,8 @@ module DSPy
|
|
115
115
|
enhanced_payload = payload.merge(
|
116
116
|
duration_ms: ((end_time - start_time) * 1000).round(2),
|
117
117
|
cpu_time_ms: ((end_cpu - start_cpu) * 1000).round(2),
|
118
|
-
status: 'success'
|
119
|
-
|
120
|
-
)
|
118
|
+
status: 'success'
|
119
|
+
).merge(generate_timestamp)
|
121
120
|
|
122
121
|
self.emit_event(event_name, enhanced_payload)
|
123
122
|
result
|
@@ -130,9 +129,8 @@ module DSPy
|
|
130
129
|
cpu_time_ms: ((end_cpu - start_cpu) * 1000).round(2),
|
131
130
|
status: 'error',
|
132
131
|
error_type: error.class.name,
|
133
|
-
error_message: error.message
|
134
|
-
|
135
|
-
)
|
132
|
+
error_message: error.message
|
133
|
+
).merge(generate_timestamp)
|
136
134
|
|
137
135
|
self.emit_event(event_name, error_payload)
|
138
136
|
raise
|
@@ -145,9 +143,8 @@ module DSPy
|
|
145
143
|
payload ||= {}
|
146
144
|
|
147
145
|
enhanced_payload = payload.merge(
|
148
|
-
timestamp: Time.now.iso8601,
|
149
146
|
status: payload[:status] || 'success'
|
150
|
-
)
|
147
|
+
).merge(generate_timestamp)
|
151
148
|
|
152
149
|
self.emit_event(event_name, enhanced_payload)
|
153
150
|
end
|
@@ -175,7 +172,102 @@ module DSPy
|
|
175
172
|
end
|
176
173
|
|
177
174
|
def self.setup_subscribers
|
178
|
-
|
175
|
+
config = DSPy.config.instrumentation
|
176
|
+
|
177
|
+
# Return early if instrumentation is disabled
|
178
|
+
return unless config.enabled
|
179
|
+
|
180
|
+
# Validate configuration first
|
181
|
+
DSPy.validate_instrumentation!
|
182
|
+
|
183
|
+
# Setup each configured subscriber
|
184
|
+
config.subscribers.each do |subscriber_type|
|
185
|
+
setup_subscriber(subscriber_type)
|
186
|
+
end
|
187
|
+
end
|
188
|
+
|
189
|
+
def self.setup_subscriber(subscriber_type)
|
190
|
+
case subscriber_type
|
191
|
+
when :logger
|
192
|
+
setup_logger_subscriber
|
193
|
+
when :otel
|
194
|
+
setup_otel_subscriber if otel_available?
|
195
|
+
when :newrelic
|
196
|
+
setup_newrelic_subscriber if newrelic_available?
|
197
|
+
when :langfuse
|
198
|
+
setup_langfuse_subscriber if langfuse_available?
|
199
|
+
else
|
200
|
+
raise ArgumentError, "Unknown subscriber type: #{subscriber_type}"
|
201
|
+
end
|
202
|
+
rescue LoadError => e
|
203
|
+
DSPy.logger.warn "Failed to setup #{subscriber_type} subscriber: #{e.message}"
|
204
|
+
end
|
205
|
+
|
206
|
+
def self.setup_logger_subscriber
|
207
|
+
# Create subscriber - it will read configuration when handling events
|
208
|
+
logger_subscriber
|
209
|
+
end
|
210
|
+
|
211
|
+
def self.setup_otel_subscriber
|
212
|
+
# Create subscriber - it will read configuration when handling events
|
213
|
+
otel_subscriber
|
214
|
+
end
|
215
|
+
|
216
|
+
def self.setup_newrelic_subscriber
|
217
|
+
# Create subscriber - it will read configuration when handling events
|
218
|
+
newrelic_subscriber
|
219
|
+
end
|
220
|
+
|
221
|
+
def self.setup_langfuse_subscriber
|
222
|
+
# Create subscriber - it will read configuration when handling events
|
223
|
+
langfuse_subscriber
|
224
|
+
end
|
225
|
+
|
226
|
+
# Dependency checking methods
|
227
|
+
def self.otel_available?
|
228
|
+
begin
|
229
|
+
require 'opentelemetry/sdk'
|
230
|
+
true
|
231
|
+
rescue LoadError
|
232
|
+
false
|
233
|
+
end
|
234
|
+
end
|
235
|
+
|
236
|
+
def self.newrelic_available?
|
237
|
+
begin
|
238
|
+
require 'newrelic_rpm'
|
239
|
+
true
|
240
|
+
rescue LoadError
|
241
|
+
false
|
242
|
+
end
|
243
|
+
end
|
244
|
+
|
245
|
+
def self.langfuse_available?
|
246
|
+
begin
|
247
|
+
require 'langfuse'
|
248
|
+
true
|
249
|
+
rescue LoadError
|
250
|
+
false
|
251
|
+
end
|
252
|
+
end
|
253
|
+
|
254
|
+
# Generate timestamp in the configured format
|
255
|
+
def self.generate_timestamp
|
256
|
+
case DSPy.config.instrumentation.timestamp_format
|
257
|
+
when DSPy::TimestampFormat::ISO8601
|
258
|
+
{ timestamp: Time.now.iso8601 }
|
259
|
+
when DSPy::TimestampFormat::RFC3339_NANO
|
260
|
+
{ timestamp: Time.now.strftime('%Y-%m-%dT%H:%M:%S.%9N%z') }
|
261
|
+
when DSPy::TimestampFormat::UNIX_NANO
|
262
|
+
{ timestamp_ns: (Time.now.to_f * 1_000_000_000).to_i }
|
263
|
+
else
|
264
|
+
{ timestamp: Time.now.iso8601 } # Fallback to iso8601
|
265
|
+
end
|
266
|
+
end
|
267
|
+
|
268
|
+
# Legacy setup method for backward compatibility
|
269
|
+
def self.setup_subscribers_legacy
|
270
|
+
# Legacy initialization - will be created when first accessed
|
179
271
|
# Force initialization of enabled subscribers
|
180
272
|
logger_subscriber
|
181
273
|
|
data/lib/dspy/lm.rb
CHANGED
@@ -39,38 +39,49 @@ module DSPy
|
|
39
39
|
input_text = messages.map { |m| m[:content] }.join(' ')
|
40
40
|
input_size = input_text.length
|
41
41
|
|
42
|
-
#
|
43
|
-
response =
|
44
|
-
|
45
|
-
gen_ai_system: provider,
|
46
|
-
gen_ai_request_model: model,
|
47
|
-
signature_class: signature_class.name,
|
48
|
-
provider: provider,
|
49
|
-
adapter_class: adapter.class.name,
|
50
|
-
input_size: input_size
|
51
|
-
}) do
|
52
|
-
adapter.chat(messages: messages, &block)
|
53
|
-
end
|
54
|
-
|
55
|
-
# Extract actual token usage from response (more accurate than estimation)
|
56
|
-
token_usage = Instrumentation::TokenTracker.extract_token_usage(response, provider)
|
42
|
+
# Use smart consolidation: emit LM events only when not in nested context
|
43
|
+
response = nil
|
44
|
+
token_usage = {}
|
57
45
|
|
58
|
-
|
59
|
-
|
60
|
-
Instrumentation.
|
46
|
+
if should_emit_lm_events?
|
47
|
+
# Emit all LM events when not in nested context
|
48
|
+
response = Instrumentation.instrument('dspy.lm.request', {
|
49
|
+
gen_ai_operation_name: 'chat',
|
61
50
|
gen_ai_system: provider,
|
62
51
|
gen_ai_request_model: model,
|
63
|
-
signature_class: signature_class.name
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
52
|
+
signature_class: signature_class.name,
|
53
|
+
provider: provider,
|
54
|
+
adapter_class: adapter.class.name,
|
55
|
+
input_size: input_size
|
56
|
+
}) do
|
57
|
+
adapter.chat(messages: messages, &block)
|
58
|
+
end
|
59
|
+
|
60
|
+
# Extract actual token usage from response (more accurate than estimation)
|
61
|
+
token_usage = Instrumentation::TokenTracker.extract_token_usage(response, provider)
|
62
|
+
|
63
|
+
# Emit token usage event if available
|
64
|
+
if token_usage.any?
|
65
|
+
Instrumentation.emit('dspy.lm.tokens', token_usage.merge({
|
66
|
+
gen_ai_system: provider,
|
67
|
+
gen_ai_request_model: model,
|
68
|
+
signature_class: signature_class.name
|
69
|
+
}))
|
70
|
+
end
|
71
|
+
|
72
|
+
# Instrument response parsing
|
73
|
+
parsed_result = Instrumentation.instrument('dspy.lm.response.parsed', {
|
74
|
+
signature_class: signature_class.name,
|
75
|
+
provider: provider,
|
76
|
+
response_length: response.content&.length || 0
|
77
|
+
}) do
|
78
|
+
parse_response(response, input_values, signature_class)
|
79
|
+
end
|
80
|
+
else
|
81
|
+
# Consolidated mode: execute without nested instrumentation
|
82
|
+
response = adapter.chat(messages: messages, &block)
|
83
|
+
token_usage = Instrumentation::TokenTracker.extract_token_usage(response, provider)
|
84
|
+
parsed_result = parse_response(response, input_values, signature_class)
|
74
85
|
end
|
75
86
|
|
76
87
|
parsed_result
|
@@ -78,6 +89,29 @@ module DSPy
|
|
78
89
|
|
79
90
|
private
|
80
91
|
|
92
|
+
# Determines if LM-level events should be emitted using smart consolidation
|
93
|
+
def should_emit_lm_events?
|
94
|
+
# Emit LM events only if we're not in a nested context (smart consolidation)
|
95
|
+
!is_nested_context?
|
96
|
+
end
|
97
|
+
|
98
|
+
# Determines if we're in a nested context where higher-level events are being emitted
|
99
|
+
def is_nested_context?
|
100
|
+
caller_locations = caller_locations(1, 30)
|
101
|
+
return false if caller_locations.nil?
|
102
|
+
|
103
|
+
# Look for higher-level DSPy modules in the call stack
|
104
|
+
# We consider ChainOfThought and ReAct as higher-level modules
|
105
|
+
higher_level_modules = caller_locations.select do |loc|
|
106
|
+
loc.path.include?('chain_of_thought') ||
|
107
|
+
loc.path.include?('re_act') ||
|
108
|
+
loc.path.include?('react')
|
109
|
+
end
|
110
|
+
|
111
|
+
# If we have higher-level modules in the call stack, we're in a nested context
|
112
|
+
higher_level_modules.any?
|
113
|
+
end
|
114
|
+
|
81
115
|
def parse_model_id(model_id)
|
82
116
|
unless model_id.include?('/')
|
83
117
|
raise ArgumentError, "model_id must include provider (e.g., 'openai/gpt-4', 'anthropic/claude-3'). Legacy format without provider is no longer supported."
|