braintrust 0.5.0 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +41 -11
- data/lib/braintrust/config.rb +7 -4
- data/lib/braintrust/contrib/roast/instrumentation/cog.rb +97 -0
- data/lib/braintrust/contrib/roast/instrumentation/workflow.rb +49 -0
- data/lib/braintrust/contrib/roast/integration.rb +32 -0
- data/lib/braintrust/contrib/roast/patcher.rb +42 -0
- data/lib/braintrust/contrib/roast/tracing.rb +15 -0
- data/lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb +63 -59
- data/lib/braintrust/contrib/ruby_llm/instrumentation/common.rb +18 -0
- data/lib/braintrust/contrib/ruby_llm/patcher.rb +1 -1
- data/lib/braintrust/contrib.rb +2 -0
- data/lib/braintrust/state.rb +4 -2
- data/lib/braintrust/trace/span_customization.rb +40 -0
- data/lib/braintrust/trace/span_customizers.rb +110 -0
- data/lib/braintrust/trace/span_export_data.rb +35 -0
- data/lib/braintrust/trace/span_exporter.rb +4 -0
- data/lib/braintrust/trace/span_origin.rb +6 -2
- data/lib/braintrust/trace.rb +14 -3
- data/lib/braintrust/version.rb +1 -1
- data/lib/braintrust.rb +3 -1
- metadata +9 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2e67bcb9e3752500ed9ac74f3ad5d46b7ee86867f648d772a4138b5491e3f405
|
|
4
|
+
data.tar.gz: 7949cd6453a74de3632a52704a13616ff620c85429f51bff1bb6405954ea9d9d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 8b07cd2821efe35fc99235dfa5fde7c06ed3dd147262c30633e1469ac0d0bd30a92fcaabef301b19be5d27dda98542b3008c01c97d2d00e4bf35ee09996c602f
|
|
7
|
+
data.tar.gz: 7b8075fc99973163cd432253c5c9edd34e51e9b79fb4dd80dca7c8fd472716567acf4b68a6008be46545414c358b192ff180a4b07c76b722d0d691b69b219d47
|
data/README.md
CHANGED
|
@@ -17,9 +17,10 @@ This is the official Ruby SDK for [Braintrust](https://www.braintrust.dev), for
|
|
|
17
17
|
- [Braintrust.init](#braintrustinit)
|
|
18
18
|
- [Environment variables](#environment-variables)
|
|
19
19
|
- [Tracing](#tracing)
|
|
20
|
-
- [Supported
|
|
20
|
+
- [Supported integrations](#supported-integrations)
|
|
21
21
|
- [Manually applying instrumentation](#manually-applying-instrumentation)
|
|
22
22
|
- [Creating custom spans](#creating-custom-spans)
|
|
23
|
+
- [Span customizers](#span-customizers)
|
|
23
24
|
- [Attachments](#attachments)
|
|
24
25
|
- [Viewing traces](#viewing-traces)
|
|
25
26
|
- [Evals](#evals)
|
|
@@ -120,6 +121,7 @@ Braintrust.init
|
|
|
120
121
|
| `filter_ai_spans` | `ENV['BRAINTRUST_OTEL_FILTER_AI_SPANS']` | Only export AI-related spans |
|
|
121
122
|
| `org_name` | `ENV['BRAINTRUST_ORG_NAME']` | Organization name |
|
|
122
123
|
| `set_global` | `true` | Set as global state. Set to `false` for isolated instances |
|
|
124
|
+
| `span_customizers` | `[]` | Redact or transform spans before export (see [Span customizers](#span-customizers)) |
|
|
123
125
|
|
|
124
126
|
**Example with options:**
|
|
125
127
|
|
|
@@ -148,16 +150,17 @@ Braintrust.init(
|
|
|
148
150
|
|
|
149
151
|
## Tracing
|
|
150
152
|
|
|
151
|
-
### Supported
|
|
153
|
+
### Supported integrations
|
|
152
154
|
|
|
153
|
-
The SDK automatically instruments these
|
|
155
|
+
The SDK automatically instruments these libraries:
|
|
154
156
|
|
|
155
|
-
|
|
|
156
|
-
| --------- | ------------- | -------- | ---------------- |
|
|
157
|
-
| Anthropic | `anthropic` | >= 0.3.0 | `:anthropic` | [Link](./examples/contrib/anthropic.rb) |
|
|
158
|
-
| OpenAI | `openai` | >= 0.1.0 | `:openai` | [Link](./examples/contrib/openai.rb) |
|
|
159
|
-
| | `ruby-openai` | >= 7.0.0 | `:ruby_openai` | [Link](./examples/contrib/
|
|
160
|
-
| Multiple | `ruby_llm` | >= 1.8.0 | `:ruby_llm` | [Link](./examples/contrib/ruby_llm.rb) |
|
|
157
|
+
| Category | Gem | Versions | Integration Name | Examples |
|
|
158
|
+
| --------- | ------------- | -------- | ---------------- | ----------------------------------------------- |
|
|
159
|
+
| Anthropic | `anthropic` | >= 0.3.0 | `:anthropic` | [Link](./examples/contrib/anthropic/basic.rb) |
|
|
160
|
+
| OpenAI | `openai` | >= 0.1.0 | `:openai` | [Link](./examples/contrib/openai/basic.rb) |
|
|
161
|
+
| | `ruby-openai` | >= 7.0.0 | `:ruby_openai` | [Link](./examples/contrib/ruby_openai/basic.rb) |
|
|
162
|
+
| Multiple | `ruby_llm` | >= 1.8.0 | `:ruby_llm` | [Link](./examples/contrib/ruby_llm/basic.rb) |
|
|
163
|
+
| Workflows | `roast-ai` | >= 1.0.0 | `:roast` | [Link](./examples/contrib/roast/basic.rb) |
|
|
161
164
|
|
|
162
165
|
### Manually applying instrumentation
|
|
163
166
|
|
|
@@ -192,6 +195,33 @@ tracer.in_span("process-request") do |span|
|
|
|
192
195
|
end
|
|
193
196
|
```
|
|
194
197
|
|
|
198
|
+
### Span customizers
|
|
199
|
+
|
|
200
|
+
Redact or transform span data before it leaves your application, e.g. to strip PII from LLM inputs and outputs:
|
|
201
|
+
|
|
202
|
+
```ruby
|
|
203
|
+
# Any object that responds to #on_span_export
|
|
204
|
+
class RedactContent
|
|
205
|
+
def on_span_export(span)
|
|
206
|
+
%w[braintrust.input_json braintrust.output_json].each do |key|
|
|
207
|
+
span.attributes[key] = JSON.generate("[redacted]") if span.attributes.key?(key)
|
|
208
|
+
end
|
|
209
|
+
span.attributes.delete("user.email")
|
|
210
|
+
span
|
|
211
|
+
end
|
|
212
|
+
end
|
|
213
|
+
|
|
214
|
+
Braintrust.init(span_customizers: [RedactContent.new])
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
**When configuring customizers...**
|
|
218
|
+
|
|
219
|
+
- Do not change the `trace_id`, `span_id`, or `parent_span_id` fields.
|
|
220
|
+
- Always return the span (or a replacement `OpenTelemetry::SDK::Trace::SpanData`).
|
|
221
|
+
- Keep them fast; otherwise they may slow down exporting.
|
|
222
|
+
- Customizers run in order on every finished span sent to Braintrust.
|
|
223
|
+
- If it raises or returns something invalid, the batch is dropped and an error is logged.
|
|
224
|
+
|
|
195
225
|
### Attachments
|
|
196
226
|
|
|
197
227
|
Log binary data (images, PDFs, audio) in your traces:
|
|
@@ -567,11 +597,11 @@ The dev server requires the `rack` gem and a Rack-compatible web server.
|
|
|
567
597
|
| [Passenger](https://www.phusionpassenger.com/) | 6.x | |
|
|
568
598
|
| [WEBrick](https://github.com/ruby/webrick) | Not supported | Does not support server-sent events. |
|
|
569
599
|
|
|
570
|
-
See examples: [server/eval.ru](./examples/server/eval.ru),
|
|
600
|
+
See examples: [server/eval.ru](./examples/server/eval.ru),
|
|
571
601
|
|
|
572
602
|
## Documentation
|
|
573
603
|
|
|
574
|
-
- [Braintrust Documentation](https://
|
|
604
|
+
- [Braintrust Documentation](https://docs.braintrust.dev)
|
|
575
605
|
- [API Reference](https://gemdocs.org/gems/braintrust/)
|
|
576
606
|
|
|
577
607
|
## Troubleshooting
|
data/lib/braintrust/config.rb
CHANGED
|
@@ -8,10 +8,10 @@ module Braintrust
|
|
|
8
8
|
# and allows overriding with explicit options
|
|
9
9
|
class Config
|
|
10
10
|
attr_reader :api_key, :org_name, :default_project, :app_url, :api_url,
|
|
11
|
-
:filter_ai_spans, :span_filter_funcs
|
|
11
|
+
:filter_ai_spans, :span_filter_funcs, :span_customizers
|
|
12
12
|
|
|
13
13
|
def initialize(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil,
|
|
14
|
-
filter_ai_spans: nil, span_filter_funcs: nil)
|
|
14
|
+
filter_ai_spans: nil, span_filter_funcs: nil, span_customizers: nil)
|
|
15
15
|
@api_key = api_key
|
|
16
16
|
@org_name = org_name
|
|
17
17
|
@default_project = default_project
|
|
@@ -19,6 +19,7 @@ module Braintrust
|
|
|
19
19
|
@api_url = api_url
|
|
20
20
|
@filter_ai_spans = filter_ai_spans
|
|
21
21
|
@span_filter_funcs = span_filter_funcs || []
|
|
22
|
+
@span_customizers = (span_customizers || []).dup.freeze
|
|
22
23
|
end
|
|
23
24
|
|
|
24
25
|
# Create a Config from environment variables, with option overrides
|
|
@@ -30,9 +31,10 @@ module Braintrust
|
|
|
30
31
|
# @param api_url [String, nil] API URL (overrides BRAINTRUST_API_URL env var)
|
|
31
32
|
# @param filter_ai_spans [Boolean, nil] Enable AI span filtering (overrides BRAINTRUST_OTEL_FILTER_AI_SPANS env var)
|
|
32
33
|
# @param span_filter_funcs [Array<Proc>, nil] Custom span filter functions
|
|
34
|
+
# @param span_customizers [Array<#on_span_export>, nil] Ordered export customizers (copied and frozen)
|
|
33
35
|
# @return [Config] the created config
|
|
34
36
|
def self.from_env(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil,
|
|
35
|
-
filter_ai_spans: nil, span_filter_funcs: nil)
|
|
37
|
+
filter_ai_spans: nil, span_filter_funcs: nil, span_customizers: nil)
|
|
36
38
|
# Parse filter_ai_spans from ENV if not explicitly provided
|
|
37
39
|
env_filter_ai_spans = ENV["BRAINTRUST_OTEL_FILTER_AI_SPANS"]
|
|
38
40
|
filter_ai_spans_value = if filter_ai_spans.nil?
|
|
@@ -48,7 +50,8 @@ module Braintrust
|
|
|
48
50
|
app_url: app_url || ENV["BRAINTRUST_APP_URL"] || "https://www.braintrust.dev",
|
|
49
51
|
api_url: api_url || ENV["BRAINTRUST_API_URL"] || "https://api.braintrust.dev",
|
|
50
52
|
filter_ai_spans: filter_ai_spans_value,
|
|
51
|
-
span_filter_funcs: span_filter_funcs
|
|
53
|
+
span_filter_funcs: span_filter_funcs,
|
|
54
|
+
span_customizers: span_customizers
|
|
52
55
|
)
|
|
53
56
|
end
|
|
54
57
|
end
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "../../support/otel"
|
|
4
|
+
require_relative "../tracing"
|
|
5
|
+
|
|
6
|
+
module Braintrust
|
|
7
|
+
module Contrib
|
|
8
|
+
module Roast
|
|
9
|
+
module Instrumentation
|
|
10
|
+
# Wrap the asynchronous work block rather than Cog#run! itself: run! only
|
|
11
|
+
# schedules a task, while the cog executes later in a different fiber.
|
|
12
|
+
module Cog
|
|
13
|
+
class BarrierProxy
|
|
14
|
+
def initialize(barrier, &around_task)
|
|
15
|
+
@barrier = barrier
|
|
16
|
+
@around_task = around_task
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def async(**options, &task)
|
|
20
|
+
@barrier.async(**options) do |async_task|
|
|
21
|
+
@around_task.call(async_task, task)
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def run!(barrier, config, input_context, executor_scope_value, executor_scope_index)
|
|
27
|
+
parent_context = roast_parent_context
|
|
28
|
+
return super unless tracing_enabled? && parent_context
|
|
29
|
+
|
|
30
|
+
tracer = Braintrust::Contrib::Roast::Tracing.tracer_for(self)
|
|
31
|
+
proxy = BarrierProxy.new(barrier) do |async_task, task|
|
|
32
|
+
::OpenTelemetry::Context.with_current(parent_context) do
|
|
33
|
+
span_name = "roast.cog.#{name}"
|
|
34
|
+
tracer.in_span(span_name) do |span|
|
|
35
|
+
# Braintrust uses span_attributes for type/name; Roast details go in metadata.
|
|
36
|
+
Support::OTel.set_json_attr(span, "braintrust.span_attributes", {type: "task", name: span_name})
|
|
37
|
+
Braintrust::Contrib::Context.set!(self, roast_span_context: ::OpenTelemetry::Context.current)
|
|
38
|
+
task.call(async_task)
|
|
39
|
+
rescue => e
|
|
40
|
+
span.record_exception(e)
|
|
41
|
+
span.status = ::OpenTelemetry::Trace::Status.error(e.message)
|
|
42
|
+
raise
|
|
43
|
+
ensure
|
|
44
|
+
record_outcome(span)
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
super(proxy, config, input_context, executor_scope_value, executor_scope_index)
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
private
|
|
52
|
+
|
|
53
|
+
def roast_parent_context
|
|
54
|
+
path = ::Roast::TaskContext.path
|
|
55
|
+
path.reverse_each do |element|
|
|
56
|
+
context = Braintrust::Contrib.context_for(element.cog)&.[](:roast_span_context)
|
|
57
|
+
return context if context
|
|
58
|
+
end
|
|
59
|
+
path.reverse_each do |element|
|
|
60
|
+
context = Braintrust::Contrib.context_for(element.execution_manager&.workflow_context)&.[](:roast_span_context)
|
|
61
|
+
return context if context
|
|
62
|
+
end
|
|
63
|
+
nil
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def tracing_enabled?
|
|
67
|
+
context = Braintrust::Contrib.context_for(self)
|
|
68
|
+
class_context = Braintrust::Contrib.context_for(self.class)
|
|
69
|
+
context&.[](:enabled) != false && class_context&.[](:enabled) != false
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
def record_outcome(span)
|
|
73
|
+
outcome = if failed?
|
|
74
|
+
"failed"
|
|
75
|
+
elsif skipped?
|
|
76
|
+
"skipped"
|
|
77
|
+
else
|
|
78
|
+
"completed"
|
|
79
|
+
end
|
|
80
|
+
Support::OTel.set_json_attr(span, "braintrust.metadata", {
|
|
81
|
+
"contrib.roast.cog.type" => type,
|
|
82
|
+
"contrib.roast.cog.name" => name.to_s,
|
|
83
|
+
"contrib.roast.cog.outcome" => outcome
|
|
84
|
+
})
|
|
85
|
+
|
|
86
|
+
output_value = case output
|
|
87
|
+
when ::Roast::Cogs::Ruby::Output then output.value
|
|
88
|
+
when ::Roast::Cogs::Chat::Output then output.response
|
|
89
|
+
end
|
|
90
|
+
Support::OTel.set_json_attr(span, "braintrust.output_json", output_value) if output_value.is_a?(String)
|
|
91
|
+
span.status = ::OpenTelemetry::Trace::Status.error("Cog failed") if failed?
|
|
92
|
+
end
|
|
93
|
+
end
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
end
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "../../support/otel"
|
|
4
|
+
require_relative "../tracing"
|
|
5
|
+
|
|
6
|
+
module Braintrust
|
|
7
|
+
module Contrib
|
|
8
|
+
module Roast
|
|
9
|
+
module Instrumentation
|
|
10
|
+
# from_file enters a new Async fiber, so carry its caller's OTel context
|
|
11
|
+
# through the params object that is handed to the workflow.
|
|
12
|
+
module WorkflowClass
|
|
13
|
+
def from_file(workflow_path, params)
|
|
14
|
+
previous_parent = Braintrust::Contrib.context_for(params)&.[](:roast_parent_context)
|
|
15
|
+
Braintrust::Contrib::Context.set!(params, roast_parent_context: ::OpenTelemetry::Context.current)
|
|
16
|
+
super
|
|
17
|
+
ensure
|
|
18
|
+
Braintrust::Contrib::Context.set!(params, roast_parent_context: previous_parent)
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
module Workflow
|
|
23
|
+
def start!
|
|
24
|
+
context = Braintrust::Contrib.context_for(self)
|
|
25
|
+
class_context = Braintrust::Contrib.context_for(self.class)
|
|
26
|
+
return super if context&.[](:enabled) == false || class_context&.[](:enabled) == false
|
|
27
|
+
|
|
28
|
+
tracer = Braintrust::Contrib::Roast::Tracing.tracer_for(self)
|
|
29
|
+
workflow_context = @workflow_context
|
|
30
|
+
workflow_path = @workflow_path
|
|
31
|
+
params = workflow_context.params
|
|
32
|
+
parent_context = Braintrust::Contrib.context_for(params)&.[](:roast_parent_context)
|
|
33
|
+
trace_workflow = proc do
|
|
34
|
+
tracer.in_span("roast.workflow") do |span|
|
|
35
|
+
Support::OTel.set_json_attr(span, "braintrust.span_attributes", {type: "task", name: "roast.workflow"})
|
|
36
|
+
Support::OTel.set_json_attr(span, "braintrust.metadata", {"contrib.roast.workflow.name" => File.basename(workflow_path.to_s)})
|
|
37
|
+
input = {targets: params.targets, args: params.args, kwargs: params.kwargs}.reject { |_, value| value.nil? || value.empty? }
|
|
38
|
+
Support::OTel.set_json_attr(span, "braintrust.input_json", input) unless input.empty?
|
|
39
|
+
Braintrust::Contrib::Context.set!(workflow_context, roast_span_context: ::OpenTelemetry::Context.current)
|
|
40
|
+
super()
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
parent_context ? ::OpenTelemetry::Context.with_current(parent_context, &trace_workflow) : trace_workflow.call
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
end
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "../integration"
|
|
4
|
+
|
|
5
|
+
module Braintrust
|
|
6
|
+
module Contrib
|
|
7
|
+
module Roast
|
|
8
|
+
# Roast workflow and cog instrumentation.
|
|
9
|
+
class Integration
|
|
10
|
+
include Braintrust::Contrib::Integration
|
|
11
|
+
|
|
12
|
+
GEM_NAMES = ["roast-ai"].freeze
|
|
13
|
+
REQUIRE_PATHS = ["roast"].freeze
|
|
14
|
+
MINIMUM_VERSION = "1.0.0"
|
|
15
|
+
|
|
16
|
+
def self.integration_name = :roast
|
|
17
|
+
def self.gem_names = GEM_NAMES
|
|
18
|
+
def self.require_paths = REQUIRE_PATHS
|
|
19
|
+
def self.minimum_version = MINIMUM_VERSION
|
|
20
|
+
|
|
21
|
+
def self.loaded?
|
|
22
|
+
(defined?(::Roast::Workflow) && defined?(::Roast::Cog)) ? true : false
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def self.patchers
|
|
26
|
+
require_relative "patcher"
|
|
27
|
+
[WorkflowPatcher, CogPatcher]
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
end
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "../patcher"
|
|
4
|
+
require_relative "integration"
|
|
5
|
+
require_relative "instrumentation/workflow"
|
|
6
|
+
require_relative "instrumentation/cog"
|
|
7
|
+
|
|
8
|
+
module Braintrust
|
|
9
|
+
module Contrib
|
|
10
|
+
module Roast
|
|
11
|
+
class WorkflowPatcher < Braintrust::Contrib::Patcher
|
|
12
|
+
def self.applicable? = defined?(::Roast::Workflow)
|
|
13
|
+
|
|
14
|
+
def self.patched?(**options)
|
|
15
|
+
klass = options[:target]&.singleton_class || ::Roast::Workflow
|
|
16
|
+
klass.ancestors.include?(Instrumentation::Workflow) &&
|
|
17
|
+
(options[:target] || ::Roast::Workflow.singleton_class.ancestors.include?(Instrumentation::WorkflowClass))
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def self.perform_patch(**options)
|
|
21
|
+
target = options[:target]
|
|
22
|
+
raise ArgumentError, "target must be a Roast::Workflow" if target && !target.is_a?(::Roast::Workflow)
|
|
23
|
+
|
|
24
|
+
(target&.singleton_class || ::Roast::Workflow).prepend(Instrumentation::Workflow)
|
|
25
|
+
::Roast::Workflow.singleton_class.prepend(Instrumentation::WorkflowClass) unless target
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
class CogPatcher < Braintrust::Contrib::Patcher
|
|
30
|
+
def self.applicable? = defined?(::Roast::Cog)
|
|
31
|
+
|
|
32
|
+
def self.patched?(**)
|
|
33
|
+
::Roast::Cog.ancestors.include?(Instrumentation::Cog)
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def self.perform_patch(**)
|
|
37
|
+
::Roast::Cog.prepend(Instrumentation::Cog)
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
end
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Braintrust
|
|
4
|
+
module Contrib
|
|
5
|
+
module Roast
|
|
6
|
+
module Tracing
|
|
7
|
+
INSTRUMENTATION_NAME = "braintrust.contrib.roast"
|
|
8
|
+
|
|
9
|
+
def self.tracer_for(target)
|
|
10
|
+
Braintrust::Contrib.tracer_for(target, name: INSTRUMENTATION_NAME)
|
|
11
|
+
end
|
|
12
|
+
end
|
|
13
|
+
end
|
|
14
|
+
end
|
|
15
|
+
end
|
|
@@ -13,43 +13,76 @@ module Braintrust
|
|
|
13
13
|
module RubyLLM
|
|
14
14
|
module Instrumentation
|
|
15
15
|
# Chat instrumentation for RubyLLM.
|
|
16
|
-
# Wraps
|
|
16
|
+
# Wraps the method that makes one chat request, and execute_tool(), to create spans.
|
|
17
17
|
module Chat
|
|
18
18
|
def self.included(base)
|
|
19
19
|
# Guard against double-wrapping
|
|
20
|
-
|
|
20
|
+
return if applied?(base)
|
|
21
|
+
|
|
22
|
+
base.prepend(InstanceMethods)
|
|
23
|
+
base.prepend(base.public_method_defined?(:generate) ? GenerateTracing : CompleteTracing)
|
|
21
24
|
end
|
|
22
25
|
|
|
23
26
|
def self.applied?(base)
|
|
24
27
|
base.ancestors.include?(InstanceMethods)
|
|
25
28
|
end
|
|
26
29
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
30
|
+
# ruby_llm 1.x: each complete() call makes one request, and recursive calls from
|
|
31
|
+
# tool execution create nested spans.
|
|
32
|
+
module CompleteTracing
|
|
33
|
+
def complete(&block)
|
|
34
|
+
return block ? super : super() unless tracing_enabled?
|
|
35
|
+
|
|
36
|
+
trace_chat_request(block) { |wrapper| super(&wrapper) }
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# ruby_llm 2.x: complete() loops over step(), which calls generate() once per
|
|
41
|
+
# request, so each generate() call gets its own span. complete() gets a parent
|
|
42
|
+
# span so a turn's chat and tool spans share one trace when the caller has no span.
|
|
43
|
+
module GenerateTracing
|
|
44
|
+
def generate(&block)
|
|
45
|
+
return block ? super : super() unless tracing_enabled?
|
|
46
|
+
|
|
47
|
+
trace_chat_request(block) { |wrapper| super(&wrapper) }
|
|
48
|
+
end
|
|
49
|
+
|
|
31
50
|
def complete(&block)
|
|
32
51
|
return block ? super : super() unless tracing_enabled?
|
|
33
52
|
|
|
34
53
|
tracer = Braintrust::Contrib.tracer_for(self)
|
|
35
54
|
|
|
55
|
+
tracer.in_span("ruby_llm.complete") do |span|
|
|
56
|
+
Support::OTel.set_json_attr(span, "braintrust.span_attributes", {type: "task"})
|
|
57
|
+
input_messages = build_input_messages
|
|
58
|
+
Support::OTel.set_json_attr(span, "braintrust.input_json", input_messages) if input_messages.any?
|
|
59
|
+
|
|
60
|
+
response = block ? super : super()
|
|
61
|
+
|
|
62
|
+
# No metrics here: each request's tokens live on its ruby_llm.chat child span.
|
|
63
|
+
Support::OTel.set_json_attr(span, "braintrust.output_json", [format_message_for_input(response)]) if response
|
|
64
|
+
response
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
module InstanceMethods
|
|
70
|
+
private
|
|
71
|
+
|
|
72
|
+
# Traces one chat request. Yields the streaming wrapper block, or nil when not
|
|
73
|
+
# streaming, for the caller to pass to super.
|
|
74
|
+
def trace_chat_request(block)
|
|
75
|
+
tracer = Braintrust::Contrib.tracer_for(self)
|
|
76
|
+
|
|
36
77
|
tracer.in_span("ruby_llm.chat") do |span|
|
|
37
78
|
if block
|
|
38
|
-
|
|
39
|
-
handle_streaming_complete(span, block) do |&wrapper|
|
|
40
|
-
super(&wrapper)
|
|
41
|
-
end
|
|
79
|
+
handle_streaming_complete(span, block) { |&wrapper| yield wrapper }
|
|
42
80
|
else
|
|
43
|
-
|
|
44
|
-
handle_non_streaming_complete(span) do
|
|
45
|
-
super()
|
|
46
|
-
end
|
|
81
|
+
handle_non_streaming_complete(span) { yield nil }
|
|
47
82
|
end
|
|
48
83
|
end
|
|
49
84
|
end
|
|
50
85
|
|
|
51
|
-
private
|
|
52
|
-
|
|
53
86
|
# Wrap execute_tool() to trace tool executions.
|
|
54
87
|
# This is a private method in RubyLLM - wrapping it avoids
|
|
55
88
|
# conflicting with user-registered on_tool_call/on_tool_result callbacks.
|
|
@@ -199,22 +232,18 @@ module Braintrust
|
|
|
199
232
|
tool_schema = tool_params_schema(tool) ? build_basic_tool_schema(tool) : build_minimal_tool_schema(tool)
|
|
200
233
|
end
|
|
201
234
|
|
|
202
|
-
# Strip RubyLLM-specific fields to match native OpenAI format
|
|
235
|
+
# Strip RubyLLM-specific fields to match native OpenAI format. Build copies: the
|
|
236
|
+
# parameters hash is the tool's cached schema, which RubyLLM sends to the provider.
|
|
203
237
|
function_key = tool_schema&.key?(:function) ? :function : "function"
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
tool_params.delete("additionalProperties")
|
|
211
|
-
tool_params.delete(:additionalProperties)
|
|
212
|
-
params_key = tool_schema[function_key].key?(:parameters) ? :parameters : "parameters"
|
|
213
|
-
tool_schema[function_key][params_key] = tool_params
|
|
214
|
-
end
|
|
215
|
-
end
|
|
238
|
+
function = tool_schema && tool_schema[function_key]
|
|
239
|
+
return tool_schema unless function
|
|
240
|
+
|
|
241
|
+
params_key = function.key?(:parameters) ? :parameters : "parameters"
|
|
242
|
+
tool_params = function[params_key]
|
|
243
|
+
return tool_schema unless tool_params.is_a?(Hash)
|
|
216
244
|
|
|
217
|
-
|
|
245
|
+
stripped = tool_params.reject { |key, _| %w[strict additionalProperties].include?(key.to_s) }
|
|
246
|
+
tool_schema.merge(function_key => function.merge(params_key => stripped))
|
|
218
247
|
end
|
|
219
248
|
|
|
220
249
|
# A tool's JSON Schema, across ruby_llm versions.
|
|
@@ -388,7 +417,8 @@ module Braintrust
|
|
|
388
417
|
|
|
389
418
|
# Format tool calls into OpenAI format
|
|
390
419
|
def format_tool_calls(tool_calls)
|
|
391
|
-
|
|
420
|
+
# Both the pre-2.2 Hash and ToolCall::Collection expose values.
|
|
421
|
+
tool_calls.values.map do |tc|
|
|
392
422
|
args = tc.arguments
|
|
393
423
|
args_string = args.is_a?(String) ? args : JSON.generate(args)
|
|
394
424
|
|
|
@@ -423,20 +453,7 @@ module Braintrust
|
|
|
423
453
|
|
|
424
454
|
# Set metrics (token usage + time_to_first_token)
|
|
425
455
|
# RubyLLM stores tokens directly in the response hash, not in a nested usage object
|
|
426
|
-
metrics =
|
|
427
|
-
if result.respond_to?(:to_h)
|
|
428
|
-
result_hash = result.to_h
|
|
429
|
-
usage = {
|
|
430
|
-
"input_tokens" => result_hash[:input_tokens],
|
|
431
|
-
"output_tokens" => result_hash[:output_tokens],
|
|
432
|
-
"cached_tokens" => result_hash[:cached_tokens],
|
|
433
|
-
"cache_creation_tokens" => result_hash[:cache_creation_tokens]
|
|
434
|
-
}.compact
|
|
435
|
-
|
|
436
|
-
unless usage.empty?
|
|
437
|
-
metrics = Common.parse_usage_tokens(usage)
|
|
438
|
-
end
|
|
439
|
-
end
|
|
456
|
+
metrics = Common.message_usage_metrics(result)
|
|
440
457
|
metrics["time_to_first_token"] = time_to_first_token if time_to_first_token
|
|
441
458
|
Support::OTel.set_json_attr(span, "braintrust.metrics", metrics) unless metrics.empty?
|
|
442
459
|
end
|
|
@@ -479,20 +496,7 @@ module Braintrust
|
|
|
479
496
|
Support::OTel.set_json_attr(span, "braintrust.output_json", output)
|
|
480
497
|
|
|
481
498
|
# Set metrics (token usage + time_to_first_token)
|
|
482
|
-
metrics =
|
|
483
|
-
if response.respond_to?(:to_h)
|
|
484
|
-
response_hash = response.to_h
|
|
485
|
-
usage = {
|
|
486
|
-
"input_tokens" => response_hash[:input_tokens],
|
|
487
|
-
"output_tokens" => response_hash[:output_tokens],
|
|
488
|
-
"cached_tokens" => response_hash[:cached_tokens],
|
|
489
|
-
"cache_creation_tokens" => response_hash[:cache_creation_tokens]
|
|
490
|
-
}.compact
|
|
491
|
-
|
|
492
|
-
unless usage.empty?
|
|
493
|
-
metrics = Common.parse_usage_tokens(usage)
|
|
494
|
-
end
|
|
495
|
-
end
|
|
499
|
+
metrics = Common.message_usage_metrics(response)
|
|
496
500
|
metrics["time_to_first_token"] = time_to_first_token if time_to_first_token
|
|
497
501
|
Support::OTel.set_json_attr(span, "braintrust.metrics", metrics) unless metrics.empty?
|
|
498
502
|
end
|
|
@@ -6,6 +6,24 @@ module Braintrust
|
|
|
6
6
|
module Instrumentation
|
|
7
7
|
# Common utilities for RubyLLM instrumentation.
|
|
8
8
|
module Common
|
|
9
|
+
# Braintrust metrics for a RubyLLM message's token counts. ruby_llm 2.x renamed the cache
|
|
10
|
+
# counts from cached_tokens/cache_creation_tokens to cache_read_tokens/cache_write_tokens.
|
|
11
|
+
# @param message [Object] a RubyLLM message (anything responding to to_h)
|
|
12
|
+
# @return [Hash<String, Integer>] normalized metrics, empty when no counts are reported
|
|
13
|
+
def self.message_usage_metrics(message)
|
|
14
|
+
return {} unless message.respond_to?(:to_h)
|
|
15
|
+
|
|
16
|
+
counts = message.to_h
|
|
17
|
+
usage = {
|
|
18
|
+
"input_tokens" => counts[:input_tokens],
|
|
19
|
+
"output_tokens" => counts[:output_tokens],
|
|
20
|
+
"cached_tokens" => counts[:cached_tokens] || counts[:cache_read_tokens],
|
|
21
|
+
"cache_creation_tokens" => counts[:cache_creation_tokens] || counts[:cache_write_tokens]
|
|
22
|
+
}.compact
|
|
23
|
+
|
|
24
|
+
usage.empty? ? {} : parse_usage_tokens(usage)
|
|
25
|
+
end
|
|
26
|
+
|
|
9
27
|
# Parse RubyLLM usage tokens into normalized Braintrust metrics.
|
|
10
28
|
# RubyLLM normalizes token fields from all providers (OpenAI, Anthropic, etc.)
|
|
11
29
|
# into a consistent format:
|
|
@@ -7,7 +7,7 @@ module Braintrust
|
|
|
7
7
|
module Contrib
|
|
8
8
|
module RubyLLM
|
|
9
9
|
# Patcher for RubyLLM chat completions.
|
|
10
|
-
# Instruments RubyLLM::Chat#complete and #execute_tool
|
|
10
|
+
# Instruments RubyLLM::Chat#complete (plus #generate on 2.x) and #execute_tool.
|
|
11
11
|
class ChatPatcher < Braintrust::Contrib::Patcher
|
|
12
12
|
class << self
|
|
13
13
|
def applicable?
|
data/lib/braintrust/contrib.rb
CHANGED
|
@@ -197,9 +197,11 @@ require_relative "contrib/openai/integration"
|
|
|
197
197
|
require_relative "contrib/ruby_openai/integration"
|
|
198
198
|
require_relative "contrib/ruby_llm/integration"
|
|
199
199
|
require_relative "contrib/anthropic/integration"
|
|
200
|
+
require_relative "contrib/roast/integration"
|
|
200
201
|
|
|
201
202
|
# Register integrations
|
|
202
203
|
Braintrust::Contrib::OpenAI::Integration.register!
|
|
203
204
|
Braintrust::Contrib::RubyOpenAI::Integration.register!
|
|
204
205
|
Braintrust::Contrib::RubyLLM::Integration.register!
|
|
205
206
|
Braintrust::Contrib::Anthropic::Integration.register!
|
|
207
|
+
Braintrust::Contrib::Roast::Integration.register!
|
data/lib/braintrust/state.rb
CHANGED
|
@@ -24,9 +24,10 @@ module Braintrust
|
|
|
24
24
|
# @param tracer_provider [TracerProvider, nil] Optional tracer provider to use
|
|
25
25
|
# @param filter_ai_spans [Boolean, nil] Enable AI span filtering
|
|
26
26
|
# @param span_filter_funcs [Array<Proc>, nil] Custom span filter functions
|
|
27
|
+
# @param span_customizers [Array<#on_span_export>, nil] Ordered synchronous export customizers
|
|
27
28
|
# @param exporter [Exporter, nil] Optional exporter override (for testing)
|
|
28
29
|
# @return [State] the created state
|
|
29
|
-
def self.from_env(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil, blocking_login: false, enable_tracing: true, tracer_provider: nil, filter_ai_spans: nil, span_filter_funcs: nil, exporter: nil)
|
|
30
|
+
def self.from_env(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil, blocking_login: false, enable_tracing: true, tracer_provider: nil, filter_ai_spans: nil, span_filter_funcs: nil, span_customizers: nil, exporter: nil)
|
|
30
31
|
require_relative "config"
|
|
31
32
|
config = Config.from_env(
|
|
32
33
|
api_key: api_key,
|
|
@@ -35,7 +36,8 @@ module Braintrust
|
|
|
35
36
|
app_url: app_url,
|
|
36
37
|
api_url: api_url,
|
|
37
38
|
filter_ai_spans: filter_ai_spans,
|
|
38
|
-
span_filter_funcs: span_filter_funcs
|
|
39
|
+
span_filter_funcs: span_filter_funcs,
|
|
40
|
+
span_customizers: span_customizers
|
|
39
41
|
)
|
|
40
42
|
new(
|
|
41
43
|
api_key: config.api_key,
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "span_customizers"
|
|
4
|
+
require_relative "../logger"
|
|
5
|
+
|
|
6
|
+
module Braintrust
|
|
7
|
+
module Trace
|
|
8
|
+
# Span customizer behavior for exporters.
|
|
9
|
+
#
|
|
10
|
+
# Prepend it onto any exporter whose +export(span_data, timeout:)+ it can
|
|
11
|
+
# +super+ into, and the exporter accepts a +span_customizers:+ option on
|
|
12
|
+
# construction and runs those customizers on every export. Prepend it
|
|
13
|
+
# before SpanOrigin, so SpanOrigin runs first and hooks see origin-enriched
|
|
14
|
+
# data:
|
|
15
|
+
#
|
|
16
|
+
# prepend SpanCustomization
|
|
17
|
+
# prepend SpanOrigin
|
|
18
|
+
#
|
|
19
|
+
# Customization is fail-closed: if any hook fails, none of the batch is
|
|
20
|
+
# passed on to the wrapped exporter.
|
|
21
|
+
module SpanCustomization
|
|
22
|
+
# Consumes +span_customizers:+ and forwards everything else to the exporter.
|
|
23
|
+
def initialize(*args, span_customizers: nil, **options, &block)
|
|
24
|
+
@span_customizers = SpanCustomizers.new(span_customizers)
|
|
25
|
+
super(*args, **options, &block)
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def export(span_data, timeout: nil)
|
|
29
|
+
super(span_customizers.customize(span_data), timeout: timeout)
|
|
30
|
+
rescue SpanCustomizers::Error => e
|
|
31
|
+
Log.error("Span customization failed; batch not sent: #{e.message}")
|
|
32
|
+
OpenTelemetry::SDK::Trace::Export::FAILURE
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
private
|
|
36
|
+
|
|
37
|
+
attr_reader :span_customizers
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
end
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "span_export_data"
|
|
4
|
+
|
|
5
|
+
module Braintrust
|
|
6
|
+
module Trace
|
|
7
|
+
# Ordered export-time transformations, independent of origin and transport.
|
|
8
|
+
class SpanCustomizers
|
|
9
|
+
# A hook failed. Messages are SDK-generated: they identify the customizer
|
|
10
|
+
# and the kind of failure, never span payloads or the hook's own exception
|
|
11
|
+
# message, either of which may contain sensitive data.
|
|
12
|
+
class Error < StandardError; end
|
|
13
|
+
|
|
14
|
+
ID_FIELDS = %i[trace_id span_id parent_span_id].freeze
|
|
15
|
+
private_constant :ID_FIELDS
|
|
16
|
+
|
|
17
|
+
def initialize(customizers = nil)
|
|
18
|
+
@customizers = (customizers || []).dup.freeze
|
|
19
|
+
@customizers.each { |customizer| validate!(customizer) }
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
# Transform the whole batch before the exporter groups or serializes it.
|
|
23
|
+
# Raises Error so the exporter can fail the batch without sending.
|
|
24
|
+
def customize(span_data)
|
|
25
|
+
return span_data if @customizers.empty?
|
|
26
|
+
|
|
27
|
+
# Hooks may call instrumented clients. Suppress tracing so those spans
|
|
28
|
+
# cannot re-enter the exporter and customize themselves forever.
|
|
29
|
+
OpenTelemetry::Common::Utilities.untraced do
|
|
30
|
+
span_data.map { |span| customize_span(span) }
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
private
|
|
35
|
+
|
|
36
|
+
# Reject objects without the hook up front, so a misspelled method name
|
|
37
|
+
# fails at registration instead of silently skipping redaction.
|
|
38
|
+
def validate!(customizer)
|
|
39
|
+
return if customizer.respond_to?(:on_span_export)
|
|
40
|
+
|
|
41
|
+
raise ArgumentError, "#{label(customizer)} must implement on_span_export"
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def customize_span(span)
|
|
45
|
+
# Snapshot IDs so in-place mutation through to_span_data cannot slip past validation.
|
|
46
|
+
ids = identity(span)
|
|
47
|
+
drops = dropped_counts(span)
|
|
48
|
+
view = nil
|
|
49
|
+
@customizers.each do |customizer|
|
|
50
|
+
view ||= writable_view(span)
|
|
51
|
+
result = invoke(customizer, view)
|
|
52
|
+
span = unwrap(result, customizer)
|
|
53
|
+
raise Error, "#{label(customizer)} changed span IDs; trace_id, span_id and parent_span_id must be preserved" unless identity(span) == ids
|
|
54
|
+
|
|
55
|
+
view = (result if result.is_a?(SpanExportData))
|
|
56
|
+
end
|
|
57
|
+
restore_counts(span, drops)
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
def invoke(customizer, view)
|
|
61
|
+
customizer.on_span_export(view)
|
|
62
|
+
# Rescue broadly: hook errors must fail the batch, not kill the export thread.
|
|
63
|
+
rescue StandardError, ScriptError, SystemStackError => e
|
|
64
|
+
raise Error, "#{label(customizer)} raised #{e.class}"
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def unwrap(result, customizer)
|
|
68
|
+
span = result.is_a?(SpanExportData) ? result.to_span_data : result
|
|
69
|
+
return span if span.is_a?(OpenTelemetry::SDK::Trace::SpanData)
|
|
70
|
+
|
|
71
|
+
raise Error, "#{label(customizer)} returned #{result.class}; hooks must return the span or a replacement SpanData"
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
def label(customizer)
|
|
75
|
+
customizer.class.name || customizer.class.inspect
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
def identity(span)
|
|
79
|
+
ID_FIELDS.map { |field| span.public_send(field).dup }
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# OTel and replacement spans may carry frozen attributes. Only the hash
|
|
83
|
+
# needs to be writable; values and other nested objects remain shared.
|
|
84
|
+
def writable_view(span)
|
|
85
|
+
span.attributes = span.attributes&.dup || {}
|
|
86
|
+
SpanExportData.new(span)
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
# Preserve drops caused by SDK limits; entries a customizer deletes are not drops.
|
|
90
|
+
def dropped_counts(span)
|
|
91
|
+
{
|
|
92
|
+
attributes: dropped(span.total_recorded_attributes, span.attributes),
|
|
93
|
+
events: dropped(span.total_recorded_events, span.events),
|
|
94
|
+
links: dropped(span.total_recorded_links, span.links)
|
|
95
|
+
}
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
def restore_counts(span, drops)
|
|
99
|
+
span.total_recorded_attributes = span.attributes&.size.to_i + drops[:attributes]
|
|
100
|
+
span.total_recorded_events = span.events&.size.to_i + drops[:events]
|
|
101
|
+
span.total_recorded_links = span.links&.size.to_i + drops[:links]
|
|
102
|
+
span
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def dropped(total, entries)
|
|
106
|
+
[total.to_i - entries&.size.to_i, 0].max
|
|
107
|
+
end
|
|
108
|
+
end
|
|
109
|
+
end
|
|
110
|
+
end
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "forwardable"
|
|
4
|
+
require "opentelemetry/sdk"
|
|
5
|
+
|
|
6
|
+
module Braintrust
|
|
7
|
+
module Trace
|
|
8
|
+
# Mutable view of completed span data with read-only identity fields.
|
|
9
|
+
# Attribute values and other nested objects are shared, not deep-copied.
|
|
10
|
+
class SpanExportData
|
|
11
|
+
extend Forwardable
|
|
12
|
+
|
|
13
|
+
ID_FIELDS = [:trace_id, :span_id, :parent_span_id].freeze
|
|
14
|
+
MUTABLE_FIELDS = (OpenTelemetry::SDK::Trace::SpanData.members - ID_FIELDS).freeze
|
|
15
|
+
private_constant :ID_FIELDS, :MUTABLE_FIELDS
|
|
16
|
+
|
|
17
|
+
attr_reader(*ID_FIELDS)
|
|
18
|
+
def_delegators :@span_data, *MUTABLE_FIELDS, *MUTABLE_FIELDS.map { |field| :"#{field}=" }
|
|
19
|
+
def_delegators :@span_data, :hex_trace_id, :hex_span_id, :hex_parent_span_id, :instrumentation_library
|
|
20
|
+
|
|
21
|
+
def initialize(span_data)
|
|
22
|
+
@span_data = span_data
|
|
23
|
+
@trace_id = span_data.trace_id.dup.freeze
|
|
24
|
+
@span_id = span_data.span_id.dup.freeze
|
|
25
|
+
@parent_span_id = span_data.parent_span_id.dup.freeze
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# Access the underlying OpenTelemetry representation, e.g. to build a replacement.
|
|
29
|
+
# Replacement IDs are validated by the exporter after each hook.
|
|
30
|
+
def to_span_data
|
|
31
|
+
@span_data
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
@@ -3,17 +3,21 @@
|
|
|
3
3
|
require "opentelemetry/exporter/otlp"
|
|
4
4
|
require_relative "../state"
|
|
5
5
|
require_relative "span_origin"
|
|
6
|
+
require_relative "span_customization"
|
|
6
7
|
|
|
7
8
|
module Braintrust
|
|
8
9
|
module Trace
|
|
9
10
|
# Custom OTLP exporter for the Braintrust backend. On export it:
|
|
10
11
|
# - stamps span origin provenance onto each SpanData (via the prepended SpanOrigin behavior)
|
|
12
|
+
# - runs optional customizers (via the prepended SpanCustomization behavior)
|
|
11
13
|
# - groups spans by braintrust.parent and sets the x-bt-parent header per group,
|
|
12
14
|
# so the backend routes them to the correct experiment/project
|
|
13
15
|
#
|
|
14
16
|
# Thread safety: BatchSpanProcessor serializes export() calls via its
|
|
15
17
|
# @export_mutex, so @headers mutation here is safe.
|
|
16
18
|
class SpanExporter < OpenTelemetry::Exporter::OTLP::Exporter
|
|
19
|
+
# The last prepended runs first: origin, then customization, then export below.
|
|
20
|
+
prepend SpanCustomization
|
|
17
21
|
prepend SpanOrigin
|
|
18
22
|
|
|
19
23
|
PARENT_ATTR_KEY = SpanProcessor::PARENT_ATTR_KEY
|
|
@@ -24,11 +24,15 @@ module Braintrust
|
|
|
24
24
|
# @param span_data [Array<OpenTelemetry::SDK::Trace::SpanData>]
|
|
25
25
|
# @return [Integer] export result from the wrapped exporter
|
|
26
26
|
def export(span_data, timeout: nil)
|
|
27
|
+
super(SpanOrigin.enrich_batch(span_data), timeout: timeout)
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
# Enrich a batch of SpanData with span origin provenance.
|
|
31
|
+
def self.enrich_batch(span_data)
|
|
27
32
|
# Environment is process-global and stable; read it once per batch
|
|
28
33
|
# rather than once per span. It is cheap (ENV reads only).
|
|
29
34
|
environment = Internal::Env.detect_environment
|
|
30
|
-
|
|
31
|
-
super(enriched, timeout: timeout)
|
|
35
|
+
span_data.map { |sd| enrich(sd, environment: environment) }
|
|
32
36
|
end
|
|
33
37
|
|
|
34
38
|
# Enrich a single SpanData with span origin provenance.
|
data/lib/braintrust/trace.rb
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require "opentelemetry/sdk"
|
|
4
|
+
require "uri"
|
|
4
5
|
require "opentelemetry/exporter/otlp"
|
|
5
6
|
require_relative "trace/span_processor"
|
|
6
7
|
require_relative "trace/span_exporter"
|
|
@@ -88,10 +89,16 @@ module Braintrust
|
|
|
88
89
|
# Get config from state if available
|
|
89
90
|
config ||= state.respond_to?(:config) ? state.config : nil
|
|
90
91
|
|
|
92
|
+
# Customizers only run in SpanExporter; refuse to silently skip them (e.g. redaction).
|
|
93
|
+
if exporter && config&.span_customizers&.any?
|
|
94
|
+
raise ArgumentError, "span_customizers are not supported with a custom exporter"
|
|
95
|
+
end
|
|
96
|
+
|
|
91
97
|
# Create OTLP HTTP exporter unless override provided
|
|
92
98
|
exporter ||= SpanExporter.new(
|
|
93
99
|
endpoint: "#{state.api_url}/otel/v1/traces",
|
|
94
|
-
api_key: state.api_key
|
|
100
|
+
api_key: state.api_key,
|
|
101
|
+
span_customizers: config&.span_customizers
|
|
95
102
|
)
|
|
96
103
|
|
|
97
104
|
# Use SimpleSpanProcessor for InMemorySpanExporter (testing), BatchSpanProcessor for production
|
|
@@ -185,14 +192,18 @@ module Braintrust
|
|
|
185
192
|
return ""
|
|
186
193
|
end
|
|
187
194
|
|
|
195
|
+
# Encode each path segment so names with spaces or slashes remain one segment.
|
|
196
|
+
escaped_org_name = URI::DEFAULT_PARSER.escape(org_name, /[^A-Za-z0-9\-._~]/)
|
|
197
|
+
escaped_parent_id = URI::DEFAULT_PARSER.escape(parent_id, /[^A-Za-z0-9\-._~]/)
|
|
198
|
+
|
|
188
199
|
# Build the permalink URL based on parent type
|
|
189
200
|
if parent_type == "experiment_id"
|
|
190
201
|
# For experiments: {app_url}/app/{org}/object?object_type=experiment&object_id={experiment_id}&r={trace_id}&s={span_id}
|
|
191
|
-
"#{app_url}/app/#{
|
|
202
|
+
"#{app_url}/app/#{escaped_org_name}/object?object_type=experiment&object_id=#{parent_id}&r=#{trace_id}&s=#{span_id}"
|
|
192
203
|
else
|
|
193
204
|
# For projects: {app_url}/app/{org}/p/{project}/logs?r={trace_id}&s={span_id}
|
|
194
205
|
# parent_type is typically "project_name"
|
|
195
|
-
"#{app_url}/app/#{
|
|
206
|
+
"#{app_url}/app/#{escaped_org_name}/p/#{escaped_parent_id}/logs?r=#{trace_id}&s=#{span_id}"
|
|
196
207
|
end
|
|
197
208
|
rescue => e
|
|
198
209
|
Log.error("Failed to generate permalink: #{e.message}")
|
data/lib/braintrust/version.rb
CHANGED
data/lib/braintrust.rb
CHANGED
|
@@ -42,6 +42,7 @@ module Braintrust
|
|
|
42
42
|
# @param tracer_provider [TracerProvider, nil] Optional tracer provider to use instead of creating one
|
|
43
43
|
# @param filter_ai_spans [Boolean, nil] Enable AI span filtering (overrides BRAINTRUST_OTEL_FILTER_AI_SPANS env var)
|
|
44
44
|
# @param span_filter_funcs [Array<Proc>, nil] Custom span filter functions
|
|
45
|
+
# @param span_customizers [Array<#on_span_export>, nil] Ordered synchronous export customizers
|
|
45
46
|
# @param exporter [Exporter, nil] Optional exporter override (for testing)
|
|
46
47
|
# @param auto_instrument [Boolean, Hash, nil] Auto-instrumentation config:
|
|
47
48
|
# - nil (default): use BRAINTRUST_AUTO_INSTRUMENT env var, default true if not set
|
|
@@ -49,7 +50,7 @@ module Braintrust
|
|
|
49
50
|
# - false: explicitly disable
|
|
50
51
|
# - Hash with :only or :except keys for filtering
|
|
51
52
|
# @return [State] the created state
|
|
52
|
-
def self.init(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil, set_global: true, blocking_login: false, enable_tracing: true, tracer_provider: nil, filter_ai_spans: nil, span_filter_funcs: nil, exporter: nil, auto_instrument: nil)
|
|
53
|
+
def self.init(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil, set_global: true, blocking_login: false, enable_tracing: true, tracer_provider: nil, filter_ai_spans: nil, span_filter_funcs: nil, span_customizers: nil, exporter: nil, auto_instrument: nil)
|
|
53
54
|
state = State.from_env(
|
|
54
55
|
api_key: api_key,
|
|
55
56
|
org_name: org_name,
|
|
@@ -61,6 +62,7 @@ module Braintrust
|
|
|
61
62
|
tracer_provider: tracer_provider,
|
|
62
63
|
filter_ai_spans: filter_ai_spans,
|
|
63
64
|
span_filter_funcs: span_filter_funcs,
|
|
65
|
+
span_customizers: span_customizers,
|
|
64
66
|
exporter: exporter
|
|
65
67
|
)
|
|
66
68
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: braintrust
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.6.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Braintrust
|
|
@@ -120,6 +120,11 @@ files:
|
|
|
120
120
|
- lib/braintrust/contrib/rails/server/routes.rb
|
|
121
121
|
- lib/braintrust/contrib/rails/server/templates/initializer.rb.tt
|
|
122
122
|
- lib/braintrust/contrib/registry.rb
|
|
123
|
+
- lib/braintrust/contrib/roast/instrumentation/cog.rb
|
|
124
|
+
- lib/braintrust/contrib/roast/instrumentation/workflow.rb
|
|
125
|
+
- lib/braintrust/contrib/roast/integration.rb
|
|
126
|
+
- lib/braintrust/contrib/roast/patcher.rb
|
|
127
|
+
- lib/braintrust/contrib/roast/tracing.rb
|
|
123
128
|
- lib/braintrust/contrib/ruby_llm/deprecated.rb
|
|
124
129
|
- lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb
|
|
125
130
|
- lib/braintrust/contrib/ruby_llm/instrumentation/common.rb
|
|
@@ -181,6 +186,9 @@ files:
|
|
|
181
186
|
- lib/braintrust/task.rb
|
|
182
187
|
- lib/braintrust/trace.rb
|
|
183
188
|
- lib/braintrust/trace/attachment.rb
|
|
189
|
+
- lib/braintrust/trace/span_customization.rb
|
|
190
|
+
- lib/braintrust/trace/span_customizers.rb
|
|
191
|
+
- lib/braintrust/trace/span_export_data.rb
|
|
184
192
|
- lib/braintrust/trace/span_exporter.rb
|
|
185
193
|
- lib/braintrust/trace/span_filter.rb
|
|
186
194
|
- lib/braintrust/trace/span_origin.rb
|