braintrust 0.5.0 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 38aa89e0aba8b3df517d42a95d016cb09cbfbaeac91d16d4caed00f0779415eb
4
- data.tar.gz: fd800de8e09a8ac7ca93342d65abd6c19dcdc5bffe2421dd5cf921e1b6b06645
3
+ metadata.gz: 2e67bcb9e3752500ed9ac74f3ad5d46b7ee86867f648d772a4138b5491e3f405
4
+ data.tar.gz: 7949cd6453a74de3632a52704a13616ff620c85429f51bff1bb6405954ea9d9d
5
5
  SHA512:
6
- metadata.gz: 82c01479c75cbb51bc79a67af1b91c46af0fdefc2895c05a54837fb38a2bd501a5e462e1cff953fc83e189ad9e1da891e39814077bb0cd03fd66970eccb285a9
7
- data.tar.gz: '0757861a2df4962b7cdcff628c5638fe361e33b78ba76417dd3c1087c557fffc7920a857a3916f95bc65e3d6f8bb676afb848a27a6fef28dc987b0d7e237c922'
6
+ metadata.gz: 8b07cd2821efe35fc99235dfa5fde7c06ed3dd147262c30633e1469ac0d0bd30a92fcaabef301b19be5d27dda98542b3008c01c97d2d00e4bf35ee09996c602f
7
+ data.tar.gz: 7b8075fc99973163cd432253c5c9edd34e51e9b79fb4dd80dca7c8fd472716567acf4b68a6008be46545414c358b192ff180a4b07c76b722d0d691b69b219d47
data/README.md CHANGED
@@ -17,9 +17,10 @@ This is the official Ruby SDK for [Braintrust](https://www.braintrust.dev), for
17
17
  - [Braintrust.init](#braintrustinit)
18
18
  - [Environment variables](#environment-variables)
19
19
  - [Tracing](#tracing)
20
- - [Supported providers](#supported-providers)
20
+ - [Supported integrations](#supported-integrations)
21
21
  - [Manually applying instrumentation](#manually-applying-instrumentation)
22
22
  - [Creating custom spans](#creating-custom-spans)
23
+ - [Span customizers](#span-customizers)
23
24
  - [Attachments](#attachments)
24
25
  - [Viewing traces](#viewing-traces)
25
26
  - [Evals](#evals)
@@ -120,6 +121,7 @@ Braintrust.init
120
121
  | `filter_ai_spans` | `ENV['BRAINTRUST_OTEL_FILTER_AI_SPANS']` | Only export AI-related spans |
121
122
  | `org_name` | `ENV['BRAINTRUST_ORG_NAME']` | Organization name |
122
123
  | `set_global` | `true` | Set as global state. Set to `false` for isolated instances |
124
+ | `span_customizers` | `[]` | Redact or transform spans before export (see [Span customizers](#span-customizers)) |
123
125
 
124
126
  **Example with options:**
125
127
 
@@ -148,16 +150,17 @@ Braintrust.init(
148
150
 
149
151
  ## Tracing
150
152
 
151
- ### Supported providers
153
+ ### Supported integrations
152
154
 
153
- The SDK automatically instruments these LLM libraries:
155
+ The SDK automatically instruments these libraries:
154
156
 
155
- | Provider | Gem | Versions | Integration Name | Examples |
156
- | --------- | ------------- | -------- | ---------------- | ----------------------------------------- |
157
- | Anthropic | `anthropic` | >= 0.3.0 | `:anthropic` | [Link](./examples/contrib/anthropic.rb) |
158
- | OpenAI | `openai` | >= 0.1.0 | `:openai` | [Link](./examples/contrib/openai.rb) |
159
- | | `ruby-openai` | >= 7.0.0 | `:ruby_openai` | [Link](./examples/contrib/ruby-openai.rb) |
160
- | Multiple | `ruby_llm` | >= 1.8.0 | `:ruby_llm` | [Link](./examples/contrib/ruby_llm.rb) |
157
+ | Category | Gem | Versions | Integration Name | Examples |
158
+ | --------- | ------------- | -------- | ---------------- | ----------------------------------------------- |
159
+ | Anthropic | `anthropic` | >= 0.3.0 | `:anthropic` | [Link](./examples/contrib/anthropic/basic.rb) |
160
+ | OpenAI | `openai` | >= 0.1.0 | `:openai` | [Link](./examples/contrib/openai/basic.rb) |
161
+ | | `ruby-openai` | >= 7.0.0 | `:ruby_openai` | [Link](./examples/contrib/ruby_openai/basic.rb) |
162
+ | Multiple | `ruby_llm` | >= 1.8.0 | `:ruby_llm` | [Link](./examples/contrib/ruby_llm/basic.rb) |
163
+ | Workflows | `roast-ai` | >= 1.0.0 | `:roast` | [Link](./examples/contrib/roast/basic.rb) |
161
164
 
162
165
  ### Manually applying instrumentation
163
166
 
@@ -192,6 +195,33 @@ tracer.in_span("process-request") do |span|
192
195
  end
193
196
  ```
194
197
 
198
+ ### Span customizers
199
+
200
+ Redact or transform span data before it leaves your application, e.g. to strip PII from LLM inputs and outputs:
201
+
202
+ ```ruby
203
+ # Any object that responds to #on_span_export
204
+ class RedactContent
205
+ def on_span_export(span)
206
+ %w[braintrust.input_json braintrust.output_json].each do |key|
207
+ span.attributes[key] = JSON.generate("[redacted]") if span.attributes.key?(key)
208
+ end
209
+ span.attributes.delete("user.email")
210
+ span
211
+ end
212
+ end
213
+
214
+ Braintrust.init(span_customizers: [RedactContent.new])
215
+ ```
216
+
217
+ **When configuring customizers...**
218
+
219
+ - Do not change the `trace_id`, `span_id`, or `parent_span_id` fields.
220
+ - Always return the span (or a replacement `OpenTelemetry::SDK::Trace::SpanData`).
221
+ - Keep them fast; otherwise they may slow down exporting.
222
+ - Customizers run in order on every finished span sent to Braintrust.
223
+ - If it raises or returns something invalid, the batch is dropped and an error is logged.
224
+
195
225
  ### Attachments
196
226
 
197
227
  Log binary data (images, PDFs, audio) in your traces:
@@ -567,11 +597,11 @@ The dev server requires the `rack` gem and a Rack-compatible web server.
567
597
  | [Passenger](https://www.phusionpassenger.com/) | 6.x | |
568
598
  | [WEBrick](https://github.com/ruby/webrick) | Not supported | Does not support server-sent events. |
569
599
 
570
- See examples: [server/eval.ru](./examples/server/eval.ru),
600
+ See examples: [server/eval.ru](./examples/server/eval.ru),
571
601
 
572
602
  ## Documentation
573
603
 
574
- - [Braintrust Documentation](https://www.braintrust.dev/docs)
604
+ - [Braintrust Documentation](https://docs.braintrust.dev)
575
605
  - [API Reference](https://gemdocs.org/gems/braintrust/)
576
606
 
577
607
  ## Troubleshooting
@@ -8,10 +8,10 @@ module Braintrust
8
8
  # and allows overriding with explicit options
9
9
  class Config
10
10
  attr_reader :api_key, :org_name, :default_project, :app_url, :api_url,
11
- :filter_ai_spans, :span_filter_funcs
11
+ :filter_ai_spans, :span_filter_funcs, :span_customizers
12
12
 
13
13
  def initialize(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil,
14
- filter_ai_spans: nil, span_filter_funcs: nil)
14
+ filter_ai_spans: nil, span_filter_funcs: nil, span_customizers: nil)
15
15
  @api_key = api_key
16
16
  @org_name = org_name
17
17
  @default_project = default_project
@@ -19,6 +19,7 @@ module Braintrust
19
19
  @api_url = api_url
20
20
  @filter_ai_spans = filter_ai_spans
21
21
  @span_filter_funcs = span_filter_funcs || []
22
+ @span_customizers = (span_customizers || []).dup.freeze
22
23
  end
23
24
 
24
25
  # Create a Config from environment variables, with option overrides
@@ -30,9 +31,10 @@ module Braintrust
30
31
  # @param api_url [String, nil] API URL (overrides BRAINTRUST_API_URL env var)
31
32
  # @param filter_ai_spans [Boolean, nil] Enable AI span filtering (overrides BRAINTRUST_OTEL_FILTER_AI_SPANS env var)
32
33
  # @param span_filter_funcs [Array<Proc>, nil] Custom span filter functions
34
+ # @param span_customizers [Array<#on_span_export>, nil] Ordered export customizers (copied and frozen)
33
35
  # @return [Config] the created config
34
36
  def self.from_env(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil,
35
- filter_ai_spans: nil, span_filter_funcs: nil)
37
+ filter_ai_spans: nil, span_filter_funcs: nil, span_customizers: nil)
36
38
  # Parse filter_ai_spans from ENV if not explicitly provided
37
39
  env_filter_ai_spans = ENV["BRAINTRUST_OTEL_FILTER_AI_SPANS"]
38
40
  filter_ai_spans_value = if filter_ai_spans.nil?
@@ -48,7 +50,8 @@ module Braintrust
48
50
  app_url: app_url || ENV["BRAINTRUST_APP_URL"] || "https://www.braintrust.dev",
49
51
  api_url: api_url || ENV["BRAINTRUST_API_URL"] || "https://api.braintrust.dev",
50
52
  filter_ai_spans: filter_ai_spans_value,
51
- span_filter_funcs: span_filter_funcs
53
+ span_filter_funcs: span_filter_funcs,
54
+ span_customizers: span_customizers
52
55
  )
53
56
  end
54
57
  end
@@ -0,0 +1,97 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative "../../support/otel"
4
+ require_relative "../tracing"
5
+
6
+ module Braintrust
7
+ module Contrib
8
+ module Roast
9
+ module Instrumentation
10
+ # Wrap the asynchronous work block rather than Cog#run! itself: run! only
11
+ # schedules a task, while the cog executes later in a different fiber.
12
+ module Cog
13
+ class BarrierProxy
14
+ def initialize(barrier, &around_task)
15
+ @barrier = barrier
16
+ @around_task = around_task
17
+ end
18
+
19
+ def async(**options, &task)
20
+ @barrier.async(**options) do |async_task|
21
+ @around_task.call(async_task, task)
22
+ end
23
+ end
24
+ end
25
+
26
+ def run!(barrier, config, input_context, executor_scope_value, executor_scope_index)
27
+ parent_context = roast_parent_context
28
+ return super unless tracing_enabled? && parent_context
29
+
30
+ tracer = Braintrust::Contrib::Roast::Tracing.tracer_for(self)
31
+ proxy = BarrierProxy.new(barrier) do |async_task, task|
32
+ ::OpenTelemetry::Context.with_current(parent_context) do
33
+ span_name = "roast.cog.#{name}"
34
+ tracer.in_span(span_name) do |span|
35
+ # Braintrust uses span_attributes for type/name; Roast details go in metadata.
36
+ Support::OTel.set_json_attr(span, "braintrust.span_attributes", {type: "task", name: span_name})
37
+ Braintrust::Contrib::Context.set!(self, roast_span_context: ::OpenTelemetry::Context.current)
38
+ task.call(async_task)
39
+ rescue => e
40
+ span.record_exception(e)
41
+ span.status = ::OpenTelemetry::Trace::Status.error(e.message)
42
+ raise
43
+ ensure
44
+ record_outcome(span)
45
+ end
46
+ end
47
+ end
48
+ super(proxy, config, input_context, executor_scope_value, executor_scope_index)
49
+ end
50
+
51
+ private
52
+
53
+ def roast_parent_context
54
+ path = ::Roast::TaskContext.path
55
+ path.reverse_each do |element|
56
+ context = Braintrust::Contrib.context_for(element.cog)&.[](:roast_span_context)
57
+ return context if context
58
+ end
59
+ path.reverse_each do |element|
60
+ context = Braintrust::Contrib.context_for(element.execution_manager&.workflow_context)&.[](:roast_span_context)
61
+ return context if context
62
+ end
63
+ nil
64
+ end
65
+
66
+ def tracing_enabled?
67
+ context = Braintrust::Contrib.context_for(self)
68
+ class_context = Braintrust::Contrib.context_for(self.class)
69
+ context&.[](:enabled) != false && class_context&.[](:enabled) != false
70
+ end
71
+
72
+ def record_outcome(span)
73
+ outcome = if failed?
74
+ "failed"
75
+ elsif skipped?
76
+ "skipped"
77
+ else
78
+ "completed"
79
+ end
80
+ Support::OTel.set_json_attr(span, "braintrust.metadata", {
81
+ "contrib.roast.cog.type" => type,
82
+ "contrib.roast.cog.name" => name.to_s,
83
+ "contrib.roast.cog.outcome" => outcome
84
+ })
85
+
86
+ output_value = case output
87
+ when ::Roast::Cogs::Ruby::Output then output.value
88
+ when ::Roast::Cogs::Chat::Output then output.response
89
+ end
90
+ Support::OTel.set_json_attr(span, "braintrust.output_json", output_value) if output_value.is_a?(String)
91
+ span.status = ::OpenTelemetry::Trace::Status.error("Cog failed") if failed?
92
+ end
93
+ end
94
+ end
95
+ end
96
+ end
97
+ end
@@ -0,0 +1,49 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative "../../support/otel"
4
+ require_relative "../tracing"
5
+
6
+ module Braintrust
7
+ module Contrib
8
+ module Roast
9
+ module Instrumentation
10
+ # from_file enters a new Async fiber, so carry its caller's OTel context
11
+ # through the params object that is handed to the workflow.
12
+ module WorkflowClass
13
+ def from_file(workflow_path, params)
14
+ previous_parent = Braintrust::Contrib.context_for(params)&.[](:roast_parent_context)
15
+ Braintrust::Contrib::Context.set!(params, roast_parent_context: ::OpenTelemetry::Context.current)
16
+ super
17
+ ensure
18
+ Braintrust::Contrib::Context.set!(params, roast_parent_context: previous_parent)
19
+ end
20
+ end
21
+
22
+ module Workflow
23
+ def start!
24
+ context = Braintrust::Contrib.context_for(self)
25
+ class_context = Braintrust::Contrib.context_for(self.class)
26
+ return super if context&.[](:enabled) == false || class_context&.[](:enabled) == false
27
+
28
+ tracer = Braintrust::Contrib::Roast::Tracing.tracer_for(self)
29
+ workflow_context = @workflow_context
30
+ workflow_path = @workflow_path
31
+ params = workflow_context.params
32
+ parent_context = Braintrust::Contrib.context_for(params)&.[](:roast_parent_context)
33
+ trace_workflow = proc do
34
+ tracer.in_span("roast.workflow") do |span|
35
+ Support::OTel.set_json_attr(span, "braintrust.span_attributes", {type: "task", name: "roast.workflow"})
36
+ Support::OTel.set_json_attr(span, "braintrust.metadata", {"contrib.roast.workflow.name" => File.basename(workflow_path.to_s)})
37
+ input = {targets: params.targets, args: params.args, kwargs: params.kwargs}.reject { |_, value| value.nil? || value.empty? }
38
+ Support::OTel.set_json_attr(span, "braintrust.input_json", input) unless input.empty?
39
+ Braintrust::Contrib::Context.set!(workflow_context, roast_span_context: ::OpenTelemetry::Context.current)
40
+ super()
41
+ end
42
+ end
43
+ parent_context ? ::OpenTelemetry::Context.with_current(parent_context, &trace_workflow) : trace_workflow.call
44
+ end
45
+ end
46
+ end
47
+ end
48
+ end
49
+ end
@@ -0,0 +1,32 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative "../integration"
4
+
5
+ module Braintrust
6
+ module Contrib
7
+ module Roast
8
+ # Roast workflow and cog instrumentation.
9
+ class Integration
10
+ include Braintrust::Contrib::Integration
11
+
12
+ GEM_NAMES = ["roast-ai"].freeze
13
+ REQUIRE_PATHS = ["roast"].freeze
14
+ MINIMUM_VERSION = "1.0.0"
15
+
16
+ def self.integration_name = :roast
17
+ def self.gem_names = GEM_NAMES
18
+ def self.require_paths = REQUIRE_PATHS
19
+ def self.minimum_version = MINIMUM_VERSION
20
+
21
+ def self.loaded?
22
+ (defined?(::Roast::Workflow) && defined?(::Roast::Cog)) ? true : false
23
+ end
24
+
25
+ def self.patchers
26
+ require_relative "patcher"
27
+ [WorkflowPatcher, CogPatcher]
28
+ end
29
+ end
30
+ end
31
+ end
32
+ end
@@ -0,0 +1,42 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative "../patcher"
4
+ require_relative "integration"
5
+ require_relative "instrumentation/workflow"
6
+ require_relative "instrumentation/cog"
7
+
8
+ module Braintrust
9
+ module Contrib
10
+ module Roast
11
+ class WorkflowPatcher < Braintrust::Contrib::Patcher
12
+ def self.applicable? = defined?(::Roast::Workflow)
13
+
14
+ def self.patched?(**options)
15
+ klass = options[:target]&.singleton_class || ::Roast::Workflow
16
+ klass.ancestors.include?(Instrumentation::Workflow) &&
17
+ (options[:target] || ::Roast::Workflow.singleton_class.ancestors.include?(Instrumentation::WorkflowClass))
18
+ end
19
+
20
+ def self.perform_patch(**options)
21
+ target = options[:target]
22
+ raise ArgumentError, "target must be a Roast::Workflow" if target && !target.is_a?(::Roast::Workflow)
23
+
24
+ (target&.singleton_class || ::Roast::Workflow).prepend(Instrumentation::Workflow)
25
+ ::Roast::Workflow.singleton_class.prepend(Instrumentation::WorkflowClass) unless target
26
+ end
27
+ end
28
+
29
+ class CogPatcher < Braintrust::Contrib::Patcher
30
+ def self.applicable? = defined?(::Roast::Cog)
31
+
32
+ def self.patched?(**)
33
+ ::Roast::Cog.ancestors.include?(Instrumentation::Cog)
34
+ end
35
+
36
+ def self.perform_patch(**)
37
+ ::Roast::Cog.prepend(Instrumentation::Cog)
38
+ end
39
+ end
40
+ end
41
+ end
42
+ end
@@ -0,0 +1,15 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Braintrust
4
+ module Contrib
5
+ module Roast
6
+ module Tracing
7
+ INSTRUMENTATION_NAME = "braintrust.contrib.roast"
8
+
9
+ def self.tracer_for(target)
10
+ Braintrust::Contrib.tracer_for(target, name: INSTRUMENTATION_NAME)
11
+ end
12
+ end
13
+ end
14
+ end
15
+ end
@@ -13,43 +13,76 @@ module Braintrust
13
13
  module RubyLLM
14
14
  module Instrumentation
15
15
  # Chat instrumentation for RubyLLM.
16
- # Wraps complete() and execute_tool() methods to create spans.
16
+ # Wraps the method that makes one chat request, and execute_tool(), to create spans.
17
17
  module Chat
18
18
  def self.included(base)
19
19
  # Guard against double-wrapping
20
- base.prepend(InstanceMethods) unless applied?(base)
20
+ return if applied?(base)
21
+
22
+ base.prepend(InstanceMethods)
23
+ base.prepend(base.public_method_defined?(:generate) ? GenerateTracing : CompleteTracing)
21
24
  end
22
25
 
23
26
  def self.applied?(base)
24
27
  base.ancestors.include?(InstanceMethods)
25
28
  end
26
29
 
27
- module InstanceMethods
28
- # Wrap complete() to trace chat completions.
29
- # Each call creates a span - recursive calls from tool execution
30
- # create nested spans (each is a separate API call).
30
+ # ruby_llm 1.x: each complete() call makes one request, and recursive calls from
31
+ # tool execution create nested spans.
32
+ module CompleteTracing
33
+ def complete(&block)
34
+ return block ? super : super() unless tracing_enabled?
35
+
36
+ trace_chat_request(block) { |wrapper| super(&wrapper) }
37
+ end
38
+ end
39
+
40
+ # ruby_llm 2.x: complete() loops over step(), which calls generate() once per
41
+ # request, so each generate() call gets its own span. complete() gets a parent
42
+ # span so a turn's chat and tool spans share one trace when the caller has no span.
43
+ module GenerateTracing
44
+ def generate(&block)
45
+ return block ? super : super() unless tracing_enabled?
46
+
47
+ trace_chat_request(block) { |wrapper| super(&wrapper) }
48
+ end
49
+
31
50
  def complete(&block)
32
51
  return block ? super : super() unless tracing_enabled?
33
52
 
34
53
  tracer = Braintrust::Contrib.tracer_for(self)
35
54
 
55
+ tracer.in_span("ruby_llm.complete") do |span|
56
+ Support::OTel.set_json_attr(span, "braintrust.span_attributes", {type: "task"})
57
+ input_messages = build_input_messages
58
+ Support::OTel.set_json_attr(span, "braintrust.input_json", input_messages) if input_messages.any?
59
+
60
+ response = block ? super : super()
61
+
62
+ # No metrics here: each request's tokens live on its ruby_llm.chat child span.
63
+ Support::OTel.set_json_attr(span, "braintrust.output_json", [format_message_for_input(response)]) if response
64
+ response
65
+ end
66
+ end
67
+ end
68
+
69
+ module InstanceMethods
70
+ private
71
+
72
+ # Traces one chat request. Yields the streaming wrapper block, or nil when not
73
+ # streaming, for the caller to pass to super.
74
+ def trace_chat_request(block)
75
+ tracer = Braintrust::Contrib.tracer_for(self)
76
+
36
77
  tracer.in_span("ruby_llm.chat") do |span|
37
78
  if block
38
- # Streaming: pass a block that calls super() with the wrapper
39
- handle_streaming_complete(span, block) do |&wrapper|
40
- super(&wrapper)
41
- end
79
+ handle_streaming_complete(span, block) { |&wrapper| yield wrapper }
42
80
  else
43
- # Non-streaming: pass a block that calls super()
44
- handle_non_streaming_complete(span) do
45
- super()
46
- end
81
+ handle_non_streaming_complete(span) { yield nil }
47
82
  end
48
83
  end
49
84
  end
50
85
 
51
- private
52
-
53
86
  # Wrap execute_tool() to trace tool executions.
54
87
  # This is a private method in RubyLLM - wrapping it avoids
55
88
  # conflicting with user-registered on_tool_call/on_tool_result callbacks.
@@ -199,22 +232,18 @@ module Braintrust
199
232
  tool_schema = tool_params_schema(tool) ? build_basic_tool_schema(tool) : build_minimal_tool_schema(tool)
200
233
  end
201
234
 
202
- # Strip RubyLLM-specific fields to match native OpenAI format
235
+ # Strip RubyLLM-specific fields to match native OpenAI format. Build copies: the
236
+ # parameters hash is the tool's cached schema, which RubyLLM sends to the provider.
203
237
  function_key = tool_schema&.key?(:function) ? :function : "function"
204
- if tool_schema && tool_schema[function_key]
205
- tool_params = tool_schema[function_key][:parameters] || tool_schema[function_key]["parameters"]
206
- if tool_params.is_a?(Hash)
207
- tool_params = tool_params.dup if tool_params.frozen?
208
- tool_params.delete("strict")
209
- tool_params.delete(:strict)
210
- tool_params.delete("additionalProperties")
211
- tool_params.delete(:additionalProperties)
212
- params_key = tool_schema[function_key].key?(:parameters) ? :parameters : "parameters"
213
- tool_schema[function_key][params_key] = tool_params
214
- end
215
- end
238
+ function = tool_schema && tool_schema[function_key]
239
+ return tool_schema unless function
240
+
241
+ params_key = function.key?(:parameters) ? :parameters : "parameters"
242
+ tool_params = function[params_key]
243
+ return tool_schema unless tool_params.is_a?(Hash)
216
244
 
217
- tool_schema
245
+ stripped = tool_params.reject { |key, _| %w[strict additionalProperties].include?(key.to_s) }
246
+ tool_schema.merge(function_key => function.merge(params_key => stripped))
218
247
  end
219
248
 
220
249
  # A tool's JSON Schema, across ruby_llm versions.
@@ -388,7 +417,8 @@ module Braintrust
388
417
 
389
418
  # Format tool calls into OpenAI format
390
419
  def format_tool_calls(tool_calls)
391
- tool_calls.map do |_id, tc|
420
+ # Both the pre-2.2 Hash and ToolCall::Collection expose values.
421
+ tool_calls.values.map do |tc|
392
422
  args = tc.arguments
393
423
  args_string = args.is_a?(String) ? args : JSON.generate(args)
394
424
 
@@ -423,20 +453,7 @@ module Braintrust
423
453
 
424
454
  # Set metrics (token usage + time_to_first_token)
425
455
  # RubyLLM stores tokens directly in the response hash, not in a nested usage object
426
- metrics = {}
427
- if result.respond_to?(:to_h)
428
- result_hash = result.to_h
429
- usage = {
430
- "input_tokens" => result_hash[:input_tokens],
431
- "output_tokens" => result_hash[:output_tokens],
432
- "cached_tokens" => result_hash[:cached_tokens],
433
- "cache_creation_tokens" => result_hash[:cache_creation_tokens]
434
- }.compact
435
-
436
- unless usage.empty?
437
- metrics = Common.parse_usage_tokens(usage)
438
- end
439
- end
456
+ metrics = Common.message_usage_metrics(result)
440
457
  metrics["time_to_first_token"] = time_to_first_token if time_to_first_token
441
458
  Support::OTel.set_json_attr(span, "braintrust.metrics", metrics) unless metrics.empty?
442
459
  end
@@ -479,20 +496,7 @@ module Braintrust
479
496
  Support::OTel.set_json_attr(span, "braintrust.output_json", output)
480
497
 
481
498
  # Set metrics (token usage + time_to_first_token)
482
- metrics = {}
483
- if response.respond_to?(:to_h)
484
- response_hash = response.to_h
485
- usage = {
486
- "input_tokens" => response_hash[:input_tokens],
487
- "output_tokens" => response_hash[:output_tokens],
488
- "cached_tokens" => response_hash[:cached_tokens],
489
- "cache_creation_tokens" => response_hash[:cache_creation_tokens]
490
- }.compact
491
-
492
- unless usage.empty?
493
- metrics = Common.parse_usage_tokens(usage)
494
- end
495
- end
499
+ metrics = Common.message_usage_metrics(response)
496
500
  metrics["time_to_first_token"] = time_to_first_token if time_to_first_token
497
501
  Support::OTel.set_json_attr(span, "braintrust.metrics", metrics) unless metrics.empty?
498
502
  end
@@ -6,6 +6,24 @@ module Braintrust
6
6
  module Instrumentation
7
7
  # Common utilities for RubyLLM instrumentation.
8
8
  module Common
9
+ # Braintrust metrics for a RubyLLM message's token counts. ruby_llm 2.x renamed the cache
10
+ # counts from cached_tokens/cache_creation_tokens to cache_read_tokens/cache_write_tokens.
11
+ # @param message [Object] a RubyLLM message (anything responding to to_h)
12
+ # @return [Hash<String, Integer>] normalized metrics, empty when no counts are reported
13
+ def self.message_usage_metrics(message)
14
+ return {} unless message.respond_to?(:to_h)
15
+
16
+ counts = message.to_h
17
+ usage = {
18
+ "input_tokens" => counts[:input_tokens],
19
+ "output_tokens" => counts[:output_tokens],
20
+ "cached_tokens" => counts[:cached_tokens] || counts[:cache_read_tokens],
21
+ "cache_creation_tokens" => counts[:cache_creation_tokens] || counts[:cache_write_tokens]
22
+ }.compact
23
+
24
+ usage.empty? ? {} : parse_usage_tokens(usage)
25
+ end
26
+
9
27
  # Parse RubyLLM usage tokens into normalized Braintrust metrics.
10
28
  # RubyLLM normalizes token fields from all providers (OpenAI, Anthropic, etc.)
11
29
  # into a consistent format:
@@ -7,7 +7,7 @@ module Braintrust
7
7
  module Contrib
8
8
  module RubyLLM
9
9
  # Patcher for RubyLLM chat completions.
10
- # Instruments RubyLLM::Chat#complete and #execute_tool methods.
10
+ # Instruments RubyLLM::Chat#complete (plus #generate on 2.x) and #execute_tool.
11
11
  class ChatPatcher < Braintrust::Contrib::Patcher
12
12
  class << self
13
13
  def applicable?
@@ -197,9 +197,11 @@ require_relative "contrib/openai/integration"
197
197
  require_relative "contrib/ruby_openai/integration"
198
198
  require_relative "contrib/ruby_llm/integration"
199
199
  require_relative "contrib/anthropic/integration"
200
+ require_relative "contrib/roast/integration"
200
201
 
201
202
  # Register integrations
202
203
  Braintrust::Contrib::OpenAI::Integration.register!
203
204
  Braintrust::Contrib::RubyOpenAI::Integration.register!
204
205
  Braintrust::Contrib::RubyLLM::Integration.register!
205
206
  Braintrust::Contrib::Anthropic::Integration.register!
207
+ Braintrust::Contrib::Roast::Integration.register!
@@ -24,9 +24,10 @@ module Braintrust
24
24
  # @param tracer_provider [TracerProvider, nil] Optional tracer provider to use
25
25
  # @param filter_ai_spans [Boolean, nil] Enable AI span filtering
26
26
  # @param span_filter_funcs [Array<Proc>, nil] Custom span filter functions
27
+ # @param span_customizers [Array<#on_span_export>, nil] Ordered synchronous export customizers
27
28
  # @param exporter [Exporter, nil] Optional exporter override (for testing)
28
29
  # @return [State] the created state
29
- def self.from_env(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil, blocking_login: false, enable_tracing: true, tracer_provider: nil, filter_ai_spans: nil, span_filter_funcs: nil, exporter: nil)
30
+ def self.from_env(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil, blocking_login: false, enable_tracing: true, tracer_provider: nil, filter_ai_spans: nil, span_filter_funcs: nil, span_customizers: nil, exporter: nil)
30
31
  require_relative "config"
31
32
  config = Config.from_env(
32
33
  api_key: api_key,
@@ -35,7 +36,8 @@ module Braintrust
35
36
  app_url: app_url,
36
37
  api_url: api_url,
37
38
  filter_ai_spans: filter_ai_spans,
38
- span_filter_funcs: span_filter_funcs
39
+ span_filter_funcs: span_filter_funcs,
40
+ span_customizers: span_customizers
39
41
  )
40
42
  new(
41
43
  api_key: config.api_key,
@@ -0,0 +1,40 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative "span_customizers"
4
+ require_relative "../logger"
5
+
6
+ module Braintrust
7
+ module Trace
8
+ # Span customizer behavior for exporters.
9
+ #
10
+ # Prepend it onto any exporter whose +export(span_data, timeout:)+ it can
11
+ # +super+ into, and the exporter accepts a +span_customizers:+ option on
12
+ # construction and runs those customizers on every export. Prepend it
13
+ # before SpanOrigin, so SpanOrigin runs first and hooks see origin-enriched
14
+ # data:
15
+ #
16
+ # prepend SpanCustomization
17
+ # prepend SpanOrigin
18
+ #
19
+ # Customization is fail-closed: if any hook fails, none of the batch is
20
+ # passed on to the wrapped exporter.
21
+ module SpanCustomization
22
+ # Consumes +span_customizers:+ and forwards everything else to the exporter.
23
+ def initialize(*args, span_customizers: nil, **options, &block)
24
+ @span_customizers = SpanCustomizers.new(span_customizers)
25
+ super(*args, **options, &block)
26
+ end
27
+
28
+ def export(span_data, timeout: nil)
29
+ super(span_customizers.customize(span_data), timeout: timeout)
30
+ rescue SpanCustomizers::Error => e
31
+ Log.error("Span customization failed; batch not sent: #{e.message}")
32
+ OpenTelemetry::SDK::Trace::Export::FAILURE
33
+ end
34
+
35
+ private
36
+
37
+ attr_reader :span_customizers
38
+ end
39
+ end
40
+ end
@@ -0,0 +1,110 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative "span_export_data"
4
+
5
+ module Braintrust
6
+ module Trace
7
+ # Ordered export-time transformations, independent of origin and transport.
8
+ class SpanCustomizers
9
+ # A hook failed. Messages are SDK-generated: they identify the customizer
10
+ # and the kind of failure, never span payloads or the hook's own exception
11
+ # message, either of which may contain sensitive data.
12
+ class Error < StandardError; end
13
+
14
+ ID_FIELDS = %i[trace_id span_id parent_span_id].freeze
15
+ private_constant :ID_FIELDS
16
+
17
+ def initialize(customizers = nil)
18
+ @customizers = (customizers || []).dup.freeze
19
+ @customizers.each { |customizer| validate!(customizer) }
20
+ end
21
+
22
+ # Transform the whole batch before the exporter groups or serializes it.
23
+ # Raises Error so the exporter can fail the batch without sending.
24
+ def customize(span_data)
25
+ return span_data if @customizers.empty?
26
+
27
+ # Hooks may call instrumented clients. Suppress tracing so those spans
28
+ # cannot re-enter the exporter and customize themselves forever.
29
+ OpenTelemetry::Common::Utilities.untraced do
30
+ span_data.map { |span| customize_span(span) }
31
+ end
32
+ end
33
+
34
+ private
35
+
36
+ # Reject objects without the hook up front, so a misspelled method name
37
+ # fails at registration instead of silently skipping redaction.
38
+ def validate!(customizer)
39
+ return if customizer.respond_to?(:on_span_export)
40
+
41
+ raise ArgumentError, "#{label(customizer)} must implement on_span_export"
42
+ end
43
+
44
+ def customize_span(span)
45
+ # Snapshot IDs so in-place mutation through to_span_data cannot slip past validation.
46
+ ids = identity(span)
47
+ drops = dropped_counts(span)
48
+ view = nil
49
+ @customizers.each do |customizer|
50
+ view ||= writable_view(span)
51
+ result = invoke(customizer, view)
52
+ span = unwrap(result, customizer)
53
+ raise Error, "#{label(customizer)} changed span IDs; trace_id, span_id and parent_span_id must be preserved" unless identity(span) == ids
54
+
55
+ view = (result if result.is_a?(SpanExportData))
56
+ end
57
+ restore_counts(span, drops)
58
+ end
59
+
60
+ def invoke(customizer, view)
61
+ customizer.on_span_export(view)
62
+ # Rescue broadly: hook errors must fail the batch, not kill the export thread.
63
+ rescue StandardError, ScriptError, SystemStackError => e
64
+ raise Error, "#{label(customizer)} raised #{e.class}"
65
+ end
66
+
67
+ def unwrap(result, customizer)
68
+ span = result.is_a?(SpanExportData) ? result.to_span_data : result
69
+ return span if span.is_a?(OpenTelemetry::SDK::Trace::SpanData)
70
+
71
+ raise Error, "#{label(customizer)} returned #{result.class}; hooks must return the span or a replacement SpanData"
72
+ end
73
+
74
+ def label(customizer)
75
+ customizer.class.name || customizer.class.inspect
76
+ end
77
+
78
+ def identity(span)
79
+ ID_FIELDS.map { |field| span.public_send(field).dup }
80
+ end
81
+
82
+ # OTel and replacement spans may carry frozen attributes. Only the hash
83
+ # needs to be writable; values and other nested objects remain shared.
84
+ def writable_view(span)
85
+ span.attributes = span.attributes&.dup || {}
86
+ SpanExportData.new(span)
87
+ end
88
+
89
+ # Preserve drops caused by SDK limits; entries a customizer deletes are not drops.
90
+ def dropped_counts(span)
91
+ {
92
+ attributes: dropped(span.total_recorded_attributes, span.attributes),
93
+ events: dropped(span.total_recorded_events, span.events),
94
+ links: dropped(span.total_recorded_links, span.links)
95
+ }
96
+ end
97
+
98
+ def restore_counts(span, drops)
99
+ span.total_recorded_attributes = span.attributes&.size.to_i + drops[:attributes]
100
+ span.total_recorded_events = span.events&.size.to_i + drops[:events]
101
+ span.total_recorded_links = span.links&.size.to_i + drops[:links]
102
+ span
103
+ end
104
+
105
+ def dropped(total, entries)
106
+ [total.to_i - entries&.size.to_i, 0].max
107
+ end
108
+ end
109
+ end
110
+ end
@@ -0,0 +1,35 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "forwardable"
4
+ require "opentelemetry/sdk"
5
+
6
+ module Braintrust
7
+ module Trace
8
+ # Mutable view of completed span data with read-only identity fields.
9
+ # Attribute values and other nested objects are shared, not deep-copied.
10
+ class SpanExportData
11
+ extend Forwardable
12
+
13
+ ID_FIELDS = [:trace_id, :span_id, :parent_span_id].freeze
14
+ MUTABLE_FIELDS = (OpenTelemetry::SDK::Trace::SpanData.members - ID_FIELDS).freeze
15
+ private_constant :ID_FIELDS, :MUTABLE_FIELDS
16
+
17
+ attr_reader(*ID_FIELDS)
18
+ def_delegators :@span_data, *MUTABLE_FIELDS, *MUTABLE_FIELDS.map { |field| :"#{field}=" }
19
+ def_delegators :@span_data, :hex_trace_id, :hex_span_id, :hex_parent_span_id, :instrumentation_library
20
+
21
+ def initialize(span_data)
22
+ @span_data = span_data
23
+ @trace_id = span_data.trace_id.dup.freeze
24
+ @span_id = span_data.span_id.dup.freeze
25
+ @parent_span_id = span_data.parent_span_id.dup.freeze
26
+ end
27
+
28
+ # Access the underlying OpenTelemetry representation, e.g. to build a replacement.
29
+ # Replacement IDs are validated by the exporter after each hook.
30
+ def to_span_data
31
+ @span_data
32
+ end
33
+ end
34
+ end
35
+ end
@@ -3,17 +3,21 @@
3
3
  require "opentelemetry/exporter/otlp"
4
4
  require_relative "../state"
5
5
  require_relative "span_origin"
6
+ require_relative "span_customization"
6
7
 
7
8
  module Braintrust
8
9
  module Trace
9
10
  # Custom OTLP exporter for the Braintrust backend. On export it:
10
11
  # - stamps span origin provenance onto each SpanData (via the prepended SpanOrigin behavior)
12
+ # - runs optional customizers (via the prepended SpanCustomization behavior)
11
13
  # - groups spans by braintrust.parent and sets the x-bt-parent header per group,
12
14
  # so the backend routes them to the correct experiment/project
13
15
  #
14
16
  # Thread safety: BatchSpanProcessor serializes export() calls via its
15
17
  # @export_mutex, so @headers mutation here is safe.
16
18
  class SpanExporter < OpenTelemetry::Exporter::OTLP::Exporter
19
+ # The last prepended runs first: origin, then customization, then export below.
20
+ prepend SpanCustomization
17
21
  prepend SpanOrigin
18
22
 
19
23
  PARENT_ATTR_KEY = SpanProcessor::PARENT_ATTR_KEY
@@ -24,11 +24,15 @@ module Braintrust
24
24
  # @param span_data [Array<OpenTelemetry::SDK::Trace::SpanData>]
25
25
  # @return [Integer] export result from the wrapped exporter
26
26
  def export(span_data, timeout: nil)
27
+ super(SpanOrigin.enrich_batch(span_data), timeout: timeout)
28
+ end
29
+
30
+ # Enrich a batch of SpanData with span origin provenance.
31
+ def self.enrich_batch(span_data)
27
32
  # Environment is process-global and stable; read it once per batch
28
33
  # rather than once per span. It is cheap (ENV reads only).
29
34
  environment = Internal::Env.detect_environment
30
- enriched = span_data.map { |sd| SpanOrigin.enrich(sd, environment: environment) }
31
- super(enriched, timeout: timeout)
35
+ span_data.map { |sd| enrich(sd, environment: environment) }
32
36
  end
33
37
 
34
38
  # Enrich a single SpanData with span origin provenance.
@@ -1,6 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require "opentelemetry/sdk"
4
+ require "uri"
4
5
  require "opentelemetry/exporter/otlp"
5
6
  require_relative "trace/span_processor"
6
7
  require_relative "trace/span_exporter"
@@ -88,10 +89,16 @@ module Braintrust
88
89
  # Get config from state if available
89
90
  config ||= state.respond_to?(:config) ? state.config : nil
90
91
 
92
+ # Customizers only run in SpanExporter; refuse to silently skip them (e.g. redaction).
93
+ if exporter && config&.span_customizers&.any?
94
+ raise ArgumentError, "span_customizers are not supported with a custom exporter"
95
+ end
96
+
91
97
  # Create OTLP HTTP exporter unless override provided
92
98
  exporter ||= SpanExporter.new(
93
99
  endpoint: "#{state.api_url}/otel/v1/traces",
94
- api_key: state.api_key
100
+ api_key: state.api_key,
101
+ span_customizers: config&.span_customizers
95
102
  )
96
103
 
97
104
  # Use SimpleSpanProcessor for InMemorySpanExporter (testing), BatchSpanProcessor for production
@@ -185,14 +192,18 @@ module Braintrust
185
192
  return ""
186
193
  end
187
194
 
195
+ # Encode each path segment so names with spaces or slashes remain one segment.
196
+ escaped_org_name = URI::DEFAULT_PARSER.escape(org_name, /[^A-Za-z0-9\-._~]/)
197
+ escaped_parent_id = URI::DEFAULT_PARSER.escape(parent_id, /[^A-Za-z0-9\-._~]/)
198
+
188
199
  # Build the permalink URL based on parent type
189
200
  if parent_type == "experiment_id"
190
201
  # For experiments: {app_url}/app/{org}/object?object_type=experiment&object_id={experiment_id}&r={trace_id}&s={span_id}
191
- "#{app_url}/app/#{org_name}/object?object_type=experiment&object_id=#{parent_id}&r=#{trace_id}&s=#{span_id}"
202
+ "#{app_url}/app/#{escaped_org_name}/object?object_type=experiment&object_id=#{parent_id}&r=#{trace_id}&s=#{span_id}"
192
203
  else
193
204
  # For projects: {app_url}/app/{org}/p/{project}/logs?r={trace_id}&s={span_id}
194
205
  # parent_type is typically "project_name"
195
- "#{app_url}/app/#{org_name}/p/#{parent_id}/logs?r=#{trace_id}&s=#{span_id}"
206
+ "#{app_url}/app/#{escaped_org_name}/p/#{escaped_parent_id}/logs?r=#{trace_id}&s=#{span_id}"
196
207
  end
197
208
  rescue => e
198
209
  Log.error("Failed to generate permalink: #{e.message}")
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Braintrust
4
- VERSION = "0.5.0"
4
+ VERSION = "0.6.1"
5
5
  end
data/lib/braintrust.rb CHANGED
@@ -42,6 +42,7 @@ module Braintrust
42
42
  # @param tracer_provider [TracerProvider, nil] Optional tracer provider to use instead of creating one
43
43
  # @param filter_ai_spans [Boolean, nil] Enable AI span filtering (overrides BRAINTRUST_OTEL_FILTER_AI_SPANS env var)
44
44
  # @param span_filter_funcs [Array<Proc>, nil] Custom span filter functions
45
+ # @param span_customizers [Array<#on_span_export>, nil] Ordered synchronous export customizers
45
46
  # @param exporter [Exporter, nil] Optional exporter override (for testing)
46
47
  # @param auto_instrument [Boolean, Hash, nil] Auto-instrumentation config:
47
48
  # - nil (default): use BRAINTRUST_AUTO_INSTRUMENT env var, default true if not set
@@ -49,7 +50,7 @@ module Braintrust
49
50
  # - false: explicitly disable
50
51
  # - Hash with :only or :except keys for filtering
51
52
  # @return [State] the created state
52
- def self.init(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil, set_global: true, blocking_login: false, enable_tracing: true, tracer_provider: nil, filter_ai_spans: nil, span_filter_funcs: nil, exporter: nil, auto_instrument: nil)
53
+ def self.init(api_key: nil, org_name: nil, default_project: nil, app_url: nil, api_url: nil, set_global: true, blocking_login: false, enable_tracing: true, tracer_provider: nil, filter_ai_spans: nil, span_filter_funcs: nil, span_customizers: nil, exporter: nil, auto_instrument: nil)
53
54
  state = State.from_env(
54
55
  api_key: api_key,
55
56
  org_name: org_name,
@@ -61,6 +62,7 @@ module Braintrust
61
62
  tracer_provider: tracer_provider,
62
63
  filter_ai_spans: filter_ai_spans,
63
64
  span_filter_funcs: span_filter_funcs,
65
+ span_customizers: span_customizers,
64
66
  exporter: exporter
65
67
  )
66
68
 
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: braintrust
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.0
4
+ version: 0.6.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Braintrust
@@ -120,6 +120,11 @@ files:
120
120
  - lib/braintrust/contrib/rails/server/routes.rb
121
121
  - lib/braintrust/contrib/rails/server/templates/initializer.rb.tt
122
122
  - lib/braintrust/contrib/registry.rb
123
+ - lib/braintrust/contrib/roast/instrumentation/cog.rb
124
+ - lib/braintrust/contrib/roast/instrumentation/workflow.rb
125
+ - lib/braintrust/contrib/roast/integration.rb
126
+ - lib/braintrust/contrib/roast/patcher.rb
127
+ - lib/braintrust/contrib/roast/tracing.rb
123
128
  - lib/braintrust/contrib/ruby_llm/deprecated.rb
124
129
  - lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb
125
130
  - lib/braintrust/contrib/ruby_llm/instrumentation/common.rb
@@ -181,6 +186,9 @@ files:
181
186
  - lib/braintrust/task.rb
182
187
  - lib/braintrust/trace.rb
183
188
  - lib/braintrust/trace/attachment.rb
189
+ - lib/braintrust/trace/span_customization.rb
190
+ - lib/braintrust/trace/span_customizers.rb
191
+ - lib/braintrust/trace/span_export_data.rb
184
192
  - lib/braintrust/trace/span_exporter.rb
185
193
  - lib/braintrust/trace/span_filter.rb
186
194
  - lib/braintrust/trace/span_origin.rb