braintrust 0.0.12 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +214 -180
  3. data/exe/braintrust +143 -0
  4. data/lib/braintrust/contrib/anthropic/deprecated.rb +24 -0
  5. data/lib/braintrust/contrib/anthropic/instrumentation/beta_messages.rb +242 -0
  6. data/lib/braintrust/contrib/anthropic/instrumentation/common.rb +53 -0
  7. data/lib/braintrust/contrib/anthropic/instrumentation/messages.rb +232 -0
  8. data/lib/braintrust/contrib/anthropic/integration.rb +53 -0
  9. data/lib/braintrust/contrib/anthropic/patcher.rb +145 -0
  10. data/lib/braintrust/contrib/context.rb +56 -0
  11. data/lib/braintrust/contrib/integration.rb +160 -0
  12. data/lib/braintrust/contrib/openai/deprecated.rb +22 -0
  13. data/lib/braintrust/contrib/openai/instrumentation/chat.rb +298 -0
  14. data/lib/braintrust/contrib/openai/instrumentation/common.rb +134 -0
  15. data/lib/braintrust/contrib/openai/instrumentation/moderations.rb +93 -0
  16. data/lib/braintrust/contrib/openai/instrumentation/responses.rb +187 -0
  17. data/lib/braintrust/contrib/openai/integration.rb +58 -0
  18. data/lib/braintrust/contrib/openai/patcher.rb +173 -0
  19. data/lib/braintrust/contrib/patcher.rb +76 -0
  20. data/lib/braintrust/contrib/rails/railtie.rb +16 -0
  21. data/lib/braintrust/contrib/registry.rb +107 -0
  22. data/lib/braintrust/contrib/ruby_llm/deprecated.rb +45 -0
  23. data/lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb +464 -0
  24. data/lib/braintrust/contrib/ruby_llm/instrumentation/common.rb +58 -0
  25. data/lib/braintrust/contrib/ruby_llm/integration.rb +54 -0
  26. data/lib/braintrust/contrib/ruby_llm/patcher.rb +44 -0
  27. data/lib/braintrust/contrib/ruby_openai/deprecated.rb +24 -0
  28. data/lib/braintrust/contrib/ruby_openai/instrumentation/chat.rb +149 -0
  29. data/lib/braintrust/contrib/ruby_openai/instrumentation/common.rb +138 -0
  30. data/lib/braintrust/contrib/ruby_openai/instrumentation/moderations.rb +94 -0
  31. data/lib/braintrust/contrib/ruby_openai/instrumentation/responses.rb +146 -0
  32. data/lib/braintrust/contrib/ruby_openai/integration.rb +58 -0
  33. data/lib/braintrust/contrib/ruby_openai/patcher.rb +120 -0
  34. data/lib/braintrust/contrib/setup.rb +168 -0
  35. data/lib/braintrust/contrib/support/openai.rb +72 -0
  36. data/lib/braintrust/contrib/support/otel.rb +23 -0
  37. data/lib/braintrust/contrib.rb +205 -0
  38. data/lib/braintrust/internal/env.rb +39 -0
  39. data/lib/braintrust/internal/time.rb +44 -0
  40. data/lib/braintrust/setup.rb +50 -0
  41. data/lib/braintrust/state.rb +6 -1
  42. data/lib/braintrust/trace.rb +41 -51
  43. data/lib/braintrust/version.rb +1 -1
  44. data/lib/braintrust.rb +10 -1
  45. metadata +41 -7
  46. data/lib/braintrust/trace/contrib/anthropic.rb +0 -316
  47. data/lib/braintrust/trace/contrib/github.com/alexrudall/ruby-openai/ruby-openai.rb +0 -377
  48. data/lib/braintrust/trace/contrib/github.com/crmne/ruby_llm.rb +0 -631
  49. data/lib/braintrust/trace/contrib/openai.rb +0 -611
  50. data/lib/braintrust/trace/tokens.rb +0 -109
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4a4dd86789a3ce80b891b88fb9b125bb7b6d85e86928adeb5fd7c1fec671ff56
4
- data.tar.gz: 171fe031f960d0a6f3abbfacbb20094b35bcd9199cf132e30063868807f95f8b
3
+ metadata.gz: 40d5c41ef999d495f9eb7b3a4cd41f1bb9ee7ba831de2074195a7d9e18da2e3a
4
+ data.tar.gz: 8e48aee25cd02a4b936da1264a9964097c6e07744faf0564d81885daa72b87da
5
5
  SHA512:
6
- metadata.gz: 3cf52e457ca5d4cdad2f8873bb45d0eeb0b79b33c5edbcd467e65ce7261caea9bcc86ea9d562835197279105f14f2787bcac6622faad4ec420a5f2fa27a003d2
7
- data.tar.gz: 5a7ccc20a3e63840e15c41dd82eaa8653a2f0dce693321e6527ace59c54255524ae1d794d2dce058127b747a514e8777729384cde9c157374039880bb3b01013
6
+ metadata.gz: 9391f2dcec3c92e032d3e7009ec4bf1d26f6d2a606f0edad78acf78078f116393d95a4a3d70a08010b89df214f92498905c67cd02b3bd6327b5aa2cbf1dda872
7
+ data.tar.gz: 5665f9bcb49a8b5ca90f5d58d9c279120a053756c6ec03ba17557b6064c7a7142ac9674789fe4c89df70e2ac61d1d1ce9318a33b897b55da1d45b9bc9df1bf15
data/README.md CHANGED
@@ -4,282 +4,316 @@
4
4
  [![Documentation](https://img.shields.io/badge/docs-gemdocs.org-blue.svg)](https://gemdocs.org/gems/braintrust/)
5
5
  ![Beta](https://img.shields.io/badge/status-beta-yellow)
6
6
 
7
- ## Overview
7
+ This is the official Ruby SDK for [Braintrust](https://www.braintrust.dev), for tracing and evaluating your AI applications.
8
+
9
+ *NOTE: This SDK is currently in BETA status and APIs may change between minor versions.*
10
+
11
+ - [Quick Start](#quick-start)
12
+ - [Installation](#installation)
13
+ - [Setup script](#setup-script)
14
+ - [CLI Command](#cli-command)
15
+ - [Braintrust.init](#braintrustinit)
16
+ - [Environment variables](#environment-variables)
17
+ - [Tracing](#tracing)
18
+ - [Supported providers](#supported-providers)
19
+ - [Manually applying instrumentation](#manually-applying-instrumentation)
20
+ - [Creating custom spans](#creating-custom-spans)
21
+ - [Attachments](#attachments)
22
+ - [Viewing traces](#viewing-traces)
23
+ - [Evals](#evals)
24
+ - [Datasets](#datasets)
25
+ - [Remote scorers](#remote-scorers)
26
+ - [Documentation](#documentation)
27
+ - [Troubleshooting](#troubleshooting)
28
+ - [Contributing](#contributing)
29
+ - [License](#license)
8
30
 
9
- This library provides tools for **evaluating** and **tracing** AI applications in [Braintrust](https://www.braintrust.dev). Use it to:
10
-
11
- - **Evaluate** your AI models with custom test cases and scoring functions
12
- - **Trace** LLM calls and monitor AI application performance with OpenTelemetry
13
- - **Integrate** seamlessly with OpenAI and other LLM providers
14
-
15
- This SDK is currently in BETA status and APIs may change.
16
-
17
- ## Installation
31
+ ## Quick Start
18
32
 
19
- Add this line to your application's Gemfile:
33
+ Add to your Gemfile:
20
34
 
21
35
  ```ruby
22
- gem 'braintrust'
36
+ gem "braintrust", require: "braintrust/setup"
23
37
  ```
24
38
 
25
- And then execute:
39
+ Set your API key and install:
26
40
 
27
41
  ```bash
42
+ export BRAINTRUST_API_KEY="your-api-key"
28
43
  bundle install
29
44
  ```
30
45
 
31
- Or install it yourself as:
46
+ Your LLM calls are now automatically traced. View them at [braintrust.dev](https://www.braintrust.dev).
32
47
 
33
- ```bash
34
- gem install braintrust
35
- ```
48
+ ## Installation
36
49
 
37
- ## Quick Start
50
+ The SDK also offers additional setup options for a variety of applications.
38
51
 
39
- ### Set up your API key
52
+ | | What it looks like | When to Use |
53
+ | -------------------------------------- | ----------------------------------- | --------------------------------------------------------------------------- |
54
+ | [**Setup script**](#setup-script) | `require 'braintrust/setup'` | You'd like to automatically setup the SDK at load time. |
55
+ | [**CLI command**](#cli-command) | `braintrust exec -- ruby app.rb` | You prefer not to modify the application source code. |
56
+ | [**Braintrust.init**](#braintrustinit) | Call `Braintrust.init` in your code | You need to control when setup occurs, or require customized configuration. |
40
57
 
41
- ```bash
42
- export BRAINTRUST_API_KEY="your-api-key"
43
- ```
58
+ See [our examples](./examples/setup/README.md) for more detail.
44
59
 
45
- ### Evals
60
+ ### Setup script
46
61
 
47
- ```ruby
48
- require "braintrust"
62
+ For most applications, we recommend adding `require: "braintrust/setup"` to your `Gemfile` or an initializer file in your Ruby application to automatically setup the SDK. This will automatically apply instrumentation to all available LLM libraries.
49
63
 
50
- Braintrust.init
64
+ You can use [environment variables](#environment-variables) to configure behavior.
51
65
 
52
- # Define task to evaluate
53
- task = ->(input) { input.include?("a") ? "fruit" : "vegetable" }
66
+ ### CLI Command
54
67
 
55
- # Run evaluation
56
- Braintrust::Eval.run(
57
- project: "my-project",
58
- experiment: "food-classifier",
59
- cases: [
60
- {input: "apple", expected: "fruit"},
61
- {input: "carrot", expected: "vegetable"}
62
- ],
63
- task: task,
64
- scorers: [
65
- ->(input, expected, output) { output == expected ? 1.0 : 0.0 }
66
- ]
67
- )
68
+ You can use this CLI command to instrument any Ruby application without modifying the source code.
69
+
70
+ First, make sure the gem is installed on the system:
71
+
72
+ ```bash
73
+ gem install braintrust
68
74
  ```
69
75
 
70
- ### Tracing
76
+ Then wrap the start up command of any Ruby application to apply:
71
77
 
72
- ```ruby
73
- require "braintrust"
74
- require "opentelemetry/sdk"
78
+ ```bash
79
+ braintrust exec -- ruby app.rb
80
+ braintrust exec -- bundle exec rails server
81
+ braintrust exec --only openai -- ruby app.rb
82
+ ```
75
83
 
76
- # Initialize Braintrust
77
- Braintrust.init
84
+ You can use [environment variables](#environment-variables) to configure behavior.
78
85
 
79
- # Get a tracer
80
- tracer = OpenTelemetry.tracer_provider.tracer("my-app")
86
+ ---
81
87
 
82
- # Create spans to track operations
83
- tracer.in_span("process-data") do |span|
84
- span.set_attribute("user.id", "123")
85
- span.set_attribute("operation.type", "data_processing")
88
+ *NOTE: Installing a package at the system-level does not guarantee compatibility with all Ruby applications on that system; conflicts with dependencies can arise.*
86
89
 
87
- # Your code here
88
- puts "Processing data..."
89
- sleep 0.1
90
+ For stronger assurance of compatibility, we recommend either:
90
91
 
91
- # Nested spans are automatically linked
92
- tracer.in_span("nested-operation") do |nested_span|
93
- nested_span.set_attribute("step", "1")
94
- puts "Nested operation..."
95
- end
96
- end
92
+ - Installing via the application's `Gemfile` and `bundle install` when possible.
93
+ - **OR** for OCI/Docker deployments, `gem install braintrust` when building images in your CI/CD pipeline (and verifying their safe function.)
97
94
 
98
- # Shutdown to flush spans
99
- OpenTelemetry.tracer_provider.shutdown
95
+ ---
100
96
 
101
- puts "View trace in Braintrust!"
102
- ```
97
+ ### Braintrust.init
103
98
 
104
- ### OpenAI Tracing
99
+ For more control over when auto-instrumentation is applied:
105
100
 
106
101
  ```ruby
107
102
  require "braintrust"
108
- require "openai"
109
103
 
110
104
  Braintrust.init
105
+ ```
111
106
 
112
- client = OpenAI::Client.new(api_key: ENV["OPENAI_API_KEY"])
107
+ **Options:**
113
108
 
114
- Braintrust::Trace::OpenAI.wrap(client)
109
+ | Option | Default | Description |
110
+ | ----------------- | ---------------------------------------- | --------------------------------------------------------------------------- |
111
+ | `api_key` | `ENV['BRAINTRUST_API_KEY']` | API key |
112
+ | `auto_instrument` | `true` | `true`, `false`, or Hash with `:only`/`:except` keys to filter integrations |
113
+ | `blocking_login` | `false` | Block until login completes (async login when `false`) |
114
+ | `default_project` | `ENV['BRAINTRUST_DEFAULT_PROJECT']` | Default project for spans |
115
+ | `enable_tracing` | `true` | Enable OpenTelemetry tracing |
116
+ | `filter_ai_spans` | `ENV['BRAINTRUST_OTEL_FILTER_AI_SPANS']` | Only export AI-related spans |
117
+ | `org_name` | `ENV['BRAINTRUST_ORG_NAME']` | Organization name |
118
+ | `set_global` | `true` | Set as global state. Set to `false` for isolated instances |
115
119
 
116
- tracer = OpenTelemetry.tracer_provider.tracer("openai-app")
117
- root_span = nil
120
+ **Example with options:**
118
121
 
119
- response = tracer.in_span("chat-completion") do |span|
120
- root_span = span
122
+ ```ruby
123
+ Braintrust.init(
124
+ default_project: "my-project",
125
+ auto_instrument: { only: [:openai] }
126
+ )
127
+ ```
121
128
 
122
- client.chat.completions.create(
123
- messages: [
124
- {role: "system", content: "You are a helpful assistant."},
125
- {role: "user", content: "Say hello!"}
126
- ],
127
- model: "gpt-4o-mini",
128
- max_tokens: 100
129
- )
130
- end
129
+ ### Environment variables
131
130
 
132
- puts "Response: #{response.choices[0].message.content}"
131
+ | Variable | Description |
132
+ | --------------------------------- | ------------------------------------------------------------------------- |
133
+ | `BRAINTRUST_API_KEY` | Required. Your Braintrust API key |
134
+ | `BRAINTRUST_API_URL` | Braintrust API URL (default: `https://api.braintrust.dev`) |
135
+ | `BRAINTRUST_APP_URL` | Braintrust app URL (default: `https://www.braintrust.dev`) |
136
+ | `BRAINTRUST_AUTO_INSTRUMENT` | Set to `false` to disable auto-instrumentation |
137
+ | `BRAINTRUST_DEBUG` | Set to `true` to enable debug logging |
138
+ | `BRAINTRUST_DEFAULT_PROJECT` | Default project for spans |
139
+ | `BRAINTRUST_FLUSH_ON_EXIT` | Set to `false` to disable automatic span flushing on program exit |
140
+ | `BRAINTRUST_INSTRUMENT_EXCEPT` | Comma-separated list of integrations to skip |
141
+ | `BRAINTRUST_INSTRUMENT_ONLY` | Comma-separated list of integrations to enable (e.g., `openai,anthropic`) |
142
+ | `BRAINTRUST_ORG_NAME` | Organization name |
143
+ | `BRAINTRUST_OTEL_FILTER_AI_SPANS` | Set to `true` to only export AI-related spans |
133
144
 
134
- puts "View trace at: #{Braintrust::Trace.permalink(root_span)}"
145
+ ## Tracing
135
146
 
136
- OpenTelemetry.tracer_provider.shutdown
137
- ```
147
+ ### Supported providers
148
+
149
+ The SDK automatically instruments these LLM libraries:
138
150
 
139
- ### Anthropic Tracing
151
+ | Provider | Gem | Versions | Integration Name | Examples |
152
+ | --------- | ------------- | -------- | ---------------- | ----------------------------------------------- |
153
+ | Anthropic | `anthropic` | >= 0.3.0 | `:anthropic` | [Link](./examples/contrib/anthropic.rb) |
154
+ | OpenAI | `openai` | >= 0.1.0 | `:openai` | [Link](./examples/contrib/openai.rb) |
155
+ | | `ruby-openai` | >= 7.0.0 | `:ruby_openai` | [Link](./examples/contrib/ruby-openai.rb) |
156
+ | Multiple | `ruby_llm` | >= 1.8.0 | `:ruby_llm` | [Link](./examples/contrib/ruby_llm.rb) |
157
+
158
+ ### Manually applying instrumentation
159
+
160
+ For fine-grained control, disable auto-instrumentation and instrument specific clients:
140
161
 
141
162
  ```ruby
142
163
  require "braintrust"
143
- require "anthropic"
144
-
145
- Braintrust.init
164
+ require "openai"
146
165
 
147
- client = Anthropic::Client.new(api_key: ENV["ANTHROPIC_API_KEY"])
166
+ Braintrust.init(auto_instrument: false) # Or BRAINTRUST_AUTO_INSTRUMENT=false
148
167
 
149
- Braintrust::Trace::Anthropic.wrap(client)
168
+ # Instrument all OpenAI clients
169
+ Braintrust.instrument!(:openai)
150
170
 
151
- tracer = OpenTelemetry.tracer_provider.tracer("anthropic-app")
152
- root_span = nil
171
+ # OR instrument a single client
172
+ client = OpenAI::Client.new
173
+ Braintrust.instrument!(:openai, target: client)
174
+ ```
153
175
 
154
- message = tracer.in_span("chat-message") do |span|
155
- root_span = span
176
+ ### Creating custom spans
156
177
 
157
- client.messages.create(
158
- model: "claude-3-haiku-20240307",
159
- max_tokens: 100,
160
- system: "You are a helpful assistant.",
161
- messages: [
162
- {role: "user", content: "Say hello!"}
163
- ]
164
- )
165
- end
178
+ Wrap business logic in spans to see it in your traces:
166
179
 
167
- puts "Response: #{message.content[0].text}"
180
+ ```ruby
181
+ tracer = OpenTelemetry.tracer_provider.tracer("my-app")
168
182
 
169
- puts "View trace at: #{Braintrust::Trace.permalink(root_span)}"
183
+ tracer.in_span("process-request") do |span|
184
+ span.set_attribute("user.id", user_id)
170
185
 
171
- OpenTelemetry.tracer_provider.shutdown
186
+ # LLM calls inside here are automatically nested under this span
187
+ response = client.chat.completions.create(...)
188
+ end
172
189
  ```
173
190
 
174
- ### RubyLLM Tracing
191
+ ### Attachments
192
+
193
+ Log binary data (images, PDFs, audio) in your traces:
175
194
 
176
195
  ```ruby
177
- require "braintrust"
178
- require "ruby_llm"
196
+ require "braintrust/trace/attachment"
179
197
 
180
- Braintrust.init
198
+ att = Braintrust::Trace::Attachment.from_file("image/png", "./photo.png")
181
199
 
182
- # Wrap RubyLLM globally (wraps all Chat instances)
183
- Braintrust::Trace::Contrib::Github::Crmne::RubyLLM.wrap
200
+ # Use in messages (OpenAI/Anthropic format)
201
+ messages = [
202
+ {
203
+ role: "user",
204
+ content: [
205
+ {type: "text", text: "What's in this image?"},
206
+ att.to_h
207
+ ]
208
+ }
209
+ ]
184
210
 
185
- tracer = OpenTelemetry.tracer_provider.tracer("ruby-llm-app")
186
- root_span = nil
211
+ # Log to span
212
+ span.set_attribute("braintrust.input_json", JSON.generate(messages))
213
+ ```
187
214
 
188
- response = tracer.in_span("chat") do |span|
189
- root_span = span
215
+ Create attachments from various sources:
190
216
 
191
- chat = RubyLLM.chat(model: "gpt-4o-mini")
192
- chat.ask("Say hello!")
193
- end
217
+ ```ruby
218
+ Braintrust::Trace::Attachment.from_bytes("image/jpeg", image_data)
219
+ Braintrust::Trace::Attachment.from_file("application/pdf", "./doc.pdf")
220
+ Braintrust::Trace::Attachment.from_url("https://example.com/image.png")
221
+ ```
222
+
223
+ See example: [trace_attachments.rb](./examples/trace/trace_attachments.rb)
194
224
 
195
- puts "Response: #{response.content}"
225
+ ### Viewing traces
196
226
 
197
- puts "View trace at: #{Braintrust::Trace.permalink(root_span)}"
227
+ Get a permalink to any span:
198
228
 
199
- OpenTelemetry.tracer_provider.shutdown
229
+ ```ruby
230
+ tracer = OpenTelemetry.tracer_provider.tracer("my-app")
231
+
232
+ tracer.in_span("my-operation") do |span|
233
+ # your code here
234
+ puts "View trace at: #{Braintrust::Trace.permalink(span)}"
235
+ end
200
236
  ```
201
237
 
202
- ### Attachments
238
+ ## Evals
203
239
 
204
- Attachments allow you to log binary data (images, PDFs, audio, etc.) as part of your traces. This is particularly useful for multimodal AI applications like vision models.
240
+ Run evaluations against your AI systems:
205
241
 
206
242
  ```ruby
207
243
  require "braintrust"
208
- require "braintrust/trace/attachment"
209
244
 
210
245
  Braintrust.init
211
246
 
212
- tracer = OpenTelemetry.tracer_provider.tracer("vision-app")
213
-
214
- tracer.in_span("analyze-image") do |span|
215
- # Create attachment from file
216
- att = Braintrust::Trace::Attachment.from_file(
217
- Braintrust::Trace::Attachment::IMAGE_PNG,
218
- "./photo.png"
219
- )
220
-
221
- # Build message with attachment (OpenAI/Anthropic format)
222
- messages = [
223
- {
224
- role: "user",
225
- content: [
226
- {type: "text", text: "What's in this image?"},
227
- att.to_h # Converts to {"type" => "base64_attachment", "content" => "data:..."}
228
- ]
229
- }
247
+ Braintrust::Eval.run(
248
+ project: "my-project",
249
+ experiment: "classifier-v1",
250
+ cases: [
251
+ {input: "apple", expected: "fruit"},
252
+ {input: "carrot", expected: "vegetable"}
253
+ ],
254
+ task: ->(input) { classify(input) },
255
+ scorers: [
256
+ ->(input, expected, output) { output == expected ? 1.0 : 0.0 }
230
257
  ]
258
+ )
259
+ ```
231
260
 
232
- # Log to trace
233
- span.set_attribute("braintrust.input_json", JSON.generate(messages))
234
- end
261
+ ### Datasets
262
+
263
+ Load test cases from a Braintrust dataset:
235
264
 
236
- OpenTelemetry.tracer_provider.shutdown
265
+ ```ruby
266
+ Braintrust::Eval.run(
267
+ project: "my-project",
268
+ dataset: "my-dataset",
269
+ task: ->(input) { classify(input) },
270
+ scorers: [...]
271
+ )
237
272
  ```
238
273
 
239
- You can create attachments from bytes, files, or URLs:
274
+ ### Remote scorers
275
+
276
+ Use scoring functions defined in Braintrust:
240
277
 
241
278
  ```ruby
242
- # From bytes
243
- att = Braintrust::Trace::Attachment.from_bytes("image/jpeg", image_data)
279
+ Braintrust::Eval.run(
280
+ project: "my-project",
281
+ cases: [...],
282
+ task: ->(input) { ... },
283
+ scorers: [
284
+ Braintrust::Scorer.remote("my-project", "accuracy-scorer")
285
+ ]
286
+ )
287
+ ```
244
288
 
245
- # From file
246
- att = Braintrust::Trace::Attachment.from_file("application/pdf", "./doc.pdf")
289
+ See examples: [eval.rb](./examples/eval.rb), [dataset.rb](./examples/eval/dataset.rb), [remote_functions.rb](./examples/eval/remote_functions.rb)
247
290
 
248
- # From URL
249
- att = Braintrust::Trace::Attachment.from_url("https://example.com/image.png")
250
- ```
291
+ ## Documentation
251
292
 
252
- ## Features
293
+ - [Braintrust Documentation](https://www.braintrust.dev/docs)
294
+ - [API Reference](https://gemdocs.org/gems/braintrust/)
253
295
 
254
- - **Evaluations**: Run systematic evaluations of your AI systems with custom scoring functions
255
- - **Tracing**: Automatic instrumentation for OpenAI and Anthropic API calls with OpenTelemetry
256
- - **Datasets**: Manage and version your evaluation datasets
257
- - **Experiments**: Track different versions and configurations of your AI systems
258
- - **Observability**: Monitor your AI applications in production
296
+ ## Troubleshooting
259
297
 
260
- ## Examples
298
+ #### No traces after adding `require 'braintrust/setup'` to the Gemfile
261
299
 
262
- Check out the [`examples/`](./examples/) directory for complete working examples:
300
+ First verify there are no errors in your logs after running with `BRAINTRUST_DEBUG=true` set.
263
301
 
264
- - [eval.rb](./examples/eval.rb) - Create and run evaluations with custom test cases and scoring functions
265
- - [trace.rb](./examples/trace.rb) - Manual span creation and tracing
266
- - [openai.rb](./examples/openai.rb) - Automatically trace OpenAI API calls
267
- - [alexrudall_openai.rb](./examples/alexrudall_openai.rb) - Automatically trace ruby-openai gem API calls
268
- - [anthropic.rb](./examples/anthropic.rb) - Automatically trace Anthropic API calls
269
- - [ruby_llm.rb](./examples/ruby_llm.rb) - Automatically trace RubyLLM API calls
270
- - [trace/trace_attachments.rb](./examples/trace/trace_attachments.rb) - Log attachments (images, PDFs) in traces
271
- - [eval/dataset.rb](./examples/eval/dataset.rb) - Run evaluations using datasets stored in Braintrust
272
- - [eval/remote_functions.rb](./examples/eval/remote_functions.rb) - Use remote scoring functions
302
+ Your application needs the following for this to work:
273
303
 
274
- ## Documentation
304
+ ```ruby
305
+ require 'bundler/setup'
306
+ Bundler.require
307
+ ```
275
308
 
276
- - [Braintrust Documentation](https://www.braintrust.dev/docs)
277
- - [API Documentation](https://gemdocs.org/gems/braintrust/)
309
+ It is present by default in Rails applications, but may not be in Sinatra, Rack, or other applications.
310
+
311
+ Alternatively, you can add `require 'braintrust/setup'` to your application initialization files.
278
312
 
279
313
  ## Contributing
280
314
 
281
- See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and contribution guidelines.
315
+ See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and guidelines.
282
316
 
283
317
  ## License
284
318
 
285
- This project is licensed under the Apache License 2.0. See the [LICENSE](./LICENSE) file for details.
319
+ Apache License 2.0 - see [LICENSE](./LICENSE).
data/exe/braintrust ADDED
@@ -0,0 +1,143 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ # Braintrust CLI - Auto-instrument Ruby applications
5
+ #
6
+ # Usage:
7
+ # braintrust exec -- ruby app.rb
8
+ # braintrust exec -- bundle exec rails server
9
+ # braintrust exec --only openai,anthropic -- ruby app.rb
10
+
11
+ require "optparse"
12
+
13
+ module Braintrust
14
+ module CLI
15
+ class << self
16
+ def run(args)
17
+ command = parse_args(args)
18
+
19
+ case command
20
+ when :exec
21
+ exec_command
22
+ when :help, nil
23
+ print_help
24
+ exit((command == :help) ? 0 : 1)
25
+ else
26
+ puts "Unknown command: #{command}"
27
+ print_help
28
+ exit 1
29
+ end
30
+ end
31
+
32
+ private
33
+
34
+ def parse_args(args)
35
+ @options = {}
36
+ @remaining_args = []
37
+
38
+ # Find -- separator
39
+ separator_index = args.index("--")
40
+ if separator_index
41
+ to_parse = args[0...separator_index]
42
+ @remaining_args = args[(separator_index + 1)..]
43
+ else
44
+ to_parse = args.dup
45
+ end
46
+
47
+ parser = OptionParser.new do |opts|
48
+ opts.banner = "Usage: braintrust <command> [options] -- COMMAND"
49
+
50
+ opts.on("--only INTEGRATIONS", "Only instrument these (comma-separated)") do |v|
51
+ @options[:only] = v
52
+ end
53
+
54
+ opts.on("--except INTEGRATIONS", "Skip these integrations (comma-separated)") do |v|
55
+ @options[:except] = v
56
+ end
57
+
58
+ opts.on("-h", "--help", "Show this help") do
59
+ return :help
60
+ end
61
+
62
+ opts.on("-v", "--version", "Show version") do
63
+ require_relative "../lib/braintrust/version"
64
+ puts "braintrust #{Braintrust::VERSION}"
65
+ exit 0
66
+ end
67
+ end
68
+
69
+ parser.parse!(to_parse)
70
+
71
+ return nil if to_parse.empty?
72
+ to_parse.first.to_sym
73
+ end
74
+
75
+ def exec_command
76
+ if @remaining_args.empty?
77
+ puts "Error: No command specified after --"
78
+ puts "Usage: braintrust exec [options] -- COMMAND"
79
+ exit 1
80
+ end
81
+
82
+ # Set environment variables for filtering
83
+ ENV["BRAINTRUST_INSTRUMENT_ONLY"] = @options[:only] if @options[:only]
84
+ ENV["BRAINTRUST_INSTRUMENT_EXCEPT"] = @options[:except] if @options[:except]
85
+
86
+ # Inject auto-instrument via RUBYOPT
87
+ set_rubyopt!
88
+
89
+ # Execute the command with error handling
90
+ exec_with_error_handling(@remaining_args)
91
+ end
92
+
93
+ def rubyopts
94
+ ["-rbraintrust/setup"]
95
+ end
96
+
97
+ def set_rubyopt!
98
+ existing = ENV["RUBYOPT"]
99
+ ENV["RUBYOPT"] = existing ? "#{existing} #{rubyopts.join(" ")}" : rubyopts.join(" ")
100
+ end
101
+
102
+ def exec_with_error_handling(args)
103
+ Kernel.exec(*args)
104
+ rescue Errno::ENOENT => e
105
+ Kernel.warn "braintrust exec failed: #{e.class.name} #{e.message}"
106
+ Kernel.exit 127
107
+ rescue Errno::EACCES, Errno::ENOEXEC => e
108
+ Kernel.warn "braintrust exec failed: #{e.class.name} #{e.message}"
109
+ Kernel.exit 126
110
+ end
111
+
112
+ def print_help
113
+ puts <<~HELP
114
+ Braintrust CLI - Auto-instrument Ruby applications
115
+
116
+ Usage:
117
+ braintrust exec [options] -- COMMAND
118
+
119
+ Commands:
120
+ exec Run a command with auto-instrumentation enabled
121
+
122
+ Options:
123
+ --only INTEGRATIONS Only instrument these (comma-separated)
124
+ --except INTEGRATIONS Skip these integrations (comma-separated)
125
+ -h, --help Show this help
126
+ -v, --version Show version
127
+
128
+ Examples:
129
+ braintrust exec -- ruby app.rb
130
+ braintrust exec -- rails server
131
+ braintrust exec --only openai -- ruby app.rb
132
+
133
+ Environment Variables:
134
+ BRAINTRUST_API_KEY API key for Braintrust
135
+ BRAINTRUST_INSTRUMENT_ONLY Comma-separated whitelist
136
+ BRAINTRUST_INSTRUMENT_EXCEPT Comma-separated blacklist
137
+ HELP
138
+ end
139
+ end
140
+ end
141
+ end
142
+
143
+ Braintrust::CLI.run(ARGV)
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Backward compatibility shim for the old Anthropic integration API.
4
+ # This file now just delegates to the new API.
5
+
6
+ require_relative "../../../braintrust"
7
+
8
+ module Braintrust
9
+ module Trace
10
+ module Anthropic
11
+ # Wrap an Anthropic::Client to automatically create spans for messages.
12
+ # This is the legacy API - delegates to the new contrib framework.
13
+ #
14
+ # @param client [Anthropic::Client] the Anthropic client to wrap
15
+ # @param tracer_provider [OpenTelemetry::SDK::Trace::TracerProvider] the tracer provider
16
+ # @return [Anthropic::Client] the wrapped client
17
+ def self.wrap(client, tracer_provider: nil)
18
+ Log.warn("Braintrust::Trace::Anthropic.wrap() is deprecated and will be removed in a future version: use Braintrust.instrument!() instead.")
19
+ Braintrust.instrument!(:anthropic, target: client, tracer_provider: tracer_provider)
20
+ client
21
+ end
22
+ end
23
+ end
24
+ end