braintrust 0.0.12 → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +213 -180
  3. data/exe/braintrust +143 -0
  4. data/lib/braintrust/contrib/anthropic/deprecated.rb +24 -0
  5. data/lib/braintrust/contrib/anthropic/instrumentation/common.rb +53 -0
  6. data/lib/braintrust/contrib/anthropic/instrumentation/messages.rb +232 -0
  7. data/lib/braintrust/contrib/anthropic/integration.rb +53 -0
  8. data/lib/braintrust/contrib/anthropic/patcher.rb +62 -0
  9. data/lib/braintrust/contrib/context.rb +56 -0
  10. data/lib/braintrust/contrib/integration.rb +160 -0
  11. data/lib/braintrust/contrib/openai/deprecated.rb +22 -0
  12. data/lib/braintrust/contrib/openai/instrumentation/chat.rb +298 -0
  13. data/lib/braintrust/contrib/openai/instrumentation/common.rb +134 -0
  14. data/lib/braintrust/contrib/openai/instrumentation/responses.rb +187 -0
  15. data/lib/braintrust/contrib/openai/integration.rb +58 -0
  16. data/lib/braintrust/contrib/openai/patcher.rb +130 -0
  17. data/lib/braintrust/contrib/patcher.rb +76 -0
  18. data/lib/braintrust/contrib/rails/railtie.rb +16 -0
  19. data/lib/braintrust/contrib/registry.rb +107 -0
  20. data/lib/braintrust/contrib/ruby_llm/deprecated.rb +45 -0
  21. data/lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb +464 -0
  22. data/lib/braintrust/contrib/ruby_llm/instrumentation/common.rb +58 -0
  23. data/lib/braintrust/contrib/ruby_llm/integration.rb +54 -0
  24. data/lib/braintrust/contrib/ruby_llm/patcher.rb +44 -0
  25. data/lib/braintrust/contrib/ruby_openai/deprecated.rb +24 -0
  26. data/lib/braintrust/contrib/ruby_openai/instrumentation/chat.rb +149 -0
  27. data/lib/braintrust/contrib/ruby_openai/instrumentation/common.rb +138 -0
  28. data/lib/braintrust/contrib/ruby_openai/instrumentation/responses.rb +146 -0
  29. data/lib/braintrust/contrib/ruby_openai/integration.rb +58 -0
  30. data/lib/braintrust/contrib/ruby_openai/patcher.rb +85 -0
  31. data/lib/braintrust/contrib/setup.rb +168 -0
  32. data/lib/braintrust/contrib/support/openai.rb +72 -0
  33. data/lib/braintrust/contrib/support/otel.rb +23 -0
  34. data/lib/braintrust/contrib.rb +205 -0
  35. data/lib/braintrust/internal/env.rb +33 -0
  36. data/lib/braintrust/internal/time.rb +44 -0
  37. data/lib/braintrust/setup.rb +50 -0
  38. data/lib/braintrust/state.rb +5 -0
  39. data/lib/braintrust/trace.rb +0 -51
  40. data/lib/braintrust/version.rb +1 -1
  41. data/lib/braintrust.rb +10 -1
  42. metadata +38 -7
  43. data/lib/braintrust/trace/contrib/anthropic.rb +0 -316
  44. data/lib/braintrust/trace/contrib/github.com/alexrudall/ruby-openai/ruby-openai.rb +0 -377
  45. data/lib/braintrust/trace/contrib/github.com/crmne/ruby_llm.rb +0 -631
  46. data/lib/braintrust/trace/contrib/openai.rb +0 -611
  47. data/lib/braintrust/trace/tokens.rb +0 -109
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4a4dd86789a3ce80b891b88fb9b125bb7b6d85e86928adeb5fd7c1fec671ff56
4
- data.tar.gz: 171fe031f960d0a6f3abbfacbb20094b35bcd9199cf132e30063868807f95f8b
3
+ metadata.gz: b28ce15a3d9599baecd29de264cd9455811cc18b1338fb948a19347f3c6327cc
4
+ data.tar.gz: f752b82b8f79ec77f72b2b95b5c98f77b5de92af77d39370eccc87af4a2835d7
5
5
  SHA512:
6
- metadata.gz: 3cf52e457ca5d4cdad2f8873bb45d0eeb0b79b33c5edbcd467e65ce7261caea9bcc86ea9d562835197279105f14f2787bcac6622faad4ec420a5f2fa27a003d2
7
- data.tar.gz: 5a7ccc20a3e63840e15c41dd82eaa8653a2f0dce693321e6527ace59c54255524ae1d794d2dce058127b747a514e8777729384cde9c157374039880bb3b01013
6
+ metadata.gz: 86b922e9252bb1e390e892458931b4fc19d495ba04842f54f37d3a0fdb52db02fb7e5237bee65df74f870fa457cfefa98d46abe65bdf855ce46bdfe7056d9c97
7
+ data.tar.gz: 7f426e8c342d57136c175dae2c69359a8103da8ec7425750c3bc86de0232ac505b78f2d82c547124070e8e7cacd6cd9c479e45b74ad06b6379118af1eb281cb8
data/README.md CHANGED
@@ -4,282 +4,315 @@
4
4
  [![Documentation](https://img.shields.io/badge/docs-gemdocs.org-blue.svg)](https://gemdocs.org/gems/braintrust/)
5
5
  ![Beta](https://img.shields.io/badge/status-beta-yellow)
6
6
 
7
- ## Overview
7
+ This is the official Ruby SDK for [Braintrust](https://www.braintrust.dev), for tracing and evaluating your AI applications.
8
+
9
+ *NOTE: This SDK is currently in BETA status and APIs may change between minor versions.*
10
+
11
+ - [Quick Start](#quick-start)
12
+ - [Installation](#installation)
13
+ - [Setup script](#setup-script)
14
+ - [CLI Command](#cli-command)
15
+ - [Braintrust.init](#braintrustinit)
16
+ - [Environment variables](#environment-variables)
17
+ - [Tracing](#tracing)
18
+ - [Supported providers](#supported-providers)
19
+ - [Manually applying instrumentation](#manually-applying-instrumentation)
20
+ - [Creating custom spans](#creating-custom-spans)
21
+ - [Attachments](#attachments)
22
+ - [Viewing traces](#viewing-traces)
23
+ - [Evals](#evals)
24
+ - [Datasets](#datasets)
25
+ - [Remote scorers](#remote-scorers)
26
+ - [Documentation](#documentation)
27
+ - [Troubleshooting](#troubleshooting)
28
+ - [Contributing](#contributing)
29
+ - [License](#license)
8
30
 
9
- This library provides tools for **evaluating** and **tracing** AI applications in [Braintrust](https://www.braintrust.dev). Use it to:
10
-
11
- - **Evaluate** your AI models with custom test cases and scoring functions
12
- - **Trace** LLM calls and monitor AI application performance with OpenTelemetry
13
- - **Integrate** seamlessly with OpenAI and other LLM providers
14
-
15
- This SDK is currently in BETA status and APIs may change.
16
-
17
- ## Installation
31
+ ## Quick Start
18
32
 
19
- Add this line to your application's Gemfile:
33
+ Add to your Gemfile:
20
34
 
21
35
  ```ruby
22
- gem 'braintrust'
36
+ gem "braintrust", require: "braintrust/setup"
23
37
  ```
24
38
 
25
- And then execute:
39
+ Set your API key and install:
26
40
 
27
41
  ```bash
42
+ export BRAINTRUST_API_KEY="your-api-key"
28
43
  bundle install
29
44
  ```
30
45
 
31
- Or install it yourself as:
46
+ Your LLM calls are now automatically traced. View them at [braintrust.dev](https://www.braintrust.dev).
32
47
 
33
- ```bash
34
- gem install braintrust
35
- ```
48
+ ## Installation
36
49
 
37
- ## Quick Start
50
+ The SDK also offers additional setup options for a variety of applications.
38
51
 
39
- ### Set up your API key
52
+ | | What it looks like | When to Use |
53
+ | -------------------------------------- | ----------------------------------- | --------------------------------------------------------------------------- |
54
+ | [**Setup script**](#setup-script) | `require 'braintrust/setup'` | You'd like to automatically setup the SDK at load time. |
55
+ | [**CLI command**](#cli-command) | `braintrust exec -- ruby app.rb` | You prefer not to modify the application source code. |
56
+ | [**Braintrust.init**](#braintrustinit) | Call `Braintrust.init` in your code | You need to control when setup occurs, or require customized configuration. |
40
57
 
41
- ```bash
42
- export BRAINTRUST_API_KEY="your-api-key"
43
- ```
58
+ See [our examples](./examples/setup/README.md) for more detail.
44
59
 
45
- ### Evals
60
+ ### Setup script
46
61
 
47
- ```ruby
48
- require "braintrust"
62
+ For most applications, we recommend adding `require: "braintrust/setup"` to your `Gemfile` or an initializer file in your Ruby application to automatically setup the SDK. This will automatically apply instrumentation to all available LLM libraries.
49
63
 
50
- Braintrust.init
64
+ You can use [environment variables](#environment-variables) to configure behavior.
51
65
 
52
- # Define task to evaluate
53
- task = ->(input) { input.include?("a") ? "fruit" : "vegetable" }
66
+ ### CLI Command
54
67
 
55
- # Run evaluation
56
- Braintrust::Eval.run(
57
- project: "my-project",
58
- experiment: "food-classifier",
59
- cases: [
60
- {input: "apple", expected: "fruit"},
61
- {input: "carrot", expected: "vegetable"}
62
- ],
63
- task: task,
64
- scorers: [
65
- ->(input, expected, output) { output == expected ? 1.0 : 0.0 }
66
- ]
67
- )
68
+ You can use this CLI command to instrument any Ruby application without modifying the source code.
69
+
70
+ First, make sure the gem is installed on the system:
71
+
72
+ ```bash
73
+ gem install braintrust
68
74
  ```
69
75
 
70
- ### Tracing
76
+ Then wrap the start up command of any Ruby application to apply:
71
77
 
72
- ```ruby
73
- require "braintrust"
74
- require "opentelemetry/sdk"
78
+ ```bash
79
+ braintrust exec -- ruby app.rb
80
+ braintrust exec -- bundle exec rails server
81
+ braintrust exec --only openai -- ruby app.rb
82
+ ```
75
83
 
76
- # Initialize Braintrust
77
- Braintrust.init
84
+ You can use [environment variables](#environment-variables) to configure behavior.
78
85
 
79
- # Get a tracer
80
- tracer = OpenTelemetry.tracer_provider.tracer("my-app")
86
+ ---
81
87
 
82
- # Create spans to track operations
83
- tracer.in_span("process-data") do |span|
84
- span.set_attribute("user.id", "123")
85
- span.set_attribute("operation.type", "data_processing")
88
+ *NOTE: Installing a package at the system-level does not guarantee compatibility with all Ruby applications on that system; conflicts with dependencies can arise.*
86
89
 
87
- # Your code here
88
- puts "Processing data..."
89
- sleep 0.1
90
+ For stronger assurance of compatibility, we recommend either:
90
91
 
91
- # Nested spans are automatically linked
92
- tracer.in_span("nested-operation") do |nested_span|
93
- nested_span.set_attribute("step", "1")
94
- puts "Nested operation..."
95
- end
96
- end
92
+ - Installing via the application's `Gemfile` and `bundle install` when possible.
93
+ - **OR** for OCI/Docker deployments, `gem install braintrust` when building images in your CI/CD pipeline (and verifying their safe function.)
97
94
 
98
- # Shutdown to flush spans
99
- OpenTelemetry.tracer_provider.shutdown
95
+ ---
100
96
 
101
- puts "View trace in Braintrust!"
102
- ```
97
+ ### Braintrust.init
103
98
 
104
- ### OpenAI Tracing
99
+ For more control over when auto-instrumentation is applied:
105
100
 
106
101
  ```ruby
107
102
  require "braintrust"
108
- require "openai"
109
103
 
110
104
  Braintrust.init
105
+ ```
111
106
 
112
- client = OpenAI::Client.new(api_key: ENV["OPENAI_API_KEY"])
107
+ **Options:**
113
108
 
114
- Braintrust::Trace::OpenAI.wrap(client)
109
+ | Option | Default | Description |
110
+ | ----------------- | ---------------------------------------- | --------------------------------------------------------------------------- |
111
+ | `api_key` | `ENV['BRAINTRUST_API_KEY']` | API key |
112
+ | `auto_instrument` | `true` | `true`, `false`, or Hash with `:only`/`:except` keys to filter integrations |
113
+ | `blocking_login` | `false` | Block until login completes (async login when `false`) |
114
+ | `default_project` | `ENV['BRAINTRUST_DEFAULT_PROJECT']` | Default project for spans |
115
+ | `enable_tracing` | `true` | Enable OpenTelemetry tracing |
116
+ | `filter_ai_spans` | `ENV['BRAINTRUST_OTEL_FILTER_AI_SPANS']` | Only export AI-related spans |
117
+ | `org_name` | `ENV['BRAINTRUST_ORG_NAME']` | Organization name |
118
+ | `set_global` | `true` | Set as global state. Set to `false` for isolated instances |
115
119
 
116
- tracer = OpenTelemetry.tracer_provider.tracer("openai-app")
117
- root_span = nil
120
+ **Example with options:**
118
121
 
119
- response = tracer.in_span("chat-completion") do |span|
120
- root_span = span
122
+ ```ruby
123
+ Braintrust.init(
124
+ default_project: "my-project",
125
+ auto_instrument: { only: [:openai] }
126
+ )
127
+ ```
121
128
 
122
- client.chat.completions.create(
123
- messages: [
124
- {role: "system", content: "You are a helpful assistant."},
125
- {role: "user", content: "Say hello!"}
126
- ],
127
- model: "gpt-4o-mini",
128
- max_tokens: 100
129
- )
130
- end
129
+ ### Environment variables
131
130
 
132
- puts "Response: #{response.choices[0].message.content}"
131
+ | Variable | Description |
132
+ | --------------------------------- | ------------------------------------------------------------------------- |
133
+ | `BRAINTRUST_API_KEY` | Required. Your Braintrust API key |
134
+ | `BRAINTRUST_API_URL` | Braintrust API URL (default: `https://api.braintrust.dev`) |
135
+ | `BRAINTRUST_APP_URL` | Braintrust app URL (default: `https://www.braintrust.dev`) |
136
+ | `BRAINTRUST_AUTO_INSTRUMENT` | Set to `false` to disable auto-instrumentation |
137
+ | `BRAINTRUST_DEBUG` | Set to `true` to enable debug logging |
138
+ | `BRAINTRUST_DEFAULT_PROJECT` | Default project for spans |
139
+ | `BRAINTRUST_INSTRUMENT_EXCEPT` | Comma-separated list of integrations to skip |
140
+ | `BRAINTRUST_INSTRUMENT_ONLY` | Comma-separated list of integrations to enable (e.g., `openai,anthropic`) |
141
+ | `BRAINTRUST_ORG_NAME` | Organization name |
142
+ | `BRAINTRUST_OTEL_FILTER_AI_SPANS` | Set to `true` to only export AI-related spans |
133
143
 
134
- puts "View trace at: #{Braintrust::Trace.permalink(root_span)}"
144
+ ## Tracing
135
145
 
136
- OpenTelemetry.tracer_provider.shutdown
137
- ```
146
+ ### Supported providers
147
+
148
+ The SDK automatically instruments these LLM libraries:
138
149
 
139
- ### Anthropic Tracing
150
+ | Provider | Gem | Versions | Integration Name | Examples |
151
+ | --------- | ------------- | -------- | ---------------- | ----------------------------------------------- |
152
+ | Anthropic | `anthropic` | >= 0.3.0 | `:anthropic` | [Link](./examples/contrib/anthropic.rb) |
153
+ | OpenAI | `openai` | >= 0.1.0 | `:openai` | [Link](./examples/contrib/openai.rb) |
154
+ | | `ruby-openai` | >= 7.0.0 | `:ruby_openai` | [Link](./examples/contrib/ruby-openai.rb) |
155
+ | Multiple | `ruby_llm` | >= 1.8.0 | `:ruby_llm` | [Link](./examples/contrib/ruby_llm.rb) |
156
+
157
+ ### Manually applying instrumentation
158
+
159
+ For fine-grained control, disable auto-instrumentation and instrument specific clients:
140
160
 
141
161
  ```ruby
142
162
  require "braintrust"
143
- require "anthropic"
144
-
145
- Braintrust.init
163
+ require "openai"
146
164
 
147
- client = Anthropic::Client.new(api_key: ENV["ANTHROPIC_API_KEY"])
165
+ Braintrust.init(auto_instrument: false) # Or BRAINTRUST_AUTO_INSTRUMENT=false
148
166
 
149
- Braintrust::Trace::Anthropic.wrap(client)
167
+ # Instrument all OpenAI clients
168
+ Braintrust.instrument!(:openai)
150
169
 
151
- tracer = OpenTelemetry.tracer_provider.tracer("anthropic-app")
152
- root_span = nil
170
+ # OR instrument a single client
171
+ client = OpenAI::Client.new
172
+ Braintrust.instrument!(:openai, target: client)
173
+ ```
153
174
 
154
- message = tracer.in_span("chat-message") do |span|
155
- root_span = span
175
+ ### Creating custom spans
156
176
 
157
- client.messages.create(
158
- model: "claude-3-haiku-20240307",
159
- max_tokens: 100,
160
- system: "You are a helpful assistant.",
161
- messages: [
162
- {role: "user", content: "Say hello!"}
163
- ]
164
- )
165
- end
177
+ Wrap business logic in spans to see it in your traces:
166
178
 
167
- puts "Response: #{message.content[0].text}"
179
+ ```ruby
180
+ tracer = OpenTelemetry.tracer_provider.tracer("my-app")
168
181
 
169
- puts "View trace at: #{Braintrust::Trace.permalink(root_span)}"
182
+ tracer.in_span("process-request") do |span|
183
+ span.set_attribute("user.id", user_id)
170
184
 
171
- OpenTelemetry.tracer_provider.shutdown
185
+ # LLM calls inside here are automatically nested under this span
186
+ response = client.chat.completions.create(...)
187
+ end
172
188
  ```
173
189
 
174
- ### RubyLLM Tracing
190
+ ### Attachments
191
+
192
+ Log binary data (images, PDFs, audio) in your traces:
175
193
 
176
194
  ```ruby
177
- require "braintrust"
178
- require "ruby_llm"
195
+ require "braintrust/trace/attachment"
179
196
 
180
- Braintrust.init
197
+ att = Braintrust::Trace::Attachment.from_file("image/png", "./photo.png")
181
198
 
182
- # Wrap RubyLLM globally (wraps all Chat instances)
183
- Braintrust::Trace::Contrib::Github::Crmne::RubyLLM.wrap
199
+ # Use in messages (OpenAI/Anthropic format)
200
+ messages = [
201
+ {
202
+ role: "user",
203
+ content: [
204
+ {type: "text", text: "What's in this image?"},
205
+ att.to_h
206
+ ]
207
+ }
208
+ ]
184
209
 
185
- tracer = OpenTelemetry.tracer_provider.tracer("ruby-llm-app")
186
- root_span = nil
210
+ # Log to span
211
+ span.set_attribute("braintrust.input_json", JSON.generate(messages))
212
+ ```
187
213
 
188
- response = tracer.in_span("chat") do |span|
189
- root_span = span
214
+ Create attachments from various sources:
190
215
 
191
- chat = RubyLLM.chat(model: "gpt-4o-mini")
192
- chat.ask("Say hello!")
193
- end
216
+ ```ruby
217
+ Braintrust::Trace::Attachment.from_bytes("image/jpeg", image_data)
218
+ Braintrust::Trace::Attachment.from_file("application/pdf", "./doc.pdf")
219
+ Braintrust::Trace::Attachment.from_url("https://example.com/image.png")
220
+ ```
221
+
222
+ See example: [trace_attachments.rb](./examples/trace/trace_attachments.rb)
194
223
 
195
- puts "Response: #{response.content}"
224
+ ### Viewing traces
196
225
 
197
- puts "View trace at: #{Braintrust::Trace.permalink(root_span)}"
226
+ Get a permalink to any span:
198
227
 
199
- OpenTelemetry.tracer_provider.shutdown
228
+ ```ruby
229
+ tracer = OpenTelemetry.tracer_provider.tracer("my-app")
230
+
231
+ tracer.in_span("my-operation") do |span|
232
+ # your code here
233
+ puts "View trace at: #{Braintrust::Trace.permalink(span)}"
234
+ end
200
235
  ```
201
236
 
202
- ### Attachments
237
+ ## Evals
203
238
 
204
- Attachments allow you to log binary data (images, PDFs, audio, etc.) as part of your traces. This is particularly useful for multimodal AI applications like vision models.
239
+ Run evaluations against your AI systems:
205
240
 
206
241
  ```ruby
207
242
  require "braintrust"
208
- require "braintrust/trace/attachment"
209
243
 
210
244
  Braintrust.init
211
245
 
212
- tracer = OpenTelemetry.tracer_provider.tracer("vision-app")
213
-
214
- tracer.in_span("analyze-image") do |span|
215
- # Create attachment from file
216
- att = Braintrust::Trace::Attachment.from_file(
217
- Braintrust::Trace::Attachment::IMAGE_PNG,
218
- "./photo.png"
219
- )
220
-
221
- # Build message with attachment (OpenAI/Anthropic format)
222
- messages = [
223
- {
224
- role: "user",
225
- content: [
226
- {type: "text", text: "What's in this image?"},
227
- att.to_h # Converts to {"type" => "base64_attachment", "content" => "data:..."}
228
- ]
229
- }
246
+ Braintrust::Eval.run(
247
+ project: "my-project",
248
+ experiment: "classifier-v1",
249
+ cases: [
250
+ {input: "apple", expected: "fruit"},
251
+ {input: "carrot", expected: "vegetable"}
252
+ ],
253
+ task: ->(input) { classify(input) },
254
+ scorers: [
255
+ ->(input, expected, output) { output == expected ? 1.0 : 0.0 }
230
256
  ]
257
+ )
258
+ ```
231
259
 
232
- # Log to trace
233
- span.set_attribute("braintrust.input_json", JSON.generate(messages))
234
- end
260
+ ### Datasets
261
+
262
+ Load test cases from a Braintrust dataset:
235
263
 
236
- OpenTelemetry.tracer_provider.shutdown
264
+ ```ruby
265
+ Braintrust::Eval.run(
266
+ project: "my-project",
267
+ dataset: "my-dataset",
268
+ task: ->(input) { classify(input) },
269
+ scorers: [...]
270
+ )
237
271
  ```
238
272
 
239
- You can create attachments from bytes, files, or URLs:
273
+ ### Remote scorers
274
+
275
+ Use scoring functions defined in Braintrust:
240
276
 
241
277
  ```ruby
242
- # From bytes
243
- att = Braintrust::Trace::Attachment.from_bytes("image/jpeg", image_data)
278
+ Braintrust::Eval.run(
279
+ project: "my-project",
280
+ cases: [...],
281
+ task: ->(input) { ... },
282
+ scorers: [
283
+ Braintrust::Scorer.remote("my-project", "accuracy-scorer")
284
+ ]
285
+ )
286
+ ```
244
287
 
245
- # From file
246
- att = Braintrust::Trace::Attachment.from_file("application/pdf", "./doc.pdf")
288
+ See examples: [eval.rb](./examples/eval.rb), [dataset.rb](./examples/eval/dataset.rb), [remote_functions.rb](./examples/eval/remote_functions.rb)
247
289
 
248
- # From URL
249
- att = Braintrust::Trace::Attachment.from_url("https://example.com/image.png")
250
- ```
290
+ ## Documentation
251
291
 
252
- ## Features
292
+ - [Braintrust Documentation](https://www.braintrust.dev/docs)
293
+ - [API Reference](https://gemdocs.org/gems/braintrust/)
253
294
 
254
- - **Evaluations**: Run systematic evaluations of your AI systems with custom scoring functions
255
- - **Tracing**: Automatic instrumentation for OpenAI and Anthropic API calls with OpenTelemetry
256
- - **Datasets**: Manage and version your evaluation datasets
257
- - **Experiments**: Track different versions and configurations of your AI systems
258
- - **Observability**: Monitor your AI applications in production
295
+ ## Troubleshooting
259
296
 
260
- ## Examples
297
+ #### No traces after adding `require 'braintrust/setup'` to the Gemfile
261
298
 
262
- Check out the [`examples/`](./examples/) directory for complete working examples:
299
+ First verify there are no errors in your logs after running with `BRAINTRUST_DEBUG=true` set.
263
300
 
264
- - [eval.rb](./examples/eval.rb) - Create and run evaluations with custom test cases and scoring functions
265
- - [trace.rb](./examples/trace.rb) - Manual span creation and tracing
266
- - [openai.rb](./examples/openai.rb) - Automatically trace OpenAI API calls
267
- - [alexrudall_openai.rb](./examples/alexrudall_openai.rb) - Automatically trace ruby-openai gem API calls
268
- - [anthropic.rb](./examples/anthropic.rb) - Automatically trace Anthropic API calls
269
- - [ruby_llm.rb](./examples/ruby_llm.rb) - Automatically trace RubyLLM API calls
270
- - [trace/trace_attachments.rb](./examples/trace/trace_attachments.rb) - Log attachments (images, PDFs) in traces
271
- - [eval/dataset.rb](./examples/eval/dataset.rb) - Run evaluations using datasets stored in Braintrust
272
- - [eval/remote_functions.rb](./examples/eval/remote_functions.rb) - Use remote scoring functions
301
+ Your application needs the following for this to work:
273
302
 
274
- ## Documentation
303
+ ```ruby
304
+ require 'bundler/setup'
305
+ Bundler.require
306
+ ```
275
307
 
276
- - [Braintrust Documentation](https://www.braintrust.dev/docs)
277
- - [API Documentation](https://gemdocs.org/gems/braintrust/)
308
+ It is present by default in Rails applications, but may not be in Sinatra, Rack, or other applications.
309
+
310
+ Alternatively, you can add `require 'braintrust/setup'` to your application initialization files.
278
311
 
279
312
  ## Contributing
280
313
 
281
- See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and contribution guidelines.
314
+ See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and guidelines.
282
315
 
283
316
  ## License
284
317
 
285
- This project is licensed under the Apache License 2.0. See the [LICENSE](./LICENSE) file for details.
318
+ Apache License 2.0 - see [LICENSE](./LICENSE).
data/exe/braintrust ADDED
@@ -0,0 +1,143 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ # Braintrust CLI - Auto-instrument Ruby applications
5
+ #
6
+ # Usage:
7
+ # braintrust exec -- ruby app.rb
8
+ # braintrust exec -- bundle exec rails server
9
+ # braintrust exec --only openai,anthropic -- ruby app.rb
10
+
11
+ require "optparse"
12
+
13
+ module Braintrust
14
+ module CLI
15
+ class << self
16
+ def run(args)
17
+ command = parse_args(args)
18
+
19
+ case command
20
+ when :exec
21
+ exec_command
22
+ when :help, nil
23
+ print_help
24
+ exit((command == :help) ? 0 : 1)
25
+ else
26
+ puts "Unknown command: #{command}"
27
+ print_help
28
+ exit 1
29
+ end
30
+ end
31
+
32
+ private
33
+
34
+ def parse_args(args)
35
+ @options = {}
36
+ @remaining_args = []
37
+
38
+ # Find -- separator
39
+ separator_index = args.index("--")
40
+ if separator_index
41
+ to_parse = args[0...separator_index]
42
+ @remaining_args = args[(separator_index + 1)..]
43
+ else
44
+ to_parse = args.dup
45
+ end
46
+
47
+ parser = OptionParser.new do |opts|
48
+ opts.banner = "Usage: braintrust <command> [options] -- COMMAND"
49
+
50
+ opts.on("--only INTEGRATIONS", "Only instrument these (comma-separated)") do |v|
51
+ @options[:only] = v
52
+ end
53
+
54
+ opts.on("--except INTEGRATIONS", "Skip these integrations (comma-separated)") do |v|
55
+ @options[:except] = v
56
+ end
57
+
58
+ opts.on("-h", "--help", "Show this help") do
59
+ return :help
60
+ end
61
+
62
+ opts.on("-v", "--version", "Show version") do
63
+ require_relative "../lib/braintrust/version"
64
+ puts "braintrust #{Braintrust::VERSION}"
65
+ exit 0
66
+ end
67
+ end
68
+
69
+ parser.parse!(to_parse)
70
+
71
+ return nil if to_parse.empty?
72
+ to_parse.first.to_sym
73
+ end
74
+
75
+ def exec_command
76
+ if @remaining_args.empty?
77
+ puts "Error: No command specified after --"
78
+ puts "Usage: braintrust exec [options] -- COMMAND"
79
+ exit 1
80
+ end
81
+
82
+ # Set environment variables for filtering
83
+ ENV["BRAINTRUST_INSTRUMENT_ONLY"] = @options[:only] if @options[:only]
84
+ ENV["BRAINTRUST_INSTRUMENT_EXCEPT"] = @options[:except] if @options[:except]
85
+
86
+ # Inject auto-instrument via RUBYOPT
87
+ set_rubyopt!
88
+
89
+ # Execute the command with error handling
90
+ exec_with_error_handling(@remaining_args)
91
+ end
92
+
93
+ def rubyopts
94
+ ["-rbraintrust/setup"]
95
+ end
96
+
97
+ def set_rubyopt!
98
+ existing = ENV["RUBYOPT"]
99
+ ENV["RUBYOPT"] = existing ? "#{existing} #{rubyopts.join(" ")}" : rubyopts.join(" ")
100
+ end
101
+
102
+ def exec_with_error_handling(args)
103
+ Kernel.exec(*args)
104
+ rescue Errno::ENOENT => e
105
+ Kernel.warn "braintrust exec failed: #{e.class.name} #{e.message}"
106
+ Kernel.exit 127
107
+ rescue Errno::EACCES, Errno::ENOEXEC => e
108
+ Kernel.warn "braintrust exec failed: #{e.class.name} #{e.message}"
109
+ Kernel.exit 126
110
+ end
111
+
112
+ def print_help
113
+ puts <<~HELP
114
+ Braintrust CLI - Auto-instrument Ruby applications
115
+
116
+ Usage:
117
+ braintrust exec [options] -- COMMAND
118
+
119
+ Commands:
120
+ exec Run a command with auto-instrumentation enabled
121
+
122
+ Options:
123
+ --only INTEGRATIONS Only instrument these (comma-separated)
124
+ --except INTEGRATIONS Skip these integrations (comma-separated)
125
+ -h, --help Show this help
126
+ -v, --version Show version
127
+
128
+ Examples:
129
+ braintrust exec -- ruby app.rb
130
+ braintrust exec -- rails server
131
+ braintrust exec --only openai -- ruby app.rb
132
+
133
+ Environment Variables:
134
+ BRAINTRUST_API_KEY API key for Braintrust
135
+ BRAINTRUST_INSTRUMENT_ONLY Comma-separated whitelist
136
+ BRAINTRUST_INSTRUMENT_EXCEPT Comma-separated blacklist
137
+ HELP
138
+ end
139
+ end
140
+ end
141
+ end
142
+
143
+ Braintrust::CLI.run(ARGV)
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Backward compatibility shim for the old Anthropic integration API.
4
+ # This file now just delegates to the new API.
5
+
6
+ require_relative "../../../braintrust"
7
+
8
+ module Braintrust
9
+ module Trace
10
+ module Anthropic
11
+ # Wrap an Anthropic::Client to automatically create spans for messages.
12
+ # This is the legacy API - delegates to the new contrib framework.
13
+ #
14
+ # @param client [Anthropic::Client] the Anthropic client to wrap
15
+ # @param tracer_provider [OpenTelemetry::SDK::Trace::TracerProvider] the tracer provider
16
+ # @return [Anthropic::Client] the wrapped client
17
+ def self.wrap(client, tracer_provider: nil)
18
+ Log.warn("Braintrust::Trace::Anthropic.wrap() is deprecated and will be removed in a future version: use Braintrust.instrument!() instead.")
19
+ Braintrust.instrument!(:anthropic, target: client, tracer_provider: tracer_provider)
20
+ client
21
+ end
22
+ end
23
+ end
24
+ end