braintrust 0.0.12 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +214 -180
- data/exe/braintrust +143 -0
- data/lib/braintrust/contrib/anthropic/deprecated.rb +24 -0
- data/lib/braintrust/contrib/anthropic/instrumentation/beta_messages.rb +242 -0
- data/lib/braintrust/contrib/anthropic/instrumentation/common.rb +53 -0
- data/lib/braintrust/contrib/anthropic/instrumentation/messages.rb +232 -0
- data/lib/braintrust/contrib/anthropic/integration.rb +53 -0
- data/lib/braintrust/contrib/anthropic/patcher.rb +145 -0
- data/lib/braintrust/contrib/context.rb +56 -0
- data/lib/braintrust/contrib/integration.rb +160 -0
- data/lib/braintrust/contrib/openai/deprecated.rb +22 -0
- data/lib/braintrust/contrib/openai/instrumentation/chat.rb +298 -0
- data/lib/braintrust/contrib/openai/instrumentation/common.rb +134 -0
- data/lib/braintrust/contrib/openai/instrumentation/moderations.rb +93 -0
- data/lib/braintrust/contrib/openai/instrumentation/responses.rb +187 -0
- data/lib/braintrust/contrib/openai/integration.rb +58 -0
- data/lib/braintrust/contrib/openai/patcher.rb +173 -0
- data/lib/braintrust/contrib/patcher.rb +76 -0
- data/lib/braintrust/contrib/rails/railtie.rb +16 -0
- data/lib/braintrust/contrib/registry.rb +107 -0
- data/lib/braintrust/contrib/ruby_llm/deprecated.rb +45 -0
- data/lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb +464 -0
- data/lib/braintrust/contrib/ruby_llm/instrumentation/common.rb +58 -0
- data/lib/braintrust/contrib/ruby_llm/integration.rb +54 -0
- data/lib/braintrust/contrib/ruby_llm/patcher.rb +44 -0
- data/lib/braintrust/contrib/ruby_openai/deprecated.rb +24 -0
- data/lib/braintrust/contrib/ruby_openai/instrumentation/chat.rb +149 -0
- data/lib/braintrust/contrib/ruby_openai/instrumentation/common.rb +138 -0
- data/lib/braintrust/contrib/ruby_openai/instrumentation/moderations.rb +94 -0
- data/lib/braintrust/contrib/ruby_openai/instrumentation/responses.rb +146 -0
- data/lib/braintrust/contrib/ruby_openai/integration.rb +58 -0
- data/lib/braintrust/contrib/ruby_openai/patcher.rb +120 -0
- data/lib/braintrust/contrib/setup.rb +168 -0
- data/lib/braintrust/contrib/support/openai.rb +72 -0
- data/lib/braintrust/contrib/support/otel.rb +23 -0
- data/lib/braintrust/contrib.rb +205 -0
- data/lib/braintrust/internal/env.rb +39 -0
- data/lib/braintrust/internal/time.rb +44 -0
- data/lib/braintrust/setup.rb +50 -0
- data/lib/braintrust/state.rb +6 -1
- data/lib/braintrust/trace.rb +41 -51
- data/lib/braintrust/version.rb +1 -1
- data/lib/braintrust.rb +10 -1
- metadata +41 -7
- data/lib/braintrust/trace/contrib/anthropic.rb +0 -316
- data/lib/braintrust/trace/contrib/github.com/alexrudall/ruby-openai/ruby-openai.rb +0 -377
- data/lib/braintrust/trace/contrib/github.com/crmne/ruby_llm.rb +0 -631
- data/lib/braintrust/trace/contrib/openai.rb +0 -611
- data/lib/braintrust/trace/tokens.rb +0 -109
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 40d5c41ef999d495f9eb7b3a4cd41f1bb9ee7ba831de2074195a7d9e18da2e3a
|
|
4
|
+
data.tar.gz: 8e48aee25cd02a4b936da1264a9964097c6e07744faf0564d81885daa72b87da
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 9391f2dcec3c92e032d3e7009ec4bf1d26f6d2a606f0edad78acf78078f116393d95a4a3d70a08010b89df214f92498905c67cd02b3bd6327b5aa2cbf1dda872
|
|
7
|
+
data.tar.gz: 5665f9bcb49a8b5ca90f5d58d9c279120a053756c6ec03ba17557b6064c7a7142ac9674789fe4c89df70e2ac61d1d1ce9318a33b897b55da1d45b9bc9df1bf15
|
data/README.md
CHANGED
|
@@ -4,282 +4,316 @@
|
|
|
4
4
|
[](https://gemdocs.org/gems/braintrust/)
|
|
5
5
|

|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
This is the official Ruby SDK for [Braintrust](https://www.braintrust.dev), for tracing and evaluating your AI applications.
|
|
8
|
+
|
|
9
|
+
*NOTE: This SDK is currently in BETA status and APIs may change between minor versions.*
|
|
10
|
+
|
|
11
|
+
- [Quick Start](#quick-start)
|
|
12
|
+
- [Installation](#installation)
|
|
13
|
+
- [Setup script](#setup-script)
|
|
14
|
+
- [CLI Command](#cli-command)
|
|
15
|
+
- [Braintrust.init](#braintrustinit)
|
|
16
|
+
- [Environment variables](#environment-variables)
|
|
17
|
+
- [Tracing](#tracing)
|
|
18
|
+
- [Supported providers](#supported-providers)
|
|
19
|
+
- [Manually applying instrumentation](#manually-applying-instrumentation)
|
|
20
|
+
- [Creating custom spans](#creating-custom-spans)
|
|
21
|
+
- [Attachments](#attachments)
|
|
22
|
+
- [Viewing traces](#viewing-traces)
|
|
23
|
+
- [Evals](#evals)
|
|
24
|
+
- [Datasets](#datasets)
|
|
25
|
+
- [Remote scorers](#remote-scorers)
|
|
26
|
+
- [Documentation](#documentation)
|
|
27
|
+
- [Troubleshooting](#troubleshooting)
|
|
28
|
+
- [Contributing](#contributing)
|
|
29
|
+
- [License](#license)
|
|
8
30
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
- **Evaluate** your AI models with custom test cases and scoring functions
|
|
12
|
-
- **Trace** LLM calls and monitor AI application performance with OpenTelemetry
|
|
13
|
-
- **Integrate** seamlessly with OpenAI and other LLM providers
|
|
14
|
-
|
|
15
|
-
This SDK is currently in BETA status and APIs may change.
|
|
16
|
-
|
|
17
|
-
## Installation
|
|
31
|
+
## Quick Start
|
|
18
32
|
|
|
19
|
-
Add
|
|
33
|
+
Add to your Gemfile:
|
|
20
34
|
|
|
21
35
|
```ruby
|
|
22
|
-
gem
|
|
36
|
+
gem "braintrust", require: "braintrust/setup"
|
|
23
37
|
```
|
|
24
38
|
|
|
25
|
-
|
|
39
|
+
Set your API key and install:
|
|
26
40
|
|
|
27
41
|
```bash
|
|
42
|
+
export BRAINTRUST_API_KEY="your-api-key"
|
|
28
43
|
bundle install
|
|
29
44
|
```
|
|
30
45
|
|
|
31
|
-
|
|
46
|
+
Your LLM calls are now automatically traced. View them at [braintrust.dev](https://www.braintrust.dev).
|
|
32
47
|
|
|
33
|
-
|
|
34
|
-
gem install braintrust
|
|
35
|
-
```
|
|
48
|
+
## Installation
|
|
36
49
|
|
|
37
|
-
|
|
50
|
+
The SDK also offers additional setup options for a variety of applications.
|
|
38
51
|
|
|
39
|
-
|
|
52
|
+
| | What it looks like | When to Use |
|
|
53
|
+
| -------------------------------------- | ----------------------------------- | --------------------------------------------------------------------------- |
|
|
54
|
+
| [**Setup script**](#setup-script) | `require 'braintrust/setup'` | You'd like to automatically setup the SDK at load time. |
|
|
55
|
+
| [**CLI command**](#cli-command) | `braintrust exec -- ruby app.rb` | You prefer not to modify the application source code. |
|
|
56
|
+
| [**Braintrust.init**](#braintrustinit) | Call `Braintrust.init` in your code | You need to control when setup occurs, or require customized configuration. |
|
|
40
57
|
|
|
41
|
-
|
|
42
|
-
export BRAINTRUST_API_KEY="your-api-key"
|
|
43
|
-
```
|
|
58
|
+
See [our examples](./examples/setup/README.md) for more detail.
|
|
44
59
|
|
|
45
|
-
###
|
|
60
|
+
### Setup script
|
|
46
61
|
|
|
47
|
-
|
|
48
|
-
require "braintrust"
|
|
62
|
+
For most applications, we recommend adding `require: "braintrust/setup"` to your `Gemfile` or an initializer file in your Ruby application to automatically setup the SDK. This will automatically apply instrumentation to all available LLM libraries.
|
|
49
63
|
|
|
50
|
-
|
|
64
|
+
You can use [environment variables](#environment-variables) to configure behavior.
|
|
51
65
|
|
|
52
|
-
|
|
53
|
-
task = ->(input) { input.include?("a") ? "fruit" : "vegetable" }
|
|
66
|
+
### CLI Command
|
|
54
67
|
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
{input: "carrot", expected: "vegetable"}
|
|
62
|
-
],
|
|
63
|
-
task: task,
|
|
64
|
-
scorers: [
|
|
65
|
-
->(input, expected, output) { output == expected ? 1.0 : 0.0 }
|
|
66
|
-
]
|
|
67
|
-
)
|
|
68
|
+
You can use this CLI command to instrument any Ruby application without modifying the source code.
|
|
69
|
+
|
|
70
|
+
First, make sure the gem is installed on the system:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
gem install braintrust
|
|
68
74
|
```
|
|
69
75
|
|
|
70
|
-
|
|
76
|
+
Then wrap the start up command of any Ruby application to apply:
|
|
71
77
|
|
|
72
|
-
```
|
|
73
|
-
|
|
74
|
-
|
|
78
|
+
```bash
|
|
79
|
+
braintrust exec -- ruby app.rb
|
|
80
|
+
braintrust exec -- bundle exec rails server
|
|
81
|
+
braintrust exec --only openai -- ruby app.rb
|
|
82
|
+
```
|
|
75
83
|
|
|
76
|
-
#
|
|
77
|
-
Braintrust.init
|
|
84
|
+
You can use [environment variables](#environment-variables) to configure behavior.
|
|
78
85
|
|
|
79
|
-
|
|
80
|
-
tracer = OpenTelemetry.tracer_provider.tracer("my-app")
|
|
86
|
+
---
|
|
81
87
|
|
|
82
|
-
|
|
83
|
-
tracer.in_span("process-data") do |span|
|
|
84
|
-
span.set_attribute("user.id", "123")
|
|
85
|
-
span.set_attribute("operation.type", "data_processing")
|
|
88
|
+
*NOTE: Installing a package at the system-level does not guarantee compatibility with all Ruby applications on that system; conflicts with dependencies can arise.*
|
|
86
89
|
|
|
87
|
-
|
|
88
|
-
puts "Processing data..."
|
|
89
|
-
sleep 0.1
|
|
90
|
+
For stronger assurance of compatibility, we recommend either:
|
|
90
91
|
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
nested_span.set_attribute("step", "1")
|
|
94
|
-
puts "Nested operation..."
|
|
95
|
-
end
|
|
96
|
-
end
|
|
92
|
+
- Installing via the application's `Gemfile` and `bundle install` when possible.
|
|
93
|
+
- **OR** for OCI/Docker deployments, `gem install braintrust` when building images in your CI/CD pipeline (and verifying their safe function.)
|
|
97
94
|
|
|
98
|
-
|
|
99
|
-
OpenTelemetry.tracer_provider.shutdown
|
|
95
|
+
---
|
|
100
96
|
|
|
101
|
-
|
|
102
|
-
```
|
|
97
|
+
### Braintrust.init
|
|
103
98
|
|
|
104
|
-
|
|
99
|
+
For more control over when auto-instrumentation is applied:
|
|
105
100
|
|
|
106
101
|
```ruby
|
|
107
102
|
require "braintrust"
|
|
108
|
-
require "openai"
|
|
109
103
|
|
|
110
104
|
Braintrust.init
|
|
105
|
+
```
|
|
111
106
|
|
|
112
|
-
|
|
107
|
+
**Options:**
|
|
113
108
|
|
|
114
|
-
|
|
109
|
+
| Option | Default | Description |
|
|
110
|
+
| ----------------- | ---------------------------------------- | --------------------------------------------------------------------------- |
|
|
111
|
+
| `api_key` | `ENV['BRAINTRUST_API_KEY']` | API key |
|
|
112
|
+
| `auto_instrument` | `true` | `true`, `false`, or Hash with `:only`/`:except` keys to filter integrations |
|
|
113
|
+
| `blocking_login` | `false` | Block until login completes (async login when `false`) |
|
|
114
|
+
| `default_project` | `ENV['BRAINTRUST_DEFAULT_PROJECT']` | Default project for spans |
|
|
115
|
+
| `enable_tracing` | `true` | Enable OpenTelemetry tracing |
|
|
116
|
+
| `filter_ai_spans` | `ENV['BRAINTRUST_OTEL_FILTER_AI_SPANS']` | Only export AI-related spans |
|
|
117
|
+
| `org_name` | `ENV['BRAINTRUST_ORG_NAME']` | Organization name |
|
|
118
|
+
| `set_global` | `true` | Set as global state. Set to `false` for isolated instances |
|
|
115
119
|
|
|
116
|
-
|
|
117
|
-
root_span = nil
|
|
120
|
+
**Example with options:**
|
|
118
121
|
|
|
119
|
-
|
|
120
|
-
|
|
122
|
+
```ruby
|
|
123
|
+
Braintrust.init(
|
|
124
|
+
default_project: "my-project",
|
|
125
|
+
auto_instrument: { only: [:openai] }
|
|
126
|
+
)
|
|
127
|
+
```
|
|
121
128
|
|
|
122
|
-
|
|
123
|
-
messages: [
|
|
124
|
-
{role: "system", content: "You are a helpful assistant."},
|
|
125
|
-
{role: "user", content: "Say hello!"}
|
|
126
|
-
],
|
|
127
|
-
model: "gpt-4o-mini",
|
|
128
|
-
max_tokens: 100
|
|
129
|
-
)
|
|
130
|
-
end
|
|
129
|
+
### Environment variables
|
|
131
130
|
|
|
132
|
-
|
|
131
|
+
| Variable | Description |
|
|
132
|
+
| --------------------------------- | ------------------------------------------------------------------------- |
|
|
133
|
+
| `BRAINTRUST_API_KEY` | Required. Your Braintrust API key |
|
|
134
|
+
| `BRAINTRUST_API_URL` | Braintrust API URL (default: `https://api.braintrust.dev`) |
|
|
135
|
+
| `BRAINTRUST_APP_URL` | Braintrust app URL (default: `https://www.braintrust.dev`) |
|
|
136
|
+
| `BRAINTRUST_AUTO_INSTRUMENT` | Set to `false` to disable auto-instrumentation |
|
|
137
|
+
| `BRAINTRUST_DEBUG` | Set to `true` to enable debug logging |
|
|
138
|
+
| `BRAINTRUST_DEFAULT_PROJECT` | Default project for spans |
|
|
139
|
+
| `BRAINTRUST_FLUSH_ON_EXIT` | Set to `false` to disable automatic span flushing on program exit |
|
|
140
|
+
| `BRAINTRUST_INSTRUMENT_EXCEPT` | Comma-separated list of integrations to skip |
|
|
141
|
+
| `BRAINTRUST_INSTRUMENT_ONLY` | Comma-separated list of integrations to enable (e.g., `openai,anthropic`) |
|
|
142
|
+
| `BRAINTRUST_ORG_NAME` | Organization name |
|
|
143
|
+
| `BRAINTRUST_OTEL_FILTER_AI_SPANS` | Set to `true` to only export AI-related spans |
|
|
133
144
|
|
|
134
|
-
|
|
145
|
+
## Tracing
|
|
135
146
|
|
|
136
|
-
|
|
137
|
-
|
|
147
|
+
### Supported providers
|
|
148
|
+
|
|
149
|
+
The SDK automatically instruments these LLM libraries:
|
|
138
150
|
|
|
139
|
-
|
|
151
|
+
| Provider | Gem | Versions | Integration Name | Examples |
|
|
152
|
+
| --------- | ------------- | -------- | ---------------- | ----------------------------------------------- |
|
|
153
|
+
| Anthropic | `anthropic` | >= 0.3.0 | `:anthropic` | [Link](./examples/contrib/anthropic.rb) |
|
|
154
|
+
| OpenAI | `openai` | >= 0.1.0 | `:openai` | [Link](./examples/contrib/openai.rb) |
|
|
155
|
+
| | `ruby-openai` | >= 7.0.0 | `:ruby_openai` | [Link](./examples/contrib/ruby-openai.rb) |
|
|
156
|
+
| Multiple | `ruby_llm` | >= 1.8.0 | `:ruby_llm` | [Link](./examples/contrib/ruby_llm.rb) |
|
|
157
|
+
|
|
158
|
+
### Manually applying instrumentation
|
|
159
|
+
|
|
160
|
+
For fine-grained control, disable auto-instrumentation and instrument specific clients:
|
|
140
161
|
|
|
141
162
|
```ruby
|
|
142
163
|
require "braintrust"
|
|
143
|
-
require "
|
|
144
|
-
|
|
145
|
-
Braintrust.init
|
|
164
|
+
require "openai"
|
|
146
165
|
|
|
147
|
-
|
|
166
|
+
Braintrust.init(auto_instrument: false) # Or BRAINTRUST_AUTO_INSTRUMENT=false
|
|
148
167
|
|
|
149
|
-
|
|
168
|
+
# Instrument all OpenAI clients
|
|
169
|
+
Braintrust.instrument!(:openai)
|
|
150
170
|
|
|
151
|
-
|
|
152
|
-
|
|
171
|
+
# OR instrument a single client
|
|
172
|
+
client = OpenAI::Client.new
|
|
173
|
+
Braintrust.instrument!(:openai, target: client)
|
|
174
|
+
```
|
|
153
175
|
|
|
154
|
-
|
|
155
|
-
root_span = span
|
|
176
|
+
### Creating custom spans
|
|
156
177
|
|
|
157
|
-
|
|
158
|
-
model: "claude-3-haiku-20240307",
|
|
159
|
-
max_tokens: 100,
|
|
160
|
-
system: "You are a helpful assistant.",
|
|
161
|
-
messages: [
|
|
162
|
-
{role: "user", content: "Say hello!"}
|
|
163
|
-
]
|
|
164
|
-
)
|
|
165
|
-
end
|
|
178
|
+
Wrap business logic in spans to see it in your traces:
|
|
166
179
|
|
|
167
|
-
|
|
180
|
+
```ruby
|
|
181
|
+
tracer = OpenTelemetry.tracer_provider.tracer("my-app")
|
|
168
182
|
|
|
169
|
-
|
|
183
|
+
tracer.in_span("process-request") do |span|
|
|
184
|
+
span.set_attribute("user.id", user_id)
|
|
170
185
|
|
|
171
|
-
|
|
186
|
+
# LLM calls inside here are automatically nested under this span
|
|
187
|
+
response = client.chat.completions.create(...)
|
|
188
|
+
end
|
|
172
189
|
```
|
|
173
190
|
|
|
174
|
-
###
|
|
191
|
+
### Attachments
|
|
192
|
+
|
|
193
|
+
Log binary data (images, PDFs, audio) in your traces:
|
|
175
194
|
|
|
176
195
|
```ruby
|
|
177
|
-
require "braintrust"
|
|
178
|
-
require "ruby_llm"
|
|
196
|
+
require "braintrust/trace/attachment"
|
|
179
197
|
|
|
180
|
-
Braintrust.
|
|
198
|
+
att = Braintrust::Trace::Attachment.from_file("image/png", "./photo.png")
|
|
181
199
|
|
|
182
|
-
#
|
|
183
|
-
|
|
200
|
+
# Use in messages (OpenAI/Anthropic format)
|
|
201
|
+
messages = [
|
|
202
|
+
{
|
|
203
|
+
role: "user",
|
|
204
|
+
content: [
|
|
205
|
+
{type: "text", text: "What's in this image?"},
|
|
206
|
+
att.to_h
|
|
207
|
+
]
|
|
208
|
+
}
|
|
209
|
+
]
|
|
184
210
|
|
|
185
|
-
|
|
186
|
-
|
|
211
|
+
# Log to span
|
|
212
|
+
span.set_attribute("braintrust.input_json", JSON.generate(messages))
|
|
213
|
+
```
|
|
187
214
|
|
|
188
|
-
|
|
189
|
-
root_span = span
|
|
215
|
+
Create attachments from various sources:
|
|
190
216
|
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
217
|
+
```ruby
|
|
218
|
+
Braintrust::Trace::Attachment.from_bytes("image/jpeg", image_data)
|
|
219
|
+
Braintrust::Trace::Attachment.from_file("application/pdf", "./doc.pdf")
|
|
220
|
+
Braintrust::Trace::Attachment.from_url("https://example.com/image.png")
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
See example: [trace_attachments.rb](./examples/trace/trace_attachments.rb)
|
|
194
224
|
|
|
195
|
-
|
|
225
|
+
### Viewing traces
|
|
196
226
|
|
|
197
|
-
|
|
227
|
+
Get a permalink to any span:
|
|
198
228
|
|
|
199
|
-
|
|
229
|
+
```ruby
|
|
230
|
+
tracer = OpenTelemetry.tracer_provider.tracer("my-app")
|
|
231
|
+
|
|
232
|
+
tracer.in_span("my-operation") do |span|
|
|
233
|
+
# your code here
|
|
234
|
+
puts "View trace at: #{Braintrust::Trace.permalink(span)}"
|
|
235
|
+
end
|
|
200
236
|
```
|
|
201
237
|
|
|
202
|
-
|
|
238
|
+
## Evals
|
|
203
239
|
|
|
204
|
-
|
|
240
|
+
Run evaluations against your AI systems:
|
|
205
241
|
|
|
206
242
|
```ruby
|
|
207
243
|
require "braintrust"
|
|
208
|
-
require "braintrust/trace/attachment"
|
|
209
244
|
|
|
210
245
|
Braintrust.init
|
|
211
246
|
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
)
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
messages = [
|
|
223
|
-
{
|
|
224
|
-
role: "user",
|
|
225
|
-
content: [
|
|
226
|
-
{type: "text", text: "What's in this image?"},
|
|
227
|
-
att.to_h # Converts to {"type" => "base64_attachment", "content" => "data:..."}
|
|
228
|
-
]
|
|
229
|
-
}
|
|
247
|
+
Braintrust::Eval.run(
|
|
248
|
+
project: "my-project",
|
|
249
|
+
experiment: "classifier-v1",
|
|
250
|
+
cases: [
|
|
251
|
+
{input: "apple", expected: "fruit"},
|
|
252
|
+
{input: "carrot", expected: "vegetable"}
|
|
253
|
+
],
|
|
254
|
+
task: ->(input) { classify(input) },
|
|
255
|
+
scorers: [
|
|
256
|
+
->(input, expected, output) { output == expected ? 1.0 : 0.0 }
|
|
230
257
|
]
|
|
258
|
+
)
|
|
259
|
+
```
|
|
231
260
|
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
261
|
+
### Datasets
|
|
262
|
+
|
|
263
|
+
Load test cases from a Braintrust dataset:
|
|
235
264
|
|
|
236
|
-
|
|
265
|
+
```ruby
|
|
266
|
+
Braintrust::Eval.run(
|
|
267
|
+
project: "my-project",
|
|
268
|
+
dataset: "my-dataset",
|
|
269
|
+
task: ->(input) { classify(input) },
|
|
270
|
+
scorers: [...]
|
|
271
|
+
)
|
|
237
272
|
```
|
|
238
273
|
|
|
239
|
-
|
|
274
|
+
### Remote scorers
|
|
275
|
+
|
|
276
|
+
Use scoring functions defined in Braintrust:
|
|
240
277
|
|
|
241
278
|
```ruby
|
|
242
|
-
|
|
243
|
-
|
|
279
|
+
Braintrust::Eval.run(
|
|
280
|
+
project: "my-project",
|
|
281
|
+
cases: [...],
|
|
282
|
+
task: ->(input) { ... },
|
|
283
|
+
scorers: [
|
|
284
|
+
Braintrust::Scorer.remote("my-project", "accuracy-scorer")
|
|
285
|
+
]
|
|
286
|
+
)
|
|
287
|
+
```
|
|
244
288
|
|
|
245
|
-
|
|
246
|
-
att = Braintrust::Trace::Attachment.from_file("application/pdf", "./doc.pdf")
|
|
289
|
+
See examples: [eval.rb](./examples/eval.rb), [dataset.rb](./examples/eval/dataset.rb), [remote_functions.rb](./examples/eval/remote_functions.rb)
|
|
247
290
|
|
|
248
|
-
|
|
249
|
-
att = Braintrust::Trace::Attachment.from_url("https://example.com/image.png")
|
|
250
|
-
```
|
|
291
|
+
## Documentation
|
|
251
292
|
|
|
252
|
-
|
|
293
|
+
- [Braintrust Documentation](https://www.braintrust.dev/docs)
|
|
294
|
+
- [API Reference](https://gemdocs.org/gems/braintrust/)
|
|
253
295
|
|
|
254
|
-
|
|
255
|
-
- **Tracing**: Automatic instrumentation for OpenAI and Anthropic API calls with OpenTelemetry
|
|
256
|
-
- **Datasets**: Manage and version your evaluation datasets
|
|
257
|
-
- **Experiments**: Track different versions and configurations of your AI systems
|
|
258
|
-
- **Observability**: Monitor your AI applications in production
|
|
296
|
+
## Troubleshooting
|
|
259
297
|
|
|
260
|
-
|
|
298
|
+
#### No traces after adding `require 'braintrust/setup'` to the Gemfile
|
|
261
299
|
|
|
262
|
-
|
|
300
|
+
First verify there are no errors in your logs after running with `BRAINTRUST_DEBUG=true` set.
|
|
263
301
|
|
|
264
|
-
|
|
265
|
-
- [trace.rb](./examples/trace.rb) - Manual span creation and tracing
|
|
266
|
-
- [openai.rb](./examples/openai.rb) - Automatically trace OpenAI API calls
|
|
267
|
-
- [alexrudall_openai.rb](./examples/alexrudall_openai.rb) - Automatically trace ruby-openai gem API calls
|
|
268
|
-
- [anthropic.rb](./examples/anthropic.rb) - Automatically trace Anthropic API calls
|
|
269
|
-
- [ruby_llm.rb](./examples/ruby_llm.rb) - Automatically trace RubyLLM API calls
|
|
270
|
-
- [trace/trace_attachments.rb](./examples/trace/trace_attachments.rb) - Log attachments (images, PDFs) in traces
|
|
271
|
-
- [eval/dataset.rb](./examples/eval/dataset.rb) - Run evaluations using datasets stored in Braintrust
|
|
272
|
-
- [eval/remote_functions.rb](./examples/eval/remote_functions.rb) - Use remote scoring functions
|
|
302
|
+
Your application needs the following for this to work:
|
|
273
303
|
|
|
274
|
-
|
|
304
|
+
```ruby
|
|
305
|
+
require 'bundler/setup'
|
|
306
|
+
Bundler.require
|
|
307
|
+
```
|
|
275
308
|
|
|
276
|
-
|
|
277
|
-
|
|
309
|
+
It is present by default in Rails applications, but may not be in Sinatra, Rack, or other applications.
|
|
310
|
+
|
|
311
|
+
Alternatively, you can add `require 'braintrust/setup'` to your application initialization files.
|
|
278
312
|
|
|
279
313
|
## Contributing
|
|
280
314
|
|
|
281
|
-
See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and
|
|
315
|
+
See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and guidelines.
|
|
282
316
|
|
|
283
317
|
## License
|
|
284
318
|
|
|
285
|
-
|
|
319
|
+
Apache License 2.0 - see [LICENSE](./LICENSE).
|
data/exe/braintrust
ADDED
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
# Braintrust CLI - Auto-instrument Ruby applications
|
|
5
|
+
#
|
|
6
|
+
# Usage:
|
|
7
|
+
# braintrust exec -- ruby app.rb
|
|
8
|
+
# braintrust exec -- bundle exec rails server
|
|
9
|
+
# braintrust exec --only openai,anthropic -- ruby app.rb
|
|
10
|
+
|
|
11
|
+
require "optparse"
|
|
12
|
+
|
|
13
|
+
module Braintrust
|
|
14
|
+
module CLI
|
|
15
|
+
class << self
|
|
16
|
+
def run(args)
|
|
17
|
+
command = parse_args(args)
|
|
18
|
+
|
|
19
|
+
case command
|
|
20
|
+
when :exec
|
|
21
|
+
exec_command
|
|
22
|
+
when :help, nil
|
|
23
|
+
print_help
|
|
24
|
+
exit((command == :help) ? 0 : 1)
|
|
25
|
+
else
|
|
26
|
+
puts "Unknown command: #{command}"
|
|
27
|
+
print_help
|
|
28
|
+
exit 1
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
private
|
|
33
|
+
|
|
34
|
+
def parse_args(args)
|
|
35
|
+
@options = {}
|
|
36
|
+
@remaining_args = []
|
|
37
|
+
|
|
38
|
+
# Find -- separator
|
|
39
|
+
separator_index = args.index("--")
|
|
40
|
+
if separator_index
|
|
41
|
+
to_parse = args[0...separator_index]
|
|
42
|
+
@remaining_args = args[(separator_index + 1)..]
|
|
43
|
+
else
|
|
44
|
+
to_parse = args.dup
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
parser = OptionParser.new do |opts|
|
|
48
|
+
opts.banner = "Usage: braintrust <command> [options] -- COMMAND"
|
|
49
|
+
|
|
50
|
+
opts.on("--only INTEGRATIONS", "Only instrument these (comma-separated)") do |v|
|
|
51
|
+
@options[:only] = v
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
opts.on("--except INTEGRATIONS", "Skip these integrations (comma-separated)") do |v|
|
|
55
|
+
@options[:except] = v
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
opts.on("-h", "--help", "Show this help") do
|
|
59
|
+
return :help
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
opts.on("-v", "--version", "Show version") do
|
|
63
|
+
require_relative "../lib/braintrust/version"
|
|
64
|
+
puts "braintrust #{Braintrust::VERSION}"
|
|
65
|
+
exit 0
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
parser.parse!(to_parse)
|
|
70
|
+
|
|
71
|
+
return nil if to_parse.empty?
|
|
72
|
+
to_parse.first.to_sym
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def exec_command
|
|
76
|
+
if @remaining_args.empty?
|
|
77
|
+
puts "Error: No command specified after --"
|
|
78
|
+
puts "Usage: braintrust exec [options] -- COMMAND"
|
|
79
|
+
exit 1
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# Set environment variables for filtering
|
|
83
|
+
ENV["BRAINTRUST_INSTRUMENT_ONLY"] = @options[:only] if @options[:only]
|
|
84
|
+
ENV["BRAINTRUST_INSTRUMENT_EXCEPT"] = @options[:except] if @options[:except]
|
|
85
|
+
|
|
86
|
+
# Inject auto-instrument via RUBYOPT
|
|
87
|
+
set_rubyopt!
|
|
88
|
+
|
|
89
|
+
# Execute the command with error handling
|
|
90
|
+
exec_with_error_handling(@remaining_args)
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def rubyopts
|
|
94
|
+
["-rbraintrust/setup"]
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
def set_rubyopt!
|
|
98
|
+
existing = ENV["RUBYOPT"]
|
|
99
|
+
ENV["RUBYOPT"] = existing ? "#{existing} #{rubyopts.join(" ")}" : rubyopts.join(" ")
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
def exec_with_error_handling(args)
|
|
103
|
+
Kernel.exec(*args)
|
|
104
|
+
rescue Errno::ENOENT => e
|
|
105
|
+
Kernel.warn "braintrust exec failed: #{e.class.name} #{e.message}"
|
|
106
|
+
Kernel.exit 127
|
|
107
|
+
rescue Errno::EACCES, Errno::ENOEXEC => e
|
|
108
|
+
Kernel.warn "braintrust exec failed: #{e.class.name} #{e.message}"
|
|
109
|
+
Kernel.exit 126
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
def print_help
|
|
113
|
+
puts <<~HELP
|
|
114
|
+
Braintrust CLI - Auto-instrument Ruby applications
|
|
115
|
+
|
|
116
|
+
Usage:
|
|
117
|
+
braintrust exec [options] -- COMMAND
|
|
118
|
+
|
|
119
|
+
Commands:
|
|
120
|
+
exec Run a command with auto-instrumentation enabled
|
|
121
|
+
|
|
122
|
+
Options:
|
|
123
|
+
--only INTEGRATIONS Only instrument these (comma-separated)
|
|
124
|
+
--except INTEGRATIONS Skip these integrations (comma-separated)
|
|
125
|
+
-h, --help Show this help
|
|
126
|
+
-v, --version Show version
|
|
127
|
+
|
|
128
|
+
Examples:
|
|
129
|
+
braintrust exec -- ruby app.rb
|
|
130
|
+
braintrust exec -- rails server
|
|
131
|
+
braintrust exec --only openai -- ruby app.rb
|
|
132
|
+
|
|
133
|
+
Environment Variables:
|
|
134
|
+
BRAINTRUST_API_KEY API key for Braintrust
|
|
135
|
+
BRAINTRUST_INSTRUMENT_ONLY Comma-separated whitelist
|
|
136
|
+
BRAINTRUST_INSTRUMENT_EXCEPT Comma-separated blacklist
|
|
137
|
+
HELP
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
end
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
Braintrust::CLI.run(ARGV)
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Backward compatibility shim for the old Anthropic integration API.
|
|
4
|
+
# This file now just delegates to the new API.
|
|
5
|
+
|
|
6
|
+
require_relative "../../../braintrust"
|
|
7
|
+
|
|
8
|
+
module Braintrust
|
|
9
|
+
module Trace
|
|
10
|
+
module Anthropic
|
|
11
|
+
# Wrap an Anthropic::Client to automatically create spans for messages.
|
|
12
|
+
# This is the legacy API - delegates to the new contrib framework.
|
|
13
|
+
#
|
|
14
|
+
# @param client [Anthropic::Client] the Anthropic client to wrap
|
|
15
|
+
# @param tracer_provider [OpenTelemetry::SDK::Trace::TracerProvider] the tracer provider
|
|
16
|
+
# @return [Anthropic::Client] the wrapped client
|
|
17
|
+
def self.wrap(client, tracer_provider: nil)
|
|
18
|
+
Log.warn("Braintrust::Trace::Anthropic.wrap() is deprecated and will be removed in a future version: use Braintrust.instrument!() instead.")
|
|
19
|
+
Braintrust.instrument!(:anthropic, target: client, tracer_provider: tracer_provider)
|
|
20
|
+
client
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
end
|