braintrust 0.0.12 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +213 -180
- data/exe/braintrust +143 -0
- data/lib/braintrust/contrib/anthropic/deprecated.rb +24 -0
- data/lib/braintrust/contrib/anthropic/instrumentation/common.rb +53 -0
- data/lib/braintrust/contrib/anthropic/instrumentation/messages.rb +232 -0
- data/lib/braintrust/contrib/anthropic/integration.rb +53 -0
- data/lib/braintrust/contrib/anthropic/patcher.rb +62 -0
- data/lib/braintrust/contrib/context.rb +56 -0
- data/lib/braintrust/contrib/integration.rb +160 -0
- data/lib/braintrust/contrib/openai/deprecated.rb +22 -0
- data/lib/braintrust/contrib/openai/instrumentation/chat.rb +298 -0
- data/lib/braintrust/contrib/openai/instrumentation/common.rb +134 -0
- data/lib/braintrust/contrib/openai/instrumentation/responses.rb +187 -0
- data/lib/braintrust/contrib/openai/integration.rb +58 -0
- data/lib/braintrust/contrib/openai/patcher.rb +130 -0
- data/lib/braintrust/contrib/patcher.rb +76 -0
- data/lib/braintrust/contrib/rails/railtie.rb +16 -0
- data/lib/braintrust/contrib/registry.rb +107 -0
- data/lib/braintrust/contrib/ruby_llm/deprecated.rb +45 -0
- data/lib/braintrust/contrib/ruby_llm/instrumentation/chat.rb +464 -0
- data/lib/braintrust/contrib/ruby_llm/instrumentation/common.rb +58 -0
- data/lib/braintrust/contrib/ruby_llm/integration.rb +54 -0
- data/lib/braintrust/contrib/ruby_llm/patcher.rb +44 -0
- data/lib/braintrust/contrib/ruby_openai/deprecated.rb +24 -0
- data/lib/braintrust/contrib/ruby_openai/instrumentation/chat.rb +149 -0
- data/lib/braintrust/contrib/ruby_openai/instrumentation/common.rb +138 -0
- data/lib/braintrust/contrib/ruby_openai/instrumentation/responses.rb +146 -0
- data/lib/braintrust/contrib/ruby_openai/integration.rb +58 -0
- data/lib/braintrust/contrib/ruby_openai/patcher.rb +85 -0
- data/lib/braintrust/contrib/setup.rb +168 -0
- data/lib/braintrust/contrib/support/openai.rb +72 -0
- data/lib/braintrust/contrib/support/otel.rb +23 -0
- data/lib/braintrust/contrib.rb +205 -0
- data/lib/braintrust/internal/env.rb +33 -0
- data/lib/braintrust/internal/time.rb +44 -0
- data/lib/braintrust/setup.rb +50 -0
- data/lib/braintrust/state.rb +5 -0
- data/lib/braintrust/trace.rb +0 -51
- data/lib/braintrust/version.rb +1 -1
- data/lib/braintrust.rb +10 -1
- metadata +38 -7
- data/lib/braintrust/trace/contrib/anthropic.rb +0 -316
- data/lib/braintrust/trace/contrib/github.com/alexrudall/ruby-openai/ruby-openai.rb +0 -377
- data/lib/braintrust/trace/contrib/github.com/crmne/ruby_llm.rb +0 -631
- data/lib/braintrust/trace/contrib/openai.rb +0 -611
- data/lib/braintrust/trace/tokens.rb +0 -109
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b28ce15a3d9599baecd29de264cd9455811cc18b1338fb948a19347f3c6327cc
|
|
4
|
+
data.tar.gz: f752b82b8f79ec77f72b2b95b5c98f77b5de92af77d39370eccc87af4a2835d7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 86b922e9252bb1e390e892458931b4fc19d495ba04842f54f37d3a0fdb52db02fb7e5237bee65df74f870fa457cfefa98d46abe65bdf855ce46bdfe7056d9c97
|
|
7
|
+
data.tar.gz: 7f426e8c342d57136c175dae2c69359a8103da8ec7425750c3bc86de0232ac505b78f2d82c547124070e8e7cacd6cd9c479e45b74ad06b6379118af1eb281cb8
|
data/README.md
CHANGED
|
@@ -4,282 +4,315 @@
|
|
|
4
4
|
[](https://gemdocs.org/gems/braintrust/)
|
|
5
5
|

|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
This is the official Ruby SDK for [Braintrust](https://www.braintrust.dev), for tracing and evaluating your AI applications.
|
|
8
|
+
|
|
9
|
+
*NOTE: This SDK is currently in BETA status and APIs may change between minor versions.*
|
|
10
|
+
|
|
11
|
+
- [Quick Start](#quick-start)
|
|
12
|
+
- [Installation](#installation)
|
|
13
|
+
- [Setup script](#setup-script)
|
|
14
|
+
- [CLI Command](#cli-command)
|
|
15
|
+
- [Braintrust.init](#braintrustinit)
|
|
16
|
+
- [Environment variables](#environment-variables)
|
|
17
|
+
- [Tracing](#tracing)
|
|
18
|
+
- [Supported providers](#supported-providers)
|
|
19
|
+
- [Manually applying instrumentation](#manually-applying-instrumentation)
|
|
20
|
+
- [Creating custom spans](#creating-custom-spans)
|
|
21
|
+
- [Attachments](#attachments)
|
|
22
|
+
- [Viewing traces](#viewing-traces)
|
|
23
|
+
- [Evals](#evals)
|
|
24
|
+
- [Datasets](#datasets)
|
|
25
|
+
- [Remote scorers](#remote-scorers)
|
|
26
|
+
- [Documentation](#documentation)
|
|
27
|
+
- [Troubleshooting](#troubleshooting)
|
|
28
|
+
- [Contributing](#contributing)
|
|
29
|
+
- [License](#license)
|
|
8
30
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
- **Evaluate** your AI models with custom test cases and scoring functions
|
|
12
|
-
- **Trace** LLM calls and monitor AI application performance with OpenTelemetry
|
|
13
|
-
- **Integrate** seamlessly with OpenAI and other LLM providers
|
|
14
|
-
|
|
15
|
-
This SDK is currently in BETA status and APIs may change.
|
|
16
|
-
|
|
17
|
-
## Installation
|
|
31
|
+
## Quick Start
|
|
18
32
|
|
|
19
|
-
Add
|
|
33
|
+
Add to your Gemfile:
|
|
20
34
|
|
|
21
35
|
```ruby
|
|
22
|
-
gem
|
|
36
|
+
gem "braintrust", require: "braintrust/setup"
|
|
23
37
|
```
|
|
24
38
|
|
|
25
|
-
|
|
39
|
+
Set your API key and install:
|
|
26
40
|
|
|
27
41
|
```bash
|
|
42
|
+
export BRAINTRUST_API_KEY="your-api-key"
|
|
28
43
|
bundle install
|
|
29
44
|
```
|
|
30
45
|
|
|
31
|
-
|
|
46
|
+
Your LLM calls are now automatically traced. View them at [braintrust.dev](https://www.braintrust.dev).
|
|
32
47
|
|
|
33
|
-
|
|
34
|
-
gem install braintrust
|
|
35
|
-
```
|
|
48
|
+
## Installation
|
|
36
49
|
|
|
37
|
-
|
|
50
|
+
The SDK also offers additional setup options for a variety of applications.
|
|
38
51
|
|
|
39
|
-
|
|
52
|
+
| | What it looks like | When to Use |
|
|
53
|
+
| -------------------------------------- | ----------------------------------- | --------------------------------------------------------------------------- |
|
|
54
|
+
| [**Setup script**](#setup-script) | `require 'braintrust/setup'` | You'd like to automatically setup the SDK at load time. |
|
|
55
|
+
| [**CLI command**](#cli-command) | `braintrust exec -- ruby app.rb` | You prefer not to modify the application source code. |
|
|
56
|
+
| [**Braintrust.init**](#braintrustinit) | Call `Braintrust.init` in your code | You need to control when setup occurs, or require customized configuration. |
|
|
40
57
|
|
|
41
|
-
|
|
42
|
-
export BRAINTRUST_API_KEY="your-api-key"
|
|
43
|
-
```
|
|
58
|
+
See [our examples](./examples/setup/README.md) for more detail.
|
|
44
59
|
|
|
45
|
-
###
|
|
60
|
+
### Setup script
|
|
46
61
|
|
|
47
|
-
|
|
48
|
-
require "braintrust"
|
|
62
|
+
For most applications, we recommend adding `require: "braintrust/setup"` to your `Gemfile` or an initializer file in your Ruby application to automatically setup the SDK. This will automatically apply instrumentation to all available LLM libraries.
|
|
49
63
|
|
|
50
|
-
|
|
64
|
+
You can use [environment variables](#environment-variables) to configure behavior.
|
|
51
65
|
|
|
52
|
-
|
|
53
|
-
task = ->(input) { input.include?("a") ? "fruit" : "vegetable" }
|
|
66
|
+
### CLI Command
|
|
54
67
|
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
{input: "carrot", expected: "vegetable"}
|
|
62
|
-
],
|
|
63
|
-
task: task,
|
|
64
|
-
scorers: [
|
|
65
|
-
->(input, expected, output) { output == expected ? 1.0 : 0.0 }
|
|
66
|
-
]
|
|
67
|
-
)
|
|
68
|
+
You can use this CLI command to instrument any Ruby application without modifying the source code.
|
|
69
|
+
|
|
70
|
+
First, make sure the gem is installed on the system:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
gem install braintrust
|
|
68
74
|
```
|
|
69
75
|
|
|
70
|
-
|
|
76
|
+
Then wrap the start up command of any Ruby application to apply:
|
|
71
77
|
|
|
72
|
-
```
|
|
73
|
-
|
|
74
|
-
|
|
78
|
+
```bash
|
|
79
|
+
braintrust exec -- ruby app.rb
|
|
80
|
+
braintrust exec -- bundle exec rails server
|
|
81
|
+
braintrust exec --only openai -- ruby app.rb
|
|
82
|
+
```
|
|
75
83
|
|
|
76
|
-
#
|
|
77
|
-
Braintrust.init
|
|
84
|
+
You can use [environment variables](#environment-variables) to configure behavior.
|
|
78
85
|
|
|
79
|
-
|
|
80
|
-
tracer = OpenTelemetry.tracer_provider.tracer("my-app")
|
|
86
|
+
---
|
|
81
87
|
|
|
82
|
-
|
|
83
|
-
tracer.in_span("process-data") do |span|
|
|
84
|
-
span.set_attribute("user.id", "123")
|
|
85
|
-
span.set_attribute("operation.type", "data_processing")
|
|
88
|
+
*NOTE: Installing a package at the system-level does not guarantee compatibility with all Ruby applications on that system; conflicts with dependencies can arise.*
|
|
86
89
|
|
|
87
|
-
|
|
88
|
-
puts "Processing data..."
|
|
89
|
-
sleep 0.1
|
|
90
|
+
For stronger assurance of compatibility, we recommend either:
|
|
90
91
|
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
nested_span.set_attribute("step", "1")
|
|
94
|
-
puts "Nested operation..."
|
|
95
|
-
end
|
|
96
|
-
end
|
|
92
|
+
- Installing via the application's `Gemfile` and `bundle install` when possible.
|
|
93
|
+
- **OR** for OCI/Docker deployments, `gem install braintrust` when building images in your CI/CD pipeline (and verifying their safe function.)
|
|
97
94
|
|
|
98
|
-
|
|
99
|
-
OpenTelemetry.tracer_provider.shutdown
|
|
95
|
+
---
|
|
100
96
|
|
|
101
|
-
|
|
102
|
-
```
|
|
97
|
+
### Braintrust.init
|
|
103
98
|
|
|
104
|
-
|
|
99
|
+
For more control over when auto-instrumentation is applied:
|
|
105
100
|
|
|
106
101
|
```ruby
|
|
107
102
|
require "braintrust"
|
|
108
|
-
require "openai"
|
|
109
103
|
|
|
110
104
|
Braintrust.init
|
|
105
|
+
```
|
|
111
106
|
|
|
112
|
-
|
|
107
|
+
**Options:**
|
|
113
108
|
|
|
114
|
-
|
|
109
|
+
| Option | Default | Description |
|
|
110
|
+
| ----------------- | ---------------------------------------- | --------------------------------------------------------------------------- |
|
|
111
|
+
| `api_key` | `ENV['BRAINTRUST_API_KEY']` | API key |
|
|
112
|
+
| `auto_instrument` | `true` | `true`, `false`, or Hash with `:only`/`:except` keys to filter integrations |
|
|
113
|
+
| `blocking_login` | `false` | Block until login completes (async login when `false`) |
|
|
114
|
+
| `default_project` | `ENV['BRAINTRUST_DEFAULT_PROJECT']` | Default project for spans |
|
|
115
|
+
| `enable_tracing` | `true` | Enable OpenTelemetry tracing |
|
|
116
|
+
| `filter_ai_spans` | `ENV['BRAINTRUST_OTEL_FILTER_AI_SPANS']` | Only export AI-related spans |
|
|
117
|
+
| `org_name` | `ENV['BRAINTRUST_ORG_NAME']` | Organization name |
|
|
118
|
+
| `set_global` | `true` | Set as global state. Set to `false` for isolated instances |
|
|
115
119
|
|
|
116
|
-
|
|
117
|
-
root_span = nil
|
|
120
|
+
**Example with options:**
|
|
118
121
|
|
|
119
|
-
|
|
120
|
-
|
|
122
|
+
```ruby
|
|
123
|
+
Braintrust.init(
|
|
124
|
+
default_project: "my-project",
|
|
125
|
+
auto_instrument: { only: [:openai] }
|
|
126
|
+
)
|
|
127
|
+
```
|
|
121
128
|
|
|
122
|
-
|
|
123
|
-
messages: [
|
|
124
|
-
{role: "system", content: "You are a helpful assistant."},
|
|
125
|
-
{role: "user", content: "Say hello!"}
|
|
126
|
-
],
|
|
127
|
-
model: "gpt-4o-mini",
|
|
128
|
-
max_tokens: 100
|
|
129
|
-
)
|
|
130
|
-
end
|
|
129
|
+
### Environment variables
|
|
131
130
|
|
|
132
|
-
|
|
131
|
+
| Variable | Description |
|
|
132
|
+
| --------------------------------- | ------------------------------------------------------------------------- |
|
|
133
|
+
| `BRAINTRUST_API_KEY` | Required. Your Braintrust API key |
|
|
134
|
+
| `BRAINTRUST_API_URL` | Braintrust API URL (default: `https://api.braintrust.dev`) |
|
|
135
|
+
| `BRAINTRUST_APP_URL` | Braintrust app URL (default: `https://www.braintrust.dev`) |
|
|
136
|
+
| `BRAINTRUST_AUTO_INSTRUMENT` | Set to `false` to disable auto-instrumentation |
|
|
137
|
+
| `BRAINTRUST_DEBUG` | Set to `true` to enable debug logging |
|
|
138
|
+
| `BRAINTRUST_DEFAULT_PROJECT` | Default project for spans |
|
|
139
|
+
| `BRAINTRUST_INSTRUMENT_EXCEPT` | Comma-separated list of integrations to skip |
|
|
140
|
+
| `BRAINTRUST_INSTRUMENT_ONLY` | Comma-separated list of integrations to enable (e.g., `openai,anthropic`) |
|
|
141
|
+
| `BRAINTRUST_ORG_NAME` | Organization name |
|
|
142
|
+
| `BRAINTRUST_OTEL_FILTER_AI_SPANS` | Set to `true` to only export AI-related spans |
|
|
133
143
|
|
|
134
|
-
|
|
144
|
+
## Tracing
|
|
135
145
|
|
|
136
|
-
|
|
137
|
-
|
|
146
|
+
### Supported providers
|
|
147
|
+
|
|
148
|
+
The SDK automatically instruments these LLM libraries:
|
|
138
149
|
|
|
139
|
-
|
|
150
|
+
| Provider | Gem | Versions | Integration Name | Examples |
|
|
151
|
+
| --------- | ------------- | -------- | ---------------- | ----------------------------------------------- |
|
|
152
|
+
| Anthropic | `anthropic` | >= 0.3.0 | `:anthropic` | [Link](./examples/contrib/anthropic.rb) |
|
|
153
|
+
| OpenAI | `openai` | >= 0.1.0 | `:openai` | [Link](./examples/contrib/openai.rb) |
|
|
154
|
+
| | `ruby-openai` | >= 7.0.0 | `:ruby_openai` | [Link](./examples/contrib/ruby-openai.rb) |
|
|
155
|
+
| Multiple | `ruby_llm` | >= 1.8.0 | `:ruby_llm` | [Link](./examples/contrib/ruby_llm.rb) |
|
|
156
|
+
|
|
157
|
+
### Manually applying instrumentation
|
|
158
|
+
|
|
159
|
+
For fine-grained control, disable auto-instrumentation and instrument specific clients:
|
|
140
160
|
|
|
141
161
|
```ruby
|
|
142
162
|
require "braintrust"
|
|
143
|
-
require "
|
|
144
|
-
|
|
145
|
-
Braintrust.init
|
|
163
|
+
require "openai"
|
|
146
164
|
|
|
147
|
-
|
|
165
|
+
Braintrust.init(auto_instrument: false) # Or BRAINTRUST_AUTO_INSTRUMENT=false
|
|
148
166
|
|
|
149
|
-
|
|
167
|
+
# Instrument all OpenAI clients
|
|
168
|
+
Braintrust.instrument!(:openai)
|
|
150
169
|
|
|
151
|
-
|
|
152
|
-
|
|
170
|
+
# OR instrument a single client
|
|
171
|
+
client = OpenAI::Client.new
|
|
172
|
+
Braintrust.instrument!(:openai, target: client)
|
|
173
|
+
```
|
|
153
174
|
|
|
154
|
-
|
|
155
|
-
root_span = span
|
|
175
|
+
### Creating custom spans
|
|
156
176
|
|
|
157
|
-
|
|
158
|
-
model: "claude-3-haiku-20240307",
|
|
159
|
-
max_tokens: 100,
|
|
160
|
-
system: "You are a helpful assistant.",
|
|
161
|
-
messages: [
|
|
162
|
-
{role: "user", content: "Say hello!"}
|
|
163
|
-
]
|
|
164
|
-
)
|
|
165
|
-
end
|
|
177
|
+
Wrap business logic in spans to see it in your traces:
|
|
166
178
|
|
|
167
|
-
|
|
179
|
+
```ruby
|
|
180
|
+
tracer = OpenTelemetry.tracer_provider.tracer("my-app")
|
|
168
181
|
|
|
169
|
-
|
|
182
|
+
tracer.in_span("process-request") do |span|
|
|
183
|
+
span.set_attribute("user.id", user_id)
|
|
170
184
|
|
|
171
|
-
|
|
185
|
+
# LLM calls inside here are automatically nested under this span
|
|
186
|
+
response = client.chat.completions.create(...)
|
|
187
|
+
end
|
|
172
188
|
```
|
|
173
189
|
|
|
174
|
-
###
|
|
190
|
+
### Attachments
|
|
191
|
+
|
|
192
|
+
Log binary data (images, PDFs, audio) in your traces:
|
|
175
193
|
|
|
176
194
|
```ruby
|
|
177
|
-
require "braintrust"
|
|
178
|
-
require "ruby_llm"
|
|
195
|
+
require "braintrust/trace/attachment"
|
|
179
196
|
|
|
180
|
-
Braintrust.
|
|
197
|
+
att = Braintrust::Trace::Attachment.from_file("image/png", "./photo.png")
|
|
181
198
|
|
|
182
|
-
#
|
|
183
|
-
|
|
199
|
+
# Use in messages (OpenAI/Anthropic format)
|
|
200
|
+
messages = [
|
|
201
|
+
{
|
|
202
|
+
role: "user",
|
|
203
|
+
content: [
|
|
204
|
+
{type: "text", text: "What's in this image?"},
|
|
205
|
+
att.to_h
|
|
206
|
+
]
|
|
207
|
+
}
|
|
208
|
+
]
|
|
184
209
|
|
|
185
|
-
|
|
186
|
-
|
|
210
|
+
# Log to span
|
|
211
|
+
span.set_attribute("braintrust.input_json", JSON.generate(messages))
|
|
212
|
+
```
|
|
187
213
|
|
|
188
|
-
|
|
189
|
-
root_span = span
|
|
214
|
+
Create attachments from various sources:
|
|
190
215
|
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
216
|
+
```ruby
|
|
217
|
+
Braintrust::Trace::Attachment.from_bytes("image/jpeg", image_data)
|
|
218
|
+
Braintrust::Trace::Attachment.from_file("application/pdf", "./doc.pdf")
|
|
219
|
+
Braintrust::Trace::Attachment.from_url("https://example.com/image.png")
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
See example: [trace_attachments.rb](./examples/trace/trace_attachments.rb)
|
|
194
223
|
|
|
195
|
-
|
|
224
|
+
### Viewing traces
|
|
196
225
|
|
|
197
|
-
|
|
226
|
+
Get a permalink to any span:
|
|
198
227
|
|
|
199
|
-
|
|
228
|
+
```ruby
|
|
229
|
+
tracer = OpenTelemetry.tracer_provider.tracer("my-app")
|
|
230
|
+
|
|
231
|
+
tracer.in_span("my-operation") do |span|
|
|
232
|
+
# your code here
|
|
233
|
+
puts "View trace at: #{Braintrust::Trace.permalink(span)}"
|
|
234
|
+
end
|
|
200
235
|
```
|
|
201
236
|
|
|
202
|
-
|
|
237
|
+
## Evals
|
|
203
238
|
|
|
204
|
-
|
|
239
|
+
Run evaluations against your AI systems:
|
|
205
240
|
|
|
206
241
|
```ruby
|
|
207
242
|
require "braintrust"
|
|
208
|
-
require "braintrust/trace/attachment"
|
|
209
243
|
|
|
210
244
|
Braintrust.init
|
|
211
245
|
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
)
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
messages = [
|
|
223
|
-
{
|
|
224
|
-
role: "user",
|
|
225
|
-
content: [
|
|
226
|
-
{type: "text", text: "What's in this image?"},
|
|
227
|
-
att.to_h # Converts to {"type" => "base64_attachment", "content" => "data:..."}
|
|
228
|
-
]
|
|
229
|
-
}
|
|
246
|
+
Braintrust::Eval.run(
|
|
247
|
+
project: "my-project",
|
|
248
|
+
experiment: "classifier-v1",
|
|
249
|
+
cases: [
|
|
250
|
+
{input: "apple", expected: "fruit"},
|
|
251
|
+
{input: "carrot", expected: "vegetable"}
|
|
252
|
+
],
|
|
253
|
+
task: ->(input) { classify(input) },
|
|
254
|
+
scorers: [
|
|
255
|
+
->(input, expected, output) { output == expected ? 1.0 : 0.0 }
|
|
230
256
|
]
|
|
257
|
+
)
|
|
258
|
+
```
|
|
231
259
|
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
260
|
+
### Datasets
|
|
261
|
+
|
|
262
|
+
Load test cases from a Braintrust dataset:
|
|
235
263
|
|
|
236
|
-
|
|
264
|
+
```ruby
|
|
265
|
+
Braintrust::Eval.run(
|
|
266
|
+
project: "my-project",
|
|
267
|
+
dataset: "my-dataset",
|
|
268
|
+
task: ->(input) { classify(input) },
|
|
269
|
+
scorers: [...]
|
|
270
|
+
)
|
|
237
271
|
```
|
|
238
272
|
|
|
239
|
-
|
|
273
|
+
### Remote scorers
|
|
274
|
+
|
|
275
|
+
Use scoring functions defined in Braintrust:
|
|
240
276
|
|
|
241
277
|
```ruby
|
|
242
|
-
|
|
243
|
-
|
|
278
|
+
Braintrust::Eval.run(
|
|
279
|
+
project: "my-project",
|
|
280
|
+
cases: [...],
|
|
281
|
+
task: ->(input) { ... },
|
|
282
|
+
scorers: [
|
|
283
|
+
Braintrust::Scorer.remote("my-project", "accuracy-scorer")
|
|
284
|
+
]
|
|
285
|
+
)
|
|
286
|
+
```
|
|
244
287
|
|
|
245
|
-
|
|
246
|
-
att = Braintrust::Trace::Attachment.from_file("application/pdf", "./doc.pdf")
|
|
288
|
+
See examples: [eval.rb](./examples/eval.rb), [dataset.rb](./examples/eval/dataset.rb), [remote_functions.rb](./examples/eval/remote_functions.rb)
|
|
247
289
|
|
|
248
|
-
|
|
249
|
-
att = Braintrust::Trace::Attachment.from_url("https://example.com/image.png")
|
|
250
|
-
```
|
|
290
|
+
## Documentation
|
|
251
291
|
|
|
252
|
-
|
|
292
|
+
- [Braintrust Documentation](https://www.braintrust.dev/docs)
|
|
293
|
+
- [API Reference](https://gemdocs.org/gems/braintrust/)
|
|
253
294
|
|
|
254
|
-
|
|
255
|
-
- **Tracing**: Automatic instrumentation for OpenAI and Anthropic API calls with OpenTelemetry
|
|
256
|
-
- **Datasets**: Manage and version your evaluation datasets
|
|
257
|
-
- **Experiments**: Track different versions and configurations of your AI systems
|
|
258
|
-
- **Observability**: Monitor your AI applications in production
|
|
295
|
+
## Troubleshooting
|
|
259
296
|
|
|
260
|
-
|
|
297
|
+
#### No traces after adding `require 'braintrust/setup'` to the Gemfile
|
|
261
298
|
|
|
262
|
-
|
|
299
|
+
First verify there are no errors in your logs after running with `BRAINTRUST_DEBUG=true` set.
|
|
263
300
|
|
|
264
|
-
|
|
265
|
-
- [trace.rb](./examples/trace.rb) - Manual span creation and tracing
|
|
266
|
-
- [openai.rb](./examples/openai.rb) - Automatically trace OpenAI API calls
|
|
267
|
-
- [alexrudall_openai.rb](./examples/alexrudall_openai.rb) - Automatically trace ruby-openai gem API calls
|
|
268
|
-
- [anthropic.rb](./examples/anthropic.rb) - Automatically trace Anthropic API calls
|
|
269
|
-
- [ruby_llm.rb](./examples/ruby_llm.rb) - Automatically trace RubyLLM API calls
|
|
270
|
-
- [trace/trace_attachments.rb](./examples/trace/trace_attachments.rb) - Log attachments (images, PDFs) in traces
|
|
271
|
-
- [eval/dataset.rb](./examples/eval/dataset.rb) - Run evaluations using datasets stored in Braintrust
|
|
272
|
-
- [eval/remote_functions.rb](./examples/eval/remote_functions.rb) - Use remote scoring functions
|
|
301
|
+
Your application needs the following for this to work:
|
|
273
302
|
|
|
274
|
-
|
|
303
|
+
```ruby
|
|
304
|
+
require 'bundler/setup'
|
|
305
|
+
Bundler.require
|
|
306
|
+
```
|
|
275
307
|
|
|
276
|
-
|
|
277
|
-
|
|
308
|
+
It is present by default in Rails applications, but may not be in Sinatra, Rack, or other applications.
|
|
309
|
+
|
|
310
|
+
Alternatively, you can add `require 'braintrust/setup'` to your application initialization files.
|
|
278
311
|
|
|
279
312
|
## Contributing
|
|
280
313
|
|
|
281
|
-
See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and
|
|
314
|
+
See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and guidelines.
|
|
282
315
|
|
|
283
316
|
## License
|
|
284
317
|
|
|
285
|
-
|
|
318
|
+
Apache License 2.0 - see [LICENSE](./LICENSE).
|
data/exe/braintrust
ADDED
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
# Braintrust CLI - Auto-instrument Ruby applications
|
|
5
|
+
#
|
|
6
|
+
# Usage:
|
|
7
|
+
# braintrust exec -- ruby app.rb
|
|
8
|
+
# braintrust exec -- bundle exec rails server
|
|
9
|
+
# braintrust exec --only openai,anthropic -- ruby app.rb
|
|
10
|
+
|
|
11
|
+
require "optparse"
|
|
12
|
+
|
|
13
|
+
module Braintrust
|
|
14
|
+
module CLI
|
|
15
|
+
class << self
|
|
16
|
+
def run(args)
|
|
17
|
+
command = parse_args(args)
|
|
18
|
+
|
|
19
|
+
case command
|
|
20
|
+
when :exec
|
|
21
|
+
exec_command
|
|
22
|
+
when :help, nil
|
|
23
|
+
print_help
|
|
24
|
+
exit((command == :help) ? 0 : 1)
|
|
25
|
+
else
|
|
26
|
+
puts "Unknown command: #{command}"
|
|
27
|
+
print_help
|
|
28
|
+
exit 1
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
private
|
|
33
|
+
|
|
34
|
+
def parse_args(args)
|
|
35
|
+
@options = {}
|
|
36
|
+
@remaining_args = []
|
|
37
|
+
|
|
38
|
+
# Find -- separator
|
|
39
|
+
separator_index = args.index("--")
|
|
40
|
+
if separator_index
|
|
41
|
+
to_parse = args[0...separator_index]
|
|
42
|
+
@remaining_args = args[(separator_index + 1)..]
|
|
43
|
+
else
|
|
44
|
+
to_parse = args.dup
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
parser = OptionParser.new do |opts|
|
|
48
|
+
opts.banner = "Usage: braintrust <command> [options] -- COMMAND"
|
|
49
|
+
|
|
50
|
+
opts.on("--only INTEGRATIONS", "Only instrument these (comma-separated)") do |v|
|
|
51
|
+
@options[:only] = v
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
opts.on("--except INTEGRATIONS", "Skip these integrations (comma-separated)") do |v|
|
|
55
|
+
@options[:except] = v
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
opts.on("-h", "--help", "Show this help") do
|
|
59
|
+
return :help
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
opts.on("-v", "--version", "Show version") do
|
|
63
|
+
require_relative "../lib/braintrust/version"
|
|
64
|
+
puts "braintrust #{Braintrust::VERSION}"
|
|
65
|
+
exit 0
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
parser.parse!(to_parse)
|
|
70
|
+
|
|
71
|
+
return nil if to_parse.empty?
|
|
72
|
+
to_parse.first.to_sym
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def exec_command
|
|
76
|
+
if @remaining_args.empty?
|
|
77
|
+
puts "Error: No command specified after --"
|
|
78
|
+
puts "Usage: braintrust exec [options] -- COMMAND"
|
|
79
|
+
exit 1
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# Set environment variables for filtering
|
|
83
|
+
ENV["BRAINTRUST_INSTRUMENT_ONLY"] = @options[:only] if @options[:only]
|
|
84
|
+
ENV["BRAINTRUST_INSTRUMENT_EXCEPT"] = @options[:except] if @options[:except]
|
|
85
|
+
|
|
86
|
+
# Inject auto-instrument via RUBYOPT
|
|
87
|
+
set_rubyopt!
|
|
88
|
+
|
|
89
|
+
# Execute the command with error handling
|
|
90
|
+
exec_with_error_handling(@remaining_args)
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def rubyopts
|
|
94
|
+
["-rbraintrust/setup"]
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
def set_rubyopt!
|
|
98
|
+
existing = ENV["RUBYOPT"]
|
|
99
|
+
ENV["RUBYOPT"] = existing ? "#{existing} #{rubyopts.join(" ")}" : rubyopts.join(" ")
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
def exec_with_error_handling(args)
|
|
103
|
+
Kernel.exec(*args)
|
|
104
|
+
rescue Errno::ENOENT => e
|
|
105
|
+
Kernel.warn "braintrust exec failed: #{e.class.name} #{e.message}"
|
|
106
|
+
Kernel.exit 127
|
|
107
|
+
rescue Errno::EACCES, Errno::ENOEXEC => e
|
|
108
|
+
Kernel.warn "braintrust exec failed: #{e.class.name} #{e.message}"
|
|
109
|
+
Kernel.exit 126
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
def print_help
|
|
113
|
+
puts <<~HELP
|
|
114
|
+
Braintrust CLI - Auto-instrument Ruby applications
|
|
115
|
+
|
|
116
|
+
Usage:
|
|
117
|
+
braintrust exec [options] -- COMMAND
|
|
118
|
+
|
|
119
|
+
Commands:
|
|
120
|
+
exec Run a command with auto-instrumentation enabled
|
|
121
|
+
|
|
122
|
+
Options:
|
|
123
|
+
--only INTEGRATIONS Only instrument these (comma-separated)
|
|
124
|
+
--except INTEGRATIONS Skip these integrations (comma-separated)
|
|
125
|
+
-h, --help Show this help
|
|
126
|
+
-v, --version Show version
|
|
127
|
+
|
|
128
|
+
Examples:
|
|
129
|
+
braintrust exec -- ruby app.rb
|
|
130
|
+
braintrust exec -- rails server
|
|
131
|
+
braintrust exec --only openai -- ruby app.rb
|
|
132
|
+
|
|
133
|
+
Environment Variables:
|
|
134
|
+
BRAINTRUST_API_KEY API key for Braintrust
|
|
135
|
+
BRAINTRUST_INSTRUMENT_ONLY Comma-separated whitelist
|
|
136
|
+
BRAINTRUST_INSTRUMENT_EXCEPT Comma-separated blacklist
|
|
137
|
+
HELP
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
end
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
Braintrust::CLI.run(ARGV)
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Backward compatibility shim for the old Anthropic integration API.
|
|
4
|
+
# This file now just delegates to the new API.
|
|
5
|
+
|
|
6
|
+
require_relative "../../../braintrust"
|
|
7
|
+
|
|
8
|
+
module Braintrust
|
|
9
|
+
module Trace
|
|
10
|
+
module Anthropic
|
|
11
|
+
# Wrap an Anthropic::Client to automatically create spans for messages.
|
|
12
|
+
# This is the legacy API - delegates to the new contrib framework.
|
|
13
|
+
#
|
|
14
|
+
# @param client [Anthropic::Client] the Anthropic client to wrap
|
|
15
|
+
# @param tracer_provider [OpenTelemetry::SDK::Trace::TracerProvider] the tracer provider
|
|
16
|
+
# @return [Anthropic::Client] the wrapped client
|
|
17
|
+
def self.wrap(client, tracer_provider: nil)
|
|
18
|
+
Log.warn("Braintrust::Trace::Anthropic.wrap() is deprecated and will be removed in a future version: use Braintrust.instrument!() instead.")
|
|
19
|
+
Braintrust.instrument!(:anthropic, target: client, tracer_provider: tracer_provider)
|
|
20
|
+
client
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
end
|