ruby_llm-agent-evaluations 0.2.1 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -0
- data/README.md +51 -13
- data/app/models/ruby_llm/evaluations/application_record.rb +0 -6
- data/app/models/ruby_llm/evaluations/case_result.rb +2 -0
- data/app/models/ruby_llm/evaluations/run.rb +2 -0
- data/db/migrate/20261011000000_create_agent_evaluation_history.rb +23 -22
- data/lib/ruby_llm/evaluations/cli.rb +3 -3
- data/lib/ruby_llm/evaluations/dashboard/layout.erb +1 -1
- data/lib/ruby_llm/evaluations/runner.rb +0 -1
- data/lib/ruby_llm/evaluations/version.rb +1 -1
- data/lib/ruby_llm/evaluations.rb +1 -2
- metadata +2 -23
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 27494e2aa94d8c50b5f680ec367b98a0cc4cb04a4d7454858a64c7dd5e7b76d5
|
|
4
|
+
data.tar.gz: 664eb1507d3a622c193bd4a50d11be565c3d9694dd14748f1f8b5241caa7ad92
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e1103c795eef6354e35a3e547cddba3334d4e88e379fb6cf260f0f396c93ee0db5c05ab834c263938c5b884036d7f017e7c4768780f9c7f66f1e451241a531ac
|
|
7
|
+
data.tar.gz: d1ab9f91ce0b83ac6522379bd20053588c265e417c69ddc53be24d859c847f0f7fc120a670acdfda243c99789faee3461279c0835d1a00ceecc7836d91cde29d
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,12 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.2.2
|
|
4
|
+
|
|
5
|
+
- Support PostgreSQL, SQLite, and MySQL 8.0+ through the ActiveRecord-selected adapter and one migration creating exactly three evaluation tables. Preserve UUID run IDs, append-only evidence, foreign keys, and chronological baseline ordering. MySQL uses binary, non-padding collation to preserve exact agent, case, and metric identities.
|
|
6
|
+
- Remove the PostgreSQL-only migration guard and mandatory `pg` dependency. Consumers add their chosen database driver; Rails continues using its normal host pool and migration tasks, while the standalone CLI defaults to `sqlite3:ruby_llm_evaluations.sqlite3`.
|
|
7
|
+
- Keep existing applied PostgreSQL migrations and saved history unchanged. Rails migrations already copied into a host are not automatically rewritten; new hosts install the current engine migration. No schema reset, history deletion, or legacy upgrade is required for portability.
|
|
8
|
+
- Add real SQLite, PostgreSQL, and MySQL consumer coverage and document driver selection, local quick starts, and isolated cross-adapter development checks.
|
|
9
|
+
|
|
3
10
|
## 0.2.1
|
|
4
11
|
|
|
5
12
|
- Simplify new-project setup to one migration creating only runs, case results, and metric results, with `minimum_scores` from the start. Remove legacy schema detection, payload copying, temporary migration models, and the intermediate column-rename migration. Legacy database upgrades are no longer supported.
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# RubyLLM Agent Evaluations
|
|
2
2
|
|
|
3
|
-
Write questions and expected answers in Ruby, declare what matters, and keep evaluation evidence in PostgreSQL. Compare compatible runs and inspect answers, metrics, errors, and action traces in a read-only dashboard.
|
|
3
|
+
Write questions and expected answers in Ruby, declare what matters, and keep evaluation evidence in PostgreSQL, SQLite, or MySQL through ActiveRecord. Compare compatible runs and inspect answers, metrics, errors, and action traces in a read-only dashboard.
|
|
4
4
|
|
|
5
5
|
Works with any callable agent. RubyLLM is optional; Rails applications use named ActiveRecord models sharing their normal host connection pool.
|
|
6
6
|
|
|
@@ -10,7 +10,7 @@ Install the package as `ruby_llm-agent-evaluations`:
|
|
|
10
10
|
gem "ruby_llm-agent-evaluations", "~> 0.2", require: "ruby_llm/evaluations"
|
|
11
11
|
~~~
|
|
12
12
|
|
|
13
|
-
The executable is `ruby_llm-agent-evaluations`. The Ruby API remains `RubyLLM::Evaluations` with `require "ruby_llm/evaluations"`; Rails migration tasks and
|
|
13
|
+
The executable is `ruby_llm-agent-evaluations`. The Ruby API remains `RubyLLM::Evaluations` with `require "ruby_llm/evaluations"`; Rails migration tasks and evaluation table names are unchanged, preserving existing history. The separately owned `ruby_llm-evaluations` package on RubyGems is not this project. The GitHub repository keeps its existing name.
|
|
14
14
|
|
|
15
15
|
## Question-first specifications
|
|
16
16
|
|
|
@@ -44,7 +44,7 @@ Questions accept optional name:, id:, and expected_actions:. Names default to qu
|
|
|
44
44
|
|
|
45
45
|
## Strict API replay
|
|
46
46
|
|
|
47
|
-
Declare fixtures inside a question block to exercise real HTTP clients without contacting the API. This complete spec needs the ordinary
|
|
47
|
+
Declare fixtures inside a question block to exercise real HTTP clients without contacting the API. This complete spec needs the ordinary database setup below, but no provider credentials:
|
|
48
48
|
|
|
49
49
|
~~~ruby
|
|
50
50
|
require "net/http"
|
|
@@ -82,7 +82,7 @@ Absolute `body_file:` paths are also accepted; prefer spec-relative paths for po
|
|
|
82
82
|
|
|
83
83
|
Replay is strict/offline by default: undeclared network requests fail. For live model requests made by the agent itself, explicitly use `api_interactions(allow_hosts: ["api.openai.com"]) do ... end`. Allowances match exact hosts, not suffixes or wildcards, and may incur provider charges. An allowed host never bypasses a declared fixture request. Judge calls need no replay allowance because judging runs outside replay. There is no recording mode.
|
|
84
84
|
|
|
85
|
-
|
|
85
|
+
Case results persist `response_metadata["api_replay"]` on success and failure: expected interactions, observed match statuses, readable mismatch differences, and missing requests. Authentication/credential fields are redacted from replay evidence. Still use synthetic fixtures: never commit credentials or sensitive payloads, and protect persisted prompts, answers, and reports.
|
|
86
86
|
|
|
87
87
|
Replay evidence uses the existing response-metadata display in reports and the dashboard; JSON reports include the persisted `response_metadata.api_replay` payload. No separate evidence store or migration is required.
|
|
88
88
|
|
|
@@ -139,6 +139,16 @@ Add the gem to your application's Gemfile. During local development, add `path:
|
|
|
139
139
|
gem "ruby_llm-agent-evaluations", "~> 0.2", require: "ruby_llm/evaluations"
|
|
140
140
|
~~~
|
|
141
141
|
|
|
142
|
+
Persistence uses the adapter selected by ActiveRecord: PostgreSQL, SQLite, or MySQL 8.0+. The gem does not require `pg` or another database driver; add your chosen driver to the application's Gemfile (or keep the one already used by your Rails host):
|
|
143
|
+
|
|
144
|
+
~~~ruby
|
|
145
|
+
gem "sqlite3" # SQLite
|
|
146
|
+
# gem "pg" # PostgreSQL instead
|
|
147
|
+
# gem "mysql2" # MySQL instead
|
|
148
|
+
~~~
|
|
149
|
+
|
|
150
|
+
Rails uses its existing database configuration and normal connection pool; no separate evaluation connection or storage configuration is needed. MySQL tables use `utf8mb4_0900_bin` so case, accents, and trailing spaces remain distinct in identifiers and baseline matching.
|
|
151
|
+
|
|
142
152
|
Install the engine migrations into the host application, then migrate normally:
|
|
143
153
|
|
|
144
154
|
~~~sh
|
|
@@ -148,7 +158,9 @@ bin/rails db:migrate
|
|
|
148
158
|
|
|
149
159
|
For standalone use, run `ruby_llm-agent-evaluations db:migrate` once before the first evaluation. Rails applications install the single engine migration and run `bin/rails db:migrate` as shown above.
|
|
150
160
|
|
|
151
|
-
|
|
161
|
+
The single migration creates exactly three evaluation tables: `ruby_llm_evaluations_runs`, `ruby_llm_evaluations_case_results`, and `ruby_llm_evaluations_metric_results`. It creates `minimum_scores` directly, with the foreign keys and indexes needed for history and baseline comparisons. No legacy tables, payload imports, or custom schema-version tables are required or inspected. ActiveRecord manages its ordinary migration bookkeeping. Execution uses the host connection pool and does not establish, migrate, or disconnect it.
|
|
162
|
+
|
|
163
|
+
Existing PostgreSQL installations continue using their applied migration and saved history unchanged; no schema reset or history deletion is needed. Rails migrations already copied into a host are not automatically updated when the gem changes. New hosts should install the current engine migration; database portability does not require a legacy upgrade migration.
|
|
152
164
|
|
|
153
165
|
~~~ruby
|
|
154
166
|
# config/routes.rb
|
|
@@ -161,12 +173,13 @@ Visit /evaluations. Assets, filters, detail/baseline links, and downloads respec
|
|
|
161
173
|
|
|
162
174
|
## Offline quick start: exact match, not semantic judging
|
|
163
175
|
|
|
164
|
-
Requirements: Ruby 3.2+,
|
|
176
|
+
Requirements: Ruby 3.2+, Bundler, and your chosen database driver (`sqlite3` for this quick start). Rails and RubyLLM are optional. This deterministic local demo runs without API keys and is deliberately separate from semantic judging. It performs a real local answer lookup and records that action; exact match is not a semantic judge.
|
|
177
|
+
|
|
178
|
+
From this checkout, the development bundle supplies the drivers. Standalone consumers add the gem and their chosen driver to their own Gemfile, then use `bundle exec ruby_llm-agent-evaluations` in place of the checkout command below.
|
|
165
179
|
|
|
166
180
|
~~~sh
|
|
167
181
|
bundle install
|
|
168
|
-
|
|
169
|
-
export DATABASE_URL='postgresql://postgres@127.0.0.1:55432/ruby_llm_evaluations'
|
|
182
|
+
unset DATABASE_URL # Defaults to sqlite3:ruby_llm_evaluations.sqlite3 in the current directory.
|
|
170
183
|
|
|
171
184
|
# Migrate once before executing specs. Runs never create tables implicitly.
|
|
172
185
|
bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations db:migrate
|
|
@@ -179,7 +192,28 @@ bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations history
|
|
|
179
192
|
bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations dashboard
|
|
180
193
|
~~~
|
|
181
194
|
|
|
182
|
-
Open **http://127.0.0.1:4567**; Ctrl-C stops the server.
|
|
195
|
+
Open **http://127.0.0.1:4567**; Ctrl-C stops the server. The local SQLite file preserves evidence across restarts. Set DATABASE_URL to use another database or SQLite file; migrations remain an explicit setup step.
|
|
196
|
+
|
|
197
|
+
### PostgreSQL or MySQL instead
|
|
198
|
+
|
|
199
|
+
Install `pg` for PostgreSQL or `mysql2` for MySQL in your consumer Gemfile. For local services from this checkout:
|
|
200
|
+
|
|
201
|
+
~~~sh
|
|
202
|
+
docker compose up -d postgres mysql
|
|
203
|
+
|
|
204
|
+
# PostgreSQL: the development database is provisioned by Compose.
|
|
205
|
+
export DATABASE_URL='postgresql://postgres@127.0.0.1:55432/ruby_llm_evaluations'
|
|
206
|
+
|
|
207
|
+
# Or MySQL: create a development database, then select it.
|
|
208
|
+
docker compose exec -T mysql mysql -uroot -pdev-only -e 'CREATE DATABASE IF NOT EXISTS ruby_llm_evaluations;'
|
|
209
|
+
export DATABASE_URL='mysql2://root:dev-only@127.0.0.1:53306/ruby_llm_evaluations'
|
|
210
|
+
|
|
211
|
+
# With the selected URL, migrate once, then run the same demo.
|
|
212
|
+
bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations db:migrate
|
|
213
|
+
bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations run examples/support_eval.rb
|
|
214
|
+
~~~
|
|
215
|
+
|
|
216
|
+
Compose binds passwordless PostgreSQL to loopback port 55432 and MySQL 8.4 with the development-only root password `dev-only` to loopback port 53306. **Do not use these credentials/configurations in production.** Named volumes preserve database history; `docker compose stop` does not delete it. For your own service, set DATABASE_URL to an existing database with an appropriate application account.
|
|
183
217
|
|
|
184
218
|
The offline example produces the following evidence:
|
|
185
219
|
|
|
@@ -193,7 +227,7 @@ The regressed agent answers the return question with 7 days instead of 30 and lo
|
|
|
193
227
|
The standalone CLI alone establishes ActiveRecord::Base's connection. Standalone library callers configure it themselves:
|
|
194
228
|
|
|
195
229
|
~~~ruby
|
|
196
|
-
ActiveRecord::Base.establish_connection(ENV.fetch("DATABASE_URL"))
|
|
230
|
+
ActiveRecord::Base.establish_connection(ENV.fetch("DATABASE_URL", "sqlite3:ruby_llm_evaluations.sqlite3"))
|
|
197
231
|
RubyLLM::Evaluations.migrate! # Once during database setup, not per run.
|
|
198
232
|
~~~
|
|
199
233
|
|
|
@@ -289,20 +323,23 @@ With no judge supplied, the actual dashboard records answers but leaves quality
|
|
|
289
323
|
|
|
290
324
|
## Development and packaging
|
|
291
325
|
|
|
292
|
-
Integration tests always: behavior changes must update end-to-end coverage. PostgreSQL
|
|
326
|
+
Integration tests always: behavior changes must update end-to-end coverage. Exercise real SQLite, PostgreSQL, and MySQL persistence and consumer smoke flows; integration tests must never silently skip an unavailable database. These are development-check requirements, not a requirement that consumers install every driver or run every server.
|
|
293
327
|
|
|
294
328
|
~~~sh
|
|
329
|
+
bundle install
|
|
330
|
+
docker compose up -d postgres mysql
|
|
295
331
|
docker compose exec -T postgres createdb -U postgres ruby_llm_evaluations_test
|
|
296
332
|
export TEST_DATABASE_URL='postgresql://postgres@127.0.0.1:55432/ruby_llm_evaluations_test'
|
|
333
|
+
export TEST_MYSQL_DATABASE_URL='mysql2://root:dev-only@127.0.0.1:53306/mysql'
|
|
297
334
|
bundle exec rake integration
|
|
298
335
|
bundle exec rake test
|
|
299
336
|
|
|
300
337
|
gem build ruby_llm-agent-evaluations.gemspec
|
|
301
|
-
gem install ./ruby_llm-agent-evaluations-0.2.
|
|
338
|
+
gem install ./ruby_llm-agent-evaluations-0.2.2.gem
|
|
302
339
|
ruby_llm-agent-evaluations --help
|
|
303
340
|
~~~
|
|
304
341
|
|
|
305
|
-
Tests isolate PostgreSQL schemas rather than clear development history
|
|
342
|
+
Tests isolate temporary SQLite files, PostgreSQL schemas, and MySQL databases rather than clear development history. The PostgreSQL test role needs schema-creation privileges; the MySQL test account needs database-creation/deletion privileges on the test server. SQLite needs no service or test URL. Local HTTP provider fixtures verify real adapter transport/orchestration, not model reasoning. Default tests never call paid providers. For UI changes inspect desktop/narrow layouts, keyboard access, filters, chart navigation, and downloads. Shared engineering guidance and specialist skills live in [.agents/](.agents/AGENTS.md).
|
|
306
343
|
|
|
307
344
|
## Publishing
|
|
308
345
|
|
|
@@ -310,6 +347,7 @@ The release version is defined in `lib/ruby_llm/evaluations/version.rb`. Bump it
|
|
|
310
347
|
|
|
311
348
|
~~~sh
|
|
312
349
|
export TEST_DATABASE_URL='postgresql://postgres@127.0.0.1:55432/ruby_llm_evaluations_test'
|
|
350
|
+
export TEST_MYSQL_DATABASE_URL='mysql2://root:dev-only@127.0.0.1:53306/mysql'
|
|
313
351
|
bundle install
|
|
314
352
|
bundle exec ruby script/publish --dry-run
|
|
315
353
|
|
|
@@ -7,12 +7,6 @@ module RubyLLM
|
|
|
7
7
|
class ApplicationRecord < ::ActiveRecord::Base
|
|
8
8
|
self.abstract_class = true
|
|
9
9
|
|
|
10
|
-
def self.require_postgresql!
|
|
11
|
-
unless connection_pool.db_config.adapter == "postgresql"
|
|
12
|
-
raise Error, "evaluation history requires PostgreSQL"
|
|
13
|
-
end
|
|
14
|
-
end
|
|
15
|
-
|
|
16
10
|
# Completed evidence is append-only through the model API.
|
|
17
11
|
def readonly?
|
|
18
12
|
persisted?
|
|
@@ -4,6 +4,8 @@ module RubyLLM
|
|
|
4
4
|
module Evaluations
|
|
5
5
|
class CaseResult < ApplicationRecord
|
|
6
6
|
self.table_name = "ruby_llm_evaluations_case_results"
|
|
7
|
+
attribute :metadata, default: -> { {} }
|
|
8
|
+
attribute :minimum_scores, default: -> { {} }
|
|
7
9
|
|
|
8
10
|
belongs_to :run, inverse_of: :case_results, class_name: "RubyLLM::Evaluations::Run"
|
|
9
11
|
has_many :metric_results, -> { order(:position) }, inverse_of: :case_result,
|
|
@@ -7,6 +7,8 @@ module RubyLLM
|
|
|
7
7
|
module Evaluations
|
|
8
8
|
class Run < ApplicationRecord
|
|
9
9
|
self.table_name = "ruby_llm_evaluations_runs"
|
|
10
|
+
self.primary_key = "id"
|
|
11
|
+
attribute :metadata, default: -> { {} }
|
|
10
12
|
|
|
11
13
|
has_many :case_results, -> { order(:position) }, inverse_of: :run,
|
|
12
14
|
class_name: "RubyLLM::Evaluations::CaseResult", dependent: :restrict_with_exception
|
|
@@ -6,51 +6,52 @@ class CreateAgentEvaluationHistory < ActiveRecord::Migration[7.1]
|
|
|
6
6
|
METRICS = "ruby_llm_evaluations_metric_results"
|
|
7
7
|
|
|
8
8
|
def change
|
|
9
|
-
|
|
9
|
+
json_type = connection.adapter_name == "PostgreSQL" ? :jsonb : :json
|
|
10
|
+
table_options = connection.adapter_name == "Mysql2" ? { charset: "utf8mb4", collation: "utf8mb4_0900_bin" } : {}
|
|
10
11
|
|
|
11
|
-
create_table RUNS,
|
|
12
|
-
table.
|
|
12
|
+
create_table RUNS, primary_key: :sequence, **table_options do |table|
|
|
13
|
+
table.string :id, null: false
|
|
13
14
|
table.string :suite, null: false
|
|
14
15
|
table.string :agent, null: false
|
|
15
16
|
table.string :version, null: false, default: "1"
|
|
16
|
-
table.string :suite_digest, null: false
|
|
17
|
-
table.
|
|
18
|
-
table.
|
|
17
|
+
table.string :suite_digest, limit: 64, null: false
|
|
18
|
+
table.datetime :started_at, null: false, precision: 6
|
|
19
|
+
table.datetime :finished_at, null: false, precision: 6
|
|
19
20
|
table.string :status, null: false
|
|
20
|
-
table.
|
|
21
|
-
table.
|
|
21
|
+
table.column :metadata, json_type, null: false
|
|
22
|
+
table.column :summary, json_type, null: false
|
|
22
23
|
table.timestamps precision: 6
|
|
23
24
|
end
|
|
24
|
-
add_index RUNS, :
|
|
25
|
+
add_index RUNS, :id, unique: true
|
|
25
26
|
add_index RUNS, [:started_at, :sequence], order: { started_at: :desc, sequence: :desc }, name: "rle_runs_recent"
|
|
26
27
|
add_index RUNS, [:suite, :agent, :suite_digest, :started_at, :sequence],
|
|
27
28
|
order: { started_at: :desc, sequence: :desc }, name: "rle_runs_comparable"
|
|
28
29
|
|
|
29
|
-
create_table CASES do |table|
|
|
30
|
-
table.references :run, type: :string, null: false, foreign_key: { to_table: RUNS }
|
|
30
|
+
create_table CASES, **table_options do |table|
|
|
31
|
+
table.references :run, type: :string, null: false, foreign_key: { to_table: RUNS, primary_key: :id }
|
|
31
32
|
table.integer :position, null: false
|
|
32
33
|
table.string :case_id, null: false
|
|
33
34
|
table.text :name, null: false
|
|
34
|
-
table.
|
|
35
|
-
table.
|
|
36
|
-
table.
|
|
37
|
-
table.
|
|
38
|
-
table.
|
|
39
|
-
table.
|
|
35
|
+
table.column :question, json_type
|
|
36
|
+
table.column :expected_answer, json_type
|
|
37
|
+
table.column :expected_actions, json_type
|
|
38
|
+
table.column :minimum_scores, json_type, null: false
|
|
39
|
+
table.column :metadata, json_type, null: false
|
|
40
|
+
table.column :output, json_type
|
|
40
41
|
table.string :status, null: false
|
|
41
42
|
table.float :duration_ms, null: false
|
|
42
43
|
table.bigint :input_tokens
|
|
43
44
|
table.bigint :output_tokens
|
|
44
45
|
table.float :cost
|
|
45
|
-
table.
|
|
46
|
-
table.
|
|
47
|
-
table.
|
|
46
|
+
table.column :actions, json_type
|
|
47
|
+
table.column :response_metadata, json_type
|
|
48
|
+
table.column :error_details, json_type
|
|
48
49
|
table.timestamps precision: 6
|
|
49
50
|
end
|
|
50
51
|
add_index CASES, [:run_id, :position], unique: true, name: "rle_case_order"
|
|
51
52
|
add_index CASES, [:run_id, :case_id], unique: true, name: "rle_case_identity"
|
|
52
53
|
|
|
53
|
-
create_table METRICS do |table|
|
|
54
|
+
create_table METRICS, **table_options do |table|
|
|
54
55
|
table.references :case_result, null: false, foreign_key: { to_table: CASES }
|
|
55
56
|
table.integer :position, null: false
|
|
56
57
|
table.string :name, null: false
|
|
@@ -58,7 +59,7 @@ class CreateAgentEvaluationHistory < ActiveRecord::Migration[7.1]
|
|
|
58
59
|
table.float :score
|
|
59
60
|
table.boolean :passed, null: false
|
|
60
61
|
table.text :reason
|
|
61
|
-
table.
|
|
62
|
+
table.column :error_details, json_type
|
|
62
63
|
table.timestamps precision: 6
|
|
63
64
|
end
|
|
64
65
|
add_index METRICS, [:case_result_id, :position], unique: true, name: "rle_metric_order"
|
|
@@ -32,7 +32,7 @@ module RubyLLM
|
|
|
32
32
|
begin
|
|
33
33
|
::ActiveRecord::Base.establish_connection(options[:database_url])
|
|
34
34
|
rescue ArgumentError, URI::InvalidURIError
|
|
35
|
-
raise Error, "Invalid database configuration. Set DATABASE_URL to
|
|
35
|
+
raise Error, "Invalid database configuration. Set DATABASE_URL to an ActiveRecord connection URL."
|
|
36
36
|
end
|
|
37
37
|
if command == "db:migrate"
|
|
38
38
|
raise ArgumentError, "db:migrate takes no positional arguments" unless args.empty?
|
|
@@ -50,7 +50,7 @@ module RubyLLM
|
|
|
50
50
|
when "dashboard" then dashboard(args, options)
|
|
51
51
|
end
|
|
52
52
|
rescue ::ActiveRecord::ActiveRecordError
|
|
53
|
-
@err.puts("Database configuration error. Check DATABASE_URL and
|
|
53
|
+
@err.puts("Database configuration error. Check DATABASE_URL and database availability; run ruby_llm-agent-evaluations db:migrate to create evaluation tables.")
|
|
54
54
|
2
|
|
55
55
|
rescue OptionParser::ParseError, ArgumentError, Error, SystemCallError, JSON::ParserError, SyntaxError => exception
|
|
56
56
|
@err.puts("#{exception.class}: #{exception.message}")
|
|
@@ -64,7 +64,7 @@ module RubyLLM
|
|
|
64
64
|
usage = { "run" => "run SPEC.rb", "history" => "history", "report" => "report [RUN_ID]",
|
|
65
65
|
"dashboard" => "dashboard", "db:migrate" => "db:migrate" }.fetch(command)
|
|
66
66
|
parser.banner = "Usage: ruby_llm-agent-evaluations #{usage} [options]"
|
|
67
|
-
parser.on("--database-url URL", "
|
|
67
|
+
parser.on("--database-url URL", "Database URL (default: DATABASE_URL or local SQLite file)") { |value| options[:database_url] = value }
|
|
68
68
|
if %w[run report].include?(command)
|
|
69
69
|
parser.on("--format FORMAT", FORMATS, "text, markdown, html, or json") { |value| options[:format] = value }
|
|
70
70
|
parser.on("--output PATH", "Write report to a file") { |value| options[:output] = value }
|
|
@@ -13,6 +13,6 @@
|
|
|
13
13
|
<nav aria-label="Main navigation"><a href="<%= ui.h(ui.path("/")) %>">Run history</a><span class="local-badge">Local workspace</span></nav>
|
|
14
14
|
</div></header>
|
|
15
15
|
<main id="main" tabindex="-1"><%= content %></main>
|
|
16
|
-
<footer class="site-footer"><span>Evaluations as code. Evidence over intuition.</span><span>Read-only ·
|
|
16
|
+
<footer class="site-footer"><span>Evaluations as code. Evidence over intuition.</span><span>Read-only · Persistent history · No external assets</span></footer>
|
|
17
17
|
</body>
|
|
18
18
|
</html>
|
|
@@ -5,7 +5,6 @@ module RubyLLM
|
|
|
5
5
|
class Runner
|
|
6
6
|
def run(suite, metadata: {})
|
|
7
7
|
suite.validate!
|
|
8
|
-
Run.require_postgresql!
|
|
9
8
|
raise Error, "Apply evaluation migrations before running a suite" unless Run.table_exists?
|
|
10
9
|
raise ArgumentError, "run metadata must be a Hash" unless metadata.is_a?(Hash)
|
|
11
10
|
run_metadata = Data.json_value(metadata)
|
data/lib/ruby_llm/evaluations.rb
CHANGED
|
@@ -21,7 +21,7 @@ require_relative "evaluations/dashboard"
|
|
|
21
21
|
|
|
22
22
|
module RubyLLM
|
|
23
23
|
module Evaluations
|
|
24
|
-
DEFAULT_DATABASE_URL = "
|
|
24
|
+
DEFAULT_DATABASE_URL = "sqlite3:ruby_llm_evaluations.sqlite3"
|
|
25
25
|
MIGRATIONS_PATH = File.expand_path("../../db/migrate", __dir__).freeze
|
|
26
26
|
autoload :Engine, File.expand_path("evaluations/engine", __dir__)
|
|
27
27
|
|
|
@@ -38,7 +38,6 @@ module RubyLLM
|
|
|
38
38
|
|
|
39
39
|
# Standalone setup uses Rails migrations; a host application uses db:migrate.
|
|
40
40
|
def self.migrate!
|
|
41
|
-
ApplicationRecord.require_postgresql!
|
|
42
41
|
::ActiveRecord::MigrationContext.new([MIGRATIONS_PATH]).migrate
|
|
43
42
|
[Run, CaseResult, MetricResult].each(&:reset_column_information)
|
|
44
43
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ruby_llm-agent-evaluations
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- RubyLLM Evaluations contributors
|
|
@@ -29,26 +29,6 @@ dependencies:
|
|
|
29
29
|
- - "<"
|
|
30
30
|
- !ruby/object:Gem::Version
|
|
31
31
|
version: '9'
|
|
32
|
-
- !ruby/object:Gem::Dependency
|
|
33
|
-
name: pg
|
|
34
|
-
requirement: !ruby/object:Gem::Requirement
|
|
35
|
-
requirements:
|
|
36
|
-
- - ">="
|
|
37
|
-
- !ruby/object:Gem::Version
|
|
38
|
-
version: '1.5'
|
|
39
|
-
- - "<"
|
|
40
|
-
- !ruby/object:Gem::Version
|
|
41
|
-
version: '2'
|
|
42
|
-
type: :runtime
|
|
43
|
-
prerelease: false
|
|
44
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
45
|
-
requirements:
|
|
46
|
-
- - ">="
|
|
47
|
-
- !ruby/object:Gem::Version
|
|
48
|
-
version: '1.5'
|
|
49
|
-
- - "<"
|
|
50
|
-
- !ruby/object:Gem::Version
|
|
51
|
-
version: '2'
|
|
52
32
|
- !ruby/object:Gem::Dependency
|
|
53
33
|
name: webrick
|
|
54
34
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -78,8 +58,7 @@ dependencies:
|
|
|
78
58
|
- !ruby/object:Gem::Version
|
|
79
59
|
version: '3.26'
|
|
80
60
|
description: Evaluate any callable Ruby LLM agent with semantic, relevance, and action
|
|
81
|
-
checks, persist runs
|
|
82
|
-
in a local dashboard.
|
|
61
|
+
checks, persist runs through ActiveRecord, and inspect regressions in a local dashboard.
|
|
83
62
|
executables:
|
|
84
63
|
- ruby_llm-agent-evaluations
|
|
85
64
|
extensions: []
|