ruby_llm-agent-evaluations 0.2.1 → 0.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3c7cae718f0af572cfe99607d7d82e896ff635c332675c8e2df6c0e4b79f6a38
4
- data.tar.gz: 43bae1d064264557c3c61293f7d1991ac951a43f7bec481e57371508e42f4726
3
+ metadata.gz: 27494e2aa94d8c50b5f680ec367b98a0cc4cb04a4d7454858a64c7dd5e7b76d5
4
+ data.tar.gz: 664eb1507d3a622c193bd4a50d11be565c3d9694dd14748f1f8b5241caa7ad92
5
5
  SHA512:
6
- metadata.gz: 070361d5bd53383114eb03ea872436af6e9501b4e19b6c0926671e948fce5e200ea7b151d4b12b15c79b8e87a6ddaf356bae858e8565c016c3e88ed826f2633e
7
- data.tar.gz: a9124295de95c93f89e16565e7f73d6ede66a6210f59e7a517675ed9f9a2c4a640d298ea7cfdf9d11011b653a95240d7843a657896007b9485b5f847b3363967
6
+ metadata.gz: e1103c795eef6354e35a3e547cddba3334d4e88e379fb6cf260f0f396c93ee0db5c05ab834c263938c5b884036d7f017e7c4768780f9c7f66f1e451241a531ac
7
+ data.tar.gz: d1ab9f91ce0b83ac6522379bd20053588c265e417c69ddc53be24d859c847f0f7fc120a670acdfda243c99789faee3461279c0835d1a00ceecc7836d91cde29d
data/CHANGELOG.md CHANGED
@@ -1,5 +1,12 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.2.2
4
+
5
+ - Support PostgreSQL, SQLite, and MySQL 8.0+ through the ActiveRecord-selected adapter and one migration creating exactly three evaluation tables. Preserve UUID run IDs, append-only evidence, foreign keys, and chronological baseline ordering. MySQL uses binary, non-padding collation to preserve exact agent, case, and metric identities.
6
+ - Remove the PostgreSQL-only migration guard and mandatory `pg` dependency. Consumers add their chosen database driver; Rails continues using its normal host pool and migration tasks, while the standalone CLI defaults to `sqlite3:ruby_llm_evaluations.sqlite3`.
7
+ - Keep existing applied PostgreSQL migrations and saved history unchanged. Rails migrations already copied into a host are not automatically rewritten; new hosts install the current engine migration. No schema reset, history deletion, or legacy upgrade is required for portability.
8
+ - Add real SQLite, PostgreSQL, and MySQL consumer coverage and document driver selection, local quick starts, and isolated cross-adapter development checks.
9
+
3
10
  ## 0.2.1
4
11
 
5
12
  - Simplify new-project setup to one migration creating only runs, case results, and metric results, with `minimum_scores` from the start. Remove legacy schema detection, payload copying, temporary migration models, and the intermediate column-rename migration. Legacy database upgrades are no longer supported.
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # RubyLLM Agent Evaluations
2
2
 
3
- Write questions and expected answers in Ruby, declare what matters, and keep evaluation evidence in PostgreSQL. Compare compatible runs and inspect answers, metrics, errors, and action traces in a read-only dashboard.
3
+ Write questions and expected answers in Ruby, declare what matters, and keep evaluation evidence in PostgreSQL, SQLite, or MySQL through ActiveRecord. Compare compatible runs and inspect answers, metrics, errors, and action traces in a read-only dashboard.
4
4
 
5
5
  Works with any callable agent. RubyLLM is optional; Rails applications use named ActiveRecord models sharing their normal host connection pool.
6
6
 
@@ -10,7 +10,7 @@ Install the package as `ruby_llm-agent-evaluations`:
10
10
  gem "ruby_llm-agent-evaluations", "~> 0.2", require: "ruby_llm/evaluations"
11
11
  ~~~
12
12
 
13
- The executable is `ruby_llm-agent-evaluations`. The Ruby API remains `RubyLLM::Evaluations` with `require "ruby_llm/evaluations"`; Rails migration tasks and PostgreSQL table names are unchanged, preserving existing history. The separately owned `ruby_llm-evaluations` package on RubyGems is not this project. The GitHub repository keeps its existing name.
13
+ The executable is `ruby_llm-agent-evaluations`. The Ruby API remains `RubyLLM::Evaluations` with `require "ruby_llm/evaluations"`; Rails migration tasks and evaluation table names are unchanged, preserving existing history. The separately owned `ruby_llm-evaluations` package on RubyGems is not this project. The GitHub repository keeps its existing name.
14
14
 
15
15
  ## Question-first specifications
16
16
 
@@ -44,7 +44,7 @@ Questions accept optional name:, id:, and expected_actions:. Names default to qu
44
44
 
45
45
  ## Strict API replay
46
46
 
47
- Declare fixtures inside a question block to exercise real HTTP clients without contacting the API. This complete spec needs the ordinary PostgreSQL setup below, but no provider credentials:
47
+ Declare fixtures inside a question block to exercise real HTTP clients without contacting the API. This complete spec needs the ordinary database setup below, but no provider credentials:
48
48
 
49
49
  ~~~ruby
50
50
  require "net/http"
@@ -82,7 +82,7 @@ Absolute `body_file:` paths are also accepted; prefer spec-relative paths for po
82
82
 
83
83
  Replay is strict/offline by default: undeclared network requests fail. For live model requests made by the agent itself, explicitly use `api_interactions(allow_hosts: ["api.openai.com"]) do ... end`. Allowances match exact hosts, not suffixes or wildcards, and may incur provider charges. An allowed host never bypasses a declared fixture request. Judge calls need no replay allowance because judging runs outside replay. There is no recording mode.
84
84
 
85
- PostgreSQL case results persist `response_metadata["api_replay"]` on success and failure: expected interactions, observed match statuses, readable mismatch differences, and missing requests. Authentication/credential fields are redacted from replay evidence. Still use synthetic fixtures: never commit credentials or sensitive payloads, and protect persisted prompts, answers, and reports.
85
+ Case results persist `response_metadata["api_replay"]` on success and failure: expected interactions, observed match statuses, readable mismatch differences, and missing requests. Authentication/credential fields are redacted from replay evidence. Still use synthetic fixtures: never commit credentials or sensitive payloads, and protect persisted prompts, answers, and reports.
86
86
 
87
87
  Replay evidence uses the existing response-metadata display in reports and the dashboard; JSON reports include the persisted `response_metadata.api_replay` payload. No separate evidence store or migration is required.
88
88
 
@@ -139,6 +139,16 @@ Add the gem to your application's Gemfile. During local development, add `path:
139
139
  gem "ruby_llm-agent-evaluations", "~> 0.2", require: "ruby_llm/evaluations"
140
140
  ~~~
141
141
 
142
+ Persistence uses the adapter selected by ActiveRecord: PostgreSQL, SQLite, or MySQL 8.0+. The gem does not require `pg` or another database driver; add your chosen driver to the application's Gemfile (or keep the one already used by your Rails host):
143
+
144
+ ~~~ruby
145
+ gem "sqlite3" # SQLite
146
+ # gem "pg" # PostgreSQL instead
147
+ # gem "mysql2" # MySQL instead
148
+ ~~~
149
+
150
+ Rails uses its existing database configuration and normal connection pool; no separate evaluation connection or storage configuration is needed. MySQL tables use `utf8mb4_0900_bin` so case, accents, and trailing spaces remain distinct in identifiers and baseline matching.
151
+
142
152
  Install the engine migrations into the host application, then migrate normally:
143
153
 
144
154
  ~~~sh
@@ -148,7 +158,9 @@ bin/rails db:migrate
148
158
 
149
159
  For standalone use, run `ruby_llm-agent-evaluations db:migrate` once before the first evaluation. Rails applications install the single engine migration and run `bin/rails db:migrate` as shown above.
150
160
 
151
- PostgreSQL is required. The single migration creates only three evaluation tables: `ruby_llm_evaluations_runs`, `ruby_llm_evaluations_case_results`, and `ruby_llm_evaluations_metric_results`. It creates `minimum_scores` directly, with the foreign keys and indexes needed for history and baseline comparisons. No legacy tables, payload imports, or custom schema-version tables are required or inspected. ActiveRecord manages its ordinary migration bookkeeping. Execution uses the host connection pool and does not establish, migrate, or disconnect it.
161
+ The single migration creates exactly three evaluation tables: `ruby_llm_evaluations_runs`, `ruby_llm_evaluations_case_results`, and `ruby_llm_evaluations_metric_results`. It creates `minimum_scores` directly, with the foreign keys and indexes needed for history and baseline comparisons. No legacy tables, payload imports, or custom schema-version tables are required or inspected. ActiveRecord manages its ordinary migration bookkeeping. Execution uses the host connection pool and does not establish, migrate, or disconnect it.
162
+
163
+ Existing PostgreSQL installations continue using their applied migration and saved history unchanged; no schema reset or history deletion is needed. Rails migrations already copied into a host are not automatically updated when the gem changes. New hosts should install the current engine migration; database portability does not require a legacy upgrade migration.
152
164
 
153
165
  ~~~ruby
154
166
  # config/routes.rb
@@ -161,12 +173,13 @@ Visit /evaluations. Assets, filters, detail/baseline links, and downloads respec
161
173
 
162
174
  ## Offline quick start: exact match, not semantic judging
163
175
 
164
- Requirements: Ruby 3.2+, PostgreSQL, Bundler. Rails and RubyLLM are optional. This deterministic local demo runs without API keys and is deliberately separate from semantic judging. It performs a real local answer lookup and records that action; exact match is not a semantic judge.
176
+ Requirements: Ruby 3.2+, Bundler, and your chosen database driver (`sqlite3` for this quick start). Rails and RubyLLM are optional. This deterministic local demo runs without API keys and is deliberately separate from semantic judging. It performs a real local answer lookup and records that action; exact match is not a semantic judge.
177
+
178
+ From this checkout, the development bundle supplies the drivers. Standalone consumers add the gem and their chosen driver to their own Gemfile, then use `bundle exec ruby_llm-agent-evaluations` in place of the checkout command below.
165
179
 
166
180
  ~~~sh
167
181
  bundle install
168
- docker compose up -d --wait
169
- export DATABASE_URL='postgresql://postgres@127.0.0.1:55432/ruby_llm_evaluations'
182
+ unset DATABASE_URL # Defaults to sqlite3:ruby_llm_evaluations.sqlite3 in the current directory.
170
183
 
171
184
  # Migrate once before executing specs. Runs never create tables implicitly.
172
185
  bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations db:migrate
@@ -179,7 +192,28 @@ bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations history
179
192
  bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations dashboard
180
193
  ~~~
181
194
 
182
- Open **http://127.0.0.1:4567**; Ctrl-C stops the server. PostgreSQL evidence survives restarts. Compose binds passwordless development PostgreSQL to loopback port 55432; **do not use this configuration in production**. Its named volume preserves history; docker compose stop does not delete it. For your own service, set DATABASE_URL to an existing database.
195
+ Open **http://127.0.0.1:4567**; Ctrl-C stops the server. The local SQLite file preserves evidence across restarts. Set DATABASE_URL to use another database or SQLite file; migrations remain an explicit setup step.
196
+
197
+ ### PostgreSQL or MySQL instead
198
+
199
+ Install `pg` for PostgreSQL or `mysql2` for MySQL in your consumer Gemfile. For local services from this checkout:
200
+
201
+ ~~~sh
202
+ docker compose up -d postgres mysql
203
+
204
+ # PostgreSQL: the development database is provisioned by Compose.
205
+ export DATABASE_URL='postgresql://postgres@127.0.0.1:55432/ruby_llm_evaluations'
206
+
207
+ # Or MySQL: create a development database, then select it.
208
+ docker compose exec -T mysql mysql -uroot -pdev-only -e 'CREATE DATABASE IF NOT EXISTS ruby_llm_evaluations;'
209
+ export DATABASE_URL='mysql2://root:dev-only@127.0.0.1:53306/ruby_llm_evaluations'
210
+
211
+ # With the selected URL, migrate once, then run the same demo.
212
+ bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations db:migrate
213
+ bundle exec ruby -Ilib exe/ruby_llm-agent-evaluations run examples/support_eval.rb
214
+ ~~~
215
+
216
+ Compose binds passwordless PostgreSQL to loopback port 55432 and MySQL 8.4 with the development-only root password `dev-only` to loopback port 53306. **Do not use these credentials/configurations in production.** Named volumes preserve database history; `docker compose stop` does not delete it. For your own service, set DATABASE_URL to an existing database with an appropriate application account.
183
217
 
184
218
  The offline example produces the following evidence:
185
219
 
@@ -193,7 +227,7 @@ The regressed agent answers the return question with 7 days instead of 30 and lo
193
227
  The standalone CLI alone establishes ActiveRecord::Base's connection. Standalone library callers configure it themselves:
194
228
 
195
229
  ~~~ruby
196
- ActiveRecord::Base.establish_connection(ENV.fetch("DATABASE_URL"))
230
+ ActiveRecord::Base.establish_connection(ENV.fetch("DATABASE_URL", "sqlite3:ruby_llm_evaluations.sqlite3"))
197
231
  RubyLLM::Evaluations.migrate! # Once during database setup, not per run.
198
232
  ~~~
199
233
 
@@ -289,20 +323,23 @@ With no judge supplied, the actual dashboard records answers but leaves quality
289
323
 
290
324
  ## Development and packaging
291
325
 
292
- Integration tests always: behavior changes must update end-to-end coverage. PostgreSQL is required; integration tests must never silently skip.
326
+ Integration tests always: behavior changes must update end-to-end coverage. Exercise real SQLite, PostgreSQL, and MySQL persistence and consumer smoke flows; integration tests must never silently skip an unavailable database. These are development-check requirements, not a requirement that consumers install every driver or run every server.
293
327
 
294
328
  ~~~sh
329
+ bundle install
330
+ docker compose up -d postgres mysql
295
331
  docker compose exec -T postgres createdb -U postgres ruby_llm_evaluations_test
296
332
  export TEST_DATABASE_URL='postgresql://postgres@127.0.0.1:55432/ruby_llm_evaluations_test'
333
+ export TEST_MYSQL_DATABASE_URL='mysql2://root:dev-only@127.0.0.1:53306/mysql'
297
334
  bundle exec rake integration
298
335
  bundle exec rake test
299
336
 
300
337
  gem build ruby_llm-agent-evaluations.gemspec
301
- gem install ./ruby_llm-agent-evaluations-0.2.1.gem
338
+ gem install ./ruby_llm-agent-evaluations-0.2.2.gem
302
339
  ruby_llm-agent-evaluations --help
303
340
  ~~~
304
341
 
305
- Tests isolate PostgreSQL schemas rather than clear development history; the test role needs schema-creation privileges. Local HTTP provider fixtures verify real adapter transport/orchestration, not model reasoning. Default tests never call paid providers. For UI changes inspect desktop/narrow layouts, keyboard access, filters, chart navigation, and downloads. Shared engineering guidance and specialist skills live in [.agents/](.agents/AGENTS.md).
342
+ Tests isolate temporary SQLite files, PostgreSQL schemas, and MySQL databases rather than clear development history. The PostgreSQL test role needs schema-creation privileges; the MySQL test account needs database-creation/deletion privileges on the test server. SQLite needs no service or test URL. Local HTTP provider fixtures verify real adapter transport/orchestration, not model reasoning. Default tests never call paid providers. For UI changes inspect desktop/narrow layouts, keyboard access, filters, chart navigation, and downloads. Shared engineering guidance and specialist skills live in [.agents/](.agents/AGENTS.md).
306
343
 
307
344
  ## Publishing
308
345
 
@@ -310,6 +347,7 @@ The release version is defined in `lib/ruby_llm/evaluations/version.rb`. Bump it
310
347
 
311
348
  ~~~sh
312
349
  export TEST_DATABASE_URL='postgresql://postgres@127.0.0.1:55432/ruby_llm_evaluations_test'
350
+ export TEST_MYSQL_DATABASE_URL='mysql2://root:dev-only@127.0.0.1:53306/mysql'
313
351
  bundle install
314
352
  bundle exec ruby script/publish --dry-run
315
353
 
@@ -7,12 +7,6 @@ module RubyLLM
7
7
  class ApplicationRecord < ::ActiveRecord::Base
8
8
  self.abstract_class = true
9
9
 
10
- def self.require_postgresql!
11
- unless connection_pool.db_config.adapter == "postgresql"
12
- raise Error, "evaluation history requires PostgreSQL"
13
- end
14
- end
15
-
16
10
  # Completed evidence is append-only through the model API.
17
11
  def readonly?
18
12
  persisted?
@@ -4,6 +4,8 @@ module RubyLLM
4
4
  module Evaluations
5
5
  class CaseResult < ApplicationRecord
6
6
  self.table_name = "ruby_llm_evaluations_case_results"
7
+ attribute :metadata, default: -> { {} }
8
+ attribute :minimum_scores, default: -> { {} }
7
9
 
8
10
  belongs_to :run, inverse_of: :case_results, class_name: "RubyLLM::Evaluations::Run"
9
11
  has_many :metric_results, -> { order(:position) }, inverse_of: :case_result,
@@ -7,6 +7,8 @@ module RubyLLM
7
7
  module Evaluations
8
8
  class Run < ApplicationRecord
9
9
  self.table_name = "ruby_llm_evaluations_runs"
10
+ self.primary_key = "id"
11
+ attribute :metadata, default: -> { {} }
10
12
 
11
13
  has_many :case_results, -> { order(:position) }, inverse_of: :run,
12
14
  class_name: "RubyLLM::Evaluations::CaseResult", dependent: :restrict_with_exception
@@ -6,51 +6,52 @@ class CreateAgentEvaluationHistory < ActiveRecord::Migration[7.1]
6
6
  METRICS = "ruby_llm_evaluations_metric_results"
7
7
 
8
8
  def change
9
- raise ActiveRecord::MigrationError, "evaluation history requires PostgreSQL" unless connection.adapter_name == "PostgreSQL"
9
+ json_type = connection.adapter_name == "PostgreSQL" ? :jsonb : :json
10
+ table_options = connection.adapter_name == "Mysql2" ? { charset: "utf8mb4", collation: "utf8mb4_0900_bin" } : {}
10
11
 
11
- create_table RUNS, id: :string do |table|
12
- table.bigserial :sequence, null: false
12
+ create_table RUNS, primary_key: :sequence, **table_options do |table|
13
+ table.string :id, null: false
13
14
  table.string :suite, null: false
14
15
  table.string :agent, null: false
15
16
  table.string :version, null: false, default: "1"
16
- table.string :suite_digest, null: false
17
- table.column :started_at, :timestamptz, null: false, precision: 6
18
- table.column :finished_at, :timestamptz, null: false, precision: 6
17
+ table.string :suite_digest, limit: 64, null: false
18
+ table.datetime :started_at, null: false, precision: 6
19
+ table.datetime :finished_at, null: false, precision: 6
19
20
  table.string :status, null: false
20
- table.jsonb :metadata, null: false, default: {}
21
- table.jsonb :summary, null: false
21
+ table.column :metadata, json_type, null: false
22
+ table.column :summary, json_type, null: false
22
23
  table.timestamps precision: 6
23
24
  end
24
- add_index RUNS, :sequence, unique: true
25
+ add_index RUNS, :id, unique: true
25
26
  add_index RUNS, [:started_at, :sequence], order: { started_at: :desc, sequence: :desc }, name: "rle_runs_recent"
26
27
  add_index RUNS, [:suite, :agent, :suite_digest, :started_at, :sequence],
27
28
  order: { started_at: :desc, sequence: :desc }, name: "rle_runs_comparable"
28
29
 
29
- create_table CASES do |table|
30
- table.references :run, type: :string, null: false, foreign_key: { to_table: RUNS }
30
+ create_table CASES, **table_options do |table|
31
+ table.references :run, type: :string, null: false, foreign_key: { to_table: RUNS, primary_key: :id }
31
32
  table.integer :position, null: false
32
33
  table.string :case_id, null: false
33
34
  table.text :name, null: false
34
- table.jsonb :question
35
- table.jsonb :expected_answer
36
- table.jsonb :expected_actions
37
- table.jsonb :minimum_scores, null: false, default: {}
38
- table.jsonb :metadata, null: false, default: {}
39
- table.jsonb :output
35
+ table.column :question, json_type
36
+ table.column :expected_answer, json_type
37
+ table.column :expected_actions, json_type
38
+ table.column :minimum_scores, json_type, null: false
39
+ table.column :metadata, json_type, null: false
40
+ table.column :output, json_type
40
41
  table.string :status, null: false
41
42
  table.float :duration_ms, null: false
42
43
  table.bigint :input_tokens
43
44
  table.bigint :output_tokens
44
45
  table.float :cost
45
- table.jsonb :actions
46
- table.jsonb :response_metadata
47
- table.jsonb :error_details
46
+ table.column :actions, json_type
47
+ table.column :response_metadata, json_type
48
+ table.column :error_details, json_type
48
49
  table.timestamps precision: 6
49
50
  end
50
51
  add_index CASES, [:run_id, :position], unique: true, name: "rle_case_order"
51
52
  add_index CASES, [:run_id, :case_id], unique: true, name: "rle_case_identity"
52
53
 
53
- create_table METRICS do |table|
54
+ create_table METRICS, **table_options do |table|
54
55
  table.references :case_result, null: false, foreign_key: { to_table: CASES }
55
56
  table.integer :position, null: false
56
57
  table.string :name, null: false
@@ -58,7 +59,7 @@ class CreateAgentEvaluationHistory < ActiveRecord::Migration[7.1]
58
59
  table.float :score
59
60
  table.boolean :passed, null: false
60
61
  table.text :reason
61
- table.jsonb :error_details
62
+ table.column :error_details, json_type
62
63
  table.timestamps precision: 6
63
64
  end
64
65
  add_index METRICS, [:case_result_id, :position], unique: true, name: "rle_metric_order"
@@ -32,7 +32,7 @@ module RubyLLM
32
32
  begin
33
33
  ::ActiveRecord::Base.establish_connection(options[:database_url])
34
34
  rescue ArgumentError, URI::InvalidURIError
35
- raise Error, "Invalid database configuration. Set DATABASE_URL to a PostgreSQL connection URL."
35
+ raise Error, "Invalid database configuration. Set DATABASE_URL to an ActiveRecord connection URL."
36
36
  end
37
37
  if command == "db:migrate"
38
38
  raise ArgumentError, "db:migrate takes no positional arguments" unless args.empty?
@@ -50,7 +50,7 @@ module RubyLLM
50
50
  when "dashboard" then dashboard(args, options)
51
51
  end
52
52
  rescue ::ActiveRecord::ActiveRecordError
53
- @err.puts("Database configuration error. Check DATABASE_URL and PostgreSQL availability; run ruby_llm-agent-evaluations db:migrate to create evaluation tables.")
53
+ @err.puts("Database configuration error. Check DATABASE_URL and database availability; run ruby_llm-agent-evaluations db:migrate to create evaluation tables.")
54
54
  2
55
55
  rescue OptionParser::ParseError, ArgumentError, Error, SystemCallError, JSON::ParserError, SyntaxError => exception
56
56
  @err.puts("#{exception.class}: #{exception.message}")
@@ -64,7 +64,7 @@ module RubyLLM
64
64
  usage = { "run" => "run SPEC.rb", "history" => "history", "report" => "report [RUN_ID]",
65
65
  "dashboard" => "dashboard", "db:migrate" => "db:migrate" }.fetch(command)
66
66
  parser.banner = "Usage: ruby_llm-agent-evaluations #{usage} [options]"
67
- parser.on("--database-url URL", "PostgreSQL URL (default: DATABASE_URL or local database)") { |value| options[:database_url] = value }
67
+ parser.on("--database-url URL", "Database URL (default: DATABASE_URL or local SQLite file)") { |value| options[:database_url] = value }
68
68
  if %w[run report].include?(command)
69
69
  parser.on("--format FORMAT", FORMATS, "text, markdown, html, or json") { |value| options[:format] = value }
70
70
  parser.on("--output PATH", "Write report to a file") { |value| options[:output] = value }
@@ -13,6 +13,6 @@
13
13
  <nav aria-label="Main navigation"><a href="<%= ui.h(ui.path("/")) %>">Run history</a><span class="local-badge">Local workspace</span></nav>
14
14
  </div></header>
15
15
  <main id="main" tabindex="-1"><%= content %></main>
16
- <footer class="site-footer"><span>Evaluations as code. Evidence over intuition.</span><span>Read-only · PostgreSQL history · No external assets</span></footer>
16
+ <footer class="site-footer"><span>Evaluations as code. Evidence over intuition.</span><span>Read-only · Persistent history · No external assets</span></footer>
17
17
  </body>
18
18
  </html>
@@ -5,7 +5,6 @@ module RubyLLM
5
5
  class Runner
6
6
  def run(suite, metadata: {})
7
7
  suite.validate!
8
- Run.require_postgresql!
9
8
  raise Error, "Apply evaluation migrations before running a suite" unless Run.table_exists?
10
9
  raise ArgumentError, "run metadata must be a Hash" unless metadata.is_a?(Hash)
11
10
  run_metadata = Data.json_value(metadata)
@@ -2,6 +2,6 @@
2
2
 
3
3
  module RubyLLM
4
4
  module Evaluations
5
- VERSION = "0.2.1"
5
+ VERSION = "0.2.2"
6
6
  end
7
7
  end
@@ -21,7 +21,7 @@ require_relative "evaluations/dashboard"
21
21
 
22
22
  module RubyLLM
23
23
  module Evaluations
24
- DEFAULT_DATABASE_URL = "postgresql://localhost/ruby_llm_evaluations"
24
+ DEFAULT_DATABASE_URL = "sqlite3:ruby_llm_evaluations.sqlite3"
25
25
  MIGRATIONS_PATH = File.expand_path("../../db/migrate", __dir__).freeze
26
26
  autoload :Engine, File.expand_path("evaluations/engine", __dir__)
27
27
 
@@ -38,7 +38,6 @@ module RubyLLM
38
38
 
39
39
  # Standalone setup uses Rails migrations; a host application uses db:migrate.
40
40
  def self.migrate!
41
- ApplicationRecord.require_postgresql!
42
41
  ::ActiveRecord::MigrationContext.new([MIGRATIONS_PATH]).migrate
43
42
  [Run, CaseResult, MetricResult].each(&:reset_column_information)
44
43
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: ruby_llm-agent-evaluations
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.1
4
+ version: 0.2.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - RubyLLM Evaluations contributors
@@ -29,26 +29,6 @@ dependencies:
29
29
  - - "<"
30
30
  - !ruby/object:Gem::Version
31
31
  version: '9'
32
- - !ruby/object:Gem::Dependency
33
- name: pg
34
- requirement: !ruby/object:Gem::Requirement
35
- requirements:
36
- - - ">="
37
- - !ruby/object:Gem::Version
38
- version: '1.5'
39
- - - "<"
40
- - !ruby/object:Gem::Version
41
- version: '2'
42
- type: :runtime
43
- prerelease: false
44
- version_requirements: !ruby/object:Gem::Requirement
45
- requirements:
46
- - - ">="
47
- - !ruby/object:Gem::Version
48
- version: '1.5'
49
- - - "<"
50
- - !ruby/object:Gem::Version
51
- version: '2'
52
32
  - !ruby/object:Gem::Dependency
53
33
  name: webrick
54
34
  requirement: !ruby/object:Gem::Requirement
@@ -78,8 +58,7 @@ dependencies:
78
58
  - !ruby/object:Gem::Version
79
59
  version: '3.26'
80
60
  description: Evaluate any callable Ruby LLM agent with semantic, relevance, and action
81
- checks, persist runs in PostgreSQL through ActiveRecord, and inspect regressions
82
- in a local dashboard.
61
+ checks, persist runs through ActiveRecord, and inspect regressions in a local dashboard.
83
62
  executables:
84
63
  - ruby_llm-agent-evaluations
85
64
  extensions: []