inquirex-llm 0.12.1 → 0.12.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ac542fe51e67d8477ceb8b7b45facb1af3e79b4794cd69d9e62a158d40931df3
-  data.tar.gz: 0a995561405b2cf3838c72a696ba2c20e55d9fb185cbd44bbf8f0a9dd02fffe2
+  metadata.gz: 1261cf90cb18c935731001fc135e896e271d63e64f84aba6fc5b45105bfcaa8c
+  data.tar.gz: 74ddcc8813674ae65e5979a3becc1c9ecbd7bdf4b0e64e71161a6ddc15c86a01
 SHA512:
-  metadata.gz: 3aa789c543358cd2a3cf12423e252fa07673fcedf6cdd950db4a205d3cea62eaebf2f493207989c3aaf4b89c8a89cc714b5d54fb5a9bfa50fc24e7737f5cded4
-  data.tar.gz: cde3f862aa1fe7f3bac1988290c93595e997df98e3fc22798e8ff146b757f3552f39d7f24129b95a19fa15ce71236f21332d290cfdbe98e8633d203aeff480f7
+  metadata.gz: 1eda2a5ae132b1027700dc058f90ff47fe3a29e5e8804669b5ad4c9c1771f76b8c01484963e8f3d63d5e3fc9f3311e568d67d73b0f35f7e3cff9c1723a4d723c
+  data.tar.gz: 7af624fbd2546c4cf1404f3ac51aecc72c670e013888fb7fdb654be5f2f3ee455023e08737dfea510295bdfa9fd783c4ffe801f2e383004f8491ed463ad02955

data/CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.12.2 (unreleased)
+
+- **The eval suite now runs on llm-evals.** `inquirex/llm/evals` requires `llm/evals` and subclasses `::LLM::Evals`, because `agent-evals` was renamed and is no longer on RubyGems. The config file is `evals/.llm-evals.config.yml` and the executable is `evals`. Inside `Inquirex::LLM`, a bare `LLM` means this gem, so the leading `::` is required.
+
 ## 0.12.0 (unreleased)
 
 Hand-written until `github_changelog_generator` regenerates this file at release. The family-wide notes are in inquirex's `INQUIREX_VERSION_LOG.md`.

data/README.md

@@ -487,15 +487,15 @@ end
 
 ## Evals
 
-Which model extracts best, measured with this gem's own prompts: ten flows, a complete and a vague text each, run on [agent-evals](https://github.com/kigster/agent-evals). The lineup and the data live in [`evals/`](evals/README.md); the adapter and the scorer are `Inquirex::LLM::Evals`, loaded only by `require "inquirex/llm/evals"`.
+Which model extracts best, measured with this gem's own prompts: ten flows, a complete and a vague text each, run on [llm-evals](https://github.com/kigster/llm-evals). The lineup and the data live in [`evals/`](evals/README.md); the adapter and the scorer are `Inquirex::LLM::Evals`, loaded only by `require "inquirex/llm/evals"`.
 
 How a run works:
 
 1. **Verify** (offline): each dataset's `schema.rb` must match the `extract` step in its `dsl.rb`, and every `expected.json` must fit that schema.
-1. **Expand the lineup**: every model under `vendors:` in `evals/.agent-eval.config.yml`, times each `effort`/`temperature` it lists, becomes one contender. Each contender runs every case.
+1. **Expand the lineup**: every model under `vendors:` in `evals/.llm-evals.config.yml`, times each `effort`/`temperature` it lists, becomes one contender. Each contender runs every case.
 1. **Call the model**: one OpenRouter chat completion per case, built from this gem's production prompt and the step's schema.
 1. **Score the raw reply** field by field: `correct` 1.0, `missed` 0.2, `hallucinated` 0.15, `incorrect` 0.1, `invalid` 0. A case's score is the mean over its fields. The rules are in [`evals/README.md`](evals/README.md#scoring).
-1. **Report**: a leaderboard on screen and a JSON log under `log/agent-evals/`. If `BRAINTRUST_API_KEY` or `LOGFIRE_TOKEN` is set, each case is also sent to that one tracker; `--local` turns that off.
+1. **Report**: a leaderboard on screen and a JSON log under `log/llm-evals/`. If `BRAINTRUST_API_KEY` or `LOGFIRE_TOKEN` is set, each case is also sent to that one tracker; `--local` turns that off.
 
 ```bash
 just evals -n                       # the whole matrix, calling nothing
@@ -503,8 +503,8 @@ just evals                          # every configured model on every dataset
 just evals --local -d 05-managed-it-services   # one dataset, no remote tracker
 just evals-verify                   # step 1 only
 
-bundle exec agent-evals cases       # the 20 cases
-bundle exec agent-evals run -m openai/gpt-4o-mini   # one model, per-case breakdown
+bundle exec evals cases       # the 20 cases
+bundle exec evals run -m openai/gpt-4o-mini   # one model, per-case breakdown
 ```
 
 `just evals` needs `OPENROUTER_API_KEY` and the `:evals` bundle group. It is a paid run: 14 contenders × 20 cases = 280 calls.

data/lib/inquirex/llm/evals/adapter.rb

@@ -9,9 +9,9 @@ module Inquirex
       # canonicalized by the gem. It never scores; {Scorer} does that.
       #
       # A reply that is not a JSON object is the model's failure and comes
-      # back as a {Agent::Evals::Reply} with `error` set. A transport failure
+      # back as a {LLM::Evals::Reply} with `error` set. A transport failure
       # raises, and the run counts it as an error rather than a wrong answer.
-      class Adapter < Agent::Evals::Adapter
+      class Adapter < ::LLM::Evals::Adapter
         # Recorded with every experiment: which prompt the models saw.
         PROMPT = "Inquirex::LLM::OpenAIAdapter"
 
@@ -24,7 +24,7 @@ module Inquirex
         end
 
         # @return [String]
-        def root = @root || Agent::Evals.configuration.root
+        def root = @root || ::LLM::Evals.configuration.root
 
         # Loads the dataset's flow and schema once, before its cases run in threads.
         #
@@ -36,14 +36,14 @@ module Inquirex
           set.schema
         end
 
-        # @param kase [Agent::Evals::Case]
-        # @param model [Agent::Evals::Model]
-        # @return [Agent::Evals::Reply]
+        # @param kase [LLM::Evals::Case]
+        # @param model [LLM::Evals::Model]
+        # @return [LLM::Evals::Reply]
         def call(kase, model:)
           set = dataset(kase.dataset)
           adapter = ModelAdapter.new(model)
           extraction = adapter.call(set.node, { set.source_step => kase.input })
-          Agent::Evals::Reply.new(
+          ::LLM::Evals::Reply.new(
             output:   extraction.output.transform_keys(&:to_s),
             raw:      extraction.raw.transform_keys(&:to_s),
             usage:    adapter.completion.usage,
@@ -52,13 +52,13 @@ module Inquirex
         rescue Errors::AdapterError => e
           raise if adapter.completion.nil? # nothing was asked: not the model's doing
 
-          Agent::Evals::Reply.new(error: e.message.lines.first.strip,
+          ::LLM::Evals::Reply.new(error: e.message.lines.first.strip,
             raw: adapter.completion.text,
             usage: adapter.completion.usage,
             metadata: { prompt: adapter.prompt })
         end
 
-        # @param kase [Agent::Evals::Case]
+        # @param kase [LLM::Evals::Case]
         # @return [Hash] the schema the scorer grades against
         def context(kase) = { schema: dataset(kase.dataset).schema }
 

data/lib/inquirex/llm/evals/dataset.rb

@@ -6,7 +6,7 @@ module Inquirex
       # What one dataset folder shares across its cases: the flow in dsl.rb,
       # whose single `extract` step supplies the prompt and the schema, and
       # schema.rb, that schema copied out verbatim so a change to either is
-      # caught by `agent-evals verify` instead of silently moving the scores.
+      # caught by `evals verify` instead of silently moving the scores.
       #
       #   evals/datasets/05-managed-it-services/
       #     dsl.rb                       the flow, loaded with unsafe: true (a file we author)
@@ -61,7 +61,7 @@ module Inquirex
         # @return [Symbol]
         def source_step = node.from_steps.first
 
-        # Offline checks of the shared files, for `agent-evals verify`.
+        # Offline checks of the shared files, for `evals verify`.
         #
         # @return [Array<String>] problems; empty when the dataset is sound
         def problems

data/lib/inquirex/llm/evals/model_adapter.rb

@@ -4,7 +4,7 @@ module Inquirex
   module LLM
     module Evals
       # The gem's {OpenAIAdapter} with its transport swapped for an
-      # agent-evals {Agent::Evals::Model}. Everything the model sees and
+      # llm-evals {LLM::Evals::Model}. Everything the model sees and
       # everything done to its answer is production's: the system prompt,
       # the schema contract, the user prompt built from the step's `from`
       # answers, fence stripping, JSON parsing and the canonicalization of
@@ -18,13 +18,13 @@ module Inquirex
         # OpenAIAdapter insists on a key; the eval model holds the real one.
         NO_KEY = "(unused: the eval model carries its own key)"
 
-        # @return [Agent::Evals::Completion, nil] what the model answered, once asked
+        # @return [LLM::Evals::Completion, nil] what the model answered, once asked
         attr_reader :completion
 
         # @return [Hash{Symbol => String}, nil] the system and user prompts, once sent
         attr_reader :prompt
 
-        # @param model [Agent::Evals::Model]
+        # @param model [LLM::Evals::Model]
         def initialize(model)
           super(api_key: NO_KEY, model: model.id)
           @model = model

data/lib/inquirex/llm/evals/scorer.rb

@@ -13,36 +13,36 @@ module Inquirex
       # share the model answered anyway; it is nil when the truth has no
       # null, so a dataset of complete texts neither earns nor loses by it.
       # `invalid_rate` is the share of fields the schema would reject.
-      class Scorer < Agent::Evals::Scorer
+      class Scorer < ::LLM::Evals::Scorer
         METRICS = [
-          Agent::Evals::Metric.new(name: "score",
+          ::LLM::Evals::Metric.new(name: "score",
             primary: true,
             description: "Mean field value: correct 1.0, missed 0.2, hallucinated 0.15, incorrect 0.1, invalid 0; " \
                          "a multi_enum is the mean of its elements."),
-          Agent::Evals::Metric.new(name: "exact", description: "1 when every field is correct."),
-          Agent::Evals::Metric.new(name: "hallucinated_rate",
+          ::LLM::Evals::Metric.new(name: "exact", description: "1 when every field is correct."),
+          ::LLM::Evals::Metric.new(name: "hallucinated_rate",
             description: "Of the fields the text never answers (truth null), the share the model answered anyway; " \
                          "nothing recorded when the truth has no null."),
-          Agent::Evals::Metric.new(name: "invalid_rate",
+          ::LLM::Evals::Metric.new(name: "invalid_rate",
             description: "Share of fields whose value the schema does not allow: off the list, wrong type.")
         ].freeze
 
-        # How the gem's field types read to {Agent::Evals::SchemaValidator}.
+        # How the gem's field types read to {LLM::Evals::SchemaValidator}.
         VALIDATOR_TYPES = { enum: :enum, multi_enum: :multi_enum, integer: :integer, boolean: :boolean,
                             decimal: :number, currency: :number }.freeze
 
-        # @return [Array<Agent::Evals::Metric>]
+        # @return [Array<LLM::Evals::Metric>]
         def metrics = METRICS
 
-        # @param kase [Agent::Evals::Case]
-        # @param reply [Agent::Evals::Reply]
+        # @param kase [LLM::Evals::Case]
+        # @param reply [LLM::Evals::Reply]
         # @param context [Hash] `:schema`, the {Inquirex::LLM::Schema} the dataset extracts
-        # @return [Agent::Evals::Score]
+        # @return [LLM::Evals::Score]
         def call(kase, reply, context:)
           schema = context.fetch(:schema)
           fields = field_scores(schema, kase.expected, reply.raw)
           open = fields.select { kase.expected[it.field].nil? }
-          Agent::Evals::Score.new(
+          ::LLM::Evals::Score.new(
             value:       mean(fields.map(&:value)),
             metrics:     { "score"             => mean(fields.map(&:value)),
                            "exact"             => fields.all?(&:correct?) ? 1.0 : 0.0,
@@ -74,7 +74,7 @@ module Inquirex
         # Checks expected.json against the schema, every field nullable: a
         # null is how the truth says the text never answers it.
         #
-        # @param kase [Agent::Evals::Case]
+        # @param kase [LLM::Evals::Case]
         # @param context [Hash]
         # @return [Array<String>] unknown keys, missing keys, off-list values, wrong types
         def verify(kase, context:)
@@ -82,10 +82,10 @@ module Inquirex
           fields = schema.field_specs.to_h do |name, spec|
             [name.to_s, { type: VALIDATOR_TYPES.fetch(spec.type, :string), enum: spec.values, nullable: true }]
           end
-          Agent::Evals::SchemaValidator.new(fields).problems(kase.expected)
+          ::LLM::Evals::SchemaValidator.new(fields).problems(kase.expected)
         end
 
-        # @param kase [Agent::Evals::Case]
+        # @param kase [LLM::Evals::Case]
         # @return [String] "answered 7/10"
         def summary(kase)
           return nil unless kase.expected.is_a?(Hash)

data/lib/inquirex/llm/evals.rb

@@ -1,15 +1,15 @@
 # frozen_string_literal: true
 
-# The extraction eval suite's Ruby: an agent-evals Adapter that runs one
+# The extraction eval suite's Ruby: an llm-evals Adapter that runs one
 # case through the gem's production prompts, and a Scorer that grades the
 # reply field by field. Loaded by `require "inquirex/llm/evals"` from
-# evals/.agent-eval.config.yml, never by `require "inquirex-llm"`, because
-# agent-evals is the suite's dependency and not the gem's.
+# evals/.llm-evals.config.yml, never by `require "inquirex-llm"`, because
+# llm-evals is the suite's dependency and not the gem's.
 begin
-  require "agent/evals"
+  require "llm/evals"
 rescue LoadError => e
   raise LoadError,
-    "inquirex/llm/evals needs the agent-evals gem, which is not a dependency of inquirex-llm: " \
+    "inquirex/llm/evals needs the llm-evals gem, which is not a dependency of inquirex-llm: " \
     "add it to the :evals group of your Gemfile (#{e.message})"
 end
 
@@ -22,7 +22,7 @@ require_relative "evals/scorer"
 
 module Inquirex
   module LLM
-    # The pieces agent-evals needs from this gem. {Adapter} owns everything
+    # The pieces llm-evals needs from this gem. {Adapter} owns everything
     # the model sees and everything done to its answer; {Scorer} owns what a
     # right answer is. Neither knows the other, and only the config file
     # names them both.

data/lib/inquirex/llm/version.rb

@@ -2,6 +2,6 @@
 
 module Inquirex
   module LLM
-    VERSION = "0.12.1"
+    VERSION = "0.12.2"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: inquirex-llm
 version: !ruby/object:Gem::Version
-  version: 0.12.1
+  version: 0.12.2
 platform: ruby
 authors:
   - Konstantin Gredeskoul