agent-evals 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 56fb5b0637a74afe1bf8be4010c7ee38f9e0838d2dc0e449e01dae294fb2fdd9
+  data.tar.gz: 7d9dd13e1d1df76b9ce6b38813775285921505b12ba863c37b730d223ef552d1
+SHA512:
+  metadata.gz: 82e38be379ec294b9f6d866ca3d2e7e83651a6790a6021a1010b430a53571389531b63c65faa1e58ba937de466f59c2a50187f04481ad2eaf3406efe5b9bedac
+  data.tar.gz: e955b8dfbdb2bb4194e7cf7712bc0cbba395b86d8534c24eb97493d66185dbbf8503c1bf62c648bd92e180274311b6a3b5d41fb57ffeb60ea6b3ab9273bb0c49

data/CHANGELOG.md

@@ -0,0 +1,16 @@
+# Changelog
+
+## 0.1.0 (2026-10-14)
+
+First release, extracted from the `extract-eval` suite of inquirex-llm and made generic.
+
+- Two contracts a suite implements: `Adapter` (runs one case against a `Model`, returns a `Reply`, never scores) and `Scorer` (returns a `Score` with `Metric`s and per-field diagnostics, never calls anything).
+- Value objects: `Case`, `Usage`, `Completion`, `Reply`, `Metric`, `Score`, `RunResult`, `Contender`.
+- `Agent::Evals.configure` and `evals/eval_helper.rb`, found upward from the working directory (`--helper`, `AGENT_EVALS_HELPER`); or a config-only suite whose `.agent-eval.config.yml` names `require:`, `adapter:` and `scorer:`, the helper winning where both exist.
+- `Discovery` of `datasets/<dataset>/cases/<variant>/[NNN/]input.txt` with `expected.json` beside; any layout through the glob.
+- `Config`: the `.agent-eval.config.yml` lineup with effort and temperature lists expanded into contenders, and the required `request_types` list, and a `direct:` section for a vendor's own API (`typesafe`), keyed by provider name; a provider declares its `request_type` and a contender is admitted only when the suite declares it, else refused by name in the warnings.
+- Providers: a registry asked last-registered first; `OpenRouter` through RubyLLM; `TypeSafe` as catalog, key and pricing with a transport stub; `Catalog` for model lists.
+- `Experiment`: parallel runs on the gem's own thread pool, watched by trackers: `Trackers::Local` always writes `log/agent-evals/*.json`; one remote tracker, `braintrust` (eval rows through the gem's exporter) or `logfire` (OTLP traces through the OpenTelemetry SDK), is chosen by `--tracker`, `tracker:` or the one env key set. braintrust, ruby_llm and the OpenTelemetry gems are optional and named when missing.
+- `Verify`: empty inputs and broken `expected.json`, then the adapter's and the scorer's own checks. `SchemaValidator` for typed, closed expected values.
+- CLI: `cases`, `verify`, `run`, `compare`, `version`, `completion`; `-n/--dry-run` and `-o/--output`; leaderboard, glossary and best value derived from the scorer's metrics; warnings held off the screen and saved to `log/`.
+- `Testing::EchoAdapter`, `Testing::FakeProvider` and `Testing::FakeModel`, and a toy suite under `spec/fixtures/toy_suite/`.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Konstantin Gredeskoul
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.