evalblink 0.1.0__tar.gz

evalblink-0.1.0/CLAUDE.md

@@ -0,0 +1,82 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this is
+
+evalblink benchmarks LLM prompts × models on your own data, routing every call through [OpenRouter](https://openrouter.ai). A run takes a YAML benchmark config, executes `model × prompt × test_case`, scores each response, and writes a results matrix (Rich table), a JSON record, and a Markdown report.
+
+## Commands
+
+```bash
+# Setup
+pip install -e ".[dev]"                  # installs evalblink + dev tools (ruff, pytest)
+export OPENROUTER_API_KEY=...             # or put it in .env (loaded via python-dotenv)
+
+# Run a benchmark
+evalblink run benchmarks/classification.yaml
+evalblink run                            # defaults to benchmarks/exact_match_classification.yaml
+evalblink run benchmarks/classification.yaml -v          # verbose: per-test-case detail
+evalblink run benchmarks/classification.yaml --no-cache  # bypass the response cache (candidate and judge calls)
+evalblink run benchmarks/classification.yaml --dry-run   # estimate cost: no completion calls, no API key (fetches the public /models catalog)
+
+# Validate a config (errors and warnings, no API calls)
+evalblink validate benchmarks/classification.yaml
+
+# Compare two finished runs (quality + cost deltas; offline, no API calls)
+evalblink compare results/<run_a>.json results/<run_b>.json
+evalblink compare results/<run_a>.json results/<run_b>.json --detailed  # per-test-case transitions + global summary
+
+# Other commands
+evalblink report results/<run>.json      # regenerate Markdown from existing JSON
+evalblink history                        # list all past runs
+evalblink models                         # list OpenRouter models + pricing
+evalblink models --provider anthropic --free --min-context 100k
+evalblink models --no-cache              # refetch the 24h catalog cache
+evalblink init                           # interactively scaffold a new benchmark YAML
+evalblink cache stats
+evalblink cache clear --yes
+
+# Tests
+pytest -q
+
+# Lint (ruff is the only dev tool; no config file, uses defaults)
+ruff check .
+ruff format .
+```
+
+Tests live in `tests/` and run offline (a fake `httpx.Client`, no API key). Beyond unit tests, verify end-to-end by running configs and inspecting the generated `results/<timestamp>_<slug>.{json,md}` files. Runs hit the live OpenRouter API (free models cost $0 but are rate-limited).
+
+## Architecture
+
+A run flows through one linear pipeline; module boundaries are deliberate, not incidental:
+
+- **`main.py`** — `argparse` subcommand dispatcher (`load_dotenv(find_dotenv(usecwd=True), encoding="utf-8-sig")` — `.env` from the cwd or a parent, never the package dir; a non-UTF-8 `.env` is ignored with a warning — then `args.func`, with `KeyboardInterrupt` → exit 130 and `EOFError` → clean exit). Subcommands: `run`, `validate`, `compare`, `report`, `history`, `models`, `init`, `cache stats`, `cache clear`. `run` validates the config via `validator.validate` before touching the API — exits 1 with error details on bad config — and (except `--dry-run`) exits 1 with a clear message before any API call if `OPENROUTER_API_KEY` is unset, empty or whitespace-only, and if `results/` can't be created/written. After the run: if any pipeline errors occurred it prints a prominent warning with their count (excluded from scores); if *every* case is a pipeline error (nothing scored) it exits 1; otherwise the top-level `quality_threshold` gate decides (absent → exit 0) on `analysis.summarize`'s fail-safe `gate_score` (passed / total cases: unscored cases count as failures). `load_config` exits with a clear message on missing/invalid YAML (including the `ValueError` PyYAML raises on a >4300-digit int and `RecursionError` on deep nesting). Write failures in `run`/`report`/`history`/`cache` exit with a one-line message. `report`/`compare` exit with `compare.RecordError`'s one-line reason on a bad results file; `history` prints `⚠  Skipping <reason>` and continues, and recomputes insights (best score, run-total errors) from `results`. `models --min-context` uses `_parse_context` as its argparse type (bad value → usage error, exit 2). `init` re-asks on invalid answers via `_ask`, finds variables with the validator's `_template_vars`, continues without suggestions if the catalog fetch fails, and runs `validate` before writing (errors → exit 1, nothing written). `main()` reconfigures stdout/stderr with `errors="replace"`. Every console comes from `reporter.make_console()` (`markup=False, emoji=False`): user/model text is printed literally, so style whole lines with `style=`, never inline `[markup]`.
+- **`runner.py`** — owns the single `httpx.Client` and the `model × prompt × test_case` triple loop. Renders Jinja2 templates (`template` and `system`) in a `jinja2.sandbox.SandboxedEnvironment` — unsafe attribute access is blocked: a lone unsafe attribute (`{{ x.__class__ }}`) renders empty, calls/chained access through one raise `SecurityError`. Dispatches work to `ThreadPoolExecutor` (default 5 workers, config key `concurrency`, 1–100; Ctrl-C sets `openrouter.CANCELLED`, cancels queued futures without waiting, and `main()` exits 130 via `os._exit`), routes each response to the right evaluator, accumulates per-prompt token/cost totals; `use_cache` is passed to both the candidate call and `evaluate_llm_judge`. A failed call never aborts the run. A candidate API failure (`openrouter_request` raising `RuntimeError`/`httpx.HTTPError` after its retries) is caught: the case is recorded with `match_score=None`, `match_result=False`, `status="api_error"`, `response=""`, an `error` message and 0 tokens/cost, a one-line warning is printed, and it counts in the `Errors` column (excluded from the score) like a judge failure. `_run_case` wraps `_evaluate_case`: any other exception becomes an `internal_error` case, so no single case can crash a run. A non-ASCII API key is a `RuntimeError` from `openrouter._auth_headers`. Per-case detail prints are gated behind `verbose` (only a `Test case: <id>` progress line prints for every case).
+- **`validator.py`** — static validation of benchmark YAML configs; no API calls, no side effects. Returns `(errors, warnings)`. Called by `cmd_run`, `cmd_validate` and `cmd_init` (before writing). Checks required fields, eval-mode-specific constraints, Jinja2 template variable coverage, and heuristic warnings (single-brace syntax, low max_tokens for llm_judge, long expected_output). Type/shape mistakes are errors with actionable messages, not later crashes: empty or non-mapping file, non-string `name`, top-level `quality_threshold` not a number in 0–100, `concurrency` not an int in 1–100, a self-referencing YAML alias (the config must `json.dumps`), `inference`/`evaluation` present but not a mapping (a null value is treated as absent everywhere in the code), non-numeric `inference.temperature`, `inference.max_tokens` not a positive int (`50.5`, `100.0`, `true` rejected; "must be a positive whole number like 100"), any number that fails `openrouter.is_number` (non-bool int/float that converts to a finite float — so `.inf`/`.nan` and a 400-digit int are rejected in every numeric field, `concurrency` included), `evaluation.judge_model` not a string whenever present, duplicate `models` entries, non-string model/prompt/test-case ids, Jinja syntax errors in `template`/`system`, `exact_match` `expected_output` not a string (YAML `yes`/`true`/`3` — quote it), and `weighted_match` `evaluation.variables` missing any of `use_case`/`percent`/`order` or `expected_output` empty / not a list / an item whose `use_case` is unhashable. Once the rest is valid, it trial-renders every prompt (`template` + `system`) against every test case with the runner's own `render_template` and sandbox; any exception is an error naming the prompt id and test case id, so a deterministic template that would fail at run time fails `validate` first.
+- **`openrouter.py`** — the *single API choke point*. Every completion call (candidate responses AND the LLM judge) goes through `openrouter_request`; the only other request is the public `GET /models` catalog in `fetch_models` (cached 24h, no key needed — `_auth_headers` omits `Authorization` when no key is set). It lives in its own leaf module specifically to avoid a `runner ↔ evaluator` import cycle. Responsibilities: build the request, consult the SHA256 cache, **retry transient failures** (`RETRYABLE_CODES = {408,429,502,503,504}` from the HTTP status line or the JSON error body, + local `httpx.TransportError` (timeouts included), `MAX_RETRIES` attempts, exponential backoff). Every other failure — any other `httpx.HTTPError` (no retry), HTTP error status, non-JSON body, missing/empty `choices` or `message`, null or empty content, missing or non-finite usage — raises `RuntimeError` with a clear message (HTTP status included when known), which both callers catch. `_result_problem` applies the same checks to fresh and cached results (a bad cache entry is a miss and is refetched); cache writes go through `_cache_set`, which warns once and continues. `clean_text` replaces lone surrogates in the outgoing model/prompt/system text (before the cache key) and in incoming/cached content and error messages. `is_number` is the single numeric check. `_price` maps a negative, non-numeric or non-finite catalog price to `None` (unknown), also on re-reading the models cache.
+- **`evaluator.py`** — three scorers: `exact_match`, `weighted_match`, `evaluate_llm_judge`, plus `_extract_json` (strips markdown code fences before parsing). Pure scoring; `evaluate_llm_judge` makes an API call but receives the `httpx.Client` as a parameter (never creates one) so it stays free of any runner dependency. Judge failures (API error, malformed JSON) return a result with `score=None` and a `status` flag — a pipeline failure is distinguished from a real score of 0. `weighted_match` returns `0.0` on unparseable candidate output (a task failure, not a pipeline error); `_extract_json` turns any `ValueError`/`RecursionError` from `json.loads` (deep nesting, a 5000-digit int) into `JSONDecodeError`. `evaluate_llm_judge` takes `use_cache`, so `--no-cache` covers judge calls too. **`exact_match` is a plain case-insensitive string comparison** (`response.strip().lower() == expected.strip().lower()`) — no JSON parsing. `_extract_json` is called only by `weighted_match` and `evaluate_llm_judge`.
+- **`analysis.py`** — pure insights over a finished run's `results` (no I/O): `best_quality` (highest score, ties → lower cost), `best_value` (score per dollar, cost floored at `1e-4` so free models fall back to the top score), and `tag_breakdown` (per-tag pass-rate over scored cases only — `match_score is None` counts as an error, not a fail — sorted warnings-first then by tag, warn below 70 %). `summarize` is the single entry point and also computes the CI `passed` verdict against the top-level `quality_threshold`; its `errors` is the run total over all combos. The reporter only formats it.
+- **`estimate.py`** — offline cost estimation for `--dry-run`. Pure logic: given a config and pricing catalog from `openrouter.fetch_models`, estimates total cost without making any completion calls. Returns per-combo rows plus the `over_budget` verdict against `max_cost_usd`. A model absent from the catalog or priced `None` there (see `openrouter._price`) goes in `missing_pricing`: excluded and warned.
+- **`cache.py`** — SHA256 file cache under `.evalblink_cache/<key>.json`. The cache key is the full request payload (model, messages, temperature, max_tokens). `temperature: 0` + cache = reproducible runs. `--no-cache` bypasses it for candidate and judge calls. `get` treats an unreadable/corrupt entry (`OSError`, `ValueError`, `RecursionError`) as a miss. `cache stats`/`cache clear` count every `.json` in the directory, including the `models.json` catalog; `cache clear` without `--yes` only prints the count.
+- **`reporter.py`** — all output (terminal Rich table, JSON, Markdown to `results/`). Formats only; makes no decisions. The JSON is serialized before the file is opened (no truncated file), and a same-second rerun gets a `-2` suffix instead of overwriting. `write` is the single entry point for a run; `render_comparison` formats the `compare` delta table; `render_estimate` formats the `--dry-run` output; `make_console()` is the one console factory. `write_from_record` recomputes insights from `results` and uses only the basename of `run_id`. Markdown is written with `errors="replace"`. Filenames are slugified (capped at 60 chars, `benchmark` when empty). In the Markdown per-case table a pipeline-error case shows its status (`API error` / `judge API error` / `judge error` / `internal error`) instead of a score; the terminal output is unchanged. JSON records carry a `schema_version` (from `schemas.SCHEMA_VERSION`).
+- **`compare.py`** — pure run-to-run diff (no I/O beyond `load_record`). `diff` keys combos by `(model, prompt_id)`, emits per-combo quality/cost deltas (latency is not tracked, so it's out of scope), and tolerates schema-version mismatch rather than hard-blocking. `load_record` raises `RecordError` (one-line reason) for a missing/unreadable/non-JSON/too-deep/wrongly-shaped file — it shape-checks every field report, compare and history read (each case needs `match_score`, and a bool `match_result` when scored). `detailed_diff` (the `compare --detailed` mode) drills into per-test-case transitions — `case_diff` classifies each case as regressed/improved/new_error/recovered/etc. (reusing the `match_score is None` = pipeline-error rule, never a real fail) and aggregates a global summary (change counts, worst-regressed cases across combos, per-tag net).
+- **`schemas.py`** — `TypedDict`s documenting the config and result dict shapes (typing only — the pipeline passes plain dicts straight from the YAML; runtime validation is `validator.py`'s job), plus the shared runtime constants `SCHEMA_VERSION`, `DEFAULT_TEMPERATURE`, `DEFAULT_MAX_TOKENS`, `DEFAULT_JUDGE_MAX_TOKENS` imported by runner, validator, reporter, evaluator and estimate.
+
+## Benchmark config schema (as actually consumed)
+
+Configs live in `benchmarks/*.yaml`. The structure the code reads:
+
+- **Templates use Jinja2** — `{{ variable }}`, NOT `{variable}`. Single braces render literally and silently produce wrong prompts (a real bug class). `evalblink validate` warns on this pattern. Variables come from the global `variables:` block merged with each `test_case.variables`.
+- **`inference:`** `temperature`, `max_tokens` (a positive int; `100.0` is an error). Reasoning models (e.g. `poolside/laguna`) consume the budget on reasoning tokens and return null content if `max_tokens` is too low — give headroom.
+- **`evaluation:` block is per-mode:**
+  - `exact_match` cases need no `evaluation:` block at all. The scorer is a plain case-insensitive string comparison — prompt the model to return the label text only, no JSON.
+  - `llm_judge` requires `evaluation.judge_model` (+ optional `judge_threshold`, default 0.70, and `judge_max_tokens`, default 4096 — the judge's own output budget; verbose reasoning judges such as nemotron-3.5-lightning need 2k–6k). Missing `judge_model` raises `ValueError`. A judge API failure (`judge_api_error`) or an unusable reply (`judge_error`) keeps its reason in the case's `error` and prints a `Judge API error on …` / `Judge error on …` warning naming `evaluation.judge_max_tokens` where relevant. Judge scores 1–5 are normalized to 0–1 via `(score - 1) / 4`.
+  - `weighted_match` requires `evaluation.quality_threshold` (0–1 per-case pass threshold) and `evaluation.variables` (list of `{name, weight, tolerance?}`). The scorer has a **fixed output schema**: expected/actual outputs must be JSON arrays of `{use_case, percent, order}` dicts. The three dimension names (`use_case`, `percent`, `order`) are hardcoded in `evaluator.weighted_match` — `evaluation.variables` defines their weights and tolerances, not arbitrary field names.
+- **Three threshold levels** — keep these distinct: `quality_threshold` at the top level (0–100 %) is the CI/CD gate compared against the best combo's pass rate with unscored (pipeline-error) cases counted as failures. `evaluation.quality_threshold` (0–1) is the per-case threshold for `weighted_match`. `evaluation.judge_threshold` (0–1) is the per-case threshold for `llm_judge`. All three coexist in the same config.
+- **`concurrency`** — optional top-level int, 1–100 (default 5); controls `ThreadPoolExecutor` worker count.
+- **`max_cost_usd`** — optional top-level float; `--dry-run` exits 1 if estimated cost exceeds it.
+- **`test_cases[].evaluation`** selects the scorer: `"exact_match"` | `"weighted_match"` | `"llm_judge"`. A single config may mix modes. `exact_match` needs `expected_output`; `llm_judge` needs `criteria`; `tags` drive per-category breakdown.
+- **Latency is not tracked.** `runner.py` records only token counts and cost. The results matrix and compare output contain no timing data.
+
+When fixing a config, keep changes in the YAML and don't alter scoring semantics in `evaluator.py`/`runner.py` unless a run reveals a genuine code defect.

evalblink-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Alex Darmon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evalblink-0.1.0/MANIFEST.in

@@ -0,0 +1,4 @@
+# The sdist carries what the test suite needs to run from it.
+include tests/conftest.py
+include PRD.md CLAUDE.md
+recursive-include benchmarks *.yaml