evalquill 0.1.0__tar.gz

evalquill-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Priteesh Karra
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evalquill-0.1.0/PKG-INFO

@@ -0,0 +1,256 @@
+Metadata-Version: 2.4
+Name: evalquill
+Version: 0.1.0
+Summary: Compare two LLM evaluation runs case by case to see which cases a prompt change broke
+Keywords: llm,evaluation,prompt-engineering,testing
+Author: Priteesh Karra
+Author-email: Priteesh Karra <WritetoPriteesh@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Dist: google-genai>=2.24.0 ; extra == 'gemini'
+Requires-Dist: openai>=1.0 ; extra == 'openai'
+Requires-Python: >=3.10
+Project-URL: Homepage, https://github.com/priteesshhh/evalquill
+Project-URL: Issues, https://github.com/priteesshhh/evalquill/issues
+Provides-Extra: gemini
+Provides-Extra: openai
+Description-Content-Type: text/markdown
+
+# evalquill
+
+[![CI](https://github.com/priteesshhh/evalquill/actions/workflows/ci.yml/badge.svg)](https://github.com/priteesshhh/evalquill/actions/workflows/ci.yml)
+
+A small Python library for checking what changed when you edit an LLM prompt.
+
+You change a prompt, rerun your evals, and the average score moves from 0.72 to 0.75. Did it get better? Maybe. It might also have fixed four cases and broken two that used to work. The average won't tell you that.
+
+evalquill compares two runs case by case and tells you which specific cases got worse.
+
+## Install
+
+Requires Python 3.10 or later. The core has no runtime dependencies.
+
+```bash
+pip install evalquill
+```
+
+The OpenAI and Gemini adapters are optional extras:
+
+```bash
+pip install "evalquill[openai]"
+pip install "evalquill[gemini]"
+```
+
+To work on it locally:
+
+```bash
+git clone https://github.com/priteesshhh/evalquill.git
+cd evalquill
+uv sync
+```
+
+## Quickstart
+
+A metric is a function `(response, expected) -> float` that returns a score from 0 to 1. An LLM is any function `(prompt) -> str`: a provider client, a wrapper around your own model, or a stub.
+
+This example classifies support tickets. Two stub functions stand in for the same model under two versions of a prompt, so it runs without an API key.
+
+```python
+from evalquill import evaluate_dataset
+from evalquill.metrics import normalized_exact_match
+from evalquill.compare import compare, diff_summary
+
+dataset = [
+    {"id": "refund", "prompt": "I was charged twice this month.", "expected": "billing"},
+    {"id": "login", "prompt": "The login page shows a 500 error.", "expected": "account"},
+    {"id": "invite", "prompt": "My teammate never got the invite email.", "expected": "account"},
+    {"id": "webhook", "prompt": "Our webhook stopped receiving events.", "expected": "integration"},
+]
+
+def llm_v1(prompt):
+    return {
+        "I was charged twice this month.": "billing",
+        "The login page shows a 500 error.": "bug",
+        "My teammate never got the invite email.": "account",
+        "Our webhook stopped receiving events.": "bug",
+    }[prompt]
+
+def llm_v2(prompt):
+    return {
+        "I was charged twice this month.": "billing",
+        "The login page shows a 500 error.": "account",
+        "My teammate never got the invite email.": "integration",
+        "Our webhook stopped receiving events.": "integration",
+    }[prompt]
+
+baseline = evaluate_dataset(dataset, llm_v1, [normalized_exact_match])
+candidate = evaluate_dataset(dataset, llm_v2, [normalized_exact_match])
+
+summary = diff_summary(compare(baseline, candidate))
+print(summary["normalized_exact_match"])
+```
+
+```
+{'baseline_mean': 0.5, 'candidate_mean': 0.75, 'mean_delta': 0.25, 'improved': ['login', 'webhook'], 'regressed': ['invite'], 'newly_failing': ['invite']}
+```
+
+Accuracy rose from 0.50 to 0.75. Two cases improved, and `invite`, which v1 classified correctly, v2 now gets wrong. The average alone would not have shown that.
+
+Each result row keeps the input, the response and its scores:
+
+```python
+{"id": "invite", "prompt": "My teammate never got the invite email.",
+ "expected": "account", "response": "integration",
+ "scores": {"normalized_exact_match": 0.0}}
+```
+
+`examples/prompt_change_demo.py` runs the same comparison through saved runs and prints the responses behind each regression.
+
+## Failing a build
+
+A regression check belongs in your test suite. Continuing from the quickstart, with `dataset` and `llm_v2` imported from wherever your project defines them:
+
+```python
+from evalquill import evaluate_dataset, summary_scores, threshold_check
+from evalquill.compare import compare, diff_summary
+from evalquill.metrics import normalized_exact_match
+from evalquill.storage import load_run
+
+def test_accuracy_meets_threshold():
+    candidate = evaluate_dataset(dataset, llm_v2, [normalized_exact_match])
+    checks = threshold_check(summary_scores(candidate), {"normalized_exact_match": 0.7})
+    assert checks == {"normalized_exact_match": "PASS"}
+
+def test_no_case_stops_passing():
+    baseline = load_run("runs/baseline.json")["results"]
+    candidate = evaluate_dataset(dataset, llm_v2, [normalized_exact_match])
+    summary = diff_summary(compare(baseline, candidate))["normalized_exact_match"]
+    assert summary["newly_failing"] == [], f"stopped passing: {summary['newly_failing']}"
+```
+
+The baseline is a saved run (see Saving runs below). With the quickstart's v2 prompt, the first test passes, since 0.75 clears 0.7, and the second fails:
+
+```
+AssertionError: stopped passing: ['invite']
+```
+
+The accuracy bar was met and a case still broke. A threshold alone lets that through.
+
+`threshold_check` returns the strings `"PASS"`, `"FAIL"` and `"NOT RUN"`, so compare them explicitly. `assert all(checks.values())` would pass on `"FAIL"`, because every non-empty string is truthy.
+
+## Saving runs
+
+Save a run once and compare against it later, without paying for the same model calls again:
+
+```python
+from evalquill.storage import save_run, load_run
+
+save_run(baseline, "runs/baseline.json", metadata={"prompt_version": "v1"})
+stored = load_run("runs/baseline.json")
+compare(stored["results"], candidate)
+```
+
+Metadata is free-form. Record the model, its settings and the prompt version; that is what makes a saved run interpretable weeks later.
+
+## Writing a metric
+
+Any named function works. This one checks a field in a JSON response:
+
+```python
+import json
+from evalquill import evaluate
+
+def json_label(response: str, expected: str) -> float:
+    try:
+        return 1.0 if json.loads(response).get("label") == expected else 0.0
+    except (json.JSONDecodeError, AttributeError):
+        return 0.0
+
+print(evaluate(expected="billing", response='{"label": "billing"}', metrics=[json_label]))
+```
+
+```
+{'json_label': 1.0}
+```
+
+The rules: the function's name is the score's key and must be unique within a run. The return value must be a finite number from 0 to 1; booleans are rejected. If a metric raises, the run stops and the error names the case.
+
+## Using a real model
+
+Wrap your model in a function that takes a prompt and returns a string. For a self-hosted model, that is usually a few lines around your inference client.
+
+For OpenAI and Gemini there are adapters. Install the matching extra; the key is read from `OPENAI_API_KEY` or `GEMINI_API_KEY`:
+
+```python
+from evalquill.providers import openai_llm
+
+llm, metadata = openai_llm(
+    "gpt-4o-mini",
+    system_prompt="Classify the support ticket as billing, account, integration, feature, bug or other. Reply with one word.",
+    temperature=0.0,
+)
+results = evaluate_dataset(dataset, llm, [normalized_exact_match])
+save_run(results, "runs/gpt-4o-mini_v2.json", metadata=metadata)
+```
+
+`gemini_llm` works the same way. The returned `metadata` records the provider, model, temperature and system prompt, so a saved run says what produced it. The adapters add no retry logic of their own: the provider SDK's default retries apply, and any error that remains stops the run and names the case.
+
+## Why case IDs matter
+
+Comparisons join on case ID, not list position. If you reorder your dataset or insert a case, a positional comparison silently diffs unrelated rows and gives you a confident wrong answer.
+
+IDs default to `case_0`, `case_1` and so on if you don't supply them. That's fine for a one-off run, but give real IDs to anything you'll compare across versions.
+
+Two runs are rejected rather than compared if they cover different cases, use different metrics, or if any case's input or expected answer differs between them. Only the response and its scores may change.
+
+## Case study
+
+[case_study/](https://github.com/priteesshhh/evalquill/tree/v0.1.0/case_study) contains a worked example: 50 fictional support tickets, hand-labeled against a written guide, classified by gpt-4o-mini under two prompt versions. Results are reported on a frozen 20-ticket held-out set, with three repeats per version on that set.
+
+Accuracy rose from 80% to 95% on the held-out set, and one previously correct ticket failed in all three observed repeats. The aggregate score does not surface that; the per-case comparison does.
+
+All runs are saved. `case_study/rescore.py` recomputes every stored score from the saved responses, with no API key. See [case_study/README.md](https://github.com/priteesshhh/evalquill/blob/v0.1.0/case_study/README.md) for method, results and limitations.
+
+## What it doesn't do
+
+- No semantic or model-graded metrics. The built-ins are string comparisons, named for what they check: `normalized_exact_match` ignores case and surrounding whitespace, and `contains_substring` checks for a substring and nothing more. `contains_substring("19", "9")` returns 1.0, and a correct answer in a different form, such as "nine" for "9", scores 0.0.
+- No RAG metrics. Metrics receive only the response and the expected value, so anything needing retrieved context doesn't fit the signature yet.
+- No concurrency. Model calls are made one at a time.
+- No repeat-run tooling in the library. The case study has scripts for its own repeats.
+- No dashboard, no hosted anything.
+
+This isn't the only tool in this space. promptfoo, DeepEval, LangSmith and Braintrust all overlap with it and most do more.
+
+## Design notes
+
+Scores are stored at full precision and rounded only for display. Rounding before a threshold check lets a true score of 0.7993 pass a 0.80 threshold, a silent false pass. There's a regression test for it.
+
+Metrics must return a finite number between 0 and 1. Booleans are rejected explicitly: `True == 1.0` in Python, so a metric returning booleans would otherwise pass validation while suggesting its author didn't mean to return a score.
+
+Duplicate metric names and duplicate case IDs are rejected. Both silently overwrite results otherwise. A missing score is rejected rather than averaged as zero.
+
+A comparison requires each case's input and expected answer to be identical in both runs. Otherwise a changed reference answer would show up as a model improvement.
+
+The whole dataset is validated before the first model call, so a bad row near the end doesn't cost every call before it. When a model call or a metric fails, the run stops and the error names the case. Recording 0.0 instead would make a broken metric look like a failing model.
+
+Saves are atomic: a failed save leaves any previous file intact. Loading validates the whole file, not just its schema version.
+
+## Tests
+
+```bash
+uv run pytest
+```
+
+`tests/test_readme.py` runs every Python example in this README and checks the output shown, so the examples can't drift from the code. Live tests against real APIs are opt-in: `uv run pytest -m live`, with the provider keys set.
+
+## Status
+
+Early. 0.1.0 is the first release; the API may change in later 0.x releases.
+
+## License
+
+MIT

evalquill-0.1.0/README.md

@@ -0,0 +1,234 @@
+# evalquill
+
+[![CI](https://github.com/priteesshhh/evalquill/actions/workflows/ci.yml/badge.svg)](https://github.com/priteesshhh/evalquill/actions/workflows/ci.yml)
+
+A small Python library for checking what changed when you edit an LLM prompt.
+
+You change a prompt, rerun your evals, and the average score moves from 0.72 to 0.75. Did it get better? Maybe. It might also have fixed four cases and broken two that used to work. The average won't tell you that.
+
+evalquill compares two runs case by case and tells you which specific cases got worse.
+
+## Install
+
+Requires Python 3.10 or later. The core has no runtime dependencies.
+
+```bash
+pip install evalquill
+```
+
+The OpenAI and Gemini adapters are optional extras:
+
+```bash
+pip install "evalquill[openai]"
+pip install "evalquill[gemini]"
+```
+
+To work on it locally:
+
+```bash
+git clone https://github.com/priteesshhh/evalquill.git
+cd evalquill
+uv sync
+```
+
+## Quickstart
+
+A metric is a function `(response, expected) -> float` that returns a score from 0 to 1. An LLM is any function `(prompt) -> str`: a provider client, a wrapper around your own model, or a stub.
+
+This example classifies support tickets. Two stub functions stand in for the same model under two versions of a prompt, so it runs without an API key.
+
+```python
+from evalquill import evaluate_dataset
+from evalquill.metrics import normalized_exact_match
+from evalquill.compare import compare, diff_summary
+
+dataset = [
+    {"id": "refund", "prompt": "I was charged twice this month.", "expected": "billing"},
+    {"id": "login", "prompt": "The login page shows a 500 error.", "expected": "account"},
+    {"id": "invite", "prompt": "My teammate never got the invite email.", "expected": "account"},
+    {"id": "webhook", "prompt": "Our webhook stopped receiving events.", "expected": "integration"},
+]
+
+def llm_v1(prompt):
+    return {
+        "I was charged twice this month.": "billing",
+        "The login page shows a 500 error.": "bug",
+        "My teammate never got the invite email.": "account",
+        "Our webhook stopped receiving events.": "bug",
+    }[prompt]
+
+def llm_v2(prompt):
+    return {
+        "I was charged twice this month.": "billing",
+        "The login page shows a 500 error.": "account",
+        "My teammate never got the invite email.": "integration",
+        "Our webhook stopped receiving events.": "integration",
+    }[prompt]
+
+baseline = evaluate_dataset(dataset, llm_v1, [normalized_exact_match])
+candidate = evaluate_dataset(dataset, llm_v2, [normalized_exact_match])
+
+summary = diff_summary(compare(baseline, candidate))
+print(summary["normalized_exact_match"])
+```
+
+```
+{'baseline_mean': 0.5, 'candidate_mean': 0.75, 'mean_delta': 0.25, 'improved': ['login', 'webhook'], 'regressed': ['invite'], 'newly_failing': ['invite']}
+```
+
+Accuracy rose from 0.50 to 0.75. Two cases improved, and `invite`, which v1 classified correctly, v2 now gets wrong. The average alone would not have shown that.
+
+Each result row keeps the input, the response and its scores:
+
+```python
+{"id": "invite", "prompt": "My teammate never got the invite email.",
+ "expected": "account", "response": "integration",
+ "scores": {"normalized_exact_match": 0.0}}
+```
+
+`examples/prompt_change_demo.py` runs the same comparison through saved runs and prints the responses behind each regression.
+
+## Failing a build
+
+A regression check belongs in your test suite. Continuing from the quickstart, with `dataset` and `llm_v2` imported from wherever your project defines them:
+
+```python
+from evalquill import evaluate_dataset, summary_scores, threshold_check
+from evalquill.compare import compare, diff_summary
+from evalquill.metrics import normalized_exact_match
+from evalquill.storage import load_run
+
+def test_accuracy_meets_threshold():
+    candidate = evaluate_dataset(dataset, llm_v2, [normalized_exact_match])
+    checks = threshold_check(summary_scores(candidate), {"normalized_exact_match": 0.7})
+    assert checks == {"normalized_exact_match": "PASS"}
+
+def test_no_case_stops_passing():
+    baseline = load_run("runs/baseline.json")["results"]
+    candidate = evaluate_dataset(dataset, llm_v2, [normalized_exact_match])
+    summary = diff_summary(compare(baseline, candidate))["normalized_exact_match"]
+    assert summary["newly_failing"] == [], f"stopped passing: {summary['newly_failing']}"
+```
+
+The baseline is a saved run (see Saving runs below). With the quickstart's v2 prompt, the first test passes, since 0.75 clears 0.7, and the second fails:
+
+```
+AssertionError: stopped passing: ['invite']
+```
+
+The accuracy bar was met and a case still broke. A threshold alone lets that through.
+
+`threshold_check` returns the strings `"PASS"`, `"FAIL"` and `"NOT RUN"`, so compare them explicitly. `assert all(checks.values())` would pass on `"FAIL"`, because every non-empty string is truthy.
+
+## Saving runs
+
+Save a run once and compare against it later, without paying for the same model calls again:
+
+```python
+from evalquill.storage import save_run, load_run
+
+save_run(baseline, "runs/baseline.json", metadata={"prompt_version": "v1"})
+stored = load_run("runs/baseline.json")
+compare(stored["results"], candidate)
+```
+
+Metadata is free-form. Record the model, its settings and the prompt version; that is what makes a saved run interpretable weeks later.
+
+## Writing a metric
+
+Any named function works. This one checks a field in a JSON response:
+
+```python
+import json
+from evalquill import evaluate
+
+def json_label(response: str, expected: str) -> float:
+    try:
+        return 1.0 if json.loads(response).get("label") == expected else 0.0
+    except (json.JSONDecodeError, AttributeError):
+        return 0.0
+
+print(evaluate(expected="billing", response='{"label": "billing"}', metrics=[json_label]))
+```
+
+```
+{'json_label': 1.0}
+```
+
+The rules: the function's name is the score's key and must be unique within a run. The return value must be a finite number from 0 to 1; booleans are rejected. If a metric raises, the run stops and the error names the case.
+
+## Using a real model
+
+Wrap your model in a function that takes a prompt and returns a string. For a self-hosted model, that is usually a few lines around your inference client.
+
+For OpenAI and Gemini there are adapters. Install the matching extra; the key is read from `OPENAI_API_KEY` or `GEMINI_API_KEY`:
+
+```python
+from evalquill.providers import openai_llm
+
+llm, metadata = openai_llm(
+    "gpt-4o-mini",
+    system_prompt="Classify the support ticket as billing, account, integration, feature, bug or other. Reply with one word.",
+    temperature=0.0,
+)
+results = evaluate_dataset(dataset, llm, [normalized_exact_match])
+save_run(results, "runs/gpt-4o-mini_v2.json", metadata=metadata)
+```
+
+`gemini_llm` works the same way. The returned `metadata` records the provider, model, temperature and system prompt, so a saved run says what produced it. The adapters add no retry logic of their own: the provider SDK's default retries apply, and any error that remains stops the run and names the case.
+
+## Why case IDs matter
+
+Comparisons join on case ID, not list position. If you reorder your dataset or insert a case, a positional comparison silently diffs unrelated rows and gives you a confident wrong answer.
+
+IDs default to `case_0`, `case_1` and so on if you don't supply them. That's fine for a one-off run, but give real IDs to anything you'll compare across versions.
+
+Two runs are rejected rather than compared if they cover different cases, use different metrics, or if any case's input or expected answer differs between them. Only the response and its scores may change.
+
+## Case study
+
+[case_study/](https://github.com/priteesshhh/evalquill/tree/v0.1.0/case_study) contains a worked example: 50 fictional support tickets, hand-labeled against a written guide, classified by gpt-4o-mini under two prompt versions. Results are reported on a frozen 20-ticket held-out set, with three repeats per version on that set.
+
+Accuracy rose from 80% to 95% on the held-out set, and one previously correct ticket failed in all three observed repeats. The aggregate score does not surface that; the per-case comparison does.
+
+All runs are saved. `case_study/rescore.py` recomputes every stored score from the saved responses, with no API key. See [case_study/README.md](https://github.com/priteesshhh/evalquill/blob/v0.1.0/case_study/README.md) for method, results and limitations.
+
+## What it doesn't do
+
+- No semantic or model-graded metrics. The built-ins are string comparisons, named for what they check: `normalized_exact_match` ignores case and surrounding whitespace, and `contains_substring` checks for a substring and nothing more. `contains_substring("19", "9")` returns 1.0, and a correct answer in a different form, such as "nine" for "9", scores 0.0.
+- No RAG metrics. Metrics receive only the response and the expected value, so anything needing retrieved context doesn't fit the signature yet.
+- No concurrency. Model calls are made one at a time.
+- No repeat-run tooling in the library. The case study has scripts for its own repeats.
+- No dashboard, no hosted anything.
+
+This isn't the only tool in this space. promptfoo, DeepEval, LangSmith and Braintrust all overlap with it and most do more.
+
+## Design notes
+
+Scores are stored at full precision and rounded only for display. Rounding before a threshold check lets a true score of 0.7993 pass a 0.80 threshold, a silent false pass. There's a regression test for it.
+
+Metrics must return a finite number between 0 and 1. Booleans are rejected explicitly: `True == 1.0` in Python, so a metric returning booleans would otherwise pass validation while suggesting its author didn't mean to return a score.
+
+Duplicate metric names and duplicate case IDs are rejected. Both silently overwrite results otherwise. A missing score is rejected rather than averaged as zero.
+
+A comparison requires each case's input and expected answer to be identical in both runs. Otherwise a changed reference answer would show up as a model improvement.
+
+The whole dataset is validated before the first model call, so a bad row near the end doesn't cost every call before it. When a model call or a metric fails, the run stops and the error names the case. Recording 0.0 instead would make a broken metric look like a failing model.
+
+Saves are atomic: a failed save leaves any previous file intact. Loading validates the whole file, not just its schema version.
+
+## Tests
+
+```bash
+uv run pytest
+```
+
+`tests/test_readme.py` runs every Python example in this README and checks the output shown, so the examples can't drift from the code. Live tests against real APIs are opt-in: `uv run pytest -m live`, with the provider keys set.
+
+## Status
+
+Early. 0.1.0 is the first release; the API may change in later 0.x releases.
+
+## License
+
+MIT

evalquill-0.1.0/pyproject.toml

@@ -0,0 +1,45 @@
+[project]
+name = "evalquill"
+version = "0.1.0"
+description = "Compare two LLM evaluation runs case by case to see which cases a prompt change broke"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = [
+    "llm",
+    "evaluation",
+    "prompt-engineering",
+    "testing",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Testing",
+]
+dependencies = []
+
+[[project.authors]]
+name = "Priteesh Karra"
+email = "WritetoPriteesh@gmail.com"
+
+[project.optional-dependencies]
+openai = ["openai>=1.0"]
+gemini = ["google-genai>=2.24.0"]
+
+[project.urls]
+Homepage = "https://github.com/priteesshhh/evalquill"
+Issues = "https://github.com/priteesshhh/evalquill/issues"
+
+[dependency-groups]
+dev = ["pytest>=8.0"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = ["live: hits a real API, costs money, deselected by default"]
+addopts = "-m 'not live'"
+
+[build-system]
+requires = ["uv_build>=0.12.10,<0.13.0"]
+build-backend = "uv_build"

evalquill-0.1.0/pyproject.toml.orig

@@ -0,0 +1,45 @@
+[project]
+name = "evalquill"
+version = "0.1.0"
+description = "Compare two LLM evaluation runs case by case to see which cases a prompt change broke"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "Priteesh Karra", email = "WritetoPriteesh@gmail.com" }
+]
+keywords = ["llm", "evaluation", "prompt-engineering", "testing"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Testing",
+]
+dependencies = []
+
+[project.optional-dependencies]
+openai = [
+    "openai>=1.0",
+]
+gemini = [
+    "google-genai>=2.24.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/priteesshhh/evalquill"
+Issues = "https://github.com/priteesshhh/evalquill/issues"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = ["live: hits a real API, costs money, deselected by default"]
+addopts = "-m 'not live'"
+
+[build-system]
+requires = ["uv_build>=0.12.10,<0.13.0"]
+build-backend = "uv_build"

evalquill-0.1.0/src/evalquill/__init__.py

@@ -0,0 +1,166 @@
+import math
+
+
+def evaluate(expected: str, response: str, metrics: list) -> dict:
+    _validate_metrics(metrics)
+    results = {}
+    for metric in metrics:
+        score = metric(response, expected)
+        _validate_score(score, metric.__name__)
+        results[metric.__name__] = score
+    return results
+
+
+def _preflight(dataset, metrics) -> list:
+    """Validate the whole dataset and metric list; return the case ids.
+
+    Runs before any model call, so a structural error anywhere in the
+    dataset costs nothing.
+    """
+    _validate_metrics(metrics)
+    if not isinstance(dataset, list):
+        raise ValueError(f"dataset must be a list of cases, got {type(dataset).__name__}.")
+
+    ids = []
+    seen = set()
+    for i, item in enumerate(dataset):
+        if not isinstance(item, dict):
+            raise ValueError(f"Case {i} is not a dict, got {type(item).__name__}.")
+        missing = [f for f in ("prompt", "expected") if f not in item]
+        if missing:
+            raise ValueError(f"Case {i} is missing required field(s): {missing}.")
+        for field in ("prompt", "expected"):
+            if not isinstance(item[field], str):
+                raise ValueError(
+                    f"Case {i}: '{field}' must be a string, got {type(item[field]).__name__}."
+                )
+        case_id = item.get("id", f"case_{i}")
+        if case_id in seen:
+            raise ValueError(f"Duplicate case id: '{case_id}'. Each case must have a unique id.")
+        seen.add(case_id)
+        ids.append(case_id)
+    return ids
+
+
+def evaluate_dataset(dataset: list, llm, metrics: list) -> list:
+    """Call llm on each case and score the response.
+
+    The dataset and metrics are validated in full before the first call.
+    LLM and metric failures are re-raised as RuntimeError naming the case;
+    the original exception is available as __cause__.
+    """
+    ids = _preflight(dataset, metrics)
+
+    results = []
+    for case_id, item in zip(ids, dataset):
+        try:
+            response = llm(item["prompt"])
+        except Exception as e:
+            raise RuntimeError(f"LLM call failed on case '{case_id}' — {e}") from e
+
+        if not isinstance(response, str):
+            raise RuntimeError(
+                f"LLM returned {type(response).__name__} for case '{case_id}'; expected a string."
+            )
+
+        try:
+            result = evaluate(
+                expected=item["expected"],
+                response=response,
+                metrics=metrics
+            )
+        except Exception as e:
+            raise RuntimeError(
+                f"Evaluation failed on case '{case_id}' "
+                f"(prompt: '{item['prompt']}') — {e}"
+            ) from e
+
+        results.append({
+            "id": case_id,
+            "prompt": item["prompt"],
+            "expected": item["expected"],
+            "response": response,
+            "scores": result
+        })
+    return results
+
+
+def summary_scores(results: list) -> dict:
+    """Mean of each metric across results, at full precision.
+
+    Every row must carry the same metric set with valid scores. A row missing
+    a metric is rejected rather than averaged as zero.
+    """
+    if not results:
+        return {}
+
+    metric_set = None
+    totals = {}
+    for index, result in enumerate(results):
+        if not isinstance(result, dict) or not isinstance(result.get("scores"), dict):
+            raise ValueError(f"Result {index} has no scores dict.")
+        scores = result["scores"]
+
+        names = set(scores)
+        if metric_set is None:
+            metric_set = names
+        elif names != metric_set:
+            raise ValueError(
+                f"Inconsistent metric sets: result 0 has {sorted(metric_set)}, "
+                f"result {index} has {sorted(names)}. A missing score would "
+                "otherwise be averaged as zero."
+            )
+
+        for name, score in scores.items():
+            _validate_score(score, name)
+            totals[name] = totals.get(name, 0.0) + score
+
+    return {name: total / len(results) for name, total in totals.items()}
+
+
+def threshold_check(summary: dict, thresholds: dict) -> dict:
+    """PASS, FAIL or NOT RUN per threshold.
+
+    NOT RUN means the metric was never measured. Invalid thresholds or
+    summary values are rejected rather than compared.
+    """
+    for metric, minimum in thresholds.items():
+        try:
+            _validate_score(minimum, metric)
+        except ValueError as e:
+            raise ValueError(f"Invalid threshold for '{metric}': {e}") from e
+
+    results = {}
+    for metric, minimum in thresholds.items():
+        if metric not in summary:
+            results[metric] = "NOT RUN"
+            continue
+        _validate_score(summary[metric], metric)
+        results[metric] = "PASS" if summary[metric] >= minimum else "FAIL"
+    return results
+
+
+def _validate_metrics(metrics) -> None:
+    if not isinstance(metrics, (list, tuple)) or not metrics:
+        raise ValueError("At least one metric is required; a run without metrics measures nothing.")
+    seen = set()
+    for metric in metrics:
+        if not callable(metric):
+            raise ValueError(f"Each metric must be callable, got {type(metric).__name__}.")
+        name = getattr(metric, "__name__", None)
+        if not isinstance(name, str):
+            raise ValueError("Each metric must have a __name__; use a named function.")
+        if name in seen:
+            raise ValueError(f"Duplicate metric name: '{name}'. Each metric must have a unique name.")
+        seen.add(name)
+
+
+def _validate_score(value: float, name: str) -> None:
+    if isinstance(value, bool):
+        raise ValueError(f"Score '{name}' must be a float, not a boolean.")
+    if not isinstance(value, (int, float)):
+        raise ValueError(f"Score '{name}' must be a number, got {type(value).__name__}.")
+    if math.isnan(value) or math.isinf(value):
+        raise ValueError(f"Score '{name}' must be finite, got {value}.")
+    if not (0.0 <= value <= 1.0):
+        raise ValueError(f"Score '{name}' must be between 0 and 1, got {value}.")

evalquill-0.1.0/src/evalquill/compare.py

@@ -0,0 +1,104 @@
+"""Compare two evaluation runs case by case.
+
+Comparison contract: for each case id, the input (prompt) and reference
+answer (expected) must be identical in both runs. Only the response and its
+scores may differ. The experimental variable - a system prompt, template or
+model - lives in the llm callable and run metadata, never in these fields.
+A changed input or reference answer is a different test case, so the runs
+are rejected rather than compared.
+"""
+
+from . import _validate_score
+from .validation import validate_results
+
+
+def compare(baseline: list, candidate: list) -> dict:
+    base_idx, base_metrics = validate_results(baseline, "baseline")
+    cand_idx, cand_metrics = validate_results(candidate, "candidate")
+
+    if base_idx.keys() != cand_idx.keys():
+        only_base = sorted(base_idx.keys() - cand_idx.keys())
+        only_cand = sorted(cand_idx.keys() - base_idx.keys())
+        raise ValueError(
+            "Runs cover different cases and cannot be compared. "
+            f"Only in baseline: {only_base}. Only in candidate: {only_cand}."
+        )
+
+    if base_metrics != cand_metrics:
+        raise ValueError(
+            "Runs used different metrics and cannot be compared. "
+            f"Baseline: {sorted(base_metrics)}. Candidate: {sorted(cand_metrics)}."
+        )
+
+    changed_input = [i for i in base_idx if base_idx[i]["prompt"] != cand_idx[i]["prompt"]]
+    changed_expected = [i for i in base_idx if base_idx[i]["expected"] != cand_idx[i]["expected"]]
+    if changed_input or changed_expected:
+        problems = []
+        if changed_input:
+            problems.append(f"different input for cases {sorted(changed_input)}")
+        if changed_expected:
+            problems.append(f"different reference answer for cases {sorted(changed_expected)}")
+        raise ValueError(
+            "Runs are not over the same dataset: " + "; ".join(problems) + ". "
+            "A changed input or reference answer would be reported as a model "
+            "improvement or regression. Compare only runs over identical cases."
+        )
+
+    metrics = sorted(base_metrics)
+    cases = []
+    for case_id in base_idx:
+        base = base_idx[case_id]
+        cand = cand_idx[case_id]
+        cases.append({
+            "id": case_id,
+            "prompt": base["prompt"],
+            "expected": base["expected"],
+            "baseline_response": base["response"],
+            "candidate_response": cand["response"],
+            "response_changed": base["response"] != cand["response"],
+            "baseline_scores": dict(base["scores"]),
+            "candidate_scores": dict(cand["scores"]),
+            "deltas": {n: cand["scores"][n] - base["scores"][n] for n in metrics},
+        })
+
+    return {"cases": cases, "metrics": metrics}
+
+
+def diff_summary(diff: dict, pass_threshold: float = 1.0) -> dict:
+    try:
+        _validate_score(pass_threshold, "pass_threshold")
+    except ValueError as e:
+        raise ValueError(
+            f"pass_threshold must be a finite number between 0 and 1, got {pass_threshold!r}."
+        ) from e
+
+    summary = {}
+    for name in diff["metrics"]:
+        improved, regressed, newly_failing = [], [], []
+
+        for case in diff["cases"]:
+            delta = case["deltas"][name]
+            if delta > 0:
+                improved.append(case["id"])
+            elif delta < 0:
+                regressed.append(case["id"])
+
+            was_passing = case["baseline_scores"][name] >= pass_threshold
+            now_passing = case["candidate_scores"][name] >= pass_threshold
+            if was_passing and not now_passing:
+                newly_failing.append(case["id"])
+
+        n = len(diff["cases"])
+        baseline_mean = sum(c["baseline_scores"][name] for c in diff["cases"]) / n
+        candidate_mean = sum(c["candidate_scores"][name] for c in diff["cases"]) / n
+
+        summary[name] = {
+            "baseline_mean": baseline_mean,
+            "candidate_mean": candidate_mean,
+            "mean_delta": candidate_mean - baseline_mean,
+            "improved": improved,
+            "regressed": regressed,
+            "newly_failing": newly_failing,
+        }
+
+    return summary

evalquill-0.1.0/src/evalquill/llm.py

@@ -0,0 +1,10 @@
+def mock_llm(prompt: str) -> str:
+    responses = {
+        "what is the capital of france?": "The capital of France is Paris.",
+        "what is 2 + 2?": "2 + 2 equals 4.",
+        "what is the capital of germany?": "The capital of Germany is Berlin.",
+        "what is the capital of india?": "The capital of India is Delhi.",
+        "what is 7 + 2?": "7 + 2 equals 9.",
+        "what is first president of united states of america?": "The first president of the United States of America is George Washington.",
+    }
+    return responses.get(prompt.strip().lower(), "I don't know.")

evalquill-0.1.0/src/evalquill/metrics.py

@@ -0,0 +1,10 @@
+def normalized_exact_match(response: str, expected: str) -> float:
+    if not expected.strip():
+        raise ValueError("expected string cannot be empty")
+    return 1.0 if response.strip().lower() == expected.strip().lower() else 0.0
+
+
+def contains_substring(response: str, expected: str) -> float:
+    if not expected.strip():
+        raise ValueError("expected string cannot be empty")
+    return 1.0 if expected.strip().lower() in response.strip().lower() else 0.0

evalquill-0.1.0/src/evalquill/providers.py

@@ -0,0 +1,109 @@
+"""Adapters turning provider clients into plain llm(prompt) -> str callables."""
+
+
+def openai_llm(
+    model: str,
+    system_prompt: str | None = None,
+    temperature: float = 0.0,
+    client=None,
+    **kwargs,
+):
+    """Build an llm(prompt) -> str callable backed by OpenAI chat completions.
+
+    Returns (llm, metadata). Pass metadata to save_run so the run records
+    which model and settings produced it.
+
+    client is injectable for testing; when omitted, a real OpenAI client is
+    constructed and the openai package must be installed.
+    """
+    if client is None:
+        try:
+            from openai import OpenAI
+        except ImportError as e:
+            raise ImportError(
+                "The openai package is required for openai_llm. "
+                "Install it with: pip install openai"
+            ) from e
+        client = OpenAI()
+
+    def llm(prompt: str) -> str:
+        messages = []
+        if system_prompt is not None:
+            messages.append({"role": "system", "content": system_prompt})
+        messages.append({"role": "user", "content": prompt})
+
+        response = client.chat.completions.create(
+            model=model,
+            messages=messages,
+            temperature=temperature,
+            **kwargs,
+        )
+
+        content = response.choices[0].message.content
+        if content is None:
+            raise RuntimeError(
+                f"Model '{model}' returned no content. "
+                "This can happen when the response was filtered or truncated."
+            )
+        return content
+
+    metadata = {
+        "provider": "openai",
+        "model": model,
+        "temperature": temperature,
+        "system_prompt": system_prompt,
+        **kwargs,
+    }
+
+    return llm, metadata
+
+def gemini_llm(
+    model: str = "gemini-3.6-flash",
+    system_prompt: str | None = None,
+    temperature: float = 0.0,
+    client=None,
+    **kwargs,
+):
+    """Build an llm(prompt) -> str callable backed by the Gemini API.
+
+    Returns (llm, metadata). Requires GEMINI_API_KEY in the environment
+    when client is not injected.
+    """
+    if client is None:
+        try:
+            from google import genai
+        except ImportError as e:
+            raise ImportError(
+                "The google-genai package is required for gemini_llm. "
+                "Install it with: pip install google-genai"
+            ) from e
+        client = genai.Client()
+
+    def llm(prompt: str) -> str:
+        config = {"temperature": temperature, **kwargs}
+        if system_prompt is not None:
+            config["system_instruction"] = system_prompt
+
+        response = client.models.generate_content(
+            model=model,
+            contents=prompt,
+            config=config,
+        )
+
+        text = response.text
+        if text is None:
+            raise RuntimeError(
+                f"Model '{model}' returned no content. "
+                "This can happen when the response was filtered or truncated."
+            )
+        return text
+
+    metadata = {
+        "provider": "gemini",
+        "model": model,
+        "temperature": temperature,
+        "system_prompt": system_prompt,
+        **kwargs,
+    }
+
+    return llm, metadata

evalquill-0.1.0/src/evalquill/storage.py

@@ -0,0 +1,101 @@
+"""Save and load evaluation runs as JSON."""
+
+import json
+import os
+import tempfile
+from datetime import datetime, timezone
+from pathlib import Path
+
+from .validation import validate_results
+
+SCHEMA_VERSION = 1
+
+
+def save_run(results: list, path: str, metadata: dict | None = None) -> None:
+    """Write a run to path atomically.
+
+    Rows are validated before anything is written. The payload is then
+    serialized in full, written to a temporary file in the same directory,
+    and moved onto path with os.replace. A failure at any stage leaves an
+    existing file at path unchanged and leaves no partial file behind. An
+    existing file at path is replaced on success.
+
+    Non-finite floats (NaN, infinity) are rejected, since they are not valid
+    JSON.
+    """
+    if not results:
+        raise ValueError("Cannot save an empty run.")
+    if metadata is not None and not isinstance(metadata, dict):
+        raise ValueError("metadata must be a dict.")
+    validate_results(results, "run")
+
+    payload = {
+        "schema_version": SCHEMA_VERSION,
+        "created_at": datetime.now(timezone.utc).isoformat(),
+        "metadata": metadata or {},
+        "results": results,
+    }
+
+    # Serialize first: any TypeError or ValueError happens before disk I/O.
+    text = json.dumps(payload, indent=2, ensure_ascii=False, allow_nan=False)
+
+    target = Path(path)
+    target.parent.mkdir(parents=True, exist_ok=True)
+
+    fd, tmp = tempfile.mkstemp(
+        dir=target.parent, prefix=f".{target.name}.", suffix=".tmp"
+    )
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8") as f:
+            f.write(text)
+            f.flush()
+            os.fsync(f.fileno())
+        os.replace(tmp, target)
+    except BaseException:
+        try:
+            os.unlink(tmp)
+        except FileNotFoundError:
+            pass
+        raise
+
+
+def load_run(path: str) -> dict:
+    """Load a run and validate its envelope and every row.
+
+    A matching schema_version is necessary but not sufficient: the file must
+    also contain valid metadata and a non-empty list of valid result rows.
+    """
+    target = Path(path)
+    if not target.exists():
+        raise FileNotFoundError(f"No run found at '{path}'.")
+
+    def reject_constant(token):
+        raise ValueError(f"'{path}' contains non-standard JSON value {token}.")
+
+    with target.open(encoding="utf-8") as f:
+        try:
+            payload = json.load(f, parse_constant=reject_constant)
+        except json.JSONDecodeError as e:
+            raise ValueError(f"'{path}' is not valid JSON: {e}") from e
+
+    if not isinstance(payload, dict) or "schema_version" not in payload:
+        raise ValueError(f"'{path}' is not an evalquill run file.")
+
+    if payload["schema_version"] != SCHEMA_VERSION:
+        raise ValueError(
+            f"Run file '{path}' uses schema version {payload['schema_version']}, "
+            f"but this version of evalquill reads version {SCHEMA_VERSION}."
+        )
+
+    for field in ("created_at", "metadata", "results"):
+        if field not in payload:
+            raise ValueError(f"'{path}' is missing required field '{field}' (results cannot be read).")
+    if not isinstance(payload["created_at"], str):
+        raise ValueError(f"'{path}': 'created_at' must be a string.")
+    if not isinstance(payload["metadata"], dict):
+        raise ValueError(f"'{path}': 'metadata' must be a dict.")
+    if not isinstance(payload["results"], list):
+        raise ValueError(f"'{path}': 'results' must be a list.")
+
+    validate_results(payload["results"], f"'{path}'")
+    return payload

evalquill-0.1.0/src/evalquill/validation.py

@@ -0,0 +1,52 @@
+"""Shared validation for result rows and runs.
+
+One definition of a valid run, used by comparison and by storage, so the two
+boundaries cannot drift apart.
+"""
+
+from . import _validate_score
+
+REQUIRED_FIELDS = ("id", "prompt", "expected", "response", "scores")
+
+
+def check_row(row, label: str, index: int) -> None:
+    if not isinstance(row, dict):
+        raise ValueError(f"{label} row {index} is not a dict.")
+    missing = [f for f in REQUIRED_FIELDS if f not in row]
+    if missing:
+        raise ValueError(f"{label} row {index} is missing required field(s): {missing}.")
+    if not isinstance(row["scores"], dict):
+        raise ValueError(f"{label} case '{row['id']}': scores must be a dict.")
+    for name, value in row["scores"].items():
+        try:
+            _validate_score(value, name)
+        except ValueError as e:
+            raise ValueError(f"{label} case '{row['id']}': {e}") from e
+
+
+def validate_results(results, label: str):
+    """Validate every row; return (rows indexed by id, the run's metric set)."""
+    if not isinstance(results, list):
+        raise ValueError(f"{label} results must be a list.")
+    if not results:
+        raise ValueError(f"{label} run must contain at least one case.")
+
+    indexed = {}
+    metric_set = None
+    first_id = None
+    for index, row in enumerate(results):
+        check_row(row, label, index)
+        case_id = row["id"]
+        if case_id in indexed:
+            raise ValueError(f"Duplicate case id '{case_id}' in {label} run.")
+        indexed[case_id] = row
+
+        names = set(row["scores"])
+        if metric_set is None:
+            metric_set, first_id = names, case_id
+        elif names != metric_set:
+            raise ValueError(
+                f"{label} run has inconsistent metric sets: case '{first_id}' "
+                f"has {sorted(metric_set)}, case '{case_id}' has {sorted(names)}."
+            )
+    return indexed, metric_set