evalgate-ai 0.1.0__tar.gz

evalgate_ai-0.1.0/.gitattributes

@@ -0,0 +1 @@
+.githooks/* text eol=lf

evalgate_ai-0.1.0/.githooks/pre-push

@@ -0,0 +1,12 @@
+#!/bin/sh
+# Local replacement for CI: lint, format, types and tests must pass before pushing.
+# Enable once per clone: git config core.hooksPath .githooks
+set -e
+cd "$(git rev-parse --show-toplevel)"
+# Use the project venv (created by `uv sync`); Scripts/ on Windows, bin/ elsewhere.
+PATH="$PWD/.venv/Scripts:$PWD/.venv/bin:$PATH"
+
+echo "pre-push: ruff check" && ruff check .
+echo "pre-push: ruff format --check" && ruff format --check .
+echo "pre-push: mypy" && mypy src
+echo "pre-push: pytest" && pytest -q

evalgate_ai-0.1.0/.gitignore

@@ -0,0 +1,14 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+graphify-out
+.claude
+CLAUDE.md
+
+evals/

evalgate_ai-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,50 @@
+# Contributing
+
+Thanks for helping. EvalGate aims to be a small, correct tool, and that shapes what gets merged.
+
+## Setup
+
+```bash
+uv sync
+git config core.hooksPath .githooks   # enables the pre-push checks
+```
+
+There is no hosted CI. The `pre-push` hook runs the same checks a CI job would, and blocks the push if any fail:
+
+```bash
+uv run ruff check .
+uv run ruff format --check .
+uv run mypy src
+uv run pytest
+```
+
+## Ground rules
+
+- **Fail closed.** Missing, malformed or incomparable input must give INVALID (exit 2), never PASS. Any change that touches validation or policy needs a test proving that bad input can't pass.
+- **Explicit semantics.** Metrics declare a direction. Never divide by a zero baseline. No weighted overall scores.
+- **Deterministic.** The same inputs, policy and version must produce byte-identical reports. Sort every list by stable keys, and keep timestamps out of results.
+- **Offline core.** No network calls, LLM APIs or databases in the core.
+- **Layering.** `comparison/` and `policies/` are pure functions over models. File I/O lives in `io/`. The CLI and SDK both go through `evalgate.api.compare`, so there is never a second code path.
+- **Small.** Add a module or option only when real code needs it. New dependencies need a strong reason.
+
+## Where things live
+
+```text
+src/evalgate/
+  models/       Pydantic models: run, case, policy, report
+  io/           read run dirs and policies; write reports; EvalRun recorder
+  validation.py comparability checks -> diagnostics
+  comparison/   matching and metric deltas (pure)
+  policies/     rule evaluation (pure)
+  engine.py     validate -> compare -> decide
+  api.py        path-based entry point shared by CLI and SDK
+  reporting/    terminal and HTML renderers
+```
+
+The product spec is [docs/SPEC.md](docs/SPEC.md), and the JSON report contract is [docs/report-format.md](docs/report-format.md). A change to the report shape must update that doc, and bump `report_version` if it breaks existing fields.
+
+## Pull requests
+
+- One topic per PR, with tests. A bug fix starts with a test that fails without the fix.
+- Explain *why* in the description, especially any change to decision semantics.
+- To release, bump `src/evalgate/_version.py`. It is the only place the version lives.

evalgate_ai-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Josep Audenis
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evalgate_ai-0.1.0/PKG-INFO

@@ -0,0 +1,194 @@
+Metadata-Version: 2.5
+Name: evalgate-ai
+Version: 0.1.0
+Summary: Local-first AI regression testing and release decision engine.
+Project-URL: Homepage, https://github.com/josep-audenis/evalgate
+Project-URL: Repository, https://github.com/josep-audenis/evalgate
+Project-URL: Issues, https://github.com/josep-audenis/evalgate/issues
+Author: Josep Audenis
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ci,evaluation,llm,regression-testing,release-gate
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: pydantic>=2.6
+Requires-Dist: pyyaml>=6
+Requires-Dist: rich>=13
+Requires-Dist: typer>=0.12
+Description-Content-Type: text/markdown
+
+# EvalGate
+
+**Local-first regression testing and release decisions for AI applications.**
+
+You changed a prompt, model, tool or retrieval step. Your eval pipeline produced scores for the old version and the new one. EvalGate answers three questions:
+
+1. **What changed?**
+2. **What got worse?**
+3. **Is this version acceptable to release?**
+
+It compares a *baseline* run against a *candidate* run case by case and applies your release policy. It returns an explainable **PASS / FAIL / INVALID**, with exit codes `0 / 1 / 2` for CI. It runs fully offline, with no LLM API key, account or database.
+
+```text
+$ evalgate compare evals/baseline evals/candidate --policy evalgate.yaml
+baseline-v1 -> candidate-v2  dataset regression-suite@v3  evaluator rubric-v2
+cases: 4 matched, 4 compared, 0 baseline errors, 0 candidate errors
+metric   direction  baseline  candidate     delta  delta %
+quality  maximize        0.8      0.865    +0.065   +8.12%
+...
+FAIL checks.required.safety_valid: required check 'safety_valid' failed in 1 case(s): abuse-001/safety_valid
+decision: FAIL
+```
+
+Average quality went up, but one case started failing a safety check, so the candidate does not ship.
+
+## Install
+
+```bash
+pip install evalgate-ai
+```
+
+The package is published as `evalgate-ai`. It imports as `evalgate`, and the command is `evalgate`. Requires Python 3.11 or newer.
+
+## Quickstart
+
+**1. Write each run as a directory.** Use `run.json` for identity and `cases.jsonl` with one JSON object per test case:
+
+```json
+{"schema_version": "1.0", "run_id": "candidate-v2", "dataset_id": "regression-suite",
+ "dataset_version": "v3", "evaluator_version": "rubric-v2"}
+```
+
+```json
+{"case_id": "refund-001", "status": "ok", "scores": {"quality": 0.91},
+ "checks": {"safety_valid": true}, "metrics": {"latency_ms": 3000, "cost_usd": 0.012},
+ "metadata": {"category": "billing"}}
+```
+
+If the pipeline failed on a case, record `"status": "error"` instead of inventing scores. Or let the Python SDK below write these files for you.
+
+**2. Create a policy:**
+
+```bash
+evalgate init          # writes a commented evalgate.yaml
+```
+
+```yaml
+version: 1
+metrics:
+  quality:
+    path: scores.quality
+    direction: maximize          # explicit: EvalGate never assumes higher is better
+    max_mean_regression_abs: 0.02
+  latency:
+    path: metrics.latency_ms
+    direction: minimize
+    max_mean_regression_pct: 10
+checks:
+  required: [safety_valid]       # must pass in every candidate case
+  max_new_failures: 0            # nothing that passed before may fail now
+```
+
+**3. Compare:**
+
+```bash
+evalgate validate evals/candidate
+evalgate compare evals/baseline evals/candidate --policy evalgate.yaml \
+  --json report.json --report report.html
+```
+
+- `--report` writes one standalone HTML file: no scripts and no external assets.
+- `--json` writes a stable, documented report: see [report format](https://github.com/josep-audenis/evalgate/blob/main/docs/report-format.md).
+- Neither report contains raw prompts or outputs.
+
+## Python SDK
+
+```python
+from evalgate import EvalRun, compare
+
+with EvalRun(
+    "candidate", dataset="support-suite", dataset_version="v3", evaluator_version="rubric-v2"
+) as run:  # writes evals/candidate/
+    for case in dataset:
+        try:
+            output = execute_agent(case)
+        except Exception as exc:
+            run.record_error(case["id"], repr(exc))  # never invent scores
+            continue
+        run.record(
+            case["id"],
+            scores={"quality": judge(output)},
+            checks={"safety_valid": is_safe(output)},
+            metrics={"latency_ms": 1234},
+        )
+
+report = compare("evals/baseline", "evals/candidate", policy="evalgate.yaml")
+print(report.summary())
+report.to_json("report.json")
+report.to_html("report.html")
+```
+
+- **Validation:** values are checked as you record them. Duplicate `case_id`s raise an error.
+- **Crash safety:** `run.json` is written only when the `with` block exits cleanly, so an interrupted run never loads as complete.
+- **Overwrites:** `EvalRun` won't replace an existing run unless you pass `overwrite=True`. Even then, the old run is replaced only when the new one finishes cleanly.
+- **Same results as the CLI:** the SDK's `compare` is the same code path as `evalgate compare`, and the reports are identical.
+
+## How decisions are made
+
+- **INVALID (exit 2):** a trustworthy decision is impossible. Causes:
+  - missing or duplicate cases
+  - a dataset or evaluator version mismatch
+  - missing metrics or checks
+  - malformed files
+  - a percent threshold on a metric whose baseline mean is 0
+
+  Invalid input never produces a PASS.
+- **FAIL (exit 1):** the comparison is valid, but at least one policy rule is violated.
+- **PASS (exit 0):** the comparison is valid and every rule holds.
+
+Further rules:
+- Means are computed over cases that ran successfully in both runs.
+- Cases with `status: "error"` count against `max_execution_errors` (default 0) and never count as successes.
+- Every metric declares its direction and at least one threshold.
+- The policy is strict. Unknown or duplicate keys, `true` where a number is expected, and a policy with no rules are all rejected.
+- There is no weighted overall score. Each rule passes or fails on its own, so one regression can't hide behind an improvement elsewhere.
+
+## Use in CI
+
+Exit codes make EvalGate a drop-in CI step. See [CI integration](https://github.com/josep-audenis/evalgate/blob/main/docs/ci.md) for a GitHub Actions example.
+
+## Examples
+
+[examples/](https://github.com/josep-audenis/evalgate/tree/main/examples) has runnable scenarios:
+- a safety regression hidden by better averages
+- a quality gain that blows the cost and latency budget
+- an SDK eval loop with a timed-out case
+
+## Status and scope
+
+Version 0.1.0 (alpha). EvalGate compares results your pipeline already produced. It does **not** run your application, call LLM judges, or host datasets.
+
+Not yet included:
+- statistical uncertainty (confidence intervals)
+- repeated-run comparison
+- adapters for other eval frameworks
+
+Planned work is in the [spec](https://github.com/josep-audenis/evalgate/blob/main/docs/SPEC.md).
+
+## Contributing
+
+See [CONTRIBUTING.md](https://github.com/josep-audenis/evalgate/blob/main/CONTRIBUTING.md).
+
+## License
+
+MIT

evalgate_ai-0.1.0/README.md

@@ -0,0 +1,165 @@
+# EvalGate
+
+**Local-first regression testing and release decisions for AI applications.**
+
+You changed a prompt, model, tool or retrieval step. Your eval pipeline produced scores for the old version and the new one. EvalGate answers three questions:
+
+1. **What changed?**
+2. **What got worse?**
+3. **Is this version acceptable to release?**
+
+It compares a *baseline* run against a *candidate* run case by case and applies your release policy. It returns an explainable **PASS / FAIL / INVALID**, with exit codes `0 / 1 / 2` for CI. It runs fully offline, with no LLM API key, account or database.
+
+```text
+$ evalgate compare evals/baseline evals/candidate --policy evalgate.yaml
+baseline-v1 -> candidate-v2  dataset regression-suite@v3  evaluator rubric-v2
+cases: 4 matched, 4 compared, 0 baseline errors, 0 candidate errors
+metric   direction  baseline  candidate     delta  delta %
+quality  maximize        0.8      0.865    +0.065   +8.12%
+...
+FAIL checks.required.safety_valid: required check 'safety_valid' failed in 1 case(s): abuse-001/safety_valid
+decision: FAIL
+```
+
+Average quality went up, but one case started failing a safety check, so the candidate does not ship.
+
+## Install
+
+```bash
+pip install evalgate-ai
+```
+
+The package is published as `evalgate-ai`. It imports as `evalgate`, and the command is `evalgate`. Requires Python 3.11 or newer.
+
+## Quickstart
+
+**1. Write each run as a directory.** Use `run.json` for identity and `cases.jsonl` with one JSON object per test case:
+
+```json
+{"schema_version": "1.0", "run_id": "candidate-v2", "dataset_id": "regression-suite",
+ "dataset_version": "v3", "evaluator_version": "rubric-v2"}
+```
+
+```json
+{"case_id": "refund-001", "status": "ok", "scores": {"quality": 0.91},
+ "checks": {"safety_valid": true}, "metrics": {"latency_ms": 3000, "cost_usd": 0.012},
+ "metadata": {"category": "billing"}}
+```
+
+If the pipeline failed on a case, record `"status": "error"` instead of inventing scores. Or let the Python SDK below write these files for you.
+
+**2. Create a policy:**
+
+```bash
+evalgate init          # writes a commented evalgate.yaml
+```
+
+```yaml
+version: 1
+metrics:
+  quality:
+    path: scores.quality
+    direction: maximize          # explicit: EvalGate never assumes higher is better
+    max_mean_regression_abs: 0.02
+  latency:
+    path: metrics.latency_ms
+    direction: minimize
+    max_mean_regression_pct: 10
+checks:
+  required: [safety_valid]       # must pass in every candidate case
+  max_new_failures: 0            # nothing that passed before may fail now
+```
+
+**3. Compare:**
+
+```bash
+evalgate validate evals/candidate
+evalgate compare evals/baseline evals/candidate --policy evalgate.yaml \
+  --json report.json --report report.html
+```
+
+- `--report` writes one standalone HTML file: no scripts and no external assets.
+- `--json` writes a stable, documented report: see [report format](https://github.com/josep-audenis/evalgate/blob/main/docs/report-format.md).
+- Neither report contains raw prompts or outputs.
+
+## Python SDK
+
+```python
+from evalgate import EvalRun, compare
+
+with EvalRun(
+    "candidate", dataset="support-suite", dataset_version="v3", evaluator_version="rubric-v2"
+) as run:  # writes evals/candidate/
+    for case in dataset:
+        try:
+            output = execute_agent(case)
+        except Exception as exc:
+            run.record_error(case["id"], repr(exc))  # never invent scores
+            continue
+        run.record(
+            case["id"],
+            scores={"quality": judge(output)},
+            checks={"safety_valid": is_safe(output)},
+            metrics={"latency_ms": 1234},
+        )
+
+report = compare("evals/baseline", "evals/candidate", policy="evalgate.yaml")
+print(report.summary())
+report.to_json("report.json")
+report.to_html("report.html")
+```
+
+- **Validation:** values are checked as you record them. Duplicate `case_id`s raise an error.
+- **Crash safety:** `run.json` is written only when the `with` block exits cleanly, so an interrupted run never loads as complete.
+- **Overwrites:** `EvalRun` won't replace an existing run unless you pass `overwrite=True`. Even then, the old run is replaced only when the new one finishes cleanly.
+- **Same results as the CLI:** the SDK's `compare` is the same code path as `evalgate compare`, and the reports are identical.
+
+## How decisions are made
+
+- **INVALID (exit 2):** a trustworthy decision is impossible. Causes:
+  - missing or duplicate cases
+  - a dataset or evaluator version mismatch
+  - missing metrics or checks
+  - malformed files
+  - a percent threshold on a metric whose baseline mean is 0
+
+  Invalid input never produces a PASS.
+- **FAIL (exit 1):** the comparison is valid, but at least one policy rule is violated.
+- **PASS (exit 0):** the comparison is valid and every rule holds.
+
+Further rules:
+- Means are computed over cases that ran successfully in both runs.
+- Cases with `status: "error"` count against `max_execution_errors` (default 0) and never count as successes.
+- Every metric declares its direction and at least one threshold.
+- The policy is strict. Unknown or duplicate keys, `true` where a number is expected, and a policy with no rules are all rejected.
+- There is no weighted overall score. Each rule passes or fails on its own, so one regression can't hide behind an improvement elsewhere.
+
+## Use in CI
+
+Exit codes make EvalGate a drop-in CI step. See [CI integration](https://github.com/josep-audenis/evalgate/blob/main/docs/ci.md) for a GitHub Actions example.
+
+## Examples
+
+[examples/](https://github.com/josep-audenis/evalgate/tree/main/examples) has runnable scenarios:
+- a safety regression hidden by better averages
+- a quality gain that blows the cost and latency budget
+- an SDK eval loop with a timed-out case
+
+## Status and scope
+
+Version 0.1.0 (alpha). EvalGate compares results your pipeline already produced. It does **not** run your application, call LLM judges, or host datasets.
+
+Not yet included:
+- statistical uncertainty (confidence intervals)
+- repeated-run comparison
+- adapters for other eval frameworks
+
+Planned work is in the [spec](https://github.com/josep-audenis/evalgate/blob/main/docs/SPEC.md).
+
+## Contributing
+
+See [CONTRIBUTING.md](https://github.com/josep-audenis/evalgate/blob/main/CONTRIBUTING.md).
+
+## License
+
+MIT