evalshift 0.3.0__tar.gz

evalshift-0.3.0/.gitignore

@@ -0,0 +1,77 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# Testing
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+htmlcov/
+.pytest_cache/
+.hypothesis/
+coverage.xml
+*.cover
+
+# Type checking
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pyre/
+.pytype/
+
+# Linting
+.ruff_cache/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+.DS_Store
+
+# EvalShift runtime
+.evalshift/
+*.db
+*.sqlite
+*.sqlite3
+
+# Docs build
+site/
+
+# uv
+uv.lock
+
+# Build plan source (kept locally, not in git)
+evalshift-cli-mvp-build-plan.md.pdf
+.claude/agents/brainstormer.md

evalshift-0.3.0/CHANGELOG.md

@@ -0,0 +1,193 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.3.0] — 2026-05-12
+
+### Added
+
+- `evalshift all` runs the full `doctor → run → evaluate → analyze →
+  report` pipeline in one command, with a single Rich Live region:
+  stacked status rows, an inline block-bar for the run stage, and a
+  final verdict block (`✓ candidate is significantly better` /
+  `✗ candidate regressed` / `~ no significant change`) plus a minor-
+  regression callout. Forwards every flag the underlying commands
+  accept (`--from/--to`, `--config`, `--suite`, `--offline`,
+  `--fixtures`, `--yes`, `--resume`, `--gate`, `--open`).
+- `evaluate`, `analyze`, and `report` now expose reusable cores
+  (`run_evaluate`, `run_analyze`, `run_report`) returning typed
+  result dataclasses; the existing Typer commands are thin wrappers.
+- Orchestrator gained an optional `on_progress` callback
+  (`ProgressEvent`) so a caller can drive its own progress UI in
+  place of the built-in Rich `Progress`. `preflight_cost(...)`
+  exposes the cost estimate without dispatching any work.
+
+### Changed
+
+- `evalshift all`: the cost preflight row is now labelled
+  `max cost ........... ≤ $X.XX` instead of `estimated cost ... $X.XX`.
+  The underlying figure is unchanged — it has always been a
+  worst-case ceiling (priced as if every call emits the model's full
+  `default_max_tokens`) — but the new wording makes that explicit so
+  users don't read it as a forecast. The interactive cost-confirmation
+  prompt also now reads "at most $X" instead of "estimated cost $X".
+- `evalshift init` now always scaffolds an agent project. The brief
+  `--agent` flag introduced in the previous unreleased entry has been
+  removed — there's only one starter, and it's the v0.2 agent flow.
+  Existing v0.1-style projects are unaffected (no breaking change to
+  the config schema or runner).
+- Agent scaffold ships 6 customer-support tools (`search_orders`,
+  `lookup_customer`, `issue_refund`, `update_order_status`,
+  `send_email`, `notify_security_team`) and 40 suite rows across 5
+  slices (security, routine, refund, customer_lookup, text_only) so
+  the analysis layer produces real severity badges on a first run
+  instead of "insufficient" / zero-variance warnings.
+- Dropped `structural.length` from the scaffolded `evalshift.yaml`.
+  Agent runs frequently produce empty `final_text` (model returned
+  only tool calls) which made length scores 0/0 across every routine
+  + security row — pure noise. Users who want length checks can
+  add the block back manually for prompts that produce text.
+
+## [0.2.0] — Tool-call evaluation
+
+Adds agent-migration support: detects regressions in *which* tools the
+new model calls, *what* arguments it passes, and *how* it sequences
+them. v0.1 backward compatible — every existing example keeps working
+without changes.
+
+### Added
+
+- `evalshift.evaluators.tool_models`: provider-agnostic `ToolSpec`,
+  `ToolCall`, `ToolTrace`. `ToolSpec.to_anthropic` / `to_openai` /
+  `from_dict` adapters keep wire-format details out of user code.
+- `evalshift.evaluators.tool_parser`: `parse_response_to_trace`
+  dispatcher with three provider parsers. Handles the LiteLLM
+  Anthropic-normalised-to-OpenAI shape transparently. Malformed
+  argument JSON marked `_parse_error` instead of crashing.
+- `evalshift.evaluators.tool_loader`: `load_tools(yaml or json)` with
+  the same plain/rich error rendering pattern as `ConfigError`.
+- `evalshift.models.client.complete_with_tools` and
+  `ToolCompletionResult` — tool-aware call path that doesn't replace
+  the existing `complete` / `CompletionResult`.
+- `evalshift test-call --tools <path>` smoke command extension.
+- `scripts/smoke_live_tools.py` for fixture capture (manual; not in CI).
+- Three new evaluators wired into `evalshift evaluate`:
+  - `ToolSelectionEvaluator` (modes: exact / set / first / expected)
+  - `ToolArgumentsEvaluator` (per-field strategies: exact / subset /
+    numeric / semantic; greedy nearest-index call matching)
+  - `ToolTraceStructureEvaluator` (call_count / parallelism /
+    refusal_alignment / expected_count_alignment; refusal mismatches
+    force severity_floor: high)
+- Suite extension: optional `expected_tools` (with per-call
+  `match_strategy`), `expected_tool_count`, `expected_no_tools`,
+  `expected_parallel`. Mutual-exclusion checks where appropriate.
+- Config extension: `prompts[].tools_path` makes a prompt agent-style;
+  `evaluators.{tool_selection, tool_arguments, tool_trace_structure}`
+  blocks added.
+- Orchestrator dispatch: agent prompts route through
+  `complete_with_tools`; the resulting `Call.trace` round-trips via
+  pydantic so `raw.jsonl` Just Works.
+- HTML report extension: per-prompt "Top regressions" section now
+  renders side-by-side trace diffs for tool-evaluator regressions
+  with green / yellow / red colour coding for matching / different-
+  args / missing-extra calls. CSS stays inlined.
+- `evalshift doctor` warns about common agent-config mistakes:
+  prompts with `tools_path` but no tool evaluators, and suite
+  examples with `expected_tools` but no agent prompts.
+- Docs: `docs/agents.md` walkthrough, plus configuration / evaluators
+  / faq updates.
+- `examples/agent/` runnable customer-routing example with 12 golden
+  rows across security / routine / text-only slices.
+
+### Quality
+
+- 512+ tests, including end-to-end integration tests covering the
+  PRD §9.3 failure-mode grid. ruff + format + mypy --strict clean.
+- v0.1 backward compatibility verified: existing 462 tests still
+  green; `examples/simple/` unchanged.
+
+## [0.1.0]
+
+First alpha release. Every command in the planned MVP pipeline is shipped.
+
+### Live commands
+
+| Command                | What it does                                                            |
+| ---------------------- | ----------------------------------------------------------------------- |
+| `evalshift doctor`     | Check Python version, provider API keys, and `evalshift.yaml` validity. |
+| `evalshift init`       | Scaffold a starter project (yaml + prompts.py + golden.jsonl).          |
+| `evalshift run`        | Call source + target models on every (prompt, example) → `raw.jsonl`.   |
+| `evalshift evaluate`   | Score every (source, target) pair → `scores.jsonl`.                     |
+| `evalshift analyze`    | Paired tests + BH correction → `analysis.json`.                         |
+| `evalshift report`     | Single-file HTML report → `report.html` + `report.json`.                |
+| `evalshift cache clear`| Wipe the local SQLite response cache.                                   |
+| `evalshift validate`   | (hidden dev) Verify config + suite + prompts are compatible.            |
+| `evalshift test-call`  | (hidden dev) One-shot smoke call to confirm provider connectivity.      |
+
+### Added — Phase 0–1
+
+- `pyproject.toml` (Python ≥3.14, hatchling, MIT, pinned deps), `ruff.toml`, `mypy.ini`, pytest config, `.pre-commit-config.yaml`, GitHub Actions CI.
+- Full `src/evalshift` package skeleton across cli/config/parsers/models/suite/runner/evaluators/analysis/reports/cache/utils with module docstrings and a `py.typed` marker.
+- `evalshift.config.models`: pydantic v2 schema for `evalshift.yaml` (prompts, evaluators, slices, defaults) with strict validation, `extra='forbid'`, and detection-mode field invariants.
+- `evalshift.config.loader`: `load_config()` plus a structured `ConfigError` with both plain-text and Rich panel rendering.
+- `evalshift doctor`: reports Python version, provider API keys, and `evalshift.yaml` validity. Exits 1 only on hard failures.
+- `evalshift init`: scaffolds `evalshift.yaml` + `prompts.py` + `golden.jsonl`. `--force` overwrite, `--directory` target dir.
+
+### Added — Phase 2 (suite + prompt loading)
+
+- `evalshift.suite` package: pydantic `SuiteExample` + `Suite` models, JSONL loader with line-numbered errors, blank-line tolerance, multi-error collection.
+- `evalshift.parsers` package: `ManualParser` (inline content) and `PythonStringParser` (AST-walks `.py` for module-level string literals; rejects f-strings, concatenation, function calls, attribute access, name references — never runs user code).
+- `evalshift.utils.templating`: `extract_variables`, `render` with strict missing-var detection, and a bulk `validate_suite_against_prompts` pre-flight check.
+- `evalshift validate` (hidden): end-to-end pre-flight that confirms config + suite + prompts are mutually compatible.
+
+### Added — Phase 3 (model client + cache)
+
+- `evalshift.models.registry`: advisory, not gating. `resolve_model()` accepts any model id, falling back to prefix-inferred provider when not in the curated registry. `get_model()` strict variant kept for tests.
+- `evalshift.cache`: SQLAlchemy 2.0 + async `CacheStore` (sha256 keys, 7-day TTL) at `~/.evalshift/cache.db`. Added `greenlet` runtime dep.
+- `evalshift cache clear` CLI command (under a `cache` sub-app).
+- `evalshift.models.client`: async `ModelClient` wrapping `litellm.acompletion` with uniform error mapping (`RateLimitError` / `AuthError` / `ModelError`), full-jitter exponential backoff retries (auth short-circuits), and per-call cost + token bookkeeping.
+- `evalshift.utils.cost`: `estimate_run_cost(...)` for pre-flight cost estimation with defensive fallbacks for missing LiteLLM pricing/token-counter data.
+- `evalshift test-call` (hidden): one-shot smoke test renderer with response/tokens/cost/latency Rich panel.
+
+### Added — Phase 4 (run orchestrator)
+
+- `evalshift.runner` package: pydantic `RunState` + `Call` models, atomic `state.json` checkpointing, append-only `raw.jsonl`, crash-safe iterator, `validate_resume` with hash-drift detection.
+- `run_orchestrator`: async loop with `asyncio.Semaphore(concurrency)`, cache-check → live-call → record per (prompt, example, role), Rich progress bar, $10 cost gate (skip with `--yes`), 50-call checkpoint cadence.
+- `evalshift run` command: `--from`, `--to`, `--config`, `--suite`, `--resume`, `--yes`. Outputs go to `.evalshift/runs/<run-id>/`. Per-call errors are recorded but don't fail the run.
+
+### Added — Phase 5 (evaluators)
+
+- `evalshift.evaluators` package: `Evaluator` Protocol, `PairedScore`, `EvalRecord`.
+- Structural: `JsonSchemaEvaluator`, `RegexEvaluator`, `LengthEvaluator` (1.0 inside bounds, distance-decayed outside).
+- Semantic: `CosineSimilarityEvaluator` (target preservation framing — source = 1.0, target = cosine to source).
+- LLM judge: `PairwiseJudgeEvaluator` with order-randomization and defensive JSON parsing.
+- `evalshift evaluate <run-id>` command: pairs source/target calls, runs every configured evaluator, writes `scores.jsonl`.
+
+### Added — Phase 6 (statistics)
+
+- `evalshift.analysis.slicing`: `build_slices` groups records by tag (with implicit `"all"` slice); `aggregates()` computes per-slice n/mean/std.
+- `evalshift.analysis.statistics`: per-comparison Shapiro-Wilk → paired t-test or Wilcoxon signed-rank, Cohen's d, 95% CI (analytical for t-test, bootstrap for Wilcoxon), Benjamini-Hochberg correction across the whole run, severity classification (critical/high/medium/low/improved/none/insufficient).
+- `evalshift analyze <run-id>` command: writes `analysis.json` and prints a Rich summary table.
+
+### Added — Phase 7 (report)
+
+- `evalshift.reports.json`: `build_report_payload` + `ReportData` stitching state + raw + scores + analysis into a single payload. Persists `report.json`.
+- `evalshift.reports.templates/report.html.j2` + `report.css`: self-contained HTML (CSS inlined, zero external assets, no JS) with executive summary, per-prompt aggregate + slice tables, top-5 regressions side-by-side, methodology appendix.
+- `evalshift report <run-id>` command, with optional `--open` to launch in the user's default browser.
+
+### Added — Phase 8 (polish + OSS)
+
+- Documentation site (MkDocs Material) covering Getting Started, Configuration, Evaluators, Methodology, FAQ.
+- `examples/simple/` runnable example.
+- `CODE_OF_CONDUCT.md` (Contributor Covenant 2.1), `SECURITY.md`, GitHub issue + PR templates.
+- README polished with badges, status, full pipeline walkthrough, non-goals.
+
+### Quality
+
+- 371+ tests, ≥95% line coverage, `mypy --strict` clean, `ruff check` clean, `ruff format` clean.
+- Mocked CI throughout (no API keys needed); `scripts/smoke_live.py` deferred for the user's manual live verification.

evalshift-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Lukas Babaliauskas
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evalshift-0.3.0/PKG-INFO

@@ -0,0 +1,185 @@
+Metadata-Version: 2.4
+Name: evalshift
+Version: 0.3.0
+Summary: Run your prompts on two LLMs and find out, with statistical confidence, what regressed.
+Project-URL: Homepage, https://github.com/babaliauskas/EvalShift
+Project-URL: Documentation, https://github.com/babaliauskas/EvalShift#readme
+Project-URL: Repository, https://github.com/babaliauskas/EvalShift
+Project-URL: Issues, https://github.com/babaliauskas/EvalShift/issues
+Project-URL: Changelog, https://github.com/babaliauskas/EvalShift/blob/main/CHANGELOG.md
+Author: Lukas Babaliauskas
+License: MIT
+License-File: LICENSE
+Keywords: anthropic,evaluation,gemini,llm,migration,openai,regression-testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.14
+Requires-Dist: aiosqlite>=0.19
+Requires-Dist: greenlet>=3.0
+Requires-Dist: httpx>=0.27
+Requires-Dist: jinja2>=3.1
+Requires-Dist: jsonschema>=4.22
+Requires-Dist: litellm>=1.40
+Requires-Dist: numpy>=1.26
+Requires-Dist: pydantic>=2.5
+Requires-Dist: pygments>=2.18
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.7
+Requires-Dist: scipy>=1.12
+Requires-Dist: sqlalchemy>=2.0
+Requires-Dist: tiktoken>=0.7
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: mkdocs-material>=9.5; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pre-commit>=3.7; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: respx>=0.21; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: types-jsonschema>=4.22; extra == 'dev'
+Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# EvalShift
+
+> Run your prompts on two LLMs and find out, with statistical confidence, what regressed.
+
+[![CI](https://github.com/babaliauskas/EvalShift/actions/workflows/ci.yml/badge.svg)](https://github.com/babaliauskas/EvalShift/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.14+](https://img.shields.io/badge/python-3.14%2B-blue.svg)](https://www.python.org/downloads/)
+[![Status: alpha](https://img.shields.io/badge/status-alpha-orange.svg)](#status)
+
+EvalShift is a local-first CLI that helps engineering teams migrate safely between
+LLM versions (e.g. `claude-4.5-sonnet` → `claude-5-sonnet`). Point it at your
+prompts and a golden suite of inputs; it runs both models, scores the outputs
+with structural / semantic / LLM-as-judge evaluators, and produces a single-file
+HTML report with **defensible statistics**: paired tests, Cohen's d, 95% CIs,
+and Benjamini–Hochberg correction across every (prompt × evaluator × slice)
+comparison.
+
+## Status
+
+**Alpha.** Every command in the pipeline is shipped and the test suite is at
+95%+ coverage. APIs may still change as feedback comes in.
+
+## Install
+
+Requires Python 3.14+.
+
+```bash
+# Recommended
+uv pip install evalshift     # or: pip install evalshift
+```
+
+From source (for contributors):
+
+```bash
+git clone https://github.com/babaliauskas/EvalShift.git
+cd EvalShift
+uv venv --python 3.14
+source .venv/bin/activate
+uv pip install -e ".[dev]"
+```
+
+## Quick start
+
+```bash
+# 1. Scaffold a starter project. Writes evalshift.yaml + prompts.py +
+#    tools.yaml + a 40-row golden.jsonl for a customer-support agent.
+mkdir my-eval && cd my-eval
+evalshift init
+
+# 2. Set whichever provider keys you'll use
+export GOOGLE_API_KEY=...   # or ANTHROPIC_API_KEY / OPENAI_API_KEY
+
+# 3. Run the whole pipeline in one command (doctor → run → evaluate
+#    → analyze → report). Pass --open to launch the report.
+evalshift all --yes --open
+```
+
+`evalshift all` drives the full five-stage pipeline under a single
+Rich Live region — stacked status rows, an inline progress bar for
+the run stage, and a final verdict block that tells you whether the
+candidate is significantly better, regressed, or showed no
+significant change.
+
+If you want to drive each stage by hand (useful in CI, or when
+re-running just one stage after fixing config):
+
+```bash
+evalshift doctor
+evalshift run --yes
+evalshift evaluate <run-id>
+evalshift analyze <run-id>
+evalshift report <run-id> --open
+```
+
+Every artefact lives under `.evalshift/runs/<run-id>/` — `state.json`,
+`raw.jsonl`, `scores.jsonl`, `analysis.json`, `report.json`,
+`report.html`. None of it leaves your machine.
+
+## Agent migrations (v0.2)
+
+Migrating an agent (a prompt that uses tools)? EvalShift v0.2 detects
+regressions in *which* tools the new model calls, *what* arguments it
+passes, and *how* it sequences them. The killer scenario: a routing
+agent that silently stops calling `notify_security_team` after the
+migration — text-only eval reports green, v0.2 marks it CRITICAL.
+
+The default `evalshift init` scaffold *is* an agent project — six tools
+plus a 40-row golden suite. Just run the quick-start above and the
+tool-call evaluators kick in automatically.
+
+See [`docs/agents.md`](docs/agents.md) for the full walkthrough and
+the [`examples/agent/`](examples/agent/) directory for a runnable
+customer-support example.
+
+## What the report looks like
+
+The HTML report (single file, no external assets, works offline) has:
+
+* **Executive summary** — one row per prompt with a severity badge.
+* **Per-prompt deep dive** — aggregate stats, per-slice breakdown,
+  top-5 worst regressions side-by-side.
+* **Methodology appendix** — every test, p-value, effect size, and
+  CI is documented.
+
+## Why local-first?
+
+Your prompts and your suite never leave your machine. The only outbound calls
+are to the LLM providers you configure (Anthropic, OpenAI, Google) using your
+own API keys. There is no EvalShift cloud.
+
+## Documentation
+
+* [Getting started](docs/getting-started.md) — install + first run walkthrough
+* [Configuration reference](docs/configuration.md) — every `evalshift.yaml` field
+* [Evaluators](docs/evaluators.md) — when to use which family
+* [Methodology](docs/methodology.md) — the statistical machinery
+* [FAQ](docs/faq.md) — common questions
+* [`MVP_TODO.md`](MVP_TODO.md) — the build checklist (every box ticked)
+
+## Non-goals (for v0.1)
+
+* Hosted backend / web UI
+* Multi-criterion judge in a single call
+* Custom evaluator plugin system
+* Comparing more than 2 models in one run
+* Auto-detection of LangChain / LlamaIndex prompt patterns
+
+These are deferred to v0.2+; see the PDF spec in the repo for the
+full deferred-features list.
+
+## License
+
+[MIT](LICENSE) — free for any use.

evalshift-0.3.0/README.md

@@ -0,0 +1,133 @@
+# EvalShift
+
+> Run your prompts on two LLMs and find out, with statistical confidence, what regressed.
+
+[![CI](https://github.com/babaliauskas/EvalShift/actions/workflows/ci.yml/badge.svg)](https://github.com/babaliauskas/EvalShift/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.14+](https://img.shields.io/badge/python-3.14%2B-blue.svg)](https://www.python.org/downloads/)
+[![Status: alpha](https://img.shields.io/badge/status-alpha-orange.svg)](#status)
+
+EvalShift is a local-first CLI that helps engineering teams migrate safely between
+LLM versions (e.g. `claude-4.5-sonnet` → `claude-5-sonnet`). Point it at your
+prompts and a golden suite of inputs; it runs both models, scores the outputs
+with structural / semantic / LLM-as-judge evaluators, and produces a single-file
+HTML report with **defensible statistics**: paired tests, Cohen's d, 95% CIs,
+and Benjamini–Hochberg correction across every (prompt × evaluator × slice)
+comparison.
+
+## Status
+
+**Alpha.** Every command in the pipeline is shipped and the test suite is at
+95%+ coverage. APIs may still change as feedback comes in.
+
+## Install
+
+Requires Python 3.14+.
+
+```bash
+# Recommended
+uv pip install evalshift     # or: pip install evalshift
+```
+
+From source (for contributors):
+
+```bash
+git clone https://github.com/babaliauskas/EvalShift.git
+cd EvalShift
+uv venv --python 3.14
+source .venv/bin/activate
+uv pip install -e ".[dev]"
+```
+
+## Quick start
+
+```bash
+# 1. Scaffold a starter project. Writes evalshift.yaml + prompts.py +
+#    tools.yaml + a 40-row golden.jsonl for a customer-support agent.
+mkdir my-eval && cd my-eval
+evalshift init
+
+# 2. Set whichever provider keys you'll use
+export GOOGLE_API_KEY=...   # or ANTHROPIC_API_KEY / OPENAI_API_KEY
+
+# 3. Run the whole pipeline in one command (doctor → run → evaluate
+#    → analyze → report). Pass --open to launch the report.
+evalshift all --yes --open
+```
+
+`evalshift all` drives the full five-stage pipeline under a single
+Rich Live region — stacked status rows, an inline progress bar for
+the run stage, and a final verdict block that tells you whether the
+candidate is significantly better, regressed, or showed no
+significant change.
+
+If you want to drive each stage by hand (useful in CI, or when
+re-running just one stage after fixing config):
+
+```bash
+evalshift doctor
+evalshift run --yes
+evalshift evaluate <run-id>
+evalshift analyze <run-id>
+evalshift report <run-id> --open
+```
+
+Every artefact lives under `.evalshift/runs/<run-id>/` — `state.json`,
+`raw.jsonl`, `scores.jsonl`, `analysis.json`, `report.json`,
+`report.html`. None of it leaves your machine.
+
+## Agent migrations (v0.2)
+
+Migrating an agent (a prompt that uses tools)? EvalShift v0.2 detects
+regressions in *which* tools the new model calls, *what* arguments it
+passes, and *how* it sequences them. The killer scenario: a routing
+agent that silently stops calling `notify_security_team` after the
+migration — text-only eval reports green, v0.2 marks it CRITICAL.
+
+The default `evalshift init` scaffold *is* an agent project — six tools
+plus a 40-row golden suite. Just run the quick-start above and the
+tool-call evaluators kick in automatically.
+
+See [`docs/agents.md`](docs/agents.md) for the full walkthrough and
+the [`examples/agent/`](examples/agent/) directory for a runnable
+customer-support example.
+
+## What the report looks like
+
+The HTML report (single file, no external assets, works offline) has:
+
+* **Executive summary** — one row per prompt with a severity badge.
+* **Per-prompt deep dive** — aggregate stats, per-slice breakdown,
+  top-5 worst regressions side-by-side.
+* **Methodology appendix** — every test, p-value, effect size, and
+  CI is documented.
+
+## Why local-first?
+
+Your prompts and your suite never leave your machine. The only outbound calls
+are to the LLM providers you configure (Anthropic, OpenAI, Google) using your
+own API keys. There is no EvalShift cloud.
+
+## Documentation
+
+* [Getting started](docs/getting-started.md) — install + first run walkthrough
+* [Configuration reference](docs/configuration.md) — every `evalshift.yaml` field
+* [Evaluators](docs/evaluators.md) — when to use which family
+* [Methodology](docs/methodology.md) — the statistical machinery
+* [FAQ](docs/faq.md) — common questions
+* [`MVP_TODO.md`](MVP_TODO.md) — the build checklist (every box ticked)
+
+## Non-goals (for v0.1)
+
+* Hosted backend / web UI
+* Multi-criterion judge in a single call
+* Custom evaluator plugin system
+* Comparing more than 2 models in one run
+* Auto-detection of LangChain / LlamaIndex prompt patterns
+
+These are deferred to v0.2+; see the PDF spec in the repo for the
+full deferred-features list.
+
+## License
+
+[MIT](LICENSE) — free for any use.

evalshift-0.3.0/examples/agent/README.md

@@ -0,0 +1,41 @@
+# Agent example (v0.2)
+
+A small customer-routing agent that EvalShift's v0.2 tool-call evaluators
+exercise end to end.
+
+## What's here
+
+| File              | Purpose                                                                |
+| ----------------- | ---------------------------------------------------------------------- |
+| `evalshift.yaml`  | Run config with a single agent prompt + the `tool_selection` evaluator.|
+| `prompts.py`      | The agent's system prompt, referenced via `python_string`.             |
+| `tools.yaml`      | Three tools: `search_orders`, `notify_security_team`, `send_email`.    |
+| `golden.jsonl`    | 12 examples mixing security, routine, and text-only cases.             |
+
+## Run it
+
+```bash
+cd examples/agent
+export GOOGLE_API_KEY=...           # or any provider you set up
+evalshift run --yes --from gemini-2.5-flash --to gemini-2.5-pro
+RUN_ID=$(ls .evalshift/runs/ | head -1)
+evalshift evaluate $RUN_ID
+evalshift analyze $RUN_ID
+evalshift report $RUN_ID --open
+```
+
+`evalshift run` notices `tools_path` on the prompt and dispatches via
+`ModelClient.complete_with_tools` — each `Call` row in `raw.jsonl` carries
+a parsed `ToolTrace`. `evalshift evaluate` then runs `tool_selection`
+against the per-example `expected_tools` ground truth.
+
+## What the example demonstrates
+
+* **Tool-call regression detection.** If you set `target_model` to a
+  weaker model, the report will flag examples where the target stops
+  calling `notify_security_team` on security-flagged inputs.
+* **Slice analysis.** The `security` and `routine` slices report
+  per-segment regressions, so a model that's fine on routine traffic
+  but worse on security can't hide.
+* **Text-only fallback.** The `text_only` examples check that the
+  model knows when *not* to call a tool — also a real regression mode.