llm-judge-kit 0.1.0__tar.gz

llm_judge_kit-0.1.0/.gitignore

@@ -0,0 +1,34 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+*.whl
+
+# Environments / secrets
+.venv/
+venv/
+.env
+.env.*
+!.env.example
+
+# Tooling caches
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+
+# OS / editors
+.DS_Store
+.idea/
+.vscode/
+
+# Local-only working notes (not part of the public project)
+CLAUDE.md
+PROMPT.md
+docs/strategy.md

llm_judge_kit-0.1.0/CHANGELOG.md

@@ -0,0 +1,75 @@
+# Changelog
+
+All notable changes to this project are documented here.
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- Project scaffold: `pyproject.toml`, tooling gate (ruff + mypy strict + pytest
+  with ≥95% coverage), CI workflow, MIT license, contributor docs.
+- **Wedge core (M1):**
+  - `Judge` — pairs a provider with a rubric; `score()` returns a typed
+    `JudgeResult`; `__call__` alias and `passed()` convenience.
+  - `JudgeResult` / `ProviderResponse` — frozen, typed; `JudgeResult` casts
+    (`__float__`) and compares (`<`, `<=`, `>`, `>=`) like its score.
+  - `Provider` protocol + `BaseProvider`; deterministic offline `MockProvider`.
+  - Provider registry with `"scheme:model"` spec parsing (`register_provider`,
+    `make_provider`, `available_providers`).
+  - `Rubric` + registry and five built-ins: factuality, groundedness,
+    relevance, instruction_following, safety (`register_rubric`, `get_rubric`,
+    `available_rubrics`).
+  - Robust `extract_json` parser (markdown fences, surrounding prose, trailing
+    commas, nested braces) with a clear `ParseError`.
+  - Typed exception hierarchy (`LLMJudgeError` and subclasses).
+  - Runnable examples and a README quickstart that executes offline.
+
+- **Real providers (M2):**
+  - `OpenAIProvider` (Chat Completions; works with any OpenAI-compatible
+    `base_url`), `AnthropicProvider` (Messages API), `OllamaProvider` (local,
+    via httpx). Registered under the `openai:`/`anthropic:`/`ollama:` schemes.
+  - SDKs are imported lazily — the core stays dependency-free; the import only
+    happens when a real client is built, with a clear error pointing at the
+    extra to install.
+  - Offline unit tests drive every provider through injected fake clients;
+    live API tests are gated behind `LLM_JUDGE_KIT_LIVE_TESTS=1` (skipped by default).
+
+- **Consensus + reliability (M3):**
+  - `ConsensusJudge` and `Judge.consensus([...], rubric=...)` — aggregate
+    several judges (mean/median); `confidence` is derived from agreement
+    (tight spread → high confidence) and member votes land in `metadata`.
+  - `RetryProvider` — composable retry-with-backoff and optional per-call
+    timeout wrapper (injectable sleep for deterministic tests).
+  - `CachingProvider` — memoizes completions; key = hash of
+    `version + provider + model + prompt + kwargs` (pluggable store).
+  - Structured logging on the `llm_judge_kit` logger (ships a `NullHandler`;
+    `enable_debug_logging()` for local debugging).
+  - Version moved to `_version.py` (single source; hatchling dynamic version).
+
+- **Adoption surface (M4):**
+  - Bundled pytest plugin (`pytest11` entry point → top-level `pytest_llm_judge_kit`
+    module): the `llm_judge_kit` fixture and `JudgeHelper.assert_passes` let you
+    write evals as ordinary pytest tests, with score/reason/violations in the
+    failure message. Choose the model with `--llm-judge-kit-provider` or
+    `$LLM_JUDGE_KIT_PROVIDER` (defaults to `mock`, so suites run offline).
+  - README "Integrations" section, including a framework-agnostic note (works
+    with LangChain / LlamaIndex / any pipeline — it judges strings).
+
+- **Platform layer (M5):**
+  - `llm_judge_kit` CLI (argparse, no new deps): `eval` (run a dataset → report,
+    with `--fail-under` for CI gating), `compare` (several providers side by
+    side), `report` (re-render a saved JSON report).
+  - Dataset loader (`load_dataset`, `Case`) for `.jsonl`/`.ndjson`/`.json`.
+  - Benchmark engine (`run_benchmark`, `Report` with count/passed/pass_rate/
+    mean_score), kept separate from reporting.
+  - Reporting (`render_json`/`render_markdown`/`render_html`, `load_report`) —
+    JSON round-trips back into a `Report`.
+
+### Hardened (M1 adversarial review)
+- Trailing-comma JSON repair is now string-aware — it no longer corrupts string
+  values that contain `,}` or `,]`.
+- Non-finite scores/confidence (`NaN`, `Infinity`) are rejected/defaulted
+  instead of silently clamping to a perfect `1.0`.
+- `JudgeResult` and `ProviderResponse` are now genuinely hashable (the `dict`
+  `metadata` field is excluded from the hash).

llm_judge_kit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 LLMJudge contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_judge_kit-0.1.0/PKG-INFO

@@ -0,0 +1,266 @@
+Metadata-Version: 2.4
+Name: llm-judge-kit
+Version: 0.1.0
+Summary: Provider-agnostic, reproducible, typed LLM-as-a-judge — a small primitive you can depend on.
+Project-URL: Homepage, https://github.com/Nikolay26ru/llm-judge-kit
+Project-URL: Repository, https://github.com/Nikolay26ru/llm-judge-kit
+Project-URL: Changelog, https://github.com/Nikolay26ru/llm-judge-kit/blob/main/CHANGELOG.md
+Author: LLMJudge contributors
+License-Expression: MIT
+License-File: LICENSE
+Keywords: eval,evaluation,llm,llm-as-a-judge,rubric,testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Provides-Extra: all
+Requires-Dist: anthropic>=0.40; extra == 'all'
+Requires-Dist: httpx>=0.27; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.40; extra == 'anthropic'
+Provides-Extra: ollama
+Requires-Dist: httpx>=0.27; extra == 'ollama'
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# LLMJudge
+
+> Provider-agnostic, reproducible, typed **LLM-as-a-judge** — a small primitive you can depend on.
+
+[![CI](https://github.com/Nikolay26ru/llm-judge-kit/actions/workflows/ci.yml/badge.svg)](https://github.com/Nikolay26ru/llm-judge-kit/actions/workflows/ci.yml)
+[![Python](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![Typed](https://img.shields.io/badge/typed-mypy%20strict-blue)](https://mypy-lang.org/)
+
+LLMJudge is one tiny, well-tested module for scoring model outputs with an LLM
+judge — the part most projects re-implement badly. The core has **zero required
+runtime dependencies**, a **stable typed API**, and runs **fully offline in
+tests** via a deterministic mock.
+
+## Install
+
+```bash
+pip install llm-judge-kit                 # core only, zero deps
+pip install "llm-judge-kit[openai]"       # + OpenAI-compatible provider
+pip install "llm-judge-kit[anthropic]"    # + Anthropic provider
+pip install "llm-judge-kit[all]"          # all providers
+```
+
+## Quickstart
+
+This runs as-is — no API key, deterministic ([`examples/quickstart.py`](examples/quickstart.py)):
+
+```python
+from llm_judge_kit import Judge, MockProvider
+
+# MockProvider(fixed_score=...) keeps this example deterministic and offline.
+judge = Judge(provider=MockProvider(fixed_score=0.9), rubric="factuality")
+
+result = judge.score(
+    prompt="What is the capital of France?",
+    response="The capital of France is Paris.",
+)
+
+assert result > 0.8  # a JudgeResult compares like its float score
+print(f"score={result.score}  confidence={result.confidence}")
+print(f"passed={result.passed()}  reason={result.reason!r}")
+```
+
+With a **real model**, swap the provider for a spec string — nothing else changes:
+
+```python
+judge = Judge(provider="openai:gpt-5", rubric="factuality")
+result = judge.score(prompt, response)
+if not result.passed(0.7):
+    print("Failed:", result.reason, result.violations)
+```
+
+## Core concepts
+
+| Piece | What it is |
+| --- | --- |
+| `Judge(provider, rubric)` | Pairs a model backend with a rubric; `score()` → `JudgeResult`. |
+| `JudgeResult` | Frozen, typed verdict: `score`, `confidence`, `reason`, `evidence`, `violations`, `raw`, `metadata`. Compares and casts like its `score`. |
+| `Provider` | A `Protocol` with one method, `complete(prompt) -> ProviderResponse`. |
+| `Rubric` | Declarative description of *what* to evaluate; renders a strict-JSON judging prompt. |
+
+### `JudgeResult` ergonomics
+
+```python
+r = judge.score(prompt, response)
+r.score            # float in [0, 1]
+float(r)           # same number
+r > 0.8            # compares like its score
+r.passed(0.7)      # bool against a threshold
+r.reason           # short justification
+r.evidence         # tuple of supporting quotes/facts
+r.violations       # tuple of failed criteria
+r.metadata         # provider, model, token usage, latency, cost
+```
+
+### Built-in rubrics
+
+`factuality`, `groundedness` (requires `context=`), `relevance`,
+`instruction_following`, `safety`. List them with
+`llm_judge_kit.available_rubrics()`.
+
+```python
+judge = Judge(provider="openai:gpt-5", rubric="groundedness")
+result = judge.score(question, answer, context=retrieved_docs)  # RAG check
+```
+
+## Consensus (vote across models)
+
+Run several judge models and aggregate — `confidence` reflects how much they
+**agree** ([`examples/consensus.py`](examples/consensus.py)):
+
+```python
+judge = Judge.consensus(
+    ["openai:gpt-5", "anthropic:claude-opus-4-8", "ollama:llama3"],
+    rubric="factuality",
+)
+result = judge.score(prompt, response)
+result.score          # mean (or median) of member scores
+result.confidence     # high when members agree, low when they diverge
+result.metadata["votes"]   # each member's score
+```
+
+## Reliability & caching
+
+Both wrappers are providers, so they compose around any backend
+([`examples/reliability_and_cache.py`](examples/reliability_and_cache.py)):
+
+```python
+from llm_judge_kit import Judge, OpenAIProvider, RetryProvider, CachingProvider
+
+provider = CachingProvider(                      # memoize identical calls
+    RetryProvider(                               # retry w/ backoff + timeout
+        OpenAIProvider(model="gpt-5"), retries=3, timeout=30,
+    )
+)
+judge = Judge(provider=provider, rubric="factuality")
+```
+
+The cache key is `version + provider + model + prompt + kwargs`, so it is stable
+and invalidates correctly across library versions. Logs are emitted on the
+`llm_judge_kit` logger (silent by default; call `enable_debug_logging()` to see them).
+
+## Integrations
+
+### pytest — eval as ordinary tests
+
+Installing `llm_judge_kit` registers a pytest plugin (no conftest wiring). The
+`llm_judge_kit` fixture turns an eval into a normal test; a failure reads like any
+other failing assertion (score, reason, violations):
+
+```python
+def test_answer_is_grounded(llm_judge_kit):
+    llm_judge_kit.assert_passes(
+        prompt="How tall is the Eiffel Tower?",
+        response=my_rag_pipeline("How tall is the Eiffel Tower?"),
+        rubric="groundedness",
+        context=retrieved_docs,
+        threshold=0.7,
+    )
+```
+
+Pick the judge model once for the whole suite — it defaults to `mock` (offline),
+so tests are green until you point them at a real model:
+
+```bash
+pytest --llm-judge-kit-provider "openai:gpt-5"      # or: export LLM_JUDGE_KIT_PROVIDER=...
+```
+
+### Any framework
+
+LLMJudge judges *strings*, so it drops into any stack — LangChain, LlamaIndex,
+DSPy, a raw script — with no adapter. Whatever produces the output, pass it in:
+
+```python
+output = my_chain.invoke(question)          # LangChain / LlamaIndex / your code
+result = Judge(provider="openai:gpt-5", rubric="relevance").score(question, output)
+```
+
+## Extend without touching the core
+
+Add a **rubric** ([`examples/custom_rubric.py`](examples/custom_rubric.py)):
+
+```python
+from llm_judge_kit import Rubric, register_rubric
+
+register_rubric(Rubric(
+    name="conciseness",
+    description="Whether the response is as short as possible while complete.",
+    criteria=("No filler or repetition.", "Every sentence earns its place."),
+))
+judge = Judge(provider="openai:gpt-5", rubric="conciseness")
+```
+
+Add a **provider** — implement one method, optionally register a scheme:
+
+```python
+from llm_judge_kit import ProviderResponse, register_provider
+
+class MyProvider:
+    name = "mine"
+    def complete(self, prompt: str, **kwargs: object) -> ProviderResponse:
+        return ProviderResponse(text=call_my_model(prompt))
+
+register_provider("mine", lambda model: MyProvider())
+judge = Judge(provider="mine:v1", rubric="relevance")
+```
+
+## CLI & batch evaluation
+
+Score a whole dataset and get a report — JSON, Markdown, or HTML. A dataset is
+JSON Lines (`prompt` + `response`, optional `context`/`reference`/`id`); see
+[`examples/sample_dataset.jsonl`](examples/sample_dataset.jsonl).
+
+```bash
+llm-judge-kit eval cases.jsonl --provider openai:gpt-5 --rubric factuality --format md
+llm-judge-kit eval cases.jsonl --fail-under 0.9            # exit non-zero in CI if pass rate drops
+llm-judge-kit compare cases.jsonl --provider openai:gpt-5 --provider anthropic:claude-opus-4-8
+llm-judge-kit report report.json --format html -o report.html
+```
+
+Same thing in code ([`examples/benchmark_report.py`](examples/benchmark_report.py)):
+
+```python
+from llm_judge_kit import Judge, load_dataset, run_benchmark, render_markdown
+
+cases = load_dataset("cases.jsonl")
+judge = Judge(provider="openai:gpt-5", rubric="factuality")
+report = run_benchmark(judge, cases, provider="openai:gpt-5", rubric="factuality")
+print(report.pass_rate, report.mean_score)
+print(render_markdown(report))
+```
+
+## Why depend on this
+
+- **Easy to depend on** — zero transitive deps in the core; provider SDKs are opt-in extras.
+- **Reproducible** — deterministic offline `MockProvider`; all unit tests run without network.
+- **Typed** — `mypy --strict` clean; ships `py.typed`.
+- **Robust parsing** — recovers JSON from markdown fences, prose, and trailing commas.
+- **Extensible** — new provider / rubric / judge without core changes.
+
+## Development
+
+```bash
+uv sync --all-extras
+uv run ruff check . && uv run ruff format --check . && uv run mypy src && uv run pytest --cov=llm_judge_kit --cov-report=term-missing
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). The plan of record is in [ROADMAP.md](ROADMAP.md).
+
+## License
+
+MIT — see [LICENSE](LICENSE).

llm_judge_kit-0.1.0/README.md

@@ -0,0 +1,232 @@
+# LLMJudge
+
+> Provider-agnostic, reproducible, typed **LLM-as-a-judge** — a small primitive you can depend on.
+
+[![CI](https://github.com/Nikolay26ru/llm-judge-kit/actions/workflows/ci.yml/badge.svg)](https://github.com/Nikolay26ru/llm-judge-kit/actions/workflows/ci.yml)
+[![Python](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![Typed](https://img.shields.io/badge/typed-mypy%20strict-blue)](https://mypy-lang.org/)
+
+LLMJudge is one tiny, well-tested module for scoring model outputs with an LLM
+judge — the part most projects re-implement badly. The core has **zero required
+runtime dependencies**, a **stable typed API**, and runs **fully offline in
+tests** via a deterministic mock.
+
+## Install
+
+```bash
+pip install llm-judge-kit                 # core only, zero deps
+pip install "llm-judge-kit[openai]"       # + OpenAI-compatible provider
+pip install "llm-judge-kit[anthropic]"    # + Anthropic provider
+pip install "llm-judge-kit[all]"          # all providers
+```
+
+## Quickstart
+
+This runs as-is — no API key, deterministic ([`examples/quickstart.py`](examples/quickstart.py)):
+
+```python
+from llm_judge_kit import Judge, MockProvider
+
+# MockProvider(fixed_score=...) keeps this example deterministic and offline.
+judge = Judge(provider=MockProvider(fixed_score=0.9), rubric="factuality")
+
+result = judge.score(
+    prompt="What is the capital of France?",
+    response="The capital of France is Paris.",
+)
+
+assert result > 0.8  # a JudgeResult compares like its float score
+print(f"score={result.score}  confidence={result.confidence}")
+print(f"passed={result.passed()}  reason={result.reason!r}")
+```
+
+With a **real model**, swap the provider for a spec string — nothing else changes:
+
+```python
+judge = Judge(provider="openai:gpt-5", rubric="factuality")
+result = judge.score(prompt, response)
+if not result.passed(0.7):
+    print("Failed:", result.reason, result.violations)
+```
+
+## Core concepts
+
+| Piece | What it is |
+| --- | --- |
+| `Judge(provider, rubric)` | Pairs a model backend with a rubric; `score()` → `JudgeResult`. |
+| `JudgeResult` | Frozen, typed verdict: `score`, `confidence`, `reason`, `evidence`, `violations`, `raw`, `metadata`. Compares and casts like its `score`. |
+| `Provider` | A `Protocol` with one method, `complete(prompt) -> ProviderResponse`. |
+| `Rubric` | Declarative description of *what* to evaluate; renders a strict-JSON judging prompt. |
+
+### `JudgeResult` ergonomics
+
+```python
+r = judge.score(prompt, response)
+r.score            # float in [0, 1]
+float(r)           # same number
+r > 0.8            # compares like its score
+r.passed(0.7)      # bool against a threshold
+r.reason           # short justification
+r.evidence         # tuple of supporting quotes/facts
+r.violations       # tuple of failed criteria
+r.metadata         # provider, model, token usage, latency, cost
+```
+
+### Built-in rubrics
+
+`factuality`, `groundedness` (requires `context=`), `relevance`,
+`instruction_following`, `safety`. List them with
+`llm_judge_kit.available_rubrics()`.
+
+```python
+judge = Judge(provider="openai:gpt-5", rubric="groundedness")
+result = judge.score(question, answer, context=retrieved_docs)  # RAG check
+```
+
+## Consensus (vote across models)
+
+Run several judge models and aggregate — `confidence` reflects how much they
+**agree** ([`examples/consensus.py`](examples/consensus.py)):
+
+```python
+judge = Judge.consensus(
+    ["openai:gpt-5", "anthropic:claude-opus-4-8", "ollama:llama3"],
+    rubric="factuality",
+)
+result = judge.score(prompt, response)
+result.score          # mean (or median) of member scores
+result.confidence     # high when members agree, low when they diverge
+result.metadata["votes"]   # each member's score
+```
+
+## Reliability & caching
+
+Both wrappers are providers, so they compose around any backend
+([`examples/reliability_and_cache.py`](examples/reliability_and_cache.py)):
+
+```python
+from llm_judge_kit import Judge, OpenAIProvider, RetryProvider, CachingProvider
+
+provider = CachingProvider(                      # memoize identical calls
+    RetryProvider(                               # retry w/ backoff + timeout
+        OpenAIProvider(model="gpt-5"), retries=3, timeout=30,
+    )
+)
+judge = Judge(provider=provider, rubric="factuality")
+```
+
+The cache key is `version + provider + model + prompt + kwargs`, so it is stable
+and invalidates correctly across library versions. Logs are emitted on the
+`llm_judge_kit` logger (silent by default; call `enable_debug_logging()` to see them).
+
+## Integrations
+
+### pytest — eval as ordinary tests
+
+Installing `llm_judge_kit` registers a pytest plugin (no conftest wiring). The
+`llm_judge_kit` fixture turns an eval into a normal test; a failure reads like any
+other failing assertion (score, reason, violations):
+
+```python
+def test_answer_is_grounded(llm_judge_kit):
+    llm_judge_kit.assert_passes(
+        prompt="How tall is the Eiffel Tower?",
+        response=my_rag_pipeline("How tall is the Eiffel Tower?"),
+        rubric="groundedness",
+        context=retrieved_docs,
+        threshold=0.7,
+    )
+```
+
+Pick the judge model once for the whole suite — it defaults to `mock` (offline),
+so tests are green until you point them at a real model:
+
+```bash
+pytest --llm-judge-kit-provider "openai:gpt-5"      # or: export LLM_JUDGE_KIT_PROVIDER=...
+```
+
+### Any framework
+
+LLMJudge judges *strings*, so it drops into any stack — LangChain, LlamaIndex,
+DSPy, a raw script — with no adapter. Whatever produces the output, pass it in:
+
+```python
+output = my_chain.invoke(question)          # LangChain / LlamaIndex / your code
+result = Judge(provider="openai:gpt-5", rubric="relevance").score(question, output)
+```
+
+## Extend without touching the core
+
+Add a **rubric** ([`examples/custom_rubric.py`](examples/custom_rubric.py)):
+
+```python
+from llm_judge_kit import Rubric, register_rubric
+
+register_rubric(Rubric(
+    name="conciseness",
+    description="Whether the response is as short as possible while complete.",
+    criteria=("No filler or repetition.", "Every sentence earns its place."),
+))
+judge = Judge(provider="openai:gpt-5", rubric="conciseness")
+```
+
+Add a **provider** — implement one method, optionally register a scheme:
+
+```python
+from llm_judge_kit import ProviderResponse, register_provider
+
+class MyProvider:
+    name = "mine"
+    def complete(self, prompt: str, **kwargs: object) -> ProviderResponse:
+        return ProviderResponse(text=call_my_model(prompt))
+
+register_provider("mine", lambda model: MyProvider())
+judge = Judge(provider="mine:v1", rubric="relevance")
+```
+
+## CLI & batch evaluation
+
+Score a whole dataset and get a report — JSON, Markdown, or HTML. A dataset is
+JSON Lines (`prompt` + `response`, optional `context`/`reference`/`id`); see
+[`examples/sample_dataset.jsonl`](examples/sample_dataset.jsonl).
+
+```bash
+llm-judge-kit eval cases.jsonl --provider openai:gpt-5 --rubric factuality --format md
+llm-judge-kit eval cases.jsonl --fail-under 0.9            # exit non-zero in CI if pass rate drops
+llm-judge-kit compare cases.jsonl --provider openai:gpt-5 --provider anthropic:claude-opus-4-8
+llm-judge-kit report report.json --format html -o report.html
+```
+
+Same thing in code ([`examples/benchmark_report.py`](examples/benchmark_report.py)):
+
+```python
+from llm_judge_kit import Judge, load_dataset, run_benchmark, render_markdown
+
+cases = load_dataset("cases.jsonl")
+judge = Judge(provider="openai:gpt-5", rubric="factuality")
+report = run_benchmark(judge, cases, provider="openai:gpt-5", rubric="factuality")
+print(report.pass_rate, report.mean_score)
+print(render_markdown(report))
+```
+
+## Why depend on this
+
+- **Easy to depend on** — zero transitive deps in the core; provider SDKs are opt-in extras.
+- **Reproducible** — deterministic offline `MockProvider`; all unit tests run without network.
+- **Typed** — `mypy --strict` clean; ships `py.typed`.
+- **Robust parsing** — recovers JSON from markdown fences, prose, and trailing commas.
+- **Extensible** — new provider / rubric / judge without core changes.
+
+## Development
+
+```bash
+uv sync --all-extras
+uv run ruff check . && uv run ruff format --check . && uv run mypy src && uv run pytest --cov=llm_judge_kit --cov-report=term-missing
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). The plan of record is in [ROADMAP.md](ROADMAP.md).
+
+## License
+
+MIT — see [LICENSE](LICENSE).