evalstone 0.2.0__tar.gz → 0.2.2__tar.gz

{evalstone-0.2.0 → evalstone-0.2.2}/CHANGELOG.md

@@ -2,14 +2,48 @@
 
 All notable changes to `python/evalstone`. Semantic versioning; tags `lib-vX.Y.Z`.
 
+## 0.2.2 — 2026-10-14
+
+Judge replies cut off by the token cap, found by the first run against a real
+vendor (Claude Haiku 5.5 via OpenRouter on the rag-bot demo: 4 of 10 verdicts
+errored).
+
+- `judge` and `comparison.pairwise` default `max_tokens` is now 1024 (was 256).
+  The cut replies needed 286–349 tokens: a verdict plus a long one-sentence reason.
+  `max_tokens` is not part of the judge version, so cached verdicts stay valid.
+- A reply the provider stopped at the cap (`length`, `max_tokens`, `MAX_TOKENS`)
+  whose JSON is unclosed or invalid now errors with "reply truncated at
+  max_tokens=N; raise the check's max_tokens" followed by the original parse error and excerpt
+  instead of only "no JSON object in judge reply". A complete verdict at the cap is
+  accepted, and a complete but wrong one (score out of range) keeps its own error.
+  `ChatResponse.truncated` exposes the check.
+- `comparison.pairwise`: an unparseable judge reply now counts that call's cost
+  (and latency) in the errored score, as `judge` already did.
+
+## 0.2.1 — 2026-10-14
+
+Attach fixes found by the docs audit.
+
+- `comparison.pairwise` built its check version (and so its cache key) from
+  `--judge-model` or the check's own `model:`, ignoring the attached project's
+  per-check override — the call itself already honoured it. Both now resolve through
+  the same precedence (`--judge-model` > center override > `model:` > default), so a
+  verdict cached under the override is found again and a changed override invalidates
+  the cache.
+- Docstrings and this changelog no longer claim prompt targets are proxied when
+  attached; they never were (`providers:` + env keys serve the system under test).
+- Known limitation, documented: when attached, `--judge-model some-typo/model` is
+  not validated up front — the center only rejects the provider at the first judge
+  call, which surfaces as an errored score. Standalone runs still fail fast.
+
 ## 0.2.0 — 2026-10-14
 
 Attach a project to a center.
 
 - `center:` in `evalstone.yaml` (`url`, `project`, `api_key_env` — an `es_…` ingest key)
-  routes judge and prompt-target calls through the center's model proxy, so no provider
-  key lives on the machine running the suite; the project's judge settings pick the
-  model (per-check overrides, then the default), `--judge-model` still wins.
+  routes judge calls through the center's model proxy, so no judge key lives on the
+  machine running the suite; the project's judge settings pick the model (per-check
+  overrides, then the default), `--judge-model` still wins.
 - Judge verdicts are cached in the center (`CenterCache`); a center hiccup is a cache
   miss, never a failed run.
 - `human` checks of an attached project look labels up in the center (`labels: center`,

{evalstone-0.2.0 → evalstone-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalstone
-Version: 0.2.0
+Version: 0.2.2
 Summary: The plug-and-play evals layer for LLM-based projects: datasets, checks, suites, runs and gates.
 Project-URL: Homepage, https://evalstone.com
 Project-URL: Documentation, https://evalstone.com/docs

{evalstone-0.2.0 → evalstone-0.2.2}/src/evalstone/__init__.py

@@ -8,7 +8,7 @@ import types
 from importlib import import_module
 from typing import TYPE_CHECKING, Any
 
-__version__ = "0.2.0"
+__version__ = "0.2.2"
 
 # Exports resolve on first access (PEP 562) so that importing the package — which
 # the pytest plugin does in every pytest session on the interpreter — costs

{evalstone-0.2.0 → evalstone-0.2.2}/src/evalstone/center.py

@@ -1,11 +1,13 @@
 """Attach a project to an Evalstone center (``center:`` in evalstone.yaml).
 
-Attached, the library never holds a provider key: judge (and prompt-target) calls go
-through the center's model proxy, which resolves the model from the project's judge
-settings and dials the provider with the organization's key server-side. The judge
-cache lives in the center (one verdict serves every machine), ``human`` checks look
-labels up there, and every ``evalstone run`` pushes its report. ``providers:`` and
-``judges:`` in the YAML remain the standalone path.
+Attached, judge calls go through the center's model proxy, which resolves the model
+from the project's judge settings and dials the provider with the organization's key
+server-side, so no judge key lives on the machine running the suite. Prompt targets
+(the system under test) are NOT proxied: they keep using the local ``providers:`` and
+their env keys. The judge cache lives in the center (one verdict serves every
+machine), ``human`` checks look labels up there, and every ``evalstone run`` pushes
+its report. ``judges:`` in the YAML is ignored while attached; ``providers:`` still
+serves prompt targets.
 """
 
 from __future__ import annotations

{evalstone-0.2.0 → evalstone-0.2.2}/src/evalstone/checks/comparison.py

@@ -12,7 +12,7 @@ from typing import Any
 
 from evalstone._json import sha256
 from evalstone.checks._base import FunctionCheck, judge_model_for
-from evalstone.checks.judge import _json_text, extract_json
+from evalstone.checks.judge import DEFAULT_MAX_TOKENS, _json_text, explain_unparsed, extract_json
 from evalstone.dataset import Sample
 from evalstone.output import Output
 from evalstone.providers import ChatMessage, ProviderError
@@ -46,6 +46,14 @@ def parse_winner(text: str) -> tuple[str, str]:
     return winner, str(data.get("reason", "")).strip()
 
 
+class _Unparsed(ValueError):
+    """An unparseable judge reply, with what the call cost."""
+
+    def __init__(self, message: str, cost_usd: float, latency_ms: float):
+        super().__init__(message)
+        self.cost_usd, self.latency_ms = cost_usd, latency_ms
+
+
 def pairwise(
     key: str,
     rubric: str,
@@ -53,7 +61,7 @@ def pairwise(
     baseline: Target | Mapping[str, Any],
     model: str | None = None,
     both_orders: bool = True,
-    max_tokens: int = 256,
+    max_tokens: int = DEFAULT_MAX_TOKENS,
 ) -> FunctionCheck:
     """The candidate output beats the ``baseline`` output for the same sample.
 
@@ -66,17 +74,25 @@ def pairwise(
     identity = {"v": 1, "rubric": rubric, "both_orders": both_orders}
     unpriced: list[str] = []  # models of this sample's calls that the price table lacks
 
-    def ask(ctx: Context, sample_input: str, a: str, b: str) -> tuple[str, str, float, float]:
+    def ask(
+        ctx: Context, chosen: str | None, sample_input: str, a: str, b: str
+    ) -> tuple[str, str, float, float]:
         user = USER.format(input=sample_input, a=a, b=b)
         resp = ctx.providers.complete(
-            judge_model_for(ctx, key, model),
+            chosen,
             {
                 "system": SYSTEM.format(rubric=rubric.strip()),
                 "messages": [ChatMessage(role="user", content=user)],
                 "max_tokens": max_tokens,
             },
         )
-        winner, reason = parse_winner(resp.text)
+        try:
+            winner, reason = parse_winner(resp.text)
+        except ValueError as exc:
+            # the failed call was still billed: carry its cost to the errored score
+            raise _Unparsed(
+                explain_unparsed(exc, resp, max_tokens), resp.cost_usd or 0.0, resp.latency_ms
+            ) from None
         if resp.cost_usd is None:
             unpriced.append(resp.model)
         return winner, reason, resp.cost_usd or 0.0, resp.latency_ms
@@ -85,8 +101,9 @@ def pairwise(
         if ctx.providers is None:
             return Score.errored("pairwise: no providers configured")
         unpriced.clear()
+        chosen = judge_model_for(ctx, key, model)  # resolved once: the version AND the call
         try:
-            resolved = "/".join(ctx.providers.split(ctx.judge_model or model))
+            resolved = "/".join(ctx.providers.split(chosen))
         except ProviderError as exc:
             return Score.errored(str(exc))
         version = sha256({**identity, "model": resolved})[:16]
@@ -111,7 +128,7 @@ def pairwise(
             if both_orders:
                 orders.append(("base_first", base.text, output.text))
             for label, a, b in orders:
-                winner, reason, c, ms = ask(ctx, text, a, b)
+                winner, reason, c, ms = ask(ctx, chosen, text, a, b)
                 cost += c
                 latency += ms
                 candidate_is_a = label == "cand_first"
@@ -124,6 +141,12 @@ def pairwise(
                 reasons.append(reason)
         except ProviderError as exc:
             return Score.errored(str(exc), cost_usd=cost, latency_ms=latency)
+        except _Unparsed as exc:
+            return Score.errored(
+                f"pairwise: {exc}",
+                cost_usd=cost + exc.cost_usd,
+                latency_ms=latency + exc.latency_ms,
+            )
         except ValueError as exc:
             return Score.errored(f"pairwise: {exc}", cost_usd=cost, latency_ms=latency)
         value = sum(points) / len(points)

{evalstone-0.2.0 → evalstone-0.2.2}/src/evalstone/checks/judge.py

@@ -16,11 +16,15 @@ from evalstone._json import dumps, sha256
 from evalstone.checks._base import FunctionCheck, judge_model_for
 from evalstone.dataset import Sample
 from evalstone.output import Output
-from evalstone.providers import ChatMessage, ProviderError
+from evalstone.providers import ChatMessage, ChatResponse, ProviderError
 from evalstone.score import CheckKind, Context, Score
 
 PROMPT_VERSION = 1
 
+# Room for a verdict plus a long reason: 256 cut real Haiku 5.5 replies mid-reason
+# (4 of 10 on the rag-bot demo). The cap only bounds a runaway; it is not in the version.
+DEFAULT_MAX_TOKENS = 1024
+
 SYSTEM_TEMPLATE = """You are a strict evaluator grading one answer produced by an AI system.
 
 RUBRIC:
@@ -67,6 +71,22 @@ def parse_verdict(text: str, scale: int) -> tuple[int, str]:
     return score, str(data.get("reason", "")).strip()
 
 
+def explain_unparsed(exc: ValueError, resp: ChatResponse, max_tokens: int) -> str:
+    """The parse error, prefixed with the cause when the cap cut the JSON itself.
+
+    Only an unclosed or invalid object is blamed on the cap: a complete object that is
+    wrong (score out of range, no winner) is the model's answer, not a truncation.
+    """
+    if resp.truncated:
+        try:
+            extract_json(resp.text)
+        except ValueError:
+            return (
+                f"reply truncated at max_tokens={max_tokens}; raise the check's max_tokens ({exc})"
+            )
+    return str(exc)
+
+
 class JudgeCheck(FunctionCheck):
     """A ``FunctionCheck`` whose function calls a model; exposes the judge version hash."""
 
@@ -99,7 +119,7 @@ def judge(
     pass_level: int | None = None,
     threshold: float | None = None,
     include_reference: bool = False,
-    max_tokens: int = 256,
+    max_tokens: int = DEFAULT_MAX_TOKENS,
 ) -> JudgeCheck:
     """Grade an output against ``rubric`` with a model.
 
@@ -190,7 +210,9 @@ def judge(
             n, reason = parse_verdict(resp.text, scale)
         except ValueError as exc:
             return Score.errored(
-                f"judge: {exc}", cost_usd=resp.cost_usd or 0.0, latency_ms=resp.latency_ms
+                f"judge: {explain_unparsed(exc, resp, max_tokens)}",
+                cost_usd=resp.cost_usd or 0.0,
+                latency_ms=resp.latency_ms,
             )
         score = to_score(
             n,
@@ -217,4 +239,12 @@ def _verdict_reason(score: Score) -> str:
     return str((score.raw or {}).get("verdict_reason", ""))
 
 
-__all__ = ["PROMPT_VERSION", "JudgeCheck", "extract_json", "judge", "parse_verdict"]
+__all__ = [
+    "DEFAULT_MAX_TOKENS",
+    "PROMPT_VERSION",
+    "JudgeCheck",
+    "explain_unparsed",
+    "extract_json",
+    "judge",
+    "parse_verdict",
+]

{evalstone-0.2.0 → evalstone-0.2.2}/src/evalstone/cli.py

@@ -145,6 +145,10 @@ def run(
     if judge_model is not None:
         try:
             registry = cfg.provider_registry()
+        except CenterError as exc:  # attached without a key: the center, not the flag
+            err.print(f"[red]{escape(str(exc))}[/red]")
+            raise typer.Exit(2) from None
+        try:
             registry.get(registry.split(judge_model)[0])
         except ProviderError as exc:
             err.print(f"[red]--judge-model: {escape(str(exc))}[/red]")

{evalstone-0.2.0 → evalstone-0.2.2}/src/evalstone/providers/__init__.py

@@ -42,6 +42,11 @@ class ChatResponse(BaseModel):
     stop_reason: str | None = None
     raw: Any = None
 
+    @property
+    def truncated(self) -> bool:
+        """The provider stopped at the max_tokens cap (OpenAI-compatible, Anthropic, Gemini)."""
+        return (self.stop_reason or "").lower() in {"length", "max_tokens"}
+
 
 class ProviderError(Exception):
     """A provider call failed (HTTP error, refusal, malformed response)."""

{evalstone-0.2.0 → evalstone-0.2.2}/src/evalstone/score.py

@@ -95,7 +95,8 @@ class ScoreCache(Protocol):
 
 
 class MemoryCache:
-    """An in-process ``ScoreCache``. The center-backed cache arrives with Phase 3."""
+    """An in-process ``ScoreCache`` (the standalone default; attached projects use
+    ``evalstone.center.CenterCache``)."""
 
     def __init__(self) -> None:
         self._data: dict[str, Score] = {}

{evalstone-0.2.0 → evalstone-0.2.2}/tests/test_center.py

@@ -312,6 +312,45 @@ def test_pairwise_honours_the_center_override(cfg):
     assert json.loads(calls.calls[0].request.content)["model"] == "other/big"
 
 
+@respx.mock
+def test_pairwise_cache_key_follows_the_center_override(cfg):
+    """The check version (and so the cache key) is built from the same resolved model as
+    the call: a verdict cached under the override is hit again, a changed override is a
+    different version and misses (regression: the key used ``ctx.judge_model or model``)."""
+    from evalstone.checks.comparison import pairwise
+    from evalstone.score import MemoryCache
+
+    calls = respx.post(f"{P}/model/complete").mock(
+        return_value=_verdict(text='{"winner": "A", "reason": "clearer"}')
+    )
+    client = CenterClient(cfg)
+    providers = CenterProviders(client, default_model="judge/small")
+    extra = {"center_client": client, "judge_models": {"better": "other/big"}}
+    cache = MemoryCache()
+    check = pairwise(
+        "better", "Which is clearer?", baseline={"s": "baseline answer"}, model="judge/small"
+    )
+    sample, output = Sample(id="s", input="q"), Output(value="candidate")
+
+    ctx = Context(providers=providers, extra=extra, cache=cache)
+    check.evaluate(sample, output, ctx)
+    filled = calls.call_count  # both orders were asked once
+    assert len(cache._data) == 1
+    key_before = next(iter(cache._data))
+    assert check.evaluate(sample, output, ctx).raw["cached"] is True
+    assert calls.call_count == filled  # served from the cache
+
+    ctx_other = Context(
+        providers=providers,
+        extra={**extra, "judge_models": {"better": "other/small"}},
+        cache=cache,
+    )
+    assert check.evaluate(sample, output, ctx_other).raw.get("cached") is not True
+    assert calls.call_count > filled  # a changed override is a different version → miss
+    assert json.loads(calls.calls[-1].request.content)["model"] == "other/small"
+    assert len(cache._data) == 2 and key_before in cache._data
+
+
 @respx.mock
 def test_prompt_targets_stay_local_when_attached(tmp_path, monkeypatch):
     from evalstone.providers import Providers

{evalstone-0.2.0 → evalstone-0.2.2}/tests/test_cli.py

@@ -143,3 +143,18 @@ def test_show_markdown(in_dir):
         "### Evalstone · `nightly` — ✅ gate passed"
     )
     assert "| `must_include` |" in res.output
+
+
+def test_missing_center_key_is_reported_as_the_center_not_the_flag(in_dir, monkeypatch):
+    """Attached without EVALSTONE_API_KEY, ``--judge-model`` must not blame the flag."""
+    monkeypatch.delenv("EVALSTONE_API_KEY", raising=False)
+    runner.invoke(app, ["init", "--project", "demo"])
+    cfg = in_dir / "evalstone.yaml"
+    cfg.write_text(
+        cfg.read_text()
+        + "center:\n  url: http://center.test\n  project: 11111111-2222-4333-8444-555555555555\n"
+    )
+    res = runner.invoke(app, ["run", "nightly", "--judge-model", "openai/gpt-x"])
+    assert res.exit_code == 2
+    assert "EVALSTONE_API_KEY" in res.output
+    assert "--judge-model" not in res.output

{evalstone-0.2.0 → evalstone-0.2.2}/tests/test_docs.py

@@ -51,3 +51,17 @@ def test_every_action_snippet_uses_the_same_ref():
             )
         )
     assert refs == {"action-v0"}, f"action refs in the docs: {sorted(refs)}"
+
+
+def test_site_changelog_is_generated_from_the_library_changelog():
+    """The site's Changelog page is `scripts/sync-changelog.mjs` output committed next to
+    the docs; a release that bumps CHANGELOG.md without re-running it is caught here."""
+    changelog = (SITE_DOCS.parents[2].parent / "python" / "evalstone" / "CHANGELOG.md").read_text(
+        encoding="utf-8"
+    )
+    page = (SITE_DOCS / "evalstone" / "changelog.mdx").read_text(encoding="utf-8")
+    body = changelog[changelog.index("\n## ") + 1 :].rstrip()
+    assert "GENERATED by scripts/sync-changelog.mjs" in page
+    assert page.rstrip().endswith(body), (
+        "run `npm run build` (or scripts/sync-changelog.mjs) in the site"
+    )

{evalstone-0.2.0 → evalstone-0.2.2}/tests/test_judge.py

@@ -16,9 +16,9 @@ class FakeProvider:
 
     name = "openai"
 
-    def __init__(self, reply=None, fail=None, judge_reply=None):
+    def __init__(self, reply=None, fail=None, judge_reply=None, stop_reason=None):
         self.reply, self.fail, self.calls = reply, fail, []
-        self.judge_reply = judge_reply
+        self.judge_reply, self.stop_reason = judge_reply, stop_reason
 
     def complete(self, request):
         self.calls.append(request)
@@ -41,6 +41,7 @@ class FakeProvider:
             output_tokens=10,
             cost_usd=0.001,
             latency_ms=12.0,
+            stop_reason=self.stop_reason,
         )
 
 
@@ -114,6 +115,44 @@ def test_judge_errors_are_never_passes():
     assert no_model.error.startswith("no model given")
 
 
+# What a real Haiku 5.5 reply looked like when max_tokens=256 cut it mid-reason.
+CUT = '{"score": 4, "reason": "The answer directly answers the question with a concrete rule (30 da'
+
+
+@pytest.mark.parametrize("stop", ["length", "max_tokens", "MAX_TOKENS"])
+def test_a_reply_cut_at_the_token_cap_says_so_and_never_passes(stop):
+    chk = judge("h", "r", max_tokens=300)
+    score = chk(S, Output(value="x"), ctx_with(FakeProvider(judge_reply=CUT, stop_reason=stop)))
+    assert score.passed is False and score.cost_usd == 0.001
+    assert score.error.startswith(
+        "judge: reply truncated at max_tokens=300; raise the check's max_tokens "
+        "(no JSON object in judge reply: "
+    )
+    assert "concrete rule (30 da" in score.error  # the excerpt of what came back survives
+
+
+def test_truncation_only_explains_a_failed_parse():
+    # A complete verdict wins even if the provider then hit the cap.
+    full = ctx_with(
+        FakeProvider(judge_reply='{"score": 5, "reason": "ok"} and', stop_reason="length")
+    )
+    assert judge("h", "r")(S, Output(value="x"), full).passed is True
+    # A complete but wrong verdict at the cap is the model's answer, not a truncation.
+    wrong = ctx_with(
+        FakeProvider(judge_reply='{"score": 7, "reason": "x"} and', stop_reason="length")
+    )
+    assert judge("h", "r")(S, Output(value="x"), wrong).error == "judge: judge score 7 outside 1..5"
+    # A normal stop keeps the parse error as it was.
+    plain = ctx_with(FakeProvider(judge_reply=CUT, stop_reason="stop"))
+    assert "no JSON object" in judge("h", "r")(S, Output(value="x"), plain).error
+
+
+def test_judge_default_token_budget_fits_a_long_reason():
+    prov = FakeProvider()
+    judge("h", "r")(S, Output(value="30 days"), ctx_with(prov))
+    assert prov.calls[0].max_tokens == 1024
+
+
 def test_judge_cache_makes_the_second_identical_call_free():
     prov = FakeProvider()
     ctx = ctx_with(prov)

{evalstone-0.2.0 → evalstone-0.2.2}/tests/test_rag_and_comparison.py

@@ -244,3 +244,35 @@ def test_config_baseline_mapping_form_and_target_form(tmp_path):
     c.providers.register("openai", JudgeFake(winner='{"winner": "A"}'))
     run = suite.run(target, ctx=c)
     assert run.summary.errors == 0 and run.summary.results == 2
+
+
+def test_pairwise_reply_cut_at_the_token_cap_says_so():
+    class Cut(JudgeFake):
+        def complete(self, request):
+            self.calls.append(request)
+            return ChatResponse(
+                text='{"winner": "A", "reason": "Answer A is more compl',
+                model=request.model,
+                cost_usd=0.001,
+                stop_reason="length",
+            )
+
+    fake = Cut()
+    score = pairwise("b", "r", baseline={"s": "x"})(S, Output(value="y"), ctx(fake))
+    assert score.passed is False
+    assert score.error.startswith("pairwise: reply truncated at max_tokens=1024")
+    assert fake.calls[0].max_tokens == 1024
+    assert score.cost_usd == pytest.approx(0.001)  # the failed call was billed: it counts
+
+
+def test_pairwise_unparseable_second_order_keeps_every_call_cost():
+    replies = iter(['{"winner": "A", "reason": "x"}', "no verdict here"])
+
+    class Half(JudgeFake):
+        def complete(self, request):
+            self.calls.append(request)
+            return ChatResponse(text=next(replies), model=request.model, cost_usd=0.001)
+
+    score = pairwise("b", "r", baseline={"s": "x"})(S, Output(value="y"), ctx(Half()))
+    assert score.passed is False and "no JSON object" in score.error
+    assert score.cost_usd == pytest.approx(0.002)