judgetap 0.2.1.dev42001__tar.gz → 0.2.1.dev44001__tar.gz

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/PKG-INFO

@@ -1,12 +1,14 @@
 Metadata-Version: 2.5
 Name: judgetap
-Version: 0.2.1.dev42001
+Version: 0.2.1.dev44001
 Summary: Fast typed decisions (choice, score, yes/no) across Jev-style engines, plus a guard for coding agents. Early development.
 Project-URL: Homepage, https://github.com/mergesafe-ai/judgetap
 Author-email: Omer Bar-Ness <omer@zsquared.io>
 License-Expression: Apache-2.0
 License-File: LICENSE
 Requires-Python: >=3.12
+Provides-Extra: gliner
+Requires-Dist: gliner2; extra == 'gliner'
 Provides-Extra: keychain
 Requires-Dist: keyring>=24; extra == 'keychain'
 Provides-Extra: laya
@@ -44,7 +46,7 @@ judgetap guard test "git push --force origin main"
 
 - **Guard.** A hook that checks every shell command (and, in Claude Code, every file write and edit) before it runs. Rules catch common destructive forms: recursive deletes outside the workspace, force-pushes and pushes to protected branches, `DROP`/`DELETE` without `WHERE`, `terraform destroy`, and secrets written to files. With an engine configured, a model judges the rest: is it irreversible? off-task? against a rule in `AGENTS.md`? Without an engine, the rules fail closed for shell commands (anything they can't vouch for asks you); file writes and edits are checked for secrets, your own rules, and writes to the guard's own configuration (which ask).
 - **Library.** One API (`choice`, `score`, `yesno`, `batch`) over every Jev-style decision engine, with a cascade that escalates low-confidence answers to a stronger engine.
-- **Eval.** `judgetap eval cases.jsonl --engines jev,laya` compares engines on your labelled cases: accuracy, calibration (ECE), latency and cost.
+- **Eval.** `judgetap eval cases.jsonl --engines jev,laya` compares engines on your labelled cases: accuracy, calibration (ECE), latency and cost. `judgetap eval --suite banking77 --limit 200 --engines jev` runs a public suite instead (`ag_news`, `banking77`): it is downloaded from Hugging Face on first use, converted to judgetap's case format and cached in `~/.judgetap/suites`; nothing is bundled. `--limit N` takes the same fixed-seed sample every run. Each suite's licence and source URL are listed in `src/judgetap/suites.py`.
 - **Dashboard.** `judgetap dashboard` is a local page with recent decisions, holds, asks, latency and cost per engine. You can mark a hold as a false alarm.
 
 ## Library
@@ -74,6 +76,7 @@ sj.configure(sj.Cascade([jev, flash]))
 | `jev@<url>` / `typesafe:<url>` | Any TypeSafe-compatible server | none on localhost; remote needs the key and https |
 | `laya` | Laya, open weights, runs in process (`pip install "judgetap[laya]"`) | none |
 | `julia` / `julia:<path>` | [Julia-1](https://huggingface.co/SupersonicLabs/Julia-1), 144M open weights, runs in process on CPU (download into `~/.judgetap/models/Julia-1` and `pip install -e` it; relative paths never come from the working directory). Loads per process, so for the guard prefer a server engine | none |
+| `gliner[:<hf model>]` | Fastino GLiNER2.5-Decide, 340M encoder, CPU or GPU (`pip install "judgetap[gliner]"`); only a full probability map over every option counts as calibrated; a winner-only score (the others share the rest evenly) or a bare label is uncalibrated. Loads in-process, so the guard refuses it: use a server engine there | none |
 | `agentjev` / `agentjev:<url>` | A local AgentJev server | none |
 | `llm:<model>` | Any LiteLLM model (`pip install "judgetap[llm]"`); probabilities self-reported | the provider's |
 

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/README.md

@@ -27,7 +27,7 @@ judgetap guard test "git push --force origin main"
 
 - **Guard.** A hook that checks every shell command (and, in Claude Code, every file write and edit) before it runs. Rules catch common destructive forms: recursive deletes outside the workspace, force-pushes and pushes to protected branches, `DROP`/`DELETE` without `WHERE`, `terraform destroy`, and secrets written to files. With an engine configured, a model judges the rest: is it irreversible? off-task? against a rule in `AGENTS.md`? Without an engine, the rules fail closed for shell commands (anything they can't vouch for asks you); file writes and edits are checked for secrets, your own rules, and writes to the guard's own configuration (which ask).
 - **Library.** One API (`choice`, `score`, `yesno`, `batch`) over every Jev-style decision engine, with a cascade that escalates low-confidence answers to a stronger engine.
-- **Eval.** `judgetap eval cases.jsonl --engines jev,laya` compares engines on your labelled cases: accuracy, calibration (ECE), latency and cost.
+- **Eval.** `judgetap eval cases.jsonl --engines jev,laya` compares engines on your labelled cases: accuracy, calibration (ECE), latency and cost. `judgetap eval --suite banking77 --limit 200 --engines jev` runs a public suite instead (`ag_news`, `banking77`): it is downloaded from Hugging Face on first use, converted to judgetap's case format and cached in `~/.judgetap/suites`; nothing is bundled. `--limit N` takes the same fixed-seed sample every run. Each suite's licence and source URL are listed in `src/judgetap/suites.py`.
 - **Dashboard.** `judgetap dashboard` is a local page with recent decisions, holds, asks, latency and cost per engine. You can mark a hold as a false alarm.
 
 ## Library
@@ -57,6 +57,7 @@ sj.configure(sj.Cascade([jev, flash]))
 | `jev@<url>` / `typesafe:<url>` | Any TypeSafe-compatible server | none on localhost; remote needs the key and https |
 | `laya` | Laya, open weights, runs in process (`pip install "judgetap[laya]"`) | none |
 | `julia` / `julia:<path>` | [Julia-1](https://huggingface.co/SupersonicLabs/Julia-1), 144M open weights, runs in process on CPU (download into `~/.judgetap/models/Julia-1` and `pip install -e` it; relative paths never come from the working directory). Loads per process, so for the guard prefer a server engine | none |
+| `gliner[:<hf model>]` | Fastino GLiNER2.5-Decide, 340M encoder, CPU or GPU (`pip install "judgetap[gliner]"`); only a full probability map over every option counts as calibrated; a winner-only score (the others share the rest evenly) or a bare label is uncalibrated. Loads in-process, so the guard refuses it: use a server engine there | none |
 | `agentjev` / `agentjev:<url>` | A local AgentJev server | none |
 | `llm:<model>` | Any LiteLLM model (`pip install "judgetap[llm]"`); probabilities self-reported | the provider's |
 

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/docs/SPEC.md

@@ -37,6 +37,7 @@ sj.batch([...questions], context=...)                -> list[Decision]  # one pa
 | `jev` | hosted (TypeSafe console, Vercel AI Gateway) | native batch; the reference shape |
 | `laya` | local, open weights (Apache-2.0) | via transformers; GPU optional |
 | `julia` | local, open weights (Apache-2.0), 144M | Julia-1 runtime from its model repo; CPU by default; Jev-shaped API; 2-20 options per question |
+| `gliner` | local, open weights (Apache-2.0), Fastino GLiNER2.5-Decide via `gliner2` | all questions as heads in one pass; full probabilities when gliner2 returns them, else the winner's probability with the rest split evenly, else the label alone (`calibrated=False`) |
 | `agentjev` | local, open weights | ~50 ms per pass |
 | `llm` | any structured-output LLM via LiteLLM | OpenAI, Gemini, Anthropic, Ollama; probabilities are the model's own JSON estimate, flagged `calibrated=False` (see `?logprobs` below) |
 | `llm:<model>?logprobs` | same, OpenAI-compatible servers exposing `logprobs` | options listed as letters, one single-token call per question; the distribution is the letters' `top_logprobs` renormalised (missing letters get 0, none present is an error); token probabilities, not calibrated (`calibrated=False`; check with `judgetap eval`); per-question calls run concurrently (max 8) and each is reported as a Call; more than 26 options is a local error raised before any request; falls back to JSON mode for good if the provider rejects logprobs |
@@ -52,7 +53,7 @@ escalate_below = 0.8      # p under this goes to the next engine
 on_exhausted = "raise"    # or "return_last", or a callback (e.g. ask a human)
 ```
 
-Each hop is recorded on the `Decision`. Engine errors and timeouts fall through the same way.
+Each hop is recorded on the `Decision`. Engine errors and timeouts fall through the same way, and so does an uncalibrated answer (`calibrated=False`) whatever its p, while an engine is left to ask; the last engine's answer is judged on p alone.
 
 ### 4. Calibration check
 

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "judgetap"
-version = "0.2.1.dev42001"
+version = "0.2.1.dev44001"
 description = "Fast typed decisions (choice, score, yes/no) across Jev-style engines, plus a guard for coding agents. Early development."
 readme = "README.md"
 license = "Apache-2.0"
@@ -17,6 +17,7 @@ snapjudge = "judgetap.cli:main"  # deprecated alias, removed after one release
 [project.optional-dependencies]
 llm = ["litellm>=1.40"]
 laya = ["laya"]
+gliner = ["gliner2"]
 keychain = ["keyring>=24"]
 
 [build-system]

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/src/judgetap/__init__.py

@@ -23,7 +23,7 @@ from judgetap.errors import (
 )
 from judgetap.types import Decision, Question
 
-__version__ = "0.2.1.dev42001"  # x-release-please-version
+__version__ = "0.2.1.dev44001"  # x-release-please-version
 
 __all__ = [
     "Cascade",

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/src/judgetap/cascade.py

@@ -175,11 +175,16 @@ class Cascade:
     def _pending(self, questions, attempts) -> list[int]:
         return [i for i in range(len(questions)) if not self._confident(attempts[i])]
 
-    def _confident(self, attempts: Sequence[Attempt]) -> bool:
+    def _confident(self, attempts: Sequence[Attempt], *, final: bool = False) -> bool:
+        """Whether the latest answer stops the cascade. An uncalibrated p
+        (calibrated=False) never stops it while another engine is left to
+        ask; once none is (final), it is judged on p like any other."""
+        last = attempts[-1] if attempts else None
         return (
-            bool(attempts)
-            and attempts[-1].p is not None
-            and (attempts[-1].p >= self.escalate_below)
+            last is not None
+            and last.p is not None
+            and last.p >= self.escalate_below
+            and (final or last.answer is None or last.answer.calibrated)
         )
 
     def _record(self, engine, questions, pending, answers, attempts) -> None:
@@ -220,7 +225,7 @@ class Cascade:
         hops = _hops(attempts)
         answered = [a for a in attempts if a.answer is not None]
         spent = _spent(attempts)
-        if self._confident(attempts):
+        if self._confident(attempts, final=True):
             return _as_result(attempts[-1], hops, spent)
         if callable(self.on_exhausted):
             raw = RawAnswer(

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/src/judgetap/engines/__init__.py

@@ -58,8 +58,13 @@ def load(spec: str | None = None) -> Engine:
         from judgetap.engines.agentjev import DEFAULT_URL, AgentJevEngine
 
         return AgentJevEngine(url=arg or DEFAULT_URL)
+    if name == "gliner":
+        from judgetap.engines.gliner import DEFAULT_MODEL as GLINER_MODEL
+        from judgetap.engines.gliner import GlinerEngine
+
+        return GlinerEngine(model=arg or GLINER_MODEL)
     raise JudgetapError(
-        f"unknown engine {name!r} in spec {spec!r}; known: jev, jev@<url>, typesafe:<url>, llm, laya, julia, agentjev"
+        f"unknown engine {name!r} in spec {spec!r}; known: jev, jev@<url>, typesafe:<url>, llm, laya, julia, agentjev, gliner"
     )
 
 

judgetap-0.2.1.dev44001/src/judgetap/engines/gliner.py

@@ -0,0 +1,132 @@
+"""GLiNER2.5-Decide (Fastino, Apache-2.0) in process, through `gliner2`.
+
+A 340M DeBERTa encoder that picks labels for several questions ("heads") in
+one forward pass, on CPU or GPU (huggingface.co/fastino/GLiNER2.5-Decide).
+Needs `pip install 'judgetap[gliner]'`; the checkpoint downloads from
+Hugging Face on first use.
+
+What comes back depends on the gliner2 code path. Only a full `probabilities`
+map covering every option is marked `calibrated`; anything less is
+`calibrated=False`, so a cascade escalates it to its next engine whatever
+its p:
+- a full `probabilities` map per head (the classifier path): renormalised
+  over the options, calibrated;
+- a map missing some options: renormalised over the ones present, uncalibrated;
+- `{"label", "confidence"}` (the extractor path): the winner gets its score
+  and the other labels split the rest evenly, uncalibrated (gliner2 doesn't
+  report them, and the score isn't a calibrated probability);
+- a bare label (older versions): p=1.0, uncalibrated.
+
+Loaded models are cached per process (by model id, at most
+MAX_CACHED_MODELS). The guard refuses this engine: its hook is a new process
+per action, so the model would load on every guarded action.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import threading
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+from judgetap.engine import Context, RawAnswer, plain_context
+from judgetap.errors import JudgetapError
+from judgetap.types import NO, YES, Question
+
+DEFAULT_MODEL = "fastino/GLiNER2.5-Decide"
+MAX_CACHED_MODELS = 2
+_models: dict[str, Any] = {}
+_models_lock = threading.Lock()
+
+
+class GlinerError(JudgetapError):
+    """gliner2 is missing, failed to load or classify, or answered in an
+    unexpected shape. The underlying exception is chained."""
+
+
+def _task(q: Question) -> dict[str, Any]:
+    """One classification head: its labels, with the question as the prompt."""
+    labels = [YES, NO] if q.kind == "yesno" else list(q.options)
+    return {"labels": labels, "prompt": q.text}
+
+
+def _text(context: Context) -> str:
+    state = plain_context(context)
+    return state if isinstance(state, str) else json.dumps(state)
+
+
+def _distribution(q: Question, answer: Any) -> tuple[dict[str, float], bool]:
+    """(distribution over the question's options, calibrated)."""
+    options = list(q.options)
+    if isinstance(answer, Mapping) and isinstance(answer.get("probabilities"), Mapping):
+        probs = {str(k): float(v) for k, v in answer["probabilities"].items()}
+        total = sum(probs.get(o, 0.0) for o in options)
+        if total > 0:
+            complete = all(o in probs for o in options)
+            return {o: probs.get(o, 0.0) / total for o in options}, complete
+    if isinstance(answer, Mapping) and "label" in answer:
+        label, p = str(answer["label"]), float(answer.get("confidence", 1.0))
+        rest = (1.0 - p) / (len(options) - 1) if len(options) > 1 else 0.0
+        return {o: (p if o == label else rest) for o in options}, False
+    if isinstance(answer, Mapping) and "value" in answer:
+        answer = answer["value"]
+    label = str(answer)
+    return {o: (1.0 if o == label else 0.0) for o in options}, False
+
+
+class GlinerEngine:
+    def __init__(self, model: str = DEFAULT_MODEL, *, extractor: Any = None) -> None:
+        self.name = "gliner"
+        self.model = model
+        self._extractor = extractor
+
+    def _get(self) -> Any:
+        if self._extractor is not None:
+            return self._extractor
+        # One load per process and model, even with concurrent first calls.
+        with _models_lock:
+            if self.model not in _models:
+                try:
+                    from gliner2 import AutoExtractor
+                except ImportError as err:
+                    raise GlinerError(
+                        "the gliner engine needs gliner2: pip install 'judgetap[gliner]'"
+                    ) from err
+                try:
+                    _models[self.model] = AutoExtractor.from_pretrained(self.model)
+                except Exception as err:
+                    raise GlinerError(f"could not load {self.model}: {err}") from err
+                while len(_models) > MAX_CACHED_MODELS:
+                    del _models[next(iter(_models))]  # oldest load first
+            else:
+                _models[self.model] = _models.pop(self.model)  # most recently used
+            self._extractor = _models[self.model]
+        return self._extractor
+
+    def decide(
+        self, questions: Sequence[Question], context: Context
+    ) -> Sequence[RawAnswer]:
+        ids = [f"q{i}" for i in range(len(questions))]
+        tasks = {i: _task(q) for i, q in zip(ids, questions, strict=True)}
+        extractor, text = self._get(), _text(context)
+        try:
+            try:
+                result = extractor.classify_text(text, tasks, include_confidence=True)
+            except TypeError:
+                result = extractor.classify_text(text, tasks)  # no confidence support
+        except Exception as err:
+            raise GlinerError(f"gliner2 classification failed: {err}") from err
+        try:
+            answers = []
+            for i, q in zip(ids, questions, strict=True):
+                dist, calibrated = _distribution(q, result[i])
+                answers.append(RawAnswer(dist, cost_usd=0.0, calibrated=calibrated))
+            return answers
+        except (KeyError, TypeError, ValueError) as err:
+            raise GlinerError(f"unexpected gliner2 result shape: {err!r}") from err
+
+    async def adecide(
+        self, questions: Sequence[Question], context: Context
+    ) -> Sequence[RawAnswer]:
+        return await asyncio.to_thread(self.decide, questions, context)

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/src/judgetap/evaluate.py

@@ -13,6 +13,7 @@ from __future__ import annotations
 
 import json
 import math
+import sys
 from collections.abc import Iterable, Sequence
 from dataclasses import asdict, dataclass, field
 from pathlib import Path
@@ -194,13 +195,35 @@ def main(argv: Sequence[str] | None = None) -> int:
     parser = argparse.ArgumentParser(
         prog="judgetap eval", description=__doc__.split("\n")[0]
     )
-    parser.add_argument("cases", help="JSONL file of labelled cases")
+    parser.add_argument("cases", nargs="?", help="JSONL file of labelled cases")
+    parser.add_argument(
+        "--suite", help="public suite instead of a file (ag_news, banking77)"
+    )
+    parser.add_argument(
+        "--limit", type=int, help="deterministic sample of N cases (fixed seed)"
+    )
     parser.add_argument(
         "--engines", required=True, help="comma-separated specs, e.g. jev,laya"
     )
     parser.add_argument("--json", action="store_true", help="JSON instead of Markdown")
     args = parser.parse_args(argv)
-    cases = load_cases(args.cases)
+    if (args.cases is None) == (args.suite is None):
+        parser.error("give exactly one of a cases file or --suite")
+    if args.suite is not None:
+        from judgetap.suites import suite_path
+
+        if not args.suite.strip():
+            parser.error("--suite needs a suite name (ag_news, banking77)")
+        try:
+            path: str | Path = suite_path(args.suite)
+        except JudgetapError as err:
+            print(f"judgetap eval: {err}", file=sys.stderr)
+            return 1
+    else:
+        path = args.cases
+    from judgetap.suites import sample
+
+    cases = sample(load_cases(path), args.limit)
     reports = [evaluate(cases, load(spec)) for spec in args.engines.split(",")]
     print(to_json(reports) if args.json else to_markdown(reports), end="")
     return 0

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/src/judgetap/guard/hook.py

@@ -167,7 +167,11 @@ def _engine():
     if not spec:
         return None
     from judgetap.engines import load
+    from judgetap.guard.install import in_process_error
 
+    if error := in_process_error(spec):
+        # Caught by the caller: rules only, failing closed.
+        raise RuntimeError(error)
     engine = load(spec)
     # The engine looked the key up once when built; reuse that, don't hit the
     # keychain a second time on every guarded action.
@@ -292,14 +296,30 @@ def run(
     record: bool = True,
     agent: str = "claude-code",
 ) -> int:
-    """Hook entry point; returns the exit code. Failures never block: they
-    allow the action and say so, since a crashing hook is ignored anyway."""
+    """Hook entry point; returns the exit code. Input the guard can't read
+    asks the user (it never saw the action, so it can't vouch for it); any
+    later failure allows the action and says so."""
     start = time.perf_counter()
     code, err_text = 0, ""
     try:
-        payload = normalise(json.load(stdin), agent)
-        action = action_from_hook(payload)
-        if action is None:
+        try:
+            payload = json.load(stdin)
+            if not isinstance(payload, dict):
+                raise TypeError(f"expected a JSON object, got {type(payload).__name__}")
+            payload = normalise(payload, agent)
+            action = action_from_hook(payload)
+        except Exception as err:  # noqa: BLE001 -- unreadable input fails closed
+            verdict = Verdict(
+                "ask",
+                "none",
+                "couldn't read the hook input",
+                error=f"{type(err).__name__}: {err}",
+            )
+            out, code, err_text = respond(verdict, agent)
+            action = payload = None
+        if payload is None:
+            pass
+        elif action is None:
             out = {"permission": "allow"} if agent == "cursor" else None
         else:
             verdict = _decide(action, start, payload.get("session_id"))

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/src/judgetap/guard/install.py

@@ -205,7 +205,24 @@ SPEC_PATTERN = re.compile(
 )  # [ ] for IPv6 hosts, ? for llm options
 
 
-KNOWN_ENGINES = frozenset({"jev", "llm", "laya", "julia", "agentjev", "typesafe"})
+KNOWN_ENGINES = frozenset(
+    {"jev", "llm", "laya", "julia", "agentjev", "typesafe", "gliner"}
+)
+
+
+# Engines that load a model into the calling process. The guard hook is a new
+# process per action, so these would reload the model on every guarded action.
+IN_PROCESS_ENGINES = frozenset({"gliner"})
+
+
+def in_process_error(spec: str) -> str | None:
+    name = spec.partition(":")[0].partition("@")[0]
+    if name in IN_PROCESS_ENGINES:
+        return (
+            f"{name} loads its model in-process, and the guard hook is a new process "
+            "per action; use a server engine for the guard"
+        )
+    return None
 
 
 def validate_engine(spec: str):
@@ -217,6 +234,8 @@ def validate_engine(spec: str):
         raise ValueError(
             f"unknown engine {spec!r}; known: {', '.join(sorted(KNOWN_ENGINES))}"
         )
+    if error := in_process_error(spec):
+        raise ValueError(error)
     from judgetap.engines import load
 
     try:

judgetap-0.2.1.dev44001/src/judgetap/suites.py

@@ -0,0 +1,177 @@
+"""Public eval suites, converted to judgetap's case format on demand.
+
+Nothing here is vendored: the first ``judgetap eval --suite <name>`` pages the
+split out of the Hugging Face datasets-server JSON API (plain HTTPS, stdlib
+only), converts each row to a ``choice`` case and caches the JSONL under
+``~/.judgetap/suites`` (override with ``JUDGETAP_SUITES_DIR``). Later runs read
+the cache. ``--limit N`` takes a deterministic sample: a fixed seed, so the same
+N cases every run and on every machine.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import random
+import urllib.parse
+import urllib.request
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+from judgetap.errors import JudgetapError
+
+ROWS_API = "https://datasets-server.huggingface.co/rows"
+PAGE = 100  # the rows API's maximum page size
+SEED = 88
+TIMEOUT = 30
+
+
+@dataclass(frozen=True)
+class Suite:
+    name: str
+    dataset: str  # Hugging Face dataset id
+    split: str
+    question: str
+    licence: str
+    source: str  # human-readable page for the dataset
+    config: str = "default"
+    text_field: str = "text"
+    label_field: str = "label"
+
+
+REGISTRY: dict[str, Suite] = {
+    s.name: s
+    for s in (
+        Suite(
+            name="ag_news",
+            dataset="fancyzhx/ag_news",
+            split="test",
+            question="Which topic is this news article about?",
+            licence="unknown; AG's corpus is provided for non-commercial research use",
+            source="https://huggingface.co/datasets/fancyzhx/ag_news",
+        ),
+        Suite(
+            name="banking77",
+            dataset="legacy-datasets/banking77",
+            split="test",
+            question="Which intent does this banking customer message express?",
+            licence="CC-BY-4.0",
+            source="https://huggingface.co/datasets/legacy-datasets/banking77",
+        ),
+    )
+}
+
+
+def cache_dir() -> Path:
+    override = os.environ.get("JUDGETAP_SUITES_DIR")
+    return Path(override) if override else Path.home() / ".judgetap" / "suites"
+
+
+def _get_json(url: str) -> dict[str, Any]:
+    req = urllib.request.Request(url, headers={"User-Agent": "judgetap-eval"})
+    with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
+        return json.load(resp)
+
+
+def _page_url(suite: Suite, offset: int) -> str:
+    query = urllib.parse.urlencode(
+        {
+            "dataset": suite.dataset,
+            "config": suite.config,
+            "split": suite.split,
+            "offset": offset,
+            "length": PAGE,
+        }
+    )
+    return f"{ROWS_API}?{query}"
+
+
+def _label_names(features: list[dict[str, Any]], field: str) -> list[str]:
+    for feature in features:
+        if feature.get("name") == field:
+            # The live rows API puts the ClassLabel under "type" (checked
+            # 2026-09-27); "feature" is accepted too, the key older docs name.
+            for key in ("type", "feature"):
+                spec = feature.get(key)
+                names = spec.get("names") if isinstance(spec, dict) else None
+                if isinstance(names, list) and names:
+                    return [str(n) for n in names]
+    raise JudgetapError(f"no class-label names for {field!r} in the dataset")
+
+
+def _to_case(suite: Suite, names: list[str], item: Any, index: int) -> dict[str, Any]:
+    try:
+        row = item["row"]
+        context = row[suite.text_field]
+        label = int(row[suite.label_field])
+        if not isinstance(context, str) or not 0 <= label < len(names):
+            raise ValueError(f"label {label} or text is out of range")
+    except (KeyError, TypeError, ValueError) as err:
+        raise JudgetapError(
+            f"suite {suite.name!r}: malformed row {index}: {err!r}"
+        ) from err
+    return {
+        "kind": "choice",
+        "question": suite.question,
+        "options": names,
+        "context": context,
+        "label": names[label],
+    }
+
+
+def download(suite: Suite) -> list[dict[str, Any]]:
+    """Every row of the split, as judgetap case dicts."""
+    cases: list[dict[str, Any]] = []
+    names: list[str] | None = None
+    offset, total = 0, None
+    while total is None or offset < total:
+        try:
+            page = _get_json(_page_url(suite, offset))
+        except (OSError, ValueError) as err:
+            raise JudgetapError(f"downloading suite {suite.name!r}: {err}") from err
+        if names is None:
+            names = _label_names(page.get("features", []), suite.label_field)
+        total = int(page.get("num_rows_total", 0))
+        rows = page.get("rows", [])
+        if not rows:
+            if offset < total:
+                raise JudgetapError(
+                    f"suite {suite.name!r}: empty page at offset {offset} "
+                    f"of {total} rows; refusing to cache a partial suite"
+                )
+            break
+        for item in rows:
+            cases.append(_to_case(suite, names, item, offset + len(cases)))
+        offset += len(rows)
+    if not cases:
+        raise JudgetapError(f"suite {suite.name!r} downloaded no rows")
+    return cases
+
+
+def suite_path(name: str) -> Path:
+    """The cached JSONL for ``name``, downloading it the first time."""
+    suite = REGISTRY.get(name)
+    if suite is None:
+        raise JudgetapError(
+            f"unknown suite {name!r}; available: {', '.join(sorted(REGISTRY))}"
+        )
+    path = cache_dir() / f"{suite.name}-{suite.split}.jsonl"
+    if path.exists():
+        return path
+    cases = download(suite)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_suffix(".jsonl.part")
+    tmp.write_text("".join(json.dumps(c) + "\n" for c in cases))
+    tmp.replace(path)  # a half-written download never looks cached
+    return path
+
+
+def sample(items: list[Any], limit: int | None) -> list[Any]:
+    """A fixed-seed sample of ``limit`` items, kept in their original order."""
+    if limit is None or limit >= len(items):
+        return items
+    if limit < 1:
+        raise JudgetapError("--limit must be at least 1")
+    picked = sorted(random.Random(SEED).sample(range(len(items)), limit))
+    return [items[i] for i in picked]

{judgetap-0.2.1.dev42001 → judgetap-0.2.1.dev44001}/tests/test_cascade.py

@@ -229,3 +229,27 @@ def test_from_config_scalar_cascade_is_a_config_error(tmp_path):
     cfg.write_text('cascade = "jev"\n')
     with pytest.raises(sj.JudgetapError):
         from_config(cfg)
+
+
+class Uncalibrated(StaticEngine):
+    def decide(self, questions, context):
+        self.calls.append((tuple(questions), context))
+        return [
+            sj.RawAnswer({"yes": 0.0, "no": 1.0}, calibrated=False) for _ in questions
+        ]
+
+
+def test_uncalibrated_answer_escalates_whatever_its_p():
+    bare, strong = Uncalibrated(lambda q, c: {}, name="bare"), eng("strong", 0.9)
+    d = sj.yesno("q", engine=sj.Cascade([bare, strong]))
+    assert (d.engine, d.meta["hops"]) == ("strong", ["bare", "strong"])
+
+
+def test_last_engine_uncalibrated_answer_is_judged_on_p():
+    d = sj.yesno(
+        "q",
+        engine=sj.Cascade(
+            [eng("cheap", 0.6), Uncalibrated(lambda q, c: {}, name="bare")]
+        ),
+    )
+    assert d.engine == "bare" and d.calibrated is False