@unotest/judge 0.24.0 → 0.26.0

package/CHANGELOG.md

@@ -1,5 +1,127 @@
 # @unotest/judge
 
+## [0.26.0] - 2026-08-30
+
+### Patch Changes
+
+- Updated dependencies [3eede7c]
+- Updated dependencies [6d1c612]
+- Updated dependencies [dbfa36f]
+- Updated dependencies [c43aa00]
+- Updated dependencies [8ec4177]
+  - @unotest/protocol@0.26.0
+
+## [0.25.0] - 2026-08-29
+
+### Minor Changes
+
+- 70f2e11: Judge: provider edge cases, a startup preflight, and logs.
+
+  **Vertex `GOOGLE_CLOUD_LOCATION=global` could not work.** The endpoint was
+  built as `https://<location>-aiplatform.googleapis.com`, so `global` asked
+  `global-aiplatform.googleapis.com` — not a Vertex host. Google's edge
+  answered 404 with an HTML page, which arrived as "model not found". The
+  global endpoint has no region prefix; it is now special-cased. `global` is
+  the default in Google's own docs, so this was the first thing a Vertex user
+  hit. A model id containing a slash
+  (`publishers/anthropic/models/claude-haiku-4-5`) is now passed through
+  instead of being forced under `publishers/google`, and a pasted
+  `models/gemini-2.5-flash` no longer doubles the prefix on the Gemini API.
+
+  **Backend replies are classified instead of lumped together.**
+
+  - `401`/`403` → `config` with the variable to fix (`OPENAI_API_KEY`, ADC,
+    an expired `UNOTEST_JUDGE_ACCESS_TOKEN`), no longer indistinguishable
+    from a model fault.
+  - `429` and `5xx` → retried transparently with a short backoff that
+    honours `Retry-After`. A single rate limit used to kill a whole scenario.
+    The retry policy is a decorator over `JudgeProvider`, not a loop inside
+    the HTTP helper, so it covers every provider — `claude` spawns a process
+    and therefore had no re-send at all. A `claude` usage limit is
+    deliberately still reported rather than retried: the window is seconds,
+    the limit is not.
+  - A non-JSON error body says the endpoint is wrong rather than quoting an
+    HTML page back as the backend's opinion of the model. The body is kept
+    for `debug` logging.
+  - OpenAI `finish_reason: length` and Anthropic `stop_reason: max_tokens`
+    are named as token-limit truncation instead of surfacing as "did not
+    reply with the requested JSON verdict".
+  - The verdict parser accepts `"PASS"` and a JSON object embedded in prose —
+    providers without a JSON mode do both.
+
+  **`UNOTEST_JUDGE_VOTE=N` no longer fails on one bad ballot.** The N calls
+  ran under `Promise.all`, so voting — the reliability feature — tripled the
+  chance that one transient error killed the request. Errored ballots are now
+  dropped as long as a majority of the N calls still voted; a tie among the
+  survivors re-raises the failure rather than picking a side.
+
+  **The service checks its credentials before it says "listening".** Each
+  provider exposes a free probe (an ADC token exchange, `GET /models`,
+  `claude --version`) — no model call, no cost. A failing probe prints the
+  reason and exits non-zero (`UNOTEST_JUDGE_SKIP_PREFLIGHT=1` waives it), and
+  `npx @unotest/judge --check` runs just the probe for start scripts.
+  `GET /health` reports the same thing instead of an unconditional
+  `{"ok":true}`: `503` + `{"ok":false,"code":"config","error":"..."}`, cached
+  for a few seconds. The waiver covers the startup gate only — a service
+  started under it still answers `/health` from a real probe, and `--check`
+  still answers honestly, because the operator who reaches for the flag is
+  precisely the one whose credentials are dead. A static
+  `UNOTEST_JUDGE_ACCESS_TOKEN` is validated too, against Google's free
+  `tokeninfo`: those tokens live about an hour and refresh themselves never,
+  so "valid at startup" is a different question from "valid now".
+
+  ADC that expires on a daily reauth policy is a normal morning state, not a
+  one-off misconfiguration — it should not cost a five-minute run to
+  discover.
+
+  **New: `UNOTEST_JUDGE_LOG_LEVEL`** (falls back to the `UNOTEST_LOG_LEVEL`
+  `@unotest/web` already reads). At the default `info`, one line per judged
+  request — verdict, attempts, model, latency — with errors on stderr; a run
+  of 21 assertions with `VOTE=3` is 63 billed model calls that previously
+  left no trace. At `debug`, per call: the effective prompt (rubric
+  **including** any `UNOTEST_JUDGE_PREAMBLE`), each ballot with its
+  reasoning, and the raw provider reply. **The judged text is application
+  content** — debug writes it to stdout verbatim, which is why it is opt-in
+  and never the default.
+
+  An oversized request body now returns `413` with a reason; the socket used
+  to be destroyed while the client was still writing, so the client saw
+  `socket hang up`.
+
+  **`@unotest/web`:** the `judge:verdict` run artifact records `preamble`
+  beside `rubric` when one is set. The model saw the concatenation, but only
+  the split says which half came from the scenario and which from the
+  environment — without it, "a red step is explainable without a re-run" held
+  only for whoever also knew the env file. A judge request that outlives
+  `UNOTEST_JUDGE_TIMEOUT_MS` now says so, and names the service-side
+  `UNOTEST_JUDGE_CALL_TIMEOUT_MS` it may be racing.
+
+  The Claude CLI provider also stopped leaking a timer: the per-call budget
+  is now our own timer, cleared on every exit path. `child_process.spawn`'s
+  `timeout` option does not clear itself when the spawn fails, so a missing
+  binary held the event loop for the whole budget — 15s for a `--version`
+  probe, 120s for a verdict.
+
+  **Breaking (0.x minor):** `@unotest/judge` no longer exports `ExecFileFn` /
+  `ExecFileResult`; the Claude CLI provider takes `runImpl: RunProcess`
+  instead of `execImpl`, because the prompt moved from `argv` to stdin — a
+  long judged text used to overflow `ARG_MAX` and fail as a bare `E2BIG`.
+
+### Patch Changes
+
+- Judge: the last `JUDGE_*` names in user-facing strings.
+
+  0.24.0 renamed every service variable to `UNOTEST_JUDGE_*`, but three
+  messages kept telling people the old name: the service's `401`, the fake
+  provider's "switch to a real provider" hint, and the Claude CLI's
+  "binary not found" error. They now name the variable that actually
+  exists. Also landed after that release: `google-auth-library` was
+  declared but missing from the lockfile.
+
+- Updated dependencies [70f2e11]
+- Updated dependencies [adb9d72]
+  - @unotest/protocol@0.25.0
+
 ## [0.24.0] - 2026-08-26
 
 ### Minor Changes

package/README.md

@@ -17,10 +17,22 @@ output.
 ```
 UNOTEST_JUDGE_PROVIDER=vertex \
 GOOGLE_CLOUD_PROJECT=my-project \
-GOOGLE_CLOUD_LOCATION=europe-west1 \
+GOOGLE_CLOUD_LOCATION=us-central1 \
 npx @unotest/judge
 ```
 
+On start the service checks the provider's credentials (an ADC token
+exchange, a free `GET /models`, a `claude --version`) and refuses to
+listen if they are dead — a judge that cannot judge should not report
+itself as ready. `npx @unotest/judge --check` runs only that check and
+exits `0`/`1`, for start scripts and CI.
+
+`UNOTEST_JUDGE_SKIP_PREFLIGHT=1` waives the **startup gate** — the service
+warns and listens anyway. It does not waive the truth: `/health` keeps
+probing and keeps reporting the failure, and `--check` still answers
+honestly. A green light on a judge that cannot judge is the fault this
+whole check exists to remove.
+
 The service listens on `127.0.0.1:8790` by default. Point `@unotest/web`
 at it with `UNOTEST_JUDGE_MODE=remote` + `UNOTEST_JUDGE_URL=http://127.0.0.1:8790`,
 or skip the HTTP hop entirely with `UNOTEST_JUDGE_MODE=local` (in-process,
@@ -35,13 +47,28 @@ unotest through `npx`, with nothing installed locally, can only use
   workload identity, or `GOOGLE_APPLICATION_CREDENTIALS`). Requires the
   `google-auth-library` (a dependency of this package, loaded only when
   this provider runs) — or a static `UNOTEST_JUDGE_ACCESS_TOKEN` from
-  `gcloud auth print-access-token`. Temperature is pinned to 0.
+  `gcloud auth print-access-token` (those live about an hour and are not
+  refreshed for you — the preflight validates one against Google's free
+  `tokeninfo` endpoint, so an expired token is caught at startup rather
+  than mid-run). Temperature is pinned to 0.
+
+  **Location and model are chosen together.** `GOOGLE_CLOUD_LOCATION` is
+  either `global` or a region (`us-central1`), and model availability
+  differs between them — `gemini-2.5-flash` is regional, newer flash-lite
+  models may be global-only. A model id containing a slash is used as-is
+  (`publishers/anthropic/models/claude-haiku-4-5`), which is how Vertex
+  addresses third-party models.
 - **`claude`** — the local Claude Code CLI (`claude -p`), spawned per
   judgement without shell interpretation. Auth comes from your Claude Code
   session, so a subscription works with no API key — this is the provider
   to reach for when no cloud credentials are at hand. `UNOTEST_JUDGE_MODEL`
   is passed as `--model` (aliases like `sonnet` work); unset uses the CLI's
-  default.
+  default. **Budget for latency, not for tokens:** every verdict spawns the
+  CLI and loads a session, which measured ~6s per call here against ~1s for
+  an HTTP provider — 20 assertions is a couple of minutes. `UNOTEST_JUDGE_VOTE`
+  costs almost nothing extra (ballots run concurrently: 3 votes measured
+  6.8s), and the per-call budget defaults to 120s for this provider for the
+  same reason. The prompt goes in on stdin, so a large judged text is fine.
 - **`gemini`** — the Gemini API with `GEMINI_API_KEY`. Same wire format as
   `vertex`, temperature pinned to 0.
 - **`openai`** — OpenAI chat completions with `OPENAI_API_KEY`
@@ -53,11 +80,35 @@ unotest through `npx`, with nothing installed locally, can only use
 - **`fake`** — deterministic, no model. For CI and smoke-testing the
   wiring. The rubric is a micro-grammar, one constraint per line:
   `must contain: <substring>` / `must not contain: <substring>`
-  (case-insensitive). Anything else fails loudly.
+  (case-insensitive). Anything else fails loudly. Note that a
+  `UNOTEST_JUDGE_PREAMBLE` set on the client side arrives glued to the
+  rubric, so constraint lines in a preamble apply to every judgement.
 
 All model providers are raw HTTP or a local process — this package ships
 zero provider SDKs.
 
+## Latency
+
+A verdict costs wall-clock time in the scenario that waits for it, so the
+choice of provider is a choice of pace:
+
+| Provider | Per verdict | Why |
+| --- | --- | --- |
+| `vertex` / `gemini` / `openai` / `anthropic` | one HTTP round-trip (~1s for a flash-class model) | a single JSON POST |
+| `claude` | ~6s, measured on a local CLI | every call spawns the binary and loads a session |
+| `fake` | none | no model behind it |
+
+`UNOTEST_JUDGE_VOTE=N` costs almost nothing on top: the N ballots run
+concurrently, so three votes take about as long as one call (measured 6.8s
+for `claude`). `UNOTEST_JUDGE_RETRIES`, by contrast, is sequential — it
+only spends a call when a verdict came back `fail`.
+
+Budgets follow the same shape: `UNOTEST_JUDGE_CALL_TIMEOUT_MS` defaults to
+30s, and to 120s for `claude`. On the client side `@unotest/web` has its
+own whole-request budget (`UNOTEST_JUDGE_TIMEOUT_MS`, 60s) — with `claude`
+plus a slow model the client can give up before the service does, and it
+now says so when that happens.
+
 ## Env
 
 | Variable | Meaning | Default |
@@ -68,7 +119,7 @@ zero provider SDKs.
 | `UNOTEST_JUDGE_VOTE` | odd N; N independent calls, the majority decides. Replaces retries rather than stacking with them | `1` (off) |
 | `UNOTEST_JUDGE_CALL_TIMEOUT_MS` | budget for ONE provider call, ms | `30000` (`120000` for claude) |
 | `GOOGLE_CLOUD_PROJECT` | vertex: ADC project | required for vertex |
-| `GOOGLE_CLOUD_LOCATION` | vertex: ADC location | required for vertex |
+| `GOOGLE_CLOUD_LOCATION` | vertex: `global` or a region (`us-central1`) | required for vertex |
 | `UNOTEST_JUDGE_ACCESS_TOKEN` | vertex: static bearer override (skips ADC) | — |
 | `GEMINI_API_KEY` | gemini: API key | required for gemini |
 | `OPENAI_API_KEY` | openai: API key | required for openai |
@@ -76,6 +127,32 @@ zero provider SDKs.
 | `UNOTEST_JUDGE_CLAUDE_BIN` | claude: binary override | `claude` |
 | `UNOTEST_JUDGE_HOST` / `UNOTEST_JUDGE_PORT` | bind address | `127.0.0.1` / `8790` |
 | `UNOTEST_JUDGE_TOKEN` | bearer token required on every `/judge` call. `@unotest/web` reads the SAME name to send it | off |
+| `UNOTEST_JUDGE_LOG_LEVEL` | `silent` \| `error` \| `warn` \| `info` \| `debug`; falls back to `UNOTEST_LOG_LEVEL` | `info` |
+| `UNOTEST_JUDGE_SKIP_PREFLIGHT` | `1` starts even when the credential check fails (`/health` still reports it) | off |
+
+## Logging
+
+At the default `info` level the service prints one line per judged
+request — verdict, attempts spent, model, latency:
+
+```
+POST /judge  pass  attempts 3  gemini-2.5-flash  1284ms
+POST /judge  ERROR config  Application Default Credentials need re-authentication (invalid_rapt) — run `gcloud auth application-default login`
+```
+
+Errors go to stderr. A run with 21 assertions and `UNOTEST_JUDGE_VOTE=3`
+is 63 billed model calls; this is the only place their cost and latency
+are visible.
+
+`UNOTEST_JUDGE_LOG_LEVEL=debug` additionally prints, per call, the
+effective prompt (rubric **including** any `UNOTEST_JUDGE_PREAMBLE` the
+client prepended), each ballot with its reasoning, and the provider's raw
+reply.
+
+> **The judged text is application content** — support answers, user
+> data, whatever the scenario put under the assertion. Debug logging
+> writes it to stdout verbatim. It is opt-in for that reason; do not
+> enable it on a shared runner without deciding that is acceptable.
 
 ## HTTP API
 
@@ -83,11 +160,31 @@ zero provider SDKs.
   `JudgeVerdict` (above). Errors: `{"error": "...", "code": "unauthorized" | "bad-request" | "provider-error" | "config" | "internal"}`.
   `config` means the operator must fix this service's setup (bad env,
   expired ADC) and the message says what to run.
-- `GET /health` — `{"ok": true}`.
+- `GET /health` — `{"ok": true}` when the provider's credentials work,
+  otherwise `503` + `{"ok": false, "code": "config", "error": "..."}` with
+  the same message and code a verdict call would have failed with. The
+  check is cached for a few seconds, so polling is cheap, and it never
+  calls the model.
 
 Wire types are shared through `@unotest/protocol` (`JudgeRequest`,
-`JudgeVerdict`, `JUDGE_ROUTES`), so the service and `@unotest/web`'s
-client cannot drift.
+`JudgeVerdict`, `JudgeHealthResponse`, `JUDGE_ROUTES`), so the service and
+`@unotest/web`'s client cannot drift.
+
+Faults that nobody chose are re-sent: rate limits (`429`), backend `5xx`,
+a dropped connection, a spawn the OS could not grant. Up to three attempts,
+500ms then 1500ms apart, `Retry-After` honoured over that curve when the
+backend names one. The policy sits on the provider contract, so it covers
+**every** provider — including `claude`, which speaks to a process rather
+than to HTTP.
+
+What is never re-sent: a rejected credential (`401`/`403`) comes back as
+`config` with the variable to fix, an unknown model is not a hiccup, and a
+`claude` usage limit is reported rather than retried — the retry window is
+seconds while a subscription limit lasts far longer, so re-spawning a ~6s
+CLI would only make the same failure slower.
+
+With `UNOTEST_JUDGE_VOTE=N`, a ballot that still errors is dropped rather
+than failing the request, as long as a majority of the N calls voted.
 
 ## License