@unotest/judge 0.23.0

package/CHANGELOG.md

@@ -0,0 +1,51 @@
+# @unotest/judge
+
+## [0.23.0] - 2026-08-26
+
+### Minor Changes
+
+- LLM-judge assertion for free-form text: `assertJudge(locator, rubric)`.
+
+  - New package `@unotest/judge` — the judge service (`npx @unotest/judge`):
+    judges text against a natural-language rubric and returns a structured
+    `{verdict, reasoning, model, attempts}`. Providers: `vertex` (Google
+    Vertex AI via ADC, no API keys; temperature pinned to 0) and `fake`
+    (deterministic, CI-safe). Non-determinism policy lives in the service:
+    `JUDGE_RETRIES` re-asks on `fail`, first `pass` wins.
+  - `@unotest/web`: new DSL assertion `assertJudge(locator, rubric)` — the
+    element's rendered text + the rubric go to the judge; a `fail` verdict
+    fails the step with the judge's reasoning. Wired via env so the config
+    sits in the `--env` overlay: `UNOTEST_JUDGE_MODE=off|local|remote`,
+    `UNOTEST_JUDGE_URL`, `UNOTEST_JUDGE_TOKEN`, `UNOTEST_JUDGE_TIMEOUT_MS`.
+    Every verdict (pass and fail) is recorded into the run's `steps.jsonl`.
+    Also documented: `shell()` runs with cwd = the project root
+    (`sandbox.shellCwd` to override) and returns `{stdout, stderr, code}`.
+  - `@unotest/protocol`: judge wire types (`JudgeRequest`, `JudgeVerdict`,
+    `JUDGE_ROUTES`) and the new `judge:verdict` run-artifact event.
+
+- Judge provider matrix: four new `JUDGE_PROVIDER` backends.
+
+  - `claude` — spawns the local Claude Code CLI (`claude -p`, no shell
+    interpretation): auth comes from the Claude Code session, so a
+    subscription works with no API key. `JUDGE_MODEL` is passed as
+    `--model` (aliases like `sonnet` work); `JUDGE_CLAUDE_BIN` overrides
+    the binary; default `JUDGE_TIMEOUT_MS` is 120000 for this provider.
+  - `gemini` — the Gemini API with `GEMINI_API_KEY` (same wire format as
+    `vertex`, temperature pinned to 0; default model `gemini-2.5-flash`).
+  - `openai` — OpenAI chat completions with `OPENAI_API_KEY`
+    (`response_format: json_object`; default model `gpt-5-mini`).
+  - `anthropic` — the Anthropic API (`/v1/messages`) with
+    `ANTHROPIC_API_KEY` (default model `claude-haiku-4-5`).
+
+  `openai` and `anthropic` deliberately send no sampling params — current
+  reasoning models reject `temperature`; determinism relies on the strict
+  JSON verdict prompt plus the `JUDGE_RETRIES` policy. Still zero provider
+  SDKs: everything is raw HTTP or a local process.
+
+  `@unotest/web`: docs only — provider matrix in the `assertJudge`
+  reference and the `.env.example` judge block.
+
+### Patch Changes
+
+- Updated dependencies
+  - @unotest/protocol@0.23.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ivan Volkov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,86 @@
+# @unotest/judge
+
+LLM-judge service for the [@unotest](https://www.npmjs.com/package/@unotest/web)
+ecosystem. It judges free-form text (a chat reply, generated content)
+against a natural-language rubric and returns a structured verdict:
+
+```json
+{ "verdict": "fail", "reasoning": "the reply promises escalation", "model": "gemini-2.5-flash", "attempts": 2 }
+```
+
+It backs the `assertJudge(locator, rubric)` assertion of `@unotest/web` —
+semantic checks where substring/regex assertions are too brittle for LLM
+output.
+
+## Run
+
+```
+JUDGE_PROVIDER=vertex \
+GOOGLE_CLOUD_PROJECT=my-project \
+GOOGLE_CLOUD_LOCATION=europe-west1 \
+npx @unotest/judge
+```
+
+The service listens on `127.0.0.1:8790` by default. Point `@unotest/web`
+at it with `UNOTEST_JUDGE_MODE=remote` + `UNOTEST_JUDGE_URL=http://127.0.0.1:8790`,
+or skip the HTTP hop entirely with `UNOTEST_JUDGE_MODE=local` (in-process,
+requires `@unotest/judge` installed in the project).
+
+## Providers
+
+- **`vertex`** — Google Vertex AI via Application Default Credentials.
+  No API keys: auth comes from ADC (`gcloud auth application-default login`,
+  workload identity, or `GOOGLE_APPLICATION_CREDENTIALS`). Requires the
+  optional peer `google-auth-library` (or a static `JUDGE_ACCESS_TOKEN`).
+  Temperature is pinned to 0.
+- **`claude`** — the local Claude Code CLI (`claude -p`), spawned per
+  judgement without shell interpretation. Auth comes from your Claude Code
+  session, so a subscription works with no API key. `JUDGE_MODEL` is passed
+  as `--model` (aliases like `sonnet` work); unset uses the CLI's default.
+- **`gemini`** — the Gemini API with `GEMINI_API_KEY`. Same wire format as
+  `vertex`, temperature pinned to 0.
+- **`openai`** — OpenAI chat completions with `OPENAI_API_KEY`
+  (`response_format: json_object`). No sampling params: reasoning models
+  reject `temperature`, so determinism relies on the strict verdict prompt
+  plus the retry policy.
+- **`anthropic`** — the Anthropic API (`/v1/messages`) with
+  `ANTHROPIC_API_KEY`. No sampling params, same reasoning as `openai`.
+- **`fake`** — deterministic, no model. For CI and smoke-testing the
+  wiring. The rubric is a micro-grammar, one constraint per line:
+  `must contain: <substring>` / `must not contain: <substring>`
+  (case-insensitive). Anything else fails loudly.
+
+All model providers are raw HTTP or a local process — this package ships
+zero provider SDKs.
+
+## Env
+
+| Variable | Meaning | Default |
+| --- | --- | --- |
+| `JUDGE_PROVIDER` | `fake` \| `vertex` \| `claude` \| `gemini` \| `openai` \| `anthropic` | required |
+| `JUDGE_MODEL` | model id | vertex/gemini `gemini-2.5-flash`, openai `gpt-5-mini`, anthropic `claude-haiku-4-5`, claude: CLI default |
+| `JUDGE_RETRIES` | extra provider calls on `fail`; first `pass` wins | `1` |
+| `JUDGE_TIMEOUT_MS` | per-call budget, ms | `30000` (`120000` for claude) |
+| `GOOGLE_CLOUD_PROJECT` | vertex: ADC project | required for vertex |
+| `GOOGLE_CLOUD_LOCATION` | vertex: ADC location | required for vertex |
+| `JUDGE_ACCESS_TOKEN` | vertex: static bearer override (skips ADC) | — |
+| `GEMINI_API_KEY` | gemini: API key | required for gemini |
+| `OPENAI_API_KEY` | openai: API key | required for openai |
+| `ANTHROPIC_API_KEY` | anthropic: API key | required for anthropic |
+| `JUDGE_CLAUDE_BIN` | claude: binary override | `claude` |
+| `JUDGE_HOST` / `JUDGE_PORT` | bind address | `127.0.0.1` / `8790` |
+| `JUDGE_TOKEN` | bearer token required on every `/judge` call | off |
+
+## HTTP API
+
+- `POST /judge` — body `{"rubric": "...", "text": "..."}` → a
+  `JudgeVerdict` (above). Errors: `{"error": "...", "code": "unauthorized" | "bad-request" | "provider-error" | "internal"}`.
+- `GET /health` — `{"ok": true}`.
+
+Wire types are shared through `@unotest/protocol` (`JudgeRequest`,
+`JudgeVerdict`, `JUDGE_ROUTES`), so the service and `@unotest/web`'s
+client cannot drift.
+
+## License
+
+MIT © Ivan Volkov

package/bin/unotest-judge.js

@@ -0,0 +1,11 @@
+#!/usr/bin/env node
+import { main } from "../dist/cli.js";
+
+main().catch((e) => {
+  if (process.env.UNOTEST_DEBUG === "1") {
+    console.error(e);
+  } else {
+    console.error(`✗ ${e instanceof Error ? e.message : String(e)}`);
+  }
+  process.exit(1);
+});

package/dist/chunk-4VUHPR26.js

@@ -0,0 +1,660 @@
+// src/errors.ts
+var JudgeError = class extends Error {
+  constructor(message, context = {}) {
+    super(message);
+    this.context = context;
+    this.name = new.target.name;
+  }
+  context;
+};
+var JudgeConfigError = class extends JudgeError {
+};
+var JudgeProviderError = class extends JudgeError {
+};
+
+// src/providers/fake.ts
+var FAKE_MODEL_ID = "fake";
+var CONSTRAINT_RE = /^must(?<not>\s+not)?\s+contain:\s*(?<needle>.+)$/i;
+function parseFakeRubric(rubric) {
+  const constraints = [];
+  for (const line of rubric.split(/\r?\n/)) {
+    const m = CONSTRAINT_RE.exec(line.trim());
+    if (!m?.groups?.needle) continue;
+    constraints.push({
+      kind: m.groups.not ? "not-contains" : "contains",
+      needle: m.groups.needle.trim()
+    });
+  }
+  return constraints;
+}
+var FakeJudgeProvider = class {
+  async judgeOnce(request) {
+    const constraints = parseFakeRubric(request.rubric);
+    if (constraints.length === 0) {
+      return {
+        pass: false,
+        reasoning: 'fake provider: rubric has no parseable constraints \u2014 use lines like "must contain: <substring>" / "must not contain: <substring>", or switch to a real provider (JUDGE_PROVIDER=vertex|claude|gemini|openai|anthropic)',
+        model: FAKE_MODEL_ID
+      };
+    }
+    const text = request.text.toLowerCase();
+    const violations = [];
+    for (const c of constraints) {
+      const hit = text.includes(c.needle.toLowerCase());
+      if (c.kind === "contains" && !hit) violations.push(`missing required "${c.needle}"`);
+      if (c.kind === "not-contains" && hit) violations.push(`contains forbidden "${c.needle}"`);
+    }
+    if (violations.length > 0) {
+      return { pass: false, reasoning: violations.join("; "), model: FAKE_MODEL_ID };
+    }
+    return {
+      pass: true,
+      reasoning: `all ${constraints.length} constraint(s) satisfied`,
+      model: FAKE_MODEL_ID
+    };
+  }
+};
+
+// src/verdict.ts
+var VERDICT_PROMPT = (rubric, text) => `You are a strict test judge. Evaluate the TEXT against the RUBRIC.
+Reply with ONLY a raw JSON object, no markdown code fences: {"verdict": "pass" | "fail", "reasoning": "<one short sentence>"}.
+RUBRIC:
+${rubric}
+
+TEXT:
+${text}`;
+var FENCED_JSON = /^```(?:json)?\s*\n([\s\S]*?)\n\s*```$/;
+function parseVerdictReply(text, model) {
+  const trimmed = text.trim();
+  const raw = FENCED_JSON.exec(trimmed)?.[1] ?? trimmed;
+  let parsed;
+  try {
+    parsed = JSON.parse(raw);
+  } catch {
+    throw new JudgeProviderError(
+      `model "${model}" did not reply with the requested JSON verdict: ${text.slice(0, 200)}`,
+      { model }
+    );
+  }
+  const { verdict, reasoning } = parsed;
+  if (verdict !== "pass" && verdict !== "fail") {
+    throw new JudgeProviderError(
+      `model "${model}" replied with an unknown verdict ${JSON.stringify(verdict)} (expected "pass"/"fail")`,
+      { model }
+    );
+  }
+  return {
+    pass: verdict === "pass",
+    reasoning: typeof reasoning === "string" && reasoning.length > 0 ? reasoning : "(no reasoning)"
+  };
+}
+
+// src/providers/generate-content.ts
+function generateContentBody(request) {
+  return {
+    contents: [
+      { role: "user", parts: [{ text: VERDICT_PROMPT(request.rubric, request.text) }] }
+    ],
+    generationConfig: {
+      temperature: 0,
+      responseMimeType: "application/json"
+    }
+  };
+}
+function extractCandidateText(body, model, backend) {
+  let parsed;
+  try {
+    parsed = JSON.parse(body);
+  } catch {
+    throw new JudgeProviderError(`${backend} reply is not JSON: ${body.slice(0, 200)}`, { model });
+  }
+  const text = parsed.candidates?.[0]?.content?.parts?.[0]?.text;
+  if (typeof text !== "string" || text.length === 0) {
+    throw new JudgeProviderError(
+      `${backend} reply carries no candidate text (blocked or empty): ${body.slice(0, 300)}`,
+      { model }
+    );
+  }
+  return text;
+}
+
+// src/providers/http-json.ts
+async function postJson(call) {
+  const { url, headers, body, timeoutMs, backend, model, fetchImpl } = call;
+  let res;
+  try {
+    res = await fetchImpl(url, {
+      method: "POST",
+      headers: { "content-type": "application/json", ...headers },
+      body: JSON.stringify(body),
+      signal: AbortSignal.timeout(timeoutMs)
+    });
+  } catch (e) {
+    throw new JudgeProviderError(
+      `cannot reach ${backend} at ${url} (${e instanceof Error ? e.message : String(e)})`,
+      { url }
+    );
+  }
+  const text = await res.text();
+  if (!res.ok) {
+    throw new JudgeProviderError(
+      `${backend} replied ${res.status} for model "${model}": ${text.slice(0, 500)}`,
+      { status: res.status, model }
+    );
+  }
+  return text;
+}
+
+// src/providers/vertex.ts
+var VertexJudgeProvider = class {
+  constructor(opts) {
+    this.opts = opts;
+    this.fetchImpl = opts.fetchImpl ?? fetch;
+  }
+  opts;
+  fetchImpl;
+  tokenSource;
+  async judgeOnce(request) {
+    const { project, location, model, timeoutMs } = this.opts;
+    const url = `https://${location}-aiplatform.googleapis.com/v1/projects/${project}/locations/${location}/publishers/google/models/${model}:generateContent`;
+    const token = await this.accessToken();
+    const body = await postJson({
+      url,
+      headers: { authorization: `Bearer ${token}` },
+      body: generateContentBody(request),
+      timeoutMs,
+      backend: "Vertex AI",
+      model,
+      fetchImpl: this.fetchImpl
+    });
+    return { ...parseVerdictReply(extractCandidateText(body, model, "Vertex AI"), model), model };
+  }
+  async accessToken() {
+    if (this.opts.accessToken) return this.opts.accessToken;
+    this.tokenSource ??= await buildAdcTokenSource();
+    return this.tokenSource();
+  }
+};
+async function buildAdcTokenSource() {
+  const specifier = "google-auth-library";
+  let mod;
+  try {
+    mod = await import(specifier);
+  } catch {
+    throw new JudgeConfigError(
+      'Vertex provider needs Application Default Credentials \u2014 install the optional peer "google-auth-library" (`npm i google-auth-library`), or pass a token via JUDGE_ACCESS_TOKEN'
+    );
+  }
+  const auth = new mod.GoogleAuth({ scopes: ["https://www.googleapis.com/auth/cloud-platform"] });
+  return async () => {
+    const token = await auth.getAccessToken();
+    if (!token) {
+      throw new JudgeConfigError(
+        "ADC produced no access token \u2014 run `gcloud auth application-default login` or set GOOGLE_APPLICATION_CREDENTIALS"
+      );
+    }
+    return token;
+  };
+}
+
+// src/providers/gemini.ts
+var GeminiJudgeProvider = class {
+  constructor(opts) {
+    this.opts = opts;
+    this.fetchImpl = opts.fetchImpl ?? fetch;
+  }
+  opts;
+  fetchImpl;
+  async judgeOnce(request) {
+    const { apiKey, model, timeoutMs } = this.opts;
+    const url = `https://generativelanguage.googleapis.com/v1beta/models/${model}:generateContent`;
+    const body = await postJson({
+      url,
+      headers: { "x-goog-api-key": apiKey },
+      body: generateContentBody(request),
+      timeoutMs,
+      backend: "Gemini API",
+      model,
+      fetchImpl: this.fetchImpl
+    });
+    return { ...parseVerdictReply(extractCandidateText(body, model, "Gemini API"), model), model };
+  }
+};
+
+// src/providers/openai.ts
+var OpenAiJudgeProvider = class {
+  constructor(opts) {
+    this.opts = opts;
+    this.fetchImpl = opts.fetchImpl ?? fetch;
+  }
+  opts;
+  fetchImpl;
+  async judgeOnce(request) {
+    const { apiKey, model, timeoutMs } = this.opts;
+    const body = await postJson({
+      url: "https://api.openai.com/v1/chat/completions",
+      headers: { authorization: `Bearer ${apiKey}` },
+      body: {
+        model,
+        messages: [{ role: "user", content: VERDICT_PROMPT(request.rubric, request.text) }],
+        response_format: { type: "json_object" }
+      },
+      timeoutMs,
+      backend: "OpenAI",
+      model,
+      fetchImpl: this.fetchImpl
+    });
+    return { ...parseVerdictReply(extractMessageContent(body, model), model), model };
+  }
+};
+function extractMessageContent(body, model) {
+  let parsed;
+  try {
+    parsed = JSON.parse(body);
+  } catch {
+    throw new JudgeProviderError(`OpenAI reply is not JSON: ${body.slice(0, 200)}`, { model });
+  }
+  const content = parsed.choices?.[0]?.message?.content;
+  if (typeof content !== "string" || content.length === 0) {
+    throw new JudgeProviderError(
+      `OpenAI reply carries no message content: ${body.slice(0, 300)}`,
+      { model }
+    );
+  }
+  return content;
+}
+
+// src/providers/anthropic.ts
+var ANTHROPIC_VERSION = "2023-06-01";
+var MAX_TOKENS = 1024;
+var AnthropicJudgeProvider = class {
+  constructor(opts) {
+    this.opts = opts;
+    this.fetchImpl = opts.fetchImpl ?? fetch;
+  }
+  opts;
+  fetchImpl;
+  async judgeOnce(request) {
+    const { apiKey, model, timeoutMs } = this.opts;
+    const body = await postJson({
+      url: "https://api.anthropic.com/v1/messages",
+      headers: { "x-api-key": apiKey, "anthropic-version": ANTHROPIC_VERSION },
+      body: {
+        model,
+        max_tokens: MAX_TOKENS,
+        messages: [{ role: "user", content: VERDICT_PROMPT(request.rubric, request.text) }]
+      },
+      timeoutMs,
+      backend: "Anthropic API",
+      model,
+      fetchImpl: this.fetchImpl
+    });
+    return { ...parseVerdictReply(extractTextBlock(body, model), model), model };
+  }
+};
+function extractTextBlock(body, model) {
+  let parsed;
+  try {
+    parsed = JSON.parse(body);
+  } catch {
+    throw new JudgeProviderError(`Anthropic API reply is not JSON: ${body.slice(0, 200)}`, {
+      model
+    });
+  }
+  const blocks = parsed.content;
+  const text = blocks?.find((b) => b.type === "text")?.text;
+  if (typeof text !== "string" || text.length === 0) {
+    throw new JudgeProviderError(
+      `Anthropic API reply carries no text block (refusal or empty): ${body.slice(0, 300)}`,
+      { model }
+    );
+  }
+  return text;
+}
+
+// src/providers/claude-cli.ts
+var MAX_STDOUT_BYTES = 10 * 1024 * 1024;
+var ClaudeCliJudgeProvider = class {
+  constructor(opts) {
+    this.opts = opts;
+  }
+  opts;
+  async judgeOnce(request) {
+    const { bin, model, timeoutMs } = this.opts;
+    const args = [
+      "-p",
+      "--output-format",
+      "json",
+      "--strict-mcp-config",
+      ...model ? ["--model", model] : [],
+      VERDICT_PROMPT(request.rubric, request.text)
+    ];
+    const exec = this.opts.execImpl ?? await defaultExec();
+    let stdout;
+    try {
+      ({ stdout } = await exec(bin, args, { timeout: timeoutMs, maxBuffer: MAX_STDOUT_BYTES }));
+    } catch (e) {
+      throw toTypedError(e, bin, timeoutMs);
+    }
+    const envelope = parseEnvelope(stdout, bin);
+    const modelLabel = firstModelId(envelope) ?? model ?? "claude";
+    if (envelope.is_error === true || typeof envelope.result !== "string" || !envelope.result) {
+      throw new JudgeProviderError(
+        `${bin} -p returned an error result: ${String(envelope.result ?? stdout).slice(0, 300)}`,
+        { model: modelLabel }
+      );
+    }
+    return { ...parseVerdictReply(envelope.result, modelLabel), model: modelLabel };
+  }
+};
+async function defaultExec() {
+  const { execFile } = await import("child_process");
+  const { promisify } = await import("util");
+  return promisify(execFile);
+}
+function toTypedError(e, bin, timeoutMs) {
+  const err = e;
+  if (err.code === "ENOENT") {
+    return new JudgeConfigError(
+      `Claude Code CLI not found ("${bin}") \u2014 install it (https://claude.com/claude-code) or point JUDGE_CLAUDE_BIN at the binary`,
+      { bin }
+    );
+  }
+  if (err.code === "EACCES") {
+    return new JudgeConfigError(`Claude Code CLI is not executable ("${bin}")`, { bin });
+  }
+  if (err.killed === true) {
+    return new JudgeProviderError(`${bin} -p timed out after ${timeoutMs}ms`, { bin, timeoutMs });
+  }
+  const stderr = typeof err.stderr === "string" && err.stderr ? err.stderr : String(err.message ?? e);
+  return new JudgeProviderError(`${bin} -p failed: ${stderr.slice(0, 500)}`, { bin });
+}
+function parseEnvelope(stdout, bin) {
+  try {
+    return JSON.parse(stdout);
+  } catch {
+    throw new JudgeProviderError(
+      `${bin} -p did not return the JSON envelope: ${stdout.slice(0, 300)}`,
+      { bin }
+    );
+  }
+}
+function firstModelId(envelope) {
+  const keys = envelope.modelUsage ? Object.keys(envelope.modelUsage) : [];
+  return keys.length > 0 ? keys[0] : void 0;
+}
+
+// src/policy.ts
+async function judgeWithRetries(provider, request, retries) {
+  const maxAttempts = Math.max(0, retries) + 1;
+  let last = null;
+  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
+    const single = await provider.judgeOnce(request);
+    last = {
+      verdict: single.pass ? "pass" : "fail",
+      reasoning: single.reasoning,
+      model: single.model,
+      attempts: attempt
+    };
+    if (single.pass) return last;
+  }
+  return last;
+}
+
+// src/env.ts
+var PROVIDER_KINDS = [
+  "fake",
+  "vertex",
+  "claude",
+  "gemini",
+  "openai",
+  "anthropic"
+];
+var DEFAULT_MODEL = {
+  fake: void 0,
+  vertex: "gemini-2.5-flash",
+  gemini: "gemini-2.5-flash",
+  openai: "gpt-5-mini",
+  anthropic: "claude-haiku-4-5",
+  claude: void 0
+};
+var API_KEY_ENV = {
+  gemini: "GEMINI_API_KEY",
+  openai: "OPENAI_API_KEY",
+  anthropic: "ANTHROPIC_API_KEY"
+};
+var DEFAULT_RETRIES = 1;
+var DEFAULT_TIMEOUT_MS = 3e4;
+var DEFAULT_CLAUDE_TIMEOUT_MS = 12e4;
+function intEnv(env, name, fallback, min, max) {
+  const raw = env[name];
+  if (raw === void 0 || raw === "") return fallback;
+  const n = Number.parseInt(raw, 10);
+  if (Number.isNaN(n) || n < min || n > max) {
+    throw new JudgeConfigError(`${name} must be an integer in [${min}, ${max}], got "${raw}"`);
+  }
+  return n;
+}
+function resolveJudgeServiceEnv(env) {
+  const provider = env.JUDGE_PROVIDER;
+  if (!provider || !PROVIDER_KINDS.includes(provider)) {
+    const list = PROVIDER_KINDS.map((k) => `"${k}"`).join(" | ");
+    throw new JudgeConfigError(
+      provider === void 0 || provider === "" ? `JUDGE_PROVIDER is not set \u2014 one of ${list} ("fake" is deterministic and CI-safe)` : `JUDGE_PROVIDER must be one of ${list}, got "${provider}"`
+    );
+  }
+  const resolved = {
+    provider,
+    retries: intEnv(env, "JUDGE_RETRIES", DEFAULT_RETRIES, 0, 10),
+    timeoutMs: intEnv(
+      env,
+      "JUDGE_TIMEOUT_MS",
+      provider === "claude" ? DEFAULT_CLAUDE_TIMEOUT_MS : DEFAULT_TIMEOUT_MS,
+      1e3,
+      6e5
+    )
+  };
+  const model = env.JUDGE_MODEL || DEFAULT_MODEL[provider];
+  if (model) resolved.model = model;
+  if (provider === "vertex") {
+    const project = env.GOOGLE_CLOUD_PROJECT;
+    const location = env.GOOGLE_CLOUD_LOCATION;
+    if (!project || !location) {
+      throw new JudgeConfigError(
+        "JUDGE_PROVIDER=vertex requires GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION (ADC auth, no API keys)"
+      );
+    }
+    resolved.project = project;
+    resolved.location = location;
+    if (env.JUDGE_ACCESS_TOKEN) resolved.accessToken = env.JUDGE_ACCESS_TOKEN;
+  }
+  const keyEnvName = API_KEY_ENV[provider];
+  if (keyEnvName) {
+    const apiKey = env[keyEnvName];
+    if (!apiKey) {
+      throw new JudgeConfigError(`JUDGE_PROVIDER=${provider} requires ${keyEnvName}`);
+    }
+    resolved.apiKey = apiKey;
+  }
+  if (provider === "claude") {
+    resolved.claudeBin = env.JUDGE_CLAUDE_BIN || "claude";
+  }
+  return resolved;
+}
+function resolveJudgeServerEnv(env) {
+  const resolved = {
+    host: env.JUDGE_HOST || "127.0.0.1",
+    port: intEnv(env, "JUDGE_PORT", 8790, 1, 65535)
+  };
+  if (env.JUDGE_TOKEN) resolved.token = env.JUDGE_TOKEN;
+  return resolved;
+}
+
+// src/judge-service.ts
+function buildProvider(env) {
+  switch (env.provider) {
+    case "fake":
+      return new FakeJudgeProvider();
+    case "vertex":
+      return new VertexJudgeProvider({
+        project: env.project,
+        location: env.location,
+        model: env.model,
+        timeoutMs: env.timeoutMs,
+        ...env.accessToken ? { accessToken: env.accessToken } : {}
+      });
+    case "gemini":
+      return new GeminiJudgeProvider({
+        apiKey: env.apiKey,
+        model: env.model,
+        timeoutMs: env.timeoutMs
+      });
+    case "openai":
+      return new OpenAiJudgeProvider({
+        apiKey: env.apiKey,
+        model: env.model,
+        timeoutMs: env.timeoutMs
+      });
+    case "anthropic":
+      return new AnthropicJudgeProvider({
+        apiKey: env.apiKey,
+        model: env.model,
+        timeoutMs: env.timeoutMs
+      });
+    case "claude":
+      return new ClaudeCliJudgeProvider({
+        bin: env.claudeBin,
+        timeoutMs: env.timeoutMs,
+        ...env.model ? { model: env.model } : {}
+      });
+  }
+}
+function createJudgeService(env, provider) {
+  const p = provider ?? buildProvider(env);
+  return {
+    judge: (request) => judgeWithRetries(p, request, env.retries)
+  };
+}
+function createJudgeServiceFromEnv(env) {
+  return createJudgeService(resolveJudgeServiceEnv(env));
+}
+
+// src/server.ts
+import { createServer } from "http";
+import { JUDGE_ROUTES } from "@unotest/protocol";
+var MAX_BODY_BYTES = 1024 * 1024;
+function startJudgeServer(service, env) {
+  const server = createServer((req, res) => {
+    void route(service, env, req, res);
+  });
+  return new Promise((resolve, reject) => {
+    server.once("error", reject);
+    server.listen(env.port, env.host, () => {
+      const addr = server.address();
+      const port = typeof addr === "object" && addr !== null ? addr.port : env.port;
+      resolve({
+        server,
+        port,
+        close: () => new Promise((res2, rej2) => server.close((e) => e ? rej2(e) : res2()))
+      });
+    });
+  });
+}
+async function route(service, env, req, res) {
+  const url = req.url ?? "/";
+  if (req.method === "GET" && url === JUDGE_ROUTES.health) {
+    sendJson(res, 200, { ok: true });
+    return;
+  }
+  if (req.method !== "POST" || url !== JUDGE_ROUTES.judge) {
+    sendError(res, 404, `unknown route ${req.method} ${url}`, "bad-request");
+    return;
+  }
+  if (env.token && req.headers.authorization !== `Bearer ${env.token}`) {
+    sendError(res, 401, "missing or wrong bearer token (JUDGE_TOKEN)", "unauthorized");
+    return;
+  }
+  let body;
+  try {
+    body = parseRequest(await readBody(req));
+  } catch (e) {
+    sendError(res, 400, e instanceof Error ? e.message : String(e), "bad-request");
+    return;
+  }
+  try {
+    sendJson(res, 200, await service.judge(body));
+  } catch (e) {
+    if (e instanceof JudgeProviderError) {
+      sendError(res, 502, e.message, "provider-error");
+    } else if (e instanceof JudgeConfigError) {
+      sendError(res, 500, e.message, "internal");
+    } else {
+      sendError(res, 500, e instanceof Error ? e.message : String(e), "internal");
+    }
+  }
+}
+function parseRequest(raw) {
+  let parsed;
+  try {
+    parsed = JSON.parse(raw);
+  } catch {
+    throw new Error("request body is not JSON");
+  }
+  const { rubric, text } = parsed;
+  if (typeof rubric !== "string" || rubric.trim() === "") {
+    throw new Error('request body needs a non-empty string "rubric"');
+  }
+  if (typeof text !== "string") {
+    throw new Error('request body needs a string "text"');
+  }
+  return { rubric, text };
+}
+function readBody(req) {
+  return new Promise((resolve, reject) => {
+    const chunks = [];
+    let size = 0;
+    req.on("data", (chunk) => {
+      size += chunk.length;
+      if (size > MAX_BODY_BYTES) {
+        reject(new Error(`request body exceeds ${MAX_BODY_BYTES} bytes`));
+        req.destroy();
+        return;
+      }
+      chunks.push(chunk);
+    });
+    req.on("end", () => resolve(Buffer.concat(chunks).toString("utf8")));
+    req.on("error", reject);
+  });
+}
+function sendJson(res, status, body) {
+  const payload = JSON.stringify(body);
+  res.writeHead(status, { "content-type": "application/json" });
+  res.end(payload);
+}
+function sendError(res, status, error, code) {
+  sendJson(res, status, { error, code });
+}
+
+export {
+  JudgeError,
+  JudgeConfigError,
+  JudgeProviderError,
+  FAKE_MODEL_ID,
+  parseFakeRubric,
+  FakeJudgeProvider,
+  VERDICT_PROMPT,
+  parseVerdictReply,
+  VertexJudgeProvider,
+  GeminiJudgeProvider,
+  OpenAiJudgeProvider,
+  AnthropicJudgeProvider,
+  ClaudeCliJudgeProvider,
+  judgeWithRetries,
+  resolveJudgeServiceEnv,
+  resolveJudgeServerEnv,
+  buildProvider,
+  createJudgeService,
+  createJudgeServiceFromEnv,
+  startJudgeServer
+};
+//# sourceMappingURL=chunk-4VUHPR26.js.map

package/dist/cli.d.ts

@@ -0,0 +1,3 @@
+declare function main(argv?: string[]): Promise<void>;
+
+export { main };

package/dist/cli.js

@@ -0,0 +1,65 @@
+import {
+  createJudgeService,
+  resolveJudgeServerEnv,
+  resolveJudgeServiceEnv,
+  startJudgeServer
+} from "./chunk-4VUHPR26.js";
+
+// src/cli.ts
+async function main(argv = process.argv.slice(2)) {
+  if (argv.includes("--help") || argv.includes("-h")) {
+    printHelp();
+    return;
+  }
+  if (argv.includes("--version") || argv.includes("-v")) {
+    const { readFileSync } = await import("fs");
+    const pkg = JSON.parse(
+      readFileSync(new URL("../package.json", import.meta.url), "utf8")
+    );
+    console.log(pkg.version);
+    return;
+  }
+  const serviceEnv = resolveJudgeServiceEnv(process.env);
+  const serverEnv = resolveJudgeServerEnv(process.env);
+  const started = await startJudgeServer(createJudgeService(serviceEnv), serverEnv);
+  console.log(
+    `unotest-judge listening on http://${serverEnv.host}:${started.port} (provider: ${serviceEnv.provider}${serviceEnv.model ? `, model: ${serviceEnv.model}` : ""})`
+  );
+  const shutdown = () => {
+    void started.close().finally(() => process.exit(0));
+  };
+  process.once("SIGINT", shutdown);
+  process.once("SIGTERM", shutdown);
+}
+function printHelp() {
+  console.log(
+    [
+      "unotest-judge \u2014 LLM-judge service for @unotest/web's assertJudge",
+      "",
+      "Usage: npx @unotest/judge [--help] [--version]",
+      "",
+      "Env:",
+      "  JUDGE_PROVIDER          fake (deterministic, CI-safe) | vertex (ADC) |",
+      "                          claude (Claude Code CLI, subscription auth) |",
+      "                          gemini | openai | anthropic (API keys)",
+      "  JUDGE_MODEL             model id; defaults per provider:",
+      "                          vertex/gemini gemini-2.5-flash, openai gpt-5-mini,",
+      "                          anthropic claude-haiku-4-5, claude: CLI default",
+      "  JUDGE_RETRIES           extra provider calls on fail (default 1)",
+      "  JUDGE_TIMEOUT_MS        per-call budget, ms (default 30000; claude 120000)",
+      "  GOOGLE_CLOUD_PROJECT    vertex: ADC project",
+      "  GOOGLE_CLOUD_LOCATION   vertex: ADC location",
+      "  JUDGE_ACCESS_TOKEN      vertex: static bearer override (skips ADC)",
+      "  GEMINI_API_KEY          gemini: API key",
+      "  OPENAI_API_KEY          openai: API key",
+      "  ANTHROPIC_API_KEY       anthropic: API key",
+      "  JUDGE_CLAUDE_BIN        claude: binary override (default claude)",
+      "  JUDGE_HOST / JUDGE_PORT bind address (default 127.0.0.1:8790)",
+      "  JUDGE_TOKEN             optional bearer required on every /judge call"
+    ].join("\n")
+  );
+}
+export {
+  main
+};
+//# sourceMappingURL=cli.js.map

package/dist/index.d.ts

@@ -0,0 +1,183 @@
+import { JudgeRequest, JudgeVerdict } from '@unotest/protocol';
+export { JUDGE_ROUTES, JudgeErrorResponse, JudgeRequest, JudgeVerdict } from '@unotest/protocol';
+import { Server } from 'node:http';
+
+declare class JudgeError extends Error {
+    readonly context: Record<string, unknown>;
+    constructor(message: string, context?: Record<string, unknown>);
+}
+/** Provider / service configuration invalid or missing (env vars). */
+declare class JudgeConfigError extends JudgeError {
+}
+/** The provider call itself failed — transport fault, non-2xx from the
+ *  model backend, or an unparseable model reply. */
+declare class JudgeProviderError extends JudgeError {
+}
+
+interface SingleVerdict {
+    pass: boolean;
+    reasoning: string;
+    /** Model id that produced this verdict (`fake` for the fake provider). */
+    model: string;
+}
+interface JudgeProvider {
+    /** One un-retried judgement. Throws JudgeProviderError on transport /
+     *  parse faults; a clean `fail` verdict is a RESULT, not an error. */
+    judgeOnce(request: JudgeRequest): Promise<SingleVerdict>;
+}
+
+declare const FAKE_MODEL_ID = "fake";
+type Constraint = {
+    kind: "contains" | "not-contains";
+    needle: string;
+};
+declare function parseFakeRubric(rubric: string): Constraint[];
+declare class FakeJudgeProvider implements JudgeProvider {
+    judgeOnce(request: JudgeRequest): Promise<SingleVerdict>;
+}
+
+interface VertexProviderOptions {
+    project: string;
+    location: string;
+    model: string;
+    /** Static bearer token override (JUDGE_ACCESS_TOKEN). Absent → ADC. */
+    accessToken?: string;
+    /** Per-call wall-clock budget, ms. */
+    timeoutMs: number;
+    /** Injected fetch (tests). Defaults to global fetch. */
+    fetchImpl?: typeof fetch;
+}
+declare class VertexJudgeProvider implements JudgeProvider {
+    private readonly opts;
+    private readonly fetchImpl;
+    private tokenSource;
+    constructor(opts: VertexProviderOptions);
+    judgeOnce(request: JudgeRequest): Promise<SingleVerdict>;
+    private accessToken;
+}
+
+interface GeminiProviderOptions {
+    apiKey: string;
+    model: string;
+    /** Per-call wall-clock budget, ms. */
+    timeoutMs: number;
+    /** Injected fetch (tests). Defaults to global fetch. */
+    fetchImpl?: typeof fetch;
+}
+declare class GeminiJudgeProvider implements JudgeProvider {
+    private readonly opts;
+    private readonly fetchImpl;
+    constructor(opts: GeminiProviderOptions);
+    judgeOnce(request: JudgeRequest): Promise<SingleVerdict>;
+}
+
+interface OpenAiProviderOptions {
+    apiKey: string;
+    model: string;
+    /** Per-call wall-clock budget, ms. */
+    timeoutMs: number;
+    /** Injected fetch (tests). Defaults to global fetch. */
+    fetchImpl?: typeof fetch;
+}
+declare class OpenAiJudgeProvider implements JudgeProvider {
+    private readonly opts;
+    private readonly fetchImpl;
+    constructor(opts: OpenAiProviderOptions);
+    judgeOnce(request: JudgeRequest): Promise<SingleVerdict>;
+}
+
+interface AnthropicProviderOptions {
+    apiKey: string;
+    model: string;
+    /** Per-call wall-clock budget, ms. */
+    timeoutMs: number;
+    /** Injected fetch (tests). Defaults to global fetch. */
+    fetchImpl?: typeof fetch;
+}
+declare class AnthropicJudgeProvider implements JudgeProvider {
+    private readonly opts;
+    private readonly fetchImpl;
+    constructor(opts: AnthropicProviderOptions);
+    judgeOnce(request: JudgeRequest): Promise<SingleVerdict>;
+}
+
+interface ExecFileResult {
+    stdout: string;
+    stderr: string;
+}
+type ExecFileFn = (file: string, args: string[], opts: {
+    timeout: number;
+    maxBuffer: number;
+}) => Promise<ExecFileResult>;
+interface ClaudeCliProviderOptions {
+    /** Binary to spawn (JUDGE_CLAUDE_BIN, default "claude"). */
+    bin: string;
+    /** Passed as `--model` only when set — otherwise the CLI's default. */
+    model?: string;
+    /** Per-call wall-clock budget, ms. */
+    timeoutMs: number;
+    /** Injected exec (tests). Defaults to node:child_process execFile. */
+    execImpl?: ExecFileFn;
+}
+declare class ClaudeCliJudgeProvider implements JudgeProvider {
+    private readonly opts;
+    constructor(opts: ClaudeCliProviderOptions);
+    judgeOnce(request: JudgeRequest): Promise<SingleVerdict>;
+}
+
+declare const VERDICT_PROMPT: (rubric: string, text: string) => string;
+/** Parse the model's JSON verdict. Exported for unit tests. */
+declare function parseVerdictReply(text: string, model: string): {
+    pass: boolean;
+    reasoning: string;
+};
+
+declare function judgeWithRetries(provider: JudgeProvider, request: JudgeRequest, retries: number): Promise<JudgeVerdict>;
+
+type JudgeProviderKind = "fake" | "vertex" | "claude" | "gemini" | "openai" | "anthropic";
+interface JudgeServiceEnv {
+    provider: JudgeProviderKind;
+    /** Model id. Absent only for `claude` without JUDGE_MODEL (CLI default)
+     *  and for `fake` (no model behind it). */
+    model?: string;
+    /** Extra provider calls allowed on `fail` (first `pass` wins). */
+    retries: number;
+    /** Per-provider-call wall-clock budget, ms. */
+    timeoutMs: number;
+    /** Vertex ADC target. */
+    project?: string;
+    location?: string;
+    /** Static bearer override (vertex) — skips google-auth-library. */
+    accessToken?: string;
+    /** API key (gemini | openai | anthropic). */
+    apiKey?: string;
+    /** Claude Code binary override (claude). */
+    claudeBin?: string;
+}
+declare function resolveJudgeServiceEnv(env: NodeJS.ProcessEnv): JudgeServiceEnv;
+interface JudgeServerEnv {
+    host: string;
+    port: number;
+    /** Optional bearer token the server requires on every /judge call. */
+    token?: string;
+}
+declare function resolveJudgeServerEnv(env: NodeJS.ProcessEnv): JudgeServerEnv;
+
+interface JudgeService {
+    judge(request: JudgeRequest): Promise<JudgeVerdict>;
+}
+declare function buildProvider(env: JudgeServiceEnv): JudgeProvider;
+declare function createJudgeService(env: JudgeServiceEnv, provider?: JudgeProvider): JudgeService;
+/** Everything from process-env in one call — the entry point @unotest/web's
+ *  local mode uses. Throws JudgeConfigError with an actionable message on
+ *  missing/malformed env. */
+declare function createJudgeServiceFromEnv(env: NodeJS.ProcessEnv): JudgeService;
+
+interface StartedJudgeServer {
+    server: Server;
+    port: number;
+    close(): Promise<void>;
+}
+declare function startJudgeServer(service: JudgeService, env: JudgeServerEnv): Promise<StartedJudgeServer>;
+
+export { AnthropicJudgeProvider, ClaudeCliJudgeProvider, type ExecFileFn, type ExecFileResult, FAKE_MODEL_ID, FakeJudgeProvider, GeminiJudgeProvider, JudgeConfigError, JudgeError, type JudgeProvider, JudgeProviderError, type JudgeProviderKind, type JudgeServerEnv, type JudgeService, type JudgeServiceEnv, OpenAiJudgeProvider, type SingleVerdict, type StartedJudgeServer, VERDICT_PROMPT, VertexJudgeProvider, buildProvider, createJudgeService, createJudgeServiceFromEnv, judgeWithRetries, parseFakeRubric, parseVerdictReply, resolveJudgeServerEnv, resolveJudgeServiceEnv, startJudgeServer };

package/dist/index.js

@@ -0,0 +1,49 @@
+import {
+  AnthropicJudgeProvider,
+  ClaudeCliJudgeProvider,
+  FAKE_MODEL_ID,
+  FakeJudgeProvider,
+  GeminiJudgeProvider,
+  JudgeConfigError,
+  JudgeError,
+  JudgeProviderError,
+  OpenAiJudgeProvider,
+  VERDICT_PROMPT,
+  VertexJudgeProvider,
+  buildProvider,
+  createJudgeService,
+  createJudgeServiceFromEnv,
+  judgeWithRetries,
+  parseFakeRubric,
+  parseVerdictReply,
+  resolveJudgeServerEnv,
+  resolveJudgeServiceEnv,
+  startJudgeServer
+} from "./chunk-4VUHPR26.js";
+
+// src/index.ts
+import { JUDGE_ROUTES } from "@unotest/protocol";
+export {
+  AnthropicJudgeProvider,
+  ClaudeCliJudgeProvider,
+  FAKE_MODEL_ID,
+  FakeJudgeProvider,
+  GeminiJudgeProvider,
+  JUDGE_ROUTES,
+  JudgeConfigError,
+  JudgeError,
+  JudgeProviderError,
+  OpenAiJudgeProvider,
+  VERDICT_PROMPT,
+  VertexJudgeProvider,
+  buildProvider,
+  createJudgeService,
+  createJudgeServiceFromEnv,
+  judgeWithRetries,
+  parseFakeRubric,
+  parseVerdictReply,
+  resolveJudgeServerEnv,
+  resolveJudgeServiceEnv,
+  startJudgeServer
+};
+//# sourceMappingURL=index.js.map

package/package.json

@@ -0,0 +1,51 @@
+{
+  "name": "@unotest/judge",
+  "version": "0.23.0",
+  "description": "LLM-judge service for the @unotest ecosystem: judges free-form text (chat replies, generated content) against a natural-language rubric and returns a structured pass/fail verdict with reasoning. Runs as a small HTTP service (`npx @unotest/judge`) or in-process. Providers: deterministic `fake` (CI-safe), Google Vertex AI via ADC, the local Claude Code CLI (subscription auth, no API key), and Gemini / OpenAI / Anthropic APIs via keys — all raw HTTP or a local process, zero provider SDKs. Backs the `assertJudge` DSL assertion of @unotest/web.",
+  "license": "MIT",
+  "publishConfig": {
+    "access": "public"
+  },
+  "author": "Ivan Volkov <ivan@volkov.io>",
+  "homepage": "https://www.npmjs.com/package/@unotest/judge",
+  "type": "module",
+  "engines": {
+    "node": ">=20"
+  },
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "bin": {
+    "unotest-judge": "bin/unotest-judge.js"
+  },
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    }
+  },
+  "files": [
+    "dist",
+    "!dist/**/*.map",
+    "bin",
+    "CHANGELOG.md",
+    "LICENSE",
+    "README.md"
+  ],
+  "dependencies": {
+    "@unotest/protocol": "^0.23.0"
+  },
+  "peerDependencies": {
+    "google-auth-library": "^9.0.0 || ^10.0.0"
+  },
+  "peerDependenciesMeta": {
+    "google-auth-library": {
+      "optional": true
+    }
+  },
+  "devDependencies": {
+    "@types/node": "^22.10.0",
+    "tsup": "^8.5.1",
+    "tsx": "^4.19.2",
+    "typescript": "^5.7.2"
+  }
+}