@novedu/cli 0.20.0 → 0.22.0

package/README.md

@@ -104,6 +104,11 @@ each one must get — and `eval` replays them through the **real grader**, then
 what it actually did. This is the one command that both **runs the model** and needs
 you signed in (`novedu-cli login`); everything else about it is local.
 
+It checks **both halves** of a grading: your `expect` gates the **verdict**, and an LLM
+**feedback judge** audits the **feedback text** the student would have read — measured
+against the quiz's own grading prompt, so there is nothing extra to author. Flagged
+feedback is **reported, never a failure**.
+
 ```yaml
 # sorting-quiz.eval.yaml
 # yaml-language-server: $schema=https://raw.githubusercontent.com/Teaching-HTL-Leonding/novedu-chat-mvp/refs/heads/main/activities/evals/eval-yaml.schema.json
@@ -134,6 +139,13 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --repeats 3
 npx @novedu/cli eval ./sorting-quiz.eval.yaml \
   --llm-provider "Azure Foundry" --llm-model gpt-5-mini
 
+# A strong judge over the quiz's own grader — the recommended pairing
+npx @novedu/cli eval ./sorting-quiz.eval.yaml \
+  --judge-llm-provider "Azure Foundry" --judge-llm-model gpt-5.6-terra
+
+# Verdicts only: half the LLM calls, for a cheap smoke run
+npx @novedu/cli eval ./sorting-quiz.eval.yaml --no-judge-feedback
+
 # Machine-readable, for CI
 npx @novedu/cli eval ./sorting-quiz.eval.yaml --json --out eval-report.json
 
@@ -163,14 +175,37 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --report eval-report.md
   a broken file instead of aborting the batch. `--json` / `--out` always carry the
   same batch shape `{ files: [...], passed, totals }`, single file or not — `passed`
   is the exit-code verdict, per batch and per file.
+- **The feedback judge.** After each successful grading, an LLM reads the feedback the
+  grader wrote and checks it against **that grading's own system prompt** — the course
+  rules and the platform frame already in it. It reports four kinds of problem:
+  `contradicts_verdict` (praise on a wrong answer, or vice versa), `misstates_facts`,
+  `ignores_instructions` (most commonly: not stating the correct answer when the verdict
+  is not `correct`, or the wrong language), and `leaks_rubric` (quoting the grading
+  criteria at the student). Flags show as **`flagged feedback`** in the terminal report,
+  a **Flagged** column plus a **"Flagged feedback"** section in the Markdown report, and
+  `totals.feedbackFlagged` / `repeats[].judge.issues` in the JSON. They never change the
+  exit code.
+- **Choosing the judge.** By default the judge runs on the same model as the grader.
+  `--judge-llm-provider` + `--judge-llm-model` (both or neither) point it at another one,
+  which is the **recommended** setup: a strong judge over a smaller grader finds real
+  problems, while a small model judging itself mostly produces noise. `--no-judge-feedback`
+  turns judging off and halves the LLM calls; combining the two is rejected as
+  contradictory. Because judging roughly doubles the cost, the run's scope line says so
+  up front: `27 case(s) × 3 repeat(s) = 81 grading + 81 judge call(s)`.
+- **If the judge itself fails**, the run **degrades instead of aborting**: after three
+  consecutive judge failures it stops judging (one warning on stderr) and finishes the
+  grading normally. Your verdict results are complete; the feedback simply was not
+  audited: files that judged nothing show an em dash in the Flagged column rather than a
+  `0`, so "unchecked" never reads as "clean".
 - **`--report <file.md>`** additionally writes a readable **Markdown** report — an
   overview table over the files, then the question, the golden answer and the grader's
-  feedback for every mismatched, errored or unstable case (passing cases stay in the
+  feedback for every mismatched, errored or unstable case, plus the "Flagged feedback"
+  section (passing, unflagged cases stay in the
   JSON). It composes with `--json` / `--out` and leaves stdout untouched.
 - **Token totals.** The reports show what a run cost —
-  `tokens: 15,420 in (12,300 cached) / 2,810 out` — summed over the grading calls that
-  **succeeded**, so it is a lower bound (a retried or failed call reports nothing), and
-  nothing at all is printed when the server reports no usage.
+  `tokens: 15,420 in (12,300 cached) / 2,810 out` — summed over the grading **and** judge
+  calls that **succeeded**, so it is a lower bound (a retried or failed call reports
+  nothing), and nothing at all is printed when the server reports no usage.
 - **Failure handling**: a 5xx or network hiccup is retried (4 attempts, linear
   backoff); any 4xx is terminal; an auth failure aborts the run with one message; and
   three consecutive errored cases trip a circuit breaker so a down server fails fast.

package/dist/main.js

@@ -2591,6 +2591,10 @@ async function loadQuizFrom(url, fetcher, opts = {}) {
 		};
 	}
 }
+const tutorToolNameSchema = z.enum(["random_number"]).meta({
+	id: "tutorToolName",
+	description: "Name of a built-in tutor tool."
+});
 //#endregion
 //#region ../lib/tutors/schemas.ts
 /**
@@ -2626,6 +2630,7 @@ const TutorSchema = z.strictObject({
 		id: "llm",
 		description: "The model and provider that back this tutor."
 	}),
+	tools: z.array(tutorToolNameSchema).default([]).meta({ description: "Optional built-in tools the tutor's model may call (e.g. random_number). Off by default; mention enabled tools in tutor_instructions so the model uses them." }),
 	prompt: z.strictObject({
 		fragment_files: z.array(FragmentFileRefSchema).default([]).meta({ description: "Optional fragment libraries used by this tutor." }),
 		text_files: z.array(TextFileRefSchema).default([]).meta({ description: "Optional plain-text files (markdown / source) embedded verbatim via {{file \"alias\"}} markers." }),
@@ -2666,6 +2671,7 @@ async function loadAndBuildTutorPrompt(url, fetchImpl, opts = {}) {
 		model: tutor.llm.model,
 		provider: tutor.llm.provider,
 		imageInput: tutor.llm.imageInput ?? true,
+		tools: tutor.tools,
 		anonymous: tutor.anonymous ?? true,
 		title: tutor.title,
 		description: tutor.description,
@@ -2837,7 +2843,8 @@ const promptDumpers = {
 					provider: result.provider,
 					model: result.model
 				},
-				system: result.prompt
+				system: result.prompt,
+				tools: result.tools
 			}
 		};
 	} },
@@ -3304,6 +3311,89 @@ async function loadAndCheckEval(url, fetchImpl, opts = {}) {
 	};
 }
 //#endregion
+//#region ../lib/quiz-feedback-judge.ts
+/**
+* The QUIZ-feedback taxonomy. The judge may only name one of these, and the endpoint
+* constrains the model to whatever the CALLER sent (see {@link judgmentSchema}) — which
+* is what keeps the route kind-agnostic for the eval kinds still to come.
+*
+* Deliberately NOT documented in code comments: every definition lives in
+* {@link FEEDBACK_JUDGE_SYSTEM}, where the model actually reads it, so the two can never
+* drift apart.
+*/
+const FEEDBACK_JUDGE_CRITERIA = [
+	"contradicts_verdict",
+	"misstates_facts",
+	"ignores_instructions",
+	"leaks_rubric"
+];
+/**
+* The judge's system prompt. Three properties are load-bearing and were validated
+* against ~100 real golden answers plus planted violations before shipping:
+*
+*   * "Do NOT judge the verdict itself" — a different check (the eval's `expect`) owns
+*     that; a judge that re-grades produces noise the report cannot act on.
+*   * "be strict about real violations, but do not invent issues … when in doubt, the
+*     feedback is ok" — without it, weak models flag matters of taste.
+*   * an EMPTY `issues` array is the way to say "acceptable". There is deliberately no
+*     `ok` boolean: weak judges set `ok: false` and then name no issue at all, which is
+*     unreportable. Flagged ⇔ an issue was named.
+*/
+const FEEDBACK_JUDGE_SYSTEM = `You are auditing the FEEDBACK a quiz-grading assistant gave to a student.
+
+You receive:
+- the complete system prompt the grader was given (it contains shared course
+  rules, the question, and the grading criteria),
+- the student's answer,
+- the verdict the grader chose (correct / partial / incorrect),
+- the feedback text the grader wrote for the student.
+
+Judge ONLY the feedback text, on these criteria:
+
+- "contradicts_verdict": the feedback's message disagrees with the verdict —
+  e.g. it celebrates the answer as right although the verdict is incorrect, or
+  corrects an answer whose verdict is correct.
+- "misstates_facts": the feedback asserts something that the grading criteria
+  in the system prompt contradict — factual errors about the subject matter.
+- "ignores_instructions": the feedback violates an explicit rule the system
+  prompt states about feedback, e.g. it fails to state the correct answer even
+  though the verdict is not correct and the prompt demands that, is written in
+  a language the prompt does not allow, or is not addressed to the student.
+- "leaks_rubric": the feedback quotes the grading criteria verbatim, refers to
+  the grading instructions ("my instructions say...", "the rubric requires..."),
+  or reveals verdict boundaries the student is not supposed to see.
+
+Do NOT judge the verdict itself — a different auditor covers that. Judge the
+feedback GIVEN the verdict. Be strict about real violations, but do not invent
+issues: stylistic taste, brevity, or a matter of tone that the system prompt
+does not regulate are NOT issues. When in doubt, the feedback is ok.
+
+Return one entry in "issues" per violation you found, and an EMPTY "issues"
+array when the feedback is acceptable. Answer with the JSON object only.`;
+/**
+* The judge's USER message: the four inputs in labeled `===` blocks, the grader's system
+* prompt first and the feedback under judgment last (so the model reads the standard
+* before the thing it measures).
+*
+* Nothing is escaped — every part is DATA for the judge, not markup, and a course prompt
+* containing `===` or Markdown must reach the judge exactly as the grader saw it.
+*/
+function buildFeedbackJudgeSubject(gradingSystem, answer, verdict, feedback) {
+	return [
+		"=== The system prompt the grader was given ===",
+		gradingSystem,
+		"",
+		"=== The student's answer ===",
+		answer,
+		"",
+		"=== The grader's verdict ===",
+		verdict,
+		"",
+		"=== The grader's feedback (JUDGE THIS) ===",
+		feedback
+	].join("\n");
+}
+//#endregion
 //#region src/retry.ts
 const defaultSleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
 /**
@@ -3357,6 +3447,14 @@ function addUsage(total, usage) {
 	total.cachedInput += usage.cachedInput;
 	total.output += usage.output;
 }
+function createJudgeBreaker() {
+	return {
+		consecutiveErrors: 0,
+		stopped: false
+	};
+}
+/** Consecutive fully-errored judge calls that mean "stop judging for the rest of the run". */
+const JUDGE_BREAKER_LIMIT = 3;
 /** Consecutive fully-errored cases that mean "the server is down, stop now". */
 const CIRCUIT_BREAKER_LIMIT = 3;
 /** Flatten questions × answers into cases, each carrying its grading prompt. */
@@ -3391,6 +3489,12 @@ function majority(graded, expected) {
 		passed: tied.every((verdict) => expected.includes(verdict))
 	};
 }
+/** The one-line message a failed judge call leaves on its repeat row. */
+function judgeErrorMessage(error) {
+	if (typeof error === "string") return error;
+	if (typeof error === "object" && error !== null && "message" in error) return String(error.message);
+	return JSON.stringify(error ?? null);
+}
 /** The seam: one runner per eval kind (mirrors `promptDumpers`). */
 const evalRunners = { quiz: { async run(checked, options) {
 	const repeats = Math.max(1, Math.floor(options.repeats ?? 1));
@@ -3398,9 +3502,53 @@ const evalRunners = { quiz: { async run(checked, options) {
 	const planned = planCases(checked);
 	const total = planned.length * repeats;
 	const questionTexts = new Map(checked.quizQuestions.map((q) => [q.id, q.text]));
+	const breaker = options.judgeBreaker ?? createJudgeBreaker();
 	let done = 0;
 	let consecutiveErrored = 0;
 	let aborted;
+	/**
+	* The `judge` field of a repeat that produced NO judgment — spread onto every row
+	* that is not a successful grading. Judging on ⇒ an explicit `null` (so a script
+	* reading `judge === null` catches every unjudged repeat, not only the degraded
+	* ones); judging off ⇒ nothing at all, since the whole run carries no judge fields.
+	*/
+	const unjudged = options.judge ? { judge: null } : {};
+	/**
+	* Judge ONE graded repeat's feedback, as a dependent step of that repeat. Judged
+	* against the repeat's OWN verdict, never the case majority. Returns the fields to
+	* merge onto the row: a judgment, or `judge: null` plus a `judgeError` when the call
+	* failed, or a bare `judge: null` when the breaker had already degraded the run.
+	*/
+	const judgeRepeat = async (system, answer, verdict, feedback) => {
+		const judge = options.judge;
+		if (!judge || breaker.stopped) return { judge: null };
+		const outcome = await withRetry(() => judge({
+			system: FEEDBACK_JUDGE_SYSTEM,
+			subject: buildFeedbackJudgeSubject(system, answer, verdict, feedback),
+			criteria: FEEDBACK_JUDGE_CRITERIA
+		}), {
+			attempts: options.retry?.attempts,
+			baseDelayMs: options.retry?.baseDelayMs,
+			sleep: options.retry?.sleep,
+			shouldRetry: (value) => !value.ok && value.retryable && !breaker.stopped
+		});
+		if (outcome.ok) {
+			breaker.consecutiveErrors = 0;
+			return { judge: {
+				issues: outcome.issues,
+				...outcome.usage ? { usage: outcome.usage } : {}
+			} };
+		}
+		breaker.consecutiveErrors += 1;
+		if (!breaker.stopped && breaker.consecutiveErrors >= JUDGE_BREAKER_LIMIT) {
+			breaker.stopped = true;
+			options.onJudgeDegraded?.();
+		}
+		return {
+			judge: null,
+			judgeError: judgeErrorMessage(outcome.error)
+		};
+	};
 	const progress = () => {
 		done += 1;
 		options.onProgress?.({
@@ -3415,7 +3563,8 @@ const evalRunners = { quiz: { async run(checked, options) {
 			if (plan.system === void 0) {
 				rows.push({
 					repeatIndex,
-					error: { message: `The quiz has no question "${plan.questionId}".` }
+					error: { message: `The quiz has no question "${plan.questionId}".` },
+					...unjudged
 				});
 				progress();
 				continue;
@@ -3429,19 +3578,23 @@ const evalRunners = { quiz: { async run(checked, options) {
 				sleep: options.retry?.sleep,
 				shouldRetry: (value) => !value.ok && value.retryable && value.auth !== true
 			});
-			progress();
 			if (outcome.ok) {
+				const judged = options.judge ? await judgeRepeat(plan.system, plan.answer, outcome.verdict, outcome.feedback) : {};
+				progress();
 				rows.push({
 					repeatIndex,
 					got: outcome.verdict,
 					feedback: outcome.feedback,
-					...outcome.usage ? { usage: outcome.usage } : {}
+					...outcome.usage ? { usage: outcome.usage } : {},
+					...judged
 				});
 				continue;
 			}
+			progress();
 			rows.push({
 				repeatIndex,
-				error: outcome.error
+				error: outcome.error,
+				...unjudged
 			});
 			if (outcome.auth) {
 				aborted ??= {
@@ -3469,11 +3622,15 @@ const evalRunners = { quiz: { async run(checked, options) {
 			status,
 			...winner ? { verdict: winner.verdict } : {},
 			unstable: new Set(graded).size > 1,
+			feedbackFlagged: rows.some((row) => (row.judge?.issues.length ?? 0) > 0),
 			repeats: rows
 		};
 	});
 	const usage = { ...ZERO_USAGE };
-	for (const result of results) for (const row of result.repeats) addUsage(usage, row.usage);
+	for (const result of results) for (const row of result.repeats) {
+		addUsage(usage, row.usage);
+		addUsage(usage, row.judge?.usage);
+	}
 	const totals = {
 		cases: results.length,
 		passed: results.filter((c) => c.status === "passed").length,
@@ -3481,6 +3638,8 @@ const evalRunners = { quiz: { async run(checked, options) {
 		errored: results.filter((c) => c.status === "errored").length,
 		skipped: results.filter((c) => c.status === "skipped").length,
 		unstable: results.filter((c) => c.unstable).length,
+		feedbackFlagged: results.filter((c) => c.feedbackFlagged).length,
+		judgeErrored: results.reduce((sum, c) => sum + c.repeats.filter((row) => row.judgeError !== void 0).length, 0),
 		repeats,
 		calls: total,
 		usage
@@ -3505,6 +3664,7 @@ const evalRunners = { quiz: { async run(checked, options) {
 		id: checked.evalFile.id,
 		target: checked.targetUrl,
 		llm: options.llm,
+		judging: !options.judge ? "off" : breaker.stopped ? "degraded" : "on",
 		totals,
 		questions: [...new Set(checked.evalFile.questions.map((question) => question.question))].map((id) => ({
 			id,
@@ -3545,6 +3705,8 @@ function summarizeBatch(files) {
 		errored: 0,
 		skipped: 0,
 		unstable: 0,
+		feedbackFlagged: 0,
+		judgeErrored: 0,
 		usage: { ...ZERO_USAGE }
 	};
 	for (const file of files) {
@@ -3555,6 +3717,8 @@ function summarizeBatch(files) {
 		totals.errored += file.result.totals.errored;
 		totals.skipped += file.result.totals.skipped;
 		totals.unstable += file.result.totals.unstable;
+		totals.feedbackFlagged += file.result.totals.feedbackFlagged;
+		totals.judgeErrored += file.result.totals.judgeErrored;
 		addUsage(totals.usage, file.result.totals.usage);
 	}
 	return {
@@ -3567,9 +3731,22 @@ function summarizeBatch(files) {
 	};
 }
 /**
+* Did this file's run produce ANY judgment? The ONE rule every renderer derives its
+* flagged count's visibility from: a file that judged nothing — judging off, or every
+* case run after the breaker degraded the run — has NOT been found clean, so its flagged
+* count renders as "not checked" (an em dash, an omitted segment), never as a `0`.
+*/
+function anyJudged(result) {
+	return result.cases.some((evalCase) => evalCase.repeats.some((repeat) => repeat.judge !== void 0 && repeat.judge !== null));
+}
+/**
 * The CI gate: every file valid, and not a single failed, errored, or skipped CASE —
 * an aborted (and therefore incomplete) run must never read as a pass. The single
 * source of truth for the exit code AND for `EvalBatchResult.passed`.
+*
+* `unstable`, `feedbackFlagged` and `judgeErrored` deliberately do NOT appear here: all
+* three are reported, none gates. (Gating is per-KIND policy, not a property of judge
+* results — the quiz kind reports; a kind whose only check is the judge would gate on it.)
 */
 function batchPassed(batch) {
 	return batch.totals.invalid === 0 && batch.totals.failed === 0 && batch.totals.errored === 0 && batch.totals.skipped === 0;
@@ -3637,6 +3814,7 @@ function formatResult(result, source) {
 		lines.push(`  model: ${result.model}`);
 		lines.push(`  system prompt: ${result.prompt.length} chars`);
 		lines.push(`  imageInput: ${result.imageInput}   anonymous: ${result.anonymous}` + (result.exampleQuestions.length ? `   exampleQuestions: ${result.exampleQuestions.length}` : ""));
+		if (result.tools.length) lines.push(`  tools: ${result.tools.join(", ")}`);
 		if (result.warnings.length) {
 			lines.push("");
 			lines.push(yellow(`${result.warnings.length} warning(s):`));
@@ -3809,18 +3987,24 @@ function formatEvalReport(result, source) {
 	];
 	const llm = result.llm.overrides ? `${result.llm.overrides.provider} / ${result.llm.overrides.model} ${yellow("→")} ${result.llm.provider} / ${result.llm.model} ${yellow("(override)")}` : `${result.llm.provider} / ${result.llm.model}`;
 	lines.push(`  llm: ${llm}`);
-	lines.push(`  cases: ${totals.cases} × ${totals.repeats} repeat(s) = ${totals.calls} grading call(s)`);
+	const judge = result.llm.judge;
+	if (judge && (judge.provider !== result.llm.provider || judge.model !== result.llm.model)) lines.push(`  judge llm: ${judge.provider} / ${judge.model}${judge.overridden ? ` ${yellow("(override)")}` : ""}`);
+	lines.push(`  cases: ${totals.cases} × ${totals.repeats} repeat(s) = ${totals.calls} grading call(s)` + (result.judging === "off" ? "" : ` + ${totals.calls} judge call(s)`));
 	if (result.aborted) {
 		lines.push("");
 		lines.push(red(`Run aborted: ${result.aborted.message}`));
 	}
+	if (result.judging === "degraded") {
+		lines.push("");
+		lines.push(yellow("Feedback judging stopped after repeated judge failures — grading was unaffected."));
+	}
 	if (result.mismatches.length) {
 		lines.push("");
 		lines.push(red(`${result.mismatches.length} mismatch(es):`));
 		lines.push(...mismatchLines(result));
 	}
 	lines.push("");
-	lines.push(`  passed: ${totals.passed}   failed: ${totals.failed}   errored: ${totals.errored}` + (totals.skipped ? red(`   skipped: ${totals.skipped} (run aborted)`) : "") + (totals.unstable ? dim(`   unstable: ${totals.unstable}`) : ""));
+	lines.push(`  passed: ${totals.passed}   failed: ${totals.failed}   errored: ${totals.errored}` + (totals.skipped ? red(`   skipped: ${totals.skipped} (run aborted)`) : "") + (totals.unstable ? dim(`   unstable: ${totals.unstable}`) : "") + (!anyJudged(result) ? "" : totals.feedbackFlagged ? yellow(`   flagged feedback: ${totals.feedbackFlagged}`) : dim("   flagged feedback: 0")) + (totals.judgeErrored ? dim(`   judge errors: ${totals.judgeErrored}`) : ""));
 	const tokens = formatUsageLine(totals.usage);
 	if (tokens) lines.push(dim(`  ${tokens}`));
 	if (result.confusion.length) {
@@ -3850,11 +4034,13 @@ function formatEvalBatchReport(batch) {
 		}
 		const t = file.result.totals;
 		const mark = t.failed === 0 && t.errored === 0 && t.skipped === 0 ? green("✔") : red("✗");
-		lines.push(`  ${mark} ${name}: ${t.cases} case(s), ${t.passed} passed, ${t.failed} failed, ${t.errored} errored` + (t.skipped ? red(`, ${t.skipped} skipped`) : "") + (t.unstable ? dim(`, ${t.unstable} unstable`) : ""));
+		lines.push(`  ${mark} ${name}: ${t.cases} case(s), ${t.passed} passed, ${t.failed} failed, ${t.errored} errored` + (t.skipped ? red(`, ${t.skipped} skipped`) : "") + (t.unstable ? dim(`, ${t.unstable} unstable`) : "") + (!anyJudged(file.result) ? "" : t.feedbackFlagged ? yellow(`, ${t.feedbackFlagged} flagged`) : dim(", 0 flagged")));
 	}
 	const g = batch.totals;
+	const judged = batch.files.some((file) => file.result && anyJudged(file.result));
 	lines.push("");
-	lines.push(`  TOTAL: ${g.cases} case(s), ${g.passed} passed, ${g.failed} failed, ${g.errored} errored` + (g.skipped ? red(`, ${g.skipped} skipped`) : "") + (g.invalid ? red(`, ${g.invalid} invalid file(s)`) : ""));
+	lines.push(`  TOTAL: ${g.cases} case(s), ${g.passed} passed, ${g.failed} failed, ${g.errored} errored` + (g.skipped ? red(`, ${g.skipped} skipped`) : "") + (judged ? g.feedbackFlagged ? yellow(`, ${g.feedbackFlagged} flagged`) : dim(", 0 flagged") : "") + (g.invalid ? red(`, ${g.invalid} invalid file(s)`) : ""));
+	if (batch.files.some((file) => file.result?.judging === "degraded")) lines.push(yellow("  Feedback judging stopped mid-run after repeated judge failures — files without a flagged count were graded but not judged."));
 	const tokens = formatUsageLine(g.usage);
 	if (tokens) lines.push(dim(`  ${tokens}`));
 	for (const file of batch.files) {
@@ -3889,6 +4075,7 @@ function formatPromptDump(dump, sections, source) {
 	const lines = [green(`✔ Prompts — ${dump.kind}`) + dim(` — ${source}`)];
 	lines.push(`  id: ${dump.id}`);
 	lines.push(`  provider: ${dump.llm.provider}   model: ${dump.llm.model}`);
+	if (dump.kind === "tutor" && dump.tools.length > 0) lines.push(`  tools: ${dump.tools.join(", ")}`);
 	lines.push(`  prompts: ${sections.length}`);
 	for (const section of sections) lines.push(`    ${section.name}: ${section.text.length} chars`);
 	lines.push("");
@@ -3928,6 +4115,17 @@ function llmText(llm) {
 	const effective = `${llm.provider} / ${llm.model}`;
 	return llm.overrides ? `${llm.overrides.provider} / ${llm.overrides.model} → ${effective} (override)` : effective;
 }
+/**
+* The judge's pair, but ONLY when it differs from the grading pair — a judge line that
+* merely repeats the grader would be noise, while a differing one is essential: judge
+* strictness varies by model, so two reports are comparable only when it matches.
+*/
+function judgeLlmText(llm) {
+	const judge = llm.judge;
+	if (!judge) return void 0;
+	if (judge.provider === llm.provider && judge.model === llm.model) return void 0;
+	return `${judge.provider} / ${judge.model}${judge.overridden ? " (override)" : ""}`;
+}
 /** `15,420 / 12,300 / 2,810`, or an em dash when nothing was reported. */
 function usageCell(usage) {
 	if (usage.input === 0 && usage.cachedInput === 0 && usage.output === 0) return "—";
@@ -3956,6 +4154,7 @@ const OVERVIEW_HEADER = [
 	"Errored",
 	"Skipped",
 	"Unstable",
+	"Flagged",
 	"False-correct",
 	"Tokens (in / cached / out)"
 ];
@@ -3975,6 +4174,7 @@ function overview(batch) {
 		"---:",
 		"---:",
 		"---:",
+		"---:",
 		"---:"
 	])];
 	for (const file of batch.files) {
@@ -3990,6 +4190,7 @@ function overview(batch) {
 				"—",
 				"—",
 				"—",
+				"—",
 				"—"
 			]));
 			continue;
@@ -4004,12 +4205,14 @@ function overview(batch) {
 			count(t.errored),
 			count(t.skipped),
 			count(t.unstable),
+			anyJudged(file.result) ? count(t.feedbackFlagged) : "—",
 			falseCorrectCell(file.result),
 			usageCell(t.usage)
 		]));
 	}
 	if (batch.files.length > 1) {
 		const g = batch.totals;
+		const judged = batch.files.some((file) => file.result && anyJudged(file.result));
 		lines.push(row([
 			"**TOTAL**",
 			g.invalid ? `${count(g.invalid)} invalid` : "",
@@ -4019,6 +4222,7 @@ function overview(batch) {
 			`**${count(g.errored)}**`,
 			`**${count(g.skipped)}**`,
 			`**${count(g.unstable)}**`,
+			judged ? `**${count(g.feedbackFlagged)}**` : "—",
 			"",
 			`**${usageCell(g.usage)}**`
 		]));
@@ -4039,6 +4243,13 @@ function verdictSummary(evalCase) {
 function needsDetail(evalCase) {
 	return evalCase.status === "failed" || evalCase.status === "errored" || evalCase.unstable;
 }
+/** The "**Question** / **Golden answer**" intro every case detail section opens with. */
+function questionAndAnswer(evalCase, questionText) {
+	const lines = [];
+	if (questionText) lines.push("**Question**", "", quote(questionText), "");
+	lines.push("**Golden answer**", "", quote(evalCase.answer), "");
+	return lines;
+}
 /**
 * One case's section: the question it belongs to, the golden answer, and what the
 * grader said — plus every repeat when they disagreed (the `--repeats` signal is
@@ -4049,16 +4260,7 @@ function caseSection(evalCase, questionText) {
 	const unstable = evalCase.unstable ? " *(unstable)*" : "";
 	lines.push(`### \`${evalCase.questionId}\` #${evalCase.answerIndex} — ${verdictSummary(evalCase)}${unstable}`);
 	lines.push("");
-	if (questionText) {
-		lines.push("**Question**");
-		lines.push("");
-		lines.push(quote(questionText));
-		lines.push("");
-	}
-	lines.push("**Golden answer**");
-	lines.push("");
-	lines.push(quote(evalCase.answer));
-	lines.push("");
+	lines.push(...questionAndAnswer(evalCase, questionText));
 	const graded = evalCase.repeats.filter((r) => r.got !== void 0);
 	const disagreed = new Set(graded.map((r) => r.got)).size > 1;
 	if (evalCase.repeats.length > 1 && (disagreed || graded.length !== evalCase.repeats.length)) {
@@ -4088,6 +4290,38 @@ function caseSection(evalCase, questionText) {
 	}
 	return lines;
 }
+/**
+* The "Flagged feedback" section: what the LLM judge found wrong with the TEXT the
+* grader wrote, per case, with each flagged repeat's verdict and feedback quoted verbatim
+* and the judge's issues as `criterion — note` items.
+*
+* Separate from the verdict sections on purpose — these cases usually PASSED (the verdict
+* was right, the wording was not), and mixing them into the mismatch list would suggest
+* the run failed on them. Empty when the file has no flags.
+*/
+function flaggedSection(result, questionText) {
+	const flagged = result.cases.filter((evalCase) => evalCase.feedbackFlagged);
+	if (flagged.length === 0) return [];
+	const lines = ["### Flagged feedback", ""];
+	lines.push("_An LLM judge audited each feedback text against the very grading prompt it was written under. Reported only — flagged feedback never fails a run._");
+	lines.push("");
+	for (const evalCase of flagged) {
+		lines.push(`#### \`${evalCase.questionId}\` #${evalCase.answerIndex}`);
+		lines.push("");
+		lines.push(...questionAndAnswer(evalCase, questionText.get(evalCase.questionId)));
+		for (const repeat of evalCase.repeats) {
+			const issues = repeat.judge?.issues ?? [];
+			if (issues.length === 0) continue;
+			lines.push(`**Repeat #${repeat.repeatIndex + 1} — \`${repeat.got ?? "?"}\`**`);
+			lines.push("");
+			lines.push(quote(repeat.feedback ?? ""));
+			lines.push("");
+			for (const issue of issues) lines.push(`- \`${cell(issue.criterion)}\` — ${inline(issue.note)}`);
+			lines.push("");
+		}
+	}
+	return lines;
+}
 /** One file's details section, or `[]` when the file has nothing to report. */
 function fileDetails(file) {
 	const name = shortSource(file.source);
@@ -4105,8 +4339,9 @@ function fileDetails(file) {
 	const result = file.result;
 	const detailed = result.cases.filter(needsDetail);
 	const skipped = result.totals.skipped;
-	if (detailed.length === 0 && skipped === 0 && !result.aborted) return [];
 	const questionText = new Map(result.questions.map((question) => [question.id, question.text]));
+	const flagged = flaggedSection(result, questionText);
+	if (detailed.length === 0 && skipped === 0 && !result.aborted && flagged.length === 0) return [];
 	const lines = [`## ${cell(name)} — \`${cell(result.id)}\``, ""];
 	if (result.aborted) {
 		lines.push("> [!WARNING]");
@@ -4119,6 +4354,7 @@ function fileDetails(file) {
 		lines.push(`**${count(skipped)} case(s) were never attempted**${reason} — the run is incomplete, so it cannot pass.`);
 		lines.push("");
 	}
+	lines.push(...flagged);
 	return lines;
 }
 /**
@@ -4132,6 +4368,8 @@ function renderEvalMarkdownReport(batch, meta) {
 	lines.push(`- **Generated** ${timestamp(meta.generatedAt)} · novedu-cli ${meta.cliVersion}`);
 	const llms = [...new Set(batch.files.filter((f) => f.result).map((f) => llmText(f.result.llm)))];
 	for (const llm of llms) lines.push(`- **LLM** ${llm}`);
+	const judges = [...new Set(batch.files.map((file) => file.result ? judgeLlmText(file.result.llm) : void 0).filter((text) => text !== void 0))];
+	for (const judge of judges) lines.push(`- **Feedback judge** ${judge}`);
 	lines.push(`- **Run** ${count(batch.totals.files)} file(s), ${count(batch.totals.cases)} case(s) × ${count(meta.repeats)} repeat(s), concurrency ${count(meta.concurrency)}`);
 	const tokens = batch.totals.usage;
 	if (tokens.input || tokens.cachedInput || tokens.output) lines.push(`- **Tokens** ${count(tokens.input)} in (${count(tokens.cachedInput)} cached) / ${count(tokens.output)} out — successful grading calls only, so a lower bound`);
@@ -4141,6 +4379,12 @@ function renderEvalMarkdownReport(batch, meta) {
 		lines.push(`> The run was ABORTED — ${count(batch.totals.skipped)} case(s) were never graded, so this report is incomplete.`);
 		lines.push("");
 	}
+	const degradedAt = batch.files.find((file) => file.result?.judging === "degraded");
+	if (degradedAt?.result) {
+		lines.push("> [!WARNING]");
+		lines.push(`> Feedback judging STOPPED during \`${cell(shortSource(degradedAt.source))}\` after 3 judge calls failed in a row — everything from there on was graded but NOT judged, and shows an em dash rather than a count in the Flagged column. Grading was unaffected.`);
+		lines.push("");
+	}
 	lines.push("## Overview");
 	lines.push("");
 	lines.push(...overview(batch));
@@ -4150,7 +4394,7 @@ function renderEvalMarkdownReport(batch, meta) {
 		lines.push("_Nothing else to report — every case matched its expected verdict. The `--json` report carries every case, including the passing ones._");
 		lines.push("");
 	} else {
-		lines.push("_Below: only the mismatched, errored and unstable cases. Passing cases live in the `--json` report._");
+		lines.push("_Below: only the mismatched, errored and unstable cases, plus any feedback the judge flagged. Passing cases live in the `--json` report._");
 		lines.push("");
 		lines.push(...details);
 	}
@@ -4485,23 +4729,27 @@ function expandSources(args) {
 		duplicates
 	};
 }
-/** The `--llm-provider`/`--llm-model` pair: strictly both-or-nothing, provider checked. */
-function parseOverride(options) {
-	const { llmProvider, llmModel } = options;
-	if (llmProvider === void 0 && llmModel === void 0) return { ok: true };
-	if (llmProvider === void 0 || llmModel === void 0) return {
+/**
+* One provider/model pair from the flags: strictly BOTH-OR-NOTHING (the `effectiveLlm`
+* rule, docs/ai-models.md) with the provider checked against the known list. Shared by
+* `--llm-*` (the grading override) and `--judge-llm-*` (the judge's own pair), so the two
+* can never drift in wording or in strictness.
+*/
+function parsePair(flag, provider, model) {
+	if (provider === void 0 && model === void 0) return { ok: true };
+	if (provider === void 0 || model === void 0) return {
 		ok: false,
-		message: "Pass --llm-provider and --llm-model together, or neither."
+		message: `Pass --${flag}-provider and --${flag}-model together, or neither.`
 	};
-	if (!LLM_PROVIDERS.includes(llmProvider)) return {
+	if (!LLM_PROVIDERS.includes(provider)) return {
 		ok: false,
-		message: `Unknown --llm-provider "${llmProvider}": expected ${LLM_PROVIDERS.map((p) => `"${p}"`).join(" or ")}.`
+		message: `Unknown --${flag}-provider "${provider}": expected ${LLM_PROVIDERS.map((p) => `"${p}"`).join(" or ")}.`
 	};
 	return {
 		ok: true,
 		llm: {
-			provider: llmProvider,
-			model: llmModel
+			provider,
+			model
 		}
 	};
 }
@@ -4571,6 +4819,101 @@ function makeGradeFn(server, llm) {
 		};
 	};
 }
+/**
+* The HTTP seam for ONE judge call, with the run's judge llm closed in. Mirrors
+* {@link makeGradeFn}'s failure classification, minus the auth branch: a judge failure
+* NEVER aborts the run — it degrades judging (see the runner's breaker) while the grading
+* half finishes untouched.
+*/
+function makeJudgeFn(server, llm) {
+	return async ({ system, subject, criteria }) => {
+		const response = await performApiRequest({
+			server,
+			path: "/api/eval/judge",
+			method: "POST",
+			body: {
+				llm,
+				system,
+				subject,
+				criteria: [...criteria]
+			},
+			quiet: true
+		});
+		if (response.ok) {
+			const payload = response.payload;
+			if (Array.isArray(payload?.issues)) {
+				const issues = payload.issues.flatMap((entry) => {
+					const { criterion, note } = entry ?? {};
+					return typeof criterion === "string" ? [{
+						criterion,
+						note: typeof note === "string" ? note : ""
+					}] : [];
+				});
+				const usage = parseUsage(payload?.usage);
+				return {
+					ok: true,
+					issues,
+					...usage ? { usage } : {}
+				};
+			}
+			return {
+				ok: false,
+				retryable: false,
+				error: { message: "The server's response is not a feedback judgment — it may not offer /api/eval/judge at all (does it run a Novedu version with the feedback judge?). Re-run with --no-judge-feedback to grade without judging." }
+			};
+		}
+		return {
+			ok: false,
+			retryable: response.status === void 0 || response.status >= 500,
+			error: response.error
+		};
+	};
+}
+/** Budget for the one version probe — a hung check must never hold up a run. */
+const VERSION_CHECK_TIMEOUT_MS = 5e3;
+/**
+* Warn when this CLI was not built from the same commit as the server it is about to
+* grade against. `eval` assembles every grading system prompt LOCALLY, from the `lib/**`
+* prompt builders frozen into this published CLI — so a stale binary can certify prompts
+* the server's activities no longer send. CLI and server live in one repo, which makes
+* the server's `cliVersion` (from `GET /api/version`, public and unauthenticated) exactly
+* the CLI release matching its bundled code.
+*
+* Deliberately EVAL-ONLY (prompt drift corrupts nothing else) and strictly advisory: one
+* fetch, no retry, never an abort, never an exit code, and never a byte on stdout — the
+* JSON output contract owns that stream. Unlike progress it prints off a TTY too: a CI
+* log is precisely where this warning has to survive. Absence is NOT silently forgiven —
+* an unreachable, non-JSON, non-2xx or `cliVersion`-less answer says so, because "could
+* not check" and "checked, fine" must not look the same.
+*/
+async function warnOnVersionMismatch(server) {
+	const local = cliVersion();
+	const unverifiable = (reason) => {
+		process.stderr.write(`Warning: could not verify that this CLI (${local}) matches the server's — ${reason}. Locally assembled grading prompts may differ from what that server's activities run.
+`);
+	};
+	const base = resolveServerUrl(server);
+	let payload;
+	try {
+		const response = await fetch(new URL("/api/version", base), { signal: AbortSignal.timeout(VERSION_CHECK_TIMEOUT_MS) });
+		if (!response.ok) {
+			unverifiable(`${base} answered HTTP ${response.status}`);
+			return;
+		}
+		payload = await response.json();
+	} catch (error) {
+		unverifiable(`${base} did not answer (${error instanceof Error ? error.message : error})`);
+		return;
+	}
+	const remote = payload?.cliVersion;
+	if (typeof remote !== "string" || remote === "") {
+		unverifiable(`${base} reports no CLI version (does it run a Novedu version that has one?)`);
+		return;
+	}
+	if (remote === local) return;
+	process.stderr.write(`Warning: this CLI is ${local} but the server was built with CLI ${remote} — locally assembled grading prompts may differ from what that server's activities run. Update: npm i -g @novedu/cli
+`);
+}
 /** stderr progress, suppressed off a TTY so CI logs stay readable. */
 function progressWriter(prefix) {
 	if (!process.stderr.isTTY) return void 0;
@@ -4579,16 +4922,41 @@ function progressWriter(prefix) {
 	};
 }
 /**
+* The off-a-TTY replacement for the spinner: ONE newline-terminated line per finished
+* file. The `\r` counter above is suppressed when stderr is redirected (it would fill a
+* log with carriage-return noise), which otherwise left a long batch printing nothing at
+* all between the scope banner and the final report — indistinguishable from a hang, and
+* an easy way to talk yourself into killing a healthy run. Coarse and greppable is
+* enough: it proves liveness and says which file the run reached.
+*
+* Deliberately no timings — a per-file duration invites extrapolating an ETA that the
+* model, the provider's load and `--concurrency` make unreliable.
+*/
+function writeFileDone(label, result) {
+	const totals = result.totals;
+	process.stderr.write(`${label}: ${totals.cases} case(s), ${totals.passed} passed, ${totals.failed} failed, ${totals.errored} errored` + (totals.skipped ? `, ${totals.skipped} skipped` : "") + (anyJudged(result) ? `, ${totals.feedbackFlagged} flagged` : "") + "\n");
+}
+/**
 * The command's core, exported for the unit tests. `seams` exists only so tests can
 * shrink the retry backoff — the CLI itself never passes it (PoC parity: 4 attempts,
 * 5 s linear).
 */
 async function runEvalCommand(pathsOrUrls, options, seams = {}) {
-	const override = parseOverride(options);
+	const override = parsePair("llm", options.llmProvider, options.llmModel);
 	if (!override.ok) {
 		failJson({ message: override.message });
 		return;
 	}
+	const judgeOverride = parsePair("judge-llm", options.judgeLlmProvider, options.judgeLlmModel);
+	if (!judgeOverride.ok) {
+		failJson({ message: judgeOverride.message });
+		return;
+	}
+	const judging = options.judgeFeedback !== false;
+	if (!judging && judgeOverride.llm) {
+		failJson({ message: "--judge-llm-provider/--judge-llm-model cannot be combined with --no-judge-feedback: the first configures the feedback judge, the second switches it off." });
+		return;
+	}
 	const expansion = expandSources(pathsOrUrls);
 	if (!expansion.ok) {
 		failJson({ message: expansion.message });
@@ -4632,8 +5000,14 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 	}
 	{
 		const totalCases = [...checked.values()].reduce((sum, file) => sum + file.caseCount, 0);
-		process.stderr.write(`${totalCases} case(s) × ${repeats} repeat(s) = ${totalCases * repeats} grading call(s)\n`);
+		const calls = totalCases * repeats;
+		process.stderr.write(`${totalCases} case(s) × ${repeats} repeat(s) = ${calls} grading` + (judging ? ` + ${calls} judge call(s)\n` : " call(s)\n"));
 	}
+	await warnOnVersionMismatch(options.server);
+	const judgeBreaker = createJudgeBreaker();
+	const onJudgeDegraded = () => {
+		process.stderr.write("Warning: feedback judging was stopped after 3 judge calls failed in a row — the rest of this run is graded but NOT judged. Grading is unaffected.\n");
+	};
 	let fileIndex = 0;
 	for (const file of files) {
 		fileIndex += 1;
@@ -4644,20 +5018,31 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 			model: check.quizDump.llm.model
 		};
 		const effective = override.llm ?? quizLlm;
+		const judgeLlm = judgeOverride.llm ?? effective;
 		const llm = {
 			...effective,
-			...override.llm ? { overrides: quizLlm } : {}
+			...override.llm ? { overrides: quizLlm } : {},
+			...judging ? { judge: {
+				...judgeLlm,
+				overridden: judgeOverride.llm !== void 0
+			} } : {}
 		};
-		const prefix = files.length > 1 ? `(${fileIndex}/${files.length}) ${check.evalFile.id}: ` : "";
-		file.result = await runEval("quiz", check, {
+		const label = files.length > 1 ? `(${fileIndex}/${files.length}) ${check.evalFile.id}` : check.evalFile.id;
+		const prefix = files.length > 1 ? `${label}: ` : "";
+		const result = await runEval("quiz", check, {
 			grade: makeGradeFn(options.server, effective),
+			...judging ? { judge: makeJudgeFn(options.server, judgeLlm) } : {},
+			judgeBreaker,
+			onJudgeDegraded,
 			concurrency,
 			repeats,
 			llm,
 			onProgress: progressWriter(prefix),
 			...seams.retry ? { retry: seams.retry } : {}
 		});
+		file.result = result;
 		if (process.stderr.isTTY) process.stderr.write("\n");
+		else writeFileDone(label, result);
 	}
 	const batch = summarizeBatch(files);
 	const payload = JSON.stringify(batch, null, 2);
@@ -4684,7 +5069,7 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 	process.exitCode = batchPassed(batch) ? 0 : 1;
 }
 function registerEval(program) {
-	program.command("eval").description("Grade a file of golden answers against its quiz's real rubric and report the result").argument("<evalPathOrUrl...>", "one or more eval YAML files (paths, http(s)/file URLs, or a quoted glob pattern)").option("--server <url>", "Novedu server base URL (defaults to the NOVEDU_SERVER env var, then production)").option("--concurrency <n>", "grading calls in flight per file", String(CONCURRENCY_DEFAULT)).option("--repeats <n>", "grade every answer N times and take the majority verdict", "1").option("--llm-provider <provider>", "grade with this provider instead of the quiz's (\"SCCH\" or \"Azure Foundry\"; needs --llm-model)").option("--llm-model <model>", "grade with this model instead of the quiz's (needs --llm-provider)").option("--json", "print the machine-readable batch report on stdout").option("--out <file>", "additionally write the machine-readable batch report to a file").option("--report <file>", "additionally write a readable Markdown report to a file").addHelpText("after", `
+	program.command("eval").description("Grade a file of golden answers against its quiz's real rubric and report the result").argument("<evalPathOrUrl...>", "one or more eval YAML files (paths, http(s)/file URLs, or a quoted glob pattern)").option("--server <url>", "Novedu server base URL (defaults to the NOVEDU_SERVER env var, then production)").option("--concurrency <n>", "grading calls in flight per file", String(CONCURRENCY_DEFAULT)).option("--repeats <n>", "grade every answer N times and take the majority verdict", "1").option("--llm-provider <provider>", "grade with this provider instead of the quiz's (\"SCCH\" or \"Azure Foundry\"; needs --llm-model)").option("--llm-model <model>", "grade with this model instead of the quiz's (needs --llm-provider)").option("--no-judge-feedback", "skip the LLM audit of the grader's feedback (halves the LLM calls)").option("--judge-llm-provider <provider>", "judge the feedback with this provider (\"SCCH\" or \"Azure Foundry\"; needs --judge-llm-model)").option("--judge-llm-model <model>", "judge the feedback with this model instead of the grading one (needs --judge-llm-provider)").option("--json", "print the machine-readable batch report on stdout").option("--out <file>", "additionally write the machine-readable batch report to a file").option("--report <file>", "additionally write a readable Markdown report to a file").addHelpText("after", `
 Examples:
   # Evaluate one quiz's golden answers
   $ novedu-cli eval ./0010-welcome-quiz.eval.yaml
@@ -4698,6 +5083,12 @@ Examples:
   # How would this rubric perform on another model? (both flags, always together)
   $ novedu-cli eval ./my-quiz.eval.yaml --llm-provider "Azure Foundry" --llm-model gpt-5-mini
 
+  # A strong judge over the quiz's own grader — the recommended pairing
+  $ novedu-cli eval ./my-quiz.eval.yaml --judge-llm-provider "Azure Foundry" --judge-llm-model gpt-5.6-terra
+
+  # Half the LLM calls: check the verdicts only, skip the feedback audit
+  $ novedu-cli eval ./my-quiz.eval.yaml --no-judge-feedback
+
   # Machine-readable, for CI
   $ novedu-cli eval ./my-quiz.eval.yaml --json --out eval-report.json
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@novedu/cli",
-  "version": "0.20.0",
+  "version": "0.22.0",
   "description": "Command-line companion for the Novedu chat app. Validates tutor, fragment, quiz, writing, coding and eval YAML definitions, dumps the exact LLM prompts an activity produces, and evaluates a quiz's grading rubric against golden answers; signs in with Entra ID and manages codes, app-hosted files and images over the app's API.",
   "type": "module",
   "repository": {