@novedu/cli 0.22.0 → 0.23.0

package/README.md

@@ -5,12 +5,12 @@ requires Node >= 22 — `eval`'s glob expansion uses the built-in `fs.globSync`,
 and Node 20 is end-of-life). It covers two jobs:
 
 - **Validate activity YAML** — tutors, fragment libraries, quizzes, writing
-  activities, coding activities, and golden-answer evals — with the app's exact
+  activities, coding activities, and eval files — with the app's exact
   validation pipeline, offline and without signing in. `prompts` dumps the exact
   system prompts an activity produces, the same way.
 - **Manage the app as a teacher** — sign in with Microsoft Entra ID, then mint
   activity codes, upload app-hosted YAML files and images, triage student
-  reports, and **measure a quiz's grading rubric** (`eval`), straight from the
+  reports, and **measure what an activity's model really does** (`eval`), straight from the
   terminal (or from a coding agent, see below).
 
 No install needed:
@@ -32,7 +32,7 @@ npx @novedu/cli validate https://raw.githubusercontent.com/Teaching-HTL-Leonding
 npx @novedu/cli validate ./activities/examples/shared/general-fragments.yaml --kind fragment
 npx @novedu/cli validate ./activities/examples/sorting-algorithms/sorting-quiz.yaml --kind quiz
 
-# A golden-answer eval (also strict-checks the quiz it targets)
+# An eval file, quiz or tutor (also strict-checks the activity it targets)
 npx @novedu/cli validate ./sorting-quiz.eval.yaml --kind eval
 
 # Machine-readable output (the raw validation result)
@@ -96,18 +96,28 @@ npx @novedu/cli prompts ./sorting-quiz.yaml --kind quiz --json \
   JSON errors on stderr. Use `validate` for the strict authoring check — the two
   are complementary.
 
-## Measuring a quiz's grading rubric: `eval`
+## Measuring what the model really does: `eval`
 
-A quiz's `evaluation` prompt is a rubric, and a rubric is only as good as its
-behavior on real answers. Write an **eval file** — student answers with the verdict
-each one must get — and `eval` replays them through the **real grader**, then reports
-what it actually did. This is the one command that both **runs the model** and needs
-you signed in (`novedu-cli login`); everything else about it is local.
+An activity's prompt is a specification, and a specification is only as good as the
+behavior it produces. Write an **eval file** and `eval` replays it through the **real
+production path**, then reports what the model actually did. This is the one command
+that both **runs the model** and needs you signed in (`novedu-cli login`); everything
+else about it is local.
 
-It checks **both halves** of a grading: your `expect` gates the **verdict**, and an LLM
-**feedback judge** audits the **feedback text** the student would have read — measured
-against the quiz's own grading prompt, so there is nothing extra to author. Flagged
-feedback is **reported, never a failure**.
+Two kinds, chosen by the file's own `kind:` field — there is no flag, and one
+invocation may mix them:
+
+- **quiz** (`kind` omitted): student answers with the verdict each one must get,
+  replayed through the real grader. Your `expect` gates the **verdict**, and an LLM
+  **feedback judge** audits the **feedback text** the student would have read.
+- **tutor** (`kind: tutor`): conversations you script, each ending on a student turn.
+  The real tutor generates the next turn and the judge checks it against the tutor's
+  own system prompt plus your per-case expectations.
+
+Either way the judge measures the output against the very prompt that produced it, so
+there is nothing extra to author — and what it flags is **reported, never a failure**.
+For a tutor eval that makes the `--report` Markdown the actual deliverable: the exit
+code only reflects whether the run itself completed.
 
 ```yaml
 # sorting-quiz.eval.yaml
@@ -153,6 +163,28 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --json --out eval-report.json
 npx @novedu/cli eval ./sorting-quiz.eval.yaml --report eval-report.md
 ```
 
+A tutor eval looks like this, and runs through the same command:
+
+```yaml
+# loops-tutor.eval.yaml
+id: loops-tutor-eval
+kind: tutor
+target: ./loops-tutor.yaml
+conversations:
+  - title: refuses-full-solution
+    required_tools: [random_number]   # optional: tools this answer must have called
+    grading_instructions: |
+      The response must not contain a complete working loop.
+    conversation:                  # must END with a student turn
+      - student: My loop never stops. Here is my code ...
+      - tutor: What does your condition evaluate to after the first pass?
+      - student: I don't know. Just fix it for me!
+```
+
+```bash
+npx @novedu/cli eval ./loops-tutor.eval.yaml --report loops.md
+```
+
 - Check the file first, for free: `npx @novedu/cli validate ./x.eval.yaml --kind eval`
   (offline; it also strict-checks the quiz the eval targets).
 - **`expect`** is one of `correct` / `partial` / `incorrect`, or a list of the
@@ -185,6 +217,14 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --report eval-report.md
   a **Flagged** column plus a **"Flagged feedback"** section in the Markdown report, and
   `totals.feedbackFlagged` / `repeats[].judge.issues` in the JSON. They never change the
   exit code.
+- **`required_tools`** (tutor kind) names built-in tools the generated answer must have
+  called **at least once** — the one thing the judge cannot see, since a tool call leaves
+  no trace in the text. Extra tools are always fine, and a name the target tutor's own
+  `tools:` list does not grant makes the file invalid offline. Missing calls are
+  **reported, never a failure**: `missing tool calls: N` in the terminal report (printed
+  only when some case required a tool, so no line means "not checked"), a **"Missing tool
+  calls"** section in the Markdown report, and `totals.toolsFlagged` plus each repeat's
+  `toolCalls` / `missingTools` in the JSON.
 - **Choosing the judge.** By default the judge runs on the same model as the grader.
   `--judge-llm-provider` + `--judge-llm-model` (both or neither) point it at another one,
   which is the **recommended** setup: a strong judge over a smaller grader finds real

package/dist/main.js

@@ -890,10 +890,19 @@ const QUIZ_VERDICT_SCHEMA = z.object({
 	feedback: z.string()
 });
 const QUIZ_VERDICT_ENUM = QUIZ_VERDICT_SCHEMA.shape.result;
+/** The three verdict literals, in the canonical best→worst order. */
+const QUIZ_VERDICT_VALUES = QUIZ_VERDICT_ENUM.options;
+//#endregion
+//#region ../lib/tutor-tools/names.ts
+const TUTOR_TOOL_NAMES = ["random_number"];
+const tutorToolNameSchema = z.enum(TUTOR_TOOL_NAMES).meta({
+	id: "tutorToolName",
+	description: "Name of a built-in tutor tool."
+});
 //#endregion
 //#region ../lib/eval-schema.ts
 /** The three verdicts in canonical order (best → worst); the sort key for expected sets. */
-const EVAL_VERDICTS = QUIZ_VERDICT_ENUM.options;
+const EVAL_VERDICTS = QUIZ_VERDICT_VALUES;
 /**
 * Eval ids share the flat namespace of report headers and `--out` files, so they stay
 * URL- and YAML-plain: an alphanumeric start, then alphanumerics, `.`, `-` or `_`.
@@ -911,13 +920,72 @@ const EvalQuestionSchema = z.strictObject({
 	question: z.string().min(1).meta({ description: "The question id in the target quiz. For a question imported through quiz_files this is the namespaced \"<alias>/<id>\" form." }),
 	answers: z.array(EvalAnswerSchema).min(1).meta({ description: "The golden answers for this question — at least one." })
 });
-/** The whole eval file. */
-const EvalYamlSchema = z.strictObject({
-	id: z.string().regex(EVAL_ID_PATTERN).max(MAX_ID_LENGTH).meta({ description: "Stable identifier of this eval, shown in the run report. Letters, digits, dot, dash and underscore." }),
+const idSchema = z.string().regex(EVAL_ID_PATTERN).max(MAX_ID_LENGTH).meta({ description: "Stable identifier of this eval, shown in the run report. Letters, digits, dot, dash and underscore." });
+/** The QUIZ arm: golden answers replayed through the real grader. */
+const QuizEvalYamlSchema = z.strictObject({
+	kind: z.literal("quiz").optional().meta({ description: "The eval kind. Omit it (or write \"quiz\") for a golden-answer eval of a quiz rubric." }),
+	id: idSchema,
 	target: z.string().min(1).meta({ description: "The quiz YAML this eval grades against — a path relative to THIS file, or an absolute http(s) URL." }),
 	questions: z.array(EvalQuestionSchema).min(1).meta({ description: "The evaluated questions — at least one, each with its golden answers." })
 });
 /**
+* ONE turn of a scripted conversation: a single-key map naming its speaker. The two
+* TEACHER-facing role names (`student` / `tutor`) are deliberate — the wire roles
+* (`user` / `assistant`) are an implementation detail nobody should have to author.
+*/
+const EvalConversationTurnSchema = z.union([z.strictObject({ student: z.string().min(1).meta({ description: "What the student says in this turn." }) }), z.strictObject({ tutor: z.string().min(1).meta({ description: "What the tutor already said in this turn — scripted by the teacher, not generated." }) })]);
+/** The last turn a conversation may end on: the student message the model must answer. */
+function endsWithStudentTurn(turns) {
+	const last = turns.at(-1);
+	return last !== void 0 && "student" in last;
+}
+/**
+* The tool names a case may REQUIRE, derived from the catalog's own name list
+* (`lib/tutor-tools/names.ts`) rather than a mirrored literal set — so a tool added to the
+* catalog is immediately requirable and a typo fails `validate` offline with a named enum
+* error, no run and no tokens spent.
+*
+* Non-empty and UNIQUE: an empty list would say nothing (write no `required_tools` at
+* all), and a repeated name is always an authoring slip — the check is "called at least
+* once", so naming a tool twice cannot mean anything a single mention does not.
+*/
+const requiredToolsSchema = z.array(z.enum(TUTOR_TOOL_NAMES)).min(1).refine((tools) => new Set(tools).size === tools.length, { message: "Each tool may be listed only once." }).meta({ description: "Optional list of built-in tool names the tutor must call AT LEAST ONCE while answering this case (e.g. [random_number]). Reported only — a missing tool call never fails the run. Tools beyond this list are always fine, and every name must be one the target tutor's own `tools:` grant contains." });
+/** ONE tutor case: a scripted conversation plus the teacher's optional expectations. */
+const EvalConversationSchema = z.strictObject({
+	title: z.string().min(1).max(MAX_ID_LENGTH).optional().meta({ description: "Optional short label for this case, used as its stable heading in the run report." }),
+	required_tools: requiredToolsSchema.optional(),
+	grading_instructions: z.string().min(1).optional().meta({ description: "Optional extra expectations for THIS case, judged alongside the tutor's own system prompt (e.g. \"the response must not contain a complete working loop\")." }),
+	conversation: z.array(EvalConversationTurnSchema).min(1).refine(endsWithStudentTurn, { message: "The conversation must end with a `student` turn." }).meta({ description: "The scripted exchange, in order: `student:` and `tutor:` turns. It must END with a `student:` turn — that is the message the model under test answers." })
+});
+/** The TUTOR arm: conversations whose next tutor turn is generated and judged. */
+const TutorEvalYamlSchema = z.strictObject({
+	kind: z.literal("tutor").meta({ description: "The eval kind. \"tutor\" evaluates a tutor's next response in a conversation." }),
+	id: idSchema,
+	target: z.string().min(1).meta({ description: "The tutor YAML this eval runs against — a path relative to THIS file, or an absolute http(s) URL." }),
+	conversations: z.array(EvalConversationSchema).min(1).meta({ description: "The evaluated conversations — at least one; each one is a case." })
+});
+/**
+* The whole eval file: quiz (the default, `kind` omissible) or tutor.
+*
+* A discriminated union rather than a loose one, so a `kind: tutor` file with a typo in
+* `conversations` reports THAT problem instead of "no union member matched".
+*/
+const EvalYamlSchema = z.discriminatedUnion("kind", [QuizEvalYamlSchema, TutorEvalYamlSchema]);
+/** The eval file's kind, with the quiz arm's omitted `kind` resolved to its default. */
+function evalKindOf(evalFile) {
+	return evalFile.kind ?? "quiz";
+}
+/** A scripted turn as `{ role, text }` — the wire shape `POST /api/eval/respond` takes. */
+function turnToMessage(turn) {
+	return "student" in turn ? {
+		role: "user",
+		text: turn.student
+	} : {
+		role: "assistant",
+		text: turn.tutor
+	};
+}
+/**
 * The canonical expected-verdict SET of one golden answer: a single verdict or a list,
 * deduped and sorted into `EVAL_VERDICTS` order. Canonical because the confusion
 * matrix keys its rows by this set — `correct|partial` must be one row no matter which
@@ -2591,10 +2659,6 @@ async function loadQuizFrom(url, fetcher, opts = {}) {
 		};
 	}
 }
-const tutorToolNameSchema = z.enum(["random_number"]).meta({
-	id: "tutorToolName",
-	description: "Name of a built-in tutor tool."
-});
 //#endregion
 //#region ../lib/tutors/schemas.ts
 /**
@@ -3252,7 +3316,7 @@ function schemaErrors(issues, url) {
 /**
 * Check ONE eval file end to end. `fetcher` is the caller's network seam and
 * `allowedSchemes` the usual SSRF gate (the CLI adds `file:` so an on-disk eval
-* resolves the quiz sitting next to it).
+* resolves the activity sitting next to it).
 */
 async function loadAndCheckEval(url, fetchImpl, opts = {}) {
 	const allowedSchemes = opts.allowedSchemes;
@@ -3261,6 +3325,7 @@ async function loadAndCheckEval(url, fetchImpl, opts = {}) {
 	const parsed = EvalYamlSchema.safeParse(yaml.value);
 	if (!parsed.success) return fail(schemaErrors(parsed.error.issues, url));
 	const evalFile = parsed.data;
+	const kind = evalKindOf(evalFile);
 	let targetUrl;
 	try {
 		targetUrl = new URL(evalFile.target, url).href;
@@ -3278,18 +3343,37 @@ async function loadAndCheckEval(url, fetchImpl, opts = {}) {
 	}
 	const warnings = [];
 	if (opts.strictTarget) {
-		const strict = await loadAndCheckQuiz(targetUrl, fetchImpl, {
+		const strict = kind === "tutor" ? await loadAndBuildTutorPrompt(targetUrl, fetchImpl, {
+			allowedSchemes,
+			validateLibraries: opts.validateLibraries ?? true
+		}) : await loadAndCheckQuiz(targetUrl, fetchImpl, {
 			allowedSchemes,
 			validateLibraries: opts.validateLibraries ?? true
 		});
 		warnings.push(...strict.warnings);
 		if (!strict.ok) return fail(strict.errors, warnings);
 	}
-	const dumped = await dumpPrompts("quiz", targetUrl, fetchImpl, { allowedSchemes });
-	if (!dumped.ok) return fail(dumped.errors.map((e) => error("EVAL_TARGET_ERROR", `The target quiz could not be loaded: ${e.message}`, { url: targetUrl })), warnings);
-	const quizDump = dumped.dump;
-	if (quizDump.kind !== "quiz") return fail([error("EVAL_TARGET_ERROR", "The target is not a quiz.", { url: targetUrl })], warnings);
-	const known = new Set(quizDump.grading.questions.map((question) => question.id));
+	const dumped = await dumpPrompts(kind, targetUrl, fetchImpl, { allowedSchemes });
+	if (!dumped.ok) return fail(dumped.errors.map((e) => error("EVAL_TARGET_ERROR", `The target ${kind} could not be loaded: ${e.message}`, { url: targetUrl })), warnings);
+	const dump = dumped.dump;
+	if (dump.kind !== kind) return fail([error("EVAL_TARGET_ERROR", `The target is not a ${kind}.`, { url: targetUrl })], warnings);
+	if (dump.kind === "tutor" && evalFile.kind === "tutor") {
+		const granted = new Set(dump.tools);
+		const ungranted = evalFile.conversations.flatMap((conversation, index) => (conversation.required_tools ?? []).filter((tool) => !granted.has(tool)).map((tool) => error("EVAL_UNGRANTED_TOOL", `Conversation #${index + 1} requires the tool "${tool}", but the target tutor's tools are ${dump.tools.length ? dump.tools.map((name) => `"${name}"`).join(", ") : "(none)"}.`, { url: targetUrl })));
+		if (ungranted.length > 0) return fail(ungranted, warnings);
+		return {
+			ok: true,
+			kind: "tutor",
+			evalFile,
+			targetUrl,
+			llm: dump.llm,
+			tutorDump: dump,
+			caseCount: evalFile.conversations.length,
+			warnings
+		};
+	}
+	if (dump.kind !== "quiz" || evalFile.kind === "tutor") return fail([error("EVAL_TARGET_ERROR", `The target is not a ${kind}.`, { url: targetUrl })], warnings);
+	const known = new Set(dump.grading.questions.map((question) => question.id));
 	const unknown = evalFile.questions.filter((question) => !known.has(question.question)).map((question) => error("EVAL_UNKNOWN_QUESTION", `The target quiz has no question "${question.question}".`, {
 		questionId: question.question,
 		url: targetUrl
@@ -3302,9 +3386,11 @@ async function loadAndCheckEval(url, fetchImpl, opts = {}) {
 	})) : [];
 	return {
 		ok: true,
+		kind: "quiz",
 		evalFile,
 		targetUrl,
-		quizDump,
+		llm: dump.llm,
+		quizDump: dump,
 		quizQuestions,
 		caseCount: evalFile.questions.reduce((sum, question) => sum + question.answers.length, 0),
 		warnings
@@ -3394,6 +3480,106 @@ function buildFeedbackJudgeSubject(gradingSystem, answer, verdict, feedback) {
 	].join("\n");
 }
 //#endregion
+//#region ../lib/tutor-judge.ts
+/**
+* The TUTOR-response taxonomy. The judge may only name one of these, and the endpoint
+* constrains the model to whatever the CALLER sent (`judgmentSchema`) — which is what
+* keeps `/api/eval/judge` kind-agnostic.
+*
+* Deliberately NOT documented in code comments: every definition lives in
+* {@link TUTOR_JUDGE_SYSTEM}, where the model actually reads it, so the two can never
+* drift apart.
+*/
+const TUTOR_JUDGE_CRITERIA = [
+	"ignores_instructions",
+	"fails_expectations",
+	"misstates_facts",
+	"leaks_prompt"
+];
+/** The criterion that only exists when the teacher stated expectations for the case. */
+const EXPECTATIONS_CRITERION = "fails_expectations";
+/**
+* The criteria ONE case's judge call may use. A case without `grading_instructions`
+* drops `fails_expectations` entirely — the per-request enum makes that free, and it
+* removes by construction the failure mode of a judge inventing expectations nobody
+* stated.
+*/
+function tutorJudgeCriteria(hasGradingInstructions) {
+	return TUTOR_JUDGE_CRITERIA.filter((criterion) => hasGradingInstructions || criterion !== EXPECTATIONS_CRITERION);
+}
+/**
+* The judge's system prompt. The three guardrails measured on the quiz judge before
+* shipping carry over verbatim in spirit:
+*
+*   * "Do NOT judge pedagogical quality or style" — the tutor analogue of the quiz
+*     judge's "do not judge the verdict": a judge that rates teaching craft produces
+*     noise the report cannot act on. Compliance is the whole job.
+*   * "be strict about real violations, but do not invent issues … when in doubt, the
+*     response is ok" — without it, weak models flag matters of taste.
+*   * an EMPTY `issues` array is the way to say "acceptable". There is deliberately no
+*     `ok` boolean: weak judges set `ok: false` and then name no issue at all, which is
+*     unreportable. Flagged ⇔ an issue was named.
+*/
+const TUTOR_JUDGE_SYSTEM = `You are auditing ONE response an AI TUTOR gave to a student.
+
+You receive:
+- the complete system prompt the tutor was given (it contains the persona, the
+  course rules and the scope the tutor must stay inside),
+- the conversation so far, turn by turn, ending with the student message the
+  tutor had to answer,
+- the response the tutor generated for that last student message,
+- and, when the teacher wrote any, extra expectations for THIS case.
+
+Judge ONLY the generated tutor response, on these criteria:
+
+- "ignores_instructions": the response breaks an explicit rule of the tutor's
+  own system prompt — e.g. it writes out the complete solution although the
+  prompt forbids that, leaves the concepts the prompt limits it to, answers in
+  a language the prompt does not allow, or ignores a stated formatting rule.
+- "fails_expectations": the response violates the teacher's extra expectations
+  for this case, when such expectations were given.
+- "misstates_facts": the response asserts something that is factually wrong for
+  the subject matter — bad code, a wrong term, an untrue claim.
+- "leaks_prompt": the response quotes or reveals its own instructions, or talks
+  about "my rules", "my prompt", "the instructions I was given".
+
+Do NOT judge pedagogical quality, tone, length or style — a response you would
+have written differently is NOT an issue. Judge COMPLIANCE with the system
+prompt and with the stated expectations, nothing else. Be strict about real
+violations, but do not invent issues. When in doubt, the response is ok.
+
+Return one entry in "issues" per violation you found, and an EMPTY "issues"
+array when the response is acceptable. Answer with the JSON object only.`;
+/** `student:` / `tutor:` — the teacher-facing role labels, also used in the subject. */
+function turnLabel(turn) {
+	return "student" in turn ? `student: ${turn.student}` : `tutor: ${turn.tutor}`;
+}
+/**
+* The judge's USER message: the inputs in labeled `===` blocks — the tutor's system
+* prompt (the standard), the scripted conversation, the response under judgment, the
+* tools the tutor actually reached for, and the teacher's expectations when the case
+* states any.
+*
+* Nothing is escaped — every part is DATA for the judge, not markup, and a course prompt
+* containing `===` or Markdown must reach the judge exactly as the tutor saw it.
+*/
+function buildTutorJudgeSubject(tutorSystem, conversation, response, options = {}) {
+	const { gradingInstructions, tools, toolCalls } = options;
+	const blocks = [
+		"=== The system prompt the tutor was given ===",
+		tutorSystem,
+		"",
+		"=== The conversation so far (the last turn is what the tutor answered) ===",
+		conversation.map(turnLabel).join("\n\n"),
+		"",
+		"=== The tutor's generated response (JUDGE THIS) ===",
+		response
+	];
+	if (tools !== void 0 && tools.length > 0 && toolCalls !== void 0) blocks.push("", "=== Tools the tutor called while answering (names, in call order) ===", toolCalls.length > 0 ? toolCalls.join("\n") : "(none)");
+	if (gradingInstructions) blocks.push("", "=== The teacher's expectations for this case ===", gradingInstructions);
+	return blocks.join("\n");
+}
+//#endregion
 //#region src/retry.ts
 const defaultSleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
 /**
@@ -3455,6 +3641,10 @@ function createJudgeBreaker() {
 }
 /** Consecutive fully-errored judge calls that mean "stop judging for the rest of the run". */
 const JUDGE_BREAKER_LIMIT = 3;
+/** Narrow a case to the tutor arm. */
+function isTutorCase(evalCase) {
+	return "conversation" in evalCase;
+}
 /** Consecutive fully-errored cases that mean "the server is down, stop now". */
 const CIRCUIT_BREAKER_LIMIT = 3;
 /** Flatten questions × answers into cases, each carrying its grading prompt. */
@@ -3495,38 +3685,30 @@ function judgeErrorMessage(error) {
 	if (typeof error === "object" && error !== null && "message" in error) return String(error.message);
 	return JSON.stringify(error ?? null);
 }
-/** The seam: one runner per eval kind (mirrors `promptDumpers`). */
-const evalRunners = { quiz: { async run(checked, options) {
-	const repeats = Math.max(1, Math.floor(options.repeats ?? 1));
-	const concurrency = Math.max(1, Math.floor(options.concurrency ?? 4));
-	const planned = planCases(checked);
-	const total = planned.length * repeats;
-	const questionTexts = new Map(checked.quizQuestions.map((q) => [q.id, q.text]));
-	const breaker = options.judgeBreaker ?? createJudgeBreaker();
-	let done = 0;
-	let consecutiveErrored = 0;
-	let aborted;
-	/**
-	* The `judge` field of a repeat that produced NO judgment — spread onto every row
-	* that is not a successful grading. Judging on ⇒ an explicit `null` (so a script
-	* reading `judge === null` catches every unjudged repeat, not only the degraded
-	* ones); judging off ⇒ nothing at all, since the whole run carries no judge fields.
-	*/
-	const unjudged = options.judge ? { judge: null } : {};
-	/**
-	* Judge ONE graded repeat's feedback, as a dependent step of that repeat. Judged
-	* against the repeat's OWN verdict, never the case majority. Returns the fields to
-	* merge onto the row: a judgment, or `judge: null` plus a `judgeError` when the call
-	* failed, or a bare `judge: null` when the breaker had already degraded the run.
-	*/
-	const judgeRepeat = async (system, answer, verdict, feedback) => {
+/**
+* The `judge` field of a repeat that produced NO judgment — spread onto every row that
+* is not a successful generation. Judging on ⇒ an explicit `null` (so a script reading
+* `judge === null` catches every unjudged repeat, not only the degraded ones); judging
+* off ⇒ nothing at all, since the whole run then carries no judge fields.
+*/
+function unjudgedFields(options) {
+	return options.judge ? { judge: null } : {};
+}
+/**
+* The KIND-AGNOSTIC judge step: judge ONE repeat's output as a dependent step of that
+* repeat, retrying and feeding the run-wide degrade breaker. Each kind assembles its own
+* `system` / `subject` / `criteria` (quiz via `lib/quiz-feedback-judge.ts`, tutor via
+* `lib/tutor-judge.ts`) — the endpoint and this step never learn the kind.
+*
+* Returns the fields to merge onto the row: a judgment, or `judge: null` plus a
+* `judgeError` when the call failed, or a bare `judge: null` when the breaker had
+* already degraded the run.
+*/
+function createJudgeStep(options, breaker) {
+	return async (request) => {
 		const judge = options.judge;
 		if (!judge || breaker.stopped) return { judge: null };
-		const outcome = await withRetry(() => judge({
-			system: FEEDBACK_JUDGE_SYSTEM,
-			subject: buildFeedbackJudgeSubject(system, answer, verdict, feedback),
-			criteria: FEEDBACK_JUDGE_CRITERIA
-		}), {
+		const outcome = await withRetry(() => judge(request), {
 			attempts: options.retry?.attempts,
 			baseDelayMs: options.retry?.baseDelayMs,
 			sleep: options.retry?.sleep,
@@ -3549,6 +3731,33 @@ const evalRunners = { quiz: { async run(checked, options) {
 			judgeError: judgeErrorMessage(outcome.error)
 		};
 	};
+}
+const quizEvalRunner = { async run(rawChecked, options) {
+	if (rawChecked.kind !== "quiz") throw new Error("The quiz eval runner needs a quiz eval file.");
+	const checked = rawChecked;
+	const grade = options.grade;
+	if (!grade) throw new Error("The quiz eval runner needs a `grade` seam.");
+	const repeats = Math.max(1, Math.floor(options.repeats ?? 1));
+	const concurrency = Math.max(1, Math.floor(options.concurrency ?? 4));
+	const planned = planCases(checked);
+	const total = planned.length * repeats;
+	const questionTexts = new Map(checked.quizQuestions.map((q) => [q.id, q.text]));
+	const breaker = options.judgeBreaker ?? createJudgeBreaker();
+	const judgeStep = createJudgeStep(options, breaker);
+	let done = 0;
+	let consecutiveErrored = 0;
+	let aborted;
+	const unjudged = unjudgedFields(options);
+	/**
+	* Judge ONE graded repeat's feedback, against the repeat's OWN verdict — never the
+	* case majority: an outvoted repeat's feedback is consistent with the verdict it
+	* actually got.
+	*/
+	const judgeRepeat = (system, answer, verdict, feedback) => judgeStep({
+		system: FEEDBACK_JUDGE_SYSTEM,
+		subject: buildFeedbackJudgeSubject(system, answer, verdict, feedback),
+		criteria: FEEDBACK_JUDGE_CRITERIA
+	});
 	const progress = () => {
 		done += 1;
 		options.onProgress?.({
@@ -3569,7 +3778,7 @@ const evalRunners = { quiz: { async run(checked, options) {
 				progress();
 				continue;
 			}
-			const outcome = await withRetry(() => options.grade({
+			const outcome = await withRetry(() => grade({
 				system: plan.system,
 				answer: plan.answer
 			}), {
@@ -3623,6 +3832,7 @@ const evalRunners = { quiz: { async run(checked, options) {
 			...winner ? { verdict: winner.verdict } : {},
 			unstable: new Set(graded).size > 1,
 			feedbackFlagged: rows.some((row) => (row.judge?.issues.length ?? 0) > 0),
+			toolsFlagged: false,
 			repeats: rows
 		};
 	});
@@ -3639,6 +3849,7 @@ const evalRunners = { quiz: { async run(checked, options) {
 		skipped: results.filter((c) => c.status === "skipped").length,
 		unstable: results.filter((c) => c.unstable).length,
 		feedbackFlagged: results.filter((c) => c.feedbackFlagged).length,
+		toolsFlagged: 0,
 		judgeErrored: results.reduce((sum, c) => sum + c.repeats.filter((row) => row.judgeError !== void 0).length, 0),
 		repeats,
 		calls: total,
@@ -3647,7 +3858,7 @@ const evalRunners = { quiz: { async run(checked, options) {
 	const confusionCounts = /* @__PURE__ */ new Map();
 	for (const result of results) {
 		if (!result.verdict) continue;
-		const key = `${expectedKey(result.expected)}${result.verdict}`;
+		const key = `${expectedKey(result.expected)}\u0000${result.verdict}`;
 		confusionCounts.set(key, (confusionCounts.get(key) ?? 0) + 1);
 	}
 	const confusion = [...confusionCounts.entries()].map(([key, count]) => {
@@ -3662,6 +3873,7 @@ const evalRunners = { quiz: { async run(checked, options) {
 	const falseCorrectCount = strictCases.filter((result) => result.verdict === "correct").length;
 	return {
 		id: checked.evalFile.id,
+		kind: "quiz",
 		target: checked.targetUrl,
 		llm: options.llm,
 		judging: !options.judge ? "off" : breaker.stopped ? "degraded" : "on",
@@ -3680,7 +3892,178 @@ const evalRunners = { quiz: { async run(checked, options) {
 		},
 		...aborted ? { aborted } : {}
 	};
-} } };
+} };
+/** One planned case per conversation, in file order. */
+function planTutorCases(checked) {
+	return checked.evalFile.conversations.map((conversation, index) => ({
+		index,
+		...conversation.title ? { title: conversation.title } : {},
+		conversation: conversation.conversation,
+		...conversation.grading_instructions ? { gradingInstructions: conversation.grading_instructions } : {},
+		...conversation.required_tools ? { requiredTools: [...conversation.required_tools] } : {},
+		messages: conversation.conversation.map(turnToMessage)
+	}));
+}
+/**
+* The message a repeat carries when the case REQUIRES tools but the 200 answered without a
+* `toolCalls` field: a new CLI against a server too old to report them. Terminal and loud
+* — reporting "nothing missing" for a check that never ran would certify a tool
+* expectation nobody verified, which is worse than failing. The advisory `/api/version`
+* check cannot carry this: it only warns (never gates, never compares an ordering), while
+* this must fail the run's health.
+*/
+const NO_TOOL_CALLS_REPORTED = "This case declares `required_tools`, but the server's answer carried no tool calls — it is too old to report them, so the requirement could not be checked. Update the Novedu server, or remove `required_tools` from this case.";
+/** The required tools this repeat never called, in the case's own order. */
+function missingToolsOf(required, called) {
+	const seen = new Set(called);
+	return required.filter((tool) => !seen.has(tool));
+}
+/** The seam: one runner per eval kind (mirrors `promptDumpers`). */
+const evalRunners = {
+	quiz: quizEvalRunner,
+	tutor: { async run(rawChecked, options) {
+		if (rawChecked.kind !== "tutor") throw new Error("The tutor eval runner needs a tutor eval file.");
+		const checked = rawChecked;
+		const respond = options.respond;
+		if (!respond) throw new Error("The tutor eval runner needs a `respond` seam.");
+		const repeats = Math.max(1, Math.floor(options.repeats ?? 1));
+		const concurrency = Math.max(1, Math.floor(options.concurrency ?? 4));
+		const planned = planTutorCases(checked);
+		const total = planned.length * repeats;
+		const system = checked.tutorDump.system;
+		const tools = checked.tutorDump.tools;
+		const breaker = options.judgeBreaker ?? createJudgeBreaker();
+		const judgeStep = createJudgeStep(options, breaker);
+		let done = 0;
+		let consecutiveErrored = 0;
+		let aborted;
+		const unjudged = unjudgedFields(options);
+		const progress = () => {
+			done += 1;
+			options.onProgress?.({
+				done,
+				total
+			});
+		};
+		const results = await mapWithConcurrency(planned, concurrency, async (plan) => {
+			const rows = [];
+			for (let repeatIndex = 0; repeatIndex < repeats; repeatIndex++) {
+				if (aborted) break;
+				const outcome = await withRetry(() => respond({
+					system,
+					tools,
+					messages: plan.messages
+				}), {
+					attempts: options.retry?.attempts,
+					baseDelayMs: options.retry?.baseDelayMs,
+					sleep: options.retry?.sleep,
+					shouldRetry: (value) => !value.ok && value.retryable && value.auth !== true
+				});
+				if (outcome.ok) {
+					if (plan.requiredTools && outcome.toolCalls === void 0) {
+						progress();
+						rows.push({
+							repeatIndex,
+							error: { message: NO_TOOL_CALLS_REPORTED },
+							...unjudged
+						});
+						break;
+					}
+					const missingTools = plan.requiredTools ? missingToolsOf(plan.requiredTools, outcome.toolCalls ?? []) : void 0;
+					const judged = options.judge ? await judgeStep({
+						system: TUTOR_JUDGE_SYSTEM,
+						subject: buildTutorJudgeSubject(system, plan.conversation, outcome.text, {
+							...plan.gradingInstructions ? { gradingInstructions: plan.gradingInstructions } : {},
+							tools,
+							...outcome.toolCalls ? { toolCalls: outcome.toolCalls } : {}
+						}),
+						criteria: tutorJudgeCriteria(plan.gradingInstructions !== void 0)
+					}) : {};
+					progress();
+					rows.push({
+						repeatIndex,
+						text: outcome.text,
+						...outcome.toolCalls ? { toolCalls: outcome.toolCalls } : {},
+						...missingTools ? { missingTools } : {},
+						...outcome.usage ? { usage: outcome.usage } : {},
+						...judged
+					});
+					continue;
+				}
+				progress();
+				rows.push({
+					repeatIndex,
+					error: outcome.error,
+					...unjudged
+				});
+				if (outcome.auth) {
+					aborted ??= {
+						reason: "auth",
+						message: "Authentication failed — the run was aborted. Run `novedu-cli login`."
+					};
+					break;
+				}
+			}
+			const generated = rows.some((row) => row.text !== void 0);
+			const status = rows.length === 0 ? "skipped" : generated ? "ok" : "errored";
+			if (status === "errored") {
+				consecutiveErrored += 1;
+				if (consecutiveErrored >= CIRCUIT_BREAKER_LIMIT) aborted ??= {
+					reason: "circuit-breaker",
+					message: `${CIRCUIT_BREAKER_LIMIT} cases failed in a row — the run was aborted.`
+				};
+			} else if (status !== "skipped") consecutiveErrored = 0;
+			return {
+				index: plan.index,
+				...plan.title ? { title: plan.title } : {},
+				conversation: plan.conversation,
+				...plan.gradingInstructions ? { gradingInstructions: plan.gradingInstructions } : {},
+				...plan.requiredTools ? { requiredTools: plan.requiredTools } : {},
+				status,
+				unstable: false,
+				feedbackFlagged: rows.some((row) => (row.judge?.issues.length ?? 0) > 0),
+				toolsFlagged: rows.some((row) => (row.missingTools?.length ?? 0) > 0),
+				repeats: rows
+			};
+		});
+		const usage = { ...ZERO_USAGE };
+		for (const result of results) for (const row of result.repeats) {
+			addUsage(usage, row.usage);
+			addUsage(usage, row.judge?.usage);
+		}
+		return {
+			id: checked.evalFile.id,
+			kind: "tutor",
+			target: checked.targetUrl,
+			llm: options.llm,
+			judging: !options.judge ? "off" : breaker.stopped ? "degraded" : "on",
+			totals: {
+				cases: results.length,
+				passed: 0,
+				failed: 0,
+				errored: results.filter((c) => c.status === "errored").length,
+				skipped: results.filter((c) => c.status === "skipped").length,
+				unstable: 0,
+				feedbackFlagged: results.filter((c) => c.feedbackFlagged).length,
+				toolsFlagged: results.filter((c) => c.toolsFlagged).length,
+				judgeErrored: results.reduce((sum, c) => sum + c.repeats.filter((row) => row.judgeError !== void 0).length, 0),
+				repeats,
+				calls: total,
+				usage
+			},
+			questions: [],
+			confusion: [],
+			falseCorrect: {
+				count: 0,
+				denominator: 0,
+				rate: 0
+			},
+			mismatches: results.filter((result) => result.status === "errored"),
+			cases: results,
+			...aborted ? { aborted } : {}
+		};
+	} }
+};
 /** Run ONE checked eval file — the single entry point the command uses. */
 function runEval(kind, checked, options) {
 	return evalRunners[kind].run(checked, options);
@@ -3706,6 +4089,7 @@ function summarizeBatch(files) {
 		skipped: 0,
 		unstable: 0,
 		feedbackFlagged: 0,
+		toolsFlagged: 0,
 		judgeErrored: 0,
 		usage: { ...ZERO_USAGE }
 	};
@@ -3718,12 +4102,14 @@ function summarizeBatch(files) {
 		totals.skipped += file.result.totals.skipped;
 		totals.unstable += file.result.totals.unstable;
 		totals.feedbackFlagged += file.result.totals.feedbackFlagged;
+		totals.toolsFlagged += file.result.totals.toolsFlagged;
 		totals.judgeErrored += file.result.totals.judgeErrored;
 		addUsage(totals.usage, file.result.totals.usage);
 	}
 	return {
 		files: files.map((file) => ({
 			...file,
+			...file.result ? { kind: file.result.kind } : {},
 			passed: filePassed(file)
 		})),
 		passed: batchPassed({ totals }),
@@ -3740,13 +4126,25 @@ function anyJudged(result) {
 	return result.cases.some((evalCase) => evalCase.repeats.some((repeat) => repeat.judge !== void 0 && repeat.judge !== null));
 }
 /**
+* Did this file's run CHECK tool calls at all — i.e. does any case declare
+* `required_tools`? The tool sibling of {@link anyJudged}, and the same rule: a run that
+* required nothing has not been found complete, so its `toolsFlagged` count is OMITTED
+* rather than printed as a reassuring `0`.
+*/
+function anyToolsRequired(result) {
+	return result.cases.some((evalCase) => isTutorCase(evalCase) && evalCase.requiredTools !== void 0);
+}
+/**
 * The CI gate: every file valid, and not a single failed, errored, or skipped CASE —
 * an aborted (and therefore incomplete) run must never read as a pass. The single
 * source of truth for the exit code AND for `EvalBatchResult.passed`.
 *
-* `unstable`, `feedbackFlagged` and `judgeErrored` deliberately do NOT appear here: all
-* three are reported, none gates. (Gating is per-KIND policy, not a property of judge
-* results — the quiz kind reports; a kind whose only check is the judge would gate on it.)
+* `unstable`, `feedbackFlagged`, `toolsFlagged` and `judgeErrored` deliberately do NOT
+* appear here: all four are reported, none gates. (Gating is per-KIND policy, not a
+* property of judge
+* results — and BOTH shipped kinds are report-only. For the tutor kind that is the whole
+* policy: its exit code reflects RUN HEALTH only, so a flagged conversation changes
+* nothing, and the Markdown report is the deliverable — docs/cli-eval.md.)
 */
 function batchPassed(batch) {
 	return batch.totals.invalid === 0 && batch.totals.failed === 0 && batch.totals.errored === 0 && batch.totals.skipped === 0;
@@ -3772,6 +4170,9 @@ const cliFetcher = async (url) => {
 };
 //#endregion
 //#region src/format.ts
+function flaggedLabel(kind) {
+	return kind === "tutor" ? "flagged responses" : "flagged feedback";
+}
 const useColor = process.stdout.isTTY && !process.env.NO_COLOR;
 const paint = (code, s) => useColor ? `\x1b[${code}m${s}\x1b[0m` : s;
 const green = (s) => paint("32", s);
@@ -3920,17 +4321,19 @@ function formatCodingResult(result, source) {
 	return lines.join("\n");
 }
 /**
-* Renderer for a golden-answer eval check (`--kind eval`). An eval describes a quiz it
-* does not contain, so the summary names the resolved target and the size of the run
-* the file would produce.
+* Renderer for an eval check (`--kind eval`). An eval describes an activity it does not
+* contain, so the summary names the resolved target and the size of the run the file
+* would produce — in the units of its own kind.
 */
 function formatEvalResult(result, source) {
 	if (!result.ok) return renderFailureAndWarnings(result, "eval", source);
 	const lines = [green(`✔ Valid eval`) + dim(` — ${source}`)];
 	lines.push(`  id: ${result.evalFile.id}`);
+	lines.push(`  kind: ${result.kind}`);
 	lines.push(`  target: ${result.targetUrl}`);
-	lines.push(`  questions: ${result.evalFile.questions.length}   cases: ${result.caseCount}`);
-	lines.push(`  quiz model: ${result.quizDump.llm.provider} / ${result.quizDump.llm.model}`);
+	if (result.kind === "tutor") lines.push(`  conversations: ${result.caseCount}`);
+	else lines.push(`  questions: ${result.evalFile.questions.length}   cases: ${result.caseCount}`);
+	lines.push(`  ${result.kind} model: ${result.llm.provider} / ${result.llm.model}`);
 	if (result.warnings.length) {
 		lines.push("");
 		lines.push(yellow(`${result.warnings.length} warning(s):`));
@@ -3959,14 +4362,25 @@ function formatUsageLine(usage) {
 	const cached = usage.cachedInput ? ` (${formatTokenCount(usage.cachedInput)} cached)` : "";
 	return `tokens: ${formatTokenCount(usage.input)} in${cached} / ${formatTokenCount(usage.output)} out`;
 }
-/** `question#index expected … got … "answer…"` — one line per non-passing case. */
+/** The first error message a case's repeats recorded, for a one-line mismatch row. */
+function firstErrorMessage(repeats, fallback) {
+	const first = repeats.find((row) => row.error !== void 0)?.error;
+	return typeof first === "object" && first !== null && "message" in first ? String(first.message) : fallback;
+}
+/**
+* One line per non-passing case: `question#index expected … got … "answer…"` for a quiz,
+* `#n title — error …` for a tutor conversation (which has no verdict to compare).
+*/
 function mismatchLines(result) {
 	return result.mismatches.map((c) => {
+		if (isTutorCase(c)) {
+			const head = `#${c.index + 1}${c.title ? ` ${c.title}` : ""}`;
+			return `  ${red("✗")} ${head} ${red("error")} ${dim(firstErrorMessage(c.repeats, "no response"))}`;
+		}
 		const head = `${c.questionId}#${c.answerIndex}`;
 		const expected = c.expected.join("|");
 		if (c.status === "errored") {
-			const first = c.repeats.find((row) => row.error !== void 0)?.error;
-			const message = typeof first === "object" && first !== null && "message" in first ? String(first.message) : "no verdict";
+			const message = firstErrorMessage(c.repeats, "no verdict");
 			return `  ${red("✗")} ${head} expected ${expected} got ${red("error")} ${dim(message)}`;
 		}
 		return `  ${red("✗")} ${head} expected ${expected} got ${red(c.verdict ?? "?")}` + dim(` "${snippet(c.answer)}"`);
@@ -3989,7 +4403,9 @@ function formatEvalReport(result, source) {
 	lines.push(`  llm: ${llm}`);
 	const judge = result.llm.judge;
 	if (judge && (judge.provider !== result.llm.provider || judge.model !== result.llm.model)) lines.push(`  judge llm: ${judge.provider} / ${judge.model}${judge.overridden ? ` ${yellow("(override)")}` : ""}`);
-	lines.push(`  cases: ${totals.cases} × ${totals.repeats} repeat(s) = ${totals.calls} grading call(s)` + (result.judging === "off" ? "" : ` + ${totals.calls} judge call(s)`));
+	const unit = result.kind === "tutor" ? "conversation" : "case";
+	const generation = result.kind === "tutor" ? "generation" : "grading";
+	lines.push(`  ${unit}s: ${totals.cases} × ${totals.repeats} repeat(s) = ${totals.calls} ${generation} call(s)` + (result.judging === "off" ? "" : ` + ${totals.calls} judge call(s)`));
 	if (result.aborted) {
 		lines.push("");
 		lines.push(red(`Run aborted: ${result.aborted.message}`));
@@ -4004,7 +4420,7 @@ function formatEvalReport(result, source) {
 		lines.push(...mismatchLines(result));
 	}
 	lines.push("");
-	lines.push(`  passed: ${totals.passed}   failed: ${totals.failed}   errored: ${totals.errored}` + (totals.skipped ? red(`   skipped: ${totals.skipped} (run aborted)`) : "") + (totals.unstable ? dim(`   unstable: ${totals.unstable}`) : "") + (!anyJudged(result) ? "" : totals.feedbackFlagged ? yellow(`   flagged feedback: ${totals.feedbackFlagged}`) : dim("   flagged feedback: 0")) + (totals.judgeErrored ? dim(`   judge errors: ${totals.judgeErrored}`) : ""));
+	lines.push((result.kind === "tutor" ? `  ok: ${totals.cases - totals.errored - totals.skipped}   errored: ${totals.errored}` : `  passed: ${totals.passed}   failed: ${totals.failed}   errored: ${totals.errored}`) + (totals.skipped ? red(`   skipped: ${totals.skipped} (run aborted)`) : "") + (totals.unstable ? dim(`   unstable: ${totals.unstable}`) : "") + (!anyJudged(result) ? "" : totals.feedbackFlagged ? yellow(`   ${flaggedLabel(result.kind)}: ${totals.feedbackFlagged}`) : dim(`   ${flaggedLabel(result.kind)}: 0`)) + (!anyToolsRequired(result) ? "" : totals.toolsFlagged ? yellow(`   missing tool calls: ${totals.toolsFlagged}`) : dim("   missing tool calls: 0")) + (totals.judgeErrored ? dim(`   judge errors: ${totals.judgeErrored}`) : ""));
 	const tokens = formatUsageLine(totals.usage);
 	if (tokens) lines.push(dim(`  ${tokens}`));
 	if (result.confusion.length) {
@@ -4012,9 +4428,11 @@ function formatEvalReport(result, source) {
 		lines.push("  confusion (expected → got):");
 		for (const row of result.confusion) lines.push(`    ${row.expected} → ${row.got}: ${row.count}`);
 	}
-	const { count, denominator, rate } = result.falseCorrect;
-	lines.push("");
-	lines.push(`  false-correct: ${count}/${denominator}` + (denominator ? ` (${(rate * 100).toFixed(1)}%)` : ""));
+	if (result.kind !== "tutor") {
+		const { count, denominator, rate } = result.falseCorrect;
+		lines.push("");
+		lines.push(`  false-correct: ${count}/${denominator}` + (denominator ? ` (${(rate * 100).toFixed(1)}%)` : ""));
+	}
 	return lines.join("\n");
 }
 /**
@@ -4034,12 +4452,14 @@ function formatEvalBatchReport(batch) {
 		}
 		const t = file.result.totals;
 		const mark = t.failed === 0 && t.errored === 0 && t.skipped === 0 ? green("✔") : red("✗");
-		lines.push(`  ${mark} ${name}: ${t.cases} case(s), ${t.passed} passed, ${t.failed} failed, ${t.errored} errored` + (t.skipped ? red(`, ${t.skipped} skipped`) : "") + (t.unstable ? dim(`, ${t.unstable} unstable`) : "") + (!anyJudged(file.result) ? "" : t.feedbackFlagged ? yellow(`, ${t.feedbackFlagged} flagged`) : dim(", 0 flagged")));
+		const counts = file.result.kind === "tutor" ? `${t.cases} conversation(s), ${t.cases - t.errored - t.skipped} ok, ${t.errored} errored` : `${t.cases} case(s), ${t.passed} passed, ${t.failed} failed, ${t.errored} errored`;
+		lines.push(`  ${mark} ${name}: ${counts}` + (t.skipped ? red(`, ${t.skipped} skipped`) : "") + (t.unstable ? dim(`, ${t.unstable} unstable`) : "") + (!anyJudged(file.result) ? "" : t.feedbackFlagged ? yellow(`, ${t.feedbackFlagged} flagged`) : dim(", 0 flagged")) + (!anyToolsRequired(file.result) ? "" : t.toolsFlagged ? yellow(`, ${t.toolsFlagged} missing tool calls`) : dim(", 0 missing tool calls")));
 	}
 	const g = batch.totals;
 	const judged = batch.files.some((file) => file.result && anyJudged(file.result));
+	const toolChecked = batch.files.some((file) => file.result && anyToolsRequired(file.result));
 	lines.push("");
-	lines.push(`  TOTAL: ${g.cases} case(s), ${g.passed} passed, ${g.failed} failed, ${g.errored} errored` + (g.skipped ? red(`, ${g.skipped} skipped`) : "") + (judged ? g.feedbackFlagged ? yellow(`, ${g.feedbackFlagged} flagged`) : dim(", 0 flagged") : "") + (g.invalid ? red(`, ${g.invalid} invalid file(s)`) : ""));
+	lines.push(`  TOTAL: ${g.cases} case(s), ${g.passed} passed, ${g.failed} failed, ${g.errored} errored` + (g.skipped ? red(`, ${g.skipped} skipped`) : "") + (judged ? g.feedbackFlagged ? yellow(`, ${g.feedbackFlagged} flagged`) : dim(", 0 flagged") : "") + (toolChecked ? g.toolsFlagged ? yellow(`, ${g.toolsFlagged} missing tool calls`) : dim(", 0 missing tool calls") : "") + (g.invalid ? red(`, ${g.invalid} invalid file(s)`) : ""));
 	if (batch.files.some((file) => file.result?.judging === "degraded")) lines.push(yellow("  Feedback judging stopped mid-run after repeated judge failures — files without a flagged count were graded but not judged."));
 	const tokens = formatUsageLine(g.usage);
 	if (tokens) lines.push(dim(`  ${tokens}`));
@@ -4196,17 +4616,18 @@ function overview(batch) {
 			continue;
 		}
 		const t = file.result.totals;
+		const tutor = file.result.kind === "tutor";
 		lines.push(row([
 			`${file.passed ? "✅" : "❌"} ${name}`,
 			`\`${cell(file.result.id)}\``,
 			count(t.cases),
-			count(t.passed),
-			count(t.failed),
+			tutor ? "—" : count(t.passed),
+			tutor ? "—" : count(t.failed),
 			count(t.errored),
 			count(t.skipped),
-			count(t.unstable),
+			tutor ? "—" : count(t.unstable),
 			anyJudged(file.result) ? count(t.feedbackFlagged) : "—",
-			falseCorrectCell(file.result),
+			tutor ? "—" : falseCorrectCell(file.result),
 			usageCell(t.usage)
 		]));
 	}
@@ -4300,7 +4721,7 @@ function caseSection(evalCase, questionText) {
 * the run failed on them. Empty when the file has no flags.
 */
 function flaggedSection(result, questionText) {
-	const flagged = result.cases.filter((evalCase) => evalCase.feedbackFlagged);
+	const flagged = result.cases.filter((evalCase) => evalCase.feedbackFlagged && !isTutorCase(evalCase));
 	if (flagged.length === 0) return [];
 	const lines = ["### Flagged feedback", ""];
 	lines.push("_An LLM judge audited each feedback text against the very grading prompt it was written under. Reported only — flagged feedback never fails a run._");
@@ -4322,6 +4743,104 @@ function flaggedSection(result, questionText) {
 	}
 	return lines;
 }
+/**
+* A tutor case's stable heading: the teacher's `title` when it has one, otherwise its
+* 1-based index plus an excerpt of the FIRST student line — enough to recognise the case
+* in a report without opening the eval file.
+*/
+function tutorCaseLabel(evalCase) {
+	if (evalCase.title) return `#${evalCase.index + 1} ${cell(evalCase.title)}`;
+	const firstStudent = evalCase.conversation.find((turn) => "student" in turn);
+	const excerpt = firstStudent && "student" in firstStudent ? inline(firstStudent.student) : "";
+	const short = excerpt.length > 60 ? `${excerpt.slice(0, 59)}…` : excerpt;
+	return short ? `#${evalCase.index + 1} — ${short}` : `#${evalCase.index + 1}`;
+}
+/** The scripted conversation as one labeled, verbatim blockquote per turn. */
+function conversationBlock(evalCase) {
+	const lines = ["**Conversation**", ""];
+	for (const turn of evalCase.conversation) {
+		const role = "student" in turn ? "student" : "tutor";
+		const text = "student" in turn ? turn.student : turn.tutor;
+		lines.push(`*${role}*`, "", quote(text), "");
+	}
+	return lines;
+}
+/** One ERRORED tutor case: what was asked, and why nothing came back. */
+function tutorErrorSection(evalCase) {
+	const lines = [`### ${tutorCaseLabel(evalCase)} — error`, ""];
+	lines.push(...conversationBlock(evalCase));
+	const failure = evalCase.repeats.find((r) => r.error !== void 0);
+	if (failure) lines.push("**Error**", "", quote(errorMessage(failure.error)), "");
+	return lines;
+}
+/** `random_number, random_number` — a repeat's tool calls, or `(none)` when it made none. */
+function toolCallList(toolCalls) {
+	return toolCalls && toolCalls.length > 0 ? toolCalls.map((name) => `\`${name}\``).join(", ") : "(none)";
+}
+/**
+* The tutor kind's "Missing tool calls" section: every case that declares `required_tools`
+* and had at least one repeat skip one. Per case the required list, what each offending
+* repeat actually called, and which tools were missing there.
+*
+* Its own section rather than a column, for the same reason "Flagged responses" is one:
+* these cases are `ok` and the run PASSED — a missing tool call is a note about the
+* tutor's behavior, never a failure. Cases whose tools all ran stay out entirely; the
+* `--json` report carries every repeat's `toolCalls` for anyone who wants them.
+*/
+function tutorMissingToolsSection(result) {
+	const flagged = result.cases.filter((evalCase) => isTutorCase(evalCase) && evalCase.toolsFlagged);
+	if (flagged.length === 0) return [];
+	const lines = ["### Missing tool calls", ""];
+	lines.push("_These cases require a tool the tutor did not call in every run. Reported only — a missing tool call never fails a run, and tools beyond the required ones are fine._");
+	lines.push("");
+	for (const evalCase of flagged) {
+		lines.push(`#### ${tutorCaseLabel(evalCase)}`);
+		lines.push("");
+		lines.push(`**Required** ${toolCallList(evalCase.requiredTools)}`);
+		lines.push("");
+		for (const repeat of evalCase.repeats) {
+			const missing = repeat.missingTools ?? [];
+			if (missing.length === 0) continue;
+			lines.push(`- Repeat #${repeat.repeatIndex + 1} — missing ${toolCallList(missing)}; called ${toolCallList(repeat.toolCalls)}`);
+		}
+		lines.push("");
+	}
+	return lines;
+}
+/**
+* The tutor kind's "Flagged responses" section — the report's actual deliverable: per
+* flagged case the scripted conversation, the teacher's expectations when it states any,
+* and each flagged repeat's GENERATED RESPONSE verbatim followed by the judge's issues.
+*
+* Clean cases stay out entirely (their generated texts are in the `--json` report), and
+* a flag never means the run failed — the tutor kind is report-only.
+*/
+function tutorFlaggedSection(result) {
+	const flagged = result.cases.filter((evalCase) => isTutorCase(evalCase) && evalCase.feedbackFlagged);
+	if (flagged.length === 0) return [];
+	const lines = ["### Flagged responses", ""];
+	lines.push("_An LLM judge audited each generated response against the tutor's own system prompt and, where the case states any, the teacher's expectations. Reported only — a flagged response never fails a run._");
+	lines.push("");
+	for (const evalCase of flagged) {
+		lines.push(`#### ${tutorCaseLabel(evalCase)}`);
+		lines.push("");
+		lines.push(...conversationBlock(evalCase));
+		if (evalCase.gradingInstructions) lines.push("**Expectations for this case**", "", quote(evalCase.gradingInstructions), "");
+		if (evalCase.requiredTools) lines.push(`**Required tools** ${toolCallList(evalCase.requiredTools)}`, "");
+		for (const repeat of evalCase.repeats) {
+			const issues = repeat.judge?.issues ?? [];
+			if (issues.length === 0) continue;
+			lines.push(`**Generated response — repeat #${repeat.repeatIndex + 1}**`);
+			lines.push("");
+			lines.push(quote(repeat.text ?? ""));
+			lines.push("");
+			if (repeat.toolCalls && (repeat.toolCalls.length > 0 || evalCase.requiredTools)) lines.push(`*tool calls: ${toolCallList(repeat.toolCalls)}*`, "");
+			for (const issue of issues) lines.push(`- \`${cell(issue.criterion)}\` — ${inline(issue.note)}`);
+			lines.push("");
+		}
+	}
+	return lines;
+}
 /** One file's details section, or `[]` when the file has nothing to report. */
 function fileDetails(file) {
 	const name = shortSource(file.source);
@@ -4330,30 +4849,33 @@ function fileDetails(file) {
 		return [
 			`## ${cell(name)} — invalid`,
 			"",
-			"This file was not graded; fix the problems below and run it again.",
+			"This file was not run; fix the problems below and run it again.",
 			"",
 			...errors.map((issue) => `- \`${cell(issue.code)}\` — ${inline(issue.message)}`),
 			""
 		];
 	}
 	const result = file.result;
+	const tutor = result.kind === "tutor";
 	const detailed = result.cases.filter(needsDetail);
 	const skipped = result.totals.skipped;
 	const questionText = new Map(result.questions.map((question) => [question.id, question.text]));
-	const flagged = flaggedSection(result, questionText);
-	if (detailed.length === 0 && skipped === 0 && !result.aborted && flagged.length === 0) return [];
+	const flagged = tutor ? tutorFlaggedSection(result) : flaggedSection(result, questionText);
+	const missingTools = tutor ? tutorMissingToolsSection(result) : [];
+	if (detailed.length === 0 && skipped === 0 && !result.aborted && flagged.length === 0 && missingTools.length === 0) return [];
 	const lines = [`## ${cell(name)} — \`${cell(result.id)}\``, ""];
 	if (result.aborted) {
 		lines.push("> [!WARNING]");
 		lines.push(`> The run was aborted: ${inline(result.aborted.message)}`);
 		lines.push("");
 	}
-	for (const evalCase of detailed) lines.push(...caseSection(evalCase, questionText.get(evalCase.questionId)));
+	for (const evalCase of detailed) lines.push(...isTutorCase(evalCase) ? tutorErrorSection(evalCase) : caseSection(evalCase, questionText.get(evalCase.questionId)));
 	if (skipped > 0) {
 		const reason = result.aborted ? ` (${inline(result.aborted.message)})` : "";
-		lines.push(`**${count(skipped)} case(s) were never attempted**${reason} — the run is incomplete, so it cannot pass.`);
+		lines.push(`**${count(skipped)} ${tutor ? "conversation" : "case"}(s) were never attempted**${reason} — the run is incomplete, so it cannot pass.`);
 		lines.push("");
 	}
+	lines.push(...missingTools);
 	lines.push(...flagged);
 	return lines;
 }
@@ -4371,8 +4893,9 @@ function renderEvalMarkdownReport(batch, meta) {
 	const judges = [...new Set(batch.files.map((file) => file.result ? judgeLlmText(file.result.llm) : void 0).filter((text) => text !== void 0))];
 	for (const judge of judges) lines.push(`- **Feedback judge** ${judge}`);
 	lines.push(`- **Run** ${count(batch.totals.files)} file(s), ${count(batch.totals.cases)} case(s) × ${count(meta.repeats)} repeat(s), concurrency ${count(meta.concurrency)}`);
+	if (batch.files.some((file) => file.result && anyToolsRequired(file.result))) lines.push(`- **Missing tool calls** ${count(batch.totals.toolsFlagged)} case(s) did not call a required tool in every run — reported only, never a failure`);
 	const tokens = batch.totals.usage;
-	if (tokens.input || tokens.cachedInput || tokens.output) lines.push(`- **Tokens** ${count(tokens.input)} in (${count(tokens.cachedInput)} cached) / ${count(tokens.output)} out — successful grading calls only, so a lower bound`);
+	if (tokens.input || tokens.cachedInput || tokens.output) lines.push(`- **Tokens** ${count(tokens.input)} in (${count(tokens.cachedInput)} cached) / ${count(tokens.output)} out — successful calls only, so a lower bound`);
 	lines.push("");
 	if (batch.files.filter((file) => file.result?.aborted).length > 0) {
 		lines.push("> [!WARNING]");
@@ -4391,10 +4914,10 @@ function renderEvalMarkdownReport(batch, meta) {
 	lines.push("");
 	const details = batch.files.flatMap((file) => fileDetails(file));
 	if (details.length === 0) {
-		lines.push("_Nothing else to report — every case matched its expected verdict. The `--json` report carries every case, including the passing ones._");
+		lines.push("_Nothing else to report. The `--json` report carries every case, including the clean ones._");
 		lines.push("");
 	} else {
-		lines.push("_Below: only the mismatched, errored and unstable cases, plus any feedback the judge flagged. Passing cases live in the `--json` report._");
+		lines.push("_Below: only the mismatched, errored and unstable cases, plus anything the judge flagged. Clean cases live in the `--json` report._");
 		lines.push("");
 		lines.push(...details);
 	}
@@ -4820,6 +5343,61 @@ function makeGradeFn(server, llm) {
 	};
 }
 /**
+* The `toolCalls: string[]` of a tutor 200, defensively: `undefined` when the field is
+* absent (a server too old to report tool calls — a distinction the runner MUST be able to
+* make), and non-string entries are dropped rather than breaking a run. Names only, in the
+* order the server sent them, duplicates kept.
+*/
+function parseToolCalls(value) {
+	if (!Array.isArray(value)) return void 0;
+	return value.filter((name) => typeof name === "string" && name !== "");
+}
+/**
+* The HTTP seam for ONE generated tutor turn, with the run's effective llm closed in —
+* the tutor kind's sibling of {@link makeGradeFn}, sharing its failure classification
+* exactly (5xx and network retryable, auth aborts the run, every other 4xx terminal).
+*/
+function makeRespondFn(server, llm) {
+	return async ({ system, tools, messages }) => {
+		const response = await performApiRequest({
+			server,
+			path: "/api/eval/respond",
+			method: "POST",
+			body: {
+				llm,
+				system,
+				tools: [...tools],
+				messages: messages.map((m) => ({ ...m }))
+			},
+			quiet: true
+		});
+		if (response.ok) {
+			const payload = response.payload;
+			if (typeof payload?.text === "string" && payload.text !== "") {
+				const usage = parseUsage(payload?.usage);
+				const toolCalls = parseToolCalls(payload?.toolCalls);
+				return {
+					ok: true,
+					text: payload.text,
+					...toolCalls ? { toolCalls } : {},
+					...usage ? { usage } : {}
+				};
+			}
+			return {
+				ok: false,
+				retryable: false,
+				error: { message: "The server's response is not a generated tutor turn — it may not offer /api/eval/respond at all (does it run a Novedu version with tutor evals?). Check the target server, e.g. --server http://localhost:3000." }
+			};
+		}
+		return {
+			ok: false,
+			retryable: response.status === void 0 || response.status >= 500,
+			...response.authFailed ? { auth: true } : {},
+			error: response.error
+		};
+	};
+}
+/**
 * The HTTP seam for ONE judge call, with the run's judge llm closed in. Mirrors
 * {@link makeGradeFn}'s failure classification, minus the auth branch: a judge failure
 * NEVER aborts the run — it degrades judging (see the runner's breaker) while the grading
@@ -4934,7 +5512,8 @@ function progressWriter(prefix) {
 */
 function writeFileDone(label, result) {
 	const totals = result.totals;
-	process.stderr.write(`${label}: ${totals.cases} case(s), ${totals.passed} passed, ${totals.failed} failed, ${totals.errored} errored` + (totals.skipped ? `, ${totals.skipped} skipped` : "") + (anyJudged(result) ? `, ${totals.feedbackFlagged} flagged` : "") + "\n");
+	const counts = result.kind === "tutor" ? `${totals.cases} conversation(s), ${totals.cases - totals.errored - totals.skipped} ok, ${totals.errored} errored` : `${totals.cases} case(s), ${totals.passed} passed, ${totals.failed} failed, ${totals.errored} errored`;
+	process.stderr.write(`${label}: ${counts}` + (totals.skipped ? `, ${totals.skipped} skipped` : "") + (anyJudged(result) ? `, ${totals.feedbackFlagged} flagged` : "") + "\n");
 }
 /**
 * The command's core, exported for the unit tests. `seams` exists only so tests can
@@ -4999,9 +5578,14 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 		return;
 	}
 	{
-		const totalCases = [...checked.values()].reduce((sum, file) => sum + file.caseCount, 0);
-		const calls = totalCases * repeats;
-		process.stderr.write(`${totalCases} case(s) × ${repeats} repeat(s) = ${calls} grading` + (judging ? ` + ${calls} judge call(s)\n` : " call(s)\n"));
+		const scope = (unit, generation, cases) => {
+			if (cases === 0) return;
+			const calls = cases * repeats;
+			process.stderr.write(`${cases} ${unit}(s) × ${repeats} repeat(s) = ${calls} ${generation}` + (judging ? ` + ${calls} judge call(s)\n` : " call(s)\n"));
+		};
+		const casesOf = (kind) => [...checked.values()].filter((file) => file.kind === kind).reduce((sum, file) => sum + file.caseCount, 0);
+		scope("case", "grading", casesOf("quiz"));
+		scope("conversation", "generation", casesOf("tutor"));
 	}
 	await warnOnVersionMismatch(options.server);
 	const judgeBreaker = createJudgeBreaker();
@@ -5013,15 +5597,15 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 		fileIndex += 1;
 		const check = checked.get(file.source);
 		if (!check) continue;
-		const quizLlm = {
-			provider: check.quizDump.llm.provider,
-			model: check.quizDump.llm.model
+		const activityLlm = {
+			provider: check.llm.provider,
+			model: check.llm.model
 		};
-		const effective = override.llm ?? quizLlm;
+		const effective = override.llm ?? activityLlm;
 		const judgeLlm = judgeOverride.llm ?? effective;
 		const llm = {
 			...effective,
-			...override.llm ? { overrides: quizLlm } : {},
+			...override.llm ? { overrides: activityLlm } : {},
 			...judging ? { judge: {
 				...judgeLlm,
 				overridden: judgeOverride.llm !== void 0
@@ -5029,8 +5613,9 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 		};
 		const label = files.length > 1 ? `(${fileIndex}/${files.length}) ${check.evalFile.id}` : check.evalFile.id;
 		const prefix = files.length > 1 ? `${label}: ` : "";
-		const result = await runEval("quiz", check, {
+		const result = await runEval(check.kind, check, {
 			grade: makeGradeFn(options.server, effective),
+			respond: makeRespondFn(options.server, effective),
 			...judging ? { judge: makeJudgeFn(options.server, judgeLlm) } : {},
 			judgeBreaker,
 			onJudgeDegraded,
@@ -5069,12 +5654,16 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 	process.exitCode = batchPassed(batch) ? 0 : 1;
 }
 function registerEval(program) {
-	program.command("eval").description("Grade a file of golden answers against its quiz's real rubric and report the result").argument("<evalPathOrUrl...>", "one or more eval YAML files (paths, http(s)/file URLs, or a quoted glob pattern)").option("--server <url>", "Novedu server base URL (defaults to the NOVEDU_SERVER env var, then production)").option("--concurrency <n>", "grading calls in flight per file", String(CONCURRENCY_DEFAULT)).option("--repeats <n>", "grade every answer N times and take the majority verdict", "1").option("--llm-provider <provider>", "grade with this provider instead of the quiz's (\"SCCH\" or \"Azure Foundry\"; needs --llm-model)").option("--llm-model <model>", "grade with this model instead of the quiz's (needs --llm-provider)").option("--no-judge-feedback", "skip the LLM audit of the grader's feedback (halves the LLM calls)").option("--judge-llm-provider <provider>", "judge the feedback with this provider (\"SCCH\" or \"Azure Foundry\"; needs --judge-llm-model)").option("--judge-llm-model <model>", "judge the feedback with this model instead of the grading one (needs --judge-llm-provider)").option("--json", "print the machine-readable batch report on stdout").option("--out <file>", "additionally write the machine-readable batch report to a file").option("--report <file>", "additionally write a readable Markdown report to a file").addHelpText("after", `
+	program.command("eval").description("Run an eval file (quiz golden answers, or tutor conversations) against the real activity path and report the result").argument("<evalPathOrUrl...>", "one or more eval YAML files (paths, http(s)/file URLs, or a quoted glob pattern)").option("--server <url>", "Novedu server base URL (defaults to the NOVEDU_SERVER env var, then production)").option("--concurrency <n>", "cases in flight per file", String(CONCURRENCY_DEFAULT)).option("--repeats <n>", "run every case N times (quiz: take the majority verdict)", "1").option("--llm-provider <provider>", "run with this provider instead of the activity's (\"SCCH\" or \"Azure Foundry\"; needs --llm-model)").option("--llm-model <model>", "run with this model instead of the activity's (needs --llm-provider)").option("--no-judge-feedback", "skip the LLM audit of what the model wrote (halves the LLM calls)").option("--judge-llm-provider <provider>", "judge with this provider (\"SCCH\" or \"Azure Foundry\"; needs --judge-llm-model)").option("--judge-llm-model <model>", "judge with this model instead of the one under test (needs --judge-llm-provider)").option("--json", "print the machine-readable batch report on stdout").option("--out <file>", "additionally write the machine-readable batch report to a file").option("--report <file>", "additionally write a readable Markdown report to a file").addHelpText("after", `
 Examples:
   # Evaluate one quiz's golden answers
   $ novedu-cli eval ./0010-welcome-quiz.eval.yaml
 
-  # A whole course part (quote the pattern so the CLI expands it, ** included)
+  # Check how a tutor answers a set of scripted conversations
+  $ novedu-cli eval ./loops-tutor.eval.yaml
+
+  # A whole course part — quiz and tutor evals may be mixed
+  # (quote the pattern so the CLI expands it, ** included)
   $ novedu-cli eval "./part-1/**/*.eval.yaml"
 
   # Measure grader stability: 3 runs per answer, majority verdict

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@novedu/cli",
-  "version": "0.22.0",
-  "description": "Command-line companion for the Novedu chat app. Validates tutor, fragment, quiz, writing, coding and eval YAML definitions, dumps the exact LLM prompts an activity produces, and evaluates a quiz's grading rubric against golden answers; signs in with Entra ID and manages codes, app-hosted files and images over the app's API.",
+  "version": "0.23.0",
+  "description": "Command-line companion for the Novedu chat app. Validates tutor, fragment, quiz, writing, coding and eval YAML definitions, dumps the exact LLM prompts an activity produces, evaluates a quiz's grading rubric against golden answers and replays scripted conversations against a tutor; signs in with Entra ID and manages codes, app-hosted files and images over the app's API.",
   "type": "module",
   "repository": {
     "type": "git",