@novedu/cli 0.23.0 → 0.24.1

package/README.md

@@ -85,7 +85,8 @@ npx @novedu/cli prompts ./sorting-quiz.yaml --kind quiz --json \
   re-implementation), so what you see is what the model gets: fragments resolved,
   and — for a compound quiz — every `quiz_files` include fetched, each imported
   question carrying its source quiz's preamble.
-- Every dump carries `{ kind, id, llm: { provider, model } }`. A **quiz** adds
+- Every dump carries `{ kind, id, llm: { provider, model, reasoning? } }` (the
+  reasoning level only when the file sets one). A **quiz** adds
   `grading` (a `system` prompt per question, the user-message templates and the
   grader's JSON-Schema response contract) and `discussion` (the chat's `system`
   prompt, the three seed-message templates and the verdict wording). A **coding**
@@ -149,6 +150,9 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --repeats 3
 npx @novedu/cli eval ./sorting-quiz.eval.yaml \
   --llm-provider "Azure Foundry" --llm-model gpt-5-mini
 
+# Same model, more thinking: the level alone keeps the activity's provider/model
+npx @novedu/cli eval ./sorting-quiz.eval.yaml --llm-reasoning high
+
 # A strong judge over the quiz's own grader — the recommended pairing
 npx @novedu/cli eval ./sorting-quiz.eval.yaml \
   --judge-llm-provider "Azure Foundry" --judge-llm-model gpt-5.6-terra
@@ -225,12 +229,20 @@ npx @novedu/cli eval ./loops-tutor.eval.yaml --report loops.md
   only when some case required a tool, so no line means "not checked"), a **"Missing tool
   calls"** section in the Markdown report, and `totals.toolsFlagged` plus each repeat's
   `toolCalls` / `missingTools` in the JSON.
-- **Choosing the judge.** By default the judge runs on the same model as the grader.
-  `--judge-llm-provider` + `--judge-llm-model` (both or neither) point it at another one,
-  which is the **recommended** setup: a strong judge over a smaller grader finds real
-  problems, while a small model judging itself mostly produces noise. `--no-judge-feedback`
-  turns judging off and halves the LLM calls; combining the two is rejected as
-  contradictory. Because judging roughly doubles the cost, the run's scope line says so
+- **Choosing what runs.** `--llm-provider` + `--llm-model` (both or neither) replace the
+  activity's **whole** `llm:` block for the run — its reasoning level included, so the
+  file's level is dropped unless `--llm-reasoning <level>` restates it.
+  `--llm-reasoning` on its own changes only the effort and keeps the file's
+  provider/model — the "same model, more thinking" comparison run.
+- **Choosing the judge.** By default the judge runs on the same model **and effort** as
+  the grader. `--judge-llm-provider` + `--judge-llm-model` (both or neither) point it at
+  another one — replacing the whole spec, exactly like the grading flags — which is the
+  **recommended** setup: a strong judge over a smaller grader finds real
+  problems, while a small model judging itself mostly produces noise.
+  `--judge-llm-reasoning` sets the judge's effort on its own, no pair needed.
+  `--no-judge-feedback` turns judging off and halves the LLM calls; combining it with any
+  of the judge flags is rejected as contradictory. Because judging roughly doubles the
+  cost, the run's scope line says so
   up front: `27 case(s) × 3 repeat(s) = 81 grading + 81 judge call(s)`.
 - **If the judge itself fails**, the run **degrades instead of aborting**: after three
   consecutive judge failures it stops judging (one warning on stderr) and finishes the
@@ -255,7 +267,7 @@ npx @novedu/cli eval ./loops-tutor.eval.yaml --report loops.md
   run localises the problem for the cost of a single grading call.
 - **Caveat**: a green run certifies **the file you ran it on**, not the app-hosted
   copy a live code serves — upload it (`files upload`) afterwards. An override run
-  certifies the override pair, not the quiz's configured `llm`.
+  certifies the override, not the quiz's configured `llm`.
 
 ## Authentication
 
@@ -300,7 +312,7 @@ authoritative — the CLI sends your input as-is and relays the server's answer.
 ```
 codes create  --module <tutor|quiz|writing|coding> --file <url>
               [--start <iso>] [--end <iso>] [--note <text>]
-              [--llm-provider <p> --llm-model <m>]
+              [--llm-provider <p> --llm-model <m>] [--llm-reasoning <level>]
 codes list    [--search <q>] [--module <m>] [--all]
 codes sync    <registry-file> [--lock <path>] [--dry-run] [--json]
 files upload  <name> [--kind <tutor|fragment|quiz|writing|coding>]
@@ -318,7 +330,10 @@ images list   [--search <q>] [--all]
   server-side before the code is stored, and the response includes the
   shareable `url`. `--start`/`--end` must be ISO 8601 **with an explicit
   offset or `Z`** (e.g. `2026-07-07T08:00:00Z`); the
-  `--llm-provider`/`--llm-model` override pair is both-or-nothing.
+  `--llm-provider`/`--llm-model` override pair is both-or-nothing, and
+  `--llm-reasoning <level>` (`minimal`, `low`, `medium` or `high`) rides on top of
+  the pair — it is rejected without it. The override replaces the activity's whole
+  `llm:` block, so leaving the level out also drops the file's.
 - `codes sync <registry-file>` mints codes for a whole **course** at once — see
   [Many activities at once](#many-activities-at-once-codes-sync) below.
 - `files upload <name>` is an **upsert**: creating a new file requires
@@ -416,11 +431,13 @@ activity-codes:
 - **Groups decide the module:** `quizzes`, `tutors`, `writing`, `coding`. Each
   entry gives either `file` (relative to `base-url`, which must end in `/`) or
   an absolute `url`, plus any of `start`/`end` (ISO 8601 **with an offset or
-  `Z`**, whole seconds), `note`, and an `llm: {provider, model}` override.
+  `Z`**, whole seconds), `note`, and an `llm: {provider, model, reasoning?}`
+  override.
 - **Keys are yours and must be unique across all groups** — lowercase letters,
   digits and hyphens. Your material references the key; the lock file maps it to
   the code.
-- **Re-runs are safe.** An entry whose activity, window and model override match
+- **Re-runs are safe.** An entry whose activity, window and LLM override
+  (provider, model and reasoning level) match
   an existing code of yours **reuses** that code; only entries without a match
   are minted. So `codes sync` after every edit is the normal workflow, and the
   first run against already-minted codes should report all-reused.

package/dist/main.js

@@ -85,17 +85,46 @@ async function acquireSilent(pca) {
 		return null;
 	}
 }
+/**
+* How the system browser is launched per platform. Exported for tests.
+*
+* Windows goes through `cmd /c start`, and cmd RE-PARSES the command line it
+* receives: an unquoted URL is cut at the first `&`, so Entra only ever saw
+* `authorize?client_id=…` and answered AADSTS900144 ("the request body must
+* contain the following parameter: 'scope'"). Node quotes an argument only
+* when it contains whitespace, so the URL is quoted here explicitly and the
+* command line handed over verbatim. The empty `""` is `start`'s window title
+* — without it, `start` would take the quoted URL as the title.
+*/
+function browserCommand(url, platform = process.platform) {
+	if (platform === "darwin") return {
+		command: "open",
+		args: [url],
+		verbatim: false
+	};
+	if (platform === "win32") return {
+		command: "cmd",
+		args: [
+			"/c",
+			"start",
+			"\"\"",
+			`"${url}"`
+		],
+		verbatim: true
+	};
+	return {
+		command: "xdg-open",
+		args: [url],
+		verbatim: false
+	};
+}
 function defaultOpenBrowser(url) {
-	const [command, args] = process.platform === "darwin" ? ["open", [url]] : process.platform === "win32" ? ["cmd", [
-		"/c",
-		"start",
-		"",
-		url
-	]] : ["xdg-open", [url]];
+	const { command, args, verbatim } = browserCommand(url);
 	try {
 		spawn(command, args, {
 			stdio: "ignore",
-			detached: true
+			detached: true,
+			windowsVerbatimArguments: verbatim
 		}).unref();
 	} catch {}
 }
@@ -284,9 +313,19 @@ async function runApiRequest(options) {
 const LLM_PROVIDERS = ["SCCH", "Azure Foundry"];
 const DEFAULT_PROVIDER = "SCCH";
 const providerSchema = z.enum(LLM_PROVIDERS).default(DEFAULT_PROVIDER).meta({ description: "The LLM provider serving the model. For Azure Foundry, model is the deployment name." });
+const REASONING_LEVELS = [
+	"minimal",
+	"low",
+	"medium",
+	"high"
+];
+const reasoningLevelSchema = z.enum(REASONING_LEVELS).optional().meta({ description: "Optional reasoning effort for reasoning models. Omit to let the model decide (the parameter is then not sent)." });
 function parseLenientProvider(value) {
 	return value === "SCCH" || value === "Azure Foundry" ? value : void 0;
 }
+function parseLenientReasoningLevel(value) {
+	return typeof value === "string" && REASONING_LEVELS.includes(value) ? value : void 0;
+}
 //#endregion
 //#region ../lib/registry-schema.ts
 /** The fixed group names and the code module each one mints for. */
@@ -307,6 +346,7 @@ function timestampField(field) {
 	}, `${field} must not carry sub-second precision — the server stores whole seconds`);
 }
 const providerField = z.enum(LLM_PROVIDERS, { error: "must be \"SCCH\" or \"Azure Foundry\"" });
+const reasoningField = z.enum(REASONING_LEVELS, { error: `must be one of ${REASONING_LEVELS.join(", ")}` });
 /**
 * One registry entry. Unknown extra properties are ACCEPTED and ignored so authors can
 * annotate freely and a newer registry keeps working with an older CLI — which is why
@@ -320,8 +360,9 @@ const RegistryEntrySchema = z.looseObject({
 	note: z.string().trim().max(200, `note must be at most 200 characters`).optional().meta({ description: `Note shown in the codes list, at most 200 characters. No effect on behaviour.` }),
 	llm: z.looseObject({
 		provider: providerField.meta({ description: "LLM provider override for this code. Required when `llm` is present." }),
-		model: z.string().trim().min(1).max(256).meta({ description: "Model id (for Azure Foundry, the deployment name). Required when `llm` is present." })
-	}).optional().meta({ description: "Per-code LLM override replacing the activity YAML's own `llm:`. Provider and model must be given together." })
+		model: z.string().trim().min(1).max(256).meta({ description: "Model id (for Azure Foundry, the deployment name). Required when `llm` is present." }),
+		reasoning: reasoningField.optional().meta({ description: "Optional reasoning effort for reasoning models, applied on top of the provider/model pair. Omit to let the model decide." })
+	}).optional().meta({ description: "Per-code LLM override replacing the activity YAML's own `llm:`. Provider and model must be given together; `reasoning` is optional on top of them." })
 }).refine((entry) => entry.file === void 0 !== (entry.url === void 0), "give exactly one of `file` (relative to base-url) or `url` (absolute)").meta({
 	id: "registryEntry",
 	description: "One activity: where its YAML lives, plus the parameters its code is minted with."
@@ -451,7 +492,8 @@ function parseRegistry(text) {
 				note: entry.note ?? null,
 				llm: entry.llm ? {
 					provider: entry.llm.provider,
-					model: entry.llm.model
+					model: entry.llm.model,
+					...entry.llm.reasoning ? { reasoning: entry.llm.reasoning } : {}
 				} : null
 			});
 		}
@@ -521,7 +563,8 @@ function parseServerCodes(payload) {
 			validUntil: typeof value.validUntil === "string" ? value.validUntil : null,
 			llm: typeof llm === "object" && llm !== null ? {
 				provider: String(llm.provider ?? ""),
-				model: String(llm.model ?? "")
+				model: String(llm.model ?? ""),
+				reasoning: typeof llm.reasoning === "string" ? llm.reasoning : null
 			} : null,
 			createdAt: typeof value.createdAt === "string" ? value.createdAt : null
 		});
@@ -539,9 +582,16 @@ function sameInstant(a, b) {
 	const right = Date.parse(b);
 	return !Number.isNaN(left) && left === right;
 }
+/**
+* The override compares WHOLE, reasoning level included: a code minted at a different
+* effort serves different behavior, so it must not be reused. A differing level therefore
+* fails the match and the entry mints a NEW code — sync never modifies an existing one
+* (docs/registry.md). An absent level on either side compares as null, so an entry
+* without `reasoning` keeps matching the codes minted before the field existed.
+*/
 function sameLlm(a, b) {
 	if (a === null || b === null) return a === b;
-	return a.provider === b.provider && a.model === b.model;
+	return a.provider === b.provider && a.model === b.model && (a.reasoning ?? null) === (b.reasoning ?? null);
 }
 /**
 * The codes that ARE this entry: same activity URL, module and availability
@@ -846,7 +896,8 @@ async function readLock(lockPath) {
 }
 function registerCodes(program) {
 	const codes = program.command("codes").description("Manage activity codes on the Novedu server");
-	codes.command("create").description("Create a code for an activity YAML (validated server-side before storing)").requiredOption("--module <module>", "activity module: tutor, quiz, writing or coding").requiredOption("--file <url>", "public http(s) URL of the activity YAML").option("--start <iso>", "window start, ISO 8601 with explicit offset (e.g. 2026-07-07T08:00:00Z)").option("--end <iso>", "window end, ISO 8601 with explicit offset").option("--note <text>", "note shown in the codes list").option("--llm-provider <provider>", "LLM override provider (\"SCCH\" or \"Azure Foundry\"; needs --llm-model)").option("--llm-model <model>", "LLM override model id (needs --llm-provider)").option(...SERVER_OPTION$3).action(async (options) => {
+	codes.command("create").description("Create a code for an activity YAML (validated server-side before storing)").requiredOption("--module <module>", "activity module: tutor, quiz, writing or coding").requiredOption("--file <url>", "public http(s) URL of the activity YAML").option("--start <iso>", "window start, ISO 8601 with explicit offset (e.g. 2026-07-07T08:00:00Z)").option("--end <iso>", "window end, ISO 8601 with explicit offset").option("--note <text>", "note shown in the codes list").option("--llm-provider <provider>", "LLM override provider (\"SCCH\" or \"Azure Foundry\"; needs --llm-model)").option("--llm-model <model>", "LLM override model id (needs --llm-provider)").option("--llm-reasoning <level>", "LLM override reasoning effort (\"minimal\", \"low\", \"medium\" or \"high\"; needs the provider/model pair)").option(...SERVER_OPTION$3).action(async (options) => {
+		const llmGiven = options.llmProvider !== void 0 || options.llmModel !== void 0 || options.llmReasoning !== void 0;
 		await runApiRequest({
 			server: options.server,
 			path: "/api/codes",
@@ -857,10 +908,11 @@ function registerCodes(program) {
 				...options.start === void 0 ? {} : { validFrom: options.start },
 				...options.end === void 0 ? {} : { validUntil: options.end },
 				...options.note === void 0 ? {} : { note: options.note },
-				...options.llmProvider === void 0 && options.llmModel === void 0 ? {} : { llm: {
+				...llmGiven ? { llm: {
 					provider: options.llmProvider ?? "",
-					model: options.llmModel ?? ""
-				} }
+					model: options.llmModel ?? "",
+					...options.llmReasoning === void 0 ? {} : { reasoning: options.llmReasoning }
+				} } : {}
 			}
 		});
 	});
@@ -1024,6 +1076,10 @@ function appendInstructions(existing, instructions) {
 * client sent no system message, a leading one carrying only the teacher's instructions
 * is added. Everything else (messages, tools, tool_choice, temperature, stream, …)
 * passes through verbatim, so client-side tools and streaming are all preserved.
+*
+* A `reasoning` level (the effective activity/code setting) is pinned exactly like
+* `model` — it OVERWRITES whatever the client sent as `reasoning_effort`. Without one
+* the client's own `reasoning_effort` passes through untouched, like any other parameter.
 */
 function buildUpstreamChatBody(clientBody, opts) {
 	const clientMessages = Array.isArray(clientBody.messages) ? clientBody.messages : [];
@@ -1046,6 +1102,7 @@ function buildUpstreamChatBody(clientBody, opts) {
 		model: opts.model,
 		messages
 	};
+	if (opts.reasoning) upstream.reasoning_effort = opts.reasoning;
 	if (clientBody.stream === true) upstream.stream_options = {
 		...isRecord(clientBody.stream_options) ? clientBody.stream_options : {},
 		include_usage: true
@@ -1108,6 +1165,11 @@ function parseCoding(content) {
 		ok: false,
 		message: "This coding activity uses an unsupported llm.provider (use \"SCCH\" or \"Azure Foundry\")."
 	};
+	const reasoning = llm?.reasoning === void 0 ? void 0 : parseLenientReasoningLevel(llm.reasoning);
+	if (llm?.reasoning !== void 0 && !reasoning) return {
+		ok: false,
+		message: "This coding activity uses an unsupported llm.reasoning (use \"minimal\", \"low\", \"medium\" or \"high\")."
+	};
 	const instructions = asString$2(root.instructions);
 	if (!instructions) return {
 		ok: false,
@@ -1120,6 +1182,7 @@ function parseCoding(content) {
 			title: asString$2(root.title),
 			model,
 			provider,
+			reasoning,
 			instructions,
 			fragmentBlock: readFragmentBlock(root)
 		}
@@ -2411,6 +2474,11 @@ function parseQuiz(content) {
 		ok: false,
 		message: "This quiz uses an unsupported llm.provider (use \"SCCH\" or \"Azure Foundry\")."
 	};
+	const reasoning = llm?.reasoning === void 0 ? void 0 : parseLenientReasoningLevel(llm.reasoning);
+	if (llm?.reasoning !== void 0 && !reasoning) return {
+		ok: false,
+		message: "This quiz uses an unsupported llm.reasoning (use \"minimal\", \"low\", \"medium\" or \"high\")."
+	};
 	const quizFiles = Array.isArray(root.quiz_files) ? root.quiz_files : [];
 	const rawQuestions = Array.isArray(root.questions) ? root.questions : [];
 	if (rawQuestions.length === 0 && quizFiles.length === 0) return {
@@ -2453,6 +2521,7 @@ function parseQuiz(content) {
 			shuffle: asBool$1(root.shuffle, true),
 			model,
 			provider,
+			reasoning,
 			questionCount,
 			imageInput: asBool$1(llm?.imageInput, false),
 			discussionInstructions: asString$1(root.discussion?.instructions),
@@ -2686,6 +2755,7 @@ const TutorSchema = z.strictObject({
 	llm: z.strictObject({
 		model: z.string().meta({ description: "Model used for this tutor." }),
 		provider: providerSchema,
+		reasoning: reasoningLevelSchema,
 		imageInput: z.boolean().optional().meta({
 			default: true,
 			description: "Image uploads are enabled by default. Set to false to hide the upload UI for text-only tutors or non-vision-capable models."
@@ -2734,6 +2804,7 @@ async function loadAndBuildTutorPrompt(url, fetchImpl, opts = {}) {
 		prompt: assembled.prompt,
 		model: tutor.llm.model,
 		provider: tutor.llm.provider,
+		reasoning: tutor.llm.reasoning,
 		imageInput: tutor.llm.imageInput ?? true,
 		tools: tutor.tools,
 		anonymous: tutor.anonymous ?? true,
@@ -2784,6 +2855,11 @@ function parseWriting(content) {
 		ok: false,
 		message: "This writing activity uses an unsupported llm.provider (use \"SCCH\" or \"Azure Foundry\")."
 	};
+	const reasoning = llm?.reasoning === void 0 ? void 0 : parseLenientReasoningLevel(llm.reasoning);
+	if (llm?.reasoning !== void 0 && !reasoning) return {
+		ok: false,
+		message: "This writing activity uses an unsupported llm.reasoning (use \"minimal\", \"low\", \"medium\" or \"high\")."
+	};
 	const instructions = asString(root.instructions);
 	if (!instructions) return {
 		ok: false,
@@ -2799,6 +2875,7 @@ function parseWriting(content) {
 			anonymous: asBool(root.anonymous, false),
 			model,
 			provider,
+			reasoning,
 			instructions,
 			fragmentBlock: readFragmentBlock(root),
 			placeholder: asString(root.placeholder)
@@ -2905,7 +2982,8 @@ const promptDumpers = {
 				id: result.id,
 				llm: {
 					provider: result.provider,
-					model: result.model
+					model: result.model,
+					...result.reasoning ? { reasoning: result.reasoning } : {}
 				},
 				system: result.prompt,
 				tools: result.tools
@@ -2923,7 +3001,8 @@ const promptDumpers = {
 				id: quiz.id,
 				llm: {
 					provider: quiz.provider,
-					model: quiz.model
+					model: quiz.model,
+					...quiz.reasoning ? { reasoning: quiz.reasoning } : {}
 				},
 				grading: {
 					userMessageTemplate: QUIZ_ANSWER_MESSAGE_TEMPLATE,
@@ -2963,7 +3042,8 @@ const promptDumpers = {
 				id: writing.id,
 				llm: {
 					provider: writing.provider,
-					model: writing.model
+					model: writing.model,
+					...writing.reasoning ? { reasoning: writing.reasoning } : {}
 				},
 				system: writing.instructions
 			}
@@ -2985,7 +3065,8 @@ const promptDumpers = {
 				id: coding.id,
 				llm: {
 					provider: coding.provider,
-					model: coding.model
+					model: coding.model,
+					...coding.reasoning ? { reasoning: coding.reasoning } : {}
 				},
 				system: coding.instructions,
 				upstreamSystemMessage: typeof system?.content === "string" ? system.content : ""
@@ -3088,6 +3169,7 @@ const QuizYamlSchema = z.strictObject({
 	llm: z.strictObject({
 		model: z.string().min(1).meta({ description: "The model that grades answers and drives the per-question discussion chat." }),
 		provider: providerSchema,
+		reasoning: reasoningLevelSchema,
 		imageInput: z.boolean().optional().meta({
 			default: false,
 			description: "Default for all questions: students may attach photos (up to 3, 5 MB each) to their answers. The model must be vision-capable. A per-question imageInput overrides it."
@@ -3641,6 +3723,31 @@ function createJudgeBreaker() {
 }
 /** Consecutive fully-errored judge calls that mean "stop judging for the rest of the run". */
 const JUDGE_BREAKER_LIMIT = 3;
+/**
+* The spec a run's calls are actually served with, out of the TARGET activity's own spec
+* and the run's two override flags. TWO independent axes (docs/cli-eval.md):
+*
+* - the PAIR (`--llm-provider`/`--llm-model`) replaces provider+model **wholesale**, so a
+*   pair given without a level DROPS the file's level — the same bundle semantics a
+*   per-code LLM override has (`effectiveLlm`, docs/ai-models.md);
+* - the LEVEL (`--llm-reasoning`) replaces only the effort, on top of whichever pair won,
+*   which is what makes "the file's own model, at high effort" a one-flag run.
+*
+* The judge's flags reuse this with the EFFECTIVE grading spec as the activity, which is
+* why "no judge flag" means "judge exactly like the model under test", level included.
+*/
+function resolveEvalSpec(activity, pair, reasoning) {
+	const base = pair ?? activity;
+	return reasoning ? {
+		provider: base.provider,
+		model: base.model,
+		reasoning
+	} : base;
+}
+/** Do two specs describe the same call? Provider, model AND effort — all three matter. */
+function sameEvalSpec(a, b) {
+	return a.provider === b.provider && a.model === b.model && a.reasoning === b.reasoning;
+}
 /** Narrow a case to the tutor arm. */
 function isTutorCase(evalCase) {
 	return "conversation" in evalCase;
@@ -4333,7 +4440,7 @@ function formatEvalResult(result, source) {
 	lines.push(`  target: ${result.targetUrl}`);
 	if (result.kind === "tutor") lines.push(`  conversations: ${result.caseCount}`);
 	else lines.push(`  questions: ${result.evalFile.questions.length}   cases: ${result.caseCount}`);
-	lines.push(`  ${result.kind} model: ${result.llm.provider} / ${result.llm.model}`);
+	lines.push(`  ${result.kind} model: ${llmSpecText(result.llm)}`);
 	if (result.warnings.length) {
 		lines.push("");
 		lines.push(yellow(`${result.warnings.length} warning(s):`));
@@ -4387,6 +4494,14 @@ function mismatchLines(result) {
 	});
 }
 /**
+* One llm spec as the reports name it: `SCCH / gemma-4`, with ` (reasoning: high)`
+* appended whenever an effort level applies — two runs of one model at different efforts
+* behave differently, so the level belongs in the header.
+*/
+function llmSpecText(spec) {
+	return `${spec.provider} / ${spec.model}${spec.reasoning ? ` (reasoning: ${spec.reasoning})` : ""}`;
+}
+/**
 * The human report for ONE eval run: header (id, target, the EFFECTIVE llm — rendered
 * as `quiz-llm → override-llm` when `--llm-provider`/`--llm-model` was used, so a
 * comparison report can never be mistaken for a baseline one), one line per
@@ -4399,10 +4514,10 @@ function formatEvalReport(result, source) {
 		`  id: ${result.id}`,
 		`  target: ${result.target}`
 	];
-	const llm = result.llm.overrides ? `${result.llm.overrides.provider} / ${result.llm.overrides.model} ${yellow("→")} ${result.llm.provider} / ${result.llm.model} ${yellow("(override)")}` : `${result.llm.provider} / ${result.llm.model}`;
+	const llm = result.llm.overrides ? `${llmSpecText(result.llm.overrides)} ${yellow("→")} ${llmSpecText(result.llm)} ${yellow("(override)")}` : llmSpecText(result.llm);
 	lines.push(`  llm: ${llm}`);
 	const judge = result.llm.judge;
-	if (judge && (judge.provider !== result.llm.provider || judge.model !== result.llm.model)) lines.push(`  judge llm: ${judge.provider} / ${judge.model}${judge.overridden ? ` ${yellow("(override)")}` : ""}`);
+	if (judge && (judge.provider !== result.llm.provider || judge.model !== result.llm.model || judge.reasoning !== result.llm.reasoning)) lines.push(`  judge llm: ${llmSpecText(judge)}${judge.overridden ? ` ${yellow("(override)")}` : ""}`);
 	const unit = result.kind === "tutor" ? "conversation" : "case";
 	const generation = result.kind === "tutor" ? "generation" : "grading";
 	lines.push(`  ${unit}s: ${totals.cases} × ${totals.repeats} repeat(s) = ${totals.calls} ${generation} call(s)` + (result.judging === "off" ? "" : ` + ${totals.calls} judge call(s)`));
@@ -4494,7 +4609,7 @@ function shortSource$1(source) {
 function formatPromptDump(dump, sections, source) {
 	const lines = [green(`✔ Prompts — ${dump.kind}`) + dim(` — ${source}`)];
 	lines.push(`  id: ${dump.id}`);
-	lines.push(`  provider: ${dump.llm.provider}   model: ${dump.llm.model}`);
+	lines.push(`  provider: ${dump.llm.provider}   model: ${dump.llm.model}` + (dump.llm.reasoning ? `   reasoning: ${dump.llm.reasoning}` : ""));
 	if (dump.kind === "tutor" && dump.tools.length > 0) lines.push(`  tools: ${dump.tools.join(", ")}`);
 	lines.push(`  prompts: ${sections.length}`);
 	for (const section of sections) lines.push(`    ${section.name}: ${section.text.length} chars`);
@@ -4530,10 +4645,18 @@ function inline(text) {
 function quote(text) {
 	return text.replace(/\s+$/, "").split(/\r?\n/).map((line) => line ? `> ${line}` : ">").join("\n");
 }
+/**
+* One spec: `SCCH / gemma-4`, and `Azure Foundry / gpt-5.6-terra (reasoning: high)` when
+* an effort level applies. The level is part of a run's identity — two runs of one model
+* at different efforts produce different behavior — so it must be readable off the report.
+*/
+function specText(spec) {
+	return `${spec.provider} / ${spec.model}${spec.reasoning ? ` (reasoning: ${spec.reasoning})` : ""}`;
+}
 /** `SCCH / gemma-4`, or `SCCH / gemma-4 → Azure Foundry / gpt-5-mini (override)`. */
 function llmText(llm) {
-	const effective = `${llm.provider} / ${llm.model}`;
-	return llm.overrides ? `${llm.overrides.provider} / ${llm.overrides.model} → ${effective} (override)` : effective;
+	const effective = specText(llm);
+	return llm.overrides ? `${specText(llm.overrides)} → ${effective} (override)` : effective;
 }
 /**
 * The judge's pair, but ONLY when it differs from the grading pair — a judge line that
@@ -4543,8 +4666,8 @@ function llmText(llm) {
 function judgeLlmText(llm) {
 	const judge = llm.judge;
 	if (!judge) return void 0;
-	if (judge.provider === llm.provider && judge.model === llm.model) return void 0;
-	return `${judge.provider} / ${judge.model}${judge.overridden ? " (override)" : ""}`;
+	if (judge.provider === llm.provider && judge.model === llm.model && judge.reasoning === llm.reasoning) return;
+	return `${specText(judge)}${judge.overridden ? " (override)" : ""}`;
 }
 /** `15,420 / 12,300 / 2,810`, or an em dash when nothing was reported. */
 function usageCell(usage) {
@@ -4945,7 +5068,8 @@ const CodingYamlSchema = z.strictObject({
 	title: z.string().optional().meta({ description: "Optional label shown to the student on the /<code> connection page." }),
 	llm: z.strictObject({
 		model: z.string().min(1).meta({ description: "The model that answers. SERVER-ONLY and PINNED: the proxy always uses this model and ignores whatever model the coding agent sends." }),
-		provider: providerSchema
+		provider: providerSchema,
+		reasoning: reasoningLevelSchema
 	}).meta({
 		id: "llm",
 		description: "The pinned model and provider that answer coding requests."
@@ -5022,7 +5146,8 @@ const WritingYamlSchema = z.strictObject({
 	}),
 	llm: z.strictObject({
 		model: z.string().min(1).meta({ description: "The model that drives the feedback chat." }),
-		provider: providerSchema
+		provider: providerSchema,
+		reasoning: reasoningLevelSchema
 	}).meta({
 		id: "llm",
 		description: "The model and provider that back the writing coach."
@@ -5277,6 +5402,26 @@ function parsePair(flag, provider, model) {
 	};
 }
 /**
+* One reasoning-effort flag: absent, or one of the four known levels. Checked here rather
+* than left to the server for the same reason `parsePair` checks the provider — a typo
+* must cost nothing, not a whole run's worth of terminal 400s.
+*
+* Deliberately INDEPENDENT of its pair flag (unlike the pair's both-or-nothing rule): the
+* common comparison run is "same model, different effort", so `--llm-reasoning` alone is
+* a first-class invocation rather than a usage error.
+*/
+function parseReasoning(flag, value) {
+	if (value === void 0) return { ok: true };
+	if (!REASONING_LEVELS.includes(value)) return {
+		ok: false,
+		message: `Unknown --${flag} "${value}": expected ${REASONING_LEVELS.map((level) => `"${level}"`).join(", ")}.`
+	};
+	return {
+		ok: true,
+		reasoning: value
+	};
+}
+/**
 * The optional `usage: { input, cachedInput, output }` of a 200 response, defensively:
 * anything that is not three finite numbers is simply absent (an older server, or one
 * whose provider reports nothing, must never break a run).
@@ -5526,14 +5671,24 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 		failJson({ message: override.message });
 		return;
 	}
+	const overrideReasoning = parseReasoning("llm-reasoning", options.llmReasoning);
+	if (!overrideReasoning.ok) {
+		failJson({ message: overrideReasoning.message });
+		return;
+	}
 	const judgeOverride = parsePair("judge-llm", options.judgeLlmProvider, options.judgeLlmModel);
 	if (!judgeOverride.ok) {
 		failJson({ message: judgeOverride.message });
 		return;
 	}
+	const judgeOverrideReasoning = parseReasoning("judge-llm-reasoning", options.judgeLlmReasoning);
+	if (!judgeOverrideReasoning.ok) {
+		failJson({ message: judgeOverrideReasoning.message });
+		return;
+	}
 	const judging = options.judgeFeedback !== false;
-	if (!judging && judgeOverride.llm) {
-		failJson({ message: "--judge-llm-provider/--judge-llm-model cannot be combined with --no-judge-feedback: the first configures the feedback judge, the second switches it off." });
+	if (!judging && (judgeOverride.llm || judgeOverrideReasoning.reasoning)) {
+		failJson({ message: "--judge-llm-provider/--judge-llm-model/--judge-llm-reasoning cannot be combined with --no-judge-feedback: the first configure the feedback judge, the second switches it off." });
 		return;
 	}
 	const expansion = expandSources(pathsOrUrls);
@@ -5599,16 +5754,17 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 		if (!check) continue;
 		const activityLlm = {
 			provider: check.llm.provider,
-			model: check.llm.model
+			model: check.llm.model,
+			...check.llm.reasoning ? { reasoning: check.llm.reasoning } : {}
 		};
-		const effective = override.llm ?? activityLlm;
-		const judgeLlm = judgeOverride.llm ?? effective;
+		const effective = resolveEvalSpec(activityLlm, override.llm, overrideReasoning.reasoning);
+		const judgeLlm = resolveEvalSpec(effective, judgeOverride.llm, judgeOverrideReasoning.reasoning);
 		const llm = {
 			...effective,
-			...override.llm ? { overrides: activityLlm } : {},
+			...sameEvalSpec(effective, activityLlm) ? {} : { overrides: activityLlm },
 			...judging ? { judge: {
 				...judgeLlm,
-				overridden: judgeOverride.llm !== void 0
+				overridden: judgeOverride.llm !== void 0 || judgeOverrideReasoning.reasoning !== void 0
 			} } : {}
 		};
 		const label = files.length > 1 ? `(${fileIndex}/${files.length}) ${check.evalFile.id}` : check.evalFile.id;
@@ -5654,7 +5810,7 @@ async function runEvalCommand(pathsOrUrls, options, seams = {}) {
 	process.exitCode = batchPassed(batch) ? 0 : 1;
 }
 function registerEval(program) {
-	program.command("eval").description("Run an eval file (quiz golden answers, or tutor conversations) against the real activity path and report the result").argument("<evalPathOrUrl...>", "one or more eval YAML files (paths, http(s)/file URLs, or a quoted glob pattern)").option("--server <url>", "Novedu server base URL (defaults to the NOVEDU_SERVER env var, then production)").option("--concurrency <n>", "cases in flight per file", String(CONCURRENCY_DEFAULT)).option("--repeats <n>", "run every case N times (quiz: take the majority verdict)", "1").option("--llm-provider <provider>", "run with this provider instead of the activity's (\"SCCH\" or \"Azure Foundry\"; needs --llm-model)").option("--llm-model <model>", "run with this model instead of the activity's (needs --llm-provider)").option("--no-judge-feedback", "skip the LLM audit of what the model wrote (halves the LLM calls)").option("--judge-llm-provider <provider>", "judge with this provider (\"SCCH\" or \"Azure Foundry\"; needs --judge-llm-model)").option("--judge-llm-model <model>", "judge with this model instead of the one under test (needs --judge-llm-provider)").option("--json", "print the machine-readable batch report on stdout").option("--out <file>", "additionally write the machine-readable batch report to a file").option("--report <file>", "additionally write a readable Markdown report to a file").addHelpText("after", `
+	program.command("eval").description("Run an eval file (quiz golden answers, or tutor conversations) against the real activity path and report the result").argument("<evalPathOrUrl...>", "one or more eval YAML files (paths, http(s)/file URLs, or a quoted glob pattern)").option("--server <url>", "Novedu server base URL (defaults to the NOVEDU_SERVER env var, then production)").option("--concurrency <n>", "cases in flight per file", String(CONCURRENCY_DEFAULT)).option("--repeats <n>", "run every case N times (quiz: take the majority verdict)", "1").option("--llm-provider <provider>", "run with this provider instead of the activity's (\"SCCH\" or \"Azure Foundry\"; needs --llm-model)").option("--llm-model <model>", "run with this model instead of the activity's (needs --llm-provider)").option("--llm-reasoning <level>", "run at this reasoning effort (\"minimal\", \"low\", \"medium\" or \"high\"); on its own it keeps the activity's model").option("--no-judge-feedback", "skip the LLM audit of what the model wrote (halves the LLM calls)").option("--judge-llm-provider <provider>", "judge with this provider (\"SCCH\" or \"Azure Foundry\"; needs --judge-llm-model)").option("--judge-llm-model <model>", "judge with this model instead of the one under test (needs --judge-llm-provider)").option("--judge-llm-reasoning <level>", "judge at this reasoning effort (\"minimal\", \"low\", \"medium\" or \"high\"); on its own it keeps the judge's model").option("--json", "print the machine-readable batch report on stdout").option("--out <file>", "additionally write the machine-readable batch report to a file").option("--report <file>", "additionally write a readable Markdown report to a file").addHelpText("after", `
 Examples:
   # Evaluate one quiz's golden answers
   $ novedu-cli eval ./0010-welcome-quiz.eval.yaml
@@ -5672,6 +5828,9 @@ Examples:
   # How would this rubric perform on another model? (both flags, always together)
   $ novedu-cli eval ./my-quiz.eval.yaml --llm-provider "Azure Foundry" --llm-model gpt-5-mini
 
+  # Same model, more thinking: the level alone keeps the activity's provider/model
+  $ novedu-cli eval ./my-quiz.eval.yaml --llm-reasoning high
+
   # A strong judge over the quiz's own grader — the recommended pairing
   $ novedu-cli eval ./my-quiz.eval.yaml --judge-llm-provider "Azure Foundry" --judge-llm-model gpt-5.6-terra
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@novedu/cli",
-  "version": "0.23.0",
+  "version": "0.24.1",
   "description": "Command-line companion for the Novedu chat app. Validates tutor, fragment, quiz, writing, coding and eval YAML definitions, dumps the exact LLM prompts an activity produces, evaluates a quiz's grading rubric against golden answers and replays scripted conversations against a tutor; signs in with Entra ID and manages codes, app-hosted files and images over the app's API.",
   "type": "module",
   "repository": {