@novedu/cli 0.22.0 → 0.24.0

package/README.md

@@ -5,12 +5,12 @@ requires Node >= 22 — `eval`'s glob expansion uses the built-in `fs.globSync`,
 and Node 20 is end-of-life). It covers two jobs:
 
 - **Validate activity YAML** — tutors, fragment libraries, quizzes, writing
-  activities, coding activities, and golden-answer evals — with the app's exact
+  activities, coding activities, and eval files — with the app's exact
   validation pipeline, offline and without signing in. `prompts` dumps the exact
   system prompts an activity produces, the same way.
 - **Manage the app as a teacher** — sign in with Microsoft Entra ID, then mint
   activity codes, upload app-hosted YAML files and images, triage student
-  reports, and **measure a quiz's grading rubric** (`eval`), straight from the
+  reports, and **measure what an activity's model really does** (`eval`), straight from the
   terminal (or from a coding agent, see below).
 
 No install needed:
@@ -32,7 +32,7 @@ npx @novedu/cli validate https://raw.githubusercontent.com/Teaching-HTL-Leonding
 npx @novedu/cli validate ./activities/examples/shared/general-fragments.yaml --kind fragment
 npx @novedu/cli validate ./activities/examples/sorting-algorithms/sorting-quiz.yaml --kind quiz
 
-# A golden-answer eval (also strict-checks the quiz it targets)
+# An eval file, quiz or tutor (also strict-checks the activity it targets)
 npx @novedu/cli validate ./sorting-quiz.eval.yaml --kind eval
 
 # Machine-readable output (the raw validation result)
@@ -85,7 +85,8 @@ npx @novedu/cli prompts ./sorting-quiz.yaml --kind quiz --json \
   re-implementation), so what you see is what the model gets: fragments resolved,
   and — for a compound quiz — every `quiz_files` include fetched, each imported
   question carrying its source quiz's preamble.
-- Every dump carries `{ kind, id, llm: { provider, model } }`. A **quiz** adds
+- Every dump carries `{ kind, id, llm: { provider, model, reasoning? } }` (the
+  reasoning level only when the file sets one). A **quiz** adds
   `grading` (a `system` prompt per question, the user-message templates and the
   grader's JSON-Schema response contract) and `discussion` (the chat's `system`
   prompt, the three seed-message templates and the verdict wording). A **coding**
@@ -96,18 +97,28 @@ npx @novedu/cli prompts ./sorting-quiz.yaml --kind quiz --json \
   JSON errors on stderr. Use `validate` for the strict authoring check — the two
   are complementary.
 
-## Measuring a quiz's grading rubric: `eval`
+## Measuring what the model really does: `eval`
 
-A quiz's `evaluation` prompt is a rubric, and a rubric is only as good as its
-behavior on real answers. Write an **eval file** — student answers with the verdict
-each one must get — and `eval` replays them through the **real grader**, then reports
-what it actually did. This is the one command that both **runs the model** and needs
-you signed in (`novedu-cli login`); everything else about it is local.
+An activity's prompt is a specification, and a specification is only as good as the
+behavior it produces. Write an **eval file** and `eval` replays it through the **real
+production path**, then reports what the model actually did. This is the one command
+that both **runs the model** and needs you signed in (`novedu-cli login`); everything
+else about it is local.
 
-It checks **both halves** of a grading: your `expect` gates the **verdict**, and an LLM
-**feedback judge** audits the **feedback text** the student would have read — measured
-against the quiz's own grading prompt, so there is nothing extra to author. Flagged
-feedback is **reported, never a failure**.
+Two kinds, chosen by the file's own `kind:` field — there is no flag, and one
+invocation may mix them:
+
+- **quiz** (`kind` omitted): student answers with the verdict each one must get,
+  replayed through the real grader. Your `expect` gates the **verdict**, and an LLM
+  **feedback judge** audits the **feedback text** the student would have read.
+- **tutor** (`kind: tutor`): conversations you script, each ending on a student turn.
+  The real tutor generates the next turn and the judge checks it against the tutor's
+  own system prompt plus your per-case expectations.
+
+Either way the judge measures the output against the very prompt that produced it, so
+there is nothing extra to author — and what it flags is **reported, never a failure**.
+For a tutor eval that makes the `--report` Markdown the actual deliverable: the exit
+code only reflects whether the run itself completed.
 
 ```yaml
 # sorting-quiz.eval.yaml
@@ -139,6 +150,9 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --repeats 3
 npx @novedu/cli eval ./sorting-quiz.eval.yaml \
   --llm-provider "Azure Foundry" --llm-model gpt-5-mini
 
+# Same model, more thinking: the level alone keeps the activity's provider/model
+npx @novedu/cli eval ./sorting-quiz.eval.yaml --llm-reasoning high
+
 # A strong judge over the quiz's own grader — the recommended pairing
 npx @novedu/cli eval ./sorting-quiz.eval.yaml \
   --judge-llm-provider "Azure Foundry" --judge-llm-model gpt-5.6-terra
@@ -153,6 +167,28 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --json --out eval-report.json
 npx @novedu/cli eval ./sorting-quiz.eval.yaml --report eval-report.md
 ```
 
+A tutor eval looks like this, and runs through the same command:
+
+```yaml
+# loops-tutor.eval.yaml
+id: loops-tutor-eval
+kind: tutor
+target: ./loops-tutor.yaml
+conversations:
+  - title: refuses-full-solution
+    required_tools: [random_number]   # optional: tools this answer must have called
+    grading_instructions: |
+      The response must not contain a complete working loop.
+    conversation:                  # must END with a student turn
+      - student: My loop never stops. Here is my code ...
+      - tutor: What does your condition evaluate to after the first pass?
+      - student: I don't know. Just fix it for me!
+```
+
+```bash
+npx @novedu/cli eval ./loops-tutor.eval.yaml --report loops.md
+```
+
 - Check the file first, for free: `npx @novedu/cli validate ./x.eval.yaml --kind eval`
   (offline; it also strict-checks the quiz the eval targets).
 - **`expect`** is one of `correct` / `partial` / `incorrect`, or a list of the
@@ -185,12 +221,28 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --report eval-report.md
   a **Flagged** column plus a **"Flagged feedback"** section in the Markdown report, and
   `totals.feedbackFlagged` / `repeats[].judge.issues` in the JSON. They never change the
   exit code.
-- **Choosing the judge.** By default the judge runs on the same model as the grader.
-  `--judge-llm-provider` + `--judge-llm-model` (both or neither) point it at another one,
-  which is the **recommended** setup: a strong judge over a smaller grader finds real
-  problems, while a small model judging itself mostly produces noise. `--no-judge-feedback`
-  turns judging off and halves the LLM calls; combining the two is rejected as
-  contradictory. Because judging roughly doubles the cost, the run's scope line says so
+- **`required_tools`** (tutor kind) names built-in tools the generated answer must have
+  called **at least once** — the one thing the judge cannot see, since a tool call leaves
+  no trace in the text. Extra tools are always fine, and a name the target tutor's own
+  `tools:` list does not grant makes the file invalid offline. Missing calls are
+  **reported, never a failure**: `missing tool calls: N` in the terminal report (printed
+  only when some case required a tool, so no line means "not checked"), a **"Missing tool
+  calls"** section in the Markdown report, and `totals.toolsFlagged` plus each repeat's
+  `toolCalls` / `missingTools` in the JSON.
+- **Choosing what runs.** `--llm-provider` + `--llm-model` (both or neither) replace the
+  activity's **whole** `llm:` block for the run — its reasoning level included, so the
+  file's level is dropped unless `--llm-reasoning <level>` restates it.
+  `--llm-reasoning` on its own changes only the effort and keeps the file's
+  provider/model — the "same model, more thinking" comparison run.
+- **Choosing the judge.** By default the judge runs on the same model **and effort** as
+  the grader. `--judge-llm-provider` + `--judge-llm-model` (both or neither) point it at
+  another one — replacing the whole spec, exactly like the grading flags — which is the
+  **recommended** setup: a strong judge over a smaller grader finds real
+  problems, while a small model judging itself mostly produces noise.
+  `--judge-llm-reasoning` sets the judge's effort on its own, no pair needed.
+  `--no-judge-feedback` turns judging off and halves the LLM calls; combining it with any
+  of the judge flags is rejected as contradictory. Because judging roughly doubles the
+  cost, the run's scope line says so
   up front: `27 case(s) × 3 repeat(s) = 81 grading + 81 judge call(s)`.
 - **If the judge itself fails**, the run **degrades instead of aborting**: after three
   consecutive judge failures it stops judging (one warning on stderr) and finishes the
@@ -215,7 +267,7 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --report eval-report.md
   run localises the problem for the cost of a single grading call.
 - **Caveat**: a green run certifies **the file you ran it on**, not the app-hosted
   copy a live code serves — upload it (`files upload`) afterwards. An override run
-  certifies the override pair, not the quiz's configured `llm`.
+  certifies the override, not the quiz's configured `llm`.
 
 ## Authentication
 
@@ -260,7 +312,7 @@ authoritative — the CLI sends your input as-is and relays the server's answer.
 ```
 codes create  --module <tutor|quiz|writing|coding> --file <url>
               [--start <iso>] [--end <iso>] [--note <text>]
-              [--llm-provider <p> --llm-model <m>]
+              [--llm-provider <p> --llm-model <m>] [--llm-reasoning <level>]
 codes list    [--search <q>] [--module <m>] [--all]
 codes sync    <registry-file> [--lock <path>] [--dry-run] [--json]
 files upload  <name> [--kind <tutor|fragment|quiz|writing|coding>]
@@ -278,7 +330,10 @@ images list   [--search <q>] [--all]
   server-side before the code is stored, and the response includes the
   shareable `url`. `--start`/`--end` must be ISO 8601 **with an explicit
   offset or `Z`** (e.g. `2026-07-07T08:00:00Z`); the
-  `--llm-provider`/`--llm-model` override pair is both-or-nothing.
+  `--llm-provider`/`--llm-model` override pair is both-or-nothing, and
+  `--llm-reasoning <level>` (`minimal`, `low`, `medium` or `high`) rides on top of
+  the pair — it is rejected without it. The override replaces the activity's whole
+  `llm:` block, so leaving the level out also drops the file's.
 - `codes sync <registry-file>` mints codes for a whole **course** at once — see
   [Many activities at once](#many-activities-at-once-codes-sync) below.
 - `files upload <name>` is an **upsert**: creating a new file requires
@@ -376,11 +431,13 @@ activity-codes:
 - **Groups decide the module:** `quizzes`, `tutors`, `writing`, `coding`. Each
   entry gives either `file` (relative to `base-url`, which must end in `/`) or
   an absolute `url`, plus any of `start`/`end` (ISO 8601 **with an offset or
-  `Z`**, whole seconds), `note`, and an `llm: {provider, model}` override.
+  `Z`**, whole seconds), `note`, and an `llm: {provider, model, reasoning?}`
+  override.
 - **Keys are yours and must be unique across all groups** — lowercase letters,
   digits and hyphens. Your material references the key; the lock file maps it to
   the code.
-- **Re-runs are safe.** An entry whose activity, window and model override match
+- **Re-runs are safe.** An entry whose activity, window and LLM override
+  (provider, model and reasoning level) match
   an existing code of yours **reuses** that code; only entries without a match
   are minted. So `codes sync` after every edit is the normal workflow, and the
   first run against already-minted codes should report all-reused.