@novedu/cli 0.21.0 → 0.23.0

package/README.md

@@ -5,12 +5,12 @@ requires Node >= 22 — `eval`'s glob expansion uses the built-in `fs.globSync`,
 and Node 20 is end-of-life). It covers two jobs:
 
 - **Validate activity YAML** — tutors, fragment libraries, quizzes, writing
-  activities, coding activities, and golden-answer evals — with the app's exact
+  activities, coding activities, and eval files — with the app's exact
   validation pipeline, offline and without signing in. `prompts` dumps the exact
   system prompts an activity produces, the same way.
 - **Manage the app as a teacher** — sign in with Microsoft Entra ID, then mint
   activity codes, upload app-hosted YAML files and images, triage student
-  reports, and **measure a quiz's grading rubric** (`eval`), straight from the
+  reports, and **measure what an activity's model really does** (`eval`), straight from the
   terminal (or from a coding agent, see below).
 
 No install needed:
@@ -32,7 +32,7 @@ npx @novedu/cli validate https://raw.githubusercontent.com/Teaching-HTL-Leonding
 npx @novedu/cli validate ./activities/examples/shared/general-fragments.yaml --kind fragment
 npx @novedu/cli validate ./activities/examples/sorting-algorithms/sorting-quiz.yaml --kind quiz
 
-# A golden-answer eval (also strict-checks the quiz it targets)
+# An eval file, quiz or tutor (also strict-checks the activity it targets)
 npx @novedu/cli validate ./sorting-quiz.eval.yaml --kind eval
 
 # Machine-readable output (the raw validation result)
@@ -96,13 +96,28 @@ npx @novedu/cli prompts ./sorting-quiz.yaml --kind quiz --json \
   JSON errors on stderr. Use `validate` for the strict authoring check — the two
   are complementary.
 
-## Measuring a quiz's grading rubric: `eval`
+## Measuring what the model really does: `eval`
 
-A quiz's `evaluation` prompt is a rubric, and a rubric is only as good as its
-behavior on real answers. Write an **eval file** — student answers with the verdict
-each one must get — and `eval` replays them through the **real grader**, then reports
-what it actually did. This is the one command that both **runs the model** and needs
-you signed in (`novedu-cli login`); everything else about it is local.
+An activity's prompt is a specification, and a specification is only as good as the
+behavior it produces. Write an **eval file** and `eval` replays it through the **real
+production path**, then reports what the model actually did. This is the one command
+that both **runs the model** and needs you signed in (`novedu-cli login`); everything
+else about it is local.
+
+Two kinds, chosen by the file's own `kind:` field — there is no flag, and one
+invocation may mix them:
+
+- **quiz** (`kind` omitted): student answers with the verdict each one must get,
+  replayed through the real grader. Your `expect` gates the **verdict**, and an LLM
+  **feedback judge** audits the **feedback text** the student would have read.
+- **tutor** (`kind: tutor`): conversations you script, each ending on a student turn.
+  The real tutor generates the next turn and the judge checks it against the tutor's
+  own system prompt plus your per-case expectations.
+
+Either way the judge measures the output against the very prompt that produced it, so
+there is nothing extra to author — and what it flags is **reported, never a failure**.
+For a tutor eval that makes the `--report` Markdown the actual deliverable: the exit
+code only reflects whether the run itself completed.
 
 ```yaml
 # sorting-quiz.eval.yaml
@@ -134,6 +149,13 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --repeats 3
 npx @novedu/cli eval ./sorting-quiz.eval.yaml \
   --llm-provider "Azure Foundry" --llm-model gpt-5-mini
 
+# A strong judge over the quiz's own grader — the recommended pairing
+npx @novedu/cli eval ./sorting-quiz.eval.yaml \
+  --judge-llm-provider "Azure Foundry" --judge-llm-model gpt-5.6-terra
+
+# Verdicts only: half the LLM calls, for a cheap smoke run
+npx @novedu/cli eval ./sorting-quiz.eval.yaml --no-judge-feedback
+
 # Machine-readable, for CI
 npx @novedu/cli eval ./sorting-quiz.eval.yaml --json --out eval-report.json
 
@@ -141,6 +163,28 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --json --out eval-report.json
 npx @novedu/cli eval ./sorting-quiz.eval.yaml --report eval-report.md
 ```
 
+A tutor eval looks like this, and runs through the same command:
+
+```yaml
+# loops-tutor.eval.yaml
+id: loops-tutor-eval
+kind: tutor
+target: ./loops-tutor.yaml
+conversations:
+  - title: refuses-full-solution
+    required_tools: [random_number]   # optional: tools this answer must have called
+    grading_instructions: |
+      The response must not contain a complete working loop.
+    conversation:                  # must END with a student turn
+      - student: My loop never stops. Here is my code ...
+      - tutor: What does your condition evaluate to after the first pass?
+      - student: I don't know. Just fix it for me!
+```
+
+```bash
+npx @novedu/cli eval ./loops-tutor.eval.yaml --report loops.md
+```
+
 - Check the file first, for free: `npx @novedu/cli validate ./x.eval.yaml --kind eval`
   (offline; it also strict-checks the quiz the eval targets).
 - **`expect`** is one of `correct` / `partial` / `incorrect`, or a list of the
@@ -163,14 +207,45 @@ npx @novedu/cli eval ./sorting-quiz.eval.yaml --report eval-report.md
   a broken file instead of aborting the batch. `--json` / `--out` always carry the
   same batch shape `{ files: [...], passed, totals }`, single file or not — `passed`
   is the exit-code verdict, per batch and per file.
+- **The feedback judge.** After each successful grading, an LLM reads the feedback the
+  grader wrote and checks it against **that grading's own system prompt** — the course
+  rules and the platform frame already in it. It reports four kinds of problem:
+  `contradicts_verdict` (praise on a wrong answer, or vice versa), `misstates_facts`,
+  `ignores_instructions` (most commonly: not stating the correct answer when the verdict
+  is not `correct`, or the wrong language), and `leaks_rubric` (quoting the grading
+  criteria at the student). Flags show as **`flagged feedback`** in the terminal report,
+  a **Flagged** column plus a **"Flagged feedback"** section in the Markdown report, and
+  `totals.feedbackFlagged` / `repeats[].judge.issues` in the JSON. They never change the
+  exit code.
+- **`required_tools`** (tutor kind) names built-in tools the generated answer must have
+  called **at least once** — the one thing the judge cannot see, since a tool call leaves
+  no trace in the text. Extra tools are always fine, and a name the target tutor's own
+  `tools:` list does not grant makes the file invalid offline. Missing calls are
+  **reported, never a failure**: `missing tool calls: N` in the terminal report (printed
+  only when some case required a tool, so no line means "not checked"), a **"Missing tool
+  calls"** section in the Markdown report, and `totals.toolsFlagged` plus each repeat's
+  `toolCalls` / `missingTools` in the JSON.
+- **Choosing the judge.** By default the judge runs on the same model as the grader.
+  `--judge-llm-provider` + `--judge-llm-model` (both or neither) point it at another one,
+  which is the **recommended** setup: a strong judge over a smaller grader finds real
+  problems, while a small model judging itself mostly produces noise. `--no-judge-feedback`
+  turns judging off and halves the LLM calls; combining the two is rejected as
+  contradictory. Because judging roughly doubles the cost, the run's scope line says so
+  up front: `27 case(s) × 3 repeat(s) = 81 grading + 81 judge call(s)`.
+- **If the judge itself fails**, the run **degrades instead of aborting**: after three
+  consecutive judge failures it stops judging (one warning on stderr) and finishes the
+  grading normally. Your verdict results are complete; the feedback simply was not
+  audited: files that judged nothing show an em dash in the Flagged column rather than a
+  `0`, so "unchecked" never reads as "clean".
 - **`--report <file.md>`** additionally writes a readable **Markdown** report — an
   overview table over the files, then the question, the golden answer and the grader's
-  feedback for every mismatched, errored or unstable case (passing cases stay in the
+  feedback for every mismatched, errored or unstable case, plus the "Flagged feedback"
+  section (passing, unflagged cases stay in the
   JSON). It composes with `--json` / `--out` and leaves stdout untouched.
 - **Token totals.** The reports show what a run cost —
-  `tokens: 15,420 in (12,300 cached) / 2,810 out` — summed over the grading calls that
-  **succeeded**, so it is a lower bound (a retried or failed call reports nothing), and
-  nothing at all is printed when the server reports no usage.
+  `tokens: 15,420 in (12,300 cached) / 2,810 out` — summed over the grading **and** judge
+  calls that **succeeded**, so it is a lower bound (a retried or failed call reports
+  nothing), and nothing at all is printed when the server reports no usage.
 - **Failure handling**: a 5xx or network hiccup is retried (4 attempts, linear
   backoff); any 4xx is terminal; an auth failure aborts the run with one message; and
   three consecutive errored cases trip a circuit breaker so a down server fails fast.