@intellectif/lk-core 1.4.0 → 1.6.0

package/CHANGELOG.md

@@ -1,5 +1,107 @@
 # @intellectif/lk-core
 
+## 1.6.0
+
+### Minor Changes
+
+- [#120](https://github.com/intellectif/learning-kit/pull/120) [`d5ee618`](https://github.com/intellectif/learning-kit/commit/d5ee6185d2a167794352b218486881c405b13b52) Thanks [@diego28e](https://github.com/diego28e)! - **Exporting and importing content**: list and re-point every file an item or a group uses, keep what each recording says for the author's side only, and describe content in one JSON Schema.
+
+  **Action required:** none. Every field is optional and every function new. Two cases to check:
+
+  - **A field you already store under these names.** If your content keeps a media `transcript` that is not a string, or a `script` that is not a list of `{ speaker, text }` turns, it now fails validation.
+  - **A `switch` over the critic's codes.** The critic has three new codes, so a `switch` over them without a `default` branch needs them.
+
+  - **Files:** `listMediaRefs(entry)` lists every media address in an activity or an item group: each one's path, kind, owner and language. It covers an item's `media`, a multiple-choice option's `media`, dictation's and read-aloud's `slowMedia`, and a stimulus's `media`, with their captions, tracks and posters. A recording with no file yet is listed with `url: null`.
+
+    ```ts
+    const files = listMediaRefs(entry).filter((ref) => ref.url !== null);
+    ```
+
+    `mapMediaRefs(entry, fn)` returns a copy with each address replaced and nothing else changed. With `(ref) => ref.url` the copy equals the input. A type you register says where it keeps files with `mediaSlots` on its descriptor.
+
+    ```ts
+    const imported = mapMediaRefs(entry, (ref) =>
+      ref.url === null ? null : storedUrlFor(ref.url)
+    );
+    ```
+
+  - **What a recording says:** an item's media, a listening option's media and a stimulus can carry an author-only `transcript` (plain text) and `script` (turns, each a `speaker` and a `text`).
+
+    - Both are removed by `redact()` and `redactItemGroup()` at every reveal level, and refused by `assertRedacted()`. No request for a learner's AI help reads them.
+    - `transcriptOf(holder)` gives the author-side text, the script as `Speaker: text` lines when there is no transcript. Pass it to `aiDraftsRequest` as its `source`.
+
+    ```jsonc
+    {
+      "type": "audio",
+      "url": "https://cdn.example/station.mp3",
+      "transcript": "Where does the train leave from? Platform four.",
+      "script": [
+        { "speaker": "Anna", "text": "Where does the train leave from?" },
+        { "speaker": "Ben", "text": "Platform four." }
+      ]
+    }
+    ```
+
+  - **Practice scored in the browser:** `withoutAuthorOnly(entry)` returns an item or a group without its author-only fields, and with its key and your own fields. Unlike `redact()`, whose output scorers refuse, it scores exactly as the stored item. Every vector of the grade-stability corpus shows it. Exams still go through `redact()`.
+
+    ```ts
+    res.json(withoutAuthorOnly(practiceItem));
+    ```
+
+  - **The critic's new codes:** `media_script_differs` (a transcript and a script that say different words), `media_transcript_differs` (a dictation's or a read-aloud's recording saying other words than the item), and `media_transcript_unused` (on a picture or an embed). The AI critic now reads a recording's transcript and script. It no longer reads a caption track's address or language, a poster, or a speaker's label.
+  - **JSON Schema:** `contentJsonSchema()` is one draft 2020-12 document whose root is a sequence entry: any registered activity type, or an item group with typed items. `jsonSchemaFor(type, { target: 'draft-2020-12' })` gives one type's schema in that draft. Draft 7 stays the default, so the existing exports do not change.
+
+    ```ts
+    writeFileSync("schema.json", JSON.stringify(contentJsonSchema(), null, 2));
+    ```
+
+  Guide: https://github.com/intellectif/learning-kit/blob/main/docs/authoring.md#exporting-and-importing-content-16
+
+## 1.5.0
+
+### Minor Changes
+
+- [#112](https://github.com/intellectif/learning-kit/pull/112) [`3fc1049`](https://github.com/intellectif/learning-kit/commit/3fc1049d67e14f6f4a12c235f19952a988472978) Thanks [@diego28e](https://github.com/diego28e)! - **Assisted grading**: a model grades a written response for the record, and in an exam no model's grade counts until it has been measured against your teachers.
+
+  **Action required:** none. Nothing grades until you call it. Four union members are new: `DeferredReason` gains `awaiting_human_review`, `AiRefusal` gains `misses-criteria` and `out-of-scale`, and `AiCheckCase['feature']` gains `grading`. A `switch` over one of them without a `default` branch needs them.
+
+  - **Levels on a rubric:** a criterion can carry `levels`, lowest first, from 2 to 10, each a `label` and an optional `description`. A level scores `index / (levels − 1)`.
+    - `validateActivity` does not check them in 1.x, so a field of that name you already use stays valid.
+    - The critic warns `wr_levels_unreadable` when it cannot read them.
+    - `redact()` passes readable levels to the learner, `label` and `description` only.
+  - **The request and its check:**
+    - `aiGradingRequest` builds what a model is told: the response verbatim, the rubric with its levels, and a group's material.
+    - `checkAiGrading` accepts a grade whole or not at all. It needs every criterion once (`misses-criteria`), each on its own scale (`out-of-scale`), with corrections that quote the response (`misquotes-answer`).
+  - **The grade:** `gradeFromJudgement` computes the total as `gradeFromRubric` does, with the item's weights and pass line. It records the model and the prompt revision on `grader`.
+  - **Calibration:** `calibrationReport` measures one grader configuration (a model, a prompt revision, a rubric and a target) against people's grades of the same responses. It reports:
+
+    - exact and adjacent agreement, quadratic kappa and a confusion matrix, on levels;
+    - Pearson's r, mean absolute error and the standardised mean difference, everywhere;
+    - calibration error and the Brier score, where grades carry level probabilities;
+    - the degradation from two people's agreement.
+
+    It ships no thresholds.
+
+  - **Routing:** `routeGrade` decides whether a grade counts or goes to a person, and gives every reason, in a fixed order:
+
+    - no matching calibration, in an exam;
+    - a report below your thresholds;
+    - a model unsure of a level, or one that asked for review;
+    - low confidence;
+    - the audit sample, chosen by `inAuditSample`.
+
+    `outcomeFromRoutedGrade` holds a grade for a person as `deferred` / `awaiting_human_review`, with the grade kept on `pendingGrade`, so `composeAssessmentScore` keeps the attempt provisional.
+
+  - **New optional fields:**
+    - `levels` on a rubric criterion, its redacted form included;
+    - `levelProbabilities` on `CriterionScore`;
+    - `routing` on `GradeRecord`;
+    - `pendingGrade` on a deferred `ItemOutcome`.
+  - **`@intellectif/lk-core/ai-check`** has five grading cases, for a `grading` port.
+  - **The grade-stability corpus** gains 123 vectors for the new functions. The 635 before them are unchanged.
+
+  Guide: https://github.com/intellectif/learning-kit/blob/main/docs/grading.md
+
 ## 1.4.0
 
 ### Minor Changes

package/README.md

@@ -25,6 +25,7 @@ npm install @intellectif/lk-core
 - **Dictation marking as data** — a dictation is graded by the character-level similarity of the typed text to the transcript, with one `ScoringDetail` per word carrying its own `score`. `alignDictation(data, text)` returns the same comparison the grade was made of (normalised strings, word pairings, which accepted transcript won), `diffDictationChars(reference, attempt)` the character-level edits for a marked display, and `dictationReferenceWords(data)` the ids and words a stored result can be checked against — so a review screen or an analytics job never re-implements the arithmetic. `score()` and `evaluate()` take an optional `{ rounding }` to compare the pass line as displayed.
 - **`evaluate(data, response)` → `ItemOutcome`** — the resilient result: `scored`, `deferred` (graded later by an AI or a human), `graded` (the grade came back), or `unscorable`. "Not graded yet" is expressible in the type system and is never conflated with a zero.
 - **Deferred grading, both directions** — `GradeRecord`, `gradeFromRubric()` (the weighted total as a pure function of the grader's judgements — never the model's arithmetic; declare `maxScore` per criterion if your grader works out of 100 or out of 9), `outcomeFromGrade()` (which refuses a record whose numbers cannot be a grade instead of mirroring them), `hasGrade()`.
+- **Assisted grading, behind a calibration gate** — `aiGradingRequest` / `checkAiGrading` hold a model's grade of a written response to every criterion of the rubric, each on its own scale (levels the author writes, or a score); `gradeFromJudgement` computes the total. `calibrationReport` measures a model, prompt, rubric and target against your teachers' grades, and `routeGrade` sends a grade to a person, for reasons it names: in an exam, until a matching calibration is supplied. No thresholds ship: they are yours.
 - **Assessment composition** — `composeAssessmentScore()` for weighted, sectioned totals, with a `provisional` status so an unmarked essay never deflates a grade — and a grade out of contract (`NaN`, 85 "out of 1") never finalises one — and `RoundingPolicy` / `roundGrade` / `gte` / `classifyBand` keeping grade rounding and band classification as the two distinct operations they are.
 - **Item groups** — `Stimulus` + `ItemGroup` (one passage or recording serving several questions), `flattenSequence()` for seeded, shuffle-atomic presentation order with stable slot identity. Slot ids are positional by default (`"3"`, `"1.0"`); author a `slotKey` on an entry and its id survives an insert, so a stored response cannot slide onto a different question.
 - **Delivery policies** — `DeliveryPolicy` says what a school lets a learner see around a question: feedback, solutions, hints and AI help. `resolveDeliveryPolicy` spells one out (an unreadable setting restricts, never allows), `validateDeliveryPolicy` checks one before it is stored, and `combineDeliveryPolicies` keeps a setting on only where every policy leaves it on.
@@ -161,6 +162,7 @@ Zod is the single runtime dependency, pinned to one exact version and private: n
 - [Delivery policies](https://github.com/intellectif/learning-kit/blob/main/docs/delivery.md) — `DeliveryPolicy`, and recording it with an attempt.
 - [Scoring policies](https://github.com/intellectif/learning-kit/blob/main/docs/scoring.md) — `ItemScoringPolicy`: tries, which counts, what hints cost, and scoring them on a server.
 - [AI help for learners](https://github.com/intellectif/learning-kit/blob/main/docs/ai.md) — the AI contract, the checks, and the `ai-check` kit for your prompts.
+- [Grading with a model](https://github.com/intellectif/learning-kit/blob/main/docs/grading.md) — a model's grade for the record: levels, calibration against your teachers, and where a grade goes.
 - [Project README](https://github.com/intellectif/learning-kit#readme) — the full picture, including the React renderers.
 - [Contributing](https://github.com/intellectif/learning-kit/blob/main/CONTRIBUTING.md) — adding a new activity type.
 

package/dist/ai-check.cjs

@@ -1,6 +1,6 @@
 Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
-const require_ai_writing = require("./ai-writing-G6tnCGzw.cjs");
-const require_scoring = require("./scoring-B2_HEhMT.cjs");
+const require_ai_writing = require("./ai-writing-C4YRVrFi.cjs");
+const require_scoring = require("./scoring-CaPS6q8b.cjs");
 //#region src/ai-check/cases.ts
 /**
 * The items the cases are built from. Small, ordinary language-course content
@@ -17,7 +17,10 @@ const require_scoring = require("./scoring-B2_HEhMT.cjs");
 *   twice — so a correction of words nobody wrote is caught;
 * - questions in item groups: a reading passage that prints a question's
 *   answer, which a hint must point to without quoting, and a run of blanks
-*   whose instructions only the group states.
+*   whose instructions only the group states;
+* - essays to grade on a rubric of three criteria, two judged on levels and
+*   one on a score, so a grade that drops a criterion or leaves a scale is
+*   caught.
 */
 const capital = {
 	schemaVersion: "1.0",
@@ -205,6 +208,117 @@ const weekend = {
 		}
 	] }
 };
+/**
+* The weekend task set as an exam question: graded on three criteria, two on
+* levels and one on a score. Three, so a prompt that names only the ones it
+* expects drops one; two scales, so a grade must keep to each.
+*/
+const weekendExam = {
+	schemaVersion: "1.0",
+	type: "written-response",
+	id: "ai-check-writing-exam",
+	title: "Your weekend",
+	prompt: "Write a few sentences about what you did last weekend.",
+	minWords: 15,
+	maxWords: 120,
+	languageTarget: "en-A2",
+	rubric: { criteria: [
+		{
+			name: "Task",
+			description: "Says what the learner did last weekend, in order",
+			weight: 1,
+			levels: [
+				{
+					label: "Not met",
+					description: "Not about last weekend"
+				},
+				{
+					label: "Partly met",
+					description: "About the weekend, but thin or out of order"
+				},
+				{
+					label: "Met",
+					description: "Says what they did, in order"
+				}
+			]
+		},
+		{
+			name: "Grammar",
+			description: "Past tenses, articles, agreement",
+			weight: 2,
+			levels: [
+				{
+					label: "Limited",
+					description: "Errors in most sentences, some blocking meaning"
+				},
+				{
+					label: "Basic",
+					description: "Frequent errors in past tenses and articles"
+				},
+				{
+					label: "Good",
+					description: "Some errors, none that blocks meaning"
+				},
+				{
+					label: "Very good",
+					description: "Past tenses, articles and agreement almost always right"
+				}
+			]
+		},
+		{
+			name: "Vocabulary",
+			description: "Everyday words, used correctly",
+			weight: 1
+		}
+	] }
+};
+/** The tides summary set as an exam question, graded against its group's passage. */
+const tidesSummaryExam = {
+	schemaVersion: "1.0",
+	type: "written-response",
+	id: "ai-check-tides-summary-exam",
+	title: "Summary",
+	prompt: "Summarise the text in one or two sentences.",
+	minWords: 10,
+	maxWords: 60,
+	languageTarget: "en-B1",
+	rubric: { criteria: [{
+		name: "Content",
+		description: "What the text says, and nothing it does not",
+		weight: 2,
+		levels: [
+			{
+				label: "Wrong",
+				description: "Misstates what the text says"
+			},
+			{
+				label: "Partly right",
+				description: "Gets one of its points wrong, or leaves it out"
+			},
+			{
+				label: "Right",
+				description: "States its points as the text does"
+			}
+		]
+	}, {
+		name: "Language",
+		description: "Clear sentences, in the learner's own words",
+		weight: 1
+	}] }
+};
+const tidesExam = {
+	schemaVersion: "1.0",
+	type: "item-group",
+	id: "ai-check-tides-exam",
+	title: "Part 2 — Reading",
+	stimulus: {
+		id: "ai-check-tides-exam-text",
+		kind: "text",
+		title: "Tides",
+		body: TIDES
+	},
+	items: [springTides, tidesSummaryExam]
+};
 const weather = {
 	schemaVersion: "1.0",
 	type: "read-aloud",
@@ -492,6 +606,21 @@ function writingCase(id, about, text, previousFeedback = [], learnerLocale, task
 		request
 	};
 }
+function gradingCase(id, about, text, extra = {}) {
+	const built = require_ai_writing.aiGradingRequest({
+		data: extra.data ?? weekendExam,
+		response: drafted(text),
+		...extra.group ? { group: extra.group } : {},
+		...extra.learnerLocale !== void 0 ? { learnerLocale: extra.learnerLocale } : {}
+	});
+	if (!built.ok) throw new Error(`ai-check: no grading request for "${id}": ${built.problem}`);
+	return {
+		id,
+		feature: "grading",
+		about,
+		request: built.request
+	};
+}
 function coachingCase(id, about, marks, learnerLocale) {
 	const assessment = marks.assessment ?? forecast(true);
 	const grade = require_scoring.gradeReadAloud(weather, {
@@ -528,8 +657,9 @@ function coachingCase(id, about, marks, learnerLocale) {
 /**
 * The cases {@link runAiCheck} runs: every type the SDK explains, answered
 * right, wrong and — where a type can be — partly right; every type it hints
-* for, before an answer and after a wrong one; and drafts of an essay, with
-* mistakes, without, and revised.
+* for, before an answer and after a wrong one; drafts of an essay, with
+* mistakes, without, and revised; and essays to grade, on the task, with
+* mistakes to quote, off the task, and against a passage.
 *
 * They are ordinary calls. What makes them a test is that the SDK knows the
 * answer to each, so it can check what your model writes about it.
@@ -652,7 +782,15 @@ function aiCheckCases() {
 			locale: "en",
 			instructions: "Focus on what Maria and her grandmother buy.",
 			settings: { "read-aloud": READ_ALOUD_SETTINGS }
-		})
+		}),
+		gradingCase("grading-on-task", "An essay that does the task, with one slip in its tenses", "Last weekend I went to the market with my sister and we bought some apples. In the afternoon we watched a film at home, and on Sunday I visit my grandmother."),
+		gradingCase("grading-mistakes", "An essay with tense, article and agreement mistakes, each to quote as written", "Last weekend I go to the market and buyed a apple. My friends was there too, so we don't go home early. Then I go to the cinema."),
+		gradingCase("grading-off-task", "An essay about something other than last weekend", "My favourite food is pizza. I like it with cheese and tomato, and I eat it every Friday with my family at a small restaurant near our house."),
+		gradingCase("grading-summary", "A summary graded against its group's passage, getting the spring tides wrong", "The tide comes in twice a day because the moon pulls the ocean. The highest tides come only at full moon.", {
+			data: tidesSummaryExam,
+			group: tidesExam
+		}),
+		gradingCase("grading-in-spanish", "The essay with mistakes, with comments asked for in Spanish", "Last weekend I go to the market and buyed a apple. My friends was there too, so we don't go home early. Then I go to the cinema.", { learnerLocale: "es" })
 	];
 }
 //#endregion
@@ -681,6 +819,7 @@ function aiCheckCases() {
 *   pronunciationCoaching: (request) => callMyModel(COACHING_PROMPT, request),
 *   critique: (request) => callMyModel(CRITIQUE_PROMPT, request),
 *   drafts: (request) => callMyModel(DRAFTS_PROMPT, request),
+*   grading: (request) => callMyModel(GRADING_PROMPT, request),
 * });
 * console.log(formatAiCheckReport(report));
 * if (report.refused > 0 || report.errors > 0) {
@@ -693,18 +832,27 @@ function aiCheckCases() {
 * to take most seriously — that prompt gives answers away. Next come
 * `misquotes-answer` on a writing case — that prompt corrects words the learner
 * never wrote — and `contradicts-marks` on a coaching case: that prompt coaches
-* words the engine heard as right, or sounds it never reported.
+* words the engine heard as right, or sounds it never reported. On a grading
+* case, `misses-criteria` and `out-of-scale` mean a grade nobody can use: one
+* that leaves a criterion out grades a different task, and one off a
+* criterion's scale grades on a scale the author did not write.
+*
+* A grading case the SDK accepts is a reply of the right shape, not a right
+* grade: whether a model grades as your teachers do is what a calibration
+* measures (`calibrationReport`), on your own responses.
 */
-const REFUSALS = [
-	"malformed",
-	"empty",
-	"too-long",
-	"contradicts-grade",
-	"reveals-answer",
-	"misquotes-answer",
-	"contradicts-marks",
-	"contradicts-item"
-];
+const REFUSALS = Object.keys({
+	malformed: true,
+	empty: true,
+	"too-long": true,
+	"contradicts-grade": true,
+	"reveals-answer": true,
+	"misquotes-answer": true,
+	"contradicts-marks": true,
+	"contradicts-item": true,
+	"misses-criteria": true,
+	"out-of-scale": true
+});
 /** The port a case is for. */
 function portFor(ports, one) {
 	switch (one.request.feature) {
@@ -713,6 +861,7 @@ function portFor(ports, one) {
 		case "pronunciation-coaching": return ports.pronunciationCoaching;
 		case "item-critique": return ports.critique;
 		case "draft-generation": return ports.drafts;
+		case "grading": return ports.grading;
 		default: return ports.writingFeedback;
 	}
 }
@@ -735,6 +884,15 @@ async function runCase(ports, one) {
 		};
 	}
 	const ms = now() - started;
+	if (one.request.feature === "grading") {
+		const checked = require_ai_writing.checkAiGrading(raw, one.request);
+		return checked.ok ? {
+			case: one,
+			ok: true,
+			grading: checked.judgement,
+			ms
+		} : refused(one, raw, checked.refusal, ms);
+	}
 	if (one.request.feature === "draft-generation") {
 		let next = 0;
 		const checked = require_ai_writing.checkAiDrafts(raw, one.request, { newId: () => {