@intellectif/lk-core 0.18.0 → 0.20.0

package/CHANGELOG.md

@@ -1,5 +1,67 @@
 # @intellectif/lk-core
 
+## 0.20.0
+
+### Minor Changes
+
+- 73a37e3: Delivery policies: what a school lets a learner see around a question — feedback, solutions, hints
+  and AI help — decided per paper rather than per item, so one question serves a practice lesson and a
+  final exam without being authored twice.
+
+  - **`DeliveryPolicy`** — `{ feedback?, solutions?, hints?, ai?: { hints?, explanations? } }`, with
+    `ai: false` as shorthand for both AI settings off. Every setting is a restriction that defaults to
+    `true`, so an empty policy is exactly the behaviour before policies existed; nothing can be switched
+    on.
+  - **`resolveDeliveryPolicy(policy)`** spells a policy out, every setting a boolean. It never throws:
+    `null` is unset, and any other value that is not exactly `true` restricts — a `"false"` from a form
+    means off, not on.
+  - **`validateDeliveryPolicy(policy)`** checks a policy before it is stored and names each problem at
+    its path, a misspelled setting included.
+  - **`combineDeliveryPolicies(...policies)`** keeps a setting on only where every policy leaves it on.
+  - **`OPEN_DELIVERY_POLICY`**, the policy that restricts nothing.
+  - **`planAttempt(entries, { delivery })`** records the policy the attempt is sat under as
+    `plan.delivery`, spelled out, and includes it in `planHash`. It throws on a policy that would not
+    validate. **A plan made without one is byte-identical to before, `planHash` included.**
+  - **`verifyAttemptPlan`** reports `deliveryChanged: true` when two plans were made under different
+    policies, and `matches` is then `false`. The field is present only when true, so every drift report
+    between plans without a policy keeps exactly its old shape.
+
+  No grade changes: nothing here reaches a scorer. Hint penalties, `ItemScoringPolicy` and retries —
+  which do — come next, with grade vectors.
+
+  See `docs/delivery.md`.
+
+## 0.19.0
+
+### Minor Changes
+
+- d576382: Groundwork under the AI line: what a call cost, what was refused, and a way to test a prompt.
+
+  - **`AiTextResult.usage`** — what the call cost, as your provider reported it to your port. It is the
+    same `GraderUsage` a returned grade already carries (`promptTokens`, `completionTokens`, `costUsd`),
+    so one shape covers help for a learner and marking by a grader. The SDK never estimates it and never
+    adds it up; it carries what you send, and drops a number that is not finite and zero or more rather
+    than let it into your totals.
+  - **`ai-help-refused`** joins the interaction kinds: a host can now see, in production, that its model
+    wrote a hint containing the answer or an explanation of a verdict the SDK did not reach. The
+    development console warning said it only to a developer. The event carries the feature and the
+    reason and **never the text** — the text of a refused hint is the answer.
+  - **`@intellectif/lk-core/ai-check`, a new entry point: a test kit for your prompts.** `runAiCheck`
+    makes the calls a learner's questions would make — on items whose answers the SDK knows — and runs
+    the same checks it runs before a learner sees anything, reporting what would have been shown and
+    what would have been refused, with the reason. `aiCheckCases` hands over the cases (every type it
+    explains, answered right, wrong and partly right; every type it hints for, before an answer and
+    after a wrong one), and takes cases of your own built on your own items. `formatAiCheckReport`
+    prints a run for a CI log.
+
+    It is a separate entry point because it is for your test run, not a learner's browser: nothing a
+    page renders imports it.
+
+  The empty `@intellectif/lk-ai` placeholder has been deleted — it never held code, and the ports it
+  was planned for live here, beside the scorers whose readings the checks are.
+
+  See `docs/ai.md`.
+
 ## 0.18.0
 
 ### Minor Changes

package/README.md

@@ -26,7 +26,8 @@ npm install @intellectif/lk-core
 - **Deferred grading, both directions** — `GradeRecord`, `gradeFromRubric()` (the weighted total as a pure function of the grader's judgements — never the model's arithmetic; declare `maxScore` per criterion if your grader works out of 100 or out of 9), `outcomeFromGrade()` (which refuses a record whose numbers cannot be a grade instead of mirroring them), `hasGrade()`.
 - **Assessment composition** — `composeAssessmentScore()` for weighted, sectioned totals, with a `provisional` status so an unmarked essay never deflates a grade — and a grade out of contract (`NaN`, 85 "out of 1") never finalises one — and `RoundingPolicy` / `roundGrade` / `gte` / `classifyBand` keeping grade rounding and band classification as the two distinct operations they are.
 - **Item groups** — `Stimulus` + `ItemGroup` (one passage or recording serving several questions), `flattenSequence()` for seeded, shuffle-atomic presentation order with stable slot identity. Slot ids are positional by default (`"3"`, `"1.0"`); author a `slotKey` on an entry and its id survives an insert, so a stored response cannot slide onto a different question.
-- **Attempt plans** — `planAttempt(entries, { seed, shuffleEntries, points })` freezes the exact paper a learner sat: slot order, per-slot points, a content hash of each item, and a `planHash` over the lot. `verifyAttemptPlan(plan, current)` reports the drift when that content is later edited (`changedSlotIds`, `changedStimulusSlotIds`, `changedPointsSlotIds`, `missingSlotIds`, …) — the thing you need during a remark or an appeal. `scoredItemsFromPlan(plan, outcomes)` turns collected outcomes into the `ScoredItem[]` composition wants, and defaults a slot with **no** outcome to `deferred` rather than silently shrinking the denominator.
+- **Delivery policies** — `DeliveryPolicy` says what a school lets a learner see around a question: feedback, solutions, hints and AI help. `resolveDeliveryPolicy` spells one out (an unreadable setting restricts, never allows), `validateDeliveryPolicy` checks one before it is stored, and `combineDeliveryPolicies` keeps a setting on only where every policy leaves it on.
+- **Attempt plans** — `planAttempt(entries, { seed, shuffleEntries, points, delivery })` freezes the exact paper a learner sat, and the delivery policy it was sat under: slot order, per-slot points, a content hash of each item, and a `planHash` over the lot. `verifyAttemptPlan(plan, current)` reports the drift when that content is later edited (`changedSlotIds`, `changedStimulusSlotIds`, `changedPointsSlotIds`, `missingSlotIds`, …) — the thing you need during a remark or an appeal. `scoredItemsFromPlan(plan, outcomes)` turns collected outcomes into the `ScoredItem[]` composition wants, and defaults a slot with **no** outcome to `deferred` rather than silently shrinking the denominator.
 - **Attempt state** — `serializeAttemptState(plan, progress)` / `restoreAttemptState(plan, snapshot)` round-trip an interrupted attempt (responses, submitted slots, position) and are **bound to the plan**: restoring answers onto a paper the learner never sat throws instead of succeeding quietly. `diffResponses(before, after)` reports what moved between two snapshots, including an answer the learner *cleared*.
 - **Media playback policy** — `media.playback` on an audio recording: `maxPlays`, `seek`, `rate`, and advisory `nativeControlHints`. `resolvePlaybackPolicy()` derives the defaults once so the schema, the plan and the renderer cannot drift. Budgets are frozen into the plan by `planAttempt`, so editing `maxPlays` mid-window cannot change what a past learner was held to, and `planAttempt` refuses a paper where one recording is budgeted under two keys.
 - **Play ledger** — `serializeMediaPlayLedger` / `restoreMediaPlayLedger` / `planMediaBudgets`, with `slotMediaKey()` / `stimulusMediaKey()` for the keys. Bound to its paper by `planHash`, and deliberately **separate** from `AttemptState`: a play is charged immediately while answers autosave on a debounce, and a build that predates the ledger would rewrite an `AttemptState` snapshot without the counts and erase them.
@@ -115,6 +116,7 @@ result.passed;   // boolean, or null while provisional. Never record a provision
 | `@intellectif/lk-core/schemas` | Zod schemas + `validateActivity` + JSON Schema export |
 | `@intellectif/lk-core/scoring` | `score`, `evaluate`, `composeAssessmentScore`, rounding, text matching |
 | `@intellectif/lk-core/xapi` | `xAPIBuilder`, `XAPIVerb`, `validateXAPIStatement` |
+| `@intellectif/lk-core/ai-check` | `runAiCheck`, `aiCheckCases`, `formatAiCheckReport` — test your AI prompts in CI |
 
 The barrel re-exports everything, and it is the **only** entry point for attempt
 plans, attempt state, item groups, redaction, the type registry, authoring drafts

package/dist/{activity-HH8dxN1v.d.cts → activity-f0bgZSTP.d.cts}

@@ -63,6 +63,147 @@ declare function levenshteinDistance(a: string, b: string, max: number): number;
  */
 declare function matchText(input: string, accepted: string | readonly string[], policy?: TextMatchPolicy): TextMatchResult;
 
+/**
+ * The return trip for deferred grading.
+ *
+ * `evaluate()` can say a submission is `deferred` — graded later by an AI or a
+ * human — but until now there was no type for the grade that comes BACK, so
+ * every consumer invented one. The shapes here are the intersection of two
+ * independent production graders (a rubric-based essay grader and a CEFR
+ * speaking grader) that converged on the same envelope: a normalised total,
+ * per-criterion scores with comments, grader-authored artefacts, a confidence
+ * signal, an explicit human-review flag, and provenance.
+ *
+ * The SDK models the SHAPE and the arithmetic. It never calls a model, never
+ * holds a key, and never decides what a rubric means — that stays yours.
+ */
+
+/**
+ * Lifecycle of a deferred grade. A grade exists only in the `graded` state;
+ * every other state means "no grade, and here is why" — which is precisely the
+ * distinction that stops an ungraded submission being rendered as a zero.
+ */
+type GradingState = 'queued' | 'running' | 'graded' | 'failed' | 'skipped';
+/**
+ * Who produced a grade.
+ *
+ * - `'auto'` — a non-generative measurement engine, for example a
+ *   pronunciation-assessment service.
+ * - `'ai'` — a generative model.
+ * - `'human'` — a person.
+ */
+type GraderKind = 'ai' | 'human' | 'auto';
+/** Provenance of a grade, so a re-grade two years later is explicable. */
+interface Grader {
+    kind: GraderKind;
+    /** Identifier of the human grader, when `kind` is `'human'`. */
+    id?: string;
+    /** Model identifier, when `kind` is `'ai'` (e.g. a model name and version). */
+    model?: string;
+    /** Hash or id of the prompt/rubric revision used, for auditability. */
+    promptHash?: string;
+}
+/** Optional cost/usage telemetry emitted by an AI grader. */
+interface GraderUsage {
+    promptTokens?: number;
+    completionTokens?: number;
+    costUsd?: number;
+}
+/** A grader's verdict on one rubric criterion. */
+interface CriterionScore {
+    /** Matches a `WrittenResponseRubricCriterion.name` when a rubric is known. */
+    name: string;
+    /**
+     * The grader's score for this criterion, out of {@link CriterionScore.maxScore}
+     * (which defaults to 1, the SDK's scaled convention). Absent when the
+     * criterion is judged on an ordinal `band`.
+     */
+    score?: number;
+    /**
+     * What `score` is out of. Defaults to `1`.
+     *
+     * Real graders rarely work in [0,1]: rubrics come back out of 100, out of 9
+     * (CEFR/IELTS bands), or out of a points total that differs between
+     * criteria. Before this field the only way to use {@link gradeFromRubric}
+     * was to normalise by hand first — and a grader that skipped that step got
+     * its scores REJECTED, which pushed integrators back to letting the model
+     * compute the weighted total itself. That is precisely the arithmetic this
+     * SDK exists to take away from a language model.
+     *
+     * Set it per criterion, so a rubric mixing a /100 criterion with a /9 band
+     * score still composes. `gradeFromRubric` normalises `score / maxScore`
+     * before weighting; the resulting ratio must land in [0,1].
+     */
+    maxScore?: number;
+    /** Ordinal verdict when the rubric is banded rather than numeric (e.g. `"B1"`). */
+    band?: string;
+    /** The grader's comment for this criterion, addressed to the learner. */
+    comment?: string;
+    /**
+     * Weight actually applied, echoed from the rubric so the total is checkable.
+     * Defaults to `1`. Zero or more: `gradeFromRubric` refuses a negative weight
+     * rather than let it carry the total outside [0,1].
+     */
+    weight?: number;
+    /** Not applicable to this submission (e.g. interaction on a monologue task). */
+    notApplicable?: boolean;
+}
+/** A suggested correction anchored in the learner's own text. */
+interface InlineCorrection {
+    /** Verbatim span from the submission. */
+    original: string;
+    /** Suggested replacement. */
+    corrected: string;
+    explanation?: string;
+    /** Character offsets into the submitted text, so a UI can anchor it inline. */
+    range?: {
+        start: number;
+        end: number;
+    };
+    /** Error-type tag (`"article"`, `"tense"`, `"register"`). */
+    category?: string;
+}
+/**
+ * A grade that came back from an asynchronous grader. `score` is scaled
+ * [0,1] against `maxScore`, matching every other score in the SDK. A grader
+ * that works in points does not have to normalise by hand: declare
+ * `maxScore` on each {@link CriterionScore} and let `gradeFromRubric` do the
+ * arithmetic, which is the whole point of handing it the judgements.
+ *
+ * `maxScore` is a positive, finite number and `score` a finite number from 0
+ * to it. `outcomeFromGrade` refuses a record that is not, rather than mirror
+ * numbers nobody can defend onto an outcome.
+ */
+interface GradeRecord {
+    score: number;
+    maxScore: number;
+    passed: boolean;
+    /** Narrative feedback addressed to the learner. */
+    feedback: string | null;
+    /** Per-criterion breakdown, when the grader worked against a rubric. */
+    criteria?: CriterionScore[];
+    /**
+     * Per-item marks carried by a grade — for example the per-word marks of a
+     * read-aloud grade. `outcomeFromGrade` does not mirror them onto the
+     * outcome; read `outcome.grade.details`.
+     */
+    details?: ScoringDetail[];
+    /** Corrections anchored in the learner's text. */
+    corrections?: InlineCorrection[];
+    /** Supporting observations the grader cited. */
+    evidence?: string[];
+    /** How the grader reached this verdict. */
+    rationale?: string;
+    /** The grader's own confidence — a signal for routing to human review. */
+    confidence?: 'high' | 'medium' | 'low';
+    /** The grader is unsure or the submission is atypical; route to a human. */
+    requiresHumanReview?: boolean;
+    grader?: Grader;
+    usage?: GraderUsage;
+    /** ISO 8601 timestamp of when the grade was produced. */
+    gradedAt?: string;
+}
+
 /**
  * The contract between an activity and a model the HOST runs, for help a
  * learner reads: an explanation of a graded answer, and hints before one.
@@ -72,6 +213,7 @@ declare function matchText(input: string, accepted: string | readonly string[],
  * SDK's own scorer — and checks what comes back. The model explains a verdict
  * the SDK reached; it never reaches one.
  */
+
 /**
  * What an author allows AI to do for one item. Absent, or a feature left out,
  * leaves the choice to the deployment. `false` switches a feature off for this
@@ -250,150 +392,20 @@ interface AiTextResult {
     /** For an explanation: the verdict the model explained. */
     verdict?: AiVerdict;
     provenance?: AiProvenance;
-}
-/** Why a result was not shown. */
-type AiRefusal = 'malformed' | 'empty' | 'too-long' | 'contradicts-grade' | 'reveals-answer';
-
-/**
- * The return trip for deferred grading.
- *
- * `evaluate()` can say a submission is `deferred` — graded later by an AI or a
- * human — but until now there was no type for the grade that comes BACK, so
- * every consumer invented one. The shapes here are the intersection of two
- * independent production graders (a rubric-based essay grader and a CEFR
- * speaking grader) that converged on the same envelope: a normalised total,
- * per-criterion scores with comments, grader-authored artefacts, a confidence
- * signal, an explicit human-review flag, and provenance.
- *
- * The SDK models the SHAPE and the arithmetic. It never calls a model, never
- * holds a key, and never decides what a rubric means — that stays yours.
- */
-
-/**
- * Lifecycle of a deferred grade. A grade exists only in the `graded` state;
- * every other state means "no grade, and here is why" — which is precisely the
- * distinction that stops an ungraded submission being rendered as a zero.
- */
-type GradingState = 'queued' | 'running' | 'graded' | 'failed' | 'skipped';
-/**
- * Who produced a grade.
- *
- * - `'auto'` — a non-generative measurement engine, for example a
- *   pronunciation-assessment service.
- * - `'ai'` — a generative model.
- * - `'human'` — a person.
- */
-type GraderKind = 'ai' | 'human' | 'auto';
-/** Provenance of a grade, so a re-grade two years later is explicable. */
-interface Grader {
-    kind: GraderKind;
-    /** Identifier of the human grader, when `kind` is `'human'`. */
-    id?: string;
-    /** Model identifier, when `kind` is `'ai'` (e.g. a model name and version). */
-    model?: string;
-    /** Hash or id of the prompt/rubric revision used, for auditability. */
-    promptHash?: string;
-}
-/** Optional cost/usage telemetry emitted by an AI grader. */
-interface GraderUsage {
-    promptTokens?: number;
-    completionTokens?: number;
-    costUsd?: number;
-}
-/** A grader's verdict on one rubric criterion. */
-interface CriterionScore {
-    /** Matches a `WrittenResponseRubricCriterion.name` when a rubric is known. */
-    name: string;
-    /**
-     * The grader's score for this criterion, out of {@link CriterionScore.maxScore}
-     * (which defaults to 1, the SDK's scaled convention). Absent when the
-     * criterion is judged on an ordinal `band`.
-     */
-    score?: number;
     /**
-     * What `score` is out of. Defaults to `1`.
-     *
-     * Real graders rarely work in [0,1]: rubrics come back out of 100, out of 9
-     * (CEFR/IELTS bands), or out of a points total that differs between
-     * criteria. Before this field the only way to use {@link gradeFromRubric}
-     * was to normalise by hand first — and a grader that skipped that step got
-     * its scores REJECTED, which pushed integrators back to letting the model
-     * compute the weighted total itself. That is precisely the arithmetic this
-     * SDK exists to take away from a language model.
+     * What the call cost, as the provider reported it to your port: the same
+     * {@link GraderUsage} a returned grade carries, so one shape covers help for
+     * a learner and marking by a grader.
      *
-     * Set it per criterion, so a rubric mixing a /100 criterion with a /9 band
-     * score still composes. `gradeFromRubric` normalises `score / maxScore`
-     * before weighting; the resulting ratio must land in [0,1].
+     * The SDK never estimates it and never adds it up. It carries what a port
+     * sends through to `onInteraction`, beside the provenance, so the record of
+     * what a learner was shown and the record of what it cost are the same
+     * record. Numbers that are not finite and zero or more are dropped.
      */
-    maxScore?: number;
-    /** Ordinal verdict when the rubric is banded rather than numeric (e.g. `"B1"`). */
-    band?: string;
-    /** The grader's comment for this criterion, addressed to the learner. */
-    comment?: string;
-    /**
-     * Weight actually applied, echoed from the rubric so the total is checkable.
-     * Defaults to `1`. Zero or more: `gradeFromRubric` refuses a negative weight
-     * rather than let it carry the total outside [0,1].
-     */
-    weight?: number;
-    /** Not applicable to this submission (e.g. interaction on a monologue task). */
-    notApplicable?: boolean;
-}
-/** A suggested correction anchored in the learner's own text. */
-interface InlineCorrection {
-    /** Verbatim span from the submission. */
-    original: string;
-    /** Suggested replacement. */
-    corrected: string;
-    explanation?: string;
-    /** Character offsets into the submitted text, so a UI can anchor it inline. */
-    range?: {
-        start: number;
-        end: number;
-    };
-    /** Error-type tag (`"article"`, `"tense"`, `"register"`). */
-    category?: string;
-}
-/**
- * A grade that came back from an asynchronous grader. `score` is scaled
- * [0,1] against `maxScore`, matching every other score in the SDK. A grader
- * that works in points does not have to normalise by hand: declare
- * `maxScore` on each {@link CriterionScore} and let `gradeFromRubric` do the
- * arithmetic, which is the whole point of handing it the judgements.
- *
- * `maxScore` is a positive, finite number and `score` a finite number from 0
- * to it. `outcomeFromGrade` refuses a record that is not, rather than mirror
- * numbers nobody can defend onto an outcome.
- */
-interface GradeRecord {
-    score: number;
-    maxScore: number;
-    passed: boolean;
-    /** Narrative feedback addressed to the learner. */
-    feedback: string | null;
-    /** Per-criterion breakdown, when the grader worked against a rubric. */
-    criteria?: CriterionScore[];
-    /**
-     * Per-item marks carried by a grade — for example the per-word marks of a
-     * read-aloud grade. `outcomeFromGrade` does not mirror them onto the
-     * outcome; read `outcome.grade.details`.
-     */
-    details?: ScoringDetail[];
-    /** Corrections anchored in the learner's text. */
-    corrections?: InlineCorrection[];
-    /** Supporting observations the grader cited. */
-    evidence?: string[];
-    /** How the grader reached this verdict. */
-    rationale?: string;
-    /** The grader's own confidence — a signal for routing to human review. */
-    confidence?: 'high' | 'medium' | 'low';
-    /** The grader is unsure or the submission is atypical; route to a human. */
-    requiresHumanReview?: boolean;
-    grader?: Grader;
     usage?: GraderUsage;
-    /** ISO 8601 timestamp of when the grade was produced. */
-    gradedAt?: string;
 }
+/** Why a result was not shown. */
+type AiRefusal = 'malformed' | 'empty' | 'too-long' | 'contradicts-grade' | 'reveals-answer';
 
 /**
  * An xAPI Activity object — the thing a statement is about.
@@ -1549,8 +1561,18 @@ type InteractionKind = 'option-selected' | 'option-deselected' | 'blank-filled'
  * is never shown and never reported here.
  */
  | 'ai-hint-shown'
-/** An AI explanation of the graded answer was shown (`provenance` when sent). */
- | 'ai-explanation-shown' | (string & {});
+/** An AI explanation of the graded answer was shown (`provenance`, `usage` when sent). */
+ | 'ai-explanation-shown'
+/**
+ * An AI answer was refused, and the learner was told help was not available
+ * (`feature`, `reason` — an {@link AiRefusal} — and `hintNumber` for a hint).
+ *
+ * **The text itself is never carried**: a hint refused for revealing the
+ * answer contains the answer, and this event is logged. It is how a host
+ * watches its own model in production, where the development console warning
+ * it accompanies is silent.
+ */
+ | 'ai-help-refused' | (string & {});
 /** Fired by activity components on every discrete learner interaction. */
 interface InteractionEvent {
     /** The kind of interaction that occurred. */