@intellectif/lk-core 1.4.0 → 1.5.0

package/CHANGELOG.md

@@ -1,5 +1,50 @@
 # @intellectif/lk-core
 
+## 1.5.0
+
+### Minor Changes
+
+- [#112](https://github.com/intellectif/learning-kit/pull/112) [`3fc1049`](https://github.com/intellectif/learning-kit/commit/3fc1049d67e14f6f4a12c235f19952a988472978) Thanks [@diego28e](https://github.com/diego28e)! - **Assisted grading**: a model grades a written response for the record, and in an exam no model's grade counts until it has been measured against your teachers.
+
+  **Action required:** none. Nothing grades until you call it. Four union members are new: `DeferredReason` gains `awaiting_human_review`, `AiRefusal` gains `misses-criteria` and `out-of-scale`, and `AiCheckCase['feature']` gains `grading`. A `switch` over one of them without a `default` branch needs them.
+
+  - **Levels on a rubric:** a criterion can carry `levels`, lowest first, from 2 to 10, each a `label` and an optional `description`. A level scores `index / (levels − 1)`.
+    - `validateActivity` does not check them in 1.x, so a field of that name you already use stays valid.
+    - The critic warns `wr_levels_unreadable` when it cannot read them.
+    - `redact()` passes readable levels to the learner, `label` and `description` only.
+  - **The request and its check:**
+    - `aiGradingRequest` builds what a model is told: the response verbatim, the rubric with its levels, and a group's material.
+    - `checkAiGrading` accepts a grade whole or not at all. It needs every criterion once (`misses-criteria`), each on its own scale (`out-of-scale`), with corrections that quote the response (`misquotes-answer`).
+  - **The grade:** `gradeFromJudgement` computes the total as `gradeFromRubric` does, with the item's weights and pass line. It records the model and the prompt revision on `grader`.
+  - **Calibration:** `calibrationReport` measures one grader configuration (a model, a prompt revision, a rubric and a target) against people's grades of the same responses. It reports:
+
+    - exact and adjacent agreement, quadratic kappa and a confusion matrix, on levels;
+    - Pearson's r, mean absolute error and the standardised mean difference, everywhere;
+    - calibration error and the Brier score, where grades carry level probabilities;
+    - the degradation from two people's agreement.
+
+    It ships no thresholds.
+
+  - **Routing:** `routeGrade` decides whether a grade counts or goes to a person, and gives every reason, in a fixed order:
+
+    - no matching calibration, in an exam;
+    - a report below your thresholds;
+    - a model unsure of a level, or one that asked for review;
+    - low confidence;
+    - the audit sample, chosen by `inAuditSample`.
+
+    `outcomeFromRoutedGrade` holds a grade for a person as `deferred` / `awaiting_human_review`, with the grade kept on `pendingGrade`, so `composeAssessmentScore` keeps the attempt provisional.
+
+  - **New optional fields:**
+    - `levels` on a rubric criterion, its redacted form included;
+    - `levelProbabilities` on `CriterionScore`;
+    - `routing` on `GradeRecord`;
+    - `pendingGrade` on a deferred `ItemOutcome`.
+  - **`@intellectif/lk-core/ai-check`** has five grading cases, for a `grading` port.
+  - **The grade-stability corpus** gains 123 vectors for the new functions. The 635 before them are unchanged.
+
+  Guide: https://github.com/intellectif/learning-kit/blob/main/docs/grading.md
+
 ## 1.4.0
 
 ### Minor Changes

package/README.md

@@ -25,6 +25,7 @@ npm install @intellectif/lk-core
 - **Dictation marking as data** — a dictation is graded by the character-level similarity of the typed text to the transcript, with one `ScoringDetail` per word carrying its own `score`. `alignDictation(data, text)` returns the same comparison the grade was made of (normalised strings, word pairings, which accepted transcript won), `diffDictationChars(reference, attempt)` the character-level edits for a marked display, and `dictationReferenceWords(data)` the ids and words a stored result can be checked against — so a review screen or an analytics job never re-implements the arithmetic. `score()` and `evaluate()` take an optional `{ rounding }` to compare the pass line as displayed.
 - **`evaluate(data, response)` → `ItemOutcome`** — the resilient result: `scored`, `deferred` (graded later by an AI or a human), `graded` (the grade came back), or `unscorable`. "Not graded yet" is expressible in the type system and is never conflated with a zero.
 - **Deferred grading, both directions** — `GradeRecord`, `gradeFromRubric()` (the weighted total as a pure function of the grader's judgements — never the model's arithmetic; declare `maxScore` per criterion if your grader works out of 100 or out of 9), `outcomeFromGrade()` (which refuses a record whose numbers cannot be a grade instead of mirroring them), `hasGrade()`.
+- **Assisted grading, behind a calibration gate** — `aiGradingRequest` / `checkAiGrading` hold a model's grade of a written response to every criterion of the rubric, each on its own scale (levels the author writes, or a score); `gradeFromJudgement` computes the total. `calibrationReport` measures a model, prompt, rubric and target against your teachers' grades, and `routeGrade` sends a grade to a person, for reasons it names: in an exam, until a matching calibration is supplied. No thresholds ship: they are yours.
 - **Assessment composition** — `composeAssessmentScore()` for weighted, sectioned totals, with a `provisional` status so an unmarked essay never deflates a grade — and a grade out of contract (`NaN`, 85 "out of 1") never finalises one — and `RoundingPolicy` / `roundGrade` / `gte` / `classifyBand` keeping grade rounding and band classification as the two distinct operations they are.
 - **Item groups** — `Stimulus` + `ItemGroup` (one passage or recording serving several questions), `flattenSequence()` for seeded, shuffle-atomic presentation order with stable slot identity. Slot ids are positional by default (`"3"`, `"1.0"`); author a `slotKey` on an entry and its id survives an insert, so a stored response cannot slide onto a different question.
 - **Delivery policies** — `DeliveryPolicy` says what a school lets a learner see around a question: feedback, solutions, hints and AI help. `resolveDeliveryPolicy` spells one out (an unreadable setting restricts, never allows), `validateDeliveryPolicy` checks one before it is stored, and `combineDeliveryPolicies` keeps a setting on only where every policy leaves it on.
@@ -161,6 +162,7 @@ Zod is the single runtime dependency, pinned to one exact version and private: n
 - [Delivery policies](https://github.com/intellectif/learning-kit/blob/main/docs/delivery.md) — `DeliveryPolicy`, and recording it with an attempt.
 - [Scoring policies](https://github.com/intellectif/learning-kit/blob/main/docs/scoring.md) — `ItemScoringPolicy`: tries, which counts, what hints cost, and scoring them on a server.
 - [AI help for learners](https://github.com/intellectif/learning-kit/blob/main/docs/ai.md) — the AI contract, the checks, and the `ai-check` kit for your prompts.
+- [Grading with a model](https://github.com/intellectif/learning-kit/blob/main/docs/grading.md) — a model's grade for the record: levels, calibration against your teachers, and where a grade goes.
 - [Project README](https://github.com/intellectif/learning-kit#readme) — the full picture, including the React renderers.
 - [Contributing](https://github.com/intellectif/learning-kit/blob/main/CONTRIBUTING.md) — adding a new activity type.
 

package/dist/{ai-CnmLPMzd.d.cts → ai-BYryFkF9.d.cts}

@@ -134,6 +134,57 @@ interface CriterionScore {
   weight?: number;
   /** Not applicable to this submission (e.g. interaction on a monologue task). */
   notApplicable?: boolean;
+  /**
+   * How likely each of the criterion's levels was, lowest first, when the
+   * grader said: a distribution, whose likeliest level is `band`. Kept on the
+   * record as the evidence a routing rule read (`routeGrade`) and a
+   * calibration measures (`calibrationReport`).
+   */
+  levelProbabilities?: number[];
+}
+/**
+ * What one calibration certifies: a model, a prompt revision, a rubric and a
+ * target language. A change to any of them is a different grader, measured
+ * afresh. Built by `graderConfiguration`.
+ */
+interface GraderConfiguration {
+  /** The model's identifier, from the grade's `grader.model`. */
+  model: string;
+  /** The prompt revision, from the grade's `grader.promptHash`. */
+  promptHash: string;
+  /** `contentHash` of the item's rubric, levels and descriptors included. */
+  rubricHash: string;
+  /** The item's `languageTarget`, or `null` when it has none. */
+  languageTarget: string | null;
+}
+/**
+ * Why `routeGrade` sent a grade to a person, in this order:
+ *
+ * - `no-calibration` — in `exam`, no calibration report for this grader's exact
+ *   configuration was supplied: a model's grade counts on a paper of record
+ *   only once it has been measured;
+ * - `calibration-below-policy` — the report misses one of the policy's
+ *   thresholds, for a criterion or for the total;
+ * - `unsure` — a criterion's likeliest level is less likely than the policy's
+ *   `minLevelProbability`;
+ * - `model-asked` — the grader set `requiresHumanReview`;
+ * - `low-confidence` — the grader's own `confidence` is `low`, and the policy
+ *   routes on it;
+ * - `audit-sample` — the response is in the audit sample: a person marks it
+ *   too, without the model's grade, to keep the calibration current.
+ */
+type GradeRoutingReason = 'no-calibration' | 'calibration-below-policy' | 'unsure' | 'model-asked' | 'low-confidence' | 'audit-sample';
+/** Where a grade went, and why: the record an appeal reads. */
+interface GradeRouting {
+  /** `model` when its grade counts; `person` when a person must mark it first. */
+  to: 'model' | 'person';
+  /** Every reason it went to a person, in {@link GradeRoutingReason}'s order; empty when it did not. */
+  reasons: GradeRoutingReason[];
+  /** The calibration report the decision read, by its hash and size, when one matched. */
+  calibration?: {
+    reportHash: string;
+    pairs: number;
+  };
 }
 /** A suggested correction anchored in the learner's own text. */
 interface InlineCorrection {
@@ -189,6 +240,8 @@ interface GradeRecord {
   usage?: GraderUsage;
   /** ISO 8601 timestamp of when the grade was produced. */
   gradedAt?: string;
+  /** Where `routeGrade` sent this grade, and why, when it was routed. */
+  routing?: GradeRouting;
 }
 //#endregion
 //#region src/types/xapi.d.ts
@@ -770,6 +823,17 @@ interface FillInTheBlanksData {
   /** What an author allows AI to do for this item: see {@link ActivityAiPermissions}. */
   ai?: ActivityAiPermissions;
 }
+/**
+ * One level of a rubric criterion: what a response at this level does. Levels
+ * run from the lowest to the highest, and a grader judging against them names
+ * one.
+ */
+interface WrittenResponseRubricLevel {
+  /** A short name, unique within its criterion: "Developing", "B1". */
+  label: string;
+  /** What a response at this level does, for the grader and the learner. */
+  description?: string;
+}
 /** A single criterion within a written-response grading rubric. */
 interface WrittenResponseRubricCriterion {
   /** Criterion name (e.g. "Task achievement", "Grammar range"). */
@@ -778,6 +842,20 @@ interface WrittenResponseRubricCriterion {
   description?: string;
   /** Non-negative weight of this criterion in the overall grade. */
   weight: number;
+  /**
+   * The criterion's levels, lowest first: 2 to 10, each with a label unique
+   * within the criterion once case and spacing are ignored. A grader judging
+   * against levels names one, and the criterion scores the level's place on
+   * the scale, `index / (levels.length - 1)`: 0 for the lowest, 1 for the
+   * highest.
+   *
+   * Read by the grading functions (`aiGradingRequest`, `gradeFromJudgement`,
+   * `calibrationReport`), which report levels they cannot use. In 1.x
+   * `validateActivity` does not check it: your own fields pass through it, and
+   * a field of yours named `levels` stays valid. The item critic warns about
+   * levels the grading functions cannot read.
+   */
+  levels?: WrittenResponseRubricLevel[];
 }
 /** Grading rubric attached to a written-response activity. */
 interface WrittenResponseRubric {
@@ -1232,8 +1310,12 @@ interface DeferredScoringPartial {
  *   a finite number, a `maxScore` of 0 or less, or a score outside 0 to
  *   `maxScore`). The record is kept on `rejectedGrade`. A real grade is still
  *   owed, so re-grade the slot: waiting will not bring one.
+ * - `awaiting_human_review` — a model graded the response, and a routing rule
+ *   (`routeGrade`) sent it to a person: the grade does not count until a
+ *   person marks it. The model's grade is kept on `pendingGrade`, for the
+ *   person and for the next calibration.
  */
-type DeferredReason = 'requires_async_grading' | 'no_response_recorded' | 'grade_rejected';
+type DeferredReason = 'requires_async_grading' | 'no_response_recorded' | 'grade_rejected' | 'awaiting_human_review';
 /**
  * The outcome of evaluating a learner response against an activity — the
  * union `evaluate()` returns. Unlike {@link ScoringResult}, it can express
@@ -1266,6 +1348,13 @@ type ItemOutcome = {
    * is mirrored onto the outcome, and nothing reads it as a grade.
    */
   rejectedGrade?: GradeRecord;
+  /**
+   * The model's grade a routing rule held for a person, when `reason` is
+   * `awaiting_human_review`, with the decision on its `routing`. Kept for
+   * the person and for the next calibration; nothing on it is mirrored
+   * onto the outcome, and nothing reads it as a grade.
+   */
+  pendingGrade?: GradeRecord;
 } | {
   status: 'graded';
   /**
@@ -1951,8 +2040,15 @@ interface AiTextResult {
  * a word the engine did not mark, or on a sound it did not report.
  * `contradicts-item` is a critique of an item that points at a field the item
  * does not have, or quotes words that field does not contain.
- */
-type AiRefusal = 'malformed' | 'empty' | 'too-long' | 'contradicts-grade' | 'reveals-answer' | 'misquotes-answer' | 'contradicts-marks' | 'contradicts-item';
+ * `misses-criteria` is a grade that leaves a rubric criterion out, renames
+ * one, judges one twice, or calls one not applicable: judging fewer criteria
+ * is judging a different task, and the total would be renormalised over the
+ * rest. `out-of-scale` is a judgement off its criterion's scale: a level the
+ * criterion does not have, a score outside 0 to 1, or level probabilities that
+ * are not a distribution over its levels, or whose likeliest level is not the
+ * one chosen.
+ */
+type AiRefusal = 'malformed' | 'empty' | 'too-long' | 'contradicts-grade' | 'reveals-answer' | 'misquotes-answer' | 'contradicts-marks' | 'contradicts-item' | 'misses-criteria' | 'out-of-scale';
 /** One criterion of a written response's rubric, as the author wrote it. */
 interface AiRubricCriterionFact {
   name: string;
@@ -2063,6 +2159,104 @@ interface AiWritingFeedback {
   provenance?: AiProvenance;
   usage?: GraderUsage;
 }
+/** One criterion of the rubric a model grades against, as the author wrote it. */
+interface AiGradingCriterionFact {
+  name: string;
+  description?: string;
+  weight: number;
+  /**
+   * The levels to choose from, lowest first, as the author wrote them; `null`
+   * when the criterion is judged on a score from 0 to 1.
+   */
+  levels: {
+    label: string;
+    description?: string;
+  }[] | null;
+}
+/**
+ * What a model is told about a submitted written response it grades: the task
+ * as the learner saw it, the response verbatim, and the rubric with its levels.
+ */
+interface AiGradingFacts {
+  activityType: 'written-response';
+  activityId: string;
+  title: string;
+  /** The language the item is written in, when the author gave one. */
+  locale?: string;
+  prompt: string;
+  /** The response, exactly as submitted. Every correction quotes it. */
+  text: string;
+  /** Recomputed with `countWords`, as the grader counts it. */
+  wordCount: number;
+  minWords: number;
+  maxWords: number;
+  withinWordBounds: boolean;
+  /** Every criterion of the item's rubric, in its order. */
+  rubric: AiGradingCriterionFact[];
+  /** The target language and level, when the author gave one (e.g. `"en-B1"`). */
+  languageTarget?: string;
+}
+/**
+ * Asks a model to grade a submitted written response. Built on the server,
+ * from the whole item: grading needs the rubric, never the learner's view.
+ */
+interface AiGradingRequest {
+  feature: 'grading';
+  facts: AiGradingFacts;
+  /** The material of the group the task sits in; absent outside a group. */
+  material?: AiMaterialFacts;
+  /** The language to write comments and feedback in, when known. */
+  learnerLocale?: string;
+}
+/**
+ * Why `aiGradingRequest` built no request:
+ *
+ * - `not-written-response` — the item is another type;
+ * - `redacted` — a `redact()` projection: grading is done on the server, from
+ *   the whole item;
+ * - `no-rubric` — no rubric, or a criterion without a name and a weight: a
+ *   model has nothing to be held to;
+ * - `duplicate-criteria` — two criteria with one name once case and spacing are
+ *   ignored: a model cannot judge them apart (`path` names the later one);
+ * - `unreadable-levels` — a criterion's `levels` are there but cannot be used
+ *   (`path` names them; the item critic's `wr_levels_unreadable` says why);
+ * - `empty-response` — nothing was written, so there is nothing to judge.
+ */
+type AiGradingRequestProblem = 'not-written-response' | 'redacted' | 'no-rubric' | 'duplicate-criteria' | 'unreadable-levels' | 'empty-response';
+/** A model's judgement of one criterion, as `checkAiGrading` accepted it. */
+interface AiGradedCriterion {
+  /** The criterion's name, as the author wrote it. */
+  name: string;
+  /** The level chosen, as the author labelled it, on a criterion with levels. */
+  level?: string;
+  /**
+   * From 0 to 1: the level's place on the scale, `index / (levels - 1)`, or the
+   * model's score on a criterion without levels.
+   */
+  score: number;
+  /** How likely each level is, lowest first, when the model said: a distribution. */
+  probabilities?: number[];
+  comment?: string;
+}
+/**
+ * A model's grade of a written response as `checkAiGrading` accepted it: every
+ * criterion of the rubric judged, on its own scale. The total is not here —
+ * `gradeFromJudgement` computes it.
+ */
+interface AiGradingJudgement {
+  /** Every criterion of the rubric, once, in the rubric's order. */
+  criteria: AiGradedCriterion[];
+  /** Narrative feedback addressed to the learner. */
+  feedback: string | null;
+  /** Corrections, each anchored in the response, as on feedback on writing. */
+  corrections: InlineCorrection[];
+  /** How the model reached its verdict; for the record, never shown. */
+  rationale?: string;
+  /** The model asked for a person to mark this response. */
+  requiresHumanReview: boolean;
+  provenance?: AiProvenance;
+  usage?: GraderUsage;
+}
 /** One sound of a word, as the engine reported it. */
 interface AiSoundFact {
   /** The sound expected, in the assessment's phoneme alphabet. */
@@ -2413,5 +2607,5 @@ interface AiVideoDraft {
   findings: ItemFinding[];
 }
 //#endregion
-export { AiWritingFeedbackRequest as $, ReadAloudDimension as $t, AiGapSelectFacts as A, InlineCorrection as An, DictationLearnerResponse as At, AiReadingFacts as B, InteractionEvent as Bt, AiDraftsCall as C, XAPIVerbObject as Cn, ActivityResult as Ct, AiFeature as D, GraderKind as Dn, DeferredScoringPartial as Dt, AiExplanationRequest as E, Grader as En, DeferredReason as Et, AiItemFacts as F, GapSelectBank as Ft, AiSupportedActivityType as G, MediaPlaybackPolicy as Gt, AiRefusal as H, ItemOutcome as Ht, AiMaterialFacts as I, GapSelectChoice as It, AiVideoDraft as J, MultipleChoiceLearnerResponse as Jt, AiTextResult as K, MediaTrack as Kt, AiMultipleChoiceFacts as L, GapSelectData as Lt, AiGrade as M, TextMatchResult as Mn, DictationTolerance as Mt, AiGroupInput as N, levenshteinDistance as Nn, FillInTheBlanksData as Nt, AiFillInTheBlanksFacts as O, GraderUsage as On, DictationData as Ot, AiHintRequest as P, matchText as Pn, FillInTheBlanksLearnerResponse as Pt, AiWritingFeedback as Q, ReadAloudData as Qt, AiOptionFact as R, GapSelectGap as Rt, AiDrafts as S, XAPIStatement as Sn, ActivityMedia as St, AiDraftsRun as T, GradeRecord as Tn, BlankConfig as Tt, AiRubricCriterionFact as U, LearnerResponse as Ut, AiReadingWordFact as V, InteractionKind as Vt, AiSoundFact as W, LearnerResponseMap as Wt, AiWritingCorrection as X, MultipleChoiceOptionMedia as Xt, AiWordFact as Y, MultipleChoiceOption as Yt, AiWritingFacts as Z, NativeControlHint as Zt, AiDraftCaption as _, XAPIContextActivities as _n, ItemFinding as _t, AiCoachingRequest as a, ScoringDetail as an, SequenceSlot as at, AiDraftSource as b, XAPIResult as bn, ActivityDataMap as bt, AiCriterionJudgement as c, ValidationError as cn, StimulusKind as ct, AiCritiqueFinding as d, WrittenResponseLearnerResponse as dn, DraftComplete as dt, ReadAloudDimensionWeight as en, AiWritingFeedbackResult as et, AiCritiqueFindingResult as f, WrittenResponseRubric as fn, DraftContext as ft, AiDictationFacts as g, XAPIContext as gn, DraftValidationResult as gt, AiCritiqueResult as h, XAPIConfig as hn, DraftSeverity as ht, AiCoaching as i, RecordingRef as in, SequenceEntry as it, AiGeneratedDraft as j, TextMatchPolicy as jn, DictationSlowMedia as jt, AiGapFact as k, GradingState as kn, DictationEquivalence as kt, AiCritique as l, ValidationResult as ln, TimelineChapter as lt, AiCritiqueRequest as m, XAPIActor as mn, DraftNotComplete as mt, AiActivityInput as n, ReadAloudSlowMedia as nn, MediaProgress as nt, AiCoachingResult as o, ScoringOutcome as on, SequenceSlotGroup as ot, AiCritiqueKind as p, WrittenResponseRubricCriterion as pn, DraftIssue as pt, AiVerdict as q, MultipleChoiceData as qt, AiBlankFact as r, RecordingBounds as rn, MediaTimeline as rt, AiCoachingWord as s, ScoringResult as sn, Stimulus as st, ActivityAiPermissions as t, ReadAloudLearnerResponse as tn, ItemGroup as tt, AiCritiqueField as u, WrittenResponseData as un, TimelineCue as ut, AiDraftProblem as v, XAPIError as vn, ItemFindingSeverity as vt, AiDraftsRequest as w, CriterionScore as wn, ActivityType as wt, AiDraftType as x, XAPIScore as xn, ActivityFeedback as xt, AiDraftSettings as y, XAPIObject as yn, ActivityData as yt, AiProvenance as z, GapSelectLearnerResponse as zt };
-//# sourceMappingURL=ai-CnmLPMzd.d.cts.map
+export { AiVerdict as $, MultipleChoiceData as $t, AiGapSelectFacts as A, XAPIVerbObject as An, ActivityType as At, AiHintRequest as B, InlineCorrection as Bn, FillInTheBlanksLearnerResponse as Bt, AiDraftsCall as C, XAPIContext as Cn, ItemFinding as Ct, AiFeature as D, XAPIResult as Dn, ActivityFeedback as Dt, AiExplanationRequest as E, XAPIObject as En, ActivityDataMap as Et, AiGradingFacts as F, Grader as Fn, DictationEquivalence as Ft, AiProvenance as G, GapSelectLearnerResponse as Gt, AiMaterialFacts as H, TextMatchResult as Hn, GapSelectChoice as Ht, AiGradingJudgement as I, GraderConfiguration as In, DictationLearnerResponse as It, AiRefusal as J, ItemOutcome as Jt, AiReadingFacts as K, InteractionEvent as Kt, AiGradingRequest as L, GraderKind as Ln, DictationSlowMedia as Lt, AiGrade as M, GradeRecord as Mn, DeferredReason as Mt, AiGradedCriterion as N, GradeRouting as Nn, DeferredScoringPartial as Nt, AiFillInTheBlanksFacts as O, XAPIScore as On, ActivityMedia as Ot, AiGradingCriterionFact as P, GradeRoutingReason as Pn, DictationData as Pt, AiTextResult as Q, MediaTrack as Qt, AiGradingRequestProblem as R, GraderUsage as Rn, DictationTolerance as Rt, AiDrafts as S, XAPIConfig as Sn, DraftValidationResult as St, AiDraftsRun as T, XAPIError as Tn, ActivityData as Tt, AiMultipleChoiceFacts as U, levenshteinDistance as Un, GapSelectData as Ut, AiItemFacts as V, TextMatchPolicy as Vn, GapSelectBank as Vt, AiOptionFact as W, matchText as Wn, GapSelectGap as Wt, AiSoundFact as X, LearnerResponseMap as Xt, AiRubricCriterionFact as Y, LearnerResponse as Yt, AiSupportedActivityType as Z, MediaPlaybackPolicy as Zt, AiDraftCaption as _, WrittenResponseLearnerResponse as _n, DraftComplete as _t, AiCoachingRequest as a, ReadAloudDimension as an, AiWritingFeedbackRequest as at, AiDraftSource as b, WrittenResponseRubricLevel as bn, DraftNotComplete as bt, AiCriterionJudgement as c, ReadAloudSlowMedia as cn, MediaProgress as ct, AiCritiqueFinding as d, ScoringDetail as dn, SequenceSlot as dt, MultipleChoiceLearnerResponse as en, AiVideoDraft as et, AiCritiqueFindingResult as f, ScoringOutcome as fn, SequenceSlotGroup as ft, AiDictationFacts as g, WrittenResponseData as gn, TimelineCue as gt, AiCritiqueResult as h, ValidationResult as hn, TimelineChapter as ht, AiCoaching as i, ReadAloudData as in, AiWritingFeedback as it, AiGeneratedDraft as j, CriterionScore as jn, BlankConfig as jt, AiGapFact as k, XAPIStatement as kn, ActivityResult as kt, AiCritique as l, RecordingBounds as ln, MediaTimeline as lt, AiCritiqueRequest as m, ValidationError as mn, StimulusKind as mt, AiActivityInput as n, MultipleChoiceOptionMedia as nn, AiWritingCorrection as nt, AiCoachingResult as o, ReadAloudDimensionWeight as on, AiWritingFeedbackResult as ot, AiCritiqueKind as p, ScoringResult as pn, Stimulus as pt, AiReadingWordFact as q, InteractionKind as qt, AiBlankFact as r, NativeControlHint as rn, AiWritingFacts as rt, AiCoachingWord as s, ReadAloudLearnerResponse as sn, ItemGroup as st, ActivityAiPermissions as t, MultipleChoiceOption as tn, AiWordFact as tt, AiCritiqueField as u, RecordingRef as un, SequenceEntry as ut, AiDraftProblem as v, WrittenResponseRubric as vn, DraftContext as vt, AiDraftsRequest as w, XAPIContextActivities as wn, ItemFindingSeverity as wt, AiDraftType as x, XAPIActor as xn, DraftSeverity as xt, AiDraftSettings as y, WrittenResponseRubricCriterion as yn, DraftIssue as yt, AiGroupInput as z, GradingState as zn, FillInTheBlanksData as zt };
+//# sourceMappingURL=ai-BYryFkF9.d.cts.map

package/dist/ai-BYryFkF9.d.cts.map

@@ -0,0 +1 @@
+{"version":3,"file":"ai-BYryFkF9.d.cts","names":[],"sources":["../src/scoring/text-match.ts","../src/types/grading.ts","../src/types/xapi.ts","../src/types/activity.ts","../src/types/authoring.ts","../src/types/item-group.ts","../src/types/ai.ts"],"mappings":";;;;;;;;;UAQiB;;EAEf;;EAEA;;;;;;;EAOA;;EAEA;;EAEA;;EAEA;;EAEA;;;;;EAKA;;;;;;;;;;;;UAae;EACf;EACA;;;;;;iBAmDc,oBAAoB,WAAW,WAAW;;;;;;;;;;;iBAoC1C,UACd,eACA,sCACA,SAAQ,kBACP;;;;;;;;KCrHS;;;;;;;;;KAUA;;UAGK;EACf,MAAM;;EAEN;;EAEA;;EAEA;;;UAIe;EACf;EACA;EACA;;;UAIe;;EAEf;;;;;;EAMA;;;;;;;;;;;;;;;;EAgBA;;EAEA;;EAEA;;;;;;EAMA;;EAEA;;;;;;;EAOA;;;;;;;UAQe;;EAEf;;EAEA;;EAEA;;EAEA;;;;;;;;;;;;;;;;;;KAmBU;;UASK;;EAEf;;EAEA,SAAS;;EAET;IAAgB;IAAoB;;;;UAIrB;;EAEf;;EAEA;EACA;;EAEA;IAAU;IAAe;;;EAEzB;;;;;;;;;;;;;UAce;EACf;EACA;EACA;;EAEA;;EAEA,WAAW;;;;;;EAMX,UAAU;;EAEV,cAAc;;EAEd;;EAEA;;EAEA;;EAEA;EACA,SAAS;EACT,QAAQ;;EAER;;EAEA,UAAU;;;;;;;;UCrMK;EACf;;EAEA;EACA;;IAEE,OAAO;;IAEP,cAAc;;IAEd;;IAEA;;IAEA;;IAEA,UAAU;MAAQ;MAAY,cAAc;;IAC5C,aAAa;;;;;;;UAQA;EACf;;EAEA;;EAEA;;EAEA;;IAEE;;IAEA;;;;UAKa;;EAEf;;EAEA,SAAS;;;UAIM;;EAEf;;EAEA;;EAEA;;EAEA;;;UAIe;EACf,QAAQ;;EAER;;EAEA;;EAEA;;EAEA;EACA,aAAa;;;UAIE;;EAEf,SAAS;;EAET,WAAW;;EAEX,WAAW;;EAEX,QAAQ;;;UAIO;;EAEf;;EAEA;;EAEA,oBAAoB;EACpB,aAAa;;;UAIE;;EAEf;EACA,OAAO;EACP,MAAM;EACN,QAAQ;EACR,SAAS;EACT,UAAU;;EAEV;EACA;;;;;;UAOe;;EAEf;;EAEA;IAAQ;IAAe;IAAkB;;IAAuB;IAAgB;;;;;;;;;;;;;;;;;EAgBhF,uBAAuB;EACvB,OAAO;;EAEP,WAAW,KAAK;;;UAID;;EAEf,WAAW;;EAEX;;EAEA;;EAEA;;;;;;;;;;;;;;;;;;;UCtIe;EACf,mBAAmB;EACnB,sBAAsB;EACtB,oBAAoB;EACpB,cAAc;EACd,WAAW;EACX,cAAc;;;;;;KAOJ,qBAAqB;;KAGrB,eAAe,gBAAgB;;UAG1B;EACf;;EAEA;;EAEA;;EAEA;;EAEA;;;;;;UAOe;;EAEf;;;;;EAKA;;EAEA;;EAEA;;;;;;;EAOA,SAAS;;EAET;;;;;;EAMA,WAAW;;;;;;;;;;KAWD;;;;;;;;;;;;UAaK;;;;;;EAMf;;;;;;;;;;;EAWA;;EAEA;;EAEA;;EAEA,qBAAqB;;;;;;;UAQN;;EAEf;;EAEA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;UAkCe;;EAEf;;EAEA;;EAEA;;EAEA;;UAGe;;EAEf;;;;;;;;;EASA;;EAEA;;EAEA;;EAEA,QAAQ;;;UAIO;EACf;EACA;;EAEA;;EAEA;;EAEA;;;;;;;;EAQA;;EAEA;;EAEA,SAAS;;EAET;;EAEA,QAAQ;;EAER,WAAW;;EAEX;;;;;;;;;;EAUA;;EAEA;;EAEA;;EAEA;;EAEA,KAAK;;;UAIU;;EAEf;;EAEA;;;;;;;EAOA;;;;;;;EAOA;;;;;;;;EAQA,QAAQ;;EAER;;EAEA;;;;UAKe;;EAEf;;EAEA;;;;;;;;;;;UAYe;;EAEf;;EAEA,SAAS;;;;;;;;;;UAWM;;EAEf;;EAEA,UAAU;;EAEV;;EAEA;;EAEA;;;;;;;;;;;;;;;UAgBe;EACf;EACA;;EAEA;;EAEA;;EAEA;;;;;;;EAOA;;EAEA,MAAM;;EAEN,QAAQ;;EAER;;;;;;;;;;;;EAYA;;;;;;;;EAQA;;EAEA,QAAQ;;EAER,WAAW;;EAEX;;EAEA;;EAEA;;EAEA;;EAEA,KAAK;;UAGU;EACf;EACA;;EAEA;;EAEA;;EAEA;;;;;;EAMA;;EAEA,QAAQ;;EAER;;EAEA,QAAQ;;EAER,WAAW;;EAEX;;EAEA;;EAEA;;EAEA;;EAEA,KAAK;;;;;;;UAQU;;EAEf;;EAEA;;;UAIe;;EAEf;;EAEA;;EAEA;;;;;;;;;;;;;;EAcA,SAAS;;;UAIM;;EAEf;;EAEA,UAAU;;;;;;;;;;;;UAaK;EACf;EACA;;EAEA;;EAEA;;EAEA;;;;;;EAMA;;EAEA;;EAEA;;EAEA,SAAS;;EAET;;EAEA,QAAQ;;EAER,WAAW;;EAEX;;EAEA;;EAEA;;EAEA;;EAEA,KAAK;;;;;;;;;UAUU;EACf;;EAEA;;EAEA;;;;;;;;;;UAWe;;EAEf;;EAEA;;;UAIe;;EAEf,eAAe;;;;;;;;;;;;;;;;UAiBA;EACf;EACA;;EAEA;;;;;EAKA;;;;;;EAMA;;;;;;;;EAQA;;;;;;;EAOA,QAAQ;;EAER,YAAY;;;;;;;EAOZ;IAAU;;;EAEV,YAAY;;EAEZ,WAAW;;EAEX;;EAEA;;EAEA;;EAEA;;EAEA,KAAK;;;KAIK;;UAGK;EACf,MAAM;;;;;EAKN;;;;;;;UAQe;;EAEf;;EAEA;;EAEA;;;;;;;;UASe;EACf;;EAEA;;EAEA;;;;;;;;;;;UAYe;EACf;EACA;;EAEA;;EAEA;;EAEA;;;;;;EAMA;;;;;EAKA;;EAEA,QAAQ;;EAER,YAAY;;EAEZ,WAAW;;;;;;;EAOX;IAAW,YAAY;;;EAEvB;;EAEA,WAAW;;EAEX;;EAEA;;EAEA,KAAK;;;;;;UAOU;EACf,mBAAmB;EACnB,sBAAsB;EACtB,oBAAoB;EACpB,cAAc;EACd,WAAW;EACX,cAAc;;;KAIJ,kBAAkB,yBAAyB;;UAGtC;EACf;;EAEA;;;;;;;;EAQA;;;UAIe;EACf;;EAEA,SAAS;;;;;;;;EAQT;;;;;;;;;;;UAYe;EACf;;EAEA,YAAY;;;;;;;;EAQZ;;;UAIe;EACf;;EAEA;;EAEA;;;;;;;;;;UAWe;EACf;;EAEA;;;;;EAKA;;;;;;;UAQe;EACf;;EAEA;EACA;;;;;;UAOe;EACf;;EAEA,WAAW;;EAEX;;;;;;;;;;KAWU;;UAGK;;EAEf;;;;;;;;;;;;EAYA,SAAS;;EAET;;EAEA;;;;;EAKA;;;;;;;;;EASA;;;UAIe;;EAEf;;EAEA;;EAEA;;;;;;EAMA;;EAEA,SAAS;;;;;;;;UASM;;EAEf;;EAEA;GACC;;;;;;;;;;;;;;;;;;;;;;;KAwBS;;;;;;;KAYA;EAEN;;EAEA;;EAEA;;EAEA;;EAEA;;EAEA,SAAS;;EAGT;;EAEA,QAAQ;;EAER;;EAEA,UAAU;;;;;;EAMV,gBAAgB;;;;;;;EAOhB,eAAe;;EAGf;;;;;;EAMA,OAAO;;EAEP;EACA;EACA;;EAEA;;EAGA;;EAEA;;EAEA;;;;;EAKA;;;UAIW;;EAEf;;EAEA;;EAEA;;;;;;;KAQU,iBAAiB;EACvB;EAAe,MAAM;;EACrB;EAAgB,QAAQ;;;UAGb;;EAEf;;EAEA;;EAEA;;EAEA;;EAEA,eAAe;;;;;;;KAQL;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;UAyGK;;EAEf,MAAM;;EAEN;;EAEA;;EAEA,SAAS;;;;;;;;;;;;;;KClpCC;;;;;UAMK,mBAAmB;EAClC,UAAU;;;UAIK,cAAc;EAC7B;;EAEA,MAAM;;EAEN,QAAQ;;;;;;;UAQO;EACf;EACA,QAAQ;;;;;;;;;;KAWE,sBAAsB,KAAK,cAAc,KAAK;;;;;;;;;;;;KAa9C;;;;;;UAOK,oBAAoB;EACnC,UAAU;;;UAIK;;;;;;;EAOf;;;;;KC9EU;;;;;;;;;;;UAYK;EACf;EACA,MAAM;;EAEN;;;;;EAKA;;;;;EAKA;;EAEA,QAAQ;;;;;EAKR;;EAEA;;EAEA;;;;;;;UAQe;;EAEf;;;;;;EAMA;;;;;EAKA;;EAEA;;;;;EAKA;;;UAIe;;EAEf;;EAEA;;;;;;;;;;UAWe;EACf,MAAM;EACN,WAAW;;;;;;EAMX;;;;;;;;UASe;EACf;;EAEA;;;;;EAKA;;;;;;;;;;;;;UAce,UAAU,QAAQ;EACjC;EACA;EACA;EACA;;;;;;;EAOA;EACA,UAAU;;;;;;;;;;EAUV,QAAQ;IAAU;;;;;;;EAMlB;;;;;;;;EAQA,WAAW;;;;;;;;;;;;;KAcD,cAAc,QAAQ,iBAAiB;EAAU;KAAsB,UAAU;;UAG5E;EACf;EACA;EACA,UAAU;;EAEV;;EAEA;;EAEA,WAAW;;EAEX,MAAM;;;UAIS,aAAa,QAAQ;;;;;;;;;;;;;;;;;;EAkBpC;;EAEA;EACA,UAAU;;EAEV,QAAQ;;;;;;;;;;;;;UCjMO;;;;;;EAMf;;EAEA;;;;;;;;UASe;WACN;WACA;WACA;WACA;WACA;WACA;;;KAIC;;KAGA;;;;;UAUK;EACf;EACA;EACA;EACA,UAAU;;;KAIA;UAEF;EACR;EACA;;EAEA;;;UAIe;EACf;EACA;EACA;;EAEA;;EAEA;;UAGe,8BAA8B;EAC7C;EACA;EACA;EACA,SAAS;;;UAIM;EACf;;EAEA;;EAEA;;EAEA;;EAEA;;EAEA;;EAEA;;UAGe,+BAA+B;EAC9C;;EAEA;EACA,QAAQ;;;UAIO;EACf;;EAEA;;EAEA;;EAEA;;EAEA;;EAEA;;EAEA;;UAGe,yBAAyB;EACxC;;EAEA;EACA,MAAM;;;UAIS;;EAEf;;EAEA;EACA;;UAGe,yBAAyB;EACxC;;EAEA;;EAEA;;EAEA,OAAO;;;;;;KAOG,cACR,wBACA,yBACA,mBACA;;;;;;;UAQa;WACN;WACA;;;;;;;;;UAUM;;EAEf,MAAM;;EAEN;;;;;;EAMA;;EAEA;;EAEA;IAAU,MAAM;IAAuB;;;EAEvC;;EAEA;;;;;;;UAQe;EACf;EACA,OAAO;EACP,OAAO;;EAEP,WAAW;;EAEX;;;;;;;UAQe;EACf;EACA,OAAO;;EAEP;;EAEA;;EAEA,WAAW;EACX;;;UAIe;;EAEf;;EAEA;;EAEA;;;;;;;UAQe;EACf;;EAEA,UAAU;EACV,aAAa;;;;;;;;;;;EAWb,QAAQ;;;;;;;;;;;;;;;;;KAkBE;;UAeK;EACf;EACA;EACA;;;;;;UAOe;EACf;EACA;EACA;;EAEA;EACA;;EAEA;;EAEA;EACA;EACA;EACA;;EAEA,QAAQ;;EAER;;;;;;UAOe;EACf;EACA,OAAO;;EAEP;;EAEA;;;;;EAKA,WAAW;;EAEX;;;;;;;;UASe;EACf;EACA;EACA;;EAEA;EACA;IAAU;IAAe;;;;;;;KAOf,uBAAuB,KAAK;EAA8B;;;;;;UAMrD;;EAEf;EACA,cAAc;EACd,WAAW;EACX,aAAa;EACb,QAAQ;;;;;UAMO;EACf;;;;;EAKA,aAAa;;EAEb,UAAU;;;;;;;;;;;EAWV;EACA,aAAa;EACb,QAAQ;;;UAMO;EACf;EACA;EACA;;;;;EAKA;IAAU;IAAe;;;;;;;UAOV;EACf;EACA;EACA;;EAEA;EACA;;EAEA;;EAEA;EACA;EACA;EACA;;EAEA,QAAQ;;EAER;;;;;;UAOe;EACf;EACA,OAAO;;EAEP,WAAW;;EAEX;;;;;;;;;;;;;;;;KAiBU;;UASK;;EAEf;;EAEA;;;;;EAKA;;EAEA;EACA;;;;;;;UAQe;;EAEf,UAAU;;EAEV;;EAEA,aAAa;;EAEb;;EAEA;EACA,aAAa;EACb,QAAQ;;;UAMO;;EAEf;;EAEA;;EAEA;IAAY;IAAgB;;;;UAIb;;EAEf;;EAEA;;EAEA;EACA;;EAEA;;EAEA,SAAS;;;;;;;UAQM;EACf;EACA;EACA;;EAEA;EACA;EACA;;EAEA;;EAEA;;EAEA;IAAU;IAAmB;IAAkB;IAAuB;;EACtE,OAAO;;;;;;UAOQ;EACf;EACA,OAAO;;EAEP;IAAU;IAAe;IAAkB;;;EAE3C;;;UAIe;;EAEf;;EAEA;;;;;EAKA;IAAU;IAAkB;;;;UAIb;;EAEf;EACA,QAAQ;EACR,aAAa;EACb,QAAQ;;;UAIO;EACf;;;;;EAKA,QAAQ;IAAmB;;EAC3B,aAAa;EACb,QAAQ;;;;;;;;;;;;;;;;;KAoBE;;UAYK;EACf;EACA;;;;;;;UAQe;EACf;EACA;IACE;;IAEA,MAAM,SAAS;;IAEf,QAAQ;;;;;IAKR;MAAY;MAAgB;MAAc;;;IAE1C;;;EAGF;;;UAIe;;EAEf;EACA,MAAM;;EAEN;;EAEA;;;UAIe;EACf,UAAU;EACV,aAAa;EACb,QAAQ;;;;;;;UAQO,0BAA0B;EACzC;EACA;;;UAIe;EACf,UAAU;EACV,aAAa;EACb,QAAQ;;;;;;KASE;;;;;;KAaA;EACN;EAAgC;;EAChC;EAAkB;IAAQ;IAAe;IAAa;;;;UAG3C;EACf;EACA;EACA;EACA;;;UAIe;EACf;EACA;EACA;EACA;;;;;;;;;KAUU,kBAAkB,QAAQ,OAAO,aAAa,SAAS;;;;;;UAOlD;EACf;EACA;;;;;;IAME,eAAe;IACf;MACM;MAAgC;;MAChC;MAAkB,UAAU;;;IAElC;;IAEA;;IAEA;;IAEA;;;;;;;;;EASF,OAAO,SAAS;;EAEhB,WAAW;;;;;;EAMX;IACE;IACA,OAAO,SAAS;IAChB,UAAU;;;;UAKG;;EAEf;EACA,MAAM;;EAEN,OAAO;;EAEP,YAAY;;EAEZ,UAAU;;;;;;EAMV;;;;;;EAMA;IAAS;IAAe;;;EAExB,WAAW,SAAS;;;UAIL;;EAEf,QAAQ;;EAER,UAAU;EACV,aAAa;EACb,QAAQ;;;KAIE;EACN;EAAU,aAAa;EAAc,QAAQ;;EAC7C;EAAW,UAAU;EAAW;;;UAGrB;;EAEf,QAAQ;;EAER,UAAU;;EAEV,OAAO;;;;;;;;UASQ;;EAEf,OAAO;;EAEP,YAAY;;EAEZ,UAAU"}

package/dist/{ai-CnmLPMzd.d.ts → ai-BYryFkF9.d.ts}

@@ -134,6 +134,57 @@ interface CriterionScore {
   weight?: number;
   /** Not applicable to this submission (e.g. interaction on a monologue task). */
   notApplicable?: boolean;
+  /**
+   * How likely each of the criterion's levels was, lowest first, when the
+   * grader said: a distribution, whose likeliest level is `band`. Kept on the
+   * record as the evidence a routing rule read (`routeGrade`) and a
+   * calibration measures (`calibrationReport`).
+   */
+  levelProbabilities?: number[];
+}
+/**
+ * What one calibration certifies: a model, a prompt revision, a rubric and a
+ * target language. A change to any of them is a different grader, measured
+ * afresh. Built by `graderConfiguration`.
+ */
+interface GraderConfiguration {
+  /** The model's identifier, from the grade's `grader.model`. */
+  model: string;
+  /** The prompt revision, from the grade's `grader.promptHash`. */
+  promptHash: string;
+  /** `contentHash` of the item's rubric, levels and descriptors included. */
+  rubricHash: string;
+  /** The item's `languageTarget`, or `null` when it has none. */
+  languageTarget: string | null;
+}
+/**
+ * Why `routeGrade` sent a grade to a person, in this order:
+ *
+ * - `no-calibration` — in `exam`, no calibration report for this grader's exact
+ *   configuration was supplied: a model's grade counts on a paper of record
+ *   only once it has been measured;
+ * - `calibration-below-policy` — the report misses one of the policy's
+ *   thresholds, for a criterion or for the total;
+ * - `unsure` — a criterion's likeliest level is less likely than the policy's
+ *   `minLevelProbability`;
+ * - `model-asked` — the grader set `requiresHumanReview`;
+ * - `low-confidence` — the grader's own `confidence` is `low`, and the policy
+ *   routes on it;
+ * - `audit-sample` — the response is in the audit sample: a person marks it
+ *   too, without the model's grade, to keep the calibration current.
+ */
+type GradeRoutingReason = 'no-calibration' | 'calibration-below-policy' | 'unsure' | 'model-asked' | 'low-confidence' | 'audit-sample';
+/** Where a grade went, and why: the record an appeal reads. */
+interface GradeRouting {
+  /** `model` when its grade counts; `person` when a person must mark it first. */
+  to: 'model' | 'person';
+  /** Every reason it went to a person, in {@link GradeRoutingReason}'s order; empty when it did not. */
+  reasons: GradeRoutingReason[];
+  /** The calibration report the decision read, by its hash and size, when one matched. */
+  calibration?: {
+    reportHash: string;
+    pairs: number;
+  };
 }
 /** A suggested correction anchored in the learner's own text. */
 interface InlineCorrection {
@@ -189,6 +240,8 @@ interface GradeRecord {
   usage?: GraderUsage;
   /** ISO 8601 timestamp of when the grade was produced. */
   gradedAt?: string;
+  /** Where `routeGrade` sent this grade, and why, when it was routed. */
+  routing?: GradeRouting;
 }
 //#endregion
 //#region src/types/xapi.d.ts
@@ -770,6 +823,17 @@ interface FillInTheBlanksData {
   /** What an author allows AI to do for this item: see {@link ActivityAiPermissions}. */
   ai?: ActivityAiPermissions;
 }
+/**
+ * One level of a rubric criterion: what a response at this level does. Levels
+ * run from the lowest to the highest, and a grader judging against them names
+ * one.
+ */
+interface WrittenResponseRubricLevel {
+  /** A short name, unique within its criterion: "Developing", "B1". */
+  label: string;
+  /** What a response at this level does, for the grader and the learner. */
+  description?: string;
+}
 /** A single criterion within a written-response grading rubric. */
 interface WrittenResponseRubricCriterion {
   /** Criterion name (e.g. "Task achievement", "Grammar range"). */
@@ -778,6 +842,20 @@ interface WrittenResponseRubricCriterion {
   description?: string;
   /** Non-negative weight of this criterion in the overall grade. */
   weight: number;
+  /**
+   * The criterion's levels, lowest first: 2 to 10, each with a label unique
+   * within the criterion once case and spacing are ignored. A grader judging
+   * against levels names one, and the criterion scores the level's place on
+   * the scale, `index / (levels.length - 1)`: 0 for the lowest, 1 for the
+   * highest.
+   *
+   * Read by the grading functions (`aiGradingRequest`, `gradeFromJudgement`,
+   * `calibrationReport`), which report levels they cannot use. In 1.x
+   * `validateActivity` does not check it: your own fields pass through it, and
+   * a field of yours named `levels` stays valid. The item critic warns about
+   * levels the grading functions cannot read.
+   */
+  levels?: WrittenResponseRubricLevel[];
 }
 /** Grading rubric attached to a written-response activity. */
 interface WrittenResponseRubric {
@@ -1232,8 +1310,12 @@ interface DeferredScoringPartial {
  *   a finite number, a `maxScore` of 0 or less, or a score outside 0 to
  *   `maxScore`). The record is kept on `rejectedGrade`. A real grade is still
  *   owed, so re-grade the slot: waiting will not bring one.
+ * - `awaiting_human_review` — a model graded the response, and a routing rule
+ *   (`routeGrade`) sent it to a person: the grade does not count until a
+ *   person marks it. The model's grade is kept on `pendingGrade`, for the
+ *   person and for the next calibration.
  */
-type DeferredReason = 'requires_async_grading' | 'no_response_recorded' | 'grade_rejected';
+type DeferredReason = 'requires_async_grading' | 'no_response_recorded' | 'grade_rejected' | 'awaiting_human_review';
 /**
  * The outcome of evaluating a learner response against an activity — the
  * union `evaluate()` returns. Unlike {@link ScoringResult}, it can express
@@ -1266,6 +1348,13 @@ type ItemOutcome = {
    * is mirrored onto the outcome, and nothing reads it as a grade.
    */
   rejectedGrade?: GradeRecord;
+  /**
+   * The model's grade a routing rule held for a person, when `reason` is
+   * `awaiting_human_review`, with the decision on its `routing`. Kept for
+   * the person and for the next calibration; nothing on it is mirrored
+   * onto the outcome, and nothing reads it as a grade.
+   */
+  pendingGrade?: GradeRecord;
 } | {
   status: 'graded';
   /**
@@ -1951,8 +2040,15 @@ interface AiTextResult {
  * a word the engine did not mark, or on a sound it did not report.
  * `contradicts-item` is a critique of an item that points at a field the item
  * does not have, or quotes words that field does not contain.
- */
-type AiRefusal = 'malformed' | 'empty' | 'too-long' | 'contradicts-grade' | 'reveals-answer' | 'misquotes-answer' | 'contradicts-marks' | 'contradicts-item';
+ * `misses-criteria` is a grade that leaves a rubric criterion out, renames
+ * one, judges one twice, or calls one not applicable: judging fewer criteria
+ * is judging a different task, and the total would be renormalised over the
+ * rest. `out-of-scale` is a judgement off its criterion's scale: a level the
+ * criterion does not have, a score outside 0 to 1, or level probabilities that
+ * are not a distribution over its levels, or whose likeliest level is not the
+ * one chosen.
+ */
+type AiRefusal = 'malformed' | 'empty' | 'too-long' | 'contradicts-grade' | 'reveals-answer' | 'misquotes-answer' | 'contradicts-marks' | 'contradicts-item' | 'misses-criteria' | 'out-of-scale';
 /** One criterion of a written response's rubric, as the author wrote it. */
 interface AiRubricCriterionFact {
   name: string;
@@ -2063,6 +2159,104 @@ interface AiWritingFeedback {
   provenance?: AiProvenance;
   usage?: GraderUsage;
 }
+/** One criterion of the rubric a model grades against, as the author wrote it. */
+interface AiGradingCriterionFact {
+  name: string;
+  description?: string;
+  weight: number;
+  /**
+   * The levels to choose from, lowest first, as the author wrote them; `null`
+   * when the criterion is judged on a score from 0 to 1.
+   */
+  levels: {
+    label: string;
+    description?: string;
+  }[] | null;
+}
+/**
+ * What a model is told about a submitted written response it grades: the task
+ * as the learner saw it, the response verbatim, and the rubric with its levels.
+ */
+interface AiGradingFacts {
+  activityType: 'written-response';
+  activityId: string;
+  title: string;
+  /** The language the item is written in, when the author gave one. */
+  locale?: string;
+  prompt: string;
+  /** The response, exactly as submitted. Every correction quotes it. */
+  text: string;
+  /** Recomputed with `countWords`, as the grader counts it. */
+  wordCount: number;
+  minWords: number;
+  maxWords: number;
+  withinWordBounds: boolean;
+  /** Every criterion of the item's rubric, in its order. */
+  rubric: AiGradingCriterionFact[];
+  /** The target language and level, when the author gave one (e.g. `"en-B1"`). */
+  languageTarget?: string;
+}
+/**
+ * Asks a model to grade a submitted written response. Built on the server,
+ * from the whole item: grading needs the rubric, never the learner's view.
+ */
+interface AiGradingRequest {
+  feature: 'grading';
+  facts: AiGradingFacts;
+  /** The material of the group the task sits in; absent outside a group. */
+  material?: AiMaterialFacts;
+  /** The language to write comments and feedback in, when known. */
+  learnerLocale?: string;
+}
+/**
+ * Why `aiGradingRequest` built no request:
+ *
+ * - `not-written-response` — the item is another type;
+ * - `redacted` — a `redact()` projection: grading is done on the server, from
+ *   the whole item;
+ * - `no-rubric` — no rubric, or a criterion without a name and a weight: a
+ *   model has nothing to be held to;
+ * - `duplicate-criteria` — two criteria with one name once case and spacing are
+ *   ignored: a model cannot judge them apart (`path` names the later one);
+ * - `unreadable-levels` — a criterion's `levels` are there but cannot be used
+ *   (`path` names them; the item critic's `wr_levels_unreadable` says why);
+ * - `empty-response` — nothing was written, so there is nothing to judge.
+ */
+type AiGradingRequestProblem = 'not-written-response' | 'redacted' | 'no-rubric' | 'duplicate-criteria' | 'unreadable-levels' | 'empty-response';
+/** A model's judgement of one criterion, as `checkAiGrading` accepted it. */
+interface AiGradedCriterion {
+  /** The criterion's name, as the author wrote it. */
+  name: string;
+  /** The level chosen, as the author labelled it, on a criterion with levels. */
+  level?: string;
+  /**
+   * From 0 to 1: the level's place on the scale, `index / (levels - 1)`, or the
+   * model's score on a criterion without levels.
+   */
+  score: number;
+  /** How likely each level is, lowest first, when the model said: a distribution. */
+  probabilities?: number[];
+  comment?: string;
+}
+/**
+ * A model's grade of a written response as `checkAiGrading` accepted it: every
+ * criterion of the rubric judged, on its own scale. The total is not here —
+ * `gradeFromJudgement` computes it.
+ */
+interface AiGradingJudgement {
+  /** Every criterion of the rubric, once, in the rubric's order. */
+  criteria: AiGradedCriterion[];
+  /** Narrative feedback addressed to the learner. */
+  feedback: string | null;
+  /** Corrections, each anchored in the response, as on feedback on writing. */
+  corrections: InlineCorrection[];
+  /** How the model reached its verdict; for the record, never shown. */
+  rationale?: string;
+  /** The model asked for a person to mark this response. */
+  requiresHumanReview: boolean;
+  provenance?: AiProvenance;
+  usage?: GraderUsage;
+}
 /** One sound of a word, as the engine reported it. */
 interface AiSoundFact {
   /** The sound expected, in the assessment's phoneme alphabet. */
@@ -2413,5 +2607,5 @@ interface AiVideoDraft {
   findings: ItemFinding[];
 }
 //#endregion
-export { AiWritingFeedbackRequest as $, ReadAloudDimension as $t, AiGapSelectFacts as A, InlineCorrection as An, DictationLearnerResponse as At, AiReadingFacts as B, InteractionEvent as Bt, AiDraftsCall as C, XAPIVerbObject as Cn, ActivityResult as Ct, AiFeature as D, GraderKind as Dn, DeferredScoringPartial as Dt, AiExplanationRequest as E, Grader as En, DeferredReason as Et, AiItemFacts as F, GapSelectBank as Ft, AiSupportedActivityType as G, MediaPlaybackPolicy as Gt, AiRefusal as H, ItemOutcome as Ht, AiMaterialFacts as I, GapSelectChoice as It, AiVideoDraft as J, MultipleChoiceLearnerResponse as Jt, AiTextResult as K, MediaTrack as Kt, AiMultipleChoiceFacts as L, GapSelectData as Lt, AiGrade as M, TextMatchResult as Mn, DictationTolerance as Mt, AiGroupInput as N, levenshteinDistance as Nn, FillInTheBlanksData as Nt, AiFillInTheBlanksFacts as O, GraderUsage as On, DictationData as Ot, AiHintRequest as P, matchText as Pn, FillInTheBlanksLearnerResponse as Pt, AiWritingFeedback as Q, ReadAloudData as Qt, AiOptionFact as R, GapSelectGap as Rt, AiDrafts as S, XAPIStatement as Sn, ActivityMedia as St, AiDraftsRun as T, GradeRecord as Tn, BlankConfig as Tt, AiRubricCriterionFact as U, LearnerResponse as Ut, AiReadingWordFact as V, InteractionKind as Vt, AiSoundFact as W, LearnerResponseMap as Wt, AiWritingCorrection as X, MultipleChoiceOptionMedia as Xt, AiWordFact as Y, MultipleChoiceOption as Yt, AiWritingFacts as Z, NativeControlHint as Zt, AiDraftCaption as _, XAPIContextActivities as _n, ItemFinding as _t, AiCoachingRequest as a, ScoringDetail as an, SequenceSlot as at, AiDraftSource as b, XAPIResult as bn, ActivityDataMap as bt, AiCriterionJudgement as c, ValidationError as cn, StimulusKind as ct, AiCritiqueFinding as d, WrittenResponseLearnerResponse as dn, DraftComplete as dt, ReadAloudDimensionWeight as en, AiWritingFeedbackResult as et, AiCritiqueFindingResult as f, WrittenResponseRubric as fn, DraftContext as ft, AiDictationFacts as g, XAPIContext as gn, DraftValidationResult as gt, AiCritiqueResult as h, XAPIConfig as hn, DraftSeverity as ht, AiCoaching as i, RecordingRef as in, SequenceEntry as it, AiGeneratedDraft as j, TextMatchPolicy as jn, DictationSlowMedia as jt, AiGapFact as k, GradingState as kn, DictationEquivalence as kt, AiCritique as l, ValidationResult as ln, TimelineChapter as lt, AiCritiqueRequest as m, XAPIActor as mn, DraftNotComplete as mt, AiActivityInput as n, ReadAloudSlowMedia as nn, MediaProgress as nt, AiCoachingResult as o, ScoringOutcome as on, SequenceSlotGroup as ot, AiCritiqueKind as p, WrittenResponseRubricCriterion as pn, DraftIssue as pt, AiVerdict as q, MultipleChoiceData as qt, AiBlankFact as r, RecordingBounds as rn, MediaTimeline as rt, AiCoachingWord as s, ScoringResult as sn, Stimulus as st, ActivityAiPermissions as t, ReadAloudLearnerResponse as tn, ItemGroup as tt, AiCritiqueField as u, WrittenResponseData as un, TimelineCue as ut, AiDraftProblem as v, XAPIError as vn, ItemFindingSeverity as vt, AiDraftsRequest as w, CriterionScore as wn, ActivityType as wt, AiDraftType as x, XAPIScore as xn, ActivityFeedback as xt, AiDraftSettings as y, XAPIObject as yn, ActivityData as yt, AiProvenance as z, GapSelectLearnerResponse as zt };
-//# sourceMappingURL=ai-CnmLPMzd.d.ts.map
+export { AiVerdict as $, MultipleChoiceData as $t, AiGapSelectFacts as A, XAPIVerbObject as An, ActivityType as At, AiHintRequest as B, InlineCorrection as Bn, FillInTheBlanksLearnerResponse as Bt, AiDraftsCall as C, XAPIContext as Cn, ItemFinding as Ct, AiFeature as D, XAPIResult as Dn, ActivityFeedback as Dt, AiExplanationRequest as E, XAPIObject as En, ActivityDataMap as Et, AiGradingFacts as F, Grader as Fn, DictationEquivalence as Ft, AiProvenance as G, GapSelectLearnerResponse as Gt, AiMaterialFacts as H, TextMatchResult as Hn, GapSelectChoice as Ht, AiGradingJudgement as I, GraderConfiguration as In, DictationLearnerResponse as It, AiRefusal as J, ItemOutcome as Jt, AiReadingFacts as K, InteractionEvent as Kt, AiGradingRequest as L, GraderKind as Ln, DictationSlowMedia as Lt, AiGrade as M, GradeRecord as Mn, DeferredReason as Mt, AiGradedCriterion as N, GradeRouting as Nn, DeferredScoringPartial as Nt, AiFillInTheBlanksFacts as O, XAPIScore as On, ActivityMedia as Ot, AiGradingCriterionFact as P, GradeRoutingReason as Pn, DictationData as Pt, AiTextResult as Q, MediaTrack as Qt, AiGradingRequestProblem as R, GraderUsage as Rn, DictationTolerance as Rt, AiDrafts as S, XAPIConfig as Sn, DraftValidationResult as St, AiDraftsRun as T, XAPIError as Tn, ActivityData as Tt, AiMultipleChoiceFacts as U, levenshteinDistance as Un, GapSelectData as Ut, AiItemFacts as V, TextMatchPolicy as Vn, GapSelectBank as Vt, AiOptionFact as W, matchText as Wn, GapSelectGap as Wt, AiSoundFact as X, LearnerResponseMap as Xt, AiRubricCriterionFact as Y, LearnerResponse as Yt, AiSupportedActivityType as Z, MediaPlaybackPolicy as Zt, AiDraftCaption as _, WrittenResponseLearnerResponse as _n, DraftComplete as _t, AiCoachingRequest as a, ReadAloudDimension as an, AiWritingFeedbackRequest as at, AiDraftSource as b, WrittenResponseRubricLevel as bn, DraftNotComplete as bt, AiCriterionJudgement as c, ReadAloudSlowMedia as cn, MediaProgress as ct, AiCritiqueFinding as d, ScoringDetail as dn, SequenceSlot as dt, MultipleChoiceLearnerResponse as en, AiVideoDraft as et, AiCritiqueFindingResult as f, ScoringOutcome as fn, SequenceSlotGroup as ft, AiDictationFacts as g, WrittenResponseData as gn, TimelineCue as gt, AiCritiqueResult as h, ValidationResult as hn, TimelineChapter as ht, AiCoaching as i, ReadAloudData as in, AiWritingFeedback as it, AiGeneratedDraft as j, CriterionScore as jn, BlankConfig as jt, AiGapFact as k, XAPIStatement as kn, ActivityResult as kt, AiCritique as l, RecordingBounds as ln, MediaTimeline as lt, AiCritiqueRequest as m, ValidationError as mn, StimulusKind as mt, AiActivityInput as n, MultipleChoiceOptionMedia as nn, AiWritingCorrection as nt, AiCoachingResult as o, ReadAloudDimensionWeight as on, AiWritingFeedbackResult as ot, AiCritiqueKind as p, ScoringResult as pn, Stimulus as pt, AiReadingWordFact as q, InteractionKind as qt, AiBlankFact as r, NativeControlHint as rn, AiWritingFacts as rt, AiCoachingWord as s, ReadAloudLearnerResponse as sn, ItemGroup as st, ActivityAiPermissions as t, MultipleChoiceOption as tn, AiWordFact as tt, AiCritiqueField as u, RecordingRef as un, SequenceEntry as ut, AiDraftProblem as v, WrittenResponseRubric as vn, DraftContext as vt, AiDraftsRequest as w, XAPIContextActivities as wn, ItemFindingSeverity as wt, AiDraftType as x, XAPIActor as xn, DraftSeverity as xt, AiDraftSettings as y, WrittenResponseRubricCriterion as yn, DraftIssue as yt, AiGroupInput as z, GradingState as zn, FillInTheBlanksData as zt };
+//# sourceMappingURL=ai-BYryFkF9.d.ts.map