quizwitz-validator 0.0.0-stage → 0.2.0

package/docs/design.md

@@ -0,0 +1,161 @@
+# Findings: rules as data, one evaluator
+
+Date: 2026-10-08 (phase one)
+Repos: `client.quizwitz.com` (declares the rules), `quizwitz-validator` (this package, published on npm from the quizwitz-findings repo, evaluates them), `quizwitz-social` (editor) and `quizwitz-mcp` (consumers, switched in a later phase).
+
+## 1. Goal
+
+The editor's "findings" (validation issues and surprising-settings warnings) were code in `quizwitz-social` (`question-anomaly.ts`, `quiz-findings.service.ts`, `question-type-descriptor.ts`), with a few rules already declared in the game client's type manifests (`completeness`, `warnings`, `settingWarnings`, `questionSettingWarnings`). The MCP server had a third, smaller set of its own. After this work:
+
+- **Every rule is data** in the game client's served manifests (`<game client>/types/manifests.json`), next to the type it belongs to. A mod declares its rules in its own `config.json`, the same way.
+- **One evaluator**, this package, turns manifests + a quiz or round (REST aggregate shape) into `{errors, warnings}`. The editor and the MCP server call it instead of carrying rule code.
+- **The fixtures are the contract**: the game client serves `types/findings-fixtures.json`, and this package's test suite asserts every fixture. A rule change in the game client that changes behaviour shows up as a failing fixture here, and vice versa.
+- **Every rule id and ignore key stays exactly as the editor produces it today**, so the `ignoredWarnings` lists stored on questions, rounds and quizzes keep working.
+
+## 2. Why the rules are data and the evaluator is code
+
+The rule *parameters* (which settings are compared, outlier factor 3, a population of at least 4, which fields count as narration, what the message says) are facts about how QuizWitz plays and are the game client's to own: the game client is the source of truth for types (types spec D1), and a mod that adds a round type is the only one who knows that its questions need a theme. Those belong in the manifest, next to `allowedQuestionTypes` and `settingsFields`.
+
+The rule *mechanics* (what "outlier against the median" means, how a tri-state setting counts, how ignore keys are formed) are shared behaviour that must be identical in the editor and the MCP server, and must stay stable for stored ignore keys. That is code, tested once, in a package both consume. A manifest cannot carry code (the editor is an Angular app that must not eval third-party JS; the MCP server is a Node service that must not either), so the vocabulary of rule *kinds* is closed and lives in the evaluator; a mod picks a kind and fills in the parameters.
+
+## 3. Manifest schema
+
+### 3.1 Format: additive on format 1
+
+The served file stays `{ "format": 1, "core": <manifest>, "mods": [<manifest>, ...] }`. The new fields (`findings` at the core manifest level, `findings` and `guidance` on every type entry) are **additive**: a reader of format 1 that does not know them ignores them (the editor's and the MCP's validators keep unknown keys; the game's `ManifestValidator` validates them when present). `format` is raised only when an old reader would *misread* the new shape (types spec 4.1); here it would not, and both current consumers refuse any `format !== 1` outright (`parseTypeManifests`, `parseServedManifests`), so a bump would take every editor and MCP offline for a change they can safely ignore. Hence format 1.
+
+### 3.2 Where rules live
+
+| Where | Key | Scope |
+|---|---|---|
+| Core manifest root (`core.findings`) | `findings` | Every round with questions, every question in it. The round-wide comparative rules the editor had in code. |
+| A round type entry | `findings` | Rounds of that type, and every question in them. |
+| A scoring strategy entry | `findings` | Rounds scored with that strategy, and every question in them. |
+| A question type entry | `findings` | Questions of that type (per-question kinds only). |
+| Any type entry | `guidance` | Prose for authors and AI assistants: what makes a good round/question of this type. A text, or a text per language. |
+
+A mod's `config.json` uses the same keys on its type entries; a mod has no root-level `findings`.
+
+The existing editor-facing fields stay and keep their meaning; the evaluator reads them too, so behaviour is identical whether a rule is written the old way or the new way:
+
+| Existing field | On | Evaluated as |
+|---|---|---|
+| `completeness` (`requireText`, `requireCorrectOption`, `requireAllOptionsFilled`, `requireAnswer`, `requireImageMap`) | question type | error, type `question` (ruleId `completeness:<name>`) |
+| `warnings` (`optionsOver`+`settingFalse`, `typeSetting`+`typeSettingBelow`) | question type | warning, ruleId `type-warning:<index>` |
+| `settingWarnings` (`setting`, `equals`, `message`) | question type | warning, ruleId `setting:type:<setting>` |
+| `questionSettingWarnings` | round type, scoring strategy | warning, ruleId `setting:round:<setting>` (a strategy rule for a setting the round type already warns about is dropped) |
+| `defaults`, `questionDefaults`, `settingsFields` | | read for the round-settings error (`canUseDefaults`) |
+
+They are not duplicated into `findings`; a later phase may move them once both consumers read `findings` only. New rules go in `findings`.
+
+### 3.3 A rule instance
+
+```json
+{ "id": "points-outlier", "rule": "outlier", "field": "points", "factor": 3, "minPopulation": 3,
+  "message": "Worth {value} points while the rest of this round is around {median}",
+  "description": "Points far above or below the rest of the round" }
+```
+
+| Key | Meaning |
+|---|---|
+| `id` | The **rule id**, stable forever: it is what `ignoredWarnings` stores (`points-outlier`, `odd:hideQuestion`, `content-gap:long`, ...). Unique within its list. |
+| `rule` | The kind, one of the vocabulary below. |
+| `severity` | `warning` (default for every kind but `require`) or `error`. A warning is a `Finding` with an id and can be ignored; an error is a `ValidationIssue`. |
+| `message` | What the author sees, with `{placeholder}`s. A text, or a text per language (`{"en": ..., "nl": ...}`). Some kinds take a second variant (below). |
+| `description` | For the "ignored checks" list (`describeRule`): the rule in one line, without values. Optional; the kind has a fallback. |
+| `questionList` | `list`, `grouped` or a list of those: the rule only runs in rounds whose type has that `questionList`. Absent: every round with questions. |
+| the rest | Kind-specific parameters. |
+
+Messages are **English keys**: the editor translates them with its own catalogue by the English text, so the core messages are exactly the strings the editor had in code. A mod can ship a text per language instead; the evaluator picks the language asked for (regional matches its base: `nl-BE` -> `nl`), then English, then the first text.
+
+### 3.4 Rule vocabulary
+
+**Round-wide (comparative) kinds** — one finding per deviating question, population taken from the round:
+
+| `rule` | Parameters | Finding |
+|---|---|---|
+| `outlier` | `field` (a numeric question setting: `points`, `duration`), `factor` (3), `minPopulation` (3) | Questions whose value is >= median x factor or <= median / factor, among questions with a value > 0, when at least `minPopulation` have one. Params `{value, median}`. |
+| `oddOneOut` | `setting` (boolean question setting), `label`, `minPopulation` (4), `unsetIsOn` (true: null/undefined count as on, for `playTimerMusic`), `message` (the only one **enabled**), `messageDisabled` | Among **same-type siblings** (at least `minPopulation`), exactly one question differs. Param `{setting}` = label. |
+| `consistency` | `field` (narration field on the primary resource: `long`, `tips`, `explanation`, `longExplanation`) **or** `slot` (question attachment slot: `before`, `during`, ...), `label`, `minPopulation` (4), `majority` (0.75) | Questions missing what at least `majority` of the round (but not all) has. Param `{field}` or `{slot}` = label. `during` counts as filled by an image map. A gap `explanationPair` already explains is skipped. |
+| `explanationPair` | `fields` (`["longExplanation", "explanation"]`), `labels` (`{field: label}`), `minPopulation` (4), `majority` (0.75) | Among questions that fill either field, the ones using the other half than the majority. Params `{expected, actual}` = labels. |
+| `optionCountMajority` | `types` (question type tokens), `minPopulation` (4), `majority` (0.5) | Same-type siblings (types listed) with a different number of filled options than the strict majority. Params `{count, expected}`. |
+| `groupPosition` | exactly one of `setting` (a numeric question setting: `points`) or `field` (a text on the primary resource: `label`, compared trimmed and ignoring case), `minPopulation` (3, groups), `majority` (0.5) | In a grouped round, the nth question of every group is compared with the nth question of the other groups: where a strict majority of the groups (but not all) agree, the nth questions that differ. Questions without a value (setting not > 0, empty text) are left out. Params `{value, expected, position}` (position from 1). |
+| `groupSize` | `minPopulation` (3, groups), `message` (several), `messageSingle` (one question) | In a grouped round, the one group whose size differs when every other group agrees. Id `<roundUuid>:group:<groupId>:group-size`, anchored on the group's first question. Params `{count, expected}`. |
+
+**Per-question kinds** — checked on every question in scope:
+
+| `rule` | Parameters | Finding |
+|---|---|---|
+| `require` | exactly one of `field` (a text on the primary resource: `text`/`name`, `label`, `explanation`, `long`, `longExplanation`, `tips`), `setting` (a question setting that must be `true`), `check` (`correctOption`, `allOptionsFilled`, `answer`, `imageMap`: the completeness checks by name) | Severity `error` unless declared `warning`. Param `{field}`/`{setting}`. |
+| `settingEquals` | `setting`, `equals` | The question's setting (default applied) equals the value. Same as `settingWarnings`, with an explicit id. |
+| `optionCountOver` | `optionsOver`, `settingFalse` | More than `optionsOver` options while the boolean setting is off. |
+| `typeSetting` | `typeSetting: {key, equals}`, `typeSettingBelow: {key, min}` | The type setting (field default applied) equals the value while the other is unset or below `min`. |
+
+Finding ids: `<questionUuid>:<ruleId>` for every per-question finding and every comparative finding except `groupSize` (above). Ignore keys: a question stores the rule id, a round stores `group:<groupId>:group-size`, a quiz stores rule ids muted for the whole quiz.
+
+Duplicate finding ids are resolved first-declared-wins, in the order round type, scoring strategy, question type, core. That is how a strategy's `questionSettingWarnings` on a setting the round type already warns about is dropped today.
+
+### 3.5 Guidance
+
+`guidance` on a round type, question type or scoring strategy: prose for the author and the AI assistant (how the type plays, what to watch for, what a good one looks like). The evaluator exposes it through `guidanceFor(manifests, kind, token, language)`; it never produces a finding. The core guidance is the text the MCP server carried in `type-guidance.ts`, reworded to name no tool.
+
+## 4. The evaluator
+
+```ts
+evaluateFindings({ manifests, quiz | round, ignoredWarnings?, ignoredRules?, language? })
+  -> { errors: ValidationIssue[], warnings: Finding[], ignored: IgnoredFinding[] }
+```
+
+- `manifests`: the served file, format 1, as fetched.
+- `quiz` / `round`: the REST aggregate (`rounds.items[]`, `questions.items[]`, `resources[]` with `name`/`options.items[]`/`attachments`, flat settings such as `points`/`timeBasedScoring`, `mode.token`, `scoringStrategy`, `modeSettings`, `ignoredWarnings`). Defaults are applied as the editor applies them on load (`points` 1000, `duration` 30, `playTimerMusic`/`autoJudge` null, ...); groups are normalised as the editor does (a run of ungrouped questions becomes one new group).
+- `ignoredWarnings`: extra finding ids to treat as ignored (the editor's legacy per-browser list); `ignoredRules`: extra rule ids muted for the whole subject. Both come on top of what the entities carry.
+- `language`: for manifest messages that come per language.
+
+Output:
+
+- `ValidationIssue { type: 'question' | 'round' | 'language', label, detail?, roundId?, questionId?, language?, canUseDefaults?, ruleId?, messageKey?, messageParams? }` — the editor's shape. `label` is the English location (`R2.Q3`, `Q3` for a round on its own, the round's name for round issues); a consumer with its own translations rebuilds it from `roundId`/`questionId` and `messageKey`/`messageParams`.
+- `Finding { id, ruleId, roundId, questionId, message, messageKey, messageParams }` — `message` is rendered in `language`; `messageKey` + `messageParams` are the template and values, so the editor keeps translating by key.
+- `IgnoredFinding = Finding & { ignoredBy: 'question' | 'round' | 'quiz' | 'caller' }`.
+
+Errors produced: incomplete questions (`completeness`, `require`; one issue per failing rule, where the editor reported one per question, so a consumer that wants the old count dedupes by `questionId`), a round short of its mode's scoring settings or with a strategy its type does not offer (`canUseDefaults: true`), a question whose type the round type does not offer. **Translation gaps** (`type: 'language'`) are **out of scope**: they need the editor's per-language completeness bookkeeping (`incompleteLanguages`, `getTranslationGaps`) which is not in the aggregate; the editor keeps producing them and concatenates. Legacy numeric enum values in round settings (the editor's `mode-setting-names`) are counted as present, not normalised. A question type the manifests do not know gets no completeness checks (the editor's fallback descriptor has none either); an unknown round type is treated as a list round with no settings, as the editor's fallback does, and gets no round-settings or combination error.
+
+Also exported: the types, `describeRule(manifests, ruleId, language?)`, `guidanceFor`, the ignore-key helpers (`findingIgnoreTarget`, `questionWarningId`, `groupSizeWarningId`, `isIgnoredOnEntity`), `localizedText`, `interpolate`, and `compileRules(manifests)` for consumers that want to list rules.
+
+## 5. The fixture contract
+
+`<game client>/types/findings-fixtures.json`, built by `grunt fileindex` from `app/types/findings-fixtures/*.json`:
+
+```json
+{ "format": 1, "manifests": <the served manifests>, "fixtures": [
+  { "name": "...", "manifests"?: <own manifests>, "input": { "quiz": ... } | { "round": ... },
+    "ignoredWarnings"?: [], "ignoredRules"?: [], "language"?: "nl",
+    "expected": { "errors": [...], "warnings": [...], "ignored"?: [...] } } ] }
+```
+
+A fixture without its own `manifests` runs against the served ones (so the real core rules are exercised); one with `manifests` is self-contained (mod rules, edge cases). `expected.warnings` and `expected.ignored` are compared as sets by id with every field strict; `expected.errors` as a set of whole objects; `ignored` is compared only when given. The game client's tests check every fixture's shape and that the set covers every rule kind, an ignore of each scope and a mod rule; this package's tests evaluate every fixture, from the game client checkout when it is there (`QUIZWITZ_CLIENT_FIXTURES` or `../client.quizwitz.com/app/types/findings-fixtures.json`) and always from the committed copy in `fixtures/`.
+
+## 6. How a mod adds a rule
+
+In the mod's `config.json`, on the type entry:
+
+```json
+"roundTypes": [{
+  "token": "mod-betting",
+  "findings": [
+    { "id": "require:label", "rule": "require", "field": "label",
+      "message": { "en": "Give every question a theme (its label): players see it before they place their bet",
+                   "nl": "Geef elke vraag een thema (haar label): spelers zien het voor ze inzetten" } }
+  ],
+  "guidance": { "en": "...", "nl": "..." }
+}]
+```
+
+Pick a kind from the vocabulary, give the instance a stable id, write the message (per language if you like) and add a fixture in `app/types/findings-fixtures/` that shows it firing. `ManifestValidator` rejects an unknown kind, a missing id or message, or a comparative kind on a question type.
+
+## 7. Package
+
+TypeScript, ESM, no runtime dependencies, no DOM, Node 20+. `dist/` is committed and published to npm as `quizwitz-validator`, so it installs without a build step in an Angular app and a Node server alike; `npm run build` before tagging, and `test/dist.test.ts` fails when `dist/` disagrees with `src/` on the fixtures.
+
+## 8. Not in this phase
+
+Switching the editor and the MCP server to the package (each gets a checklist in the hand-off), moving the legacy fields into `findings`, round-level `require` rules, translation gaps.