@fabi-play/protocol 0.3.0 → 0.4.0

package/README.md

@@ -92,3 +92,39 @@ child's round ends (a crash too) and turns remain, it asks `break('turn')`, and
 the last round the game ends with every child's scores. `level` and `gameover` breaks never swap players; hosts
 before turns answer `turn` as an unknown reason, at once. Games keep scores by player id. See "Playing with the TV remote" in the
 [`@fabi-play/sdk` README](https://www.npmjs.com/package/@fabi-play/sdk).
+
+### The arrows as a cursor
+
+A manifest's optional `"remoteCursor": true` (valid only with `remote` in `inputs`) asks the screen, in remote mode,
+to turn the arrows into a free cursor instead of gestures. The screen keeps which arrows are held and runs
+`KeyCursor` every animation frame (`update(held, dtMs)` → `{ x, y }`, 0–1 of the game's frame: slow at first, faster
+the longer an arrow is held, opposite arrows cancel, diagonals no faster, clamped to the edges, a long frame counted
+as 100 ms; `reset()` at a turn), draws the cursor above the game, and sends `{ type: 'cursor', x, y, down }`
+(`parseCursor`) when it moves and when OK goes down or up. init says `pointer: true`, as for a Magic Remote. The
+SDK turns the messages into pointer and mouse events in the game's page, so the game sees a mouse either way. Back
+stays the platform's pause; OK is the cursor's button, not a jump.
+
+## Pictures to the phone
+
+A game can save a picture on the family's phone. The path, and the messages:
+
+1. The screen tells the game whether it can: `init.pictures` and, when it changes, `{ type: 'capabilities',
+   pictures }`. It is true while a phone whose `hello` said `pictures: true` is linked (the app; the web phone
+   doesn't say it).
+2. The game sends `{ type: 'picture', data, title?, id? }` (GameMessage): `data` a base64 data URL,
+   `data:image/jpeg` or `data:image/png`, at most `MAX_PICTURE_BYTES` (3 MiB of image), whose bytes really start as
+   that type (`checkPicture`; `parseGameMessage` drops anything else); `title` up to `MAX_PICTURE_TITLE` (60)
+   characters, cleaned by `parsePictureTitle`.
+3. The screen sends it to the phone over their reliable, ordered data channel, in pieces, because data channels
+   don't carry multi-megabyte messages (16 KiB is what every browser takes): `chunkPicture(dataUrl, id, title)`
+   gives `{ type: 'picture-start', id, mime, size, title? }` (`size` in bytes of the image), then
+   `{ type: 'picture-chunk', id, seq, data }` (`PICTURE_CHUNK_CHARS`, 16 000 base64 characters, each but the last,
+   `seq` from 0; each JSON message stays under 16 KiB), then `{ type: 'picture-end', id }` (ScreenMessage).
+4. The phone feeds them to a `PictureAssembler` (chunks in any order; a missing chunk, a picture over the limit, a
+   chunk past its size, bytes that are not the type the start said, all fail with a reason) and answers
+   `{ type: 'picture-saved', id }` or `{ type: 'picture-failed', id }` (ControllerMessage).
+5. The screen answers the game `{ type: 'picture-result', id, result }` (`parsePictureResult`): `'sent'`,
+   `'unavailable'`, `'too-big'` or `'error'`.
+
+Privacy: the picture travels only TV → phone over the direct link (WebRTC, encrypted), never through or to a server,
+and no copy is kept anywhere else. Games can send only a JPEG or PNG image this way.

package/dist/index.d.ts

@@ -2,13 +2,195 @@
 // Generated by scripts/build-lib.mjs (fabiplay-kit) from the sources: do not edit.
 
 /**
- * The messages of Fabi Play, between three parties:
+ * A picture from the game to the family's phone (a drawing, the photo of a wall), saved in the parent's gallery.
  *
- *   phone (controller) ──link──▶ screen (host) ──postMessage──▶ game (sandboxed frame)
+ *   game ──postMessage { type: 'picture', data }──▶ screen ──data channel, in chunks──▶ phone (gallery)
  *
- * The camera stays on the phone: it sends only body landmarks, never images. The game gets the landmarks and the
- * movements recognised by the host, and can only answer with what is listed in GameMessage.
+ * Privacy: the picture goes only from the TV to the phone, over the direct link they already share (WebRTC, encrypted
+ * end to end), never through or to a server, and no copy is kept anywhere else. A game can send only a JPEG or PNG
+ * image this way (a base64 data URL whose bytes really start as one), at most MAX_PICTURE_BYTES: nothing else, no
+ * text, no other file. The phone side decides where it goes (the gallery, with the parent's permission).
+ *
+ * WebRTC data channels do not carry multi-megabyte messages reliably (16 KiB is the size every browser takes), so
+ * the screen sends a picture as picture-start, picture-chunk × n (PICTURE_CHUNK_CHARS of base64 each, the last
+ * shorter), picture-end; chunkPicture makes them, PictureAssembler puts them back together on the phone, which
+ * answers picture-saved or picture-failed.
+ */
+/** The image types a picture may be. No WebP, GIF or SVG: JPEG for paintings and photos, PNG for flat art. */
+export type PictureMime = "image/jpeg" | "image/png";
+export declare const PICTURE_MIMES: readonly PictureMime[];
+/** The largest picture, in bytes of the image (3 MiB; its base64 is a third bigger). */
+export declare const MAX_PICTURE_BYTES: number;
+/** The SDK scales a canvas down to this many pixels on its longer side before encoding it. */
+export declare const MAX_PICTURE_SIDE = 1920;
+/** The JPEG quality the SDK encodes a canvas with first (lower ones only if the picture would be too big). */
+export declare const PICTURE_JPEG_QUALITY = 0.85;
+/** The longest title of a picture (its file name on the phone), in characters. */
+export declare const MAX_PICTURE_TITLE = 60;
+/**
+ * Base64 characters per picture-chunk: a multiple of 4 (each chunk decodes on its own), small enough that the whole
+ * JSON message stays under 16 KiB.
+ */
+export declare const PICTURE_CHUNK_CHARS = 16000;
+/** What became of a picture the game asked to save (HostMessage picture-result, Play.savePicture). */
+export type PictureResult = "sent" | "unavailable" | "too-big" | "error";
+export declare const PICTURE_RESULTS: readonly PictureResult[];
+/** A picture, checked: its type, its base64 (no data: prefix) and its size in bytes. */
+export interface PictureData {
+	mime: PictureMime;
+	base64: string;
+	size: number;
+}
+/**
+ * A picture's data URL checked: 'ok' with its parts, 'too-big' over MAX_PICTURE_BYTES, 'invalid' for anything that
+ * is not a base64 JPEG or PNG data URL whose bytes start as that type.
+ */
+export declare function checkPicture(v: unknown): {
+	ok: PictureData;
+} | {
+	error: "too-big" | "invalid";
+};
+/** A picture's title, or undefined: a string, trimmed, without control characters, cut to MAX_PICTURE_TITLE. */
+export declare function parsePictureTitle(v: unknown): string | undefined;
+/** The three messages of a picture from the screen to the phone (members of ScreenMessage). */
+export type PictureStart = {
+	type: "picture-start";
+	id: number;
+	mime: PictureMime;
+	size: number;
+	title?: string;
+};
+export type PictureChunk = {
+	type: "picture-chunk";
+	id: number;
+	seq: number;
+	data: string;
+};
+export type PictureEnd = {
+	type: "picture-end";
+	id: number;
+};
+export type PictureMessage = PictureStart | PictureChunk | PictureEnd;
+/** A picture's id on the link: an integer from 0 to 2³¹ − 1, chosen by the screen. */
+export declare const isPictureId: (v: unknown) => v is number;
+/** How many chunks a picture of this many bytes travels in. */
+export declare const pictureChunks: (size: number) => number;
+/**
+ * A picture as the messages the screen sends the phone, in order: picture-start, its chunks (seq from 0), picture-end.
+ * Throws RangeError when dataUrl is not a picture checkPicture takes, or id not a picture id. The screen sends them on
+ * the reliable, ordered channel, waiting while its bufferedAmount is high.
  */
+export declare function chunkPicture(dataUrl: string, id: number, title?: string): PictureMessage[];
+/** The picture-* screen messages checked (see parseScreenMessage), or null. */
+export declare function parsePictureMessage(data: Record<string, unknown>): PictureMessage | null;
+/** A picture put back together on the phone: dataUrl for an <Image>, base64 to write the file. */
+export interface AssembledPicture {
+	id: number;
+	mime: PictureMime;
+	size: number;
+	title?: string;
+	base64: string;
+	dataUrl: string;
+}
+/**
+ * Why a picture could not be put back together: 'too-big' (its start says over MAX_PICTURE_BYTES, or chunks run
+ * past its size), 'missing' (a chunk never came, or the picture ended short), 'invalid' (a chunk out of place, a
+ * repeated chunk that differs, bytes that are not the type its start said), 'unknown' (a chunk or end with no start),
+ * 'replaced' (dropped for newer pictures: at most PICTURE_ASSEMBLER_MAX at once).
+ */
+export type PictureFailure = "too-big" | "missing" | "invalid" | "unknown" | "replaced";
+/** What PictureAssembler.push says: nothing yet (null), a picture, or a failed one (answer picture-failed with its id). */
+export type AssemblerResult = null | {
+	ok: AssembledPicture;
+} | {
+	failed: PictureFailure;
+	id: number;
+};
+/** Pictures put together at once; a start beyond this drops the oldest ('replaced'). */
+export declare const PICTURE_ASSEMBLER_MAX = 2;
+/**
+ * The phone's side: feed it every picture-* message (parsed with parseScreenMessage), in any order; it gives back the
+ * picture at its picture-end, or why it failed. Chunks may come out of order or twice (the same data); a picture
+ * whose end comes before all its chunks fails 'missing'. Pure, no DOM: decoding and saving are the app's.
+ *
+ * @example
+ * const pictures = new PictureAssembler()
+ * channel.onmessage = (e) => {
+ *   const m = parseScreenMessage(JSON.parse(e.data))
+ *   if (!m || !m.type.startsWith('picture-')) return
+ *   const r = pictures.push(m as PictureMessage)
+ *   if (r && 'ok' in r) saveToGallery(r.ok).then(() => send({ type: 'picture-saved', id: r.ok.id }), () => send({ type: 'picture-failed', id: r.ok.id }))
+ *   else if (r) send({ type: 'picture-failed', id: r.id })
+ * }
+ */
+export declare class PictureAssembler {
+	private readonly pending;
+	/** Pictures being put together now. */
+	get size(): number;
+	push(m: PictureMessage): AssemblerResult;
+	/** Forgets every picture being put together (the link closed). */
+	clear(): void;
+	private fail;
+}
+/**
+ * The TV remote's arrows as a free cursor, for games whose manifest says "remoteCursor": true (see
+ * GameManifest.remoteCursor). The screen keeps which arrows are held and calls update every animation frame; the
+ * cursor glides, slowly at first for precision, then faster the longer an arrow is held, and stops at the edges.
+ * Pure: no DOM, no clock of its own (the screen passes the time between frames), so it is tested on its own.
+ *
+ *   const cursor = new KeyCursor({ aspect: innerWidth / innerHeight })
+ *   // every frame, with the arrows held now (the screen's KeyGate knows them):
+ *   const { x, y } = cursor.update(held, dt)
+ *   host.cursor(x, y, okHeld)
+ */
+/** The arrows, as the screen names its keys. */
+export type CursorKey = "UP" | "DOWN" | "LEFT" | "RIGHT";
+/** The arrows held now: the key names (other names are ignored), or one flag per arrow. */
+export type CursorKeys = Iterable<string> | Partial<Record<CursorKey, boolean>>;
+/** Options of KeyCursor; the defaults are what the TV uses. */
+export interface KeyCursorOptions {
+	/** The screen's width / height, so the cursor moves as fast across as down (default 16 / 9). */
+	aspect?: number;
+	/** Speed when an arrow is first held, in screen heights per second (default 0.12: a tap moves a few pixels). */
+	slow?: number;
+	/** Top speed, in screen heights per second (default 1.1: a 1080p screen's width in about 1.5 s). */
+	fast?: number;
+	/** How long an arrow stays at the slow speed before it speeds up, ms (default 180). */
+	holdMs?: number;
+	/** How long the speed then takes to reach fast, ms (default 1000; it eases in). */
+	rampMs?: number;
+	/** Where it starts, 0–1 (default the centre). */
+	x?: number;
+	y?: number;
+}
+export declare class KeyCursor {
+	/** 0 left, 1 right: fractions of the game's frame. */
+	x: number;
+	/** 0 top, 1 bottom. */
+	y: number;
+	private readonly aspect;
+	private readonly slow;
+	private readonly fast;
+	private readonly holdMs;
+	private readonly rampMs;
+	/** How long the current direction has been held, ms; the direction it is. */
+	private held;
+	private dir;
+	constructor(o?: KeyCursorOptions);
+	/** The speed after holding a direction this long, in screen heights per second. */
+	speed(heldMs: number): number;
+	/**
+	 * One frame: keys, the arrows held now; dt, ms since the last update (over 100 ms counts as 100). Opposite arrows
+	 * cancel; a diagonal is as fast as a straight move. The speed starts slow again whenever the direction changes or
+	 * no arrow is held. Returns where the cursor is now, 0–1, clamped to the edges.
+	 */
+	update(keys: CursorKeys, dt: number): {
+		x: number;
+		y: number;
+	};
+	/** Puts the cursor at x, y (clamped), slow again: a new turn, a game that starts. Default the centre. */
+	reset(x?: number, y?: number): void;
+}
 export declare const PROTOCOL_VERSION = 1;
 /** MediaPipe Pose landmarks: 33 points, the ones the games use by name. */
 export declare const LANDMARKS = 33;
@@ -96,6 +278,8 @@ export type ControllerMessage =
  * aspect: width / height of the camera image, so the screen draws the bodies undistorted. fov: the camera's field
  * of view for that image (degrees, h across its width, v down its height), when the phone knows it (the app reads
  * it from the camera); the screen estimates each child's distance with it, or with DEFAULT_FOV when absent.
+ * pictures: true when this phone can receive pictures from games and save them (the app, with its gallery); absent
+ * on phones that cannot (the web phone, apps before pictures): the screen then tells games it can't save.
  */
 {
 	type: "hello";
@@ -104,6 +288,7 @@ export type ControllerMessage =
 	device: string;
 	aspect: number;
 	fov?: Fov;
+	pictures?: boolean;
 }
 /** A camera frame read on the phone: t in ms on the phone's clock, one body per player seen. */
  | {
@@ -156,6 +341,15 @@ export type ControllerMessage =
 	type: "video";
 	state: VideoState;
 	reason?: VideoBlock;
+}
+/**
+ * The answer to a picture the screen sent (picture-start … picture-end, same id): saved (in the gallery, or kept to
+ * save once the parent allows it), or failed (it could not be put together, or saving failed for good). When saving
+ * fails the phone tells the parent itself; the children are never told.
+ */
+ | {
+	type: "picture-saved" | "picture-failed";
+	id: number;
 };
 export type ScreenMessage = 
 /**
@@ -211,7 +405,14 @@ export type ScreenMessage =
  * confirmed band (absent: not known, e.g. a phone answered elsewhere and the TV forgot), zone the distances (metres)
  * the children should play at, for the parents' area.
  */
- | ScreenSizeNews;
+ | ScreenSizeNews
+/**
+ * A picture from the game for the phone to save (see chunkPicture, PictureAssembler): picture-start (its type, its
+ * size in bytes, a title for the file), then picture-chunk × n (base64, PICTURE_CHUNK_CHARS each but the last, seq
+ * from 0), then picture-end; the phone answers picture-saved or picture-failed. Only over the direct link to the
+ * phone, never through a server; only to a phone whose hello said pictures.
+ */
+ | PictureMessage;
 /**
  * Mirror mode on or off, for every app at build time: the one switch for the phone's camera video on the TV. On:
  * the parent's switch on the phone decides, within the rules of docs/test.md section 13. Off: the phones (web page and app) never send video and never open a camera for it, their video switch and
@@ -439,7 +640,13 @@ export type HostMessage =
  */
 /**
  * pointer: only with input 'remote', true when the TV's remote also points (LG's Magic Remote): it reaches the game
- * as ordinary pointer and mouse events (hover, click with OK). Absent = false.
+ * as ordinary pointer and mouse events (hover, click with OK); or, for a game whose manifest says "remoteCursor":
+ * true, when the screen moves a cursor with the arrows (cursor messages, which the SDK turns into the same pointer
+ * and mouse events). Absent = false.
+ */
+/**
+ * pictures: a phone that can save pictures is linked now (its hello said pictures), so the game may send picture
+ * messages; it changes with a capabilities message. Absent = false.
  */
 {
 	type: "init";
@@ -456,6 +663,7 @@ export type HostMessage =
 	input?: GameInput;
 	turns?: number[];
 	pointer?: boolean;
+	pictures?: boolean;
 }
 /** The backdrop or the view changed mid-game (video switched on or off, the camera image's fit or framing changed). */
  | {
@@ -509,7 +717,45 @@ export type HostMessage =
  | {
 	type: "turn";
 	players: number[];
+}
+/**
+ * What the screen can do for the game changed mid-game: pictures, whether a phone that can save pictures is linked
+ * now (one joined, or the last one left). init carries the first value.
+ */
+ | {
+	type: "capabilities";
+	pictures: boolean;
+}
+/**
+ * The answer to a picture message, with its id (absent if the picture had none): 'sent', the phone has it (saved,
+ * or saved once the parent allows it); 'unavailable', no phone that saves pictures is linked; 'too-big', over
+ * MAX_PICTURE_BYTES; 'error', anything else (the link dropped, the phone failed, no answer in time).
+ */
+ | {
+	type: "picture-result";
+	id?: number;
+	result: PictureResult;
+}
+/**
+ * Remote mode with "remoteCursor": true: the cursor the arrows move (see KeyCursor), x and y in fractions of the
+ * game's frame (0–1), down while OK is held. Sent when it moves and when OK goes down or up; the SDK turns it into
+ * pointer and mouse events in the game's page (pointermove, pointerdown, pointerup, click, …), as a Magic Remote's.
+ * The screen draws the cursor itself, above the game.
+ */
+ | {
+	type: "cursor";
+	x: number;
+	y: number;
+	down: boolean;
 };
+/** A cursor message's position and button, or undefined: x, y finite (clamped to 0–1), down a boolean. */
+export declare function parseCursor(v: unknown): Extract<HostMessage, {
+	type: "cursor";
+}> | undefined;
+/** A picture-result message, or undefined: result one of PICTURE_RESULTS, id (if any) an integer. */
+export declare function parsePictureResult(v: unknown): Extract<HostMessage, {
+	type: "picture-result";
+}> | undefined;
 /** The players of a turn message, or undefined if they are not valid: exactly one integer id from 1 to MAX_PLAYERS. A copy. */
 export declare function parseTurnPlayers(v: unknown): number[] | undefined;
 /** init's turns, or undefined if they are not valid: 1 to MAX_PLAYERS distinct integer ids from 1 to MAX_PLAYERS, in order. A copy. */
@@ -555,7 +801,20 @@ export type GameMessage =
  | ({
 	type: "stats";
 	renderFps: number;
-} & GameStats);
+} & GameStats)
+/**
+ * Save a picture on the family's phone (a drawing, the photo of a wall). data: a base64 data URL, data:image/jpeg
+ * or data:image/png, at most MAX_PICTURE_BYTES, whose bytes really are that type (checkPicture); title: a name for
+ * it (MAX_PICTURE_TITLE characters); id: the game's own, echoed in picture-result. Anything else is dropped: a game
+ * can send only an image this way. The screen sends it to the phone over their direct link only, never to a
+ * server, and answers picture-result.
+ */
+ | {
+	type: "picture";
+	data: string;
+	title?: string;
+	id?: number;
+};
 /** The movements a game is played with, as its manifest lists them. */
 export type Movement = "hands" | "jump" | "squat" | "clap" | "freeze" | "lanes" | "shapes";
 /**
@@ -625,6 +884,14 @@ export interface GameManifest {
 	 * a game drawn on exact spots of the live video).
 	 */
 	framing?: GameFraming;
+	/**
+	 * Optional, only with 'remote' in inputs: true and, in remote mode, the arrows move a free cursor instead of making
+	 * gestures (they speed up when held, see KeyCursor) and OK presses and releases it: the game gets it as pointer and
+	 * mouse events, the same as a Magic Remote's (init.pointer true), so a game written for the pointer works with
+	 * both. Back is still the platform's pause. For games picked or drawn with a pointer (a painting game). Absent or
+	 * false: arrows and OK make gestures.
+	 */
+	remoteCursor?: boolean;
 }
 /** How the screen frames the camera image for a game: see GameManifest.framing. */
 export type GameFraming = "auto" | "fixed";

package/dist/index.js

@@ -1,3 +1,128 @@
+var __defProp = Object.defineProperty;
+var __defNormalProp = (obj, key, value) => key in obj ? __defProp(obj, key, { enumerable: true, configurable: true, writable: true, value }) : obj[key] = value;
+var __publicField = (obj, key, value) => __defNormalProp(obj, typeof key !== "symbol" ? key + "" : key, value);
+
+// src/picture.ts
+var PICTURE_MIMES = ["image/jpeg", "image/png"];
+var MAX_PICTURE_BYTES = 3 * 1024 * 1024;
+var MAX_PICTURE_SIDE = 1920;
+var PICTURE_JPEG_QUALITY = 0.85;
+var MAX_PICTURE_TITLE = 60;
+var PICTURE_CHUNK_CHARS = 16e3;
+var PICTURE_RESULTS = ["sent", "unavailable", "too-big", "error"];
+var BASE64 = /^[A-Za-z0-9+/]*={0,2}$/;
+var MAGIC = { "image/jpeg": "/9j/", "image/png": "iVBORw0KGg" };
+var PREFIX = /^data:(image\/(?:jpeg|png));base64,/;
+var decodedSize = (b64) => b64.length / 4 * 3 - (b64.endsWith("==") ? 2 : b64.endsWith("=") ? 1 : 0);
+var MAX_BASE64 = Math.ceil(MAX_PICTURE_BYTES / 3) * 4;
+var isImage = (mime, b64) => b64.length % 4 === 0 && b64.length > 0 && b64.startsWith(MAGIC[mime]) && BASE64.test(b64);
+function checkPicture(v) {
+  if (typeof v !== "string") return { error: "invalid" };
+  const m = PREFIX.exec(v);
+  if (!m) return { error: "invalid" };
+  const mime = m[1];
+  const base64 = v.slice(m[0].length);
+  if (base64.length > MAX_BASE64) return { error: "too-big" };
+  if (!isImage(mime, base64)) return { error: "invalid" };
+  const size = decodedSize(base64);
+  return size > MAX_PICTURE_BYTES ? { error: "too-big" } : { ok: { mime, base64, size } };
+}
+function parsePictureTitle(v) {
+  if (typeof v !== "string") return void 0;
+  const t = Array.from(v.replace(/[\p{Cc}\p{Cf}]/gu, "").trim()).slice(0, MAX_PICTURE_TITLE).join("").trim();
+  return t || void 0;
+}
+var isPictureId = (v) => Number.isInteger(v) && v >= 0 && v <= 2147483647;
+var pictureChunks = (size) => Math.ceil(Math.ceil(size / 3) * 4 / PICTURE_CHUNK_CHARS);
+function chunkPicture(dataUrl, id, title) {
+  if (!isPictureId(id)) throw new RangeError("picture: id is not an integer from 0 to 2^31 - 1");
+  const checked = checkPicture(dataUrl);
+  if ("error" in checked) throw new RangeError(`picture: ${checked.error === "too-big" ? `over ${MAX_PICTURE_BYTES} bytes` : "not a base64 JPEG or PNG data URL"}`);
+  const { mime, base64, size } = checked.ok;
+  const name = parsePictureTitle(title);
+  const out = [{ type: "picture-start", id, mime, size, ...name ? { title: name } : {} }];
+  for (let at = 0, seq = 0; at < base64.length; at += PICTURE_CHUNK_CHARS, seq++) out.push({ type: "picture-chunk", id, seq, data: base64.slice(at, at + PICTURE_CHUNK_CHARS) });
+  out.push({ type: "picture-end", id });
+  return out;
+}
+function parsePictureMessage(data) {
+  if (!isPictureId(data.id)) return null;
+  const id = data.id;
+  switch (data.type) {
+    case "picture-start": {
+      const { mime, size } = data;
+      if (!PICTURE_MIMES.includes(mime) || !Number.isInteger(size) || size < 1) return null;
+      const title = parsePictureTitle(data.title);
+      return { type: "picture-start", id, mime, size, ...title ? { title } : {} };
+    }
+    case "picture-chunk": {
+      const { seq, data: d } = data;
+      return Number.isInteger(seq) && seq >= 0 && typeof d === "string" && d.length > 0 && d.length <= PICTURE_CHUNK_CHARS && BASE64.test(d) ? { type: "picture-chunk", id, seq, data: d } : null;
+    }
+    case "picture-end":
+      return { type: "picture-end", id };
+    default:
+      return null;
+  }
+}
+var PICTURE_ASSEMBLER_MAX = 2;
+var PictureAssembler = class {
+  constructor() {
+    __publicField(this, "pending", /* @__PURE__ */ new Map());
+  }
+  /** Pictures being put together now. */
+  get size() {
+    return this.pending.size;
+  }
+  push(m) {
+    switch (m.type) {
+      case "picture-start": {
+        this.pending.delete(m.id);
+        if (m.size > MAX_PICTURE_BYTES) return { failed: "too-big", id: m.id };
+        let dropped = null;
+        if (this.pending.size >= PICTURE_ASSEMBLER_MAX) {
+          const oldest = this.pending.keys().next().value;
+          this.pending.delete(oldest);
+          dropped = { failed: "replaced", id: oldest };
+        }
+        this.pending.set(m.id, { start: m, chunks: new Array(pictureChunks(m.size)), received: 0 });
+        return dropped;
+      }
+      case "picture-chunk": {
+        const p = this.pending.get(m.id);
+        if (!p) return { failed: "unknown", id: m.id };
+        const last = p.chunks.length - 1;
+        const expected = m.seq < last ? PICTURE_CHUNK_CHARS : Math.ceil(p.start.size / 3) * 4 - last * PICTURE_CHUNK_CHARS;
+        if (m.seq > last) return this.fail(m.id, "too-big");
+        if (m.data.length !== expected) return this.fail(m.id, m.data.length > expected ? "too-big" : "invalid");
+        const had = p.chunks[m.seq];
+        if (had !== void 0) return had === m.data ? null : this.fail(m.id, "invalid");
+        p.chunks[m.seq] = m.data;
+        p.received++;
+        return null;
+      }
+      case "picture-end": {
+        const p = this.pending.get(m.id);
+        if (!p) return { failed: "unknown", id: m.id };
+        this.pending.delete(m.id);
+        if (p.received < p.chunks.length) return { failed: "missing", id: m.id };
+        const base64 = p.chunks.join("");
+        const { id, mime, size, title } = p.start;
+        if (!isImage(mime, base64) || decodedSize(base64) !== size) return { failed: "invalid", id };
+        return { ok: { id, mime, size, ...title ? { title } : {}, base64, dataUrl: `data:${mime};base64,${base64}` } };
+      }
+    }
+  }
+  /** Forgets every picture being put together (the link closed). */
+  clear() {
+    this.pending.clear();
+  }
+  fail(id, why) {
+    this.pending.delete(id);
+    return { failed: why, id };
+  }
+};
+
 // src/sha256.ts
 function utf8(s) {
   const out = [];
@@ -133,6 +258,76 @@ function sha256(bytes) {
   return out;
 }
 
+// src/key-cursor.ts
+var MAX_DT_MS = 100;
+var clamp01 = (v) => v <= 0 ? 0 : v >= 1 ? 1 : v;
+var finite = (v, d) => typeof v === "number" && Number.isFinite(v) && v > 0 ? v : d;
+var KeyCursor = class {
+  constructor(o = {}) {
+    /** 0 left, 1 right: fractions of the game's frame. */
+    __publicField(this, "x");
+    /** 0 top, 1 bottom. */
+    __publicField(this, "y");
+    __publicField(this, "aspect");
+    __publicField(this, "slow");
+    __publicField(this, "fast");
+    __publicField(this, "holdMs");
+    __publicField(this, "rampMs");
+    /** How long the current direction has been held, ms; the direction it is. */
+    __publicField(this, "held", 0);
+    __publicField(this, "dir", "");
+    this.aspect = finite(o.aspect, 16 / 9);
+    this.slow = finite(o.slow, 0.12);
+    this.fast = Math.max(finite(o.fast, 1.1), this.slow);
+    this.holdMs = typeof o.holdMs === "number" && o.holdMs >= 0 ? o.holdMs : 180;
+    this.rampMs = finite(o.rampMs, 1e3);
+    this.x = clamp01(o.x ?? 0.5);
+    this.y = clamp01(o.y ?? 0.5);
+  }
+  /** The speed after holding a direction this long, in screen heights per second. */
+  speed(heldMs) {
+    const t = clamp01((heldMs - this.holdMs) / this.rampMs);
+    return this.slow + (this.fast - this.slow) * t * t;
+  }
+  /**
+   * One frame: keys, the arrows held now; dt, ms since the last update (over 100 ms counts as 100). Opposite arrows
+   * cancel; a diagonal is as fast as a straight move. The speed starts slow again whenever the direction changes or
+   * no arrow is held. Returns where the cursor is now, 0–1, clamped to the edges.
+   */
+  update(keys, dt) {
+    const k = held(keys);
+    const dx = (k.RIGHT ? 1 : 0) - (k.LEFT ? 1 : 0);
+    const dy = (k.DOWN ? 1 : 0) - (k.UP ? 1 : 0);
+    const dir = `${dx},${dy}`;
+    if (dir !== this.dir) {
+      this.dir = dir;
+      this.held = 0;
+    }
+    const ms = Number.isFinite(dt) && dt > 0 ? Math.min(dt, MAX_DT_MS) : 0;
+    if ((dx || dy) && ms) {
+      const v = (this.speed(this.held) + this.speed(this.held + ms)) / 2;
+      this.held += ms;
+      const step = v * ms / 1e3 / (dx && dy ? Math.SQRT2 : 1);
+      this.x = clamp01(this.x + dx * step / this.aspect);
+      this.y = clamp01(this.y + dy * step);
+    }
+    return { x: this.x, y: this.y };
+  }
+  /** Puts the cursor at x, y (clamped), slow again: a new turn, a game that starts. Default the centre. */
+  reset(x = 0.5, y = 0.5) {
+    this.x = clamp01(Number.isFinite(x) ? x : 0.5);
+    this.y = clamp01(Number.isFinite(y) ? y : 0.5);
+    this.held = 0;
+    this.dir = "";
+  }
+};
+function held(keys) {
+  if (typeof keys[Symbol.iterator] !== "function") return keys;
+  const out = {};
+  for (const name of keys) if (name === "UP" || name === "DOWN" || name === "LEFT" || name === "RIGHT") out[name] = true;
+  return out;
+}
+
 // src/index.ts
 var PROTOCOL_VERSION = 1;
 var LANDMARKS = 33;
@@ -227,6 +422,15 @@ function parseGameStats(v) {
   if (DEVICE_TIERS.includes(v.tier)) out.tier = v.tier;
   return out;
 }
+function parseCursor(v) {
+  if (!isObject(v) || !isNumber(v.x) || !isNumber(v.y) || typeof v.down !== "boolean") return void 0;
+  const c = (n) => Math.min(Math.max(n, 0), 1);
+  return { type: "cursor", x: c(v.x), y: c(v.y), down: v.down };
+}
+function parsePictureResult(v) {
+  if (!isObject(v) || !PICTURE_RESULTS.includes(v.result)) return void 0;
+  return { type: "picture-result", ...Number.isInteger(v.id) ? { id: v.id } : {}, result: v.result };
+}
 var isPlayerId = (id) => Number.isInteger(id) && id >= 1 && id <= MAX_PLAYERS;
 function parseTurnPlayers(v) {
   return Array.isArray(v) && v.length === 1 && isPlayerId(v[0]) ? [v[0]] : void 0;
@@ -268,6 +472,8 @@ function parseManifest(data) {
     errors.push(`inputs: optional, from ${GAME_INPUTS.join(", ")}, without repeats, always with body`);
   if (m.turns !== void 0 && !(typeof m.turns === "boolean" && (!m.turns || Array.isArray(m.inputs) && m.inputs.includes("remote"))))
     errors.push('turns: optional, true or false; true only with "remote" in inputs');
+  if (m.remoteCursor !== void 0 && !(typeof m.remoteCursor === "boolean" && (!m.remoteCursor || Array.isArray(m.inputs) && m.inputs.includes("remote"))))
+    errors.push('remoteCursor: optional, true or false; true only with "remote" in inputs');
   if (m.framing !== void 0 && !GAME_FRAMINGS.includes(m.framing)) errors.push(`framing: optional, ${GAME_FRAMINGS.join(" or ")}`);
   if (m.intro !== void 0) errors.push(...introErrors(m.intro));
   return errors.length ? { errors } : { manifest: m };
@@ -363,6 +569,11 @@ function parseGameMessage(data) {
     }
     case "stats":
       return isNumber(data.renderFps) && data.renderFps >= 0 && data.renderFps < 1e3 ? { type: "stats", renderFps: data.renderFps, ...parseGameStats(data) } : null;
+    case "picture": {
+      if (!("ok" in checkPicture(data.data))) return null;
+      const title = parsePictureTitle(data.title);
+      return { type: "picture", data: data.data, ...title ? { title } : {}, ...Number.isInteger(data.id) ? { id: data.id } : {} };
+    }
     default:
       return null;
   }
@@ -382,7 +593,8 @@ function parseControllerMessage(data) {
           camera: data.camera,
           device: data.device.slice(0, 80),
           aspect: isNumber(data.aspect) ? Math.min(Math.max(data.aspect, 0.3), 3) : 0.75,
-          ...fov ? { fov } : {}
+          ...fov ? { fov } : {},
+          ...data.pictures === true ? { pictures: true } : {}
         };
       }
     case "pose":
@@ -406,6 +618,9 @@ function parseControllerMessage(data) {
       return INTRO_ACTIONS.includes(data.action) ? { type: "intro", action: data.action } : null;
     case "video":
       return VIDEO_STATES.includes(data.state) ? { type: "video", state: data.state, ...VIDEO_BLOCKS.includes(data.reason) ? { reason: data.reason } : {} } : null;
+    case "picture-saved":
+    case "picture-failed":
+      return Number.isInteger(data.id) && data.id >= 0 ? { type: data.type, id: data.id } : null;
     default:
       return null;
   }
@@ -455,6 +670,10 @@ function parseScreenMessage(data) {
         ...zone ? { zone } : {}
       };
     }
+    case "picture-start":
+    case "picture-chunk":
+    case "picture-end":
+      return parsePictureMessage(data);
     default:
       return null;
   }
@@ -571,8 +790,12 @@ export {
   INTRO_MAX_CHARS,
   INTRO_MAX_STEPS,
   INTRO_MOVES,
+  KeyCursor,
   LANDMARKS,
   MAX_AVATAR_LENGTH,
+  MAX_PICTURE_BYTES,
+  MAX_PICTURE_SIDE,
+  MAX_PICTURE_TITLE,
   MAX_PLAYERS,
   MIRROR_ENABLED,
   MOVEMENTS,
@@ -580,17 +803,25 @@ export {
   P,
   PAUSE_REASONS,
   PHASES,
+  PICTURE_ASSEMBLER_MAX,
+  PICTURE_CHUNK_CHARS,
+  PICTURE_JPEG_QUALITY,
+  PICTURE_MIMES,
+  PICTURE_RESULTS,
   PLAYER_COLORS,
   POSE_CODEC_VERSION,
   POSE_CODEC_VERSIONS,
   POSE_MAX_BYTES,
   PROTOCOL_VERSION,
+  PictureAssembler,
   SCREEN_SIZES,
   SCREEN_SIZE_KEY,
   TOO_CLOSE_KEY,
   VIDEO_BLOCKS,
   VIDEO_STATES,
   acceptsBinaryPose,
+  checkPicture,
+  chunkPicture,
   decodePose,
   defaultAvatarFor,
   encodePose,
@@ -602,22 +833,28 @@ export {
   isAvatarName,
   isBody,
   isGameId,
+  isPictureId,
   isScreenSizeId,
   legacyStorageKey,
   migrateStorageKey,
   migrateStorageKeyAsync,
   parseControllerMessage,
+  parseCursor,
   parseDeviceHint,
   parseFov,
   parseGameMessage,
   parseGameStats,
   parseIntro,
   parseManifest,
+  parsePictureMessage,
+  parsePictureResult,
+  parsePictureTitle,
   parseScreenMessage,
   parseTurnOrder,
   parseTurnPlayers,
   parseView,
   parseZone,
+  pictureChunks,
   poseBytes,
   screenSize,
   screenSizesOf,

package/fabi-play.schema.json

@@ -5,9 +5,19 @@
   "type": "object",
   "required": ["id", "version", "title", "description", "ageRanges", "movements", "players", "minutes", "entry", "sdk"],
   "additionalProperties": false,
-  "anyOf": [
-    { "not": { "required": ["turns"], "properties": { "turns": { "const": true } } } },
-    { "required": ["inputs"], "properties": { "inputs": { "contains": { "const": "remote" } } } }
+  "allOf": [
+    {
+      "anyOf": [
+        { "not": { "required": ["turns"], "properties": { "turns": { "const": true } } } },
+        { "required": ["inputs"], "properties": { "inputs": { "contains": { "const": "remote" } } } }
+      ]
+    },
+    {
+      "anyOf": [
+        { "not": { "required": ["remoteCursor"], "properties": { "remoteCursor": { "const": true } } } },
+        { "required": ["inputs"], "properties": { "inputs": { "contains": { "const": "remote" } } } }
+      ]
+    }
   ],
   "properties": {
     "$schema": {
@@ -95,6 +105,10 @@
       "type": "boolean",
       "description": "Optional, true only with \"remote\" in inputs: the game follows the turns contract, so several children may play with the remote, a round each. When a child's round ends (a crash too) and turns remain, the game asks break('turn') (the screen answers break-start, turn, break-done); after the last round it ends with every child's scores. Absent or false: remote mode is one child only."
     },
+    "remoteCursor": {
+      "type": "boolean",
+      "description": "Optional, true only with \"remote\" in inputs: in remote mode the arrows move a free cursor (it speeds up while held) and OK presses it, instead of making gestures. The game gets it as pointer and mouse events, the same as a Magic Remote's (init.pointer true), so a game written for the pointer works with both. Back is still Fabi Play's pause. Absent or false: arrows and OK make gestures."
+    },
     "framing": {
       "enum": ["auto", "fixed"],
       "description": "Optional, \"auto\" by default: the screen zooms in on children standing far away (up to 1.8×), so every child's avatar is about the same size near or far; the view changes only at calibration and in breaks, never during play. \"fixed\": always the whole camera image (for a game drawn on exact spots of the live video)."

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fabi-play/protocol",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Messages between the phone, the screen and the games, with their checks.",
   "keywords": [
     "fabi-play",