tapline-host 0.0.0-stage → 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tapline contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,90 @@
-# Temporary Holding Version
+# tapline-host
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+The computer side of [Tapline](https://github.com/bo5amis/tapline#readme): the server your app's
+`react-native-tapline` connects to over the same Wi-Fi, an MCP server for agents, and a CLI.
+Node.js 20 or later. Full setup: [docs/setup.md](https://github.com/bo5amis/tapline/blob/main/docs/setup.md).
+
+## CLI
+
+| Command | |
+|---|---|
+| `tapline mcp` | the host plus an MCP server on stdio (what agents run) |
+| `tapline host` | the host alone, logging connections and app logs (`--scheme` also prints a pairing QR code) |
+| `tapline pair --scheme <s>` | a pairing link and QR code in the terminal |
+| `tapline info` | this computer's name and address, and the code the app needs |
+| `tapline doctor` | check Node, this computer's Wi-Fi or LAN address, the token, ports and the Windows firewall (Tailscale and Metro over https are optional: warnings only) |
+| `tapline dev [-- <metro command>]` | optional, for a phone away from this Wi-Fi: serve Metro over https in your tailnet and run Metro with `EXPO_PACKAGER_PROXY_URL` set |
+| `tapline replay <name>` | replay a recorded script without an agent (exit code 1 when a step fails) |
+| `tapline token` | print the token (only to connect without pairing, e.g. a simulator) |
+| `tapline skill install` / `skill path` | install the agent skill for Claude Code, Codex and other agents / print where it is |
+
+Options: `--port <n>` for a fixed port (or `TAPLINE_PORT`; otherwise 7210, moving up to
+7219 when taken, so each agent's host gets its own); `--host <addr>[,<addr>…]` to listen
+on other addresses; `--screenshots file` (or `TAPLINE_SCREENSHOTS=file`) makes `look` save
+screenshots in `~/.tapline/shots` (newest 50 kept) and return the path instead of the image,
+for agents that hand images to smaller models; `--viewer <local|remote|off>` (or
+`TAPLINE_VIEWER`) sets where the live viewer opens: this computer only (default), also your
+own devices on Tailscale, or nowhere. `TAPLINE_HOME` moves the data directory (default
+`~/.tapline`).
+
+Claude Code: `claude mcp add tapline -- tapline mcp`. Other MCP clients: run `tapline mcp`
+as a stdio server; it logs to stderr only.
+
+## MCP tools
+
+`pair`, `apps`, `open_app`, `look`, `check_layout`, `tap`, `long_press`, `type_text`,
+`scroll`, `swipe`, `toggle`, `open_url`, `back`, `app_command`, `wait_for`, `record`, `replay`,
+`compare_screen`, `filmstrip`, `viewer`, `logs`, `reload`, `wait`. Targets: `{ref}`, `{id}`
+(the `testID` prop), `{label}`, `{text}` or `{x, y}`; `index` goes with `id`, `label` and
+`text`. Every action answers with the verb and the element, then `Same screen. Changes:`
+(or the full list when the screen changed), then `Settled in N s.`, then any `Note:` lines;
+`look: false` returns only the first line. No image is sent by default: `screenshot: true`
+on any action or on `look` adds one (`width` in pixels, default 600). Errors read
+`Error <code>: <message>`; a `not_actionable` message starts with its reason (`disabled`,
+`off_screen`, `covered`, `under_tab_bar`, `behind_alert`, `no_action`). After a person acts
+in the live viewer, `{ref}` targets fail with `stale` until the next `look`. `scroll.amount`
+is a share of the scroll area (0.1 to 1, default 0.8); `long_press` takes `seconds`, which
+Tapline ignores.
+
+- `pair {scheme}` returns a pairing link for this host (with the port it actually got),
+  its QR code as an image, and the forget link; the QR code is also saved as
+  `~/.tapline/pair-<port>.png`. A code works once, for 10 minutes.
+- With several apps connected, the latest one is used until `open_app {app}` picks another (`app` is a session id, app
+  id or name; several matches give `ambiguous`).
+  Phones that send no name (iPhones) show their install id.
+- If an app can't take screenshots, the host remembers it for that session and stops asking.
+- `reload` waits for the app to reconnect, then returns the element list.
+- `logs` returns what the host has buffered since the app connected (500 lines); with
+  nothing buffered it asks the app for its own.
+- `viewer` returns the live viewer's link, plus the app's capture and stutter numbers while
+  someone watches. The viewer rides on the host's port (`/viewer`, `/stream`, `/ws`) and
+  speaks [docs/VIEWER-PROTOCOL.md](https://github.com/bo5amis/tapline/blob/main/docs/VIEWER-PROTOCOL.md). The app captures frames
+  only while someone watches. A person's input goes ahead of the agent's actions, which
+  wait until the person has been idle for 3 s (`busy` after 30 s; `paused` while paused).
+
+## Security
+
+- The token (32 random hex characters) is created on the first run in `~/.tapline/token`
+  (`%USERPROFILE%\.tapline\token` on Windows). Treat it like a password. Pairing hands it
+  to the app, which keeps it in the keychain; the MCP server never shows it to the agent.
+  Delete the file to make a new one: every phone then has to pair again.
+- Pairing codes are signed with the token, expire after 10 minutes, and each host accepts
+  a code once.
+- The host listens on this computer's Wi-Fi or LAN address (private addresses on real
+  adapters only), on its Tailscale address when there is one, and on 127.0.0.1. Phones pair
+  with the Wi-Fi or LAN address. `--host <addr>` chooses other addresses.
+- Connections without the right token are closed (4001), and bad pairing codes too
+  (4004). Traffic is plain WebSocket: use a Wi-Fi you trust (Tailscale, when used, encrypts it).
+- The live viewer has its own random key, new on each start, in the link the `viewer` tool
+  gives. Only that first page load takes the key in the address: it sets an HttpOnly,
+  SameSite=Strict cookie and the page removes the key from the address bar. The stream and
+  the WebSocket take the cookie or an `Authorization: Bearer` header, never the key in the
+  URL. A cookie sent from another site (even another port on this computer:
+  `Sec-Fetch-Site`) is refused, so are requests from another Host or (for the WebSocket)
+  another Origin, and by default only this computer can open it.
+
+## Develop
+
+```sh
+npm run typecheck && npm run build && npm test
+```

package/dist/annotate.d.ts

@@ -0,0 +1,20 @@
+import type { TreeElement } from "./protocol.js";
+/**
+ * Labeled screenshots: draws each actionable element's box and its ref (for example "e12")
+ * onto a JPEG screenshot, so a vision model can name elements by ref.
+ */
+export interface ShotImage {
+    mime: string;
+    data: string;
+    width?: number;
+    height?: number;
+}
+export interface ScreenSize {
+    width: number;
+    height: number;
+}
+type Rgb = readonly [number, number, number];
+/** Saturated, high-contrast colours (white text stays readable on all of them). */
+export declare const ANNOTATION_PALETTE: readonly Rgb[];
+export declare function annotateScreenshot(shot: ShotImage, elements: TreeElement[], screen: ScreenSize): ShotImage;
+export {};

package/dist/annotate.js

@@ -0,0 +1,139 @@
+import jpeg from "jpeg-js";
+/** Saturated, high-contrast colours (white text stays readable on all of them). */
+export const ANNOTATION_PALETTE = [
+    [220, 20, 60], // crimson
+    [0, 90, 255], // blue
+    [0, 140, 60], // green
+    [150, 40, 200], // purple
+    [230, 100, 0], // orange
+    [0, 130, 140], // teal
+];
+const JPEG_QUALITY = 80;
+const OUTLINE = 2;
+const WIDE_IMAGE = 500;
+const MID_IMAGE = 280;
+const GLYPH_W = 5;
+const GLYPH_H = 7;
+// 5x7 glyphs, one string per row, '#' = ink.
+const GLYPHS = {
+    e: [".....", ".....", ".###.", "#...#", "#####", "#....", ".###."],
+    "0": [".###.", "#...#", "#..##", "#.#.#", "##..#", "#...#", ".###."],
+    "1": ["..#..", ".##..", "..#..", "..#..", "..#..", "..#..", ".###."],
+    "2": [".###.", "#...#", "....#", "...#.", "..#..", ".#...", "#####"],
+    "3": ["#####", "...#.", "..#..", "...#.", "....#", "#...#", ".###."],
+    "4": ["...#.", "..##.", ".#.#.", "#..#.", "#####", "...#.", "...#."],
+    "5": ["#####", "#....", "####.", "....#", "....#", "#...#", ".###."],
+    "6": ["..##.", ".#...", "#....", "####.", "#...#", "#...#", ".###."],
+    "7": ["#####", "....#", "...#.", "..#..", ".#...", ".#...", ".#..."],
+    "8": [".###.", "#...#", "#...#", ".###.", "#...#", "#...#", ".###."],
+    "9": [".###.", "#...#", "#...#", ".####", "....#", "...#.", ".##.."],
+};
+function setPixel(canvas, x, y, colour) {
+    if (x < 0 || y < 0 || x >= canvas.width || y >= canvas.height)
+        return;
+    const i = (y * canvas.width + x) * 4;
+    canvas.data[i] = colour[0];
+    canvas.data[i + 1] = colour[1];
+    canvas.data[i + 2] = colour[2];
+    canvas.data[i + 3] = 255;
+}
+function fillRect(canvas, x, y, w, h, colour) {
+    const x0 = Math.max(0, x);
+    const y0 = Math.max(0, y);
+    const x1 = Math.min(canvas.width, x + w);
+    const y1 = Math.min(canvas.height, y + h);
+    for (let yy = y0; yy < y1; yy++) {
+        for (let xx = x0; xx < x1; xx++)
+            setPixel(canvas, xx, yy, colour);
+    }
+}
+/** Outline drawn inside the rectangle's edges. */
+function strokeRect(canvas, x, y, w, h, thickness, colour) {
+    const t = Math.min(thickness, Math.floor(w / 2), Math.floor(h / 2));
+    if (t <= 0) {
+        fillRect(canvas, x, y, w, h, colour);
+        return;
+    }
+    fillRect(canvas, x, y, w, t, colour); // top
+    fillRect(canvas, x, y + h - t, w, t, colour); // bottom
+    fillRect(canvas, x, y + t, t, h - 2 * t, colour); // left
+    fillRect(canvas, x + w - t, y + t, t, h - 2 * t, colour); // right
+}
+const WHITE = [255, 255, 255];
+function drawLabel(canvas, text, frameX, frameY, colour) {
+    const glyphs = [...text].map((ch) => GLYPHS[ch]);
+    if (glyphs.some((g) => g === undefined))
+        return;
+    // Big enough to read on a phone screenshot (a vision model found 2x on 600 px too small).
+    const scale = canvas.width >= WIDE_IMAGE ? 3 : canvas.width >= MID_IMAGE ? 2 : 1;
+    const pad = scale;
+    const spacing = scale; // 1 px of the 1x grid between glyphs
+    const textW = glyphs.length * GLYPH_W * scale + Math.max(0, glyphs.length - 1) * spacing;
+    const boxW = textW + 2 * pad;
+    const boxH = GLYPH_H * scale + 2 * pad;
+    // Keep the whole label inside the image: shift left/down when the frame touches an edge.
+    const boxX = Math.max(0, Math.min(frameX, canvas.width - boxW));
+    const boxY = Math.max(0, Math.min(frameY, canvas.height - boxH));
+    fillRect(canvas, boxX, boxY, boxW, boxH, colour);
+    let cursor = boxX + pad;
+    for (const glyph of glyphs) {
+        if (glyph === undefined)
+            continue;
+        for (let row = 0; row < GLYPH_H; row++) {
+            const line = glyph[row] ?? "";
+            for (let col = 0; col < GLYPH_W; col++) {
+                if (line[col] === "#")
+                    fillRect(canvas, cursor + col * scale, boxY + pad + row * scale, scale, scale, WHITE);
+            }
+        }
+        cursor += GLYPH_W * scale + spacing;
+    }
+}
+function isLabelled(element) {
+    if (!element.frame)
+        return false;
+    return (element.actions ?? []).some((action) => action !== "scroll");
+}
+export function annotateScreenshot(shot, elements, screen) {
+    if (shot.mime !== "image/jpeg")
+        return shot;
+    if (!(screen.width > 0) || !(screen.height > 0))
+        return shot;
+    let decoded;
+    try {
+        decoded = jpeg.decode(Buffer.from(shot.data, "base64"), { useTArray: true });
+    }
+    catch {
+        return shot;
+    }
+    if (decoded.width <= 0 || decoded.height <= 0)
+        return shot;
+    const canvas = { width: decoded.width, height: decoded.height, data: decoded.data };
+    const scale = canvas.width / screen.width;
+    let index = 0;
+    for (const element of elements) {
+        const frame = element.frame;
+        if (!frame || !isLabelled(element))
+            continue;
+        const x = Math.round(frame.x * scale);
+        const y = Math.round(frame.y * scale);
+        const w = Math.max(1, Math.round(frame.width * scale));
+        const h = Math.max(1, Math.round(frame.height * scale));
+        if (x >= canvas.width || y >= canvas.height || x + w <= 0 || y + h <= 0)
+            continue;
+        const colour = ANNOTATION_PALETTE[index % ANNOTATION_PALETTE.length];
+        index++;
+        if (!colour)
+            continue;
+        strokeRect(canvas, x, y, w, h, OUTLINE, colour);
+        drawLabel(canvas, element.ref, x, y, colour);
+    }
+    const encoded = jpeg.encode({ data: canvas.data, width: canvas.width, height: canvas.height }, JPEG_QUALITY);
+    return {
+        mime: "image/jpeg",
+        data: encoded.data.toString("base64"),
+        width: canvas.width,
+        height: canvas.height,
+    };
+}
+//# sourceMappingURL=annotate.js.map

package/dist/annotate.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"annotate.js","sourceRoot":"","sources":["../src/annotate.ts"],"names":[],"mappings":"AAAA,OAAO,IAAI,MAAM,SAAS,CAAC;AAsB3B,mFAAmF;AACnF,MAAM,CAAC,MAAM,kBAAkB,GAAmB;IAChD,CAAC,GAAG,EAAE,EAAE,EAAE,EAAE,CAAC,EAAE,UAAU;IACzB,CAAC,CAAC,EAAE,EAAE,EAAE,GAAG,CAAC,EAAE,OAAO;IACrB,CAAC,CAAC,EAAE,GAAG,EAAE,EAAE,CAAC,EAAE,QAAQ;IACtB,CAAC,GAAG,EAAE,EAAE,EAAE,GAAG,CAAC,EAAE,SAAS;IACzB,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,CAAC,EAAE,SAAS;IACxB,CAAC,CAAC,EAAE,GAAG,EAAE,GAAG,CAAC,EAAE,OAAO;CACvB,CAAC;AAEF,MAAM,YAAY,GAAG,EAAE,CAAC;AACxB,MAAM,OAAO,GAAG,CAAC,CAAC;AAClB,MAAM,UAAU,GAAG,GAAG,CAAC;AACvB,MAAM,SAAS,GAAG,GAAG,CAAC;AACtB,MAAM,OAAO,GAAG,CAAC,CAAC;AAClB,MAAM,OAAO,GAAG,CAAC,CAAC;AAElB,6CAA6C;AAC7C,MAAM,MAAM,GAAgD;IAC1D,CAAC,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IAClE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IACpE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IACpE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IACpE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IACpE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IACpE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IACpE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IACpE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IACpE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;IACpE,GAAG,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,CAAC;CACrE,CAAC;AAQF,SAAS,QAAQ,CAAC,MAAc,EAAE,CAAS,EAAE,CAAS,EAAE,MAAW;IACjE,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,MAAM,CAAC,KAAK,IAAI,CAAC,IAAI,MAAM,CAAC,MAAM;QAAE,OAAO;IACtE,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,CAAC,KAAK,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC;IACrC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,CAAC;IAC3B,MAAM,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,CAAC;IAC/B,MAAM,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,CAAC;IAC/B,MAAM,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,GAAG,CAAC;AAC3B,CAAC;AAED,SAAS,QAAQ,CAAC,MAAc,EAAE,CAAS,EAAE,CAAS,EAAE,CAAS,EAAE,CAAS,EAAE,MAAW;IACvF,MAAM,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC1B,MAAM,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IAC1B,MAAM,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;IACzC,MAAM,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;IAC1C,KAAK,IAAI,EAAE,GAAG,EAAE,EAAE,EAAE,GAAG,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC;QAChC,KAAK,IAAI,EAAE,GAAG,EAAE,EAAE,EAAE,GAAG,EAAE,EAAE,EAAE,EAAE;YAAE,QAAQ,CAAC,MAAM,EAAE,EAAE,EAAE,EAAE,EAAE,MAAM,CAAC,CAAC;IACpE,CAAC;AACH,CAAC;AAED,kDAAkD;AAClD,SAAS,UAAU,CAAC,MAAc,EAAE,CAAS,EAAE,CAAS,EAAE,CAAS,EAAE,CAAS,EAAE,SAAiB,EAAE,MAAW;IAC5G,MAAM,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;IACpE,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QACX,QAAQ,CAAC,MAAM,EAAE,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,CAAC;QACrC,OAAO;IACT,CAAC;IACD,QAAQ,CAAC,MAAM,EAAE,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,MAAM;IAC5C,QAAQ,CAAC,MAAM,EAAE,CAAC,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,SAAS;IACvD,QAAQ,CAAC,MAAM,EAAE,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,OAAO;IACzD,QAAQ,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,QAAQ;AACpE,CAAC;AAED,MAAM,KAAK,GAAQ,CAAC,GAAG,EAAE,GAAG,EAAE,GAAG,CAAC,CAAC;AAEnC,SAAS,SAAS,CAAC,MAAc,EAAE,IAAY,EAAE,MAAc,EAAE,MAAc,EAAE,MAAW;IAC1F,MAAM,MAAM,GAAG,CAAC,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IACjD,IAAI,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,SAAS,CAAC;QAAE,OAAO;IAChD,0FAA0F;IAC1F,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,IAAI,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,IAAI,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACjF,MAAM,GAAG,GAAG,KAAK,CAAC;IAClB,MAAM,OAAO,GAAG,KAAK,CAAC,CAAC,qCAAqC;IAC5D,MAAM,KAAK,GAAG,MAAM,CAAC,MAAM,GAAG,OAAO,GAAG,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC,GAAG,OAAO,CAAC;IACzF,MAAM,IAAI,GAAG,KAAK,GAAG,CAAC,GAAG,GAAG,CAAC;IAC7B,MAAM,IAAI,GAAG,OAAO,GAAG,KAAK,GAAG,CAAC,GAAG,GAAG,CAAC;IACvC,yFAAyF;IACzF,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,KAAK,GAAG,IAAI,CAAC,CAAC,CAAC;IAChE,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,GAAG,IAAI,CAAC,CAAC,CAAC;IACjE,QAAQ,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,CAAC;IACjD,IAAI,MAAM,GAAG,IAAI,GAAG,GAAG,CAAC;IACxB,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;QAC3B,IAAI,KAAK,KAAK,SAAS;YAAE,SAAS;QAClC,KAAK,IAAI,GAAG,GAAG,CAAC,EAAE,GAAG,GAAG,OAAO,EAAE,GAAG,EAAE,EAAE,CAAC;YACvC,MAAM,IAAI,GAAG,KAAK,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC;YAC9B,KAAK,IAAI,GAAG,GAAG,CAAC,EAAE,GAAG,GAAG,OAAO,EAAE,GAAG,EAAE,EAAE,CAAC;gBACvC,IAAI,IAAI,CAAC,GAAG,CAAC,KAAK,GAAG;oBAAE,QAAQ,CAAC,MAAM,EAAE,MAAM,GAAG,GAAG,GAAG,KAAK,EAAE,IAAI,GAAG,GAAG,GAAG,GAAG,GAAG,KAAK,EAAE,KAAK,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC;YAC/G,CAAC;QACH,CAAC;QACD,MAAM,IAAI,OAAO,GAAG,KAAK,GAAG,OAAO,CAAC;IACtC,CAAC;AACH,CAAC;AAED,SAAS,UAAU,CAAC,OAAoB;IACtC,IAAI,CAAC,OAAO,CAAC,KAAK;QAAE,OAAO,KAAK,CAAC;IACjC,OAAO,CAAC,OAAO,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,KAAK,QAAQ,CAAC,CAAC;AACvE,CAAC;AAED,MAAM,UAAU,kBAAkB,CAAC,IAAe,EAAE,QAAuB,EAAE,MAAkB;IAC7F,IAAI,IAAI,CAAC,IAAI,KAAK,YAAY;QAAE,OAAO,IAAI,CAAC;IAC5C,IAAI,CAAC,CAAC,MAAM,CAAC,KAAK,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC;QAAE,OAAO,IAAI,CAAC;IAE7D,IAAI,OAA4D,CAAC;IACjE,IAAI,CAAC;QACH,OAAO,GAAG,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,QAAQ,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IAC/E,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,IAAI,CAAC;IACd,CAAC;IACD,IAAI,OAAO,CAAC,KAAK,IAAI,CAAC,IAAI,OAAO,CAAC,MAAM,IAAI,CAAC;QAAE,OAAO,IAAI,CAAC;IAE3D,MAAM,MAAM,GAAW,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,IAAI,EAAE,OAAO,CAAC,IAAI,EAAE,CAAC;IAC5F,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC;IAE1C,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,KAAK,MAAM,OAAO,IAAI,QAAQ,EAAE,CAAC;QAC/B,MAAM,KAAK,GAAG,OAAO,CAAC,KAAK,CAAC;QAC5B,IAAI,CAAC,KAAK,IAAI,CAAC,UAAU,CAAC,OAAO,CAAC;YAAE,SAAS;QAC7C,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QACtC,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QACtC,MAAM,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,KAAK,GAAG,KAAK,CAAC,CAAC,CAAC;QACvD,MAAM,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,KAAK,CAAC,CAAC,CAAC;QACxD,IAAI,CAAC,IAAI,MAAM,CAAC,KAAK,IAAI,CAAC,IAAI,MAAM,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC;YAAE,SAAS;QAClF,MAAM,MAAM,GAAG,kBAAkB,CAAC,KAAK,GAAG,kBAAkB,CAAC,MAAM,CAAC,CAAC;QACrE,KAAK,EAAE,CAAC;QACR,IAAI,CAAC,MAAM;YAAE,SAAS;QACtB,UAAU,CAAC,MAAM,EAAE,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,OAAO,EAAE,MAAM,CAAC,CAAC;QAChD,SAAS,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,EAAE,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,CAAC;IAC/C,CAAC;IAED,MAAM,OAAO,GAAG,IAAI,CAAC,MAAM,CAAC,EAAE,IAAI,EAAE,MAAM,CAAC,IAAI,EAAE,KAAK,EAAE,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,EAAE,YAAY,CAAC,CAAC;IAC7G,OAAO;QACL,IAAI,EAAE,YAAY;QAClB,IAAI,EAAE,OAAO,CAAC,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC;QACrC,KAAK,EAAE,MAAM,CAAC,KAAK;QACnB,MAAM,EAAE,MAAM,CAAC,MAAM;KACtB,CAAC;AACJ,CAAC"}

package/dist/captions.d.ts

@@ -0,0 +1,14 @@
+/**
+ * VIEWER-PROTOCOL 1.4: the `kind` of an action the viewer is shown, and the plain-language `caption`
+ * for the agent's steps. A caption never carries refs, coordinates, typed text or a link's address,
+ * and never the failure reason (the page adds that). A person's steps get no caption from here:
+ * the page words them itself ("You tapped …").
+ */
+import type { PersonInput } from "./viewer.js";
+export declare function agentKind(method: string): string;
+/** A person's input → the viewer's input name (tap, longPress, drag, scroll, text, enter, backspace). */
+export declare function personKind(input: PersonInput, method: string): string;
+/** The reason word of a refusal, for the viewer (1.4 `reason`); undefined for any other error. */
+export declare function reasonOf(code: string | undefined, message: string | undefined): string | undefined;
+/** The agent's step in plain words, or undefined for steps a viewer needn't see (wait_for). */
+export declare function captionFor(kind: string, label?: string, params?: unknown): string | undefined;

package/dist/captions.js

@@ -0,0 +1,101 @@
+/** The app's method → the AGENT-TOOLS tool the agent called. */
+const AGENT_KINDS = {
+    press: "tap",
+    longPress: "long_press",
+    type: "type_text",
+    scroll: "scroll",
+    swipe: "swipe",
+    toggle: "toggle",
+    back: "back",
+    openURL: "open_url",
+    waitFor: "wait_for",
+    command: "app_command",
+    reload: "reload",
+};
+export function agentKind(method) {
+    return AGENT_KINDS[method] ?? method;
+}
+/** A person's input → the viewer's input name (tap, longPress, drag, scroll, text, enter, backspace). */
+export function personKind(input, method) {
+    switch (input.type) {
+        case "text":
+            return "text";
+        case "key":
+            return input.key;
+        case "scroll":
+            return "scroll";
+        case "gesture":
+            if (method === "press")
+                return "tap";
+            if (method === "longPress")
+                return "longPress";
+            if (method === "swipe")
+                return "drag";
+            return "scroll";
+    }
+}
+/** AGENT-TOOLS §4's reason words: the first word of a not_actionable message. */
+const REASONS = new Set(["disabled", "off_screen", "covered", "under_tab_bar", "behind_alert", "no_action"]);
+/** The reason word of a refusal, for the viewer (1.4 `reason`); undefined for any other error. */
+export function reasonOf(code, message) {
+    if (code !== "not_actionable" || message === undefined)
+        return undefined;
+    const word = /^([a-z_]+):/.exec(message.trim())?.[1];
+    return word !== undefined && REASONS.has(word) ? word : undefined;
+}
+const NAME_MAX = 32;
+/** An element's name for a caption: 32 characters at most, then an ellipsis. */
+function shortName(label) {
+    const clean = label.replace(/\s+/g, " ").trim();
+    return clean.length <= NAME_MAX ? clean : `${clean.slice(0, NAME_MAX).trimEnd()}…`;
+}
+function field(params, key) {
+    return typeof params === "object" && params !== null ? params[key] : undefined;
+}
+const DIRECTIONS = new Set(["up", "down", "left", "right"]);
+/** The agent's step in plain words, or undefined for steps a viewer needn't see (wait_for). */
+export function captionFor(kind, label, params) {
+    const name = label !== undefined && label.trim() !== "" ? shortName(label) : undefined;
+    const quoted = name !== undefined ? ` "${name}"` : "";
+    switch (kind) {
+        case "tap":
+            return `Tap${quoted}`;
+        case "long_press":
+            return `Press and hold${quoted}`;
+        case "type_text":
+            return name !== undefined ? `Type into${quoted}` : "Type";
+        case "scroll": {
+            const direction = field(params, "direction");
+            return typeof direction === "string" && DIRECTIONS.has(direction) ? `Scroll ${direction}` : "Scroll";
+        }
+        case "swipe":
+            return "Swipe";
+        case "toggle": {
+            const value = field(params, "value");
+            if (value === true)
+                return `Turn on${quoted}`;
+            if (value === false)
+                return `Turn off${quoted}`;
+            return `Switch${quoted}`;
+        }
+        case "back":
+            return "Go back";
+        case "open_url":
+            return "Open link";
+        case "reload":
+            return "Reload the app";
+        case "app_command": {
+            const command = field(params, "name");
+            const scheme = field(field(params, "args"), "scheme");
+            if (command === "appearance" && (scheme === "dark" || scheme === "light"))
+                return `Switch to ${scheme} mode`;
+            const named = typeof command === "string" && command !== "" ? shortName(command) : name;
+            return named !== undefined ? `Run "${named}"` : "Run a test command";
+        }
+        case "wait_for":
+            return undefined;
+        default:
+            return kind.replace(/_/g, " ");
+    }
+}
+//# sourceMappingURL=captions.js.map

package/dist/captions.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"captions.js","sourceRoot":"","sources":["../src/captions.ts"],"names":[],"mappings":"AAQA,gEAAgE;AAChE,MAAM,WAAW,GAA2B;IAC1C,KAAK,EAAE,KAAK;IACZ,SAAS,EAAE,YAAY;IACvB,IAAI,EAAE,WAAW;IACjB,MAAM,EAAE,QAAQ;IAChB,KAAK,EAAE,OAAO;IACd,MAAM,EAAE,QAAQ;IAChB,IAAI,EAAE,MAAM;IACZ,OAAO,EAAE,UAAU;IACnB,OAAO,EAAE,UAAU;IACnB,OAAO,EAAE,aAAa;IACtB,MAAM,EAAE,QAAQ;CACjB,CAAC;AAEF,MAAM,UAAU,SAAS,CAAC,MAAc;IACtC,OAAO,WAAW,CAAC,MAAM,CAAC,IAAI,MAAM,CAAC;AACvC,CAAC;AAED,yGAAyG;AACzG,MAAM,UAAU,UAAU,CAAC,KAAkB,EAAE,MAAc;IAC3D,QAAQ,KAAK,CAAC,IAAI,EAAE,CAAC;QACnB,KAAK,MAAM;YACT,OAAO,MAAM,CAAC;QAChB,KAAK,KAAK;YACR,OAAO,KAAK,CAAC,GAAG,CAAC;QACnB,KAAK,QAAQ;YACX,OAAO,QAAQ,CAAC;QAClB,KAAK,SAAS;YACZ,IAAI,MAAM,KAAK,OAAO;gBAAE,OAAO,KAAK,CAAC;YACrC,IAAI,MAAM,KAAK,WAAW;gBAAE,OAAO,WAAW,CAAC;YAC/C,IAAI,MAAM,KAAK,OAAO;gBAAE,OAAO,MAAM,CAAC;YACtC,OAAO,QAAQ,CAAC;IACpB,CAAC;AACH,CAAC;AAED,iFAAiF;AACjF,MAAM,OAAO,GAAG,IAAI,GAAG,CAAC,CAAC,UAAU,EAAE,YAAY,EAAE,SAAS,EAAE,eAAe,EAAE,cAAc,EAAE,WAAW,CAAC,CAAC,CAAC;AAE7G,kGAAkG;AAClG,MAAM,UAAU,QAAQ,CAAC,IAAwB,EAAE,OAA2B;IAC5E,IAAI,IAAI,KAAK,gBAAgB,IAAI,OAAO,KAAK,SAAS;QAAE,OAAO,SAAS,CAAC;IACzE,MAAM,IAAI,GAAG,aAAa,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;IACrD,OAAO,IAAI,KAAK,SAAS,IAAI,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;AACpE,CAAC;AAED,MAAM,QAAQ,GAAG,EAAE,CAAC;AAEpB,gFAAgF;AAChF,SAAS,SAAS,CAAC,KAAa;IAC9B,MAAM,KAAK,GAAG,KAAK,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;IAChD,OAAO,KAAK,CAAC,MAAM,IAAI,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,QAAQ,CAAC,CAAC,OAAO,EAAE,GAAG,CAAC;AACrF,CAAC;AAED,SAAS,KAAK,CAAC,MAAe,EAAE,GAAW;IACzC,OAAO,OAAO,MAAM,KAAK,QAAQ,IAAI,MAAM,KAAK,IAAI,CAAC,CAAC,CAAE,MAAkC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;AAC9G,CAAC;AAED,MAAM,UAAU,GAAG,IAAI,GAAG,CAAC,CAAC,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,CAAC,CAAC,CAAC;AAE5D,+FAA+F;AAC/F,MAAM,UAAU,UAAU,CAAC,IAAY,EAAE,KAAc,EAAE,MAAgB;IACvE,MAAM,IAAI,GAAG,KAAK,KAAK,SAAS,IAAI,KAAK,CAAC,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;IACvF,MAAM,MAAM,GAAG,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,KAAK,IAAI,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC;IACtD,QAAQ,IAAI,EAAE,CAAC;QACb,KAAK,KAAK;YACR,OAAO,MAAM,MAAM,EAAE,CAAC;QACxB,KAAK,YAAY;YACf,OAAO,iBAAiB,MAAM,EAAE,CAAC;QACnC,KAAK,WAAW;YACd,OAAO,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,YAAY,MAAM,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC;QAC5D,KAAK,QAAQ,CAAC,CAAC,CAAC;YACd,MAAM,SAAS,GAAG,KAAK,CAAC,MAAM,EAAE,WAAW,CAAC,CAAC;YAC7C,OAAO,OAAO,SAAS,KAAK,QAAQ,IAAI,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,UAAU,SAAS,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC;QACvG,CAAC;QACD,KAAK,OAAO;YACV,OAAO,OAAO,CAAC;QACjB,KAAK,QAAQ,CAAC,CAAC,CAAC;YACd,MAAM,KAAK,GAAG,KAAK,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;YACrC,IAAI,KAAK,KAAK,IAAI;gBAAE,OAAO,UAAU,MAAM,EAAE,CAAC;YAC9C,IAAI,KAAK,KAAK,KAAK;gBAAE,OAAO,WAAW,MAAM,EAAE,CAAC;YAChD,OAAO,SAAS,MAAM,EAAE,CAAC;QAC3B,CAAC;QACD,KAAK,MAAM;YACT,OAAO,SAAS,CAAC;QACnB,KAAK,UAAU;YACb,OAAO,WAAW,CAAC;QACrB,KAAK,QAAQ;YACX,OAAO,gBAAgB,CAAC;QAC1B,KAAK,aAAa,CAAC,CAAC,CAAC;YACnB,MAAM,OAAO,GAAG,KAAK,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;YACtC,MAAM,MAAM,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,QAAQ,CAAC,CAAC;YACtD,IAAI,OAAO,KAAK,YAAY,IAAI,CAAC,MAAM,KAAK,MAAM,IAAI,MAAM,KAAK,OAAO,CAAC;gBAAE,OAAO,aAAa,MAAM,OAAO,CAAC;YAC7G,MAAM,KAAK,GAAG,OAAO,OAAO,KAAK,QAAQ,IAAI,OAAO,KAAK,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;YACxF,OAAO,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,QAAQ,KAAK,GAAG,CAAC,CAAC,CAAC,oBAAoB,CAAC;QACvE,CAAC;QACD,KAAK,UAAU;YACb,OAAO,SAAS,CAAC;QACnB;YACE,OAAO,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,GAAG,CAAC,CAAC;IACnC,CAAC;AACH,CAAC"}

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};