pw-repl 0.7.2 → 0.8.1

package/AGENTS.md

@@ -13,10 +13,11 @@ If you hit a limitation or write a workaround, consider adding the capability to
   needs no tab, only reads, or takes `--all` is also listed in `lib/runner.js`'s `NO_TAB_NEEDED`,
   `READ_ONLY` or `INSPECTION`, and one that takes fixed words in `complete` (`lib/commands.js`).
 - The modules, one per facility, each with its commands: `lib/tabs.js` (tabs, navigating),
-  `lib/elements.js` (choosing a match; the interact and mouse commands), `lib/inspect.js` (reading the
+  `lib/elements.js` (choosing a match; the interact and mouse commands; highlight), `lib/cursor.js` (the pointer
+  drawn over the page, which the element and mouse commands glide first), `lib/inspect.js` (reading the
   page, screenshots, eval, CDP, waits), `lib/watch.js`, `lib/requestlog.js` (requests, bodies, console),
   `lib/routes.js`, `lib/network.js`, `lib/emulation.js` (and viewport), `lib/capture.js`, `lib/record.js`
-  (video, through ffmpeg), `lib/dialogs.js` and `lib/modes.js`. Beneath them, `lib/tabstate.js` is the one record kept per tab, `lib/cdp.js` the CDP
+  (video, through ffmpeg, and the steps file saved with it), `lib/dialogs.js` and `lib/modes.js`. Beneath them, `lib/tabstate.js` is the one record kept per tab, `lib/cdp.js` the CDP
   session kept per tab, and `lib/util.js` small shared helpers. No facility requires `lib/modes.js` or
   `lib/commands.js`, and a unit test fails on a require cycle, which CommonJS would not report.
 - `bin/pw-repl.js` is the command line; `lib/start.js` connects and runs the prompt; `lib/runner.js`
@@ -27,10 +28,11 @@ If you hit a limitation or write a workaround, consider adding the capability to
   names to these; `lib/state.js` holds the session state, with each client's selected tab;
   `lib/output.js` routes all output so the server can return it; `lib/skill.js` stamps the skill
   `pw-repl skill` prints, and gives `where` its hash.
-- Match playwright-cli (Microsoft's CLI for agents) where it is reasonable: agents are likelier to be
-  trained on it. A new command, option or output that does what one of its does takes its name and form,
-  or accepts them too (`lib/cli-names.js`, `help playwright-cli`). Differ where sharing a browser with a
-  person needs it, and say why in the help.
+- Match Playwright and playwright-cli (Microsoft's CLI for agents) as closely as is reasonable: agents are
+  likelier to be trained on them. A new command, option or output that does what one of theirs does takes
+  its name, form and API style (an option named as Playwright's API names it, e.g. type's `delay`), or
+  accepts theirs too (`lib/cli-names.js`, `help playwright-cli`). Deviate only where it is necessary, as
+  where sharing a browser with a user needs it, and say why in the help.
 - Comment the *why* when it isn't obvious from the code.
 - `npm test` runs the suite (under two minutes): a private headless Chromium, a local test site
   (`test/harness.js`), and the real REPL with its server. Nothing is mocked, and the shared browser is
@@ -60,7 +62,10 @@ If you hit a limitation or write a workaround, consider adding the capability to
 
 Work goes in waves, and each one ends with a release that is ready to push.
 
-1. Build what was asked, with tests, and try it in a running REPL.
+1. Build what was asked, with tests, and try it in a running REPL. For anything the user will look at
+   (a cursor, an overlay, a video), show them a demo while it is still a prototype (a recording or
+   screenshots they can open) and get their go-ahead on how it looks before writing tests or
+   building on it.
 2. Try it on a fresh agent that has not seen the code or the change (a guinea pig), on something real: a
    small local page whose planted bugs need the change to find.
    - Brief it with goals only: what to find out or do in the browser, never the commands.

package/README.md

@@ -99,6 +99,8 @@ the Docker bridge gateway, `ip route | awk '/default/ {print $3}'`).
 | see each step as I click             | `watch on --live`                                                                     |
 | record requests and console together | `capture on`, then `capture off`                                                      |
 | record a video of the page           | `record on`, then `record off` (`help video` for the other ways)                      |
+| show a pointer where the agent acts  | `cursor on`, then `cursor off` (in screenshots and videos)                            |
+| point something out on the page      | `highlight <ref>` (`--style=<css>` to restyle it), then `highlight off`               |
 | break the backend on purpose         | `route <glob> <status> <json>` (fake a response), `route <glob> abort`, `network off` |
 | change or slow an API response       | `route <glob> patch <json>`, `route <glob> delay <secs>`                              |
 | slow the whole network               | `network slow`                                                                        |
@@ -108,13 +110,15 @@ Everything else is in `help <topic>`; `help <command>` has usage and caveats.
 
 ### Modes
 
-`watch`, `capture`, `route`, `network off` or `slow`, and `emulate` stay on until you turn them off:
-`watch on|off`, `capture on|off`, `route ...|route off`, `network off|slow|on`, `emulate ...|emulate off`.
+`watch`, `capture`, `route`, `network off` or `slow`, `emulate`, `viewport`, `record`, `highlight` and `cursor`
+stay on until you turn them off: `watch on|off`, `capture on|off`, `route ...|route off`, `network off|slow|on`,
+`emulate ...|emulate off`, `viewport <WxH>|viewport off`, `record on|off`, `highlight <selector>|highlight off`,
+`cursor on|off`.
 While any are on in the selected tab, the prompt shows them: `(watch network:off routes:2) pw>`. `modes`
 lists them for every tab, and `modes off` turns them all off.
 
-`tab`, `watch`, `capture`, `route`, `network`, `emulate` and `modes` on their own show their state and what
-you can run next.
+`tab`, `watch`, `capture`, `route`, `network`, `emulate`, `viewport`, `record`, `highlight`, `cursor` and `modes` on
+their own show their state and what you can run next.
 
 ## How it differs from playwright-cli
 
@@ -131,8 +135,9 @@ pw-repl differs, it is because someone else may be using the browser:
 - `watch` records what a person does as steps with their requests, not as generated code.
 
 For a browser of its own per agent, isolated sessions, traces, video, PDFs or generated test code,
-playwright-cli is the better fit. `record on` records a tab's page, without the pointer; for the shared
-browser as you see it, use a screen recorder (`ffmpeg -f x11grab`, macOS `screencapture -v`, OBS).
+playwright-cli is the better fit. `record on` records a tab's page, and `cursor on` draws a pointer where the
+agent acts; for the shared browser as you see it, use a screen recorder (`ffmpeg -f x11grab`, macOS
+`screencapture -v`, OBS).
 
 ## From scripts and agents
 

package/lib/commands.js

@@ -6,6 +6,7 @@ const HELP = require('./help');
 const { tabState } = require('./tabstate');
 const { commands: dialogsCommands, ensureDialogHandler } = require('./dialogs');
 const { commands: elementsCommands } = require('./elements');
+const { commands: cursorCommands } = require('./cursor');
 const { commands: requestlogCommands, ensureRecentLog } = require('./requestlog');
 const { commands: routesCommands, routesFor } = require('./routes');
 const { commands: networkCommands } = require('./network');
@@ -21,6 +22,7 @@ const { commands: modesCommands, allModesOff } = require('./modes');
 const commands = {
   ...tabsCommands,
   ...elementsCommands,
+  ...cursorCommands,
   ...inspectCommands,
   ...watchCommands,
   ...requestlogCommands,
@@ -113,6 +115,7 @@ function complete(line) {
     if (command === 'watch') return match(['on', 'off', 'new', '--all']);
     if (command === 'capture') return match(['on', 'off']);
     if (command === 'record') return match(['on', 'off']);
+    if (command === 'cursor') return match(['on', 'off']);
     if (command === 'viewport') return match(['off']);
     if (command === 'route') return match(['off']);
   }

package/lib/cursor.js

@@ -0,0 +1,302 @@
+// A pointer drawn over the page, for videos: the REPL's input moves no pointer on screen. With it on,
+// the commands that act on an element or at a point glide it there first and then act, and a
+// click shows where it landed. It shows the REPL's own input only: the user's pointer is theirs.
+// The REPL moves it, not the page's own mouse events, which the user's mouse fires as well.
+const { state, onShutdown } = require('./state');
+const out = require('./output');
+const { tabState } = require('./tabstate');
+const { unquote, toSelector } = require('./syntax');
+const { recordingOf, stepTarget, cursorStep } = require('./record');
+
+// Scaled with the distance, so a short hop is quick and a long one does not drag. Slower while the tab
+// records, so a video (and a zoom that follows the cursor) can be followed.
+const DWELL_MS = 300;
+const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
+
+const glideTime = (distance, recording) => {
+  if (distance < 2) return 0;
+  const [min, max, base, perPixel] = recording ? [300, 1000, 200, 0.8] : [150, 600, 100, 0.5];
+  return Math.round(Math.min(max, Math.max(min, base + distance * perPixel)));
+};
+
+// Runs in the page. Its own element under <html>, not <body>, which frameworks replace, with a closed
+// shadow root, so the page's selectors, the snapshot and text do not find what is in it. It takes no
+// pointer events, so what is under it is clicked, and it is built with the DOM and inline styles,
+// which a page's Trusted Types or style-src rules do not block. It is shown as a popover, in the top
+// layer, above the page's own dialogs, as Playwright's highlight is. Kept on the document under a
+// symbol, so each call finds it again, and one the page removed is drawn again.
+function pageCursor({ x, y, action, ms, count, fade, box, items, index }) {
+  const KEY = Symbol.for('pw-repl-cursor');
+  // Important on the element in the page, whose own rules could reach it; inside, the page's cannot,
+  // and an important transform would win over the glide's animation.
+  const set = (el, styles, important = '') => { for (const [name, value] of Object.entries(styles)) el.style.setProperty(name, value, important); };
+  const place = (el, px, py) => set(el, { transform: `translate(${px}px, ${py}px)` });
+  let cursor = document[KEY];
+  if (action === 'remove') {
+    if (!cursor) return;
+    delete document[KEY];
+    if (!fade || !cursor.host.isConnected) { cursor.host.remove(); return; }
+    // Waited on, as a glide is, but not past its time in a tab that draws no frames.
+    const fading = cursor.arrow.animate([{ opacity: 1 }, { opacity: 0 }], { duration: 200, fill: 'forwards' });
+    return Promise.race([fading.finished, new Promise(resolve => setTimeout(resolve, 500))]).then(() => cursor.host.remove(), () => cursor.host.remove());
+  }
+  if (!cursor || !cursor.host.isConnected) {
+    const host = document.createElement('pw-repl-cursor');
+    host.setAttribute('aria-hidden', 'true');
+    host.setAttribute('popover', 'manual');
+    set(host, {
+      display: 'block', position: 'fixed', inset: '0', width: '100%', height: '100%', 'max-width': 'none',
+      'max-height': 'none', margin: '0', padding: '0', border: 'none', background: 'transparent',
+      overflow: 'visible', 'pointer-events': 'none', 'z-index': '2147483647',
+    }, 'important');
+    const root = host.attachShadow({ mode: 'closed' });
+    const NS = 'http://www.w3.org/2000/svg';
+    const arrow = document.createElementNS(NS, 'svg');
+    arrow.setAttribute('width', '20');
+    arrow.setAttribute('height', '28');
+    arrow.setAttribute('viewBox', '0 0 15 21');
+    set(arrow, { position: 'absolute', left: '0', top: '0', overflow: 'visible', filter: 'drop-shadow(0 1px 1.5px rgba(0,0,0,.4))' });
+    const shape = document.createElementNS(NS, 'path');
+    // The tip at 0,0 is the point.
+    shape.setAttribute('d', 'M0 0 L0 17 L4.2 13.2 L7 19.6 L9.8 18.4 L7.1 12.1 L12.6 12.1 Z');
+    shape.setAttribute('fill', '#111');
+    shape.setAttribute('stroke', '#fff');
+    shape.setAttribute('stroke-width', '1.4');
+    shape.setAttribute('stroke-linejoin', 'round');
+    arrow.append(shape);
+    root.append(arrow);
+    document.documentElement.append(host);
+    cursor = document[KEY] = { host, root, arrow, x, y };
+    place(arrow, x, y);
+    // Faded in when turned on; drawn again on a new page, it is there at once, as it was.
+    if (fade) arrow.animate([{ opacity: 0 }, { opacity: 1 }], { duration: 200 });
+  }
+  // Shown again, so it is above a dialog the page opened since.
+  try { cursor.host.hidePopover(); cursor.host.showPopover(); } catch {}
+  if (action === 'move') {
+    const from = `translate(${cursor.x}px, ${cursor.y}px)`;
+    const to = `translate(${x}px, ${y}px)`;
+    cursor.x = x;
+    cursor.y = y;
+    place(cursor.arrow, x, y);
+    if (!ms) return;
+    // Smoothstep, exactly (x = t with these control points, and y = 3t² - 2t³): the easing a zoom that
+    // follows the cursor uses, so it knows where the cursor is on every frame.
+    const glide = cursor.arrow.animate([{ transform: from }, { transform: to }], { duration: ms, easing: 'cubic-bezier(0.3333, 0, 0.6667, 1)' });
+    // A hidden tab draws no frames, and its animation may never finish: not waited on past its time.
+    return Promise.race([glide.finished, new Promise(resolve => setTimeout(resolve, ms + 300))]).then(() => {}, () => {});
+  }
+  // A select's list, drawn under it: a native one opens outside the page, where no recording sees it.
+  if (action === 'pickerOpen') {
+    cursor.picker?.remove();
+    const list = document.createElement('div');
+    const rowHeight = 28;
+    set(list, {
+      position: 'absolute', left: `${box.x}px`, top: `${box.y + box.height + 2}px`, 'min-width': `${Math.max(box.width, 120)}px`,
+      background: '#fff', border: '1px solid #9aa0a6', 'border-radius': '4px', 'box-shadow': '0 4px 14px rgba(0,0,0,.25)',
+      font: '15px system-ui, sans-serif', color: '#202124', padding: '4px 0', 'box-sizing': 'border-box', overflow: 'hidden',
+    });
+    const rows = items.slice(0, 10).map(text => {
+      const row = document.createElement('div');
+      row.textContent = text;
+      set(row, { height: `${rowHeight}px`, 'line-height': `${rowHeight}px`, padding: '0 12px', 'white-space': 'nowrap' });
+      list.append(row);
+      return row;
+    });
+    cursor.picker = list;
+    cursor.pickerRows = rows;
+    cursor.root.insertBefore(list, cursor.arrow);
+    const mark = i => rows.forEach((row, j) => set(row, { background: j === i ? '#1a73e8' : 'transparent', color: j === i ? '#fff' : '#202124' }));
+    cursor.pickerMark = mark;
+    mark(index);
+    list.animate([{ opacity: 0, transform: 'translateY(-4px)' }, { opacity: 1, transform: 'none' }], { duration: 150 });
+    // Where the chosen row will be, for the cursor to go to.
+    return { rowY: box.y + box.height + 2 + 4 + rowHeight / 2, rowHeight };
+  }
+  if (action === 'pickerChoose') {
+    cursor.pickerMark?.(index);
+    return;
+  }
+  if (action === 'pickerClose') {
+    const list = cursor.picker;
+    if (!list) return;
+    cursor.picker = null;
+    return Promise.race([list.animate([{ opacity: 1 }, { opacity: 0 }], { duration: 150, fill: 'forwards' }).finished, new Promise(resolve => setTimeout(resolve, 400))])
+      .then(() => list.remove(), () => list.remove());
+  }
+  if (action === 'click') {
+    // A ring that grows and fades where the click landed, one for each click of a double-click.
+    for (let i = 0; i < count; i += 1) {
+      const ring = document.createElement('div');
+      set(ring, {
+        position: 'absolute', left: `${x - 14}px`, top: `${y - 14}px`, width: '28px', height: '28px', 'box-sizing': 'border-box',
+        'border-radius': '50%', border: '2px solid rgba(230, 40, 40, .9)', background: 'rgba(230, 40, 40, .3)', opacity: '0',
+      });
+      cursor.root.insertBefore(ring, cursor.arrow);
+      ring.animate([{ transform: 'scale(.3)', opacity: 1 }, { transform: 'scale(1.5)', opacity: 0 }], { duration: 450, delay: i * 140, easing: 'ease-out' })
+        .finished.then(() => ring.remove(), () => ring.remove());
+    }
+  }
+}
+
+function cursorOf(p) {
+  return tabState(p).cursor || null;
+}
+
+// A new document drops the cursor: it is drawn again where it was.
+function keepAcrossPages(p) {
+  if (tabState(p).cursorKept) return;
+  tabState(p).cursorKept = true;
+  p.on('domcontentloaded', () => {
+    const cursor = cursorOf(p);
+    if (cursor) p.evaluate(pageCursor, { x: cursor.x, y: cursor.y, action: 'draw' }).catch(() => {});
+  });
+}
+
+// Glides the tab's cursor to a point, if it is on, and waits for it to get there.
+async function glideTo(p, x, y) {
+  const cursor = cursorOf(p);
+  if (!cursor) return;
+  const ms = glideTime(Math.hypot(x - cursor.x, y - cursor.y), !!recordingOf(p));
+  // A recording notes where the pointer went and when, for a zoom to follow it.
+  if (ms) cursorStep(p, `(cursor from ${cursor.x} ${cursor.y})`, { x, y }, ms);
+  cursor.x = x;
+  cursor.y = y;
+  await p.evaluate(pageCursor, { x, y, action: 'move', ms }).catch(() => {});
+}
+
+// Whether a command should find its element first: to glide the cursor there, or to note it in a
+// recording's steps.
+function aiming(p) {
+  return !!(cursorOf(p) || recordingOf(p));
+}
+
+// Finds the centre of an element (a locator or a handle), where Playwright clicks, after scrolling it
+// into view as Playwright would, and glides the cursor there if it is on: the pointer is there before
+// the click, not after. A recording notes the element in its steps.
+// It waits for a locator's element to be there, as the command would, and throws Playwright's timeout
+// if none comes, for the command to report as its own: the command does not wait all over again. One
+// there but hidden (a file input under its button) is not glided to, and the command says why.
+async function aimAt(p, target, timeout) {
+  if (!aiming(p)) return null;
+  if (target.waitFor) await target.waitFor({ state: 'attached', timeout });
+  try {
+    if (!await target.isVisible()) return null;
+    await target.scrollIntoViewIfNeeded({ timeout: 2000 });
+    const box = await target.boundingBox({ timeout: 1000 });
+    if (!box) return null;
+    const point = { x: Math.round(box.x + box.width / 2), y: Math.round(box.y + box.height / 2) };
+    stepTarget(p, box, point);
+    await glideTo(p, point.x, point.y);
+    // While recording, a moment on the element before acting on it, for the viewer to see where it is.
+    if (cursorOf(p) && recordingOf(p)) await sleep(DWELL_MS);
+    return point;
+  } catch {
+    return null;
+  }
+}
+
+// Shows a click where the cursor is; the input has landed by then. A click that loads a new page
+// may take the ring with it.
+async function showClick(p, count = 1) {
+  const cursor = cursorOf(p);
+  if (!cursor) return;
+  await p.evaluate(pageCursor, { x: cursor.x, y: cursor.y, action: 'click', count }).catch(() => {});
+}
+
+// select with the cursor on: a native select's list opens outside the page, where no recording (or
+// headless Chrome) shows it, so its list is drawn over the page instead. It opens under the select with
+// the current option marked, the mark moves to the option to be chosen (by value or label, as
+// Playwright's selectOption matches it), the cursor clicks it and the choice is made (act), and the list
+// closes. Drawn only: the page gets the same input as without the cursor.
+async function showSelect(p, target, value, act) {
+  const cursor = cursorOf(p);
+  const box = cursor && await target.boundingBox().catch(() => null);
+  // Not drawn for a disabled select or option, which selectOption refuses: the video would show a choice
+  // that was never made.
+  const before = box && await target.evaluate((el, v) => {
+    if (el.tagName !== 'SELECT' || el.multiple || el.size > 1 || el.disabled) return null;
+    const chosen = [...el.options].findIndex(o => o.value === v || o.label === v);
+    if (chosen >= 0 && (el.options[chosen].disabled || el.options[chosen].parentElement.disabled)) return null;
+    return { items: [...el.options].map(o => o.label || o.text), index: el.selectedIndex, chosen };
+  }, value).catch(() => null);
+  if (!before || before.chosen < 0 || before.chosen >= 10) return act();
+  const pace = recordingOf(p) ? 1 : 0.5;
+  await showClick(p);
+  const opened = await p.evaluate(pageCursor, { action: 'pickerOpen', box, items: before.items, index: before.index }).catch(() => null);
+  if (!opened) return act();
+  await sleep(350 * pace);
+  await glideTo(p, Math.round(cursor.x), Math.round(opened.rowY + before.chosen * opened.rowHeight));
+  await p.evaluate(pageCursor, { action: 'pickerChoose', index: before.chosen }).catch(() => {});
+  await showClick(p);
+  try {
+    return await act();
+  } finally {
+    await sleep(250 * pace);
+    await p.evaluate(pageCursor, { action: 'pickerClose' }).catch(() => {});
+  }
+}
+
+async function cursorOff(p) {
+  if (!cursorOf(p)) return false;
+  const { x, y } = cursorOf(p);
+  delete tabState(p).cursor;
+  cursorStep(p, '(cursor off)', { x, y }, 0);
+  await p.evaluate(pageCursor, { action: 'remove', fade: true }).catch(() => {});
+  return true;
+}
+
+onShutdown(async () => {
+  if (!state.browser || state.connectionLost) return;
+  const pages = state.browser.contexts().flatMap(c => c.pages()).filter(p => !p.isClosed());
+  await Promise.all(pages.map(p => cursorOff(p).catch(() => {})));
+});
+
+const commands = {
+  async cursor(args) {
+    const arg = (args || '').trim();
+    const p = state.page;
+    const cursor = cursorOf(p);
+    if (!arg) {
+      out.log(cursor
+        ? `The cursor is on in the selected tab, at ${cursor.x}, ${cursor.y}${cursor.owner ? ` (by ${cursor.owner})` : ''}; cursor off hides it`
+        : 'The cursor is off in the selected tab; cursor on shows it');
+      return;
+    }
+    if (arg === 'off') {
+      out.log(await cursorOff(p) ? 'The cursor is off' : 'The cursor was not on in the selected tab');
+      return;
+    }
+    const usage = 'Usage: cursor [on [<selector> | <x> <y>] | off]';
+    const on = /^on(?:\s+([\s\S]+))?$/.exec(arg);
+    if (!on) throw new Error(usage);
+    if (cursor) { out.log(`The cursor is already on in the selected tab, at ${cursor.x}, ${cursor.y}; mousemove or a command on an element moves it`); return; }
+    // Where it is asked to appear (a video's first element, say), or where the mouse last went, if the
+    // REPL knows; otherwise the middle of the viewport.
+    const where = (on[1] || '').trim();
+    let at;
+    if (/^-?\d+(?:\.\d+)?\s+-?\d+(?:\.\d+)?$/.test(where)) {
+      const [x, y] = where.split(/\s+/).map(n => Math.round(Number(n)));
+      at = { x, y };
+    } else if (where) {
+      const target = p.locator(toSelector(unquote(where))).first();
+      try {
+        await target.scrollIntoViewIfNeeded({ timeout: 5000 });
+      } catch {
+        throw new Error(`No element matches ${where} that can be shown (waited 5s); the cursor is still off`);
+      }
+      const box = await target.boundingBox();
+      if (!box) throw new Error(`${where} is not on the page to show the cursor at; the cursor is still off`);
+      at = { x: Math.round(box.x + box.width / 2), y: Math.round(box.y + box.height / 2) };
+    } else {
+      at = tabState(p).mouseAt || await p.evaluate(() => ({ x: Math.round(innerWidth / 2), y: Math.round(innerHeight / 2) }));
+    }
+    await p.evaluate(pageCursor, { x: at.x, y: at.y, action: 'draw', fade: true });
+    tabState(p).cursor = { owner: state.client, x: at.x, y: at.y };
+    cursorStep(p, '(cursor on)', at, 0);
+    keepAcrossPages(p);
+    out.log(`The cursor is on, at ${at.x}, ${at.y}: commands on elements and the mouse glide it there first, and clicks show where they land; cursor off hides it`);
+  },
+};
+
+module.exports = { commands, cursorOf, cursorOff, aiming, aimAt, glideTo, showClick, showSelect };

package/lib/elements.js

@@ -7,6 +7,8 @@ const { takeOptions } = require('./cli-names');
 const { unquote, toSelector, keyName, splitSelector, REF } = require('./syntax');
 const { numbers, hints } = require('./util');
 const { tabState } = require('./tabstate');
+const { cursorOf, aiming, aimAt, glideTo, showClick, showSelect } = require('./cursor');
+const { recordingOf, stepTarget } = require('./record');
 
 // The type a page sees for a file it is given, from its extension.
 const MIME_TYPES = {
@@ -177,19 +179,25 @@ function mouseButton(args, name) {
 // click, dblclick and hover: on the match chooseMatch picks, or as Playwright does.
 // A page that renders the chosen element again (as React lists do) replaces it before it is
 // acted on; it is chosen again then, as Playwright finds a locator's element again.
-async function pointerAction(selector, act, options = {}) {
+async function pointerAction(selector, act, options = {}, clicks = 0) {
   const TIMEOUT = 5000;
   const deadline = Date.now() + TIMEOUT;
   const left = () => Math.max(1000, deadline - Date.now());
   for (;;) {
     const chosen = await onElement(selector, () => chooseMatch(selector, left()));
     if (!chosen) {
-      // Only counted so far, so the whole time is Playwright's, as it was before.
-      await onElement(selector, () => act(state.page, selector, { ...options, timeout: TIMEOUT }));
+      // Only counted so far, so the whole time is Playwright's, as it was before; the cursor's glide, if it
+      // is on, waits for the element as part of it.
+      const glided = Date.now();
+      const point = await onElement(selector, () => aimAt(state.page, state.page.locator(selector).first(), TIMEOUT));
+      await onElement(selector, () => act(state.page, selector, { ...options, timeout: Math.max(1000, TIMEOUT - (Date.now() - glided)) }));
+      await landed(point, clicks);
       return '';
     }
     try {
+      const point = await aimAt(state.page, chosen.handle, left());
       await onElement(selector, () => act(chosen.handle, null, { ...options, timeout: left() }));
+      await landed(point, clicks);
       return chosen.position === 1 ? '' : ` (match ${chosen.position} of ${chosen.count}; the ones before it are hidden or covered)`;
     } catch (error) {
       if (!/not attached/i.test(error.message) || Date.now() >= deadline) throw error;
@@ -199,6 +207,63 @@ async function pointerAction(selector, act, options = {}) {
   }
 }
 
+// After a pointer action: Playwright's mouse is where it acted, at the centre the cursor glided to,
+// or somewhere the REPL does not know without the cursor. A click shows where it landed.
+async function landed(point, clicks) {
+  tabState(state.page).mouseAt = point;
+  if (clicks) await showClick(state.page, clicks);
+}
+
+// Finds the element a command without a pointer acts on (fill, select...), for the cursor to glide
+// to, as someone would go to it first, and for a recording's steps; the mouse itself stays where it is. What is left of the command's
+// 5s is its timeout, so the glide does not make it wait longer for an element that never comes.
+async function aimFor(selector) {
+  if (!aiming(state.page)) return 5000;
+  const started = Date.now();
+  await onElement(selector, () => aimAt(state.page, state.page.locator(selector).first(), 5000));
+  return Math.max(1000, 5000 - (Date.now() - started));
+}
+
+// While the tab records, a wheel turn goes in small steps, eased, over 400ms to 1s by its length, so the
+// video scrolls rather than jumps; the same distance in all.
+async function smoothWheel(dx, dy) {
+  const ms = Math.min(1000, Math.max(400, Math.hypot(dx, dy) * 0.8));
+  const steps = Math.max(1, Math.round(ms / 25));
+  const ease = t => t * t * (3 - 2 * t);
+  let sentX = 0;
+  let sentY = 0;
+  for (let i = 1; i <= steps; i += 1) {
+    const x = Math.round(dx * ease(i / steps));
+    const y = Math.round(dy * ease(i / steps));
+    if (x !== sentX || y !== sentY) await state.page.mouse.wheel(x - sentX, y - sentY);
+    sentX = x;
+    sentY = y;
+    await new Promise(resolve => setTimeout(resolve, ms / steps));
+  }
+}
+
+// While the tab records, text is typed at a pace a viewer can follow, steady rather than human-like.
+// Otherwise at once, as before.
+function typingPace() {
+  return recordingOf(state.page) ? 60 : 0;
+}
+
+// check, uncheck and upload's button click it at its centre, where aimFor put the cursor.
+async function clickedHere() {
+  const cursor = cursorOf(state.page);
+  if (!cursor) return;
+  tabState(state.page).mouseAt = { x: cursor.x, y: cursor.y };
+  await showClick(state.page);
+}
+
+// The mouse commands act where the mouse is: the cursor goes back there first, if it went elsewhere.
+async function glideToMouse() {
+  const at = tabState(state.page).mouseAt;
+  if (!at) return;
+  stepTarget(state.page, null, at);
+  await glideTo(state.page, at.x, at.y);
+}
+
 // Highlights: Playwright's own overlay, as playwright-cli's highlight draws it. Each is a mode of its
 // tab, so modes shows it and modes off hides it, and none is left on the page when the REPL stops.
 function highlightsOf(p) {
@@ -230,13 +295,13 @@ onShutdown(async () => {
 const commands = {
   async click(args) {
     const { selector, shown, options } = clickArgs(args, 'click <selector> [left|right|middle] [--modifiers=<key>[,<key>]]');
-    const which = await pointerAction(selector, (on, sel, o) => (sel ? on.click(sel, o) : on.click(o)), options);
+    const which = await pointerAction(selector, (on, sel, o) => (sel ? on.click(sel, o) : on.click(o)), options, 1);
     out.log(`Clicked: ${shown}${which}`);
   },
 
   async dblclick(args) {
     const { selector, shown, options } = clickArgs(args, 'dblclick <selector> [left|right|middle] [--modifiers=<key>[,<key>]]');
-    const which = await pointerAction(selector, (on, sel, o) => (sel ? on.dblclick(sel, o) : on.dblclick(o)), options);
+    const which = await pointerAction(selector, (on, sel, o) => (sel ? on.dblclick(sel, o) : on.dblclick(o)), options, 2);
     out.log(`Double-clicked: ${shown}${which}`);
   },
 
@@ -286,18 +351,24 @@ const commands = {
   async mousemove(args) {
     const [x, y, extra] = numbers(args, 'mousemove <x> <y>');
     if (y === undefined || extra !== undefined) throw new Error('Usage: mousemove <x> <y>, in CSS pixels from the top left of the viewport');
+    stepTarget(state.page, null, { x, y });
+    await glideTo(state.page, x, y);
     await state.page.mouse.move(x, y);
+    tabState(state.page).mouseAt = { x, y };
     out.log(`Moved the mouse to ${x}, ${y}`);
   },
 
   async mousedown(args) {
     const button = mouseButton(args, 'mousedown');
+    await glideToMouse();
     await state.page.mouse.down({ button });
+    await showClick(state.page);
     out.log(`Pressed the ${button} button`);
   },
 
   async mouseup(args) {
     const button = mouseButton(args, 'mouseup');
+    await glideToMouse();
     await state.page.mouse.up({ button });
     out.log(`Released the ${button} button`);
   },
@@ -306,20 +377,27 @@ const commands = {
   async mousewheel(args) {
     const [dx, dy, extra] = numbers(args, 'mousewheel <dx> <dy>');
     if (dy === undefined || extra !== undefined) throw new Error('Usage: mousewheel <dx> <dy>, e.g. mousewheel 0 300 scrolls down (negative dy up)');
-    await state.page.mouse.wheel(dx, dy);
+    await glideToMouse();
+    if (recordingOf(state.page)) await smoothWheel(dx, dy);
+    else await state.page.mouse.wheel(dx, dy);
     out.log(`Turned the wheel by ${dx}, ${dy}`);
   },
 
   async fill(line) {
     const { args, submit } = submitFlag(line);
     const { selector, value, shown } = selectorAndValue(args, 'fill <selector> <value> [--submit]', 'fill #name Ada Lovelace');
-    await onElement(selector, () => state.page.fill(selector, value, { timeout: 5000 }));
+    const timeout = await aimFor(selector);
+    await onElement(selector, () => state.page.fill(selector, value, { timeout }));
     if (submit) await state.page.press(selector, 'Enter', { timeout: 5000 });
     out.log(`Filled${submit ? ' and submitted' : ''}: ${shown}`);
   },
 
   async type(line) {
-    const { args, submit } = submitFlag(line);
+    // --delay=<ms> goes first, so text with --delay in it is still text.
+    const paced = /^\s*--delay(?:=|\s+)(\S*)(?:\s+|$)/.exec(line || '');
+    if (paced && !/^\d+$/.test(paced[1])) throw new Error('Usage: type --delay=<ms> [<selector>] <text>, the pause between keys, e.g. type --delay=80 #search garden hose');
+    const { args, submit } = submitFlag(paced ? line.slice(paced[0].length) : line);
+    const delay = () => (paced ? Number(paced[1]) : typingPace());
     // One word, or one quoted, is typed into the focused element, as playwright-cli's type does.
     const sole = splitSelector(args);
     if (sole && sole.rest === undefined) {
@@ -334,12 +412,13 @@ const commands = {
       const usage = 'type <selector> <text> types into an element, e.g. type #search garden hose';
       if (!focused) throw new Error(`Nothing on the page has focus: ${usage}`);
       if (!focused.editable) throw new Error(`The focused element, ${focused.name}, takes no text: ${usage}`);
-      await state.page.keyboard.type(sole.word);
+      await state.page.keyboard.type(sole.word, { delay: delay() });
       if (submit) await state.page.keyboard.press('Enter');
       out.log(`Typed into the focused ${focused.name}${submit ? ', and submitted' : ''}`);
       return;
     }
     const { selector, value, shown } = selectorAndValue(args, 'type [<selector>] <text> [--submit]', 'type #search garden hose');
+    const timeout = await aimFor(selector);
     // Typed after what the field already holds, as someone clicking into it
     // at the end would; focusing it alone leaves the caret at the start.
     await onElement(selector, () => state.page.locator(selector).first().evaluate(el => {
@@ -353,8 +432,8 @@ const commands = {
         getSelection().removeAllRanges();
         getSelection().addRange(range);
       }
-    }, null, { timeout: 5000 }));
-    await state.page.keyboard.type(value);
+    }, null, { timeout }));
+    await state.page.keyboard.type(value, { delay: delay() });
     if (submit) await state.page.keyboard.press('Enter');
     out.log(`Typed into${submit ? ' and submitted' : ''}: ${shown}`);
   },
@@ -368,25 +447,32 @@ const commands = {
       return;
     }
     const key = keyName(unquote(parsed.rest));
-    await onElement(parsed.selector, () => state.page.press(parsed.selector, key, { timeout: 5000 }));
+    const timeout = await aimFor(parsed.selector);
+    await onElement(parsed.selector, () => state.page.press(parsed.selector, key, { timeout }));
     out.log(`Pressed ${key} on ${parsed.word}`);
   },
 
   async select(args) {
     const { selector, value, shown } = selectorAndValue(args, 'select <selector> <value>', 'select #country Canada');
-    await onElement(selector, () => state.page.selectOption(selector, value, { timeout: 5000 }));
+    const timeout = await aimFor(selector);
+    const target = state.page.locator(selector).first();
+    await onElement(selector, () => showSelect(state.page, target, value, () => state.page.selectOption(selector, value, { timeout })));
     out.log(`Selected "${value}" in ${shown}`);
   },
 
   async check(args) {
     const selector = soleSelector(args, 'check <selector>');
-    await onElement(selector, () => state.page.check(selector, { timeout: 5000 }));
+    const timeout = await aimFor(selector);
+    await onElement(selector, () => state.page.check(selector, { timeout }));
+    await clickedHere();
     out.log(`Checked: ${args.trim()}`);
   },
 
   async uncheck(args) {
     const selector = soleSelector(args, 'uncheck <selector>');
-    await onElement(selector, () => state.page.uncheck(selector, { timeout: 5000 }));
+    const timeout = await aimFor(selector);
+    await onElement(selector, () => state.page.uncheck(selector, { timeout }));
+    await clickedHere();
     out.log(`Unchecked: ${args.trim()}`);
   },
 
@@ -408,6 +494,7 @@ const commands = {
       return { name: path.basename(file), mimeType: mimeType(file), buffer };
     });
     const { selector } = parsed;
+    await aimFor(selector);
     await onElement(selector, async () => {
       try {
         await state.page.setInputFiles(selector, payloads, { timeout: 5000 });
@@ -416,6 +503,7 @@ const commands = {
         // A button that opens the file picker itself: click it, and answer the picker.
         const [chooser] = await Promise.all([state.page.waitForEvent('filechooser', { timeout: 5000 }), state.page.click(selector, { timeout: 5000 })]);
         await chooser.setFiles(payloads);
+        await clickedHere();
       }
     });
     out.log(`Chose ${files.length === 1 ? 'a file' : `${files.length} files`} in ${parsed.word}: ${files.join(', ')}`);

package/lib/help.js

@@ -74,7 +74,7 @@ fill for selectors with spaces. mousemove and the other mouse commands act at a
   },
   inspect: {
     intro: `Output is capped; --all right after the command, or at the end, shows everything (text --all body).`,
-    commands: ['snapshot', 'watch', 'text', 'html', 'attrs', 'listeners', 'count', 'visible', 'links', 'inputs', 'screenshot', 'highlight', 'record', 'viewport', 'emulate', 'wait', 'sleep'],
+    commands: ['snapshot', 'watch', 'text', 'html', 'attrs', 'listeners', 'count', 'visible', 'links', 'inputs', 'screenshot', 'highlight', 'cursor', 'record', 'viewport', 'emulate', 'wait', 'sleep'],
   },
   network: {
     intro: `requests and console record all the time; a capture records only while it runs.`,
@@ -196,10 +196,22 @@ Examples: fill #name Ada Lovelace, fill "text=Your name" Ada, fill e7 Ada`,
     usage: 'type [<selector>] <text> [--submit]',
     summary: 'type key by key after what the field holds; text as for fill',
     detail: `With one word, or one quoted, it is typed into the element that has focus: type "garden hose".
---submit at the end presses Enter afterwards.`,
+--submit at the end presses Enter afterwards.
+
+Keys are sent all at once. While the tab is recording (help record), they are 60ms apart, so the video
+shows the text being typed. type --delay=<ms> <selector> <text> sets the gap between keys yourself. The
+whole text takes its length times the gap, and send waits 20s by default: about 300 characters at 60ms
+need a longer send -t, or a shorter --delay. It is a gap between keys, not a wait before starting like
+screenshot's --delay. A page that reacts to each key, such as search as you type, may react to each one.`,
   },
   press: { usage: 'press <key> | press <selector> <key>', summary: 'press a key (Enter, Escape, Control+A; any case), maybe on an element' },
-  select: { usage: 'select <selector> <value>', summary: 'choose an option in a select (the value is the rest of the line)' },
+  select: {
+    usage: 'select <selector> <value>',
+    summary: 'choose an option in a select (the value is the rest of the line)',
+    detail: `The value is an option's value or its label. With the cursor on (help cursor), select also draws the
+select's list opening and the cursor choosing the option, since a native select's list opens outside
+the page, where no screenshot or recording sees it; the page gets the same input either way.`,
+  },
   check: { usage: 'check <selector>', summary: 'check a checkbox' },
   uncheck: { usage: 'uncheck <selector>', summary: 'uncheck a checkbox' },
   upload: {
@@ -322,6 +334,28 @@ is at CSS pixel size, one image pixel per CSS pixel. While emulate mobile is on,
 phone shows: a page laid out wider than the phone (no viewport meta tag) comes out shrunk to fit.
 
 --delay counts down out loud first (maximum 60s), so someone can hold a hover or open a menu.`,
+  },
+  cursor: {
+    usage: 'cursor [on [<selector> | <x> <y>] | off]',
+    summary: 'draw a pointer where the REPL acts, for videos',
+    detail: `The REPL's clicks and typing don't move a pointer on screen, so a recording shows things
+happening with nothing pointing at them. cursor on draws a pointer in the selected tab. After that, these
+commands move it smoothly to their element or point before they act: click, dblclick, hover, fill, type
+<selector>, press <selector>, select, check, uncheck, upload and mousemove. Each move takes 150 to 600ms,
+depending on the distance (300 to 1000ms while the tab is recording), and the command waits for it. Clicks (click, dblclick, check, uncheck,
+mousedown, and upload when it clicks a button) show a ring where they land.
+
+It shows only the REPL's input, from every client, never the user's own mouse. Only the drawing moves: no
+mouse events are sent along the way, so the page gets the same input as with the cursor off. fill, select
+and the like move the drawing to their element but leave the real mouse where it was; mousedown, mouseup
+and mousewheel act at the real mouse, so the pointer moves back there first.
+
+cursor on fades it in where the mouse last went (or the middle), cursor on <selector> or <x> <y> fades it in
+there instead (a video's first element, say), cursor off fades it out, and cursor alone says whether it
+is on and where. It is a
+mode of its tab: modes lists it and modes off turns it off. It stays on across page loads, at the same
+spot, and shows in screenshots and recordings. Selectors, snapshot and text don't see it, it takes no
+clicks, and it points at elements inside frames too.`,
   },
   record: {
     usage: 'record [on [file] [seconds] | off [file]]',
@@ -341,11 +375,20 @@ it stops by itself after them (at most 3600, which is also the default); command
 It brings the tab to the front of its window first, as screenshot does. It records the page only,
 from inside the browser, at the viewport's size in CSS pixels, scrollbars included, made even (an
 odd width or height loses a pixel), a phone's too under emulate mobile: a Pixel 7's is 412 wide, less
-sharp than the phone draws it. It has no pointer and no browser toolbars,
+sharp than the phone draws it. It has no pointer (cursor on draws one) and no browser toolbars,
 and a tab behind another, hidden or headless keeps recording. Set the size first with viewport
 (1280x720 fills a 16:9 player); a page that changes size while recording is fitted into the first size.
 Chrome sends a frame as the page draws, and a still page keeps its real length.
 
+While a tab records, type sends keys at a pace the video can show (help type), and mousewheel scrolls in
+small eased steps over up to 1s rather than at once; fill still sets the whole
+value at once, so use type for text the video should show being typed. record off also saves the steps
+next to the video, as clip.steps.txt for clip.webm: one line for each command run on the tab while it
+recorded, with when it began and ended in seconds into the video, the element or point it acted on in the
+video's pixels, and the command with what it acts on, never what it types or chooses, the code it runs or
+the URL it goes to. A (page) line marks each new page, loaded or routed to by a single-page app. They
+are for editing the video afterwards, such as zooming in on a step. After the page changes size, the steps leave out where they acted.
+
 It needs ffmpeg: $PW_FFMPEG, and only it when it is set (the REPL's, not send's); otherwise Playwright's own (npx playwright-core install ffmpeg, in $PLAYWRIGHT_BROWSERS_PATH or ~/.cache/ms-playwright), then one on the PATH. Playwright's writes only WebM; an .mp4 needs a system ffmpeg, with H.264. record on fails at once, recording nothing, without an ffmpeg that can write its format. If ffmpeg exits while recording, the recording ends then, the pane says what was saved, and the next record off fails with it. ffmpeg writes in blocks, so what it has not written yet is lost: the last seconds, or at first all of it. WebM plays in browsers,
 GitHub, GitLab and Slack; MP4 plays everywhere. It takes about 15 MB a minute.
 
@@ -520,10 +563,10 @@ Browser.close and Target.closeTarget are refused, and so is Page.handleJavaScrip
   modes: {
     usage: 'modes [off [--mine]]',
     summary: 'the modes on in every tab; modes off turns them all off',
-    detail: `The modes are watch, network off or slow, route, capture, record, emulate, viewport and highlight.
+    detail: `The modes are watch, network off or slow, route, capture, record, emulate, viewport, highlight and cursor.
 Each is turned on and off with its own command: watch on|off, network off|slow|on, route <glob> ... |
 route off <glob>|--all, capture on|off, record on|off, emulate ... | emulate off, viewport <WxH> |
-viewport off, highlight <selector> | highlight off.
+viewport off, highlight <selector> | highlight off, cursor on|off.
 
 modes off turns off every one in every tab; a capture it stops is kept for capture to show, and a
 recording it stops is saved, as record off would. With
@@ -577,8 +620,9 @@ function renderAll() {
 const VIDEO = `There are three ways to record a video of a page, and each gives something up.
 
 record on (help record) records the selected tab's page from inside the browser: headless, hidden and
-background tabs too, with nothing to set up but ffmpeg. It has no pointer and no browser toolbars. For
-quick clips, and for an agent recording its own work.
+background tabs too, with nothing to set up but ffmpeg. It has no pointer and no browser toolbars:
+cursor on draws a pointer that shows where the REPL clicks. For quick clips, and for an agent recording
+its own work.
 
 A screen recorder on the person's machine (OBS, ffmpeg -f x11grab, macOS screencapture -v) records their
 screen as they see it, pointer included, smooth and in real time. It records the screen or a window, not

package/lib/modes.js

@@ -10,6 +10,7 @@ const { recordingOf, finishRecording } = require('./record');
 const { currentCapture, endCapture } = require('./capture');
 const { nextTabWanted, waitingText, stopWaitingForNextTab, stopWatching } = require('./watch');
 const { hideHighlights } = require('./elements');
+const { cursorOf, cursorOff } = require('./cursor');
 
 // The modes turned on in a tab, in the order the prompt shows them.
 function activeModes(p) {
@@ -25,6 +26,7 @@ function activeModes(p) {
   if (routes) modes.push(`routes:${routes}`);
   const highlights = tabState(p).highlights?.size;
   if (highlights) modes.push(`highlight:${highlights}`);
+  if (cursorOf(p)) modes.push('cursor');
   if (currentCapture()?.page === p) modes.push('capture');
   if (recordingOf(p)) modes.push('record');
   return modes;
@@ -74,6 +76,8 @@ function modeOwners(p) {
   if (viewport) owners.push([viewport.owner ?? null, 'viewport']);
   for (const [glob, route] of tabState(p).routes || []) owners.push([route.owner ?? null, `route ${glob}`]);
   for (const [selector, h] of tabState(p).highlights || []) owners.push([h.owner ?? null, `highlight ${selector}`]);
+  const cursor = cursorOf(p);
+  if (cursor) owners.push([cursor.owner ?? null, 'cursor']);
   const capture = currentCapture();
   if (capture?.page === p) owners.push([capture.owner ?? null, 'capture']);
   const recording = recordingOf(p);
@@ -110,6 +114,7 @@ async function allModesOff(mine = false) {
       if ('network' in tabState(p) && ours(tabState(p).network.owner)) { await setNetwork(p, null); done.push('network on'); }
       const hidden = await hideHighlights(p, mine);
       if (hidden) done.push(`${hidden} highlight${hidden === 1 ? '' : 's'} hidden`);
+      if (cursorOf(p) && ours(cursorOf(p).owner) && await cursorOff(p)) done.push('cursor off');
       // Before the page's size is reset, which the recording would otherwise end with.
       const recording = recordingOf(p);
       if (recording && ours(recording.owner)) done.push(`record off (${(await finishRecording(recording)).replace(/\n/g, '; ')})`);

package/lib/record.js

@@ -7,7 +7,7 @@ const { state, onShutdown } = require('./state');
 const out = require('./output');
 const { tabState } = require('./tabstate');
 const { clock, NO_TAB, STOPPED_RECORDING, ENDED_UNSEEN } = require('./util');
-const { shellWords } = require('./syntax');
+const { shellWords, splitSelector } = require('./syntax');
 
 const FPS = 25;
 const MAX_SECONDS = 3600;
@@ -142,7 +142,7 @@ async function startRecording(p, file, seconds, format) {
   const rec = {
     page: p, file, format, exe, session, owner: state.client, startedAt: Date.now(), firstFrameAt: null,
     size: null, resizedAt: null, written: 0, dropped: 0, latest: null, ff: null, timer: null, limit: null, finishing: null,
-    ffmpegError: '', starting: true,
+    ffmpegError: '', starting: true, scale: 1, steps: [], step: null, stepsFile: null,
   };
   // The size is the page's as the screen shows it, in CSS pixels (a phone's too, whose frames come in
   // its device pixels), and Chrome scales each frame to fit it. Not the first frame's: just after a
@@ -153,7 +153,8 @@ async function startRecording(p, file, seconds, format) {
     const { cssVisualViewport: vv } = await session.send('Page.getLayoutMetrics');
     const { result } = await session.send('Runtime.evaluate', { expression: '[innerWidth, innerHeight]', returnByValue: true });
     const [width, height] = result.value;
-    rec.size = { width: even(width * (vv.scale || 1)), height: even(height * (vv.scale || 1)) };
+    rec.scale = vv.scale || 1;
+    rec.size = { width: even(width * rec.scale), height: even(height * rec.scale) };
   } catch (error) {
     session.detach().catch(() => {});
     throw error;
@@ -208,6 +209,17 @@ async function startRecording(p, file, seconds, format) {
   }, seconds * 1000);
   rec.onClose = () => finishRecording(rec, TAB_CLOSED).then(text => out.notice(text, p));
   p.once('close', rec.onClose);
+  // A new page, loaded or routed to by a single-page app, is a step of its own: a zoom that follows the
+  // cursor ends there. Its URL is left out, as a step's values are.
+  rec.onNavigated = frame => {
+    if (frame !== p.mainFrame() || rec.finishing) return;
+    const t = videoTime(rec);
+    // An app that sets its URL twice in a row (push, then replace) is one new page.
+    const last = rec.steps[rec.steps.length - 1];
+    if (last?.command === '(page)' && t - last.t0 < 0.5) return;
+    rec.steps.push({ rec, t0: t, t1: t, command: '(page)', box: null, point: null, offSize: false, failed: false });
+  };
+  p.on('framenavigated', rec.onNavigated);
   return rec;
 }
 
@@ -267,6 +279,7 @@ function finishRecording(rec, why = null, { shuttingDown = false } = {}) {
 async function finishFile(rec, shuttingDown) {
   clearTimeout(rec.limit);
   if (rec.onClose) rec.page.off('close', rec.onClose);
+  if (rec.onNavigated) rec.page.off('framenavigated', rec.onNavigated);
   if (tabState(rec.page).recording === rec) delete tabState(rec.page).recording;
   // The last frame runs to now, and a recording has one frame at least.
   if (rec.ff && rec.latest && !rec.ffExited) {
@@ -314,7 +327,97 @@ async function finishFile(rec, shuttingDown) {
   else if (rec.exitedEarly) notes.push(`ffmpeg ${exitText(code)}${rec.ffmpegError ? ` (${rec.ffmpegError.trim()})` : ''}, ${short}`);
   else if (code !== 0) notes.push(`ffmpeg ${exitText(code)}${rec.ffmpegError ? `: ${rec.ffmpegError.trim()}` : ''}`);
   const length = cut || rec.exitedEarly ? '' : `${seconds.toFixed(1)}s, `;
-  return `Saved: ${rec.file} (${length}${rec.size.width}x${rec.size.height}, ${sizeText(saved)})${notes.length ? `\n${notes.join('; ')}` : ''}`;
+  const steps = writeSteps(rec, seconds);
+  const stepsLine = !steps ? '' : steps.error ? `\nIts steps were not saved: ${steps.error}` : `\nSteps: ${steps.file} (${steps.count}; when each command ran, and where)`;
+  return `Saved: ${rec.file} (${length}${rec.size.width}x${rec.size.height}, ${sizeText(saved)})${notes.length ? `\n${notes.join('; ')}` : ''}${stepsLine}`;
+}
+
+// The steps: when each command the REPL ran on a recording tab began and ended, in the video's own
+// seconds, and the element or point it acted on in the video's pixels, saved next to the video for
+// whoever edits it (a zoom on a step, say). Only that: watch is the log of what happened.
+
+// The video's clock: frames are written against the wall clock since the first, less those dropped.
+function videoTime(rec) {
+  return rec.firstFrameAt === null ? 0 : Math.max(0, (Date.now() - rec.firstFrameAt) / 1000 - rec.dropped / FPS);
+}
+
+// What a step is, for its line: the command and what it acts on, never what it types or chooses, the
+// code it runs or the URL it goes to, which can hold a password or a token. A steps file goes where the
+// video goes, and the video shows a password as dots.
+const WHOLE = new Set(['click', 'dblclick', 'hover', 'check', 'uncheck', 'mousemove', 'mousedown', 'mouseup', 'mousewheel']);
+const TARGET_FIRST = new Set(['fill', 'type', 'select', 'press', 'upload']);
+function stepLabel(cmd, args) {
+  if (WHOLE.has(cmd)) return `${cmd} ${args.replace(/\s+--modifiers=\S+/, '')}`.trim();
+  if (!TARGET_FIRST.has(cmd)) return cmd;
+  const rest = cmd === 'type' ? args.replace(/^\s*--delay(?:=|\s+)\S*\s*/, '') : args;
+  const parsed = splitSelector(rest);
+  // type and press with one word act on the focused element: the word is the text or the key.
+  return parsed && parsed.rest !== undefined ? `${cmd} ${parsed.word}` : cmd;
+}
+
+function stepStarted(p, command) {
+  const rec = recordingOf(p);
+  if (!rec || rec.finishing) return null;
+  // After the page changed size, its frames are fitted into the first size, so its pixels are not the video's.
+  rec.step = { rec, t0: videoTime(rec), t1: null, command, box: null, point: null, offSize: rec.resizedAt !== null, failed: false };
+  rec.steps.push(rec.step);
+  return rec.step;
+}
+
+function stepEnded(step, failed) {
+  if (!step) return;
+  step.t1 = videoTime(step.rec);
+  step.failed = failed;
+  if (step.rec.step === step) step.rec.step = null;
+}
+
+// The element (box) or point the running step acts on, in CSS pixels.
+function stepTarget(p, box, point) {
+  const rec = recordingOf(p);
+  if (!rec?.step) return;
+  const scaled = o => o && Object.fromEntries(Object.entries(o).map(([k, v]) => [k, Math.round(v * rec.scale)]));
+  rec.step.box = scaled(box);
+  rec.step.point = scaled(point);
+}
+
+// Where the cursor went (cursor on), and when: a line of its own, not a command's step.
+function cursorStep(p, what, point, ms) {
+  const rec = recordingOf(p);
+  if (!rec || rec.finishing) return;
+  const t0 = videoTime(rec);
+  const scaled = Math.round(point.x * rec.scale);
+  rec.steps.push({ rec, t0, t1: t0 + ms / 1000, command: what.replace(/-?\d+(?:\.\d+)?/g, n => String(Math.round(Number(n) * rec.scale))), box: null, point: { x: scaled, y: Math.round(point.y * rec.scale) }, offSize: rec.resizedAt !== null, failed: false });
+}
+
+function stepsPath(file) {
+  return file.replace(/\.[^./]+$/, '') + '.steps.txt';
+}
+
+// One line a step: t0 t1, then x y w h of its element (a point is x y 0 0; none, or not in the video's
+// pixels, is - - - -), then the command. Plain words, for a shell script to read.
+function writeSteps(rec, seconds) {
+  if (!rec.steps.length) return null;
+  const file = stepsPath(rec.file);
+  const lines = [
+    `# pw-repl record steps, for a video of ${rec.size.width}x${rec.size.height}, ${seconds.toFixed(2)}s long`,
+    '# t0 t1: when the command began and ended, in seconds into the video',
+    '# x y w h: its element in the video\'s pixels (a point: w and h are 0; none: -)',
+    '# (cursor from X Y): the cursor (cursor on) glided from X Y to x y between t0 and t1',
+    '# (page): the tab went to a new page, loaded or routed to by a single-page app',
+    ...(rec.resizedAt !== null ? [`# the page changed size at ${rec.resizedAt.toFixed(2)}s: boxes after it are left out`] : []),
+  ];
+  for (const s of rec.steps) {
+    const where = s.offSize ? null : s.box || (s.point && { x: s.point.x, y: s.point.y, width: 0, height: 0 });
+    const xywh = where ? `${where.x} ${where.y} ${where.width} ${where.height}` : '- - - -';
+    lines.push(`${s.t0.toFixed(2)} ${(s.t1 ?? videoTime(rec)).toFixed(2)} ${xywh} ${s.command}${s.failed ? '  # failed' : ''}`);
+  }
+  try {
+    fs.writeFileSync(file, `${lines.join('\n')}\n`, { flag: 'wx', mode: 0o600 });
+  } catch (error) {
+    return { file, error: error.code === 'EEXIST' ? `${file} already exists` : error.message };
+  }
+  rec.stepsFile = file;
+  return { file, count: rec.steps.length };
 }
 
 const TAB_CLOSED = 'Its tab closed, which ended it';
@@ -439,9 +542,18 @@ const commands = {
       } catch (error) {
         throw new Error(`${text}\nIt could not be renamed to ${rename} (${error.message}), so it stays where it was saved`);
       }
+      // Its steps go with it, unless a file has the new name already.
+      let renamed = t => t.split(rec.file).join(rename);
+      if (rec.stepsFile && !fs.existsSync(stepsPath(rename))) {
+        try {
+          moveRecording(rec.stepsFile, stepsPath(rename));
+          const before = renamed;
+          renamed = t => before(t.split(rec.stepsFile).join(stepsPath(rename)));
+        } catch {}
+      }
       const last = lastEnded.get(rec.owner ?? null);
-      if (last) last.text = last.text.split(rec.file).join(rename);
-      out.log(text.split(rec.file).join(rename));
+      if (last) last.text = renamed(last.text);
+      out.log(renamed(text));
       return;
     }
     if (numbers.length > 1) throw new Error(usage);
@@ -481,4 +593,4 @@ const commands = {
   },
 };
 
-module.exports = { commands, recordingOf, finishRecording, endedByTabClose, findFfmpeg, jpegSize };
+module.exports = { commands, recordingOf, finishRecording, endedByTabClose, findFfmpeg, jpegSize, stepStarted, stepEnded, stepTarget, cursorStep, stepLabel };

package/lib/runner.js

@@ -7,6 +7,7 @@ const { commands, cliCommands, needsTab } = require('./commands');
 const { dialogCommand, CDP_ANSWER, CDP_REFUSED } = require('./dialogs');
 const { activeModes } = require('./modes');
 const { NO_TAB } = require('./util');
+const { stepStarted, stepEnded, stepLabel } = require('./record');
 const cliNames = require('./cli-names');
 
 // playwright-cli names already explained, per client: each client is told once.
@@ -105,9 +106,13 @@ async function executeCommand(text) {
     out.error(`Error: ${NO_TAB}`);
     return { status: 'error', cmd };
   }
+  // A recording tab notes when each command that may change it ran, for its steps file (help record).
+  const step = READ_ONLY.has(cmd) || NO_TAB_NEEDED.has(cmd) || !state.page || state.page.isClosed() ? null : stepStarted(state.page, stepLabel(cmd, args));
   try {
     await run(args, all);
+    stepEnded(step, false);
   } catch (e) {
+    stepEnded(step, true);
     // Playwright colours its call log; the codes are noise anywhere but a terminal.
     e.message = String(e.message || '').replace(/\x1b\[[0-9;]*m/g, '');
     out.error(`Error: ${e.message}`);

package/lib/tabstate.js

@@ -11,6 +11,9 @@
 //                  with the owner
 //   viewport       the size viewport set: { width, height }, with the owner
 //   recording      its recording (record on): the ffmpeg writing it, its file, its owner
+//   cursor         its cursor (cursor on): where it is drawn { x, y }, with the owner
+//   cursorKept     its cursor is drawn again on each new page
+//   mouseAt        where the REPL last put the mouse { x, y }, when it knows
 //   routes         Map(glob -> { status, body, handler, owner })
 //   log            its recent requests (requests)
 //   consoleLog     its recent console messages and page errors (console)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pw-repl",
-  "version": "0.7.2",
+  "version": "0.8.1",
   "description": "Text REPL for driving an existing Chromium through Playwright over CDP",
   "keywords": [
     "playwright",

package/skill/SKILL.md

@@ -205,8 +205,9 @@ pw-repl send -c <name> close                        # your modes off; tab close
 - To point something out, to the person or in a screenshot, `highlight <ref|selector>` draws a box over it
   without changing the page (`--style=<css>` restyles it; `highlight off` hides them). Its label can
   cover what is below it; `help highlight` says how to mark a screenshot without one.
-- For a video, `help video` says which way suits what you want: `record on` records a tab's page, and a
-  screen recorder on the person's machine records what they see.
+- For a video, `help video` says which way suits what you want: `record on` records a tab's page (`cursor
+  on` draws a pointer where you act, for the viewer), and a screen recorder on the person's machine records
+  what they see.
 
 ## Environment