@vosjs/cli 0.70.0 → 0.71.0

package/README.md

@@ -173,7 +173,9 @@ A session never enters a take directory and never reaches vos.so: `vos push` ref
 
 **Rehearse before you record.** `vos record … --dry-run` runs every step against the real page, in order, because a later selector usually exists only after an earlier click. Nothing is captured and nothing is written: the pointer lands instead of travelling, every pause is cut to a beat, and the take directory beside it keeps its footage, its cut and its script exactly as they were (a real re-record moves `doc.json` aside; a rehearsal does not). Selector lookups keep their whole timeout, so a miss here is a miss in the take. It prints each step with the rect it resolved, in capture px, which are the rects a pin or `vos callout --step` reads, and exits 2 on any miss or a first load that never settled. Add `--dry-run` to the exact command you were about to run; `--storage-state` and `--browser-arg=` apply to it too, and the `Next:` line it prints carries them, so the command it hands you records what it rehearsed.
 
-**Digest first.** `vos digest <take>` is how an agent sees a recording without reading the video. It writes `digest/digest.json`: one moment per thing the cursor track says mattered (click clusters, typing sessions, scroll runs, dwells, idle gaps, head, tail, and frame-diff scene changes), each with source and output extents, a normalized `focus` and `rect` you can copy into a zoom span, per-second `activity`, and the planners' `proposed` span ids; plus one footage frame and a crop around the target per moment, and `sheet.png`, the contact sheet. Read the JSON, then the sheet, then a crop only where you must decide. `vos validate` then warns when a zoom does not contain what was clicked under it, and `vos frames --at-moments` renders the composed output at every moment so a still and its footage crop share an id.
+**The planners read the flow's own structure.** On a scripted take the step timeline is a planner input: a `wait` step between the first and the last doing step, or a hover's dwell, in which the digest's activity measured nothing moving is proposed at 2× (`speedParams.stillRate`; the opening and closing waits are the clip's establishing and resting beats and stay at 1×), the presses inside one step (a click and the dialog it opened, a drag's presses) are one zoom while their targets share a window, and a park before the first step (the cursor arriving on the logo) is never a dwell. Manual and rejected spans stay as they are.
+
+**Digest first.** `vos digest <take>` is how an agent sees a recording without reading the video. It writes `digest/digest.json`: one moment per thing the cursor track says mattered (click clusters, typing sessions, scroll runs, dwells, idle gaps, head, tail, and frame-diff scene changes), and after every click and typing run a `changed` moment, the region of the page that answered it (a dialog opening, a 200 landing, a panel re-laying out, a card going dark), with `seed` naming the moment it answers, its `rect` the changed region and its `focus` the centre, so a cut can hold on the consequence and a zoom can frame it, each with source and output extents, a normalized `focus` and `rect` you can copy into a zoom span, per-second `activity`, and the planners' `proposed` span ids; `steps`, every script step with its source and output windows, the page it left and the words its `read` kept, so a layer is placed by the table and quotes the take; plus one footage frame and a crop around the target per moment, and `sheet.png`, the contact sheet. Read the JSON, then the sheet, then a crop only where you must decide. `vos validate` then warns when a zoom does not contain what was clicked under it, and `vos frames --at-moments` renders the composed output at every moment so a still and its footage crop share an id.
 
 **Many media in one take (concat).** `doc.json` may carry `media: [{ id, videoKey, cursor, meta, … }]`, the take's OTHER recordings or uploads, each the `source` shape with an `id`; a segment on one (`{ in, out, media: "m1" }`) plays it, and a zoom, tilt, speed, freeze or cam-move span names the media its source seconds belong to (absent = the primary, `source`). The Video row shows the clips in order, a span is drawn where its media plays, `vos plan` proposes zoom and speed spans on every media from its own cursor track, `vos validate` measures each span against its own media's length, and `vos push` and `vos pull --media` carry every media through the recording door. A media wears its OWN card: `media[].frame` carries the card-owned fields (its placement and size as `inset`, the browser bar, the corner, the shadow, the border, the cover fit and its focus) over the take's frame while it plays; the bar names that media's recorded page, an upload with no page wears none, and the frame-wide fields (the aspect, the padding, the ground, the backdrop, the card's animation) stay the take's. Sound: the primary's tracks play at the primary's moments; another media's own audio is not spliced into the cut yet. **Many cards**: an image or video overlay may show a document media by reference (`key: "media:<id>"`, `media:` alone the primary; its cursor track and recorded page come with it) and wear a card (`frame`: `browserBar`, a `lean` `{rx, ry}` in degrees, `shadow`, `shadowContact`, `shadowColor`, `cursor`), drawn by the card painter on its own plane above the primary card with the media's cursor dot and click rings inside it; a layer without `frame` stays the flat picture. The primary card stays primary: the sequence, the camera and the cut are its.
 
@@ -251,7 +253,7 @@ A re-record replaces the footage, the cursor track, the frames and everything de
 }
 ```
 
-Seven verbs: `wait`, `hover` (`ms` 700, the dwell), `click` (`ms` the settle after the press, 150), `type` (`delayMs` 40 per key, paced by the clock; `focus: false` skips the focusing click, for a submitting Enter; `ms` the settle after, 150), `scroll` (`ms` the settle, 200), `move`, `drag` (press, move, release: a range input, a canvas element, a timeline clip; `ms` the travel, 700). The recorder's pace is the script's: a gesture is driven by the clock, so a page that answers each pointer sample slowly costs samples, never seconds, and the only pauses are the ones the script names (`wait`, a hover's dwell, a settle). `vos record` ends with a pace line, what the script asked, what the gestures added, what the page cost, and the steps that ran slow, and the `--json` done event carries it as `pace`. Every step takes an optional unique `id`; give steps ids so a span anchored to a step survives script edits and `vos plan --reuse` can follow it, and so a layer can name the element a step touched (`overlays[].pin.step`; the recorder keeps each hover, click, type and drag step's element rect on `meta.steps[].rect`). Because the CLI issues every input itself, the cursor track is synthesized with exact coordinates, exact timing and fresh element rects, which is what powers element-aware auto-zoom and click effects downstream. The schema is [`schema/actions.schema.json`](./schema/actions.schema.json); `vos validate actions.json` checks a script without running anything.
+Eight verbs: `wait`, `hover` (`ms` 700, the dwell), `click` (`ms` the settle after the press, 150), `type` (`delayMs` 40 per key, paced by the clock; `focus: false` skips the focusing click; `replace: true` selects the field's value first so the typing replaces it; `at: "end" | "start"` puts the caret there before the first character, for a code editor line; `ms` the settle after, 150), `press` (one named key, `Escape`, `Enter`, `Tab`, `Backspace`, `Delete`, the arrows, `Home`, `End`, `PageUp`, `PageDown`, `Space`, recorded as a typing ping where the pointer is or on `selector` with no click, so an Escape closes what a click opened and an Enter submits what was typed without a second click ringing on the field; `ms` the settle, 150), `scroll` (`ms` the settle, 200), `move`, `drag` (press, move, release: a range input, a canvas element, a timeline clip; `ms` the travel, 700). The recorder's pace is the script's: a gesture is driven by the clock, so a page that answers each pointer sample slowly costs samples, never seconds, and the only pauses are the ones the script names (`wait`, a hover's dwell, a settle). `vos record` ends with a pace line, what the script asked, what the gestures added, what the page cost, and the steps that ran slow, and the `--json` done event carries it as `pace`. Every step takes an optional unique `id`; give steps ids so a span anchored to a step survives script edits and `vos plan --reuse` can follow it, and so a layer can name the element a step touched (`overlays[].pin.step`; the recorder keeps each hover, click, type and drag step's element rect on `meta.steps[].rect`). **The take carries its payload**: every step keeps the page it left (`meta.steps[].url`, origin and path), and a step with `read: "<selector>"` keeps that element's words once the step has settled (`meta.steps[].text`, at most 2 KB; on a click, what the click opened: the URL a share button minted, the code a dialog printed), so a layer quotes the take instead of a replay of the flow. Because the CLI issues every input itself, the cursor track is synthesized with exact coordinates, exact timing and fresh element rects, which is what powers element-aware auto-zoom and click effects downstream. The schema is [`schema/actions.schema.json`](./schema/actions.schema.json); `vos validate actions.json` checks a script without running anything.
 
 Verified the flow in agent-browser already? `vos actions from-agent-browser steps.jsonl [--out actions.json] [--url] [--viewport WxH]` writes the script from that walk (each command kept beside its `--json` result, the batch record shape; refs resolve through the last `snapshot -i`), and names every step the recorder cannot follow rather than dropping it.
 

package/dist/{chunk-KIFXVPBR.js → chunk-XB5XRE4Q.js}

@@ -2710,15 +2710,21 @@ import { mkdir, readFile as readFile2, rename, writeFile as writeFile2 } from "f
 import { existsSync as existsSync2 } from "fs";
 import { join as join2, resolve } from "path";
 import {
+  CHANGE_GRID,
+  CHANGE_LEAD,
+  CHANGE_OFFSETS,
   DIGEST_CROP_MAX,
   DIGEST_FULL_MAX,
   DIGEST_VERSION,
   MOTION_DELTA,
+  ambientDrift,
   buildDigest,
+  changedRegion,
   cropBox,
   frameGeometry,
   momentsFromDoc,
   outputDurationOf,
+  pickChange,
   planForDigest,
   sceneChanges
 } from "@vosjs/studio-core";
@@ -2762,7 +2768,40 @@ async function binsInPage(opts) {
       prev = luma;
       w.__progress = (i + 1) / n;
     }
-    w.__done = { ok: true, bins, width: size.width, height: size.height };
+    const g = document.createElement("canvas");
+    g.width = opts.grid.w;
+    g.height = opts.grid.h;
+    const gctx = g.getContext("2d", { willReadFrequently: true });
+    const gridAt = async (t) => {
+      const el = await getFrame(Math.max(0, Math.min(opts.durationS, t)));
+      gctx.drawImage(el, 0, 0, opts.grid.w, opts.grid.h);
+      const d = gctx.getImageData(0, 0, opts.grid.w, opts.grid.h).data;
+      const out = new Array(opts.grid.w * opts.grid.h);
+      for (let p = 0; p < out.length; p++) {
+        const o = p * 4;
+        out[p] = (d[o] * 299 + d[o + 1] * 587 + d[o + 2] * 114) / 1e3;
+      }
+      return out;
+    };
+    const changes = [];
+    for (const seed of opts.seeds) {
+      const samples = [];
+      if (seed.t - opts.lead >= 0)
+        samples.push({ dt: -opts.lead, luma: await gridAt(seed.t - opts.lead) });
+      samples.push({ dt: 0, luma: await gridAt(seed.t) });
+      for (const dt of opts.offsets) {
+        if (seed.t + dt > opts.durationS) break;
+        samples.push({ dt, luma: await gridAt(seed.t + dt) });
+      }
+      changes.push({ id: seed.id, samples });
+    }
+    w.__done = {
+      ok: true,
+      bins,
+      width: size.width,
+      height: size.height,
+      changes
+    };
   } catch (e) {
     w.__error = String(e instanceof Error && e.stack || e);
   }
@@ -2985,23 +3024,66 @@ async function digestTake(browser, dir, opts = {}) {
       });
       await page.goto(`${server.base}/digest.html`);
       await page.waitForFunction("window.__pageReady__ === true");
-      void page.evaluate(binsInPage, { source, durationS: durS, delta: MOTION_DELTA }).catch(() => {
+      const seeds = momentsFromDoc(doc, plan, {}).filter(
+        (m) => m.at !== null && (m.kind === "click" || m.kind === "typing")
+      ).map((m) => ({
+        id: m.id,
+        t: m.kind === "click" ? m.at : m.source.out
+      }));
+      void page.evaluate(binsInPage, {
+        source,
+        durationS: durS,
+        delta: MOTION_DELTA,
+        seeds,
+        grid: { ...CHANGE_GRID },
+        offsets: [...CHANGE_OFFSETS],
+        lead: CHANGE_LEAD
+      }).catch(() => {
       });
       const b = await waitForPageDone(
         page,
         "digest bins",
         () => {
         },
-        6e4 + durS * 400
+        6e4 + durS * 400 + seeds.length * 2e3
       );
       bins = b.bins;
       plan = planForDigest(doc, bins);
       frameW = b.width;
       frameH = b.height;
+      const sampled = b.changes ?? [];
+      const changes = seeds.flatMap((seed) => {
+        const row = sampled.find((c) => c.id === seed.id);
+        const at2 = row?.samples.find((smp) => smp.dt === 0);
+        const earlier = row?.samples.find((smp) => smp.dt < 0);
+        const after = row?.samples.filter((smp) => smp.dt > 0) ?? [];
+        if (!row || !at2 || !after.length) return [];
+        const before = { ...CHANGE_GRID, luma: at2.luma };
+        const rate = earlier ? ambientDrift(
+          { ...CHANGE_GRID, luma: earlier.luma },
+          before,
+          -earlier.dt
+        ) : null;
+        const w = pickChange(
+          { id: seed.id, at: seed.t },
+          after.map((smp) => ({
+            dt: smp.dt,
+            region: changedRegion(
+              before,
+              { ...CHANGE_GRID, luma: smp.luma },
+              MOTION_DELTA,
+              1,
+              rate ? { rate, dt: smp.dt } : null
+            )
+          }))
+        );
+        return w ? [w] : [];
+      });
       moments = momentsFromDoc(doc, plan, {
         bins,
         scenes: sceneChanges(bins),
-        transcript
+        transcript,
+        changes
       });
       const geo = frameGeometry(doc, frameW, frameH);
       const shots = moments.filter((m) => m.at !== null).map((m) => {
@@ -5699,7 +5781,8 @@ function planOtherMedia(doc, activity) {
         height: m.meta.height,
         duration: m.meta.durationMs / 1e3,
         style: doc.zoomStyle,
-        params: doc.zoomParams
+        params: doc.zoomParams,
+        steps: m.meta.steps
       }).filter((z) => !manualZoom.some((x) => overlaps(z, x))).filter((z) => !isRejected("zoom", z, rejected)).map((z) => ({ ...z, id: `${z.id}-${m.id}`, media: m.id }))
     );
     const manualSpeed = (doc.speed ?? []).filter(
@@ -5709,7 +5792,8 @@ function planOtherMedia(doc, activity) {
       ...planAutoSpeed(m.cursor, {
         durationMs: m.meta.durationMs,
         params: doc.speedParams,
-        activity
+        activity,
+        steps: m.meta.steps
       }).filter((x) => !manualSpeed.some((y) => x.in < y.out && x.out > y.in)).filter((x) => !isRejected("speed", x, rejected)).map((x) => ({ ...x, id: `${x.id}-${m.id}`, media: m.id }))
     );
   }
@@ -5742,14 +5826,16 @@ async function planTake(dir, opts = {}) {
       height: doc.source.meta.height,
       duration: doc.source.meta.durationMs / 1e3,
       style: doc.zoomStyle,
-      params: doc.zoomParams
+      params: doc.zoomParams,
+      steps: doc.source.meta.steps
     }).filter((z) => !manualZoom.some((m) => overlaps(z, m))).filter((z) => !isRejected("zoom", z, doc.rejected));
     doc.zoom = [...manualZoom, ...autoZoom].sort((a, b) => a.in - b.in);
     const manualSpeed = rt.speed;
     const autoSpeed = planAutoSpeed(doc.source.cursor, {
       durationMs: doc.source.meta.durationMs,
       params: doc.speedParams,
-      activity
+      activity,
+      steps: doc.source.meta.steps
     }).filter((s) => !manualSpeed.some((m) => s.in < m.out && s.out > m.in)).filter((s) => !isRejected("speed", s, doc.rejected));
     doc.speed = [...manualSpeed, ...autoSpeed].sort((a, b) => a.in - b.in);
     {
@@ -5797,7 +5883,8 @@ async function planTake(dir, opts = {}) {
       height: doc.source.meta.height,
       duration: doc.source.meta.durationMs / 1e3,
       style: doc.zoomStyle,
-      params: doc.zoomParams
+      params: doc.zoomParams,
+      steps: doc.source.meta.steps
     }).filter((z) => !manual.some((m) => overlaps(z, m))).filter((z) => !isRejected("zoom", z, doc.rejected));
     doc.zoom = [...manual, ...auto].sort((a, b) => a.in - b.in);
     const manualSpeed = (doc.speed ?? []).filter(
@@ -5806,7 +5893,8 @@ async function planTake(dir, opts = {}) {
     const autoSpeed = planAutoSpeed(doc.source.cursor, {
       durationMs: doc.source.meta.durationMs,
       params: doc.speedParams,
-      activity
+      activity,
+      steps: doc.source.meta.steps
     }).filter((s) => !manualSpeed.some((m) => s.in < m.out && s.out > m.in)).filter((s) => !isRejected("speed", s, doc.rejected));
     doc.speed = [...manualSpeed, ...autoSpeed].sort((a, b) => a.in - b.in);
     {
@@ -5837,12 +5925,14 @@ async function planTake(dir, opts = {}) {
       height: doc.source.meta.height,
       duration: doc.source.meta.durationMs / 1e3,
       style: doc.zoomStyle,
-      params: doc.zoomParams
+      params: doc.zoomParams,
+      steps: doc.source.meta.steps
     });
     doc.speed = planAutoSpeed(doc.source.cursor, {
       durationMs: doc.source.meta.durationMs,
       params: doc.speedParams,
-      activity
+      activity,
+      steps: doc.source.meta.steps
     });
     fresh = true;
   }
@@ -11386,7 +11476,7 @@ function summary2(result) {
 async function cmdActionsScript(input, flags) {
   const r = createReporter(flags.json === true);
   const { readFile: readFile14 } = await import("fs/promises");
-  const { shotList, formatShotList } = await import("./shotList-KH6FTX4L.js");
+  const { shotList, formatShotList } = await import("./shotList-MAQVLZBY.js");
   const raw = JSON.parse(await readFile14(resolve11(input), "utf8"));
   const errors = validateActions(raw);
   if (errors.length) throw new UsageError(`${input}:
@@ -14238,4 +14328,4 @@ export {
   verbHelp,
   run
 };
-//# sourceMappingURL=chunk-KIFXVPBR.js.map
+//# sourceMappingURL=chunk-XB5XRE4Q.js.map