@vosjs/cli 0.31.1 → 0.32.0

package/README.md

@@ -143,7 +143,7 @@ A re-record replaces the footage, the cursor track, the frames and everything de
 }
 ```
 
-Seven verbs: `wait`, `hover` (`ms` 700), `click`, `type` (`delayMs` 40 per key; `focus: false` skips the focusing click, for a submitting Enter), `scroll`, `move`, `drag` (press, move, release: a range input, a canvas element, a timeline clip). Every step takes an optional unique `id`; give steps ids so a span anchored to a step survives script edits and `vos plan --reuse` can follow it. Because the CLI issues every input itself, the cursor track is synthesized with exact coordinates, exact timing and fresh element rects, which is what powers element-aware auto-zoom and click effects downstream. The schema is [`schema/actions.schema.json`](./schema/actions.schema.json); `vos validate actions.json` checks a script without running anything.
+Seven verbs: `wait`, `hover` (`ms` 700), `click`, `type` (`delayMs` 40 per key; `focus: false` skips the focusing click, for a submitting Enter), `scroll`, `move`, `drag` (press, move, release: a range input, a canvas element, a timeline clip). Every step takes an optional unique `id`; give steps ids so a span anchored to a step survives script edits and `vos plan --reuse` can follow it, and so a layer can name the element a step touched (`overlays[].pin.step`; the recorder keeps each hover, click, type and drag step's element rect on `meta.steps[].rect`). Because the CLI issues every input itself, the cursor track is synthesized with exact coordinates, exact timing and fresh element rects, which is what powers element-aware auto-zoom and click effects downstream. The schema is [`schema/actions.schema.json`](./schema/actions.schema.json); `vos validate actions.json` checks a script without running anything.
 
 Verified the flow in agent-browser already? `vos actions from-agent-browser steps.jsonl [--out actions.json] [--url] [--viewport WxH]` writes the script from that walk (each command kept beside its `--json` result, the batch record shape; refs resolve through the last `snapshot -i`), and names every step the recorder cannot follow rather than dropping it.
 
@@ -151,13 +151,15 @@ Verified the flow in agent-browser already? `vos actions from-agent-browser step
 
 `doc.json` is a `ProjectDoc` from [`@vosjs/studio-core`](../studio-core), all plain JSON. Zoom is `zoom: [{ id, in, out, level, cx, cy, source }]`, trims are `segments`, pacing is `speed`. Edit and re-render; nothing re-runs the browser. The full shape ships as a JSON Schema at [`schema/doc.schema.json`](./schema/doc.schema.json) (a `oneOf`: the recording document and the program document, sharing the layer definitions), and `vos validate <dir>` lints the semantics of either (span overlap, footage bounds, coordinate ranges, export honesty) before you spend a render.
 
-**Contracts that bite.** Time is source seconds in `zoom`, `segments`, `speed` and `tilt`, and output seconds in `overlays` and `audio`. Zoom `cx`/`cy` and overlay `transform.x`/`y` are normalized fractions of the frame in `[0, 1]` (`0.5, 0.5` is the centre), never pixels. `level` is 1..5. The planners write `source: "auto"`; spans you add or edit carry `source: "manual"`, which a re-plan never touches, and a deleted automatic span is recorded in `doc.rejected` so a re-plan does not bring it back.
+**Contracts that bite.** Time is source seconds in `zoom`, `segments`, `speed` and `tilt`, and output seconds in `overlays` and `audio`. Zoom `cx`/`cy`, overlay `transform.x`/`y` and a pin's `rect` are normalized fractions of the frame in `[0, 1]` (`0.5, 0.5` is the centre), never pixels. `level` is 1..5. The planners write `source: "auto"`; spans you add or edit carry `source: "manual"`, which a re-plan never touches, and a deleted automatic span is recorded in `doc.rejected` so a re-plan does not bring it back.
 
 **Camera styles.** `doc.zoomStyle` is one of `glide` (default), `focus`, `cinema`, `snappy`, `cut`, `keynote` (gliding zooms plus a medium lean toward each focus, the launch-film register), `drift` (slow ambient zooms with a subtle lean) or `none`. In the studio, picking a style also stamps `tiltStyle` and plans the tilt spans; in `doc.json` set `tiltStyle` and `tilt` yourself.
 
 **Tilt.** `doc.tilt`: source-anchored, non-overlapping spans where the card leans to a pose and returns to rest: `[{ "id": "u0", "in": 4, "out": 8, "rx": 6, "ry": -9, "source": "manual" }]`. `rx`/`ry` are degrees (±45 hard limit; ±5..18 reads well); `+rx` brings the top edge toward the camera, `+ry` the left edge, so a lean toward a right-side focus is a negative `ry`. Rest is flat. `doc.tiltStyle` (`off`, `subtle`, `medium`, `strong`) is the intensity the planner derives automatic spans from.
 
-**Text overlays.** `doc.overlays`: screen-space clips above the card, outside the zoom, output-anchored. `{ "id": "t0", "kind": "text", "start": 1, "duration": 3, "text": "Ship it", "preset": "title", "transform": { "x": 0.5, "y": 0.82, "scale": 1, "rotation": 0 }, "enter": "rise", "exit": "fade" }`. Presets `title`, `caption`, `label`, overridable with `size` (12..200 design px) and `color`; `\n` breaks lines; a lower third sits at `y` ≈ 0.82. Enter and exit: `rise`, `fade`, `none`. Fonts load at render start, fail-open to system stacks.
+**Text overlays.** `doc.overlays`: screen-space clips above the card, outside the zoom, output-anchored. `{ "id": "t0", "kind": "text", "start": 1, "duration": 3, "text": "Ship it", "preset": "title", "transform": { "x": 0.5, "y": 0.82, "scale": 1, "rotation": 0 }, "enter": "rise", "exit": "fade" }`. Presets `title`, `caption`, `label`, overridable with `size` (12..200 design px) and `color`; `\n` breaks lines; a caption (a layer with no referent) is a lower third at `y` ≈ 0.82. Enter and exit: `rise`, `fade`, `none`. Fonts load at render start, fail-open to system stacks.
+
+**Pinned layers.** A layer ABOUT something on the page names it, and the lowering keeps the two together: `"pin": { "step": "copy", "side": "auto", "mark": "ring", "leader": true }` on any overlay kind places the layer a gap off one side of that step's element and carries it with the element as the camera moves (the layer keeps its screen size; only its place follows). One of `step` (a recorder step's `id`, else its index), `press` (SOURCE seconds; the nearest press names the element, for a human recording) or `rect` (video fractions) names the referent. `side` is `auto` (the first of right, left, below, above whose box fits inside the frame through the layer's life) or a side by name; `gap` is design px (24); `mark` is `ring` or `underline`, a standing highlight on the referent for the layer's life; `leader` draws a hairline from the layer to the referent; `color` is their ink. `transform.x/y` stay the fallback for a pin that cannot resolve, and `vos validate` says why (an unknown step, a step that touched nothing, a scroll inside the layer's window). A layer that cannot name its referent is a caption: place it in the margin, never over the app.
 
 **Image and video overlays.** Media kinds on the same lane: `{ "id": "m0", "kind": "image" | "video", "start": 2, "duration": 4, "key": "/logo.png", "width": 0.35, "radius": 12, "opacity": 1, "loop": false, "transform": { … } }`. `key` is a file inside the take directory (`"/logo.png"`) or a URL; `width` is a fraction of the frame width, height follows the media's aspect; corners in design px. Video time is clip-local and muted by design; soundtracks belong to `doc.audio`.
 

package/dist/{chunk-APVJ4MDJ.js → chunk-QJORXTJM.js}

@@ -210,9 +210,15 @@ import {
   ZOOM_LEVEL_MIN,
   ZOOM_SPAN_MIN,
   docCardLayout,
+  docZoomTrack,
+  findStep,
   outputEnd,
+  pinBox,
+  pinRectOnScreen,
+  pinReferent,
   ratedSegments,
   recommendedExportResolution,
+  resolvePins,
   spanOutputExtent,
   zoomCoversRect,
   cameraModel
@@ -944,6 +950,7 @@ function lintDoc(docIn) {
     if (!isNum(o.duration) || o.duration <= 0) {
       problems.push(`${name}.duration must be > 0 (seconds)`);
     }
+    checkPin(o, name, docIn, recording, problems, warnings);
     if (o.kind === "text") {
       if (typeof o.text !== "string")
         problems.push(`${name}.text must be a string`);
@@ -1455,6 +1462,106 @@ function clicksWithRects(source) {
   }
   return out;
 }
+var PIN_SIDES = ["auto", "right", "left", "below", "above"];
+var PIN_MARKS = ["none", "ring", "underline"];
+function checkPin(o, name, docIn, recording, problems, warnings) {
+  if (o.pin === void 0) return;
+  if (!isObj(o.pin)) {
+    problems.push(
+      `${name}.pin must be { step | press | rect, side?, gap?, mark?, leader?, color? }`
+    );
+    return;
+  }
+  const p = o.pin;
+  const named = ["step", "press", "rect"].filter((k) => p[k] !== void 0);
+  if (named.length !== 1) {
+    problems.push(
+      `${name}.pin names its referent by exactly one of step (a recorder step id or index), press (a source second) or rect (video fractions)${named.length ? ` \u2014 got ${named.join(" and ")}` : ""}`
+    );
+    return;
+  }
+  if (p.step !== void 0 && typeof p.step !== "string" && !(isNum(p.step) && Number.isInteger(p.step) && p.step >= 0)) {
+    problems.push(
+      `${name}.pin.step must be a step id (string) or index (integer \u2265 0)`
+    );
+  }
+  if (p.press !== void 0 && !isNum(p.press))
+    problems.push(`${name}.pin.press must be a source second (number)`);
+  if (p.rect !== void 0) {
+    const r = isObj(p.rect) ? p.rect : null;
+    if (!r || !isNum(r.x) || !isNum(r.y) || !isNum(r.w) || !isNum(r.h)) {
+      problems.push(`${name}.pin.rect must be { x, y, w, h }`);
+    } else if (Math.abs(r.x) > 2 || Math.abs(r.y) > 2 || r.w > 2 || r.h > 2) {
+      problems.push(
+        `${name}.pin.rect looks like PIXELS (${String(r.x)}, ${String(r.y)}, ${String(r.w)}\xD7${String(r.h)}) \u2014 it is FRACTIONS of the video frame [0..1] (the zoom cx/cy convention)`
+      );
+    } else if (r.w <= 0 || r.h <= 0) {
+      problems.push(`${name}.pin.rect must have w > 0 and h > 0`);
+    }
+  }
+  if (p.side !== void 0 && !PIN_SIDES.includes(p.side))
+    problems.push(`${name}.pin.side must be one of ${PIN_SIDES.join(" | ")}`);
+  if (p.gap !== void 0 && (!isNum(p.gap) || p.gap < 0))
+    problems.push(`${name}.pin.gap must be design px \u2265 0`);
+  if (p.mark !== void 0 && !PIN_MARKS.includes(p.mark))
+    problems.push(`${name}.pin.mark must be one of ${PIN_MARKS.join(" | ")}`);
+  if (p.leader !== void 0 && typeof p.leader !== "boolean")
+    problems.push(`${name}.pin.leader must be true or false`);
+  if (p.color !== void 0 && typeof p.color !== "string")
+    problems.push(`${name}.pin.color must be a CSS colour string`);
+  if (problems.some((m) => m.startsWith(`${name}.pin`))) return;
+  if (!recording) {
+    problems.push(
+      `${name}.pin names something on a recording's page; a program has no referents \u2014 place the layer with transform`
+    );
+    return;
+  }
+  const doc = docIn;
+  const steps = doc.source.meta.steps ?? [];
+  if (p.step !== void 0) {
+    const step = findStep(steps, p.step);
+    if (!step) {
+      const known = steps.filter((s) => s.selector).map((s) => s.id !== void 0 ? s.id : String(s.step));
+      problems.push(
+        `${name}.pin.step "${String(p.step)}" is not a step of this take${known.length ? ` \u2014 steps with an element: ${known.join(", ")}` : " \u2014 the take has no step timeline (a human recording); pin by press or rect"}`
+      );
+      return;
+    }
+    if (step.skipped) {
+      problems.push(
+        `${name}.pin.step "${String(p.step)}" was skipped at record time (its selector never became visible), so it has no element to point at`
+      );
+      return;
+    }
+  }
+  let ref;
+  try {
+    ref = pinReferent(doc, p);
+  } catch {
+    ref = null;
+  }
+  if (!ref) {
+    problems.push(
+      p.step !== void 0 ? `${name}.pin.step "${String(p.step)}" has no element rect and no press inside its window (a wait, scroll or move step touches nothing) \u2014 pin a hover, click, type or drag step, or a press` : p.press !== void 0 ? `${name}.pin.press ${String(p.press)}s has no press with an element rect within half a second \u2014 the presses are the cursor track's down events` : `${name}.pin.rect could not be read`
+    );
+    return;
+  }
+  if (ref.at !== null && isNum(o.start) && isNum(o.duration)) {
+    const rated = ratedSegments(doc);
+    for (const s of steps) {
+      if (s.tStart <= ref.at) continue;
+      if (s.do !== "scroll" && !s.navigated) continue;
+      const out = spanOutputExtent(rated, s.tStart, s.tEnd);
+      if (!out) continue;
+      if (out.start < o.start + o.duration && out.end > o.start) {
+        warnings.push(
+          `${name} is pinned to something measured at ${ref.at.toFixed(1)}s, but ${s.do === "scroll" ? "a scroll" : "a navigation"} at ${s.tStart.toFixed(1)}s (output ${out.start.toFixed(1)}s) falls inside the layer's window \u2014 the referent may have moved; end the layer before it`
+        );
+        break;
+      }
+    }
+  }
+}
 function framingWarnings(docIn, doc, zoom, overlays, warnings) {
   try {
     const source = isObj(doc.source) ? doc.source : {};
@@ -1496,24 +1603,54 @@ function framingWarnings(docIn, doc, zoom, overlays, warnings) {
       }
     }
     const rated = ratedSegments(docIn);
-    const toFrame = (nx, ny) => ({
-      x: (layout.dx + nx * layout.dw) / layout.W,
-      y: (layout.dy + ny * layout.dh) / layout.H
-    });
+    const camera = cameraModel(docIn.frame);
+    const zoomTrack = docZoomTrack(docIn);
+    const pins = resolvePins(docIn, layout, camera, zoomTrack);
     overlays.forEach((o, i) => {
-      if (!isObj(o) || o.kind !== void 0 && o.kind !== "text") return;
+      if (!isObj(o) || !isNum(o.start) || !isNum(o.duration)) return;
       const tf = isObj(o.transform) ? o.transform : null;
-      if (!tf || !isNum(tf.x) || !isNum(tf.y) || !isNum(o.start) || !isNum(o.duration))
+      if (!tf || !isNum(tf.x) || !isNum(tf.y)) return;
+      const pinned = pins.get(String(o.id));
+      if (pinned) {
+        if (pinned.clamped > 4) {
+          warnings.push(
+            `overlays[${i}] is pinned ${pinned.side} of its referent but the frame has no room there \u2014 it sits ${Math.round(pinned.clamped)} design px off; a smaller box, another side, or a lower zoom level keeps it beside what it names`
+          );
+        }
+        return;
+      }
+      let box;
+      try {
+        box = pinBox(o, layout);
+      } catch {
         return;
+      }
       for (const c of clicks) {
         const out = spanOutputExtent(rated, c.t, c.t + 1e-3);
         if (!out || out.start < o.start || out.start > o.start + o.duration)
           continue;
-        const a = toFrame(c.rect.x / w, c.rect.y / h);
-        const b = toFrame((c.rect.x + c.rect.w) / w, (c.rect.y + c.rect.h) / h);
-        if (tf.x >= a.x && tf.x <= b.x && tf.y >= a.y && tf.y <= b.y) {
+        const target = pinRectOnScreen(
+          {
+            x: c.rect.x / w,
+            y: c.rect.y / h,
+            w: c.rect.w / w,
+            h: c.rect.h / h
+          },
+          out.start,
+          layout,
+          camera,
+          zoomTrack
+        );
+        const L = {
+          x: tf.x * layout.W - box.w / 2,
+          y: tf.y * layout.H - box.h / 2,
+          w: box.w,
+          h: box.h
+        };
+        const overlaps2 = L.x < target.x + target.w && L.x + L.w > target.x && L.y < target.y + target.h && L.y + L.h > target.y;
+        if (overlaps2) {
           warnings.push(
-            `overlays[${i}] sits over the thing being clicked at ${c.t.toFixed(1)}s (output ${out.start.toFixed(1)}s) \u2014 move it off the target`
+            `overlays[${i}] sits over the thing being clicked at ${c.t.toFixed(1)}s (output ${out.start.toFixed(1)}s) \u2014 pin it to that step (pin: { step }) so it sits beside the target and follows it, or move it off`
           );
           break;
         }
@@ -5894,6 +6031,7 @@ async function recordTake(browser, url, actions, paths, log, opts = {}) {
     const stepStart = now();
     const skippedBefore = skipped.length;
     const urlBefore = page.url();
+    let stepRect = null;
     switch (step.do) {
       case "wait":
         await sleep(clampWait(step.ms, now(), maxSeconds));
@@ -5903,6 +6041,7 @@ async function recordTake(browser, url, actions, paths, log, opts = {}) {
         break;
       case "hover": {
         const rect = await boxOf(step.selector);
+        stepRect = rect;
         if (rect) {
           await moveTo(rect.x + rect.w / 2, rect.y + rect.h / 2);
           log(`hover ${step.selector}`);
@@ -5913,6 +6052,7 @@ async function recordTake(browser, url, actions, paths, log, opts = {}) {
       }
       case "click": {
         const rect = await boxOf(step.selector);
+        stepRect = rect;
         if (rect) {
           await clickAt(rect);
           log(`click ${step.selector}`);
@@ -5923,6 +6063,7 @@ async function recordTake(browser, url, actions, paths, log, opts = {}) {
       }
       case "type": {
         const rect = await boxOf(step.selector);
+        stepRect = rect;
         if (rect) {
           if (step.focus !== false) await clickAt(rect);
           const delay = step.delayMs ?? 40;
@@ -5964,6 +6105,7 @@ async function recordTake(browser, url, actions, paths, log, opts = {}) {
         let rect;
         if (step.selector) {
           const box = await boxOf(step.selector);
+          stepRect = box;
           if (box) {
             rect = box;
             start = { x: box.x + box.w / 2, y: box.y + box.h / 2 };
@@ -6026,7 +6168,15 @@ async function recordTake(browser, url, actions, paths, log, opts = {}) {
       tStart: +(stepStart / 1e3).toFixed(3),
       tEnd: +(now() / 1e3).toFixed(3),
       ...skipped.length > skippedBefore ? { skipped: true } : {},
-      ...page.url() !== urlBefore ? { navigated: true } : {}
+      ...page.url() !== urlBefore ? { navigated: true } : {},
+      ...stepRect ? {
+        rect: {
+          x: +stepRect.x.toFixed(1),
+          y: +stepRect.y.toFixed(1),
+          w: +stepRect.w.toFixed(1),
+          h: +stepRect.h.toFixed(1)
+        }
+      } : {}
     });
   }
   if (!capped) await sleep(600);
@@ -10628,4 +10778,4 @@ export {
   convertAgentBrowser,
   run
 };
-//# sourceMappingURL=chunk-APVJ4MDJ.js.map
+//# sourceMappingURL=chunk-QJORXTJM.js.map