@vosjs/cli 0.69.1 → 0.71.0

package/README.md

@@ -90,6 +90,8 @@ vos record --actions actions.json --out take --strict   # footage replaced; the
 vos plan take --reuse                                   # re-time that cut onto the new recording; what could not follow is named
 ```
 
+Every manual span follows the step it was made on (the gesture whose window held it, or the one just before it), keeping its offset exactly; a span whose step is missing or skipped in the new recording is dropped and named with what it framed, never re-timed to a place with nothing under it. A layer pinned to a step (`pin: { step }`) plays when that step plays in the new cut; a layer that is not pinned keeps its output second, and when the step under it moved the report names it with the step's new time and the pin to write. A take fetched from vos.so can be recorded again: the script rides the document and `vos fetch --media` writes it back as `actions.json`.
+
 **Already have a video?** A Cursor cloud agent's PR video, a Playwright run's `video.webm`, a Loom export: `vos ingest` makes it a take. The file becomes `recording.<container>` (stream-copied into a seekable container, or copied as it is), `meta.json` comes from the file's own dimensions and duration, and a trace recorded beside it becomes `cursor.json`, so the planner has clicks to zoom on.
 
 ```bash
@@ -171,7 +173,9 @@ A session never enters a take directory and never reaches vos.so: `vos push` ref
 
 **Rehearse before you record.** `vos record … --dry-run` runs every step against the real page, in order, because a later selector usually exists only after an earlier click. Nothing is captured and nothing is written: the pointer lands instead of travelling, every pause is cut to a beat, and the take directory beside it keeps its footage, its cut and its script exactly as they were (a real re-record moves `doc.json` aside; a rehearsal does not). Selector lookups keep their whole timeout, so a miss here is a miss in the take. It prints each step with the rect it resolved, in capture px, which are the rects a pin or `vos callout --step` reads, and exits 2 on any miss or a first load that never settled. Add `--dry-run` to the exact command you were about to run; `--storage-state` and `--browser-arg=` apply to it too, and the `Next:` line it prints carries them, so the command it hands you records what it rehearsed.
 
-**Digest first.** `vos digest <take>` is how an agent sees a recording without reading the video. It writes `digest/digest.json`: one moment per thing the cursor track says mattered (click clusters, typing sessions, scroll runs, dwells, idle gaps, head, tail, and frame-diff scene changes), each with source and output extents, a normalized `focus` and `rect` you can copy into a zoom span, per-second `activity`, and the planners' `proposed` span ids; plus one footage frame and a crop around the target per moment, and `sheet.png`, the contact sheet. Read the JSON, then the sheet, then a crop only where you must decide. `vos validate` then warns when a zoom does not contain what was clicked under it, and `vos frames --at-moments` renders the composed output at every moment so a still and its footage crop share an id.
+**The planners read the flow's own structure.** On a scripted take the step timeline is a planner input: a `wait` step between the first and the last doing step, or a hover's dwell, in which the digest's activity measured nothing moving is proposed at 2× (`speedParams.stillRate`; the opening and closing waits are the clip's establishing and resting beats and stay at 1×), the presses inside one step (a click and the dialog it opened, a drag's presses) are one zoom while their targets share a window, and a park before the first step (the cursor arriving on the logo) is never a dwell. Manual and rejected spans stay as they are.
+
+**Digest first.** `vos digest <take>` is how an agent sees a recording without reading the video. It writes `digest/digest.json`: one moment per thing the cursor track says mattered (click clusters, typing sessions, scroll runs, dwells, idle gaps, head, tail, and frame-diff scene changes), and after every click and typing run a `changed` moment, the region of the page that answered it (a dialog opening, a 200 landing, a panel re-laying out, a card going dark), with `seed` naming the moment it answers, its `rect` the changed region and its `focus` the centre, so a cut can hold on the consequence and a zoom can frame it, each with source and output extents, a normalized `focus` and `rect` you can copy into a zoom span, per-second `activity`, and the planners' `proposed` span ids; `steps`, every script step with its source and output windows, the page it left and the words its `read` kept, so a layer is placed by the table and quotes the take; plus one footage frame and a crop around the target per moment, and `sheet.png`, the contact sheet. Read the JSON, then the sheet, then a crop only where you must decide. `vos validate` then warns when a zoom does not contain what was clicked under it, and `vos frames --at-moments` renders the composed output at every moment so a still and its footage crop share an id.
 
 **Many media in one take (concat).** `doc.json` may carry `media: [{ id, videoKey, cursor, meta, … }]`, the take's OTHER recordings or uploads, each the `source` shape with an `id`; a segment on one (`{ in, out, media: "m1" }`) plays it, and a zoom, tilt, speed, freeze or cam-move span names the media its source seconds belong to (absent = the primary, `source`). The Video row shows the clips in order, a span is drawn where its media plays, `vos plan` proposes zoom and speed spans on every media from its own cursor track, `vos validate` measures each span against its own media's length, and `vos push` and `vos pull --media` carry every media through the recording door. A media wears its OWN card: `media[].frame` carries the card-owned fields (its placement and size as `inset`, the browser bar, the corner, the shadow, the border, the cover fit and its focus) over the take's frame while it plays; the bar names that media's recorded page, an upload with no page wears none, and the frame-wide fields (the aspect, the padding, the ground, the backdrop, the card's animation) stay the take's. Sound: the primary's tracks play at the primary's moments; another media's own audio is not spliced into the cut yet. **Many cards**: an image or video overlay may show a document media by reference (`key: "media:<id>"`, `media:` alone the primary; its cursor track and recorded page come with it) and wear a card (`frame`: `browserBar`, a `lean` `{rx, ry}` in degrees, `shadow`, `shadowContact`, `shadowColor`, `cursor`), drawn by the card painter on its own plane above the primary card with the media's cursor dot and click rings inside it; a layer without `frame` stays the flat picture. The primary card stays primary: the sequence, the camera and the cut are its.
 
@@ -208,7 +212,7 @@ take/
   frames/          raw screencast JPEGs, indexed by frames.json (kept for re-encode)
   cursor.json      the synthesized CursorTrack: exact coordinates, element rects
   meta.json        RecordingMeta (producer: "cli", per-step source extents)
-  actions.json     the script that produced it, with the resolved url
+  actions.json     the script that produced it, with the resolved url; pushed inside the document and written back by a fetch or pull with --media
   doc.json         ProjectDoc, the editable cut
   doc.prev.json    the previous cut, kept by a re-record for `vos plan --reuse`
   vos.json         hosted tracking (vos id + base version), written by push, fetch and pull
@@ -249,7 +253,7 @@ A re-record replaces the footage, the cursor track, the frames and everything de
 }
 ```
 
-Seven verbs: `wait`, `hover` (`ms` 700, the dwell), `click` (`ms` the settle after the press, 150), `type` (`delayMs` 40 per key, paced by the clock; `focus: false` skips the focusing click, for a submitting Enter; `ms` the settle after, 150), `scroll` (`ms` the settle, 200), `move`, `drag` (press, move, release: a range input, a canvas element, a timeline clip; `ms` the travel, 700). The recorder's pace is the script's: a gesture is driven by the clock, so a page that answers each pointer sample slowly costs samples, never seconds, and the only pauses are the ones the script names (`wait`, a hover's dwell, a settle). `vos record` ends with a pace line, what the script asked, what the gestures added, what the page cost, and the steps that ran slow, and the `--json` done event carries it as `pace`. Every step takes an optional unique `id`; give steps ids so a span anchored to a step survives script edits and `vos plan --reuse` can follow it, and so a layer can name the element a step touched (`overlays[].pin.step`; the recorder keeps each hover, click, type and drag step's element rect on `meta.steps[].rect`). Because the CLI issues every input itself, the cursor track is synthesized with exact coordinates, exact timing and fresh element rects, which is what powers element-aware auto-zoom and click effects downstream. The schema is [`schema/actions.schema.json`](./schema/actions.schema.json); `vos validate actions.json` checks a script without running anything.
+Eight verbs: `wait`, `hover` (`ms` 700, the dwell), `click` (`ms` the settle after the press, 150), `type` (`delayMs` 40 per key, paced by the clock; `focus: false` skips the focusing click; `replace: true` selects the field's value first so the typing replaces it; `at: "end" | "start"` puts the caret there before the first character, for a code editor line; `ms` the settle after, 150), `press` (one named key, `Escape`, `Enter`, `Tab`, `Backspace`, `Delete`, the arrows, `Home`, `End`, `PageUp`, `PageDown`, `Space`, recorded as a typing ping where the pointer is or on `selector` with no click, so an Escape closes what a click opened and an Enter submits what was typed without a second click ringing on the field; `ms` the settle, 150), `scroll` (`ms` the settle, 200), `move`, `drag` (press, move, release: a range input, a canvas element, a timeline clip; `ms` the travel, 700). The recorder's pace is the script's: a gesture is driven by the clock, so a page that answers each pointer sample slowly costs samples, never seconds, and the only pauses are the ones the script names (`wait`, a hover's dwell, a settle). `vos record` ends with a pace line, what the script asked, what the gestures added, what the page cost, and the steps that ran slow, and the `--json` done event carries it as `pace`. Every step takes an optional unique `id`; give steps ids so a span anchored to a step survives script edits and `vos plan --reuse` can follow it, and so a layer can name the element a step touched (`overlays[].pin.step`; the recorder keeps each hover, click, type and drag step's element rect on `meta.steps[].rect`). **The take carries its payload**: every step keeps the page it left (`meta.steps[].url`, origin and path), and a step with `read: "<selector>"` keeps that element's words once the step has settled (`meta.steps[].text`, at most 2 KB; on a click, what the click opened: the URL a share button minted, the code a dialog printed), so a layer quotes the take instead of a replay of the flow. Because the CLI issues every input itself, the cursor track is synthesized with exact coordinates, exact timing and fresh element rects, which is what powers element-aware auto-zoom and click effects downstream. The schema is [`schema/actions.schema.json`](./schema/actions.schema.json); `vos validate actions.json` checks a script without running anything.
 
 Verified the flow in agent-browser already? `vos actions from-agent-browser steps.jsonl [--out actions.json] [--url] [--viewport WxH]` writes the script from that walk (each command kept beside its `--json` result, the batch record shape; refs resolve through the last `snapshot -i`), and names every step the recorder cannot follow rather than dropping it.