@fortemate/vega-vvd-driver 0.2.0 → 0.2.1

package/README.md

@@ -6,7 +6,7 @@ Drive the Vega Virtual Device from scripts and AI agents: press remote keys, tak
 
 ## Why
 
-The Vega SDK's `vega` command installs and launches apps on the Vega Virtual Device (VVD), but has no command to press a remote key or take a screenshot. Amazon's [Appium Vega driver](https://developer.amazon.com/docs/vega/0.24/appium-install.html) does both, and much more: it runs UI tests that find elements, on the VVD and on a Fire TV Stick. It needs an Appium 2 server, the driver package and the device's automation toolkit switched on, with Node 22 or earlier.
+The Vega SDK's `vega` command installs and launches apps on the Vega Virtual Device (VVD), but has no command to press a remote key or take a screenshot. On a Fire TV Stick, the device's own `gwsi-tool-screenshooter` takes one through `vega exec vda shell`, but the [VDA reference](https://developer.amazon.com/docs/vega/0.24/vda-tools.html) notes that the VVD doesn't support it. Amazon's [Appium Vega driver](https://developer.amazon.com/docs/vega/0.24/appium-install.html) does both, and much more: it runs UI tests that find elements, on the VVD and on a Fire TV Stick. It needs an Appium 2 server, the driver package and the device's automation toolkit switched on, with Node 22 or earlier.
 
 This driver is for lighter jobs on the VVD: a key press or a screenshot from a shell script, a video with sound, every frame of an animation, and an AI coding agent that can see what it built. It talks to the Android emulator that the VVD is built on, through the emulator's own gRPC API and console, so there is nothing to install on the device and no server to run.
 
@@ -14,15 +14,17 @@ Its methods were worked out while building [Dice Chess for Fire TV](https://gith
 
 ## Appium or this driver?
 
-| To…                                                 | Use                    |
-| --------------------------------------------------- | ---------------------- |
-| find elements, read their text, run a test suite    | Appium                 |
-| test on a Fire TV Stick                             | Appium                 |
-| press keys or take a screenshot from a shell script | this driver, or Appium |
-| record a video with sound                           | this driver            |
-| capture every frame of an animation                 | this driver            |
-| let an AI agent see and operate the VVD over MCP    | this driver            |
-| check the TV safe area                              | this driver            |
+| To…                                                 | Appium | this driver |
+| :-------------------------------------------------- | :----: | :---------: |
+| find elements, read their text, run a test suite    |   ✅   |      —      |
+| test on a Fire TV Stick                             |   ✅   |      —      |
+| press keys or take a screenshot from a shell script |   ✅   |     ✅      |
+| record a video with sound                           |   —    |     ✅      |
+| capture every frame of an animation                 |   —    |     ✅      |
+| let an AI agent see and operate the VVD over MCP    |   —    |     ✅      |
+| check the TV safe area                              |   ◯    |     ✅      |
+
+✅ a good fit · ◯ possible with your own code (take a screenshot, then check its edges) · — not supported
 
 ## What you need
 
@@ -129,8 +131,8 @@ Measured on the VVD with Vega SDK 0.24.12112 on macOS:
 - **Keys.** The emulator's gRPC `sendKey`, with Linux evdev codes, reaches apps: it is the path the VVD's own on-screen remote uses. OK is `KEY_KPENTER`, and apps receive it as `kpenter`, not the `select` the remote's documentation names. `KEY_SELECT` and `KEY_OK` never arrive, because the emulator's virtual keyboard does not declare them. Back is `KEY_BACK`; `KEY_ESC` does not reach an app as Back.
 - **Dead ends.** The emulator console's `event send`, QEMU's `send-key` and `inputd-cli` on the device all report success and never reach an app.
 - **Home cannot be sent.** `KEY_HOMEPAGE` (172), `KEY_F1` and 170, the code Amazon's Appium documentation gives for Home, all leave the app on screen. To get back to the launcher, run `vega device launch-app -d VirtualDevice -a com.amazon.keplerlauncherapp.main`.
-- **Screenshots.** `getScreenshot` returns 1920x1080. A running process polls it in RGB at about 57 screenshots a second of a still screen, 17 ms each, and at 16 to 43 a second while the screen changes, 23 to 61 ms each (measured on 28 September 2026). `vvd screenshot` takes about half a second, most of it starting up. The emulator's `streamScreenshot` has delivered only its first frame while the screen kept changing, so the driver polls instead.
-- **Audio.** `streamAudio` sends nothing while the device is silent. `record` rebuilds the track on the video's clock from each packet's capture time and fills the gaps with silence, so a sound effect stays in sync.
+- **Screenshots.** `getScreenshot` returns 1920x1080. A running process polls it in RGB at about 57 screenshots a second of a still screen, 17 ms each, and at 16 to 43 a second while the screen changes, 23 to 61 ms each (measured on 28 September 2026). `vvd screenshot` takes about half a second, most of it starting up. The emulator's `streamScreenshot` has delivered only its first frame while the screen kept changing, so the driver polls instead. The device's `gwsi-tool-screenshooter` is for a Fire TV Stick: the VDA reference notes that the VVD doesn't support it, and on the VVD it wrote a 0-byte PNG or hung, with no error (28 September 2026).
+- **Audio.** `streamAudio` sends nothing while the device is silent, and while a sound plays each packet's capture time wanders by about ±10 ms around where the packet before it ends. `record` lays the packets of one sound back to back and places the sound where their capture times agree best, then fills the gaps between sounds with silence. A sound stays in sync, and a long one, such as speech or music, plays without holes ([#13](https://github.com/fortemate/vega-vvd-driver/issues/13)). Measured on the VVD.
 - **gRPC.** The endpoint is off after every start of the VVD, and the discovery file that the driver reads appears only once `grpc <port>` has been sent to the console. `vvd enable-grpc` does that.
 
 ## Troubleshooting

package/dist/record.js

@@ -7,8 +7,8 @@
 // screen keeps changing. Polling does not stop: on the VVD a 1080p screenshot
 // takes about 17 ms, or 23 to 61 ms while the screen changes. The audio comes
 // from streamAudio. The emulator sends nothing while the device is silent, so
-// the track is rebuilt on the video's clock from each packet's capture time,
-// with silence in the gaps.
+// the track is rebuilt on the video's clock from the packets' capture times,
+// with silence in the gaps; see assembleAudio for how.
 //
 // However a recording ends, it cleans up after itself: ffmpeg is stopped, the
 // audio stream is cancelled and the working directory is removed.
@@ -49,23 +49,91 @@ const abortable = (promise, signal) => {
 };
 const SAMPLE_RATE = 44100;
 const FRAME_BYTES = 4; // 16-bit stereo
+// A packet's capture time wanders by about ±10 ms around where the packet
+// before it ends, and a stream starts with smaller packets (#13). Laid each at
+// its own time, the packets of one sound would leave holes and cut into each
+// other several times a second, which is heard as a rattle. So consecutive
+// packets are laid back to back, the way the device played them, and each run
+// of them is placed as a whole where the capture times agree best: their
+// median. A run ends where the device sent nothing, a silence that leaves a
+// step in time longer than SILENCE_US, and where it has drifted from the clock
+// by more than DRIFT_US. Its first SETTLE_US are not held to the clock: there
+// the small packets' capture times run ahead of their audio, by more than
+// DRIFT_US at the start of some sounds, and splitting there would cut a hole
+// just after the sound begins.
+const SILENCE_US = 50_000;
+const DRIFT_US = 100_000;
+const SETTLE_US = 500_000;
+const framesOf = (packet) => Math.floor(packet.pcm.length / FRAME_BYTES);
+const usOf = (frames) => (frames / SAMPLE_RATE) * 1e6;
+// The packets in the order they arrived, which is the order they were played,
+// cut into runs at each silence.
+const runsOf = (packets) => {
+    const runs = [];
+    let run = [];
+    for (const packet of packets) {
+        const last = run.at(-1);
+        if (last &&
+            Math.abs(packet.timestampUs - (last.timestampUs + usOf(framesOf(last)))) >
+                SILENCE_US) {
+            runs.push(run);
+            run = [];
+        }
+        run.push(packet);
+    }
+    if (run.length)
+        runs.push(run);
+    return runs;
+};
+// Where a run's first frame belongs on a clock that starts at `startUs`, in
+// microseconds: each packet's capture time less the audio before it in the
+// run, and the median of those, which jitter and the first small packets do
+// not move.
+const anchorOf = (run, startUs) => {
+    let before = 0;
+    const offsets = run.map((packet) => {
+        const offset = packet.timestampUs - startUs - usOf(before);
+        before += framesOf(packet);
+        return offset;
+    });
+    offsets.sort((a, b) => a - b);
+    return offsets[Math.floor(offsets.length / 2)];
+};
+// The run split where a packet, past the run's first SETTLE_US of audio, has
+// drifted more than DRIFT_US from the place the run gives it, so a long sound
+// follows the clock: the part before the first such packet, and the rest.
+const steady = (run, startUs) => {
+    const anchor = anchorOf(run, startUs);
+    let before = 0;
+    for (let i = 0; i < run.length; i++) {
+        const drift = run[i].timestampUs - startUs - (anchor + usOf(before));
+        if (usOf(before) > SETTLE_US && Math.abs(drift) > DRIFT_US)
+            return [run.slice(0, i), run.slice(i)];
+        before += framesOf(run[i]);
+    }
+    return [[...run], []];
+};
 // Lays audio packets on a timeline that starts at `startUs` and lasts
-// `seconds`, as 16-bit stereo PCM: silence where nothing arrived, and a packet
-// that starts earlier than the audio before it overwrites the overlap. Packets
-// captured before the start are cut; the result is exactly as long as asked.
+// `seconds`, as 16-bit stereo PCM, with silence where nothing arrived. Audio
+// placed before the start is cut, and past the end is dropped; the result is
+// exactly as long as asked. Where two runs overlap, the later one is heard.
 export const assembleAudio = (packets, startUs, seconds) => {
     const track = Buffer.alloc(Math.round(seconds * SAMPLE_RATE) * FRAME_BYTES);
-    for (const packet of packets) {
-        const frames = Math.floor(packet.pcm.length / FRAME_BYTES);
-        let offset = Math.round(((packet.timestampUs - startUs) / 1e6) * SAMPLE_RATE);
-        let from = 0;
-        if (offset < 0) {
-            from = -offset;
-            offset = 0;
+    const pending = runsOf(packets);
+    while (pending.length) {
+        const [run, rest] = steady(pending.shift(), startUs);
+        if (rest.length)
+            pending.unshift(rest);
+        let offset = Math.round((anchorOf(run, startUs) / 1e6) * SAMPLE_RATE);
+        for (const packet of run) {
+            const frames = framesOf(packet);
+            const from = Math.max(0, -offset);
+            const at = offset + from;
+            offset += frames;
+            if (from >= frames || at * FRAME_BYTES >= track.length)
+                continue;
+            packet.pcm.copy(track, at * FRAME_BYTES, from * FRAME_BYTES, frames * FRAME_BYTES);
         }
-        if (from >= frames || offset * FRAME_BYTES >= track.length)
-            continue;
-        packet.pcm.copy(track, offset * FRAME_BYTES, from * FRAME_BYTES, frames * FRAME_BYTES);
     }
     return track;
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fortemate/vega-vvd-driver",
-  "version": "0.2.0",
+  "version": "0.2.1",
   "description": "Unofficial driver for the Vega Virtual Device: remote keys, screenshots and video with sound, from scripts and AI agents. CLI, Node library and MCP server.",
   "keywords": [
     "vega",