jev-cdp 0.1.6 → 0.1.7

package/README.md

@@ -27,8 +27,8 @@ This is an early experimental port. See [NOTICE.md](NOTICE.md) for source attrib
 Run the published CLI without adding it to a project. Bun must be installed for either command:
 
 ```bash
-bunx jev-cdp@0.1.6 help run
-npx -y jev-cdp@0.1.6 help run
+bunx jev-cdp@0.1.7 help run
+npx -y jev-cdp@0.1.7 help run
 ```
 
 Chrome with a CDP endpoint and `TYPESAFE_API_KEY` are required for browser runs. FFmpeg with H.264 encoding (`libx264`) is required for `--recording`. Jev selects `-fps_mode vfr` when FFmpeg supports it and falls back to `-vsync vfr` for older builds. Run `jev-cdp doctor` to check the installed encoder with a short MP4 encode and decode.
@@ -96,13 +96,13 @@ bun run run -- run \
   --final-state
 ```
 
-`--recording` captures Chrome's compositor screencast stream for the full goal and renders an H.264 MP4. Because Chrome's native pointer is not part of that stream, the adapter draws a high-contrast cursor that starts at the viewport center, glides to each target, and pulses on clicks. `--interaction-pauses` adds a deterministic delay after opening or loading a page, switching to an attached tab, or changing the URL within a page. Jev chooses during that delay, and the adapter waits only for any time left before acting. The same flag also pauses after moving to a click target and before pressing the mouse. Jev does not choose or observe these delays. `--screenshot` saves the final viewport after the goal stops; when recording is also enabled, it reuses the final screencast frame.
+`--recording` captures Chrome's compositor screencast stream for the full goal and renders a 1280×720 H.264 MP4. A 40-pixel browser bar shows the active tab's URL above the page. Chrome uses a 1506×800 viewport, scaled to fit the video beneath the bar, so controls remain at their observed coordinates while more of the page fits on screen. Because Chrome's native pointer is not part of the stream, the adapter draws a high-contrast cursor that starts at the viewport center, glides to each target, and pulses on clicks. `--interaction-pauses` adds a deterministic delay after opening or loading a page, changing the URL within a page, or moving to a click target before pressing the mouse. Jev chooses during page pauses, and the adapter waits only for any time left before acting. Jev does not choose or observe those delays. When an action opens a new tab, the recording shows a "New tab opened" notice over the previous tab for the full configured pause, then switches to the new tab. With zero pause, the notice appears for one frame without delaying Jev. `--screenshot` saves a 1280×720 image with the same browser bar, even when recording is off. If a new tab opened, the final screenshot also includes the notice.
 
 `jev-cdp run` writes JSON Lines to standard output: one `type:"action"` object per executed action, followed by one `type:"result"` object with the final status and action budget. Action objects include elapsed execution time, the operation, page and tab URLs, the viewport, and a CSS selector, role, name, frame, and coordinates for the element. Fill actions include the entered text; password fields and `--field-value-env` values are redacted and must be supplied separately for replay. Select and scroll actions include their option value or wheel delta. The CSS selector and page URL can be used as Playwright replay targets, with the coordinates as a fallback at the recorded viewport size. `--final-state` adds the final semantic page snapshot: a frame tree with URLs and loading state, actionable elements with frame identity, screen bounds, nearby text, region, clickability, and any element covering the click point, plus new-tab and redirect transitions. Automatic navigation and embedded-content waits use `--wait-budget-ms` (15 seconds by default) and do not consume `--max-steps`; a timeout returns `wait_timeout`, its pending condition and elapsed wait time, and the latest semantic state. Runtime failures emit a `type:"result"` object with `status:"error"`; diagnostic text uses standard error.
 
 Each action object includes `consoleErrors` observed during that step. The result object includes `initialConsoleErrors` already present when attaching to the tab and `consoleErrors` for the full run. These fields include browser console errors, uncaught exceptions, and error-level DevTools log entries from the page and its frames. Review them alongside the semantic state; a console error does not by itself establish that the goal failed.
 
-The snapshot includes visible controls in nested iframes, including cross-origin frames. Clicking a link there uses its frame coordinates and checks that the observed control is still current. If that click opens a new tab, Jev switches to it, returns its target ID for the next goal, and keeps the recording and 1120×780 viewport consistent across the switch.
+The snapshot includes visible controls in nested iframes, including cross-origin frames. Clicking a link there uses its frame coordinates and checks that the observed control is still current. If that click opens a new tab, Jev switches to it, returns its target ID for the next goal, and keeps the 1506×800 viewport consistent across the switch.
 
 ## Supply known field values without another LLM
 
@@ -184,9 +184,9 @@ The context is disposed after evidence capture by default. Combine it with `--ke
 | `--cdp URL` | `CHROME_CDP_URL` | `http://127.0.0.1:9222` | Chrome DevTools HTTP endpoint |
 | `--tab TARGET_ID` | — | create a new tab | Attach to one exact existing Chrome page target |
 | `tabs` | — | — | Print target IDs, titles, and URLs for open page tabs |
-| `--recording PATH.mp4` | — | off | Record the complete goal with an animated cursor |
+| `--recording PATH.mp4` | — | off | Record a 1280×720 video with a URL bar, animated cursor, and tab notices |
 | `--interaction-pauses MS` | — | `0` | Pause after page loads and before clicks, overlapping page pauses with Jev decisions |
-| `--screenshot PATH.jpg` | — | off | Save the final browser viewport |
+| `--screenshot PATH.jpg` | — | off | Save a 1280×720 image with the URL bar and any new-tab notice |
 | `--final-state` | — | off | Include the final semantic page state in stdout JSON |
 | `--field-value LABEL=VALUE` | — | Luna fallback | Type caller-provided test data into the exactly labeled field |
 | `--field-value-env LABEL=NAME` | — | off | Read a sensitive field value from an environment variable |

package/dist/cli.js

@@ -3,7 +3,7 @@
 // package.json
 var package_default = {
   name: "jev-cdp",
-  version: "0.1.6",
+  version: "0.1.7",
   description: "A small Jev-powered bridge to Chrome through the Chrome DevTools Protocol.",
   type: "module",
   license: "MIT",
@@ -225,6 +225,82 @@ file '${quoted}'
   }
 }
 
+// src/recording-visual.ts
+var VIDEO_WIDTH = 1280;
+var VIDEO_HEIGHT = 720;
+var BAR_HEIGHT = 40;
+var PAGE_WIDTH = 1506;
+var PAGE_HEIGHT = 800;
+function composeFrameExpression(jpeg, url, newTabUrl) {
+  return `(async () => {
+    const bytes = Uint8Array.from(atob(${JSON.stringify(jpeg)}), character => character.charCodeAt(0));
+    const bitmap = await createImageBitmap(new Blob([bytes], { type: 'image/jpeg' }));
+    const canvas = document.createElement('canvas');
+    canvas.width = ${VIDEO_WIDTH}; canvas.height = ${VIDEO_HEIGHT};
+    const context = canvas.getContext('2d');
+    if (!context) throw new Error('Could not create recording canvas');
+    context.fillStyle = '#f5f7fa';
+    context.fillRect(0, 0, canvas.width, ${BAR_HEIGHT});
+    context.fillStyle = '#ffffff';
+    context.beginPath();
+    context.roundRect(12, 6, canvas.width - 24, 28, 7);
+    context.fill();
+    context.strokeStyle = '#d7dce4';
+    context.lineWidth = 1;
+    context.stroke();
+    context.fillStyle = '#617185';
+    context.beginPath(); context.arc(28, 20, 4, 0, Math.PI * 2); context.fill();
+    const fit = (value, maxWidth) => {
+      if (context.measureText(value).width <= maxWidth) return value;
+      while (value.length && context.measureText(value + '\u2026').width > maxWidth) value = value.slice(0, -1);
+      return value + '\u2026';
+    };
+    const url = ${JSON.stringify(url)};
+    context.font = '14px system-ui, sans-serif';
+    context.fillStyle = '#263343';
+    context.textBaseline = 'middle';
+    context.fillText(fit(url, canvas.width - 72), 42, 20);
+    const chromeBorder = '#61738a';
+    context.fillStyle = chromeBorder;
+    context.fillRect(0, ${BAR_HEIGHT - 2}, canvas.width, 2);
+    context.drawImage(bitmap, 0, ${BAR_HEIGHT}, canvas.width, canvas.height - ${BAR_HEIGHT});
+    bitmap.close();
+    const newTabUrl = ${JSON.stringify(newTabUrl)};
+    if (newTabUrl) {
+      const cardWidth = 620, cardHeight = 104, radius = 12, x = (canvas.width - cardWidth) / 2;
+      const cardPath = () => {
+        context.beginPath();
+        context.moveTo(x, 0);
+        context.lineTo(x + cardWidth, 0);
+        context.lineTo(x + cardWidth, cardHeight - radius);
+        context.quadraticCurveTo(x + cardWidth, cardHeight, x + cardWidth - radius, cardHeight);
+        context.lineTo(x + radius, cardHeight);
+        context.quadraticCurveTo(x, cardHeight, x, cardHeight - radius);
+        context.closePath();
+      };
+      context.save();
+      context.shadowColor = 'rgba(15, 23, 42, .24)';
+      context.shadowBlur = 14;
+      context.shadowOffsetY = 5;
+      context.fillStyle = '#ffffff';
+      cardPath(); context.fill();
+      context.restore();
+      context.strokeStyle = chromeBorder;
+      context.lineWidth = 1.5;
+      cardPath(); context.stroke();
+      context.fillStyle = '#2563eb';
+      context.beginPath(); context.arc(x + 28, 35, 7, 0, Math.PI * 2); context.fill();
+      context.fillStyle = '#172033';
+      context.font = '600 19px system-ui, sans-serif';
+      context.fillText('New tab opened', x + 48, 35);
+      context.fillStyle = '#536277';
+      context.font = '14px system-ui, sans-serif';
+      context.fillText(fit(newTabUrl, cardWidth - 52), x + 24, 72);
+    }
+    return canvas.toDataURL('image/jpeg', .88).split(',')[1];
+  })()`;
+}
+
 // src/snapshot.js
 var snapshot_default = `// Ported from browser-use/jev-ultrafast at commit 452c1ad2 under the MIT License.
 (() => {
@@ -417,6 +493,12 @@ class Browser {
   #recordingStartedAt = 0;
   #recordingSequence = 0;
   #recordingWrites = Promise.resolve();
+  #activeUrl = "about:blank";
+  #recordingUrlEvents = [];
+  #newTabNoticeUntil = 0;
+  #newTabNoticeUrl = null;
+  #newTabNoticeWindows = [];
+  #hadNewTab = false;
   #stopRecordingEvents;
   #stopPageEvents = [];
   #mainFrameId;
@@ -512,6 +594,8 @@ class Browser {
         }
         if (!info?.targetId || !info.url || info.url === "about:blank")
           return;
+        if (info.targetId === browser.#targetId)
+          browser.setActiveUrl(info.url);
         const urls = browser.#targetUrls.get(info.targetId) ?? [];
         if (urls.at(-1) !== info.url)
           urls.push(info.url);
@@ -537,8 +621,8 @@ class Browser {
       await browser.watchConsole();
       await browser.call("Page.enable");
       await browser.call("Emulation.setDeviceMetricsOverride", {
-        width: 1120,
-        height: 780,
+        width: PAGE_WIDTH,
+        height: PAGE_HEIGHT,
         deviceScaleFactor: 1,
         mobile: false
       });
@@ -547,6 +631,7 @@ class Browser {
       if (options.url)
         await browser.call("Page.navigate", { url: options.url });
       await browser.waitForReady();
+      browser.#activeUrl = await browser.evaluate("location.href") ?? browser.#activeUrl;
       browser.#pauseUntil = performance.now() + browser.#interactionPauses;
       await browser.startRecording();
       return browser;
@@ -563,6 +648,18 @@ class Browser {
   get targetId() {
     return this.#targetId;
   }
+  setActiveUrl(url) {
+    if (!url || url === this.#activeUrl)
+      return;
+    this.#activeUrl = url;
+    if (this.#recordingDirectory && this.#recordingFrames.length) {
+      this.#recordingUrlEvents.push({
+        elapsedMs: performance.now() - this.#recordingStartedAt,
+        targetId: this.#targetId,
+        url
+      });
+    }
+  }
   takeConsoleErrors() {
     return this.#consoleErrors.splice(0);
   }
@@ -606,6 +703,8 @@ class Browser {
     await this.call("Page.enable");
     const tree = await this.call("Page.getFrameTree");
     this.#mainFrameId = tree.frameTree.frame.id;
+    if (tree.frameTree.frame.url)
+      this.setActiveUrl(tree.frameTree.frame.url);
     await this.call("Runtime.addBinding", { name: "__jevDomChanged" });
     const observeDom = `(() => {
       if (window.__jevDomWatching) return;
@@ -642,8 +741,11 @@ class Browser {
     this.#stopPageEvents.push(this.#cdp.on("Page.frameStartedLoading", this.#sessionId, loading), this.#cdp.on("Page.frameNavigated", this.#sessionId, (params) => {
       const frame = params.frame;
       this.signalChange();
-      if (frame && frame.id === this.#mainFrameId && !frame.parentId)
+      if (frame && frame.id === this.#mainFrameId && !frame.parentId) {
+        if (frame.url)
+          this.setActiveUrl(frame.url);
         loading({ frameId: frame.id });
+      }
     }), this.#cdp.on("Page.loadEventFired", this.#sessionId, loaded), this.#cdp.on("Page.frameStoppedLoading", this.#sessionId, (params) => {
       if (typeof params.frameId === "string")
         this.#loadingFrames.delete(params.frameId);
@@ -653,6 +755,8 @@ class Browser {
     }), this.#cdp.on("Page.navigatedWithinDocument", this.#sessionId, (params) => {
       this.signalChange();
       if (params.frameId === this.#mainFrameId) {
+        if (typeof params.url === "string")
+          this.setActiveUrl(params.url);
         this.#pauseUntil = performance.now() + this.#interactionPauses;
         this.resetRecordingCursor().catch(() => {
           return;
@@ -728,7 +832,14 @@ class Browser {
     await Bun.write(initialPath, Buffer.from(initial.data, "base64"));
     if (firstSegment)
       this.#recordingStartedAt = performance.now();
-    this.#recordingFrames.push({ path: initialPath, elapsedMs: firstSegment ? 0 : performance.now() - this.#recordingStartedAt });
+    const initialElapsedMs = firstSegment ? 0 : performance.now() - this.#recordingStartedAt;
+    this.#recordingFrames.push({
+      path: initialPath,
+      elapsedMs: initialElapsedMs,
+      targetId: this.#targetId,
+      url: this.#activeUrl,
+      newTab: false
+    });
     this.#stopRecordingEvents = this.#cdp.on("Page.screencastFrame", this.#sessionId, (params) => {
       const frame = params;
       this.call("Page.screencastFrameAck", { sessionId: frame.sessionId }).catch(() => {
@@ -736,20 +847,49 @@ class Browser {
       });
       const sequence = this.#recordingSequence++;
       const path = join2(this.#recordingDirectory, `${String(sequence).padStart(6, "0")}.jpg`);
-      const elapsedMs = sequence ? performance.now() - this.#recordingStartedAt : 0;
+      const capturedAt = performance.now();
+      const elapsedMs = sequence ? capturedAt - this.#recordingStartedAt : 0;
+      const url = this.#activeUrl;
+      const targetId = this.#targetId;
+      const notice = capturedAt < this.#newTabNoticeUntil;
+      const noticeUrl = notice ? this.#newTabNoticeUrl ?? undefined : undefined;
       this.#recordingWrites = this.#recordingWrites.then(async () => {
         await Bun.write(path, Buffer.from(frame.data, "base64"));
-        this.#recordingFrames.push({ path, elapsedMs });
+        this.#recordingFrames.push({
+          path,
+          elapsedMs,
+          targetId,
+          url,
+          newTab: notice,
+          ...noticeUrl ? { noticeUrl } : {}
+        });
       });
     });
     await this.call("Page.startScreencast", {
       format: "jpeg",
       quality: 70,
-      maxWidth: 1120,
-      maxHeight: 780,
+      maxWidth: PAGE_WIDTH,
+      maxHeight: PAGE_HEIGHT,
       everyNthFrame: 3
     });
   }
+  async recordNewTabNotice(url) {
+    if (!this.#recordingPath || !this.#recordingDirectory)
+      return;
+    const capture = await this.call("Page.captureScreenshot", { format: "jpeg", quality: 70 });
+    const elapsedMs = performance.now() - this.#recordingStartedAt;
+    const path = join2(this.#recordingDirectory, `${String(this.#recordingSequence++).padStart(6, "0")}.jpg`);
+    await Bun.write(path, Buffer.from(capture.data, "base64"));
+    this.#recordingFrames.push({
+      path,
+      elapsedMs,
+      targetId: this.#targetId,
+      url: this.#activeUrl,
+      newTab: true,
+      noticeUrl: url
+    });
+    return elapsedMs;
+  }
   async finishRecording() {
     if (!this.#recordingPath || !this.#recordingDirectory)
       return;
@@ -762,8 +902,53 @@ class Browser {
     await this.#recordingWrites;
     if (!this.#recordingFrames.length)
       throw new Error("Recording produced no browser frames");
+    const stoppedAt = performance.now() - this.#recordingStartedAt;
+    this.#recordingFrames.sort((a, b) => a.elapsedMs - b.elapsedMs);
+    for (const event of this.#recordingUrlEvents) {
+      let index = this.#recordingFrames.length - 1;
+      while (index >= 0 && (this.#recordingFrames[index].elapsedMs >= event.elapsedMs || this.#recordingFrames[index].targetId !== event.targetId))
+        index--;
+      const before = this.#recordingFrames[index];
+      if (!before || before.url === event.url)
+        continue;
+      const path = join2(this.#recordingDirectory, `${String(this.#recordingSequence++).padStart(6, "0")}.jpg`);
+      await Bun.write(path, Bun.file(before.path));
+      const frame = {
+        path,
+        elapsedMs: event.elapsedMs,
+        targetId: event.targetId,
+        url: event.url,
+        newTab: this.#newTabNoticeWindows.some((window) => event.elapsedMs >= window.startMs && event.elapsedMs < window.endMs),
+        noticeUrl: before.noticeUrl
+      };
+      this.#recordingFrames.push(frame);
+      this.#recordingFrames.sort((a, b) => a.elapsedMs - b.elapsedMs);
+    }
+    for (const { endMs } of this.#newTabNoticeWindows) {
+      if (this.#interactionPauses > 0)
+        continue;
+      let index = this.#recordingFrames.length - 1;
+      while (index >= 0 && this.#recordingFrames[index].elapsedMs >= endMs)
+        index--;
+      const before = this.#recordingFrames[index];
+      if (!before?.newTab || this.#recordingFrames[index + 1]?.elapsedMs === endMs)
+        continue;
+      const path = join2(this.#recordingDirectory, `${String(this.#recordingSequence++).padStart(6, "0")}.jpg`);
+      await Bun.write(path, Bun.file(before.path));
+      this.#recordingFrames.splice(index + 1, 0, {
+        path,
+        elapsedMs: endMs,
+        targetId: before.targetId,
+        url: before.url,
+        newTab: false
+      });
+    }
+    for (const frame of this.#recordingFrames) {
+      const jpeg = Buffer.from(await Bun.file(frame.path).arrayBuffer()).toString("base64");
+      await Bun.write(frame.path, Buffer.from(await this.composeFrame(jpeg, frame.url, frame.newTab ? frame.noticeUrl ?? frame.url : null), "base64"));
+    }
     await mkdir(dirname(this.#recordingPath), { recursive: true });
-    const finishedAt = performance.now() - this.#recordingStartedAt;
+    const finishedAt = Math.max(stoppedAt, this.#recordingFrames.at(-1).elapsedMs + 40);
     const quoted = (path) => path.replaceAll("'", "'\\''");
     const lines = ["ffconcat version 1.0"];
     for (let index = 0;index < this.#recordingFrames.length; index++) {
@@ -785,16 +970,15 @@ class Browser {
     if (!this.#screenshotPath)
       return;
     await mkdir(dirname(this.#screenshotPath), { recursive: true });
-    if (this.#recordingDirectory) {
-      await this.#recordingWrites;
-      const finalFrame = this.#recordingFrames.at(-1);
-      if (finalFrame) {
-        await Bun.write(this.#screenshotPath, Bun.file(finalFrame.path));
-        return;
-      }
-    }
     const capture = await this.call("Page.captureScreenshot", { format: "jpeg", quality: 90 });
-    await Bun.write(this.#screenshotPath, Buffer.from(capture.data, "base64"));
+    const url = await this.evaluate("location.href") ?? this.#activeUrl;
+    await Bun.write(this.#screenshotPath, Buffer.from(await this.composeFrame(capture.data, url, this.#hadNewTab ? url : null), "base64"));
+  }
+  async composeFrame(jpeg, url, newTabUrl) {
+    const composed = await this.evaluate(composeFrameExpression(jpeg, url, newTabUrl), true);
+    if (!composed)
+      throw new Error("Could not compose browser recording frame");
+    return composed;
   }
   async animateCursor(x, y, click = false) {
     if (!this.#recordingPath)
@@ -941,13 +1125,25 @@ class Browser {
       const attached = await this.#cdp.command("Target.attachToTarget", { targetId: target.id, flatten: true });
       try {
         await this.#cdp.command("Emulation.setDeviceMetricsOverride", {
-          width: 1120,
-          height: 780,
+          width: PAGE_WIDTH,
+          height: PAGE_HEIGHT,
           deviceScaleFactor: 1,
           mobile: false
         }, attached.sessionId);
       } finally {
         if (this.#switchOnPopup) {
+          const noticeStart = await this.recordNewTabNotice(target.url);
+          const noticeStartAt = noticeStart === undefined ? performance.now() : this.#recordingStartedAt + noticeStart;
+          this.#newTabNoticeUrl = target.url;
+          this.#newTabNoticeUntil = this.#interactionPauses > 0 ? Infinity : 0;
+          const noticeWindow = noticeStart === undefined ? undefined : {
+            startMs: noticeStart,
+            endMs: noticeStart + Math.max(40, this.#interactionPauses)
+          };
+          if (noticeWindow)
+            this.#newTabNoticeWindows.push(noticeWindow);
+          if (this.#interactionPauses > 0)
+            await Bun.sleep(Math.max(0, noticeStartAt + this.#interactionPauses - performance.now()));
           if (this.#recordingPath) {
             await this.call("Page.stopScreencast").catch(() => {
               return;
@@ -956,13 +1152,20 @@ class Browser {
             this.#stopRecordingEvents = undefined;
             await this.#recordingWrites;
           }
+          if (noticeWindow && this.#interactionPauses > 0)
+            noticeWindow.endMs = performance.now() - this.#recordingStartedAt;
+          this.#newTabNoticeUrl = null;
+          this.#newTabNoticeUntil = 0;
           this.#sessionId = attached.sessionId;
           this.#targetId = target.id;
+          this.#activeUrl = target.url;
           this.#loadingFrames.clear();
           await this.watchConsole();
           await this.watchPageLoads();
           this.#switchOnPopup = false;
+          this.#hadNewTab = true;
           await this.startRecording();
+          this.#pauseUntil = performance.now();
         } else {
           await this.#cdp.command("Target.detachFromTarget", { sessionId: attached.sessionId });
         }
@@ -1140,7 +1343,7 @@ ${child.text}`.slice(0, 6000);
     const page = { ...info, frames, transitions: [...this.#transitions], fingerprint: fingerprint(info) };
     if (screenshot) {
       const capture = await this.call("Page.captureScreenshot", { format: "jpeg", quality: 72 });
-      page.screenshot = capture.data;
+      page.screenshot = await this.composeFrame(capture.data, info.url, performance.now() < this.#newTabNoticeUntil ? this.#newTabNoticeUrl : null);
     }
     return page;
   }
@@ -2045,7 +2248,7 @@ Known field values:
 Evidence and output:
   --recording <file.mp4>         Record the complete goal with an animated cursor.
                                  Requires FFmpeg.
-  --screenshot <file.jpg>        Save the final browser viewport.
+  --screenshot <file.jpg>        Save a 1280x720 image with the URL bar.
   --final-state                  Add the final semantic page state to stdout JSON.
   -h, --help                     Show this help and exit.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "jev-cdp",
-  "version": "0.1.6",
+  "version": "0.1.7",
   "description": "A small Jev-powered bridge to Chrome through the Chrome DevTools Protocol.",
   "type": "module",
   "license": "MIT",