pawbrowse 0.6.2 → 0.6.4

package/README.md

@@ -136,30 +136,27 @@ Tips:
 ## Benchmark
 
 <p align="center">
-  <img src="assets/pawbrowse-vs-claude-in-chrome.gif" alt="Real side-by-side recording: PawBrowse vs Claude-in-Chrome on the same task" width="100%">
+  <a href="assets/benchmark.mp4"><img src="assets/benchmark.gif" alt="Side-by-side recording on live Booking.com: the same agent searches a Lisbon hotel through PawBrowse (102.5 s, 8 tool calls) and through Claude in Chrome (112.9 s, 11 tool calls)" width="100%"></a>
 </p>
 
-<p align="center"><em>Real side-by-side screen recording — same task (3 Wikipedia section jumps), same brain (Claude). PawBrowse acts in one call per click and finishes first; Claude-in-Chrome perceives-then-clicks. Measured numbers below.</em></p>
-
-Because PawBrowse keeps **stable element refs** and its `navigate`/`act` already return the fresh
-table, the agent clicks a known target in **one** round trip. Screenshot/accessibility-tree drivers
-do **perceive-then-act** — a read (or screenshot) *then* a click — paying an extra agent round trip
-and a larger payload every action.
-
-Measured task: click 5 different section links on the same Wikipedia page, averaged, same machine,
-same agent (Claude):
+<p align="center"><sub>
+Same task on live Booking.com (Lisbon, Oct 20–22, free cancellation, open the first hotel), same agent (Claude), same real Chrome.
+Each run filmed from its own tab and timed from its first action; played at 8×, clocks in real time. <a href="assets/benchmark.mp4">MP4</a>
+</sub></p>
 
-| | PawBrowse | Claude-in-Chrome |
+| | PawBrowse | Claude in Chrome |
 | --- | --- | --- |
-| Calls per click | **1** (`act` by stable ref) | 2 (`read_page` → click) |
-| Avg wall-clock per click | **~7.6 s** | ~17.0 s |
-| Perception payload | compact, viewport-only | full a11y tree w/ URLs (up to 50 KB) |
-
-> **Honest caveat:** with Claude as the shared brain, absolute wall-clock is dominated by agent
-> latency and is noisy — treat the **~2.2× ratio** as the signal, not the exact seconds. The win is
-> *structural* (fewer round trips + smaller payloads), which also means fewer tokens per step. It's
-> **not** the sub-second speed of a small, dedicated click-picking model — PawBrowse trades that
-> raw speed for a smart, general brain (Claude) with no keys and no per-click cost.
+| Wall-clock (incl. the agent's thinking) | **102.5 s** | 112.9 s |
+| Tool calls | **8** (12 actions) | 11 (19 actions) |
+| Screenshots needed to check state | **0** — every call returns the page's fresh element table | 3 |
+| Surprises handled | sign-in popup reported ("element is disabled") and dismissed; new tab followed automatically | a filter click that silently didn't apply, caught from a screenshot and retried |
+
+> **Honest caveats:** most of the time on both sides is the agent thinking between calls, so the
+> wall-clock gap is modest. The structural signal is **fewer calls and no screenshots** — PawBrowse
+> returns the page's state with every action, so the agent never has to "look again". Run 2 (Claude in
+> Chrome) also started with Booking remembering run 1's destination and dates. One run each; treat it
+> as an illustration, not a statistic. Reproduce: `scripts/demo/peek-record.mjs` films a tab,
+> `scripts/demo/render_compare.py` renders the comparison.
 
 ## How it compares
 

package/extension/background.js

@@ -18,7 +18,8 @@ let ws = null;
 let reconnectTimer = null;
 
 // Per-session state. Each session drives its own tab(s) inside its own tab group.
-const attachedTabs = new Set();           // tabIds we currently hold a debugger on
+const attachedTabs = new Set();
+const peekOnly = new Set();               // tabs attached ONLY for dev frame capture (peek)           // tabIds we currently hold a debugger on
 const sessions = new Map();               // sessionId -> { activeTabId, createdTabs:Set, groupId, num, color }
 const tabOwner = new Map();               // tabId -> sessionId (so sessions don't steal each other's tabs)
 const chains = new Map();
@@ -732,6 +733,18 @@ const SNAPSHOT = `(function(seed, opts){
     var asel=e.getAttribute('aria-selected'); if(asel!=null) base.selected=(asel==='true');
     if(e.required || e.getAttribute('aria-required')==='true') base.required=true;
     if(e.getAttribute('draggable')==='true') base.draggable=true;
+    // Site chrome (global header/nav — "Home", language picker, sign-in...) is real and stays
+    // clickable, just demoted to the end of the in-view rows so task content comes first.
+    try{ if(e.closest('nav,[role="navigation"]')) base.chrome=true; }catch(_){}
+    // Repeated links to the exact same destination (a card's photo, title, and "Opens X info"
+    // overlay often all point at the same URL): dedup key, hash dropped, bare "#"/javascript: skipped.
+    if(e.tagName==='A'){
+      // Same-PAGE hash-only hrefs ("#", "#1", "#panel-a") are cheap and collide across totally
+      // unrelated widgets (a sidebar's "#1" and a panel's "#1" resolve to the identical URL) — never
+      // dedup those. Only a link to a genuinely different page/resource is a safe, real duplicate.
+      var hattr=e.getAttribute('href')||'';
+      if(hattr && hattr.charAt(0)!=='#' && !/^javascript:/i.test(hattr)) base._dk=e.href;
+    }
     // Only surface validation errors on fields the user (or agent) has put a value in, or that the
     // page itself flags, so an untouched required form isn't a wall of warnings.
     if(e.getAttribute('aria-invalid')==='true') base.invalid='invalid';
@@ -782,10 +795,75 @@ const SNAPSHOT = `(function(seed, opts){
   var omitted=Math.max(0, inView.length-250); inView.splice(250);
   // Keep the 25 off-screen controls NEAREST the visible area (not the first 25 in DOM order, which
   // for a scrolled list are the rows furthest behind), then restore page order.
-  var OFFCAP=opts.all?400:25;
+  var OFFCAP=opts.all?400:12; // default trimmed further: the nearest few off-screen rows are rarely the next target
   var offMore=Math.max(0, offView.length-OFFCAP)+farOff;
   if(offView.length>OFFCAP){ offView.forEach(function(a,k){ a.ord=k; }); offView.sort(function(a,b){ return a.dist-b.dist; }); offView.splice(OFFCAP); offView.sort(function(a,b){ return a.ord-b.ord; }); }
+  inView.sort(function(a,b){ return (a.chrome?1:0)-(b.chrome?1:0); }); // stable: keeps relative order within each group
   var actions=inView.concat(offView);
+  // Two rows with the IDENTICAL displayed label where one element contains the other, sitting at
+  // essentially the same spot, are the same control seen twice by our own heuristics — an outer
+  // clickable "cell" (aria-label="Tuesday, October 20, 2026") wrapping the real checkbox/radio/link
+  // with the same name inside it (a calendar day, a selectable list row). A small pixel tolerance
+  // absorbs the cell's own border/padding around an absolutely-positioned inner control. Keep the
+  // one with richer state (checked/selected/expanded) when only one side has it, else the innermost.
+  try{
+    var byLbl={};
+    actions.forEach(function(a){ var k=a.kind+'|'+a.label; (byLbl[k]=byLbl[k]||[]).push(a); });
+    var dropR=new Set();
+    Object.keys(byLbl).forEach(function(key){
+      var grp=byLbl[key]; if(grp.length<2) return;
+      for(var gi=0; gi<grp.length; gi++){
+        for(var gj=0; gj<grp.length; gj++){
+          if(gi===gj) continue;
+          var A=grp[gi], B=grp[gj]; if(dropR.has(A)||dropR.has(B)) continue;
+          var ea=cache.nodes.get(A.node), eb=cache.nodes.get(B.node);
+          if(!ea||!eb||ea===eb) continue;
+          var contains; try{ contains=eb.contains(ea); }catch(_){ contains=false; }
+          if(!contains) continue; // B is an ancestor of A, same label
+          if(Math.abs(A.x-B.x)>8 || Math.abs(A.y-B.y)>8) continue; // not the same visual spot: leave both
+          var richA=A.checked!=null||A.selected!=null||A.expanded!=null;
+          var richB=B.checked!=null||B.selected!=null||B.expanded!=null;
+          if(richB && !richA) dropR.add(A); else dropR.add(B); // prefer the innermost, unless only the ancestor carries state
+        }
+      }
+    });
+    if(dropR.size) actions=actions.filter(function(a){ return !dropR.has(a); });
+  }catch(_){}
+  // Repeated links to the same destination (see _dk above): keep the clearest copy (in view, not
+  // covered, shortest label), drop the rest.
+  try{
+    var byHref={};
+    actions.forEach(function(a){ if(a._dk) (byHref[a._dk]=byHref[a._dk]||[]).push(a); });
+    var dropH=new Set();
+    Object.keys(byHref).forEach(function(href){
+      var grp=byHref[href]; if(grp.length<2) return;
+      // Labels get combined below, so there's no information reason to prefer the shorter one —
+      // prefer the LARGER, more prominent element as the surviving click target's geometry instead
+      // (a tiny trailing ")" fragment next to a wide title link would otherwise become the anchor).
+      var score=function(a){ return (a.off?2:0)+(a.covered?2:0)-Math.min((a.w||0)*(a.h||0),40000)/40000; };
+      // Cluster by PROXIMITY first (a navbar shortcut and a hero CTA can legitimately share a
+      // destination while being two different, intentionally separate controls); only within the
+      // same small visual area is one of them a genuine repeat of the other.
+      var used=new Array(grp.length).fill(false);
+      for(var gi=0; gi<grp.length; gi++){
+        if(used[gi]) continue;
+        var cluster=[grp[gi]]; used[gi]=true;
+        for(var gj=gi+1; gj<grp.length; gj++){
+          if(used[gj]) continue;
+          if(Math.abs(grp[gi].x-grp[gj].x)<=200 && Math.abs(grp[gi].y-grp[gj].y)<=200){ cluster.push(grp[gj]); used[gj]=true; }
+        }
+        if(cluster.length<2) continue;
+        var keep=cluster.reduce(function(b,a){ return score(a)<score(b) ? a : b; });
+        // Different labels on the same destination ("7 hours ago" and "197 comments" both open the
+        // same story) each carry real information — combine them into the surviving row instead of
+        // silently discarding one, so the agent never loses "197 comments" for "7 hours ago".
+        var uniq=[]; cluster.forEach(function(a){ if(uniq.indexOf(a.label)<0) uniq.push(a.label); });
+        if(uniq.length>1) keep.label=clean(uniq.join(' · '),140);
+        cluster.forEach(function(a){ if(a!==keep) dropH.add(a); });
+      }
+    });
+    if(dropH.size) actions=actions.filter(function(a){ return !dropH.has(a); });
+  }catch(_){}
   // The nearest ancestor text that isn't just the label itself: which row/item a control is in.
   cache.ctxOf=function(el, lab){ return ctxOf(el, lab); };
   function ctxOf(el, lab){
@@ -1199,12 +1277,13 @@ async function observe(tabId, opts) {
 
 // Re-resolve a ref to its live element, re-check it, and hit-test the center
 // (elementFromPoint containment) so we never click a stale/covered/wrong target.
-async function resolveHit(tabId, ref, opts) {
+async function resolveHit(tabId, ref, opts, _retries) {
+  _retries = _retries || 0;
   const forFill = opts && opts.fill ? 'true' : 'false';
   const noScroll = opts && opts.noScroll ? 'true' : 'false', noHit = opts && opts.noHit ? 'true' : 'false';
   const TXT = opts && opts.text != null ? JSON.stringify(String(opts.text)) : 'null';
   const R = JSON.stringify(String(ref));
-  return evaluate(tabId, `(function(){
+  const result = await evaluate(tabId, `(function(){
     var c=window.__pawbrowse; if(!c||!c.byId) return {error:'no snapshot yet; observe first'};
     if(c.byId[${R}]==null) return {error:'unknown ref (observe again)'};
     var e=c.get?c.get(${R}):null;
@@ -1260,6 +1339,27 @@ async function resolveHit(tabId, ref, opts) {
     if(!p.hit) return {error:'element is covered by another element (dismiss the overlay/dialog first)'};
     return {x:p.x, y:p.y};
   })()`);
+  // Any "(observe again)"-flavored failure can mean the page is mid-re-render — from a just-prior
+  // action, or from something async and independent of us entirely (a sidebar widget that hydrates
+  // on its own timer) — rather than the target being truly gone. Up to two free retries with a short
+  // growing delay, transparent to the caller: costs nothing on success, and still surfaces a real
+  // failure once retries run out.
+  //
+  // 'unknown ref' means byId[ref] itself is missing — there is no fingerprint left to match against,
+  // so only a fresh full snapshot() (which runs the "gone element -> matching new element" rebind for
+  // EVERY ref) has any chance of reviving it. For every other case (element/row disconnected or
+  // changed), byId[ref]'s fingerprint is still intact, and c.get()'s own live-DOM fallback (above)
+  // already re-scans the page fresh on every call — so we must NOT call snapshot() here: a full
+  // rebuild resets byId/guards/fps for ALL refs from whatever's on the page at that instant, and if it
+  // happens to land while our target is still absent (e.g. mid-hydration), it erases the very
+  // fingerprint a LATER, successful retry would need — permanently orphaning a ref that would
+  // otherwise have recovered on its own.
+  if(result && result.error && /observe again/i.test(result.error) && _retries < 2){
+    await sleep(_retries === 0 ? 120 : 300);
+    if(result.error === 'unknown ref (observe again)'){ try { await snapshot(tabId, 1); } catch {} }
+    return resolveHit(tabId, ref, opts, _retries + 1);
+  }
+  return result;
 }
 
 // Set the value of a date/time/month/week/color/range input the way a user's picker would: via the
@@ -2145,6 +2245,25 @@ async function handleCommand(cmd, args, token, session) {
         return `ran ${ops.length} op(s) [${note}]:\n${logLines.join('\n')}\n${takeDialogLog(tabId)}\n${(changed && deltaTable(seenFull, table)) || table}`;
       });
     }
+    case 'peek': {
+      // Dev-only frame capture for recordings: screenshot ANY tab by id without adopting it into a
+      // session, grouping it, or enabling domains on it (so another tool driving that tab — e.g. for
+      // a side-by-side benchmark — is not disturbed). Attaches the debugger only if needed.
+      const tabId = Number(args.tabId);
+      let t; try { t = await chrome.tabs.get(tabId); } catch { throw new Error(`tab ${args.tabId} not found`); }
+      if (restrictedPage(t.url)) throw new Error('that tab is a browser page that cannot be captured');
+      if (!attachedTabs.has(tabId)) {
+        await new Promise((res, rej) => chrome.debugger.attach({ tabId }, '1.3', () => { const e = chrome.runtime.lastError; if (e && !/already attached/i.test(e.message)) rej(new Error(e.message)); else res(); }));
+        peekOnly.add(tabId);
+      }
+      const q = Math.max(20, Math.min(90, Number(args.quality) || 60));
+      const shot = await sendCdp(tabId, 'Page.captureScreenshot', { format: 'jpeg', quality: q });
+      return { data: shot.data, t: Date.now(), url: t.url };
+    }
+    case 'peek_end': {
+      for (const id of [...peekOnly]) { peekOnly.delete(id); if (!attachedTabs.has(id)) await new Promise((r) => chrome.debugger.detach({ tabId: id }, () => { void chrome.runtime.lastError; r(); })); }
+      return { ok: true };
+    }
     case 'screenshot': {
       const tabId = await resolveTabId(session, args, 'inspect');
       await attach(tabId);

package/extension/manifest.json

@@ -1,7 +1,7 @@
 {
   "manifest_version": 3,
   "name": "PawBrowse",
-  "version": "0.6.2",
+  "version": "0.6.4",
   "description": "Let your AI agent (Claude Code) drive your real Chrome. Fast element-table control. Open source, no keys.",
   "minimum_chrome_version": "125",
   "permissions": [

package/mcp/server.mjs

@@ -211,6 +211,16 @@ const TOOLS = [
   },
 ];
 
+// Dev-only tools (recording benchmarks): hidden unless PAWBROWSE_DEV_TOOLS=1.
+if (process.env.PAWBROWSE_DEV_TOOLS === '1') {
+  TOOLS.push({
+    name: 'browser_peek',
+    description: 'DEV: screenshot any tab by id (JPEG base64 + capture time) without adopting or changing it. For recording side-by-side benchmarks.',
+    inputSchema: { type: 'object', properties: { tabId: { type: 'number' }, quality: { type: 'number' }, end: { type: 'boolean' } } },
+    annotations: { title: 'Peek at a tab (dev)', readOnlyHint: true, openWorldHint: true },
+  });
+}
+
 function textResult(obj) {
   const text = typeof obj === 'string' ? obj : JSON.stringify(obj, null, 2);
   return { content: [{ type: 'text', text }] };
@@ -237,6 +247,10 @@ async function callTool(name, args) {
       return textResult(await callExtension('act', args));
     }
     case 'browser_assert':  return textResult(await callExtension('assert', args));
+    case 'browser_peek': {
+      if (process.env.PAWBROWSE_DEV_TOOLS !== '1') throw new Error('unknown tool: browser_peek');
+      return textResult(await callExtension(args.end ? 'peek_end' : 'peek', args));
+    }
     case 'browser_screenshot': {
       const r = await callExtension('screenshot', args);
       return { content: [{ type: 'image', data: r.data, mimeType: r.mimeType }, { type: 'text', text: r.note }] };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pawbrowse",
-  "version": "0.6.2",
+  "version": "0.6.4",
   "description": "Let Claude Code drive your real, logged-in Chrome. A zero-dependency MCP server + a Chrome MV3 extension that uses element-table perception over CDP. The calling agent is the policy: no second model, no API keys, page snapshots never leave for a third party.",
   "type": "module",
   "bin": {