barebrowse 0.5.4 → 0.5.6

package/CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## 0.5.6
+
+Assess now works on bot-blocking EU sites. Headed fallback + consent fix.
+
+### Assess headed fallback (`mcp-server.js`)
+- Assess tries headless first; if result looks bot-blocked (score ≤5, all zeros), retries with a separate `connect({ mode: 'headed' })` session
+- Previously all assess scans ran headless-only — sites like Lufthansa, Coolblue, Rabobank returned score 5 (empty page behind bot wall)
+- Now: Lufthansa 50/high, Coolblue 55/high, Rabobank 75/critical
+
+### Consent dismissal improvements (`src/consent.js`)
+- Tab `goto()` now runs `dismissConsent()` (was missing — consent walls blocked all trackers from loading, making assess see a clean page)
+- Added `/\baccepteren\b/i` Dutch pattern (Rabobank uses bare "ACCEPTEREN" without "alles")
+- realClick fallback: if jsClick doesn't dismiss the CMP (button disappears from ARIA but overlay stays), retries with real `Input.dispatchMouseEvent` mouse click
+- Both dialog-scoped and global consent paths now have the jsClick→realClick fallback
+
+### createTab consent (`src/index.js`)
+- `createTab().goto()` now dismisses consent after navigation (same as main page `goto()`)
+
+## 0.5.5
+
+Fix assess tab leak and Linux shared memory crash.
+
+### Assess tab leak (`mcp-server.js`)
+- Fixed: successful assess calls never closed the tab — tabs accumulated, eating RAM until Chromium crashed
+- `tab.close()` now called in the success path (was only in error/timeout paths)
+- This was the root cause of crashes on heavy EU cookie consent sites (zalando.de, otto.de, etc.) — not the CMPs themselves
+
+### Chromium launch (`src/chromium.js`)
+- Added `--disable-dev-shm-usage` flag — prevents shared memory crashes on Linux systems with limited `/dev/shm`
+
+### Tests
+- New: `createTab()` suite in browse.test.js (2 tests)
+  - Creates 5 tabs, navigates each, closes all, verifies no zombie tabs remain
+  - Tab close is idempotent (double-close doesn't throw)
+- 71/71 passing (was 69)
+
 ## 0.5.4
 
 Assess tool hardened: session reuse, concurrency, self-healing.

package/README.md

@@ -105,7 +105,7 @@ For code examples, API reference, and wiring instructions, see **[barebrowse.con
 
 ## What it handles automatically
 
-Cookie consent walls (29 languages), login walls (cookie extraction from your browsers), bot detection (stealth patches + automatic headed fallback on challenge pages, error pages, and near-empty responses), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
+Cookie consent walls (29 languages, with real mouse click fallback for stubborn CMPs), login walls (cookie extraction from your browsers), bot detection (stealth patches + automatic headed fallback on challenge pages, error pages, and near-empty responses), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
 
 ## What the agent sees
 
@@ -139,7 +139,7 @@ Everything the agent can do through barebrowse:
 | **Upload** | Set files on a file input element |
 | **Screenshot** | Page capture as base64 PNG/JPEG/WebP |
 | **PDF** | Export page as PDF |
-| **Assess** | Privacy scan: score (0-100), risk level, 10-category breakdown. Reuses session browser via tabs (max 3 concurrent), auto-retries on CDP crash, 30s timeout per scan. Requires `npm install wearehere`. |
+| **Assess** | Privacy scan: score (0-100), risk level, 10-category breakdown. Tries headless first, falls back to headed if bot-blocked. Consent auto-dismissed before scan. Max 3 concurrent, 30s timeout, tabs cleaned up. Requires `npm install wearehere`. |
 | **Tabs** | List open tabs, switch between them |
 | **Wait for content** | Poll for text or CSS selector to appear on page |
 | **Wait for navigation** | SPA-aware: works for full page loads and pushState |

package/barebrowse.context.md

@@ -1,7 +1,7 @@
 # barebrowse -- Integration Guide
 
 > For AI assistants and developers wiring barebrowse into a project.
-> v0.5.4 | Node.js >= 22 | 0 required deps | MIT
+> v0.5.6 | Node.js >= 22 | 0 required deps | MIT
 
 ## What this is
 
@@ -146,7 +146,7 @@ barebrowse can inject cookies from the user's real browser sessions, bypassing l
 | Obstacle | How | Mode |
 |---|---|---|
 | Cookie consent | ARIA scan + jsClick accept button, 29 languages | Both |
-| Consent behind iframes | JS `.click()` via DOM.resolveNode bypasses overlays | Both |
+| Consent behind iframes | JS `.click()` via DOM.resolveNode bypasses overlays, real mouse click fallback for CMPs that ignore synthetic clicks | Both |
 | Permission prompts | Launch flags + CDP Browser.setPermission auto-deny | Both |
 | Media autoplay blocked | `--autoplay-policy=no-user-gesture-required` | Both |
 | Login walls | Cookie extraction from Firefox/Chromium + CDP injection | Both |
@@ -157,6 +157,7 @@ barebrowse can inject cookies from the user's real browser sessions, bypassing l
 | Bot detection | Hybrid fallback: detects challenge pages, error pages, and near-empty responses, then switches to headed | Hybrid |
 | `navigator.webdriver` | Stealth patches in headless (webdriver, plugins, chrome obj) | Headless |
 | Profile locking | Unique temp dir per headless instance | Headless |
+| Shared memory crash (Linux) | `--disable-dev-shm-usage` flag prevents `/dev/shm` exhaustion | Headless |
 | ARIA noise | 9-step pruning: wrapper collapse, noise removal, landmark promotion | Both |
 
 ## bareagent wiring
@@ -242,7 +243,7 @@ Action tools return `'ok'` -- the agent calls `snapshot` explicitly to observe.
 
 Session runs in hybrid mode (headless with automatic headed fallback on bot detection). `goto` injects cookies from the user's browser before navigation for authenticated access.
 
-Session tools share a singleton page, lazy-created on first use. Assess runs in isolated tabs within the session (max 3 concurrent, with retry and CDP crash recovery).
+Session tools share a singleton page, lazy-created on first use. Assess tries headless first; if bot-blocked (score ≤5 with all zeros), retries with a separate headed session. Tabs dismissed for consent and closed after every scan. Max 3 concurrent, with CDP crash recovery.
 
 ## Architecture
 

package/mcp-server.js

@@ -294,9 +294,14 @@ async function handleToolCall(name, args) {
       if (!assessFn) throw new Error('wearehere is not installed. Run: npm install wearehere');
       await acquireAssessSlot();
       try {
-        const runAssess = async () => {
-          const page = await getPage();
-          const tab = await page.createTab();
+        const runAssess = async (headed) => {
+          let tab;
+          if (headed) {
+            tab = await connect({ mode: 'headed' });
+          } else {
+            const page = await getPage();
+            tab = await page.createTab();
+          }
           let timer;
           try {
             const result = await Promise.race([
@@ -312,6 +317,7 @@ async function handleToolCall(name, args) {
               }),
             ]);
             clearTimeout(timer);
+            await tab.close().catch(() => {});
             return JSON.stringify(result, null, 2);
           } catch (err) {
             clearTimeout(timer);
@@ -319,15 +325,33 @@ async function handleToolCall(name, args) {
             throw err;
           }
         };
+
+        // Try headless first
         try {
-          return await runAssess();
+          const result = await runAssess(false);
+          // Check if result looks bot-blocked (score 0-5, no trackers, few cookies)
+          try {
+            const parsed = JSON.parse(result);
+            const { network, trackers, profiling } = parsed.categories || {};
+            const allZero = (network?.score || 0) === 0
+              && (trackers?.score || 0) === 0
+              && (profiling?.score || 0) === 0;
+            if (allZero && (parsed.score || 0) <= 5) {
+              // Likely bot-blocked — retry headed
+              try {
+                return await runAssess(true);
+              } catch {
+                return result; // headed failed, return headless result
+              }
+            }
+          } catch {}
+          return result;
         } catch (err) {
           if (isCdpDead(err)) _page = null;
-          await new Promise((r) => setTimeout(r, 2000));
+          // Headless crashed — try headed
           try {
-            return await runAssess();
+            return await runAssess(true);
           } catch (retryErr) {
-            if (isCdpDead(retryErr)) _page = null;
             throw retryErr;
           }
         }
@@ -355,7 +379,7 @@ async function handleMessage(msg) {
     return jsonrpcResponse(id, {
       protocolVersion: '2024-11-05',
       capabilities: { tools: {} },
-      serverInfo: { name: 'barebrowse', version: '0.5.4' },
+      serverInfo: { name: 'barebrowse', version: '0.5.6' },
     });
   }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "barebrowse",
-  "version": "0.5.4",
+  "version": "0.5.6",
   "description": "Authenticated web browsing for autonomous agents via CDP. URL in, pruned ARIA snapshot out.",
   "type": "module",
   "main": "src/index.js",

package/src/chromium.js

@@ -82,6 +82,7 @@ export async function launch(opts = {}) {
     '--use-fake-device-for-media-stream',
     '--use-fake-ui-for-media-stream',
     '--disable-features=MediaRouter',
+    '--disable-dev-shm-usage',
   ];
 
   if (opts.proxy) {

package/src/consent.js

@@ -21,6 +21,7 @@ const ACCEPT_PATTERNS = [
   // Dutch
   /\balles\s*accepteren\b/i,
   /\balles\s*toestaan\b/i,
+  /\baccepteren\b/i,
   /\bakkoord\b/i,
   // German
   /\balle\s*akzeptieren\b/i,
@@ -162,6 +163,20 @@ async function jsClick(session, backendNodeId) {
   });
 }
 
+/**
+ * Click a node via real mouse events (scrollIntoView → getBoxModel → mousePressed/Released).
+ * Some CMPs ignore synthetic .click() and only respond to real Input events.
+ */
+async function realClick(session, backendNodeId) {
+  await session.send('DOM.scrollIntoViewIfNeeded', { backendNodeId });
+  const { model } = await session.send('DOM.getBoxModel', { backendNodeId });
+  const [x1, y1, x2, y2, x3, y3, x4, y4] = model.content;
+  const cx = (x1 + x3) / 2;
+  const cy = (y1 + y3) / 2;
+  await session.send('Input.dispatchMouseEvent', { type: 'mousePressed', x: cx, y: cy, button: 'left', clickCount: 1 });
+  await session.send('Input.dispatchMouseEvent', { type: 'mouseReleased', x: cx, y: cy, button: 'left', clickCount: 1 });
+}
+
 /**
  * Try to dismiss a cookie consent dialog on the current page.
  * Inspects the ARIA tree for dialog elements with consent-related content,
@@ -212,8 +227,19 @@ export async function dismissConsent(session) {
     const button = findAcceptButton(dialogId, nodes, nodeMap, parentMap);
     if (button?.backendDOMNodeId) {
       try {
+        // Try jsClick first (bypasses iframe overlays)
         await jsClick(session, button.backendDOMNodeId);
         await new Promise((r) => setTimeout(r, 1000));
+        // Check if consent actually dismissed — some CMPs ignore synthetic clicks
+        const { nodes: nodesAfter } = await session.send('Accessibility.getFullAXTree');
+        const stillThere = nodesAfter.some((n) =>
+          n.role?.value === 'button' && n.name?.value === button.name?.value
+        );
+        if (stillThere) {
+          // Retry with real mouse event
+          await realClick(session, button.backendDOMNodeId);
+          await new Promise((r) => setTimeout(r, 1000));
+        }
         return true;
       } catch {
         // Click failed — try next dialog
@@ -278,10 +304,24 @@ function tryGlobalConsentButton(nodes, session) {
       if (node.role?.value !== 'button') continue;
       const name = node.name?.value || '';
       if (name && pattern.test(name) && node.backendDOMNodeId) {
-        return jsClick(session, node.backendDOMNodeId)
-          .then(() => new Promise((r) => setTimeout(r, 1000)))
-          .then(() => true)
-          .catch(() => false);
+        return (async () => {
+          try {
+            await jsClick(session, node.backendDOMNodeId);
+            await new Promise((r) => setTimeout(r, 1000));
+            // Check if button still exists — retry with real click if so
+            const { nodes: nodesAfter } = await session.send('Accessibility.getFullAXTree');
+            const stillThere = nodesAfter.some((n) =>
+              n.role?.value === 'button' && n.name?.value === name
+            );
+            if (stillThere) {
+              await realClick(session, node.backendDOMNodeId);
+              await new Promise((r) => setTimeout(r, 1000));
+            }
+            return true;
+          } catch {
+            return false;
+          }
+        })();
       }
     }
   }

package/src/index.js

@@ -371,6 +371,9 @@ export async function connect(opts = {}) {
       return {
         async goto(url, timeout = 30000) {
           await navigate(tab, url, timeout);
+          if (opts.consent !== false) {
+            await dismissConsent(tab.session);
+          }
         },
         async injectCookies(url, cookieOpts) {
           await authenticate(tab.session, url, { browser: cookieOpts?.browser });