barebrowse 0.5.7 → 0.5.9

package/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## 0.5.8
+
+Bot challenge detection for all browsing, not just assess.
+
+### Bot detection (`src/index.js`)
+- `isChallengePage()` now checks ARIA node count (<50 = bot-blocked) in addition to text length and phrase matching
+- `botBlocked` property exposed on both `connect()` pages and `createTab()` tabs
+- `goto()` on main page and tabs sets `botBlocked` after every navigation
+- `snapshot()` prepends `[BOT CHALLENGE DETECTED]` warning line when flagged
+- Hybrid fallback on main page now uses node count for more reliable detection
+
+### Assess handler (`mcp-server.js`)
+- Headed fallback now uses `tab.botBlocked` flag instead of naive score threshold (≤5 + all zeros)
+- Previously: sites like Reuters, Home Depot, Leboncoin returned fake-clean scores because the bot challenge page looked "clean" to the scanner
+- Now: node count catches every bot-blocked page regardless of score
+
+### Tested
+- reuters.com, homedepot.com, leboncoin.fr, idealista.com all correctly flagged `botBlocked: true`
+- svt.se, whatsapp.com, google.com correctly flagged `false`
+- 71/71 tests passing
+
 ## 0.5.7
 
 MCP server crash resilience + process hardening.

package/README.md

@@ -105,7 +105,7 @@ For code examples, API reference, and wiring instructions, see **[barebrowse.con
 
 ## What it handles automatically
 
-Cookie consent walls (29 languages, with real mouse click fallback for stubborn CMPs), login walls (cookie extraction from your browsers), bot detection (stealth patches + automatic headed fallback on challenge pages, error pages, and near-empty responses), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
+Cookie consent walls (29 languages, with real mouse click fallback for stubborn CMPs), login walls (cookie extraction from your browsers), bot detection (ARIA node count heuristic + stealth patches + automatic headed fallback — snapshot shows `[BOT CHALLENGE DETECTED]` warning when blocked), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
 
 ## What the agent sees
 

package/barebrowse.context.md

@@ -1,7 +1,7 @@
 # barebrowse -- Integration Guide
 
 > For AI assistants and developers wiring barebrowse into a project.
-> v0.5.7 | Node.js >= 22 | 0 required deps | MIT
+> v0.5.8 | Node.js >= 22 | 0 required deps | MIT
 
 ## What this is
 
@@ -76,9 +76,10 @@ const snapshot = await browse('https://example.com', {
 | `waitForNetworkIdle(opts?)` | { timeout?: number, idle?: number } | void | Wait until no pending requests for `idle` ms (default 500) |
 | `saveState(filePath)` | filePath: string | void | Export cookies + localStorage to JSON file |
 | `injectCookies(url, opts?)` | url: string, { browser?: string } | void | Extract cookies from user's browser and inject via CDP |
+| `botBlocked` | -- | boolean | True if last `goto()` hit a bot challenge (ARIA node count <50). Resets on each navigation. |
 | `dialogLog` | -- | Array<{type, message, timestamp}> | Auto-dismissed JS dialog history |
 | `cdp` | -- | object | Raw CDP session for escape hatch: `page.cdp.send(method, params)` |
-| `createTab()` | -- | tab handle | New tab in same browser. Returns `{ goto, injectCookies, waitForNetworkIdle, cdp, close }`. Tab close doesn't affect session. |
+| `createTab()` | -- | tab handle | New tab in same browser. Returns `{ goto, botBlocked, injectCookies, waitForNetworkIdle, cdp, close }`. Tab close doesn't affect session. |
 | `close()` | -- | void | Close page, disconnect CDP, kill browser (if headless) |
 
 **connect() options** (in addition to mode/port/consent):
@@ -154,7 +155,7 @@ barebrowse can inject cookies from the user's real browser sessions, bypassing l
 | Off-screen elements | `DOM.scrollIntoViewIfNeeded` before every click | Both |
 | Form submission | `press('Enter')` triggers onsubmit | Both |
 | SPA navigation | `waitForNavigation()` uses loadEventFired + frameNavigated | Both |
-| Bot detection | Hybrid fallback: detects challenge pages, error pages, and near-empty responses, then switches to headed | Hybrid |
+| Bot detection | ARIA node count (<50 = bot-blocked) + text heuristics. `botBlocked` flag set after every `goto()`. Hybrid fallback switches to headed. Snapshot shows `[BOT CHALLENGE DETECTED]` warning. | Hybrid |
 | `navigator.webdriver` | Stealth patches in headless (webdriver, plugins, chrome obj) | Headless |
 | Profile locking | Unique temp dir per headless instance | Headless |
 | Shared memory crash (Linux) | `--disable-dev-shm-usage` flag prevents `/dev/shm` exhaustion | Headless |

package/mcp-server.js

@@ -54,26 +54,18 @@ async function getPage() {
   }
 }
 
-// Concurrency limiter for assess — max 3 tabs at once
-const ASSESS_MAX = 3;
-let _assessRunning = 0;
-const _assessQueue = [];
+// Concurrency limiter — one assess at a time.
+// Headless tabs are fast, but headed fallback uses the user's single browser.
+// Running multiple headed navigations simultaneously hangs the browser.
+let _assessLock = Promise.resolve();
 
 function acquireAssessSlot() {
-  if (_assessRunning < ASSESS_MAX) {
-    _assessRunning++;
-    return Promise.resolve();
-  }
-  return new Promise((resolve) => _assessQueue.push(resolve));
+  let release;
+  const prev = _assessLock;
+  _assessLock = new Promise((r) => { release = r; });
+  return prev.then(() => release);
 }
 
-function releaseAssessSlot() {
-  if (_assessQueue.length > 0) {
-    _assessQueue.shift()();
-  } else {
-    _assessRunning--;
-  }
-}
 
 const TOOLS = [
   {
@@ -292,7 +284,7 @@ async function handleToolCall(name, args) {
     }
     case 'assess': {
       if (!assessFn) throw new Error('wearehere is not installed. Run: npm install wearehere');
-      await acquireAssessSlot();
+      const releaseSlot = await acquireAssessSlot();
       try {
         const runAssess = async (headed) => {
           let tab;
@@ -317,8 +309,9 @@ async function handleToolCall(name, args) {
               }),
             ]);
             clearTimeout(timer);
+            const wasBotBlocked = tab.botBlocked;
             await tab.close().catch(() => {});
-            return JSON.stringify(result, null, 2);
+            return { result, botBlocked: wasBotBlocked };
           } catch (err) {
             clearTimeout(timer);
             await tab.close().catch(() => {});
@@ -328,35 +321,29 @@ async function handleToolCall(name, args) {
 
         // Try headless first
         try {
-          const result = await runAssess(false);
-          // Check if result looks bot-blocked (score 0-5, no trackers, few cookies)
-          try {
-            const parsed = JSON.parse(result);
-            const { network, trackers, profiling } = parsed.categories || {};
-            const allZero = (network?.score || 0) === 0
-              && (trackers?.score || 0) === 0
-              && (profiling?.score || 0) === 0;
-            if (allZero && (parsed.score || 0) <= 5) {
-              // Likely bot-blocked — retry headed
-              try {
-                return await runAssess(true);
-              } catch {
-                return result; // headed failed, return headless result
-              }
+          const { result, botBlocked } = await runAssess(false);
+          if (botBlocked) {
+            // Bot-blocked in headless — retry headed
+            try {
+              const headed = await runAssess(true);
+              return JSON.stringify(headed.result, null, 2);
+            } catch {
+              return JSON.stringify(result, null, 2); // headed failed, return headless result
             }
-          } catch {}
-          return result;
+          }
+          return JSON.stringify(result, null, 2);
         } catch (err) {
           if (isCdpDead(err)) _page = null;
           // Headless crashed — try headed
           try {
-            return await runAssess(true);
+            const headed = await runAssess(true);
+            return JSON.stringify(headed.result, null, 2);
           } catch (retryErr) {
             throw retryErr;
           }
         }
       } finally {
-        releaseAssessSlot();
+        releaseSlot();
       }
     }
     default:
@@ -379,7 +366,7 @@ async function handleMessage(msg) {
     return jsonrpcResponse(id, {
       protocolVersion: '2024-11-05',
       capabilities: { tools: {} },
-      serverInfo: { name: 'barebrowse', version: '0.5.7' },
+      serverInfo: { name: 'barebrowse', version: '0.5.9' },
     });
   }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "barebrowse",
-  "version": "0.5.7",
+  "version": "0.5.9",
   "description": "Authenticated web browsing for autonomous agents via CDP. URL in, pruned ARIA snapshot out.",
   "type": "module",
   "main": "src/index.js",

package/src/index.js

@@ -73,10 +73,10 @@ export async function browse(url, opts = {}) {
     }
 
     // Step 5: Get ARIA tree
-    let { tree } = await ariaTree(page);
+    let { tree, nodeCount } = await ariaTree(page);
 
     // Step 5.5: Hybrid fallback — if headless was bot-blocked, retry headed
-    if (mode === 'hybrid' && isChallengePage(tree)) {
+    if (mode === 'hybrid' && isChallengePage(tree, nodeCount)) {
       await cdp.send('Target.closeTarget', { targetId: page.targetId });
       cdp.close();
       if (browser) { browser.process.kill(); browser = null; }
@@ -140,6 +140,7 @@ export async function connect(opts = {}) {
 
   let page = await createPage(cdp, mode !== 'headed', { viewport: opts.viewport });
   let refMap = new Map();
+  let botBlocked = false;
 
   // Suppress permission prompts for all modes
   await suppressPermissions(cdp);
@@ -179,23 +180,28 @@ export async function connect(opts = {}) {
         await dismissConsent(page.session);
       }
 
+      // Check for bot challenge
+      const { tree, nodeCount } = await ariaTree(page);
+      botBlocked = isChallengePage(tree, nodeCount);
+
       // Hybrid fallback: if bot-blocked, retry with headed browser
-      if (mode === 'hybrid') {
-        const { tree } = await ariaTree(page);
-        if (isChallengePage(tree)) {
-          await cdp.send('Target.closeTarget', { targetId: page.targetId });
-          cdp.close();
-          if (browser) { browser.process.kill(); browser = null; }
-
-          const port = opts.port || 9222;
-          const wsUrl = await getDebugUrl(port);
-          cdp = await createCDP(wsUrl);
-          page = await createPage(cdp, false, { viewport: opts.viewport });
-          setupDialogHandler(page.session);
-          await suppressPermissions(cdp);
-          await navigate(page, url, timeout);
-          if (opts.consent !== false) await dismissConsent(page.session);
-        }
+      if (botBlocked && mode === 'hybrid') {
+        await cdp.send('Target.closeTarget', { targetId: page.targetId });
+        cdp.close();
+        if (browser) { browser.process.kill(); browser = null; }
+
+        const port = opts.port || 9222;
+        const wsUrl = await getDebugUrl(port);
+        cdp = await createCDP(wsUrl);
+        page = await createPage(cdp, false, { viewport: opts.viewport });
+        setupDialogHandler(page.session);
+        await suppressPermissions(cdp);
+        await navigate(page, url, timeout);
+        if (opts.consent !== false) await dismissConsent(page.session);
+
+        // Re-check after headed fallback
+        const after = await ariaTree(page);
+        botBlocked = isChallengePage(after.tree, after.nodeCount);
       }
     },
 
@@ -223,11 +229,12 @@ export async function connect(opts = {}) {
       const raw = formatTree(result.tree);
       const { currentIndex, entries } = await page.session.send('Page.getNavigationHistory');
       const pageUrl = entries[currentIndex]?.url || '';
-      if (pruneOpts === false) return `url: ${pageUrl}\n` + raw;
+      const warn = botBlocked ? '[BOT CHALLENGE DETECTED — page content may be incomplete or blocked]\n' : '';
+      if (pruneOpts === false) return `url: ${pageUrl}\n` + warn + raw;
       const pruned = pruneTree(result.tree, { mode: pruneOpts?.mode || 'act' });
       const out = formatTree(pruned);
       const stats = `url: ${pageUrl}\n${raw.length.toLocaleString()} chars → ${out.length.toLocaleString()} chars (${Math.round((1 - out.length / raw.length) * 100)}% pruned)`;
-      return stats + '\n' + out;
+      return stats + '\n' + warn + out;
     },
 
     async click(ref) {
@@ -334,6 +341,8 @@ export async function connect(opts = {}) {
       writeFileSync(filePath, JSON.stringify(state, null, 2));
     },
 
+    get botBlocked() { return botBlocked; },
+
     dialogLog,
 
     async screenshot(screenshotOpts = {}) {
@@ -368,13 +377,17 @@ export async function connect(opts = {}) {
     async createTab() {
       const tab = await createPage(cdp, mode !== 'headed', { viewport: opts.viewport });
       await suppressPermissions(cdp);
+      let tabBotBlocked = false;
       return {
         async goto(url, timeout = 30000) {
           await navigate(tab, url, timeout);
           if (opts.consent !== false) {
             await dismissConsent(tab.session);
           }
+          const { tree, nodeCount } = await ariaTree(tab);
+          tabBotBlocked = isChallengePage(tree, nodeCount);
         },
+        get botBlocked() { return tabBotBlocked; },
         async injectCookies(url, cookieOpts) {
           await authenticate(tab.session, url, { browser: cookieOpts?.browser });
         },
@@ -485,7 +498,7 @@ async function ariaTree(page) {
     }
   }
 
-  return { tree, refMap };
+  return { tree, refMap, nodeCount: nodes.length };
 }
 
 /**
@@ -583,10 +596,14 @@ function waitForNetworkIdle(session, opts = {}) {
 
 /**
  * Detect if a page is a bot-challenge page (Cloudflare, etc.).
- * Heuristic: very short ARIA tree + known challenge phrases.
+ * Heuristic: low ARIA node count, short text, or known challenge phrases.
+ * @param {object} tree - Nested ARIA tree (from buildTree)
+ * @param {number} [nodeCount] - Raw CDP node count (from Accessibility.getFullAXTree)
  */
-function isChallengePage(tree) {
+function isChallengePage(tree, nodeCount) {
   if (!tree) return true;
+  // Real pages have 50+ ARIA nodes. Bot challenges have <20.
+  if (nodeCount !== undefined && nodeCount < 50) return true;
   const text = flattenTreeText(tree);
   // Near-empty pages are almost certainly blocks
   if (text.trim().length < 50) return true;

package/rescan.mjs

@@ -1,107 +0,0 @@
-import { connect } from '/home/hamr/PycharmProjects/barebrowse/src/index.js';
-import { assess } from 'wearehere';
-
-const SITES = [
-  // NL timeouts
-  'rtv.nl', 'bijenkorf.nl', 'jumbo.com', 'klm.nl',
-  // EU timeouts  
-  'zalando.de', 'otto.de', 'allegro.pl',
-  // Suspected bot-blocked zeros (NL)
-  'coolblue.nl', 'rabobank.nl', 'telegraaf.nl', 'wehkamp.nl',
-  'ing.nl', 'kvk.nl', 'thuisbezorgd.nl', 'transavia.com', 'schiphol.nl',
-  // Suspected bot-blocked zeros (EU)
-  'lufthansa.com', 'fnac.com', 'svt.se', 'revolut.com',
-  'leboncoin.fr', 'subito.it', 'idealista.com',
-  // Suspected bot-blocked zeros (US)
-  'washingtonpost.com', 'usatoday.com', 'etsy.com', 'costco.com',
-  'homedepot.com', 'chatgpt.com', 'doordash.com', 'yelp.com', 'reuters.com',
-  'snapchat.com', 'cnn.com'
-];
-
-async function scanSite(url) {
-  const full = 'https://www.' + url;
-  
-  // Try headless first
-  let page;
-  try {
-    page = await connect({ mode: 'hybrid' });
-    const tab = await page.createTab();
-    try {
-      await tab.injectCookies(full).catch(() => {});
-      const r = await Promise.race([
-        assess(tab, full, { timeout: 30000, settle: 3000 }),
-        new Promise((_, rej) => setTimeout(() => rej(new Error('timeout')), 35000))
-      ]);
-      await tab.close().catch(() => {});
-      
-      // Check if bot-blocked
-      const { network, trackers, profiling } = r.categories;
-      const allZero = (network?.score||0) === 0 && (trackers?.score||0) === 0 && (profiling?.score||0) === 0;
-      if (allZero && r.score <= 5) {
-        await page.close().catch(() => {});
-        // Retry headed
-        const hp = await connect({ mode: 'headed' });
-        try {
-          await hp.injectCookies(full).catch(() => {});
-          const r2 = await Promise.race([
-            assess(hp, full, { timeout: 30000, settle: 3000 }),
-            new Promise((_, rej) => setTimeout(() => rej(new Error('timeout')), 35000))
-          ]);
-          console.log(url + '|' + r2.score + '|' + r2.risk + '|HEADED|' + summarize(r2));
-          return;
-        } finally {
-          await hp.close().catch(() => {});
-        }
-      }
-      console.log(url + '|' + r.score + '|' + r.risk + '|HEADLESS|' + summarize(r));
-    } catch (e) {
-      await tab.close().catch(() => {});
-      if (e.message === 'timeout') {
-        // Try headed on timeout too
-        await page.close().catch(() => {});
-        try {
-          const hp = await connect({ mode: 'headed' });
-          try {
-            await hp.injectCookies(full).catch(() => {});
-            const r2 = await Promise.race([
-              assess(hp, full, { timeout: 30000, settle: 3000 }),
-              new Promise((_, rej) => setTimeout(() => rej(new Error('timeout')), 35000))
-            ]);
-            console.log(url + '|' + r2.score + '|' + r2.risk + '|HEADED|' + summarize(r2));
-            return;
-          } finally {
-            await hp.close().catch(() => {});
-          }
-        } catch {
-          console.log(url + '|—|timeout|BOTH|Failed both modes');
-          return;
-        }
-      }
-      console.log(url + '|—|error|—|' + e.message);
-    } finally {
-      await page?.close().catch(() => {});
-    }
-  } catch (e) {
-    console.log(url + '|—|error|—|' + e.message);
-  }
-}
-
-function summarize(r) {
-  const parts = [];
-  const c = r.categories;
-  if (c.cookies?.score > 0) parts.push(c.cookies.summary);
-  if (c.network?.score > 0) parts.push(c.network.summary);
-  if (c.trackers?.score > 0) parts.push(c.trackers.summary);
-  if (c.profiling?.score > 0) parts.push(c.profiling.summary);
-  if (c.selling_data?.score > 0) parts.push(c.selling_data.summary);
-  if (c.pressure?.score > 0) parts.push('pressure:' + c.pressure.score);
-  if (c.watching?.score > 0) parts.push(c.watching.summary);
-  if (c.stored_data?.score > 0) parts.push(c.stored_data.summary);
-  if (c.terms?.score > 0) parts.push('terms:' + c.terms.summary);
-  if (parts.length === 0) parts.push('Clean');
-  return parts.join('; ');
-}
-
-for (const site of SITES) {
-  await scanSite(site);
-}