barebrowse 0.5.2 → 0.5.4

package/.aurora/plans/active/harden-assess/design.md

@@ -0,0 +1,68 @@
+# Design: Harden assess tool
+
+## Architecture
+
+### Current flow (broken)
+```
+assess(url) → connect({ mode: 'hybrid' })  ← NEW browser, no cookies
+            → assessFn(page, url)
+            → page.close()                  ← kills browser
+```
+
+### New flow
+```
+assess(url) → getPage()                     ← reuse session browser
+            → page.createTab()              ← new tab in same browser
+            → tab.injectCookies(url)        ← cookie injection
+            → assessFn(tab, url)            ← assess uses tab
+            → tab.close()                   ← close tab only
+            timeout guard wraps entire flow
+            retry wraps entire flow
+```
+
+## Key design decisions
+
+### Why a new tab, not the session page?
+wearehere's `assess()` calls:
+- `session.send('Page.addScriptToEvaluateOnNewDocument', ...)` — injects fingerprint detection scripts
+- `networkSession.on('Network.requestWillBeSent', ...)` — monitors all network traffic
+
+These would pollute the session page. A separate tab has its own CDP session with isolated Page/Network domains.
+
+### createTab() page-like interface
+wearehere expects a page object with: `goto()`, `cdp` (raw session), `waitForNetworkIdle()`. createTab() returns exactly this interface:
+
+```javascript
+{
+  goto(url, timeout)        // navigate tab
+  cdp                       // raw CDP session for this tab
+  waitForNetworkIdle(opts)  // reuses existing waitForNetworkIdle()
+  injectCookies(url)        // cookie injection for this tab
+  close()                   // close tab, NOT the browser
+}
+```
+
+### Retry strategy
+```
+attempt 1: assess with 45s timeout
+  fail → wait 2s
+attempt 2: assess with 45s timeout (if browser crashed, reset _page first)
+  fail → return { error: "assessment_failed", ... }
+```
+
+### Timeout implementation
+```javascript
+const result = await Promise.race([
+  doAssess(page, url, opts),
+  new Promise((_, reject) => setTimeout(() => reject(new Error('timeout')), 45000))
+]);
+```
+On timeout, the tab is closed in a finally block.
+
+## Files changed
+| File | Change |
+|------|--------|
+| `src/index.js` | Add `createTab()` and tab's `close()` to connect() return object |
+| `mcp-server.js` | Rewrite assess case: use getPage().createTab(), add retry + timeout |
+| `README.md` | Update assess description, add self-healing mention |
+| `barebrowse.context.md` | Update assess section, document createTab() |

package/.aurora/plans/active/harden-assess/plan.md

@@ -0,0 +1,71 @@
+# Plan: Harden assess tool — session reuse + self-healing
+
+## Plan ID
+`harden-assess`
+
+## Summary
+Make the assess tool reuse the MCP session's browser instance (cookies, headed fallback) instead of spawning throwaway browsers, add retry logic for transient failures, and add a timeout guard so no single assessment can hang forever.
+
+## Problem Statement
+The `assess` tool in `mcp-server.js` calls `connect({ mode: 'hybrid' })` directly — creating a fresh headless browser per call with no cookies, no session state, and no headed fallback. Every other MCP tool uses the `getPage()` singleton. This causes:
+
+1. **No cookies** — assess browses as a stranger, getting blocked by consent walls and bot detection that cookies would bypass
+2. **No headed fallback reuse** — if the singleton already fell back to headed mode, assess still starts fresh headless and hits the same blocks
+3. **No retry** — any failure (browser crash, navigation timeout, CDP disconnect) kills the assessment with no recovery
+4. **No timeout guard** — if `wearehere`'s `assess()` hangs (e.g. network idle never resolves), the MCP call blocks indefinitely
+
+However, assess can't simply use the shared `_page` singleton directly because `wearehere` injects init scripts (`addScriptToEvaluateOnNewDocument`) and network listeners that would pollute the session page for subsequent calls. The solution is to create a **second page tab** within the same browser instance.
+
+## Proposed Solution
+1. **Session-aware page creation** — Instead of `connect()`, assess opens a new CDP tab within the existing browser. This shares the browser process (same cookies, same headed/headless state) but isolates assess's script injections to its own tab.
+2. **Retry with backoff** — Wrap the assess call in a retry loop (max 2 attempts, 2s backoff). On browser crash, reconnect the singleton.
+3. **Timeout guard** — Wrap each assess call in `Promise.race` with a 45s hard deadline. If exceeded, return an error result (not hang).
+
+## Benefits
+- Assess gets cookies and headed fallback for free — no separate browser instance
+- Failed assessments auto-retry instead of dying
+- Hanging assessments time out gracefully instead of blocking the MCP server forever
+- Eliminates the 10+ second cold-start per assessment (browser launch)
+
+## Scope
+### In Scope
+- Modify `mcp-server.js` assess handler to create tabs within existing browser
+- Add `createTab()` / `closeTab()` helper to `src/index.js` connect() page handle
+- Add retry wrapper in mcp-server.js
+- Add timeout guard in mcp-server.js
+- Update docs: README.md, barebrowse.context.md, CLAUDE.md
+
+### Out of Scope
+- Changing wearehere's internal logic
+- Adding retry/self-healing to other MCP tools (future work)
+- Batch/queue mode for multiple assessments
+- Changing the assess tool's MCP interface (same inputs/outputs)
+
+## Dependencies
+- `wearehere` package (assess function signature unchanged)
+- `src/index.js` connect() API (adding createTab/closeTab methods)
+- `src/cdp.js` (Target.createTarget / closeTarget already available)
+
+## Implementation Strategy
+Phase 1: Add tab management to connect() page handle (createTab, closeTab)
+Phase 2: Rewrite assess handler to use session tab + retry + timeout
+Phase 3: Update documentation
+
+## Risks and Mitigations
+| Risk | Impact | Mitigation |
+|------|--------|------------|
+| Init script injection leaks across tabs | Pollutes session page | Each tab gets its own Page domain; addScriptToEvaluateOnNewDocument is per-target |
+| Browser crash during assess kills session too | Session page lost | getPage() already handles reconnection lazily (set _page = null, next call recreates) |
+| wearehere expects full page handle, not raw CDP session | API mismatch | createTab() returns a page-like object with goto, cdp, waitForNetworkIdle — same interface |
+
+## Success Criteria
+- [ ] `assess` reuses the session browser (no separate `connect()` call)
+- [ ] `assess` inherits cookies from the session
+- [ ] `assess` works when session is in headed mode (hybrid fallback already triggered)
+- [ ] Failed assessment retries once before returning error
+- [ ] Assessment hanging > 45s returns timeout error, doesn't block server
+- [ ] All existing tests pass
+- [ ] Documentation updated
+
+## Open Questions
+1. Should createTab() also inject cookies? — **Recommendation**: Yes, call `authenticate()` for the target URL before navigation, same as `goto` does.

package/.aurora/plans/active/harden-assess/prd.md

@@ -0,0 +1,38 @@
+# PRD: Harden assess tool
+
+## Overview
+The `assess` MCP tool must reuse the session browser, retry on failure, and time out gracefully.
+
+## Requirements
+
+### R1: Session-aware tab creation
+The assess tool MUST create a new browser tab within the existing MCP session browser instead of spawning a separate browser via `connect()`. The tab MUST:
+- Share the same browser process (inheriting headless/headed state)
+- Have access to the browser's cookie jar
+- Isolate its CDP domains (Page, Network, DOM) from the session page
+- Be closed after each assessment completes or fails
+
+### R2: Cookie injection
+Before navigating, the assess tab MUST inject cookies from the user's browser for the target URL, using the same `authenticate()` mechanism as `goto`.
+
+### R3: Retry on failure
+If an assessment fails (navigation timeout, CDP error, browser crash), the tool MUST:
+- Retry once after a 2-second delay
+- If the browser crashed, reset the session singleton (`_page = null`) so getPage() reconnects
+- If retry also fails, return a structured error result (not throw)
+
+### R4: Timeout guard
+Each assessment MUST have a hard timeout of 45 seconds. If exceeded:
+- The tab is force-closed
+- A structured error result is returned: `{ site, url, error: "timeout", scanned_at }`
+- The session page is NOT affected
+
+### R5: Backwards compatibility
+- The assess tool's MCP interface (inputs/outputs) MUST NOT change
+- Successful assessments return the same JSON format as before
+- The tool appears/disappears based on wearehere availability (unchanged)
+
+## Non-functional
+- No new dependencies
+- No changes to wearehere package
+- createTab()/closeTab() exposed on connect() page handle for library users too

package/CHANGELOG.md

@@ -1,5 +1,40 @@
 # Changelog
 
+## 0.5.4
+
+Assess tool hardened: session reuse, concurrency, self-healing.
+
+### Assess handler (`mcp-server.js`)
+- Session reuse: assess now opens tabs in the existing session browser via `createTab()` instead of spawning a new browser per scan
+- Concurrency: semaphore limits to 3 concurrent assess tabs, queues the rest
+- 30s hard timeout per scan with force tab close on expiry
+- Auto-retry once on failure with 2s backoff; resets session on CDP crash
+- Removed debug logging added during development
+
+### CDP resilience (`src/cdp.js`)
+- `ws.onclose` handler rejects all pending CDP promises — prevents zombie promises on browser crash
+
+### Library API (`src/index.js`)
+- `createTab()` added to `connect()` return object — creates new tab in same browser session
+- Returns `{ goto, injectCookies, waitForNetworkIdle, cdp, close }`, tab close doesn't affect main page
+
+### Message loop (`mcp-server.js`)
+- `getPage()` is concurrency-safe with promise dedup (prevents duplicate browser launches)
+- Message loop changed from sequential `await` to concurrent `.then()` fire-and-forget — multiple tool calls no longer block each other
+
+### Tests
+- 69/69 passing
+- Verified: 10 concurrent assess calls through MCP client, all succeed (3-at-a-time semaphore confirmed)
+
+## 0.5.3
+
+Improved bot detection for hybrid mode fallback.
+
+### Challenge detection (`src/index.js`)
+- `isChallengePage()` now catches near-empty pages (< 50 chars of text) as blocks
+- Added challenge phrases: "unknown error", "access denied", "permission denied", "request blocked"
+- Sites like CNN that return generic error pages instead of Cloudflare challenges now correctly trigger hybrid → headed fallback
+
 ## 0.5.2
 
 Clean npm tarball + bareagent tool parity.

package/README.md

@@ -105,7 +105,7 @@ For code examples, API reference, and wiring instructions, see **[barebrowse.con
 
 ## What it handles automatically
 
-Cookie consent walls (29 languages), login walls (cookie extraction from your browsers), bot detection (stealth patches + automatic headed fallback), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
+Cookie consent walls (29 languages), login walls (cookie extraction from your browsers), bot detection (stealth patches + automatic headed fallback on challenge pages, error pages, and near-empty responses), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
 
 ## What the agent sees
 
@@ -139,7 +139,7 @@ Everything the agent can do through barebrowse:
 | **Upload** | Set files on a file input element |
 | **Screenshot** | Page capture as base64 PNG/JPEG/WebP |
 | **PDF** | Export page as PDF |
-| **Assess** | Privacy scan: score (0-100), risk level, 10-category breakdown. Requires `npm install wearehere`. |
+| **Assess** | Privacy scan: score (0-100), risk level, 10-category breakdown. Reuses session browser via tabs (max 3 concurrent), auto-retries on CDP crash, 30s timeout per scan. Requires `npm install wearehere`. |
 | **Tabs** | List open tabs, switch between them |
 | **Wait for content** | Poll for text or CSS selector to appear on page |
 | **Wait for navigation** | SPA-aware: works for full page loads and pushState |

package/barebrowse.context.md

@@ -1,7 +1,7 @@
 # barebrowse -- Integration Guide
 
 > For AI assistants and developers wiring barebrowse into a project.
-> v0.5.2 | Node.js >= 22 | 0 required deps | MIT
+> v0.5.4 | Node.js >= 22 | 0 required deps | MIT
 
 ## What this is
 
@@ -78,6 +78,7 @@ const snapshot = await browse('https://example.com', {
 | `injectCookies(url, opts?)` | url: string, { browser?: string } | void | Extract cookies from user's browser and inject via CDP |
 | `dialogLog` | -- | Array<{type, message, timestamp}> | Auto-dismissed JS dialog history |
 | `cdp` | -- | object | Raw CDP session for escape hatch: `page.cdp.send(method, params)` |
+| `createTab()` | -- | tab handle | New tab in same browser. Returns `{ goto, injectCookies, waitForNetworkIdle, cdp, close }`. Tab close doesn't affect session. |
 | `close()` | -- | void | Close page, disconnect CDP, kill browser (if headless) |
 
 **connect() options** (in addition to mode/port/consent):
@@ -153,7 +154,7 @@ barebrowse can inject cookies from the user's real browser sessions, bypassing l
 | Off-screen elements | `DOM.scrollIntoViewIfNeeded` before every click | Both |
 | Form submission | `press('Enter')` triggers onsubmit | Both |
 | SPA navigation | `waitForNavigation()` uses loadEventFired + frameNavigated | Both |
-| Bot detection | Headed mode with real cookies bypasses most checks | Headed |
+| Bot detection | Hybrid fallback: detects challenge pages, error pages, and near-empty responses, then switches to headed | Hybrid |
 | `navigator.webdriver` | Stealth patches in headless (webdriver, plugins, chrome obj) | Headless |
 | Profile locking | Unique temp dir per headless instance | Headless |
 | ARIA noise | 9-step pruning: wrapper collapse, noise removal, landmark promotion | Both |
@@ -241,7 +242,7 @@ Action tools return `'ok'` -- the agent calls `snapshot` explicitly to observe.
 
 Session runs in hybrid mode (headless with automatic headed fallback on bot detection). `goto` injects cookies from the user's browser before navigation for authenticated access.
 
-Session tools share a singleton page, lazy-created on first use.
+Session tools share a singleton page, lazy-created on first use. Assess runs in isolated tabs within the session (max 3 concurrent, with retry and CDP crash recovery).
 
 ## Architecture
 
@@ -271,7 +272,7 @@ URL -> chromium.js (find/launch browser, permission flags)
 | `src/consent.js` | 200 | Auto-dismiss cookie consent dialogs across languages |
 | `src/stealth.js` | ~40 | Navigator patches for headless anti-detection |
 | `src/bareagent.js` | ~250 | Tool adapter for bareagent Loop |
-| `mcp-server.js` | ~170 | MCP server (JSON-RPC over stdio) |
+| `mcp-server.js` | ~340 | MCP server (JSON-RPC over stdio, assess session reuse + concurrency) |
 
 ## Privacy assessment (optional)
 
@@ -281,7 +282,7 @@ Install `wearehere` to add an `assess` tool to both the MCP server and bareagent
 npm install wearehere
 ```
 
-The `assess` tool navigates to a URL in an isolated browser page, scans for 10 privacy categories (cookies, trackers, fingerprinting, dark patterns, data brokers, form surveillance, link tracking, toxic terms, stored data, network traffic), and returns a compact JSON assessment:
+The `assess` tool opens a new tab in the session browser via `createTab()` (reusing cookies and headed fallback), scans for 10 privacy categories (cookies, trackers, fingerprinting, dark patterns, data brokers, form surveillance, link tracking, toxic terms, stored data, network traffic), and returns a compact JSON assessment. Max 3 concurrent scans (queued beyond that), 30s hard timeout per scan, auto-retry once on failure with 2s backoff (session reset on CDP crash):
 
 ```json
 {

package/mcp-server.js

@@ -20,6 +20,11 @@ try {
 } catch {}
 
 
+function isCdpDead(err) {
+  const m = err.message || '';
+  return m.includes('WebSocket') || m.includes('Target closed') || m.includes('Session closed') || m.includes('CDP');
+}
+
 const MAX_CHARS_DEFAULT = 30000;
 const OUTPUT_DIR = join(process.cwd(), '.barebrowse');
 
@@ -32,10 +37,42 @@ function saveSnapshot(text) {
 }
 
 let _page = null;
+let _pageConnecting = null;
 
 async function getPage() {
-  if (!_page) _page = await connect({ mode: 'hybrid' });
-  return _page;
+  if (_page) return _page;
+  if (_pageConnecting) return _pageConnecting;
+  _pageConnecting = connect({ mode: 'hybrid' });
+  try {
+    _page = await _pageConnecting;
+    return _page;
+  } catch (err) {
+    _page = null;
+    throw err;
+  } finally {
+    _pageConnecting = null;
+  }
+}
+
+// Concurrency limiter for assess — max 3 tabs at once
+const ASSESS_MAX = 3;
+let _assessRunning = 0;
+const _assessQueue = [];
+
+function acquireAssessSlot() {
+  if (_assessRunning < ASSESS_MAX) {
+    _assessRunning++;
+    return Promise.resolve();
+  }
+  return new Promise((resolve) => _assessQueue.push(resolve));
+}
+
+function releaseAssessSlot() {
+  if (_assessQueue.length > 0) {
+    _assessQueue.shift()();
+  } else {
+    _assessRunning--;
+  }
 }
 
 const TOOLS = [
@@ -255,15 +292,47 @@ async function handleToolCall(name, args) {
     }
     case 'assess': {
       if (!assessFn) throw new Error('wearehere is not installed. Run: npm install wearehere');
-      const page = await connect({ mode: 'hybrid' });
+      await acquireAssessSlot();
       try {
-        const result = await assessFn(page, args.url, {
-          timeout: args.timeout,
-          settle: args.settle,
-        });
-        return JSON.stringify(result, null, 2);
+        const runAssess = async () => {
+          const page = await getPage();
+          const tab = await page.createTab();
+          let timer;
+          try {
+            const result = await Promise.race([
+              (async () => {
+                await tab.injectCookies(args.url).catch(() => {});
+                return await assessFn(tab, args.url, { timeout: args.timeout, settle: args.settle });
+              })(),
+              new Promise((_, reject) => {
+                timer = setTimeout(() => {
+                  tab.close().catch(() => {});
+                  reject(new Error('assess timeout'));
+                }, 30000);
+              }),
+            ]);
+            clearTimeout(timer);
+            return JSON.stringify(result, null, 2);
+          } catch (err) {
+            clearTimeout(timer);
+            await tab.close().catch(() => {});
+            throw err;
+          }
+        };
+        try {
+          return await runAssess();
+        } catch (err) {
+          if (isCdpDead(err)) _page = null;
+          await new Promise((r) => setTimeout(r, 2000));
+          try {
+            return await runAssess();
+          } catch (retryErr) {
+            if (isCdpDead(retryErr)) _page = null;
+            throw retryErr;
+          }
+        }
       } finally {
-        await page.close().catch(() => {});
+        releaseAssessSlot();
       }
     }
     default:
@@ -286,7 +355,7 @@ async function handleMessage(msg) {
     return jsonrpcResponse(id, {
       protocolVersion: '2024-11-05',
       capabilities: { tools: {} },
-      serverInfo: { name: 'barebrowse', version: '0.5.2' },
+      serverInfo: { name: 'barebrowse', version: '0.5.4' },
     });
   }
 
@@ -321,8 +390,9 @@ async function handleMessage(msg) {
 let buffer = '';
 
 process.stdin.setEncoding('utf8');
-process.stdin.on('data', async (chunk) => {
+process.stdin.on('data', (chunk) => {
   buffer += chunk;
+
   let newlineIdx;
   while ((newlineIdx = buffer.indexOf('\n')) !== -1) {
     const line = buffer.slice(0, newlineIdx).trim();
@@ -331,11 +401,19 @@ process.stdin.on('data', async (chunk) => {
 
     try {
       const msg = JSON.parse(line);
-      const response = await handleMessage(msg);
-      if (response) {
-        process.stdout.write(response + '\n');
-      }
+
+      handleMessage(msg).then((response) => {
+        if (response) {
+
+          process.stdout.write(response + '\n');
+
+        }
+      }).catch((err) => {
+
+        process.stdout.write(jsonrpcError(msg.id, -32700, `Error: ${err.message}`) + '\n');
+      });
     } catch (err) {
+
       process.stdout.write(jsonrpcError(null, -32700, `Parse error: ${err.message}`) + '\n');
     }
   }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "barebrowse",
-  "version": "0.5.2",
+  "version": "0.5.4",
   "description": "Authenticated web browsing for autonomous agents via CDP. URL in, pruned ARIA snapshot out.",
   "type": "module",
   "main": "src/index.js",

package/src/cdp.js

@@ -71,6 +71,13 @@ export async function createCDP(wsUrl) {
     }
   };
 
+  ws.onclose = () => {
+    for (const [id, handler] of pending) {
+      handler.reject(new Error('CDP WebSocket closed'));
+      pending.delete(id);
+    }
+  };
+
   const client = {
     /**
      * Send a CDP command and wait for the response.

package/src/index.js

@@ -365,6 +365,26 @@ export async function connect(opts = {}) {
     /** Raw CDP session for escape hatch */
     cdp: page.session,
 
+    async createTab() {
+      const tab = await createPage(cdp, mode !== 'headed', { viewport: opts.viewport });
+      await suppressPermissions(cdp);
+      return {
+        async goto(url, timeout = 30000) {
+          await navigate(tab, url, timeout);
+        },
+        async injectCookies(url, cookieOpts) {
+          await authenticate(tab.session, url, { browser: cookieOpts?.browser });
+        },
+        waitForNetworkIdle(idleOpts = {}) {
+          return waitForNetworkIdle(tab.session, idleOpts);
+        },
+        cdp: tab.session,
+        async close() {
+          await cdp.send('Target.closeTarget', { targetId: tab.targetId });
+        },
+      };
+    },
+
     async close() {
       await cdp.send('Target.closeTarget', { targetId: page.targetId });
       cdp.close();
@@ -565,6 +585,8 @@ function waitForNetworkIdle(session, opts = {}) {
 function isChallengePage(tree) {
   if (!tree) return true;
   const text = flattenTreeText(tree);
+  // Near-empty pages are almost certainly blocks
+  if (text.trim().length < 50) return true;
   const challengePhrases = [
     'just a moment',
     'checking if the site connection is secure',
@@ -574,6 +596,10 @@ function isChallengePage(tree) {
     'prove your humanity',
     'attention required',
     'file a ticket',
+    'unknown error',
+    'access denied',
+    'permission denied',
+    'request blocked',
   ];
   const lower = text.toLowerCase();
   return challengePhrases.some((p) => lower.includes(p));