barebrowse 0.5.8 → 0.6.1

package/CHANGELOG.md

@@ -1,5 +1,62 @@
 # Changelog
 
+## 0.6.1
+
+Headed fallback is now a per-navigation escape hatch, not a permanent mode switch. Graceful degradation when headed is unavailable.
+
+### Switch-back to headless (`src/index.js`)
+- `connect().goto()` in hybrid mode: if currently headed from a previous fallback, kills the headed browser and launches fresh headless before navigating
+- New `currentlyHeaded` runtime state variable tracks actual browser mode (vs `mode` which is user config)
+- `createPage()` stealth decision uses runtime mode (`!currentlyHeaded`) instead of config mode (`mode !== 'headed'`)
+- `createTab()` also uses `currentlyHeaded` for correct stealth application
+
+### Graceful degradation (`src/index.js`)
+- `connect().goto()` hybrid fallback wrapped in try/catch — if `launch({ headed: true })` fails (no `$DISPLAY`, no Wayland, CI/Docker), keeps the headless result with `botBlocked: true` and `[BOT CHALLENGE DETECTED]` warning
+- `browse()` hybrid fallback also wrapped in try/catch — same graceful degradation for one-shot browsing
+- No crash on headless-only environments
+
+### Flow after changes
+```
+goto(url) in hybrid mode:
+  1. If currently headed → kill headed, launch headless, reset currentlyHeaded
+  2. Navigate to url
+  3. Check bot-blocked
+  4. If bot-blocked → TRY launch headed (set currentlyHeaded=true)
+                    → CATCH: headed unavailable, keep headless result
+```
+
+### Docs
+- Updated hybrid mode descriptions in barebrowse.context.md, system-state.md, prd.md
+
+### Tests
+- All existing tests pass (tests use headless mode, unaffected by hybrid logic)
+
+## 0.6.0
+
+Self-launching headed fallback. Headed and hybrid modes no longer require a manually-launched browser on port 9222 — barebrowse auto-launches a visible Chromium window via `launch({ headed: true })`.
+
+### Headed mode auto-launch (`src/chromium.js`)
+- `launch()` accepts `headed` option — skips `--headless=new` and `--hide-scrollbars` flags
+- Same temp profile, same random port, same CDP parsing, same process return
+
+### Hybrid fallback fix (`src/index.js`)
+- All 4 `getDebugUrl(port)` call sites replaced with `launch({ headed: true, proxy })` + `createCDP(browser.wsUrl)`
+- `browse()` headed branch, `browse()` hybrid fallback, `connect()` headed branch, `connect().goto()` hybrid fallback
+- `getDebugUrl` import removed from index.js (still exported from chromium.js for external use)
+- Hybrid mode now actually works — previously it tried to connect to port 9222 which nobody ran
+
+### Assess handler simplified (`mcp-server.js`)
+- Removed dual-path `runAssess(headed)` function (~60 lines of broken headed fallback)
+- Assess now uses the session's hybrid mode: if tab is bot-blocked, triggers headed fallback via main page `goto()`, then retries in a new tab
+- One flow, no separate `connect({ mode: 'headed' })` call
+
+### Docs
+- Removed all "launch browser with --remote-debugging-port=9222" instructions
+- Updated headed/hybrid mode descriptions across barebrowse.context.md, README.md, system-state.md, prd.md
+
+### Tests
+- 71/71 passing — no test changes needed (all tests use headless mode)
+
 ## 0.5.8
 
 Bot challenge detection for all browsing, not just assess.

package/README.md

@@ -100,12 +100,12 @@ For code examples, API reference, and wiring instructions, see **[barebrowse.con
 | Mode | What happens | Best for |
 |------|-------------|----------|
 | **Headless** (default) | Launches a fresh Chromium, no UI | Fast automation, scraping, reading pages |
-| **Headed** | Connects to your running browser on CDP port | Bot-detected sites, visual debugging, CAPTCHAs |
-| **Hybrid** | Tries headless first, falls back to headed if blocked | General-purpose agent browsing |
+| **Headed** | Auto-launches a visible Chromium window | Bot-detected sites, visual debugging, CAPTCHAs |
+| **Hybrid** | Tries headless first, auto-launches headed if blocked | General-purpose agent browsing |
 
 ## What it handles automatically
 
-Cookie consent walls (29 languages, with real mouse click fallback for stubborn CMPs), login walls (cookie extraction from your browsers), bot detection (stealth patches + automatic headed fallback on challenge pages, error pages, and near-empty responses), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
+Cookie consent walls (29 languages, with real mouse click fallback for stubborn CMPs), login walls (cookie extraction from your browsers), bot detection (ARIA node count heuristic + stealth patches + automatic headed fallback — snapshot shows `[BOT CHALLENGE DETECTED]` warning when blocked), permission prompts, SPA navigation, JS dialogs, off-screen elements, pre-filled inputs, ARIA noise, and profile locking. The agent doesn't think about any of it.
 
 ## What the agent sees
 

package/barebrowse.context.md

@@ -1,7 +1,7 @@
 # barebrowse -- Integration Guide
 
 > For AI assistants and developers wiring barebrowse into a project.
-> v0.5.8 | Node.js >= 22 | 0 required deps | MIT
+> v0.6.1 | Node.js >= 22 | 0 required deps | MIT
 
 ## What this is
 
@@ -23,10 +23,8 @@ Three integration paths:
 | Mode | What it does | When to use |
 |---|---|---|
 | `headless` (default) | Launches a fresh Chromium, no UI | Scraping, reading, fast automation |
-| `headed` | Connects to user's running browser on CDP port | Bot-detected sites, debugging, visual tasks |
-| `hybrid` | Tries headless first, falls back to headed if blocked | General-purpose agent browsing |
-
-Headed mode requires the browser to be launched with `--remote-debugging-port=9222`.
+| `headed` | Auto-launches a visible Chromium window | Bot-detected sites, debugging, visual tasks |
+| `hybrid` | Tries headless first, headed fallback per-navigation (switches back to headless next time) | General-purpose agent browsing |
 
 ## Minimal usage: one-shot browse
 
@@ -45,13 +43,12 @@ const snapshot = await browse('https://example.com', {
   pruneMode: 'act',      // 'act' (interactive elements) | 'read' (all content)
   consent: true,         // auto-dismiss cookie consent dialogs
   timeout: 30000,        // navigation timeout in ms
-  port: 9222,            // CDP port for headed/hybrid mode
 });
 ```
 
 ## connect() API
 
-`connect(opts)` returns a page handle for interactive sessions. Same opts as `browse()` for mode/port. Supports `hybrid` mode — starts headless, falls back to headed on bot detection (same as `browse()`).
+`connect(opts)` returns a page handle for interactive sessions. Same opts as `browse()` for mode. Supports `hybrid` mode — starts headless, auto-launches headed on bot detection (same as `browse()`).
 
 | Method | Args | Returns | Notes |
 |---|---|---|---|
@@ -76,9 +73,10 @@ const snapshot = await browse('https://example.com', {
 | `waitForNetworkIdle(opts?)` | { timeout?: number, idle?: number } | void | Wait until no pending requests for `idle` ms (default 500) |
 | `saveState(filePath)` | filePath: string | void | Export cookies + localStorage to JSON file |
 | `injectCookies(url, opts?)` | url: string, { browser?: string } | void | Extract cookies from user's browser and inject via CDP |
+| `botBlocked` | -- | boolean | True if last `goto()` hit a bot challenge (ARIA node count <50). Resets on each navigation. |
 | `dialogLog` | -- | Array<{type, message, timestamp}> | Auto-dismissed JS dialog history |
 | `cdp` | -- | object | Raw CDP session for escape hatch: `page.cdp.send(method, params)` |
-| `createTab()` | -- | tab handle | New tab in same browser. Returns `{ goto, injectCookies, waitForNetworkIdle, cdp, close }`. Tab close doesn't affect session. |
+| `createTab()` | -- | tab handle | New tab in same browser. Returns `{ goto, botBlocked, injectCookies, waitForNetworkIdle, cdp, close }`. Tab close doesn't affect session. |
 | `close()` | -- | void | Close page, disconnect CDP, kill browser (if headless) |
 
 **connect() options** (in addition to mode/port/consent):
@@ -154,7 +152,7 @@ barebrowse can inject cookies from the user's real browser sessions, bypassing l
 | Off-screen elements | `DOM.scrollIntoViewIfNeeded` before every click | Both |
 | Form submission | `press('Enter')` triggers onsubmit | Both |
 | SPA navigation | `waitForNavigation()` uses loadEventFired + frameNavigated | Both |
-| Bot detection | Hybrid fallback: detects challenge pages, error pages, and near-empty responses, then switches to headed | Hybrid |
+| Bot detection | ARIA node count (<50 = bot-blocked) + text heuristics. `botBlocked` flag set after every `goto()`. Hybrid fallback switches to headed. Snapshot shows `[BOT CHALLENGE DETECTED]` warning. | Hybrid |
 | `navigator.webdriver` | Stealth patches in headless (webdriver, plugins, chrome obj) | Headless |
 | Profile locking | Unique temp dir per headless instance | Headless |
 | Shared memory crash (Linux) | `--disable-dev-shm-usage` flag prevents `/dev/shm` exhaustion | Headless |
@@ -311,13 +309,13 @@ Useful for agent threshold decisions: "skip sites above score 40", "warn if term
 
 3. **Pruning modes matter.** `act` mode (default) keeps interactive elements + visible labels. `read` mode keeps all text content. Use `read` for content extraction, `act` for form filling and navigation.
 
-4. **Headed mode requires manual browser launch.** Start your browser with `--remote-debugging-port=9222`. barebrowse connects to it -- it does not launch it.
+4. **Headed mode auto-launches Chromium.** No need to start a browser manually — barebrowse launches a headed Chromium instance with CDP enabled automatically.
 
 5. **Cookie extraction needs unlocked profile.** Chromium cookies are AES-encrypted with a keyring key. If Chromium is running, the profile may be locked. Firefox cookies are plaintext and always accessible.
 
-6. **Hybrid mode kills and relaunches.** If headless is bot-blocked, hybrid mode kills the headless browser and connects to headed on port 9222. The headed browser must already be running.
+6. **Hybrid mode is per-navigation.** If headless is bot-blocked, hybrid kills headless and launches headed for that URL. On the next `goto()`, it switches back to headless automatically. If headed can't launch (no display — CI, Docker), it degrades gracefully with the headless result and a `[BOT CHALLENGE DETECTED]` warning.
 
-7. **One page per connect().** Each `connect()` call creates one page. For multiple tabs, call `connect()` multiple times.
+7. **One page per connect(), but tabs are supported.** Each `connect()` call creates one page. Use `createTab()` for additional tabs in the same browser.
 
 8. **Consent dismiss is best-effort.** It handles 16+ tested sites across 29 languages but novel consent implementations may need manual handling. Disable with `{ consent: false }`.
 

package/mcp-server.js

@@ -54,26 +54,18 @@ async function getPage() {
   }
 }
 
-// Concurrency limiter for assess — max 3 tabs at once
-const ASSESS_MAX = 3;
-let _assessRunning = 0;
-const _assessQueue = [];
+// Concurrency limiter — one assess at a time.
+// Headless tabs are fast, but headed fallback uses the user's single browser.
+// Running multiple headed navigations simultaneously hangs the browser.
+let _assessLock = Promise.resolve();
 
 function acquireAssessSlot() {
-  if (_assessRunning < ASSESS_MAX) {
-    _assessRunning++;
-    return Promise.resolve();
-  }
-  return new Promise((resolve) => _assessQueue.push(resolve));
+  let release;
+  const prev = _assessLock;
+  _assessLock = new Promise((r) => { release = r; });
+  return prev.then(() => release);
 }
 
-function releaseAssessSlot() {
-  if (_assessQueue.length > 0) {
-    _assessQueue.shift()();
-  } else {
-    _assessRunning--;
-  }
-}
 
 const TOOLS = [
   {
@@ -292,66 +284,50 @@ async function handleToolCall(name, args) {
     }
     case 'assess': {
       if (!assessFn) throw new Error('wearehere is not installed. Run: npm install wearehere');
-      await acquireAssessSlot();
+      const releaseSlot = await acquireAssessSlot();
       try {
-        const runAssess = async (headed) => {
-          let tab;
-          if (headed) {
-            tab = await connect({ mode: 'headed' });
-          } else {
-            const page = await getPage();
-            tab = await page.createTab();
-          }
-          let timer;
-          try {
-            const result = await Promise.race([
-              (async () => {
-                await tab.injectCookies(args.url).catch(() => {});
-                return await assessFn(tab, args.url, { timeout: args.timeout, settle: args.settle });
-              })(),
-              new Promise((_, reject) => {
-                timer = setTimeout(() => {
-                  tab.close().catch(() => {});
-                  reject(new Error('assess timeout'));
-                }, 30000);
-              }),
-            ]);
-            clearTimeout(timer);
-            const wasBotBlocked = tab.botBlocked;
-            await tab.close().catch(() => {});
-            return { result, botBlocked: wasBotBlocked };
-          } catch (err) {
-            clearTimeout(timer);
-            await tab.close().catch(() => {});
-            throw err;
-          }
-        };
-
-        // Try headless first
+        const page = await getPage();
+        const tab = await page.createTab();
+        let timer;
         try {
-          const { result, botBlocked } = await runAssess(false);
-          if (botBlocked) {
-            // Bot-blocked in headless — retry headed
+          await tab.injectCookies(args.url).catch(() => {});
+          const result = await Promise.race([
+            assessFn(tab, args.url, { timeout: args.timeout, settle: args.settle }),
+            new Promise((_, rej) => { timer = setTimeout(() => rej(new Error('assess timeout')), 30000); }),
+          ]);
+          clearTimeout(timer);
+          if (tab.botBlocked) {
+            // Bot-blocked — trigger hybrid fallback via main page, retry in new tab
+            await tab.close().catch(() => {});
+            await page.goto(args.url);
+            const tab2 = await page.createTab();
+            let timer2;
             try {
-              const headed = await runAssess(true);
-              return JSON.stringify(headed.result, null, 2);
-            } catch {
-              return JSON.stringify(result, null, 2); // headed failed, return headless result
+              await tab2.injectCookies(args.url).catch(() => {});
+              const r2 = await Promise.race([
+                assessFn(tab2, args.url, { timeout: args.timeout, settle: args.settle }),
+                new Promise((_, rej) => { timer2 = setTimeout(() => rej(new Error('assess timeout')), 30000); }),
+              ]);
+              clearTimeout(timer2);
+              if (tab2.botBlocked) r2._warning = 'Bot-blocked in both modes. Score may be unreliable.';
+              await tab2.close().catch(() => {});
+              return JSON.stringify(r2, null, 2);
+            } catch (err2) {
+              clearTimeout(timer2);
+              await tab2.close().catch(() => {});
+              throw err2;
             }
           }
+          await tab.close().catch(() => {});
           return JSON.stringify(result, null, 2);
         } catch (err) {
+          clearTimeout(timer);
+          await tab.close().catch(() => {});
           if (isCdpDead(err)) _page = null;
-          // Headless crashed — try headed
-          try {
-            const headed = await runAssess(true);
-            return JSON.stringify(headed.result, null, 2);
-          } catch (retryErr) {
-            throw retryErr;
-          }
+          throw err;
         }
       } finally {
-        releaseAssessSlot();
+        releaseSlot();
       }
     }
     default:
@@ -374,7 +350,7 @@ async function handleMessage(msg) {
     return jsonrpcResponse(id, {
       protocolVersion: '2024-11-05',
       capabilities: { tools: {} },
-      serverInfo: { name: 'barebrowse', version: '0.5.8' },
+      serverInfo: { name: 'barebrowse', version: '0.6.0' },
     });
   }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "barebrowse",
-  "version": "0.5.8",
+  "version": "0.6.1",
   "description": "Authenticated web browsing for autonomous agents via CDP. URL in, pruned ARIA snapshot out.",
   "type": "module",
   "main": "src/index.js",

package/src/chromium.js

@@ -2,7 +2,7 @@
  * chromium.js — Find, launch, and connect to Chromium-based browsers.
  *
  * Supports: Chrome, Chromium, Brave, Edge, Vivaldi, Arc, Opera.
- * Modes: headless (launch new), headed (connect to running).
+ * Modes: headless (launch new, no UI), headed (launch new, visible window).
  */
 
 import { execSync, spawn } from 'node:child_process';
@@ -55,11 +55,12 @@ export function findBrowser() {
 }
 
 /**
- * Launch a headless Chromium instance with CDP enabled.
+ * Launch a Chromium instance with CDP enabled.
  * @param {object} [opts]
  * @param {string} [opts.binary] - Path to browser binary (auto-detected if omitted)
  * @param {number} [opts.port=0] - CDP port (0 = random available port)
  * @param {string} [opts.userDataDir] - Browser profile directory
+ * @param {boolean} [opts.headed=false] - Launch in headed mode (with visible window)
  * @returns {Promise<{wsUrl: string, process: ChildProcess, port: number}>}
  */
 export async function launch(opts = {}) {
@@ -67,7 +68,6 @@ export async function launch(opts = {}) {
   const port = opts.port || 0;
 
   const args = [
-    '--headless=new',
     `--remote-debugging-port=${port}`,
     '--no-first-run',
     '--no-default-browser-check',
@@ -75,7 +75,8 @@ export async function launch(opts = {}) {
     '--disable-sync',
     '--disable-translate',
     '--mute-audio',
-    '--hide-scrollbars',
+    // Headless-only flags
+    ...(!opts.headed ? ['--headless=new', '--hide-scrollbars'] : []),
     // Suppress permission prompts (location, notifications, camera, mic, etc.)
     '--disable-notifications',
     '--autoplay-policy=no-user-gesture-required',

package/src/index.js

@@ -8,7 +8,7 @@
  *   const snapshot = await browse('https://example.com');
  */
 
-import { launch, getDebugUrl } from './chromium.js';
+import { launch } from './chromium.js';
 import { createCDP } from './cdp.js';
 import { formatTree } from './aria.js';
 import { authenticate } from './auth.js';
@@ -27,7 +27,6 @@ import { applyStealth } from './stealth.js';
  * @param {boolean} [opts.cookies=true] - Inject user's cookies (Phase 2)
  * @param {boolean} [opts.prune=true] - Apply ARIA pruning (Phase 2)
  * @param {number} [opts.timeout=30000] - Navigation timeout in ms
- * @param {number} [opts.port] - CDP port for headed mode
  * @returns {Promise<string>} ARIA snapshot text
  */
 export async function browse(url, opts = {}) {
@@ -40,9 +39,8 @@ export async function browse(url, opts = {}) {
   try {
     // Step 1: Get a CDP connection
     if (mode === 'headed') {
-      const port = opts.port || 9222;
-      const wsUrl = await getDebugUrl(port);
-      cdp = await createCDP(wsUrl);
+      browser = await launch({ headed: true, proxy: opts.proxy });
+      cdp = await createCDP(browser.wsUrl);
     } else {
       // headless or hybrid (start headless)
       browser = await launch({ proxy: opts.proxy });
@@ -81,17 +79,20 @@ export async function browse(url, opts = {}) {
       cdp.close();
       if (browser) { browser.process.kill(); browser = null; }
 
-      const port = opts.port || 9222;
-      const wsUrl = await getDebugUrl(port);
-      cdp = await createCDP(wsUrl);
-      page = await createPage(cdp, false, { viewport: opts.viewport });
-      await suppressPermissions(cdp);
-      if (opts.cookies !== false) {
-        try { await authenticate(page.session, url, { browser: opts.browser }); } catch {}
+      try {
+        browser = await launch({ headed: true, proxy: opts.proxy });
+        cdp = await createCDP(browser.wsUrl);
+        page = await createPage(cdp, false, { viewport: opts.viewport });
+        await suppressPermissions(cdp);
+        if (opts.cookies !== false) {
+          try { await authenticate(page.session, url, { browser: opts.browser }); } catch {}
+        }
+        await navigate(page, url, timeout);
+        if (opts.consent !== false) await dismissConsent(page.session);
+        ({ tree } = await ariaTree(page));
+      } catch {
+        // Headed launch failed (no display?) — return headless result as-is
       }
-      await navigate(page, url, timeout);
-      if (opts.consent !== false) await dismissConsent(page.session);
-      ({ tree } = await ariaTree(page));
     }
 
     // Step 6: Prune for agent consumption
@@ -121,7 +122,6 @@ export async function browse(url, opts = {}) {
  *
  * @param {object} [opts]
  * @param {'headless'|'headed'|'hybrid'} [opts.mode='headless'] - Browser mode
- * @param {number} [opts.port=9222] - CDP port for headed mode
  * @returns {Promise<object>} Page handle with goto, snapshot, close
  */
 export async function connect(opts = {}) {
@@ -130,15 +130,15 @@ export async function connect(opts = {}) {
   let cdp;
 
   if (mode === 'headed') {
-    const port = opts.port || 9222;
-    const wsUrl = await getDebugUrl(port);
-    cdp = await createCDP(wsUrl);
+    browser = await launch({ headed: true, proxy: opts.proxy });
+    cdp = await createCDP(browser.wsUrl);
   } else {
     browser = await launch({ proxy: opts.proxy });
     cdp = await createCDP(browser.wsUrl);
   }
 
-  let page = await createPage(cdp, mode !== 'headed', { viewport: opts.viewport });
+  let currentlyHeaded = (mode === 'headed');
+  let page = await createPage(cdp, !currentlyHeaded, { viewport: opts.viewport });
   let refMap = new Map();
   let botBlocked = false;
 
@@ -175,6 +175,20 @@ export async function connect(opts = {}) {
 
   return {
     async goto(url, timeout = 30000) {
+      // Switch back to headless if we fell back to headed previously
+      if (currentlyHeaded && mode === 'hybrid') {
+        await cdp.send('Target.closeTarget', { targetId: page.targetId });
+        cdp.close();
+        if (browser) { browser.process.kill(); browser = null; }
+
+        browser = await launch({ proxy: opts.proxy });
+        cdp = await createCDP(browser.wsUrl);
+        page = await createPage(cdp, true, { viewport: opts.viewport });
+        setupDialogHandler(page.session);
+        await suppressPermissions(cdp);
+        currentlyHeaded = false;
+      }
+
       await navigate(page, url, timeout);
       if (opts.consent !== false) {
         await dismissConsent(page.session);
@@ -190,18 +204,22 @@ export async function connect(opts = {}) {
         cdp.close();
         if (browser) { browser.process.kill(); browser = null; }
 
-        const port = opts.port || 9222;
-        const wsUrl = await getDebugUrl(port);
-        cdp = await createCDP(wsUrl);
-        page = await createPage(cdp, false, { viewport: opts.viewport });
-        setupDialogHandler(page.session);
-        await suppressPermissions(cdp);
-        await navigate(page, url, timeout);
-        if (opts.consent !== false) await dismissConsent(page.session);
-
-        // Re-check after headed fallback
-        const after = await ariaTree(page);
-        botBlocked = isChallengePage(after.tree, after.nodeCount);
+        try {
+          browser = await launch({ headed: true, proxy: opts.proxy });
+          cdp = await createCDP(browser.wsUrl);
+          page = await createPage(cdp, false, { viewport: opts.viewport });
+          setupDialogHandler(page.session);
+          await suppressPermissions(cdp);
+          await navigate(page, url, timeout);
+          if (opts.consent !== false) await dismissConsent(page.session);
+
+          // Re-check after headed fallback
+          const after = await ariaTree(page);
+          botBlocked = isChallengePage(after.tree, after.nodeCount);
+          currentlyHeaded = true;
+        } catch {
+          // Headed launch failed (no display?) — keep headless result, botBlocked stays true
+        }
       }
     },
 
@@ -375,7 +393,7 @@ export async function connect(opts = {}) {
     cdp: page.session,
 
     async createTab() {
-      const tab = await createPage(cdp, mode !== 'headed', { viewport: opts.viewport });
+      const tab = await createPage(cdp, !currentlyHeaded, { viewport: opts.viewport });
       await suppressPermissions(cdp);
       let tabBotBlocked = false;
       return {